Max level shown:
1 starting October 13, 2023 at 06:08:53 PM UTC
2 Creating downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.51920396
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.32296038
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.21638484
8 Region set with 3 regions was created
9 Starting downstairs
10 Using existing output directory "/tmp/dsc"
11 start ds: 8810
12 start ds: 8820
13 start ds: 8830
14 start access at:127.0.0.1:9998
15 Starting downstairs at port 8810
16 Starting downstairs at port 8820
17 Make output file at "/tmp/dsc/downstairs-8810.txt"
18 Starting downstairs at port 8830
19 Make output file at "/tmp/dsc/downstairs-8820.txt"
20 Make output file at "/tmp/dsc/downstairs-8830.txt"
21 [8810][0] reports Starting
22 [8820][1] reports Starting
23 [8830][2] reports Starting
24 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1166)
25 [8810][0] reports Running
26 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1167)
27 [8820][1] reports Running
28 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1168)
29 [8830][2] reports Running
30 Oct 13 18:08:54.904 INFO listening, local_addr: 127.0.0.1:9998
31 Control access at:127.0.0.1:9998
32 Running initial fill
332023-10-13T18:09:00.016ZINFOcrucible: Upstairs starts
342023-10-13T18:09:00.018ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "a730f3b26afaf115f27149ef849470fd85caef41", git_commit_timestamp: "2023-10-13T17:42:12.000000000Z", git_branch: "main", rustc_semver: "1.70.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "90c541806f23a127002de5b4038be731ba1458ca", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
352023-10-13T18:09:00.018ZINFOcrucible: Upstairs <-> Downstairs Message Version: 4
362023-10-13T18:09:00.018ZINFOcrucible: Upstairs opts: Upstairs UUID: c998ae83-bb6a-45d9-b08f-d75152575576, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false
372023-10-13T18:09:00.019ZINFOcrucible: Crucible stats registered with UUID: c998ae83-bb6a-45d9-b08f-d75152575576
382023-10-13T18:09:00.019ZINFOcrucible: Crucible c998ae83-bb6a-45d9-b08f-d75152575576 has session id: aa8d641f-a151-4743-ae77-a3f29be24015
39 Crucible runtime is spawned
40 {"msg":"up_listen starts","v":0,"name":"crucible","level":30The guest has requested activation with gen:1
41 ,"time":"2023-10-13T18:09:00.020976442Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1172,"task":"up_listen"}
422023-10-13T18:09:00.021ZINFOcrucible: Wait for all three downstairs to come online
432023-10-13T18:09:00.021ZINFOcrucible: Flush timeout: 0.5
442023-10-13T18:09:00.021ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
452023-10-13T18:09:00.021ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
462023-10-13T18:09:00.021ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
472023-10-13T18:09:00.021ZINFOcrucible: c998ae83-bb6a-45d9-b08f-d75152575576 active request set
482023-10-13T18:09:00.021ZINFOcrucible: Set desired generation to :1
492023-10-13T18:09:00.023ZINFOcrucible: [0] c998ae83-bb6a-45d9-b08f-d75152575576 looper connected looper = 0
502023-10-13T18:09:00.023ZINFOcrucible: [2] c998ae83-bb6a-45d9-b08f-d75152575576 looper connected looper = 2
512023-10-13T18:09:00.023ZINFOcrucible: [1] c998ae83-bb6a-45d9-b08f-d75152575576 looper connected looper = 1
522023-10-13T18:09:00.023ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state New
532023-10-13T18:09:00.023ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state New
542023-10-13T18:09:00.025ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state New
552023-10-13T18:09:00.028ZINFOcrucible: [0] c998ae83-bb6a-45d9-b08f-d75152575576 (68ce8e24-5067-4686-8c25-070ff31d6e4c) New New New ds_transition to WaitActive
562023-10-13T18:09:00.028ZINFOcrucible: [0] Transition from New to WaitActive
572023-10-13T18:09:00.028ZINFOcrucible: [1] c998ae83-bb6a-45d9-b08f-d75152575576 (68ce8e24-5067-4686-8c25-070ff31d6e4c) WaitActive New New ds_transition to WaitActive
582023-10-13T18:09:00.028ZINFOcrucible: [1] Transition from New to WaitActive
592023-10-13T18:09:00.028ZINFOcrucible: [2] c998ae83-bb6a-45d9-b08f-d75152575576 (68ce8e24-5067-4686-8c25-070ff31d6e4c) WaitActive WaitActive New ds_transition to WaitActive
602023-10-13T18:09:00.028ZINFOcrucible: [2] Transition from New to WaitActive
612023-10-13T18:09:00.028ZINFOcrucible: [0] client is_active_req TRUE, promote! session 68ce8e24-5067-4686-8c25-070ff31d6e4c
622023-10-13T18:09:00.028ZINFOcrucible: [1] client is_active_req TRUE, promote! session 68ce8e24-5067-4686-8c25-070ff31d6e4c
632023-10-13T18:09:00.028ZINFOcrucible: [2] client is_active_req TRUE, promote! session 68ce8e24-5067-4686-8c25-070ff31d6e4c
642023-10-13T18:09:00.032ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
652023-10-13T18:09:00.032ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
662023-10-13T18:09:00.032ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
672023-10-13T18:09:00.032ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
682023-10-13T18:09:00.032ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
692023-10-13T18:09:00.032ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
702023-10-13T18:09:00.033ZINFOcrucible: c998ae83-bb6a-45d9-b08f-d75152575576 WaitActive WaitActive WaitActive
712023-10-13T18:09:00.033ZINFOcrucible: c998ae83-bb6a-45d9-b08f-d75152575576 WaitActive WaitActive WaitActive
722023-10-13T18:09:00.033ZINFOcrucible: c998ae83-bb6a-45d9-b08f-d75152575576 WaitActive WaitActive WaitActive
732023-10-13T18:09:00.035ZINFOcrucible: [0] c998ae83-bb6a-45d9-b08f-d75152575576 (68ce8e24-5067-4686-8c25-070ff31d6e4c) WaitActive WaitActive WaitActive ds_transition to WaitQuorum
742023-10-13T18:09:00.035ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum
752023-10-13T18:09:00.036ZWARNcrucible: [0] new RM replaced this: None
762023-10-13T18:09:00.036ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
772023-10-13T18:09:00.036ZINFOcrucible: [2] c998ae83-bb6a-45d9-b08f-d75152575576 (68ce8e24-5067-4686-8c25-070ff31d6e4c) WaitQuorum WaitActive WaitActive ds_transition to WaitQuorum
782023-10-13T18:09:00.036ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum
792023-10-13T18:09:00.036ZWARNcrucible: [2] new RM replaced this: None
802023-10-13T18:09:00.036ZINFOcrucible: c998ae83-bb6a-45d9-b08f-d75152575576 WaitQuorum WaitActive WaitQuorum
812023-10-13T18:09:00.036ZINFOcrucible: [1] c998ae83-bb6a-45d9-b08f-d75152575576 (68ce8e24-5067-4686-8c25-070ff31d6e4c) WaitQuorum WaitActive WaitQuorum ds_transition to WaitQuorum
822023-10-13T18:09:00.036ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum
832023-10-13T18:09:00.037ZWARNcrucible: [1] new RM replaced this: None
842023-10-13T18:09:00.037ZINFOcrucible: [0] Starts reconcile loop
852023-10-13T18:09:00.037ZINFOcrucible: [2] Starts reconcile loop
862023-10-13T18:09:00.037ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
872023-10-13T18:09:00.037ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
882023-10-13T18:09:00.037ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
892023-10-13T18:09:00.037ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
902023-10-13T18:09:00.037ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
912023-10-13T18:09:00.037ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
922023-10-13T18:09:00.037ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
932023-10-13T18:09:00.037ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
942023-10-13T18:09:00.037ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
952023-10-13T18:09:00.037ZINFOcrucible: Max found gen is 1
962023-10-13T18:09:00.037ZINFOcrucible: Generation requested: 1 >= found:1
972023-10-13T18:09:00.037ZINFOcrucible: Next flush: 1
982023-10-13T18:09:00.038ZINFOcrucible: All extents match
992023-10-13T18:09:00.038ZINFOcrucible: No downstairs repair required
1002023-10-13T18:09:00.038ZINFOcrucible: [1] Starts reconcile loop
1012023-10-13T18:09:00.038ZINFOcrucible: No initial repair work was required
1022023-10-13T18:09:00.038ZINFOcrucible: Set Downstairs and Upstairs active
103 {"msg":"The guest has finished waiting for activation with:1
104 c998ae83-bb6a-45d9-b08f-d75152575576 is now active with session: 68ce8e24-5067-4686-8c25-070ff31d6e4c","v":Activate successful
105 0Wait for a query_work_queue command to finish before sending IO
106 ,"name":"crucible","level":30,"time":"2023-10-13T18:09:00.038939948Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1172}
1072023-10-13T18:09:00.039ZINFOcrucible: c998ae83-bb6a-45d9-b08f-d75152575576 Set Active after no repair
1082023-10-13T18:09:00.039ZINFOcrucible: Notify all downstairs, region set compare is done.
1092023-10-13T18:09:00.039ZINFOcrucible: Set check for repair
1102023-10-13T18:09:00.039ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
1112023-10-13T18:09:00.039ZINFOcrucible: c998ae83-bb6a-45d9-b08f-d75152575576 Active Active Active
1122023-10-13T18:09:00.039ZINFOcrucible: Set check for repair
1132023-10-13T18:09:00.039ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
1142023-10-13T18:09:00.039ZINFOcrucible: c998ae83-bb6a-45d9-b08f-d75152575576 Active Active Active
1152023-10-13T18:09:00.039ZINFOcrucible: Set check for repair
1162023-10-13T18:09:00.039ZINFOcrucible: [0] received reconcile message
1172023-10-13T18:09:00.039ZINFOcrucible: [0] All repairs completed, exit
1182023-10-13T18:09:00.039ZINFOcrucible: [0] Starts cmd_loop
1192023-10-13T18:09:00.039ZINFOcrucible: [1] received reconcile message
1202023-10-13T18:09:00.039ZINFOcrucible: [2] received reconcile message
1212023-10-13T18:09:00.039ZINFOcrucible: [1] All repairs completed, exit
1222023-10-13T18:09:00.039ZINFOcrucible: [1] Starts cmd_loop
1232023-10-13T18:09:00.040ZINFOcrucible: [2] All repairs completed, exit
1242023-10-13T18:09:00.040ZINFOcrucible: [2] Starts cmd_loop
125 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
126 Fill test
127 Read and Verify all blocks (0..5000 range:false)
1282023-10-13T18:09:01.044ZINFOcrucible: Checking if live repair is needed
1292023-10-13T18:09:01.044ZINFOcrucible: No Live Repair required at this time
130 Wrote out file "alan"
131 CLIENT: Tests done. All submitted work has been ACK'd
132 ----------------------------------------------------------------
133 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:10
134 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
135 97 Acked 1096 Read 100 Done Done Done false
136 98 Acked 1097 Read 100 Done Done Done false
137 99 Acked 1098 Read 100 Done Done Done false
138 100 Acked 1099 Read 100 Done Done Done false
139 101 Acked 1100 Read 100 Done Done Done false
140 102 Acked 1101 Read 100 Done Done Done false
141 103 Acked 1102 Read 100 Done Done Done false
142 104 Acked 1103 Read 100 Done Done Done false
143 105 Acked 1104 Read 100 Done Done Done false
144 106 Acked 1105 Read 100 Done Done Done false
145 STATES DS:0 DS:1 DS:2 TOTAL
146 New 0 0 0 0
147 Sent 0 0 0 0
148 Done 10 10 10 30
149 Skipped 0 0 0 0
150 Error 0 0 0 0
151 Last Flush: 1095 1095 1095
152 Downstairs last five completed: 1095 1094 1093 1092 1091 1090
153 Upstairs last five completed: 106 105 104 103 102 101
154 CLIENT: Up:0 ds:10 act:3
155 ----------------------------------------------------------------
156 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
157 Downstairs last five completed: 1106 1105 1104 1103 1102 1101
158 Upstairs last five completed: 107 106 105 104 103 102
159 CLIENT: Up:0 ds:0 act:3
160 CLIENT: All crucible jobs finished, exiting program
161 Replay loop starts now October 13, 2023 at 06:09:07 PM UTC
1622023-10-13T18:09:07.034ZINFOcrucible: Upstairs starts
1632023-10-13T18:09:07.035ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "a730f3b26afaf115f27149ef849470fd85caef41", git_commit_timestamp: "2023-10-13T17:42:12.000000000Z", git_branch: "main", rustc_semver: "1.70.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "90c541806f23a127002de5b4038be731ba1458ca", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1642023-10-13T18:09:07.035ZINFOcrucible: Upstairs <-> Downstairs Message Version: 4
1652023-10-13T18:09:07.035ZINFOcrucible: Upstairs opts: Upstairs UUID: fe698f06-923e-45d4-b592-2bcd1ce0484c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false
1662023-10-13T18:09:07.035ZINFOcrucible: Crucible stats registered with UUID: fe698f06-923e-45d4-b592-2bcd1ce0484c
1672023-10-13T18:09:07.035ZINFOcrucible: Crucible fe698f06-923e-45d4-b592-2bcd1ce0484c has session id: de0bd0e5-3d48-4ee0-8ee7-40554c4a5e92
168 Crucible runtime is spawned
169 The guest has requested activation with gen:2
1702023-10-13T18:09:07.035ZINFOcrucible: up_listen starts task = up_listen
1712023-10-13T18:09:07.035ZINFOcrucible: Wait for all three downstairs to come online
1722023-10-13T18:09:07.035ZINFOcrucible: Flush timeout: 0.5
1732023-10-13T18:09:07.035ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
1742023-10-13T18:09:07.036ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
1752023-10-13T18:09:07.036ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 1
1762023-10-13T18:09:07.036ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
1772023-10-13T18:09:07.036ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 0
1782023-10-13T18:09:07.036ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 2
1792023-10-13T18:09:07.036ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c active request set
1802023-10-13T18:09:07.036ZINFOcrucible: Set desired generation to :2
1812023-10-13T18:09:07.037ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state New
1822023-10-13T18:09:07.037ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state New
1832023-10-13T18:09:07.037ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state New
1842023-10-13T18:09:07.037ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) New New New ds_transition to WaitActive
1852023-10-13T18:09:07.037ZINFOcrucible: [0] Transition from New to WaitActive
1862023-10-13T18:09:07.038ZINFOcrucible: [0] client is_active_req TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
1872023-10-13T18:09:07.038ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) WaitActive New New ds_transition to WaitActive
1882023-10-13T18:09:07.038ZINFOcrucible: [1] Transition from New to WaitActive
1892023-10-13T18:09:07.038ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) WaitActive WaitActive New ds_transition to WaitActive
1902023-10-13T18:09:07.038ZINFOcrucible: [2] Transition from New to WaitActive
1912023-10-13T18:09:07.038ZINFOcrucible: [1] client is_active_req TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
1922023-10-13T18:09:07.038ZINFOcrucible: [2] client is_active_req TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
1932023-10-13T18:09:07.038ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
1942023-10-13T18:09:07.038ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
1952023-10-13T18:09:07.038ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c WaitActive WaitActive WaitActive
1962023-10-13T18:09:07.039ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
1972023-10-13T18:09:07.039ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
1982023-10-13T18:09:07.039ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
1992023-10-13T18:09:07.039ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
2002023-10-13T18:09:07.039ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c WaitActive WaitActive WaitActive
2012023-10-13T18:09:07.039ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c WaitActive WaitActive WaitActive
2022023-10-13T18:09:07.040ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) WaitActive WaitActive WaitActive ds_transition to WaitQuorum
2032023-10-13T18:09:07.040ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum
2042023-10-13T18:09:07.040ZWARNcrucible: [0] new RM replaced this: None
2052023-10-13T18:09:07.040ZINFOcrucible: [0] Starts reconcile loop
2062023-10-13T18:09:07.040ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
2072023-10-13T18:09:07.041ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c WaitQuorum WaitActive WaitActive
2082023-10-13T18:09:07.041ZINFOcrucible: Waiting for 2 more clients to be ready
2092023-10-13T18:09:07.041ZINFOcrucible: Set check for repair
2102023-10-13T18:09:07.041ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) WaitQuorum WaitActive WaitActive ds_transition to WaitQuorum
2112023-10-13T18:09:07.041ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum
2122023-10-13T18:09:07.041ZWARNcrucible: [2] new RM replaced this: None
2132023-10-13T18:09:07.041ZINFOcrucible: [2] Starts reconcile loop
2142023-10-13T18:09:07.041ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
2152023-10-13T18:09:07.041ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c WaitQuorum WaitActive WaitQuorum
2162023-10-13T18:09:07.041ZINFOcrucible: Waiting for 1 more clients to be ready
2172023-10-13T18:09:07.041ZINFOcrucible: Set check for repair
2182023-10-13T18:09:07.041ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) WaitQuorum WaitActive WaitQuorum ds_transition to WaitQuorum
2192023-10-13T18:09:07.041ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum
2202023-10-13T18:09:07.041ZWARNcrucible: [1] new RM replaced this: None
2212023-10-13T18:09:07.041ZINFOcrucible: [1] Starts reconcile loop
2222023-10-13T18:09:07.041ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
2232023-10-13T18:09:07.041ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c WaitQuorum WaitQuorum WaitQuorum
2242023-10-13T18:09:07.041ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
2252023-10-13T18:09:07.041ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
2262023-10-13T18:09:07.041ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
2272023-10-13T18:09:07.041ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
2282023-10-13T18:09:07.041ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
2292023-10-13T18:09:07.041ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
2302023-10-13T18:09:07.041ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
2312023-10-13T18:09:07.041ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
2322023-10-13T18:09:07.041ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
233 {"msg":"Max found gen is 2","v":0,"name":"crucible","level":The guest has finished waiting for activation with:2
234 30Activate successful
235 Wait for a query_work_queue command to finish before sending IO
236 ,"time":"2023-10-13T18:09:07.041634505Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1176}
2372023-10-13T18:09:07.041ZINFOcrucible: Generation requested: 2 >= found:2
2382023-10-13T18:09:07.041ZINFOcrucible: Next flush: 3
2392023-10-13T18:09:07.041ZINFOcrucible: All extents match
2402023-10-13T18:09:07.041ZINFOcrucible: No downstairs repair required
2412023-10-13T18:09:07.041ZINFOcrucible: No initial repair work was required
2422023-10-13T18:09:07.041ZINFOcrucible: Set Downstairs and Upstairs active
2432023-10-13T18:09:07.041ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c is now active with session: c526496c-9bef-4440-8022-c243881533f9
2442023-10-13T18:09:07.041ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Set Active after no repair
2452023-10-13T18:09:07.041ZINFOcrucible: Notify all downstairs, region set compare is done.
2462023-10-13T18:09:07.041ZINFOcrucible: Set check for repair
2472023-10-13T18:09:07.041ZINFOcrucible: [0] received reconcile message
2482023-10-13T18:09:07.041ZINFOcrucible: [1] received reconcile message
2492023-10-13T18:09:07.041ZINFOcrucible: [2] received reconcile message
2502023-10-13T18:09:07.042ZINFOcrucible: [0] All repairs completed, exit
2512023-10-13T18:09:07.042ZINFOcrucible: [0] Starts cmd_loop
2522023-10-13T18:09:07.042ZINFOcrucible: [1] All repairs completed, exit
2532023-10-13T18:09:07.042ZINFOcrucible: [1] Starts cmd_loop
2542023-10-13T18:09:07.042ZINFOcrucible: [2] All repairs completed, exit
2552023-10-13T18:09:07.042ZINFOcrucible: [2] Starts cmd_loop
256 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
257 Loading write count information from file "alan"
258 Oct 13 18:09:07.134 INFO accepted connection, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
259 Main task has work to do, go find it
260 got dsc Stop(2)
261 stop 2
262 Oct 13 18:09:07.137 INFO request completed, latency_us: 704, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 5529ace6-0ac0-434c-ad12-224db2105f38[8830] Got stop action so:false kr:false
263 , remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
2642023-10-13T18:09:07.138ZWARNcrucible: [2] downstairs disconnected
2652023-10-13T18:09:07.138ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
2662023-10-13T18:09:07.138ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8830 closed looper = 2
2672023-10-13T18:09:07.138ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
2682023-10-13T18:09:07.138ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
2692023-10-13T18:09:07.138ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
2702023-10-13T18:09:07.139ZWARNcrucible: [2] pm_task rx.recv() is None
271 Oct 13 18:09:07.139 INFO request completed, latency_us: 409, response_code: 200, uri: /state/cid/2, method: GET, req_id: 52008710-9efb-4c3a-81d4-5d308ea622d1, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
272 [8830] Exited with: ExitStatus(unix_wait_status(9))
273 [8830][2] reports Exit
2742023-10-13T18:09:08.042ZINFOcrucible: Checking if live repair is needed
2752023-10-13T18:09:08.042ZINFOcrucible: No Live Repair required at this time
276 Oct 13 18:09:11.142 INFO request completed, latency_us: 258, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5b1b6bdf-710d-4517-b188-00b9fa32e177, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
277 001/300 Write block 2885 len 8192 data: 2 2
278 002/300 Write block 2835 len 12288 data: 2 2 2
279 003/300 Read block 51 len 4096
280 004/300 Read block 4434 len 28672
281 005/300 Write block 2539 len 16384 data: 2 2 2 2
282 006/300 Write block 2885 len 16384 data: 3 3 2 2
283 007/300 Read block 4987 len 20480
284 008/300 Read block 2471 len 32768
285 009/300 Write block 2216 len 28672 data: 2 2 2 2 2 2 2
286 010/300 Write block 2553 len 8192 data: 2 2
287 011/300 Write block 3173 len 28672 data: 2 2 2 2 2 2 2
288 012/300 Read block 1685 len 16384
289 013/300 Write block 1646 len 36864 data: 2 2 2 2 2 2 2 2 2
290 014/300 Write block 2327 len 24576 data: 2 2 2 2 2 2
291 015/300 Read block 2191 len 36864
292 016/300 Write block 1995 len 12288 data: 2 2 2
293 017/300 Write block 4379 len 4096 data: 2
294 018/300 Read block 594 len 24576
295 019/300 Write block 190 len 20480 data: 2 2 2 2 2
296 020/300 Write block 4229 len 12288 data: 2 2 2
297 021/300 Write block 610 len 4096 data: 2
298 022/300 Read block 1887 len 24576
299 023/300 Read block 1667 len 20480
300 024/300 Read block 3706 len 40960
301 025/300 Write block 417 len 20480 data: 2 2 2 2 2
302 026/300 Read block 1168 len 16384
303 027/300 Read block 2165 len 28672
304 028/300 Write block 2566 len 28672 data: 2 2 2 2 2 2 2
305 029/300 Read block 932 len 16384
306 030/300 Write block 2465 len 36864 data: 2 2 2 2 2 2 2 2 2
307 031/300 Write block 1735 len 40960 data: 2 2 2 2 2 2 2 2 2 2
308 032/300 Flush
309 033/300 Write block 3208 len 40960 data: 2 2 2 2 2 2 2 2 2 2
310 034/300 Write block 2056 len 8192 data: 2 2
311 035/300 Read block 3581 len 20480
312 036/300 Read block 3909 len 32768
313 037/300 Write block 1208 len 20480 data: 2 2 2 2 2
314 038/300 Write block 2396 len 40960 data: 2 2 2 2 2 2 2 2 2 2
315 039/300 Read block 2310 len 20480
316 040/300 Read block 3552 len 28672
317 041/300 Write block 1116 len 36864 data: 2 2 2 2 2 2 2 2 2
318 042/300 Flush
319 043/300 Write block 3710 len 8192 data: 2 2
320 044/300 Write block 2644 len 4096 data: 2
321 045/300 Read block 2214 len 40960
322 046/300 Read block 850 len 4096
323 047/300 Write block 3440 len 8192 data: 2 2
324 048/300 Read block 3263 len 28672
325 049/300 Read block 4906 len 24576
326 050/300 Write block 1095 len 16384 data: 2 2 2 2
327 051/300 Read block 597 len 28672
328 052/300 Write block 2057 len 16384 data: 3 2 2 2
329 053/300 Write block 2088 len 28672 data: 2 2 2 2 2 2 2
330 054/300 Read block 1942 len 4096
331 055/300 Write block 2397 len 16384 data: 3 3 3 3
332 056/300 Read block 1037 len 12288
333 057/300 Read block 2589 len 8192
334 058/300 Write block 2501 len 32768 data: 2 2 2 2 2 2 2 2
335 059/300 Read block 2629 len 40960
336 060/300 Write block 3505 len 28672 data: 2 2 2 2 2 2 2
337 061/300 Write block 2531 len 36864 data: 2 2 2 2 2 2 2 2 3
338 062/300 Write block 2440 len 28672 data: 2 2 2 2 2 2 2
339 063/300 Write block 3644 len 40960 data: 2 2 2 2 2 2 2 2 2 2
340 064/300 Write block 3870 len 8192 data: 2 2
341 065/300 Write block 3666 len 32768 data: 2 2 2 2 2 2 2 2
342 066/300 Read block 3692 len 4096
343 067/300 Write block 2485 len 36864 data: 2 2 2 2 2 2 2 2 2
344 068/300 Read block 558 len 4096
345 069/300 Read block 4675 len 24576
346 070/300 Read block 127 len 12288
347 071/300 Flush
348 072/300 Read block 979 len 40960
349 073/300 Write block 2306 len 24576 data: 2 2 2 2 2 2
350 074/300 Flush
351 075/300 Read block 1686 len 4096
352 076/300 Write block 321 len 40960 data: 2 2 2 2 2 2 2 2 2 2
353 077/300 Read block 3082 len 8192
354 078/300 Write block 3752 len 20480 data: 2 2 2 2 2
355 079/300 Write block 3906 len 40960 data: 2 2 2 2 2 2 2 2 2 2
356 080/300 Flush
357 081/300 Flush
358 082/300 Read block 3868 len 24576
359 083/300 Flush
360 084/300 Write block 2211 len 4096 data: 2
361 085/300 Flush
362 086/300 Flush
363 087/300 Write block 658 len 4096 data: 2
364 088/300 Flush
365 089/300 Flush
366 090/300 Write block 4822 len 32768 data: 2 2 2 2 2 2 2 2
367 091/300 Read block 1996 len 12288
368 092/300 Read block 3790 len 24576
369 093/300 Read block 1120 len 40960
370 094/300 Read block 4682 len 8192
371 095/300 Read block 4211 len 28672
372 096/300 Flush
373 097/300 Read block 3885 len 32768
374 098/300 Write block 1383 len 36864 data: 2 2 2 2 2 2 2 2 2
375 099/300 Flush
376 100/300 Write block 666 len 4096 data: 2
377 101/300 Read block 705 len 12288
378 102/300 Read block 2878 len 32768
379 103/300 Read block 2140 len 20480
380 104/300 Write block 2007 len 24576 data: 2 2 2 2 2 2
381 105/300 Read block 3225 len 4096
382 106/300 Write block 2937 len 20480 data: 2 2 2 2 2
383 107/300 Read block 4666 len 36864
384 108/300 Read block 1500 len 4096
385 109/300 Read block 2139 len 12288
386 110/300 Read block 294 len 20480
387 111/300 Read block 46 len 24576
388 112/300 Write block 3095 len 4096 data: 2
389 113/300 Write block 628 len 12288 data: 2 2 2
390 114/300 Read block 4643 len 28672
391 115/300 Write block 2279 len 16384 data: 2 2 2 2
392 116/300 Write block 3741 len 16384 data: 2 2 2 2
393 117/300 Write block 4188 len 40960 data: 2 2 2 2 2 2 2 2 2 2
394 118/300 Write block 23 len 28672 data: 2 2 2 2 2 2 2
395 119/300 Flush
396 120/300 Read block 243 len 12288
397 121/300 Read block 870 len 4096
398 122/300 Read block 2448 len 24576
399 123/300 Read block 3961 len 4096
400 124/300 Write block 223 len 32768 data: 2 2 2 2 2 2 2 2
401 125/300 Write block 2715 len 28672 data: 2 2 2 2 2 2 2
402 126/300 Read block 2467 len 20480
403 127/300 Read block 3962 len 40960
404 128/300 Flush
405 129/300 Read block 4389 len 24576
406 130/300 Read block 736 len 36864
407 131/300 Read block 520 len 4096
408 132/300 Read block 417 len 32768
409 133/300 Write block 4711 len 28672 data: 2 2 2 2 2 2 2
410 134/300 Write block 1931 len 16384 data: 2 2 2 2
411 135/300 Write block 170 len 12288 data: 2 2 2
412 136/300 Read block 1432 len 32768
413 137/300 Write block 2705 len 16384 data: 2 2 2 2
414 138/300 Flush
415 139/300 Read block 4585 len 8192
416 140/300 Write block 3533 len 40960 data: 2 2 2 2 2 2 2 2 2 2
417 141/300 Write block 1989 len 24576 data: 2 2 2 2 2 2
418 142/300 Write block 297 len 20480 data: 2 2 2 2 2
419 143/300 Write block 2316 len 12288 data: 2 2 2
420 144/300 Flush
421 145/300 Read block 1093 len 8192
422 146/300 Read block 1209 len 20480
423 147/300 Write block 3519 len 24576 data: 2 2 2 2 2 2
424 148/300 Read block 3454 len 20480
425 149/300 Write block 4200 len 36864 data: 2 2 2 2 2 2 2 2 2
426 150/300 Read block 1396 len 24576
427 151/300 Flush
428 152/300 Write block 4617 len 4096 data: 2
429 153/300 Read block 4587 len 28672
430 154/300 Read block 4313 len 20480
431 155/300 Write block 3908 len 4096 data: 3
432 156/300 Read block 1872 len 16384
433 157/300 Write block 2209 len 20480 data: 2 2 3 2 2
434 158/300 Write block 319 len 8192 data: 2 2
435 159/300 Read block 4118 len 12288
436 160/300 Read block 4782 len 16384
437 161/300 Read block 4983 len 28672
438 162/300 Read block 4055 len 36864
439 163/300 Write block 218 len 12288 data: 2 2 2
440 164/300 Read block 895 len 4096
441 165/300 Read block 1192 len 32768
442 166/300 Write block 2808 len 36864 data: 2 2 2 2 2 2 2 2 2
443 167/300 Write block 1376 len 24576 data: 2 2 2 2 2 2
444 168/300 Read block 3040 len 24576
445 169/300 Read block 3248 len 40960
446 170/300 Write block 150 len 20480 data: 2 2 2 2 2
447 171/300 Read block 95 len 28672
448 172/300 Flush
449 173/300 Write block 1974 len 12288 data: 2 2 2
450 174/300 Write block 3911 len 28672 data: 3 3 3 3 3 2 2
451 175/300 Read block 2044 len 4096
452 176/300 Read block 4382 len 36864
453 177/300 Read block 4513 len 32768
454 178/300 Read block 371 len 24576
455 179/300 Flush
456 180/300 Read block 3300 len 40960
457 181/300 Read block 1382 len 32768
458 182/300 Read block 4877 len 40960
459 183/300 Read block 4693 len 20480
460 184/300 Write block 4800 len 36864 data: 2 2 2 2 2 2 2 2 2
461 185/300 Read block 3315 len 20480
462 186/300 Write block 1706 len 16384 data: 2 2 2 2
463 187/300 Write block 2210 len 28672 data: 3 4 3 3 2 2 3
464 188/300 Write block 2367 len 20480 data: 2 2 2 2 2
465 189/300 Read block 2950 len 24576
466 190/300 Read block 4751 len 32768
467 191/300 Read block 2675 len 24576
468 192/300 Write block 1433 len 8192 data: 2 2
469 193/300 Read block 4144 len 16384
470 194/300 Flush
471 195/300 Flush
472 196/300 Write block 2760 len 40960 data: 2 2 2 2 2 2 2 2 2 2
473 197/300 Write block 4755 len 32768 data: 2 2 2 2 2 2 2 2
474 198/300 Write block 2492 len 32768 data: 3 3 2 2 2 2 2 2
475 199/300 Write block 6 len 28672 data: 2 2 2 2 2 2 2
476 200/300 Read block 4491 len 12288
477 201/300 Read block 4375 len 32768
478 202/300 Write block 3406 len 36864 data: 2 2 2 2 2 2 2 2 2
479 203/300 Read block 2244 len 28672
480 204/300 Read block 1483 len 24576
481 205/300 Read block 3806 len 4096
482 206/300 Write block 3954 len 28672 data: 2 2 2 2 2 2 2
483 207/300 Write block 3714 len 36864 data: 2 2 2 2 2 2 2 2 2
484 208/300 Read block 3981 len 20480
485 209/300 Read block 1580 len 4096
486 210/300 Write block 3762 len 16384 data: 2 2 2 2
487 211/300 Read block 3943 len 36864
488 212/300 Read block 3999 len 4096
489 213/300 Flush
490 214/300 Flush
491 215/300 Write block 600 len 36864 data: 2 2 2 2 2 2 2 2 2
492 216/300 Read block 3622 len 16384
493 217/300 Read block 3586 len 8192
494 218/300 Read block 3380 len 32768
495 219/300 Write block 4800 len 20480 data: 3 3 3 3 3
496 220/300 Read block 1700 len 4096
497 221/300 Read block 1947 len 4096
498 222/300 Write block 773 len 28672 data: 2 2 2 2 2 2 2
499 223/300 Read block 1559 len 16384
500 224/300 Read block 853 len 28672
501 225/300 Read block 657 len 20480
502 226/300 Read block 1462 len 24576
503 227/300 Write block 4101 len 32768 data: 2 2 2 2 2 2 2 2
504 228/300 Read block 1683 len 4096
505 229/300 Read block 4780 len 20480
506 230/300 Read block 2840 len 40960
507 231/300 Write block 1883 len 40960 data: 2 2 2 2 2 2 2 2 2 2
508 232/300 Read block 3052 len 12288
509 233/300 Read block 1973 len 20480
510 234/300 Read block 216 len 24576
511 235/300 Read block 817 len 20480
512 236/300 Read block 448 len 4096
513 237/300 Read block 3425 len 4096
514 238/300 Write block 484 len 20480 data: 2 2 2 2 2
515 239/300 Write block 1947 len 32768 data: 2 2 2 2 2 2 2 2
516 240/300 Write block 4263 len 8192 data: 2 2
517 241/300 Write block 3692 len 32768 data: 2 2 2 2 2 2 2 2
518 242/300 Flush
519 243/300 Read block 4183 len 32768
520 244/300 Write block 705 len 8192 data: 2 2
521 245/300 Read block 778 len 24576
522 246/300 Flush
523 247/300 Read block 1813 len 28672
524 248/300 Write block 2585 len 8192 data: 2 2
525 249/300 Write block 3260 len 12288 data: 2 2 2
526 250/300 Read block 1667 len 28672
527 251/300 Flush
528 252/300 Read block 360 len 20480
529 253/300 Write block 1476 len 16384 data: 2 2 2 2
530 254/300 Read block 4111 len 4096
531 255/300 Read block 1729 len 4096
532 256/300 Write block 1367 len 40960 data: 2 2 2 2 2 2 2 2 2 3
533 257/300 Write block 3624 len 24576 data: 2 2 2 2 2 2
534 258/300 Read block 2254 len 4096
535 259/300 Read block 817 len 4096
536 260/300 Read block 1580 len 36864
537 261/300 Write block 3028 len 36864 data: 2 2 2 2 2 2 2 2 2
538 262/300 Write block 4528 len 4096 data: 2
539 263/300 Read block 2948 len 20480
540 264/300 Flush
541 265/300 Write block 3205 len 16384 data: 2 2 2 3
542 266/300 Write block 3453 len 32768 data: 2 2 2 2 2 2 2 2
543 267/300 Write block 3383 len 12288 data: 2 2 2
544 268/300 Read block 2618 len 4096
545 269/300 Write block 1099 len 28672 data: 2 2 2 2 2 2 2
546 270/300 Read block 506 len 32768
547 271/300 Read block 8 len 36864
548 272/300 Flush
549 273/300 Write block 1012 len 4096 data: 2
550 274/300 Read block 863 len 4096
551 275/300 Flush
552 276/300 Write block 2553 len 16384 data: 3 3 2 2
553 277/300 Read block 1257 len 12288
554 278/300 Read block 3004 len 32768
555 279/300 Flush
556 280/300 Read block 1704 len 28672
557 281/300 Read block 1267 len 24576
558 282/300 Read block 740 len 20480
559 283/300 Read block 1474 len 36864
560 284/300 Write block 2610 len 32768 data: 2 2 2 2 2 2 2 2
561 285/300 Read block 4353 len 20480
562 286/300 Flush
563 287/300 Write block 4941 len 28672 data: 2 2 2 2 2 2 2
564 288/300 Read block 454 len 12288
565 289/300 Write block 2761 len 20480 data: 3 3 3 3 3
566 290/300 Flush
567 291/300 Write block 4838 len 24576 data: 2 2 2 2 2 2
568 292/300 Read block 3708 len 20480
569 293/300 Read block 266 len 4096
570 294/300 Read block 2338 len 16384
571 295/300 Write block 3836 len 32768 data: 2 2 2 2 2 2 2 2
572 296/300 Write block 2146 len 8192 data: 2 2
573 297/300 Write block 3934 len 24576 data: 2 2 2 2 2 2
574 298/300 Write block 4428 len 16384 data: 2 2 2 2
575 299/300 Read block 826 len 8192
576 300/300 Read block 2596 len 28672
577 Main task has work to do, go find it
578 got dsc Start(2)
579 start 2
580 Oct 13 18:09:12.088 INFO request completed, latency_us: 324, response_code: 204, uri: /start/cid/2, method: POST, req_id: d11d430e-431a-42d8-98c4-0b41a49319cc, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
581 Replay: started 2, returned:Ok(())
582 ----------------------------------------------------------------
583 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
584 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
585 1 Acked 1000 Write 2 Done Done New false
586 2 Acked 1001 Write 3 Done Done New false
587 3 Acked 1002 Read 1 Done Done New false
588 4 Acked 1003 Read 7 Done Done New false
589 5 Acked 1004 Write 4 Done Done New false
590 6 Acked 1005 Write 4 Done Done New false
591 7 Acked 1006 Read 5 Done Done New false
592 8 Acked 1007 Read 8 Done Done New false
593 9 Acked 1008 Write 7 Done Done New false
594 10 Acked 1009 Write 2 Done Done New false
595 11 Acked 1010 Write 7 Done Done New false
596 12 Acked 1011 Read 4 Done Done New false
597 13 Acked 1012 Write 9 Done Done New false
598 14 Acked 1013 Write 6 Done Done New false
599 15 Acked 1014 Read 9 Done Done New false
600 16 Acked 1015 Write 3 Done Done New false
601 17 Acked 1016 Write 1 Done Done New false
602 18 Acked 1017 Read 6 Done Done New false
603 19 Acked 1018 Write 5 Done Done New false
604 20 Acked 1019 Write 3 Done Done New false
605 21 Acked 1020 Write 1 Done Done New false
606 22 Acked 1021 Read 6 Done Done New false
607 23 Acked 1022 Read 5 Done Done New false
608 24 Acked 1023 Read 10 Done Done New false
609 25 Acked 1024 Write 5 Done Done New false
610 26 Acked 1025 Read 4 Done Done New false
611 27 Acked 1026 Read 7 Done Done New false
612 28 Acked 1027 Write 7 Done Done New false
613 29 Acked 1028 Read 4 Done Done New false
614 30 Acked 1029 Write 9 Done Done New false
615 31 Acked 1030 Write 10 Done Done New false
616 32 Acked 1031 Flush 0 Done Done New false
617 33 Acked 1032 Write 10 Done Done New false
618 34 Acked 1033 Write 2 Done Done New false
619 35 Acked 1034 Read 5 Done Done New false
620 36 Acked 1035 Read 8 Done Done New false
621 37 Acked 1036 Write 5 Done Done New false
622 38 Acked 1037 Write 10 Done Done New false
623 39 Acked 1038 Read 5 Done Done New false
624 40 Acked 1039 Read 7 Done Done New false
625 41 Acked 1040 Write 9 Done Done New false
626 42 Acked 1041 Flush 0 Done Done New false
627 43 Acked 1042 Write 2 Done Done New false
628 44 Acked 1043 Write 1 Done Done New false
629 45 Acked 1044 Read 10 Done Done New false
630 46 Acked 1045 Read 1 Done Done New false
631 47 Acked 1046 Write 2 Done Done New false
632 48 Acked 1047 Read 7 Done Done New false
633 49 Acked 1048 Read 6 Done Done New false
634 50 Acked 1049 Write 4 Done Done New false
635 51 Acked 1050 Read 7 Done Done New false
636 52 Acked 1051 Write 4 Done Done New false
637 53 Acked 1052 Write 7 Done Done New false
638 54 Acked 1053 Read 1 Done Done New false
639 55 Acked 1054 Write 4 Done Done New false
640 56 Acked 1055 Read 3 Done Done New false
641 57 Acked 1056 Read 2 Done Done New false
642 58 Acked 1057 Write 8 Done Done New false
643 59 Acked 1058 Read 10 Done Done New false
644 60 Acked 1059 Write 7 Done Done New false
645 61 Acked 1060 Write 9 Done Done New false
646 62 Acked 1061 Write 7 Done Done New false
647 63 Acked 1062 Write 10 Done Done New false
648 64 Acked 1063 Write 2 Done Done New false
649 65 Acked 1064 Write 8 Done Done New false
650 66 Acked 1065 Read 1 Done Done New false
651 67 Acked 1066 Write 9 Done Done New false
652 68 Acked 1067 Read 1 Done Done New false
653 69 Acked 1068 Read 6 Done Done New false
654 70 Acked 1069 Read 3 Done Done New false
655 71 Acked 1070 Flush 0 Done Done New false
656 72 Acked 1071 Read 10 Done Done New false
657 73 Acked 1072 Write 6 Done Done New false
658 74 Acked 1073 Flush 0 Done Done New false
659 75 Acked 1074 Read 1 Done Done New false
660 76 Acked 1075 Write 10 Done Done New false
661 77 Acked 1076 Read 2 Done Done New false
662 78 Acked 1077 Write 5 Done Done New false
663 79 Acked 1078 Write 10 Done Done New false
664 80 Acked 1079 Flush 0 Done Done New false
665 81 Acked 1080 Flush 0 Done Done New false
666 82 Acked 1081 Read 6 Done Done New false
667 83 Acked 1082 Flush 0 Done Done New false
668 84 Acked 1083 Write 1 Done Done New false
669 85 Acked 1084 Flush 0 Done Done New false
670 86 Acked 1085 Flush 0 Done Done New false
671 87 Acked 1086 Write 1 Done Done New false
672 88 Acked 1087 Flush 0 Done Done New false
673 89 Acked 1088 Flush 0 Done Done New false
674 90 Acked 1089 Write 8 Done Done New false
675 91 Acked 1090 Read 3 Done Done New false
676 92 Acked 1091 Read 6 Done Done New false
677 93 Acked 1092 Read 10 Done Done New false
678 94 Acked 1093 Read 2 Done Done New false
679 95 Acked 1094 Read 7 Done Done New false
680 96 Acked 1095 Flush 0 Done Done New false
681 97 Acked 1096 Read 8 Done Done New false
682 98 Acked 1097 Write 9 Done Done New false
683 99 Acked 1098 Flush 0 Done Done New false
684 100 Acked 1099 Write 1 Done Done New false
685 101 Acked 1100 Read 3 Done Done New false
686 102 Acked 1101 Read 8 Done Done New false
687 103 Acked 1102 Read 5 Done Done New false
688 104 Acked 1103 Write 6 Done Done New false
689 105 Acked 1104 Read 1 Done Done New false
690 106 Acked 1105 Write 5 Done Done New false
691 107 Acked 1106 Read 9 Done Done New false
692 108 Acked 1107 Read 1 Done Done New false
693 109 Acked 1108 Read 3 Done Done New false
694 110 Acked 1109 Read 5 Done Done New false
695 111 Acked 1110 Read 6 Done Done New false
696 112 Acked 1111 Write 1 Done Done New false
697 113 Acked 1112 Write 3 Done Done New false
698 114 Acked 1113 Read 7 Done Done New false
699 115 Acked 1114 Write 4 Done Done New false
700 116 Acked 1115 Write 4 Done Done New false
701 117 Acked 1116 Write 10 Done Done New false
702 118 Acked 1117 Write 7 Done Done New false
703 119 Acked 1118 Flush 0 Done Done New false
704 120 Acked 1119 Read 3 Done Done New false
705 121 Acked 1120 Read 1 Done Done New false
706 122 Acked 1121 Read 6 Done Done New false
707 123 Acked 1122 Read 1 Done Done New false
708 124 Acked 1123 Write 8 Done Done New false
709 125 Acked 1124 Write 7 Done Done New false
710 126 Acked 1125 Read 5 Done Done New false
711 127 Acked 1126 Read 10 Done Done New false
712 128 Acked 1127 Flush 0 Done Done New false
713 129 Acked 1128 Read 6 Done Done New false
714 130 Acked 1129 Flush 0 Done Done New false
715 131 Acked 1130 Read 9 Done Done New false
716 132 Acked 1131 Read 1 Done Done New false
717 133 Acked 1132 Read 8 Done Done New false
718 134 Acked 1133 Write 7 Done Done New false
719 135 Acked 1134 Write 4 Done Done New false
720 136 Acked 1135 Write 3 Done Done New false
721 137 Acked 1136 Read 8 Done Done New false
722 138 Acked 1137 Write 4 Done Done New false
723 139 Acked 1138 Flush 0 Done Done New false
724 140 Acked 1139 Read 2 Done Done New false
725 141 Acked 1140 Write 10 Done Done New false
726 142 Acked 1141 Write 6 Done Done New false
727 143 Acked 1142 Write 5 Done Done New false
728 144 Acked 1143 Write 3 Done Done New false
729 145 Acked 1144 Flush 0 Done Done New false
730 146 Acked 1145 Read 2 Done Done New false
731 147 Acked 1146 Read 5 Done Done New false
732 148 Acked 1147 Write 6 Done Done New false
733 149 Acked 1148 Read 5 Done Done New false
734 150 Acked 1149 Write 9 Done Done New false
735 151 Acked 1150 Read 6 Done Done New false
736 152 Acked 1151 Flush 0 Done Done New false
737 153 Acked 1152 Write 1 Done Done New false
738 154 Acked 1153 Read 7 Done Done New false
739 155 Acked 1154 Read 5 Done Done New false
740 156 Acked 1155 Write 1 Done Done New false
741 157 Acked 1156 Read 4 Done Done New false
742 158 Acked 1157 Write 5 Done Done New false
743 159 Acked 1158 Write 2 Done Done New false
744 160 Acked 1159 Read 3 Done Done New false
745 161 Acked 1160 Read 4 Done Done New false
746 162 Acked 1161 Read 7 Done Done New false
747 163 Acked 1162 Read 9 Done Done New false
748 164 Acked 1163 Write 3 Done Done New false
749 165 Acked 1164 Read 1 Done Done New false
750 166 Acked 1165 Read 8 Done Done New false
751 167 Acked 1166 Write 9 Done Done New false
752 168 Acked 1167 Write 6 Done Done New false
753 169 Acked 1168 Read 6 Done Done New false
754 170 Acked 1169 Read 10 Done Done New false
755 171 Acked 1170 Write 5 Done Done New false
756 172 Acked 1171 Read 7 Done Done New false
757 173 Acked 1172 Flush 0 Done Done New false
758 174 Acked 1173 Write 3 Done Done New false
759 175 Acked 1174 Write 7 Done Done New false
760 176 Acked 1175 Read 1 Done Done New false
761 177 Acked 1176 Read 9 Done Done New false
762 178 Acked 1177 Read 8 Done Done New false
763 179 Acked 1178 Read 6 Done Done New false
764 180 Acked 1179 Flush 0 Done Done New false
765 181 Acked 1180 Read 10 Done Done New false
766 182 Acked 1181 Read 8 Done Done New false
767 183 Acked 1182 Read 10 Done Done New false
768 184 Acked 1183 Read 5 Done Done New false
769 185 Acked 1184 Write 9 Done Done New false
770 186 Acked 1185 Read 5 Done Done New false
771 187 Acked 1186 Write 4 Done Done New false
772 188 Acked 1187 Write 7 Done Done New false
773 189 Acked 1188 Write 5 Done Done New false
774 190 Acked 1189 Read 6 Done Done New false
775 191 Acked 1190 Read 8 Done Done New false
776 192 Acked 1191 Read 6 Done Done New false
777 193 Acked 1192 Write 2 Done Done New false
778 194 Acked 1193 Read 4 Done Done New false
779 195 Acked 1194 Flush 0 Done Done New false
780 196 Acked 1195 Flush 0 Done Done New false
781 197 Acked 1196 Write 10 Done Done New false
782 198 Acked 1197 Write 8 Done Done New false
783 199 Acked 1198 Write 8 Done Done New false
784 200 Acked 1199 Write 7 Done Done New false
785 201 Acked 1200 Read 3 Done Done New false
786 202 Acked 1201 Read 8 Done Done New false
787 203 Acked 1202 Write 9 Done Done New false
788 204 Acked 1203 Read 7 Done Done New false
789 205 Acked 1204 Read 6 Done Done New false
790 206 Acked 1205 Read 1 Done Done New false
791 207 Acked 1206 Write 7 Done Done New false
792 208 Acked 1207 Write 9 Done Done New false
793 209 Acked 1208 Read 5 Done Done New false
794 210 Acked 1209 Read 1 Done Done New false
795 211 Acked 1210 Write 4 Done Done New false
796 212 Acked 1211 Read 9 Done Done New false
797 213 Acked 1212 Read 1 Done Done New false
798 214 Acked 1213 Flush 0 Done Done New false
799 215 Acked 1214 Flush 0 Done Done New false
800 216 Acked 1215 Write 9 Done Done New false
801 217 Acked 1216 Read 4 Done Done New false
802 218 Acked 1217 Read 2 Done Done New false
803 219 Acked 1218 Read 8 Done Done New false
804 220 Acked 1219 Write 5 Done Done New false
805 221 Acked 1220 Read 1 Done Done New false
806 222 Acked 1221 Read 1 Done Done New false
807 223 Acked 1222 Write 7 Done Done New false
808 224 Acked 1223 Read 4 Done Done New false
809 225 Acked 1224 Read 7 Done Done New false
810 226 Acked 1225 Read 5 Done Done New false
811 227 Acked 1226 Read 6 Done Done New false
812 228 Acked 1227 Write 8 Done Done New false
813 229 Acked 1228 Read 1 Done Done New false
814 230 Acked 1229 Read 5 Done Done New false
815 231 Acked 1230 Read 10 Done Done New false
816 232 Acked 1231 Write 10 Done Done New false
817 233 Acked 1232 Read 3 Done Done New false
818 234 Acked 1233 Read 5 Done Done New false
819 235 Acked 1234 Read 6 Done Done New false
820 236 Acked 1235 Read 5 Done Done New false
821 237 Acked 1236 Read 1 Done Done New false
822 238 Acked 1237 Read 1 Done Done New false
823 239 Acked 1238 Write 5 Done Done New false
824 240 Acked 1239 Write 8 Done Done New false
825 241 Acked 1240 Write 2 Done Done New false
826 242 Acked 1241 Write 8 Done Done New false
827 243 Acked 1242 Flush 0 Done Done New false
828 244 Acked 1243 Read 8 Done Done New false
829 245 Acked 1244 Write 2 Done Done New false
830 246 Acked 1245 Read 6 Done Done New false
831 247 Acked 1246 Flush 0 Done Done New false
832 248 Acked 1247 Read 7 Done Done New false
833 249 Acked 1248 Write 2 Done Done New false
834 250 Acked 1249 Write 3 Done Done New false
835 251 Acked 1250 Read 7 Done Done New false
836 252 Acked 1251 Flush 0 Done Done New false
837 253 Acked 1252 Read 5 Done Done New false
838 254 Acked 1253 Write 4 Done Done New false
839 255 Acked 1254 Read 1 Done Done New false
840 256 Acked 1255 Read 1 Done Done New false
841 257 Acked 1256 Write 10 Done Done New false
842 258 Acked 1257 Write 6 Done Done New false
843 259 Acked 1258 Read 1 Done Done New false
844 260 Acked 1259 Read 1 Done Done New false
845 261 Acked 1260 Read 9 Done Done New false
846 262 Acked 1261 Write 9 Done Done New false
847 263 Acked 1262 Write 1 Done Done New false
848 264 Acked 1263 Read 5 Done Done New false
849 265 Acked 1264 Flush 0 Done Done New false
850 266 Acked 1265 Write 4 Done Done New false
851 267 Acked 1266 Write 8 Done Done New false
852 268 Acked 1267 Write 3 Done Done New false
853 269 Acked 1268 Read 1 Done Done New false
854 270 Acked 1269 Write 7 Done Done New false
855 271 Acked 1270 Read 8 Done Done New false
856 272 Acked 1271 Read 9 Done Done New false
857 273 Acked 1272 Flush 0 Done Done New false
858 274 Acked 1273 Write 1 Done Done New false
859 275 Acked 1274 Read 1 Done Done New false
860 276 Acked 1275 Flush 0 Done Done New false
861 277 Acked 1276 Write 4 Done Done New false
862 278 Acked 1277 Read 3 Done Done New false
863 279 Acked 1278 Read 8 Done Done New false
864 280 Acked 1279 Flush 0 Done Done New false
865 281 Acked 1280 Read 7 Done Done New false
866 282 Acked 1281 Read 6 Done Done New false
867 283 Acked 1282 Read 5 Done Done New false
868 284 Acked 1283 Read 9 Done Done New false
869 285 Acked 1284 Write 8 Done Done New false
870 286 Acked 1285 Read 5 Done Done New false
871 287 Acked 1286 Flush 0 Done Done New false
872 288 Acked 1287 Write 7 Done Done New false
873 289 Acked 1288 Read 3 Done Done New false
874 290 Acked 1289 Flush 0 Done Done New false
875 291 Acked 1290 Write 5 Done Done New false
876 292 Acked 1291 Flush 0 Done Done New false
877 293 Acked 1292 Write 6 Done Done New false
878 294 Acked 1293 Read 5 Done Done New false
879 295 Acked 1294 Read 1 Done Done New false
880 296 Acked 1295 Read 4 Done Done New false
881 297 Acked 1296 Write 8 Done Done New false
882 298 Acked 1297 Write 2 Done Done New false
883 299 Acked 1298 Write 6 Done Done New false
884 300 Acked 1299 Write 4 Done Done New false
885 301 Acked 1300 Read 2 Done Done New false
886 302 Acked 1301 Read 7 Done Done New false
887 STATES DS:0 DS:1 DS:2 TOTAL
888 New 0 0 302 302
889 Sent 0 0 0 0
890 Done 302 302 0 604
891 Skipped 0 0 0 0
892 Error 0 0 0 0
893 Last Flush: 1291 1291 0
894 Downstairs last five completed:
895 Upstairs last five completed: 302 301 300 299 298 297
896 CLIENT: Up:0 ds:302 act:2
897 [8830] got start action
898 [8830] I am going to restart
899 Starting downstairs at port 8830
900 Make output file at "/tmp/dsc/downstairs-8830.txt"
901 [8830][2] reports Starting
902 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1177)
903 [8830][2] reports Running
9042023-10-13T18:09:14.145ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 2
9052023-10-13T18:09:14.145ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
9062023-10-13T18:09:14.145ZINFOcrucible: [2] client re-new 303 jobs since flush 0 = downstairs
9072023-10-13T18:09:14.146ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
9082023-10-13T18:09:14.147ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
9092023-10-13T18:09:14.147ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
9102023-10-13T18:09:14.147ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
9112023-10-13T18:09:14.147ZINFOcrucible: [2] send last flush ID to this DS: 0
9122023-10-13T18:09:14.147ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
9132023-10-13T18:09:14.147ZINFOcrucible: [2] Replied this last flush ID: 0
9142023-10-13T18:09:14.147ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Offline ds_transition to Replay
9152023-10-13T18:09:14.147ZINFOcrucible: [2] Transition from Offline to Replay
9162023-10-13T18:09:14.147ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
9172023-10-13T18:09:14.147ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Replay ds_transition to Active
9182023-10-13T18:09:14.147ZINFOcrucible: [2] Transition from Replay to Active
9192023-10-13T18:09:14.147ZINFOcrucible: [2] Starts cmd_loop
9202023-10-13T18:09:14.148ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
9212023-10-13T18:09:14.148ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
9222023-10-13T18:09:14.148ZINFOcrucible: Set check for repair
9232023-10-13T18:09:15.148ZWARNcrucible: [2] flow control end
9242023-10-13T18:09:15.149ZINFOcrucible: Checking if live repair is needed
9252023-10-13T18:09:15.149ZINFOcrucible: No Live Repair required at this time
926 ----------------------------------------------------------------
927 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
928 Downstairs last five completed: 1302 1301 1300 1299 1298 1297
929 Upstairs last five completed: 303 302 301 300 299 298
930 CLIENT: Up:0 ds:0 act:3
931 Replay: All jobs finished, all DS active.
932 Main task has work to do, go find it
933 got dsc Stop(0)
934 stop 0
935 Oct 13 18:09:16.092 INFO request completed, latency_us: 512, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 88191ef8-d4ac-4beb-ad4f-e0f7f4d96af5, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
936 [8810] Got stop action so:false kr:false
937 Oct 13 18:09:16.093 INFO request completed, latency_us: 241, response_code: 200, uri: /state/cid/0, method: GET, req_id: f92e0a09-c2d2-4c6c-b73d-88d3c3cbc314, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
9382023-10-13T18:09:16.093ZWARNcrucible: [0] downstairs disconnected
9392023-10-13T18:09:16.094ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
9402023-10-13T18:09:16.094ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8810 closed looper = 0
9412023-10-13T18:09:16.094ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:false
9422023-10-13T18:09:16.094ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
9432023-10-13T18:09:16.094ZINFOcrucible: [0] 127.0.0.1:8810 task reports offline
9442023-10-13T18:09:16.094ZWARNcrucible: [0] pm_task rx.recv() is None
945 [8810] Exited with: ExitStatus(unix_wait_status(9))
946 [8810][0] reports Exit
947 Oct 13 18:09:20.096 INFO request completed, latency_us: 262, response_code: 200, uri: /state/cid/0, method: GET, req_id: a1f601ef-e6c3-4a57-97fb-077879bee5b6, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
948 001/300 Read block 3275 len 16384
949 002/300 Write block 2520 len 32768 data: 2 2 2 2 2 2 2 2
950 003/300 Write block 3663 len 16384 data: 2 2 2 3
951 004/300 Write block 759 len 32768 data: 2 2 2 2 2 2 2 2
952 005/300 Write block 4933 len 32768 data: 2 2 2 2 2 2 2 2
953 006/300 Write block 698 len 32768 data: 2 2 2 2 2 2 2 3
954 007/300 Read block 1594 len 12288
955 008/300 Read block 3248 len 16384
956 009/300 Read block 2454 len 36864
957 010/300 Read block 734 len 32768
958 011/300 Read block 2238 len 24576
959 012/300 Read block 3684 len 32768
960 013/300 Read block 168 len 4096
961 014/300 Read block 4556 len 36864
962 015/300 Read block 119 len 16384
963 016/300 Write block 3847 len 12288 data: 2 2 2
964 017/300 Write block 3862 len 32768 data: 2 2 2 2 2 2 2 2
965 018/300 Read block 1855 len 4096
966 019/300 Read block 1398 len 24576
967 020/300 Read block 4205 len 24576
968 021/300 Read block 3702 len 20480
969 022/300 Write block 4624 len 24576 data: 2 2 2 2 2 2
970 023/300 Read block 567 len 16384
971 024/300 Read block 4778 len 24576
972 025/300 Read block 4307 len 32768
973 026/300 Read block 2070 len 40960
974 027/300 Read block 4945 len 24576
975 028/300 Write block 1050 len 16384 data: 2 2 2 2
976 029/300 Read block 1913 len 24576
977 030/300 Read block 627 len 20480
978 031/300 Flush
979 032/300 Read block 1392 len 32768
980 033/300 Write block 4619 len 20480 data: 2 2 2 2 2
981 034/300 Write block 3351 len 4096 data: 2
982 035/300 Read block 2546 len 28672
983 036/300 Write block 2222 len 16384 data: 3 2 2 2
984 037/300 Flush
985 038/300 Write block 4676 len 8192 data: 2 2
986 039/300 Read block 4875 len 32768
987 040/300 Flush
988 041/300 Read block 4626 len 32768
989 042/300 Write block 3706 len 28672 data: 2 2 2 2 3 3 2
990 043/300 Read block 3670 len 24576
991 044/300 Flush
992 045/300 Write block 557 len 40960 data: 2 2 2 2 2 2 2 2 2 2
993 046/300 Write block 4373 len 32768 data: 2 2 2 2 2 2 3 2
994 047/300 Read block 3555 len 20480
995 048/300 Read block 3494 len 20480
996 049/300 Write block 1937 len 4096 data: 2
997 050/300 Read block 715 len 32768
998 051/300 Write block 4614 len 8192 data: 2 2
999 052/300 Read block 1330 len 8192
1000 053/300 Read block 3777 len 28672
1001 054/300 Read block 493 len 36864
1002 055/300 Write block 2242 len 12288 data: 2 2 2
1003 056/300 Write block 1857 len 16384 data: 2 2 2 2
1004 057/300 Read block 1643 len 16384
1005 058/300 Read block 1007 len 20480
1006 059/300 Read block 4150 len 24576
1007 060/300 Write block 87 len 12288 data: 2 2 2
1008 061/300 Write block 4878 len 36864 data: 2 2 2 2 2 2 2 2 2
1009 062/300 Read block 1702 len 32768
1010 063/300 Read block 367 len 40960
1011 064/300 Write block 4416 len 20480 data: 2 2 2 2 2
1012 065/300 Write block 1924 len 16384 data: 2 2 2 2
1013 066/300 Write block 2806 len 12288 data: 2 2 3
1014 067/300 Flush
1015 068/300 Write block 2478 len 36864 data: 2 2 2 2 2 2 2 3 3
1016 069/300 Read block 1122 len 4096
1017 070/300 Read block 4593 len 32768
1018 071/300 Read block 2648 len 24576
1019 072/300 Flush
1020 073/300 Write block 4524 len 28672 data: 2 2 2 2 3 2 2
1021 074/300 Write block 1148 len 20480 data: 2 2 2 2 2
1022 075/300 Read block 3761 len 28672
1023 076/300 Read block 2248 len 32768
1024 077/300 Flush
1025 078/300 Write block 1131 len 16384 data: 2 2 2 2
1026 079/300 Read block 4682 len 20480
1027 080/300 Read block 4346 len 36864
1028 081/300 Read block 4637 len 4096
1029 082/300 Write block 1129 len 28672 data: 2 2 3 3 3 3 2
1030 083/300 Write block 146 len 40960 data: 2 2 2 2 3 3 3 3 3 2
1031 084/300 Read block 1823 len 16384
1032 085/300 Write block 3595 len 12288 data: 2 2 2
1033 086/300 Write block 3329 len 16384 data: 2 2 2 2
1034 087/300 Flush
1035 088/300 Write block 2943 len 20480 data: 2 2 2 2 2
1036 089/300 Write block 799 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1037 090/300 Write block 3568 len 8192 data: 2 2
1038 091/300 Write block 2158 len 8192 data: 2 2
1039 092/300 Read block 3102 len 32768
1040 093/300 Flush
1041 094/300 Read block 2117 len 36864
1042 095/300 Read block 2299 len 36864
1043 096/300 Write block 1322 len 8192 data: 2 2
1044 097/300 Write block 1491 len 36864 data: 2 2 2 2 2 2 2 2 2
1045 098/300 Read block 3992 len 12288
1046 099/300 Read block 2921 len 28672
1047 100/300 Write block 3872 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1048 101/300 Read block 2137 len 24576
1049 102/300 Flush
1050 103/300 Write block 620 len 8192 data: 2 2
1051 104/300 Read block 4176 len 20480
1052 105/300 Read block 4472 len 16384
1053 106/300 Write block 1897 len 8192 data: 2 2
1054 107/300 Write block 3444 len 4096 data: 2
1055 108/300 Flush
1056 109/300 Read block 312 len 8192
1057 110/300 Read block 574 len 16384
1058 111/300 Read block 4272 len 4096
1059 112/300 Read block 3219 len 8192
1060 113/300 Read block 153 len 32768
1061 114/300 Flush
1062 115/300 Read block 2796 len 16384
1063 116/300 Write block 2219 len 28672 data: 3 3 3 4 3 3 3
1064 117/300 Write block 4928 len 20480 data: 2 2 2 2 2
1065 118/300 Read block 1327 len 16384
1066 119/300 Read block 2904 len 8192
1067 120/300 Read block 2851 len 8192
1068 121/300 Write block 1660 len 20480 data: 2 2 2 2 2
1069 122/300 Read block 1123 len 4096
1070 123/300 Read block 3218 len 24576
1071 124/300 Read block 275 len 12288
1072 125/300 Write block 753 len 8192 data: 2 2
1073 126/300 Write block 1032 len 28672 data: 2 2 2 2 2 2 2
1074 127/300 Read block 3242 len 12288
1075 128/300 Read block 3245 len 20480
1076 129/300 Flush
1077 130/300 Flush
1078 131/300 Write block 1520 len 36864 data: 2 2 2 2 2 2 2 2 2
1079 132/300 Write block 3946 len 40960 data: 2 2 2 2 2 2 2 2 3 3
1080 133/300 Flush
1081 134/300 Read block 2364 len 24576
1082 135/300 Read block 410 len 36864
1083 136/300 Read block 3542 len 16384
1084 137/300 Read block 627 len 8192
1085 138/300 Read block 3539 len 8192
1086 139/300 Read block 480 len 36864
1087 140/300 Read block 1055 len 40960
1088 141/300 Write block 4721 len 36864 data: 2 2 2 2 2 2 2 2 2
1089 142/300 Write block 1994 len 24576 data: 3 3 3 3 2 2
1090 143/300 Read block 323 len 24576
1091 144/300 Read block 2065 len 28672
1092 145/300 Read block 2226 len 28672
1093 146/300 Read block 1831 len 16384
1094 147/300 Write block 2004 len 28672 data: 2 2 2 3 3 3 3
1095 148/300 Write block 4528 len 24576 data: 4 3 3 2 2 2
1096 149/300 Read block 1050 len 4096
1097 150/300 Flush
1098 151/300 Write block 4193 len 4096 data: 3
1099 152/300 Read block 2519 len 4096
1100 153/300 Read block 2838 len 16384
1101 154/300 Write block 1801 len 4096 data: 2
1102 155/300 Read block 4416 len 4096
1103 156/300 Write block 4398 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1104 157/300 Flush
1105 158/300 Write block 2015 len 4096 data: 2
1106 159/300 Write block 3328 len 12288 data: 2 3 3
1107 160/300 Read block 2196 len 24576
1108 161/300 Flush
1109 162/300 Read block 3181 len 4096
1110 163/300 Read block 1309 len 40960
1111 164/300 Flush
1112 165/300 Read block 1619 len 4096
1113 166/300 Read block 1462 len 20480
1114 167/300 Read block 518 len 28672
1115 168/300 Read block 1401 len 32768
1116 169/300 Write block 2119 len 36864 data: 2 2 2 2 2 2 2 2 2
1117 170/300 Read block 2021 len 32768
1118 171/300 Read block 1390 len 40960
1119 172/300 Write block 661 len 28672 data: 2 2 2 2 2 3 2
1120 173/300 Write block 3285 len 32768 data: 2 2 2 2 2 2 2 2
1121 174/300 Write block 1815 len 24576 data: 2 2 2 2 2 2
1122 175/300 Write block 2597 len 24576 data: 2 2 2 2 2 2
1123 176/300 Write block 1429 len 28672 data: 2 2 2 2 3 3 2
1124 177/300 Write block 136 len 28672 data: 2 2 2 2 2 2 2
1125 178/300 Read block 930 len 40960
1126 179/300 Write block 1937 len 8192 data: 3 2
1127 180/300 Write block 663 len 4096 data: 3
1128 181/300 Read block 3153 len 24576
1129 182/300 Flush
1130 183/300 Read block 3245 len 40960
1131 184/300 Read block 1175 len 40960
1132 185/300 Read block 2010 len 20480
1133 186/300 Read block 1904 len 36864
1134 187/300 Read block 3688 len 8192
1135 188/300 Read block 4938 len 16384
1136 189/300 Flush
1137 190/300 Read block 704 len 20480
1138 191/300 Read block 3589 len 40960
1139 192/300 Read block 2040 len 8192
1140 193/300 Read block 245 len 12288
1141 194/300 Write block 4021 len 24576 data: 2 2 2 2 2 2
1142 195/300 Write block 4333 len 12288 data: 2 2 2
1143 196/300 Read block 2436 len 8192
1144 197/300 Write block 3137 len 28672 data: 2 2 2 2 2 2 2
1145 198/300 Write block 1294 len 36864 data: 2 2 2 2 2 2 2 2 2
1146 199/300 Read block 1830 len 36864
1147 200/300 Read block 2297 len 8192
1148 201/300 Write block 3937 len 32768 data: 3 3 3 2 2 2 2 2
1149 202/300 Read block 489 len 32768
1150 203/300 Read block 905 len 32768
1151 204/300 Write block 1561 len 16384 data: 2 2 2 2
1152 205/300 Flush
1153 206/300 Read block 1976 len 40960
1154 207/300 Read block 1443 len 24576
1155 208/300 Read block 2121 len 16384
1156 209/300 Read block 1068 len 28672
1157 210/300 Read block 3612 len 4096
1158 211/300 Write block 1670 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1159 212/300 Write block 3447 len 40960 data: 2 2 2 2 2 2 3 3 3 3
1160 213/300 Write block 127 len 32768 data: 2 2 2 2 2 2 2 2
1161 214/300 Read block 35 len 36864
1162 215/300 Read block 4297 len 12288
1163 216/300 Read block 3905 len 16384
1164 217/300 Write block 3518 len 36864 data: 2 3 3 3 3 3 3 2 2
1165 218/300 Read block 1672 len 8192
1166 219/300 Write block 2569 len 8192 data: 3 3
1167 220/300 Read block 3049 len 4096
1168 221/300 Read block 1173 len 24576
1169 222/300 Write block 1712 len 8192 data: 2 2
1170 223/300 Write block 840 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1171 224/300 Write block 4755 len 12288 data: 3 3 3
1172 225/300 Read block 1898 len 20480
1173 226/300 Read block 2800 len 8192
1174 227/300 Write block 4163 len 32768 data: 2 2 2 2 2 2 2 2
1175 228/300 Read block 1414 len 16384
1176 229/300 Read block 4742 len 8192
1177 230/300 Read block 2876 len 32768
1178 231/300 Write block 4845 len 32768 data: 2 2 2 2 2 2 2 2
1179 232/300 Write block 2200 len 12288 data: 2 2 2
1180 233/300 Write block 345 len 20480 data: 2 2 2 2 2
1181 234/300 Write block 760 len 4096 data: 3
1182 235/300 Read block 4049 len 28672
1183 236/300 Read block 1428 len 40960
1184 237/300 Read block 4125 len 4096
1185 238/300 Write block 3792 len 8192 data: 2 2
1186 239/300 Read block 2131 len 12288
1187 240/300 Write block 1970 len 16384 data: 2 2 2 2
1188 241/300 Write block 2552 len 40960 data: 2 4 4 3 3 2 2 2 2 2
1189 242/300 Flush
1190 243/300 Read block 4566 len 40960
1191 244/300 Write block 35 len 24576 data: 2 2 2 2 2 2
1192 245/300 Read block 336 len 16384
1193 246/300 Write block 2243 len 12288 data: 3 3 2
1194 247/300 Flush
1195 248/300 Write block 243 len 28672 data: 2 2 2 2 2 2 2
1196 249/300 Write block 1162 len 20480 data: 2 2 2 2 2
1197 250/300 Read block 113 len 4096
1198 251/300 Read block 307 len 20480
1199 252/300 Read block 4103 len 12288
1200 253/300 Read block 1619 len 16384
1201 254/300 Write block 2032 len 4096 data: 2
1202 255/300 Read block 3809 len 12288
1203 256/300 Write block 4939 len 8192 data: 3 3
1204 257/300 Read block 3748 len 20480
1205 258/300 Read block 4932 len 28672
1206 259/300 Read block 1267 len 8192
1207 260/300 Read block 3757 len 4096
1208 261/300 Write block 3618 len 24576 data: 2 2 2 2 2 2
1209 262/300 Write block 1322 len 24576 data: 3 3 2 2 2 2
1210 263/300 Write block 2863 len 32768 data: 2 2 2 2 2 2 2 2
1211 264/300 Flush
1212 265/300 Write block 3680 len 32768 data: 2 2 2 2 2 2 2 2
1213 266/300 Flush
1214 267/300 Read block 4465 len 36864
1215 268/300 Flush
1216 269/300 Read block 2141 len 24576
1217 270/300 Write block 1896 len 24576 data: 2 3 3 2 2 2
1218 271/300 Read block 4764 len 28672
1219 272/300 Read block 2563 len 40960
1220 273/300 Flush
1221 274/300 Read block 1628 len 12288
1222 275/300 Read block 4731 len 32768
1223 276/300 Write block 491 len 4096 data: 2
1224 277/300 Write block 1783 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1225 278/300 Write block 2329 len 36864 data: 3 3 3 3 2 2 2 2 2
1226 279/300 Read block 3837 len 12288
1227 280/300 Write block 2619 len 8192 data: 2 2
1228 281/300 Write block 3387 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1229 282/300 Read block 4214 len 32768
1230 283/300 Write block 2237 len 16384 data: 2 2 2 2
1231 284/300 Read block 1652 len 4096
1232 285/300 Write block 4511 len 16384 data: 2 2 2 2
1233 286/300 Write block 3095 len 24576 data: 3 2 2 2 2 2
1234 287/300 Read block 4694 len 36864
1235 288/300 Read block 1102 len 36864
1236 289/300 Flush
1237 290/300 Read block 764 len 36864
1238 291/300 Write block 2218 len 4096 data: 3
1239 292/300 Write block 4608 len 4096 data: 2
1240 293/300 Read block 2009 len 32768
1241 294/300 Read block 1614 len 40960
1242 295/300 Read block 293 len 4096
1243 296/300 Flush
1244 297/300 Read block 676 len 24576
1245 298/300 Write block 4890 len 4096 data: 2
1246 299/300 Flush
1247 300/300 Read block 2532 len 20480
1248 Main task has work to do, go find it
1249 got dsc Start(0)
1250 start 0
1251 Oct 13 18:09:21.049 INFO request completed, latency_us: 297, response_code: 204, uri: /start/cid/0, method: POST, req_id: 482a3630-69c7-406d-abf4-3c2465dc0b73, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
1252 Replay: started 0, returned:Ok(())
1253 ----------------------------------------------------------------
1254 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1255 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1256 304 Acked 1303 Read 4 New Done Done false
1257 305 Acked 1304 Write 8 New Done Done false
1258 306 Acked 1305 Write 4 New Done Done false
1259 307 Acked 1306 Write 8 New Done Done false
1260 308 Acked 1307 Write 8 New Done Done false
1261 309 Acked 1308 Write 8 New Done Done false
1262 310 Acked 1309 Read 3 New Done Done false
1263 311 Acked 1310 Read 4 New Done Done false
1264 312 Acked 1311 Read 9 New Done Done false
1265 313 Acked 1312 Read 8 New Done Done false
1266 314 Acked 1313 Read 6 New Done Done false
1267 315 Acked 1314 Read 8 New Done Done false
1268 316 Acked 1315 Read 1 New Done Done false
1269 317 Acked 1316 Read 9 New Done Done false
1270 318 Acked 1317 Read 4 New Done Done false
1271 319 Acked 1318 Write 3 New Done Done false
1272 320 Acked 1319 Write 8 New Done Done false
1273 321 Acked 1320 Read 1 New Done Done false
1274 322 Acked 1321 Read 6 New Done Done false
1275 323 Acked 1322 Read 6 New Done Done false
1276 324 Acked 1323 Read 5 New Done Done false
1277 325 Acked 1324 Write 6 New Done Done false
1278 326 Acked 1325 Read 4 New Done Done false
1279 327 Acked 1326 Read 6 New Done Done false
1280 328 Acked 1327 Read 8 New Done Done false
1281 329 Acked 1328 Read 10 New Done Done false
1282 330 Acked 1329 Read 6 New Done Done false
1283 331 Acked 1330 Write 4 New Done Done false
1284 332 Acked 1331 Read 6 New Done Done false
1285 333 Acked 1332 Read 5 New Done Done false
1286 334 Acked 1333 Flush 0 New Done Done false
1287 335 Acked 1334 Read 8 New Done Done false
1288 336 Acked 1335 Write 5 New Done Done false
1289 337 Acked 1336 Write 1 New Done Done false
1290 338 Acked 1337 Read 7 New Done Done false
1291 339 Acked 1338 Write 4 New Done Done false
1292 340 Acked 1339 Flush 0 New Done Done false
1293 341 Acked 1340 Write 2 New Done Done false
1294 342 Acked 1341 Read 8 New Done Done false
1295 343 Acked 1342 Flush 0 New Done Done false
1296 344 Acked 1343 Read 8 New Done Done false
1297 345 Acked 1344 Write 7 New Done Done false
1298 346 Acked 1345 Read 6 New Done Done false
1299 347 Acked 1346 Flush 0 New Done Done false
1300 348 Acked 1347 Write 10 New Done Done false
1301 349 Acked 1348 Write 8 New Done Done false
1302 350 Acked 1349 Read 5 New Done Done false
1303 351 Acked 1350 Read 5 New Done Done false
1304 352 Acked 1351 Write 1 New Done Done false
1305 353 Acked 1352 Read 8 New Done Done false
1306 354 Acked 1353 Write 2 New Done Done false
1307 355 Acked 1354 Read 2 New Done Done false
1308 356 Acked 1355 Read 7 New Done Done false
1309 357 Acked 1356 Read 9 New Done Done false
1310 358 Acked 1357 Write 3 New Done Done false
1311 359 Acked 1358 Write 4 New Done Done false
1312 360 Acked 1359 Read 4 New Done Done false
1313 361 Acked 1360 Read 5 New Done Done false
1314 362 Acked 1361 Read 6 New Done Done false
1315 363 Acked 1362 Write 3 New Done Done false
1316 364 Acked 1363 Write 9 New Done Done false
1317 365 Acked 1364 Read 8 New Done Done false
1318 366 Acked 1365 Read 10 New Done Done false
1319 367 Acked 1366 Write 5 New Done Done false
1320 368 Acked 1367 Write 4 New Done Done false
1321 369 Acked 1368 Write 3 New Done Done false
1322 370 Acked 1369 Flush 0 New Done Done false
1323 371 Acked 1370 Write 9 New Done Done false
1324 372 Acked 1371 Read 1 New Done Done false
1325 373 Acked 1372 Read 8 New Done Done false
1326 374 Acked 1373 Read 6 New Done Done false
1327 375 Acked 1374 Flush 0 New Done Done false
1328 376 Acked 1375 Write 7 New Done Done false
1329 377 Acked 1376 Write 5 New Done Done false
1330 378 Acked 1377 Read 7 New Done Done false
1331 379 Acked 1378 Read 8 New Done Done false
1332 380 Acked 1379 Flush 0 New Done Done false
1333 381 Acked 1380 Write 4 New Done Done false
1334 382 Acked 1381 Read 5 New Done Done false
1335 383 Acked 1382 Read 9 New Done Done false
1336 384 Acked 1383 Read 1 New Done Done false
1337 385 Acked 1384 Write 7 New Done Done false
1338 386 Acked 1385 Write 10 New Done Done false
1339 387 Acked 1386 Read 4 New Done Done false
1340 388 Acked 1387 Write 3 New Done Done false
1341 389 Acked 1388 Write 4 New Done Done false
1342 390 Acked 1389 Flush 0 New Done Done false
1343 391 Acked 1390 Write 5 New Done Done false
1344 392 Acked 1391 Write 10 New Done Done false
1345 393 Acked 1392 Write 2 New Done Done false
1346 394 Acked 1393 Write 2 New Done Done false
1347 395 Acked 1394 Read 8 New Done Done false
1348 396 Acked 1395 Flush 0 New Done Done false
1349 397 Acked 1396 Read 9 New Done Done false
1350 398 Acked 1397 Read 9 New Done Done false
1351 399 Acked 1398 Write 2 New Done Done false
1352 400 Acked 1399 Write 9 New Done Done false
1353 401 Acked 1400 Read 3 New Done Done false
1354 402 Acked 1401 Read 7 New Done Done false
1355 403 Acked 1402 Write 10 New Done Done false
1356 404 Acked 1403 Read 6 New Done Done false
1357 405 Acked 1404 Flush 0 New Done Done false
1358 406 Acked 1405 Write 2 New Done Done false
1359 407 Acked 1406 Read 5 New Done Done false
1360 408 Acked 1407 Read 4 New Done Done false
1361 409 Acked 1408 Write 2 New Done Done false
1362 410 Acked 1409 Write 1 New Done Done false
1363 411 Acked 1410 Flush 0 New Done Done false
1364 412 Acked 1411 Read 2 New Done Done false
1365 413 Acked 1412 Read 4 New Done Done false
1366 414 Acked 1413 Read 1 New Done Done false
1367 415 Acked 1414 Read 2 New Done Done false
1368 416 Acked 1415 Read 8 New Done Done false
1369 417 Acked 1416 Flush 0 New Done Done false
1370 418 Acked 1417 Read 4 New Done Done false
1371 419 Acked 1418 Write 7 New Done Done false
1372 420 Acked 1419 Write 5 New Done Done false
1373 421 Acked 1420 Read 4 New Done Done false
1374 422 Acked 1421 Read 2 New Done Done false
1375 423 Acked 1422 Read 2 New Done Done false
1376 424 Acked 1423 Write 5 New Done Done false
1377 425 Acked 1424 Read 1 New Done Done false
1378 426 Acked 1425 Read 6 New Done Done false
1379 427 Acked 1426 Read 3 New Done Done false
1380 428 Acked 1427 Write 2 New Done Done false
1381 429 Acked 1428 Write 7 New Done Done false
1382 430 Acked 1429 Read 3 New Done Done false
1383 431 Acked 1430 Read 5 New Done Done false
1384 432 Acked 1431 Flush 0 New Done Done false
1385 433 Acked 1432 Flush 0 New Done Done false
1386 434 Acked 1433 Write 9 New Done Done false
1387 435 Acked 1434 Write 10 New Done Done false
1388 436 Acked 1435 Flush 0 New Done Done false
1389 437 Acked 1436 Read 6 New Done Done false
1390 438 Acked 1437 Read 9 New Done Done false
1391 439 Acked 1438 Read 4 New Done Done false
1392 440 Acked 1439 Read 2 New Done Done false
1393 441 Acked 1440 Read 2 New Done Done false
1394 442 Acked 1441 Read 9 New Done Done false
1395 443 Acked 1442 Read 10 New Done Done false
1396 444 Acked 1443 Write 9 New Done Done false
1397 445 Acked 1444 Write 6 New Done Done false
1398 446 Acked 1445 Read 6 New Done Done false
1399 447 Acked 1446 Read 7 New Done Done false
1400 448 Acked 1447 Read 7 New Done Done false
1401 449 Acked 1448 Read 4 New Done Done false
1402 450 Acked 1449 Write 7 New Done Done false
1403 451 Acked 1450 Write 6 New Done Done false
1404 452 Acked 1451 Read 1 New Done Done false
1405 453 Acked 1452 Flush 0 New Done Done false
1406 454 Acked 1453 Flush 0 New Done Done false
1407 455 Acked 1454 Write 1 New Done Done false
1408 456 Acked 1455 Read 1 New Done Done false
1409 457 Acked 1456 Read 4 New Done Done false
1410 458 Acked 1457 Write 1 New Done Done false
1411 459 Acked 1458 Read 1 New Done Done false
1412 460 Acked 1459 Write 10 New Done Done false
1413 461 Acked 1460 Flush 0 New Done Done false
1414 462 Acked 1461 Write 1 New Done Done false
1415 463 Acked 1462 Write 3 New Done Done false
1416 464 Acked 1463 Read 6 New Done Done false
1417 465 Acked 1464 Flush 0 New Done Done false
1418 466 Acked 1465 Read 1 New Done Done false
1419 467 Acked 1466 Read 10 New Done Done false
1420 468 Acked 1467 Flush 0 New Done Done false
1421 469 Acked 1468 Read 1 New Done Done false
1422 470 Acked 1469 Read 5 New Done Done false
1423 471 Acked 1470 Read 7 New Done Done false
1424 472 Acked 1471 Read 8 New Done Done false
1425 473 Acked 1472 Write 9 New Done Done false
1426 474 Acked 1473 Read 8 New Done Done false
1427 475 Acked 1474 Read 10 New Done Done false
1428 476 Acked 1475 Write 7 New Done Done false
1429 477 Acked 1476 Write 8 New Done Done false
1430 478 Acked 1477 Write 6 New Done Done false
1431 479 Acked 1478 Write 6 New Done Done false
1432 480 Acked 1479 Write 7 New Done Done false
1433 481 Acked 1480 Write 7 New Done Done false
1434 482 Acked 1481 Read 10 New Done Done false
1435 483 Acked 1482 Write 2 New Done Done false
1436 484 Acked 1483 Write 1 New Done Done false
1437 485 Acked 1484 Read 6 New Done Done false
1438 486 Acked 1485 Flush 0 New Done Done false
1439 487 Acked 1486 Read 10 New Done Done false
1440 488 Acked 1487 Read 10 New Done Done false
1441 489 Acked 1488 Read 5 New Done Done false
1442 490 Acked 1489 Read 9 New Done Done false
1443 491 Acked 1490 Read 2 New Done Done false
1444 492 Acked 1491 Read 4 New Done Done false
1445 493 Acked 1492 Flush 0 New Done Done false
1446 494 Acked 1493 Read 5 New Done Done false
1447 495 Acked 1494 Read 10 New Done Done false
1448 496 Acked 1495 Read 2 New Done Done false
1449 497 Acked 1496 Read 3 New Done Done false
1450 498 Acked 1497 Write 6 New Done Done false
1451 499 Acked 1498 Write 3 New Done Done false
1452 500 Acked 1499 Read 2 New Done Done false
1453 501 Acked 1500 Write 7 New Done Done false
1454 502 Acked 1501 Write 9 New Done Done false
1455 503 Acked 1502 Read 9 New Done Done false
1456 504 Acked 1503 Read 2 New Done Done false
1457 505 Acked 1504 Write 8 New Done Done false
1458 506 Acked 1505 Read 8 New Done Done false
1459 507 Acked 1506 Read 8 New Done Done false
1460 508 Acked 1507 Write 4 New Done Done false
1461 509 Acked 1508 Flush 0 New Done Done false
1462 510 Acked 1509 Read 10 New Done Done false
1463 511 Acked 1510 Read 6 New Done Done false
1464 512 Acked 1511 Read 4 New Done Done false
1465 513 Acked 1512 Read 7 New Done Done false
1466 514 Acked 1513 Read 1 New Done Done false
1467 515 Acked 1514 Write 10 New Done Done false
1468 516 Acked 1515 Write 10 New Done Done false
1469 517 Acked 1516 Write 8 New Done Done false
1470 518 Acked 1517 Read 9 New Done Done false
1471 519 Acked 1518 Read 3 New Done Done false
1472 520 Acked 1519 Read 4 New Done Done false
1473 521 Acked 1520 Write 9 New Done Done false
1474 522 Acked 1521 Read 2 New Done Done false
1475 523 Acked 1522 Write 2 New Done Done false
1476 524 Acked 1523 Read 1 New Done Done false
1477 525 Acked 1524 Read 6 New Done Done false
1478 526 Acked 1525 Write 2 New Done Done false
1479 527 Acked 1526 Write 10 New Done Done false
1480 528 Acked 1527 Write 3 New Done Done false
1481 529 Acked 1528 Read 5 New Done Done false
1482 530 Acked 1529 Read 2 New Done Done false
1483 531 Acked 1530 Write 8 New Done Done false
1484 532 Acked 1531 Read 4 New Done Done false
1485 533 Acked 1532 Read 2 New Done Done false
1486 534 Acked 1533 Read 8 New Done Done false
1487 535 Acked 1534 Write 8 New Done Done false
1488 536 Acked 1535 Write 3 New Done Done false
1489 537 Acked 1536 Write 5 New Done Done false
1490 538 Acked 1537 Write 1 New Done Done false
1491 539 Acked 1538 Read 7 New Done Done false
1492 540 Acked 1539 Read 10 New Done Done false
1493 541 Acked 1540 Read 1 New Done Done false
1494 542 Acked 1541 Write 2 New Done Done false
1495 543 Acked 1542 Read 3 New Done Done false
1496 544 Acked 1543 Write 4 New Done Done false
1497 545 Acked 1544 Write 10 New Done Done false
1498 546 Acked 1545 Flush 0 New Done Done false
1499 547 Acked 1546 Read 10 New Done Done false
1500 548 Acked 1547 Write 6 New Done Done false
1501 549 Acked 1548 Read 4 New Done Done false
1502 550 Acked 1549 Write 3 New Done Done false
1503 551 Acked 1550 Flush 0 New Done Done false
1504 552 Acked 1551 Write 7 New Done Done false
1505 553 Acked 1552 Write 5 New Done Done false
1506 554 Acked 1553 Read 1 New Done Done false
1507 555 Acked 1554 Read 5 New Done Done false
1508 556 Acked 1555 Read 3 New Done Done false
1509 557 Acked 1556 Read 4 New Done Done false
1510 558 Acked 1557 Write 1 New Done Done false
1511 559 Acked 1558 Read 3 New Done Done false
1512 560 Acked 1559 Write 2 New Done Done false
1513 561 Acked 1560 Read 5 New Done Done false
1514 562 Acked 1561 Read 7 New Done Done false
1515 563 Acked 1562 Read 2 New Done Done false
1516 564 Acked 1563 Read 1 New Done Done false
1517 565 Acked 1564 Write 6 New Done Done false
1518 566 Acked 1565 Write 6 New Done Done false
1519 567 Acked 1566 Write 8 New Done Done false
1520 568 Acked 1567 Flush 0 New Done Done false
1521 569 Acked 1568 Write 8 New Done Done false
1522 570 Acked 1569 Flush 0 New Done Done false
1523 571 Acked 1570 Read 9 New Done Done false
1524 572 Acked 1571 Flush 0 New Done Done false
1525 573 Acked 1572 Read 6 New Done Done false
1526 574 Acked 1573 Write 6 New Done Done false
1527 575 Acked 1574 Read 7 New Done Done false
1528 576 Acked 1575 Read 10 New Done Done false
1529 577 Acked 1576 Flush 0 New Done Done false
1530 578 Acked 1577 Read 3 New Done Done false
1531 579 Acked 1578 Read 8 New Done Done false
1532 580 Acked 1579 Write 1 New Done Done false
1533 581 Acked 1580 Write 10 New Done Done false
1534 582 Acked 1581 Write 9 New Done Done false
1535 583 Acked 1582 Read 3 New Done Done false
1536 584 Acked 1583 Write 2 New Done Done false
1537 585 Acked 1584 Write 10 New Done Done false
1538 586 Acked 1585 Read 8 New Done Done false
1539 587 Acked 1586 Write 4 New Done Done false
1540 588 Acked 1587 Read 1 New Done Done false
1541 589 Acked 1588 Write 4 New Done Done false
1542 590 Acked 1589 Write 6 New Done Done false
1543 591 Acked 1590 Read 9 New Done Done false
1544 592 Acked 1591 Read 9 New Done Done false
1545 593 Acked 1592 Flush 0 New Done Done false
1546 594 Acked 1593 Read 9 New Done Done false
1547 595 Acked 1594 Write 1 New Done Done false
1548 596 Acked 1595 Write 1 New Done Done false
1549 597 Acked 1596 Read 8 New Done Done false
1550 598 Acked 1597 Read 10 New Done Done false
1551 599 Acked 1598 Read 1 New Done Done false
1552 600 Acked 1599 Flush 0 New Done Done false
1553 601 Acked 1600 Read 6 New Done Done false
1554 602 Acked 1601 Write 1 New Done Done false
1555 603 Acked 1602 Flush 0 New Done Done false
1556 604 Acked 1603 Read 5 New Done Done false
1557 STATES DS:0 DS:1 DS:2 TOTAL
1558 New 301 0 0 301
1559 Sent 0 0 0 0
1560 Done 0 301 301 602
1561 Skipped 0 0 0 0
1562 Error 0 0 0 0
1563 Last Flush: 1302 1602 1602
1564 Downstairs last five completed: 1302 1301 1300 1299 1298 1297
1565 Upstairs last five completed: 604 603 602 601 600 599
1566 CLIENT: Up:0 ds:301 act:2
1567 [8810] got start action
1568 [8810] I am going to restart
1569 Starting downstairs at port 8810
1570 Make output file at "/tmp/dsc/downstairs-8810.txt"
1571 [8810][0] reports Starting
1572 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1178)
1573 [8810][0] reports Running
15742023-10-13T18:09:23.101ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 0
15752023-10-13T18:09:23.101ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state Offline
15762023-10-13T18:09:23.101ZINFOcrucible: [0] client re-new 302 jobs since flush 1302 = downstairs
15772023-10-13T18:09:23.102ZINFOcrucible: [0] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
15782023-10-13T18:09:23.103ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
15792023-10-13T18:09:23.103ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
15802023-10-13T18:09:23.103ZINFOcrucible: Returning client:0 UUID:12345678-0000-0000-0000-000000008810 matches
15812023-10-13T18:09:23.103ZINFOcrucible: [0] send last flush ID to this DS: 1302
15822023-10-13T18:09:23.103ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
15832023-10-13T18:09:23.103ZINFOcrucible: [0] Replied this last flush ID: 1302
15842023-10-13T18:09:23.103ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Offline Active Active ds_transition to Replay
15852023-10-13T18:09:23.103ZINFOcrucible: [0] Transition from Offline to Replay
15862023-10-13T18:09:23.103ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
15872023-10-13T18:09:23.103ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Replay Active Active ds_transition to Active
15882023-10-13T18:09:23.103ZINFOcrucible: [0] Transition from Replay to Active
15892023-10-13T18:09:23.103ZINFOcrucible: [0] Starts cmd_loop
15902023-10-13T18:09:23.103ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
15912023-10-13T18:09:23.103ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
15922023-10-13T18:09:23.103ZINFOcrucible: Set check for repair
15932023-10-13T18:09:24.104ZINFOcrucible: Checking if live repair is needed
15942023-10-13T18:09:24.104ZINFOcrucible: No Live Repair required at this time
15952023-10-13T18:09:24.104ZWARNcrucible: [0] flow control end
1596 ----------------------------------------------------------------
1597 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1598 Downstairs last five completed: 1604 1603 1602 1601 1600 1599
1599 Upstairs last five completed: 605 604 603 602 601 600
1600 CLIENT: Up:0 ds:0 act:3
1601 Replay: All jobs finished, all DS active.
1602 Main task has work to do, go find it
1603 got dsc Stop(0)
1604 stop 0
1605 Oct 13 18:09:25.053 INFO request completed, latency_us: 498, response_code: 204[8810] Got stop action so:false kr:false
1606 , uri: /stop/cid/0, method: POST, req_id: 536d55d4-13d5-4338-a2ae-796e90ceee62, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
1607 Oct 13 18:09:25.054 INFO request completed, latency_us: 201, response_code: 200, uri: /state/cid/0, method: GET, req_id: ce1ecb0f-dde8-4e11-8e2c-dfeb42eee356, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
16082023-10-13T18:09:25.055ZWARNcrucible: [0] downstairs disconnected
16092023-10-13T18:09:25.055ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
16102023-10-13T18:09:25.055ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8810 closed looper = 0
16112023-10-13T18:09:25.056ZWARNcrucible: [0] pm_task rx.recv() is None
16122023-10-13T18:09:25.056ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:false
16132023-10-13T18:09:25.056ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
16142023-10-13T18:09:25.056ZINFOcrucible: [0] 127.0.0.1:8810 task reports offline
1615 [8810] Exited with: ExitStatus(unix_wait_status(9))
1616 [8810][0] reports Exit
16172023-10-13T18:09:28.060ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
1618 Oct 13 18:09:29.057 INFO request completed, latency_us: 280, response_code: 200, uri: /state/cid/0, method: GET, req_id: 750a9243-1bf2-4267-a330-903f72ecb8af, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
1619 001/300 Read block 1452 len 12288
1620 002/300 Read block 2790 len 24576
1621 003/300 Write block 3055 len 32768 data: 2 2 2 2 2 2 2 2
1622 004/300 Flush
1623 005/300 Write block 3143 len 20480 data: 3 2 2 2 2
1624 006/300 Read block 4964 len 28672
1625 007/300 Read block 2443 len 16384
1626 008/300 Write block 95 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1627 009/300 Read block 3926 len 8192
1628 010/300 Write block 1227 len 28672 data: 2 2 2 2 2 2 2
1629 011/300 Flush
1630 012/300 Read block 1214 len 28672
1631 013/300 Read block 3727 len 8192
1632 014/300 Read block 1191 len 36864
1633 015/300 Read block 4361 len 20480
1634 016/300 Read block 3819 len 24576
1635 017/300 Read block 918 len 24576
1636 018/300 Read block 3379 len 12288
1637 019/300 Flush
1638 020/300 Write block 2136 len 28672 data: 2 2 2 2 2 2 2
1639 021/300 Write block 3336 len 16384 data: 2 2 2 2
1640 022/300 Read block 1052 len 4096
1641 023/300 Write block 1934 len 20480 data: 3 2 2 4 3
1642 024/300 Read block 4851 len 12288
1643 025/300 Write block 3435 len 12288 data: 2 2 2
1644 026/300 Write block 4673 len 40960 data: 2 2 2 3 3 2 2 2 2 2
1645 027/300 Write block 70 len 20480 data: 2 2 2 2 2
1646 028/300 Flush
1647 029/300 Write block 1618 len 20480 data: 2 2 2 2 2
1648 030/300 Write block 1154 len 28672 data: 2 2 2 2 2 2 2
1649 031/300 Read block 1530 len 32768
1650 032/300 Write block 2588 len 12288 data: 2 2 2
1651 033/300 Flush
1652 034/300 Write block 2424 len 24576 data: 2 2 2 2 2 2
1653 035/300 Read block 3981 len 24576
1654 036/300 Write block 2575 len 4096 data: 2
1655 037/300 Read block 2650 len 24576
1656 038/300 Read block 2322 len 40960
1657 039/300 Flush
1658 040/300 Write block 4858 len 36864 data: 2 2 2 2 2 2 2 2 2
1659 041/300 Write block 3013 len 36864 data: 2 2 2 2 2 2 2 2 2
1660 042/300 Read block 3485 len 4096
1661 043/300 Write block 2112 len 12288 data: 2 2 2
1662 044/300 Read block 3262 len 24576
1663 045/300 Write block 374 len 8192 data: 2 2
1664 046/300 Read block 2135 len 28672
1665 047/300 Read block 2052 len 28672
1666 048/300 Read block 1018 len 40960
1667 049/300 Write block 665 len 8192 data: 3 4
1668 050/300 Write block 2406 len 32768 data: 2 2 2 2 2 2 2 2
1669 051/300 Read block 1056 len 28672
1670 052/300 Write block 4405 len 40960 data: 3 3 3 2 2 2 2 2 2 2
1671 053/300 Write block 2235 len 8192 data: 2 2
1672 054/300 Read block 209 len 20480
1673 055/300 Flush
1674 056/300 Write block 2866 len 36864 data: 3 3 3 3 3 2 2 2 2
1675 057/300 Read block 1374 len 28672
1676 058/300 Read block 2764 len 16384
1677 059/300 Read block 2009 len 12288
1678 060/300 Read block 71 len 20480
1679 061/300 Write block 4231 len 8192 data: 3 2
1680 062/300 Read block 168 len 4096
1681 063/300 Write block 2952 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1682 064/300 Write block 2508 len 32768 data: 3 2 2 2 2 2 2 2
1683 065/300 Read block 1339 len 16384
1684 066/300 Flush
1685 067/300 Read block 4555 len 24576
1686 068/300 Read block 2755 len 32768
1687 069/300 Read block 4894 len 4096
1688 070/300 Read block 552 len 8192
1689 071/300 Read block 3803 len 4096
1690 072/300 Write block 2015 len 32768 data: 3 2 2 2 2 2 2 2
1691 073/300 Read block 8 len 20480
1692 074/300 Write block 2277 len 28672 data: 2 2 3 3 3 3 2
1693 075/300 Read block 153 len 28672
1694 076/300 Read block 2895 len 36864
1695 077/300 Write block 1160 len 36864 data: 3 2 3 3 3 3 3 2 2
1696 078/300 Write block 4422 len 12288 data: 2 2 2
1697 079/300 Write block 4678 len 16384 data: 3 3 3 3
1698 080/300 Read block 3248 len 32768
1699 081/300 Read block 1279 len 28672
1700 082/300 Read block 693 len 40960
1701 083/300 Write block 4842 len 4096 data: 3
1702 084/300 Read block 3858 len 4096
1703 085/300 Write block 188 len 24576 data: 2 2 3 3 3 3
1704 086/300 Write block 4495 len 28672 data: 2 2 2 2 2 2 2
1705 087/300 Write block 3306 len 20480 data: 2 2 2 2 2
1706 088/300 Read block 4143 len 40960
1707 089/300 Read block 4678 len 20480
1708 090/300 Read block 3405 len 40960
1709 091/300 Write block 4335 len 36864 data: 3 2 2 2 2 2 2 2 2
1710 092/300 Read block 4398 len 12288
1711 093/300 Write block 2280 len 12288 data: 4 4 4
1712 094/300 Write block 102 len 4096 data: 3
1713 095/300 Read block 4558 len 8192
1714 096/300 Read block 1907 len 40960
1715 097/300 Write block 1169 len 20480 data: 2 2 2 2 2
1716 098/300 Read block 494 len 32768
1717 099/300 Write block 2639 len 20480 data: 2 2 2 2 2
1718 100/300 Write block 3969 len 8192 data: 2 2
1719 101/300 Read block 2004 len 40960
1720 102/300 Read block 973 len 28672
1721 103/300 Read block 166 len 40960
1722 104/300 Write block 923 len 12288 data: 2 2 2
1723 105/300 Flush
1724 106/300 Read block 2548 len 36864
1725 107/300 Read block 713 len 36864
1726 108/300 Read block 3292 len 8192
1727 109/300 Read block 3304 len 24576
1728 110/300 Flush
1729 111/300 Read block 2830 len 28672
1730 112/300 Flush
1731 113/300 Read block 759 len 40960
1732 114/300 Read block 179 len 36864
1733 115/300 Read block 3037 len 16384
1734 116/300 Read block 2518 len 36864
1735 117/300 Read block 2920 len 28672
1736 118/300 Flush
1737 119/300 Write block 212 len 28672 data: 2 2 2 2 2 2 3
1738 120/300 Read block 3338 len 12288
1739 121/300 Write block 4387 len 8192 data: 2 2
1740 122/300 Flush
1741 123/300 Read block 3562 len 12288
1742 124/300 Write block 1809 len 4096 data: 2
1743 125/300 Write block 2666 len 24576 data: 2 2 2 2 2 2
1744 126/300 Read block 4695 len 36864
1745 127/300 Write block 3233 len 36864 data: 2 2 2 2 2 2 2 2 2
1746 128/300 Write block 4329 len 28672 data: 2 2 2 2 3 3 4
1747 129/300 Read block 4095 len 40960
1748 130/300 Write block 102 len 16384 data: 4 3 3 2
1749 131/300 Read block 595 len 16384
1750 132/300 Read block 4424 len 4096
1751 133/300 Write block 984 len 24576 data: 2 2 2 2 2 2
1752 134/300 Read block 4057 len 12288
1753 135/300 Write block 4794 len 16384 data: 2 2 2 2
1754 136/300 Read block 929 len 12288
1755 137/300 Read block 718 len 8192
1756 138/300 Read block 2975 len 12288
1757 139/300 Read block 2971 len 36864
1758 140/300 Write block 2388 len 20480 data: 2 2 2 2 2
1759 141/300 Read block 396 len 12288
1760 142/300 Write block 4677 len 16384 data: 4 4 4 4
1761 143/300 Read block 4593 len 16384
1762 144/300 Write block 1412 len 12288 data: 2 2 2
1763 145/300 Read block 3327 len 32768
1764 146/300 Write block 719 len 28672 data: 2 2 2 2 2 2 2
1765 147/300 Read block 3090 len 16384
1766 148/300 Write block 3359 len 4096 data: 2
1767 149/300 Write block 4845 len 4096 data: 3
1768 150/300 Write block 2343 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1769 151/300 Read block 2473 len 4096
1770 152/300 Read block 3513 len 20480
1771 153/300 Read block 597 len 36864
1772 154/300 Read block 1153 len 20480
1773 155/300 Flush
1774 156/300 Read block 252 len 8192
1775 157/300 Read block 803 len 36864
1776 158/300 Write block 3711 len 32768 data: 4 3 2 3 3 3 3 3
1777 159/300 Read block 1410 len 20480
1778 160/300 Write block 3338 len 8192 data: 3 3
1779 161/300 Write block 2883 len 40960 data: 2 2 4 4 3 3 2 2 2 2
1780 162/300 Read block 4846 len 20480
1781 163/300 Write block 2392 len 32768 data: 3 2 2 2 3 4 4 4
1782 164/300 Flush
1783 165/300 Read block 3118 len 8192
1784 166/300 Read block 2522 len 8192
1785 167/300 Write block 476 len 8192 data: 2 2
1786 168/300 Write block 1948 len 20480 data: 3 3 3 3 3
1787 169/300 Write block 2200 len 40960 data: 3 3 3 2 2 2 2 2 2 3
1788 170/300 Write block 4526 len 8192 data: 3 3
1789 171/300 Flush
1790 172/300 Write block 597 len 32768 data: 2 2 2 3 3 3 3 3
1791 173/300 Write block 751 len 36864 data: 2 2 3 3 2 2 2 2 3
1792 174/300 Flush
1793 175/300 Read block 929 len 40960
1794 176/300 Write block 1234 len 28672 data: 2 2 2 2 2 2 2
1795 177/300 Read block 3277 len 36864
1796 178/300 Read block 312 len 24576
1797 179/300 Read block 3871 len 8192
1798 180/300 Write block 2869 len 40960 data: 4 4 3 3 3 3 2 2 2 2
1799 181/300 Write block 553 len 32768 data: 2 2 2 2 3 3 3 3
1800 182/300 Read block 2314 len 28672
1801 183/300 Write block 2461 len 28672 data: 2 2 2 2 3 3 3
1802 184/300 Write block 2942 len 28672 data: 2 3 3 3 3 3 2
1803 185/300 Read block 4246 len 24576
1804 186/300 Read block 2734 len 4096
1805 187/300 Write block 211 len 24576 data: 2 3 3 3 3 3
1806 188/300 Write block 4810 len 20480 data: 2 2 2 2 2
1807 189/300 Read block 751 len 20480
1808 190/300 Read block 1002 len 24576
1809 191/300 Write block 3096 len 28672 data: 3 3 3 3 3 2 2
1810 192/300 Read block 4191 len 4096
1811 193/300 Write block 1662 len 16384 data: 3 3 3 2
1812 194/300 Write block 2050 len 20480 data: 2 2 2 2 2
1813 195/300 Read block 1789 len 8192
1814 196/300 Write block 3330 len 24576 data: 4 3 3 2 2 2
1815 197/300 Write block 1084 len 16384 data: 2 2 2 2
1816 198/300 Read block 4593 len 36864
1817 199/300 Read block 3254 len 20480
1818 200/300 Read block 1983 len 28672
1819 201/300 Read block 2121 len 4096
1820 202/300 Read block 3719 len 24576
1821 203/300 Flush
1822 204/300 Write block 4922 len 16384 data: 2 2 2 2
1823 205/300 Flush
1824 206/300 Read block 262 len 36864
1825 207/300 Read block 2530 len 12288
1826 208/300 Read block 2210 len 16384
1827 209/300 Write block 3100 len 12288 data: 4 3 3
1828 210/300 Write block 1416 len 36864 data: 2 2 2 2 2 2 2 2 2
1829 211/300 Read block 4136 len 12288
1830 212/300 Read block 1017 len 36864
1831 213/300 Write block 3083 len 8192 data: 2 2
1832 214/300 Write block 3162 len 12288 data: 2 2 2
1833 215/300 Write block 3105 len 36864 data: 2 2 2 2 2 2 2 2 2
1834 216/300 Read block 1874 len 20480
1835 217/300 Flush
1836 218/300 Read block 398 len 32768
1837 219/300 Read block 4318 len 16384
1838 220/300 Read block 2920 len 40960
1839 221/300 Read block 3344 len 40960
1840 222/300 Read block 3165 len 8192
1841 223/300 Read block 2886 len 4096
1842 224/300 Read block 4410 len 40960
1843 225/300 Read block 229 len 28672
1844 226/300 Read block 3184 len 24576
1845 227/300 Write block 4283 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1846 228/300 Read block 1265 len 16384
1847 229/300 Write block 1486 len 36864 data: 2 2 2 2 2 3 3 3 3
1848 230/300 Write block 401 len 4096 data: 2
1849 231/300 Read block 253 len 16384
1850 232/300 Write block 1147 len 12288 data: 2 3 3
1851 233/300 Read block 2694 len 4096
1852 234/300 Read block 3112 len 40960
1853 235/300 Read block 4082 len 28672
1854 236/300 Write block 4461 len 4096 data: 2
1855 237/300 Read block 3965 len 4096
1856 238/300 Write block 1483 len 28672 data: 2 2 2 3 3 3 3
1857 239/300 Read block 299 len 16384
1858 240/300 Flush
1859 241/300 Read block 2064 len 36864
1860 242/300 Write block 1809 len 20480 data: 3 2 2 2 2
1861 243/300 Write block 4519 len 20480 data: 2 2 2 2 2
1862 244/300 Read block 1425 len 12288
1863 245/300 Read block 3615 len 32768
1864 246/300 Write block 4189 len 4096 data: 3
1865 247/300 Write block 3821 len 12288 data: 2 2 2
1866 248/300 Write block 3708 len 32768 data: 3 3 4 5 4 3 4 4
1867 249/300 Read block 1464 len 32768
1868 250/300 Write block 3802 len 16384 data: 2 2 2 2
1869 251/300 Write block 3210 len 20480 data: 3 3 3 3 3
1870 252/300 Write block 2339 len 24576 data: 2 2 2 2 3 3
1871 253/300 Write block 2935 len 40960 data: 2 2 3 3 3 3 3 3 4 4
1872 254/300 Flush
1873 255/300 Write block 3769 len 12288 data: 2 2 2
1874 256/300 Flush
1875 257/300 Flush
1876 258/300 Read block 3856 len 12288
1877 259/300 Write block 2806 len 20480 data: 3 3 4 3 3
1878 260/300 Flush
1879 261/300 Read block 4671 len 8192
1880 262/300 Read block 3027 len 40960
1881 263/300 Write block 1197 len 32768 data: 2 2 2 2 2 2 2 2
1882 264/300 Write block 4815 len 40960 data: 2 2 2 2 2 2 2 3 3 3
1883 265/300 Read block 870 len 12288
1884 266/300 Write block 441 len 12288 data: 2 2 2
1885 267/300 Flush
1886 268/300 Read block 710 len 16384
1887 269/300 Write block 4684 len 20480 data: 2 2 2 2 2
1888 270/300 Write block 535 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1889 271/300 Read block 343 len 12288
1890 272/300 Write block 4800 len 16384 data: 4 4 4 4
1891 273/300 Read block 406 len 24576
1892 274/300 Read block 3194 len 8192
1893 275/300 Read block 1366 len 12288
1894 276/300 Write block 1155 len 40960 data: 3 3 3 3 3 4 3 4 4 4
1895 277/300 Read block 4420 len 20480
1896 278/300 Read block 2674 len 12288
1897 279/300 Write block 3937 len 40960 data: 4 4 4 3 3 3 3 3 2 3
1898 280/300 Read block 3359 len 40960
1899 281/300 Read block 3016 len 8192
1900 282/300 Read block 2359 len 4096
1901 283/300 Flush
1902 284/300 Read block 3391 len 20480
1903 285/300 Write block 572 len 16384 data: 2 2 2 2
1904 286/300 Read block 808 len 36864
1905 287/300 Write block 1611 len 24576 data: 2 2 2 2 2 2
1906 288/300 Write block 3459 len 16384 data: 3 3 2 2
1907 289/300 Read block 974 len 28672
1908 290/300 Read block 4043 len 12288
1909 291/300 Flush
1910 292/300 Read block 2869 len 20480
1911 293/300 Read block 4172 len 28672
1912 294/300 Flush
1913 295/300 Write block 3631 len 4096 data: 2
1914 296/300 Read block 4523 len 4096
1915 297/300 Read block 4694 len 16384
1916 298/300 Write block 4550 len 24576 data: 2 2 2 2 2 2
1917 299/300 Read block 2583 len 8192
1918 300/300 Write block 286 len 8192 data: 2 2
1919 Main task has work to do, go find it
1920 got dsc Start(0)
1921 start 0
1922 Oct 13 18:09:30.016 INFO request completed, latency_us: 300, response_code: 204, uri: /start/cid/0, method: POST, req_id: bc3e0503-be62-48b5-b272-7798d376fce5, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
1923 Replay: started 0, returned:Ok(())
1924 ----------------------------------------------------------------
1925 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1926 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1927 606 Acked 1605 Read 3 New Done Done false
1928 607 Acked 1606 Read 6 New Done Done false
1929 608 Acked 1607 Write 8 New Done Done false
1930 609 Acked 1608 Flush 0 New Done Done false
1931 610 Acked 1609 Write 5 New Done Done false
1932 611 Acked 1610 Read 7 New Done Done false
1933 612 Acked 1611 Read 4 New Done Done false
1934 613 Acked 1612 Write 10 New Done Done false
1935 614 Acked 1613 Read 2 New Done Done false
1936 615 Acked 1614 Flush 0 New Done Done false
1937 616 Acked 1615 Write 7 New Done Done false
1938 617 Acked 1616 Flush 0 New Done Done false
1939 618 Acked 1617 Read 7 New Done Done false
1940 619 Acked 1618 Read 2 New Done Done false
1941 620 Acked 1619 Read 9 New Done Done false
1942 621 Acked 1620 Read 5 New Done Done false
1943 622 Acked 1621 Read 6 New Done Done false
1944 623 Acked 1622 Read 6 New Done Done false
1945 624 Acked 1623 Read 3 New Done Done false
1946 625 Acked 1624 Flush 0 New Done Done false
1947 626 Acked 1625 Write 7 New Done Done false
1948 627 Acked 1626 Write 4 New Done Done false
1949 628 Acked 1627 Read 1 New Done Done false
1950 629 Acked 1628 Write 5 New Done Done false
1951 630 Acked 1629 Read 3 New Done Done false
1952 631 Acked 1630 Write 3 New Done Done false
1953 632 Acked 1631 Write 10 New Done Done false
1954 633 Acked 1632 Write 5 New Done Done false
1955 634 Acked 1633 Flush 0 New Done Done false
1956 635 Acked 1634 Write 5 New Done Done false
1957 636 Acked 1635 Write 7 New Done Done false
1958 637 Acked 1636 Read 8 New Done Done false
1959 638 Acked 1637 Write 3 New Done Done false
1960 639 Acked 1638 Flush 0 New Done Done false
1961 640 Acked 1639 Write 6 New Done Done false
1962 641 Acked 1640 Read 6 New Done Done false
1963 642 Acked 1641 Write 1 New Done Done false
1964 643 Acked 1642 Read 6 New Done Done false
1965 644 Acked 1643 Read 10 New Done Done false
1966 645 Acked 1644 Flush 0 New Done Done false
1967 646 Acked 1645 Write 9 New Done Done false
1968 647 Acked 1646 Write 9 New Done Done false
1969 648 Acked 1647 Read 1 New Done Done false
1970 649 Acked 1648 Write 3 New Done Done false
1971 650 Acked 1649 Read 6 New Done Done false
1972 651 Acked 1650 Write 2 New Done Done false
1973 652 Acked 1651 Read 7 New Done Done false
1974 653 Acked 1652 Read 7 New Done Done false
1975 654 Acked 1653 Read 10 New Done Done false
1976 655 Acked 1654 Write 2 New Done Done false
1977 656 Acked 1655 Write 8 New Done Done false
1978 657 Acked 1656 Read 7 New Done Done false
1979 658 Acked 1657 Write 10 New Done Done false
1980 659 Acked 1658 Write 2 New Done Done false
1981 660 Acked 1659 Read 5 New Done Done false
1982 661 Acked 1660 Flush 0 New Done Done false
1983 662 Acked 1661 Write 9 New Done Done false
1984 663 Acked 1662 Read 7 New Done Done false
1985 664 Acked 1663 Read 4 New Done Done false
1986 665 Acked 1664 Read 3 New Done Done false
1987 666 Acked 1665 Read 5 New Done Done false
1988 667 Acked 1666 Write 2 New Done Done false
1989 668 Acked 1667 Read 1 New Done Done false
1990 669 Acked 1668 Write 10 New Done Done false
1991 670 Acked 1669 Write 8 New Done Done false
1992 671 Acked 1670 Read 4 New Done Done false
1993 672 Acked 1671 Flush 0 New Done Done false
1994 673 Acked 1672 Read 6 New Done Done false
1995 674 Acked 1673 Read 8 New Done Done false
1996 675 Acked 1674 Read 1 New Done Done false
1997 676 Acked 1675 Read 2 New Done Done false
1998 677 Acked 1676 Read 1 New Done Done false
1999 678 Acked 1677 Write 8 New Done Done false
2000 679 Acked 1678 Read 5 New Done Done false
2001 680 Acked 1679 Write 7 New Done Done false
2002 681 Acked 1680 Read 7 New Done Done false
2003 682 Acked 1681 Read 9 New Done Done false
2004 683 Acked 1682 Write 9 New Done Done false
2005 684 Acked 1683 Write 3 New Done Done false
2006 685 Acked 1684 Write 4 New Done Done false
2007 686 Acked 1685 Read 8 New Done Done false
2008 687 Acked 1686 Read 7 New Done Done false
2009 688 Acked 1687 Read 10 New Done Done false
2010 689 Acked 1688 Write 1 New Done Done false
2011 690 Acked 1689 Read 1 New Done Done false
2012 691 Acked 1690 Write 6 New Done Done false
2013 692 Acked 1691 Write 7 New Done Done false
2014 693 Acked 1692 Write 5 New Done Done false
2015 694 Acked 1693 Read 10 New Done Done false
2016 695 Acked 1694 Read 5 New Done Done false
2017 696 Acked 1695 Read 10 New Done Done false
2018 697 Acked 1696 Write 9 New Done Done false
2019 698 Acked 1697 Read 3 New Done Done false
2020 699 Acked 1698 Write 3 New Done Done false
2021 700 Acked 1699 Write 1 New Done Done false
2022 701 Acked 1700 Read 2 New Done Done false
2023 702 Acked 1701 Read 10 New Done Done false
2024 703 Acked 1702 Write 5 New Done Done false
2025 704 Acked 1703 Read 8 New Done Done false
2026 705 Acked 1704 Write 5 New Done Done false
2027 706 Acked 1705 Write 2 New Done Done false
2028 707 Acked 1706 Read 10 New Done Done false
2029 708 Acked 1707 Read 7 New Done Done false
2030 709 Acked 1708 Read 10 New Done Done false
2031 710 Acked 1709 Write 3 New Done Done false
2032 711 Acked 1710 Flush 0 New Done Done false
2033 712 Acked 1711 Read 9 New Done Done false
2034 713 Acked 1712 Read 9 New Done Done false
2035 714 Acked 1713 Read 2 New Done Done false
2036 715 Acked 1714 Read 6 New Done Done false
2037 716 Acked 1715 Flush 0 New Done Done false
2038 717 Acked 1716 Read 7 New Done Done false
2039 718 Acked 1717 Flush 0 New Done Done false
2040 719 Acked 1718 Read 10 New Done Done false
2041 720 Acked 1719 Read 9 New Done Done false
2042 721 Acked 1720 Read 4 New Done Done false
2043 722 Acked 1721 Read 9 New Done Done false
2044 723 Acked 1722 Read 7 New Done Done false
2045 724 Acked 1723 Flush 0 New Done Done false
2046 725 Acked 1724 Write 7 New Done Done false
2047 726 Acked 1725 Read 3 New Done Done false
2048 727 Acked 1726 Write 2 New Done Done false
2049 728 Acked 1727 Flush 0 New Done Done false
2050 729 Acked 1728 Read 3 New Done Done false
2051 730 Acked 1729 Write 1 New Done Done false
2052 731 Acked 1730 Write 6 New Done Done false
2053 732 Acked 1731 Read 9 New Done Done false
2054 733 Acked 1732 Write 9 New Done Done false
2055 734 Acked 1733 Write 7 New Done Done false
2056 735 Acked 1734 Read 10 New Done Done false
2057 736 Acked 1735 Write 4 New Done Done false
2058 737 Acked 1736 Read 4 New Done Done false
2059 738 Acked 1737 Read 1 New Done Done false
2060 739 Acked 1738 Write 6 New Done Done false
2061 740 Acked 1739 Read 3 New Done Done false
2062 741 Acked 1740 Write 4 New Done Done false
2063 742 Acked 1741 Read 3 New Done Done false
2064 743 Acked 1742 Read 2 New Done Done false
2065 744 Acked 1743 Read 3 New Done Done false
2066 745 Acked 1744 Read 9 New Done Done false
2067 746 Acked 1745 Write 5 New Done Done false
2068 747 Acked 1746 Read 3 New Done Done false
2069 748 Acked 1747 Write 4 New Done Done false
2070 749 Acked 1748 Read 4 New Done Done false
2071 750 Acked 1749 Write 3 New Done Done false
2072 751 Acked 1750 Read 8 New Done Done false
2073 752 Acked 1751 Write 7 New Done Done false
2074 753 Acked 1752 Read 4 New Done Done false
2075 754 Acked 1753 Write 1 New Done Done false
2076 755 Acked 1754 Write 1 New Done Done false
2077 756 Acked 1755 Write 10 New Done Done false
2078 757 Acked 1756 Read 1 New Done Done false
2079 758 Acked 1757 Read 5 New Done Done false
2080 759 Acked 1758 Read 9 New Done Done false
2081 760 Acked 1759 Read 5 New Done Done false
2082 761 Acked 1760 Flush 0 New Done Done false
2083 762 Acked 1761 Read 2 New Done Done false
2084 763 Acked 1762 Read 9 New Done Done false
2085 764 Acked 1763 Write 8 New Done Done false
2086 765 Acked 1764 Read 5 New Done Done false
2087 766 Acked 1765 Write 2 New Done Done false
2088 767 Acked 1766 Write 10 New Done Done false
2089 768 Acked 1767 Read 5 New Done Done false
2090 769 Acked 1768 Write 8 New Done Done false
2091 770 Acked 1769 Flush 0 New Done Done false
2092 771 Acked 1770 Read 2 New Done Done false
2093 772 Acked 1771 Read 2 New Done Done false
2094 773 Acked 1772 Write 2 New Done Done false
2095 774 Acked 1773 Write 5 New Done Done false
2096 775 Acked 1774 Write 10 New Done Done false
2097 776 Acked 1775 Write 2 New Done Done false
2098 777 Acked 1776 Flush 0 New Done Done false
2099 778 Acked 1777 Write 8 New Done Done false
2100 779 Acked 1778 Write 9 New Done Done false
2101 780 Acked 1779 Flush 0 New Done Done false
2102 781 Acked 1780 Read 10 New Done Done false
2103 782 Acked 1781 Write 7 New Done Done false
2104 783 Acked 1782 Read 9 New Done Done false
2105 784 Acked 1783 Read 6 New Done Done false
2106 785 Acked 1784 Read 2 New Done Done false
2107 786 Acked 1785 Write 10 New Done Done false
2108 787 Acked 1786 Write 8 New Done Done false
2109 788 Acked 1787 Read 7 New Done Done false
2110 789 Acked 1788 Write 7 New Done Done false
2111 790 Acked 1789 Write 7 New Done Done false
2112 791 Acked 1790 Read 6 New Done Done false
2113 792 Acked 1791 Read 1 New Done Done false
2114 793 Acked 1792 Write 6 New Done Done false
2115 794 Acked 1793 Write 5 New Done Done false
2116 795 Acked 1794 Read 5 New Done Done false
2117 796 Acked 1795 Read 6 New Done Done false
2118 797 Acked 1796 Write 7 New Done Done false
2119 798 Acked 1797 Read 1 New Done Done false
2120 799 Acked 1798 Write 4 New Done Done false
2121 800 Acked 1799 Write 5 New Done Done false
2122 801 Acked 1800 Read 2 New Done Done false
2123 802 Acked 1801 Write 6 New Done Done false
2124 803 Acked 1802 Write 4 New Done Done false
2125 804 Acked 1803 Read 9 New Done Done false
2126 805 Acked 1804 Read 5 New Done Done false
2127 806 Acked 1805 Read 7 New Done Done false
2128 807 Acked 1806 Read 1 New Done Done false
2129 808 Acked 1807 Read 6 New Done Done false
2130 809 Acked 1808 Flush 0 New Done Done false
2131 810 Acked 1809 Write 4 New Done Done false
2132 811 Acked 1810 Flush 0 New Done Done false
2133 812 Acked 1811 Read 9 New Done Done false
2134 813 Acked 1812 Read 3 New Done Done false
2135 814 Acked 1813 Read 4 New Done Done false
2136 815 Acked 1814 Write 3 New Done Done false
2137 816 Acked 1815 Write 9 New Done Done false
2138 817 Acked 1816 Read 3 New Done Done false
2139 818 Acked 1817 Read 9 New Done Done false
2140 819 Acked 1818 Write 2 New Done Done false
2141 820 Acked 1819 Write 3 New Done Done false
2142 821 Acked 1820 Write 9 New Done Done false
2143 822 Acked 1821 Read 5 New Done Done false
2144 823 Acked 1822 Flush 0 New Done Done false
2145 824 Acked 1823 Read 8 New Done Done false
2146 825 Acked 1824 Read 4 New Done Done false
2147 826 Acked 1825 Read 10 New Done Done false
2148 827 Acked 1826 Read 10 New Done Done false
2149 828 Acked 1827 Read 2 New Done Done false
2150 829 Acked 1828 Read 1 New Done Done false
2151 830 Acked 1829 Read 10 New Done Done false
2152 831 Acked 1830 Read 7 New Done Done false
2153 832 Acked 1831 Read 6 New Done Done false
2154 833 Acked 1832 Write 10 New Done Done false
2155 834 Acked 1833 Read 4 New Done Done false
2156 835 Acked 1834 Write 9 New Done Done false
2157 836 Acked 1835 Write 1 New Done Done false
2158 837 Acked 1836 Read 4 New Done Done false
2159 838 Acked 1837 Write 3 New Done Done false
2160 839 Acked 1838 Read 1 New Done Done false
2161 840 Acked 1839 Read 10 New Done Done false
2162 841 Acked 1840 Read 7 New Done Done false
2163 842 Acked 1841 Write 1 New Done Done false
2164 843 Acked 1842 Read 1 New Done Done false
2165 844 Acked 1843 Write 7 New Done Done false
2166 845 Acked 1844 Read 4 New Done Done false
2167 846 Acked 1845 Flush 0 New Done Done false
2168 847 Acked 1846 Read 9 New Done Done false
2169 848 Acked 1847 Write 5 New Done Done false
2170 849 Acked 1848 Write 5 New Done Done false
2171 850 Acked 1849 Read 3 New Done Done false
2172 851 Acked 1850 Read 8 New Done Done false
2173 852 Acked 1851 Write 1 New Done Done false
2174 853 Acked 1852 Write 3 New Done Done false
2175 854 Acked 1853 Write 8 New Done Done false
2176 855 Acked 1854 Read 8 New Done Done false
2177 856 Acked 1855 Write 4 New Done Done false
2178 857 Acked 1856 Write 5 New Done Done false
2179 858 Acked 1857 Write 6 New Done Done false
2180 859 Acked 1858 Write 10 New Done Done false
2181 860 Acked 1859 Flush 0 New Done Done false
2182 861 Acked 1860 Write 3 New Done Done false
2183 862 Acked 1861 Flush 0 New Done Done false
2184 863 Acked 1862 Flush 0 New Done Done false
2185 864 Acked 1863 Read 3 New Done Done false
2186 865 Acked 1864 Write 5 New Done Done false
2187 866 Acked 1865 Flush 0 New Done Done false
2188 867 Acked 1866 Read 2 New Done Done false
2189 868 Acked 1867 Read 10 New Done Done false
2190 869 Acked 1868 Write 8 New Done Done false
2191 870 Acked 1869 Write 10 New Done Done false
2192 871 Acked 1870 Read 3 New Done Done false
2193 872 Acked 1871 Write 3 New Done Done false
2194 873 Acked 1872 Flush 0 New Done Done false
2195 874 Acked 1873 Read 4 New Done Done false
2196 875 Acked 1874 Write 5 New Done Done false
2197 876 Acked 1875 Write 10 New Done Done false
2198 877 Acked 1876 Read 3 New Done Done false
2199 878 Acked 1877 Write 4 New Done Done false
2200 879 Acked 1878 Read 6 New Done Done false
2201 880 Acked 1879 Read 2 New Done Done false
2202 881 Acked 1880 Read 3 New Done Done false
2203 882 Acked 1881 Write 10 New Done Done false
2204 883 Acked 1882 Read 5 New Done Done false
2205 884 Acked 1883 Read 3 New Done Done false
2206 885 Acked 1884 Write 10 New Done Done false
2207 886 Acked 1885 Read 10 New Done Done false
2208 887 Acked 1886 Read 2 New Done Done false
2209 888 Acked 1887 Read 1 New Done Done false
2210 889 Acked 1888 Flush 0 New Done Done false
2211 890 Acked 1889 Read 5 New Done Done false
2212 891 Acked 1890 Write 4 New Done Done false
2213 892 Acked 1891 Read 9 New Done Done false
2214 893 Acked 1892 Write 6 New Done Done false
2215 894 Acked 1893 Write 4 New Done Done false
2216 895 Acked 1894 Read 7 New Done Done false
2217 896 Acked 1895 Read 3 New Done Done false
2218 897 Acked 1896 Flush 0 New Done Done false
2219 898 Acked 1897 Read 5 New Done Done false
2220 899 Acked 1898 Read 7 New Done Done false
2221 900 Acked 1899 Flush 0 New Done Done false
2222 901 Acked 1900 Write 1 New Done Done false
2223 902 Acked 1901 Read 1 New Done Done false
2224 903 Acked 1902 Read 4 New Done Done false
2225 904 Acked 1903 Write 6 New Done Done false
2226 905 Acked 1904 Read 2 New Done Done false
2227 906 Acked 1905 Write 2 New Sent Sent false
2228 STATES DS:0 DS:1 DS:2 TOTAL
2229 New 301 0 0 301
2230 Sent 0 1 1 2
2231 Done 0 300 300 600
2232 Skipped 0 0 0 0
2233 Error 0 0 0 0
2234 Last Flush: 1604 1899 1899
2235 Downstairs last five completed: 1604 1603 1602 1601 1600 1599
2236 Upstairs last five completed: 906 905 904 903 902 901
2237 CLIENT: Up:0 ds:301 act:2
2238 [8810] got start action
2239 [8810] I am going to restart
2240 Starting downstairs at port 8810
2241 Make output file at "/tmp/dsc/downstairs-8810.txt"
2242 [8810][0] reports Starting
2243 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1179)
2244 [8810][0] reports Running
22452023-10-13T18:09:32.065ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 0
22462023-10-13T18:09:32.065ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state Offline
22472023-10-13T18:09:32.065ZINFOcrucible: [0] client re-new 302 jobs since flush 1604 = downstairs
22482023-10-13T18:09:32.066ZINFOcrucible: [0] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
22492023-10-13T18:09:32.067ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
22502023-10-13T18:09:32.067ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
22512023-10-13T18:09:32.067ZINFOcrucible: Returning client:0 UUID:12345678-0000-0000-0000-000000008810 matches
22522023-10-13T18:09:32.067ZINFOcrucible: [0] send last flush ID to this DS: 1604
22532023-10-13T18:09:32.067ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
22542023-10-13T18:09:32.067ZINFOcrucible: [0] Replied this last flush ID: 1604
22552023-10-13T18:09:32.067ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Offline Active Active ds_transition to Replay
22562023-10-13T18:09:32.067ZINFOcrucible: [0] Transition from Offline to Replay
22572023-10-13T18:09:32.067ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
22582023-10-13T18:09:32.067ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Replay Active Active ds_transition to Active
22592023-10-13T18:09:32.067ZINFOcrucible: [0] Transition from Replay to Active
22602023-10-13T18:09:32.067ZINFOcrucible: [0] Starts cmd_loop
22612023-10-13T18:09:32.067ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
22622023-10-13T18:09:32.067ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
22632023-10-13T18:09:32.067ZINFOcrucible: Set check for repair
22642023-10-13T18:09:33.068ZINFOcrucible: Checking if live repair is needed
22652023-10-13T18:09:33.068ZINFOcrucible: No Live Repair required at this time
22662023-10-13T18:09:33.068ZWARNcrucible: [0] flow control end
2267 ----------------------------------------------------------------
2268 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2269 Downstairs last five completed: 1906 1905 1904 1903 1902 1901
2270 Upstairs last five completed: 907 906 905 904 903 902
2271 CLIENT: Up:0 ds:0 act:3
2272 Replay: All jobs finished, all DS active.
2273 Main task has work to do, go find it
2274 got dsc Stop(2)
2275 stop 2
2276 Oct 13 18:09:34.020 INFO request completed, latency_us: 472, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 7d394bd1-25c3-4982-a71a-dd1902a0b53a, remote_addr[8830] Got stop action so:false kr:false
2277 : 127.0.0.1:64571, local_addr: 127.0.0.1:9998
2278 Oct 13 18:09:34.021 INFO request completed, latency_us: 190, response_code: 200, uri: /state/cid/2, method: GET, req_id: ddc45acb-d2f7-4e6d-b776-f01399bfd41a, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
22792023-10-13T18:09:34.022ZWARNcrucible: [2] downstairs disconnected
22802023-10-13T18:09:34.022ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
22812023-10-13T18:09:34.022ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8830 closed looper = 2
22822023-10-13T18:09:34.022ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
22832023-10-13T18:09:34.022ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
22842023-10-13T18:09:34.022ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
22852023-10-13T18:09:34.022ZWARNcrucible: [2] pm_task rx.recv() is None
2286 [8830] Exited with: ExitStatus(unix_wait_status(9))
2287 [8830][2] reports Exit
22882023-10-13T18:09:37.026ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
2289 Oct 13 18:09:38.024 INFO request completed, latency_us: 270, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2c0f4619-3fab-4b23-8eb2-16340f6664af, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
2290 001/300 Read block 2822 len 24576
2291 002/300 Read block 118 len 4096
2292 003/300 Flush
2293 004/300 Write block 2461 len 24576 data: 3 3 3 3 4 4
2294 005/300 Read block 57 len 24576
2295 006/300 Write block 3726 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2296 007/300 Write block 3440 len 8192 data: 3 3
2297 008/300 Read block 4722 len 4096
2298 009/300 Read block 4414 len 40960
2299 010/300 Write block 693 len 8192 data: 2 2
2300 011/300 Read block 1040 len 36864
2301 012/300 Flush
2302 013/300 Read block 1898 len 4096
2303 014/300 Write block 4303 len 24576 data: 2 2 2 2 2 2
2304 015/300 Write block 2974 len 8192 data: 2 2
2305 016/300 Write block 770 len 36864 data: 2 2 2 3 3 3 3 3 3
2306 017/300 Write block 2260 len 4096 data: 2
2307 018/300 Read block 2759 len 20480
2308 019/300 Flush
2309 020/300 Read block 4444 len 12288
2310 021/300 Write block 2325 len 28672 data: 2 2 3 3 4 4 4
2311 022/300 Write block 222 len 20480 data: 2 3 3 3 3
2312 023/300 Write block 1020 len 8192 data: 2 2
2313 024/300 Read block 3425 len 4096
2314 025/300 Read block 4232 len 20480
2315 026/300 Read block 4014 len 24576
2316 027/300 Read block 3052 len 8192
2317 028/300 Write block 2899 len 24576 data: 2 2 2 2 2 2
2318 029/300 Write block 4112 len 16384 data: 2 2 2 2
2319 030/300 Read block 3160 len 16384
2320 031/300 Flush
2321 032/300 Write block 2158 len 16384 data: 3 3 2 2
2322 033/300 Write block 1638 len 28672 data: 2 2 2 2 2 2 2
2323 034/300 Read block 1226 len 32768
2324 035/300 Read block 1762 len 8192
2325 036/300 Read block 1128 len 32768
2326 037/300 Read block 594 len 16384
2327 038/300 Write block 2960 len 28672 data: 3 3 2 2 2 2 2
2328 039/300 Read block 3992 len 8192
2329 040/300 Read block 4224 len 12288
2330 041/300 Read block 1984 len 28672
2331 042/300 Write block 1995 len 8192 data: 4 4
2332 043/300 Write block 1692 len 24576 data: 2 2 2 2 2 2
2333 044/300 Write block 1198 len 32768 data: 3 3 3 3 3 3 3 2
2334 045/300 Write block 4211 len 20480 data: 2 2 2 2 2
2335 046/300 Read block 2735 len 12288
2336 047/300 Read block 155 len 36864
2337 048/300 Write block 1466 len 12288 data: 2 2 2
2338 049/300 Write block 4803 len 36864 data: 5 4 3 3 3 3 2 3 3
2339 050/300 Read block 4579 len 40960
2340 051/300 Read block 341 len 24576
2341 052/300 Write block 126 len 32768 data: 2 3 3 3 3 3 3 3
2342 053/300 Write block 4701 len 8192 data: 2 2
2343 054/300 Read block 1781 len 4096
2344 055/300 Read block 4704 len 16384
2345 056/300 Read block 3958 len 24576
2346 057/300 Read block 1012 len 4096
2347 058/300 Write block 4752 len 8192 data: 2 2
2348 059/300 Read block 1064 len 36864
2349 060/300 Write block 4694 len 36864 data: 2 2 2 2 2 2 2 3 3
2350 061/300 Write block 576 len 8192 data: 2 2
2351 062/300 Read block 4416 len 24576
2352 063/300 Read block 2872 len 40960
2353 064/300 Read block 2860 len 12288
2354 065/300 Write block 2833 len 8192 data: 2 2
2355 066/300 Write block 3965 len 4096 data: 2
2356 067/300 Flush
2357 068/300 Flush
2358 069/300 Write block 2584 len 32768 data: 2 3 3 2 3 3 3 2
2359 070/300 Flush
2360 071/300 Read block 40 len 32768
2361 072/300 Write block 1632 len 36864 data: 2 2 2 2 2 2 3 3 3
2362 073/300 Write block 1448 len 12288 data: 2 2 2
2363 074/300 Write block 370 len 40960 data: 2 2 2 2 3 3 2 2 2 2
2364 075/300 Flush
2365 076/300 Read block 1275 len 20480
2366 077/300 Write block 1485 len 36864 data: 3 4 4 4 4 3 4 4 4
2367 078/300 Read block 1605 len 24576
2368 079/300 Read block 1142 len 32768
2369 080/300 Write block 4901 len 4096 data: 2
2370 081/300 Read block 4391 len 40960
2371 082/300 Read block 1356 len 12288
2372 083/300 Read block 3836 len 28672
2373 084/300 Write block 4217 len 36864 data: 2 2 2 2 2 2 2 2 2
2374 085/300 Write block 2301 len 32768 data: 2 2 2 2 2 3 3 3
2375 086/300 Read block 3673 len 40960
2376 087/300 Read block 62 len 40960
2377 088/300 Read block 4729 len 20480
2378 089/300 Write block 3249 len 36864 data: 2 2 2 2 2 2 2 2 2
2379 090/300 Write block 557 len 8192 data: 4 4
2380 091/300 Read block 2807 len 24576
2381 092/300 Read block 1589 len 36864
2382 093/300 Read block 1762 len 16384
2383 094/300 Read block 4342 len 32768
2384 095/300 Write block 691 len 16384 data: 2 2 3 3
2385 096/300 Write block 1979 len 8192 data: 2 2
2386 097/300 Write block 1125 len 20480 data: 2 2 2 2 3
2387 098/300 Read block 1463 len 12288
2388 099/300 Read block 2159 len 12288
2389 100/300 Read block 2027 len 4096
2390 101/300 Flush
2391 102/300 Read block 184 len 4096
2392 103/300 Read block 743 len 32768
2393 104/300 Read block 2926 len 28672
2394 105/300 Write block 1918 len 40960 data: 2 2 2 2 2 2 3 3 3 3
2395 106/300 Write block 3155 len 24576 data: 2 2 2 2 2 2
2396 107/300 Read block 162 len 32768
2397 108/300 Write block 4777 len 24576 data: 2 2 2 2 2 2
2398 109/300 Flush
2399 110/300 Write block 3573 len 12288 data: 2 2 2
2400 111/300 Read block 127 len 32768
2401 112/300 Read block 344 len 8192
2402 113/300 Write block 3874 len 12288 data: 3 3 3
2403 114/300 Flush
2404 115/300 Write block 603 len 16384 data: 4 4 3 3
2405 116/300 Write block 487 len 32768 data: 3 3 2 2 3 2 2 2
2406 117/300 Read block 2984 len 28672
2407 118/300 Read block 3808 len 28672
2408 119/300 Write block 2649 len 8192 data: 2 2
2409 120/300 Write block 494 len 12288 data: 3 2 2
2410 121/300 Read block 2895 len 36864
2411 122/300 Read block 1517 len 24576
2412 123/300 Read block 3373 len 8192
2413 124/300 Write block 3912 len 12288 data: 4 4 4
2414 125/300 Read block 4346 len 16384
2415 126/300 Write block 707 len 28672 data: 2 2 2 2 2 2 2
2416 127/300 Read block 3952 len 4096
2417 128/300 Read block 305 len 16384
2418 129/300 Write block 2757 len 16384 data: 2 2 2 3
2419 130/300 Write block 1005 len 40960 data: 2 2 2 2 2 2 2 3 2 2
2420 131/300 Write block 722 len 4096 data: 3
2421 132/300 Read block 446 len 32768
2422 133/300 Read block 878 len 20480
2423 134/300 Write block 3916 len 40960 data: 3 3 2 2 2 2 2 2 2 2
2424 135/300 Write block 1789 len 20480 data: 3 3 3 3 2
2425 136/300 Write block 2733 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2426 137/300 Write block 2000 len 36864 data: 2 2 2 2 3 3 3 4 4
2427 138/300 Read block 1769 len 12288
2428 139/300 Read block 2951 len 4096
2429 140/300 Write block 4118 len 4096 data: 2
2430 141/300 Flush
2431 142/300 Read block 2912 len 8192
2432 143/300 Flush
2433 144/300 Read block 3453 len 28672
2434 145/300 Read block 3378 len 12288
2435 146/300 Write block 4960 len 20480 data: 2 2 2 2 2
2436 147/300 Read block 1938 len 32768
2437 148/300 Read block 234 len 20480
2438 149/300 Flush
2439 150/300 Write block 1451 len 28672 data: 2 2 2 2 2 2 2
2440 151/300 Read block 261 len 16384
2441 152/300 Flush
2442 153/300 Read block 3576 len 36864
2443 154/300 Read block 804 len 36864
2444 155/300 Read block 4799 len 36864
2445 156/300 Read block 1811 len 36864
2446 157/300 Read block 4061 len 4096
2447 158/300 Write block 980 len 16384 data: 2 2 2 2
2448 159/300 Write block 672 len 4096 data: 2
2449 160/300 Write block 2874 len 4096 data: 4
2450 161/300 Read block 4064 len 24576
2451 162/300 Read block 3514 len 12288
2452 163/300 Read block 4215 len 8192
2453 164/300 Write block 1691 len 36864 data: 2 3 3 3 3 3 3 2 2
2454 165/300 Write block 130 len 16384 data: 4 4 4 4
2455 166/300 Read block 2104 len 12288
2456 167/300 Read block 4158 len 32768
2457 168/300 Read block 4976 len 20480
2458 169/300 Read block 3687 len 20480
2459 170/300 Write block 929 len 12288 data: 2 2 2
2460 171/300 Write block 3802 len 40960 data: 3 3 3 3 2 2 2 2 2 2
2461 172/300 Write block 2993 len 32768 data: 2 2 2 2 2 2 2 2
2462 173/300 Write block 1114 len 36864 data: 2 2 3 3 3 3 3 3 3
2463 174/300 Read block 4212 len 24576
2464 175/300 Read block 2339 len 32768
2465 176/300 Write block 2863 len 12288 data: 3 3 3
2466 177/300 Read block 2388 len 32768
2467 178/300 Read block 2306 len 4096
2468 179/300 Write block 1876 len 12288 data: 2 2 2
2469 180/300 Read block 4304 len 32768
2470 181/300 Read block 383 len 32768
2471 182/300 Read block 1943 len 16384
2472 183/300 Read block 1739 len 8192
2473 184/300 Read block 1114 len 16384
2474 185/300 Read block 4031 len 8192
2475 186/300 Read block 3324 len 12288
2476 187/300 Read block 3831 len 36864
2477 188/300 Write block 1028 len 12288 data: 2 2 2
2478 189/300 Read block 1396 len 16384
2479 190/300 Read block 3915 len 12288
2480 191/300 Write block 2129 len 4096 data: 2
2481 192/300 Read block 79 len 40960
2482 193/300 Write block 806 len 16384 data: 3 3 3 2
2483 194/300 Read block 2589 len 8192
2484 195/300 Read block 3959 len 4096
2485 196/300 Read block 3346 len 16384
2486 197/300 Read block 320 len 20480
2487 198/300 Write block 1500 len 16384 data: 2 2 2 2
2488 199/300 Read block 1728 len 4096
2489 200/300 Read block 2592 len 36864
2490 201/300 Write block 4424 len 20480 data: 3 2 2 2 3
2491 202/300 Write block 3569 len 36864 data: 3 2 2 2 3 3 3 2 2
2492 203/300 Read block 1065 len 24576
2493 204/300 Write block 3545 len 28672 data: 2 2 2 2 2 2 2
2494 205/300 Read block 3053 len 32768
2495 206/300 Read block 1616 len 8192
2496 207/300 Write block 320 len 32768 data: 3 3 3 3 3 3 3 3
2497 208/300 Read block 3002 len 16384
2498 209/300 Write block 2516 len 8192 data: 2 2
2499 210/300 Write block 2599 len 32768 data: 3 3 3 3 2 2 2 2
2500 211/300 Read block 1001 len 36864
2501 212/300 Write block 4160 len 40960 data: 2 2 2 3 3 3 3 3 3 3
2502 213/300 Read block 335 len 12288
2503 214/300 Write block 86 len 40960 data: 2 3 3 3 2 2 2 2 2 3
2504 215/300 Read block 1766 len 28672
2505 216/300 Write block 1163 len 4096 data: 5
2506 217/300 Flush
2507 218/300 Flush
2508 219/300 Write block 1047 len 20480 data: 2 2 2 3 3
2509 220/300 Read block 4402 len 24576
2510 221/300 Write block 1096 len 28672 data: 3 3 3 3 3 3 3
2511 222/300 Write block 1279 len 24576 data: 2 2 2 2 2 2
2512 223/300 Write block 2350 len 32768 data: 3 3 3 2 2 2 2 2
2513 224/300 Read block 61 len 28672
2514 225/300 Read block 3439 len 8192
2515 226/300 Write block 460 len 16384 data: 2 2 2 2
2516 227/300 Read block 476 len 4096
2517 228/300 Read block 2024 len 32768
2518 229/300 Write block 3109 len 32768 data: 3 3 3 3 3 2 2 2
2519 230/300 Read block 607 len 36864
2520 231/300 Flush
2521 232/300 Write block 4606 len 24576 data: 2 2 3 2 2 2
2522 233/300 Read block 1150 len 40960
2523 234/300 Read block 3370 len 40960
2524 235/300 Read block 4031 len 16384
2525 236/300 Write block 2845 len 20480 data: 2 2 2 2 2
2526 237/300 Write block 1292 len 8192 data: 2 2
2527 238/300 Flush
2528 239/300 Read block 969 len 4096
2529 240/300 Read block 4881 len 16384
2530 241/300 Read block 4898 len 16384
2531 242/300 Write block 1064 len 20480 data: 2 2 2 2 2
2532 243/300 Write block 2104 len 8192 data: 2 2
2533 244/300 Read block 4157 len 20480
2534 245/300 Flush
2535 246/300 Write block 4176 len 12288 data: 2 2 2
2536 247/300 Write block 424 len 32768 data: 2 2 2 2 2 2 2 2
2537 248/300 Read block 814 len 32768
2538 249/300 Flush
2539 250/300 Write block 688 len 28672 data: 2 2 2 3 3 4 4
2540 251/300 Write block 3818 len 24576 data: 2 2 2 3 3 3
2541 252/300 Read block 432 len 36864
2542 253/300 Write block 2671 len 36864 data: 3 2 2 2 2 2 2 2 2
2543 254/300 Read block 373 len 32768
2544 255/300 Read block 1107 len 36864
2545 256/300 Write block 3874 len 40960 data: 4 4 4 3 3 3 3 3 2 2
2546 257/300 Write block 4859 len 8192 data: 3 3
2547 258/300 Write block 1500 len 4096 data: 3
2548 259/300 Write block 3115 len 16384 data: 3 3 2 2
2549 260/300 Read block 3829 len 8192
2550 261/300 Read block 388 len 28672
2551 262/300 Read block 2831 len 32768
2552 263/300 Read block 223 len 16384
2553 264/300 Read block 4968 len 12288
2554 265/300 Read block 181 len 4096
2555 266/300 Flush
2556 267/300 Read block 2858 len 28672
2557 268/300 Write block 4567 len 36864 data: 2 2 2 2 2 2 2 2 2
2558 269/300 Read block 3828 len 4096
2559 270/300 Write block 4310 len 28672 data: 2 2 2 2 2 2 2
2560 271/300 Write block 1841 len 12288 data: 2 2 2
2561 272/300 Write block 3160 len 8192 data: 3 2
2562 273/300 Flush
2563 274/300 Read block 786 len 12288
2564 275/300 Write block 2120 len 12288 data: 3 3 3
2565 276/300 Write block 282 len 32768 data: 2 2 2 2 3 3 2 2
2566 277/300 Flush
2567 278/300 Write block 2824 len 36864 data: 2 2 2 2 2 2 2 2 2
2568 279/300 Write block 917 len 24576 data: 2 2 2 2 2 2
2569 280/300 Write block 568 len 16384 data: 2 2 2 2
2570 281/300 Write block 1088 len 8192 data: 2 2
2571 282/300 Read block 781 len 28672
2572 283/300 Read block 828 len 16384
2573 284/300 Read block 4566 len 28672
2574 285/300 Write block 4280 len 28672 data: 2 2 2 3 3 3 3
2575 286/300 Read block 812 len 8192
2576 287/300 Read block 1177 len 28672
2577 288/300 Read block 2796 len 36864
2578 289/300 Read block 1204 len 36864
2579 290/300 Write block 193 len 16384 data: 4 3 2 2
2580 291/300 Read block 4970 len 36864
2581 292/300 Read block 3129 len 32768
2582 293/300 Write block 2364 len 16384 data: 2 2 2 3
2583 294/300 Read block 1678 len 28672
2584 295/300 Read block 4429 len 12288
2585 296/300 Read block 4887 len 20480
2586 297/300 Read block 3351 len 8192
2587 298/300 Read block 1802 len 40960
2588 299/300 Write block 284 len 12288 data: 3 3 4
2589 300/300 Read block 871 len 4096
2590 Main task has work to do, go find it
2591 got dsc Start(2)
2592 start 2
2593 Oct 13 18:09:38.983 INFO request completed, latency_us: 302, response_code: 204, uri: /start/cid/2, method: POST, req_id: fa7350bf-9ae5-4168-ae9e-ee729cfc3a52, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
2594 Replay: started 2, returned:Ok(())
2595 ----------------------------------------------------------------
2596 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
2597 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2598 908 Acked 1907 Read 6 Done Done New false
2599 909 Acked 1908 Read 1 Done Done New false
2600 910 Acked 1909 Flush 0 Done Done New false
2601 911 Acked 1910 Write 6 Done Done New false
2602 912 Acked 1911 Read 6 Done Done New false
2603 913 Acked 1912 Write 10 Done Done New false
2604 914 Acked 1913 Write 2 Done Done New false
2605 915 Acked 1914 Read 1 Done Done New false
2606 916 Acked 1915 Read 10 Done Done New false
2607 917 Acked 1916 Write 2 Done Done New false
2608 918 Acked 1917 Read 9 Done Done New false
2609 919 Acked 1918 Flush 0 Done Done New false
2610 920 Acked 1919 Read 1 Done Done New false
2611 921 Acked 1920 Write 6 Done Done New false
2612 922 Acked 1921 Write 2 Done Done New false
2613 923 Acked 1922 Write 9 Done Done New false
2614 924 Acked 1923 Write 1 Done Done New false
2615 925 Acked 1924 Read 5 Done Done New false
2616 926 Acked 1925 Flush 0 Done Done New false
2617 927 Acked 1926 Read 3 Done Done New false
2618 928 Acked 1927 Write 7 Done Done New false
2619 929 Acked 1928 Write 5 Done Done New false
2620 930 Acked 1929 Write 2 Done Done New false
2621 931 Acked 1930 Read 1 Done Done New false
2622 932 Acked 1931 Read 5 Done Done New false
2623 933 Acked 1932 Read 6 Done Done New false
2624 934 Acked 1933 Read 2 Done Done New false
2625 935 Acked 1934 Write 6 Done Done New false
2626 936 Acked 1935 Write 4 Done Done New false
2627 937 Acked 1936 Read 4 Done Done New false
2628 938 Acked 1937 Flush 0 Done Done New false
2629 939 Acked 1938 Flush 0 Done Done New false
2630 940 Acked 1939 Write 4 Done Done New false
2631 941 Acked 1940 Write 7 Done Done New false
2632 942 Acked 1941 Read 8 Done Done New false
2633 943 Acked 1942 Read 2 Done Done New false
2634 944 Acked 1943 Read 8 Done Done New false
2635 945 Acked 1944 Read 4 Done Done New false
2636 946 Acked 1945 Write 7 Done Done New false
2637 947 Acked 1946 Read 2 Done Done New false
2638 948 Acked 1947 Read 3 Done Done New false
2639 949 Acked 1948 Read 7 Done Done New false
2640 950 Acked 1949 Write 2 Done Done New false
2641 951 Acked 1950 Write 6 Done Done New false
2642 952 Acked 1951 Write 8 Done Done New false
2643 953 Acked 1952 Write 5 Done Done New false
2644 954 Acked 1953 Read 3 Done Done New false
2645 955 Acked 1954 Read 9 Done Done New false
2646 956 Acked 1955 Write 3 Done Done New false
2647 957 Acked 1956 Write 9 Done Done New false
2648 958 Acked 1957 Read 10 Done Done New false
2649 959 Acked 1958 Read 6 Done Done New false
2650 960 Acked 1959 Write 8 Done Done New false
2651 961 Acked 1960 Write 2 Done Done New false
2652 962 Acked 1961 Read 1 Done Done New false
2653 963 Acked 1962 Read 4 Done Done New false
2654 964 Acked 1963 Read 6 Done Done New false
2655 965 Acked 1964 Read 1 Done Done New false
2656 966 Acked 1965 Write 2 Done Done New false
2657 967 Acked 1966 Read 9 Done Done New false
2658 968 Acked 1967 Write 9 Done Done New false
2659 969 Acked 1968 Write 2 Done Done New false
2660 970 Acked 1969 Read 6 Done Done New false
2661 971 Acked 1970 Read 10 Done Done New false
2662 972 Acked 1971 Read 3 Done Done New false
2663 973 Acked 1972 Write 2 Done Done New false
2664 974 Acked 1973 Write 1 Done Done New false
2665 975 Acked 1974 Flush 0 Done Done New false
2666 976 Acked 1975 Flush 0 Done Done New false
2667 977 Acked 1976 Write 8 Done Done New false
2668 978 Acked 1977 Flush 0 Done Done New false
2669 979 Acked 1978 Read 8 Done Done New false
2670 980 Acked 1979 Write 9 Done Done New false
2671 981 Acked 1980 Write 3 Done Done New false
2672 982 Acked 1981 Write 10 Done Done New false
2673 983 Acked 1982 Flush 0 Done Done New false
2674 984 Acked 1983 Read 5 Done Done New false
2675 985 Acked 1984 Write 9 Done Done New false
2676 986 Acked 1985 Read 6 Done Done New false
2677 987 Acked 1986 Read 8 Done Done New false
2678 988 Acked 1987 Write 1 Done Done New false
2679 989 Acked 1988 Read 10 Done Done New false
2680 990 Acked 1989 Read 3 Done Done New false
2681 991 Acked 1990 Read 7 Done Done New false
2682 992 Acked 1991 Write 9 Done Done New false
2683 993 Acked 1992 Write 8 Done Done New false
2684 994 Acked 1993 Read 10 Done Done New false
2685 995 Acked 1994 Read 10 Done Done New false
2686 996 Acked 1995 Read 5 Done Done New false
2687 997 Acked 1996 Write 9 Done Done New false
2688 998 Acked 1997 Write 2 Done Done New false
2689 999 Acked 1998 Read 6 Done Done New false
2690 1000 Acked 1999 Read 9 Done Done New false
2691 1001 Acked 2000 Read 4 Done Done New false
2692 1002 Acked 2001 Read 8 Done Done New false
2693 1003 Acked 2002 Write 4 Done Done New false
2694 1004 Acked 2003 Write 2 Done Done New false
2695 1005 Acked 2004 Write 5 Done Done New false
2696 1006 Acked 2005 Read 3 Done Done New false
2697 1007 Acked 2006 Read 3 Done Done New false
2698 1008 Acked 2007 Read 1 Done Done New false
2699 1009 Acked 2008 Flush 0 Done Done New false
2700 1010 Acked 2009 Read 1 Done Done New false
2701 1011 Acked 2010 Read 8 Done Done New false
2702 1012 Acked 2011 Read 7 Done Done New false
2703 1013 Acked 2012 Write 10 Done Done New false
2704 1014 Acked 2013 Write 6 Done Done New false
2705 1015 Acked 2014 Read 8 Done Done New false
2706 1016 Acked 2015 Write 6 Done Done New false
2707 1017 Acked 2016 Flush 0 Done Done New false
2708 1018 Acked 2017 Write 3 Done Done New false
2709 1019 Acked 2018 Read 8 Done Done New false
2710 1020 Acked 2019 Read 2 Done Done New false
2711 1021 Acked 2020 Write 3 Done Done New false
2712 1022 Acked 2021 Flush 0 Done Done New false
2713 1023 Acked 2022 Write 4 Done Done New false
2714 1024 Acked 2023 Write 8 Done Done New false
2715 1025 Acked 2024 Read 7 Done Done New false
2716 1026 Acked 2025 Read 7 Done Done New false
2717 1027 Acked 2026 Write 2 Done Done New false
2718 1028 Acked 2027 Write 3 Done Done New false
2719 1029 Acked 2028 Read 9 Done Done New false
2720 1030 Acked 2029 Read 6 Done Done New false
2721 1031 Acked 2030 Read 2 Done Done New false
2722 1032 Acked 2031 Write 3 Done Done New false
2723 1033 Acked 2032 Read 4 Done Done New false
2724 1034 Acked 2033 Write 7 Done Done New false
2725 1035 Acked 2034 Read 1 Done Done New false
2726 1036 Acked 2035 Read 4 Done Done New false
2727 1037 Acked 2036 Write 4 Done Done New false
2728 1038 Acked 2037 Write 10 Done Done New false
2729 1039 Acked 2038 Write 1 Done Done New false
2730 1040 Acked 2039 Read 8 Done Done New false
2731 1041 Acked 2040 Read 5 Done Done New false
2732 1042 Acked 2041 Write 10 Done Done New false
2733 1043 Acked 2042 Write 5 Done Done New false
2734 1044 Acked 2043 Write 10 Done Done New false
2735 1045 Acked 2044 Write 9 Done Done New false
2736 1046 Acked 2045 Read 3 Done Done New false
2737 1047 Acked 2046 Read 1 Done Done New false
2738 1048 Acked 2047 Write 1 Done Done New false
2739 1049 Acked 2048 Flush 0 Done Done New false
2740 1050 Acked 2049 Read 2 Done Done New false
2741 1051 Acked 2050 Flush 0 Done Done New false
2742 1052 Acked 2051 Read 7 Done Done New false
2743 1053 Acked 2052 Read 3 Done Done New false
2744 1054 Acked 2053 Write 5 Done Done New false
2745 1055 Acked 2054 Read 8 Done Done New false
2746 1056 Acked 2055 Read 5 Done Done New false
2747 1057 Acked 2056 Flush 0 Done Done New false
2748 1058 Acked 2057 Write 7 Done Done New false
2749 1059 Acked 2058 Read 4 Done Done New false
2750 1060 Acked 2059 Flush 0 Done Done New false
2751 1061 Acked 2060 Read 9 Done Done New false
2752 1062 Acked 2061 Read 9 Done Done New false
2753 1063 Acked 2062 Read 9 Done Done New false
2754 1064 Acked 2063 Read 9 Done Done New false
2755 1065 Acked 2064 Read 1 Done Done New false
2756 1066 Acked 2065 Write 4 Done Done New false
2757 1067 Acked 2066 Write 1 Done Done New false
2758 1068 Acked 2067 Write 1 Done Done New false
2759 1069 Acked 2068 Read 6 Done Done New false
2760 1070 Acked 2069 Read 3 Done Done New false
2761 1071 Acked 2070 Read 2 Done Done New false
2762 1072 Acked 2071 Write 9 Done Done New false
2763 1073 Acked 2072 Write 4 Done Done New false
2764 1074 Acked 2073 Read 3 Done Done New false
2765 1075 Acked 2074 Read 8 Done Done New false
2766 1076 Acked 2075 Read 5 Done Done New false
2767 1077 Acked 2076 Read 5 Done Done New false
2768 1078 Acked 2077 Write 3 Done Done New false
2769 1079 Acked 2078 Write 10 Done Done New false
2770 1080 Acked 2079 Write 8 Done Done New false
2771 1081 Acked 2080 Write 9 Done Done New false
2772 1082 Acked 2081 Read 6 Done Done New false
2773 1083 Acked 2082 Read 8 Done Done New false
2774 1084 Acked 2083 Write 3 Done Done New false
2775 1085 Acked 2084 Read 8 Done Done New false
2776 1086 Acked 2085 Read 1 Done Done New false
2777 1087 Acked 2086 Write 3 Done Done New false
2778 1088 Acked 2087 Read 8 Done Done New false
2779 1089 Acked 2088 Read 8 Done Done New false
2780 1090 Acked 2089 Read 4 Done Done New false
2781 1091 Acked 2090 Flush 0 Done Done New false
2782 1092 Acked 2091 Read 2 Done Done New false
2783 1093 Acked 2092 Read 4 Done Done New false
2784 1094 Acked 2093 Read 2 Done Done New false
2785 1095 Acked 2094 Read 3 Done Done New false
2786 1096 Acked 2095 Read 9 Done Done New false
2787 1097 Acked 2096 Write 3 Done Done New false
2788 1098 Acked 2097 Read 4 Done Done New false
2789 1099 Acked 2098 Read 3 Done Done New false
2790 1100 Acked 2099 Write 1 Done Done New false
2791 1101 Acked 2100 Read 10 Done Done New false
2792 1102 Acked 2101 Write 4 Done Done New false
2793 1103 Acked 2102 Read 2 Done Done New false
2794 1104 Acked 2103 Read 1 Done Done New false
2795 1105 Acked 2104 Read 4 Done Done New false
2796 1106 Acked 2105 Read 5 Done Done New false
2797 1107 Acked 2106 Write 4 Done Done New false
2798 1108 Acked 2107 Read 1 Done Done New false
2799 1109 Acked 2108 Read 9 Done Done New false
2800 1110 Acked 2109 Write 5 Done Done New false
2801 1111 Acked 2110 Write 9 Done Done New false
2802 1112 Acked 2111 Read 6 Done Done New false
2803 1113 Acked 2112 Write 7 Done Done New false
2804 1114 Acked 2113 Read 8 Done Done New false
2805 1115 Acked 2114 Read 2 Done Done New false
2806 1116 Acked 2115 Write 8 Done Done New false
2807 1117 Acked 2116 Read 4 Done Done New false
2808 1118 Acked 2117 Write 2 Done Done New false
2809 1119 Acked 2118 Write 8 Done Done New false
2810 1120 Acked 2119 Read 9 Done Done New false
2811 1121 Acked 2120 Write 10 Done Done New false
2812 1122 Acked 2121 Read 3 Done Done New false
2813 1123 Acked 2122 Write 10 Done Done New false
2814 1124 Acked 2123 Read 7 Done Done New false
2815 1125 Acked 2124 Write 1 Done Done New false
2816 1126 Acked 2125 Flush 0 Done Done New false
2817 1127 Acked 2126 Flush 0 Done Done New false
2818 1128 Acked 2127 Write 5 Done Done New false
2819 1129 Acked 2128 Read 6 Done Done New false
2820 1130 Acked 2129 Write 7 Done Done New false
2821 1131 Acked 2130 Write 6 Done Done New false
2822 1132 Acked 2131 Write 8 Done Done New false
2823 1133 Acked 2132 Read 7 Done Done New false
2824 1134 Acked 2133 Read 2 Done Done New false
2825 1135 Acked 2134 Write 4 Done Done New false
2826 1136 Acked 2135 Read 1 Done Done New false
2827 1137 Acked 2136 Read 8 Done Done New false
2828 1138 Acked 2137 Write 8 Done Done New false
2829 1139 Acked 2138 Read 9 Done Done New false
2830 1140 Acked 2139 Flush 0 Done Done New false
2831 1141 Acked 2140 Write 6 Done Done New false
2832 1142 Acked 2141 Read 10 Done Done New false
2833 1143 Acked 2142 Read 10 Done Done New false
2834 1144 Acked 2143 Read 4 Done Done New false
2835 1145 Acked 2144 Write 5 Done Done New false
2836 1146 Acked 2145 Write 2 Done Done New false
2837 1147 Acked 2146 Flush 0 Done Done New false
2838 1148 Acked 2147 Read 1 Done Done New false
2839 1149 Acked 2148 Read 4 Done Done New false
2840 1150 Acked 2149 Read 4 Done Done New false
2841 1151 Acked 2150 Write 5 Done Done New false
2842 1152 Acked 2151 Write 2 Done Done New false
2843 1153 Acked 2152 Read 5 Done Done New false
2844 1154 Acked 2153 Flush 0 Done Done New false
2845 1155 Acked 2154 Write 3 Done Done New false
2846 1156 Acked 2155 Write 8 Done Done New false
2847 1157 Acked 2156 Read 8 Done Done New false
2848 1158 Acked 2157 Flush 0 Done Done New false
2849 1159 Acked 2158 Write 7 Done Done New false
2850 1160 Acked 2159 Write 6 Done Done New false
2851 1161 Acked 2160 Read 9 Done Done New false
2852 1162 Acked 2161 Write 9 Done Done New false
2853 1163 Acked 2162 Read 8 Done Done New false
2854 1164 Acked 2163 Read 9 Done Done New false
2855 1165 Acked 2164 Write 10 Done Done New false
2856 1166 Acked 2165 Write 2 Done Done New false
2857 1167 Acked 2166 Write 1 Done Done New false
2858 1168 Acked 2167 Write 4 Done Done New false
2859 1169 Acked 2168 Read 2 Done Done New false
2860 1170 Acked 2169 Read 7 Done Done New false
2861 1171 Acked 2170 Read 8 Done Done New false
2862 1172 Acked 2171 Read 4 Done Done New false
2863 1173 Acked 2172 Read 3 Done Done New false
2864 1174 Acked 2173 Read 1 Done Done New false
2865 1175 Acked 2174 Flush 0 Done Done New false
2866 1176 Acked 2175 Read 7 Done Done New false
2867 1177 Acked 2176 Write 9 Done Done New false
2868 1178 Acked 2177 Read 1 Done Done New false
2869 1179 Acked 2178 Write 7 Done Done New false
2870 1180 Acked 2179 Write 3 Done Done New false
2871 1181 Acked 2180 Write 2 Done Done New false
2872 1182 Acked 2181 Flush 0 Done Done New false
2873 1183 Acked 2182 Read 3 Done Done New false
2874 1184 Acked 2183 Write 3 Done Done New false
2875 1185 Acked 2184 Write 8 Done Done New false
2876 1186 Acked 2185 Flush 0 Done Done New false
2877 1187 Acked 2186 Write 9 Done Done New false
2878 1188 Acked 2187 Write 6 Done Done New false
2879 1189 Acked 2188 Write 4 Done Done New false
2880 1190 Acked 2189 Write 2 Done Done New false
2881 1191 Acked 2190 Read 7 Done Done New false
2882 1192 Acked 2191 Read 4 Done Done New false
2883 1193 Acked 2192 Read 7 Done Done New false
2884 1194 Acked 2193 Write 7 Done Done New false
2885 1195 Acked 2194 Read 2 Done Done New false
2886 1196 Acked 2195 Read 7 Done Done New false
2887 1197 Acked 2196 Read 9 Done Done New false
2888 1198 Acked 2197 Read 9 Done Done New false
2889 1199 Acked 2198 Write 4 Done Done New false
2890 1200 Acked 2199 Read 9 Done Done New false
2891 1201 Acked 2200 Read 8 Done Done New false
2892 1202 Acked 2201 Write 4 Done Done New false
2893 1203 Acked 2202 Read 7 Done Done New false
2894 1204 Acked 2203 Read 3 Done Done New false
2895 1205 Acked 2204 Read 5 Done Done New false
2896 1206 Acked 2205 Read 2 Done Done New false
2897 1207 Acked 2206 Read 10 Done Done New false
2898 1208 Acked 2207 Write 3 Done Done New false
2899 1209 Acked 2208 Read 1 Done Done New false
2900 STATES DS:0 DS:1 DS:2 TOTAL
2901 New 0 0 302 302
2902 Sent 0 0 0 0
2903 Done 302 302 0 604
2904 Skipped 0 0 0 0
2905 Error 0 0 0 0
2906 Last Flush: 2185 2185 1906
2907 Downstairs last five completed: 1906 1905 1904 1903 1902 1901
2908 Upstairs last five completed: 1209 1208 1207 1206 1205 1204
2909 CLIENT: Up:0 ds:302 act:2
2910 [8830] got start action
2911 [8830] I am going to restart
2912 Starting downstairs at port 8830
2913 Make output file at "/tmp/dsc/downstairs-8830.txt"
2914 [8830][2] reports Starting
2915 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1180)
2916 [8830][2] reports Running
2917 ----------------------------------------------------------------
2918 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
2919 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2920 908 Acked 1907 Read 6 Done Done New false
2921 909 Acked 1908 Read 1 Done Done New false
2922 910 Acked 1909 Flush 0 Done Done New false
2923 911 Acked 1910 Write 6 Done Done New false
2924 912 Acked 1911 Read 6 Done Done New false
2925 913 Acked 1912 Write 10 Done Done New false
2926 914 Acked 1913 Write 2 Done Done New false
2927 915 Acked 1914 Read 1 Done Done New false
2928 916 Acked 1915 Read 10 Done Done New false
2929 917 Acked 1916 Write 2 Done Done New false
2930 918 Acked 1917 Read 9 Done Done New false
2931 919 Acked 1918 Flush 0 Done Done New false
2932 920 Acked 1919 Read 1 Done Done New false
2933 921 Acked 1920 Write 6 Done Done New false
2934 922 Acked 1921 Write 2 Done Done New false
2935 923 Acked 1922 Write 9 Done Done New false
2936 924 Acked 1923 Write 1 Done Done New false
2937 925 Acked 1924 Read 5 Done Done New false
2938 926 Acked 1925 Flush 0 Done Done New false
2939 927 Acked 1926 Read 3 Done Done New false
2940 928 Acked 1927 Write 7 Done Done New false
2941 929 Acked 1928 Write 5 Done Done New false
2942 930 Acked 1929 Write 2 Done Done New false
2943 931 Acked 1930 Read 1 Done Done New false
2944 932 Acked 1931 Read 5 Done Done New false
2945 933 Acked 1932 Read 6 Done Done New false
2946 934 Acked 1933 Read 2 Done Done New false
2947 935 Acked 1934 Write 6 Done Done New false
2948 936 Acked 1935 Write 4 Done Done New false
2949 937 Acked 1936 Read 4 Done Done New false
2950 938 Acked 1937 Flush 0 Done Done New false
2951 939 Acked 1938 Flush 0 Done Done New false
2952 940 Acked 1939 Write 4 Done Done New false
2953 941 Acked 1940 Write 7 Done Done New false
2954 942 Acked 1941 Read 8 Done Done New false
2955 943 Acked 1942 Read 2 Done Done New false
2956 944 Acked 1943 Read 8 Done Done New false
2957 945 Acked 1944 Read 4 Done Done New false
2958 946 Acked 1945 Write 7 Done Done New false
2959 947 Acked 1946 Read 2 Done Done New false
2960 948 Acked 1947 Read 3 Done Done New false
2961 949 Acked 1948 Read 7 Done Done New false
2962 950 Acked 1949 Write 2 Done Done New false
2963 951 Acked 1950 Write 6 Done Done New false
2964 952 Acked 1951 Write 8 Done Done New false
2965 953 Acked 1952 Write 5 Done Done New false
2966 954 Acked 1953 Read 3 Done Done New false
2967 955 Acked 1954 Read 9 Done Done New false
2968 956 Acked 1955 Write 3 Done Done New false
2969 957 Acked 1956 Write 9 Done Done New false
2970 958 Acked 1957 Read 10 Done Done New false
2971 959 Acked 1958 Read 6 Done Done New false
2972 960 Acked 1959 Write 8 Done Done New false
2973 961 Acked 1960 Write 2 Done Done New false
2974 962 Acked 1961 Read 1 Done Done New false
2975 963 Acked 1962 Read 4 Done Done New false
2976 964 Acked 1963 Read 6 Done Done New false
2977 965 Acked 1964 Read 1 Done Done New false
2978 966 Acked 1965 Write 2 Done Done New false
2979 967 Acked 1966 Read 9 Done Done New false
2980 968 Acked 1967 Write 9 Done Done New false
2981 969 Acked 1968 Write 2 Done Done New false
2982 970 Acked 1969 Read 6 Done Done New false
2983 971 Acked 1970 Read 10 Done Done New false
2984 972 Acked 1971 Read 3 Done Done New false
2985 973 Acked 1972 Write 2 Done Done New false
2986 974 Acked 1973 Write 1 Done Done New false
2987 975 Acked 1974 Flush 0 Done Done New false
2988 976 Acked 1975 Flush 0 Done Done New false
2989 977 Acked 1976 Write 8 Done Done New false
2990 978 Acked 1977 Flush 0 Done Done New false
2991 979 Acked 1978 Read 8 Done Done New false
2992 980 Acked 1979 Write 9 Done Done New false
2993 981 Acked 1980 Write 3 Done Done New false
2994 982 Acked 1981 Write 10 Done Done New false
2995 983 Acked 1982 Flush 0 Done Done New false
2996 984 Acked 1983 Read 5 Done Done New false
2997 985 Acked 1984 Write 9 Done Done New false
2998 986 Acked 1985 Read 6 Done Done New false
2999 987 Acked 1986 Read 8 Done Done New false
3000 988 Acked 1987 Write 1 Done Done New false
3001 989 Acked 1988 Read 10 Done Done New false
3002 990 Acked 1989 Read 3 Done Done New false
3003 991 Acked 1990 Read 7 Done Done New false
3004 992 Acked 1991 Write 9 Done Done New false
3005 993 Acked 1992 Write 8 Done Done New false
3006 994 Acked 1993 Read 10 Done Done New false
3007 995 Acked 1994 Read 10 Done Done New false
3008 996 Acked 1995 Read 5 Done Done New false
3009 997 Acked 1996 Write 9 Done Done New false
3010 998 Acked 1997 Write 2 Done Done New false
3011 999 Acked 1998 Read 6 Done Done New false
3012 1000 Acked 1999 Read 9 Done Done New false
3013 1001 Acked 2000 Read 4 Done Done New false
3014 1002 Acked 2001 Read 8 Done Done New false
3015 1003 Acked 2002 Write 4 Done Done New false
3016 1004 Acked 2003 Write 2 Done Done New false
3017 1005 Acked 2004 Write 5 Done Done New false
3018 1006 Acked 2005 Read 3 Done Done New false
3019 1007 Acked 2006 Read 3 Done Done New false
3020 1008 Acked 2007 Read 1 Done Done New false
3021 1009 Acked 2008 Flush 0 Done Done New false
3022 1010 Acked 2009 Read 1 Done Done New false
3023 1011 Acked 2010 Read 8 Done Done New false
3024 1012 Acked 2011 Read 7 Done Done New false
3025 1013 Acked 2012 Write 10 Done Done New false
3026 1014 Acked 2013 Write 6 Done Done New false
3027 1015 Acked 2014 Read 8 Done Done New false
3028 1016 Acked 2015 Write 6 Done Done New false
3029 1017 Acked 2016 Flush 0 Done Done New false
3030 1018 Acked 2017 Write 3 Done Done New false
3031 1019 Acked 2018 Read 8 Done Done New false
3032 1020 Acked 2019 Read 2 Done Done New false
3033 1021 Acked 2020 Write 3 Done Done New false
3034 1022 Acked 2021 Flush 0 Done Done New false
3035 1023 Acked 2022 Write 4 Done Done New false
3036 1024 Acked 2023 Write 8 Done Done New false
3037 1025 Acked 2024 Read 7 Done Done New false
3038 1026 Acked 2025 Read 7 Done Done New false
3039 1027 Acked 2026 Write 2 Done Done New false
3040 1028 Acked 2027 Write 3 Done Done New false
3041 1029 Acked 2028 Read 9 Done Done New false
3042 1030 Acked 2029 Read 6 Done Done New false
3043 1031 Acked 2030 Read 2 Done Done New false
3044 1032 Acked 2031 Write 3 Done Done New false
3045 1033 Acked 2032 Read 4 Done Done New false
3046 1034 Acked 2033 Write 7 Done Done New false
3047 1035 Acked 2034 Read 1 Done Done New false
3048 1036 Acked 2035 Read 4 Done Done New false
3049 1037 Acked 2036 Write 4 Done Done New false
3050 1038 Acked 2037 Write 10 Done Done New false
3051 1039 Acked 2038 Write 1 Done Done New false
3052 1040 Acked 2039 Read 8 Done Done New false
3053 1041 Acked 2040 Read 5 Done Done New false
3054 1042 Acked 2041 Write 10 Done Done New false
3055 1043 Acked 2042 Write 5 Done Done New false
3056 1044 Acked 2043 Write 10 Done Done New false
3057 1045 Acked 2044 Write 9 Done Done New false
3058 1046 Acked 2045 Read 3 Done Done New false
3059 1047 Acked 2046 Read 1 Done Done New false
3060 1048 Acked 2047 Write 1 Done Done New false
3061 1049 Acked 2048 Flush 0 Done Done New false
3062 1050 Acked 2049 Read 2 Done Done New false
3063 1051 Acked 2050 Flush 0 Done Done New false
3064 1052 Acked 2051 Read 7 Done Done New false
3065 1053 Acked 2052 Read 3 Done Done New false
3066 1054 Acked 2053 Write 5 Done Done New false
3067 1055 Acked 2054 Read 8 Done Done New false
3068 1056 Acked 2055 Read 5 Done Done New false
3069 1057 Acked 2056 Flush 0 Done Done New false
3070 1058 Acked 2057 Write 7 Done Done New false
3071 1059 Acked 2058 Read 4 Done Done New false
3072 1060 Acked 2059 Flush 0 Done Done New false
3073 1061 Acked 2060 Read 9 Done Done New false
3074 1062 Acked 2061 Read 9 Done Done New false
3075 1063 Acked 2062 Read 9 Done Done New false
3076 1064 Acked 2063 Read 9 Done Done New false
3077 1065 Acked 2064 Read 1 Done Done New false
3078 1066 Acked 2065 Write 4 Done Done New false
3079 1067 Acked 2066 Write 1 Done Done New false
3080 1068 Acked 2067 Write 1 Done Done New false
3081 1069 Acked 2068 Read 6 Done Done New false
3082 1070 Acked 2069 Read 3 Done Done New false
3083 1071 Acked 2070 Read 2 Done Done New false
3084 1072 Acked 2071 Write 9 Done Done New false
3085 1073 Acked 2072 Write 4 Done Done New false
3086 1074 Acked 2073 Read 3 Done Done New false
3087 1075 Acked 2074 Read 8 Done Done New false
3088 1076 Acked 2075 Read 5 Done Done New false
3089 1077 Acked 2076 Read 5 Done Done New false
3090 1078 Acked 2077 Write 3 Done Done New false
3091 1079 Acked 2078 Write 10 Done Done New false
3092 1080 Acked 2079 Write 8 Done Done New false
3093 1081 Acked 2080 Write 9 Done Done New false
3094 1082 Acked 2081 Read 6 Done Done New false
3095 1083 Acked 2082 Read 8 Done Done New false
3096 1084 Acked 2083 Write 3 Done Done New false
3097 1085 Acked 2084 Read 8 Done Done New false
3098 1086 Acked 2085 Read 1 Done Done New false
3099 1087 Acked 2086 Write 3 Done Done New false
3100 1088 Acked 2087 Read 8 Done Done New false
3101 1089 Acked 2088 Read 8 Done Done New false
3102 1090 Acked 2089 Read 4 Done Done New false
3103 1091 Acked 2090 Flush 0 Done Done New false
3104 1092 Acked 2091 Read 2 Done Done New false
3105 1093 Acked 2092 Read 4 Done Done New false
3106 1094 Acked 2093 Read 2 Done Done New false
3107 1095 Acked 2094 Read 3 Done Done New false
3108 1096 Acked 2095 Read 9 Done Done New false
3109 1097 Acked 2096 Write 3 Done Done New false
3110 1098 Acked 2097 Read 4 Done Done New false
3111 1099 Acked 2098 Read 3 Done Done New false
3112 1100 Acked 2099 Write 1 Done Done New false
3113 1101 Acked 2100 Read 10 Done Done New false
3114 1102 Acked 2101 Write 4 Done Done New false
3115 1103 Acked 2102 Read 2 Done Done New false
3116 1104 Acked 2103 Read 1 Done Done New false
3117 1105 Acked 2104 Read 4 Done Done New false
3118 1106 Acked 2105 Read 5 Done Done New false
3119 1107 Acked 2106 Write 4 Done Done New false
3120 1108 Acked 2107 Read 1 Done Done New false
3121 1109 Acked 2108 Read 9 Done Done New false
3122 1110 Acked 2109 Write 5 Done Done New false
3123 1111 Acked 2110 Write 9 Done Done New false
3124 1112 Acked 2111 Read 6 Done Done New false
3125 1113 Acked 2112 Write 7 Done Done New false
3126 1114 Acked 2113 Read 8 Done Done New false
3127 1115 Acked 2114 Read 2 Done Done New false
3128 1116 Acked 2115 Write 8 Done Done New false
3129 1117 Acked 2116 Read 4 Done Done New false
3130 1118 Acked 2117 Write 2 Done Done New false
3131 1119 Acked 2118 Write 8 Done Done New false
3132 1120 Acked 2119 Read 9 Done Done New false
3133 1121 Acked 2120 Write 10 Done Done New false
3134 1122 Acked 2121 Read 3 Done Done New false
3135 1123 Acked 2122 Write 10 Done Done New false
3136 1124 Acked 2123 Read 7 Done Done New false
3137 1125 Acked 2124 Write 1 Done Done New false
3138 1126 Acked 2125 Flush 0 Done Done New false
3139 1127 Acked 2126 Flush 0 Done Done New false
3140 1128 Acked 2127 Write 5 Done Done New false
3141 1129 Acked 2128 Read 6 Done Done New false
3142 1130 Acked 2129 Write 7 Done Done New false
3143 1131 Acked 2130 Write 6 Done Done New false
3144 1132 Acked 2131 Write 8 Done Done New false
3145 1133 Acked 2132 Read 7 Done Done New false
3146 1134 Acked 2133 Read 2 Done Done New false
3147 1135 Acked 2134 Write 4 Done Done New false
3148 1136 Acked 2135 Read 1 Done Done New false
3149 1137 Acked 2136 Read 8 Done Done New false
3150 1138 Acked 2137 Write 8 Done Done New false
3151 1139 Acked 2138 Read 9 Done Done New false
3152 1140 Acked 2139 Flush 0 Done Done New false
3153 1141 Acked 2140 Write 6 Done Done New false
3154 1142 Acked 2141 Read 10 Done Done New false
3155 1143 Acked 2142 Read 10 Done Done New false
3156 1144 Acked 2143 Read 4 Done Done New false
3157 1145 Acked 2144 Write 5 Done Done New false
3158 1146 Acked 2145 Write 2 Done Done New false
3159 1147 Acked 2146 Flush 0 Done Done New false
3160 1148 Acked 2147 Read 1 Done Done New false
3161 1149 Acked 2148 Read 4 Done Done New false
3162 1150 Acked 2149 Read 4 Done Done New false
3163 1151 Acked 2150 Write 5 Done Done New false
3164 1152 Acked 2151 Write 2 Done Done New false
3165 1153 Acked 2152 Read 5 Done Done New false
3166 1154 Acked 2153 Flush 0 Done Done New false
3167 1155 Acked 2154 Write 3 Done Done New false
3168 1156 Acked 2155 Write 8 Done Done New false
3169 1157 Acked 2156 Read 8 Done Done New false
3170 1158 Acked 2157 Flush 0 Done Done New false
3171 1159 Acked 2158 Write 7 Done Done New false
3172 1160 Acked 2159 Write 6 Done Done New false
3173 1161 Acked 2160 Read 9 Done Done New false
3174 1162 Acked 2161 Write 9 Done Done New false
3175 1163 Acked 2162 Read 8 Done Done New false
3176 1164 Acked 2163 Read 9 Done Done New false
3177 1165 Acked 2164 Write 10 Done Done New false
3178 1166 Acked 2165 Write 2 Done Done New false
3179 1167 Acked 2166 Write 1 Done Done New false
3180 1168 Acked 2167 Write 4 Done Done New false
3181 1169 Acked 2168 Read 2 Done Done New false
3182 1170 Acked 2169 Read 7 Done Done New false
3183 1171 Acked 2170 Read 8 Done Done New false
3184 1172 Acked 2171 Read 4 Done Done New false
3185 1173 Acked 2172 Read 3 Done Done New false
3186 1174 Acked 2173 Read 1 Done Done New false
3187 1175 Acked 2174 Flush 0 Done Done New false
3188 1176 Acked 2175 Read 7 Done Done New false
3189 1177 Acked 2176 Write 9 Done Done New false
3190 1178 Acked 2177 Read 1 Done Done New false
3191 1179 Acked 2178 Write 7 Done Done New false
3192 1180 Acked 2179 Write 3 Done Done New false
3193 1181 Acked 2180 Write 2 Done Done New false
3194 1182 Acked 2181 Flush 0 Done Done New false
3195 1183 Acked 2182 Read 3 Done Done New false
3196 1184 Acked 2183 Write 3 Done Done New false
3197 1185 Acked 2184 Write 8 Done Done New false
3198 1186 Acked 2185 Flush 0 Done Done New false
3199 1187 Acked 2186 Write 9 Done Done New false
3200 1188 Acked 2187 Write 6 Done Done New false
3201 1189 Acked 2188 Write 4 Done Done New false
3202 1190 Acked 2189 Write 2 Done Done New false
3203 1191 Acked 2190 Read 7 Done Done New false
3204 1192 Acked 2191 Read 4 Done Done New false
3205 1193 Acked 2192 Read 7 Done Done New false
3206 1194 Acked 2193 Write 7 Done Done New false
3207 1195 Acked 2194 Read 2 Done Done New false
3208 1196 Acked 2195 Read 7 Done Done New false
3209 1197 Acked 2196 Read 9 Done Done New false
3210 1198 Acked 2197 Read 9 Done Done New false
3211 1199 Acked 2198 Write 4 Done Done New false
3212 1200 Acked 2199 Read 9 Done Done New false
3213 1201 Acked 2200 Read 8 Done Done New false
3214 1202 Acked 2201 Write 4 Done Done New false
3215 1203 Acked 2202 Read 7 Done Done New false
3216 1204 Acked 2203 Read 3 Done Done New false
3217 1205 Acked 2204 Read 5 Done Done New false
3218 1206 Acked 2205 Read 2 Done Done New false
3219 1207 Acked 2206 Read 10 Done Done New false
3220 1208 Acked 2207 Write 3 Done Done New false
3221 1209 Acked 2208 Read 1 Done Done New false
3222 1210 Acked 2209 Flush 0 Done Done New false
3223 STATES DS:0 DS:1 DS:2 TOTAL
3224 New 0 0 303 303
3225 Sent 0 0 0 0
3226 Done 303 303 0 606
3227 Skipped 0 0 0 0
3228 Error 0 0 0 0
3229 Last Flush: 2209 2209 1906
3230 Downstairs last five completed: 1906 1905 1904 1903 1902 1901
3231 Upstairs last five completed: 1210 1209 1208 1207 1206 1205
3232 CLIENT: Up:0 ds:303 act:2
32332023-10-13T18:09:43.034ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 2
32342023-10-13T18:09:43.034ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
32352023-10-13T18:09:43.034ZINFOcrucible: [2] client re-new 303 jobs since flush 1906 = downstairs
32362023-10-13T18:09:43.035ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
32372023-10-13T18:09:43.036ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
32382023-10-13T18:09:43.036ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
32392023-10-13T18:09:43.036ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
32402023-10-13T18:09:43.036ZINFOcrucible: [2] send last flush ID to this DS: 1906
32412023-10-13T18:09:43.036ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
32422023-10-13T18:09:43.036ZINFOcrucible: [2] Replied this last flush ID: 1906
32432023-10-13T18:09:43.036ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Offline ds_transition to Replay
32442023-10-13T18:09:43.036ZINFOcrucible: [2] Transition from Offline to Replay
32452023-10-13T18:09:43.036ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
32462023-10-13T18:09:43.036ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Replay ds_transition to Active
32472023-10-13T18:09:43.036ZINFOcrucible: [2] Transition from Replay to Active
32482023-10-13T18:09:43.036ZINFOcrucible: [2] Starts cmd_loop
32492023-10-13T18:09:43.036ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
32502023-10-13T18:09:43.036ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
32512023-10-13T18:09:43.036ZINFOcrucible: Set check for repair
32522023-10-13T18:09:44.037ZINFOcrucible: Checking if live repair is needed
32532023-10-13T18:09:44.037ZINFOcrucible: No Live Repair required at this time
32542023-10-13T18:09:44.037ZWARNcrucible: [2] flow control end
3255 ----------------------------------------------------------------
3256 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3257 Downstairs last five completed: 2209 2208 2207 2206 2205 2204
3258 Upstairs last five completed: 1210 1209 1208 1207 1206 1205
3259 CLIENT: Up:0 ds:0 act:3
3260 Replay: All jobs finished, all DS active.
3261 Main task has work to do, go find it
3262 got dsc Stop(0)
3263 stop 0
3264 Oct 13 18:09:46.990 INFO request completed, latency_us: 527, response_code: 204, uri: /stop/cid/0, method[8810] Got stop action so:false kr:false
3265 : POST, req_id: d960db11-0c1a-4565-80ab-9bd08fc1843b, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
3266 Oct 13 18:09:46.991 INFO request completed, latency_us: 215, response_code: 200, uri: /state/cid/0, method: GET, req_id: 0eb79b24-a5d9-424b-a524-407b81a58f0f, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
32672023-10-13T18:09:46.992ZWARNcrucible: [0] downstairs disconnected
32682023-10-13T18:09:46.992ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
32692023-10-13T18:09:46.992ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8810 closed looper = 0
32702023-10-13T18:09:46.992ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:false
32712023-10-13T18:09:46.992ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
32722023-10-13T18:09:46.992ZINFOcrucible: [0] 127.0.0.1:8810 task reports offline
32732023-10-13T18:09:46.992ZWARNcrucible: [0] pm_task rx.recv() is None
3274 [8810] Exited with: ExitStatus(unix_wait_status(9))
3275 [8810][0] reports Exit
3276 Oct 13 18:09:50.994 INFO request completed, latency_us: 268, response_code: 200, uri: /state/cid/0, method: GET, req_id: c2429032-12e6-4cf9-a93b-6ef5a7144826, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
3277 001/300 Read block 543 len 40960
3278 002/300 Read block 4562 len 40960
3279 003/300 Read block 2616 len 8192
3280 004/300 Read block 3532 len 28672
3281 005/300 Read block 3063 len 4096
3282 006/300 Write block 3736 len 28672 data: 2 2 2 2 2 3 3
3283 007/300 Write block 3443 len 24576 data: 2 3 2 2 3 3
3284 008/300 Write block 2471 len 20480 data: 3 3 3 2 2
3285 009/300 Read block 3471 len 12288
3286 010/300 Read block 3596 len 28672
3287 011/300 Read block 785 len 4096
3288 012/300 Write block 832 len 8192 data: 2 2
3289 013/300 Write block 2971 len 32768 data: 2 2 2 3 3 2 2 2
3290 014/300 Write block 4779 len 20480 data: 3 3 3 3 2
3291 015/300 Read block 4099 len 32768
3292 016/300 Read block 1929 len 20480
3293 017/300 Write block 1309 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3294 018/300 Read block 2398 len 16384
3295 019/300 Read block 3692 len 36864
3296 020/300 Write block 1683 len 8192 data: 2 2
3297 021/300 Read block 4470 len 4096
3298 022/300 Write block 3989 len 16384 data: 2 2 2 2
3299 023/300 Write block 2523 len 40960 data: 3 3 3 3 3 2 2 2 3 3
3300 024/300 Read block 4439 len 4096
3301 025/300 Read block 713 len 20480
3302 026/300 Read block 1779 len 12288
3303 027/300 Read block 472 len 8192
3304 028/300 Read block 915 len 16384
3305 029/300 Read block 4823 len 16384
3306 030/300 Read block 3832 len 32768
3307 031/300 Read block 1569 len 4096
3308 032/300 Write block 1472 len 36864 data: 2 2 2 2 3 3 3 3 2
3309 033/300 Flush
3310 034/300 Write block 3926 len 32768 data: 2 2 2 2 2 2 2 2
3311 035/300 Write block 2938 len 4096 data: 4
3312 036/300 Read block 1119 len 12288
3313 037/300 Read block 2590 len 32768
3314 038/300 Read block 2030 len 16384
3315 039/300 Read block 4692 len 12288
3316 040/300 Flush
3317 041/300 Write block 3839 len 20480 data: 3 3 3 3 3
3318 042/300 Write block 3476 len 12288 data: 2 2 2
3319 043/300 Read block 3777 len 40960
3320 044/300 Write block 4254 len 12288 data: 2 2 2
3321 045/300 Write block 3026 len 4096 data: 2
3322 046/300 Read block 2236 len 12288
3323 047/300 Read block 102 len 20480
3324 048/300 Read block 4164 len 36864
3325 049/300 Read block 3002 len 32768
3326 050/300 Write block 2432 len 4096 data: 2
3327 051/300 Read block 607 len 12288
3328 052/300 Write block 3070 len 32768 data: 2 2 2 2 2 2 2 2
3329 053/300 Read block 1377 len 32768
3330 054/300 Read block 3813 len 20480
3331 055/300 Read block 2431 len 16384
3332 056/300 Write block 912 len 36864 data: 2 2 2 2 2 3 3 3 3
3333 057/300 Read block 1032 len 28672
3334 058/300 Write block 2398 len 12288 data: 5 5 4
3335 059/300 Read block 2311 len 8192
3336 060/300 Write block 3145 len 12288 data: 3 3 3
3337 061/300 Read block 3966 len 32768
3338 062/300 Write block 1711 len 4096 data: 2
3339 063/300 Read block 818 len 8192
3340 064/300 Read block 1609 len 32768
3341 065/300 Flush
3342 066/300 Write block 6 len 20480 data: 3 3 3 3 3
3343 067/300 Flush
3344 068/300 Read block 2869 len 32768
3345 069/300 Flush
3346 070/300 Read block 1294 len 40960
3347 071/300 Write block 3854 len 20480 data: 2 2 2 2 2
3348 072/300 Read block 4554 len 28672
3349 073/300 Write block 4180 len 36864 data: 2 2 2 2 2 2 2 2 3
3350 074/300 Write block 2470 len 8192 data: 3 4
3351 075/300 Read block 2669 len 40960
3352 076/300 Flush
3353 077/300 Write block 4119 len 32768 data: 2 2 2 2 2 2 2 2
3354 078/300 Read block 2230 len 40960
3355 079/300 Write block 1343 len 8192 data: 2 2
3356 080/300 Write block 827 len 16384 data: 2 2 2 2
3357 081/300 Read block 1027 len 28672
3358 082/300 Read block 2507 len 36864
3359 083/300 Flush
3360 084/300 Flush
3361 085/300 Read block 3124 len 20480
3362 086/300 Read block 1113 len 40960
3363 087/300 Write block 3520 len 4096 data: 4
3364 088/300 Write block 321 len 36864 data: 4 4 4 4 4 4 4 3 3
3365 089/300 Read block 3108 len 4096
3366 090/300 Read block 3377 len 16384
3367 091/300 Write block 2994 len 28672 data: 3 3 3 3 3 3 3
3368 092/300 Read block 2551 len 8192
3369 093/300 Write block 3458 len 12288 data: 3 4 4
3370 094/300 Read block 3589 len 40960
3371 095/300 Read block 4903 len 12288
3372 096/300 Read block 368 len 32768
3373 097/300 Write block 4855 len 28672 data: 2 2 2 3 4 4 3
3374 098/300 Read block 615 len 36864
3375 099/300 Write block 1378 len 28672 data: 3 3 3 3 2 3 3
3376 100/300 Read block 4871 len 36864
3377 101/300 Write block 2138 len 4096 data: 3
3378 102/300 Write block 1753 len 12288 data: 2 2 2
3379 103/300 Write block 3715 len 40960 data: 5 4 4 4 3 3 3 3 2 2
3380 104/300 Write block 3704 len 40960 data: 2 2 3 3 4 4 5 6 5 4
3381 105/300 Read block 1460 len 16384
3382 106/300 Write block 2556 len 24576 data: 4 3 3 3 3 3
3383 107/300 Read block 325 len 16384
3384 108/300 Read block 3590 len 24576
3385 109/300 Read block 1162 len 28672
3386 110/300 Read block 4919 len 16384
3387 111/300 Write block 3591 len 24576 data: 2 2 2 2 3 3
3388 112/300 Read block 278 len 28672
3389 113/300 Read block 474 len 4096
3390 114/300 Flush
3391 115/300 Flush
3392 116/300 Read block 4251 len 28672
3393 117/300 Write block 3230 len 12288 data: 2 2 2
3394 118/300 Flush
3395 119/300 Read block 3193 len 4096
3396 120/300 Read block 3933 len 40960
3397 121/300 Flush
3398 122/300 Read block 4547 len 32768
3399 123/300 Flush
3400 124/300 Read block 4133 len 12288
3401 125/300 Read block 1327 len 32768
3402 126/300 Write block 4538 len 32768 data: 2 2 2 2 2 2 2 2
3403 127/300 Flush
3404 128/300 Read block 2197 len 4096
3405 129/300 Read block 2737 len 28672
3406 130/300 Write block 1802 len 4096 data: 2
3407 131/300 Read block 3990 len 36864
3408 132/300 Read block 4898 len 28672
3409 133/300 Write block 250 len 28672 data: 2 2 2 2 2 2 2
3410 134/300 Read block 1022 len 32768
3411 135/300 Read block 2324 len 8192
3412 136/300 Read block 2290 len 8192
3413 137/300 Read block 1989 len 20480
3414 138/300 Write block 597 len 32768 data: 3 3 3 4 4 4 5 5
3415 139/300 Read block 2196 len 40960
3416 140/300 Read block 1211 len 4096
3417 141/300 Write block 2202 len 4096 data: 4
3418 142/300 Read block 3530 len 8192
3419 143/300 Read block 1829 len 28672
3420 144/300 Write block 886 len 24576 data: 2 2 2 2 2 2
3421 145/300 Read block 4956 len 40960
3422 146/300 Write block 4382 len 24576 data: 2 2 2 2 2 3
3423 147/300 Write block 20 len 24576 data: 2 2 2 3 3 3
3424 148/300 Write block 3753 len 28672 data: 3 3 3 3 2 2 2
3425 149/300 Read block 2964 len 40960
3426 150/300 Read block 3725 len 16384
3427 151/300 Write block 113 len 12288 data: 2 2 2
3428 152/300 Flush
3429 153/300 Write block 4506 len 40960 data: 2 2 2 2 2 3 3 3 3 2
3430 154/300 Write block 3342 len 8192 data: 2 2
3431 155/300 Read block 1000 len 20480
3432 156/300 Flush
3433 157/300 Read block 3734 len 32768
3434 158/300 Write block 1650 len 8192 data: 3 3
3435 159/300 Write block 3810 len 20480 data: 3 3 2 2 2
3436 160/300 Read block 4391 len 32768
3437 161/300 Flush
3438 162/300 Read block 1924 len 36864
3439 163/300 Read block 4713 len 12288
3440 164/300 Read block 434 len 8192
3441 165/300 Write block 1058 len 40960 data: 2 2 2 2 2 2 3 3 3 3
3442 166/300 Write block 1086 len 24576 data: 3 3 3 3 2 2
3443 167/300 Write block 3293 len 32768 data: 2 2 2 2 2 2 2 2
3444 168/300 Flush
3445 169/300 Read block 2310 len 36864
3446 170/300 Write block 3092 len 20480 data: 2 2 2 4 4
3447 171/300 Read block 1698 len 4096
3448 172/300 Write block 3113 len 28672 data: 4 3 4 4 3 3 2
3449 173/300 Read block 1520 len 16384
3450 174/300 Read block 2225 len 16384
3451 175/300 Read block 2819 len 16384
3452 176/300 Read block 1265 len 32768
3453 177/300 Read block 1238 len 32768
3454 178/300 Read block 1761 len 16384
3455 179/300 Write block 4077 len 32768 data: 2 2 2 2 2 2 2 2
3456 180/300 Read block 2915 len 4096
3457 181/300 Read block 972 len 28672
3458 182/300 Write block 417 len 36864 data: 3 3 3 3 3 2 2 3 3
3459 183/300 Write block 2348 len 40960 data: 3 3 4 4 4 3 3 3 3 3
3460 184/300 Read block 1506 len 12288
3461 185/300 Flush
3462 186/300 Write block 4345 len 20480 data: 2 2 2 2 2
3463 187/300 Read block 4541 len 40960
3464 188/300 Write block 1723 len 8192 data: 2 2
3465 189/300 Read block 3841 len 28672
3466 190/300 Read block 192 len 8192
3467 191/300 Write block 912 len 36864 data: 3 3 3 3 3 4 4 4 4
3468 192/300 Read block 3069 len 8192
3469 193/300 Write block 415 len 16384 data: 2 2 4 4
3470 194/300 Read block 4819 len 36864
3471 195/300 Write block 935 len 20480 data: 2 2 2 2 2
3472 196/300 Write block 2471 len 20480 data: 5 4 4 3 3
3473 197/300 Write block 745 len 4096 data: 2
3474 198/300 Read block 4749 len 4096
3475 199/300 Read block 2463 len 20480
3476 200/300 Read block 155 len 4096
3477 201/300 Read block 2313 len 20480
3478 202/300 Read block 3300 len 32768
3479 203/300 Write block 2602 len 8192 data: 4 3
3480 204/300 Read block 1129 len 20480
3481 205/300 Flush
3482 206/300 Write block 4329 len 24576 data: 3 3 3 3 4 4
3483 207/300 Read block 2682 len 32768
3484 208/300 Write block 1728 len 20480 data: 2 2 2 2 2
3485 209/300 Write block 1953 len 16384 data: 3 3 2 2
3486 210/300 Write block 2682 len 8192 data: 2 2
3487 211/300 Write block 4667 len 24576 data: 2 2 2 2 2 2
3488 212/300 Flush
3489 213/300 Write block 2695 len 28672 data: 2 2 2 2 2 2 2
3490 214/300 Read block 3319 len 4096
3491 215/300 Write block 1789 len 28672 data: 4 4 4 4 3 2 2
3492 216/300 Write block 2137 len 32768 data: 3 4 3 3 3 3 2 2
3493 217/300 Read block 1819 len 36864
3494 218/300 Read block 1600 len 40960
3495 219/300 Read block 1973 len 32768
3496 220/300 Read block 36 len 24576
3497 221/300 Read block 1462 len 40960
3498 222/300 Write block 3001 len 20480 data: 2 2 2 2 2
3499 223/300 Read block 369 len 20480
3500 224/300 Flush
3501 225/300 Read block 3319 len 36864
3502 226/300 Read block 1000 len 12288
3503 227/300 Read block 2250 len 32768
3504 228/300 Write block 4509 len 20480 data: 3 3 4 4 4
3505 229/300 Read block 3537 len 20480
3506 230/300 Write block 3883 len 4096 data: 3
3507 231/300 Write block 3897 len 4096 data: 2
3508 232/300 Write block 3200 len 24576 data: 2 2 2 2 2 3
3509 233/300 Read block 3396 len 12288
3510 234/300 Read block 706 len 12288
3511 235/300 Write block 1488 len 32768 data: 5 5 4 5 5 5 4 3
3512 236/300 Read block 486 len 4096
3513 237/300 Read block 843 len 36864
3514 238/300 Write block 4908 len 4096 data: 2
3515 239/300 Read block 1771 len 8192
3516 240/300 Flush
3517 241/300 Read block 3299 len 12288
3518 242/300 Write block 1366 len 36864 data: 2 3 3 3 3 3 3 3 3
3519 243/300 Read block 3509 len 24576
3520 244/300 Read block 2608 len 12288
3521 245/300 Write block 4184 len 4096 data: 3
3522 246/300 Read block 4089 len 16384
3523 247/300 Write block 2746 len 36864 data: 2 2 2 2 2 2 2 2 2
3524 248/300 Read block 18 len 20480
3525 249/300 Read block 48 len 4096
3526 250/300 Read block 3224 len 24576
3527 251/300 Read block 3322 len 20480
3528 252/300 Read block 1442 len 12288
3529 253/300 Read block 1917 len 4096
3530 254/300 Read block 1703 len 36864
3531 255/300 Write block 4353 len 8192 data: 2 2
3532 256/300 Write block 3139 len 28672 data: 3 3 3 3 4 3 4
3533 257/300 Write block 1800 len 28672 data: 2 3 3 2 2 2 2
3534 258/300 Read block 3816 len 16384
3535 259/300 Write block 846 len 36864 data: 3 3 3 3 2 2 2 2 2
3536 260/300 Read block 3097 len 20480
3537 261/300 Read block 2356 len 12288
3538 262/300 Write block 2266 len 16384 data: 2 2 2 2
3539 263/300 Read block 2719 len 4096
3540 264/300 Flush
3541 265/300 Read block 3505 len 8192
3542 266/300 Flush
3543 267/300 Write block 1399 len 8192 data: 2 2
3544 268/300 Read block 2502 len 20480
3545 269/300 Read block 4080 len 16384
3546 270/300 Write block 928 len 40960 data: 2 3 3 3 2 2 2 3 3 3
3547 271/300 Read block 4440 len 4096
3548 272/300 Write block 1643 len 28672 data: 3 3 2 3 3 3 3
3549 273/300 Flush
3550 274/300 Read block 2947 len 32768
3551 275/300 Write block 4099 len 12288 data: 2 2 3
3552 276/300 Read block 4541 len 24576
3553 277/300 Read block 616 len 32768
3554 278/300 Read block 801 len 8192
3555 279/300 Write block 1565 len 16384 data: 2 2 2 2
3556 280/300 Read block 785 len 40960
3557 281/300 Read block 1259 len 4096
3558 282/300 Read block 31 len 32768
3559 283/300 Read block 109 len 28672
3560 284/300 Flush
3561 285/300 Read block 1568 len 36864
3562 286/300 Write block 278 len 4096 data: 2
3563 287/300 Read block 1512 len 4096
3564 288/300 Write block 4324 len 16384 data: 2 2 2 2
3565 289/300 Read block 4677 len 4096
3566 290/300 Read block 3225 len 20480
3567 291/300 Flush
3568 292/300 Write block 827 len 16384 data: 3 3 3 3
3569 293/300 Read block 3506 len 20480
3570 294/300 Read block 2580 len 28672
3571 295/300 Read block 3445 len 20480
3572 296/300 Write block 4619 len 24576 data: 3 3 3 3 3 3
3573 297/300 Write block 1097 len 16384 data: 4 4 4 4
3574 298/300 Write block 3489 len 12288 data: 2 2 2
3575 299/300 Write block 3935 len 4096 data: 3
3576 300/300 Write block 3460 len 16384 data: 5 3 3 2
3577 Main task has work to do, go find it
3578 got dsc Start(0)
3579 start 0
3580 Oct 13 18:09:51.956 INFO request completed, latency_us: 314, response_code: 204, uri: /start/cid/0, method: POST, req_id: 8d800cf8-bf6a-4d47-9368-63f6885babda, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
3581 Replay: started 0, returned:Ok(())
3582 ----------------------------------------------------------------
3583 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
3584 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3585 1211 Acked 2210 Read 10 New Done Done false
3586 1212 Acked 2211 Read 10 New Done Done false
3587 1213 Acked 2212 Read 2 New Done Done false
3588 1214 Acked 2213 Read 7 New Done Done false
3589 1215 Acked 2214 Read 1 New Done Done false
3590 1216 Acked 2215 Write 7 New Done Done false
3591 1217 Acked 2216 Write 6 New Done Done false
3592 1218 Acked 2217 Write 5 New Done Done false
3593 1219 Acked 2218 Read 3 New Done Done false
3594 1220 Acked 2219 Read 7 New Done Done false
3595 1221 Acked 2220 Read 1 New Done Done false
3596 1222 Acked 2221 Write 2 New Done Done false
3597 1223 Acked 2222 Write 8 New Done Done false
3598 1224 Acked 2223 Write 5 New Done Done false
3599 1225 Acked 2224 Read 8 New Done Done false
3600 1226 Acked 2225 Read 5 New Done Done false
3601 1227 Acked 2226 Write 10 New Done Done false
3602 1228 Acked 2227 Read 4 New Done Done false
3603 1229 Acked 2228 Read 9 New Done Done false
3604 1230 Acked 2229 Write 2 New Done Done false
3605 1231 Acked 2230 Read 1 New Done Done false
3606 1232 Acked 2231 Write 4 New Done Done false
3607 1233 Acked 2232 Write 10 New Done Done false
3608 1234 Acked 2233 Read 1 New Done Done false
3609 1235 Acked 2234 Read 5 New Done Done false
3610 1236 Acked 2235 Read 3 New Done Done false
3611 1237 Acked 2236 Read 2 New Done Done false
3612 1238 Acked 2237 Read 4 New Done Done false
3613 1239 Acked 2238 Read 4 New Done Done false
3614 1240 Acked 2239 Read 8 New Done Done false
3615 1241 Acked 2240 Read 1 New Done Done false
3616 1242 Acked 2241 Write 9 New Done Done false
3617 1243 Acked 2242 Flush 0 New Done Done false
3618 1244 Acked 2243 Write 8 New Done Done false
3619 1245 Acked 2244 Write 1 New Done Done false
3620 1246 Acked 2245 Read 3 New Done Done false
3621 1247 Acked 2246 Read 8 New Done Done false
3622 1248 Acked 2247 Read 4 New Done Done false
3623 1249 Acked 2248 Read 3 New Done Done false
3624 1250 Acked 2249 Flush 0 New Done Done false
3625 1251 Acked 2250 Write 5 New Done Done false
3626 1252 Acked 2251 Write 3 New Done Done false
3627 1253 Acked 2252 Read 10 New Done Done false
3628 1254 Acked 2253 Write 3 New Done Done false
3629 1255 Acked 2254 Flush 0 New Done Done false
3630 1256 Acked 2255 Write 1 New Done Done false
3631 1257 Acked 2256 Read 3 New Done Done false
3632 1258 Acked 2257 Read 5 New Done Done false
3633 1259 Acked 2258 Read 9 New Done Done false
3634 1260 Acked 2259 Read 8 New Done Done false
3635 1261 Acked 2260 Write 1 New Done Done false
3636 1262 Acked 2261 Read 3 New Done Done false
3637 1263 Acked 2262 Write 8 New Done Done false
3638 1264 Acked 2263 Read 8 New Done Done false
3639 1265 Acked 2264 Read 5 New Done Done false
3640 1266 Acked 2265 Read 4 New Done Done false
3641 1267 Acked 2266 Write 9 New Done Done false
3642 1268 Acked 2267 Read 7 New Done Done false
3643 1269 Acked 2268 Write 3 New Done Done false
3644 1270 Acked 2269 Read 2 New Done Done false
3645 1271 Acked 2270 Write 3 New Done Done false
3646 1272 Acked 2271 Read 8 New Done Done false
3647 1273 Acked 2272 Write 1 New Done Done false
3648 1274 Acked 2273 Read 2 New Done Done false
3649 1275 Acked 2274 Read 8 New Done Done false
3650 1276 Acked 2275 Flush 0 New Done Done false
3651 1277 Acked 2276 Write 5 New Done Done false
3652 1278 Acked 2277 Flush 0 New Done Done false
3653 1279 Acked 2278 Read 8 New Done Done false
3654 1280 Acked 2279 Flush 0 New Done Done false
3655 1281 Acked 2280 Read 10 New Done Done false
3656 1282 Acked 2281 Write 5 New Done Done false
3657 1283 Acked 2282 Read 7 New Done Done false
3658 1284 Acked 2283 Write 9 New Done Done false
3659 1285 Acked 2284 Write 2 New Done Done false
3660 1286 Acked 2285 Read 10 New Done Done false
3661 1287 Acked 2286 Flush 0 New Done Done false
3662 1288 Acked 2287 Write 8 New Done Done false
3663 1289 Acked 2288 Read 10 New Done Done false
3664 1290 Acked 2289 Write 2 New Done Done false
3665 1291 Acked 2290 Write 4 New Done Done false
3666 1292 Acked 2291 Read 7 New Done Done false
3667 1293 Acked 2292 Read 9 New Done Done false
3668 1294 Acked 2293 Flush 0 New Done Done false
3669 1295 Acked 2294 Flush 0 New Done Done false
3670 1296 Acked 2295 Read 5 New Done Done false
3671 1297 Acked 2296 Read 10 New Done Done false
3672 1298 Acked 2297 Write 1 New Done Done false
3673 1299 Acked 2298 Write 9 New Done Done false
3674 1300 Acked 2299 Read 1 New Done Done false
3675 1301 Acked 2300 Read 4 New Done Done false
3676 1302 Acked 2301 Write 7 New Done Done false
3677 1303 Acked 2302 Read 2 New Done Done false
3678 1304 Acked 2303 Write 3 New Done Done false
3679 1305 Acked 2304 Read 10 New Done Done false
3680 1306 Acked 2305 Read 3 New Done Done false
3681 1307 Acked 2306 Read 8 New Done Done false
3682 1308 Acked 2307 Write 7 New Done Done false
3683 1309 Acked 2308 Read 9 New Done Done false
3684 1310 Acked 2309 Write 7 New Done Done false
3685 1311 Acked 2310 Read 9 New Done Done false
3686 1312 Acked 2311 Write 1 New Done Done false
3687 1313 Acked 2312 Write 3 New Done Done false
3688 1314 Acked 2313 Write 10 New Done Done false
3689 1315 Acked 2314 Write 10 New Done Done false
3690 1316 Acked 2315 Read 4 New Done Done false
3691 1317 Acked 2316 Write 6 New Done Done false
3692 1318 Acked 2317 Read 4 New Done Done false
3693 1319 Acked 2318 Read 6 New Done Done false
3694 1320 Acked 2319 Read 7 New Done Done false
3695 1321 Acked 2320 Read 4 New Done Done false
3696 1322 Acked 2321 Write 6 New Done Done false
3697 1323 Acked 2322 Read 7 New Done Done false
3698 1324 Acked 2323 Read 1 New Done Done false
3699 1325 Acked 2324 Flush 0 New Done Done false
3700 1326 Acked 2325 Flush 0 New Done Done false
3701 1327 Acked 2326 Read 7 New Done Done false
3702 1328 Acked 2327 Write 3 New Done Done false
3703 1329 Acked 2328 Flush 0 New Done Done false
3704 1330 Acked 2329 Read 1 New Done Done false
3705 1331 Acked 2330 Read 10 New Done Done false
3706 1332 Acked 2331 Flush 0 New Done Done false
3707 1333 Acked 2332 Read 8 New Done Done false
3708 1334 Acked 2333 Flush 0 New Done Done false
3709 1335 Acked 2334 Read 3 New Done Done false
3710 1336 Acked 2335 Read 8 New Done Done false
3711 1337 Acked 2336 Write 8 New Done Done false
3712 1338 Acked 2337 Flush 0 New Done Done false
3713 1339 Acked 2338 Read 1 New Done Done false
3714 1340 Acked 2339 Read 7 New Done Done false
3715 1341 Acked 2340 Write 1 New Done Done false
3716 1342 Acked 2341 Read 9 New Done Done false
3717 1343 Acked 2342 Read 7 New Done Done false
3718 1344 Acked 2343 Write 7 New Done Done false
3719 1345 Acked 2344 Read 8 New Done Done false
3720 1346 Acked 2345 Read 2 New Done Done false
3721 1347 Acked 2346 Read 2 New Done Done false
3722 1348 Acked 2347 Read 5 New Done Done false
3723 1349 Acked 2348 Write 8 New Done Done false
3724 1350 Acked 2349 Read 10 New Done Done false
3725 1351 Acked 2350 Read 1 New Done Done false
3726 1352 Acked 2351 Write 1 New Done Done false
3727 1353 Acked 2352 Read 2 New Done Done false
3728 1354 Acked 2353 Read 7 New Done Done false
3729 1355 Acked 2354 Write 6 New Done Done false
3730 1356 Acked 2355 Read 10 New Done Done false
3731 1357 Acked 2356 Write 6 New Done Done false
3732 1358 Acked 2357 Write 6 New Done Done false
3733 1359 Acked 2358 Write 7 New Done Done false
3734 1360 Acked 2359 Read 10 New Done Done false
3735 1361 Acked 2360 Read 4 New Done Done false
3736 1362 Acked 2361 Write 3 New Done Done false
3737 1363 Acked 2362 Flush 0 New Done Done false
3738 1364 Acked 2363 Write 10 New Done Done false
3739 1365 Acked 2364 Write 2 New Done Done false
3740 1366 Acked 2365 Read 5 New Done Done false
3741 1367 Acked 2366 Flush 0 New Done Done false
3742 1368 Acked 2367 Read 8 New Done Done false
3743 1369 Acked 2368 Write 2 New Done Done false
3744 1370 Acked 2369 Write 5 New Done Done false
3745 1371 Acked 2370 Read 8 New Done Done false
3746 1372 Acked 2371 Flush 0 New Done Done false
3747 1373 Acked 2372 Read 9 New Done Done false
3748 1374 Acked 2373 Read 3 New Done Done false
3749 1375 Acked 2374 Read 2 New Done Done false
3750 1376 Acked 2375 Write 10 New Done Done false
3751 1377 Acked 2376 Write 6 New Done Done false
3752 1378 Acked 2377 Write 8 New Done Done false
3753 1379 Acked 2378 Flush 0 New Done Done false
3754 1380 Acked 2379 Read 9 New Done Done false
3755 1381 Acked 2380 Write 5 New Done Done false
3756 1382 Acked 2381 Read 1 New Done Done false
3757 1383 Acked 2382 Write 7 New Done Done false
3758 1384 Acked 2383 Read 4 New Done Done false
3759 1385 Acked 2384 Read 4 New Done Done false
3760 1386 Acked 2385 Read 4 New Done Done false
3761 1387 Acked 2386 Read 8 New Done Done false
3762 1388 Acked 2387 Read 8 New Done Done false
3763 1389 Acked 2388 Read 4 New Done Done false
3764 1390 Acked 2389 Write 8 New Done Done false
3765 1391 Acked 2390 Read 1 New Done Done false
3766 1392 Acked 2391 Read 7 New Done Done false
3767 1393 Acked 2392 Write 9 New Done Done false
3768 1394 Acked 2393 Write 10 New Done Done false
3769 1395 Acked 2394 Read 3 New Done Done false
3770 1396 Acked 2395 Flush 0 New Done Done false
3771 1397 Acked 2396 Write 5 New Done Done false
3772 1398 Acked 2397 Read 10 New Done Done false
3773 1399 Acked 2398 Write 2 New Done Done false
3774 1400 Acked 2399 Read 7 New Done Done false
3775 1401 Acked 2400 Read 2 New Done Done false
3776 1402 Acked 2401 Write 9 New Done Done false
3777 1403 Acked 2402 Read 2 New Done Done false
3778 1404 Acked 2403 Write 4 New Done Done false
3779 1405 Acked 2404 Read 9 New Done Done false
3780 1406 Acked 2405 Flush 0 New Done Done false
3781 1407 Acked 2406 Write 5 New Done Done false
3782 1408 Acked 2407 Write 5 New Done Done false
3783 1409 Acked 2408 Write 1 New Done Done false
3784 1410 Acked 2409 Read 1 New Done Done false
3785 1411 Acked 2410 Read 5 New Done Done false
3786 1412 Acked 2411 Read 1 New Done Done false
3787 1413 Acked 2412 Read 5 New Done Done false
3788 1414 Acked 2413 Read 8 New Done Done false
3789 1415 Acked 2414 Write 2 New Done Done false
3790 1416 Acked 2415 Read 5 New Done Done false
3791 1417 Acked 2416 Flush 0 New Done Done false
3792 1418 Acked 2417 Write 6 New Done Done false
3793 1419 Acked 2418 Read 8 New Done Done false
3794 1420 Acked 2419 Write 5 New Done Done false
3795 1421 Acked 2420 Write 4 New Done Done false
3796 1422 Acked 2421 Write 2 New Done Done false
3797 1423 Acked 2422 Write 6 New Done Done false
3798 1424 Acked 2423 Flush 0 New Done Done false
3799 1425 Acked 2424 Write 7 New Done Done false
3800 1426 Acked 2425 Read 1 New Done Done false
3801 1427 Acked 2426 Write 7 New Done Done false
3802 1428 Acked 2427 Write 8 New Done Done false
3803 1429 Acked 2428 Read 9 New Done Done false
3804 1430 Acked 2429 Read 10 New Done Done false
3805 1431 Acked 2430 Read 8 New Done Done false
3806 1432 Acked 2431 Read 6 New Done Done false
3807 1433 Acked 2432 Read 10 New Done Done false
3808 1434 Acked 2433 Write 5 New Done Done false
3809 1435 Acked 2434 Read 5 New Done Done false
3810 1436 Acked 2435 Flush 0 New Done Done false
3811 1437 Acked 2436 Read 9 New Done Done false
3812 1438 Acked 2437 Read 3 New Done Done false
3813 1439 Acked 2438 Read 8 New Done Done false
3814 1440 Acked 2439 Write 5 New Done Done false
3815 1441 Acked 2440 Read 5 New Done Done false
3816 1442 Acked 2441 Write 1 New Done Done false
3817 1443 Acked 2442 Write 1 New Done Done false
3818 1444 Acked 2443 Write 6 New Done Done false
3819 1445 Acked 2444 Read 3 New Done Done false
3820 1446 Acked 2445 Read 3 New Done Done false
3821 1447 Acked 2446 Write 8 New Done Done false
3822 1448 Acked 2447 Read 1 New Done Done false
3823 1449 Acked 2448 Read 9 New Done Done false
3824 1450 Acked 2449 Write 1 New Done Done false
3825 1451 Acked 2450 Read 2 New Done Done false
3826 1452 Acked 2451 Flush 0 New Done Done false
3827 1453 Acked 2452 Read 3 New Done Done false
3828 1454 Acked 2453 Write 9 New Done Done false
3829 1455 Acked 2454 Read 6 New Done Done false
3830 1456 Acked 2455 Read 3 New Done Done false
3831 1457 Acked 2456 Write 1 New Done Done false
3832 1458 Acked 2457 Read 4 New Done Done false
3833 1459 Acked 2458 Write 9 New Done Done false
3834 1460 Acked 2459 Read 5 New Done Done false
3835 1461 Acked 2460 Read 1 New Done Done false
3836 1462 Acked 2461 Read 6 New Done Done false
3837 1463 Acked 2462 Read 5 New Done Done false
3838 1464 Acked 2463 Read 3 New Done Done false
3839 1465 Acked 2464 Read 1 New Done Done false
3840 1466 Acked 2465 Read 9 New Done Done false
3841 1467 Acked 2466 Write 2 New Done Done false
3842 1468 Acked 2467 Write 7 New Done Done false
3843 1469 Acked 2468 Write 7 New Done Done false
3844 1470 Acked 2469 Read 4 New Done Done false
3845 1471 Acked 2470 Write 9 New Done Done false
3846 1472 Acked 2471 Read 5 New Done Done false
3847 1473 Acked 2472 Read 3 New Done Done false
3848 1474 Acked 2473 Write 4 New Done Done false
3849 1475 Acked 2474 Read 1 New Done Done false
3850 1476 Acked 2475 Flush 0 New Done Done false
3851 1477 Acked 2476 Read 2 New Done Done false
3852 1478 Acked 2477 Flush 0 New Done Done false
3853 1479 Acked 2478 Write 2 New Done Done false
3854 1480 Acked 2479 Read 5 New Done Done false
3855 1481 Acked 2480 Read 4 New Done Done false
3856 1482 Acked 2481 Write 10 New Done Done false
3857 1483 Acked 2482 Read 1 New Done Done false
3858 1484 Acked 2483 Write 7 New Done Done false
3859 1485 Acked 2484 Flush 0 New Done Done false
3860 1486 Acked 2485 Read 8 New Done Done false
3861 1487 Acked 2486 Write 3 New Done Done false
3862 1488 Acked 2487 Read 6 New Done Done false
3863 1489 Acked 2488 Read 8 New Done Done false
3864 1490 Acked 2489 Read 2 New Done Done false
3865 1491 Acked 2490 Write 4 New Done Done false
3866 1492 Acked 2491 Read 10 New Done Done false
3867 1493 Acked 2492 Read 1 New Done Done false
3868 1494 Acked 2493 Read 8 New Done Done false
3869 1495 Acked 2494 Read 7 New Done Done false
3870 1496 Acked 2495 Flush 0 New Done Done false
3871 1497 Acked 2496 Read 9 New Done Done false
3872 1498 Acked 2497 Write 1 New Done Done false
3873 1499 Acked 2498 Read 1 New Done Done false
3874 1500 Acked 2499 Write 4 New Done Done false
3875 1501 Acked 2500 Read 1 New Done Done false
3876 1502 Acked 2501 Read 5 New Done Done false
3877 1503 Acked 2502 Flush 0 New Done Done false
3878 1504 Acked 2503 Write 4 New Done Done false
3879 1505 Acked 2504 Read 5 New Done Done false
3880 1506 Acked 2505 Read 7 New Done Done false
3881 1507 Acked 2506 Read 5 New Done Done false
3882 1508 Acked 2507 Write 6 New Done Done false
3883 1509 Acked 2508 Write 4 New Sent Sent false
3884 1510 Acked 2509 Write 3 New Sent Sent false
3885 1511 Acked 2510 Write 1 New Sent Sent false
3886 1512 Acked 2511 Write 4 New Sent Sent false
3887 STATES DS:0 DS:1 DS:2 TOTAL
3888 New 302 0 0 302
3889 Sent 0 4 4 8
3890 Done 0 298 298 596
3891 Skipped 0 0 0 0
3892 Error 0 0 0 0
3893 Last Flush: 2209 2502 2502
3894 Downstairs last five completed: 2209 2208 2207 2206 2205 2204
3895 Upstairs last five completed: 1512 1511 1510 1509 1508 1507
3896 CLIENT: Up:0 ds:302 act:2
38972023-10-13T18:09:52.998ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
3898 [8810] got start action
3899 [8810] I am going to restart
3900 Starting downstairs at port 8810
3901 Make output file at "/tmp/dsc/downstairs-8810.txt"
3902 [8810][0] reports Starting
3903 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1181)
3904 [8810][0] reports Running
3905 ----------------------------------------------------------------
3906 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
3907 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3908 1211 Acked 2210 Read 10 New Done Done false
3909 1212 Acked 2211 Read 10 New Done Done false
3910 1213 Acked 2212 Read 2 New Done Done false
3911 1214 Acked 2213 Read 7 New Done Done false
3912 1215 Acked 2214 Read 1 New Done Done false
3913 1216 Acked 2215 Write 7 New Done Done false
3914 1217 Acked 2216 Write 6 New Done Done false
3915 1218 Acked 2217 Write 5 New Done Done false
3916 1219 Acked 2218 Read 3 New Done Done false
3917 1220 Acked 2219 Read 7 New Done Done false
3918 1221 Acked 2220 Read 1 New Done Done false
3919 1222 Acked 2221 Write 2 New Done Done false
3920 1223 Acked 2222 Write 8 New Done Done false
3921 1224 Acked 2223 Write 5 New Done Done false
3922 1225 Acked 2224 Read 8 New Done Done false
3923 1226 Acked 2225 Read 5 New Done Done false
3924 1227 Acked 2226 Write 10 New Done Done false
3925 1228 Acked 2227 Read 4 New Done Done false
3926 1229 Acked 2228 Read 9 New Done Done false
3927 1230 Acked 2229 Write 2 New Done Done false
3928 1231 Acked 2230 Read 1 New Done Done false
3929 1232 Acked 2231 Write 4 New Done Done false
3930 1233 Acked 2232 Write 10 New Done Done false
3931 1234 Acked 2233 Read 1 New Done Done false
3932 1235 Acked 2234 Read 5 New Done Done false
3933 1236 Acked 2235 Read 3 New Done Done false
3934 1237 Acked 2236 Read 2 New Done Done false
3935 1238 Acked 2237 Read 4 New Done Done false
3936 1239 Acked 2238 Read 4 New Done Done false
3937 1240 Acked 2239 Read 8 New Done Done false
3938 1241 Acked 2240 Read 1 New Done Done false
3939 1242 Acked 2241 Write 9 New Done Done false
3940 1243 Acked 2242 Flush 0 New Done Done false
3941 1244 Acked 2243 Write 8 New Done Done false
3942 1245 Acked 2244 Write 1 New Done Done false
3943 1246 Acked 2245 Read 3 New Done Done false
3944 1247 Acked 2246 Read 8 New Done Done false
3945 1248 Acked 2247 Read 4 New Done Done false
3946 1249 Acked 2248 Read 3 New Done Done false
3947 1250 Acked 2249 Flush 0 New Done Done false
3948 1251 Acked 2250 Write 5 New Done Done false
3949 1252 Acked 2251 Write 3 New Done Done false
3950 1253 Acked 2252 Read 10 New Done Done false
3951 1254 Acked 2253 Write 3 New Done Done false
3952 1255 Acked 2254 Flush 0 New Done Done false
3953 1256 Acked 2255 Write 1 New Done Done false
3954 1257 Acked 2256 Read 3 New Done Done false
3955 1258 Acked 2257 Read 5 New Done Done false
3956 1259 Acked 2258 Read 9 New Done Done false
3957 1260 Acked 2259 Read 8 New Done Done false
3958 1261 Acked 2260 Write 1 New Done Done false
3959 1262 Acked 2261 Read 3 New Done Done false
3960 1263 Acked 2262 Write 8 New Done Done false
3961 1264 Acked 2263 Read 8 New Done Done false
3962 1265 Acked 2264 Read 5 New Done Done false
3963 1266 Acked 2265 Read 4 New Done Done false
3964 1267 Acked 2266 Write 9 New Done Done false
3965 1268 Acked 2267 Read 7 New Done Done false
3966 1269 Acked 2268 Write 3 New Done Done false
3967 1270 Acked 2269 Read 2 New Done Done false
3968 1271 Acked 2270 Write 3 New Done Done false
3969 1272 Acked 2271 Read 8 New Done Done false
3970 1273 Acked 2272 Write 1 New Done Done false
3971 1274 Acked 2273 Read 2 New Done Done false
3972 1275 Acked 2274 Read 8 New Done Done false
3973 1276 Acked 2275 Flush 0 New Done Done false
3974 1277 Acked 2276 Write 5 New Done Done false
3975 1278 Acked 2277 Flush 0 New Done Done false
3976 1279 Acked 2278 Read 8 New Done Done false
3977 1280 Acked 2279 Flush 0 New Done Done false
3978 1281 Acked 2280 Read 10 New Done Done false
3979 1282 Acked 2281 Write 5 New Done Done false
3980 1283 Acked 2282 Read 7 New Done Done false
3981 1284 Acked 2283 Write 9 New Done Done false
3982 1285 Acked 2284 Write 2 New Done Done false
3983 1286 Acked 2285 Read 10 New Done Done false
3984 1287 Acked 2286 Flush 0 New Done Done false
3985 1288 Acked 2287 Write 8 New Done Done false
3986 1289 Acked 2288 Read 10 New Done Done false
3987 1290 Acked 2289 Write 2 New Done Done false
3988 1291 Acked 2290 Write 4 New Done Done false
3989 1292 Acked 2291 Read 7 New Done Done false
3990 1293 Acked 2292 Read 9 New Done Done false
3991 1294 Acked 2293 Flush 0 New Done Done false
3992 1295 Acked 2294 Flush 0 New Done Done false
3993 1296 Acked 2295 Read 5 New Done Done false
3994 1297 Acked 2296 Read 10 New Done Done false
3995 1298 Acked 2297 Write 1 New Done Done false
3996 1299 Acked 2298 Write 9 New Done Done false
3997 1300 Acked 2299 Read 1 New Done Done false
3998 1301 Acked 2300 Read 4 New Done Done false
3999 1302 Acked 2301 Write 7 New Done Done false
4000 1303 Acked 2302 Read 2 New Done Done false
4001 1304 Acked 2303 Write 3 New Done Done false
4002 1305 Acked 2304 Read 10 New Done Done false
4003 1306 Acked 2305 Read 3 New Done Done false
4004 1307 Acked 2306 Read 8 New Done Done false
4005 1308 Acked 2307 Write 7 New Done Done false
4006 1309 Acked 2308 Read 9 New Done Done false
4007 1310 Acked 2309 Write 7 New Done Done false
4008 1311 Acked 2310 Read 9 New Done Done false
4009 1312 Acked 2311 Write 1 New Done Done false
4010 1313 Acked 2312 Write 3 New Done Done false
4011 1314 Acked 2313 Write 10 New Done Done false
4012 1315 Acked 2314 Write 10 New Done Done false
4013 1316 Acked 2315 Read 4 New Done Done false
4014 1317 Acked 2316 Write 6 New Done Done false
4015 1318 Acked 2317 Read 4 New Done Done false
4016 1319 Acked 2318 Read 6 New Done Done false
4017 1320 Acked 2319 Read 7 New Done Done false
4018 1321 Acked 2320 Read 4 New Done Done false
4019 1322 Acked 2321 Write 6 New Done Done false
4020 1323 Acked 2322 Read 7 New Done Done false
4021 1324 Acked 2323 Read 1 New Done Done false
4022 1325 Acked 2324 Flush 0 New Done Done false
4023 1326 Acked 2325 Flush 0 New Done Done false
4024 1327 Acked 2326 Read 7 New Done Done false
4025 1328 Acked 2327 Write 3 New Done Done false
4026 1329 Acked 2328 Flush 0 New Done Done false
4027 1330 Acked 2329 Read 1 New Done Done false
4028 1331 Acked 2330 Read 10 New Done Done false
4029 1332 Acked 2331 Flush 0 New Done Done false
4030 1333 Acked 2332 Read 8 New Done Done false
4031 1334 Acked 2333 Flush 0 New Done Done false
4032 1335 Acked 2334 Read 3 New Done Done false
4033 1336 Acked 2335 Read 8 New Done Done false
4034 1337 Acked 2336 Write 8 New Done Done false
4035 1338 Acked 2337 Flush 0 New Done Done false
4036 1339 Acked 2338 Read 1 New Done Done false
4037 1340 Acked 2339 Read 7 New Done Done false
4038 1341 Acked 2340 Write 1 New Done Done false
4039 1342 Acked 2341 Read 9 New Done Done false
4040 1343 Acked 2342 Read 7 New Done Done false
4041 1344 Acked 2343 Write 7 New Done Done false
4042 1345 Acked 2344 Read 8 New Done Done false
4043 1346 Acked 2345 Read 2 New Done Done false
4044 1347 Acked 2346 Read 2 New Done Done false
4045 1348 Acked 2347 Read 5 New Done Done false
4046 1349 Acked 2348 Write 8 New Done Done false
4047 1350 Acked 2349 Read 10 New Done Done false
4048 1351 Acked 2350 Read 1 New Done Done false
4049 1352 Acked 2351 Write 1 New Done Done false
4050 1353 Acked 2352 Read 2 New Done Done false
4051 1354 Acked 2353 Read 7 New Done Done false
4052 1355 Acked 2354 Write 6 New Done Done false
4053 1356 Acked 2355 Read 10 New Done Done false
4054 1357 Acked 2356 Write 6 New Done Done false
4055 1358 Acked 2357 Write 6 New Done Done false
4056 1359 Acked 2358 Write 7 New Done Done false
4057 1360 Acked 2359 Read 10 New Done Done false
4058 1361 Acked 2360 Read 4 New Done Done false
4059 1362 Acked 2361 Write 3 New Done Done false
4060 1363 Acked 2362 Flush 0 New Done Done false
4061 1364 Acked 2363 Write 10 New Done Done false
4062 1365 Acked 2364 Write 2 New Done Done false
4063 1366 Acked 2365 Read 5 New Done Done false
4064 1367 Acked 2366 Flush 0 New Done Done false
4065 1368 Acked 2367 Read 8 New Done Done false
4066 1369 Acked 2368 Write 2 New Done Done false
4067 1370 Acked 2369 Write 5 New Done Done false
4068 1371 Acked 2370 Read 8 New Done Done false
4069 1372 Acked 2371 Flush 0 New Done Done false
4070 1373 Acked 2372 Read 9 New Done Done false
4071 1374 Acked 2373 Read 3 New Done Done false
4072 1375 Acked 2374 Read 2 New Done Done false
4073 1376 Acked 2375 Write 10 New Done Done false
4074 1377 Acked 2376 Write 6 New Done Done false
4075 1378 Acked 2377 Write 8 New Done Done false
4076 1379 Acked 2378 Flush 0 New Done Done false
4077 1380 Acked 2379 Read 9 New Done Done false
4078 1381 Acked 2380 Write 5 New Done Done false
4079 1382 Acked 2381 Read 1 New Done Done false
4080 1383 Acked 2382 Write 7 New Done Done false
4081 1384 Acked 2383 Read 4 New Done Done false
4082 1385 Acked 2384 Read 4 New Done Done false
4083 1386 Acked 2385 Read 4 New Done Done false
4084 1387 Acked 2386 Read 8 New Done Done false
4085 1388 Acked 2387 Read 8 New Done Done false
4086 1389 Acked 2388 Read 4 New Done Done false
4087 1390 Acked 2389 Write 8 New Done Done false
4088 1391 Acked 2390 Read 1 New Done Done false
4089 1392 Acked 2391 Read 7 New Done Done false
4090 1393 Acked 2392 Write 9 New Done Done false
4091 1394 Acked 2393 Write 10 New Done Done false
4092 1395 Acked 2394 Read 3 New Done Done false
4093 1396 Acked 2395 Flush 0 New Done Done false
4094 1397 Acked 2396 Write 5 New Done Done false
4095 1398 Acked 2397 Read 10 New Done Done false
4096 1399 Acked 2398 Write 2 New Done Done false
4097 1400 Acked 2399 Read 7 New Done Done false
4098 1401 Acked 2400 Read 2 New Done Done false
4099 1402 Acked 2401 Write 9 New Done Done false
4100 1403 Acked 2402 Read 2 New Done Done false
4101 1404 Acked 2403 Write 4 New Done Done false
4102 1405 Acked 2404 Read 9 New Done Done false
4103 1406 Acked 2405 Flush 0 New Done Done false
4104 1407 Acked 2406 Write 5 New Done Done false
4105 1408 Acked 2407 Write 5 New Done Done false
4106 1409 Acked 2408 Write 1 New Done Done false
4107 1410 Acked 2409 Read 1 New Done Done false
4108 1411 Acked 2410 Read 5 New Done Done false
4109 1412 Acked 2411 Read 1 New Done Done false
4110 1413 Acked 2412 Read 5 New Done Done false
4111 1414 Acked 2413 Read 8 New Done Done false
4112 1415 Acked 2414 Write 2 New Done Done false
4113 1416 Acked 2415 Read 5 New Done Done false
4114 1417 Acked 2416 Flush 0 New Done Done false
4115 1418 Acked 2417 Write 6 New Done Done false
4116 1419 Acked 2418 Read 8 New Done Done false
4117 1420 Acked 2419 Write 5 New Done Done false
4118 1421 Acked 2420 Write 4 New Done Done false
4119 1422 Acked 2421 Write 2 New Done Done false
4120 1423 Acked 2422 Write 6 New Done Done false
4121 1424 Acked 2423 Flush 0 New Done Done false
4122 1425 Acked 2424 Write 7 New Done Done false
4123 1426 Acked 2425 Read 1 New Done Done false
4124 1427 Acked 2426 Write 7 New Done Done false
4125 1428 Acked 2427 Write 8 New Done Done false
4126 1429 Acked 2428 Read 9 New Done Done false
4127 1430 Acked 2429 Read 10 New Done Done false
4128 1431 Acked 2430 Read 8 New Done Done false
4129 1432 Acked 2431 Read 6 New Done Done false
4130 1433 Acked 2432 Read 10 New Done Done false
4131 1434 Acked 2433 Write 5 New Done Done false
4132 1435 Acked 2434 Read 5 New Done Done false
4133 1436 Acked 2435 Flush 0 New Done Done false
4134 1437 Acked 2436 Read 9 New Done Done false
4135 1438 Acked 2437 Read 3 New Done Done false
4136 1439 Acked 2438 Read 8 New Done Done false
4137 1440 Acked 2439 Write 5 New Done Done false
4138 1441 Acked 2440 Read 5 New Done Done false
4139 1442 Acked 2441 Write 1 New Done Done false
4140 1443 Acked 2442 Write 1 New Done Done false
4141 1444 Acked 2443 Write 6 New Done Done false
4142 1445 Acked 2444 Read 3 New Done Done false
4143 1446 Acked 2445 Read 3 New Done Done false
4144 1447 Acked 2446 Write 8 New Done Done false
4145 1448 Acked 2447 Read 1 New Done Done false
4146 1449 Acked 2448 Read 9 New Done Done false
4147 1450 Acked 2449 Write 1 New Done Done false
4148 1451 Acked 2450 Read 2 New Done Done false
4149 1452 Acked 2451 Flush 0 New Done Done false
4150 1453 Acked 2452 Read 3 New Done Done false
4151 1454 Acked 2453 Write 9 New Done Done false
4152 1455 Acked 2454 Read 6 New Done Done false
4153 1456 Acked 2455 Read 3 New Done Done false
4154 1457 Acked 2456 Write 1 New Done Done false
4155 1458 Acked 2457 Read 4 New Done Done false
4156 1459 Acked 2458 Write 9 New Done Done false
4157 1460 Acked 2459 Read 5 New Done Done false
4158 1461 Acked 2460 Read 1 New Done Done false
4159 1462 Acked 2461 Read 6 New Done Done false
4160 1463 Acked 2462 Read 5 New Done Done false
4161 1464 Acked 2463 Read 3 New Done Done false
4162 1465 Acked 2464 Read 1 New Done Done false
4163 1466 Acked 2465 Read 9 New Done Done false
4164 1467 Acked 2466 Write 2 New Done Done false
4165 1468 Acked 2467 Write 7 New Done Done false
4166 1469 Acked 2468 Write 7 New Done Done false
4167 1470 Acked 2469 Read 4 New Done Done false
4168 1471 Acked 2470 Write 9 New Done Done false
4169 1472 Acked 2471 Read 5 New Done Done false
4170 1473 Acked 2472 Read 3 New Done Done false
4171 1474 Acked 2473 Write 4 New Done Done false
4172 1475 Acked 2474 Read 1 New Done Done false
4173 1476 Acked 2475 Flush 0 New Done Done false
4174 1477 Acked 2476 Read 2 New Done Done false
4175 1478 Acked 2477 Flush 0 New Done Done false
4176 1479 Acked 2478 Write 2 New Done Done false
4177 1480 Acked 2479 Read 5 New Done Done false
4178 1481 Acked 2480 Read 4 New Done Done false
4179 1482 Acked 2481 Write 10 New Done Done false
4180 1483 Acked 2482 Read 1 New Done Done false
4181 1484 Acked 2483 Write 7 New Done Done false
4182 1485 Acked 2484 Flush 0 New Done Done false
4183 1486 Acked 2485 Read 8 New Done Done false
4184 1487 Acked 2486 Write 3 New Done Done false
4185 1488 Acked 2487 Read 6 New Done Done false
4186 1489 Acked 2488 Read 8 New Done Done false
4187 1490 Acked 2489 Read 2 New Done Done false
4188 1491 Acked 2490 Write 4 New Done Done false
4189 1492 Acked 2491 Read 10 New Done Done false
4190 1493 Acked 2492 Read 1 New Done Done false
4191 1494 Acked 2493 Read 8 New Done Done false
4192 1495 Acked 2494 Read 7 New Done Done false
4193 1496 Acked 2495 Flush 0 New Done Done false
4194 1497 Acked 2496 Read 9 New Done Done false
4195 1498 Acked 2497 Write 1 New Done Done false
4196 1499 Acked 2498 Read 1 New Done Done false
4197 1500 Acked 2499 Write 4 New Done Done false
4198 1501 Acked 2500 Read 1 New Done Done false
4199 1502 Acked 2501 Read 5 New Done Done false
4200 1503 Acked 2502 Flush 0 New Done Done false
4201 1504 Acked 2503 Write 4 New Done Done false
4202 1505 Acked 2504 Read 5 New Done Done false
4203 1506 Acked 2505 Read 7 New Done Done false
4204 1507 Acked 2506 Read 5 New Done Done false
4205 1508 Acked 2507 Write 6 New Done Done false
4206 1509 Acked 2508 Write 4 New Done Done false
4207 1510 Acked 2509 Write 3 New Done Done false
4208 1511 Acked 2510 Write 1 New Done Done false
4209 1512 Acked 2511 Write 4 New Done Done false
4210 1513 Acked 2512 Flush 0 New Done Done false
4211 STATES DS:0 DS:1 DS:2 TOTAL
4212 New 303 0 0 303
4213 Sent 0 0 0 0
4214 Done 0 303 303 606
4215 Skipped 0 0 0 0
4216 Error 0 0 0 0
4217 Last Flush: 2209 2512 2512
4218 Downstairs last five completed: 2209 2208 2207 2206 2205 2204
4219 Upstairs last five completed: 1513 1512 1511 1510 1509 1508
4220 CLIENT: Up:0 ds:303 act:2
42212023-10-13T18:09:56.001ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 0
42222023-10-13T18:09:56.001ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state Offline
42232023-10-13T18:09:56.001ZINFOcrucible: [0] client re-new 303 jobs since flush 2209 = downstairs
42242023-10-13T18:09:56.002ZINFOcrucible: [0] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
42252023-10-13T18:09:56.003ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
42262023-10-13T18:09:56.003ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
42272023-10-13T18:09:56.003ZINFOcrucible: Returning client:0 UUID:12345678-0000-0000-0000-000000008810 matches
42282023-10-13T18:09:56.003ZINFOcrucible: [0] send last flush ID to this DS: 2209
42292023-10-13T18:09:56.003ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
42302023-10-13T18:09:56.003ZINFOcrucible: [0] Replied this last flush ID: 2209
42312023-10-13T18:09:56.003ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Offline Active Active ds_transition to Replay
42322023-10-13T18:09:56.003ZINFOcrucible: [0] Transition from Offline to Replay
42332023-10-13T18:09:56.003ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
42342023-10-13T18:09:56.003ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Replay Active Active ds_transition to Active
42352023-10-13T18:09:56.003ZINFOcrucible: [0] Transition from Replay to Active
42362023-10-13T18:09:56.003ZINFOcrucible: [0] Starts cmd_loop
42372023-10-13T18:09:56.003ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
42382023-10-13T18:09:56.003ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
42392023-10-13T18:09:56.003ZINFOcrucible: Set check for repair
42402023-10-13T18:09:57.004ZINFOcrucible: Checking if live repair is needed
42412023-10-13T18:09:57.004ZINFOcrucible: No Live Repair required at this time
42422023-10-13T18:09:57.004ZWARNcrucible: [0] flow control end
4243 ----------------------------------------------------------------
4244 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4245 Downstairs last five completed: 2512 2511 2510 2509 2508 2507
4246 Upstairs last five completed: 1513 1512 1511 1510 1509 1508
4247 CLIENT: Up:0 ds:0 act:3
4248 Replay: All jobs finished, all DS active.
4249 Main task has work to do, go find it
4250 got dsc Stop(1)
4251 stop 1
4252 Oct 13 18:09:59.964 INFO request completed, latency_us: 499, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 47669bbd-43af-4173-855e-7a77422ac14a, remote_addr: 127.0.0.1[8820] Got stop action so:false kr:false
4253 :64571, local_addr: 127.0.0.1:9998
4254 Oct 13 18:09:59.965 INFO request completed, latency_us: 175, response_code: 200, uri: /state/cid/1, method: GET, req_id: 3e3b55ea-ea8d-4a12-87da-8214d916391f, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
42552023-10-13T18:09:59.965ZWARNcrucible: [1] downstairs disconnected
42562023-10-13T18:09:59.965ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
42572023-10-13T18:09:59.965ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8820 closed looper = 1
42582023-10-13T18:09:59.966ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
42592023-10-13T18:09:59.966ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
42602023-10-13T18:09:59.966ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
42612023-10-13T18:09:59.966ZWARNcrucible: [1] pm_task rx.recv() is None
4262 [8820] Exited with: ExitStatus(unix_wait_status(9))
4263 [8820][1] reports Exit
4264 Oct 13 18:10:03.968 INFO request completed, latency_us: 256, response_code: 200, uri: /state/cid/1, method: GET, req_id: d1b2e70e-f2d0-4867-9dff-56b5c4c5635a, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
4265 001/300 Write block 551 len 4096 data: 2
4266 002/300 Write block 4405 len 8192 data: 4 4
4267 003/300 Flush
4268 004/300 Read block 2345 len 8192
4269 005/300 Flush
4270 006/300 Write block 2366 len 20480 data: 3 4 3 3 3
4271 007/300 Flush
4272 008/300 Read block 4539 len 20480
4273 009/300 Flush
4274 010/300 Read block 3337 len 16384
4275 011/300 Flush
4276 012/300 Read block 2978 len 20480
4277 013/300 Read block 385 len 16384
4278 014/300 Read block 1169 len 20480
4279 015/300 Write block 444 len 12288 data: 2 2 2
4280 016/300 Write block 4753 len 20480 data: 3 2 4 4 4
4281 017/300 Flush
4282 018/300 Write block 1993 len 36864 data: 3 4 5 5 4 3 3 3 3
4283 019/300 Read block 38 len 16384
4284 020/300 Write block 2889 len 16384 data: 3 3 3 3
4285 021/300 Read block 1584 len 4096
4286 022/300 Write block 2581 len 8192 data: 2 2
4287 023/300 Read block 3598 len 12288
4288 024/300 Read block 4947 len 20480
4289 025/300 Read block 3344 len 28672
4290 026/300 Write block 2411 len 24576 data: 3 3 3 2 2 2
4291 027/300 Write block 3666 len 24576 data: 4 3 3 3 3 3
4292 028/300 Write block 2332 len 24576 data: 4 3 3 3 3 3
4293 029/300 Write block 314 len 40960 data: 2 2 2 2 2 3 4 5 5 5
4294 030/300 Write block 2797 len 8192 data: 2 2
4295 031/300 Read block 4513 len 12288
4296 032/300 Write block 3986 len 28672 data: 2 2 2 3 3 3 3
4297 033/300 Read block 1329 len 24576
4298 034/300 Write block 3049 len 12288 data: 2 2 2
4299 035/300 Read block 1510 len 16384
4300 036/300 Write block 4042 len 8192 data: 2 2
4301 037/300 Flush
4302 038/300 Read block 4034 len 36864
4303 039/300 Read block 4440 len 16384
4304 040/300 Write block 2362 len 24576 data: 2 2 3 3 4 5
4305 041/300 Read block 2760 len 16384
4306 042/300 Write block 330 len 20480 data: 3 2 2 2 2
4307 043/300 Write block 881 len 36864 data: 2 2 2 2 2 3 3 3 3
4308 044/300 Read block 1825 len 4096
4309 045/300 Read block 2206 len 36864
4310 046/300 Read block 2988 len 32768
4311 047/300 Write block 1563 len 4096 data: 3
4312 048/300 Read block 2876 len 4096
4313 049/300 Write block 952 len 8192 data: 2 2
4314 050/300 Write block 776 len 20480 data: 4 4 4 3 2
4315 051/300 Write block 1053 len 32768 data: 3 2 2 2 2 3 3 3
4316 052/300 Read block 15 len 12288
4317 053/300 Read block 1540 len 20480
4318 054/300 Write block 1307 len 20480 data: 2 2 3 3 3
4319 055/300 Write block 3805 len 36864 data: 4 3 3 3 3 4 4 3 3
4320 056/300 Write block 1651 len 8192 data: 4 3
4321 057/300 Flush
4322 058/300 Read block 776 len 24576
4323 059/300 Read block 1465 len 28672
4324 060/300 Read block 3328 len 32768
4325 061/300 Read block 2533 len 12288
4326 062/300 Write block 4702 len 12288 data: 4 2 2
4327 063/300 Read block 3226 len 20480
4328 064/300 Read block 2936 len 12288
4329 065/300 Read block 2169 len 40960
4330 066/300 Write block 2074 len 12288 data: 2 2 2
4331 067/300 Write block 4180 len 12288 data: 3 3 3
4332 068/300 Write block 3593 len 4096 data: 3
4333 069/300 Write block 3338 len 24576 data: 4 4 2 2 3 3
4334 070/300 Read block 4073 len 20480
4335 071/300 Write block 1825 len 8192 data: 2 2
4336 072/300 Write block 3082 len 12288 data: 2 3 3
4337 073/300 Read block 1196 len 36864
4338 074/300 Write block 2984 len 28672 data: 2 2 2 2 2 2 2
4339 075/300 Read block 3493 len 4096
4340 076/300 Flush
4341 077/300 Write block 4361 len 24576 data: 2 2 2 2 2 2
4342 078/300 Read block 4923 len 28672
4343 079/300 Read block 1662 len 20480
4344 080/300 Read block 2156 len 36864
4345 081/300 Write block 1708 len 16384 data: 3 3 2 3
4346 082/300 Read block 4815 len 24576
4347 083/300 Write block 671 len 4096 data: 2
4348 084/300 Read block 4020 len 28672
4349 085/300 Read block 4691 len 28672
4350 086/300 Read block 2262 len 8192
4351 087/300 Write block 175 len 24576 data: 2 2 2 2 2 2
4352 088/300 Read block 4421 len 36864
4353 089/300 Write block 3689 len 16384 data: 2 2 2 3
4354 090/300 Write block 928 len 16384 data: 3 4 4 4
4355 091/300 Read block 1797 len 24576
4356 092/300 Flush
4357 093/300 Read block 3132 len 20480
4358 094/300 Write block 4297 len 8192 data: 2 2
4359 095/300 Write block 4627 len 20480 data: 3 3 3 2 2
4360 096/300 Write block 1666 len 28672 data: 2 2 2 2 3 3 3
4361 097/300 Write block 1805 len 36864 data: 3 3 2 2 4 3 3 3 3
4362 098/300 Write block 1376 len 28672 data: 4 3 4 4 4 4 3
4363 099/300 Read block 2212 len 28672
4364 100/300 Write block 1397 len 36864 data: 2 2 3 3 2 2 2 2 2
4365 101/300 Read block 1661 len 4096
4366 102/300 Read block 3201 len 24576
4367 103/300 Write block 1442 len 4096 data: 2
4368 104/300 Write block 269 len 8192 data: 2 2
4369 105/300 Read block 2234 len 24576
4370 106/300 Flush
4371 107/300 Write block 4503 len 16384 data: 2 2 2 3
4372 108/300 Read block 3064 len 36864
4373 109/300 Write block 179 len 4096 data: 3
4374 110/300 Write block 2731 len 28672 data: 2 2 3 3 3 3 3
4375 111/300 Write block 2791 len 32768 data: 2 2 2 2 2 2 3 3
4376 112/300 Read block 152 len 24576
4377 113/300 Write block 1604 len 16384 data: 2 2 2 2
4378 114/300 Flush
4379 115/300 Read block 3298 len 32768
4380 116/300 Read block 2724 len 20480
4381 117/300 Read block 1033 len 32768
4382 118/300 Read block 3198 len 20480
4383 119/300 Write block 850 len 12288 data: 3 3 3
4384 120/300 Read block 2347 len 12288
4385 121/300 Write block 1076 len 24576 data: 2 2 2 2 2 2
4386 122/300 Write block 1386 len 8192 data: 3 3
4387 123/300 Write block 3011 len 32768 data: 2 2 3 3 3 3 3 3
4388 124/300 Read block 1908 len 40960
4389 125/300 Read block 866 len 12288
4390 126/300 Write block 2008 len 20480 data: 5 4 4 3 3
4391 127/300 Flush
4392 128/300 Read block 4965 len 32768
4393 129/300 Read block 1560 len 4096
4394 130/300 Read block 3838 len 32768
4395 131/300 Read block 2638 len 12288
4396 132/300 Write block 544 len 12288 data: 3 2 2
4397 133/300 Flush
4398 134/300 Write block 4484 len 36864 data: 2 2 2 2 2 2 2 2 2
4399 135/300 Read block 3729 len 12288
4400 136/300 Write block 137 len 20480 data: 3 3 3 3 3
4401 137/300 Read block 1840 len 8192
4402 138/300 Write block 362 len 40960 data: 2 2 2 2 2 2 2 2 3 3
4403 139/300 Read block 4855 len 8192
4404 140/300 Write block 3040 len 32768 data: 2 2 2 2 2 2 2 2
4405 141/300 Write block 2556 len 12288 data: 5 4 4
4406 142/300 Flush
4407 143/300 Read block 1272 len 32768
4408 144/300 Read block 1012 len 32768
4409 145/300 Read block 1396 len 8192
4410 146/300 Write block 2380 len 20480 data: 2 2 2 2 2
4411 147/300 Read block 693 len 4096
4412 148/300 Write block 4856 len 20480 data: 3 3 4 5 5
4413 149/300 Flush
4414 150/300 Read block 457 len 32768
4415 151/300 Write block 4947 len 8192 data: 3 2
4416 152/300 Read block 4156 len 32768
4417 153/300 Write block 2259 len 8192 data: 2 3
4418 154/300 Read block 1856 len 4096
4419 155/300 Read block 4565 len 20480
4420 156/300 Flush
4421 157/300 Flush
4422 158/300 Read block 1642 len 36864
4423 159/300 Read block 3348 len 36864
4424 160/300 Read block 794 len 24576
4425 161/300 Read block 1025 len 12288
4426 162/300 Write block 1215 len 4096 data: 2
4427 163/300 Flush
4428 164/300 Write block 93 len 36864 data: 3 3 4 3 3 3 3 3 3
4429 165/300 Read block 2731 len 24576
4430 166/300 Read block 1932 len 16384
4431 167/300 Read block 1187 len 32768
4432 168/300 Write block 4045 len 32768 data: 2 2 2 2 2 2 2 2
4433 169/300 Write block 1949 len 36864 data: 4 4 4 4 4 4 3 3 2
4434 170/300 Write block 4870 len 20480 data: 2 2 2 2 2
4435 171/300 Read block 48 len 8192
4436 172/300 Flush
4437 173/300 Write block 1041 len 12288 data: 2 2 2
4438 174/300 Read block 4214 len 16384
4439 175/300 Flush
4440 176/300 Flush
4441 177/300 Read block 4194 len 28672
4442 178/300 Write block 4242 len 12288 data: 2 2 2
4443 179/300 Write block 1634 len 36864 data: 3 3 3 3 4 4 4 3 3
4444 180/300 Read block 808 len 36864
4445 181/300 Read block 4835 len 8192
4446 182/300 Read block 1849 len 28672
4447 183/300 Write block 902 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4448 184/300 Write block 556 len 36864 data: 3 5 5 4 4 3 3 3 3
4449 185/300 Read block 14 len 24576
4450 186/300 Read block 2117 len 16384
4451 187/300 Read block 2616 len 24576
4452 188/300 Read block 4865 len 12288
4453 189/300 Flush
4454 190/300 Read block 1214 len 8192
4455 191/300 Read block 3234 len 4096
4456 192/300 Read block 967 len 8192
4457 193/300 Write block 3721 len 12288 data: 4 4 3
4458 194/300 Write block 4892 len 4096 data: 2
4459 195/300 Read block 2206 len 8192
4460 196/300 Write block 1183 len 12288 data: 2 2 2
4461 197/300 Read block 1880 len 4096
4462 198/300 Write block 1448 len 28672 data: 3 3 3 3 3 3 3
4463 199/300 Read block 4947 len 28672
4464 200/300 Read block 2745 len 16384
4465 201/300 Write block 2469 len 16384 data: 3 4 6 5
4466 202/300 Write block 4552 len 40960 data: 3 3 3 3 2 2 2 2 2 2
4467 203/300 Write block 3298 len 8192 data: 3 3
4468 204/300 Write block 1196 len 12288 data: 2 3 4
4469 205/300 Write block 1583 len 12288 data: 2 2 2
4470 206/300 Write block 703 len 28672 data: 3 3 4 3 3 3 3
4471 207/300 Read block 4119 len 40960
4472 208/300 Write block 2994 len 28672 data: 4 4 4 4 4 4 4
4473 209/300 Write block 3396 len 4096 data: 3
4474 210/300 Read block 3240 len 24576
4475 211/300 Flush
4476 212/300 Read block 1645 len 24576
4477 213/300 Read block 1077 len 8192
4478 214/300 Read block 2206 len 16384
4479 215/300 Read block 2201 len 28672
4480 216/300 Write block 3000 len 32768 data: 5 3 3 3 3 3 2 2
4481 217/300 Write block 3133 len 4096 data: 2
4482 218/300 Write block 4297 len 24576 data: 3 3 2 2 2 2
4483 219/300 Read block 1007 len 36864
4484 220/300 Write block 4304 len 36864 data: 3 3 3 3 3 2 3 3 3
4485 221/300 Read block 697 len 36864
4486 222/300 Flush
4487 223/300 Write block 2305 len 8192 data: 3 4
4488 224/300 Write block 3186 len 32768 data: 2 2 2 2 2 2 2 2
4489 225/300 Write block 1616 len 8192 data: 3 2
4490 226/300 Read block 1023 len 24576
4491 227/300 Write block 1096 len 28672 data: 4 5 5 5 5 4 4
4492 228/300 Write block 2949 len 20480 data: 2 2 2 3 3
4493 229/300 Write block 1720 len 16384 data: 2 2 2 3
4494 230/300 Flush
4495 231/300 Read block 2327 len 32768
4496 232/300 Write block 3941 len 24576 data: 4 4 4 4 3 4
4497 233/300 Read block 2574 len 28672
4498 234/300 Write block 1657 len 40960 data: 2 2 2 3 3 4 4 4 3 3
4499 235/300 Read block 1159 len 32768
4500 236/300 Read block 1999 len 4096
4501 237/300 Read block 4718 len 28672
4502 238/300 Read block 233 len 16384
4503 239/300 Write block 1238 len 8192 data: 3 3
4504 240/300 Read block 1896 len 28672
4505 241/300 Read block 151 len 20480
4506 242/300 Read block 120 len 8192
4507 243/300 Read block 210 len 20480
4508 244/300 Read block 2084 len 32768
4509 245/300 Read block 165 len 16384
4510 246/300 Read block 2555 len 24576
4511 247/300 Write block 3386 len 20480 data: 2 3 3 3 3
4512 248/300 Flush
4513 249/300 Flush
4514 250/300 Write block 79 len 12288 data: 2 2 2
4515 251/300 Read block 3679 len 16384
4516 252/300 Write block 4372 len 12288 data: 2 3 3
4517 253/300 Write block 2486 len 12288 data: 4 3 3
4518 254/300 Read block 2191 len 28672
4519 255/300 Read block 361 len 24576
4520 256/300 Write block 4540 len 8192 data: 3 3
4521 257/300 Write block 2575 len 36864 data: 3 2 2 2 2 2 3 3 2
4522 258/300 Flush
4523 259/300 Write block 2548 len 16384 data: 2 2 2 2
4524 260/300 Read block 431 len 16384
4525 261/300 Read block 3972 len 4096
4526 262/300 Read block 1803 len 4096
4527 263/300 Write block 4450 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4528 264/300 Read block 2347 len 28672
4529 265/300 Write block 1449 len 28672 data: 4 4 4 4 4 4 3
4530 266/300 Read block 2172 len 36864
4531 267/300 Write block 4467 len 20480 data: 2 2 2 2 2
4532 268/300 Read block 4529 len 8192
4533 269/300 Write block 4384 len 28672 data: 3 3 3 4 3 2 2
4534 270/300 Read block 603 len 40960
4535 271/300 Write block 172 len 28672 data: 3 2 2 3 3 3 3
4536 272/300 Flush
4537 273/300 Read block 3787 len 8192
4538 274/300 Flush
4539 275/300 Write block 2533 len 4096 data: 3
4540 276/300 Write block 866 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4541 277/300 Read block 1656 len 16384
4542 278/300 Read block 3475 len 12288
4543 279/300 Read block 1087 len 16384
4544 280/300 Write block 3304 len 16384 data: 2 2 3 3
4545 281/300 Read block 775 len 20480
4546 282/300 Write block 4102 len 8192 data: 3 3
4547 283/300 Write block 1345 len 20480 data: 2 2 2 2 2
4548 284/300 Write block 2191 len 40960 data: 2 2 2 2 2 2 2 2 2 4
4549 285/300 Write block 2506 len 16384 data: 3 3 4 3
4550 286/300 Read block 2695 len 4096
4551 287/300 Write block 2759 len 36864 data: 3 4 4 4 4 4 4 3 3
4552 288/300 Write block 2055 len 20480 data: 2 3 4 3 3
4553 289/300 Write block 2190 len 36864 data: 2 3 3 3 3 3 3 3 3
4554 290/300 Write block 641 len 12288 data: 2 2 2
4555 291/300 Write block 2520 len 8192 data: 3 3
4556 292/300 Write block 1977 len 40960 data: 2 2 3 3 2 2 2 2 2 2
4557 293/300 Flush
4558 294/300 Flush
4559 295/300 Read block 4374 len 16384
4560 296/300 Write block 958 len 16384 data: 2 2 2 2
4561 297/300 Write block 401 len 20480 data: 3 2 2 2 2
4562 298/300 Read block 118 len 8192
4563 299/300 Write block 3773 len 12288 data: 2 2 2
4564 300/300 Write block 1448 len 8192 data: 4 5
4565 Main task has work to do, go find it
4566 got dsc Start(1)
4567 start 1
4568 Oct 13 18:10:04.896 INFO request completed, latency_us: 294, response_code: 204, uri: /start/cid/1, method: POST, req_id: 761ffdb7-419a-479c-89f3-f18a03bfd070, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
4569 Replay: started 1, returned:Ok(())
4570 ----------------------------------------------------------------
4571 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
4572 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4573 1514 Acked 2513 Write 1 Done New Done false
4574 1515 Acked 2514 Write 2 Done New Done false
4575 1516 Acked 2515 Flush 0 Done New Done false
4576 1517 Acked 2516 Read 2 Done New Done false
4577 1518 Acked 2517 Flush 0 Done New Done false
4578 1519 Acked 2518 Write 5 Done New Done false
4579 1520 Acked 2519 Flush 0 Done New Done false
4580 1521 Acked 2520 Read 5 Done New Done false
4581 1522 Acked 2521 Flush 0 Done New Done false
4582 1523 Acked 2522 Read 4 Done New Done false
4583 1524 Acked 2523 Flush 0 Done New Done false
4584 1525 Acked 2524 Read 5 Done New Done false
4585 1526 Acked 2525 Read 4 Done New Done false
4586 1527 Acked 2526 Read 5 Done New Done false
4587 1528 Acked 2527 Write 3 Done New Done false
4588 1529 Acked 2528 Write 5 Done New Done false
4589 1530 Acked 2529 Flush 0 Done New Done false
4590 1531 Acked 2530 Write 9 Done New Done false
4591 1532 Acked 2531 Read 4 Done New Done false
4592 1533 Acked 2532 Write 4 Done New Done false
4593 1534 Acked 2533 Read 1 Done New Done false
4594 1535 Acked 2534 Write 2 Done New Done false
4595 1536 Acked 2535 Read 3 Done New Done false
4596 1537 Acked 2536 Read 5 Done New Done false
4597 1538 Acked 2537 Read 7 Done New Done false
4598 1539 Acked 2538 Write 6 Done New Done false
4599 1540 Acked 2539 Write 6 Done New Done false
4600 1541 Acked 2540 Write 6 Done New Done false
4601 1542 Acked 2541 Write 10 Done New Done false
4602 1543 Acked 2542 Write 2 Done New Done false
4603 1544 Acked 2543 Read 3 Done New Done false
4604 1545 Acked 2544 Write 7 Done New Done false
4605 1546 Acked 2545 Read 6 Done New Done false
4606 1547 Acked 2546 Write 3 Done New Done false
4607 1548 Acked 2547 Read 4 Done New Done false
4608 1549 Acked 2548 Write 2 Done New Done false
4609 1550 Acked 2549 Flush 0 Done New Done false
4610 1551 Acked 2550 Read 9 Done New Done false
4611 1552 Acked 2551 Read 4 Done New Done false
4612 1553 Acked 2552 Write 6 Done New Done false
4613 1554 Acked 2553 Read 4 Done New Done false
4614 1555 Acked 2554 Write 5 Done New Done false
4615 1556 Acked 2555 Write 9 Done New Done false
4616 1557 Acked 2556 Read 1 Done New Done false
4617 1558 Acked 2557 Read 9 Done New Done false
4618 1559 Acked 2558 Read 8 Done New Done false
4619 1560 Acked 2559 Write 1 Done New Done false
4620 1561 Acked 2560 Read 1 Done New Done false
4621 1562 Acked 2561 Write 2 Done New Done false
4622 1563 Acked 2562 Write 5 Done New Done false
4623 1564 Acked 2563 Write 8 Done New Done false
4624 1565 Acked 2564 Read 3 Done New Done false
4625 1566 Acked 2565 Read 5 Done New Done false
4626 1567 Acked 2566 Write 5 Done New Done false
4627 1568 Acked 2567 Write 9 Done New Done false
4628 1569 Acked 2568 Write 2 Done New Done false
4629 1570 Acked 2569 Flush 0 Done New Done false
4630 1571 Acked 2570 Read 6 Done New Done false
4631 1572 Acked 2571 Read 7 Done New Done false
4632 1573 Acked 2572 Read 8 Done New Done false
4633 1574 Acked 2573 Read 3 Done New Done false
4634 1575 Acked 2574 Write 3 Done New Done false
4635 1576 Acked 2575 Read 5 Done New Done false
4636 1577 Acked 2576 Read 3 Done New Done false
4637 1578 Acked 2577 Flush 0 Done New Done false
4638 1579 Acked 2578 Read 10 Done New Done false
4639 1580 Acked 2579 Write 3 Done New Done false
4640 1581 Acked 2580 Write 3 Done New Done false
4641 1582 Acked 2581 Write 1 Done New Done false
4642 1583 Acked 2582 Write 6 Done New Done false
4643 1584 Acked 2583 Read 5 Done New Done false
4644 1585 Acked 2584 Write 2 Done New Done false
4645 1586 Acked 2585 Write 3 Done New Done false
4646 1587 Acked 2586 Read 9 Done New Done false
4647 1588 Acked 2587 Write 7 Done New Done false
4648 1589 Acked 2588 Read 1 Done New Done false
4649 1590 Acked 2589 Flush 0 Done New Done false
4650 1591 Acked 2590 Write 6 Done New Done false
4651 1592 Acked 2591 Read 7 Done New Done false
4652 1593 Acked 2592 Read 5 Done New Done false
4653 1594 Acked 2593 Read 9 Done New Done false
4654 1595 Acked 2594 Write 4 Done New Done false
4655 1596 Acked 2595 Read 6 Done New Done false
4656 1597 Acked 2596 Write 1 Done New Done false
4657 1598 Acked 2597 Read 7 Done New Done false
4658 1599 Acked 2598 Read 7 Done New Done false
4659 1600 Acked 2599 Read 2 Done New Done false
4660 1601 Acked 2600 Write 6 Done New Done false
4661 1602 Acked 2601 Read 9 Done New Done false
4662 1603 Acked 2602 Write 4 Done New Done false
4663 1604 Acked 2603 Write 4 Done New Done false
4664 1605 Acked 2604 Read 6 Done New Done false
4665 1606 Acked 2605 Flush 0 Done New Done false
4666 1607 Acked 2606 Read 5 Done New Done false
4667 1608 Acked 2607 Write 2 Done New Done false
4668 1609 Acked 2608 Write 5 Done New Done false
4669 1610 Acked 2609 Write 7 Done New Done false
4670 1611 Acked 2610 Write 9 Done New Done false
4671 1612 Acked 2611 Write 7 Done New Done false
4672 1613 Acked 2612 Read 7 Done New Done false
4673 1614 Acked 2613 Write 9 Done New Done false
4674 1615 Acked 2614 Read 1 Done New Done false
4675 1616 Acked 2615 Read 6 Done New Done false
4676 1617 Acked 2616 Write 1 Done New Done false
4677 1618 Acked 2617 Write 2 Done New Done false
4678 1619 Acked 2618 Read 6 Done New Done false
4679 1620 Acked 2619 Flush 0 Done New Done false
4680 1621 Acked 2620 Write 4 Done New Done false
4681 1622 Acked 2621 Read 9 Done New Done false
4682 1623 Acked 2622 Write 1 Done New Done false
4683 1624 Acked 2623 Write 7 Done New Done false
4684 1625 Acked 2624 Write 8 Done New Done false
4685 1626 Acked 2625 Read 6 Done New Done false
4686 1627 Acked 2626 Write 4 Done New Done false
4687 1628 Acked 2627 Flush 0 Done New Done false
4688 1629 Acked 2628 Read 8 Done New Done false
4689 1630 Acked 2629 Read 5 Done New Done false
4690 1631 Acked 2630 Read 8 Done New Done false
4691 1632 Acked 2631 Read 5 Done New Done false
4692 1633 Acked 2632 Write 3 Done New Done false
4693 1634 Acked 2633 Read 3 Done New Done false
4694 1635 Acked 2634 Write 6 Done New Done false
4695 1636 Acked 2635 Write 2 Done New Done false
4696 1637 Acked 2636 Write 8 Done New Done false
4697 1638 Acked 2637 Read 10 Done New Done false
4698 1639 Acked 2638 Read 3 Done New Done false
4699 1640 Acked 2639 Write 5 Done New Done false
4700 1641 Acked 2640 Flush 0 Done New Done false
4701 1642 Acked 2641 Read 8 Done New Done false
4702 1643 Acked 2642 Read 1 Done New Done false
4703 1644 Acked 2643 Read 8 Done New Done false
4704 1645 Acked 2644 Read 3 Done New Done false
4705 1646 Acked 2645 Write 3 Done New Done false
4706 1647 Acked 2646 Flush 0 Done New Done false
4707 1648 Acked 2647 Write 9 Done New Done false
4708 1649 Acked 2648 Read 3 Done New Done false
4709 1650 Acked 2649 Write 5 Done New Done false
4710 1651 Acked 2650 Read 2 Done New Done false
4711 1652 Acked 2651 Write 10 Done New Done false
4712 1653 Acked 2652 Read 2 Done New Done false
4713 1654 Acked 2653 Write 8 Done New Done false
4714 1655 Acked 2654 Write 3 Done New Done false
4715 1656 Acked 2655 Flush 0 Done New Done false
4716 1657 Acked 2656 Read 8 Done New Done false
4717 1658 Acked 2657 Read 8 Done New Done false
4718 1659 Acked 2658 Read 2 Done New Done false
4719 1660 Acked 2659 Write 5 Done New Done false
4720 1661 Acked 2660 Read 1 Done New Done false
4721 1662 Acked 2661 Write 5 Done New Done false
4722 1663 Acked 2662 Flush 0 Done New Done false
4723 1664 Acked 2663 Read 8 Done New Done false
4724 1665 Acked 2664 Write 2 Done New Done false
4725 1666 Acked 2665 Read 8 Done New Done false
4726 1667 Acked 2666 Write 2 Done New Done false
4727 1668 Acked 2667 Read 1 Done New Done false
4728 1669 Acked 2668 Read 5 Done New Done false
4729 1670 Acked 2669 Flush 0 Done New Done false
4730 1671 Acked 2670 Flush 0 Done New Done false
4731 1672 Acked 2671 Read 9 Done New Done false
4732 1673 Acked 2672 Read 9 Done New Done false
4733 1674 Acked 2673 Read 6 Done New Done false
4734 1675 Acked 2674 Read 3 Done New Done false
4735 1676 Acked 2675 Write 1 Done New Done false
4736 1677 Acked 2676 Flush 0 Done New Done false
4737 1678 Acked 2677 Write 9 Done New Done false
4738 1679 Acked 2678 Read 6 Done New Done false
4739 1680 Acked 2679 Read 4 Done New Done false
4740 1681 Acked 2680 Read 8 Done New Done false
4741 1682 Acked 2681 Write 8 Done New Done false
4742 1683 Acked 2682 Write 9 Done New Done false
4743 1684 Acked 2683 Write 5 Done New Done false
4744 1685 Acked 2684 Read 2 Done New Done false
4745 1686 Acked 2685 Flush 0 Done New Done false
4746 1687 Acked 2686 Write 3 Done New Done false
4747 1688 Acked 2687 Read 4 Done New Done false
4748 1689 Acked 2688 Flush 0 Done New Done false
4749 1690 Acked 2689 Flush 0 Done New Done false
4750 1691 Acked 2690 Read 7 Done New Done false
4751 1692 Acked 2691 Write 3 Done New Done false
4752 1693 Acked 2692 Write 9 Done New Done false
4753 1694 Acked 2693 Read 9 Done New Done false
4754 1695 Acked 2694 Read 2 Done New Done false
4755 1696 Acked 2695 Read 7 Done New Done false
4756 1697 Acked 2696 Write 10 Done New Done false
4757 1698 Acked 2697 Write 9 Done New Done false
4758 1699 Acked 2698 Read 6 Done New Done false
4759 1700 Acked 2699 Read 4 Done New Done false
4760 1701 Acked 2700 Read 6 Done New Done false
4761 1702 Acked 2701 Read 3 Done New Done false
4762 1703 Acked 2702 Flush 0 Done New Done false
4763 1704 Acked 2703 Read 2 Done New Done false
4764 1705 Acked 2704 Read 1 Done New Done false
4765 1706 Acked 2705 Read 2 Done New Done false
4766 1707 Acked 2706 Write 3 Done New Done false
4767 1708 Acked 2707 Write 1 Done New Done false
4768 1709 Acked 2708 Read 2 Done New Done false
4769 1710 Acked 2709 Write 3 Done New Done false
4770 1711 Acked 2710 Read 1 Done New Done false
4771 1712 Acked 2711 Write 7 Done New Done false
4772 1713 Acked 2712 Read 7 Done New Done false
4773 1714 Acked 2713 Read 4 Done New Done false
4774 1715 Acked 2714 Write 4 Done New Done false
4775 1716 Acked 2715 Write 10 Done New Done false
4776 1717 Acked 2716 Write 2 Done New Done false
4777 1718 Acked 2717 Write 3 Done New Done false
4778 1719 Acked 2718 Write 3 Done New Done false
4779 1720 Acked 2719 Write 7 Done New Done false
4780 1721 Acked 2720 Read 10 Done New Done false
4781 1722 Acked 2721 Write 7 Done New Done false
4782 1723 Acked 2722 Write 1 Done New Done false
4783 1724 Acked 2723 Read 6 Done New Done false
4784 1725 Acked 2724 Flush 0 Done New Done false
4785 1726 Acked 2725 Read 6 Done New Done false
4786 1727 Acked 2726 Read 2 Done New Done false
4787 1728 Acked 2727 Read 4 Done New Done false
4788 1729 Acked 2728 Read 7 Done New Done false
4789 1730 Acked 2729 Write 8 Done New Done false
4790 1731 Acked 2730 Write 1 Done New Done false
4791 1732 Acked 2731 Write 6 Done New Done false
4792 1733 Acked 2732 Read 9 Done New Done false
4793 1734 Acked 2733 Write 9 Done New Done false
4794 1735 Acked 2734 Read 9 Done New Done false
4795 1736 Acked 2735 Flush 0 Done New Done false
4796 1737 Acked 2736 Write 2 Done New Done false
4797 1738 Acked 2737 Flush 0 Done New Done false
4798 1739 Acked 2738 Write 8 Done New Done false
4799 1740 Acked 2739 Write 2 Done New Done false
4800 1741 Acked 2740 Read 6 Done New Done false
4801 1742 Acked 2741 Write 7 Done New Done false
4802 1743 Acked 2742 Write 5 Done New Done false
4803 1744 Acked 2743 Write 4 Done New Done false
4804 1745 Acked 2744 Flush 0 Done New Done false
4805 1746 Acked 2745 Read 8 Done New Done false
4806 1747 Acked 2746 Write 6 Done New Done false
4807 1748 Acked 2747 Read 7 Done New Done false
4808 1749 Acked 2748 Write 10 Done New Done false
4809 1750 Acked 2749 Read 8 Done New Done false
4810 1751 Acked 2750 Read 1 Done New Done false
4811 1752 Acked 2751 Read 7 Done New Done false
4812 1753 Acked 2752 Read 4 Done New Done false
4813 1754 Acked 2753 Write 2 Done New Done false
4814 1755 Acked 2754 Read 7 Done New Done false
4815 1756 Acked 2755 Read 5 Done New Done false
4816 1757 Acked 2756 Read 2 Done New Done false
4817 1758 Acked 2757 Read 5 Done New Done false
4818 1759 Acked 2758 Read 8 Done New Done false
4819 1760 Acked 2759 Read 4 Done New Done false
4820 1761 Acked 2760 Read 6 Done New Done false
4821 1762 Acked 2761 Write 5 Done New Done false
4822 1763 Acked 2762 Flush 0 Done New Done false
4823 1764 Acked 2763 Flush 0 Done New Done false
4824 1765 Acked 2764 Write 3 Done New Done false
4825 1766 Acked 2765 Read 4 Done New Done false
4826 1767 Acked 2766 Write 3 Done New Done false
4827 1768 Acked 2767 Write 3 Done New Done false
4828 1769 Acked 2768 Read 7 Done New Done false
4829 1770 Acked 2769 Read 6 Done New Done false
4830 1771 Acked 2770 Write 2 Done New Done false
4831 1772 Acked 2771 Write 9 Done New Done false
4832 1773 Acked 2772 Flush 0 Done New Done false
4833 1774 Acked 2773 Write 4 Done New Done false
4834 1775 Acked 2774 Read 4 Done New Done false
4835 1776 Acked 2775 Read 1 Done New Done false
4836 1777 Acked 2776 Read 1 Done New Done false
4837 1778 Acked 2777 Write 10 Done New Done false
4838 1779 Acked 2778 Read 7 Done New Done false
4839 1780 Acked 2779 Write 7 Done New Done false
4840 1781 Acked 2780 Read 9 Done New Done false
4841 1782 Acked 2781 Write 5 Done New Done false
4842 1783 Acked 2782 Read 2 Done New Done false
4843 1784 Acked 2783 Write 7 Done New Done false
4844 1785 Acked 2784 Read 10 Done New Done false
4845 1786 Acked 2785 Write 7 Done New Done false
4846 1787 Acked 2786 Flush 0 Done New Done false
4847 1788 Acked 2787 Read 2 Done New Done false
4848 1789 Acked 2788 Flush 0 Done New Done false
4849 1790 Acked 2789 Write 1 Done New Done false
4850 1791 Acked 2790 Write 10 Done New Done false
4851 1792 Acked 2791 Read 4 Done New Done false
4852 1793 Acked 2792 Read 3 Done New Done false
4853 1794 Acked 2793 Read 4 Done New Done false
4854 1795 Acked 2794 Write 4 Done New Done false
4855 1796 Acked 2795 Read 5 Done New Done false
4856 1797 Acked 2796 Write 2 Done New Done false
4857 1798 Acked 2797 Write 5 Done New Done false
4858 1799 Acked 2798 Write 10 Done New Done false
4859 1800 Acked 2799 Write 4 Done New Done false
4860 1801 Acked 2800 Read 1 Done New Done false
4861 1802 Acked 2801 Write 9 Done New Done false
4862 1803 Acked 2802 Write 5 Done New Done false
4863 1804 Acked 2803 Write 9 Done New Done false
4864 1805 Acked 2804 Write 3 Done New Done false
4865 1806 Acked 2805 Write 2 Done New Done false
4866 1807 Acked 2806 Write 10 Done New Done false
4867 1808 Acked 2807 Flush 0 Done New Done false
4868 1809 Acked 2808 Flush 0 Done New Done false
4869 1810 Acked 2809 Read 4 Done New Done false
4870 1811 Acked 2810 Write 4 Done New Done false
4871 1812 Acked 2811 Write 5 Done New Done false
4872 1813 Acked 2812 Read 2 Done New Done false
4873 1814 Acked 2813 Write 3 Sent New Sent false
4874 1815 Acked 2814 Write 2 Sent New Sent false
4875 STATES DS:0 DS:1 DS:2 TOTAL
4876 New 0 302 0 302
4877 Sent 2 0 2 4
4878 Done 300 0 300 600
4879 Skipped 0 0 0 0
4880 Error 0 0 0 0
4881 Last Flush: 2808 2512 2808
4882 Downstairs last five completed: 2512 2511 2510 2509 2508 2507
4883 Upstairs last five completed: 1815 1814 1813 1812 1811 1810
4884 CLIENT: Up:0 ds:302 act:2
4885 [8820] got start action
4886 [8820] I am going to restart
4887 Starting downstairs at port 8820
4888 Make output file at "/tmp/dsc/downstairs-8820.txt"
4889 [8820][1] reports Starting
4890 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1182)
4891 [8820][1] reports Running
4892 ----------------------------------------------------------------
4893 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
4894 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4895 1514 Acked 2513 Write 1 Done New Done false
4896 1515 Acked 2514 Write 2 Done New Done false
4897 1516 Acked 2515 Flush 0 Done New Done false
4898 1517 Acked 2516 Read 2 Done New Done false
4899 1518 Acked 2517 Flush 0 Done New Done false
4900 1519 Acked 2518 Write 5 Done New Done false
4901 1520 Acked 2519 Flush 0 Done New Done false
4902 1521 Acked 2520 Read 5 Done New Done false
4903 1522 Acked 2521 Flush 0 Done New Done false
4904 1523 Acked 2522 Read 4 Done New Done false
4905 1524 Acked 2523 Flush 0 Done New Done false
4906 1525 Acked 2524 Read 5 Done New Done false
4907 1526 Acked 2525 Read 4 Done New Done false
4908 1527 Acked 2526 Read 5 Done New Done false
4909 1528 Acked 2527 Write 3 Done New Done false
4910 1529 Acked 2528 Write 5 Done New Done false
4911 1530 Acked 2529 Flush 0 Done New Done false
4912 1531 Acked 2530 Write 9 Done New Done false
4913 1532 Acked 2531 Read 4 Done New Done false
4914 1533 Acked 2532 Write 4 Done New Done false
4915 1534 Acked 2533 Read 1 Done New Done false
4916 1535 Acked 2534 Write 2 Done New Done false
4917 1536 Acked 2535 Read 3 Done New Done false
4918 1537 Acked 2536 Read 5 Done New Done false
4919 1538 Acked 2537 Read 7 Done New Done false
4920 1539 Acked 2538 Write 6 Done New Done false
4921 1540 Acked 2539 Write 6 Done New Done false
4922 1541 Acked 2540 Write 6 Done New Done false
4923 1542 Acked 2541 Write 10 Done New Done false
4924 1543 Acked 2542 Write 2 Done New Done false
4925 1544 Acked 2543 Read 3 Done New Done false
4926 1545 Acked 2544 Write 7 Done New Done false
4927 1546 Acked 2545 Read 6 Done New Done false
4928 1547 Acked 2546 Write 3 Done New Done false
4929 1548 Acked 2547 Read 4 Done New Done false
4930 1549 Acked 2548 Write 2 Done New Done false
4931 1550 Acked 2549 Flush 0 Done New Done false
4932 1551 Acked 2550 Read 9 Done New Done false
4933 1552 Acked 2551 Read 4 Done New Done false
4934 1553 Acked 2552 Write 6 Done New Done false
4935 1554 Acked 2553 Read 4 Done New Done false
4936 1555 Acked 2554 Write 5 Done New Done false
4937 1556 Acked 2555 Write 9 Done New Done false
4938 1557 Acked 2556 Read 1 Done New Done false
4939 1558 Acked 2557 Read 9 Done New Done false
4940 1559 Acked 2558 Read 8 Done New Done false
4941 1560 Acked 2559 Write 1 Done New Done false
4942 1561 Acked 2560 Read 1 Done New Done false
4943 1562 Acked 2561 Write 2 Done New Done false
4944 1563 Acked 2562 Write 5 Done New Done false
4945 1564 Acked 2563 Write 8 Done New Done false
4946 1565 Acked 2564 Read 3 Done New Done false
4947 1566 Acked 2565 Read 5 Done New Done false
4948 1567 Acked 2566 Write 5 Done New Done false
4949 1568 Acked 2567 Write 9 Done New Done false
4950 1569 Acked 2568 Write 2 Done New Done false
4951 1570 Acked 2569 Flush 0 Done New Done false
4952 1571 Acked 2570 Read 6 Done New Done false
4953 1572 Acked 2571 Read 7 Done New Done false
4954 1573 Acked 2572 Read 8 Done New Done false
4955 1574 Acked 2573 Read 3 Done New Done false
4956 1575 Acked 2574 Write 3 Done New Done false
4957 1576 Acked 2575 Read 5 Done New Done false
4958 1577 Acked 2576 Read 3 Done New Done false
4959 1578 Acked 2577 Flush 0 Done New Done false
4960 1579 Acked 2578 Read 10 Done New Done false
4961 1580 Acked 2579 Write 3 Done New Done false
4962 1581 Acked 2580 Write 3 Done New Done false
4963 1582 Acked 2581 Write 1 Done New Done false
4964 1583 Acked 2582 Write 6 Done New Done false
4965 1584 Acked 2583 Read 5 Done New Done false
4966 1585 Acked 2584 Write 2 Done New Done false
4967 1586 Acked 2585 Write 3 Done New Done false
4968 1587 Acked 2586 Read 9 Done New Done false
4969 1588 Acked 2587 Write 7 Done New Done false
4970 1589 Acked 2588 Read 1 Done New Done false
4971 1590 Acked 2589 Flush 0 Done New Done false
4972 1591 Acked 2590 Write 6 Done New Done false
4973 1592 Acked 2591 Read 7 Done New Done false
4974 1593 Acked 2592 Read 5 Done New Done false
4975 1594 Acked 2593 Read 9 Done New Done false
4976 1595 Acked 2594 Write 4 Done New Done false
4977 1596 Acked 2595 Read 6 Done New Done false
4978 1597 Acked 2596 Write 1 Done New Done false
4979 1598 Acked 2597 Read 7 Done New Done false
4980 1599 Acked 2598 Read 7 Done New Done false
4981 1600 Acked 2599 Read 2 Done New Done false
4982 1601 Acked 2600 Write 6 Done New Done false
4983 1602 Acked 2601 Read 9 Done New Done false
4984 1603 Acked 2602 Write 4 Done New Done false
4985 1604 Acked 2603 Write 4 Done New Done false
4986 1605 Acked 2604 Read 6 Done New Done false
4987 1606 Acked 2605 Flush 0 Done New Done false
4988 1607 Acked 2606 Read 5 Done New Done false
4989 1608 Acked 2607 Write 2 Done New Done false
4990 1609 Acked 2608 Write 5 Done New Done false
4991 1610 Acked 2609 Write 7 Done New Done false
4992 1611 Acked 2610 Write 9 Done New Done false
4993 1612 Acked 2611 Write 7 Done New Done false
4994 1613 Acked 2612 Read 7 Done New Done false
4995 1614 Acked 2613 Write 9 Done New Done false
4996 1615 Acked 2614 Read 1 Done New Done false
4997 1616 Acked 2615 Read 6 Done New Done false
4998 1617 Acked 2616 Write 1 Done New Done false
4999 1618 Acked 2617 Write 2 Done New Done false
5000 1619 Acked 2618 Read 6 Done New Done false
5001 1620 Acked 2619 Flush 0 Done New Done false
5002 1621 Acked 2620 Write 4 Done New Done false
5003 1622 Acked 2621 Read 9 Done New Done false
5004 1623 Acked 2622 Write 1 Done New Done false
5005 1624 Acked 2623 Write 7 Done New Done false
5006 1625 Acked 2624 Write 8 Done New Done false
5007 1626 Acked 2625 Read 6 Done New Done false
5008 1627 Acked 2626 Write 4 Done New Done false
5009 1628 Acked 2627 Flush 0 Done New Done false
5010 1629 Acked 2628 Read 8 Done New Done false
5011 1630 Acked 2629 Read 5 Done New Done false
5012 1631 Acked 2630 Read 8 Done New Done false
5013 1632 Acked 2631 Read 5 Done New Done false
5014 1633 Acked 2632 Write 3 Done New Done false
5015 1634 Acked 2633 Read 3 Done New Done false
5016 1635 Acked 2634 Write 6 Done New Done false
5017 1636 Acked 2635 Write 2 Done New Done false
5018 1637 Acked 2636 Write 8 Done New Done false
5019 1638 Acked 2637 Read 10 Done New Done false
5020 1639 Acked 2638 Read 3 Done New Done false
5021 1640 Acked 2639 Write 5 Done New Done false
5022 1641 Acked 2640 Flush 0 Done New Done false
5023 1642 Acked 2641 Read 8 Done New Done false
5024 1643 Acked 2642 Read 1 Done New Done false
5025 1644 Acked 2643 Read 8 Done New Done false
5026 1645 Acked 2644 Read 3 Done New Done false
5027 1646 Acked 2645 Write 3 Done New Done false
5028 1647 Acked 2646 Flush 0 Done New Done false
5029 1648 Acked 2647 Write 9 Done New Done false
5030 1649 Acked 2648 Read 3 Done New Done false
5031 1650 Acked 2649 Write 5 Done New Done false
5032 1651 Acked 2650 Read 2 Done New Done false
5033 1652 Acked 2651 Write 10 Done New Done false
5034 1653 Acked 2652 Read 2 Done New Done false
5035 1654 Acked 2653 Write 8 Done New Done false
5036 1655 Acked 2654 Write 3 Done New Done false
5037 1656 Acked 2655 Flush 0 Done New Done false
5038 1657 Acked 2656 Read 8 Done New Done false
5039 1658 Acked 2657 Read 8 Done New Done false
5040 1659 Acked 2658 Read 2 Done New Done false
5041 1660 Acked 2659 Write 5 Done New Done false
5042 1661 Acked 2660 Read 1 Done New Done false
5043 1662 Acked 2661 Write 5 Done New Done false
5044 1663 Acked 2662 Flush 0 Done New Done false
5045 1664 Acked 2663 Read 8 Done New Done false
5046 1665 Acked 2664 Write 2 Done New Done false
5047 1666 Acked 2665 Read 8 Done New Done false
5048 1667 Acked 2666 Write 2 Done New Done false
5049 1668 Acked 2667 Read 1 Done New Done false
5050 1669 Acked 2668 Read 5 Done New Done false
5051 1670 Acked 2669 Flush 0 Done New Done false
5052 1671 Acked 2670 Flush 0 Done New Done false
5053 1672 Acked 2671 Read 9 Done New Done false
5054 1673 Acked 2672 Read 9 Done New Done false
5055 1674 Acked 2673 Read 6 Done New Done false
5056 1675 Acked 2674 Read 3 Done New Done false
5057 1676 Acked 2675 Write 1 Done New Done false
5058 1677 Acked 2676 Flush 0 Done New Done false
5059 1678 Acked 2677 Write 9 Done New Done false
5060 1679 Acked 2678 Read 6 Done New Done false
5061 1680 Acked 2679 Read 4 Done New Done false
5062 1681 Acked 2680 Read 8 Done New Done false
5063 1682 Acked 2681 Write 8 Done New Done false
5064 1683 Acked 2682 Write 9 Done New Done false
5065 1684 Acked 2683 Write 5 Done New Done false
5066 1685 Acked 2684 Read 2 Done New Done false
5067 1686 Acked 2685 Flush 0 Done New Done false
5068 1687 Acked 2686 Write 3 Done New Done false
5069 1688 Acked 2687 Read 4 Done New Done false
5070 1689 Acked 2688 Flush 0 Done New Done false
5071 1690 Acked 2689 Flush 0 Done New Done false
5072 1691 Acked 2690 Read 7 Done New Done false
5073 1692 Acked 2691 Write 3 Done New Done false
5074 1693 Acked 2692 Write 9 Done New Done false
5075 1694 Acked 2693 Read 9 Done New Done false
5076 1695 Acked 2694 Read 2 Done New Done false
5077 1696 Acked 2695 Read 7 Done New Done false
5078 1697 Acked 2696 Write 10 Done New Done false
5079 1698 Acked 2697 Write 9 Done New Done false
5080 1699 Acked 2698 Read 6 Done New Done false
5081 1700 Acked 2699 Read 4 Done New Done false
5082 1701 Acked 2700 Read 6 Done New Done false
5083 1702 Acked 2701 Read 3 Done New Done false
5084 1703 Acked 2702 Flush 0 Done New Done false
5085 1704 Acked 2703 Read 2 Done New Done false
5086 1705 Acked 2704 Read 1 Done New Done false
5087 1706 Acked 2705 Read 2 Done New Done false
5088 1707 Acked 2706 Write 3 Done New Done false
5089 1708 Acked 2707 Write 1 Done New Done false
5090 1709 Acked 2708 Read 2 Done New Done false
5091 1710 Acked 2709 Write 3 Done New Done false
5092 1711 Acked 2710 Read 1 Done New Done false
5093 1712 Acked 2711 Write 7 Done New Done false
5094 1713 Acked 2712 Read 7 Done New Done false
5095 1714 Acked 2713 Read 4 Done New Done false
5096 1715 Acked 2714 Write 4 Done New Done false
5097 1716 Acked 2715 Write 10 Done New Done false
5098 1717 Acked 2716 Write 2 Done New Done false
5099 1718 Acked 2717 Write 3 Done New Done false
5100 1719 Acked 2718 Write 3 Done New Done false
5101 1720 Acked 2719 Write 7 Done New Done false
5102 1721 Acked 2720 Read 10 Done New Done false
5103 1722 Acked 2721 Write 7 Done New Done false
5104 1723 Acked 2722 Write 1 Done New Done false
5105 1724 Acked 2723 Read 6 Done New Done false
5106 1725 Acked 2724 Flush 0 Done New Done false
5107 1726 Acked 2725 Read 6 Done New Done false
5108 1727 Acked 2726 Read 2 Done New Done false
5109 1728 Acked 2727 Read 4 Done New Done false
5110 1729 Acked 2728 Read 7 Done New Done false
5111 1730 Acked 2729 Write 8 Done New Done false
5112 1731 Acked 2730 Write 1 Done New Done false
5113 1732 Acked 2731 Write 6 Done New Done false
5114 1733 Acked 2732 Read 9 Done New Done false
5115 1734 Acked 2733 Write 9 Done New Done false
5116 1735 Acked 2734 Read 9 Done New Done false
5117 1736 Acked 2735 Flush 0 Done New Done false
5118 1737 Acked 2736 Write 2 Done New Done false
5119 1738 Acked 2737 Flush 0 Done New Done false
5120 1739 Acked 2738 Write 8 Done New Done false
5121 1740 Acked 2739 Write 2 Done New Done false
5122 1741 Acked 2740 Read 6 Done New Done false
5123 1742 Acked 2741 Write 7 Done New Done false
5124 1743 Acked 2742 Write 5 Done New Done false
5125 1744 Acked 2743 Write 4 Done New Done false
5126 1745 Acked 2744 Flush 0 Done New Done false
5127 1746 Acked 2745 Read 8 Done New Done false
5128 1747 Acked 2746 Write 6 Done New Done false
5129 1748 Acked 2747 Read 7 Done New Done false
5130 1749 Acked 2748 Write 10 Done New Done false
5131 1750 Acked 2749 Read 8 Done New Done false
5132 1751 Acked 2750 Read 1 Done New Done false
5133 1752 Acked 2751 Read 7 Done New Done false
5134 1753 Acked 2752 Read 4 Done New Done false
5135 1754 Acked 2753 Write 2 Done New Done false
5136 1755 Acked 2754 Read 7 Done New Done false
5137 1756 Acked 2755 Read 5 Done New Done false
5138 1757 Acked 2756 Read 2 Done New Done false
5139 1758 Acked 2757 Read 5 Done New Done false
5140 1759 Acked 2758 Read 8 Done New Done false
5141 1760 Acked 2759 Read 4 Done New Done false
5142 1761 Acked 2760 Read 6 Done New Done false
5143 1762 Acked 2761 Write 5 Done New Done false
5144 1763 Acked 2762 Flush 0 Done New Done false
5145 1764 Acked 2763 Flush 0 Done New Done false
5146 1765 Acked 2764 Write 3 Done New Done false
5147 1766 Acked 2765 Read 4 Done New Done false
5148 1767 Acked 2766 Write 3 Done New Done false
5149 1768 Acked 2767 Write 3 Done New Done false
5150 1769 Acked 2768 Read 7 Done New Done false
5151 1770 Acked 2769 Read 6 Done New Done false
5152 1771 Acked 2770 Write 2 Done New Done false
5153 1772 Acked 2771 Write 9 Done New Done false
5154 1773 Acked 2772 Flush 0 Done New Done false
5155 1774 Acked 2773 Write 4 Done New Done false
5156 1775 Acked 2774 Read 4 Done New Done false
5157 1776 Acked 2775 Read 1 Done New Done false
5158 1777 Acked 2776 Read 1 Done New Done false
5159 1778 Acked 2777 Write 10 Done New Done false
5160 1779 Acked 2778 Read 7 Done New Done false
5161 1780 Acked 2779 Write 7 Done New Done false
5162 1781 Acked 2780 Read 9 Done New Done false
5163 1782 Acked 2781 Write 5 Done New Done false
5164 1783 Acked 2782 Read 2 Done New Done false
5165 1784 Acked 2783 Write 7 Done New Done false
5166 1785 Acked 2784 Read 10 Done New Done false
5167 1786 Acked 2785 Write 7 Done New Done false
5168 1787 Acked 2786 Flush 0 Done New Done false
5169 1788 Acked 2787 Read 2 Done New Done false
5170 1789 Acked 2788 Flush 0 Done New Done false
5171 1790 Acked 2789 Write 1 Done New Done false
5172 1791 Acked 2790 Write 10 Done New Done false
5173 1792 Acked 2791 Read 4 Done New Done false
5174 1793 Acked 2792 Read 3 Done New Done false
5175 1794 Acked 2793 Read 4 Done New Done false
5176 1795 Acked 2794 Write 4 Done New Done false
5177 1796 Acked 2795 Read 5 Done New Done false
5178 1797 Acked 2796 Write 2 Done New Done false
5179 1798 Acked 2797 Write 5 Done New Done false
5180 1799 Acked 2798 Write 10 Done New Done false
5181 1800 Acked 2799 Write 4 Done New Done false
5182 1801 Acked 2800 Read 1 Done New Done false
5183 1802 Acked 2801 Write 9 Done New Done false
5184 1803 Acked 2802 Write 5 Done New Done false
5185 1804 Acked 2803 Write 9 Done New Done false
5186 1805 Acked 2804 Write 3 Done New Done false
5187 1806 Acked 2805 Write 2 Done New Done false
5188 1807 Acked 2806 Write 10 Done New Done false
5189 1808 Acked 2807 Flush 0 Done New Done false
5190 1809 Acked 2808 Flush 0 Done New Done false
5191 1810 Acked 2809 Read 4 Done New Done false
5192 1811 Acked 2810 Write 4 Done New Done false
5193 1812 Acked 2811 Write 5 Done New Done false
5194 1813 Acked 2812 Read 2 Done New Done false
5195 1814 Acked 2813 Write 3 Done New Done false
5196 1815 Acked 2814 Write 2 Done New Done false
5197 1816 Acked 2815 Flush 0 Done New Done false
5198 STATES DS:0 DS:1 DS:2 TOTAL
5199 New 0 303 0 303
5200 Sent 0 0 0 0
5201 Done 303 0 303 606
5202 Skipped 0 0 0 0
5203 Error 0 0 0 0
5204 Last Flush: 2815 2512 2815
5205 Downstairs last five completed: 2512 2511 2510 2509 2508 2507
5206 Upstairs last five completed: 1816 1815 1814 1813 1812 1811
5207 CLIENT: Up:0 ds:303 act:2
52082023-10-13T18:10:08.978ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 1
52092023-10-13T18:10:08.978ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
52102023-10-13T18:10:08.978ZINFOcrucible: [1] client re-new 303 jobs since flush 2512 = downstairs
52112023-10-13T18:10:08.979ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
52122023-10-13T18:10:08.979ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
52132023-10-13T18:10:08.979ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
52142023-10-13T18:10:08.980ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
52152023-10-13T18:10:08.980ZINFOcrucible: [1] send last flush ID to this DS: 2512
52162023-10-13T18:10:08.980ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
52172023-10-13T18:10:08.980ZINFOcrucible: [1] Replied this last flush ID: 2512
52182023-10-13T18:10:08.980ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Offline Active ds_transition to Replay
52192023-10-13T18:10:08.980ZINFOcrucible: [1] Transition from Offline to Replay
52202023-10-13T18:10:08.980ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
52212023-10-13T18:10:08.980ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Replay Active ds_transition to Active
52222023-10-13T18:10:08.980ZINFOcrucible: [1] Transition from Replay to Active
52232023-10-13T18:10:08.980ZINFOcrucible: [1] Starts cmd_loop
52242023-10-13T18:10:08.980ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
52252023-10-13T18:10:08.980ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
52262023-10-13T18:10:08.980ZINFOcrucible: Set check for repair
52272023-10-13T18:10:09.981ZINFOcrucible: Checking if live repair is needed
52282023-10-13T18:10:09.981ZINFOcrucible: No Live Repair required at this time
52292023-10-13T18:10:09.981ZWARNcrucible: [1] flow control end
5230 ----------------------------------------------------------------
5231 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5232 Downstairs last five completed: 2815 2814 2813 2812 2811 2810
5233 Upstairs last five completed: 1816 1815 1814 1813 1812 1811
5234 CLIENT: Up:0 ds:0 act:3
5235 Replay: All jobs finished, all DS active.
5236 Main task has work to do, go find it
5237 got dsc Stop(1)
5238 stop 1
5239 Oct 13 18:10:12.904 INFO request completed, latency_us: 529, response_code: 204, uri: /stop/cid/1, method: POST, req_id: e1a0322a-5540-4ff4-8116-e6dd7554d794, remote_addr: 127.[8820] Got stop action so:false kr:false
5240 0.0.1:64571, local_addr: 127.0.0.1:9998
5241 Oct 13 18:10:12.905 INFO request completed, latency_us: 181, response_code: 200, uri: /state/cid/1, method: GET, req_id: 428a180f-1b40-4719-baee-f128b4c8b174, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
52422023-10-13T18:10:12.905ZWARNcrucible: [1] downstairs disconnected
52432023-10-13T18:10:12.905ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
52442023-10-13T18:10:12.905ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8820 closed looper = 1
52452023-10-13T18:10:12.906ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
52462023-10-13T18:10:12.906ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
52472023-10-13T18:10:12.906ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
52482023-10-13T18:10:12.906ZWARNcrucible: [1] pm_task rx.recv() is None
5249 [8820] Exited with: ExitStatus(unix_wait_status(9))
5250 [8820][1] reports Exit
52512023-10-13T18:10:13.908ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
5252 Oct 13 18:10:16.908 INFO request completed, latency_us: 270, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6360d395-8d7f-4006-8be1-c62595843097, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
5253 001/300 Read block 4058 len 4096
5254 002/300 Write block 1632 len 12288 data: 3 3 4
5255 003/300 Write block 4152 len 28672 data: 2 2 2 2 2 2 2
5256 004/300 Read block 2406 len 28672
5257 005/300 Write block 2634 len 24576 data: 2 2 2 2 2 3
5258 006/300 Read block 3132 len 20480
5259 007/300 Read block 4869 len 12288
5260 008/300 Read block 2198 len 16384
5261 009/300 Read block 2100 len 8192
5262 010/300 Write block 1704 len 8192 data: 2 2
5263 011/300 Read block 38 len 4096
5264 012/300 Read block 2415 len 36864
5265 013/300 Read block 1748 len 36864
5266 014/300 Read block 844 len 12288
5267 015/300 Flush
5268 016/300 Read block 3522 len 36864
5269 017/300 Read block 345 len 36864
5270 018/300 Write block 1171 len 28672 data: 3 3 3 2 2 2 2
5271 019/300 Read block 3197 len 16384
5272 020/300 Read block 3327 len 12288
5273 021/300 Write block 922 len 8192 data: 3 3
5274 022/300 Write block 3468 len 16384 data: 2 2 2 2
5275 023/300 Write block 2677 len 28672 data: 3 3 3 2 2 3 3
5276 024/300 Write block 3259 len 32768 data: 2 3 3 3 2 2 2 2
5277 025/300 Read block 2608 len 40960
5278 026/300 Read block 457 len 28672
5279 027/300 Write block 4098 len 8192 data: 2 3
5280 028/300 Write block 1019 len 20480 data: 2 3 3 2 2
5281 029/300 Read block 615 len 20480
5282 030/300 Read block 2072 len 24576
5283 031/300 Write block 2696 len 32768 data: 3 3 3 3 3 3 2 2
5284 032/300 Read block 1837 len 24576
5285 033/300 Read block 1954 len 8192
5286 034/300 Read block 2285 len 40960
5287 035/300 Read block 1711 len 40960
5288 036/300 Write block 3730 len 40960 data: 3 3 3 3 3 3 3 3 3 3
5289 037/300 Write block 1834 len 8192 data: 2 2
5290 038/300 Read block 2966 len 40960
5291 039/300 Write block 3980 len 32768 data: 2 2 2 2 2 2 3 3
5292 040/300 Read block 3158 len 8192
5293 041/300 Write block 3804 len 20480 data: 4 5 4 4 4
5294 042/300 Flush
5295 043/300 Read block 1688 len 28672
5296 044/300 Read block 1540 len 20480
5297 045/300 Read block 2421 len 8192
5298 046/300 Read block 17 len 36864
5299 047/300 Write block 1412 len 28672 data: 3 3 3 2 3 3 3
5300 048/300 Read block 4890 len 8192
5301 049/300 Read block 37 len 36864
5302 050/300 Write block 2555 len 12288 data: 4 6 5
5303 051/300 Read block 1616 len 4096
5304 052/300 Read block 3551 len 8192
5305 053/300 Read block 248 len 12288
5306 054/300 Write block 3472 len 36864 data: 2 2 2 2 3 3 3 2 2
5307 055/300 Flush
5308 056/300 Read block 766 len 36864
5309 057/300 Write block 3487 len 32768 data: 2 2 3 3 3 2 2 2
5310 058/300 Write block 4139 len 4096 data: 2
5311 059/300 Read block 2003 len 8192
5312 060/300 Read block 1125 len 40960
5313 061/300 Flush
5314 062/300 Flush
5315 063/300 Read block 3910 len 40960
5316 064/300 Flush
5317 065/300 Flush
5318 066/300 Read block 4193 len 36864
5319 067/300 Flush
5320 068/300 Read block 1099 len 12288
5321 069/300 Read block 2108 len 36864
5322 070/300 Write block 2663 len 4096 data: 2
5323 071/300 Read block 3792 len 12288
5324 072/300 Read block 2060 len 32768
5325 073/300 Read block 4497 len 40960
5326 074/300 Write block 4332 len 24576 data: 4 5 5 5 3 3
5327 075/300 Flush
5328 076/300 Read block 658 len 16384
5329 077/300 Write block 1146 len 16384 data: 2 3 4 4
5330 078/300 Write block 1780 len 24576 data: 2 2 2 3 3 3
5331 079/300 Read block 1158 len 20480
5332 080/300 Read block 805 len 36864
5333 081/300 Write block 4149 len 24576 data: 2 2 2 3 3 3
5334 082/300 Write block 4164 len 36864 data: 4 4 4 4 4 4 3 2 2
5335 083/300 Write block 3639 len 16384 data: 2 2 2 2
5336 084/300 Read block 4903 len 12288
5337 085/300 Read block 365 len 8192
5338 086/300 Write block 3637 len 16384 data: 2 2 3 3
5339 087/300 Write block 3660 len 12288 data: 2 2 2
5340 088/300 Read block 3877 len 40960
5341 089/300 Read block 1347 len 36864
5342 090/300 Flush
5343 091/300 Write block 21 len 8192 data: 3 3
5344 092/300 Write block 4442 len 32768 data: 2 2 2 2 2 2 2 2
5345 093/300 Read block 1926 len 40960
5346 094/300 Read block 3871 len 36864
5347 095/300 Flush
5348 096/300 Read block 4720 len 16384
5349 097/300 Flush
5350 098/300 Write block 2104 len 8192 data: 3 3
5351 099/300 Read block 4334 len 8192
5352 100/300 Write block 1277 len 4096 data: 2
5353 101/300 Read block 2483 len 36864
5354 102/300 Write block 3425 len 36864 data: 2 2 2 2 2 2 2 2 2
5355 103/300 Write block 726 len 8192 data: 2 2
5356 104/300 Write block 2785 len 8192 data: 2 2
5357 105/300 Read block 4694 len 24576
5358 106/300 Read block 3928 len 32768
5359 107/300 Write block 3620 len 4096 data: 3
5360 108/300 Write block 4720 len 12288 data: 2 3 3
5361 109/300 Read block 2997 len 36864
5362 110/300 Read block 1318 len 40960
5363 111/300 Read block 990 len 16384
5364 112/300 Flush
5365 113/300 Read block 4039 len 32768
5366 114/300 Write block 3950 len 40960 data: 3 3 3 3 4 4 3 3 3 3
5367 115/300 Read block 393 len 20480
5368 116/300 Read block 381 len 12288
5369 117/300 Flush
5370 118/300 Read block 3189 len 28672
5371 119/300 Write block 4661 len 40960 data: 2 2 2 2 2 2 3 3 3 3
5372 120/300 Read block 1328 len 12288
5373 121/300 Read block 1698 len 12288
5374 122/300 Read block 1899 len 36864
5375 123/300 Flush
5376 124/300 Read block 3517 len 16384
5377 125/300 Read block 752 len 4096
5378 126/300 Write block 3862 len 28672 data: 3 3 3 3 3 3 3
5379 127/300 Write block 3704 len 16384 data: 3 3 4 4
5380 128/300 Read block 4307 len 16384
5381 129/300 Read block 1039 len 20480
5382 130/300 Write block 3693 len 24576 data: 3 3 3 3 3 3
5383 131/300 Read block 83 len 8192
5384 132/300 Write block 1041 len 20480 data: 3 3 3 2 2
5385 133/300 Write block 2659 len 16384 data: 2 2 2 2
5386 134/300 Read block 3798 len 16384
5387 135/300 Read block 4336 len 36864
5388 136/300 Write block 1055 len 20480 data: 3 3 3 4 4
5389 137/300 Write block 4701 len 24576 data: 4 5 3 3 2 2
5390 138/300 Read block 139 len 20480
5391 139/300 Read block 3619 len 36864
5392 140/300 Read block 1328 len 16384
5393 141/300 Read block 2533 len 32768
5394 142/300 Read block 3505 len 8192
5395 143/300 Write block 3902 len 36864 data: 2 2 2 2 3 3 4 3 3
5396 144/300 Read block 322 len 12288
5397 145/300 Read block 4651 len 16384
5398 146/300 Write block 1091 len 40960 data: 3 2 2 2 3 5 6 6 6 6
5399 147/300 Write block 4589 len 8192 data: 2 2
5400 148/300 Write block 4320 len 4096 data: 2
5401 149/300 Write block 3747 len 32768 data: 2 2 2 2 2 3 4 4
5402 150/300 Read block 1742 len 12288
5403 151/300 Flush
5404 152/300 Read block 1602 len 16384
5405 153/300 Flush
5406 154/300 Read block 3421 len 16384
5407 155/300 Read block 2735 len 32768
5408 156/300 Read block 430 len 32768
5409 157/300 Read block 1241 len 36864
5410 158/300 Read block 4371 len 28672
5411 159/300 Write block 4662 len 28672 data: 3 3 3 3 3 4 4
5412 160/300 Write block 931 len 40960 data: 5 3 3 3 4 4 4 3 3 2
5413 161/300 Read block 3623 len 36864
5414 162/300 Read block 1631 len 28672
5415 163/300 Read block 3110 len 32768
5416 164/300 Read block 751 len 32768
5417 165/300 Write block 3193 len 12288 data: 3 2 2
5418 166/300 Read block 4780 len 36864
5419 167/300 Read block 1179 len 32768
5420 168/300 Read block 1806 len 20480
5421 169/300 Write block 3234 len 32768 data: 3 3 3 3 3 3 3 3
5422 170/300 Write block 3688 len 24576 data: 2 3 3 3 4 4
5423 171/300 Write block 3963 len 16384 data: 2 2 3 2
5424 172/300 Flush
5425 173/300 Write block 3775 len 40960 data: 3 2 2 2 2 2 2 2 2 2
5426 174/300 Read block 864 len 8192
5427 175/300 Write block 481 len 8192 data: 2 2
5428 176/300 Write block 1826 len 32768 data: 3 2 2 2 2 2 2 2
5429 177/300 Write block 4914 len 40960 data: 2 2 2 2 2 2 2 2 3 3
5430 178/300 Read block 2530 len 16384
5431 179/300 Write block 4754 len 40960 data: 3 5 5 5 3 3 3 3 3 2
5432 180/300 Flush
5433 181/300 Write block 1557 len 4096 data: 2
5434 182/300 Read block 117 len 24576
5435 183/300 Write block 161 len 40960 data: 2 2 2 2 2 2 2 2 2 3
5436 184/300 Write block 2205 len 32768 data: 3 3 3 3 4 4 5 4
5437 185/300 Read block 4026 len 16384
5438 186/300 Write block 3398 len 20480 data: 2 2 2 2 2
5439 187/300 Read block 2307 len 16384
5440 188/300 Read block 2812 len 4096
5441 189/300 Read block 782 len 16384
5442 190/300 Read block 3217 len 36864
5443 191/300 Read block 4661 len 4096
5444 192/300 Flush
5445 193/300 Read block 2159 len 20480
5446 194/300 Read block 3162 len 4096
5447 195/300 Read block 1673 len 4096
5448 196/300 Read block 4521 len 8192
5449 197/300 Flush
5450 198/300 Read block 3209 len 28672
5451 199/300 Write block 4413 len 8192 data: 3 3
5452 200/300 Write block 2949 len 40960 data: 3 3 3 4 4 3 3 3 3 3
5453 201/300 Read block 98 len 20480
5454 202/300 Write block 1044 len 36864 data: 3 3 2 3 3 3 4 4 3
5455 203/300 Read block 4096 len 32768
5456 204/300 Read block 2448 len 32768
5457 205/300 Read block 1361 len 16384
5458 206/300 Write block 1737 len 36864 data: 3 3 3 3 3 3 3 3 2
5459 207/300 Write block 43 len 24576 data: 2 2 2 2 2 2
5460 208/300 Read block 763 len 16384
5461 209/300 Write block 3533 len 28672 data: 3 3 3 3 3 3 3
5462 210/300 Write block 607 len 16384 data: 3 3 2 3
5463 211/300 Flush
5464 212/300 Read block 3817 len 36864
5465 213/300 Flush
5466 214/300 Read block 4372 len 8192
5467 215/300 Flush
5468 216/300 Read block 195 len 8192
5469 217/300 Read block 4859 len 40960
5470 218/300 Flush
5471 219/300 Read block 3416 len 24576
5472 220/300 Write block 550 len 32768 data: 2 3 2 3 3 3 4 6
5473 221/300 Write block 835 len 32768 data: 2 2 2 2 2 3 3 3
5474 222/300 Read block 3283 len 12288
5475 223/300 Read block 2270 len 32768
5476 224/300 Write block 4324 len 36864 data: 3 3 3 3 2 4 4 4 5
5477 225/300 Read block 3626 len 4096
5478 226/300 Read block 1502 len 4096
5479 227/300 Read block 2460 len 28672
5480 228/300 Write block 2321 len 16384 data: 2 2 2 2
5481 229/300 Read block 3042 len 24576
5482 230/300 Write block 46 len 20480 data: 3 3 3 2 2
5483 231/300 Flush
5484 232/300 Flush
5485 233/300 Read block 1891 len 24576
5486 234/300 Read block 2334 len 24576
5487 235/300 Write block 743 len 32768 data: 2 2 3 2 2 2 2 2
5488 236/300 Write block 4845 len 12288 data: 4 3 3
5489 237/300 Read block 784 len 4096
5490 238/300 Read block 2254 len 20480
5491 239/300 Read block 813 len 20480
5492 240/300 Write block 4592 len 24576 data: 2 2 2 2 2 2
5493 241/300 Flush
5494 242/300 Write block 1767 len 16384 data: 2 2 2 2
5495 243/300 Write block 1008 len 8192 data: 3 3
5496 244/300 Write block 3536 len 20480 data: 4 4 4 4 3
5497 245/300 Read block 3011 len 16384
5498 246/300 Write block 2247 len 8192 data: 2 2
5499 247/300 Write block 1500 len 8192 data: 4 3
5500 248/300 Read block 2544 len 28672
5501 249/300 Flush
5502 250/300 Write block 3429 len 24576 data: 3 3 3 3 3 2
5503 251/300 Write block 2590 len 24576 data: 4 3 2 2 2 2
5504 252/300 Write block 3375 len 4096 data: 2
5505 253/300 Write block 3033 len 40960 data: 3 3 3 3 2 2 2 3 3 3
5506 254/300 Read block 3440 len 28672
5507 255/300 Write block 4311 len 36864 data: 4 4 3 3 3 3 2 2 2
5508 256/300 Write block 2215 len 28672 data: 3 4 3 4 4 4 4
5509 257/300 Write block 4057 len 32768 data: 2 2 2 2 2 2 2 2
5510 258/300 Read block 2694 len 28672
5511 259/300 Write block 4863 len 24576 data: 3 3 3 3 2 2
5512 260/300 Read block 4429 len 40960
5513 261/300 Write block 1994 len 12288 data: 5 6 6
5514 262/300 Write block 4929 len 32768 data: 3 3 3 3 3 3 3 3
5515 263/300 Write block 2958 len 36864 data: 4 3 4 4 3 3 3 3 3
5516 264/300 Read block 4755 len 16384
5517 265/300 Write block 3815 len 4096 data: 2
5518 266/300 Write block 1753 len 16384 data: 3 3 3 2
5519 267/300 Read block 1369 len 20480
5520 268/300 Read block 1048 len 32768
5521 269/300 Write block 2868 len 24576 data: 4 5 5 4 4 4
5522 270/300 Read block 3073 len 12288
5523 271/300 Read block 3175 len 12288
5524 272/300 Write block 2196 len 16384 data: 4 4 4 3
5525 273/300 Write block 4580 len 32768 data: 2 2 2 2 2 2 2 2
5526 274/300 Write block 1303 len 4096 data: 2
5527 275/300 Write block 4875 len 16384 data: 2 2 2 3
5528 276/300 Read block 142 len 16384
5529 277/300 Write block 4846 len 40960 data: 4 4 3 3 3 3 3 2 2 3
5530 278/300 Write block 2899 len 24576 data: 3 3 3 3 3 3
5531 279/300 Write block 3018 len 32768 data: 4 3 3 3 2 2 2 2
5532 280/300 Read block 2142 len 28672
5533 281/300 Write block 1838 len 4096 data: 2
5534 282/300 Read block 3010 len 4096
5535 283/300 Write block 755 len 28672 data: 3 3 3 3 4 4 3
5536 284/300 Write block 4371 len 40960 data: 2 3 4 4 3 3 3 3 4 3
5537 285/300 Write block 1387 len 36864 data: 4 3 3 3 3 2 2 2 2
5538 286/300 Read block 4232 len 40960
5539 287/300 Read block 646 len 20480
5540 288/300 Flush
5541 289/300 Write block 746 len 4096 data: 3
5542 290/300 Read block 4354 len 4096
5543 291/300 Flush
5544 292/300 Read block 2913 len 8192
5545 293/300 Flush
5546 294/300 Read block 817 len 32768
5547 295/300 Write block 2818 len 16384 data: 2 2 2 2
5548 296/300 Write block 3424 len 12288 data: 2 3 3
5549 297/300 Write block 3695 len 32768 data: 4 4 4 4 3 2 2 2
5550 298/300 Read block 789 len 24576
5551 299/300 Read block 2827 len 4096
5552 300/300 Read block 4219 len 12288
5553 Main task has work to do, go find it
5554 got dsc Start(1)
5555 start 1
5556 Oct 13 18:10:17.893 INFO request completed, latency_us: 296, response_code: 204, uri: /start/cid/1, method: POST, req_id: 3c5218fa-d110-4a4a-acc4-8cfee30d309c, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
5557 Replay: started 1, returned:Ok(())
5558 ----------------------------------------------------------------
5559 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
5560 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5561 1817 Acked 2816 Read 1 Done New Done false
5562 1818 Acked 2817 Write 3 Done New Done false
5563 1819 Acked 2818 Write 7 Done New Done false
5564 1820 Acked 2819 Read 7 Done New Done false
5565 1821 Acked 2820 Write 6 Done New Done false
5566 1822 Acked 2821 Read 5 Done New Done false
5567 1823 Acked 2822 Read 3 Done New Done false
5568 1824 Acked 2823 Read 4 Done New Done false
5569 1825 Acked 2824 Read 2 Done New Done false
5570 1826 Acked 2825 Write 2 Done New Done false
5571 1827 Acked 2826 Read 1 Done New Done false
5572 1828 Acked 2827 Read 9 Done New Done false
5573 1829 Acked 2828 Read 9 Done New Done false
5574 1830 Acked 2829 Read 3 Done New Done false
5575 1831 Acked 2830 Flush 0 Done New Done false
5576 1832 Acked 2831 Read 9 Done New Done false
5577 1833 Acked 2832 Read 9 Done New Done false
5578 1834 Acked 2833 Write 7 Done New Done false
5579 1835 Acked 2834 Read 4 Done New Done false
5580 1836 Acked 2835 Read 3 Done New Done false
5581 1837 Acked 2836 Write 2 Done New Done false
5582 1838 Acked 2837 Write 4 Done New Done false
5583 1839 Acked 2838 Write 7 Done New Done false
5584 1840 Acked 2839 Write 8 Done New Done false
5585 1841 Acked 2840 Read 10 Done New Done false
5586 1842 Acked 2841 Read 7 Done New Done false
5587 1843 Acked 2842 Write 2 Done New Done false
5588 1844 Acked 2843 Write 5 Done New Done false
5589 1845 Acked 2844 Read 5 Done New Done false
5590 1846 Acked 2845 Read 6 Done New Done false
5591 1847 Acked 2846 Write 8 Done New Done false
5592 1848 Acked 2847 Read 6 Done New Done false
5593 1849 Acked 2848 Read 2 Done New Done false
5594 1850 Acked 2849 Read 10 Done New Done false
5595 1851 Acked 2850 Read 10 Done New Done false
5596 1852 Acked 2851 Write 10 Done New Done false
5597 1853 Acked 2852 Write 2 Done New Done false
5598 1854 Acked 2853 Read 10 Done New Done false
5599 1855 Acked 2854 Write 8 Done New Done false
5600 1856 Acked 2855 Read 2 Done New Done false
5601 1857 Acked 2856 Write 5 Done New Done false
5602 1858 Acked 2857 Flush 0 Done New Done false
5603 1859 Acked 2858 Read 7 Done New Done false
5604 1860 Acked 2859 Read 5 Done New Done false
5605 1861 Acked 2860 Read 2 Done New Done false
5606 1862 Acked 2861 Read 9 Done New Done false
5607 1863 Acked 2862 Write 7 Done New Done false
5608 1864 Acked 2863 Read 2 Done New Done false
5609 1865 Acked 2864 Read 9 Done New Done false
5610 1866 Acked 2865 Write 3 Done New Done false
5611 1867 Acked 2866 Read 1 Done New Done false
5612 1868 Acked 2867 Read 2 Done New Done false
5613 1869 Acked 2868 Read 3 Done New Done false
5614 1870 Acked 2869 Write 9 Done New Done false
5615 1871 Acked 2870 Flush 0 Done New Done false
5616 1872 Acked 2871 Read 9 Done New Done false
5617 1873 Acked 2872 Write 8 Done New Done false
5618 1874 Acked 2873 Write 1 Done New Done false
5619 1875 Acked 2874 Read 2 Done New Done false
5620 1876 Acked 2875 Read 10 Done New Done false
5621 1877 Acked 2876 Flush 0 Done New Done false
5622 1878 Acked 2877 Flush 0 Done New Done false
5623 1879 Acked 2878 Read 10 Done New Done false
5624 1880 Acked 2879 Flush 0 Done New Done false
5625 1881 Acked 2880 Flush 0 Done New Done false
5626 1882 Acked 2881 Read 9 Done New Done false
5627 1883 Acked 2882 Flush 0 Done New Done false
5628 1884 Acked 2883 Read 3 Done New Done false
5629 1885 Acked 2884 Read 9 Done New Done false
5630 1886 Acked 2885 Write 1 Done New Done false
5631 1887 Acked 2886 Read 3 Done New Done false
5632 1888 Acked 2887 Read 8 Done New Done false
5633 1889 Acked 2888 Read 10 Done New Done false
5634 1890 Acked 2889 Write 6 Done New Done false
5635 1891 Acked 2890 Flush 0 Done New Done false
5636 1892 Acked 2891 Read 4 Done New Done false
5637 1893 Acked 2892 Write 4 Done New Done false
5638 1894 Acked 2893 Write 6 Done New Done false
5639 1895 Acked 2894 Read 5 Done New Done false
5640 1896 Acked 2895 Read 9 Done New Done false
5641 1897 Acked 2896 Write 6 Done New Done false
5642 1898 Acked 2897 Write 9 Done New Done false
5643 1899 Acked 2898 Write 4 Done New Done false
5644 1900 Acked 2899 Read 3 Done New Done false
5645 1901 Acked 2900 Read 2 Done New Done false
5646 1902 Acked 2901 Write 4 Done New Done false
5647 1903 Acked 2902 Write 3 Done New Done false
5648 1904 Acked 2903 Read 10 Done New Done false
5649 1905 Acked 2904 Flush 0 Done New Done false
5650 1906 Acked 2905 Read 9 Done New Done false
5651 1907 Acked 2906 Flush 0 Done New Done false
5652 1908 Acked 2907 Write 2 Done New Done false
5653 1909 Acked 2908 Write 8 Done New Done false
5654 1910 Acked 2909 Read 10 Done New Done false
5655 1911 Acked 2910 Read 9 Done New Done false
5656 1912 Acked 2911 Flush 0 Done New Done false
5657 1913 Acked 2912 Read 4 Done New Done false
5658 1914 Acked 2913 Flush 0 Done New Done false
5659 1915 Acked 2914 Write 2 Done New Done false
5660 1916 Acked 2915 Read 2 Done New Done false
5661 1917 Acked 2916 Write 1 Done New Done false
5662 1918 Acked 2917 Read 9 Done New Done false
5663 1919 Acked 2918 Write 9 Done New Done false
5664 1920 Acked 2919 Write 2 Done New Done false
5665 1921 Acked 2920 Write 2 Done New Done false
5666 1922 Acked 2921 Read 6 Done New Done false
5667 1923 Acked 2922 Read 8 Done New Done false
5668 1924 Acked 2923 Write 1 Done New Done false
5669 1925 Acked 2924 Write 3 Done New Done false
5670 1926 Acked 2925 Read 9 Done New Done false
5671 1927 Acked 2926 Read 10 Done New Done false
5672 1928 Acked 2927 Read 4 Done New Done false
5673 1929 Acked 2928 Flush 0 Done New Done false
5674 1930 Acked 2929 Read 8 Done New Done false
5675 1931 Acked 2930 Write 10 Done New Done false
5676 1932 Acked 2931 Read 5 Done New Done false
5677 1933 Acked 2932 Read 3 Done New Done false
5678 1934 Acked 2933 Flush 0 Done New Done false
5679 1935 Acked 2934 Read 7 Done New Done false
5680 1936 Acked 2935 Write 10 Done New Done false
5681 1937 Acked 2936 Read 3 Done New Done false
5682 1938 Acked 2937 Read 3 Done New Done false
5683 1939 Acked 2938 Read 9 Done New Done false
5684 1940 Acked 2939 Flush 0 Done New Done false
5685 1941 Acked 2940 Read 4 Done New Done false
5686 1942 Acked 2941 Read 1 Done New Done false
5687 1943 Acked 2942 Write 7 Done New Done false
5688 1944 Acked 2943 Write 4 Done New Done false
5689 1945 Acked 2944 Read 4 Done New Done false
5690 1946 Acked 2945 Read 5 Done New Done false
5691 1947 Acked 2946 Write 6 Done New Done false
5692 1948 Acked 2947 Read 2 Done New Done false
5693 1949 Acked 2948 Write 5 Done New Done false
5694 1950 Acked 2949 Write 4 Done New Done false
5695 1951 Acked 2950 Read 4 Done New Done false
5696 1952 Acked 2951 Read 9 Done New Done false
5697 1953 Acked 2952 Write 5 Done New Done false
5698 1954 Acked 2953 Write 6 Done New Done false
5699 1955 Acked 2954 Read 5 Done New Done false
5700 1956 Acked 2955 Read 9 Done New Done false
5701 1957 Acked 2956 Read 4 Done New Done false
5702 1958 Acked 2957 Read 8 Done New Done false
5703 1959 Acked 2958 Read 2 Done New Done false
5704 1960 Acked 2959 Write 9 Done New Done false
5705 1961 Acked 2960 Read 3 Done New Done false
5706 1962 Acked 2961 Read 4 Done New Done false
5707 1963 Acked 2962 Write 10 Done New Done false
5708 1964 Acked 2963 Write 2 Done New Done false
5709 1965 Acked 2964 Write 1 Done New Done false
5710 1966 Acked 2965 Write 8 Done New Done false
5711 1967 Acked 2966 Read 3 Done New Done false
5712 1968 Acked 2967 Flush 0 Done New Done false
5713 1969 Acked 2968 Read 4 Done New Done false
5714 1970 Acked 2969 Flush 0 Done New Done false
5715 1971 Acked 2970 Read 4 Done New Done false
5716 1972 Acked 2971 Read 8 Done New Done false
5717 1973 Acked 2972 Read 8 Done New Done false
5718 1974 Acked 2973 Read 9 Done New Done false
5719 1975 Acked 2974 Read 7 Done New Done false
5720 1976 Acked 2975 Write 7 Done New Done false
5721 1977 Acked 2976 Write 10 Done New Done false
5722 1978 Acked 2977 Read 9 Done New Done false
5723 1979 Acked 2978 Read 7 Done New Done false
5724 1980 Acked 2979 Read 8 Done New Done false
5725 1981 Acked 2980 Read 8 Done New Done false
5726 1982 Acked 2981 Write 3 Done New Done false
5727 1983 Acked 2982 Read 9 Done New Done false
5728 1984 Acked 2983 Read 8 Done New Done false
5729 1985 Acked 2984 Read 5 Done New Done false
5730 1986 Acked 2985 Write 8 Done New Done false
5731 1987 Acked 2986 Write 6 Done New Done false
5732 1988 Acked 2987 Write 4 Done New Done false
5733 1989 Acked 2988 Flush 0 Done New Done false
5734 1990 Acked 2989 Write 10 Done New Done false
5735 1991 Acked 2990 Read 2 Done New Done false
5736 1992 Acked 2991 Write 2 Done New Done false
5737 1993 Acked 2992 Write 8 Done New Done false
5738 1994 Acked 2993 Write 10 Done New Done false
5739 1995 Acked 2994 Read 4 Done New Done false
5740 1996 Acked 2995 Write 10 Done New Done false
5741 1997 Acked 2996 Flush 0 Done New Done false
5742 1998 Acked 2997 Write 1 Done New Done false
5743 1999 Acked 2998 Read 6 Done New Done false
5744 2000 Acked 2999 Write 10 Done New Done false
5745 2001 Acked 3000 Write 8 Done New Done false
5746 2002 Acked 3001 Read 4 Done New Done false
5747 2003 Acked 3002 Write 5 Done New Done false
5748 2004 Acked 3003 Read 4 Done New Done false
5749 2005 Acked 3004 Read 1 Done New Done false
5750 2006 Acked 3005 Read 4 Done New Done false
5751 2007 Acked 3006 Read 9 Done New Done false
5752 2008 Acked 3007 Read 1 Done New Done false
5753 2009 Acked 3008 Flush 0 Done New Done false
5754 2010 Acked 3009 Read 5 Done New Done false
5755 2011 Acked 3010 Read 1 Done New Done false
5756 2012 Acked 3011 Read 1 Done New Done false
5757 2013 Acked 3012 Read 2 Done New Done false
5758 2014 Acked 3013 Flush 0 Done New Done false
5759 2015 Acked 3014 Read 7 Done New Done false
5760 2016 Acked 3015 Write 2 Done New Done false
5761 2017 Acked 3016 Write 10 Done New Done false
5762 2018 Acked 3017 Read 5 Done New Done false
5763 2019 Acked 3018 Write 9 Done New Done false
5764 2020 Acked 3019 Read 8 Done New Done false
5765 2021 Acked 3020 Read 8 Done New Done false
5766 2022 Acked 3021 Read 4 Done New Done false
5767 2023 Acked 3022 Write 9 Done New Done false
5768 2024 Acked 3023 Write 6 Done New Done false
5769 2025 Acked 3024 Read 4 Done New Done false
5770 2026 Acked 3025 Write 7 Done New Done false
5771 2027 Acked 3026 Write 4 Done New Done false
5772 2028 Acked 3027 Flush 0 Done New Done false
5773 2029 Acked 3028 Read 9 Done New Done false
5774 2030 Acked 3029 Flush 0 Done New Done false
5775 2031 Acked 3030 Read 2 Done New Done false
5776 2032 Acked 3031 Flush 0 Done New Done false
5777 2033 Acked 3032 Read 2 Done New Done false
5778 2034 Acked 3033 Read 10 Done New Done false
5779 2035 Acked 3034 Flush 0 Done New Done false
5780 2036 Acked 3035 Read 6 Done New Done false
5781 2037 Acked 3036 Write 8 Done New Done false
5782 2038 Acked 3037 Write 8 Done New Done false
5783 2039 Acked 3038 Read 3 Done New Done false
5784 2040 Acked 3039 Read 8 Done New Done false
5785 2041 Acked 3040 Write 9 Done New Done false
5786 2042 Acked 3041 Read 1 Done New Done false
5787 2043 Acked 3042 Read 1 Done New Done false
5788 2044 Acked 3043 Read 7 Done New Done false
5789 2045 Acked 3044 Write 4 Done New Done false
5790 2046 Acked 3045 Read 6 Done New Done false
5791 2047 Acked 3046 Write 5 Done New Done false
5792 2048 Acked 3047 Flush 0 Done New Done false
5793 2049 Acked 3048 Flush 0 Done New Done false
5794 2050 Acked 3049 Read 6 Done New Done false
5795 2051 Acked 3050 Read 6 Done New Done false
5796 2052 Acked 3051 Flush 0 Done New Done false
5797 2053 Acked 3052 Write 8 Done New Done false
5798 2054 Acked 3053 Write 3 Done New Done false
5799 2055 Acked 3054 Read 1 Done New Done false
5800 2056 Acked 3055 Read 5 Done New Done false
5801 2057 Acked 3056 Read 5 Done New Done false
5802 2058 Acked 3057 Write 6 Done New Done false
5803 2059 Acked 3058 Flush 0 Done New Done false
5804 2060 Acked 3059 Write 4 Done New Done false
5805 2061 Acked 3060 Write 2 Done New Done false
5806 2062 Acked 3061 Write 5 Done New Done false
5807 2063 Acked 3062 Read 4 Done New Done false
5808 2064 Acked 3063 Write 2 Done New Done false
5809 2065 Acked 3064 Write 2 Done New Done false
5810 2066 Acked 3065 Read 7 Done New Done false
5811 2067 Acked 3066 Flush 0 Done New Done false
5812 2068 Acked 3067 Write 6 Done New Done false
5813 2069 Acked 3068 Write 6 Done New Done false
5814 2070 Acked 3069 Write 1 Done New Done false
5815 2071 Acked 3070 Write 10 Done New Done false
5816 2072 Acked 3071 Read 7 Done New Done false
5817 2073 Acked 3072 Write 9 Done New Done false
5818 2074 Acked 3073 Write 7 Done New Done false
5819 2075 Acked 3074 Write 8 Done New Done false
5820 2076 Acked 3075 Read 7 Done New Done false
5821 2077 Acked 3076 Write 6 Done New Done false
5822 2078 Acked 3077 Read 10 Done New Done false
5823 2079 Acked 3078 Write 3 Done New Done false
5824 2080 Acked 3079 Write 8 Done New Done false
5825 2081 Acked 3080 Write 9 Done New Done false
5826 2082 Acked 3081 Read 4 Done New Done false
5827 2083 Acked 3082 Write 1 Done New Done false
5828 2084 Acked 3083 Write 4 Done New Done false
5829 2085 Acked 3084 Read 5 Done New Done false
5830 2086 Acked 3085 Read 8 Done New Done false
5831 2087 Acked 3086 Write 6 Done New Done false
5832 2088 Acked 3087 Read 3 Done New Done false
5833 2089 Acked 3088 Read 3 Done New Done false
5834 2090 Acked 3089 Write 4 Done New Done false
5835 2091 Acked 3090 Write 8 Done New Done false
5836 2092 Acked 3091 Write 1 Done New Done false
5837 2093 Acked 3092 Write 4 Done New Done false
5838 2094 Acked 3093 Read 4 Done New Done false
5839 2095 Acked 3094 Write 10 Done New Done false
5840 2096 Acked 3095 Write 6 Done New Done false
5841 2097 Acked 3096 Write 8 Done New Done false
5842 2098 Acked 3097 Read 7 Done New Done false
5843 2099 Acked 3098 Write 1 Done New Done false
5844 2100 Acked 3099 Read 1 Done New Done false
5845 2101 Acked 3100 Write 7 Done New Done false
5846 2102 Acked 3101 Write 10 Done New Done false
5847 2103 Acked 3102 Write 9 Done New Done false
5848 2104 Acked 3103 Read 10 Done New Done false
5849 2105 Acked 3104 Read 5 Done New Done false
5850 2106 Acked 3105 Flush 0 Done New Done false
5851 2107 Acked 3106 Write 1 Done New Done false
5852 2108 Acked 3107 Read 1 Done New Done false
5853 2109 Acked 3108 Flush 0 Done New Done false
5854 2110 Acked 3109 Read 2 Done New Done false
5855 2111 Acked 3110 Flush 0 Done New Done false
5856 2112 Acked 3111 Read 8 Done New Done false
5857 2113 Acked 3112 Write 4 Done New Done false
5858 2114 Acked 3113 Write 3 Done New Done false
5859 2115 Acked 3114 Write 8 Done New Done false
5860 2116 Acked 3115 Read 6 Done New Done false
5861 2117 Acked 3116 Read 1 Done New Done false
5862 2118 Acked 3117 Read 3 Done New Done false
5863 STATES DS:0 DS:1 DS:2 TOTAL
5864 New 0 302 0 302
5865 Sent 0 0 0 0
5866 Done 302 0 302 604
5867 Skipped 0 0 0 0
5868 Error 0 0 0 0
5869 Last Flush: 3110 2815 3110
5870 Downstairs last five completed: 2815 2814 2813 2812 2811 2810
5871 Upstairs last five completed: 2118 2117 2116 2115 2114 2113
5872 CLIENT: Up:0 ds:302 act:2
5873 [8820] got start action
5874 [8820] I am going to restart
5875 Starting downstairs at port 8820
5876 Make output file at "/tmp/dsc/downstairs-8820.txt"
5877 [8820][1] reports Starting
5878 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1183)
5879 [8820][1] reports Running
58802023-10-13T18:10:19.915ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 1
58812023-10-13T18:10:19.915ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
58822023-10-13T18:10:19.915ZINFOcrucible: [1] client re-new 303 jobs since flush 2815 = downstairs
58832023-10-13T18:10:19.916ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
58842023-10-13T18:10:19.917ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
58852023-10-13T18:10:19.917ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
58862023-10-13T18:10:19.917ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
58872023-10-13T18:10:19.917ZINFOcrucible: [1] send last flush ID to this DS: 2815
58882023-10-13T18:10:19.917ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
58892023-10-13T18:10:19.917ZINFOcrucible: [1] Replied this last flush ID: 2815
58902023-10-13T18:10:19.917ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Offline Active ds_transition to Replay
58912023-10-13T18:10:19.917ZINFOcrucible: [1] Transition from Offline to Replay
58922023-10-13T18:10:19.917ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
58932023-10-13T18:10:19.917ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Replay Active ds_transition to Active
58942023-10-13T18:10:19.917ZINFOcrucible: [1] Transition from Replay to Active
58952023-10-13T18:10:19.917ZINFOcrucible: [1] Starts cmd_loop
58962023-10-13T18:10:19.917ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
58972023-10-13T18:10:19.917ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
58982023-10-13T18:10:19.917ZINFOcrucible: Set check for repair
58992023-10-13T18:10:20.918ZWARNcrucible: [1] flow control end
59002023-10-13T18:10:20.919ZINFOcrucible: Checking if live repair is needed
59012023-10-13T18:10:20.919ZINFOcrucible: No Live Repair required at this time
5902 ----------------------------------------------------------------
5903 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5904 Downstairs last five completed: 3118 3117 3116 3115 3114 3113
5905 Upstairs last five completed: 2119 2118 2117 2116 2115 2114
5906 CLIENT: Up:0 ds:0 act:3
5907 Replay: All jobs finished, all DS active.
5908 Main task has work to do, go find it
5909 got dsc Stop(1)
5910 stop 1
5911 Oct 13 18:10:21.898 INFO request completed, latency_us: 507, response_code: 204, uri: [8820] Got stop action so:false kr:false
5912 /stop/cid/1, method: POST, req_id: 123732c5-caf7-4eb0-8c87-af6cf8fe6933, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
5913 Oct 13 18:10:21.899 INFO request completed, latency_us: 194, response_code: 200, uri: /state/cid/1, method: GET, req_id: 35a4049a-fda4-4339-b27e-92b5adc2fd6e, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
59142023-10-13T18:10:21.899ZWARNcrucible: [1] downstairs disconnected
59152023-10-13T18:10:21.899ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
59162023-10-13T18:10:21.899ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8820 closed looper = 1
59172023-10-13T18:10:21.899ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
59182023-10-13T18:10:21.899ZWARNcrucible: [1] pm_task rx.recv() is None
59192023-10-13T18:10:21.899ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
59202023-10-13T18:10:21.899ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
5921 [8820] Exited with: ExitStatus(unix_wait_status(9))
5922 [8820][1] reports Exit
5923 Oct 13 18:10:25.902 INFO request completed, latency_us: 281, response_code: 200, uri: /state/cid/1, method: GET, req_id: c2ab3875-c11f-45ce-be1c-c73822dede51, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
5924 001/300 Write block 4835 len 40960 data: 2 2 2 3 3 3 3 4 3 2
59252023-10-13T18:10:25.904ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
5926 002/300 Read block 3749 len 16384
5927 003/300 Write block 1334 len 8192 data: 2 2
5928 004/300 Write block 190 len 8192 data: 4 4
5929 005/300 Write block 2741 len 36864 data: 3 3 2 2 2 3 3 3 3
5930 006/300 Write block 4862 len 4096 data: 3
5931 007/300 Write block 1571 len 36864 data: 2 2 2 2 2 2 2 2 2
5932 008/300 Read block 1244 len 12288
5933 009/300 Write block 2617 len 12288 data: 3 2 3
5934 010/300 Flush
5935 011/300 Read block 4630 len 16384
5936 012/300 Read block 3171 len 4096
5937 013/300 Read block 945 len 28672
5938 014/300 Read block 71 len 36864
5939 015/300 Read block 2635 len 8192
5940 016/300 Write block 1719 len 32768 data: 2 3 3 3 4 3 2 2
5941 017/300 Read block 3049 len 24576
5942 018/300 Read block 2310 len 8192
5943 019/300 Write block 1272 len 36864 data: 2 2 2 2 2 3 2 3 3
5944 020/300 Read block 3988 len 24576
5945 021/300 Write block 4883 len 32768 data: 3 3 3 3 2 2 2 3
5946 022/300 Read block 493 len 36864
5947 023/300 Write block 4903 len 40960 data: 2 2 2 2 2 3 2 2 2 2
5948 024/300 Write block 4743 len 36864 data: 2 2 2 2 2 2 2 2 2
5949 025/300 Write block 4312 len 20480 data: 5 4 4 4 4
5950 026/300 Write block 208 len 16384 data: 2 2 2 3
5951 027/300 Read block 3132 len 20480
5952 028/300 Read block 3049 len 12288
5953 029/300 Read block 3994 len 8192
5954 030/300 Read block 3222 len 4096
5955 031/300 Read block 4536 len 4096
5956 032/300 Write block 90 len 36864 data: 3 3 3 4 4 5 4 4 4
5957 033/300 Read block 3246 len 8192
5958 034/300 Read block 4847 len 32768
5959 035/300 Write block 1954 len 24576 data: 5 4 4 3 2 2
5960 036/300 Flush
5961 037/300 Flush
5962 038/300 Read block 650 len 4096
5963 039/300 Write block 3611 len 28672 data: 2 2 2 2 2 2 2
5964 040/300 Read block 1142 len 36864
5965 041/300 Read block 4566 len 12288
5966 042/300 Flush
5967 043/300 Write block 1023 len 12288 data: 3 2 2
5968 044/300 Read block 336 len 32768
5969 045/300 Read block 4221 len 16384
5970 046/300 Read block 4623 len 8192
5971 047/300 Read block 247 len 8192
5972 048/300 Write block 3892 len 4096 data: 2
5973 049/300 Read block 649 len 8192
5974 050/300 Read block 4696 len 16384
5975 051/300 Read block 1899 len 40960
5976 052/300 Write block 4579 len 12288 data: 2 3 3
5977 053/300 Write block 128 len 4096 data: 4
5978 054/300 Write block 1828 len 40960 data: 3 3 3 3 3 3 3 3 2 2
5979 055/300 Write block 2855 len 24576 data: 2 2 2 2 2 2
5980 056/300 Read block 3853 len 32768
5981 057/300 Write block 3959 len 12288 data: 4 3 2
5982 058/300 Write block 4242 len 4096 data: 3
5983 059/300 Flush
5984 060/300 Read block 810 len 12288
5985 061/300 Write block 4967 len 4096 data: 2
5986 062/300 Write block 1406 len 40960 data: 2 2 2 2 2 2 4 4 4 3
5987 063/300 Flush
5988 064/300 Write block 3863 len 20480 data: 4 4 4 4 4
5989 065/300 Write block 4860 len 12288 data: 6 4 4
5990 066/300 Read block 4749 len 20480
5991 067/300 Read block 3679 len 12288
5992 068/300 Read block 481 len 20480
5993 069/300 Read block 2267 len 32768
5994 070/300 Flush
5995 071/300 Read block 3692 len 40960
5996 072/300 Read block 3962 len 12288
5997 073/300 Write block 3474 len 8192 data: 3 3
5998 074/300 Write block 403 len 28672 data: 3 3 3 2 2 2 2
5999 075/300 Read block 1292 len 36864
6000 076/300 Read block 4935 len 4096
6001 077/300 Read block 3624 len 32768
6002 078/300 Read block 832 len 32768
6003 079/300 Write block 2508 len 20480 data: 5 4 3 3 3
6004 080/300 Flush
6005 081/300 Flush
6006 082/300 Read block 1 len 12288
6007 083/300 Write block 2153 len 20480 data: 2 2 2 2 2
6008 084/300 Write block 1489 len 8192 data: 6 5
6009 085/300 Write block 378 len 28672 data: 3 3 2 2 2 2 2
6010 086/300 Read block 2882 len 20480
6011 087/300 Flush
6012 088/300 Write block 4985 len 12288 data: 2 2 2
6013 089/300 Flush
6014 090/300 Read block 4812 len 4096
6015 091/300 Write block 1496 len 20480 data: 3 3 3 3 5
6016 092/300 Write block 3501 len 20480 data: 2 2 2 2 3
6017 093/300 Write block 4041 len 20480 data: 2 3 3 2 3
6018 094/300 Flush
6019 095/300 Flush
6020 096/300 Write block 3999 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6021 097/300 Read block 2829 len 16384
6022 098/300 Flush
6023 099/300 Write block 3750 len 12288 data: 3 3 4
6024 100/300 Read block 2321 len 36864
6025 101/300 Read block 799 len 32768
6026 102/300 Read block 4263 len 32768
6027 103/300 Flush
6028 104/300 Write block 1498 len 36864 data: 4 4 6 4 3 3 2 2 2
6029 105/300 Read block 2248 len 4096
6030 106/300 Read block 4490 len 32768
6031 107/300 Read block 2564 len 24576
6032 108/300 Write block 2303 len 20480 data: 3 3 4 5 4
6033 109/300 Write block 2966 len 28672 data: 4 2 2 2 2 3 3
6034 110/300 Read block 2427 len 8192
6035 111/300 Read block 1079 len 20480
6036 112/300 Flush
6037 113/300 Flush
6038 114/300 Flush
6039 115/300 Write block 1585 len 28672 data: 3 2 2 2 2 2 2
6040 116/300 Read block 4159 len 32768
6041 117/300 Write block 1165 len 24576 data: 4 4 3 3 3 3
6042 118/300 Write block 673 len 36864 data: 2 2 2 2 2 2 2 2 2
6043 119/300 Read block 190 len 32768
6044 120/300 Write block 738 len 40960 data: 2 2 2 2 2 3 3 4 4 3
6045 121/300 Flush
6046 122/300 Flush
6047 123/300 Read block 1062 len 20480
6048 124/300 Read block 1351 len 24576
6049 125/300 Write block 4461 len 8192 data: 3 2
6050 126/300 Write block 318 len 28672 data: 3 4 5 6 6 6 5
6051 127/300 Write block 2622 len 20480 data: 2 2 2 2 2
6052 128/300 Read block 1125 len 12288
6053 129/300 Read block 2897 len 12288
6054 130/300 Write block 4299 len 40960 data: 3 3 3 3 3 4 4 4 4 4
6055 131/300 Write block 1936 len 24576 data: 3 5 4 2 2 2
6056 132/300 Write block 3943 len 32768 data: 5 5 4 5 3 3 3 4
6057 133/300 Write block 3997 len 8192 data: 2 2
6058 134/300 Read block 3153 len 32768
6059 135/300 Read block 975 len 36864
6060 136/300 Write block 2192 len 16384 data: 4 4 4 4
6061 137/300 Write block 3632 len 8192 data: 2 2
6062 138/300 Read block 2275 len 24576
6063 139/300 Read block 1639 len 20480
6064 140/300 Read block 1328 len 36864
6065 141/300 Read block 3374 len 20480
6066 142/300 Write block 2081 len 8192 data: 2 2
6067 143/300 Read block 2466 len 12288
6068 144/300 Write block 313 len 4096 data: 2
6069 145/300 Write block 3369 len 20480 data: 2 2 2 2 2
6070 146/300 Write block 4384 len 40960 data: 4 4 4 5 4 3 3 2 2 2
6071 147/300 Read block 3912 len 16384
6072 148/300 Read block 4362 len 4096
6073 149/300 Read block 3003 len 36864
6074 150/300 Write block 736 len 36864 data: 2 2 3 3 3 3 3 4 4
6075 151/300 Read block 548 len 12288
6076 152/300 Read block 4190 len 36864
6077 153/300 Flush
6078 154/300 Read block 1784 len 20480
6079 155/300 Read block 222 len 28672
6080 156/300 Read block 3653 len 28672
6081 157/300 Read block 1674 len 28672
6082 158/300 Write block 1551 len 12288 data: 2 2 2
6083 159/300 Read block 743 len 36864
6084 160/300 Write block 844 len 36864 data: 3 3 4 4 4 4 4 4 4
6085 161/300 Flush
6086 162/300 Write block 2636 len 16384 data: 3 3 3 4
6087 163/300 Write block 3032 len 4096 data: 3
6088 164/300 Flush
6089 165/300 Write block 948 len 40960 data: 2 2 2 2 3 3 2 2 2 2
6090 166/300 Read block 1694 len 40960
6091 167/300 Write block 1668 len 4096 data: 3
6092 168/300 Read block 2725 len 20480
6093 169/300 Write block 4251 len 32768 data: 2 2 2 3 3 3 2 2
6094 170/300 Read block 82 len 32768
6095 171/300 Read block 1662 len 8192
6096 172/300 Read block 2072 len 12288
6097 173/300 Flush
6098 174/300 Read block 2598 len 8192
6099 175/300 Read block 3534 len 32768
6100 176/300 Flush
6101 177/300 Write block 4160 len 36864 data: 3 3 3 4 5 5 5 5 5
6102 178/300 Write block 4481 len 36864 data: 2 2 2 3 3 3 3 3 3
6103 179/300 Read block 65 len 40960
6104 180/300 Flush
6105 181/300 Write block 318 len 12288 data: 4 5 6
6106 182/300 Write block 4972 len 8192 data: 2 2
6107 183/300 Read block 1242 len 24576
6108 184/300 Read block 4791 len 32768
6109 185/300 Read block 3419 len 40960
6110 186/300 Write block 74 len 32768 data: 3 2 2 2 2 3 3 3
6111 187/300 Read block 1764 len 40960
6112 188/300 Write block 4694 len 40960 data: 3 3 3 3 3 3 3 5 6 4
6113 189/300 Flush
6114 190/300 Write block 3679 len 8192 data: 2 3
6115 191/300 Flush
6116 192/300 Read block 3838 len 40960
6117 193/300 Write block 580 len 4096 data: 2
6118 194/300 Write block 3856 len 12288 data: 3 3 3
6119 195/300 Read block 4752 len 24576
6120 196/300 Flush
6121 197/300 Write block 2662 len 28672 data: 3 3 2 2 3 3 3
6122 198/300 Read block 3299 len 12288
6123 199/300 Read block 4186 len 12288
6124 200/300 Write block 336 len 36864 data: 2 2 2 2 2 2 2 2 2
6125 201/300 Flush
6126 202/300 Read block 1154 len 36864
6127 203/300 Read block 3657 len 16384
6128 204/300 Write block 981 len 20480 data: 3 3 3 3 3
6129 205/300 Read block 4028 len 8192
6130 206/300 Read block 4849 len 20480
6131 207/300 Flush
6132 208/300 Write block 3020 len 36864 data: 4 4 3 3 3 3 3 2 3
6133 209/300 Write block 1098 len 12288 data: 7 7 7
6134 210/300 Read block 673 len 8192
6135 211/300 Write block 2920 len 32768 data: 2 2 2 2 2 2 2 2
6136 212/300 Write block 268 len 12288 data: 2 3 3
6137 213/300 Write block 3011 len 12288 data: 3 3 4
6138 214/300 Write block 1477 len 24576 data: 4 4 4 3 2 2
6139 215/300 Write block 2696 len 4096 data: 4
6140 216/300 Write block 4330 len 24576 data: 5 5 6 6 6 6
6141 217/300 Read block 1986 len 8192
6142 218/300 Write block 813 len 4096 data: 2
6143 219/300 Flush
6144 220/300 Flush
6145 221/300 Write block 2766 len 28672 data: 4 4 3 3 2 2 2
6146 222/300 Write block 14 len 36864 data: 2 2 2 2 2 2 3 4 4
6147 223/300 Read block 4763 len 16384
6148 224/300 Write block 2346 len 28672 data: 3 3 4 4 5 5 5
6149 225/300 Write block 512 len 24576 data: 2 2 2 2 2 2
6150 226/300 Read block 2863 len 20480
6151 227/300 Read block 3773 len 16384
6152 228/300 Write block 2026 len 32768 data: 2 2 2 2 2 2 3 2
6153 229/300 Read block 85 len 4096
6154 230/300 Read block 3220 len 4096
6155 231/300 Write block 677 len 40960 data: 3 3 3 3 3 2 2 2 2 2
6156 232/300 Read block 3909 len 12288
6157 233/300 Write block 3001 len 40960 data: 4 4 4 4 4 3 3 2 2 2
6158 234/300 Write block 587 len 32768 data: 2 2 2 2 2 2 2 2
6159 235/300 Read block 3074 len 36864
6160 236/300 Write block 3198 len 24576 data: 2 2 3 3 3 3
6161 237/300 Read block 1203 len 20480
6162 238/300 Write block 4592 len 40960 data: 3 3 3 3 3 3 2 2 2 2
6163 239/300 Read block 2857 len 8192
6164 240/300 Read block 1180 len 12288
6165 241/300 Read block 3297 len 32768
6166 242/300 Read block 2808 len 28672
6167 243/300 Write block 4791 len 28672 data: 2 2 2 3 3 3 3
6168 244/300 Write block 3107 len 20480 data: 3 3 4 4 4
6169 245/300 Read block 4787 len 28672
6170 246/300 Write block 2132 len 40960 data: 2 2 2 2 3 4 5 4 4 4
6171 247/300 Read block 3904 len 32768
6172 248/300 Read block 505 len 16384
6173 249/300 Flush
6174 250/300 Read block 3152 len 40960
6175 251/300 Write block 2185 len 12288 data: 2 2 2
6176 252/300 Write block 571 len 4096 data: 3
6177 253/300 Write block 1761 len 32768 data: 2 2 2 2 2 2 3 3
6178 254/300 Read block 4695 len 4096
6179 255/300 Flush
6180 256/300 Write block 3158 len 16384 data: 3 3 4 3
6181 257/300 Write block 1591 len 24576 data: 3 2 2 2 2 2
6182 258/300 Read block 2385 len 32768
6183 259/300 Read block 2492 len 16384
6184 260/300 Read block 4224 len 40960
6185 261/300 Read block 1207 len 28672
6186 262/300 Write block 2334 len 20480 data: 4 4 4 4 2
6187 263/300 Write block 697 len 40960 data: 2 3 3 3 3 3 4 4 5 4
6188 264/300 Write block 2102 len 8192 data: 2 2
6189 265/300 Read block 3879 len 32768
6190 266/300 Flush
6191 267/300 Read block 3190 len 12288
6192 268/300 Read block 2441 len 20480
6193 269/300 Flush
6194 270/300 Read block 4187 len 36864
6195 271/300 Write block 2728 len 24576 data: 2 2 2 3 3 4
6196 272/300 Read block 813 len 28672
6197 273/300 Read block 3155 len 40960
6198 274/300 Read block 3031 len 36864
6199 275/300 Read block 1311 len 20480
6200 276/300 Read block 2816 len 8192
6201 277/300 Read block 159 len 32768
6202 278/300 Write block 1395 len 8192 data: 3 2
6203 279/300 Write block 2477 len 40960 data: 2 3 3 3 3 3 3 3 4 5
6204 280/300 Flush
6205 281/300 Read block 3259 len 36864
6206 282/300 Write block 3176 len 32768 data: 3 3 3 3 2 2 2 2
6207 283/300 Read block 3200 len 40960
6208 284/300 Write block 3904 len 28672 data: 3 3 4 4 5 4 4
6209 285/300 Read block 4189 len 4096
6210 286/300 Read block 688 len 36864
6211 287/300 Write block 4882 len 12288 data: 3 4 4
6212 288/300 Write block 3234 len 12288 data: 4 4 4
6213 289/300 Read block 2409 len 8192
6214 290/300 Write block 615 len 20480 data: 2 2 2 2 2
6215 291/300 Read block 3692 len 24576
6216 292/300 Read block 2191 len 12288
6217 293/300 Read block 822 len 24576
6218 294/300 Read block 1135 len 40960
6219 295/300 Read block 1196 len 16384
6220 296/300 Read block 1825 len 8192
6221 297/300 Write block 1472 len 16384 data: 3 3 3 3
6222 298/300 Flush
6223 299/300 Write block 2272 len 24576 data: 2 2 2 2 2 3
6224 300/300 Read block 174 len 8192
6225 Main task has work to do, go find it
6226 got dsc Start(1)
6227 start 1
6228 Oct 13 18:10:26.892 INFO request completed, latency_us: 297, response_code: 204, uri: /start/cid/1, method: POST, req_id: 67e24277-8b49-4b11-ace2-caf0fc947f94, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
6229 Replay: started 1, returned:Ok(())
6230 ----------------------------------------------------------------
6231 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6232 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6233 2120 Acked 3119 Write 10 Done New Done false
6234 2121 Acked 3120 Read 4 Done New Done false
6235 2122 Acked 3121 Write 2 Done New Done false
6236 2123 Acked 3122 Write 2 Done New Done false
6237 2124 Acked 3123 Write 9 Done New Done false
6238 2125 Acked 3124 Write 1 Done New Done false
6239 2126 Acked 3125 Write 9 Done New Done false
6240 2127 Acked 3126 Read 3 Done New Done false
6241 2128 Acked 3127 Write 3 Done New Done false
6242 2129 Acked 3128 Flush 0 Done New Done false
6243 2130 Acked 3129 Read 4 Done New Done false
6244 2131 Acked 3130 Read 1 Done New Done false
6245 2132 Acked 3131 Read 7 Done New Done false
6246 2133 Acked 3132 Read 9 Done New Done false
6247 2134 Acked 3133 Read 2 Done New Done false
6248 2135 Acked 3134 Write 8 Done New Done false
6249 2136 Acked 3135 Read 6 Done New Done false
6250 2137 Acked 3136 Read 2 Done New Done false
6251 2138 Acked 3137 Write 9 Done New Done false
6252 2139 Acked 3138 Read 6 Done New Done false
6253 2140 Acked 3139 Write 8 Done New Done false
6254 2141 Acked 3140 Read 9 Done New Done false
6255 2142 Acked 3141 Write 10 Done New Done false
6256 2143 Acked 3142 Write 9 Done New Done false
6257 2144 Acked 3143 Write 5 Done New Done false
6258 2145 Acked 3144 Write 4 Done New Done false
6259 2146 Acked 3145 Read 5 Done New Done false
6260 2147 Acked 3146 Read 3 Done New Done false
6261 2148 Acked 3147 Read 2 Done New Done false
6262 2149 Acked 3148 Read 1 Done New Done false
6263 2150 Acked 3149 Read 1 Done New Done false
6264 2151 Acked 3150 Write 9 Done New Done false
6265 2152 Acked 3151 Read 2 Done New Done false
6266 2153 Acked 3152 Read 8 Done New Done false
6267 2154 Acked 3153 Write 6 Done New Done false
6268 2155 Acked 3154 Flush 0 Done New Done false
6269 2156 Acked 3155 Flush 0 Done New Done false
6270 2157 Acked 3156 Read 1 Done New Done false
6271 2158 Acked 3157 Write 7 Done New Done false
6272 2159 Acked 3158 Read 9 Done New Done false
6273 2160 Acked 3159 Read 3 Done New Done false
6274 2161 Acked 3160 Flush 0 Done New Done false
6275 2162 Acked 3161 Write 3 Done New Done false
6276 2163 Acked 3162 Read 8 Done New Done false
6277 2164 Acked 3163 Read 4 Done New Done false
6278 2165 Acked 3164 Read 2 Done New Done false
6279 2166 Acked 3165 Read 2 Done New Done false
6280 2167 Acked 3166 Write 1 Done New Done false
6281 2168 Acked 3167 Read 2 Done New Done false
6282 2169 Acked 3168 Read 4 Done New Done false
6283 2170 Acked 3169 Read 10 Done New Done false
6284 2171 Acked 3170 Write 3 Done New Done false
6285 2172 Acked 3171 Write 1 Done New Done false
6286 2173 Acked 3172 Write 10 Done New Done false
6287 2174 Acked 3173 Write 6 Done New Done false
6288 2175 Acked 3174 Read 8 Done New Done false
6289 2176 Acked 3175 Write 3 Done New Done false
6290 2177 Acked 3176 Write 1 Done New Done false
6291 2178 Acked 3177 Flush 0 Done New Done false
6292 2179 Acked 3178 Read 3 Done New Done false
6293 2180 Acked 3179 Write 1 Done New Done false
6294 2181 Acked 3180 Write 10 Done New Done false
6295 2182 Acked 3181 Flush 0 Done New Done false
6296 2183 Acked 3182 Write 5 Done New Done false
6297 2184 Acked 3183 Write 3 Done New Done false
6298 2185 Acked 3184 Read 5 Done New Done false
6299 2186 Acked 3185 Read 3 Done New Done false
6300 2187 Acked 3186 Read 5 Done New Done false
6301 2188 Acked 3187 Read 8 Done New Done false
6302 2189 Acked 3188 Flush 0 Done New Done false
6303 2190 Acked 3189 Read 10 Done New Done false
6304 2191 Acked 3190 Read 3 Done New Done false
6305 2192 Acked 3191 Write 2 Done New Done false
6306 2193 Acked 3192 Write 7 Done New Done false
6307 2194 Acked 3193 Read 9 Done New Done false
6308 2195 Acked 3194 Read 1 Done New Done false
6309 2196 Acked 3195 Read 8 Done New Done false
6310 2197 Acked 3196 Read 8 Done New Done false
6311 2198 Acked 3197 Write 5 Done New Done false
6312 2199 Acked 3198 Flush 0 Done New Done false
6313 2200 Acked 3199 Flush 0 Done New Done false
6314 2201 Acked 3200 Read 3 Done New Done false
6315 2202 Acked 3201 Write 5 Done New Done false
6316 2203 Acked 3202 Write 2 Done New Done false
6317 2204 Acked 3203 Write 7 Done New Done false
6318 2205 Acked 3204 Read 5 Done New Done false
6319 2206 Acked 3205 Flush 0 Done New Done false
6320 2207 Acked 3206 Write 3 Done New Done false
6321 2208 Acked 3207 Flush 0 Done New Done false
6322 2209 Acked 3208 Read 1 Done New Done false
6323 2210 Acked 3209 Write 5 Done New Done false
6324 2211 Acked 3210 Write 5 Done New Done false
6325 2212 Acked 3211 Write 5 Done New Done false
6326 2213 Acked 3212 Flush 0 Done New Done false
6327 2214 Acked 3213 Flush 0 Done New Done false
6328 2215 Acked 3214 Write 10 Done New Done false
6329 2216 Acked 3215 Read 4 Done New Done false
6330 2217 Acked 3216 Flush 0 Done New Done false
6331 2218 Acked 3217 Write 3 Done New Done false
6332 2219 Acked 3218 Read 9 Done New Done false
6333 2220 Acked 3219 Read 8 Done New Done false
6334 2221 Acked 3220 Flush 0 Done New Done false
6335 2222 Acked 3221 Read 8 Done New Done false
6336 2223 Acked 3222 Flush 0 Done New Done false
6337 2224 Acked 3223 Write 9 Done New Done false
6338 2225 Acked 3224 Read 1 Done New Done false
6339 2226 Acked 3225 Read 8 Done New Done false
6340 2227 Acked 3226 Read 6 Done New Done false
6341 2228 Acked 3227 Write 5 Done New Done false
6342 2229 Acked 3228 Write 7 Done New Done false
6343 2230 Acked 3229 Read 2 Done New Done false
6344 2231 Acked 3230 Read 5 Done New Done false
6345 2232 Acked 3231 Flush 0 Done New Done false
6346 2233 Acked 3232 Flush 0 Done New Done false
6347 2234 Acked 3233 Flush 0 Done New Done false
6348 2235 Acked 3234 Write 7 Done New Done false
6349 2236 Acked 3235 Read 8 Done New Done false
6350 2237 Acked 3236 Write 6 Done New Done false
6351 2238 Acked 3237 Write 9 Done New Done false
6352 2239 Acked 3238 Read 8 Done New Done false
6353 2240 Acked 3239 Write 10 Done New Done false
6354 2241 Acked 3240 Flush 0 Done New Done false
6355 2242 Acked 3241 Flush 0 Done New Done false
6356 2243 Acked 3242 Read 5 Done New Done false
6357 2244 Acked 3243 Read 6 Done New Done false
6358 2245 Acked 3244 Write 2 Done New Done false
6359 2246 Acked 3245 Write 7 Done New Done false
6360 2247 Acked 3246 Write 5 Done New Done false
6361 2248 Acked 3247 Read 3 Done New Done false
6362 2249 Acked 3248 Read 3 Done New Done false
6363 2250 Acked 3249 Write 10 Done New Done false
6364 2251 Acked 3250 Write 6 Done New Done false
6365 2252 Acked 3251 Write 8 Done New Done false
6366 2253 Acked 3252 Write 2 Done New Done false
6367 2254 Acked 3253 Read 8 Done New Done false
6368 2255 Acked 3254 Read 9 Done New Done false
6369 2256 Acked 3255 Write 4 Done New Done false
6370 2257 Acked 3256 Write 2 Done New Done false
6371 2258 Acked 3257 Read 6 Done New Done false
6372 2259 Acked 3258 Read 5 Done New Done false
6373 2260 Acked 3259 Read 9 Done New Done false
6374 2261 Acked 3260 Read 5 Done New Done false
6375 2262 Acked 3261 Write 2 Done New Done false
6376 2263 Acked 3262 Read 3 Done New Done false
6377 2264 Acked 3263 Write 1 Done New Done false
6378 2265 Acked 3264 Write 5 Done New Done false
6379 2266 Acked 3265 Write 10 Done New Done false
6380 2267 Acked 3266 Read 4 Done New Done false
6381 2268 Acked 3267 Read 1 Done New Done false
6382 2269 Acked 3268 Read 9 Done New Done false
6383 2270 Acked 3269 Write 9 Done New Done false
6384 2271 Acked 3270 Read 3 Done New Done false
6385 2272 Acked 3271 Read 9 Done New Done false
6386 2273 Acked 3272 Flush 0 Done New Done false
6387 2274 Acked 3273 Read 5 Done New Done false
6388 2275 Acked 3274 Read 7 Done New Done false
6389 2276 Acked 3275 Read 7 Done New Done false
6390 2277 Acked 3276 Read 7 Done New Done false
6391 2278 Acked 3277 Write 3 Done New Done false
6392 2279 Acked 3278 Read 9 Done New Done false
6393 2280 Acked 3279 Write 9 Done New Done false
6394 2281 Acked 3280 Flush 0 Done New Done false
6395 2282 Acked 3281 Write 4 Done New Done false
6396 2283 Acked 3282 Write 1 Done New Done false
6397 2284 Acked 3283 Flush 0 Done New Done false
6398 2285 Acked 3284 Write 10 Done New Done false
6399 2286 Acked 3285 Read 10 Done New Done false
6400 2287 Acked 3286 Write 1 Done New Done false
6401 2288 Acked 3287 Read 5 Done New Done false
6402 2289 Acked 3288 Write 8 Done New Done false
6403 2290 Acked 3289 Read 8 Done New Done false
6404 2291 Acked 3290 Read 2 Done New Done false
6405 2292 Acked 3291 Read 3 Done New Done false
6406 2293 Acked 3292 Flush 0 Done New Done false
6407 2294 Acked 3293 Read 2 Done New Done false
6408 2295 Acked 3294 Read 8 Done New Done false
6409 2296 Acked 3295 Flush 0 Done New Done false
6410 2297 Acked 3296 Write 9 Done New Done false
6411 2298 Acked 3297 Write 9 Done New Done false
6412 2299 Acked 3298 Read 10 Done New Done false
6413 2300 Acked 3299 Flush 0 Done New Done false
6414 2301 Acked 3300 Write 3 Done New Done false
6415 2302 Acked 3301 Write 2 Done New Done false
6416 2303 Acked 3302 Read 6 Done New Done false
6417 2304 Acked 3303 Read 8 Done New Done false
6418 2305 Acked 3304 Read 10 Done New Done false
6419 2306 Acked 3305 Write 8 Done New Done false
6420 2307 Acked 3306 Read 10 Done New Done false
6421 2308 Acked 3307 Write 10 Done New Done false
6422 2309 Acked 3308 Flush 0 Done New Done false
6423 2310 Acked 3309 Write 2 Done New Done false
6424 2311 Acked 3310 Flush 0 Done New Done false
6425 2312 Acked 3311 Read 10 Done New Done false
6426 2313 Acked 3312 Write 1 Done New Done false
6427 2314 Acked 3313 Write 3 Done New Done false
6428 2315 Acked 3314 Read 6 Done New Done false
6429 2316 Acked 3315 Flush 0 Done New Done false
6430 2317 Acked 3316 Write 7 Done New Done false
6431 2318 Acked 3317 Read 3 Done New Done false
6432 2319 Acked 3318 Read 3 Done New Done false
6433 2320 Acked 3319 Write 9 Done New Done false
6434 2321 Acked 3320 Flush 0 Done New Done false
6435 2322 Acked 3321 Read 9 Done New Done false
6436 2323 Acked 3322 Read 4 Done New Done false
6437 2324 Acked 3323 Write 5 Done New Done false
6438 2325 Acked 3324 Read 2 Done New Done false
6439 2326 Acked 3325 Read 5 Done New Done false
6440 2327 Acked 3326 Flush 0 Done New Done false
6441 2328 Acked 3327 Write 9 Done New Done false
6442 2329 Acked 3328 Write 3 Done New Done false
6443 2330 Acked 3329 Read 2 Done New Done false
6444 2331 Acked 3330 Write 8 Done New Done false
6445 2332 Acked 3331 Write 3 Done New Done false
6446 2333 Acked 3332 Write 3 Done New Done false
6447 2334 Acked 3333 Write 6 Done New Done false
6448 2335 Acked 3334 Write 1 Done New Done false
6449 2336 Acked 3335 Write 6 Done New Done false
6450 2337 Acked 3336 Read 2 Done New Done false
6451 2338 Acked 3337 Write 1 Done New Done false
6452 2339 Acked 3338 Flush 0 Done New Done false
6453 2340 Acked 3339 Flush 0 Done New Done false
6454 2341 Acked 3340 Write 7 Done New Done false
6455 2342 Acked 3341 Write 9 Done New Done false
6456 2343 Acked 3342 Read 4 Done New Done false
6457 2344 Acked 3343 Write 7 Done New Done false
6458 2345 Acked 3344 Write 6 Done New Done false
6459 2346 Acked 3345 Read 5 Done New Done false
6460 2347 Acked 3346 Read 4 Done New Done false
6461 2348 Acked 3347 Write 8 Done New Done false
6462 2349 Acked 3348 Read 1 Done New Done false
6463 2350 Acked 3349 Read 1 Done New Done false
6464 2351 Acked 3350 Write 10 Done New Done false
6465 2352 Acked 3351 Read 3 Done New Done false
6466 2353 Acked 3352 Write 10 Done New Done false
6467 2354 Acked 3353 Write 8 Done New Done false
6468 2355 Acked 3354 Read 9 Done New Done false
6469 2356 Acked 3355 Write 6 Done New Done false
6470 2357 Acked 3356 Read 5 Done New Done false
6471 2358 Acked 3357 Write 10 Done New Done false
6472 2359 Acked 3358 Read 2 Done New Done false
6473 2360 Acked 3359 Read 3 Done New Done false
6474 2361 Acked 3360 Read 8 Done New Done false
6475 2362 Acked 3361 Read 7 Done New Done false
6476 2363 Acked 3362 Write 7 Done New Done false
6477 2364 Acked 3363 Write 5 Done New Done false
6478 2365 Acked 3364 Read 7 Done New Done false
6479 2366 Acked 3365 Write 10 Done New Done false
6480 2367 Acked 3366 Read 8 Done New Done false
6481 2368 Acked 3367 Read 4 Done New Done false
6482 2369 Acked 3368 Flush 0 Done New Done false
6483 2370 Acked 3369 Read 10 Done New Done false
6484 2371 Acked 3370 Write 3 Done New Done false
6485 2372 Acked 3371 Write 1 Done New Done false
6486 2373 Acked 3372 Write 8 Done New Done false
6487 2374 Acked 3373 Read 1 Done New Done false
6488 2375 Acked 3374 Flush 0 Done New Done false
6489 2376 Acked 3375 Write 4 Done New Done false
6490 2377 Acked 3376 Write 6 Done New Done false
6491 2378 Acked 3377 Read 8 Done New Done false
6492 2379 Acked 3378 Read 4 Done New Done false
6493 2380 Acked 3379 Read 10 Done New Done false
6494 2381 Acked 3380 Read 7 Done New Done false
6495 2382 Acked 3381 Write 5 Done New Done false
6496 2383 Acked 3382 Write 10 Done New Done false
6497 2384 Acked 3383 Write 2 Done New Done false
6498 2385 Acked 3384 Read 8 Done New Done false
6499 2386 Acked 3385 Flush 0 Done New Done false
6500 2387 Acked 3386 Read 3 Done New Done false
6501 2388 Acked 3387 Read 5 Done New Done false
6502 2389 Acked 3388 Flush 0 Done New Done false
6503 2390 Acked 3389 Read 9 Done New Done false
6504 2391 Acked 3390 Write 6 Done New Done false
6505 2392 Acked 3391 Read 7 Done New Done false
6506 2393 Acked 3392 Read 10 Done New Done false
6507 2394 Acked 3393 Read 9 Done New Done false
6508 2395 Acked 3394 Read 5 Done New Done false
6509 2396 Acked 3395 Read 2 Done New Done false
6510 2397 Acked 3396 Read 8 Done New Done false
6511 2398 Acked 3397 Write 2 Done New Done false
6512 2399 Acked 3398 Write 10 Done New Done false
6513 2400 Acked 3399 Flush 0 Done New Done false
6514 2401 Acked 3400 Read 9 Done New Done false
6515 2402 Acked 3401 Write 8 Done New Done false
6516 2403 Acked 3402 Read 10 Done New Done false
6517 2404 Acked 3403 Write 7 Done New Done false
6518 2405 Acked 3404 Read 1 Done New Done false
6519 2406 Acked 3405 Read 9 Done New Done false
6520 2407 Acked 3406 Write 3 Done New Done false
6521 2408 Acked 3407 Write 3 Done New Done false
6522 2409 Acked 3408 Read 2 Done New Done false
6523 2410 Acked 3409 Write 5 Done New Done false
6524 2411 Acked 3410 Read 6 Done New Done false
6525 2412 Acked 3411 Read 3 Done New Done false
6526 2413 Acked 3412 Read 6 Done New Done false
6527 2414 Acked 3413 Read 10 Done New Done false
6528 2415 Acked 3414 Read 4 Done New Done false
6529 2416 Acked 3415 Read 2 Done New Done false
6530 2417 Acked 3416 Write 4 Done New Done false
6531 2418 Acked 3417 Flush 0 Done New Done false
6532 2419 Acked 3418 Write 6 Done New Done false
6533 2420 Acked 3419 Read 2 Done New Done false
6534 STATES DS:0 DS:1 DS:2 TOTAL
6535 New 0 301 0 301
6536 Sent 0 0 0 0
6537 Done 301 0 301 602
6538 Skipped 0 0 0 0
6539 Error 0 0 0 0
6540 Last Flush: 3417 3118 3417
6541 Downstairs last five completed: 3118 3117 3116 3115 3114 3113
6542 Upstairs last five completed: 2420 2419 2418 2417 2416 2415
6543 CLIENT: Up:0 ds:301 act:2
6544 [8820] got start action
6545 [8820] I am going to restart
6546 Starting downstairs at port 8820
6547 Make output file at "/tmp/dsc/downstairs-8820.txt"
6548 [8820][1] reports Starting
6549 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1184)
6550 [8820][1] reports Running
65512023-10-13T18:10:28.907ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 1
65522023-10-13T18:10:28.907ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
65532023-10-13T18:10:28.907ZINFOcrucible: [1] client re-new 302 jobs since flush 3118 = downstairs
65542023-10-13T18:10:28.908ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
65552023-10-13T18:10:28.909ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
65562023-10-13T18:10:28.909ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
65572023-10-13T18:10:28.909ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
65582023-10-13T18:10:28.909ZINFOcrucible: [1] send last flush ID to this DS: 3118
65592023-10-13T18:10:28.909ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
65602023-10-13T18:10:28.909ZINFOcrucible: [1] Replied this last flush ID: 3118
65612023-10-13T18:10:28.909ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Offline Active ds_transition to Replay
65622023-10-13T18:10:28.909ZINFOcrucible: [1] Transition from Offline to Replay
65632023-10-13T18:10:28.909ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
65642023-10-13T18:10:28.909ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Replay Active ds_transition to Active
65652023-10-13T18:10:28.909ZINFOcrucible: [1] Transition from Replay to Active
65662023-10-13T18:10:28.909ZINFOcrucible: [1] Starts cmd_loop
65672023-10-13T18:10:28.909ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
65682023-10-13T18:10:28.909ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
65692023-10-13T18:10:28.909ZINFOcrucible: Set check for repair
65702023-10-13T18:10:29.910ZINFOcrucible: Checking if live repair is needed
65712023-10-13T18:10:29.910ZINFOcrucible: No Live Repair required at this time
65722023-10-13T18:10:29.910ZWARNcrucible: [1] flow control end
6573 ----------------------------------------------------------------
6574 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6575 Downstairs last five completed: 3420 3419 3418 3417 3416 3415
6576 Upstairs last five completed: 2421 2420 2419 2418 2417 2416
6577 CLIENT: Up:0 ds:0 act:3
6578 Replay: All jobs finished, all DS active.
6579 Main task has work to do, go find it
6580 got dsc Stop(1)
6581 stop 1
6582 Oct 13 18:10:30.897 INFO request completed, latency_us: 523, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 9533c080-de7a-424e-8e79-f159c35e2163, remote_addr: 127.0.0.1:64571[8820] Got stop action so:false kr:false
6583 , local_addr: 127.0.0.1:9998
6584 Oct 13 18:10:30.898 INFO request completed, latency_us: 263, response_code: 200, uri: /state/cid/1, method: GET, req_id: cffee0e9-0ead-430e-bbba-c7b70c29f023, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
65852023-10-13T18:10:30.899ZWARNcrucible: [1] downstairs disconnected
65862023-10-13T18:10:30.899ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
65872023-10-13T18:10:30.899ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8820 closed looper = 1
65882023-10-13T18:10:30.899ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
65892023-10-13T18:10:30.899ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
65902023-10-13T18:10:30.899ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
65912023-10-13T18:10:30.899ZWARNcrucible: [1] pm_task rx.recv() is None
6592 [8820] Exited with: ExitStatus(unix_wait_status(9))
6593 [8820][1] reports Exit
6594 Oct 13 18:10:34.901 INFO request completed, latency_us: 314, response_code: 200, uri: /state/cid/1, method: GET, req_id: c2381cd4-3c75-4862-8b9f-04076303084d, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
6595 001/300 Read block 2663 len 28672
6596 002/300 Write block 810 len 40960 data: 2 2 2 3 2 2 2 2 2 2
6597 003/300 Write block 1853 len 16384 data: 2 2 2 2
6598 004/300 Read block 4303 len 20480
6599 005/300 Write block 185 len 16384 data: 2 2 2 3
6600 006/300 Read block 2061 len 24576
6601 007/300 Read block 2580 len 24576
6602 008/300 Read block 1772 len 40960
6603 009/300 Flush
6604 010/300 Write block 34 len 16384 data: 2 3 3 3
6605 011/300 Read block 1249 len 20480
6606 012/300 Read block 790 len 40960
6607 013/300 Write block 4445 len 16384 data: 3 3 3 3
6608 014/300 Flush
6609 015/300 Flush
6610 016/300 Read block 3007 len 36864
6611 017/300 Read block 2271 len 24576
6612 018/300 Write block 2585 len 40960 data: 4 4 3 4 4 5 4 3 3 3
6613 019/300 Read block 550 len 28672
6614 020/300 Write block 3350 len 20480 data: 2 3 2 2 2
6615 021/300 Flush
6616 022/300 Read block 3092 len 4096
6617 023/300 Read block 227 len 24576
6618 024/300 Read block 495 len 20480
6619 025/300 Write block 1631 len 20480 data: 2 4 4 5 4
6620 026/300 Read block 4129 len 20480
6621 027/300 Read block 4722 len 20480
6622 028/300 Read block 616 len 12288
6623 029/300 Read block 450 len 24576
6624 030/300 Read block 2349 len 40960
6625 031/300 Read block 4170 len 12288
6626 032/300 Read block 3415 len 32768
6627 033/300 Read block 2315 len 4096
6628 034/300 Read block 2295 len 8192
6629 035/300 Write block 2163 len 12288 data: 2 2 2
6630 036/300 Write block 4224 len 36864 data: 3 3 2 2 2 3 3 4 3
6631 037/300 Write block 2128 len 20480 data: 2 3 2 2 3
6632 038/300 Flush
6633 039/300 Write block 311 len 16384 data: 2 2 3 3
6634 040/300 Read block 3170 len 36864
6635 041/300 Read block 2495 len 24576
6636 042/300 Flush
6637 043/300 Read block 1598 len 8192
6638 044/300 Write block 3430 len 28672 data: 4 4 4 4 3 3 3
6639 045/300 Read block 3864 len 32768
6640 046/300 Flush
6641 047/300 Write block 2004 len 40960 data: 4 4 4 5 6 5 5 4 4 2
6642 048/300 Read block 2976 len 12288
6643 049/300 Write block 4544 len 20480 data: 3 3 2 2 2
6644 050/300 Read block 4478 len 4096
6645 051/300 Write block 2525 len 20480 data: 4 4 4 3 3
6646 052/300 Read block 734 len 8192
6647 053/300 Read block 1842 len 8192
6648 054/300 Read block 2848 len 24576
6649 055/300 Read block 4776 len 32768
6650 056/300 Read block 2140 len 32768
6651 057/300 Write block 2234 len 4096 data: 2
6652 058/300 Write block 3042 len 16384 data: 4 3 3 3
6653 059/300 Read block 219 len 20480
6654 060/300 Read block 3487 len 8192
6655 061/300 Write block 2619 len 32768 data: 4 3 2 3 3 3 3 3
6656 062/300 Write block 2159 len 8192 data: 4 3
6657 063/300 Read block 2696 len 28672
6658 064/300 Write block 48 len 12288 data: 4 3 3
6659 065/300 Read block 3451 len 4096
6660 066/300 Read block 2897 len 8192
6661 067/300 Write block 1172 len 20480 data: 4 4 3 3 3
6662 068/300 Write block 4556 len 12288 data: 3 3 3
6663 069/300 Read block 1681 len 4096
6664 070/300 Write block 3333 len 4096 data: 3
6665 071/300 Write block 373 len 16384 data: 3 4 4 3
6666 072/300 Write block 1203 len 36864 data: 4 4 3 2 2 3 3 3 3
6667 073/300 Write block 4398 len 24576 data: 3 3 3 3 3 3
6668 074/300 Write block 2171 len 24576 data: 2 2 2 2 2 2
6669 075/300 Read block 35 len 16384
6670 076/300 Read block 4383 len 8192
6671 077/300 Write block 2023 len 32768 data: 2 2 2 3 3 3 3 3
6672 078/300 Read block 2943 len 40960
6673 079/300 Read block 472 len 20480
6674 080/300 Write block 2396 len 40960 data: 4 5 6 6 5 3 3 3 3 3
6675 081/300 Write block 3489 len 32768 data: 4 4 4 3 3 3 2 2
6676 082/300 Read block 3919 len 24576
6677 083/300 Write block 1574 len 8192 data: 3 3
6678 084/300 Read block 1185 len 24576
6679 085/300 Read block 4203 len 24576
6680 086/300 Flush
6681 087/300 Read block 3164 len 32768
6682 088/300 Write block 3920 len 20480 data: 3 3 3 3 3
6683 089/300 Write block 4760 len 40960 data: 4 4 4 3 2 2 2 2 2 2
6684 090/300 Write block 4833 len 4096 data: 2
6685 091/300 Read block 2114 len 16384
6686 092/300 Read block 139 len 12288
6687 093/300 Read block 1109 len 16384
6688 094/300 Read block 1222 len 32768
6689 095/300 Read block 2991 len 4096
6690 096/300 Read block 662 len 16384
6691 097/300 Read block 3640 len 16384
6692 098/300 Write block 2787 len 8192 data: 2 2
6693 099/300 Read block 2405 len 12288
6694 100/300 Write block 2637 len 28672 data: 4 4 5 3 3 3 3
6695 101/300 Read block 322 len 4096
6696 102/300 Read block 4118 len 4096
6697 103/300 Write block 4498 len 32768 data: 3 3 3 3 2 3 3 3
6698 104/300 Read block 3629 len 8192
6699 105/300 Read block 1123 len 24576
6700 106/300 Read block 3828 len 8192
6701 107/300 Read block 3428 len 16384
6702 108/300 Read block 1588 len 12288
6703 109/300 Read block 1475 len 32768
6704 110/300 Read block 2533 len 16384
6705 111/300 Read block 3820 len 40960
6706 112/300 Read block 1503 len 12288
6707 113/300 Read block 3304 len 36864
6708 114/300 Read block 1150 len 40960
6709 115/300 Read block 3509 len 16384
6710 116/300 Write block 44 len 40960 data: 3 3 4 4 5 4 4 2 2 2
6711 117/300 Read block 2805 len 4096
6712 118/300 Read block 1788 len 4096
6713 119/300 Write block 1215 len 36864 data: 3 2 2 2 2 2 2 2 2
6714 120/300 Read block 3646 len 20480
6715 121/300 Flush
6716 122/300 Read block 2710 len 40960
6717 123/300 Read block 702 len 32768
6718 124/300 Flush
6719 125/300 Read block 4429 len 16384
6720 126/300 Read block 3117 len 36864
6721 127/300 Read block 1166 len 40960
6722 128/300 Read block 3394 len 28672
6723 129/300 Write block 2354 len 28672 data: 4 4 4 4 2 2 2
6724 130/300 Read block 3472 len 12288
6725 131/300 Read block 3085 len 36864
6726 132/300 Write block 4716 len 40960 data: 3 3 2 2 3 4 4 3 3 3
6727 133/300 Read block 1000 len 24576
6728 134/300 Read block 2459 len 24576
6729 135/300 Read block 1300 len 16384
6730 136/300 Write block 4625 len 28672 data: 3 3 4 4 4 3 3
6731 137/300 Write block 3047 len 24576 data: 3 2 3 3 3 2
6732 138/300 Write block 4521 len 12288 data: 3 3 3
6733 139/300 Flush
6734 140/300 Read block 4397 len 12288
6735 141/300 Write block 1040 len 20480 data: 2 4 4 4 4
6736 142/300 Flush
6737 143/300 Write block 1463 len 32768 data: 2 2 2 3 3 3 2 2
6738 144/300 Write block 4196 len 8192 data: 3 3
6739 145/300 Write block 3279 len 40960 data: 2 2 2 2 2 2 3 3 3 3
6740 146/300 Read block 3646 len 12288
6741 147/300 Write block 3428 len 28672 data: 3 4 5 5 5 5 4
6742 148/300 Flush
6743 149/300 Write block 4152 len 28672 data: 4 4 4 3 3 3 3
6744 150/300 Flush
6745 151/300 Read block 1162 len 12288
6746 152/300 Write block 2751 len 8192 data: 3 3
6747 153/300 Read block 1219 len 36864
6748 154/300 Flush
6749 155/300 Read block 1605 len 12288
6750 156/300 Read block 4509 len 40960
6751 157/300 Write block 4363 len 40960 data: 3 3 3 3 2 2 2 2 3 4
6752 158/300 Read block 4682 len 8192
6753 159/300 Read block 4237 len 32768
6754 160/300 Write block 173 len 32768 data: 3 3 4 4 4 4 4 3
6755 161/300 Write block 817 len 16384 data: 3 3 3 2
6756 162/300 Read block 2103 len 8192
6757 163/300 Read block 280 len 20480
6758 164/300 Write block 181 len 32768 data: 2 2 2 2 3 3 3 4
6759 165/300 Flush
6760 166/300 Read block 1763 len 8192
6761 167/300 Write block 2280 len 28672 data: 5 5 5 3 2 2 2
6762 168/300 Read block 2969 len 20480
6763 169/300 Read block 1768 len 24576
6764 170/300 Write block 1030 len 12288 data: 3 2 3
6765 171/300 Read block 438 len 28672
6766 172/300 Write block 1067 len 40960 data: 4 3 2 2 2 2 2 2 2 3
6767 173/300 Read block 3555 len 4096
6768 174/300 Write block 2891 len 8192 data: 4 4
6769 175/300 Write block 4398 len 36864 data: 4 4 4 4 4 4 3 5 5
6770 176/300 Read block 4647 len 40960
6771 177/300 Read block 3936 len 8192
6772 178/300 Write block 585 len 12288 data: 2 2 3
6773 179/300 Read block 2193 len 32768
6774 180/300 Write block 1632 len 32768 data: 5 5 6 5 4 4 5 5
6775 181/300 Read block 4352 len 20480
6776 182/300 Read block 2578 len 40960
6777 183/300 Read block 1687 len 32768
6778 184/300 Flush
6779 185/300 Read block 148 len 8192
6780 186/300 Flush
6781 187/300 Write block 2066 len 8192 data: 2 2
6782 188/300 Write block 3396 len 8192 data: 4 2
6783 189/300 Write block 3093 len 32768 data: 3 3 5 5 4 4 4 5
6784 190/300 Read block 3720 len 28672
6785 191/300 Write block 84 len 4096 data: 2
6786 192/300 Write block 4592 len 36864 data: 4 4 4 4 4 4 3 3 3
6787 193/300 Read block 1103 len 36864
6788 194/300 Read block 871 len 12288
6789 195/300 Write block 3933 len 32768 data: 3 3 4 3 5 5 5 4
6790 196/300 Flush
6791 197/300 Write block 3596 len 12288 data: 4 3 2
6792 198/300 Read block 4808 len 24576
6793 199/300 Flush
6794 200/300 Write block 3079 len 40960 data: 2 2 2 3 4 4 2 2 2 2
6795 201/300 Read block 1859 len 32768
6796 202/300 Read block 2645 len 36864
6797 203/300 Flush
6798 204/300 Read block 4322 len 36864
6799 205/300 Write block 1403 len 32768 data: 3 3 3 3 3 3 3 3
6800 206/300 Read block 4037 len 28672
6801 207/300 Write block 2900 len 16384 data: 4 4 4 4
6802 208/300 Read block 1414 len 28672
6803 209/300 Write block 1875 len 24576 data: 2 3 3 3 2 2
6804 210/300 Write block 3498 len 12288 data: 2 2 2
6805 211/300 Read block 2211 len 16384
6806 212/300 Write block 2069 len 32768 data: 2 2 2 2 2 3 3 3
6807 213/300 Read block 748 len 32768
6808 214/300 Read block 2099 len 32768
6809 215/300 Write block 2770 len 24576 data: 3 3 3 2 2 2
6810 216/300 Write block 3737 len 28672 data: 4 4 4 3 4 4 3
6811 217/300 Read block 1635 len 24576
6812 218/300 Write block 3509 len 16384 data: 3 3 3 2
6813 219/300 Read block 2785 len 32768
6814 220/300 Write block 938 len 40960 data: 4 4 3 2 2 2 2 2 2 2
6815 221/300 Read block 3904 len 36864
6816 222/300 Read block 576 len 16384
6817 223/300 Write block 2104 len 32768 data: 4 4 2 2 2 2 2 2
6818 224/300 Write block 4096 len 12288 data: 2 2 3
6819 225/300 Write block 1878 len 32768 data: 4 3 3 2 2 3 3 3
6820 226/300 Write block 2771 len 16384 data: 4 4 3 3
6821 227/300 Read block 809 len 24576
6822 228/300 Write block 521 len 20480 data: 2 2 2 2 2
6823 229/300 Write block 798 len 28672 data: 2 3 3 3 3 3 3
6824 230/300 Read block 3738 len 20480
6825 231/300 Read block 4867 len 36864
6826 232/300 Read block 1081 len 20480
6827 233/300 Write block 1255 len 36864 data: 2 2 2 2 2 2 2 2 2
6828 234/300 Read block 217 len 16384
6829 235/300 Read block 2359 len 12288
6830 236/300 Read block 1510 len 4096
6831 237/300 Flush
6832 238/300 Read block 1296 len 16384
6833 239/300 Read block 1743 len 36864
6834 240/300 Write block 4621 len 16384 data: 4 4 4 4
6835 241/300 Read block 4748 len 16384
6836 242/300 Read block 866 len 4096
6837 243/300 Read block 823 len 40960
6838 244/300 Read block 319 len 4096
6839 245/300 Write block 1034 len 4096 data: 3
6840 246/300 Read block 3964 len 4096
6841 247/300 Flush
6842 248/300 Read block 4967 len 12288
6843 249/300 Write block 2777 len 28672 data: 2 2 2 2 2 2 2
6844 250/300 Flush
6845 251/300 Write block 2161 len 24576 data: 3 2 3 3 3 2
6846 252/300 Write block 3708 len 32768 data: 5 5 6 7 6 5 5 6
6847 253/300 Write block 1623 len 36864 data: 2 2 2 2 2 2 2 2 3
6848 254/300 Read block 2917 len 20480
6849 255/300 Read block 116 len 4096
6850 256/300 Read block 2911 len 28672
6851 257/300 Read block 3579 len 24576
6852 258/300 Write block 4241 len 40960 data: 2 4 3 3 2 2 2 2 2 2
6853 259/300 Write block 4057 len 8192 data: 3 3
6854 260/300 Read block 2466 len 12288
6855 261/300 Read block 2988 len 36864
6856 262/300 Read block 3699 len 40960
6857 263/300 Read block 4565 len 24576
6858 264/300 Write block 116 len 24576 data: 2 2 2 2 2 2
6859 265/300 Read block 2460 len 20480
6860 266/300 Write block 4062 len 40960 data: 3 3 3 2 2 2 2 2 2 2
6861 267/300 Write block 4324 len 36864 data: 4 4 4 4 3 5 6 6 7
6862 268/300 Read block 1403 len 36864
6863 269/300 Flush
6864 270/300 Read block 1723 len 20480
6865 271/300 Flush
6866 272/300 Write block 2380 len 24576 data: 3 3 3 3 3 2
6867 273/300 Read block 3068 len 36864
6868 274/300 Write block 1059 len 20480 data: 5 4 3 3 3
6869 275/300 Read block 3462 len 32768
6870 276/300 Write block 1338 len 36864 data: 2 2 2 2 2 3 3 3 3
6871 277/300 Write block 1230 len 12288 data: 3 3 3
6872 278/300 Read block 1640 len 16384
6873 279/300 Write block 3538 len 32768 data: 5 5 4 3 3 2 2 3
6874 280/300 Read block 4055 len 40960
6875 281/300 Flush
6876 282/300 Write block 3404 len 36864 data: 2 2 3 3 3 3 3 3 3
6877 283/300 Read block 4108 len 28672
6878 284/300 Read block 4306 len 20480
6879 285/300 Write block 4408 len 28672 data: 3 3 3 3 3 4 4
6880 286/300 Flush
6881 287/300 Flush
6882 288/300 Read block 4760 len 28672
6883 289/300 Write block 1812 len 16384 data: 4 4 2 3
6884 290/300 Flush
6885 291/300 Read block 4003 len 4096
6886 292/300 Write block 3153 len 4096 data: 2
6887 293/300 Write block 818 len 36864 data: 4 4 3 2 2 2 2 2 2
6888 294/300 Write block 2833 len 16384 data: 3 3 3 3
6889 295/300 Write block 3496 len 28672 data: 3 2 3 3 3 3 3
6890 296/300 Write block 2344 len 12288 data: 4 3 4
6891 297/300 Read block 3327 len 40960
6892 298/300 Write block 1087 len 8192 data: 4 4
6893 299/300 Read block 1890 len 20480
6894 300/300 Read block 3757 len 28672
6895 Main task has work to do, go find it
6896 got dsc Start(1)
6897 start 1
6898 Oct 13 18:10:35.915 INFO request completed, latency_us: 313, response_code: 204, uri: /start/cid/1, method: POST, req_id: ae50985c-bf45-4024-a693-32087452deef, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
6899 Replay: started 1, returned:Ok(())
6900 ----------------------------------------------------------------
6901 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
6902 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6903 2422 Acked 3421 Read 7 Done New Done false
6904 2423 Acked 3422 Write 10 Done New Done false
6905 2424 Acked 3423 Write 4 Done New Done false
6906 2425 Acked 3424 Read 5 Done New Done false
6907 2426 Acked 3425 Write 4 Done New Done false
6908 2427 Acked 3426 Read 6 Done New Done false
6909 2428 Acked 3427 Read 6 Done New Done false
6910 2429 Acked 3428 Read 10 Done New Done false
6911 2430 Acked 3429 Flush 0 Done New Done false
6912 2431 Acked 3430 Write 4 Done New Done false
6913 2432 Acked 3431 Read 5 Done New Done false
6914 2433 Acked 3432 Read 10 Done New Done false
6915 2434 Acked 3433 Write 4 Done New Done false
6916 2435 Acked 3434 Flush 0 Done New Done false
6917 2436 Acked 3435 Flush 0 Done New Done false
6918 2437 Acked 3436 Read 9 Done New Done false
6919 2438 Acked 3437 Read 6 Done New Done false
6920 2439 Acked 3438 Write 10 Done New Done false
6921 2440 Acked 3439 Read 7 Done New Done false
6922 2441 Acked 3440 Write 5 Done New Done false
6923 2442 Acked 3441 Flush 0 Done New Done false
6924 2443 Acked 3442 Read 1 Done New Done false
6925 2444 Acked 3443 Read 6 Done New Done false
6926 2445 Acked 3444 Read 5 Done New Done false
6927 2446 Acked 3445 Write 5 Done New Done false
6928 2447 Acked 3446 Read 5 Done New Done false
6929 2448 Acked 3447 Read 5 Done New Done false
6930 2449 Acked 3448 Read 3 Done New Done false
6931 2450 Acked 3449 Read 6 Done New Done false
6932 2451 Acked 3450 Read 10 Done New Done false
6933 2452 Acked 3451 Read 3 Done New Done false
6934 2453 Acked 3452 Read 8 Done New Done false
6935 2454 Acked 3453 Read 1 Done New Done false
6936 2455 Acked 3454 Read 2 Done New Done false
6937 2456 Acked 3455 Write 3 Done New Done false
6938 2457 Acked 3456 Write 9 Done New Done false
6939 2458 Acked 3457 Write 5 Done New Done false
6940 2459 Acked 3458 Flush 0 Done New Done false
6941 2460 Acked 3459 Write 4 Done New Done false
6942 2461 Acked 3460 Read 9 Done New Done false
6943 2462 Acked 3461 Read 6 Done New Done false
6944 2463 Acked 3462 Flush 0 Done New Done false
6945 2464 Acked 3463 Read 2 Done New Done false
6946 2465 Acked 3464 Write 7 Done New Done false
6947 2466 Acked 3465 Read 8 Done New Done false
6948 2467 Acked 3466 Flush 0 Done New Done false
6949 2468 Acked 3467 Write 10 Done New Done false
6950 2469 Acked 3468 Read 3 Done New Done false
6951 2470 Acked 3469 Write 5 Done New Done false
6952 2471 Acked 3470 Read 1 Done New Done false
6953 2472 Acked 3471 Write 5 Done New Done false
6954 2473 Acked 3472 Read 2 Done New Done false
6955 2474 Acked 3473 Read 2 Done New Done false
6956 2475 Acked 3474 Read 6 Done New Done false
6957 2476 Acked 3475 Read 8 Done New Done false
6958 2477 Acked 3476 Read 8 Done New Done false
6959 2478 Acked 3477 Write 1 Done New Done false
6960 2479 Acked 3478 Write 4 Done New Done false
6961 2480 Acked 3479 Read 5 Done New Done false
6962 2481 Acked 3480 Read 2 Done New Done false
6963 2482 Acked 3481 Write 8 Done New Done false
6964 2483 Acked 3482 Write 2 Done New Done false
6965 2484 Acked 3483 Read 7 Done New Done false
6966 2485 Acked 3484 Write 3 Done New Done false
6967 2486 Acked 3485 Read 1 Done New Done false
6968 2487 Acked 3486 Read 2 Done New Done false
6969 2488 Acked 3487 Write 5 Done New Done false
6970 2489 Acked 3488 Write 3 Done New Done false
6971 2490 Acked 3489 Read 1 Done New Done false
6972 2491 Acked 3490 Write 1 Done New Done false
6973 2492 Acked 3491 Write 4 Done New Done false
6974 2493 Acked 3492 Write 9 Done New Done false
6975 2494 Acked 3493 Write 6 Done New Done false
6976 2495 Acked 3494 Write 6 Done New Done false
6977 2496 Acked 3495 Read 4 Done New Done false
6978 2497 Acked 3496 Read 2 Done New Done false
6979 2498 Acked 3497 Write 8 Done New Done false
6980 2499 Acked 3498 Read 10 Done New Done false
6981 2500 Acked 3499 Read 5 Done New Done false
6982 2501 Acked 3500 Write 10 Done New Done false
6983 2502 Acked 3501 Write 8 Done New Done false
6984 2503 Acked 3502 Read 6 Done New Done false
6985 2504 Acked 3503 Write 2 Done New Done false
6986 2505 Acked 3504 Read 6 Done New Done false
6987 2506 Acked 3505 Read 6 Done New Done false
6988 2507 Acked 3506 Flush 0 Done New Done false
6989 2508 Acked 3507 Read 8 Done New Done false
6990 2509 Acked 3508 Write 5 Done New Done false
6991 2510 Acked 3509 Write 10 Done New Done false
6992 2511 Acked 3510 Write 1 Done New Done false
6993 2512 Acked 3511 Read 4 Done New Done false
6994 2513 Acked 3512 Read 3 Done New Done false
6995 2514 Acked 3513 Read 4 Done New Done false
6996 2515 Acked 3514 Read 8 Done New Done false
6997 2516 Acked 3515 Read 1 Done New Done false
6998 2517 Acked 3516 Read 4 Done New Done false
6999 2518 Acked 3517 Read 4 Done New Done false
7000 2519 Acked 3518 Write 2 Done New Done false
7001 2520 Acked 3519 Read 3 Done New Done false
7002 2521 Acked 3520 Write 7 Done New Done false
7003 2522 Acked 3521 Read 1 Done New Done false
7004 2523 Acked 3522 Read 1 Done New Done false
7005 2524 Acked 3523 Write 8 Done New Done false
7006 2525 Acked 3524 Read 2 Done New Done false
7007 2526 Acked 3525 Read 6 Done New Done false
7008 2527 Acked 3526 Flush 0 Done New Done false
7009 2528 Acked 3527 Read 2 Done New Done false
7010 2529 Acked 3528 Read 4 Done New Done false
7011 2530 Acked 3529 Read 3 Done New Done false
7012 2531 Acked 3530 Read 8 Done New Done false
7013 2532 Acked 3531 Read 4 Done New Done false
7014 2533 Acked 3532 Read 10 Done New Done false
7015 2534 Acked 3533 Read 3 Done New Done false
7016 2535 Acked 3534 Read 9 Done New Done false
7017 2536 Acked 3535 Read 10 Done New Done false
7018 2537 Acked 3536 Read 4 Done New Done false
7019 2538 Acked 3537 Write 10 Done New Done false
7020 2539 Acked 3538 Read 1 Done New Done false
7021 2540 Acked 3539 Read 1 Done New Done false
7022 2541 Acked 3540 Write 9 Done New Done false
7023 2542 Acked 3541 Read 5 Done New Done false
7024 2543 Acked 3542 Flush 0 Done New Done false
7025 2544 Acked 3543 Read 10 Done New Done false
7026 2545 Acked 3544 Read 8 Done New Done false
7027 2546 Acked 3545 Flush 0 Done New Done false
7028 2547 Acked 3546 Read 4 Done New Done false
7029 2548 Acked 3547 Read 9 Done New Done false
7030 2549 Acked 3548 Read 10 Done New Done false
7031 2550 Acked 3549 Read 7 Done New Done false
7032 2551 Acked 3550 Write 7 Done New Done false
7033 2552 Acked 3551 Read 3 Done New Done false
7034 2553 Acked 3552 Read 9 Done New Done false
7035 2554 Acked 3553 Write 10 Done New Done false
7036 2555 Acked 3554 Read 6 Done New Done false
7037 2556 Acked 3555 Read 6 Done New Done false
7038 2557 Acked 3556 Read 4 Done New Done false
7039 2558 Acked 3557 Write 7 Done New Done false
7040 2559 Acked 3558 Write 6 Done New Done false
7041 2560 Acked 3559 Write 3 Done New Done false
7042 2561 Acked 3560 Flush 0 Done New Done false
7043 2562 Acked 3561 Read 3 Done New Done false
7044 2563 Acked 3562 Write 5 Done New Done false
7045 2564 Acked 3563 Flush 0 Done New Done false
7046 2565 Acked 3564 Write 8 Done New Done false
7047 2566 Acked 3565 Write 2 Done New Done false
7048 2567 Acked 3566 Write 10 Done New Done false
7049 2568 Acked 3567 Read 3 Done New Done false
7050 2569 Acked 3568 Write 7 Done New Done false
7051 2570 Acked 3569 Flush 0 Done New Done false
7052 2571 Acked 3570 Write 7 Done New Done false
7053 2572 Acked 3571 Flush 0 Done New Done false
7054 2573 Acked 3572 Read 3 Done New Done false
7055 2574 Acked 3573 Write 2 Done New Done false
7056 2575 Acked 3574 Read 9 Done New Done false
7057 2576 Acked 3575 Flush 0 Done New Done false
7058 2577 Acked 3576 Read 3 Done New Done false
7059 2578 Acked 3577 Read 10 Done New Done false
7060 2579 Acked 3578 Write 10 Done New Done false
7061 2580 Acked 3579 Read 2 Done New Done false
7062 2581 Acked 3580 Read 8 Done New Done false
7063 2582 Acked 3581 Write 8 Done New Done false
7064 2583 Acked 3582 Write 4 Done New Done false
7065 2584 Acked 3583 Read 2 Done New Done false
7066 2585 Acked 3584 Read 5 Done New Done false
7067 2586 Acked 3585 Write 8 Done New Done false
7068 2587 Acked 3586 Flush 0 Done New Done false
7069 2588 Acked 3587 Read 2 Done New Done false
7070 2589 Acked 3588 Write 7 Done New Done false
7071 2590 Acked 3589 Read 5 Done New Done false
7072 2591 Acked 3590 Read 6 Done New Done false
7073 2592 Acked 3591 Write 3 Done New Done false
7074 2593 Acked 3592 Read 7 Done New Done false
7075 2594 Acked 3593 Write 10 Done New Done false
7076 2595 Acked 3594 Read 1 Done New Done false
7077 2596 Acked 3595 Write 2 Done New Done false
7078 2597 Acked 3596 Write 9 Done New Done false
7079 2598 Acked 3597 Read 10 Done New Done false
7080 2599 Acked 3598 Read 2 Done New Done false
7081 2600 Acked 3599 Write 3 Done New Done false
7082 2601 Acked 3600 Read 8 Done New Done false
7083 2602 Acked 3601 Write 8 Done New Done false
7084 2603 Acked 3602 Read 5 Done New Done false
7085 2604 Acked 3603 Read 10 Done New Done false
7086 2605 Acked 3604 Read 8 Done New Done false
7087 2606 Acked 3605 Flush 0 Done New Done false
7088 2607 Acked 3606 Read 2 Done New Done false
7089 2608 Acked 3607 Flush 0 Done New Done false
7090 2609 Acked 3608 Write 2 Done New Done false
7091 2610 Acked 3609 Write 2 Done New Done false
7092 2611 Acked 3610 Write 8 Done New Done false
7093 2612 Acked 3611 Read 7 Done New Done false
7094 2613 Acked 3612 Write 1 Done New Done false
7095 2614 Acked 3613 Write 9 Done New Done false
7096 2615 Acked 3614 Read 9 Done New Done false
7097 2616 Acked 3615 Read 3 Done New Done false
7098 2617 Acked 3616 Write 8 Done New Done false
7099 2618 Acked 3617 Flush 0 Done New Done false
7100 2619 Acked 3618 Write 3 Done New Done false
7101 2620 Acked 3619 Read 6 Done New Done false
7102 2621 Acked 3620 Flush 0 Done New Done false
7103 2622 Acked 3621 Write 10 Done New Done false
7104 2623 Acked 3622 Read 8 Done New Done false
7105 2624 Acked 3623 Read 9 Done New Done false
7106 2625 Acked 3624 Flush 0 Done New Done false
7107 2626 Acked 3625 Read 9 Done New Done false
7108 2627 Acked 3626 Write 8 Done New Done false
7109 2628 Acked 3627 Read 7 Done New Done false
7110 2629 Acked 3628 Write 4 Done New Done false
7111 2630 Acked 3629 Read 7 Done New Done false
7112 2631 Acked 3630 Write 6 Done New Done false
7113 2632 Acked 3631 Write 3 Done New Done false
7114 2633 Acked 3632 Read 4 Done New Done false
7115 2634 Acked 3633 Write 8 Done New Done false
7116 2635 Acked 3634 Read 8 Done New Done false
7117 2636 Acked 3635 Read 8 Done New Done false
7118 2637 Acked 3636 Write 6 Done New Done false
7119 2638 Acked 3637 Write 7 Done New Done false
7120 2639 Acked 3638 Read 6 Done New Done false
7121 2640 Acked 3639 Write 4 Done New Done false
7122 2641 Acked 3640 Read 8 Done New Done false
7123 2642 Acked 3641 Write 10 Done New Done false
7124 2643 Acked 3642 Read 9 Done New Done false
7125 2644 Acked 3643 Read 4 Done New Done false
7126 2645 Acked 3644 Write 8 Done New Done false
7127 2646 Acked 3645 Write 3 Done New Done false
7128 2647 Acked 3646 Write 8 Done New Done false
7129 2648 Acked 3647 Write 4 Done New Done false
7130 2649 Acked 3648 Read 6 Done New Done false
7131 2650 Acked 3649 Write 5 Done New Done false
7132 2651 Acked 3650 Write 7 Done New Done false
7133 2652 Acked 3651 Read 5 Done New Done false
7134 2653 Acked 3652 Read 9 Done New Done false
7135 2654 Acked 3653 Read 5 Done New Done false
7136 2655 Acked 3654 Write 9 Done New Done false
7137 2656 Acked 3655 Read 4 Done New Done false
7138 2657 Acked 3656 Read 3 Done New Done false
7139 2658 Acked 3657 Read 1 Done New Done false
7140 2659 Acked 3658 Flush 0 Done New Done false
7141 2660 Acked 3659 Read 4 Done New Done false
7142 2661 Acked 3660 Read 9 Done New Done false
7143 2662 Acked 3661 Write 4 Done New Done false
7144 2663 Acked 3662 Read 4 Done New Done false
7145 2664 Acked 3663 Read 1 Done New Done false
7146 2665 Acked 3664 Read 10 Done New Done false
7147 2666 Acked 3665 Read 1 Done New Done false
7148 2667 Acked 3666 Write 1 Done New Done false
7149 2668 Acked 3667 Read 1 Done New Done false
7150 2669 Acked 3668 Flush 0 Done New Done false
7151 2670 Acked 3669 Read 3 Done New Done false
7152 2671 Acked 3670 Write 7 Done New Done false
7153 2672 Acked 3671 Flush 0 Done New Done false
7154 2673 Acked 3672 Write 6 Done New Done false
7155 2674 Acked 3673 Write 8 Done New Done false
7156 2675 Acked 3674 Write 9 Done New Done false
7157 2676 Acked 3675 Read 5 Done New Done false
7158 2677 Acked 3676 Flush 0 Done New Done false
7159 2678 Acked 3677 Read 1 Done New Done false
7160 2679 Acked 3678 Read 7 Done New Done false
7161 2680 Acked 3679 Read 6 Done New Done false
7162 2681 Acked 3680 Write 10 Done New Done false
7163 2682 Acked 3681 Write 2 Done New Done false
7164 2683 Acked 3682 Read 3 Done New Done false
7165 2684 Acked 3683 Read 9 Done New Done false
7166 2685 Acked 3684 Read 10 Done New Done false
7167 2686 Acked 3685 Read 6 Done New Done false
7168 2687 Acked 3686 Write 6 Done New Done false
7169 2688 Acked 3687 Read 5 Done New Done false
7170 2689 Acked 3688 Write 10 Done New Done false
7171 2690 Acked 3689 Write 9 Done New Done false
7172 2691 Acked 3690 Read 9 Done New Done false
7173 2692 Acked 3691 Flush 0 Done New Done false
7174 2693 Acked 3692 Read 5 Done New Done false
7175 2694 Acked 3693 Flush 0 Done New Done false
7176 2695 Acked 3694 Write 6 Done New Done false
7177 2696 Acked 3695 Read 9 Done New Done false
7178 2697 Acked 3696 Write 5 Done New Done false
7179 2698 Acked 3697 Read 8 Done New Done false
7180 2699 Acked 3698 Write 9 Done New Done false
7181 2700 Acked 3699 Write 3 Done New Done false
7182 2701 Acked 3700 Read 4 Done New Done false
7183 2702 Acked 3701 Write 8 Done New Done false
7184 2703 Acked 3702 Read 10 Done New Done false
7185 2704 Acked 3703 Flush 0 Done New Done false
7186 2705 Acked 3704 Write 9 Done New Done false
7187 2706 Acked 3705 Read 7 Done New Done false
7188 2707 Acked 3706 Read 5 Done New Done false
7189 2708 Acked 3707 Write 7 Done New Done false
7190 2709 Acked 3708 Flush 0 Done New Done false
7191 2710 Acked 3709 Flush 0 Done New Done false
7192 2711 Acked 3710 Read 7 Done New Done false
7193 2712 Acked 3711 Write 4 Done New Done false
7194 2713 Acked 3712 Flush 0 Done New Done false
7195 2714 Acked 3713 Read 1 Done New Done false
7196 2715 Acked 3714 Write 1 Done New Done false
7197 2716 Acked 3715 Write 9 Done New Done false
7198 2717 Acked 3716 Write 4 Done New Done false
7199 2718 Acked 3717 Write 7 Done New Done false
7200 2719 Acked 3718 Write 3 Done New Done false
7201 2720 Acked 3719 Read 10 Done New Done false
7202 2721 Acked 3720 Write 2 Done New Done false
7203 2722 Acked 3721 Read 5 Done New Done false
7204 2723 Acked 3722 Read 7 Done New Done false
7205 STATES DS:0 DS:1 DS:2 TOTAL
7206 New 0 302 0 302
7207 Sent 0 0 0 0
7208 Done 302 0 302 604
7209 Skipped 0 0 0 0
7210 Error 0 0 0 0
7211 Last Flush: 3712 3420 3712
7212 Downstairs last five completed: 3420 3419 3418 3417 3416 3415
7213 Upstairs last five completed: 2723 2722 2721 2720 2719 2718
7214 CLIENT: Up:0 ds:302 act:2
7215 [8820] got start action
7216 [8820] I am going to restart
7217 Starting downstairs at port 8820
7218 Make output file at "/tmp/dsc/downstairs-8820.txt"
7219 [8820][1] reports Starting
7220 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1185)
7221 [8820][1] reports Running
72222023-10-13T18:10:37.908ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
72232023-10-13T18:10:37.908ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 1
72242023-10-13T18:10:37.908ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
72252023-10-13T18:10:37.908ZINFOcrucible: [1] client re-new 303 jobs since flush 3420 = downstairs
72262023-10-13T18:10:37.909ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
72272023-10-13T18:10:37.910ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
72282023-10-13T18:10:37.910ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
72292023-10-13T18:10:37.910ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
72302023-10-13T18:10:37.910ZINFOcrucible: [1] send last flush ID to this DS: 3420
72312023-10-13T18:10:37.910ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
72322023-10-13T18:10:37.910ZINFOcrucible: [1] Replied this last flush ID: 3420
72332023-10-13T18:10:37.910ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Offline Active ds_transition to Replay
72342023-10-13T18:10:37.910ZINFOcrucible: [1] Transition from Offline to Replay
72352023-10-13T18:10:37.910ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
72362023-10-13T18:10:37.910ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Replay Active ds_transition to Active
72372023-10-13T18:10:37.910ZINFOcrucible: [1] Transition from Replay to Active
72382023-10-13T18:10:37.910ZINFOcrucible: [1] Starts cmd_loop
72392023-10-13T18:10:37.910ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
72402023-10-13T18:10:37.910ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
72412023-10-13T18:10:37.910ZINFOcrucible: Set check for repair
72422023-10-13T18:10:38.911ZINFOcrucible: Checking if live repair is needed
72432023-10-13T18:10:38.911ZINFOcrucible: No Live Repair required at this time
72442023-10-13T18:10:38.911ZWARNcrucible: [1] flow control end
7245 ----------------------------------------------------------------
7246 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7247 Downstairs last five completed: 3723 3722 3721 3720 3719 3718
7248 Upstairs last five completed: 2724 2723 2722 2721 2720 2719
7249 CLIENT: Up:0 ds:0 act:3
7250 Replay: All jobs finished, all DS active.
7251 Main task has work to do, go find it
7252 got dsc Stop(2)
7253 stop 2
7254 Oct 13 18:10:39.920 INFO request completed, latency_us: 517, response_code: 204, uri[8830] Got stop action so:false kr:false
7255 : /stop/cid/2, method: POST, req_id: 01d1c047-14cb-440c-9ae5-c03baf382ae8, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
7256 Oct 13 18:10:39.921 INFO request completed, latency_us: 181, response_code: 200, uri: /state/cid/2, method: GET, req_id: 6d6ad309-ccb4-474b-9714-638d3b1cd1e0, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
72572023-10-13T18:10:39.921ZWARNcrucible: [2] downstairs disconnected
72582023-10-13T18:10:39.921ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
72592023-10-13T18:10:39.921ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8830 closed looper = 2
72602023-10-13T18:10:39.921ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
72612023-10-13T18:10:39.921ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
72622023-10-13T18:10:39.921ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
72632023-10-13T18:10:39.921ZWARNcrucible: [2] pm_task rx.recv() is None
7264 [8830] Exited with: ExitStatus(unix_wait_status(9))
7265 [8830][2] reports Exit
7266 Oct 13 18:10:43.924 INFO request completed, latency_us: 275, response_code: 200, uri: /state/cid/2, method: GET, req_id: 6617656f-1a0a-4632-b1fa-7dd827a7de5a, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
7267 001/300 Read block 1238 len 16384
7268 002/300 Read block 1788 len 36864
72692023-10-13T18:10:43.928ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
7270 003/300 Read block 380 len 36864
7271 004/300 Read block 584 len 4096
7272 005/300 Read block 635 len 24576
7273 006/300 Read block 2799 len 36864
7274 007/300 Write block 156 len 16384 data: 2 2 2 2
7275 008/300 Write block 1221 len 8192 data: 3 3
7276 009/300 Write block 2307 len 28672 data: 5 4 3 3 3 2 2
7277 010/300 Read block 3732 len 36864
7278 011/300 Write block 4632 len 4096 data: 2
7279 012/300 Write block 3146 len 20480 data: 4 4 2 2 2
7280 013/300 Read block 2235 len 8192
7281 014/300 Write block 1026 len 4096 data: 2
7282 015/300 Write block 4742 len 40960 data: 2 3 3 3 3 3 3 3 3 3
7283 016/300 Write block 461 len 12288 data: 3 3 3
7284 017/300 Read block 812 len 16384
7285 018/300 Write block 2446 len 24576 data: 3 2 2 2 2 2
7286 019/300 Read block 4751 len 20480
7287 020/300 Read block 4106 len 20480
7288 021/300 Write block 2143 len 4096 data: 3
7289 022/300 Read block 3 len 16384
7290 023/300 Write block 3662 len 28672 data: 3 3 3 3 5 4 4
7291 024/300 Read block 3292 len 12288
7292 025/300 Flush
7293 026/300 Write block 1809 len 40960 data: 5 4 4 5 5 3 4 3 3 3
7294 027/300 Write block 1252 len 36864 data: 2 2 2 3 3 3 3 3 3
7295 028/300 Read block 2843 len 28672
7296 029/300 Read block 871 len 20480
7297 030/300 Write block 3313 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7298 031/300 Read block 2800 len 16384
7299 032/300 Read block 4364 len 40960
7300 033/300 Read block 1573 len 8192
7301 034/300 Write block 1624 len 36864 data: 3 3 3 3 3 3 3 4 6
7302 035/300 Write block 3775 len 36864 data: 4 3 3 3 3 3 3 3 3
7303 036/300 Read block 3291 len 12288
7304 037/300 Write block 4699 len 20480 data: 4 4 6 7 5
7305 038/300 Read block 2142 len 8192
7306 039/300 Write block 491 len 8192 data: 4 3
7307 040/300 Read block 433 len 32768
7308 041/300 Read block 1491 len 8192
7309 042/300 Read block 3633 len 36864
7310 043/300 Read block 2920 len 12288
7311 044/300 Flush
7312 045/300 Read block 285 len 32768
7313 046/300 Write block 1457 len 4096 data: 3
7314 047/300 Write block 227 len 12288 data: 3 3 3
7315 048/300 Read block 3401 len 20480
7316 049/300 Read block 1182 len 28672
7317 050/300 Read block 4650 len 40960
7318 051/300 Read block 3281 len 16384
7319 052/300 Read block 135 len 8192
7320 053/300 Write block 2904 len 4096 data: 4
7321 054/300 Read block 3195 len 24576
7322 055/300 Write block 33 len 20480 data: 2 3 4 4 4
7323 056/300 Read block 4631 len 20480
7324 057/300 Read block 1044 len 24576
7325 058/300 Write block 3680 len 40960 data: 4 3 3 3 3 3 3 3 3 4
7326 059/300 Flush
7327 060/300 Read block 2094 len 12288
7328 061/300 Write block 4724 len 36864 data: 4 4 3 3 3 3 2 2 2
7329 062/300 Read block 662 len 4096
7330 063/300 Write block 4049 len 16384 data: 3 3 3 3
7331 064/300 Write block 4740 len 24576 data: 2 2 3 4 4 4
7332 065/300 Flush
7333 066/300 Read block 1602 len 12288
7334 067/300 Read block 2032 len 4096
7335 068/300 Write block 297 len 36864 data: 3 3 3 3 3 2 2 2 2
7336 069/300 Read block 1071 len 24576
7337 070/300 Flush
7338 071/300 Write block 4330 len 40960 data: 7 7 8 7 7 7 4 4 3 3
7339 072/300 Read block 3331 len 4096
7340 073/300 Write block 1981 len 8192 data: 3 3
7341 074/300 Read block 2703 len 36864
7342 075/300 Write block 3619 len 8192 data: 3 4
7343 076/300 Read block 1882 len 32768
7344 077/300 Read block 4796 len 28672
7345 078/300 Read block 4233 len 36864
7346 079/300 Write block 2678 len 28672 data: 4 4 3 3 4 4 2
7347 080/300 Write block 3662 len 20480 data: 4 4 4 4 6
7348 081/300 Read block 3912 len 8192
7349 082/300 Read block 2704 len 16384
7350 083/300 Read block 1129 len 8192
7351 084/300 Write block 4384 len 4096 data: 5
7352 085/300 Read block 1423 len 20480
7353 086/300 Read block 445 len 20480
7354 087/300 Write block 2997 len 40960 data: 5 5 5 6 5 5 5 5 5 4
7355 088/300 Write block 642 len 16384 data: 3 3 2 2
7356 089/300 Write block 1390 len 40960 data: 4 4 3 3 3 4 3 3 3 4
7357 090/300 Write block 1012 len 4096 data: 4
7358 091/300 Write block 836 len 20480 data: 3 3 3 3 4
7359 092/300 Read block 3427 len 32768
7360 093/300 Read block 2926 len 36864
7361 094/300 Read block 4151 len 28672
7362 095/300 Read block 430 len 36864
7363 096/300 Read block 2112 len 16384
7364 097/300 Write block 1331 len 28672 data: 2 2 2 3 3 2 2
7365 098/300 Read block 588 len 4096
7366 099/300 Read block 1369 len 40960
7367 100/300 Read block 3930 len 20480
7368 101/300 Read block 951 len 16384
7369 102/300 Write block 3850 len 28672 data: 2 2 2 2 3 3 4
7370 103/300 Write block 4296 len 28672 data: 2 4 4 4 4 4 4
7371 104/300 Read block 1785 len 36864
7372 105/300 Flush
7373 106/300 Flush
7374 107/300 Write block 3868 len 20480 data: 4 3 3 3 3
7375 108/300 Read block 456 len 28672
7376 109/300 Read block 1036 len 20480
7377 110/300 Write block 2798 len 16384 data: 4 2 2 2
7378 111/300 Write block 1530 len 12288 data: 2 2 2
7379 112/300 Read block 1182 len 20480
7380 113/300 Read block 3656 len 32768
7381 114/300 Read block 109 len 32768
7382 115/300 Read block 4952 len 16384
7383 116/300 Write block 2144 len 12288 data: 3 2 3
7384 117/300 Read block 1207 len 8192
7385 118/300 Write block 3518 len 16384 data: 3 4 5 4
7386 119/300 Write block 3724 len 28672 data: 3 2 3 3 3 3 4
7387 120/300 Read block 732 len 8192
7388 121/300 Write block 1909 len 28672 data: 2 2 2 2 2 2 2
7389 122/300 Read block 1261 len 20480
7390 123/300 Read block 3760 len 4096
7391 124/300 Write block 3778 len 16384 data: 4 4 4 4
7392 125/300 Write block 4139 len 12288 data: 3 2 2
7393 126/300 Flush
7394 127/300 Read block 4916 len 24576
7395 128/300 Read block 2109 len 40960
7396 129/300 Write block 1932 len 36864 data: 3 3 4 3 4 6 5 3 3
7397 130/300 Flush
7398 131/300 Flush
7399 132/300 Read block 2105 len 28672
7400 133/300 Read block 971 len 12288
7401 134/300 Write block 2253 len 40960 data: 2 2 2 2 2 2 3 4 2 2
7402 135/300 Read block 4065 len 36864
7403 136/300 Read block 920 len 12288
7404 137/300 Write block 4042 len 8192 data: 4 4
7405 138/300 Write block 2324 len 4096 data: 3
7406 139/300 Write block 4797 len 36864 data: 4 2 2 5 5 5 6 5 4
7407 140/300 Flush
7408 141/300 Write block 4604 len 12288 data: 2 2 3
7409 142/300 Read block 4295 len 32768
7410 143/300 Read block 266 len 28672
7411 144/300 Read block 2121 len 4096
7412 145/300 Read block 2390 len 20480
7413 146/300 Read block 2596 len 20480
7414 147/300 Read block 2563 len 40960
7415 148/300 Read block 999 len 32768
7416 149/300 Write block 1713 len 16384 data: 3 2 2 2
7417 150/300 Write block 3459 len 28672 data: 5 6 4 4 3 2 2
7418 151/300 Read block 1496 len 32768
7419 152/300 Write block 1874 len 32768 data: 2 3 4 4 5 4 4 3
7420 153/300 Read block 2499 len 4096
7421 154/300 Read block 4429 len 32768
7422 155/300 Write block 5 len 20480 data: 2 4 4 4 4
7423 156/300 Read block 2419 len 32768
7424 157/300 Write block 4690 len 20480 data: 2 2 2 2 4
7425 158/300 Read block 3063 len 36864
7426 159/300 Read block 587 len 12288
7427 160/300 Read block 2138 len 12288
7428 161/300 Write block 1603 len 20480 data: 2 3 3 3 3
7429 162/300 Write block 3991 len 28672 data: 4 4 2 2 2 2 3
7430 163/300 Read block 2666 len 8192
7431 164/300 Read block 4776 len 24576
7432 165/300 Read block 2545 len 32768
7433 166/300 Write block 2549 len 20480 data: 3 3 3 3 5
7434 167/300 Read block 4594 len 8192
7435 168/300 Read block 3165 len 40960
7436 169/300 Write block 2815 len 4096 data: 3
7437 170/300 Write block 975 len 16384 data: 2 2 2 2
7438 171/300 Read block 1636 len 12288
7439 172/300 Flush
7440 173/300 Read block 1649 len 32768
7441 174/300 Write block 2536 len 20480 data: 3 3 3 4 3
7442 175/300 Write block 466 len 32768 data: 2 2 2 2 2 2 2 2
7443 176/300 Write block 2096 len 20480 data: 2 2 2 2 2
7444 177/300 Flush
7445 178/300 Read block 2132 len 4096
7446 179/300 Read block 1494 len 28672
7447 180/300 Flush
7448 181/300 Read block 4487 len 24576
7449 182/300 Write block 2015 len 40960 data: 4 3 3 3 3 3 3 3 3 3
7450 183/300 Read block 2824 len 28672
7451 184/300 Write block 2420 len 24576 data: 2 2 2 2 3 3
7452 185/300 Read block 3696 len 8192
7453 186/300 Write block 3216 len 8192 data: 3 3
7454 187/300 Read block 2676 len 20480
7455 188/300 Flush
7456 189/300 Read block 162 len 20480
7457 190/300 Read block 3357 len 24576
7458 191/300 Read block 1762 len 32768
7459 192/300 Read block 291 len 12288
7460 193/300 Write block 3330 len 36864 data: 5 4 4 4 3 3 3 3 5
7461 194/300 Read block 3017 len 28672
7462 195/300 Write block 2547 len 32768 data: 2 3 4 4 4 4 6 5
7463 196/300 Write block 3425 len 24576 data: 4 4 3 4 5 6
7464 197/300 Read block 4809 len 16384
7465 198/300 Read block 3924 len 12288
7466 199/300 Read block 3271 len 40960
7467 200/300 Flush
7468 201/300 Flush
7469 202/300 Read block 3692 len 28672
7470 203/300 Read block 4948 len 4096
7471 204/300 Read block 4018 len 12288
7472 205/300 Read block 2226 len 32768
7473 206/300 Read block 133 len 40960
7474 207/300 Read block 3208 len 12288
7475 208/300 Write block 2378 len 12288 data: 2 2 4
7476 209/300 Flush
7477 210/300 Read block 1620 len 32768
7478 211/300 Flush
7479 212/300 Read block 2422 len 28672
7480 213/300 Read block 4934 len 12288
7481 214/300 Write block 830 len 8192 data: 4 2
7482 215/300 Read block 2190 len 4096
7483 216/300 Read block 1296 len 16384
7484 217/300 Read block 272 len 36864
7485 218/300 Read block 1863 len 20480
7486 219/300 Read block 1900 len 36864
7487 220/300 Read block 390 len 24576
7488 221/300 Flush
7489 222/300 Read block 3427 len 16384
7490 223/300 Flush
7491 224/300 Write block 2422 len 16384 data: 3 3 4 4
7492 225/300 Read block 4513 len 36864
7493 226/300 Read block 655 len 16384
7494 227/300 Flush
7495 228/300 Read block 2488 len 16384
7496 229/300 Read block 2535 len 32768
7497 230/300 Flush
7498 231/300 Flush
7499 232/300 Read block 3474 len 36864
7500 233/300 Write block 4064 len 36864 data: 4 3 3 3 3 3 3 3 2
7501 234/300 Read block 4801 len 32768
7502 235/300 Read block 3597 len 20480
7503 236/300 Read block 3542 len 20480
7504 237/300 Read block 441 len 12288
7505 238/300 Write block 1157 len 20480 data: 4 4 4 5 4
7506 239/300 Read block 2039 len 8192
7507 240/300 Write block 2546 len 16384 data: 2 3 4 5
7508 241/300 Read block 4871 len 8192
7509 242/300 Read block 322 len 32768
7510 243/300 Write block 1525 len 40960 data: 3 3 3 3 2 3 3 3 2 2
7511 244/300 Write block 4172 len 36864 data: 3 2 2 2 3 3 3 2 4
7512 245/300 Write block 3648 len 36864 data: 3 3 3 3 3 3 2 2 2
7513 246/300 Write block 2085 len 4096 data: 2
7514 247/300 Read block 4218 len 16384
7515 248/300 Flush
7516 249/300 Flush
7517 250/300 Write block 372 len 16384 data: 3 4 5 5
7518 251/300 Read block 2336 len 16384
7519 252/300 Write block 1001 len 36864 data: 2 2 2 2 3 3 3 4 4
7520 253/300 Read block 1918 len 4096
7521 254/300 Flush
7522 255/300 Read block 1776 len 32768
7523 256/300 Read block 4248 len 8192
7524 257/300 Write block 2812 len 12288 data: 3 3 3
7525 258/300 Read block 1116 len 36864
7526 259/300 Read block 2775 len 24576
7527 260/300 Read block 4270 len 32768
7528 261/300 Write block 1807 len 28672 data: 3 3 6 5 5 6 6
7529 262/300 Read block 4482 len 4096
7530 263/300 Read block 2208 len 24576
7531 264/300 Read block 4905 len 24576
7532 265/300 Read block 3814 len 28672
7533 266/300 Write block 4933 len 36864 data: 4 4 4 4 3 3 4 4 3
7534 267/300 Read block 1005 len 28672
7535 268/300 Read block 4289 len 40960
7536 269/300 Read block 2187 len 24576
7537 270/300 Read block 4541 len 20480
7538 271/300 Read block 60 len 36864
7539 272/300 Write block 3703 len 40960 data: 2 4 4 5 5 6 6 7 8 7
7540 273/300 Write block 2562 len 16384 data: 2 2 2 2
7541 274/300 Flush
7542 275/300 Flush
7543 276/300 Write block 1960 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7544 277/300 Read block 1156 len 8192
7545 278/300 Flush
7546 279/300 Read block 988 len 36864
7547 280/300 Write block 1866 len 32768 data: 2 2 2 2 2 2 2 2
7548 281/300 Read block 4154 len 28672
7549 282/300 Write block 4569 len 20480 data: 3 3 3 3 3
7550 283/300 Read block 3119 len 28672
7551 284/300 Read block 201 len 24576
7552 285/300 Write block 2882 len 24576 data: 2 3 3 5 5 4
7553 286/300 Write block 4453 len 16384 data: 3 3 3 3
7554 287/300 Write block 1311 len 8192 data: 4 3
7555 288/300 Read block 3571 len 20480
7556 289/300 Write block 4238 len 24576 data: 2 2 2 3 5 4
7557 290/300 Read block 2013 len 40960
7558 291/300 Read block 113 len 24576
7559 292/300 Read block 334 len 4096
7560 293/300 Write block 3733 len 4096 data: 4
7561 294/300 Write block 1618 len 12288 data: 3 3 3
7562 295/300 Read block 2637 len 28672
7563 296/300 Read block 3828 len 20480
7564 297/300 Read block 2326 len 8192
7565 298/300 Read block 3731 len 28672
7566 299/300 Read block 267 len 32768
7567 300/300 Read block 3908 len 28672
7568 Main task has work to do, go find it
7569 got dsc Start(2)
7570 start 2
7571 Oct 13 18:10:44.878 INFO request completed, latency_us: 300, response_code: 204, uri: /start/cid/2, method: POST, req_id: eeca4ff4-dbf4-434c-b581-ff7a42666cd6, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
7572 Replay: started 2, returned:Ok(())
7573 ----------------------------------------------------------------
7574 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
7575 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7576 2725 Acked 3724 Read 4 Done Done New false
7577 2726 Acked 3725 Read 9 Done Done New false
7578 2727 Acked 3726 Read 9 Done Done New false
7579 2728 Acked 3727 Read 1 Done Done New false
7580 2729 Acked 3728 Read 6 Done Done New false
7581 2730 Acked 3729 Read 9 Done Done New false
7582 2731 Acked 3730 Write 4 Done Done New false
7583 2732 Acked 3731 Write 2 Done Done New false
7584 2733 Acked 3732 Write 7 Done Done New false
7585 2734 Acked 3733 Read 9 Done Done New false
7586 2735 Acked 3734 Write 1 Done Done New false
7587 2736 Acked 3735 Write 5 Done Done New false
7588 2737 Acked 3736 Read 2 Done Done New false
7589 2738 Acked 3737 Write 1 Done Done New false
7590 2739 Acked 3738 Write 10 Done Done New false
7591 2740 Acked 3739 Write 3 Done Done New false
7592 2741 Acked 3740 Read 4 Done Done New false
7593 2742 Acked 3741 Write 6 Done Done New false
7594 2743 Acked 3742 Read 5 Done Done New false
7595 2744 Acked 3743 Read 5 Done Done New false
7596 2745 Acked 3744 Write 1 Done Done New false
7597 2746 Acked 3745 Read 4 Done Done New false
7598 2747 Acked 3746 Write 7 Done Done New false
7599 2748 Acked 3747 Read 3 Done Done New false
7600 2749 Acked 3748 Flush 0 Done Done New false
7601 2750 Acked 3749 Write 10 Done Done New false
7602 2751 Acked 3750 Write 9 Done Done New false
7603 2752 Acked 3751 Read 7 Done Done New false
7604 2753 Acked 3752 Read 5 Done Done New false
7605 2754 Acked 3753 Write 10 Done Done New false
7606 2755 Acked 3754 Read 4 Done Done New false
7607 2756 Acked 3755 Read 10 Done Done New false
7608 2757 Acked 3756 Read 2 Done Done New false
7609 2758 Acked 3757 Write 9 Done Done New false
7610 2759 Acked 3758 Write 9 Done Done New false
7611 2760 Acked 3759 Read 3 Done Done New false
7612 2761 Acked 3760 Write 5 Done Done New false
7613 2762 Acked 3761 Read 2 Done Done New false
7614 2763 Acked 3762 Write 2 Done Done New false
7615 2764 Acked 3763 Read 8 Done Done New false
7616 2765 Acked 3764 Read 2 Done Done New false
7617 2766 Acked 3765 Read 9 Done Done New false
7618 2767 Acked 3766 Read 3 Done Done New false
7619 2768 Acked 3767 Flush 0 Done Done New false
7620 2769 Acked 3768 Read 8 Done Done New false
7621 2770 Acked 3769 Write 1 Done Done New false
7622 2771 Acked 3770 Write 3 Done Done New false
7623 2772 Acked 3771 Read 5 Done Done New false
7624 2773 Acked 3772 Read 7 Done Done New false
7625 2774 Acked 3773 Read 10 Done Done New false
7626 2775 Acked 3774 Read 4 Done Done New false
7627 2776 Acked 3775 Read 2 Done Done New false
7628 2777 Acked 3776 Write 1 Done Done New false
7629 2778 Acked 3777 Read 6 Done Done New false
7630 2779 Acked 3778 Write 5 Done Done New false
7631 2780 Acked 3779 Read 5 Done Done New false
7632 2781 Acked 3780 Read 6 Done Done New false
7633 2782 Acked 3781 Write 10 Done Done New false
7634 2783 Acked 3782 Flush 0 Done Done New false
7635 2784 Acked 3783 Read 3 Done Done New false
7636 2785 Acked 3784 Write 9 Done Done New false
7637 2786 Acked 3785 Read 1 Done Done New false
7638 2787 Acked 3786 Write 4 Done Done New false
7639 2788 Acked 3787 Write 6 Done Done New false
7640 2789 Acked 3788 Flush 0 Done Done New false
7641 2790 Acked 3789 Read 3 Done Done New false
7642 2791 Acked 3790 Read 1 Done Done New false
7643 2792 Acked 3791 Write 9 Done Done New false
7644 2793 Acked 3792 Read 6 Done Done New false
7645 2794 Acked 3793 Flush 0 Done Done New false
7646 2795 Acked 3794 Write 10 Done Done New false
7647 2796 Acked 3795 Read 1 Done Done New false
7648 2797 Acked 3796 Write 2 Done Done New false
7649 2798 Acked 3797 Read 9 Done Done New false
7650 2799 Acked 3798 Write 2 Done Done New false
7651 2800 Acked 3799 Read 8 Done Done New false
7652 2801 Acked 3800 Read 7 Done Done New false
7653 2802 Acked 3801 Read 9 Done Done New false
7654 2803 Acked 3802 Write 7 Done Done New false
7655 2804 Acked 3803 Write 5 Done Done New false
7656 2805 Acked 3804 Read 2 Done Done New false
7657 2806 Acked 3805 Read 4 Done Done New false
7658 2807 Acked 3806 Read 2 Done Done New false
7659 2808 Acked 3807 Write 1 Done Done New false
7660 2809 Acked 3808 Read 5 Done Done New false
7661 2810 Acked 3809 Read 5 Done Done New false
7662 2811 Acked 3810 Write 10 Done Done New false
7663 2812 Acked 3811 Write 4 Done Done New false
7664 2813 Acked 3812 Write 10 Done Done New false
7665 2814 Acked 3813 Write 1 Done Done New false
7666 2815 Acked 3814 Write 5 Done Done New false
7667 2816 Acked 3815 Read 8 Done Done New false
7668 2817 Acked 3816 Read 9 Done Done New false
7669 2818 Acked 3817 Read 7 Done Done New false
7670 2819 Acked 3818 Read 9 Done Done New false
7671 2820 Acked 3819 Read 4 Done Done New false
7672 2821 Acked 3820 Write 7 Done Done New false
7673 2822 Acked 3821 Read 1 Done Done New false
7674 2823 Acked 3822 Read 10 Done Done New false
7675 2824 Acked 3823 Read 5 Done Done New false
7676 2825 Acked 3824 Read 4 Done Done New false
7677 2826 Acked 3825 Write 7 Done Done New false
7678 2827 Acked 3826 Write 7 Done Done New false
7679 2828 Acked 3827 Read 9 Done Done New false
7680 2829 Acked 3828 Flush 0 Done Done New false
7681 2830 Acked 3829 Flush 0 Done Done New false
7682 2831 Acked 3830 Flush 0 Done Done New false
7683 2832 Acked 3831 Write 5 Done Done New false
7684 2833 Acked 3832 Read 7 Done Done New false
7685 2834 Acked 3833 Read 5 Done Done New false
7686 2835 Acked 3834 Write 4 Done Done New false
7687 2836 Acked 3835 Write 3 Done Done New false
7688 2837 Acked 3836 Read 5 Done Done New false
7689 2838 Acked 3837 Read 8 Done Done New false
7690 2839 Acked 3838 Read 8 Done Done New false
7691 2840 Acked 3839 Read 4 Done Done New false
7692 2841 Acked 3840 Write 3 Done Done New false
7693 2842 Acked 3841 Read 2 Done Done New false
7694 2843 Acked 3842 Write 4 Done Done New false
7695 2844 Acked 3843 Write 7 Done Done New false
7696 2845 Acked 3844 Read 2 Done Done New false
7697 2846 Acked 3845 Write 7 Done Done New false
7698 2847 Acked 3846 Read 5 Done Done New false
7699 2848 Acked 3847 Read 1 Done Done New false
7700 2849 Acked 3848 Write 4 Done Done New false
7701 2850 Acked 3849 Write 3 Done Done New false
7702 2851 Acked 3850 Flush 0 Done Done New false
7703 2852 Acked 3851 Read 6 Done Done New false
7704 2853 Acked 3852 Read 10 Done Done New false
7705 2854 Acked 3853 Write 9 Done Done New false
7706 2855 Acked 3854 Flush 0 Done Done New false
7707 2856 Acked 3855 Flush 0 Done Done New false
7708 2857 Acked 3856 Read 7 Done Done New false
7709 2858 Acked 3857 Read 3 Done Done New false
7710 2859 Acked 3858 Write 10 Done Done New false
7711 2860 Acked 3859 Read 9 Done Done New false
7712 2861 Acked 3860 Read 3 Done Done New false
7713 2862 Acked 3861 Write 2 Done Done New false
7714 2863 Acked 3862 Write 1 Done Done New false
7715 2864 Acked 3863 Write 9 Done Done New false
7716 2865 Acked 3864 Flush 0 Done Done New false
7717 2866 Acked 3865 Write 3 Done Done New false
7718 2867 Acked 3866 Read 8 Done Done New false
7719 2868 Acked 3867 Read 7 Done Done New false
7720 2869 Acked 3868 Read 1 Done Done New false
7721 2870 Acked 3869 Read 5 Done Done New false
7722 2871 Acked 3870 Read 5 Done Done New false
7723 2872 Acked 3871 Read 10 Done Done New false
7724 2873 Acked 3872 Read 8 Done Done New false
7725 2874 Acked 3873 Write 4 Done Done New false
7726 2875 Acked 3874 Write 7 Done Done New false
7727 2876 Acked 3875 Read 8 Done Done New false
7728 2877 Acked 3876 Write 8 Done Done New false
7729 2878 Acked 3877 Read 1 Done Done New false
7730 2879 Acked 3878 Read 8 Done Done New false
7731 2880 Acked 3879 Write 5 Done Done New false
7732 2881 Acked 3880 Read 8 Done Done New false
7733 2882 Acked 3881 Write 5 Done Done New false
7734 2883 Acked 3882 Read 9 Done Done New false
7735 2884 Acked 3883 Read 3 Done Done New false
7736 2885 Acked 3884 Read 3 Done Done New false
7737 2886 Acked 3885 Write 5 Done Done New false
7738 2887 Acked 3886 Write 7 Done Done New false
7739 2888 Acked 3887 Read 2 Done Done New false
7740 2889 Acked 3888 Read 6 Done Done New false
7741 2890 Acked 3889 Read 8 Done Done New false
7742 2891 Acked 3890 Write 5 Done Done New false
7743 2892 Acked 3891 Read 2 Done Done New false
7744 2893 Acked 3892 Read 10 Done Done New false
7745 2894 Acked 3893 Write 1 Done Done New false
7746 2895 Acked 3894 Write 4 Done Done New false
7747 2896 Acked 3895 Read 3 Done Done New false
7748 2897 Acked 3896 Flush 0 Done Done New false
7749 2898 Acked 3897 Read 8 Done Done New false
7750 2899 Acked 3898 Write 5 Done Done New false
7751 2900 Acked 3899 Write 8 Done Done New false
7752 2901 Acked 3900 Write 5 Done Done New false
7753 2902 Acked 3901 Flush 0 Done Done New false
7754 2903 Acked 3902 Read 1 Done Done New false
7755 2904 Acked 3903 Read 7 Done Done New false
7756 2905 Acked 3904 Flush 0 Done Done New false
7757 2906 Acked 3905 Read 6 Done Done New false
7758 2907 Acked 3906 Write 10 Done Done New false
7759 2908 Acked 3907 Read 7 Done Done New false
7760 2909 Acked 3908 Write 6 Done Done New false
7761 2910 Acked 3909 Read 2 Done Done New false
7762 2911 Acked 3910 Write 2 Done Done New false
7763 2912 Acked 3911 Read 5 Done Done New false
7764 2913 Acked 3912 Flush 0 Done Done New false
7765 2914 Acked 3913 Read 5 Done Done New false
7766 2915 Acked 3914 Read 6 Done Done New false
7767 2916 Acked 3915 Read 8 Done Done New false
7768 2917 Acked 3916 Read 3 Done Done New false
7769 2918 Acked 3917 Write 9 Done Done New false
7770 2919 Acked 3918 Read 7 Done Done New false
7771 2920 Acked 3919 Write 8 Done Done New false
7772 2921 Acked 3920 Write 6 Done Done New false
7773 2922 Acked 3921 Read 4 Done Done New false
7774 2923 Acked 3922 Read 3 Done Done New false
7775 2924 Acked 3923 Read 10 Done Done New false
7776 2925 Acked 3924 Flush 0 Done Done New false
7777 2926 Acked 3925 Flush 0 Done Done New false
7778 2927 Acked 3926 Read 7 Done Done New false
7779 2928 Acked 3927 Read 1 Done Done New false
7780 2929 Acked 3928 Read 3 Done Done New false
7781 2930 Acked 3929 Read 8 Done Done New false
7782 2931 Acked 3930 Read 10 Done Done New false
7783 2932 Acked 3931 Read 3 Done Done New false
7784 2933 Acked 3932 Write 3 Done Done New false
7785 2934 Acked 3933 Flush 0 Done Done New false
7786 2935 Acked 3934 Read 8 Done Done New false
7787 2936 Acked 3935 Flush 0 Done Done New false
7788 2937 Acked 3936 Read 7 Done Done New false
7789 2938 Acked 3937 Read 3 Done Done New false
7790 2939 Acked 3938 Write 2 Done Done New false
7791 2940 Acked 3939 Read 1 Done Done New false
7792 2941 Acked 3940 Read 4 Done Done New false
7793 2942 Acked 3941 Read 9 Done Done New false
7794 2943 Acked 3942 Read 5 Done Done New false
7795 2944 Acked 3943 Read 9 Done Done New false
7796 2945 Acked 3944 Read 6 Done Done New false
7797 2946 Acked 3945 Flush 0 Done Done New false
7798 2947 Acked 3946 Read 4 Done Done New false
7799 2948 Acked 3947 Flush 0 Done Done New false
7800 2949 Acked 3948 Write 4 Done Done New false
7801 2950 Acked 3949 Read 9 Done Done New false
7802 2951 Acked 3950 Read 4 Done Done New false
7803 2952 Acked 3951 Flush 0 Done Done New false
7804 2953 Acked 3952 Read 4 Done Done New false
7805 2954 Acked 3953 Read 8 Done Done New false
7806 2955 Acked 3954 Flush 0 Done Done New false
7807 2956 Acked 3955 Flush 0 Done Done New false
7808 2957 Acked 3956 Read 9 Done Done New false
7809 2958 Acked 3957 Write 9 Done Done New false
7810 2959 Acked 3958 Read 8 Done Done New false
7811 2960 Acked 3959 Read 5 Done Done New false
7812 2961 Acked 3960 Read 5 Done Done New false
7813 2962 Acked 3961 Read 3 Done Done New false
7814 2963 Acked 3962 Write 5 Done Done New false
7815 2964 Acked 3963 Read 2 Done Done New false
7816 2965 Acked 3964 Write 4 Done Done New false
7817 2966 Acked 3965 Read 2 Done Done New false
7818 2967 Acked 3966 Read 8 Done Done New false
7819 2968 Acked 3967 Write 10 Done Done New false
7820 2969 Acked 3968 Write 9 Done Done New false
7821 2970 Acked 3969 Write 9 Done Done New false
7822 2971 Acked 3970 Write 1 Done Done New false
7823 2972 Acked 3971 Read 4 Done Done New false
7824 2973 Acked 3972 Flush 0 Done Done New false
7825 2974 Acked 3973 Flush 0 Done Done New false
7826 2975 Acked 3974 Write 4 Done Done New false
7827 2976 Acked 3975 Read 4 Done Done New false
7828 2977 Acked 3976 Write 9 Done Done New false
7829 2978 Acked 3977 Read 1 Done Done New false
7830 2979 Acked 3978 Flush 0 Done Done New false
7831 2980 Acked 3979 Read 8 Done Done New false
7832 2981 Acked 3980 Read 2 Done Done New false
7833 2982 Acked 3981 Write 3 Done Done New false
7834 2983 Acked 3982 Read 9 Done Done New false
7835 2984 Acked 3983 Read 6 Done Done New false
7836 2985 Acked 3984 Read 8 Done Done New false
7837 2986 Acked 3985 Write 7 Done Done New false
7838 2987 Acked 3986 Read 1 Done Done New false
7839 2988 Acked 3987 Flush 0 Done Done New false
7840 2989 Acked 3988 Read 6 Done Done New false
7841 2990 Acked 3989 Read 6 Done Done New false
7842 2991 Acked 3990 Read 7 Done Done New false
7843 2992 Acked 3991 Write 9 Done Done New false
7844 2993 Acked 3992 Read 7 Done Done New false
7845 2994 Acked 3993 Read 10 Done Done New false
7846 2995 Acked 3994 Read 6 Done Done New false
7847 2996 Acked 3995 Read 5 Done Done New false
7848 2997 Acked 3996 Read 9 Done Done New false
7849 2998 Acked 3997 Write 10 Done Done New false
7850 2999 Acked 3998 Write 4 Done Done New false
7851 3000 Acked 3999 Flush 0 Done Done New false
7852 3001 Acked 4000 Flush 0 Done Done New false
7853 3002 Acked 4001 Write 10 Done Done New false
7854 3003 Acked 4002 Read 2 Done Done New false
7855 3004 Acked 4003 Flush 0 Done Done New false
7856 3005 Acked 4004 Read 9 Done Done New false
7857 3006 Acked 4005 Write 8 Done Done New false
7858 3007 Acked 4006 Read 7 Done Done New false
7859 3008 Acked 4007 Write 5 Done Done New false
7860 3009 Acked 4008 Read 7 Done Done New false
7861 3010 Acked 4009 Read 6 Done Done New false
7862 3011 Acked 4010 Write 6 Done Done New false
7863 3012 Acked 4011 Write 4 Done Done New false
7864 3013 Acked 4012 Write 2 Done Done New false
7865 3014 Acked 4013 Read 5 Done Done New false
7866 3015 Acked 4014 Write 6 Done Done New false
7867 3016 Acked 4015 Read 10 Done Done New false
7868 3017 Acked 4016 Read 6 Done Done New false
7869 3018 Acked 4017 Read 1 Done Done New false
7870 3019 Acked 4018 Write 1 Done Done New false
7871 3020 Acked 4019 Write 3 Done Done New false
7872 3021 Acked 4020 Read 7 Done Done New false
7873 3022 Acked 4021 Read 5 Done Done New false
7874 3023 Acked 4022 Read 2 Done Done New false
7875 3024 Acked 4023 Read 7 Done Done New false
7876 3025 Acked 4024 Read 8 Done Done New false
7877 3026 Acked 4025 Read 7 Done Done New false
7878 STATES DS:0 DS:1 DS:2 TOTAL
7879 New 0 0 302 302
7880 Sent 0 0 0 0
7881 Done 302 302 0 604
7882 Skipped 0 0 0 0
7883 Error 0 0 0 0
7884 Last Flush: 4003 4003 3723
7885 Downstairs last five completed: 3723 3722 3721 3720 3719 3718
7886 Upstairs last five completed: 3026 3025 3024 3023 3022 3021
7887 CLIENT: Up:0 ds:302 act:2
7888 [8830] got start action
7889 [8830] I am going to restart
7890 Starting downstairs at port 8830
7891 Make output file at "/tmp/dsc/downstairs-8830.txt"
7892 [8830][2] reports Starting
7893 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1186)
7894 [8830][2] reports Running
78952023-10-13T18:10:46.931ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 2
78962023-10-13T18:10:46.931ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
78972023-10-13T18:10:46.931ZINFOcrucible: [2] client re-new 303 jobs since flush 3723 = downstairs
78982023-10-13T18:10:46.932ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
78992023-10-13T18:10:46.933ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
79002023-10-13T18:10:46.933ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
79012023-10-13T18:10:46.933ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
79022023-10-13T18:10:46.933ZINFOcrucible: [2] send last flush ID to this DS: 3723
79032023-10-13T18:10:46.933ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
79042023-10-13T18:10:46.933ZINFOcrucible: [2] Replied this last flush ID: 3723
79052023-10-13T18:10:46.933ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Offline ds_transition to Replay
79062023-10-13T18:10:46.933ZINFOcrucible: [2] Transition from Offline to Replay
79072023-10-13T18:10:46.933ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
79082023-10-13T18:10:46.933ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Replay ds_transition to Active
79092023-10-13T18:10:46.933ZINFOcrucible: [2] Transition from Replay to Active
79102023-10-13T18:10:46.933ZINFOcrucible: [2] Starts cmd_loop
79112023-10-13T18:10:46.933ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
79122023-10-13T18:10:46.933ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
79132023-10-13T18:10:46.933ZINFOcrucible: Set check for repair
79142023-10-13T18:10:47.934ZINFOcrucible: Checking if live repair is needed
79152023-10-13T18:10:47.934ZWARNcrucible: [2] flow control end
79162023-10-13T18:10:47.934ZINFOcrucible: No Live Repair required at this time
7917 ----------------------------------------------------------------
7918 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7919 Downstairs last five completed: 4026 4025 4024 4023 4022 4021
7920 Upstairs last five completed: 3027 3026 3025 3024 3023 3022
7921 CLIENT: Up:0 ds:0 act:3
7922 Replay: All jobs finished, all DS active.
7923 Main task has work to do, go find it
7924 got dsc Stop(2)
7925 stop 2
7926 Oct 13 18:10:48.883 INFO request completed, latency_us: 541, response_code:[8830] Got stop action so:false kr:false
7927 204, uri: /stop/cid/2, method: POST, req_id: 3bf732df-3511-40da-a5bc-456dad1ae7ef, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
7928 Oct 13 18:10:48.885 INFO request completed, latency_us: 182, response_code: 200, uri: /state/cid/2, method: GET, req_id: 71abbd4d-4ba1-43e5-9294-44337d8d532d, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
79292023-10-13T18:10:48.885ZWARNcrucible: [2] downstairs disconnected
79302023-10-13T18:10:48.885ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
79312023-10-13T18:10:48.885ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8830 closed looper = 2
79322023-10-13T18:10:48.885ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
79332023-10-13T18:10:48.885ZWARNcrucible: [2] pm_task rx.recv() is None
79342023-10-13T18:10:48.885ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
79352023-10-13T18:10:48.885ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
7936 [8830] Exited with: ExitStatus(unix_wait_status(9))
7937 [8830][2] reports Exit
7938 Oct 13 18:10:52.888 INFO request completed, latency_us: 262, response_code: 200, uri: /state/cid/2, method: GET, req_id: c262ea0a-9fd2-4f7a-bb0b-82ee4401fd81, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
7939 001/300 Read block 3941 len 28672
7940 002/300 Read block 1423 len 28672
7941 003/300 Write block 2733 len 16384 data: 5 4 4 4
7942 004/300 Read block 4283 len 32768
7943 005/300 Read block 4826 len 16384
7944 006/300 Read block 2249 len 40960
7945 007/300 Write block 2406 len 40960 data: 3 3 3 3 3 4 4 4 3 3
7946 008/300 Read block 1316 len 24576
7947 009/300 Read block 3184 len 24576
7948 010/300 Read block 3851 len 12288
7949 011/300 Write block 2897 len 12288 data: 2 2 4
7950 012/300 Read block 779 len 24576
7951 013/300 Read block 469 len 36864
7952 014/300 Write block 6 len 4096 data: 5
7953 015/300 Flush
7954 016/300 Read block 4590 len 32768
7955 017/300 Write block 48 len 4096 data: 6
7956 018/300 Read block 1742 len 32768
7957 019/300 Read block 3174 len 40960
7958 020/300 Read block 4830 len 4096
7959 021/300 Flush
7960 022/300 Flush
7961 023/300 Flush
7962 024/300 Flush
7963 025/300 Write block 4890 len 24576 data: 4 2 3 2 2 2
7964 026/300 Write block 639 len 36864 data: 2 2 3 4 4 3 3 2 2
7965 027/300 Read block 1518 len 12288
7966 028/300 Write block 2928 len 32768 data: 2 2 2 2 2 2 2 3
7967 029/300 Write block 3379 len 28672 data: 2 2 2 2 3 3 3
7968 030/300 Write block 1902 len 32768 data: 2 2 2 2 2 2 2 3
7969 031/300 Write block 2327 len 24576 data: 4 4 5 5 5 5
7970 032/300 Write block 1668 len 12288 data: 4 3 4
7971 033/300 Write block 3710 len 32768 data: 8 9 8 6 6 7 5 5
7972 034/300 Read block 607 len 20480
7973 035/300 Read block 4621 len 40960
7974 036/300 Write block 3878 len 8192 data: 4 4
7975 037/300 Read block 2226 len 40960
7976 038/300 Read block 2739 len 36864
7977 039/300 Read block 792 len 40960
7978 040/300 Write block 4193 len 28672 data: 4 3 3 4 4 2 2
7979 041/300 Write block 3660 len 8192 data: 3 3
7980 042/300 Read block 655 len 28672
7981 043/300 Write block 3713 len 32768 data: 7 7 8 6 6 5 4 4
7982 044/300 Flush
7983 045/300 Read block 4800 len 16384
7984 046/300 Read block 3051 len 16384
7985 047/300 Write block 358 len 36864 data: 2 2 2 2 3 3 3 3 3
7986 048/300 Read block 357 len 8192
7987 049/300 Write block 2387 len 20480 data: 2 3 3 3 3
7988 050/300 Flush
7989 051/300 Read block 1002 len 8192
7990 052/300 Read block 53 len 20480
7991 053/300 Read block 4035 len 20480
7992 054/300 Write block 350 len 36864 data: 2 2 2 2 2 2 2 2 3
7993 055/300 Write block 2014 len 16384 data: 2 5 4 4
7994 056/300 Read block 645 len 8192
7995 057/300 Read block 4733 len 24576
7996 058/300 Read block 584 len 28672
7997 059/300 Read block 1691 len 4096
7998 060/300 Read block 1361 len 20480
7999 061/300 Write block 47 len 16384 data: 5 7 5 5
8000 062/300 Write block 162 len 20480 data: 3 3 3 3 3
8001 063/300 Read block 2982 len 4096
8002 064/300 Read block 3725 len 24576
8003 065/300 Read block 351 len 36864
8004 066/300 Read block 381 len 4096
8005 067/300 Read block 545 len 36864
8006 068/300 Write block 4326 len 4096 data: 5
8007 069/300 Read block 1607 len 28672
8008 070/300 Write block 4545 len 8192 data: 4 3
8009 071/300 Read block 1123 len 8192
8010 072/300 Write block 3765 len 32768 data: 3 2 2 2 3 3 3 2
8011 073/300 Read block 2722 len 16384
8012 074/300 Read block 1087 len 12288
8013 075/300 Write block 4993 len 8192 data: 2 2
8014 076/300 Write block 3141 len 40960 data: 4 4 5 4 5 5 5 3 3 3
8015 077/300 Read block 1059 len 24576
8016 078/300 Write block 2448 len 40960 data: 3 3 3 3 2 2 2 2 2 2
8017 079/300 Flush
8018 080/300 Write block 1005 len 16384 data: 4 4 4 5
8019 081/300 Read block 2161 len 20480
8020 082/300 Read block 1067 len 8192
8021 083/300 Read block 898 len 8192
8022 084/300 Write block 1363 len 36864 data: 2 2 2 3 4 4 4 4 4
8023 085/300 Flush
8024 086/300 Read block 2508 len 32768
8025 087/300 Read block 1122 len 8192
8026 088/300 Read block 2903 len 20480
8027 089/300 Read block 996 len 36864
8028 090/300 Write block 1193 len 32768 data: 2 2 2 3 4 5 4 4
8029 091/300 Write block 1940 len 8192 data: 4 3
8030 092/300 Read block 325 len 28672
8031 093/300 Read block 625 len 32768
8032 094/300 Read block 3859 len 40960
8033 095/300 Read block 514 len 24576
8034 096/300 Flush
8035 097/300 Flush
8036 098/300 Write block 2701 len 16384 data: 4 3 3 2
8037 099/300 Read block 4878 len 24576
8038 100/300 Write block 3092 len 24576 data: 3 4 4 6 6 5
8039 101/300 Read block 1558 len 24576
8040 102/300 Read block 469 len 40960
8041 103/300 Read block 144 len 40960
8042 104/300 Write block 400 len 4096 data: 2
8043 105/300 Read block 2291 len 40960
8044 106/300 Read block 2063 len 36864
8045 107/300 Read block 1957 len 28672
8046 108/300 Write block 4883 len 16384 data: 5 5 4 4
8047 109/300 Read block 3437 len 20480
8048 110/300 Flush
8049 111/300 Write block 3042 len 36864 data: 5 4 4 4 3 4 3 4 4
8050 112/300 Read block 3171 len 12288
8051 113/300 Read block 3339 len 24576
8052 114/300 Flush
8053 115/300 Write block 4750 len 36864 data: 4 4 3 4 4 6 6 6 4
8054 116/300 Read block 2317 len 8192
8055 117/300 Write block 1871 len 28672 data: 3 3 3 3 4 5 5
8056 118/300 Write block 4975 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8057 119/300 Read block 645 len 16384
8058 120/300 Write block 3876 len 28672 data: 5 4 5 5 4 4 3
8059 121/300 Read block 634 len 4096
8060 122/300 Read block 1006 len 40960
8061 123/300 Write block 1307 len 36864 data: 3 3 4 4 5 4 3 3 3
8062 124/300 Write block 280 len 12288 data: 2 2 3
8063 125/300 Read block 4780 len 12288
8064 126/300 Write block 833 len 12288 data: 3 2 3
8065 127/300 Read block 3019 len 20480
8066 128/300 Read block 3201 len 32768
8067 129/300 Read block 3847 len 40960
8068 130/300 Read block 3743 len 8192
8069 131/300 Write block 4006 len 4096 data: 3
8070 132/300 Write block 3617 len 36864 data: 3 3 4 5 3 3 3 3 3
8071 133/300 Read block 4461 len 4096
8072 134/300 Read block 1528 len 16384
8073 135/300 Write block 3872 len 32768 data: 4 3 5 5 6 5 6 6
8074 136/300 Read block 4675 len 16384
8075 137/300 Flush
8076 138/300 Read block 4865 len 40960
8077 139/300 Read block 1057 len 20480
8078 140/300 Read block 3102 len 8192
8079 141/300 Write block 1273 len 8192 data: 3 3
8080 142/300 Write block 2817 len 24576 data: 2 3 3 3 3 2
8081 143/300 Write block 118 len 32768 data: 3 3 3 3 2 2 2 2
8082 144/300 Read block 2230 len 40960
8083 145/300 Write block 992 len 16384 data: 2 2 2 2
8084 146/300 Read block 273 len 12288
8085 147/300 Read block 4104 len 16384
8086 148/300 Read block 3865 len 36864
8087 149/300 Read block 1340 len 36864
8088 150/300 Read block 2412 len 24576
8089 151/300 Read block 1837 len 32768
8090 152/300 Read block 1214 len 8192
8091 153/300 Read block 4970 len 36864
8092 154/300 Flush
8093 155/300 Read block 1184 len 4096
8094 156/300 Write block 4164 len 40960 data: 6 6 6 6 6 5 4 3 4 3
8095 157/300 Read block 184 len 24576
8096 158/300 Read block 719 len 20480
8097 159/300 Write block 446 len 28672 data: 3 2 2 2 2 2 2
8098 160/300 Write block 930 len 4096 data: 5
8099 161/300 Read block 1956 len 4096
8100 162/300 Read block 3579 len 32768
8101 163/300 Read block 3011 len 40960
8102 164/300 Read block 2009 len 36864
8103 165/300 Write block 176 len 28672 data: 5 5 5 5 4 3 3
8104 166/300 Write block 2397 len 28672 data: 6 7 7 6 4 4 4
8105 167/300 Read block 1427 len 8192
8106 168/300 Write block 625 len 24576 data: 2 2 2 3 3 3
8107 169/300 Read block 68 len 32768
8108 170/300 Read block 411 len 36864
8109 171/300 Write block 653 len 36864 data: 2 2 2 2 2 3 2 2 3
8110 172/300 Read block 22 len 8192
8111 173/300 Read block 2166 len 4096
8112 174/300 Write block 259 len 4096 data: 2
8113 175/300 Read block 638 len 12288
8114 176/300 Read block 4123 len 24576
8115 177/300 Write block 4822 len 20480 data: 4 4 4 3 3
8116 178/300 Read block 496 len 32768
8117 179/300 Flush
8118 180/300 Read block 2295 len 20480
8119 181/300 Write block 4787 len 40960 data: 2 2 2 2 3 3 3 4 4 4
8120 182/300 Read block 302 len 24576
8121 183/300 Read block 2409 len 32768
8122 184/300 Read block 835 len 12288
8123 185/300 Write block 2238 len 12288 data: 3 3 3
8124 186/300 Write block 969 len 36864 data: 2 2 2 2 2 2 3 3 3
8125 187/300 Write block 953 len 40960 data: 4 3 3 3 3 3 3 3 3 2
8126 188/300 Flush
8127 189/300 Write block 84 len 16384 data: 3 2 3 4
8128 190/300 Write block 4167 len 20480 data: 7 7 6 5 4
8129 191/300 Read block 3414 len 16384
8130 192/300 Read block 1324 len 4096
8131 193/300 Read block 1012 len 40960
8132 194/300 Write block 4438 len 16384 data: 2 2 2 2
8133 195/300 Read block 4267 len 8192
8134 196/300 Write block 4146 len 28672 data: 2 2 2 3 3 3 5
8135 197/300 Write block 4786 len 16384 data: 2 3 3 3
8136 198/300 Read block 4347 len 32768
8137 199/300 Write block 2244 len 36864 data: 4 3 2 3 3 2 2 2 2
8138 200/300 Read block 3903 len 8192
8139 201/300 Write block 1190 len 28672 data: 2 2 2 3 3 3 4
8140 202/300 Read block 1880 len 28672
8141 203/300 Write block 4579 len 16384 data: 3 4 4 3
8142 204/300 Write block 3575 len 4096 data: 4
8143 205/300 Flush
8144 206/300 Write block 3158 len 24576 data: 4 4 5 4 3 3
8145 207/300 Write block 4219 len 32768 data: 3 3 3 3 3 4 4 3
8146 208/300 Read block 1750 len 12288
8147 209/300 Write block 4726 len 16384 data: 4 4 4 4
8148 210/300 Write block 1982 len 4096 data: 4
8149 211/300 Read block 1363 len 12288
8150 212/300 Flush
8151 213/300 Read block 2086 len 28672
8152 214/300 Write block 1565 len 28672 data: 3 3 3 3 2 2 3
8153 215/300 Write block 206 len 36864 data: 2 2 3 3 3 4 4 4 4
8154 216/300 Read block 4758 len 8192
8155 217/300 Write block 2602 len 20480 data: 5 4 3 3 3
8156 218/300 Write block 3728 len 24576 data: 4 4 5 4 4 5
8157 219/300 Flush
8158 220/300 Read block 2046 len 16384
8159 221/300 Read block 2374 len 40960
8160 222/300 Read block 4766 len 40960
8161 223/300 Read block 3359 len 40960
8162 224/300 Write block 2625 len 28672 data: 4 4 2 2 2 2 2
8163 225/300 Read block 2772 len 32768
8164 226/300 Read block 1342 len 24576
8165 227/300 Write block 2155 len 12288 data: 3 3 3
8166 228/300 Write block 461 len 16384 data: 4 4 4 2
8167 229/300 Write block 1425 len 36864 data: 2 2 2 2 3 3 3 3 4
8168 230/300 Read block 4657 len 12288
8169 231/300 Write block 2198 len 12288 data: 5 4 5
8170 232/300 Read block 1242 len 32768
8171 233/300 Write block 4307 len 36864 data: 5 5 3 4 5 6 5 5 5
8172 234/300 Read block 4738 len 40960
8173 235/300 Read block 2399 len 28672
8174 236/300 Read block 3119 len 12288
8175 237/300 Write block 2900 len 16384 data: 5 5 5 5
8176 238/300 Read block 3851 len 28672
8177 239/300 Read block 323 len 8192
8178 240/300 Write block 1792 len 20480 data: 5 4 3 3 2
8179 241/300 Read block 389 len 24576
8180 242/300 Write block 4322 len 20480 data: 2 2 5 5 6
8181 243/300 Flush
8182 244/300 Read block 2327 len 16384
8183 245/300 Read block 163 len 20480
8184 246/300 Read block 4681 len 28672
8185 247/300 Read block 4602 len 8192
8186 248/300 Read block 4906 len 12288
8187 249/300 Write block 1925 len 40960 data: 4 4 4 2 2 2 3 4 4 5
8188 250/300 Read block 623 len 28672
8189 251/300 Read block 3914 len 40960
8190 252/300 Read block 3748 len 20480
8191 253/300 Read block 121 len 20480
8192 254/300 Read block 831 len 36864
8193 255/300 Read block 4592 len 28672
8194 256/300 Read block 3272 len 28672
8195 257/300 Write block 3087 len 36864 data: 3 3 2 2 2 4 5 5 7
8196 258/300 Read block 1992 len 36864
8197 259/300 Read block 1035 len 12288
8198 260/300 Write block 2470 len 28672 data: 5 7 6 5 4 4 2
8199 261/300 Write block 2931 len 8192 data: 3 3
8200 262/300 Write block 1046 len 24576 data: 3 4 4 4 5 5
8201 263/300 Read block 4653 len 32768
8202 264/300 Flush
8203 265/300 Read block 517 len 4096
8204 266/300 Read block 3905 len 40960
8205 267/300 Flush
8206 268/300 Write block 410 len 16384 data: 2 2 2 2
8207 269/300 Read block 1743 len 20480
8208 270/300 Read block 1851 len 28672
8209 271/300 Flush
8210 272/300 Read block 1801 len 8192
8211 273/300 Flush
8212 274/300 Write block 331 len 24576 data: 3 3 3 3 2 3
8213 275/300 Write block 1882 len 12288 data: 3 4 4
8214 276/300 Read block 2880 len 28672
8215 277/300 Read block 103 len 32768
8216 278/300 Write block 872 len 4096 data: 3
8217 279/300 Read block 2332 len 8192
8218 280/300 Write block 486 len 12288 data: 3 4 4
8219 281/300 Write block 4963 len 16384 data: 3 3 2 2
8220 282/300 Flush
8221 283/300 Read block 1575 len 40960
8222 284/300 Write block 370 len 24576 data: 4 4 4 5 6 6
8223 285/300 Write block 2517 len 8192 data: 3 2
8224 286/300 Write block 1776 len 20480 data: 2 2 2 2 3
8225 287/300 Read block 4769 len 16384
8226 288/300 Write block 4272 len 12288 data: 2 2 2
8227 289/300 Write block 3003 len 16384 data: 6 6 6 5
8228 290/300 Write block 4393 len 36864 data: 3 2 2 2 2 5 5 5 5
8229 291/300 Write block 4431 len 16384 data: 3 2 2 2
8230 292/300 Read block 3888 len 40960
8231 293/300 Write block 2784 len 4096 data: 2
8232 294/300 Write block 1835 len 24576 data: 4 3 3 3 2 2
8233 295/300 Read block 2330 len 4096
8234 296/300 Read block 2938 len 8192
8235 297/300 Read block 2287 len 32768
8236 298/300 Write block 3192 len 40960 data: 3 4 3 3 2 2 3 3 4 4
8237 299/300 Flush
8238 300/300 Write block 4988 len 8192 data: 2 2
8239 Main task has work to do, go find it
8240 got dsc Start(2)
8241 start 2
8242 Oct 13 18:10:53.857 INFO request completed, latency_us: 304, response_code: 204, uri: /start/cid/2, method: POST, req_id: ac39fd68-32e8-49bb-92bb-dc030cd02699, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
8243 Replay: started 2, returned:Ok(())
8244 ----------------------------------------------------------------
8245 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8246 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8247 3028 Acked 4027 Read 7 Done Done New false
8248 3029 Acked 4028 Read 7 Done Done New false
8249 3030 Acked 4029 Write 4 Done Done New false
8250 3031 Acked 4030 Read 8 Done Done New false
8251 3032 Acked 4031 Read 4 Done Done New false
8252 3033 Acked 4032 Read 10 Done Done New false
8253 3034 Acked 4033 Write 10 Done Done New false
8254 3035 Acked 4034 Read 6 Done Done New false
8255 3036 Acked 4035 Read 6 Done Done New false
8256 3037 Acked 4036 Read 3 Done Done New false
8257 3038 Acked 4037 Write 3 Done Done New false
8258 3039 Acked 4038 Read 6 Done Done New false
8259 3040 Acked 4039 Read 9 Done Done New false
8260 3041 Acked 4040 Write 1 Done Done New false
8261 3042 Acked 4041 Flush 0 Done Done New false
8262 3043 Acked 4042 Read 8 Done Done New false
8263 3044 Acked 4043 Write 1 Done Done New false
8264 3045 Acked 4044 Read 8 Done Done New false
8265 3046 Acked 4045 Read 10 Done Done New false
8266 3047 Acked 4046 Read 1 Done Done New false
8267 3048 Acked 4047 Flush 0 Done Done New false
8268 3049 Acked 4048 Flush 0 Done Done New false
8269 3050 Acked 4049 Flush 0 Done Done New false
8270 3051 Acked 4050 Flush 0 Done Done New false
8271 3052 Acked 4051 Write 6 Done Done New false
8272 3053 Acked 4052 Write 9 Done Done New false
8273 3054 Acked 4053 Read 3 Done Done New false
8274 3055 Acked 4054 Write 8 Done Done New false
8275 3056 Acked 4055 Write 7 Done Done New false
8276 3057 Acked 4056 Write 8 Done Done New false
8277 3058 Acked 4057 Write 6 Done Done New false
8278 3059 Acked 4058 Write 3 Done Done New false
8279 3060 Acked 4059 Write 8 Done Done New false
8280 3061 Acked 4060 Read 5 Done Done New false
8281 3062 Acked 4061 Read 10 Done Done New false
8282 3063 Acked 4062 Write 2 Done Done New false
8283 3064 Acked 4063 Read 10 Done Done New false
8284 3065 Acked 4064 Read 9 Done Done New false
8285 3066 Acked 4065 Read 10 Done Done New false
8286 3067 Acked 4066 Write 7 Done Done New false
8287 3068 Acked 4067 Write 2 Done Done New false
8288 3069 Acked 4068 Read 7 Done Done New false
8289 3070 Acked 4069 Write 8 Done Done New false
8290 3071 Acked 4070 Flush 0 Done Done New false
8291 3072 Acked 4071 Read 4 Done Done New false
8292 3073 Acked 4072 Read 4 Done Done New false
8293 3074 Acked 4073 Write 9 Done Done New false
8294 3075 Acked 4074 Read 2 Done Done New false
8295 3076 Acked 4075 Write 5 Done Done New false
8296 3077 Acked 4076 Flush 0 Done Done New false
8297 3078 Acked 4077 Read 2 Done Done New false
8298 3079 Acked 4078 Read 5 Done Done New false
8299 3080 Acked 4079 Read 5 Done Done New false
8300 3081 Acked 4080 Write 9 Done Done New false
8301 3082 Acked 4081 Write 4 Done Done New false
8302 3083 Acked 4082 Read 2 Done Done New false
8303 3084 Acked 4083 Read 6 Done Done New false
8304 3085 Acked 4084 Read 7 Done Done New false
8305 3086 Acked 4085 Read 1 Done Done New false
8306 3087 Acked 4086 Read 5 Done Done New false
8307 3088 Acked 4087 Write 4 Done Done New false
8308 3089 Acked 4088 Write 5 Done Done New false
8309 3090 Acked 4089 Read 1 Done Done New false
8310 3091 Acked 4090 Read 6 Done Done New false
8311 3092 Acked 4091 Read 9 Done Done New false
8312 3093 Acked 4092 Read 1 Done Done New false
8313 3094 Acked 4093 Read 9 Done Done New false
8314 3095 Acked 4094 Write 1 Done Done New false
8315 3096 Acked 4095 Read 7 Done Done New false
8316 3097 Acked 4096 Write 2 Done Done New false
8317 3098 Acked 4097 Read 2 Done Done New false
8318 3099 Acked 4098 Write 8 Done Done New false
8319 3100 Acked 4099 Read 4 Done Done New false
8320 3101 Acked 4100 Read 3 Done Done New false
8321 3102 Acked 4101 Write 2 Done Done New false
8322 3103 Acked 4102 Write 10 Done Done New false
8323 3104 Acked 4103 Read 6 Done Done New false
8324 3105 Acked 4104 Write 10 Done Done New false
8325 3106 Acked 4105 Flush 0 Done Done New false
8326 3107 Acked 4106 Write 4 Done Done New false
8327 3108 Acked 4107 Read 5 Done Done New false
8328 3109 Acked 4108 Read 2 Done Done New false
8329 3110 Acked 4109 Read 2 Done Done New false
8330 3111 Acked 4110 Write 9 Done Done New false
8331 3112 Acked 4111 Flush 0 Done Done New false
8332 3113 Acked 4112 Read 8 Done Done New false
8333 3114 Acked 4113 Read 2 Done Done New false
8334 3115 Acked 4114 Read 5 Done Done New false
8335 3116 Acked 4115 Read 9 Done Done New false
8336 3117 Acked 4116 Write 8 Done Done New false
8337 3118 Acked 4117 Write 2 Done Done New false
8338 3119 Acked 4118 Read 7 Done Done New false
8339 3120 Acked 4119 Read 8 Done Done New false
8340 3121 Acked 4120 Read 10 Done Done New false
8341 3122 Acked 4121 Read 6 Done Done New false
8342 3123 Acked 4122 Flush 0 Done Done New false
8343 3124 Acked 4123 Flush 0 Done Done New false
8344 3125 Acked 4124 Write 4 Done Done New false
8345 3126 Acked 4125 Read 6 Done Done New false
8346 3127 Acked 4126 Write 6 Done Done New false
8347 3128 Acked 4127 Read 6 Done Done New false
8348 3129 Acked 4128 Read 10 Done Done New false
8349 3130 Acked 4129 Read 10 Done Done New false
8350 3131 Acked 4130 Write 1 Done Done New false
8351 3132 Acked 4131 Read 10 Done Done New false
8352 3133 Acked 4132 Read 9 Done Done New false
8353 3134 Acked 4133 Read 7 Done Done New false
8354 3135 Acked 4134 Write 4 Done Done New false
8355 3136 Acked 4135 Read 5 Done Done New false
8356 3137 Acked 4136 Flush 0 Done Done New false
8357 3138 Acked 4137 Write 9 Done Done New false
8358 3139 Acked 4138 Read 3 Done Done New false
8359 3140 Acked 4139 Read 6 Done Done New false
8360 3141 Acked 4140 Flush 0 Done Done New false
8361 3142 Acked 4141 Write 9 Done Done New false
8362 3143 Acked 4142 Read 2 Done Done New false
8363 3144 Acked 4143 Flush 0 Done Done New false
8364 3145 Acked 4144 Write 7 Done Done New false
8365 3146 Acked 4145 Write 10 Done Done New false
8366 3147 Acked 4146 Read 4 Done Done New false
8367 3148 Acked 4147 Write 7 Done Done New false
8368 3149 Acked 4148 Read 1 Done Done New false
8369 3150 Acked 4149 Read 10 Done Done New false
8370 3151 Acked 4150 Write 9 Done Done New false
8371 3152 Acked 4151 Write 3 Done Done New false
8372 3153 Acked 4152 Read 3 Done Done New false
8373 3154 Acked 4153 Write 3 Done Done New false
8374 3155 Acked 4154 Read 5 Done Done New false
8375 3156 Acked 4155 Read 8 Done Done New false
8376 3157 Acked 4156 Read 10 Done Done New false
8377 3158 Acked 4157 Read 2 Done Done New false
8378 3159 Acked 4158 Write 1 Done Done New false
8379 3160 Acked 4159 Write 9 Done Done New false
8380 3161 Acked 4160 Read 1 Done Done New false
8381 3162 Acked 4161 Read 4 Done Done New false
8382 3163 Acked 4162 Write 8 Done Done New false
8383 3164 Acked 4163 Read 4 Done Done New false
8384 3165 Acked 4164 Flush 0 Done Done New false
8385 3166 Acked 4165 Read 10 Done Done New false
8386 3167 Acked 4166 Read 5 Done Done New false
8387 3168 Acked 4167 Read 2 Done Done New false
8388 3169 Acked 4168 Write 2 Done Done New false
8389 3170 Acked 4169 Write 6 Done Done New false
8390 3171 Acked 4170 Write 8 Done Done New false
8391 3172 Acked 4171 Read 10 Done Done New false
8392 3173 Acked 4172 Write 4 Done Done New false
8393 3174 Acked 4173 Read 3 Done Done New false
8394 3175 Acked 4174 Read 4 Done Done New false
8395 3176 Acked 4175 Read 9 Done Done New false
8396 3177 Acked 4176 Read 9 Done Done New false
8397 3178 Acked 4177 Read 6 Done Done New false
8398 3179 Acked 4178 Read 8 Done Done New false
8399 3180 Acked 4179 Read 2 Done Done New false
8400 3181 Acked 4180 Read 9 Done Done New false
8401 3182 Acked 4181 Flush 0 Done Done New false
8402 3183 Acked 4182 Read 1 Done Done New false
8403 3184 Acked 4183 Write 10 Done Done New false
8404 3185 Acked 4184 Read 6 Done Done New false
8405 3186 Acked 4185 Read 5 Done Done New false
8406 3187 Acked 4186 Write 7 Done Done New false
8407 3188 Acked 4187 Write 1 Done Done New false
8408 3189 Acked 4188 Read 1 Done Done New false
8409 3190 Acked 4189 Read 8 Done Done New false
8410 3191 Acked 4190 Read 10 Done Done New false
8411 3192 Acked 4191 Read 9 Done Done New false
8412 3193 Acked 4192 Write 7 Done Done New false
8413 3194 Acked 4193 Write 7 Done Done New false
8414 3195 Acked 4194 Read 2 Done Done New false
8415 3196 Acked 4195 Write 6 Done Done New false
8416 3197 Acked 4196 Read 8 Done Done New false
8417 3198 Acked 4197 Read 9 Done Done New false
8418 3199 Acked 4198 Write 9 Done Done New false
8419 3200 Acked 4199 Read 2 Done Done New false
8420 3201 Acked 4200 Read 1 Done Done New false
8421 3202 Acked 4201 Write 1 Done Done New false
8422 3203 Acked 4202 Read 3 Done Done New false
8423 3204 Acked 4203 Read 6 Done Done New false
8424 3205 Acked 4204 Write 5 Done Done New false
8425 3206 Acked 4205 Read 8 Done Done New false
8426 3207 Acked 4206 Flush 0 Done Done New false
8427 3208 Acked 4207 Read 5 Done Done New false
8428 3209 Acked 4208 Write 10 Done Done New false
8429 3210 Acked 4209 Read 6 Done Done New false
8430 3211 Acked 4210 Read 8 Done Done New false
8431 3212 Acked 4211 Read 3 Done Done New false
8432 3213 Acked 4212 Write 3 Done Done New false
8433 3214 Acked 4213 Write 9 Done Done New false
8434 3215 Acked 4214 Write 10 Done Done New false
8435 3216 Acked 4215 Flush 0 Done Done New false
8436 3217 Acked 4216 Write 4 Done Done New false
8437 3218 Acked 4217 Write 5 Done Done New false
8438 3219 Acked 4218 Read 4 Done Done New false
8439 3220 Acked 4219 Read 1 Done Done New false
8440 3221 Acked 4220 Read 10 Done Done New false
8441 3222 Acked 4221 Write 4 Done Done New false
8442 3223 Acked 4222 Read 2 Done Done New false
8443 3224 Acked 4223 Write 7 Done Done New false
8444 3225 Acked 4224 Write 4 Done Done New false
8445 3226 Acked 4225 Read 8 Done Done New false
8446 3227 Acked 4226 Write 9 Done Done New false
8447 3228 Acked 4227 Read 2 Done Done New false
8448 3229 Acked 4228 Write 7 Done Done New false
8449 3230 Acked 4229 Read 7 Done Done New false
8450 3231 Acked 4230 Write 4 Done Done New false
8451 3232 Acked 4231 Write 1 Done Done New false
8452 3233 Acked 4232 Flush 0 Done Done New false
8453 3234 Acked 4233 Write 6 Done Done New false
8454 3235 Acked 4234 Write 8 Done Done New false
8455 3236 Acked 4235 Read 3 Done Done New false
8456 3237 Acked 4236 Write 4 Done Done New false
8457 3238 Acked 4237 Write 1 Done Done New false
8458 3239 Acked 4238 Read 3 Done Done New false
8459 3240 Acked 4239 Flush 0 Done Done New false
8460 3241 Acked 4240 Read 7 Done Done New false
8461 3242 Acked 4241 Write 7 Done Done New false
8462 3243 Acked 4242 Write 9 Done Done New false
8463 3244 Acked 4243 Read 2 Done Done New false
8464 3245 Acked 4244 Write 5 Done Done New false
8465 3246 Acked 4245 Write 6 Done Done New false
8466 3247 Acked 4246 Flush 0 Done Done New false
8467 3248 Acked 4247 Read 4 Done Done New false
8468 3249 Acked 4248 Read 10 Done Done New false
8469 3250 Acked 4249 Read 10 Done Done New false
8470 3251 Acked 4250 Read 10 Done Done New false
8471 3252 Acked 4251 Write 7 Done Done New false
8472 3253 Acked 4252 Read 8 Done Done New false
8473 3254 Acked 4253 Read 6 Done Done New false
8474 3255 Acked 4254 Write 3 Done Done New false
8475 3256 Acked 4255 Write 4 Done Done New false
8476 3257 Acked 4256 Write 9 Done Done New false
8477 3258 Acked 4257 Read 3 Done Done New false
8478 3259 Acked 4258 Write 3 Done Done New false
8479 3260 Acked 4259 Read 8 Done Done New false
8480 3261 Acked 4260 Write 9 Done Done New false
8481 3262 Acked 4261 Read 10 Done Done New false
8482 3263 Acked 4262 Read 7 Done Done New false
8483 3264 Acked 4263 Read 3 Done Done New false
8484 3265 Acked 4264 Write 4 Done Done New false
8485 3266 Acked 4265 Read 7 Done Done New false
8486 3267 Acked 4266 Read 2 Done Done New false
8487 3268 Acked 4267 Write 5 Done Done New false
8488 3269 Acked 4268 Read 6 Done Done New false
8489 3270 Acked 4269 Write 5 Done Done New false
8490 3271 Acked 4270 Flush 0 Done Done New false
8491 3272 Acked 4271 Read 4 Done Done New false
8492 3273 Acked 4272 Read 5 Done Done New false
8493 3274 Acked 4273 Read 7 Done Done New false
8494 3275 Acked 4274 Read 2 Done Done New false
8495 3276 Acked 4275 Read 3 Done Done New false
8496 3277 Acked 4276 Write 10 Done Done New false
8497 3278 Acked 4277 Read 7 Done Done New false
8498 3279 Acked 4278 Read 10 Done Done New false
8499 3280 Acked 4279 Read 5 Done Done New false
8500 3281 Acked 4280 Read 5 Done Done New false
8501 3282 Acked 4281 Read 9 Done Done New false
8502 3283 Acked 4282 Read 7 Done Done New false
8503 3284 Acked 4283 Read 7 Done Done New false
8504 3285 Acked 4284 Write 9 Done Done New false
8505 3286 Acked 4285 Read 9 Done Done New false
8506 3287 Acked 4286 Read 3 Done Done New false
8507 3288 Acked 4287 Write 7 Done Done New false
8508 3289 Acked 4288 Write 2 Done Done New false
8509 3290 Acked 4289 Write 6 Done Done New false
8510 3291 Acked 4290 Read 8 Done Done New false
8511 3292 Acked 4291 Flush 0 Done Done New false
8512 3293 Acked 4292 Read 1 Done Done New false
8513 3294 Acked 4293 Read 10 Done Done New false
8514 3295 Acked 4294 Flush 0 Done Done New false
8515 3296 Acked 4295 Write 4 Done Done New false
8516 3297 Acked 4296 Read 5 Done Done New false
8517 3298 Acked 4297 Read 7 Done Done New false
8518 3299 Acked 4298 Flush 0 Done Done New false
8519 3300 Acked 4299 Read 2 Done Done New false
8520 3301 Acked 4300 Flush 0 Done Done New false
8521 3302 Acked 4301 Write 6 Done Done New false
8522 3303 Acked 4302 Write 3 Done Done New false
8523 3304 Acked 4303 Read 7 Done Done New false
8524 3305 Acked 4304 Read 8 Done Done New false
8525 3306 Acked 4305 Write 1 Done Done New false
8526 3307 Acked 4306 Read 2 Done Done New false
8527 3308 Acked 4307 Write 3 Done Done New false
8528 3309 Acked 4308 Write 4 Done Done New false
8529 3310 Acked 4309 Flush 0 Done Done New false
8530 3311 Acked 4310 Read 10 Done Done New false
8531 3312 Acked 4311 Write 6 Done Done New false
8532 3313 Acked 4312 Write 2 Done Done New false
8533 3314 Acked 4313 Write 5 Done Done New false
8534 3315 Acked 4314 Read 4 Done Done New false
8535 3316 Acked 4315 Write 3 Done Done New false
8536 3317 Acked 4316 Write 4 Done Done New false
8537 3318 Acked 4317 Write 9 Done Done New false
8538 3319 Acked 4318 Write 4 Done Done New false
8539 3320 Acked 4319 Read 10 Done Done New false
8540 3321 Acked 4320 Write 1 Done Done New false
8541 3322 Acked 4321 Write 6 Done Done New false
8542 3323 Acked 4322 Read 1 Done Done New false
8543 3324 Acked 4323 Read 2 Done Done New false
8544 3325 Acked 4324 Read 8 Done Done New false
8545 3326 Acked 4325 Write 10 Done Done New false
8546 3327 Acked 4326 Flush 0 Done Done New false
8547 3328 Acked 4327 Write 2 Sent Sent New false
8548 STATES DS:0 DS:1 DS:2 TOTAL
8549 New 0 0 301 301
8550 Sent 1 1 0 2
8551 Done 300 300 0 600
8552 Skipped 0 0 0 0
8553 Error 0 0 0 0
8554 Last Flush: 4326 4326 4026
8555 Downstairs last five completed: 4026 4025 4024 4023 4022 4021
8556 Upstairs last five completed: 3328 3327 3326 3325 3324 3323
8557 CLIENT: Up:0 ds:301 act:2
8558 [8830] got start action
8559 [8830] I am going to restart
8560 Starting downstairs at port 8830
8561 Make output file at "/tmp/dsc/downstairs-8830.txt"
8562 [8830][2] reports Starting
8563 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1187)
8564 [8830][2] reports Running
85652023-10-13T18:10:55.895ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
85662023-10-13T18:10:55.896ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 2
85672023-10-13T18:10:55.896ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
85682023-10-13T18:10:55.896ZINFOcrucible: [2] client re-new 302 jobs since flush 4026 = downstairs
85692023-10-13T18:10:55.897ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
85702023-10-13T18:10:55.897ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
85712023-10-13T18:10:55.897ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
85722023-10-13T18:10:55.897ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
85732023-10-13T18:10:55.897ZINFOcrucible: [2] send last flush ID to this DS: 4026
85742023-10-13T18:10:55.897ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
85752023-10-13T18:10:55.898ZINFOcrucible: [2] Replied this last flush ID: 4026
85762023-10-13T18:10:55.898ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Offline ds_transition to Replay
85772023-10-13T18:10:55.898ZINFOcrucible: [2] Transition from Offline to Replay
85782023-10-13T18:10:55.898ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
85792023-10-13T18:10:55.898ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Replay ds_transition to Active
85802023-10-13T18:10:55.898ZINFOcrucible: [2] Transition from Replay to Active
85812023-10-13T18:10:55.898ZINFOcrucible: [2] Starts cmd_loop
85822023-10-13T18:10:55.898ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
85832023-10-13T18:10:55.898ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
85842023-10-13T18:10:55.898ZINFOcrucible: Set check for repair
85852023-10-13T18:10:56.899ZINFOcrucible: Checking if live repair is needed
85862023-10-13T18:10:56.899ZINFOcrucible: No Live Repair required at this time
85872023-10-13T18:10:56.899ZWARNcrucible: [2] flow control end
8588 ----------------------------------------------------------------
8589 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8590 Downstairs last five completed: 4328 4327 4326 4325 4324 4323
8591 Upstairs last five completed: 3329 3328 3327 3326 3325 3324
8592 CLIENT: Up:0 ds:0 act:3
8593 Replay: All jobs finished, all DS active.
8594 Main task has work to do, go find it
8595 got dsc Stop(0)
8596 stop 0
8597 Oct 13 18:10:57.862 INFO request completed, latency_us: 522, response_code: 204[8810] Got stop action so:false kr:false
8598 , uri: /stop/cid/0, method: POST, req_id: fac5cc70-ceed-4e42-86bf-aa587e19ea67, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
8599 Oct 13 18:10:57.863 INFO request completed, latency_us: 185, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4ca55261-eb29-49cd-b097-b7c1062c614f, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
86002023-10-13T18:10:57.863ZWARNcrucible: [0] downstairs disconnected
86012023-10-13T18:10:57.863ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
86022023-10-13T18:10:57.863ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8810 closed looper = 0
86032023-10-13T18:10:57.863ZWARNcrucible: [0] pm_task rx.recv() is None
86042023-10-13T18:10:57.863ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:false
86052023-10-13T18:10:57.863ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
86062023-10-13T18:10:57.863ZINFOcrucible: [0] 127.0.0.1:8810 task reports offline
8607 [8810] Exited with: ExitStatus(unix_wait_status(9))
8608 [8810][0] reports Exit
8609 Oct 13 18:11:01.865 INFO request completed, latency_us: 253, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2738316c-d42b-4165-b764-3f0607345257, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
8610 001/300 Read block 3354 len 36864
8611 002/300 Write block 3594 len 20480 data: 3 4 5 4 3
8612 003/300 Read block 2164 len 8192
8613 004/300 Read block 187 len 36864
8614 005/300 Write block 1614 len 4096 data: 3
8615 006/300 Read block 3552 len 4096
8616 007/300 Read block 1893 len 40960
8617 008/300 Read block 2891 len 40960
8618 009/300 Flush
8619 010/300 Read block 2539 len 4096
8620 011/300 Flush
8621 012/300 Flush
8622 013/300 Read block 3396 len 28672
8623 014/300 Write block 2754 len 4096 data: 3
8624 015/300 Write block 451 len 36864 data: 3 3 2 2 2 2 2 2 2
8625 016/300 Read block 1218 len 36864
8626 017/300 Write block 662 len 4096 data: 3
8627 018/300 Write block 995 len 20480 data: 3 2 2 2 2
8628 019/300 Read block 489 len 32768
8629 020/300 Read block 2664 len 28672
8630 021/300 Read block 3913 len 36864
8631 022/300 Read block 1235 len 4096
8632 023/300 Write block 1691 len 24576 data: 3 4 4 4 4 4
8633 024/300 Write block 4811 len 40960 data: 4 3 3 3 3 3 3 3 3 3
8634 025/300 Read block 601 len 32768
8635 026/300 Flush
8636 027/300 Write block 3816 len 28672 data: 2 2 3 3 3 4 4
8637 028/300 Read block 3093 len 32768
8638 029/300 Write block 4215 len 40960 data: 3 2 3 3 4 4 4 4 4 5
8639 030/300 Read block 895 len 4096
8640 031/300 Write block 47 len 8192 data: 6 8
8641 032/300 Write block 2314 len 12288 data: 2 2 3
8642 033/300 Read block 2221 len 28672
8643 034/300 Write block 3962 len 40960 data: 2 3 3 4 3 2 2 3 3 2
8644 035/300 Flush
8645 036/300 Read block 4417 len 20480
8646 037/300 Read block 3190 len 20480
8647 038/300 Write block 1993 len 12288 data: 4 6 7
8648 039/300 Write block 4724 len 40960 data: 5 5 5 5 5 5 3 3 3 2
8649 040/300 Write block 1934 len 8192 data: 6 4
8650 041/300 Read block 3552 len 28672
8651 042/300 Write block 2234 len 36864 data: 3 3 3 3 4 4 4 2 3
8652 043/300 Read block 2190 len 4096
8653 044/300 Read block 3931 len 12288
8654 045/300 Write block 4347 len 8192 data: 3 3
8655 046/300 Write block 3793 len 32768 data: 3 2 2 2 2 2 2 2
8656 047/300 Read block 2671 len 36864
8657 048/300 Write block 2240 len 16384 data: 5 3 4 4
8658 049/300 Write block 4151 len 8192 data: 4 6
8659 050/300 Write block 3302 len 40960 data: 2 2 3 3 4 4 3 3 3 2
8660 051/300 Read block 2845 len 24576
8661 052/300 Read block 4653 len 12288
8662 053/300 Read block 962 len 16384
8663 054/300 Read block 410 len 16384
8664 055/300 Write block 604 len 32768 data: 6 4 4 4 4 3 4 2
8665 056/300 Read block 2331 len 4096
8666 057/300 Read block 3980 len 36864
8667 058/300 Read block 163 len 36864
8668 059/300 Write block 4150 len 4096 data: 4
8669 060/300 Flush
8670 061/300 Read block 4926 len 12288
8671 062/300 Read block 600 len 28672
8672 063/300 Read block 3785 len 28672
8673 064/300 Write block 1036 len 40960 data: 3 3 3 2 3 5 5 5 5 4
8674 065/300 Write block 4648 len 8192 data: 2 2
8675 066/300 Write block 3569 len 24576 data: 4 3 3 3 4 4
8676 067/300 Flush
8677 068/300 Read block 2534 len 16384
8678 069/300 Flush
8679 070/300 Read block 4422 len 32768
8680 071/300 Read block 1780 len 32768
8681 072/300 Read block 3021 len 36864
8682 073/300 Write block 2459 len 24576 data: 2 2 4 4 4 4
8683 074/300 Write block 985 len 40960 data: 4 3 3 3 3 2 2 3 3 3
8684 075/300 Read block 1013 len 28672
8685 076/300 Write block 3455 len 40960 data: 4 4 3 4 6 7 5 5 4 3
8686 077/300 Read block 4429 len 20480
8687 078/300 Read block 2419 len 24576
8688 079/300 Write block 4702 len 16384 data: 8 6 4 3
8689 080/300 Write block 2521 len 8192 data: 4 3
8690 081/300 Read block 3394 len 40960
8691 082/300 Read block 4751 len 4096
8692 083/300 Read block 4702 len 28672
8693 084/300 Write block 1089 len 16384 data: 4 3 4 3
8694 085/300 Write block 2613 len 20480 data: 3 3 3 3 4
8695 086/300 Write block 212 len 4096 data: 5
8696 087/300 Read block 2040 len 4096
8697 088/300 Flush
8698 089/300 Write block 2310 len 32768 data: 4 4 3 3 3 3 4 3
8699 090/300 Flush
8700 091/300 Read block 4468 len 20480
8701 092/300 Read block 4366 len 12288
8702 093/300 Write block 4633 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8703 094/300 Read block 917 len 16384
8704 095/300 Read block 2926 len 32768
8705 096/300 Write block 3566 len 40960 data: 2 2 3 5 4 4 4 5 5 5
8706 097/300 Read block 3897 len 4096
8707 098/300 Read block 4393 len 8192
8708 099/300 Flush
8709 100/300 Write block 2276 len 20480 data: 3 4 3 4 6
8710 101/300 Read block 1591 len 20480
8711 102/300 Read block 1245 len 4096
8712 103/300 Write block 1818 len 32768 data: 4 3 3 2 2 2 2 3
8713 104/300 Read block 676 len 8192
8714 105/300 Write block 1330 len 32768 data: 2 3 3 3 4 4 3 3
8715 106/300 Write block 32 len 40960 data: 2 3 4 5 5 5 3 3 3 2
8716 107/300 Read block 3933 len 4096
8717 108/300 Write block 2629 len 12288 data: 3 3 3
8718 109/300 Read block 3147 len 24576
8719 110/300 Read block 4521 len 12288
8720 111/300 Flush
8721 112/300 Flush
8722 113/300 Read block 998 len 24576
8723 114/300 Flush
8724 115/300 Write block 2807 len 24576 data: 4 5 4 4 3 4
8725 116/300 Write block 4422 len 16384 data: 3 3 4 3
8726 117/300 Write block 3667 len 32768 data: 5 5 4 4 4 3 3 2
8727 118/300 Write block 2555 len 20480 data: 5 7 6 5 4
8728 119/300 Write block 598 len 4096 data: 4
8729 120/300 Write block 14 len 16384 data: 3 3 3 3
8730 121/300 Write block 4486 len 20480 data: 4 4 4 4 3
8731 122/300 Read block 3951 len 40960
8732 123/300 Write block 73 len 8192 data: 3 4
8733 124/300 Read block 4375 len 24576
8734 125/300 Read block 2812 len 16384
8735 126/300 Write block 2361 len 36864 data: 2 3 3 4 4 5 6 4 4
8736 127/300 Read block 359 len 4096
8737 128/300 Flush
8738 129/300 Read block 4003 len 8192
8739 130/300 Read block 4036 len 32768
8740 131/300 Read block 4646 len 20480
8741 132/300 Write block 1098 len 16384 data: 8 8 8 5
8742 133/300 Read block 833 len 36864
8743 134/300 Write block 3564 len 24576 data: 2 2 3 3 4 6
8744 135/300 Read block 1807 len 28672
8745 136/300 Write block 2990 len 40960 data: 3 2 2 3 5 5 5 6 6 6
8746 137/300 Read block 1803 len 8192
8747 138/300 Read block 2248 len 4096
8748 139/300 Read block 2917 len 20480
8749 140/300 Read block 2427 len 28672
8750 141/300 Write block 3067 len 28672 data: 2 2 2 3 3 3 3
8751 142/300 Read block 2392 len 12288
8752 143/300 Write block 2009 len 36864 data: 6 6 5 5 3 3 6 5 5
8753 144/300 Read block 2147 len 32768
8754 145/300 Read block 812 len 8192
8755 146/300 Read block 3178 len 4096
8756 147/300 Read block 2985 len 32768
8757 148/300 Write block 1741 len 20480 data: 4 4 4 4 3
8758 149/300 Write block 4804 len 28672 data: 6 5 4 4 4 3 4
8759 150/300 Read block 4681 len 32768
8760 151/300 Read block 3617 len 16384
8761 152/300 Read block 2598 len 28672
8762 153/300 Read block 2421 len 8192
8763 154/300 Read block 1022 len 8192
8764 155/300 Read block 0 len 32768
8765 156/300 Read block 4233 len 40960
8766 157/300 Read block 811 len 20480
8767 158/300 Read block 2358 len 8192
8768 159/300 Read block 1369 len 12288
8769 160/300 Read block 3125 len 20480
8770 161/300 Write block 1339 len 24576 data: 3 3 3 3 4 4
8771 162/300 Write block 3926 len 16384 data: 3 3 3 3
8772 163/300 Read block 4077 len 20480
8773 164/300 Read block 810 len 28672
8774 165/300 Write block 3402 len 16384 data: 3 2 3 3
8775 166/300 Read block 3665 len 24576
8776 167/300 Read block 735 len 24576
8777 168/300 Write block 3618 len 16384 data: 4 5 6 4
8778 169/300 Flush
8779 170/300 Write block 4996 len 8192 data: 2 2
8780 171/300 Read block 1586 len 16384
8781 172/300 Write block 3432 len 20480 data: 6 6 5 4 4
8782 173/300 Read block 4667 len 12288
8783 174/300 Read block 4865 len 8192
8784 175/300 Flush
8785 176/300 Write block 3755 len 32768 data: 4 4 3 3 3 2 2 3
8786 177/300 Write block 3091 len 36864 data: 3 5 6 6 8 7 6 5 5
8787 178/300 Write block 1063 len 36864 data: 4 4 4 4 5 4 3 3 3
8788 179/300 Write block 1296 len 16384 data: 3 3 3 3
8789 180/300 Write block 2524 len 40960 data: 4 5 5 5 4 4 3 4 4 4
8790 181/300 Write block 3765 len 20480 data: 4 3 3 3 4
8791 182/300 Write block 1604 len 8192 data: 4 4
8792 183/300 Write block 3953 len 20480 data: 4 5 5 4 4
8793 184/300 Write block 4022 len 16384 data: 3 3 3 3
8794 185/300 Write block 3660 len 28672 data: 4 4 5 5 5 5 7
8795 186/300 Read block 3983 len 28672
8796 187/300 Write block 1430 len 12288 data: 4 4 4
8797 188/300 Write block 4189 len 36864 data: 4 3 3 3 5 4 4 5 5
8798 189/300 Read block 3206 len 28672
8799 190/300 Read block 4949 len 32768
8800 191/300 Write block 4579 len 20480 data: 4 5 5 4 3
8801 192/300 Write block 2601 len 20480 data: 4 6 5 4 4
8802 193/300 Write block 124 len 40960 data: 3 3 3 4 5 4 5 5 5 5
8803 194/300 Read block 195 len 32768
8804 195/300 Read block 3001 len 8192
8805 196/300 Write block 160 len 32768 data: 2 3 4 4 4 4 4 3
8806 197/300 Write block 4649 len 32768 data: 3 2 2 2 2 2 2 2
8807 198/300 Read block 4774 len 12288
8808 199/300 Read block 1789 len 40960
8809 200/300 Read block 1451 len 28672
8810 201/300 Read block 1330 len 40960
8811 202/300 Read block 4098 len 16384
8812 203/300 Read block 3389 len 24576
8813 204/300 Write block 4914 len 36864 data: 3 3 3 3 3 3 3 3 4
8814 205/300 Read block 604 len 12288
8815 206/300 Write block 2556 len 32768 data: 8 7 6 5 4 4 3 3
8816 207/300 Read block 57 len 40960
8817 208/300 Read block 4510 len 32768
8818 209/300 Read block 2202 len 32768
8819 210/300 Write block 2495 len 4096 data: 3
8820 211/300 Read block 1679 len 28672
8821 212/300 Read block 2208 len 20480
8822 213/300 Read block 2480 len 4096
8823 214/300 Read block 857 len 4096
8824 215/300 Read block 3358 len 20480
8825 216/300 Read block 656 len 32768
8826 217/300 Write block 3917 len 8192 data: 4 3
8827 218/300 Flush
8828 219/300 Write block 3006 len 36864 data: 6 4 3 3 3 4 4 5 4
8829 220/300 Read block 1873 len 4096
8830 221/300 Write block 3476 len 24576 data: 4 4 4 3 3 2
8831 222/300 Read block 4714 len 40960
8832 223/300 Write block 2931 len 20480 data: 4 4 3 3 4
8833 224/300 Write block 3993 len 12288 data: 3 3 3
8834 225/300 Read block 899 len 28672
8835 226/300 Write block 206 len 16384 data: 3 3 4 4
8836 227/300 Write block 2416 len 4096 data: 3
8837 228/300 Read block 3415 len 8192
8838 229/300 Flush
8839 230/300 Read block 1642 len 20480
8840 231/300 Read block 4893 len 20480
8841 232/300 Read block 3098 len 32768
8842 233/300 Read block 1852 len 8192
8843 234/300 Read block 3757 len 20480
8844 235/300 Read block 650 len 20480
8845 236/300 Read block 4771 len 36864
8846 237/300 Read block 720 len 24576
8847 238/300 Read block 4145 len 4096
8848 239/300 Read block 3813 len 12288
8849 240/300 Read block 2895 len 24576
8850 241/300 Write block 104 len 36864 data: 4 3 2 2 2 2 2 2 2
8851 242/300 Flush
8852 243/300 Read block 4366 len 40960
8853 244/300 Read block 2865 len 4096
8854 245/300 Write block 3506 len 36864 data: 3 3 3 4 4 4 3 2 2
8855 246/300 Write block 4170 len 8192 data: 6 5
8856 247/300 Write block 2968 len 16384 data: 3 3 3 4
8857 248/300 Write block 1328 len 40960 data: 2 2 3 4 4 4 5 5 4 4
8858 249/300 Read block 3427 len 24576
8859 250/300 Read block 4548 len 20480
8860 251/300 Read block 4702 len 28672
8861 252/300 Read block 4546 len 28672
8862 253/300 Write block 1256 len 32768 data: 4 4 4 4 4 3 3 3
8863 254/300 Read block 1057 len 8192
8864 255/300 Read block 3994 len 4096
8865 256/300 Read block 755 len 40960
8866 257/300 Read block 3795 len 36864
8867 258/300 Flush
8868 259/300 Write block 2270 len 4096 data: 2
8869 260/300 Write block 1914 len 24576 data: 3 3 2 2 3 3
8870 261/300 Read block 668 len 28672
8871 262/300 Write block 4840 len 12288 data: 4 4 5
8872 263/300 Write block 3515 len 28672 data: 2 2 2 4 5 6 5
8873 264/300 Read block 3607 len 28672
8874 265/300 Write block 2143 len 36864 data: 4 4 3 4 3 2 2 2 2
8875 266/300 Read block 4921 len 4096
8876 267/300 Read block 3510 len 32768
8877 268/300 Write block 1710 len 4096 data: 3
8878 269/300 Read block 2107 len 32768
8879 270/300 Write block 1126 len 12288 data: 3 3 3
8880 271/300 Read block 309 len 32768
8881 272/300 Write block 4916 len 28672 data: 4 4 4 4 4 4 5
8882 273/300 Read block 3292 len 36864
8883 274/300 Flush
8884 275/300 Write block 4372 len 28672 data: 5 5 5 4 4 4 4
8885 276/300 Write block 2117 len 28672 data: 2 2 3 4 4 4 3
8886 277/300 Write block 4751 len 20480 data: 5 4 5 5 7
8887 278/300 Read block 3042 len 12288
8888 279/300 Read block 2040 len 20480
8889 280/300 Write block 1442 len 32768 data: 3 2 2 2 2 2 5 6
8890 281/300 Write block 4015 len 24576 data: 2 2 2 2 2 2
8891 282/300 Read block 201 len 12288
8892 283/300 Read block 534 len 4096
8893 284/300 Read block 2748 len 20480
8894 285/300 Write block 446 len 40960 data: 4 3 3 3 3 4 4 3 3 3
8895 286/300 Write block 3127 len 36864 data: 2 2 2 2 2 2 3 2 2
8896 287/300 Read block 1742 len 20480
8897 288/300 Read block 4751 len 28672
8898 289/300 Flush
8899 290/300 Write block 1782 len 12288 data: 3 4 4
8900 291/300 Read block 361 len 4096
8901 292/300 Read block 4912 len 36864
8902 293/300 Write block 595 len 40960 data: 2 2 4 5 4 5 5 5 6 7
8903 294/300 Read block 769 len 36864
8904 295/300 Read block 4400 len 24576
8905 296/300 Write block 1882 len 36864 data: 4 5 5 4 3 3 3 3 3
8906 297/300 Write block 268 len 32768 data: 3 4 4 2 2 2 2 2
8907 298/300 Read block 734 len 32768
8908 299/300 Flush
8909 300/300 Write block 73 len 8192 data: 4 5
8910 Main task has work to do, go find it
8911 got dsc Start(0)
8912 start 0
8913 Oct 13 18:11:02.810 INFO request completed, latency_us: 299, response_code: 204, uri: /start/cid/0, method: POST, req_id: 23ac038b-dc0d-4aad-a81b-63326c926e18, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
8914 Replay: started 0, returned:Ok(())
8915 ----------------------------------------------------------------
8916 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
8917 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8918 3330 Acked 4329 Read 9 New Done Done false
8919 3331 Acked 4330 Write 5 New Done Done false
8920 3332 Acked 4331 Read 2 New Done Done false
8921 3333 Acked 4332 Read 9 New Done Done false
8922 3334 Acked 4333 Write 1 New Done Done false
8923 3335 Acked 4334 Read 1 New Done Done false
8924 3336 Acked 4335 Read 10 New Done Done false
8925 3337 Acked 4336 Read 10 New Done Done false
8926 3338 Acked 4337 Flush 0 New Done Done false
8927 3339 Acked 4338 Read 1 New Done Done false
8928 3340 Acked 4339 Flush 0 New Done Done false
8929 3341 Acked 4340 Flush 0 New Done Done false
8930 3342 Acked 4341 Read 7 New Done Done false
8931 3343 Acked 4342 Write 1 New Done Done false
8932 3344 Acked 4343 Write 9 New Done Done false
8933 3345 Acked 4344 Read 9 New Done Done false
8934 3346 Acked 4345 Write 1 New Done Done false
8935 3347 Acked 4346 Write 5 New Done Done false
8936 3348 Acked 4347 Read 8 New Done Done false
8937 3349 Acked 4348 Read 7 New Done Done false
8938 3350 Acked 4349 Read 9 New Done Done false
8939 3351 Acked 4350 Read 1 New Done Done false
8940 3352 Acked 4351 Write 6 New Done Done false
8941 3353 Acked 4352 Write 10 New Done Done false
8942 3354 Acked 4353 Read 8 New Done Done false
8943 3355 Acked 4354 Flush 0 New Done Done false
8944 3356 Acked 4355 Write 7 New Done Done false
8945 3357 Acked 4356 Read 8 New Done Done false
8946 3358 Acked 4357 Write 10 New Done Done false
8947 3359 Acked 4358 Read 1 New Done Done false
8948 3360 Acked 4359 Write 2 New Done Done false
8949 3361 Acked 4360 Write 3 New Done Done false
8950 3362 Acked 4361 Read 7 New Done Done false
8951 3363 Acked 4362 Write 10 New Done Done false
8952 3364 Acked 4363 Flush 0 New Done Done false
8953 3365 Acked 4364 Read 5 New Done Done false
8954 3366 Acked 4365 Read 5 New Done Done false
8955 3367 Acked 4366 Write 3 New Done Done false
8956 3368 Acked 4367 Write 10 New Done Done false
8957 3369 Acked 4368 Write 2 New Done Done false
8958 3370 Acked 4369 Read 7 New Done Done false
8959 3371 Acked 4370 Write 9 New Done Done false
8960 3372 Acked 4371 Read 1 New Done Done false
8961 3373 Acked 4372 Read 3 New Done Done false
8962 3374 Acked 4373 Write 2 New Done Done false
8963 3375 Acked 4374 Write 8 New Done Done false
8964 3376 Acked 4375 Read 9 New Done Done false
8965 3377 Acked 4376 Write 4 New Done Done false
8966 3378 Acked 4377 Write 2 New Done Done false
8967 3379 Acked 4378 Write 10 New Done Done false
8968 3380 Acked 4379 Read 6 New Done Done false
8969 3381 Acked 4380 Read 3 New Done Done false
8970 3382 Acked 4381 Read 4 New Done Done false
8971 3383 Acked 4382 Read 4 New Done Done false
8972 3384 Acked 4383 Write 8 New Done Done false
8973 3385 Acked 4384 Read 1 New Done Done false
8974 3386 Acked 4385 Read 9 New Done Done false
8975 3387 Acked 4386 Read 9 New Done Done false
8976 3388 Acked 4387 Write 1 New Done Done false
8977 3389 Acked 4388 Flush 0 New Done Done false
8978 3390 Acked 4389 Read 3 New Done Done false
8979 3391 Acked 4390 Read 7 New Done Done false
8980 3392 Acked 4391 Read 7 New Done Done false
8981 3393 Acked 4392 Write 10 New Done Done false
8982 3394 Acked 4393 Write 2 New Done Done false
8983 3395 Acked 4394 Write 6 New Done Done false
8984 3396 Acked 4395 Flush 0 New Done Done false
8985 3397 Acked 4396 Read 4 New Done Done false
8986 3398 Acked 4397 Flush 0 New Done Done false
8987 3399 Acked 4398 Read 8 New Done Done false
8988 3400 Acked 4399 Read 8 New Done Done false
8989 3401 Acked 4400 Read 9 New Done Done false
8990 3402 Acked 4401 Write 6 New Done Done false
8991 3403 Acked 4402 Write 10 New Done Done false
8992 3404 Acked 4403 Read 7 New Done Done false
8993 3405 Acked 4404 Write 10 New Done Done false
8994 3406 Acked 4405 Read 5 New Done Done false
8995 3407 Acked 4406 Read 6 New Done Done false
8996 3408 Acked 4407 Write 4 New Done Done false
8997 3409 Acked 4408 Write 2 New Done Done false
8998 3410 Acked 4409 Read 10 New Done Done false
8999 3411 Acked 4410 Read 1 New Done Done false
9000 3412 Acked 4411 Read 7 New Done Done false
9001 3413 Acked 4412 Write 4 New Done Done false
9002 3414 Acked 4413 Write 5 New Done Done false
9003 3415 Acked 4414 Write 1 New Done Done false
9004 3416 Acked 4415 Read 1 New Done Done false
9005 3417 Acked 4416 Flush 0 New Done Done false
9006 3418 Acked 4417 Write 8 New Done Done false
9007 3419 Acked 4418 Flush 0 New Done Done false
9008 3420 Acked 4419 Read 5 New Done Done false
9009 3421 Acked 4420 Read 3 New Done Done false
9010 3422 Acked 4421 Write 10 New Done Done false
9011 3423 Acked 4422 Read 4 New Done Done false
9012 3424 Acked 4423 Read 8 New Done Done false
9013 3425 Acked 4424 Write 10 New Done Done false
9014 3426 Acked 4425 Read 1 New Done Done false
9015 3427 Acked 4426 Read 2 New Done Done false
9016 3428 Acked 4427 Flush 0 New Done Done false
9017 3429 Acked 4428 Write 5 New Done Done false
9018 3430 Acked 4429 Read 5 New Done Done false
9019 3431 Acked 4430 Read 1 New Done Done false
9020 3432 Acked 4431 Write 8 New Done Done false
9021 3433 Acked 4432 Read 2 New Done Done false
9022 3434 Acked 4433 Write 8 New Done Done false
9023 3435 Acked 4434 Write 10 New Done Done false
9024 3436 Acked 4435 Read 1 New Done Done false
9025 3437 Acked 4436 Write 3 New Done Done false
9026 3438 Acked 4437 Read 6 New Done Done false
9027 3439 Acked 4438 Read 3 New Done Done false
9028 3440 Acked 4439 Flush 0 New Done Done false
9029 3441 Acked 4440 Flush 0 New Done Done false
9030 3442 Acked 4441 Read 6 New Done Done false
9031 3443 Acked 4442 Flush 0 New Done Done false
9032 3444 Acked 4443 Write 6 New Done Done false
9033 3445 Acked 4444 Write 4 New Done Done false
9034 3446 Acked 4445 Write 8 New Done Done false
9035 3447 Acked 4446 Write 5 New Done Done false
9036 3448 Acked 4447 Write 1 New Done Done false
9037 3449 Acked 4448 Write 4 New Done Done false
9038 3450 Acked 4449 Write 5 New Done Done false
9039 3451 Acked 4450 Read 10 New Done Done false
9040 3452 Acked 4451 Write 2 New Done Done false
9041 3453 Acked 4452 Read 6 New Done Done false
9042 3454 Acked 4453 Read 4 New Done Done false
9043 3455 Acked 4454 Write 9 New Done Done false
9044 3456 Acked 4455 Read 1 New Done Done false
9045 3457 Acked 4456 Flush 0 New Done Done false
9046 3458 Acked 4457 Read 2 New Done Done false
9047 3459 Acked 4458 Read 8 New Done Done false
9048 3460 Acked 4459 Read 5 New Done Done false
9049 3461 Acked 4460 Write 4 New Done Done false
9050 3462 Acked 4461 Read 9 New Done Done false
9051 3463 Acked 4462 Write 6 New Done Done false
9052 3464 Acked 4463 Read 7 New Done Done false
9053 3465 Acked 4464 Write 10 New Done Done false
9054 3466 Acked 4465 Read 2 New Done Done false
9055 3467 Acked 4466 Flush 0 New Done Done false
9056 3468 Acked 4467 Read 1 New Done Done false
9057 3469 Acked 4468 Read 5 New Done Done false
9058 3470 Acked 4469 Read 7 New Done Done false
9059 3471 Acked 4470 Write 7 New Done Done false
9060 3472 Acked 4471 Read 3 New Done Done false
9061 3473 Acked 4472 Write 9 New Done Done false
9062 3474 Acked 4473 Read 8 New Done Done false
9063 3475 Acked 4474 Read 2 New Done Done false
9064 3476 Acked 4475 Read 1 New Done Done false
9065 3477 Acked 4476 Read 8 New Done Done false
9066 3478 Acked 4477 Write 5 New Done Done false
9067 3479 Acked 4478 Write 7 New Done Done false
9068 3480 Acked 4479 Read 8 New Done Done false
9069 3481 Acked 4480 Read 4 New Done Done false
9070 3482 Acked 4481 Read 7 New Done Done false
9071 3483 Acked 4482 Read 2 New Done Done false
9072 3484 Acked 4483 Read 2 New Done Done false
9073 3485 Acked 4484 Read 8 New Done Done false
9074 3486 Acked 4485 Read 10 New Done Done false
9075 3487 Acked 4486 Read 5 New Done Done false
9076 3488 Acked 4487 Read 2 New Done Done false
9077 3489 Acked 4488 Read 3 New Done Done false
9078 3490 Acked 4489 Read 5 New Done Done false
9079 3491 Acked 4490 Write 6 New Done Done false
9080 3492 Acked 4491 Write 4 New Done Done false
9081 3493 Acked 4492 Read 5 New Done Done false
9082 3494 Acked 4493 Read 7 New Done Done false
9083 3495 Acked 4494 Write 4 New Done Done false
9084 3496 Acked 4495 Read 6 New Done Done false
9085 3497 Acked 4496 Read 6 New Done Done false
9086 3498 Acked 4497 Write 4 New Done Done false
9087 3499 Acked 4498 Flush 0 New Done Done false
9088 3500 Acked 4499 Write 2 New Done Done false
9089 3501 Acked 4500 Read 4 New Done Done false
9090 3502 Acked 4501 Write 5 New Done Done false
9091 3503 Acked 4502 Read 3 New Done Done false
9092 3504 Acked 4503 Read 2 New Done Done false
9093 3505 Acked 4504 Flush 0 New Done Done false
9094 3506 Acked 4505 Write 8 New Done Done false
9095 3507 Acked 4506 Write 9 New Done Done false
9096 3508 Acked 4507 Write 9 New Done Done false
9097 3509 Acked 4508 Write 4 New Done Done false
9098 3510 Acked 4509 Write 10 New Done Done false
9099 3511 Acked 4510 Write 5 New Done Done false
9100 3512 Acked 4511 Write 2 New Done Done false
9101 3513 Acked 4512 Write 5 New Done Done false
9102 3514 Acked 4513 Write 4 New Done Done false
9103 3515 Acked 4514 Write 7 New Done Done false
9104 3516 Acked 4515 Read 7 New Done Done false
9105 3517 Acked 4516 Write 3 New Done Done false
9106 3518 Acked 4517 Write 9 New Done Done false
9107 3519 Acked 4518 Read 7 New Done Done false
9108 3520 Acked 4519 Read 8 New Done Done false
9109 3521 Acked 4520 Write 5 New Done Done false
9110 3522 Acked 4521 Write 5 New Done Done false
9111 3523 Acked 4522 Write 10 New Done Done false
9112 3524 Acked 4523 Read 8 New Done Done false
9113 3525 Acked 4524 Read 2 New Done Done false
9114 3526 Acked 4525 Write 8 New Done Done false
9115 3527 Acked 4526 Write 8 New Done Done false
9116 3528 Acked 4527 Read 3 New Done Done false
9117 3529 Acked 4528 Read 10 New Done Done false
9118 3530 Acked 4529 Read 7 New Done Done false
9119 3531 Acked 4530 Read 10 New Done Done false
9120 3532 Acked 4531 Read 4 New Done Done false
9121 3533 Acked 4532 Read 6 New Done Done false
9122 3534 Acked 4533 Write 9 New Done Done false
9123 3535 Acked 4534 Read 3 New Done Done false
9124 3536 Acked 4535 Write 8 New Done Done false
9125 3537 Acked 4536 Read 10 New Done Done false
9126 3538 Acked 4537 Read 8 New Done Done false
9127 3539 Acked 4538 Read 8 New Done Done false
9128 3540 Acked 4539 Write 1 New Done Done false
9129 3541 Acked 4540 Read 7 New Done Done false
9130 3542 Acked 4541 Read 5 New Done Done false
9131 3543 Acked 4542 Read 1 New Done Done false
9132 3544 Acked 4543 Read 1 New Done Done false
9133 3545 Acked 4544 Read 5 New Done Done false
9134 3546 Acked 4545 Read 8 New Done Done false
9135 3547 Acked 4546 Write 2 New Done Done false
9136 3548 Acked 4547 Flush 0 New Done Done false
9137 3549 Acked 4548 Write 9 New Done Done false
9138 3550 Acked 4549 Read 1 New Done Done false
9139 3551 Acked 4550 Write 6 New Done Done false
9140 3552 Acked 4551 Read 10 New Done Done false
9141 3553 Acked 4552 Write 5 New Done Done false
9142 3554 Acked 4553 Write 3 New Done Done false
9143 3555 Acked 4554 Read 7 New Done Done false
9144 3556 Acked 4555 Write 4 New Done Done false
9145 3557 Acked 4556 Write 1 New Done Done false
9146 3558 Acked 4557 Read 2 New Done Done false
9147 3559 Acked 4558 Flush 0 New Done Done false
9148 3560 Acked 4559 Read 5 New Done Done false
9149 3561 Acked 4560 Read 5 New Done Done false
9150 3562 Acked 4561 Read 8 New Done Done false
9151 3563 Acked 4562 Read 2 New Done Done false
9152 3564 Acked 4563 Read 5 New Done Done false
9153 3565 Acked 4564 Read 5 New Done Done false
9154 3566 Acked 4565 Read 9 New Done Done false
9155 3567 Acked 4566 Read 6 New Done Done false
9156 3568 Acked 4567 Read 1 New Done Done false
9157 3569 Acked 4568 Read 3 New Done Done false
9158 3570 Acked 4569 Read 6 New Done Done false
9159 3571 Acked 4570 Write 9 New Done Done false
9160 3572 Acked 4571 Flush 0 New Done Done false
9161 3573 Acked 4572 Read 10 New Done Done false
9162 3574 Acked 4573 Read 1 New Done Done false
9163 3575 Acked 4574 Write 9 New Done Done false
9164 3576 Acked 4575 Write 2 New Done Done false
9165 3577 Acked 4576 Write 4 New Done Done false
9166 3578 Acked 4577 Write 10 New Done Done false
9167 3579 Acked 4578 Read 6 New Done Done false
9168 3580 Acked 4579 Read 5 New Done Done false
9169 3581 Acked 4580 Read 7 New Done Done false
9170 3582 Acked 4581 Read 7 New Done Done false
9171 3583 Acked 4582 Write 8 New Done Done false
9172 3584 Acked 4583 Read 2 New Done Done false
9173 3585 Acked 4584 Read 1 New Done Done false
9174 3586 Acked 4585 Read 10 New Done Done false
9175 3587 Acked 4586 Read 9 New Done Done false
9176 3588 Acked 4587 Flush 0 New Done Done false
9177 3589 Acked 4588 Write 1 New Done Done false
9178 3590 Acked 4589 Write 6 New Done Done false
9179 3591 Acked 4590 Read 7 New Done Done false
9180 3592 Acked 4591 Write 3 New Done Done false
9181 3593 Acked 4592 Write 7 New Done Done false
9182 3594 Acked 4593 Read 7 New Done Done false
9183 3595 Acked 4594 Write 9 New Done Done false
9184 3596 Acked 4595 Read 1 New Done Done false
9185 3597 Acked 4596 Read 8 New Done Done false
9186 3598 Acked 4597 Write 1 New Done Done false
9187 3599 Acked 4598 Read 8 New Done Done false
9188 3600 Acked 4599 Write 3 New Done Done false
9189 3601 Acked 4600 Read 8 New Done Done false
9190 3602 Acked 4601 Write 7 New Done Done false
9191 3603 Acked 4602 Read 9 New Done Done false
9192 3604 Acked 4603 Flush 0 New Done Done false
9193 3605 Acked 4604 Write 7 New Done Done false
9194 3606 Acked 4605 Write 7 New Done Done false
9195 3607 Acked 4606 Write 5 New Done Done false
9196 3608 Acked 4607 Read 3 New Done Done false
9197 3609 Acked 4608 Read 5 New Done Done false
9198 3610 Acked 4609 Write 8 New Done Done false
9199 3611 Acked 4610 Write 6 New Done Done false
9200 3612 Acked 4611 Read 3 New Done Done false
9201 3613 Acked 4612 Read 1 New Done Done false
9202 3614 Acked 4613 Read 5 New Done Done false
9203 3615 Acked 4614 Write 10 New Done Done false
9204 3616 Acked 4615 Write 9 New Done Done false
9205 3617 Acked 4616 Read 5 New Done Done false
9206 3618 Acked 4617 Read 7 New Done Done false
9207 3619 Acked 4618 Flush 0 New Done Done false
9208 3620 Acked 4619 Write 3 New Done Done false
9209 3621 Acked 4620 Read 1 New Done Done false
9210 3622 Acked 4621 Read 9 New Done Done false
9211 3623 Acked 4622 Write 10 New Done Done false
9212 3624 Acked 4623 Read 9 New Done Done false
9213 3625 Acked 4624 Flush 0 New Done Done false
9214 3626 Acked 4625 Read 6 New Done Done false
9215 3627 Acked 4626 Write 9 New Done Done false
9216 3628 Acked 4627 Write 8 New Done Done false
9217 3629 Acked 4628 Read 8 New Done Done false
9218 3630 Acked 4629 Flush 0 New Done Done false
9219 3631 Acked 4630 Write 2 New Sent Sent false
9220 STATES DS:0 DS:1 DS:2 TOTAL
9221 New 302 0 0 302
9222 Sent 0 1 1 2
9223 Done 0 301 301 602
9224 Skipped 0 0 0 0
9225 Error 0 0 0 0
9226 Last Flush: 4328 4629 4629
9227 Downstairs last five completed: 4328 4327 4326 4325 4324 4323
9228 Upstairs last five completed: 3631 3630 3629 3628 3627 3626
9229 CLIENT: Up:0 ds:302 act:2
9230 [8810] got start action
9231 [8810] I am going to restart
9232 Starting downstairs at port 8810
9233 Make output file at "/tmp/dsc/downstairs-8810.txt"
9234 [8810][0] reports Starting
9235 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1188)
9236 [8810][0] reports Running
92372023-10-13T18:11:04.873ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
92382023-10-13T18:11:04.874ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 0
92392023-10-13T18:11:04.874ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state Offline
92402023-10-13T18:11:04.874ZINFOcrucible: [0] client re-new 303 jobs since flush 4328 = downstairs
92412023-10-13T18:11:04.875ZINFOcrucible: [0] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
92422023-10-13T18:11:04.875ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
92432023-10-13T18:11:04.875ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
92442023-10-13T18:11:04.875ZINFOcrucible: Returning client:0 UUID:12345678-0000-0000-0000-000000008810 matches
92452023-10-13T18:11:04.875ZINFOcrucible: [0] send last flush ID to this DS: 4328
92462023-10-13T18:11:04.875ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
92472023-10-13T18:11:04.876ZINFOcrucible: [0] Replied this last flush ID: 4328
92482023-10-13T18:11:04.876ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Offline Active Active ds_transition to Replay
92492023-10-13T18:11:04.876ZINFOcrucible: [0] Transition from Offline to Replay
92502023-10-13T18:11:04.876ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
92512023-10-13T18:11:04.876ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Replay Active Active ds_transition to Active
92522023-10-13T18:11:04.876ZINFOcrucible: [0] Transition from Replay to Active
92532023-10-13T18:11:04.876ZINFOcrucible: [0] Starts cmd_loop
92542023-10-13T18:11:04.876ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
92552023-10-13T18:11:04.876ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
92562023-10-13T18:11:04.876ZINFOcrucible: Set check for repair
92572023-10-13T18:11:05.877ZINFOcrucible: Checking if live repair is needed
92582023-10-13T18:11:05.877ZINFOcrucible: No Live Repair required at this time
92592023-10-13T18:11:05.877ZWARNcrucible: [0] flow control end
9260 ----------------------------------------------------------------
9261 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9262 Downstairs last five completed: 4631 4630 4629 4628 4627 4626
9263 Upstairs last five completed: 3632 3631 3630 3629 3628 3627
9264 CLIENT: Up:0 ds:0 act:3
9265 Replay: All jobs finished, all DS active.
9266 Main task has work to do, go find it
9267 got dsc Stop(0)
9268 stop 0
9269 Oct 13 18:11:06.815 INFO request completed, latency_us: 554, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 0142e38f-7d20-4de7-87e4-1740f21b58f3, [8810] Got stop action so:false kr:false
9270 remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
9271 Oct 13 18:11:06.816 INFO request completed, latency_us: 373, response_code: 200, uri: /state/cid/0, method: GET, req_id: b73068a3-b79f-4037-9431-d149cbddd933, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
92722023-10-13T18:11:06.816ZWARNcrucible: [0] downstairs disconnected
92732023-10-13T18:11:06.817ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
92742023-10-13T18:11:06.817ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8810 closed looper = 0
92752023-10-13T18:11:06.817ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:false
92762023-10-13T18:11:06.817ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
92772023-10-13T18:11:06.817ZINFOcrucible: [0] 127.0.0.1:8810 task reports offline
92782023-10-13T18:11:06.817ZWARNcrucible: [0] pm_task rx.recv() is None
9279 [8810] Exited with: ExitStatus(unix_wait_status(9))
9280 [8810][0] reports Exit
9281 Oct 13 18:11:10.819 INFO request completed, latency_us: 255, response_code: 200, uri: /state/cid/0, method: GET, req_id: bf57fe2e-3eba-4db8-92ea-df3f5f0c2d66, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
9282 001/300 Flush
9283 002/300 Read block 4658 len 28672
9284 003/300 Write block 2599 len 40960 data: 4 4 5 7 6 5 5 4 2 2
9285 004/300 Read block 3731 len 32768
9286 005/300 Read block 4880 len 4096
9287 006/300 Write block 2014 len 36864 data: 4 7 6 6 4 4 4 4 4
9288 007/300 Write block 1741 len 32768 data: 5 5 5 5 4 2 2 2
9289 008/300 Flush
9290 009/300 Read block 1767 len 24576
9291 010/300 Read block 4164 len 4096
9292 011/300 Read block 545 len 24576
9293 012/300 Read block 2558 len 28672
9294 013/300 Write block 4082 len 4096 data: 3
9295 014/300 Read block 2458 len 4096
9296 015/300 Read block 1356 len 28672
9297 016/300 Write block 170 len 40960 data: 4 3 4 4 4 5 6 6 6 6
9298 017/300 Read block 4524 len 20480
9299 018/300 Flush
9300 019/300 Write block 2170 len 40960 data: 2 3 3 3 3 3 3 2 2 2
9301 020/300 Write block 2299 len 32768 data: 2 2 3 3 4 4 5 6
9302 021/300 Read block 4384 len 12288
9303 022/300 Write block 2960 len 40960 data: 5 5 4 4 4 4 5 3 4 4
9304 023/300 Read block 363 len 8192
9305 024/300 Write block 2745 len 40960 data: 3 4 4 4 4 3 4 4 3 4
9306 025/300 Write block 551 len 36864 data: 4 3 4 4 4 5 7 6 5
9307 026/300 Flush
9308 027/300 Write block 2393 len 20480 data: 3 3 3 5 7
9309 028/300 Read block 1308 len 36864
9310 029/300 Read block 1482 len 40960
9311 030/300 Write block 4983 len 4096 data: 3
9312 031/300 Read block 797 len 36864
9313 032/300 Read block 3246 len 12288
9314 033/300 Read block 1906 len 36864
9315 034/300 Read block 3210 len 12288
9316 035/300 Write block 890 len 40960 data: 3 3 2 2 2 2 2 2 2 2
9317 036/300 Write block 457 len 32768 data: 3 3 3 3 5 5 5 3
9318 037/300 Read block 3295 len 20480
9319 038/300 Write block 2796 len 40960 data: 3 4 5 3 3 3 2 2 2 2
9320 039/300 Flush
9321 040/300 Read block 4803 len 36864
9322 041/300 Write block 4775 len 40960 data: 2 2 3 3 4 4 4 4 3 2
9323 042/300 Flush
9324 043/300 Read block 441 len 20480
9325 044/300 Write block 3600 len 28672 data: 2 2 2 2 2 2 2
9326 045/300 Write block 2242 len 20480 data: 5 5 5 4 3
9327 046/300 Write block 130 len 12288 data: 6 6 6
9328 047/300 Flush
9329 048/300 Read block 846 len 32768
9330 049/300 Write block 1035 len 8192 data: 3 4
9331 050/300 Read block 559 len 32768
9332 051/300 Read block 2156 len 20480
9333 052/300 Read block 4744 len 36864
9334 053/300 Write block 1798 len 36864 data: 2 2 3 4 4 3 3 4 4
9335 054/300 Read block 3030 len 32768
9336 055/300 Write block 3906 len 36864 data: 5 5 6 5 5 4 5 5 5
9337 056/300 Read block 3499 len 4096
9338 057/300 Read block 3476 len 8192
9339 058/300 Write block 2133 len 8192 data: 3 3
9340 059/300 Write block 1655 len 12288 data: 2 2 3
9341 060/300 Read block 2966 len 12288
9342 061/300 Write block 3246 len 24576 data: 2 2 2 3 3 3
9343 062/300 Read block 439 len 16384
9344 063/300 Read block 2182 len 8192
9345 064/300 Read block 1692 len 24576
9346 065/300 Read block 4145 len 20480
9347 066/300 Flush
9348 067/300 Read block 3145 len 28672
9349 068/300 Write block 4595 len 32768 data: 5 5 5 4 4 4 3 2
9350 069/300 Read block 4578 len 40960
9351 070/300 Read block 3359 len 40960
9352 071/300 Read block 752 len 12288
9353 072/300 Flush
9354 073/300 Write block 2092 len 4096 data: 3
9355 074/300 Write block 3934 len 8192 data: 4 5
9356 075/300 Write block 2673 len 40960 data: 3 3 3 3 4 5 5 4 4 5
9357 076/300 Write block 4760 len 20480 data: 5 5 5 4 3
9358 077/300 Write block 3911 len 24576 data: 5 6 6 6 4 4
9359 078/300 Write block 4277 len 16384 data: 2 2 2 3
9360 079/300 Write block 3172 len 4096 data: 2
9361 080/300 Write block 4206 len 28672 data: 3 3 3 2 2 3 3
9362 081/300 Read block 1751 len 24576
9363 082/300 Read block 1352 len 12288
9364 083/300 Read block 1513 len 16384
9365 084/300 Read block 1992 len 16384
9366 085/300 Read block 427 len 32768
9367 086/300 Read block 3315 len 12288
9368 087/300 Write block 3901 len 12288 data: 2 3 3
9369 088/300 Write block 2693 len 24576 data: 2 2 3 5 4 4
9370 089/300 Flush
9371 090/300 Write block 1758 len 28672 data: 2 2 2 3 3 3 3
9372 091/300 Write block 3361 len 20480 data: 2 2 2 2 2
9373 092/300 Flush
9374 093/300 Write block 4935 len 28672 data: 5 5 4 4 5 5 4
9375 094/300 Flush
9376 095/300 Write block 2905 len 36864 data: 2 2 2 2 2 2 2 2 2
9377 096/300 Flush
9378 097/300 Write block 1557 len 24576 data: 3 2 2 2 3 3
9379 098/300 Write block 3778 len 4096 data: 5
9380 099/300 Write block 4611 len 12288 data: 3 2 2
9381 100/300 Read block 4158 len 40960
9382 101/300 Flush
9383 102/300 Read block 3100 len 24576
9384 103/300 Write block 3131 len 36864 data: 3 3 4 3 3 2 3 3 4
9385 104/300 Read block 2438 len 28672
9386 105/300 Read block 3547 len 20480
9387 106/300 Write block 3134 len 40960 data: 4 4 3 4 4 5 4 5 5 6
9388 107/300 Read block 2275 len 40960
9389 108/300 Read block 3712 len 28672
9390 109/300 Read block 1510 len 32768
9391 110/300 Write block 1928 len 8192 data: 3 3
9392 111/300 Read block 1800 len 8192
9393 112/300 Read block 1948 len 32768
9394 113/300 Write block 2561 len 40960 data: 5 4 4 3 3 3 3 3 4 4
9395 114/300 Read block 2294 len 4096
9396 115/300 Read block 4235 len 20480
9397 116/300 Read block 2050 len 24576
9398 117/300 Flush
9399 118/300 Write block 1762 len 24576 data: 4 4 4 3 3 4
9400 119/300 Write block 2086 len 12288 data: 2 2 3
9401 120/300 Read block 1031 len 32768
9402 121/300 Read block 1855 len 8192
9403 122/300 Write block 4945 len 16384 data: 3 3 4 3
9404 123/300 Write block 4176 len 4096 data: 4
9405 124/300 Flush
9406 125/300 Write block 2209 len 32768 data: 5 5 6 5 4 3 4 5
9407 126/300 Read block 1784 len 24576
9408 127/300 Flush
9409 128/300 Read block 4811 len 36864
9410 129/300 Write block 4979 len 16384 data: 3 3 3 3
9411 130/300 Write block 1732 len 28672 data: 3 2 2 3 3 4 4
9412 131/300 Write block 993 len 20480 data: 4 4 4 3 3
9413 132/300 Write block 4733 len 16384 data: 3 2 2 2
9414 133/300 Read block 4167 len 28672
9415 134/300 Write block 2491 len 20480 data: 3 4 4 3 4
9416 135/300 Write block 4740 len 40960 data: 3 3 4 5 5 5 4 4 4 4
9417 136/300 Read block 481 len 4096
9418 137/300 Read block 3599 len 28672
9419 138/300 Read block 3117 len 32768
9420 139/300 Write block 3225 len 24576 data: 2 2 2 2 2 3
9421 140/300 Write block 135 len 28672 data: 2 3 4 4 4 4 4
9422 141/300 Write block 1702 len 4096 data: 2
9423 142/300 Write block 2719 len 12288 data: 3 3 3
9424 143/300 Read block 1298 len 12288
9425 144/300 Write block 1725 len 40960 data: 3 3 2 3 3 3 3 4 3 3
9426 145/300 Read block 1036 len 12288
9427 146/300 Read block 3771 len 12288
9428 147/300 Read block 1252 len 36864
9429 148/300 Write block 1234 len 8192 data: 3 3
9430 149/300 Read block 4144 len 16384
9431 150/300 Write block 4927 len 4096 data: 2
9432 151/300 Read block 2726 len 28672
9433 152/300 Read block 2184 len 8192
9434 153/300 Read block 283 len 40960
9435 154/300 Read block 2173 len 16384
9436 155/300 Read block 3623 len 36864
9437 156/300 Read block 3184 len 40960
9438 157/300 Write block 2107 len 36864 data: 3 3 3 3 3 3 3 3 2
9439 158/300 Read block 825 len 40960
9440 159/300 Read block 234 len 12288
9441 160/300 Flush
9442 161/300 Read block 279 len 28672
9443 162/300 Write block 1848 len 36864 data: 2 2 2 2 2 3 3 3 3
9444 163/300 Read block 3565 len 40960
9445 164/300 Read block 2134 len 28672
9446 165/300 Read block 3817 len 32768
9447 166/300 Write block 129 len 12288 data: 5 7 7
9448 167/300 Read block 2247 len 40960
9449 168/300 Flush
9450 169/300 Flush
9451 170/300 Write block 1411 len 16384 data: 3 5 5 5
9452 171/300 Read block 1778 len 20480
9453 172/300 Write block 4959 len 20480 data: 2 3 3 3 4
9454 173/300 Flush
9455 174/300 Flush
9456 175/300 Write block 3654 len 16384 data: 3 3 3 2
9457 176/300 Write block 235 len 8192 data: 2 2
9458 177/300 Read block 3402 len 32768
9459 178/300 Read block 3262 len 12288
9460 179/300 Read block 4709 len 16384
9461 180/300 Flush
9462 181/300 Read block 4485 len 16384
9463 182/300 Flush
9464 183/300 Read block 2879 len 16384
9465 184/300 Write block 3307 len 8192 data: 5 4
9466 185/300 Write block 4549 len 12288 data: 2 3 3
9467 186/300 Write block 1943 len 28672 data: 2 2 2 2 3 4 5
9468 187/300 Write block 1892 len 4096 data: 3
9469 188/300 Read block 4088 len 36864
9470 189/300 Read block 1879 len 24576
9471 190/300 Write block 245 len 40960 data: 3 3 3 3 3 3 3 3 3 3
9472 191/300 Flush
9473 192/300 Read block 2537 len 40960
9474 193/300 Flush
9475 194/300 Read block 232 len 24576
9476 195/300 Write block 2351 len 8192 data: 6 6
9477 196/300 Write block 2820 len 8192 data: 4 4
9478 197/300 Read block 4563 len 24576
9479 198/300 Read block 4111 len 8192
9480 199/300 Flush
9481 200/300 Read block 4151 len 4096
9482 201/300 Write block 4871 len 32768 data: 3 3 3 3 3 3 3 4
9483 202/300 Write block 967 len 20480 data: 2 2 3 3 3
9484 203/300 Write block 3756 len 12288 data: 5 4 4
9485 204/300 Read block 350 len 8192
9486 205/300 Write block 4395 len 24576 data: 3 3 3 6 6 6
9487 206/300 Write block 3412 len 32768 data: 4 3 3 2 2 2 2 2
9488 207/300 Write block 2718 len 20480 data: 3 4 4 4 2
9489 208/300 Read block 2131 len 24576
9490 209/300 Read block 2967 len 4096
9491 210/300 Flush
9492 211/300 Read block 3080 len 12288
9493 212/300 Read block 1334 len 32768
9494 213/300 Write block 3814 len 4096 data: 3
9495 214/300 Write block 2024 len 36864 data: 4 3 4 4 4 4 4 3 4
9496 215/300 Write block 3086 len 24576 data: 3 4 4 3 3 4
9497 216/300 Read block 761 len 40960
9498 217/300 Write block 1376 len 20480 data: 5 4 5 5 5
9499 218/300 Read block 348 len 36864
9500 219/300 Write block 1696 len 4096 data: 5
9501 220/300 Write block 3383 len 12288 data: 4 4 4
9502 221/300 Read block 2030 len 24576
9503 222/300 Flush
9504 223/300 Read block 1977 len 4096
9505 224/300 Flush
9506 225/300 Read block 3544 len 4096
9507 226/300 Write block 3133 len 12288 data: 5 5 5
9508 227/300 Write block 2776 len 32768 data: 2 3 3 3 3 3 3 3
9509 228/300 Write block 2015 len 8192 data: 8 7
9510 229/300 Read block 768 len 32768
9511 230/300 Write block 2558 len 20480 data: 7 6 5 6 5
9512 231/300 Read block 2885 len 12288
9513 232/300 Write block 2069 len 8192 data: 3 3
9514 233/300 Read block 4043 len 36864
9515 234/300 Read block 1418 len 24576
9516 235/300 Flush
9517 236/300 Write block 1663 len 40960 data: 5 5 4 4 3 5 4 5 4 4
9518 237/300 Read block 3140 len 36864
9519 238/300 Read block 4069 len 28672
9520 239/300 Read block 2383 len 16384
9521 240/300 Flush
9522 241/300 Read block 1022 len 40960
9523 242/300 Write block 1861 len 36864 data: 2 2 2 2 2 3 3 3 3
9524 243/300 Read block 1584 len 4096
9525 244/300 Write block 3987 len 12288 data: 4 3 4
9526 245/300 Flush
9527 246/300 Write block 2415 len 32768 data: 4 4 2 2 2 3 3 4
9528 247/300 Write block 2994 len 4096 data: 6
9529 248/300 Read block 873 len 28672
9530 249/300 Flush
9531 250/300 Write block 3742 len 24576 data: 5 4 3 2 2 3
9532 251/300 Read block 3682 len 24576
9533 252/300 Write block 3173 len 40960 data: 3 3 3 4 4 4 4 3 3 3
9534 253/300 Write block 4404 len 16384 data: 4 6 6 4
9535 254/300 Write block 82 len 12288 data: 2 2 4
9536 255/300 Read block 2840 len 20480
9537 256/300 Write block 3365 len 8192 data: 3 2
9538 257/300 Flush
9539 258/300 Flush
9540 259/300 Flush
9541 260/300 Write block 1282 len 40960 data: 3 3 3 2 2 2 2 2 2 2
9542 261/300 Read block 1756 len 32768
9543 262/300 Flush
9544 263/300 Write block 2769 len 20480 data: 4 4 5 5 4
9545 264/300 Flush
9546 265/300 Write block 1508 len 36864 data: 2 2 2 2 2 2 2 2 2
9547 266/300 Write block 127 len 28672 data: 5 6 6 8 8 7 6
9548 267/300 Read block 2596 len 16384
9549 268/300 Write block 4784 len 24576 data: 3 2 3 4 4 4
9550 269/300 Write block 3108 len 4096 data: 4
9551 270/300 Write block 2560 len 40960 data: 6 7 6 5 4 4 4 4 4 5
9552 271/300 Write block 2314 len 24576 data: 4 4 5 4 3 2
9553 272/300 Read block 4424 len 8192
9554 273/300 Read block 710 len 24576
9555 274/300 Read block 607 len 40960
9556 275/300 Read block 2616 len 4096
9557 276/300 Read block 1312 len 24576
9558 277/300 Write block 612 len 28672 data: 2 2 2 3 3 3 3
9559 278/300 Write block 382 len 40960 data: 3 3 3 2 2 2 2 2 2 2
9560 279/300 Write block 188 len 4096 data: 5
9561 280/300 Read block 162 len 24576
9562 281/300 Read block 3337 len 16384
9563 282/300 Write block 1069 len 8192 data: 4 4
9564 283/300 Write block 3912 len 28672 data: 7 7 7 5 5 5 4
9565 284/300 Flush
9566 285/300 Read block 2255 len 32768
9567 286/300 Flush
9568 287/300 Flush
9569 288/300 Read block 4342 len 36864
9570 289/300 Read block 2105 len 8192
9571 290/300 Write block 837 len 24576 data: 4 4 4 5 4 4
9572 291/300 Read block 3934 len 20480
9573 292/300 Read block 572 len 16384
9574 293/300 Write block 2074 len 24576 data: 4 4 4 2 2 2
9575 294/300 Read block 602 len 32768
9576 295/300 Write block 4807 len 32768 data: 5 5 4 5 5 4 4 4
9577 296/300 Write block 4962 len 12288 data: 4 5 4
9578 297/300 Read block 3940 len 36864
9579 298/300 Write block 1951 len 12288 data: 5 5 5
9580 299/300 Write block 2036 len 16384 data: 2 2 2 2
9581 300/300 Write block 2489 len 20480 data: 3 3 4 5 5
9582 Main task has work to do, go find it
9583 got dsc Start(0)
9584 start 0
9585 Oct 13 18:11:11.792 INFO request completed, latency_us: 401, response_code: 204, uri: /start/cid/0, method: POST, req_id: 81b0453b-9922-4895-b8da-312acf39c7c9, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
9586 Replay: started 0, returned:Ok(())
9587 ----------------------------------------------------------------
9588 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9589 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
9590 3633 Acked 4632 Flush 0 New Done Done false
9591 3634 Acked 4633 Read 7 New Done Done false
9592 3635 Acked 4634 Write 10 New Done Done false
9593 3636 Acked 4635 Read 8 New Done Done false
9594 3637 Acked 4636 Read 1 New Done Done false
9595 3638 Acked 4637 Write 9 New Done Done false
9596 3639 Acked 4638 Write 8 New Done Done false
9597 3640 Acked 4639 Flush 0 New Done Done false
9598 3641 Acked 4640 Read 6 New Done Done false
9599 3642 Acked 4641 Read 1 New Done Done false
9600 3643 Acked 4642 Read 6 New Done Done false
9601 3644 Acked 4643 Read 7 New Done Done false
9602 3645 Acked 4644 Write 1 New Done Done false
9603 3646 Acked 4645 Read 1 New Done Done false
9604 3647 Acked 4646 Read 7 New Done Done false
9605 3648 Acked 4647 Write 10 New Done Done false
9606 3649 Acked 4648 Read 5 New Done Done false
9607 3650 Acked 4649 Flush 0 New Done Done false
9608 3651 Acked 4650 Write 10 New Done Done false
9609 3652 Acked 4651 Write 8 New Done Done false
9610 3653 Acked 4652 Read 3 New Done Done false
9611 3654 Acked 4653 Write 10 New Done Done false
9612 3655 Acked 4654 Read 2 New Done Done false
9613 3656 Acked 4655 Write 10 New Done Done false
9614 3657 Acked 4656 Write 9 New Done Done false
9615 3658 Acked 4657 Flush 0 New Done Done false
9616 3659 Acked 4658 Write 5 New Done Done false
9617 3660 Acked 4659 Read 9 New Done Done false
9618 3661 Acked 4660 Read 10 New Done Done false
9619 3662 Acked 4661 Write 1 New Done Done false
9620 3663 Acked 4662 Read 9 New Done Done false
9621 3664 Acked 4663 Read 3 New Done Done false
9622 3665 Acked 4664 Read 9 New Done Done false
9623 3666 Acked 4665 Read 3 New Done Done false
9624 3667 Acked 4666 Write 10 New Done Done false
9625 3668 Acked 4667 Write 8 New Done Done false
9626 3669 Acked 4668 Read 5 New Done Done false
9627 3670 Acked 4669 Write 10 New Done Done false
9628 3671 Acked 4670 Flush 0 New Done Done false
9629 3672 Acked 4671 Read 9 New Done Done false
9630 3673 Acked 4672 Write 10 New Done Done false
9631 3674 Acked 4673 Flush 0 New Done Done false
9632 3675 Acked 4674 Read 5 New Done Done false
9633 3676 Acked 4675 Write 7 New Done Done false
9634 3677 Acked 4676 Write 5 New Done Done false
9635 3678 Acked 4677 Write 3 New Done Done false
9636 3679 Acked 4678 Flush 0 New Done Done false
9637 3680 Acked 4679 Read 8 New Done Done false
9638 3681 Acked 4680 Write 2 New Done Done false
9639 3682 Acked 4681 Read 8 New Done Done false
9640 3683 Acked 4682 Read 5 New Done Done false
9641 3684 Acked 4683 Read 9 New Done Done false
9642 3685 Acked 4684 Write 9 New Done Done false
9643 3686 Acked 4685 Read 8 New Done Done false
9644 3687 Acked 4686 Write 9 New Done Done false
9645 3688 Acked 4687 Read 1 New Done Done false
9646 3689 Acked 4688 Read 2 New Done Done false
9647 3690 Acked 4689 Write 2 New Done Done false
9648 3691 Acked 4690 Write 3 New Done Done false
9649 3692 Acked 4691 Read 3 New Done Done false
9650 3693 Acked 4692 Write 6 New Done Done false
9651 3694 Acked 4693 Read 4 New Done Done false
9652 3695 Acked 4694 Read 2 New Done Done false
9653 3696 Acked 4695 Read 6 New Done Done false
9654 3697 Acked 4696 Read 5 New Done Done false
9655 3698 Acked 4697 Flush 0 New Done Done false
9656 3699 Acked 4698 Read 7 New Done Done false
9657 3700 Acked 4699 Write 8 New Done Done false
9658 3701 Acked 4700 Read 10 New Done Done false
9659 3702 Acked 4701 Read 10 New Done Done false
9660 3703 Acked 4702 Read 3 New Done Done false
9661 3704 Acked 4703 Flush 0 New Done Done false
9662 3705 Acked 4704 Write 1 New Done Done false
9663 3706 Acked 4705 Write 2 New Done Done false
9664 3707 Acked 4706 Write 10 New Done Done false
9665 3708 Acked 4707 Write 5 New Done Done false
9666 3709 Acked 4708 Write 6 New Done Done false
9667 3710 Acked 4709 Write 4 New Done Done false
9668 3711 Acked 4710 Write 1 New Done Done false
9669 3712 Acked 4711 Write 7 New Done Done false
9670 3713 Acked 4712 Read 6 New Done Done false
9671 3714 Acked 4713 Read 3 New Done Done false
9672 3715 Acked 4714 Read 4 New Done Done false
9673 3716 Acked 4715 Read 4 New Done Done false
9674 3717 Acked 4716 Read 8 New Done Done false
9675 3718 Acked 4717 Read 3 New Done Done false
9676 3719 Acked 4718 Write 3 New Done Done false
9677 3720 Acked 4719 Write 6 New Done Done false
9678 3721 Acked 4720 Flush 0 New Done Done false
9679 3722 Acked 4721 Write 7 New Done Done false
9680 3723 Acked 4722 Write 5 New Done Done false
9681 3724 Acked 4723 Flush 0 New Done Done false
9682 3725 Acked 4724 Write 7 New Done Done false
9683 3726 Acked 4725 Flush 0 New Done Done false
9684 3727 Acked 4726 Write 9 New Done Done false
9685 3728 Acked 4727 Flush 0 New Done Done false
9686 3729 Acked 4728 Write 6 New Done Done false
9687 3730 Acked 4729 Write 1 New Done Done false
9688 3731 Acked 4730 Write 3 New Done Done false
9689 3732 Acked 4731 Read 10 New Done Done false
9690 3733 Acked 4732 Flush 0 New Done Done false
9691 3734 Acked 4733 Read 6 New Done Done false
9692 3735 Acked 4734 Write 9 New Done Done false
9693 3736 Acked 4735 Read 7 New Done Done false
9694 3737 Acked 4736 Read 5 New Done Done false
9695 3738 Acked 4737 Write 10 New Done Done false
9696 3739 Acked 4738 Read 10 New Done Done false
9697 3740 Acked 4739 Read 7 New Done Done false
9698 3741 Acked 4740 Read 8 New Done Done false
9699 3742 Acked 4741 Write 2 New Done Done false
9700 3743 Acked 4742 Read 2 New Done Done false
9701 3744 Acked 4743 Read 8 New Done Done false
9702 3745 Acked 4744 Write 10 New Done Done false
9703 3746 Acked 4745 Read 1 New Done Done false
9704 3747 Acked 4746 Read 5 New Done Done false
9705 3748 Acked 4747 Read 6 New Done Done false
9706 3749 Acked 4748 Flush 0 New Done Done false
9707 3750 Acked 4749 Write 6 New Done Done false
9708 3751 Acked 4750 Write 3 New Done Done false
9709 3752 Acked 4751 Read 8 New Done Done false
9710 3753 Acked 4752 Read 2 New Done Done false
9711 3754 Acked 4753 Write 4 New Done Done false
9712 3755 Acked 4754 Write 1 New Done Done false
9713 3756 Acked 4755 Flush 0 New Done Done false
9714 3757 Acked 4756 Write 8 New Done Done false
9715 3758 Acked 4757 Read 6 New Done Done false
9716 3759 Acked 4758 Flush 0 New Done Done false
9717 3760 Acked 4759 Read 9 New Done Done false
9718 3761 Acked 4760 Write 4 New Done Done false
9719 3762 Acked 4761 Write 7 New Done Done false
9720 3763 Acked 4762 Write 5 New Done Done false
9721 3764 Acked 4763 Write 4 New Done Done false
9722 3765 Acked 4764 Read 7 New Done Done false
9723 3766 Acked 4765 Write 5 New Done Done false
9724 3767 Acked 4766 Write 10 New Done Done false
9725 3768 Acked 4767 Read 1 New Done Done false
9726 3769 Acked 4768 Read 7 New Done Done false
9727 3770 Acked 4769 Read 8 New Done Done false
9728 3771 Acked 4770 Write 6 New Done Done false
9729 3772 Acked 4771 Write 7 New Done Done false
9730 3773 Acked 4772 Write 1 New Done Done false
9731 3774 Acked 4773 Write 3 New Done Done false
9732 3775 Acked 4774 Read 3 New Done Done false
9733 3776 Acked 4775 Flush 0 New Done Done false
9734 3777 Acked 4776 Write 10 New Done Done false
9735 3778 Acked 4777 Read 3 New Done Done false
9736 3779 Acked 4778 Read 3 New Done Done false
9737 3780 Acked 4779 Read 9 New Done Done false
9738 3781 Acked 4780 Write 2 New Done Done false
9739 3782 Acked 4781 Read 4 New Done Done false
9740 3783 Acked 4782 Write 1 New Done Done false
9741 3784 Acked 4783 Read 7 New Done Done false
9742 3785 Acked 4784 Read 2 New Done Done false
9743 3786 Acked 4785 Read 10 New Done Done false
9744 3787 Acked 4786 Read 4 New Done Done false
9745 3788 Acked 4787 Read 9 New Done Done false
9746 3789 Acked 4788 Read 10 New Done Done false
9747 3790 Acked 4789 Write 9 New Done Done false
9748 3791 Acked 4790 Read 10 New Done Done false
9749 3792 Acked 4791 Read 3 New Done Done false
9750 3793 Acked 4792 Flush 0 New Done Done false
9751 3794 Acked 4793 Read 7 New Done Done false
9752 3795 Acked 4794 Write 9 New Done Done false
9753 3796 Acked 4795 Read 10 New Done Done false
9754 3797 Acked 4796 Read 7 New Done Done false
9755 3798 Acked 4797 Read 8 New Done Done false
9756 3799 Acked 4798 Write 3 New Done Done false
9757 3800 Acked 4799 Read 10 New Done Done false
9758 3801 Acked 4800 Flush 0 New Done Done false
9759 3802 Acked 4801 Flush 0 New Done Done false
9760 3803 Acked 4802 Write 4 New Done Done false
9761 3804 Acked 4803 Read 5 New Done Done false
9762 3805 Acked 4804 Write 5 New Done Done false
9763 3806 Acked 4805 Flush 0 New Done Done false
9764 3807 Acked 4806 Flush 0 New Done Done false
9765 3808 Acked 4807 Write 4 New Done Done false
9766 3809 Acked 4808 Write 2 New Done Done false
9767 3810 Acked 4809 Read 8 New Done Done false
9768 3811 Acked 4810 Read 3 New Done Done false
9769 3812 Acked 4811 Read 4 New Done Done false
9770 3813 Acked 4812 Flush 0 New Done Done false
9771 3814 Acked 4813 Read 4 New Done Done false
9772 3815 Acked 4814 Flush 0 New Done Done false
9773 3816 Acked 4815 Read 4 New Done Done false
9774 3817 Acked 4816 Write 2 New Done Done false
9775 3818 Acked 4817 Write 3 New Done Done false
9776 3819 Acked 4818 Write 7 New Done Done false
9777 3820 Acked 4819 Write 1 New Done Done false
9778 3821 Acked 4820 Read 9 New Done Done false
9779 3822 Acked 4821 Read 6 New Done Done false
9780 3823 Acked 4822 Write 10 New Done Done false
9781 3824 Acked 4823 Flush 0 New Done Done false
9782 3825 Acked 4824 Read 10 New Done Done false
9783 3826 Acked 4825 Flush 0 New Done Done false
9784 3827 Acked 4826 Read 6 New Done Done false
9785 3828 Acked 4827 Write 2 New Done Done false
9786 3829 Acked 4828 Write 2 New Done Done false
9787 3830 Acked 4829 Read 6 New Done Done false
9788 3831 Acked 4830 Read 2 New Done Done false
9789 3832 Acked 4831 Flush 0 New Done Done false
9790 3833 Acked 4832 Read 1 New Done Done false
9791 3834 Acked 4833 Write 8 New Done Done false
9792 3835 Acked 4834 Write 5 New Done Done false
9793 3836 Acked 4835 Write 3 New Done Done false
9794 3837 Acked 4836 Read 2 New Done Done false
9795 3838 Acked 4837 Write 6 New Done Done false
9796 3839 Acked 4838 Write 8 New Done Done false
9797 3840 Acked 4839 Write 5 New Done Done false
9798 3841 Acked 4840 Read 6 New Done Done false
9799 3842 Acked 4841 Read 1 New Done Done false
9800 3843 Acked 4842 Flush 0 New Done Done false
9801 3844 Acked 4843 Read 3 New Done Done false
9802 3845 Acked 4844 Read 8 New Done Done false
9803 3846 Acked 4845 Write 1 New Done Done false
9804 3847 Acked 4846 Write 9 New Done Done false
9805 3848 Acked 4847 Write 6 New Done Done false
9806 3849 Acked 4848 Read 10 New Done Done false
9807 3850 Acked 4849 Write 5 New Done Done false
9808 3851 Acked 4850 Read 9 New Done Done false
9809 3852 Acked 4851 Write 1 New Done Done false
9810 3853 Acked 4852 Write 3 New Done Done false
9811 3854 Acked 4853 Read 6 New Done Done false
9812 3855 Acked 4854 Flush 0 New Done Done false
9813 3856 Acked 4855 Read 1 New Done Done false
9814 3857 Acked 4856 Flush 0 New Done Done false
9815 3858 Acked 4857 Read 1 New Done Done false
9816 3859 Acked 4858 Write 3 New Done Done false
9817 3860 Acked 4859 Write 8 New Done Done false
9818 3861 Acked 4860 Write 2 New Done Done false
9819 3862 Acked 4861 Read 8 New Done Done false
9820 3863 Acked 4862 Write 5 New Done Done false
9821 3864 Acked 4863 Read 3 New Done Done false
9822 3865 Acked 4864 Write 2 New Done Done false
9823 3866 Acked 4865 Read 9 New Done Done false
9824 3867 Acked 4866 Read 6 New Done Done false
9825 3868 Acked 4867 Flush 0 New Done Done false
9826 3869 Acked 4868 Write 10 New Done Done false
9827 3870 Acked 4869 Read 9 New Done Done false
9828 3871 Acked 4870 Read 7 New Done Done false
9829 3872 Acked 4871 Read 4 New Done Done false
9830 3873 Acked 4872 Flush 0 New Done Done false
9831 3874 Acked 4873 Read 10 New Done Done false
9832 3875 Acked 4874 Write 9 New Done Done false
9833 3876 Acked 4875 Read 1 New Done Done false
9834 3877 Acked 4876 Write 3 New Done Done false
9835 3878 Acked 4877 Flush 0 New Done Done false
9836 3879 Acked 4878 Write 8 New Done Done false
9837 3880 Acked 4879 Write 1 New Done Done false
9838 3881 Acked 4880 Read 7 New Done Done false
9839 3882 Acked 4881 Flush 0 New Done Done false
9840 3883 Acked 4882 Write 6 New Done Done false
9841 3884 Acked 4883 Read 6 New Done Done false
9842 3885 Acked 4884 Write 10 New Done Done false
9843 3886 Acked 4885 Write 4 New Done Done false
9844 3887 Acked 4886 Write 3 New Done Done false
9845 3888 Acked 4887 Read 5 New Done Done false
9846 3889 Acked 4888 Write 2 New Done Done false
9847 3890 Acked 4889 Flush 0 New Done Done false
9848 3891 Acked 4890 Flush 0 New Done Done false
9849 3892 Acked 4891 Flush 0 New Done Done false
9850 3893 Acked 4892 Write 10 New Done Done false
9851 3894 Acked 4893 Read 8 New Done Done false
9852 3895 Acked 4894 Flush 0 New Done Done false
9853 3896 Acked 4895 Write 5 New Done Done false
9854 3897 Acked 4896 Flush 0 New Done Done false
9855 3898 Acked 4897 Write 9 New Done Done false
9856 3899 Acked 4898 Write 7 New Done Done false
9857 3900 Acked 4899 Read 4 New Done Done false
9858 3901 Acked 4900 Write 6 New Done Done false
9859 3902 Acked 4901 Write 1 New Done Done false
9860 3903 Acked 4902 Write 10 New Done Done false
9861 3904 Acked 4903 Write 6 New Done Done false
9862 3905 Acked 4904 Read 2 New Done Done false
9863 3906 Acked 4905 Read 6 New Done Done false
9864 3907 Acked 4906 Read 10 New Done Done false
9865 3908 Acked 4907 Read 1 New Done Done false
9866 3909 Acked 4908 Read 6 New Done Done false
9867 3910 Acked 4909 Write 7 New Done Done false
9868 3911 Acked 4910 Write 10 New Done Done false
9869 3912 Acked 4911 Write 1 New Done Done false
9870 3913 Acked 4912 Read 6 New Done Done false
9871 3914 Acked 4913 Read 4 New Done Done false
9872 3915 Acked 4914 Write 2 New Done Done false
9873 3916 Acked 4915 Write 7 New Done Done false
9874 3917 Acked 4916 Flush 0 New Done Done false
9875 3918 Acked 4917 Read 8 New Done Done false
9876 3919 Acked 4918 Flush 0 New Done Done false
9877 3920 Acked 4919 Flush 0 New Done Done false
9878 3921 Acked 4920 Read 9 New Done Done false
9879 3922 Acked 4921 Read 2 New Done Done false
9880 3923 Acked 4922 Write 6 New Done Done false
9881 3924 Acked 4923 Read 5 New Done Done false
9882 3925 Acked 4924 Read 4 New Done Done false
9883 3926 Acked 4925 Write 6 New Done Done false
9884 3927 Acked 4926 Read 8 New Done Done false
9885 3928 Acked 4927 Write 8 New Done Done false
9886 3929 Acked 4928 Write 3 New Done Done false
9887 3930 Acked 4929 Read 9 New Done Done false
9888 3931 Acked 4930 Write 3 New Done Done false
9889 3932 Acked 4931 Write 4 New Sent Sent false
9890 3933 Acked 4932 Write 5 New Sent Sent false
9891 STATES DS:0 DS:1 DS:2 TOTAL
9892 New 301 0 0 301
9893 Sent 0 2 2 4
9894 Done 0 299 299 598
9895 Skipped 0 0 0 0
9896 Error 0 0 0 0
9897 Last Flush: 4631 4919 4919
9898 Downstairs last five completed: 4631 4630 4629 4628 4627 4626
9899 Upstairs last five completed: 3933 3932 3931 3930 3929 3928
9900 CLIENT: Up:0 ds:301 act:2
9901 [8810] got start action
9902 [8810] I am going to restart
9903 Starting downstairs at port 8810
9904 Make output file at "/tmp/dsc/downstairs-8810.txt"
9905 [8810][0] reports Starting
9906 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1189)
9907 [8810][0] reports Running
9908 ----------------------------------------------------------------
9909 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
9910 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
9911 3633 Acked 4632 Flush 0 New Done Done false
9912 3634 Acked 4633 Read 7 New Done Done false
9913 3635 Acked 4634 Write 10 New Done Done false
9914 3636 Acked 4635 Read 8 New Done Done false
9915 3637 Acked 4636 Read 1 New Done Done false
9916 3638 Acked 4637 Write 9 New Done Done false
9917 3639 Acked 4638 Write 8 New Done Done false
9918 3640 Acked 4639 Flush 0 New Done Done false
9919 3641 Acked 4640 Read 6 New Done Done false
9920 3642 Acked 4641 Read 1 New Done Done false
9921 3643 Acked 4642 Read 6 New Done Done false
9922 3644 Acked 4643 Read 7 New Done Done false
9923 3645 Acked 4644 Write 1 New Done Done false
9924 3646 Acked 4645 Read 1 New Done Done false
9925 3647 Acked 4646 Read 7 New Done Done false
9926 3648 Acked 4647 Write 10 New Done Done false
9927 3649 Acked 4648 Read 5 New Done Done false
9928 3650 Acked 4649 Flush 0 New Done Done false
9929 3651 Acked 4650 Write 10 New Done Done false
9930 3652 Acked 4651 Write 8 New Done Done false
9931 3653 Acked 4652 Read 3 New Done Done false
9932 3654 Acked 4653 Write 10 New Done Done false
9933 3655 Acked 4654 Read 2 New Done Done false
9934 3656 Acked 4655 Write 10 New Done Done false
9935 3657 Acked 4656 Write 9 New Done Done false
9936 3658 Acked 4657 Flush 0 New Done Done false
9937 3659 Acked 4658 Write 5 New Done Done false
9938 3660 Acked 4659 Read 9 New Done Done false
9939 3661 Acked 4660 Read 10 New Done Done false
9940 3662 Acked 4661 Write 1 New Done Done false
9941 3663 Acked 4662 Read 9 New Done Done false
9942 3664 Acked 4663 Read 3 New Done Done false
9943 3665 Acked 4664 Read 9 New Done Done false
9944 3666 Acked 4665 Read 3 New Done Done false
9945 3667 Acked 4666 Write 10 New Done Done false
9946 3668 Acked 4667 Write 8 New Done Done false
9947 3669 Acked 4668 Read 5 New Done Done false
9948 3670 Acked 4669 Write 10 New Done Done false
9949 3671 Acked 4670 Flush 0 New Done Done false
9950 3672 Acked 4671 Read 9 New Done Done false
9951 3673 Acked 4672 Write 10 New Done Done false
9952 3674 Acked 4673 Flush 0 New Done Done false
9953 3675 Acked 4674 Read 5 New Done Done false
9954 3676 Acked 4675 Write 7 New Done Done false
9955 3677 Acked 4676 Write 5 New Done Done false
9956 3678 Acked 4677 Write 3 New Done Done false
9957 3679 Acked 4678 Flush 0 New Done Done false
9958 3680 Acked 4679 Read 8 New Done Done false
9959 3681 Acked 4680 Write 2 New Done Done false
9960 3682 Acked 4681 Read 8 New Done Done false
9961 3683 Acked 4682 Read 5 New Done Done false
9962 3684 Acked 4683 Read 9 New Done Done false
9963 3685 Acked 4684 Write 9 New Done Done false
9964 3686 Acked 4685 Read 8 New Done Done false
9965 3687 Acked 4686 Write 9 New Done Done false
9966 3688 Acked 4687 Read 1 New Done Done false
9967 3689 Acked 4688 Read 2 New Done Done false
9968 3690 Acked 4689 Write 2 New Done Done false
9969 3691 Acked 4690 Write 3 New Done Done false
9970 3692 Acked 4691 Read 3 New Done Done false
9971 3693 Acked 4692 Write 6 New Done Done false
9972 3694 Acked 4693 Read 4 New Done Done false
9973 3695 Acked 4694 Read 2 New Done Done false
9974 3696 Acked 4695 Read 6 New Done Done false
9975 3697 Acked 4696 Read 5 New Done Done false
9976 3698 Acked 4697 Flush 0 New Done Done false
9977 3699 Acked 4698 Read 7 New Done Done false
9978 3700 Acked 4699 Write 8 New Done Done false
9979 3701 Acked 4700 Read 10 New Done Done false
9980 3702 Acked 4701 Read 10 New Done Done false
9981 3703 Acked 4702 Read 3 New Done Done false
9982 3704 Acked 4703 Flush 0 New Done Done false
9983 3705 Acked 4704 Write 1 New Done Done false
9984 3706 Acked 4705 Write 2 New Done Done false
9985 3707 Acked 4706 Write 10 New Done Done false
9986 3708 Acked 4707 Write 5 New Done Done false
9987 3709 Acked 4708 Write 6 New Done Done false
9988 3710 Acked 4709 Write 4 New Done Done false
9989 3711 Acked 4710 Write 1 New Done Done false
9990 3712 Acked 4711 Write 7 New Done Done false
9991 3713 Acked 4712 Read 6 New Done Done false
9992 3714 Acked 4713 Read 3 New Done Done false
9993 3715 Acked 4714 Read 4 New Done Done false
9994 3716 Acked 4715 Read 4 New Done Done false
9995 3717 Acked 4716 Read 8 New Done Done false
9996 3718 Acked 4717 Read 3 New Done Done false
9997 3719 Acked 4718 Write 3 New Done Done false
9998 3720 Acked 4719 Write 6 New Done Done false
9999 3721 Acked 4720 Flush 0 New Done Done false
10000 3722 Acked 4721 Write 7 New Done Done false
10001 3723 Acked 4722 Write 5 New Done Done false
10002 3724 Acked 4723 Flush 0 New Done Done false
10003 3725 Acked 4724 Write 7 New Done Done false
10004 3726 Acked 4725 Flush 0 New Done Done false
10005 3727 Acked 4726 Write 9 New Done Done false
10006 3728 Acked 4727 Flush 0 New Done Done false
10007 3729 Acked 4728 Write 6 New Done Done false
10008 3730 Acked 4729 Write 1 New Done Done false
10009 3731 Acked 4730 Write 3 New Done Done false
10010 3732 Acked 4731 Read 10 New Done Done false
10011 3733 Acked 4732 Flush 0 New Done Done false
10012 3734 Acked 4733 Read 6 New Done Done false
10013 3735 Acked 4734 Write 9 New Done Done false
10014 3736 Acked 4735 Read 7 New Done Done false
10015 3737 Acked 4736 Read 5 New Done Done false
10016 3738 Acked 4737 Write 10 New Done Done false
10017 3739 Acked 4738 Read 10 New Done Done false
10018 3740 Acked 4739 Read 7 New Done Done false
10019 3741 Acked 4740 Read 8 New Done Done false
10020 3742 Acked 4741 Write 2 New Done Done false
10021 3743 Acked 4742 Read 2 New Done Done false
10022 3744 Acked 4743 Read 8 New Done Done false
10023 3745 Acked 4744 Write 10 New Done Done false
10024 3746 Acked 4745 Read 1 New Done Done false
10025 3747 Acked 4746 Read 5 New Done Done false
10026 3748 Acked 4747 Read 6 New Done Done false
10027 3749 Acked 4748 Flush 0 New Done Done false
10028 3750 Acked 4749 Write 6 New Done Done false
10029 3751 Acked 4750 Write 3 New Done Done false
10030 3752 Acked 4751 Read 8 New Done Done false
10031 3753 Acked 4752 Read 2 New Done Done false
10032 3754 Acked 4753 Write 4 New Done Done false
10033 3755 Acked 4754 Write 1 New Done Done false
10034 3756 Acked 4755 Flush 0 New Done Done false
10035 3757 Acked 4756 Write 8 New Done Done false
10036 3758 Acked 4757 Read 6 New Done Done false
10037 3759 Acked 4758 Flush 0 New Done Done false
10038 3760 Acked 4759 Read 9 New Done Done false
10039 3761 Acked 4760 Write 4 New Done Done false
10040 3762 Acked 4761 Write 7 New Done Done false
10041 3763 Acked 4762 Write 5 New Done Done false
10042 3764 Acked 4763 Write 4 New Done Done false
10043 3765 Acked 4764 Read 7 New Done Done false
10044 3766 Acked 4765 Write 5 New Done Done false
10045 3767 Acked 4766 Write 10 New Done Done false
10046 3768 Acked 4767 Read 1 New Done Done false
10047 3769 Acked 4768 Read 7 New Done Done false
10048 3770 Acked 4769 Read 8 New Done Done false
10049 3771 Acked 4770 Write 6 New Done Done false
10050 3772 Acked 4771 Write 7 New Done Done false
10051 3773 Acked 4772 Write 1 New Done Done false
10052 3774 Acked 4773 Write 3 New Done Done false
10053 3775 Acked 4774 Read 3 New Done Done false
10054 3776 Acked 4775 Flush 0 New Done Done false
10055 3777 Acked 4776 Write 10 New Done Done false
10056 3778 Acked 4777 Read 3 New Done Done false
10057 3779 Acked 4778 Read 3 New Done Done false
10058 3780 Acked 4779 Read 9 New Done Done false
10059 3781 Acked 4780 Write 2 New Done Done false
10060 3782 Acked 4781 Read 4 New Done Done false
10061 3783 Acked 4782 Write 1 New Done Done false
10062 3784 Acked 4783 Read 7 New Done Done false
10063 3785 Acked 4784 Read 2 New Done Done false
10064 3786 Acked 4785 Read 10 New Done Done false
10065 3787 Acked 4786 Read 4 New Done Done false
10066 3788 Acked 4787 Read 9 New Done Done false
10067 3789 Acked 4788 Read 10 New Done Done false
10068 3790 Acked 4789 Write 9 New Done Done false
10069 3791 Acked 4790 Read 10 New Done Done false
10070 3792 Acked 4791 Read 3 New Done Done false
10071 3793 Acked 4792 Flush 0 New Done Done false
10072 3794 Acked 4793 Read 7 New Done Done false
10073 3795 Acked 4794 Write 9 New Done Done false
10074 3796 Acked 4795 Read 10 New Done Done false
10075 3797 Acked 4796 Read 7 New Done Done false
10076 3798 Acked 4797 Read 8 New Done Done false
10077 3799 Acked 4798 Write 3 New Done Done false
10078 3800 Acked 4799 Read 10 New Done Done false
10079 3801 Acked 4800 Flush 0 New Done Done false
10080 3802 Acked 4801 Flush 0 New Done Done false
10081 3803 Acked 4802 Write 4 New Done Done false
10082 3804 Acked 4803 Read 5 New Done Done false
10083 3805 Acked 4804 Write 5 New Done Done false
10084 3806 Acked 4805 Flush 0 New Done Done false
10085 3807 Acked 4806 Flush 0 New Done Done false
10086 3808 Acked 4807 Write 4 New Done Done false
10087 3809 Acked 4808 Write 2 New Done Done false
10088 3810 Acked 4809 Read 8 New Done Done false
10089 3811 Acked 4810 Read 3 New Done Done false
10090 3812 Acked 4811 Read 4 New Done Done false
10091 3813 Acked 4812 Flush 0 New Done Done false
10092 3814 Acked 4813 Read 4 New Done Done false
10093 3815 Acked 4814 Flush 0 New Done Done false
10094 3816 Acked 4815 Read 4 New Done Done false
10095 3817 Acked 4816 Write 2 New Done Done false
10096 3818 Acked 4817 Write 3 New Done Done false
10097 3819 Acked 4818 Write 7 New Done Done false
10098 3820 Acked 4819 Write 1 New Done Done false
10099 3821 Acked 4820 Read 9 New Done Done false
10100 3822 Acked 4821 Read 6 New Done Done false
10101 3823 Acked 4822 Write 10 New Done Done false
10102 3824 Acked 4823 Flush 0 New Done Done false
10103 3825 Acked 4824 Read 10 New Done Done false
10104 3826 Acked 4825 Flush 0 New Done Done false
10105 3827 Acked 4826 Read 6 New Done Done false
10106 3828 Acked 4827 Write 2 New Done Done false
10107 3829 Acked 4828 Write 2 New Done Done false
10108 3830 Acked 4829 Read 6 New Done Done false
10109 3831 Acked 4830 Read 2 New Done Done false
10110 3832 Acked 4831 Flush 0 New Done Done false
10111 3833 Acked 4832 Read 1 New Done Done false
10112 3834 Acked 4833 Write 8 New Done Done false
10113 3835 Acked 4834 Write 5 New Done Done false
10114 3836 Acked 4835 Write 3 New Done Done false
10115 3837 Acked 4836 Read 2 New Done Done false
10116 3838 Acked 4837 Write 6 New Done Done false
10117 3839 Acked 4838 Write 8 New Done Done false
10118 3840 Acked 4839 Write 5 New Done Done false
10119 3841 Acked 4840 Read 6 New Done Done false
10120 3842 Acked 4841 Read 1 New Done Done false
10121 3843 Acked 4842 Flush 0 New Done Done false
10122 3844 Acked 4843 Read 3 New Done Done false
10123 3845 Acked 4844 Read 8 New Done Done false
10124 3846 Acked 4845 Write 1 New Done Done false
10125 3847 Acked 4846 Write 9 New Done Done false
10126 3848 Acked 4847 Write 6 New Done Done false
10127 3849 Acked 4848 Read 10 New Done Done false
10128 3850 Acked 4849 Write 5 New Done Done false
10129 3851 Acked 4850 Read 9 New Done Done false
10130 3852 Acked 4851 Write 1 New Done Done false
10131 3853 Acked 4852 Write 3 New Done Done false
10132 3854 Acked 4853 Read 6 New Done Done false
10133 3855 Acked 4854 Flush 0 New Done Done false
10134 3856 Acked 4855 Read 1 New Done Done false
10135 3857 Acked 4856 Flush 0 New Done Done false
10136 3858 Acked 4857 Read 1 New Done Done false
10137 3859 Acked 4858 Write 3 New Done Done false
10138 3860 Acked 4859 Write 8 New Done Done false
10139 3861 Acked 4860 Write 2 New Done Done false
10140 3862 Acked 4861 Read 8 New Done Done false
10141 3863 Acked 4862 Write 5 New Done Done false
10142 3864 Acked 4863 Read 3 New Done Done false
10143 3865 Acked 4864 Write 2 New Done Done false
10144 3866 Acked 4865 Read 9 New Done Done false
10145 3867 Acked 4866 Read 6 New Done Done false
10146 3868 Acked 4867 Flush 0 New Done Done false
10147 3869 Acked 4868 Write 10 New Done Done false
10148 3870 Acked 4869 Read 9 New Done Done false
10149 3871 Acked 4870 Read 7 New Done Done false
10150 3872 Acked 4871 Read 4 New Done Done false
10151 3873 Acked 4872 Flush 0 New Done Done false
10152 3874 Acked 4873 Read 10 New Done Done false
10153 3875 Acked 4874 Write 9 New Done Done false
10154 3876 Acked 4875 Read 1 New Done Done false
10155 3877 Acked 4876 Write 3 New Done Done false
10156 3878 Acked 4877 Flush 0 New Done Done false
10157 3879 Acked 4878 Write 8 New Done Done false
10158 3880 Acked 4879 Write 1 New Done Done false
10159 3881 Acked 4880 Read 7 New Done Done false
10160 3882 Acked 4881 Flush 0 New Done Done false
10161 3883 Acked 4882 Write 6 New Done Done false
10162 3884 Acked 4883 Read 6 New Done Done false
10163 3885 Acked 4884 Write 10 New Done Done false
10164 3886 Acked 4885 Write 4 New Done Done false
10165 3887 Acked 4886 Write 3 New Done Done false
10166 3888 Acked 4887 Read 5 New Done Done false
10167 3889 Acked 4888 Write 2 New Done Done false
10168 3890 Acked 4889 Flush 0 New Done Done false
10169 3891 Acked 4890 Flush 0 New Done Done false
10170 3892 Acked 4891 Flush 0 New Done Done false
10171 3893 Acked 4892 Write 10 New Done Done false
10172 3894 Acked 4893 Read 8 New Done Done false
10173 3895 Acked 4894 Flush 0 New Done Done false
10174 3896 Acked 4895 Write 5 New Done Done false
10175 3897 Acked 4896 Flush 0 New Done Done false
10176 3898 Acked 4897 Write 9 New Done Done false
10177 3899 Acked 4898 Write 7 New Done Done false
10178 3900 Acked 4899 Read 4 New Done Done false
10179 3901 Acked 4900 Write 6 New Done Done false
10180 3902 Acked 4901 Write 1 New Done Done false
10181 3903 Acked 4902 Write 10 New Done Done false
10182 3904 Acked 4903 Write 6 New Done Done false
10183 3905 Acked 4904 Read 2 New Done Done false
10184 3906 Acked 4905 Read 6 New Done Done false
10185 3907 Acked 4906 Read 10 New Done Done false
10186 3908 Acked 4907 Read 1 New Done Done false
10187 3909 Acked 4908 Read 6 New Done Done false
10188 3910 Acked 4909 Write 7 New Done Done false
10189 3911 Acked 4910 Write 10 New Done Done false
10190 3912 Acked 4911 Write 1 New Done Done false
10191 3913 Acked 4912 Read 6 New Done Done false
10192 3914 Acked 4913 Read 4 New Done Done false
10193 3915 Acked 4914 Write 2 New Done Done false
10194 3916 Acked 4915 Write 7 New Done Done false
10195 3917 Acked 4916 Flush 0 New Done Done false
10196 3918 Acked 4917 Read 8 New Done Done false
10197 3919 Acked 4918 Flush 0 New Done Done false
10198 3920 Acked 4919 Flush 0 New Done Done false
10199 3921 Acked 4920 Read 9 New Done Done false
10200 3922 Acked 4921 Read 2 New Done Done false
10201 3923 Acked 4922 Write 6 New Done Done false
10202 3924 Acked 4923 Read 5 New Done Done false
10203 3925 Acked 4924 Read 4 New Done Done false
10204 3926 Acked 4925 Write 6 New Done Done false
10205 3927 Acked 4926 Read 8 New Done Done false
10206 3928 Acked 4927 Write 8 New Done Done false
10207 3929 Acked 4928 Write 3 New Done Done false
10208 3930 Acked 4929 Read 9 New Done Done false
10209 3931 Acked 4930 Write 3 New Done Done false
10210 3932 Acked 4931 Write 4 New Done Done false
10211 3933 Acked 4932 Write 5 New Done Done false
10212 3934 Acked 4933 Flush 0 New Done Done false
10213 STATES DS:0 DS:1 DS:2 TOTAL
10214 New 302 0 0 302
10215 Sent 0 0 0 0
10216 Done 0 302 302 604
10217 Skipped 0 0 0 0
10218 Error 0 0 0 0
10219 Last Flush: 4631 4933 4933
10220 Downstairs last five completed: 4631 4630 4629 4628 4627 4626
10221 Upstairs last five completed: 3934 3933 3932 3931 3930 3929
10222 CLIENT: Up:0 ds:302 act:2
102232023-10-13T18:11:15.828ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 0
102242023-10-13T18:11:15.828ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state Offline
102252023-10-13T18:11:15.828ZINFOcrucible: [0] client re-new 302 jobs since flush 4631 = downstairs
102262023-10-13T18:11:15.829ZINFOcrucible: [0] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
102272023-10-13T18:11:15.830ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
102282023-10-13T18:11:15.830ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
102292023-10-13T18:11:15.830ZINFOcrucible: Returning client:0 UUID:12345678-0000-0000-0000-000000008810 matches
102302023-10-13T18:11:15.830ZINFOcrucible: [0] send last flush ID to this DS: 4631
102312023-10-13T18:11:15.830ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
102322023-10-13T18:11:15.830ZINFOcrucible: [0] Replied this last flush ID: 4631
102332023-10-13T18:11:15.830ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Offline Active Active ds_transition to Replay
102342023-10-13T18:11:15.830ZINFOcrucible: [0] Transition from Offline to Replay
102352023-10-13T18:11:15.830ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
102362023-10-13T18:11:15.830ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Replay Active Active ds_transition to Active
102372023-10-13T18:11:15.830ZINFOcrucible: [0] Transition from Replay to Active
102382023-10-13T18:11:15.830ZINFOcrucible: [0] Starts cmd_loop
102392023-10-13T18:11:15.831ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
102402023-10-13T18:11:15.831ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
102412023-10-13T18:11:15.831ZINFOcrucible: Set check for repair
102422023-10-13T18:11:16.831ZWARNcrucible: [0] flow control end
102432023-10-13T18:11:16.832ZINFOcrucible: Checking if live repair is needed
102442023-10-13T18:11:16.832ZINFOcrucible: No Live Repair required at this time
10245 ----------------------------------------------------------------
10246 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10247 Downstairs last five completed: 4933 4932 4931 4930 4929 4928
10248 Upstairs last five completed: 3934 3933 3932 3931 3930 3929
10249 CLIENT: Up:0 ds:0 act:3
10250 Replay: All jobs finished, all DS active.
10251 Main task has work to do, go find it
10252 got dsc Stop(1)
10253 stop 1
10254 Oct 13 18:11:19.799 INFO request completed, latency_us: 520, response_code: [8820] Got stop action so:false kr:false
10255 204, uri: /stop/cid/1, method: POST, req_id: 51f62e66-ffc7-4cc6-8136-7d4455e76338, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
10256 Oct 13 18:11:19.801 INFO request completed, latency_us: 234, response_code: 200, uri: /state/cid/1, method: GET, req_id: 281427d5-cd8c-4a05-8f28-a4ebf70fde5c, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
102572023-10-13T18:11:19.801ZWARNcrucible: [1] downstairs disconnected
102582023-10-13T18:11:19.801ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
102592023-10-13T18:11:19.801ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8820 closed looper = 1
102602023-10-13T18:11:19.801ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
102612023-10-13T18:11:19.801ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
102622023-10-13T18:11:19.802ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
102632023-10-13T18:11:19.802ZWARNcrucible: [1] pm_task rx.recv() is None
10264 [8820] Exited with: ExitStatus(unix_wait_status(9))
10265 [8820][1] reports Exit
10266 Oct 13 18:11:23.803 INFO request completed, latency_us: 403, response_code: 200, uri: /state/cid/1, method: GET, req_id: 57ba69b6-93ab-4dca-9881-3a89ca8db7a9, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
10267 001/300 Read block 1615 len 24576
10268 002/300 Flush
10269 003/300 Read block 3691 len 36864
10270 004/300 Write block 1271 len 8192 data: 2 3
10271 005/300 Flush
10272 006/300 Read block 3781 len 20480
10273 007/300 Write block 476 len 20480 data: 3 3 2 2 2
10274 008/300 Read block 313 len 28672
10275 009/300 Read block 4962 len 24576
10276 010/300 Write block 3206 len 24576 data: 3 3 4 3 4 4
10277 011/300 Read block 676 len 28672
10278 012/300 Read block 4216 len 16384
10279 013/300 Read block 4373 len 16384
10280 014/300 Write block 209 len 8192 data: 5 4
10281 015/300 Read block 2596 len 4096
10282 016/300 Flush
10283 017/300 Write block 274 len 24576 data: 3 3 2 2 3 2
10284 018/300 Write block 3406 len 12288 data: 4 4 4
10285 019/300 Read block 1879 len 4096
10286 020/300 Read block 3999 len 12288
10287 021/300 Read block 1140 len 36864
10288 022/300 Write block 1106 len 36864 data: 2 2 2 2 2 2 2 2 3
10289 023/300 Write block 148 len 40960 data: 3 3 4 4 4 4 4 3 3 3
10290 024/300 Write block 2525 len 32768 data: 6 6 6 5 5 4 5 5
10291 025/300 Read block 684 len 24576
10292 026/300 Write block 162 len 4096 data: 5
10293 027/300 Read block 1001 len 8192
10294 028/300 Read block 538 len 28672
10295 029/300 Read block 1933 len 20480
10296 030/300 Read block 890 len 12288
10297 031/300 Read block 30 len 32768
10298 032/300 Read block 4121 len 40960
10299 033/300 Read block 607 len 24576
10300 034/300 Write block 1680 len 40960 data: 2 2 2 3 3 2 2 2 2 2
10301 035/300 Flush
10302 036/300 Write block 1347 len 40960 data: 3 3 3 2 2 2 2 2 2 2
10303 037/300 Read block 2314 len 40960
10304 038/300 Read block 3085 len 40960
10305 039/300 Write block 1514 len 8192 data: 3 3
10306 040/300 Write block 1614 len 36864 data: 4 3 4 3 4 4 4 3 3
10307 041/300 Read block 3826 len 36864
10308 042/300 Read block 1892 len 36864
10309 043/300 Write block 1750 len 12288 data: 2 2 2
10310 044/300 Write block 3955 len 28672 data: 6 5 5 4 5 4 3
10311 045/300 Read block 366 len 40960
10312 046/300 Write block 460 len 32768 data: 4 6 6 6 4 2 3 3
10313 047/300 Write block 1392 len 32768 data: 4 4 4 5 4 4 4 5
10314 048/300 Read block 4892 len 32768
10315 049/300 Write block 4908 len 40960 data: 4 3 3 3 3 2 4 4 5 5
10316 050/300 Flush
10317 051/300 Write block 1634 len 16384 data: 7 6 5 5
10318 052/300 Read block 2107 len 32768
10319 053/300 Read block 4591 len 24576
10320 054/300 Write block 1339 len 32768 data: 4 4 4 4 5 5 4 4
10321 055/300 Write block 4341 len 28672 data: 3 3 3 2 3 3 4
10322 056/300 Read block 1155 len 36864
10323 057/300 Write block 4952 len 36864 data: 2 2 2 2 2 2 2 3 4
10324 058/300 Read block 1638 len 8192
10325 059/300 Read block 3337 len 20480
10326 060/300 Read block 1565 len 20480
10327 061/300 Write block 894 len 28672 data: 3 3 3 3 3 3 2
10328 062/300 Flush
10329 063/300 Read block 323 len 36864
10330 064/300 Write block 215 len 16384 data: 4 4 3 4
10331 065/300 Read block 147 len 16384
10332 066/300 Write block 1896 len 28672 data: 3 4 4 3 3 3 3
10333 067/300 Write block 3857 len 16384 data: 4 4 2 2
10334 068/300 Read block 1974 len 16384
10335 069/300 Write block 2435 len 36864 data: 2 2 2 2 2 3 3 3 3
10336 070/300 Read block 1088 len 40960
10337 071/300 Flush
10338 072/300 Write block 74 len 40960 data: 6 3 3 3 3 4 4 4 3 3
10339 073/300 Read block 3055 len 36864
10340 074/300 Read block 1654 len 24576
10341 075/300 Write block 1354 len 8192 data: 3 3
10342 076/300 Read block 2221 len 36864
10343 077/300 Write block 664 len 24576 data: 3 4 5 3 2 2
10344 078/300 Write block 4374 len 12288 data: 6 5 5
10345 079/300 Write block 583 len 32768 data: 2 2 3 3 4 3 3 3
10346 080/300 Write block 2983 len 12288 data: 2 3 3
10347 081/300 Read block 3808 len 20480
10348 082/300 Read block 4215 len 24576
10349 083/300 Read block 4647 len 4096
10350 084/300 Write block 3738 len 36864 data: 5 5 4 5 6 5 4 3 3
10351 085/300 Read block 1943 len 36864
10352 086/300 Read block 1621 len 36864
10353 087/300 Write block 3235 len 24576 data: 5 5 4 4 4 4
10354 088/300 Read block 2247 len 36864
10355 089/300 Read block 1639 len 36864
10356 090/300 Read block 1361 len 20480
10357 091/300 Read block 1004 len 36864
10358 092/300 Read block 2919 len 12288
10359 093/300 Read block 3499 len 16384
10360 094/300 Flush
10361 095/300 Read block 1171 len 28672
10362 096/300 Write block 3384 len 24576 data: 5 5 3 4 4 4
10363 097/300 Read block 725 len 32768
10364 098/300 Read block 2817 len 36864
10365 099/300 Write block 922 len 4096 data: 4
10366 100/300 Write block 925 len 4096 data: 3
10367 101/300 Read block 3431 len 20480
10368 102/300 Read block 75 len 8192
10369 103/300 Write block 254 len 16384 data: 4 3 3 2
10370 104/300 Read block 4498 len 20480
10371 105/300 Write block 1000 len 16384 data: 2 3 3 3
10372 106/300 Write block 3293 len 8192 data: 3 3
10373 107/300 Read block 609 len 40960
10374 108/300 Write block 4761 len 4096 data: 6
10375 109/300 Read block 2373 len 8192
10376 110/300 Read block 706 len 4096
10377 111/300 Write block 2166 len 12288 data: 3 2 2
10378 112/300 Read block 1021 len 12288
10379 113/300 Write block 3537 len 24576 data: 5 6 6 5 4 4
10380 114/300 Write block 3228 len 36864 data: 3 3 4 3 3 3 5 6 6
10381 115/300 Write block 2589 len 4096 data: 5
10382 116/300 Read block 3908 len 40960
10383 117/300 Read block 4524 len 12288
10384 118/300 Write block 4612 len 20480 data: 3 3 3 3 2
10385 119/300 Write block 712 len 20480 data: 3 3 2 2 2
10386 120/300 Read block 692 len 28672
10387 121/300 Write block 2356 len 16384 data: 5 5 3 3
10388 122/300 Write block 451 len 8192 data: 5 5
10389 123/300 Flush
10390 124/300 Read block 3448 len 36864
10391 125/300 Flush
10392 126/300 Write block 1123 len 40960 data: 3 3 3 4 4 4 4 3 4 4
10393 127/300 Write block 1593 len 32768 data: 3 3 3 3 2 2 2 2
10394 128/300 Write block 3494 len 40960 data: 4 3 4 3 4 4 4 4 4 3
10395 129/300 Write block 3183 len 16384 data: 3 2 2 3
10396 130/300 Read block 2936 len 4096
10397 131/300 Read block 2248 len 40960
10398 132/300 Read block 210 len 36864
10399 133/300 Flush
10400 134/300 Write block 2011 len 28672 data: 6 6 4 5 9 8 7
10401 135/300 Write block 3827 len 28672 data: 2 2 2 2 2 2 2
10402 136/300 Read block 1921 len 24576
10403 137/300 Write block 186 len 16384 data: 4 4 6 3
10404 138/300 Read block 916 len 20480
10405 139/300 Flush
10406 140/300 Read block 2837 len 12288
10407 141/300 Read block 1780 len 20480
10408 142/300 Read block 3289 len 16384
10409 143/300 Read block 4359 len 28672
10410 144/300 Read block 4684 len 4096
10411 145/300 Read block 4973 len 36864
10412 146/300 Write block 1116 len 16384 data: 4 4 4 4
10413 147/300 Write block 2031 len 24576 data: 4 5 3 2 2 3
10414 148/300 Read block 2801 len 40960
10415 149/300 Write block 4365 len 40960 data: 4 4 3 3 3 3 4 6 6 7
10416 150/300 Read block 4014 len 16384
10417 151/300 Read block 92 len 8192
10418 152/300 Read block 4973 len 32768
10419 153/300 Read block 2518 len 36864
10420 154/300 Read block 3654 len 28672
10421 155/300 Write block 4322 len 32768 data: 3 3 6 6 7 5 4 6
10422 156/300 Read block 3660 len 40960
10423 157/300 Write block 3708 len 40960 data: 7 7 9 10 9 8 8 9 7 7
10424 158/300 Read block 48 len 36864
10425 159/300 Read block 2255 len 24576
10426 160/300 Write block 1368 len 4096 data: 5
10427 161/300 Read block 3505 len 16384
10428 162/300 Read block 2948 len 12288
10429 163/300 Read block 3702 len 32768
10430 164/300 Read block 531 len 16384
10431 165/300 Write block 4673 len 40960 data: 3 3 3 4 5 5 5 5 4 3
10432 166/300 Flush
10433 167/300 Read block 2566 len 12288
10434 168/300 Read block 1869 len 16384
10435 169/300 Write block 915 len 40960 data: 4 4 5 5 5 5 3 5 4 3
10436 170/300 Read block 1537 len 36864
10437 171/300 Write block 3538 len 8192 data: 7 7
10438 172/300 Read block 1773 len 16384
10439 173/300 Read block 3636 len 4096
10440 174/300 Read block 3098 len 36864
10441 175/300 Read block 3796 len 36864
10442 176/300 Write block 1251 len 32768 data: 2 3 3 3 4 5 5 5
10443 177/300 Read block 295 len 36864
10444 178/300 Write block 2839 len 24576 data: 2 2 2 2 2 2
10445 179/300 Read block 4509 len 4096
10446 180/300 Write block 2359 len 36864 data: 4 3 3 4 4 5 5 6 7
10447 181/300 Write block 4302 len 28672 data: 5 4 5 5 5 6 6
10448 182/300 Read block 1905 len 8192
10449 183/300 Read block 1688 len 4096
10450 184/300 Read block 3422 len 20480
10451 185/300 Write block 1872 len 20480 data: 4 4 4 5 6
10452 186/300 Read block 2191 len 8192
10453 187/300 Read block 3347 len 28672
10454 188/300 Read block 2224 len 8192
10455 189/300 Write block 4891 len 4096 data: 3
10456 190/300 Write block 3150 len 8192 data: 4 2
10457 191/300 Read block 4640 len 16384
10458 192/300 Write block 4127 len 8192 data: 2 2
10459 193/300 Read block 2783 len 12288
10460 194/300 Read block 433 len 36864
10461 195/300 Read block 4162 len 40960
10462 196/300 Write block 122 len 20480 data: 3 3 4 4 4
10463 197/300 Read block 4367 len 16384
10464 198/300 Read block 382 len 36864
10465 199/300 Read block 3253 len 20480
10466 200/300 Write block 2945 len 16384 data: 4 4 4 3
10467 201/300 Read block 4079 len 24576
10468 202/300 Read block 3922 len 36864
10469 203/300 Write block 4264 len 36864 data: 3 2 2 2 2 2 2 2 3
10470 204/300 Read block 2657 len 4096
10471 205/300 Flush
10472 206/300 Read block 4457 len 12288
10473 207/300 Read block 1624 len 12288
10474 208/300 Read block 4939 len 16384
10475 209/300 Write block 4206 len 20480 data: 4 4 4 3 3
10476 210/300 Write block 2027 len 24576 data: 5 5 5 5 5 6
10477 211/300 Write block 1054 len 8192 data: 3 4
10478 212/300 Read block 1750 len 20480
10479 213/300 Read block 3487 len 20480
10480 214/300 Read block 394 len 8192
10481 215/300 Write block 3418 len 32768 data: 3 3 2 2 2 2 3 5
10482 216/300 Write block 2353 len 12288 data: 4 5 5
10483 217/300 Write block 3263 len 4096 data: 3
10484 218/300 Write block 819 len 40960 data: 5 4 3 3 3 3 3 3 4 4
10485 219/300 Write block 1054 len 20480 data: 4 5 4 4 5
10486 220/300 Write block 1561 len 12288 data: 4 4 4
10487 221/300 Read block 4855 len 8192
10488 222/300 Read block 4263 len 12288
10489 223/300 Read block 1977 len 40960
10490 224/300 Read block 1513 len 28672
10491 225/300 Read block 939 len 36864
10492 226/300 Read block 3900 len 40960
10493 227/300 Write block 30 len 20480 data: 2 2 3 4 5
10494 228/300 Read block 3608 len 12288
10495 229/300 Write block 238 len 32768 data: 2 2 2 2 2 3 3 4
10496 230/300 Read block 2090 len 16384
10497 231/300 Write block 958 len 24576 data: 4 4 4 4 3 2
10498 232/300 Read block 420 len 12288
10499 233/300 Write block 3371 len 12288 data: 3 3 3
10500 234/300 Write block 2497 len 24576 data: 3 3 3 2 3 3
10501 235/300 Flush
10502 236/300 Write block 2190 len 32768 data: 3 4 5 5 5 5 5 5
10503 237/300 Write block 2759 len 4096 data: 4
10504 238/300 Read block 234 len 28672
10505 239/300 Read block 4069 len 32768
10506 240/300 Write block 3518 len 36864 data: 5 6 7 6 4 4 4 3 3
10507 241/300 Write block 2121 len 32768 data: 5 5 4 3 3 3 3 3
10508 242/300 Read block 309 len 32768
10509 243/300 Read block 4499 len 40960
10510 244/300 Read block 319 len 20480
10511 245/300 Read block 4207 len 24576
10512 246/300 Read block 4260 len 16384
10513 247/300 Read block 1667 len 8192
10514 248/300 Read block 1396 len 20480
10515 249/300 Read block 2941 len 28672
10516 250/300 Write block 3218 len 24576 data: 2 2 2 2 2 2
10517 251/300 Read block 1351 len 4096
10518 252/300 Read block 2290 len 28672
10519 253/300 Read block 485 len 8192
10520 254/300 Read block 2474 len 24576
10521 255/300 Read block 1525 len 20480
10522 256/300 Flush
10523 257/300 Write block 2437 len 4096 data: 3
10524 258/300 Read block 2261 len 36864
10525 259/300 Read block 4881 len 12288
10526 260/300 Read block 4949 len 12288
10527 261/300 Flush
10528 262/300 Read block 2933 len 24576
10529 263/300 Flush
10530 264/300 Read block 2135 len 4096
10531 265/300 Read block 3172 len 16384
10532 266/300 Write block 4528 len 40960 data: 5 4 4 3 3 3 2 2 2 2
10533 267/300 Flush
10534 268/300 Read block 1151 len 28672
10535 269/300 Read block 47 len 8192
10536 270/300 Read block 2441 len 16384
10537 271/300 Write block 4612 len 40960 data: 4 4 4 4 3 3 2 4 4 5
10538 272/300 Read block 1732 len 32768
10539 273/300 Read block 4309 len 32768
10540 274/300 Read block 2296 len 20480
10541 275/300 Read block 2851 len 24576
10542 276/300 Write block 3254 len 36864 data: 3 3 3 3 2 3 4 4 4
10543 277/300 Write block 3776 len 40960 data: 4 4 6 5 5 5 4 4 3 2
10544 278/300 Write block 4207 len 12288 data: 5 5 4
10545 279/300 Write block 3827 len 36864 data: 3 3 3 3 3 3 3 2 2
10546 280/300 Read block 2368 len 36864
10547 281/300 Flush
10548 282/300 Flush
10549 283/300 Read block 4561 len 12288
10550 284/300 Read block 4648 len 4096
10551 285/300 Write block 24 len 4096 data: 4
10552 286/300 Flush
10553 287/300 Read block 594 len 12288
10554 288/300 Flush
10555 289/300 Write block 911 len 40960 data: 3 4 4 4 5 5 6 6 6 6
10556 290/300 Write block 852 len 16384 data: 5 3 3 2
10557 291/300 Write block 2918 len 16384 data: 2 2 3 3
10558 292/300 Read block 3285 len 8192
10559 293/300 Read block 1816 len 40960
10560 294/300 Read block 3593 len 8192
10561 295/300 Write block 314 len 28672 data: 4 3 3 3 5 6 7
10562 296/300 Read block 900 len 12288
10563 297/300 Read block 2745 len 36864
10564 298/300 Write block 3192 len 40960 data: 4 5 4 4 3 3 4 4 5 5
10565 299/300 Read block 3834 len 16384
10566 300/300 Write block 4608 len 12288 data: 4 3 3
10567 Main task has work to do, go find it
10568 got dsc Start(1)
10569 start 1
10570 Oct 13 18:11:24.786 INFO request completed, latency_us: 297, response_code: 204, uri: /start/cid/1, method: POST, req_id: 1c1841d8-94cd-44b8-b2ce-f20b1eaba22c, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
10571 Replay: started 1, returned:Ok(())
10572 ----------------------------------------------------------------
10573 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
10574 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
10575 3935 Acked 4934 Read 6 Done New Done false
10576 3936 Acked 4935 Flush 0 Done New Done false
10577 3937 Acked 4936 Read 9 Done New Done false
10578 3938 Acked 4937 Write 2 Done New Done false
10579 3939 Acked 4938 Flush 0 Done New Done false
10580 3940 Acked 4939 Read 5 Done New Done false
10581 3941 Acked 4940 Write 5 Done New Done false
10582 3942 Acked 4941 Read 7 Done New Done false
10583 3943 Acked 4942 Read 6 Done New Done false
10584 3944 Acked 4943 Flush 0 Done New Done false
10585 3945 Acked 4944 Write 6 Done New Done false
10586 3946 Acked 4945 Read 7 Done New Done false
10587 3947 Acked 4946 Read 4 Done New Done false
10588 3948 Acked 4947 Read 4 Done New Done false
10589 3949 Acked 4948 Write 2 Done New Done false
10590 3950 Acked 4949 Read 1 Done New Done false
10591 3951 Acked 4950 Flush 0 Done New Done false
10592 3952 Acked 4951 Write 6 Done New Done false
10593 3953 Acked 4952 Write 3 Done New Done false
10594 3954 Acked 4953 Read 1 Done New Done false
10595 3955 Acked 4954 Read 3 Done New Done false
10596 3956 Acked 4955 Read 9 Done New Done false
10597 3957 Acked 4956 Write 9 Done New Done false
10598 3958 Acked 4957 Write 10 Done New Done false
10599 3959 Acked 4958 Write 8 Done New Done false
10600 3960 Acked 4959 Read 6 Done New Done false
10601 3961 Acked 4960 Write 1 Done New Done false
10602 3962 Acked 4961 Read 2 Done New Done false
10603 3963 Acked 4962 Read 7 Done New Done false
10604 3964 Acked 4963 Read 5 Done New Done false
10605 3965 Acked 4964 Read 3 Done New Done false
10606 3966 Acked 4965 Read 8 Done New Done false
10607 3967 Acked 4966 Read 10 Done New Done false
10608 3968 Acked 4967 Read 6 Done New Done false
10609 3969 Acked 4968 Write 10 Done New Done false
10610 3970 Acked 4969 Flush 0 Done New Done false
10611 3971 Acked 4970 Write 10 Done New Done false
10612 3972 Acked 4971 Read 10 Done New Done false
10613 3973 Acked 4972 Read 10 Done New Done false
10614 3974 Acked 4973 Write 2 Done New Done false
10615 3975 Acked 4974 Write 9 Done New Done false
10616 3976 Acked 4975 Read 9 Done New Done false
10617 3977 Acked 4976 Read 9 Done New Done false
10618 3978 Acked 4977 Write 3 Done New Done false
10619 3979 Acked 4978 Write 7 Done New Done false
10620 3980 Acked 4979 Read 10 Done New Done false
10621 3981 Acked 4980 Write 8 Done New Done false
10622 3982 Acked 4981 Write 8 Done New Done false
10623 3983 Acked 4982 Read 8 Done New Done false
10624 3984 Acked 4983 Write 10 Done New Done false
10625 3985 Acked 4984 Flush 0 Done New Done false
10626 3986 Acked 4985 Write 4 Done New Done false
10627 3987 Acked 4986 Read 8 Done New Done false
10628 3988 Acked 4987 Read 6 Done New Done false
10629 3989 Acked 4988 Write 8 Done New Done false
10630 3990 Acked 4989 Write 7 Done New Done false
10631 3991 Acked 4990 Read 9 Done New Done false
10632 3992 Acked 4991 Write 9 Done New Done false
10633 3993 Acked 4992 Read 2 Done New Done false
10634 3994 Acked 4993 Read 5 Done New Done false
10635 3995 Acked 4994 Read 5 Done New Done false
10636 3996 Acked 4995 Write 7 Done New Done false
10637 3997 Acked 4996 Flush 0 Done New Done false
10638 3998 Acked 4997 Read 9 Done New Done false
10639 3999 Acked 4998 Write 4 Done New Done false
10640 4000 Acked 4999 Read 4 Done New Done false
10641 4001 Acked 5000 Write 7 Done New Done false
10642 4002 Acked 5001 Write 4 Done New Done false
10643 4003 Acked 5002 Read 4 Done New Done false
10644 4004 Acked 5003 Write 9 Done New Done false
10645 4005 Acked 5004 Read 10 Done New Done false
10646 4006 Acked 5005 Flush 0 Done New Done false
10647 4007 Acked 5006 Write 10 Done New Done false
10648 4008 Acked 5007 Read 9 Done New Done false
10649 4009 Acked 5008 Read 6 Done New Done false
10650 4010 Acked 5009 Write 2 Done New Done false
10651 4011 Acked 5010 Read 9 Done New Done false
10652 4012 Acked 5011 Write 6 Done New Done false
10653 4013 Acked 5012 Write 3 Done New Done false
10654 4014 Acked 5013 Write 8 Done New Done false
10655 4015 Acked 5014 Write 3 Done New Done false
10656 4016 Acked 5015 Read 5 Done New Done false
10657 4017 Acked 5016 Read 6 Done New Done false
10658 4018 Acked 5017 Read 1 Done New Done false
10659 4019 Acked 5018 Write 9 Done New Done false
10660 4020 Acked 5019 Read 9 Done New Done false
10661 4021 Acked 5020 Read 9 Done New Done false
10662 4022 Acked 5021 Write 6 Done New Done false
10663 4023 Acked 5022 Read 9 Done New Done false
10664 4024 Acked 5023 Read 9 Done New Done false
10665 4025 Acked 5024 Read 5 Done New Done false
10666 4026 Acked 5025 Read 9 Done New Done false
10667 4027 Acked 5026 Read 3 Done New Done false
10668 4028 Acked 5027 Read 4 Done New Done false
10669 4029 Acked 5028 Flush 0 Done New Done false
10670 4030 Acked 5029 Read 7 Done New Done false
10671 4031 Acked 5030 Write 6 Done New Done false
10672 4032 Acked 5031 Read 8 Done New Done false
10673 4033 Acked 5032 Read 9 Done New Done false
10674 4034 Acked 5033 Write 1 Done New Done false
10675 4035 Acked 5034 Write 1 Done New Done false
10676 4036 Acked 5035 Read 5 Done New Done false
10677 4037 Acked 5036 Read 2 Done New Done false
10678 4038 Acked 5037 Write 4 Done New Done false
10679 4039 Acked 5038 Read 5 Done New Done false
10680 4040 Acked 5039 Write 4 Done New Done false
10681 4041 Acked 5040 Write 2 Done New Done false
10682 4042 Acked 5041 Read 10 Done New Done false
10683 4043 Acked 5042 Write 1 Done New Done false
10684 4044 Acked 5043 Read 2 Done New Done false
10685 4045 Acked 5044 Read 1 Done New Done false
10686 4046 Acked 5045 Write 3 Done New Done false
10687 4047 Acked 5046 Read 3 Done New Done false
10688 4048 Acked 5047 Write 6 Done New Done false
10689 4049 Acked 5048 Write 9 Done New Done false
10690 4050 Acked 5049 Write 1 Done New Done false
10691 4051 Acked 5050 Read 10 Done New Done false
10692 4052 Acked 5051 Read 3 Done New Done false
10693 4053 Acked 5052 Write 5 Done New Done false
10694 4054 Acked 5053 Write 5 Done New Done false
10695 4055 Acked 5054 Read 7 Done New Done false
10696 4056 Acked 5055 Write 4 Done New Done false
10697 4057 Acked 5056 Write 2 Done New Done false
10698 4058 Acked 5057 Flush 0 Done New Done false
10699 4059 Acked 5058 Read 9 Done New Done false
10700 4060 Acked 5059 Flush 0 Done New Done false
10701 4061 Acked 5060 Write 10 Done New Done false
10702 4062 Acked 5061 Write 8 Done New Done false
10703 4063 Acked 5062 Write 10 Done New Done false
10704 4064 Acked 5063 Write 4 Done New Done false
10705 4065 Acked 5064 Read 1 Done New Done false
10706 4066 Acked 5065 Read 10 Done New Done false
10707 4067 Acked 5066 Read 9 Done New Done false
10708 4068 Acked 5067 Flush 0 Done New Done false
10709 4069 Acked 5068 Write 7 Done New Done false
10710 4070 Acked 5069 Write 7 Done New Done false
10711 4071 Acked 5070 Read 6 Done New Done false
10712 4072 Acked 5071 Write 4 Done New Done false
10713 4073 Acked 5072 Read 5 Done New Done false
10714 4074 Acked 5073 Flush 0 Done New Done false
10715 4075 Acked 5074 Read 3 Done New Done false
10716 4076 Acked 5075 Read 5 Done New Done false
10717 4077 Acked 5076 Read 4 Done New Done false
10718 4078 Acked 5077 Read 7 Done New Done false
10719 4079 Acked 5078 Read 1 Done New Done false
10720 4080 Acked 5079 Read 9 Done New Done false
10721 4081 Acked 5080 Write 4 Done New Done false
10722 4082 Acked 5081 Write 6 Done New Done false
10723 4083 Acked 5082 Read 10 Done New Done false
10724 4084 Acked 5083 Write 10 Done New Done false
10725 4085 Acked 5084 Read 4 Done New Done false
10726 4086 Acked 5085 Read 2 Done New Done false
10727 4087 Acked 5086 Read 8 Done New Done false
10728 4088 Acked 5087 Read 9 Done New Done false
10729 4089 Acked 5088 Read 7 Done New Done false
10730 4090 Acked 5089 Write 8 Done New Done false
10731 4091 Acked 5090 Read 10 Done New Done false
10732 4092 Acked 5091 Flush 0 Done New Done false
10733 4093 Acked 5092 Write 10 Done New Done false
10734 4094 Acked 5093 Read 9 Done New Done false
10735 4095 Acked 5094 Read 6 Done New Done false
10736 4096 Acked 5095 Write 1 Done New Done false
10737 4097 Acked 5096 Read 4 Done New Done false
10738 4098 Acked 5097 Read 3 Done New Done false
10739 4099 Acked 5098 Read 8 Done New Done false
10740 4100 Acked 5099 Read 4 Done New Done false
10741 4101 Acked 5100 Write 10 Done New Done false
10742 4102 Acked 5101 Flush 0 Done New Done false
10743 4103 Acked 5102 Read 3 Done New Done false
10744 4104 Acked 5103 Read 4 Done New Done false
10745 4105 Acked 5104 Write 10 Done New Done false
10746 4106 Acked 5105 Read 9 Done New Done false
10747 4107 Acked 5106 Write 2 Done New Done false
10748 4108 Acked 5107 Read 4 Done New Done false
10749 4109 Acked 5108 Read 1 Done New Done false
10750 4110 Acked 5109 Read 9 Done New Done false
10751 4111 Acked 5110 Read 9 Done New Done false
10752 4112 Acked 5111 Write 8 Done New Done false
10753 4113 Acked 5112 Read 9 Done New Done false
10754 4114 Acked 5113 Write 6 Done New Done false
10755 4115 Acked 5114 Read 1 Done New Done false
10756 4116 Acked 5115 Write 9 Done New Done false
10757 4117 Acked 5116 Write 7 Done New Done false
10758 4118 Acked 5117 Read 2 Done New Done false
10759 4119 Acked 5118 Read 1 Done New Done false
10760 4120 Acked 5119 Read 5 Done New Done false
10761 4121 Acked 5120 Write 5 Done New Done false
10762 4122 Acked 5121 Read 2 Done New Done false
10763 4123 Acked 5122 Read 7 Done New Done false
10764 4124 Acked 5123 Read 2 Done New Done false
10765 4125 Acked 5124 Write 1 Done New Done false
10766 4126 Acked 5125 Write 2 Done New Done false
10767 4127 Acked 5126 Read 4 Done New Done false
10768 4128 Acked 5127 Write 2 Done New Done false
10769 4129 Acked 5128 Read 3 Done New Done false
10770 4130 Acked 5129 Read 9 Done New Done false
10771 4131 Acked 5130 Read 10 Done New Done false
10772 4132 Acked 5131 Write 5 Done New Done false
10773 4133 Acked 5132 Read 4 Done New Done false
10774 4134 Acked 5133 Read 9 Done New Done false
10775 4135 Acked 5134 Read 5 Done New Done false
10776 4136 Acked 5135 Write 4 Done New Done false
10777 4137 Acked 5136 Read 6 Done New Done false
10778 4138 Acked 5137 Read 9 Done New Done false
10779 4139 Acked 5138 Write 9 Done New Done false
10780 4140 Acked 5139 Read 1 Done New Done false
10781 4141 Acked 5140 Flush 0 Done New Done false
10782 4142 Acked 5141 Read 3 Done New Done false
10783 4143 Acked 5142 Read 3 Done New Done false
10784 4144 Acked 5143 Read 4 Done New Done false
10785 4145 Acked 5144 Write 5 Done New Done false
10786 4146 Acked 5145 Write 6 Done New Done false
10787 4147 Acked 5146 Write 2 Done New Done false
10788 4148 Acked 5147 Read 5 Done New Done false
10789 4149 Acked 5148 Read 5 Done New Done false
10790 4150 Acked 5149 Read 2 Done New Done false
10791 4151 Acked 5150 Write 8 Done New Done false
10792 4152 Acked 5151 Write 3 Done New Done false
10793 4153 Acked 5152 Write 1 Done New Done false
10794 4154 Acked 5153 Write 10 Done New Done false
10795 4155 Acked 5154 Write 5 Done New Done false
10796 4156 Acked 5155 Write 3 Done New Done false
10797 4157 Acked 5156 Read 2 Done New Done false
10798 4158 Acked 5157 Read 3 Done New Done false
10799 4159 Acked 5158 Read 10 Done New Done false
10800 4160 Acked 5159 Read 7 Done New Done false
10801 4161 Acked 5160 Read 9 Done New Done false
10802 4162 Acked 5161 Read 10 Done New Done false
10803 4163 Acked 5162 Write 5 Done New Done false
10804 4164 Acked 5163 Read 3 Done New Done false
10805 4165 Acked 5164 Write 8 Done New Done false
10806 4166 Acked 5165 Read 4 Done New Done false
10807 4167 Acked 5166 Write 6 Done New Done false
10808 4168 Acked 5167 Read 3 Done New Done false
10809 4169 Acked 5168 Write 3 Done New Done false
10810 4170 Acked 5169 Write 6 Done New Done false
10811 4171 Acked 5170 Flush 0 Done New Done false
10812 4172 Acked 5171 Write 8 Done New Done false
10813 4173 Acked 5172 Write 1 Done New Done false
10814 4174 Acked 5173 Read 7 Done New Done false
10815 4175 Acked 5174 Read 8 Done New Done false
10816 4176 Acked 5175 Write 9 Done New Done false
10817 4177 Acked 5176 Write 8 Done New Done false
10818 4178 Acked 5177 Read 8 Done New Done false
10819 4179 Acked 5178 Read 10 Done New Done false
10820 4180 Acked 5179 Read 5 Done New Done false
10821 4181 Acked 5180 Read 6 Done New Done false
10822 4182 Acked 5181 Read 4 Done New Done false
10823 4183 Acked 5182 Read 2 Done New Done false
10824 4184 Acked 5183 Read 5 Done New Done false
10825 4185 Acked 5184 Read 7 Done New Done false
10826 4186 Acked 5185 Write 6 Done New Done false
10827 4187 Acked 5186 Read 1 Done New Done false
10828 4188 Acked 5187 Read 7 Done New Done false
10829 4189 Acked 5188 Read 2 Done New Done false
10830 4190 Acked 5189 Read 6 Done New Done false
10831 4191 Acked 5190 Read 5 Done New Done false
10832 4192 Acked 5191 Flush 0 Done New Done false
10833 4193 Acked 5192 Write 1 Done New Done false
10834 4194 Acked 5193 Read 9 Done New Done false
10835 4195 Acked 5194 Read 3 Done New Done false
10836 4196 Acked 5195 Read 3 Done New Done false
10837 4197 Acked 5196 Flush 0 Done New Done false
10838 4198 Acked 5197 Read 6 Done New Done false
10839 4199 Acked 5198 Flush 0 Done New Done false
10840 4200 Acked 5199 Read 1 Done New Done false
10841 4201 Acked 5200 Read 4 Done New Done false
10842 4202 Acked 5201 Write 10 Done New Done false
10843 4203 Acked 5202 Flush 0 Done New Done false
10844 4204 Acked 5203 Read 7 Done New Done false
10845 4205 Acked 5204 Read 2 Done New Done false
10846 4206 Acked 5205 Read 4 Done New Done false
10847 4207 Acked 5206 Write 10 Done New Done false
10848 4208 Acked 5207 Read 8 Done New Done false
10849 4209 Acked 5208 Read 8 Done New Done false
10850 4210 Acked 5209 Read 5 Done New Done false
10851 4211 Acked 5210 Read 6 Done New Done false
10852 4212 Acked 5211 Write 9 Done New Done false
10853 4213 Acked 5212 Write 10 Done New Done false
10854 4214 Acked 5213 Write 3 Done New Done false
10855 4215 Acked 5214 Write 9 Done New Done false
10856 4216 Acked 5215 Read 9 Done New Done false
10857 4217 Acked 5216 Flush 0 Done New Done false
10858 4218 Acked 5217 Flush 0 Done New Done false
10859 4219 Acked 5218 Read 3 Done New Done false
10860 4220 Acked 5219 Read 1 Done New Done false
10861 4221 Acked 5220 Write 1 Done New Done false
10862 4222 Acked 5221 Flush 0 Done New Done false
10863 4223 Acked 5222 Read 3 Done New Done false
10864 4224 Acked 5223 Flush 0 Done New Done false
10865 4225 Acked 5224 Write 10 Done New Done false
10866 4226 Acked 5225 Write 4 Done New Done false
10867 4227 Acked 5226 Write 4 Done New Done false
10868 4228 Acked 5227 Read 2 Done New Done false
10869 4229 Acked 5228 Read 10 Done New Done false
10870 4230 Acked 5229 Read 2 Done New Done false
10871 4231 Acked 5230 Write 7 Done New Done false
10872 4232 Acked 5231 Read 3 Done New Done false
10873 4233 Acked 5232 Read 9 Done New Done false
10874 4234 Acked 5233 Write 10 Done New Done false
10875 4235 Acked 5234 Read 4 Done New Done false
10876 4236 Acked 5235 Write 3 Sent New Sent false
10877 STATES DS:0 DS:1 DS:2 TOTAL
10878 New 0 302 0 302
10879 Sent 1 0 1 2
10880 Done 301 0 301 602
10881 Skipped 0 0 0 0
10882 Error 0 0 0 0
10883 Last Flush: 5223 4933 5223
10884 Downstairs last five completed: 4933 4932 4931 4930 4929 4928
10885 Upstairs last five completed: 4236 4235 4234 4233 4232 4231
10886 CLIENT: Up:0 ds:302 act:2
10887 [8820] got start action
10888 [8820] I am going to restart
10889 Starting downstairs at port 8820
10890 Make output file at "/tmp/dsc/downstairs-8820.txt"
10891 [8820][1] reports Starting
10892 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1190)
10893 [8820][1] reports Running
108942023-10-13T18:11:26.809ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 1
108952023-10-13T18:11:26.809ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
108962023-10-13T18:11:26.809ZINFOcrucible: [1] client re-new 303 jobs since flush 4933 = downstairs
108972023-10-13T18:11:26.810ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
108982023-10-13T18:11:26.811ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
108992023-10-13T18:11:26.811ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
109002023-10-13T18:11:26.811ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
109012023-10-13T18:11:26.811ZINFOcrucible: [1] send last flush ID to this DS: 4933
109022023-10-13T18:11:26.811ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
109032023-10-13T18:11:26.811ZINFOcrucible: [1] Replied this last flush ID: 4933
109042023-10-13T18:11:26.811ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Offline Active ds_transition to Replay
109052023-10-13T18:11:26.811ZINFOcrucible: [1] Transition from Offline to Replay
109062023-10-13T18:11:26.811ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
109072023-10-13T18:11:26.811ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Replay Active ds_transition to Active
109082023-10-13T18:11:26.811ZINFOcrucible: [1] Transition from Replay to Active
109092023-10-13T18:11:26.811ZINFOcrucible: [1] Starts cmd_loop
109102023-10-13T18:11:26.811ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
109112023-10-13T18:11:26.811ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
109122023-10-13T18:11:26.811ZINFOcrucible: Set check for repair
109132023-10-13T18:11:27.812ZINFOcrucible: Checking if live repair is needed
109142023-10-13T18:11:27.812ZINFOcrucible: No Live Repair required at this time
109152023-10-13T18:11:27.812ZWARNcrucible: [1] flow control end
10916 ----------------------------------------------------------------
10917 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10918 Downstairs last five completed: 5236 5235 5234 5233 5232 5231
10919 Upstairs last five completed: 4237 4236 4235 4234 4233 4232
10920 CLIENT: Up:0 ds:0 act:3
10921 Replay: All jobs finished, all DS active.
10922 Main task has work to do, go find it
10923 got dsc Stop(1)
10924 stop 1
10925 Oct 13 18:11:28.791 INFO request completed, latency_us: 506, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 74888a11-1796-4c6a-8c78-d375f9a4ade2, remote_addr: 127.0.0[8820] Got stop action so:false kr:false
10926 .1:64571, local_addr: 127.0.0.1:9998
10927 Oct 13 18:11:28.792 INFO request completed, latency_us: 308, response_code: 200, uri: /state/cid/1, method: GET, req_id: 381c4154-fd88-43ef-bfaa-4faa9fcda8a4, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
109282023-10-13T18:11:28.792ZWARNcrucible: [1] downstairs disconnected
109292023-10-13T18:11:28.792ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
109302023-10-13T18:11:28.792ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8820 closed looper = 1
109312023-10-13T18:11:28.793ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
109322023-10-13T18:11:28.793ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
109332023-10-13T18:11:28.793ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
109342023-10-13T18:11:28.793ZWARNcrucible: [1] pm_task rx.recv() is None
10935 [8820] Exited with: ExitStatus(unix_wait_status(9))
10936 [8820][1] reports Exit
109372023-10-13T18:11:31.796ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
10938 Oct 13 18:11:32.794 INFO request completed, latency_us: 256, response_code: 200, uri: /state/cid/1, method: GET, req_id: 5b599b67-fffb-4542-8215-cc981367bb57, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
10939 001/300 Write block 813 len 16384 data: 4 3 3 3
10940 002/300 Write block 3671 len 4096 data: 5
10941 003/300 Write block 2916 len 12288 data: 2 2 3
10942 004/300 Write block 102 len 20480 data: 5 4 5 4 3
10943 005/300 Write block 2601 len 12288 data: 6 8 7
10944 006/300 Write block 4789 len 4096 data: 5
10945 007/300 Flush
10946 008/300 Write block 2295 len 40960 data: 2 2 2 2 3 3 4 4 5 5
10947 009/300 Flush
10948 010/300 Read block 1455 len 8192
10949 011/300 Read block 1737 len 36864
10950 012/300 Read block 3676 len 4096
10951 013/300 Read block 3657 len 36864
10952 014/300 Read block 3362 len 32768
10953 015/300 Read block 4150 len 24576
10954 016/300 Flush
10955 017/300 Read block 86 len 24576
10956 018/300 Read block 1595 len 8192
10957 019/300 Write block 2127 len 36864 data: 4 4 4 3 3 4 4 4 3
10958 020/300 Read block 1719 len 16384
10959 021/300 Read block 3852 len 16384
10960 022/300 Write block 3746 len 4096 data: 4
10961 023/300 Write block 2181 len 28672 data: 2 2 2 2 3 3 3
10962 024/300 Read block 909 len 28672
10963 025/300 Read block 554 len 40960
10964 026/300 Write block 599 len 36864 data: 5 6 6 6 7 8 5 5 5
10965 027/300 Flush
10966 028/300 Write block 808 len 28672 data: 4 3 3 3 3 5 4
10967 029/300 Flush
10968 030/300 Write block 4743 len 24576 data: 6 6 6 5 5 5
10969 031/300 Flush
10970 032/300 Write block 4390 len 36864 data: 4 3 3 4 3 4 4 4 7
10971 033/300 Write block 301 len 8192 data: 4 3
10972 034/300 Read block 1581 len 12288
10973 035/300 Read block 449 len 36864
10974 036/300 Write block 616 len 20480 data: 4 4 4 3 3
10975 037/300 Read block 2461 len 12288
10976 038/300 Read block 798 len 32768
10977 039/300 Read block 673 len 8192
10978 040/300 Write block 3144 len 16384 data: 5 6 6 6
10979 041/300 Write block 1906 len 32768 data: 3 3 3 4 3 3 3 3
10980 042/300 Read block 1696 len 32768
10981 043/300 Read block 3087 len 16384
10982 044/300 Read block 4227 len 32768
10983 045/300 Write block 2390 len 16384 data: 4 4 4 4
10984 046/300 Write block 2667 len 36864 data: 4 4 3 3 4 3 4 4 4
10985 047/300 Write block 4744 len 24576 data: 7 7 6 6 6 5
10986 048/300 Read block 2102 len 4096
10987 049/300 Flush
10988 050/300 Read block 3058 len 20480
10989 051/300 Write block 1689 len 28672 data: 3 2 4 5 5 5 5
10990 052/300 Read block 2505 len 20480
10991 053/300 Flush
10992 054/300 Read block 4192 len 40960
10993 055/300 Write block 1170 len 40960 data: 4 4 5 5 4 4 4 3 2 2
10994 056/300 Read block 1126 len 28672
10995 057/300 Write block 1107 len 24576 data: 3 3 3 3 3 3
10996 058/300 Write block 4196 len 4096 data: 6
10997 059/300 Read block 3733 len 8192
10998 060/300 Read block 249 len 12288
10999 061/300 Read block 127 len 24576
11000 062/300 Read block 2132 len 16384
11001 063/300 Write block 4417 len 28672 data: 3 3 3 3 2 4 4
11002 064/300 Write block 1853 len 36864 data: 4 4 4 4 3 3 3 3 3
11003 065/300 Read block 1787 len 36864
11004 066/300 Flush
11005 067/300 Read block 4617 len 8192
11006 068/300 Read block 4007 len 24576
11007 069/300 Read block 3464 len 28672
11008 070/300 Read block 1130 len 32768
11009 071/300 Write block 2150 len 24576 data: 3 3 2 3 3 4
11010 072/300 Read block 1376 len 16384
11011 073/300 Read block 3147 len 28672
11012 074/300 Read block 3176 len 20480
11013 075/300 Read block 1006 len 16384
11014 076/300 Read block 216 len 8192
11015 077/300 Write block 4507 len 40960 data: 3 3 4 4 5 5 5 4 3 2
11016 078/300 Write block 905 len 20480 data: 3 3 3 3 3
11017 079/300 Read block 4575 len 40960
11018 080/300 Write block 506 len 40960 data: 2 2 2 2 2 2 3 3 3 3
11019 081/300 Read block 497 len 8192
11020 082/300 Read block 1918 len 16384
11021 083/300 Write block 1442 len 28672 data: 4 3 3 3 3 3 6
11022 084/300 Read block 4257 len 24576
11023 085/300 Write block 2491 len 12288 data: 5 6 6
11024 086/300 Write block 629 len 12288 data: 4 4 2
11025 087/300 Write block 1386 len 36864 data: 4 5 4 4 5 5 5 5 5
11026 088/300 Write block 4286 len 12288 data: 4 3 3
11027 089/300 Read block 3138 len 28672
11028 090/300 Write block 4235 len 24576 data: 2 2 2 3 3 3
11029 091/300 Read block 4767 len 16384
11030 092/300 Write block 2787 len 28672 data: 3 3 2 2 3 3 3
11031 093/300 Read block 4737 len 12288
11032 094/300 Read block 253 len 20480
11033 095/300 Read block 2233 len 12288
11034 096/300 Read block 4899 len 36864
11035 097/300 Write block 1069 len 32768 data: 5 5 4 3 3 3 3 4
11036 098/300 Write block 279 len 20480 data: 3 3 3 4 3
11037 099/300 Write block 1234 len 28672 data: 4 4 3 3 4 4 3
11038 100/300 Write block 543 len 20480 data: 3 4 3 3 2
11039 101/300 Write block 4130 len 16384 data: 2 2 2 2
11040 102/300 Write block 644 len 12288 data: 4 4 3
11041 103/300 Flush
11042 104/300 Read block 3696 len 16384
11043 105/300 Read block 2000 len 40960
11044 106/300 Write block 1862 len 12288 data: 3 3 3
11045 107/300 Read block 1596 len 32768
11046 108/300 Write block 4970 len 24576 data: 2 2 3 3 2 3
11047 109/300 Flush
11048 110/300 Read block 3039 len 4096
11049 111/300 Write block 4978 len 32768 data: 3 4 4 4 4 4 3 3
11050 112/300 Write block 4170 len 12288 data: 7 6 5
11051 113/300 Read block 3960 len 16384
11052 114/300 Read block 4732 len 28672
11053 115/300 Read block 3837 len 36864
11054 116/300 Write block 2350 len 4096 data: 6
11055 117/300 Write block 1771 len 32768 data: 2 2 2 2 2 3 3 3
11056 118/300 Read block 910 len 20480
11057 119/300 Read block 829 len 28672
11058 120/300 Flush
11059 121/300 Read block 69 len 40960
11060 122/300 Read block 2790 len 28672
11061 123/300 Write block 1754 len 32768 data: 4 4 3 2 3 3 3 4
11062 124/300 Write block 3646 len 4096 data: 3
11063 125/300 Flush
11064 126/300 Flush
11065 127/300 Write block 3983 len 28672 data: 3 3 3 4 5 4 5
11066 128/300 Write block 4507 len 4096 data: 4
11067 129/300 Read block 4750 len 20480
11068 130/300 Read block 384 len 20480
11069 131/300 Read block 2117 len 28672
11070 132/300 Read block 4140 len 32768
11071 133/300 Write block 4462 len 4096 data: 3
11072 134/300 Write block 1017 len 24576 data: 2 2 3 4 4 3
11073 135/300 Read block 486 len 32768
11074 136/300 Read block 3869 len 36864
11075 137/300 Write block 1357 len 40960 data: 2 2 2 2 2 2 3 3 3 4
11076 138/300 Read block 2315 len 36864
11077 139/300 Read block 3197 len 24576
11078 140/300 Read block 72 len 16384
11079 141/300 Write block 4394 len 20480 data: 4 5 5 5 8
11080 142/300 Write block 4457 len 16384 data: 3 3 3 2
11081 143/300 Write block 4498 len 32768 data: 4 4 4 4 3 4 4 4
11082 144/300 Read block 4284 len 16384
11083 145/300 Write block 3534 len 32768 data: 4 4 5 6 8 8 6 5
11084 146/300 Read block 4044 len 20480
11085 147/300 Write block 3844 len 16384 data: 2 2 2 3
11086 148/300 Flush
11087 149/300 Read block 433 len 32768
11088 150/300 Write block 4100 len 32768 data: 3 4 4 4 3 3 3 3
11089 151/300 Read block 644 len 28672
11090 152/300 Write block 785 len 16384 data: 2 2 2 2
11091 153/300 Write block 299 len 4096 data: 4
11092 154/300 Flush
11093 155/300 Read block 4173 len 40960
11094 156/300 Write block 3187 len 40960 data: 3 3 3 3 3 5 6 5 5 4
11095 157/300 Write block 108 len 36864 data: 3 3 3 3 3 3 3 3 3
11096 158/300 Read block 4659 len 40960
11097 159/300 Read block 1172 len 4096
11098 160/300 Read block 1119 len 12288
11099 161/300 Write block 1490 len 12288 data: 6 6 6
11100 162/300 Write block 3366 len 16384 data: 3 2 2 3
11101 163/300 Read block 1929 len 24576
11102 164/300 Flush
11103 165/300 Write block 698 len 28672 data: 4 4 4 4 4 5 5
11104 166/300 Write block 1719 len 12288 data: 3 4 4
11105 167/300 Read block 3260 len 32768
11106 168/300 Read block 2698 len 32768
11107 169/300 Write block 2553 len 4096 data: 7
11108 170/300 Write block 2753 len 32768 data: 4 5 2 2 3 3 5 5
11109 171/300 Write block 2935 len 24576 data: 5 3 4 5 4 4
11110 172/300 Read block 3552 len 40960
11111 173/300 Write block 463 len 24576 data: 7 5 3 4 4 3
11112 174/300 Read block 1484 len 8192
11113 175/300 Write block 166 len 20480 data: 5 4 3 3 5
11114 176/300 Flush
11115 177/300 Write block 3145 len 8192 data: 7 7
11116 178/300 Write block 1651 len 24576 data: 5 4 3 3 3 3
11117 179/300 Write block 3139 len 20480 data: 6 5 6 6 7
11118 180/300 Write block 3617 len 16384 data: 4 5 6 7
11119 181/300 Write block 4153 len 40960 data: 5 5 4 4 4 4 2 4 4 4
11120 182/300 Read block 873 len 16384
11121 183/300 Read block 3457 len 32768
11122 184/300 Read block 325 len 36864
11123 185/300 Read block 698 len 12288
11124 186/300 Read block 4470 len 20480
11125 187/300 Read block 1466 len 40960
11126 188/300 Read block 4226 len 40960
11127 189/300 Flush
11128 190/300 Read block 2869 len 20480
11129 191/300 Read block 1133 len 32768
11130 192/300 Write block 3484 len 12288 data: 2 2 2
11131 193/300 Write block 3351 len 28672 data: 4 3 3 3 2 2 2
11132 194/300 Read block 1872 len 28672
11133 195/300 Flush
11134 196/300 Flush
11135 197/300 Read block 4338 len 24576
11136 198/300 Read block 4651 len 20480
11137 199/300 Write block 2303 len 32768 data: 6 6 6 7 6 5 4 5
11138 200/300 Read block 628 len 32768
11139 201/300 Write block 953 len 40960 data: 5 4 4 4 4 5 5 5 5 4
11140 202/300 Write block 1068 len 24576 data: 5 6 6 5 4 4
11141 203/300 Read block 2363 len 36864
11142 204/300 Read block 2533 len 4096
11143 205/300 Write block 2879 len 36864 data: 2 2 2 3 4 4 6 6 5
11144 206/300 Read block 630 len 12288
11145 207/300 Flush
11146 208/300 Read block 98 len 4096
11147 209/300 Flush
11148 210/300 Write block 3324 len 16384 data: 2 2 2 2
11149 211/300 Read block 3793 len 40960
11150 212/300 Write block 131 len 36864 data: 9 8 7 3 3 4 5 5 5
11151 213/300 Write block 4981 len 8192 data: 5 5
11152 214/300 Write block 593 len 16384 data: 3 3 3 3
11153 215/300 Flush
11154 216/300 Read block 4971 len 8192
11155 217/300 Write block 335 len 20480 data: 3 4 3 3 3
11156 218/300 Flush
11157 219/300 Write block 1522 len 32768 data: 3 3 3 4 4 4 4 3
11158 220/300 Read block 899 len 24576
11159 221/300 Flush
11160 222/300 Read block 3000 len 40960
11161 223/300 Write block 278 len 32768 data: 4 4 4 4 5 4 4 4
11162 224/300 Write block 4217 len 24576 data: 4 4 5 5 5 5
11163 225/300 Write block 3894 len 8192 data: 2 2
11164 226/300 Write block 4825 len 24576 data: 4 4 3 3 3 2
11165 227/300 Flush
11166 228/300 Read block 3564 len 40960
11167 229/300 Read block 1602 len 20480
11168 230/300 Read block 4334 len 24576
11169 231/300 Read block 2702 len 24576
11170 232/300 Write block 3937 len 36864 data: 6 6 6 5 5 5 6 6 5
11171 233/300 Write block 2892 len 12288 data: 5 2 2
11172 234/300 Read block 715 len 32768
11173 235/300 Read block 1019 len 24576
11174 236/300 Read block 4371 len 4096
11175 237/300 Write block 2405 len 4096 data: 4
11176 238/300 Read block 705 len 40960
11177 239/300 Write block 2299 len 36864 data: 4 4 5 5 7 7 7 8 7
11178 240/300 Read block 2598 len 16384
11179 241/300 Write block 4772 len 40960 data: 2 2 2 3 3 4 4 5 5 5
11180 242/300 Write block 4252 len 20480 data: 3 3 4 4 4
11181 243/300 Flush
11182 244/300 Read block 4912 len 20480
11183 245/300 Read block 660 len 12288
11184 246/300 Read block 2271 len 32768
11185 247/300 Read block 347 len 12288
11186 248/300 Read block 4344 len 20480
11187 249/300 Read block 1936 len 24576
11188 250/300 Write block 1852 len 32768 data: 3 5 5 5 5 4 4 4
11189 251/300 Read block 3846 len 8192
11190 252/300 Flush
11191 253/300 Write block 3255 len 32768 data: 4 4 4 3 4 5 5 5
11192 254/300 Read block 1784 len 36864
11193 255/300 Read block 380 len 24576
11194 256/300 Read block 3446 len 12288
11195 257/300 Flush
11196 258/300 Read block 925 len 16384
11197 259/300 Flush
11198 260/300 Flush
11199 261/300 Read block 4851 len 8192
11200 262/300 Flush
11201 263/300 Flush
11202 264/300 Flush
11203 265/300 Read block 3920 len 32768
11204 266/300 Flush
11205 267/300 Read block 3564 len 40960
11206 268/300 Write block 1893 len 4096 data: 2
11207 269/300 Read block 4982 len 28672
11208 270/300 Read block 4736 len 36864
11209 271/300 Write block 457 len 8192 data: 4 4
11210 272/300 Write block 4280 len 24576 data: 4 3 3 4 4 4
11211 273/300 Write block 1453 len 8192 data: 5 5
11212 274/300 Write block 4932 len 12288 data: 4 5 5
11213 275/300 Write block 2865 len 20480 data: 4 4 4 5 6
11214 276/300 Write block 3245 len 36864 data: 2 3 3 3 4 4 4 3 3
11215 277/300 Read block 579 len 8192
11216 278/300 Write block 1833 len 36864 data: 4 4 5 4 4 4 3 3 3
11217 279/300 Read block 4927 len 16384
11218 280/300 Write block 888 len 24576 data: 4 4 4 4 3 3
11219 281/300 Read block 902 len 20480
11220 282/300 Read block 826 len 24576
11221 283/300 Read block 2679 len 28672
11222 284/300 Read block 4934 len 28672
11223 285/300 Write block 1785 len 20480 data: 4 3 3 3 5
11224 286/300 Read block 2669 len 32768
11225 287/300 Write block 4311 len 16384 data: 6 7 6 6
11226 288/300 Read block 4244 len 24576
11227 289/300 Read block 4914 len 28672
11228 290/300 Read block 3091 len 32768
11229 291/300 Read block 831 len 32768
11230 292/300 Read block 3298 len 16384
11231 293/300 Write block 1487 len 24576 data: 5 6 7 7 7 7
11232 294/300 Write block 2601 len 32768 data: 7 9 8 6 6 5 3 3
11233 295/300 Flush
11234 296/300 Read block 985 len 32768
11235 297/300 Write block 2823 len 28672 data: 2 3 3 3 3 3 3
11236 298/300 Read block 3270 len 40960
11237 299/300 Read block 1320 len 32768
11238 300/300 Read block 2657 len 4096
11239 Main task has work to do, go find it
11240 got dsc Start(1)
11241 start 1
11242 Oct 13 18:11:33.756 INFO request completed, latency_us: 295, response_code: 204, uri: /start/cid/1, method: POST, req_id: a23a41b7-bcbb-44bb-859e-dae3dbb9f611, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
11243 Replay: started 1, returned:Ok(())
11244 ----------------------------------------------------------------
11245 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
11246 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
11247 4238 Acked 5237 Write 4 Done New Done false
11248 4239 Acked 5238 Write 1 Done New Done false
11249 4240 Acked 5239 Write 3 Done New Done false
11250 4241 Acked 5240 Write 5 Done New Done false
11251 4242 Acked 5241 Write 3 Done New Done false
11252 4243 Acked 5242 Write 1 Done New Done false
11253 4244 Acked 5243 Flush 0 Done New Done false
11254 4245 Acked 5244 Write 10 Done New Done false
11255 4246 Acked 5245 Flush 0 Done New Done false
11256 4247 Acked 5246 Read 2 Done New Done false
11257 4248 Acked 5247 Read 9 Done New Done false
11258 4249 Acked 5248 Read 1 Done New Done false
11259 4250 Acked 5249 Read 9 Done New Done false
11260 4251 Acked 5250 Read 8 Done New Done false
11261 4252 Acked 5251 Read 6 Done New Done false
11262 4253 Acked 5252 Flush 0 Done New Done false
11263 4254 Acked 5253 Read 6 Done New Done false
11264 4255 Acked 5254 Read 2 Done New Done false
11265 4256 Acked 5255 Write 9 Done New Done false
11266 4257 Acked 5256 Read 4 Done New Done false
11267 4258 Acked 5257 Flush 0 Done New Done false
11268 4259 Acked 5258 Read 4 Done New Done false
11269 4260 Acked 5259 Write 1 Done New Done false
11270 4261 Acked 5260 Write 7 Done New Done false
11271 4262 Acked 5261 Read 7 Done New Done false
11272 4263 Acked 5262 Read 10 Done New Done false
11273 4264 Acked 5263 Write 9 Done New Done false
11274 4265 Acked 5264 Flush 0 Done New Done false
11275 4266 Acked 5265 Write 7 Done New Done false
11276 4267 Acked 5266 Flush 0 Done New Done false
11277 4268 Acked 5267 Write 6 Done New Done false
11278 4269 Acked 5268 Flush 0 Done New Done false
11279 4270 Acked 5269 Write 9 Done New Done false
11280 4271 Acked 5270 Write 2 Done New Done false
11281 4272 Acked 5271 Read 3 Done New Done false
11282 4273 Acked 5272 Read 9 Done New Done false
11283 4274 Acked 5273 Write 5 Done New Done false
11284 4275 Acked 5274 Read 3 Done New Done false
11285 4276 Acked 5275 Read 8 Done New Done false
11286 4277 Acked 5276 Read 2 Done New Done false
11287 4278 Acked 5277 Write 4 Done New Done false
11288 4279 Acked 5278 Write 8 Done New Done false
11289 4280 Acked 5279 Read 8 Done New Done false
11290 4281 Acked 5280 Read 4 Done New Done false
11291 4282 Acked 5281 Read 8 Done New Done false
11292 4283 Acked 5282 Write 4 Done New Done false
11293 4284 Acked 5283 Write 9 Done New Done false
11294 4285 Acked 5284 Write 6 Done New Done false
11295 4286 Acked 5285 Read 1 Done New Done false
11296 4287 Acked 5286 Flush 0 Done New Done false
11297 4288 Acked 5287 Read 5 Done New Done false
11298 4289 Acked 5288 Write 7 Done New Done false
11299 4290 Acked 5289 Read 5 Done New Done false
11300 4291 Acked 5290 Flush 0 Done New Done false
11301 4292 Acked 5291 Read 10 Done New Done false
11302 4293 Acked 5292 Write 10 Done New Done false
11303 4294 Acked 5293 Read 7 Done New Done false
11304 4295 Acked 5294 Write 6 Done New Done false
11305 4296 Acked 5295 Write 1 Done New Done false
11306 4297 Acked 5296 Read 2 Done New Done false
11307 4298 Acked 5297 Read 3 Done New Done false
11308 4299 Acked 5298 Read 6 Done New Done false
11309 4300 Acked 5299 Read 4 Done New Done false
11310 4301 Acked 5300 Write 7 Done New Done false
11311 4302 Acked 5301 Write 9 Done New Done false
11312 4303 Acked 5302 Read 9 Done New Done false
11313 4304 Acked 5303 Flush 0 Done New Done false
11314 4305 Acked 5304 Read 2 Done New Done false
11315 4306 Acked 5305 Read 6 Done New Done false
11316 4307 Acked 5306 Read 7 Done New Done false
11317 4308 Acked 5307 Read 8 Done New Done false
11318 4309 Acked 5308 Write 6 Done New Done false
11319 4310 Acked 5309 Read 4 Done New Done false
11320 4311 Acked 5310 Read 7 Done New Done false
11321 4312 Acked 5311 Read 5 Done New Done false
11322 4313 Acked 5312 Read 4 Done New Done false
11323 4314 Acked 5313 Read 2 Done New Done false
11324 4315 Acked 5314 Write 10 Done New Done false
11325 4316 Acked 5315 Write 5 Done New Done false
11326 4317 Acked 5316 Read 10 Done New Done false
11327 4318 Acked 5317 Write 10 Done New Done false
11328 4319 Acked 5318 Read 2 Done New Done false
11329 4320 Acked 5319 Read 4 Done New Done false
11330 4321 Acked 5320 Write 7 Done New Done false
11331 4322 Acked 5321 Read 6 Done New Done false
11332 4323 Acked 5322 Write 3 Done New Done false
11333 4324 Acked 5323 Write 3 Done New Done false
11334 4325 Acked 5324 Write 9 Done New Done false
11335 4326 Acked 5325 Write 3 Done New Done false
11336 4327 Acked 5326 Read 7 Done New Done false
11337 4328 Acked 5327 Write 6 Done New Done false
11338 4329 Acked 5328 Read 4 Done New Done false
11339 4330 Acked 5329 Write 7 Done New Done false
11340 4331 Acked 5330 Read 3 Done New Done false
11341 4332 Acked 5331 Read 5 Done New Done false
11342 4333 Acked 5332 Read 3 Done New Done false
11343 4334 Acked 5333 Read 9 Done New Done false
11344 4335 Acked 5334 Write 8 Done New Done false
11345 4336 Acked 5335 Write 5 Done New Done false
11346 4337 Acked 5336 Write 7 Done New Done false
11347 4338 Acked 5337 Write 5 Done New Done false
11348 4339 Acked 5338 Write 4 Done New Done false
11349 4340 Acked 5339 Write 3 Done New Done false
11350 4341 Acked 5340 Flush 0 Done New Done false
11351 4342 Acked 5341 Read 4 Done New Done false
11352 4343 Acked 5342 Read 10 Done New Done false
11353 4344 Acked 5343 Write 3 Done New Done false
11354 4345 Acked 5344 Read 8 Done New Done false
11355 4346 Acked 5345 Write 6 Done New Done false
11356 4347 Acked 5346 Flush 0 Done New Done false
11357 4348 Acked 5347 Read 1 Done New Done false
11358 4349 Acked 5348 Write 8 Done New Done false
11359 4350 Acked 5349 Write 3 Done New Done false
11360 4351 Acked 5350 Read 4 Done New Done false
11361 4352 Acked 5351 Read 7 Done New Done false
11362 4353 Acked 5352 Read 9 Done New Done false
11363 4354 Acked 5353 Write 1 Done New Done false
11364 4355 Acked 5354 Write 8 Done New Done false
11365 4356 Acked 5355 Read 5 Done New Done false
11366 4357 Acked 5356 Read 7 Done New Done false
11367 4358 Acked 5357 Flush 0 Done New Done false
11368 4359 Acked 5358 Read 10 Done New Done false
11369 4360 Acked 5359 Read 7 Done New Done false
11370 4361 Acked 5360 Write 8 Done New Done false
11371 4362 Acked 5361 Write 1 Done New Done false
11372 4363 Acked 5362 Flush 0 Done New Done false
11373 4364 Acked 5363 Flush 0 Done New Done false
11374 4365 Acked 5364 Write 7 Done New Done false
11375 4366 Acked 5365 Write 1 Done New Done false
11376 4367 Acked 5366 Read 5 Done New Done false
11377 4368 Acked 5367 Read 5 Done New Done false
11378 4369 Acked 5368 Read 7 Done New Done false
11379 4370 Acked 5369 Read 8 Done New Done false
11380 4371 Acked 5370 Write 1 Done New Done false
11381 4372 Acked 5371 Write 6 Done New Done false
11382 4373 Acked 5372 Read 8 Done New Done false
11383 4374 Acked 5373 Read 9 Done New Done false
11384 4375 Acked 5374 Write 10 Done New Done false
11385 4376 Acked 5375 Read 9 Done New Done false
11386 4377 Acked 5376 Read 6 Done New Done false
11387 4378 Acked 5377 Read 4 Done New Done false
11388 4379 Acked 5378 Write 5 Done New Done false
11389 4380 Acked 5379 Write 4 Done New Done false
11390 4381 Acked 5380 Write 8 Done New Done false
11391 4382 Acked 5381 Read 4 Done New Done false
11392 4383 Acked 5382 Write 8 Done New Done false
11393 4384 Acked 5383 Read 5 Done New Done false
11394 4385 Acked 5384 Write 4 Done New Done false
11395 4386 Acked 5385 Flush 0 Done New Done false
11396 4387 Acked 5386 Read 8 Done New Done false
11397 4388 Acked 5387 Write 8 Done New Done false
11398 4389 Acked 5388 Read 7 Done New Done false
11399 4390 Acked 5389 Write 4 Done New Done false
11400 4391 Acked 5390 Write 1 Done New Done false
11401 4392 Acked 5391 Flush 0 Done New Done false
11402 4393 Acked 5392 Read 10 Done New Done false
11403 4394 Acked 5393 Write 10 Done New Done false
11404 4395 Acked 5394 Write 9 Done New Done false
11405 4396 Acked 5395 Read 10 Done New Done false
11406 4397 Acked 5396 Read 1 Done New Done false
11407 4398 Acked 5397 Read 3 Done New Done false
11408 4399 Acked 5398 Write 3 Done New Done false
11409 4400 Acked 5399 Write 4 Done New Done false
11410 4401 Acked 5400 Read 6 Done New Done false
11411 4402 Acked 5401 Flush 0 Done New Done false
11412 4403 Acked 5402 Write 7 Done New Done false
11413 4404 Acked 5403 Write 3 Done New Done false
11414 4405 Acked 5404 Read 8 Done New Done false
11415 4406 Acked 5405 Read 8 Done New Done false
11416 4407 Acked 5406 Write 1 Done New Done false
11417 4408 Acked 5407 Write 8 Done New Done false
11418 4409 Acked 5408 Write 6 Done New Done false
11419 4410 Acked 5409 Read 10 Done New Done false
11420 4411 Acked 5410 Flush 0 Done New Done false
11421 4412 Acked 5411 Write 6 Done New Done false
11422 4413 Acked 5412 Read 2 Done New Done false
11423 4414 Acked 5413 Write 5 Done New Done false
11424 4415 Acked 5414 Flush 0 Done New Done false
11425 4416 Acked 5415 Write 2 Done New Done false
11426 4417 Acked 5416 Write 6 Done New Done false
11427 4418 Acked 5417 Write 5 Done New Done false
11428 4419 Acked 5418 Write 4 Done New Done false
11429 4420 Acked 5419 Write 10 Done New Done false
11430 4421 Acked 5420 Read 4 Done New Done false
11431 4422 Acked 5421 Read 8 Done New Done false
11432 4423 Acked 5422 Read 9 Done New Done false
11433 4424 Acked 5423 Read 3 Done New Done false
11434 4425 Acked 5424 Read 5 Done New Done false
11435 4426 Acked 5425 Read 10 Done New Done false
11436 4427 Acked 5426 Read 10 Done New Done false
11437 4428 Acked 5427 Flush 0 Done New Done false
11438 4429 Acked 5428 Read 5 Done New Done false
11439 4430 Acked 5429 Read 8 Done New Done false
11440 4431 Acked 5430 Write 3 Done New Done false
11441 4432 Acked 5431 Write 7 Done New Done false
11442 4433 Acked 5432 Read 7 Done New Done false
11443 4434 Acked 5433 Flush 0 Done New Done false
11444 4435 Acked 5434 Flush 0 Done New Done false
11445 4436 Acked 5435 Read 6 Done New Done false
11446 4437 Acked 5436 Read 5 Done New Done false
11447 4438 Acked 5437 Write 8 Done New Done false
11448 4439 Acked 5438 Read 8 Done New Done false
11449 4440 Acked 5439 Write 10 Done New Done false
11450 4441 Acked 5440 Write 6 Done New Done false
11451 4442 Acked 5441 Read 9 Done New Done false
11452 4443 Acked 5442 Read 1 Done New Done false
11453 4444 Acked 5443 Write 9 Done New Done false
11454 4445 Acked 5444 Read 3 Done New Done false
11455 4446 Acked 5445 Flush 0 Done New Done false
11456 4447 Acked 5446 Read 1 Done New Done false
11457 4448 Acked 5447 Flush 0 Done New Done false
11458 4449 Acked 5448 Write 4 Done New Done false
11459 4450 Acked 5449 Read 10 Done New Done false
11460 4451 Acked 5450 Write 9 Done New Done false
11461 4452 Acked 5451 Write 2 Done New Done false
11462 4453 Acked 5452 Write 4 Done New Done false
11463 4454 Acked 5453 Flush 0 Done New Done false
11464 4455 Acked 5454 Read 2 Done New Done false
11465 4456 Acked 5455 Write 5 Done New Done false
11466 4457 Acked 5456 Flush 0 Done New Done false
11467 4458 Acked 5457 Write 8 Done New Done false
11468 4459 Acked 5458 Read 6 Done New Done false
11469 4460 Acked 5459 Flush 0 Done New Done false
11470 4461 Acked 5460 Read 10 Done New Done false
11471 4462 Acked 5461 Write 8 Done New Done false
11472 4463 Acked 5462 Write 6 Done New Done false
11473 4464 Acked 5463 Write 2 Done New Done false
11474 4465 Acked 5464 Write 6 Done New Done false
11475 4466 Acked 5465 Flush 0 Done New Done false
11476 4467 Acked 5466 Read 10 Done New Done false
11477 4468 Acked 5467 Read 5 Done New Done false
11478 4469 Acked 5468 Read 6 Done New Done false
11479 4470 Acked 5469 Read 6 Done New Done false
11480 4471 Acked 5470 Write 9 Done New Done false
11481 4472 Acked 5471 Write 3 Done New Done false
11482 4473 Acked 5472 Read 8 Done New Done false
11483 4474 Acked 5473 Read 6 Done New Done false
11484 4475 Acked 5474 Read 1 Done New Done false
11485 4476 Acked 5475 Write 1 Done New Done false
11486 4477 Acked 5476 Read 10 Done New Done false
11487 4478 Acked 5477 Write 9 Done New Done false
11488 4479 Acked 5478 Read 4 Done New Done false
11489 4480 Acked 5479 Write 10 Done New Done false
11490 4481 Acked 5480 Write 5 Done New Done false
11491 4482 Acked 5481 Flush 0 Done New Done false
11492 4483 Acked 5482 Read 5 Done New Done false
11493 4484 Acked 5483 Read 3 Done New Done false
11494 4485 Acked 5484 Read 8 Done New Done false
11495 4486 Acked 5485 Read 3 Done New Done false
11496 4487 Acked 5486 Read 5 Done New Done false
11497 4488 Acked 5487 Read 6 Done New Done false
11498 4489 Acked 5488 Write 8 Done New Done false
11499 4490 Acked 5489 Read 2 Done New Done false
11500 4491 Acked 5490 Flush 0 Done New Done false
11501 4492 Acked 5491 Write 8 Done New Done false
11502 4493 Acked 5492 Read 9 Done New Done false
11503 4494 Acked 5493 Read 6 Done New Done false
11504 4495 Acked 5494 Read 3 Done New Done false
11505 4496 Acked 5495 Flush 0 Done New Done false
11506 4497 Acked 5496 Read 4 Done New Done false
11507 4498 Acked 5497 Flush 0 Done New Done false
11508 4499 Acked 5498 Flush 0 Done New Done false
11509 4500 Acked 5499 Read 2 Done New Done false
11510 4501 Acked 5500 Flush 0 Done New Done false
11511 4502 Acked 5501 Flush 0 Done New Done false
11512 4503 Acked 5502 Flush 0 Done New Done false
11513 4504 Acked 5503 Read 8 Done New Done false
11514 4505 Acked 5504 Flush 0 Done New Done false
11515 4506 Acked 5505 Read 10 Done New Done false
11516 4507 Acked 5506 Write 1 Done New Done false
11517 4508 Acked 5507 Read 7 Done New Done false
11518 4509 Acked 5508 Read 9 Done New Done false
11519 4510 Acked 5509 Write 2 Done New Done false
11520 4511 Acked 5510 Write 6 Done New Done false
11521 4512 Acked 5511 Write 2 Done New Done false
11522 4513 Acked 5512 Write 3 Done New Done false
11523 4514 Acked 5513 Write 5 Done New Done false
11524 4515 Acked 5514 Write 9 Done New Done false
11525 4516 Acked 5515 Read 2 Done New Done false
11526 4517 Acked 5516 Write 9 Done New Done false
11527 4518 Acked 5517 Read 4 Done New Done false
11528 4519 Acked 5518 Write 6 Done New Done false
11529 4520 Acked 5519 Read 5 Done New Done false
11530 4521 Acked 5520 Read 6 Done New Done false
11531 4522 Acked 5521 Read 7 Done New Done false
11532 4523 Acked 5522 Read 7 Done New Done false
11533 4524 Acked 5523 Write 5 Done New Done false
11534 4525 Acked 5524 Read 8 Done New Done false
11535 4526 Acked 5525 Write 4 Done New Done false
11536 4527 Acked 5526 Read 6 Done New Done false
11537 4528 Acked 5527 Read 7 Done New Done false
11538 4529 Acked 5528 Read 8 Done New Done false
11539 4530 Acked 5529 Read 8 Done New Done false
11540 4531 Acked 5530 Read 4 Done New Done false
11541 4532 Acked 5531 Write 6 Done New Done false
11542 4533 Acked 5532 Write 8 Done New Done false
11543 4534 Acked 5533 Flush 0 Done New Done false
11544 4535 Acked 5534 Read 8 Done New Done false
11545 4536 Acked 5535 Write 7 Done New Done false
11546 4537 Acked 5536 Read 10 Done New Done false
11547 4538 Acked 5537 Read 8 Done New Done false
11548 4539 Acked 5538 Read 1 Done New Done false
11549 STATES DS:0 DS:1 DS:2 TOTAL
11550 New 0 302 0 302
11551 Sent 0 0 0 0
11552 Done 302 0 302 604
11553 Skipped 0 0 0 0
11554 Error 0 0 0 0
11555 Last Flush: 5533 5236 5533
11556 Downstairs last five completed: 5236 5235 5234 5233 5232 5231
11557 Upstairs last five completed: 4539 4538 4537 4536 4535 4534
11558 CLIENT: Up:0 ds:302 act:2
11559 [8820] got start action
11560 [8820] I am going to restart
11561 Starting downstairs at port 8820
11562 Make output file at "/tmp/dsc/downstairs-8820.txt"
11563 [8820][1] reports Starting
11564 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1191)
11565 [8820][1] reports Running
115662023-10-13T18:11:35.800ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 1
115672023-10-13T18:11:35.800ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
115682023-10-13T18:11:35.800ZINFOcrucible: [1] client re-new 303 jobs since flush 5236 = downstairs
115692023-10-13T18:11:35.801ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
115702023-10-13T18:11:35.802ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
115712023-10-13T18:11:35.802ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
115722023-10-13T18:11:35.802ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
115732023-10-13T18:11:35.802ZINFOcrucible: [1] send last flush ID to this DS: 5236
115742023-10-13T18:11:35.802ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
115752023-10-13T18:11:35.802ZINFOcrucible: [1] Replied this last flush ID: 5236
115762023-10-13T18:11:35.802ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Offline Active ds_transition to Replay
115772023-10-13T18:11:35.802ZINFOcrucible: [1] Transition from Offline to Replay
115782023-10-13T18:11:35.802ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
115792023-10-13T18:11:35.802ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Replay Active ds_transition to Active
115802023-10-13T18:11:35.802ZINFOcrucible: [1] Transition from Replay to Active
115812023-10-13T18:11:35.802ZINFOcrucible: [1] Starts cmd_loop
115822023-10-13T18:11:35.802ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
115832023-10-13T18:11:35.802ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
115842023-10-13T18:11:35.802ZINFOcrucible: Set check for repair
115852023-10-13T18:11:36.803ZINFOcrucible: Checking if live repair is needed
115862023-10-13T18:11:36.803ZINFOcrucible: No Live Repair required at this time
115872023-10-13T18:11:36.803ZWARNcrucible: [1] flow control end
11588 ----------------------------------------------------------------
11589 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11590 Downstairs last five completed: 5539 5538 5537 5536 5535 5534
11591 Upstairs last five completed: 4540 4539 4538 4537 4536 4535
11592 CLIENT: Up:0 ds:0 act:3
11593 Replay: All jobs finished, all DS active.
11594 Main task has work to do, go find it
11595 got dsc Stop(2)
11596 stop 2
11597 Oct 13 18:11:37.760 INFO request completed, latency_us: 511, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 09249007-aa3f-4807-9ebd-a2a13a13a2b7, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.[8830] Got stop action so:false kr:false
11598 1:9998
11599 Oct 13 18:11:37.762 INFO request completed, latency_us: 166, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0a8aeece-3e0e-4a26-9f19-306067c4dad5, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
116002023-10-13T18:11:37.762ZWARNcrucible: [2] downstairs disconnected
116012023-10-13T18:11:37.762ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
116022023-10-13T18:11:37.762ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8830 closed looper = 2
116032023-10-13T18:11:37.762ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
116042023-10-13T18:11:37.762ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
116052023-10-13T18:11:37.762ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
116062023-10-13T18:11:37.762ZWARNcrucible: [2] pm_task rx.recv() is None
11607 [8830] Exited with: ExitStatus(unix_wait_status(9))
11608 [8830][2] reports Exit
11609 Oct 13 18:11:41.764 INFO request completed, latency_us: 255, response_code: 200, uri: /state/cid/2, method: GET, req_id: d3460e2b-58bf-4527-8966-33eb4a172195, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
11610 001/300 Read block 3789 len 16384
11611 002/300 Write block 1177 len 40960 data: 4 3 3 2 2 2 3 3 3 2
11612 003/300 Write block 2605 len 24576 data: 7 6 4 4 2 3
11613 004/300 Write block 3040 len 24576 data: 4 4 6 5 5 5
11614 005/300 Write block 767 len 36864 data: 2 2 2 3 3 3 4 4 4
11615 006/300 Read block 1317 len 8192
11616 007/300 Read block 4963 len 36864
11617 008/300 Flush
11618 009/300 Read block 818 len 36864
11619 010/300 Write block 4058 len 32768 data: 4 3 3 3 4 4 5 4
11620 011/300 Write block 1973 len 8192 data: 3 3
11621 012/300 Read block 1886 len 36864
11622 013/300 Write block 3041 len 12288 data: 5 7 6
11623 014/300 Read block 4809 len 8192
11624 015/300 Read block 4000 len 36864
11625 016/300 Read block 4413 len 36864
11626 017/300 Read block 3667 len 28672
11627 018/300 Write block 794 len 24576 data: 2 2 2 2 3 4
11628 019/300 Read block 241 len 4096
11629 020/300 Read block 3957 len 8192
11630 021/300 Flush
11631 022/300 Write block 314 len 20480 data: 5 4 4 4 6
11632 023/300 Read block 3493 len 40960
11633 024/300 Flush
11634 025/300 Write block 2848 len 20480 data: 3 3 2 2 2
11635 026/300 Read block 3429 len 32768
11636 027/300 Flush
11637 028/300 Read block 184 len 36864
11638 029/300 Read block 1487 len 8192
11639 030/300 Read block 47 len 28672
11640 031/300 Read block 891 len 16384
11641 032/300 Read block 4211 len 40960
11642 033/300 Flush
11643 034/300 Write block 1336 len 4096 data: 5
11644 035/300 Read block 433 len 16384
11645 036/300 Flush
11646 037/300 Read block 518 len 16384
11647 038/300 Write block 4324 len 24576 data: 7 7 8 6 5 7
11648 039/300 Write block 3301 len 12288 data: 2 3 3
11649 040/300 Write block 1099 len 24576 data: 9 9 6 5 3 3
11650 041/300 Write block 1616 len 12288 data: 5 4 5
11651 042/300 Flush
11652 043/300 Read block 3167 len 4096
11653 044/300 Read block 2028 len 28672
11654 045/300 Read block 492 len 24576
11655 046/300 Write block 4585 len 28672 data: 3 3 3 2 3 3 2
11656 047/300 Write block 344 len 28672 data: 3 3 3 3 3 3 3
11657 048/300 Write block 938 len 16384 data: 5 5 4 3
11658 049/300 Flush
11659 050/300 Read block 1482 len 4096
11660 051/300 Read block 1315 len 12288
11661 052/300 Write block 3171 len 12288 data: 2 3 4
11662 053/300 Read block 1897 len 4096
11663 054/300 Read block 4605 len 40960
11664 055/300 Read block 4060 len 16384
11665 056/300 Write block 2938 len 12288 data: 6 5 5
11666 057/300 Read block 3554 len 40960
11667 058/300 Read block 1222 len 12288
11668 059/300 Read block 2528 len 16384
11669 060/300 Write block 319 len 4096 data: 7
11670 061/300 Read block 518 len 28672
11671 062/300 Read block 3173 len 40960
11672 063/300 Flush
11673 064/300 Read block 2069 len 32768
11674 065/300 Read block 625 len 8192
11675 066/300 Read block 1842 len 28672
11676 067/300 Write block 555 len 20480 data: 5 6 8 7 6
11677 068/300 Flush
11678 069/300 Read block 2635 len 8192
11679 070/300 Read block 2065 len 12288
11680 071/300 Write block 1973 len 8192 data: 4 4
11681 072/300 Read block 3235 len 20480
11682 073/300 Write block 4014 len 8192 data: 2 3
11683 074/300 Write block 4970 len 28672 data: 3 3 4 4 3 4 3
11684 075/300 Read block 1991 len 8192
11685 076/300 Write block 1244 len 16384 data: 2 2 2 2
11686 077/300 Read block 117 len 16384
11687 078/300 Read block 3450 len 36864
11688 079/300 Flush
11689 080/300 Write block 4321 len 28672 data: 2 4 4 8 8 9 7
11690 081/300 Flush
11691 082/300 Write block 2256 len 32768 data: 3 3 3 4 5 3 3 2
11692 083/300 Write block 3917 len 8192 data: 6 5
11693 084/300 Write block 1208 len 32768 data: 4 4 4 4 3 2 2 4
11694 085/300 Read block 2675 len 8192
11695 086/300 Read block 523 len 20480
11696 087/300 Read block 3076 len 16384
11697 088/300 Flush
11698 089/300 Write block 3964 len 28672 data: 4 5 4 3 3 4 4
11699 090/300 Read block 23 len 16384
11700 091/300 Read block 3598 len 12288
11701 092/300 Write block 2381 len 20480 data: 4 4 4 4 3
11702 093/300 Write block 167 len 4096 data: 5
11703 094/300 Read block 2369 len 16384
11704 095/300 Read block 1253 len 32768
11705 096/300 Read block 4915 len 36864
11706 097/300 Read block 875 len 32768
11707 098/300 Read block 3302 len 28672
11708 099/300 Read block 2067 len 32768
11709 100/300 Write block 885 len 16384 data: 3 4 4 5
11710 101/300 Read block 4077 len 4096
11711 102/300 Flush
11712 103/300 Read block 3747 len 16384
11713 104/300 Write block 1102 len 16384 data: 6 4 4 3
11714 105/300 Write block 2688 len 20480 data: 2 2 2 2 2
11715 106/300 Read block 677 len 8192
11716 107/300 Write block 3164 len 28672 data: 3 2 2 2 2 2 2
11717 108/300 Write block 2580 len 36864 data: 3 4 4 3 3 5 5 4 5
11718 109/300 Read block 2966 len 40960
11719 110/300 Write block 19 len 20480 data: 3 4 5 5 4
11720 111/300 Write block 1457 len 40960 data: 4 2 2 2 2 2 3 3 3 4
11721 112/300 Write block 2144 len 8192 data: 5 4
11722 113/300 Write block 4199 len 20480 data: 3 3 3 3 3
11723 114/300 Read block 2387 len 28672
11724 115/300 Read block 4507 len 8192
11725 116/300 Write block 2269 len 12288 data: 3 3 2
11726 117/300 Read block 3512 len 12288
11727 118/300 Read block 4970 len 24576
11728 119/300 Read block 4889 len 36864
11729 120/300 Read block 2888 len 28672
11730 121/300 Write block 3706 len 24576 data: 6 6 8 8 10 11
11731 122/300 Write block 4673 len 4096 data: 4
11732 123/300 Read block 2124 len 24576
11733 124/300 Write block 2110 len 28672 data: 4 4 4 4 4 3 2
11734 125/300 Read block 202 len 16384
11735 126/300 Write block 4892 len 8192 data: 4 3
11736 127/300 Read block 2744 len 36864
11737 128/300 Read block 2733 len 28672
11738 129/300 Flush
11739 130/300 Flush
11740 131/300 Read block 3435 len 36864
11741 132/300 Read block 3662 len 16384
11742 133/300 Write block 3405 len 16384 data: 4 5 5 5
11743 134/300 Read block 4811 len 32768
11744 135/300 Flush
11745 136/300 Flush
11746 137/300 Read block 4769 len 36864
11747 138/300 Read block 3380 len 12288
11748 139/300 Write block 351 len 8192 data: 3 3
11749 140/300 Write block 4101 len 20480 data: 5 5 5 4 4
11750 141/300 Write block 1541 len 24576 data: 2 2 2 2 2 2
11751 142/300 Write block 3153 len 4096 data: 3
11752 143/300 Write block 4745 len 8192 data: 8 7
11753 144/300 Write block 1931 len 28672 data: 4 5 5 7 5 5 7
11754 145/300 Read block 943 len 40960
11755 146/300 Read block 4962 len 40960
11756 147/300 Flush
11757 148/300 Read block 596 len 4096
11758 149/300 Flush
11759 150/300 Write block 3514 len 20480 data: 3 3 3 3 6
11760 151/300 Flush
11761 152/300 Flush
11762 153/300 Read block 706 len 28672
11763 154/300 Write block 695 len 4096 data: 2
11764 155/300 Flush
11765 156/300 Write block 1972 len 32768 data: 3 5 5 3 3 3 3 4
11766 157/300 Write block 1878 len 20480 data: 6 5 5 4 5
11767 158/300 Read block 1614 len 4096
11768 159/300 Write block 4129 len 40960 data: 2 3 3 3 3 2 2 2 2 2
11769 160/300 Read block 1049 len 12288
11770 161/300 Write block 287 len 32768 data: 4 3 3 2 2 2 2 2
11771 162/300 Write block 3577 len 32768 data: 3 2 2 2 2 2 2 2
11772 163/300 Write block 4321 len 8192 data: 3 5
11773 164/300 Read block 1743 len 16384
11774 165/300 Read block 1679 len 40960
11775 166/300 Read block 163 len 8192
11776 167/300 Read block 1845 len 36864
11777 168/300 Read block 1049 len 24576
11778 169/300 Flush
11779 170/300 Read block 1009 len 20480
11780 171/300 Read block 1467 len 20480
11781 172/300 Read block 1716 len 32768
11782 173/300 Write block 4803 len 40960 data: 7 7 6 5 6 6 5 6 6 5
11783 174/300 Read block 1650 len 12288
11784 175/300 Write block 4217 len 8192 data: 5 5
11785 176/300 Flush
11786 177/300 Read block 1157 len 4096
11787 178/300 Read block 3978 len 16384
11788 179/300 Write block 669 len 12288 data: 3 2 3
11789 180/300 Read block 481 len 40960
11790 181/300 Read block 3767 len 12288
11791 182/300 Write block 1556 len 12288 data: 2 4 3
11792 183/300 Read block 4611 len 20480
11793 184/300 Read block 193 len 32768
11794 185/300 Read block 135 len 20480
11795 186/300 Flush
11796 187/300 Write block 2097 len 16384 data: 3 3 3 3
11797 188/300 Flush
11798 189/300 Write block 3372 len 16384 data: 4 4 2 3
11799 190/300 Read block 3166 len 16384
11800 191/300 Read block 4691 len 12288
11801 192/300 Flush
11802 193/300 Flush
11803 194/300 Write block 4989 len 8192 data: 3 2
11804 195/300 Read block 69 len 36864
11805 196/300 Read block 4412 len 40960
11806 197/300 Write block 622 len 24576 data: 2 2 2 3 3 3
11807 198/300 Write block 2107 len 20480 data: 4 4 4 5 5
11808 199/300 Write block 1160 len 32768 data: 6 5 5 6 5 5 5 4
11809 200/300 Read block 1247 len 12288
11810 201/300 Write block 1220 len 16384 data: 3 4 4 3
11811 202/300 Read block 574 len 32768
11812 203/300 Read block 3642 len 8192
11813 204/300 Write block 4917 len 32768 data: 6 5 5 5 5 6 4 3
11814 205/300 Flush
11815 206/300 Read block 4365 len 4096
11816 207/300 Read block 2038 len 40960
11817 208/300 Write block 4146 len 24576 data: 3 3 3 4 5 5
11818 209/300 Write block 3514 len 16384 data: 4 4 4 4
11819 210/300 Write block 326 len 12288 data: 5 5 4
11820 211/300 Read block 4986 len 16384
11821 212/300 Read block 3786 len 4096
11822 213/300 Flush
11823 214/300 Write block 140 len 32768 data: 5 5 3 2 2 2 3 3
11824 215/300 Write block 2777 len 28672 data: 4 4 4 4 4 4 4
11825 216/300 Read block 4262 len 32768
11826 217/300 Read block 3124 len 4096
11827 218/300 Flush
11828 219/300 Flush
11829 220/300 Read block 2318 len 16384
11830 221/300 Read block 3385 len 28672
11831 222/300 Read block 1654 len 12288
11832 223/300 Read block 4428 len 24576
11833 224/300 Write block 253 len 12288 data: 4 5 4
11834 225/300 Write block 3526 len 8192 data: 4 2
11835 226/300 Read block 4734 len 24576
11836 227/300 Read block 201 len 36864
11837 228/300 Read block 4665 len 20480
11838 229/300 Read block 1666 len 40960
11839 230/300 Read block 751 len 8192
11840 231/300 Write block 4787 len 28672 data: 5 5 6 3 4 4 4
11841 232/300 Read block 64 len 24576
11842 233/300 Read block 2416 len 20480
11843 234/300 Write block 4693 len 4096 data: 3
11844 235/300 Read block 3513 len 4096
11845 236/300 Write block 2231 len 36864 data: 2 2 2 4 4 4 4 5 5
11846 237/300 Write block 2969 len 16384 data: 5 4 5 4
11847 238/300 Read block 1122 len 40960
11848 239/300 Write block 4851 len 40960 data: 4 4 3 3 4 4 4 5 6 7
11849 240/300 Write block 3013 len 12288 data: 6 5 4
11850 241/300 Read block 1602 len 16384
11851 242/300 Write block 2403 len 4096 data: 5
11852 243/300 Write block 4631 len 32768 data: 4 3 3 3 3 3 3 3
11853 244/300 Write block 2390 len 36864 data: 5 5 5 5 4 4 6 8 8
11854 245/300 Write block 3592 len 8192 data: 3 4
11855 246/300 Write block 1226 len 4096 data: 2
11856 247/300 Write block 4030 len 12288 data: 2 2 2
11857 248/300 Read block 1694 len 12288
11858 249/300 Write block 2685 len 32768 data: 2 2 2 3 3 3 3 3
11859 250/300 Write block 2500 len 40960 data: 3 4 4 3 3 3 4 4 6 5
11860 251/300 Read block 3965 len 16384
11861 252/300 Read block 934 len 8192
11862 253/300 Read block 2466 len 24576
11863 254/300 Flush
11864 255/300 Read block 995 len 32768
11865 256/300 Write block 2341 len 4096 data: 3
11866 257/300 Read block 3154 len 12288
11867 258/300 Read block 2658 len 20480
11868 259/300 Write block 1553 len 8192 data: 3 2
11869 260/300 Write block 1014 len 16384 data: 3 2 2 3
11870 261/300 Read block 2432 len 24576
11871 262/300 Write block 4731 len 24576 data: 4 4 4 3 3 3
11872 263/300 Write block 1796 len 12288 data: 3 2 3
11873 264/300 Flush
11874 265/300 Read block 4596 len 8192
11875 266/300 Read block 4328 len 28672
11876 267/300 Read block 1075 len 32768
11877 268/300 Write block 3832 len 8192 data: 4 4
11878 269/300 Read block 3576 len 16384
11879 270/300 Read block 629 len 8192
11880 271/300 Read block 4872 len 8192
11881 272/300 Read block 143 len 24576
11882 273/300 Write block 3937 len 36864 data: 7 7 7 6 6 6 7 7 6
11883 274/300 Write block 3555 len 20480 data: 2 2 2 2 2
11884 275/300 Read block 4244 len 20480
11885 276/300 Write block 4509 len 36864 data: 5 5 6 6 6 5 4 3 2
11886 277/300 Write block 1370 len 40960 data: 5 5 4 4 4 3 6 5 6 6
11887 278/300 Flush
11888 279/300 Write block 4976 len 28672 data: 4 3 4 5 5 6 6
11889 280/300 Read block 2934 len 40960
11890 281/300 Read block 1847 len 20480
11891 282/300 Read block 3707 len 20480
11892 283/300 Write block 3929 len 12288 data: 4 3 3
11893 284/300 Flush
11894 285/300 Read block 1079 len 36864
11895 286/300 Read block 2903 len 4096
11896 287/300 Write block 1908 len 40960 data: 4 5 4 4 4 4 4 4 3 3
11897 288/300 Write block 3267 len 24576 data: 2 2 2 2 2 2
11898 289/300 Write block 997 len 40960 data: 4 3 3 3 4 4 4 3 5 5
11899 290/300 Write block 1378 len 24576 data: 7 7 6 5 4 4
11900 291/300 Write block 840 len 28672 data: 6 5 5 3 4 4 5
11901 292/300 Flush
11902 293/300 Read block 84 len 16384
11903 294/300 Flush
11904 295/300 Write block 4195 len 24576 data: 5 7 6 3 4 4
11905 296/300 Read block 464 len 36864
11906 297/300 Read block 834 len 32768
11907 298/300 Write block 3675 len 36864 data: 2 2 2 2 3 5 4 4 4
11908 299/300 Read block 3696 len 8192
11909 300/300 Read block 2762 len 36864
11910 Main task has work to do, go find it
11911 got dsc Start(2)
11912 start 2
11913 Oct 13 18:11:42.710 INFO request completed, latency_us: 297, response_code: 204, uri: /start/cid/2, method: POST, req_id: 2d9941a8-c31a-432e-87e9-8bfeb8d63b43, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
11914 Replay: started 2, returned:Ok(())
11915 ----------------------------------------------------------------
11916 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
11917 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
11918 4541 Acked 5540 Read 4 Done Done New false
11919 4542 Acked 5541 Write 10 Done Done New false
11920 4543 Acked 5542 Write 6 Done Done New false
11921 4544 Acked 5543 Write 6 Done Done New false
11922 4545 Acked 5544 Write 9 Done Done New false
11923 4546 Acked 5545 Read 2 Done Done New false
11924 4547 Acked 5546 Read 9 Done Done New false
11925 4548 Acked 5547 Flush 0 Done Done New false
11926 4549 Acked 5548 Read 9 Done Done New false
11927 4550 Acked 5549 Write 8 Done Done New false
11928 4551 Acked 5550 Write 2 Done Done New false
11929 4552 Acked 5551 Read 9 Done Done New false
11930 4553 Acked 5552 Write 3 Done Done New false
11931 4554 Acked 5553 Read 2 Done Done New false
11932 4555 Acked 5554 Read 9 Done Done New false
11933 4556 Acked 5555 Read 9 Done Done New false
11934 4557 Acked 5556 Read 7 Done Done New false
11935 4558 Acked 5557 Write 6 Done Done New false
11936 4559 Acked 5558 Read 1 Done Done New false
11937 4560 Acked 5559 Read 2 Done Done New false
11938 4561 Acked 5560 Flush 0 Done Done New false
11939 4562 Acked 5561 Write 5 Done Done New false
11940 4563 Acked 5562 Read 10 Done Done New false
11941 4564 Acked 5563 Flush 0 Done Done New false
11942 4565 Acked 5564 Write 5 Done Done New false
11943 4566 Acked 5565 Read 8 Done Done New false
11944 4567 Acked 5566 Flush 0 Done Done New false
11945 4568 Acked 5567 Read 9 Done Done New false
11946 4569 Acked 5568 Read 2 Done Done New false
11947 4570 Acked 5569 Read 7 Done Done New false
11948 4571 Acked 5570 Read 4 Done Done New false
11949 4572 Acked 5571 Flush 0 Done Done New false
11950 4573 Acked 5572 Read 10 Done Done New false
11951 4574 Acked 5573 Flush 0 Done Done New false
11952 4575 Acked 5574 Write 1 Done Done New false
11953 4576 Acked 5575 Read 4 Done Done New false
11954 4577 Acked 5576 Flush 0 Done Done New false
11955 4578 Acked 5577 Read 4 Done Done New false
11956 4579 Acked 5578 Write 6 Done Done New false
11957 4580 Acked 5579 Write 3 Done Done New false
11958 4581 Acked 5580 Write 6 Done Done New false
11959 4582 Acked 5581 Write 3 Done Done New false
11960 4583 Acked 5582 Flush 0 Done Done New false
11961 4584 Acked 5583 Read 1 Done Done New false
11962 4585 Acked 5584 Read 7 Done Done New false
11963 4586 Acked 5585 Read 6 Done Done New false
11964 4587 Acked 5586 Write 7 Done Done New false
11965 4588 Acked 5587 Write 7 Done Done New false
11966 4589 Acked 5588 Write 4 Done Done New false
11967 4590 Acked 5589 Flush 0 Done Done New false
11968 4591 Acked 5590 Read 1 Done Done New false
11969 4592 Acked 5591 Read 3 Done Done New false
11970 4593 Acked 5592 Write 3 Done Done New false
11971 4594 Acked 5593 Read 1 Done Done New false
11972 4595 Acked 5594 Read 10 Done Done New false
11973 4596 Acked 5595 Read 4 Done Done New false
11974 4597 Acked 5596 Write 3 Done Done New false
11975 4598 Acked 5597 Read 10 Done Done New false
11976 4599 Acked 5598 Read 3 Done Done New false
11977 4600 Acked 5599 Read 4 Done Done New false
11978 4601 Acked 5600 Write 1 Done Done New false
11979 4602 Acked 5601 Read 7 Done Done New false
11980 4603 Acked 5602 Read 10 Done Done New false
11981 4604 Acked 5603 Flush 0 Done Done New false
11982 4605 Acked 5604 Read 8 Done Done New false
11983 4606 Acked 5605 Read 2 Done Done New false
11984 4607 Acked 5606 Read 7 Done Done New false
11985 4608 Acked 5607 Write 5 Done Done New false
11986 4609 Acked 5608 Flush 0 Done Done New false
11987 4610 Acked 5609 Read 2 Done Done New false
11988 4611 Acked 5610 Read 3 Done Done New false
11989 4612 Acked 5611 Write 2 Done Done New false
11990 4613 Acked 5612 Read 5 Done Done New false
11991 4614 Acked 5613 Write 2 Done Done New false
11992 4615 Acked 5614 Write 7 Done Done New false
11993 4616 Acked 5615 Read 2 Done Done New false
11994 4617 Acked 5616 Write 4 Done Done New false
11995 4618 Acked 5617 Read 4 Done Done New false
11996 4619 Acked 5618 Read 9 Done Done New false
11997 4620 Acked 5619 Flush 0 Done Done New false
11998 4621 Acked 5620 Write 7 Done Done New false
11999 4622 Acked 5621 Flush 0 Done Done New false
12000 4623 Acked 5622 Write 8 Done Done New false
12001 4624 Acked 5623 Write 2 Done Done New false
12002 4625 Acked 5624 Write 8 Done Done New false
12003 4626 Acked 5625 Read 2 Done Done New false
12004 4627 Acked 5626 Read 5 Done Done New false
12005 4628 Acked 5627 Read 4 Done Done New false
12006 4629 Acked 5628 Flush 0 Done Done New false
12007 4630 Acked 5629 Write 7 Done Done New false
12008 4631 Acked 5630 Read 4 Done Done New false
12009 4632 Acked 5631 Read 3 Done Done New false
12010 4633 Acked 5632 Write 5 Done Done New false
12011 4634 Acked 5633 Write 1 Done Done New false
12012 4635 Acked 5634 Read 4 Done Done New false
12013 4636 Acked 5635 Read 8 Done Done New false
12014 4637 Acked 5636 Read 9 Done Done New false
12015 4638 Acked 5637 Read 8 Done Done New false
12016 4639 Acked 5638 Read 7 Done Done New false
12017 4640 Acked 5639 Read 8 Done Done New false
12018 4641 Acked 5640 Write 4 Done Done New false
12019 4642 Acked 5641 Read 1 Done Done New false
12020 4643 Acked 5642 Flush 0 Done Done New false
12021 4644 Acked 5643 Read 4 Done Done New false
12022 4645 Acked 5644 Write 4 Done Done New false
12023 4646 Acked 5645 Write 5 Done Done New false
12024 4647 Acked 5646 Read 2 Done Done New false
12025 4648 Acked 5647 Write 7 Done Done New false
12026 4649 Acked 5648 Write 9 Done Done New false
12027 4650 Acked 5649 Read 10 Done Done New false
12028 4651 Acked 5650 Write 5 Done Done New false
12029 4652 Acked 5651 Write 10 Done Done New false
12030 4653 Acked 5652 Write 2 Done Done New false
12031 4654 Acked 5653 Write 5 Done Done New false
12032 4655 Acked 5654 Read 7 Done Done New false
12033 4656 Acked 5655 Read 2 Done Done New false
12034 4657 Acked 5656 Write 3 Done Done New false
12035 4658 Acked 5657 Read 3 Done Done New false
12036 4659 Acked 5658 Read 6 Done Done New false
12037 4660 Acked 5659 Read 9 Done Done New false
12038 4661 Acked 5660 Read 7 Done Done New false
12039 4662 Acked 5661 Write 6 Done Done New false
12040 4663 Acked 5662 Write 1 Done Done New false
12041 4664 Acked 5663 Read 6 Done Done New false
12042 4665 Acked 5664 Write 7 Done Done New false
12043 4666 Acked 5665 Read 4 Done Done New false
12044 4667 Acked 5666 Write 2 Done Done New false
12045 4668 Acked 5667 Read 9 Done Done New false
12046 4669 Acked 5668 Read 7 Done Done New false
12047 4670 Acked 5669 Flush 0 Done Done New false
12048 4671 Acked 5670 Flush 0 Done Done New false
12049 4672 Acked 5671 Read 9 Done Done New false
12050 4673 Acked 5672 Read 4 Done Done New false
12051 4674 Acked 5673 Write 4 Done Done New false
12052 4675 Acked 5674 Read 8 Done Done New false
12053 4676 Acked 5675 Flush 0 Done Done New false
12054 4677 Acked 5676 Flush 0 Done Done New false
12055 4678 Acked 5677 Read 9 Done Done New false
12056 4679 Acked 5678 Read 3 Done Done New false
12057 4680 Acked 5679 Write 2 Done Done New false
12058 4681 Acked 5680 Write 5 Done Done New false
12059 4682 Acked 5681 Write 6 Done Done New false
12060 4683 Acked 5682 Write 1 Done Done New false
12061 4684 Acked 5683 Write 2 Done Done New false
12062 4685 Acked 5684 Write 7 Done Done New false
12063 4686 Acked 5685 Read 10 Done Done New false
12064 4687 Acked 5686 Read 10 Done Done New false
12065 4688 Acked 5687 Flush 0 Done Done New false
12066 4689 Acked 5688 Read 1 Done Done New false
12067 4690 Acked 5689 Flush 0 Done Done New false
12068 4691 Acked 5690 Write 5 Done Done New false
12069 4692 Acked 5691 Flush 0 Done Done New false
12070 4693 Acked 5692 Flush 0 Done Done New false
12071 4694 Acked 5693 Read 7 Done Done New false
12072 4695 Acked 5694 Write 1 Done Done New false
12073 4696 Acked 5695 Flush 0 Done Done New false
12074 4697 Acked 5696 Write 8 Done Done New false
12075 4698 Acked 5697 Write 5 Done Done New false
12076 4699 Acked 5698 Read 1 Done Done New false
12077 4700 Acked 5699 Write 10 Done Done New false
12078 4701 Acked 5700 Read 3 Done Done New false
12079 4702 Acked 5701 Write 8 Done Done New false
12080 4703 Acked 5702 Write 8 Done Done New false
12081 4704 Acked 5703 Write 2 Done Done New false
12082 4705 Acked 5704 Read 4 Done Done New false
12083 4706 Acked 5705 Read 10 Done Done New false
12084 4707 Acked 5706 Read 2 Done Done New false
12085 4708 Acked 5707 Read 9 Done Done New false
12086 4709 Acked 5708 Read 6 Done Done New false
12087 4710 Acked 5709 Flush 0 Done Done New false
12088 4711 Acked 5710 Read 5 Done Done New false
12089 4712 Acked 5711 Read 5 Done Done New false
12090 4713 Acked 5712 Read 8 Done Done New false
12091 4714 Acked 5713 Write 10 Done Done New false
12092 4715 Acked 5714 Read 3 Done Done New false
12093 4716 Acked 5715 Write 2 Done Done New false
12094 4717 Acked 5716 Flush 0 Done Done New false
12095 4718 Acked 5717 Read 1 Done Done New false
12096 4719 Acked 5718 Read 4 Done Done New false
12097 4720 Acked 5719 Write 3 Done Done New false
12098 4721 Acked 5720 Read 10 Done Done New false
12099 4722 Acked 5721 Read 3 Done Done New false
12100 4723 Acked 5722 Write 3 Done Done New false
12101 4724 Acked 5723 Read 5 Done Done New false
12102 4725 Acked 5724 Read 8 Done Done New false
12103 4726 Acked 5725 Read 5 Done Done New false
12104 4727 Acked 5726 Flush 0 Done Done New false
12105 4728 Acked 5727 Write 4 Done Done New false
12106 4729 Acked 5728 Flush 0 Done Done New false
12107 4730 Acked 5729 Write 4 Done Done New false
12108 4731 Acked 5730 Read 4 Done Done New false
12109 4732 Acked 5731 Flush 0 Done Done New false
12110 4733 Acked 5732 Read 3 Done Done New false
12111 4734 Acked 5733 Flush 0 Done Done New false
12112 4735 Acked 5734 Flush 0 Done Done New false
12113 4736 Acked 5735 Write 2 Done Done New false
12114 4737 Acked 5736 Read 9 Done Done New false
12115 4738 Acked 5737 Read 10 Done Done New false
12116 4739 Acked 5738 Write 6 Done Done New false
12117 4740 Acked 5739 Write 5 Done Done New false
12118 4741 Acked 5740 Write 8 Done Done New false
12119 4742 Acked 5741 Read 3 Done Done New false
12120 4743 Acked 5742 Write 4 Done Done New false
12121 4744 Acked 5743 Read 8 Done Done New false
12122 4745 Acked 5744 Read 2 Done Done New false
12123 4746 Acked 5745 Write 8 Done Done New false
12124 4747 Acked 5746 Flush 0 Done Done New false
12125 4748 Acked 5747 Read 1 Done Done New false
12126 4749 Acked 5748 Read 10 Done Done New false
12127 4750 Acked 5749 Write 6 Done Done New false
12128 4751 Acked 5750 Write 4 Done Done New false
12129 4752 Acked 5751 Write 3 Done Done New false
12130 4753 Acked 5752 Read 4 Done Done New false
12131 4754 Acked 5753 Read 1 Done Done New false
12132 4755 Acked 5754 Flush 0 Done Done New false
12133 4756 Acked 5755 Write 8 Done Done New false
12134 4757 Acked 5756 Write 7 Done Done New false
12135 4758 Acked 5757 Read 8 Done Done New false
12136 4759 Acked 5758 Read 1 Done Done New false
12137 4760 Acked 5759 Flush 0 Done Done New false
12138 4761 Acked 5760 Flush 0 Done Done New false
12139 4762 Acked 5761 Read 4 Done Done New false
12140 4763 Acked 5762 Read 7 Done Done New false
12141 4764 Acked 5763 Read 3 Done Done New false
12142 4765 Acked 5764 Read 6 Done Done New false
12143 4766 Acked 5765 Write 3 Done Done New false
12144 4767 Acked 5766 Write 2 Done Done New false
12145 4768 Acked 5767 Read 6 Done Done New false
12146 4769 Acked 5768 Read 9 Done Done New false
12147 4770 Acked 5769 Read 5 Done Done New false
12148 4771 Acked 5770 Read 10 Done Done New false
12149 4772 Acked 5771 Read 2 Done Done New false
12150 4773 Acked 5772 Write 7 Done Done New false
12151 4774 Acked 5773 Read 6 Done Done New false
12152 4775 Acked 5774 Read 5 Done Done New false
12153 4776 Acked 5775 Write 1 Done Done New false
12154 4777 Acked 5776 Read 1 Done Done New false
12155 4778 Acked 5777 Write 9 Done Done New false
12156 4779 Acked 5778 Write 4 Done Done New false
12157 4780 Acked 5779 Read 10 Done Done New false
12158 4781 Acked 5780 Write 10 Done Done New false
12159 4782 Acked 5781 Write 3 Done Done New false
12160 4783 Acked 5782 Read 4 Done Done New false
12161 4784 Acked 5783 Write 1 Done Done New false
12162 4785 Acked 5784 Write 8 Done Done New false
12163 4786 Acked 5785 Write 9 Done Done New false
12164 4787 Acked 5786 Write 2 Done Done New false
12165 4788 Acked 5787 Write 1 Done Done New false
12166 4789 Acked 5788 Write 3 Done Done New false
12167 4790 Acked 5789 Read 3 Done Done New false
12168 4791 Acked 5790 Write 8 Done Done New false
12169 4792 Acked 5791 Write 10 Done Done New false
12170 4793 Acked 5792 Read 4 Done Done New false
12171 4794 Acked 5793 Read 2 Done Done New false
12172 4795 Acked 5794 Read 6 Done Done New false
12173 4796 Acked 5795 Flush 0 Done Done New false
12174 4797 Acked 5796 Read 8 Done Done New false
12175 4798 Acked 5797 Write 1 Done Done New false
12176 4799 Acked 5798 Read 3 Done Done New false
12177 4800 Acked 5799 Read 5 Done Done New false
12178 4801 Acked 5800 Write 2 Done Done New false
12179 4802 Acked 5801 Write 4 Done Done New false
12180 4803 Acked 5802 Read 6 Done Done New false
12181 4804 Acked 5803 Write 6 Done Done New false
12182 4805 Acked 5804 Write 3 Done Done New false
12183 4806 Acked 5805 Flush 0 Done Done New false
12184 4807 Acked 5806 Read 2 Done Done New false
12185 4808 Acked 5807 Read 7 Done Done New false
12186 4809 Acked 5808 Read 8 Done Done New false
12187 4810 Acked 5809 Write 2 Done Done New false
12188 4811 Acked 5810 Read 4 Done Done New false
12189 4812 Acked 5811 Read 2 Done Done New false
12190 4813 Acked 5812 Read 2 Done Done New false
12191 4814 Acked 5813 Read 6 Done Done New false
12192 4815 Acked 5814 Write 9 Done Done New false
12193 4816 Acked 5815 Write 5 Done Done New false
12194 4817 Acked 5816 Read 5 Done Done New false
12195 4818 Acked 5817 Write 9 Done Done New false
12196 4819 Acked 5818 Write 10 Done Done New false
12197 4820 Acked 5819 Flush 0 Done Done New false
12198 4821 Acked 5820 Write 7 Done Done New false
12199 4822 Acked 5821 Read 10 Done Done New false
12200 4823 Acked 5822 Read 5 Done Done New false
12201 4824 Acked 5823 Read 5 Done Done New false
12202 4825 Acked 5824 Write 3 Done Done New false
12203 4826 Acked 5825 Flush 0 Done Done New false
12204 4827 Acked 5826 Read 9 Done Done New false
12205 4828 Acked 5827 Read 1 Done Done New false
12206 4829 Acked 5828 Write 10 Done Done New false
12207 4830 Acked 5829 Write 6 Done Done New false
12208 4831 Acked 5830 Write 10 Done Done New false
12209 4832 Acked 5831 Write 6 Done Done New false
12210 4833 Acked 5832 Write 7 Done Done New false
12211 4834 Acked 5833 Flush 0 Done Done New false
12212 4835 Acked 5834 Read 4 Done Done New false
12213 4836 Acked 5835 Flush 0 Done Done New false
12214 4837 Acked 5836 Write 6 Done Done New false
12215 4838 Acked 5837 Read 9 Done Done New false
12216 4839 Acked 5838 Read 8 Done Done New false
12217 4840 Acked 5839 Write 9 Done Done New false
12218 4841 Acked 5840 Read 2 Done Done New false
12219 4842 Acked 5841 Read 9 Done Done New false
12220 STATES DS:0 DS:1 DS:2 TOTAL
12221 New 0 0 302 302
12222 Sent 0 0 0 0
12223 Done 302 302 0 604
12224 Skipped 0 0 0 0
12225 Error 0 0 0 0
12226 Last Flush: 5835 5835 5539
12227 Downstairs last five completed: 5539 5538 5537 5536 5535 5534
12228 Upstairs last five completed: 4842 4841 4840 4839 4838 4837
12229 CLIENT: Up:0 ds:302 act:2
12230 [8830] got start action
12231 [8830] I am going to restart
12232 Starting downstairs at port 8830
12233 Make output file at "/tmp/dsc/downstairs-8830.txt"
12234 [8830][2] reports Starting
12235 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1192)
12236 [8830][2] reports Running
122372023-10-13T18:11:44.770ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 2
122382023-10-13T18:11:44.770ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
122392023-10-13T18:11:44.770ZINFOcrucible: [2] client re-new 303 jobs since flush 5539 = downstairs
122402023-10-13T18:11:44.771ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
122412023-10-13T18:11:44.772ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
122422023-10-13T18:11:44.772ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
122432023-10-13T18:11:44.772ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
122442023-10-13T18:11:44.772ZINFOcrucible: [2] send last flush ID to this DS: 5539
122452023-10-13T18:11:44.772ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
122462023-10-13T18:11:44.772ZINFOcrucible: [2] Replied this last flush ID: 5539
122472023-10-13T18:11:44.772ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Offline ds_transition to Replay
122482023-10-13T18:11:44.772ZINFOcrucible: [2] Transition from Offline to Replay
122492023-10-13T18:11:44.772ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
122502023-10-13T18:11:44.772ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Replay ds_transition to Active
122512023-10-13T18:11:44.772ZINFOcrucible: [2] Transition from Replay to Active
122522023-10-13T18:11:44.772ZINFOcrucible: [2] Starts cmd_loop
122532023-10-13T18:11:44.772ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
122542023-10-13T18:11:44.772ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
122552023-10-13T18:11:44.772ZINFOcrucible: Set check for repair
122562023-10-13T18:11:45.773ZINFOcrucible: Checking if live repair is needed
122572023-10-13T18:11:45.773ZINFOcrucible: No Live Repair required at this time
122582023-10-13T18:11:45.773ZWARNcrucible: [2] flow control end
12259 ----------------------------------------------------------------
12260 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12261 Downstairs last five completed: 5842 5841 5840 5839 5838 5837
12262 Upstairs last five completed: 4843 4842 4841 4840 4839 4838
12263 CLIENT: Up:0 ds:0 act:3
12264 Replay: All jobs finished, all DS active.
12265 Main task has work to do, go find it
12266 got dsc Stop(1)
12267 stop 1
12268 Oct 13 18:11:46.714 INFO request completed, latency_us: 506, response_code: 204, uri: /stop/cid/1, method: POST, req_id:[8820] Got stop action so:false kr:false
12269 eaa23f57-594b-4f0c-af83-3de4ea9bd774, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
12270 Oct 13 18:11:46.716 INFO request completed, latency_us: 179, response_code: 200, uri: /state/cid/1, method: GET, req_id: d9e838b3-2a2e-4d73-8b38-c1e9ead6f155, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
122712023-10-13T18:11:46.716ZWARNcrucible: [1] downstairs disconnected
122722023-10-13T18:11:46.716ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
122732023-10-13T18:11:46.716ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8820 closed looper = 1
122742023-10-13T18:11:46.716ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
122752023-10-13T18:11:46.716ZWARNcrucible: [1] pm_task rx.recv() is None
122762023-10-13T18:11:46.716ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
122772023-10-13T18:11:46.716ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
12278 [8820] Exited with: ExitStatus(unix_wait_status(9))
12279 [8820][1] reports Exit
12280 Oct 13 18:11:50.718 INFO request completed, latency_us: 277, response_code: 200, uri: /state/cid/1, method: GET, req_id: 38ce660f-cd45-45c4-833f-044633999836, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
12281 001/300 Read block 4445 len 16384
12282 002/300 Write block 4565 len 28672 data: 2 2 3 3 4 4 4
12283 003/300 Write block 1177 len 4096 data: 5
12284 004/300 Write block 1146 len 24576 data: 3 4 5 5 3 3
12285 005/300 Read block 4389 len 40960
12286 006/300 Read block 4253 len 24576
12287 007/300 Write block 1244 len 8192 data: 3 3
12288 008/300 Flush
12289 009/300 Write block 200 len 24576 data: 2 2 2 2 2 2
12290 010/300 Read block 2262 len 4096
12291 011/300 Read block 3134 len 32768
12292 012/300 Read block 3420 len 24576
12293 013/300 Write block 3487 len 40960 data: 3 3 5 5 5 4 4 5 4 5
12294 014/300 Write block 921 len 36864 data: 4 6 5 4 4 2 2 4 5
12295 015/300 Read block 2695 len 4096
12296 016/300 Flush
12297 017/300 Write block 20 len 16384 data: 5 6 6 5
12298 018/300 Read block 1171 len 4096
12299 019/300 Flush
12300 020/300 Flush
12301 021/300 Read block 387 len 20480
12302 022/300 Write block 1441 len 8192 data: 2 5
12303 023/300 Write block 4947 len 4096 data: 5
12304 024/300 Read block 2480 len 4096
12305 025/300 Write block 4115 len 36864 data: 3 2 2 3 3 3 3 3 3
12306 026/300 Read block 2434 len 20480
12307 027/300 Read block 4233 len 8192
12308 028/300 Read block 3801 len 16384
12309 029/300 Write block 140 len 32768 data: 6 6 4 3 3 3 4 4
12310 030/300 Write block 1816 len 24576 data: 4 4 5 4 4 3
12311 031/300 Write block 2022 len 36864 data: 5 4 5 4 5 6 6 6 6
12312 032/300 Flush
12313 033/300 Write block 4760 len 36864 data: 6 7 6 5 4 3 3 3 3
12314 034/300 Read block 594 len 16384
12315 035/300 Read block 3077 len 16384
12316 036/300 Read block 4515 len 32768
12317 037/300 Flush
12318 038/300 Write block 3639 len 40960 data: 4 4 3 3 2 3 3 4 3 4
12319 039/300 Read block 2253 len 24576
12320 040/300 Write block 4219 len 8192 data: 6 6
12321 041/300 Read block 3470 len 4096
12322 042/300 Read block 1389 len 32768
12323 043/300 Write block 83 len 20480 data: 4 5 3 4 5
12324 044/300 Write block 2758 len 20480 data: 4 6 6 5 5
12325 045/300 Read block 426 len 36864
12326 046/300 Write block 1837 len 4096 data: 5
12327 047/300 Read block 276 len 32768
12328 048/300 Read block 3078 len 40960
12329 049/300 Write block 1931 len 20480 data: 5 6 6 8 6
12330 050/300 Write block 2763 len 20480 data: 5 5 5 5 5
12331 051/300 Read block 2339 len 12288
12332 052/300 Read block 1670 len 16384
12333 053/300 Read block 2570 len 16384
12334 054/300 Write block 859 len 12288 data: 2 2 2
12335 055/300 Read block 3229 len 8192
12336 056/300 Write block 3347 len 36864 data: 2 2 2 3 5 4 4 4 3
12337 057/300 Read block 1818 len 16384
12338 058/300 Write block 67 len 4096 data: 2
12339 059/300 Read block 1896 len 32768
12340 060/300 Write block 4777 len 20480 data: 5 5 6 6 6
12341 061/300 Read block 4024 len 36864
12342 062/300 Write block 308 len 28672 data: 2 2 2 3 3 4 6
12343 063/300 Write block 343 len 36864 data: 3 4 4 4 4 4 4 4 4
12344 064/300 Read block 3600 len 12288
12345 065/300 Write block 1463 len 40960 data: 4 4 4 5 4 4 3 3 2 4
12346 066/300 Read block 695 len 12288
12347 067/300 Read block 2642 len 40960
12348 068/300 Read block 1430 len 28672
12349 069/300 Read block 3799 len 12288
12350 070/300 Read block 4762 len 4096
12351 071/300 Write block 2202 len 24576 data: 5 3 3 4 4 4
12352 072/300 Read block 832 len 20480
12353 073/300 Write block 4200 len 24576 data: 5 4 4 4 3 3
12354 074/300 Read block 1951 len 16384
12355 075/300 Flush
12356 076/300 Read block 4435 len 16384
12357 077/300 Read block 2234 len 12288
12358 078/300 Write block 2954 len 12288 data: 4 4 4
12359 079/300 Read block 2668 len 16384
12360 080/300 Read block 2329 len 16384
12361 081/300 Write block 1037 len 40960 data: 4 4 3 4 6 6 6 6 5 4
12362 082/300 Write block 2615 len 20480 data: 4 4 5 3 5
12363 083/300 Read block 774 len 24576
12364 084/300 Write block 1747 len 40960 data: 3 3 2 3 3 3 4 5 5 4
12365 085/300 Write block 915 len 24576 data: 6 6 7 7 7 7
12366 086/300 Write block 1652 len 20480 data: 5 4 4 4 4
12367 087/300 Write block 4283 len 12288 data: 5 5 5
12368 088/300 Write block 2322 len 32768 data: 3 3 4 3 3 5 5 6
12369 089/300 Read block 2036 len 20480
12370 090/300 Read block 1455 len 8192
12371 091/300 Read block 1886 len 36864
12372 092/300 Write block 3043 len 28672 data: 7 6 6 4 5 4 5
12373 093/300 Write block 1780 len 4096 data: 4
12374 094/300 Flush
12375 095/300 Read block 2743 len 12288
12376 096/300 Flush
12377 097/300 Flush
12378 098/300 Write block 4013 len 28672 data: 2 3 4 3 3 3 3
12379 099/300 Write block 2876 len 28672 data: 3 3 3 3 3 3 4
12380 100/300 Flush
12381 101/300 Write block 3145 len 40960 data: 8 8 7 4 4 5 3 2 4 2
12382 102/300 Write block 1001 len 28672 data: 5 5 5 4 6 6 5
12383 103/300 Flush
12384 104/300 Write block 1541 len 12288 data: 3 3 3
12385 105/300 Write block 302 len 40960 data: 4 3 3 3 2 2 3 3 3 4
12386 106/300 Read block 2451 len 8192
12387 107/300 Flush
12388 108/300 Read block 3992 len 28672
12389 109/300 Write block 3185 len 16384 data: 3 4 4 4
12390 110/300 Read block 3867 len 32768
12391 111/300 Write block 4406 len 8192 data: 7 5
12392 112/300 Write block 3606 len 28672 data: 3 2 2 2 2 3 3
12393 113/300 Read block 4808 len 28672
12394 114/300 Read block 2364 len 36864
12395 115/300 Flush
12396 116/300 Write block 3240 len 36864 data: 5 4 2 2 2 3 4 4 4
12397 117/300 Flush
12398 118/300 Write block 2613 len 24576 data: 4 4 5 5 6 4
12399 119/300 Write block 1481 len 36864 data: 3 3 3 3 4 5 6 7 8
12400 120/300 Read block 2597 len 40960
12401 121/300 Read block 1417 len 24576
12402 122/300 Read block 3270 len 40960
12403 123/300 Read block 415 len 24576
12404 124/300 Write block 3895 len 12288 data: 3 2 3
12405 125/300 Read block 3438 len 40960
12406 126/300 Write block 445 len 32768 data: 3 5 4 4 4 4 6 6
12407 127/300 Write block 3592 len 16384 data: 4 5 4 5
12408 128/300 Read block 1946 len 12288
12409 129/300 Write block 3374 len 40960 data: 3 4 2 2 2 3 3 3 3 5
12410 130/300 Write block 1270 len 40960 data: 2 3 4 4 4 3 3 4 3 4
12411 131/300 Write block 377 len 24576 data: 3 4 4 3 3 4
12412 132/300 Write block 1960 len 28672 data: 3 3 3 3 3 3 3
12413 133/300 Read block 4072 len 4096
12414 134/300 Write block 2883 len 4096 data: 5
12415 135/300 Read block 4492 len 12288
12416 136/300 Flush
12417 137/300 Write block 3590 len 40960 data: 2 3 5 6 5 6 6 5 4 2
12418 138/300 Read block 3987 len 28672
12419 139/300 Read block 4612 len 24576
12420 140/300 Flush
12421 141/300 Read block 3883 len 8192
12422 142/300 Read block 731 len 36864
12423 143/300 Write block 1325 len 16384 data: 3 3 3 3
12424 144/300 Write block 2028 len 12288 data: 7 7 7
12425 145/300 Read block 354 len 20480
12426 146/300 Read block 1624 len 20480
12427 147/300 Write block 2432 len 28672 data: 3 2 2 3 3 4 3
12428 148/300 Read block 563 len 4096
12429 149/300 Read block 3683 len 40960
12430 150/300 Read block 391 len 20480
12431 151/300 Flush
12432 152/300 Write block 3101 len 4096 data: 4
12433 153/300 Read block 4254 len 32768
12434 154/300 Write block 3547 len 36864 data: 3 3 3 3 3 2 2 2 3
12435 155/300 Read block 1689 len 12288
12436 156/300 Write block 2169 len 16384 data: 2 3 4 4
12437 157/300 Read block 2524 len 20480
12438 158/300 Read block 2052 len 32768
12439 159/300 Flush
12440 160/300 Write block 3742 len 36864 data: 7 6 5 4 5 4 3 3 4
12441 161/300 Write block 4345 len 12288 data: 4 4 5
12442 162/300 Read block 1617 len 36864
12443 163/300 Read block 4485 len 36864
12444 164/300 Read block 3896 len 36864
12445 165/300 Read block 1638 len 20480
12446 166/300 Read block 2865 len 28672
12447 167/300 Read block 3717 len 32768
12448 168/300 Write block 2837 len 16384 data: 3 2 3 3
12449 169/300 Read block 4911 len 16384
12450 170/300 Read block 2920 len 28672
12451 171/300 Read block 1215 len 4096
12452 172/300 Read block 137 len 16384
12453 173/300 Read block 2031 len 8192
12454 174/300 Write block 3111 len 28672 data: 5 4 5 4 5 5 4
12455 175/300 Write block 279 len 32768 data: 5 5 5 6 5 5 5 5
12456 176/300 Write block 2240 len 28672 data: 6 4 6 6 6 5 4
12457 177/300 Read block 2797 len 36864
12458 178/300 Read block 2600 len 4096
12459 179/300 Write block 123 len 12288 data: 4 5 5
12460 180/300 Read block 2280 len 16384
12461 181/300 Read block 4328 len 36864
12462 182/300 Write block 3526 len 36864 data: 5 3 2 2 2 2 2 4 5
12463 183/300 Write block 1448 len 8192 data: 7 7
12464 184/300 Read block 596 len 32768
12465 185/300 Write block 4876 len 20480 data: 4 4 5 3 3
12466 186/300 Write block 1215 len 12288 data: 5 3 3
12467 187/300 Read block 429 len 40960
12468 188/300 Write block 3158 len 8192 data: 5 5
12469 189/300 Read block 4966 len 16384
12470 190/300 Read block 2465 len 40960
12471 191/300 Read block 2054 len 20480
12472 192/300 Write block 68 len 20480 data: 2 2 3 3 3
12473 193/300 Read block 4793 len 32768
12474 194/300 Read block 864 len 24576
12475 195/300 Read block 410 len 8192
12476 196/300 Read block 4747 len 20480
12477 197/300 Read block 2666 len 12288
12478 198/300 Write block 2463 len 28672 data: 5 5 5 5 4 3 4
12479 199/300 Read block 1698 len 28672
12480 200/300 Write block 1701 len 12288 data: 2 3 2
12481 201/300 Write block 3930 len 4096 data: 4
12482 202/300 Read block 4988 len 36864
12483 203/300 Flush
12484 204/300 Read block 2304 len 12288
12485 205/300 Write block 143 len 8192 data: 4 4
12486 206/300 Read block 1212 len 32768
12487 207/300 Read block 3902 len 16384
12488 208/300 Write block 2867 len 24576 data: 5 6 7 6 5 5
12489 209/300 Read block 3898 len 12288
12490 210/300 Write block 2823 len 24576 data: 3 4 4 4 4 4
12491 211/300 Write block 1847 len 8192 data: 2 3
12492 212/300 Read block 1490 len 12288
12493 213/300 Read block 3884 len 40960
12494 214/300 Write block 3935 len 40960 data: 6 4 8 8 8 7 7 7 8 8
12495 215/300 Read block 2253 len 4096
12496 216/300 Write block 3534 len 32768 data: 6 5 6 7 9 9 7 6
12497 217/300 Read block 4486 len 32768
12498 218/300 Read block 4703 len 40960
12499 219/300 Flush
12500 220/300 Write block 4825 len 32768 data: 5 5 4 4 4 3 2 2
12501 221/300 Flush
12502 222/300 Read block 1948 len 40960
12503 223/300 Flush
12504 224/300 Write block 4685 len 8192 data: 3 3
12505 225/300 Write block 94 len 4096 data: 5
12506 226/300 Write block 2346 len 4096 data: 5
12507 227/300 Read block 999 len 32768
12508 228/300 Write block 3178 len 28672 data: 5 5 4 4 4 4 3
12509 229/300 Read block 548 len 36864
12510 230/300 Read block 2825 len 8192
12511 231/300 Write block 471 len 12288 data: 3 3 3
12512 232/300 Flush
12513 233/300 Read block 3022 len 4096
12514 234/300 Write block 4737 len 20480 data: 2 2 2 4 4
12515 235/300 Read block 3953 len 16384
12516 236/300 Read block 879 len 40960
12517 237/300 Write block 4113 len 28672 data: 3 3 4 3 3 4 4
12518 238/300 Write block 1342 len 32768 data: 5 6 6 5 5 4 4 4
12519 239/300 Write block 3758 len 20480 data: 5 4 3 3 4
12520 240/300 Read block 3017 len 24576
12521 241/300 Read block 3445 len 36864
12522 242/300 Read block 4719 len 12288
12523 243/300 Flush
12524 244/300 Write block 2122 len 16384 data: 6 5 4 4
12525 245/300 Read block 518 len 36864
12526 246/300 Write block 1055 len 4096 data: 6
12527 247/300 Flush
12528 248/300 Write block 3178 len 8192 data: 6 6
12529 249/300 Write block 1695 len 12288 data: 6 6 4
12530 250/300 Write block 2310 len 36864 data: 6 5 4 4 5 5 6 5 4
12531 251/300 Write block 4447 len 8192 data: 4 4
12532 252/300 Write block 3471 len 28672 data: 3 3 3 4 4 5 5
12533 253/300 Read block 121 len 12288
12534 254/300 Flush
12535 255/300 Read block 2118 len 40960
12536 256/300 Write block 833 len 8192 data: 4 3
12537 257/300 Write block 1185 len 40960 data: 4 3 2 2 2 3 3 3 4 4
12538 258/300 Write block 1871 len 40960 data: 4 5 5 5 6 7 6 7 6 6
12539 259/300 Read block 4724 len 8192
12540 260/300 Write block 3835 len 32768 data: 3 3 3 3 4 4 4 4
12541 261/300 Read block 4324 len 8192
12542 262/300 Read block 4748 len 36864
12543 263/300 Write block 2945 len 16384 data: 5 5 5 4
12544 264/300 Write block 1715 len 24576 data: 3 3 2 2 4 5
12545 265/300 Write block 1451 len 16384 data: 5 5 6 6
12546 266/300 Write block 1459 len 24576 data: 3 3 3 3 5 5
12547 267/300 Flush
12548 268/300 Read block 3597 len 28672
12549 269/300 Read block 3951 len 32768
12550 270/300 Read block 2154 len 16384
12551 271/300 Read block 956 len 12288
12552 272/300 Read block 332 len 40960
12553 273/300 Write block 1643 len 32768 data: 4 4 3 4 4 4 4 4
12554 274/300 Read block 481 len 16384
12555 275/300 Read block 285 len 16384
12556 276/300 Read block 1817 len 28672
12557 277/300 Read block 578 len 4096
12558 278/300 Read block 3587 len 40960
12559 279/300 Read block 88 len 12288
12560 280/300 Write block 205 len 8192 data: 3 4
12561 281/300 Write block 3009 len 12288 data: 4 4 5
12562 282/300 Write block 4353 len 8192 data: 3 3
12563 283/300 Read block 1467 len 4096
12564 284/300 Read block 946 len 32768
12565 285/300 Write block 2420 len 4096 data: 4
12566 286/300 Read block 3474 len 16384
12567 287/300 Write block 4391 len 4096 data: 4
12568 288/300 Write block 2636 len 40960 data: 4 5 5 6 4 4 4 4 3 2
12569 289/300 Read block 3986 len 24576
12570 290/300 Read block 4592 len 40960
12571 291/300 Read block 2546 len 8192
12572 292/300 Read block 907 len 16384
12573 293/300 Read block 1277 len 40960
12574 294/300 Write block 2239 len 40960 data: 6 7 5 7 7 7 6 5 4 4
12575 295/300 Write block 2011 len 20480 data: 7 7 5 6 10
12576 296/300 Flush
12577 297/300 Write block 1157 len 24576 data: 5 5 5 7 6 6
12578 298/300 Flush
12579 299/300 Write block 3346 len 32768 data: 2 3 3 3 4 6 5 5
12580 300/300 Read block 1794 len 24576
12581 Main task has work to do, go find it
12582 got dsc Start(1)
12583 start 1
12584 Oct 13 18:11:51.691 INFO request completed, latency_us: 300, response_code: 204, uri: /start/cid/1, method: POST, req_id: b54ef4f8-9652-459b-8755-bf4375001135, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
12585 Replay: started 1, returned:Ok(())
12586 ----------------------------------------------------------------
12587 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
12588 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
12589 4844 Acked 5843 Read 4 Done New Done false
12590 4845 Acked 5844 Write 7 Done New Done false
12591 4846 Acked 5845 Write 1 Done New Done false
12592 4847 Acked 5846 Write 6 Done New Done false
12593 4848 Acked 5847 Read 10 Done New Done false
12594 4849 Acked 5848 Read 6 Done New Done false
12595 4850 Acked 5849 Write 2 Done New Done false
12596 4851 Acked 5850 Flush 0 Done New Done false
12597 4852 Acked 5851 Write 6 Done New Done false
12598 4853 Acked 5852 Read 1 Done New Done false
12599 4854 Acked 5853 Read 8 Done New Done false
12600 4855 Acked 5854 Read 6 Done New Done false
12601 4856 Acked 5855 Write 10 Done New Done false
12602 4857 Acked 5856 Write 9 Done New Done false
12603 4858 Acked 5857 Read 1 Done New Done false
12604 4859 Acked 5858 Flush 0 Done New Done false
12605 4860 Acked 5859 Write 4 Done New Done false
12606 4861 Acked 5860 Read 1 Done New Done false
12607 4862 Acked 5861 Flush 0 Done New Done false
12608 4863 Acked 5862 Flush 0 Done New Done false
12609 4864 Acked 5863 Read 5 Done New Done false
12610 4865 Acked 5864 Write 2 Done New Done false
12611 4866 Acked 5865 Write 1 Done New Done false
12612 4867 Acked 5866 Read 1 Done New Done false
12613 4868 Acked 5867 Write 9 Done New Done false
12614 4869 Acked 5868 Read 5 Done New Done false
12615 4870 Acked 5869 Read 2 Done New Done false
12616 4871 Acked 5870 Read 4 Done New Done false
12617 4872 Acked 5871 Write 8 Done New Done false
12618 4873 Acked 5872 Write 6 Done New Done false
12619 4874 Acked 5873 Write 9 Done New Done false
12620 4875 Acked 5874 Flush 0 Done New Done false
12621 4876 Acked 5875 Write 9 Done New Done false
12622 4877 Acked 5876 Read 4 Done New Done false
12623 4878 Acked 5877 Read 4 Done New Done false
12624 4879 Acked 5878 Read 8 Done New Done false
12625 4880 Acked 5879 Flush 0 Done New Done false
12626 4881 Acked 5880 Write 10 Done New Done false
12627 4882 Acked 5881 Read 6 Done New Done false
12628 4883 Acked 5882 Write 2 Done New Done false
12629 4884 Acked 5883 Read 1 Done New Done false
12630 4885 Acked 5884 Read 8 Done New Done false
12631 4886 Acked 5885 Write 5 Done New Done false
12632 4887 Acked 5886 Write 5 Done New Done false
12633 4888 Acked 5887 Read 9 Done New Done false
12634 4889 Acked 5888 Write 1 Done New Done false
12635 4890 Acked 5889 Read 8 Done New Done false
12636 4891 Acked 5890 Read 10 Done New Done false
12637 4892 Acked 5891 Write 5 Done New Done false
12638 4893 Acked 5892 Write 5 Done New Done false
12639 4894 Acked 5893 Read 3 Done New Done false
12640 4895 Acked 5894 Read 4 Done New Done false
12641 4896 Acked 5895 Read 4 Done New Done false
12642 4897 Acked 5896 Write 3 Done New Done false
12643 4898 Acked 5897 Read 2 Done New Done false
12644 4899 Acked 5898 Write 9 Done New Done false
12645 4900 Acked 5899 Read 4 Done New Done false
12646 4901 Acked 5900 Write 1 Done New Done false
12647 4902 Acked 5901 Read 8 Done New Done false
12648 4903 Acked 5902 Flush 0 Done New Done false
12649 4904 Acked 5903 Write 5 Done New Done false
12650 4905 Acked 5904 Read 9 Done New Done false
12651 4906 Acked 5905 Write 7 Done New Done false
12652 4907 Acked 5906 Write 9 Done New Done false
12653 4908 Acked 5907 Read 3 Done New Done false
12654 4909 Acked 5908 Write 10 Done New Done false
12655 4910 Acked 5909 Read 3 Done New Done false
12656 4911 Acked 5910 Read 10 Done New Done false
12657 4912 Acked 5911 Read 7 Done New Done false
12658 4913 Acked 5912 Read 3 Done New Done false
12659 4914 Acked 5913 Read 1 Done New Done false
12660 4915 Acked 5914 Write 6 Done New Done false
12661 4916 Acked 5915 Read 5 Done New Done false
12662 4917 Acked 5916 Write 6 Done New Done false
12663 4918 Acked 5917 Read 4 Done New Done false
12664 4919 Acked 5918 Flush 0 Done New Done false
12665 4920 Acked 5919 Read 4 Done New Done false
12666 4921 Acked 5920 Read 3 Done New Done false
12667 4922 Acked 5921 Write 3 Done New Done false
12668 4923 Acked 5922 Read 4 Done New Done false
12669 4924 Acked 5923 Read 4 Done New Done false
12670 4925 Acked 5924 Write 10 Done New Done false
12671 4926 Acked 5925 Write 5 Done New Done false
12672 4927 Acked 5926 Read 6 Done New Done false
12673 4928 Acked 5927 Write 10 Done New Done false
12674 4929 Acked 5928 Write 6 Done New Done false
12675 4930 Acked 5929 Write 5 Done New Done false
12676 4931 Acked 5930 Write 3 Done New Done false
12677 4932 Acked 5931 Write 8 Done New Done false
12678 4933 Acked 5932 Read 5 Done New Done false
12679 4934 Acked 5933 Read 2 Done New Done false
12680 4935 Acked 5934 Read 9 Done New Done false
12681 4936 Acked 5935 Write 7 Done New Done false
12682 4937 Acked 5936 Write 1 Done New Done false
12683 4938 Acked 5937 Flush 0 Done New Done false
12684 4939 Acked 5938 Read 3 Done New Done false
12685 4940 Acked 5939 Flush 0 Done New Done false
12686 4941 Acked 5940 Flush 0 Done New Done false
12687 4942 Acked 5941 Write 7 Done New Done false
12688 4943 Acked 5942 Write 7 Done New Done false
12689 4944 Acked 5943 Flush 0 Done New Done false
12690 4945 Acked 5944 Write 10 Done New Done false
12691 4946 Acked 5945 Write 7 Done New Done false
12692 4947 Acked 5946 Flush 0 Done New Done false
12693 4948 Acked 5947 Write 3 Done New Done false
12694 4949 Acked 5948 Write 10 Done New Done false
12695 4950 Acked 5949 Read 2 Done New Done false
12696 4951 Acked 5950 Flush 0 Done New Done false
12697 4952 Acked 5951 Read 7 Done New Done false
12698 4953 Acked 5952 Write 4 Done New Done false
12699 4954 Acked 5953 Read 8 Done New Done false
12700 4955 Acked 5954 Write 2 Done New Done false
12701 4956 Acked 5955 Write 7 Done New Done false
12702 4957 Acked 5956 Read 7 Done New Done false
12703 4958 Acked 5957 Read 9 Done New Done false
12704 4959 Acked 5958 Flush 0 Done New Done false
12705 4960 Acked 5959 Write 9 Done New Done false
12706 4961 Acked 5960 Flush 0 Done New Done false
12707 4962 Acked 5961 Write 6 Done New Done false
12708 4963 Acked 5962 Write 9 Done New Done false
12709 4964 Acked 5963 Read 10 Done New Done false
12710 4965 Acked 5964 Read 6 Done New Done false
12711 4966 Acked 5965 Read 10 Done New Done false
12712 4967 Acked 5966 Read 6 Done New Done false
12713 4968 Acked 5967 Write 3 Done New Done false
12714 4969 Acked 5968 Read 10 Done New Done false
12715 4970 Acked 5969 Write 8 Done New Done false
12716 4971 Acked 5970 Write 4 Done New Done false
12717 4972 Acked 5971 Read 3 Done New Done false
12718 4973 Acked 5972 Write 10 Done New Done false
12719 4974 Acked 5973 Write 10 Done New Done false
12720 4975 Acked 5974 Write 6 Done New Done false
12721 4976 Acked 5975 Write 7 Done New Done false
12722 4977 Acked 5976 Read 1 Done New Done false
12723 4978 Acked 5977 Write 1 Done New Done false
12724 4979 Acked 5978 Read 3 Done New Done false
12725 4980 Acked 5979 Flush 0 Done New Done false
12726 4981 Acked 5980 Write 10 Done New Done false
12727 4982 Acked 5981 Read 7 Done New Done false
12728 4983 Acked 5982 Read 6 Done New Done false
12729 4984 Acked 5983 Flush 0 Done New Done false
12730 4985 Acked 5984 Read 2 Done New Done false
12731 4986 Acked 5985 Read 9 Done New Done false
12732 4987 Acked 5986 Write 4 Done New Done false
12733 4988 Acked 5987 Write 3 Done New Done false
12734 4989 Acked 5988 Read 5 Done New Done false
12735 4990 Acked 5989 Read 5 Done New Done false
12736 4991 Acked 5990 Write 7 Done New Done false
12737 4992 Acked 5991 Read 1 Done New Done false
12738 4993 Acked 5992 Read 10 Done New Done false
12739 4994 Acked 5993 Read 5 Done New Done false
12740 4995 Acked 5994 Flush 0 Done New Done false
12741 4996 Acked 5995 Write 1 Done New Done false
12742 4997 Acked 5996 Read 8 Done New Done false
12743 4998 Acked 5997 Write 9 Done New Done false
12744 4999 Acked 5998 Read 3 Done New Done false
12745 5000 Acked 5999 Write 4 Done New Done false
12746 5001 Acked 6000 Read 5 Done New Done false
12747 5002 Acked 6001 Read 8 Done New Done false
12748 5003 Acked 6002 Flush 0 Done New Done false
12749 5004 Acked 6003 Write 9 Done New Done false
12750 5005 Acked 6004 Write 3 Done New Done false
12751 5006 Acked 6005 Read 9 Done New Done false
12752 5007 Acked 6006 Read 9 Done New Done false
12753 5008 Acked 6007 Read 9 Done New Done false
12754 5009 Acked 6008 Read 5 Done New Done false
12755 5010 Acked 6009 Read 7 Done New Done false
12756 5011 Acked 6010 Read 8 Done New Done false
12757 5012 Acked 6011 Write 4 Done New Done false
12758 5013 Acked 6012 Read 4 Done New Done false
12759 5014 Acked 6013 Read 7 Done New Done false
12760 5015 Acked 6014 Read 1 Done New Done false
12761 5016 Acked 6015 Read 4 Done New Done false
12762 5017 Acked 6016 Read 2 Done New Done false
12763 5018 Acked 6017 Write 7 Done New Done false
12764 5019 Acked 6018 Write 8 Done New Done false
12765 5020 Acked 6019 Write 7 Done New Done false
12766 5021 Acked 6020 Read 9 Done New Done false
12767 5022 Acked 6021 Read 1 Done New Done false
12768 5023 Acked 6022 Write 3 Done New Done false
12769 5024 Acked 6023 Read 4 Done New Done false
12770 5025 Acked 6024 Read 9 Done New Done false
12771 5026 Acked 6025 Write 9 Done New Done false
12772 5027 Acked 6026 Write 2 Done New Done false
12773 5028 Acked 6027 Read 8 Done New Done false
12774 5029 Acked 6028 Write 5 Done New Done false
12775 5030 Acked 6029 Write 3 Done New Done false
12776 5031 Acked 6030 Read 10 Done New Done false
12777 5032 Acked 6031 Write 2 Done New Done false
12778 5033 Acked 6032 Read 4 Done New Done false
12779 5034 Acked 6033 Read 10 Done New Done false
12780 5035 Acked 6034 Read 5 Done New Done false
12781 5036 Acked 6035 Write 5 Done New Done false
12782 5037 Acked 6036 Read 8 Done New Done false
12783 5038 Acked 6037 Read 6 Done New Done false
12784 5039 Acked 6038 Read 2 Done New Done false
12785 5040 Acked 6039 Read 5 Done New Done false
12786 5041 Acked 6040 Read 3 Done New Done false
12787 5042 Acked 6041 Write 7 Done New Done false
12788 5043 Acked 6042 Read 7 Done New Done false
12789 5044 Acked 6043 Write 3 Done New Done false
12790 5045 Acked 6044 Write 1 Done New Done false
12791 5046 Acked 6045 Read 9 Done New Done false
12792 5047 Acked 6046 Flush 0 Done New Done false
12793 5048 Acked 6047 Read 3 Done New Done false
12794 5049 Acked 6048 Write 2 Done New Done false
12795 5050 Acked 6049 Read 8 Done New Done false
12796 5051 Acked 6050 Read 4 Done New Done false
12797 5052 Acked 6051 Write 6 Done New Done false
12798 5053 Acked 6052 Read 3 Done New Done false
12799 5054 Acked 6053 Write 6 Done New Done false
12800 5055 Acked 6054 Flush 0 Done New Done false
12801 5056 Acked 6055 Write 2 Done New Done false
12802 5057 Acked 6056 Read 3 Done New Done false
12803 5058 Acked 6057 Read 10 Done New Done false
12804 5059 Acked 6058 Write 10 Done New Done false
12805 5060 Acked 6059 Read 1 Done New Done false
12806 5061 Acked 6060 Write 8 Done New Done false
12807 5062 Acked 6061 Read 8 Done New Done false
12808 5063 Acked 6062 Read 10 Done New Done false
12809 5064 Acked 6063 Flush 0 Done New Done false
12810 5065 Acked 6064 Write 8 Done New Done false
12811 5066 Acked 6065 Flush 0 Done New Done false
12812 5067 Acked 6066 Read 10 Done New Done false
12813 5068 Acked 6067 Flush 0 Done New Done false
12814 5069 Acked 6068 Write 2 Done New Done false
12815 5070 Acked 6069 Write 1 Done New Done false
12816 5071 Acked 6070 Write 1 Done New Done false
12817 5072 Acked 6071 Read 8 Done New Done false
12818 5073 Acked 6072 Write 7 Done New Done false
12819 5074 Acked 6073 Read 9 Done New Done false
12820 5075 Acked 6074 Read 2 Done New Done false
12821 5076 Acked 6075 Write 3 Done New Done false
12822 5077 Acked 6076 Flush 0 Done New Done false
12823 5078 Acked 6077 Read 1 Done New Done false
12824 5079 Acked 6078 Write 5 Done New Done false
12825 5080 Acked 6079 Read 4 Done New Done false
12826 5081 Acked 6080 Read 10 Done New Done false
12827 5082 Acked 6081 Write 7 Done New Done false
12828 5083 Acked 6082 Write 8 Done New Done false
12829 5084 Acked 6083 Write 5 Done New Done false
12830 5085 Acked 6084 Read 6 Done New Done false
12831 5086 Acked 6085 Read 9 Done New Done false
12832 5087 Acked 6086 Read 3 Done New Done false
12833 5088 Acked 6087 Flush 0 Done New Done false
12834 5089 Acked 6088 Write 4 Done New Done false
12835 5090 Acked 6089 Read 9 Done New Done false
12836 5091 Acked 6090 Write 1 Done New Done false
12837 5092 Acked 6091 Flush 0 Done New Done false
12838 5093 Acked 6092 Write 2 Done New Done false
12839 5094 Acked 6093 Write 3 Done New Done false
12840 5095 Acked 6094 Write 9 Done New Done false
12841 5096 Acked 6095 Write 2 Done New Done false
12842 5097 Acked 6096 Write 7 Done New Done false
12843 5098 Acked 6097 Read 3 Done New Done false
12844 5099 Acked 6098 Flush 0 Done New Done false
12845 5100 Acked 6099 Read 10 Done New Done false
12846 5101 Acked 6100 Write 2 Done New Done false
12847 5102 Acked 6101 Write 10 Done New Done false
12848 5103 Acked 6102 Write 10 Done New Done false
12849 5104 Acked 6103 Read 2 Done New Done false
12850 5105 Acked 6104 Write 8 Done New Done false
12851 5106 Acked 6105 Read 2 Done New Done false
12852 5107 Acked 6106 Read 9 Done New Done false
12853 5108 Acked 6107 Write 4 Done New Done false
12854 5109 Acked 6108 Write 6 Done New Done false
12855 5110 Acked 6109 Write 4 Done New Done false
12856 5111 Acked 6110 Write 6 Done New Done false
12857 5112 Acked 6111 Flush 0 Done New Done false
12858 5113 Acked 6112 Read 7 Done New Done false
12859 5114 Acked 6113 Read 8 Done New Done false
12860 5115 Acked 6114 Read 4 Done New Done false
12861 5116 Acked 6115 Read 3 Done New Done false
12862 5117 Acked 6116 Read 10 Done New Done false
12863 5118 Acked 6117 Write 8 Done New Done false
12864 5119 Acked 6118 Read 4 Done New Done false
12865 5120 Acked 6119 Read 4 Done New Done false
12866 5121 Acked 6120 Read 7 Done New Done false
12867 5122 Acked 6121 Read 1 Done New Done false
12868 5123 Acked 6122 Read 10 Done New Done false
12869 5124 Acked 6123 Read 3 Done New Done false
12870 5125 Acked 6124 Write 2 Done New Done false
12871 5126 Acked 6125 Write 3 Done New Done false
12872 5127 Acked 6126 Write 2 Done New Done false
12873 5128 Acked 6127 Read 1 Done New Done false
12874 5129 Acked 6128 Read 8 Done New Done false
12875 5130 Acked 6129 Write 1 Done New Done false
12876 5131 Acked 6130 Read 4 Done New Done false
12877 5132 Acked 6131 Write 1 Done New Done false
12878 5133 Acked 6132 Write 10 Done New Done false
12879 5134 Acked 6133 Read 6 Done New Done false
12880 5135 Acked 6134 Read 10 Done New Done false
12881 5136 Acked 6135 Read 2 Done New Done false
12882 5137 Acked 6136 Read 4 Done New Done false
12883 5138 Acked 6137 Read 10 Done New Done false
12884 5139 Acked 6138 Write 10 Done New Done false
12885 5140 Acked 6139 Write 5 Done New Done false
12886 5141 Acked 6140 Flush 0 Done New Done false
12887 5142 Acked 6141 Write 6 Done New Done false
12888 5143 Acked 6142 Flush 0 Done New Done false
12889 5144 Acked 6143 Write 8 Done New Done false
12890 5145 Acked 6144 Read 6 Done New Done false
12891 STATES DS:0 DS:1 DS:2 TOTAL
12892 New 0 302 0 302
12893 Sent 0 0 0 0
12894 Done 302 0 302 604
12895 Skipped 0 0 0 0
12896 Error 0 0 0 0
12897 Last Flush: 6142 5842 6142
12898 Downstairs last five completed: 5842 5841 5840 5839 5838 5837
12899 Upstairs last five completed: 5145 5144 5143 5142 5141 5140
12900 CLIENT: Up:0 ds:302 act:2
129012023-10-13T18:11:52.722ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
12902 [8820] got start action
12903 [8820] I am going to restart
12904 Starting downstairs at port 8820
12905 Make output file at "/tmp/dsc/downstairs-8820.txt"
12906 [8820][1] reports Starting
12907 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1193)
12908 [8820][1] reports Running
129092023-10-13T18:11:53.723ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 1
129102023-10-13T18:11:53.723ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
129112023-10-13T18:11:53.723ZINFOcrucible: [1] client re-new 303 jobs since flush 5842 = downstairs
129122023-10-13T18:11:53.724ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
129132023-10-13T18:11:53.725ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
129142023-10-13T18:11:53.725ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
129152023-10-13T18:11:53.725ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
129162023-10-13T18:11:53.725ZINFOcrucible: [1] send last flush ID to this DS: 5842
129172023-10-13T18:11:53.725ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
129182023-10-13T18:11:53.725ZINFOcrucible: [1] Replied this last flush ID: 5842
129192023-10-13T18:11:53.725ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Offline Active ds_transition to Replay
129202023-10-13T18:11:53.725ZINFOcrucible: [1] Transition from Offline to Replay
129212023-10-13T18:11:53.725ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
129222023-10-13T18:11:53.725ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Replay Active ds_transition to Active
129232023-10-13T18:11:53.725ZINFOcrucible: [1] Transition from Replay to Active
129242023-10-13T18:11:53.725ZINFOcrucible: [1] Starts cmd_loop
129252023-10-13T18:11:53.725ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
129262023-10-13T18:11:53.726ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
129272023-10-13T18:11:53.726ZINFOcrucible: Set check for repair
129282023-10-13T18:11:54.726ZWARNcrucible: [1] flow control end
129292023-10-13T18:11:54.727ZINFOcrucible: Checking if live repair is needed
129302023-10-13T18:11:54.727ZINFOcrucible: No Live Repair required at this time
12931 ----------------------------------------------------------------
12932 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12933 Downstairs last five completed: 6145 6144 6143 6142 6141 6140
12934 Upstairs last five completed: 5146 5145 5144 5143 5142 5141
12935 CLIENT: Up:0 ds:0 act:3
12936 Replay: All jobs finished, all DS active.
12937 Main task has work to do, go find it
12938 got dsc Stop(2)
12939 stop 2
12940 Oct 13 18:11:55.695 INFO request completed, latency_us: 500, response_code: 204, uri: /stop/cid/2, method: POST, req_id: aa54c420-a960-4857-8040-1c87c9a10a4f, remote_addr: 127.0.0.1:64571, local_addr:[8830] Got stop action so:false kr:false
12941 127.0.0.1:9998
12942 Oct 13 18:11:55.697 INFO request completed, latency_us: 192, response_code: 200, uri: /state/cid/2, method: GET, req_id:{"msg":"[2] downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2023-10-13T18:11:55.69744832Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1176}
12943 b9c620c6-7f95-4e62-8323-0ac575c19661, remote_addr: 127.0.0.1:64571, local_addr: 127.{"msg":"[2] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2023-10-13T18:11:55.697571805Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1176}
12944 0.0.1:9998
129452023-10-13T18:11:55.697ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8830 closed looper = 2
129462023-10-13T18:11:55.697ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
129472023-10-13T18:11:55.697ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
129482023-10-13T18:11:55.697ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
129492023-10-13T18:11:55.697ZWARNcrucible: [2] pm_task rx.recv() is None
12950 [8830] Exited with: ExitStatus(unix_wait_status(9))
12951 [8830][2] reports Exit
129522023-10-13T18:11:58.700ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
12953 Oct 13 18:11:59.700 INFO request completed, latency_us: 267, response_code: 200, uri: /state/cid/2, method: GET, req_id: d083d6f2-da45-4b9d-96a9-9e520f5651c4, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
12954 001/300 Read block 528 len 32768
12955 002/300 Read block 1340 len 16384
12956 003/300 Read block 1987 len 40960
12957 004/300 Read block 152 len 24576
12958 005/300 Flush
12959 006/300 Read block 1399 len 20480
12960 007/300 Write block 1672 len 16384 data: 5 3 3 3
12961 008/300 Read block 4498 len 16384
12962 009/300 Read block 712 len 12288
12963 010/300 Read block 3988 len 32768
12964 011/300 Flush
12965 012/300 Flush
12966 013/300 Flush
12967 014/300 Read block 3847 len 40960
12968 015/300 Read block 2005 len 40960
12969 016/300 Write block 3981 len 12288 data: 3 3 4
12970 017/300 Read block 2760 len 40960
12971 018/300 Write block 1353 len 8192 data: 3 4
12972 019/300 Write block 2732 len 24576 data: 4 6 5 5 5 4
12973 020/300 Read block 4845 len 12288
12974 021/300 Write block 561 len 28672 data: 4 4 4 4 3 3 2
12975 022/300 Write block 4264 len 16384 data: 4 3 3 3
12976 023/300 Read block 285 len 28672
12977 024/300 Write block 4784 len 40960 data: 4 3 4 6 6 7 4 5 5 5
12978 025/300 Read block 1039 len 24576
12979 026/300 Read block 1663 len 32768
12980 027/300 Read block 1336 len 8192
12981 028/300 Write block 4012 len 20480 data: 2 3 4 5 4
12982 029/300 Write block 1769 len 12288 data: 3 3 3
12983 030/300 Read block 4233 len 32768
12984 031/300 Write block 3304 len 12288 data: 4 4 5
12985 032/300 Read block 4489 len 24576
12986 033/300 Write block 3541 len 32768 data: 7 5 3 3 4 3 4 4
12987 034/300 Read block 595 len 28672
12988 035/300 Flush
12989 036/300 Read block 2171 len 36864
12990 037/300 Read block 4394 len 8192
12991 038/300 Read block 4522 len 32768
12992 039/300 Flush
12993 040/300 Read block 4683 len 4096
12994 041/300 Write block 1445 len 12288 data: 4 4 4
12995 042/300 Flush
12996 043/300 Read block 2405 len 16384
12997 044/300 Read block 2815 len 4096
12998 045/300 Read block 1712 len 24576
12999 046/300 Read block 4604 len 20480
13000 047/300 Write block 3455 len 36864 data: 5 5 4 5 7 8 6 6 5
13001 048/300 Read block 289 len 32768
13002 049/300 Write block 1094 len 16384 data: 3 4 6 7
13003 050/300 Write block 826 len 24576 data: 4 5 5 4 5 3
13004 051/300 Flush
13005 052/300 Read block 510 len 28672
13006 053/300 Write block 3675 len 12288 data: 3 3 3
13007 054/300 Write block 3884 len 36864 data: 2 2 2 2 2 2 2 2 3
13008 055/300 Write block 4158 len 32768 data: 5 3 5 5 5 5 7 7
13009 056/300 Read block 2181 len 28672
13010 057/300 Read block 1163 len 4096
13011 058/300 Read block 4929 len 40960
13012 059/300 Write block 446 len 16384 data: 6 5 5 5
13013 060/300 Read block 556 len 32768
13014 061/300 Write block 2033 len 8192 data: 4 3
13015 062/300 Read block 2890 len 12288
13016 063/300 Read block 3526 len 4096
13017 064/300 Write block 3510 len 28672 data: 5 5 4 3 5 5 5
13018 065/300 Read block 4126 len 32768
13019 066/300 Flush
13020 067/300 Read block 1812 len 8192
13021 068/300 Read block 3008 len 4096
13022 069/300 Write block 2143 len 12288 data: 5 6 5
13023 070/300 Write block 64 len 24576 data: 2 2 2 3 3 3
13024 071/300 Write block 4395 len 8192 data: 6 6
13025 072/300 Write block 3976 len 4096 data: 2
13026 073/300 Write block 3900 len 32768 data: 2 3 4 4 4 4 6 6
13027 074/300 Write block 1068 len 20480 data: 6 7 7 6 5
13028 075/300 Write block 2146 len 36864 data: 5 4 3 3 4 4 3 4 4
13029 076/300 Read block 3935 len 28672
13030 077/300 Read block 157 len 4096
13031 078/300 Read block 829 len 24576
13032 079/300 Read block 2190 len 32768
13033 080/300 Read block 3792 len 4096
13034 081/300 Write block 2854 len 4096 data: 2
13035 082/300 Read block 2917 len 4096
13036 083/300 Write block 332 len 24576 data: 4 4 4 4 5 4
13037 084/300 Read block 3015 len 40960
13038 085/300 Flush
13039 086/300 Write block 1715 len 40960 data: 4 4 3 3 5 6 5 4 5 4
13040 087/300 Write block 4281 len 8192 data: 4 4
13041 088/300 Read block 637 len 16384
13042 089/300 Read block 3087 len 36864
13043 090/300 Read block 2846 len 32768
13044 091/300 Write block 1656 len 4096 data: 5
13045 092/300 Write block 1940 len 12288 data: 5 4 2
13046 093/300 Write block 378 len 40960 data: 5 5 4 4 5 4 4 3 3 3
13047 094/300 Write block 2990 len 8192 data: 4 3
13048 095/300 Read block 4285 len 4096
13049 096/300 Read block 1654 len 8192
13050 097/300 Read block 4722 len 8192
13051 098/300 Read block 3815 len 12288
13052 099/300 Read block 1950 len 32768
13053 100/300 Write block 149 len 8192 data: 4 5
13054 101/300 Write block 1366 len 12288 data: 5 5 6
13055 102/300 Read block 4110 len 24576
13056 103/300 Read block 1540 len 28672
13057 104/300 Read block 71 len 32768
13058 105/300 Read block 2706 len 28672
13059 106/300 Write block 3993 len 24576 data: 4 4 4 3 4 3
13060 107/300 Read block 26 len 28672
13061 108/300 Read block 3053 len 28672
13062 109/300 Read block 3107 len 24576
13063 110/300 Read block 4228 len 40960
13064 111/300 Write block 1341 len 16384 data: 5 6 7 7
13065 112/300 Read block 755 len 36864
13066 113/300 Flush
13067 114/300 Write block 2794 len 4096 data: 3
13068 115/300 Read block 1745 len 20480
13069 116/300 Flush
13070 117/300 Write block 2348 len 40960 data: 5 5 7 7 7 5 6 6 6 6
13071 118/300 Write block 4171 len 28672 data: 7 6 4 3 3 5 4
13072 119/300 Write block 228 len 20480 data: 4 4 3 2 2
13073 120/300 Write block 2168 len 16384 data: 3 3 4 5
13074 121/300 Write block 102 len 40960 data: 6 5 6 5 4 3 4 4 4 4
13075 122/300 Write block 1652 len 4096 data: 6
13076 123/300 Read block 148 len 24576
13077 124/300 Write block 2556 len 28672 data: 9 8 8 7 7 8 7
13078 125/300 Write block 4334 len 36864 data: 8 8 5 5 4 4 3 4 4
13079 126/300 Read block 3507 len 40960
13080 127/300 Flush
13081 128/300 Read block 4485 len 36864
13082 129/300 Flush
13083 130/300 Flush
13084 131/300 Read block 184 len 40960
13085 132/300 Write block 2277 len 12288 data: 5 4 5
13086 133/300 Read block 3126 len 32768
13087 134/300 Write block 2368 len 8192 data: 5 5
13088 135/300 Write block 4476 len 24576 data: 2 2 2 2 2 3
13089 136/300 Read block 4492 len 16384
13090 137/300 Read block 4593 len 8192
13091 138/300 Read block 955 len 36864
13092 139/300 Read block 2754 len 36864
13093 140/300 Write block 2859 len 12288 data: 3 3 2
13094 141/300 Read block 2031 len 36864
13095 142/300 Read block 707 len 32768
13096 143/300 Read block 1431 len 20480
13097 144/300 Read block 214 len 16384
13098 145/300 Read block 889 len 16384
13099 146/300 Write block 507 len 12288 data: 3 3 3
13100 147/300 Write block 3616 len 20480 data: 3 5 6 7 8
13101 148/300 Write block 4771 len 4096 data: 2
13102 149/300 Read block 170 len 24576
13103 150/300 Write block 2712 len 32768 data: 2 2 2 3 3 3 4 5
13104 151/300 Write block 3315 len 40960 data: 3 3 3 3 3 3 3 3 2 3
13105 152/300 Write block 4280 len 20480 data: 5 5 5 6 6
13106 153/300 Flush
13107 154/300 Write block 1366 len 36864 data: 6 6 7 5 6 6 5 5 5
13108 155/300 Flush
13109 156/300 Write block 3997 len 4096 data: 5
13110 157/300 Read block 3866 len 24576
13111 158/300 Write block 3977 len 32768 data: 2 2 2 3 4 4 5 4
13112 159/300 Read block 72 len 8192
13113 160/300 Read block 1815 len 8192
13114 161/300 Read block 4687 len 28672
13115 162/300 Write block 904 len 16384 data: 3 4 4 4
13116 163/300 Write block 3287 len 36864 data: 4 4 3 3 3 3 4 4 3
13117 164/300 Flush
13118 165/300 Write block 642 len 24576 data: 5 5 5 5 4 3
13119 166/300 Flush
13120 167/300 Flush
13121 168/300 Write block 1534 len 8192 data: 3 2
13122 169/300 Write block 590 len 4096 data: 4
13123 170/300 Write block 4173 len 32768 data: 5 4 4 6 5 4 3 5
13124 171/300 Write block 1575 len 12288 data: 4 3 3
13125 172/300 Read block 4977 len 20480
13126 173/300 Read block 2717 len 32768
13127 174/300 Write block 472 len 40960 data: 4 4 2 2 4 4 3 3 3 3
13128 175/300 Write block 4322 len 12288 data: 6 5 9
13129 176/300 Read block 818 len 8192
13130 177/300 Read block 347 len 20480
13131 178/300 Write block 1054 len 4096 data: 5
13132 179/300 Read block 1719 len 12288
13133 180/300 Flush
13134 181/300 Write block 4799 len 32768 data: 3 6 6 6 8 8 7 6
13135 182/300 Read block 1362 len 28672
13136 183/300 Read block 4301 len 32768
13137 184/300 Read block 2614 len 24576
13138 185/300 Write block 3816 len 4096 data: 3
13139 186/300 Read block 828 len 16384
13140 187/300 Write block 2942 len 28672 data: 4 5 5 6 6 6 5
13141 188/300 Read block 4090 len 4096
13142 189/300 Write block 3608 len 28672 data: 3 3 3 4 4 3 3
13143 190/300 Write block 4992 len 20480 data: 2 3 3 2 3
13144 191/300 Write block 1170 len 32768 data: 5 5 6 6 5 5 5 6
13145 192/300 Flush
13146 193/300 Read block 2481 len 4096
13147 194/300 Read block 2560 len 16384
13148 195/300 Write block 1425 len 24576 data: 3 3 3 3 4 5
13149 196/300 Read block 4172 len 16384
13150 197/300 Flush
13151 198/300 Read block 1680 len 16384
13152 199/300 Write block 2960 len 16384 data: 6 6 5 5
13153 200/300 Flush
13154 201/300 Write block 885 len 28672 data: 4 5 5 6 5 5 5
13155 202/300 Read block 3581 len 28672
13156 203/300 Read block 3431 len 36864
13157 204/300 Write block 2850 len 28672 data: 3 3 3 2 3 3 3
13158 205/300 Read block 3057 len 8192
13159 206/300 Write block 1730 len 20480 data: 4 4 5 4 4
13160 207/300 Read block 788 len 16384
13161 208/300 Read block 3944 len 32768
13162 209/300 Write block 752 len 8192 data: 3 4
13163 210/300 Read block 3993 len 28672
13164 211/300 Write block 4948 len 20480 data: 4 2 2 2 3
13165 212/300 Read block 4393 len 28672
13166 213/300 Read block 2476 len 28672
13167 214/300 Read block 504 len 24576
13168 215/300 Read block 4838 len 4096
13169 216/300 Read block 2627 len 20480
13170 217/300 Read block 815 len 12288
13171 218/300 Write block 4947 len 12288 data: 6 5 3
13172 219/300 Read block 3300 len 32768
13173 220/300 Write block 867 len 28672 data: 3 3 3 3 3 4 3
13174 221/300 Write block 343 len 20480 data: 4 5 5 5 5
13175 222/300 Flush
13176 223/300 Read block 2896 len 12288
13177 224/300 Write block 1487 len 20480 data: 7 8 9 8 8
13178 225/300 Write block 4354 len 20480 data: 4 2 2 2 2
13179 226/300 Write block 4655 len 28672 data: 3 3 2 2 2 2 3
13180 227/300 Write block 4383 len 20480 data: 3 6 5 5 6
13181 228/300 Write block 1057 len 28672 data: 5 6 6 5 4 4 5
13182 229/300 Read block 2477 len 16384
13183 230/300 Write block 25 len 32768 data: 4 3 3 3 3 3 3 4
13184 231/300 Read block 4192 len 20480
13185 232/300 Write block 1501 len 16384 data: 5 4 4 3
13186 233/300 Write block 3776 len 8192 data: 5 5
13187 234/300 Read block 4380 len 16384
13188 235/300 Flush
13189 236/300 Read block 797 len 36864
13190 237/300 Write block 140 len 24576 data: 7 7 5 5 5 4
13191 238/300 Write block 2914 len 40960 data: 2 2 3 3 4 3 4 4 3 3
13192 239/300 Write block 317 len 8192 data: 5 7
13193 240/300 Write block 2889 len 40960 data: 4 4 5 6 3 3 2 2 3 3
13194 241/300 Read block 2090 len 16384
13195 242/300 Write block 3065 len 40960 data: 2 2 3 3 3 4 4 4 4 3
13196 243/300 Write block 3891 len 36864 data: 3 4 2 3 4 3 4 2 2
13197 244/300 Read block 1692 len 36864
13198 245/300 Write block 106 len 12288 data: 5 4 5
13199 246/300 Flush
13200 247/300 Write block 1651 len 4096 data: 6
13201 248/300 Flush
13202 249/300 Read block 658 len 36864
13203 250/300 Write block 185 len 28672 data: 4 5 5 7 4 5 5
13204 251/300 Read block 4583 len 24576
13205 252/300 Flush
13206 253/300 Read block 2216 len 4096
13207 254/300 Read block 1276 len 20480
13208 255/300 Write block 4489 len 20480 data: 5 4 3 3 2
13209 256/300 Read block 3533 len 32768
13210 257/300 Write block 2227 len 4096 data: 2
13211 258/300 Read block 2204 len 40960
13212 259/300 Write block 2278 len 36864 data: 5 6 7 6 6 4 3 3 3
13213 260/300 Read block 4829 len 24576
13214 261/300 Read block 3071 len 40960
13215 262/300 Read block 263 len 4096
13216 263/300 Write block 1469 len 8192 data: 4 4
13217 264/300 Read block 611 len 4096
13218 265/300 Read block 4632 len 32768
13219 266/300 Read block 2952 len 24576
13220 267/300 Write block 4497 len 32768 data: 3 5 5 5 5 4 5 5
13221 268/300 Read block 3174 len 24576
13222 269/300 Read block 4295 len 16384
13223 270/300 Read block 732 len 8192
13224 271/300 Read block 3197 len 24576
13225 272/300 Write block 210 len 28672 data: 5 5 6 5 5 5 5
13226 273/300 Write block 851 len 32768 data: 5 6 4 4 3 2 2 2
13227 274/300 Write block 799 len 36864 data: 5 4 4 4 4 4 3 4 4
13228 275/300 Read block 990 len 16384
13229 276/300 Read block 3238 len 40960
13230 277/300 Read block 1289 len 28672
13231 278/300 Read block 4024 len 8192
13232 279/300 Read block 2629 len 40960
13233 280/300 Write block 3348 len 20480 data: 4 4 5 7 6
13234 281/300 Write block 2695 len 8192 data: 4 6
13235 282/300 Write block 1265 len 20480 data: 2 2 2 2 2
13236 283/300 Read block 304 len 32768
13237 284/300 Read block 3207 len 4096
13238 285/300 Write block 1661 len 36864 data: 4 5 6 6 5 5 4 6 5
13239 286/300 Read block 4739 len 8192
13240 287/300 Write block 3302 len 8192 data: 4 4
13241 288/300 Read block 1570 len 32768
13242 289/300 Read block 2263 len 24576
13243 290/300 Read block 3812 len 32768
13244 291/300 Read block 3790 len 20480
13245 292/300 Read block 2101 len 4096
13246 293/300 Write block 3497 len 32768 data: 4 5 5 5 5 5 4 3
13247 294/300 Read block 733 len 16384
13248 295/300 Write block 1562 len 24576 data: 5 5 3 4 4 4
13249 296/300 Read block 2045 len 32768
13250 297/300 Flush
13251 298/300 Write block 2194 len 12288 data: 6 6 6
13252 299/300 Read block 3205 len 4096
13253 300/300 Write block 2721 len 28672 data: 5 3 2 2 2 2 2
13254 Main task has work to do, go find it
13255 got dsc Start(2)
13256 start 2
13257 Oct 13 18:12:00.660 INFO request completed, latency_us: 313, response_code: 204, uri: /start/cid/2, method: POST, req_id: b009ec12-92b9-4683-90e0-68292968fa26, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
13258 Replay: started 2, returned:Ok(())
13259 ----------------------------------------------------------------
13260 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
13261 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
13262 5147 Acked 6146 Read 8 Done Done New false
13263 5148 Acked 6147 Read 4 Done Done New false
13264 5149 Acked 6148 Read 10 Done Done New false
13265 5150 Acked 6149 Read 6 Done Done New false
13266 5151 Acked 6150 Flush 0 Done Done New false
13267 5152 Acked 6151 Read 5 Done Done New false
13268 5153 Acked 6152 Write 4 Done Done New false
13269 5154 Acked 6153 Read 4 Done Done New false
13270 5155 Acked 6154 Read 3 Done Done New false
13271 5156 Acked 6155 Read 8 Done Done New false
13272 5157 Acked 6156 Flush 0 Done Done New false
13273 5158 Acked 6157 Flush 0 Done Done New false
13274 5159 Acked 6158 Flush 0 Done Done New false
13275 5160 Acked 6159 Read 10 Done Done New false
13276 5161 Acked 6160 Read 10 Done Done New false
13277 5162 Acked 6161 Write 3 Done Done New false
13278 5163 Acked 6162 Read 10 Done Done New false
13279 5164 Acked 6163 Write 2 Done Done New false
13280 5165 Acked 6164 Write 6 Done Done New false
13281 5166 Acked 6165 Read 3 Done Done New false
13282 5167 Acked 6166 Write 7 Done Done New false
13283 5168 Acked 6167 Write 4 Done Done New false
13284 5169 Acked 6168 Read 7 Done Done New false
13285 5170 Acked 6169 Write 10 Done Done New false
13286 5171 Acked 6170 Read 6 Done Done New false
13287 5172 Acked 6171 Read 8 Done Done New false
13288 5173 Acked 6172 Read 2 Done Done New false
13289 5174 Acked 6173 Write 5 Done Done New false
13290 5175 Acked 6174 Write 3 Done Done New false
13291 5176 Acked 6175 Read 8 Done Done New false
13292 5177 Acked 6176 Write 3 Done Done New false
13293 5178 Acked 6177 Read 6 Done Done New false
13294 5179 Acked 6178 Write 8 Done Done New false
13295 5180 Acked 6179 Read 7 Done Done New false
13296 5181 Acked 6180 Flush 0 Done Done New false
13297 5182 Acked 6181 Read 9 Done Done New false
13298 5183 Acked 6182 Read 2 Done Done New false
13299 5184 Acked 6183 Read 8 Done Done New false
13300 5185 Acked 6184 Flush 0 Done Done New false
13301 5186 Acked 6185 Read 1 Done Done New false
13302 5187 Acked 6186 Write 3 Done Done New false
13303 5188 Acked 6187 Flush 0 Done Done New false
13304 5189 Acked 6188 Read 4 Done Done New false
13305 5190 Acked 6189 Read 1 Done Done New false
13306 5191 Acked 6190 Read 6 Done Done New false
13307 5192 Acked 6191 Read 5 Done Done New false
13308 5193 Acked 6192 Write 9 Done Done New false
13309 5194 Acked 6193 Read 8 Done Done New false
13310 5195 Acked 6194 Write 4 Done Done New false
13311 5196 Acked 6195 Write 6 Done Done New false
13312 5197 Acked 6196 Flush 0 Done Done New false
13313 5198 Acked 6197 Read 7 Done Done New false
13314 5199 Acked 6198 Write 3 Done Done New false
13315 5200 Acked 6199 Write 9 Done Done New false
13316 5201 Acked 6200 Write 8 Done Done New false
13317 5202 Acked 6201 Read 7 Done Done New false
13318 5203 Acked 6202 Read 1 Done Done New false
13319 5204 Acked 6203 Read 10 Done Done New false
13320 5205 Acked 6204 Write 4 Done Done New false
13321 5206 Acked 6205 Read 8 Done Done New false
13322 5207 Acked 6206 Write 2 Done Done New false
13323 5208 Acked 6207 Read 3 Done Done New false
13324 5209 Acked 6208 Read 1 Done Done New false
13325 5210 Acked 6209 Write 7 Done Done New false
13326 5211 Acked 6210 Read 8 Done Done New false
13327 5212 Acked 6211 Flush 0 Done Done New false
13328 5213 Acked 6212 Read 2 Done Done New false
13329 5214 Acked 6213 Read 1 Done Done New false
13330 5215 Acked 6214 Write 3 Done Done New false
13331 5216 Acked 6215 Write 6 Done Done New false
13332 5217 Acked 6216 Write 2 Done Done New false
13333 5218 Acked 6217 Write 1 Done Done New false
13334 5219 Acked 6218 Flush 0 Done Done New false
13335 5220 Acked 6219 Write 8 Done Done New false
13336 5221 Acked 6220 Write 5 Done Done New false
13337 5222 Acked 6221 Write 9 Done Done New false
13338 5223 Acked 6222 Read 7 Done Done New false
13339 5224 Acked 6223 Read 1 Done Done New false
13340 5225 Acked 6224 Read 6 Done Done New false
13341 5226 Acked 6225 Read 8 Done Done New false
13342 5227 Acked 6226 Read 1 Done Done New false
13343 5228 Acked 6227 Write 1 Done Done New false
13344 5229 Acked 6228 Read 1 Done Done New false
13345 5230 Acked 6229 Write 6 Done Done New false
13346 5231 Acked 6230 Read 10 Done Done New false
13347 5232 Acked 6231 Flush 0 Done Done New false
13348 5233 Acked 6232 Write 10 Done Done New false
13349 5234 Acked 6233 Write 2 Done Done New false
13350 5235 Acked 6234 Read 4 Done Done New false
13351 5236 Acked 6235 Read 9 Done Done New false
13352 5237 Acked 6236 Read 8 Done Done New false
13353 5238 Acked 6237 Write 1 Done Done New false
13354 5239 Acked 6238 Write 3 Done Done New false
13355 5240 Acked 6239 Write 10 Done Done New false
13356 5241 Acked 6240 Write 2 Done Done New false
13357 5242 Acked 6241 Read 1 Done Done New false
13358 5243 Acked 6242 Read 2 Done Done New false
13359 5244 Acked 6243 Read 2 Done Done New false
13360 5245 Acked 6244 Read 3 Done Done New false
13361 5246 Acked 6245 Read 8 Done Done New false
13362 5247 Acked 6246 Write 2 Done Done New false
13363 5248 Acked 6247 Write 3 Done Done New false
13364 5249 Acked 6248 Read 6 Done Done New false
13365 5250 Acked 6249 Read 7 Done Done New false
13366 5251 Acked 6250 Read 8 Done Done New false
13367 5252 Acked 6251 Read 7 Done Done New false
13368 5253 Acked 6252 Write 6 Done Done New false
13369 5254 Acked 6253 Read 7 Done Done New false
13370 5255 Acked 6254 Read 7 Done Done New false
13371 5256 Acked 6255 Read 6 Done Done New false
13372 5257 Acked 6256 Read 10 Done Done New false
13373 5258 Acked 6257 Write 4 Done Done New false
13374 5259 Acked 6258 Read 9 Done Done New false
13375 5260 Acked 6259 Flush 0 Done Done New false
13376 5261 Acked 6260 Write 1 Done Done New false
13377 5262 Acked 6261 Read 5 Done Done New false
13378 5263 Acked 6262 Flush 0 Done Done New false
13379 5264 Acked 6263 Write 10 Done Done New false
13380 5265 Acked 6264 Write 7 Done Done New false
13381 5266 Acked 6265 Write 5 Done Done New false
13382 5267 Acked 6266 Write 4 Done Done New false
13383 5268 Acked 6267 Write 10 Done Done New false
13384 5269 Acked 6268 Write 1 Done Done New false
13385 5270 Acked 6269 Read 6 Done Done New false
13386 5271 Acked 6270 Write 7 Done Done New false
13387 5272 Acked 6271 Write 9 Done Done New false
13388 5273 Acked 6272 Read 10 Done Done New false
13389 5274 Acked 6273 Flush 0 Done Done New false
13390 5275 Acked 6274 Read 9 Done Done New false
13391 5276 Acked 6275 Flush 0 Done Done New false
13392 5277 Acked 6276 Flush 0 Done Done New false
13393 5278 Acked 6277 Read 10 Done Done New false
13394 5279 Acked 6278 Write 3 Done Done New false
13395 5280 Acked 6279 Read 8 Done Done New false
13396 5281 Acked 6280 Write 2 Done Done New false
13397 5282 Acked 6281 Write 6 Done Done New false
13398 5283 Acked 6282 Read 4 Done Done New false
13399 5284 Acked 6283 Read 2 Done Done New false
13400 5285 Acked 6284 Read 9 Done Done New false
13401 5286 Acked 6285 Read 9 Done Done New false
13402 5287 Acked 6286 Write 3 Done Done New false
13403 5288 Acked 6287 Read 9 Done Done New false
13404 5289 Acked 6288 Read 8 Done Done New false
13405 5290 Acked 6289 Read 5 Done Done New false
13406 5291 Acked 6290 Read 4 Done Done New false
13407 5292 Acked 6291 Read 4 Done Done New false
13408 5293 Acked 6292 Write 3 Done Done New false
13409 5294 Acked 6293 Write 5 Done Done New false
13410 5295 Acked 6294 Write 1 Done Done New false
13411 5296 Acked 6295 Read 6 Done Done New false
13412 5297 Acked 6296 Write 8 Done Done New false
13413 5298 Acked 6297 Write 10 Done Done New false
13414 5299 Acked 6298 Write 5 Done Done New false
13415 5300 Acked 6299 Flush 0 Done Done New false
13416 5301 Acked 6300 Write 9 Done Done New false
13417 5302 Acked 6301 Flush 0 Done Done New false
13418 5303 Acked 6302 Write 1 Done Done New false
13419 5304 Acked 6303 Read 6 Done Done New false
13420 5305 Acked 6304 Write 8 Done Done New false
13421 5306 Acked 6305 Read 2 Done Done New false
13422 5307 Acked 6306 Read 2 Done Done New false
13423 5308 Acked 6307 Read 7 Done Done New false
13424 5309 Acked 6308 Write 4 Done Done New false
13425 5310 Acked 6309 Write 9 Done Done New false
13426 5311 Acked 6310 Flush 0 Done Done New false
13427 5312 Acked 6311 Write 6 Done Done New false
13428 5313 Acked 6312 Flush 0 Done Done New false
13429 5314 Acked 6313 Flush 0 Done Done New false
13430 5315 Acked 6314 Write 2 Done Done New false
13431 5316 Acked 6315 Write 1 Done Done New false
13432 5317 Acked 6316 Write 8 Done Done New false
13433 5318 Acked 6317 Write 3 Done Done New false
13434 5319 Acked 6318 Read 5 Done Done New false
13435 5320 Acked 6319 Read 8 Done Done New false
13436 5321 Acked 6320 Write 10 Done Done New false
13437 5322 Acked 6321 Write 3 Done Done New false
13438 5323 Acked 6322 Read 2 Done Done New false
13439 5324 Acked 6323 Read 5 Done Done New false
13440 5325 Acked 6324 Write 1 Done Done New false
13441 5326 Acked 6325 Read 3 Done Done New false
13442 5327 Acked 6326 Flush 0 Done Done New false
13443 5328 Acked 6327 Write 8 Done Done New false
13444 5329 Acked 6328 Read 7 Done Done New false
13445 5330 Acked 6329 Read 8 Done Done New false
13446 5331 Acked 6330 Read 6 Done Done New false
13447 5332 Acked 6331 Write 1 Done Done New false
13448 5333 Acked 6332 Read 4 Done Done New false
13449 5334 Acked 6333 Write 7 Done Done New false
13450 5335 Acked 6334 Read 1 Done Done New false
13451 5336 Acked 6335 Write 7 Done Done New false
13452 5337 Acked 6336 Write 5 Done Done New false
13453 5338 Acked 6337 Write 8 Done Done New false
13454 5339 Acked 6338 Flush 0 Done Done New false
13455 5340 Acked 6339 Read 1 Done Done New false
13456 5341 Acked 6340 Read 4 Done Done New false
13457 5342 Acked 6341 Write 6 Done Done New false
13458 5343 Acked 6342 Read 4 Done Done New false
13459 5344 Acked 6343 Flush 0 Done Done New false
13460 5345 Acked 6344 Read 4 Done Done New false
13461 5346 Acked 6345 Write 4 Done Done New false
13462 5347 Acked 6346 Flush 0 Done Done New false
13463 5348 Acked 6347 Write 7 Done Done New false
13464 5349 Acked 6348 Read 7 Done Done New false
13465 5350 Acked 6349 Read 9 Done Done New false
13466 5351 Acked 6350 Write 7 Done Done New false
13467 5352 Acked 6351 Read 2 Done Done New false
13468 5353 Acked 6352 Write 5 Done Done New false
13469 5354 Acked 6353 Read 4 Done Done New false
13470 5355 Acked 6354 Read 8 Done Done New false
13471 5356 Acked 6355 Write 2 Done Done New false
13472 5357 Acked 6356 Read 7 Done Done New false
13473 5358 Acked 6357 Write 5 Done Done New false
13474 5359 Acked 6358 Read 7 Done Done New false
13475 5360 Acked 6359 Read 7 Done Done New false
13476 5361 Acked 6360 Read 6 Done Done New false
13477 5362 Acked 6361 Read 1 Done Done New false
13478 5363 Acked 6362 Read 5 Done Done New false
13479 5364 Acked 6363 Read 3 Done Done New false
13480 5365 Acked 6364 Write 3 Done Done New false
13481 5366 Acked 6365 Read 8 Done Done New false
13482 5367 Acked 6366 Write 7 Done Done New false
13483 5368 Acked 6367 Write 5 Done Done New false
13484 5369 Acked 6368 Flush 0 Done Done New false
13485 5370 Acked 6369 Read 3 Done Done New false
13486 5371 Acked 6370 Write 5 Done Done New false
13487 5372 Acked 6371 Write 5 Done Done New false
13488 5373 Acked 6372 Write 7 Done Done New false
13489 5374 Acked 6373 Write 5 Done Done New false
13490 5375 Acked 6374 Write 7 Done Done New false
13491 5376 Acked 6375 Read 4 Done Done New false
13492 5377 Acked 6376 Flush 0 Done Done New false
13493 5378 Acked 6377 Write 8 Done Done New false
13494 5379 Acked 6378 Read 5 Done Done New false
13495 5380 Acked 6379 Write 4 Done Done New false
13496 5381 Acked 6380 Write 2 Done Done New false
13497 5382 Acked 6381 Read 4 Done Done New false
13498 5383 Acked 6382 Flush 0 Done Done New false
13499 5384 Acked 6383 Read 9 Done Done New false
13500 5385 Acked 6384 Write 6 Done Done New false
13501 5386 Acked 6385 Write 10 Done Done New false
13502 5387 Acked 6386 Write 2 Done Done New false
13503 5388 Acked 6387 Write 10 Done Done New false
13504 5389 Acked 6388 Read 4 Done Done New false
13505 5390 Acked 6389 Write 10 Done Done New false
13506 5391 Acked 6390 Write 9 Done Done New false
13507 5392 Acked 6391 Read 9 Done Done New false
13508 5393 Acked 6392 Write 3 Done Done New false
13509 5394 Acked 6393 Flush 0 Done Done New false
13510 5395 Acked 6394 Write 1 Done Done New false
13511 5396 Acked 6395 Flush 0 Done Done New false
13512 5397 Acked 6396 Read 9 Done Done New false
13513 5398 Acked 6397 Write 7 Done Done New false
13514 5399 Acked 6398 Read 6 Done Done New false
13515 5400 Acked 6399 Flush 0 Done Done New false
13516 5401 Acked 6400 Read 1 Done Done New false
13517 5402 Acked 6401 Read 5 Done Done New false
13518 5403 Acked 6402 Write 5 Done Done New false
13519 5404 Acked 6403 Read 8 Done Done New false
13520 5405 Acked 6404 Write 1 Done Done New false
13521 5406 Acked 6405 Read 10 Done Done New false
13522 5407 Acked 6406 Write 9 Done Done New false
13523 5408 Acked 6407 Read 6 Done Done New false
13524 5409 Acked 6408 Read 10 Done Done New false
13525 5410 Acked 6409 Read 1 Done Done New false
13526 5411 Acked 6410 Write 2 Done Done New false
13527 5412 Acked 6411 Read 1 Done Done New false
13528 5413 Acked 6412 Read 8 Done Done New false
13529 5414 Acked 6413 Read 6 Done Done New false
13530 5415 Acked 6414 Write 8 Done Done New false
13531 5416 Acked 6415 Read 6 Done Done New false
13532 5417 Acked 6416 Read 4 Done Done New false
13533 5418 Acked 6417 Read 2 Done Done New false
13534 5419 Acked 6418 Read 6 Done Done New false
13535 5420 Acked 6419 Write 7 Done Done New false
13536 5421 Acked 6420 Write 8 Done Done New false
13537 5422 Acked 6421 Write 9 Done Done New false
13538 5423 Acked 6422 Read 4 Done Done New false
13539 5424 Acked 6423 Read 10 Done Done New false
13540 5425 Acked 6424 Read 7 Done Done New false
13541 5426 Acked 6425 Read 2 Done Done New false
13542 5427 Acked 6426 Read 10 Done Done New false
13543 5428 Acked 6427 Write 5 Done Done New false
13544 5429 Acked 6428 Write 2 Done Done New false
13545 5430 Acked 6429 Write 5 Done Done New false
13546 5431 Acked 6430 Read 8 Done Done New false
13547 5432 Acked 6431 Read 1 Done Done New false
13548 5433 Acked 6432 Write 9 Done Done New false
13549 5434 Acked 6433 Read 2 Done Done New false
13550 5435 Acked 6434 Write 2 Done Done New false
13551 5436 Acked 6435 Read 8 Done Done New false
13552 5437 Acked 6436 Read 6 Done Done New false
13553 5438 Acked 6437 Read 8 Done Done New false
13554 5439 Acked 6438 Read 5 Done Done New false
13555 5440 Acked 6439 Read 1 Done Done New false
13556 5441 Acked 6440 Write 8 Done Done New false
13557 5442 Acked 6441 Read 4 Done Done New false
13558 5443 Acked 6442 Write 6 Done Done New false
13559 5444 Acked 6443 Read 8 Done Done New false
13560 5445 Acked 6444 Flush 0 Done Done New false
13561 5446 Acked 6445 Write 3 Done Done New false
13562 5447 Acked 6446 Read 1 Done Done New false
13563 5448 Acked 6447 Write 7 Sent Sent New false
13564 STATES DS:0 DS:1 DS:2 TOTAL
13565 New 0 0 302 302
13566 Sent 1 1 0 2
13567 Done 301 301 0 602
13568 Skipped 0 0 0 0
13569 Error 0 0 0 0
13570 Last Flush: 6444 6444 6145
13571 Downstairs last five completed: 6145 6144 6143 6142 6141 6140
13572 Upstairs last five completed: 5448 5447 5446 5445 5444 5443
13573 CLIENT: Up:0 ds:302 act:2
13574 [8830] got start action
13575 [8830] I am going to restart
13576 Starting downstairs at port 8830
13577 Make output file at "/tmp/dsc/downstairs-8830.txt"
13578 [8830][2] reports Starting
13579 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1194)
13580 [8830][2] reports Running
13581 ----------------------------------------------------------------
13582 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
13583 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
13584 5147 Acked 6146 Read 8 Done Done New false
13585 5148 Acked 6147 Read 4 Done Done New false
13586 5149 Acked 6148 Read 10 Done Done New false
13587 5150 Acked 6149 Read 6 Done Done New false
13588 5151 Acked 6150 Flush 0 Done Done New false
13589 5152 Acked 6151 Read 5 Done Done New false
13590 5153 Acked 6152 Write 4 Done Done New false
13591 5154 Acked 6153 Read 4 Done Done New false
13592 5155 Acked 6154 Read 3 Done Done New false
13593 5156 Acked 6155 Read 8 Done Done New false
13594 5157 Acked 6156 Flush 0 Done Done New false
13595 5158 Acked 6157 Flush 0 Done Done New false
13596 5159 Acked 6158 Flush 0 Done Done New false
13597 5160 Acked 6159 Read 10 Done Done New false
13598 5161 Acked 6160 Read 10 Done Done New false
13599 5162 Acked 6161 Write 3 Done Done New false
13600 5163 Acked 6162 Read 10 Done Done New false
13601 5164 Acked 6163 Write 2 Done Done New false
13602 5165 Acked 6164 Write 6 Done Done New false
13603 5166 Acked 6165 Read 3 Done Done New false
13604 5167 Acked 6166 Write 7 Done Done New false
13605 5168 Acked 6167 Write 4 Done Done New false
13606 5169 Acked 6168 Read 7 Done Done New false
13607 5170 Acked 6169 Write 10 Done Done New false
13608 5171 Acked 6170 Read 6 Done Done New false
13609 5172 Acked 6171 Read 8 Done Done New false
13610 5173 Acked 6172 Read 2 Done Done New false
13611 5174 Acked 6173 Write 5 Done Done New false
13612 5175 Acked 6174 Write 3 Done Done New false
13613 5176 Acked 6175 Read 8 Done Done New false
13614 5177 Acked 6176 Write 3 Done Done New false
13615 5178 Acked 6177 Read 6 Done Done New false
13616 5179 Acked 6178 Write 8 Done Done New false
13617 5180 Acked 6179 Read 7 Done Done New false
13618 5181 Acked 6180 Flush 0 Done Done New false
13619 5182 Acked 6181 Read 9 Done Done New false
13620 5183 Acked 6182 Read 2 Done Done New false
13621 5184 Acked 6183 Read 8 Done Done New false
13622 5185 Acked 6184 Flush 0 Done Done New false
13623 5186 Acked 6185 Read 1 Done Done New false
13624 5187 Acked 6186 Write 3 Done Done New false
13625 5188 Acked 6187 Flush 0 Done Done New false
13626 5189 Acked 6188 Read 4 Done Done New false
13627 5190 Acked 6189 Read 1 Done Done New false
13628 5191 Acked 6190 Read 6 Done Done New false
13629 5192 Acked 6191 Read 5 Done Done New false
13630 5193 Acked 6192 Write 9 Done Done New false
13631 5194 Acked 6193 Read 8 Done Done New false
13632 5195 Acked 6194 Write 4 Done Done New false
13633 5196 Acked 6195 Write 6 Done Done New false
13634 5197 Acked 6196 Flush 0 Done Done New false
13635 5198 Acked 6197 Read 7 Done Done New false
13636 5199 Acked 6198 Write 3 Done Done New false
13637 5200 Acked 6199 Write 9 Done Done New false
13638 5201 Acked 6200 Write 8 Done Done New false
13639 5202 Acked 6201 Read 7 Done Done New false
13640 5203 Acked 6202 Read 1 Done Done New false
13641 5204 Acked 6203 Read 10 Done Done New false
13642 5205 Acked 6204 Write 4 Done Done New false
13643 5206 Acked 6205 Read 8 Done Done New false
13644 5207 Acked 6206 Write 2 Done Done New false
13645 5208 Acked 6207 Read 3 Done Done New false
13646 5209 Acked 6208 Read 1 Done Done New false
13647 5210 Acked 6209 Write 7 Done Done New false
13648 5211 Acked 6210 Read 8 Done Done New false
13649 5212 Acked 6211 Flush 0 Done Done New false
13650 5213 Acked 6212 Read 2 Done Done New false
13651 5214 Acked 6213 Read 1 Done Done New false
13652 5215 Acked 6214 Write 3 Done Done New false
13653 5216 Acked 6215 Write 6 Done Done New false
13654 5217 Acked 6216 Write 2 Done Done New false
13655 5218 Acked 6217 Write 1 Done Done New false
13656 5219 Acked 6218 Flush 0 Done Done New false
13657 5220 Acked 6219 Write 8 Done Done New false
13658 5221 Acked 6220 Write 5 Done Done New false
13659 5222 Acked 6221 Write 9 Done Done New false
13660 5223 Acked 6222 Read 7 Done Done New false
13661 5224 Acked 6223 Read 1 Done Done New false
13662 5225 Acked 6224 Read 6 Done Done New false
13663 5226 Acked 6225 Read 8 Done Done New false
13664 5227 Acked 6226 Read 1 Done Done New false
13665 5228 Acked 6227 Write 1 Done Done New false
13666 5229 Acked 6228 Read 1 Done Done New false
13667 5230 Acked 6229 Write 6 Done Done New false
13668 5231 Acked 6230 Read 10 Done Done New false
13669 5232 Acked 6231 Flush 0 Done Done New false
13670 5233 Acked 6232 Write 10 Done Done New false
13671 5234 Acked 6233 Write 2 Done Done New false
13672 5235 Acked 6234 Read 4 Done Done New false
13673 5236 Acked 6235 Read 9 Done Done New false
13674 5237 Acked 6236 Read 8 Done Done New false
13675 5238 Acked 6237 Write 1 Done Done New false
13676 5239 Acked 6238 Write 3 Done Done New false
13677 5240 Acked 6239 Write 10 Done Done New false
13678 5241 Acked 6240 Write 2 Done Done New false
13679 5242 Acked 6241 Read 1 Done Done New false
13680 5243 Acked 6242 Read 2 Done Done New false
13681 5244 Acked 6243 Read 2 Done Done New false
13682 5245 Acked 6244 Read 3 Done Done New false
13683 5246 Acked 6245 Read 8 Done Done New false
13684 5247 Acked 6246 Write 2 Done Done New false
13685 5248 Acked 6247 Write 3 Done Done New false
13686 5249 Acked 6248 Read 6 Done Done New false
13687 5250 Acked 6249 Read 7 Done Done New false
13688 5251 Acked 6250 Read 8 Done Done New false
13689 5252 Acked 6251 Read 7 Done Done New false
13690 5253 Acked 6252 Write 6 Done Done New false
13691 5254 Acked 6253 Read 7 Done Done New false
13692 5255 Acked 6254 Read 7 Done Done New false
13693 5256 Acked 6255 Read 6 Done Done New false
13694 5257 Acked 6256 Read 10 Done Done New false
13695 5258 Acked 6257 Write 4 Done Done New false
13696 5259 Acked 6258 Read 9 Done Done New false
13697 5260 Acked 6259 Flush 0 Done Done New false
13698 5261 Acked 6260 Write 1 Done Done New false
13699 5262 Acked 6261 Read 5 Done Done New false
13700 5263 Acked 6262 Flush 0 Done Done New false
13701 5264 Acked 6263 Write 10 Done Done New false
13702 5265 Acked 6264 Write 7 Done Done New false
13703 5266 Acked 6265 Write 5 Done Done New false
13704 5267 Acked 6266 Write 4 Done Done New false
13705 5268 Acked 6267 Write 10 Done Done New false
13706 5269 Acked 6268 Write 1 Done Done New false
13707 5270 Acked 6269 Read 6 Done Done New false
13708 5271 Acked 6270 Write 7 Done Done New false
13709 5272 Acked 6271 Write 9 Done Done New false
13710 5273 Acked 6272 Read 10 Done Done New false
13711 5274 Acked 6273 Flush 0 Done Done New false
13712 5275 Acked 6274 Read 9 Done Done New false
13713 5276 Acked 6275 Flush 0 Done Done New false
13714 5277 Acked 6276 Flush 0 Done Done New false
13715 5278 Acked 6277 Read 10 Done Done New false
13716 5279 Acked 6278 Write 3 Done Done New false
13717 5280 Acked 6279 Read 8 Done Done New false
13718 5281 Acked 6280 Write 2 Done Done New false
13719 5282 Acked 6281 Write 6 Done Done New false
13720 5283 Acked 6282 Read 4 Done Done New false
13721 5284 Acked 6283 Read 2 Done Done New false
13722 5285 Acked 6284 Read 9 Done Done New false
13723 5286 Acked 6285 Read 9 Done Done New false
13724 5287 Acked 6286 Write 3 Done Done New false
13725 5288 Acked 6287 Read 9 Done Done New false
13726 5289 Acked 6288 Read 8 Done Done New false
13727 5290 Acked 6289 Read 5 Done Done New false
13728 5291 Acked 6290 Read 4 Done Done New false
13729 5292 Acked 6291 Read 4 Done Done New false
13730 5293 Acked 6292 Write 3 Done Done New false
13731 5294 Acked 6293 Write 5 Done Done New false
13732 5295 Acked 6294 Write 1 Done Done New false
13733 5296 Acked 6295 Read 6 Done Done New false
13734 5297 Acked 6296 Write 8 Done Done New false
13735 5298 Acked 6297 Write 10 Done Done New false
13736 5299 Acked 6298 Write 5 Done Done New false
13737 5300 Acked 6299 Flush 0 Done Done New false
13738 5301 Acked 6300 Write 9 Done Done New false
13739 5302 Acked 6301 Flush 0 Done Done New false
13740 5303 Acked 6302 Write 1 Done Done New false
13741 5304 Acked 6303 Read 6 Done Done New false
13742 5305 Acked 6304 Write 8 Done Done New false
13743 5306 Acked 6305 Read 2 Done Done New false
13744 5307 Acked 6306 Read 2 Done Done New false
13745 5308 Acked 6307 Read 7 Done Done New false
13746 5309 Acked 6308 Write 4 Done Done New false
13747 5310 Acked 6309 Write 9 Done Done New false
13748 5311 Acked 6310 Flush 0 Done Done New false
13749 5312 Acked 6311 Write 6 Done Done New false
13750 5313 Acked 6312 Flush 0 Done Done New false
13751 5314 Acked 6313 Flush 0 Done Done New false
13752 5315 Acked 6314 Write 2 Done Done New false
13753 5316 Acked 6315 Write 1 Done Done New false
13754 5317 Acked 6316 Write 8 Done Done New false
13755 5318 Acked 6317 Write 3 Done Done New false
13756 5319 Acked 6318 Read 5 Done Done New false
13757 5320 Acked 6319 Read 8 Done Done New false
13758 5321 Acked 6320 Write 10 Done Done New false
13759 5322 Acked 6321 Write 3 Done Done New false
13760 5323 Acked 6322 Read 2 Done Done New false
13761 5324 Acked 6323 Read 5 Done Done New false
13762 5325 Acked 6324 Write 1 Done Done New false
13763 5326 Acked 6325 Read 3 Done Done New false
13764 5327 Acked 6326 Flush 0 Done Done New false
13765 5328 Acked 6327 Write 8 Done Done New false
13766 5329 Acked 6328 Read 7 Done Done New false
13767 5330 Acked 6329 Read 8 Done Done New false
13768 5331 Acked 6330 Read 6 Done Done New false
13769 5332 Acked 6331 Write 1 Done Done New false
13770 5333 Acked 6332 Read 4 Done Done New false
13771 5334 Acked 6333 Write 7 Done Done New false
13772 5335 Acked 6334 Read 1 Done Done New false
13773 5336 Acked 6335 Write 7 Done Done New false
13774 5337 Acked 6336 Write 5 Done Done New false
13775 5338 Acked 6337 Write 8 Done Done New false
13776 5339 Acked 6338 Flush 0 Done Done New false
13777 5340 Acked 6339 Read 1 Done Done New false
13778 5341 Acked 6340 Read 4 Done Done New false
13779 5342 Acked 6341 Write 6 Done Done New false
13780 5343 Acked 6342 Read 4 Done Done New false
13781 5344 Acked 6343 Flush 0 Done Done New false
13782 5345 Acked 6344 Read 4 Done Done New false
13783 5346 Acked 6345 Write 4 Done Done New false
13784 5347 Acked 6346 Flush 0 Done Done New false
13785 5348 Acked 6347 Write 7 Done Done New false
13786 5349 Acked 6348 Read 7 Done Done New false
13787 5350 Acked 6349 Read 9 Done Done New false
13788 5351 Acked 6350 Write 7 Done Done New false
13789 5352 Acked 6351 Read 2 Done Done New false
13790 5353 Acked 6352 Write 5 Done Done New false
13791 5354 Acked 6353 Read 4 Done Done New false
13792 5355 Acked 6354 Read 8 Done Done New false
13793 5356 Acked 6355 Write 2 Done Done New false
13794 5357 Acked 6356 Read 7 Done Done New false
13795 5358 Acked 6357 Write 5 Done Done New false
13796 5359 Acked 6358 Read 7 Done Done New false
13797 5360 Acked 6359 Read 7 Done Done New false
13798 5361 Acked 6360 Read 6 Done Done New false
13799 5362 Acked 6361 Read 1 Done Done New false
13800 5363 Acked 6362 Read 5 Done Done New false
13801 5364 Acked 6363 Read 3 Done Done New false
13802 5365 Acked 6364 Write 3 Done Done New false
13803 5366 Acked 6365 Read 8 Done Done New false
13804 5367 Acked 6366 Write 7 Done Done New false
13805 5368 Acked 6367 Write 5 Done Done New false
13806 5369 Acked 6368 Flush 0 Done Done New false
13807 5370 Acked 6369 Read 3 Done Done New false
13808 5371 Acked 6370 Write 5 Done Done New false
13809 5372 Acked 6371 Write 5 Done Done New false
13810 5373 Acked 6372 Write 7 Done Done New false
13811 5374 Acked 6373 Write 5 Done Done New false
13812 5375 Acked 6374 Write 7 Done Done New false
13813 5376 Acked 6375 Read 4 Done Done New false
13814 5377 Acked 6376 Flush 0 Done Done New false
13815 5378 Acked 6377 Write 8 Done Done New false
13816 5379 Acked 6378 Read 5 Done Done New false
13817 5380 Acked 6379 Write 4 Done Done New false
13818 5381 Acked 6380 Write 2 Done Done New false
13819 5382 Acked 6381 Read 4 Done Done New false
13820 5383 Acked 6382 Flush 0 Done Done New false
13821 5384 Acked 6383 Read 9 Done Done New false
13822 5385 Acked 6384 Write 6 Done Done New false
13823 5386 Acked 6385 Write 10 Done Done New false
13824 5387 Acked 6386 Write 2 Done Done New false
13825 5388 Acked 6387 Write 10 Done Done New false
13826 5389 Acked 6388 Read 4 Done Done New false
13827 5390 Acked 6389 Write 10 Done Done New false
13828 5391 Acked 6390 Write 9 Done Done New false
13829 5392 Acked 6391 Read 9 Done Done New false
13830 5393 Acked 6392 Write 3 Done Done New false
13831 5394 Acked 6393 Flush 0 Done Done New false
13832 5395 Acked 6394 Write 1 Done Done New false
13833 5396 Acked 6395 Flush 0 Done Done New false
13834 5397 Acked 6396 Read 9 Done Done New false
13835 5398 Acked 6397 Write 7 Done Done New false
13836 5399 Acked 6398 Read 6 Done Done New false
13837 5400 Acked 6399 Flush 0 Done Done New false
13838 5401 Acked 6400 Read 1 Done Done New false
13839 5402 Acked 6401 Read 5 Done Done New false
13840 5403 Acked 6402 Write 5 Done Done New false
13841 5404 Acked 6403 Read 8 Done Done New false
13842 5405 Acked 6404 Write 1 Done Done New false
13843 5406 Acked 6405 Read 10 Done Done New false
13844 5407 Acked 6406 Write 9 Done Done New false
13845 5408 Acked 6407 Read 6 Done Done New false
13846 5409 Acked 6408 Read 10 Done Done New false
13847 5410 Acked 6409 Read 1 Done Done New false
13848 5411 Acked 6410 Write 2 Done Done New false
13849 5412 Acked 6411 Read 1 Done Done New false
13850 5413 Acked 6412 Read 8 Done Done New false
13851 5414 Acked 6413 Read 6 Done Done New false
13852 5415 Acked 6414 Write 8 Done Done New false
13853 5416 Acked 6415 Read 6 Done Done New false
13854 5417 Acked 6416 Read 4 Done Done New false
13855 5418 Acked 6417 Read 2 Done Done New false
13856 5419 Acked 6418 Read 6 Done Done New false
13857 5420 Acked 6419 Write 7 Done Done New false
13858 5421 Acked 6420 Write 8 Done Done New false
13859 5422 Acked 6421 Write 9 Done Done New false
13860 5423 Acked 6422 Read 4 Done Done New false
13861 5424 Acked 6423 Read 10 Done Done New false
13862 5425 Acked 6424 Read 7 Done Done New false
13863 5426 Acked 6425 Read 2 Done Done New false
13864 5427 Acked 6426 Read 10 Done Done New false
13865 5428 Acked 6427 Write 5 Done Done New false
13866 5429 Acked 6428 Write 2 Done Done New false
13867 5430 Acked 6429 Write 5 Done Done New false
13868 5431 Acked 6430 Read 8 Done Done New false
13869 5432 Acked 6431 Read 1 Done Done New false
13870 5433 Acked 6432 Write 9 Done Done New false
13871 5434 Acked 6433 Read 2 Done Done New false
13872 5435 Acked 6434 Write 2 Done Done New false
13873 5436 Acked 6435 Read 8 Done Done New false
13874 5437 Acked 6436 Read 6 Done Done New false
13875 5438 Acked 6437 Read 8 Done Done New false
13876 5439 Acked 6438 Read 5 Done Done New false
13877 5440 Acked 6439 Read 1 Done Done New false
13878 5441 Acked 6440 Write 8 Done Done New false
13879 5442 Acked 6441 Read 4 Done Done New false
13880 5443 Acked 6442 Write 6 Done Done New false
13881 5444 Acked 6443 Read 8 Done Done New false
13882 5445 Acked 6444 Flush 0 Done Done New false
13883 5446 Acked 6445 Write 3 Done Done New false
13884 5447 Acked 6446 Read 1 Done Done New false
13885 5448 Acked 6447 Write 7 Done Done New false
13886 5449 Acked 6448 Flush 0 Done Done New false
13887 STATES DS:0 DS:1 DS:2 TOTAL
13888 New 0 0 303 303
13889 Sent 0 0 0 0
13890 Done 303 303 0 606
13891 Skipped 0 0 0 0
13892 Error 0 0 0 0
13893 Last Flush: 6448 6448 6145
13894 Downstairs last five completed: 6145 6144 6143 6142 6141 6140
13895 Upstairs last five completed: 5449 5448 5447 5446 5445 5444
13896 CLIENT: Up:0 ds:303 act:2
138972023-10-13T18:12:04.706ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 2
138982023-10-13T18:12:04.706ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
138992023-10-13T18:12:04.706ZINFOcrucible: [2] client re-new 303 jobs since flush 6145 = downstairs
139002023-10-13T18:12:04.707ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
139012023-10-13T18:12:04.708ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
139022023-10-13T18:12:04.708ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
139032023-10-13T18:12:04.708ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
139042023-10-13T18:12:04.708ZINFOcrucible: [2] send last flush ID to this DS: 6145
139052023-10-13T18:12:04.708ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
139062023-10-13T18:12:04.708ZINFOcrucible: [2] Replied this last flush ID: 6145
139072023-10-13T18:12:04.708ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Offline ds_transition to Replay
139082023-10-13T18:12:04.708ZINFOcrucible: [2] Transition from Offline to Replay
139092023-10-13T18:12:04.708ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
139102023-10-13T18:12:04.708ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Replay ds_transition to Active
139112023-10-13T18:12:04.708ZINFOcrucible: [2] Transition from Replay to Active
139122023-10-13T18:12:04.708ZINFOcrucible: [2] Starts cmd_loop
139132023-10-13T18:12:04.709ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
139142023-10-13T18:12:04.709ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
139152023-10-13T18:12:04.709ZINFOcrucible: Set check for repair
139162023-10-13T18:12:05.709ZWARNcrucible: [2] flow control end
139172023-10-13T18:12:05.710ZINFOcrucible: Checking if live repair is needed
139182023-10-13T18:12:05.710ZINFOcrucible: No Live Repair required at this time
13919 ----------------------------------------------------------------
13920 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13921 Downstairs last five completed: 6448 6447 6446 6445 6444 6443
13922 Upstairs last five completed: 5449 5448 5447 5446 5445 5444
13923 CLIENT: Up:0 ds:0 act:3
13924 Replay: All jobs finished, all DS active.
13925 Main task has work to do, go find it
13926 got dsc Stop(1)
13927 stop 1
13928 Oct 13 18:12:08.668 INFO request completed, latency_us: 537, response_code: 204, uri: /stop/cid/1, method: POST, req_id: e25b501b-7e8f-4648-8ca8-9edf4313e13d, [8820] Got stop action so:false kr:false
13929 remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
13930 Oct 13 18:12:08.669 INFO request completed, latency_us: 238, response_code: 200, uri: /state/cid/1, method: GET, req_id: e7b0fc78-c179-4f86-9898-56f8dc245d91, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
139312023-10-13T18:12:08.670ZWARNcrucible: [1] downstairs disconnected
139322023-10-13T18:12:08.670ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
139332023-10-13T18:12:08.670ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8820 closed looper = 1
139342023-10-13T18:12:08.671ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
139352023-10-13T18:12:08.671ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
139362023-10-13T18:12:08.671ZWARNcrucible: [1] pm_task rx.recv() is None
139372023-10-13T18:12:08.671ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
13938 [8820] Exited with: ExitStatus(unix_wait_status(9))
13939 [8820][1] reports Exit
13940 Oct 13 18:12:12.671 INFO request completed, latency_us: 257, response_code: 200, uri: /state/cid/1, method: GET, req_id: fd1ef10d-dffa-4bfc-90c0-00058e6c4ba5, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
13941 001/300 Read block 2693 len 20480
13942 002/300 Write block 4942 len 28672 data: 3 3 3 4 4 7 6
13943 003/300 Write block 431 len 36864 data: 3 2 2 2 2 2 2 2 2
13944 004/300 Write block 2252 len 20480 data: 3 3 3 3 4
13945 005/300 Write block 402 len 36864 data: 3 4 4 4 3 3 3 3 3
13946 006/300 Write block 595 len 36864 data: 4 4 5 6 6 7 7 7 8
13947 007/300 Read block 3834 len 36864
13948 008/300 Write block 1833 len 24576 data: 5 5 6 5 6 5
13949 009/300 Write block 4955 len 32768 data: 3 3 3 3 4 5 4 5
13950 010/300 Read block 197 len 36864
13951 011/300 Read block 1288 len 20480
13952 012/300 Read block 3409 len 8192
13953 013/300 Read block 4458 len 28672
13954 014/300 Read block 865 len 36864
13955 015/300 Flush
13956 016/300 Read block 1296 len 20480
13957 017/300 Write block 935 len 16384 data: 5 5 5 6
13958 018/300 Write block 1325 len 36864 data: 4 4 4 4 3 4 5 5 5
13959 019/300 Flush
13960 020/300 Read block 4506 len 16384
13961 021/300 Read block 3451 len 28672
13962 022/300 Write block 3473 len 16384 data: 4 5 5 6
13963 023/300 Read block 675 len 12288
13964 024/300 Read block 3903 len 12288
13965 025/300 Read block 23 len 12288
13966 026/300 Write block 1896 len 36864 data: 4 5 5 4 4 4 4 3 3
13967 027/300 Read block 1702 len 28672
13968 028/300 Write block 1734 len 8192 data: 5 4
13969 029/300 Read block 4576 len 40960
13970 030/300 Read block 172 len 8192
13971 031/300 Write block 438 len 24576 data: 3 3 2 3 3 3
13972 032/300 Flush
13973 033/300 Read block 3228 len 28672
13974 034/300 Write block 1492 len 20480 data: 8 6 5 4 4
13975 035/300 Read block 4659 len 16384
13976 036/300 Write block 3531 len 36864 data: 3 3 5 7 6 7 8 10 10
13977 037/300 Read block 2697 len 24576
13978 038/300 Read block 1541 len 32768
13979 039/300 Write block 691 len 36864 data: 4 4 5 5 3 2 3 5 5
13980 040/300 Read block 283 len 16384
13981 041/300 Read block 4614 len 32768
13982 042/300 Read block 3145 len 36864
13983 043/300 Write block 2709 len 40960 data: 2 2 2 3 3 3 4 4 4 5
13984 044/300 Read block 234 len 32768
13985 045/300 Read block 2949 len 36864
13986 046/300 Read block 760 len 8192
13987 047/300 Flush
13988 048/300 Write block 4665 len 12288 data: 4 4 5
13989 049/300 Read block 3903 len 12288
13990 050/300 Read block 4746 len 8192
13991 051/300 Read block 1228 len 32768
13992 052/300 Flush
13993 053/300 Read block 2805 len 36864
13994 054/300 Write block 2473 len 28672 data: 6 5 5 3 3 4 4
13995 055/300 Write block 3192 len 28672 data: 6 7 6 6 5 4 5
13996 056/300 Read block 4715 len 8192
13997 057/300 Read block 843 len 20480
13998 058/300 Read block 1970 len 40960
13999 059/300 Write block 456 len 40960 data: 3 5 5 4 5 7 7 8 6 4
14000 060/300 Flush
14001 061/300 Read block 4693 len 32768
14002 062/300 Flush
14003 063/300 Read block 3242 len 8192
14004 064/300 Read block 1183 len 24576
14005 065/300 Write block 4600 len 36864 data: 5 4 3 2 3 3 4 3 5
14006 066/300 Write block 755 len 8192 data: 4 4
14007 067/300 Write block 4740 len 20480 data: 5 5 5 7 8
14008 068/300 Write block 4803 len 28672 data: 9 9 8 7 7 7 6
14009 069/300 Read block 2248 len 32768
14010 070/300 Write block 3088 len 28672 data: 5 4 4 5 6 7 7
14011 071/300 Flush
14012 072/300 Write block 908 len 12288 data: 4 4 3
14013 073/300 Read block 284 len 16384
14014 074/300 Read block 2107 len 28672
14015 075/300 Write block 4811 len 24576 data: 7 6 5 5 4 4
14016 076/300 Write block 1477 len 12288 data: 5 5 5
14017 077/300 Write block 1781 len 36864 data: 3 4 5 5 5 4 4 4 6
14018 078/300 Read block 1151 len 8192
14019 079/300 Read block 165 len 36864
14020 080/300 Flush
14021 081/300 Write block 318 len 40960 data: 8 8 8 7 7 7 6 5 6 6
14022 082/300 Read block 4575 len 20480
14023 083/300 Read block 339 len 20480
14024 084/300 Read block 4616 len 16384
14025 085/300 Write block 2782 len 36864 data: 5 5 3 3 3 4 4 3 3
14026 086/300 Read block 1384 len 8192
14027 087/300 Read block 1741 len 20480
14028 088/300 Read block 1235 len 20480
14029 089/300 Write block 2242 len 12288 data: 8 8 8
14030 090/300 Read block 104 len 16384
14031 091/300 Write block 3543 len 16384 data: 4 4 5 4
14032 092/300 Read block 4520 len 40960
14033 093/300 Read block 2365 len 4096
14034 094/300 Read block 620 len 16384
14035 095/300 Write block 3011 len 16384 data: 6 5 7 6
14036 096/300 Read block 4439 len 28672
14037 097/300 Read block 193 len 28672
14038 098/300 Read block 1784 len 8192
14039 099/300 Read block 3205 len 12288
14040 100/300 Write block 2099 len 40960 data: 4 4 2 3 3 5 5 3 5 5
14041 101/300 Write block 3089 len 28672 data: 5 5 6 7 8 8 9
14042 102/300 Write block 590 len 16384 data: 5 3 3 4
14043 103/300 Flush
14044 104/300 Write block 385 len 24576 data: 4 4 4 3 3 3
14045 105/300 Write block 4056 len 24576 data: 2 4 5 4 4 4
14046 106/300 Read block 4804 len 12288
14047 107/300 Read block 3579 len 40960
14048 108/300 Read block 1807 len 32768
14049 109/300 Read block 409 len 4096
14050 110/300 Read block 4584 len 36864
14051 111/300 Write block 2771 len 4096 data: 6
14052 112/300 Flush
14053 113/300 Read block 1669 len 8192
14054 114/300 Write block 1795 len 24576 data: 4 4 3 4 3 4
14055 115/300 Write block 4268 len 40960 data: 3 3 3 3 4 3 3 2 2 3
14056 116/300 Write block 4394 len 32768 data: 5 7 7 6 9 7 7 6
14057 117/300 Read block 3148 len 16384
14058 118/300 Read block 2900 len 12288
14059 119/300 Read block 541 len 20480
14060 120/300 Write block 4676 len 40960 data: 5 6 6 6 6 5 4 2 3 4
14061 121/300 Read block 3599 len 16384
14062 122/300 Read block 665 len 40960
14063 123/300 Write block 3282 len 28672 data: 3 3 3 4 4 5 5
14064 124/300 Read block 194 len 20480
14065 125/300 Write block 481 len 20480 data: 4 3 2 3 3
14066 126/300 Read block 59 len 8192
14067 127/300 Flush
14068 128/300 Read block 2599 len 20480
14069 129/300 Write block 4661 len 40960 data: 4 4 4 4 5 5 6 5 4 4
14070 130/300 Write block 1892 len 36864 data: 4 3 2 2 5 6 6 5 5
14071 131/300 Write block 2385 len 4096 data: 4
14072 132/300 Write block 4824 len 8192 data: 5 6
14073 133/300 Flush
14074 134/300 Write block 1105 len 36864 data: 4 3 4 4 4 4 4 4 3
14075 135/300 Write block 3049 len 28672 data: 6 5 4 3 2 2 3
14076 136/300 Read block 4374 len 36864
14077 137/300 Read block 4559 len 12288
14078 138/300 Write block 3146 len 4096 data: 9
14079 139/300 Write block 907 len 40960 data: 5 5 5 4 4 5 5 5 7 7
14080 140/300 Write block 4542 len 8192 data: 3 3
14081 141/300 Read block 4056 len 20480
14082 142/300 Write block 1909 len 16384 data: 6 5 5 5
14083 143/300 Read block 283 len 8192
14084 144/300 Read block 2966 len 12288
14085 145/300 Write block 128 len 8192 data: 7 7
14086 146/300 Read block 3354 len 4096
14087 147/300 Read block 2712 len 12288
14088 148/300 Read block 1739 len 8192
14089 149/300 Write block 170 len 28672 data: 6 4 5 5 5 6 7
14090 150/300 Flush
14091 151/300 Flush
14092 152/300 Read block 4432 len 40960
14093 153/300 Write block 200 len 28672 data: 3 3 3 3 3 4 5
14094 154/300 Write block 4488 len 36864 data: 5 6 5 4 4 3 2 3 3
14095 155/300 Read block 4187 len 20480
14096 156/300 Read block 906 len 40960
14097 157/300 Read block 2052 len 32768
14098 158/300 Write block 630 len 40960 data: 5 3 2 2 2 2 2 2 2 3
14099 159/300 Write block 2658 len 28672 data: 2 3 3 3 4 4 3
14100 160/300 Read block 1112 len 20480
14101 161/300 Write block 4065 len 36864 data: 5 4 4 4 4 4 4 3 2
14102 162/300 Read block 4862 len 20480
14103 163/300 Read block 3568 len 20480
14104 164/300 Read block 3557 len 8192
14105 165/300 Read block 4257 len 20480
14106 166/300 Write block 4134 len 32768 data: 3 3 3 3 3 4 3 3
14107 167/300 Flush
14108 168/300 Write block 1657 len 36864 data: 4 3 3 4 5 6 7 7 6
14109 169/300 Read block 1430 len 8192
14110 170/300 Read block 116 len 20480
14111 171/300 Read block 480 len 12288
14112 172/300 Write block 1759 len 40960 data: 4 4 5 5 5 5 4 4 5 4
14113 173/300 Read block 3192 len 12288
14114 174/300 Read block 4465 len 20480
14115 175/300 Read block 3195 len 4096
14116 176/300 Read block 4321 len 36864
14117 177/300 Flush
14118 178/300 Read block 3220 len 4096
14119 179/300 Read block 654 len 8192
14120 180/300 Read block 3640 len 40960
14121 181/300 Read block 3928 len 8192
14122 182/300 Read block 1996 len 32768
14123 183/300 Read block 2579 len 32768
14124 184/300 Write block 881 len 20480 data: 3 3 3 3 5
14125 185/300 Flush
14126 186/300 Write block 1678 len 32768 data: 3 3 3 3 3 4 4 3
14127 187/300 Read block 1731 len 28672
14128 188/300 Write block 1625 len 20480 data: 4 4 4 4 4
14129 189/300 Read block 3555 len 4096
14130 190/300 Read block 4180 len 36864
14131 191/300 Write block 4986 len 24576 data: 3 3 3 4 3 2
14132 192/300 Read block 1970 len 16384
14133 193/300 Read block 591 len 16384
14134 194/300 Flush
14135 195/300 Read block 1141 len 20480
14136 196/300 Read block 3357 len 8192
14137 197/300 Write block 3481 len 12288 data: 3 2 2
14138 198/300 Read block 4032 len 12288
14139 199/300 Read block 1521 len 36864
14140 200/300 Read block 187 len 40960
14141 201/300 Write block 3758 len 4096 data: 6
14142 202/300 Write block 4810 len 8192 data: 7 8
14143 203/300 Read block 4094 len 36864
14144 204/300 Read block 4657 len 12288
14145 205/300 Write block 4857 len 40960 data: 5 6 7 8 5 5 4 4 4 4
14146 206/300 Write block 1370 len 20480 data: 7 7 6 6 6
14147 207/300 Read block 1054 len 20480
14148 208/300 Write block 4901 len 28672 data: 3 2 3 3 3 3 3
14149 209/300 Write block 2826 len 28672 data: 5 5 5 4 3 3 3
14150 210/300 Read block 3197 len 24576
14151 211/300 Write block 1675 len 32768 data: 4 3 3 4 4 4 4 4
14152 212/300 Read block 170 len 20480
14153 213/300 Write block 1028 len 8192 data: 3 3
14154 214/300 Write block 2805 len 8192 data: 3 4
14155 215/300 Read block 1497 len 36864
14156 216/300 Read block 4375 len 8192
14157 217/300 Flush
14158 218/300 Read block 1721 len 32768
14159 219/300 Write block 3496 len 40960 data: 6 5 6 6 6 6 6 5 4 4
14160 220/300 Write block 3562 len 16384 data: 2 2 3 3
14161 221/300 Read block 3909 len 20480
14162 222/300 Flush
14163 223/300 Flush
14164 224/300 Read block 4430 len 8192
14165 225/300 Write block 4388 len 20480 data: 5 4 5 5 4
14166 226/300 Write block 2740 len 8192 data: 3 4
14167 227/300 Read block 1282 len 8192
14168 228/300 Read block 3017 len 24576
14169 229/300 Read block 4744 len 16384
14170 230/300 Read block 3741 len 36864
14171 231/300 Read block 3033 len 16384
14172 232/300 Flush
14173 233/300 Write block 4798 len 24576 data: 3 4 7 7 7 10
14174 234/300 Write block 4501 len 32768 data: 6 5 6 6 5 4 5 4
14175 235/300 Flush
14176 236/300 Read block 390 len 32768
14177 237/300 Read block 4619 len 36864
14178 238/300 Read block 2383 len 4096
14179 239/300 Read block 1921 len 12288
14180 240/300 Write block 1402 len 24576 data: 3 4 4 4 4 4
14181 241/300 Write block 573 len 28672 data: 3 3 3 3 3 2 2
14182 242/300 Read block 1556 len 8192
14183 243/300 Write block 688 len 36864 data: 3 3 3 5 5 6 6 4 3
14184 244/300 Write block 4771 len 4096 data: 3
14185 245/300 Write block 4318 len 12288 data: 3 3 3
14186 246/300 Flush
14187 247/300 Read block 1726 len 16384
14188 248/300 Read block 3888 len 40960
14189 249/300 Read block 3126 len 16384
14190 250/300 Read block 2030 len 12288
14191 251/300 Write block 2561 len 20480 data: 9 8 6 5 5
14192 252/300 Read block 1411 len 36864
14193 253/300 Write block 3111 len 4096 data: 6
14194 254/300 Read block 3595 len 12288
14195 255/300 Read block 3438 len 8192
14196 256/300 Read block 4978 len 36864
14197 257/300 Read block 1704 len 20480
14198 258/300 Read block 1646 len 28672
14199 259/300 Read block 2873 len 4096
14200 260/300 Flush
14201 261/300 Write block 4364 len 32768 data: 4 5 5 4 4 4 4 5
14202 262/300 Write block 3218 len 24576 data: 3 3 3 3 3 3
14203 263/300 Read block 4811 len 28672
14204 264/300 Write block 1692 len 28672 data: 6 6 6 7 7 5 3
14205 265/300 Write block 2973 len 12288 data: 3 4 4
14206 266/300 Read block 4839 len 12288
14207 267/300 Read block 3920 len 24576
14208 268/300 Write block 2237 len 20480 data: 5 6 7 8 6
14209 269/300 Read block 2789 len 8192
14210 270/300 Write block 1899 len 8192 data: 6 6
14211 271/300 Write block 756 len 36864 data: 5 4 4 5 5 4 3 3 3
14212 272/300 Read block 2569 len 8192
14213 273/300 Write block 236 len 12288 data: 3 2 3
14214 274/300 Read block 3104 len 36864
14215 275/300 Read block 3968 len 28672
14216 276/300 Read block 3173 len 40960
14217 277/300 Read block 3596 len 8192
14218 278/300 Read block 635 len 20480
14219 279/300 Write block 3937 len 12288 data: 9 9 9
14220 280/300 Flush
14221 281/300 Read block 2437 len 28672
14222 282/300 Read block 2743 len 28672
14223 283/300 Flush
14224 284/300 Write block 1462 len 40960 data: 4 6 6 5 6 5 5 5 5 3
14225 285/300 Read block 169 len 40960
14226 286/300 Read block 2148 len 32768
14227 287/300 Write block 489 len 32768 data: 3 3 5 4 3 4 3 3
14228 288/300 Read block 2358 len 36864
14229 289/300 Read block 542 len 32768
14230 290/300 Write block 4955 len 40960 data: 4 4 4 4 5 6 5 6 6 5
14231 291/300 Flush
14232 292/300 Write block 569 len 4096 data: 3
14233 293/300 Write block 3050 len 8192 data: 6 5
14234 294/300 Write block 3343 len 16384 data: 4 2 2 3
14235 295/300 Write block 2924 len 40960 data: 3 3 3 3 3 3 3 5 5 4
14236 296/300 Write block 4747 len 36864 data: 7 7 6 5 6 5 6 6 8
14237 297/300 Write block 2559 len 16384 data: 8 8 10 9
14238 298/300 Write block 4534 len 40960 data: 3 3 3 3 3 3 4 4 4 4
14239 299/300 Write block 4768 len 40960 data: 4 3 2 4 3 3 3 4 4 6
14240 300/300 Write block 3362 len 32768 data: 3 3 3 4 4 3 3 4
14241 Main task has work to do, go find it
14242 got dsc Start(1)
14243 start 1
14244 Oct 13 18:12:13.642 INFO request completed, latency_us: 294, response_code: 204, uri: /start/cid/1, method: POST, req_id: a7496541-6a2e-4148-bfb9-6da9533a6224, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
14245 Replay: started 1, returned:Ok(())
14246 ----------------------------------------------------------------
14247 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14248 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14249 5450 Acked 6449 Read 5 Done New Done false
14250 5451 Acked 6450 Write 7 Done New Done false
14251 5452 Acked 6451 Write 9 Done New Done false
14252 5453 Acked 6452 Write 5 Done New Done false
14253 5454 Acked 6453 Write 9 Done New Done false
14254 5455 Acked 6454 Write 9 Done New Done false
14255 5456 Acked 6455 Read 9 Done New Done false
14256 5457 Acked 6456 Write 6 Done New Done false
14257 5458 Acked 6457 Write 8 Done New Done false
14258 5459 Acked 6458 Read 9 Done New Done false
14259 5460 Acked 6459 Read 5 Done New Done false
14260 5461 Acked 6460 Read 2 Done New Done false
14261 5462 Acked 6461 Read 7 Done New Done false
14262 5463 Acked 6462 Read 9 Done New Done false
14263 5464 Acked 6463 Flush 0 Done New Done false
14264 5465 Acked 6464 Read 5 Done New Done false
14265 5466 Acked 6465 Write 4 Done New Done false
14266 5467 Acked 6466 Write 9 Done New Done false
14267 5468 Acked 6467 Flush 0 Done New Done false
14268 5469 Acked 6468 Read 4 Done New Done false
14269 5470 Acked 6469 Read 7 Done New Done false
14270 5471 Acked 6470 Write 4 Done New Done false
14271 5472 Acked 6471 Read 3 Done New Done false
14272 5473 Acked 6472 Read 3 Done New Done false
14273 5474 Acked 6473 Read 3 Done New Done false
14274 5475 Acked 6474 Write 9 Done New Done false
14275 5476 Acked 6475 Read 7 Done New Done false
14276 5477 Acked 6476 Write 2 Done New Done false
14277 5478 Acked 6477 Read 10 Done New Done false
14278 5479 Acked 6478 Read 2 Done New Done false
14279 5480 Acked 6479 Write 6 Done New Done false
14280 5481 Acked 6480 Flush 0 Done New Done false
14281 5482 Acked 6481 Read 7 Done New Done false
14282 5483 Acked 6482 Write 5 Done New Done false
14283 5484 Acked 6483 Read 4 Done New Done false
14284 5485 Acked 6484 Write 9 Done New Done false
14285 5486 Acked 6485 Read 6 Done New Done false
14286 5487 Acked 6486 Read 8 Done New Done false
14287 5488 Acked 6487 Write 9 Done New Done false
14288 5489 Acked 6488 Read 4 Done New Done false
14289 5490 Acked 6489 Read 8 Done New Done false
14290 5491 Acked 6490 Read 9 Done New Done false
14291 5492 Acked 6491 Write 10 Done New Done false
14292 5493 Acked 6492 Read 8 Done New Done false
14293 5494 Acked 6493 Read 9 Done New Done false
14294 5495 Acked 6494 Read 2 Done New Done false
14295 5496 Acked 6495 Flush 0 Done New Done false
14296 5497 Acked 6496 Write 3 Done New Done false
14297 5498 Acked 6497 Read 3 Done New Done false
14298 5499 Acked 6498 Read 2 Done New Done false
14299 5500 Acked 6499 Read 8 Done New Done false
14300 5501 Acked 6500 Flush 0 Done New Done false
14301 5502 Acked 6501 Read 9 Done New Done false
14302 5503 Acked 6502 Write 7 Done New Done false
14303 5504 Acked 6503 Write 7 Done New Done false
14304 5505 Acked 6504 Read 2 Done New Done false
14305 5506 Acked 6505 Read 5 Done New Done false
14306 5507 Acked 6506 Read 10 Done New Done false
14307 5508 Acked 6507 Write 10 Done New Done false
14308 5509 Acked 6508 Flush 0 Done New Done false
14309 5510 Acked 6509 Read 8 Done New Done false
14310 5511 Acked 6510 Flush 0 Done New Done false
14311 5512 Acked 6511 Read 2 Done New Done false
14312 5513 Acked 6512 Read 6 Done New Done false
14313 5514 Acked 6513 Write 9 Done New Done false
14314 5515 Acked 6514 Write 2 Done New Done false
14315 5516 Acked 6515 Write 5 Done New Done false
14316 5517 Acked 6516 Write 7 Done New Done false
14317 5518 Acked 6517 Read 8 Done New Done false
14318 5519 Acked 6518 Write 7 Done New Done false
14319 5520 Acked 6519 Flush 0 Done New Done false
14320 5521 Acked 6520 Write 3 Done New Done false
14321 5522 Acked 6521 Read 4 Done New Done false
14322 5523 Acked 6522 Read 7 Done New Done false
14323 5524 Acked 6523 Write 6 Done New Done false
14324 5525 Acked 6524 Write 3 Done New Done false
14325 5526 Acked 6525 Write 9 Done New Done false
14326 5527 Acked 6526 Read 2 Done New Done false
14327 5528 Acked 6527 Read 9 Done New Done false
14328 5529 Acked 6528 Flush 0 Done New Done false
14329 5530 Acked 6529 Write 10 Done New Done false
14330 5531 Acked 6530 Read 5 Done New Done false
14331 5532 Acked 6531 Read 5 Done New Done false
14332 5533 Acked 6532 Read 4 Done New Done false
14333 5534 Acked 6533 Write 9 Done New Done false
14334 5535 Acked 6534 Read 2 Done New Done false
14335 5536 Acked 6535 Read 5 Done New Done false
14336 5537 Acked 6536 Read 5 Done New Done false
14337 5538 Acked 6537 Write 3 Done New Done false
14338 5539 Acked 6538 Read 4 Done New Done false
14339 5540 Acked 6539 Write 4 Done New Done false
14340 5541 Acked 6540 Read 10 Done New Done false
14341 5542 Acked 6541 Read 1 Done New Done false
14342 5543 Acked 6542 Read 4 Done New Done false
14343 5544 Acked 6543 Write 4 Done New Done false
14344 5545 Acked 6544 Read 7 Done New Done false
14345 5546 Acked 6545 Read 7 Done New Done false
14346 5547 Acked 6546 Read 2 Done New Done false
14347 5548 Acked 6547 Read 3 Done New Done false
14348 5549 Acked 6548 Write 10 Done New Done false
14349 5550 Acked 6549 Write 7 Done New Done false
14350 5551 Acked 6550 Write 4 Done New Done false
14351 5552 Acked 6551 Flush 0 Done New Done false
14352 5553 Acked 6552 Write 6 Done New Done false
14353 5554 Acked 6553 Write 6 Done New Done false
14354 5555 Acked 6554 Read 3 Done New Done false
14355 5556 Acked 6555 Read 10 Done New Done false
14356 5557 Acked 6556 Read 8 Done New Done false
14357 5558 Acked 6557 Read 1 Done New Done false
14358 5559 Acked 6558 Read 9 Done New Done false
14359 5560 Acked 6559 Write 1 Done New Done false
14360 5561 Acked 6560 Flush 0 Done New Done false
14361 5562 Acked 6561 Read 2 Done New Done false
14362 5563 Acked 6562 Write 6 Done New Done false
14363 5564 Acked 6563 Write 10 Done New Done false
14364 5565 Acked 6564 Write 8 Done New Done false
14365 5566 Acked 6565 Read 4 Done New Done false
14366 5567 Acked 6566 Read 3 Done New Done false
14367 5568 Acked 6567 Read 5 Done New Done false
14368 5569 Acked 6568 Write 10 Done New Done false
14369 5570 Acked 6569 Read 4 Done New Done false
14370 5571 Acked 6570 Read 10 Done New Done false
14371 5572 Acked 6571 Write 7 Done New Done false
14372 5573 Acked 6572 Read 5 Done New Done false
14373 5574 Acked 6573 Write 5 Done New Done false
14374 5575 Acked 6574 Read 2 Done New Done false
14375 5576 Acked 6575 Flush 0 Done New Done false
14376 5577 Acked 6576 Read 5 Done New Done false
14377 5578 Acked 6577 Write 10 Done New Done false
14378 5579 Acked 6578 Write 9 Done New Done false
14379 5580 Acked 6579 Write 1 Done New Done false
14380 5581 Acked 6580 Write 2 Done New Done false
14381 5582 Acked 6581 Flush 0 Done New Done false
14382 5583 Acked 6582 Write 9 Done New Done false
14383 5584 Acked 6583 Write 7 Done New Done false
14384 5585 Acked 6584 Read 9 Done New Done false
14385 5586 Acked 6585 Read 3 Done New Done false
14386 5587 Acked 6586 Write 1 Done New Done false
14387 5588 Acked 6587 Write 10 Done New Done false
14388 5589 Acked 6588 Write 2 Done New Done false
14389 5590 Acked 6589 Read 5 Done New Done false
14390 5591 Acked 6590 Write 4 Done New Done false
14391 5592 Acked 6591 Read 2 Done New Done false
14392 5593 Acked 6592 Read 3 Done New Done false
14393 5594 Acked 6593 Write 2 Done New Done false
14394 5595 Acked 6594 Read 1 Done New Done false
14395 5596 Acked 6595 Read 3 Done New Done false
14396 5597 Acked 6596 Read 2 Done New Done false
14397 5598 Acked 6597 Write 7 Done New Done false
14398 5599 Acked 6598 Flush 0 Done New Done false
14399 5600 Acked 6599 Flush 0 Done New Done false
14400 5601 Acked 6600 Read 10 Done New Done false
14401 5602 Acked 6601 Write 7 Done New Done false
14402 5603 Acked 6602 Write 9 Done New Done false
14403 5604 Acked 6603 Read 5 Done New Done false
14404 5605 Acked 6604 Read 10 Done New Done false
14405 5606 Acked 6605 Read 8 Done New Done false
14406 5607 Acked 6606 Write 10 Done New Done false
14407 5608 Acked 6607 Write 7 Done New Done false
14408 5609 Acked 6608 Read 5 Done New Done false
14409 5610 Acked 6609 Write 9 Done New Done false
14410 5611 Acked 6610 Read 5 Done New Done false
14411 5612 Acked 6611 Read 5 Done New Done false
14412 5613 Acked 6612 Read 2 Done New Done false
14413 5614 Acked 6613 Read 5 Done New Done false
14414 5615 Acked 6614 Write 8 Done New Done false
14415 5616 Acked 6615 Flush 0 Done New Done false
14416 5617 Acked 6616 Write 9 Done New Done false
14417 5618 Acked 6617 Read 2 Done New Done false
14418 5619 Acked 6618 Read 5 Done New Done false
14419 5620 Acked 6619 Read 3 Done New Done false
14420 5621 Acked 6620 Write 10 Done New Done false
14421 5622 Acked 6621 Read 3 Done New Done false
14422 5623 Acked 6622 Read 5 Done New Done false
14423 5624 Acked 6623 Read 1 Done New Done false
14424 5625 Acked 6624 Read 9 Done New Done false
14425 5626 Acked 6625 Flush 0 Done New Done false
14426 5627 Acked 6626 Read 1 Done New Done false
14427 5628 Acked 6627 Read 2 Done New Done false
14428 5629 Acked 6628 Read 10 Done New Done false
14429 5630 Acked 6629 Read 2 Done New Done false
14430 5631 Acked 6630 Read 8 Done New Done false
14431 5632 Acked 6631 Read 8 Done New Done false
14432 5633 Acked 6632 Write 5 Done New Done false
14433 5634 Acked 6633 Flush 0 Done New Done false
14434 5635 Acked 6634 Write 8 Done New Done false
14435 5636 Acked 6635 Read 7 Done New Done false
14436 5637 Acked 6636 Write 5 Done New Done false
14437 5638 Acked 6637 Read 1 Done New Done false
14438 5639 Acked 6638 Read 9 Done New Done false
14439 5640 Acked 6639 Write 6 Done New Done false
14440 5641 Acked 6640 Read 4 Done New Done false
14441 5642 Acked 6641 Read 4 Done New Done false
14442 5643 Acked 6642 Flush 0 Done New Done false
14443 5644 Acked 6643 Read 5 Done New Done false
14444 5645 Acked 6644 Read 2 Done New Done false
14445 5646 Acked 6645 Write 3 Done New Done false
14446 5647 Acked 6646 Read 3 Done New Done false
14447 5648 Acked 6647 Read 9 Done New Done false
14448 5649 Acked 6648 Read 10 Done New Done false
14449 5650 Acked 6649 Write 1 Done New Done false
14450 5651 Acked 6650 Write 2 Done New Done false
14451 5652 Acked 6651 Read 9 Done New Done false
14452 5653 Acked 6652 Read 3 Done New Done false
14453 5654 Acked 6653 Write 10 Done New Done false
14454 5655 Acked 6654 Write 5 Done New Done false
14455 5656 Acked 6655 Read 5 Done New Done false
14456 5657 Acked 6656 Write 7 Done New Done false
14457 5658 Acked 6657 Write 7 Done New Done false
14458 5659 Acked 6658 Read 6 Done New Done false
14459 5660 Acked 6659 Write 8 Done New Done false
14460 5661 Acked 6660 Read 5 Done New Done false
14461 5662 Acked 6661 Write 2 Done New Done false
14462 5663 Acked 6662 Write 2 Done New Done false
14463 5664 Acked 6663 Read 9 Done New Done false
14464 5665 Acked 6664 Read 2 Done New Done false
14465 5666 Acked 6665 Flush 0 Done New Done false
14466 5667 Acked 6666 Read 8 Done New Done false
14467 5668 Acked 6667 Write 10 Done New Done false
14468 5669 Acked 6668 Write 4 Done New Done false
14469 5670 Acked 6669 Read 5 Done New Done false
14470 5671 Acked 6670 Flush 0 Done New Done false
14471 5672 Acked 6671 Flush 0 Done New Done false
14472 5673 Acked 6672 Read 2 Done New Done false
14473 5674 Acked 6673 Write 5 Done New Done false
14474 5675 Acked 6674 Write 2 Done New Done false
14475 5676 Acked 6675 Read 2 Done New Done false
14476 5677 Acked 6676 Read 6 Done New Done false
14477 5678 Acked 6677 Read 4 Done New Done false
14478 5679 Acked 6678 Read 9 Done New Done false
14479 5680 Acked 6679 Read 4 Done New Done false
14480 5681 Acked 6680 Flush 0 Done New Done false
14481 5682 Acked 6681 Write 6 Done New Done false
14482 5683 Acked 6682 Write 8 Done New Done false
14483 5684 Acked 6683 Flush 0 Done New Done false
14484 5685 Acked 6684 Read 8 Done New Done false
14485 5686 Acked 6685 Flush 0 Done New Done false
14486 5687 Acked 6686 Read 9 Done New Done false
14487 5688 Acked 6687 Read 1 Done New Done false
14488 5689 Acked 6688 Read 3 Done New Done false
14489 5690 Acked 6689 Write 6 Done New Done false
14490 5691 Acked 6690 Write 7 Done New Done false
14491 5692 Acked 6691 Read 2 Done New Done false
14492 5693 Acked 6692 Write 9 Done New Done false
14493 5694 Acked 6693 Write 1 Done New Done false
14494 5695 Acked 6694 Write 3 Done New Done false
14495 5696 Acked 6695 Flush 0 Done New Done false
14496 5697 Acked 6696 Read 4 Done New Done false
14497 5698 Acked 6697 Read 10 Done New Done false
14498 5699 Acked 6698 Read 4 Done New Done false
14499 5700 Acked 6699 Read 3 Done New Done false
14500 5701 Acked 6700 Write 5 Done New Done false
14501 5702 Acked 6701 Read 9 Done New Done false
14502 5703 Acked 6702 Write 1 Done New Done false
14503 5704 Acked 6703 Read 3 Done New Done false
14504 5705 Acked 6704 Read 2 Done New Done false
14505 5706 Acked 6705 Read 9 Done New Done false
14506 5707 Acked 6706 Read 5 Done New Done false
14507 5708 Acked 6707 Read 7 Done New Done false
14508 5709 Acked 6708 Read 1 Done New Done false
14509 5710 Acked 6709 Flush 0 Done New Done false
14510 5711 Acked 6710 Write 8 Done New Done false
14511 5712 Acked 6711 Write 6 Done New Done false
14512 5713 Acked 6712 Read 7 Done New Done false
14513 5714 Acked 6713 Write 7 Done New Done false
14514 5715 Acked 6714 Write 3 Done New Done false
14515 5716 Acked 6715 Read 3 Done New Done false
14516 5717 Acked 6716 Read 6 Done New Done false
14517 5718 Acked 6717 Write 5 Done New Done false
14518 5719 Acked 6718 Read 2 Done New Done false
14519 5720 Acked 6719 Write 2 Done New Done false
14520 5721 Acked 6720 Write 9 Done New Done false
14521 5722 Acked 6721 Read 2 Done New Done false
14522 5723 Acked 6722 Write 3 Done New Done false
14523 5724 Acked 6723 Read 9 Done New Done false
14524 5725 Acked 6724 Read 7 Done New Done false
14525 5726 Acked 6725 Read 10 Done New Done false
14526 5727 Acked 6726 Read 2 Done New Done false
14527 5728 Acked 6727 Read 5 Done New Done false
14528 5729 Acked 6728 Write 3 Done New Done false
14529 5730 Acked 6729 Flush 0 Done New Done false
14530 5731 Acked 6730 Read 7 Done New Done false
14531 5732 Acked 6731 Read 7 Done New Done false
14532 5733 Acked 6732 Flush 0 Done New Done false
14533 5734 Acked 6733 Write 10 Done New Done false
14534 5735 Acked 6734 Read 10 Done New Done false
14535 5736 Acked 6735 Read 8 Done New Done false
14536 5737 Acked 6736 Write 8 Done New Done false
14537 5738 Acked 6737 Read 9 Done New Done false
14538 5739 Acked 6738 Read 8 Done New Done false
14539 5740 Acked 6739 Write 10 Done New Done false
14540 5741 Acked 6740 Flush 0 Done New Done false
14541 5742 Acked 6741 Write 1 Done New Done false
14542 5743 Acked 6742 Write 2 Done New Done false
14543 5744 Acked 6743 Write 4 Done New Done false
14544 5745 Acked 6744 Write 10 Sent New Sent false
14545 5746 Acked 6745 Write 9 Sent New Sent false
14546 5747 Acked 6746 Write 4 Sent New Sent false
14547 5748 Acked 6747 Write 10 Sent New Sent false
14548 5749 Acked 6748 Write 10 Sent New Sent false
14549 5750 Acked 6749 Write 8 Sent New Sent false
14550 STATES DS:0 DS:1 DS:2 TOTAL
14551 New 0 301 0 301
14552 Sent 6 0 6 12
14553 Done 295 0 295 590
14554 Skipped 0 0 0 0
14555 Error 0 0 0 0
14556 Last Flush: 6740 6448 6740
14557 Downstairs last five completed: 6448 6447 6446 6445 6444 6443
14558 Upstairs last five completed: 5750 5749 5748 5747 5746 5745
14559 CLIENT: Up:0 ds:301 act:2
14560 [8820] got start action
14561 [8820] I am going to restart
14562 Starting downstairs at port 8820
14563 Make output file at "/tmp/dsc/downstairs-8820.txt"
14564 [8820][1] reports Starting
14565 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1195)
14566 [8820][1] reports Running
145672023-10-13T18:12:15.681ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 1
145682023-10-13T18:12:15.681ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
145692023-10-13T18:12:15.681ZINFOcrucible: [1] client re-new 302 jobs since flush 6448 = downstairs
145702023-10-13T18:12:15.682ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
145712023-10-13T18:12:15.683ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
145722023-10-13T18:12:15.683ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
145732023-10-13T18:12:15.683ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
145742023-10-13T18:12:15.683ZINFOcrucible: [1] send last flush ID to this DS: 6448
145752023-10-13T18:12:15.683ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
145762023-10-13T18:12:15.683ZINFOcrucible: [1] Replied this last flush ID: 6448
145772023-10-13T18:12:15.683ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Offline Active ds_transition to Replay
145782023-10-13T18:12:15.683ZINFOcrucible: [1] Transition from Offline to Replay
145792023-10-13T18:12:15.683ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
145802023-10-13T18:12:15.683ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Replay Active ds_transition to Active
145812023-10-13T18:12:15.683ZINFOcrucible: [1] Transition from Replay to Active
145822023-10-13T18:12:15.683ZINFOcrucible: [1] Starts cmd_loop
145832023-10-13T18:12:15.683ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
145842023-10-13T18:12:15.684ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
145852023-10-13T18:12:15.684ZINFOcrucible: Set check for repair
145862023-10-13T18:12:16.685ZINFOcrucible: Checking if live repair is needed
145872023-10-13T18:12:16.685ZINFOcrucible: No Live Repair required at this time
145882023-10-13T18:12:16.685ZWARNcrucible: [1] flow control end
14589 ----------------------------------------------------------------
14590 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14591 Downstairs last five completed: 6750 6749 6748 6747 6746 6745
14592 Upstairs last five completed: 5751 5750 5749 5748 5747 5746
14593 CLIENT: Up:0 ds:0 act:3
14594 Replay: All jobs finished, all DS active.
14595 Main task has work to do, go find it
14596 got dsc Stop(2)
14597 stop 2
14598 Oct 13 18:12:17.647 INFO request completed, latency_us[8830] Got stop action so:false kr:false
14599 : 519, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 5db08aad-d926-4dac-b6f4-d44ea66bd250, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
14600 Oct 13 18:12:17.649 INFO request completed, latency_us: 181, response_code: 200, uri: /state/cid/2, method: GET, req_id: 17c12812-21c2-4d1c-9589-494ec36000b3, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
146012023-10-13T18:12:17.649ZWARNcrucible: [2] downstairs disconnected
146022023-10-13T18:12:17.650ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
146032023-10-13T18:12:17.650ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8830 closed looper = 2
146042023-10-13T18:12:17.650ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
146052023-10-13T18:12:17.650ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
146062023-10-13T18:12:17.650ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
146072023-10-13T18:12:17.650ZWARNcrucible: [2] pm_task rx.recv() is None
14608 [8830] Exited with: ExitStatus(unix_wait_status(9))
14609 [8830][2] reports Exit
14610 Oct 13 18:12:21.652 INFO request completed, latency_us: 257, response_code: 200, uri: /state/cid/2, method: GET, req_id: 52e5c5a4-b188-4534-a45c-a3c06498bf62, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
14611 001/300 Read block 518 len 12288
146122023-10-13T18:12:21.655ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
14613 002/300 Write block 1803 len 40960 data: 4 4 5 5 4 4 7 6 6 7
14614 003/300 Read block 4715 len 40960
14615 004/300 Read block 4436 len 24576
14616 005/300 Read block 4844 len 36864
14617 006/300 Read block 4158 len 32768
14618 007/300 Flush
14619 008/300 Read block 3749 len 16384
14620 009/300 Write block 3179 len 24576 data: 7 5 5 5 5 4
14621 010/300 Write block 2507 len 28672 data: 5 7 6 4 4 4 3
14622 011/300 Write block 1743 len 40960 data: 6 6 5 3 4 4 3 4 4 4
14623 012/300 Read block 118 len 40960
14624 013/300 Read block 3923 len 24576
14625 014/300 Write block 1524 len 24576 data: 4 5 5 5 5 4
14626 015/300 Write block 1304 len 12288 data: 2 2 2
14627 016/300 Write block 2260 len 24576 data: 6 4 4 3 2 2
14628 017/300 Read block 4421 len 32768
14629 018/300 Read block 1057 len 40960
14630 019/300 Read block 1580 len 4096
14631 020/300 Write block 4622 len 20480 data: 5 5 5 4 4
14632 021/300 Flush
14633 022/300 Read block 164 len 28672
14634 023/300 Write block 377 len 24576 data: 4 6 6 5 5 6
14635 024/300 Read block 3259 len 8192
14636 025/300 Read block 3540 len 8192
14637 026/300 Write block 1754 len 24576 data: 6 6 5 3 4 5
14638 027/300 Write block 4151 len 28672 data: 6 7 6 6 5 5 5
14639 028/300 Write block 705 len 12288 data: 6 5 4
14640 029/300 Write block 2145 len 20480 data: 6 6 5 4 4
14641 030/300 Write block 4463 len 12288 data: 2 2 2
14642 031/300 Read block 3256 len 4096
14643 032/300 Write block 2597 len 40960 data: 3 3 5 5 8 10 9 7 8 7
14644 033/300 Read block 878 len 32768
14645 034/300 Write block 2195 len 4096 data: 7
14646 035/300 Read block 1841 len 32768
14647 036/300 Read block 3868 len 8192
14648 037/300 Read block 983 len 24576
14649 038/300 Read block 1135 len 24576
14650 039/300 Write block 3145 len 12288 data: 9 10 8
14651 040/300 Write block 1215 len 28672 data: 6 4 4 3 3 4 5
14652 041/300 Write block 2580 len 36864 data: 4 5 5 4 4 6 6 5 6
14653 042/300 Read block 778 len 32768
14654 043/300 Flush
14655 044/300 Read block 207 len 36864
14656 045/300 Read block 4397 len 40960
14657 046/300 Write block 4448 len 16384 data: 5 3 3 3
14658 047/300 Write block 4742 len 28672 data: 6 8 9 9 8 8 8
14659 048/300 Read block 3599 len 36864
14660 049/300 Write block 4447 len 36864 data: 5 6 4 4 4 3 4 4 4
14661 050/300 Read block 3050 len 28672
14662 051/300 Flush
14663 052/300 Write block 3433 len 8192 data: 7 6
14664 053/300 Flush
14665 054/300 Write block 1663 len 36864 data: 8 8 7 6 5 7 6 6 5
14666 055/300 Write block 1196 len 20480 data: 5 5 6 5 5
14667 056/300 Read block 4726 len 24576
14668 057/300 Read block 20 len 12288
14669 058/300 Write block 278 len 4096 data: 5
14670 059/300 Flush
14671 060/300 Read block 4102 len 40960
14672 061/300 Write block 4506 len 32768 data: 5 6 5 6 6 7 7 7
14673 062/300 Flush
14674 063/300 Read block 27 len 28672
14675 064/300 Flush
14676 065/300 Write block 4890 len 20480 data: 5 4 5 4 3
14677 066/300 Read block 353 len 8192
14678 067/300 Read block 2388 len 36864
14679 068/300 Read block 3297 len 4096
14680 069/300 Read block 1285 len 28672
14681 070/300 Read block 4528 len 32768
14682 071/300 Write block 41 len 40960 data: 3 2 3 4 4 5 7 9 6 6
14683 072/300 Write block 2463 len 32768 data: 6 6 6 6 5 4 5 6
14684 073/300 Read block 524 len 32768
14685 074/300 Read block 1348 len 12288
14686 075/300 Write block 3160 len 16384 data: 6 5 4 4
14687 076/300 Read block 3855 len 4096
14688 077/300 Read block 1598 len 8192
14689 078/300 Write block 595 len 24576 data: 5 5 6 7 7 8
14690 079/300 Read block 2413 len 4096
14691 080/300 Write block 2998 len 32768 data: 7 7 7 6 6 7 7 7
14692 081/300 Read block 3668 len 20480
14693 082/300 Read block 1979 len 36864
14694 083/300 Write block 509 len 28672 data: 4 3 3 4 4 4 4
14695 084/300 Write block 4449 len 16384 data: 5 5 5 4
14696 085/300 Read block 2982 len 36864
14697 086/300 Read block 2954 len 24576
14698 087/300 Read block 973 len 24576
14699 088/300 Write block 4106 len 16384 data: 4 4 3 2
14700 089/300 Read block 2368 len 40960
14701 090/300 Write block 3511 len 40960 data: 6 5 4 6 6 6 5 7 7 8
14702 091/300 Write block 1730 len 12288 data: 5 5 6
14703 092/300 Read block 1028 len 40960
14704 093/300 Write block 452 len 40960 data: 7 4 4 4 4 6 6 5 6 8
14705 094/300 Read block 981 len 32768
14706 095/300 Write block 4624 len 28672 data: 6 5 5 5 5 5 4
14707 096/300 Read block 4423 len 40960
14708 097/300 Write block 1527 len 12288 data: 6 6 5
14709 098/300 Read block 3228 len 16384
14710 099/300 Write block 1622 len 28672 data: 4 3 4 5 5 5 5
14711 100/300 Read block 3302 len 28672
14712 101/300 Read block 749 len 24576
14713 102/300 Write block 2840 len 20480 data: 4 3 3 3 3
14714 103/300 Write block 788 len 8192 data: 3 2
14715 104/300 Read block 3892 len 36864
14716 105/300 Read block 4695 len 8192
14717 106/300 Write block 676 len 4096 data: 3
14718 107/300 Read block 2094 len 4096
14719 108/300 Write block 4053 len 4096 data: 2
14720 109/300 Read block 4621 len 12288
14721 110/300 Write block 1062 len 28672 data: 5 6 5 5 5 6 7
14722 111/300 Read block 1592 len 20480
14723 112/300 Write block 1415 len 16384 data: 4 4 4 4
14724 113/300 Read block 3025 len 40960
14725 114/300 Write block 4283 len 24576 data: 7 7 6 5 4 4
14726 115/300 Read block 1034 len 4096
14727 116/300 Read block 4345 len 4096
14728 117/300 Write block 3020 len 12288 data: 5 5 4
14729 118/300 Write block 413 len 4096 data: 3
14730 119/300 Write block 1186 len 28672 data: 4 3 3 3 4 4 4
14731 120/300 Write block 589 len 28672 data: 4 6 4 4 5 4 6
14732 121/300 Write block 3551 len 4096 data: 4
14733 122/300 Read block 2046 len 4096
14734 123/300 Read block 4385 len 4096
14735 124/300 Read block 470 len 16384
14736 125/300 Read block 704 len 28672
14737 126/300 Write block 4604 len 40960 data: 4 4 5 4 6 4 4 4 5 5
14738 127/300 Read block 3978 len 12288
14739 128/300 Read block 3230 len 20480
14740 129/300 Write block 4043 len 28672 data: 5 3 4 3 3 3 4
14741 130/300 Flush
14742 131/300 Write block 747 len 40960 data: 4 3 3 3 3 4 5 4 5 6
14743 132/300 Read block 2358 len 20480
14744 133/300 Read block 955 len 8192
14745 134/300 Read block 3161 len 8192
14746 135/300 Read block 3288 len 24576
14747 136/300 Read block 3031 len 36864
14748 137/300 Flush
14749 138/300 Read block 1588 len 8192
14750 139/300 Flush
14751 140/300 Write block 1578 len 20480 data: 3 3 2 2 2
14752 141/300 Read block 1596 len 20480
14753 142/300 Read block 3736 len 24576
14754 143/300 Read block 3621 len 16384
14755 144/300 Read block 1735 len 32768
14756 145/300 Write block 2023 len 4096 data: 5
14757 146/300 Flush
14758 147/300 Read block 2230 len 40960
14759 148/300 Flush
14760 149/300 Write block 4459 len 4096 data: 4
14761 150/300 Read block 3676 len 24576
14762 151/300 Write block 4978 len 16384 data: 5 6 6 7
14763 152/300 Write block 3567 len 28672 data: 4 5 7 5 5 5 6
14764 153/300 Read block 1323 len 12288
14765 154/300 Read block 2261 len 32768
14766 155/300 Read block 680 len 32768
14767 156/300 Write block 3654 len 20480 data: 4 4 4 3 2
14768 157/300 Write block 2142 len 16384 data: 4 6 7 7
14769 158/300 Write block 4458 len 36864 data: 4 5 3 4 4 3 3 3 2
14770 159/300 Read block 3886 len 32768
14771 160/300 Write block 1350 len 24576 data: 3 3 3 4 5 4
14772 161/300 Write block 884 len 28672 data: 4 6 6 6 7 6 6
14773 162/300 Write block 385 len 40960 data: 5 5 5 4 4 4 3 2 2 2
14774 163/300 Write block 4297 len 8192 data: 5 5
14775 164/300 Read block 4544 len 36864
14776 165/300 Write block 2342 len 8192 data: 3 4
14777 166/300 Flush
14778 167/300 Read block 2070 len 36864
14779 168/300 Read block 4837 len 28672
14780 169/300 Read block 3271 len 4096
14781 170/300 Write block 721 len 40960 data: 3 4 3 3 3 3 3 2 2 2
14782 171/300 Read block 3539 len 12288
14783 172/300 Write block 1112 len 36864 data: 5 4 4 3 5 5 5 5 4
14784 173/300 Write block 4120 len 12288 data: 4 4 4
14785 174/300 Write block 3092 len 40960 data: 8 9 9 10 8 7 6 6 6 5
14786 175/300 Write block 204 len 36864 data: 4 5 6 4 5 6 6 6 7
14787 176/300 Read block 876 len 4096
14788 177/300 Read block 3906 len 36864
14789 178/300 Read block 1430 len 32768
14790 179/300 Read block 3386 len 12288
14791 180/300 Read block 202 len 32768
14792 181/300 Read block 2501 len 12288
14793 182/300 Write block 4011 len 36864 data: 2 3 4 5 6 5 4 4 4
14794 183/300 Read block 4039 len 16384
14795 184/300 Read block 4616 len 4096
14796 185/300 Read block 1889 len 16384
14797 186/300 Write block 2176 len 28672 data: 4 3 3 3 2 3 3
14798 187/300 Write block 4343 len 28672 data: 4 3 5 5 6 4 3
14799 188/300 Read block 882 len 32768
14800 189/300 Write block 1290 len 12288 data: 3 3 3
14801 190/300 Write block 360 len 28672 data: 3 3 4 4 4 4 4
14802 191/300 Write block 3432 len 32768 data: 7 8 7 5 5 3 2 2
14803 192/300 Write block 2791 len 20480 data: 4 4 4 4 3
14804 193/300 Read block 2572 len 28672
14805 194/300 Read block 2613 len 20480
14806 195/300 Read block 969 len 20480
14807 196/300 Read block 1109 len 24576
14808 197/300 Read block 4689 len 8192
14809 198/300 Read block 1412 len 20480
14810 199/300 Read block 1831 len 12288
14811 200/300 Read block 3360 len 4096
14812 201/300 Write block 972 len 24576 data: 3 3 3 4 4 4
14813 202/300 Read block 303 len 24576
14814 203/300 Read block 4610 len 12288
14815 204/300 Write block 1186 len 20480 data: 5 4 4 4 5
14816 205/300 Read block 844 len 8192
14817 206/300 Read block 1655 len 16384
14818 207/300 Read block 2277 len 32768
14819 208/300 Write block 914 len 36864 data: 6 8 8 8 8 8 8 5 7
14820 209/300 Read block 509 len 4096
14821 210/300 Write block 4681 len 24576 data: 6 5 3 4 5 4
14822 211/300 Write block 283 len 16384 data: 6 6 6 6
14823 212/300 Read block 3276 len 12288
14824 213/300 Write block 1128 len 12288 data: 5 5 4
14825 214/300 Write block 151 len 8192 data: 5 5
14826 215/300 Write block 2125 len 12288 data: 5 4 5
14827 216/300 Read block 3715 len 20480
14828 217/300 Read block 4992 len 20480
14829 218/300 Write block 3236 len 24576 data: 7 5 5 5 6 5
14830 219/300 Read block 224 len 20480
14831 220/300 Read block 1113 len 32768
14832 221/300 Read block 1609 len 12288
14833 222/300 Read block 811 len 36864
14834 223/300 Write block 3433 len 8192 data: 9 8
14835 224/300 Read block 3273 len 28672
14836 225/300 Write block 194 len 12288 data: 4 3 3
14837 226/300 Read block 3305 len 12288
14838 227/300 Read block 2112 len 16384
14839 228/300 Read block 55 len 28672
14840 229/300 Read block 4903 len 12288
14841 230/300 Write block 1940 len 28672 data: 6 5 3 3 3 3 3
14842 231/300 Read block 3730 len 36864
14843 232/300 Read block 4456 len 24576
14844 233/300 Flush
14845 234/300 Write block 2677 len 24576 data: 5 6 6 5 5 6
14846 235/300 Read block 4765 len 4096
14847 236/300 Write block 3917 len 8192 data: 7 6
14848 237/300 Read block 2684 len 40960
14849 238/300 Write block 834 len 24576 data: 4 4 4 5 5 5
14850 239/300 Read block 3867 len 24576
14851 240/300 Read block 48 len 4096
14852 241/300 Flush
14853 242/300 Read block 2350 len 16384
14854 243/300 Read block 3141 len 8192
14855 244/300 Read block 3027 len 24576
14856 245/300 Write block 2243 len 16384 data: 9 9 7 6
14857 246/300 Read block 1636 len 4096
14858 247/300 Write block 58 len 12288 data: 2 2 2
14859 248/300 Read block 4688 len 36864
14860 249/300 Read block 641 len 20480
14861 250/300 Write block 2659 len 8192 data: 4 4
14862 251/300 Write block 2918 len 24576 data: 5 4 5 5 4 4
14863 252/300 Write block 3745 len 36864 data: 5 6 5 4 4 5 4 5 5
14864 253/300 Write block 3376 len 32768 data: 3 3 3 4 4 4 4 6
14865 254/300 Read block 3057 len 24576
14866 255/300 Write block 3801 len 36864 data: 2 4 4 5 6 5 5 5 4
14867 256/300 Read block 3261 len 16384
14868 257/300 Write block 3169 len 24576 data: 3 3 3 4 5 4
14869 258/300 Read block 3932 len 4096
14870 259/300 Read block 3988 len 28672
14871 260/300 Flush
14872 261/300 Write block 207 len 4096 data: 5
14873 262/300 Write block 2473 len 20480 data: 7 6 6 4 4
14874 263/300 Read block 3601 len 12288
14875 264/300 Flush
14876 265/300 Read block 845 len 16384
14877 266/300 Read block 2679 len 36864
14878 267/300 Write block 3329 len 36864 data: 4 6 5 5 5 4 4 4 4
14879 268/300 Write block 1230 len 40960 data: 4 4 4 3 5 5 4 4 5 5
14880 269/300 Write block 3542 len 36864 data: 6 5 5 6 5 5 5 4 4
14881 270/300 Read block 1812 len 28672
14882 271/300 Write block 1548 len 24576 data: 2 2 2 3 3 4
14883 272/300 Read block 4484 len 40960
14884 273/300 Read block 2267 len 4096
14885 274/300 Write block 4094 len 8192 data: 2 2
14886 275/300 Read block 2771 len 8192
14887 276/300 Write block 2530 len 28672 data: 5 6 6 5 3 3 4
14888 277/300 Write block 4906 len 4096 data: 4
14889 278/300 Write block 350 len 12288 data: 5 5 4
14890 279/300 Read block 4125 len 28672
14891 280/300 Write block 3984 len 32768 data: 5 4 5 6 5 6 4 5
14892 281/300 Read block 423 len 32768
14893 282/300 Write block 3750 len 28672 data: 6 5 6 6 5 5 6
14894 283/300 Flush
14895 284/300 Read block 4390 len 16384
14896 285/300 Write block 990 len 40960 data: 3 3 4 5 5 5 4 5 4 4
14897 286/300 Write block 2729 len 32768 data: 3 3 4 5 7 6 6 6
14898 287/300 Write block 15 len 28672 data: 4 4 4 3 4 6 7
14899 288/300 Write block 4351 len 28672 data: 2 2 4 5 3 3 3
14900 289/300 Flush
14901 290/300 Write block 850 len 32768 data: 5 6 7 5 5 4 3 3
14902 291/300 Read block 617 len 16384
14903 292/300 Read block 4522 len 40960
14904 293/300 Read block 1524 len 16384
14905 294/300 Write block 1100 len 32768 data: 10 7 7 5 5 5 4 5
14906 295/300 Write block 1789 len 36864 data: 7 5 5 6 5 4 5 5 4
14907 296/300 Read block 1857 len 32768
14908 297/300 Write block 4337 len 20480 data: 6 5 5 4 5
14909 298/300 Write block 4236 len 8192 data: 3 3
14910 299/300 Read block 216 len 36864
14911 300/300 Write block 4797 len 8192 data: 5 4
14912 Main task has work to do, go find it
14913 got dsc Start(2)
14914 start 2
14915 Oct 13 18:12:22.617 INFO request completed, latency_us: 321, response_code: 204, uri: /start/cid/2, method: POST, req_id: b7d69381-8659-4c09-b52c-e42d0fd6cbf5, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
14916 Replay: started 2, returned:Ok(())
14917 ----------------------------------------------------------------
14918 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
14919 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14920 5752 Acked 6751 Read 3 Done Done New false
14921 5753 Acked 6752 Write 10 Done Done New false
14922 5754 Acked 6753 Read 10 Done Done New false
14923 5755 Acked 6754 Read 6 Done Done New false
14924 5756 Acked 6755 Read 9 Done Done New false
14925 5757 Acked 6756 Read 8 Done Done New false
14926 5758 Acked 6757 Flush 0 Done Done New false
14927 5759 Acked 6758 Read 4 Done Done New false
14928 5760 Acked 6759 Write 6 Done Done New false
14929 5761 Acked 6760 Write 7 Done Done New false
14930 5762 Acked 6761 Write 10 Done Done New false
14931 5763 Acked 6762 Read 10 Done Done New false
14932 5764 Acked 6763 Read 6 Done Done New false
14933 5765 Acked 6764 Write 6 Done Done New false
14934 5766 Acked 6765 Write 3 Done Done New false
14935 5767 Acked 6766 Write 6 Done Done New false
14936 5768 Acked 6767 Read 8 Done Done New false
14937 5769 Acked 6768 Read 10 Done Done New false
14938 5770 Acked 6769 Read 1 Done Done New false
14939 5771 Acked 6770 Write 5 Done Done New false
14940 5772 Acked 6771 Flush 0 Done Done New false
14941 5773 Acked 6772 Read 7 Done Done New false
14942 5774 Acked 6773 Write 6 Done Done New false
14943 5775 Acked 6774 Read 2 Done Done New false
14944 5776 Acked 6775 Read 2 Done Done New false
14945 5777 Acked 6776 Write 6 Done Done New false
14946 5778 Acked 6777 Write 7 Done Done New false
14947 5779 Acked 6778 Write 3 Done Done New false
14948 5780 Acked 6779 Write 5 Done Done New false
14949 5781 Acked 6780 Write 3 Done Done New false
14950 5782 Acked 6781 Read 1 Done Done New false
14951 5783 Acked 6782 Write 10 Done Done New false
14952 5784 Acked 6783 Read 8 Done Done New false
14953 5785 Acked 6784 Write 1 Done Done New false
14954 5786 Acked 6785 Read 8 Done Done New false
14955 5787 Acked 6786 Read 2 Done Done New false
14956 5788 Acked 6787 Read 6 Done Done New false
14957 5789 Acked 6788 Read 6 Done Done New false
14958 5790 Acked 6789 Write 3 Done Done New false
14959 5791 Acked 6790 Write 7 Done Done New false
14960 5792 Acked 6791 Write 9 Done Done New false
14961 5793 Acked 6792 Read 8 Done Done New false
14962 5794 Acked 6793 Flush 0 Done Done New false
14963 5795 Acked 6794 Read 9 Done Done New false
14964 5796 Acked 6795 Read 10 Done Done New false
14965 5797 Acked 6796 Write 4 Done Done New false
14966 5798 Acked 6797 Write 7 Done Done New false
14967 5799 Acked 6798 Read 9 Done Done New false
14968 5800 Acked 6799 Write 9 Done Done New false
14969 5801 Acked 6800 Read 7 Done Done New false
14970 5802 Acked 6801 Flush 0 Done Done New false
14971 5803 Acked 6802 Write 2 Done Done New false
14972 5804 Acked 6803 Flush 0 Done Done New false
14973 5805 Acked 6804 Write 9 Done Done New false
14974 5806 Acked 6805 Write 5 Done Done New false
14975 5807 Acked 6806 Read 6 Done Done New false
14976 5808 Acked 6807 Read 3 Done Done New false
14977 5809 Acked 6808 Write 1 Done Done New false
14978 5810 Acked 6809 Flush 0 Done Done New false
14979 5811 Acked 6810 Read 10 Done Done New false
14980 5812 Acked 6811 Write 8 Done Done New false
14981 5813 Acked 6812 Flush 0 Done Done New false
14982 5814 Acked 6813 Read 7 Done Done New false
14983 5815 Acked 6814 Flush 0 Done Done New false
14984 5816 Acked 6815 Write 5 Done Done New false
14985 5817 Acked 6816 Read 2 Done Done New false
14986 5818 Acked 6817 Read 9 Done Done New false
14987 5819 Acked 6818 Read 1 Done Done New false
14988 5820 Acked 6819 Read 7 Done Done New false
14989 5821 Acked 6820 Read 8 Done Done New false
14990 5822 Acked 6821 Write 10 Done Done New false
14991 5823 Acked 6822 Write 8 Done Done New false
14992 5824 Acked 6823 Read 8 Done Done New false
14993 5825 Acked 6824 Read 3 Done Done New false
14994 5826 Acked 6825 Write 4 Done Done New false
14995 5827 Acked 6826 Read 1 Done Done New false
14996 5828 Acked 6827 Read 2 Done Done New false
14997 5829 Acked 6828 Write 6 Done Done New false
14998 5830 Acked 6829 Read 1 Done Done New false
14999 5831 Acked 6830 Write 8 Done Done New false
15000 5832 Acked 6831 Read 5 Done Done New false
15001 5833 Acked 6832 Read 9 Done Done New false
15002 5834 Acked 6833 Write 7 Done Done New false
15003 5835 Acked 6834 Write 4 Done Done New false
15004 5836 Acked 6835 Read 9 Done Done New false
15005 5837 Acked 6836 Read 6 Done Done New false
15006 5838 Acked 6837 Read 6 Done Done New false
15007 5839 Acked 6838 Write 4 Done Done New false
15008 5840 Acked 6839 Read 10 Done Done New false
15009 5841 Acked 6840 Write 10 Done Done New false
15010 5842 Acked 6841 Write 3 Done Done New false
15011 5843 Acked 6842 Read 10 Done Done New false
15012 5844 Acked 6843 Flush 0 Done Done New false
15013 5845 Acked 6844 Write 10 Done Done New false
15014 5846 Acked 6845 Read 8 Done Done New false
15015 5847 Acked 6846 Write 7 Done Done New false
15016 5848 Acked 6847 Read 10 Done Done New false
15017 5849 Acked 6848 Write 3 Done Done New false
15018 5850 Acked 6849 Read 4 Done Done New false
15019 5851 Acked 6850 Write 7 Done Done New false
15020 5852 Acked 6851 Read 7 Done Done New false
15021 5853 Acked 6852 Read 6 Done Done New false
15022 5854 Acked 6853 Write 5 Done Done New false
15023 5855 Acked 6854 Write 2 Done Done New false
15024 5856 Acked 6855 Read 9 Done Done New false
15025 5857 Acked 6856 Read 2 Done Done New false
15026 5858 Acked 6857 Write 1 Done Done New false
15027 5859 Acked 6858 Read 1 Done Done New false
15028 5860 Acked 6859 Write 1 Done Done New false
15029 5861 Acked 6860 Read 3 Done Done New false
15030 5862 Acked 6861 Write 7 Done Done New false
15031 5863 Acked 6862 Read 5 Done Done New false
15032 5864 Acked 6863 Write 4 Done Done New false
15033 5865 Acked 6864 Read 10 Done Done New false
15034 5866 Acked 6865 Write 6 Done Done New false
15035 5867 Acked 6866 Read 1 Done Done New false
15036 5868 Acked 6867 Read 1 Done Done New false
15037 5869 Acked 6868 Write 3 Done Done New false
15038 5870 Acked 6869 Write 1 Done Done New false
15039 5871 Acked 6870 Write 7 Done Done New false
15040 5872 Acked 6871 Write 7 Done Done New false
15041 5873 Acked 6872 Write 1 Done Done New false
15042 5874 Acked 6873 Read 1 Done Done New false
15043 5875 Acked 6874 Read 1 Done Done New false
15044 5876 Acked 6875 Read 4 Done Done New false
15045 5877 Acked 6876 Read 7 Done Done New false
15046 5878 Acked 6877 Write 10 Done Done New false
15047 5879 Acked 6878 Read 3 Done Done New false
15048 5880 Acked 6879 Read 5 Done Done New false
15049 5881 Acked 6880 Write 7 Done Done New false
15050 5882 Acked 6881 Flush 0 Done Done New false
15051 5883 Acked 6882 Write 10 Done Done New false
15052 5884 Acked 6883 Read 5 Done Done New false
15053 5885 Acked 6884 Read 2 Done Done New false
15054 5886 Acked 6885 Read 2 Done Done New false
15055 5887 Acked 6886 Read 6 Done Done New false
15056 5888 Acked 6887 Read 9 Done Done New false
15057 5889 Acked 6888 Flush 0 Done Done New false
15058 5890 Acked 6889 Read 2 Done Done New false
15059 5891 Acked 6890 Flush 0 Done Done New false
15060 5892 Acked 6891 Write 5 Done Done New false
15061 5893 Acked 6892 Read 5 Done Done New false
15062 5894 Acked 6893 Read 6 Done Done New false
15063 5895 Acked 6894 Read 4 Done Done New false
15064 5896 Acked 6895 Read 8 Done Done New false
15065 5897 Acked 6896 Write 1 Done Done New false
15066 5898 Acked 6897 Flush 0 Done Done New false
15067 5899 Acked 6898 Read 10 Done Done New false
15068 5900 Acked 6899 Flush 0 Done Done New false
15069 5901 Acked 6900 Write 1 Done Done New false
15070 5902 Acked 6901 Read 6 Done Done New false
15071 5903 Acked 6902 Write 4 Done Done New false
15072 5904 Acked 6903 Write 7 Done Done New false
15073 5905 Acked 6904 Read 3 Done Done New false
15074 5906 Acked 6905 Read 8 Done Done New false
15075 5907 Acked 6906 Read 8 Done Done New false
15076 5908 Acked 6907 Write 5 Done Done New false
15077 5909 Acked 6908 Write 4 Done Done New false
15078 5910 Acked 6909 Write 9 Done Done New false
15079 5911 Acked 6910 Read 8 Done Done New false
15080 5912 Acked 6911 Write 6 Done Done New false
15081 5913 Acked 6912 Write 7 Done Done New false
15082 5914 Acked 6913 Write 10 Done Done New false
15083 5915 Acked 6914 Write 2 Done Done New false
15084 5916 Acked 6915 Read 9 Done Done New false
15085 5917 Acked 6916 Write 2 Done Done New false
15086 5918 Acked 6917 Flush 0 Done Done New false
15087 5919 Acked 6918 Read 9 Done Done New false
15088 5920 Acked 6919 Read 7 Done Done New false
15089 5921 Acked 6920 Read 1 Done Done New false
15090 5922 Acked 6921 Write 10 Done Done New false
15091 5923 Acked 6922 Read 3 Done Done New false
15092 5924 Acked 6923 Write 9 Done Done New false
15093 5925 Acked 6924 Write 3 Done Done New false
15094 5926 Acked 6925 Write 10 Done Done New false
15095 5927 Acked 6926 Write 9 Done Done New false
15096 5928 Acked 6927 Read 1 Done Done New false
15097 5929 Acked 6928 Read 9 Done Done New false
15098 5930 Acked 6929 Read 8 Done Done New false
15099 5931 Acked 6930 Read 3 Done Done New false
15100 5932 Acked 6931 Read 8 Done Done New false
15101 5933 Acked 6932 Read 3 Done Done New false
15102 5934 Acked 6933 Write 9 Done Done New false
15103 5935 Acked 6934 Read 4 Done Done New false
15104 5936 Acked 6935 Read 1 Done Done New false
15105 5937 Acked 6936 Read 4 Done Done New false
15106 5938 Acked 6937 Write 7 Done Done New false
15107 5939 Acked 6938 Write 7 Done Done New false
15108 5940 Acked 6939 Read 8 Done Done New false
15109 5941 Acked 6940 Write 3 Done Done New false
15110 5942 Acked 6941 Write 7 Done Done New false
15111 5943 Acked 6942 Write 8 Done Done New false
15112 5944 Acked 6943 Write 5 Done Done New false
15113 5945 Acked 6944 Read 7 Done Done New false
15114 5946 Acked 6945 Read 5 Done Done New false
15115 5947 Acked 6946 Read 5 Done Done New false
15116 5948 Acked 6947 Read 6 Done Done New false
15117 5949 Acked 6948 Read 2 Done Done New false
15118 5950 Acked 6949 Read 5 Done Done New false
15119 5951 Acked 6950 Read 3 Done Done New false
15120 5952 Acked 6951 Read 1 Done Done New false
15121 5953 Acked 6952 Write 6 Done Done New false
15122 5954 Acked 6953 Read 6 Done Done New false
15123 5955 Acked 6954 Read 3 Done Done New false
15124 5956 Acked 6955 Write 5 Done Done New false
15125 5957 Acked 6956 Read 2 Done Done New false
15126 5958 Acked 6957 Read 4 Done Done New false
15127 5959 Acked 6958 Read 8 Done Done New false
15128 5960 Acked 6959 Write 9 Done Done New false
15129 5961 Acked 6960 Read 1 Done Done New false
15130 5962 Acked 6961 Write 6 Done Done New false
15131 5963 Acked 6962 Write 4 Done Done New false
15132 5964 Acked 6963 Read 3 Done Done New false
15133 5965 Acked 6964 Write 3 Done Done New false
15134 5966 Acked 6965 Write 2 Done Done New false
15135 5967 Acked 6966 Write 3 Done Done New false
15136 5968 Acked 6967 Read 5 Done Done New false
15137 5969 Acked 6968 Read 5 Done Done New false
15138 5970 Acked 6969 Write 6 Done Done New false
15139 5971 Acked 6970 Read 5 Done Done New false
15140 5972 Acked 6971 Read 8 Done Done New false
15141 5973 Acked 6972 Read 3 Done Done New false
15142 5974 Acked 6973 Read 9 Done Done New false
15143 5975 Acked 6974 Write 2 Done Done New false
15144 5976 Acked 6975 Read 7 Done Done New false
15145 5977 Acked 6976 Write 3 Done Done New false
15146 5978 Acked 6977 Read 3 Done Done New false
15147 5979 Acked 6978 Read 4 Done Done New false
15148 5980 Acked 6979 Read 7 Done Done New false
15149 5981 Acked 6980 Read 3 Done Done New false
15150 5982 Acked 6981 Write 7 Done Done New false
15151 5983 Acked 6982 Read 9 Done Done New false
15152 5984 Acked 6983 Read 6 Done Done New false
15153 5985 Acked 6984 Flush 0 Done Done New false
15154 5986 Acked 6985 Write 6 Done Done New false
15155 5987 Acked 6986 Read 1 Done Done New false
15156 5988 Acked 6987 Write 2 Done Done New false
15157 5989 Acked 6988 Read 10 Done Done New false
15158 5990 Acked 6989 Write 6 Done Done New false
15159 5991 Acked 6990 Read 6 Done Done New false
15160 5992 Acked 6991 Read 1 Done Done New false
15161 5993 Acked 6992 Flush 0 Done Done New false
15162 5994 Acked 6993 Read 4 Done Done New false
15163 5995 Acked 6994 Read 2 Done Done New false
15164 5996 Acked 6995 Read 6 Done Done New false
15165 5997 Acked 6996 Write 4 Done Done New false
15166 5998 Acked 6997 Read 1 Done Done New false
15167 5999 Acked 6998 Write 3 Done Done New false
15168 6000 Acked 6999 Read 9 Done Done New false
15169 6001 Acked 7000 Read 5 Done Done New false
15170 6002 Acked 7001 Write 2 Done Done New false
15171 6003 Acked 7002 Write 6 Done Done New false
15172 6004 Acked 7003 Write 9 Done Done New false
15173 6005 Acked 7004 Write 8 Done Done New false
15174 6006 Acked 7005 Read 6 Done Done New false
15175 6007 Acked 7006 Flush 0 Done Done New false
15176 6008 Acked 7007 Write 9 Done Done New false
15177 6009 Acked 7008 Read 4 Done Done New false
15178 6010 Acked 7009 Write 6 Done Done New false
15179 6011 Acked 7010 Read 1 Done Done New false
15180 6012 Acked 7011 Read 7 Done Done New false
15181 6013 Acked 7012 Flush 0 Done Done New false
15182 6014 Acked 7013 Write 1 Done Done New false
15183 6015 Acked 7014 Write 5 Done Done New false
15184 6016 Acked 7015 Read 3 Done Done New false
15185 6017 Acked 7016 Flush 0 Done Done New false
15186 6018 Acked 7017 Read 4 Done Done New false
15187 6019 Acked 7018 Read 9 Done Done New false
15188 6020 Acked 7019 Write 9 Done Done New false
15189 6021 Acked 7020 Write 10 Done Done New false
15190 6022 Acked 7021 Write 9 Done Done New false
15191 6023 Acked 7022 Read 7 Done Done New false
15192 6024 Acked 7023 Write 6 Done Done New false
15193 6025 Acked 7024 Read 10 Done Done New false
15194 6026 Acked 7025 Read 1 Done Done New false
15195 6027 Acked 7026 Write 2 Done Done New false
15196 6028 Acked 7027 Read 2 Done Done New false
15197 6029 Acked 7028 Write 7 Done Done New false
15198 6030 Acked 7029 Write 1 Done Done New false
15199 6031 Acked 7030 Write 3 Done Done New false
15200 6032 Acked 7031 Read 7 Done Done New false
15201 6033 Acked 7032 Write 8 Done Done New false
15202 6034 Acked 7033 Read 8 Done Done New false
15203 6035 Acked 7034 Write 7 Done Done New false
15204 6036 Acked 7035 Flush 0 Done Done New false
15205 6037 Acked 7036 Read 4 Done Done New false
15206 6038 Acked 7037 Write 10 Done Done New false
15207 6039 Acked 7038 Write 8 Done Done New false
15208 6040 Acked 7039 Write 7 Done Done New false
15209 6041 Acked 7040 Write 7 Done Done New false
15210 6042 Acked 7041 Flush 0 Done Done New false
15211 6043 Acked 7042 Write 8 Done Done New false
15212 6044 Acked 7043 Read 4 Done Done New false
15213 6045 Acked 7044 Read 10 Done Done New false
15214 6046 Acked 7045 Read 4 Done Done New false
15215 6047 Acked 7046 Write 8 Done Done New false
15216 6048 Acked 7047 Write 9 Done Done New false
15217 6049 Acked 7048 Read 8 Done Done New false
15218 6050 Acked 7049 Write 5 Done Done New false
15219 6051 Acked 7050 Write 2 Done Done New false
15220 6052 Acked 7051 Read 9 Done Done New false
15221 6053 Acked 7052 Write 2 Sent Sent New false
15222 STATES DS:0 DS:1 DS:2 TOTAL
15223 New 0 0 302 302
15224 Sent 1 1 0 2
15225 Done 301 301 0 602
15226 Skipped 0 0 0 0
15227 Error 0 0 0 0
15228 Last Flush: 7041 7041 6750
15229 Downstairs last five completed: 6750 6749 6748 6747 6746 6745
15230 Upstairs last five completed: 6053 6052 6051 6050 6049 6048
15231 CLIENT: Up:0 ds:302 act:2
15232 [8830] got start action
15233 [8830] I am going to restart
15234 Starting downstairs at port 8830
15235 Make output file at "/tmp/dsc/downstairs-8830.txt"
15236 [8830][2] reports Starting
15237 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1196)
15238 [8830][2] reports Running
15239 ----------------------------------------------------------------
15240 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
15241 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15242 5752 Acked 6751 Read 3 Done Done New false
15243 5753 Acked 6752 Write 10 Done Done New false
15244 5754 Acked 6753 Read 10 Done Done New false
15245 5755 Acked 6754 Read 6 Done Done New false
15246 5756 Acked 6755 Read 9 Done Done New false
15247 5757 Acked 6756 Read 8 Done Done New false
15248 5758 Acked 6757 Flush 0 Done Done New false
15249 5759 Acked 6758 Read 4 Done Done New false
15250 5760 Acked 6759 Write 6 Done Done New false
15251 5761 Acked 6760 Write 7 Done Done New false
15252 5762 Acked 6761 Write 10 Done Done New false
15253 5763 Acked 6762 Read 10 Done Done New false
15254 5764 Acked 6763 Read 6 Done Done New false
15255 5765 Acked 6764 Write 6 Done Done New false
15256 5766 Acked 6765 Write 3 Done Done New false
15257 5767 Acked 6766 Write 6 Done Done New false
15258 5768 Acked 6767 Read 8 Done Done New false
15259 5769 Acked 6768 Read 10 Done Done New false
15260 5770 Acked 6769 Read 1 Done Done New false
15261 5771 Acked 6770 Write 5 Done Done New false
15262 5772 Acked 6771 Flush 0 Done Done New false
15263 5773 Acked 6772 Read 7 Done Done New false
15264 5774 Acked 6773 Write 6 Done Done New false
15265 5775 Acked 6774 Read 2 Done Done New false
15266 5776 Acked 6775 Read 2 Done Done New false
15267 5777 Acked 6776 Write 6 Done Done New false
15268 5778 Acked 6777 Write 7 Done Done New false
15269 5779 Acked 6778 Write 3 Done Done New false
15270 5780 Acked 6779 Write 5 Done Done New false
15271 5781 Acked 6780 Write 3 Done Done New false
15272 5782 Acked 6781 Read 1 Done Done New false
15273 5783 Acked 6782 Write 10 Done Done New false
15274 5784 Acked 6783 Read 8 Done Done New false
15275 5785 Acked 6784 Write 1 Done Done New false
15276 5786 Acked 6785 Read 8 Done Done New false
15277 5787 Acked 6786 Read 2 Done Done New false
15278 5788 Acked 6787 Read 6 Done Done New false
15279 5789 Acked 6788 Read 6 Done Done New false
15280 5790 Acked 6789 Write 3 Done Done New false
15281 5791 Acked 6790 Write 7 Done Done New false
15282 5792 Acked 6791 Write 9 Done Done New false
15283 5793 Acked 6792 Read 8 Done Done New false
15284 5794 Acked 6793 Flush 0 Done Done New false
15285 5795 Acked 6794 Read 9 Done Done New false
15286 5796 Acked 6795 Read 10 Done Done New false
15287 5797 Acked 6796 Write 4 Done Done New false
15288 5798 Acked 6797 Write 7 Done Done New false
15289 5799 Acked 6798 Read 9 Done Done New false
15290 5800 Acked 6799 Write 9 Done Done New false
15291 5801 Acked 6800 Read 7 Done Done New false
15292 5802 Acked 6801 Flush 0 Done Done New false
15293 5803 Acked 6802 Write 2 Done Done New false
15294 5804 Acked 6803 Flush 0 Done Done New false
15295 5805 Acked 6804 Write 9 Done Done New false
15296 5806 Acked 6805 Write 5 Done Done New false
15297 5807 Acked 6806 Read 6 Done Done New false
15298 5808 Acked 6807 Read 3 Done Done New false
15299 5809 Acked 6808 Write 1 Done Done New false
15300 5810 Acked 6809 Flush 0 Done Done New false
15301 5811 Acked 6810 Read 10 Done Done New false
15302 5812 Acked 6811 Write 8 Done Done New false
15303 5813 Acked 6812 Flush 0 Done Done New false
15304 5814 Acked 6813 Read 7 Done Done New false
15305 5815 Acked 6814 Flush 0 Done Done New false
15306 5816 Acked 6815 Write 5 Done Done New false
15307 5817 Acked 6816 Read 2 Done Done New false
15308 5818 Acked 6817 Read 9 Done Done New false
15309 5819 Acked 6818 Read 1 Done Done New false
15310 5820 Acked 6819 Read 7 Done Done New false
15311 5821 Acked 6820 Read 8 Done Done New false
15312 5822 Acked 6821 Write 10 Done Done New false
15313 5823 Acked 6822 Write 8 Done Done New false
15314 5824 Acked 6823 Read 8 Done Done New false
15315 5825 Acked 6824 Read 3 Done Done New false
15316 5826 Acked 6825 Write 4 Done Done New false
15317 5827 Acked 6826 Read 1 Done Done New false
15318 5828 Acked 6827 Read 2 Done Done New false
15319 5829 Acked 6828 Write 6 Done Done New false
15320 5830 Acked 6829 Read 1 Done Done New false
15321 5831 Acked 6830 Write 8 Done Done New false
15322 5832 Acked 6831 Read 5 Done Done New false
15323 5833 Acked 6832 Read 9 Done Done New false
15324 5834 Acked 6833 Write 7 Done Done New false
15325 5835 Acked 6834 Write 4 Done Done New false
15326 5836 Acked 6835 Read 9 Done Done New false
15327 5837 Acked 6836 Read 6 Done Done New false
15328 5838 Acked 6837 Read 6 Done Done New false
15329 5839 Acked 6838 Write 4 Done Done New false
15330 5840 Acked 6839 Read 10 Done Done New false
15331 5841 Acked 6840 Write 10 Done Done New false
15332 5842 Acked 6841 Write 3 Done Done New false
15333 5843 Acked 6842 Read 10 Done Done New false
15334 5844 Acked 6843 Flush 0 Done Done New false
15335 5845 Acked 6844 Write 10 Done Done New false
15336 5846 Acked 6845 Read 8 Done Done New false
15337 5847 Acked 6846 Write 7 Done Done New false
15338 5848 Acked 6847 Read 10 Done Done New false
15339 5849 Acked 6848 Write 3 Done Done New false
15340 5850 Acked 6849 Read 4 Done Done New false
15341 5851 Acked 6850 Write 7 Done Done New false
15342 5852 Acked 6851 Read 7 Done Done New false
15343 5853 Acked 6852 Read 6 Done Done New false
15344 5854 Acked 6853 Write 5 Done Done New false
15345 5855 Acked 6854 Write 2 Done Done New false
15346 5856 Acked 6855 Read 9 Done Done New false
15347 5857 Acked 6856 Read 2 Done Done New false
15348 5858 Acked 6857 Write 1 Done Done New false
15349 5859 Acked 6858 Read 1 Done Done New false
15350 5860 Acked 6859 Write 1 Done Done New false
15351 5861 Acked 6860 Read 3 Done Done New false
15352 5862 Acked 6861 Write 7 Done Done New false
15353 5863 Acked 6862 Read 5 Done Done New false
15354 5864 Acked 6863 Write 4 Done Done New false
15355 5865 Acked 6864 Read 10 Done Done New false
15356 5866 Acked 6865 Write 6 Done Done New false
15357 5867 Acked 6866 Read 1 Done Done New false
15358 5868 Acked 6867 Read 1 Done Done New false
15359 5869 Acked 6868 Write 3 Done Done New false
15360 5870 Acked 6869 Write 1 Done Done New false
15361 5871 Acked 6870 Write 7 Done Done New false
15362 5872 Acked 6871 Write 7 Done Done New false
15363 5873 Acked 6872 Write 1 Done Done New false
15364 5874 Acked 6873 Read 1 Done Done New false
15365 5875 Acked 6874 Read 1 Done Done New false
15366 5876 Acked 6875 Read 4 Done Done New false
15367 5877 Acked 6876 Read 7 Done Done New false
15368 5878 Acked 6877 Write 10 Done Done New false
15369 5879 Acked 6878 Read 3 Done Done New false
15370 5880 Acked 6879 Read 5 Done Done New false
15371 5881 Acked 6880 Write 7 Done Done New false
15372 5882 Acked 6881 Flush 0 Done Done New false
15373 5883 Acked 6882 Write 10 Done Done New false
15374 5884 Acked 6883 Read 5 Done Done New false
15375 5885 Acked 6884 Read 2 Done Done New false
15376 5886 Acked 6885 Read 2 Done Done New false
15377 5887 Acked 6886 Read 6 Done Done New false
15378 5888 Acked 6887 Read 9 Done Done New false
15379 5889 Acked 6888 Flush 0 Done Done New false
15380 5890 Acked 6889 Read 2 Done Done New false
15381 5891 Acked 6890 Flush 0 Done Done New false
15382 5892 Acked 6891 Write 5 Done Done New false
15383 5893 Acked 6892 Read 5 Done Done New false
15384 5894 Acked 6893 Read 6 Done Done New false
15385 5895 Acked 6894 Read 4 Done Done New false
15386 5896 Acked 6895 Read 8 Done Done New false
15387 5897 Acked 6896 Write 1 Done Done New false
15388 5898 Acked 6897 Flush 0 Done Done New false
15389 5899 Acked 6898 Read 10 Done Done New false
15390 5900 Acked 6899 Flush 0 Done Done New false
15391 5901 Acked 6900 Write 1 Done Done New false
15392 5902 Acked 6901 Read 6 Done Done New false
15393 5903 Acked 6902 Write 4 Done Done New false
15394 5904 Acked 6903 Write 7 Done Done New false
15395 5905 Acked 6904 Read 3 Done Done New false
15396 5906 Acked 6905 Read 8 Done Done New false
15397 5907 Acked 6906 Read 8 Done Done New false
15398 5908 Acked 6907 Write 5 Done Done New false
15399 5909 Acked 6908 Write 4 Done Done New false
15400 5910 Acked 6909 Write 9 Done Done New false
15401 5911 Acked 6910 Read 8 Done Done New false
15402 5912 Acked 6911 Write 6 Done Done New false
15403 5913 Acked 6912 Write 7 Done Done New false
15404 5914 Acked 6913 Write 10 Done Done New false
15405 5915 Acked 6914 Write 2 Done Done New false
15406 5916 Acked 6915 Read 9 Done Done New false
15407 5917 Acked 6916 Write 2 Done Done New false
15408 5918 Acked 6917 Flush 0 Done Done New false
15409 5919 Acked 6918 Read 9 Done Done New false
15410 5920 Acked 6919 Read 7 Done Done New false
15411 5921 Acked 6920 Read 1 Done Done New false
15412 5922 Acked 6921 Write 10 Done Done New false
15413 5923 Acked 6922 Read 3 Done Done New false
15414 5924 Acked 6923 Write 9 Done Done New false
15415 5925 Acked 6924 Write 3 Done Done New false
15416 5926 Acked 6925 Write 10 Done Done New false
15417 5927 Acked 6926 Write 9 Done Done New false
15418 5928 Acked 6927 Read 1 Done Done New false
15419 5929 Acked 6928 Read 9 Done Done New false
15420 5930 Acked 6929 Read 8 Done Done New false
15421 5931 Acked 6930 Read 3 Done Done New false
15422 5932 Acked 6931 Read 8 Done Done New false
15423 5933 Acked 6932 Read 3 Done Done New false
15424 5934 Acked 6933 Write 9 Done Done New false
15425 5935 Acked 6934 Read 4 Done Done New false
15426 5936 Acked 6935 Read 1 Done Done New false
15427 5937 Acked 6936 Read 4 Done Done New false
15428 5938 Acked 6937 Write 7 Done Done New false
15429 5939 Acked 6938 Write 7 Done Done New false
15430 5940 Acked 6939 Read 8 Done Done New false
15431 5941 Acked 6940 Write 3 Done Done New false
15432 5942 Acked 6941 Write 7 Done Done New false
15433 5943 Acked 6942 Write 8 Done Done New false
15434 5944 Acked 6943 Write 5 Done Done New false
15435 5945 Acked 6944 Read 7 Done Done New false
15436 5946 Acked 6945 Read 5 Done Done New false
15437 5947 Acked 6946 Read 5 Done Done New false
15438 5948 Acked 6947 Read 6 Done Done New false
15439 5949 Acked 6948 Read 2 Done Done New false
15440 5950 Acked 6949 Read 5 Done Done New false
15441 5951 Acked 6950 Read 3 Done Done New false
15442 5952 Acked 6951 Read 1 Done Done New false
15443 5953 Acked 6952 Write 6 Done Done New false
15444 5954 Acked 6953 Read 6 Done Done New false
15445 5955 Acked 6954 Read 3 Done Done New false
15446 5956 Acked 6955 Write 5 Done Done New false
15447 5957 Acked 6956 Read 2 Done Done New false
15448 5958 Acked 6957 Read 4 Done Done New false
15449 5959 Acked 6958 Read 8 Done Done New false
15450 5960 Acked 6959 Write 9 Done Done New false
15451 5961 Acked 6960 Read 1 Done Done New false
15452 5962 Acked 6961 Write 6 Done Done New false
15453 5963 Acked 6962 Write 4 Done Done New false
15454 5964 Acked 6963 Read 3 Done Done New false
15455 5965 Acked 6964 Write 3 Done Done New false
15456 5966 Acked 6965 Write 2 Done Done New false
15457 5967 Acked 6966 Write 3 Done Done New false
15458 5968 Acked 6967 Read 5 Done Done New false
15459 5969 Acked 6968 Read 5 Done Done New false
15460 5970 Acked 6969 Write 6 Done Done New false
15461 5971 Acked 6970 Read 5 Done Done New false
15462 5972 Acked 6971 Read 8 Done Done New false
15463 5973 Acked 6972 Read 3 Done Done New false
15464 5974 Acked 6973 Read 9 Done Done New false
15465 5975 Acked 6974 Write 2 Done Done New false
15466 5976 Acked 6975 Read 7 Done Done New false
15467 5977 Acked 6976 Write 3 Done Done New false
15468 5978 Acked 6977 Read 3 Done Done New false
15469 5979 Acked 6978 Read 4 Done Done New false
15470 5980 Acked 6979 Read 7 Done Done New false
15471 5981 Acked 6980 Read 3 Done Done New false
15472 5982 Acked 6981 Write 7 Done Done New false
15473 5983 Acked 6982 Read 9 Done Done New false
15474 5984 Acked 6983 Read 6 Done Done New false
15475 5985 Acked 6984 Flush 0 Done Done New false
15476 5986 Acked 6985 Write 6 Done Done New false
15477 5987 Acked 6986 Read 1 Done Done New false
15478 5988 Acked 6987 Write 2 Done Done New false
15479 5989 Acked 6988 Read 10 Done Done New false
15480 5990 Acked 6989 Write 6 Done Done New false
15481 5991 Acked 6990 Read 6 Done Done New false
15482 5992 Acked 6991 Read 1 Done Done New false
15483 5993 Acked 6992 Flush 0 Done Done New false
15484 5994 Acked 6993 Read 4 Done Done New false
15485 5995 Acked 6994 Read 2 Done Done New false
15486 5996 Acked 6995 Read 6 Done Done New false
15487 5997 Acked 6996 Write 4 Done Done New false
15488 5998 Acked 6997 Read 1 Done Done New false
15489 5999 Acked 6998 Write 3 Done Done New false
15490 6000 Acked 6999 Read 9 Done Done New false
15491 6001 Acked 7000 Read 5 Done Done New false
15492 6002 Acked 7001 Write 2 Done Done New false
15493 6003 Acked 7002 Write 6 Done Done New false
15494 6004 Acked 7003 Write 9 Done Done New false
15495 6005 Acked 7004 Write 8 Done Done New false
15496 6006 Acked 7005 Read 6 Done Done New false
15497 6007 Acked 7006 Flush 0 Done Done New false
15498 6008 Acked 7007 Write 9 Done Done New false
15499 6009 Acked 7008 Read 4 Done Done New false
15500 6010 Acked 7009 Write 6 Done Done New false
15501 6011 Acked 7010 Read 1 Done Done New false
15502 6012 Acked 7011 Read 7 Done Done New false
15503 6013 Acked 7012 Flush 0 Done Done New false
15504 6014 Acked 7013 Write 1 Done Done New false
15505 6015 Acked 7014 Write 5 Done Done New false
15506 6016 Acked 7015 Read 3 Done Done New false
15507 6017 Acked 7016 Flush 0 Done Done New false
15508 6018 Acked 7017 Read 4 Done Done New false
15509 6019 Acked 7018 Read 9 Done Done New false
15510 6020 Acked 7019 Write 9 Done Done New false
15511 6021 Acked 7020 Write 10 Done Done New false
15512 6022 Acked 7021 Write 9 Done Done New false
15513 6023 Acked 7022 Read 7 Done Done New false
15514 6024 Acked 7023 Write 6 Done Done New false
15515 6025 Acked 7024 Read 10 Done Done New false
15516 6026 Acked 7025 Read 1 Done Done New false
15517 6027 Acked 7026 Write 2 Done Done New false
15518 6028 Acked 7027 Read 2 Done Done New false
15519 6029 Acked 7028 Write 7 Done Done New false
15520 6030 Acked 7029 Write 1 Done Done New false
15521 6031 Acked 7030 Write 3 Done Done New false
15522 6032 Acked 7031 Read 7 Done Done New false
15523 6033 Acked 7032 Write 8 Done Done New false
15524 6034 Acked 7033 Read 8 Done Done New false
15525 6035 Acked 7034 Write 7 Done Done New false
15526 6036 Acked 7035 Flush 0 Done Done New false
15527 6037 Acked 7036 Read 4 Done Done New false
15528 6038 Acked 7037 Write 10 Done Done New false
15529 6039 Acked 7038 Write 8 Done Done New false
15530 6040 Acked 7039 Write 7 Done Done New false
15531 6041 Acked 7040 Write 7 Done Done New false
15532 6042 Acked 7041 Flush 0 Done Done New false
15533 6043 Acked 7042 Write 8 Done Done New false
15534 6044 Acked 7043 Read 4 Done Done New false
15535 6045 Acked 7044 Read 10 Done Done New false
15536 6046 Acked 7045 Read 4 Done Done New false
15537 6047 Acked 7046 Write 8 Done Done New false
15538 6048 Acked 7047 Write 9 Done Done New false
15539 6049 Acked 7048 Read 8 Done Done New false
15540 6050 Acked 7049 Write 5 Done Done New false
15541 6051 Acked 7050 Write 2 Done Done New false
15542 6052 Acked 7051 Read 9 Done Done New false
15543 6053 Acked 7052 Write 2 Done Done New false
15544 6054 Acked 7053 Flush 0 Done Done New false
15545 STATES DS:0 DS:1 DS:2 TOTAL
15546 New 0 0 303 303
15547 Sent 0 0 0 0
15548 Done 303 303 0 606
15549 Skipped 0 0 0 0
15550 Error 0 0 0 0
15551 Last Flush: 7053 7053 6750
15552 Downstairs last five completed: 6750 6749 6748 6747 6746 6745
15553 Upstairs last five completed: 6054 6053 6052 6051 6050 6049
15554 CLIENT: Up:0 ds:303 act:2
155552023-10-13T18:12:26.661ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 2
155562023-10-13T18:12:26.661ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
155572023-10-13T18:12:26.662ZINFOcrucible: [2] client re-new 303 jobs since flush 6750 = downstairs
155582023-10-13T18:12:26.662ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
155592023-10-13T18:12:26.663ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
155602023-10-13T18:12:26.663ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
155612023-10-13T18:12:26.663ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
155622023-10-13T18:12:26.663ZINFOcrucible: [2] send last flush ID to this DS: 6750
155632023-10-13T18:12:26.663ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
155642023-10-13T18:12:26.663ZINFOcrucible: [2] Replied this last flush ID: 6750
155652023-10-13T18:12:26.663ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Offline ds_transition to Replay
155662023-10-13T18:12:26.663ZINFOcrucible: [2] Transition from Offline to Replay
155672023-10-13T18:12:26.663ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
155682023-10-13T18:12:26.664ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Replay ds_transition to Active
155692023-10-13T18:12:26.664ZINFOcrucible: [2] Transition from Replay to Active
155702023-10-13T18:12:26.664ZINFOcrucible: [2] Starts cmd_loop
155712023-10-13T18:12:26.664ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
155722023-10-13T18:12:26.664ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
155732023-10-13T18:12:26.664ZINFOcrucible: Set check for repair
155742023-10-13T18:12:27.665ZINFOcrucible: Checking if live repair is needed
155752023-10-13T18:12:27.665ZINFOcrucible: No Live Repair required at this time
155762023-10-13T18:12:27.665ZWARNcrucible: [2] flow control end
15577 ----------------------------------------------------------------
15578 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15579 Downstairs last five completed: 7053 7052 7051 7050 7049 7048
15580 Upstairs last five completed: 6054 6053 6052 6051 6050 6049
15581 CLIENT: Up:0 ds:0 act:3
15582 Replay: All jobs finished, all DS active.
15583 Main task has work to do, go find it
15584 got dsc Stop(1)
15585 stop 1
15586 Oct 13 18:12:30.624 INFO request completed, latency_us: 507, response_code: 204, uri: /stop/cid/1, method: POST, req_id: be831142-de6e-4c4c-aeeb-41b84392b20a, remote_addr: 127.0.0.1:[8820] Got stop action so:false kr:false
15587 64571, local_addr: 127.0.0.1:9998
15588 Oct 13 18:12:30.626 INFO request completed, latency_us: 215, response_code: 200, uri: /state/cid/1, method: GET, req_id: 646a10b6-db64-4843-a88f-60eec884faaa, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
155892023-10-13T18:12:30.626ZWARNcrucible: [1] downstairs disconnected
155902023-10-13T18:12:30.626ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
155912023-10-13T18:12:30.626ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8820 closed looper = 1
155922023-10-13T18:12:30.626ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
155932023-10-13T18:12:30.626ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
155942023-10-13T18:12:30.626ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
155952023-10-13T18:12:30.626ZWARNcrucible: [1] pm_task rx.recv() is None
15596 [8820] Exited with: ExitStatus(unix_wait_status(9))
15597 [8820][1] reports Exit
155982023-10-13T18:12:32.629ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
15599 Oct 13 18:12:34.628 INFO request completed, latency_us: 257, response_code: 200, uri: /state/cid/1, method: GET, req_id: e7acd1f6-b443-4b94-b9e9-26eba898f9ba, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
15600 001/300 Write block 932 len 36864 data: 4 4 4 6 6 6 7 6 5
15601 002/300 Read block 716 len 36864
15602 003/300 Write block 1920 len 40960 data: 3 3 3 3 4 5 5 5 4 4
15603 004/300 Read block 710 len 8192
15604 005/300 Read block 43 len 16384
15605 006/300 Write block 3710 len 4096 data: 11
15606 007/300 Read block 3828 len 16384
15607 008/300 Flush
15608 009/300 Read block 4251 len 16384
15609 010/300 Flush
15610 011/300 Write block 4677 len 32768 data: 7 7 7 7 7 6 4 5
15611 012/300 Read block 720 len 28672
15612 013/300 Read block 1440 len 24576
15613 014/300 Read block 3269 len 36864
15614 015/300 Write block 221 len 12288 data: 2 3 4
15615 016/300 Read block 3403 len 40960
15616 017/300 Read block 3049 len 28672
15617 018/300 Read block 866 len 8192
15618 019/300 Read block 3425 len 16384
15619 020/300 Read block 324 len 20480
15620 021/300 Read block 3543 len 12288
15621 022/300 Write block 4474 len 4096 data: 2
15622 023/300 Flush
15623 024/300 Read block 483 len 20480
15624 025/300 Write block 629 len 16384 data: 5 6 4 3
15625 026/300 Read block 2061 len 36864
15626 027/300 Read block 1693 len 28672
15627 028/300 Read block 4433 len 32768
15628 029/300 Read block 3043 len 32768
15629 030/300 Read block 2165 len 16384
15630 031/300 Read block 2505 len 12288
15631 032/300 Write block 1636 len 8192 data: 6 6
15632 033/300 Flush
15633 034/300 Read block 2235 len 4096
15634 035/300 Write block 4038 len 40960 data: 2 2 2 3 5 6 4 5 4 4
15635 036/300 Read block 4487 len 36864
15636 037/300 Write block 2794 len 32768 data: 5 4 4 5 6 4 4 4
15637 038/300 Read block 4621 len 12288
15638 039/300 Read block 4503 len 8192
15639 040/300 Flush
15640 041/300 Read block 3966 len 12288
15641 042/300 Write block 4115 len 24576 data: 5 4 4 5 5 5
15642 043/300 Read block 19 len 12288
15643 044/300 Read block 79 len 12288
15644 045/300 Read block 4128 len 24576
15645 046/300 Write block 2923 len 8192 data: 5 4
15646 047/300 Read block 1547 len 24576
15647 048/300 Write block 4601 len 8192 data: 5 4
15648 049/300 Read block 1989 len 20480
15649 050/300 Write block 4667 len 40960 data: 7 6 5 5 3 3 5 4 4 6
15650 051/300 Write block 746 len 24576 data: 5 5 4 4 4 4
15651 052/300 Write block 4386 len 4096 data: 6
15652 053/300 Write block 1657 len 24576 data: 5 4 4 5 6 7
15653 054/300 Read block 3388 len 40960
15654 055/300 Write block 925 len 36864 data: 5 3 3 5 6 6 6 5 5
15655 056/300 Read block 3192 len 20480
15656 057/300 Read block 2868 len 20480
15657 058/300 Read block 2231 len 32768
15658 059/300 Read block 4991 len 24576
15659 060/300 Read block 218 len 28672
15660 061/300 Write block 4433 len 12288 data: 3 3 2
15661 062/300 Write block 3020 len 32768 data: 6 6 5 4 4 4 4 3
15662 063/300 Write block 3878 len 4096 data: 7
15663 064/300 Write block 1432 len 16384 data: 5 5 4 3
15664 065/300 Write block 3056 len 12288 data: 3 3 3
15665 066/300 Write block 2435 len 36864 data: 4 4 5 4 3 4 4 4 4
15666 067/300 Write block 492 len 36864 data: 5 4 5 4 4 2 2 2 2
15667 068/300 Write block 4756 len 28672 data: 7 7 5 4 7 8 7
15668 069/300 Write block 1849 len 40960 data: 3 3 3 4 6 6 6 6 5 5
15669 070/300 Read block 1504 len 40960
15670 071/300 Flush
15671 072/300 Write block 1477 len 20480 data: 6 6 6 4 4
15672 073/300 Write block 1247 len 12288 data: 3 2 2
15673 074/300 Flush
15674 075/300 Write block 3545 len 20480 data: 7 6 6 6 5
15675 076/300 Read block 750 len 4096
15676 077/300 Write block 2852 len 28672 data: 4 3 4 4 4 3 3
15677 078/300 Read block 3136 len 16384
15678 079/300 Read block 50 len 8192
15679 080/300 Read block 635 len 32768
15680 081/300 Write block 4868 len 20480 data: 3 2 3 4 4
15681 082/300 Write block 4410 len 20480 data: 4 4 4 5 5
15682 083/300 Read block 3982 len 12288
15683 084/300 Write block 3883 len 28672 data: 4 3 3 3 3 3 3
15684 085/300 Write block 2059 len 40960 data: 4 3 2 2 2 2 2 3 3 2
15685 086/300 Read block 1398 len 16384
15686 087/300 Write block 2437 len 16384 data: 6 5 4 5
15687 088/300 Read block 4098 len 24576
15688 089/300 Write block 311 len 24576 data: 5 4 5 7 5 5
15689 090/300 Write block 3588 len 4096 data: 2
15690 091/300 Flush
15691 092/300 Write block 350 len 4096 data: 6
15692 093/300 Read block 557 len 12288
15693 094/300 Read block 2887 len 24576
15694 095/300 Read block 4885 len 4096
15695 096/300 Write block 3710 len 12288 data: 12 12 10
15696 097/300 Read block 1446 len 16384
15697 098/300 Read block 2853 len 16384
15698 099/300 Read block 4774 len 16384
15699 100/300 Read block 2074 len 12288
15700 101/300 Write block 1697 len 16384 data: 6 4 3 2
15701 102/300 Write block 2654 len 12288 data: 2 2 2
15702 103/300 Write block 2978 len 20480 data: 3 2 2 2 2
15703 104/300 Read block 232 len 24576
15704 105/300 Write block 4071 len 4096 data: 5
15705 106/300 Read block 1741 len 32768
15706 107/300 Write block 4443 len 12288 data: 3 3 4
15707 108/300 Read block 2013 len 28672
15708 109/300 Write block 3140 len 12288 data: 6 7 7
15709 110/300 Read block 2912 len 36864
15710 111/300 Read block 3550 len 32768
15711 112/300 Read block 3789 len 40960
15712 113/300 Read block 181 len 24576
15713 114/300 Write block 1160 len 28672 data: 8 7 7 7 6 6 6
15714 115/300 Read block 1766 len 36864
15715 116/300 Read block 3746 len 24576
15716 117/300 Write block 4100 len 24576 data: 4 6 6 6 5 5
15717 118/300 Write block 3617 len 16384 data: 6 7 8 9
15718 119/300 Read block 1884 len 28672
15719 120/300 Flush
15720 121/300 Flush
15721 122/300 Write block 4398 len 40960 data: 10 8 8 7 5 5 5 7 8 6
15722 123/300 Read block 388 len 24576
15723 124/300 Write block 486 len 36864 data: 4 5 5 4 4 6 6 5 6
15724 125/300 Write block 2879 len 40960 data: 4 4 4 5 6 5 7 7 6 4
15725 126/300 Write block 2416 len 24576 data: 5 3 3 3 5 4
15726 127/300 Write block 3539 len 32768 data: 11 8 8 7 6 6 8 7
15727 128/300 Write block 852 len 8192 data: 8 6
15728 129/300 Write block 2977 len 24576 data: 3 4 3 3 3 3
15729 130/300 Flush
15730 131/300 Read block 622 len 20480
15731 132/300 Write block 881 len 20480 data: 4 4 4 5 7
15732 133/300 Write block 4355 len 16384 data: 4 4 4 3
15733 134/300 Write block 3870 len 20480 data: 4 4 5 4 6
15734 135/300 Read block 2131 len 28672
15735 136/300 Write block 4357 len 16384 data: 5 4 2 2
15736 137/300 Write block 2142 len 20480 data: 5 7 8 8 7
15737 138/300 Write block 531 len 4096 data: 2
15738 139/300 Write block 830 len 16384 data: 6 4 3 5
15739 140/300 Write block 1834 len 20480 data: 6 7 6 7 6
15740 141/300 Write block 3652 len 36864 data: 4 4 5 5 5 4 3 2 5
15741 142/300 Read block 2459 len 40960
15742 143/300 Read block 381 len 24576
15743 144/300 Write block 306 len 8192 data: 3 3
15744 145/300 Write block 1421 len 24576 data: 3 3 3 3 4 4
15745 146/300 Write block 3600 len 8192 data: 3 3
15746 147/300 Read block 3670 len 16384
15747 148/300 Read block 2707 len 8192
15748 149/300 Read block 3461 len 36864
15749 150/300 Read block 915 len 8192
15750 151/300 Read block 2314 len 4096
15751 152/300 Write block 2281 len 40960 data: 7 7 5 4 4 4 2 2 2 2
15752 153/300 Write block 4528 len 36864 data: 6 5 5 4 4 4 4 4 4
15753 154/300 Write block 1623 len 28672 data: 4 5 6 6 6 6 5
15754 155/300 Flush
15755 156/300 Write block 90 len 8192 data: 4 4
15756 157/300 Read block 3016 len 16384
15757 158/300 Read block 909 len 8192
15758 159/300 Write block 3784 len 20480 data: 4 3 2 2 2
15759 160/300 Write block 66 len 4096 data: 3
15760 161/300 Write block 1273 len 4096 data: 5
15761 162/300 Write block 1148 len 20480 data: 6 6 4 4 3
15762 163/300 Read block 4576 len 12288
15763 164/300 Read block 2894 len 4096
15764 165/300 Write block 2031 len 20480 data: 6 7 5 4 3
15765 166/300 Read block 4094 len 12288
15766 167/300 Read block 3473 len 24576
15767 168/300 Read block 241 len 8192
15768 169/300 Write block 1030 len 16384 data: 4 3 4 3
15769 170/300 Write block 2794 len 24576 data: 6 5 5 6 7 5
15770 171/300 Write block 1167 len 12288 data: 5 4 4
15771 172/300 Read block 2154 len 20480
15772 173/300 Write block 4013 len 12288 data: 5 6 7
15773 174/300 Write block 470 len 20480 data: 3 4 5 5 3
15774 175/300 Read block 4105 len 32768
15775 176/300 Read block 1504 len 24576
15776 177/300 Read block 4270 len 20480
15777 178/300 Read block 4246 len 40960
15778 179/300 Write block 2224 len 8192 data: 4 4
15779 180/300 Write block 4307 len 36864 data: 7 7 4 5 7 8 7 7 6
15780 181/300 Write block 2291 len 28672 data: 2 2 2 2 3 3 3
15781 182/300 Write block 3721 len 24576 data: 5 5 4 4 3 4
15782 183/300 Read block 4322 len 8192
15783 184/300 Flush
15784 185/300 Write block 3216 len 32768 data: 4 4 4 4 4 4 4 4
15785 186/300 Write block 279 len 8192 data: 6 6
15786 187/300 Flush
15787 188/300 Write block 361 len 8192 data: 4 5
15788 189/300 Read block 1173 len 28672
15789 190/300 Read block 4344 len 20480
15790 191/300 Write block 346 len 16384 data: 6 6 5 5
15791 192/300 Write block 3882 len 4096 data: 4
15792 193/300 Read block 4040 len 16384
15793 194/300 Read block 3704 len 12288
15794 195/300 Write block 3817 len 36864 data: 3 4 4 4 5 5 4 2 2
15795 196/300 Read block 4626 len 16384
15796 197/300 Read block 3527 len 8192
15797 198/300 Write block 1264 len 36864 data: 2 3 3 3 3 3 3 4 5
15798 199/300 Read block 1213 len 8192
15799 200/300 Read block 3127 len 12288
15800 201/300 Read block 2639 len 4096
15801 202/300 Write block 2761 len 12288 data: 6 6 6
15802 203/300 Read block 2432 len 16384
15803 204/300 Write block 321 len 4096 data: 8
15804 205/300 Write block 2461 len 28672 data: 5 5 7 7 7 7 6
15805 206/300 Write block 3331 len 40960 data: 6 6 6 5 5 5 5 6 5 3
15806 207/300 Read block 2047 len 12288
15807 208/300 Flush
15808 209/300 Flush
15809 210/300 Flush
15810 211/300 Read block 3949 len 8192
15811 212/300 Write block 2166 len 28672 data: 4 3 4 4 5 6 5
15812 213/300 Read block 463 len 4096
15813 214/300 Read block 1239 len 20480
15814 215/300 Read block 781 len 32768
15815 216/300 Write block 3782 len 20480 data: 5 5 5 4 3
15816 217/300 Read block 4622 len 24576
15817 218/300 Read block 2894 len 12288
15818 219/300 Write block 1708 len 28672 data: 4 4 4 4 3 4 3
15819 220/300 Flush
15820 221/300 Read block 1468 len 28672
15821 222/300 Read block 2441 len 36864
15822 223/300 Read block 3701 len 8192
15823 224/300 Write block 3525 len 4096 data: 4
15824 225/300 Write block 3988 len 28672 data: 6 7 5 6 5 5 5
15825 226/300 Read block 3265 len 4096
15826 227/300 Read block 441 len 16384
15827 228/300 Read block 219 len 16384
15828 229/300 Read block 1042 len 40960
15829 230/300 Read block 2779 len 28672
15830 231/300 Write block 1113 len 4096 data: 5
15831 232/300 Read block 1963 len 32768
15832 233/300 Read block 1164 len 36864
15833 234/300 Write block 4281 len 20480 data: 6 6 8 8 7
15834 235/300 Write block 3056 len 32768 data: 4 4 4 3 3 3 3 2
15835 236/300 Read block 1063 len 24576
15836 237/300 Flush
15837 238/300 Read block 1664 len 12288
15838 239/300 Write block 4968 len 36864 data: 2 2 4 4 5 5 4 5 5
15839 240/300 Write block 3743 len 16384 data: 7 6 6 7
15840 241/300 Write block 2570 len 4096 data: 5
15841 242/300 Read block 4460 len 40960
15842 243/300 Read block 3380 len 8192
15843 244/300 Write block 3886 len 36864 data: 4 4 4 4 3 4 5 3 4
15844 245/300 Write block 11 len 28672 data: 3 3 2 4 5 5 5
15845 246/300 Write block 3587 len 20480 data: 2 3 2 3 4
15846 247/300 Read block 3961 len 24576
15847 248/300 Read block 3286 len 40960
15848 249/300 Read block 1521 len 16384
15849 250/300 Write block 3957 len 16384 data: 6 5 6 5
15850 251/300 Read block 4937 len 4096
15851 252/300 Read block 3208 len 36864
15852 253/300 Read block 4634 len 36864
15853 254/300 Read block 1564 len 24576
15854 255/300 Write block 2350 len 40960 data: 8 8 8 6 7 7 7 7 4 5
15855 256/300 Write block 1026 len 20480 data: 3 2 4 4 5
15856 257/300 Write block 3010 len 20480 data: 5 7 6 8 7
15857 258/300 Read block 537 len 36864
15858 259/300 Flush
15859 260/300 Write block 1187 len 28672 data: 5 5 5 6 5 5 5
15860 261/300 Read block 3387 len 40960
15861 262/300 Write block 3300 len 8192 data: 3 3
15862 263/300 Write block 1242 len 16384 data: 2 2 4 4
15863 264/300 Flush
15864 265/300 Write block 2509 len 28672 data: 7 5 5 5 4 3 3
15865 266/300 Read block 723 len 8192
15866 267/300 Flush
15867 268/300 Read block 4194 len 20480
15868 269/300 Read block 1436 len 4096
15869 270/300 Flush
15870 271/300 Read block 4470 len 28672
15871 272/300 Read block 1980 len 40960
15872 273/300 Read block 4865 len 16384
15873 274/300 Read block 4679 len 20480
15874 275/300 Write block 1412 len 36864 data: 6 6 6 5 5 5 5 3 3
15875 276/300 Flush
15876 277/300 Write block 1765 len 8192 data: 5 5
15877 278/300 Read block 730 len 32768
15878 279/300 Read block 722 len 4096
15879 280/300 Write block 201 len 40960 data: 4 4 4 5 6 7 6 6 7 7
15880 281/300 Read block 4985 len 4096
15881 282/300 Write block 3841 len 16384 data: 5 5 4 3
15882 283/300 Read block 525 len 28672
15883 284/300 Write block 1081 len 12288 data: 3 2 2
15884 285/300 Write block 4143 len 36864 data: 2 2 2 4 4 4 5 6 7
15885 286/300 Flush
15886 287/300 Write block 4572 len 36864 data: 4 4 3 3 2 2 2 5 6
15887 288/300 Write block 4591 len 24576 data: 3 5 5 5 6 6
15888 289/300 Read block 2455 len 8192
15889 290/300 Write block 2965 len 20480 data: 5 6 4 5 6
15890 291/300 Flush
15891 292/300 Read block 2686 len 16384
15892 293/300 Write block 3269 len 40960 data: 3 3 3 3 2 2 2 2 2 2
15893 294/300 Write block 1399 len 4096 data: 6
15894 295/300 Read block 1503 len 4096
15895 296/300 Write block 3684 len 24576 data: 4 4 4 4 4 5
15896 297/300 Write block 160 len 32768 data: 3 4 6 5 5 5 6 6
15897 298/300 Write block 4859 len 16384 data: 8 9 6 6
15898 299/300 Flush
15899 300/300 Write block 662 len 20480 data: 4 4 4 5 6
15900 Main task has work to do, go find it
15901 got dsc Start(1)
15902 start 1
15903 Oct 13 18:12:35.552 INFO request completed, latency_us: 309, response_code: 204, uri: /start/cid/1, method: POST, req_id: 094da28a-fda7-46c2-aa82-9cef804c9843, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
15904 Replay: started 1, returned:Ok(())
15905 ----------------------------------------------------------------
15906 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
15907 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15908 6055 Acked 7054 Write 9 Done New Done false
15909 6056 Acked 7055 Read 9 Done New Done false
15910 6057 Acked 7056 Write 10 Done New Done false
15911 6058 Acked 7057 Read 2 Done New Done false
15912 6059 Acked 7058 Read 4 Done New Done false
15913 6060 Acked 7059 Write 1 Done New Done false
15914 6061 Acked 7060 Read 4 Done New Done false
15915 6062 Acked 7061 Flush 0 Done New Done false
15916 6063 Acked 7062 Read 4 Done New Done false
15917 6064 Acked 7063 Flush 0 Done New Done false
15918 6065 Acked 7064 Write 8 Done New Done false
15919 6066 Acked 7065 Read 7 Done New Done false
15920 6067 Acked 7066 Read 6 Done New Done false
15921 6068 Acked 7067 Read 9 Done New Done false
15922 6069 Acked 7068 Write 3 Done New Done false
15923 6070 Acked 7069 Read 10 Done New Done false
15924 6071 Acked 7070 Read 7 Done New Done false
15925 6072 Acked 7071 Read 2 Done New Done false
15926 6073 Acked 7072 Read 4 Done New Done false
15927 6074 Acked 7073 Read 5 Done New Done false
15928 6075 Acked 7074 Read 3 Done New Done false
15929 6076 Acked 7075 Write 1 Done New Done false
15930 6077 Acked 7076 Flush 0 Done New Done false
15931 6078 Acked 7077 Read 5 Done New Done false
15932 6079 Acked 7078 Write 4 Done New Done false
15933 6080 Acked 7079 Read 9 Done New Done false
15934 6081 Acked 7080 Read 7 Done New Done false
15935 6082 Acked 7081 Read 8 Done New Done false
15936 6083 Acked 7082 Read 8 Done New Done false
15937 6084 Acked 7083 Read 4 Done New Done false
15938 6085 Acked 7084 Read 3 Done New Done false
15939 6086 Acked 7085 Write 2 Done New Done false
15940 6087 Acked 7086 Flush 0 Done New Done false
15941 6088 Acked 7087 Read 1 Done New Done false
15942 6089 Acked 7088 Write 10 Done New Done false
15943 6090 Acked 7089 Read 9 Done New Done false
15944 6091 Acked 7090 Write 8 Done New Done false
15945 6092 Acked 7091 Read 3 Done New Done false
15946 6093 Acked 7092 Read 2 Done New Done false
15947 6094 Acked 7093 Flush 0 Done New Done false
15948 6095 Acked 7094 Read 3 Done New Done false
15949 6096 Acked 7095 Write 6 Done New Done false
15950 6097 Acked 7096 Read 3 Done New Done false
15951 6098 Acked 7097 Read 3 Done New Done false
15952 6099 Acked 7098 Read 6 Done New Done false
15953 6100 Acked 7099 Write 2 Done New Done false
15954 6101 Acked 7100 Read 6 Done New Done false
15955 6102 Acked 7101 Write 2 Done New Done false
15956 6103 Acked 7102 Read 5 Done New Done false
15957 6104 Acked 7103 Write 10 Done New Done false
15958 6105 Acked 7104 Write 6 Done New Done false
15959 6106 Acked 7105 Write 1 Done New Done false
15960 6107 Acked 7106 Write 6 Done New Done false
15961 6108 Acked 7107 Read 10 Done New Done false
15962 6109 Acked 7108 Write 9 Done New Done false
15963 6110 Acked 7109 Read 5 Done New Done false
15964 6111 Acked 7110 Read 5 Done New Done false
15965 6112 Acked 7111 Read 8 Done New Done false
15966 6113 Acked 7112 Read 6 Done New Done false
15967 6114 Acked 7113 Read 7 Done New Done false
15968 6115 Acked 7114 Write 3 Done New Done false
15969 6116 Acked 7115 Write 8 Done New Done false
15970 6117 Acked 7116 Write 1 Done New Done false
15971 6118 Acked 7117 Write 4 Done New Done false
15972 6119 Acked 7118 Write 3 Done New Done false
15973 6120 Acked 7119 Write 9 Done New Done false
15974 6121 Acked 7120 Write 9 Done New Done false
15975 6122 Acked 7121 Write 7 Done New Done false
15976 6123 Acked 7122 Write 10 Done New Done false
15977 6124 Acked 7123 Read 10 Done New Done false
15978 6125 Acked 7124 Flush 0 Done New Done false
15979 6126 Acked 7125 Write 5 Done New Done false
15980 6127 Acked 7126 Write 3 Done New Done false
15981 6128 Acked 7127 Flush 0 Done New Done false
15982 6129 Acked 7128 Write 5 Done New Done false
15983 6130 Acked 7129 Read 1 Done New Done false
15984 6131 Acked 7130 Write 7 Done New Done false
15985 6132 Acked 7131 Read 4 Done New Done false
15986 6133 Acked 7132 Read 2 Done New Done false
15987 6134 Acked 7133 Read 8 Done New Done false
15988 6135 Acked 7134 Write 5 Done New Done false
15989 6136 Acked 7135 Write 5 Done New Done false
15990 6137 Acked 7136 Read 3 Done New Done false
15991 6138 Acked 7137 Write 7 Done New Done false
15992 6139 Acked 7138 Write 10 Done New Done false
15993 6140 Acked 7139 Read 4 Done New Done false
15994 6141 Acked 7140 Write 4 Done New Done false
15995 6142 Acked 7141 Read 6 Done New Done false
15996 6143 Acked 7142 Write 6 Done New Done false
15997 6144 Acked 7143 Write 1 Done New Done false
15998 6145 Acked 7144 Flush 0 Done New Done false
15999 6146 Acked 7145 Write 1 Done New Done false
16000 6147 Acked 7146 Read 3 Done New Done false
16001 6148 Acked 7147 Read 6 Done New Done false
16002 6149 Acked 7148 Read 1 Done New Done false
16003 6150 Acked 7149 Write 3 Done New Done false
16004 6151 Acked 7150 Read 4 Done New Done false
16005 6152 Acked 7151 Read 4 Done New Done false
16006 6153 Acked 7152 Read 4 Done New Done false
16007 6154 Acked 7153 Read 3 Done New Done false
16008 6155 Acked 7154 Write 4 Done New Done false
16009 6156 Acked 7155 Write 3 Done New Done false
16010 6157 Acked 7156 Write 5 Done New Done false
16011 6158 Acked 7157 Read 6 Done New Done false
16012 6159 Acked 7158 Write 1 Done New Done false
16013 6160 Acked 7159 Read 8 Done New Done false
16014 6161 Acked 7160 Write 3 Done New Done false
16015 6162 Acked 7161 Read 7 Done New Done false
16016 6163 Acked 7162 Write 3 Done New Done false
16017 6164 Acked 7163 Read 9 Done New Done false
16018 6165 Acked 7164 Read 8 Done New Done false
16019 6166 Acked 7165 Read 10 Done New Done false
16020 6167 Acked 7166 Read 6 Done New Done false
16021 6168 Acked 7167 Write 7 Done New Done false
16022 6169 Acked 7168 Read 9 Done New Done false
16023 6170 Acked 7169 Flush 0 Done New Done false
16024 6171 Acked 7170 Read 6 Done New Done false
16025 6172 Acked 7171 Write 6 Done New Done false
16026 6173 Acked 7172 Write 4 Done New Done false
16027 6174 Acked 7173 Read 7 Done New Done false
16028 6175 Acked 7174 Flush 0 Done New Done false
16029 6176 Acked 7175 Flush 0 Done New Done false
16030 6177 Acked 7176 Write 10 Done New Done false
16031 6178 Acked 7177 Read 6 Done New Done false
16032 6179 Acked 7178 Write 9 Done New Done false
16033 6180 Acked 7179 Write 10 Done New Done false
16034 6181 Acked 7180 Write 6 Done New Done false
16035 6182 Acked 7181 Write 8 Done New Done false
16036 6183 Acked 7182 Write 2 Done New Done false
16037 6184 Acked 7183 Write 6 Done New Done false
16038 6185 Acked 7184 Flush 0 Done New Done false
16039 6186 Acked 7185 Read 5 Done New Done false
16040 6187 Acked 7186 Write 5 Done New Done false
16041 6188 Acked 7187 Write 4 Done New Done false
16042 6189 Acked 7188 Write 5 Done New Done false
16043 6190 Acked 7189 Read 7 Done New Done false
16044 6191 Acked 7190 Write 4 Done New Done false
16045 6192 Acked 7191 Write 5 Done New Done false
16046 6193 Acked 7192 Write 1 Done New Done false
16047 6194 Acked 7193 Write 4 Done New Done false
16048 6195 Acked 7194 Write 5 Done New Done false
16049 6196 Acked 7195 Write 9 Done New Done false
16050 6197 Acked 7196 Read 10 Done New Done false
16051 6198 Acked 7197 Read 6 Done New Done false
16052 6199 Acked 7198 Write 2 Done New Done false
16053 6200 Acked 7199 Write 6 Done New Done false
16054 6201 Acked 7200 Write 2 Done New Done false
16055 6202 Acked 7201 Read 4 Done New Done false
16056 6203 Acked 7202 Read 2 Done New Done false
16057 6204 Acked 7203 Read 9 Done New Done false
16058 6205 Acked 7204 Read 2 Done New Done false
16059 6206 Acked 7205 Read 1 Done New Done false
16060 6207 Acked 7206 Write 10 Done New Done false
16061 6208 Acked 7207 Write 9 Done New Done false
16062 6209 Acked 7208 Write 7 Done New Done false
16063 6210 Acked 7209 Flush 0 Done New Done false
16064 6211 Acked 7210 Write 2 Done New Done false
16065 6212 Acked 7211 Read 4 Done New Done false
16066 6213 Acked 7212 Read 2 Done New Done false
16067 6214 Acked 7213 Write 5 Done New Done false
16068 6215 Acked 7214 Write 1 Done New Done false
16069 6216 Acked 7215 Write 1 Done New Done false
16070 6217 Acked 7216 Write 5 Done New Done false
16071 6218 Acked 7217 Read 3 Done New Done false
16072 6219 Acked 7218 Read 1 Done New Done false
16073 6220 Acked 7219 Write 5 Done New Done false
16074 6221 Acked 7220 Read 3 Done New Done false
16075 6222 Acked 7221 Read 6 Done New Done false
16076 6223 Acked 7222 Read 2 Done New Done false
16077 6224 Acked 7223 Write 4 Done New Done false
16078 6225 Acked 7224 Write 6 Done New Done false
16079 6226 Acked 7225 Write 3 Done New Done false
16080 6227 Acked 7226 Read 5 Done New Done false
16081 6228 Acked 7227 Write 3 Done New Done false
16082 6229 Acked 7228 Write 5 Done New Done false
16083 6230 Acked 7229 Read 8 Done New Done false
16084 6231 Acked 7230 Read 6 Done New Done false
16085 6232 Acked 7231 Read 5 Done New Done false
16086 6233 Acked 7232 Read 10 Done New Done false
16087 6234 Acked 7233 Write 2 Done New Done false
16088 6235 Acked 7234 Write 9 Done New Done false
16089 6236 Acked 7235 Write 7 Done New Done false
16090 6237 Acked 7236 Write 6 Done New Done false
16091 6238 Acked 7237 Read 2 Done New Done false
16092 6239 Acked 7238 Flush 0 Done New Done false
16093 6240 Acked 7239 Write 8 Done New Done false
16094 6241 Acked 7240 Write 2 Done New Done false
16095 6242 Acked 7241 Flush 0 Done New Done false
16096 6243 Acked 7242 Write 2 Done New Done false
16097 6244 Acked 7243 Read 7 Done New Done false
16098 6245 Acked 7244 Read 5 Done New Done false
16099 6246 Acked 7245 Write 4 Done New Done false
16100 6247 Acked 7246 Write 1 Done New Done false
16101 6248 Acked 7247 Read 4 Done New Done false
16102 6249 Acked 7248 Read 3 Done New Done false
16103 6250 Acked 7249 Write 9 Done New Done false
16104 6251 Acked 7250 Read 4 Done New Done false
16105 6252 Acked 7251 Read 2 Done New Done false
16106 6253 Acked 7252 Write 9 Done New Done false
16107 6254 Acked 7253 Read 2 Done New Done false
16108 6255 Acked 7254 Read 3 Done New Done false
16109 6256 Acked 7255 Read 1 Done New Done false
16110 6257 Acked 7256 Write 3 Done New Done false
16111 6258 Acked 7257 Read 4 Done New Done false
16112 6259 Acked 7258 Write 1 Done New Done false
16113 6260 Acked 7259 Write 7 Done New Done false
16114 6261 Acked 7260 Write 10 Done New Done false
16115 6262 Acked 7261 Read 3 Done New Done false
16116 6263 Acked 7262 Flush 0 Done New Done false
16117 6264 Acked 7263 Flush 0 Done New Done false
16118 6265 Acked 7264 Flush 0 Done New Done false
16119 6266 Acked 7265 Read 2 Done New Done false
16120 6267 Acked 7266 Write 7 Done New Done false
16121 6268 Acked 7267 Read 1 Done New Done false
16122 6269 Acked 7268 Read 5 Done New Done false
16123 6270 Acked 7269 Read 8 Done New Done false
16124 6271 Acked 7270 Write 5 Done New Done false
16125 6272 Acked 7271 Read 6 Done New Done false
16126 6273 Acked 7272 Read 3 Done New Done false
16127 6274 Acked 7273 Write 7 Done New Done false
16128 6275 Acked 7274 Flush 0 Done New Done false
16129 6276 Acked 7275 Read 7 Done New Done false
16130 6277 Acked 7276 Read 9 Done New Done false
16131 6278 Acked 7277 Read 2 Done New Done false
16132 6279 Acked 7278 Write 1 Done New Done false
16133 6280 Acked 7279 Write 7 Done New Done false
16134 6281 Acked 7280 Read 1 Done New Done false
16135 6282 Acked 7281 Read 4 Done New Done false
16136 6283 Acked 7282 Read 4 Done New Done false
16137 6284 Acked 7283 Read 10 Done New Done false
16138 6285 Acked 7284 Read 7 Done New Done false
16139 6286 Acked 7285 Write 1 Done New Done false
16140 6287 Acked 7286 Read 8 Done New Done false
16141 6288 Acked 7287 Read 9 Done New Done false
16142 6289 Acked 7288 Write 5 Done New Done false
16143 6290 Acked 7289 Write 8 Done New Done false
16144 6291 Acked 7290 Read 6 Done New Done false
16145 6292 Acked 7291 Flush 0 Done New Done false
16146 6293 Acked 7292 Read 3 Done New Done false
16147 6294 Acked 7293 Write 9 Done New Done false
16148 6295 Acked 7294 Write 4 Done New Done false
16149 6296 Acked 7295 Write 1 Done New Done false
16150 6297 Acked 7296 Read 10 Done New Done false
16151 6298 Acked 7297 Read 2 Done New Done false
16152 6299 Acked 7298 Write 9 Done New Done false
16153 6300 Acked 7299 Write 7 Done New Done false
16154 6301 Acked 7300 Write 5 Done New Done false
16155 6302 Acked 7301 Read 6 Done New Done false
16156 6303 Acked 7302 Read 10 Done New Done false
16157 6304 Acked 7303 Read 4 Done New Done false
16158 6305 Acked 7304 Write 4 Done New Done false
16159 6306 Acked 7305 Read 1 Done New Done false
16160 6307 Acked 7306 Read 9 Done New Done false
16161 6308 Acked 7307 Read 9 Done New Done false
16162 6309 Acked 7308 Read 6 Done New Done false
16163 6310 Acked 7309 Write 10 Done New Done false
16164 6311 Acked 7310 Write 5 Done New Done false
16165 6312 Acked 7311 Write 5 Done New Done false
16166 6313 Acked 7312 Read 9 Done New Done false
16167 6314 Acked 7313 Flush 0 Done New Done false
16168 6315 Acked 7314 Write 7 Done New Done false
16169 6316 Acked 7315 Read 10 Done New Done false
16170 6317 Acked 7316 Write 2 Done New Done false
16171 6318 Acked 7317 Write 4 Done New Done false
16172 6319 Acked 7318 Flush 0 Done New Done false
16173 6320 Acked 7319 Write 7 Done New Done false
16174 6321 Acked 7320 Read 2 Done New Done false
16175 6322 Acked 7321 Flush 0 Done New Done false
16176 6323 Acked 7322 Read 5 Done New Done false
16177 6324 Acked 7323 Read 1 Done New Done false
16178 6325 Acked 7324 Flush 0 Done New Done false
16179 6326 Acked 7325 Read 7 Done New Done false
16180 6327 Acked 7326 Read 10 Done New Done false
16181 6328 Acked 7327 Read 4 Done New Done false
16182 6329 Acked 7328 Read 5 Done New Done false
16183 6330 Acked 7329 Write 9 Done New Done false
16184 6331 Acked 7330 Flush 0 Done New Done false
16185 6332 Acked 7331 Write 2 Done New Done false
16186 6333 Acked 7332 Read 8 Done New Done false
16187 6334 Acked 7333 Read 1 Done New Done false
16188 6335 Acked 7334 Write 10 Done New Done false
16189 6336 Acked 7335 Read 1 Done New Done false
16190 6337 Acked 7336 Flush 0 Done New Done false
16191 6338 Acked 7337 Write 4 Done New Done false
16192 6339 Acked 7338 Read 7 Done New Done false
16193 6340 Acked 7339 Write 3 Done New Done false
16194 6341 Acked 7340 Write 9 Done New Done false
16195 6342 Acked 7341 Flush 0 Done New Done false
16196 6343 Acked 7342 Write 9 Done New Done false
16197 6344 Acked 7343 Write 6 Done New Done false
16198 6345 Acked 7344 Read 2 Done New Done false
16199 6346 Acked 7345 Write 5 Done New Done false
16200 6347 Acked 7346 Flush 0 Done New Done false
16201 6348 Acked 7347 Read 4 Done New Done false
16202 6349 Acked 7348 Write 10 Done New Done false
16203 6350 Acked 7349 Write 1 Done New Done false
16204 6351 Acked 7350 Read 1 Done New Done false
16205 6352 Acked 7351 Write 6 Done New Done false
16206 6353 Acked 7352 Write 8 Done New Done false
16207 6354 Acked 7353 Write 4 Done New Done false
16208 6355 Acked 7354 Flush 0 Done New Done false
16209 6356 Acked 7355 Write 5 Sent New Sent false
16210 STATES DS:0 DS:1 DS:2 TOTAL
16211 New 0 302 0 302
16212 Sent 1 0 1 2
16213 Done 301 0 301 602
16214 Skipped 0 0 0 0
16215 Error 0 0 0 0
16216 Last Flush: 7354 7053 7354
16217 Downstairs last five completed: 7053 7052 7051 7050 7049 7048
16218 Upstairs last five completed: 6356 6355 6354 6353 6352 6351
16219 CLIENT: Up:0 ds:302 act:2
16220 [8820] got start action
16221 [8820] I am going to restart
16222 Starting downstairs at port 8820
16223 Make output file at "/tmp/dsc/downstairs-8820.txt"
16224 [8820][1] reports Starting
16225 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1197)
16226 [8820][1] reports Running
162272023-10-13T18:12:37.634ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 1
162282023-10-13T18:12:37.634ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
162292023-10-13T18:12:37.634ZINFOcrucible: [1] client re-new 303 jobs since flush 7053 = downstairs
162302023-10-13T18:12:37.635ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
162312023-10-13T18:12:37.636ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
162322023-10-13T18:12:37.636ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
162332023-10-13T18:12:37.636ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
162342023-10-13T18:12:37.636ZINFOcrucible: [1] send last flush ID to this DS: 7053
162352023-10-13T18:12:37.636ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
162362023-10-13T18:12:37.636ZINFOcrucible: [1] Replied this last flush ID: 7053
162372023-10-13T18:12:37.636ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Offline Active ds_transition to Replay
162382023-10-13T18:12:37.636ZINFOcrucible: [1] Transition from Offline to Replay
162392023-10-13T18:12:37.636ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
162402023-10-13T18:12:37.636ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Replay Active ds_transition to Active
162412023-10-13T18:12:37.636ZINFOcrucible: [1] Transition from Replay to Active
162422023-10-13T18:12:37.636ZINFOcrucible: [1] Starts cmd_loop
162432023-10-13T18:12:37.636ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
162442023-10-13T18:12:37.636ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
162452023-10-13T18:12:37.636ZINFOcrucible: Set check for repair
162462023-10-13T18:12:38.637ZINFOcrucible: Checking if live repair is needed
162472023-10-13T18:12:38.637ZINFOcrucible: No Live Repair required at this time
162482023-10-13T18:12:38.637ZWARNcrucible: [1] flow control end
16249 ----------------------------------------------------------------
16250 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16251 Downstairs last five completed: 7356 7355 7354 7353 7352 7351
16252 Upstairs last five completed: 6357 6356 6355 6354 6353 6352
16253 CLIENT: Up:0 ds:0 act:3
16254 Replay: All jobs finished, all DS active.
16255 Main task has work to do, go find it
16256 got dsc Stop(2)
16257 stop 2
16258 Oct 13 18:12:39.556 INFO request completed, latency_us: 488[8830] Got stop action so:false kr:false
16259 , response_code: 204, uri: /stop/cid/2, method: POST, req_id: e4f7cd29-e01a-404a-8214-b82c34390a9e, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
16260 Oct 13 18:12:39.558 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/2, method: GET, req_id: 40f9dede-1f4a-41e7-bd22-59079ab93cdb, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
162612023-10-13T18:12:39.558ZWARNcrucible: [2] downstairs disconnected
162622023-10-13T18:12:39.558ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
162632023-10-13T18:12:39.558ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8830 closed looper = 2
162642023-10-13T18:12:39.558ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
162652023-10-13T18:12:39.558ZWARNcrucible: [2] pm_task rx.recv() is None
162662023-10-13T18:12:39.558ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
162672023-10-13T18:12:39.558ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
16268 [8830] Exited with: ExitStatus(unix_wait_status(9))
16269 [8830][2] reports Exit
16270 Oct 13 18:12:43.560 INFO request completed, latency_us: 267, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1c705525-8f36-41c0-b14a-b41c6f305bda, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
16271 001/300 Write block 1104 len 16384 data: 6 6 5 6
16272 002/300 Read block 4204 len 20480
16273 003/300 Read block 3109 len 4096
16274 004/300 Read block 245 len 28672
16275 005/300 Read block 3808 len 20480
16276 006/300 Read block 2655 len 20480
16277 007/300 Write block 4801 len 20480 data: 8 8 11 10 9
16278 008/300 Write block 257 len 36864 data: 3 2 3 2 2 2 2 2 2
16279 009/300 Write block 3056 len 16384 data: 5 5 5 4
16280 010/300 Read block 1849 len 32768
16281 011/300 Flush
16282 012/300 Read block 220 len 20480
16283 013/300 Read block 1920 len 40960
16284 014/300 Flush
16285 015/300 Read block 3055 len 32768
16286 016/300 Write block 2733 len 36864 data: 8 7 7 7 5 3 3 4 5
16287 017/300 Read block 155 len 12288
16288 018/300 Flush
16289 019/300 Flush
16290 020/300 Read block 3059 len 12288
16291 021/300 Write block 3165 len 8192 data: 3 3
16292 022/300 Flush
16293 023/300 Flush
16294 024/300 Read block 4880 len 4096
16295 025/300 Write block 2156 len 4096 data: 4
16296 026/300 Write block 3655 len 24576 data: 6 6 5 4 3 6
16297 027/300 Read block 3166 len 12288
16298 028/300 Read block 2744 len 4096
16299 029/300 Read block 4615 len 36864
16300 030/300 Write block 1637 len 24576 data: 7 6 6 5 4 4
16301 031/300 Write block 42 len 36864 data: 3 4 5 5 6 8 10 7 7
16302 032/300 Write block 4880 len 40960 data: 4 3 4 6 6 5 5 3 3 3
16303 033/300 Write block 3923 len 8192 data: 4 4
16304 034/300 Flush
16305 035/300 Write block 4648 len 12288 data: 3 4 3
16306 036/300 Write block 1309 len 32768 data: 5 5 6 5 4 4 4 3
16307 037/300 Flush
16308 038/300 Write block 4727 len 28672 data: 6 6 6 4 5 5 5
16309 039/300 Read block 2852 len 4096
16310 040/300 Write block 3570 len 20480 data: 6 6 6 7 6
16311 041/300 Read block 3590 len 32768
16312 042/300 Flush
16313 043/300 Write block 1758 len 36864 data: 5 6 5 6 6 6 6 6 6
16314 044/300 Write block 2895 len 40960 data: 3 3 4 4 5 6 6 6 6 5
16315 045/300 Read block 889 len 4096
16316 046/300 Write block 3207 len 40960 data: 4 5 4 5 5 4 4 4 3 5
16317 047/300 Write block 2502 len 16384 data: 5 4 4 4
16318 048/300 Write block 443 len 20480 data: 4 3 4 7 6
16319 049/300 Write block 4901 len 8192 data: 4 3
16320 050/300 Read block 3739 len 32768
16321 051/300 Read block 247 len 32768
16322 052/300 Read block 897 len 4096
16323 053/300 Read block 2704 len 4096
16324 054/300 Read block 4732 len 32768
16325 055/300 Read block 937 len 20480
16326 056/300 Write block 1374 len 12288 data: 7 4 7
16327 057/300 Read block 3247 len 4096
16328 058/300 Read block 2842 len 20480
16329 059/300 Write block 2463 len 28672 data: 8 8 8 8 7 5 6
16330 060/300 Write block 870 len 20480 data: 4 4 5 4 3
16331 061/300 Write block 1480 len 4096 data: 5
16332 062/300 Write block 461 len 36864 data: 9 8 9 7 5 5 5 4 3
16333 063/300 Read block 2899 len 24576
16334 064/300 Write block 2471 len 40960 data: 8 7 8 7 7 5 5 5 5 4
16335 065/300 Read block 925 len 40960
16336 066/300 Write block 2560 len 12288 data: 9 11 10
16337 067/300 Write block 1402 len 36864 data: 4 5 5 5 5 5 4 4 4
16338 068/300 Flush
16339 069/300 Write block 2789 len 36864 data: 4 4 5 5 5 7 6 6 7
16340 070/300 Read block 1587 len 32768
16341 071/300 Read block 3151 len 24576
16342 072/300 Read block 4875 len 32768
16343 073/300 Write block 1869 len 32768 data: 4 3 5 6 6 6 7 8
16344 074/300 Read block 2635 len 32768
16345 075/300 Write block 1403 len 16384 data: 6 6 6 6
16346 076/300 Read block 2948 len 24576
16347 077/300 Read block 2147 len 12288
16348 078/300 Read block 2373 len 40960
16349 079/300 Write block 4447 len 28672 data: 6 7 6 6 6 5 5
16350 080/300 Flush
16351 081/300 Read block 3111 len 36864
16352 082/300 Read block 2646 len 28672
16353 083/300 Read block 4914 len 8192
16354 084/300 Write block 3613 len 24576 data: 4 4 3 4 7 8
16355 085/300 Flush
16356 086/300 Read block 445 len 16384
16357 087/300 Flush
16358 088/300 Read block 3877 len 28672
16359 089/300 Flush
16360 090/300 Read block 4530 len 20480
16361 091/300 Read block 3455 len 12288
16362 092/300 Write block 1110 len 32768 data: 5 5 6 6 5 4 6 6
16363 093/300 Write block 612 len 36864 data: 3 3 3 4 5 5 5 4 4
16364 094/300 Write block 3106 len 40960 data: 3 4 5 5 5 7 5 6 5 6
16365 095/300 Read block 3157 len 8192
16366 096/300 Write block 3933 len 40960 data: 4 5 7 5 10 10 10 8 8 8
16367 097/300 Read block 3743 len 32768
16368 098/300 Read block 3401 len 24576
16369 099/300 Read block 2577 len 28672
16370 100/300 Write block 1902 len 12288 data: 5 4 4
16371 101/300 Read block 702 len 36864
16372 102/300 Write block 4425 len 28672 data: 4 3 3 4 3 3 4
16373 103/300 Write block 4798 len 40960 data: 5 5 8 9 9 12 11 10 8 8
16374 104/300 Read block 2308 len 28672
16375 105/300 Write block 2508 len 28672 data: 8 8 6 6 6 5 4
16376 106/300 Flush
16377 107/300 Read block 2084 len 4096
16378 108/300 Write block 3874 len 16384 data: 7 6 7 6
16379 109/300 Read block 979 len 16384
16380 110/300 Write block 4735 len 40960 data: 4 4 3 3 3 6 6 7 9 10
16381 111/300 Write block 4896 len 32768 data: 2 2 2 2 2 5 4 4
16382 112/300 Read block 4288 len 4096
16383 113/300 Write block 2451 len 24576 data: 4 3 3 3 3 3
16384 114/300 Write block 1332 len 32768 data: 6 6 6 6 6 5 3 5
16385 115/300 Read block 4908 len 28672
16386 116/300 Read block 2307 len 12288
16387 117/300 Read block 3827 len 12288
16388 118/300 Read block 4257 len 32768
16389 119/300 Read block 754 len 16384
16390 120/300 Read block 4869 len 8192
16391 121/300 Write block 3325 len 16384 data: 3 3 3 3
16392 122/300 Read block 3414 len 4096
16393 123/300 Write block 1130 len 20480 data: 5 5 5 4 4
16394 124/300 Write block 2156 len 28672 data: 5 4 4 5 4 4 3
16395 125/300 Read block 730 len 12288
16396 126/300 Write block 4957 len 32768 data: 5 5 6 7 6 7 7 6
16397 127/300 Read block 3705 len 12288
16398 128/300 Write block 2639 len 24576 data: 7 5 5 5 5 4
16399 129/300 Read block 2049 len 36864
16400 130/300 Write block 2834 len 36864 data: 4 4 4 4 3 4 5 4 4
16401 131/300 Write block 3257 len 36864 data: 5 4 5 6 6 6 4 3 3
16402 132/300 Read block 4147 len 24576
16403 133/300 Write block 643 len 16384 data: 6 6 6 5
16404 134/300 Read block 4227 len 20480
16405 135/300 Flush
16406 136/300 Read block 2398 len 12288
16407 137/300 Read block 4695 len 8192
16408 138/300 Read block 1841 len 4096
16409 139/300 Read block 1550 len 16384
16410 140/300 Read block 2306 len 32768
16411 141/300 Write block 1475 len 40960 data: 4 4 7 7 7 6 5 4 4 4
16412 142/300 Read block 2677 len 4096
16413 143/300 Read block 2081 len 4096
16414 144/300 Read block 1409 len 8192
16415 145/300 Read block 4670 len 4096
16416 146/300 Read block 3203 len 24576
16417 147/300 Write block 4330 len 12288 data: 8 8 9
16418 148/300 Write block 2208 len 40960 data: 4 6 6 7 6 5 4 5 6 4
16419 149/300 Read block 1389 len 36864
16420 150/300 Flush
16421 151/300 Read block 4948 len 4096
16422 152/300 Read block 3993 len 12288
16423 153/300 Read block 2939 len 12288
16424 154/300 Write block 3630 len 28672 data: 2 3 3 3 2 2 2
16425 155/300 Write block 1543 len 8192 data: 4 3
16426 156/300 Write block 66 len 36864 data: 4 4 4 4 4 4 4 5 7
16427 157/300 Write block 414 len 16384 data: 2 3 3 5
16428 158/300 Read block 4076 len 40960
16429 159/300 Write block 1305 len 24576 data: 3 3 4 4 6 6
16430 160/300 Read block 1686 len 40960
16431 161/300 Write block 2092 len 20480 data: 4 3 3 2 3
16432 162/300 Write block 501 len 8192 data: 2 2
16433 163/300 Read block 2593 len 40960
16434 164/300 Read block 4318 len 36864
16435 165/300 Write block 2094 len 24576 data: 4 3 4 4 4 5
16436 166/300 Read block 1411 len 28672
16437 167/300 Read block 247 len 24576
16438 168/300 Read block 1329 len 4096
16439 169/300 Read block 3503 len 24576
16440 170/300 Read block 656 len 40960
16441 171/300 Write block 3095 len 8192 data: 11 9
16442 172/300 Read block 1208 len 24576
16443 173/300 Read block 3309 len 36864
16444 174/300 Write block 1940 len 32768 data: 7 6 4 4 4 4 4 4
16445 175/300 Read block 1621 len 16384
16446 176/300 Read block 4544 len 40960
16447 177/300 Read block 3467 len 20480
16448 178/300 Read block 2358 len 20480
16449 179/300 Read block 743 len 16384
16450 180/300 Write block 147 len 40960 data: 5 4 5 6 6 6 5 5 4 4
16451 181/300 Write block 2476 len 12288 data: 6 6 6
16452 182/300 Write block 4538 len 4096 data: 4
16453 183/300 Read block 4247 len 28672
16454 184/300 Read block 671 len 16384
16455 185/300 Read block 4052 len 4096
16456 186/300 Flush
16457 187/300 Write block 183 len 28672 data: 3 3 5 6 6 8 5
16458 188/300 Write block 3794 len 28672 data: 3 3 3 3 3 3 3
16459 189/300 Write block 3411 len 4096 data: 4
16460 190/300 Read block 4043 len 20480
16461 191/300 Flush
16462 192/300 Flush
16463 193/300 Write block 4690 len 4096 data: 3
16464 194/300 Read block 3459 len 36864
16465 195/300 Read block 3491 len 8192
16466 196/300 Read block 3785 len 24576
16467 197/300 Write block 957 len 32768 data: 5 6 6 6 6 5 3 2
16468 198/300 Write block 731 len 36864 data: 2 2 2 2 2 3 3 4 4
16469 199/300 Flush
16470 200/300 Read block 604 len 36864
16471 201/300 Read block 2979 len 40960
16472 202/300 Read block 2601 len 40960
16473 203/300 Read block 2519 len 24576
16474 204/300 Write block 3367 len 8192 data: 4 4
16475 205/300 Read block 4303 len 4096
16476 206/300 Write block 2597 len 20480 data: 4 4 6 6 9
16477 207/300 Write block 2860 len 36864 data: 4 3 2 4 4 5 5 6 7
16478 208/300 Read block 2131 len 36864
16479 209/300 Flush
16480 210/300 Read block 4464 len 12288
16481 211/300 Read block 4370 len 36864
16482 212/300 Read block 2523 len 16384
16483 213/300 Read block 1202 len 28672
16484 214/300 Write block 1593 len 16384 data: 4 4 4 4
16485 215/300 Write block 3477 len 28672 data: 6 5 4 4 4 3 3
16486 216/300 Write block 3286 len 12288 data: 5 6 6
16487 217/300 Flush
16488 218/300 Write block 4235 len 16384 data: 3 4 4 4
16489 219/300 Read block 3749 len 36864
16490 220/300 Read block 633 len 32768
16491 221/300 Write block 4812 len 24576 data: 7 6 6 5 5 4
16492 222/300 Read block 487 len 32768
16493 223/300 Flush
16494 224/300 Read block 3782 len 28672
16495 225/300 Read block 2858 len 24576
16496 226/300 Write block 1668 len 20480 data: 8 7 7 6 6
16497 227/300 Read block 1206 len 28672
16498 228/300 Write block 2088 len 4096 data: 4
16499 229/300 Read block 230 len 36864
16500 230/300 Read block 1497 len 24576
16501 231/300 Write block 3693 len 36864 data: 5 4 5 5 5 5 4 3 3
16502 232/300 Flush
16503 233/300 Write block 3143 len 24576 data: 8 6 10 11 9 5
16504 234/300 Write block 4559 len 8192 data: 3 3
16505 235/300 Write block 3680 len 12288 data: 6 5 5
16506 236/300 Read block 4070 len 28672
16507 237/300 Write block 1137 len 24576 data: 2 2 2 2 2 2
16508 238/300 Write block 2952 len 12288 data: 5 5 5
16509 239/300 Write block 2446 len 8192 data: 4 3
16510 240/300 Read block 1357 len 24576
16511 241/300 Write block 4149 len 36864 data: 6 7 8 8 7 7 6 6 6
16512 242/300 Flush
16513 243/300 Read block 2782 len 12288
16514 244/300 Write block 1028 len 40960 data: 5 5 6 4 5 4 4 4 5 5
16515 245/300 Read block 2037 len 28672
16516 246/300 Write block 1451 len 8192 data: 6 6
16517 247/300 Write block 2357 len 20480 data: 8 5 6 4 4
16518 248/300 Read block 1865 len 8192
16519 249/300 Flush
16520 250/300 Flush
16521 251/300 Read block 3015 len 32768
16522 252/300 Read block 2797 len 12288
16523 253/300 Write block 4339 len 8192 data: 6 5
16524 254/300 Write block 1063 len 28672 data: 7 6 6 6 7 8 8
16525 255/300 Write block 4043 len 8192 data: 7 5
16526 256/300 Write block 2330 len 40960 data: 6 6 6 4 5 5 5 5 3 3
16527 257/300 Flush
16528 258/300 Read block 524 len 16384
16529 259/300 Write block 377 len 4096 data: 5
16530 260/300 Read block 2648 len 12288
16531 261/300 Read block 4748 len 40960
16532 262/300 Read block 4009 len 40960
16533 263/300 Write block 4609 len 4096 data: 5
16534 264/300 Read block 643 len 24576
16535 265/300 Read block 2851 len 4096
16536 266/300 Read block 4965 len 4096
16537 267/300 Write block 2853 len 32768 data: 4 5 5 5 4 4 4 5
16538 268/300 Read block 2589 len 32768
16539 269/300 Write block 236 len 20480 data: 4 3 4 3 3
16540 270/300 Read block 1899 len 20480
16541 271/300 Flush
16542 272/300 Write block 2348 len 32768 data: 6 6 9 9 9 7 8 8
16543 273/300 Read block 3340 len 8192
16544 274/300 Read block 4903 len 32768
16545 275/300 Write block 3023 len 12288 data: 5 5 5
16546 276/300 Write block 1275 len 16384 data: 4 4 5 4
16547 277/300 Flush
16548 278/300 Read block 3541 len 16384
16549 279/300 Read block 4798 len 36864
16550 280/300 Read block 4611 len 40960
16551 281/300 Read block 3165 len 12288
16552 282/300 Read block 4418 len 40960
16553 283/300 Flush
16554 284/300 Read block 3770 len 40960
16555 285/300 Flush
16556 286/300 Flush
16557 287/300 Write block 1620 len 28672 data: 5 4 5 5 6 7 7
16558 288/300 Read block 2098 len 32768
16559 289/300 Write block 1556 len 24576 data: 3 5 4 3 3 5
16560 290/300 Write block 2463 len 36864 data: 9 9 9 9 8 6 7 7 9
16561 291/300 Write block 4587 len 32768 data: 4 3 4 4 4 6 6 6
16562 292/300 Write block 4865 len 20480 data: 5 5 3 4 3
16563 293/300 Write block 3281 len 20480 data: 3 4 4 4 5
16564 294/300 Write block 1714 len 36864 data: 4 5 5 4 4 6 7 6 5
16565 295/300 Write block 4328 len 20480 data: 6 8 9 9 10
16566 296/300 Write block 1925 len 4096 data: 6
16567 297/300 Read block 4290 len 24576
16568 298/300 Flush
16569 299/300 Read block 3557 len 16384
16570 300/300 Read block 2921 len 40960
16571 Main task has work to do, go find it
16572 got dsc Start(2)
16573 start 2
16574 Oct 13 18:12:44.530 INFO request completed, latency_us: 300, response_code: 204, uri: /start/cid/2, method: POST, req_id: db3602a6-1553-4617-8041-1f0e64fb7bfd, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
16575 Replay: started 2, returned:Ok(())
16576 ----------------------------------------------------------------
16577 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16578 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
16579 6358 Acked 7357 Write 4 Done Done New false
16580 6359 Acked 7358 Read 5 Done Done New false
16581 6360 Acked 7359 Read 1 Done Done New false
16582 6361 Acked 7360 Read 7 Done Done New false
16583 6362 Acked 7361 Read 5 Done Done New false
16584 6363 Acked 7362 Read 5 Done Done New false
16585 6364 Acked 7363 Write 5 Done Done New false
16586 6365 Acked 7364 Write 9 Done Done New false
16587 6366 Acked 7365 Write 4 Done Done New false
16588 6367 Acked 7366 Read 8 Done Done New false
16589 6368 Acked 7367 Flush 0 Done Done New false
16590 6369 Acked 7368 Read 5 Done Done New false
16591 6370 Acked 7369 Read 10 Done Done New false
16592 6371 Acked 7370 Flush 0 Done Done New false
16593 6372 Acked 7371 Read 8 Done Done New false
16594 6373 Acked 7372 Write 9 Done Done New false
16595 6374 Acked 7373 Read 3 Done Done New false
16596 6375 Acked 7374 Flush 0 Done Done New false
16597 6376 Acked 7375 Flush 0 Done Done New false
16598 6377 Acked 7376 Read 3 Done Done New false
16599 6378 Acked 7377 Write 2 Done Done New false
16600 6379 Acked 7378 Flush 0 Done Done New false
16601 6380 Acked 7379 Flush 0 Done Done New false
16602 6381 Acked 7380 Read 1 Done Done New false
16603 6382 Acked 7381 Write 1 Done Done New false
16604 6383 Acked 7382 Write 6 Done Done New false
16605 6384 Acked 7383 Read 3 Done Done New false
16606 6385 Acked 7384 Read 1 Done Done New false
16607 6386 Acked 7385 Read 9 Done Done New false
16608 6387 Acked 7386 Write 6 Done Done New false
16609 6388 Acked 7387 Write 9 Done Done New false
16610 6389 Acked 7388 Write 10 Done Done New false
16611 6390 Acked 7389 Write 2 Done Done New false
16612 6391 Acked 7390 Flush 0 Done Done New false
16613 6392 Acked 7391 Write 3 Done Done New false
16614 6393 Acked 7392 Write 8 Done Done New false
16615 6394 Acked 7393 Flush 0 Done Done New false
16616 6395 Acked 7394 Write 7 Done Done New false
16617 6396 Acked 7395 Read 1 Done Done New false
16618 6397 Acked 7396 Write 5 Done Done New false
16619 6398 Acked 7397 Read 8 Done Done New false
16620 6399 Acked 7398 Flush 0 Done Done New false
16621 6400 Acked 7399 Write 9 Done Done New false
16622 6401 Acked 7400 Write 10 Done Done New false
16623 6402 Acked 7401 Read 1 Done Done New false
16624 6403 Acked 7402 Write 10 Done Done New false
16625 6404 Acked 7403 Write 4 Done Done New false
16626 6405 Acked 7404 Write 5 Done Done New false
16627 6406 Acked 7405 Write 2 Done Done New false
16628 6407 Acked 7406 Read 8 Done Done New false
16629 6408 Acked 7407 Read 8 Done Done New false
16630 6409 Acked 7408 Read 1 Done Done New false
16631 6410 Acked 7409 Read 1 Done Done New false
16632 6411 Acked 7410 Read 8 Done Done New false
16633 6412 Acked 7411 Read 5 Done Done New false
16634 6413 Acked 7412 Write 3 Done Done New false
16635 6414 Acked 7413 Read 1 Done Done New false
16636 6415 Acked 7414 Read 5 Done Done New false
16637 6416 Acked 7415 Write 7 Done Done New false
16638 6417 Acked 7416 Write 5 Done Done New false
16639 6418 Acked 7417 Write 1 Done Done New false
16640 6419 Acked 7418 Write 9 Done Done New false
16641 6420 Acked 7419 Read 6 Done Done New false
16642 6421 Acked 7420 Write 10 Done Done New false
16643 6422 Acked 7421 Read 10 Done Done New false
16644 6423 Acked 7422 Write 3 Done Done New false
16645 6424 Acked 7423 Write 9 Done Done New false
16646 6425 Acked 7424 Flush 0 Done Done New false
16647 6426 Acked 7425 Write 9 Done Done New false
16648 6427 Acked 7426 Read 8 Done Done New false
16649 6428 Acked 7427 Read 6 Done Done New false
16650 6429 Acked 7428 Read 8 Done Done New false
16651 6430 Acked 7429 Write 8 Done Done New false
16652 6431 Acked 7430 Read 8 Done Done New false
16653 6432 Acked 7431 Write 4 Done Done New false
16654 6433 Acked 7432 Read 6 Done Done New false
16655 6434 Acked 7433 Read 3 Done Done New false
16656 6435 Acked 7434 Read 10 Done Done New false
16657 6436 Acked 7435 Write 7 Done Done New false
16658 6437 Acked 7436 Flush 0 Done Done New false
16659 6438 Acked 7437 Read 9 Done Done New false
16660 6439 Acked 7438 Read 7 Done Done New false
16661 6440 Acked 7439 Read 2 Done Done New false
16662 6441 Acked 7440 Write 6 Done Done New false
16663 6442 Acked 7441 Flush 0 Done Done New false
16664 6443 Acked 7442 Read 4 Done Done New false
16665 6444 Acked 7443 Flush 0 Done Done New false
16666 6445 Acked 7444 Read 7 Done Done New false
16667 6446 Acked 7445 Flush 0 Done Done New false
16668 6447 Acked 7446 Read 5 Done Done New false
16669 6448 Acked 7447 Read 3 Done Done New false
16670 6449 Acked 7448 Write 8 Done Done New false
16671 6450 Acked 7449 Write 9 Done Done New false
16672 6451 Acked 7450 Write 10 Done Done New false
16673 6452 Acked 7451 Read 2 Done Done New false
16674 6453 Acked 7452 Write 10 Done Done New false
16675 6454 Acked 7453 Read 8 Done Done New false
16676 6455 Acked 7454 Read 6 Done Done New false
16677 6456 Acked 7455 Read 7 Done Done New false
16678 6457 Acked 7456 Write 3 Done Done New false
16679 6458 Acked 7457 Read 9 Done Done New false
16680 6459 Acked 7458 Write 7 Done Done New false
16681 6460 Acked 7459 Write 10 Done Done New false
16682 6461 Acked 7460 Read 7 Done Done New false
16683 6462 Acked 7461 Write 7 Done Done New false
16684 6463 Acked 7462 Flush 0 Done Done New false
16685 6464 Acked 7463 Read 1 Done Done New false
16686 6465 Acked 7464 Write 4 Done Done New false
16687 6466 Acked 7465 Read 4 Done Done New false
16688 6467 Acked 7466 Write 10 Done Done New false
16689 6468 Acked 7467 Write 8 Done Done New false
16690 6469 Acked 7468 Read 1 Done Done New false
16691 6470 Acked 7469 Write 6 Done Done New false
16692 6471 Acked 7470 Write 8 Done Done New false
16693 6472 Acked 7471 Read 7 Done Done New false
16694 6473 Acked 7472 Read 3 Done Done New false
16695 6474 Acked 7473 Read 3 Done Done New false
16696 6475 Acked 7474 Read 8 Done Done New false
16697 6476 Acked 7475 Read 4 Done Done New false
16698 6477 Acked 7476 Read 2 Done Done New false
16699 6478 Acked 7477 Write 4 Done Done New false
16700 6479 Acked 7478 Read 1 Done Done New false
16701 6480 Acked 7479 Write 5 Done Done New false
16702 6481 Acked 7480 Write 7 Done Done New false
16703 6482 Acked 7481 Read 3 Done Done New false
16704 6483 Acked 7482 Write 8 Done Done New false
16705 6484 Acked 7483 Read 3 Done Done New false
16706 6485 Acked 7484 Write 6 Done Done New false
16707 6486 Acked 7485 Read 9 Done Done New false
16708 6487 Acked 7486 Write 9 Done Done New false
16709 6488 Acked 7487 Write 9 Done Done New false
16710 6489 Acked 7488 Read 6 Done Done New false
16711 6490 Acked 7489 Write 4 Done Done New false
16712 6491 Acked 7490 Read 5 Done Done New false
16713 6492 Acked 7491 Flush 0 Done Done New false
16714 6493 Acked 7492 Read 3 Done Done New false
16715 6494 Acked 7493 Read 2 Done Done New false
16716 6495 Acked 7494 Read 1 Done Done New false
16717 6496 Acked 7495 Read 4 Done Done New false
16718 6497 Acked 7496 Read 8 Done Done New false
16719 6498 Acked 7497 Write 10 Done Done New false
16720 6499 Acked 7498 Read 1 Done Done New false
16721 6500 Acked 7499 Read 1 Done Done New false
16722 6501 Acked 7500 Read 2 Done Done New false
16723 6502 Acked 7501 Read 1 Done Done New false
16724 6503 Acked 7502 Read 6 Done Done New false
16725 6504 Acked 7503 Flush 0 Done Done New false
16726 6505 Acked 7504 Write 3 Done Done New false
16727 6506 Acked 7505 Write 10 Done Done New false
16728 6507 Acked 7506 Read 9 Done Done New false
16729 6508 Acked 7507 Flush 0 Done Done New false
16730 6509 Acked 7508 Read 1 Done Done New false
16731 6510 Acked 7509 Read 3 Done Done New false
16732 6511 Acked 7510 Read 3 Done Done New false
16733 6512 Acked 7511 Write 7 Done Done New false
16734 6513 Acked 7512 Write 2 Done Done New false
16735 6514 Acked 7513 Write 9 Done Done New false
16736 6515 Acked 7514 Write 4 Done Done New false
16737 6516 Acked 7515 Read 10 Done Done New false
16738 6517 Acked 7516 Write 6 Done Done New false
16739 6518 Acked 7517 Read 10 Done Done New false
16740 6519 Acked 7518 Write 5 Done Done New false
16741 6520 Acked 7519 Write 2 Done Done New false
16742 6521 Acked 7520 Read 10 Done Done New false
16743 6522 Acked 7521 Read 9 Done Done New false
16744 6523 Acked 7522 Write 6 Done Done New false
16745 6524 Acked 7523 Read 7 Done Done New false
16746 6525 Acked 7524 Read 6 Done Done New false
16747 6526 Acked 7525 Read 1 Done Done New false
16748 6527 Acked 7526 Read 6 Done Done New false
16749 6528 Acked 7527 Read 10 Done Done New false
16750 6529 Acked 7528 Write 2 Done Done New false
16751 6530 Acked 7529 Read 6 Done Done New false
16752 6531 Acked 7530 Read 9 Done Done New false
16753 6532 Acked 7531 Write 8 Done Done New false
16754 6533 Acked 7532 Read 4 Done Done New false
16755 6534 Acked 7533 Read 10 Done Done New false
16756 6535 Acked 7534 Read 5 Done Done New false
16757 6536 Acked 7535 Read 5 Done Done New false
16758 6537 Acked 7536 Read 4 Done Done New false
16759 6538 Acked 7537 Write 10 Done Done New false
16760 6539 Acked 7538 Write 3 Done Done New false
16761 6540 Acked 7539 Write 1 Done Done New false
16762 6541 Acked 7540 Read 7 Done Done New false
16763 6542 Acked 7541 Read 4 Done Done New false
16764 6543 Acked 7542 Read 1 Done Done New false
16765 6544 Acked 7543 Flush 0 Done Done New false
16766 6545 Acked 7544 Write 7 Done Done New false
16767 6546 Acked 7545 Write 7 Done Done New false
16768 6547 Acked 7546 Write 1 Done Done New false
16769 6548 Acked 7547 Read 5 Done Done New false
16770 6549 Acked 7548 Flush 0 Done Done New false
16771 6550 Acked 7549 Flush 0 Done Done New false
16772 6551 Acked 7550 Write 1 Done Done New false
16773 6552 Acked 7551 Read 9 Done Done New false
16774 6553 Acked 7552 Read 2 Done Done New false
16775 6554 Acked 7553 Read 6 Done Done New false
16776 6555 Acked 7554 Write 8 Done Done New false
16777 6556 Acked 7555 Write 9 Done Done New false
16778 6557 Acked 7556 Flush 0 Done Done New false
16779 6558 Acked 7557 Read 9 Done Done New false
16780 6559 Acked 7558 Read 10 Done Done New false
16781 6560 Acked 7559 Read 10 Done Done New false
16782 6561 Acked 7560 Read 6 Done Done New false
16783 6562 Acked 7561 Write 2 Done Done New false
16784 6563 Acked 7562 Read 1 Done Done New false
16785 6564 Acked 7563 Write 5 Done Done New false
16786 6565 Acked 7564 Write 9 Done Done New false
16787 6566 Acked 7565 Read 9 Done Done New false
16788 6567 Acked 7566 Flush 0 Done Done New false
16789 6568 Acked 7567 Read 3 Done Done New false
16790 6569 Acked 7568 Read 9 Done Done New false
16791 6570 Acked 7569 Read 4 Done Done New false
16792 6571 Acked 7570 Read 7 Done Done New false
16793 6572 Acked 7571 Write 4 Done Done New false
16794 6573 Acked 7572 Write 7 Done Done New false
16795 6574 Acked 7573 Write 3 Done Done New false
16796 6575 Acked 7574 Flush 0 Done Done New false
16797 6576 Acked 7575 Write 4 Done Done New false
16798 6577 Acked 7576 Read 9 Done Done New false
16799 6578 Acked 7577 Read 8 Done Done New false
16800 6579 Acked 7578 Write 6 Done Done New false
16801 6580 Acked 7579 Read 8 Done Done New false
16802 6581 Acked 7580 Flush 0 Done Done New false
16803 6582 Acked 7581 Read 7 Done Done New false
16804 6583 Acked 7582 Read 6 Done Done New false
16805 6584 Acked 7583 Write 5 Done Done New false
16806 6585 Acked 7584 Read 7 Done Done New false
16807 6586 Acked 7585 Write 1 Done Done New false
16808 6587 Acked 7586 Read 9 Done Done New false
16809 6588 Acked 7587 Read 6 Done Done New false
16810 6589 Acked 7588 Write 9 Done Done New false
16811 6590 Acked 7589 Flush 0 Done Done New false
16812 6591 Acked 7590 Write 6 Done Done New false
16813 6592 Acked 7591 Write 2 Done Done New false
16814 6593 Acked 7592 Write 3 Done Done New false
16815 6594 Acked 7593 Read 7 Done Done New false
16816 6595 Acked 7594 Write 6 Done Done New false
16817 6596 Acked 7595 Write 3 Done Done New false
16818 6597 Acked 7596 Write 2 Done Done New false
16819 6598 Acked 7597 Read 6 Done Done New false
16820 6599 Acked 7598 Write 9 Done Done New false
16821 6600 Acked 7599 Flush 0 Done Done New false
16822 6601 Acked 7600 Read 3 Done Done New false
16823 6602 Acked 7601 Write 10 Done Done New false
16824 6603 Acked 7602 Read 7 Done Done New false
16825 6604 Acked 7603 Write 2 Done Done New false
16826 6605 Acked 7604 Write 5 Done Done New false
16827 6606 Acked 7605 Read 2 Done Done New false
16828 6607 Acked 7606 Flush 0 Done Done New false
16829 6608 Acked 7607 Flush 0 Done Done New false
16830 6609 Acked 7608 Read 8 Done Done New false
16831 6610 Acked 7609 Read 3 Done Done New false
16832 6611 Acked 7610 Write 2 Done Done New false
16833 6612 Acked 7611 Write 7 Done Done New false
16834 6613 Acked 7612 Write 2 Done Done New false
16835 6614 Acked 7613 Write 10 Done Done New false
16836 6615 Acked 7614 Flush 0 Done Done New false
16837 6616 Acked 7615 Read 4 Done Done New false
16838 6617 Acked 7616 Write 1 Done Done New false
16839 6618 Acked 7617 Read 3 Done Done New false
16840 6619 Acked 7618 Read 10 Done Done New false
16841 6620 Acked 7619 Read 10 Done Done New false
16842 6621 Acked 7620 Write 1 Done Done New false
16843 6622 Acked 7621 Read 6 Done Done New false
16844 6623 Acked 7622 Read 1 Done Done New false
16845 6624 Acked 7623 Read 1 Done Done New false
16846 6625 Acked 7624 Write 8 Done Done New false
16847 6626 Acked 7625 Read 8 Done Done New false
16848 6627 Acked 7626 Write 5 Done Done New false
16849 6628 Acked 7627 Read 5 Done Done New false
16850 6629 Acked 7628 Flush 0 Done Done New false
16851 6630 Acked 7629 Write 8 Done Done New false
16852 6631 Acked 7630 Read 2 Done Done New false
16853 6632 Acked 7631 Read 8 Done Done New false
16854 6633 Acked 7632 Write 3 Done Done New false
16855 6634 Acked 7633 Write 4 Done Done New false
16856 6635 Acked 7634 Flush 0 Done Done New false
16857 6636 Acked 7635 Read 4 Done Done New false
16858 6637 Acked 7636 Read 9 Done Done New false
16859 6638 Acked 7637 Read 10 Done Done New false
16860 6639 Acked 7638 Read 3 Done Done New false
16861 6640 Acked 7639 Read 10 Done Done New false
16862 6641 Acked 7640 Flush 0 Done Done New false
16863 6642 Acked 7641 Read 10 Done Done New false
16864 6643 Acked 7642 Flush 0 Done Done New false
16865 6644 Acked 7643 Flush 0 Done Done New false
16866 6645 Acked 7644 Write 7 Done Done New false
16867 6646 Acked 7645 Read 8 Done Done New false
16868 6647 Acked 7646 Write 6 Done Done New false
16869 6648 Acked 7647 Write 9 Done Done New false
16870 6649 Acked 7648 Write 8 Done Done New false
16871 6650 Acked 7649 Write 5 Done Done New false
16872 6651 Acked 7650 Write 5 Done Done New false
16873 6652 Acked 7651 Write 9 Done Done New false
16874 6653 Acked 7652 Write 5 Done Done New false
16875 6654 Acked 7653 Write 1 Done Done New false
16876 6655 Acked 7654 Read 6 Done Done New false
16877 6656 Acked 7655 Flush 0 Done Done New false
16878 6657 Acked 7656 Read 4 Done Done New false
16879 6658 Acked 7657 Read 10 Done Done New false
16880 STATES DS:0 DS:1 DS:2 TOTAL
16881 New 0 0 301 301
16882 Sent 0 0 0 0
16883 Done 301 301 0 602
16884 Skipped 0 0 0 0
16885 Error 0 0 0 0
16886 Last Flush: 7655 7655 7356
16887 Downstairs last five completed: 7356 7355 7354 7353 7352 7351
16888 Upstairs last five completed: 6658 6657 6656 6655 6654 6653
16889 CLIENT: Up:0 ds:301 act:2
168902023-10-13T18:12:44.564ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
16891 [8830] got start action
16892 [8830] I am going to restart
16893 Starting downstairs at port 8830
16894 Make output file at "/tmp/dsc/downstairs-8830.txt"
16895 [8830][2] reports Starting
16896 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1198)
16897 [8830][2] reports Running
168982023-10-13T18:12:46.567ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 2
168992023-10-13T18:12:46.567ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
169002023-10-13T18:12:46.567ZINFOcrucible: [2] client re-new 302 jobs since flush 7356 = downstairs
169012023-10-13T18:12:46.568ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
169022023-10-13T18:12:46.569ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
169032023-10-13T18:12:46.569ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
169042023-10-13T18:12:46.569ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
169052023-10-13T18:12:46.569ZINFOcrucible: [2] send last flush ID to this DS: 7356
169062023-10-13T18:12:46.569ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
169072023-10-13T18:12:46.569ZINFOcrucible: [2] Replied this last flush ID: 7356
169082023-10-13T18:12:46.569ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Offline ds_transition to Replay
169092023-10-13T18:12:46.569ZINFOcrucible: [2] Transition from Offline to Replay
169102023-10-13T18:12:46.569ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
169112023-10-13T18:12:46.569ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Replay ds_transition to Active
169122023-10-13T18:12:46.569ZINFOcrucible: [2] Transition from Replay to Active
169132023-10-13T18:12:46.569ZINFOcrucible: [2] Starts cmd_loop
169142023-10-13T18:12:46.569ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
169152023-10-13T18:12:46.570ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
169162023-10-13T18:12:46.570ZINFOcrucible: Set check for repair
169172023-10-13T18:12:47.570ZWARNcrucible: [2] flow control end
169182023-10-13T18:12:47.571ZINFOcrucible: Checking if live repair is needed
169192023-10-13T18:12:47.571ZINFOcrucible: No Live Repair required at this time
16920 ----------------------------------------------------------------
16921 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16922 Downstairs last five completed: 7658 7657 7656 7655 7654 7653
16923 Upstairs last five completed: 6659 6658 6657 6656 6655 6654
16924 CLIENT: Up:0 ds:0 act:3
16925 Replay: All jobs finished, all DS active.
16926 Main task has work to do, go find it
16927 got dsc Stop(2)
16928 stop 2
16929 Oct 13 18:12:48.535 INFO request completed, [8830] Got stop action so:false kr:false
16930 latency_us: 511, response_code: 204, uri: /stop/cid/2, method: POST, req_id: d18a58c2-0e9f-4e68-af62-a306f56979ab, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
16931 Oct 13 18:12:48.537 INFO request completed, latency_us: 295, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1830dca0-1635-4451-afbe-676b608281c9, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
169322023-10-13T18:12:48.537ZWARNcrucible: [2] downstairs disconnected
169332023-10-13T18:12:48.537ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
169342023-10-13T18:12:48.537ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8830 closed looper = 2
169352023-10-13T18:12:48.538ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
169362023-10-13T18:12:48.538ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
169372023-10-13T18:12:48.538ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
169382023-10-13T18:12:48.538ZWARNcrucible: [2] pm_task rx.recv() is None
16939 [8830] Exited with: ExitStatus(unix_wait_status(9))
16940 [8830][2] reports Exit
16941 Oct 13 18:12:52.539 INFO request completed, latency_us: 261, response_code: 200, uri: /state/cid/2, method: GET, req_id: 88717329-13bc-4032-a853-d460b5eec322, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
16942 001/300 Flush
16943 002/300 Write block 3926 len 24576 data: 4 4 4 5 5 4
16944 003/300 Flush
16945 004/300 Write block 3703 len 32768 data: 3 5 5 7 7 9 9 13
16946 005/300 Write block 2467 len 32768 data: 9 7 8 8 10 8 9 8
16947 006/300 Write block 4457 len 16384 data: 4 5 6 4
16948 007/300 Read block 4422 len 32768
16949 008/300 Write block 1071 len 8192 data: 7 6
16950 009/300 Write block 1836 len 40960 data: 7 8 7 4 4 4 3 3 2 2
16951 010/300 Write block 2780 len 36864 data: 5 5 6 6 4 4 4 5 5
16952 011/300 Read block 499 len 24576
16953 012/300 Write block 1204 len 32768 data: 5 4 3 3 5 5 5 5
16954 013/300 Write block 1959 len 40960 data: 3 4 4 4 4 4 4 4 3 3
16955 014/300 Flush
16956 015/300 Write block 1784 len 4096 data: 6
16957 016/300 Read block 1246 len 28672
16958 017/300 Flush
16959 018/300 Write block 4818 len 40960 data: 4 4 4 3 5 5 6 7 6 5
16960 019/300 Read block 550 len 20480
16961 020/300 Read block 1875 len 12288
16962 021/300 Write block 123 len 28672 data: 5 6 6 5 6 8 8
16963 022/300 Flush
16964 023/300 Write block 4424 len 32768 data: 5 5 4 4 5 4 4 5
16965 024/300 Read block 3597 len 16384
16966 025/300 Read block 2024 len 20480
16967 026/300 Read block 3149 len 16384
16968 027/300 Read block 4334 len 32768
16969 028/300 Write block 4889 len 28672 data: 4 6 5 6 5 4 3
16970 029/300 Read block 2869 len 40960
16971 030/300 Write block 583 len 28672 data: 3 3 4 4 5 4 5
16972 031/300 Read block 4868 len 40960
16973 032/300 Read block 502 len 36864
16974 033/300 Read block 3817 len 24576
16975 034/300 Write block 2781 len 20480 data: 6 7 7 5 5
16976 035/300 Write block 12 len 16384 data: 4 3 5 6
16977 036/300 Read block 1121 len 16384
16978 037/300 Read block 3955 len 32768
16979 038/300 Read block 1914 len 36864
16980 039/300 Read block 3209 len 32768
16981 040/300 Read block 4558 len 24576
16982 041/300 Read block 4009 len 12288
16983 042/300 Read block 718 len 40960
16984 043/300 Read block 3884 len 24576
16985 044/300 Write block 4857 len 40960 data: 6 7 9 10 7 7 5 5 6 6
16986 045/300 Flush
16987 046/300 Read block 3724 len 20480
16988 047/300 Write block 1060 len 12288 data: 6 5 6
16989 048/300 Read block 2135 len 24576
16990 049/300 Read block 3715 len 16384
16991 050/300 Write block 904 len 20480 data: 4 5 5 6 6
16992 051/300 Read block 4828 len 20480
16993 052/300 Read block 1760 len 40960
16994 053/300 Write block 1303 len 32768 data: 3 3 4 4 5 5 7 7
16995 054/300 Write block 4023 len 28672 data: 4 4 4 3 2 2 2
16996 055/300 Write block 3872 len 24576 data: 6 5 8 7 8 7
16997 056/300 Write block 4144 len 36864 data: 3 3 5 5 5 7 8 9 9
16998 057/300 Read block 544 len 24576
16999 058/300 Read block 3349 len 12288
17000 059/300 Read block 4193 len 12288
17001 060/300 Write block 4670 len 4096 data: 6
17002 061/300 Read block 136 len 8192
17003 062/300 Write block 1159 len 12288 data: 6 9 8
17004 063/300 Write block 688 len 20480 data: 4 4 4 6 6
17005 064/300 Flush
17006 065/300 Read block 4675 len 24576
17007 066/300 Read block 1455 len 24576
17008 067/300 Write block 3832 len 28672 data: 5 5 3 4 4 4 4
17009 068/300 Flush
17010 069/300 Read block 4948 len 24576
17011 070/300 Read block 3432 len 40960
17012 071/300 Read block 1728 len 8192
17013 072/300 Read block 4949 len 20480
17014 073/300 Write block 3327 len 8192 data: 4 4
17015 074/300 Read block 3729 len 4096
17016 075/300 Write block 607 len 40960 data: 6 5 4 5 3 4 4 4 5 6
17017 076/300 Read block 1847 len 24576
17018 077/300 Read block 658 len 16384
17019 078/300 Read block 4549 len 20480
17020 079/300 Read block 4143 len 12288
17021 080/300 Write block 2374 len 20480 data: 2 2 2 2 3
17022 081/300 Write block 189 len 28672 data: 6 6 6 4 5 5 4
17023 082/300 Flush
17024 083/300 Write block 4161 len 4096 data: 6
17025 084/300 Write block 4774 len 4096 data: 4
17026 085/300 Write block 1432 len 16384 data: 6 6 5 4
17027 086/300 Write block 3423 len 20480 data: 3 4 6 5 4
17028 087/300 Read block 4817 len 4096
17029 088/300 Write block 4385 len 24576 data: 6 7 7 6 5 6
17030 089/300 Write block 378 len 8192 data: 7 7
17031 090/300 Flush
17032 091/300 Write block 3322 len 28672 data: 4 3 4 4 4 5 5
17033 092/300 Read block 260 len 36864
17034 093/300 Read block 1408 len 12288
17035 094/300 Write block 1077 len 24576 data: 3 3 3 3 4 3
17036 095/300 Read block 2952 len 4096
17037 096/300 Flush
17038 097/300 Flush
17039 098/300 Write block 2558 len 24576 data: 9 9 10 12 11 7
17040 099/300 Read block 1727 len 36864
17041 100/300 Read block 4184 len 36864
17042 101/300 Read block 793 len 40960
17043 102/300 Write block 721 len 32768 data: 4 5 4 4 4 4 4 3
17044 103/300 Read block 4145 len 32768
17045 104/300 Flush
17046 105/300 Read block 4873 len 32768
17047 106/300 Read block 2013 len 40960
17048 107/300 Write block 4198 len 16384 data: 4 5 6 5
17049 108/300 Read block 2073 len 32768
17050 109/300 Write block 2442 len 16384 data: 5 5 3 3
17051 110/300 Flush
17052 111/300 Flush
17053 112/300 Write block 1603 len 28672 data: 3 5 5 4 4 2 2
17054 113/300 Read block 4677 len 12288
17055 114/300 Read block 4565 len 32768
17056 115/300 Read block 2229 len 16384
17057 116/300 Read block 4321 len 4096
17058 117/300 Write block 877 len 20480 data: 2 2 2 2 5
17059 118/300 Read block 327 len 32768
17060 119/300 Read block 2417 len 36864
17061 120/300 Write block 998 len 8192 data: 5 5
17062 121/300 Read block 1885 len 20480
17063 122/300 Read block 816 len 32768
17064 123/300 Read block 478 len 8192
17065 124/300 Read block 2894 len 24576
17066 125/300 Write block 4039 len 40960 data: 3 3 4 6 8 6 6 5 5 4
17067 126/300 Write block 423 len 16384 data: 3 4 4 3
17068 127/300 Write block 2234 len 32768 data: 5 5 5 6 7 8 9 7
17069 128/300 Read block 298 len 32768
17070 129/300 Read block 1204 len 4096
17071 130/300 Read block 77 len 40960
17072 131/300 Read block 1255 len 40960
17073 132/300 Flush
17074 133/300 Write block 2531 len 32768 data: 7 7 6 4 4 5 4 4
17075 134/300 Read block 63 len 40960
17076 135/300 Write block 366 len 24576 data: 5 3 3 3 5 5
17077 136/300 Read block 2195 len 4096
17078 137/300 Read block 1401 len 24576
17079 138/300 Write block 1424 len 8192 data: 4 5
17080 139/300 Read block 1177 len 20480
17081 140/300 Write block 29 len 12288 data: 4 4 4
17082 141/300 Read block 1943 len 32768
17083 142/300 Read block 4020 len 40960
17084 143/300 Read block 1374 len 40960
17085 144/300 Write block 350 len 12288 data: 7 6 5
17086 145/300 Read block 2725 len 40960
17087 146/300 Read block 4696 len 28672
17088 147/300 Write block 2765 len 20480 data: 6 6 6 4 5
17089 148/300 Read block 1822 len 16384
17090 149/300 Read block 2193 len 8192
17091 150/300 Write block 2230 len 20480 data: 2 3 3 3 6
17092 151/300 Write block 4217 len 24576 data: 6 6 7 7 6 6
17093 152/300 Write block 306 len 16384 data: 4 4 4 4
17094 153/300 Read block 3080 len 20480
17095 154/300 Read block 1406 len 4096
17096 155/300 Read block 4872 len 32768
17097 156/300 Flush
17098 157/300 Read block 2040 len 4096
17099 158/300 Read block 85 len 12288
17100 159/300 Read block 273 len 24576
17101 160/300 Flush
17102 161/300 Read block 2877 len 12288
17103 162/300 Write block 2019 len 16384 data: 5 5 5 6
17104 163/300 Read block 1166 len 20480
17105 164/300 Read block 2501 len 40960
17106 165/300 Write block 2698 len 12288 data: 5 4 4
17107 166/300 Read block 4798 len 36864
17108 167/300 Write block 37 len 4096 data: 6
17109 168/300 Read block 8 len 16384
17110 169/300 Flush
17111 170/300 Read block 1740 len 20480
17112 171/300 Flush
17113 172/300 Write block 3550 len 4096 data: 5
17114 173/300 Read block 4450 len 36864
17115 174/300 Read block 2690 len 20480
17116 175/300 Read block 2052 len 40960
17117 176/300 Write block 3977 len 32768 data: 3 3 3 4 5 5 6 6
17118 177/300 Read block 1379 len 20480
17119 178/300 Read block 1458 len 32768
17120 179/300 Read block 2091 len 20480
17121 180/300 Read block 2988 len 12288
17122 181/300 Write block 4688 len 24576 data: 3 2 4 3 3 4
17123 182/300 Write block 4778 len 28672 data: 6 7 7 7 5 4 5
17124 183/300 Read block 4060 len 28672
17125 184/300 Read block 326 len 20480
17126 185/300 Read block 4012 len 20480
17127 186/300 Read block 3608 len 32768
17128 187/300 Write block 1926 len 28672 data: 6 6 5 5 3 6 7
17129 188/300 Read block 2076 len 20480
17130 189/300 Write block 1075 len 32768 data: 4 5 4 4 4 4 5 4
17131 190/300 Write block 4287 len 40960 data: 5 5 3 3 3 3 2 2 2 3
17132 191/300 Write block 443 len 16384 data: 5 4 5 8
17133 192/300 Write block 2899 len 4096 data: 6
17134 193/300 Read block 3206 len 40960
17135 194/300 Flush
17136 195/300 Read block 1464 len 16384
17137 196/300 Write block 3687 len 40960 data: 5 5 6 4 4 5 6 5 6 6
17138 197/300 Write block 2158 len 4096 data: 5
17139 198/300 Write block 885 len 16384 data: 8 7 7 8
17140 199/300 Flush
17141 200/300 Read block 1033 len 20480
17142 201/300 Write block 503 len 4096 data: 2
17143 202/300 Read block 4731 len 24576
17144 203/300 Write block 2346 len 4096 data: 6
17145 204/300 Read block 1057 len 28672
17146 205/300 Read block 3581 len 20480
17147 206/300 Write block 3380 len 12288 data: 5 5 5
17148 207/300 Read block 76 len 28672
17149 208/300 Read block 3067 len 40960
17150 209/300 Read block 1803 len 24576
17151 210/300 Read block 1005 len 8192
17152 211/300 Write block 1148 len 12288 data: 7 7 5
17153 212/300 Read block 3179 len 24576
17154 213/300 Write block 2082 len 36864 data: 3 2 2 3 3 3 5 3 3
17155 214/300 Flush
17156 215/300 Write block 362 len 20480 data: 6 5 5 5 6
17157 216/300 Write block 2362 len 16384 data: 5 5 6 6
17158 217/300 Write block 2143 len 24576 data: 8 9 9 8 6 5
17159 218/300 Read block 802 len 24576
17160 219/300 Write block 4269 len 24576 data: 4 4 4 5 4 4
17161 220/300 Read block 1367 len 28672
17162 221/300 Write block 1436 len 24576 data: 2 2 2 2 2 3
17163 222/300 Read block 1231 len 8192
17164 223/300 Read block 956 len 40960
17165 224/300 Read block 2400 len 4096
17166 225/300 Write block 1439 len 40960 data: 3 3 4 6 4 4 5 5 5 8
17167 226/300 Read block 1241 len 32768
17168 227/300 Read block 4477 len 12288
17169 228/300 Read block 1382 len 20480
17170 229/300 Write block 4176 len 24576 data: 7 6 5 4 6 4
17171 230/300 Flush
17172 231/300 Flush
17173 232/300 Write block 2563 len 32768 data: 8 6 6 5 5 5 6 6
17174 233/300 Write block 3739 len 28672 data: 6 5 6 8 8 7 7
17175 234/300 Flush
17176 235/300 Read block 2897 len 8192
17177 236/300 Read block 887 len 36864
17178 237/300 Read block 4769 len 28672
17179 238/300 Write block 239 len 40960 data: 4 4 3 3 4 4 5 4 4 4
17180 239/300 Write block 2464 len 24576 data: 10 10 10 10 8 9
17181 240/300 Read block 219 len 40960
17182 241/300 Read block 254 len 20480
17183 242/300 Write block 1628 len 36864 data: 7 6 4 5 7 6 8 7 7
17184 243/300 Flush
17185 244/300 Read block 2357 len 16384
17186 245/300 Write block 2002 len 24576 data: 3 3 5 5 5 6
17187 246/300 Read block 1603 len 40960
17188 247/300 Read block 579 len 36864
17189 248/300 Write block 4287 len 24576 data: 6 6 4 4 4 4
17190 249/300 Read block 3121 len 36864
17191 250/300 Read block 1281 len 28672
17192 251/300 Write block 4838 len 4096 data: 4
17193 252/300 Write block 1627 len 36864 data: 7 8 7 5 6 8 7 9 8
17194 253/300 Read block 2087 len 36864
17195 254/300 Write block 675 len 24576 data: 3 4 4 4 4 4
17196 255/300 Flush
17197 256/300 Read block 433 len 20480
17198 257/300 Read block 2379 len 20480
17199 258/300 Read block 3028 len 40960
17200 259/300 Write block 2190 len 40960 data: 4 5 6 6 7 8 7 6 6 5
17201 260/300 Flush
17202 261/300 Read block 2007 len 40960
17203 262/300 Read block 1939 len 40960
17204 263/300 Read block 3673 len 28672
17205 264/300 Write block 3909 len 4096 data: 6
17206 265/300 Write block 4970 len 8192 data: 5 5
17207 266/300 Read block 4928 len 20480
17208 267/300 Write block 3530 len 28672 data: 3 4 4 6 8 7 8
17209 268/300 Read block 1251 len 24576
17210 269/300 Write block 1282 len 24576 data: 4 4 4 3 3 3
17211 270/300 Write block 2528 len 8192 data: 6 6
17212 271/300 Read block 2485 len 16384
17213 272/300 Read block 4651 len 16384
17214 273/300 Flush
17215 274/300 Read block 951 len 36864
17216 275/300 Read block 3851 len 16384
17217 276/300 Write block 1387 len 20480 data: 6 5 5 6 6
17218 277/300 Read block 4347 len 20480
17219 278/300 Read block 1015 len 32768
17220 279/300 Flush
17221 280/300 Read block 4906 len 4096
17222 281/300 Read block 3607 len 12288
17223 282/300 Read block 4021 len 12288
17224 283/300 Flush
17225 284/300 Write block 2426 len 4096 data: 3
17226 285/300 Write block 273 len 32768 data: 3 4 4 3 3 6 7 7
17227 286/300 Flush
17228 287/300 Write block 3569 len 16384 data: 8 7 7 7
17229 288/300 Write block 2381 len 20480 data: 5 5 5 5 5
17230 289/300 Write block 270 len 40960 data: 5 3 3 4 5 5 4 4 7 8
17231 290/300 Read block 4417 len 32768
17232 291/300 Write block 1836 len 4096 data: 8
17233 292/300 Read block 4443 len 28672
17234 293/300 Read block 4551 len 36864
17235 294/300 Write block 4697 len 32768 data: 4 4 5 5 7 9 7 5
17236 295/300 Read block 533 len 28672
17237 296/300 Read block 948 len 36864
17238 297/300 Read block 3378 len 12288
17239 298/300 Read block 4644 len 16384
17240 299/300 Write block 2833 len 40960 data: 4 5 5 5 5 4 5 6 5 5
17241 300/300 Write block 886 len 16384 data: 8 8 9 7
17242 Main task has work to do, go find it
17243 got dsc Start(2)
17244 start 2
17245 Oct 13 18:12:53.546 INFO request completed, latency_us: 303, response_code: 204, uri: /start/cid/2, method: POST, req_id: cd9877ef-c26c-4a4f-8758-0eeb118e3899, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
17246 Replay: started 2, returned:Ok(())
17247 ----------------------------------------------------------------
17248 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
17249 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
17250 6660 Acked 7659 Flush 0 Done Done New false
17251 6661 Acked 7660 Write 6 Done Done New false
17252 6662 Acked 7661 Flush 0 Done Done New false
17253 6663 Acked 7662 Write 8 Done Done New false
17254 6664 Acked 7663 Write 8 Done Done New false
17255 6665 Acked 7664 Write 4 Done Done New false
17256 6666 Acked 7665 Read 8 Done Done New false
17257 6667 Acked 7666 Write 2 Done Done New false
17258 6668 Acked 7667 Write 10 Done Done New false
17259 6669 Acked 7668 Write 9 Done Done New false
17260 6670 Acked 7669 Read 6 Done Done New false
17261 6671 Acked 7670 Write 8 Done Done New false
17262 6672 Acked 7671 Write 10 Done Done New false
17263 6673 Acked 7672 Flush 0 Done Done New false
17264 6674 Acked 7673 Write 1 Done Done New false
17265 6675 Acked 7674 Read 7 Done Done New false
17266 6676 Acked 7675 Flush 0 Done Done New false
17267 6677 Acked 7676 Write 10 Done Done New false
17268 6678 Acked 7677 Read 5 Done Done New false
17269 6679 Acked 7678 Read 3 Done Done New false
17270 6680 Acked 7679 Write 7 Done Done New false
17271 6681 Acked 7680 Flush 0 Done Done New false
17272 6682 Acked 7681 Write 8 Done Done New false
17273 6683 Acked 7682 Read 4 Done Done New false
17274 6684 Acked 7683 Read 5 Done Done New false
17275 6685 Acked 7684 Read 4 Done Done New false
17276 6686 Acked 7685 Read 8 Done Done New false
17277 6687 Acked 7686 Write 7 Done Done New false
17278 6688 Acked 7687 Read 10 Done Done New false
17279 6689 Acked 7688 Write 7 Done Done New false
17280 6690 Acked 7689 Read 10 Done Done New false
17281 6691 Acked 7690 Read 9 Done Done New false
17282 6692 Acked 7691 Read 6 Done Done New false
17283 6693 Acked 7692 Write 5 Done Done New false
17284 6694 Acked 7693 Write 4 Done Done New false
17285 6695 Acked 7694 Read 4 Done Done New false
17286 6696 Acked 7695 Read 8 Done Done New false
17287 6697 Acked 7696 Read 9 Done Done New false
17288 6698 Acked 7697 Read 8 Done Done New false
17289 6699 Acked 7698 Read 6 Done Done New false
17290 6700 Acked 7699 Read 3 Done Done New false
17291 6701 Acked 7700 Read 10 Done Done New false
17292 6702 Acked 7701 Read 6 Done Done New false
17293 6703 Acked 7702 Write 10 Done Done New false
17294 6704 Acked 7703 Flush 0 Done Done New false
17295 6705 Acked 7704 Read 5 Done Done New false
17296 6706 Acked 7705 Write 3 Done Done New false
17297 6707 Acked 7706 Read 6 Done Done New false
17298 6708 Acked 7707 Read 4 Done Done New false
17299 6709 Acked 7708 Write 5 Done Done New false
17300 6710 Acked 7709 Read 5 Done Done New false
17301 6711 Acked 7710 Read 10 Done Done New false
17302 6712 Acked 7711 Write 8 Done Done New false
17303 6713 Acked 7712 Write 7 Done Done New false
17304 6714 Acked 7713 Write 6 Done Done New false
17305 6715 Acked 7714 Write 9 Done Done New false
17306 6716 Acked 7715 Read 6 Done Done New false
17307 6717 Acked 7716 Read 3 Done Done New false
17308 6718 Acked 7717 Read 3 Done Done New false
17309 6719 Acked 7718 Write 1 Done Done New false
17310 6720 Acked 7719 Read 2 Done Done New false
17311 6721 Acked 7720 Write 3 Done Done New false
17312 6722 Acked 7721 Write 5 Done Done New false
17313 6723 Acked 7722 Flush 0 Done Done New false
17314 6724 Acked 7723 Read 6 Done Done New false
17315 6725 Acked 7724 Read 6 Done Done New false
17316 6726 Acked 7725 Write 7 Done Done New false
17317 6727 Acked 7726 Flush 0 Done Done New false
17318 6728 Acked 7727 Read 6 Done Done New false
17319 6729 Acked 7728 Read 10 Done Done New false
17320 6730 Acked 7729 Read 2 Done Done New false
17321 6731 Acked 7730 Read 5 Done Done New false
17322 6732 Acked 7731 Write 2 Done Done New false
17323 6733 Acked 7732 Read 1 Done Done New false
17324 6734 Acked 7733 Write 10 Done Done New false
17325 6735 Acked 7734 Read 6 Done Done New false
17326 6736 Acked 7735 Read 4 Done Done New false
17327 6737 Acked 7736 Read 5 Done Done New false
17328 6738 Acked 7737 Read 3 Done Done New false
17329 6739 Acked 7738 Write 5 Done Done New false
17330 6740 Acked 7739 Write 7 Done Done New false
17331 6741 Acked 7740 Flush 0 Done Done New false
17332 6742 Acked 7741 Write 1 Done Done New false
17333 6743 Acked 7742 Write 1 Done Done New false
17334 6744 Acked 7743 Write 4 Done Done New false
17335 6745 Acked 7744 Write 5 Done Done New false
17336 6746 Acked 7745 Read 1 Done Done New false
17337 6747 Acked 7746 Write 6 Done Done New false
17338 6748 Acked 7747 Write 2 Done Done New false
17339 6749 Acked 7748 Flush 0 Done Done New false
17340 6750 Acked 7749 Write 7 Done Done New false
17341 6751 Acked 7750 Read 9 Done Done New false
17342 6752 Acked 7751 Read 3 Done Done New false
17343 6753 Acked 7752 Write 6 Done Done New false
17344 6754 Acked 7753 Read 1 Done Done New false
17345 6755 Acked 7754 Flush 0 Done Done New false
17346 6756 Acked 7755 Flush 0 Done Done New false
17347 6757 Acked 7756 Write 6 Done Done New false
17348 6758 Acked 7757 Read 9 Done Done New false
17349 6759 Acked 7758 Read 9 Done Done New false
17350 6760 Acked 7759 Read 10 Done Done New false
17351 6761 Acked 7760 Write 8 Done Done New false
17352 6762 Acked 7761 Read 8 Done Done New false
17353 6763 Acked 7762 Flush 0 Done Done New false
17354 6764 Acked 7763 Read 8 Done Done New false
17355 6765 Acked 7764 Read 10 Done Done New false
17356 6766 Acked 7765 Write 4 Done Done New false
17357 6767 Acked 7766 Read 8 Done Done New false
17358 6768 Acked 7767 Write 4 Done Done New false
17359 6769 Acked 7768 Flush 0 Done Done New false
17360 6770 Acked 7769 Flush 0 Done Done New false
17361 6771 Acked 7770 Write 7 Done Done New false
17362 6772 Acked 7771 Read 3 Done Done New false
17363 6773 Acked 7772 Read 8 Done Done New false
17364 6774 Acked 7773 Read 4 Done Done New false
17365 6775 Acked 7774 Read 1 Done Done New false
17366 6776 Acked 7775 Write 5 Done Done New false
17367 6777 Acked 7776 Read 8 Done Done New false
17368 6778 Acked 7777 Read 9 Done Done New false
17369 6779 Acked 7778 Write 2 Done Done New false
17370 6780 Acked 7779 Read 5 Done Done New false
17371 6781 Acked 7780 Read 8 Done Done New false
17372 6782 Acked 7781 Read 2 Done Done New false
17373 6783 Acked 7782 Read 6 Done Done New false
17374 6784 Acked 7783 Write 10 Done Done New false
17375 6785 Acked 7784 Write 4 Done Done New false
17376 6786 Acked 7785 Write 8 Done Done New false
17377 6787 Acked 7786 Read 8 Done Done New false
17378 6788 Acked 7787 Read 1 Done Done New false
17379 6789 Acked 7788 Read 10 Done Done New false
17380 6790 Acked 7789 Read 10 Done Done New false
17381 6791 Acked 7790 Flush 0 Done Done New false
17382 6792 Acked 7791 Write 8 Done Done New false
17383 6793 Acked 7792 Read 10 Done Done New false
17384 6794 Acked 7793 Write 6 Done Done New false
17385 6795 Acked 7794 Read 1 Done Done New false
17386 6796 Acked 7795 Read 6 Done Done New false
17387 6797 Acked 7796 Write 2 Done Done New false
17388 6798 Acked 7797 Read 5 Done Done New false
17389 6799 Acked 7798 Write 3 Done Done New false
17390 6800 Acked 7799 Read 8 Done Done New false
17391 6801 Acked 7800 Read 10 Done Done New false
17392 6802 Acked 7801 Read 10 Done Done New false
17393 6803 Acked 7802 Write 3 Done Done New false
17394 6804 Acked 7803 Read 10 Done Done New false
17395 6805 Acked 7804 Read 7 Done Done New false
17396 6806 Acked 7805 Write 5 Done Done New false
17397 6807 Acked 7806 Read 4 Done Done New false
17398 6808 Acked 7807 Read 2 Done Done New false
17399 6809 Acked 7808 Write 5 Done Done New false
17400 6810 Acked 7809 Flush 0 Done Done New false
17401 6811 Acked 7810 Write 6 Done Done New false
17402 6812 Acked 7811 Write 4 Done Done New false
17403 6813 Acked 7812 Read 5 Done Done New false
17404 6814 Acked 7813 Read 1 Done Done New false
17405 6815 Acked 7814 Read 8 Done Done New false
17406 6816 Acked 7815 Flush 0 Done Done New false
17407 6817 Acked 7816 Read 1 Done Done New false
17408 6818 Acked 7817 Read 3 Done Done New false
17409 6819 Acked 7818 Read 6 Done Done New false
17410 6820 Acked 7819 Flush 0 Done Done New false
17411 6821 Acked 7820 Read 3 Done Done New false
17412 6822 Acked 7821 Write 4 Done Done New false
17413 6823 Acked 7822 Read 5 Done Done New false
17414 6824 Acked 7823 Read 10 Done Done New false
17415 6825 Acked 7824 Write 3 Done Done New false
17416 6826 Acked 7825 Read 9 Done Done New false
17417 6827 Acked 7826 Write 1 Done Done New false
17418 6828 Acked 7827 Read 4 Done Done New false
17419 6829 Acked 7828 Flush 0 Done Done New false
17420 6830 Acked 7829 Read 5 Done Done New false
17421 6831 Acked 7830 Flush 0 Done Done New false
17422 6832 Acked 7831 Write 1 Done Done New false
17423 6833 Acked 7832 Read 9 Done Done New false
17424 6834 Acked 7833 Read 5 Done Done New false
17425 6835 Acked 7834 Read 10 Done Done New false
17426 6836 Acked 7835 Write 8 Done Done New false
17427 6837 Acked 7836 Read 5 Done Done New false
17428 6838 Acked 7837 Read 8 Done Done New false
17429 6839 Acked 7838 Read 5 Done Done New false
17430 6840 Acked 7839 Read 3 Done Done New false
17431 6841 Acked 7840 Write 6 Done Done New false
17432 6842 Acked 7841 Write 7 Done Done New false
17433 6843 Acked 7842 Read 7 Done Done New false
17434 6844 Acked 7843 Read 5 Done Done New false
17435 6845 Acked 7844 Read 5 Done Done New false
17436 6846 Acked 7845 Read 8 Done Done New false
17437 6847 Acked 7846 Write 7 Done Done New false
17438 6848 Acked 7847 Read 5 Done Done New false
17439 6849 Acked 7848 Write 8 Done Done New false
17440 6850 Acked 7849 Write 10 Done Done New false
17441 6851 Acked 7850 Write 4 Done Done New false
17442 6852 Acked 7851 Write 1 Done Done New false
17443 6853 Acked 7852 Read 10 Done Done New false
17444 6854 Acked 7853 Flush 0 Done Done New false
17445 6855 Acked 7854 Read 4 Done Done New false
17446 6856 Acked 7855 Write 10 Done Done New false
17447 6857 Acked 7856 Write 1 Done Done New false
17448 6858 Acked 7857 Write 4 Done Done New false
17449 6859 Acked 7858 Flush 0 Done Done New false
17450 6860 Acked 7859 Read 5 Done Done New false
17451 6861 Acked 7860 Write 1 Done Done New false
17452 6862 Acked 7861 Read 6 Done Done New false
17453 6863 Acked 7862 Write 1 Done Done New false
17454 6864 Acked 7863 Read 7 Done Done New false
17455 6865 Acked 7864 Read 5 Done Done New false
17456 6866 Acked 7865 Write 3 Done Done New false
17457 6867 Acked 7866 Read 7 Done Done New false
17458 6868 Acked 7867 Read 10 Done Done New false
17459 6869 Acked 7868 Read 6 Done Done New false
17460 6870 Acked 7869 Read 2 Done Done New false
17461 6871 Acked 7870 Write 3 Done Done New false
17462 6872 Acked 7871 Read 6 Done Done New false
17463 6873 Acked 7872 Write 9 Done Done New false
17464 6874 Acked 7873 Flush 0 Done Done New false
17465 6875 Acked 7874 Write 5 Done Done New false
17466 6876 Acked 7875 Write 4 Done Done New false
17467 6877 Acked 7876 Write 6 Done Done New false
17468 6878 Acked 7877 Read 6 Done Done New false
17469 6879 Acked 7878 Write 6 Done Done New false
17470 6880 Acked 7879 Read 7 Done Done New false
17471 6881 Acked 7880 Write 6 Done Done New false
17472 6882 Acked 7881 Read 2 Done Done New false
17473 6883 Acked 7882 Read 10 Done Done New false
17474 6884 Acked 7883 Read 1 Done Done New false
17475 6885 Acked 7884 Write 10 Done Done New false
17476 6886 Acked 7885 Read 8 Done Done New false
17477 6887 Acked 7886 Read 3 Done Done New false
17478 6888 Acked 7887 Read 5 Done Done New false
17479 6889 Acked 7888 Write 6 Done Done New false
17480 6890 Acked 7889 Flush 0 Done Done New false
17481 6891 Acked 7890 Flush 0 Done Done New false
17482 6892 Acked 7891 Write 8 Done Done New false
17483 6893 Acked 7892 Write 7 Done Done New false
17484 6894 Acked 7893 Flush 0 Done Done New false
17485 6895 Acked 7894 Read 2 Done Done New false
17486 6896 Acked 7895 Read 9 Done Done New false
17487 6897 Acked 7896 Read 7 Done Done New false
17488 6898 Acked 7897 Write 10 Done Done New false
17489 6899 Acked 7898 Write 6 Done Done New false
17490 6900 Acked 7899 Read 10 Done Done New false
17491 6901 Acked 7900 Read 5 Done Done New false
17492 6902 Acked 7901 Write 9 Done Done New false
17493 6903 Acked 7902 Flush 0 Done Done New false
17494 6904 Acked 7903 Read 4 Done Done New false
17495 6905 Acked 7904 Write 6 Done Done New false
17496 6906 Acked 7905 Read 10 Done Done New false
17497 6907 Acked 7906 Read 9 Done Done New false
17498 6908 Acked 7907 Write 6 Done Done New false
17499 6909 Acked 7908 Read 9 Done Done New false
17500 6910 Acked 7909 Read 7 Done Done New false
17501 6911 Acked 7910 Write 1 Done Done New false
17502 6912 Acked 7911 Write 9 Done Done New false
17503 6913 Acked 7912 Read 9 Done Done New false
17504 6914 Acked 7913 Write 6 Done Done New false
17505 6915 Acked 7914 Flush 0 Done Done New false
17506 6916 Acked 7915 Read 5 Done Done New false
17507 6917 Acked 7916 Read 5 Done Done New false
17508 6918 Acked 7917 Read 10 Done Done New false
17509 6919 Acked 7918 Write 10 Done Done New false
17510 6920 Acked 7919 Flush 0 Done Done New false
17511 6921 Acked 7920 Read 10 Done Done New false
17512 6922 Acked 7921 Read 10 Done Done New false
17513 6923 Acked 7922 Read 7 Done Done New false
17514 6924 Acked 7923 Write 1 Done Done New false
17515 6925 Acked 7924 Write 2 Done Done New false
17516 6926 Acked 7925 Read 5 Done Done New false
17517 6927 Acked 7926 Write 7 Done Done New false
17518 6928 Acked 7927 Read 6 Done Done New false
17519 6929 Acked 7928 Write 6 Done Done New false
17520 6930 Acked 7929 Write 2 Done Done New false
17521 6931 Acked 7930 Read 4 Done Done New false
17522 6932 Acked 7931 Read 4 Done Done New false
17523 6933 Acked 7932 Flush 0 Done Done New false
17524 6934 Acked 7933 Read 9 Done Done New false
17525 6935 Acked 7934 Read 4 Done Done New false
17526 6936 Acked 7935 Write 5 Done Done New false
17527 6937 Acked 7936 Read 5 Done Done New false
17528 6938 Acked 7937 Read 8 Done Done New false
17529 6939 Acked 7938 Flush 0 Done Done New false
17530 6940 Acked 7939 Read 1 Done Done New false
17531 6941 Acked 7940 Read 3 Done Done New false
17532 6942 Acked 7941 Read 3 Done Done New false
17533 6943 Acked 7942 Flush 0 Done Done New false
17534 6944 Acked 7943 Write 1 Done Done New false
17535 6945 Acked 7944 Write 8 Done Done New false
17536 6946 Acked 7945 Flush 0 Done Done New false
17537 6947 Acked 7946 Write 4 Done Done New false
17538 6948 Acked 7947 Write 5 Done Done New false
17539 6949 Acked 7948 Write 10 Done Done New false
17540 6950 Acked 7949 Read 8 Done Done New false
17541 6951 Acked 7950 Write 1 Done Done New false
17542 6952 Acked 7951 Read 7 Done Done New false
17543 6953 Acked 7952 Read 9 Done Done New false
17544 6954 Acked 7953 Write 8 Done Done New false
17545 6955 Acked 7954 Read 7 Done Done New false
17546 6956 Acked 7955 Read 9 Done Done New false
17547 6957 Acked 7956 Read 3 Done Done New false
17548 6958 Acked 7957 Flush 0 Done Done New false
17549 6959 Acked 7958 Read 4 Done Done New false
17550 6960 Acked 7959 Write 10 Sent Sent New false
17551 6961 Acked 7960 Write 4 Sent Sent New false
17552 STATES DS:0 DS:1 DS:2 TOTAL
17553 New 0 0 302 302
17554 Sent 2 2 0 4
17555 Done 300 300 0 600
17556 Skipped 0 0 0 0
17557 Error 0 0 0 0
17558 Last Flush: 7957 7957 7658
17559 Downstairs last five completed: 7658 7657 7656 7655 7654 7653
17560 Upstairs last five completed: 6961 6960 6959 6958 6957 6956
17561 CLIENT: Up:0 ds:302 act:2
175622023-10-13T18:12:56.549ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
17563 [8830] got start action
17564 [8830] I am going to restart
17565 Starting downstairs at port 8830
17566 Make output file at "/tmp/dsc/downstairs-8830.txt"
17567 [8830][2] reports Starting
17568 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1199)
17569 [8830][2] reports Running
17570 ----------------------------------------------------------------
17571 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
17572 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
17573 6660 Acked 7659 Flush 0 Done Done New false
17574 6661 Acked 7660 Write 6 Done Done New false
17575 6662 Acked 7661 Flush 0 Done Done New false
17576 6663 Acked 7662 Write 8 Done Done New false
17577 6664 Acked 7663 Write 8 Done Done New false
17578 6665 Acked 7664 Write 4 Done Done New false
17579 6666 Acked 7665 Read 8 Done Done New false
17580 6667 Acked 7666 Write 2 Done Done New false
17581 6668 Acked 7667 Write 10 Done Done New false
17582 6669 Acked 7668 Write 9 Done Done New false
17583 6670 Acked 7669 Read 6 Done Done New false
17584 6671 Acked 7670 Write 8 Done Done New false
17585 6672 Acked 7671 Write 10 Done Done New false
17586 6673 Acked 7672 Flush 0 Done Done New false
17587 6674 Acked 7673 Write 1 Done Done New false
17588 6675 Acked 7674 Read 7 Done Done New false
17589 6676 Acked 7675 Flush 0 Done Done New false
17590 6677 Acked 7676 Write 10 Done Done New false
17591 6678 Acked 7677 Read 5 Done Done New false
17592 6679 Acked 7678 Read 3 Done Done New false
17593 6680 Acked 7679 Write 7 Done Done New false
17594 6681 Acked 7680 Flush 0 Done Done New false
17595 6682 Acked 7681 Write 8 Done Done New false
17596 6683 Acked 7682 Read 4 Done Done New false
17597 6684 Acked 7683 Read 5 Done Done New false
17598 6685 Acked 7684 Read 4 Done Done New false
17599 6686 Acked 7685 Read 8 Done Done New false
17600 6687 Acked 7686 Write 7 Done Done New false
17601 6688 Acked 7687 Read 10 Done Done New false
17602 6689 Acked 7688 Write 7 Done Done New false
17603 6690 Acked 7689 Read 10 Done Done New false
17604 6691 Acked 7690 Read 9 Done Done New false
17605 6692 Acked 7691 Read 6 Done Done New false
17606 6693 Acked 7692 Write 5 Done Done New false
17607 6694 Acked 7693 Write 4 Done Done New false
17608 6695 Acked 7694 Read 4 Done Done New false
17609 6696 Acked 7695 Read 8 Done Done New false
17610 6697 Acked 7696 Read 9 Done Done New false
17611 6698 Acked 7697 Read 8 Done Done New false
17612 6699 Acked 7698 Read 6 Done Done New false
17613 6700 Acked 7699 Read 3 Done Done New false
17614 6701 Acked 7700 Read 10 Done Done New false
17615 6702 Acked 7701 Read 6 Done Done New false
17616 6703 Acked 7702 Write 10 Done Done New false
17617 6704 Acked 7703 Flush 0 Done Done New false
17618 6705 Acked 7704 Read 5 Done Done New false
17619 6706 Acked 7705 Write 3 Done Done New false
17620 6707 Acked 7706 Read 6 Done Done New false
17621 6708 Acked 7707 Read 4 Done Done New false
17622 6709 Acked 7708 Write 5 Done Done New false
17623 6710 Acked 7709 Read 5 Done Done New false
17624 6711 Acked 7710 Read 10 Done Done New false
17625 6712 Acked 7711 Write 8 Done Done New false
17626 6713 Acked 7712 Write 7 Done Done New false
17627 6714 Acked 7713 Write 6 Done Done New false
17628 6715 Acked 7714 Write 9 Done Done New false
17629 6716 Acked 7715 Read 6 Done Done New false
17630 6717 Acked 7716 Read 3 Done Done New false
17631 6718 Acked 7717 Read 3 Done Done New false
17632 6719 Acked 7718 Write 1 Done Done New false
17633 6720 Acked 7719 Read 2 Done Done New false
17634 6721 Acked 7720 Write 3 Done Done New false
17635 6722 Acked 7721 Write 5 Done Done New false
17636 6723 Acked 7722 Flush 0 Done Done New false
17637 6724 Acked 7723 Read 6 Done Done New false
17638 6725 Acked 7724 Read 6 Done Done New false
17639 6726 Acked 7725 Write 7 Done Done New false
17640 6727 Acked 7726 Flush 0 Done Done New false
17641 6728 Acked 7727 Read 6 Done Done New false
17642 6729 Acked 7728 Read 10 Done Done New false
17643 6730 Acked 7729 Read 2 Done Done New false
17644 6731 Acked 7730 Read 5 Done Done New false
17645 6732 Acked 7731 Write 2 Done Done New false
17646 6733 Acked 7732 Read 1 Done Done New false
17647 6734 Acked 7733 Write 10 Done Done New false
17648 6735 Acked 7734 Read 6 Done Done New false
17649 6736 Acked 7735 Read 4 Done Done New false
17650 6737 Acked 7736 Read 5 Done Done New false
17651 6738 Acked 7737 Read 3 Done Done New false
17652 6739 Acked 7738 Write 5 Done Done New false
17653 6740 Acked 7739 Write 7 Done Done New false
17654 6741 Acked 7740 Flush 0 Done Done New false
17655 6742 Acked 7741 Write 1 Done Done New false
17656 6743 Acked 7742 Write 1 Done Done New false
17657 6744 Acked 7743 Write 4 Done Done New false
17658 6745 Acked 7744 Write 5 Done Done New false
17659 6746 Acked 7745 Read 1 Done Done New false
17660 6747 Acked 7746 Write 6 Done Done New false
17661 6748 Acked 7747 Write 2 Done Done New false
17662 6749 Acked 7748 Flush 0 Done Done New false
17663 6750 Acked 7749 Write 7 Done Done New false
17664 6751 Acked 7750 Read 9 Done Done New false
17665 6752 Acked 7751 Read 3 Done Done New false
17666 6753 Acked 7752 Write 6 Done Done New false
17667 6754 Acked 7753 Read 1 Done Done New false
17668 6755 Acked 7754 Flush 0 Done Done New false
17669 6756 Acked 7755 Flush 0 Done Done New false
17670 6757 Acked 7756 Write 6 Done Done New false
17671 6758 Acked 7757 Read 9 Done Done New false
17672 6759 Acked 7758 Read 9 Done Done New false
17673 6760 Acked 7759 Read 10 Done Done New false
17674 6761 Acked 7760 Write 8 Done Done New false
17675 6762 Acked 7761 Read 8 Done Done New false
17676 6763 Acked 7762 Flush 0 Done Done New false
17677 6764 Acked 7763 Read 8 Done Done New false
17678 6765 Acked 7764 Read 10 Done Done New false
17679 6766 Acked 7765 Write 4 Done Done New false
17680 6767 Acked 7766 Read 8 Done Done New false
17681 6768 Acked 7767 Write 4 Done Done New false
17682 6769 Acked 7768 Flush 0 Done Done New false
17683 6770 Acked 7769 Flush 0 Done Done New false
17684 6771 Acked 7770 Write 7 Done Done New false
17685 6772 Acked 7771 Read 3 Done Done New false
17686 6773 Acked 7772 Read 8 Done Done New false
17687 6774 Acked 7773 Read 4 Done Done New false
17688 6775 Acked 7774 Read 1 Done Done New false
17689 6776 Acked 7775 Write 5 Done Done New false
17690 6777 Acked 7776 Read 8 Done Done New false
17691 6778 Acked 7777 Read 9 Done Done New false
17692 6779 Acked 7778 Write 2 Done Done New false
17693 6780 Acked 7779 Read 5 Done Done New false
17694 6781 Acked 7780 Read 8 Done Done New false
17695 6782 Acked 7781 Read 2 Done Done New false
17696 6783 Acked 7782 Read 6 Done Done New false
17697 6784 Acked 7783 Write 10 Done Done New false
17698 6785 Acked 7784 Write 4 Done Done New false
17699 6786 Acked 7785 Write 8 Done Done New false
17700 6787 Acked 7786 Read 8 Done Done New false
17701 6788 Acked 7787 Read 1 Done Done New false
17702 6789 Acked 7788 Read 10 Done Done New false
17703 6790 Acked 7789 Read 10 Done Done New false
17704 6791 Acked 7790 Flush 0 Done Done New false
17705 6792 Acked 7791 Write 8 Done Done New false
17706 6793 Acked 7792 Read 10 Done Done New false
17707 6794 Acked 7793 Write 6 Done Done New false
17708 6795 Acked 7794 Read 1 Done Done New false
17709 6796 Acked 7795 Read 6 Done Done New false
17710 6797 Acked 7796 Write 2 Done Done New false
17711 6798 Acked 7797 Read 5 Done Done New false
17712 6799 Acked 7798 Write 3 Done Done New false
17713 6800 Acked 7799 Read 8 Done Done New false
17714 6801 Acked 7800 Read 10 Done Done New false
17715 6802 Acked 7801 Read 10 Done Done New false
17716 6803 Acked 7802 Write 3 Done Done New false
17717 6804 Acked 7803 Read 10 Done Done New false
17718 6805 Acked 7804 Read 7 Done Done New false
17719 6806 Acked 7805 Write 5 Done Done New false
17720 6807 Acked 7806 Read 4 Done Done New false
17721 6808 Acked 7807 Read 2 Done Done New false
17722 6809 Acked 7808 Write 5 Done Done New false
17723 6810 Acked 7809 Flush 0 Done Done New false
17724 6811 Acked 7810 Write 6 Done Done New false
17725 6812 Acked 7811 Write 4 Done Done New false
17726 6813 Acked 7812 Read 5 Done Done New false
17727 6814 Acked 7813 Read 1 Done Done New false
17728 6815 Acked 7814 Read 8 Done Done New false
17729 6816 Acked 7815 Flush 0 Done Done New false
17730 6817 Acked 7816 Read 1 Done Done New false
17731 6818 Acked 7817 Read 3 Done Done New false
17732 6819 Acked 7818 Read 6 Done Done New false
17733 6820 Acked 7819 Flush 0 Done Done New false
17734 6821 Acked 7820 Read 3 Done Done New false
17735 6822 Acked 7821 Write 4 Done Done New false
17736 6823 Acked 7822 Read 5 Done Done New false
17737 6824 Acked 7823 Read 10 Done Done New false
17738 6825 Acked 7824 Write 3 Done Done New false
17739 6826 Acked 7825 Read 9 Done Done New false
17740 6827 Acked 7826 Write 1 Done Done New false
17741 6828 Acked 7827 Read 4 Done Done New false
17742 6829 Acked 7828 Flush 0 Done Done New false
17743 6830 Acked 7829 Read 5 Done Done New false
17744 6831 Acked 7830 Flush 0 Done Done New false
17745 6832 Acked 7831 Write 1 Done Done New false
17746 6833 Acked 7832 Read 9 Done Done New false
17747 6834 Acked 7833 Read 5 Done Done New false
17748 6835 Acked 7834 Read 10 Done Done New false
17749 6836 Acked 7835 Write 8 Done Done New false
17750 6837 Acked 7836 Read 5 Done Done New false
17751 6838 Acked 7837 Read 8 Done Done New false
17752 6839 Acked 7838 Read 5 Done Done New false
17753 6840 Acked 7839 Read 3 Done Done New false
17754 6841 Acked 7840 Write 6 Done Done New false
17755 6842 Acked 7841 Write 7 Done Done New false
17756 6843 Acked 7842 Read 7 Done Done New false
17757 6844 Acked 7843 Read 5 Done Done New false
17758 6845 Acked 7844 Read 5 Done Done New false
17759 6846 Acked 7845 Read 8 Done Done New false
17760 6847 Acked 7846 Write 7 Done Done New false
17761 6848 Acked 7847 Read 5 Done Done New false
17762 6849 Acked 7848 Write 8 Done Done New false
17763 6850 Acked 7849 Write 10 Done Done New false
17764 6851 Acked 7850 Write 4 Done Done New false
17765 6852 Acked 7851 Write 1 Done Done New false
17766 6853 Acked 7852 Read 10 Done Done New false
17767 6854 Acked 7853 Flush 0 Done Done New false
17768 6855 Acked 7854 Read 4 Done Done New false
17769 6856 Acked 7855 Write 10 Done Done New false
17770 6857 Acked 7856 Write 1 Done Done New false
17771 6858 Acked 7857 Write 4 Done Done New false
17772 6859 Acked 7858 Flush 0 Done Done New false
17773 6860 Acked 7859 Read 5 Done Done New false
17774 6861 Acked 7860 Write 1 Done Done New false
17775 6862 Acked 7861 Read 6 Done Done New false
17776 6863 Acked 7862 Write 1 Done Done New false
17777 6864 Acked 7863 Read 7 Done Done New false
17778 6865 Acked 7864 Read 5 Done Done New false
17779 6866 Acked 7865 Write 3 Done Done New false
17780 6867 Acked 7866 Read 7 Done Done New false
17781 6868 Acked 7867 Read 10 Done Done New false
17782 6869 Acked 7868 Read 6 Done Done New false
17783 6870 Acked 7869 Read 2 Done Done New false
17784 6871 Acked 7870 Write 3 Done Done New false
17785 6872 Acked 7871 Read 6 Done Done New false
17786 6873 Acked 7872 Write 9 Done Done New false
17787 6874 Acked 7873 Flush 0 Done Done New false
17788 6875 Acked 7874 Write 5 Done Done New false
17789 6876 Acked 7875 Write 4 Done Done New false
17790 6877 Acked 7876 Write 6 Done Done New false
17791 6878 Acked 7877 Read 6 Done Done New false
17792 6879 Acked 7878 Write 6 Done Done New false
17793 6880 Acked 7879 Read 7 Done Done New false
17794 6881 Acked 7880 Write 6 Done Done New false
17795 6882 Acked 7881 Read 2 Done Done New false
17796 6883 Acked 7882 Read 10 Done Done New false
17797 6884 Acked 7883 Read 1 Done Done New false
17798 6885 Acked 7884 Write 10 Done Done New false
17799 6886 Acked 7885 Read 8 Done Done New false
17800 6887 Acked 7886 Read 3 Done Done New false
17801 6888 Acked 7887 Read 5 Done Done New false
17802 6889 Acked 7888 Write 6 Done Done New false
17803 6890 Acked 7889 Flush 0 Done Done New false
17804 6891 Acked 7890 Flush 0 Done Done New false
17805 6892 Acked 7891 Write 8 Done Done New false
17806 6893 Acked 7892 Write 7 Done Done New false
17807 6894 Acked 7893 Flush 0 Done Done New false
17808 6895 Acked 7894 Read 2 Done Done New false
17809 6896 Acked 7895 Read 9 Done Done New false
17810 6897 Acked 7896 Read 7 Done Done New false
17811 6898 Acked 7897 Write 10 Done Done New false
17812 6899 Acked 7898 Write 6 Done Done New false
17813 6900 Acked 7899 Read 10 Done Done New false
17814 6901 Acked 7900 Read 5 Done Done New false
17815 6902 Acked 7901 Write 9 Done Done New false
17816 6903 Acked 7902 Flush 0 Done Done New false
17817 6904 Acked 7903 Read 4 Done Done New false
17818 6905 Acked 7904 Write 6 Done Done New false
17819 6906 Acked 7905 Read 10 Done Done New false
17820 6907 Acked 7906 Read 9 Done Done New false
17821 6908 Acked 7907 Write 6 Done Done New false
17822 6909 Acked 7908 Read 9 Done Done New false
17823 6910 Acked 7909 Read 7 Done Done New false
17824 6911 Acked 7910 Write 1 Done Done New false
17825 6912 Acked 7911 Write 9 Done Done New false
17826 6913 Acked 7912 Read 9 Done Done New false
17827 6914 Acked 7913 Write 6 Done Done New false
17828 6915 Acked 7914 Flush 0 Done Done New false
17829 6916 Acked 7915 Read 5 Done Done New false
17830 6917 Acked 7916 Read 5 Done Done New false
17831 6918 Acked 7917 Read 10 Done Done New false
17832 6919 Acked 7918 Write 10 Done Done New false
17833 6920 Acked 7919 Flush 0 Done Done New false
17834 6921 Acked 7920 Read 10 Done Done New false
17835 6922 Acked 7921 Read 10 Done Done New false
17836 6923 Acked 7922 Read 7 Done Done New false
17837 6924 Acked 7923 Write 1 Done Done New false
17838 6925 Acked 7924 Write 2 Done Done New false
17839 6926 Acked 7925 Read 5 Done Done New false
17840 6927 Acked 7926 Write 7 Done Done New false
17841 6928 Acked 7927 Read 6 Done Done New false
17842 6929 Acked 7928 Write 6 Done Done New false
17843 6930 Acked 7929 Write 2 Done Done New false
17844 6931 Acked 7930 Read 4 Done Done New false
17845 6932 Acked 7931 Read 4 Done Done New false
17846 6933 Acked 7932 Flush 0 Done Done New false
17847 6934 Acked 7933 Read 9 Done Done New false
17848 6935 Acked 7934 Read 4 Done Done New false
17849 6936 Acked 7935 Write 5 Done Done New false
17850 6937 Acked 7936 Read 5 Done Done New false
17851 6938 Acked 7937 Read 8 Done Done New false
17852 6939 Acked 7938 Flush 0 Done Done New false
17853 6940 Acked 7939 Read 1 Done Done New false
17854 6941 Acked 7940 Read 3 Done Done New false
17855 6942 Acked 7941 Read 3 Done Done New false
17856 6943 Acked 7942 Flush 0 Done Done New false
17857 6944 Acked 7943 Write 1 Done Done New false
17858 6945 Acked 7944 Write 8 Done Done New false
17859 6946 Acked 7945 Flush 0 Done Done New false
17860 6947 Acked 7946 Write 4 Done Done New false
17861 6948 Acked 7947 Write 5 Done Done New false
17862 6949 Acked 7948 Write 10 Done Done New false
17863 6950 Acked 7949 Read 8 Done Done New false
17864 6951 Acked 7950 Write 1 Done Done New false
17865 6952 Acked 7951 Read 7 Done Done New false
17866 6953 Acked 7952 Read 9 Done Done New false
17867 6954 Acked 7953 Write 8 Done Done New false
17868 6955 Acked 7954 Read 7 Done Done New false
17869 6956 Acked 7955 Read 9 Done Done New false
17870 6957 Acked 7956 Read 3 Done Done New false
17871 6958 Acked 7957 Flush 0 Done Done New false
17872 6959 Acked 7958 Read 4 Done Done New false
17873 6960 Acked 7959 Write 10 Done Done New false
17874 6961 Acked 7960 Write 4 Done Done New false
17875 6962 Acked 7961 Flush 0 Done Done New false
17876 STATES DS:0 DS:1 DS:2 TOTAL
17877 New 0 0 303 303
17878 Sent 0 0 0 0
17879 Done 303 303 0 606
17880 Skipped 0 0 0 0
17881 Error 0 0 0 0
17882 Last Flush: 7961 7961 7658
17883 Downstairs last five completed: 7658 7657 7656 7655 7654 7653
17884 Upstairs last five completed: 6962 6961 6960 6959 6958 6957
17885 CLIENT: Up:0 ds:303 act:2
178862023-10-13T18:12:57.551ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 2
178872023-10-13T18:12:57.551ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
178882023-10-13T18:12:57.551ZINFOcrucible: [2] client re-new 303 jobs since flush 7658 = downstairs
178892023-10-13T18:12:57.552ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
178902023-10-13T18:12:57.553ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
178912023-10-13T18:12:57.553ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
178922023-10-13T18:12:57.553ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
178932023-10-13T18:12:57.553ZINFOcrucible: [2] send last flush ID to this DS: 7658
178942023-10-13T18:12:57.553ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
178952023-10-13T18:12:57.553ZINFOcrucible: [2] Replied this last flush ID: 7658
178962023-10-13T18:12:57.553ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Offline ds_transition to Replay
178972023-10-13T18:12:57.553ZINFOcrucible: [2] Transition from Offline to Replay
178982023-10-13T18:12:57.553ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
178992023-10-13T18:12:57.553ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Replay ds_transition to Active
179002023-10-13T18:12:57.553ZINFOcrucible: [2] Transition from Replay to Active
179012023-10-13T18:12:57.553ZINFOcrucible: [2] Starts cmd_loop
179022023-10-13T18:12:57.553ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
179032023-10-13T18:12:57.553ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
179042023-10-13T18:12:57.553ZINFOcrucible: Set check for repair
179052023-10-13T18:12:58.554ZINFOcrucible: Checking if live repair is needed
179062023-10-13T18:12:58.554ZINFOcrucible: No Live Repair required at this time
179072023-10-13T18:12:58.554ZWARNcrucible: [2] flow control end
17908 ----------------------------------------------------------------
17909 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17910 Downstairs last five completed: 7961 7960 7959 7958 7957 7956
17911 Upstairs last five completed: 6962 6961 6960 6959 6958 6957
17912 CLIENT: Up:0 ds:0 act:3
17913 Replay: All jobs finished, all DS active.
17914 Main task has work to do, go find it
17915 got dsc Stop(0)
17916 stop 0
17917 Oct 13 18:13:01.554 INFO request completed, latency_us: 520, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 65d37cf4-4dbf-4a88-b59e-621a7f527261, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
17918 [8810] Got stop action so:false kr:false
17919 Oct 13 18:13:01.555 INFO request completed, latency_us: 297, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4d05cb2f-00c0-42e9-8468-4ab8cd54af07, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
179202023-10-13T18:13:01.556ZWARNcrucible: [0] downstairs disconnected
179212023-10-13T18:13:01.556ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
179222023-10-13T18:13:01.556ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8810 closed looper = 0
179232023-10-13T18:13:01.556ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:false
179242023-10-13T18:13:01.556ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
179252023-10-13T18:13:01.556ZINFOcrucible: [0] 127.0.0.1:8810 task reports offline
179262023-10-13T18:13:01.556ZWARNcrucible: [0] pm_task rx.recv() is None
17927 [8810] Exited with: ExitStatus(unix_wait_status(9))
17928 [8810][0] reports Exit
179292023-10-13T18:13:02.557ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
17930 Oct 13 18:13:05.558 INFO request completed, latency_us: 269, response_code: 200, uri: /state/cid/0, method: GET, req_id: a7025fa7-3b7a-46c7-bb9c-4a2208875a59, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
17931 001/300 Write block 1940 len 36864 data: 8 7 5 5 5 5 5 5 5
17932 002/300 Read block 2270 len 16384
17933 003/300 Read block 3291 len 12288
17934 004/300 Read block 4346 len 40960
17935 005/300 Write block 551 len 8192 data: 5 4
17936 006/300 Write block 1531 len 36864 data: 4 4 3 4 3 2 2 2 2
17937 007/300 Read block 2293 len 12288
17938 008/300 Read block 2576 len 28672
17939 009/300 Write block 3695 len 12288 data: 7 7 6
17940 010/300 Read block 4365 len 40960
17941 011/300 Read block 973 len 40960
17942 012/300 Read block 1386 len 4096
17943 013/300 Read block 4617 len 28672
17944 014/300 Write block 2698 len 40960 data: 6 5 5 5 4 4 3 3 3 3
17945 015/300 Write block 4252 len 36864 data: 4 4 5 5 5 3 3 2 2
17946 016/300 Read block 2706 len 4096
17947 017/300 Write block 2035 len 36864 data: 4 4 3 3 3 2 2 2 2
17948 018/300 Read block 3607 len 12288
17949 019/300 Read block 4132 len 20480
17950 020/300 Read block 2552 len 36864
17951 021/300 Read block 1809 len 32768
17952 022/300 Write block 2316 len 28672 data: 7 6 5 3 2 3 4
17953 023/300 Write block 4616 len 16384 data: 4 4 3 5
17954 024/300 Read block 3913 len 40960
17955 025/300 Read block 2891 len 32768
17956 026/300 Read block 4536 len 40960
17957 027/300 Flush
17958 028/300 Read block 1222 len 36864
17959 029/300 Read block 3215 len 28672
17960 030/300 Read block 1325 len 36864
17961 031/300 Read block 3493 len 8192
17962 032/300 Read block 3586 len 32768
17963 033/300 Write block 2542 len 4096 data: 3
17964 034/300 Read block 3234 len 28672
17965 035/300 Write block 3992 len 12288 data: 6 6 6
17966 036/300 Read block 2764 len 8192
17967 037/300 Read block 688 len 36864
17968 038/300 Read block 3392 len 16384
17969 039/300 Write block 846 len 4096 data: 6
17970 040/300 Flush
17971 041/300 Flush
17972 042/300 Read block 2630 len 12288
17973 043/300 Read block 2286 len 8192
17974 044/300 Flush
17975 045/300 Write block 4584 len 4096 data: 3
17976 046/300 Write block 4233 len 24576 data: 2 2 4 5 5 5
17977 047/300 Read block 3430 len 24576
17978 048/300 Read block 178 len 4096
17979 049/300 Read block 853 len 36864
17980 050/300 Write block 4031 len 4096 data: 3
17981 051/300 Read block 1093 len 28672
17982 052/300 Flush
17983 053/300 Write block 3430 len 40960 data: 7 6 8 10 9 6 6 4 3 3
17984 054/300 Read block 4043 len 28672
17985 055/300 Read block 3930 len 24576
17986 056/300 Write block 1798 len 28672 data: 5 4 5 5 5 5 5
17987 057/300 Read block 1991 len 40960
17988 058/300 Write block 836 len 28672 data: 5 6 6 6 7 6 6
17989 059/300 Write block 1261 len 8192 data: 4 4
17990 060/300 Write block 706 len 4096 data: 6
17991 061/300 Write block 783 len 8192 data: 2 2
17992 062/300 Flush
17993 063/300 Read block 3679 len 28672
17994 064/300 Write block 432 len 20480 data: 3 3 3 3 3
17995 065/300 Write block 2565 len 16384 data: 7 6 6 6
17996 066/300 Read block 3475 len 24576
17997 067/300 Write block 2473 len 20480 data: 10 9 8 7 7
17998 068/300 Write block 598 len 20480 data: 8 8 9 8 8
17999 069/300 Read block 4986 len 40960
18000 070/300 Flush
18001 071/300 Write block 4387 len 20480 data: 8 7 6 7 6
18002 072/300 Write block 2618 len 32768 data: 5 6 4 3 4 4 4 5
18003 073/300 Read block 4336 len 4096
18004 074/300 Read block 2265 len 12288
18005 075/300 Write block 4670 len 28672 data: 7 4 4 6 5 5 7
18006 076/300 Read block 3952 len 8192
18007 077/300 Flush
18008 078/300 Write block 231 len 24576 data: 3 3 2 2 3 5
18009 079/300 Write block 4968 len 36864 data: 3 3 6 6 6 6 5 6 6
18010 080/300 Write block 4859 len 36864 data: 10 11 8 8 6 6 7 7 4
18011 081/300 Read block 1645 len 36864
18012 082/300 Read block 3060 len 20480
18013 083/300 Flush
18014 084/300 Read block 4960 len 8192
18015 085/300 Read block 1697 len 12288
18016 086/300 Read block 3782 len 32768
18017 087/300 Write block 4901 len 12288 data: 6 5 5
18018 088/300 Read block 4673 len 40960
18019 089/300 Flush
18020 090/300 Read block 2639 len 20480
18021 091/300 Write block 4895 len 28672 data: 4 3 3 3 3 3 7
18022 092/300 Write block 3498 len 28672 data: 7 7 7 7 7 6 5
18023 093/300 Read block 2720 len 24576
18024 094/300 Write block 3194 len 16384 data: 7 7 6 5
18025 095/300 Write block 4808 len 20480 data: 8 7 8 9 8
18026 096/300 Write block 1533 len 28672 data: 4 5 4 3 3 3 3
18027 097/300 Read block 954 len 40960
18028 098/300 Read block 829 len 40960
18029 099/300 Write block 437 len 28672 data: 3 4 4 3 4 4 6
18030 100/300 Read block 4574 len 12288
18031 101/300 Read block 2208 len 28672
18032 102/300 Write block 4306 len 16384 data: 6 8 8 5
18033 103/300 Flush
18034 104/300 Flush
18035 105/300 Write block 645 len 8192 data: 7 6
18036 106/300 Write block 4633 len 24576 data: 4 4 4 4 4 4
18037 107/300 Read block 2209 len 36864
18038 108/300 Read block 3731 len 40960
18039 109/300 Read block 3164 len 12288
18040 110/300 Read block 3061 len 32768
18041 111/300 Flush
18042 112/300 Read block 4726 len 20480
18043 113/300 Write block 2124 len 8192 data: 5 6
18044 114/300 Read block 422 len 36864
18045 115/300 Write block 3929 len 24576 data: 6 6 5 3 5 6
18046 116/300 Read block 2006 len 32768
18047 117/300 Read block 964 len 36864
18048 118/300 Read block 3997 len 12288
18049 119/300 Read block 4797 len 24576
18050 120/300 Read block 3298 len 16384
18051 121/300 Read block 65 len 16384
18052 122/300 Flush
18053 123/300 Write block 1716 len 20480 data: 6 5 5 7 8
18054 124/300 Read block 1380 len 16384
18055 125/300 Read block 2614 len 36864
18056 126/300 Read block 3313 len 16384
18057 127/300 Flush
18058 128/300 Read block 2454 len 8192
18059 129/300 Write block 2130 len 40960 data: 4 4 5 5 5 4 4 5 6 5
18060 130/300 Flush
18061 131/300 Write block 2952 len 32768 data: 6 6 6 5 5 4 5 4
18062 132/300 Write block 2524 len 28672 data: 5 7 7 7 7 7 6
18063 133/300 Write block 787 len 4096 data: 3
18064 134/300 Read block 2266 len 24576
18065 135/300 Write block 1876 len 16384 data: 9 7 8 7
18066 136/300 Write block 4754 len 40960 data: 7 9 8 8 6 5 8 9 8 6
18067 137/300 Write block 3789 len 32768 data: 2 2 2 3 4 4 4 4
18068 138/300 Read block 3908 len 32768
18069 139/300 Write block 307 len 32768 data: 5 5 5 4 6 5 6 8
18070 140/300 Write block 1135 len 36864 data: 3 2 3 3 3 3 3 3 2
18071 141/300 Write block 3682 len 8192 data: 6 5
18072 142/300 Read block 196 len 4096
18073 143/300 Read block 1788 len 4096
18074 144/300 Write block 1238 len 24576 data: 6 6 4 2 3 3
18075 145/300 Read block 1867 len 8192
18076 146/300 Write block 1019 len 32768 data: 4 5 5 4 4 3 3 4
18077 147/300 Write block 4607 len 8192 data: 5 7
18078 148/300 Write block 4373 len 32768 data: 7 8 6 6 5 5 5 4
18079 149/300 Read block 3258 len 8192
18080 150/300 Flush
18081 151/300 Read block 4177 len 28672
18082 152/300 Read block 3349 len 40960
18083 153/300 Read block 617 len 36864
18084 154/300 Read block 2829 len 16384
18085 155/300 Read block 1108 len 36864
18086 156/300 Read block 2785 len 28672
18087 157/300 Read block 1718 len 32768
18088 158/300 Read block 3520 len 36864
18089 159/300 Read block 2675 len 32768
18090 160/300 Read block 2959 len 24576
18091 161/300 Write block 1311 len 20480 data: 7 6 5 5 5
18092 162/300 Read block 1004 len 12288
18093 163/300 Read block 3316 len 20480
18094 164/300 Write block 3994 len 40960 data: 7 5 4 6 4 3 3 3 3 3
18095 165/300 Write block 3298 len 24576 data: 4 4 4 4 5 5
18096 166/300 Read block 865 len 32768
18097 167/300 Write block 143 len 4096 data: 6
18098 168/300 Write block 3845 len 12288 data: 3 3 4
18099 169/300 Write block 2733 len 28672 data: 9 8 8 8 6 4 4
18100 170/300 Write block 118 len 16384 data: 4 4 4 4
18101 171/300 Write block 1246 len 20480 data: 3 4 3 3 2
18102 172/300 Read block 2817 len 16384
18103 173/300 Read block 4317 len 12288
18104 174/300 Write block 903 len 40960 data: 3 5 6 6 7 7 6 5 5 6
18105 175/300 Read block 972 len 4096
18106 176/300 Write block 1156 len 36864 data: 4 6 6 7 10 9 8 8 7
18107 177/300 Read block 1794 len 28672
18108 178/300 Write block 2687 len 16384 data: 3 4 4 4
18109 179/300 Write block 2693 len 40960 data: 3 3 5 7 5 7 6 6 6 5
18110 180/300 Write block 3176 len 12288 data: 5 5 7
18111 181/300 Read block 618 len 32768
18112 182/300 Read block 634 len 24576
18113 183/300 Write block 1392 len 20480 data: 6 6 6 6 5
18114 184/300 Write block 3316 len 36864 data: 4 4 4 4 4 4 5 4 5
18115 185/300 Read block 4179 len 16384
18116 186/300 Write block 1306 len 24576 data: 5 6 6 8 8 8
18117 187/300 Write block 4701 len 8192 data: 8 10
18118 188/300 Read block 1533 len 28672
18119 189/300 Read block 4760 len 40960
18120 190/300 Read block 585 len 28672
18121 191/300 Read block 4021 len 4096
18122 192/300 Write block 3448 len 24576 data: 4 3 3 3 3 4
18123 193/300 Read block 2938 len 24576
18124 194/300 Write block 907 len 24576 data: 8 8 7 6 6 7
18125 195/300 Read block 222 len 20480
18126 196/300 Write block 1549 len 32768 data: 3 3 4 4 5 3 2 4
18127 197/300 Read block 1436 len 16384
18128 198/300 Flush
18129 199/300 Flush
18130 200/300 Flush
18131 201/300 Read block 3210 len 4096
18132 202/300 Write block 4172 len 16384 data: 7 6 5 5
18133 203/300 Write block 2930 len 8192 data: 4 6
18134 204/300 Flush
18135 205/300 Read block 3713 len 24576
18136 206/300 Write block 4819 len 16384 data: 5 5 4 6
18137 207/300 Read block 495 len 8192
18138 208/300 Read block 842 len 4096
18139 209/300 Read block 152 len 24576
18140 210/300 Write block 4927 len 4096 data: 3
18141 211/300 Read block 4964 len 32768
18142 212/300 Read block 3289 len 32768
18143 213/300 Read block 2828 len 16384
18144 214/300 Read block 2353 len 32768
18145 215/300 Read block 369 len 4096
18146 216/300 Write block 1535 len 36864 data: 5 4 4 4 4 2 4 4 5
18147 217/300 Read block 2672 len 16384
18148 218/300 Read block 212 len 20480
18149 219/300 Read block 1905 len 40960
18150 220/300 Write block 4747 len 32768 data: 9 9 7 6 7 6 7 8
18151 221/300 Write block 806 len 36864 data: 5 5 5 4 4 4 4 6 5
18152 222/300 Flush
18153 223/300 Flush
18154 224/300 Flush
18155 225/300 Read block 1964 len 36864
18156 226/300 Write block 3538 len 4096 data: 11
18157 227/300 Write block 2810 len 24576 data: 5 4 5 4 4 4
18158 228/300 Read block 2629 len 40960
18159 229/300 Read block 482 len 8192
18160 230/300 Read block 1006 len 28672
18161 231/300 Write block 968 len 4096 data: 3
18162 232/300 Write block 1237 len 12288 data: 5 7 7
18163 233/300 Write block 2291 len 36864 data: 3 3 3 3 4 4 4 3 5
18164 234/300 Flush
18165 235/300 Write block 1833 len 12288 data: 6 7 8
18166 236/300 Read block 799 len 32768
18167 237/300 Write block 1400 len 36864 data: 4 3 5 7 7 7 7 6 5
18168 238/300 Read block 97 len 20480
18169 239/300 Write block 953 len 36864 data: 6 5 5 5 6 7 7 7 7
18170 240/300 Read block 261 len 4096
18171 241/300 Write block 820 len 36864 data: 5 4 4 4 4 4 5 6 6
18172 242/300 Read block 374 len 28672
18173 243/300 Write block 3368 len 4096 data: 5
18174 244/300 Read block 3679 len 12288
18175 245/300 Read block 1099 len 32768
18176 246/300 Read block 4557 len 12288
18177 247/300 Read block 2186 len 28672
18178 248/300 Flush
18179 249/300 Read block 3724 len 20480
18180 250/300 Read block 4679 len 24576
18181 251/300 Read block 4579 len 8192
18182 252/300 Write block 4057 len 32768 data: 5 6 5 5 5 5 5 6
18183 253/300 Read block 2807 len 40960
18184 254/300 Write block 1554 len 32768 data: 4 3 5 6 5 4 4 6
18185 255/300 Flush
18186 256/300 Read block 1264 len 28672
18187 257/300 Read block 180 len 16384
18188 258/300 Write block 2376 len 28672 data: 3 3 4 3 5 6 6
18189 259/300 Read block 4636 len 32768
18190 260/300 Read block 916 len 4096
18191 261/300 Read block 2206 len 20480
18192 262/300 Write block 3260 len 8192 data: 7 7
18193 263/300 Read block 2973 len 28672
18194 264/300 Read block 164 len 20480
18195 265/300 Read block 2032 len 12288
18196 266/300 Write block 2888 len 24576 data: 5 5 5 6 7 4
18197 267/300 Write block 2377 len 12288 data: 4 5 4
18198 268/300 Flush
18199 269/300 Write block 4450 len 24576 data: 7 7 6 6 5 5
18200 270/300 Read block 2591 len 16384
18201 271/300 Read block 68 len 12288
18202 272/300 Read block 2100 len 4096
18203 273/300 Read block 817 len 20480
18204 274/300 Read block 2801 len 20480
18205 275/300 Read block 3368 len 28672
18206 276/300 Read block 898 len 12288
18207 277/300 Flush
18208 278/300 Read block 302 len 20480
18209 279/300 Write block 848 len 12288 data: 5 5 6
18210 280/300 Write block 2283 len 12288 data: 6 5 5
18211 281/300 Read block 2906 len 28672
18212 282/300 Flush
18213 283/300 Write block 658 len 8192 data: 4 3
18214 284/300 Write block 2016 len 36864 data: 9 8 5 6 6 6 7 6 6
18215 285/300 Flush
18216 286/300 Write block 1634 len 36864 data: 10 9 8 8 7 7 6 5 5
18217 287/300 Write block 792 len 24576 data: 2 2 3 3 3 3
18218 288/300 Read block 4801 len 40960
18219 289/300 Read block 4289 len 12288
18220 290/300 Read block 2970 len 32768
18221 291/300 Read block 2007 len 28672
18222 292/300 Read block 4758 len 20480
18223 293/300 Read block 432 len 40960
18224 294/300 Read block 3625 len 32768
18225 295/300 Read block 2299 len 8192
18226 296/300 Write block 2066 len 12288 data: 4 4 3
18227 297/300 Write block 4970 len 16384 data: 7 7 7 7
18228 298/300 Flush
18229 299/300 Write block 913 len 24576 data: 6 7 9 9 9 9
18230 300/300 Write block 3748 len 12288 data: 5 5 7
18231 Main task has work to do, go find it
18232 got dsc Start(0)
18233 start 0
18234 Oct 13 18:13:06.542 INFO request completed, latency_us: 303, response_code: 204, uri: /start/cid/0, method: POST, req_id: d65d8d80-12f2-4fc4-ae19-32cb0139c7f3, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
18235 Replay: started 0, returned:Ok(())
18236 ----------------------------------------------------------------
18237 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
18238 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
18239 6963 Acked 7962 Write 9 New Done Done false
18240 6964 Acked 7963 Read 4 New Done Done false
18241 6965 Acked 7964 Flush 0 New Done Done false
18242 6966 Acked 7965 Read 3 New Done Done false
18243 6967 Acked 7966 Read 10 New Done Done false
18244 6968 Acked 7967 Write 2 New Done Done false
18245 6969 Acked 7968 Write 9 New Done Done false
18246 6970 Acked 7969 Read 3 New Done Done false
18247 6971 Acked 7970 Read 7 New Done Done false
18248 6972 Acked 7971 Write 3 New Done Done false
18249 6973 Acked 7972 Read 10 New Done Done false
18250 6974 Acked 7973 Read 10 New Done Done false
18251 6975 Acked 7974 Read 1 New Done Done false
18252 6976 Acked 7975 Read 7 New Done Done false
18253 6977 Acked 7976 Write 10 New Done Done false
18254 6978 Acked 7977 Write 9 New Done Done false
18255 6979 Acked 7978 Read 1 New Done Done false
18256 6980 Acked 7979 Write 9 New Done Done false
18257 6981 Acked 7980 Read 3 New Done Done false
18258 6982 Acked 7981 Read 5 New Done Done false
18259 6983 Acked 7982 Read 9 New Done Done false
18260 6984 Acked 7983 Read 8 New Done Done false
18261 6985 Acked 7984 Write 7 New Done Done false
18262 6986 Acked 7985 Write 4 New Done Done false
18263 6987 Acked 7986 Read 10 New Done Done false
18264 6988 Acked 7987 Read 8 New Done Done false
18265 6989 Acked 7988 Read 10 New Done Done false
18266 6990 Acked 7989 Flush 0 New Done Done false
18267 6991 Acked 7990 Read 9 New Done Done false
18268 6992 Acked 7991 Read 7 New Done Done false
18269 6993 Acked 7992 Read 9 New Done Done false
18270 6994 Acked 7993 Read 2 New Done Done false
18271 6995 Acked 7994 Read 8 New Done Done false
18272 6996 Acked 7995 Write 1 New Done Done false
18273 6997 Acked 7996 Read 7 New Done Done false
18274 6998 Acked 7997 Write 3 New Done Done false
18275 6999 Acked 7998 Read 2 New Done Done false
18276 7000 Acked 7999 Read 9 New Done Done false
18277 7001 Acked 8000 Read 4 New Done Done false
18278 7002 Acked 8001 Write 1 New Done Done false
18279 7003 Acked 8002 Flush 0 New Done Done false
18280 7004 Acked 8003 Flush 0 New Done Done false
18281 7005 Acked 8004 Read 3 New Done Done false
18282 7006 Acked 8005 Read 2 New Done Done false
18283 7007 Acked 8006 Flush 0 New Done Done false
18284 7008 Acked 8007 Write 1 New Done Done false
18285 7009 Acked 8008 Write 6 New Done Done false
18286 7010 Acked 8009 Read 6 New Done Done false
18287 7011 Acked 8010 Read 1 New Done Done false
18288 7012 Acked 8011 Read 9 New Done Done false
18289 7013 Acked 8012 Write 1 New Done Done false
18290 7014 Acked 8013 Read 7 New Done Done false
18291 7015 Acked 8014 Flush 0 New Done Done false
18292 7016 Acked 8015 Write 10 New Done Done false
18293 7017 Acked 8016 Read 7 New Done Done false
18294 7018 Acked 8017 Read 6 New Done Done false
18295 7019 Acked 8018 Write 7 New Done Done false
18296 7020 Acked 8019 Read 10 New Done Done false
18297 7021 Acked 8020 Write 7 New Done Done false
18298 7022 Acked 8021 Write 2 New Done Done false
18299 7023 Acked 8022 Write 1 New Done Done false
18300 7024 Acked 8023 Write 2 New Done Done false
18301 7025 Acked 8024 Flush 0 New Done Done false
18302 7026 Acked 8025 Read 7 New Done Done false
18303 7027 Acked 8026 Write 5 New Done Done false
18304 7028 Acked 8027 Write 4 New Done Done false
18305 7029 Acked 8028 Read 6 New Done Done false
18306 7030 Acked 8029 Write 5 New Done Done false
18307 7031 Acked 8030 Write 5 New Done Done false
18308 7032 Acked 8031 Read 10 New Done Done false
18309 7033 Acked 8032 Flush 0 New Done Done false
18310 7034 Acked 8033 Write 5 New Done Done false
18311 7035 Acked 8034 Write 8 New Done Done false
18312 7036 Acked 8035 Read 1 New Done Done false
18313 7037 Acked 8036 Read 3 New Done Done false
18314 7038 Acked 8037 Write 7 New Done Done false
18315 7039 Acked 8038 Read 2 New Done Done false
18316 7040 Acked 8039 Flush 0 New Done Done false
18317 7041 Acked 8040 Write 6 New Done Done false
18318 7042 Acked 8041 Write 9 New Done Done false
18319 7043 Acked 8042 Write 9 New Done Done false
18320 7044 Acked 8043 Read 9 New Done Done false
18321 7045 Acked 8044 Read 5 New Done Done false
18322 7046 Acked 8045 Flush 0 New Done Done false
18323 7047 Acked 8046 Read 2 New Done Done false
18324 7048 Acked 8047 Read 3 New Done Done false
18325 7049 Acked 8048 Read 8 New Done Done false
18326 7050 Acked 8049 Write 3 New Done Done false
18327 7051 Acked 8050 Read 10 New Done Done false
18328 7052 Acked 8051 Flush 0 New Done Done false
18329 7053 Acked 8052 Read 5 New Done Done false
18330 7054 Acked 8053 Write 7 New Done Done false
18331 7055 Acked 8054 Write 7 New Done Done false
18332 7056 Acked 8055 Read 6 New Done Done false
18333 7057 Acked 8056 Write 4 New Done Done false
18334 7058 Acked 8057 Write 5 New Done Done false
18335 7059 Acked 8058 Write 7 New Done Done false
18336 7060 Acked 8059 Read 10 New Done Done false
18337 7061 Acked 8060 Read 10 New Done Done false
18338 7062 Acked 8061 Write 7 New Done Done false
18339 7063 Acked 8062 Read 3 New Done Done false
18340 7064 Acked 8063 Read 7 New Done Done false
18341 7065 Acked 8064 Write 4 New Done Done false
18342 7066 Acked 8065 Flush 0 New Done Done false
18343 7067 Acked 8066 Flush 0 New Done Done false
18344 7068 Acked 8067 Write 2 New Done Done false
18345 7069 Acked 8068 Write 6 New Done Done false
18346 7070 Acked 8069 Read 9 New Done Done false
18347 7071 Acked 8070 Read 10 New Done Done false
18348 7072 Acked 8071 Read 3 New Done Done false
18349 7073 Acked 8072 Read 8 New Done Done false
18350 7074 Acked 8073 Flush 0 New Done Done false
18351 7075 Acked 8074 Read 5 New Done Done false
18352 7076 Acked 8075 Write 2 New Done Done false
18353 7077 Acked 8076 Read 9 New Done Done false
18354 7078 Acked 8077 Write 6 New Done Done false
18355 7079 Acked 8078 Read 8 New Done Done false
18356 7080 Acked 8079 Read 9 New Done Done false
18357 7081 Acked 8080 Read 3 New Done Done false
18358 7082 Acked 8081 Read 6 New Done Done false
18359 7083 Acked 8082 Read 4 New Done Done false
18360 7084 Acked 8083 Read 4 New Done Done false
18361 7085 Acked 8084 Flush 0 New Done Done false
18362 7086 Acked 8085 Write 5 New Done Done false
18363 7087 Acked 8086 Read 4 New Done Done false
18364 7088 Acked 8087 Read 9 New Done Done false
18365 7089 Acked 8088 Read 4 New Done Done false
18366 7090 Acked 8089 Flush 0 New Done Done false
18367 7091 Acked 8090 Read 2 New Done Done false
18368 7092 Acked 8091 Write 10 New Done Done false
18369 7093 Acked 8092 Flush 0 New Done Done false
18370 7094 Acked 8093 Write 8 New Done Done false
18371 7095 Acked 8094 Write 7 New Done Done false
18372 7096 Acked 8095 Write 1 New Done Done false
18373 7097 Acked 8096 Read 6 New Done Done false
18374 7098 Acked 8097 Write 4 New Done Done false
18375 7099 Acked 8098 Write 10 New Done Done false
18376 7100 Acked 8099 Write 8 New Done Done false
18377 7101 Acked 8100 Read 8 New Done Done false
18378 7102 Acked 8101 Write 8 New Done Done false
18379 7103 Acked 8102 Write 9 New Done Done false
18380 7104 Acked 8103 Write 2 New Done Done false
18381 7105 Acked 8104 Read 1 New Done Done false
18382 7106 Acked 8105 Read 1 New Done Done false
18383 7107 Acked 8106 Write 6 New Done Done false
18384 7108 Acked 8107 Read 2 New Done Done false
18385 7109 Acked 8108 Write 8 New Done Done false
18386 7110 Acked 8109 Write 2 New Done Done false
18387 7111 Acked 8110 Write 8 New Done Done false
18388 7112 Acked 8111 Read 2 New Done Done false
18389 7113 Acked 8112 Flush 0 New Done Done false
18390 7114 Acked 8113 Read 7 New Done Done false
18391 7115 Acked 8114 Flush 0 New Done Done false
18392 7116 Acked 8115 Read 10 New Done Done false
18393 7117 Acked 8116 Read 9 New Done Done false
18394 7118 Acked 8117 Read 4 New Done Done false
18395 7119 Acked 8118 Read 9 New Done Done false
18396 7120 Acked 8119 Read 7 New Done Done false
18397 7121 Acked 8120 Read 8 New Done Done false
18398 7122 Acked 8121 Read 9 New Done Done false
18399 7123 Acked 8122 Read 8 New Done Done false
18400 7124 Acked 8123 Read 6 New Done Done false
18401 7125 Acked 8124 Write 5 New Done Done false
18402 7126 Acked 8125 Read 3 New Done Done false
18403 7127 Acked 8126 Read 5 New Done Done false
18404 7128 Acked 8127 Write 10 New Done Done false
18405 7129 Acked 8128 Write 6 New Done Done false
18406 7130 Acked 8129 Read 8 New Done Done false
18407 7131 Acked 8130 Write 1 New Done Done false
18408 7132 Acked 8131 Write 3 New Done Done false
18409 7133 Acked 8132 Write 7 New Done Done false
18410 7134 Acked 8133 Write 4 New Done Done false
18411 7135 Acked 8134 Write 5 New Done Done false
18412 7136 Acked 8135 Read 4 New Done Done false
18413 7137 Acked 8136 Read 3 New Done Done false
18414 7138 Acked 8137 Write 10 New Done Done false
18415 7139 Acked 8138 Read 1 New Done Done false
18416 7140 Acked 8139 Write 9 New Done Done false
18417 7141 Acked 8140 Read 7 New Done Done false
18418 7142 Acked 8141 Write 4 New Done Done false
18419 7143 Acked 8142 Write 10 New Done Done false
18420 7144 Acked 8143 Write 3 New Done Done false
18421 7145 Acked 8144 Read 8 New Done Done false
18422 7146 Acked 8145 Read 6 New Done Done false
18423 7147 Acked 8146 Write 5 New Done Done false
18424 7148 Acked 8147 Write 9 New Done Done false
18425 7149 Acked 8148 Read 4 New Done Done false
18426 7150 Acked 8149 Write 6 New Done Done false
18427 7151 Acked 8150 Write 2 New Done Done false
18428 7152 Acked 8151 Read 7 New Done Done false
18429 7153 Acked 8152 Read 10 New Done Done false
18430 7154 Acked 8153 Read 7 New Done Done false
18431 7155 Acked 8154 Read 1 New Done Done false
18432 7156 Acked 8155 Write 6 New Done Done false
18433 7157 Acked 8156 Read 6 New Done Done false
18434 7158 Acked 8157 Write 6 New Done Done false
18435 7159 Acked 8158 Read 5 New Done Done false
18436 7160 Acked 8159 Write 8 New Done Done false
18437 7161 Acked 8160 Read 4 New Done Done false
18438 7162 Acked 8161 Flush 0 New Done Done false
18439 7163 Acked 8162 Flush 0 New Done Done false
18440 7164 Acked 8163 Flush 0 New Done Done false
18441 7165 Acked 8164 Read 1 New Done Done false
18442 7166 Acked 8165 Write 4 New Done Done false
18443 7167 Acked 8166 Write 2 New Done Done false
18444 7168 Acked 8167 Flush 0 New Done Done false
18445 7169 Acked 8168 Read 6 New Done Done false
18446 7170 Acked 8169 Write 4 New Done Done false
18447 7171 Acked 8170 Read 2 New Done Done false
18448 7172 Acked 8171 Read 1 New Done Done false
18449 7173 Acked 8172 Read 6 New Done Done false
18450 7174 Acked 8173 Write 1 New Done Done false
18451 7175 Acked 8174 Read 8 New Done Done false
18452 7176 Acked 8175 Read 8 New Done Done false
18453 7177 Acked 8176 Read 4 New Done Done false
18454 7178 Acked 8177 Read 8 New Done Done false
18455 7179 Acked 8178 Read 1 New Done Done false
18456 7180 Acked 8179 Write 9 New Done Done false
18457 7181 Acked 8180 Read 4 New Done Done false
18458 7182 Acked 8181 Read 5 New Done Done false
18459 7183 Acked 8182 Read 10 New Done Done false
18460 7184 Acked 8183 Write 8 New Done Done false
18461 7185 Acked 8184 Write 9 New Done Done false
18462 7186 Acked 8185 Flush 0 New Done Done false
18463 7187 Acked 8186 Flush 0 New Done Done false
18464 7188 Acked 8187 Flush 0 New Done Done false
18465 7189 Acked 8188 Read 9 New Done Done false
18466 7190 Acked 8189 Write 1 New Done Done false
18467 7191 Acked 8190 Write 6 New Done Done false
18468 7192 Acked 8191 Read 10 New Done Done false
18469 7193 Acked 8192 Read 2 New Done Done false
18470 7194 Acked 8193 Read 7 New Done Done false
18471 7195 Acked 8194 Write 1 New Done Done false
18472 7196 Acked 8195 Write 3 New Done Done false
18473 7197 Acked 8196 Write 9 New Done Done false
18474 7198 Acked 8197 Flush 0 New Done Done false
18475 7199 Acked 8198 Write 3 New Done Done false
18476 7200 Acked 8199 Read 8 New Done Done false
18477 7201 Acked 8200 Write 9 New Done Done false
18478 7202 Acked 8201 Read 5 New Done Done false
18479 7203 Acked 8202 Write 9 New Done Done false
18480 7204 Acked 8203 Read 1 New Done Done false
18481 7205 Acked 8204 Write 9 New Done Done false
18482 7206 Acked 8205 Read 7 New Done Done false
18483 7207 Acked 8206 Write 1 New Done Done false
18484 7208 Acked 8207 Read 3 New Done Done false
18485 7209 Acked 8208 Read 8 New Done Done false
18486 7210 Acked 8209 Read 3 New Done Done false
18487 7211 Acked 8210 Read 7 New Done Done false
18488 7212 Acked 8211 Flush 0 New Done Done false
18489 7213 Acked 8212 Read 5 New Done Done false
18490 7214 Acked 8213 Read 6 New Done Done false
18491 7215 Acked 8214 Read 2 New Done Done false
18492 7216 Acked 8215 Write 8 New Done Done false
18493 7217 Acked 8216 Read 10 New Done Done false
18494 7218 Acked 8217 Write 8 New Done Done false
18495 7219 Acked 8218 Flush 0 New Done Done false
18496 7220 Acked 8219 Read 7 New Done Done false
18497 7221 Acked 8220 Read 4 New Done Done false
18498 7222 Acked 8221 Write 7 New Done Done false
18499 7223 Acked 8222 Read 8 New Done Done false
18500 7224 Acked 8223 Read 1 New Done Done false
18501 7225 Acked 8224 Read 5 New Done Done false
18502 7226 Acked 8225 Write 2 New Done Done false
18503 7227 Acked 8226 Read 7 New Done Done false
18504 7228 Acked 8227 Read 5 New Done Done false
18505 7229 Acked 8228 Read 3 New Done Done false
18506 7230 Acked 8229 Write 6 New Done Done false
18507 7231 Acked 8230 Write 3 New Done Done false
18508 7232 Acked 8231 Flush 0 New Done Done false
18509 7233 Acked 8232 Write 6 New Done Done false
18510 7234 Acked 8233 Read 4 New Done Done false
18511 7235 Acked 8234 Read 3 New Done Done false
18512 7236 Acked 8235 Read 1 New Done Done false
18513 7237 Acked 8236 Read 5 New Done Done false
18514 7238 Acked 8237 Read 5 New Done Done false
18515 7239 Acked 8238 Read 7 New Done Done false
18516 7240 Acked 8239 Read 3 New Done Done false
18517 7241 Acked 8240 Flush 0 New Done Done false
18518 7242 Acked 8241 Read 5 New Done Done false
18519 7243 Acked 8242 Write 3 New Done Done false
18520 7244 Acked 8243 Write 3 New Done Done false
18521 7245 Acked 8244 Read 7 New Done Done false
18522 7246 Acked 8245 Flush 0 New Done Done false
18523 7247 Acked 8246 Write 2 New Done Done false
18524 7248 Acked 8247 Write 9 New Done Done false
18525 7249 Acked 8248 Flush 0 New Done Done false
18526 7250 Acked 8249 Write 9 New Done Done false
18527 7251 Acked 8250 Write 6 New Done Done false
18528 7252 Acked 8251 Read 10 New Done Done false
18529 7253 Acked 8252 Read 3 New Done Done false
18530 7254 Acked 8253 Read 8 New Done Done false
18531 7255 Acked 8254 Read 7 New Done Done false
18532 7256 Acked 8255 Read 5 New Done Done false
18533 7257 Acked 8256 Read 10 New Done Done false
18534 7258 Acked 8257 Read 8 New Done Done false
18535 7259 Acked 8258 Read 2 New Done Done false
18536 7260 Acked 8259 Write 3 New Done Done false
18537 7261 Acked 8260 Write 4 New Done Done false
18538 7262 Acked 8261 Flush 0 New Done Done false
18539 7263 Acked 8262 Write 6 New Sent Sent false
18540 7264 Acked 8263 Write 3 New Sent Sent false
18541 STATES DS:0 DS:1 DS:2 TOTAL
18542 New 302 0 0 302
18543 Sent 0 2 2 4
18544 Done 0 300 300 600
18545 Skipped 0 0 0 0
18546 Error 0 0 0 0
18547 Last Flush: 7961 8261 8261
18548 Downstairs last five completed: 7961 7960 7959 7958 7957 7956
18549 Upstairs last five completed: 7264 7263 7262 7261 7260 7259
18550 CLIENT: Up:0 ds:302 act:2
18551 ----------------------------------------------------------------
18552 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
18553 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
18554 6963 Acked 7962 Write 9 New Done Done false
18555 6964 Acked 7963 Read 4 New Done Done false
18556 6965 Acked 7964 Flush 0 New Done Done false
18557 6966 Acked 7965 Read 3 New Done Done false
18558 6967 Acked 7966 Read 10 New Done Done false
18559 6968 Acked 7967 Write 2 New Done Done false
18560 6969 Acked 7968 Write 9 New Done Done false
18561 6970 Acked 7969 Read 3 New Done Done false
18562 6971 Acked 7970 Read 7 New Done Done false
18563 6972 Acked 7971 Write 3 New Done Done false
18564 6973 Acked 7972 Read 10 New Done Done false
18565 6974 Acked 7973 Read 10 New Done Done false
18566 6975 Acked 7974 Read 1 New Done Done false
18567 6976 Acked 7975 Read 7 New Done Done false
18568 6977 Acked 7976 Write 10 New Done Done false
18569 6978 Acked 7977 Write 9 New Done Done false
18570 6979 Acked 7978 Read 1 New Done Done false
18571 6980 Acked 7979 Write 9 New Done Done false
18572 6981 Acked 7980 Read 3 New Done Done false
18573 6982 Acked 7981 Read 5 New Done Done false
18574 6983 Acked 7982 Read 9 New Done Done false
18575 6984 Acked 7983 Read 8 New Done Done false
18576 6985 Acked 7984 Write 7 New Done Done false
18577 6986 Acked 7985 Write 4 New Done Done false
18578 6987 Acked 7986 Read 10 New Done Done false
18579 6988 Acked 7987 Read 8 New Done Done false
18580 6989 Acked 7988 Read 10 New Done Done false
18581 6990 Acked 7989 Flush 0 New Done Done false
18582 6991 Acked 7990 Read 9 New Done Done false
18583 6992 Acked 7991 Read 7 New Done Done false
18584 6993 Acked 7992 Read 9 New Done Done false
18585 6994 Acked 7993 Read 2 New Done Done false
18586 6995 Acked 7994 Read 8 New Done Done false
18587 6996 Acked 7995 Write 1 New Done Done false
18588 6997 Acked 7996 Read 7 New Done Done false
18589 6998 Acked 7997 Write 3 New Done Done false
18590 6999 Acked 7998 Read 2 New Done Done false
18591 7000 Acked 7999 Read 9 New Done Done false
18592 7001 Acked 8000 Read 4 New Done Done false
18593 7002 Acked 8001 Write 1 New Done Done false
18594 7003 Acked 8002 Flush 0 New Done Done false
18595 7004 Acked 8003 Flush 0 New Done Done false
18596 7005 Acked 8004 Read 3 New Done Done false
18597 7006 Acked 8005 Read 2 New Done Done false
18598 7007 Acked 8006 Flush 0 New Done Done false
18599 7008 Acked 8007 Write 1 New Done Done false
18600 7009 Acked 8008 Write 6 New Done Done false
18601 7010 Acked 8009 Read 6 New Done Done false
18602 7011 Acked 8010 Read 1 New Done Done false
18603 7012 Acked 8011 Read 9 New Done Done false
18604 7013 Acked 8012 Write 1 New Done Done false
18605 7014 Acked 8013 Read 7 New Done Done false
18606 7015 Acked 8014 Flush 0 New Done Done false
18607 7016 Acked 8015 Write 10 New Done Done false
18608 7017 Acked 8016 Read 7 New Done Done false
18609 7018 Acked 8017 Read 6 New Done Done false
18610 7019 Acked 8018 Write 7 New Done Done false
18611 7020 Acked 8019 Read 10 New Done Done false
18612 7021 Acked 8020 Write 7 New Done Done false
18613 7022 Acked 8021 Write 2 New Done Done false
18614 7023 Acked 8022 Write 1 New Done Done false
18615 7024 Acked 8023 Write 2 New Done Done false
18616 7025 Acked 8024 Flush 0 New Done Done false
18617 7026 Acked 8025 Read 7 New Done Done false
18618 7027 Acked 8026 Write 5 New Done Done false
18619 7028 Acked 8027 Write 4 New Done Done false
18620 7029 Acked 8028 Read 6 New Done Done false
18621 7030 Acked 8029 Write 5 New Done Done false
18622 7031 Acked 8030 Write 5 New Done Done false
18623 7032 Acked 8031 Read 10 New Done Done false
18624 7033 Acked 8032 Flush 0 New Done Done false
18625 7034 Acked 8033 Write 5 New Done Done false
18626 7035 Acked 8034 Write 8 New Done Done false
18627 7036 Acked 8035 Read 1 New Done Done false
18628 7037 Acked 8036 Read 3 New Done Done false
18629 7038 Acked 8037 Write 7 New Done Done false
18630 7039 Acked 8038 Read 2 New Done Done false
18631 7040 Acked 8039 Flush 0 New Done Done false
18632 7041 Acked 8040 Write 6 New Done Done false
18633 7042 Acked 8041 Write 9 New Done Done false
18634 7043 Acked 8042 Write 9 New Done Done false
18635 7044 Acked 8043 Read 9 New Done Done false
18636 7045 Acked 8044 Read 5 New Done Done false
18637 7046 Acked 8045 Flush 0 New Done Done false
18638 7047 Acked 8046 Read 2 New Done Done false
18639 7048 Acked 8047 Read 3 New Done Done false
18640 7049 Acked 8048 Read 8 New Done Done false
18641 7050 Acked 8049 Write 3 New Done Done false
18642 7051 Acked 8050 Read 10 New Done Done false
18643 7052 Acked 8051 Flush 0 New Done Done false
18644 7053 Acked 8052 Read 5 New Done Done false
18645 7054 Acked 8053 Write 7 New Done Done false
18646 7055 Acked 8054 Write 7 New Done Done false
18647 7056 Acked 8055 Read 6 New Done Done false
18648 7057 Acked 8056 Write 4 New Done Done false
18649 7058 Acked 8057 Write 5 New Done Done false
18650 7059 Acked 8058 Write 7 New Done Done false
18651 7060 Acked 8059 Read 10 New Done Done false
18652 7061 Acked 8060 Read 10 New Done Done false
18653 7062 Acked 8061 Write 7 New Done Done false
18654 7063 Acked 8062 Read 3 New Done Done false
18655 7064 Acked 8063 Read 7 New Done Done false
18656 7065 Acked 8064 Write 4 New Done Done false
18657 7066 Acked 8065 Flush 0 New Done Done false
18658 7067 Acked 8066 Flush 0 New Done Done false
18659 7068 Acked 8067 Write 2 New Done Done false
18660 7069 Acked 8068 Write 6 New Done Done false
18661 7070 Acked 8069 Read 9 New Done Done false
18662 7071 Acked 8070 Read 10 New Done Done false
18663 7072 Acked 8071 Read 3 New Done Done false
18664 7073 Acked 8072 Read 8 New Done Done false
18665 7074 Acked 8073 Flush 0 New Done Done false
18666 7075 Acked 8074 Read 5 New Done Done false
18667 7076 Acked 8075 Write 2 New Done Done false
18668 7077 Acked 8076 Read 9 New Done Done false
18669 7078 Acked 8077 Write 6 New Done Done false
18670 7079 Acked 8078 Read 8 New Done Done false
18671 7080 Acked 8079 Read 9 New Done Done false
18672 7081 Acked 8080 Read 3 New Done Done false
18673 7082 Acked 8081 Read 6 New Done Done false
18674 7083 Acked 8082 Read 4 New Done Done false
18675 7084 Acked 8083 Read 4 New Done Done false
18676 7085 Acked 8084 Flush 0 New Done Done false
18677 7086 Acked 8085 Write 5 New Done Done false
18678 7087 Acked 8086 Read 4 New Done Done false
18679 7088 Acked 8087 Read 9 New Done Done false
18680 7089 Acked 8088 Read 4 New Done Done false
18681 7090 Acked 8089 Flush 0 New Done Done false
18682 7091 Acked 8090 Read 2 New Done Done false
18683 7092 Acked 8091 Write 10 New Done Done false
18684 7093 Acked 8092 Flush 0 New Done Done false
18685 7094 Acked 8093 Write 8 New Done Done false
18686 7095 Acked 8094 Write 7 New Done Done false
18687 7096 Acked 8095 Write 1 New Done Done false
18688 7097 Acked 8096 Read 6 New Done Done false
18689 7098 Acked 8097 Write 4 New Done Done false
18690 7099 Acked 8098 Write 10 New Done Done false
18691 7100 Acked 8099 Write 8 New Done Done false
18692 7101 Acked 8100 Read 8 New Done Done false
18693 7102 Acked 8101 Write 8 New Done Done false
18694 7103 Acked 8102 Write 9 New Done Done false
18695 7104 Acked 8103 Write 2 New Done Done false
18696 7105 Acked 8104 Read 1 New Done Done false
18697 7106 Acked 8105 Read 1 New Done Done false
18698 7107 Acked 8106 Write 6 New Done Done false
18699 7108 Acked 8107 Read 2 New Done Done false
18700 7109 Acked 8108 Write 8 New Done Done false
18701 7110 Acked 8109 Write 2 New Done Done false
18702 7111 Acked 8110 Write 8 New Done Done false
18703 7112 Acked 8111 Read 2 New Done Done false
18704 7113 Acked 8112 Flush 0 New Done Done false
18705 7114 Acked 8113 Read 7 New Done Done false
18706 7115 Acked 8114 Flush 0 New Done Done false
18707 7116 Acked 8115 Read 10 New Done Done false
18708 7117 Acked 8116 Read 9 New Done Done false
18709 7118 Acked 8117 Read 4 New Done Done false
18710 7119 Acked 8118 Read 9 New Done Done false
18711 7120 Acked 8119 Read 7 New Done Done false
18712 7121 Acked 8120 Read 8 New Done Done false
18713 7122 Acked 8121 Read 9 New Done Done false
18714 7123 Acked 8122 Read 8 New Done Done false
18715 7124 Acked 8123 Read 6 New Done Done false
18716 7125 Acked 8124 Write 5 New Done Done false
18717 7126 Acked 8125 Read 3 New Done Done false
18718 7127 Acked 8126 Read 5 New Done Done false
18719 7128 Acked 8127 Write 10 New Done Done false
18720 7129 Acked 8128 Write 6 New Done Done false
18721 7130 Acked 8129 Read 8 New Done Done false
18722 7131 Acked 8130 Write 1 New Done Done false
18723 7132 Acked 8131 Write 3 New Done Done false
18724 7133 Acked 8132 Write 7 New Done Done false
18725 7134 Acked 8133 Write 4 New Done Done false
18726 7135 Acked 8134 Write 5 New Done Done false
18727 7136 Acked 8135 Read 4 New Done Done false
18728 7137 Acked 8136 Read 3 New Done Done false
18729 7138 Acked 8137 Write 10 New Done Done false
18730 7139 Acked 8138 Read 1 New Done Done false
18731 7140 Acked 8139 Write 9 New Done Done false
18732 7141 Acked 8140 Read 7 New Done Done false
18733 7142 Acked 8141 Write 4 New Done Done false
18734 7143 Acked 8142 Write 10 New Done Done false
18735 7144 Acked 8143 Write 3 New Done Done false
18736 7145 Acked 8144 Read 8 New Done Done false
18737 7146 Acked 8145 Read 6 New Done Done false
18738 7147 Acked 8146 Write 5 New Done Done false
18739 7148 Acked 8147 Write 9 New Done Done false
18740 7149 Acked 8148 Read 4 New Done Done false
18741 7150 Acked 8149 Write 6 New Done Done false
18742 7151 Acked 8150 Write 2 New Done Done false
18743 7152 Acked 8151 Read 7 New Done Done false
18744 7153 Acked 8152 Read 10 New Done Done false
18745 7154 Acked 8153 Read 7 New Done Done false
18746 7155 Acked 8154 Read 1 New Done Done false
18747 7156 Acked 8155 Write 6 New Done Done false
18748 7157 Acked 8156 Read 6 New Done Done false
18749 7158 Acked 8157 Write 6 New Done Done false
18750 7159 Acked 8158 Read 5 New Done Done false
18751 7160 Acked 8159 Write 8 New Done Done false
18752 7161 Acked 8160 Read 4 New Done Done false
18753 7162 Acked 8161 Flush 0 New Done Done false
18754 7163 Acked 8162 Flush 0 New Done Done false
18755 7164 Acked 8163 Flush 0 New Done Done false
18756 7165 Acked 8164 Read 1 New Done Done false
18757 7166 Acked 8165 Write 4 New Done Done false
18758 7167 Acked 8166 Write 2 New Done Done false
18759 7168 Acked 8167 Flush 0 New Done Done false
18760 7169 Acked 8168 Read 6 New Done Done false
18761 7170 Acked 8169 Write 4 New Done Done false
18762 7171 Acked 8170 Read 2 New Done Done false
18763 7172 Acked 8171 Read 1 New Done Done false
18764 7173 Acked 8172 Read 6 New Done Done false
18765 7174 Acked 8173 Write 1 New Done Done false
18766 7175 Acked 8174 Read 8 New Done Done false
18767 7176 Acked 8175 Read 8 New Done Done false
18768 7177 Acked 8176 Read 4 New Done Done false
18769 7178 Acked 8177 Read 8 New Done Done false
18770 7179 Acked 8178 Read 1 New Done Done false
18771 7180 Acked 8179 Write 9 New Done Done false
18772 7181 Acked 8180 Read 4 New Done Done false
18773 7182 Acked 8181 Read 5 New Done Done false
18774 7183 Acked 8182 Read 10 New Done Done false
18775 7184 Acked 8183 Write 8 New Done Done false
18776 7185 Acked 8184 Write 9 New Done Done false
18777 7186 Acked 8185 Flush 0 New Done Done false
18778 7187 Acked 8186 Flush 0 New Done Done false
18779 7188 Acked 8187 Flush 0 New Done Done false
18780 7189 Acked 8188 Read 9 New Done Done false
18781 7190 Acked 8189 Write 1 New Done Done false
18782 7191 Acked 8190 Write 6 New Done Done false
18783 7192 Acked 8191 Read 10 New Done Done false
18784 7193 Acked 8192 Read 2 New Done Done false
18785 7194 Acked 8193 Read 7 New Done Done false
18786 7195 Acked 8194 Write 1 New Done Done false
18787 7196 Acked 8195 Write 3 New Done Done false
18788 7197 Acked 8196 Write 9 New Done Done false
18789 7198 Acked 8197 Flush 0 New Done Done false
18790 7199 Acked 8198 Write 3 New Done Done false
18791 7200 Acked 8199 Read 8 New Done Done false
18792 7201 Acked 8200 Write 9 New Done Done false
18793 7202 Acked 8201 Read 5 New Done Done false
18794 7203 Acked 8202 Write 9 New Done Done false
18795 7204 Acked 8203 Read 1 New Done Done false
18796 7205 Acked 8204 Write 9 New Done Done false
18797 7206 Acked 8205 Read 7 New Done Done false
18798 7207 Acked 8206 Write 1 New Done Done false
18799 7208 Acked 8207 Read 3 New Done Done false
18800 7209 Acked 8208 Read 8 New Done Done false
18801 7210 Acked 8209 Read 3 New Done Done false
18802 7211 Acked 8210 Read 7 New Done Done false
18803 7212 Acked 8211 Flush 0 New Done Done false
18804 7213 Acked 8212 Read 5 New Done Done false
18805 7214 Acked 8213 Read 6 New Done Done false
18806 7215 Acked 8214 Read 2 New Done Done false
18807 7216 Acked 8215 Write 8 New Done Done false
18808 7217 Acked 8216 Read 10 New Done Done false
18809 7218 Acked 8217 Write 8 New Done Done false
18810 7219 Acked 8218 Flush 0 New Done Done false
18811 7220 Acked 8219 Read 7 New Done Done false
18812 7221 Acked 8220 Read 4 New Done Done false
18813 7222 Acked 8221 Write 7 New Done Done false
18814 7223 Acked 8222 Read 8 New Done Done false
18815 7224 Acked 8223 Read 1 New Done Done false
18816 7225 Acked 8224 Read 5 New Done Done false
18817 7226 Acked 8225 Write 2 New Done Done false
18818 7227 Acked 8226 Read 7 New Done Done false
18819 7228 Acked 8227 Read 5 New Done Done false
18820 7229 Acked 8228 Read 3 New Done Done false
18821 7230 Acked 8229 Write 6 New Done Done false
18822 7231 Acked 8230 Write 3 New Done Done false
18823 7232 Acked 8231 Flush 0 New Done Done false
18824 7233 Acked 8232 Write 6 New Done Done false
18825 7234 Acked 8233 Read 4 New Done Done false
18826 7235 Acked 8234 Read 3 New Done Done false
18827 7236 Acked 8235 Read 1 New Done Done false
18828 7237 Acked 8236 Read 5 New Done Done false
18829 7238 Acked 8237 Read 5 New Done Done false
18830 7239 Acked 8238 Read 7 New Done Done false
18831 7240 Acked 8239 Read 3 New Done Done false
18832 7241 Acked 8240 Flush 0 New Done Done false
18833 7242 Acked 8241 Read 5 New Done Done false
18834 7243 Acked 8242 Write 3 New Done Done false
18835 7244 Acked 8243 Write 3 New Done Done false
18836 7245 Acked 8244 Read 7 New Done Done false
18837 7246 Acked 8245 Flush 0 New Done Done false
18838 7247 Acked 8246 Write 2 New Done Done false
18839 7248 Acked 8247 Write 9 New Done Done false
18840 7249 Acked 8248 Flush 0 New Done Done false
18841 7250 Acked 8249 Write 9 New Done Done false
18842 7251 Acked 8250 Write 6 New Done Done false
18843 7252 Acked 8251 Read 10 New Done Done false
18844 7253 Acked 8252 Read 3 New Done Done false
18845 7254 Acked 8253 Read 8 New Done Done false
18846 7255 Acked 8254 Read 7 New Done Done false
18847 7256 Acked 8255 Read 5 New Done Done false
18848 7257 Acked 8256 Read 10 New Done Done false
18849 7258 Acked 8257 Read 8 New Done Done false
18850 7259 Acked 8258 Read 2 New Done Done false
18851 7260 Acked 8259 Write 3 New Done Done false
18852 7261 Acked 8260 Write 4 New Done Done false
18853 7262 Acked 8261 Flush 0 New Done Done false
18854 7263 Acked 8262 Write 6 New Done Done false
18855 7264 Acked 8263 Write 3 New Done Done false
18856 7265 Acked 8264 Flush 0 New Done Done false
18857 STATES DS:0 DS:1 DS:2 TOTAL
18858 New 303 0 0 303
18859 Sent 0 0 0 0
18860 Done 0 303 303 606
18861 Skipped 0 0 0 0
18862 Error 0 0 0 0
18863 Last Flush: 7961 8264 8264
18864 Downstairs last five completed: 7961 7960 7959 7958 7957 7956
18865 Upstairs last five completed: 7265 7264 7263 7262 7261 7260
18866 CLIENT: Up:0 ds:303 act:2
18867 [8810] got start action
18868 [8810] I am going to restart
18869 Starting downstairs at port 8810
18870 Make output file at "/tmp/dsc/downstairs-8810.txt"
18871 [8810][0] reports Starting
18872 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1200)
18873 [8810][0] reports Running
188742023-10-13T18:13:12.567ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
188752023-10-13T18:13:12.567ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 0
188762023-10-13T18:13:12.567ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state Offline
188772023-10-13T18:13:12.567ZINFOcrucible: [0] client re-new 303 jobs since flush 7961 = downstairs
188782023-10-13T18:13:12.568ZINFOcrucible: [0] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
188792023-10-13T18:13:12.569ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
188802023-10-13T18:13:12.569ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
188812023-10-13T18:13:12.569ZINFOcrucible: Returning client:0 UUID:12345678-0000-0000-0000-000000008810 matches
188822023-10-13T18:13:12.569ZINFOcrucible: [0] send last flush ID to this DS: 7961
188832023-10-13T18:13:12.569ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
188842023-10-13T18:13:12.569ZINFOcrucible: [0] Replied this last flush ID: 7961
188852023-10-13T18:13:12.569ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Offline Active Active ds_transition to Replay
188862023-10-13T18:13:12.569ZINFOcrucible: [0] Transition from Offline to Replay
188872023-10-13T18:13:12.569ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
188882023-10-13T18:13:12.569ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Replay Active Active ds_transition to Active
188892023-10-13T18:13:12.569ZINFOcrucible: [0] Transition from Replay to Active
188902023-10-13T18:13:12.569ZINFOcrucible: [0] Starts cmd_loop
188912023-10-13T18:13:12.569ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
188922023-10-13T18:13:12.569ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
188932023-10-13T18:13:12.570ZINFOcrucible: Set check for repair
188942023-10-13T18:13:13.570ZWARNcrucible: [0] flow control end
188952023-10-13T18:13:13.571ZINFOcrucible: Checking if live repair is needed
188962023-10-13T18:13:13.571ZINFOcrucible: No Live Repair required at this time
18897 ----------------------------------------------------------------
18898 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18899 Downstairs last five completed: 8264 8263 8262 8261 8260 8259
18900 Upstairs last five completed: 7265 7264 7263 7262 7261 7260
18901 CLIENT: Up:0 ds:0 act:3
18902 Replay: All jobs finished, all DS active.
18903 Main task has work to do, go find it
18904 got dsc Stop(0)
18905 stop 0
18906 Oct 13 18:13:14.550 INFO request completed, latency_us: 525, response_code: 204, uri[8810] Got stop action so:false kr:false
18907 : /stop/cid/0, method: POST, req_id: f96e5722-5d37-4b60-9026-0b598473bd93, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
18908 Oct 13 18:13:14.551 INFO request completed, latency_us: 178, response_code: 200, uri: /state/cid/0, method: GET, req_id: 825b6efe-1001-4549-8d74-8bbc949604de, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
189092023-10-13T18:13:14.552ZWARNcrucible: [0] downstairs disconnected
189102023-10-13T18:13:14.552ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
189112023-10-13T18:13:14.552ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8810 closed looper = 0
189122023-10-13T18:13:14.552ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:false
189132023-10-13T18:13:14.552ZWARNcrucible: [0] pm_task rx.recv() is None
189142023-10-13T18:13:14.552ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
189152023-10-13T18:13:14.552ZINFOcrucible: [0] 127.0.0.1:8810 task reports offline
18916 [8810] Exited with: ExitStatus(unix_wait_status(9))
18917 [8810][0] reports Exit
18918 Oct 13 18:13:18.554 INFO request completed, latency_us: 259, response_code: 200, uri: /state/cid/0, method: GET, req_id: eee44f4e-9e73-4e6c-bb2c-5d02db70fee6, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
18919 001/300 Write block 82 len 12288 data: 4 5 6
18920 002/300 Write block 4194 len 24576 data: 5 6 8 7 5 6
18921 003/300 Read block 3414 len 4096
18922 004/300 Flush
18923 005/300 Flush
18924 006/300 Read block 4940 len 4096
18925 007/300 Read block 4948 len 36864
18926 008/300 Write block 3046 len 8192 data: 5 6
18927 009/300 Read block 700 len 4096
18928 010/300 Write block 1535 len 12288 data: 6 5 5
18929 011/300 Write block 4195 len 12288 data: 7 9 8
18930 012/300 Write block 1739 len 8192 data: 4 4
18931 013/300 Read block 2296 len 28672
18932 014/300 Read block 3634 len 4096
18933 015/300 Read block 2073 len 24576
18934 016/300 Read block 3571 len 16384
18935 017/300 Read block 263 len 20480
18936 018/300 Write block 327 len 20480 data: 7 5 4 4 4
18937 019/300 Read block 4469 len 12288
18938 020/300 Flush
18939 021/300 Write block 4465 len 24576 data: 4 3 3 3 3 3
18940 022/300 Write block 305 len 36864 data: 4 5 6 6 6 5 7 6 7
18941 023/300 Flush
18942 024/300 Write block 4718 len 16384 data: 3 3 4 5
18943 025/300 Write block 4027 len 8192 data: 3 3
18944 026/300 Flush
18945 027/300 Read block 3885 len 8192
18946 028/300 Read block 1838 len 28672
18947 029/300 Write block 4980 len 40960 data: 7 8 7 5 4 4 4 4 4 5
18948 030/300 Read block 4078 len 8192
18949 031/300 Flush
18950 032/300 Flush
18951 033/300 Flush
18952 034/300 Flush
18953 035/300 Write block 1812 len 32768 data: 8 7 4 5 5 5 6 5
18954 036/300 Write block 2326 len 16384 data: 4 6 6 7
18955 037/300 Write block 4787 len 28672 data: 7 7 8 5 6 6 6
18956 038/300 Read block 2759 len 24576
18957 039/300 Write block 1421 len 12288 data: 4 4 4
18958 040/300 Read block 874 len 20480
18959 041/300 Read block 3877 len 32768
18960 042/300 Write block 4571 len 40960 data: 5 5 5 4 4 3 3 3 6 7
18961 043/300 Write block 3020 len 36864 data: 7 7 6 6 6 6 5 4 4
18962 044/300 Read block 1596 len 40960
18963 045/300 Write block 4261 len 12288 data: 2 2 3
18964 046/300 Read block 3107 len 12288
18965 047/300 Read block 1946 len 8192
18966 048/300 Write block 1044 len 28672 data: 7 6 5 5 5 5 6
18967 049/300 Read block 3549 len 36864
18968 050/300 Flush
18969 051/300 Read block 3244 len 4096
18970 052/300 Read block 3066 len 12288
18971 053/300 Flush
18972 054/300 Write block 3695 len 20480 data: 8 8 7 6 5
18973 055/300 Read block 1195 len 32768
18974 056/300 Write block 2400 len 36864 data: 7 5 5 6 4 5 4 4 4
18975 057/300 Read block 4099 len 20480
18976 058/300 Read block 3905 len 16384
18977 059/300 Flush
18978 060/300 Read block 2598 len 8192
18979 061/300 Write block 2678 len 4096 data: 7
18980 062/300 Write block 3227 len 12288 data: 3 4 4
18981 063/300 Write block 4356 len 16384 data: 5 6 5 3
18982 064/300 Write block 3547 len 20480 data: 7 7 6 6 5
18983 065/300 Write block 355 len 16384 data: 3 3 3 4
18984 066/300 Write block 3159 len 32768 data: 6 7 6 5 5 4 4 4
18985 067/300 Read block 4446 len 40960
18986 068/300 Read block 4843 len 28672
18987 069/300 Write block 1857 len 8192 data: 6 6
18988 070/300 Read block 4333 len 32768
18989 071/300 Write block 707 len 8192 data: 5 4
18990 072/300 Read block 2435 len 32768
18991 073/300 Write block 2696 len 12288 data: 8 6 8
18992 074/300 Read block 3901 len 28672
18993 075/300 Read block 2797 len 4096
18994 076/300 Write block 3040 len 12288 data: 5 6 8
18995 077/300 Read block 4044 len 12288
18996 078/300 Read block 4791 len 20480
18997 079/300 Write block 596 len 28672 data: 6 7 9 9 10 9 9
18998 080/300 Read block 380 len 8192
18999 081/300 Read block 4625 len 40960
19000 082/300 Read block 4992 len 20480
19001 083/300 Read block 2232 len 24576
19002 084/300 Read block 856 len 20480
19003 085/300 Read block 1965 len 32768
19004 086/300 Read block 302 len 32768
19005 087/300 Write block 3265 len 20480 data: 4 3 3 3 4
19006 088/300 Read block 2233 len 40960
19007 089/300 Write block 545 len 36864 data: 4 4 3 2 2 3 6 5 5
19008 090/300 Write block 870 len 28672 data: 5 5 6 5 4 3 2
19009 091/300 Read block 4433 len 8192
19010 092/300 Read block 1590 len 20480
19011 093/300 Write block 4430 len 24576 data: 5 6 3 4 4 3
19012 094/300 Read block 4953 len 4096
19013 095/300 Flush
19014 096/300 Write block 2370 len 40960 data: 4 3 2 2 3 3 4 5 6 5
19015 097/300 Read block 4057 len 40960
19016 098/300 Read block 2148 len 4096
19017 099/300 Write block 54 len 40960 data: 2 2 2 2 3 3 3 2 2 2
19018 100/300 Write block 3700 len 24576 data: 4 4 3 4 6 6
19019 101/300 Flush
19020 102/300 Read block 2579 len 36864
19021 103/300 Write block 4315 len 12288 data: 7 5 3
19022 104/300 Read block 1594 len 4096
19023 105/300 Read block 446 len 24576
19024 106/300 Read block 3168 len 16384
19025 107/300 Flush
19026 108/300 Write block 3885 len 28672 data: 4 5 5 5 5 4 5
19027 109/300 Read block 548 len 8192
19028 110/300 Read block 2784 len 8192
19029 111/300 Read block 320 len 24576
19030 112/300 Write block 757 len 24576 data: 5 5 6 6 5 4
19031 113/300 Read block 2217 len 16384
19032 114/300 Read block 3182 len 8192
19033 115/300 Read block 4811 len 16384
19034 116/300 Write block 779 len 28672 data: 4 3 2 2 3 3 3
19035 117/300 Read block 1901 len 32768
19036 118/300 Read block 3346 len 36864
19037 119/300 Read block 2516 len 28672
19038 120/300 Read block 35 len 12288
19039 121/300 Write block 1181 len 28672 data: 3 3 4 4 5 6 6
19040 122/300 Write block 1788 len 32768 data: 5 8 6 6 7 6 5 6
19041 123/300 Read block 1495 len 12288
19042 124/300 Read block 149 len 20480
19043 125/300 Read block 3541 len 28672
19044 126/300 Read block 3014 len 32768
19045 127/300 Write block 3208 len 4096 data: 6
19046 128/300 Read block 2687 len 20480
19047 129/300 Read block 3865 len 32768
19048 130/300 Read block 3990 len 8192
19049 131/300 Write block 2541 len 28672 data: 3 4 2 2 2 3 4
19050 132/300 Write block 3106 len 12288 data: 4 5 6
19051 133/300 Flush
19052 134/300 Write block 2194 len 12288 data: 8 9 8
19053 135/300 Write block 119 len 32768 data: 5 5 5 4 6 7 7 6
19054 136/300 Flush
19055 137/300 Read block 2828 len 8192
19056 138/300 Write block 846 len 8192 data: 7 5
19057 139/300 Write block 3848 len 36864 data: 3 3 3 3 3 3 4 4 5
19058 140/300 Read block 938 len 12288
19059 141/300 Write block 2018 len 20480 data: 6 7 7 7 8
19060 142/300 Flush
19061 143/300 Write block 2718 len 4096 data: 6
19062 144/300 Write block 377 len 40960 data: 6 8 8 6 6 7 5 5 6 6
19063 145/300 Read block 215 len 36864
19064 146/300 Read block 3967 len 8192
19065 147/300 Read block 945 len 28672
19066 148/300 Read block 3373 len 40960
19067 149/300 Read block 1319 len 20480
19068 150/300 Write block 4142 len 8192 data: 2 3
19069 151/300 Write block 2433 len 4096 data: 3
19070 152/300 Read block 1923 len 28672
19071 153/300 Read block 3523 len 32768
19072 154/300 Read block 3790 len 8192
19073 155/300 Write block 2937 len 24576 data: 5 7 6 6 4 5
19074 156/300 Read block 2577 len 40960
19075 157/300 Write block 126 len 8192 data: 7 7
19076 158/300 Read block 3451 len 32768
19077 159/300 Write block 2804 len 16384 data: 3 4 5 5
19078 160/300 Read block 1246 len 8192
19079 161/300 Read block 4648 len 24576
19080 162/300 Flush
19081 163/300 Write block 2440 len 24576 data: 6 5 6 6 4 4
19082 164/300 Write block 683 len 8192 data: 3 3
19083 165/300 Read block 3403 len 36864
19084 166/300 Write block 3169 len 12288 data: 4 4 4
19085 167/300 Write block 3834 len 24576 data: 4 5 5 5 5 5
19086 168/300 Read block 451 len 36864
19087 169/300 Read block 2608 len 8192
19088 170/300 Write block 2167 len 20480 data: 4 5 5 6 7
19089 171/300 Write block 1142 len 24576 data: 4 3 2 2 4 5
19090 172/300 Write block 2164 len 12288 data: 4 4 5
19091 173/300 Read block 1898 len 40960
19092 174/300 Write block 1072 len 32768 data: 7 5 4 5 6 5 5 5
19093 175/300 Write block 1790 len 12288 data: 7 7 8
19094 176/300 Read block 366 len 40960
19095 177/300 Read block 1118 len 24576
19096 178/300 Write block 1030 len 12288 data: 7 5 6
19097 179/300 Write block 4593 len 4096 data: 7
19098 180/300 Write block 3505 len 4096 data: 5
19099 181/300 Read block 3934 len 40960
19100 182/300 Read block 1347 len 4096
19101 183/300 Write block 2528 len 20480 data: 8 8 7 8 8
19102 184/300 Read block 1678 len 20480
19103 185/300 Write block 4050 len 40960 data: 4 4 4 3 2 2 3 6 7 6
19104 186/300 Read block 3634 len 24576
19105 187/300 Flush
19106 188/300 Read block 2407 len 4096
19107 189/300 Read block 2707 len 8192
19108 190/300 Write block 495 len 28672 data: 5 5 3 3 3 3 3
19109 191/300 Read block 3368 len 36864
19110 192/300 Read block 1685 len 24576
19111 193/300 Flush
19112 194/300 Flush
19113 195/300 Read block 122 len 36864
19114 196/300 Read block 2787 len 40960
19115 197/300 Read block 2505 len 8192
19116 198/300 Write block 1333 len 4096 data: 7
19117 199/300 Flush
19118 200/300 Write block 438 len 36864 data: 5 5 4 5 5 7 5 6 9
19119 201/300 Read block 296 len 8192
19120 202/300 Write block 3533 len 4096 data: 7
19121 203/300 Read block 4275 len 24576
19122 204/300 Read block 4002 len 40960
19123 205/300 Read block 3631 len 32768
19124 206/300 Read block 1199 len 16384
19125 207/300 Read block 1827 len 20480
19126 208/300 Write block 1240 len 20480 data: 5 3 4 4 5
19127 209/300 Write block 3531 len 8192 data: 5 5
19128 210/300 Write block 4180 len 8192 data: 7 5
19129 211/300 Read block 2060 len 12288
19130 212/300 Write block 1652 len 8192 data: 7 5
19131 213/300 Write block 3250 len 4096 data: 5
19132 214/300 Write block 3150 len 8192 data: 6 4
19133 215/300 Read block 99 len 24576
19134 216/300 Read block 2868 len 4096
19135 217/300 Read block 1852 len 36864
19136 218/300 Read block 4167 len 40960
19137 219/300 Write block 3870 len 20480 data: 5 5 7 6 9
19138 220/300 Read block 917 len 36864
19139 221/300 Write block 1183 len 16384 data: 5 5 6 7
19140 222/300 Write block 3556 len 36864 data: 3 3 3 3 2 2 3 3 4
19141 223/300 Read block 2541 len 40960
19142 224/300 Read block 2713 len 24576
19143 225/300 Write block 2445 len 4096 data: 5
19144 226/300 Write block 2857 len 4096 data: 5
19145 227/300 Flush
19146 228/300 Read block 3424 len 36864
19147 229/300 Read block 901 len 28672
19148 230/300 Read block 2213 len 24576
19149 231/300 Read block 4588 len 20480
19150 232/300 Read block 4843 len 20480
19151 233/300 Read block 2767 len 24576
19152 234/300 Write block 4892 len 24576 data: 7 6 5 5 4 4
19153 235/300 Write block 4406 len 4096 data: 9
19154 236/300 Read block 2562 len 36864
19155 237/300 Read block 3289 len 20480
19156 238/300 Write block 4254 len 4096 data: 6
19157 239/300 Flush
19158 240/300 Read block 2089 len 16384
19159 241/300 Write block 1336 len 40960 data: 7 6 4 6 5 6 7 8 8 6
19160 242/300 Write block 2103 len 4096 data: 4
19161 243/300 Write block 2782 len 24576 data: 8 8 6 6 5 6
19162 244/300 Flush
19163 245/300 Read block 3820 len 28672
19164 246/300 Read block 1046 len 40960
19165 247/300 Read block 289 len 4096
19166 248/300 Read block 3821 len 8192
19167 249/300 Write block 63 len 40960 data: 3 3 3 5 5 5 5 5 5 5
19168 250/300 Write block 611 len 40960 data: 4 5 5 5 6 7 6 6 5 5
19169 251/300 Write block 1469 len 16384 data: 6 6 4 5
19170 252/300 Read block 517 len 36864
19171 253/300 Read block 4051 len 16384
19172 254/300 Read block 3523 len 32768
19173 255/300 Read block 1333 len 36864
19174 256/300 Read block 2241 len 28672
19175 257/300 Write block 845 len 36864 data: 5 8 6 6 6 7 7 9 7
19176 258/300 Read block 197 len 32768
19177 259/300 Flush
19178 260/300 Write block 936 len 8192 data: 7 7
19179 261/300 Write block 4306 len 28672 data: 7 9 9 6 6 8 9
19180 262/300 Read block 1722 len 4096
19181 263/300 Write block 3864 len 20480 data: 5 5 5 5 5
19182 264/300 Read block 1506 len 40960
19183 265/300 Write block 2587 len 16384 data: 6 7 6 6
19184 266/300 Read block 2474 len 36864
19185 267/300 Read block 3803 len 24576
19186 268/300 Read block 649 len 20480
19187 269/300 Flush
19188 270/300 Write block 2447 len 40960 data: 4 4 4 4 5 4 4 4 4 4
19189 271/300 Write block 3888 len 28672 data: 6 6 5 6 6 4 5
19190 272/300 Read block 4198 len 40960
19191 273/300 Write block 1288 len 8192 data: 3 3
19192 274/300 Read block 3307 len 8192
19193 275/300 Write block 2623 len 28672 data: 5 5 6 5 3 3 4
19194 276/300 Write block 3978 len 16384 data: 4 4 5 6
19195 277/300 Read block 204 len 40960
19196 278/300 Write block 2214 len 28672 data: 5 6 7 5 5 5 5
19197 279/300 Flush
19198 280/300 Read block 4775 len 40960
19199 281/300 Write block 2367 len 28672 data: 8 6 6 5 4 3 3
19200 282/300 Read block 1314 len 32768
19201 283/300 Read block 2364 len 36864
19202 284/300 Read block 4086 len 20480
19203 285/300 Read block 2274 len 28672
19204 286/300 Write block 1997 len 12288 data: 5 4 4
19205 287/300 Flush
19206 288/300 Read block 376 len 24576
19207 289/300 Write block 3014 len 36864 data: 8 5 4 4 5 4 8 8 7
19208 290/300 Read block 4349 len 32768
19209 291/300 Flush
19210 292/300 Read block 2584 len 24576
19211 293/300 Read block 3790 len 40960
19212 294/300 Read block 3453 len 16384
19213 295/300 Read block 699 len 24576
19214 296/300 Read block 2556 len 40960
19215 297/300 Read block 4498 len 8192
19216 298/300 Read block 3164 len 36864
19217 299/300 Read block 3059 len 16384
19218 300/300 Read block 1396 len 28672
19219 Main task has work to do, go find it
19220 got dsc Start(0)
19221 start 0
19222 Oct 13 18:13:19.519 INFO request completed, latency_us: 302, response_code: 204, uri: /start/cid/0, method: POST, req_id: 9b647856-7b21-4bda-84b8-39a3cbd0275f, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
19223 Replay: started 0, returned:Ok(())
19224 ----------------------------------------------------------------
19225 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
19226 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
19227 7266 Acked 8265 Write 3 New Done Done false
19228 7267 Acked 8266 Write 6 New Done Done false
19229 7268 Acked 8267 Read 1 New Done Done false
19230 7269 Acked 8268 Flush 0 New Done Done false
19231 7270 Acked 8269 Flush 0 New Done Done false
19232 7271 Acked 8270 Read 1 New Done Done false
19233 7272 Acked 8271 Read 9 New Done Done false
19234 7273 Acked 8272 Write 2 New Done Done false
19235 7274 Acked 8273 Read 1 New Done Done false
19236 7275 Acked 8274 Write 3 New Done Done false
19237 7276 Acked 8275 Write 3 New Done Done false
19238 7277 Acked 8276 Write 2 New Done Done false
19239 7278 Acked 8277 Read 7 New Done Done false
19240 7279 Acked 8278 Read 1 New Done Done false
19241 7280 Acked 8279 Read 6 New Done Done false
19242 7281 Acked 8280 Flush 0 New Done Done false
19243 7282 Acked 8281 Read 4 New Done Done false
19244 7283 Acked 8282 Read 5 New Done Done false
19245 7284 Acked 8283 Write 5 New Done Done false
19246 7285 Acked 8284 Read 3 New Done Done false
19247 7286 Acked 8285 Flush 0 New Done Done false
19248 7287 Acked 8286 Write 6 New Done Done false
19249 7288 Acked 8287 Write 9 New Done Done false
19250 7289 Acked 8288 Flush 0 New Done Done false
19251 7290 Acked 8289 Write 4 New Done Done false
19252 7291 Acked 8290 Write 2 New Done Done false
19253 7292 Acked 8291 Flush 0 New Done Done false
19254 7293 Acked 8292 Read 2 New Done Done false
19255 7294 Acked 8293 Read 7 New Done Done false
19256 7295 Acked 8294 Write 10 New Done Done false
19257 7296 Acked 8295 Read 2 New Done Done false
19258 7297 Acked 8296 Flush 0 New Done Done false
19259 7298 Acked 8297 Flush 0 New Done Done false
19260 7299 Acked 8298 Flush 0 New Done Done false
19261 7300 Acked 8299 Flush 0 New Done Done false
19262 7301 Acked 8300 Write 8 New Done Done false
19263 7302 Acked 8301 Write 4 New Done Done false
19264 7303 Acked 8302 Write 7 New Done Done false
19265 7304 Acked 8303 Read 6 New Done Done false
19266 7305 Acked 8304 Write 3 New Done Done false
19267 7306 Acked 8305 Read 5 New Done Done false
19268 7307 Acked 8306 Read 8 New Done Done false
19269 7308 Acked 8307 Write 10 New Done Done false
19270 7309 Acked 8308 Write 9 New Done Done false
19271 7310 Acked 8309 Read 10 New Done Done false
19272 7311 Acked 8310 Write 3 New Done Done false
19273 7312 Acked 8311 Read 3 New Done Done false
19274 7313 Acked 8312 Read 2 New Done Done false
19275 7314 Acked 8313 Write 7 New Done Done false
19276 7315 Acked 8314 Read 9 New Done Done false
19277 7316 Acked 8315 Flush 0 New Done Done false
19278 7317 Acked 8316 Read 1 New Done Done false
19279 7318 Acked 8317 Read 3 New Done Done false
19280 7319 Acked 8318 Flush 0 New Done Done false
19281 7320 Acked 8319 Write 5 New Done Done false
19282 7321 Acked 8320 Read 8 New Done Done false
19283 7322 Acked 8321 Write 9 New Done Done false
19284 7323 Acked 8322 Read 5 New Done Done false
19285 7324 Acked 8323 Read 4 New Done Done false
19286 7325 Acked 8324 Flush 0 New Done Done false
19287 7326 Acked 8325 Read 2 New Done Done false
19288 7327 Acked 8326 Write 1 New Done Done false
19289 7328 Acked 8327 Write 3 New Done Done false
19290 7329 Acked 8328 Write 4 New Done Done false
19291 7330 Acked 8329 Write 5 New Done Done false
19292 7331 Acked 8330 Write 4 New Done Done false
19293 7332 Acked 8331 Write 8 New Done Done false
19294 7333 Acked 8332 Read 10 New Done Done false
19295 7334 Acked 8333 Read 7 New Done Done false
19296 7335 Acked 8334 Write 2 New Done Done false
19297 7336 Acked 8335 Read 8 New Done Done false
19298 7337 Acked 8336 Write 2 New Done Done false
19299 7338 Acked 8337 Read 8 New Done Done false
19300 7339 Acked 8338 Write 3 New Done Done false
19301 7340 Acked 8339 Read 7 New Done Done false
19302 7341 Acked 8340 Read 1 New Done Done false
19303 7342 Acked 8341 Write 3 New Done Done false
19304 7343 Acked 8342 Read 3 New Done Done false
19305 7344 Acked 8343 Read 5 New Done Done false
19306 7345 Acked 8344 Write 7 New Done Done false
19307 7346 Acked 8345 Read 2 New Done Done false
19308 7347 Acked 8346 Read 10 New Done Done false
19309 7348 Acked 8347 Read 5 New Done Done false
19310 7349 Acked 8348 Read 6 New Done Done false
19311 7350 Acked 8349 Read 5 New Done Done false
19312 7351 Acked 8350 Read 8 New Done Done false
19313 7352 Acked 8351 Read 8 New Done Done false
19314 7353 Acked 8352 Write 5 New Done Done false
19315 7354 Acked 8353 Read 10 New Done Done false
19316 7355 Acked 8354 Write 9 New Done Done false
19317 7356 Acked 8355 Write 7 New Done Done false
19318 7357 Acked 8356 Read 2 New Done Done false
19319 7358 Acked 8357 Read 5 New Done Done false
19320 7359 Acked 8358 Write 6 New Done Done false
19321 7360 Acked 8359 Read 1 New Done Done false
19322 7361 Acked 8360 Flush 0 New Done Done false
19323 7362 Acked 8361 Write 10 New Done Done false
19324 7363 Acked 8362 Read 10 New Done Done false
19325 7364 Acked 8363 Read 1 New Done Done false
19326 7365 Acked 8364 Write 10 New Done Done false
19327 7366 Acked 8365 Write 6 New Done Done false
19328 7367 Acked 8366 Flush 0 New Done Done false
19329 7368 Acked 8367 Read 9 New Done Done false
19330 7369 Acked 8368 Write 3 New Done Done false
19331 7370 Acked 8369 Read 1 New Done Done false
19332 7371 Acked 8370 Read 6 New Done Done false
19333 7372 Acked 8371 Read 4 New Done Done false
19334 7373 Acked 8372 Flush 0 New Done Done false
19335 7374 Acked 8373 Write 7 New Done Done false
19336 7375 Acked 8374 Read 2 New Done Done false
19337 7376 Acked 8375 Read 2 New Done Done false
19338 7377 Acked 8376 Read 6 New Done Done false
19339 7378 Acked 8377 Write 6 New Done Done false
19340 7379 Acked 8378 Read 4 New Done Done false
19341 7380 Acked 8379 Read 2 New Done Done false
19342 7381 Acked 8380 Read 4 New Done Done false
19343 7382 Acked 8381 Write 7 New Done Done false
19344 7383 Acked 8382 Read 8 New Done Done false
19345 7384 Acked 8383 Read 9 New Done Done false
19346 7385 Acked 8384 Read 7 New Done Done false
19347 7386 Acked 8385 Read 3 New Done Done false
19348 7387 Acked 8386 Write 7 New Done Done false
19349 7388 Acked 8387 Write 8 New Done Done false
19350 7389 Acked 8388 Read 3 New Done Done false
19351 7390 Acked 8389 Read 5 New Done Done false
19352 7391 Acked 8390 Read 7 New Done Done false
19353 7392 Acked 8391 Read 8 New Done Done false
19354 7393 Acked 8392 Write 1 New Done Done false
19355 7394 Acked 8393 Read 5 New Done Done false
19356 7395 Acked 8394 Read 8 New Done Done false
19357 7396 Acked 8395 Read 2 New Done Done false
19358 7397 Acked 8396 Write 7 New Done Done false
19359 7398 Acked 8397 Write 3 New Done Done false
19360 7399 Acked 8398 Flush 0 New Done Done false
19361 7400 Acked 8399 Write 3 New Done Done false
19362 7401 Acked 8400 Write 8 New Done Done false
19363 7402 Acked 8401 Flush 0 New Done Done false
19364 7403 Acked 8402 Read 2 New Done Done false
19365 7404 Acked 8403 Write 2 New Done Done false
19366 7405 Acked 8404 Write 9 New Done Done false
19367 7406 Acked 8405 Read 3 New Done Done false
19368 7407 Acked 8406 Write 5 New Done Done false
19369 7408 Acked 8407 Flush 0 New Done Done false
19370 7409 Acked 8408 Write 1 New Done Done false
19371 7410 Acked 8409 Write 10 New Done Done false
19372 7411 Acked 8410 Read 9 New Done Done false
19373 7412 Acked 8411 Read 2 New Done Done false
19374 7413 Acked 8412 Read 7 New Done Done false
19375 7414 Acked 8413 Read 10 New Done Done false
19376 7415 Acked 8414 Read 5 New Done Done false
19377 7416 Acked 8415 Write 2 New Done Done false
19378 7417 Acked 8416 Write 1 New Done Done false
19379 7418 Acked 8417 Read 7 New Done Done false
19380 7419 Acked 8418 Read 8 New Done Done false
19381 7420 Acked 8419 Read 2 New Done Done false
19382 7421 Acked 8420 Write 6 New Done Done false
19383 7422 Acked 8421 Read 10 New Done Done false
19384 7423 Acked 8422 Write 2 New Done Done false
19385 7424 Acked 8423 Read 8 New Done Done false
19386 7425 Acked 8424 Write 4 New Done Done false
19387 7426 Acked 8425 Read 2 New Done Done false
19388 7427 Acked 8426 Read 6 New Done Done false
19389 7428 Acked 8427 Flush 0 New Done Done false
19390 7429 Acked 8428 Write 6 New Done Done false
19391 7430 Acked 8429 Write 2 New Done Done false
19392 7431 Acked 8430 Read 9 New Done Done false
19393 7432 Acked 8431 Write 3 New Done Done false
19394 7433 Acked 8432 Write 6 New Done Done false
19395 7434 Acked 8433 Read 9 New Done Done false
19396 7435 Acked 8434 Read 2 New Done Done false
19397 7436 Acked 8435 Write 5 New Done Done false
19398 7437 Acked 8436 Write 6 New Done Done false
19399 7438 Acked 8437 Write 3 New Done Done false
19400 7439 Acked 8438 Read 10 New Done Done false
19401 7440 Acked 8439 Write 8 New Done Done false
19402 7441 Acked 8440 Flush 0 New Done Done false
19403 7442 Acked 8441 Write 3 New Done Done false
19404 7443 Acked 8442 Read 10 New Done Done false
19405 7444 Acked 8443 Read 6 New Done Done false
19406 7445 Acked 8444 Write 3 New Done Done false
19407 7446 Acked 8445 Write 1 New Done Done false
19408 7447 Acked 8446 Write 1 New Done Done false
19409 7448 Acked 8447 Read 10 New Done Done false
19410 7449 Acked 8448 Read 1 New Done Done false
19411 7450 Acked 8449 Write 5 New Done Done false
19412 7451 Acked 8450 Read 5 New Done Done false
19413 7452 Acked 8451 Write 10 New Done Done false
19414 7453 Acked 8452 Read 6 New Done Done false
19415 7454 Acked 8453 Flush 0 New Done Done false
19416 7455 Acked 8454 Read 1 New Done Done false
19417 7456 Acked 8455 Read 2 New Done Done false
19418 7457 Acked 8456 Write 7 New Done Done false
19419 7458 Acked 8457 Read 9 New Done Done false
19420 7459 Acked 8458 Read 6 New Done Done false
19421 7460 Acked 8459 Flush 0 New Done Done false
19422 7461 Acked 8460 Flush 0 New Done Done false
19423 7462 Acked 8461 Read 9 New Done Done false
19424 7463 Acked 8462 Read 10 New Done Done false
19425 7464 Acked 8463 Read 2 New Done Done false
19426 7465 Acked 8464 Write 1 New Done Done false
19427 7466 Acked 8465 Flush 0 New Done Done false
19428 7467 Acked 8466 Write 9 New Done Done false
19429 7468 Acked 8467 Read 2 New Done Done false
19430 7469 Acked 8468 Write 1 New Done Done false
19431 7470 Acked 8469 Read 6 New Done Done false
19432 7471 Acked 8470 Read 10 New Done Done false
19433 7472 Acked 8471 Read 8 New Done Done false
19434 7473 Acked 8472 Read 4 New Done Done false
19435 7474 Acked 8473 Read 5 New Done Done false
19436 7475 Acked 8474 Write 5 New Done Done false
19437 7476 Acked 8475 Write 2 New Done Done false
19438 7477 Acked 8476 Write 2 New Done Done false
19439 7478 Acked 8477 Read 3 New Done Done false
19440 7479 Acked 8478 Write 2 New Done Done false
19441 7480 Acked 8479 Write 1 New Done Done false
19442 7481 Acked 8480 Write 2 New Done Done false
19443 7482 Acked 8481 Read 6 New Done Done false
19444 7483 Acked 8482 Read 1 New Done Done false
19445 7484 Acked 8483 Read 9 New Done Done false
19446 7485 Acked 8484 Read 10 New Done Done false
19447 7486 Acked 8485 Write 5 New Done Done false
19448 7487 Acked 8486 Read 9 New Done Done false
19449 7488 Acked 8487 Write 4 New Done Done false
19450 7489 Acked 8488 Write 9 New Done Done false
19451 7490 Acked 8489 Read 10 New Done Done false
19452 7491 Acked 8490 Read 6 New Done Done false
19453 7492 Acked 8491 Write 1 New Done Done false
19454 7493 Acked 8492 Write 1 New Done Done false
19455 7494 Acked 8493 Flush 0 New Done Done false
19456 7495 Acked 8494 Read 9 New Done Done false
19457 7496 Acked 8495 Read 7 New Done Done false
19458 7497 Acked 8496 Read 6 New Done Done false
19459 7498 Acked 8497 Read 5 New Done Done false
19460 7499 Acked 8498 Read 5 New Done Done false
19461 7500 Acked 8499 Read 6 New Done Done false
19462 7501 Acked 8500 Write 6 New Done Done false
19463 7502 Acked 8501 Write 1 New Done Done false
19464 7503 Acked 8502 Read 9 New Done Done false
19465 7504 Acked 8503 Read 5 New Done Done false
19466 7505 Acked 8504 Write 1 New Done Done false
19467 7506 Acked 8505 Flush 0 New Done Done false
19468 7507 Acked 8506 Read 4 New Done Done false
19469 7508 Acked 8507 Write 10 New Done Done false
19470 7509 Acked 8508 Write 1 New Done Done false
19471 7510 Acked 8509 Write 6 New Done Done false
19472 7511 Acked 8510 Flush 0 New Done Done false
19473 7512 Acked 8511 Read 7 New Done Done false
19474 7513 Acked 8512 Read 10 New Done Done false
19475 7514 Acked 8513 Read 1 New Done Done false
19476 7515 Acked 8514 Read 2 New Done Done false
19477 7516 Acked 8515 Write 10 New Done Done false
19478 7517 Acked 8516 Write 10 New Done Done false
19479 7518 Acked 8517 Write 4 New Done Done false
19480 7519 Acked 8518 Read 9 New Done Done false
19481 7520 Acked 8519 Read 4 New Done Done false
19482 7521 Acked 8520 Read 8 New Done Done false
19483 7522 Acked 8521 Read 9 New Done Done false
19484 7523 Acked 8522 Read 7 New Done Done false
19485 7524 Acked 8523 Write 9 New Done Done false
19486 7525 Acked 8524 Read 8 New Done Done false
19487 7526 Acked 8525 Flush 0 New Done Done false
19488 7527 Acked 8526 Write 2 New Done Done false
19489 7528 Acked 8527 Write 7 New Done Done false
19490 7529 Acked 8528 Read 1 New Done Done false
19491 7530 Acked 8529 Write 5 New Done Done false
19492 7531 Acked 8530 Read 10 New Done Done false
19493 7532 Acked 8531 Write 4 New Done Done false
19494 7533 Acked 8532 Read 9 New Done Done false
19495 7534 Acked 8533 Read 6 New Done Done false
19496 7535 Acked 8534 Read 5 New Done Done false
19497 7536 Acked 8535 Flush 0 New Done Done false
19498 7537 Acked 8536 Write 10 New Done Done false
19499 7538 Acked 8537 Write 7 New Done Done false
19500 7539 Acked 8538 Read 10 New Done Done false
19501 7540 Acked 8539 Write 2 New Done Done false
19502 7541 Acked 8540 Read 2 New Done Done false
19503 7542 Acked 8541 Write 7 New Done Done false
19504 7543 Acked 8542 Write 4 New Done Done false
19505 7544 Acked 8543 Read 10 New Done Done false
19506 7545 Acked 8544 Write 7 New Done Done false
19507 7546 Acked 8545 Flush 0 New Done Done false
19508 7547 Acked 8546 Read 10 New Done Done false
19509 7548 Acked 8547 Write 7 New Done Done false
19510 7549 Acked 8548 Read 8 New Done Done false
19511 7550 Acked 8549 Read 9 New Done Done false
19512 7551 Acked 8550 Read 5 New Done Done false
19513 7552 Acked 8551 Read 7 New Done Done false
19514 7553 Acked 8552 Write 3 New Done Done false
19515 7554 Acked 8553 Flush 0 New Done Done false
19516 7555 Acked 8554 Read 6 New Done Done false
19517 7556 Acked 8555 Write 9 New Done Done false
19518 7557 Acked 8556 Read 8 New Done Done false
19519 7558 Acked 8557 Flush 0 New Done Done false
19520 7559 Acked 8558 Read 6 New Done Done false
19521 7560 Acked 8559 Read 10 New Done Done false
19522 7561 Acked 8560 Read 4 New Done Done false
19523 7562 Acked 8561 Read 6 New Done Done false
19524 7563 Acked 8562 Read 10 New Done Done false
19525 7564 Acked 8563 Read 2 New Done Done false
19526 7565 Acked 8564 Read 9 New Done Done false
19527 7566 Acked 8565 Read 4 New Done Done false
19528 7567 Acked 8566 Read 7 New Done Done false
19529 STATES DS:0 DS:1 DS:2 TOTAL
19530 New 302 0 0 302
19531 Sent 0 0 0 0
19532 Done 0 302 302 604
19533 Skipped 0 0 0 0
19534 Error 0 0 0 0
19535 Last Flush: 8264 8557 8557
19536 Downstairs last five completed: 8264 8263 8262 8261 8260 8259
19537 Upstairs last five completed: 7567 7566 7565 7564 7563 7562
19538 CLIENT: Up:0 ds:302 act:2
19539 [8810] got start action
19540 [8810] I am going to restart
19541 Starting downstairs at port 8810
19542 Make output file at "/tmp/dsc/downstairs-8810.txt"
19543 [8810][0] reports Starting
19544 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1201)
19545 [8810][0] reports Running
195462023-10-13T18:13:21.559ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 0
195472023-10-13T18:13:21.559ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state Offline
195482023-10-13T18:13:21.559ZINFOcrucible: [0] client re-new 303 jobs since flush 8264 = downstairs
195492023-10-13T18:13:21.560ZINFOcrucible: [0] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
195502023-10-13T18:13:21.560ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
195512023-10-13T18:13:21.561ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
195522023-10-13T18:13:21.561ZINFOcrucible: Returning client:0 UUID:12345678-0000-0000-0000-000000008810 matches
195532023-10-13T18:13:21.561ZINFOcrucible: [0] send last flush ID to this DS: 8264
195542023-10-13T18:13:21.561ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
195552023-10-13T18:13:21.561ZINFOcrucible: [0] Replied this last flush ID: 8264
195562023-10-13T18:13:21.561ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Offline Active Active ds_transition to Replay
195572023-10-13T18:13:21.561ZINFOcrucible: [0] Transition from Offline to Replay
195582023-10-13T18:13:21.561ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
195592023-10-13T18:13:21.561ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Replay Active Active ds_transition to Active
195602023-10-13T18:13:21.561ZINFOcrucible: [0] Transition from Replay to Active
195612023-10-13T18:13:21.561ZINFOcrucible: [0] Starts cmd_loop
195622023-10-13T18:13:21.561ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
195632023-10-13T18:13:21.561ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
195642023-10-13T18:13:21.561ZINFOcrucible: Set check for repair
195652023-10-13T18:13:22.563ZINFOcrucible: Checking if live repair is needed
195662023-10-13T18:13:22.563ZINFOcrucible: No Live Repair required at this time
195672023-10-13T18:13:22.563ZWARNcrucible: [0] flow control end
19568 ----------------------------------------------------------------
19569 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19570 Downstairs last five completed: 8567 8566 8565 8564 8563 8562
19571 Upstairs last five completed: 7568 7567 7566 7565 7564 7563
19572 CLIENT: Up:0 ds:0 act:3
19573 Replay: All jobs finished, all DS active.
19574 Main task has work to do, go find it
19575 got dsc Stop(1)
19576 stop 1
19577 Oct 13 18:13:23.524 INFO request completed, latency_us: 525, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 83d2c062-99cf-414e-8382-97550e19115a, remote_addr: 127.0.0.1:64571, local_addr[8820] Got stop action so:false kr:false
19578 : 127.0.0.1:9998
19579 Oct 13 18:13:23.525 INFO request completed, latency_us: 180, response_code: 200, uri: /state/cid/1, method: GET, req_id: 64a16066-6c7b-43f1-add6-3ec90111c86b, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
195802023-10-13T18:13:23.526ZWARNcrucible: [1] downstairs disconnected
195812023-10-13T18:13:23.526ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
195822023-10-13T18:13:23.526ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8820 closed looper = 1
195832023-10-13T18:13:23.526ZWARNcrucible: [1] pm_task rx.recv() is None
195842023-10-13T18:13:23.526ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
195852023-10-13T18:13:23.526ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
195862023-10-13T18:13:23.526ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
19587 [8820] Exited with: ExitStatus(unix_wait_status(9))
19588 [8820][1] reports Exit
19589 Oct 13 18:13:27.528 INFO request completed, latency_us: 259, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2e05b660-939a-4df7-94f4-9ec4ad19453f, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
19590 001/300 Write block 2743 len 32768 data: 3 3 4 5 5 5 5 4
19591 002/300 Read block 1949 len 28672
19592 003/300 Read block 3234 len 24576
19593 004/300 Read block 4581 len 24576
19594 005/300 Read block 3906 len 36864
19595 006/300 Read block 1808 len 4096
19596 007/300 Read block 2897 len 36864
19597 008/300 Read block 3038 len 8192
19598 009/300 Write block 1078 len 20480 data: 6 6 5 6 5
19599 010/300 Read block 281 len 16384
19600 011/300 Write block 3338 len 12288 data: 7 6 4
19601 012/300 Write block 4533 len 20480 data: 5 5 5 5 4
19602 013/300 Write block 2106 len 32768 data: 4 6 6 5 6 6 5 5
19603 014/300 Read block 3304 len 20480
19604 015/300 Write block 1 len 32768 data: 2 2 2 2 3 6 5 5
19605 016/300 Write block 2609 len 20480 data: 3 4 3 3 5
19606 017/300 Flush
19607 018/300 Write block 3918 len 8192 data: 7 3
19608 019/300 Write block 4704 len 16384 data: 6 4 3 2
19609 020/300 Read block 4035 len 32768
19610 021/300 Write block 985 len 36864 data: 5 4 4 4 4 4 4 5 6
19611 022/300 Read block 4964 len 4096
19612 023/300 Read block 16 len 28672
19613 024/300 Write block 4531 len 20480 data: 5 5 6 6 6
19614 025/300 Read block 1308 len 16384
19615 026/300 Read block 4221 len 4096
19616 027/300 Flush
19617 028/300 Write block 182 len 24576 data: 4 4 4 6 7 7
19618 029/300 Read block 3334 len 8192
19619 030/300 Read block 1926 len 32768
19620 031/300 Flush
19621 032/300 Write block 3020 len 36864 data: 9 9 8 7 7 7 6 5 5
19622 033/300 Read block 2842 len 40960
19623 034/300 Read block 3997 len 40960
19624 035/300 Write block 4900 len 12288 data: 4 8 6
19625 036/300 Write block 2343 len 4096 data: 5
19626 037/300 Flush
19627 038/300 Write block 1847 len 4096 data: 3
19628 039/300 Flush
19629 040/300 Write block 1482 len 24576 data: 5 5 5 5 6 8
19630 041/300 Write block 2503 len 40960 data: 5 5 5 5 6 9 9 7 7 7
19631 042/300 Read block 3381 len 40960
19632 043/300 Read block 470 len 20480
19633 044/300 Read block 1502 len 28672
19634 045/300 Write block 3833 len 40960 data: 6 5 6 6 6 6 6 5 6 6
19635 046/300 Read block 1320 len 4096
19636 047/300 Write block 3176 len 36864 data: 6 6 8 8 6 6 6 6 5
19637 048/300 Write block 1444 len 28672 data: 5 6 6 6 9 8 5
19638 049/300 Write block 3158 len 20480 data: 6 7 8 7 6
19639 050/300 Write block 1372 len 4096 data: 7
19640 051/300 Read block 868 len 4096
19641 052/300 Read block 1739 len 4096
19642 053/300 Write block 2788 len 36864 data: 6 5 5 6 6 6 8 7 7
19643 054/300 Flush
19644 055/300 Read block 3205 len 4096
19645 056/300 Read block 1394 len 28672
19646 057/300 Read block 1230 len 4096
19647 058/300 Write block 4117 len 32768 data: 5 6 6 6 5 5 4 3
19648 059/300 Flush
19649 060/300 Read block 1177 len 28672
19650 061/300 Write block 4308 len 8192 data: 10 7
19651 062/300 Flush
19652 063/300 Write block 4397 len 24576 data: 7 11 9 9 8 6
19653 064/300 Read block 725 len 40960
19654 065/300 Flush
19655 066/300 Write block 2504 len 32768 data: 6 6 6 7 10 10 8 8
19656 067/300 Read block 951 len 12288
19657 068/300 Read block 2789 len 40960
19658 069/300 Write block 3236 len 40960 data: 8 6 6 6 7 6 3 3 3 4
19659 070/300 Read block 534 len 8192
19660 071/300 Read block 4809 len 40960
19661 072/300 Write block 2656 len 28672 data: 3 2 3 5 5 4 5
19662 073/300 Write block 4652 len 16384 data: 3 3 3 4
19663 074/300 Read block 4497 len 24576
19664 075/300 Write block 3404 len 40960 data: 4 5 6 6 6 4 4 5 5 4
19665 076/300 Write block 3480 len 12288 data: 5 5 4
19666 077/300 Read block 580 len 32768
19667 078/300 Write block 572 len 20480 data: 3 4 4 4 4
19668 079/300 Read block 4243 len 32768
19669 080/300 Read block 439 len 24576
19670 081/300 Write block 2194 len 8192 data: 9 10
19671 082/300 Read block 2047 len 36864
19672 083/300 Flush
19673 084/300 Read block 4679 len 4096
19674 085/300 Write block 577 len 40960 data: 4 3 3 3 2 2 4 4 5 5
19675 086/300 Write block 1846 len 36864 data: 2 4 4 4 4 4 5 7 7
19676 087/300 Read block 1571 len 28672
19677 088/300 Read block 3189 len 16384
19678 089/300 Read block 92 len 4096
19679 090/300 Write block 2902 len 16384 data: 7 7 6 3
19680 091/300 Read block 2181 len 32768
19681 092/300 Read block 603 len 16384
19682 093/300 Read block 4237 len 24576
19683 094/300 Flush
19684 095/300 Read block 1428 len 36864
19685 096/300 Write block 3573 len 20480 data: 8 7 6 3 4
19686 097/300 Flush
19687 098/300 Read block 3265 len 40960
19688 099/300 Write block 2048 len 20480 data: 2 2 3 3 3
19689 100/300 Write block 2568 len 8192 data: 7 7
19690 101/300 Write block 1851 len 36864 data: 5 6 8 8 7 7 7 7 5
19691 102/300 Read block 1362 len 40960
19692 103/300 Write block 3574 len 20480 data: 8 7 4 5 3
19693 104/300 Flush
19694 105/300 Flush
19695 106/300 Write block 3960 len 16384 data: 6 4 3 4
19696 107/300 Read block 3687 len 32768
19697 108/300 Flush
19698 109/300 Read block 4801 len 36864
19699 110/300 Write block 85 len 28672 data: 4 5 6 4 4 5 5
19700 111/300 Write block 4562 len 32768 data: 2 2 2 3 3 4 4 5
19701 112/300 Write block 1025 len 28672 data: 4 5 3 6 6 8 6
19702 113/300 Write block 3652 len 32768 data: 5 5 6 7 7 6 5 4
19703 114/300 Write block 3797 len 40960 data: 4 4 4 4 3 5 5 6 7 6
19704 115/300 Write block 3208 len 36864 data: 7 5 6 6 5 5 5 4 6
19705 116/300 Read block 4065 len 4096
19706 117/300 Write block 438 len 28672 data: 6 6 5 6 6 8 6
19707 118/300 Write block 1188 len 24576 data: 6 6 7 6 6 6
19708 119/300 Read block 4180 len 24576
19709 120/300 Read block 1640 len 8192
19710 121/300 Write block 3600 len 36864 data: 4 4 3 3 3 3 4 3 4
19711 122/300 Write block 4423 len 8192 data: 5 6
19712 123/300 Write block 4561 len 20480 data: 3 3 3 3 4
19713 124/300 Read block 2286 len 40960
19714 125/300 Write block 624 len 16384 data: 3 4 4 4
19715 126/300 Read block 2003 len 36864
19716 127/300 Read block 1235 len 20480
19717 128/300 Read block 4385 len 28672
19718 129/300 Write block 4014 len 28672 data: 7 8 6 5 5 5 3
19719 130/300 Read block 717 len 36864
19720 131/300 Read block 4103 len 8192
19721 132/300 Write block 3633 len 28672 data: 4 3 3 3 3 3 5
19722 133/300 Read block 760 len 12288
19723 134/300 Read block 3955 len 28672
19724 135/300 Read block 4180 len 8192
19725 136/300 Read block 2405 len 32768
19726 137/300 Read block 1352 len 40960
19727 138/300 Read block 2672 len 36864
19728 139/300 Read block 4309 len 12288
19729 140/300 Write block 949 len 40960 data: 3 3 3 4 7 6 6 6 7 8
19730 141/300 Read block 1879 len 12288
19731 142/300 Read block 446 len 4096
19732 143/300 Read block 1767 len 4096
19733 144/300 Write block 1178 len 24576 data: 4 4 3 4 4 6
19734 145/300 Write block 418 len 40960 data: 5 4 4 4 3 4 5 5 4 3
19735 146/300 Read block 4639 len 12288
19736 147/300 Flush
19737 148/300 Read block 516 len 16384
19738 149/300 Read block 2851 len 24576
19739 150/300 Read block 828 len 40960
19740 151/300 Read block 1080 len 28672
19741 152/300 Write block 4447 len 8192 data: 7 8
19742 153/300 Flush
19743 154/300 Read block 1797 len 8192
19744 155/300 Flush
19745 156/300 Read block 1880 len 16384
19746 157/300 Read block 4910 len 12288
19747 158/300 Read block 4087 len 24576
19748 159/300 Read block 4903 len 4096
19749 160/300 Write block 1810 len 20480 data: 7 7 9 8 5
19750 161/300 Read block 4659 len 12288
19751 162/300 Write block 835 len 24576 data: 5 6 7 7 7 8
19752 163/300 Write block 1765 len 40960 data: 7 7 6 5 4 4 4 3 3 3
19753 164/300 Write block 3739 len 24576 data: 7 6 7 9 9 8
19754 165/300 Read block 2140 len 32768
19755 166/300 Write block 116 len 8192 data: 4 3
19756 167/300 Write block 4360 len 32768 data: 3 3 3 4 5 6 6 5
19757 168/300 Write block 2318 len 24576 data: 6 4 3 4 5 4
19758 169/300 Write block 2269 len 16384 data: 4 4 3 3
19759 170/300 Read block 604 len 20480
19760 171/300 Read block 2349 len 16384
19761 172/300 Read block 24 len 8192
19762 173/300 Read block 4945 len 20480
19763 174/300 Write block 2528 len 24576 data: 9 9 8 9 9 7
19764 175/300 Read block 2821 len 32768
19765 176/300 Write block 4789 len 20480 data: 9 6 7 7 7
19766 177/300 Read block 594 len 28672
19767 178/300 Write block 721 len 12288 data: 5 6 5
19768 179/300 Write block 238 len 20480 data: 5 5 5 4 4
19769 180/300 Read block 1968 len 12288
19770 181/300 Read block 1654 len 12288
19771 182/300 Read block 2802 len 12288
19772 183/300 Write block 413 len 36864 data: 4 3 4 4 6 6 5 5 5
19773 184/300 Read block 3402 len 40960
19774 185/300 Flush
19775 186/300 Read block 1087 len 4096
19776 187/300 Write block 2318 len 4096 data: 7
19777 188/300 Write block 306 len 40960 data: 6 7 7 7 6 8 7 8 9 6
19778 189/300 Write block 4764 len 24576 data: 5 4 4 4 5 4
19779 190/300 Write block 3419 len 36864 data: 4 3 3 3 4 5 7 6 5
19780 191/300 Read block 2262 len 20480
19781 192/300 Write block 1447 len 12288 data: 7 10 9
19782 193/300 Write block 1181 len 32768 data: 5 5 7 6 7 8 7 7
19783 194/300 Read block 2102 len 40960
19784 195/300 Write block 3596 len 20480 data: 7 6 5 3 5
19785 196/300 Flush
19786 197/300 Read block 3894 len 16384
19787 198/300 Read block 4543 len 20480
19788 199/300 Read block 1762 len 24576
19789 200/300 Write block 2052 len 24576 data: 4 3 3 3 4 5
19790 201/300 Read block 4978 len 20480
19791 202/300 Read block 2439 len 12288
19792 203/300 Read block 2846 len 36864
19793 204/300 Read block 3743 len 8192
19794 205/300 Write block 3637 len 32768 data: 4 4 6 5 4 4 3 4
19795 206/300 Read block 4092 len 36864
19796 207/300 Write block 1139 len 12288 data: 4 4 4
19797 208/300 Write block 1367 len 20480 data: 7 8 6 8 8
19798 209/300 Write block 4912 len 28672 data: 4 3 5 5 6 7 6
19799 210/300 Write block 4916 len 36864 data: 7 8 7 6 6 6 7 5 4
19800 211/300 Read block 2602 len 28672
19801 212/300 Read block 4311 len 16384
19802 213/300 Write block 2076 len 24576 data: 5 3 3 3 2 3
19803 214/300 Flush
19804 215/300 Write block 2937 len 24576 data: 6 8 7 7 5 6
19805 216/300 Flush
19806 217/300 Read block 2972 len 32768
19807 218/300 Read block 2220 len 36864
19808 219/300 Write block 2565 len 4096 data: 8
19809 220/300 Read block 1156 len 4096
19810 221/300 Read block 3089 len 24576
19811 222/300 Write block 885 len 28672 data: 9 9 9 10 8 7 6
19812 223/300 Read block 1282 len 40960
19813 224/300 Read block 3607 len 40960
19814 225/300 Read block 4556 len 16384
19815 226/300 Read block 250 len 16384
19816 227/300 Write block 173 len 24576 data: 6 6 7 8 7 7
19817 228/300 Read block 4171 len 8192
19818 229/300 Read block 3815 len 28672
19819 230/300 Write block 3146 len 4096 data: 12
19820 231/300 Read block 3461 len 32768
19821 232/300 Write block 3820 len 4096 data: 5
19822 233/300 Read block 3114 len 28672
19823 234/300 Write block 4778 len 32768 data: 7 8 8 8 6 5 6 4
19824 235/300 Read block 1538 len 4096
19825 236/300 Write block 3602 len 40960 data: 4 4 4 4 5 4 5 4 4 5
19826 237/300 Read block 1971 len 4096
19827 238/300 Write block 157 len 32768 data: 4 3 3 4 5 7 6 6
19828 239/300 Read block 2805 len 28672
19829 240/300 Read block 832 len 12288
19830 241/300 Write block 2808 len 16384 data: 6 5 6 5
19831 242/300 Write block 1380 len 32768 data: 7 6 5 5 4 3 5 7
19832 243/300 Read block 3371 len 28672
19833 244/300 Write block 3524 len 40960 data: 5 5 6 4 3 3 4 6 6 8
19834 245/300 Read block 789 len 16384
19835 246/300 Write block 1652 len 20480 data: 8 6 5 5 6
19836 247/300 Read block 3270 len 16384
19837 248/300 Read block 3572 len 12288
19838 249/300 Write block 3710 len 12288 data: 14 13 11
19839 250/300 Read block 1435 len 8192
19840 251/300 Write block 2644 len 16384 data: 5 3 2 2
19841 252/300 Read block 4511 len 4096
19842 253/300 Read block 3487 len 36864
19843 254/300 Read block 1373 len 40960
19844 255/300 Read block 16 len 32768
19845 256/300 Read block 4058 len 32768
19846 257/300 Read block 439 len 8192
19847 258/300 Read block 1251 len 36864
19848 259/300 Write block 2196 len 32768 data: 9 7 7 6 6 4 6 4
19849 260/300 Write block 1872 len 24576 data: 7 7 7 8 10 8
19850 261/300 Read block 1434 len 24576
19851 262/300 Read block 4624 len 32768
19852 263/300 Read block 3612 len 32768
19853 264/300 Read block 4642 len 12288
19854 265/300 Write block 4451 len 16384 data: 8 7 7 6
19855 266/300 Write block 836 len 12288 data: 7 8 8
19856 267/300 Read block 3511 len 32768
19857 268/300 Write block 1347 len 32768 data: 5 5 5 4 4 4 5 6
19858 269/300 Read block 2288 len 8192
19859 270/300 Write block 4727 len 8192 data: 7 7
19860 271/300 Read block 1418 len 40960
19861 272/300 Read block 2002 len 12288
19862 273/300 Flush
19863 274/300 Read block 3624 len 32768
19864 275/300 Read block 390 len 28672
19865 276/300 Flush
19866 277/300 Read block 3458 len 12288
19867 278/300 Flush
19868 279/300 Write block 881 len 16384 data: 6 5 5 6
19869 280/300 Read block 3963 len 36864
19870 281/300 Read block 243 len 4096
19871 282/300 Read block 3891 len 36864
19872 283/300 Read block 2617 len 12288
19873 284/300 Read block 2917 len 8192
19874 285/300 Write block 87 len 40960 data: 7 5 5 6 6 4 5 6 6 5
19875 286/300 Write block 4290 len 40960 data: 5 5 5 3 3 3 4 6 6 5
19876 287/300 Read block 4379 len 20480
19877 288/300 Read block 2873 len 16384
19878 289/300 Read block 4908 len 36864
19879 290/300 Write block 2790 len 12288 data: 6 7 7
19880 291/300 Write block 1698 len 32768 data: 5 4 3 3 4 3 3 3
19881 292/300 Read block 1464 len 32768
19882 293/300 Write block 4775 len 40960 data: 5 5 7 8 9 9 9 7 6 7
19883 294/300 Write block 509 len 36864 data: 5 4 4 5 5 5 5 3 3
19884 295/300 Flush
19885 296/300 Read block 3911 len 4096
19886 297/300 Write block 859 len 4096 data: 3
19887 298/300 Read block 1648 len 4096
19888 299/300 Write block 139 len 40960 data: 6 8 8 6 7 6 5 5 6 5
19889 300/300 Read block 4128 len 12288
19890 Main task has work to do, go find it
19891 got dsc Start(1)
19892 start 1
19893 Oct 13 18:13:28.507 INFO request completed, latency_us: 298, response_code: 204, uri: /start/cid/1, method: POST, req_id: ebb49668-3c35-41b2-8926-2a4e13b62997, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
19894 Replay: started 1, returned:Ok(())
19895 ----------------------------------------------------------------
19896 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19897 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
19898 7569 Acked 8568 Write 8 Done New Done false
19899 7570 Acked 8569 Read 7 Done New Done false
19900 7571 Acked 8570 Read 6 Done New Done false
19901 7572 Acked 8571 Read 6 Done New Done false
19902 7573 Acked 8572 Read 9 Done New Done false
19903 7574 Acked 8573 Read 1 Done New Done false
19904 7575 Acked 8574 Read 9 Done New Done false
19905 7576 Acked 8575 Read 2 Done New Done false
19906 7577 Acked 8576 Write 5 Done New Done false
19907 7578 Acked 8577 Read 4 Done New Done false
19908 7579 Acked 8578 Write 3 Done New Done false
19909 7580 Acked 8579 Write 5 Done New Done false
19910 7581 Acked 8580 Write 8 Done New Done false
19911 7582 Acked 8581 Read 5 Done New Done false
19912 7583 Acked 8582 Write 8 Done New Done false
19913 7584 Acked 8583 Write 5 Done New Done false
19914 7585 Acked 8584 Flush 0 Done New Done false
19915 7586 Acked 8585 Write 2 Done New Done false
19916 7587 Acked 8586 Write 4 Done New Done false
19917 7588 Acked 8587 Read 8 Done New Done false
19918 7589 Acked 8588 Write 9 Done New Done false
19919 7590 Acked 8589 Read 1 Done New Done false
19920 7591 Acked 8590 Read 7 Done New Done false
19921 7592 Acked 8591 Write 5 Done New Done false
19922 7593 Acked 8592 Read 4 Done New Done false
19923 7594 Acked 8593 Read 1 Done New Done false
19924 7595 Acked 8594 Flush 0 Done New Done false
19925 7596 Acked 8595 Write 6 Done New Done false
19926 7597 Acked 8596 Read 2 Done New Done false
19927 7598 Acked 8597 Read 8 Done New Done false
19928 7599 Acked 8598 Flush 0 Done New Done false
19929 7600 Acked 8599 Write 9 Done New Done false
19930 7601 Acked 8600 Read 10 Done New Done false
19931 7602 Acked 8601 Read 10 Done New Done false
19932 7603 Acked 8602 Write 3 Done New Done false
19933 7604 Acked 8603 Write 1 Done New Done false
19934 7605 Acked 8604 Flush 0 Done New Done false
19935 7606 Acked 8605 Write 1 Done New Done false
19936 7607 Acked 8606 Flush 0 Done New Done false
19937 7608 Acked 8607 Write 6 Done New Done false
19938 7609 Acked 8608 Write 10 Done New Done false
19939 7610 Acked 8609 Read 10 Done New Done false
19940 7611 Acked 8610 Read 5 Done New Done false
19941 7612 Acked 8611 Read 7 Done New Done false
19942 7613 Acked 8612 Write 10 Done New Done false
19943 7614 Acked 8613 Read 1 Done New Done false
19944 7615 Acked 8614 Write 9 Done New Done false
19945 7616 Acked 8615 Write 7 Done New Done false
19946 7617 Acked 8616 Write 5 Done New Done false
19947 7618 Acked 8617 Write 1 Done New Done false
19948 7619 Acked 8618 Read 1 Done New Done false
19949 7620 Acked 8619 Read 1 Done New Done false
19950 7621 Acked 8620 Write 9 Done New Done false
19951 7622 Acked 8621 Flush 0 Done New Done false
19952 7623 Acked 8622 Read 1 Done New Done false
19953 7624 Acked 8623 Read 7 Done New Done false
19954 7625 Acked 8624 Read 1 Done New Done false
19955 7626 Acked 8625 Write 8 Done New Done false
19956 7627 Acked 8626 Flush 0 Done New Done false
19957 7628 Acked 8627 Read 7 Done New Done false
19958 7629 Acked 8628 Write 2 Done New Done false
19959 7630 Acked 8629 Flush 0 Done New Done false
19960 7631 Acked 8630 Write 6 Done New Done false
19961 7632 Acked 8631 Read 10 Done New Done false
19962 7633 Acked 8632 Flush 0 Done New Done false
19963 7634 Acked 8633 Write 8 Done New Done false
19964 7635 Acked 8634 Read 3 Done New Done false
19965 7636 Acked 8635 Read 10 Done New Done false
19966 7637 Acked 8636 Write 10 Done New Done false
19967 7638 Acked 8637 Read 2 Done New Done false
19968 7639 Acked 8638 Read 10 Done New Done false
19969 7640 Acked 8639 Write 7 Done New Done false
19970 7641 Acked 8640 Write 4 Done New Done false
19971 7642 Acked 8641 Read 6 Done New Done false
19972 7643 Acked 8642 Write 10 Done New Done false
19973 7644 Acked 8643 Write 3 Done New Done false
19974 7645 Acked 8644 Read 8 Done New Done false
19975 7646 Acked 8645 Write 5 Done New Done false
19976 7647 Acked 8646 Read 8 Done New Done false
19977 7648 Acked 8647 Read 6 Done New Done false
19978 7649 Acked 8648 Write 2 Done New Done false
19979 7650 Acked 8649 Read 9 Done New Done false
19980 7651 Acked 8650 Flush 0 Done New Done false
19981 7652 Acked 8651 Read 1 Done New Done false
19982 7653 Acked 8652 Write 10 Done New Done false
19983 7654 Acked 8653 Write 9 Done New Done false
19984 7655 Acked 8654 Read 7 Done New Done false
19985 7656 Acked 8655 Read 4 Done New Done false
19986 7657 Acked 8656 Read 1 Done New Done false
19987 7658 Acked 8657 Write 4 Done New Done false
19988 7659 Acked 8658 Read 8 Done New Done false
19989 7660 Acked 8659 Read 4 Done New Done false
19990 7661 Acked 8660 Read 6 Done New Done false
19991 7662 Acked 8661 Flush 0 Done New Done false
19992 7663 Acked 8662 Read 9 Done New Done false
19993 7664 Acked 8663 Write 5 Done New Done false
19994 7665 Acked 8664 Flush 0 Done New Done false
19995 7666 Acked 8665 Read 10 Done New Done false
19996 7667 Acked 8666 Write 5 Done New Done false
19997 7668 Acked 8667 Write 2 Done New Done false
19998 7669 Acked 8668 Write 9 Done New Done false
19999 7670 Acked 8669 Read 10 Done New Done false
20000 7671 Acked 8670 Write 5 Done New Done false
20001 7672 Acked 8671 Flush 0 Done New Done false
20002 7673 Acked 8672 Flush 0 Done New Done false
20003 7674 Acked 8673 Write 4 Done New Done false
20004 7675 Acked 8674 Read 8 Done New Done false
20005 7676 Acked 8675 Flush 0 Done New Done false
20006 7677 Acked 8676 Read 9 Done New Done false
20007 7678 Acked 8677 Write 7 Done New Done false
20008 7679 Acked 8678 Write 8 Done New Done false
20009 7680 Acked 8679 Write 7 Done New Done false
20010 7681 Acked 8680 Write 8 Done New Done false
20011 7682 Acked 8681 Write 10 Done New Done false
20012 7683 Acked 8682 Write 9 Done New Done false
20013 7684 Acked 8683 Read 1 Done New Done false
20014 7685 Acked 8684 Write 7 Done New Done false
20015 7686 Acked 8685 Write 6 Done New Done false
20016 7687 Acked 8686 Read 6 Done New Done false
20017 7688 Acked 8687 Read 2 Done New Done false
20018 7689 Acked 8688 Write 9 Done New Done false
20019 7690 Acked 8689 Write 2 Done New Done false
20020 7691 Acked 8690 Write 5 Done New Done false
20021 7692 Acked 8691 Read 10 Done New Done false
20022 7693 Acked 8692 Write 4 Done New Done false
20023 7694 Acked 8693 Read 9 Done New Done false
20024 7695 Acked 8694 Read 5 Done New Done false
20025 7696 Acked 8695 Read 7 Done New Done false
20026 7697 Acked 8696 Write 7 Done New Done false
20027 7698 Acked 8697 Read 9 Done New Done false
20028 7699 Acked 8698 Read 2 Done New Done false
20029 7700 Acked 8699 Write 7 Done New Done false
20030 7701 Acked 8700 Read 3 Done New Done false
20031 7702 Acked 8701 Read 7 Done New Done false
20032 7703 Acked 8702 Read 2 Done New Done false
20033 7704 Acked 8703 Read 8 Done New Done false
20034 7705 Acked 8704 Read 10 Done New Done false
20035 7706 Acked 8705 Read 9 Done New Done false
20036 7707 Acked 8706 Read 3 Done New Done false
20037 7708 Acked 8707 Write 10 Done New Done false
20038 7709 Acked 8708 Read 3 Done New Done false
20039 7710 Acked 8709 Read 1 Done New Done false
20040 7711 Acked 8710 Read 1 Done New Done false
20041 7712 Acked 8711 Write 6 Done New Done false
20042 7713 Acked 8712 Write 10 Done New Done false
20043 7714 Acked 8713 Read 3 Done New Done false
20044 7715 Acked 8714 Flush 0 Done New Done false
20045 7716 Acked 8715 Read 4 Done New Done false
20046 7717 Acked 8716 Read 6 Done New Done false
20047 7718 Acked 8717 Read 10 Done New Done false
20048 7719 Acked 8718 Read 7 Done New Done false
20049 7720 Acked 8719 Write 2 Done New Done false
20050 7721 Acked 8720 Flush 0 Done New Done false
20051 7722 Acked 8721 Read 2 Done New Done false
20052 7723 Acked 8722 Flush 0 Done New Done false
20053 7724 Acked 8723 Read 4 Done New Done false
20054 7725 Acked 8724 Read 3 Done New Done false
20055 7726 Acked 8725 Read 6 Done New Done false
20056 7727 Acked 8726 Read 1 Done New Done false
20057 7728 Acked 8727 Write 5 Done New Done false
20058 7729 Acked 8728 Read 3 Done New Done false
20059 7730 Acked 8729 Write 6 Done New Done false
20060 7731 Acked 8730 Write 10 Done New Done false
20061 7732 Acked 8731 Write 6 Done New Done false
20062 7733 Acked 8732 Read 8 Done New Done false
20063 7734 Acked 8733 Write 2 Done New Done false
20064 7735 Acked 8734 Write 8 Done New Done false
20065 7736 Acked 8735 Write 6 Done New Done false
20066 7737 Acked 8736 Write 4 Done New Done false
20067 7738 Acked 8737 Read 5 Done New Done false
20068 7739 Acked 8738 Read 4 Done New Done false
20069 7740 Acked 8739 Read 2 Done New Done false
20070 7741 Acked 8740 Read 5 Done New Done false
20071 7742 Acked 8741 Write 6 Done New Done false
20072 7743 Acked 8742 Read 8 Done New Done false
20073 7744 Acked 8743 Write 5 Done New Done false
20074 7745 Acked 8744 Read 7 Done New Done false
20075 7746 Acked 8745 Write 3 Done New Done false
20076 7747 Acked 8746 Write 5 Done New Done false
20077 7748 Acked 8747 Read 3 Done New Done false
20078 7749 Acked 8748 Flush 0 Done New Done false
20079 7750 Acked 8749 Read 3 Done New Done false
20080 7751 Acked 8750 Read 3 Done New Done false
20081 7752 Acked 8751 Write 9 Done New Done false
20082 7753 Acked 8752 Read 10 Done New Done false
20083 7754 Acked 8753 Flush 0 Done New Done false
20084 7755 Acked 8754 Read 1 Done New Done false
20085 7756 Acked 8755 Write 1 Done New Done false
20086 7757 Acked 8756 Write 10 Done New Done false
20087 7758 Acked 8757 Write 6 Done New Done false
20088 7759 Acked 8758 Write 9 Done New Done false
20089 7760 Acked 8759 Read 5 Done New Done false
20090 7761 Acked 8760 Write 3 Done New Done false
20091 7762 Acked 8761 Write 8 Done New Done false
20092 7763 Acked 8762 Read 10 Done New Done false
20093 7764 Acked 8763 Write 5 Done New Done false
20094 7765 Acked 8764 Flush 0 Done New Done false
20095 7766 Acked 8765 Read 4 Done New Done false
20096 7767 Acked 8766 Read 5 Done New Done false
20097 7768 Acked 8767 Read 6 Done New Done false
20098 7769 Acked 8768 Write 6 Done New Done false
20099 7770 Acked 8769 Read 5 Done New Done false
20100 7771 Acked 8770 Read 3 Done New Done false
20101 7772 Acked 8771 Read 9 Done New Done false
20102 7773 Acked 8772 Read 2 Done New Done false
20103 7774 Acked 8773 Write 8 Done New Done false
20104 7775 Acked 8774 Read 9 Done New Done false
20105 7776 Acked 8775 Write 3 Done New Done false
20106 7777 Acked 8776 Write 5 Done New Done false
20107 7778 Acked 8777 Write 7 Done New Done false
20108 7779 Acked 8778 Write 9 Done New Done false
20109 7780 Acked 8779 Read 7 Done New Done false
20110 7781 Acked 8780 Read 4 Done New Done false
20111 7782 Acked 8781 Write 6 Done New Done false
20112 7783 Acked 8782 Flush 0 Done New Done false
20113 7784 Acked 8783 Write 6 Done New Done false
20114 7785 Acked 8784 Flush 0 Done New Done false
20115 7786 Acked 8785 Read 8 Done New Done false
20116 7787 Acked 8786 Read 9 Done New Done false
20117 7788 Acked 8787 Write 1 Done New Done false
20118 7789 Acked 8788 Read 1 Done New Done false
20119 7790 Acked 8789 Read 6 Done New Done false
20120 7791 Acked 8790 Write 7 Done New Done false
20121 7792 Acked 8791 Read 10 Done New Done false
20122 7793 Acked 8792 Read 10 Done New Done false
20123 7794 Acked 8793 Read 4 Done New Done false
20124 7795 Acked 8794 Read 4 Done New Done false
20125 7796 Acked 8795 Write 6 Done New Done false
20126 7797 Acked 8796 Read 2 Done New Done false
20127 7798 Acked 8797 Read 7 Done New Done false
20128 7799 Acked 8798 Write 1 Done New Done false
20129 7800 Acked 8799 Read 8 Done New Done false
20130 7801 Acked 8800 Write 1 Done New Done false
20131 7802 Acked 8801 Read 7 Done New Done false
20132 7803 Acked 8802 Write 8 Done New Done false
20133 7804 Acked 8803 Read 1 Done New Done false
20134 7805 Acked 8804 Write 10 Done New Done false
20135 7806 Acked 8805 Read 1 Done New Done false
20136 7807 Acked 8806 Write 8 Done New Done false
20137 7808 Acked 8807 Read 7 Done New Done false
20138 7809 Acked 8808 Read 3 Done New Done false
20139 7810 Acked 8809 Write 4 Done New Done false
20140 7811 Acked 8810 Write 8 Done New Done false
20141 7812 Acked 8811 Read 7 Done New Done false
20142 7813 Acked 8812 Write 10 Done New Done false
20143 7814 Acked 8813 Read 4 Done New Done false
20144 7815 Acked 8814 Write 5 Done New Done false
20145 7816 Acked 8815 Read 4 Done New Done false
20146 7817 Acked 8816 Read 3 Done New Done false
20147 7818 Acked 8817 Write 3 Done New Done false
20148 7819 Acked 8818 Read 2 Done New Done false
20149 7820 Acked 8819 Write 4 Done New Done false
20150 7821 Acked 8820 Read 1 Done New Done false
20151 7822 Acked 8821 Read 9 Done New Done false
20152 7823 Acked 8822 Read 10 Done New Done false
20153 7824 Acked 8823 Read 8 Done New Done false
20154 7825 Acked 8824 Read 8 Done New Done false
20155 7826 Acked 8825 Read 2 Done New Done false
20156 7827 Acked 8826 Read 9 Done New Done false
20157 7828 Acked 8827 Write 8 Done New Done false
20158 7829 Acked 8828 Write 6 Done New Done false
20159 7830 Acked 8829 Read 6 Done New Done false
20160 7831 Acked 8830 Read 8 Done New Done false
20161 7832 Acked 8831 Read 8 Done New Done false
20162 7833 Acked 8832 Read 3 Done New Done false
20163 7834 Acked 8833 Write 4 Done New Done false
20164 7835 Acked 8834 Write 3 Done New Done false
20165 7836 Acked 8835 Read 8 Done New Done false
20166 7837 Acked 8836 Write 8 Done New Done false
20167 7838 Acked 8837 Read 2 Done New Done false
20168 7839 Acked 8838 Write 2 Done New Done false
20169 7840 Acked 8839 Read 10 Done New Done false
20170 7841 Acked 8840 Read 3 Done New Done false
20171 7842 Acked 8841 Flush 0 Done New Done false
20172 7843 Acked 8842 Read 8 Done New Done false
20173 7844 Acked 8843 Read 7 Done New Done false
20174 7845 Acked 8844 Flush 0 Done New Done false
20175 7846 Acked 8845 Read 3 Done New Done false
20176 7847 Acked 8846 Flush 0 Done New Done false
20177 7848 Acked 8847 Write 4 Done New Done false
20178 7849 Acked 8848 Read 9 Done New Done false
20179 7850 Acked 8849 Read 1 Done New Done false
20180 7851 Acked 8850 Read 9 Done New Done false
20181 7852 Acked 8851 Read 3 Done New Done false
20182 7853 Acked 8852 Read 2 Done New Done false
20183 7854 Acked 8853 Write 10 Done New Done false
20184 7855 Acked 8854 Write 10 Done New Done false
20185 7856 Acked 8855 Read 5 Done New Done false
20186 7857 Acked 8856 Read 4 Done New Done false
20187 7858 Acked 8857 Read 9 Done New Done false
20188 7859 Acked 8858 Write 3 Done New Done false
20189 7860 Acked 8859 Write 8 Done New Done false
20190 7861 Acked 8860 Read 8 Done New Done false
20191 7862 Acked 8861 Write 10 Done New Done false
20192 7863 Acked 8862 Write 9 Done New Done false
20193 7864 Acked 8863 Flush 0 Done New Done false
20194 7865 Acked 8864 Read 1 Done New Done false
20195 7866 Acked 8865 Write 1 Done New Done false
20196 7867 Acked 8866 Read 1 Done New Done false
20197 7868 Acked 8867 Write 10 Done New Done false
20198 7869 Acked 8868 Read 3 Done New Done false
20199 STATES DS:0 DS:1 DS:2 TOTAL
20200 New 0 301 0 301
20201 Sent 0 0 0 0
20202 Done 301 0 301 602
20203 Skipped 0 0 0 0
20204 Error 0 0 0 0
20205 Last Flush: 8863 8567 8863
20206 Downstairs last five completed: 8567 8566 8565 8564 8563 8562
20207 Upstairs last five completed: 7869 7868 7867 7866 7865 7864
20208 CLIENT: Up:0 ds:301 act:2
202092023-10-13T18:13:28.533ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
20210 [8820] got start action
20211 [8820] I am going to restart
20212 Starting downstairs at port 8820
20213 Make output file at "/tmp/dsc/downstairs-8820.txt"
20214 [8820][1] reports Starting
20215 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1202)
20216 [8820][1] reports Running
202172023-10-13T18:13:30.535ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 1
202182023-10-13T18:13:30.535ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
202192023-10-13T18:13:30.535ZINFOcrucible: [1] client re-new 302 jobs since flush 8567 = downstairs
202202023-10-13T18:13:30.536ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
202212023-10-13T18:13:30.537ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
202222023-10-13T18:13:30.537ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
202232023-10-13T18:13:30.537ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
202242023-10-13T18:13:30.537ZINFOcrucible: [1] send last flush ID to this DS: 8567
202252023-10-13T18:13:30.537ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Offline Active
202262023-10-13T18:13:30.537ZINFOcrucible: [1] Replied this last flush ID: 8567
202272023-10-13T18:13:30.537ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Offline Active ds_transition to Replay
202282023-10-13T18:13:30.537ZINFOcrucible: [1] Transition from Offline to Replay
202292023-10-13T18:13:30.537ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
202302023-10-13T18:13:30.537ZINFOcrucible: [1] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Replay Active ds_transition to Active
202312023-10-13T18:13:30.537ZINFOcrucible: [1] Transition from Replay to Active
202322023-10-13T18:13:30.537ZINFOcrucible: [1] Starts cmd_loop
202332023-10-13T18:13:30.538ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
202342023-10-13T18:13:30.538ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
202352023-10-13T18:13:30.538ZINFOcrucible: Set check for repair
202362023-10-13T18:13:31.538ZWARNcrucible: [1] flow control end
202372023-10-13T18:13:31.539ZINFOcrucible: Checking if live repair is needed
202382023-10-13T18:13:31.539ZINFOcrucible: No Live Repair required at this time
20239 ----------------------------------------------------------------
20240 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20241 Downstairs last five completed: 8869 8868 8867 8866 8865 8864
20242 Upstairs last five completed: 7870 7869 7868 7867 7866 7865
20243 CLIENT: Up:0 ds:0 act:3
20244 Replay: All jobs finished, all DS active.
20245 Main task has work to do, go find it
20246 got dsc Stop(2)
20247 stop 2
20248 [8830] Got stop action so:false kr:false
20249 Oct 13 18:13:32.512 INFO request completed, latency_us: 516, response_code: 204, uri: /stop/cid/2, method: POST, req_id: e47ecd96-70b0-480c-8619-8dd0b649b1b6, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
20250 Oct 13 18:13:32.513 INFO request completed, latency_us: 179, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4f012bca-938f-47a3-802d-82b0ef056546, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
202512023-10-13T18:13:32.513ZWARNcrucible: [2] downstairs disconnected
202522023-10-13T18:13:32.513ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
202532023-10-13T18:13:32.513ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8830 closed looper = 2
202542023-10-13T18:13:32.513ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
202552023-10-13T18:13:32.513ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
202562023-10-13T18:13:32.514ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
202572023-10-13T18:13:32.514ZWARNcrucible: [2] pm_task rx.recv() is None
20258 [8830] Exited with: ExitStatus(unix_wait_status(9))
20259 [8830][2] reports Exit
20260 Oct 13 18:13:36.515 INFO request completed, latency_us: 260, response_code: 200, uri: /state/cid/2, method: GET, req_id: a4da0bf7-6fe4-49a2-8a70-4de7183551dc, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
20261 001/300 Flush
20262 002/300 Write block 2179 len 16384 data: 4 3 4 4
20263 003/300 Write block 3220 len 12288 data: 5 5 5
20264 004/300 Read block 3269 len 8192
20265 005/300 Write block 4227 len 24576 data: 3 3 4 4 5 4
20266 006/300 Read block 3066 len 32768
20267 007/300 Read block 1229 len 16384
20268 008/300 Read block 2511 len 32768
20269 009/300 Write block 1473 len 12288 data: 4 4 5
20270 010/300 Read block 1729 len 32768
20271 011/300 Write block 619 len 32768 data: 6 6 3 3 3 4 5 5
20272 012/300 Read block 3099 len 12288
20273 013/300 Write block 3950 len 28672 data: 5 4 4 5 6 7 6
20274 014/300 Read block 4162 len 28672
20275 015/300 Write block 4892 len 16384 data: 8 7 6 6
20276 016/300 Write block 2513 len 16384 data: 6 5 4 3
20277 017/300 Read block 3150 len 8192
20278 018/300 Read block 4993 len 20480
20279 019/300 Read block 1161 len 28672
20280 020/300 Write block 442 len 40960 data: 7 9 7 7 10 7 6 6 5 7
20281 021/300 Read block 4573 len 28672
20282 022/300 Read block 1240 len 12288
20283 023/300 Write block 4373 len 12288 data: 8 9 7
20284 024/300 Write block 1738 len 8192 data: 5 5
20285 025/300 Write block 4640 len 28672 data: 3 3 3 2 2 2 2
20286 026/300 Write block 595 len 8192 data: 7 7
20287 027/300 Write block 711 len 8192 data: 3 4
20288 028/300 Read block 1425 len 40960
20289 029/300 Read block 4511 len 8192
20290 030/300 Write block 3229 len 16384 data: 5 5 4 4
20291 031/300 Read block 1076 len 16384
20292 032/300 Read block 4312 len 20480
20293 033/300 Read block 801 len 36864
20294 034/300 Flush
20295 035/300 Read block 4245 len 40960
20296 036/300 Read block 774 len 28672
20297 037/300 Flush
20298 038/300 Read block 1556 len 4096
20299 039/300 Write block 1356 len 12288 data: 3 3 3
20300 040/300 Read block 151 len 40960
20301 041/300 Flush
20302 042/300 Write block 99 len 4096 data: 4
20303 043/300 Read block 4301 len 36864
20304 044/300 Read block 4096 len 24576
20305 045/300 Write block 1537 len 32768 data: 6 5 5 3 5 5 6 4
20306 046/300 Read block 1186 len 40960
20307 047/300 Flush
20308 048/300 Write block 4110 len 16384 data: 2 2 3 4
20309 049/300 Write block 1653 len 8192 data: 7 6
20310 050/300 Read block 4310 len 28672
20311 051/300 Read block 1299 len 32768
20312 052/300 Flush
20313 053/300 Flush
20314 054/300 Read block 1791 len 12288
20315 055/300 Write block 25 len 40960 data: 5 4 4 4 5 5 5 5 5 6
20316 056/300 Read block 276 len 32768
20317 057/300 Read block 3550 len 4096
20318 058/300 Write block 641 len 8192 data: 4 6
20319 059/300 Write block 4472 len 32768 data: 2 2 3 2 3 3 3 3
20320 060/300 Read block 1154 len 40960
20321 061/300 Flush
20322 062/300 Read block 3186 len 40960
20323 063/300 Flush
20324 064/300 Read block 2523 len 4096
20325 065/300 Write block 52 len 20480 data: 3 3 3 3 3
20326 066/300 Read block 912 len 28672
20327 067/300 Write block 3383 len 32768 data: 7 6 6 4 5 5 5 4
20328 068/300 Read block 2265 len 24576
20329 069/300 Write block 1658 len 36864 data: 5 5 6 7 8 9 9 8 7
20330 070/300 Read block 1944 len 24576
20331 071/300 Write block 2888 len 8192 data: 6 6
20332 072/300 Read block 4444 len 16384
20333 073/300 Read block 3719 len 36864
20334 074/300 Read block 3896 len 12288
20335 075/300 Read block 1071 len 16384
20336 076/300 Write block 961 len 32768 data: 8 6 4 3 2 2 3 4
20337 077/300 Read block 669 len 24576
20338 078/300 Write block 1368 len 4096 data: 9
20339 079/300 Read block 4216 len 36864
20340 080/300 Read block 3115 len 32768
20341 081/300 Read block 1300 len 20480
20342 082/300 Flush
20343 083/300 Read block 39 len 24576
20344 084/300 Read block 2029 len 4096
20345 085/300 Read block 1419 len 24576
20346 086/300 Read block 1465 len 24576
20347 087/300 Read block 2863 len 36864
20348 088/300 Write block 1898 len 24576 data: 7 7 7 5 6 5
20349 089/300 Read block 3552 len 24576
20350 090/300 Write block 2502 len 12288 data: 6 6 7
20351 091/300 Read block 2842 len 20480
20352 092/300 Flush
20353 093/300 Write block 1827 len 4096 data: 3
20354 094/300 Write block 78 len 4096 data: 4
20355 095/300 Read block 1095 len 4096
20356 096/300 Write block 1419 len 4096 data: 4
20357 097/300 Read block 303 len 8192
20358 098/300 Write block 2879 len 24576 data: 5 5 5 6 7 6
20359 099/300 Write block 440 len 36864 data: 6 7 8 10 8 8 11 8 7
20360 100/300 Flush
20361 101/300 Read block 2523 len 4096
20362 102/300 Read block 2390 len 36864
20363 103/300 Write block 4616 len 24576 data: 5 5 4 6 5 6
20364 104/300 Read block 4100 len 4096
20365 105/300 Write block 3049 len 24576 data: 7 7 6 4 3 3
20366 106/300 Read block 917 len 4096
20367 107/300 Read block 1706 len 16384
20368 108/300 Write block 4369 len 16384 data: 5 5 6 7
20369 109/300 Write block 2353 len 40960 data: 8 9 9 8 9 6 7 5 5 6
20370 110/300 Read block 2818 len 40960
20371 111/300 Write block 4463 len 28672 data: 4 4 5 4 4 4 4
20372 112/300 Write block 3473 len 28672 data: 5 6 6 7 7 6 5
20373 113/300 Write block 492 len 20480 data: 7 6 7 6 6
20374 114/300 Read block 1148 len 8192
20375 115/300 Read block 149 len 32768
20376 116/300 Write block 151 len 36864 data: 7 7 6 6 5 5 5 4 4
20377 117/300 Write block 706 len 24576 data: 7 6 5 4 3 4
20378 118/300 Write block 1784 len 32768 data: 7 6 5 5 6 9 8 8
20379 119/300 Write block 1742 len 4096 data: 6
20380 120/300 Write block 4903 len 16384 data: 6 4 4 5
20381 121/300 Write block 3892 len 4096 data: 7
20382 122/300 Write block 4920 len 32768 data: 7 7 8 6 5 3 2 4
20383 123/300 Read block 2623 len 32768
20384 124/300 Flush
20385 125/300 Read block 3118 len 24576
20386 126/300 Read block 581 len 36864
20387 127/300 Write block 3562 len 40960 data: 4 4 5 4 4 5 6 9 8 8
20388 128/300 Read block 1111 len 12288
20389 129/300 Read block 2404 len 12288
20390 130/300 Write block 3090 len 36864 data: 6 7 9 10 10 12 10 8 7
20391 131/300 Read block 2096 len 24576
20392 132/300 Read block 1085 len 28672
20393 133/300 Write block 2630 len 32768 data: 4 4 2 2 3 3 5 6
20394 134/300 Read block 4378 len 24576
20395 135/300 Read block 2598 len 24576
20396 136/300 Read block 4141 len 24576
20397 137/300 Write block 243 len 36864 data: 5 5 6 5 5 5 4 4 4
20398 138/300 Read block 3475 len 12288
20399 139/300 Flush
20400 140/300 Write block 970 len 40960 data: 4 4 4 4 4 5 5 5 3 2
20401 141/300 Read block 26 len 12288
20402 142/300 Read block 3176 len 12288
20403 143/300 Write block 2470 len 8192 data: 9 11
20404 144/300 Read block 2696 len 4096
20405 145/300 Read block 786 len 8192
20406 146/300 Read block 502 len 4096
20407 147/300 Read block 2398 len 28672
20408 148/300 Write block 95 len 12288 data: 7 6 5
20409 149/300 Write block 2643 len 20480 data: 6 6 4 3 3
20410 150/300 Write block 3450 len 36864 data: 4 4 4 5 4 6 6 5 6
20411 151/300 Write block 4055 len 24576 data: 3 4 7 8 7 6
20412 152/300 Read block 2560 len 28672
20413 153/300 Read block 2073 len 36864
20414 154/300 Read block 3851 len 20480
20415 155/300 Read block 1122 len 8192
20416 156/300 Read block 4226 len 12288
20417 157/300 Write block 4903 len 16384 data: 7 5 5 6
20418 158/300 Write block 3990 len 20480 data: 6 7 7 7 8
20419 159/300 Read block 1614 len 28672
20420 160/300 Read block 3830 len 24576
20421 161/300 Read block 2989 len 24576
20422 162/300 Write block 4770 len 12288 data: 3 5 4
20423 163/300 Read block 1414 len 16384
20424 164/300 Read block 1130 len 8192
20425 165/300 Write block 3671 len 40960 data: 6 4 4 3 4 4 4 3 4 7
20426 166/300 Flush
20427 167/300 Read block 846 len 20480
20428 168/300 Flush
20429 169/300 Write block 3582 len 28672 data: 3 3 3 2 2 3 4
20430 170/300 Read block 291 len 40960
20431 171/300 Read block 1414 len 4096
20432 172/300 Write block 480 len 8192 data: 4 5
20433 173/300 Flush
20434 174/300 Write block 2624 len 4096 data: 6
20435 175/300 Write block 374 len 40960 data: 7 7 4 7 9 9 7 7 8 6
20436 176/300 Write block 2529 len 40960 data: 10 9 10 10 8 5 5 6 5 5
20437 177/300 Write block 60 len 8192 data: 4 3
20438 178/300 Flush
20439 179/300 Read block 4817 len 12288
20440 180/300 Read block 4382 len 4096
20441 181/300 Flush
20442 182/300 Write block 4198 len 8192 data: 6 7
20443 183/300 Write block 4435 len 12288 data: 4 2 2
20444 184/300 Read block 1365 len 28672
20445 185/300 Write block 691 len 32768 data: 7 7 7 7 5 4 4 6
20446 186/300 Read block 4431 len 28672
20447 187/300 Write block 1581 len 12288 data: 3 3 3
20448 188/300 Flush
20449 189/300 Read block 999 len 40960
20450 190/300 Write block 3125 len 24576 data: 2 2 3 3 3 3
20451 191/300 Read block 1554 len 16384
20452 192/300 Write block 2909 len 12288 data: 3 3 3
20453 193/300 Read block 2782 len 12288
20454 194/300 Read block 3174 len 40960
20455 195/300 Write block 4223 len 4096 data: 5
20456 196/300 Read block 870 len 36864
20457 197/300 Read block 2197 len 24576
20458 198/300 Read block 2777 len 16384
20459 199/300 Flush
20460 200/300 Write block 2355 len 16384 data: 10 9 10 7
20461 201/300 Read block 628 len 16384
20462 202/300 Read block 3654 len 32768
20463 203/300 Write block 744 len 28672 data: 5 5 6 6 5 5 5
20464 204/300 Write block 4388 len 4096 data: 8
20465 205/300 Write block 4890 len 24576 data: 7 6 9 8 7 7
20466 206/300 Read block 925 len 4096
20467 207/300 Read block 2313 len 16384
20468 208/300 Write block 4846 len 8192 data: 5 5
20469 209/300 Write block 2097 len 20480 data: 5 5 6 5 3
20470 210/300 Read block 4401 len 20480
20471 211/300 Write block 1333 len 20480 data: 8 7 7 8 7
20472 212/300 Write block 3261 len 16384 data: 8 7 5 4
20473 213/300 Read block 2193 len 28672
20474 214/300 Read block 3925 len 4096
20475 215/300 Write block 817 len 32768 data: 4 5 6 6 5 5 5 5
20476 216/300 Read block 1854 len 20480
20477 217/300 Read block 2603 len 28672
20478 218/300 Read block 23 len 32768
20479 219/300 Read block 3560 len 36864
20480 220/300 Flush
20481 221/300 Read block 3923 len 8192
20482 222/300 Read block 2429 len 28672
20483 223/300 Read block 146 len 16384
20484 224/300 Write block 4484 len 40960 data: 4 4 5 5 6 7 6 5 5 4
20485 225/300 Flush
20486 226/300 Write block 2230 len 20480 data: 3 4 4 4 7
20487 227/300 Read block 2739 len 40960
20488 228/300 Read block 4056 len 40960
20489 229/300 Read block 2521 len 36864
20490 230/300 Write block 512 len 32768 data: 6 6 6 6 4 4 2 2
20491 231/300 Flush
20492 232/300 Write block 4453 len 8192 data: 8 7
20493 233/300 Flush
20494 234/300 Read block 3444 len 20480
20495 235/300 Write block 124 len 36864 data: 8 8 8 8 9 9 9 10 9
20496 236/300 Write block 3542 len 36864 data: 8 7 7 9 8 8 8 7 7
20497 237/300 Write block 2848 len 16384 data: 4 4 4 4
20498 238/300 Write block 1680 len 40960 data: 5 5 5 5 5 4 3 3 3 4
20499 239/300 Write block 4250 len 32768 data: 3 3 5 5 7 6 6 4
20500 240/300 Write block 3614 len 36864 data: 5 4 5 8 9 9 10 5 4
20501 241/300 Read block 873 len 28672
20502 242/300 Write block 218 len 32768 data: 5 3 3 3 4 5 4 4
20503 243/300 Read block 4107 len 12288
20504 244/300 Write block 2871 len 8192 data: 6 6
20505 245/300 Write block 4067 len 24576 data: 5 5 5 5 6 4
20506 246/300 Read block 312 len 40960
20507 247/300 Write block 2706 len 8192 data: 4 4
20508 248/300 Read block 47 len 28672
20509 249/300 Read block 3520 len 16384
20510 250/300 Write block 1703 len 40960 data: 4 4 4 3 3 5 5 5 5 4
20511 251/300 Write block 130 len 8192 data: 10 11
20512 252/300 Read block 129 len 40960
20513 253/300 Write block 1983 len 40960 data: 3 3 3 3 2 2 3 3 3 3
20514 254/300 Read block 1518 len 16384
20515 255/300 Flush
20516 256/300 Read block 4082 len 20480
20517 257/300 Read block 392 len 24576
20518 258/300 Read block 2742 len 16384
20519 259/300 Read block 3538 len 40960
20520 260/300 Read block 1696 len 40960
20521 261/300 Write block 531 len 40960 data: 3 2 2 2 3 3 3 3 3 3
20522 262/300 Write block 3900 len 12288 data: 3 4 5
20523 263/300 Flush
20524 264/300 Write block 670 len 4096 data: 3
20525 265/300 Write block 978 len 16384 data: 4 3 3 4
20526 266/300 Write block 409 len 20480 data: 4 4 3 3 5
20527 267/300 Write block 2581 len 20480 data: 6 6 5 5 7
20528 268/300 Write block 4066 len 28672 data: 5 6 6 6 6 7 5
20529 269/300 Read block 466 len 16384
20530 270/300 Read block 2422 len 36864
20531 271/300 Write block 4723 len 20480 data: 4 6 6 6 8
20532 272/300 Write block 1693 len 4096 data: 7
20533 273/300 Read block 2267 len 16384
20534 274/300 Read block 4889 len 32768
20535 275/300 Write block 699 len 32768 data: 6 5 5 5 6 6 7 8
20536 276/300 Write block 792 len 28672 data: 3 3 4 4 4 4 4
20537 277/300 Write block 2802 len 24576 data: 3 3 4 5 6 6
20538 278/300 Write block 179 len 36864 data: 7 5 4 5 5 5 7 8 8
20539 279/300 Flush
20540 280/300 Read block 2202 len 40960
20541 281/300 Write block 3380 len 28672 data: 6 6 6 8 7 7 5
20542 282/300 Write block 1475 len 8192 data: 6 5
20543 283/300 Write block 3132 len 28672 data: 4 6 6 6 4 5 5
20544 284/300 Write block 1304 len 28672 data: 4 5 6 7 7 9 9
20545 285/300 Write block 1288 len 24576 data: 4 4 4 4 4 3
20546 286/300 Read block 544 len 24576
20547 287/300 Read block 1476 len 12288
20548 288/300 Write block 4309 len 4096 data: 8
20549 289/300 Read block 4982 len 12288
20550 290/300 Read block 1534 len 20480
20551 291/300 Read block 2511 len 28672
20552 292/300 Write block 1139 len 20480 data: 5 5 5 5 4
20553 293/300 Read block 450 len 20480
20554 294/300 Write block 667 len 4096 data: 4
20555 295/300 Write block 508 len 36864 data: 4 6 5 5 7 7 7 7 5
20556 296/300 Read block 1741 len 28672
20557 297/300 Read block 1051 len 40960
20558 298/300 Read block 685 len 28672
20559 299/300 Write block 2572 len 4096 data: 3
20560 300/300 Write block 3833 len 32768 data: 7 6 7 7 7 7 7 6
20561 Main task has work to do, go find it
20562 got dsc Start(2)
20563 start 2
20564 Oct 13 18:13:37.475 INFO request completed, latency_us: 321, response_code: 204, uri: /start/cid/2, method: POST, req_id: c28f21e0-ce8c-49a1-92b2-5d124f770789, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
20565 Replay: started 2, returned:Ok(())
20566 ----------------------------------------------------------------
20567 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
20568 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
20569 7871 Acked 8870 Flush 0 Done Done New false
20570 7872 Acked 8871 Write 4 Done Done New false
20571 7873 Acked 8872 Write 3 Done Done New false
20572 7874 Acked 8873 Read 2 Done Done New false
20573 7875 Acked 8874 Write 6 Done Done New false
20574 7876 Acked 8875 Read 8 Done Done New false
20575 7877 Acked 8876 Read 4 Done Done New false
20576 7878 Acked 8877 Read 8 Done Done New false
20577 7879 Acked 8878 Write 3 Done Done New false
20578 7880 Acked 8879 Read 8 Done Done New false
20579 7881 Acked 8880 Write 8 Done Done New false
20580 7882 Acked 8881 Read 3 Done Done New false
20581 7883 Acked 8882 Write 7 Done Done New false
20582 7884 Acked 8883 Read 7 Done Done New false
20583 7885 Acked 8884 Write 4 Done Done New false
20584 7886 Acked 8885 Write 4 Done Done New false
20585 7887 Acked 8886 Read 2 Done Done New false
20586 7888 Acked 8887 Read 5 Done Done New false
20587 7889 Acked 8888 Read 7 Done Done New false
20588 7890 Acked 8889 Write 10 Done Done New false
20589 7891 Acked 8890 Read 7 Done Done New false
20590 7892 Acked 8891 Read 3 Done Done New false
20591 7893 Acked 8892 Write 3 Done Done New false
20592 7894 Acked 8893 Write 2 Done Done New false
20593 7895 Acked 8894 Write 7 Done Done New false
20594 7896 Acked 8895 Write 2 Done Done New false
20595 7897 Acked 8896 Write 2 Done Done New false
20596 7898 Acked 8897 Read 10 Done Done New false
20597 7899 Acked 8898 Read 2 Done Done New false
20598 7900 Acked 8899 Write 4 Done Done New false
20599 7901 Acked 8900 Read 4 Done Done New false
20600 7902 Acked 8901 Read 5 Done Done New false
20601 7903 Acked 8902 Read 9 Done Done New false
20602 7904 Acked 8903 Flush 0 Done Done New false
20603 7905 Acked 8904 Read 10 Done Done New false
20604 7906 Acked 8905 Read 7 Done Done New false
20605 7907 Acked 8906 Flush 0 Done Done New false
20606 7908 Acked 8907 Flush 0 Done Done New false
20607 7909 Acked 8908 Read 1 Done Done New false
20608 7910 Acked 8909 Write 3 Done Done New false
20609 7911 Acked 8910 Read 10 Done Done New false
20610 7912 Acked 8911 Flush 0 Done Done New false
20611 7913 Acked 8912 Write 1 Done Done New false
20612 7914 Acked 8913 Read 9 Done Done New false
20613 7915 Acked 8914 Read 6 Done Done New false
20614 7916 Acked 8915 Write 8 Done Done New false
20615 7917 Acked 8916 Read 10 Done Done New false
20616 7918 Acked 8917 Flush 0 Done Done New false
20617 7919 Acked 8918 Write 4 Done Done New false
20618 7920 Acked 8919 Write 2 Done Done New false
20619 7921 Acked 8920 Read 7 Done Done New false
20620 7922 Acked 8921 Read 8 Done Done New false
20621 7923 Acked 8922 Flush 0 Done Done New false
20622 7924 Acked 8923 Flush 0 Done Done New false
20623 7925 Acked 8924 Read 3 Done Done New false
20624 7926 Acked 8925 Write 10 Done Done New false
20625 7927 Acked 8926 Read 8 Done Done New false
20626 7928 Acked 8927 Read 1 Done Done New false
20627 7929 Acked 8928 Write 2 Done Done New false
20628 7930 Acked 8929 Write 8 Done Done New false
20629 7931 Acked 8930 Read 10 Done Done New false
20630 7932 Acked 8931 Flush 0 Done Done New false
20631 7933 Acked 8932 Read 10 Done Done New false
20632 7934 Acked 8933 Flush 0 Done Done New false
20633 7935 Acked 8934 Read 1 Done Done New false
20634 7936 Acked 8935 Write 5 Done Done New false
20635 7937 Acked 8936 Read 7 Done Done New false
20636 7938 Acked 8937 Write 8 Done Done New false
20637 7939 Acked 8938 Read 6 Done Done New false
20638 7940 Acked 8939 Write 9 Done Done New false
20639 7941 Acked 8940 Read 6 Done Done New false
20640 7942 Acked 8941 Write 2 Done Done New false
20641 7943 Acked 8942 Read 4 Done Done New false
20642 7944 Acked 8943 Read 9 Done Done New false
20643 7945 Acked 8944 Read 3 Done Done New false
20644 7946 Acked 8945 Read 4 Done Done New false
20645 7947 Acked 8946 Write 8 Done Done New false
20646 7948 Acked 8947 Read 6 Done Done New false
20647 7949 Acked 8948 Write 1 Done Done New false
20648 7950 Acked 8949 Read 9 Done Done New false
20649 7951 Acked 8950 Read 8 Done Done New false
20650 7952 Acked 8951 Read 5 Done Done New false
20651 7953 Acked 8952 Flush 0 Done Done New false
20652 7954 Acked 8953 Read 6 Done Done New false
20653 7955 Acked 8954 Read 1 Done Done New false
20654 7956 Acked 8955 Read 6 Done Done New false
20655 7957 Acked 8956 Read 6 Done Done New false
20656 7958 Acked 8957 Read 9 Done Done New false
20657 7959 Acked 8958 Write 6 Done Done New false
20658 7960 Acked 8959 Read 6 Done Done New false
20659 7961 Acked 8960 Write 3 Done Done New false
20660 7962 Acked 8961 Read 5 Done Done New false
20661 7963 Acked 8962 Flush 0 Done Done New false
20662 7964 Acked 8963 Write 1 Done Done New false
20663 7965 Acked 8964 Write 1 Done Done New false
20664 7966 Acked 8965 Read 1 Done Done New false
20665 7967 Acked 8966 Write 1 Done Done New false
20666 7968 Acked 8967 Read 2 Done Done New false
20667 7969 Acked 8968 Write 6 Done Done New false
20668 7970 Acked 8969 Write 9 Done Done New false
20669 7971 Acked 8970 Flush 0 Done Done New false
20670 7972 Acked 8971 Read 1 Done Done New false
20671 7973 Acked 8972 Read 9 Done Done New false
20672 7974 Acked 8973 Write 6 Done Done New false
20673 7975 Acked 8974 Read 1 Done Done New false
20674 7976 Acked 8975 Write 6 Done Done New false
20675 7977 Acked 8976 Read 1 Done Done New false
20676 7978 Acked 8977 Read 4 Done Done New false
20677 7979 Acked 8978 Write 4 Done Done New false
20678 7980 Acked 8979 Write 10 Done Done New false
20679 7981 Acked 8980 Read 10 Done Done New false
20680 7982 Acked 8981 Write 7 Done Done New false
20681 7983 Acked 8982 Write 7 Done Done New false
20682 7984 Acked 8983 Write 5 Done Done New false
20683 7985 Acked 8984 Read 2 Done Done New false
20684 7986 Acked 8985 Read 8 Done Done New false
20685 7987 Acked 8986 Write 9 Done Done New false
20686 7988 Acked 8987 Write 6 Done Done New false
20687 7989 Acked 8988 Write 8 Done Done New false
20688 7990 Acked 8989 Write 1 Done Done New false
20689 7991 Acked 8990 Write 4 Done Done New false
20690 7992 Acked 8991 Write 1 Done Done New false
20691 7993 Acked 8992 Write 8 Done Done New false
20692 7994 Acked 8993 Read 8 Done Done New false
20693 7995 Acked 8994 Flush 0 Done Done New false
20694 7996 Acked 8995 Read 6 Done Done New false
20695 7997 Acked 8996 Read 9 Done Done New false
20696 7998 Acked 8997 Write 10 Done Done New false
20697 7999 Acked 8998 Read 3 Done Done New false
20698 8000 Acked 8999 Read 3 Done Done New false
20699 8001 Acked 9000 Write 9 Done Done New false
20700 8002 Acked 9001 Read 6 Done Done New false
20701 8003 Acked 9002 Read 7 Done Done New false
20702 8004 Acked 9003 Write 8 Done Done New false
20703 8005 Acked 9004 Read 6 Done Done New false
20704 8006 Acked 9005 Read 6 Done Done New false
20705 8007 Acked 9006 Read 6 Done Done New false
20706 8008 Acked 9007 Write 9 Done Done New false
20707 8009 Acked 9008 Read 3 Done Done New false
20708 8010 Acked 9009 Flush 0 Done Done New false
20709 8011 Acked 9010 Write 10 Done Done New false
20710 8012 Acked 9011 Read 3 Done Done New false
20711 8013 Acked 9012 Read 3 Done Done New false
20712 8014 Acked 9013 Write 2 Done Done New false
20713 8015 Acked 9014 Read 1 Done Done New false
20714 8016 Acked 9015 Read 2 Done Done New false
20715 8017 Acked 9016 Read 1 Done Done New false
20716 8018 Acked 9017 Read 7 Done Done New false
20717 8019 Acked 9018 Write 3 Done Done New false
20718 8020 Acked 9019 Write 5 Done Done New false
20719 8021 Acked 9020 Write 9 Done Done New false
20720 8022 Acked 9021 Write 6 Done Done New false
20721 8023 Acked 9022 Read 7 Done Done New false
20722 8024 Acked 9023 Read 9 Done Done New false
20723 8025 Acked 9024 Read 5 Done Done New false
20724 8026 Acked 9025 Read 2 Done Done New false
20725 8027 Acked 9026 Read 3 Done Done New false
20726 8028 Acked 9027 Write 4 Done Done New false
20727 8029 Acked 9028 Write 5 Done Done New false
20728 8030 Acked 9029 Read 7 Done Done New false
20729 8031 Acked 9030 Read 6 Done Done New false
20730 8032 Acked 9031 Read 6 Done Done New false
20731 8033 Acked 9032 Write 3 Done Done New false
20732 8034 Acked 9033 Read 4 Done Done New false
20733 8035 Acked 9034 Read 2 Done Done New false
20734 8036 Acked 9035 Write 10 Done Done New false
20735 8037 Acked 9036 Flush 0 Done Done New false
20736 8038 Acked 9037 Read 5 Done Done New false
20737 8039 Acked 9038 Flush 0 Done Done New false
20738 8040 Acked 9039 Write 7 Done Done New false
20739 8041 Acked 9040 Read 10 Done Done New false
20740 8042 Acked 9041 Read 1 Done Done New false
20741 8043 Acked 9042 Write 2 Done Done New false
20742 8044 Acked 9043 Flush 0 Done Done New false
20743 8045 Acked 9044 Write 1 Done Done New false
20744 8046 Acked 9045 Write 10 Done Done New false
20745 8047 Acked 9046 Write 10 Done Done New false
20746 8048 Acked 9047 Write 2 Done Done New false
20747 8049 Acked 9048 Flush 0 Done Done New false
20748 8050 Acked 9049 Read 3 Done Done New false
20749 8051 Acked 9050 Read 1 Done Done New false
20750 8052 Acked 9051 Flush 0 Done Done New false
20751 8053 Acked 9052 Write 2 Done Done New false
20752 8054 Acked 9053 Write 3 Done Done New false
20753 8055 Acked 9054 Read 7 Done Done New false
20754 8056 Acked 9055 Write 8 Done Done New false
20755 8057 Acked 9056 Read 7 Done Done New false
20756 8058 Acked 9057 Write 3 Done Done New false
20757 8059 Acked 9058 Flush 0 Done Done New false
20758 8060 Acked 9059 Read 10 Done Done New false
20759 8061 Acked 9060 Write 6 Done Done New false
20760 8062 Acked 9061 Read 4 Done Done New false
20761 8063 Acked 9062 Write 3 Done Done New false
20762 8064 Acked 9063 Read 3 Done Done New false
20763 8065 Acked 9064 Read 10 Done Done New false
20764 8066 Acked 9065 Write 1 Done Done New false
20765 8067 Acked 9066 Read 9 Done Done New false
20766 8068 Acked 9067 Flush 0 Done Done New false
20767 8069 Acked 9068 Read 6 Done Done New false
20768 8070 Acked 9069 Read 4 Done Done New false
20769 8071 Acked 9070 Flush 0 Done Done New false
20770 8072 Acked 9071 Write 4 Done Done New false
20771 8073 Acked 9072 Read 4 Done Done New false
20772 8074 Acked 9073 Read 8 Done Done New false
20773 8075 Acked 9074 Write 7 Done Done New false
20774 8076 Acked 9075 Write 1 Done Done New false
20775 8077 Acked 9076 Write 6 Done Done New false
20776 8078 Acked 9077 Read 1 Done Done New false
20777 8079 Acked 9078 Read 4 Done Done New false
20778 8080 Acked 9079 Write 2 Done Done New false
20779 8081 Acked 9080 Write 5 Done Done New false
20780 8082 Acked 9081 Read 5 Done Done New false
20781 8083 Acked 9082 Write 5 Done Done New false
20782 8084 Acked 9083 Write 4 Done Done New false
20783 8085 Acked 9084 Read 7 Done Done New false
20784 8086 Acked 9085 Read 1 Done Done New false
20785 8087 Acked 9086 Write 8 Done Done New false
20786 8088 Acked 9087 Read 5 Done Done New false
20787 8089 Acked 9088 Read 7 Done Done New false
20788 8090 Acked 9089 Read 8 Done Done New false
20789 8091 Acked 9090 Read 9 Done Done New false
20790 8092 Acked 9091 Flush 0 Done Done New false
20791 8093 Acked 9092 Read 2 Done Done New false
20792 8094 Acked 9093 Read 7 Done Done New false
20793 8095 Acked 9094 Read 4 Done Done New false
20794 8096 Acked 9095 Write 10 Done Done New false
20795 8097 Acked 9096 Flush 0 Done Done New false
20796 8098 Acked 9097 Write 5 Done Done New false
20797 8099 Acked 9098 Read 10 Done Done New false
20798 8100 Acked 9099 Read 10 Done Done New false
20799 8101 Acked 9100 Read 9 Done Done New false
20800 8102 Acked 9101 Write 8 Done Done New false
20801 8103 Acked 9102 Flush 0 Done Done New false
20802 8104 Acked 9103 Write 2 Done Done New false
20803 8105 Acked 9104 Flush 0 Done Done New false
20804 8106 Acked 9105 Read 5 Done Done New false
20805 8107 Acked 9106 Write 9 Done Done New false
20806 8108 Acked 9107 Write 9 Done Done New false
20807 8109 Acked 9108 Write 4 Done Done New false
20808 8110 Acked 9109 Write 10 Done Done New false
20809 8111 Acked 9110 Write 8 Done Done New false
20810 8112 Acked 9111 Write 9 Done Done New false
20811 8113 Acked 9112 Read 7 Done Done New false
20812 8114 Acked 9113 Write 8 Done Done New false
20813 8115 Acked 9114 Read 3 Done Done New false
20814 8116 Acked 9115 Write 2 Done Done New false
20815 8117 Acked 9116 Write 6 Done Done New false
20816 8118 Acked 9117 Read 10 Done Done New false
20817 8119 Acked 9118 Write 2 Done Done New false
20818 8120 Acked 9119 Read 7 Done Done New false
20819 8121 Acked 9120 Read 4 Done Done New false
20820 8122 Acked 9121 Write 10 Done Done New false
20821 8123 Acked 9122 Write 2 Done Done New false
20822 8124 Acked 9123 Read 10 Done Done New false
20823 8125 Acked 9124 Write 10 Done Done New false
20824 8126 Acked 9125 Read 4 Done Done New false
20825 8127 Acked 9126 Flush 0 Done Done New false
20826 8128 Acked 9127 Read 5 Done Done New false
20827 8129 Acked 9128 Read 6 Done Done New false
20828 8130 Acked 9129 Read 4 Done Done New false
20829 8131 Acked 9130 Read 10 Done Done New false
20830 8132 Acked 9131 Read 10 Done Done New false
20831 8133 Acked 9132 Write 10 Done Done New false
20832 8134 Acked 9133 Write 3 Done Done New false
20833 8135 Acked 9134 Flush 0 Done Done New false
20834 8136 Acked 9135 Write 1 Done Done New false
20835 8137 Acked 9136 Write 4 Done Done New false
20836 8138 Acked 9137 Write 5 Done Done New false
20837 8139 Acked 9138 Write 5 Done Done New false
20838 8140 Acked 9139 Write 7 Done Done New false
20839 8141 Acked 9140 Read 4 Done Done New false
20840 8142 Acked 9141 Read 9 Done Done New false
20841 8143 Acked 9142 Write 5 Done Done New false
20842 8144 Acked 9143 Write 1 Done Done New false
20843 8145 Acked 9144 Read 4 Done Done New false
20844 8146 Acked 9145 Read 8 Done Done New false
20845 8147 Acked 9146 Write 8 Done Done New false
20846 8148 Acked 9147 Write 7 Done Done New false
20847 8149 Acked 9148 Write 6 Done Done New false
20848 8150 Acked 9149 Write 9 Done Done New false
20849 8151 Acked 9150 Flush 0 Done Done New false
20850 8152 Acked 9151 Read 10 Done Done New false
20851 8153 Acked 9152 Write 7 Done Done New false
20852 8154 Acked 9153 Write 2 Done Done New false
20853 8155 Acked 9154 Write 7 Done Done New false
20854 8156 Acked 9155 Write 7 Done Done New false
20855 8157 Acked 9156 Write 6 Done Done New false
20856 8158 Acked 9157 Read 6 Done Done New false
20857 8159 Acked 9158 Read 3 Done Done New false
20858 8160 Acked 9159 Write 1 Done Done New false
20859 8161 Acked 9160 Read 3 Done Done New false
20860 8162 Acked 9161 Read 5 Done Done New false
20861 8163 Acked 9162 Read 7 Done Done New false
20862 8164 Acked 9163 Write 5 Done Done New false
20863 8165 Acked 9164 Read 5 Done Done New false
20864 8166 Acked 9165 Write 1 Done Done New false
20865 8167 Acked 9166 Write 9 Done Done New false
20866 8168 Acked 9167 Read 7 Done Done New false
20867 8169 Acked 9168 Read 10 Done Done New false
20868 8170 Acked 9169 Read 7 Done Done New false
20869 8171 Acked 9170 Write 1 Done Done New false
20870 8172 Acked 9171 Write 8 Sent Sent New false
20871 STATES DS:0 DS:1 DS:2 TOTAL
20872 New 0 0 302 302
20873 Sent 1 1 0 2
20874 Done 301 301 0 602
20875 Skipped 0 0 0 0
20876 Error 0 0 0 0
20877 Last Flush: 9150 9150 8869
20878 Downstairs last five completed: 8869 8868 8867 8866 8865 8864
20879 Upstairs last five completed: 8172 8171 8170 8169 8168 8167
20880 CLIENT: Up:0 ds:302 act:2
20881 ----------------------------------------------------------------
20882 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
20883 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
20884 7871 Acked 8870 Flush 0 Done Done New false
20885 7872 Acked 8871 Write 4 Done Done New false
20886 7873 Acked 8872 Write 3 Done Done New false
20887 7874 Acked 8873 Read 2 Done Done New false
20888 7875 Acked 8874 Write 6 Done Done New false
20889 7876 Acked 8875 Read 8 Done Done New false
20890 7877 Acked 8876 Read 4 Done Done New false
20891 7878 Acked 8877 Read 8 Done Done New false
20892 7879 Acked 8878 Write 3 Done Done New false
20893 7880 Acked 8879 Read 8 Done Done New false
20894 7881 Acked 8880 Write 8 Done Done New false
20895 7882 Acked 8881 Read 3 Done Done New false
20896 7883 Acked 8882 Write 7 Done Done New false
20897 7884 Acked 8883 Read 7 Done Done New false
20898 7885 Acked 8884 Write 4 Done Done New false
20899 7886 Acked 8885 Write 4 Done Done New false
20900 7887 Acked 8886 Read 2 Done Done New false
20901 7888 Acked 8887 Read 5 Done Done New false
20902 7889 Acked 8888 Read 7 Done Done New false
20903 7890 Acked 8889 Write 10 Done Done New false
20904 7891 Acked 8890 Read 7 Done Done New false
20905 7892 Acked 8891 Read 3 Done Done New false
20906 7893 Acked 8892 Write 3 Done Done New false
20907 7894 Acked 8893 Write 2 Done Done New false
20908 7895 Acked 8894 Write 7 Done Done New false
20909 7896 Acked 8895 Write 2 Done Done New false
20910 7897 Acked 8896 Write 2 Done Done New false
20911 7898 Acked 8897 Read 10 Done Done New false
20912 7899 Acked 8898 Read 2 Done Done New false
20913 7900 Acked 8899 Write 4 Done Done New false
20914 7901 Acked 8900 Read 4 Done Done New false
20915 7902 Acked 8901 Read 5 Done Done New false
20916 7903 Acked 8902 Read 9 Done Done New false
20917 7904 Acked 8903 Flush 0 Done Done New false
20918 7905 Acked 8904 Read 10 Done Done New false
20919 7906 Acked 8905 Read 7 Done Done New false
20920 7907 Acked 8906 Flush 0 Done Done New false
20921 7908 Acked 8907 Flush 0 Done Done New false
20922 7909 Acked 8908 Read 1 Done Done New false
20923 7910 Acked 8909 Write 3 Done Done New false
20924 7911 Acked 8910 Read 10 Done Done New false
20925 7912 Acked 8911 Flush 0 Done Done New false
20926 7913 Acked 8912 Write 1 Done Done New false
20927 7914 Acked 8913 Read 9 Done Done New false
20928 7915 Acked 8914 Read 6 Done Done New false
20929 7916 Acked 8915 Write 8 Done Done New false
20930 7917 Acked 8916 Read 10 Done Done New false
20931 7918 Acked 8917 Flush 0 Done Done New false
20932 7919 Acked 8918 Write 4 Done Done New false
20933 7920 Acked 8919 Write 2 Done Done New false
20934 7921 Acked 8920 Read 7 Done Done New false
20935 7922 Acked 8921 Read 8 Done Done New false
20936 7923 Acked 8922 Flush 0 Done Done New false
20937 7924 Acked 8923 Flush 0 Done Done New false
20938 7925 Acked 8924 Read 3 Done Done New false
20939 7926 Acked 8925 Write 10 Done Done New false
20940 7927 Acked 8926 Read 8 Done Done New false
20941 7928 Acked 8927 Read 1 Done Done New false
20942 7929 Acked 8928 Write 2 Done Done New false
20943 7930 Acked 8929 Write 8 Done Done New false
20944 7931 Acked 8930 Read 10 Done Done New false
20945 7932 Acked 8931 Flush 0 Done Done New false
20946 7933 Acked 8932 Read 10 Done Done New false
20947 7934 Acked 8933 Flush 0 Done Done New false
20948 7935 Acked 8934 Read 1 Done Done New false
20949 7936 Acked 8935 Write 5 Done Done New false
20950 7937 Acked 8936 Read 7 Done Done New false
20951 7938 Acked 8937 Write 8 Done Done New false
20952 7939 Acked 8938 Read 6 Done Done New false
20953 7940 Acked 8939 Write 9 Done Done New false
20954 7941 Acked 8940 Read 6 Done Done New false
20955 7942 Acked 8941 Write 2 Done Done New false
20956 7943 Acked 8942 Read 4 Done Done New false
20957 7944 Acked 8943 Read 9 Done Done New false
20958 7945 Acked 8944 Read 3 Done Done New false
20959 7946 Acked 8945 Read 4 Done Done New false
20960 7947 Acked 8946 Write 8 Done Done New false
20961 7948 Acked 8947 Read 6 Done Done New false
20962 7949 Acked 8948 Write 1 Done Done New false
20963 7950 Acked 8949 Read 9 Done Done New false
20964 7951 Acked 8950 Read 8 Done Done New false
20965 7952 Acked 8951 Read 5 Done Done New false
20966 7953 Acked 8952 Flush 0 Done Done New false
20967 7954 Acked 8953 Read 6 Done Done New false
20968 7955 Acked 8954 Read 1 Done Done New false
20969 7956 Acked 8955 Read 6 Done Done New false
20970 7957 Acked 8956 Read 6 Done Done New false
20971 7958 Acked 8957 Read 9 Done Done New false
20972 7959 Acked 8958 Write 6 Done Done New false
20973 7960 Acked 8959 Read 6 Done Done New false
20974 7961 Acked 8960 Write 3 Done Done New false
20975 7962 Acked 8961 Read 5 Done Done New false
20976 7963 Acked 8962 Flush 0 Done Done New false
20977 7964 Acked 8963 Write 1 Done Done New false
20978 7965 Acked 8964 Write 1 Done Done New false
20979 7966 Acked 8965 Read 1 Done Done New false
20980 7967 Acked 8966 Write 1 Done Done New false
20981 7968 Acked 8967 Read 2 Done Done New false
20982 7969 Acked 8968 Write 6 Done Done New false
20983 7970 Acked 8969 Write 9 Done Done New false
20984 7971 Acked 8970 Flush 0 Done Done New false
20985 7972 Acked 8971 Read 1 Done Done New false
20986 7973 Acked 8972 Read 9 Done Done New false
20987 7974 Acked 8973 Write 6 Done Done New false
20988 7975 Acked 8974 Read 1 Done Done New false
20989 7976 Acked 8975 Write 6 Done Done New false
20990 7977 Acked 8976 Read 1 Done Done New false
20991 7978 Acked 8977 Read 4 Done Done New false
20992 7979 Acked 8978 Write 4 Done Done New false
20993 7980 Acked 8979 Write 10 Done Done New false
20994 7981 Acked 8980 Read 10 Done Done New false
20995 7982 Acked 8981 Write 7 Done Done New false
20996 7983 Acked 8982 Write 7 Done Done New false
20997 7984 Acked 8983 Write 5 Done Done New false
20998 7985 Acked 8984 Read 2 Done Done New false
20999 7986 Acked 8985 Read 8 Done Done New false
21000 7987 Acked 8986 Write 9 Done Done New false
21001 7988 Acked 8987 Write 6 Done Done New false
21002 7989 Acked 8988 Write 8 Done Done New false
21003 7990 Acked 8989 Write 1 Done Done New false
21004 7991 Acked 8990 Write 4 Done Done New false
21005 7992 Acked 8991 Write 1 Done Done New false
21006 7993 Acked 8992 Write 8 Done Done New false
21007 7994 Acked 8993 Read 8 Done Done New false
21008 7995 Acked 8994 Flush 0 Done Done New false
21009 7996 Acked 8995 Read 6 Done Done New false
21010 7997 Acked 8996 Read 9 Done Done New false
21011 7998 Acked 8997 Write 10 Done Done New false
21012 7999 Acked 8998 Read 3 Done Done New false
21013 8000 Acked 8999 Read 3 Done Done New false
21014 8001 Acked 9000 Write 9 Done Done New false
21015 8002 Acked 9001 Read 6 Done Done New false
21016 8003 Acked 9002 Read 7 Done Done New false
21017 8004 Acked 9003 Write 8 Done Done New false
21018 8005 Acked 9004 Read 6 Done Done New false
21019 8006 Acked 9005 Read 6 Done Done New false
21020 8007 Acked 9006 Read 6 Done Done New false
21021 8008 Acked 9007 Write 9 Done Done New false
21022 8009 Acked 9008 Read 3 Done Done New false
21023 8010 Acked 9009 Flush 0 Done Done New false
21024 8011 Acked 9010 Write 10 Done Done New false
21025 8012 Acked 9011 Read 3 Done Done New false
21026 8013 Acked 9012 Read 3 Done Done New false
21027 8014 Acked 9013 Write 2 Done Done New false
21028 8015 Acked 9014 Read 1 Done Done New false
21029 8016 Acked 9015 Read 2 Done Done New false
21030 8017 Acked 9016 Read 1 Done Done New false
21031 8018 Acked 9017 Read 7 Done Done New false
21032 8019 Acked 9018 Write 3 Done Done New false
21033 8020 Acked 9019 Write 5 Done Done New false
21034 8021 Acked 9020 Write 9 Done Done New false
21035 8022 Acked 9021 Write 6 Done Done New false
21036 8023 Acked 9022 Read 7 Done Done New false
21037 8024 Acked 9023 Read 9 Done Done New false
21038 8025 Acked 9024 Read 5 Done Done New false
21039 8026 Acked 9025 Read 2 Done Done New false
21040 8027 Acked 9026 Read 3 Done Done New false
21041 8028 Acked 9027 Write 4 Done Done New false
21042 8029 Acked 9028 Write 5 Done Done New false
21043 8030 Acked 9029 Read 7 Done Done New false
21044 8031 Acked 9030 Read 6 Done Done New false
21045 8032 Acked 9031 Read 6 Done Done New false
21046 8033 Acked 9032 Write 3 Done Done New false
21047 8034 Acked 9033 Read 4 Done Done New false
21048 8035 Acked 9034 Read 2 Done Done New false
21049 8036 Acked 9035 Write 10 Done Done New false
21050 8037 Acked 9036 Flush 0 Done Done New false
21051 8038 Acked 9037 Read 5 Done Done New false
21052 8039 Acked 9038 Flush 0 Done Done New false
21053 8040 Acked 9039 Write 7 Done Done New false
21054 8041 Acked 9040 Read 10 Done Done New false
21055 8042 Acked 9041 Read 1 Done Done New false
21056 8043 Acked 9042 Write 2 Done Done New false
21057 8044 Acked 9043 Flush 0 Done Done New false
21058 8045 Acked 9044 Write 1 Done Done New false
21059 8046 Acked 9045 Write 10 Done Done New false
21060 8047 Acked 9046 Write 10 Done Done New false
21061 8048 Acked 9047 Write 2 Done Done New false
21062 8049 Acked 9048 Flush 0 Done Done New false
21063 8050 Acked 9049 Read 3 Done Done New false
21064 8051 Acked 9050 Read 1 Done Done New false
21065 8052 Acked 9051 Flush 0 Done Done New false
21066 8053 Acked 9052 Write 2 Done Done New false
21067 8054 Acked 9053 Write 3 Done Done New false
21068 8055 Acked 9054 Read 7 Done Done New false
21069 8056 Acked 9055 Write 8 Done Done New false
21070 8057 Acked 9056 Read 7 Done Done New false
21071 8058 Acked 9057 Write 3 Done Done New false
21072 8059 Acked 9058 Flush 0 Done Done New false
21073 8060 Acked 9059 Read 10 Done Done New false
21074 8061 Acked 9060 Write 6 Done Done New false
21075 8062 Acked 9061 Read 4 Done Done New false
21076 8063 Acked 9062 Write 3 Done Done New false
21077 8064 Acked 9063 Read 3 Done Done New false
21078 8065 Acked 9064 Read 10 Done Done New false
21079 8066 Acked 9065 Write 1 Done Done New false
21080 8067 Acked 9066 Read 9 Done Done New false
21081 8068 Acked 9067 Flush 0 Done Done New false
21082 8069 Acked 9068 Read 6 Done Done New false
21083 8070 Acked 9069 Read 4 Done Done New false
21084 8071 Acked 9070 Flush 0 Done Done New false
21085 8072 Acked 9071 Write 4 Done Done New false
21086 8073 Acked 9072 Read 4 Done Done New false
21087 8074 Acked 9073 Read 8 Done Done New false
21088 8075 Acked 9074 Write 7 Done Done New false
21089 8076 Acked 9075 Write 1 Done Done New false
21090 8077 Acked 9076 Write 6 Done Done New false
21091 8078 Acked 9077 Read 1 Done Done New false
21092 8079 Acked 9078 Read 4 Done Done New false
21093 8080 Acked 9079 Write 2 Done Done New false
21094 8081 Acked 9080 Write 5 Done Done New false
21095 8082 Acked 9081 Read 5 Done Done New false
21096 8083 Acked 9082 Write 5 Done Done New false
21097 8084 Acked 9083 Write 4 Done Done New false
21098 8085 Acked 9084 Read 7 Done Done New false
21099 8086 Acked 9085 Read 1 Done Done New false
21100 8087 Acked 9086 Write 8 Done Done New false
21101 8088 Acked 9087 Read 5 Done Done New false
21102 8089 Acked 9088 Read 7 Done Done New false
21103 8090 Acked 9089 Read 8 Done Done New false
21104 8091 Acked 9090 Read 9 Done Done New false
21105 8092 Acked 9091 Flush 0 Done Done New false
21106 8093 Acked 9092 Read 2 Done Done New false
21107 8094 Acked 9093 Read 7 Done Done New false
21108 8095 Acked 9094 Read 4 Done Done New false
21109 8096 Acked 9095 Write 10 Done Done New false
21110 8097 Acked 9096 Flush 0 Done Done New false
21111 8098 Acked 9097 Write 5 Done Done New false
21112 8099 Acked 9098 Read 10 Done Done New false
21113 8100 Acked 9099 Read 10 Done Done New false
21114 8101 Acked 9100 Read 9 Done Done New false
21115 8102 Acked 9101 Write 8 Done Done New false
21116 8103 Acked 9102 Flush 0 Done Done New false
21117 8104 Acked 9103 Write 2 Done Done New false
21118 8105 Acked 9104 Flush 0 Done Done New false
21119 8106 Acked 9105 Read 5 Done Done New false
21120 8107 Acked 9106 Write 9 Done Done New false
21121 8108 Acked 9107 Write 9 Done Done New false
21122 8109 Acked 9108 Write 4 Done Done New false
21123 8110 Acked 9109 Write 10 Done Done New false
21124 8111 Acked 9110 Write 8 Done Done New false
21125 8112 Acked 9111 Write 9 Done Done New false
21126 8113 Acked 9112 Read 7 Done Done New false
21127 8114 Acked 9113 Write 8 Done Done New false
21128 8115 Acked 9114 Read 3 Done Done New false
21129 8116 Acked 9115 Write 2 Done Done New false
21130 8117 Acked 9116 Write 6 Done Done New false
21131 8118 Acked 9117 Read 10 Done Done New false
21132 8119 Acked 9118 Write 2 Done Done New false
21133 8120 Acked 9119 Read 7 Done Done New false
21134 8121 Acked 9120 Read 4 Done Done New false
21135 8122 Acked 9121 Write 10 Done Done New false
21136 8123 Acked 9122 Write 2 Done Done New false
21137 8124 Acked 9123 Read 10 Done Done New false
21138 8125 Acked 9124 Write 10 Done Done New false
21139 8126 Acked 9125 Read 4 Done Done New false
21140 8127 Acked 9126 Flush 0 Done Done New false
21141 8128 Acked 9127 Read 5 Done Done New false
21142 8129 Acked 9128 Read 6 Done Done New false
21143 8130 Acked 9129 Read 4 Done Done New false
21144 8131 Acked 9130 Read 10 Done Done New false
21145 8132 Acked 9131 Read 10 Done Done New false
21146 8133 Acked 9132 Write 10 Done Done New false
21147 8134 Acked 9133 Write 3 Done Done New false
21148 8135 Acked 9134 Flush 0 Done Done New false
21149 8136 Acked 9135 Write 1 Done Done New false
21150 8137 Acked 9136 Write 4 Done Done New false
21151 8138 Acked 9137 Write 5 Done Done New false
21152 8139 Acked 9138 Write 5 Done Done New false
21153 8140 Acked 9139 Write 7 Done Done New false
21154 8141 Acked 9140 Read 4 Done Done New false
21155 8142 Acked 9141 Read 9 Done Done New false
21156 8143 Acked 9142 Write 5 Done Done New false
21157 8144 Acked 9143 Write 1 Done Done New false
21158 8145 Acked 9144 Read 4 Done Done New false
21159 8146 Acked 9145 Read 8 Done Done New false
21160 8147 Acked 9146 Write 8 Done Done New false
21161 8148 Acked 9147 Write 7 Done Done New false
21162 8149 Acked 9148 Write 6 Done Done New false
21163 8150 Acked 9149 Write 9 Done Done New false
21164 8151 Acked 9150 Flush 0 Done Done New false
21165 8152 Acked 9151 Read 10 Done Done New false
21166 8153 Acked 9152 Write 7 Done Done New false
21167 8154 Acked 9153 Write 2 Done Done New false
21168 8155 Acked 9154 Write 7 Done Done New false
21169 8156 Acked 9155 Write 7 Done Done New false
21170 8157 Acked 9156 Write 6 Done Done New false
21171 8158 Acked 9157 Read 6 Done Done New false
21172 8159 Acked 9158 Read 3 Done Done New false
21173 8160 Acked 9159 Write 1 Done Done New false
21174 8161 Acked 9160 Read 3 Done Done New false
21175 8162 Acked 9161 Read 5 Done Done New false
21176 8163 Acked 9162 Read 7 Done Done New false
21177 8164 Acked 9163 Write 5 Done Done New false
21178 8165 Acked 9164 Read 5 Done Done New false
21179 8166 Acked 9165 Write 1 Done Done New false
21180 8167 Acked 9166 Write 9 Done Done New false
21181 8168 Acked 9167 Read 7 Done Done New false
21182 8169 Acked 9168 Read 10 Done Done New false
21183 8170 Acked 9169 Read 7 Done Done New false
21184 8171 Acked 9170 Write 1 Done Done New false
21185 8172 Acked 9171 Write 8 Done Done New false
21186 8173 Acked 9172 Flush 0 Done Done New false
21187 STATES DS:0 DS:1 DS:2 TOTAL
21188 New 0 0 303 303
21189 Sent 0 0 0 0
21190 Done 303 303 0 606
21191 Skipped 0 0 0 0
21192 Error 0 0 0 0
21193 Last Flush: 9172 9172 8869
21194 Downstairs last five completed: 8869 8868 8867 8866 8865 8864
21195 Upstairs last five completed: 8173 8172 8171 8170 8169 8168
21196 CLIENT: Up:0 ds:303 act:2
211972023-10-13T18:13:41.526ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
21198 ----------------------------------------------------------------
21199 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
21200 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21201 7871 Acked 8870 Flush 0 Done Done New false
21202 7872 Acked 8871 Write 4 Done Done New false
21203 7873 Acked 8872 Write 3 Done Done New false
21204 7874 Acked 8873 Read 2 Done Done New false
21205 7875 Acked 8874 Write 6 Done Done New false
21206 7876 Acked 8875 Read 8 Done Done New false
21207 7877 Acked 8876 Read 4 Done Done New false
21208 7878 Acked 8877 Read 8 Done Done New false
21209 7879 Acked 8878 Write 3 Done Done New false
21210 7880 Acked 8879 Read 8 Done Done New false
21211 7881 Acked 8880 Write 8 Done Done New false
21212 7882 Acked 8881 Read 3 Done Done New false
21213 7883 Acked 8882 Write 7 Done Done New false
21214 7884 Acked 8883 Read 7 Done Done New false
21215 7885 Acked 8884 Write 4 Done Done New false
21216 7886 Acked 8885 Write 4 Done Done New false
21217 7887 Acked 8886 Read 2 Done Done New false
21218 7888 Acked 8887 Read 5 Done Done New false
21219 7889 Acked 8888 Read 7 Done Done New false
21220 7890 Acked 8889 Write 10 Done Done New false
21221 7891 Acked 8890 Read 7 Done Done New false
21222 7892 Acked 8891 Read 3 Done Done New false
21223 7893 Acked 8892 Write 3 Done Done New false
21224 7894 Acked 8893 Write 2 Done Done New false
21225 7895 Acked 8894 Write 7 Done Done New false
21226 7896 Acked 8895 Write 2 Done Done New false
21227 7897 Acked 8896 Write 2 Done Done New false
21228 7898 Acked 8897 Read 10 Done Done New false
21229 7899 Acked 8898 Read 2 Done Done New false
21230 7900 Acked 8899 Write 4 Done Done New false
21231 7901 Acked 8900 Read 4 Done Done New false
21232 7902 Acked 8901 Read 5 Done Done New false
21233 7903 Acked 8902 Read 9 Done Done New false
21234 7904 Acked 8903 Flush 0 Done Done New false
21235 7905 Acked 8904 Read 10 Done Done New false
21236 7906 Acked 8905 Read 7 Done Done New false
21237 7907 Acked 8906 Flush 0 Done Done New false
21238 7908 Acked 8907 Flush 0 Done Done New false
21239 7909 Acked 8908 Read 1 Done Done New false
21240 7910 Acked 8909 Write 3 Done Done New false
21241 7911 Acked 8910 Read 10 Done Done New false
21242 7912 Acked 8911 Flush 0 Done Done New false
21243 7913 Acked 8912 Write 1 Done Done New false
21244 7914 Acked 8913 Read 9 Done Done New false
21245 7915 Acked 8914 Read 6 Done Done New false
21246 7916 Acked 8915 Write 8 Done Done New false
21247 7917 Acked 8916 Read 10 Done Done New false
21248 7918 Acked 8917 Flush 0 Done Done New false
21249 7919 Acked 8918 Write 4 Done Done New false
21250 7920 Acked 8919 Write 2 Done Done New false
21251 7921 Acked 8920 Read 7 Done Done New false
21252 7922 Acked 8921 Read 8 Done Done New false
21253 7923 Acked 8922 Flush 0 Done Done New false
21254 7924 Acked 8923 Flush 0 Done Done New false
21255 7925 Acked 8924 Read 3 Done Done New false
21256 7926 Acked 8925 Write 10 Done Done New false
21257 7927 Acked 8926 Read 8 Done Done New false
21258 7928 Acked 8927 Read 1 Done Done New false
21259 7929 Acked 8928 Write 2 Done Done New false
21260 7930 Acked 8929 Write 8 Done Done New false
21261 7931 Acked 8930 Read 10 Done Done New false
21262 7932 Acked 8931 Flush 0 Done Done New false
21263 7933 Acked 8932 Read 10 Done Done New false
21264 7934 Acked 8933 Flush 0 Done Done New false
21265 7935 Acked 8934 Read 1 Done Done New false
21266 7936 Acked 8935 Write 5 Done Done New false
21267 7937 Acked 8936 Read 7 Done Done New false
21268 7938 Acked 8937 Write 8 Done Done New false
21269 7939 Acked 8938 Read 6 Done Done New false
21270 7940 Acked 8939 Write 9 Done Done New false
21271 7941 Acked 8940 Read 6 Done Done New false
21272 7942 Acked 8941 Write 2 Done Done New false
21273 7943 Acked 8942 Read 4 Done Done New false
21274 7944 Acked 8943 Read 9 Done Done New false
21275 7945 Acked 8944 Read 3 Done Done New false
21276 7946 Acked 8945 Read 4 Done Done New false
21277 7947 Acked 8946 Write 8 Done Done New false
21278 7948 Acked 8947 Read 6 Done Done New false
21279 7949 Acked 8948 Write 1 Done Done New false
21280 7950 Acked 8949 Read 9 Done Done New false
21281 7951 Acked 8950 Read 8 Done Done New false
21282 7952 Acked 8951 Read 5 Done Done New false
21283 7953 Acked 8952 Flush 0 Done Done New false
21284 7954 Acked 8953 Read 6 Done Done New false
21285 7955 Acked 8954 Read 1 Done Done New false
21286 7956 Acked 8955 Read 6 Done Done New false
21287 7957 Acked 8956 Read 6 Done Done New false
21288 7958 Acked 8957 Read 9 Done Done New false
21289 7959 Acked 8958 Write 6 Done Done New false
21290 7960 Acked 8959 Read 6 Done Done New false
21291 7961 Acked 8960 Write 3 Done Done New false
21292 7962 Acked 8961 Read 5 Done Done New false
21293 7963 Acked 8962 Flush 0 Done Done New false
21294 7964 Acked 8963 Write 1 Done Done New false
21295 7965 Acked 8964 Write 1 Done Done New false
21296 7966 Acked 8965 Read 1 Done Done New false
21297 7967 Acked 8966 Write 1 Done Done New false
21298 7968 Acked 8967 Read 2 Done Done New false
21299 7969 Acked 8968 Write 6 Done Done New false
21300 7970 Acked 8969 Write 9 Done Done New false
21301 7971 Acked 8970 Flush 0 Done Done New false
21302 7972 Acked 8971 Read 1 Done Done New false
21303 7973 Acked 8972 Read 9 Done Done New false
21304 7974 Acked 8973 Write 6 Done Done New false
21305 7975 Acked 8974 Read 1 Done Done New false
21306 7976 Acked 8975 Write 6 Done Done New false
21307 7977 Acked 8976 Read 1 Done Done New false
21308 7978 Acked 8977 Read 4 Done Done New false
21309 7979 Acked 8978 Write 4 Done Done New false
21310 7980 Acked 8979 Write 10 Done Done New false
21311 7981 Acked 8980 Read 10 Done Done New false
21312 7982 Acked 8981 Write 7 Done Done New false
21313 7983 Acked 8982 Write 7 Done Done New false
21314 7984 Acked 8983 Write 5 Done Done New false
21315 7985 Acked 8984 Read 2 Done Done New false
21316 7986 Acked 8985 Read 8 Done Done New false
21317 7987 Acked 8986 Write 9 Done Done New false
21318 7988 Acked 8987 Write 6 Done Done New false
21319 7989 Acked 8988 Write 8 Done Done New false
21320 7990 Acked 8989 Write 1 Done Done New false
21321 7991 Acked 8990 Write 4 Done Done New false
21322 7992 Acked 8991 Write 1 Done Done New false
21323 7993 Acked 8992 Write 8 Done Done New false
21324 7994 Acked 8993 Read 8 Done Done New false
21325 7995 Acked 8994 Flush 0 Done Done New false
21326 7996 Acked 8995 Read 6 Done Done New false
21327 7997 Acked 8996 Read 9 Done Done New false
21328 7998 Acked 8997 Write 10 Done Done New false
21329 7999 Acked 8998 Read 3 Done Done New false
21330 8000 Acked 8999 Read 3 Done Done New false
21331 8001 Acked 9000 Write 9 Done Done New false
21332 8002 Acked 9001 Read 6 Done Done New false
21333 8003 Acked 9002 Read 7 Done Done New false
21334 8004 Acked 9003 Write 8 Done Done New false
21335 8005 Acked 9004 Read 6 Done Done New false
21336 8006 Acked 9005 Read 6 Done Done New false
21337 8007 Acked 9006 Read 6 Done Done New false
21338 8008 Acked 9007 Write 9 Done Done New false
21339 8009 Acked 9008 Read 3 Done Done New false
21340 8010 Acked 9009 Flush 0 Done Done New false
21341 8011 Acked 9010 Write 10 Done Done New false
21342 8012 Acked 9011 Read 3 Done Done New false
21343 8013 Acked 9012 Read 3 Done Done New false
21344 8014 Acked 9013 Write 2 Done Done New false
21345 8015 Acked 9014 Read 1 Done Done New false
21346 8016 Acked 9015 Read 2 Done Done New false
21347 8017 Acked 9016 Read 1 Done Done New false
21348 8018 Acked 9017 Read 7 Done Done New false
21349 8019 Acked 9018 Write 3 Done Done New false
21350 8020 Acked 9019 Write 5 Done Done New false
21351 8021 Acked 9020 Write 9 Done Done New false
21352 8022 Acked 9021 Write 6 Done Done New false
21353 8023 Acked 9022 Read 7 Done Done New false
21354 8024 Acked 9023 Read 9 Done Done New false
21355 8025 Acked 9024 Read 5 Done Done New false
21356 8026 Acked 9025 Read 2 Done Done New false
21357 8027 Acked 9026 Read 3 Done Done New false
21358 8028 Acked 9027 Write 4 Done Done New false
21359 8029 Acked 9028 Write 5 Done Done New false
21360 8030 Acked 9029 Read 7 Done Done New false
21361 8031 Acked 9030 Read 6 Done Done New false
21362 8032 Acked 9031 Read 6 Done Done New false
21363 8033 Acked 9032 Write 3 Done Done New false
21364 8034 Acked 9033 Read 4 Done Done New false
21365 8035 Acked 9034 Read 2 Done Done New false
21366 8036 Acked 9035 Write 10 Done Done New false
21367 8037 Acked 9036 Flush 0 Done Done New false
21368 8038 Acked 9037 Read 5 Done Done New false
21369 8039 Acked 9038 Flush 0 Done Done New false
21370 8040 Acked 9039 Write 7 Done Done New false
21371 8041 Acked 9040 Read 10 Done Done New false
21372 8042 Acked 9041 Read 1 Done Done New false
21373 8043 Acked 9042 Write 2 Done Done New false
21374 8044 Acked 9043 Flush 0 Done Done New false
21375 8045 Acked 9044 Write 1 Done Done New false
21376 8046 Acked 9045 Write 10 Done Done New false
21377 8047 Acked 9046 Write 10 Done Done New false
21378 8048 Acked 9047 Write 2 Done Done New false
21379 8049 Acked 9048 Flush 0 Done Done New false
21380 8050 Acked 9049 Read 3 Done Done New false
21381 8051 Acked 9050 Read 1 Done Done New false
21382 8052 Acked 9051 Flush 0 Done Done New false
21383 8053 Acked 9052 Write 2 Done Done New false
21384 8054 Acked 9053 Write 3 Done Done New false
21385 8055 Acked 9054 Read 7 Done Done New false
21386 8056 Acked 9055 Write 8 Done Done New false
21387 8057 Acked 9056 Read 7 Done Done New false
21388 8058 Acked 9057 Write 3 Done Done New false
21389 8059 Acked 9058 Flush 0 Done Done New false
21390 8060 Acked 9059 Read 10 Done Done New false
21391 8061 Acked 9060 Write 6 Done Done New false
21392 8062 Acked 9061 Read 4 Done Done New false
21393 8063 Acked 9062 Write 3 Done Done New false
21394 8064 Acked 9063 Read 3 Done Done New false
21395 8065 Acked 9064 Read 10 Done Done New false
21396 8066 Acked 9065 Write 1 Done Done New false
21397 8067 Acked 9066 Read 9 Done Done New false
21398 8068 Acked 9067 Flush 0 Done Done New false
21399 8069 Acked 9068 Read 6 Done Done New false
21400 8070 Acked 9069 Read 4 Done Done New false
21401 8071 Acked 9070 Flush 0 Done Done New false
21402 8072 Acked 9071 Write 4 Done Done New false
21403 8073 Acked 9072 Read 4 Done Done New false
21404 8074 Acked 9073 Read 8 Done Done New false
21405 8075 Acked 9074 Write 7 Done Done New false
21406 8076 Acked 9075 Write 1 Done Done New false
21407 8077 Acked 9076 Write 6 Done Done New false
21408 8078 Acked 9077 Read 1 Done Done New false
21409 8079 Acked 9078 Read 4 Done Done New false
21410 8080 Acked 9079 Write 2 Done Done New false
21411 8081 Acked 9080 Write 5 Done Done New false
21412 8082 Acked 9081 Read 5 Done Done New false
21413 8083 Acked 9082 Write 5 Done Done New false
21414 8084 Acked 9083 Write 4 Done Done New false
21415 8085 Acked 9084 Read 7 Done Done New false
21416 8086 Acked 9085 Read 1 Done Done New false
21417 8087 Acked 9086 Write 8 Done Done New false
21418 8088 Acked 9087 Read 5 Done Done New false
21419 8089 Acked 9088 Read 7 Done Done New false
21420 8090 Acked 9089 Read 8 Done Done New false
21421 8091 Acked 9090 Read 9 Done Done New false
21422 8092 Acked 9091 Flush 0 Done Done New false
21423 8093 Acked 9092 Read 2 Done Done New false
21424 8094 Acked 9093 Read 7 Done Done New false
21425 8095 Acked 9094 Read 4 Done Done New false
21426 8096 Acked 9095 Write 10 Done Done New false
21427 8097 Acked 9096 Flush 0 Done Done New false
21428 8098 Acked 9097 Write 5 Done Done New false
21429 8099 Acked 9098 Read 10 Done Done New false
21430 8100 Acked 9099 Read 10 Done Done New false
21431 8101 Acked 9100 Read 9 Done Done New false
21432 8102 Acked 9101 Write 8 Done Done New false
21433 8103 Acked 9102 Flush 0 Done Done New false
21434 8104 Acked 9103 Write 2 Done Done New false
21435 8105 Acked 9104 Flush 0 Done Done New false
21436 8106 Acked 9105 Read 5 Done Done New false
21437 8107 Acked 9106 Write 9 Done Done New false
21438 8108 Acked 9107 Write 9 Done Done New false
21439 8109 Acked 9108 Write 4 Done Done New false
21440 8110 Acked 9109 Write 10 Done Done New false
21441 8111 Acked 9110 Write 8 Done Done New false
21442 8112 Acked 9111 Write 9 Done Done New false
21443 8113 Acked 9112 Read 7 Done Done New false
21444 8114 Acked 9113 Write 8 Done Done New false
21445 8115 Acked 9114 Read 3 Done Done New false
21446 8116 Acked 9115 Write 2 Done Done New false
21447 8117 Acked 9116 Write 6 Done Done New false
21448 8118 Acked 9117 Read 10 Done Done New false
21449 8119 Acked 9118 Write 2 Done Done New false
21450 8120 Acked 9119 Read 7 Done Done New false
21451 8121 Acked 9120 Read 4 Done Done New false
21452 8122 Acked 9121 Write 10 Done Done New false
21453 8123 Acked 9122 Write 2 Done Done New false
21454 8124 Acked 9123 Read 10 Done Done New false
21455 8125 Acked 9124 Write 10 Done Done New false
21456 8126 Acked 9125 Read 4 Done Done New false
21457 8127 Acked 9126 Flush 0 Done Done New false
21458 8128 Acked 9127 Read 5 Done Done New false
21459 8129 Acked 9128 Read 6 Done Done New false
21460 8130 Acked 9129 Read 4 Done Done New false
21461 8131 Acked 9130 Read 10 Done Done New false
21462 8132 Acked 9131 Read 10 Done Done New false
21463 8133 Acked 9132 Write 10 Done Done New false
21464 8134 Acked 9133 Write 3 Done Done New false
21465 8135 Acked 9134 Flush 0 Done Done New false
21466 8136 Acked 9135 Write 1 Done Done New false
21467 8137 Acked 9136 Write 4 Done Done New false
21468 8138 Acked 9137 Write 5 Done Done New false
21469 8139 Acked 9138 Write 5 Done Done New false
21470 8140 Acked 9139 Write 7 Done Done New false
21471 8141 Acked 9140 Read 4 Done Done New false
21472 8142 Acked 9141 Read 9 Done Done New false
21473 8143 Acked 9142 Write 5 Done Done New false
21474 8144 Acked 9143 Write 1 Done Done New false
21475 8145 Acked 9144 Read 4 Done Done New false
21476 8146 Acked 9145 Read 8 Done Done New false
21477 8147 Acked 9146 Write 8 Done Done New false
21478 8148 Acked 9147 Write 7 Done Done New false
21479 8149 Acked 9148 Write 6 Done Done New false
21480 8150 Acked 9149 Write 9 Done Done New false
21481 8151 Acked 9150 Flush 0 Done Done New false
21482 8152 Acked 9151 Read 10 Done Done New false
21483 8153 Acked 9152 Write 7 Done Done New false
21484 8154 Acked 9153 Write 2 Done Done New false
21485 8155 Acked 9154 Write 7 Done Done New false
21486 8156 Acked 9155 Write 7 Done Done New false
21487 8157 Acked 9156 Write 6 Done Done New false
21488 8158 Acked 9157 Read 6 Done Done New false
21489 8159 Acked 9158 Read 3 Done Done New false
21490 8160 Acked 9159 Write 1 Done Done New false
21491 8161 Acked 9160 Read 3 Done Done New false
21492 8162 Acked 9161 Read 5 Done Done New false
21493 8163 Acked 9162 Read 7 Done Done New false
21494 8164 Acked 9163 Write 5 Done Done New false
21495 8165 Acked 9164 Read 5 Done Done New false
21496 8166 Acked 9165 Write 1 Done Done New false
21497 8167 Acked 9166 Write 9 Done Done New false
21498 8168 Acked 9167 Read 7 Done Done New false
21499 8169 Acked 9168 Read 10 Done Done New false
21500 8170 Acked 9169 Read 7 Done Done New false
21501 8171 Acked 9170 Write 1 Done Done New false
21502 8172 Acked 9171 Write 8 Done Done New false
21503 8173 Acked 9172 Flush 0 Done Done New false
21504 STATES DS:0 DS:1 DS:2 TOTAL
21505 New 0 0 303 303
21506 Sent 0 0 0 0
21507 Done 303 303 0 606
21508 Skipped 0 0 0 0
21509 Error 0 0 0 0
21510 Last Flush: 9172 9172 8869
21511 Downstairs last five completed: 8869 8868 8867 8866 8865 8864
21512 Upstairs last five completed: 8173 8172 8171 8170 8169 8168
21513 CLIENT: Up:0 ds:303 act:2
21514 ----------------------------------------------------------------
21515 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
21516 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21517 7871 Acked 8870 Flush 0 Done Done New false
21518 7872 Acked 8871 Write 4 Done Done New false
21519 7873 Acked 8872 Write 3 Done Done New false
21520 7874 Acked 8873 Read 2 Done Done New false
21521 7875 Acked 8874 Write 6 Done Done New false
21522 7876 Acked 8875 Read 8 Done Done New false
21523 7877 Acked 8876 Read 4 Done Done New false
21524 7878 Acked 8877 Read 8 Done Done New false
21525 7879 Acked 8878 Write 3 Done Done New false
21526 7880 Acked 8879 Read 8 Done Done New false
21527 7881 Acked 8880 Write 8 Done Done New false
21528 7882 Acked 8881 Read 3 Done Done New false
21529 7883 Acked 8882 Write 7 Done Done New false
21530 7884 Acked 8883 Read 7 Done Done New false
21531 7885 Acked 8884 Write 4 Done Done New false
21532 7886 Acked 8885 Write 4 Done Done New false
21533 7887 Acked 8886 Read 2 Done Done New false
21534 7888 Acked 8887 Read 5 Done Done New false
21535 7889 Acked 8888 Read 7 Done Done New false
21536 7890 Acked 8889 Write 10 Done Done New false
21537 7891 Acked 8890 Read 7 Done Done New false
21538 7892 Acked 8891 Read 3 Done Done New false
21539 7893 Acked 8892 Write 3 Done Done New false
21540 7894 Acked 8893 Write 2 Done Done New false
21541 7895 Acked 8894 Write 7 Done Done New false
21542 7896 Acked 8895 Write 2 Done Done New false
21543 7897 Acked 8896 Write 2 Done Done New false
21544 7898 Acked 8897 Read 10 Done Done New false
21545 7899 Acked 8898 Read 2 Done Done New false
21546 7900 Acked 8899 Write 4 Done Done New false
21547 7901 Acked 8900 Read 4 Done Done New false
21548 7902 Acked 8901 Read 5 Done Done New false
21549 7903 Acked 8902 Read 9 Done Done New false
21550 7904 Acked 8903 Flush 0 Done Done New false
21551 7905 Acked 8904 Read 10 Done Done New false
21552 7906 Acked 8905 Read 7 Done Done New false
21553 7907 Acked 8906 Flush 0 Done Done New false
21554 7908 Acked 8907 Flush 0 Done Done New false
21555 7909 Acked 8908 Read 1 Done Done New false
21556 7910 Acked 8909 Write 3 Done Done New false
21557 7911 Acked 8910 Read 10 Done Done New false
21558 7912 Acked 8911 Flush 0 Done Done New false
21559 7913 Acked 8912 Write 1 Done Done New false
21560 7914 Acked 8913 Read 9 Done Done New false
21561 7915 Acked 8914 Read 6 Done Done New false
21562 7916 Acked 8915 Write 8 Done Done New false
21563 7917 Acked 8916 Read 10 Done Done New false
21564 7918 Acked 8917 Flush 0 Done Done New false
21565 7919 Acked 8918 Write 4 Done Done New false
21566 7920 Acked 8919 Write 2 Done Done New false
21567 7921 Acked 8920 Read 7 Done Done New false
21568 7922 Acked 8921 Read 8 Done Done New false
21569 7923 Acked 8922 Flush 0 Done Done New false
21570 7924 Acked 8923 Flush 0 Done Done New false
21571 7925 Acked 8924 Read 3 Done Done New false
21572 7926 Acked 8925 Write 10 Done Done New false
21573 7927 Acked 8926 Read 8 Done Done New false
21574 7928 Acked 8927 Read 1 Done Done New false
21575 7929 Acked 8928 Write 2 Done Done New false
21576 7930 Acked 8929 Write 8 Done Done New false
21577 7931 Acked 8930 Read 10 Done Done New false
21578 7932 Acked 8931 Flush 0 Done Done New false
21579 7933 Acked 8932 Read 10 Done Done New false
21580 7934 Acked 8933 Flush 0 Done Done New false
21581 7935 Acked 8934 Read 1 Done Done New false
21582 7936 Acked 8935 Write 5 Done Done New false
21583 7937 Acked 8936 Read 7 Done Done New false
21584 7938 Acked 8937 Write 8 Done Done New false
21585 7939 Acked 8938 Read 6 Done Done New false
21586 7940 Acked 8939 Write 9 Done Done New false
21587 7941 Acked 8940 Read 6 Done Done New false
21588 7942 Acked 8941 Write 2 Done Done New false
21589 7943 Acked 8942 Read 4 Done Done New false
21590 7944 Acked 8943 Read 9 Done Done New false
21591 7945 Acked 8944 Read 3 Done Done New false
21592 7946 Acked 8945 Read 4 Done Done New false
21593 7947 Acked 8946 Write 8 Done Done New false
21594 7948 Acked 8947 Read 6 Done Done New false
21595 7949 Acked 8948 Write 1 Done Done New false
21596 7950 Acked 8949 Read 9 Done Done New false
21597 7951 Acked 8950 Read 8 Done Done New false
21598 7952 Acked 8951 Read 5 Done Done New false
21599 7953 Acked 8952 Flush 0 Done Done New false
21600 7954 Acked 8953 Read 6 Done Done New false
21601 7955 Acked 8954 Read 1 Done Done New false
21602 7956 Acked 8955 Read 6 Done Done New false
21603 7957 Acked 8956 Read 6 Done Done New false
21604 7958 Acked 8957 Read 9 Done Done New false
21605 7959 Acked 8958 Write 6 Done Done New false
21606 7960 Acked 8959 Read 6 Done Done New false
21607 7961 Acked 8960 Write 3 Done Done New false
21608 7962 Acked 8961 Read 5 Done Done New false
21609 7963 Acked 8962 Flush 0 Done Done New false
21610 7964 Acked 8963 Write 1 Done Done New false
21611 7965 Acked 8964 Write 1 Done Done New false
21612 7966 Acked 8965 Read 1 Done Done New false
21613 7967 Acked 8966 Write 1 Done Done New false
21614 7968 Acked 8967 Read 2 Done Done New false
21615 7969 Acked 8968 Write 6 Done Done New false
21616 7970 Acked 8969 Write 9 Done Done New false
21617 7971 Acked 8970 Flush 0 Done Done New false
21618 7972 Acked 8971 Read 1 Done Done New false
21619 7973 Acked 8972 Read 9 Done Done New false
21620 7974 Acked 8973 Write 6 Done Done New false
21621 7975 Acked 8974 Read 1 Done Done New false
21622 7976 Acked 8975 Write 6 Done Done New false
21623 7977 Acked 8976 Read 1 Done Done New false
21624 7978 Acked 8977 Read 4 Done Done New false
21625 7979 Acked 8978 Write 4 Done Done New false
21626 7980 Acked 8979 Write 10 Done Done New false
21627 7981 Acked 8980 Read 10 Done Done New false
21628 7982 Acked 8981 Write 7 Done Done New false
21629 7983 Acked 8982 Write 7 Done Done New false
21630 7984 Acked 8983 Write 5 Done Done New false
21631 7985 Acked 8984 Read 2 Done Done New false
21632 7986 Acked 8985 Read 8 Done Done New false
21633 7987 Acked 8986 Write 9 Done Done New false
21634 7988 Acked 8987 Write 6 Done Done New false
21635 7989 Acked 8988 Write 8 Done Done New false
21636 7990 Acked 8989 Write 1 Done Done New false
21637 7991 Acked 8990 Write 4 Done Done New false
21638 7992 Acked 8991 Write 1 Done Done New false
21639 7993 Acked 8992 Write 8 Done Done New false
21640 7994 Acked 8993 Read 8 Done Done New false
21641 7995 Acked 8994 Flush 0 Done Done New false
21642 7996 Acked 8995 Read 6 Done Done New false
21643 7997 Acked 8996 Read 9 Done Done New false
21644 7998 Acked 8997 Write 10 Done Done New false
21645 7999 Acked 8998 Read 3 Done Done New false
21646 8000 Acked 8999 Read 3 Done Done New false
21647 8001 Acked 9000 Write 9 Done Done New false
21648 8002 Acked 9001 Read 6 Done Done New false
21649 8003 Acked 9002 Read 7 Done Done New false
21650 8004 Acked 9003 Write 8 Done Done New false
21651 8005 Acked 9004 Read 6 Done Done New false
21652 8006 Acked 9005 Read 6 Done Done New false
21653 8007 Acked 9006 Read 6 Done Done New false
21654 8008 Acked 9007 Write 9 Done Done New false
21655 8009 Acked 9008 Read 3 Done Done New false
21656 8010 Acked 9009 Flush 0 Done Done New false
21657 8011 Acked 9010 Write 10 Done Done New false
21658 8012 Acked 9011 Read 3 Done Done New false
21659 8013 Acked 9012 Read 3 Done Done New false
21660 8014 Acked 9013 Write 2 Done Done New false
21661 8015 Acked 9014 Read 1 Done Done New false
21662 8016 Acked 9015 Read 2 Done Done New false
21663 8017 Acked 9016 Read 1 Done Done New false
21664 8018 Acked 9017 Read 7 Done Done New false
21665 8019 Acked 9018 Write 3 Done Done New false
21666 8020 Acked 9019 Write 5 Done Done New false
21667 8021 Acked 9020 Write 9 Done Done New false
21668 8022 Acked 9021 Write 6 Done Done New false
21669 8023 Acked 9022 Read 7 Done Done New false
21670 8024 Acked 9023 Read 9 Done Done New false
21671 8025 Acked 9024 Read 5 Done Done New false
21672 8026 Acked 9025 Read 2 Done Done New false
21673 8027 Acked 9026 Read 3 Done Done New false
21674 8028 Acked 9027 Write 4 Done Done New false
21675 8029 Acked 9028 Write 5 Done Done New false
21676 8030 Acked 9029 Read 7 Done Done New false
21677 8031 Acked 9030 Read 6 Done Done New false
21678 8032 Acked 9031 Read 6 Done Done New false
21679 8033 Acked 9032 Write 3 Done Done New false
21680 8034 Acked 9033 Read 4 Done Done New false
21681 8035 Acked 9034 Read 2 Done Done New false
21682 8036 Acked 9035 Write 10 Done Done New false
21683 8037 Acked 9036 Flush 0 Done Done New false
21684 8038 Acked 9037 Read 5 Done Done New false
21685 8039 Acked 9038 Flush 0 Done Done New false
21686 8040 Acked 9039 Write 7 Done Done New false
21687 8041 Acked 9040 Read 10 Done Done New false
21688 8042 Acked 9041 Read 1 Done Done New false
21689 8043 Acked 9042 Write 2 Done Done New false
21690 8044 Acked 9043 Flush 0 Done Done New false
21691 8045 Acked 9044 Write 1 Done Done New false
21692 8046 Acked 9045 Write 10 Done Done New false
21693 8047 Acked 9046 Write 10 Done Done New false
21694 8048 Acked 9047 Write 2 Done Done New false
21695 8049 Acked 9048 Flush 0 Done Done New false
21696 8050 Acked 9049 Read 3 Done Done New false
21697 8051 Acked 9050 Read 1 Done Done New false
21698 8052 Acked 9051 Flush 0 Done Done New false
21699 8053 Acked 9052 Write 2 Done Done New false
21700 8054 Acked 9053 Write 3 Done Done New false
21701 8055 Acked 9054 Read 7 Done Done New false
21702 8056 Acked 9055 Write 8 Done Done New false
21703 8057 Acked 9056 Read 7 Done Done New false
21704 8058 Acked 9057 Write 3 Done Done New false
21705 8059 Acked 9058 Flush 0 Done Done New false
21706 8060 Acked 9059 Read 10 Done Done New false
21707 8061 Acked 9060 Write 6 Done Done New false
21708 8062 Acked 9061 Read 4 Done Done New false
21709 8063 Acked 9062 Write 3 Done Done New false
21710 8064 Acked 9063 Read 3 Done Done New false
21711 8065 Acked 9064 Read 10 Done Done New false
21712 8066 Acked 9065 Write 1 Done Done New false
21713 8067 Acked 9066 Read 9 Done Done New false
21714 8068 Acked 9067 Flush 0 Done Done New false
21715 8069 Acked 9068 Read 6 Done Done New false
21716 8070 Acked 9069 Read 4 Done Done New false
21717 8071 Acked 9070 Flush 0 Done Done New false
21718 8072 Acked 9071 Write 4 Done Done New false
21719 8073 Acked 9072 Read 4 Done Done New false
21720 8074 Acked 9073 Read 8 Done Done New false
21721 8075 Acked 9074 Write 7 Done Done New false
21722 8076 Acked 9075 Write 1 Done Done New false
21723 8077 Acked 9076 Write 6 Done Done New false
21724 8078 Acked 9077 Read 1 Done Done New false
21725 8079 Acked 9078 Read 4 Done Done New false
21726 8080 Acked 9079 Write 2 Done Done New false
21727 8081 Acked 9080 Write 5 Done Done New false
21728 8082 Acked 9081 Read 5 Done Done New false
21729 8083 Acked 9082 Write 5 Done Done New false
21730 8084 Acked 9083 Write 4 Done Done New false
21731 8085 Acked 9084 Read 7 Done Done New false
21732 8086 Acked 9085 Read 1 Done Done New false
21733 8087 Acked 9086 Write 8 Done Done New false
21734 8088 Acked 9087 Read 5 Done Done New false
21735 8089 Acked 9088 Read 7 Done Done New false
21736 8090 Acked 9089 Read 8 Done Done New false
21737 8091 Acked 9090 Read 9 Done Done New false
21738 8092 Acked 9091 Flush 0 Done Done New false
21739 8093 Acked 9092 Read 2 Done Done New false
21740 8094 Acked 9093 Read 7 Done Done New false
21741 8095 Acked 9094 Read 4 Done Done New false
21742 8096 Acked 9095 Write 10 Done Done New false
21743 8097 Acked 9096 Flush 0 Done Done New false
21744 8098 Acked 9097 Write 5 Done Done New false
21745 8099 Acked 9098 Read 10 Done Done New false
21746 8100 Acked 9099 Read 10 Done Done New false
21747 8101 Acked 9100 Read 9 Done Done New false
21748 8102 Acked 9101 Write 8 Done Done New false
21749 8103 Acked 9102 Flush 0 Done Done New false
21750 8104 Acked 9103 Write 2 Done Done New false
21751 8105 Acked 9104 Flush 0 Done Done New false
21752 8106 Acked 9105 Read 5 Done Done New false
21753 8107 Acked 9106 Write 9 Done Done New false
21754 8108 Acked 9107 Write 9 Done Done New false
21755 8109 Acked 9108 Write 4 Done Done New false
21756 8110 Acked 9109 Write 10 Done Done New false
21757 8111 Acked 9110 Write 8 Done Done New false
21758 8112 Acked 9111 Write 9 Done Done New false
21759 8113 Acked 9112 Read 7 Done Done New false
21760 8114 Acked 9113 Write 8 Done Done New false
21761 8115 Acked 9114 Read 3 Done Done New false
21762 8116 Acked 9115 Write 2 Done Done New false
21763 8117 Acked 9116 Write 6 Done Done New false
21764 8118 Acked 9117 Read 10 Done Done New false
21765 8119 Acked 9118 Write 2 Done Done New false
21766 8120 Acked 9119 Read 7 Done Done New false
21767 8121 Acked 9120 Read 4 Done Done New false
21768 8122 Acked 9121 Write 10 Done Done New false
21769 8123 Acked 9122 Write 2 Done Done New false
21770 8124 Acked 9123 Read 10 Done Done New false
21771 8125 Acked 9124 Write 10 Done Done New false
21772 8126 Acked 9125 Read 4 Done Done New false
21773 8127 Acked 9126 Flush 0 Done Done New false
21774 8128 Acked 9127 Read 5 Done Done New false
21775 8129 Acked 9128 Read 6 Done Done New false
21776 8130 Acked 9129 Read 4 Done Done New false
21777 8131 Acked 9130 Read 10 Done Done New false
21778 8132 Acked 9131 Read 10 Done Done New false
21779 8133 Acked 9132 Write 10 Done Done New false
21780 8134 Acked 9133 Write 3 Done Done New false
21781 8135 Acked 9134 Flush 0 Done Done New false
21782 8136 Acked 9135 Write 1 Done Done New false
21783 8137 Acked 9136 Write 4 Done Done New false
21784 8138 Acked 9137 Write 5 Done Done New false
21785 8139 Acked 9138 Write 5 Done Done New false
21786 8140 Acked 9139 Write 7 Done Done New false
21787 8141 Acked 9140 Read 4 Done Done New false
21788 8142 Acked 9141 Read 9 Done Done New false
21789 8143 Acked 9142 Write 5 Done Done New false
21790 8144 Acked 9143 Write 1 Done Done New false
21791 8145 Acked 9144 Read 4 Done Done New false
21792 8146 Acked 9145 Read 8 Done Done New false
21793 8147 Acked 9146 Write 8 Done Done New false
21794 8148 Acked 9147 Write 7 Done Done New false
21795 8149 Acked 9148 Write 6 Done Done New false
21796 8150 Acked 9149 Write 9 Done Done New false
21797 8151 Acked 9150 Flush 0 Done Done New false
21798 8152 Acked 9151 Read 10 Done Done New false
21799 8153 Acked 9152 Write 7 Done Done New false
21800 8154 Acked 9153 Write 2 Done Done New false
21801 8155 Acked 9154 Write 7 Done Done New false
21802 8156 Acked 9155 Write 7 Done Done New false
21803 8157 Acked 9156 Write 6 Done Done New false
21804 8158 Acked 9157 Read 6 Done Done New false
21805 8159 Acked 9158 Read 3 Done Done New false
21806 8160 Acked 9159 Write 1 Done Done New false
21807 8161 Acked 9160 Read 3 Done Done New false
21808 8162 Acked 9161 Read 5 Done Done New false
21809 8163 Acked 9162 Read 7 Done Done New false
21810 8164 Acked 9163 Write 5 Done Done New false
21811 8165 Acked 9164 Read 5 Done Done New false
21812 8166 Acked 9165 Write 1 Done Done New false
21813 8167 Acked 9166 Write 9 Done Done New false
21814 8168 Acked 9167 Read 7 Done Done New false
21815 8169 Acked 9168 Read 10 Done Done New false
21816 8170 Acked 9169 Read 7 Done Done New false
21817 8171 Acked 9170 Write 1 Done Done New false
21818 8172 Acked 9171 Write 8 Done Done New false
21819 8173 Acked 9172 Flush 0 Done Done New false
21820 STATES DS:0 DS:1 DS:2 TOTAL
21821 New 0 0 303 303
21822 Sent 0 0 0 0
21823 Done 303 303 0 606
21824 Skipped 0 0 0 0
21825 Error 0 0 0 0
21826 Last Flush: 9172 9172 8869
21827 Downstairs last five completed: 8869 8868 8867 8866 8865 8864
21828 Upstairs last five completed: 8173 8172 8171 8170 8169 8168
21829 CLIENT: Up:0 ds:303 act:2
218302023-10-13T18:13:51.536ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
21831 ----------------------------------------------------------------
21832 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
21833 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21834 7871 Acked 8870 Flush 0 Done Done New false
21835 7872 Acked 8871 Write 4 Done Done New false
21836 7873 Acked 8872 Write 3 Done Done New false
21837 7874 Acked 8873 Read 2 Done Done New false
21838 7875 Acked 8874 Write 6 Done Done New false
21839 7876 Acked 8875 Read 8 Done Done New false
21840 7877 Acked 8876 Read 4 Done Done New false
21841 7878 Acked 8877 Read 8 Done Done New false
21842 7879 Acked 8878 Write 3 Done Done New false
21843 7880 Acked 8879 Read 8 Done Done New false
21844 7881 Acked 8880 Write 8 Done Done New false
21845 7882 Acked 8881 Read 3 Done Done New false
21846 7883 Acked 8882 Write 7 Done Done New false
21847 7884 Acked 8883 Read 7 Done Done New false
21848 7885 Acked 8884 Write 4 Done Done New false
21849 7886 Acked 8885 Write 4 Done Done New false
21850 7887 Acked 8886 Read 2 Done Done New false
21851 7888 Acked 8887 Read 5 Done Done New false
21852 7889 Acked 8888 Read 7 Done Done New false
21853 7890 Acked 8889 Write 10 Done Done New false
21854 7891 Acked 8890 Read 7 Done Done New false
21855 7892 Acked 8891 Read 3 Done Done New false
21856 7893 Acked 8892 Write 3 Done Done New false
21857 7894 Acked 8893 Write 2 Done Done New false
21858 7895 Acked 8894 Write 7 Done Done New false
21859 7896 Acked 8895 Write 2 Done Done New false
21860 7897 Acked 8896 Write 2 Done Done New false
21861 7898 Acked 8897 Read 10 Done Done New false
21862 7899 Acked 8898 Read 2 Done Done New false
21863 7900 Acked 8899 Write 4 Done Done New false
21864 7901 Acked 8900 Read 4 Done Done New false
21865 7902 Acked 8901 Read 5 Done Done New false
21866 7903 Acked 8902 Read 9 Done Done New false
21867 7904 Acked 8903 Flush 0 Done Done New false
21868 7905 Acked 8904 Read 10 Done Done New false
21869 7906 Acked 8905 Read 7 Done Done New false
21870 7907 Acked 8906 Flush 0 Done Done New false
21871 7908 Acked 8907 Flush 0 Done Done New false
21872 7909 Acked 8908 Read 1 Done Done New false
21873 7910 Acked 8909 Write 3 Done Done New false
21874 7911 Acked 8910 Read 10 Done Done New false
21875 7912 Acked 8911 Flush 0 Done Done New false
21876 7913 Acked 8912 Write 1 Done Done New false
21877 7914 Acked 8913 Read 9 Done Done New false
21878 7915 Acked 8914 Read 6 Done Done New false
21879 7916 Acked 8915 Write 8 Done Done New false
21880 7917 Acked 8916 Read 10 Done Done New false
21881 7918 Acked 8917 Flush 0 Done Done New false
21882 7919 Acked 8918 Write 4 Done Done New false
21883 7920 Acked 8919 Write 2 Done Done New false
21884 7921 Acked 8920 Read 7 Done Done New false
21885 7922 Acked 8921 Read 8 Done Done New false
21886 7923 Acked 8922 Flush 0 Done Done New false
21887 7924 Acked 8923 Flush 0 Done Done New false
21888 7925 Acked 8924 Read 3 Done Done New false
21889 7926 Acked 8925 Write 10 Done Done New false
21890 7927 Acked 8926 Read 8 Done Done New false
21891 7928 Acked 8927 Read 1 Done Done New false
21892 7929 Acked 8928 Write 2 Done Done New false
21893 7930 Acked 8929 Write 8 Done Done New false
21894 7931 Acked 8930 Read 10 Done Done New false
21895 7932 Acked 8931 Flush 0 Done Done New false
21896 7933 Acked 8932 Read 10 Done Done New false
21897 7934 Acked 8933 Flush 0 Done Done New false
21898 7935 Acked 8934 Read 1 Done Done New false
21899 7936 Acked 8935 Write 5 Done Done New false
21900 7937 Acked 8936 Read 7 Done Done New false
21901 7938 Acked 8937 Write 8 Done Done New false
21902 7939 Acked 8938 Read 6 Done Done New false
21903 7940 Acked 8939 Write 9 Done Done New false
21904 7941 Acked 8940 Read 6 Done Done New false
21905 7942 Acked 8941 Write 2 Done Done New false
21906 7943 Acked 8942 Read 4 Done Done New false
21907 7944 Acked 8943 Read 9 Done Done New false
21908 7945 Acked 8944 Read 3 Done Done New false
21909 7946 Acked 8945 Read 4 Done Done New false
21910 7947 Acked 8946 Write 8 Done Done New false
21911 7948 Acked 8947 Read 6 Done Done New false
21912 7949 Acked 8948 Write 1 Done Done New false
21913 7950 Acked 8949 Read 9 Done Done New false
21914 7951 Acked 8950 Read 8 Done Done New false
21915 7952 Acked 8951 Read 5 Done Done New false
21916 7953 Acked 8952 Flush 0 Done Done New false
21917 7954 Acked 8953 Read 6 Done Done New false
21918 7955 Acked 8954 Read 1 Done Done New false
21919 7956 Acked 8955 Read 6 Done Done New false
21920 7957 Acked 8956 Read 6 Done Done New false
21921 7958 Acked 8957 Read 9 Done Done New false
21922 7959 Acked 8958 Write 6 Done Done New false
21923 7960 Acked 8959 Read 6 Done Done New false
21924 7961 Acked 8960 Write 3 Done Done New false
21925 7962 Acked 8961 Read 5 Done Done New false
21926 7963 Acked 8962 Flush 0 Done Done New false
21927 7964 Acked 8963 Write 1 Done Done New false
21928 7965 Acked 8964 Write 1 Done Done New false
21929 7966 Acked 8965 Read 1 Done Done New false
21930 7967 Acked 8966 Write 1 Done Done New false
21931 7968 Acked 8967 Read 2 Done Done New false
21932 7969 Acked 8968 Write 6 Done Done New false
21933 7970 Acked 8969 Write 9 Done Done New false
21934 7971 Acked 8970 Flush 0 Done Done New false
21935 7972 Acked 8971 Read 1 Done Done New false
21936 7973 Acked 8972 Read 9 Done Done New false
21937 7974 Acked 8973 Write 6 Done Done New false
21938 7975 Acked 8974 Read 1 Done Done New false
21939 7976 Acked 8975 Write 6 Done Done New false
21940 7977 Acked 8976 Read 1 Done Done New false
21941 7978 Acked 8977 Read 4 Done Done New false
21942 7979 Acked 8978 Write 4 Done Done New false
21943 7980 Acked 8979 Write 10 Done Done New false
21944 7981 Acked 8980 Read 10 Done Done New false
21945 7982 Acked 8981 Write 7 Done Done New false
21946 7983 Acked 8982 Write 7 Done Done New false
21947 7984 Acked 8983 Write 5 Done Done New false
21948 7985 Acked 8984 Read 2 Done Done New false
21949 7986 Acked 8985 Read 8 Done Done New false
21950 7987 Acked 8986 Write 9 Done Done New false
21951 7988 Acked 8987 Write 6 Done Done New false
21952 7989 Acked 8988 Write 8 Done Done New false
21953 7990 Acked 8989 Write 1 Done Done New false
21954 7991 Acked 8990 Write 4 Done Done New false
21955 7992 Acked 8991 Write 1 Done Done New false
21956 7993 Acked 8992 Write 8 Done Done New false
21957 7994 Acked 8993 Read 8 Done Done New false
21958 7995 Acked 8994 Flush 0 Done Done New false
21959 7996 Acked 8995 Read 6 Done Done New false
21960 7997 Acked 8996 Read 9 Done Done New false
21961 7998 Acked 8997 Write 10 Done Done New false
21962 7999 Acked 8998 Read 3 Done Done New false
21963 8000 Acked 8999 Read 3 Done Done New false
21964 8001 Acked 9000 Write 9 Done Done New false
21965 8002 Acked 9001 Read 6 Done Done New false
21966 8003 Acked 9002 Read 7 Done Done New false
21967 8004 Acked 9003 Write 8 Done Done New false
21968 8005 Acked 9004 Read 6 Done Done New false
21969 8006 Acked 9005 Read 6 Done Done New false
21970 8007 Acked 9006 Read 6 Done Done New false
21971 8008 Acked 9007 Write 9 Done Done New false
21972 8009 Acked 9008 Read 3 Done Done New false
21973 8010 Acked 9009 Flush 0 Done Done New false
21974 8011 Acked 9010 Write 10 Done Done New false
21975 8012 Acked 9011 Read 3 Done Done New false
21976 8013 Acked 9012 Read 3 Done Done New false
21977 8014 Acked 9013 Write 2 Done Done New false
21978 8015 Acked 9014 Read 1 Done Done New false
21979 8016 Acked 9015 Read 2 Done Done New false
21980 8017 Acked 9016 Read 1 Done Done New false
21981 8018 Acked 9017 Read 7 Done Done New false
21982 8019 Acked 9018 Write 3 Done Done New false
21983 8020 Acked 9019 Write 5 Done Done New false
21984 8021 Acked 9020 Write 9 Done Done New false
21985 8022 Acked 9021 Write 6 Done Done New false
21986 8023 Acked 9022 Read 7 Done Done New false
21987 8024 Acked 9023 Read 9 Done Done New false
21988 8025 Acked 9024 Read 5 Done Done New false
21989 8026 Acked 9025 Read 2 Done Done New false
21990 8027 Acked 9026 Read 3 Done Done New false
21991 8028 Acked 9027 Write 4 Done Done New false
21992 8029 Acked 9028 Write 5 Done Done New false
21993 8030 Acked 9029 Read 7 Done Done New false
21994 8031 Acked 9030 Read 6 Done Done New false
21995 8032 Acked 9031 Read 6 Done Done New false
21996 8033 Acked 9032 Write 3 Done Done New false
21997 8034 Acked 9033 Read 4 Done Done New false
21998 8035 Acked 9034 Read 2 Done Done New false
21999 8036 Acked 9035 Write 10 Done Done New false
22000 8037 Acked 9036 Flush 0 Done Done New false
22001 8038 Acked 9037 Read 5 Done Done New false
22002 8039 Acked 9038 Flush 0 Done Done New false
22003 8040 Acked 9039 Write 7 Done Done New false
22004 8041 Acked 9040 Read 10 Done Done New false
22005 8042 Acked 9041 Read 1 Done Done New false
22006 8043 Acked 9042 Write 2 Done Done New false
22007 8044 Acked 9043 Flush 0 Done Done New false
22008 8045 Acked 9044 Write 1 Done Done New false
22009 8046 Acked 9045 Write 10 Done Done New false
22010 8047 Acked 9046 Write 10 Done Done New false
22011 8048 Acked 9047 Write 2 Done Done New false
22012 8049 Acked 9048 Flush 0 Done Done New false
22013 8050 Acked 9049 Read 3 Done Done New false
22014 8051 Acked 9050 Read 1 Done Done New false
22015 8052 Acked 9051 Flush 0 Done Done New false
22016 8053 Acked 9052 Write 2 Done Done New false
22017 8054 Acked 9053 Write 3 Done Done New false
22018 8055 Acked 9054 Read 7 Done Done New false
22019 8056 Acked 9055 Write 8 Done Done New false
22020 8057 Acked 9056 Read 7 Done Done New false
22021 8058 Acked 9057 Write 3 Done Done New false
22022 8059 Acked 9058 Flush 0 Done Done New false
22023 8060 Acked 9059 Read 10 Done Done New false
22024 8061 Acked 9060 Write 6 Done Done New false
22025 8062 Acked 9061 Read 4 Done Done New false
22026 8063 Acked 9062 Write 3 Done Done New false
22027 8064 Acked 9063 Read 3 Done Done New false
22028 8065 Acked 9064 Read 10 Done Done New false
22029 8066 Acked 9065 Write 1 Done Done New false
22030 8067 Acked 9066 Read 9 Done Done New false
22031 8068 Acked 9067 Flush 0 Done Done New false
22032 8069 Acked 9068 Read 6 Done Done New false
22033 8070 Acked 9069 Read 4 Done Done New false
22034 8071 Acked 9070 Flush 0 Done Done New false
22035 8072 Acked 9071 Write 4 Done Done New false
22036 8073 Acked 9072 Read 4 Done Done New false
22037 8074 Acked 9073 Read 8 Done Done New false
22038 8075 Acked 9074 Write 7 Done Done New false
22039 8076 Acked 9075 Write 1 Done Done New false
22040 8077 Acked 9076 Write 6 Done Done New false
22041 8078 Acked 9077 Read 1 Done Done New false
22042 8079 Acked 9078 Read 4 Done Done New false
22043 8080 Acked 9079 Write 2 Done Done New false
22044 8081 Acked 9080 Write 5 Done Done New false
22045 8082 Acked 9081 Read 5 Done Done New false
22046 8083 Acked 9082 Write 5 Done Done New false
22047 8084 Acked 9083 Write 4 Done Done New false
22048 8085 Acked 9084 Read 7 Done Done New false
22049 8086 Acked 9085 Read 1 Done Done New false
22050 8087 Acked 9086 Write 8 Done Done New false
22051 8088 Acked 9087 Read 5 Done Done New false
22052 8089 Acked 9088 Read 7 Done Done New false
22053 8090 Acked 9089 Read 8 Done Done New false
22054 8091 Acked 9090 Read 9 Done Done New false
22055 8092 Acked 9091 Flush 0 Done Done New false
22056 8093 Acked 9092 Read 2 Done Done New false
22057 8094 Acked 9093 Read 7 Done Done New false
22058 8095 Acked 9094 Read 4 Done Done New false
22059 8096 Acked 9095 Write 10 Done Done New false
22060 8097 Acked 9096 Flush 0 Done Done New false
22061 8098 Acked 9097 Write 5 Done Done New false
22062 8099 Acked 9098 Read 10 Done Done New false
22063 8100 Acked 9099 Read 10 Done Done New false
22064 8101 Acked 9100 Read 9 Done Done New false
22065 8102 Acked 9101 Write 8 Done Done New false
22066 8103 Acked 9102 Flush 0 Done Done New false
22067 8104 Acked 9103 Write 2 Done Done New false
22068 8105 Acked 9104 Flush 0 Done Done New false
22069 8106 Acked 9105 Read 5 Done Done New false
22070 8107 Acked 9106 Write 9 Done Done New false
22071 8108 Acked 9107 Write 9 Done Done New false
22072 8109 Acked 9108 Write 4 Done Done New false
22073 8110 Acked 9109 Write 10 Done Done New false
22074 8111 Acked 9110 Write 8 Done Done New false
22075 8112 Acked 9111 Write 9 Done Done New false
22076 8113 Acked 9112 Read 7 Done Done New false
22077 8114 Acked 9113 Write 8 Done Done New false
22078 8115 Acked 9114 Read 3 Done Done New false
22079 8116 Acked 9115 Write 2 Done Done New false
22080 8117 Acked 9116 Write 6 Done Done New false
22081 8118 Acked 9117 Read 10 Done Done New false
22082 8119 Acked 9118 Write 2 Done Done New false
22083 8120 Acked 9119 Read 7 Done Done New false
22084 8121 Acked 9120 Read 4 Done Done New false
22085 8122 Acked 9121 Write 10 Done Done New false
22086 8123 Acked 9122 Write 2 Done Done New false
22087 8124 Acked 9123 Read 10 Done Done New false
22088 8125 Acked 9124 Write 10 Done Done New false
22089 8126 Acked 9125 Read 4 Done Done New false
22090 8127 Acked 9126 Flush 0 Done Done New false
22091 8128 Acked 9127 Read 5 Done Done New false
22092 8129 Acked 9128 Read 6 Done Done New false
22093 8130 Acked 9129 Read 4 Done Done New false
22094 8131 Acked 9130 Read 10 Done Done New false
22095 8132 Acked 9131 Read 10 Done Done New false
22096 8133 Acked 9132 Write 10 Done Done New false
22097 8134 Acked 9133 Write 3 Done Done New false
22098 8135 Acked 9134 Flush 0 Done Done New false
22099 8136 Acked 9135 Write 1 Done Done New false
22100 8137 Acked 9136 Write 4 Done Done New false
22101 8138 Acked 9137 Write 5 Done Done New false
22102 8139 Acked 9138 Write 5 Done Done New false
22103 8140 Acked 9139 Write 7 Done Done New false
22104 8141 Acked 9140 Read 4 Done Done New false
22105 8142 Acked 9141 Read 9 Done Done New false
22106 8143 Acked 9142 Write 5 Done Done New false
22107 8144 Acked 9143 Write 1 Done Done New false
22108 8145 Acked 9144 Read 4 Done Done New false
22109 8146 Acked 9145 Read 8 Done Done New false
22110 8147 Acked 9146 Write 8 Done Done New false
22111 8148 Acked 9147 Write 7 Done Done New false
22112 8149 Acked 9148 Write 6 Done Done New false
22113 8150 Acked 9149 Write 9 Done Done New false
22114 8151 Acked 9150 Flush 0 Done Done New false
22115 8152 Acked 9151 Read 10 Done Done New false
22116 8153 Acked 9152 Write 7 Done Done New false
22117 8154 Acked 9153 Write 2 Done Done New false
22118 8155 Acked 9154 Write 7 Done Done New false
22119 8156 Acked 9155 Write 7 Done Done New false
22120 8157 Acked 9156 Write 6 Done Done New false
22121 8158 Acked 9157 Read 6 Done Done New false
22122 8159 Acked 9158 Read 3 Done Done New false
22123 8160 Acked 9159 Write 1 Done Done New false
22124 8161 Acked 9160 Read 3 Done Done New false
22125 8162 Acked 9161 Read 5 Done Done New false
22126 8163 Acked 9162 Read 7 Done Done New false
22127 8164 Acked 9163 Write 5 Done Done New false
22128 8165 Acked 9164 Read 5 Done Done New false
22129 8166 Acked 9165 Write 1 Done Done New false
22130 8167 Acked 9166 Write 9 Done Done New false
22131 8168 Acked 9167 Read 7 Done Done New false
22132 8169 Acked 9168 Read 10 Done Done New false
22133 8170 Acked 9169 Read 7 Done Done New false
22134 8171 Acked 9170 Write 1 Done Done New false
22135 8172 Acked 9171 Write 8 Done Done New false
22136 8173 Acked 9172 Flush 0 Done Done New false
22137 STATES DS:0 DS:1 DS:2 TOTAL
22138 New 0 0 303 303
22139 Sent 0 0 0 0
22140 Done 303 303 0 606
22141 Skipped 0 0 0 0
22142 Error 0 0 0 0
22143 Last Flush: 9172 9172 8869
22144 Downstairs last five completed: 8869 8868 8867 8866 8865 8864
22145 Upstairs last five completed: 8173 8172 8171 8170 8169 8168
22146 CLIENT: Up:0 ds:303 act:2
22147 ----------------------------------------------------------------
22148 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
22149 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
22150 7871 Acked 8870 Flush 0 Done Done New false
22151 7872 Acked 8871 Write 4 Done Done New false
22152 7873 Acked 8872 Write 3 Done Done New false
22153 7874 Acked 8873 Read 2 Done Done New false
22154 7875 Acked 8874 Write 6 Done Done New false
22155 7876 Acked 8875 Read 8 Done Done New false
22156 7877 Acked 8876 Read 4 Done Done New false
22157 7878 Acked 8877 Read 8 Done Done New false
22158 7879 Acked 8878 Write 3 Done Done New false
22159 7880 Acked 8879 Read 8 Done Done New false
22160 7881 Acked 8880 Write 8 Done Done New false
22161 7882 Acked 8881 Read 3 Done Done New false
22162 7883 Acked 8882 Write 7 Done Done New false
22163 7884 Acked 8883 Read 7 Done Done New false
22164 7885 Acked 8884 Write 4 Done Done New false
22165 7886 Acked 8885 Write 4 Done Done New false
22166 7887 Acked 8886 Read 2 Done Done New false
22167 7888 Acked 8887 Read 5 Done Done New false
22168 7889 Acked 8888 Read 7 Done Done New false
22169 7890 Acked 8889 Write 10 Done Done New false
22170 7891 Acked 8890 Read 7 Done Done New false
22171 7892 Acked 8891 Read 3 Done Done New false
22172 7893 Acked 8892 Write 3 Done Done New false
22173 7894 Acked 8893 Write 2 Done Done New false
22174 7895 Acked 8894 Write 7 Done Done New false
22175 7896 Acked 8895 Write 2 Done Done New false
22176 7897 Acked 8896 Write 2 Done Done New false
22177 7898 Acked 8897 Read 10 Done Done New false
22178 7899 Acked 8898 Read 2 Done Done New false
22179 7900 Acked 8899 Write 4 Done Done New false
22180 7901 Acked 8900 Read 4 Done Done New false
22181 7902 Acked 8901 Read 5 Done Done New false
22182 7903 Acked 8902 Read 9 Done Done New false
22183 7904 Acked 8903 Flush 0 Done Done New false
22184 7905 Acked 8904 Read 10 Done Done New false
22185 7906 Acked 8905 Read 7 Done Done New false
22186 7907 Acked 8906 Flush 0 Done Done New false
22187 7908 Acked 8907 Flush 0 Done Done New false
22188 7909 Acked 8908 Read 1 Done Done New false
22189 7910 Acked 8909 Write 3 Done Done New false
22190 7911 Acked 8910 Read 10 Done Done New false
22191 7912 Acked 8911 Flush 0 Done Done New false
22192 7913 Acked 8912 Write 1 Done Done New false
22193 7914 Acked 8913 Read 9 Done Done New false
22194 7915 Acked 8914 Read 6 Done Done New false
22195 7916 Acked 8915 Write 8 Done Done New false
22196 7917 Acked 8916 Read 10 Done Done New false
22197 7918 Acked 8917 Flush 0 Done Done New false
22198 7919 Acked 8918 Write 4 Done Done New false
22199 7920 Acked 8919 Write 2 Done Done New false
22200 7921 Acked 8920 Read 7 Done Done New false
22201 7922 Acked 8921 Read 8 Done Done New false
22202 7923 Acked 8922 Flush 0 Done Done New false
22203 7924 Acked 8923 Flush 0 Done Done New false
22204 7925 Acked 8924 Read 3 Done Done New false
22205 7926 Acked 8925 Write 10 Done Done New false
22206 7927 Acked 8926 Read 8 Done Done New false
22207 7928 Acked 8927 Read 1 Done Done New false
22208 7929 Acked 8928 Write 2 Done Done New false
22209 7930 Acked 8929 Write 8 Done Done New false
22210 7931 Acked 8930 Read 10 Done Done New false
22211 7932 Acked 8931 Flush 0 Done Done New false
22212 7933 Acked 8932 Read 10 Done Done New false
22213 7934 Acked 8933 Flush 0 Done Done New false
22214 7935 Acked 8934 Read 1 Done Done New false
22215 7936 Acked 8935 Write 5 Done Done New false
22216 7937 Acked 8936 Read 7 Done Done New false
22217 7938 Acked 8937 Write 8 Done Done New false
22218 7939 Acked 8938 Read 6 Done Done New false
22219 7940 Acked 8939 Write 9 Done Done New false
22220 7941 Acked 8940 Read 6 Done Done New false
22221 7942 Acked 8941 Write 2 Done Done New false
22222 7943 Acked 8942 Read 4 Done Done New false
22223 7944 Acked 8943 Read 9 Done Done New false
22224 7945 Acked 8944 Read 3 Done Done New false
22225 7946 Acked 8945 Read 4 Done Done New false
22226 7947 Acked 8946 Write 8 Done Done New false
22227 7948 Acked 8947 Read 6 Done Done New false
22228 7949 Acked 8948 Write 1 Done Done New false
22229 7950 Acked 8949 Read 9 Done Done New false
22230 7951 Acked 8950 Read 8 Done Done New false
22231 7952 Acked 8951 Read 5 Done Done New false
22232 7953 Acked 8952 Flush 0 Done Done New false
22233 7954 Acked 8953 Read 6 Done Done New false
22234 7955 Acked 8954 Read 1 Done Done New false
22235 7956 Acked 8955 Read 6 Done Done New false
22236 7957 Acked 8956 Read 6 Done Done New false
22237 7958 Acked 8957 Read 9 Done Done New false
22238 7959 Acked 8958 Write 6 Done Done New false
22239 7960 Acked 8959 Read 6 Done Done New false
22240 7961 Acked 8960 Write 3 Done Done New false
22241 7962 Acked 8961 Read 5 Done Done New false
22242 7963 Acked 8962 Flush 0 Done Done New false
22243 7964 Acked 8963 Write 1 Done Done New false
22244 7965 Acked 8964 Write 1 Done Done New false
22245 7966 Acked 8965 Read 1 Done Done New false
22246 7967 Acked 8966 Write 1 Done Done New false
22247 7968 Acked 8967 Read 2 Done Done New false
22248 7969 Acked 8968 Write 6 Done Done New false
22249 7970 Acked 8969 Write 9 Done Done New false
22250 7971 Acked 8970 Flush 0 Done Done New false
22251 7972 Acked 8971 Read 1 Done Done New false
22252 7973 Acked 8972 Read 9 Done Done New false
22253 7974 Acked 8973 Write 6 Done Done New false
22254 7975 Acked 8974 Read 1 Done Done New false
22255 7976 Acked 8975 Write 6 Done Done New false
22256 7977 Acked 8976 Read 1 Done Done New false
22257 7978 Acked 8977 Read 4 Done Done New false
22258 7979 Acked 8978 Write 4 Done Done New false
22259 7980 Acked 8979 Write 10 Done Done New false
22260 7981 Acked 8980 Read 10 Done Done New false
22261 7982 Acked 8981 Write 7 Done Done New false
22262 7983 Acked 8982 Write 7 Done Done New false
22263 7984 Acked 8983 Write 5 Done Done New false
22264 7985 Acked 8984 Read 2 Done Done New false
22265 7986 Acked 8985 Read 8 Done Done New false
22266 7987 Acked 8986 Write 9 Done Done New false
22267 7988 Acked 8987 Write 6 Done Done New false
22268 7989 Acked 8988 Write 8 Done Done New false
22269 7990 Acked 8989 Write 1 Done Done New false
22270 7991 Acked 8990 Write 4 Done Done New false
22271 7992 Acked 8991 Write 1 Done Done New false
22272 7993 Acked 8992 Write 8 Done Done New false
22273 7994 Acked 8993 Read 8 Done Done New false
22274 7995 Acked 8994 Flush 0 Done Done New false
22275 7996 Acked 8995 Read 6 Done Done New false
22276 7997 Acked 8996 Read 9 Done Done New false
22277 7998 Acked 8997 Write 10 Done Done New false
22278 7999 Acked 8998 Read 3 Done Done New false
22279 8000 Acked 8999 Read 3 Done Done New false
22280 8001 Acked 9000 Write 9 Done Done New false
22281 8002 Acked 9001 Read 6 Done Done New false
22282 8003 Acked 9002 Read 7 Done Done New false
22283 8004 Acked 9003 Write 8 Done Done New false
22284 8005 Acked 9004 Read 6 Done Done New false
22285 8006 Acked 9005 Read 6 Done Done New false
22286 8007 Acked 9006 Read 6 Done Done New false
22287 8008 Acked 9007 Write 9 Done Done New false
22288 8009 Acked 9008 Read 3 Done Done New false
22289 8010 Acked 9009 Flush 0 Done Done New false
22290 8011 Acked 9010 Write 10 Done Done New false
22291 8012 Acked 9011 Read 3 Done Done New false
22292 8013 Acked 9012 Read 3 Done Done New false
22293 8014 Acked 9013 Write 2 Done Done New false
22294 8015 Acked 9014 Read 1 Done Done New false
22295 8016 Acked 9015 Read 2 Done Done New false
22296 8017 Acked 9016 Read 1 Done Done New false
22297 8018 Acked 9017 Read 7 Done Done New false
22298 8019 Acked 9018 Write 3 Done Done New false
22299 8020 Acked 9019 Write 5 Done Done New false
22300 8021 Acked 9020 Write 9 Done Done New false
22301 8022 Acked 9021 Write 6 Done Done New false
22302 8023 Acked 9022 Read 7 Done Done New false
22303 8024 Acked 9023 Read 9 Done Done New false
22304 8025 Acked 9024 Read 5 Done Done New false
22305 8026 Acked 9025 Read 2 Done Done New false
22306 8027 Acked 9026 Read 3 Done Done New false
22307 8028 Acked 9027 Write 4 Done Done New false
22308 8029 Acked 9028 Write 5 Done Done New false
22309 8030 Acked 9029 Read 7 Done Done New false
22310 8031 Acked 9030 Read 6 Done Done New false
22311 8032 Acked 9031 Read 6 Done Done New false
22312 8033 Acked 9032 Write 3 Done Done New false
22313 8034 Acked 9033 Read 4 Done Done New false
22314 8035 Acked 9034 Read 2 Done Done New false
22315 8036 Acked 9035 Write 10 Done Done New false
22316 8037 Acked 9036 Flush 0 Done Done New false
22317 8038 Acked 9037 Read 5 Done Done New false
22318 8039 Acked 9038 Flush 0 Done Done New false
22319 8040 Acked 9039 Write 7 Done Done New false
22320 8041 Acked 9040 Read 10 Done Done New false
22321 8042 Acked 9041 Read 1 Done Done New false
22322 8043 Acked 9042 Write 2 Done Done New false
22323 8044 Acked 9043 Flush 0 Done Done New false
22324 8045 Acked 9044 Write 1 Done Done New false
22325 8046 Acked 9045 Write 10 Done Done New false
22326 8047 Acked 9046 Write 10 Done Done New false
22327 8048 Acked 9047 Write 2 Done Done New false
22328 8049 Acked 9048 Flush 0 Done Done New false
22329 8050 Acked 9049 Read 3 Done Done New false
22330 8051 Acked 9050 Read 1 Done Done New false
22331 8052 Acked 9051 Flush 0 Done Done New false
22332 8053 Acked 9052 Write 2 Done Done New false
22333 8054 Acked 9053 Write 3 Done Done New false
22334 8055 Acked 9054 Read 7 Done Done New false
22335 8056 Acked 9055 Write 8 Done Done New false
22336 8057 Acked 9056 Read 7 Done Done New false
22337 8058 Acked 9057 Write 3 Done Done New false
22338 8059 Acked 9058 Flush 0 Done Done New false
22339 8060 Acked 9059 Read 10 Done Done New false
22340 8061 Acked 9060 Write 6 Done Done New false
22341 8062 Acked 9061 Read 4 Done Done New false
22342 8063 Acked 9062 Write 3 Done Done New false
22343 8064 Acked 9063 Read 3 Done Done New false
22344 8065 Acked 9064 Read 10 Done Done New false
22345 8066 Acked 9065 Write 1 Done Done New false
22346 8067 Acked 9066 Read 9 Done Done New false
22347 8068 Acked 9067 Flush 0 Done Done New false
22348 8069 Acked 9068 Read 6 Done Done New false
22349 8070 Acked 9069 Read 4 Done Done New false
22350 8071 Acked 9070 Flush 0 Done Done New false
22351 8072 Acked 9071 Write 4 Done Done New false
22352 8073 Acked 9072 Read 4 Done Done New false
22353 8074 Acked 9073 Read 8 Done Done New false
22354 8075 Acked 9074 Write 7 Done Done New false
22355 8076 Acked 9075 Write 1 Done Done New false
22356 8077 Acked 9076 Write 6 Done Done New false
22357 8078 Acked 9077 Read 1 Done Done New false
22358 8079 Acked 9078 Read 4 Done Done New false
22359 8080 Acked 9079 Write 2 Done Done New false
22360 8081 Acked 9080 Write 5 Done Done New false
22361 8082 Acked 9081 Read 5 Done Done New false
22362 8083 Acked 9082 Write 5 Done Done New false
22363 8084 Acked 9083 Write 4 Done Done New false
22364 8085 Acked 9084 Read 7 Done Done New false
22365 8086 Acked 9085 Read 1 Done Done New false
22366 8087 Acked 9086 Write 8 Done Done New false
22367 8088 Acked 9087 Read 5 Done Done New false
22368 8089 Acked 9088 Read 7 Done Done New false
22369 8090 Acked 9089 Read 8 Done Done New false
22370 8091 Acked 9090 Read 9 Done Done New false
22371 8092 Acked 9091 Flush 0 Done Done New false
22372 8093 Acked 9092 Read 2 Done Done New false
22373 8094 Acked 9093 Read 7 Done Done New false
22374 8095 Acked 9094 Read 4 Done Done New false
22375 8096 Acked 9095 Write 10 Done Done New false
22376 8097 Acked 9096 Flush 0 Done Done New false
22377 8098 Acked 9097 Write 5 Done Done New false
22378 8099 Acked 9098 Read 10 Done Done New false
22379 8100 Acked 9099 Read 10 Done Done New false
22380 8101 Acked 9100 Read 9 Done Done New false
22381 8102 Acked 9101 Write 8 Done Done New false
22382 8103 Acked 9102 Flush 0 Done Done New false
22383 8104 Acked 9103 Write 2 Done Done New false
22384 8105 Acked 9104 Flush 0 Done Done New false
22385 8106 Acked 9105 Read 5 Done Done New false
22386 8107 Acked 9106 Write 9 Done Done New false
22387 8108 Acked 9107 Write 9 Done Done New false
22388 8109 Acked 9108 Write 4 Done Done New false
22389 8110 Acked 9109 Write 10 Done Done New false
22390 8111 Acked 9110 Write 8 Done Done New false
22391 8112 Acked 9111 Write 9 Done Done New false
22392 8113 Acked 9112 Read 7 Done Done New false
22393 8114 Acked 9113 Write 8 Done Done New false
22394 8115 Acked 9114 Read 3 Done Done New false
22395 8116 Acked 9115 Write 2 Done Done New false
22396 8117 Acked 9116 Write 6 Done Done New false
22397 8118 Acked 9117 Read 10 Done Done New false
22398 8119 Acked 9118 Write 2 Done Done New false
22399 8120 Acked 9119 Read 7 Done Done New false
22400 8121 Acked 9120 Read 4 Done Done New false
22401 8122 Acked 9121 Write 10 Done Done New false
22402 8123 Acked 9122 Write 2 Done Done New false
22403 8124 Acked 9123 Read 10 Done Done New false
22404 8125 Acked 9124 Write 10 Done Done New false
22405 8126 Acked 9125 Read 4 Done Done New false
22406 8127 Acked 9126 Flush 0 Done Done New false
22407 8128 Acked 9127 Read 5 Done Done New false
22408 8129 Acked 9128 Read 6 Done Done New false
22409 8130 Acked 9129 Read 4 Done Done New false
22410 8131 Acked 9130 Read 10 Done Done New false
22411 8132 Acked 9131 Read 10 Done Done New false
22412 8133 Acked 9132 Write 10 Done Done New false
22413 8134 Acked 9133 Write 3 Done Done New false
22414 8135 Acked 9134 Flush 0 Done Done New false
22415 8136 Acked 9135 Write 1 Done Done New false
22416 8137 Acked 9136 Write 4 Done Done New false
22417 8138 Acked 9137 Write 5 Done Done New false
22418 8139 Acked 9138 Write 5 Done Done New false
22419 8140 Acked 9139 Write 7 Done Done New false
22420 8141 Acked 9140 Read 4 Done Done New false
22421 8142 Acked 9141 Read 9 Done Done New false
22422 8143 Acked 9142 Write 5 Done Done New false
22423 8144 Acked 9143 Write 1 Done Done New false
22424 8145 Acked 9144 Read 4 Done Done New false
22425 8146 Acked 9145 Read 8 Done Done New false
22426 8147 Acked 9146 Write 8 Done Done New false
22427 8148 Acked 9147 Write 7 Done Done New false
22428 8149 Acked 9148 Write 6 Done Done New false
22429 8150 Acked 9149 Write 9 Done Done New false
22430 8151 Acked 9150 Flush 0 Done Done New false
22431 8152 Acked 9151 Read 10 Done Done New false
22432 8153 Acked 9152 Write 7 Done Done New false
22433 8154 Acked 9153 Write 2 Done Done New false
22434 8155 Acked 9154 Write 7 Done Done New false
22435 8156 Acked 9155 Write 7 Done Done New false
22436 8157 Acked 9156 Write 6 Done Done New false
22437 8158 Acked 9157 Read 6 Done Done New false
22438 8159 Acked 9158 Read 3 Done Done New false
22439 8160 Acked 9159 Write 1 Done Done New false
22440 8161 Acked 9160 Read 3 Done Done New false
22441 8162 Acked 9161 Read 5 Done Done New false
22442 8163 Acked 9162 Read 7 Done Done New false
22443 8164 Acked 9163 Write 5 Done Done New false
22444 8165 Acked 9164 Read 5 Done Done New false
22445 8166 Acked 9165 Write 1 Done Done New false
22446 8167 Acked 9166 Write 9 Done Done New false
22447 8168 Acked 9167 Read 7 Done Done New false
22448 8169 Acked 9168 Read 10 Done Done New false
22449 8170 Acked 9169 Read 7 Done Done New false
22450 8171 Acked 9170 Write 1 Done Done New false
22451 8172 Acked 9171 Write 8 Done Done New false
22452 8173 Acked 9172 Flush 0 Done Done New false
22453 STATES DS:0 DS:1 DS:2 TOTAL
22454 New 0 0 303 303
22455 Sent 0 0 0 0
22456 Done 303 303 0 606
22457 Skipped 0 0 0 0
22458 Error 0 0 0 0
22459 Last Flush: 9172 9172 8869
22460 Downstairs last five completed: 8869 8868 8867 8866 8865 8864
22461 Upstairs last five completed: 8173 8172 8171 8170 8169 8168
22462 CLIENT: Up:0 ds:303 act:2
22463 [8830] got start action
22464 [8830] I am going to restart
22465 Starting downstairs at port 8830
22466 Make output file at "/tmp/dsc/downstairs-8830.txt"
22467 [8830][2] reports Starting
22468 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1203)
22469 [8830][2] reports Running
224702023-10-13T18:13:59.544ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 2
224712023-10-13T18:13:59.544ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
224722023-10-13T18:13:59.544ZINFOcrucible: [2] client re-new 303 jobs since flush 8869 = downstairs
224732023-10-13T18:13:59.545ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
224742023-10-13T18:13:59.546ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
224752023-10-13T18:13:59.546ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
224762023-10-13T18:13:59.546ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
224772023-10-13T18:13:59.546ZINFOcrucible: [2] send last flush ID to this DS: 8869
224782023-10-13T18:13:59.546ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
224792023-10-13T18:13:59.546ZINFOcrucible: [2] Replied this last flush ID: 8869
224802023-10-13T18:13:59.546ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Offline ds_transition to Replay
224812023-10-13T18:13:59.546ZINFOcrucible: [2] Transition from Offline to Replay
224822023-10-13T18:13:59.546ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
224832023-10-13T18:13:59.546ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Replay ds_transition to Active
224842023-10-13T18:13:59.546ZINFOcrucible: [2] Transition from Replay to Active
224852023-10-13T18:13:59.546ZINFOcrucible: [2] Starts cmd_loop
224862023-10-13T18:13:59.546ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
224872023-10-13T18:13:59.546ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
224882023-10-13T18:13:59.546ZINFOcrucible: Set check for repair
224892023-10-13T18:14:00.547ZWARNcrucible: [2] flow control end
224902023-10-13T18:14:00.548ZINFOcrucible: Checking if live repair is needed
224912023-10-13T18:14:00.548ZINFOcrucible: No Live Repair required at this time
22492 ----------------------------------------------------------------
22493 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22494 Downstairs last five completed: 9172 9171 9170 9169 9168 9167
22495 Upstairs last five completed: 8173 8172 8171 8170 8169 8168
22496 CLIENT: Up:0 ds:0 act:3
22497 Replay: All jobs finished, all DS active.
22498 Main task has work to do, go find it
22499 got dsc Stop(2)
22500 stop 2
22501 Oct 13 18:14:01.495 INFO request completed, latency_us: [8830] Got stop action so:false kr:false
22502 512, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 1c52cd20-fc56-493e-88a7-ac0cd4f67584, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
22503 Oct 13 18:14:01.497{"msg":"[2] downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2023-10-13T18:14:01.497542095Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1176}
22504 INFO request completed, latency_us: 188, response_code: 200, {"msg":"[2] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2023-10-13T18:14:01.497676625Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1176}
22505 uri: /state/cid/2, method: {"msg":"[2] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8830 closed","v":0,"name":"crucible","level":30,"time":"2023-10-13T18:14:01.497737052Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1176,"looper":"2"}
22506 GET, req_id: 9622f289-2995-47c3-a1a5-fab103b8e69a, remote_addr: 127{"msg":"[2] 127.0.0.1:8830 task reports connection:false","v":0,"name":"crucible","level":30,"time":"2023-10-13T18:14:01.497789107Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1176}
22507 .0.0.{"msg":"fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline","v":0,"name":"crucible","level":30,"time":"2023-10-13T18:14:01.497837961Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1176}
22508 1:64571, local_addr: 127.0.{"msg":"[2] 127.0.0.1:8830 task reports offline","v":0,"name":"crucible","level":30,"time":"2023-10-13T18:14:01.497886975Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1176}
22509 0.1:9998
225102023-10-13T18:14:01.497ZWARNcrucible: [2] pm_task rx.recv() is None
22511 [8830] Exited with: ExitStatus(unix_wait_status(9))
22512 [8830][2] reports Exit
225132023-10-13T18:14:03.500ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
22514 Oct 13 18:14:05.500 INFO request completed, latency_us: 264, response_code: 200, uri: /state/cid/2, method: GET, req_id: 245ac573-7406-44fa-a245-ff721fe688dd, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
22515 001/300 Read block 2357 len 36864
22516 002/300 Write block 3594 len 28672 data: 6 7 8 7 6 4 6
22517 003/300 Read block 379 len 8192
22518 004/300 Read block 1020 len 4096
22519 005/300 Write block 3949 len 40960 data: 4 6 5 5 6 7 8 7 7 6
22520 006/300 Read block 340 len 12288
22521 007/300 Read block 3347 len 32768
22522 008/300 Write block 693 len 28672 data: 8 8 6 5 5 7 7
22523 009/300 Write block 1758 len 8192 data: 6 7
22524 010/300 Read block 186 len 4096
22525 011/300 Flush
22526 012/300 Write block 1689 len 20480 data: 5 3 5 7 8
22527 013/300 Read block 2055 len 32768
22528 014/300 Read block 3741 len 8192
22529 015/300 Write block 2322 len 12288 data: 6 5 5
22530 016/300 Write block 338 len 12288 data: 4 4 3
22531 017/300 Write block 1055 len 24576 data: 7 5 6 7 7 7
22532 018/300 Flush
22533 019/300 Read block 2553 len 20480
22534 020/300 Read block 650 len 32768
22535 021/300 Read block 3022 len 8192
22536 022/300 Read block 4480 len 24576
22537 023/300 Flush
22538 024/300 Read block 3944 len 20480
22539 025/300 Write block 4983 len 12288 data: 6 5 5
22540 026/300 Read block 2775 len 12288
22541 027/300 Write block 2041 len 36864 data: 3 3 3 2 2 2 2 3 3
22542 028/300 Write block 4267 len 12288 data: 4 4 5
22543 029/300 Write block 2334 len 12288 data: 6 6 6
22544 030/300 Flush
22545 031/300 Flush
22546 032/300 Write block 3175 len 28672 data: 4 7 7 9 9 7 7
22547 033/300 Write block 4916 len 28672 data: 8 9 8 7 8 8 9
22548 034/300 Read block 667 len 8192
22549 035/300 Write block 3438 len 8192 data: 4 4
22550 036/300 Write block 2318 len 16384 data: 8 5 4 5
22551 037/300 Write block 4911 len 24576 data: 4 5 4 6 6 9
22552 038/300 Read block 4959 len 16384
22553 039/300 Read block 755 len 24576
22554 040/300 Write block 2059 len 16384 data: 5 4 3 3
22555 041/300 Read block 4357 len 16384
22556 042/300 Read block 4901 len 36864
22557 043/300 Flush
22558 044/300 Read block 1715 len 20480
22559 045/300 Read block 1968 len 36864
22560 046/300 Flush
22561 047/300 Read block 4859 len 40960
22562 048/300 Read block 4802 len 36864
22563 049/300 Read block 1301 len 36864
22564 050/300 Read block 295 len 24576
22565 051/300 Write block 1351 len 28672 data: 5 5 6 7 5 4 4
22566 052/300 Read block 3989 len 28672
22567 053/300 Write block 2642 len 32768 data: 6 7 7 5 4 4 2 3
22568 054/300 Read block 4689 len 40960
22569 055/300 Read block 3226 len 4096
22570 056/300 Read block 3946 len 24576
22571 057/300 Read block 800 len 28672
22572 058/300 Write block 2869 len 4096 data: 8
22573 059/300 Read block 1337 len 36864
22574 060/300 Read block 759 len 4096
22575 061/300 Flush
22576 062/300 Write block 3434 len 32768 data: 10 7 7 5 5 5 4 4
22577 063/300 Write block 1798 len 16384 data: 6 5 6 6
22578 064/300 Read block 2515 len 32768
22579 065/300 Read block 1366 len 32768
22580 066/300 Read block 343 len 16384
22581 067/300 Read block 3054 len 12288
22582 068/300 Read block 3412 len 20480
22583 069/300 Write block 2501 len 16384 data: 5 7 7 8
22584 070/300 Write block 3296 len 40960 data: 3 3 5 5 5 5 6 6 5 5
22585 071/300 Read block 1128 len 24576
22586 072/300 Write block 949 len 24576 data: 4 4 4 5 8 7
22587 073/300 Read block 1527 len 32768
22588 074/300 Read block 3356 len 8192
22589 075/300 Write block 1837 len 8192 data: 9 8
22590 076/300 Read block 2867 len 20480
22591 077/300 Flush
22592 078/300 Flush
22593 079/300 Flush
22594 080/300 Write block 808 len 4096 data: 6
22595 081/300 Write block 1012 len 8192 data: 5 3
22596 082/300 Write block 1574 len 4096 data: 4
22597 083/300 Read block 3975 len 16384
22598 084/300 Flush
22599 085/300 Read block 875 len 20480
22600 086/300 Read block 3583 len 40960
22601 087/300 Read block 1877 len 40960
22602 088/300 Write block 4418 len 16384 data: 4 4 4 3
22603 089/300 Read block 3655 len 32768
22604 090/300 Read block 462 len 36864
22605 091/300 Write block 2372 len 20480 data: 4 4 4 4 5
22606 092/300 Read block 1362 len 20480
22607 093/300 Write block 1289 len 20480 data: 5 5 5 5 4
22608 094/300 Write block 4978 len 24576 data: 6 7 8 9 8 7
22609 095/300 Read block 1334 len 8192
22610 096/300 Read block 2599 len 8192
22611 097/300 Write block 3020 len 16384 data: 10 10 9 8
22612 098/300 Write block 4534 len 24576 data: 7 7 6 5 5 4
22613 099/300 Write block 4677 len 8192 data: 8 8
22614 100/300 Write block 1792 len 40960 data: 9 7 6 7 6 5 7 6 7 7
22615 101/300 Flush
22616 102/300 Read block 2000 len 28672
22617 103/300 Write block 4521 len 32768 data: 4 4 4 3 3 4 4 7
22618 104/300 Write block 3139 len 32768 data: 7 7 8 8 9 7 11 13
22619 105/300 Write block 1411 len 36864 data: 4 7 7 7 6 6 6 6 5
22620 106/300 Read block 3723 len 8192
22621 107/300 Read block 1084 len 32768
22622 108/300 Read block 718 len 16384
22623 109/300 Read block 822 len 8192
22624 110/300 Read block 4521 len 28672
22625 111/300 Read block 4226 len 28672
22626 112/300 Read block 4968 len 8192
22627 113/300 Read block 3997 len 28672
22628 114/300 Write block 1225 len 40960 data: 2 3 3 3 3 5 5 5 4 6
22629 115/300 Write block 454 len 4096 data: 5
22630 116/300 Write block 4460 len 16384 data: 5 5 5 5
22631 117/300 Write block 2225 len 24576 data: 5 2 3 2 2 4
22632 118/300 Read block 4950 len 28672
22633 119/300 Read block 2147 len 24576
22634 120/300 Read block 2560 len 32768
22635 121/300 Read block 228 len 24576
22636 122/300 Flush
22637 123/300 Read block 4268 len 12288
22638 124/300 Read block 3589 len 28672
22639 125/300 Flush
22640 126/300 Read block 1347 len 28672
22641 127/300 Read block 1014 len 8192
22642 128/300 Write block 4136 len 36864 data: 4 4 4 5 4 4 3 4 4
22643 129/300 Flush
22644 130/300 Read block 4790 len 32768
22645 131/300 Read block 1756 len 12288
22646 132/300 Write block 2561 len 16384 data: 13 12 9 7
22647 133/300 Write block 575 len 36864 data: 5 5 5 4 4 4 3 3 5
22648 134/300 Read block 1013 len 4096
22649 135/300 Write block 1571 len 8192 data: 4 3
22650 136/300 Flush
22651 137/300 Read block 1318 len 12288
22652 138/300 Write block 3795 len 32768 data: 5 5 5 5 5 5 4 6
22653 139/300 Read block 2535 len 4096
22654 140/300 Write block 330 len 32768 data: 5 5 5 5 5 5 6 5
22655 141/300 Write block 250 len 24576 data: 5 5 4 5 6 5
22656 142/300 Read block 2713 len 8192
22657 143/300 Write block 2258 len 28672 data: 4 5 7 5 5 4 3
22658 144/300 Read block 3020 len 36864
22659 145/300 Read block 2212 len 32768
22660 146/300 Write block 3928 len 16384 data: 5 7 7 6
22661 147/300 Read block 2174 len 12288
22662 148/300 Read block 431 len 4096
22663 149/300 Read block 4958 len 4096
22664 150/300 Read block 1217 len 20480
22665 151/300 Flush
22666 152/300 Write block 119 len 12288 data: 6 6 6
22667 153/300 Read block 3518 len 40960
22668 154/300 Read block 1138 len 16384
22669 155/300 Write block 1024 len 12288 data: 4 5 6
22670 156/300 Write block 2789 len 4096 data: 6
22671 157/300 Read block 1315 len 40960
22672 158/300 Read block 2487 len 8192
22673 159/300 Read block 4913 len 32768
22674 160/300 Write block 2717 len 16384 data: 5 7 6 5
22675 161/300 Write block 1779 len 8192 data: 3 5
22676 162/300 Read block 2343 len 32768
22677 163/300 Read block 1914 len 20480
22678 164/300 Write block 1680 len 4096 data: 6
22679 165/300 Read block 4929 len 32768
22680 166/300 Write block 2447 len 12288 data: 5 5 5
22681 167/300 Read block 1722 len 8192
22682 168/300 Write block 3709 len 40960 data: 10 15 14 12 9 9 10 8 8 6
22683 169/300 Write block 1122 len 40960 data: 4 4 4 4 5 5 6 6 6 6
22684 170/300 Write block 4107 len 28672 data: 5 4 3 3 3 4 5
22685 171/300 Write block 1644 len 12288 data: 5 4 5
22686 172/300 Read block 3268 len 12288
22687 173/300 Write block 677 len 8192 data: 5 5
22688 174/300 Write block 2600 len 32768 data: 7 10 11 10 8 9 8 5
22689 175/300 Write block 1475 len 24576 data: 7 6 8 8 8 7
22690 176/300 Flush
22691 177/300 Write block 3277 len 12288 data: 3 3 3
22692 178/300 Write block 4604 len 28672 data: 5 5 6 6 8 6 5
22693 179/300 Read block 3276 len 32768
22694 180/300 Read block 397 len 12288
22695 181/300 Read block 870 len 24576
22696 182/300 Write block 2680 len 4096 data: 6
22697 183/300 Read block 2601 len 24576
22698 184/300 Read block 2054 len 32768
22699 185/300 Read block 2190 len 40960
22700 186/300 Write block 2352 len 32768 data: 10 9 10 11 10 11 8 8
22701 187/300 Read block 18 len 40960
22702 188/300 Read block 2310 len 12288
22703 189/300 Read block 1754 len 20480
22704 190/300 Read block 2820 len 8192
22705 191/300 Read block 1633 len 8192
22706 192/300 Write block 70 len 36864 data: 6 6 6 6 8 4 4 4 5
22707 193/300 Read block 2347 len 40960
22708 194/300 Read block 3015 len 24576
22709 195/300 Read block 23 len 32768
22710 196/300 Write block 339 len 16384 data: 5 4 3 3
22711 197/300 Read block 3188 len 36864
22712 198/300 Write block 3474 len 12288 data: 7 7 8
22713 199/300 Read block 2266 len 36864
22714 200/300 Write block 1594 len 40960 data: 5 5 5 3 3 3 3 2 2 4
22715 201/300 Write block 135 len 24576 data: 4 5 6 6 7 9
22716 202/300 Write block 4735 len 12288 data: 5 5 4
22717 203/300 Flush
22718 204/300 Read block 4657 len 24576
22719 205/300 Read block 1754 len 36864
22720 206/300 Write block 4751 len 28672 data: 8 7 8 9 10 9 9
22721 207/300 Read block 2230 len 4096
22722 208/300 Read block 2671 len 28672
22723 209/300 Write block 2908 len 12288 data: 3 4 4
22724 210/300 Read block 746 len 4096
22725 211/300 Write block 3609 len 32768 data: 5 5 6 5 5 6 5 6
22726 212/300 Write block 4313 len 36864 data: 8 8 8 6 4 4 4 4 4
22727 213/300 Write block 3792 len 32768 data: 4 5 5 6 6 6 6 6
22728 214/300 Flush
22729 215/300 Write block 4374 len 8192 data: 10 8
22730 216/300 Read block 3826 len 4096
22731 217/300 Read block 2676 len 40960
22732 218/300 Write block 3196 len 28672 data: 7 6 6 5 6 6 4
22733 219/300 Write block 1179 len 8192 data: 5 4
22734 220/300 Write block 3294 len 28672 data: 5 4 4 4 6 6 6
22735 221/300 Read block 1647 len 28672
22736 222/300 Read block 1129 len 40960
22737 223/300 Flush
22738 224/300 Write block 3258 len 8192 data: 5 6
22739 225/300 Read block 809 len 20480
22740 226/300 Write block 3463 len 20480 data: 6 4 3 2 2
22741 227/300 Flush
22742 228/300 Write block 1551 len 36864 data: 5 5 6 5 4 6 7 6 5
22743 229/300 Flush
22744 230/300 Write block 2754 len 4096 data: 6
22745 231/300 Write block 1789 len 28672 data: 10 9 9 10 8 7 8
22746 232/300 Read block 436 len 16384
22747 233/300 Read block 1089 len 40960
22748 234/300 Write block 4689 len 16384 data: 3 5 4 4
22749 235/300 Write block 836 len 40960 data: 8 9 9 8 9 7 7 4 5 6
22750 236/300 Write block 4131 len 32768 data: 4 4 4 4 4 5 5 5
22751 237/300 Read block 4327 len 12288
22752 238/300 Write block 4206 len 32768 data: 5 6 6 5 4 4 4 3
22753 239/300 Write block 1706 len 32768 data: 4 4 6 6 6 6 5 5
22754 240/300 Read block 3501 len 12288
22755 241/300 Write block 2165 len 4096 data: 5
22756 242/300 Write block 2567 len 32768 data: 7 8 8 7 3 4 2 2
22757 243/300 Flush
22758 244/300 Read block 4320 len 36864
22759 245/300 Read block 1857 len 40960
22760 246/300 Flush
22761 247/300 Write block 1635 len 32768 data: 10 9 9 8 8 7 6 6
22762 248/300 Read block 2400 len 32768
22763 249/300 Read block 2719 len 16384
22764 250/300 Read block 734 len 28672
22765 251/300 Write block 3209 len 12288 data: 6 7 7
22766 252/300 Read block 2724 len 36864
22767 253/300 Read block 4273 len 4096
22768 254/300 Flush
22769 255/300 Read block 2403 len 12288
22770 256/300 Read block 3900 len 28672
22771 257/300 Read block 2032 len 8192
22772 258/300 Write block 2768 len 8192 data: 5 6
22773 259/300 Read block 1714 len 28672
22774 260/300 Read block 4704 len 28672
22775 261/300 Read block 4282 len 12288
22776 262/300 Flush
22777 263/300 Read block 1999 len 8192
22778 264/300 Write block 4923 len 36864 data: 7 6 4 3 5 3 4 4 4
22779 265/300 Write block 3401 len 40960 data: 3 4 3 5 6 7 7 7 5 5
22780 266/300 Read block 1048 len 20480
22781 267/300 Read block 1775 len 20480
22782 268/300 Read block 747 len 40960
22783 269/300 Flush
22784 270/300 Read block 241 len 28672
22785 271/300 Write block 2073 len 24576 data: 3 5 5 6 4 4
22786 272/300 Write block 4113 len 24576 data: 6 4 6 5 6 7
22787 273/300 Read block 3159 len 24576
22788 274/300 Write block 4128 len 4096 data: 3
22789 275/300 Write block 4980 len 8192 data: 9 10
22790 276/300 Read block 4228 len 36864
22791 277/300 Write block 3024 len 16384 data: 8 8 7 6
22792 278/300 Write block 18 len 4096 data: 4
22793 279/300 Read block 4634 len 20480
22794 280/300 Read block 3948 len 36864
22795 281/300 Read block 1584 len 24576
22796 282/300 Read block 3854 len 40960
22797 283/300 Read block 2481 len 32768
22798 284/300 Read block 1960 len 24576
22799 285/300 Read block 3106 len 20480
22800 286/300 Read block 543 len 32768
22801 287/300 Write block 1578 len 32768 data: 4 4 3 4 4 4 3 4
22802 288/300 Read block 2416 len 24576
22803 289/300 Read block 4066 len 12288
22804 290/300 Write block 4458 len 4096 data: 6
22805 291/300 Read block 4323 len 40960
22806 292/300 Write block 547 len 20480 data: 4 3 3 4 7
22807 293/300 Write block 2161 len 12288 data: 5 4 4
22808 294/300 Write block 485 len 12288 data: 4 5 6
22809 295/300 Write block 2314 len 40960 data: 6 6 8 7 9 6 5 6 7 6
22810 296/300 Write block 805 len 32768 data: 4 6 6 7 5 5 5 5
22811 297/300 Read block 1806 len 16384
22812 298/300 Write block 3348 len 40960 data: 5 5 6 8 7 6 5 4 3 3
22813 299/300 Read block 2454 len 4096
22814 300/300 Write block 4727 len 4096 data: 9
22815 Main task has work to do, go find it
22816 got dsc Start(2)
22817 start 2
22818 Oct 13 18:14:06.467 INFO request completed, latency_us: 323, response_code: 204, uri: /start/cid/2, method: POST, req_id: 3a1b2e04-65c3-4be8-ba8e-8d4809d245cc, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
22819 Replay: started 2, returned:Ok(())
22820 ----------------------------------------------------------------
22821 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
22822 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
22823 8174 Acked 9173 Read 9 Done Done New false
22824 8175 Acked 9174 Write 7 Done Done New false
22825 8176 Acked 9175 Read 2 Done Done New false
22826 8177 Acked 9176 Read 1 Done Done New false
22827 8178 Acked 9177 Write 10 Done Done New false
22828 8179 Acked 9178 Read 3 Done Done New false
22829 8180 Acked 9179 Read 8 Done Done New false
22830 8181 Acked 9180 Write 7 Done Done New false
22831 8182 Acked 9181 Write 2 Done Done New false
22832 8183 Acked 9182 Read 1 Done Done New false
22833 8184 Acked 9183 Flush 0 Done Done New false
22834 8185 Acked 9184 Write 5 Done Done New false
22835 8186 Acked 9185 Read 8 Done Done New false
22836 8187 Acked 9186 Read 2 Done Done New false
22837 8188 Acked 9187 Write 3 Done Done New false
22838 8189 Acked 9188 Write 3 Done Done New false
22839 8190 Acked 9189 Write 6 Done Done New false
22840 8191 Acked 9190 Flush 0 Done Done New false
22841 8192 Acked 9191 Read 5 Done Done New false
22842 8193 Acked 9192 Read 8 Done Done New false
22843 8194 Acked 9193 Read 2 Done Done New false
22844 8195 Acked 9194 Read 6 Done Done New false
22845 8196 Acked 9195 Flush 0 Done Done New false
22846 8197 Acked 9196 Read 5 Done Done New false
22847 8198 Acked 9197 Write 3 Done Done New false
22848 8199 Acked 9198 Read 3 Done Done New false
22849 8200 Acked 9199 Write 9 Done Done New false
22850 8201 Acked 9200 Write 3 Done Done New false
22851 8202 Acked 9201 Write 3 Done Done New false
22852 8203 Acked 9202 Flush 0 Done Done New false
22853 8204 Acked 9203 Flush 0 Done Done New false
22854 8205 Acked 9204 Write 7 Done Done New false
22855 8206 Acked 9205 Write 7 Done Done New false
22856 8207 Acked 9206 Read 2 Done Done New false
22857 8208 Acked 9207 Write 2 Done Done New false
22858 8209 Acked 9208 Write 4 Done Done New false
22859 8210 Acked 9209 Write 6 Done Done New false
22860 8211 Acked 9210 Read 4 Done Done New false
22861 8212 Acked 9211 Read 6 Done Done New false
22862 8213 Acked 9212 Write 4 Done Done New false
22863 8214 Acked 9213 Read 4 Done Done New false
22864 8215 Acked 9214 Read 9 Done Done New false
22865 8216 Acked 9215 Flush 0 Done Done New false
22866 8217 Acked 9216 Read 5 Done Done New false
22867 8218 Acked 9217 Read 9 Done Done New false
22868 8219 Acked 9218 Flush 0 Done Done New false
22869 8220 Acked 9219 Read 10 Done Done New false
22870 8221 Acked 9220 Read 9 Done Done New false
22871 8222 Acked 9221 Read 9 Done Done New false
22872 8223 Acked 9222 Read 6 Done Done New false
22873 8224 Acked 9223 Write 7 Done Done New false
22874 8225 Acked 9224 Read 7 Done Done New false
22875 8226 Acked 9225 Write 8 Done Done New false
22876 8227 Acked 9226 Read 10 Done Done New false
22877 8228 Acked 9227 Read 1 Done Done New false
22878 8229 Acked 9228 Read 6 Done Done New false
22879 8230 Acked 9229 Read 7 Done Done New false
22880 8231 Acked 9230 Write 1 Done Done New false
22881 8232 Acked 9231 Read 9 Done Done New false
22882 8233 Acked 9232 Read 1 Done Done New false
22883 8234 Acked 9233 Flush 0 Done Done New false
22884 8235 Acked 9234 Write 8 Done Done New false
22885 8236 Acked 9235 Flush 0 Done Done New false
22886 8237 Acked 9236 Write 4 Done Done New false
22887 8238 Acked 9237 Read 8 Done Done New false
22888 8239 Acked 9238 Read 8 Done Done New false
22889 8240 Acked 9239 Read 4 Done Done New false
22890 8241 Acked 9240 Read 3 Done Done New false
22891 8242 Acked 9241 Read 5 Done Done New false
22892 8243 Acked 9242 Write 4 Done Done New false
22893 8244 Acked 9243 Write 10 Done Done New false
22894 8245 Acked 9244 Read 6 Done Done New false
22895 8246 Acked 9245 Write 6 Done Done New false
22896 8247 Acked 9246 Read 8 Done Done New false
22897 8248 Acked 9247 Read 2 Done Done New false
22898 8249 Acked 9248 Write 2 Done Done New false
22899 8250 Acked 9249 Read 5 Done Done New false
22900 8251 Acked 9250 Flush 0 Done Done New false
22901 8252 Acked 9251 Flush 0 Done Done New false
22902 8253 Acked 9252 Flush 0 Done Done New false
22903 8254 Acked 9253 Write 1 Done Done New false
22904 8255 Acked 9254 Write 2 Done Done New false
22905 8256 Acked 9255 Write 1 Done Done New false
22906 8257 Acked 9256 Read 4 Done Done New false
22907 8258 Acked 9257 Flush 0 Done Done New false
22908 8259 Acked 9258 Read 5 Done Done New false
22909 8260 Acked 9259 Read 10 Done Done New false
22910 8261 Acked 9260 Read 10 Done Done New false
22911 8262 Acked 9261 Write 4 Done Done New false
22912 8263 Acked 9262 Read 8 Done Done New false
22913 8264 Acked 9263 Read 9 Done Done New false
22914 8265 Acked 9264 Write 5 Done Done New false
22915 8266 Acked 9265 Read 5 Done Done New false
22916 8267 Acked 9266 Write 5 Done Done New false
22917 8268 Acked 9267 Write 6 Done Done New false
22918 8269 Acked 9268 Read 2 Done Done New false
22919 8270 Acked 9269 Read 2 Done Done New false
22920 8271 Acked 9270 Write 4 Done Done New false
22921 8272 Acked 9271 Write 6 Done Done New false
22922 8273 Acked 9272 Write 2 Done Done New false
22923 8274 Acked 9273 Write 10 Done Done New false
22924 8275 Acked 9274 Flush 0 Done Done New false
22925 8276 Acked 9275 Read 7 Done Done New false
22926 8277 Acked 9276 Write 8 Done Done New false
22927 8278 Acked 9277 Write 8 Done Done New false
22928 8279 Acked 9278 Write 9 Done Done New false
22929 8280 Acked 9279 Read 2 Done Done New false
22930 8281 Acked 9280 Read 8 Done Done New false
22931 8282 Acked 9281 Read 4 Done Done New false
22932 8283 Acked 9282 Read 2 Done Done New false
22933 8284 Acked 9283 Read 7 Done Done New false
22934 8285 Acked 9284 Read 7 Done Done New false
22935 8286 Acked 9285 Read 2 Done Done New false
22936 8287 Acked 9286 Read 7 Done Done New false
22937 8288 Acked 9287 Write 10 Done Done New false
22938 8289 Acked 9288 Write 1 Done Done New false
22939 8290 Acked 9289 Write 4 Done Done New false
22940 8291 Acked 9290 Write 6 Done Done New false
22941 8292 Acked 9291 Read 7 Done Done New false
22942 8293 Acked 9292 Read 6 Done Done New false
22943 8294 Acked 9293 Read 8 Done Done New false
22944 8295 Acked 9294 Read 6 Done Done New false
22945 8296 Acked 9295 Flush 0 Done Done New false
22946 8297 Acked 9296 Read 3 Done Done New false
22947 8298 Acked 9297 Read 7 Done Done New false
22948 8299 Acked 9298 Flush 0 Done Done New false
22949 8300 Acked 9299 Read 7 Done Done New false
22950 8301 Acked 9300 Read 2 Done Done New false
22951 8302 Acked 9301 Write 9 Done Done New false
22952 8303 Acked 9302 Flush 0 Done Done New false
22953 8304 Acked 9303 Read 8 Done Done New false
22954 8305 Acked 9304 Read 3 Done Done New false
22955 8306 Acked 9305 Write 4 Done Done New false
22956 8307 Acked 9306 Write 9 Done Done New false
22957 8308 Acked 9307 Read 1 Done Done New false
22958 8309 Acked 9308 Write 2 Done Done New false
22959 8310 Acked 9309 Flush 0 Done Done New false
22960 8311 Acked 9310 Read 3 Done Done New false
22961 8312 Acked 9311 Write 8 Done Done New false
22962 8313 Acked 9312 Read 1 Done Done New false
22963 8314 Acked 9313 Write 8 Done Done New false
22964 8315 Acked 9314 Write 6 Done Done New false
22965 8316 Acked 9315 Read 2 Done Done New false
22966 8317 Acked 9316 Write 7 Done Done New false
22967 8318 Acked 9317 Read 9 Done Done New false
22968 8319 Acked 9318 Read 8 Done Done New false
22969 8320 Acked 9319 Write 4 Done Done New false
22970 8321 Acked 9320 Read 3 Done Done New false
22971 8322 Acked 9321 Read 1 Done Done New false
22972 8323 Acked 9322 Read 1 Done Done New false
22973 8324 Acked 9323 Read 5 Done Done New false
22974 8325 Acked 9324 Flush 0 Done Done New false
22975 8326 Acked 9325 Write 3 Done Done New false
22976 8327 Acked 9326 Read 10 Done Done New false
22977 8328 Acked 9327 Read 4 Done Done New false
22978 8329 Acked 9328 Write 3 Done Done New false
22979 8330 Acked 9329 Write 1 Done Done New false
22980 8331 Acked 9330 Read 10 Done Done New false
22981 8332 Acked 9331 Read 2 Done Done New false
22982 8333 Acked 9332 Read 8 Done Done New false
22983 8334 Acked 9333 Write 4 Done Done New false
22984 8335 Acked 9334 Write 2 Done Done New false
22985 8336 Acked 9335 Read 8 Done Done New false
22986 8337 Acked 9336 Read 5 Done Done New false
22987 8338 Acked 9337 Write 1 Done Done New false
22988 8339 Acked 9338 Read 8 Done Done New false
22989 8340 Acked 9339 Write 3 Done Done New false
22990 8341 Acked 9340 Read 2 Done Done New false
22991 8342 Acked 9341 Write 10 Done Done New false
22992 8343 Acked 9342 Write 10 Done Done New false
22993 8344 Acked 9343 Write 7 Done Done New false
22994 8345 Acked 9344 Write 3 Done Done New false
22995 8346 Acked 9345 Read 3 Done Done New false
22996 8347 Acked 9346 Write 2 Done Done New false
22997 8348 Acked 9347 Write 8 Done Done New false
22998 8349 Acked 9348 Write 6 Done Done New false
22999 8350 Acked 9349 Flush 0 Done Done New false
23000 8351 Acked 9350 Write 3 Done Done New false
23001 8352 Acked 9351 Write 7 Done Done New false
23002 8353 Acked 9352 Read 8 Done Done New false
23003 8354 Acked 9353 Read 3 Done Done New false
23004 8355 Acked 9354 Read 6 Done Done New false
23005 8356 Acked 9355 Write 1 Done Done New false
23006 8357 Acked 9356 Read 6 Done Done New false
23007 8358 Acked 9357 Read 8 Done Done New false
23008 8359 Acked 9358 Read 10 Done Done New false
23009 8360 Acked 9359 Write 8 Done Done New false
23010 8361 Acked 9360 Read 10 Done Done New false
23011 8362 Acked 9361 Read 3 Done Done New false
23012 8363 Acked 9362 Read 5 Done Done New false
23013 8364 Acked 9363 Read 2 Done Done New false
23014 8365 Acked 9364 Read 2 Done Done New false
23015 8366 Acked 9365 Write 9 Done Done New false
23016 8367 Acked 9366 Read 10 Done Done New false
23017 8368 Acked 9367 Read 6 Done Done New false
23018 8369 Acked 9368 Read 8 Done Done New false
23019 8370 Acked 9369 Write 4 Done Done New false
23020 8371 Acked 9370 Read 9 Done Done New false
23021 8372 Acked 9371 Write 3 Done Done New false
23022 8373 Acked 9372 Read 9 Done Done New false
23023 8374 Acked 9373 Write 10 Done Done New false
23024 8375 Acked 9374 Write 6 Done Done New false
23025 8376 Acked 9375 Write 3 Done Done New false
23026 8377 Acked 9376 Flush 0 Done Done New false
23027 8378 Acked 9377 Read 6 Done Done New false
23028 8379 Acked 9378 Read 9 Done Done New false
23029 8380 Acked 9379 Write 7 Done Done New false
23030 8381 Acked 9380 Read 1 Done Done New false
23031 8382 Acked 9381 Read 7 Done Done New false
23032 8383 Acked 9382 Write 3 Done Done New false
23033 8384 Acked 9383 Read 1 Done Done New false
23034 8385 Acked 9384 Write 8 Done Done New false
23035 8386 Acked 9385 Write 9 Done Done New false
23036 8387 Acked 9386 Write 8 Done Done New false
23037 8388 Acked 9387 Flush 0 Done Done New false
23038 8389 Acked 9388 Write 2 Done Done New false
23039 8390 Acked 9389 Read 1 Done Done New false
23040 8391 Acked 9390 Flush 0 Done Done New false
23041 8392 Acked 9391 Read 10 Done Done New false
23042 8393 Acked 9392 Write 7 Done Done New false
23043 8394 Acked 9393 Write 2 Done Done New false
23044 8395 Acked 9394 Write 7 Done Done New false
23045 8396 Acked 9395 Read 7 Done Done New false
23046 8397 Acked 9396 Read 10 Done Done New false
23047 8398 Acked 9397 Flush 0 Done Done New false
23048 8399 Acked 9398 Write 2 Done Done New false
23049 8400 Acked 9399 Read 5 Done Done New false
23050 8401 Acked 9400 Write 5 Done Done New false
23051 8402 Acked 9401 Flush 0 Done Done New false
23052 8403 Acked 9402 Write 9 Done Done New false
23053 8404 Acked 9403 Flush 0 Done Done New false
23054 8405 Acked 9404 Write 1 Done Done New false
23055 8406 Acked 9405 Write 7 Done Done New false
23056 8407 Acked 9406 Read 4 Done Done New false
23057 8408 Acked 9407 Read 10 Done Done New false
23058 8409 Acked 9408 Write 4 Done Done New false
23059 8410 Acked 9409 Write 10 Done Done New false
23060 8411 Acked 9410 Write 8 Done Done New false
23061 8412 Acked 9411 Read 3 Done Done New false
23062 8413 Acked 9412 Write 8 Done Done New false
23063 8414 Acked 9413 Write 8 Done Done New false
23064 8415 Acked 9414 Read 3 Done Done New false
23065 8416 Acked 9415 Write 1 Done Done New false
23066 8417 Acked 9416 Write 8 Done Done New false
23067 8418 Acked 9417 Flush 0 Done Done New false
23068 8419 Acked 9418 Read 9 Done Done New false
23069 8420 Acked 9419 Read 10 Done Done New false
23070 8421 Acked 9420 Flush 0 Done Done New false
23071 8422 Acked 9421 Write 8 Done Done New false
23072 8423 Acked 9422 Read 8 Done Done New false
23073 8424 Acked 9423 Read 4 Done Done New false
23074 8425 Acked 9424 Read 7 Done Done New false
23075 8426 Acked 9425 Write 3 Done Done New false
23076 8427 Acked 9426 Read 9 Done Done New false
23077 8428 Acked 9427 Read 1 Done Done New false
23078 8429 Acked 9428 Flush 0 Done Done New false
23079 8430 Acked 9429 Read 3 Done Done New false
23080 8431 Acked 9430 Read 7 Done Done New false
23081 8432 Acked 9431 Read 2 Done Done New false
23082 8433 Acked 9432 Write 2 Done Done New false
23083 8434 Acked 9433 Read 7 Done Done New false
23084 8435 Acked 9434 Read 7 Done Done New false
23085 8436 Acked 9435 Read 3 Done Done New false
23086 8437 Acked 9436 Flush 0 Done Done New false
23087 8438 Acked 9437 Read 2 Done Done New false
23088 8439 Acked 9438 Write 9 Done Done New false
23089 8440 Acked 9439 Write 10 Done Done New false
23090 8441 Acked 9440 Read 5 Done Done New false
23091 8442 Acked 9441 Read 5 Done Done New false
23092 8443 Acked 9442 Read 10 Done Done New false
23093 8444 Acked 9443 Flush 0 Done Done New false
23094 8445 Acked 9444 Read 7 Done Done New false
23095 8446 Acked 9445 Write 6 Done Done New false
23096 8447 Acked 9446 Write 6 Done Done New false
23097 8448 Acked 9447 Read 6 Done Done New false
23098 8449 Acked 9448 Write 1 Done Done New false
23099 8450 Acked 9449 Write 2 Done Done New false
23100 8451 Acked 9450 Read 9 Done Done New false
23101 8452 Acked 9451 Write 4 Done Done New false
23102 8453 Acked 9452 Write 1 Done Done New false
23103 8454 Acked 9453 Read 5 Done Done New false
23104 8455 Acked 9454 Read 9 Done Done New false
23105 8456 Acked 9455 Read 6 Done Done New false
23106 8457 Acked 9456 Read 10 Done Done New false
23107 8458 Acked 9457 Read 8 Done Done New false
23108 8459 Acked 9458 Read 6 Done Done New false
23109 8460 Acked 9459 Read 5 Done Done New false
23110 8461 Acked 9460 Read 8 Done Done New false
23111 8462 Acked 9461 Write 8 Done Done New false
23112 8463 Acked 9462 Read 6 Done Done New false
23113 8464 Acked 9463 Read 3 Done Done New false
23114 8465 Acked 9464 Write 1 Done Done New false
23115 8466 Acked 9465 Read 10 Done Done New false
23116 8467 Acked 9466 Write 5 Done Done New false
23117 8468 Acked 9467 Write 3 Done Done New false
23118 8469 Acked 9468 Write 3 Done Done New false
23119 8470 Acked 9469 Write 10 Done Done New false
23120 8471 Acked 9470 Write 8 Done Done New false
23121 8472 Acked 9471 Read 4 Done Done New false
23122 8473 Acked 9472 Write 10 Done Done New false
23123 8474 Acked 9473 Read 1 Done Done New false
23124 8475 Acked 9474 Write 1 Sent Sent New false
23125 STATES DS:0 DS:1 DS:2 TOTAL
23126 New 0 0 302 302
23127 Sent 1 1 0 2
23128 Done 301 301 0 602
23129 Skipped 0 0 0 0
23130 Error 0 0 0 0
23131 Last Flush: 9443 9443 9172
23132 Downstairs last five completed: 9172 9171 9170 9169 9168 9167
23133 Upstairs last five completed: 8475 8474 8473 8472 8471 8470
23134 CLIENT: Up:0 ds:302 act:2
23135 [8830] got start action
23136 [8830] I am going to restart
23137 Starting downstairs at port 8830
23138 Make output file at "/tmp/dsc/downstairs-8830.txt"
23139 [8830][2] reports Starting
23140 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1204)
23141 [8830][2] reports Running
231422023-10-13T18:14:08.506ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 2
231432023-10-13T18:14:08.506ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
231442023-10-13T18:14:08.506ZINFOcrucible: [2] client re-new 303 jobs since flush 9172 = downstairs
231452023-10-13T18:14:08.507ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
231462023-10-13T18:14:08.508ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
231472023-10-13T18:14:08.508ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
231482023-10-13T18:14:08.508ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
231492023-10-13T18:14:08.508ZINFOcrucible: [2] send last flush ID to this DS: 9172
231502023-10-13T18:14:08.508ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
231512023-10-13T18:14:08.508ZINFOcrucible: [2] Replied this last flush ID: 9172
231522023-10-13T18:14:08.508ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Offline ds_transition to Replay
231532023-10-13T18:14:08.508ZINFOcrucible: [2] Transition from Offline to Replay
231542023-10-13T18:14:08.508ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
231552023-10-13T18:14:08.508ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Replay ds_transition to Active
231562023-10-13T18:14:08.508ZINFOcrucible: [2] Transition from Replay to Active
231572023-10-13T18:14:08.508ZINFOcrucible: [2] Starts cmd_loop
231582023-10-13T18:14:08.508ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
231592023-10-13T18:14:08.508ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
231602023-10-13T18:14:08.508ZINFOcrucible: Set check for repair
231612023-10-13T18:14:09.509ZINFOcrucible: Checking if live repair is needed
231622023-10-13T18:14:09.509ZINFOcrucible: No Live Repair required at this time
231632023-10-13T18:14:09.509ZWARNcrucible: [2] flow control end
23164 ----------------------------------------------------------------
23165 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23166 Downstairs last five completed: 9475 9474 9473 9472 9471 9470
23167 Upstairs last five completed: 8476 8475 8474 8473 8472 8471
23168 CLIENT: Up:0 ds:0 act:3
23169 Replay: All jobs finished, all DS active.
23170 Main task has work to do, go find it
23171 got dsc Stop(0)
23172 stop 0
23173 Oct 13 18:14:10.472 INFO request completed, latency_us: 558, response_code: [8810] Got stop action so:false kr:false
23174 204, uri: /stop/cid/0, method: POST, req_id: e3b3c291-364a-4107-bc53-05adc2a79e0a, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
23175 Oct 13 18:14:10.473 INFO accepted connection, remote_addr: 127.0.0.1:36157, local_addr: 127.0.0.1:9998
23176 Oct 13 18:14:10.473 INFO request completed, latency_us: 174, response_code{"msg":"[0] downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2023-10-13T18:14:10.473865435Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1176}
23177 : 200, uri: /state/cid/0, method: GET, req_id: 22d37233-99ed-44fc-9a2a-f5f6b6627814, remote_addr{"msg":"[0] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2023-10-13T18:14:10.473972649Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1176}
23178 :{"msg":"[0] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8810 closed","v":0,"name":"crucible","level":30,"time":"2023-10-13T18:14:10.474044815Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1176,"looper":"0"}
23179 127.0{"msg":"[0] 127.0.0.1:8810 task reports connection:false","v":0,"name":"crucible","level":30,"time":"2023-10-13T18:14:10.474191168Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1176}
23180 .0.1:64571, local_addr: 127.0.0.1:9998
231812023-10-13T18:14:10.474ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
231822023-10-13T18:14:10.474ZINFOcrucible: [0] 127.0.0.1:8810 task reports offline
231832023-10-13T18:14:10.474ZWARNcrucible: [0] pm_task rx.recv() is None
23184 [8810] Exited with: ExitStatus(unix_wait_status(9))
23185 [8810][0] reports Exit
231862023-10-13T18:14:13.479ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
23187 Oct 13 18:14:14.475 INFO request completed, latency_us: 259, response_code: 200, uri: /state/cid/0, method: GET, req_id: 265fd86c-2a6a-4ce3-ab60-db1168cf0ca5, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
23188 001/300 Flush
23189 002/300 Write block 4466 len 4096 data: 5
23190 003/300 Flush
23191 004/300 Write block 4749 len 40960 data: 8 7 9 8 9 10 11 10 10 7
23192 005/300 Read block 468 len 8192
23193 006/300 Read block 258 len 24576
23194 007/300 Read block 1851 len 16384
23195 008/300 Write block 1166 len 36864 data: 7 6 5 5 6 6 7 7 6
23196 009/300 Read block 3993 len 20480
23197 010/300 Write block 248 len 40960 data: 6 5 6 6 5 6 7 6 4 4
23198 011/300 Read block 1769 len 24576
23199 012/300 Write block 1848 len 32768 data: 5 5 5 6 7 9 9 8
23200 013/300 Write block 4344 len 28672 data: 4 6 6 7 5 4 2
23201 014/300 Write block 4058 len 16384 data: 9 8 7 6
23202 015/300 Read block 4117 len 4096
23203 016/300 Read block 3141 len 24576
23204 017/300 Read block 1765 len 32768
23205 018/300 Write block 2092 len 32768 data: 5 4 5 4 5 6 6 7
23206 019/300 Flush
23207 020/300 Read block 1061 len 36864
23208 021/300 Read block 1591 len 24576
23209 022/300 Write block 1280 len 32768 data: 4 3 5 5 5 4 4 4
23210 023/300 Write block 978 len 8192 data: 5 4
23211 024/300 Write block 29 len 20480 data: 6 6 6 6 6
23212 025/300 Write block 335 len 24576 data: 6 7 6 5 6 5
23213 026/300 Read block 3480 len 4096
23214 027/300 Read block 4921 len 40960
23215 028/300 Write block 4035 len 12288 data: 2 2 2
23216 029/300 Flush
23217 030/300 Read block 3637 len 36864
23218 031/300 Read block 3644 len 20480
23219 032/300 Read block 807 len 40960
23220 033/300 Write block 4326 len 24576 data: 10 8 7 9 10 10
23221 034/300 Write block 561 len 20480 data: 5 5 5 5 4
23222 035/300 Write block 3657 len 16384 data: 7 6 5 7
23223 036/300 Read block 2380 len 8192
23224 037/300 Read block 4013 len 12288
23225 038/300 Read block 2172 len 32768
23226 039/300 Read block 451 len 36864
23227 040/300 Write block 3181 len 28672 data: 8 7 7 6 4 5 5
23228 041/300 Read block 3668 len 4096
23229 042/300 Read block 2091 len 36864
23230 043/300 Write block 1369 len 36864 data: 7 9 9 8 7 8 5 8 6
23231 044/300 Write block 4022 len 16384 data: 4 5 5 5
23232 045/300 Read block 3899 len 20480
23233 046/300 Write block 1389 len 20480 data: 6 7 7 7 7
23234 047/300 Read block 422 len 12288
23235 048/300 Read block 4243 len 16384
23236 049/300 Flush
23237 050/300 Flush
23238 051/300 Read block 4347 len 4096
23239 052/300 Flush
23240 053/300 Read block 3595 len 20480
23241 054/300 Write block 1513 len 16384 data: 3 4 4 3
23242 055/300 Read block 4691 len 36864
23243 056/300 Write block 2747 len 28672 data: 6 6 6 5 5 5 5
23244 057/300 Write block 469 len 40960 data: 4 4 5 6 6 4 3 5 5 4
23245 058/300 Flush
23246 059/300 Read block 4802 len 40960
23247 060/300 Read block 1421 len 24576
23248 061/300 Read block 2644 len 40960
23249 062/300 Read block 3218 len 28672
23250 063/300 Write block 2018 len 24576 data: 7 8 8 8 9 7
23251 064/300 Write block 1573 len 8192 data: 3 5
23252 065/300 Read block 127 len 16384
23253 066/300 Read block 3267 len 28672
23254 067/300 Read block 4451 len 32768
23255 068/300 Read block 3571 len 16384
23256 069/300 Read block 681 len 8192
23257 070/300 Read block 2553 len 4096
23258 071/300 Read block 973 len 8192
23259 072/300 Write block 4766 len 20480 data: 5 5 6 5 4
23260 073/300 Flush
23261 074/300 Write block 927 len 16384 data: 4 6 7 7
23262 075/300 Read block 617 len 40960
23263 076/300 Read block 2757 len 12288
23264 077/300 Read block 280 len 32768
23265 078/300 Read block 3191 len 28672
23266 079/300 Write block 3027 len 4096 data: 7
23267 080/300 Write block 3847 len 36864 data: 5 4 4 4 4 4 4 5 5
23268 081/300 Read block 4227 len 8192
23269 082/300 Read block 2304 len 24576
23270 083/300 Flush
23271 084/300 Read block 4068 len 20480
23272 085/300 Read block 3640 len 36864
23273 086/300 Read block 4988 len 40960
23274 087/300 Read block 787 len 28672
23275 088/300 Read block 438 len 40960
23276 089/300 Read block 3674 len 16384
23277 090/300 Write block 2926 len 32768 data: 4 4 4 4 5 7 6 5
23278 091/300 Write block 3270 len 40960 data: 4 4 4 3 3 3 3 4 4 4
23279 092/300 Write block 340 len 36864 data: 6 4 4 5 6 6 7 7 6
23280 093/300 Read block 3549 len 36864
23281 094/300 Write block 4568 len 32768 data: 5 6 5 6 6 6 5 5
23282 095/300 Write block 2506 len 32768 data: 7 8 11 11 9 9 8 7
23283 096/300 Write block 2097 len 8192 data: 7 7
23284 097/300 Read block 782 len 36864
23285 098/300 Write block 512 len 24576 data: 8 8 8 8 6 5
23286 099/300 Write block 3069 len 16384 data: 4 5 5 5
23287 100/300 Read block 3920 len 32768
23288 101/300 Flush
23289 102/300 Write block 4367 len 20480 data: 6 5 6 6 7
23290 103/300 Read block 3453 len 8192
23291 104/300 Read block 3216 len 16384
23292 105/300 Read block 1082 len 24576
23293 106/300 Write block 2553 len 12288 data: 8 6 6
23294 107/300 Read block 3518 len 20480
23295 108/300 Read block 2444 len 8192
23296 109/300 Write block 1699 len 12288 data: 5 4 4
23297 110/300 Read block 2040 len 12288
23298 111/300 Write block 3235 len 8192 data: 7 9
23299 112/300 Read block 3498 len 4096
23300 113/300 Read block 2597 len 8192
23301 114/300 Flush
23302 115/300 Read block 4935 len 32768
23303 116/300 Read block 3736 len 32768
23304 117/300 Read block 427 len 4096
23305 118/300 Read block 1598 len 16384
23306 119/300 Read block 4257 len 16384
23307 120/300 Read block 1164 len 12288
23308 121/300 Write block 1481 len 40960 data: 6 6 6 6 6 7 9 9 10 9
23309 122/300 Write block 4662 len 8192 data: 5 5
23310 123/300 Write block 2278 len 32768 data: 6 7 8 8 8 7 6 6
23311 124/300 Write block 2669 len 16384 data: 4 4 5 4
23312 125/300 Write block 1192 len 32768 data: 7 7 5 4 6 6 7 6
23313 126/300 Flush
23314 127/300 Flush
23315 128/300 Write block 1481 len 36864 data: 7 7 7 7 7 8 10 10 11
23316 129/300 Write block 725 len 20480 data: 5 5 5 4 3
23317 130/300 Read block 2326 len 24576
23318 131/300 Read block 1347 len 16384
23319 132/300 Read block 4212 len 20480
23320 133/300 Read block 4144 len 36864
23321 134/300 Write block 3968 len 40960 data: 4 5 5 3 2 2 2 2 3 4
23322 135/300 Read block 1726 len 4096
23323 136/300 Read block 4230 len 28672
23324 137/300 Read block 4436 len 4096
23325 138/300 Write block 2516 len 28672 data: 4 4 3 2 4 5 4
23326 139/300 Read block 3906 len 8192
23327 140/300 Read block 25 len 8192
23328 141/300 Read block 138 len 36864
23329 142/300 Flush
23330 143/300 Read block 922 len 20480
23331 144/300 Flush
23332 145/300 Read block 2649 len 28672
23333 146/300 Flush
23334 147/300 Flush
23335 148/300 Write block 3464 len 32768 data: 5 4 3 3 3 3 3 4
23336 149/300 Read block 1185 len 32768
23337 150/300 Write block 1234 len 24576 data: 7 6 5 6 8 8
23338 151/300 Write block 3141 len 36864 data: 9 9 10 8 12 14 10 6 5
23339 152/300 Read block 2543 len 16384
23340 153/300 Read block 1887 len 36864
23341 154/300 Read block 1926 len 40960
23342 155/300 Read block 4818 len 36864
23343 156/300 Read block 363 len 36864
23344 157/300 Write block 984 len 32768 data: 4 6 5 5 5 5 5 5
23345 158/300 Write block 1538 len 28672 data: 6 6 4 6 6 7 5
23346 159/300 Read block 1528 len 32768
23347 160/300 Write block 826 len 28672 data: 6 7 7 5 7 5 4
23348 161/300 Write block 4405 len 12288 data: 8 10 7
23349 162/300 Read block 1453 len 36864
23350 163/300 Write block 3958 len 16384 data: 7 7 7 5
23351 164/300 Read block 206 len 28672
23352 165/300 Read block 3861 len 28672
23353 166/300 Read block 4443 len 36864
23354 167/300 Write block 3571 len 24576 data: 9 8 9 9 8 5
23355 168/300 Read block 1138 len 40960
23356 169/300 Flush
23357 170/300 Read block 866 len 20480
23358 171/300 Write block 2936 len 4096 data: 4
23359 172/300 Write block 3890 len 8192 data: 6 7
23360 173/300 Write block 256 len 12288 data: 5 5 3
23361 174/300 Write block 3338 len 32768 data: 8 7 5 3 4 5 3 3
23362 175/300 Read block 607 len 40960
23363 176/300 Read block 3152 len 20480
23364 177/300 Write block 811 len 36864 data: 6 6 7 6 4 4 5 6 7
23365 178/300 Read block 4363 len 28672
23366 179/300 Read block 939 len 4096
23367 180/300 Write block 396 len 12288 data: 2 2 2
23368 181/300 Write block 725 len 20480 data: 6 6 6 5 4
23369 182/300 Read block 4088 len 28672
23370 183/300 Write block 3601 len 32768 data: 5 5 5 5 5 6 5 6
23371 184/300 Read block 3669 len 12288
23372 185/300 Read block 1970 len 28672
23373 186/300 Write block 2752 len 28672 data: 6 6 7 3 3 4 5
23374 187/300 Read block 124 len 28672
23375 188/300 Read block 1589 len 16384
23376 189/300 Read block 3680 len 28672
23377 190/300 Write block 2160 len 16384 data: 5 6 5 5
23378 191/300 Flush
23379 192/300 Write block 3373 len 12288 data: 5 4 5
23380 193/300 Read block 3495 len 4096
23381 194/300 Read block 1475 len 24576
23382 195/300 Read block 1994 len 32768
23383 196/300 Flush
23384 197/300 Read block 3633 len 16384
23385 198/300 Write block 1434 len 8192 data: 6 5
23386 199/300 Read block 2194 len 4096
23387 200/300 Read block 3016 len 36864
23388 201/300 Read block 2780 len 16384
23389 202/300 Write block 1324 len 36864 data: 3 5 5 5 5 4 5 6 7
23390 203/300 Read block 3806 len 40960
23391 204/300 Read block 1456 len 32768
23392 205/300 Flush
23393 206/300 Read block 4471 len 4096
23394 207/300 Read block 3754 len 24576
23395 208/300 Read block 3068 len 12288
23396 209/300 Read block 1461 len 36864
23397 210/300 Read block 1799 len 24576
23398 211/300 Read block 3944 len 8192
23399 212/300 Read block 4581 len 36864
23400 213/300 Read block 1463 len 20480
23401 214/300 Flush
23402 215/300 Read block 3094 len 40960
23403 216/300 Read block 3293 len 36864
23404 217/300 Write block 3214 len 24576 data: 6 5 7 5 5 5
23405 218/300 Flush
23406 219/300 Read block 679 len 24576
23407 220/300 Write block 3069 len 4096 data: 5
23408 221/300 Write block 3783 len 20480 data: 6 6 5 4 3
23409 222/300 Read block 1582 len 4096
23410 223/300 Read block 3368 len 4096
23411 224/300 Read block 1741 len 20480
23412 225/300 Write block 4678 len 32768 data: 9 8 8 8 7 5 6 6
23413 226/300 Read block 2842 len 40960
23414 227/300 Read block 4876 len 32768
23415 228/300 Flush
23416 229/300 Read block 1319 len 4096
23417 230/300 Read block 4821 len 12288
23418 231/300 Read block 3710 len 32768
23419 232/300 Write block 160 len 16384 data: 5 6 8 7
23420 233/300 Read block 3824 len 32768
23421 234/300 Read block 1229 len 40960
23422 235/300 Read block 4432 len 20480
23423 236/300 Read block 2791 len 40960
23424 237/300 Write block 3081 len 4096 data: 3
23425 238/300 Read block 2742 len 20480
23426 239/300 Write block 327 len 28672 data: 8 6 5 6 6 6 6
23427 240/300 Read block 100 len 36864
23428 241/300 Read block 1609 len 12288
23429 242/300 Write block 1358 len 36864 data: 4 3 3 3 3 4 4 4 7
23430 243/300 Read block 1857 len 20480
23431 244/300 Read block 87 len 28672
23432 245/300 Read block 1616 len 32768
23433 246/300 Read block 2560 len 20480
23434 247/300 Write block 4975 len 8192 data: 7 7
23435 248/300 Flush
23436 249/300 Write block 3287 len 28672 data: 7 7 4 4 4 4 5
23437 250/300 Write block 3471 len 32768 data: 5 4 6 8 8 9 8 7
23438 251/300 Write block 3786 len 16384 data: 5 4 3 3
23439 252/300 Write block 3469 len 36864 data: 4 4 6 5 7 9 9 10 9
23440 253/300 Write block 221 len 24576 data: 4 5 6 5 5 4
23441 254/300 Read block 3241 len 36864
23442 255/300 Read block 309 len 4096
23443 256/300 Read block 2468 len 32768
23444 257/300 Write block 3977 len 32768 data: 5 5 5 6 7 6 7 7
23445 258/300 Write block 2601 len 32768 data: 11 12 11 9 10 9 6 5
23446 259/300 Write block 4734 len 28672 data: 4 6 6 5 4 4 7
23447 260/300 Flush
23448 261/300 Flush
23449 262/300 Read block 4364 len 28672
23450 263/300 Read block 1147 len 28672
23451 264/300 Read block 310 len 8192
23452 265/300 Write block 2000 len 4096 data: 4
23453 266/300 Write block 3769 len 40960 data: 5 4 4 3 3 3 5 6 6 7
23454 267/300 Read block 4857 len 4096
23455 268/300 Read block 4931 len 8192
23456 269/300 Read block 2770 len 40960
23457 270/300 Write block 3420 len 20480 data: 4 4 4 5 6
23458 271/300 Read block 2059 len 24576
23459 272/300 Write block 413 len 32768 data: 6 4 5 5 7 7 6 6
23460 273/300 Read block 4607 len 8192
23461 274/300 Flush
23462 275/300 Write block 288 len 24576 data: 4 4 3 3 3 3
23463 276/300 Read block 4328 len 8192
23464 277/300 Write block 3643 len 8192 data: 4 5
23465 278/300 Flush
23466 279/300 Read block 4480 len 12288
23467 280/300 Flush
23468 281/300 Write block 4042 len 4096 data: 7
23469 282/300 Flush
23470 283/300 Write block 3812 len 40960 data: 4 4 4 3 4 4 5 5 6 6
23471 284/300 Read block 3207 len 40960
23472 285/300 Read block 927 len 32768
23473 286/300 Write block 4688 len 36864 data: 4 4 6 5 5 5 5 4 4
23474 287/300 Flush
23475 288/300 Write block 1338 len 24576 data: 5 7 6 7 8 9
23476 289/300 Write block 3537 len 40960 data: 9 12 12 9 9 9 8 8 10 9
23477 290/300 Read block 2701 len 12288
23478 291/300 Flush
23479 292/300 Read block 2843 len 16384
23480 293/300 Write block 4517 len 8192 data: 3 2
23481 294/300 Write block 2450 len 28672 data: 5 6 5 5 5 5 5
23482 295/300 Write block 4788 len 24576 data: 8 10 7 8 8 8
23483 296/300 Read block 3179 len 16384
23484 297/300 Read block 3446 len 24576
23485 298/300 Write block 4368 len 24576 data: 6 7 7 8 8 9
23486 299/300 Read block 1716 len 20480
23487 300/300 Read block 4495 len 4096
23488 Main task has work to do, go find it
23489 got dsc Start(0)
23490 start 0
23491 Oct 13 18:14:15.441 INFO request completed, latency_us: 305, response_code: 204, uri: /start/cid/0, method: POST, req_id: 49815e0e-6b56-442d-88be-4a3052a721ba, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
23492 Replay: started 0, returned:Ok(())
23493 ----------------------------------------------------------------
23494 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23495 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
23496 8477 Acked 9476 Flush 0 New Done Done false
23497 8478 Acked 9477 Write 1 New Done Done false
23498 8479 Acked 9478 Flush 0 New Done Done false
23499 8480 Acked 9479 Write 10 New Done Done false
23500 8481 Acked 9480 Read 2 New Done Done false
23501 8482 Acked 9481 Read 6 New Done Done false
23502 8483 Acked 9482 Read 4 New Done Done false
23503 8484 Acked 9483 Write 9 New Done Done false
23504 8485 Acked 9484 Read 5 New Done Done false
23505 8486 Acked 9485 Write 10 New Done Done false
23506 8487 Acked 9486 Read 6 New Done Done false
23507 8488 Acked 9487 Write 8 New Done Done false
23508 8489 Acked 9488 Write 7 New Done Done false
23509 8490 Acked 9489 Write 4 New Done Done false
23510 8491 Acked 9490 Read 1 New Done Done false
23511 8492 Acked 9491 Read 6 New Done Done false
23512 8493 Acked 9492 Read 8 New Done Done false
23513 8494 Acked 9493 Write 8 New Done Done false
23514 8495 Acked 9494 Flush 0 New Done Done false
23515 8496 Acked 9495 Read 9 New Done Done false
23516 8497 Acked 9496 Read 6 New Done Done false
23517 8498 Acked 9497 Write 8 New Done Done false
23518 8499 Acked 9498 Write 2 New Done Done false
23519 8500 Acked 9499 Write 5 New Done Done false
23520 8501 Acked 9500 Write 6 New Done Done false
23521 8502 Acked 9501 Read 1 New Done Done false
23522 8503 Acked 9502 Read 10 New Done Done false
23523 8504 Acked 9503 Write 3 New Done Done false
23524 8505 Acked 9504 Flush 0 New Done Done false
23525 8506 Acked 9505 Read 9 New Done Done false
23526 8507 Acked 9506 Read 5 New Done Done false
23527 8508 Acked 9507 Read 10 New Done Done false
23528 8509 Acked 9508 Write 6 New Done Done false
23529 8510 Acked 9509 Write 5 New Done Done false
23530 8511 Acked 9510 Write 4 New Done Done false
23531 8512 Acked 9511 Read 2 New Done Done false
23532 8513 Acked 9512 Read 3 New Done Done false
23533 8514 Acked 9513 Read 8 New Done Done false
23534 8515 Acked 9514 Read 9 New Done Done false
23535 8516 Acked 9515 Write 7 New Done Done false
23536 8517 Acked 9516 Read 1 New Done Done false
23537 8518 Acked 9517 Read 9 New Done Done false
23538 8519 Acked 9518 Write 9 New Done Done false
23539 8520 Acked 9519 Write 4 New Done Done false
23540 8521 Acked 9520 Read 5 New Done Done false
23541 8522 Acked 9521 Write 5 New Done Done false
23542 8523 Acked 9522 Read 3 New Done Done false
23543 8524 Acked 9523 Read 4 New Done Done false
23544 8525 Acked 9524 Flush 0 New Done Done false
23545 8526 Acked 9525 Flush 0 New Done Done false
23546 8527 Acked 9526 Read 1 New Done Done false
23547 8528 Acked 9527 Flush 0 New Done Done false
23548 8529 Acked 9528 Read 5 New Done Done false
23549 8530 Acked 9529 Write 4 New Done Done false
23550 8531 Acked 9530 Read 9 New Done Done false
23551 8532 Acked 9531 Write 7 New Done Done false
23552 8533 Acked 9532 Write 10 New Done Done false
23553 8534 Acked 9533 Flush 0 New Done Done false
23554 8535 Acked 9534 Read 10 New Done Done false
23555 8536 Acked 9535 Read 6 New Done Done false
23556 8537 Acked 9536 Read 10 New Done Done false
23557 8538 Acked 9537 Read 7 New Done Done false
23558 8539 Acked 9538 Write 6 New Done Done false
23559 8540 Acked 9539 Write 2 New Done Done false
23560 8541 Acked 9540 Read 4 New Done Done false
23561 8542 Acked 9541 Read 7 New Done Done false
23562 8543 Acked 9542 Read 8 New Done Done false
23563 8544 Acked 9543 Read 4 New Done Done false
23564 8545 Acked 9544 Read 2 New Done Done false
23565 8546 Acked 9545 Read 1 New Done Done false
23566 8547 Acked 9546 Read 2 New Done Done false
23567 8548 Acked 9547 Write 5 New Done Done false
23568 8549 Acked 9548 Flush 0 New Done Done false
23569 8550 Acked 9549 Write 4 New Done Done false
23570 8551 Acked 9550 Read 10 New Done Done false
23571 8552 Acked 9551 Flush 0 New Done Done false
23572 8553 Acked 9552 Read 3 New Done Done false
23573 8554 Acked 9553 Read 8 New Done Done false
23574 8555 Acked 9554 Read 7 New Done Done false
23575 8556 Acked 9555 Write 1 New Done Done false
23576 8557 Acked 9556 Write 9 New Done Done false
23577 8558 Acked 9557 Read 2 New Done Done false
23578 8559 Acked 9558 Read 6 New Done Done false
23579 8560 Acked 9559 Flush 0 New Done Done false
23580 8561 Acked 9560 Read 5 New Done Done false
23581 8562 Acked 9561 Read 9 New Done Done false
23582 8563 Acked 9562 Read 10 New Done Done false
23583 8564 Acked 9563 Read 7 New Done Done false
23584 8565 Acked 9564 Read 10 New Done Done false
23585 8566 Acked 9565 Read 4 New Done Done false
23586 8567 Acked 9566 Write 8 New Done Done false
23587 8568 Acked 9567 Write 10 New Done Done false
23588 8569 Acked 9568 Write 9 New Done Done false
23589 8570 Acked 9569 Read 9 New Done Done false
23590 8571 Acked 9570 Write 8 New Done Done false
23591 8572 Acked 9571 Write 8 New Done Done false
23592 8573 Acked 9572 Write 2 New Done Done false
23593 8574 Acked 9573 Read 9 New Done Done false
23594 8575 Acked 9574 Write 6 New Done Done false
23595 8576 Acked 9575 Write 4 New Done Done false
23596 8577 Acked 9576 Read 8 New Done Done false
23597 8578 Acked 9577 Flush 0 New Done Done false
23598 8579 Acked 9578 Write 5 New Done Done false
23599 8580 Acked 9579 Read 2 New Done Done false
23600 8581 Acked 9580 Read 4 New Done Done false
23601 8582 Acked 9581 Read 6 New Done Done false
23602 8583 Acked 9582 Write 3 New Done Done false
23603 8584 Acked 9583 Read 5 New Done Done false
23604 8585 Acked 9584 Read 2 New Done Done false
23605 8586 Acked 9585 Write 3 New Done Done false
23606 8587 Acked 9586 Read 3 New Done Done false
23607 8588 Acked 9587 Write 2 New Done Done false
23608 8589 Acked 9588 Read 1 New Done Done false
23609 8590 Acked 9589 Read 2 New Done Done false
23610 8591 Acked 9590 Flush 0 New Done Done false
23611 8592 Acked 9591 Read 8 New Done Done false
23612 8593 Acked 9592 Read 8 New Done Done false
23613 8594 Acked 9593 Read 1 New Done Done false
23614 8595 Acked 9594 Read 4 New Done Done false
23615 8596 Acked 9595 Read 4 New Done Done false
23616 8597 Acked 9596 Read 3 New Done Done false
23617 8598 Acked 9597 Write 10 New Done Done false
23618 8599 Acked 9598 Write 2 New Done Done false
23619 8600 Acked 9599 Write 8 New Done Done false
23620 8601 Acked 9600 Write 4 New Done Done false
23621 8602 Acked 9601 Write 8 New Done Done false
23622 8603 Acked 9602 Flush 0 New Done Done false
23623 8604 Acked 9603 Flush 0 New Done Done false
23624 8605 Acked 9604 Write 9 New Done Done false
23625 8606 Acked 9605 Write 5 New Done Done false
23626 8607 Acked 9606 Read 6 New Done Done false
23627 8608 Acked 9607 Read 4 New Done Done false
23628 8609 Acked 9608 Read 5 New Done Done false
23629 8610 Acked 9609 Read 9 New Done Done false
23630 8611 Acked 9610 Write 10 New Done Done false
23631 8612 Acked 9611 Read 1 New Done Done false
23632 8613 Acked 9612 Read 7 New Done Done false
23633 8614 Acked 9613 Read 1 New Done Done false
23634 8615 Acked 9614 Write 7 New Done Done false
23635 8616 Acked 9615 Read 2 New Done Done false
23636 8617 Acked 9616 Read 2 New Done Done false
23637 8618 Acked 9617 Read 9 New Done Done false
23638 8619 Acked 9618 Flush 0 New Done Done false
23639 8620 Acked 9619 Read 5 New Done Done false
23640 8621 Acked 9620 Flush 0 New Done Done false
23641 8622 Acked 9621 Read 7 New Done Done false
23642 8623 Acked 9622 Flush 0 New Done Done false
23643 8624 Acked 9623 Flush 0 New Done Done false
23644 8625 Acked 9624 Write 8 New Done Done false
23645 8626 Acked 9625 Read 8 New Done Done false
23646 8627 Acked 9626 Write 6 New Done Done false
23647 8628 Acked 9627 Write 9 New Done Done false
23648 8629 Acked 9628 Read 4 New Done Done false
23649 8630 Acked 9629 Read 9 New Done Done false
23650 8631 Acked 9630 Read 10 New Done Done false
23651 8632 Acked 9631 Read 9 New Done Done false
23652 8633 Acked 9632 Read 9 New Done Done false
23653 8634 Acked 9633 Write 8 New Done Done false
23654 8635 Acked 9634 Write 7 New Done Done false
23655 8636 Acked 9635 Read 8 New Done Done false
23656 8637 Acked 9636 Write 7 New Done Done false
23657 8638 Acked 9637 Write 3 New Done Done false
23658 8639 Acked 9638 Read 9 New Done Done false
23659 8640 Acked 9639 Write 4 New Done Done false
23660 8641 Acked 9640 Read 7 New Done Done false
23661 8642 Acked 9641 Read 7 New Done Done false
23662 8643 Acked 9642 Read 9 New Done Done false
23663 8644 Acked 9643 Write 6 New Done Done false
23664 8645 Acked 9644 Read 10 New Done Done false
23665 8646 Acked 9645 Flush 0 New Done Done false
23666 8647 Acked 9646 Read 5 New Done Done false
23667 8648 Acked 9647 Write 1 New Done Done false
23668 8649 Acked 9648 Write 2 New Done Done false
23669 8650 Acked 9649 Write 3 New Done Done false
23670 8651 Acked 9650 Write 8 New Done Done false
23671 8652 Acked 9651 Read 10 New Done Done false
23672 8653 Acked 9652 Read 5 New Done Done false
23673 8654 Acked 9653 Write 9 New Done Done false
23674 8655 Acked 9654 Read 7 New Done Done false
23675 8656 Acked 9655 Read 1 New Done Done false
23676 8657 Acked 9656 Write 3 New Done Done false
23677 8658 Acked 9657 Write 5 New Done Done false
23678 8659 Acked 9658 Read 7 New Done Done false
23679 8660 Acked 9659 Write 8 New Done Done false
23680 8661 Acked 9660 Read 3 New Done Done false
23681 8662 Acked 9661 Read 7 New Done Done false
23682 8663 Acked 9662 Write 7 New Done Done false
23683 8664 Acked 9663 Read 7 New Done Done false
23684 8665 Acked 9664 Read 4 New Done Done false
23685 8666 Acked 9665 Read 7 New Done Done false
23686 8667 Acked 9666 Write 4 New Done Done false
23687 8668 Acked 9667 Flush 0 New Done Done false
23688 8669 Acked 9668 Write 3 New Done Done false
23689 8670 Acked 9669 Read 1 New Done Done false
23690 8671 Acked 9670 Read 6 New Done Done false
23691 8672 Acked 9671 Read 8 New Done Done false
23692 8673 Acked 9672 Flush 0 New Done Done false
23693 8674 Acked 9673 Read 4 New Done Done false
23694 8675 Acked 9674 Write 2 New Done Done false
23695 8676 Acked 9675 Read 1 New Done Done false
23696 8677 Acked 9676 Read 9 New Done Done false
23697 8678 Acked 9677 Read 4 New Done Done false
23698 8679 Acked 9678 Write 9 New Done Done false
23699 8680 Acked 9679 Read 10 New Done Done false
23700 8681 Acked 9680 Read 8 New Done Done false
23701 8682 Acked 9681 Flush 0 New Done Done false
23702 8683 Acked 9682 Read 1 New Done Done false
23703 8684 Acked 9683 Read 6 New Done Done false
23704 8685 Acked 9684 Read 3 New Done Done false
23705 8686 Acked 9685 Read 9 New Done Done false
23706 8687 Acked 9686 Read 6 New Done Done false
23707 8688 Acked 9687 Read 2 New Done Done false
23708 8689 Acked 9688 Read 9 New Done Done false
23709 8690 Acked 9689 Read 5 New Done Done false
23710 8691 Acked 9690 Flush 0 New Done Done false
23711 8692 Acked 9691 Read 10 New Done Done false
23712 8693 Acked 9692 Read 9 New Done Done false
23713 8694 Acked 9693 Write 6 New Done Done false
23714 8695 Acked 9694 Flush 0 New Done Done false
23715 8696 Acked 9695 Read 6 New Done Done false
23716 8697 Acked 9696 Write 1 New Done Done false
23717 8698 Acked 9697 Write 5 New Done Done false
23718 8699 Acked 9698 Read 1 New Done Done false
23719 8700 Acked 9699 Read 1 New Done Done false
23720 8701 Acked 9700 Read 5 New Done Done false
23721 8702 Acked 9701 Write 8 New Done Done false
23722 8703 Acked 9702 Read 10 New Done Done false
23723 8704 Acked 9703 Read 8 New Done Done false
23724 8705 Acked 9704 Flush 0 New Done Done false
23725 8706 Acked 9705 Read 1 New Done Done false
23726 8707 Acked 9706 Read 3 New Done Done false
23727 8708 Acked 9707 Read 8 New Done Done false
23728 8709 Acked 9708 Write 4 New Done Done false
23729 8710 Acked 9709 Read 8 New Done Done false
23730 8711 Acked 9710 Read 10 New Done Done false
23731 8712 Acked 9711 Read 5 New Done Done false
23732 8713 Acked 9712 Read 10 New Done Done false
23733 8714 Acked 9713 Write 1 New Done Done false
23734 8715 Acked 9714 Read 5 New Done Done false
23735 8716 Acked 9715 Write 7 New Done Done false
23736 8717 Acked 9716 Read 9 New Done Done false
23737 8718 Acked 9717 Read 3 New Done Done false
23738 8719 Acked 9718 Write 9 New Done Done false
23739 8720 Acked 9719 Read 5 New Done Done false
23740 8721 Acked 9720 Read 7 New Done Done false
23741 8722 Acked 9721 Read 8 New Done Done false
23742 8723 Acked 9722 Read 5 New Done Done false
23743 8724 Acked 9723 Write 2 New Done Done false
23744 8725 Acked 9724 Flush 0 New Done Done false
23745 8726 Acked 9725 Write 7 New Done Done false
23746 8727 Acked 9726 Write 8 New Done Done false
23747 8728 Acked 9727 Write 4 New Done Done false
23748 8729 Acked 9728 Write 9 New Done Done false
23749 8730 Acked 9729 Write 6 New Done Done false
23750 8731 Acked 9730 Read 9 New Done Done false
23751 8732 Acked 9731 Read 1 New Done Done false
23752 8733 Acked 9732 Read 8 New Done Done false
23753 8734 Acked 9733 Write 8 New Done Done false
23754 8735 Acked 9734 Write 8 New Done Done false
23755 8736 Acked 9735 Write 7 New Done Done false
23756 8737 Acked 9736 Flush 0 New Done Done false
23757 8738 Acked 9737 Flush 0 New Done Done false
23758 8739 Acked 9738 Read 7 New Done Done false
23759 8740 Acked 9739 Read 7 New Done Done false
23760 8741 Acked 9740 Read 2 New Done Done false
23761 8742 Acked 9741 Write 1 New Done Done false
23762 8743 Acked 9742 Write 10 New Done Done false
23763 8744 Acked 9743 Read 1 New Done Done false
23764 8745 Acked 9744 Read 2 New Done Done false
23765 8746 Acked 9745 Read 10 New Done Done false
23766 8747 Acked 9746 Write 5 New Done Done false
23767 8748 Acked 9747 Read 6 New Done Done false
23768 8749 Acked 9748 Write 8 New Done Done false
23769 8750 Acked 9749 Read 2 New Done Done false
23770 8751 Acked 9750 Flush 0 New Done Done false
23771 8752 Acked 9751 Write 6 New Done Done false
23772 8753 Acked 9752 Read 2 New Done Done false
23773 8754 Acked 9753 Write 2 New Done Done false
23774 8755 Acked 9754 Flush 0 New Done Done false
23775 8756 Acked 9755 Read 3 New Done Done false
23776 8757 Acked 9756 Flush 0 New Done Done false
23777 8758 Acked 9757 Write 1 New Done Done false
23778 8759 Acked 9758 Flush 0 New Done Done false
23779 8760 Acked 9759 Write 10 New Done Done false
23780 8761 Acked 9760 Read 10 New Done Done false
23781 8762 Acked 9761 Read 8 New Done Done false
23782 8763 Acked 9762 Write 9 New Done Done false
23783 8764 Acked 9763 Flush 0 New Done Done false
23784 8765 Acked 9764 Write 6 New Done Done false
23785 8766 Acked 9765 Write 10 New Done Done false
23786 8767 Acked 9766 Read 3 New Done Done false
23787 8768 Acked 9767 Flush 0 New Done Done false
23788 8769 Acked 9768 Read 4 New Done Done false
23789 8770 Acked 9769 Write 2 New Done Done false
23790 8771 Acked 9770 Write 7 New Done Done false
23791 8772 Acked 9771 Write 6 New Done Done false
23792 8773 Acked 9772 Read 4 New Done Done false
23793 8774 Acked 9773 Read 6 New Done Done false
23794 8775 Acked 9774 Write 6 New Done Done false
23795 8776 Acked 9775 Read 5 New Done Done false
23796 8777 Acked 9776 Read 1 New Done Done false
23797 STATES DS:0 DS:1 DS:2 TOTAL
23798 New 301 0 0 301
23799 Sent 0 0 0 0
23800 Done 0 301 301 602
23801 Skipped 0 0 0 0
23802 Error 0 0 0 0
23803 Last Flush: 9475 9767 9767
23804 Downstairs last five completed: 9475 9474 9473 9472 9471 9470
23805 Upstairs last five completed: 8777 8776 8775 8774 8773 8772
23806 CLIENT: Up:0 ds:301 act:2
23807 ----------------------------------------------------------------
23808 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
23809 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
23810 8477 Acked 9476 Flush 0 New Done Done false
23811 8478 Acked 9477 Write 1 New Done Done false
23812 8479 Acked 9478 Flush 0 New Done Done false
23813 8480 Acked 9479 Write 10 New Done Done false
23814 8481 Acked 9480 Read 2 New Done Done false
23815 8482 Acked 9481 Read 6 New Done Done false
23816 8483 Acked 9482 Read 4 New Done Done false
23817 8484 Acked 9483 Write 9 New Done Done false
23818 8485 Acked 9484 Read 5 New Done Done false
23819 8486 Acked 9485 Write 10 New Done Done false
23820 8487 Acked 9486 Read 6 New Done Done false
23821 8488 Acked 9487 Write 8 New Done Done false
23822 8489 Acked 9488 Write 7 New Done Done false
23823 8490 Acked 9489 Write 4 New Done Done false
23824 8491 Acked 9490 Read 1 New Done Done false
23825 8492 Acked 9491 Read 6 New Done Done false
23826 8493 Acked 9492 Read 8 New Done Done false
23827 8494 Acked 9493 Write 8 New Done Done false
23828 8495 Acked 9494 Flush 0 New Done Done false
23829 8496 Acked 9495 Read 9 New Done Done false
23830 8497 Acked 9496 Read 6 New Done Done false
23831 8498 Acked 9497 Write 8 New Done Done false
23832 8499 Acked 9498 Write 2 New Done Done false
23833 8500 Acked 9499 Write 5 New Done Done false
23834 8501 Acked 9500 Write 6 New Done Done false
23835 8502 Acked 9501 Read 1 New Done Done false
23836 8503 Acked 9502 Read 10 New Done Done false
23837 8504 Acked 9503 Write 3 New Done Done false
23838 8505 Acked 9504 Flush 0 New Done Done false
23839 8506 Acked 9505 Read 9 New Done Done false
23840 8507 Acked 9506 Read 5 New Done Done false
23841 8508 Acked 9507 Read 10 New Done Done false
23842 8509 Acked 9508 Write 6 New Done Done false
23843 8510 Acked 9509 Write 5 New Done Done false
23844 8511 Acked 9510 Write 4 New Done Done false
23845 8512 Acked 9511 Read 2 New Done Done false
23846 8513 Acked 9512 Read 3 New Done Done false
23847 8514 Acked 9513 Read 8 New Done Done false
23848 8515 Acked 9514 Read 9 New Done Done false
23849 8516 Acked 9515 Write 7 New Done Done false
23850 8517 Acked 9516 Read 1 New Done Done false
23851 8518 Acked 9517 Read 9 New Done Done false
23852 8519 Acked 9518 Write 9 New Done Done false
23853 8520 Acked 9519 Write 4 New Done Done false
23854 8521 Acked 9520 Read 5 New Done Done false
23855 8522 Acked 9521 Write 5 New Done Done false
23856 8523 Acked 9522 Read 3 New Done Done false
23857 8524 Acked 9523 Read 4 New Done Done false
23858 8525 Acked 9524 Flush 0 New Done Done false
23859 8526 Acked 9525 Flush 0 New Done Done false
23860 8527 Acked 9526 Read 1 New Done Done false
23861 8528 Acked 9527 Flush 0 New Done Done false
23862 8529 Acked 9528 Read 5 New Done Done false
23863 8530 Acked 9529 Write 4 New Done Done false
23864 8531 Acked 9530 Read 9 New Done Done false
23865 8532 Acked 9531 Write 7 New Done Done false
23866 8533 Acked 9532 Write 10 New Done Done false
23867 8534 Acked 9533 Flush 0 New Done Done false
23868 8535 Acked 9534 Read 10 New Done Done false
23869 8536 Acked 9535 Read 6 New Done Done false
23870 8537 Acked 9536 Read 10 New Done Done false
23871 8538 Acked 9537 Read 7 New Done Done false
23872 8539 Acked 9538 Write 6 New Done Done false
23873 8540 Acked 9539 Write 2 New Done Done false
23874 8541 Acked 9540 Read 4 New Done Done false
23875 8542 Acked 9541 Read 7 New Done Done false
23876 8543 Acked 9542 Read 8 New Done Done false
23877 8544 Acked 9543 Read 4 New Done Done false
23878 8545 Acked 9544 Read 2 New Done Done false
23879 8546 Acked 9545 Read 1 New Done Done false
23880 8547 Acked 9546 Read 2 New Done Done false
23881 8548 Acked 9547 Write 5 New Done Done false
23882 8549 Acked 9548 Flush 0 New Done Done false
23883 8550 Acked 9549 Write 4 New Done Done false
23884 8551 Acked 9550 Read 10 New Done Done false
23885 8552 Acked 9551 Flush 0 New Done Done false
23886 8553 Acked 9552 Read 3 New Done Done false
23887 8554 Acked 9553 Read 8 New Done Done false
23888 8555 Acked 9554 Read 7 New Done Done false
23889 8556 Acked 9555 Write 1 New Done Done false
23890 8557 Acked 9556 Write 9 New Done Done false
23891 8558 Acked 9557 Read 2 New Done Done false
23892 8559 Acked 9558 Read 6 New Done Done false
23893 8560 Acked 9559 Flush 0 New Done Done false
23894 8561 Acked 9560 Read 5 New Done Done false
23895 8562 Acked 9561 Read 9 New Done Done false
23896 8563 Acked 9562 Read 10 New Done Done false
23897 8564 Acked 9563 Read 7 New Done Done false
23898 8565 Acked 9564 Read 10 New Done Done false
23899 8566 Acked 9565 Read 4 New Done Done false
23900 8567 Acked 9566 Write 8 New Done Done false
23901 8568 Acked 9567 Write 10 New Done Done false
23902 8569 Acked 9568 Write 9 New Done Done false
23903 8570 Acked 9569 Read 9 New Done Done false
23904 8571 Acked 9570 Write 8 New Done Done false
23905 8572 Acked 9571 Write 8 New Done Done false
23906 8573 Acked 9572 Write 2 New Done Done false
23907 8574 Acked 9573 Read 9 New Done Done false
23908 8575 Acked 9574 Write 6 New Done Done false
23909 8576 Acked 9575 Write 4 New Done Done false
23910 8577 Acked 9576 Read 8 New Done Done false
23911 8578 Acked 9577 Flush 0 New Done Done false
23912 8579 Acked 9578 Write 5 New Done Done false
23913 8580 Acked 9579 Read 2 New Done Done false
23914 8581 Acked 9580 Read 4 New Done Done false
23915 8582 Acked 9581 Read 6 New Done Done false
23916 8583 Acked 9582 Write 3 New Done Done false
23917 8584 Acked 9583 Read 5 New Done Done false
23918 8585 Acked 9584 Read 2 New Done Done false
23919 8586 Acked 9585 Write 3 New Done Done false
23920 8587 Acked 9586 Read 3 New Done Done false
23921 8588 Acked 9587 Write 2 New Done Done false
23922 8589 Acked 9588 Read 1 New Done Done false
23923 8590 Acked 9589 Read 2 New Done Done false
23924 8591 Acked 9590 Flush 0 New Done Done false
23925 8592 Acked 9591 Read 8 New Done Done false
23926 8593 Acked 9592 Read 8 New Done Done false
23927 8594 Acked 9593 Read 1 New Done Done false
23928 8595 Acked 9594 Read 4 New Done Done false
23929 8596 Acked 9595 Read 4 New Done Done false
23930 8597 Acked 9596 Read 3 New Done Done false
23931 8598 Acked 9597 Write 10 New Done Done false
23932 8599 Acked 9598 Write 2 New Done Done false
23933 8600 Acked 9599 Write 8 New Done Done false
23934 8601 Acked 9600 Write 4 New Done Done false
23935 8602 Acked 9601 Write 8 New Done Done false
23936 8603 Acked 9602 Flush 0 New Done Done false
23937 8604 Acked 9603 Flush 0 New Done Done false
23938 8605 Acked 9604 Write 9 New Done Done false
23939 8606 Acked 9605 Write 5 New Done Done false
23940 8607 Acked 9606 Read 6 New Done Done false
23941 8608 Acked 9607 Read 4 New Done Done false
23942 8609 Acked 9608 Read 5 New Done Done false
23943 8610 Acked 9609 Read 9 New Done Done false
23944 8611 Acked 9610 Write 10 New Done Done false
23945 8612 Acked 9611 Read 1 New Done Done false
23946 8613 Acked 9612 Read 7 New Done Done false
23947 8614 Acked 9613 Read 1 New Done Done false
23948 8615 Acked 9614 Write 7 New Done Done false
23949 8616 Acked 9615 Read 2 New Done Done false
23950 8617 Acked 9616 Read 2 New Done Done false
23951 8618 Acked 9617 Read 9 New Done Done false
23952 8619 Acked 9618 Flush 0 New Done Done false
23953 8620 Acked 9619 Read 5 New Done Done false
23954 8621 Acked 9620 Flush 0 New Done Done false
23955 8622 Acked 9621 Read 7 New Done Done false
23956 8623 Acked 9622 Flush 0 New Done Done false
23957 8624 Acked 9623 Flush 0 New Done Done false
23958 8625 Acked 9624 Write 8 New Done Done false
23959 8626 Acked 9625 Read 8 New Done Done false
23960 8627 Acked 9626 Write 6 New Done Done false
23961 8628 Acked 9627 Write 9 New Done Done false
23962 8629 Acked 9628 Read 4 New Done Done false
23963 8630 Acked 9629 Read 9 New Done Done false
23964 8631 Acked 9630 Read 10 New Done Done false
23965 8632 Acked 9631 Read 9 New Done Done false
23966 8633 Acked 9632 Read 9 New Done Done false
23967 8634 Acked 9633 Write 8 New Done Done false
23968 8635 Acked 9634 Write 7 New Done Done false
23969 8636 Acked 9635 Read 8 New Done Done false
23970 8637 Acked 9636 Write 7 New Done Done false
23971 8638 Acked 9637 Write 3 New Done Done false
23972 8639 Acked 9638 Read 9 New Done Done false
23973 8640 Acked 9639 Write 4 New Done Done false
23974 8641 Acked 9640 Read 7 New Done Done false
23975 8642 Acked 9641 Read 7 New Done Done false
23976 8643 Acked 9642 Read 9 New Done Done false
23977 8644 Acked 9643 Write 6 New Done Done false
23978 8645 Acked 9644 Read 10 New Done Done false
23979 8646 Acked 9645 Flush 0 New Done Done false
23980 8647 Acked 9646 Read 5 New Done Done false
23981 8648 Acked 9647 Write 1 New Done Done false
23982 8649 Acked 9648 Write 2 New Done Done false
23983 8650 Acked 9649 Write 3 New Done Done false
23984 8651 Acked 9650 Write 8 New Done Done false
23985 8652 Acked 9651 Read 10 New Done Done false
23986 8653 Acked 9652 Read 5 New Done Done false
23987 8654 Acked 9653 Write 9 New Done Done false
23988 8655 Acked 9654 Read 7 New Done Done false
23989 8656 Acked 9655 Read 1 New Done Done false
23990 8657 Acked 9656 Write 3 New Done Done false
23991 8658 Acked 9657 Write 5 New Done Done false
23992 8659 Acked 9658 Read 7 New Done Done false
23993 8660 Acked 9659 Write 8 New Done Done false
23994 8661 Acked 9660 Read 3 New Done Done false
23995 8662 Acked 9661 Read 7 New Done Done false
23996 8663 Acked 9662 Write 7 New Done Done false
23997 8664 Acked 9663 Read 7 New Done Done false
23998 8665 Acked 9664 Read 4 New Done Done false
23999 8666 Acked 9665 Read 7 New Done Done false
24000 8667 Acked 9666 Write 4 New Done Done false
24001 8668 Acked 9667 Flush 0 New Done Done false
24002 8669 Acked 9668 Write 3 New Done Done false
24003 8670 Acked 9669 Read 1 New Done Done false
24004 8671 Acked 9670 Read 6 New Done Done false
24005 8672 Acked 9671 Read 8 New Done Done false
24006 8673 Acked 9672 Flush 0 New Done Done false
24007 8674 Acked 9673 Read 4 New Done Done false
24008 8675 Acked 9674 Write 2 New Done Done false
24009 8676 Acked 9675 Read 1 New Done Done false
24010 8677 Acked 9676 Read 9 New Done Done false
24011 8678 Acked 9677 Read 4 New Done Done false
24012 8679 Acked 9678 Write 9 New Done Done false
24013 8680 Acked 9679 Read 10 New Done Done false
24014 8681 Acked 9680 Read 8 New Done Done false
24015 8682 Acked 9681 Flush 0 New Done Done false
24016 8683 Acked 9682 Read 1 New Done Done false
24017 8684 Acked 9683 Read 6 New Done Done false
24018 8685 Acked 9684 Read 3 New Done Done false
24019 8686 Acked 9685 Read 9 New Done Done false
24020 8687 Acked 9686 Read 6 New Done Done false
24021 8688 Acked 9687 Read 2 New Done Done false
24022 8689 Acked 9688 Read 9 New Done Done false
24023 8690 Acked 9689 Read 5 New Done Done false
24024 8691 Acked 9690 Flush 0 New Done Done false
24025 8692 Acked 9691 Read 10 New Done Done false
24026 8693 Acked 9692 Read 9 New Done Done false
24027 8694 Acked 9693 Write 6 New Done Done false
24028 8695 Acked 9694 Flush 0 New Done Done false
24029 8696 Acked 9695 Read 6 New Done Done false
24030 8697 Acked 9696 Write 1 New Done Done false
24031 8698 Acked 9697 Write 5 New Done Done false
24032 8699 Acked 9698 Read 1 New Done Done false
24033 8700 Acked 9699 Read 1 New Done Done false
24034 8701 Acked 9700 Read 5 New Done Done false
24035 8702 Acked 9701 Write 8 New Done Done false
24036 8703 Acked 9702 Read 10 New Done Done false
24037 8704 Acked 9703 Read 8 New Done Done false
24038 8705 Acked 9704 Flush 0 New Done Done false
24039 8706 Acked 9705 Read 1 New Done Done false
24040 8707 Acked 9706 Read 3 New Done Done false
24041 8708 Acked 9707 Read 8 New Done Done false
24042 8709 Acked 9708 Write 4 New Done Done false
24043 8710 Acked 9709 Read 8 New Done Done false
24044 8711 Acked 9710 Read 10 New Done Done false
24045 8712 Acked 9711 Read 5 New Done Done false
24046 8713 Acked 9712 Read 10 New Done Done false
24047 8714 Acked 9713 Write 1 New Done Done false
24048 8715 Acked 9714 Read 5 New Done Done false
24049 8716 Acked 9715 Write 7 New Done Done false
24050 8717 Acked 9716 Read 9 New Done Done false
24051 8718 Acked 9717 Read 3 New Done Done false
24052 8719 Acked 9718 Write 9 New Done Done false
24053 8720 Acked 9719 Read 5 New Done Done false
24054 8721 Acked 9720 Read 7 New Done Done false
24055 8722 Acked 9721 Read 8 New Done Done false
24056 8723 Acked 9722 Read 5 New Done Done false
24057 8724 Acked 9723 Write 2 New Done Done false
24058 8725 Acked 9724 Flush 0 New Done Done false
24059 8726 Acked 9725 Write 7 New Done Done false
24060 8727 Acked 9726 Write 8 New Done Done false
24061 8728 Acked 9727 Write 4 New Done Done false
24062 8729 Acked 9728 Write 9 New Done Done false
24063 8730 Acked 9729 Write 6 New Done Done false
24064 8731 Acked 9730 Read 9 New Done Done false
24065 8732 Acked 9731 Read 1 New Done Done false
24066 8733 Acked 9732 Read 8 New Done Done false
24067 8734 Acked 9733 Write 8 New Done Done false
24068 8735 Acked 9734 Write 8 New Done Done false
24069 8736 Acked 9735 Write 7 New Done Done false
24070 8737 Acked 9736 Flush 0 New Done Done false
24071 8738 Acked 9737 Flush 0 New Done Done false
24072 8739 Acked 9738 Read 7 New Done Done false
24073 8740 Acked 9739 Read 7 New Done Done false
24074 8741 Acked 9740 Read 2 New Done Done false
24075 8742 Acked 9741 Write 1 New Done Done false
24076 8743 Acked 9742 Write 10 New Done Done false
24077 8744 Acked 9743 Read 1 New Done Done false
24078 8745 Acked 9744 Read 2 New Done Done false
24079 8746 Acked 9745 Read 10 New Done Done false
24080 8747 Acked 9746 Write 5 New Done Done false
24081 8748 Acked 9747 Read 6 New Done Done false
24082 8749 Acked 9748 Write 8 New Done Done false
24083 8750 Acked 9749 Read 2 New Done Done false
24084 8751 Acked 9750 Flush 0 New Done Done false
24085 8752 Acked 9751 Write 6 New Done Done false
24086 8753 Acked 9752 Read 2 New Done Done false
24087 8754 Acked 9753 Write 2 New Done Done false
24088 8755 Acked 9754 Flush 0 New Done Done false
24089 8756 Acked 9755 Read 3 New Done Done false
24090 8757 Acked 9756 Flush 0 New Done Done false
24091 8758 Acked 9757 Write 1 New Done Done false
24092 8759 Acked 9758 Flush 0 New Done Done false
24093 8760 Acked 9759 Write 10 New Done Done false
24094 8761 Acked 9760 Read 10 New Done Done false
24095 8762 Acked 9761 Read 8 New Done Done false
24096 8763 Acked 9762 Write 9 New Done Done false
24097 8764 Acked 9763 Flush 0 New Done Done false
24098 8765 Acked 9764 Write 6 New Done Done false
24099 8766 Acked 9765 Write 10 New Done Done false
24100 8767 Acked 9766 Read 3 New Done Done false
24101 8768 Acked 9767 Flush 0 New Done Done false
24102 8769 Acked 9768 Read 4 New Done Done false
24103 8770 Acked 9769 Write 2 New Done Done false
24104 8771 Acked 9770 Write 7 New Done Done false
24105 8772 Acked 9771 Write 6 New Done Done false
24106 8773 Acked 9772 Read 4 New Done Done false
24107 8774 Acked 9773 Read 6 New Done Done false
24108 8775 Acked 9774 Write 6 New Done Done false
24109 8776 Acked 9775 Read 5 New Done Done false
24110 8777 Acked 9776 Read 1 New Done Done false
24111 8778 Acked 9777 Flush 0 New Done Done false
24112 STATES DS:0 DS:1 DS:2 TOTAL
24113 New 302 0 0 302
24114 Sent 0 0 0 0
24115 Done 0 302 302 604
24116 Skipped 0 0 0 0
24117 Error 0 0 0 0
24118 Last Flush: 9475 9777 9777
24119 Downstairs last five completed: 9475 9474 9473 9472 9471 9470
24120 Upstairs last five completed: 8778 8777 8776 8775 8774 8773
24121 CLIENT: Up:0 ds:302 act:2
24122 ----------------------------------------------------------------
24123 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24124 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24125 8477 Acked 9476 Flush 0 New Done Done false
24126 8478 Acked 9477 Write 1 New Done Done false
24127 8479 Acked 9478 Flush 0 New Done Done false
24128 8480 Acked 9479 Write 10 New Done Done false
24129 8481 Acked 9480 Read 2 New Done Done false
24130 8482 Acked 9481 Read 6 New Done Done false
24131 8483 Acked 9482 Read 4 New Done Done false
24132 8484 Acked 9483 Write 9 New Done Done false
24133 8485 Acked 9484 Read 5 New Done Done false
24134 8486 Acked 9485 Write 10 New Done Done false
24135 8487 Acked 9486 Read 6 New Done Done false
24136 8488 Acked 9487 Write 8 New Done Done false
24137 8489 Acked 9488 Write 7 New Done Done false
24138 8490 Acked 9489 Write 4 New Done Done false
24139 8491 Acked 9490 Read 1 New Done Done false
24140 8492 Acked 9491 Read 6 New Done Done false
24141 8493 Acked 9492 Read 8 New Done Done false
24142 8494 Acked 9493 Write 8 New Done Done false
24143 8495 Acked 9494 Flush 0 New Done Done false
24144 8496 Acked 9495 Read 9 New Done Done false
24145 8497 Acked 9496 Read 6 New Done Done false
24146 8498 Acked 9497 Write 8 New Done Done false
24147 8499 Acked 9498 Write 2 New Done Done false
24148 8500 Acked 9499 Write 5 New Done Done false
24149 8501 Acked 9500 Write 6 New Done Done false
24150 8502 Acked 9501 Read 1 New Done Done false
24151 8503 Acked 9502 Read 10 New Done Done false
24152 8504 Acked 9503 Write 3 New Done Done false
24153 8505 Acked 9504 Flush 0 New Done Done false
24154 8506 Acked 9505 Read 9 New Done Done false
24155 8507 Acked 9506 Read 5 New Done Done false
24156 8508 Acked 9507 Read 10 New Done Done false
24157 8509 Acked 9508 Write 6 New Done Done false
24158 8510 Acked 9509 Write 5 New Done Done false
24159 8511 Acked 9510 Write 4 New Done Done false
24160 8512 Acked 9511 Read 2 New Done Done false
24161 8513 Acked 9512 Read 3 New Done Done false
24162 8514 Acked 9513 Read 8 New Done Done false
24163 8515 Acked 9514 Read 9 New Done Done false
24164 8516 Acked 9515 Write 7 New Done Done false
24165 8517 Acked 9516 Read 1 New Done Done false
24166 8518 Acked 9517 Read 9 New Done Done false
24167 8519 Acked 9518 Write 9 New Done Done false
24168 8520 Acked 9519 Write 4 New Done Done false
24169 8521 Acked 9520 Read 5 New Done Done false
24170 8522 Acked 9521 Write 5 New Done Done false
24171 8523 Acked 9522 Read 3 New Done Done false
24172 8524 Acked 9523 Read 4 New Done Done false
24173 8525 Acked 9524 Flush 0 New Done Done false
24174 8526 Acked 9525 Flush 0 New Done Done false
24175 8527 Acked 9526 Read 1 New Done Done false
24176 8528 Acked 9527 Flush 0 New Done Done false
24177 8529 Acked 9528 Read 5 New Done Done false
24178 8530 Acked 9529 Write 4 New Done Done false
24179 8531 Acked 9530 Read 9 New Done Done false
24180 8532 Acked 9531 Write 7 New Done Done false
24181 8533 Acked 9532 Write 10 New Done Done false
24182 8534 Acked 9533 Flush 0 New Done Done false
24183 8535 Acked 9534 Read 10 New Done Done false
24184 8536 Acked 9535 Read 6 New Done Done false
24185 8537 Acked 9536 Read 10 New Done Done false
24186 8538 Acked 9537 Read 7 New Done Done false
24187 8539 Acked 9538 Write 6 New Done Done false
24188 8540 Acked 9539 Write 2 New Done Done false
24189 8541 Acked 9540 Read 4 New Done Done false
24190 8542 Acked 9541 Read 7 New Done Done false
24191 8543 Acked 9542 Read 8 New Done Done false
24192 8544 Acked 9543 Read 4 New Done Done false
24193 8545 Acked 9544 Read 2 New Done Done false
24194 8546 Acked 9545 Read 1 New Done Done false
24195 8547 Acked 9546 Read 2 New Done Done false
24196 8548 Acked 9547 Write 5 New Done Done false
24197 8549 Acked 9548 Flush 0 New Done Done false
24198 8550 Acked 9549 Write 4 New Done Done false
24199 8551 Acked 9550 Read 10 New Done Done false
24200 8552 Acked 9551 Flush 0 New Done Done false
24201 8553 Acked 9552 Read 3 New Done Done false
24202 8554 Acked 9553 Read 8 New Done Done false
24203 8555 Acked 9554 Read 7 New Done Done false
24204 8556 Acked 9555 Write 1 New Done Done false
24205 8557 Acked 9556 Write 9 New Done Done false
24206 8558 Acked 9557 Read 2 New Done Done false
24207 8559 Acked 9558 Read 6 New Done Done false
24208 8560 Acked 9559 Flush 0 New Done Done false
24209 8561 Acked 9560 Read 5 New Done Done false
24210 8562 Acked 9561 Read 9 New Done Done false
24211 8563 Acked 9562 Read 10 New Done Done false
24212 8564 Acked 9563 Read 7 New Done Done false
24213 8565 Acked 9564 Read 10 New Done Done false
24214 8566 Acked 9565 Read 4 New Done Done false
24215 8567 Acked 9566 Write 8 New Done Done false
24216 8568 Acked 9567 Write 10 New Done Done false
24217 8569 Acked 9568 Write 9 New Done Done false
24218 8570 Acked 9569 Read 9 New Done Done false
24219 8571 Acked 9570 Write 8 New Done Done false
24220 8572 Acked 9571 Write 8 New Done Done false
24221 8573 Acked 9572 Write 2 New Done Done false
24222 8574 Acked 9573 Read 9 New Done Done false
24223 8575 Acked 9574 Write 6 New Done Done false
24224 8576 Acked 9575 Write 4 New Done Done false
24225 8577 Acked 9576 Read 8 New Done Done false
24226 8578 Acked 9577 Flush 0 New Done Done false
24227 8579 Acked 9578 Write 5 New Done Done false
24228 8580 Acked 9579 Read 2 New Done Done false
24229 8581 Acked 9580 Read 4 New Done Done false
24230 8582 Acked 9581 Read 6 New Done Done false
24231 8583 Acked 9582 Write 3 New Done Done false
24232 8584 Acked 9583 Read 5 New Done Done false
24233 8585 Acked 9584 Read 2 New Done Done false
24234 8586 Acked 9585 Write 3 New Done Done false
24235 8587 Acked 9586 Read 3 New Done Done false
24236 8588 Acked 9587 Write 2 New Done Done false
24237 8589 Acked 9588 Read 1 New Done Done false
24238 8590 Acked 9589 Read 2 New Done Done false
24239 8591 Acked 9590 Flush 0 New Done Done false
24240 8592 Acked 9591 Read 8 New Done Done false
24241 8593 Acked 9592 Read 8 New Done Done false
24242 8594 Acked 9593 Read 1 New Done Done false
24243 8595 Acked 9594 Read 4 New Done Done false
24244 8596 Acked 9595 Read 4 New Done Done false
24245 8597 Acked 9596 Read 3 New Done Done false
24246 8598 Acked 9597 Write 10 New Done Done false
24247 8599 Acked 9598 Write 2 New Done Done false
24248 8600 Acked 9599 Write 8 New Done Done false
24249 8601 Acked 9600 Write 4 New Done Done false
24250 8602 Acked 9601 Write 8 New Done Done false
24251 8603 Acked 9602 Flush 0 New Done Done false
24252 8604 Acked 9603 Flush 0 New Done Done false
24253 8605 Acked 9604 Write 9 New Done Done false
24254 8606 Acked 9605 Write 5 New Done Done false
24255 8607 Acked 9606 Read 6 New Done Done false
24256 8608 Acked 9607 Read 4 New Done Done false
24257 8609 Acked 9608 Read 5 New Done Done false
24258 8610 Acked 9609 Read 9 New Done Done false
24259 8611 Acked 9610 Write 10 New Done Done false
24260 8612 Acked 9611 Read 1 New Done Done false
24261 8613 Acked 9612 Read 7 New Done Done false
24262 8614 Acked 9613 Read 1 New Done Done false
24263 8615 Acked 9614 Write 7 New Done Done false
24264 8616 Acked 9615 Read 2 New Done Done false
24265 8617 Acked 9616 Read 2 New Done Done false
24266 8618 Acked 9617 Read 9 New Done Done false
24267 8619 Acked 9618 Flush 0 New Done Done false
24268 8620 Acked 9619 Read 5 New Done Done false
24269 8621 Acked 9620 Flush 0 New Done Done false
24270 8622 Acked 9621 Read 7 New Done Done false
24271 8623 Acked 9622 Flush 0 New Done Done false
24272 8624 Acked 9623 Flush 0 New Done Done false
24273 8625 Acked 9624 Write 8 New Done Done false
24274 8626 Acked 9625 Read 8 New Done Done false
24275 8627 Acked 9626 Write 6 New Done Done false
24276 8628 Acked 9627 Write 9 New Done Done false
24277 8629 Acked 9628 Read 4 New Done Done false
24278 8630 Acked 9629 Read 9 New Done Done false
24279 8631 Acked 9630 Read 10 New Done Done false
24280 8632 Acked 9631 Read 9 New Done Done false
24281 8633 Acked 9632 Read 9 New Done Done false
24282 8634 Acked 9633 Write 8 New Done Done false
24283 8635 Acked 9634 Write 7 New Done Done false
24284 8636 Acked 9635 Read 8 New Done Done false
24285 8637 Acked 9636 Write 7 New Done Done false
24286 8638 Acked 9637 Write 3 New Done Done false
24287 8639 Acked 9638 Read 9 New Done Done false
24288 8640 Acked 9639 Write 4 New Done Done false
24289 8641 Acked 9640 Read 7 New Done Done false
24290 8642 Acked 9641 Read 7 New Done Done false
24291 8643 Acked 9642 Read 9 New Done Done false
24292 8644 Acked 9643 Write 6 New Done Done false
24293 8645 Acked 9644 Read 10 New Done Done false
24294 8646 Acked 9645 Flush 0 New Done Done false
24295 8647 Acked 9646 Read 5 New Done Done false
24296 8648 Acked 9647 Write 1 New Done Done false
24297 8649 Acked 9648 Write 2 New Done Done false
24298 8650 Acked 9649 Write 3 New Done Done false
24299 8651 Acked 9650 Write 8 New Done Done false
24300 8652 Acked 9651 Read 10 New Done Done false
24301 8653 Acked 9652 Read 5 New Done Done false
24302 8654 Acked 9653 Write 9 New Done Done false
24303 8655 Acked 9654 Read 7 New Done Done false
24304 8656 Acked 9655 Read 1 New Done Done false
24305 8657 Acked 9656 Write 3 New Done Done false
24306 8658 Acked 9657 Write 5 New Done Done false
24307 8659 Acked 9658 Read 7 New Done Done false
24308 8660 Acked 9659 Write 8 New Done Done false
24309 8661 Acked 9660 Read 3 New Done Done false
24310 8662 Acked 9661 Read 7 New Done Done false
24311 8663 Acked 9662 Write 7 New Done Done false
24312 8664 Acked 9663 Read 7 New Done Done false
24313 8665 Acked 9664 Read 4 New Done Done false
24314 8666 Acked 9665 Read 7 New Done Done false
24315 8667 Acked 9666 Write 4 New Done Done false
24316 8668 Acked 9667 Flush 0 New Done Done false
24317 8669 Acked 9668 Write 3 New Done Done false
24318 8670 Acked 9669 Read 1 New Done Done false
24319 8671 Acked 9670 Read 6 New Done Done false
24320 8672 Acked 9671 Read 8 New Done Done false
24321 8673 Acked 9672 Flush 0 New Done Done false
24322 8674 Acked 9673 Read 4 New Done Done false
24323 8675 Acked 9674 Write 2 New Done Done false
24324 8676 Acked 9675 Read 1 New Done Done false
24325 8677 Acked 9676 Read 9 New Done Done false
24326 8678 Acked 9677 Read 4 New Done Done false
24327 8679 Acked 9678 Write 9 New Done Done false
24328 8680 Acked 9679 Read 10 New Done Done false
24329 8681 Acked 9680 Read 8 New Done Done false
24330 8682 Acked 9681 Flush 0 New Done Done false
24331 8683 Acked 9682 Read 1 New Done Done false
24332 8684 Acked 9683 Read 6 New Done Done false
24333 8685 Acked 9684 Read 3 New Done Done false
24334 8686 Acked 9685 Read 9 New Done Done false
24335 8687 Acked 9686 Read 6 New Done Done false
24336 8688 Acked 9687 Read 2 New Done Done false
24337 8689 Acked 9688 Read 9 New Done Done false
24338 8690 Acked 9689 Read 5 New Done Done false
24339 8691 Acked 9690 Flush 0 New Done Done false
24340 8692 Acked 9691 Read 10 New Done Done false
24341 8693 Acked 9692 Read 9 New Done Done false
24342 8694 Acked 9693 Write 6 New Done Done false
24343 8695 Acked 9694 Flush 0 New Done Done false
24344 8696 Acked 9695 Read 6 New Done Done false
24345 8697 Acked 9696 Write 1 New Done Done false
24346 8698 Acked 9697 Write 5 New Done Done false
24347 8699 Acked 9698 Read 1 New Done Done false
24348 8700 Acked 9699 Read 1 New Done Done false
24349 8701 Acked 9700 Read 5 New Done Done false
24350 8702 Acked 9701 Write 8 New Done Done false
24351 8703 Acked 9702 Read 10 New Done Done false
24352 8704 Acked 9703 Read 8 New Done Done false
24353 8705 Acked 9704 Flush 0 New Done Done false
24354 8706 Acked 9705 Read 1 New Done Done false
24355 8707 Acked 9706 Read 3 New Done Done false
24356 8708 Acked 9707 Read 8 New Done Done false
24357 8709 Acked 9708 Write 4 New Done Done false
24358 8710 Acked 9709 Read 8 New Done Done false
24359 8711 Acked 9710 Read 10 New Done Done false
24360 8712 Acked 9711 Read 5 New Done Done false
24361 8713 Acked 9712 Read 10 New Done Done false
24362 8714 Acked 9713 Write 1 New Done Done false
24363 8715 Acked 9714 Read 5 New Done Done false
24364 8716 Acked 9715 Write 7 New Done Done false
24365 8717 Acked 9716 Read 9 New Done Done false
24366 8718 Acked 9717 Read 3 New Done Done false
24367 8719 Acked 9718 Write 9 New Done Done false
24368 8720 Acked 9719 Read 5 New Done Done false
24369 8721 Acked 9720 Read 7 New Done Done false
24370 8722 Acked 9721 Read 8 New Done Done false
24371 8723 Acked 9722 Read 5 New Done Done false
24372 8724 Acked 9723 Write 2 New Done Done false
24373 8725 Acked 9724 Flush 0 New Done Done false
24374 8726 Acked 9725 Write 7 New Done Done false
24375 8727 Acked 9726 Write 8 New Done Done false
24376 8728 Acked 9727 Write 4 New Done Done false
24377 8729 Acked 9728 Write 9 New Done Done false
24378 8730 Acked 9729 Write 6 New Done Done false
24379 8731 Acked 9730 Read 9 New Done Done false
24380 8732 Acked 9731 Read 1 New Done Done false
24381 8733 Acked 9732 Read 8 New Done Done false
24382 8734 Acked 9733 Write 8 New Done Done false
24383 8735 Acked 9734 Write 8 New Done Done false
24384 8736 Acked 9735 Write 7 New Done Done false
24385 8737 Acked 9736 Flush 0 New Done Done false
24386 8738 Acked 9737 Flush 0 New Done Done false
24387 8739 Acked 9738 Read 7 New Done Done false
24388 8740 Acked 9739 Read 7 New Done Done false
24389 8741 Acked 9740 Read 2 New Done Done false
24390 8742 Acked 9741 Write 1 New Done Done false
24391 8743 Acked 9742 Write 10 New Done Done false
24392 8744 Acked 9743 Read 1 New Done Done false
24393 8745 Acked 9744 Read 2 New Done Done false
24394 8746 Acked 9745 Read 10 New Done Done false
24395 8747 Acked 9746 Write 5 New Done Done false
24396 8748 Acked 9747 Read 6 New Done Done false
24397 8749 Acked 9748 Write 8 New Done Done false
24398 8750 Acked 9749 Read 2 New Done Done false
24399 8751 Acked 9750 Flush 0 New Done Done false
24400 8752 Acked 9751 Write 6 New Done Done false
24401 8753 Acked 9752 Read 2 New Done Done false
24402 8754 Acked 9753 Write 2 New Done Done false
24403 8755 Acked 9754 Flush 0 New Done Done false
24404 8756 Acked 9755 Read 3 New Done Done false
24405 8757 Acked 9756 Flush 0 New Done Done false
24406 8758 Acked 9757 Write 1 New Done Done false
24407 8759 Acked 9758 Flush 0 New Done Done false
24408 8760 Acked 9759 Write 10 New Done Done false
24409 8761 Acked 9760 Read 10 New Done Done false
24410 8762 Acked 9761 Read 8 New Done Done false
24411 8763 Acked 9762 Write 9 New Done Done false
24412 8764 Acked 9763 Flush 0 New Done Done false
24413 8765 Acked 9764 Write 6 New Done Done false
24414 8766 Acked 9765 Write 10 New Done Done false
24415 8767 Acked 9766 Read 3 New Done Done false
24416 8768 Acked 9767 Flush 0 New Done Done false
24417 8769 Acked 9768 Read 4 New Done Done false
24418 8770 Acked 9769 Write 2 New Done Done false
24419 8771 Acked 9770 Write 7 New Done Done false
24420 8772 Acked 9771 Write 6 New Done Done false
24421 8773 Acked 9772 Read 4 New Done Done false
24422 8774 Acked 9773 Read 6 New Done Done false
24423 8775 Acked 9774 Write 6 New Done Done false
24424 8776 Acked 9775 Read 5 New Done Done false
24425 8777 Acked 9776 Read 1 New Done Done false
24426 8778 Acked 9777 Flush 0 New Done Done false
24427 STATES DS:0 DS:1 DS:2 TOTAL
24428 New 302 0 0 302
24429 Sent 0 0 0 0
24430 Done 0 302 302 604
24431 Skipped 0 0 0 0
24432 Error 0 0 0 0
24433 Last Flush: 9475 9777 9777
24434 Downstairs last five completed: 9475 9474 9473 9472 9471 9470
24435 Upstairs last five completed: 8778 8777 8776 8775 8774 8773
24436 CLIENT: Up:0 ds:302 act:2
244372023-10-13T18:14:23.491ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
24438 [8810] got start action
24439 [8810] I am going to restart
24440 Starting downstairs at port 8810
24441 Make output file at "/tmp/dsc/downstairs-8810.txt"
24442 [8810][0] reports Starting
24443 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1205)
24444 [8810][0] reports Running
244452023-10-13T18:14:25.494ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 0
244462023-10-13T18:14:25.494ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state Offline
244472023-10-13T18:14:25.494ZINFOcrucible: [0] client re-new 302 jobs since flush 9475 = downstairs
244482023-10-13T18:14:25.495ZINFOcrucible: [0] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
244492023-10-13T18:14:25.495ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
244502023-10-13T18:14:25.495ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
244512023-10-13T18:14:25.495ZINFOcrucible: Returning client:0 UUID:12345678-0000-0000-0000-000000008810 matches
244522023-10-13T18:14:25.495ZINFOcrucible: [0] send last flush ID to this DS: 9475
244532023-10-13T18:14:25.495ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Offline Active Active
244542023-10-13T18:14:25.496ZINFOcrucible: [0] Replied this last flush ID: 9475
244552023-10-13T18:14:25.496ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Offline Active Active ds_transition to Replay
244562023-10-13T18:14:25.496ZINFOcrucible: [0] Transition from Offline to Replay
244572023-10-13T18:14:25.496ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
244582023-10-13T18:14:25.496ZINFOcrucible: [0] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Replay Active Active ds_transition to Active
244592023-10-13T18:14:25.496ZINFOcrucible: [0] Transition from Replay to Active
244602023-10-13T18:14:25.496ZINFOcrucible: [0] Starts cmd_loop
244612023-10-13T18:14:25.496ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
244622023-10-13T18:14:25.496ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
244632023-10-13T18:14:25.496ZINFOcrucible: Set check for repair
244642023-10-13T18:14:26.497ZINFOcrucible: Checking if live repair is needed
244652023-10-13T18:14:26.497ZINFOcrucible: No Live Repair required at this time
244662023-10-13T18:14:26.497ZWARNcrucible: [0] flow control end
24467 ----------------------------------------------------------------
24468 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24469 Downstairs last five completed: 9777 9776 9775 9774 9773 9772
24470 Upstairs last five completed: 8778 8777 8776 8775 8774 8773
24471 CLIENT: Up:0 ds:0 act:3
24472 Replay: All jobs finished, all DS active.
24473 Main task has work to do, go find it
24474 got dsc Stop(2)
24475 stop 2
24476 Oct 13 18:14:27.452 INFO request completed, [8830] Got stop action so:false kr:false
24477 latency_us: 521, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 7ee60baf-87f7-4aea-8f64-08ff0115ac9c, remote_addr: 127.0.0.1:64571, local_addr: 127.0.0.1:9998
24478 Oct 13 18:14:27.453 INFO request completed, latency_us: 238, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3cb2e359-8bfd-44a3-bb39-43068c9352c8, remote_addr: 127.0.0.1:36157, local_addr: 127.0.0.1:9998
244792023-10-13T18:14:27.453ZWARNcrucible: [2] downstairs disconnected
244802023-10-13T18:14:27.453ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Gone missing, transition from Active to Offline
244812023-10-13T18:14:27.453ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c connection to 127.0.0.1:8830 closed looper = 2
244822023-10-13T18:14:27.453ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
244832023-10-13T18:14:27.453ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
244842023-10-13T18:14:27.453ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
244852023-10-13T18:14:27.453ZWARNcrucible: [2] pm_task rx.recv() is None
24486 [8830] Exited with: ExitStatus(unix_wait_status(9))
24487 [8830][2] reports Exit
24488 Oct 13 18:14:31.455 INFO request completed, latency_us: 313, response_code: 200, uri: /state/cid/2, method: GET, req_id: f7ac2f11-4f2c-4d17-a69e-efc3fd79d00a, remote_addr: 127.0.0.1:36157, local_addr: 127.0.0.1:9998
24489 001/300 Read block 2703 len 4096
24490 002/300 Read block 3027 len 20480
24491 003/300 Write block 3248 len 8192 data: 5 5
24492 004/300 Read block 4486 len 8192
24493 005/300 Write block 2929 len 24576 data: 5 6 8 7 6 4
24494 006/300 Read block 2691 len 4096
24495 007/300 Read block 103 len 24576
24496 008/300 Write block 1824 len 36864 data: 3 4 4 4 4 4 4 4 4
24497 009/300 Read block 2960 len 40960
24498 010/300 Read block 888 len 36864
24499 011/300 Write block 48 len 28672 data: 11 8 8 3 4 4 4
24500 012/300 Flush
24501 013/300 Read block 2928 len 12288
24502 014/300 Read block 1710 len 20480
24503 015/300 Write block 4452 len 12288 data: 8 9 8
24504 016/300 Read block 4729 len 28672
24505 017/300 Write block 992 len 40960 data: 6 7 6 6 5 6 6 6 4 6
24506 018/300 Write block 4391 len 40960 data: 7 5 5 6 8 8 8 12 10 10
24507 019/300 Read block 1206 len 40960
24508 020/300 Flush
24509 021/300 Read block 3269 len 36864
24510 022/300 Read block 4282 len 28672
24511 023/300 Write block 4795 len 24576 data: 5 5 6 6 6 9
24512 024/300 Write block 3896 len 24576 data: 4 5 3 3 4 5
24513 025/300 Write block 1927 len 16384 data: 7 6 6 4
24514 026/300 Read block 3406 len 20480
24515 027/300 Read block 2047 len 32768
24516 028/300 Read block 4864 len 20480
24517 029/300 Write block 2553 len 40960 data: 9 7 7 10 9 10 10 11 14 13
24518 030/300 Write block 65 len 32768 data: 4 6 6 6 6 7 7 7
24519 031/300 Flush
24520 032/300 Write block 3487 len 24576 data: 4 4 6 6 6 5
24521 033/300 Read block 444 len 28672
24522 034/300 Flush
24523 035/300 Write block 2312 len 8192 data: 5 5
24524 036/300 Read block 945 len 4096
24525 037/300 Read block 4549 len 36864
24526 038/300 Write block 78 len 8192 data: 6 5
24527 039/300 Write block 921 len 40960 data: 6 8 6 5 6 4 5 7 8 8
24528 040/300 Read block 4 len 8192
24529 041/300 Read block 4783 len 36864
24530 042/300 Read block 3668 len 32768
24531 043/300 Write block 2333 len 40960 data: 5 7 7 7 6 4 4 3 4 4
24532 044/300 Write block 566 len 4096 data: 4
24533 045/300 Read block 662 len 24576
24534 046/300 Read block 743 len 24576
24535 047/300 Read block 4329 len 16384
24536 048/300 Read block 1274 len 20480
24537 049/300 Read block 2039 len 16384
24538 050/300 Write block 4733 len 24576 data: 6 5 7 7 6 5
24539 051/300 Read block 1717 len 4096
24540 052/300 Read block 1543 len 16384
24541 053/300 Read block 421 len 20480
24542 054/300 Read block 801 len 36864
24543 055/300 Write block 3847 len 36864 data: 6 5 5 5 5 5 5 6 6
24544 056/300 Read block 645 len 36864
24545 057/300 Write block 1115 len 20480 data: 5 7 7 6 6
24546 058/300 Flush
24547 059/300 Read block 2949 len 8192
24548 060/300 Read block 3945 len 20480
24549 061/300 Write block 2814 len 40960 data: 5 5 3 3 4 4 5 5 3 4
24550 062/300 Write block 220 len 8192 data: 4 5
24551 063/300 Read block 3328 len 40960
24552 064/300 Write block 2068 len 40960 data: 4 4 4 3 3 4 6 6 7 5
24553 065/300 Write block 4496 len 32768 data: 4 4 6 6 6 7 6 7
24554 066/300 Read block 4864 len 28672
24555 067/300 Write block 2873 len 40960 data: 5 5 3 4 4 4 6 6 6 7
24556 068/300 Write block 676 len 12288 data: 5 6 6
24557 069/300 Flush
24558 070/300 Read block 793 len 4096
24559 071/300 Read block 1126 len 36864
24560 072/300 Read block 880 len 4096
24561 073/300 Read block 1007 len 36864
24562 074/300 Read block 1615 len 4096
24563 075/300 Write block 1240 len 4096 data: 6
24564 076/300 Flush
24565 077/300 Write block 3782 len 12288 data: 6 7 7
24566 078/300 Read block 2568 len 16384
24567 079/300 Read block 3521 len 28672
24568 080/300 Write block 938 len 20480 data: 8 7 6 4 3
24569 081/300 Write block 4129 len 32768 data: 3 4 5 5 5 5 5 6
24570 082/300 Write block 2518 len 8192 data: 4 3
24571 083/300 Read block 297 len 28672
24572 084/300 Write block 2879 len 24576 data: 7 7 7 8 8 7
24573 085/300 Read block 2594 len 12288
24574 086/300 Read block 1532 len 36864
24575 087/300 Read block 1813 len 40960
24576 088/300 Read block 2949 len 28672
24577 089/300 Write block 2451 len 16384 data: 7 6 6 6
24578 090/300 Write block 3620 len 8192 data: 11 6
24579 091/300 Read block 2165 len 24576
24580 092/300 Read block 4032 len 20480
24581 093/300 Read block 2802 len 8192
24582 094/300 Read block 3342 len 4096
24583 095/300 Read block 4071 len 36864
24584 096/300 Read block 4083 len 40960
24585 097/300 Read block 3714 len 20480
24586 098/300 Read block 3668 len 28672
24587 099/300 Read block 4902 len 36864
24588 100/300 Read block 1290 len 32768
24589 101/300 Read block 1824 len 32768
24590 102/300 Read block 1196 len 12288
24591 103/300 Read block 1943 len 28672
24592 104/300 Flush
24593 105/300 Write block 1705 len 32768 data: 5 5 5 7 7 7 7 6
24594 106/300 Read block 2260 len 16384
24595 107/300 Write block 2726 len 16384 data: 3 3 3 4
24596 108/300 Read block 1896 len 12288
24597 109/300 Read block 1865 len 12288
24598 110/300 Write block 2514 len 8192 data: 6 5
24599 111/300 Flush
24600 112/300 Read block 2545 len 4096
24601 113/300 Read block 1185 len 4096
24602 114/300 Read block 1347 len 36864
24603 115/300 Write block 1517 len 4096 data: 2
24604 116/300 Write block 1468 len 12288 data: 6 7 7
24605 117/300 Write block 2041 len 36864 data: 4 4 4 3 3 3 3 4 4
24606 118/300 Write block 4408 len 24576 data: 4 4 5 5 5 6
24607 119/300 Read block 2156 len 12288
24608 120/300 Write block 2633 len 4096 data: 3
24609 121/300 Read block 93 len 24576
24610 122/300 Write block 4415 len 24576 data: 2 3 4 5 5 5
24611 123/300 Read block 52 len 32768
24612 124/300 Write block 4121 len 20480 data: 6 6 5 4 3
24613 125/300 Write block 1239 len 32768 data: 9 7 4 5 5 6 5 4
24614 126/300 Read block 2206 len 16384
24615 127/300 Read block 4763 len 36864
24616 128/300 Write block 128 len 4096 data: 10
24617 129/300 Read block 901 len 40960
24618 130/300 Read block 2809 len 12288
24619 131/300 Write block 18 len 4096 data: 5
24620 132/300 Flush
24621 133/300 Read block 1437 len 20480
24622 134/300 Flush
24623 135/300 Read block 2066 len 4096
24624 136/300 Write block 2170 len 24576 data: 7 8 6 4 4 4
24625 137/300 Read block 921 len 8192
24626 138/300 Read block 1565 len 8192
24627 139/300 Read block 2952 len 36864
24628 140/300 Flush
24629 141/300 Read block 3135 len 36864
24630 142/300 Read block 1322 len 36864
24631 143/300 Read block 4704 len 8192
24632 144/300 Write block 4056 len 32768 data: 5 8 10 9 8 7 6 6
24633 145/300 Write block 4554 len 28672 data: 4 4 4 4 4 4 4
24634 146/300 Read block 3773 len 28672
24635 147/300 Write block 2726 len 32768 data: 4 4 4 5 4 5 6 10
24636 148/300 Read block 455 len 32768
24637 149/300 Write block 2538 len 16384 data: 6 5 4 4
24638 150/300 Write block 932 len 12288 data: 6 6 5
24639 151/300 Write block 2389 len 20480 data: 4 6 6 6 6
24640 152/300 Write block 372 len 16384 data: 5 6 8 8
24641 153/300 Write block 3154 len 24576 data: 3 3 3 3 7 8
24642 154/300 Write block 1099 len 32768 data: 10 11 8 8 6 7 7 6
24643 155/300 Read block 2456 len 28672
24644 156/300 Write block 74 len 28672 data: 9 5 5 5 7 6 5
24645 157/300 Read block 1024 len 4096
24646 158/300 Write block 2891 len 24576 data: 7 8 5 4 4 4
24647 159/300 Write block 609 len 36864 data: 5 6 5 6 6 6 7 8 7
24648 160/300 Read block 2019 len 40960
24649 161/300 Read block 4958 len 32768
24650 162/300 Write block 1380 len 40960 data: 8 7 6 6 5 4 6 8 6 7
24651 163/300 Read block 3016 len 16384
24652 164/300 Write block 1735 len 24576 data: 5 4 5 6 6 5
24653 165/300 Write block 1172 len 40960 data: 8 8 7 6 6 7 5 6 5 6
24654 166/300 Read block 3931 len 28672
24655 167/300 Read block 4468 len 4096
24656 168/300 Read block 2796 len 28672
24657 169/300 Write block 1087 len 12288 data: 5 5 5
24658 170/300 Read block 1871 len 8192
24659 171/300 Write block 2562 len 20480 data: 14 10 8 9 7
24660 172/300 Write block 1516 len 12288 data: 4 3 2
24661 173/300 Read block 4345 len 4096
24662 174/300 Read block 1568 len 12288
24663 175/300 Read block 2308 len 40960
24664 176/300 Read block 2163 len 40960
24665 177/300 Read block 2331 len 4096
24666 178/300 Read block 2170 len 16384
24667 179/300 Read block 387 len 4096
24668 180/300 Read block 3720 len 36864
24669 181/300 Read block 1863 len 28672
24670 182/300 Write block 2665 len 28672 data: 3 4 5 5 5 5 6
24671 183/300 Read block 4327 len 20480
24672 184/300 Write block 3098 len 16384 data: 8 7 7 6
24673 185/300 Write block 1112 len 36864 data: 7 7 6 6 8 8 7 7 5
24674 186/300 Read block 3439 len 12288
24675 187/300 Read block 3872 len 32768
24676 188/300 Flush
24677 189/300 Flush
24678 190/300 Flush
24679 191/300 Read block 2150 len 20480
24680 192/300 Write block 2460 len 20480 data: 3 6 6 10 11
24681 193/300 Flush
24682 194/300 Write block 3948 len 8192 data: 4 5
24683 195/300 Read block 3605 len 40960
24684 196/300 Read block 3159 len 24576
24685 197/300 Write block 772 len 12288 data: 4 5 5
24686 198/300 Read block 3801 len 32768
24687 199/300 Read block 3268 len 28672
24688 200/300 Flush
24689 201/300 Write block 2279 len 24576 data: 8 9 9 9 8 7
24690 202/300 Read block 645 len 32768
24691 203/300 Write block 3078 len 8192 data: 2 3
24692 204/300 Write block 801 len 16384 data: 5 5 5 5
24693 205/300 Write block 2073 len 36864 data: 5 7 7 8 6 5 4 3 4
24694 206/300 Read block 1789 len 24576
24695 207/300 Read block 4753 len 40960
24696 208/300 Flush
24697 209/300 Read block 3098 len 36864
24698 210/300 Write block 2554 len 12288 data: 8 8 11
24699 211/300 Write block 1997 len 32768 data: 6 5 5 5 4 4 4 6
24700 212/300 Read block 1857 len 8192
24701 213/300 Write block 3838 len 20480 data: 8 8 7 7 7
24702 214/300 Read block 2126 len 16384
24703 215/300 Write block 2153 len 40960 data: 5 5 5 6 5 6 6 6 7 6
24704 216/300 Write block 4889 len 40960 data: 5 8 7 10 9 8 8 5 5 4
24705 217/300 Write block 157 len 20480 data: 6 5 5 6 7
24706 218/300 Write block 151 len 8192 data: 8 8
24707 219/300 Write block 1702 len 4096 data: 5
24708 220/300 Flush
24709 221/300 Read block 931 len 20480
24710 222/300 Write block 3271 len 8192 data: 5 5
24711 223/300 Write block 4015 len 4096 data: 9
24712 224/300 Read block 2396 len 24576
24713 225/300 Read block 150 len 28672
24714 226/300 Read block 4925 len 20480
24715 227/300 Read block 561 len 24576
24716 228/300 Read block 270 len 4096
24717 229/300 Write block 4549 len 28672 data: 3 4 4 4 4 5 5
24718 230/300 Read block 3111 len 12288
24719 231/300 Read block 2580 len 36864
24720 232/300 Read block 2727 len 36864
24721 233/300 Read block 3016 len 4096
24722 234/300 Read block 606 len 12288
24723 235/300 Read block 3657 len 16384
24724 236/300 Read block 4944 len 36864
24725 237/300 Read block 1843 len 8192
24726 238/300 Read block 3633 len 4096
24727 239/300 Write block 4552 len 12288 data: 5 5 6
24728 240/300 Write block 1788 len 4096 data: 7
24729 241/300 Read block 3074 len 24576
24730 242/300 Read block 4942 len 32768
24731 243/300 Read block 3415 len 16384
24732 244/300 Write block 3031 len 8192 data: 3 4
24733 245/300 Read block 1462 len 32768
24734 246/300 Write block 630 len 28672 data: 7 5 4 3 3 3 3
24735 247/300 Write block 4867 len 4096 data: 5
24736 248/300 Read block 2647 len 8192
24737 249/300 Read block 4243 len 20480
24738 250/300 Write block 645 len 12288 data: 8 7 4
24739 251/300 Write block 860 len 40960 data: 3 3 2 2 2 2 3 4 4 4
24740 252/300 Write block 1497 len 8192 data: 4 5
24741 253/300 Write block 4530 len 40960 data: 6 6 6 7 8 8 7 6 6 5
24742 254/300 Read block 4758 len 4096
24743 255/300 Read block 2218 len 16384
24744 256/300 Write block 1527 len 36864 data: 7 7 6 4 5 5 5 6 7
24745 257/300 Write block 543 len 24576 data: 4 5 5 5 5 4
24746 258/300 Write block 4591 len 8192 data: 5 7
24747 259/300 Read block 2804 len 4096
24748 260/300 Write block 1352 len 36864 data: 6 7 8 6 5 5 5 4 4
24749 261/300 Write block 4872 len 24576 data: 5 4 4 4 5 5
24750 262/300 Read block 3123 len 20480
24751 263/300 Write block 1656 len 36864 data: 7 6 6 6 7 8 9 10 10
24752 264/300 Read block 1506 len 8192
24753 265/300 Read block 3223 len 28672
24754 266/300 Read block 3632 len 12288
24755 267/300 Read block 1796 len 36864
24756 268/300 Read block 1243 len 24576
24757 269/300 Read block 381 len 8192
24758 270/300 Write block 4855 len 24576 data: 5 5 7 8 11 12
24759 271/300 Write block 4254 len 4096 data: 8
24760 272/300 Write block 2620 len 16384 data: 5 4 5 6
24761 273/300 Write block 2311 len 20480 data: 6 6 6 7 7
24762 274/300 Read block 88 len 28672
24763 275/300 Write block 931 len 24576 data: 7 7 7 6 7 8
24764 276/300 Write block 1476 len 40960 data: 7 9 9 9 8 8 8 8 8 8
24765 277/300 Read block 2770 len 24576
24766 278/300 Read block 612 len 40960
24767 279/300 Read block 3220 len 4096
24768 280/300 Read block 4073 len 12288
24769 281/300 Flush
24770 282/300 Read block 546 len 12288
24771 283/300 Write block 2644 len 36864 data: 8 6 5 5 3 4 3 2 2
24772 284/300 Read block 3322 len 4096
24773 285/300 Write block 3476 len 36864 data: 11 10 8 6 6 6 5 4 3
24774 286/300 Read block 3793 len 16384
24775 287/300 Read block 4765 len 24576
24776 288/300 Flush
24777 289/300 Write block 1328 len 16384 data: 6 5 6 7
24778 290/300 Read block 880 len 36864
24779 291/300 Flush
24780 292/300 Write block 502 len 8192 data: 3 3
24781 293/300 Read block 1908 len 24576
24782 294/300 Read block 4560 len 36864
24783 295/300 Write block 342 len 24576 data: 5 6 7 7 8 8
24784 296/300 Read block 4882 len 28672
24785 297/300 Write block 3352 len 20480 data: 8 7 6 5 4
24786 298/300 Read block 3992 len 20480
24787 299/300 Write block 2027 len 28672 data: 7 8 8 8 7 8 6
24788 300/300 Read block 2491 len 20480
24789 Main task has work to do, go find it
24790 got dsc Start(2)
24791 start 2
24792 Oct 13 18:14:32.435 INFO request completed, latency_us: 300, response_code: 204, uri: /start/cid/2, method: POST, req_id: b6ee13c5-1491-41dd-979d-c74cc8aef626, remote_addr: 127.0.0.1:36157, local_addr: 127.0.0.1:9998
24793 Replay: started 2, returned:Ok(())
24794 ----------------------------------------------------------------
24795 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24796 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24797 8779 Acked 9778 Read 1 Done Done New false
24798 8780 Acked 9779 Read 5 Done Done New false
24799 8781 Acked 9780 Write 2 Done Done New false
24800 8782 Acked 9781 Read 2 Done Done New false
24801 8783 Acked 9782 Write 6 Done Done New false
24802 8784 Acked 9783 Read 1 Done Done New false
24803 8785 Acked 9784 Read 6 Done Done New false
24804 8786 Acked 9785 Write 9 Done Done New false
24805 8787 Acked 9786 Read 10 Done Done New false
24806 8788 Acked 9787 Read 9 Done Done New false
24807 8789 Acked 9788 Write 7 Done Done New false
24808 8790 Acked 9789 Flush 0 Done Done New false
24809 8791 Acked 9790 Read 3 Done Done New false
24810 8792 Acked 9791 Read 5 Done Done New false
24811 8793 Acked 9792 Write 3 Done Done New false
24812 8794 Acked 9793 Read 7 Done Done New false
24813 8795 Acked 9794 Write 10 Done Done New false
24814 8796 Acked 9795 Write 10 Done Done New false
24815 8797 Acked 9796 Read 10 Done Done New false
24816 8798 Acked 9797 Flush 0 Done Done New false
24817 8799 Acked 9798 Read 9 Done Done New false
24818 8800 Acked 9799 Read 7 Done Done New false
24819 8801 Acked 9800 Write 6 Done Done New false
24820 8802 Acked 9801 Write 6 Done Done New false
24821 8803 Acked 9802 Write 4 Done Done New false
24822 8804 Acked 9803 Read 5 Done Done New false
24823 8805 Acked 9804 Read 8 Done Done New false
24824 8806 Acked 9805 Read 5 Done Done New false
24825 8807 Acked 9806 Write 10 Done Done New false
24826 8808 Acked 9807 Write 8 Done Done New false
24827 8809 Acked 9808 Flush 0 Done Done New false
24828 8810 Acked 9809 Write 6 Done Done New false
24829 8811 Acked 9810 Read 7 Done Done New false
24830 8812 Acked 9811 Flush 0 Done Done New false
24831 8813 Acked 9812 Write 2 Done Done New false
24832 8814 Acked 9813 Read 1 Done Done New false
24833 8815 Acked 9814 Read 9 Done Done New false
24834 8816 Acked 9815 Write 2 Done Done New false
24835 8817 Acked 9816 Write 10 Done Done New false
24836 8818 Acked 9817 Read 2 Done Done New false
24837 8819 Acked 9818 Read 9 Done Done New false
24838 8820 Acked 9819 Read 8 Done Done New false
24839 8821 Acked 9820 Write 10 Done Done New false
24840 8822 Acked 9821 Write 1 Done Done New false
24841 8823 Acked 9822 Read 6 Done Done New false
24842 8824 Acked 9823 Read 6 Done Done New false
24843 8825 Acked 9824 Read 4 Done Done New false
24844 8826 Acked 9825 Read 5 Done Done New false
24845 8827 Acked 9826 Read 4 Done Done New false
24846 8828 Acked 9827 Write 6 Done Done New false
24847 8829 Acked 9828 Read 1 Done Done New false
24848 8830 Acked 9829 Read 4 Done Done New false
24849 8831 Acked 9830 Read 5 Done Done New false
24850 8832 Acked 9831 Read 9 Done Done New false
24851 8833 Acked 9832 Write 9 Done Done New false
24852 8834 Acked 9833 Read 9 Done Done New false
24853 8835 Acked 9834 Write 5 Done Done New false
24854 8836 Acked 9835 Flush 0 Done Done New false
24855 8837 Acked 9836 Read 2 Done Done New false
24856 8838 Acked 9837 Read 5 Done Done New false
24857 8839 Acked 9838 Write 10 Done Done New false
24858 8840 Acked 9839 Write 2 Done Done New false
24859 8841 Acked 9840 Read 10 Done Done New false
24860 8842 Acked 9841 Write 10 Done Done New false
24861 8843 Acked 9842 Write 8 Done Done New false
24862 8844 Acked 9843 Read 7 Done Done New false
24863 8845 Acked 9844 Write 10 Done Done New false
24864 8846 Acked 9845 Write 3 Done Done New false
24865 8847 Acked 9846 Flush 0 Done Done New false
24866 8848 Acked 9847 Read 1 Done Done New false
24867 8849 Acked 9848 Read 9 Done Done New false
24868 8850 Acked 9849 Read 1 Done Done New false
24869 8851 Acked 9850 Read 9 Done Done New false
24870 8852 Acked 9851 Read 1 Done Done New false
24871 8853 Acked 9852 Write 1 Done Done New false
24872 8854 Acked 9853 Flush 0 Done Done New false
24873 8855 Acked 9854 Write 3 Done Done New false
24874 8856 Acked 9855 Read 4 Done Done New false
24875 8857 Acked 9856 Read 7 Done Done New false
24876 8858 Acked 9857 Write 5 Done Done New false
24877 8859 Acked 9858 Write 8 Done Done New false
24878 8860 Acked 9859 Write 2 Done Done New false
24879 8861 Acked 9860 Read 7 Done Done New false
24880 8862 Acked 9861 Write 6 Done Done New false
24881 8863 Acked 9862 Read 3 Done Done New false
24882 8864 Acked 9863 Read 9 Done Done New false
24883 8865 Acked 9864 Read 10 Done Done New false
24884 8866 Acked 9865 Flush 0 Done Done New false
24885 8867 Acked 9866 Read 7 Done Done New false
24886 8868 Acked 9867 Write 4 Done Done New false
24887 8869 Acked 9868 Write 2 Done Done New false
24888 8870 Acked 9869 Read 6 Done Done New false
24889 8871 Acked 9870 Read 5 Done Done New false
24890 8872 Acked 9871 Read 2 Done Done New false
24891 8873 Acked 9872 Read 1 Done Done New false
24892 8874 Acked 9873 Read 9 Done Done New false
24893 8875 Acked 9874 Read 10 Done Done New false
24894 8876 Acked 9875 Read 5 Done Done New false
24895 8877 Acked 9876 Read 7 Done Done New false
24896 8878 Acked 9877 Read 9 Done Done New false
24897 8879 Acked 9878 Read 8 Done Done New false
24898 8880 Acked 9879 Read 8 Done Done New false
24899 8881 Acked 9880 Read 3 Done Done New false
24900 8882 Acked 9881 Read 7 Done Done New false
24901 8883 Acked 9882 Flush 0 Done Done New false
24902 8884 Acked 9883 Write 8 Done Done New false
24903 8885 Acked 9884 Read 4 Done Done New false
24904 8886 Acked 9885 Write 4 Done Done New false
24905 8887 Acked 9886 Read 3 Done Done New false
24906 8888 Acked 9887 Read 3 Done Done New false
24907 8889 Acked 9888 Write 2 Done Done New false
24908 8890 Acked 9889 Flush 0 Done Done New false
24909 8891 Acked 9890 Read 1 Done Done New false
24910 8892 Acked 9891 Read 1 Done Done New false
24911 8893 Acked 9892 Read 9 Done Done New false
24912 8894 Acked 9893 Write 1 Done Done New false
24913 8895 Acked 9894 Write 3 Done Done New false
24914 8896 Acked 9895 Write 9 Done Done New false
24915 8897 Acked 9896 Write 6 Done Done New false
24916 8898 Acked 9897 Read 3 Done Done New false
24917 8899 Acked 9898 Write 1 Done Done New false
24918 8900 Acked 9899 Read 6 Done Done New false
24919 8901 Acked 9900 Write 6 Done Done New false
24920 8902 Acked 9901 Read 8 Done Done New false
24921 8903 Acked 9902 Write 5 Done Done New false
24922 8904 Acked 9903 Write 8 Done Done New false
24923 8905 Acked 9904 Read 4 Done Done New false
24924 8906 Acked 9905 Read 9 Done Done New false
24925 8907 Acked 9906 Write 1 Done Done New false
24926 8908 Acked 9907 Read 10 Done Done New false
24927 8909 Acked 9908 Read 3 Done Done New false
24928 8910 Acked 9909 Write 1 Done Done New false
24929 8911 Acked 9910 Flush 0 Done Done New false
24930 8912 Acked 9911 Read 5 Done Done New false
24931 8913 Acked 9912 Flush 0 Done Done New false
24932 8914 Acked 9913 Read 1 Done Done New false
24933 8915 Acked 9914 Write 6 Done Done New false
24934 8916 Acked 9915 Read 2 Done Done New false
24935 8917 Acked 9916 Read 2 Done Done New false
24936 8918 Acked 9917 Read 9 Done Done New false
24937 8919 Acked 9918 Flush 0 Done Done New false
24938 8920 Acked 9919 Read 9 Done Done New false
24939 8921 Acked 9920 Read 9 Done Done New false
24940 8922 Acked 9921 Read 2 Done Done New false
24941 8923 Acked 9922 Write 8 Done Done New false
24942 8924 Acked 9923 Write 7 Done Done New false
24943 8925 Acked 9924 Read 7 Done Done New false
24944 8926 Acked 9925 Write 8 Done Done New false
24945 8927 Acked 9926 Read 8 Done Done New false
24946 8928 Acked 9927 Write 4 Done Done New false
24947 8929 Acked 9928 Write 3 Done Done New false
24948 8930 Acked 9929 Write 5 Done Done New false
24949 8931 Acked 9930 Write 4 Done Done New false
24950 8932 Acked 9931 Write 6 Done Done New false
24951 8933 Acked 9932 Write 8 Done Done New false
24952 8934 Acked 9933 Read 7 Done Done New false
24953 8935 Acked 9934 Write 7 Done Done New false
24954 8936 Acked 9935 Read 1 Done Done New false
24955 8937 Acked 9936 Write 6 Done Done New false
24956 8938 Acked 9937 Write 9 Done Done New false
24957 8939 Acked 9938 Read 10 Done Done New false
24958 8940 Acked 9939 Read 8 Done Done New false
24959 8941 Acked 9940 Write 10 Done Done New false
24960 8942 Acked 9941 Read 4 Done Done New false
24961 8943 Acked 9942 Write 6 Done Done New false
24962 8944 Acked 9943 Write 10 Done Done New false
24963 8945 Acked 9944 Read 7 Done Done New false
24964 8946 Acked 9945 Read 1 Done Done New false
24965 8947 Acked 9946 Read 7 Done Done New false
24966 8948 Acked 9947 Write 3 Done Done New false
24967 8949 Acked 9948 Read 2 Done Done New false
24968 8950 Acked 9949 Write 5 Done Done New false
24969 8951 Acked 9950 Write 3 Done Done New false
24970 8952 Acked 9951 Read 1 Done Done New false
24971 8953 Acked 9952 Read 3 Done Done New false
24972 8954 Acked 9953 Read 10 Done Done New false
24973 8955 Acked 9954 Read 10 Done Done New false
24974 8956 Acked 9955 Read 1 Done Done New false
24975 8957 Acked 9956 Read 4 Done Done New false
24976 8958 Acked 9957 Read 1 Done Done New false
24977 8959 Acked 9958 Read 9 Done Done New false
24978 8960 Acked 9959 Read 7 Done Done New false
24979 8961 Acked 9960 Write 7 Done Done New false
24980 8962 Acked 9961 Read 5 Done Done New false
24981 8963 Acked 9962 Write 4 Done Done New false
24982 8964 Acked 9963 Write 9 Done Done New false
24983 8965 Acked 9964 Read 3 Done Done New false
24984 8966 Acked 9965 Read 8 Done Done New false
24985 8967 Acked 9966 Flush 0 Done Done New false
24986 8968 Acked 9967 Flush 0 Done Done New false
24987 8969 Acked 9968 Flush 0 Done Done New false
24988 8970 Acked 9969 Read 5 Done Done New false
24989 8971 Acked 9970 Write 5 Done Done New false
24990 8972 Acked 9971 Flush 0 Done Done New false
24991 8973 Acked 9972 Write 2 Done Done New false
24992 8974 Acked 9973 Read 10 Done Done New false
24993 8975 Acked 9974 Read 6 Done Done New false
24994 8976 Acked 9975 Write 3 Done Done New false
24995 8977 Acked 9976 Read 8 Done Done New false
24996 8978 Acked 9977 Read 7 Done Done New false
24997 8979 Acked 9978 Flush 0 Done Done New false
24998 8980 Acked 9979 Write 6 Done Done New false
24999 8981 Acked 9980 Read 8 Done Done New false
25000 8982 Acked 9981 Write 2 Done Done New false
25001 8983 Acked 9982 Write 4 Done Done New false
25002 8984 Acked 9983 Write 9 Done Done New false
25003 8985 Acked 9984 Read 6 Done Done New false
25004 8986 Acked 9985 Read 10 Done Done New false
25005 8987 Acked 9986 Flush 0 Done Done New false
25006 8988 Acked 9987 Read 9 Done Done New false
25007 8989 Acked 9988 Write 3 Done Done New false
25008 8990 Acked 9989 Write 8 Done Done New false
25009 8991 Acked 9990 Read 2 Done Done New false
25010 8992 Acked 9991 Write 5 Done Done New false
25011 8993 Acked 9992 Read 4 Done Done New false
25012 8994 Acked 9993 Write 10 Done Done New false
25013 8995 Acked 9994 Write 10 Done Done New false
25014 8996 Acked 9995 Write 5 Done Done New false
25015 8997 Acked 9996 Write 2 Done Done New false
25016 8998 Acked 9997 Write 1 Done Done New false
25017 8999 Acked 9998 Flush 0 Done Done New false
25018 9000 Acked 9999 Read 5 Done Done New false
25019 9001 Acked 10000 Write 2 Done Done New false
25020 9002 Acked 10001 Write 1 Done Done New false
25021 9003 Acked 10002 Read 6 Done Done New false
25022 9004 Acked 10003 Read 7 Done Done New false
25023 9005 Acked 10004 Read 5 Done Done New false
25024 9006 Acked 10005 Read 6 Done Done New false
25025 9007 Acked 10006 Read 1 Done Done New false
25026 9008 Acked 10007 Write 7 Done Done New false
25027 9009 Acked 10008 Read 3 Done Done New false
25028 9010 Acked 10009 Read 9 Done Done New false
25029 9011 Acked 10010 Read 9 Done Done New false
25030 9012 Acked 10011 Read 1 Done Done New false
25031 9013 Acked 10012 Read 3 Done Done New false
25032 9014 Acked 10013 Read 4 Done Done New false
25033 9015 Acked 10014 Read 9 Done Done New false
25034 9016 Acked 10015 Read 2 Done Done New false
25035 9017 Acked 10016 Read 1 Done Done New false
25036 9018 Acked 10017 Write 3 Done Done New false
25037 9019 Acked 10018 Write 1 Done Done New false
25038 9020 Acked 10019 Read 6 Done Done New false
25039 9021 Acked 10020 Read 8 Done Done New false
25040 9022 Acked 10021 Flush 0 Done Done New false
25041 9023 Acked 10022 Read 4 Done Done New false
25042 9024 Acked 10023 Write 2 Done Done New false
25043 9025 Acked 10024 Read 8 Done Done New false
25044 9026 Acked 10025 Write 7 Done Done New false
25045 9027 Acked 10026 Write 1 Done Done New false
25046 9028 Acked 10027 Read 2 Done Done New false
25047 9029 Acked 10028 Read 5 Done Done New false
25048 9030 Acked 10029 Write 3 Done Done New false
25049 9031 Acked 10030 Write 10 Done Done New false
25050 9032 Acked 10031 Write 2 Done Done New false
25051 9033 Acked 10032 Write 10 Done Done New false
25052 9034 Acked 10033 Read 1 Done Done New false
25053 9035 Acked 10034 Read 4 Done Done New false
25054 9036 Acked 10035 Write 9 Done Done New false
25055 9037 Acked 10036 Write 6 Done Done New false
25056 9038 Acked 10037 Write 2 Done Done New false
25057 9039 Acked 10038 Read 1 Done Done New false
25058 9040 Acked 10039 Write 9 Done Done New false
25059 9041 Acked 10040 Write 6 Done Done New false
25060 9042 Acked 10041 Read 5 Done Done New false
25061 9043 Acked 10042 Write 9 Done Done New false
25062 9044 Acked 10043 Read 2 Done Done New false
25063 9045 Acked 10044 Read 7 Done Done New false
25064 9046 Acked 10045 Read 3 Done Done New false
25065 9047 Acked 10046 Read 9 Done Done New false
25066 9048 Acked 10047 Read 6 Done Done New false
25067 9049 Acked 10048 Read 2 Done Done New false
25068 9050 Acked 10049 Write 6 Done Done New false
25069 9051 Acked 10050 Write 1 Done Done New false
25070 9052 Acked 10051 Write 4 Done Done New false
25071 9053 Acked 10052 Write 5 Done Done New false
25072 9054 Acked 10053 Read 7 Done Done New false
25073 9055 Acked 10054 Write 6 Done Done New false
25074 9056 Acked 10055 Write 10 Done Done New false
25075 9057 Acked 10056 Read 6 Done Done New false
25076 9058 Acked 10057 Read 10 Done Done New false
25077 9059 Acked 10058 Read 1 Done Done New false
25078 9060 Acked 10059 Read 3 Done Done New false
25079 9061 Acked 10060 Flush 0 Done Done New false
25080 9062 Acked 10061 Read 3 Done Done New false
25081 9063 Acked 10062 Write 9 Done Done New false
25082 9064 Acked 10063 Read 1 Done Done New false
25083 9065 Acked 10064 Write 9 Done Done New false
25084 9066 Acked 10065 Read 4 Done Done New false
25085 9067 Acked 10066 Read 6 Done Done New false
25086 9068 Acked 10067 Flush 0 Done Done New false
25087 9069 Acked 10068 Write 4 Done Done New false
25088 9070 Acked 10069 Read 9 Done Done New false
25089 9071 Acked 10070 Flush 0 Done Done New false
25090 9072 Acked 10071 Write 2 Done Done New false
25091 9073 Acked 10072 Read 6 Done Done New false
25092 9074 Acked 10073 Read 9 Done Done New false
25093 9075 Acked 10074 Write 6 Done Done New false
25094 9076 Acked 10075 Read 7 Done Done New false
25095 9077 Acked 10076 Write 5 Done Done New false
25096 9078 Acked 10077 Read 5 Done Done New false
25097 9079 Acked 10078 Write 7 Done Done New false
25098 9080 Acked 10079 Read 5 Done Done New false
25099 STATES DS:0 DS:1 DS:2 TOTAL
25100 New 0 0 302 302
25101 Sent 0 0 0 0
25102 Done 302 302 0 604
25103 Skipped 0 0 0 0
25104 Error 0 0 0 0
25105 Last Flush: 10070 10070 9777
25106 Downstairs last five completed: 9777 9776 9775 9774 9773 9772
25107 Upstairs last five completed: 9080 9079 9078 9077 9076 9075
25108 CLIENT: Up:0 ds:302 act:2
251092023-10-13T18:14:32.459ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
25110 ----------------------------------------------------------------
25111 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
25112 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
25113 8779 Acked 9778 Read 1 Done Done New false
25114 8780 Acked 9779 Read 5 Done Done New false
25115 8781 Acked 9780 Write 2 Done Done New false
25116 8782 Acked 9781 Read 2 Done Done New false
25117 8783 Acked 9782 Write 6 Done Done New false
25118 8784 Acked 9783 Read 1 Done Done New false
25119 8785 Acked 9784 Read 6 Done Done New false
25120 8786 Acked 9785 Write 9 Done Done New false
25121 8787 Acked 9786 Read 10 Done Done New false
25122 8788 Acked 9787 Read 9 Done Done New false
25123 8789 Acked 9788 Write 7 Done Done New false
25124 8790 Acked 9789 Flush 0 Done Done New false
25125 8791 Acked 9790 Read 3 Done Done New false
25126 8792 Acked 9791 Read 5 Done Done New false
25127 8793 Acked 9792 Write 3 Done Done New false
25128 8794 Acked 9793 Read 7 Done Done New false
25129 8795 Acked 9794 Write 10 Done Done New false
25130 8796 Acked 9795 Write 10 Done Done New false
25131 8797 Acked 9796 Read 10 Done Done New false
25132 8798 Acked 9797 Flush 0 Done Done New false
25133 8799 Acked 9798 Read 9 Done Done New false
25134 8800 Acked 9799 Read 7 Done Done New false
25135 8801 Acked 9800 Write 6 Done Done New false
25136 8802 Acked 9801 Write 6 Done Done New false
25137 8803 Acked 9802 Write 4 Done Done New false
25138 8804 Acked 9803 Read 5 Done Done New false
25139 8805 Acked 9804 Read 8 Done Done New false
25140 8806 Acked 9805 Read 5 Done Done New false
25141 8807 Acked 9806 Write 10 Done Done New false
25142 8808 Acked 9807 Write 8 Done Done New false
25143 8809 Acked 9808 Flush 0 Done Done New false
25144 8810 Acked 9809 Write 6 Done Done New false
25145 8811 Acked 9810 Read 7 Done Done New false
25146 8812 Acked 9811 Flush 0 Done Done New false
25147 8813 Acked 9812 Write 2 Done Done New false
25148 8814 Acked 9813 Read 1 Done Done New false
25149 8815 Acked 9814 Read 9 Done Done New false
25150 8816 Acked 9815 Write 2 Done Done New false
25151 8817 Acked 9816 Write 10 Done Done New false
25152 8818 Acked 9817 Read 2 Done Done New false
25153 8819 Acked 9818 Read 9 Done Done New false
25154 8820 Acked 9819 Read 8 Done Done New false
25155 8821 Acked 9820 Write 10 Done Done New false
25156 8822 Acked 9821 Write 1 Done Done New false
25157 8823 Acked 9822 Read 6 Done Done New false
25158 8824 Acked 9823 Read 6 Done Done New false
25159 8825 Acked 9824 Read 4 Done Done New false
25160 8826 Acked 9825 Read 5 Done Done New false
25161 8827 Acked 9826 Read 4 Done Done New false
25162 8828 Acked 9827 Write 6 Done Done New false
25163 8829 Acked 9828 Read 1 Done Done New false
25164 8830 Acked 9829 Read 4 Done Done New false
25165 8831 Acked 9830 Read 5 Done Done New false
25166 8832 Acked 9831 Read 9 Done Done New false
25167 8833 Acked 9832 Write 9 Done Done New false
25168 8834 Acked 9833 Read 9 Done Done New false
25169 8835 Acked 9834 Write 5 Done Done New false
25170 8836 Acked 9835 Flush 0 Done Done New false
25171 8837 Acked 9836 Read 2 Done Done New false
25172 8838 Acked 9837 Read 5 Done Done New false
25173 8839 Acked 9838 Write 10 Done Done New false
25174 8840 Acked 9839 Write 2 Done Done New false
25175 8841 Acked 9840 Read 10 Done Done New false
25176 8842 Acked 9841 Write 10 Done Done New false
25177 8843 Acked 9842 Write 8 Done Done New false
25178 8844 Acked 9843 Read 7 Done Done New false
25179 8845 Acked 9844 Write 10 Done Done New false
25180 8846 Acked 9845 Write 3 Done Done New false
25181 8847 Acked 9846 Flush 0 Done Done New false
25182 8848 Acked 9847 Read 1 Done Done New false
25183 8849 Acked 9848 Read 9 Done Done New false
25184 8850 Acked 9849 Read 1 Done Done New false
25185 8851 Acked 9850 Read 9 Done Done New false
25186 8852 Acked 9851 Read 1 Done Done New false
25187 8853 Acked 9852 Write 1 Done Done New false
25188 8854 Acked 9853 Flush 0 Done Done New false
25189 8855 Acked 9854 Write 3 Done Done New false
25190 8856 Acked 9855 Read 4 Done Done New false
25191 8857 Acked 9856 Read 7 Done Done New false
25192 8858 Acked 9857 Write 5 Done Done New false
25193 8859 Acked 9858 Write 8 Done Done New false
25194 8860 Acked 9859 Write 2 Done Done New false
25195 8861 Acked 9860 Read 7 Done Done New false
25196 8862 Acked 9861 Write 6 Done Done New false
25197 8863 Acked 9862 Read 3 Done Done New false
25198 8864 Acked 9863 Read 9 Done Done New false
25199 8865 Acked 9864 Read 10 Done Done New false
25200 8866 Acked 9865 Flush 0 Done Done New false
25201 8867 Acked 9866 Read 7 Done Done New false
25202 8868 Acked 9867 Write 4 Done Done New false
25203 8869 Acked 9868 Write 2 Done Done New false
25204 8870 Acked 9869 Read 6 Done Done New false
25205 8871 Acked 9870 Read 5 Done Done New false
25206 8872 Acked 9871 Read 2 Done Done New false
25207 8873 Acked 9872 Read 1 Done Done New false
25208 8874 Acked 9873 Read 9 Done Done New false
25209 8875 Acked 9874 Read 10 Done Done New false
25210 8876 Acked 9875 Read 5 Done Done New false
25211 8877 Acked 9876 Read 7 Done Done New false
25212 8878 Acked 9877 Read 9 Done Done New false
25213 8879 Acked 9878 Read 8 Done Done New false
25214 8880 Acked 9879 Read 8 Done Done New false
25215 8881 Acked 9880 Read 3 Done Done New false
25216 8882 Acked 9881 Read 7 Done Done New false
25217 8883 Acked 9882 Flush 0 Done Done New false
25218 8884 Acked 9883 Write 8 Done Done New false
25219 8885 Acked 9884 Read 4 Done Done New false
25220 8886 Acked 9885 Write 4 Done Done New false
25221 8887 Acked 9886 Read 3 Done Done New false
25222 8888 Acked 9887 Read 3 Done Done New false
25223 8889 Acked 9888 Write 2 Done Done New false
25224 8890 Acked 9889 Flush 0 Done Done New false
25225 8891 Acked 9890 Read 1 Done Done New false
25226 8892 Acked 9891 Read 1 Done Done New false
25227 8893 Acked 9892 Read 9 Done Done New false
25228 8894 Acked 9893 Write 1 Done Done New false
25229 8895 Acked 9894 Write 3 Done Done New false
25230 8896 Acked 9895 Write 9 Done Done New false
25231 8897 Acked 9896 Write 6 Done Done New false
25232 8898 Acked 9897 Read 3 Done Done New false
25233 8899 Acked 9898 Write 1 Done Done New false
25234 8900 Acked 9899 Read 6 Done Done New false
25235 8901 Acked 9900 Write 6 Done Done New false
25236 8902 Acked 9901 Read 8 Done Done New false
25237 8903 Acked 9902 Write 5 Done Done New false
25238 8904 Acked 9903 Write 8 Done Done New false
25239 8905 Acked 9904 Read 4 Done Done New false
25240 8906 Acked 9905 Read 9 Done Done New false
25241 8907 Acked 9906 Write 1 Done Done New false
25242 8908 Acked 9907 Read 10 Done Done New false
25243 8909 Acked 9908 Read 3 Done Done New false
25244 8910 Acked 9909 Write 1 Done Done New false
25245 8911 Acked 9910 Flush 0 Done Done New false
25246 8912 Acked 9911 Read 5 Done Done New false
25247 8913 Acked 9912 Flush 0 Done Done New false
25248 8914 Acked 9913 Read 1 Done Done New false
25249 8915 Acked 9914 Write 6 Done Done New false
25250 8916 Acked 9915 Read 2 Done Done New false
25251 8917 Acked 9916 Read 2 Done Done New false
25252 8918 Acked 9917 Read 9 Done Done New false
25253 8919 Acked 9918 Flush 0 Done Done New false
25254 8920 Acked 9919 Read 9 Done Done New false
25255 8921 Acked 9920 Read 9 Done Done New false
25256 8922 Acked 9921 Read 2 Done Done New false
25257 8923 Acked 9922 Write 8 Done Done New false
25258 8924 Acked 9923 Write 7 Done Done New false
25259 8925 Acked 9924 Read 7 Done Done New false
25260 8926 Acked 9925 Write 8 Done Done New false
25261 8927 Acked 9926 Read 8 Done Done New false
25262 8928 Acked 9927 Write 4 Done Done New false
25263 8929 Acked 9928 Write 3 Done Done New false
25264 8930 Acked 9929 Write 5 Done Done New false
25265 8931 Acked 9930 Write 4 Done Done New false
25266 8932 Acked 9931 Write 6 Done Done New false
25267 8933 Acked 9932 Write 8 Done Done New false
25268 8934 Acked 9933 Read 7 Done Done New false
25269 8935 Acked 9934 Write 7 Done Done New false
25270 8936 Acked 9935 Read 1 Done Done New false
25271 8937 Acked 9936 Write 6 Done Done New false
25272 8938 Acked 9937 Write 9 Done Done New false
25273 8939 Acked 9938 Read 10 Done Done New false
25274 8940 Acked 9939 Read 8 Done Done New false
25275 8941 Acked 9940 Write 10 Done Done New false
25276 8942 Acked 9941 Read 4 Done Done New false
25277 8943 Acked 9942 Write 6 Done Done New false
25278 8944 Acked 9943 Write 10 Done Done New false
25279 8945 Acked 9944 Read 7 Done Done New false
25280 8946 Acked 9945 Read 1 Done Done New false
25281 8947 Acked 9946 Read 7 Done Done New false
25282 8948 Acked 9947 Write 3 Done Done New false
25283 8949 Acked 9948 Read 2 Done Done New false
25284 8950 Acked 9949 Write 5 Done Done New false
25285 8951 Acked 9950 Write 3 Done Done New false
25286 8952 Acked 9951 Read 1 Done Done New false
25287 8953 Acked 9952 Read 3 Done Done New false
25288 8954 Acked 9953 Read 10 Done Done New false
25289 8955 Acked 9954 Read 10 Done Done New false
25290 8956 Acked 9955 Read 1 Done Done New false
25291 8957 Acked 9956 Read 4 Done Done New false
25292 8958 Acked 9957 Read 1 Done Done New false
25293 8959 Acked 9958 Read 9 Done Done New false
25294 8960 Acked 9959 Read 7 Done Done New false
25295 8961 Acked 9960 Write 7 Done Done New false
25296 8962 Acked 9961 Read 5 Done Done New false
25297 8963 Acked 9962 Write 4 Done Done New false
25298 8964 Acked 9963 Write 9 Done Done New false
25299 8965 Acked 9964 Read 3 Done Done New false
25300 8966 Acked 9965 Read 8 Done Done New false
25301 8967 Acked 9966 Flush 0 Done Done New false
25302 8968 Acked 9967 Flush 0 Done Done New false
25303 8969 Acked 9968 Flush 0 Done Done New false
25304 8970 Acked 9969 Read 5 Done Done New false
25305 8971 Acked 9970 Write 5 Done Done New false
25306 8972 Acked 9971 Flush 0 Done Done New false
25307 8973 Acked 9972 Write 2 Done Done New false
25308 8974 Acked 9973 Read 10 Done Done New false
25309 8975 Acked 9974 Read 6 Done Done New false
25310 8976 Acked 9975 Write 3 Done Done New false
25311 8977 Acked 9976 Read 8 Done Done New false
25312 8978 Acked 9977 Read 7 Done Done New false
25313 8979 Acked 9978 Flush 0 Done Done New false
25314 8980 Acked 9979 Write 6 Done Done New false
25315 8981 Acked 9980 Read 8 Done Done New false
25316 8982 Acked 9981 Write 2 Done Done New false
25317 8983 Acked 9982 Write 4 Done Done New false
25318 8984 Acked 9983 Write 9 Done Done New false
25319 8985 Acked 9984 Read 6 Done Done New false
25320 8986 Acked 9985 Read 10 Done Done New false
25321 8987 Acked 9986 Flush 0 Done Done New false
25322 8988 Acked 9987 Read 9 Done Done New false
25323 8989 Acked 9988 Write 3 Done Done New false
25324 8990 Acked 9989 Write 8 Done Done New false
25325 8991 Acked 9990 Read 2 Done Done New false
25326 8992 Acked 9991 Write 5 Done Done New false
25327 8993 Acked 9992 Read 4 Done Done New false
25328 8994 Acked 9993 Write 10 Done Done New false
25329 8995 Acked 9994 Write 10 Done Done New false
25330 8996 Acked 9995 Write 5 Done Done New false
25331 8997 Acked 9996 Write 2 Done Done New false
25332 8998 Acked 9997 Write 1 Done Done New false
25333 8999 Acked 9998 Flush 0 Done Done New false
25334 9000 Acked 9999 Read 5 Done Done New false
25335 9001 Acked 10000 Write 2 Done Done New false
25336 9002 Acked 10001 Write 1 Done Done New false
25337 9003 Acked 10002 Read 6 Done Done New false
25338 9004 Acked 10003 Read 7 Done Done New false
25339 9005 Acked 10004 Read 5 Done Done New false
25340 9006 Acked 10005 Read 6 Done Done New false
25341 9007 Acked 10006 Read 1 Done Done New false
25342 9008 Acked 10007 Write 7 Done Done New false
25343 9009 Acked 10008 Read 3 Done Done New false
25344 9010 Acked 10009 Read 9 Done Done New false
25345 9011 Acked 10010 Read 9 Done Done New false
25346 9012 Acked 10011 Read 1 Done Done New false
25347 9013 Acked 10012 Read 3 Done Done New false
25348 9014 Acked 10013 Read 4 Done Done New false
25349 9015 Acked 10014 Read 9 Done Done New false
25350 9016 Acked 10015 Read 2 Done Done New false
25351 9017 Acked 10016 Read 1 Done Done New false
25352 9018 Acked 10017 Write 3 Done Done New false
25353 9019 Acked 10018 Write 1 Done Done New false
25354 9020 Acked 10019 Read 6 Done Done New false
25355 9021 Acked 10020 Read 8 Done Done New false
25356 9022 Acked 10021 Flush 0 Done Done New false
25357 9023 Acked 10022 Read 4 Done Done New false
25358 9024 Acked 10023 Write 2 Done Done New false
25359 9025 Acked 10024 Read 8 Done Done New false
25360 9026 Acked 10025 Write 7 Done Done New false
25361 9027 Acked 10026 Write 1 Done Done New false
25362 9028 Acked 10027 Read 2 Done Done New false
25363 9029 Acked 10028 Read 5 Done Done New false
25364 9030 Acked 10029 Write 3 Done Done New false
25365 9031 Acked 10030 Write 10 Done Done New false
25366 9032 Acked 10031 Write 2 Done Done New false
25367 9033 Acked 10032 Write 10 Done Done New false
25368 9034 Acked 10033 Read 1 Done Done New false
25369 9035 Acked 10034 Read 4 Done Done New false
25370 9036 Acked 10035 Write 9 Done Done New false
25371 9037 Acked 10036 Write 6 Done Done New false
25372 9038 Acked 10037 Write 2 Done Done New false
25373 9039 Acked 10038 Read 1 Done Done New false
25374 9040 Acked 10039 Write 9 Done Done New false
25375 9041 Acked 10040 Write 6 Done Done New false
25376 9042 Acked 10041 Read 5 Done Done New false
25377 9043 Acked 10042 Write 9 Done Done New false
25378 9044 Acked 10043 Read 2 Done Done New false
25379 9045 Acked 10044 Read 7 Done Done New false
25380 9046 Acked 10045 Read 3 Done Done New false
25381 9047 Acked 10046 Read 9 Done Done New false
25382 9048 Acked 10047 Read 6 Done Done New false
25383 9049 Acked 10048 Read 2 Done Done New false
25384 9050 Acked 10049 Write 6 Done Done New false
25385 9051 Acked 10050 Write 1 Done Done New false
25386 9052 Acked 10051 Write 4 Done Done New false
25387 9053 Acked 10052 Write 5 Done Done New false
25388 9054 Acked 10053 Read 7 Done Done New false
25389 9055 Acked 10054 Write 6 Done Done New false
25390 9056 Acked 10055 Write 10 Done Done New false
25391 9057 Acked 10056 Read 6 Done Done New false
25392 9058 Acked 10057 Read 10 Done Done New false
25393 9059 Acked 10058 Read 1 Done Done New false
25394 9060 Acked 10059 Read 3 Done Done New false
25395 9061 Acked 10060 Flush 0 Done Done New false
25396 9062 Acked 10061 Read 3 Done Done New false
25397 9063 Acked 10062 Write 9 Done Done New false
25398 9064 Acked 10063 Read 1 Done Done New false
25399 9065 Acked 10064 Write 9 Done Done New false
25400 9066 Acked 10065 Read 4 Done Done New false
25401 9067 Acked 10066 Read 6 Done Done New false
25402 9068 Acked 10067 Flush 0 Done Done New false
25403 9069 Acked 10068 Write 4 Done Done New false
25404 9070 Acked 10069 Read 9 Done Done New false
25405 9071 Acked 10070 Flush 0 Done Done New false
25406 9072 Acked 10071 Write 2 Done Done New false
25407 9073 Acked 10072 Read 6 Done Done New false
25408 9074 Acked 10073 Read 9 Done Done New false
25409 9075 Acked 10074 Write 6 Done Done New false
25410 9076 Acked 10075 Read 7 Done Done New false
25411 9077 Acked 10076 Write 5 Done Done New false
25412 9078 Acked 10077 Read 5 Done Done New false
25413 9079 Acked 10078 Write 7 Done Done New false
25414 9080 Acked 10079 Read 5 Done Done New false
25415 9081 Acked 10080 Flush 0 Done Done New false
25416 STATES DS:0 DS:1 DS:2 TOTAL
25417 New 0 0 303 303
25418 Sent 0 0 0 0
25419 Done 303 303 0 606
25420 Skipped 0 0 0 0
25421 Error 0 0 0 0
25422 Last Flush: 10080 10080 9777
25423 Downstairs last five completed: 9777 9776 9775 9774 9773 9772
25424 Upstairs last five completed: 9081 9080 9079 9078 9077 9076
25425 CLIENT: Up:0 ds:303 act:2
25426 [8830] got start action
25427 [8830] I am going to restart
25428 Starting downstairs at port 8830
25429 Make output file at "/tmp/dsc/downstairs-8830.txt"
25430 [8830][2] reports Starting
25431 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1206)
25432 [8830][2] reports Running
254332023-10-13T18:14:38.466ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c looper connected looper = 2
254342023-10-13T18:14:38.466ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
254352023-10-13T18:14:38.466ZINFOcrucible: [2] client re-new 303 jobs since flush 9777 = downstairs
254362023-10-13T18:14:38.467ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session c526496c-9bef-4440-8022-c243881533f9
254372023-10-13T18:14:38.467ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
254382023-10-13T18:14:38.467ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
254392023-10-13T18:14:38.467ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
254402023-10-13T18:14:38.467ZINFOcrucible: [2] send last flush ID to this DS: 9777
254412023-10-13T18:14:38.467ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Offline
254422023-10-13T18:14:38.468ZINFOcrucible: [2] Replied this last flush ID: 9777
254432023-10-13T18:14:38.468ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Offline ds_transition to Replay
254442023-10-13T18:14:38.468ZINFOcrucible: [2] Transition from Offline to Replay
254452023-10-13T18:14:38.468ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c Transition from Replay to Active
254462023-10-13T18:14:38.468ZINFOcrucible: [2] fe698f06-923e-45d4-b592-2bcd1ce0484c (c526496c-9bef-4440-8022-c243881533f9) Active Active Replay ds_transition to Active
254472023-10-13T18:14:38.468ZINFOcrucible: [2] Transition from Replay to Active
254482023-10-13T18:14:38.468ZINFOcrucible: [2] Starts cmd_loop
254492023-10-13T18:14:38.468ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
254502023-10-13T18:14:38.468ZINFOcrucible: fe698f06-923e-45d4-b592-2bcd1ce0484c Active Active Active
254512023-10-13T18:14:38.468ZINFOcrucible: Set check for repair
254522023-10-13T18:14:39.469ZINFOcrucible: Checking if live repair is needed
254532023-10-13T18:14:39.469ZINFOcrucible: No Live Repair required at this time
254542023-10-13T18:14:39.469ZWARNcrucible: [2] flow control end
25455 ----------------------------------------------------------------
25456 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25457 Downstairs last five completed: 10080 10079 10078 10077 10076 10075
25458 Upstairs last five completed: 9081 9080 9079 9078 9077 9076
25459 CLIENT: Up:0 ds:0 act:3
25460 Replay: All jobs finished, all DS active.
25461 Test replay has completed
25462 Wrote out file "alan"
25463 CLIENT: Tests done. All submitted work has been ACK'd
25464 ----------------------------------------------------------------
25465 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25466 Downstairs last five completed: 10080 10079 10078 10077 10076 10075
25467 Upstairs last five completed: 9081 9080 9079 9078 9077 9076
25468 CLIENT: Up:0 ds:0 act:3
25469 CLIENT: All jobs finished, all DS active.
25470 Replays:30 time: 5:33
25471 Do final verify
254722023-10-13T18:14:40.490ZINFOcrucible: Upstairs starts
254732023-10-13T18:14:40.491ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "a730f3b26afaf115f27149ef849470fd85caef41", git_commit_timestamp: "2023-10-13T17:42:12.000000000Z", git_branch: "main", rustc_semver: "1.70.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "90c541806f23a127002de5b4038be731ba1458ca", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
254742023-10-13T18:14:40.491ZINFOcrucible: Upstairs <-> Downstairs Message Version: 4
254752023-10-13T18:14:40.491ZINFOcrucible: Upstairs opts: Upstairs UUID: 2a900e23-6d38-4b9d-aa47-98ceb10607d2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false
254762023-10-13T18:14:40.491ZINFOcrucible: Crucible stats registered with UUID: 2a900e23-6d38-4b9d-aa47-98ceb10607d2
254772023-10-13T18:14:40.491ZINFOcrucible: Crucible 2a900e23-6d38-4b9d-aa47-98ceb10607d2 has session id: 73fbc9dd-aa95-44f0-984a-5624ba11f6be
25478 Crucible runtime is spawned
25479 The guest has requested activation with gen:3
254802023-10-13T18:14:40.491ZINFOcrucible: up_listen starts task = up_listen
254812023-10-13T18:14:40.491ZINFOcrucible: Wait for all three downstairs to come online
254822023-10-13T18:14:40.491ZINFOcrucible: Flush timeout: 0.5
254832023-10-13T18:14:40.491ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
254842023-10-13T18:14:40.492ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
254852023-10-13T18:14:40.492ZINFOcrucible: [0] 2a900e23-6d38-4b9d-aa47-98ceb10607d2 looper connected looper = 0
254862023-10-13T18:14:40.492ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
254872023-10-13T18:14:40.492ZINFOcrucible: [1] 2a900e23-6d38-4b9d-aa47-98ceb10607d2 looper connected looper = 1
254882023-10-13T18:14:40.492ZINFOcrucible: [2] 2a900e23-6d38-4b9d-aa47-98ceb10607d2 looper connected looper = 2
254892023-10-13T18:14:40.493ZINFOcrucible: 2a900e23-6d38-4b9d-aa47-98ceb10607d2 active request set
254902023-10-13T18:14:40.493ZINFOcrucible: Set desired generation to :3
254912023-10-13T18:14:40.493ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state New
254922023-10-13T18:14:40.493ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state New
254932023-10-13T18:14:40.493ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state New
254942023-10-13T18:14:40.493ZINFOcrucible: [1] 2a900e23-6d38-4b9d-aa47-98ceb10607d2 (66b233b9-5ea3-4738-9040-f72d1fd20914) New New New ds_transition to WaitActive
254952023-10-13T18:14:40.494ZINFOcrucible: [1] Transition from New to WaitActive
254962023-10-13T18:14:40.494ZINFOcrucible: [1] client is_active_req TRUE, promote! session 66b233b9-5ea3-4738-9040-f72d1fd20914
254972023-10-13T18:14:40.494ZINFOcrucible: [0] 2a900e23-6d38-4b9d-aa47-98ceb10607d2 (66b233b9-5ea3-4738-9040-f72d1fd20914) New WaitActive New ds_transition to WaitActive
254982023-10-13T18:14:40.494ZINFOcrucible: [0] Transition from New to WaitActive
254992023-10-13T18:14:40.494ZINFOcrucible: [0] client is_active_req TRUE, promote! session 66b233b9-5ea3-4738-9040-f72d1fd20914
255002023-10-13T18:14:40.494ZINFOcrucible: [2] 2a900e23-6d38-4b9d-aa47-98ceb10607d2 (66b233b9-5ea3-4738-9040-f72d1fd20914) WaitActive WaitActive New ds_transition to WaitActive
255012023-10-13T18:14:40.494ZINFOcrucible: [2] Transition from New to WaitActive
255022023-10-13T18:14:40.494ZINFOcrucible: [2] client is_active_req TRUE, promote! session 66b233b9-5ea3-4738-9040-f72d1fd20914
255032023-10-13T18:14:40.494ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
255042023-10-13T18:14:40.494ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
255052023-10-13T18:14:40.495ZINFOcrucible: 2a900e23-6d38-4b9d-aa47-98ceb10607d2 WaitActive WaitActive WaitActive
255062023-10-13T18:14:40.495ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
255072023-10-13T18:14:40.495ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
255082023-10-13T18:14:40.495ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
255092023-10-13T18:14:40.495ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
255102023-10-13T18:14:40.495ZINFOcrucible: 2a900e23-6d38-4b9d-aa47-98ceb10607d2 WaitActive WaitActive WaitActive
255112023-10-13T18:14:40.495ZINFOcrucible: 2a900e23-6d38-4b9d-aa47-98ceb10607d2 WaitActive WaitActive WaitActive
255122023-10-13T18:14:40.498ZINFOcrucible: [1] 2a900e23-6d38-4b9d-aa47-98ceb10607d2 (66b233b9-5ea3-4738-9040-f72d1fd20914) WaitActive WaitActive WaitActive ds_transition to WaitQuorum
255132023-10-13T18:14:40.498ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum
255142023-10-13T18:14:40.498ZWARNcrucible: [1] new RM replaced this: None
255152023-10-13T18:14:40.498ZINFOcrucible: [1] Starts reconcile loop
255162023-10-13T18:14:40.498ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
255172023-10-13T18:14:40.498ZINFOcrucible: 2a900e23-6d38-4b9d-aa47-98ceb10607d2 WaitActive WaitQuorum WaitActive
255182023-10-13T18:14:40.498ZINFOcrucible: Waiting for 2 more clients to be ready
255192023-10-13T18:14:40.498ZINFOcrucible: Set check for repair
255202023-10-13T18:14:40.498ZINFOcrucible: [2] 2a900e23-6d38-4b9d-aa47-98ceb10607d2 (66b233b9-5ea3-4738-9040-f72d1fd20914) WaitActive WaitQuorum WaitActive ds_transition to WaitQuorum
255212023-10-13T18:14:40.498ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum
255222023-10-13T18:14:40.498ZWARNcrucible: [2] new RM replaced this: None
255232023-10-13T18:14:40.498ZINFOcrucible: [2] Starts reconcile loop
255242023-10-13T18:14:40.498ZINFOcrucible: [0] 2a900e23-6d38-4b9d-aa47-98ceb10607d2 (66b233b9-5ea3-4738-9040-f72d1fd20914) WaitActive WaitQuorum WaitQuorum ds_transition to WaitQuorum
255252023-10-13T18:14:40.498ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
255262023-10-13T18:14:40.498ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum
255272023-10-13T18:14:40.498ZWARNcrucible: [0] new RM replaced this: None
255282023-10-13T18:14:40.498ZINFOcrucible: 2a900e23-6d38-4b9d-aa47-98ceb10607d2 WaitQuorum WaitQuorum WaitQuorum
255292023-10-13T18:14:40.498ZINFOcrucible: [0] Starts reconcile loop
255302023-10-13T18:14:40.498ZINFOcrucible: [0]R flush_numbers[0..12]: [979, 985, 971, 990, 959, 990, 987, 983, 987, 987, 979, 979]
255312023-10-13T18:14:40.498ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]
255322023-10-13T18:14:40.498ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
255332023-10-13T18:14:40.498ZINFOcrucible: [1]R flush_numbers[0..12]: [979, 985, 971, 990, 959, 990, 987, 983, 987, 987, 979, 979]
255342023-10-13T18:14:40.498ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]
255352023-10-13T18:14:40.498ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
255362023-10-13T18:14:40.498ZINFOcrucible: [2]R flush_numbers[0..12]: [979, 985, 971, 990, 959, 990, 987, 983, 987, 987, 979, 979]
255372023-10-13T18:14:40.498ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]
255382023-10-13T18:14:40.498ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
255392023-10-13T18:14:40.498ZINFOcrucible: Max found gen is 3
255402023-10-13T18:14:40.498ZINFOcrucible: Generation requested: 3 >= found:3
255412023-10-13T18:14:40.498ZINFOcrucible: Next flush: 991
25542 {"msg":"All extents match","v":0,"name":"crucible","level":30The guest has finished waiting for activation with:3
25543 ,"timeWait for a query_work_queue command to finish before sending IO
25544 ":"2023-10-13T18:14:40.498900257Z","hostname":"ip-10-150-1-136.us-west-2.compute.internal","pid":1211}
255452023-10-13T18:14:40.498ZINFOcrucible: No downstairs repair required
255462023-10-13T18:14:40.498ZINFOcrucible: No initial repair work was required
255472023-10-13T18:14:40.498ZINFOcrucible: Set Downstairs and Upstairs active
255482023-10-13T18:14:40.498ZINFOcrucible: 2a900e23-6d38-4b9d-aa47-98ceb10607d2 is now active with session: 66b233b9-5ea3-4738-9040-f72d1fd20914
255492023-10-13T18:14:40.498ZINFOcrucible: 2a900e23-6d38-4b9d-aa47-98ceb10607d2 Set Active after no repair
255502023-10-13T18:14:40.499ZINFOcrucible: Notify all downstairs, region set compare is done.
255512023-10-13T18:14:40.499ZINFOcrucible: Set check for repair
255522023-10-13T18:14:40.499ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
255532023-10-13T18:14:40.499ZINFOcrucible: 2a900e23-6d38-4b9d-aa47-98ceb10607d2 Active Active Active
255542023-10-13T18:14:40.499ZINFOcrucible: Set check for repair
255552023-10-13T18:14:40.499ZINFOcrucible: [1] received reconcile message
255562023-10-13T18:14:40.499ZINFOcrucible: [2] received reconcile message
255572023-10-13T18:14:40.499ZINFOcrucible: [0] received reconcile message
255582023-10-13T18:14:40.499ZINFOcrucible: [1] All repairs completed, exit
255592023-10-13T18:14:40.499ZINFOcrucible: [1] Starts cmd_loop
255602023-10-13T18:14:40.499ZINFOcrucible: [2] All repairs completed, exit
255612023-10-13T18:14:40.499ZINFOcrucible: [2] Starts cmd_loop
255622023-10-13T18:14:40.499ZINFOcrucible: [0] All repairs completed, exit
255632023-10-13T18:14:40.499ZINFOcrucible: [0] Starts cmd_loop
25564 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
25565 Loading write count information from file "alan"
25566 Read and Verify all blocks (0..5000 range:false)
255672023-10-13T18:14:41.525ZINFOcrucible: Checking if live repair is needed
255682023-10-13T18:14:41.525ZINFOcrucible: No Live Repair required at this time
25569 Wrote out file "alan"
25570 Verify test completed
25571 Wrote out file "alan"
25572 CLIENT: Tests done. All submitted work has been ACK'd
25573 ----------------------------------------------------------------
25574 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
25575 Downstairs last five completed: 1053 1052 1051 1050 1049 1048
25576 Upstairs last five completed: 54 53 52 51 50 49
25577 CLIENT: Up:0 ds:0 act:3
25578 CLIENT: All crucible jobs finished, exiting program
25579 Oct 13 18:14:42.683 INFO accepted connection, remote_addr: 127.0.0.1:59240, local_addr: 127.0.0.1:9998
25580 Main task has work to do, go find it
25581 got dsc Shutdown
25582 Shutdown
25583 Shut it down
25584 Oct 13 18:14:42.684 INFO request completed, latency_us: 291, response_code: 204, uri: /shutdown, method: POST, req_id: 51a6faea-6c20-4acf-a8b5-2493df21e1bd, remote_addr[8810] Disable keep_running
25585 :[8830] Disable keep_running
25586 127.0.0.1:59240, local_addr: 127.0.0.1:9998
25587 [8830] Got stop action so:false kr:false
25588 [8820] Disable keep_running
25589 [8810] Got stop action so:false kr:false
25590 [8820] Got stop action so:false kr:false
25591 [8830] Exited with: ExitStatus(unix_wait_status(9))
25592 [8830][2] reports Exit
25593 [8820] Exited with: ExitStatus(unix_wait_status(9))
25594 [8820][1] reports Exit
25595 [8810] Exited with: ExitStatus(unix_wait_status(9))
25596 [8810][0] reports Exit
25597 October 13, 2023 at 06:14:47 PM UTC Test ends with 0