Max level shown:
1 starting December 9, 2024 at 03:07:44 PM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.26954886
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.023068173
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.023128232
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start access at:127.0.0.1:9998
14 start ds: 8820
15 start ds: 8830
16 Starting downstairs at port 8810
17 Starting downstairs at port 8820
18 Make output file at "/tmp/dsc/downstairs-8820.txt"
19 Make output file at "/tmp/dsc/downstairs-8810.txt"
20 Starting downstairs at port 8830
21 Make output file at "/tmp/dsc/downstairs-8830.txt"
22 [8810][0] initial start wait reports Starting
23 [8820][1] initial start wait reports Starting
24 [8830][2] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1191)
26 [8820][1] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1192)
28 [8830][2] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1193)
30 [8810][0] initial start wait reports Running
31 All downstairs are running
32 Dec 09 15:07:45.243 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Dec 09 15:07:50.329 INFO accepted connection, remote_addr: 127.0.0.1:55986, local_addr: 127.0.0.1:9998
36 Dec 09 15:07:50.332 INFO request completed, latency_us: 544, response_code: 200, uri: /regioninfo, method: GET, req_id: 4e4facec-eaf4-4cf1-90de-7e59dbd7d2b5, remote_addr: 127.0.0.1:55986, local_addr: 127.0.0.1:9998
37 Dec 09 15:07:50.334 INFO request completed, latency_us: 152, response_code: 200, uri: /regioncount, method: GET, req_id: 21395b2b-6539-4566-a7e8-d88e19aa9920, remote_addr: 127.0.0.1:55986, local_addr: 127.0.0.1:9998
382024-12-09T15:07:50.334ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392024-12-09T15:07:50.334ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Dec 09 15:07:50.335 INFO request completed, latency_us: 170, response_code: 200, uri: /port/cid/0, method: GET, req_id: 5459779e-c81f-4e52-bb7e-b621ad0dcc99, remote_addr: 127.0.0.1:55986, local_addr: 127.0.0.1:9998
41 Dec 09 15:07:50.335 INFO request completed, latency_us: 98, response_code: 200, uri: /port/cid/1, method: GET, req_id: 6a90599b-1707-44e5-9bb1-efef5fc37242, remote_addr: 127.0.0.1:55986, local_addr: 127.0.0.1:9998
42 Dec 09 15:07:50.336 INFO request completed, latency_us: 97, response_code: 200, uri: /port/cid/2, method: GET, req_id: b397fbc6-3622-4886-bec0-fe1bd2bac891, remote_addr: 127.0.0.1:55986, local_addr: 127.0.0.1:9998
432024-12-09T15:07:50.336ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442024-12-09T15:07:50.337ZINFOcrucible: Upstairs starts
452024-12-09T15:07:50.338ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "2978a42101da810efa0e51ea3f370f1f90c0069b", git_commit_timestamp: "2024-12-09T14:48:58.000000000Z", git_branch: "mkeeter/simplify-faults", rustc_semver: "1.82.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "f6e511eec7342f59a25f7c0534f1dbea00d01b14", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462024-12-09T15:07:50.338ZINFOcrucible: Upstairs <-> Downstairs Message Version: 12
472024-12-09T15:07:50.338ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482024-12-09T15:07:50.338ZINFOcrucible: Crucible c2fc35a3-3bd8-45e9-aa58-70f889a4b0a4 has session id: e80579d2-4ffb-4efa-81eb-05898168ffc0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
492024-12-09T15:07:50.339ZINFOcrucible: Upstairs opts: Upstairs UUID: c2fc35a3-3bd8-45e9-aa58-70f889a4b0a4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
502024-12-09T15:07:50.339ZINFOcrucible: Crucible stats registered with UUID: c2fc35a3-3bd8-45e9-aa58-70f889a4b0a4 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
512024-12-09T15:07:50.377ZINFOcrucible: Downstairs in volume = 3 task = crutest
522024-12-09T15:07:50.377ZINFOcrucible: The guest has requested activation with gen:1
532024-12-09T15:07:50.377ZINFOcrucible: c2fc35a3-3bd8-45e9-aa58-70f889a4b0a4 active request set session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
542024-12-09T15:07:50.377ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
552024-12-09T15:07:50.377ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
562024-12-09T15:07:50.377ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
572024-12-09T15:07:50.377ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
582024-12-09T15:07:50.377ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
592024-12-09T15:07:50.377ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
602024-12-09T15:07:50.377ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
612024-12-09T15:07:50.377ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
622024-12-09T15:07:50.377ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
632024-12-09T15:07:50.377ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63519) = io task client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
642024-12-09T15:07:50.377ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61431) = io task client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
652024-12-09T15:07:50.377ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44851) = io task client = 2 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
662024-12-09T15:07:50.379ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
672024-12-09T15:07:50.379ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
682024-12-09T15:07:50.379ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
692024-12-09T15:07:50.379ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
702024-12-09T15:07:50.379ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
712024-12-09T15:07:50.379ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
722024-12-09T15:07:50.380ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
732024-12-09T15:07:50.380ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
742024-12-09T15:07:50.380ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
752024-12-09T15:07:50.381ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
762024-12-09T15:07:50.381ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
772024-12-09T15:07:50.381ZINFOcrucible: Waiting for more clients to be ready session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
782024-12-09T15:07:50.381ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
792024-12-09T15:07:50.381ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
802024-12-09T15:07:50.381ZINFOcrucible: Waiting for more clients to be ready session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
812024-12-09T15:07:50.381ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
822024-12-09T15:07:50.381ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
832024-12-09T15:07:50.382ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
842024-12-09T15:07:50.382ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
852024-12-09T15:07:50.382ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
862024-12-09T15:07:50.382ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
872024-12-09T15:07:50.382ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
882024-12-09T15:07:50.382ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
892024-12-09T15:07:50.382ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
902024-12-09T15:07:50.382ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
912024-12-09T15:07:50.382ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
922024-12-09T15:07:50.382ZINFOcrucible: Max found gen is 1 = downstairs session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
932024-12-09T15:07:50.382ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
942024-12-09T15:07:50.382ZINFOcrucible: Next flush: 1 = downstairs session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
952024-12-09T15:07:50.382ZINFOcrucible: All extents match = downstairs session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
962024-12-09T15:07:50.382ZINFOcrucible: No downstairs reconciliation required session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
972024-12-09T15:07:50.382ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
982024-12-09T15:07:50.382ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
992024-12-09T15:07:50.382ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
1002024-12-09T15:07:50.382ZINFOcrucible: All required reconciliation work is completed session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
1012024-12-09T15:07:50.382ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
1022024-12-09T15:07:50.382ZINFOcrucible: c2fc35a3-3bd8-45e9-aa58-70f889a4b0a4 is now active with session: e80579d2-4ffb-4efa-81eb-05898168ffc0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
1032024-12-09T15:07:50.382ZINFOcrucible: Set Active after no reconciliation session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
1042024-12-09T15:07:50.382ZINFOcrucible: The guest has finished waiting for activation with:1
105 Activate successful
106 Wait for a query_work_queue command to finish before sending IO
107 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
108 Fill test
109 Read and Verify all blocks (0..5000 range:false)
110 Wrote out file "/tmp/test_replay_verify.log"
111 CLIENT: Tests done. All submitted work has been ACK'd
112 ----------------------------------------------------------------
113 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
114 States: Active Active Active
115 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
116 1021 Acked Read 256 Done Done Done false
117 1022 Acked Read 256 Done Done Done false
118 1023 Acked Read 256 Done Done Done false
119 1024 Acked Read 256 Done Done Done false
120 1025 Acked Read 256 Done Done Done false
121 1026 Acked Read 256 Done Done Done false
122 1027 Acked Read 256 Done Done Done false
123 1028 Acked Read 256 Done Done Done false
124 1029 Acked Read 256 Done Done Done false
125 1030 Acked Read 256 Done Done Done false
126 1031 Acked Read 256 Done Done Done false
127 1032 Acked Read 256 Done Done Done false
128 1033 Acked Read 256 Done Done Done false
129 1034 Acked Read 256 Done Done Done false
130 1035 Acked Read 256 Done Done Done false
131 1036 Acked Read 256 Done Done Done false
132 1037 Acked Read 256 Done Done Done false
133 1038 Acked Read 256 Done Done Done false
134 1039 Acked Read 136 Done Done Done false
135 1040 Acked Read 256 Done Done Done false
136 STATES DS:0 DS:1 DS:2 TOTAL
137 Sent 0 0 0 0
138 Done 20 20 20 60
139 Skipped 0 0 0 0
140 Error 0 0 0 0
141 Last Flush: 1020 1020 1020
142 Downstairs last five completed: 1020 1019 1018 1017 1016
143 Upstairs last five completed: 1040 1039 1038 1037 1036
144 CLIENT: Up:0 ds:20 act:3
145 ----------------------------------------------------------------
146 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
147 Downstairs last five completed: 1041 1040 1039 1038 1037
148 Upstairs last five completed: 1041 1040 1039 1038 1037
149 CLIENT: Up:0 ds:0 act:3
150 CLIENT: All crucible jobs finished, exiting program
1512024-12-09T15:07:54.952ZWARNcrucible: Guest handle has been dropped
1522024-12-09T15:07:54.952ZWARNcrucible: client_stop_rx closed unexpectedly: RecvError(()) = io task client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
1532024-12-09T15:07:54.952ZWARNcrucible: client task is sending Done(QueueClosed) = io task client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
1542024-12-09T15:07:54.952ZWARNcrucible: client task could not reply to main task; shutting down? = io task client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
1552024-12-09T15:07:54.952ZWARNcrucible: client_stop_rx closed unexpectedly: RecvError(()) = io task client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
1562024-12-09T15:07:54.952ZINFOcrucible: client task is exiting = io task client = 0 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
1572024-12-09T15:07:54.952ZWARNcrucible: client task is sending Done(QueueClosed) = io task client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
1582024-12-09T15:07:54.952ZWARNcrucible: client task could not reply to main task; shutting down? = io task client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
1592024-12-09T15:07:54.952ZINFOcrucible: client task is exiting = io task client = 1 session_id = e80579d2-4ffb-4efa-81eb-05898168ffc0
160 Replay loop starts now December 9, 2024 at 03:07:54 PM UTC
161 Dec 09 15:07:55.029 INFO accepted connection, remote_addr: 127.0.0.1:47424, local_addr: 127.0.0.1:9998
162 Dec 09 15:07:55.030 INFO request completed, latency_us: 186, response_code: 200, uri: /regioninfo, method: GET, req_id: 0e80f920-c249-4bbd-9917-f3544dbadaae, remote_addr: 127.0.0.1:47424, local_addr: 127.0.0.1:9998
1632024-12-09T15:07:55.030ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
164 Dec 09 15:07:55.031 INFO request completed, latency_us: 99, response_code: 200, uri: /regioncount, method: GET, req_id: aed579fb-f549-43a5-8dac-5a048ed56f16, remote_addr: 127.0.0.1:47424, local_addr: 127.0.0.1:9998
1652024-12-09T15:07:55.031ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
166 Dec 09 15:07:55.032 INFO request completed, latency_us: 125, response_code: 200, uri: /port/cid/0, method: GET, req_id: 6e628eef-f7d3-4ad2-93d9-bc8ce7978be6, remote_addr: 127.0.0.1:47424, local_addr: 127.0.0.1:9998
167 Dec 09 15:07:55.032 INFO request completed, latency_us: 92, response_code: 200, uri: /port/cid/1, method: GET, req_id: ded99304-c9ff-4d3d-ad13-c99c9c4e1b93, remote_addr: 127.0.0.1:47424, local_addr: 127.0.0.1:9998
168 Dec 09 15:07:55.033 INFO request completed, latency_us: 96, response_code: 200, uri: /port/cid/2, method: GET, req_id: a275343d-d037-43f5-9a1b-c01dab0dead4, remote_addr: 127.0.0.1:47424, local_addr: 127.0.0.1:9998
1692024-12-09T15:07:55.033ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1702024-12-09T15:07:55.033ZINFOcrucible: Upstairs starts
1712024-12-09T15:07:55.033ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "2978a42101da810efa0e51ea3f370f1f90c0069b", git_commit_timestamp: "2024-12-09T14:48:58.000000000Z", git_branch: "mkeeter/simplify-faults", rustc_semver: "1.82.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "f6e511eec7342f59a25f7c0534f1dbea00d01b14", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1722024-12-09T15:07:55.033ZINFOcrucible: Upstairs <-> Downstairs Message Version: 12
1732024-12-09T15:07:55.033ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1742024-12-09T15:07:55.034ZINFOcrucible: Crucible 38c902f2-438a-4643-974f-c822bb91347b has session id: 585553f3-c0fa-4eaa-8619-2b4b00d00492 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1752024-12-09T15:07:55.034ZINFOcrucible: Upstairs opts: Upstairs UUID: 38c902f2-438a-4643-974f-c822bb91347b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1762024-12-09T15:07:55.034ZINFOcrucible: Crucible stats registered with UUID: 38c902f2-438a-4643-974f-c822bb91347b session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1772024-12-09T15:07:55.068ZINFOcrucible: Downstairs in volume = 3 task = crutest
1782024-12-09T15:07:55.068ZINFOcrucible: The guest has requested activation with gen:2
1792024-12-09T15:07:55.068ZINFOcrucible: 38c902f2-438a-4643-974f-c822bb91347b active request set session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1802024-12-09T15:07:55.068ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1812024-12-09T15:07:55.068ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1822024-12-09T15:07:55.068ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1832024-12-09T15:07:55.069ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1842024-12-09T15:07:55.069ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1852024-12-09T15:07:55.069ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1862024-12-09T15:07:55.069ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1872024-12-09T15:07:55.069ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1882024-12-09T15:07:55.069ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1892024-12-09T15:07:55.069ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50987) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1902024-12-09T15:07:55.069ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51499) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1912024-12-09T15:07:55.069ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55648) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1922024-12-09T15:07:55.070ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1932024-12-09T15:07:55.070ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1942024-12-09T15:07:55.070ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1952024-12-09T15:07:55.070ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1962024-12-09T15:07:55.070ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1972024-12-09T15:07:55.070ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1982024-12-09T15:07:55.070ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1992024-12-09T15:07:55.071ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2002024-12-09T15:07:55.071ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2012024-12-09T15:07:55.071ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2022024-12-09T15:07:55.071ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2032024-12-09T15:07:55.071ZINFOcrucible: Waiting for more clients to be ready session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2042024-12-09T15:07:55.071ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2052024-12-09T15:07:55.071ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2062024-12-09T15:07:55.071ZINFOcrucible: Waiting for more clients to be ready session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2072024-12-09T15:07:55.071ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2082024-12-09T15:07:55.071ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2092024-12-09T15:07:55.071ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2102024-12-09T15:07:55.071ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2112024-12-09T15:07:55.071ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2122024-12-09T15:07:55.071ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2132024-12-09T15:07:55.071ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2142024-12-09T15:07:55.071ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2152024-12-09T15:07:55.071ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2162024-12-09T15:07:55.071ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2172024-12-09T15:07:55.071ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2182024-12-09T15:07:55.072ZINFOcrucible: Max found gen is 2 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2192024-12-09T15:07:55.072ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2202024-12-09T15:07:55.072ZINFOcrucible: Next flush: 2 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
221 {"msg":"All extents match","v":0,"name":"crucible","level":30Activate successful
222 ,"Wait for a query_work_queue command to finish before sending IO
223 time":"2024-12-09T15:07:55.072049758Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"downstairs","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
2242024-12-09T15:07:55.072ZINFOcrucible: No downstairs reconciliation required session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2252024-12-09T15:07:55.072ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2262024-12-09T15:07:55.072ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2272024-12-09T15:07:55.072ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2282024-12-09T15:07:55.072ZINFOcrucible: All required reconciliation work is completed session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2292024-12-09T15:07:55.072ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2302024-12-09T15:07:55.072ZINFOcrucible: 38c902f2-438a-4643-974f-c822bb91347b is now active with session: 585553f3-c0fa-4eaa-8619-2b4b00d00492 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2312024-12-09T15:07:55.072ZINFOcrucible: Set Active after no reconciliation session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2322024-12-09T15:07:55.072ZINFOcrucible: The guest has finished waiting for activation with:2
233 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
234 Loading write count information from file "/tmp/test_replay_verify.log"
235 Dec 09 15:07:55.106 INFO accepted connection, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
236 Main task has work to do, go find it
237 got dsc Stop(2)
238 stop 2
239 Dec 09 15:07:55.107 INFO request completed, latency_us: 228, response_code: 204, uri: /stop/cid/2, method: POST, req_id: fef9c455-3294-4be9-ada6-51efaa1169dc, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
240 [8830] Got stop action so:false kr:false
2412024-12-09T15:07:55.108ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2422024-12-09T15:07:55.108ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
243 Dec 09 15:07:55.108 INFO request completed, latency_us: 123, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2ea3709a-75c7-4b61-b8f7-049840f7b0ee, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
2442024-12-09T15:07:55.108ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2452024-12-09T15:07:55.108ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2462024-12-09T15:07:55.108ZINFOcrucible: [2] Transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2472024-12-09T15:07:55.108ZINFOcrucible: client task is exiting = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
248 [8830] Exited with: ExitStatus(unix_wait_status(9))
249 [8830][2] reports Exit
250 Dec 09 15:07:59.110 INFO request completed, latency_us: 249, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8dca856f-6ce9-4d85-9e5b-afac1ad36283, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
251 001/300 Read block 3765 len 16384
252 002/300 Write block 2369 len 4096 data: 2
253 003/300 Flush
254 004/300 Read block 3522 len 4096
255 005/300 Write block 133 len 40960 data: 2 2 2 2 2 2 2 2 2 2
256 006/300 Write block 3393 len 28672 data: 2 2 2 2 2 2 2
257 007/300 Flush
258 008/300 Read block 1516 len 40960
259 009/300 Write block 1568 len 16384 data: 2 2 2 2
260 010/300 Write block 932 len 4096 data: 2
261 011/300 Write block 3846 len 28672 data: 2 2 2 2 2 2 2
262 012/300 Write block 2336 len 40960 data: 2 2 2 2 2 2 2 2 2 2
263 013/300 Read block 128 len 4096
264 014/300 Write block 2297 len 8192 data: 2 2
265 015/300 Read block 3565 len 40960
266 016/300 Write block 1568 len 8192 data: 3 3
267 017/300 Read block 4015 len 4096
268 018/300 Read block 1342 len 24576
269 019/300 Write block 2618 len 32768 data: 2 2 2 2 2 2 2 2
270 020/300 Read block 4598 len 28672
271 021/300 Write block 4120 len 20480 data: 2 2 2 2 2
272 022/300 Write block 1651 len 40960 data: 2 2 2 2 2 2 2 2 2 2
273 023/300 Read block 1726 len 32768
274 024/300 Read block 1785 len 28672
275 025/300 Write block 3398 len 24576 data: 3 3 2 2 2 2
276 026/300 Read block 2891 len 20480
277 027/300 Write block 417 len 4096 data: 2
278 028/300 Read block 3183 len 32768
279 029/300 Write block 1926 len 32768 data: 2 2 2 2 2 2 2 2
280 030/300 Write block 607 len 12288 data: 2 2 2
281 031/300 Flush
282 032/300 Read block 344 len 20480
283 033/300 Read block 2303 len 20480
284 034/300 Write block 478 len 20480 data: 2 2 2 2 2
285 035/300 Read block 2850 len 40960
286 036/300 Read block 1275 len 12288
287 037/300 Read block 1278 len 16384
288 038/300 Write block 1929 len 24576 data: 3 3 3 3 3 2
289 039/300 Read block 2710 len 12288
290 040/300 Write block 1368 len 32768 data: 2 2 2 2 2 2 2 2
291 041/300 Write block 1842 len 4096 data: 2
292 042/300 Flush
293 043/300 Read block 1484 len 36864
294 044/300 Read block 2062 len 12288
295 045/300 Flush
296 046/300 Read block 1921 len 28672
297 047/300 Read block 4379 len 36864
298 048/300 Write block 3155 len 32768 data: 2 2 2 2 2 2 2 2
299 049/300 Read block 4327 len 32768
300 050/300 Write block 544 len 36864 data: 2 2 2 2 2 2 2 2 2
301 051/300 Read block 851 len 16384
302 052/300 Read block 3105 len 20480
303 053/300 Flush
304 054/300 Write block 573 len 16384 data: 2 2 2 2
305 055/300 Read block 2427 len 28672
306 056/300 Read block 34 len 40960
307 057/300 Read block 1194 len 8192
308 058/300 Read block 170 len 36864
309 059/300 Read block 4704 len 4096
310 060/300 Read block 2107 len 12288
311 061/300 Write block 482 len 16384 data: 3 2 2 2
312 062/300 Write block 2939 len 20480 data: 2 2 2 2 2
313 063/300 Write block 2475 len 20480 data: 2 2 2 2 2
314 064/300 Write block 3470 len 40960 data: 2 2 2 2 2 2 2 2 2 2
315 065/300 Read block 3474 len 28672
316 066/300 Write block 1728 len 12288 data: 2 2 2
317 067/300 Read block 1570 len 16384
318 068/300 Flush
319 069/300 Read block 3039 len 36864
320 070/300 Read block 1332 len 16384
321 071/300 Read block 4284 len 8192
322 072/300 Write block 2773 len 20480 data: 2 2 2 2 2
323 073/300 Read block 4127 len 28672
324 074/300 Read block 3357 len 4096
325 075/300 Read block 1279 len 32768
326 076/300 Read block 216 len 24576
327 077/300 Write block 2728 len 20480 data: 2 2 2 2 2
328 078/300 Write block 4378 len 16384 data: 2 2 2 2
329 079/300 Read block 4065 len 24576
330 080/300 Read block 846 len 32768
331 081/300 Flush
332 082/300 Read block 413 len 16384
333 083/300 Write block 3039 len 28672 data: 2 2 2 2 2 2 2
334 084/300 Read block 1985 len 4096
335 085/300 Flush
336 086/300 Flush
337 087/300 Read block 1500 len 8192
338 088/300 Write block 4405 len 36864 data: 2 2 2 2 2 2 2 2 2
339 089/300 Read block 1207 len 28672
340 090/300 Read block 1600 len 12288
341 091/300 Read block 4607 len 40960
342 092/300 Read block 1810 len 12288
343 093/300 Write block 543 len 36864 data: 2 3 3 3 3 3 3 3 3
344 094/300 Read block 4389 len 28672
345 095/300 Write block 3654 len 12288 data: 2 2 2
346 096/300 Write block 2333 len 32768 data: 2 2 2 3 3 3 3 3
347 097/300 Read block 732 len 28672
348 098/300 Write block 4574 len 40960 data: 2 2 2 2 2 2 2 2 2 2
349 099/300 Read block 3595 len 28672
350 100/300 Write block 2549 len 16384 data: 2 2 2 2
351 101/300 Write block 3520 len 20480 data: 2 2 2 2 2
352 102/300 Write block 1076 len 40960 data: 2 2 2 2 2 2 2 2 2 2
353 103/300 Write block 1370 len 16384 data: 3 3 3 3
354 104/300 Read block 4607 len 12288
355 105/300 Read block 3839 len 40960
356 106/300 Write block 1484 len 16384 data: 2 2 2 2
357 107/300 Write block 154 len 4096 data: 2
358 108/300 Write block 4259 len 36864 data: 2 2 2 2 2 2 2 2 2
359 109/300 Read block 2692 len 24576
360 110/300 Read block 2102 len 4096
361 111/300 Read block 4165 len 8192
362 112/300 Write block 4356 len 32768 data: 2 2 2 2 2 2 2 2
363 113/300 Read block 3722 len 8192
364 114/300 Read block 510 len 12288
365 115/300 Read block 3345 len 20480
366 116/300 Read block 1423 len 12288
367 117/300 Read block 3994 len 12288
368 118/300 Write block 2335 len 4096 data: 3
369 119/300 Write block 798 len 4096 data: 2
370 120/300 Read block 2055 len 16384
371 121/300 Read block 3057 len 28672
372 122/300 Flush
373 123/300 Read block 295 len 4096
374 124/300 Write block 3580 len 20480 data: 2 2 2 2 2
375 125/300 Read block 477 len 12288
376 126/300 Read block 2078 len 36864
377 127/300 Read block 4910 len 12288
378 128/300 Read block 3762 len 16384
379 129/300 Write block 4991 len 20480 data: 2 2 2 2 2
380 130/300 Read block 4804 len 20480
381 131/300 Read block 4712 len 24576
382 132/300 Read block 1157 len 28672
383 133/300 Read block 4392 len 28672
384 134/300 Read block 1152 len 20480
385 135/300 Read block 897 len 28672
386 136/300 Read block 323 len 20480
387 137/300 Read block 1706 len 20480
388 138/300 Read block 3892 len 4096
389 139/300 Read block 795 len 12288
390 140/300 Flush
391 141/300 Write block 2006 len 24576 data: 2 2 2 2 2 2
392 142/300 Read block 4826 len 36864
393 143/300 Write block 2521 len 24576 data: 2 2 2 2 2 2
394 144/300 Read block 2863 len 40960
395 145/300 Flush
396 146/300 Write block 2917 len 32768 data: 2 2 2 2 2 2 2 2
397 147/300 Write block 754 len 40960 data: 2 2 2 2 2 2 2 2 2 2
398 148/300 Read block 2371 len 8192
399 149/300 Read block 1227 len 12288
400 150/300 Write block 2548 len 4096 data: 2
401 151/300 Read block 1631 len 12288
402 152/300 Write block 1609 len 28672 data: 2 2 2 2 2 2 2
403 153/300 Read block 4511 len 24576
404 154/300 Write block 1709 len 12288 data: 2 2 2
405 155/300 Flush
406 156/300 Write block 4202 len 4096 data: 2
407 157/300 Read block 589 len 4096
408 158/300 Write block 953 len 12288 data: 2 2 2
409 159/300 Read block 1037 len 12288
410 160/300 Read block 1636 len 24576
411 161/300 Write block 2334 len 36864 data: 3 4 4 4 4 4 4 3 3
412 162/300 Write block 857 len 32768 data: 2 2 2 2 2 2 2 2
413 163/300 Read block 1677 len 16384
414 164/300 Write block 3416 len 40960 data: 2 2 2 2 2 2 2 2 2 2
415 165/300 Read block 665 len 16384
416 166/300 Read block 748 len 20480
417 167/300 Write block 4375 len 40960 data: 2 2 2 3 3 3 3 2 2 2
418 168/300 Flush
419 169/300 Write block 1497 len 28672 data: 2 2 2 2 2 2 2
420 170/300 Write block 1270 len 40960 data: 2 2 2 2 2 2 2 2 2 2
421 171/300 Write block 3398 len 8192 data: 4 4
422 172/300 Write block 2810 len 40960 data: 2 2 2 2 2 2 2 2 2 2
423 173/300 Read block 1350 len 40960
424 174/300 Write block 3145 len 12288 data: 2 2 2
425 175/300 Read block 897 len 40960
426 176/300 Write block 4823 len 24576 data: 2 2 2 2 2 2
427 177/300 Write block 1556 len 36864 data: 2 2 2 2 2 2 2 2 2
428 178/300 Read block 2705 len 24576
429 179/300 Write block 860 len 4096 data: 3
430 180/300 Flush
431 181/300 Write block 2640 len 28672 data: 2 2 2 2 2 2 2
432 182/300 Write block 369 len 12288 data: 2 2 2
433 183/300 Write block 3612 len 32768 data: 2 2 2 2 2 2 2 2
434 184/300 Read block 2153 len 36864
435 185/300 Read block 3376 len 12288
436 186/300 Read block 4951 len 24576
437 187/300 Write block 1339 len 40960 data: 2 2 2 2 2 2 2 2 2 2
438 188/300 Read block 3150 len 32768
439 189/300 Write block 4185 len 24576 data: 2 2 2 2 2 2
440 190/300 Write block 513 len 24576 data: 2 2 2 2 2 2
441 191/300 Write block 1376 len 16384 data: 2 2 2 2
442 192/300 Read block 1143 len 16384
443 193/300 Read block 1915 len 20480
444 194/300 Write block 989 len 12288 data: 2 2 2
445 195/300 Flush
446 196/300 Write block 2751 len 28672 data: 2 2 2 2 2 2 2
447 197/300 Read block 2791 len 40960
448 198/300 Read block 3331 len 4096
449 199/300 Read block 4493 len 40960
450 200/300 Read block 300 len 4096
451 201/300 Write block 227 len 20480 data: 2 2 2 2 2
452 202/300 Read block 723 len 8192
453 203/300 Read block 3621 len 36864
454 204/300 Read block 3454 len 20480
455 205/300 Write block 1270 len 32768 data: 3 3 3 3 3 3 3 3
456 206/300 Flush
457 207/300 Write block 4222 len 20480 data: 2 2 2 2 2
458 208/300 Write block 82 len 20480 data: 2 2 2 2 2
459 209/300 Read block 3113 len 8192
460 210/300 Write block 2867 len 28672 data: 2 2 2 2 2 2 2
461 211/300 Read block 1313 len 36864
462 212/300 Write block 3620 len 20480 data: 2 2 2 2 2
463 213/300 Read block 42 len 16384
464 214/300 Read block 970 len 36864
465 215/300 Read block 1625 len 20480
466 216/300 Flush
467 217/300 Read block 3559 len 4096
468 218/300 Write block 1404 len 20480 data: 2 2 2 2 2
469 219/300 Read block 585 len 4096
470 220/300 Read block 1473 len 4096
471 221/300 Write block 3502 len 24576 data: 2 2 2 2 2 2
472 222/300 Flush
473 223/300 Read block 4568 len 36864
474 224/300 Write block 921 len 8192 data: 2 2
475 225/300 Write block 784 len 36864 data: 2 2 2 2 2 2 2 2 2
476 226/300 Read block 164 len 16384
477 227/300 Write block 4627 len 12288 data: 2 2 2
478 228/300 Read block 3146 len 36864
479 229/300 Write block 4343 len 4096 data: 2
480 230/300 Write block 2299 len 4096 data: 2
481 231/300 Read block 3694 len 24576
482 232/300 Flush
483 233/300 Read block 1218 len 8192
484 234/300 Write block 2890 len 40960 data: 2 2 2 2 2 2 2 2 2 2
485 235/300 Read block 749 len 8192
486 236/300 Write block 852 len 20480 data: 2 2 2 2 2
487 237/300 Write block 3735 len 8192 data: 2 2
488 238/300 Write block 188 len 20480 data: 2 2 2 2 2
489 239/300 Read block 342 len 24576
490 240/300 Read block 801 len 40960
491 241/300 Write block 3853 len 28672 data: 2 2 2 2 2 2 2
492 242/300 Write block 861 len 40960 data: 3 3 3 3 2 2 2 2 2 2
493 243/300 Write block 1442 len 24576 data: 2 2 2 2 2 2
494 244/300 Read block 1066 len 32768
495 245/300 Write block 3320 len 12288 data: 2 2 2
496 246/300 Write block 2229 len 40960 data: 2 2 2 2 2 2 2 2 2 2
497 247/300 Read block 1334 len 4096
498 248/300 Write block 2130 len 36864 data: 2 2 2 2 2 2 2 2 2
499 249/300 Write block 520 len 16384 data: 2 2 2 2
500 250/300 Write block 4630 len 28672 data: 2 2 2 2 2 2 2
501 251/300 Flush
502 252/300 Flush
503 253/300 Write block 1532 len 24576 data: 2 2 2 2 2 2
504 254/300 Write block 2381 len 12288 data: 2 2 2
505 255/300 Read block 2868 len 20480
506 256/300 Read block 3466 len 40960
507 257/300 Read block 2842 len 40960
508 258/300 Write block 1126 len 40960 data: 2 2 2 2 2 2 2 2 2 2
509 259/300 Write block 1513 len 8192 data: 2 2
510 260/300 Read block 1906 len 40960
511 261/300 Read block 288 len 36864
512 262/300 Write block 4597 len 24576 data: 2 2 2 2 2 2
513 263/300 Write block 4697 len 36864 data: 2 2 2 2 2 2 2 2 2
514 264/300 Write block 2492 len 16384 data: 2 2 2 2
515 265/300 Read block 552 len 36864
516 266/300 Read block 1780 len 24576
517 267/300 Write block 507 len 36864 data: 2 2 2 2 2 2 3 3 3
518 268/300 Read block 2638 len 20480
519 269/300 Read block 2052 len 20480
520 270/300 Flush
521 271/300 Write block 1846 len 16384 data: 2 2 2 2
522 272/300 Write block 4930 len 4096 data: 2
523 273/300 Read block 1145 len 8192
524 274/300 Read block 298 len 16384
525 275/300 Read block 49 len 32768
526 276/300 Write block 4724 len 8192 data: 2 2
527 277/300 Read block 1911 len 4096
528 278/300 Read block 2740 len 16384
529 279/300 Read block 3431 len 24576
530 280/300 Read block 1779 len 12288
531 281/300 Write block 3012 len 8192 data: 2 2
532 282/300 Write block 1394 len 28672 data: 2 2 2 2 2 2 2
533 283/300 Read block 4193 len 40960
534 284/300 Read block 1445 len 8192
535 285/300 Write block 4001 len 40960 data: 2 2 2 2 2 2 2 2 2 2
536 286/300 Write block 2883 len 8192 data: 2 2
537 287/300 Write block 3970 len 4096 data: 2
538 288/300 Read block 344 len 16384
539 289/300 Flush
540 290/300 Write block 4406 len 32768 data: 3 3 3 3 3 3 3 3
541 291/300 Read block 2073 len 16384
542 292/300 Write block 4032 len 4096 data: 2
543 293/300 Write block 1822 len 8192 data: 2 2
544 294/300 Read block 1804 len 24576
545 295/300 Read block 3878 len 12288
546 296/300 Read block 1797 len 4096
547 297/300 Write block 4158 len 8192 data: 2 2
548 298/300 Read block 1518 len 32768
549 299/300 Write block 3355 len 16384 data: 2 2 2 2
550 300/300 Read block 4936 len 32768
551 301/300 Read block 3653 len 32768
552 Main task has work to do, go find it
553 got dsc Start(2)
554 start 2
555 Dec 09 15:07:59.351 INFO request completed, latency_us: 275, response_code: 204, uri: /start/cid/2, method: POST, req_id: 6d5901e9-f1a4-48d7-b2bf-7d7f72def3d3, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
556 Replay: started 2, returned:Ok(())
557 ----------------------------------------------------------------
558 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
559 States: Active Active Offline
560 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
561 1000 Acked Read 4 Done Done Sent false
562 1001 Acked Write 1 Done Done Sent false
563 1002 Acked Flush 0 Done Done Sent false
564 1003 Acked Read 1 Done Done Sent false
565 1004 Acked Write 10 Done Done Sent false
566 1005 Acked Write 7 Done Done Sent false
567 1006 Acked Flush 0 Done Done Sent false
568 1007 Acked Read 10 Done Done Sent false
569 1008 Acked Write 4 Done Done Sent false
570 1009 Acked Write 1 Done Done Sent false
571 1010 Acked Write 7 Done Done Sent false
572 1011 Acked Write 10 Done Done Sent false
573 1012 Acked Read 1 Done Done Sent false
574 1013 Acked Write 2 Done Done Sent false
575 1014 Acked Read 10 Done Done Sent false
576 1015 Acked Write 2 Done Done Sent false
577 1016 Acked Read 1 Done Done Sent false
578 1017 Acked Read 6 Done Done Sent false
579 1018 Acked Write 8 Done Done Sent false
580 1019 Acked Read 7 Done Done Sent false
581 1020 Acked Write 5 Done Done Sent false
582 1021 Acked Write 10 Done Done Sent false
583 1022 Acked Read 8 Done Done Sent false
584 1023 Acked Read 7 Done Done Sent false
585 1024 Acked Write 6 Done Done Sent false
586 1025 Acked Read 5 Done Done Sent false
587 1026 Acked Write 1 Done Done Sent false
588 1027 Acked Read 8 Done Done Sent false
589 1028 Acked Write 8 Done Done Sent false
590 1029 Acked Write 3 Done Done Sent false
591 1030 Acked Flush 0 Done Done Sent false
592 1031 Acked Read 5 Done Done Sent false
593 1032 Acked Read 5 Done Done Sent false
594 1033 Acked Write 5 Done Done Sent false
595 1034 Acked Read 10 Done Done Sent false
596 1035 Acked Read 3 Done Done Sent false
597 1036 Acked Read 4 Done Done Sent false
598 1037 Acked Write 6 Done Done Sent false
599 1038 Acked Read 3 Done Done Sent false
600 1039 Acked Write 8 Done Done Sent false
601 1040 Acked Write 1 Done Done Sent false
602 1041 Acked Flush 0 Done Done Sent false
603 1042 Acked Read 9 Done Done Sent false
604 1043 Acked Read 3 Done Done Sent false
605 1044 Acked Flush 0 Done Done Sent false
606 1045 Acked Read 7 Done Done Sent false
607 1046 Acked Read 9 Done Done Sent false
608 1047 Acked Write 8 Done Done Sent false
609 1048 Acked Read 8 Done Done Sent false
610 1049 Acked Write 9 Done Done Sent false
611 1050 Acked Read 4 Done Done Sent false
612 1051 Acked Read 5 Done Done Sent false
613 1052 Acked Flush 0 Done Done Sent false
614 1053 Acked Write 4 Done Done Sent false
615 1054 Acked Read 7 Done Done Sent false
616 1055 Acked Read 10 Done Done Sent false
617 1056 Acked Read 2 Done Done Sent false
618 1057 Acked Read 9 Done Done Sent false
619 1058 Acked Read 1 Done Done Sent false
620 1059 Acked Read 3 Done Done Sent false
621 1060 Acked Write 4 Done Done Sent false
622 1061 Acked Write 5 Done Done Sent false
623 1062 Acked Write 5 Done Done Sent false
624 1063 Acked Write 10 Done Done Sent false
625 1064 Acked Read 7 Done Done Sent false
626 1065 Acked Write 3 Done Done Sent false
627 1066 Acked Read 4 Done Done Sent false
628 1067 Acked Flush 0 Done Done Sent false
629 1068 Acked Read 9 Done Done Sent false
630 1069 Acked Read 4 Done Done Sent false
631 1070 Acked Read 2 Done Done Sent false
632 1071 Acked Write 5 Done Done Sent false
633 1072 Acked Read 7 Done Done Sent false
634 1073 Acked Read 1 Done Done Sent false
635 1074 Acked Read 8 Done Done Sent false
636 1075 Acked Read 6 Done Done Sent false
637 1076 Acked Write 5 Done Done Sent false
638 1077 Acked Write 4 Done Done Sent false
639 1078 Acked Read 6 Done Done Sent false
640 1079 Acked Read 8 Done Done Sent false
641 1080 Acked Flush 0 Done Done Sent false
642 1081 Acked Read 4 Done Done Sent false
643 1082 Acked Write 7 Done Done Sent false
644 1083 Acked Read 1 Done Done Sent false
645 1084 Acked Flush 0 Done Done Sent false
646 1085 Acked Flush 0 Done Done Sent false
647 1086 Acked Read 2 Done Done Sent false
648 1087 Acked Write 9 Done Done Sent false
649 1088 Acked Read 7 Done Done Sent false
650 1089 Acked Read 3 Done Done Sent false
651 1090 Acked Read 10 Done Done Sent false
652 1091 Acked Read 3 Done Done Sent false
653 1092 Acked Write 9 Done Done Sent false
654 1093 Acked Read 7 Done Done Sent false
655 1094 Acked Write 3 Done Done Sent false
656 1095 Acked Write 8 Done Done Sent false
657 1096 Acked Read 7 Done Done Sent false
658 1097 Acked Write 10 Done Done Sent false
659 1098 Acked Read 7 Done Done Sent false
660 1099 Acked Write 4 Done Done Sent false
661 1100 Acked Write 5 Done Done Sent false
662 1101 Acked Write 10 Done Done Sent false
663 1102 Acked Write 4 Done Done Sent false
664 1103 Acked Read 3 Done Done Sent false
665 1104 Acked Read 10 Done Done Sent false
666 1105 Acked Write 4 Done Done Sent false
667 1106 Acked Write 1 Done Done Sent false
668 1107 Acked Write 9 Done Done Sent false
669 1108 Acked Read 6 Done Done Sent false
670 1109 Acked Read 1 Done Done Sent false
671 1110 Acked Read 2 Done Done Sent false
672 1111 Acked Write 8 Done Done Sent false
673 1112 Acked Read 2 Done Done Sent false
674 1113 Acked Read 3 Done Done Sent false
675 1114 Acked Read 5 Done Done Sent false
676 1115 Acked Read 3 Done Done Sent false
677 1116 Acked Read 3 Done Done Sent false
678 1117 Acked Write 1 Done Done Sent false
679 1118 Acked Write 1 Done Done Sent false
680 1119 Acked Read 4 Done Done Sent false
681 1120 Acked Read 7 Done Done Sent false
682 1121 Acked Flush 0 Done Done Sent false
683 1122 Acked Read 1 Done Done Sent false
684 1123 Acked Write 5 Done Done Sent false
685 1124 Acked Read 3 Done Done Sent false
686 1125 Acked Read 9 Done Done Sent false
687 1126 Acked Read 3 Done Done Sent false
688 1127 Acked Read 4 Done Done Sent false
689 1128 Acked Write 5 Done Done Sent false
690 1129 Acked Read 5 Done Done Sent false
691 1130 Acked Read 6 Done Done Sent false
692 1131 Acked Read 7 Done Done Sent false
693 1132 Acked Read 7 Done Done Sent false
694 1133 Acked Read 5 Done Done Sent false
695 1134 Acked Read 7 Done Done Sent false
696 1135 Acked Read 5 Done Done Sent false
697 1136 Acked Read 5 Done Done Sent false
698 1137 Acked Read 1 Done Done Sent false
699 1138 Acked Read 3 Done Done Sent false
700 1139 Acked Flush 0 Done Done Sent false
701 1140 Acked Write 6 Done Done Sent false
702 1141 Acked Read 9 Done Done Sent false
703 1142 Acked Write 6 Done Done Sent false
704 1143 Acked Read 10 Done Done Sent false
705 1144 Acked Flush 0 Done Done Sent false
706 1145 Acked Write 8 Done Done Sent false
707 1146 Acked Write 10 Done Done Sent false
708 1147 Acked Read 2 Done Done Sent false
709 1148 Acked Read 3 Done Done Sent false
710 1149 Acked Write 1 Done Done Sent false
711 1150 Acked Read 3 Done Done Sent false
712 1151 Acked Write 7 Done Done Sent false
713 1152 Acked Read 6 Done Done Sent false
714 1153 Acked Write 3 Done Done Sent false
715 1154 Acked Flush 0 Done Done Sent false
716 1155 Acked Write 1 Done Done Sent false
717 1156 Acked Read 1 Done Done Sent false
718 1157 Acked Write 3 Done Done Sent false
719 1158 Acked Read 3 Done Done Sent false
720 1159 Acked Read 6 Done Done Sent false
721 1160 Acked Write 9 Done Done Sent false
722 1161 Acked Write 8 Done Done Sent false
723 1162 Acked Read 4 Done Done Sent false
724 1163 Acked Write 10 Done Done Sent false
725 1164 Acked Read 4 Done Done Sent false
726 1165 Acked Read 5 Done Done Sent false
727 1166 Acked Write 10 Done Done Sent false
728 1167 Acked Flush 0 Done Done Sent false
729 1168 Acked Write 7 Done Done Sent false
730 1169 Acked Write 10 Done Done Sent false
731 1170 Acked Write 2 Done Done Sent false
732 1171 Acked Write 10 Done Done Sent false
733 1172 Acked Read 10 Done Done Sent false
734 1173 Acked Write 3 Done Done Sent false
735 1174 Acked Read 10 Done Done Sent false
736 1175 Acked Write 6 Done Done Sent false
737 1176 Acked Write 9 Done Done Sent false
738 1177 Acked Read 6 Done Done Sent false
739 1178 Acked Write 1 Done Done Sent false
740 1179 Acked Flush 0 Done Done Sent false
741 1180 Acked Write 7 Done Done Sent false
742 1181 Acked Write 3 Done Done Sent false
743 1182 Acked Write 8 Done Done Sent false
744 1183 Acked Read 9 Done Done Sent false
745 1184 Acked Read 3 Done Done Sent false
746 1185 Acked Read 6 Done Done Sent false
747 1186 Acked Write 10 Done Done Sent false
748 1187 Acked Read 8 Done Done Sent false
749 1188 Acked Write 6 Done Done Sent false
750 1189 Acked Write 6 Done Done Sent false
751 1190 Acked Write 4 Done Done Sent false
752 1191 Acked Read 4 Done Done Sent false
753 1192 Acked Read 5 Done Done Sent false
754 1193 Acked Write 3 Done Done Sent false
755 1194 Acked Flush 0 Done Done Sent false
756 1195 Acked Write 7 Done Done Sent false
757 1196 Acked Read 10 Done Done Sent false
758 1197 Acked Read 1 Done Done Sent false
759 1198 Acked Read 10 Done Done Sent false
760 1199 Acked Read 1 Done Done Sent false
761 1200 Acked Write 5 Done Done Sent false
762 1201 Acked Read 2 Done Done Sent false
763 1202 Acked Read 9 Done Done Sent false
764 1203 Acked Read 5 Done Done Sent false
765 1204 Acked Write 8 Done Done Sent false
766 1205 Acked Flush 0 Done Done Sent false
767 1206 Acked Write 5 Done Done Sent false
768 1207 Acked Write 5 Done Done Sent false
769 1208 Acked Read 2 Done Done Sent false
770 1209 Acked Write 7 Done Done Sent false
771 1210 Acked Read 9 Done Done Sent false
772 1211 Acked Write 5 Done Done Sent false
773 1212 Acked Read 4 Done Done Sent false
774 1213 Acked Read 9 Done Done Sent false
775 1214 Acked Read 5 Done Done Sent false
776 1215 Acked Flush 0 Done Done Sent false
777 1216 Acked Read 1 Done Done Sent false
778 1217 Acked Write 5 Done Done Sent false
779 1218 Acked Read 1 Done Done Sent false
780 1219 Acked Read 1 Done Done Sent false
781 1220 Acked Write 6 Done Done Sent false
782 1221 Acked Flush 0 Done Done Sent false
783 1222 Acked Read 9 Done Done Sent false
784 1223 Acked Write 2 Done Done Sent false
785 1224 Acked Write 9 Done Done Sent false
786 1225 Acked Read 4 Done Done Sent false
787 1226 Acked Write 3 Done Done Sent false
788 1227 Acked Read 9 Done Done Sent false
789 1228 Acked Write 1 Done Done Sent false
790 1229 Acked Write 1 Done Done Sent false
791 1230 Acked Read 6 Done Done Sent false
792 1231 Acked Flush 0 Done Done Sent false
793 1232 Acked Read 2 Done Done Sent false
794 1233 Acked Write 10 Done Done Sent false
795 1234 Acked Read 2 Done Done Sent false
796 1235 Acked Write 5 Done Done Sent false
797 1236 Acked Write 2 Done Done Sent false
798 1237 Acked Write 5 Done Done Sent false
799 1238 Acked Read 6 Done Done Sent false
800 1239 Acked Read 10 Done Done Sent false
801 1240 Acked Write 7 Done Done Sent false
802 1241 Acked Write 10 Done Done Sent false
803 1242 Acked Write 6 Done Done Sent false
804 1243 Acked Read 8 Done Done Sent false
805 1244 Acked Write 3 Done Done Sent false
806 1245 Acked Write 10 Done Done Sent false
807 1246 Acked Read 1 Done Done Sent false
808 1247 Acked Write 9 Done Done Sent false
809 1248 Acked Write 4 Done Done Sent false
810 1249 Acked Write 7 Done Done Sent false
811 1250 Acked Flush 0 Done Done Sent false
812 1251 Acked Flush 0 Done Done Sent false
813 1252 Acked Write 6 Done Done Sent false
814 1253 Acked Write 3 Done Done Sent false
815 1254 Acked Read 5 Done Done Sent false
816 1255 Acked Read 10 Done Done Sent false
817 1256 Acked Read 10 Done Done Sent false
818 1257 Acked Write 10 Done Done Sent false
819 1258 Acked Write 2 Done Done Sent false
820 1259 Acked Read 10 Done Done Sent false
821 1260 Acked Read 9 Done Done Sent false
822 1261 Acked Write 6 Done Done Sent false
823 1262 Acked Write 9 Done Done Sent false
824 1263 Acked Write 4 Done Done Sent false
825 1264 Acked Read 9 Done Done Sent false
826 1265 Acked Read 6 Done Done Sent false
827 1266 Acked Write 9 Done Done Sent false
828 1267 Acked Read 5 Done Done Sent false
829 1268 Acked Read 5 Done Done Sent false
830 1269 Acked Flush 0 Done Done Sent false
831 1270 Acked Write 4 Done Done Sent false
832 1271 Acked Write 1 Done Done Sent false
833 1272 Acked Read 2 Done Done Sent false
834 1273 Acked Read 4 Done Done Sent false
835 1274 Acked Read 8 Done Done Sent false
836 1275 Acked Write 2 Done Done Sent false
837 1276 Acked Read 1 Done Done Sent false
838 1277 Acked Read 4 Done Done Sent false
839 1278 Acked Read 6 Done Done Sent false
840 1279 Acked Read 3 Done Done Sent false
841 1280 Acked Write 2 Done Done Sent false
842 1281 Acked Write 7 Done Done Sent false
843 1282 Acked Read 10 Done Done Sent false
844 1283 Acked Read 2 Done Done Sent false
845 1284 Acked Write 10 Done Done Sent false
846 1285 Acked Write 2 Done Done Sent false
847 1286 Acked Write 1 Done Done Sent false
848 1287 Acked Read 4 Done Done Sent false
849 1288 Acked Flush 0 Done Done Sent false
850 1289 Acked Write 8 Done Done Sent false
851 1290 Acked Read 4 Done Done Sent false
852 1291 Acked Write 1 Done Done Sent false
853 1292 Acked Write 2 Done Done Sent false
854 1293 Acked Read 6 Done Done Sent false
855 1294 Acked Read 3 Done Done Sent false
856 1295 Acked Read 1 Done Done Sent false
857 1296 Acked Write 2 Done Done Sent false
858 1297 Acked Read 8 Done Done Sent false
859 1298 Acked Write 4 Done Done Sent false
860 1299 Acked Read 8 Done Done Sent false
861 1300 Acked Read 8 Done Done Sent false
862 STATES DS:0 DS:1 DS:2 TOTAL
863 Sent 0 0 301 301
864 Done 301 301 0 602
865 Skipped 0 0 0 0
866 Error 0 0 0 0
867 Last Flush: 1288 1288 0
868 Downstairs last five completed:
869 Upstairs last five completed: 1300 1299 1298 1297 1296
870 CLIENT: Up:0 ds:301 act:2
871 [8830] got start action
872 [8830] I am going to restart
873 Starting downstairs at port 8830
874 Make output file at "/tmp/dsc/downstairs-8830.txt"
875 [8830][2] reports Starting
876 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1202)
877 [8830][2] reports Running
878 ----------------------------------------------------------------
879 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
880 States: Active Active Offline
881 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
882 1000 Acked Read 4 Done Done Sent false
883 1001 Acked Write 1 Done Done Sent false
884 1002 Acked Flush 0 Done Done Sent false
885 1003 Acked Read 1 Done Done Sent false
886 1004 Acked Write 10 Done Done Sent false
887 1005 Acked Write 7 Done Done Sent false
888 1006 Acked Flush 0 Done Done Sent false
889 1007 Acked Read 10 Done Done Sent false
890 1008 Acked Write 4 Done Done Sent false
891 1009 Acked Write 1 Done Done Sent false
892 1010 Acked Write 7 Done Done Sent false
893 1011 Acked Write 10 Done Done Sent false
894 1012 Acked Read 1 Done Done Sent false
895 1013 Acked Write 2 Done Done Sent false
896 1014 Acked Read 10 Done Done Sent false
897 1015 Acked Write 2 Done Done Sent false
898 1016 Acked Read 1 Done Done Sent false
899 1017 Acked Read 6 Done Done Sent false
900 1018 Acked Write 8 Done Done Sent false
901 1019 Acked Read 7 Done Done Sent false
902 1020 Acked Write 5 Done Done Sent false
903 1021 Acked Write 10 Done Done Sent false
904 1022 Acked Read 8 Done Done Sent false
905 1023 Acked Read 7 Done Done Sent false
906 1024 Acked Write 6 Done Done Sent false
907 1025 Acked Read 5 Done Done Sent false
908 1026 Acked Write 1 Done Done Sent false
909 1027 Acked Read 8 Done Done Sent false
910 1028 Acked Write 8 Done Done Sent false
911 1029 Acked Write 3 Done Done Sent false
912 1030 Acked Flush 0 Done Done Sent false
913 1031 Acked Read 5 Done Done Sent false
914 1032 Acked Read 5 Done Done Sent false
915 1033 Acked Write 5 Done Done Sent false
916 1034 Acked Read 10 Done Done Sent false
917 1035 Acked Read 3 Done Done Sent false
918 1036 Acked Read 4 Done Done Sent false
919 1037 Acked Write 6 Done Done Sent false
920 1038 Acked Read 3 Done Done Sent false
921 1039 Acked Write 8 Done Done Sent false
922 1040 Acked Write 1 Done Done Sent false
923 1041 Acked Flush 0 Done Done Sent false
924 1042 Acked Read 9 Done Done Sent false
925 1043 Acked Read 3 Done Done Sent false
926 1044 Acked Flush 0 Done Done Sent false
927 1045 Acked Read 7 Done Done Sent false
928 1046 Acked Read 9 Done Done Sent false
929 1047 Acked Write 8 Done Done Sent false
930 1048 Acked Read 8 Done Done Sent false
931 1049 Acked Write 9 Done Done Sent false
932 1050 Acked Read 4 Done Done Sent false
933 1051 Acked Read 5 Done Done Sent false
934 1052 Acked Flush 0 Done Done Sent false
935 1053 Acked Write 4 Done Done Sent false
936 1054 Acked Read 7 Done Done Sent false
937 1055 Acked Read 10 Done Done Sent false
938 1056 Acked Read 2 Done Done Sent false
939 1057 Acked Read 9 Done Done Sent false
940 1058 Acked Read 1 Done Done Sent false
941 1059 Acked Read 3 Done Done Sent false
942 1060 Acked Write 4 Done Done Sent false
943 1061 Acked Write 5 Done Done Sent false
944 1062 Acked Write 5 Done Done Sent false
945 1063 Acked Write 10 Done Done Sent false
946 1064 Acked Read 7 Done Done Sent false
947 1065 Acked Write 3 Done Done Sent false
948 1066 Acked Read 4 Done Done Sent false
949 1067 Acked Flush 0 Done Done Sent false
950 1068 Acked Read 9 Done Done Sent false
951 1069 Acked Read 4 Done Done Sent false
952 1070 Acked Read 2 Done Done Sent false
953 1071 Acked Write 5 Done Done Sent false
954 1072 Acked Read 7 Done Done Sent false
955 1073 Acked Read 1 Done Done Sent false
956 1074 Acked Read 8 Done Done Sent false
957 1075 Acked Read 6 Done Done Sent false
958 1076 Acked Write 5 Done Done Sent false
959 1077 Acked Write 4 Done Done Sent false
960 1078 Acked Read 6 Done Done Sent false
961 1079 Acked Read 8 Done Done Sent false
962 1080 Acked Flush 0 Done Done Sent false
963 1081 Acked Read 4 Done Done Sent false
964 1082 Acked Write 7 Done Done Sent false
965 1083 Acked Read 1 Done Done Sent false
966 1084 Acked Flush 0 Done Done Sent false
967 1085 Acked Flush 0 Done Done Sent false
968 1086 Acked Read 2 Done Done Sent false
969 1087 Acked Write 9 Done Done Sent false
970 1088 Acked Read 7 Done Done Sent false
971 1089 Acked Read 3 Done Done Sent false
972 1090 Acked Read 10 Done Done Sent false
973 1091 Acked Read 3 Done Done Sent false
974 1092 Acked Write 9 Done Done Sent false
975 1093 Acked Read 7 Done Done Sent false
976 1094 Acked Write 3 Done Done Sent false
977 1095 Acked Write 8 Done Done Sent false
978 1096 Acked Read 7 Done Done Sent false
979 1097 Acked Write 10 Done Done Sent false
980 1098 Acked Read 7 Done Done Sent false
981 1099 Acked Write 4 Done Done Sent false
982 1100 Acked Write 5 Done Done Sent false
983 1101 Acked Write 10 Done Done Sent false
984 1102 Acked Write 4 Done Done Sent false
985 1103 Acked Read 3 Done Done Sent false
986 1104 Acked Read 10 Done Done Sent false
987 1105 Acked Write 4 Done Done Sent false
988 1106 Acked Write 1 Done Done Sent false
989 1107 Acked Write 9 Done Done Sent false
990 1108 Acked Read 6 Done Done Sent false
991 1109 Acked Read 1 Done Done Sent false
992 1110 Acked Read 2 Done Done Sent false
993 1111 Acked Write 8 Done Done Sent false
994 1112 Acked Read 2 Done Done Sent false
995 1113 Acked Read 3 Done Done Sent false
996 1114 Acked Read 5 Done Done Sent false
997 1115 Acked Read 3 Done Done Sent false
998 1116 Acked Read 3 Done Done Sent false
999 1117 Acked Write 1 Done Done Sent false
1000 1118 Acked Write 1 Done Done Sent false
1001 1119 Acked Read 4 Done Done Sent false
1002 1120 Acked Read 7 Done Done Sent false
1003 1121 Acked Flush 0 Done Done Sent false
1004 1122 Acked Read 1 Done Done Sent false
1005 1123 Acked Write 5 Done Done Sent false
1006 1124 Acked Read 3 Done Done Sent false
1007 1125 Acked Read 9 Done Done Sent false
1008 1126 Acked Read 3 Done Done Sent false
1009 1127 Acked Read 4 Done Done Sent false
1010 1128 Acked Write 5 Done Done Sent false
1011 1129 Acked Read 5 Done Done Sent false
1012 1130 Acked Read 6 Done Done Sent false
1013 1131 Acked Read 7 Done Done Sent false
1014 1132 Acked Read 7 Done Done Sent false
1015 1133 Acked Read 5 Done Done Sent false
1016 1134 Acked Read 7 Done Done Sent false
1017 1135 Acked Read 5 Done Done Sent false
1018 1136 Acked Read 5 Done Done Sent false
1019 1137 Acked Read 1 Done Done Sent false
1020 1138 Acked Read 3 Done Done Sent false
1021 1139 Acked Flush 0 Done Done Sent false
1022 1140 Acked Write 6 Done Done Sent false
1023 1141 Acked Read 9 Done Done Sent false
1024 1142 Acked Write 6 Done Done Sent false
1025 1143 Acked Read 10 Done Done Sent false
1026 1144 Acked Flush 0 Done Done Sent false
1027 1145 Acked Write 8 Done Done Sent false
1028 1146 Acked Write 10 Done Done Sent false
1029 1147 Acked Read 2 Done Done Sent false
1030 1148 Acked Read 3 Done Done Sent false
1031 1149 Acked Write 1 Done Done Sent false
1032 1150 Acked Read 3 Done Done Sent false
1033 1151 Acked Write 7 Done Done Sent false
1034 1152 Acked Read 6 Done Done Sent false
1035 1153 Acked Write 3 Done Done Sent false
1036 1154 Acked Flush 0 Done Done Sent false
1037 1155 Acked Write 1 Done Done Sent false
1038 1156 Acked Read 1 Done Done Sent false
1039 1157 Acked Write 3 Done Done Sent false
1040 1158 Acked Read 3 Done Done Sent false
1041 1159 Acked Read 6 Done Done Sent false
1042 1160 Acked Write 9 Done Done Sent false
1043 1161 Acked Write 8 Done Done Sent false
1044 1162 Acked Read 4 Done Done Sent false
1045 1163 Acked Write 10 Done Done Sent false
1046 1164 Acked Read 4 Done Done Sent false
1047 1165 Acked Read 5 Done Done Sent false
1048 1166 Acked Write 10 Done Done Sent false
1049 1167 Acked Flush 0 Done Done Sent false
1050 1168 Acked Write 7 Done Done Sent false
1051 1169 Acked Write 10 Done Done Sent false
1052 1170 Acked Write 2 Done Done Sent false
1053 1171 Acked Write 10 Done Done Sent false
1054 1172 Acked Read 10 Done Done Sent false
1055 1173 Acked Write 3 Done Done Sent false
1056 1174 Acked Read 10 Done Done Sent false
1057 1175 Acked Write 6 Done Done Sent false
1058 1176 Acked Write 9 Done Done Sent false
1059 1177 Acked Read 6 Done Done Sent false
1060 1178 Acked Write 1 Done Done Sent false
1061 1179 Acked Flush 0 Done Done Sent false
1062 1180 Acked Write 7 Done Done Sent false
1063 1181 Acked Write 3 Done Done Sent false
1064 1182 Acked Write 8 Done Done Sent false
1065 1183 Acked Read 9 Done Done Sent false
1066 1184 Acked Read 3 Done Done Sent false
1067 1185 Acked Read 6 Done Done Sent false
1068 1186 Acked Write 10 Done Done Sent false
1069 1187 Acked Read 8 Done Done Sent false
1070 1188 Acked Write 6 Done Done Sent false
1071 1189 Acked Write 6 Done Done Sent false
1072 1190 Acked Write 4 Done Done Sent false
1073 1191 Acked Read 4 Done Done Sent false
1074 1192 Acked Read 5 Done Done Sent false
1075 1193 Acked Write 3 Done Done Sent false
1076 1194 Acked Flush 0 Done Done Sent false
1077 1195 Acked Write 7 Done Done Sent false
1078 1196 Acked Read 10 Done Done Sent false
1079 1197 Acked Read 1 Done Done Sent false
1080 1198 Acked Read 10 Done Done Sent false
1081 1199 Acked Read 1 Done Done Sent false
1082 1200 Acked Write 5 Done Done Sent false
1083 1201 Acked Read 2 Done Done Sent false
1084 1202 Acked Read 9 Done Done Sent false
1085 1203 Acked Read 5 Done Done Sent false
1086 1204 Acked Write 8 Done Done Sent false
1087 1205 Acked Flush 0 Done Done Sent false
1088 1206 Acked Write 5 Done Done Sent false
1089 1207 Acked Write 5 Done Done Sent false
1090 1208 Acked Read 2 Done Done Sent false
1091 1209 Acked Write 7 Done Done Sent false
1092 1210 Acked Read 9 Done Done Sent false
1093 1211 Acked Write 5 Done Done Sent false
1094 1212 Acked Read 4 Done Done Sent false
1095 1213 Acked Read 9 Done Done Sent false
1096 1214 Acked Read 5 Done Done Sent false
1097 1215 Acked Flush 0 Done Done Sent false
1098 1216 Acked Read 1 Done Done Sent false
1099 1217 Acked Write 5 Done Done Sent false
1100 1218 Acked Read 1 Done Done Sent false
1101 1219 Acked Read 1 Done Done Sent false
1102 1220 Acked Write 6 Done Done Sent false
1103 1221 Acked Flush 0 Done Done Sent false
1104 1222 Acked Read 9 Done Done Sent false
1105 1223 Acked Write 2 Done Done Sent false
1106 1224 Acked Write 9 Done Done Sent false
1107 1225 Acked Read 4 Done Done Sent false
1108 1226 Acked Write 3 Done Done Sent false
1109 1227 Acked Read 9 Done Done Sent false
1110 1228 Acked Write 1 Done Done Sent false
1111 1229 Acked Write 1 Done Done Sent false
1112 1230 Acked Read 6 Done Done Sent false
1113 1231 Acked Flush 0 Done Done Sent false
1114 1232 Acked Read 2 Done Done Sent false
1115 1233 Acked Write 10 Done Done Sent false
1116 1234 Acked Read 2 Done Done Sent false
1117 1235 Acked Write 5 Done Done Sent false
1118 1236 Acked Write 2 Done Done Sent false
1119 1237 Acked Write 5 Done Done Sent false
1120 1238 Acked Read 6 Done Done Sent false
1121 1239 Acked Read 10 Done Done Sent false
1122 1240 Acked Write 7 Done Done Sent false
1123 1241 Acked Write 10 Done Done Sent false
1124 1242 Acked Write 6 Done Done Sent false
1125 1243 Acked Read 8 Done Done Sent false
1126 1244 Acked Write 3 Done Done Sent false
1127 1245 Acked Write 10 Done Done Sent false
1128 1246 Acked Read 1 Done Done Sent false
1129 1247 Acked Write 9 Done Done Sent false
1130 1248 Acked Write 4 Done Done Sent false
1131 1249 Acked Write 7 Done Done Sent false
1132 1250 Acked Flush 0 Done Done Sent false
1133 1251 Acked Flush 0 Done Done Sent false
1134 1252 Acked Write 6 Done Done Sent false
1135 1253 Acked Write 3 Done Done Sent false
1136 1254 Acked Read 5 Done Done Sent false
1137 1255 Acked Read 10 Done Done Sent false
1138 1256 Acked Read 10 Done Done Sent false
1139 1257 Acked Write 10 Done Done Sent false
1140 1258 Acked Write 2 Done Done Sent false
1141 1259 Acked Read 10 Done Done Sent false
1142 1260 Acked Read 9 Done Done Sent false
1143 1261 Acked Write 6 Done Done Sent false
1144 1262 Acked Write 9 Done Done Sent false
1145 1263 Acked Write 4 Done Done Sent false
1146 1264 Acked Read 9 Done Done Sent false
1147 1265 Acked Read 6 Done Done Sent false
1148 1266 Acked Write 9 Done Done Sent false
1149 1267 Acked Read 5 Done Done Sent false
1150 1268 Acked Read 5 Done Done Sent false
1151 1269 Acked Flush 0 Done Done Sent false
1152 1270 Acked Write 4 Done Done Sent false
1153 1271 Acked Write 1 Done Done Sent false
1154 1272 Acked Read 2 Done Done Sent false
1155 1273 Acked Read 4 Done Done Sent false
1156 1274 Acked Read 8 Done Done Sent false
1157 1275 Acked Write 2 Done Done Sent false
1158 1276 Acked Read 1 Done Done Sent false
1159 1277 Acked Read 4 Done Done Sent false
1160 1278 Acked Read 6 Done Done Sent false
1161 1279 Acked Read 3 Done Done Sent false
1162 1280 Acked Write 2 Done Done Sent false
1163 1281 Acked Write 7 Done Done Sent false
1164 1282 Acked Read 10 Done Done Sent false
1165 1283 Acked Read 2 Done Done Sent false
1166 1284 Acked Write 10 Done Done Sent false
1167 1285 Acked Write 2 Done Done Sent false
1168 1286 Acked Write 1 Done Done Sent false
1169 1287 Acked Read 4 Done Done Sent false
1170 1288 Acked Flush 0 Done Done Sent false
1171 1289 Acked Write 8 Done Done Sent false
1172 1290 Acked Read 4 Done Done Sent false
1173 1291 Acked Write 1 Done Done Sent false
1174 1292 Acked Write 2 Done Done Sent false
1175 1293 Acked Read 6 Done Done Sent false
1176 1294 Acked Read 3 Done Done Sent false
1177 1295 Acked Read 1 Done Done Sent false
1178 1296 Acked Write 2 Done Done Sent false
1179 1297 Acked Read 8 Done Done Sent false
1180 1298 Acked Write 4 Done Done Sent false
1181 1299 Acked Read 8 Done Done Sent false
1182 1300 Acked Read 8 Done Done Sent false
1183 1301 Acked Flush 0 Done Done Sent false
1184 STATES DS:0 DS:1 DS:2 TOTAL
1185 Sent 0 0 302 302
1186 Done 302 302 0 604
1187 Skipped 0 0 0 0
1188 Error 0 0 0 0
1189 Last Flush: 1301 1301 0
1190 Downstairs last five completed:
1191 Upstairs last five completed: 1301 1300 1299 1298 1297
1192 CLIENT: Up:0 ds:302 act:2
11932024-12-09T15:08:05.110ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
11942024-12-09T15:08:05.110ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51311) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
11952024-12-09T15:08:05.111ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
11962024-12-09T15:08:05.111ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
11972024-12-09T15:08:05.111ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
11982024-12-09T15:08:05.111ZINFOcrucible: send last flush ID to this DS: 0 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
11992024-12-09T15:08:05.112ZINFOcrucible: Replied this last flush ID: 0 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
12002024-12-09T15:08:05.112ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
12012024-12-09T15:08:05.112ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
12022024-12-09T15:08:05.112ZINFOcrucible: [2] client re-new 302 jobs since flush 0 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
12032024-12-09T15:08:05.112ZINFOcrucible: [2] Replayed 302 jobs since flush: 0 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1204 ----------------------------------------------------------------
1205 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1206 Downstairs last five completed: 1301 1300 1299 1298 1297
1207 Upstairs last five completed: 1301 1300 1299 1298 1297
1208 CLIENT: Up:0 ds:0 act:3
1209 Replay: All jobs finished
1210 Main task has work to do, go find it
1211 got dsc Stop(1)
1212 stop 1
1213 Dec 09 15:08:07.356 INFO request completed, latency_us: 358, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 17170be9-d49a-4719-b102-6f813cb0cc2a, remote_addr: 127.0.0.1:47510, local_addr: 127.[8820] Got stop action so:false kr:false
1214 0.0.1:9998
1215 Dec 09 15:08:07.357 INFO request completed, latency_us: 184, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4af8b485-d7f6-454a-9aef-136e89639b50, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
12162024-12-09T15:08:07.357ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
12172024-12-09T15:08:07.357ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
12182024-12-09T15:08:07.357ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
12192024-12-09T15:08:07.357ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
12202024-12-09T15:08:07.357ZINFOcrucible: [1] Transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
12212024-12-09T15:08:07.357ZINFOcrucible: client task is exiting = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
1222 [8820] Exited with: ExitStatus(unix_wait_status(9))
1223 [8820][1] reports Exit
1224 Dec 09 15:08:11.359 INFO request completed, latency_us: 231, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6c007036-dcbb-4cb4-b42e-cea3c25df2d8, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
1225 001/300 Read block 998 len 40960
1226 002/300 Write block 2094 len 28672 data: 2 2 2 2 2 2 2
1227 003/300 Write block 3762 len 32768 data: 2 2 2 2 2 2 2 2
1228 004/300 Flush
1229 005/300 Write block 4593 len 36864 data: 2 2 2 2 3 3 3 3 3
1230 006/300 Read block 1555 len 28672
1231 007/300 Read block 975 len 40960
1232 008/300 Write block 4425 len 28672 data: 2 2 2 2 2 2 2
1233 009/300 Write block 2441 len 32768 data: 2 2 2 2 2 2 2 2
1234 010/300 Read block 2143 len 16384
1235 011/300 Read block 1123 len 28672
1236 012/300 Write block 3301 len 36864 data: 2 2 2 2 2 2 2 2 2
1237 013/300 Read block 1764 len 28672
1238 014/300 Read block 2702 len 8192
1239 015/300 Write block 4274 len 8192 data: 2 2
1240 016/300 Write block 2484 len 12288 data: 2 2 2
1241 017/300 Write block 404 len 12288 data: 2 2 2
1242 018/300 Read block 816 len 4096
1243 019/300 Write block 3525 len 4096 data: 2
1244 020/300 Read block 1960 len 16384
1245 021/300 Write block 1428 len 8192 data: 2 2
1246 022/300 Read block 3415 len 28672
1247 023/300 Write block 2980 len 36864 data: 2 2 2 2 2 2 2 2 2
1248 024/300 Read block 4515 len 8192
1249 025/300 Write block 3583 len 24576 data: 3 3 2 2 2 2
1250 026/300 Write block 2555 len 20480 data: 2 2 2 2 2
1251 027/300 Read block 2104 len 12288
1252 028/300 Read block 1860 len 36864
1253 029/300 Read block 1120 len 28672
1254 030/300 Flush
1255 031/300 Write block 156 len 16384 data: 2 2 2 2
1256 032/300 Read block 1032 len 20480
1257 033/300 Read block 3751 len 20480
1258 034/300 Write block 2548 len 32768 data: 3 3 3 3 3 2 2 3
1259 035/300 Read block 2816 len 4096
1260 036/300 Write block 4834 len 32768 data: 2 2 2 2 2 2 2 2
1261 037/300 Write block 4759 len 28672 data: 2 2 2 2 2 2 2
1262 038/300 Write block 3346 len 28672 data: 2 2 2 2 2 2 2
1263 039/300 Write block 206 len 32768 data: 2 2 2 2 2 2 2 2
1264 040/300 Flush
1265 041/300 Read block 4066 len 40960
1266 042/300 Write block 661 len 32768 data: 2 2 2 2 2 2 2 2
1267 043/300 Read block 837 len 8192
1268 044/300 Read block 176 len 36864
1269 045/300 Write block 956 len 24576 data: 2 2 2 2 2 2
1270 046/300 Read block 3807 len 36864
1271 047/300 Write block 4083 len 32768 data: 2 2 2 2 2 2 2 2
1272 048/300 Write block 245 len 36864 data: 2 2 2 2 2 2 2 2 2
1273 049/300 Read block 1510 len 32768
1274 050/300 Read block 612 len 12288
1275 051/300 Write block 2101 len 36864 data: 2 2 2 2 2 2 2 2 2
1276 052/300 Write block 1917 len 28672 data: 2 2 2 2 2 2 2
1277 053/300 Flush
1278 054/300 Read block 3512 len 8192
1279 055/300 Write block 1786 len 16384 data: 2 2 2 2
1280 056/300 Write block 2961 len 16384 data: 2 2 2 2
1281 057/300 Read block 4448 len 28672
1282 058/300 Write block 460 len 32768 data: 2 2 2 2 2 2 2 2
1283 059/300 Write block 4252 len 32768 data: 2 2 2 2 2 2 2 3
1284 060/300 Read block 4672 len 8192
1285 061/300 Write block 3150 len 24576 data: 2 2 2 2 2 3
1286 062/300 Write block 960 len 40960 data: 3 3 2 2 2 2 2 2 2 2
1287 063/300 Read block 2368 len 32768
1288 064/300 Read block 2386 len 12288
1289 065/300 Write block 2456 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1290 066/300 Read block 395 len 12288
1291 067/300 Write block 2188 len 28672 data: 2 2 2 2 2 2 2
1292 068/300 Read block 588 len 16384
1293 069/300 Read block 2254 len 40960
1294 070/300 Write block 2795 len 28672 data: 2 2 2 2 2 2 2
1295 071/300 Write block 3341 len 32768 data: 2 2 2 2 2 3 3 3
1296 072/300 Read block 1808 len 20480
1297 073/300 Write block 2141 len 4096 data: 2
1298 074/300 Write block 2388 len 12288 data: 2 2 2
1299 075/300 Write block 1211 len 16384 data: 2 2 2 2
1300 076/300 Write block 538 len 28672 data: 2 2 2 2 2 3 4
1301 077/300 Write block 4861 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1302 078/300 Write block 825 len 16384 data: 2 2 2 2
1303 079/300 Write block 360 len 24576 data: 2 2 2 2 2 2
1304 080/300 Read block 824 len 20480
1305 081/300 Flush
1306 082/300 Read block 3388 len 4096
1307 083/300 Write block 408 len 12288 data: 2 2 2
1308 084/300 Write block 3985 len 8192 data: 2 2
1309 085/300 Read block 3162 len 36864
1310 086/300 Read block 719 len 36864
1311 087/300 Read block 1638 len 8192
1312 088/300 Write block 4443 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1313 089/300 Read block 4506 len 12288
1314 090/300 Write block 4811 len 16384 data: 2 2 2 2
1315 091/300 Write block 4759 len 16384 data: 3 3 3 3
1316 092/300 Flush
1317 093/300 Write block 1311 len 16384 data: 2 2 2 2
1318 094/300 Read block 4375 len 4096
1319 095/300 Read block 528 len 40960
1320 096/300 Write block 3009 len 40960 data: 2 2 2 3 3 2 2 2 2 2
1321 097/300 Read block 3213 len 20480
1322 098/300 Read block 2599 len 16384
1323 099/300 Read block 1827 len 36864
1324 100/300 Read block 3605 len 12288
1325 101/300 Write block 3555 len 28672 data: 2 2 2 2 2 2 2
1326 102/300 Write block 100 len 36864 data: 2 2 2 2 2 2 2 2 2
1327 103/300 Read block 4820 len 28672
1328 104/300 Write block 3246 len 36864 data: 2 2 2 2 2 2 2 2 2
1329 105/300 Write block 1743 len 8192 data: 2 2
1330 106/300 Read block 4517 len 4096
1331 107/300 Write block 4090 len 40960 data: 3 2 2 2 2 2 2 2 2 2
1332 108/300 Flush
1333 109/300 Write block 4029 len 40960 data: 2 2 2 3 2 2 2 2 2 2
1334 110/300 Write block 1840 len 8192 data: 2 2
1335 111/300 Read block 1468 len 40960
1336 112/300 Read block 1269 len 20480
1337 113/300 Read block 1549 len 32768
1338 114/300 Write block 1966 len 4096 data: 2
1339 115/300 Read block 72 len 12288
1340 116/300 Read block 3379 len 16384
1341 117/300 Write block 1145 len 4096 data: 2
1342 118/300 Write block 326 len 24576 data: 2 2 2 2 2 2
1343 119/300 Write block 2324 len 12288 data: 2 2 2
1344 120/300 Read block 2100 len 12288
1345 121/300 Read block 806 len 20480
1346 122/300 Read block 2411 len 28672
1347 123/300 Read block 548 len 12288
1348 124/300 Read block 3144 len 8192
1349 125/300 Write block 2797 len 12288 data: 3 3 3
1350 126/300 Write block 3463 len 36864 data: 2 2 2 2 2 2 2 3 3
1351 127/300 Flush
1352 128/300 Read block 3594 len 24576
1353 129/300 Write block 939 len 24576 data: 2 2 2 2 2 2
1354 130/300 Read block 4065 len 32768
1355 131/300 Write block 4840 len 20480 data: 3 3 2 2 2
1356 132/300 Read block 2124 len 12288
1357 133/300 Flush
1358 134/300 Write block 827 len 4096 data: 3
1359 135/300 Read block 1971 len 12288
1360 136/300 Read block 4048 len 4096
1361 137/300 Read block 2123 len 16384
1362 138/300 Flush
1363 139/300 Write block 3843 len 16384 data: 2 2 2 3
1364 140/300 Write block 1909 len 24576 data: 2 2 2 2 2 2
1365 141/300 Read block 888 len 4096
1366 142/300 Read block 3201 len 32768
1367 143/300 Read block 136 len 8192
1368 144/300 Read block 2328 len 28672
1369 145/300 Write block 3927 len 24576 data: 2 2 2 2 2 2
1370 146/300 Read block 2590 len 36864
1371 147/300 Read block 2831 len 4096
1372 148/300 Read block 779 len 8192
1373 149/300 Flush
1374 150/300 Write block 637 len 16384 data: 2 2 2 2
1375 151/300 Read block 2321 len 16384
1376 152/300 Read block 1629 len 40960
1377 153/300 Write block 3974 len 24576 data: 2 2 2 2 2 2
1378 154/300 Read block 1460 len 32768
1379 155/300 Read block 2646 len 20480
1380 156/300 Read block 2021 len 36864
1381 157/300 Write block 4784 len 4096 data: 2
1382 158/300 Write block 4360 len 8192 data: 3 3
1383 159/300 Read block 1443 len 28672
1384 160/300 Flush
1385 161/300 Write block 4016 len 24576 data: 2 2 2 2 2 2
1386 162/300 Write block 1981 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1387 163/300 Write block 3738 len 36864 data: 2 2 2 2 2 2 2 2 2
1388 164/300 Flush
1389 165/300 Flush
1390 166/300 Read block 3234 len 20480
1391 167/300 Write block 947 len 36864 data: 2 2 2 2 2 2 3 3 3
1392 168/300 Write block 4377 len 20480 data: 3 4 4 4 4
1393 169/300 Write block 3361 len 32768 data: 2 2 2 2 2 2 2 2
1394 170/300 Write block 2202 len 24576 data: 2 2 2 2 2 2
1395 171/300 Read block 4006 len 32768
1396 172/300 Write block 758 len 40960 data: 3 3 3 3 3 3 2 2 2 2
1397 173/300 Read block 3735 len 24576
1398 174/300 Write block 1575 len 28672 data: 2 2 2 2 2 2 2
1399 175/300 Write block 3025 len 4096 data: 2
1400 176/300 Read block 1232 len 28672
1401 177/300 Write block 4298 len 12288 data: 2 2 2
1402 178/300 Write block 3961 len 24576 data: 2 2 2 2 2 2
1403 179/300 Write block 1856 len 28672 data: 2 2 2 2 2 2 2
1404 180/300 Read block 345 len 40960
1405 181/300 Write block 4096 len 16384 data: 3 3 3 3
1406 182/300 Read block 4446 len 40960
1407 183/300 Write block 813 len 28672 data: 2 2 2 2 2 2 2
1408 184/300 Write block 3971 len 40960 data: 2 2 2 3 3 3 3 3 3 2
1409 185/300 Read block 1619 len 28672
1410 186/300 Read block 3488 len 24576
1411 187/300 Read block 3904 len 36864
1412 188/300 Write block 2026 len 12288 data: 2 2 2
1413 189/300 Read block 3658 len 40960
1414 190/300 Read block 989 len 20480
1415 191/300 Flush
1416 192/300 Write block 4228 len 12288 data: 2 2 2
1417 193/300 Read block 4038 len 4096
1418 194/300 Write block 3708 len 4096 data: 2
1419 195/300 Write block 4524 len 24576 data: 2 2 2 2 2 2
1420 196/300 Read block 1507 len 36864
1421 197/300 Flush
1422 198/300 Write block 2960 len 8192 data: 2 3
1423 199/300 Write block 2985 len 20480 data: 3 3 3 3 2
1424 200/300 Write block 4138 len 28672 data: 2 2 2 2 2 2 2
1425 201/300 Write block 2540 len 16384 data: 2 2 2 2
1426 202/300 Write block 3135 len 28672 data: 2 2 2 2 2 2 2
1427 203/300 Read block 44 len 32768
1428 204/300 Read block 3865 len 4096
1429 205/300 Read block 3769 len 40960
1430 206/300 Flush
1431 207/300 Flush
1432 208/300 Read block 1943 len 28672
1433 209/300 Read block 2702 len 32768
1434 210/300 Write block 1386 len 20480 data: 2 2 2 2 2
1435 211/300 Write block 1526 len 28672 data: 2 2 2 2 2 2 3
1436 212/300 Read block 4639 len 36864
1437 213/300 Read block 3735 len 16384
1438 214/300 Write block 2735 len 24576 data: 2 2 2 2 2 2
1439 215/300 Read block 2967 len 16384
1440 216/300 Write block 1733 len 8192 data: 2 2
1441 217/300 Read block 3053 len 24576
1442 218/300 Write block 3564 len 20480 data: 2 2 2 2 2
1443 219/300 Write block 1497 len 36864 data: 3 3 3 3 3 3 3 2 2
1444 220/300 Write block 1602 len 8192 data: 2 2
1445 221/300 Flush
1446 222/300 Read block 4766 len 16384
1447 223/300 Read block 4550 len 12288
1448 224/300 Read block 1384 len 12288
1449 225/300 Write block 207 len 16384 data: 3 3 3 3
1450 226/300 Write block 2074 len 4096 data: 2
1451 227/300 Read block 1418 len 4096
1452 228/300 Read block 1780 len 12288
1453 229/300 Flush
1454 230/300 Read block 4358 len 12288
1455 231/300 Read block 1192 len 24576
1456 232/300 Write block 2684 len 20480 data: 2 2 2 2 2
1457 233/300 Flush
1458 234/300 Write block 1753 len 4096 data: 2
1459 235/300 Write block 3807 len 16384 data: 2 2 2 2
1460 236/300 Write block 2503 len 36864 data: 2 2 2 2 2 2 2 2 2
1461 237/300 Write block 132 len 8192 data: 2 3
1462 238/300 Read block 3052 len 40960
1463 239/300 Read block 2304 len 28672
1464 240/300 Write block 1740 len 28672 data: 2 2 2 3 3 2 2
1465 241/300 Read block 1772 len 20480
1466 242/300 Read block 3914 len 24576
1467 243/300 Write block 2702 len 20480 data: 2 2 2 2 2
1468 244/300 Read block 1417 len 20480
1469 245/300 Write block 4946 len 4096 data: 2
1470 246/300 Write block 2886 len 40960 data: 2 2 2 2 3 3 3 3 3 3
1471 247/300 Flush
1472 248/300 Write block 3430 len 24576 data: 2 2 2 2 2 2
1473 249/300 Write block 4619 len 32768 data: 2 2 2 2 2 2 2 2
1474 250/300 Write block 233 len 24576 data: 2 2 2 2 2 2
1475 251/300 Read block 3585 len 8192
1476 252/300 Flush
1477 253/300 Read block 349 len 24576
1478 254/300 Write block 517 len 4096 data: 3
1479 255/300 Flush
1480 256/300 Read block 1120 len 8192
1481 257/300 Write block 771 len 24576 data: 2 2 2 2 2 2
1482 258/300 Write block 1802 len 24576 data: 2 2 2 2 2 2
1483 259/300 Write block 4282 len 24576 data: 2 2 2 2 2 2
1484 260/300 Read block 440 len 24576
1485 261/300 Write block 899 len 16384 data: 2 2 2 2
1486 262/300 Read block 4245 len 28672
1487 263/300 Write block 2817 len 4096 data: 3
1488 264/300 Read block 294 len 28672
1489 265/300 Read block 201 len 8192
1490 266/300 Flush
1491 267/300 Write block 1819 len 40960 data: 2 2 2 3 3 2 2 2 2 2
1492 268/300 Read block 4685 len 12288
1493 269/300 Read block 1105 len 8192
1494 270/300 Read block 2035 len 32768
1495 271/300 Read block 1196 len 12288
1496 272/300 Flush
1497 273/300 Read block 93 len 36864
1498 274/300 Read block 2829 len 36864
1499 275/300 Read block 4582 len 8192
1500 276/300 Read block 4635 len 28672
1501 277/300 Read block 3043 len 24576
1502 278/300 Write block 1382 len 24576 data: 2 2 2 2 3 3
1503 279/300 Read block 4646 len 32768
1504 280/300 Write block 4812 len 28672 data: 3 3 3 2 2 2 2
1505 281/300 Write block 3044 len 16384 data: 3 3 2 2
1506 282/300 Write block 2625 len 20480 data: 3 2 2 2 2
1507 283/300 Write block 801 len 8192 data: 2 2
1508 284/300 Flush
1509 285/300 Write block 2210 len 36864 data: 2 2 2 2 2 2 2 2 2
1510 286/300 Write block 3160 len 40960 data: 3 3 3 2 2 2 2 2 2 2
1511 287/300 Flush
1512 288/300 Read block 3591 len 8192
1513 289/300 Read block 2379 len 32768
1514 290/300 Write block 3517 len 24576 data: 2 2 2 3 3 3
1515 291/300 Flush
1516 292/300 Read block 10 len 40960
1517 293/300 Read block 205 len 40960
1518 294/300 Read block 4071 len 4096
1519 295/300 Write block 803 len 8192 data: 2 2
1520 296/300 Write block 1894 len 8192 data: 2 2
1521 297/300 Read block 131 len 24576
1522 298/300 Write block 211 len 36864 data: 3 3 3 2 2 2 2 2 2
1523 299/300 Read block 965 len 40960
1524 300/300 Read block 3100 len 36864
1525 301/300 Write block 429 len 20480 data: 2 2 2 2 2
1526 Main task has work to do, go find it
1527 got dsc Start(1)
1528 start 1
1529 Dec 09 15:08:11.588 INFO request completed, latency_us: 278, response_code: 204, uri: /start/cid/1, method: POST, req_id: 588686bb-89f7-40af-b1ea-6a701f3aff76, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
1530 Replay: started 1, returned:Ok(())
1531 ----------------------------------------------------------------
1532 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1533 States: Active Offline Active
1534 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1535 1302 Acked Read 10 Done Sent Done false
1536 1303 Acked Write 7 Done Sent Done false
1537 1304 Acked Write 8 Done Sent Done false
1538 1305 Acked Flush 0 Done Sent Done false
1539 1306 Acked Write 9 Done Sent Done false
1540 1307 Acked Read 7 Done Sent Done false
1541 1308 Acked Read 10 Done Sent Done false
1542 1309 Acked Write 7 Done Sent Done false
1543 1310 Acked Write 8 Done Sent Done false
1544 1311 Acked Read 4 Done Sent Done false
1545 1312 Acked Read 7 Done Sent Done false
1546 1313 Acked Write 9 Done Sent Done false
1547 1314 Acked Read 7 Done Sent Done false
1548 1315 Acked Read 2 Done Sent Done false
1549 1316 Acked Write 2 Done Sent Done false
1550 1317 Acked Write 3 Done Sent Done false
1551 1318 Acked Write 3 Done Sent Done false
1552 1319 Acked Read 1 Done Sent Done false
1553 1320 Acked Write 1 Done Sent Done false
1554 1321 Acked Read 4 Done Sent Done false
1555 1322 Acked Write 2 Done Sent Done false
1556 1323 Acked Read 7 Done Sent Done false
1557 1324 Acked Write 9 Done Sent Done false
1558 1325 Acked Read 2 Done Sent Done false
1559 1326 Acked Write 6 Done Sent Done false
1560 1327 Acked Write 5 Done Sent Done false
1561 1328 Acked Read 3 Done Sent Done false
1562 1329 Acked Read 9 Done Sent Done false
1563 1330 Acked Read 7 Done Sent Done false
1564 1331 Acked Flush 0 Done Sent Done false
1565 1332 Acked Write 4 Done Sent Done false
1566 1333 Acked Read 5 Done Sent Done false
1567 1334 Acked Read 5 Done Sent Done false
1568 1335 Acked Write 8 Done Sent Done false
1569 1336 Acked Read 1 Done Sent Done false
1570 1337 Acked Write 8 Done Sent Done false
1571 1338 Acked Write 7 Done Sent Done false
1572 1339 Acked Write 7 Done Sent Done false
1573 1340 Acked Write 8 Done Sent Done false
1574 1341 Acked Flush 0 Done Sent Done false
1575 1342 Acked Read 10 Done Sent Done false
1576 1343 Acked Write 8 Done Sent Done false
1577 1344 Acked Read 2 Done Sent Done false
1578 1345 Acked Read 9 Done Sent Done false
1579 1346 Acked Write 6 Done Sent Done false
1580 1347 Acked Read 9 Done Sent Done false
1581 1348 Acked Write 8 Done Sent Done false
1582 1349 Acked Write 9 Done Sent Done false
1583 1350 Acked Read 8 Done Sent Done false
1584 1351 Acked Read 3 Done Sent Done false
1585 1352 Acked Write 9 Done Sent Done false
1586 1353 Acked Write 7 Done Sent Done false
1587 1354 Acked Flush 0 Done Sent Done false
1588 1355 Acked Read 2 Done Sent Done false
1589 1356 Acked Write 4 Done Sent Done false
1590 1357 Acked Write 4 Done Sent Done false
1591 1358 Acked Read 7 Done Sent Done false
1592 1359 Acked Write 8 Done Sent Done false
1593 1360 Acked Write 8 Done Sent Done false
1594 1361 Acked Read 2 Done Sent Done false
1595 1362 Acked Write 6 Done Sent Done false
1596 1363 Acked Write 10 Done Sent Done false
1597 1364 Acked Read 8 Done Sent Done false
1598 1365 Acked Read 3 Done Sent Done false
1599 1366 Acked Write 10 Done Sent Done false
1600 1367 Acked Read 3 Done Sent Done false
1601 1368 Acked Write 7 Done Sent Done false
1602 1369 Acked Read 4 Done Sent Done false
1603 1370 Acked Read 10 Done Sent Done false
1604 1371 Acked Write 7 Done Sent Done false
1605 1372 Acked Write 8 Done Sent Done false
1606 1373 Acked Read 5 Done Sent Done false
1607 1374 Acked Write 1 Done Sent Done false
1608 1375 Acked Write 3 Done Sent Done false
1609 1376 Acked Write 4 Done Sent Done false
1610 1377 Acked Write 7 Done Sent Done false
1611 1378 Acked Write 10 Done Sent Done false
1612 1379 Acked Write 4 Done Sent Done false
1613 1380 Acked Write 6 Done Sent Done false
1614 1381 Acked Read 5 Done Sent Done false
1615 1382 Acked Flush 0 Done Sent Done false
1616 1383 Acked Read 1 Done Sent Done false
1617 1384 Acked Write 3 Done Sent Done false
1618 1385 Acked Write 2 Done Sent Done false
1619 1386 Acked Read 9 Done Sent Done false
1620 1387 Acked Read 9 Done Sent Done false
1621 1388 Acked Read 2 Done Sent Done false
1622 1389 Acked Write 10 Done Sent Done false
1623 1390 Acked Read 3 Done Sent Done false
1624 1391 Acked Write 4 Done Sent Done false
1625 1392 Acked Write 4 Done Sent Done false
1626 1393 Acked Flush 0 Done Sent Done false
1627 1394 Acked Write 4 Done Sent Done false
1628 1395 Acked Read 1 Done Sent Done false
1629 1396 Acked Read 10 Done Sent Done false
1630 1397 Acked Write 10 Done Sent Done false
1631 1398 Acked Read 5 Done Sent Done false
1632 1399 Acked Read 4 Done Sent Done false
1633 1400 Acked Read 9 Done Sent Done false
1634 1401 Acked Read 3 Done Sent Done false
1635 1402 Acked Write 7 Done Sent Done false
1636 1403 Acked Write 9 Done Sent Done false
1637 1404 Acked Read 7 Done Sent Done false
1638 1405 Acked Write 9 Done Sent Done false
1639 1406 Acked Write 2 Done Sent Done false
1640 1407 Acked Read 1 Done Sent Done false
1641 1408 Acked Write 10 Done Sent Done false
1642 1409 Acked Flush 0 Done Sent Done false
1643 1410 Acked Write 10 Done Sent Done false
1644 1411 Acked Write 2 Done Sent Done false
1645 1412 Acked Read 10 Done Sent Done false
1646 1413 Acked Read 5 Done Sent Done false
1647 1414 Acked Read 8 Done Sent Done false
1648 1415 Acked Write 1 Done Sent Done false
1649 1416 Acked Read 3 Done Sent Done false
1650 1417 Acked Read 4 Done Sent Done false
1651 1418 Acked Write 1 Done Sent Done false
1652 1419 Acked Write 6 Done Sent Done false
1653 1420 Acked Write 3 Done Sent Done false
1654 1421 Acked Read 3 Done Sent Done false
1655 1422 Acked Read 5 Done Sent Done false
1656 1423 Acked Read 7 Done Sent Done false
1657 1424 Acked Read 3 Done Sent Done false
1658 1425 Acked Read 2 Done Sent Done false
1659 1426 Acked Write 3 Done Sent Done false
1660 1427 Acked Write 9 Done Sent Done false
1661 1428 Acked Flush 0 Done Sent Done false
1662 1429 Acked Read 6 Done Sent Done false
1663 1430 Acked Write 6 Done Sent Done false
1664 1431 Acked Read 8 Done Sent Done false
1665 1432 Acked Write 5 Done Sent Done false
1666 1433 Acked Read 3 Done Sent Done false
1667 1434 Acked Flush 0 Done Sent Done false
1668 1435 Acked Write 1 Done Sent Done false
1669 1436 Acked Read 3 Done Sent Done false
1670 1437 Acked Read 1 Done Sent Done false
1671 1438 Acked Read 4 Done Sent Done false
1672 1439 Acked Flush 0 Done Sent Done false
1673 1440 Acked Write 4 Done Sent Done false
1674 1441 Acked Write 6 Done Sent Done false
1675 1442 Acked Read 1 Done Sent Done false
1676 1443 Acked Read 8 Done Sent Done false
1677 1444 Acked Read 2 Done Sent Done false
1678 1445 Acked Read 7 Done Sent Done false
1679 1446 Acked Write 6 Done Sent Done false
1680 1447 Acked Read 9 Done Sent Done false
1681 1448 Acked Read 1 Done Sent Done false
1682 1449 Acked Read 2 Done Sent Done false
1683 1450 Acked Flush 0 Done Sent Done false
1684 1451 Acked Write 4 Done Sent Done false
1685 1452 Acked Read 4 Done Sent Done false
1686 1453 Acked Read 10 Done Sent Done false
1687 1454 Acked Write 6 Done Sent Done false
1688 1455 Acked Read 8 Done Sent Done false
1689 1456 Acked Read 5 Done Sent Done false
1690 1457 Acked Read 9 Done Sent Done false
1691 1458 Acked Write 1 Done Sent Done false
1692 1459 Acked Write 2 Done Sent Done false
1693 1460 Acked Read 7 Done Sent Done false
1694 1461 Acked Flush 0 Done Sent Done false
1695 1462 Acked Write 6 Done Sent Done false
1696 1463 Acked Write 10 Done Sent Done false
1697 1464 Acked Write 9 Done Sent Done false
1698 1465 Acked Flush 0 Done Sent Done false
1699 1466 Acked Flush 0 Done Sent Done false
1700 1467 Acked Read 5 Done Sent Done false
1701 1468 Acked Write 9 Done Sent Done false
1702 1469 Acked Write 5 Done Sent Done false
1703 1470 Acked Write 8 Done Sent Done false
1704 1471 Acked Write 6 Done Sent Done false
1705 1472 Acked Read 8 Done Sent Done false
1706 1473 Acked Write 10 Done Sent Done false
1707 1474 Acked Read 6 Done Sent Done false
1708 1475 Acked Write 7 Done Sent Done false
1709 1476 Acked Write 1 Done Sent Done false
1710 1477 Acked Read 7 Done Sent Done false
1711 1478 Acked Write 3 Done Sent Done false
1712 1479 Acked Write 6 Done Sent Done false
1713 1480 Acked Write 7 Done Sent Done false
1714 1481 Acked Read 10 Done Sent Done false
1715 1482 Acked Write 4 Done Sent Done false
1716 1483 Acked Read 10 Done Sent Done false
1717 1484 Acked Write 7 Done Sent Done false
1718 1485 Acked Write 10 Done Sent Done false
1719 1486 Acked Read 7 Done Sent Done false
1720 1487 Acked Read 6 Done Sent Done false
1721 1488 Acked Read 9 Done Sent Done false
1722 1489 Acked Write 3 Done Sent Done false
1723 1490 Acked Read 10 Done Sent Done false
1724 1491 Acked Read 5 Done Sent Done false
1725 1492 Acked Flush 0 Done Sent Done false
1726 1493 Acked Write 3 Done Sent Done false
1727 1494 Acked Read 1 Done Sent Done false
1728 1495 Acked Write 1 Done Sent Done false
1729 1496 Acked Write 6 Done Sent Done false
1730 1497 Acked Read 9 Done Sent Done false
1731 1498 Acked Flush 0 Done Sent Done false
1732 1499 Acked Write 2 Done Sent Done false
1733 1500 Acked Write 5 Done Sent Done false
1734 1501 Acked Write 7 Done Sent Done false
1735 1502 Acked Write 4 Done Sent Done false
1736 1503 Acked Write 7 Done Sent Done false
1737 1504 Acked Read 8 Done Sent Done false
1738 1505 Acked Read 1 Done Sent Done false
1739 1506 Acked Read 10 Done Sent Done false
1740 1507 Acked Flush 0 Done Sent Done false
1741 1508 Acked Flush 0 Done Sent Done false
1742 1509 Acked Read 7 Done Sent Done false
1743 1510 Acked Read 8 Done Sent Done false
1744 1511 Acked Write 5 Done Sent Done false
1745 1512 Acked Write 7 Done Sent Done false
1746 1513 Acked Read 9 Done Sent Done false
1747 1514 Acked Read 4 Done Sent Done false
1748 1515 Acked Write 6 Done Sent Done false
1749 1516 Acked Read 4 Done Sent Done false
1750 1517 Acked Write 2 Done Sent Done false
1751 1518 Acked Read 6 Done Sent Done false
1752 1519 Acked Write 5 Done Sent Done false
1753 1520 Acked Write 9 Done Sent Done false
1754 1521 Acked Write 2 Done Sent Done false
1755 1522 Acked Flush 0 Done Sent Done false
1756 1523 Acked Read 4 Done Sent Done false
1757 1524 Acked Read 3 Done Sent Done false
1758 1525 Acked Read 3 Done Sent Done false
1759 1526 Acked Write 4 Done Sent Done false
1760 1527 Acked Write 1 Done Sent Done false
1761 1528 Acked Read 1 Done Sent Done false
1762 1529 Acked Read 3 Done Sent Done false
1763 1530 Acked Flush 0 Done Sent Done false
1764 1531 Acked Read 3 Done Sent Done false
1765 1532 Acked Read 6 Done Sent Done false
1766 1533 Acked Write 5 Done Sent Done false
1767 1534 Acked Flush 0 Done Sent Done false
1768 1535 Acked Write 1 Done Sent Done false
1769 1536 Acked Write 4 Done Sent Done false
1770 1537 Acked Write 9 Done Sent Done false
1771 1538 Acked Write 2 Done Sent Done false
1772 1539 Acked Read 10 Done Sent Done false
1773 1540 Acked Read 7 Done Sent Done false
1774 1541 Acked Write 7 Done Sent Done false
1775 1542 Acked Read 5 Done Sent Done false
1776 1543 Acked Read 6 Done Sent Done false
1777 1544 Acked Write 5 Done Sent Done false
1778 1545 Acked Read 5 Done Sent Done false
1779 1546 Acked Write 1 Done Sent Done false
1780 1547 Acked Write 10 Done Sent Done false
1781 1548 Acked Flush 0 Done Sent Done false
1782 1549 Acked Write 6 Done Sent Done false
1783 1550 Acked Write 8 Done Sent Done false
1784 1551 Acked Write 6 Done Sent Done false
1785 1552 Acked Read 2 Done Sent Done false
1786 1553 Acked Flush 0 Done Sent Done false
1787 1554 Acked Read 6 Done Sent Done false
1788 1555 Acked Write 1 Done Sent Done false
1789 1556 Acked Flush 0 Done Sent Done false
1790 1557 Acked Read 2 Done Sent Done false
1791 1558 Acked Write 6 Done Sent Done false
1792 1559 Acked Write 6 Done Sent Done false
1793 1560 Acked Write 6 Done Sent Done false
1794 1561 Acked Read 6 Done Sent Done false
1795 1562 Acked Write 4 Done Sent Done false
1796 1563 Acked Read 7 Done Sent Done false
1797 1564 Acked Write 1 Done Sent Done false
1798 1565 Acked Read 7 Done Sent Done false
1799 1566 Acked Read 2 Done Sent Done false
1800 1567 Acked Flush 0 Done Sent Done false
1801 1568 Acked Write 10 Done Sent Done false
1802 1569 Acked Read 3 Done Sent Done false
1803 1570 Acked Read 2 Done Sent Done false
1804 1571 Acked Read 8 Done Sent Done false
1805 1572 Acked Read 3 Done Sent Done false
1806 1573 Acked Flush 0 Done Sent Done false
1807 1574 Acked Read 9 Done Sent Done false
1808 1575 Acked Read 9 Done Sent Done false
1809 1576 Acked Read 2 Done Sent Done false
1810 1577 Acked Read 7 Done Sent Done false
1811 1578 Acked Read 6 Done Sent Done false
1812 1579 Acked Write 6 Done Sent Done false
1813 1580 Acked Read 8 Done Sent Done false
1814 1581 Acked Write 7 Done Sent Done false
1815 1582 Acked Write 4 Done Sent Done false
1816 1583 Acked Write 5 Done Sent Done false
1817 1584 Acked Write 2 Done Sent Done false
1818 1585 Acked Flush 0 Done Sent Done false
1819 1586 Acked Write 9 Done Sent Done false
1820 1587 Acked Write 10 Done Sent Done false
1821 1588 Acked Flush 0 Done Sent Done false
1822 1589 Acked Read 2 Done Sent Done false
1823 1590 Acked Read 8 Done Sent Done false
1824 1591 Acked Write 6 Done Sent Done false
1825 1592 Acked Flush 0 Done Sent Done false
1826 1593 Acked Read 10 Done Sent Done false
1827 1594 Acked Read 10 Done Sent Done false
1828 1595 Acked Read 1 Done Sent Done false
1829 1596 Acked Write 2 Done Sent Done false
1830 1597 Acked Write 2 Done Sent Done false
1831 1598 Acked Read 6 Done Sent Done false
1832 1599 Acked Write 9 Done Sent Done false
1833 1600 Acked Read 10 Done Sent Done false
1834 1601 Acked Read 9 Done Sent Done false
1835 1602 Acked Write 5 Done Sent Done false
1836 STATES DS:0 DS:1 DS:2 TOTAL
1837 Sent 0 301 0 301
1838 Done 301 0 301 602
1839 Skipped 0 0 0 0
1840 Error 0 0 0 0
1841 Last Flush: 1592 1301 1592
1842 Downstairs last five completed: 1301 1300 1299 1298 1297
1843 Upstairs last five completed: 1602 1601 1600 1599 1598
1844 CLIENT: Up:0 ds:301 act:2
1845 [8820] got start action
1846 [8820] I am going to restart
1847 Starting downstairs at port 8820
1848 Make output file at "/tmp/dsc/downstairs-8820.txt"
1849 [8820][1] reports Starting
1850 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1203)
1851 [8820][1] reports Running
1852 ----------------------------------------------------------------
1853 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
1854 States: Active Offline Active
1855 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1856 1302 Acked Read 10 Done Sent Done false
1857 1303 Acked Write 7 Done Sent Done false
1858 1304 Acked Write 8 Done Sent Done false
1859 1305 Acked Flush 0 Done Sent Done false
1860 1306 Acked Write 9 Done Sent Done false
1861 1307 Acked Read 7 Done Sent Done false
1862 1308 Acked Read 10 Done Sent Done false
1863 1309 Acked Write 7 Done Sent Done false
1864 1310 Acked Write 8 Done Sent Done false
1865 1311 Acked Read 4 Done Sent Done false
1866 1312 Acked Read 7 Done Sent Done false
1867 1313 Acked Write 9 Done Sent Done false
1868 1314 Acked Read 7 Done Sent Done false
1869 1315 Acked Read 2 Done Sent Done false
1870 1316 Acked Write 2 Done Sent Done false
1871 1317 Acked Write 3 Done Sent Done false
1872 1318 Acked Write 3 Done Sent Done false
1873 1319 Acked Read 1 Done Sent Done false
1874 1320 Acked Write 1 Done Sent Done false
1875 1321 Acked Read 4 Done Sent Done false
1876 1322 Acked Write 2 Done Sent Done false
1877 1323 Acked Read 7 Done Sent Done false
1878 1324 Acked Write 9 Done Sent Done false
1879 1325 Acked Read 2 Done Sent Done false
1880 1326 Acked Write 6 Done Sent Done false
1881 1327 Acked Write 5 Done Sent Done false
1882 1328 Acked Read 3 Done Sent Done false
1883 1329 Acked Read 9 Done Sent Done false
1884 1330 Acked Read 7 Done Sent Done false
1885 1331 Acked Flush 0 Done Sent Done false
1886 1332 Acked Write 4 Done Sent Done false
1887 1333 Acked Read 5 Done Sent Done false
1888 1334 Acked Read 5 Done Sent Done false
1889 1335 Acked Write 8 Done Sent Done false
1890 1336 Acked Read 1 Done Sent Done false
1891 1337 Acked Write 8 Done Sent Done false
1892 1338 Acked Write 7 Done Sent Done false
1893 1339 Acked Write 7 Done Sent Done false
1894 1340 Acked Write 8 Done Sent Done false
1895 1341 Acked Flush 0 Done Sent Done false
1896 1342 Acked Read 10 Done Sent Done false
1897 1343 Acked Write 8 Done Sent Done false
1898 1344 Acked Read 2 Done Sent Done false
1899 1345 Acked Read 9 Done Sent Done false
1900 1346 Acked Write 6 Done Sent Done false
1901 1347 Acked Read 9 Done Sent Done false
1902 1348 Acked Write 8 Done Sent Done false
1903 1349 Acked Write 9 Done Sent Done false
1904 1350 Acked Read 8 Done Sent Done false
1905 1351 Acked Read 3 Done Sent Done false
1906 1352 Acked Write 9 Done Sent Done false
1907 1353 Acked Write 7 Done Sent Done false
1908 1354 Acked Flush 0 Done Sent Done false
1909 1355 Acked Read 2 Done Sent Done false
1910 1356 Acked Write 4 Done Sent Done false
1911 1357 Acked Write 4 Done Sent Done false
1912 1358 Acked Read 7 Done Sent Done false
1913 1359 Acked Write 8 Done Sent Done false
1914 1360 Acked Write 8 Done Sent Done false
1915 1361 Acked Read 2 Done Sent Done false
1916 1362 Acked Write 6 Done Sent Done false
1917 1363 Acked Write 10 Done Sent Done false
1918 1364 Acked Read 8 Done Sent Done false
1919 1365 Acked Read 3 Done Sent Done false
1920 1366 Acked Write 10 Done Sent Done false
1921 1367 Acked Read 3 Done Sent Done false
1922 1368 Acked Write 7 Done Sent Done false
1923 1369 Acked Read 4 Done Sent Done false
1924 1370 Acked Read 10 Done Sent Done false
1925 1371 Acked Write 7 Done Sent Done false
1926 1372 Acked Write 8 Done Sent Done false
1927 1373 Acked Read 5 Done Sent Done false
1928 1374 Acked Write 1 Done Sent Done false
1929 1375 Acked Write 3 Done Sent Done false
1930 1376 Acked Write 4 Done Sent Done false
1931 1377 Acked Write 7 Done Sent Done false
1932 1378 Acked Write 10 Done Sent Done false
1933 1379 Acked Write 4 Done Sent Done false
1934 1380 Acked Write 6 Done Sent Done false
1935 1381 Acked Read 5 Done Sent Done false
1936 1382 Acked Flush 0 Done Sent Done false
1937 1383 Acked Read 1 Done Sent Done false
1938 1384 Acked Write 3 Done Sent Done false
1939 1385 Acked Write 2 Done Sent Done false
1940 1386 Acked Read 9 Done Sent Done false
1941 1387 Acked Read 9 Done Sent Done false
1942 1388 Acked Read 2 Done Sent Done false
1943 1389 Acked Write 10 Done Sent Done false
1944 1390 Acked Read 3 Done Sent Done false
1945 1391 Acked Write 4 Done Sent Done false
1946 1392 Acked Write 4 Done Sent Done false
1947 1393 Acked Flush 0 Done Sent Done false
1948 1394 Acked Write 4 Done Sent Done false
1949 1395 Acked Read 1 Done Sent Done false
1950 1396 Acked Read 10 Done Sent Done false
1951 1397 Acked Write 10 Done Sent Done false
1952 1398 Acked Read 5 Done Sent Done false
1953 1399 Acked Read 4 Done Sent Done false
1954 1400 Acked Read 9 Done Sent Done false
1955 1401 Acked Read 3 Done Sent Done false
1956 1402 Acked Write 7 Done Sent Done false
1957 1403 Acked Write 9 Done Sent Done false
1958 1404 Acked Read 7 Done Sent Done false
1959 1405 Acked Write 9 Done Sent Done false
1960 1406 Acked Write 2 Done Sent Done false
1961 1407 Acked Read 1 Done Sent Done false
1962 1408 Acked Write 10 Done Sent Done false
1963 1409 Acked Flush 0 Done Sent Done false
1964 1410 Acked Write 10 Done Sent Done false
1965 1411 Acked Write 2 Done Sent Done false
1966 1412 Acked Read 10 Done Sent Done false
1967 1413 Acked Read 5 Done Sent Done false
1968 1414 Acked Read 8 Done Sent Done false
1969 1415 Acked Write 1 Done Sent Done false
1970 1416 Acked Read 3 Done Sent Done false
1971 1417 Acked Read 4 Done Sent Done false
1972 1418 Acked Write 1 Done Sent Done false
1973 1419 Acked Write 6 Done Sent Done false
1974 1420 Acked Write 3 Done Sent Done false
1975 1421 Acked Read 3 Done Sent Done false
1976 1422 Acked Read 5 Done Sent Done false
1977 1423 Acked Read 7 Done Sent Done false
1978 1424 Acked Read 3 Done Sent Done false
1979 1425 Acked Read 2 Done Sent Done false
1980 1426 Acked Write 3 Done Sent Done false
1981 1427 Acked Write 9 Done Sent Done false
1982 1428 Acked Flush 0 Done Sent Done false
1983 1429 Acked Read 6 Done Sent Done false
1984 1430 Acked Write 6 Done Sent Done false
1985 1431 Acked Read 8 Done Sent Done false
1986 1432 Acked Write 5 Done Sent Done false
1987 1433 Acked Read 3 Done Sent Done false
1988 1434 Acked Flush 0 Done Sent Done false
1989 1435 Acked Write 1 Done Sent Done false
1990 1436 Acked Read 3 Done Sent Done false
1991 1437 Acked Read 1 Done Sent Done false
1992 1438 Acked Read 4 Done Sent Done false
1993 1439 Acked Flush 0 Done Sent Done false
1994 1440 Acked Write 4 Done Sent Done false
1995 1441 Acked Write 6 Done Sent Done false
1996 1442 Acked Read 1 Done Sent Done false
1997 1443 Acked Read 8 Done Sent Done false
1998 1444 Acked Read 2 Done Sent Done false
1999 1445 Acked Read 7 Done Sent Done false
2000 1446 Acked Write 6 Done Sent Done false
2001 1447 Acked Read 9 Done Sent Done false
2002 1448 Acked Read 1 Done Sent Done false
2003 1449 Acked Read 2 Done Sent Done false
2004 1450 Acked Flush 0 Done Sent Done false
2005 1451 Acked Write 4 Done Sent Done false
2006 1452 Acked Read 4 Done Sent Done false
2007 1453 Acked Read 10 Done Sent Done false
2008 1454 Acked Write 6 Done Sent Done false
2009 1455 Acked Read 8 Done Sent Done false
2010 1456 Acked Read 5 Done Sent Done false
2011 1457 Acked Read 9 Done Sent Done false
2012 1458 Acked Write 1 Done Sent Done false
2013 1459 Acked Write 2 Done Sent Done false
2014 1460 Acked Read 7 Done Sent Done false
2015 1461 Acked Flush 0 Done Sent Done false
2016 1462 Acked Write 6 Done Sent Done false
2017 1463 Acked Write 10 Done Sent Done false
2018 1464 Acked Write 9 Done Sent Done false
2019 1465 Acked Flush 0 Done Sent Done false
2020 1466 Acked Flush 0 Done Sent Done false
2021 1467 Acked Read 5 Done Sent Done false
2022 1468 Acked Write 9 Done Sent Done false
2023 1469 Acked Write 5 Done Sent Done false
2024 1470 Acked Write 8 Done Sent Done false
2025 1471 Acked Write 6 Done Sent Done false
2026 1472 Acked Read 8 Done Sent Done false
2027 1473 Acked Write 10 Done Sent Done false
2028 1474 Acked Read 6 Done Sent Done false
2029 1475 Acked Write 7 Done Sent Done false
2030 1476 Acked Write 1 Done Sent Done false
2031 1477 Acked Read 7 Done Sent Done false
2032 1478 Acked Write 3 Done Sent Done false
2033 1479 Acked Write 6 Done Sent Done false
2034 1480 Acked Write 7 Done Sent Done false
2035 1481 Acked Read 10 Done Sent Done false
2036 1482 Acked Write 4 Done Sent Done false
2037 1483 Acked Read 10 Done Sent Done false
2038 1484 Acked Write 7 Done Sent Done false
2039 1485 Acked Write 10 Done Sent Done false
2040 1486 Acked Read 7 Done Sent Done false
2041 1487 Acked Read 6 Done Sent Done false
2042 1488 Acked Read 9 Done Sent Done false
2043 1489 Acked Write 3 Done Sent Done false
2044 1490 Acked Read 10 Done Sent Done false
2045 1491 Acked Read 5 Done Sent Done false
2046 1492 Acked Flush 0 Done Sent Done false
2047 1493 Acked Write 3 Done Sent Done false
2048 1494 Acked Read 1 Done Sent Done false
2049 1495 Acked Write 1 Done Sent Done false
2050 1496 Acked Write 6 Done Sent Done false
2051 1497 Acked Read 9 Done Sent Done false
2052 1498 Acked Flush 0 Done Sent Done false
2053 1499 Acked Write 2 Done Sent Done false
2054 1500 Acked Write 5 Done Sent Done false
2055 1501 Acked Write 7 Done Sent Done false
2056 1502 Acked Write 4 Done Sent Done false
2057 1503 Acked Write 7 Done Sent Done false
2058 1504 Acked Read 8 Done Sent Done false
2059 1505 Acked Read 1 Done Sent Done false
2060 1506 Acked Read 10 Done Sent Done false
2061 1507 Acked Flush 0 Done Sent Done false
2062 1508 Acked Flush 0 Done Sent Done false
2063 1509 Acked Read 7 Done Sent Done false
2064 1510 Acked Read 8 Done Sent Done false
2065 1511 Acked Write 5 Done Sent Done false
2066 1512 Acked Write 7 Done Sent Done false
2067 1513 Acked Read 9 Done Sent Done false
2068 1514 Acked Read 4 Done Sent Done false
2069 1515 Acked Write 6 Done Sent Done false
2070 1516 Acked Read 4 Done Sent Done false
2071 1517 Acked Write 2 Done Sent Done false
2072 1518 Acked Read 6 Done Sent Done false
2073 1519 Acked Write 5 Done Sent Done false
2074 1520 Acked Write 9 Done Sent Done false
2075 1521 Acked Write 2 Done Sent Done false
2076 1522 Acked Flush 0 Done Sent Done false
2077 1523 Acked Read 4 Done Sent Done false
2078 1524 Acked Read 3 Done Sent Done false
2079 1525 Acked Read 3 Done Sent Done false
2080 1526 Acked Write 4 Done Sent Done false
2081 1527 Acked Write 1 Done Sent Done false
2082 1528 Acked Read 1 Done Sent Done false
2083 1529 Acked Read 3 Done Sent Done false
2084 1530 Acked Flush 0 Done Sent Done false
2085 1531 Acked Read 3 Done Sent Done false
2086 1532 Acked Read 6 Done Sent Done false
2087 1533 Acked Write 5 Done Sent Done false
2088 1534 Acked Flush 0 Done Sent Done false
2089 1535 Acked Write 1 Done Sent Done false
2090 1536 Acked Write 4 Done Sent Done false
2091 1537 Acked Write 9 Done Sent Done false
2092 1538 Acked Write 2 Done Sent Done false
2093 1539 Acked Read 10 Done Sent Done false
2094 1540 Acked Read 7 Done Sent Done false
2095 1541 Acked Write 7 Done Sent Done false
2096 1542 Acked Read 5 Done Sent Done false
2097 1543 Acked Read 6 Done Sent Done false
2098 1544 Acked Write 5 Done Sent Done false
2099 1545 Acked Read 5 Done Sent Done false
2100 1546 Acked Write 1 Done Sent Done false
2101 1547 Acked Write 10 Done Sent Done false
2102 1548 Acked Flush 0 Done Sent Done false
2103 1549 Acked Write 6 Done Sent Done false
2104 1550 Acked Write 8 Done Sent Done false
2105 1551 Acked Write 6 Done Sent Done false
2106 1552 Acked Read 2 Done Sent Done false
2107 1553 Acked Flush 0 Done Sent Done false
2108 1554 Acked Read 6 Done Sent Done false
2109 1555 Acked Write 1 Done Sent Done false
2110 1556 Acked Flush 0 Done Sent Done false
2111 1557 Acked Read 2 Done Sent Done false
2112 1558 Acked Write 6 Done Sent Done false
2113 1559 Acked Write 6 Done Sent Done false
2114 1560 Acked Write 6 Done Sent Done false
2115 1561 Acked Read 6 Done Sent Done false
2116 1562 Acked Write 4 Done Sent Done false
2117 1563 Acked Read 7 Done Sent Done false
2118 1564 Acked Write 1 Done Sent Done false
2119 1565 Acked Read 7 Done Sent Done false
2120 1566 Acked Read 2 Done Sent Done false
2121 1567 Acked Flush 0 Done Sent Done false
2122 1568 Acked Write 10 Done Sent Done false
2123 1569 Acked Read 3 Done Sent Done false
2124 1570 Acked Read 2 Done Sent Done false
2125 1571 Acked Read 8 Done Sent Done false
2126 1572 Acked Read 3 Done Sent Done false
2127 1573 Acked Flush 0 Done Sent Done false
2128 1574 Acked Read 9 Done Sent Done false
2129 1575 Acked Read 9 Done Sent Done false
2130 1576 Acked Read 2 Done Sent Done false
2131 1577 Acked Read 7 Done Sent Done false
2132 1578 Acked Read 6 Done Sent Done false
2133 1579 Acked Write 6 Done Sent Done false
2134 1580 Acked Read 8 Done Sent Done false
2135 1581 Acked Write 7 Done Sent Done false
2136 1582 Acked Write 4 Done Sent Done false
2137 1583 Acked Write 5 Done Sent Done false
2138 1584 Acked Write 2 Done Sent Done false
2139 1585 Acked Flush 0 Done Sent Done false
2140 1586 Acked Write 9 Done Sent Done false
2141 1587 Acked Write 10 Done Sent Done false
2142 1588 Acked Flush 0 Done Sent Done false
2143 1589 Acked Read 2 Done Sent Done false
2144 1590 Acked Read 8 Done Sent Done false
2145 1591 Acked Write 6 Done Sent Done false
2146 1592 Acked Flush 0 Done Sent Done false
2147 1593 Acked Read 10 Done Sent Done false
2148 1594 Acked Read 10 Done Sent Done false
2149 1595 Acked Read 1 Done Sent Done false
2150 1596 Acked Write 2 Done Sent Done false
2151 1597 Acked Write 2 Done Sent Done false
2152 1598 Acked Read 6 Done Sent Done false
2153 1599 Acked Write 9 Done Sent Done false
2154 1600 Acked Read 10 Done Sent Done false
2155 1601 Acked Read 9 Done Sent Done false
2156 1602 Acked Write 5 Done Sent Done false
2157 1603 Acked Flush 0 Done Sent Done false
2158 STATES DS:0 DS:1 DS:2 TOTAL
2159 Sent 0 302 0 302
2160 Done 302 0 302 604
2161 Skipped 0 0 0 0
2162 Error 0 0 0 0
2163 Last Flush: 1603 1301 1603
2164 Downstairs last five completed: 1301 1300 1299 1298 1297
2165 Upstairs last five completed: 1603 1602 1601 1600 1599
2166 CLIENT: Up:0 ds:302 act:2
21672024-12-09T15:08:17.358ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21682024-12-09T15:08:17.358ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34814) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21692024-12-09T15:08:17.359ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21702024-12-09T15:08:17.360ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21712024-12-09T15:08:17.360ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21722024-12-09T15:08:17.360ZINFOcrucible: send last flush ID to this DS: 1301 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21732024-12-09T15:08:17.360ZINFOcrucible: Replied this last flush ID: 1301 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21742024-12-09T15:08:17.360ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21752024-12-09T15:08:17.360ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21762024-12-09T15:08:17.360ZINFOcrucible: [1] client re-new 302 jobs since flush 1301 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21772024-12-09T15:08:17.361ZINFOcrucible: [1] Replayed 302 jobs since flush: 1301 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2178 ----------------------------------------------------------------
2179 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2180 Downstairs last five completed: 1603 1602 1601 1600 1599
2181 Upstairs last five completed: 1603 1602 1601 1600 1599
2182 CLIENT: Up:0 ds:0 act:3
2183 Replay: All jobs finished
2184 Main task has work to do, go find it
2185 got dsc Stop(2)
2186 stop 2
2187 Dec 09 15:08:19.593 INFO request completed, latency_us: 358, response_code: 204[8830] Got stop action so:false kr:false
2188 , uri: /stop/cid/2, method: POST, req_id: 2e7f8de3-db80-4733-aacb-980597d81db6, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
21892024-12-09T15:08:19.594ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21902024-12-09T15:08:19.594ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21912024-12-09T15:08:19.594ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2192 Dec 09 15:08:19.594 INFO request completed, latency_us: 155, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1b9dddde-dfcf-4954-b2d7-d4cca52a70bf, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
21932024-12-09T15:08:19.594ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21942024-12-09T15:08:19.594ZINFOcrucible: [2] Transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21952024-12-09T15:08:19.594ZINFOcrucible: client task is exiting = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
2196 [8830] Exited with: ExitStatus(unix_wait_status(9))
2197 [8830][2] reports Exit
2198 Dec 09 15:08:23.596 INFO request completed, latency_us: 233, response_code: 200, uri: /state/cid/2, method: GET, req_id: 356218ca-1f69-4d3f-b3f3-0eb266d05659, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
2199 001/300 Write block 136 len 20480 data: 3 3 3 3 3
2200 002/300 Write block 3286 len 36864 data: 2 2 2 2 2 2 2 2 2
2201 003/300 Flush
2202 004/300 Write block 3832 len 24576 data: 2 2 2 2 2 2
2203 005/300 Read block 205 len 12288
2204 006/300 Read block 2376 len 16384
2205 007/300 Read block 1057 len 40960
2206 008/300 Read block 2902 len 20480
2207 009/300 Flush
2208 010/300 Read block 1397 len 8192
2209 011/300 Read block 4490 len 36864
2210 012/300 Read block 1367 len 20480
2211 013/300 Read block 1426 len 20480
2212 014/300 Read block 3251 len 40960
2213 015/300 Read block 2506 len 16384
2214 016/300 Write block 4601 len 40960 data: 4 3 2 2 2 2 2 2 2 2
2215 017/300 Write block 1112 len 20480 data: 2 2 2 2 2
2216 018/300 Read block 3678 len 28672
2217 019/300 Write block 4505 len 32768 data: 2 2 2 2 2 2 2 2
2218 020/300 Flush
2219 021/300 Write block 239 len 40960 data: 2 2 2 2 2 2 3 3 3 3
2220 022/300 Read block 2340 len 36864
2221 023/300 Write block 3175 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2222 024/300 Write block 577 len 36864 data: 2 2 2 2 2 2 2 2 2
2223 025/300 Write block 3758 len 36864 data: 2 2 2 2 3 3 3 3 3
2224 026/300 Read block 2953 len 40960
2225 027/300 Write block 3100 len 24576 data: 2 2 2 2 2 2
2226 028/300 Read block 3471 len 8192
2227 029/300 Write block 2116 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2228 030/300 Read block 4117 len 28672
2229 031/300 Read block 1221 len 8192
2230 032/300 Write block 3359 len 24576 data: 2 2 3 3 3 3
2231 033/300 Flush
2232 034/300 Write block 2232 len 16384 data: 3 3 3 3
2233 035/300 Write block 3089 len 24576 data: 2 2 2 2 2 2
2234 036/300 Write block 1852 len 16384 data: 2 2 2 2
2235 037/300 Write block 1141 len 8192 data: 2 2
2236 038/300 Write block 4156 len 36864 data: 2 2 3 3 2 2 2 2 2
2237 039/300 Write block 4395 len 28672 data: 2 2 2 2 2 2 2
2238 040/300 Write block 2222 len 16384 data: 2 2 2 2
2239 041/300 Read block 3749 len 16384
2240 042/300 Write block 3709 len 20480 data: 2 2 2 2 2
2241 043/300 Read block 1061 len 36864
2242 044/300 Write block 3331 len 28672 data: 2 2 2 2 2 2 2
2243 045/300 Write block 1561 len 16384 data: 3 3 3 3
2244 046/300 Read block 1208 len 20480
2245 047/300 Write block 3801 len 32768 data: 2 2 2 2 2 2 3 3
2246 048/300 Read block 205 len 8192
2247 049/300 Flush
2248 050/300 Write block 926 len 12288 data: 2 2 2
2249 051/300 Flush
2250 052/300 Read block 1161 len 24576
2251 053/300 Write block 1363 len 8192 data: 2 2
2252 054/300 Read block 1868 len 24576
2253 055/300 Write block 801 len 16384 data: 3 3 3 3
2254 056/300 Write block 446 len 36864 data: 2 2 2 2 2 2 2 2 2
2255 057/300 Read block 2228 len 40960
2256 058/300 Read block 261 len 32768
2257 059/300 Read block 4132 len 16384
2258 060/300 Flush
2259 061/300 Flush
2260 062/300 Write block 4991 len 24576 data: 3 3 3 3 3 2
2261 063/300 Flush
2262 064/300 Read block 2743 len 20480
2263 065/300 Write block 4963 len 8192 data: 2 2
2264 066/300 Read block 1528 len 12288
2265 067/300 Write block 4713 len 8192 data: 2 2
2266 068/300 Write block 4583 len 4096 data: 3
2267 069/300 Read block 301 len 4096
2268 070/300 Read block 1968 len 4096
2269 071/300 Read block 3947 len 20480
2270 072/300 Flush
2271 073/300 Read block 3756 len 40960
2272 074/300 Write block 4439 len 24576 data: 2 2 2 2 3 3
2273 075/300 Write block 2938 len 8192 data: 2 3
2274 076/300 Write block 4744 len 32768 data: 2 2 2 2 2 2 2 2
2275 077/300 Write block 2135 len 36864 data: 3 3 3 3 2 2 3 2 2
2276 078/300 Read block 259 len 32768
2277 079/300 Read block 2716 len 40960
2278 080/300 Write block 219 len 40960 data: 3 2 2 2 2 2 2 2 3 3
2279 081/300 Read block 2781 len 36864
2280 082/300 Write block 4378 len 24576 data: 5 5 5 5 3 3
2281 083/300 Read block 323 len 36864
2282 084/300 Write block 3711 len 24576 data: 3 3 3 2 2 2
2283 085/300 Read block 1028 len 40960
2284 086/300 Write block 4572 len 12288 data: 2 2 3
2285 087/300 Write block 3799 len 16384 data: 2 2 3 3
2286 088/300 Read block 4290 len 32768
2287 089/300 Write block 1132 len 12288 data: 3 3 3
2288 090/300 Read block 2125 len 32768
2289 091/300 Write block 853 len 12288 data: 3 3 3
2290 092/300 Write block 4461 len 28672 data: 2 2 2 2 2 2 2
2291 093/300 Write block 4280 len 4096 data: 2
2292 094/300 Read block 4737 len 16384
2293 095/300 Read block 1593 len 28672
2294 096/300 Write block 603 len 16384 data: 2 2 2 2
2295 097/300 Read block 3835 len 4096
2296 098/300 Write block 2865 len 40960 data: 2 2 3 3 3 3 3 3 3 2
2297 099/300 Read block 2852 len 12288
2298 100/300 Read block 3365 len 12288
2299 101/300 Write block 1289 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2300 102/300 Read block 4557 len 24576
2301 103/300 Read block 3860 len 28672
2302 104/300 Write block 3748 len 28672 data: 2 2 2 2 2 2 2
2303 105/300 Read block 849 len 28672
2304 106/300 Write block 122 len 28672 data: 2 2 2 2 2 2 2
2305 107/300 Read block 2260 len 4096
2306 108/300 Read block 3780 len 24576
2307 109/300 Read block 479 len 28672
2308 110/300 Read block 306 len 24576
2309 111/300 Write block 4963 len 8192 data: 3 3
2310 112/300 Read block 2705 len 4096
2311 113/300 Read block 4360 len 16384
2312 114/300 Write block 1543 len 24576 data: 2 2 2 2 2 2
2313 115/300 Write block 1407 len 40960 data: 3 3 2 2 2 2 2 2 2 2
2314 116/300 Read block 119 len 36864
2315 117/300 Write block 3269 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2316 118/300 Write block 4171 len 8192 data: 2 2
2317 119/300 Read block 3923 len 16384
2318 120/300 Read block 367 len 24576
2319 121/300 Read block 2771 len 36864
2320 122/300 Write block 2717 len 12288 data: 2 2 2
2321 123/300 Write block 2791 len 20480 data: 2 2 2 2 3
2322 124/300 Read block 3749 len 20480
2323 125/300 Read block 3832 len 36864
2324 126/300 Read block 2764 len 16384
2325 127/300 Write block 425 len 8192 data: 2 2
2326 128/300 Write block 1742 len 4096 data: 3
2327 129/300 Flush
2328 130/300 Read block 1302 len 8192
2329 131/300 Write block 1094 len 12288 data: 2 2 2
2330 132/300 Flush
2331 133/300 Write block 2296 len 4096 data: 2
2332 134/300 Write block 4856 len 20480 data: 2 2 2 2 2
2333 135/300 Write block 3243 len 20480 data: 2 2 2 3 3
2334 136/300 Read block 1287 len 36864
2335 137/300 Read block 1892 len 32768
2336 138/300 Read block 1303 len 40960
2337 139/300 Read block 4523 len 40960
2338 140/300 Write block 4912 len 36864 data: 2 2 2 2 2 2 2 2 2
2339 141/300 Read block 786 len 40960
2340 142/300 Read block 2024 len 16384
2341 143/300 Read block 957 len 40960
2342 144/300 Read block 3041 len 28672
2343 145/300 Write block 3561 len 4096 data: 3
2344 146/300 Read block 133 len 16384
2345 147/300 Write block 2616 len 36864 data: 2 2 3 3 3 3 3 3 3
2346 148/300 Read block 3191 len 16384
2347 149/300 Write block 2698 len 20480 data: 2 2 2 2 3
2348 150/300 Write block 1362 len 36864 data: 2 3 3 2 2 2 3 3 4
2349 151/300 Read block 4020 len 4096
2350 152/300 Flush
2351 153/300 Read block 4762 len 4096
2352 154/300 Read block 1425 len 40960
2353 155/300 Flush
2354 156/300 Write block 1479 len 16384 data: 2 2 2 2
2355 157/300 Write block 4839 len 28672 data: 3 4 4 3 3 3 2
2356 158/300 Read block 4744 len 32768
2357 159/300 Write block 1501 len 12288 data: 4 4 4
2358 160/300 Write block 45 len 24576 data: 2 2 2 2 2 2
2359 161/300 Read block 2850 len 24576
2360 162/300 Read block 2105 len 36864
2361 163/300 Read block 3429 len 16384
2362 164/300 Read block 569 len 32768
2363 165/300 Write block 4346 len 24576 data: 2 2 2 2 2 2
2364 166/300 Write block 4814 len 4096 data: 4
2365 167/300 Read block 4370 len 24576
2366 168/300 Flush
2367 169/300 Read block 274 len 16384
2368 170/300 Write block 1560 len 28672 data: 3 4 4 4 4 2 2
2369 171/300 Flush
2370 172/300 Write block 1039 len 32768 data: 2 2 2 2 2 2 2 2
2371 173/300 Write block 3074 len 32768 data: 2 2 2 2 2 2 2 2
2372 174/300 Flush
2373 175/300 Read block 4712 len 8192
2374 176/300 Read block 111 len 8192
2375 177/300 Read block 1016 len 40960
2376 178/300 Read block 4660 len 4096
2377 179/300 Read block 2362 len 16384
2378 180/300 Read block 636 len 32768
2379 181/300 Flush
2380 182/300 Read block 3835 len 28672
2381 183/300 Write block 1952 len 8192 data: 2 2
2382 184/300 Write block 2987 len 40960 data: 4 4 3 2 2 2 2 2 2 2
2383 185/300 Read block 1206 len 16384
2384 186/300 Write block 1668 len 16384 data: 2 2 2 2
2385 187/300 Read block 2279 len 24576
2386 188/300 Flush
2387 189/300 Write block 1374 len 4096 data: 3
2388 190/300 Read block 33 len 4096
2389 191/300 Read block 1665 len 4096
2390 192/300 Write block 185 len 32768 data: 2 2 2 3 3 3 3 3
2391 193/300 Read block 4526 len 16384
2392 194/300 Read block 497 len 8192
2393 195/300 Write block 178 len 12288 data: 2 2 2
2394 196/300 Read block 2142 len 8192
2395 197/300 Read block 323 len 28672
2396 198/300 Read block 537 len 24576
2397 199/300 Write block 4694 len 32768 data: 2 2 2 3 3 3 3 3
2398 200/300 Flush
2399 201/300 Read block 441 len 20480
2400 202/300 Read block 3625 len 4096
2401 203/300 Read block 694 len 12288
2402 204/300 Read block 1949 len 4096
2403 205/300 Write block 1528 len 28672 data: 3 3 3 3 4 3 3
2404 206/300 Read block 765 len 40960
2405 207/300 Read block 4823 len 28672
2406 208/300 Write block 3657 len 32768 data: 2 2 2 2 2 2 2 2
2407 209/300 Read block 4146 len 28672
2408 210/300 Read block 3053 len 20480
2409 211/300 Flush
2410 212/300 Read block 2849 len 40960
2411 213/300 Read block 3933 len 20480
2412 214/300 Read block 1261 len 4096
2413 215/300 Read block 10 len 4096
2414 216/300 Read block 4596 len 16384
2415 217/300 Write block 3530 len 8192 data: 2 2
2416 218/300 Read block 2038 len 36864
2417 219/300 Write block 4443 len 8192 data: 4 4
2418 220/300 Read block 2835 len 36864
2419 221/300 Read block 1127 len 20480
2420 222/300 Read block 3854 len 40960
2421 223/300 Write block 696 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2422 224/300 Read block 1304 len 4096
2423 225/300 Read block 4486 len 32768
2424 226/300 Write block 4321 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2425 227/300 Read block 4576 len 8192
2426 228/300 Write block 2088 len 16384 data: 2 2 2 2
2427 229/300 Read block 358 len 12288
2428 230/300 Write block 3549 len 24576 data: 2 2 2 2 2 2
2429 231/300 Read block 3866 len 32768
2430 232/300 Flush
2431 233/300 Write block 4535 len 8192 data: 2 2
2432 234/300 Read block 2571 len 24576
2433 235/300 Read block 748 len 16384
2434 236/300 Write block 3518 len 16384 data: 3 3 4 4
2435 237/300 Read block 3786 len 4096
2436 238/300 Write block 1335 len 20480 data: 2 2 2 2 3
2437 239/300 Write block 3487 len 16384 data: 2 2 2 2
2438 240/300 Read block 4990 len 20480
2439 241/300 Read block 2036 len 20480
2440 242/300 Read block 95 len 12288
2441 243/300 Write block 278 len 24576 data: 2 2 2 2 2 2
2442 244/300 Flush
2443 245/300 Write block 2218 len 40960 data: 3 2 2 2 3 3 3 3 2 2
2444 246/300 Write block 1997 len 36864 data: 2 2 2 2 2 2 2 2 2
2445 247/300 Write block 1219 len 36864 data: 2 2 2 2 2 2 2 2 2
2446 248/300 Read block 4523 len 28672
2447 249/300 Flush
2448 250/300 Write block 3483 len 28672 data: 2 2 2 2 3 3 3
2449 251/300 Read block 4122 len 28672
2450 252/300 Read block 3149 len 8192
2451 253/300 Write block 778 len 40960 data: 2 2 2 2 2 2 3 3 3 3
2452 254/300 Flush
2453 255/300 Read block 2477 len 32768
2454 256/300 Write block 304 len 8192 data: 2 2
2455 257/300 Read block 4865 len 16384
2456 258/300 Read block 2085 len 40960
2457 259/300 Read block 3230 len 8192
2458 260/300 Write block 2140 len 36864 data: 3 4 3 3 2 2 2 2 2
2459 261/300 Read block 32 len 32768
2460 262/300 Read block 2222 len 8192
2461 263/300 Write block 133 len 40960 data: 4 3 3 4 4 4 4 4 3 3
2462 264/300 Read block 4464 len 40960
2463 265/300 Write block 1291 len 20480 data: 3 3 3 3 3
2464 266/300 Write block 3336 len 36864 data: 3 3 2 2 2 3 3 3 3
2465 267/300 Read block 2174 len 20480
2466 268/300 Read block 30 len 36864
2467 269/300 Flush
2468 270/300 Write block 3439 len 28672 data: 2 2 2 2 2 2 2
2469 271/300 Read block 750 len 32768
2470 272/300 Read block 1514 len 4096
2471 273/300 Read block 3445 len 16384
2472 274/300 Read block 2097 len 36864
2473 275/300 Write block 1148 len 28672 data: 2 2 2 2 2 2 2
2474 276/300 Write block 2128 len 32768 data: 2 2 3 3 3 3 3 4
2475 277/300 Write block 4375 len 4096 data: 3
2476 278/300 Write block 3955 len 28672 data: 2 2 2 2 2 2 3
2477 279/300 Read block 1758 len 40960
2478 280/300 Flush
2479 281/300 Write block 1447 len 28672 data: 3 2 2 2 2 2 2
2480 282/300 Write block 4879 len 12288 data: 2 2 2
2481 283/300 Read block 2311 len 32768
2482 284/300 Read block 218 len 12288
2483 285/300 Read block 1544 len 40960
2484 286/300 Read block 3500 len 4096
2485 287/300 Read block 2022 len 36864
2486 288/300 Read block 4607 len 12288
2487 289/300 Write block 1026 len 8192 data: 2 2
2488 290/300 Read block 4379 len 20480
2489 291/300 Read block 4390 len 12288
2490 292/300 Write block 3540 len 24576 data: 2 2 2 2 2 2
2491 293/300 Write block 3434 len 36864 data: 3 3 2 2 2 3 3 3 3
2492 294/300 Flush
2493 295/300 Write block 288 len 8192 data: 2 2
2494 296/300 Write block 4168 len 36864 data: 2 2 2 3 3 2 2 2 2
2495 297/300 Flush
2496 298/300 Read block 1932 len 40960
2497 299/300 Read block 4484 len 28672
2498 300/300 Read block 449 len 36864
2499 301/300 Write block 3024 len 12288 data: 2 3 2
2500 Main task has work to do, go find it
2501 got dsc Start(2)
2502 start 2
2503 Dec 09 15:08:23.841 INFO request completed, latency_us: 239, response_code: 204, uri: /start/cid/2, method: POST, req_id: bd772672-43cd-4e40-b20a-da3334b9ac43, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
2504 Replay: started 2, returned:Ok(())
2505 ----------------------------------------------------------------
2506 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2507 States: Active Active Offline
2508 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2509 1604 Acked Write 5 Done Done Sent false
2510 1605 Acked Write 9 Done Done Sent false
2511 1606 Acked Flush 0 Done Done Sent false
2512 1607 Acked Write 6 Done Done Sent false
2513 1608 Acked Read 3 Done Done Sent false
2514 1609 Acked Read 4 Done Done Sent false
2515 1610 Acked Read 10 Done Done Sent false
2516 1611 Acked Read 5 Done Done Sent false
2517 1612 Acked Flush 0 Done Done Sent false
2518 1613 Acked Read 2 Done Done Sent false
2519 1614 Acked Read 9 Done Done Sent false
2520 1615 Acked Read 5 Done Done Sent false
2521 1616 Acked Read 5 Done Done Sent false
2522 1617 Acked Read 10 Done Done Sent false
2523 1618 Acked Read 4 Done Done Sent false
2524 1619 Acked Write 10 Done Done Sent false
2525 1620 Acked Write 5 Done Done Sent false
2526 1621 Acked Read 7 Done Done Sent false
2527 1622 Acked Write 8 Done Done Sent false
2528 1623 Acked Flush 0 Done Done Sent false
2529 1624 Acked Write 10 Done Done Sent false
2530 1625 Acked Read 9 Done Done Sent false
2531 1626 Acked Write 10 Done Done Sent false
2532 1627 Acked Write 9 Done Done Sent false
2533 1628 Acked Write 9 Done Done Sent false
2534 1629 Acked Read 10 Done Done Sent false
2535 1630 Acked Write 6 Done Done Sent false
2536 1631 Acked Read 2 Done Done Sent false
2537 1632 Acked Write 10 Done Done Sent false
2538 1633 Acked Read 7 Done Done Sent false
2539 1634 Acked Read 2 Done Done Sent false
2540 1635 Acked Write 6 Done Done Sent false
2541 1636 Acked Flush 0 Done Done Sent false
2542 1637 Acked Write 4 Done Done Sent false
2543 1638 Acked Write 6 Done Done Sent false
2544 1639 Acked Write 4 Done Done Sent false
2545 1640 Acked Write 2 Done Done Sent false
2546 1641 Acked Write 9 Done Done Sent false
2547 1642 Acked Write 7 Done Done Sent false
2548 1643 Acked Write 4 Done Done Sent false
2549 1644 Acked Read 4 Done Done Sent false
2550 1645 Acked Write 5 Done Done Sent false
2551 1646 Acked Read 9 Done Done Sent false
2552 1647 Acked Write 7 Done Done Sent false
2553 1648 Acked Write 4 Done Done Sent false
2554 1649 Acked Read 5 Done Done Sent false
2555 1650 Acked Write 8 Done Done Sent false
2556 1651 Acked Read 2 Done Done Sent false
2557 1652 Acked Flush 0 Done Done Sent false
2558 1653 Acked Write 3 Done Done Sent false
2559 1654 Acked Flush 0 Done Done Sent false
2560 1655 Acked Read 6 Done Done Sent false
2561 1656 Acked Write 2 Done Done Sent false
2562 1657 Acked Read 6 Done Done Sent false
2563 1658 Acked Write 4 Done Done Sent false
2564 1659 Acked Write 9 Done Done Sent false
2565 1660 Acked Read 10 Done Done Sent false
2566 1661 Acked Read 8 Done Done Sent false
2567 1662 Acked Read 4 Done Done Sent false
2568 1663 Acked Flush 0 Done Done Sent false
2569 1664 Acked Flush 0 Done Done Sent false
2570 1665 Acked Write 6 Done Done Sent false
2571 1666 Acked Flush 0 Done Done Sent false
2572 1667 Acked Read 5 Done Done Sent false
2573 1668 Acked Write 2 Done Done Sent false
2574 1669 Acked Read 3 Done Done Sent false
2575 1670 Acked Write 2 Done Done Sent false
2576 1671 Acked Write 1 Done Done Sent false
2577 1672 Acked Read 1 Done Done Sent false
2578 1673 Acked Read 1 Done Done Sent false
2579 1674 Acked Read 5 Done Done Sent false
2580 1675 Acked Flush 0 Done Done Sent false
2581 1676 Acked Read 10 Done Done Sent false
2582 1677 Acked Write 6 Done Done Sent false
2583 1678 Acked Write 2 Done Done Sent false
2584 1679 Acked Write 8 Done Done Sent false
2585 1680 Acked Write 9 Done Done Sent false
2586 1681 Acked Read 8 Done Done Sent false
2587 1682 Acked Read 10 Done Done Sent false
2588 1683 Acked Write 10 Done Done Sent false
2589 1684 Acked Read 9 Done Done Sent false
2590 1685 Acked Write 6 Done Done Sent false
2591 1686 Acked Read 9 Done Done Sent false
2592 1687 Acked Write 6 Done Done Sent false
2593 1688 Acked Read 10 Done Done Sent false
2594 1689 Acked Write 3 Done Done Sent false
2595 1690 Acked Write 4 Done Done Sent false
2596 1691 Acked Read 8 Done Done Sent false
2597 1692 Acked Write 3 Done Done Sent false
2598 1693 Acked Read 8 Done Done Sent false
2599 1694 Acked Write 3 Done Done Sent false
2600 1695 Acked Write 7 Done Done Sent false
2601 1696 Acked Write 1 Done Done Sent false
2602 1697 Acked Read 4 Done Done Sent false
2603 1698 Acked Read 7 Done Done Sent false
2604 1699 Acked Write 4 Done Done Sent false
2605 1700 Acked Read 1 Done Done Sent false
2606 1701 Acked Write 10 Done Done Sent false
2607 1702 Acked Read 3 Done Done Sent false
2608 1703 Acked Read 3 Done Done Sent false
2609 1704 Acked Write 10 Done Done Sent false
2610 1705 Acked Read 6 Done Done Sent false
2611 1706 Acked Read 7 Done Done Sent false
2612 1707 Acked Write 7 Done Done Sent false
2613 1708 Acked Read 7 Done Done Sent false
2614 1709 Acked Write 7 Done Done Sent false
2615 1710 Acked Read 1 Done Done Sent false
2616 1711 Acked Read 6 Done Done Sent false
2617 1712 Acked Read 7 Done Done Sent false
2618 1713 Acked Read 6 Done Done Sent false
2619 1714 Acked Write 2 Done Done Sent false
2620 1715 Acked Read 1 Done Done Sent false
2621 1716 Acked Read 4 Done Done Sent false
2622 1717 Acked Write 6 Done Done Sent false
2623 1718 Acked Write 10 Done Done Sent false
2624 1719 Acked Read 9 Done Done Sent false
2625 1720 Acked Write 10 Done Done Sent false
2626 1721 Acked Write 2 Done Done Sent false
2627 1722 Acked Read 4 Done Done Sent false
2628 1723 Acked Read 6 Done Done Sent false
2629 1724 Acked Read 9 Done Done Sent false
2630 1725 Acked Write 3 Done Done Sent false
2631 1726 Acked Write 5 Done Done Sent false
2632 1727 Acked Read 5 Done Done Sent false
2633 1728 Acked Read 9 Done Done Sent false
2634 1729 Acked Read 4 Done Done Sent false
2635 1730 Acked Write 2 Done Done Sent false
2636 1731 Acked Write 1 Done Done Sent false
2637 1732 Acked Flush 0 Done Done Sent false
2638 1733 Acked Read 2 Done Done Sent false
2639 1734 Acked Write 3 Done Done Sent false
2640 1735 Acked Flush 0 Done Done Sent false
2641 1736 Acked Write 1 Done Done Sent false
2642 1737 Acked Write 5 Done Done Sent false
2643 1738 Acked Write 5 Done Done Sent false
2644 1739 Acked Read 9 Done Done Sent false
2645 1740 Acked Read 8 Done Done Sent false
2646 1741 Acked Read 10 Done Done Sent false
2647 1742 Acked Read 10 Done Done Sent false
2648 1743 Acked Write 9 Done Done Sent false
2649 1744 Acked Read 10 Done Done Sent false
2650 1745 Acked Read 4 Done Done Sent false
2651 1746 Acked Read 10 Done Done Sent false
2652 1747 Acked Read 7 Done Done Sent false
2653 1748 Acked Write 1 Done Done Sent false
2654 1749 Acked Read 4 Done Done Sent false
2655 1750 Acked Write 9 Done Done Sent false
2656 1751 Acked Read 4 Done Done Sent false
2657 1752 Acked Write 5 Done Done Sent false
2658 1753 Acked Write 9 Done Done Sent false
2659 1754 Acked Read 1 Done Done Sent false
2660 1755 Acked Flush 0 Done Done Sent false
2661 1756 Acked Read 1 Done Done Sent false
2662 1757 Acked Read 10 Done Done Sent false
2663 1758 Acked Flush 0 Done Done Sent false
2664 1759 Acked Write 4 Done Done Sent false
2665 1760 Acked Write 7 Done Done Sent false
2666 1761 Acked Read 8 Done Done Sent false
2667 1762 Acked Write 3 Done Done Sent false
2668 1763 Acked Write 6 Done Done Sent false
2669 1764 Acked Read 6 Done Done Sent false
2670 1765 Acked Read 9 Done Done Sent false
2671 1766 Acked Read 4 Done Done Sent false
2672 1767 Acked Read 8 Done Done Sent false
2673 1768 Acked Write 6 Done Done Sent false
2674 1769 Acked Write 1 Done Done Sent false
2675 1770 Acked Read 6 Done Done Sent false
2676 1771 Acked Flush 0 Done Done Sent false
2677 1772 Acked Read 4 Done Done Sent false
2678 1773 Acked Write 7 Done Done Sent false
2679 1774 Acked Flush 0 Done Done Sent false
2680 1775 Acked Write 8 Done Done Sent false
2681 1776 Acked Write 8 Done Done Sent false
2682 1777 Acked Flush 0 Done Done Sent false
2683 1778 Acked Read 2 Done Done Sent false
2684 1779 Acked Read 2 Done Done Sent false
2685 1780 Acked Read 10 Done Done Sent false
2686 1781 Acked Read 1 Done Done Sent false
2687 1782 Acked Read 4 Done Done Sent false
2688 1783 Acked Read 8 Done Done Sent false
2689 1784 Acked Flush 0 Done Done Sent false
2690 1785 Acked Read 7 Done Done Sent false
2691 1786 Acked Write 2 Done Done Sent false
2692 1787 Acked Write 10 Done Done Sent false
2693 1788 Acked Read 4 Done Done Sent false
2694 1789 Acked Write 4 Done Done Sent false
2695 1790 Acked Read 6 Done Done Sent false
2696 1791 Acked Flush 0 Done Done Sent false
2697 1792 Acked Write 1 Done Done Sent false
2698 1793 Acked Read 1 Done Done Sent false
2699 1794 Acked Read 1 Done Done Sent false
2700 1795 Acked Write 8 Done Done Sent false
2701 1796 Acked Read 4 Done Done Sent false
2702 1797 Acked Read 2 Done Done Sent false
2703 1798 Acked Write 3 Done Done Sent false
2704 1799 Acked Read 2 Done Done Sent false
2705 1800 Acked Read 7 Done Done Sent false
2706 1801 Acked Read 6 Done Done Sent false
2707 1802 Acked Write 8 Done Done Sent false
2708 1803 Acked Flush 0 Done Done Sent false
2709 1804 Acked Read 5 Done Done Sent false
2710 1805 Acked Read 1 Done Done Sent false
2711 1806 Acked Read 3 Done Done Sent false
2712 1807 Acked Read 1 Done Done Sent false
2713 1808 Acked Write 7 Done Done Sent false
2714 1809 Acked Read 10 Done Done Sent false
2715 1810 Acked Read 7 Done Done Sent false
2716 1811 Acked Write 8 Done Done Sent false
2717 1812 Acked Read 7 Done Done Sent false
2718 1813 Acked Read 5 Done Done Sent false
2719 1814 Acked Flush 0 Done Done Sent false
2720 1815 Acked Read 10 Done Done Sent false
2721 1816 Acked Read 5 Done Done Sent false
2722 1817 Acked Read 1 Done Done Sent false
2723 1818 Acked Read 1 Done Done Sent false
2724 1819 Acked Read 4 Done Done Sent false
2725 1820 Acked Write 2 Done Done Sent false
2726 1821 Acked Read 9 Done Done Sent false
2727 1822 Acked Write 2 Done Done Sent false
2728 1823 Acked Read 9 Done Done Sent false
2729 1824 Acked Read 5 Done Done Sent false
2730 1825 Acked Read 10 Done Done Sent false
2731 1826 Acked Write 10 Done Done Sent false
2732 1827 Acked Read 1 Done Done Sent false
2733 1828 Acked Read 8 Done Done Sent false
2734 1829 Acked Write 10 Done Done Sent false
2735 1830 Acked Read 2 Done Done Sent false
2736 1831 Acked Write 4 Done Done Sent false
2737 1832 Acked Read 3 Done Done Sent false
2738 1833 Acked Write 6 Done Done Sent false
2739 1834 Acked Read 8 Done Done Sent false
2740 1835 Acked Flush 0 Done Done Sent false
2741 1836 Acked Write 2 Done Done Sent false
2742 1837 Acked Read 6 Done Done Sent false
2743 1838 Acked Read 4 Done Done Sent false
2744 1839 Acked Write 4 Done Done Sent false
2745 1840 Acked Read 1 Done Done Sent false
2746 1841 Acked Write 5 Done Done Sent false
2747 1842 Acked Write 4 Done Done Sent false
2748 1843 Acked Read 5 Done Done Sent false
2749 1844 Acked Read 5 Done Done Sent false
2750 1845 Acked Read 3 Done Done Sent false
2751 1846 Acked Write 6 Done Done Sent false
2752 1847 Acked Flush 0 Done Done Sent false
2753 1848 Acked Write 10 Done Done Sent false
2754 1849 Acked Write 9 Done Done Sent false
2755 1850 Acked Write 9 Done Done Sent false
2756 1851 Acked Read 7 Done Done Sent false
2757 1852 Acked Flush 0 Done Done Sent false
2758 1853 Acked Write 7 Done Done Sent false
2759 1854 Acked Read 7 Done Done Sent false
2760 1855 Acked Read 2 Done Done Sent false
2761 1856 Acked Write 10 Done Done Sent false
2762 1857 Acked Flush 0 Done Done Sent false
2763 1858 Acked Read 8 Done Done Sent false
2764 1859 Acked Write 2 Done Done Sent false
2765 1860 Acked Read 4 Done Done Sent false
2766 1861 Acked Read 10 Done Done Sent false
2767 1862 Acked Read 2 Done Done Sent false
2768 1863 Acked Write 9 Done Done Sent false
2769 1864 Acked Read 8 Done Done Sent false
2770 1865 Acked Read 2 Done Done Sent false
2771 1866 Acked Write 10 Done Done Sent false
2772 1867 Acked Read 10 Done Done Sent false
2773 1868 Acked Write 5 Done Done Sent false
2774 1869 Acked Write 9 Done Done Sent false
2775 1870 Acked Read 5 Done Done Sent false
2776 1871 Acked Read 9 Done Done Sent false
2777 1872 Acked Flush 0 Done Done Sent false
2778 1873 Acked Write 7 Done Done Sent false
2779 1874 Acked Read 8 Done Done Sent false
2780 1875 Acked Read 1 Done Done Sent false
2781 1876 Acked Read 4 Done Done Sent false
2782 1877 Acked Read 9 Done Done Sent false
2783 1878 Acked Write 7 Done Done Sent false
2784 1879 Acked Write 8 Done Done Sent false
2785 1880 Acked Write 1 Done Done Sent false
2786 1881 Acked Write 7 Done Done Sent false
2787 1882 Acked Read 10 Done Done Sent false
2788 1883 Acked Flush 0 Done Done Sent false
2789 1884 Acked Write 7 Done Done Sent false
2790 1885 Acked Write 3 Done Done Sent false
2791 1886 Acked Read 8 Done Done Sent false
2792 1887 Acked Read 3 Done Done Sent false
2793 1888 Acked Read 10 Done Done Sent false
2794 1889 Acked Read 1 Done Done Sent false
2795 1890 Acked Read 9 Done Done Sent false
2796 1891 Acked Read 3 Done Done Sent false
2797 1892 Acked Write 2 Done Done Sent false
2798 1893 Acked Read 5 Done Done Sent false
2799 1894 Acked Read 3 Done Done Sent false
2800 1895 Acked Write 6 Done Done Sent false
2801 1896 Acked Write 9 Done Done Sent false
2802 1897 Acked Flush 0 Done Done Sent false
2803 1898 Acked Write 2 Done Done Sent false
2804 1899 Acked Write 9 Done Done Sent false
2805 1900 Acked Flush 0 Done Done Sent false
2806 1901 Acked Read 10 Done Done Sent false
2807 1902 Acked Read 7 Done Done Sent false
2808 1903 Acked Read 9 Done Done Sent false
2809 1904 Acked Write 3 Done Done Sent false
2810 STATES DS:0 DS:1 DS:2 TOTAL
2811 Sent 0 0 301 301
2812 Done 301 301 0 602
2813 Skipped 0 0 0 0
2814 Error 0 0 0 0
2815 Last Flush: 1900 1900 1603
2816 Downstairs last five completed: 1603 1602 1601 1600 1599
2817 Upstairs last five completed: 1904 1903 1902 1901 1900
2818 CLIENT: Up:0 ds:301 act:2
2819 [8830] got start action
2820 [8830] I am going to restart
2821 Starting downstairs at port 8830
2822 Make output file at "/tmp/dsc/downstairs-8830.txt"
2823 [8830][2] reports Starting
2824 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1204)
2825 [8830][2] reports Running
2826 ----------------------------------------------------------------
2827 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
2828 States: Active Active Offline
2829 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2830 1604 Acked Write 5 Done Done Sent false
2831 1605 Acked Write 9 Done Done Sent false
2832 1606 Acked Flush 0 Done Done Sent false
2833 1607 Acked Write 6 Done Done Sent false
2834 1608 Acked Read 3 Done Done Sent false
2835 1609 Acked Read 4 Done Done Sent false
2836 1610 Acked Read 10 Done Done Sent false
2837 1611 Acked Read 5 Done Done Sent false
2838 1612 Acked Flush 0 Done Done Sent false
2839 1613 Acked Read 2 Done Done Sent false
2840 1614 Acked Read 9 Done Done Sent false
2841 1615 Acked Read 5 Done Done Sent false
2842 1616 Acked Read 5 Done Done Sent false
2843 1617 Acked Read 10 Done Done Sent false
2844 1618 Acked Read 4 Done Done Sent false
2845 1619 Acked Write 10 Done Done Sent false
2846 1620 Acked Write 5 Done Done Sent false
2847 1621 Acked Read 7 Done Done Sent false
2848 1622 Acked Write 8 Done Done Sent false
2849 1623 Acked Flush 0 Done Done Sent false
2850 1624 Acked Write 10 Done Done Sent false
2851 1625 Acked Read 9 Done Done Sent false
2852 1626 Acked Write 10 Done Done Sent false
2853 1627 Acked Write 9 Done Done Sent false
2854 1628 Acked Write 9 Done Done Sent false
2855 1629 Acked Read 10 Done Done Sent false
2856 1630 Acked Write 6 Done Done Sent false
2857 1631 Acked Read 2 Done Done Sent false
2858 1632 Acked Write 10 Done Done Sent false
2859 1633 Acked Read 7 Done Done Sent false
2860 1634 Acked Read 2 Done Done Sent false
2861 1635 Acked Write 6 Done Done Sent false
2862 1636 Acked Flush 0 Done Done Sent false
2863 1637 Acked Write 4 Done Done Sent false
2864 1638 Acked Write 6 Done Done Sent false
2865 1639 Acked Write 4 Done Done Sent false
2866 1640 Acked Write 2 Done Done Sent false
2867 1641 Acked Write 9 Done Done Sent false
2868 1642 Acked Write 7 Done Done Sent false
2869 1643 Acked Write 4 Done Done Sent false
2870 1644 Acked Read 4 Done Done Sent false
2871 1645 Acked Write 5 Done Done Sent false
2872 1646 Acked Read 9 Done Done Sent false
2873 1647 Acked Write 7 Done Done Sent false
2874 1648 Acked Write 4 Done Done Sent false
2875 1649 Acked Read 5 Done Done Sent false
2876 1650 Acked Write 8 Done Done Sent false
2877 1651 Acked Read 2 Done Done Sent false
2878 1652 Acked Flush 0 Done Done Sent false
2879 1653 Acked Write 3 Done Done Sent false
2880 1654 Acked Flush 0 Done Done Sent false
2881 1655 Acked Read 6 Done Done Sent false
2882 1656 Acked Write 2 Done Done Sent false
2883 1657 Acked Read 6 Done Done Sent false
2884 1658 Acked Write 4 Done Done Sent false
2885 1659 Acked Write 9 Done Done Sent false
2886 1660 Acked Read 10 Done Done Sent false
2887 1661 Acked Read 8 Done Done Sent false
2888 1662 Acked Read 4 Done Done Sent false
2889 1663 Acked Flush 0 Done Done Sent false
2890 1664 Acked Flush 0 Done Done Sent false
2891 1665 Acked Write 6 Done Done Sent false
2892 1666 Acked Flush 0 Done Done Sent false
2893 1667 Acked Read 5 Done Done Sent false
2894 1668 Acked Write 2 Done Done Sent false
2895 1669 Acked Read 3 Done Done Sent false
2896 1670 Acked Write 2 Done Done Sent false
2897 1671 Acked Write 1 Done Done Sent false
2898 1672 Acked Read 1 Done Done Sent false
2899 1673 Acked Read 1 Done Done Sent false
2900 1674 Acked Read 5 Done Done Sent false
2901 1675 Acked Flush 0 Done Done Sent false
2902 1676 Acked Read 10 Done Done Sent false
2903 1677 Acked Write 6 Done Done Sent false
2904 1678 Acked Write 2 Done Done Sent false
2905 1679 Acked Write 8 Done Done Sent false
2906 1680 Acked Write 9 Done Done Sent false
2907 1681 Acked Read 8 Done Done Sent false
2908 1682 Acked Read 10 Done Done Sent false
2909 1683 Acked Write 10 Done Done Sent false
2910 1684 Acked Read 9 Done Done Sent false
2911 1685 Acked Write 6 Done Done Sent false
2912 1686 Acked Read 9 Done Done Sent false
2913 1687 Acked Write 6 Done Done Sent false
2914 1688 Acked Read 10 Done Done Sent false
2915 1689 Acked Write 3 Done Done Sent false
2916 1690 Acked Write 4 Done Done Sent false
2917 1691 Acked Read 8 Done Done Sent false
2918 1692 Acked Write 3 Done Done Sent false
2919 1693 Acked Read 8 Done Done Sent false
2920 1694 Acked Write 3 Done Done Sent false
2921 1695 Acked Write 7 Done Done Sent false
2922 1696 Acked Write 1 Done Done Sent false
2923 1697 Acked Read 4 Done Done Sent false
2924 1698 Acked Read 7 Done Done Sent false
2925 1699 Acked Write 4 Done Done Sent false
2926 1700 Acked Read 1 Done Done Sent false
2927 1701 Acked Write 10 Done Done Sent false
2928 1702 Acked Read 3 Done Done Sent false
2929 1703 Acked Read 3 Done Done Sent false
2930 1704 Acked Write 10 Done Done Sent false
2931 1705 Acked Read 6 Done Done Sent false
2932 1706 Acked Read 7 Done Done Sent false
2933 1707 Acked Write 7 Done Done Sent false
2934 1708 Acked Read 7 Done Done Sent false
2935 1709 Acked Write 7 Done Done Sent false
2936 1710 Acked Read 1 Done Done Sent false
2937 1711 Acked Read 6 Done Done Sent false
2938 1712 Acked Read 7 Done Done Sent false
2939 1713 Acked Read 6 Done Done Sent false
2940 1714 Acked Write 2 Done Done Sent false
2941 1715 Acked Read 1 Done Done Sent false
2942 1716 Acked Read 4 Done Done Sent false
2943 1717 Acked Write 6 Done Done Sent false
2944 1718 Acked Write 10 Done Done Sent false
2945 1719 Acked Read 9 Done Done Sent false
2946 1720 Acked Write 10 Done Done Sent false
2947 1721 Acked Write 2 Done Done Sent false
2948 1722 Acked Read 4 Done Done Sent false
2949 1723 Acked Read 6 Done Done Sent false
2950 1724 Acked Read 9 Done Done Sent false
2951 1725 Acked Write 3 Done Done Sent false
2952 1726 Acked Write 5 Done Done Sent false
2953 1727 Acked Read 5 Done Done Sent false
2954 1728 Acked Read 9 Done Done Sent false
2955 1729 Acked Read 4 Done Done Sent false
2956 1730 Acked Write 2 Done Done Sent false
2957 1731 Acked Write 1 Done Done Sent false
2958 1732 Acked Flush 0 Done Done Sent false
2959 1733 Acked Read 2 Done Done Sent false
2960 1734 Acked Write 3 Done Done Sent false
2961 1735 Acked Flush 0 Done Done Sent false
2962 1736 Acked Write 1 Done Done Sent false
2963 1737 Acked Write 5 Done Done Sent false
2964 1738 Acked Write 5 Done Done Sent false
2965 1739 Acked Read 9 Done Done Sent false
2966 1740 Acked Read 8 Done Done Sent false
2967 1741 Acked Read 10 Done Done Sent false
2968 1742 Acked Read 10 Done Done Sent false
2969 1743 Acked Write 9 Done Done Sent false
2970 1744 Acked Read 10 Done Done Sent false
2971 1745 Acked Read 4 Done Done Sent false
2972 1746 Acked Read 10 Done Done Sent false
2973 1747 Acked Read 7 Done Done Sent false
2974 1748 Acked Write 1 Done Done Sent false
2975 1749 Acked Read 4 Done Done Sent false
2976 1750 Acked Write 9 Done Done Sent false
2977 1751 Acked Read 4 Done Done Sent false
2978 1752 Acked Write 5 Done Done Sent false
2979 1753 Acked Write 9 Done Done Sent false
2980 1754 Acked Read 1 Done Done Sent false
2981 1755 Acked Flush 0 Done Done Sent false
2982 1756 Acked Read 1 Done Done Sent false
2983 1757 Acked Read 10 Done Done Sent false
2984 1758 Acked Flush 0 Done Done Sent false
2985 1759 Acked Write 4 Done Done Sent false
2986 1760 Acked Write 7 Done Done Sent false
2987 1761 Acked Read 8 Done Done Sent false
2988 1762 Acked Write 3 Done Done Sent false
2989 1763 Acked Write 6 Done Done Sent false
2990 1764 Acked Read 6 Done Done Sent false
2991 1765 Acked Read 9 Done Done Sent false
2992 1766 Acked Read 4 Done Done Sent false
2993 1767 Acked Read 8 Done Done Sent false
2994 1768 Acked Write 6 Done Done Sent false
2995 1769 Acked Write 1 Done Done Sent false
2996 1770 Acked Read 6 Done Done Sent false
2997 1771 Acked Flush 0 Done Done Sent false
2998 1772 Acked Read 4 Done Done Sent false
2999 1773 Acked Write 7 Done Done Sent false
3000 1774 Acked Flush 0 Done Done Sent false
3001 1775 Acked Write 8 Done Done Sent false
3002 1776 Acked Write 8 Done Done Sent false
3003 1777 Acked Flush 0 Done Done Sent false
3004 1778 Acked Read 2 Done Done Sent false
3005 1779 Acked Read 2 Done Done Sent false
3006 1780 Acked Read 10 Done Done Sent false
3007 1781 Acked Read 1 Done Done Sent false
3008 1782 Acked Read 4 Done Done Sent false
3009 1783 Acked Read 8 Done Done Sent false
3010 1784 Acked Flush 0 Done Done Sent false
3011 1785 Acked Read 7 Done Done Sent false
3012 1786 Acked Write 2 Done Done Sent false
3013 1787 Acked Write 10 Done Done Sent false
3014 1788 Acked Read 4 Done Done Sent false
3015 1789 Acked Write 4 Done Done Sent false
3016 1790 Acked Read 6 Done Done Sent false
3017 1791 Acked Flush 0 Done Done Sent false
3018 1792 Acked Write 1 Done Done Sent false
3019 1793 Acked Read 1 Done Done Sent false
3020 1794 Acked Read 1 Done Done Sent false
3021 1795 Acked Write 8 Done Done Sent false
3022 1796 Acked Read 4 Done Done Sent false
3023 1797 Acked Read 2 Done Done Sent false
3024 1798 Acked Write 3 Done Done Sent false
3025 1799 Acked Read 2 Done Done Sent false
3026 1800 Acked Read 7 Done Done Sent false
3027 1801 Acked Read 6 Done Done Sent false
3028 1802 Acked Write 8 Done Done Sent false
3029 1803 Acked Flush 0 Done Done Sent false
3030 1804 Acked Read 5 Done Done Sent false
3031 1805 Acked Read 1 Done Done Sent false
3032 1806 Acked Read 3 Done Done Sent false
3033 1807 Acked Read 1 Done Done Sent false
3034 1808 Acked Write 7 Done Done Sent false
3035 1809 Acked Read 10 Done Done Sent false
3036 1810 Acked Read 7 Done Done Sent false
3037 1811 Acked Write 8 Done Done Sent false
3038 1812 Acked Read 7 Done Done Sent false
3039 1813 Acked Read 5 Done Done Sent false
3040 1814 Acked Flush 0 Done Done Sent false
3041 1815 Acked Read 10 Done Done Sent false
3042 1816 Acked Read 5 Done Done Sent false
3043 1817 Acked Read 1 Done Done Sent false
3044 1818 Acked Read 1 Done Done Sent false
3045 1819 Acked Read 4 Done Done Sent false
3046 1820 Acked Write 2 Done Done Sent false
3047 1821 Acked Read 9 Done Done Sent false
3048 1822 Acked Write 2 Done Done Sent false
3049 1823 Acked Read 9 Done Done Sent false
3050 1824 Acked Read 5 Done Done Sent false
3051 1825 Acked Read 10 Done Done Sent false
3052 1826 Acked Write 10 Done Done Sent false
3053 1827 Acked Read 1 Done Done Sent false
3054 1828 Acked Read 8 Done Done Sent false
3055 1829 Acked Write 10 Done Done Sent false
3056 1830 Acked Read 2 Done Done Sent false
3057 1831 Acked Write 4 Done Done Sent false
3058 1832 Acked Read 3 Done Done Sent false
3059 1833 Acked Write 6 Done Done Sent false
3060 1834 Acked Read 8 Done Done Sent false
3061 1835 Acked Flush 0 Done Done Sent false
3062 1836 Acked Write 2 Done Done Sent false
3063 1837 Acked Read 6 Done Done Sent false
3064 1838 Acked Read 4 Done Done Sent false
3065 1839 Acked Write 4 Done Done Sent false
3066 1840 Acked Read 1 Done Done Sent false
3067 1841 Acked Write 5 Done Done Sent false
3068 1842 Acked Write 4 Done Done Sent false
3069 1843 Acked Read 5 Done Done Sent false
3070 1844 Acked Read 5 Done Done Sent false
3071 1845 Acked Read 3 Done Done Sent false
3072 1846 Acked Write 6 Done Done Sent false
3073 1847 Acked Flush 0 Done Done Sent false
3074 1848 Acked Write 10 Done Done Sent false
3075 1849 Acked Write 9 Done Done Sent false
3076 1850 Acked Write 9 Done Done Sent false
3077 1851 Acked Read 7 Done Done Sent false
3078 1852 Acked Flush 0 Done Done Sent false
3079 1853 Acked Write 7 Done Done Sent false
3080 1854 Acked Read 7 Done Done Sent false
3081 1855 Acked Read 2 Done Done Sent false
3082 1856 Acked Write 10 Done Done Sent false
3083 1857 Acked Flush 0 Done Done Sent false
3084 1858 Acked Read 8 Done Done Sent false
3085 1859 Acked Write 2 Done Done Sent false
3086 1860 Acked Read 4 Done Done Sent false
3087 1861 Acked Read 10 Done Done Sent false
3088 1862 Acked Read 2 Done Done Sent false
3089 1863 Acked Write 9 Done Done Sent false
3090 1864 Acked Read 8 Done Done Sent false
3091 1865 Acked Read 2 Done Done Sent false
3092 1866 Acked Write 10 Done Done Sent false
3093 1867 Acked Read 10 Done Done Sent false
3094 1868 Acked Write 5 Done Done Sent false
3095 1869 Acked Write 9 Done Done Sent false
3096 1870 Acked Read 5 Done Done Sent false
3097 1871 Acked Read 9 Done Done Sent false
3098 1872 Acked Flush 0 Done Done Sent false
3099 1873 Acked Write 7 Done Done Sent false
3100 1874 Acked Read 8 Done Done Sent false
3101 1875 Acked Read 1 Done Done Sent false
3102 1876 Acked Read 4 Done Done Sent false
3103 1877 Acked Read 9 Done Done Sent false
3104 1878 Acked Write 7 Done Done Sent false
3105 1879 Acked Write 8 Done Done Sent false
3106 1880 Acked Write 1 Done Done Sent false
3107 1881 Acked Write 7 Done Done Sent false
3108 1882 Acked Read 10 Done Done Sent false
3109 1883 Acked Flush 0 Done Done Sent false
3110 1884 Acked Write 7 Done Done Sent false
3111 1885 Acked Write 3 Done Done Sent false
3112 1886 Acked Read 8 Done Done Sent false
3113 1887 Acked Read 3 Done Done Sent false
3114 1888 Acked Read 10 Done Done Sent false
3115 1889 Acked Read 1 Done Done Sent false
3116 1890 Acked Read 9 Done Done Sent false
3117 1891 Acked Read 3 Done Done Sent false
3118 1892 Acked Write 2 Done Done Sent false
3119 1893 Acked Read 5 Done Done Sent false
3120 1894 Acked Read 3 Done Done Sent false
3121 1895 Acked Write 6 Done Done Sent false
3122 1896 Acked Write 9 Done Done Sent false
3123 1897 Acked Flush 0 Done Done Sent false
3124 1898 Acked Write 2 Done Done Sent false
3125 1899 Acked Write 9 Done Done Sent false
3126 1900 Acked Flush 0 Done Done Sent false
3127 1901 Acked Read 10 Done Done Sent false
3128 1902 Acked Read 7 Done Done Sent false
3129 1903 Acked Read 9 Done Done Sent false
3130 1904 Acked Write 3 Done Done Sent false
3131 1905 Acked Flush 0 Done Done Sent false
3132 STATES DS:0 DS:1 DS:2 TOTAL
3133 Sent 0 0 302 302
3134 Done 302 302 0 604
3135 Skipped 0 0 0 0
3136 Error 0 0 0 0
3137 Last Flush: 1905 1905 1603
3138 Downstairs last five completed: 1603 1602 1601 1600 1599
3139 Upstairs last five completed: 1905 1904 1903 1902 1901
3140 CLIENT: Up:0 ds:302 act:2
31412024-12-09T15:08:29.595ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31422024-12-09T15:08:29.595ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61107) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31432024-12-09T15:08:29.596ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31442024-12-09T15:08:29.596ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31452024-12-09T15:08:29.596ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31462024-12-09T15:08:29.596ZINFOcrucible: send last flush ID to this DS: 1603 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31472024-12-09T15:08:29.597ZINFOcrucible: Replied this last flush ID: 1603 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31482024-12-09T15:08:29.597ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31492024-12-09T15:08:29.597ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31502024-12-09T15:08:29.597ZINFOcrucible: [2] client re-new 302 jobs since flush 1603 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31512024-12-09T15:08:29.597ZINFOcrucible: [2] Replayed 302 jobs since flush: 1603 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
3152 ----------------------------------------------------------------
3153 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3154 Downstairs last five completed: 1905 1904 1903 1902 1901
3155 Upstairs last five completed: 1905 1904 1903 1902 1901
3156 CLIENT: Up:0 ds:0 act:3
3157 Replay: All jobs finished
3158 Main task has work to do, go find it
3159 got dsc Stop(1)
3160 stop 1
3161 Dec 09 15:08:31.845 INFO request completed, latency_us: 373, response_code[8820] Got stop action so:false kr:false
3162 : 204, uri: /stop/cid/1, method: POST, req_id: 102dd0b9-3a7f-42ef-9bf8-6f69d7135da0, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
3163 Dec 09 15:08:31.846 INFO request completed, latency_us: 128, response_code: 200, uri: /state/cid/1, method: GET, req_id: ae3a21dc-61ea-4d0c-bb63-7bfe7c128816, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
31642024-12-09T15:08:31.846ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31652024-12-09T15:08:31.846ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31662024-12-09T15:08:31.846ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31672024-12-09T15:08:31.846ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31682024-12-09T15:08:31.846ZINFOcrucible: [1] Transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31692024-12-09T15:08:31.846ZINFOcrucible: client task is exiting = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
3170 [8820] Exited with: ExitStatus(unix_wait_status(9))
3171 [8820][1] reports Exit
3172 Dec 09 15:08:35.848 INFO request completed, latency_us: 246, response_code: 200, uri: /state/cid/1, method: GET, req_id: c9a97e50-f598-4bac-a468-a56e727e025a, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
3173 001/300 Write block 2863 len 36864 data: 2 2 3 3 4 4 4 4 4
3174 002/300 Write block 4169 len 36864 data: 3 3 4 4 3 3 3 3 2
3175 003/300 Read block 4466 len 8192
3176 004/300 Read block 2592 len 32768
3177 005/300 Read block 970 len 28672
3178 006/300 Read block 2491 len 24576
3179 007/300 Write block 1416 len 16384 data: 3 2 2 2
3180 008/300 Write block 4869 len 16384 data: 3 3 2 2
3181 009/300 Write block 2667 len 32768 data: 2 2 2 2 2 2 2 2
3182 010/300 Write block 1835 len 32768 data: 2 2 2 2 2 3 3 3
3183 011/300 Write block 4429 len 4096 data: 3
3184 012/300 Read block 3046 len 32768
3185 013/300 Write block 1570 len 20480 data: 3 3 2 2 2
3186 014/300 Read block 1742 len 16384
3187 015/300 Read block 63 len 20480
3188 016/300 Read block 2818 len 8192
3189 017/300 Write block 672 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3190 018/300 Write block 3735 len 20480 data: 3 3 2 3 3
3191 019/300 Read block 3345 len 28672
3192 020/300 Read block 3577 len 8192
3193 021/300 Read block 1479 len 4096
3194 022/300 Write block 1363 len 40960 data: 4 4 3 3 3 4 4 5 4 4
3195 023/300 Read block 2060 len 40960
3196 024/300 Read block 1364 len 36864
3197 025/300 Write block 2831 len 28672 data: 2 2 2 2 2 2 2
3198 026/300 Write block 4411 len 36864 data: 4 4 4 2 2 2 2 2 2
3199 027/300 Read block 7 len 28672
3200 028/300 Write block 494 len 8192 data: 2 2
3201 029/300 Read block 943 len 36864
3202 030/300 Write block 1924 len 28672 data: 2 2 3 3 3 4 4
3203 031/300 Flush
3204 032/300 Write block 1242 len 36864 data: 2 2 2 2 2 2 2 2 2
3205 033/300 Read block 4134 len 16384
3206 034/300 Read block 4837 len 32768
3207 035/300 Read block 1384 len 40960
3208 036/300 Read block 2434 len 40960
3209 037/300 Write block 1142 len 40960 data: 3 2 2 3 2 2 3 3 3 3
3210 038/300 Read block 2297 len 32768
3211 039/300 Write block 4640 len 16384 data: 2 2 2 2
3212 040/300 Write block 3596 len 16384 data: 2 2 2 2
3213 041/300 Flush
3214 042/300 Write block 2711 len 12288 data: 2 2 2
3215 043/300 Read block 563 len 36864
3216 044/300 Read block 240 len 28672
3217 045/300 Read block 2655 len 24576
3218 046/300 Write block 1580 len 16384 data: 3 3 2 2
3219 047/300 Write block 2351 len 24576 data: 2 2 2 2 2 2
3220 048/300 Flush
3221 049/300 Read block 3783 len 28672
3222 050/300 Read block 2140 len 12288
3223 051/300 Read block 3811 len 8192
3224 052/300 Read block 2320 len 24576
3225 053/300 Read block 2130 len 36864
3226 054/300 Read block 2008 len 8192
3227 055/300 Read block 2948 len 36864
3228 056/300 Write block 214 len 24576 data: 3 3 3 3 3 4
3229 057/300 Read block 1626 len 4096
3230 058/300 Write block 4978 len 24576 data: 2 2 2 2 2 2
3231 059/300 Read block 2343 len 28672
3232 060/300 Read block 3599 len 28672
3233 061/300 Write block 2028 len 24576 data: 3 2 2 2 2 2
3234 062/300 Read block 3598 len 8192
3235 063/300 Write block 2992 len 24576 data: 3 3 3 3 3 2
3236 064/300 Read block 4722 len 24576
3237 065/300 Flush
3238 066/300 Write block 1977 len 40960 data: 2 2 2 2 3 3 3 3 3 3
3239 067/300 Read block 3007 len 20480
3240 068/300 Read block 964 len 4096
3241 069/300 Write block 4312 len 28672 data: 2 2 2 2 2 2 2
3242 070/300 Flush
3243 071/300 Read block 1197 len 12288
3244 072/300 Write block 4889 len 4096 data: 2
3245 073/300 Read block 324 len 12288
3246 074/300 Write block 3823 len 28672 data: 2 2 2 2 2 2 2
3247 075/300 Write block 2493 len 12288 data: 3 3 3
3248 076/300 Read block 1342 len 12288
3249 077/300 Write block 1279 len 36864 data: 3 2 2 2 2 2 2 2 2
3250 078/300 Read block 2728 len 16384
3251 079/300 Read block 2940 len 40960
3252 080/300 Write block 1057 len 28672 data: 2 2 2 2 2 2 2
3253 081/300 Read block 4880 len 40960
3254 082/300 Read block 3452 len 4096
3255 083/300 Write block 716 len 24576 data: 2 2 2 2 2 2
3256 084/300 Read block 2294 len 40960
3257 085/300 Read block 2194 len 8192
3258 086/300 Read block 3418 len 36864
3259 087/300 Read block 1151 len 16384
3260 088/300 Read block 4907 len 40960
3261 089/300 Read block 4849 len 32768
3262 090/300 Write block 4771 len 4096 data: 2
3263 091/300 Write block 2829 len 16384 data: 2 2 3 3
3264 092/300 Write block 2045 len 4096 data: 2
3265 093/300 Read block 2797 len 24576
3266 094/300 Read block 1959 len 40960
3267 095/300 Read block 3860 len 8192
3268 096/300 Read block 3023 len 8192
3269 097/300 Read block 4423 len 8192
3270 098/300 Write block 923 len 28672 data: 2 2 2 3 3 3 2
3271 099/300 Write block 2 len 8192 data: 2 2
3272 100/300 Write block 2583 len 12288 data: 2 2 2
3273 101/300 Read block 305 len 8192
3274 102/300 Read block 2263 len 12288
3275 103/300 Write block 2641 len 20480 data: 3 3 3 3 3
3276 104/300 Read block 3831 len 28672
3277 105/300 Read block 1894 len 40960
3278 106/300 Read block 2009 len 4096
3279 107/300 Write block 3987 len 36864 data: 2 2 2 2 2 2 2 2 2
3280 108/300 Read block 1487 len 28672
3281 109/300 Read block 1119 len 4096
3282 110/300 Write block 4493 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3283 111/300 Flush
3284 112/300 Read block 3975 len 12288
3285 113/300 Write block 3320 len 4096 data: 3
3286 114/300 Write block 178 len 40960 data: 3 3 3 2 2 2 2 3 3 3
3287 115/300 Write block 2080 len 4096 data: 2
3288 116/300 Write block 1574 len 32768 data: 3 3 3 3 3 3 4 4
3289 117/300 Write block 1312 len 28672 data: 3 3 3 2 2 2 2
3290 118/300 Flush
3291 119/300 Read block 3100 len 28672
3292 120/300 Write block 915 len 16384 data: 2 2 2 2
3293 121/300 Write block 1785 len 12288 data: 2 3 3
3294 122/300 Flush
3295 123/300 Write block 2254 len 12288 data: 2 2 2
3296 124/300 Write block 4161 len 4096 data: 3
3297 125/300 Write block 3640 len 8192 data: 2 2
3298 126/300 Write block 2667 len 24576 data: 3 3 3 3 3 3
3299 127/300 Read block 1550 len 32768
3300 128/300 Read block 989 len 4096
3301 129/300 Write block 2276 len 20480 data: 2 2 2 2 2
3302 130/300 Flush
3303 131/300 Write block 4623 len 4096 data: 3
3304 132/300 Read block 4566 len 36864
3305 133/300 Read block 4880 len 36864
3306 134/300 Write block 1508 len 4096 data: 2
3307 135/300 Read block 3335 len 32768
3308 136/300 Write block 4222 len 28672 data: 3 3 3 3 3 2 3
3309 137/300 Read block 2636 len 12288
3310 138/300 Read block 21 len 24576
3311 139/300 Read block 566 len 4096
3312 140/300 Write block 4800 len 20480 data: 2 2 2 2 2
3313 141/300 Read block 1837 len 4096
3314 142/300 Write block 2941 len 28672 data: 3 3 3 2 2 2 2
3315 143/300 Write block 4799 len 20480 data: 2 3 3 3 3
3316 144/300 Write block 2948 len 36864 data: 2 2 2 2 2 2 2 2 2
3317 145/300 Read block 1763 len 32768
3318 146/300 Write block 947 len 20480 data: 3 3 3 3 3
3319 147/300 Write block 4172 len 8192 data: 5 4
3320 148/300 Read block 2873 len 12288
3321 149/300 Write block 3072 len 16384 data: 2 2 3 3
3322 150/300 Write block 845 len 4096 data: 2
3323 151/300 Read block 3195 len 16384
3324 152/300 Write block 3804 len 4096 data: 3
3325 153/300 Flush
3326 154/300 Read block 442 len 32768
3327 155/300 Read block 131 len 36864
3328 156/300 Write block 2177 len 32768 data: 2 2 2 2 2 2 2 2
3329 157/300 Flush
3330 158/300 Flush
3331 159/300 Read block 3734 len 28672
3332 160/300 Read block 2595 len 36864
3333 161/300 Read block 508 len 20480
3334 162/300 Read block 2356 len 4096
3335 163/300 Flush
3336 164/300 Read block 272 len 12288
3337 165/300 Read block 2577 len 8192
3338 166/300 Read block 840 len 16384
3339 167/300 Write block 4512 len 32768 data: 3 2 2 2 2 2 2 2
3340 168/300 Write block 3607 len 4096 data: 2
3341 169/300 Read block 1336 len 36864
3342 170/300 Read block 3782 len 40960
3343 171/300 Write block 3880 len 28672 data: 2 2 2 2 2 2 2
3344 172/300 Write block 881 len 28672 data: 2 2 2 2 2 2 2
3345 173/300 Read block 4944 len 40960
3346 174/300 Write block 25 len 32768 data: 2 2 2 2 2 2 2 2
3347 175/300 Write block 724 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3348 176/300 Read block 207 len 8192
3349 177/300 Read block 3187 len 24576
3350 178/300 Write block 2143 len 20480 data: 4 3 3 3 3
3351 179/300 Read block 514 len 36864
3352 180/300 Write block 1996 len 20480 data: 2 3 3 3 3
3353 181/300 Read block 4782 len 36864
3354 182/300 Flush
3355 183/300 Read block 402 len 32768
3356 184/300 Read block 3090 len 28672
3357 185/300 Write block 2004 len 24576 data: 3 3 3 3 3 3
3358 186/300 Read block 4566 len 40960
3359 187/300 Read block 2413 len 4096
3360 188/300 Read block 4873 len 20480
3361 189/300 Read block 3451 len 24576
3362 190/300 Read block 3017 len 40960
3363 191/300 Write block 547 len 40960 data: 4 4 4 4 4 3 2 2 2 2
3364 192/300 Write block 1825 len 32768 data: 3 3 3 3 2 2 2 2
3365 193/300 Write block 169 len 12288 data: 2 2 2
3366 194/300 Write block 1906 len 36864 data: 2 2 2 3 3 3 3 3 3
3367 195/300 Write block 4654 len 4096 data: 2
3368 196/300 Write block 4639 len 16384 data: 2 3 3 3
3369 197/300 Write block 2496 len 24576 data: 2 2 2 2 2 2
3370 198/300 Read block 2039 len 20480
3371 199/300 Write block 1683 len 4096 data: 2
3372 200/300 Read block 3418 len 8192
3373 201/300 Flush
3374 202/300 Read block 275 len 20480
3375 203/300 Read block 1726 len 20480
3376 204/300 Read block 3138 len 12288
3377 205/300 Read block 251 len 4096
3378 206/300 Read block 2492 len 20480
3379 207/300 Write block 2014 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3380 208/300 Write block 320 len 20480 data: 2 2 2 2 2
3381 209/300 Write block 2612 len 36864 data: 2 2 2 2 3 3 4 4 4
3382 210/300 Flush
3383 211/300 Read block 4432 len 8192
3384 212/300 Write block 2556 len 20480 data: 3 3 3 3 2
3385 213/300 Flush
3386 214/300 Write block 3679 len 12288 data: 2 2 2
3387 215/300 Read block 3574 len 16384
3388 216/300 Read block 1784 len 24576
3389 217/300 Read block 1313 len 16384
3390 218/300 Write block 1713 len 16384 data: 2 2 2 2
3391 219/300 Read block 2697 len 40960
3392 220/300 Read block 1163 len 36864
3393 221/300 Read block 3491 len 28672
3394 222/300 Write block 3777 len 28672 data: 2 2 2 2 2 2 2
3395 223/300 Read block 3873 len 12288
3396 224/300 Read block 4742 len 24576
3397 225/300 Read block 1615 len 16384
3398 226/300 Read block 4388 len 36864
3399 227/300 Read block 2738 len 28672
3400 228/300 Write block 1037 len 4096 data: 2
3401 229/300 Write block 575 len 36864 data: 3 3 3 3 3 3 3 3 3
3402 230/300 Write block 3453 len 4096 data: 2
3403 231/300 Write block 1002 len 12288 data: 2 2 2
3404 232/300 Read block 3973 len 28672
3405 233/300 Write block 4534 len 8192 data: 2 3
3406 234/300 Read block 4468 len 36864
3407 235/300 Write block 2733 len 24576 data: 2 2 3 3 3 3
3408 236/300 Read block 13 len 12288
3409 237/300 Write block 4153 len 8192 data: 2 2
3410 238/300 Write block 3255 len 20480 data: 2 2 2 2 2
3411 239/300 Read block 2978 len 32768
3412 240/300 Write block 1662 len 20480 data: 2 2 2 2 2
3413 241/300 Read block 1783 len 36864
3414 242/300 Read block 419 len 40960
3415 243/300 Write block 4763 len 24576 data: 3 3 3 2 2 2
3416 244/300 Read block 4332 len 28672
3417 245/300 Write block 4701 len 36864 data: 4 3 3 3 3 2 2 2 2
3418 246/300 Flush
3419 247/300 Read block 1715 len 36864
3420 248/300 Read block 4846 len 16384
3421 249/300 Write block 4625 len 8192 data: 3 3
3422 250/300 Write block 759 len 4096 data: 4
3423 251/300 Read block 3420 len 8192
3424 252/300 Read block 2479 len 24576
3425 253/300 Write block 1149 len 16384 data: 4 4 4 3
3426 254/300 Flush
3427 255/300 Flush
3428 256/300 Read block 2589 len 28672
3429 257/300 Write block 4287 len 16384 data: 3 2 2 2
3430 258/300 Flush
3431 259/300 Read block 678 len 8192
3432 260/300 Write block 2346 len 4096 data: 2
3433 261/300 Flush
3434 262/300 Write block 1786 len 24576 data: 4 4 3 3 2 2
3435 263/300 Write block 256 len 32768 data: 2 2 2 2 2 2 2 2
3436 264/300 Write block 4237 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3437 265/300 Write block 4556 len 24576 data: 2 2 2 2 2 2
3438 266/300 Write block 3904 len 20480 data: 2 2 2 2 2
3439 267/300 Write block 4835 len 36864 data: 3 3 3 3 4 5 5 4 4
3440 268/300 Read block 321 len 16384
3441 269/300 Write block 1806 len 36864 data: 3 3 2 2 2 2 2 2 2
3442 270/300 Read block 2444 len 20480
3443 271/300 Read block 2725 len 32768
3444 272/300 Write block 2254 len 20480 data: 3 3 3 2 2
3445 273/300 Write block 1197 len 20480 data: 2 2 2 2 2
3446 274/300 Read block 2162 len 16384
3447 275/300 Write block 3075 len 20480 data: 4 3 3 3 3
3448 276/300 Write block 1076 len 20480 data: 3 3 3 3 3
3449 277/300 Read block 911 len 36864
3450 278/300 Read block 1872 len 40960
3451 279/300 Read block 3743 len 36864
3452 280/300 Flush
3453 281/300 Read block 4658 len 12288
3454 282/300 Read block 329 len 4096
3455 283/300 Write block 3061 len 12288 data: 2 2 2
3456 284/300 Write block 1827 len 12288 data: 4 4 3
3457 285/300 Read block 3744 len 28672
3458 286/300 Read block 4923 len 4096
3459 287/300 Write block 280 len 12288 data: 3 3 3
3460 288/300 Read block 2376 len 32768
3461 289/300 Read block 2962 len 24576
3462 290/300 Write block 4433 len 32768 data: 2 2 2 2 2 2 3 3
3463 291/300 Read block 3280 len 28672
3464 292/300 Read block 3994 len 12288
3465 293/300 Read block 4478 len 8192
3466 294/300 Write block 1769 len 4096 data: 2
3467 295/300 Write block 2886 len 40960 data: 3 3 3 3 4 4 4 4 4 4
3468 296/300 Write block 3411 len 40960 data: 2 2 2 2 2 3 3 3 3 3
3469 297/300 Read block 2221 len 24576
3470 298/300 Write block 3880 len 16384 data: 3 3 3 3
3471 299/300 Write block 466 len 8192 data: 3 3
3472 300/300 Write block 3144 len 36864 data: 2 3 3 3 2 2 3 3 3
3473 301/300 Read block 1615 len 40960
3474 Main task has work to do, go find it
3475 got dsc Start(1)
3476 start 1
3477 Dec 09 15:08:36.088 INFO request completed, latency_us: 230, response_code: 204, uri: /start/cid/1, method: POST, req_id: 318eaf4b-59a6-4042-80e1-943af1bbd7b7, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
3478 Replay: started 1, returned:Ok(())
3479 ----------------------------------------------------------------
3480 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3481 States: Active Offline Active
3482 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3483 1906 Acked Write 9 Done Sent Done false
3484 1907 Acked Write 9 Done Sent Done false
3485 1908 Acked Read 2 Done Sent Done false
3486 1909 Acked Read 8 Done Sent Done false
3487 1910 Acked Read 7 Done Sent Done false
3488 1911 Acked Read 6 Done Sent Done false
3489 1912 Acked Write 4 Done Sent Done false
3490 1913 Acked Write 4 Done Sent Done false
3491 1914 Acked Write 8 Done Sent Done false
3492 1915 Acked Write 8 Done Sent Done false
3493 1916 Acked Write 1 Done Sent Done false
3494 1917 Acked Read 8 Done Sent Done false
3495 1918 Acked Write 5 Done Sent Done false
3496 1919 Acked Read 4 Done Sent Done false
3497 1920 Acked Read 5 Done Sent Done false
3498 1921 Acked Read 2 Done Sent Done false
3499 1922 Acked Write 10 Done Sent Done false
3500 1923 Acked Write 5 Done Sent Done false
3501 1924 Acked Read 7 Done Sent Done false
3502 1925 Acked Read 2 Done Sent Done false
3503 1926 Acked Read 1 Done Sent Done false
3504 1927 Acked Write 10 Done Sent Done false
3505 1928 Acked Read 10 Done Sent Done false
3506 1929 Acked Read 9 Done Sent Done false
3507 1930 Acked Write 7 Done Sent Done false
3508 1931 Acked Write 9 Done Sent Done false
3509 1932 Acked Read 7 Done Sent Done false
3510 1933 Acked Write 2 Done Sent Done false
3511 1934 Acked Read 9 Done Sent Done false
3512 1935 Acked Write 7 Done Sent Done false
3513 1936 Acked Flush 0 Done Sent Done false
3514 1937 Acked Write 9 Done Sent Done false
3515 1938 Acked Read 4 Done Sent Done false
3516 1939 Acked Read 8 Done Sent Done false
3517 1940 Acked Read 10 Done Sent Done false
3518 1941 Acked Read 10 Done Sent Done false
3519 1942 Acked Write 10 Done Sent Done false
3520 1943 Acked Read 8 Done Sent Done false
3521 1944 Acked Write 4 Done Sent Done false
3522 1945 Acked Write 4 Done Sent Done false
3523 1946 Acked Flush 0 Done Sent Done false
3524 1947 Acked Write 3 Done Sent Done false
3525 1948 Acked Read 9 Done Sent Done false
3526 1949 Acked Read 7 Done Sent Done false
3527 1950 Acked Read 6 Done Sent Done false
3528 1951 Acked Write 4 Done Sent Done false
3529 1952 Acked Write 6 Done Sent Done false
3530 1953 Acked Flush 0 Done Sent Done false
3531 1954 Acked Read 7 Done Sent Done false
3532 1955 Acked Read 3 Done Sent Done false
3533 1956 Acked Read 2 Done Sent Done false
3534 1957 Acked Read 6 Done Sent Done false
3535 1958 Acked Read 9 Done Sent Done false
3536 1959 Acked Read 2 Done Sent Done false
3537 1960 Acked Read 9 Done Sent Done false
3538 1961 Acked Write 6 Done Sent Done false
3539 1962 Acked Read 1 Done Sent Done false
3540 1963 Acked Write 6 Done Sent Done false
3541 1964 Acked Read 7 Done Sent Done false
3542 1965 Acked Read 7 Done Sent Done false
3543 1966 Acked Write 6 Done Sent Done false
3544 1967 Acked Read 2 Done Sent Done false
3545 1968 Acked Write 6 Done Sent Done false
3546 1969 Acked Read 6 Done Sent Done false
3547 1970 Acked Flush 0 Done Sent Done false
3548 1971 Acked Write 10 Done Sent Done false
3549 1972 Acked Read 5 Done Sent Done false
3550 1973 Acked Read 1 Done Sent Done false
3551 1974 Acked Write 7 Done Sent Done false
3552 1975 Acked Flush 0 Done Sent Done false
3553 1976 Acked Read 3 Done Sent Done false
3554 1977 Acked Write 1 Done Sent Done false
3555 1978 Acked Read 3 Done Sent Done false
3556 1979 Acked Write 7 Done Sent Done false
3557 1980 Acked Write 3 Done Sent Done false
3558 1981 Acked Read 3 Done Sent Done false
3559 1982 Acked Write 9 Done Sent Done false
3560 1983 Acked Read 4 Done Sent Done false
3561 1984 Acked Read 10 Done Sent Done false
3562 1985 Acked Write 7 Done Sent Done false
3563 1986 Acked Read 10 Done Sent Done false
3564 1987 Acked Read 1 Done Sent Done false
3565 1988 Acked Write 6 Done Sent Done false
3566 1989 Acked Read 10 Done Sent Done false
3567 1990 Acked Read 2 Done Sent Done false
3568 1991 Acked Read 9 Done Sent Done false
3569 1992 Acked Read 4 Done Sent Done false
3570 1993 Acked Read 10 Done Sent Done false
3571 1994 Acked Read 8 Done Sent Done false
3572 1995 Acked Write 1 Done Sent Done false
3573 1996 Acked Write 4 Done Sent Done false
3574 1997 Acked Write 1 Done Sent Done false
3575 1998 Acked Read 6 Done Sent Done false
3576 1999 Acked Read 10 Done Sent Done false
3577 2000 Acked Read 2 Done Sent Done false
3578 2001 Acked Read 2 Done Sent Done false
3579 2002 Acked Read 2 Done Sent Done false
3580 2003 Acked Write 7 Done Sent Done false
3581 2004 Acked Write 2 Done Sent Done false
3582 2005 Acked Write 3 Done Sent Done false
3583 2006 Acked Read 2 Done Sent Done false
3584 2007 Acked Read 3 Done Sent Done false
3585 2008 Acked Write 5 Done Sent Done false
3586 2009 Acked Read 7 Done Sent Done false
3587 2010 Acked Read 10 Done Sent Done false
3588 2011 Acked Read 1 Done Sent Done false
3589 2012 Acked Write 9 Done Sent Done false
3590 2013 Acked Read 7 Done Sent Done false
3591 2014 Acked Read 1 Done Sent Done false
3592 2015 Acked Write 10 Done Sent Done false
3593 2016 Acked Flush 0 Done Sent Done false
3594 2017 Acked Read 3 Done Sent Done false
3595 2018 Acked Write 1 Done Sent Done false
3596 2019 Acked Write 10 Done Sent Done false
3597 2020 Acked Write 1 Done Sent Done false
3598 2021 Acked Write 8 Done Sent Done false
3599 2022 Acked Write 7 Done Sent Done false
3600 2023 Acked Flush 0 Done Sent Done false
3601 2024 Acked Read 7 Done Sent Done false
3602 2025 Acked Write 4 Done Sent Done false
3603 2026 Acked Write 3 Done Sent Done false
3604 2027 Acked Flush 0 Done Sent Done false
3605 2028 Acked Write 3 Done Sent Done false
3606 2029 Acked Write 1 Done Sent Done false
3607 2030 Acked Write 2 Done Sent Done false
3608 2031 Acked Write 6 Done Sent Done false
3609 2032 Acked Read 8 Done Sent Done false
3610 2033 Acked Read 1 Done Sent Done false
3611 2034 Acked Write 5 Done Sent Done false
3612 2035 Acked Flush 0 Done Sent Done false
3613 2036 Acked Write 1 Done Sent Done false
3614 2037 Acked Read 9 Done Sent Done false
3615 2038 Acked Read 9 Done Sent Done false
3616 2039 Acked Write 1 Done Sent Done false
3617 2040 Acked Read 8 Done Sent Done false
3618 2041 Acked Write 7 Done Sent Done false
3619 2042 Acked Read 3 Done Sent Done false
3620 2043 Acked Read 6 Done Sent Done false
3621 2044 Acked Read 1 Done Sent Done false
3622 2045 Acked Write 5 Done Sent Done false
3623 2046 Acked Read 1 Done Sent Done false
3624 2047 Acked Write 7 Done Sent Done false
3625 2048 Acked Write 5 Done Sent Done false
3626 2049 Acked Write 9 Done Sent Done false
3627 2050 Acked Read 8 Done Sent Done false
3628 2051 Acked Write 5 Done Sent Done false
3629 2052 Acked Write 2 Done Sent Done false
3630 2053 Acked Read 3 Done Sent Done false
3631 2054 Acked Write 4 Done Sent Done false
3632 2055 Acked Write 1 Done Sent Done false
3633 2056 Acked Read 4 Done Sent Done false
3634 2057 Acked Write 1 Done Sent Done false
3635 2058 Acked Flush 0 Done Sent Done false
3636 2059 Acked Read 8 Done Sent Done false
3637 2060 Acked Read 9 Done Sent Done false
3638 2061 Acked Write 8 Done Sent Done false
3639 2062 Acked Flush 0 Done Sent Done false
3640 2063 Acked Flush 0 Done Sent Done false
3641 2064 Acked Read 7 Done Sent Done false
3642 2065 Acked Read 9 Done Sent Done false
3643 2066 Acked Read 5 Done Sent Done false
3644 2067 Acked Read 1 Done Sent Done false
3645 2068 Acked Flush 0 Done Sent Done false
3646 2069 Acked Read 3 Done Sent Done false
3647 2070 Acked Read 2 Done Sent Done false
3648 2071 Acked Read 4 Done Sent Done false
3649 2072 Acked Write 8 Done Sent Done false
3650 2073 Acked Write 1 Done Sent Done false
3651 2074 Acked Read 9 Done Sent Done false
3652 2075 Acked Read 10 Done Sent Done false
3653 2076 Acked Write 7 Done Sent Done false
3654 2077 Acked Write 7 Done Sent Done false
3655 2078 Acked Read 10 Done Sent Done false
3656 2079 Acked Write 8 Done Sent Done false
3657 2080 Acked Write 10 Done Sent Done false
3658 2081 Acked Read 2 Done Sent Done false
3659 2082 Acked Read 6 Done Sent Done false
3660 2083 Acked Write 5 Done Sent Done false
3661 2084 Acked Read 9 Done Sent Done false
3662 2085 Acked Write 5 Done Sent Done false
3663 2086 Acked Read 9 Done Sent Done false
3664 2087 Acked Flush 0 Done Sent Done false
3665 2088 Acked Read 8 Done Sent Done false
3666 2089 Acked Read 7 Done Sent Done false
3667 2090 Acked Write 6 Done Sent Done false
3668 2091 Acked Read 10 Done Sent Done false
3669 2092 Acked Read 1 Done Sent Done false
3670 2093 Acked Read 5 Done Sent Done false
3671 2094 Acked Read 6 Done Sent Done false
3672 2095 Acked Read 10 Done Sent Done false
3673 2096 Acked Write 10 Done Sent Done false
3674 2097 Acked Write 8 Done Sent Done false
3675 2098 Acked Write 3 Done Sent Done false
3676 2099 Acked Write 9 Done Sent Done false
3677 2100 Acked Write 1 Done Sent Done false
3678 2101 Acked Write 4 Done Sent Done false
3679 2102 Acked Write 6 Done Sent Done false
3680 2103 Acked Read 5 Done Sent Done false
3681 2104 Acked Write 1 Done Sent Done false
3682 2105 Acked Read 2 Done Sent Done false
3683 2106 Acked Flush 0 Done Sent Done false
3684 2107 Acked Read 5 Done Sent Done false
3685 2108 Acked Read 5 Done Sent Done false
3686 2109 Acked Read 3 Done Sent Done false
3687 2110 Acked Read 1 Done Sent Done false
3688 2111 Acked Read 5 Done Sent Done false
3689 2112 Acked Write 10 Done Sent Done false
3690 2113 Acked Write 5 Done Sent Done false
3691 2114 Acked Write 9 Done Sent Done false
3692 2115 Acked Flush 0 Done Sent Done false
3693 2116 Acked Read 2 Done Sent Done false
3694 2117 Acked Write 5 Done Sent Done false
3695 2118 Acked Flush 0 Done Sent Done false
3696 2119 Acked Write 3 Done Sent Done false
3697 2120 Acked Read 4 Done Sent Done false
3698 2121 Acked Read 6 Done Sent Done false
3699 2122 Acked Read 4 Done Sent Done false
3700 2123 Acked Write 4 Done Sent Done false
3701 2124 Acked Read 10 Done Sent Done false
3702 2125 Acked Read 9 Done Sent Done false
3703 2126 Acked Read 7 Done Sent Done false
3704 2127 Acked Write 7 Done Sent Done false
3705 2128 Acked Read 3 Done Sent Done false
3706 2129 Acked Read 6 Done Sent Done false
3707 2130 Acked Read 4 Done Sent Done false
3708 2131 Acked Read 9 Done Sent Done false
3709 2132 Acked Read 7 Done Sent Done false
3710 2133 Acked Write 1 Done Sent Done false
3711 2134 Acked Write 9 Done Sent Done false
3712 2135 Acked Write 1 Done Sent Done false
3713 2136 Acked Write 3 Done Sent Done false
3714 2137 Acked Read 7 Done Sent Done false
3715 2138 Acked Write 2 Done Sent Done false
3716 2139 Acked Read 9 Done Sent Done false
3717 2140 Acked Write 6 Done Sent Done false
3718 2141 Acked Read 3 Done Sent Done false
3719 2142 Acked Write 2 Done Sent Done false
3720 2143 Acked Write 5 Done Sent Done false
3721 2144 Acked Read 8 Done Sent Done false
3722 2145 Acked Write 5 Done Sent Done false
3723 2146 Acked Read 9 Done Sent Done false
3724 2147 Acked Read 10 Done Sent Done false
3725 2148 Acked Write 6 Done Sent Done false
3726 2149 Acked Read 7 Done Sent Done false
3727 2150 Acked Write 9 Done Sent Done false
3728 2151 Acked Flush 0 Done Sent Done false
3729 2152 Acked Read 9 Done Sent Done false
3730 2153 Acked Read 4 Done Sent Done false
3731 2154 Acked Write 2 Done Sent Done false
3732 2155 Acked Write 1 Done Sent Done false
3733 2156 Acked Read 2 Done Sent Done false
3734 2157 Acked Read 6 Done Sent Done false
3735 2158 Acked Write 4 Done Sent Done false
3736 2159 Acked Flush 0 Done Sent Done false
3737 2160 Acked Flush 0 Done Sent Done false
3738 2161 Acked Read 7 Done Sent Done false
3739 2162 Acked Write 4 Done Sent Done false
3740 2163 Acked Flush 0 Done Sent Done false
3741 2164 Acked Read 2 Done Sent Done false
3742 2165 Acked Write 1 Done Sent Done false
3743 2166 Acked Flush 0 Done Sent Done false
3744 2167 Acked Write 6 Done Sent Done false
3745 2168 Acked Write 8 Done Sent Done false
3746 2169 Acked Write 10 Done Sent Done false
3747 2170 Acked Write 6 Done Sent Done false
3748 2171 Acked Write 5 Done Sent Done false
3749 2172 Acked Write 9 Done Sent Done false
3750 2173 Acked Read 4 Done Sent Done false
3751 2174 Acked Write 9 Done Sent Done false
3752 2175 Acked Read 5 Done Sent Done false
3753 2176 Acked Read 8 Done Sent Done false
3754 2177 Acked Write 5 Done Sent Done false
3755 2178 Acked Write 5 Done Sent Done false
3756 2179 Acked Read 4 Done Sent Done false
3757 2180 Acked Write 5 Done Sent Done false
3758 2181 Acked Write 5 Done Sent Done false
3759 2182 Acked Read 9 Done Sent Done false
3760 2183 Acked Read 10 Done Sent Done false
3761 2184 Acked Read 9 Done Sent Done false
3762 2185 Acked Flush 0 Done Sent Done false
3763 2186 Acked Read 3 Done Sent Done false
3764 2187 Acked Read 1 Done Sent Done false
3765 2188 Acked Write 3 Done Sent Done false
3766 2189 Acked Write 3 Done Sent Done false
3767 2190 Acked Read 7 Done Sent Done false
3768 2191 Acked Read 1 Done Sent Done false
3769 2192 Acked Write 3 Done Sent Done false
3770 2193 Acked Read 8 Done Sent Done false
3771 2194 Acked Read 6 Done Sent Done false
3772 2195 Acked Write 8 Done Sent Done false
3773 2196 Acked Read 7 Done Sent Done false
3774 2197 Acked Read 3 Done Sent Done false
3775 2198 Acked Read 2 Done Sent Done false
3776 2199 Acked Write 1 Done Sent Done false
3777 2200 Acked Write 10 Done Sent Done false
3778 2201 Acked Write 10 Done Sent Done false
3779 2202 Acked Read 6 Done Sent Done false
3780 2203 Acked Write 4 Done Sent Done false
3781 2204 Acked Write 2 Done Sent Done false
3782 2205 Acked Write 9 Done Sent Done false
3783 2206 Acked Read 10 Done Sent Done false
3784 STATES DS:0 DS:1 DS:2 TOTAL
3785 Sent 0 301 0 301
3786 Done 301 0 301 602
3787 Skipped 0 0 0 0
3788 Error 0 0 0 0
3789 Last Flush: 2185 1905 2185
3790 Downstairs last five completed: 1905 1904 1903 1902 1901
3791 Upstairs last five completed: 2206 2205 2204 2203 2202
3792 CLIENT: Up:0 ds:301 act:2
3793 [8820] got start action
3794 [8820] I am going to restart
3795 Starting downstairs at port 8820
3796 Make output file at "/tmp/dsc/downstairs-8820.txt"
3797 [8820][1] reports Starting
3798 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1205)
3799 [8820][1] reports Running
3800 ----------------------------------------------------------------
3801 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
3802 States: Active Offline Active
3803 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3804 1906 Acked Write 9 Done Sent Done false
3805 1907 Acked Write 9 Done Sent Done false
3806 1908 Acked Read 2 Done Sent Done false
3807 1909 Acked Read 8 Done Sent Done false
3808 1910 Acked Read 7 Done Sent Done false
3809 1911 Acked Read 6 Done Sent Done false
3810 1912 Acked Write 4 Done Sent Done false
3811 1913 Acked Write 4 Done Sent Done false
3812 1914 Acked Write 8 Done Sent Done false
3813 1915 Acked Write 8 Done Sent Done false
3814 1916 Acked Write 1 Done Sent Done false
3815 1917 Acked Read 8 Done Sent Done false
3816 1918 Acked Write 5 Done Sent Done false
3817 1919 Acked Read 4 Done Sent Done false
3818 1920 Acked Read 5 Done Sent Done false
3819 1921 Acked Read 2 Done Sent Done false
3820 1922 Acked Write 10 Done Sent Done false
3821 1923 Acked Write 5 Done Sent Done false
3822 1924 Acked Read 7 Done Sent Done false
3823 1925 Acked Read 2 Done Sent Done false
3824 1926 Acked Read 1 Done Sent Done false
3825 1927 Acked Write 10 Done Sent Done false
3826 1928 Acked Read 10 Done Sent Done false
3827 1929 Acked Read 9 Done Sent Done false
3828 1930 Acked Write 7 Done Sent Done false
3829 1931 Acked Write 9 Done Sent Done false
3830 1932 Acked Read 7 Done Sent Done false
3831 1933 Acked Write 2 Done Sent Done false
3832 1934 Acked Read 9 Done Sent Done false
3833 1935 Acked Write 7 Done Sent Done false
3834 1936 Acked Flush 0 Done Sent Done false
3835 1937 Acked Write 9 Done Sent Done false
3836 1938 Acked Read 4 Done Sent Done false
3837 1939 Acked Read 8 Done Sent Done false
3838 1940 Acked Read 10 Done Sent Done false
3839 1941 Acked Read 10 Done Sent Done false
3840 1942 Acked Write 10 Done Sent Done false
3841 1943 Acked Read 8 Done Sent Done false
3842 1944 Acked Write 4 Done Sent Done false
3843 1945 Acked Write 4 Done Sent Done false
3844 1946 Acked Flush 0 Done Sent Done false
3845 1947 Acked Write 3 Done Sent Done false
3846 1948 Acked Read 9 Done Sent Done false
3847 1949 Acked Read 7 Done Sent Done false
3848 1950 Acked Read 6 Done Sent Done false
3849 1951 Acked Write 4 Done Sent Done false
3850 1952 Acked Write 6 Done Sent Done false
3851 1953 Acked Flush 0 Done Sent Done false
3852 1954 Acked Read 7 Done Sent Done false
3853 1955 Acked Read 3 Done Sent Done false
3854 1956 Acked Read 2 Done Sent Done false
3855 1957 Acked Read 6 Done Sent Done false
3856 1958 Acked Read 9 Done Sent Done false
3857 1959 Acked Read 2 Done Sent Done false
3858 1960 Acked Read 9 Done Sent Done false
3859 1961 Acked Write 6 Done Sent Done false
3860 1962 Acked Read 1 Done Sent Done false
3861 1963 Acked Write 6 Done Sent Done false
3862 1964 Acked Read 7 Done Sent Done false
3863 1965 Acked Read 7 Done Sent Done false
3864 1966 Acked Write 6 Done Sent Done false
3865 1967 Acked Read 2 Done Sent Done false
3866 1968 Acked Write 6 Done Sent Done false
3867 1969 Acked Read 6 Done Sent Done false
3868 1970 Acked Flush 0 Done Sent Done false
3869 1971 Acked Write 10 Done Sent Done false
3870 1972 Acked Read 5 Done Sent Done false
3871 1973 Acked Read 1 Done Sent Done false
3872 1974 Acked Write 7 Done Sent Done false
3873 1975 Acked Flush 0 Done Sent Done false
3874 1976 Acked Read 3 Done Sent Done false
3875 1977 Acked Write 1 Done Sent Done false
3876 1978 Acked Read 3 Done Sent Done false
3877 1979 Acked Write 7 Done Sent Done false
3878 1980 Acked Write 3 Done Sent Done false
3879 1981 Acked Read 3 Done Sent Done false
3880 1982 Acked Write 9 Done Sent Done false
3881 1983 Acked Read 4 Done Sent Done false
3882 1984 Acked Read 10 Done Sent Done false
3883 1985 Acked Write 7 Done Sent Done false
3884 1986 Acked Read 10 Done Sent Done false
3885 1987 Acked Read 1 Done Sent Done false
3886 1988 Acked Write 6 Done Sent Done false
3887 1989 Acked Read 10 Done Sent Done false
3888 1990 Acked Read 2 Done Sent Done false
3889 1991 Acked Read 9 Done Sent Done false
3890 1992 Acked Read 4 Done Sent Done false
3891 1993 Acked Read 10 Done Sent Done false
3892 1994 Acked Read 8 Done Sent Done false
3893 1995 Acked Write 1 Done Sent Done false
3894 1996 Acked Write 4 Done Sent Done false
3895 1997 Acked Write 1 Done Sent Done false
3896 1998 Acked Read 6 Done Sent Done false
3897 1999 Acked Read 10 Done Sent Done false
3898 2000 Acked Read 2 Done Sent Done false
3899 2001 Acked Read 2 Done Sent Done false
3900 2002 Acked Read 2 Done Sent Done false
3901 2003 Acked Write 7 Done Sent Done false
3902 2004 Acked Write 2 Done Sent Done false
3903 2005 Acked Write 3 Done Sent Done false
3904 2006 Acked Read 2 Done Sent Done false
3905 2007 Acked Read 3 Done Sent Done false
3906 2008 Acked Write 5 Done Sent Done false
3907 2009 Acked Read 7 Done Sent Done false
3908 2010 Acked Read 10 Done Sent Done false
3909 2011 Acked Read 1 Done Sent Done false
3910 2012 Acked Write 9 Done Sent Done false
3911 2013 Acked Read 7 Done Sent Done false
3912 2014 Acked Read 1 Done Sent Done false
3913 2015 Acked Write 10 Done Sent Done false
3914 2016 Acked Flush 0 Done Sent Done false
3915 2017 Acked Read 3 Done Sent Done false
3916 2018 Acked Write 1 Done Sent Done false
3917 2019 Acked Write 10 Done Sent Done false
3918 2020 Acked Write 1 Done Sent Done false
3919 2021 Acked Write 8 Done Sent Done false
3920 2022 Acked Write 7 Done Sent Done false
3921 2023 Acked Flush 0 Done Sent Done false
3922 2024 Acked Read 7 Done Sent Done false
3923 2025 Acked Write 4 Done Sent Done false
3924 2026 Acked Write 3 Done Sent Done false
3925 2027 Acked Flush 0 Done Sent Done false
3926 2028 Acked Write 3 Done Sent Done false
3927 2029 Acked Write 1 Done Sent Done false
3928 2030 Acked Write 2 Done Sent Done false
3929 2031 Acked Write 6 Done Sent Done false
3930 2032 Acked Read 8 Done Sent Done false
3931 2033 Acked Read 1 Done Sent Done false
3932 2034 Acked Write 5 Done Sent Done false
3933 2035 Acked Flush 0 Done Sent Done false
3934 2036 Acked Write 1 Done Sent Done false
3935 2037 Acked Read 9 Done Sent Done false
3936 2038 Acked Read 9 Done Sent Done false
3937 2039 Acked Write 1 Done Sent Done false
3938 2040 Acked Read 8 Done Sent Done false
3939 2041 Acked Write 7 Done Sent Done false
3940 2042 Acked Read 3 Done Sent Done false
3941 2043 Acked Read 6 Done Sent Done false
3942 2044 Acked Read 1 Done Sent Done false
3943 2045 Acked Write 5 Done Sent Done false
3944 2046 Acked Read 1 Done Sent Done false
3945 2047 Acked Write 7 Done Sent Done false
3946 2048 Acked Write 5 Done Sent Done false
3947 2049 Acked Write 9 Done Sent Done false
3948 2050 Acked Read 8 Done Sent Done false
3949 2051 Acked Write 5 Done Sent Done false
3950 2052 Acked Write 2 Done Sent Done false
3951 2053 Acked Read 3 Done Sent Done false
3952 2054 Acked Write 4 Done Sent Done false
3953 2055 Acked Write 1 Done Sent Done false
3954 2056 Acked Read 4 Done Sent Done false
3955 2057 Acked Write 1 Done Sent Done false
3956 2058 Acked Flush 0 Done Sent Done false
3957 2059 Acked Read 8 Done Sent Done false
3958 2060 Acked Read 9 Done Sent Done false
3959 2061 Acked Write 8 Done Sent Done false
3960 2062 Acked Flush 0 Done Sent Done false
3961 2063 Acked Flush 0 Done Sent Done false
3962 2064 Acked Read 7 Done Sent Done false
3963 2065 Acked Read 9 Done Sent Done false
3964 2066 Acked Read 5 Done Sent Done false
3965 2067 Acked Read 1 Done Sent Done false
3966 2068 Acked Flush 0 Done Sent Done false
3967 2069 Acked Read 3 Done Sent Done false
3968 2070 Acked Read 2 Done Sent Done false
3969 2071 Acked Read 4 Done Sent Done false
3970 2072 Acked Write 8 Done Sent Done false
3971 2073 Acked Write 1 Done Sent Done false
3972 2074 Acked Read 9 Done Sent Done false
3973 2075 Acked Read 10 Done Sent Done false
3974 2076 Acked Write 7 Done Sent Done false
3975 2077 Acked Write 7 Done Sent Done false
3976 2078 Acked Read 10 Done Sent Done false
3977 2079 Acked Write 8 Done Sent Done false
3978 2080 Acked Write 10 Done Sent Done false
3979 2081 Acked Read 2 Done Sent Done false
3980 2082 Acked Read 6 Done Sent Done false
3981 2083 Acked Write 5 Done Sent Done false
3982 2084 Acked Read 9 Done Sent Done false
3983 2085 Acked Write 5 Done Sent Done false
3984 2086 Acked Read 9 Done Sent Done false
3985 2087 Acked Flush 0 Done Sent Done false
3986 2088 Acked Read 8 Done Sent Done false
3987 2089 Acked Read 7 Done Sent Done false
3988 2090 Acked Write 6 Done Sent Done false
3989 2091 Acked Read 10 Done Sent Done false
3990 2092 Acked Read 1 Done Sent Done false
3991 2093 Acked Read 5 Done Sent Done false
3992 2094 Acked Read 6 Done Sent Done false
3993 2095 Acked Read 10 Done Sent Done false
3994 2096 Acked Write 10 Done Sent Done false
3995 2097 Acked Write 8 Done Sent Done false
3996 2098 Acked Write 3 Done Sent Done false
3997 2099 Acked Write 9 Done Sent Done false
3998 2100 Acked Write 1 Done Sent Done false
3999 2101 Acked Write 4 Done Sent Done false
4000 2102 Acked Write 6 Done Sent Done false
4001 2103 Acked Read 5 Done Sent Done false
4002 2104 Acked Write 1 Done Sent Done false
4003 2105 Acked Read 2 Done Sent Done false
4004 2106 Acked Flush 0 Done Sent Done false
4005 2107 Acked Read 5 Done Sent Done false
4006 2108 Acked Read 5 Done Sent Done false
4007 2109 Acked Read 3 Done Sent Done false
4008 2110 Acked Read 1 Done Sent Done false
4009 2111 Acked Read 5 Done Sent Done false
4010 2112 Acked Write 10 Done Sent Done false
4011 2113 Acked Write 5 Done Sent Done false
4012 2114 Acked Write 9 Done Sent Done false
4013 2115 Acked Flush 0 Done Sent Done false
4014 2116 Acked Read 2 Done Sent Done false
4015 2117 Acked Write 5 Done Sent Done false
4016 2118 Acked Flush 0 Done Sent Done false
4017 2119 Acked Write 3 Done Sent Done false
4018 2120 Acked Read 4 Done Sent Done false
4019 2121 Acked Read 6 Done Sent Done false
4020 2122 Acked Read 4 Done Sent Done false
4021 2123 Acked Write 4 Done Sent Done false
4022 2124 Acked Read 10 Done Sent Done false
4023 2125 Acked Read 9 Done Sent Done false
4024 2126 Acked Read 7 Done Sent Done false
4025 2127 Acked Write 7 Done Sent Done false
4026 2128 Acked Read 3 Done Sent Done false
4027 2129 Acked Read 6 Done Sent Done false
4028 2130 Acked Read 4 Done Sent Done false
4029 2131 Acked Read 9 Done Sent Done false
4030 2132 Acked Read 7 Done Sent Done false
4031 2133 Acked Write 1 Done Sent Done false
4032 2134 Acked Write 9 Done Sent Done false
4033 2135 Acked Write 1 Done Sent Done false
4034 2136 Acked Write 3 Done Sent Done false
4035 2137 Acked Read 7 Done Sent Done false
4036 2138 Acked Write 2 Done Sent Done false
4037 2139 Acked Read 9 Done Sent Done false
4038 2140 Acked Write 6 Done Sent Done false
4039 2141 Acked Read 3 Done Sent Done false
4040 2142 Acked Write 2 Done Sent Done false
4041 2143 Acked Write 5 Done Sent Done false
4042 2144 Acked Read 8 Done Sent Done false
4043 2145 Acked Write 5 Done Sent Done false
4044 2146 Acked Read 9 Done Sent Done false
4045 2147 Acked Read 10 Done Sent Done false
4046 2148 Acked Write 6 Done Sent Done false
4047 2149 Acked Read 7 Done Sent Done false
4048 2150 Acked Write 9 Done Sent Done false
4049 2151 Acked Flush 0 Done Sent Done false
4050 2152 Acked Read 9 Done Sent Done false
4051 2153 Acked Read 4 Done Sent Done false
4052 2154 Acked Write 2 Done Sent Done false
4053 2155 Acked Write 1 Done Sent Done false
4054 2156 Acked Read 2 Done Sent Done false
4055 2157 Acked Read 6 Done Sent Done false
4056 2158 Acked Write 4 Done Sent Done false
4057 2159 Acked Flush 0 Done Sent Done false
4058 2160 Acked Flush 0 Done Sent Done false
4059 2161 Acked Read 7 Done Sent Done false
4060 2162 Acked Write 4 Done Sent Done false
4061 2163 Acked Flush 0 Done Sent Done false
4062 2164 Acked Read 2 Done Sent Done false
4063 2165 Acked Write 1 Done Sent Done false
4064 2166 Acked Flush 0 Done Sent Done false
4065 2167 Acked Write 6 Done Sent Done false
4066 2168 Acked Write 8 Done Sent Done false
4067 2169 Acked Write 10 Done Sent Done false
4068 2170 Acked Write 6 Done Sent Done false
4069 2171 Acked Write 5 Done Sent Done false
4070 2172 Acked Write 9 Done Sent Done false
4071 2173 Acked Read 4 Done Sent Done false
4072 2174 Acked Write 9 Done Sent Done false
4073 2175 Acked Read 5 Done Sent Done false
4074 2176 Acked Read 8 Done Sent Done false
4075 2177 Acked Write 5 Done Sent Done false
4076 2178 Acked Write 5 Done Sent Done false
4077 2179 Acked Read 4 Done Sent Done false
4078 2180 Acked Write 5 Done Sent Done false
4079 2181 Acked Write 5 Done Sent Done false
4080 2182 Acked Read 9 Done Sent Done false
4081 2183 Acked Read 10 Done Sent Done false
4082 2184 Acked Read 9 Done Sent Done false
4083 2185 Acked Flush 0 Done Sent Done false
4084 2186 Acked Read 3 Done Sent Done false
4085 2187 Acked Read 1 Done Sent Done false
4086 2188 Acked Write 3 Done Sent Done false
4087 2189 Acked Write 3 Done Sent Done false
4088 2190 Acked Read 7 Done Sent Done false
4089 2191 Acked Read 1 Done Sent Done false
4090 2192 Acked Write 3 Done Sent Done false
4091 2193 Acked Read 8 Done Sent Done false
4092 2194 Acked Read 6 Done Sent Done false
4093 2195 Acked Write 8 Done Sent Done false
4094 2196 Acked Read 7 Done Sent Done false
4095 2197 Acked Read 3 Done Sent Done false
4096 2198 Acked Read 2 Done Sent Done false
4097 2199 Acked Write 1 Done Sent Done false
4098 2200 Acked Write 10 Done Sent Done false
4099 2201 Acked Write 10 Done Sent Done false
4100 2202 Acked Read 6 Done Sent Done false
4101 2203 Acked Write 4 Done Sent Done false
4102 2204 Acked Write 2 Done Sent Done false
4103 2205 Acked Write 9 Done Sent Done false
4104 2206 Acked Read 10 Done Sent Done false
4105 2207 Acked Flush 0 Done Sent Done false
4106 STATES DS:0 DS:1 DS:2 TOTAL
4107 Sent 0 302 0 302
4108 Done 302 0 302 604
4109 Skipped 0 0 0 0
4110 Error 0 0 0 0
4111 Last Flush: 2207 1905 2207
4112 Downstairs last five completed: 1905 1904 1903 1902 1901
4113 Upstairs last five completed: 2207 2206 2205 2204 2203
4114 CLIENT: Up:0 ds:302 act:2
41152024-12-09T15:08:41.847ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
41162024-12-09T15:08:41.847ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52383) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
41172024-12-09T15:08:41.848ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
41182024-12-09T15:08:41.849ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
41192024-12-09T15:08:41.849ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
41202024-12-09T15:08:41.849ZINFOcrucible: send last flush ID to this DS: 1905 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
41212024-12-09T15:08:41.849ZINFOcrucible: Replied this last flush ID: 1905 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
41222024-12-09T15:08:41.849ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
41232024-12-09T15:08:41.849ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
41242024-12-09T15:08:41.849ZINFOcrucible: [1] client re-new 302 jobs since flush 1905 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
41252024-12-09T15:08:41.849ZINFOcrucible: [1] Replayed 302 jobs since flush: 1905 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
4126 ----------------------------------------------------------------
4127 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4128 Downstairs last five completed: 2207 2206 2205 2204 2203
4129 Upstairs last five completed: 2207 2206 2205 2204 2203
4130 CLIENT: Up:0 ds:0 act:3
4131 Replay: All jobs finished
4132 Main task has work to do, go find it
4133 got dsc Stop(0)
4134 stop 0
4135 Dec 09 15:08:44.092 INFO request completed, latency_us: 363, response_code: 204, uri: /stop/cid/0, method: POST, req_id: ac1a357c-947c-45e7-8ca4-6583559cbbd2[8810] Got stop action so:false kr:false
4136 , remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
41372024-12-09T15:08:44.093ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
4138 Dec 09 15:08:44.093 INFO request completed, latency_us: 120, response_code: 200{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:08:44.093642911Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"0","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
4139 , uri: /state/cid/0, method: GET, req_id: 86f40be0-7d3b-4d82-bb06-0a8a32b5d5d7, remote_addr: 127.0.{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:08:44.093677125Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
4140 0{"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:08:44.093713678Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"0","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
4141 .1:47510, local_addr: 127.0.0.1:9998
41422024-12-09T15:08:44.093ZINFOcrucible: [0] Transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
41432024-12-09T15:08:44.093ZINFOcrucible: client task is exiting = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
4144 [8810] Exited with: ExitStatus(unix_wait_status(9))
4145 [8810][0] reports Exit
4146 Dec 09 15:08:48.095 INFO request completed, latency_us: 273, response_code: 200, uri: /state/cid/0, method: GET, req_id: 5cb3d65e-aaed-4f41-9c1d-eb4430f34074, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
4147 001/300 Read block 1881 len 8192
4148 002/300 Read block 4517 len 12288
4149 003/300 Flush
4150 004/300 Flush
4151 005/300 Write block 3116 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4152 006/300 Read block 4996 len 16384
4153 007/300 Read block 2609 len 40960
4154 008/300 Read block 68 len 32768
4155 009/300 Write block 3140 len 20480 data: 3 3 2 2 3
4156 010/300 Flush
4157 011/300 Write block 3157 len 28672 data: 3 3 3 4 4 4 3
4158 012/300 Read block 2603 len 36864
4159 013/300 Read block 4781 len 28672
4160 014/300 Read block 4928 len 12288
4161 015/300 Read block 352 len 32768
4162 016/300 Write block 3797 len 28672 data: 2 2 3 3 4 4 3
4163 017/300 Read block 749 len 40960
4164 018/300 Write block 3548 len 8192 data: 2 3
4165 019/300 Flush
4166 020/300 Write block 2134 len 20480 data: 4 5 4 4 4
4167 021/300 Read block 3716 len 12288
4168 022/300 Write block 750 len 12288 data: 2 2 2
4169 023/300 Read block 1532 len 28672
4170 024/300 Read block 3250 len 20480
4171 025/300 Write block 2217 len 40960 data: 3 4 3 3 3 4 4 4 4 3
4172 026/300 Read block 958 len 28672
4173 027/300 Read block 2117 len 36864
4174 028/300 Write block 3490 len 24576 data: 3 2 2 2 2 2
4175 029/300 Write block 3921 len 16384 data: 2 2 2 2
4176 030/300 Read block 3934 len 24576
4177 031/300 Read block 4337 len 4096
4178 032/300 Read block 363 len 12288
4179 033/300 Read block 1416 len 28672
4180 034/300 Read block 1102 len 36864
4181 035/300 Read block 4455 len 24576
4182 036/300 Flush
4183 037/300 Read block 4712 len 16384
4184 038/300 Write block 53 len 36864 data: 2 2 2 2 2 2 2 2 2
4185 039/300 Read block 4907 len 40960
4186 040/300 Write block 1274 len 24576 data: 4 4 4 4 3 4
4187 041/300 Read block 3910 len 24576
4188 042/300 Write block 2373 len 28672 data: 2 2 2 2 2 2 2
4189 043/300 Read block 1090 len 28672
4190 044/300 Read block 946 len 24576
4191 045/300 Flush
4192 046/300 Read block 1448 len 16384
4193 047/300 Write block 2018 len 24576 data: 3 3 3 3 3 3
4194 048/300 Read block 2971 len 20480
4195 049/300 Read block 1854 len 12288
4196 050/300 Write block 1710 len 36864 data: 3 3 2 3 3 3 3 2 2
4197 051/300 Flush
4198 052/300 Read block 4687 len 4096
4199 053/300 Write block 27 len 36864 data: 3 3 3 3 3 3 2 2 2
4200 054/300 Read block 2794 len 20480
4201 055/300 Read block 2694 len 24576
4202 056/300 Read block 1230 len 8192
4203 057/300 Read block 4366 len 8192
4204 058/300 Read block 2072 len 28672
4205 059/300 Write block 3209 len 4096 data: 2
4206 060/300 Write block 1978 len 32768 data: 3 3 3 4 4 4 4 4
4207 061/300 Read block 1175 len 32768
4208 062/300 Write block 2727 len 36864 data: 2 3 3 3 3 3 3 3 4
4209 063/300 Write block 1792 len 12288 data: 2 2 2
4210 064/300 Read block 3157 len 32768
4211 065/300 Read block 2733 len 8192
4212 066/300 Read block 1774 len 20480
4213 067/300 Read block 3517 len 24576
4214 068/300 Write block 2525 len 32768 data: 3 3 2 2 2 2 2 2
4215 069/300 Read block 28 len 36864
4216 070/300 Read block 853 len 4096
4217 071/300 Write block 2983 len 20480 data: 3 3 4 4 5
4218 072/300 Write block 2073 len 20480 data: 2 3 2 2 2
4219 073/300 Write block 4616 len 16384 data: 2 2 2 3
4220 074/300 Write block 3745 len 12288 data: 3 3 2
4221 075/300 Read block 3924 len 8192
4222 076/300 Read block 2181 len 20480
4223 077/300 Read block 2473 len 20480
4224 078/300 Write block 2684 len 20480 data: 3 3 3 3 3
4225 079/300 Write block 901 len 16384 data: 3 3 2 2
4226 080/300 Write block 1807 len 12288 data: 4 3 3
4227 081/300 Read block 2192 len 32768
4228 082/300 Flush
4229 083/300 Write block 3682 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4230 084/300 Write block 4485 len 28672 data: 2 2 2 2 2 2 2
4231 085/300 Write block 4964 len 8192 data: 4 2
4232 086/300 Read block 788 len 36864
4233 087/300 Write block 2711 len 24576 data: 3 3 3 2 2 2
4234 088/300 Write block 3885 len 20480 data: 3 3 2 2 2
4235 089/300 Read block 2768 len 40960
4236 090/300 Read block 1591 len 40960
4237 091/300 Read block 1521 len 12288
4238 092/300 Write block 4644 len 36864 data: 2 2 2 2 2 2 2 2 2
4239 093/300 Write block 1731 len 40960 data: 2 2 3 3 2 2 2 2 2 3
4240 094/300 Flush
4241 095/300 Read block 2155 len 16384
4242 096/300 Write block 4046 len 28672 data: 2 2 2 2 2 2 2
4243 097/300 Read block 4117 len 16384
4244 098/300 Read block 451 len 40960
4245 099/300 Write block 1751 len 32768 data: 2 2 3 2 2 2 2 2
4246 100/300 Write block 4872 len 12288 data: 3 2 2
4247 101/300 Read block 3843 len 40960
4248 102/300 Read block 3070 len 8192
4249 103/300 Read block 4847 len 28672
4250 104/300 Read block 4019 len 24576
4251 105/300 Write block 2778 len 32768 data: 2 2 2 2 2 2 2 2
4252 106/300 Read block 3473 len 16384
4253 107/300 Write block 1867 len 20480 data: 2 2 2 2 2
4254 108/300 Flush
4255 109/300 Read block 617 len 20480
4256 110/300 Read block 1260 len 40960
4257 111/300 Write block 1595 len 36864 data: 2 2 2 2 2 2 2 3 3
4258 112/300 Write block 2317 len 20480 data: 2 2 2 2 2
4259 113/300 Read block 2899 len 36864
4260 114/300 Read block 1492 len 8192
4261 115/300 Write block 3844 len 28672 data: 3 3 4 3 3 3 3
4262 116/300 Write block 3543 len 36864 data: 3 3 3 2 2 3 4 3 3
4263 117/300 Read block 1529 len 4096
4264 118/300 Read block 526 len 4096
4265 119/300 Write block 124 len 4096 data: 3
4266 120/300 Read block 999 len 32768
4267 121/300 Write block 3581 len 28672 data: 3 3 4 4 3 3 3
4268 122/300 Read block 829 len 36864
4269 123/300 Write block 1282 len 16384 data: 3 3 3 3
4270 124/300 Write block 2179 len 20480 data: 3 3 3 3 3
4271 125/300 Write block 3932 len 28672 data: 3 2 2 2 2 2 2
4272 126/300 Read block 3672 len 40960
4273 127/300 Read block 3593 len 16384
4274 128/300 Write block 838 len 20480 data: 2 2 2 2 2
4275 129/300 Flush
4276 130/300 Write block 3197 len 28672 data: 2 2 2 2 2 2 2
4277 131/300 Read block 1622 len 4096
4278 132/300 Read block 3786 len 12288
4279 133/300 Read block 754 len 12288
4280 134/300 Write block 1834 len 8192 data: 2 3
4281 135/300 Write block 1688 len 36864 data: 2 2 2 2 2 2 2 2 2
4282 136/300 Write block 4643 len 32768 data: 3 3 3 3 3 3 3 3
4283 137/300 Write block 4550 len 28672 data: 2 2 2 2 2 2 3
4284 138/300 Write block 2105 len 16384 data: 3 3 3 3
4285 139/300 Flush
4286 140/300 Write block 3431 len 20480 data: 3 3 3 4 4
4287 141/300 Write block 1534 len 24576 data: 4 3 3 3 2 2
4288 142/300 Read block 4340 len 40960
4289 143/300 Read block 646 len 16384
4290 144/300 Flush
4291 145/300 Write block 2916 len 28672 data: 2 3 3 3 3 3 3
4292 146/300 Write block 3339 len 24576 data: 3 3 4 4 4 4
4293 147/300 Write block 1768 len 36864 data: 2 3 2 2 2 2 2 2 2
4294 148/300 Read block 2780 len 40960
4295 149/300 Write block 1105 len 40960 data: 2 2 2 2 2 2 2 3 3 3
4296 150/300 Write block 420 len 28672 data: 2 2 2 2 2 3 3
4297 151/300 Read block 1488 len 4096
4298 152/300 Read block 3013 len 36864
4299 153/300 Write block 2474 len 40960 data: 2 3 3 3 3 3 2 2 2 2
4300 154/300 Write block 4707 len 36864 data: 3 3 3 2 2 2 3 3 2
4301 155/300 Read block 4176 len 4096
4302 156/300 Write block 149 len 32768 data: 2 2 2 2 2 3 2 3
4303 157/300 Read block 2911 len 4096
4304 158/300 Read block 1723 len 16384
4305 159/300 Write block 3305 len 32768 data: 3 3 3 3 3 2 2 2
4306 160/300 Read block 4640 len 8192
4307 161/300 Flush
4308 162/300 Read block 2013 len 40960
4309 163/300 Write block 2128 len 32768 data: 3 3 4 4 4 4 5 6
4310 164/300 Read block 3355 len 4096
4311 165/300 Write block 1553 len 4096 data: 2
4312 166/300 Write block 810 len 12288 data: 2 2 2
4313 167/300 Write block 1045 len 4096 data: 3
4314 168/300 Write block 3394 len 36864 data: 3 3 3 3 5 5 3 3 3
4315 169/300 Write block 18 len 28672 data: 2 2 2 2 2 2 2
4316 170/300 Read block 3600 len 24576
4317 171/300 Read block 3175 len 36864
4318 172/300 Read block 1472 len 8192
4319 173/300 Read block 2494 len 24576
4320 174/300 Read block 4955 len 20480
4321 175/300 Flush
4322 176/300 Flush
4323 177/300 Write block 2771 len 8192 data: 2 2
4324 178/300 Read block 1903 len 36864
4325 179/300 Flush
4326 180/300 Write block 3992 len 16384 data: 3 3 3 3
4327 181/300 Read block 3586 len 16384
4328 182/300 Write block 4433 len 24576 data: 3 3 3 3 3 3
4329 183/300 Read block 728 len 28672
4330 184/300 Write block 3454 len 12288 data: 2 2 2
4331 185/300 Read block 2069 len 36864
4332 186/300 Read block 4078 len 4096
4333 187/300 Read block 1812 len 20480
4334 188/300 Read block 1659 len 32768
4335 189/300 Read block 1693 len 8192
4336 190/300 Write block 2165 len 20480 data: 2 2 2 2 2
4337 191/300 Read block 4404 len 4096
4338 192/300 Read block 1488 len 4096
4339 193/300 Write block 3215 len 12288 data: 2 2 2
4340 194/300 Read block 2409 len 36864
4341 195/300 Write block 1607 len 40960 data: 2 2 3 3 3 3 3 3 3 2
4342 196/300 Write block 1926 len 8192 data: 4 4
4343 197/300 Read block 3888 len 8192
4344 198/300 Read block 3299 len 24576
4345 199/300 Flush
4346 200/300 Read block 4877 len 28672
4347 201/300 Write block 4337 len 24576 data: 2 2 2 2 2 2
4348 202/300 Read block 4926 len 20480
4349 203/300 Read block 4940 len 36864
4350 204/300 Write block 4133 len 4096 data: 2
4351 205/300 Write block 2348 len 12288 data: 2 2 2
4352 206/300 Read block 2147 len 12288
4353 207/300 Read block 2968 len 20480
4354 208/300 Write block 915 len 36864 data: 3 3 3 3 2 2 3 3 3
4355 209/300 Write block 3607 len 32768 data: 3 2 2 2 2 3 3 3
4356 210/300 Read block 2034 len 20480
4357 211/300 Read block 842 len 8192
4358 212/300 Write block 3829 len 28672 data: 3 2 2 3 3 3 3
4359 213/300 Read block 3728 len 16384
4360 214/300 Read block 2944 len 8192
4361 215/300 Read block 4114 len 8192
4362 216/300 Read block 4141 len 16384
4363 217/300 Write block 482 len 28672 data: 4 3 3 3 2 2 2
4364 218/300 Read block 2271 len 4096
4365 219/300 Flush
4366 220/300 Read block 1625 len 24576
4367 221/300 Read block 4748 len 20480
4368 222/300 Write block 4650 len 40960 data: 4 3 3 2 3 2 2 2 2 2
4369 223/300 Read block 337 len 4096
4370 224/300 Write block 3322 len 32768 data: 3 2 2 2 2 2 2 2
4371 225/300 Write block 2769 len 8192 data: 2 2
4372 226/300 Read block 4850 len 40960
4373 227/300 Read block 522 len 8192
4374 228/300 Flush
4375 229/300 Write block 231 len 4096 data: 3
4376 230/300 Write block 3309 len 20480 data: 4 3 3 3 2
4377 231/300 Write block 2008 len 36864 data: 4 4 3 3 2 2 3 3 3
4378 232/300 Read block 1869 len 8192
4379 233/300 Write block 2107 len 4096 data: 4
4380 234/300 Write block 3283 len 32768 data: 2 2 2 3 3 3 3 3
4381 235/300 Write block 3800 len 4096 data: 4
4382 236/300 Write block 4116 len 8192 data: 2 2
4383 237/300 Read block 760 len 16384
4384 238/300 Read block 2983 len 40960
4385 239/300 Read block 2664 len 4096
4386 240/300 Write block 2131 len 36864 data: 5 5 5 6 7 5 5 5 3
4387 241/300 Write block 1561 len 40960 data: 5 5 5 5 3 3 2 4 4 4
4388 242/300 Write block 1332 len 8192 data: 2 2
4389 243/300 Write block 2946 len 24576 data: 3 3 3 3 3 3
4390 244/300 Read block 3139 len 32768
4391 245/300 Flush
4392 246/300 Read block 652 len 16384
4393 247/300 Write block 648 len 32768 data: 2 2 2 2 2 2 2 2
4394 248/300 Read block 4167 len 40960
4395 249/300 Read block 506 len 36864
4396 250/300 Read block 3400 len 12288
4397 251/300 Write block 1417 len 8192 data: 3 3
4398 252/300 Read block 1846 len 16384
4399 253/300 Read block 4946 len 20480
4400 254/300 Read block 4271 len 40960
4401 255/300 Read block 4802 len 36864
4402 256/300 Write block 2016 len 8192 data: 4 3
4403 257/300 Write block 3483 len 12288 data: 3 3 3
4404 258/300 Read block 276 len 40960
4405 259/300 Read block 3317 len 36864
4406 260/300 Write block 1354 len 28672 data: 2 2 2 2 2 2 2
4407 261/300 Read block 1710 len 28672
4408 262/300 Read block 4063 len 28672
4409 263/300 Write block 495 len 16384 data: 3 2 2 2
4410 264/300 Read block 930 len 32768
4411 265/300 Read block 2637 len 24576
4412 266/300 Read block 2087 len 8192
4413 267/300 Read block 711 len 12288
4414 268/300 Read block 3155 len 36864
4415 269/300 Write block 1300 len 24576 data: 2 2 2 2 2 2
4416 270/300 Write block 756 len 12288 data: 3 3 4
4417 271/300 Write block 629 len 36864 data: 2 2 2 2 2 2 2 2 3
4418 272/300 Read block 2246 len 36864
4419 273/300 Write block 3614 len 40960 data: 4 3 3 3 3 3 3 3 3 3
4420 274/300 Write block 2760 len 8192 data: 2 2
4421 275/300 Write block 3592 len 32768 data: 2 2 2 2 3 3 3 3
4422 276/300 Write block 1648 len 36864 data: 2 2 2 3 3 3 3 3 3
4423 277/300 Read block 242 len 36864
4424 278/300 Read block 946 len 8192
4425 279/300 Write block 4196 len 16384 data: 2 2 2 2
4426 280/300 Write block 3629 len 24576 data: 2 2 2 2 2 2
4427 281/300 Read block 3371 len 40960
4428 282/300 Write block 2312 len 36864 data: 2 2 2 2 2 3 3 3 3
4429 283/300 Read block 17 len 28672
4430 284/300 Write block 3263 len 12288 data: 2 2 2
4431 285/300 Read block 4722 len 32768
4432 286/300 Read block 1716 len 36864
4433 287/300 Read block 736 len 12288
4434 288/300 Read block 2931 len 40960
4435 289/300 Read block 3259 len 32768
4436 290/300 Write block 1147 len 20480 data: 3 4 5 5 5
4437 291/300 Write block 1516 len 24576 data: 2 2 2 2 2 2
4438 292/300 Read block 4431 len 20480
4439 293/300 Read block 685 len 24576
4440 294/300 Read block 2598 len 8192
4441 295/300 Flush
4442 296/300 Write block 3242 len 16384 data: 2 3 3 3
4443 297/300 Read block 965 len 28672
4444 298/300 Write block 131 len 20480 data: 2 3 5 4 4
4445 299/300 Read block 2382 len 24576
4446 300/300 Read block 3912 len 8192
4447 301/300 Write block 671 len 4096 data: 2
4448 Main task has work to do, go find it
4449 got dsc Start(0)
4450 start 0
4451 Dec 09 15:08:48.341 INFO request completed, latency_us: 221, response_code: 204, uri: /start/cid/0, method: POST, req_id: 1fdfa397-cb44-4202-b528-e4a3a53d16c8, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
4452 Replay: started 0, returned:Ok(())
4453 ----------------------------------------------------------------
4454 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4455 States: Offline Active Active
4456 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4457 2208 Acked Read 2 Sent Done Done false
4458 2209 Acked Read 3 Sent Done Done false
4459 2210 Acked Flush 0 Sent Done Done false
4460 2211 Acked Flush 0 Sent Done Done false
4461 2212 Acked Write 10 Sent Done Done false
4462 2213 Acked Read 4 Sent Done Done false
4463 2214 Acked Read 10 Sent Done Done false
4464 2215 Acked Read 8 Sent Done Done false
4465 2216 Acked Write 5 Sent Done Done false
4466 2217 Acked Flush 0 Sent Done Done false
4467 2218 Acked Write 7 Sent Done Done false
4468 2219 Acked Read 9 Sent Done Done false
4469 2220 Acked Read 7 Sent Done Done false
4470 2221 Acked Read 3 Sent Done Done false
4471 2222 Acked Read 8 Sent Done Done false
4472 2223 Acked Write 7 Sent Done Done false
4473 2224 Acked Read 10 Sent Done Done false
4474 2225 Acked Write 2 Sent Done Done false
4475 2226 Acked Flush 0 Sent Done Done false
4476 2227 Acked Write 5 Sent Done Done false
4477 2228 Acked Read 3 Sent Done Done false
4478 2229 Acked Write 3 Sent Done Done false
4479 2230 Acked Read 7 Sent Done Done false
4480 2231 Acked Read 5 Sent Done Done false
4481 2232 Acked Write 10 Sent Done Done false
4482 2233 Acked Read 7 Sent Done Done false
4483 2234 Acked Read 9 Sent Done Done false
4484 2235 Acked Write 6 Sent Done Done false
4485 2236 Acked Write 4 Sent Done Done false
4486 2237 Acked Read 6 Sent Done Done false
4487 2238 Acked Read 1 Sent Done Done false
4488 2239 Acked Read 3 Sent Done Done false
4489 2240 Acked Read 7 Sent Done Done false
4490 2241 Acked Read 9 Sent Done Done false
4491 2242 Acked Read 6 Sent Done Done false
4492 2243 Acked Flush 0 Sent Done Done false
4493 2244 Acked Read 4 Sent Done Done false
4494 2245 Acked Write 9 Sent Done Done false
4495 2246 Acked Read 10 Sent Done Done false
4496 2247 Acked Write 6 Sent Done Done false
4497 2248 Acked Read 6 Sent Done Done false
4498 2249 Acked Write 7 Sent Done Done false
4499 2250 Acked Read 7 Sent Done Done false
4500 2251 Acked Read 6 Sent Done Done false
4501 2252 Acked Flush 0 Sent Done Done false
4502 2253 Acked Read 4 Sent Done Done false
4503 2254 Acked Write 6 Sent Done Done false
4504 2255 Acked Read 5 Sent Done Done false
4505 2256 Acked Read 3 Sent Done Done false
4506 2257 Acked Write 9 Sent Done Done false
4507 2258 Acked Flush 0 Sent Done Done false
4508 2259 Acked Read 1 Sent Done Done false
4509 2260 Acked Write 9 Sent Done Done false
4510 2261 Acked Read 5 Sent Done Done false
4511 2262 Acked Read 6 Sent Done Done false
4512 2263 Acked Read 2 Sent Done Done false
4513 2264 Acked Read 2 Sent Done Done false
4514 2265 Acked Read 7 Sent Done Done false
4515 2266 Acked Write 1 Sent Done Done false
4516 2267 Acked Write 8 Sent Done Done false
4517 2268 Acked Read 8 Sent Done Done false
4518 2269 Acked Write 9 Sent Done Done false
4519 2270 Acked Write 3 Sent Done Done false
4520 2271 Acked Read 8 Sent Done Done false
4521 2272 Acked Read 2 Sent Done Done false
4522 2273 Acked Read 5 Sent Done Done false
4523 2274 Acked Read 6 Sent Done Done false
4524 2275 Acked Write 8 Sent Done Done false
4525 2276 Acked Read 9 Sent Done Done false
4526 2277 Acked Read 1 Sent Done Done false
4527 2278 Acked Write 5 Sent Done Done false
4528 2279 Acked Write 5 Sent Done Done false
4529 2280 Acked Write 4 Sent Done Done false
4530 2281 Acked Write 3 Sent Done Done false
4531 2282 Acked Read 2 Sent Done Done false
4532 2283 Acked Read 5 Sent Done Done false
4533 2284 Acked Read 5 Sent Done Done false
4534 2285 Acked Write 5 Sent Done Done false
4535 2286 Acked Write 4 Sent Done Done false
4536 2287 Acked Write 3 Sent Done Done false
4537 2288 Acked Read 8 Sent Done Done false
4538 2289 Acked Flush 0 Sent Done Done false
4539 2290 Acked Write 10 Sent Done Done false
4540 2291 Acked Write 7 Sent Done Done false
4541 2292 Acked Write 2 Sent Done Done false
4542 2293 Acked Read 9 Sent Done Done false
4543 2294 Acked Write 6 Sent Done Done false
4544 2295 Acked Write 5 Sent Done Done false
4545 2296 Acked Read 10 Sent Done Done false
4546 2297 Acked Read 10 Sent Done Done false
4547 2298 Acked Read 3 Sent Done Done false
4548 2299 Acked Write 9 Sent Done Done false
4549 2300 Acked Write 10 Sent Done Done false
4550 2301 Acked Flush 0 Sent Done Done false
4551 2302 Acked Read 4 Sent Done Done false
4552 2303 Acked Write 7 Sent Done Done false
4553 2304 Acked Read 4 Sent Done Done false
4554 2305 Acked Read 10 Sent Done Done false
4555 2306 Acked Write 8 Sent Done Done false
4556 2307 Acked Write 3 Sent Done Done false
4557 2308 Acked Read 10 Sent Done Done false
4558 2309 Acked Read 2 Sent Done Done false
4559 2310 Acked Read 7 Sent Done Done false
4560 2311 Acked Read 6 Sent Done Done false
4561 2312 Acked Write 8 Sent Done Done false
4562 2313 Acked Read 4 Sent Done Done false
4563 2314 Acked Write 5 Sent Done Done false
4564 2315 Acked Flush 0 Sent Done Done false
4565 2316 Acked Read 5 Sent Done Done false
4566 2317 Acked Read 10 Sent Done Done false
4567 2318 Acked Write 9 Sent Done Done false
4568 2319 Acked Write 5 Sent Done Done false
4569 2320 Acked Read 9 Sent Done Done false
4570 2321 Acked Read 2 Sent Done Done false
4571 2322 Acked Write 7 Sent Done Done false
4572 2323 Acked Write 9 Sent Done Done false
4573 2324 Acked Read 1 Sent Done Done false
4574 2325 Acked Read 1 Sent Done Done false
4575 2326 Acked Write 1 Sent Done Done false
4576 2327 Acked Read 8 Sent Done Done false
4577 2328 Acked Write 7 Sent Done Done false
4578 2329 Acked Read 9 Sent Done Done false
4579 2330 Acked Write 4 Sent Done Done false
4580 2331 Acked Write 5 Sent Done Done false
4581 2332 Acked Write 7 Sent Done Done false
4582 2333 Acked Read 10 Sent Done Done false
4583 2334 Acked Read 4 Sent Done Done false
4584 2335 Acked Write 5 Sent Done Done false
4585 2336 Acked Flush 0 Sent Done Done false
4586 2337 Acked Write 7 Sent Done Done false
4587 2338 Acked Read 1 Sent Done Done false
4588 2339 Acked Read 3 Sent Done Done false
4589 2340 Acked Read 3 Sent Done Done false
4590 2341 Acked Write 2 Sent Done Done false
4591 2342 Acked Write 9 Sent Done Done false
4592 2343 Acked Write 8 Sent Done Done false
4593 2344 Acked Write 7 Sent Done Done false
4594 2345 Acked Write 4 Sent Done Done false
4595 2346 Acked Flush 0 Sent Done Done false
4596 2347 Acked Write 5 Sent Done Done false
4597 2348 Acked Write 6 Sent Done Done false
4598 2349 Acked Read 10 Sent Done Done false
4599 2350 Acked Read 4 Sent Done Done false
4600 2351 Acked Flush 0 Sent Done Done false
4601 2352 Acked Write 7 Sent Done Done false
4602 2353 Acked Write 6 Sent Done Done false
4603 2354 Acked Write 9 Sent Done Done false
4604 2355 Acked Read 10 Sent Done Done false
4605 2356 Acked Write 10 Sent Done Done false
4606 2357 Acked Write 7 Sent Done Done false
4607 2358 Acked Read 1 Sent Done Done false
4608 2359 Acked Read 9 Sent Done Done false
4609 2360 Acked Write 10 Sent Done Done false
4610 2361 Acked Write 9 Sent Done Done false
4611 2362 Acked Read 1 Sent Done Done false
4612 2363 Acked Write 8 Sent Done Done false
4613 2364 Acked Read 1 Sent Done Done false
4614 2365 Acked Read 4 Sent Done Done false
4615 2366 Acked Write 8 Sent Done Done false
4616 2367 Acked Read 2 Sent Done Done false
4617 2368 Acked Flush 0 Sent Done Done false
4618 2369 Acked Read 10 Sent Done Done false
4619 2370 Acked Write 8 Sent Done Done false
4620 2371 Acked Read 1 Sent Done Done false
4621 2372 Acked Write 1 Sent Done Done false
4622 2373 Acked Write 3 Sent Done Done false
4623 2374 Acked Write 1 Sent Done Done false
4624 2375 Acked Write 9 Sent Done Done false
4625 2376 Acked Write 7 Sent Done Done false
4626 2377 Acked Read 6 Sent Done Done false
4627 2378 Acked Read 9 Sent Done Done false
4628 2379 Acked Read 2 Sent Done Done false
4629 2380 Acked Read 6 Sent Done Done false
4630 2381 Acked Read 5 Sent Done Done false
4631 2382 Acked Flush 0 Sent Done Done false
4632 2383 Acked Flush 0 Sent Done Done false
4633 2384 Acked Write 2 Sent Done Done false
4634 2385 Acked Read 9 Sent Done Done false
4635 2386 Acked Flush 0 Sent Done Done false
4636 2387 Acked Write 4 Sent Done Done false
4637 2388 Acked Read 4 Sent Done Done false
4638 2389 Acked Write 6 Sent Done Done false
4639 2390 Acked Read 7 Sent Done Done false
4640 2391 Acked Write 3 Sent Done Done false
4641 2392 Acked Read 9 Sent Done Done false
4642 2393 Acked Read 1 Sent Done Done false
4643 2394 Acked Read 5 Sent Done Done false
4644 2395 Acked Read 8 Sent Done Done false
4645 2396 Acked Read 2 Sent Done Done false
4646 2397 Acked Write 5 Sent Done Done false
4647 2398 Acked Read 1 Sent Done Done false
4648 2399 Acked Read 1 Sent Done Done false
4649 2400 Acked Write 3 Sent Done Done false
4650 2401 Acked Read 9 Sent Done Done false
4651 2402 Acked Write 10 Sent Done Done false
4652 2403 Acked Write 2 Sent Done Done false
4653 2404 Acked Read 2 Sent Done Done false
4654 2405 Acked Read 6 Sent Done Done false
4655 2406 Acked Flush 0 Sent Done Done false
4656 2407 Acked Read 7 Sent Done Done false
4657 2408 Acked Write 6 Sent Done Done false
4658 2409 Acked Read 5 Sent Done Done false
4659 2410 Acked Read 9 Sent Done Done false
4660 2411 Acked Write 1 Sent Done Done false
4661 2412 Acked Write 3 Sent Done Done false
4662 2413 Acked Read 3 Sent Done Done false
4663 2414 Acked Read 5 Sent Done Done false
4664 2415 Acked Write 9 Sent Done Done false
4665 2416 Acked Write 8 Sent Done Done false
4666 2417 Acked Read 5 Sent Done Done false
4667 2418 Acked Read 2 Sent Done Done false
4668 2419 Acked Write 7 Sent Done Done false
4669 2420 Acked Read 4 Sent Done Done false
4670 2421 Acked Read 2 Sent Done Done false
4671 2422 Acked Read 2 Sent Done Done false
4672 2423 Acked Read 4 Sent Done Done false
4673 2424 Acked Write 7 Sent Done Done false
4674 2425 Acked Read 1 Sent Done Done false
4675 2426 Acked Flush 0 Sent Done Done false
4676 2427 Acked Read 6 Sent Done Done false
4677 2428 Acked Read 5 Sent Done Done false
4678 2429 Acked Write 10 Sent Done Done false
4679 2430 Acked Read 1 Sent Done Done false
4680 2431 Acked Write 8 Sent Done Done false
4681 2432 Acked Write 2 Sent Done Done false
4682 2433 Acked Read 10 Sent Done Done false
4683 2434 Acked Read 2 Sent Done Done false
4684 2435 Acked Flush 0 Sent Done Done false
4685 2436 Acked Write 1 Sent Done Done false
4686 2437 Acked Write 5 Sent Done Done false
4687 2438 Acked Write 9 Sent Done Done false
4688 2439 Acked Read 2 Sent Done Done false
4689 2440 Acked Write 1 Sent Done Done false
4690 2441 Acked Write 8 Sent Done Done false
4691 2442 Acked Write 1 Sent Done Done false
4692 2443 Acked Write 2 Sent Done Done false
4693 2444 Acked Read 4 Sent Done Done false
4694 2445 Acked Read 10 Sent Done Done false
4695 2446 Acked Read 1 Sent Done Done false
4696 2447 Acked Write 9 Sent Done Done false
4697 2448 Acked Write 10 Sent Done Done false
4698 2449 Acked Write 2 Sent Done Done false
4699 2450 Acked Write 6 Sent Done Done false
4700 2451 Acked Read 8 Sent Done Done false
4701 2452 Acked Flush 0 Sent Done Done false
4702 2453 Acked Read 4 Sent Done Done false
4703 2454 Acked Write 8 Sent Done Done false
4704 2455 Acked Read 10 Sent Done Done false
4705 2456 Acked Read 9 Sent Done Done false
4706 2457 Acked Read 3 Sent Done Done false
4707 2458 Acked Write 2 Sent Done Done false
4708 2459 Acked Read 4 Sent Done Done false
4709 2460 Acked Read 5 Sent Done Done false
4710 2461 Acked Read 10 Sent Done Done false
4711 2462 Acked Read 9 Sent Done Done false
4712 2463 Acked Write 2 Sent Done Done false
4713 2464 Acked Write 3 Sent Done Done false
4714 2465 Acked Read 10 Sent Done Done false
4715 2466 Acked Read 9 Sent Done Done false
4716 2467 Acked Write 7 Sent Done Done false
4717 2468 Acked Read 7 Sent Done Done false
4718 2469 Acked Read 7 Sent Done Done false
4719 2470 Acked Write 4 Sent Done Done false
4720 2471 Acked Read 8 Sent Done Done false
4721 2472 Acked Read 6 Sent Done Done false
4722 2473 Acked Read 2 Sent Done Done false
4723 2474 Acked Read 3 Sent Done Done false
4724 2475 Acked Read 9 Sent Done Done false
4725 2476 Acked Write 6 Sent Done Done false
4726 2477 Acked Write 3 Sent Done Done false
4727 2478 Acked Write 9 Sent Done Done false
4728 2479 Acked Read 9 Sent Done Done false
4729 2480 Acked Write 10 Sent Done Done false
4730 2481 Acked Write 2 Sent Done Done false
4731 2482 Acked Write 8 Sent Done Done false
4732 2483 Acked Write 9 Sent Done Done false
4733 2484 Acked Read 9 Sent Done Done false
4734 2485 Acked Read 2 Sent Done Done false
4735 2486 Acked Write 4 Sent Done Done false
4736 2487 Acked Write 6 Sent Done Done false
4737 2488 Acked Read 10 Sent Done Done false
4738 2489 Acked Write 9 Sent Done Done false
4739 2490 Acked Read 7 Sent Done Done false
4740 2491 Acked Write 3 Sent Done Done false
4741 2492 Acked Read 8 Sent Done Done false
4742 2493 Acked Read 9 Sent Done Done false
4743 2494 Acked Read 3 Sent Done Done false
4744 2495 Acked Read 10 Sent Done Done false
4745 2496 Acked Read 8 Sent Done Done false
4746 2497 Acked Write 5 Sent Done Done false
4747 2498 Acked Write 6 Sent Done Done false
4748 2499 Acked Read 5 Sent Done Done false
4749 2500 Acked Read 6 Sent Done Done false
4750 2501 Acked Read 2 Sent Done Done false
4751 2502 Acked Flush 0 Sent Done Done false
4752 2503 Acked Write 4 Sent Done Done false
4753 2504 Acked Read 7 Sent Done Done false
4754 2505 Acked Write 5 Sent Done Done false
4755 2506 Acked Read 6 Sent Done Done false
4756 2507 Acked Read 2 Sent Done Done false
4757 2508 Acked Write 1 Sent Done Done false
4758 STATES DS:0 DS:1 DS:2 TOTAL
4759 Sent 301 0 0 301
4760 Done 0 301 301 602
4761 Skipped 0 0 0 0
4762 Error 0 0 0 0
4763 Last Flush: 2207 2502 2502
4764 Downstairs last five completed: 2207 2206 2205 2204 2203
4765 Upstairs last five completed: 2508 2507 2506 2505 2504
4766 CLIENT: Up:0 ds:301 act:2
4767 [8810] got start action
4768 [8810] I am going to restart
4769 Starting downstairs at port 8810
4770 Make output file at "/tmp/dsc/downstairs-8810.txt"
4771 [8810][0] reports Starting
4772 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1206)
4773 [8810][0] reports Running
4774 ----------------------------------------------------------------
4775 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
4776 States: Offline Active Active
4777 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4778 2208 Acked Read 2 Sent Done Done false
4779 2209 Acked Read 3 Sent Done Done false
4780 2210 Acked Flush 0 Sent Done Done false
4781 2211 Acked Flush 0 Sent Done Done false
4782 2212 Acked Write 10 Sent Done Done false
4783 2213 Acked Read 4 Sent Done Done false
4784 2214 Acked Read 10 Sent Done Done false
4785 2215 Acked Read 8 Sent Done Done false
4786 2216 Acked Write 5 Sent Done Done false
4787 2217 Acked Flush 0 Sent Done Done false
4788 2218 Acked Write 7 Sent Done Done false
4789 2219 Acked Read 9 Sent Done Done false
4790 2220 Acked Read 7 Sent Done Done false
4791 2221 Acked Read 3 Sent Done Done false
4792 2222 Acked Read 8 Sent Done Done false
4793 2223 Acked Write 7 Sent Done Done false
4794 2224 Acked Read 10 Sent Done Done false
4795 2225 Acked Write 2 Sent Done Done false
4796 2226 Acked Flush 0 Sent Done Done false
4797 2227 Acked Write 5 Sent Done Done false
4798 2228 Acked Read 3 Sent Done Done false
4799 2229 Acked Write 3 Sent Done Done false
4800 2230 Acked Read 7 Sent Done Done false
4801 2231 Acked Read 5 Sent Done Done false
4802 2232 Acked Write 10 Sent Done Done false
4803 2233 Acked Read 7 Sent Done Done false
4804 2234 Acked Read 9 Sent Done Done false
4805 2235 Acked Write 6 Sent Done Done false
4806 2236 Acked Write 4 Sent Done Done false
4807 2237 Acked Read 6 Sent Done Done false
4808 2238 Acked Read 1 Sent Done Done false
4809 2239 Acked Read 3 Sent Done Done false
4810 2240 Acked Read 7 Sent Done Done false
4811 2241 Acked Read 9 Sent Done Done false
4812 2242 Acked Read 6 Sent Done Done false
4813 2243 Acked Flush 0 Sent Done Done false
4814 2244 Acked Read 4 Sent Done Done false
4815 2245 Acked Write 9 Sent Done Done false
4816 2246 Acked Read 10 Sent Done Done false
4817 2247 Acked Write 6 Sent Done Done false
4818 2248 Acked Read 6 Sent Done Done false
4819 2249 Acked Write 7 Sent Done Done false
4820 2250 Acked Read 7 Sent Done Done false
4821 2251 Acked Read 6 Sent Done Done false
4822 2252 Acked Flush 0 Sent Done Done false
4823 2253 Acked Read 4 Sent Done Done false
4824 2254 Acked Write 6 Sent Done Done false
4825 2255 Acked Read 5 Sent Done Done false
4826 2256 Acked Read 3 Sent Done Done false
4827 2257 Acked Write 9 Sent Done Done false
4828 2258 Acked Flush 0 Sent Done Done false
4829 2259 Acked Read 1 Sent Done Done false
4830 2260 Acked Write 9 Sent Done Done false
4831 2261 Acked Read 5 Sent Done Done false
4832 2262 Acked Read 6 Sent Done Done false
4833 2263 Acked Read 2 Sent Done Done false
4834 2264 Acked Read 2 Sent Done Done false
4835 2265 Acked Read 7 Sent Done Done false
4836 2266 Acked Write 1 Sent Done Done false
4837 2267 Acked Write 8 Sent Done Done false
4838 2268 Acked Read 8 Sent Done Done false
4839 2269 Acked Write 9 Sent Done Done false
4840 2270 Acked Write 3 Sent Done Done false
4841 2271 Acked Read 8 Sent Done Done false
4842 2272 Acked Read 2 Sent Done Done false
4843 2273 Acked Read 5 Sent Done Done false
4844 2274 Acked Read 6 Sent Done Done false
4845 2275 Acked Write 8 Sent Done Done false
4846 2276 Acked Read 9 Sent Done Done false
4847 2277 Acked Read 1 Sent Done Done false
4848 2278 Acked Write 5 Sent Done Done false
4849 2279 Acked Write 5 Sent Done Done false
4850 2280 Acked Write 4 Sent Done Done false
4851 2281 Acked Write 3 Sent Done Done false
4852 2282 Acked Read 2 Sent Done Done false
4853 2283 Acked Read 5 Sent Done Done false
4854 2284 Acked Read 5 Sent Done Done false
4855 2285 Acked Write 5 Sent Done Done false
4856 2286 Acked Write 4 Sent Done Done false
4857 2287 Acked Write 3 Sent Done Done false
4858 2288 Acked Read 8 Sent Done Done false
4859 2289 Acked Flush 0 Sent Done Done false
4860 2290 Acked Write 10 Sent Done Done false
4861 2291 Acked Write 7 Sent Done Done false
4862 2292 Acked Write 2 Sent Done Done false
4863 2293 Acked Read 9 Sent Done Done false
4864 2294 Acked Write 6 Sent Done Done false
4865 2295 Acked Write 5 Sent Done Done false
4866 2296 Acked Read 10 Sent Done Done false
4867 2297 Acked Read 10 Sent Done Done false
4868 2298 Acked Read 3 Sent Done Done false
4869 2299 Acked Write 9 Sent Done Done false
4870 2300 Acked Write 10 Sent Done Done false
4871 2301 Acked Flush 0 Sent Done Done false
4872 2302 Acked Read 4 Sent Done Done false
4873 2303 Acked Write 7 Sent Done Done false
4874 2304 Acked Read 4 Sent Done Done false
4875 2305 Acked Read 10 Sent Done Done false
4876 2306 Acked Write 8 Sent Done Done false
4877 2307 Acked Write 3 Sent Done Done false
4878 2308 Acked Read 10 Sent Done Done false
4879 2309 Acked Read 2 Sent Done Done false
4880 2310 Acked Read 7 Sent Done Done false
4881 2311 Acked Read 6 Sent Done Done false
4882 2312 Acked Write 8 Sent Done Done false
4883 2313 Acked Read 4 Sent Done Done false
4884 2314 Acked Write 5 Sent Done Done false
4885 2315 Acked Flush 0 Sent Done Done false
4886 2316 Acked Read 5 Sent Done Done false
4887 2317 Acked Read 10 Sent Done Done false
4888 2318 Acked Write 9 Sent Done Done false
4889 2319 Acked Write 5 Sent Done Done false
4890 2320 Acked Read 9 Sent Done Done false
4891 2321 Acked Read 2 Sent Done Done false
4892 2322 Acked Write 7 Sent Done Done false
4893 2323 Acked Write 9 Sent Done Done false
4894 2324 Acked Read 1 Sent Done Done false
4895 2325 Acked Read 1 Sent Done Done false
4896 2326 Acked Write 1 Sent Done Done false
4897 2327 Acked Read 8 Sent Done Done false
4898 2328 Acked Write 7 Sent Done Done false
4899 2329 Acked Read 9 Sent Done Done false
4900 2330 Acked Write 4 Sent Done Done false
4901 2331 Acked Write 5 Sent Done Done false
4902 2332 Acked Write 7 Sent Done Done false
4903 2333 Acked Read 10 Sent Done Done false
4904 2334 Acked Read 4 Sent Done Done false
4905 2335 Acked Write 5 Sent Done Done false
4906 2336 Acked Flush 0 Sent Done Done false
4907 2337 Acked Write 7 Sent Done Done false
4908 2338 Acked Read 1 Sent Done Done false
4909 2339 Acked Read 3 Sent Done Done false
4910 2340 Acked Read 3 Sent Done Done false
4911 2341 Acked Write 2 Sent Done Done false
4912 2342 Acked Write 9 Sent Done Done false
4913 2343 Acked Write 8 Sent Done Done false
4914 2344 Acked Write 7 Sent Done Done false
4915 2345 Acked Write 4 Sent Done Done false
4916 2346 Acked Flush 0 Sent Done Done false
4917 2347 Acked Write 5 Sent Done Done false
4918 2348 Acked Write 6 Sent Done Done false
4919 2349 Acked Read 10 Sent Done Done false
4920 2350 Acked Read 4 Sent Done Done false
4921 2351 Acked Flush 0 Sent Done Done false
4922 2352 Acked Write 7 Sent Done Done false
4923 2353 Acked Write 6 Sent Done Done false
4924 2354 Acked Write 9 Sent Done Done false
4925 2355 Acked Read 10 Sent Done Done false
4926 2356 Acked Write 10 Sent Done Done false
4927 2357 Acked Write 7 Sent Done Done false
4928 2358 Acked Read 1 Sent Done Done false
4929 2359 Acked Read 9 Sent Done Done false
4930 2360 Acked Write 10 Sent Done Done false
4931 2361 Acked Write 9 Sent Done Done false
4932 2362 Acked Read 1 Sent Done Done false
4933 2363 Acked Write 8 Sent Done Done false
4934 2364 Acked Read 1 Sent Done Done false
4935 2365 Acked Read 4 Sent Done Done false
4936 2366 Acked Write 8 Sent Done Done false
4937 2367 Acked Read 2 Sent Done Done false
4938 2368 Acked Flush 0 Sent Done Done false
4939 2369 Acked Read 10 Sent Done Done false
4940 2370 Acked Write 8 Sent Done Done false
4941 2371 Acked Read 1 Sent Done Done false
4942 2372 Acked Write 1 Sent Done Done false
4943 2373 Acked Write 3 Sent Done Done false
4944 2374 Acked Write 1 Sent Done Done false
4945 2375 Acked Write 9 Sent Done Done false
4946 2376 Acked Write 7 Sent Done Done false
4947 2377 Acked Read 6 Sent Done Done false
4948 2378 Acked Read 9 Sent Done Done false
4949 2379 Acked Read 2 Sent Done Done false
4950 2380 Acked Read 6 Sent Done Done false
4951 2381 Acked Read 5 Sent Done Done false
4952 2382 Acked Flush 0 Sent Done Done false
4953 2383 Acked Flush 0 Sent Done Done false
4954 2384 Acked Write 2 Sent Done Done false
4955 2385 Acked Read 9 Sent Done Done false
4956 2386 Acked Flush 0 Sent Done Done false
4957 2387 Acked Write 4 Sent Done Done false
4958 2388 Acked Read 4 Sent Done Done false
4959 2389 Acked Write 6 Sent Done Done false
4960 2390 Acked Read 7 Sent Done Done false
4961 2391 Acked Write 3 Sent Done Done false
4962 2392 Acked Read 9 Sent Done Done false
4963 2393 Acked Read 1 Sent Done Done false
4964 2394 Acked Read 5 Sent Done Done false
4965 2395 Acked Read 8 Sent Done Done false
4966 2396 Acked Read 2 Sent Done Done false
4967 2397 Acked Write 5 Sent Done Done false
4968 2398 Acked Read 1 Sent Done Done false
4969 2399 Acked Read 1 Sent Done Done false
4970 2400 Acked Write 3 Sent Done Done false
4971 2401 Acked Read 9 Sent Done Done false
4972 2402 Acked Write 10 Sent Done Done false
4973 2403 Acked Write 2 Sent Done Done false
4974 2404 Acked Read 2 Sent Done Done false
4975 2405 Acked Read 6 Sent Done Done false
4976 2406 Acked Flush 0 Sent Done Done false
4977 2407 Acked Read 7 Sent Done Done false
4978 2408 Acked Write 6 Sent Done Done false
4979 2409 Acked Read 5 Sent Done Done false
4980 2410 Acked Read 9 Sent Done Done false
4981 2411 Acked Write 1 Sent Done Done false
4982 2412 Acked Write 3 Sent Done Done false
4983 2413 Acked Read 3 Sent Done Done false
4984 2414 Acked Read 5 Sent Done Done false
4985 2415 Acked Write 9 Sent Done Done false
4986 2416 Acked Write 8 Sent Done Done false
4987 2417 Acked Read 5 Sent Done Done false
4988 2418 Acked Read 2 Sent Done Done false
4989 2419 Acked Write 7 Sent Done Done false
4990 2420 Acked Read 4 Sent Done Done false
4991 2421 Acked Read 2 Sent Done Done false
4992 2422 Acked Read 2 Sent Done Done false
4993 2423 Acked Read 4 Sent Done Done false
4994 2424 Acked Write 7 Sent Done Done false
4995 2425 Acked Read 1 Sent Done Done false
4996 2426 Acked Flush 0 Sent Done Done false
4997 2427 Acked Read 6 Sent Done Done false
4998 2428 Acked Read 5 Sent Done Done false
4999 2429 Acked Write 10 Sent Done Done false
5000 2430 Acked Read 1 Sent Done Done false
5001 2431 Acked Write 8 Sent Done Done false
5002 2432 Acked Write 2 Sent Done Done false
5003 2433 Acked Read 10 Sent Done Done false
5004 2434 Acked Read 2 Sent Done Done false
5005 2435 Acked Flush 0 Sent Done Done false
5006 2436 Acked Write 1 Sent Done Done false
5007 2437 Acked Write 5 Sent Done Done false
5008 2438 Acked Write 9 Sent Done Done false
5009 2439 Acked Read 2 Sent Done Done false
5010 2440 Acked Write 1 Sent Done Done false
5011 2441 Acked Write 8 Sent Done Done false
5012 2442 Acked Write 1 Sent Done Done false
5013 2443 Acked Write 2 Sent Done Done false
5014 2444 Acked Read 4 Sent Done Done false
5015 2445 Acked Read 10 Sent Done Done false
5016 2446 Acked Read 1 Sent Done Done false
5017 2447 Acked Write 9 Sent Done Done false
5018 2448 Acked Write 10 Sent Done Done false
5019 2449 Acked Write 2 Sent Done Done false
5020 2450 Acked Write 6 Sent Done Done false
5021 2451 Acked Read 8 Sent Done Done false
5022 2452 Acked Flush 0 Sent Done Done false
5023 2453 Acked Read 4 Sent Done Done false
5024 2454 Acked Write 8 Sent Done Done false
5025 2455 Acked Read 10 Sent Done Done false
5026 2456 Acked Read 9 Sent Done Done false
5027 2457 Acked Read 3 Sent Done Done false
5028 2458 Acked Write 2 Sent Done Done false
5029 2459 Acked Read 4 Sent Done Done false
5030 2460 Acked Read 5 Sent Done Done false
5031 2461 Acked Read 10 Sent Done Done false
5032 2462 Acked Read 9 Sent Done Done false
5033 2463 Acked Write 2 Sent Done Done false
5034 2464 Acked Write 3 Sent Done Done false
5035 2465 Acked Read 10 Sent Done Done false
5036 2466 Acked Read 9 Sent Done Done false
5037 2467 Acked Write 7 Sent Done Done false
5038 2468 Acked Read 7 Sent Done Done false
5039 2469 Acked Read 7 Sent Done Done false
5040 2470 Acked Write 4 Sent Done Done false
5041 2471 Acked Read 8 Sent Done Done false
5042 2472 Acked Read 6 Sent Done Done false
5043 2473 Acked Read 2 Sent Done Done false
5044 2474 Acked Read 3 Sent Done Done false
5045 2475 Acked Read 9 Sent Done Done false
5046 2476 Acked Write 6 Sent Done Done false
5047 2477 Acked Write 3 Sent Done Done false
5048 2478 Acked Write 9 Sent Done Done false
5049 2479 Acked Read 9 Sent Done Done false
5050 2480 Acked Write 10 Sent Done Done false
5051 2481 Acked Write 2 Sent Done Done false
5052 2482 Acked Write 8 Sent Done Done false
5053 2483 Acked Write 9 Sent Done Done false
5054 2484 Acked Read 9 Sent Done Done false
5055 2485 Acked Read 2 Sent Done Done false
5056 2486 Acked Write 4 Sent Done Done false
5057 2487 Acked Write 6 Sent Done Done false
5058 2488 Acked Read 10 Sent Done Done false
5059 2489 Acked Write 9 Sent Done Done false
5060 2490 Acked Read 7 Sent Done Done false
5061 2491 Acked Write 3 Sent Done Done false
5062 2492 Acked Read 8 Sent Done Done false
5063 2493 Acked Read 9 Sent Done Done false
5064 2494 Acked Read 3 Sent Done Done false
5065 2495 Acked Read 10 Sent Done Done false
5066 2496 Acked Read 8 Sent Done Done false
5067 2497 Acked Write 5 Sent Done Done false
5068 2498 Acked Write 6 Sent Done Done false
5069 2499 Acked Read 5 Sent Done Done false
5070 2500 Acked Read 6 Sent Done Done false
5071 2501 Acked Read 2 Sent Done Done false
5072 2502 Acked Flush 0 Sent Done Done false
5073 2503 Acked Write 4 Sent Done Done false
5074 2504 Acked Read 7 Sent Done Done false
5075 2505 Acked Write 5 Sent Done Done false
5076 2506 Acked Read 6 Sent Done Done false
5077 2507 Acked Read 2 Sent Done Done false
5078 2508 Acked Write 1 Sent Done Done false
5079 2509 Acked Flush 0 Sent Done Done false
5080 STATES DS:0 DS:1 DS:2 TOTAL
5081 Sent 302 0 0 302
5082 Done 0 302 302 604
5083 Skipped 0 0 0 0
5084 Error 0 0 0 0
5085 Last Flush: 2207 2509 2509
5086 Downstairs last five completed: 2207 2206 2205 2204 2203
5087 Upstairs last five completed: 2509 2508 2507 2506 2505
5088 CLIENT: Up:0 ds:302 act:2
50892024-12-09T15:08:54.094ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
50902024-12-09T15:08:54.094ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46652) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
50912024-12-09T15:08:54.095ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
50922024-12-09T15:08:54.096ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
50932024-12-09T15:08:54.096ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
50942024-12-09T15:08:54.096ZINFOcrucible: send last flush ID to this DS: 2207 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
50952024-12-09T15:08:54.096ZINFOcrucible: Replied this last flush ID: 2207 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
50962024-12-09T15:08:54.096ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
50972024-12-09T15:08:54.096ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
50982024-12-09T15:08:54.096ZINFOcrucible: [0] client re-new 302 jobs since flush 2207 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
50992024-12-09T15:08:54.096ZINFOcrucible: [0] Replayed 302 jobs since flush: 2207 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
5100 ----------------------------------------------------------------
5101 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5102 Downstairs last five completed: 2509 2508 2507 2506 2505
5103 Upstairs last five completed: 2509 2508 2507 2506 2505
5104 CLIENT: Up:0 ds:0 act:3
5105 Replay: All jobs finished
5106 Main task has work to do, go find it
5107 got dsc Stop(2)
5108 stop 2
5109 Dec 09 15:08:56.345 INFO request completed[8830] Got stop action so:false kr:false
5110 , latency_us: 398, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 8e318502-dee0-4b1b-a99c-f7fa30b6490b, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
51112024-12-09T15:08:56.346ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
51122024-12-09T15:08:56.346ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
5113 Dec 09 15:08:56.346 INFO request completed, latency_us: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:08:56.346510389Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
5114 182, response_code: 200, uri: /state/cid/2, method: GET, req_id: c121b3e0-4e4b-4e20-af84-6c21882ab3f2, remote_addr: 127.0{"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:08:56.346531296Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
5115 .0.1:47510, local_addr: 127.0.0.1:9998
51162024-12-09T15:08:56.346ZINFOcrucible: [2] Transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
51172024-12-09T15:08:56.346ZINFOcrucible: client task is exiting = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
5118 [8830] Exited with: ExitStatus(unix_wait_status(9))
5119 [8830][2] reports Exit
5120 Dec 09 15:09:00.348 INFO request completed, latency_us: 232, response_code: 200, uri: /state/cid/2, method: GET, req_id: d7d8740a-c470-49b2-86da-c48a6c6364cc, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
5121 001/300 Flush
5122 002/300 Write block 213 len 24576 data: 4 4 4 4 4 4
5123 003/300 Write block 1245 len 24576 data: 3 3 3 3 3 3
5124 004/300 Write block 2899 len 12288 data: 3 2 2
5125 005/300 Read block 644 len 40960
5126 006/300 Read block 4963 len 8192
5127 007/300 Write block 3243 len 32768 data: 4 4 4 4 4 3 3 3
5128 008/300 Write block 1463 len 4096 data: 2
5129 009/300 Write block 364 len 20480 data: 3 3 2 2 2
5130 010/300 Read block 1289 len 4096
5131 011/300 Read block 4211 len 28672
5132 012/300 Read block 4839 len 16384
5133 013/300 Read block 4766 len 4096
5134 014/300 Write block 4015 len 36864 data: 2 3 3 3 3 3 3 2 2
5135 015/300 Read block 1834 len 12288
5136 016/300 Write block 1086 len 40960 data: 2 2 2 2 2 2 2 2 3 3
5137 017/300 Write block 1350 len 36864 data: 2 2 2 2 3 3 3 3 3
5138 018/300 Write block 1808 len 4096 data: 4
5139 019/300 Read block 850 len 4096
5140 020/300 Flush
5141 021/300 Read block 2892 len 24576
5142 022/300 Flush
5143 023/300 Read block 3144 len 12288
5144 024/300 Write block 4985 len 28672 data: 2 2 2 2 2 2 4
5145 025/300 Write block 2653 len 36864 data: 2 2 2 2 2 2 2 2 2
5146 026/300 Write block 2387 len 4096 data: 2
5147 027/300 Read block 4890 len 32768
5148 028/300 Read block 3809 len 28672
5149 029/300 Read block 2715 len 40960
5150 030/300 Read block 3900 len 28672
5151 031/300 Write block 588 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5152 032/300 Flush
5153 033/300 Read block 3852 len 20480
5154 034/300 Read block 4732 len 24576
5155 035/300 Read block 1292 len 32768
5156 036/300 Write block 2199 len 20480 data: 2 2 2 3 3
5157 037/300 Read block 2109 len 32768
5158 038/300 Write block 149 len 36864 data: 3 3 3 3 3 4 3 4 3
5159 039/300 Read block 3283 len 12288
5160 040/300 Read block 3544 len 40960
5161 041/300 Write block 1225 len 40960 data: 3 3 3 2 2 2 2 2 2 2
5162 042/300 Write block 3673 len 20480 data: 2 2 2 2 2
5163 043/300 Read block 4682 len 16384
5164 044/300 Read block 3369 len 12288
5165 045/300 Write block 4083 len 32768 data: 3 3 3 3 3 3 3 4
5166 046/300 Read block 1314 len 24576
5167 047/300 Write block 3535 len 24576 data: 2 2 2 2 2 3
5168 048/300 Write block 2201 len 12288 data: 3 4 4
5169 049/300 Write block 2112 len 28672 data: 2 2 2 2 3 3 3
5170 050/300 Flush
5171 051/300 Read block 1682 len 4096
5172 052/300 Write block 4527 len 28672 data: 3 3 3 2 2 2 2
5173 053/300 Read block 1314 len 4096
5174 054/300 Read block 2223 len 32768
5175 055/300 Write block 571 len 40960 data: 2 2 3 3 4 4 4 4 4 4
5176 056/300 Write block 1618 len 32768 data: 2 2 2 2 2 2 2 2
5177 057/300 Read block 454 len 28672
5178 058/300 Write block 3383 len 16384 data: 2 2 2 2
5179 059/300 Flush
5180 060/300 Read block 3450 len 40960
5181 061/300 Read block 3135 len 24576
5182 062/300 Write block 3132 len 16384 data: 2 2 2 3
5183 063/300 Read block 2045 len 28672
5184 064/300 Read block 3340 len 8192
5185 065/300 Write block 314 len 12288 data: 2 2 2
5186 066/300 Flush
5187 067/300 Read block 2449 len 32768
5188 068/300 Write block 1560 len 16384 data: 4 6 6 6
5189 069/300 Read block 3866 len 12288
5190 070/300 Write block 1804 len 16384 data: 3 3 4 5
5191 071/300 Write block 3119 len 16384 data: 3 3 3 3
5192 072/300 Read block 1500 len 24576
5193 073/300 Read block 2082 len 8192
5194 074/300 Write block 4347 len 40960 data: 3 3 3 3 3 2 2 2 2 3
5195 075/300 Read block 564 len 16384
5196 076/300 Write block 1450 len 20480 data: 3 3 3 3 2
5197 077/300 Write block 4343 len 8192 data: 3 2
5198 078/300 Write block 2234 len 12288 data: 4 4 3
5199 079/300 Read block 2281 len 24576
5200 080/300 Write block 3747 len 36864 data: 3 3 3 3 3 3 3 3 2
5201 081/300 Write block 4776 len 40960 data: 2 2 2 2 2 2 2 2 3 2
5202 082/300 Write block 4345 len 20480 data: 2 3 4 4 4
5203 083/300 Write block 1114 len 8192 data: 4 3
5204 084/300 Write block 4506 len 40960 data: 3 3 3 3 3 3 4 3 3 3
5205 085/300 Write block 3965 len 32768 data: 3 3 2 2 2 3 3 3
5206 086/300 Write block 3500 len 8192 data: 2 2
5207 087/300 Read block 3593 len 36864
5208 088/300 Write block 4426 len 20480 data: 3 3 3 4 3
5209 089/300 Read block 4605 len 32768
5210 090/300 Write block 4285 len 24576 data: 3 3 4 3 3 3
5211 091/300 Write block 2729 len 4096 data: 4
5212 092/300 Flush
5213 093/300 Flush
5214 094/300 Read block 4218 len 8192
5215 095/300 Write block 4435 len 4096 data: 4
5216 096/300 Read block 3389 len 12288
5217 097/300 Write block 1198 len 40960 data: 3 3 3 3 2 2 2 2 2 2
5218 098/300 Read block 2527 len 40960
5219 099/300 Read block 2717 len 32768
5220 100/300 Read block 2606 len 32768
5221 101/300 Write block 2161 len 12288 data: 2 2 2
5222 102/300 Write block 3669 len 4096 data: 2
5223 103/300 Flush
5224 104/300 Flush
5225 105/300 Write block 1457 len 32768 data: 2 2 2 2 2 2 3 2
5226 106/300 Read block 4329 len 8192
5227 107/300 Read block 3327 len 16384
5228 108/300 Write block 3886 len 12288 data: 4 3 3
5229 109/300 Read block 2591 len 20480
5230 110/300 Read block 3315 len 16384
5231 111/300 Write block 2513 len 36864 data: 2 2 2 2 2 2 2 2 3
5232 112/300 Write block 4912 len 16384 data: 3 3 3 3
5233 113/300 Read block 352 len 4096
5234 114/300 Flush
5235 115/300 Write block 2257 len 32768 data: 3 3 2 2 2 2 2 2
5236 116/300 Read block 357 len 20480
5237 117/300 Read block 1117 len 16384
5238 118/300 Write block 3855 len 12288 data: 3 3 3
5239 119/300 Write block 3464 len 40960 data: 3 3 3 3 3 3 4 4 3 3
5240 120/300 Write block 3211 len 28672 data: 2 2 2 2 3 3 3
5241 121/300 Write block 2333 len 40960 data: 3 4 5 5 5 5 5 5 4 4
5242 122/300 Write block 4199 len 4096 data: 3
5243 123/300 Flush
5244 124/300 Read block 2063 len 20480
5245 125/300 Read block 4999 len 4096
5246 126/300 Read block 4197 len 24576
5247 127/300 Read block 3210 len 36864
5248 128/300 Read block 3782 len 24576
5249 129/300 Write block 300 len 28672 data: 2 2 2 2 3 3 2
5250 130/300 Flush
5251 131/300 Write block 4256 len 36864 data: 3 3 3 4 3 3 3 3 3
5252 132/300 Write block 1431 len 32768 data: 2 2 2 2 2 2 2 2
5253 133/300 Read block 715 len 16384
5254 134/300 Flush
5255 135/300 Read block 1076 len 4096
5256 136/300 Read block 2007 len 8192
5257 137/300 Write block 3590 len 32768 data: 2 2 3 3 3 3 4 4
5258 138/300 Write block 2341 len 28672 data: 5 5 3 3 3 3 2
5259 139/300 Read block 4328 len 8192
5260 140/300 Read block 4605 len 40960
5261 141/300 Write block 3821 len 16384 data: 2 2 3 3
5262 142/300 Read block 3904 len 4096
5263 143/300 Write block 4715 len 36864 data: 3 2 2 2 2 2 2 2 2
5264 144/300 Read block 1779 len 8192
5265 145/300 Write block 1691 len 28672 data: 3 3 3 3 3 3 2
5266 146/300 Write block 1297 len 36864 data: 3 3 2 3 3 3 3 3 3
5267 147/300 Flush
5268 148/300 Read block 188 len 24576
5269 149/300 Read block 3425 len 16384
5270 150/300 Read block 496 len 8192
5271 151/300 Read block 4737 len 20480
5272 152/300 Read block 817 len 4096
5273 153/300 Write block 972 len 36864 data: 2 2 2 2 2 2 2 2 2
5274 154/300 Write block 258 len 20480 data: 3 3 3 3 3
5275 155/300 Read block 1259 len 40960
5276 156/300 Read block 3307 len 36864
5277 157/300 Write block 2752 len 8192 data: 3 3
5278 158/300 Write block 2036 len 32768 data: 2 2 2 2 2 2 2 2
5279 159/300 Write block 4243 len 20480 data: 3 3 3 3 2
5280 160/300 Read block 368 len 28672
5281 161/300 Write block 1217 len 40960 data: 2 2 3 3 3 3 3 3 4 4
5282 162/300 Write block 4385 len 24576 data: 2 2 2 2 2 2
5283 163/300 Read block 3420 len 36864
5284 164/300 Write block 4374 len 32768 data: 2 4 3 4 6 6 6 6
5285 165/300 Read block 2137 len 28672
5286 166/300 Read block 368 len 12288
5287 167/300 Write block 318 len 28672 data: 2 2 3 3 3 3 3
5288 168/300 Write block 1866 len 24576 data: 2 3 3 3 3 3
5289 169/300 Write block 1840 len 16384 data: 4 4 4 2
5290 170/300 Read block 3167 len 8192
5291 171/300 Flush
5292 172/300 Read block 4301 len 36864
5293 173/300 Read block 3618 len 16384
5294 174/300 Write block 209 len 24576 data: 4 4 4 4 5 5
5295 175/300 Write block 2999 len 32768 data: 2 2 2 2 2 2 2 2
5296 176/300 Read block 4801 len 16384
5297 177/300 Read block 712 len 16384
5298 178/300 Read block 3836 len 12288
5299 179/300 Flush
5300 180/300 Read block 4089 len 16384
5301 181/300 Write block 3709 len 40960 data: 3 3 4 4 4 3 3 3 2 2
5302 182/300 Write block 627 len 20480 data: 2 2 3 3 3
5303 183/300 Write block 4010 len 36864 data: 3 2 2 2 2 3 4 4 4
5304 184/300 Flush
5305 185/300 Write block 627 len 28672 data: 3 3 4 4 4 3 3
5306 186/300 Write block 149 len 20480 data: 4 4 4 4 4
5307 187/300 Flush
5308 188/300 Write block 1560 len 40960 data: 5 7 7 7 6 4 4 3 5 5
5309 189/300 Flush
5310 190/300 Write block 1551 len 12288 data: 2 2 3
5311 191/300 Read block 2491 len 36864
5312 192/300 Read block 1096 len 36864
5313 193/300 Write block 1468 len 36864 data: 2 2 2 2 2 2 2 2 2
5314 194/300 Read block 1458 len 16384
5315 195/300 Write block 4678 len 8192 data: 2 2
5316 196/300 Read block 4972 len 8192
5317 197/300 Write block 635 len 32768 data: 3 3 4 3 3 3 2 2
5318 198/300 Read block 2468 len 24576
5319 199/300 Read block 727 len 36864
5320 200/300 Read block 2731 len 36864
5321 201/300 Write block 4082 len 24576 data: 2 4 4 4 4 4
5322 202/300 Write block 246 len 4096 data: 4
5323 203/300 Read block 1962 len 40960
5324 204/300 Flush
5325 205/300 Read block 2085 len 40960
5326 206/300 Write block 1762 len 20480 data: 2 2 2 2 2
5327 207/300 Flush
5328 208/300 Write block 1316 len 4096 data: 3
5329 209/300 Write block 2443 len 28672 data: 3 3 3 3 3 3 2
5330 210/300 Read block 4977 len 20480
5331 211/300 Write block 44 len 4096 data: 2
5332 212/300 Write block 3741 len 4096 data: 3
5333 213/300 Flush
5334 214/300 Flush
5335 215/300 Write block 2249 len 20480 data: 2 2 2 2 2
5336 216/300 Read block 492 len 16384
5337 217/300 Read block 169 len 36864
5338 218/300 Write block 2839 len 32768 data: 2 2 2 2 2 2 2 2
5339 219/300 Read block 493 len 40960
5340 220/300 Write block 4867 len 16384 data: 3 3 4 4
5341 221/300 Read block 3892 len 28672
5342 222/300 Read block 2775 len 16384
5343 223/300 Write block 578 len 24576 data: 5 5 5 4 4 4
5344 224/300 Read block 478 len 40960
5345 225/300 Read block 2345 len 8192
5346 226/300 Read block 4900 len 24576
5347 227/300 Read block 2567 len 8192
5348 228/300 Read block 3967 len 16384
5349 229/300 Read block 3578 len 28672
5350 230/300 Write block 2601 len 24576 data: 2 2 2 2 2 2
5351 231/300 Write block 306 len 16384 data: 3 2 2 2
5352 232/300 Write block 270 len 12288 data: 2 2 2
5353 233/300 Read block 4252 len 36864
5354 234/300 Write block 1577 len 12288 data: 4 4 4
5355 235/300 Read block 3002 len 24576
5356 236/300 Write block 1779 len 40960 data: 2 2 2 2 2 2 3 5 5 4
5357 237/300 Write block 3123 len 8192 data: 3 3
5358 238/300 Write block 1456 len 24576 data: 2 3 3 3 3 3
5359 239/300 Read block 4497 len 28672
5360 240/300 Read block 3453 len 8192
5361 241/300 Write block 1640 len 36864 data: 2 2 2 2 2 2 2 2 3
5362 242/300 Write block 1174 len 24576 data: 2 2 2 2 2 2
5363 243/300 Write block 1335 len 16384 data: 3 3 3 3
5364 244/300 Write block 3066 len 4096 data: 2
5365 245/300 Read block 1437 len 16384
5366 246/300 Write block 3823 len 28672 data: 4 4 3 3 3 3 4
5367 247/300 Write block 4218 len 16384 data: 2 2 2 2
5368 248/300 Read block 2640 len 36864
5369 249/300 Read block 3852 len 24576
5370 250/300 Write block 848 len 4096 data: 2
5371 251/300 Write block 2041 len 20480 data: 3 3 3 2 3
5372 252/300 Read block 1592 len 12288
5373 253/300 Write block 2949 len 36864 data: 4 4 4 3 3 3 3 3 2
5374 254/300 Write block 2951 len 12288 data: 5 4 4
5375 255/300 Read block 4233 len 24576
5376 256/300 Write block 2818 len 36864 data: 3 3 2 2 2 2 2 2 2
5377 257/300 Read block 1632 len 36864
5378 258/300 Read block 146 len 28672
5379 259/300 Read block 3290 len 16384
5380 260/300 Write block 2353 len 40960 data: 3 3 3 3 2 2 2 2 2 2
5381 261/300 Write block 3466 len 16384 data: 4 4 4 4
5382 262/300 Write block 3787 len 24576 data: 2 2 2 2 2 2
5383 263/300 Flush
5384 264/300 Write block 2867 len 12288 data: 5 5 5
5385 265/300 Write block 3529 len 32768 data: 2 3 3 2 2 2 3 3
5386 266/300 Read block 1334 len 40960
5387 267/300 Read block 1436 len 28672
5388 268/300 Write block 3715 len 28672 data: 4 4 3 3 2 2 2
5389 269/300 Read block 3989 len 28672
5390 270/300 Flush
5391 271/300 Read block 2623 len 16384
5392 272/300 Read block 1450 len 12288
5393 273/300 Read block 3941 len 32768
5394 274/300 Write block 4716 len 24576 data: 3 3 3 3 3 3
5395 275/300 Read block 3444 len 20480
5396 276/300 Write block 4698 len 12288 data: 4 4 4
5397 277/300 Read block 1759 len 40960
5398 278/300 Read block 2515 len 16384
5399 279/300 Write block 2848 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5400 280/300 Read block 4200 len 16384
5401 281/300 Read block 4105 len 28672
5402 282/300 Flush
5403 283/300 Read block 4136 len 8192
5404 284/300 Write block 4119 len 36864 data: 2 3 3 3 3 3 2 2 2
5405 285/300 Write block 1409 len 36864 data: 3 3 3 3 3 3 3 4 4
5406 286/300 Read block 301 len 12288
5407 287/300 Read block 1164 len 12288
5408 288/300 Read block 549 len 16384
5409 289/300 Write block 366 len 8192 data: 3 3
5410 290/300 Write block 3422 len 32768 data: 3 3 3 3 2 2 2 2
5411 291/300 Write block 4028 len 8192 data: 2 3
5412 292/300 Read block 767 len 36864
5413 293/300 Flush
5414 294/300 Read block 1620 len 8192
5415 295/300 Write block 2530 len 32768 data: 3 3 3 2 2 2 2 2
5416 296/300 Write block 2080 len 12288 data: 3 2 2
5417 297/300 Write block 498 len 8192 data: 3 2
5418 298/300 Write block 2793 len 20480 data: 3 3 4 3 4
5419 299/300 Read block 4730 len 20480
5420 300/300 Read block 1128 len 24576
5421 301/300 Write block 4987 len 24576 data: 3 3 3 3 5 4
5422 Main task has work to do, go find it
5423 got dsc Start(2)
5424 start 2
5425 Dec 09 15:09:00.577 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/2, method: POST, req_id: 627998fc-89a9-48b8-b869-8b8a4768a545, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
5426 Replay: started 2, returned:Ok(())
5427 ----------------------------------------------------------------
5428 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5429 States: Active Active Offline
5430 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5431 2510 Acked Flush 0 Done Done Sent false
5432 2511 Acked Write 6 Done Done Sent false
5433 2512 Acked Write 6 Done Done Sent false
5434 2513 Acked Write 3 Done Done Sent false
5435 2514 Acked Read 10 Done Done Sent false
5436 2515 Acked Read 2 Done Done Sent false
5437 2516 Acked Write 8 Done Done Sent false
5438 2517 Acked Write 1 Done Done Sent false
5439 2518 Acked Write 5 Done Done Sent false
5440 2519 Acked Read 1 Done Done Sent false
5441 2520 Acked Read 7 Done Done Sent false
5442 2521 Acked Read 4 Done Done Sent false
5443 2522 Acked Read 1 Done Done Sent false
5444 2523 Acked Write 9 Done Done Sent false
5445 2524 Acked Read 3 Done Done Sent false
5446 2525 Acked Write 10 Done Done Sent false
5447 2526 Acked Write 9 Done Done Sent false
5448 2527 Acked Write 1 Done Done Sent false
5449 2528 Acked Read 1 Done Done Sent false
5450 2529 Acked Flush 0 Done Done Sent false
5451 2530 Acked Read 6 Done Done Sent false
5452 2531 Acked Flush 0 Done Done Sent false
5453 2532 Acked Read 3 Done Done Sent false
5454 2533 Acked Write 7 Done Done Sent false
5455 2534 Acked Write 9 Done Done Sent false
5456 2535 Acked Write 1 Done Done Sent false
5457 2536 Acked Read 8 Done Done Sent false
5458 2537 Acked Read 7 Done Done Sent false
5459 2538 Acked Read 10 Done Done Sent false
5460 2539 Acked Read 7 Done Done Sent false
5461 2540 Acked Write 10 Done Done Sent false
5462 2541 Acked Flush 0 Done Done Sent false
5463 2542 Acked Read 5 Done Done Sent false
5464 2543 Acked Read 6 Done Done Sent false
5465 2544 Acked Read 8 Done Done Sent false
5466 2545 Acked Write 5 Done Done Sent false
5467 2546 Acked Read 8 Done Done Sent false
5468 2547 Acked Write 9 Done Done Sent false
5469 2548 Acked Read 3 Done Done Sent false
5470 2549 Acked Read 10 Done Done Sent false
5471 2550 Acked Write 10 Done Done Sent false
5472 2551 Acked Write 5 Done Done Sent false
5473 2552 Acked Read 4 Done Done Sent false
5474 2553 Acked Read 3 Done Done Sent false
5475 2554 Acked Write 8 Done Done Sent false
5476 2555 Acked Read 6 Done Done Sent false
5477 2556 Acked Write 6 Done Done Sent false
5478 2557 Acked Write 3 Done Done Sent false
5479 2558 Acked Write 7 Done Done Sent false
5480 2559 Acked Flush 0 Done Done Sent false
5481 2560 Acked Read 1 Done Done Sent false
5482 2561 Acked Write 7 Done Done Sent false
5483 2562 Acked Read 1 Done Done Sent false
5484 2563 Acked Read 8 Done Done Sent false
5485 2564 Acked Write 10 Done Done Sent false
5486 2565 Acked Write 8 Done Done Sent false
5487 2566 Acked Read 7 Done Done Sent false
5488 2567 Acked Write 4 Done Done Sent false
5489 2568 Acked Flush 0 Done Done Sent false
5490 2569 Acked Read 10 Done Done Sent false
5491 2570 Acked Read 6 Done Done Sent false
5492 2571 Acked Write 4 Done Done Sent false
5493 2572 Acked Read 7 Done Done Sent false
5494 2573 Acked Read 2 Done Done Sent false
5495 2574 Acked Write 3 Done Done Sent false
5496 2575 Acked Flush 0 Done Done Sent false
5497 2576 Acked Read 8 Done Done Sent false
5498 2577 Acked Write 4 Done Done Sent false
5499 2578 Acked Read 3 Done Done Sent false
5500 2579 Acked Write 4 Done Done Sent false
5501 2580 Acked Write 4 Done Done Sent false
5502 2581 Acked Read 6 Done Done Sent false
5503 2582 Acked Read 2 Done Done Sent false
5504 2583 Acked Write 10 Done Done Sent false
5505 2584 Acked Read 4 Done Done Sent false
5506 2585 Acked Write 5 Done Done Sent false
5507 2586 Acked Write 2 Done Done Sent false
5508 2587 Acked Write 3 Done Done Sent false
5509 2588 Acked Read 6 Done Done Sent false
5510 2589 Acked Write 9 Done Done Sent false
5511 2590 Acked Write 10 Done Done Sent false
5512 2591 Acked Write 5 Done Done Sent false
5513 2592 Acked Write 2 Done Done Sent false
5514 2593 Acked Write 10 Done Done Sent false
5515 2594 Acked Write 8 Done Done Sent false
5516 2595 Acked Write 2 Done Done Sent false
5517 2596 Acked Read 9 Done Done Sent false
5518 2597 Acked Write 5 Done Done Sent false
5519 2598 Acked Read 8 Done Done Sent false
5520 2599 Acked Write 6 Done Done Sent false
5521 2600 Acked Write 1 Done Done Sent false
5522 2601 Acked Flush 0 Done Done Sent false
5523 2602 Acked Flush 0 Done Done Sent false
5524 2603 Acked Read 2 Done Done Sent false
5525 2604 Acked Write 1 Done Done Sent false
5526 2605 Acked Read 3 Done Done Sent false
5527 2606 Acked Write 10 Done Done Sent false
5528 2607 Acked Read 10 Done Done Sent false
5529 2608 Acked Read 8 Done Done Sent false
5530 2609 Acked Read 8 Done Done Sent false
5531 2610 Acked Write 3 Done Done Sent false
5532 2611 Acked Write 1 Done Done Sent false
5533 2612 Acked Flush 0 Done Done Sent false
5534 2613 Acked Flush 0 Done Done Sent false
5535 2614 Acked Write 8 Done Done Sent false
5536 2615 Acked Read 2 Done Done Sent false
5537 2616 Acked Read 4 Done Done Sent false
5538 2617 Acked Write 3 Done Done Sent false
5539 2618 Acked Read 5 Done Done Sent false
5540 2619 Acked Read 4 Done Done Sent false
5541 2620 Acked Write 9 Done Done Sent false
5542 2621 Acked Write 4 Done Done Sent false
5543 2622 Acked Read 1 Done Done Sent false
5544 2623 Acked Flush 0 Done Done Sent false
5545 2624 Acked Write 8 Done Done Sent false
5546 2625 Acked Read 5 Done Done Sent false
5547 2626 Acked Read 4 Done Done Sent false
5548 2627 Acked Write 3 Done Done Sent false
5549 2628 Acked Write 10 Done Done Sent false
5550 2629 Acked Write 7 Done Done Sent false
5551 2630 Acked Write 10 Done Done Sent false
5552 2631 Acked Write 1 Done Done Sent false
5553 2632 Acked Flush 0 Done Done Sent false
5554 2633 Acked Read 5 Done Done Sent false
5555 2634 Acked Read 1 Done Done Sent false
5556 2635 Acked Read 6 Done Done Sent false
5557 2636 Acked Read 9 Done Done Sent false
5558 2637 Acked Read 6 Done Done Sent false
5559 2638 Acked Write 7 Done Done Sent false
5560 2639 Acked Flush 0 Done Done Sent false
5561 2640 Acked Write 9 Done Done Sent false
5562 2641 Acked Write 8 Done Done Sent false
5563 2642 Acked Read 4 Done Done Sent false
5564 2643 Acked Flush 0 Done Done Sent false
5565 2644 Acked Read 1 Done Done Sent false
5566 2645 Acked Read 2 Done Done Sent false
5567 2646 Acked Write 8 Done Done Sent false
5568 2647 Acked Write 7 Done Done Sent false
5569 2648 Acked Read 2 Done Done Sent false
5570 2649 Acked Read 10 Done Done Sent false
5571 2650 Acked Write 4 Done Done Sent false
5572 2651 Acked Read 1 Done Done Sent false
5573 2652 Acked Write 9 Done Done Sent false
5574 2653 Acked Read 2 Done Done Sent false
5575 2654 Acked Write 7 Done Done Sent false
5576 2655 Acked Write 9 Done Done Sent false
5577 2656 Acked Flush 0 Done Done Sent false
5578 2657 Acked Read 6 Done Done Sent false
5579 2658 Acked Read 4 Done Done Sent false
5580 2659 Acked Read 2 Done Done Sent false
5581 2660 Acked Read 5 Done Done Sent false
5582 2661 Acked Read 1 Done Done Sent false
5583 2662 Acked Write 9 Done Done Sent false
5584 2663 Acked Write 5 Done Done Sent false
5585 2664 Acked Read 10 Done Done Sent false
5586 2665 Acked Read 9 Done Done Sent false
5587 2666 Acked Write 2 Done Done Sent false
5588 2667 Acked Write 8 Done Done Sent false
5589 2668 Acked Write 5 Done Done Sent false
5590 2669 Acked Read 7 Done Done Sent false
5591 2670 Acked Write 10 Done Done Sent false
5592 2671 Acked Write 6 Done Done Sent false
5593 2672 Acked Read 9 Done Done Sent false
5594 2673 Acked Write 8 Done Done Sent false
5595 2674 Acked Read 7 Done Done Sent false
5596 2675 Acked Read 3 Done Done Sent false
5597 2676 Acked Write 7 Done Done Sent false
5598 2677 Acked Write 6 Done Done Sent false
5599 2678 Acked Write 4 Done Done Sent false
5600 2679 Acked Read 2 Done Done Sent false
5601 2680 Acked Flush 0 Done Done Sent false
5602 2681 Acked Read 9 Done Done Sent false
5603 2682 Acked Read 4 Done Done Sent false
5604 2683 Acked Write 6 Done Done Sent false
5605 2684 Acked Write 8 Done Done Sent false
5606 2685 Acked Read 4 Done Done Sent false
5607 2686 Acked Read 4 Done Done Sent false
5608 2687 Acked Read 3 Done Done Sent false
5609 2688 Acked Flush 0 Done Done Sent false
5610 2689 Acked Read 4 Done Done Sent false
5611 2690 Acked Write 10 Done Done Sent false
5612 2691 Acked Write 5 Done Done Sent false
5613 2692 Acked Write 9 Done Done Sent false
5614 2693 Acked Flush 0 Done Done Sent false
5615 2694 Acked Write 7 Done Done Sent false
5616 2695 Acked Write 5 Done Done Sent false
5617 2696 Acked Flush 0 Done Done Sent false
5618 2697 Acked Write 10 Done Done Sent false
5619 2698 Acked Flush 0 Done Done Sent false
5620 2699 Acked Write 3 Done Done Sent false
5621 2700 Acked Read 9 Done Done Sent false
5622 2701 Acked Read 9 Done Done Sent false
5623 2702 Acked Write 9 Done Done Sent false
5624 2703 Acked Read 4 Done Done Sent false
5625 2704 Acked Write 2 Done Done Sent false
5626 2705 Acked Read 2 Done Done Sent false
5627 2706 Acked Write 8 Done Done Sent false
5628 2707 Acked Read 6 Done Done Sent false
5629 2708 Acked Read 9 Done Done Sent false
5630 2709 Acked Read 9 Done Done Sent false
5631 2710 Acked Write 6 Done Done Sent false
5632 2711 Acked Write 1 Done Done Sent false
5633 2712 Acked Read 10 Done Done Sent false
5634 2713 Acked Flush 0 Done Done Sent false
5635 2714 Acked Read 10 Done Done Sent false
5636 2715 Acked Write 5 Done Done Sent false
5637 2716 Acked Flush 0 Done Done Sent false
5638 2717 Acked Write 1 Done Done Sent false
5639 2718 Acked Write 7 Done Done Sent false
5640 2719 Acked Read 5 Done Done Sent false
5641 2720 Acked Write 1 Done Done Sent false
5642 2721 Acked Write 1 Done Done Sent false
5643 2722 Acked Flush 0 Done Done Sent false
5644 2723 Acked Flush 0 Done Done Sent false
5645 2724 Acked Write 5 Done Done Sent false
5646 2725 Acked Read 4 Done Done Sent false
5647 2726 Acked Read 9 Done Done Sent false
5648 2727 Acked Write 8 Done Done Sent false
5649 2728 Acked Read 10 Done Done Sent false
5650 2729 Acked Write 4 Done Done Sent false
5651 2730 Acked Read 7 Done Done Sent false
5652 2731 Acked Read 4 Done Done Sent false
5653 2732 Acked Write 6 Done Done Sent false
5654 2733 Acked Read 10 Done Done Sent false
5655 2734 Acked Read 2 Done Done Sent false
5656 2735 Acked Read 6 Done Done Sent false
5657 2736 Acked Read 2 Done Done Sent false
5658 2737 Acked Read 4 Done Done Sent false
5659 2738 Acked Read 7 Done Done Sent false
5660 2739 Acked Write 6 Done Done Sent false
5661 2740 Acked Write 4 Done Done Sent false
5662 2741 Acked Write 3 Done Done Sent false
5663 2742 Acked Read 9 Done Done Sent false
5664 2743 Acked Write 3 Done Done Sent false
5665 2744 Acked Read 6 Done Done Sent false
5666 2745 Acked Write 10 Done Done Sent false
5667 2746 Acked Write 2 Done Done Sent false
5668 2747 Acked Write 6 Done Done Sent false
5669 2748 Acked Read 7 Done Done Sent false
5670 2749 Acked Read 2 Done Done Sent false
5671 2750 Acked Write 9 Done Done Sent false
5672 2751 Acked Write 6 Done Done Sent false
5673 2752 Acked Write 4 Done Done Sent false
5674 2753 Acked Write 1 Done Done Sent false
5675 2754 Acked Read 4 Done Done Sent false
5676 2755 Acked Write 7 Done Done Sent false
5677 2756 Acked Write 4 Done Done Sent false
5678 2757 Acked Read 9 Done Done Sent false
5679 2758 Acked Read 6 Done Done Sent false
5680 2759 Acked Write 1 Done Done Sent false
5681 2760 Acked Write 5 Done Done Sent false
5682 2761 Acked Read 3 Done Done Sent false
5683 2762 Acked Write 9 Done Done Sent false
5684 2763 Acked Write 3 Done Done Sent false
5685 2764 Acked Read 6 Done Done Sent false
5686 2765 Acked Write 9 Done Done Sent false
5687 2766 Acked Read 9 Done Done Sent false
5688 2767 Acked Read 7 Done Done Sent false
5689 2768 Acked Read 4 Done Done Sent false
5690 2769 Acked Write 10 Done Done Sent false
5691 2770 Acked Write 4 Done Done Sent false
5692 2771 Acked Write 6 Done Done Sent false
5693 2772 Acked Flush 0 Done Done Sent false
5694 2773 Acked Write 3 Done Done Sent false
5695 2774 Acked Write 8 Done Done Sent false
5696 2775 Acked Read 10 Done Done Sent false
5697 2776 Acked Read 7 Done Done Sent false
5698 2777 Acked Write 7 Done Done Sent false
5699 2778 Acked Read 7 Done Done Sent false
5700 2779 Acked Flush 0 Done Done Sent false
5701 2780 Acked Read 4 Done Done Sent false
5702 2781 Acked Read 3 Done Done Sent false
5703 2782 Acked Read 8 Done Done Sent false
5704 2783 Acked Write 6 Done Done Sent false
5705 2784 Acked Read 5 Done Done Sent false
5706 2785 Acked Write 3 Done Done Sent false
5707 2786 Acked Read 10 Done Done Sent false
5708 2787 Acked Read 4 Done Done Sent false
5709 2788 Acked Write 10 Done Done Sent false
5710 2789 Acked Read 4 Done Done Sent false
5711 2790 Acked Read 7 Done Done Sent false
5712 2791 Acked Flush 0 Done Done Sent false
5713 2792 Acked Read 2 Done Done Sent false
5714 2793 Acked Write 9 Done Done Sent false
5715 2794 Acked Write 9 Done Done Sent false
5716 2795 Acked Read 3 Done Done Sent false
5717 2796 Acked Read 3 Done Done Sent false
5718 2797 Acked Read 4 Done Done Sent false
5719 2798 Acked Write 2 Done Done Sent false
5720 2799 Acked Write 8 Done Done Sent false
5721 2800 Acked Write 2 Done Done Sent false
5722 2801 Acked Read 9 Done Done Sent false
5723 2802 Acked Flush 0 Done Done Sent false
5724 2803 Acked Read 2 Done Done Sent false
5725 2804 Acked Write 8 Done Done Sent false
5726 2805 Acked Write 3 Done Done Sent false
5727 2806 Acked Write 2 Done Done Sent false
5728 2807 Acked Write 5 Done Done Sent false
5729 2808 Acked Read 5 Done Done Sent false
5730 2809 Acked Read 6 Done Done Sent false
5731 2810 Acked Write 6 Done Done Sent false
5732 STATES DS:0 DS:1 DS:2 TOTAL
5733 Sent 0 0 301 301
5734 Done 301 301 0 602
5735 Skipped 0 0 0 0
5736 Error 0 0 0 0
5737 Last Flush: 2802 2802 2509
5738 Downstairs last five completed: 2509 2508 2507 2506 2505
5739 Upstairs last five completed: 2810 2809 2808 2807 2806
5740 CLIENT: Up:0 ds:301 act:2
5741 [8830] got start action
5742 [8830] I am going to restart
5743 Starting downstairs at port 8830
5744 Make output file at "/tmp/dsc/downstairs-8830.txt"
5745 [8830][2] reports Starting
5746 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1207)
5747 [8830][2] reports Running
5748 ----------------------------------------------------------------
5749 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
5750 States: Active Active Offline
5751 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5752 2510 Acked Flush 0 Done Done Sent false
5753 2511 Acked Write 6 Done Done Sent false
5754 2512 Acked Write 6 Done Done Sent false
5755 2513 Acked Write 3 Done Done Sent false
5756 2514 Acked Read 10 Done Done Sent false
5757 2515 Acked Read 2 Done Done Sent false
5758 2516 Acked Write 8 Done Done Sent false
5759 2517 Acked Write 1 Done Done Sent false
5760 2518 Acked Write 5 Done Done Sent false
5761 2519 Acked Read 1 Done Done Sent false
5762 2520 Acked Read 7 Done Done Sent false
5763 2521 Acked Read 4 Done Done Sent false
5764 2522 Acked Read 1 Done Done Sent false
5765 2523 Acked Write 9 Done Done Sent false
5766 2524 Acked Read 3 Done Done Sent false
5767 2525 Acked Write 10 Done Done Sent false
5768 2526 Acked Write 9 Done Done Sent false
5769 2527 Acked Write 1 Done Done Sent false
5770 2528 Acked Read 1 Done Done Sent false
5771 2529 Acked Flush 0 Done Done Sent false
5772 2530 Acked Read 6 Done Done Sent false
5773 2531 Acked Flush 0 Done Done Sent false
5774 2532 Acked Read 3 Done Done Sent false
5775 2533 Acked Write 7 Done Done Sent false
5776 2534 Acked Write 9 Done Done Sent false
5777 2535 Acked Write 1 Done Done Sent false
5778 2536 Acked Read 8 Done Done Sent false
5779 2537 Acked Read 7 Done Done Sent false
5780 2538 Acked Read 10 Done Done Sent false
5781 2539 Acked Read 7 Done Done Sent false
5782 2540 Acked Write 10 Done Done Sent false
5783 2541 Acked Flush 0 Done Done Sent false
5784 2542 Acked Read 5 Done Done Sent false
5785 2543 Acked Read 6 Done Done Sent false
5786 2544 Acked Read 8 Done Done Sent false
5787 2545 Acked Write 5 Done Done Sent false
5788 2546 Acked Read 8 Done Done Sent false
5789 2547 Acked Write 9 Done Done Sent false
5790 2548 Acked Read 3 Done Done Sent false
5791 2549 Acked Read 10 Done Done Sent false
5792 2550 Acked Write 10 Done Done Sent false
5793 2551 Acked Write 5 Done Done Sent false
5794 2552 Acked Read 4 Done Done Sent false
5795 2553 Acked Read 3 Done Done Sent false
5796 2554 Acked Write 8 Done Done Sent false
5797 2555 Acked Read 6 Done Done Sent false
5798 2556 Acked Write 6 Done Done Sent false
5799 2557 Acked Write 3 Done Done Sent false
5800 2558 Acked Write 7 Done Done Sent false
5801 2559 Acked Flush 0 Done Done Sent false
5802 2560 Acked Read 1 Done Done Sent false
5803 2561 Acked Write 7 Done Done Sent false
5804 2562 Acked Read 1 Done Done Sent false
5805 2563 Acked Read 8 Done Done Sent false
5806 2564 Acked Write 10 Done Done Sent false
5807 2565 Acked Write 8 Done Done Sent false
5808 2566 Acked Read 7 Done Done Sent false
5809 2567 Acked Write 4 Done Done Sent false
5810 2568 Acked Flush 0 Done Done Sent false
5811 2569 Acked Read 10 Done Done Sent false
5812 2570 Acked Read 6 Done Done Sent false
5813 2571 Acked Write 4 Done Done Sent false
5814 2572 Acked Read 7 Done Done Sent false
5815 2573 Acked Read 2 Done Done Sent false
5816 2574 Acked Write 3 Done Done Sent false
5817 2575 Acked Flush 0 Done Done Sent false
5818 2576 Acked Read 8 Done Done Sent false
5819 2577 Acked Write 4 Done Done Sent false
5820 2578 Acked Read 3 Done Done Sent false
5821 2579 Acked Write 4 Done Done Sent false
5822 2580 Acked Write 4 Done Done Sent false
5823 2581 Acked Read 6 Done Done Sent false
5824 2582 Acked Read 2 Done Done Sent false
5825 2583 Acked Write 10 Done Done Sent false
5826 2584 Acked Read 4 Done Done Sent false
5827 2585 Acked Write 5 Done Done Sent false
5828 2586 Acked Write 2 Done Done Sent false
5829 2587 Acked Write 3 Done Done Sent false
5830 2588 Acked Read 6 Done Done Sent false
5831 2589 Acked Write 9 Done Done Sent false
5832 2590 Acked Write 10 Done Done Sent false
5833 2591 Acked Write 5 Done Done Sent false
5834 2592 Acked Write 2 Done Done Sent false
5835 2593 Acked Write 10 Done Done Sent false
5836 2594 Acked Write 8 Done Done Sent false
5837 2595 Acked Write 2 Done Done Sent false
5838 2596 Acked Read 9 Done Done Sent false
5839 2597 Acked Write 5 Done Done Sent false
5840 2598 Acked Read 8 Done Done Sent false
5841 2599 Acked Write 6 Done Done Sent false
5842 2600 Acked Write 1 Done Done Sent false
5843 2601 Acked Flush 0 Done Done Sent false
5844 2602 Acked Flush 0 Done Done Sent false
5845 2603 Acked Read 2 Done Done Sent false
5846 2604 Acked Write 1 Done Done Sent false
5847 2605 Acked Read 3 Done Done Sent false
5848 2606 Acked Write 10 Done Done Sent false
5849 2607 Acked Read 10 Done Done Sent false
5850 2608 Acked Read 8 Done Done Sent false
5851 2609 Acked Read 8 Done Done Sent false
5852 2610 Acked Write 3 Done Done Sent false
5853 2611 Acked Write 1 Done Done Sent false
5854 2612 Acked Flush 0 Done Done Sent false
5855 2613 Acked Flush 0 Done Done Sent false
5856 2614 Acked Write 8 Done Done Sent false
5857 2615 Acked Read 2 Done Done Sent false
5858 2616 Acked Read 4 Done Done Sent false
5859 2617 Acked Write 3 Done Done Sent false
5860 2618 Acked Read 5 Done Done Sent false
5861 2619 Acked Read 4 Done Done Sent false
5862 2620 Acked Write 9 Done Done Sent false
5863 2621 Acked Write 4 Done Done Sent false
5864 2622 Acked Read 1 Done Done Sent false
5865 2623 Acked Flush 0 Done Done Sent false
5866 2624 Acked Write 8 Done Done Sent false
5867 2625 Acked Read 5 Done Done Sent false
5868 2626 Acked Read 4 Done Done Sent false
5869 2627 Acked Write 3 Done Done Sent false
5870 2628 Acked Write 10 Done Done Sent false
5871 2629 Acked Write 7 Done Done Sent false
5872 2630 Acked Write 10 Done Done Sent false
5873 2631 Acked Write 1 Done Done Sent false
5874 2632 Acked Flush 0 Done Done Sent false
5875 2633 Acked Read 5 Done Done Sent false
5876 2634 Acked Read 1 Done Done Sent false
5877 2635 Acked Read 6 Done Done Sent false
5878 2636 Acked Read 9 Done Done Sent false
5879 2637 Acked Read 6 Done Done Sent false
5880 2638 Acked Write 7 Done Done Sent false
5881 2639 Acked Flush 0 Done Done Sent false
5882 2640 Acked Write 9 Done Done Sent false
5883 2641 Acked Write 8 Done Done Sent false
5884 2642 Acked Read 4 Done Done Sent false
5885 2643 Acked Flush 0 Done Done Sent false
5886 2644 Acked Read 1 Done Done Sent false
5887 2645 Acked Read 2 Done Done Sent false
5888 2646 Acked Write 8 Done Done Sent false
5889 2647 Acked Write 7 Done Done Sent false
5890 2648 Acked Read 2 Done Done Sent false
5891 2649 Acked Read 10 Done Done Sent false
5892 2650 Acked Write 4 Done Done Sent false
5893 2651 Acked Read 1 Done Done Sent false
5894 2652 Acked Write 9 Done Done Sent false
5895 2653 Acked Read 2 Done Done Sent false
5896 2654 Acked Write 7 Done Done Sent false
5897 2655 Acked Write 9 Done Done Sent false
5898 2656 Acked Flush 0 Done Done Sent false
5899 2657 Acked Read 6 Done Done Sent false
5900 2658 Acked Read 4 Done Done Sent false
5901 2659 Acked Read 2 Done Done Sent false
5902 2660 Acked Read 5 Done Done Sent false
5903 2661 Acked Read 1 Done Done Sent false
5904 2662 Acked Write 9 Done Done Sent false
5905 2663 Acked Write 5 Done Done Sent false
5906 2664 Acked Read 10 Done Done Sent false
5907 2665 Acked Read 9 Done Done Sent false
5908 2666 Acked Write 2 Done Done Sent false
5909 2667 Acked Write 8 Done Done Sent false
5910 2668 Acked Write 5 Done Done Sent false
5911 2669 Acked Read 7 Done Done Sent false
5912 2670 Acked Write 10 Done Done Sent false
5913 2671 Acked Write 6 Done Done Sent false
5914 2672 Acked Read 9 Done Done Sent false
5915 2673 Acked Write 8 Done Done Sent false
5916 2674 Acked Read 7 Done Done Sent false
5917 2675 Acked Read 3 Done Done Sent false
5918 2676 Acked Write 7 Done Done Sent false
5919 2677 Acked Write 6 Done Done Sent false
5920 2678 Acked Write 4 Done Done Sent false
5921 2679 Acked Read 2 Done Done Sent false
5922 2680 Acked Flush 0 Done Done Sent false
5923 2681 Acked Read 9 Done Done Sent false
5924 2682 Acked Read 4 Done Done Sent false
5925 2683 Acked Write 6 Done Done Sent false
5926 2684 Acked Write 8 Done Done Sent false
5927 2685 Acked Read 4 Done Done Sent false
5928 2686 Acked Read 4 Done Done Sent false
5929 2687 Acked Read 3 Done Done Sent false
5930 2688 Acked Flush 0 Done Done Sent false
5931 2689 Acked Read 4 Done Done Sent false
5932 2690 Acked Write 10 Done Done Sent false
5933 2691 Acked Write 5 Done Done Sent false
5934 2692 Acked Write 9 Done Done Sent false
5935 2693 Acked Flush 0 Done Done Sent false
5936 2694 Acked Write 7 Done Done Sent false
5937 2695 Acked Write 5 Done Done Sent false
5938 2696 Acked Flush 0 Done Done Sent false
5939 2697 Acked Write 10 Done Done Sent false
5940 2698 Acked Flush 0 Done Done Sent false
5941 2699 Acked Write 3 Done Done Sent false
5942 2700 Acked Read 9 Done Done Sent false
5943 2701 Acked Read 9 Done Done Sent false
5944 2702 Acked Write 9 Done Done Sent false
5945 2703 Acked Read 4 Done Done Sent false
5946 2704 Acked Write 2 Done Done Sent false
5947 2705 Acked Read 2 Done Done Sent false
5948 2706 Acked Write 8 Done Done Sent false
5949 2707 Acked Read 6 Done Done Sent false
5950 2708 Acked Read 9 Done Done Sent false
5951 2709 Acked Read 9 Done Done Sent false
5952 2710 Acked Write 6 Done Done Sent false
5953 2711 Acked Write 1 Done Done Sent false
5954 2712 Acked Read 10 Done Done Sent false
5955 2713 Acked Flush 0 Done Done Sent false
5956 2714 Acked Read 10 Done Done Sent false
5957 2715 Acked Write 5 Done Done Sent false
5958 2716 Acked Flush 0 Done Done Sent false
5959 2717 Acked Write 1 Done Done Sent false
5960 2718 Acked Write 7 Done Done Sent false
5961 2719 Acked Read 5 Done Done Sent false
5962 2720 Acked Write 1 Done Done Sent false
5963 2721 Acked Write 1 Done Done Sent false
5964 2722 Acked Flush 0 Done Done Sent false
5965 2723 Acked Flush 0 Done Done Sent false
5966 2724 Acked Write 5 Done Done Sent false
5967 2725 Acked Read 4 Done Done Sent false
5968 2726 Acked Read 9 Done Done Sent false
5969 2727 Acked Write 8 Done Done Sent false
5970 2728 Acked Read 10 Done Done Sent false
5971 2729 Acked Write 4 Done Done Sent false
5972 2730 Acked Read 7 Done Done Sent false
5973 2731 Acked Read 4 Done Done Sent false
5974 2732 Acked Write 6 Done Done Sent false
5975 2733 Acked Read 10 Done Done Sent false
5976 2734 Acked Read 2 Done Done Sent false
5977 2735 Acked Read 6 Done Done Sent false
5978 2736 Acked Read 2 Done Done Sent false
5979 2737 Acked Read 4 Done Done Sent false
5980 2738 Acked Read 7 Done Done Sent false
5981 2739 Acked Write 6 Done Done Sent false
5982 2740 Acked Write 4 Done Done Sent false
5983 2741 Acked Write 3 Done Done Sent false
5984 2742 Acked Read 9 Done Done Sent false
5985 2743 Acked Write 3 Done Done Sent false
5986 2744 Acked Read 6 Done Done Sent false
5987 2745 Acked Write 10 Done Done Sent false
5988 2746 Acked Write 2 Done Done Sent false
5989 2747 Acked Write 6 Done Done Sent false
5990 2748 Acked Read 7 Done Done Sent false
5991 2749 Acked Read 2 Done Done Sent false
5992 2750 Acked Write 9 Done Done Sent false
5993 2751 Acked Write 6 Done Done Sent false
5994 2752 Acked Write 4 Done Done Sent false
5995 2753 Acked Write 1 Done Done Sent false
5996 2754 Acked Read 4 Done Done Sent false
5997 2755 Acked Write 7 Done Done Sent false
5998 2756 Acked Write 4 Done Done Sent false
5999 2757 Acked Read 9 Done Done Sent false
6000 2758 Acked Read 6 Done Done Sent false
6001 2759 Acked Write 1 Done Done Sent false
6002 2760 Acked Write 5 Done Done Sent false
6003 2761 Acked Read 3 Done Done Sent false
6004 2762 Acked Write 9 Done Done Sent false
6005 2763 Acked Write 3 Done Done Sent false
6006 2764 Acked Read 6 Done Done Sent false
6007 2765 Acked Write 9 Done Done Sent false
6008 2766 Acked Read 9 Done Done Sent false
6009 2767 Acked Read 7 Done Done Sent false
6010 2768 Acked Read 4 Done Done Sent false
6011 2769 Acked Write 10 Done Done Sent false
6012 2770 Acked Write 4 Done Done Sent false
6013 2771 Acked Write 6 Done Done Sent false
6014 2772 Acked Flush 0 Done Done Sent false
6015 2773 Acked Write 3 Done Done Sent false
6016 2774 Acked Write 8 Done Done Sent false
6017 2775 Acked Read 10 Done Done Sent false
6018 2776 Acked Read 7 Done Done Sent false
6019 2777 Acked Write 7 Done Done Sent false
6020 2778 Acked Read 7 Done Done Sent false
6021 2779 Acked Flush 0 Done Done Sent false
6022 2780 Acked Read 4 Done Done Sent false
6023 2781 Acked Read 3 Done Done Sent false
6024 2782 Acked Read 8 Done Done Sent false
6025 2783 Acked Write 6 Done Done Sent false
6026 2784 Acked Read 5 Done Done Sent false
6027 2785 Acked Write 3 Done Done Sent false
6028 2786 Acked Read 10 Done Done Sent false
6029 2787 Acked Read 4 Done Done Sent false
6030 2788 Acked Write 10 Done Done Sent false
6031 2789 Acked Read 4 Done Done Sent false
6032 2790 Acked Read 7 Done Done Sent false
6033 2791 Acked Flush 0 Done Done Sent false
6034 2792 Acked Read 2 Done Done Sent false
6035 2793 Acked Write 9 Done Done Sent false
6036 2794 Acked Write 9 Done Done Sent false
6037 2795 Acked Read 3 Done Done Sent false
6038 2796 Acked Read 3 Done Done Sent false
6039 2797 Acked Read 4 Done Done Sent false
6040 2798 Acked Write 2 Done Done Sent false
6041 2799 Acked Write 8 Done Done Sent false
6042 2800 Acked Write 2 Done Done Sent false
6043 2801 Acked Read 9 Done Done Sent false
6044 2802 Acked Flush 0 Done Done Sent false
6045 2803 Acked Read 2 Done Done Sent false
6046 2804 Acked Write 8 Done Done Sent false
6047 2805 Acked Write 3 Done Done Sent false
6048 2806 Acked Write 2 Done Done Sent false
6049 2807 Acked Write 5 Done Done Sent false
6050 2808 Acked Read 5 Done Done Sent false
6051 2809 Acked Read 6 Done Done Sent false
6052 2810 Acked Write 6 Done Done Sent false
6053 2811 Acked Flush 0 Done Done Sent false
6054 STATES DS:0 DS:1 DS:2 TOTAL
6055 Sent 0 0 302 302
6056 Done 302 302 0 604
6057 Skipped 0 0 0 0
6058 Error 0 0 0 0
6059 Last Flush: 2811 2811 2509
6060 Downstairs last five completed: 2509 2508 2507 2506 2505
6061 Upstairs last five completed: 2811 2810 2809 2808 2807
6062 CLIENT: Up:0 ds:302 act:2
60632024-12-09T15:09:06.348ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
60642024-12-09T15:09:06.348ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43640) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
60652024-12-09T15:09:06.349ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
60662024-12-09T15:09:06.349ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
60672024-12-09T15:09:06.349ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
60682024-12-09T15:09:06.349ZINFOcrucible: send last flush ID to this DS: 2509 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
60692024-12-09T15:09:06.350ZINFOcrucible: Replied this last flush ID: 2509 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
60702024-12-09T15:09:06.350ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
60712024-12-09T15:09:06.350ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
60722024-12-09T15:09:06.350ZINFOcrucible: [2] client re-new 302 jobs since flush 2509 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
60732024-12-09T15:09:06.350ZINFOcrucible: [2] Replayed 302 jobs since flush: 2509 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
6074 ----------------------------------------------------------------
6075 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6076 Downstairs last five completed: 2811 2810 2809 2808 2807
6077 Upstairs last five completed: 2811 2810 2809 2808 2807
6078 CLIENT: Up:0 ds:0 act:3
6079 Replay: All jobs finished
6080 Main task has work to do, go find it
6081 got dsc Stop(2)
6082 stop 2
6083 Dec 09 15:09:08.580 INFO request completed, latency_us: 354[8830] Got stop action so:false kr:false
6084 , response_code: 204, uri: /stop/cid/2, method: POST, req_id: 04b50b17-b3d3-42e0-8639-122074382853, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
60852024-12-09T15:09:08.581ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
60862024-12-09T15:09:08.581ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
6087 Dec 09 15:09:08.581 INFO request completed{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:09:08.581576571Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
6088 , latency_us: 138, response_code: 200, uri: /state/cid/2, method: {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:09:08.581611055Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
6089 GET, req_id: cda9ae04-4044-4abd-8cae-f0106874c833, remote_addr: {"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:09:08.5816383Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
6090 127.0.0.1:47510, local_addr: 127.0.0.1:9998
60912024-12-09T15:09:08.581ZINFOcrucible: client task is exiting = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
6092 [8830] Exited with: ExitStatus(unix_wait_status(9))
6093 [8830][2] reports Exit
6094 Dec 09 15:09:12.583 INFO request completed, latency_us: 273, response_code: 200, uri: /state/cid/2, method: GET, req_id: be4d93b4-ccf0-464a-8255-8261fa685c7f, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
6095 001/300 Read block 971 len 32768
6096 002/300 Read block 1217 len 40960
6097 003/300 Read block 2118 len 12288
6098 004/300 Write block 4387 len 8192 data: 3 3
6099 005/300 Write block 3979 len 36864 data: 4 3 2 2 2 2 3 3 3
6100 006/300 Read block 2038 len 4096
6101 007/300 Read block 1966 len 40960
6102 008/300 Read block 1534 len 16384
6103 009/300 Read block 2712 len 32768
6104 010/300 Read block 2963 len 32768
6105 011/300 Write block 3375 len 4096 data: 2
6106 012/300 Read block 3031 len 12288
6107 013/300 Write block 3457 len 32768 data: 2 2 2 2 2 2 3 4
6108 014/300 Write block 1544 len 16384 data: 3 3 3 3
6109 015/300 Read block 2377 len 8192
6110 016/300 Read block 1889 len 8192
6111 017/300 Read block 3471 len 32768
6112 018/300 Read block 2995 len 20480
6113 019/300 Read block 964 len 40960
6114 020/300 Write block 1840 len 16384 data: 5 5 5 3
6115 021/300 Read block 2644 len 24576
6116 022/300 Read block 1530 len 20480
6117 023/300 Write block 2930 len 16384 data: 2 2 2 2
6118 024/300 Flush
6119 025/300 Read block 3091 len 12288
6120 026/300 Read block 4140 len 4096
6121 027/300 Read block 1788 len 28672
6122 028/300 Read block 3273 len 16384
6123 029/300 Read block 4813 len 32768
6124 030/300 Read block 1032 len 16384
6125 031/300 Read block 4331 len 24576
6126 032/300 Write block 1848 len 16384 data: 3 3 2 2
6127 033/300 Read block 4360 len 20480
6128 034/300 Write block 4980 len 40960 data: 3 3 3 3 2 3 3 4 4 4
6129 035/300 Read block 1470 len 8192
6130 036/300 Write block 793 len 28672 data: 2 2 2 2 2 3 2
6131 037/300 Flush
6132 038/300 Read block 1914 len 20480
6133 039/300 Read block 2170 len 16384
6134 040/300 Write block 2182 len 12288 data: 4 4 3
6135 041/300 Write block 713 len 12288 data: 2 2 2
6136 042/300 Read block 357 len 20480
6137 043/300 Read block 2504 len 32768
6138 044/300 Read block 3044 len 12288
6139 045/300 Read block 841 len 20480
6140 046/300 Write block 4445 len 20480 data: 3 3 3 3 3
6141 047/300 Write block 4177 len 32768 data: 3 2 2 2 2 2 2 2
6142 048/300 Write block 3676 len 40960 data: 3 3 2 3 3 3 3 3 3 3
6143 049/300 Read block 1585 len 4096
6144 050/300 Write block 1752 len 32768 data: 3 4 3 3 3 3 3 2
6145 051/300 Write block 3174 len 36864 data: 2 3 3 3 3 3 3 3 3
6146 052/300 Write block 2013 len 24576 data: 3 4 4 5 4 4
6147 053/300 Read block 2063 len 40960
6148 054/300 Read block 4441 len 8192
6149 055/300 Read block 803 len 28672
6150 056/300 Write block 2512 len 28672 data: 2 3 3 3 3 3 3
6151 057/300 Write block 4780 len 32768 data: 3 3 3 3 4 3 2 2
6152 058/300 Read block 2176 len 20480
6153 059/300 Read block 2461 len 40960
6154 060/300 Read block 1309 len 36864
6155 061/300 Write block 2226 len 20480 data: 4 3 2 3 3
6156 062/300 Read block 1469 len 32768
6157 063/300 Read block 736 len 32768
6158 064/300 Read block 4431 len 24576
6159 065/300 Write block 799 len 32768 data: 3 2 4 4 4 4 2 2
6160 066/300 Write block 4499 len 24576 data: 3 3 3 3 2 2
6161 067/300 Read block 3179 len 32768
6162 068/300 Write block 2436 len 36864 data: 2 2 2 2 2 3 3 4 4
6163 069/300 Write block 4508 len 4096 data: 4
6164 070/300 Read block 3024 len 24576
6165 071/300 Write block 3144 len 32768 data: 4 4 4 4 3 3 4 4
6166 072/300 Write block 4303 len 12288 data: 2 2 2
6167 073/300 Read block 4003 len 16384
6168 074/300 Read block 1250 len 4096
6169 075/300 Write block 4538 len 12288 data: 2 2 2
6170 076/300 Write block 1318 len 28672 data: 3 2 2 2 2 2 2
6171 077/300 Read block 2813 len 16384
6172 078/300 Write block 2393 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6173 079/300 Write block 4549 len 40960 data: 2 3 3 3 3 3 3 4 3 3
6174 080/300 Write block 2768 len 28672 data: 2 3 3 3 3 3 3
6175 081/300 Read block 4917 len 16384
6176 082/300 Write block 4678 len 32768 data: 3 3 2 2 2 2 2 2
6177 083/300 Read block 1686 len 24576
6178 084/300 Read block 3877 len 8192
6179 085/300 Write block 4570 len 28672 data: 2 2 3 3 4 3 3
6180 086/300 Read block 1112 len 20480
6181 087/300 Write block 1039 len 4096 data: 3
6182 088/300 Write block 808 len 40960 data: 2 2 3 3 3 3 3 3 3 3
6183 089/300 Read block 3601 len 28672
6184 090/300 Read block 4678 len 40960
6185 091/300 Write block 3285 len 32768 data: 3 4 4 4 4 4 3 3
6186 092/300 Write block 1419 len 16384 data: 3 2 2 2
6187 093/300 Write block 4689 len 12288 data: 2 2 2
6188 094/300 Read block 1052 len 28672
6189 095/300 Flush
6190 096/300 Write block 2063 len 24576 data: 2 2 2 2 2 2
6191 097/300 Read block 4312 len 20480
6192 098/300 Read block 2482 len 12288
6193 099/300 Read block 738 len 8192
6194 100/300 Write block 533 len 8192 data: 2 2
6195 101/300 Read block 2034 len 12288
6196 102/300 Write block 4600 len 12288 data: 4 5 4
6197 103/300 Read block 3848 len 36864
6198 104/300 Write block 522 len 28672 data: 3 3 2 2 2 2 2
6199 105/300 Write block 2104 len 24576 data: 3 4 4 5 4 3
6200 106/300 Read block 544 len 16384
6201 107/300 Write block 4183 len 40960 data: 3 3 3 3 3 3 3 3 2 2
6202 108/300 Write block 1008 len 4096 data: 2
6203 109/300 Write block 137 len 40960 data: 5 5 5 5 4 4 2 2 2 2
6204 110/300 Read block 1878 len 16384
6205 111/300 Write block 2711 len 40960 data: 4 4 4 3 3 3 3 3 3 2
6206 112/300 Read block 3498 len 8192
6207 113/300 Write block 2428 len 36864 data: 2 2 2 2 2 2 2 2 3
6208 114/300 Write block 1322 len 4096 data: 3
6209 115/300 Write block 2858 len 8192 data: 2 2
6210 116/300 Write block 2241 len 20480 data: 2 2 2 2 2
6211 117/300 Write block 1089 len 20480 data: 3 3 3 3 3
6212 118/300 Read block 3011 len 16384
6213 119/300 Write block 467 len 28672 data: 4 2 2 2 2 2 2
6214 120/300 Write block 683 len 24576 data: 2 2 2 2 2 2
6215 121/300 Read block 4589 len 28672
6216 122/300 Write block 410 len 28672 data: 3 2 2 2 2 2 2
6217 123/300 Read block 3802 len 12288
6218 124/300 Read block 4778 len 32768
6219 125/300 Write block 819 len 24576 data: 3 2 2 2 2 2
6220 126/300 Read block 673 len 4096
6221 127/300 Write block 4958 len 4096 data: 2
6222 128/300 Write block 3368 len 4096 data: 3
6223 129/300 Read block 2840 len 12288
6224 130/300 Read block 3375 len 8192
6225 131/300 Read block 350 len 12288
6226 132/300 Write block 1311 len 32768 data: 3 4 4 4 3 4 3 4
6227 133/300 Read block 1677 len 36864
6228 134/300 Read block 1254 len 8192
6229 135/300 Write block 405 len 12288 data: 3 3 2
6230 136/300 Read block 3092 len 40960
6231 137/300 Read block 1562 len 12288
6232 138/300 Read block 2024 len 4096
6233 139/300 Write block 3433 len 8192 data: 4 5
6234 140/300 Write block 3665 len 16384 data: 2 2 2 2
6235 141/300 Flush
6236 142/300 Write block 2261 len 4096 data: 3
6237 143/300 Read block 4741 len 40960
6238 144/300 Write block 2494 len 40960 data: 4 4 3 3 3 3 3 3 2 3
6239 145/300 Write block 1967 len 24576 data: 2 2 2 2 2 2
6240 146/300 Read block 2696 len 24576
6241 147/300 Read block 2434 len 28672
6242 148/300 Write block 2649 len 32768 data: 2 2 2 2 3 3 3 3
6243 149/300 Write block 336 len 12288 data: 2 2 2
6244 150/300 Read block 940 len 36864
6245 151/300 Read block 1169 len 12288
6246 152/300 Write block 948 len 28672 data: 4 4 4 4 3 4 4
6247 153/300 Write block 372 len 28672 data: 2 2 2 2 2 2 2
6248 154/300 Read block 590 len 8192
6249 155/300 Write block 1756 len 40960 data: 4 4 4 3 2 2 3 3 3 3
6250 156/300 Read block 2388 len 40960
6251 157/300 Flush
6252 158/300 Read block 2411 len 12288
6253 159/300 Read block 3366 len 28672
6254 160/300 Write block 2331 len 28672 data: 2 2 4 5 6 6 6
6255 161/300 Read block 1730 len 24576
6256 162/300 Write block 3465 len 16384 data: 4 5 5 5
6257 163/300 Flush
6258 164/300 Flush
6259 165/300 Write block 708 len 16384 data: 2 2 2 2
6260 166/300 Read block 1549 len 16384
6261 167/300 Write block 985 len 32768 data: 2 2 2 2 3 3 3 2
6262 168/300 Write block 3024 len 24576 data: 3 4 3 2 2 2
6263 169/300 Read block 2138 len 8192
6264 170/300 Read block 1240 len 40960
6265 171/300 Write block 2058 len 16384 data: 2 2 2 2
6266 172/300 Write block 3223 len 24576 data: 2 2 2 2 2 2
6267 173/300 Write block 4094 len 32768 data: 3 3 4 4 4 4 2 2
6268 174/300 Write block 4660 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6269 175/300 Flush
6270 176/300 Read block 1332 len 20480
6271 177/300 Write block 1775 len 32768 data: 3 3 2 2 3 3 3 3
6272 178/300 Write block 3870 len 32768 data: 2 2 2 2 2 2 2 2
6273 179/300 Read block 4978 len 12288
6274 180/300 Read block 3326 len 24576
6275 181/300 Read block 734 len 12288
6276 182/300 Write block 1377 len 12288 data: 3 3 3
6277 183/300 Read block 3068 len 8192
6278 184/300 Flush
6279 185/300 Write block 475 len 40960 data: 2 2 2 3 3 3 3 5 4 4
6280 186/300 Read block 1259 len 16384
6281 187/300 Read block 2412 len 32768
6282 188/300 Flush
6283 189/300 Read block 4004 len 32768
6284 190/300 Write block 2896 len 28672 data: 3 3 3 4 3 3 2
6285 191/300 Write block 267 len 32768 data: 2 2 2 3 3 3 2 2
6286 192/300 Read block 3425 len 12288
6287 193/300 Read block 3264 len 36864
6288 194/300 Write block 4984 len 40960 data: 3 4 4 5 5 5 4 6 5 4
6289 195/300 Write block 536 len 12288 data: 2 2 3
6290 196/300 Flush
6291 197/300 Write block 2001 len 8192 data: 3 3
6292 198/300 Write block 2788 len 28672 data: 2 2 2 3 3 4 4
6293 199/300 Read block 4167 len 12288
6294 200/300 Read block 2132 len 40960
6295 201/300 Write block 582 len 4096 data: 5
6296 202/300 Write block 912 len 12288 data: 2 2 2
6297 203/300 Read block 985 len 28672
6298 204/300 Read block 3391 len 28672
6299 205/300 Read block 2725 len 16384
6300 206/300 Write block 4480 len 4096 data: 2
6301 207/300 Read block 4635 len 32768
6302 208/300 Flush
6303 209/300 Write block 1229 len 36864 data: 3 3 3 3 3 3 2 2 2
6304 210/300 Flush
6305 211/300 Read block 10 len 40960
6306 212/300 Write block 3447 len 36864 data: 2 2 2 2 2 2 3 3 3
6307 213/300 Read block 3284 len 40960
6308 214/300 Write block 2466 len 32768 data: 2 2 2 2 2 2 2 2
6309 215/300 Read block 3226 len 32768
6310 216/300 Write block 2282 len 36864 data: 2 2 2 2 2 2 2 2 2
6311 217/300 Flush
6312 218/300 Flush
6313 219/300 Read block 1243 len 20480
6314 220/300 Write block 3550 len 40960 data: 4 4 3 3 3 3 3 3 3 3
6315 221/300 Write block 2749 len 40960 data: 2 2 3 4 4 3 3 3 3 2
6316 222/300 Write block 1035 len 12288 data: 2 2 3
6317 223/300 Read block 2231 len 28672
6318 224/300 Write block 2401 len 4096 data: 3
6319 225/300 Read block 1104 len 8192
6320 226/300 Read block 1618 len 32768
6321 227/300 Read block 942 len 36864
6322 228/300 Read block 1750 len 8192
6323 229/300 Write block 3104 len 28672 data: 3 3 2 2 2 2 2
6324 230/300 Write block 2321 len 12288 data: 3 2 2
6325 231/300 Write block 2740 len 28672 data: 3 2 2 2 2 2 2
6326 232/300 Write block 2193 len 16384 data: 3 3 2 2
6327 233/300 Write block 1510 len 36864 data: 2 2 2 3 3 2 3 3 3
6328 234/300 Write block 1010 len 28672 data: 2 2 2 2 2 2 2
6329 235/300 Write block 3232 len 36864 data: 2 2 2 2 2 2 2 2 2
6330 236/300 Write block 93 len 24576 data: 2 2 2 2 2 2
6331 237/300 Write block 2986 len 24576 data: 5 6 5 4 3 3
6332 238/300 Flush
6333 239/300 Write block 3488 len 40960 data: 4 4 4 3 3 3 3 3 2 2
6334 240/300 Write block 145 len 36864 data: 3 3 2 2 5 5 5 5 5
6335 241/300 Flush
6336 242/300 Read block 2579 len 28672
6337 243/300 Read block 4791 len 24576
6338 244/300 Read block 2446 len 4096
6339 245/300 Write block 4089 len 16384 data: 4 5 3 3
6340 246/300 Read block 2943 len 4096
6341 247/300 Write block 428 len 40960 data: 2 3 3 3 3 3 2 2 2 2
6342 248/300 Read block 2046 len 36864
6343 249/300 Write block 2611 len 16384 data: 2 3 3 3
6344 250/300 Read block 709 len 40960
6345 251/300 Read block 19 len 32768
6346 252/300 Read block 1003 len 40960
6347 253/300 Write block 4932 len 16384 data: 2 2 2 2
6348 254/300 Write block 4154 len 24576 data: 3 2 3 3 4 4
6349 255/300 Write block 2022 len 4096 data: 4
6350 256/300 Flush
6351 257/300 Write block 4592 len 28672 data: 2 3 3 3 3 4 4
6352 258/300 Write block 4561 len 12288 data: 3 2 2
6353 259/300 Read block 4365 len 4096
6354 260/300 Write block 1956 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6355 261/300 Read block 4552 len 4096
6356 262/300 Write block 536 len 32768 data: 3 3 4 3 3 3 3 4
6357 263/300 Read block 3745 len 40960
6358 264/300 Write block 1969 len 12288 data: 3 3 3
6359 265/300 Write block 395 len 40960 data: 2 2 2 2 2 2 2 2 2 3
6360 266/300 Read block 1196 len 4096
6361 267/300 Read block 334 len 36864
6362 268/300 Read block 4448 len 4096
6363 269/300 Read block 958 len 16384
6364 270/300 Write block 4447 len 20480 data: 4 4 4 3 3
6365 271/300 Write block 1438 len 40960 data: 3 2 2 2 3 3 3 3 3 4
6366 272/300 Write block 2559 len 16384 data: 4 3 2 2
6367 273/300 Read block 142 len 24576
6368 274/300 Write block 809 len 20480 data: 3 4 4 4 4
6369 275/300 Read block 4504 len 12288
6370 276/300 Read block 3553 len 36864
6371 277/300 Read block 3468 len 40960
6372 278/300 Read block 761 len 24576
6373 279/300 Write block 2095 len 40960 data: 3 3 3 3 3 3 3 3 3 4
6374 280/300 Flush
6375 281/300 Read block 882 len 12288
6376 282/300 Read block 4183 len 32768
6377 283/300 Read block 2659 len 36864
6378 284/300 Read block 3012 len 36864
6379 285/300 Write block 546 len 12288 data: 4 5 5
6380 286/300 Read block 4381 len 24576
6381 287/300 Write block 1664 len 24576 data: 3 3 3 2 3 3
6382 288/300 Write block 410 len 4096 data: 4
6383 289/300 Write block 2384 len 40960 data: 2 2 2 3 3 3 3 2 2 3
6384 290/300 Write block 3170 len 8192 data: 2 2
6385 291/300 Write block 4626 len 12288 data: 4 3 3
6386 292/300 Flush
6387 293/300 Write block 317 len 16384 data: 2 3 3 4
6388 294/300 Read block 2847 len 40960
6389 295/300 Read block 4380 len 28672
6390 296/300 Write block 4622 len 32768 data: 3 4 3 4 5 4 4 3
6391 297/300 Read block 4845 len 28672
6392 298/300 Read block 2691 len 20480
6393 299/300 Read block 3130 len 28672
6394 300/300 Read block 2280 len 24576
6395 301/300 Read block 4330 len 36864
6396 Main task has work to do, go find it
6397 got dsc Start(2)
6398 start 2
6399 Dec 09 15:09:12.822 INFO request completed, latency_us: 240, response_code: 204, uri: /start/cid/2, method: POST, req_id: a0c6f339-8a2c-4221-93f1-8b93b7a7d348, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
6400 Replay: started 2, returned:Ok(())
6401 ----------------------------------------------------------------
6402 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6403 States: Active Active Offline
6404 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6405 2812 Acked Read 8 Done Done Sent false
6406 2813 Acked Read 10 Done Done Sent false
6407 2814 Acked Read 3 Done Done Sent false
6408 2815 Acked Write 2 Done Done Sent false
6409 2816 Acked Write 9 Done Done Sent false
6410 2817 Acked Read 1 Done Done Sent false
6411 2818 Acked Read 10 Done Done Sent false
6412 2819 Acked Read 4 Done Done Sent false
6413 2820 Acked Read 8 Done Done Sent false
6414 2821 Acked Read 8 Done Done Sent false
6415 2822 Acked Write 1 Done Done Sent false
6416 2823 Acked Read 3 Done Done Sent false
6417 2824 Acked Write 8 Done Done Sent false
6418 2825 Acked Write 4 Done Done Sent false
6419 2826 Acked Read 2 Done Done Sent false
6420 2827 Acked Read 2 Done Done Sent false
6421 2828 Acked Read 8 Done Done Sent false
6422 2829 Acked Read 5 Done Done Sent false
6423 2830 Acked Read 10 Done Done Sent false
6424 2831 Acked Write 4 Done Done Sent false
6425 2832 Acked Read 6 Done Done Sent false
6426 2833 Acked Read 5 Done Done Sent false
6427 2834 Acked Write 4 Done Done Sent false
6428 2835 Acked Flush 0 Done Done Sent false
6429 2836 Acked Read 3 Done Done Sent false
6430 2837 Acked Read 1 Done Done Sent false
6431 2838 Acked Read 7 Done Done Sent false
6432 2839 Acked Read 4 Done Done Sent false
6433 2840 Acked Read 8 Done Done Sent false
6434 2841 Acked Read 4 Done Done Sent false
6435 2842 Acked Read 6 Done Done Sent false
6436 2843 Acked Write 4 Done Done Sent false
6437 2844 Acked Read 5 Done Done Sent false
6438 2845 Acked Write 10 Done Done Sent false
6439 2846 Acked Read 2 Done Done Sent false
6440 2847 Acked Write 7 Done Done Sent false
6441 2848 Acked Flush 0 Done Done Sent false
6442 2849 Acked Read 5 Done Done Sent false
6443 2850 Acked Read 4 Done Done Sent false
6444 2851 Acked Write 3 Done Done Sent false
6445 2852 Acked Write 3 Done Done Sent false
6446 2853 Acked Read 5 Done Done Sent false
6447 2854 Acked Read 8 Done Done Sent false
6448 2855 Acked Read 3 Done Done Sent false
6449 2856 Acked Read 5 Done Done Sent false
6450 2857 Acked Write 5 Done Done Sent false
6451 2858 Acked Write 8 Done Done Sent false
6452 2859 Acked Write 10 Done Done Sent false
6453 2860 Acked Read 1 Done Done Sent false
6454 2861 Acked Write 8 Done Done Sent false
6455 2862 Acked Write 9 Done Done Sent false
6456 2863 Acked Write 6 Done Done Sent false
6457 2864 Acked Read 10 Done Done Sent false
6458 2865 Acked Read 2 Done Done Sent false
6459 2866 Acked Read 7 Done Done Sent false
6460 2867 Acked Write 7 Done Done Sent false
6461 2868 Acked Write 8 Done Done Sent false
6462 2869 Acked Read 5 Done Done Sent false
6463 2870 Acked Read 10 Done Done Sent false
6464 2871 Acked Read 9 Done Done Sent false
6465 2872 Acked Write 5 Done Done Sent false
6466 2873 Acked Read 8 Done Done Sent false
6467 2874 Acked Read 8 Done Done Sent false
6468 2875 Acked Read 6 Done Done Sent false
6469 2876 Acked Write 8 Done Done Sent false
6470 2877 Acked Write 6 Done Done Sent false
6471 2878 Acked Read 8 Done Done Sent false
6472 2879 Acked Write 9 Done Done Sent false
6473 2880 Acked Write 1 Done Done Sent false
6474 2881 Acked Read 6 Done Done Sent false
6475 2882 Acked Write 8 Done Done Sent false
6476 2883 Acked Write 3 Done Done Sent false
6477 2884 Acked Read 4 Done Done Sent false
6478 2885 Acked Read 1 Done Done Sent false
6479 2886 Acked Write 3 Done Done Sent false
6480 2887 Acked Write 7 Done Done Sent false
6481 2888 Acked Read 4 Done Done Sent false
6482 2889 Acked Write 10 Done Done Sent false
6483 2890 Acked Write 10 Done Done Sent false
6484 2891 Acked Write 7 Done Done Sent false
6485 2892 Acked Read 4 Done Done Sent false
6486 2893 Acked Write 8 Done Done Sent false
6487 2894 Acked Read 6 Done Done Sent false
6488 2895 Acked Read 2 Done Done Sent false
6489 2896 Acked Write 7 Done Done Sent false
6490 2897 Acked Read 5 Done Done Sent false
6491 2898 Acked Write 1 Done Done Sent false
6492 2899 Acked Write 10 Done Done Sent false
6493 2900 Acked Read 7 Done Done Sent false
6494 2901 Acked Read 10 Done Done Sent false
6495 2902 Acked Write 8 Done Done Sent false
6496 2903 Acked Write 4 Done Done Sent false
6497 2904 Acked Write 3 Done Done Sent false
6498 2905 Acked Read 7 Done Done Sent false
6499 2906 Acked Flush 0 Done Done Sent false
6500 2907 Acked Write 6 Done Done Sent false
6501 2908 Acked Read 5 Done Done Sent false
6502 2909 Acked Read 3 Done Done Sent false
6503 2910 Acked Read 2 Done Done Sent false
6504 2911 Acked Write 2 Done Done Sent false
6505 2912 Acked Read 3 Done Done Sent false
6506 2913 Acked Write 3 Done Done Sent false
6507 2914 Acked Read 9 Done Done Sent false
6508 2915 Acked Write 7 Done Done Sent false
6509 2916 Acked Write 6 Done Done Sent false
6510 2917 Acked Read 4 Done Done Sent false
6511 2918 Acked Write 10 Done Done Sent false
6512 2919 Acked Write 1 Done Done Sent false
6513 2920 Acked Write 10 Done Done Sent false
6514 2921 Acked Read 4 Done Done Sent false
6515 2922 Acked Write 10 Done Done Sent false
6516 2923 Acked Read 2 Done Done Sent false
6517 2924 Acked Write 9 Done Done Sent false
6518 2925 Acked Write 1 Done Done Sent false
6519 2926 Acked Write 2 Done Done Sent false
6520 2927 Acked Write 5 Done Done Sent false
6521 2928 Acked Write 5 Done Done Sent false
6522 2929 Acked Read 4 Done Done Sent false
6523 2930 Acked Write 7 Done Done Sent false
6524 2931 Acked Write 6 Done Done Sent false
6525 2932 Acked Read 7 Done Done Sent false
6526 2933 Acked Write 7 Done Done Sent false
6527 2934 Acked Read 3 Done Done Sent false
6528 2935 Acked Read 8 Done Done Sent false
6529 2936 Acked Write 6 Done Done Sent false
6530 2937 Acked Read 1 Done Done Sent false
6531 2938 Acked Write 1 Done Done Sent false
6532 2939 Acked Write 1 Done Done Sent false
6533 2940 Acked Read 3 Done Done Sent false
6534 2941 Acked Read 2 Done Done Sent false
6535 2942 Acked Read 3 Done Done Sent false
6536 2943 Acked Write 8 Done Done Sent false
6537 2944 Acked Read 9 Done Done Sent false
6538 2945 Acked Read 2 Done Done Sent false
6539 2946 Acked Write 3 Done Done Sent false
6540 2947 Acked Read 10 Done Done Sent false
6541 2948 Acked Read 3 Done Done Sent false
6542 2949 Acked Read 1 Done Done Sent false
6543 2950 Acked Write 2 Done Done Sent false
6544 2951 Acked Write 4 Done Done Sent false
6545 2952 Acked Flush 0 Done Done Sent false
6546 2953 Acked Write 1 Done Done Sent false
6547 2954 Acked Read 10 Done Done Sent false
6548 2955 Acked Write 10 Done Done Sent false
6549 2956 Acked Write 6 Done Done Sent false
6550 2957 Acked Read 6 Done Done Sent false
6551 2958 Acked Read 7 Done Done Sent false
6552 2959 Acked Write 8 Done Done Sent false
6553 2960 Acked Write 3 Done Done Sent false
6554 2961 Acked Read 9 Done Done Sent false
6555 2962 Acked Read 3 Done Done Sent false
6556 2963 Acked Write 7 Done Done Sent false
6557 2964 Acked Write 7 Done Done Sent false
6558 2965 Acked Read 2 Done Done Sent false
6559 2966 Acked Write 10 Done Done Sent false
6560 2967 Acked Read 10 Done Done Sent false
6561 2968 Acked Flush 0 Done Done Sent false
6562 2969 Acked Read 3 Done Done Sent false
6563 2970 Acked Read 7 Done Done Sent false
6564 2971 Acked Write 7 Done Done Sent false
6565 2972 Acked Read 6 Done Done Sent false
6566 2973 Acked Write 4 Done Done Sent false
6567 2974 Acked Flush 0 Done Done Sent false
6568 2975 Acked Flush 0 Done Done Sent false
6569 2976 Acked Write 4 Done Done Sent false
6570 2977 Acked Read 4 Done Done Sent false
6571 2978 Acked Write 8 Done Done Sent false
6572 2979 Acked Write 6 Done Done Sent false
6573 2980 Acked Read 2 Done Done Sent false
6574 2981 Acked Read 10 Done Done Sent false
6575 2982 Acked Write 4 Done Done Sent false
6576 2983 Acked Write 6 Done Done Sent false
6577 2984 Acked Write 8 Done Done Sent false
6578 2985 Acked Write 10 Done Done Sent false
6579 2986 Acked Flush 0 Done Done Sent false
6580 2987 Acked Read 5 Done Done Sent false
6581 2988 Acked Write 8 Done Done Sent false
6582 2989 Acked Write 8 Done Done Sent false
6583 2990 Acked Read 3 Done Done Sent false
6584 2991 Acked Read 6 Done Done Sent false
6585 2992 Acked Read 3 Done Done Sent false
6586 2993 Acked Write 3 Done Done Sent false
6587 2994 Acked Read 2 Done Done Sent false
6588 2995 Acked Flush 0 Done Done Sent false
6589 2996 Acked Write 10 Done Done Sent false
6590 2997 Acked Read 4 Done Done Sent false
6591 2998 Acked Read 8 Done Done Sent false
6592 2999 Acked Flush 0 Done Done Sent false
6593 3000 Acked Read 8 Done Done Sent false
6594 3001 Acked Write 7 Done Done Sent false
6595 3002 Acked Write 8 Done Done Sent false
6596 3003 Acked Read 3 Done Done Sent false
6597 3004 Acked Read 9 Done Done Sent false
6598 3005 Acked Write 10 Done Done Sent false
6599 3006 Acked Write 3 Done Done Sent false
6600 3007 Acked Flush 0 Done Done Sent false
6601 3008 Acked Write 2 Done Done Sent false
6602 3009 Acked Write 7 Done Done Sent false
6603 3010 Acked Read 3 Done Done Sent false
6604 3011 Acked Read 10 Done Done Sent false
6605 3012 Acked Write 1 Done Done Sent false
6606 3013 Acked Write 3 Done Done Sent false
6607 3014 Acked Read 7 Done Done Sent false
6608 3015 Acked Read 7 Done Done Sent false
6609 3016 Acked Read 4 Done Done Sent false
6610 3017 Acked Write 1 Done Done Sent false
6611 3018 Acked Read 8 Done Done Sent false
6612 3019 Acked Flush 0 Done Done Sent false
6613 3020 Acked Write 9 Done Done Sent false
6614 3021 Acked Flush 0 Done Done Sent false
6615 3022 Acked Read 10 Done Done Sent false
6616 3023 Acked Write 9 Done Done Sent false
6617 3024 Acked Read 10 Done Done Sent false
6618 3025 Acked Write 8 Done Done Sent false
6619 3026 Acked Read 8 Done Done Sent false
6620 3027 Acked Write 9 Done Done Sent false
6621 3028 Acked Flush 0 Done Done Sent false
6622 3029 Acked Flush 0 Done Done Sent false
6623 3030 Acked Read 5 Done Done Sent false
6624 3031 Acked Write 10 Done Done Sent false
6625 3032 Acked Write 10 Done Done Sent false
6626 3033 Acked Write 3 Done Done Sent false
6627 3034 Acked Read 7 Done Done Sent false
6628 3035 Acked Write 1 Done Done Sent false
6629 3036 Acked Read 2 Done Done Sent false
6630 3037 Acked Read 8 Done Done Sent false
6631 3038 Acked Read 9 Done Done Sent false
6632 3039 Acked Read 2 Done Done Sent false
6633 3040 Acked Write 7 Done Done Sent false
6634 3041 Acked Write 3 Done Done Sent false
6635 3042 Acked Write 7 Done Done Sent false
6636 3043 Acked Write 4 Done Done Sent false
6637 3044 Acked Write 9 Done Done Sent false
6638 3045 Acked Write 7 Done Done Sent false
6639 3046 Acked Write 9 Done Done Sent false
6640 3047 Acked Write 6 Done Done Sent false
6641 3048 Acked Write 6 Done Done Sent false
6642 3049 Acked Flush 0 Done Done Sent false
6643 3050 Acked Write 10 Done Done Sent false
6644 3051 Acked Write 9 Done Done Sent false
6645 3052 Acked Flush 0 Done Done Sent false
6646 3053 Acked Read 7 Done Done Sent false
6647 3054 Acked Read 6 Done Done Sent false
6648 3055 Acked Read 1 Done Done Sent false
6649 3056 Acked Write 4 Done Done Sent false
6650 3057 Acked Read 1 Done Done Sent false
6651 3058 Acked Write 10 Done Done Sent false
6652 3059 Acked Read 9 Done Done Sent false
6653 3060 Acked Write 4 Done Done Sent false
6654 3061 Acked Read 10 Done Done Sent false
6655 3062 Acked Read 8 Done Done Sent false
6656 3063 Acked Read 10 Done Done Sent false
6657 3064 Acked Write 4 Done Done Sent false
6658 3065 Acked Write 6 Done Done Sent false
6659 3066 Acked Write 1 Done Done Sent false
6660 3067 Acked Flush 0 Done Done Sent false
6661 3068 Acked Write 7 Done Done Sent false
6662 3069 Acked Write 3 Done Done Sent false
6663 3070 Acked Read 1 Done Done Sent false
6664 3071 Acked Write 10 Done Done Sent false
6665 3072 Acked Read 1 Done Done Sent false
6666 3073 Acked Write 8 Done Done Sent false
6667 3074 Acked Read 10 Done Done Sent false
6668 3075 Acked Write 3 Done Done Sent false
6669 3076 Acked Write 10 Done Done Sent false
6670 3077 Acked Read 1 Done Done Sent false
6671 3078 Acked Read 9 Done Done Sent false
6672 3079 Acked Read 1 Done Done Sent false
6673 3080 Acked Read 4 Done Done Sent false
6674 3081 Acked Write 5 Done Done Sent false
6675 3082 Acked Write 10 Done Done Sent false
6676 3083 Acked Write 4 Done Done Sent false
6677 3084 Acked Read 6 Done Done Sent false
6678 3085 Acked Write 5 Done Done Sent false
6679 3086 Acked Read 3 Done Done Sent false
6680 3087 Acked Read 9 Done Done Sent false
6681 3088 Acked Read 10 Done Done Sent false
6682 3089 Acked Read 6 Done Done Sent false
6683 3090 Acked Write 10 Done Done Sent false
6684 3091 Acked Flush 0 Done Done Sent false
6685 3092 Acked Read 3 Done Done Sent false
6686 3093 Acked Read 8 Done Done Sent false
6687 3094 Acked Read 9 Done Done Sent false
6688 3095 Acked Read 9 Done Done Sent false
6689 3096 Acked Write 3 Done Done Sent false
6690 3097 Acked Read 6 Done Done Sent false
6691 3098 Acked Write 6 Done Done Sent false
6692 3099 Acked Write 1 Done Done Sent false
6693 3100 Acked Write 10 Done Done Sent false
6694 3101 Acked Write 2 Done Done Sent false
6695 3102 Acked Write 3 Done Done Sent false
6696 3103 Acked Flush 0 Done Done Sent false
6697 3104 Acked Write 4 Done Done Sent false
6698 3105 Acked Read 10 Done Done Sent false
6699 3106 Acked Read 7 Done Done Sent false
6700 3107 Acked Write 8 Done Done Sent false
6701 3108 Acked Read 7 Done Done Sent false
6702 3109 Acked Read 5 Done Done Sent false
6703 3110 Acked Read 7 Done Done Sent false
6704 3111 Acked Read 6 Done Done Sent false
6705 3112 Acked Read 9 Done Done Sent false
6706 STATES DS:0 DS:1 DS:2 TOTAL
6707 Sent 0 0 301 301
6708 Done 301 301 0 602
6709 Skipped 0 0 0 0
6710 Error 0 0 0 0
6711 Last Flush: 3103 3103 2811
6712 Downstairs last five completed: 2811 2810 2809 2808 2807
6713 Upstairs last five completed: 3112 3111 3110 3109 3108
6714 CLIENT: Up:0 ds:301 act:2
6715 [8830] got start action
6716 [8830] I am going to restart
6717 Starting downstairs at port 8830
6718 Make output file at "/tmp/dsc/downstairs-8830.txt"
6719 [8830][2] reports Starting
6720 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1208)
6721 [8830][2] reports Running
6722 ----------------------------------------------------------------
6723 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
6724 States: Active Active Offline
6725 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6726 2812 Acked Read 8 Done Done Sent false
6727 2813 Acked Read 10 Done Done Sent false
6728 2814 Acked Read 3 Done Done Sent false
6729 2815 Acked Write 2 Done Done Sent false
6730 2816 Acked Write 9 Done Done Sent false
6731 2817 Acked Read 1 Done Done Sent false
6732 2818 Acked Read 10 Done Done Sent false
6733 2819 Acked Read 4 Done Done Sent false
6734 2820 Acked Read 8 Done Done Sent false
6735 2821 Acked Read 8 Done Done Sent false
6736 2822 Acked Write 1 Done Done Sent false
6737 2823 Acked Read 3 Done Done Sent false
6738 2824 Acked Write 8 Done Done Sent false
6739 2825 Acked Write 4 Done Done Sent false
6740 2826 Acked Read 2 Done Done Sent false
6741 2827 Acked Read 2 Done Done Sent false
6742 2828 Acked Read 8 Done Done Sent false
6743 2829 Acked Read 5 Done Done Sent false
6744 2830 Acked Read 10 Done Done Sent false
6745 2831 Acked Write 4 Done Done Sent false
6746 2832 Acked Read 6 Done Done Sent false
6747 2833 Acked Read 5 Done Done Sent false
6748 2834 Acked Write 4 Done Done Sent false
6749 2835 Acked Flush 0 Done Done Sent false
6750 2836 Acked Read 3 Done Done Sent false
6751 2837 Acked Read 1 Done Done Sent false
6752 2838 Acked Read 7 Done Done Sent false
6753 2839 Acked Read 4 Done Done Sent false
6754 2840 Acked Read 8 Done Done Sent false
6755 2841 Acked Read 4 Done Done Sent false
6756 2842 Acked Read 6 Done Done Sent false
6757 2843 Acked Write 4 Done Done Sent false
6758 2844 Acked Read 5 Done Done Sent false
6759 2845 Acked Write 10 Done Done Sent false
6760 2846 Acked Read 2 Done Done Sent false
6761 2847 Acked Write 7 Done Done Sent false
6762 2848 Acked Flush 0 Done Done Sent false
6763 2849 Acked Read 5 Done Done Sent false
6764 2850 Acked Read 4 Done Done Sent false
6765 2851 Acked Write 3 Done Done Sent false
6766 2852 Acked Write 3 Done Done Sent false
6767 2853 Acked Read 5 Done Done Sent false
6768 2854 Acked Read 8 Done Done Sent false
6769 2855 Acked Read 3 Done Done Sent false
6770 2856 Acked Read 5 Done Done Sent false
6771 2857 Acked Write 5 Done Done Sent false
6772 2858 Acked Write 8 Done Done Sent false
6773 2859 Acked Write 10 Done Done Sent false
6774 2860 Acked Read 1 Done Done Sent false
6775 2861 Acked Write 8 Done Done Sent false
6776 2862 Acked Write 9 Done Done Sent false
6777 2863 Acked Write 6 Done Done Sent false
6778 2864 Acked Read 10 Done Done Sent false
6779 2865 Acked Read 2 Done Done Sent false
6780 2866 Acked Read 7 Done Done Sent false
6781 2867 Acked Write 7 Done Done Sent false
6782 2868 Acked Write 8 Done Done Sent false
6783 2869 Acked Read 5 Done Done Sent false
6784 2870 Acked Read 10 Done Done Sent false
6785 2871 Acked Read 9 Done Done Sent false
6786 2872 Acked Write 5 Done Done Sent false
6787 2873 Acked Read 8 Done Done Sent false
6788 2874 Acked Read 8 Done Done Sent false
6789 2875 Acked Read 6 Done Done Sent false
6790 2876 Acked Write 8 Done Done Sent false
6791 2877 Acked Write 6 Done Done Sent false
6792 2878 Acked Read 8 Done Done Sent false
6793 2879 Acked Write 9 Done Done Sent false
6794 2880 Acked Write 1 Done Done Sent false
6795 2881 Acked Read 6 Done Done Sent false
6796 2882 Acked Write 8 Done Done Sent false
6797 2883 Acked Write 3 Done Done Sent false
6798 2884 Acked Read 4 Done Done Sent false
6799 2885 Acked Read 1 Done Done Sent false
6800 2886 Acked Write 3 Done Done Sent false
6801 2887 Acked Write 7 Done Done Sent false
6802 2888 Acked Read 4 Done Done Sent false
6803 2889 Acked Write 10 Done Done Sent false
6804 2890 Acked Write 10 Done Done Sent false
6805 2891 Acked Write 7 Done Done Sent false
6806 2892 Acked Read 4 Done Done Sent false
6807 2893 Acked Write 8 Done Done Sent false
6808 2894 Acked Read 6 Done Done Sent false
6809 2895 Acked Read 2 Done Done Sent false
6810 2896 Acked Write 7 Done Done Sent false
6811 2897 Acked Read 5 Done Done Sent false
6812 2898 Acked Write 1 Done Done Sent false
6813 2899 Acked Write 10 Done Done Sent false
6814 2900 Acked Read 7 Done Done Sent false
6815 2901 Acked Read 10 Done Done Sent false
6816 2902 Acked Write 8 Done Done Sent false
6817 2903 Acked Write 4 Done Done Sent false
6818 2904 Acked Write 3 Done Done Sent false
6819 2905 Acked Read 7 Done Done Sent false
6820 2906 Acked Flush 0 Done Done Sent false
6821 2907 Acked Write 6 Done Done Sent false
6822 2908 Acked Read 5 Done Done Sent false
6823 2909 Acked Read 3 Done Done Sent false
6824 2910 Acked Read 2 Done Done Sent false
6825 2911 Acked Write 2 Done Done Sent false
6826 2912 Acked Read 3 Done Done Sent false
6827 2913 Acked Write 3 Done Done Sent false
6828 2914 Acked Read 9 Done Done Sent false
6829 2915 Acked Write 7 Done Done Sent false
6830 2916 Acked Write 6 Done Done Sent false
6831 2917 Acked Read 4 Done Done Sent false
6832 2918 Acked Write 10 Done Done Sent false
6833 2919 Acked Write 1 Done Done Sent false
6834 2920 Acked Write 10 Done Done Sent false
6835 2921 Acked Read 4 Done Done Sent false
6836 2922 Acked Write 10 Done Done Sent false
6837 2923 Acked Read 2 Done Done Sent false
6838 2924 Acked Write 9 Done Done Sent false
6839 2925 Acked Write 1 Done Done Sent false
6840 2926 Acked Write 2 Done Done Sent false
6841 2927 Acked Write 5 Done Done Sent false
6842 2928 Acked Write 5 Done Done Sent false
6843 2929 Acked Read 4 Done Done Sent false
6844 2930 Acked Write 7 Done Done Sent false
6845 2931 Acked Write 6 Done Done Sent false
6846 2932 Acked Read 7 Done Done Sent false
6847 2933 Acked Write 7 Done Done Sent false
6848 2934 Acked Read 3 Done Done Sent false
6849 2935 Acked Read 8 Done Done Sent false
6850 2936 Acked Write 6 Done Done Sent false
6851 2937 Acked Read 1 Done Done Sent false
6852 2938 Acked Write 1 Done Done Sent false
6853 2939 Acked Write 1 Done Done Sent false
6854 2940 Acked Read 3 Done Done Sent false
6855 2941 Acked Read 2 Done Done Sent false
6856 2942 Acked Read 3 Done Done Sent false
6857 2943 Acked Write 8 Done Done Sent false
6858 2944 Acked Read 9 Done Done Sent false
6859 2945 Acked Read 2 Done Done Sent false
6860 2946 Acked Write 3 Done Done Sent false
6861 2947 Acked Read 10 Done Done Sent false
6862 2948 Acked Read 3 Done Done Sent false
6863 2949 Acked Read 1 Done Done Sent false
6864 2950 Acked Write 2 Done Done Sent false
6865 2951 Acked Write 4 Done Done Sent false
6866 2952 Acked Flush 0 Done Done Sent false
6867 2953 Acked Write 1 Done Done Sent false
6868 2954 Acked Read 10 Done Done Sent false
6869 2955 Acked Write 10 Done Done Sent false
6870 2956 Acked Write 6 Done Done Sent false
6871 2957 Acked Read 6 Done Done Sent false
6872 2958 Acked Read 7 Done Done Sent false
6873 2959 Acked Write 8 Done Done Sent false
6874 2960 Acked Write 3 Done Done Sent false
6875 2961 Acked Read 9 Done Done Sent false
6876 2962 Acked Read 3 Done Done Sent false
6877 2963 Acked Write 7 Done Done Sent false
6878 2964 Acked Write 7 Done Done Sent false
6879 2965 Acked Read 2 Done Done Sent false
6880 2966 Acked Write 10 Done Done Sent false
6881 2967 Acked Read 10 Done Done Sent false
6882 2968 Acked Flush 0 Done Done Sent false
6883 2969 Acked Read 3 Done Done Sent false
6884 2970 Acked Read 7 Done Done Sent false
6885 2971 Acked Write 7 Done Done Sent false
6886 2972 Acked Read 6 Done Done Sent false
6887 2973 Acked Write 4 Done Done Sent false
6888 2974 Acked Flush 0 Done Done Sent false
6889 2975 Acked Flush 0 Done Done Sent false
6890 2976 Acked Write 4 Done Done Sent false
6891 2977 Acked Read 4 Done Done Sent false
6892 2978 Acked Write 8 Done Done Sent false
6893 2979 Acked Write 6 Done Done Sent false
6894 2980 Acked Read 2 Done Done Sent false
6895 2981 Acked Read 10 Done Done Sent false
6896 2982 Acked Write 4 Done Done Sent false
6897 2983 Acked Write 6 Done Done Sent false
6898 2984 Acked Write 8 Done Done Sent false
6899 2985 Acked Write 10 Done Done Sent false
6900 2986 Acked Flush 0 Done Done Sent false
6901 2987 Acked Read 5 Done Done Sent false
6902 2988 Acked Write 8 Done Done Sent false
6903 2989 Acked Write 8 Done Done Sent false
6904 2990 Acked Read 3 Done Done Sent false
6905 2991 Acked Read 6 Done Done Sent false
6906 2992 Acked Read 3 Done Done Sent false
6907 2993 Acked Write 3 Done Done Sent false
6908 2994 Acked Read 2 Done Done Sent false
6909 2995 Acked Flush 0 Done Done Sent false
6910 2996 Acked Write 10 Done Done Sent false
6911 2997 Acked Read 4 Done Done Sent false
6912 2998 Acked Read 8 Done Done Sent false
6913 2999 Acked Flush 0 Done Done Sent false
6914 3000 Acked Read 8 Done Done Sent false
6915 3001 Acked Write 7 Done Done Sent false
6916 3002 Acked Write 8 Done Done Sent false
6917 3003 Acked Read 3 Done Done Sent false
6918 3004 Acked Read 9 Done Done Sent false
6919 3005 Acked Write 10 Done Done Sent false
6920 3006 Acked Write 3 Done Done Sent false
6921 3007 Acked Flush 0 Done Done Sent false
6922 3008 Acked Write 2 Done Done Sent false
6923 3009 Acked Write 7 Done Done Sent false
6924 3010 Acked Read 3 Done Done Sent false
6925 3011 Acked Read 10 Done Done Sent false
6926 3012 Acked Write 1 Done Done Sent false
6927 3013 Acked Write 3 Done Done Sent false
6928 3014 Acked Read 7 Done Done Sent false
6929 3015 Acked Read 7 Done Done Sent false
6930 3016 Acked Read 4 Done Done Sent false
6931 3017 Acked Write 1 Done Done Sent false
6932 3018 Acked Read 8 Done Done Sent false
6933 3019 Acked Flush 0 Done Done Sent false
6934 3020 Acked Write 9 Done Done Sent false
6935 3021 Acked Flush 0 Done Done Sent false
6936 3022 Acked Read 10 Done Done Sent false
6937 3023 Acked Write 9 Done Done Sent false
6938 3024 Acked Read 10 Done Done Sent false
6939 3025 Acked Write 8 Done Done Sent false
6940 3026 Acked Read 8 Done Done Sent false
6941 3027 Acked Write 9 Done Done Sent false
6942 3028 Acked Flush 0 Done Done Sent false
6943 3029 Acked Flush 0 Done Done Sent false
6944 3030 Acked Read 5 Done Done Sent false
6945 3031 Acked Write 10 Done Done Sent false
6946 3032 Acked Write 10 Done Done Sent false
6947 3033 Acked Write 3 Done Done Sent false
6948 3034 Acked Read 7 Done Done Sent false
6949 3035 Acked Write 1 Done Done Sent false
6950 3036 Acked Read 2 Done Done Sent false
6951 3037 Acked Read 8 Done Done Sent false
6952 3038 Acked Read 9 Done Done Sent false
6953 3039 Acked Read 2 Done Done Sent false
6954 3040 Acked Write 7 Done Done Sent false
6955 3041 Acked Write 3 Done Done Sent false
6956 3042 Acked Write 7 Done Done Sent false
6957 3043 Acked Write 4 Done Done Sent false
6958 3044 Acked Write 9 Done Done Sent false
6959 3045 Acked Write 7 Done Done Sent false
6960 3046 Acked Write 9 Done Done Sent false
6961 3047 Acked Write 6 Done Done Sent false
6962 3048 Acked Write 6 Done Done Sent false
6963 3049 Acked Flush 0 Done Done Sent false
6964 3050 Acked Write 10 Done Done Sent false
6965 3051 Acked Write 9 Done Done Sent false
6966 3052 Acked Flush 0 Done Done Sent false
6967 3053 Acked Read 7 Done Done Sent false
6968 3054 Acked Read 6 Done Done Sent false
6969 3055 Acked Read 1 Done Done Sent false
6970 3056 Acked Write 4 Done Done Sent false
6971 3057 Acked Read 1 Done Done Sent false
6972 3058 Acked Write 10 Done Done Sent false
6973 3059 Acked Read 9 Done Done Sent false
6974 3060 Acked Write 4 Done Done Sent false
6975 3061 Acked Read 10 Done Done Sent false
6976 3062 Acked Read 8 Done Done Sent false
6977 3063 Acked Read 10 Done Done Sent false
6978 3064 Acked Write 4 Done Done Sent false
6979 3065 Acked Write 6 Done Done Sent false
6980 3066 Acked Write 1 Done Done Sent false
6981 3067 Acked Flush 0 Done Done Sent false
6982 3068 Acked Write 7 Done Done Sent false
6983 3069 Acked Write 3 Done Done Sent false
6984 3070 Acked Read 1 Done Done Sent false
6985 3071 Acked Write 10 Done Done Sent false
6986 3072 Acked Read 1 Done Done Sent false
6987 3073 Acked Write 8 Done Done Sent false
6988 3074 Acked Read 10 Done Done Sent false
6989 3075 Acked Write 3 Done Done Sent false
6990 3076 Acked Write 10 Done Done Sent false
6991 3077 Acked Read 1 Done Done Sent false
6992 3078 Acked Read 9 Done Done Sent false
6993 3079 Acked Read 1 Done Done Sent false
6994 3080 Acked Read 4 Done Done Sent false
6995 3081 Acked Write 5 Done Done Sent false
6996 3082 Acked Write 10 Done Done Sent false
6997 3083 Acked Write 4 Done Done Sent false
6998 3084 Acked Read 6 Done Done Sent false
6999 3085 Acked Write 5 Done Done Sent false
7000 3086 Acked Read 3 Done Done Sent false
7001 3087 Acked Read 9 Done Done Sent false
7002 3088 Acked Read 10 Done Done Sent false
7003 3089 Acked Read 6 Done Done Sent false
7004 3090 Acked Write 10 Done Done Sent false
7005 3091 Acked Flush 0 Done Done Sent false
7006 3092 Acked Read 3 Done Done Sent false
7007 3093 Acked Read 8 Done Done Sent false
7008 3094 Acked Read 9 Done Done Sent false
7009 3095 Acked Read 9 Done Done Sent false
7010 3096 Acked Write 3 Done Done Sent false
7011 3097 Acked Read 6 Done Done Sent false
7012 3098 Acked Write 6 Done Done Sent false
7013 3099 Acked Write 1 Done Done Sent false
7014 3100 Acked Write 10 Done Done Sent false
7015 3101 Acked Write 2 Done Done Sent false
7016 3102 Acked Write 3 Done Done Sent false
7017 3103 Acked Flush 0 Done Done Sent false
7018 3104 Acked Write 4 Done Done Sent false
7019 3105 Acked Read 10 Done Done Sent false
7020 3106 Acked Read 7 Done Done Sent false
7021 3107 Acked Write 8 Done Done Sent false
7022 3108 Acked Read 7 Done Done Sent false
7023 3109 Acked Read 5 Done Done Sent false
7024 3110 Acked Read 7 Done Done Sent false
7025 3111 Acked Read 6 Done Done Sent false
7026 3112 Acked Read 9 Done Done Sent false
7027 3113 Acked Flush 0 Done Done Sent false
7028 STATES DS:0 DS:1 DS:2 TOTAL
7029 Sent 0 0 302 302
7030 Done 302 302 0 604
7031 Skipped 0 0 0 0
7032 Error 0 0 0 0
7033 Last Flush: 3113 3113 2811
7034 Downstairs last five completed: 2811 2810 2809 2808 2807
7035 Upstairs last five completed: 3113 3112 3111 3110 3109
7036 CLIENT: Up:0 ds:302 act:2
70372024-12-09T15:09:18.582ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70382024-12-09T15:09:18.582ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63632) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70392024-12-09T15:09:18.583ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70402024-12-09T15:09:18.584ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70412024-12-09T15:09:18.584ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70422024-12-09T15:09:18.584ZINFOcrucible: send last flush ID to this DS: 2811 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70432024-12-09T15:09:18.584ZINFOcrucible: Replied this last flush ID: 2811 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70442024-12-09T15:09:18.584ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70452024-12-09T15:09:18.584ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70462024-12-09T15:09:18.584ZINFOcrucible: [2] client re-new 302 jobs since flush 2811 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70472024-12-09T15:09:18.585ZINFOcrucible: [2] Replayed 302 jobs since flush: 2811 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
7048 ----------------------------------------------------------------
7049 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7050 Downstairs last five completed: 3113 3112 3111 3110 3109
7051 Upstairs last five completed: 3113 3112 3111 3110 3109
7052 CLIENT: Up:0 ds:0 act:3
7053 Replay: All jobs finished
7054 Main task has work to do, go find it
7055 got dsc Stop(0)
7056 stop 0
7057 Dec 09 15:09:20.827 INFO request completed, latency_us: 362, response_code: 204, uri: /stop/cid/0, method[8810] Got stop action so:false kr:false
7058 : POST, req_id: 5a551e17-99d6-4143-8b71-f0f2e88e2de6, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
70592024-12-09T15:09:20.828ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
7060 Dec 09 15:09:20.828 INFO request completed, latency_us: 137, response_code: 200, uri: /state/cid/0, method: GET, req_id: 28f008ac-f81a-4a05-b5f7-42332088ecb5, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
70612024-12-09T15:09:20.828ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70622024-12-09T15:09:20.828ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70632024-12-09T15:09:20.828ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70642024-12-09T15:09:20.828ZINFOcrucible: [0] Transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
70652024-12-09T15:09:20.828ZINFOcrucible: client task is exiting = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
7066 [8810] Exited with: ExitStatus(unix_wait_status(9))
7067 [8810][0] reports Exit
7068 Dec 09 15:09:24.830 INFO request completed, latency_us: 226, response_code: 200, uri: /state/cid/0, method: GET, req_id: fba84303-b7dc-4e8b-ad27-68cdad02913e, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
7069 001/300 Read block 3747 len 4096
7070 002/300 Write block 4865 len 36864 data: 3 3 4 4 5 5 3 4 3
7071 003/300 Write block 925 len 36864 data: 3 4 4 4 3 2 2 3 2
7072 004/300 Write block 3558 len 4096 data: 4
7073 005/300 Write block 775 len 32768 data: 3 3 2 3 3 3 3 3
7074 006/300 Read block 578 len 20480
7075 007/300 Read block 64 len 8192
7076 008/300 Write block 4465 len 8192 data: 3 3
7077 009/300 Write block 3382 len 32768 data: 2 3 3 3 3 2 2 2
7078 010/300 Read block 2715 len 24576
7079 011/300 Read block 2397 len 28672
7080 012/300 Read block 1589 len 16384
7081 013/300 Read block 3086 len 12288
7082 014/300 Read block 3127 len 36864
7083 015/300 Read block 4217 len 4096
7084 016/300 Write block 2525 len 24576 data: 4 4 3 3 3 4
7085 017/300 Flush
7086 018/300 Read block 3963 len 40960
7087 019/300 Read block 4823 len 24576
7088 020/300 Flush
7089 021/300 Read block 4061 len 4096
7090 022/300 Read block 3125 len 8192
7091 023/300 Write block 4088 len 20480 data: 4 5 6 4 4
7092 024/300 Read block 3359 len 4096
7093 025/300 Read block 4680 len 12288
7094 026/300 Flush
7095 027/300 Flush
7096 028/300 Flush
7097 029/300 Read block 2805 len 16384
7098 030/300 Read block 3918 len 28672
7099 031/300 Write block 2172 len 20480 data: 2 2 2 2 2
7100 032/300 Read block 824 len 28672
7101 033/300 Read block 2243 len 8192
7102 034/300 Read block 1574 len 12288
7103 035/300 Write block 987 len 28672 data: 3 3 4 4 4 3 2
7104 036/300 Write block 3741 len 4096 data: 4
7105 037/300 Write block 37 len 12288 data: 2 2 2
7106 038/300 Read block 3921 len 12288
7107 039/300 Write block 3195 len 16384 data: 2 2 3 3
7108 040/300 Write block 2701 len 20480 data: 3 4 3 3 3
7109 041/300 Read block 1030 len 20480
7110 042/300 Read block 4917 len 32768
7111 043/300 Read block 3425 len 16384
7112 044/300 Flush
7113 045/300 Write block 2437 len 12288 data: 3 3 3
7114 046/300 Write block 381 len 12288 data: 2 2 2
7115 047/300 Read block 1054 len 24576
7116 048/300 Write block 4182 len 28672 data: 3 4 4 4 4 4 4
7117 049/300 Write block 1790 len 28672 data: 3 3 3 3 3 2 2
7118 050/300 Read block 4022 len 20480
7119 051/300 Read block 2595 len 8192
7120 052/300 Write block 671 len 32768 data: 3 3 3 3 3 3 3 3
7121 053/300 Flush
7122 054/300 Write block 740 len 12288 data: 2 2 2
7123 055/300 Read block 2076 len 16384
7124 056/300 Read block 541 len 12288
7125 057/300 Write block 3596 len 16384 data: 5 5 4 4
7126 058/300 Read block 247 len 8192
7127 059/300 Write block 2040 len 32768 data: 3 4 4 4 3 4 2 2
7128 060/300 Write block 820 len 28672 data: 3 3 3 3 3 3 3
7129 061/300 Write block 3274 len 36864 data: 3 3 3 3 3 2 2 2 2
7130 062/300 Read block 1759 len 28672
7131 063/300 Read block 1706 len 12288
7132 064/300 Read block 3316 len 40960
7133 065/300 Write block 2529 len 32768 data: 4 5 4 4 3 3 3 3
7134 066/300 Write block 2478 len 12288 data: 4 4 3
7135 067/300 Write block 4179 len 16384 data: 3 3 3 4
7136 068/300 Read block 3671 len 36864
7137 069/300 Flush
7138 070/300 Read block 3203 len 40960
7139 071/300 Flush
7140 072/300 Read block 1725 len 28672
7141 073/300 Read block 525 len 4096
7142 074/300 Read block 4845 len 12288
7143 075/300 Write block 2844 len 24576 data: 3 3 3 2 3 3
7144 076/300 Flush
7145 077/300 Read block 2366 len 12288
7146 078/300 Read block 886 len 4096
7147 079/300 Read block 2396 len 20480
7148 080/300 Read block 2033 len 24576
7149 081/300 Read block 3961 len 40960
7150 082/300 Write block 524 len 36864 data: 3 3 3 3 3 2 2 2 2
7151 083/300 Read block 126 len 24576
7152 084/300 Read block 2638 len 40960
7153 085/300 Write block 4020 len 32768 data: 4 4 3 3 2 2 2 2
7154 086/300 Read block 4285 len 20480
7155 087/300 Read block 114 len 20480
7156 088/300 Read block 4095 len 20480
7157 089/300 Write block 3685 len 36864 data: 4 3 3 3 3 3 3 2 2
7158 090/300 Read block 4218 len 24576
7159 091/300 Read block 4883 len 8192
7160 092/300 Read block 695 len 4096
7161 093/300 Write block 492 len 8192 data: 2 2
7162 094/300 Write block 4945 len 24576 data: 2 3 2 2 2 2
7163 095/300 Read block 1594 len 16384
7164 096/300 Write block 4111 len 36864 data: 2 2 2 2 2 3 3 2 3
7165 097/300 Read block 123 len 36864
7166 098/300 Write block 2454 len 28672 data: 2 2 3 3 3 3 3
7167 099/300 Read block 396 len 28672
7168 100/300 Read block 2121 len 36864
7169 101/300 Write block 98 len 12288 data: 3 2 3
7170 102/300 Read block 1230 len 12288
7171 103/300 Flush
7172 104/300 Write block 1751 len 40960 data: 3 4 5 4 4 5 5 5 4 3
7173 105/300 Read block 1087 len 40960
7174 106/300 Write block 979 len 28672 data: 3 3 2 2 2 2 3
7175 107/300 Read block 435 len 4096
7176 108/300 Write block 2458 len 40960 data: 4 4 4 3 3 3 3 3 3 3
7177 109/300 Write block 1816 len 16384 data: 2 2 2 3
7178 110/300 Flush
7179 111/300 Write block 1207 len 16384 data: 3 2 2 2
7180 112/300 Read block 1098 len 20480
7181 113/300 Read block 853 len 16384
7182 114/300 Read block 2132 len 16384
7183 115/300 Flush
7184 116/300 Write block 2579 len 12288 data: 2 2 2
7185 117/300 Write block 522 len 36864 data: 4 4 4 4 4 4 4 3 3
7186 118/300 Write block 2287 len 40960 data: 3 3 3 3 2 2 2 2 2 3
7187 119/300 Read block 2096 len 8192
7188 120/300 Read block 4766 len 36864
7189 121/300 Read block 3204 len 8192
7190 122/300 Read block 406 len 24576
7191 123/300 Read block 4185 len 8192
7192 124/300 Write block 2394 len 36864 data: 3 3 3 3 3 3 3 4 3
7193 125/300 Write block 1315 len 28672 data: 4 5 4 5 3 3 3
7194 126/300 Write block 316 len 32768 data: 3 3 4 4 5 4 4 4
7195 127/300 Write block 4989 len 24576 data: 6 5 7 6 5 4
7196 128/300 Read block 2110 len 24576
7197 129/300 Read block 224 len 36864
7198 130/300 Write block 3898 len 4096 data: 2
7199 131/300 Write block 3938 len 36864 data: 3 2 2 2 2 2 2 2 2
7200 132/300 Flush
7201 133/300 Read block 2139 len 24576
7202 134/300 Write block 1239 len 40960 data: 2 2 2 3 3 3 4 4 4 4
7203 135/300 Read block 1345 len 36864
7204 136/300 Flush
7205 137/300 Flush
7206 138/300 Read block 282 len 28672
7207 139/300 Read block 2308 len 20480
7208 140/300 Read block 583 len 32768
7209 141/300 Flush
7210 142/300 Read block 210 len 28672
7211 143/300 Read block 417 len 16384
7212 144/300 Read block 1189 len 16384
7213 145/300 Read block 884 len 24576
7214 146/300 Read block 4201 len 20480
7215 147/300 Write block 4934 len 40960 data: 3 3 2 2 2 2 2 2 2 2
7216 148/300 Read block 4255 len 32768
7217 149/300 Write block 2075 len 36864 data: 3 3 3 2 2 4 3 3 2
7218 150/300 Read block 3904 len 40960
7219 151/300 Write block 1570 len 12288 data: 5 4 3
7220 152/300 Read block 3923 len 12288
7221 153/300 Read block 3413 len 40960
7222 154/300 Read block 983 len 8192
7223 155/300 Read block 3263 len 12288
7224 156/300 Read block 828 len 8192
7225 157/300 Read block 4638 len 32768
7226 158/300 Flush
7227 159/300 Write block 4706 len 4096 data: 3
7228 160/300 Read block 4791 len 20480
7229 161/300 Read block 275 len 24576
7230 162/300 Write block 883 len 16384 data: 3 3 3 3
7231 163/300 Write block 4582 len 28672 data: 3 4 2 2 2 2 2
7232 164/300 Write block 1508 len 8192 data: 3 2
7233 165/300 Write block 3721 len 36864 data: 3 2 2 2 2 2 2 2 2
7234 166/300 Write block 3407 len 20480 data: 2 2 2 2 3
7235 167/300 Read block 1976 len 4096
7236 168/300 Read block 4850 len 4096
7237 169/300 Read block 3620 len 24576
7238 170/300 Read block 4823 len 20480
7239 171/300 Read block 2709 len 28672
7240 172/300 Read block 4668 len 24576
7241 173/300 Read block 4055 len 4096
7242 174/300 Read block 3375 len 8192
7243 175/300 Write block 3565 len 16384 data: 3 3 3 3
7244 176/300 Write block 3753 len 8192 data: 4 4
7245 177/300 Read block 3026 len 24576
7246 178/300 Read block 4101 len 12288
7247 179/300 Read block 4498 len 16384
7248 180/300 Write block 4796 len 24576 data: 2 2 2 3 4 4
7249 181/300 Write block 4431 len 20480 data: 3 2 4 4 5
7250 182/300 Read block 4942 len 8192
7251 183/300 Read block 3100 len 36864
7252 184/300 Read block 963 len 4096
7253 185/300 Write block 2635 len 8192 data: 2 2
7254 186/300 Read block 216 len 16384
7255 187/300 Write block 4949 len 12288 data: 3 3 2
7256 188/300 Write block 1722 len 24576 data: 2 2 2 2 2 2
7257 189/300 Read block 1475 len 16384
7258 190/300 Read block 757 len 28672
7259 191/300 Write block 2131 len 8192 data: 6 6
7260 192/300 Read block 1918 len 12288
7261 193/300 Read block 3870 len 32768
7262 194/300 Write block 3256 len 24576 data: 3 3 3 3 2 2
7263 195/300 Flush
7264 196/300 Read block 3252 len 36864
7265 197/300 Read block 1687 len 28672
7266 198/300 Read block 1125 len 12288
7267 199/300 Write block 1803 len 16384 data: 3 4 4 5
7268 200/300 Read block 3159 len 12288
7269 201/300 Write block 2246 len 20480 data: 2 2 2 3 3
7270 202/300 Read block 1357 len 12288
7271 203/300 Write block 3523 len 36864 data: 3 3 3 2 2 2 3 4 4
7272 204/300 Write block 2927 len 8192 data: 2 2
7273 205/300 Read block 107 len 4096
7274 206/300 Read block 2497 len 28672
7275 207/300 Write block 420 len 32768 data: 3 3 3 3 3 4 4 2
7276 208/300 Write block 829 len 40960 data: 2 2 2 2 2 2 2 2 2 3
7277 209/300 Write block 3273 len 20480 data: 3 4 4 4 4
7278 210/300 Read block 3176 len 4096
7279 211/300 Write block 4199 len 12288 data: 4 2 2
7280 212/300 Read block 664 len 40960
7281 213/300 Read block 805 len 40960
7282 214/300 Write block 192 len 36864 data: 4 2 2 2 2 2 2 2 2
7283 215/300 Write block 3321 len 24576 data: 3 4 3 3 3 3
7284 216/300 Write block 2751 len 40960 data: 4 5 5 4 4 4 4 3 2 3
7285 217/300 Flush
7286 218/300 Write block 3584 len 20480 data: 5 4 4 4 3
7287 219/300 Write block 1211 len 32768 data: 3 3 3 3 2 2 3 3
7288 220/300 Write block 1443 len 16384 data: 4 4 4 4
7289 221/300 Read block 3345 len 4096
7290 222/300 Flush
7291 223/300 Read block 1071 len 24576
7292 224/300 Read block 4704 len 8192
7293 225/300 Write block 2135 len 40960 data: 8 6 6 6 4 4 5 4 5 4
7294 226/300 Read block 3219 len 16384
7295 227/300 Write block 3843 len 36864 data: 3 4 4 5 4 4 4 4 3
7296 228/300 Write block 493 len 16384 data: 3 3 4 3
7297 229/300 Read block 4770 len 36864
7298 230/300 Read block 346 len 40960
7299 231/300 Read block 1753 len 28672
7300 232/300 Flush
7301 233/300 Write block 909 len 4096 data: 2
7302 234/300 Read block 3083 len 12288
7303 235/300 Flush
7304 236/300 Write block 1023 len 40960 data: 2 2 2 3 3 2 2 2 2 2
7305 237/300 Flush
7306 238/300 Flush
7307 239/300 Read block 3516 len 4096
7308 240/300 Write block 3156 len 36864 data: 3 4 4 4 5 5 5 4 3
7309 241/300 Write block 2852 len 12288 data: 3 3 3
7310 242/300 Flush
7311 243/300 Read block 4617 len 16384
7312 244/300 Read block 1796 len 40960
7313 245/300 Write block 895 len 4096 data: 2
7314 246/300 Flush
7315 247/300 Read block 4879 len 12288
7316 248/300 Write block 3537 len 4096 data: 3
7317 249/300 Write block 3982 len 4096 data: 3
7318 250/300 Write block 1865 len 8192 data: 2 3
7319 251/300 Read block 4212 len 28672
7320 252/300 Write block 1746 len 16384 data: 3 2 2 2
7321 253/300 Write block 2173 len 16384 data: 3 3 3 3
7322 254/300 Read block 4850 len 12288
7323 255/300 Read block 2713 len 20480
7324 256/300 Read block 2081 len 8192
7325 257/300 Write block 163 len 40960 data: 2 2 2 2 2 2 3 3 3 2
7326 258/300 Read block 752 len 4096
7327 259/300 Write block 4051 len 16384 data: 3 3 2 2
7328 260/300 Flush
7329 261/300 Read block 4413 len 8192
7330 262/300 Write block 801 len 32768 data: 5 5 5 5 3 3 2 3
7331 263/300 Read block 3280 len 20480
7332 264/300 Write block 3022 len 24576 data: 2 2 4 5 4 3
7333 265/300 Flush
7334 266/300 Flush
7335 267/300 Write block 2533 len 36864 data: 4 4 4 4 3 2 2 3 3
7336 268/300 Write block 2640 len 28672 data: 3 4 4 4 4 4 3
7337 269/300 Write block 3161 len 40960 data: 6 6 5 4 3 3 3 3 3 3
7338 270/300 Write block 3912 len 20480 data: 2 2 2 2 2
7339 271/300 Read block 2824 len 20480
7340 272/300 Read block 186 len 12288
7341 273/300 Write block 4204 len 16384 data: 2 2 2 2
7342 274/300 Read block 1572 len 12288
7343 275/300 Write block 3894 len 16384 data: 2 2 2 2
7344 276/300 Read block 3886 len 40960
7345 277/300 Write block 4453 len 16384 data: 2 2 2 2
7346 278/300 Read block 4819 len 4096
7347 279/300 Read block 419 len 16384
7348 280/300 Read block 290 len 40960
7349 281/300 Read block 4016 len 24576
7350 282/300 Read block 892 len 20480
7351 283/300 Write block 1159 len 36864 data: 2 2 2 2 2 2 2 2 2
7352 284/300 Flush
7353 285/300 Write block 2768 len 32768 data: 3 4 4 4 4 4 4 3
7354 286/300 Write block 3536 len 4096 data: 4
7355 287/300 Read block 3051 len 40960
7356 288/300 Read block 1655 len 24576
7357 289/300 Write block 1089 len 8192 data: 4 4
7358 290/300 Read block 3349 len 28672
7359 291/300 Write block 113 len 40960 data: 2 2 2 2 2 2 2 2 2 3
7360 292/300 Read block 3210 len 20480
7361 293/300 Read block 3823 len 24576
7362 294/300 Read block 1177 len 8192
7363 295/300 Write block 3816 len 28672 data: 2 2 2 2 2 3 3
7364 296/300 Read block 1188 len 16384
7365 297/300 Read block 2374 len 36864
7366 298/300 Read block 3207 len 24576
7367 299/300 Read block 3438 len 32768
7368 300/300 Write block 515 len 20480 data: 4 3 4 3 2
7369 301/300 Read block 257 len 12288
7370 Main task has work to do, go find it
7371 got dsc Start(0)
7372 start 0
7373 Dec 09 15:09:25.064 INFO request completed, latency_us: 226, response_code: 204, uri: /start/cid/0, method: POST, req_id: c34f95d8-2241-4999-a4ee-3924e928db3f, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
7374 Replay: started 0, returned:Ok(())
7375 ----------------------------------------------------------------
7376 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7377 States: Offline Active Active
7378 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7379 3114 Acked Read 1 Sent Done Done false
7380 3115 Acked Write 9 Sent Done Done false
7381 3116 Acked Write 9 Sent Done Done false
7382 3117 Acked Write 1 Sent Done Done false
7383 3118 Acked Write 8 Sent Done Done false
7384 3119 Acked Read 5 Sent Done Done false
7385 3120 Acked Read 2 Sent Done Done false
7386 3121 Acked Write 2 Sent Done Done false
7387 3122 Acked Write 8 Sent Done Done false
7388 3123 Acked Read 6 Sent Done Done false
7389 3124 Acked Read 7 Sent Done Done false
7390 3125 Acked Read 4 Sent Done Done false
7391 3126 Acked Read 3 Sent Done Done false
7392 3127 Acked Read 9 Sent Done Done false
7393 3128 Acked Read 1 Sent Done Done false
7394 3129 Acked Write 6 Sent Done Done false
7395 3130 Acked Flush 0 Sent Done Done false
7396 3131 Acked Read 10 Sent Done Done false
7397 3132 Acked Read 6 Sent Done Done false
7398 3133 Acked Flush 0 Sent Done Done false
7399 3134 Acked Read 1 Sent Done Done false
7400 3135 Acked Read 2 Sent Done Done false
7401 3136 Acked Write 5 Sent Done Done false
7402 3137 Acked Read 1 Sent Done Done false
7403 3138 Acked Read 3 Sent Done Done false
7404 3139 Acked Flush 0 Sent Done Done false
7405 3140 Acked Flush 0 Sent Done Done false
7406 3141 Acked Flush 0 Sent Done Done false
7407 3142 Acked Read 4 Sent Done Done false
7408 3143 Acked Read 7 Sent Done Done false
7409 3144 Acked Write 5 Sent Done Done false
7410 3145 Acked Read 7 Sent Done Done false
7411 3146 Acked Read 2 Sent Done Done false
7412 3147 Acked Read 3 Sent Done Done false
7413 3148 Acked Write 7 Sent Done Done false
7414 3149 Acked Write 1 Sent Done Done false
7415 3150 Acked Write 3 Sent Done Done false
7416 3151 Acked Read 3 Sent Done Done false
7417 3152 Acked Write 4 Sent Done Done false
7418 3153 Acked Write 5 Sent Done Done false
7419 3154 Acked Read 5 Sent Done Done false
7420 3155 Acked Read 8 Sent Done Done false
7421 3156 Acked Read 4 Sent Done Done false
7422 3157 Acked Flush 0 Sent Done Done false
7423 3158 Acked Write 3 Sent Done Done false
7424 3159 Acked Write 3 Sent Done Done false
7425 3160 Acked Read 6 Sent Done Done false
7426 3161 Acked Write 7 Sent Done Done false
7427 3162 Acked Write 7 Sent Done Done false
7428 3163 Acked Read 5 Sent Done Done false
7429 3164 Acked Read 2 Sent Done Done false
7430 3165 Acked Write 8 Sent Done Done false
7431 3166 Acked Flush 0 Sent Done Done false
7432 3167 Acked Write 3 Sent Done Done false
7433 3168 Acked Read 4 Sent Done Done false
7434 3169 Acked Read 3 Sent Done Done false
7435 3170 Acked Write 4 Sent Done Done false
7436 3171 Acked Read 2 Sent Done Done false
7437 3172 Acked Write 8 Sent Done Done false
7438 3173 Acked Write 7 Sent Done Done false
7439 3174 Acked Write 9 Sent Done Done false
7440 3175 Acked Read 7 Sent Done Done false
7441 3176 Acked Read 3 Sent Done Done false
7442 3177 Acked Read 10 Sent Done Done false
7443 3178 Acked Write 8 Sent Done Done false
7444 3179 Acked Write 3 Sent Done Done false
7445 3180 Acked Write 4 Sent Done Done false
7446 3181 Acked Read 9 Sent Done Done false
7447 3182 Acked Flush 0 Sent Done Done false
7448 3183 Acked Read 10 Sent Done Done false
7449 3184 Acked Flush 0 Sent Done Done false
7450 3185 Acked Read 7 Sent Done Done false
7451 3186 Acked Read 1 Sent Done Done false
7452 3187 Acked Read 3 Sent Done Done false
7453 3188 Acked Write 6 Sent Done Done false
7454 3189 Acked Flush 0 Sent Done Done false
7455 3190 Acked Read 3 Sent Done Done false
7456 3191 Acked Read 1 Sent Done Done false
7457 3192 Acked Read 5 Sent Done Done false
7458 3193 Acked Read 6 Sent Done Done false
7459 3194 Acked Read 10 Sent Done Done false
7460 3195 Acked Write 9 Sent Done Done false
7461 3196 Acked Read 6 Sent Done Done false
7462 3197 Acked Read 10 Sent Done Done false
7463 3198 Acked Write 8 Sent Done Done false
7464 3199 Acked Read 5 Sent Done Done false
7465 3200 Acked Read 5 Sent Done Done false
7466 3201 Acked Read 5 Sent Done Done false
7467 3202 Acked Write 9 Sent Done Done false
7468 3203 Acked Read 6 Sent Done Done false
7469 3204 Acked Read 2 Sent Done Done false
7470 3205 Acked Read 1 Sent Done Done false
7471 3206 Acked Write 2 Sent Done Done false
7472 3207 Acked Write 6 Sent Done Done false
7473 3208 Acked Read 4 Sent Done Done false
7474 3209 Acked Write 9 Sent Done Done false
7475 3210 Acked Read 9 Sent Done Done false
7476 3211 Acked Write 7 Sent Done Done false
7477 3212 Acked Read 7 Sent Done Done false
7478 3213 Acked Read 9 Sent Done Done false
7479 3214 Acked Write 3 Sent Done Done false
7480 3215 Acked Read 3 Sent Done Done false
7481 3216 Acked Flush 0 Sent Done Done false
7482 3217 Acked Write 10 Sent Done Done false
7483 3218 Acked Read 10 Sent Done Done false
7484 3219 Acked Write 7 Sent Done Done false
7485 3220 Acked Read 1 Sent Done Done false
7486 3221 Acked Write 10 Sent Done Done false
7487 3222 Acked Write 4 Sent Done Done false
7488 3223 Acked Flush 0 Sent Done Done false
7489 3224 Acked Write 4 Sent Done Done false
7490 3225 Acked Read 5 Sent Done Done false
7491 3226 Acked Read 4 Sent Done Done false
7492 3227 Acked Read 4 Sent Done Done false
7493 3228 Acked Flush 0 Sent Done Done false
7494 3229 Acked Write 3 Sent Done Done false
7495 3230 Acked Write 9 Sent Done Done false
7496 3231 Acked Write 10 Sent Done Done false
7497 3232 Acked Read 2 Sent Done Done false
7498 3233 Acked Read 9 Sent Done Done false
7499 3234 Acked Read 2 Sent Done Done false
7500 3235 Acked Read 6 Sent Done Done false
7501 3236 Acked Read 2 Sent Done Done false
7502 3237 Acked Write 9 Sent Done Done false
7503 3238 Acked Write 7 Sent Done Done false
7504 3239 Acked Write 8 Sent Done Done false
7505 3240 Acked Write 6 Sent Done Done false
7506 3241 Acked Read 6 Sent Done Done false
7507 3242 Acked Read 9 Sent Done Done false
7508 3243 Acked Write 1 Sent Done Done false
7509 3244 Acked Write 9 Sent Done Done false
7510 3245 Acked Flush 0 Sent Done Done false
7511 3246 Acked Read 6 Sent Done Done false
7512 3247 Acked Write 10 Sent Done Done false
7513 3248 Acked Read 9 Sent Done Done false
7514 3249 Acked Flush 0 Sent Done Done false
7515 3250 Acked Flush 0 Sent Done Done false
7516 3251 Acked Read 7 Sent Done Done false
7517 3252 Acked Read 5 Sent Done Done false
7518 3253 Acked Read 8 Sent Done Done false
7519 3254 Acked Flush 0 Sent Done Done false
7520 3255 Acked Read 7 Sent Done Done false
7521 3256 Acked Read 4 Sent Done Done false
7522 3257 Acked Read 4 Sent Done Done false
7523 3258 Acked Read 6 Sent Done Done false
7524 3259 Acked Read 5 Sent Done Done false
7525 3260 Acked Write 10 Sent Done Done false
7526 3261 Acked Read 8 Sent Done Done false
7527 3262 Acked Write 9 Sent Done Done false
7528 3263 Acked Read 10 Sent Done Done false
7529 3264 Acked Write 3 Sent Done Done false
7530 3265 Acked Read 3 Sent Done Done false
7531 3266 Acked Read 10 Sent Done Done false
7532 3267 Acked Read 2 Sent Done Done false
7533 3268 Acked Read 3 Sent Done Done false
7534 3269 Acked Read 2 Sent Done Done false
7535 3270 Acked Read 8 Sent Done Done false
7536 3271 Acked Flush 0 Sent Done Done false
7537 3272 Acked Write 1 Sent Done Done false
7538 3273 Acked Read 5 Sent Done Done false
7539 3274 Acked Read 6 Sent Done Done false
7540 3275 Acked Write 4 Sent Done Done false
7541 3276 Acked Write 7 Sent Done Done false
7542 3277 Acked Write 2 Sent Done Done false
7543 3278 Acked Write 9 Sent Done Done false
7544 3279 Acked Write 5 Sent Done Done false
7545 3280 Acked Read 1 Sent Done Done false
7546 3281 Acked Read 1 Sent Done Done false
7547 3282 Acked Read 6 Sent Done Done false
7548 3283 Acked Read 5 Sent Done Done false
7549 3284 Acked Read 7 Sent Done Done false
7550 3285 Acked Read 6 Sent Done Done false
7551 3286 Acked Read 1 Sent Done Done false
7552 3287 Acked Read 2 Sent Done Done false
7553 3288 Acked Write 4 Sent Done Done false
7554 3289 Acked Write 2 Sent Done Done false
7555 3290 Acked Read 6 Sent Done Done false
7556 3291 Acked Read 3 Sent Done Done false
7557 3292 Acked Read 4 Sent Done Done false
7558 3293 Acked Write 6 Sent Done Done false
7559 3294 Acked Write 5 Sent Done Done false
7560 3295 Acked Read 2 Sent Done Done false
7561 3296 Acked Read 9 Sent Done Done false
7562 3297 Acked Read 1 Sent Done Done false
7563 3298 Acked Write 2 Sent Done Done false
7564 3299 Acked Read 4 Sent Done Done false
7565 3300 Acked Write 3 Sent Done Done false
7566 3301 Acked Write 6 Sent Done Done false
7567 3302 Acked Read 4 Sent Done Done false
7568 3303 Acked Read 7 Sent Done Done false
7569 3304 Acked Write 2 Sent Done Done false
7570 3305 Acked Read 3 Sent Done Done false
7571 3306 Acked Read 8 Sent Done Done false
7572 3307 Acked Write 6 Sent Done Done false
7573 3308 Acked Flush 0 Sent Done Done false
7574 3309 Acked Read 9 Sent Done Done false
7575 3310 Acked Read 7 Sent Done Done false
7576 3311 Acked Read 3 Sent Done Done false
7577 3312 Acked Write 4 Sent Done Done false
7578 3313 Acked Read 3 Sent Done Done false
7579 3314 Acked Write 5 Sent Done Done false
7580 3315 Acked Read 3 Sent Done Done false
7581 3316 Acked Write 9 Sent Done Done false
7582 3317 Acked Write 2 Sent Done Done false
7583 3318 Acked Read 1 Sent Done Done false
7584 3319 Acked Read 7 Sent Done Done false
7585 3320 Acked Write 8 Sent Done Done false
7586 3321 Acked Write 10 Sent Done Done false
7587 3322 Acked Write 5 Sent Done Done false
7588 3323 Acked Read 1 Sent Done Done false
7589 3324 Acked Write 3 Sent Done Done false
7590 3325 Acked Read 10 Sent Done Done false
7591 3326 Acked Read 10 Sent Done Done false
7592 3327 Acked Write 9 Sent Done Done false
7593 3328 Acked Write 6 Sent Done Done false
7594 3329 Acked Write 10 Sent Done Done false
7595 3330 Acked Flush 0 Sent Done Done false
7596 3331 Acked Write 5 Sent Done Done false
7597 3332 Acked Write 8 Sent Done Done false
7598 3333 Acked Write 4 Sent Done Done false
7599 3334 Acked Read 1 Sent Done Done false
7600 3335 Acked Flush 0 Sent Done Done false
7601 3336 Acked Read 6 Sent Done Done false
7602 3337 Acked Read 2 Sent Done Done false
7603 3338 Acked Write 10 Sent Done Done false
7604 3339 Acked Read 4 Sent Done Done false
7605 3340 Acked Write 9 Sent Done Done false
7606 3341 Acked Write 4 Sent Done Done false
7607 3342 Acked Read 9 Sent Done Done false
7608 3343 Acked Read 10 Sent Done Done false
7609 3344 Acked Read 7 Sent Done Done false
7610 3345 Acked Flush 0 Sent Done Done false
7611 3346 Acked Write 1 Sent Done Done false
7612 3347 Acked Read 3 Sent Done Done false
7613 3348 Acked Flush 0 Sent Done Done false
7614 3349 Acked Write 10 Sent Done Done false
7615 3350 Acked Flush 0 Sent Done Done false
7616 3351 Acked Flush 0 Sent Done Done false
7617 3352 Acked Read 1 Sent Done Done false
7618 3353 Acked Write 9 Sent Done Done false
7619 3354 Acked Write 3 Sent Done Done false
7620 3355 Acked Flush 0 Sent Done Done false
7621 3356 Acked Read 4 Sent Done Done false
7622 3357 Acked Read 10 Sent Done Done false
7623 3358 Acked Write 1 Sent Done Done false
7624 3359 Acked Flush 0 Sent Done Done false
7625 3360 Acked Read 3 Sent Done Done false
7626 3361 Acked Write 1 Sent Done Done false
7627 3362 Acked Write 1 Sent Done Done false
7628 3363 Acked Write 2 Sent Done Done false
7629 3364 Acked Read 7 Sent Done Done false
7630 3365 Acked Write 4 Sent Done Done false
7631 3366 Acked Write 4 Sent Done Done false
7632 3367 Acked Read 3 Sent Done Done false
7633 3368 Acked Read 5 Sent Done Done false
7634 3369 Acked Read 2 Sent Done Done false
7635 3370 Acked Write 10 Sent Done Done false
7636 3371 Acked Read 1 Sent Done Done false
7637 3372 Acked Write 4 Sent Done Done false
7638 3373 Acked Flush 0 Sent Done Done false
7639 3374 Acked Read 2 Sent Done Done false
7640 3375 Acked Write 8 Sent Done Done false
7641 3376 Acked Read 5 Sent Done Done false
7642 3377 Acked Write 6 Sent Done Done false
7643 3378 Acked Flush 0 Sent Done Done false
7644 3379 Acked Flush 0 Sent Done Done false
7645 3380 Acked Write 9 Sent Done Done false
7646 3381 Acked Write 7 Sent Done Done false
7647 3382 Acked Write 10 Sent Done Done false
7648 3383 Acked Write 5 Sent Done Done false
7649 3384 Acked Read 5 Sent Done Done false
7650 3385 Acked Read 3 Sent Done Done false
7651 3386 Acked Write 4 Sent Done Done false
7652 3387 Acked Read 3 Sent Done Done false
7653 3388 Acked Write 4 Sent Done Done false
7654 3389 Acked Read 10 Sent Done Done false
7655 3390 Acked Write 4 Sent Done Done false
7656 3391 Acked Read 1 Sent Done Done false
7657 3392 Acked Read 4 Sent Done Done false
7658 3393 Acked Read 10 Sent Done Done false
7659 3394 Acked Read 6 Sent Done Done false
7660 3395 Acked Read 5 Sent Done Done false
7661 3396 Acked Write 9 Sent Done Done false
7662 3397 Acked Flush 0 Sent Done Done false
7663 3398 Acked Write 8 Sent Done Done false
7664 3399 Acked Write 1 Sent Done Done false
7665 3400 Acked Read 10 Sent Done Done false
7666 3401 Acked Read 6 Sent Done Done false
7667 3402 Acked Write 2 Sent Done Done false
7668 3403 Acked Read 7 Sent Done Done false
7669 3404 Acked Write 10 Sent Done Done false
7670 3405 Acked Read 5 Sent Done Done false
7671 3406 Acked Read 6 Sent Done Done false
7672 3407 Acked Read 2 Sent Done Done false
7673 3408 Acked Write 7 Sent Done Done false
7674 3409 Acked Read 4 Sent Done Done false
7675 3410 Acked Read 9 Sent Done Done false
7676 3411 Acked Read 6 Sent Done Done false
7677 3412 Acked Read 8 Sent Done Done false
7678 3413 Acked Write 5 Sent Done Done false
7679 3414 Acked Read 3 Sent Done Done false
7680 STATES DS:0 DS:1 DS:2 TOTAL
7681 Sent 301 0 0 301
7682 Done 0 301 301 602
7683 Skipped 0 0 0 0
7684 Error 0 0 0 0
7685 Last Flush: 3113 3397 3397
7686 Downstairs last five completed: 3113 3112 3111 3110 3109
7687 Upstairs last five completed: 3414 3413 3412 3411 3410
7688 CLIENT: Up:0 ds:301 act:2
7689 [8810] got start action
7690 [8810] I am going to restart
7691 Starting downstairs at port 8810
7692 Make output file at "/tmp/dsc/downstairs-8810.txt"
7693 [8810][0] reports Starting
7694 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1209)
7695 [8810][0] reports Running
7696 ----------------------------------------------------------------
7697 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
7698 States: Offline Active Active
7699 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7700 3114 Acked Read 1 Sent Done Done false
7701 3115 Acked Write 9 Sent Done Done false
7702 3116 Acked Write 9 Sent Done Done false
7703 3117 Acked Write 1 Sent Done Done false
7704 3118 Acked Write 8 Sent Done Done false
7705 3119 Acked Read 5 Sent Done Done false
7706 3120 Acked Read 2 Sent Done Done false
7707 3121 Acked Write 2 Sent Done Done false
7708 3122 Acked Write 8 Sent Done Done false
7709 3123 Acked Read 6 Sent Done Done false
7710 3124 Acked Read 7 Sent Done Done false
7711 3125 Acked Read 4 Sent Done Done false
7712 3126 Acked Read 3 Sent Done Done false
7713 3127 Acked Read 9 Sent Done Done false
7714 3128 Acked Read 1 Sent Done Done false
7715 3129 Acked Write 6 Sent Done Done false
7716 3130 Acked Flush 0 Sent Done Done false
7717 3131 Acked Read 10 Sent Done Done false
7718 3132 Acked Read 6 Sent Done Done false
7719 3133 Acked Flush 0 Sent Done Done false
7720 3134 Acked Read 1 Sent Done Done false
7721 3135 Acked Read 2 Sent Done Done false
7722 3136 Acked Write 5 Sent Done Done false
7723 3137 Acked Read 1 Sent Done Done false
7724 3138 Acked Read 3 Sent Done Done false
7725 3139 Acked Flush 0 Sent Done Done false
7726 3140 Acked Flush 0 Sent Done Done false
7727 3141 Acked Flush 0 Sent Done Done false
7728 3142 Acked Read 4 Sent Done Done false
7729 3143 Acked Read 7 Sent Done Done false
7730 3144 Acked Write 5 Sent Done Done false
7731 3145 Acked Read 7 Sent Done Done false
7732 3146 Acked Read 2 Sent Done Done false
7733 3147 Acked Read 3 Sent Done Done false
7734 3148 Acked Write 7 Sent Done Done false
7735 3149 Acked Write 1 Sent Done Done false
7736 3150 Acked Write 3 Sent Done Done false
7737 3151 Acked Read 3 Sent Done Done false
7738 3152 Acked Write 4 Sent Done Done false
7739 3153 Acked Write 5 Sent Done Done false
7740 3154 Acked Read 5 Sent Done Done false
7741 3155 Acked Read 8 Sent Done Done false
7742 3156 Acked Read 4 Sent Done Done false
7743 3157 Acked Flush 0 Sent Done Done false
7744 3158 Acked Write 3 Sent Done Done false
7745 3159 Acked Write 3 Sent Done Done false
7746 3160 Acked Read 6 Sent Done Done false
7747 3161 Acked Write 7 Sent Done Done false
7748 3162 Acked Write 7 Sent Done Done false
7749 3163 Acked Read 5 Sent Done Done false
7750 3164 Acked Read 2 Sent Done Done false
7751 3165 Acked Write 8 Sent Done Done false
7752 3166 Acked Flush 0 Sent Done Done false
7753 3167 Acked Write 3 Sent Done Done false
7754 3168 Acked Read 4 Sent Done Done false
7755 3169 Acked Read 3 Sent Done Done false
7756 3170 Acked Write 4 Sent Done Done false
7757 3171 Acked Read 2 Sent Done Done false
7758 3172 Acked Write 8 Sent Done Done false
7759 3173 Acked Write 7 Sent Done Done false
7760 3174 Acked Write 9 Sent Done Done false
7761 3175 Acked Read 7 Sent Done Done false
7762 3176 Acked Read 3 Sent Done Done false
7763 3177 Acked Read 10 Sent Done Done false
7764 3178 Acked Write 8 Sent Done Done false
7765 3179 Acked Write 3 Sent Done Done false
7766 3180 Acked Write 4 Sent Done Done false
7767 3181 Acked Read 9 Sent Done Done false
7768 3182 Acked Flush 0 Sent Done Done false
7769 3183 Acked Read 10 Sent Done Done false
7770 3184 Acked Flush 0 Sent Done Done false
7771 3185 Acked Read 7 Sent Done Done false
7772 3186 Acked Read 1 Sent Done Done false
7773 3187 Acked Read 3 Sent Done Done false
7774 3188 Acked Write 6 Sent Done Done false
7775 3189 Acked Flush 0 Sent Done Done false
7776 3190 Acked Read 3 Sent Done Done false
7777 3191 Acked Read 1 Sent Done Done false
7778 3192 Acked Read 5 Sent Done Done false
7779 3193 Acked Read 6 Sent Done Done false
7780 3194 Acked Read 10 Sent Done Done false
7781 3195 Acked Write 9 Sent Done Done false
7782 3196 Acked Read 6 Sent Done Done false
7783 3197 Acked Read 10 Sent Done Done false
7784 3198 Acked Write 8 Sent Done Done false
7785 3199 Acked Read 5 Sent Done Done false
7786 3200 Acked Read 5 Sent Done Done false
7787 3201 Acked Read 5 Sent Done Done false
7788 3202 Acked Write 9 Sent Done Done false
7789 3203 Acked Read 6 Sent Done Done false
7790 3204 Acked Read 2 Sent Done Done false
7791 3205 Acked Read 1 Sent Done Done false
7792 3206 Acked Write 2 Sent Done Done false
7793 3207 Acked Write 6 Sent Done Done false
7794 3208 Acked Read 4 Sent Done Done false
7795 3209 Acked Write 9 Sent Done Done false
7796 3210 Acked Read 9 Sent Done Done false
7797 3211 Acked Write 7 Sent Done Done false
7798 3212 Acked Read 7 Sent Done Done false
7799 3213 Acked Read 9 Sent Done Done false
7800 3214 Acked Write 3 Sent Done Done false
7801 3215 Acked Read 3 Sent Done Done false
7802 3216 Acked Flush 0 Sent Done Done false
7803 3217 Acked Write 10 Sent Done Done false
7804 3218 Acked Read 10 Sent Done Done false
7805 3219 Acked Write 7 Sent Done Done false
7806 3220 Acked Read 1 Sent Done Done false
7807 3221 Acked Write 10 Sent Done Done false
7808 3222 Acked Write 4 Sent Done Done false
7809 3223 Acked Flush 0 Sent Done Done false
7810 3224 Acked Write 4 Sent Done Done false
7811 3225 Acked Read 5 Sent Done Done false
7812 3226 Acked Read 4 Sent Done Done false
7813 3227 Acked Read 4 Sent Done Done false
7814 3228 Acked Flush 0 Sent Done Done false
7815 3229 Acked Write 3 Sent Done Done false
7816 3230 Acked Write 9 Sent Done Done false
7817 3231 Acked Write 10 Sent Done Done false
7818 3232 Acked Read 2 Sent Done Done false
7819 3233 Acked Read 9 Sent Done Done false
7820 3234 Acked Read 2 Sent Done Done false
7821 3235 Acked Read 6 Sent Done Done false
7822 3236 Acked Read 2 Sent Done Done false
7823 3237 Acked Write 9 Sent Done Done false
7824 3238 Acked Write 7 Sent Done Done false
7825 3239 Acked Write 8 Sent Done Done false
7826 3240 Acked Write 6 Sent Done Done false
7827 3241 Acked Read 6 Sent Done Done false
7828 3242 Acked Read 9 Sent Done Done false
7829 3243 Acked Write 1 Sent Done Done false
7830 3244 Acked Write 9 Sent Done Done false
7831 3245 Acked Flush 0 Sent Done Done false
7832 3246 Acked Read 6 Sent Done Done false
7833 3247 Acked Write 10 Sent Done Done false
7834 3248 Acked Read 9 Sent Done Done false
7835 3249 Acked Flush 0 Sent Done Done false
7836 3250 Acked Flush 0 Sent Done Done false
7837 3251 Acked Read 7 Sent Done Done false
7838 3252 Acked Read 5 Sent Done Done false
7839 3253 Acked Read 8 Sent Done Done false
7840 3254 Acked Flush 0 Sent Done Done false
7841 3255 Acked Read 7 Sent Done Done false
7842 3256 Acked Read 4 Sent Done Done false
7843 3257 Acked Read 4 Sent Done Done false
7844 3258 Acked Read 6 Sent Done Done false
7845 3259 Acked Read 5 Sent Done Done false
7846 3260 Acked Write 10 Sent Done Done false
7847 3261 Acked Read 8 Sent Done Done false
7848 3262 Acked Write 9 Sent Done Done false
7849 3263 Acked Read 10 Sent Done Done false
7850 3264 Acked Write 3 Sent Done Done false
7851 3265 Acked Read 3 Sent Done Done false
7852 3266 Acked Read 10 Sent Done Done false
7853 3267 Acked Read 2 Sent Done Done false
7854 3268 Acked Read 3 Sent Done Done false
7855 3269 Acked Read 2 Sent Done Done false
7856 3270 Acked Read 8 Sent Done Done false
7857 3271 Acked Flush 0 Sent Done Done false
7858 3272 Acked Write 1 Sent Done Done false
7859 3273 Acked Read 5 Sent Done Done false
7860 3274 Acked Read 6 Sent Done Done false
7861 3275 Acked Write 4 Sent Done Done false
7862 3276 Acked Write 7 Sent Done Done false
7863 3277 Acked Write 2 Sent Done Done false
7864 3278 Acked Write 9 Sent Done Done false
7865 3279 Acked Write 5 Sent Done Done false
7866 3280 Acked Read 1 Sent Done Done false
7867 3281 Acked Read 1 Sent Done Done false
7868 3282 Acked Read 6 Sent Done Done false
7869 3283 Acked Read 5 Sent Done Done false
7870 3284 Acked Read 7 Sent Done Done false
7871 3285 Acked Read 6 Sent Done Done false
7872 3286 Acked Read 1 Sent Done Done false
7873 3287 Acked Read 2 Sent Done Done false
7874 3288 Acked Write 4 Sent Done Done false
7875 3289 Acked Write 2 Sent Done Done false
7876 3290 Acked Read 6 Sent Done Done false
7877 3291 Acked Read 3 Sent Done Done false
7878 3292 Acked Read 4 Sent Done Done false
7879 3293 Acked Write 6 Sent Done Done false
7880 3294 Acked Write 5 Sent Done Done false
7881 3295 Acked Read 2 Sent Done Done false
7882 3296 Acked Read 9 Sent Done Done false
7883 3297 Acked Read 1 Sent Done Done false
7884 3298 Acked Write 2 Sent Done Done false
7885 3299 Acked Read 4 Sent Done Done false
7886 3300 Acked Write 3 Sent Done Done false
7887 3301 Acked Write 6 Sent Done Done false
7888 3302 Acked Read 4 Sent Done Done false
7889 3303 Acked Read 7 Sent Done Done false
7890 3304 Acked Write 2 Sent Done Done false
7891 3305 Acked Read 3 Sent Done Done false
7892 3306 Acked Read 8 Sent Done Done false
7893 3307 Acked Write 6 Sent Done Done false
7894 3308 Acked Flush 0 Sent Done Done false
7895 3309 Acked Read 9 Sent Done Done false
7896 3310 Acked Read 7 Sent Done Done false
7897 3311 Acked Read 3 Sent Done Done false
7898 3312 Acked Write 4 Sent Done Done false
7899 3313 Acked Read 3 Sent Done Done false
7900 3314 Acked Write 5 Sent Done Done false
7901 3315 Acked Read 3 Sent Done Done false
7902 3316 Acked Write 9 Sent Done Done false
7903 3317 Acked Write 2 Sent Done Done false
7904 3318 Acked Read 1 Sent Done Done false
7905 3319 Acked Read 7 Sent Done Done false
7906 3320 Acked Write 8 Sent Done Done false
7907 3321 Acked Write 10 Sent Done Done false
7908 3322 Acked Write 5 Sent Done Done false
7909 3323 Acked Read 1 Sent Done Done false
7910 3324 Acked Write 3 Sent Done Done false
7911 3325 Acked Read 10 Sent Done Done false
7912 3326 Acked Read 10 Sent Done Done false
7913 3327 Acked Write 9 Sent Done Done false
7914 3328 Acked Write 6 Sent Done Done false
7915 3329 Acked Write 10 Sent Done Done false
7916 3330 Acked Flush 0 Sent Done Done false
7917 3331 Acked Write 5 Sent Done Done false
7918 3332 Acked Write 8 Sent Done Done false
7919 3333 Acked Write 4 Sent Done Done false
7920 3334 Acked Read 1 Sent Done Done false
7921 3335 Acked Flush 0 Sent Done Done false
7922 3336 Acked Read 6 Sent Done Done false
7923 3337 Acked Read 2 Sent Done Done false
7924 3338 Acked Write 10 Sent Done Done false
7925 3339 Acked Read 4 Sent Done Done false
7926 3340 Acked Write 9 Sent Done Done false
7927 3341 Acked Write 4 Sent Done Done false
7928 3342 Acked Read 9 Sent Done Done false
7929 3343 Acked Read 10 Sent Done Done false
7930 3344 Acked Read 7 Sent Done Done false
7931 3345 Acked Flush 0 Sent Done Done false
7932 3346 Acked Write 1 Sent Done Done false
7933 3347 Acked Read 3 Sent Done Done false
7934 3348 Acked Flush 0 Sent Done Done false
7935 3349 Acked Write 10 Sent Done Done false
7936 3350 Acked Flush 0 Sent Done Done false
7937 3351 Acked Flush 0 Sent Done Done false
7938 3352 Acked Read 1 Sent Done Done false
7939 3353 Acked Write 9 Sent Done Done false
7940 3354 Acked Write 3 Sent Done Done false
7941 3355 Acked Flush 0 Sent Done Done false
7942 3356 Acked Read 4 Sent Done Done false
7943 3357 Acked Read 10 Sent Done Done false
7944 3358 Acked Write 1 Sent Done Done false
7945 3359 Acked Flush 0 Sent Done Done false
7946 3360 Acked Read 3 Sent Done Done false
7947 3361 Acked Write 1 Sent Done Done false
7948 3362 Acked Write 1 Sent Done Done false
7949 3363 Acked Write 2 Sent Done Done false
7950 3364 Acked Read 7 Sent Done Done false
7951 3365 Acked Write 4 Sent Done Done false
7952 3366 Acked Write 4 Sent Done Done false
7953 3367 Acked Read 3 Sent Done Done false
7954 3368 Acked Read 5 Sent Done Done false
7955 3369 Acked Read 2 Sent Done Done false
7956 3370 Acked Write 10 Sent Done Done false
7957 3371 Acked Read 1 Sent Done Done false
7958 3372 Acked Write 4 Sent Done Done false
7959 3373 Acked Flush 0 Sent Done Done false
7960 3374 Acked Read 2 Sent Done Done false
7961 3375 Acked Write 8 Sent Done Done false
7962 3376 Acked Read 5 Sent Done Done false
7963 3377 Acked Write 6 Sent Done Done false
7964 3378 Acked Flush 0 Sent Done Done false
7965 3379 Acked Flush 0 Sent Done Done false
7966 3380 Acked Write 9 Sent Done Done false
7967 3381 Acked Write 7 Sent Done Done false
7968 3382 Acked Write 10 Sent Done Done false
7969 3383 Acked Write 5 Sent Done Done false
7970 3384 Acked Read 5 Sent Done Done false
7971 3385 Acked Read 3 Sent Done Done false
7972 3386 Acked Write 4 Sent Done Done false
7973 3387 Acked Read 3 Sent Done Done false
7974 3388 Acked Write 4 Sent Done Done false
7975 3389 Acked Read 10 Sent Done Done false
7976 3390 Acked Write 4 Sent Done Done false
7977 3391 Acked Read 1 Sent Done Done false
7978 3392 Acked Read 4 Sent Done Done false
7979 3393 Acked Read 10 Sent Done Done false
7980 3394 Acked Read 6 Sent Done Done false
7981 3395 Acked Read 5 Sent Done Done false
7982 3396 Acked Write 9 Sent Done Done false
7983 3397 Acked Flush 0 Sent Done Done false
7984 3398 Acked Write 8 Sent Done Done false
7985 3399 Acked Write 1 Sent Done Done false
7986 3400 Acked Read 10 Sent Done Done false
7987 3401 Acked Read 6 Sent Done Done false
7988 3402 Acked Write 2 Sent Done Done false
7989 3403 Acked Read 7 Sent Done Done false
7990 3404 Acked Write 10 Sent Done Done false
7991 3405 Acked Read 5 Sent Done Done false
7992 3406 Acked Read 6 Sent Done Done false
7993 3407 Acked Read 2 Sent Done Done false
7994 3408 Acked Write 7 Sent Done Done false
7995 3409 Acked Read 4 Sent Done Done false
7996 3410 Acked Read 9 Sent Done Done false
7997 3411 Acked Read 6 Sent Done Done false
7998 3412 Acked Read 8 Sent Done Done false
7999 3413 Acked Write 5 Sent Done Done false
8000 3414 Acked Read 3 Sent Done Done false
8001 3415 Acked Flush 0 Sent Done Done false
8002 STATES DS:0 DS:1 DS:2 TOTAL
8003 Sent 302 0 0 302
8004 Done 0 302 302 604
8005 Skipped 0 0 0 0
8006 Error 0 0 0 0
8007 Last Flush: 3113 3415 3415
8008 Downstairs last five completed: 3113 3112 3111 3110 3109
8009 Upstairs last five completed: 3415 3414 3413 3412 3411
8010 CLIENT: Up:0 ds:302 act:2
80112024-12-09T15:09:30.830ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
80122024-12-09T15:09:30.830ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37943) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
80132024-12-09T15:09:30.831ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
80142024-12-09T15:09:30.831ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
80152024-12-09T15:09:30.831ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
80162024-12-09T15:09:30.831ZINFOcrucible: send last flush ID to this DS: 3113 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
80172024-12-09T15:09:30.832ZINFOcrucible: Replied this last flush ID: 3113 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
80182024-12-09T15:09:30.832ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
80192024-12-09T15:09:30.832ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
80202024-12-09T15:09:30.832ZINFOcrucible: [0] client re-new 302 jobs since flush 3113 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
80212024-12-09T15:09:30.832ZINFOcrucible: [0] Replayed 302 jobs since flush: 3113 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
8022 ----------------------------------------------------------------
8023 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8024 Downstairs last five completed: 3415 3414 3413 3412 3411
8025 Upstairs last five completed: 3415 3414 3413 3412 3411
8026 CLIENT: Up:0 ds:0 act:3
8027 Replay: All jobs finished
8028 Main task has work to do, go find it
8029 got dsc Stop(2)
8030 stop 2
8031 Dec 09 15:09:33.069 INFO request completed, latency_us: 370, response_code: [8830] Got stop action so:false kr:false
8032 204, uri: /stop/cid/2, method: POST, req_id: 4992d123-b48b-4337-a05b-0ff141e950c6, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
80332024-12-09T15:09:33.070ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
80342024-12-09T15:09:33.070ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
8035 Dec 09 15:09:33.070 INFO request completed, latency_us:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:09:33.07047847Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
8036 150, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7bd38e2b-c9c4-465b-8ce1-7b42560e91c3, remote_addr: {"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:09:33.070496546Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
8037 127.0.0.1:47510, {"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:09:33.070512484Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
8038 local_addr: 127.0.0.1:9998
80392024-12-09T15:09:33.070ZINFOcrucible: client task is exiting = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
8040 [8830] Exited with: ExitStatus(unix_wait_status(9))
8041 [8830][2] reports Exit
8042 Dec 09 15:09:37.072 INFO request completed, latency_us: 226, response_code: 200, uri: /state/cid/2, method: GET, req_id: 212cab69-bb8c-41e4-bb26-4f034877488b, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
8043 001/300 Flush
8044 002/300 Write block 1370 len 32768 data: 6 5 5 4 4 3 3 4
8045 003/300 Write block 2147 len 32768 data: 4 3 2 2 2 2 2 2
8046 004/300 Read block 96 len 4096
8047 005/300 Read block 865 len 12288
8048 006/300 Write block 1391 len 28672 data: 2 2 2 3 3 3 3
8049 007/300 Write block 364 len 4096 data: 4
8050 008/300 Write block 1371 len 20480 data: 6 6 5 5 4
8051 009/300 Write block 655 len 8192 data: 3 2
8052 010/300 Read block 4400 len 20480
8053 011/300 Write block 924 len 12288 data: 3 4 5
8054 012/300 Read block 3221 len 12288
8055 013/300 Read block 1784 len 40960
8056 014/300 Read block 3134 len 12288
8057 015/300 Write block 1967 len 36864 data: 3 3 4 4 4 3 2 2 2
8058 016/300 Write block 683 len 28672 data: 3 3 3 3 3 3 2
8059 017/300 Read block 177 len 20480
8060 018/300 Read block 2412 len 12288
8061 019/300 Read block 240 len 40960
8062 020/300 Read block 1621 len 40960
8063 021/300 Write block 1869 len 20480 data: 4 4 4 2 2
8064 022/300 Read block 3822 len 8192
8065 023/300 Flush
8066 024/300 Read block 2664 len 20480
8067 025/300 Write block 898 len 28672 data: 2 3 3 4 4 3 3
8068 026/300 Flush
8069 027/300 Write block 2167 len 4096 data: 3
8070 028/300 Read block 359 len 4096
8071 029/300 Write block 4912 len 20480 data: 4 4 4 4 3
8072 030/300 Flush
8073 031/300 Write block 2970 len 16384 data: 2 2 2 2
8074 032/300 Read block 2885 len 20480
8075 033/300 Read block 4950 len 36864
8076 034/300 Read block 4764 len 16384
8077 035/300 Flush
8078 036/300 Read block 1468 len 16384
8079 037/300 Write block 3561 len 36864 data: 4 2 2 3 4 4 4 4 2
8080 038/300 Read block 1157 len 12288
8081 039/300 Read block 2804 len 32768
8082 040/300 Write block 1957 len 28672 data: 3 3 3 3 3 3 3
8083 041/300 Read block 499 len 16384
8084 042/300 Flush
8085 043/300 Read block 2321 len 40960
8086 044/300 Read block 4969 len 16384
8087 045/300 Read block 63 len 28672
8088 046/300 Write block 324 len 20480 data: 4 2 3 3 3
8089 047/300 Read block 1206 len 32768
8090 048/300 Read block 2221 len 28672
8091 049/300 Read block 381 len 36864
8092 050/300 Write block 4466 len 24576 data: 4 3 2 2 2 2
8093 051/300 Write block 1941 len 8192 data: 2 2
8094 052/300 Read block 377 len 24576
8095 053/300 Write block 3337 len 32768 data: 4 3 4 4 5 5 5 5
8096 054/300 Read block 3579 len 20480
8097 055/300 Read block 66 len 20480
8098 056/300 Read block 3248 len 8192
8099 057/300 Write block 1743 len 28672 data: 4 4 3 4 3 3 3
8100 058/300 Read block 2363 len 28672
8101 059/300 Write block 1587 len 24576 data: 2 2 2 2 2 2
8102 060/300 Read block 1444 len 8192
8103 061/300 Read block 2941 len 8192
8104 062/300 Read block 1850 len 40960
8105 063/300 Read block 2088 len 32768
8106 064/300 Read block 1819 len 40960
8107 065/300 Write block 769 len 24576 data: 2 2 3 3 3 3
8108 066/300 Write block 3314 len 12288 data: 2 2 2
8109 067/300 Read block 4239 len 16384
8110 068/300 Read block 3329 len 36864
8111 069/300 Flush
8112 070/300 Read block 4823 len 4096
8113 071/300 Read block 2846 len 4096
8114 072/300 Read block 4180 len 28672
8115 073/300 Write block 1886 len 24576 data: 2 2 2 2 2 2
8116 074/300 Write block 3222 len 4096 data: 2
8117 075/300 Read block 1345 len 8192
8118 076/300 Write block 1349 len 32768 data: 2 3 3 3 3 4 4 4
8119 077/300 Flush
8120 078/300 Write block 900 len 28672 data: 4 5 5 4 4 2 2
8121 079/300 Write block 1106 len 16384 data: 3 3 3 3
8122 080/300 Read block 4748 len 40960
8123 081/300 Read block 2768 len 24576
8124 082/300 Flush
8125 083/300 Write block 2036 len 4096 data: 3
8126 084/300 Read block 3573 len 28672
8127 085/300 Read block 1946 len 24576
8128 086/300 Read block 1803 len 4096
8129 087/300 Write block 988 len 12288 data: 4 5 5
8130 088/300 Write block 3002 len 28672 data: 3 3 3 3 3 2 2
8131 089/300 Read block 1240 len 12288
8132 090/300 Read block 1850 len 16384
8133 091/300 Read block 4 len 36864
8134 092/300 Write block 1899 len 4096 data: 2
8135 093/300 Flush
8136 094/300 Read block 4923 len 16384
8137 095/300 Read block 852 len 40960
8138 096/300 Read block 590 len 28672
8139 097/300 Read block 4612 len 12288
8140 098/300 Read block 858 len 28672
8141 099/300 Read block 4280 len 16384
8142 100/300 Write block 2927 len 28672 data: 3 3 2 3 3 3 3
8143 101/300 Read block 767 len 4096
8144 102/300 Read block 4843 len 40960
8145 103/300 Read block 4469 len 28672
8146 104/300 Write block 1006 len 12288 data: 2 2 3
8147 105/300 Read block 1108 len 8192
8148 106/300 Write block 4662 len 20480 data: 3 3 3 3 3
8149 107/300 Read block 132 len 24576
8150 108/300 Write block 4510 len 40960 data: 4 4 5 4 4 4 3 3 3 3
8151 109/300 Write block 75 len 40960 data: 2 2 2 2 2 2 2 3 3 3
8152 110/300 Read block 353 len 36864
8153 111/300 Read block 549 len 40960
8154 112/300 Read block 3667 len 28672
8155 113/300 Write block 3517 len 16384 data: 3 4 4 5
8156 114/300 Read block 72 len 40960
8157 115/300 Write block 1722 len 20480 data: 3 3 3 3 3
8158 116/300 Flush
8159 117/300 Read block 2563 len 4096
8160 118/300 Write block 4981 len 36864 data: 4 4 4 4 5 5 6 6 7
8161 119/300 Flush
8162 120/300 Read block 4083 len 8192
8163 121/300 Write block 1509 len 4096 data: 3
8164 122/300 Write block 3365 len 12288 data: 3 3 3
8165 123/300 Flush
8166 124/300 Write block 608 len 4096 data: 3
8167 125/300 Read block 3182 len 32768
8168 126/300 Read block 1263 len 32768
8169 127/300 Read block 4235 len 20480
8170 128/300 Write block 2771 len 40960 data: 5 5 5 5 4 3 3 3 3 3
8171 129/300 Write block 415 len 24576 data: 3 3 3 2 2 4
8172 130/300 Write block 1886 len 8192 data: 3 3
8173 131/300 Read block 3372 len 32768
8174 132/300 Read block 971 len 36864
8175 133/300 Write block 1079 len 40960 data: 4 4 3 3 3 3 3 3 3 3
8176 134/300 Flush
8177 135/300 Write block 2608 len 40960 data: 2 2 2 3 4 4 4 3 4 4
8178 136/300 Read block 1027 len 8192
8179 137/300 Write block 1865 len 28672 data: 3 4 4 4 5 5 5
8180 138/300 Write block 3474 len 20480 data: 3 3 3 3 3
8181 139/300 Write block 3416 len 32768 data: 4 4 4 4 4 3 4 4
8182 140/300 Write block 876 len 28672 data: 2 2 2 2 2 3 3
8183 141/300 Flush
8184 142/300 Write block 4079 len 20480 data: 2 2 2 3 5
8185 143/300 Read block 3554 len 28672
8186 144/300 Read block 892 len 28672
8187 145/300 Read block 3377 len 4096
8188 146/300 Write block 1793 len 24576 data: 4 4 3 3 2 2
8189 147/300 Read block 3206 len 4096
8190 148/300 Write block 547 len 4096 data: 6
8191 149/300 Write block 663 len 24576 data: 3 3 3 3 3 3
8192 150/300 Write block 4468 len 28672 data: 3 3 3 3 2 2 2
8193 151/300 Write block 3531 len 20480 data: 5 3 3 3 4
8194 152/300 Read block 3489 len 40960
8195 153/300 Read block 2448 len 36864
8196 154/300 Write block 1474 len 12288 data: 3 3 3
8197 155/300 Read block 426 len 4096
8198 156/300 Flush
8199 157/300 Flush
8200 158/300 Write block 3781 len 16384 data: 3 3 3 2
8201 159/300 Write block 4748 len 36864 data: 3 3 3 3 2 2 2 2 2
8202 160/300 Write block 2616 len 32768 data: 5 5 5 5 5 4 4 4
8203 161/300 Read block 3150 len 8192
8204 162/300 Read block 1376 len 36864
8205 163/300 Read block 324 len 32768
8206 164/300 Write block 1597 len 12288 data: 3 3 3
8207 165/300 Read block 4086 len 24576
8208 166/300 Flush
8209 167/300 Read block 4569 len 28672
8210 168/300 Read block 2701 len 32768
8211 169/300 Read block 3276 len 40960
8212 170/300 Write block 1852 len 20480 data: 3 3 3 3 3
8213 171/300 Read block 63 len 28672
8214 172/300 Write block 3445 len 40960 data: 3 2 3 3 3 3 3 3 4 4
8215 173/300 Read block 391 len 32768
8216 174/300 Write block 2416 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8217 175/300 Flush
8218 176/300 Read block 4505 len 36864
8219 177/300 Write block 1693 len 4096 data: 4
8220 178/300 Write block 3162 len 8192 data: 7 6
8221 179/300 Write block 4126 len 24576 data: 3 3 2 2 2 2
8222 180/300 Write block 3844 len 8192 data: 5 5
8223 181/300 Flush
8224 182/300 Write block 354 len 40960 data: 2 2 2 2 2 2 3 3 3 3
8225 183/300 Write block 2195 len 12288 data: 3 3 2
8226 184/300 Flush
8227 185/300 Write block 1963 len 20480 data: 4 3 3 3 4
8228 186/300 Flush
8229 187/300 Write block 3149 len 20480 data: 4 5 5 4 3
8230 188/300 Flush
8231 189/300 Read block 2387 len 4096
8232 190/300 Read block 4653 len 40960
8233 191/300 Write block 4503 len 28672 data: 3 3 3 4 4 5 4
8234 192/300 Read block 4548 len 8192
8235 193/300 Write block 3128 len 24576 data: 2 2 2 2 3 3
8236 194/300 Read block 394 len 36864
8237 195/300 Read block 2732 len 12288
8238 196/300 Read block 1559 len 40960
8239 197/300 Write block 55 len 28672 data: 3 3 3 3 3 3 3
8240 198/300 Read block 2274 len 28672
8241 199/300 Read block 1562 len 40960
8242 200/300 Write block 1779 len 36864 data: 4 4 4 4 3 3 4 6 6
8243 201/300 Read block 259 len 32768
8244 202/300 Flush
8245 203/300 Read block 1715 len 28672
8246 204/300 Read block 2461 len 24576
8247 205/300 Write block 2648 len 36864 data: 2 3 3 3 3 4 4 4 4
8248 206/300 Read block 2907 len 32768
8249 207/300 Write block 10 len 8192 data: 2 2
8250 208/300 Flush
8251 209/300 Write block 3474 len 28672 data: 4 4 4 4 4 3 2
8252 210/300 Read block 3530 len 28672
8253 211/300 Read block 1150 len 32768
8254 212/300 Write block 2601 len 24576 data: 3 3 3 3 3 3
8255 213/300 Read block 2563 len 20480
8256 214/300 Flush
8257 215/300 Read block 1924 len 40960
8258 216/300 Read block 2504 len 32768
8259 217/300 Write block 3886 len 16384 data: 5 4 4 3
8260 218/300 Flush
8261 219/300 Read block 1364 len 12288
8262 220/300 Read block 1412 len 16384
8263 221/300 Read block 3574 len 32768
8264 222/300 Write block 1629 len 28672 data: 2 2 2 2 2 2 2
8265 223/300 Read block 4317 len 16384
8266 224/300 Read block 3185 len 20480
8267 225/300 Write block 3141 len 40960 data: 4 3 3 5 5 5 5 4 5 6
8268 226/300 Read block 4708 len 8192
8269 227/300 Read block 932 len 12288
8270 228/300 Write block 1556 len 12288 data: 3 3 3
8271 229/300 Write block 4362 len 12288 data: 3 3 2
8272 230/300 Write block 155 len 32768 data: 4 5 4 3 3 2 2 2
8273 231/300 Write block 4131 len 12288 data: 3 2 3
8274 232/300 Read block 3047 len 32768
8275 233/300 Read block 468 len 40960
8276 234/300 Write block 4921 len 8192 data: 2 2
8277 235/300 Flush
8278 236/300 Write block 1198 len 32768 data: 4 4 4 4 3 3 3 3
8279 237/300 Read block 2440 len 12288
8280 238/300 Read block 2628 len 12288
8281 239/300 Read block 459 len 20480
8282 240/300 Read block 1449 len 8192
8283 241/300 Read block 4788 len 8192
8284 242/300 Read block 2776 len 40960
8285 243/300 Flush
8286 244/300 Read block 2233 len 4096
8287 245/300 Write block 3409 len 24576 data: 3 3 4 3 3 3
8288 246/300 Read block 3228 len 4096
8289 247/300 Read block 1083 len 20480
8290 248/300 Read block 77 len 16384
8291 249/300 Write block 3651 len 12288 data: 2 2 2
8292 250/300 Write block 4832 len 4096 data: 2
8293 251/300 Write block 1850 len 24576 data: 3 3 4 4 4 4
8294 252/300 Read block 3002 len 12288
8295 253/300 Flush
8296 254/300 Write block 4124 len 12288 data: 4 3 4
8297 255/300 Read block 1754 len 12288
8298 256/300 Flush
8299 257/300 Read block 1477 len 32768
8300 258/300 Write block 1041 len 8192 data: 3 3
8301 259/300 Read block 4645 len 16384
8302 260/300 Read block 1693 len 40960
8303 261/300 Flush
8304 262/300 Read block 3272 len 32768
8305 263/300 Write block 4024 len 40960 data: 3 3 3 3 3 4 3 3 4 3
8306 264/300 Flush
8307 265/300 Read block 1679 len 12288
8308 266/300 Write block 4201 len 16384 data: 3 3 2 3
8309 267/300 Write block 4463 len 16384 data: 3 3 4 5
8310 268/300 Read block 415 len 28672
8311 269/300 Read block 3267 len 36864
8312 270/300 Flush
8313 271/300 Read block 2441 len 4096
8314 272/300 Read block 4399 len 4096
8315 273/300 Read block 4527 len 4096
8316 274/300 Read block 3044 len 16384
8317 275/300 Write block 3383 len 16384 data: 4 4 4 4
8318 276/300 Write block 2609 len 8192 data: 3 3
8319 277/300 Write block 3554 len 36864 data: 4 4 4 4 5 4 3 5 3
8320 278/300 Write block 4068 len 16384 data: 2 2 2 2
8321 279/300 Read block 4533 len 28672
8322 280/300 Read block 924 len 20480
8323 281/300 Read block 1976 len 8192
8324 282/300 Write block 4247 len 4096 data: 3
8325 283/300 Read block 4703 len 8192
8326 284/300 Read block 2329 len 16384
8327 285/300 Read block 3258 len 40960
8328 286/300 Read block 94 len 36864
8329 287/300 Read block 310 len 24576
8330 288/300 Read block 130 len 40960
8331 289/300 Read block 3114 len 8192
8332 290/300 Write block 3154 len 24576 data: 3 4 4 5 5 5
8333 291/300 Read block 4532 len 20480
8334 292/300 Read block 3326 len 4096
8335 293/300 Read block 676 len 40960
8336 294/300 Write block 2532 len 32768 data: 5 5 5 5 5 4 3 3
8337 295/300 Write block 1354 len 28672 data: 5 5 5 4 4 3 3
8338 296/300 Flush
8339 297/300 Read block 4352 len 24576
8340 298/300 Write block 4996 len 12288 data: 3 2 2
8341 299/300 Write block 3088 len 16384 data: 2 3 3 3
8342 300/300 Read block 3470 len 20480
8343 301/300 Write block 3929 len 8192 data: 3 3
8344 Main task has work to do, go find it
8345 got dsc Start(2)
8346 start 2
8347 Dec 09 15:09:37.316 INFO request completed, latency_us: 226, response_code: 204, uri: /start/cid/2, method: POST, req_id: 7c67a117-51c9-4b9d-8620-1dff0fd77166, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
8348 Replay: started 2, returned:Ok(())
8349 ----------------------------------------------------------------
8350 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8351 States: Active Active Offline
8352 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8353 3416 Acked Flush 0 Done Done Sent false
8354 3417 Acked Write 8 Done Done Sent false
8355 3418 Acked Write 8 Done Done Sent false
8356 3419 Acked Read 1 Done Done Sent false
8357 3420 Acked Read 3 Done Done Sent false
8358 3421 Acked Write 7 Done Done Sent false
8359 3422 Acked Write 1 Done Done Sent false
8360 3423 Acked Write 5 Done Done Sent false
8361 3424 Acked Write 2 Done Done Sent false
8362 3425 Acked Read 5 Done Done Sent false
8363 3426 Acked Write 3 Done Done Sent false
8364 3427 Acked Read 3 Done Done Sent false
8365 3428 Acked Read 10 Done Done Sent false
8366 3429 Acked Read 3 Done Done Sent false
8367 3430 Acked Write 9 Done Done Sent false
8368 3431 Acked Write 7 Done Done Sent false
8369 3432 Acked Read 5 Done Done Sent false
8370 3433 Acked Read 3 Done Done Sent false
8371 3434 Acked Read 10 Done Done Sent false
8372 3435 Acked Read 10 Done Done Sent false
8373 3436 Acked Write 5 Done Done Sent false
8374 3437 Acked Read 2 Done Done Sent false
8375 3438 Acked Flush 0 Done Done Sent false
8376 3439 Acked Read 5 Done Done Sent false
8377 3440 Acked Write 7 Done Done Sent false
8378 3441 Acked Flush 0 Done Done Sent false
8379 3442 Acked Write 1 Done Done Sent false
8380 3443 Acked Read 1 Done Done Sent false
8381 3444 Acked Write 5 Done Done Sent false
8382 3445 Acked Flush 0 Done Done Sent false
8383 3446 Acked Write 4 Done Done Sent false
8384 3447 Acked Read 5 Done Done Sent false
8385 3448 Acked Read 9 Done Done Sent false
8386 3449 Acked Read 4 Done Done Sent false
8387 3450 Acked Flush 0 Done Done Sent false
8388 3451 Acked Read 4 Done Done Sent false
8389 3452 Acked Write 9 Done Done Sent false
8390 3453 Acked Read 3 Done Done Sent false
8391 3454 Acked Read 8 Done Done Sent false
8392 3455 Acked Write 7 Done Done Sent false
8393 3456 Acked Read 4 Done Done Sent false
8394 3457 Acked Flush 0 Done Done Sent false
8395 3458 Acked Read 10 Done Done Sent false
8396 3459 Acked Read 4 Done Done Sent false
8397 3460 Acked Read 7 Done Done Sent false
8398 3461 Acked Write 5 Done Done Sent false
8399 3462 Acked Read 8 Done Done Sent false
8400 3463 Acked Read 7 Done Done Sent false
8401 3464 Acked Read 9 Done Done Sent false
8402 3465 Acked Write 6 Done Done Sent false
8403 3466 Acked Write 2 Done Done Sent false
8404 3467 Acked Read 6 Done Done Sent false
8405 3468 Acked Write 8 Done Done Sent false
8406 3469 Acked Read 5 Done Done Sent false
8407 3470 Acked Read 5 Done Done Sent false
8408 3471 Acked Read 2 Done Done Sent false
8409 3472 Acked Write 7 Done Done Sent false
8410 3473 Acked Read 7 Done Done Sent false
8411 3474 Acked Write 6 Done Done Sent false
8412 3475 Acked Read 2 Done Done Sent false
8413 3476 Acked Read 2 Done Done Sent false
8414 3477 Acked Read 10 Done Done Sent false
8415 3478 Acked Read 8 Done Done Sent false
8416 3479 Acked Read 10 Done Done Sent false
8417 3480 Acked Write 6 Done Done Sent false
8418 3481 Acked Write 3 Done Done Sent false
8419 3482 Acked Read 4 Done Done Sent false
8420 3483 Acked Read 9 Done Done Sent false
8421 3484 Acked Flush 0 Done Done Sent false
8422 3485 Acked Read 1 Done Done Sent false
8423 3486 Acked Read 1 Done Done Sent false
8424 3487 Acked Read 7 Done Done Sent false
8425 3488 Acked Write 6 Done Done Sent false
8426 3489 Acked Write 1 Done Done Sent false
8427 3490 Acked Read 2 Done Done Sent false
8428 3491 Acked Write 8 Done Done Sent false
8429 3492 Acked Flush 0 Done Done Sent false
8430 3493 Acked Write 7 Done Done Sent false
8431 3494 Acked Write 4 Done Done Sent false
8432 3495 Acked Read 10 Done Done Sent false
8433 3496 Acked Read 6 Done Done Sent false
8434 3497 Acked Flush 0 Done Done Sent false
8435 3498 Acked Write 1 Done Done Sent false
8436 3499 Acked Read 7 Done Done Sent false
8437 3500 Acked Read 6 Done Done Sent false
8438 3501 Acked Read 1 Done Done Sent false
8439 3502 Acked Write 3 Done Done Sent false
8440 3503 Acked Write 7 Done Done Sent false
8441 3504 Acked Read 3 Done Done Sent false
8442 3505 Acked Read 4 Done Done Sent false
8443 3506 Acked Read 9 Done Done Sent false
8444 3507 Acked Write 1 Done Done Sent false
8445 3508 Acked Flush 0 Done Done Sent false
8446 3509 Acked Read 4 Done Done Sent false
8447 3510 Acked Read 10 Done Done Sent false
8448 3511 Acked Read 7 Done Done Sent false
8449 3512 Acked Read 3 Done Done Sent false
8450 3513 Acked Read 7 Done Done Sent false
8451 3514 Acked Read 4 Done Done Sent false
8452 3515 Acked Write 7 Done Done Sent false
8453 3516 Acked Read 1 Done Done Sent false
8454 3517 Acked Read 10 Done Done Sent false
8455 3518 Acked Read 7 Done Done Sent false
8456 3519 Acked Write 3 Done Done Sent false
8457 3520 Acked Read 2 Done Done Sent false
8458 3521 Acked Write 5 Done Done Sent false
8459 3522 Acked Read 6 Done Done Sent false
8460 3523 Acked Write 10 Done Done Sent false
8461 3524 Acked Write 10 Done Done Sent false
8462 3525 Acked Read 9 Done Done Sent false
8463 3526 Acked Read 10 Done Done Sent false
8464 3527 Acked Read 7 Done Done Sent false
8465 3528 Acked Write 4 Done Done Sent false
8466 3529 Acked Read 10 Done Done Sent false
8467 3530 Acked Write 5 Done Done Sent false
8468 3531 Acked Flush 0 Done Done Sent false
8469 3532 Acked Read 1 Done Done Sent false
8470 3533 Acked Write 9 Done Done Sent false
8471 3534 Acked Flush 0 Done Done Sent false
8472 3535 Acked Read 2 Done Done Sent false
8473 3536 Acked Write 1 Done Done Sent false
8474 3537 Acked Write 3 Done Done Sent false
8475 3538 Acked Flush 0 Done Done Sent false
8476 3539 Acked Write 1 Done Done Sent false
8477 3540 Acked Read 8 Done Done Sent false
8478 3541 Acked Read 8 Done Done Sent false
8479 3542 Acked Read 5 Done Done Sent false
8480 3543 Acked Write 10 Done Done Sent false
8481 3544 Acked Write 6 Done Done Sent false
8482 3545 Acked Write 2 Done Done Sent false
8483 3546 Acked Read 8 Done Done Sent false
8484 3547 Acked Read 9 Done Done Sent false
8485 3548 Acked Write 10 Done Done Sent false
8486 3549 Acked Flush 0 Done Done Sent false
8487 3550 Acked Write 10 Done Done Sent false
8488 3551 Acked Read 2 Done Done Sent false
8489 3552 Acked Write 7 Done Done Sent false
8490 3553 Acked Write 5 Done Done Sent false
8491 3554 Acked Write 8 Done Done Sent false
8492 3555 Acked Write 7 Done Done Sent false
8493 3556 Acked Flush 0 Done Done Sent false
8494 3557 Acked Write 5 Done Done Sent false
8495 3558 Acked Read 7 Done Done Sent false
8496 3559 Acked Read 7 Done Done Sent false
8497 3560 Acked Read 1 Done Done Sent false
8498 3561 Acked Write 6 Done Done Sent false
8499 3562 Acked Read 1 Done Done Sent false
8500 3563 Acked Write 1 Done Done Sent false
8501 3564 Acked Write 6 Done Done Sent false
8502 3565 Acked Write 7 Done Done Sent false
8503 3566 Acked Write 5 Done Done Sent false
8504 3567 Acked Read 10 Done Done Sent false
8505 3568 Acked Read 9 Done Done Sent false
8506 3569 Acked Write 3 Done Done Sent false
8507 3570 Acked Read 1 Done Done Sent false
8508 3571 Acked Flush 0 Done Done Sent false
8509 3572 Acked Flush 0 Done Done Sent false
8510 3573 Acked Write 4 Done Done Sent false
8511 3574 Acked Write 9 Done Done Sent false
8512 3575 Acked Write 8 Done Done Sent false
8513 3576 Acked Read 2 Done Done Sent false
8514 3577 Acked Read 9 Done Done Sent false
8515 3578 Acked Read 8 Done Done Sent false
8516 3579 Acked Write 3 Done Done Sent false
8517 3580 Acked Read 6 Done Done Sent false
8518 3581 Acked Flush 0 Done Done Sent false
8519 3582 Acked Read 7 Done Done Sent false
8520 3583 Acked Read 8 Done Done Sent false
8521 3584 Acked Read 10 Done Done Sent false
8522 3585 Acked Write 5 Done Done Sent false
8523 3586 Acked Read 7 Done Done Sent false
8524 3587 Acked Write 10 Done Done Sent false
8525 3588 Acked Read 8 Done Done Sent false
8526 3589 Acked Write 10 Done Done Sent false
8527 3590 Acked Flush 0 Done Done Sent false
8528 3591 Acked Read 9 Done Done Sent false
8529 3592 Acked Write 1 Done Done Sent false
8530 3593 Acked Write 2 Done Done Sent false
8531 3594 Acked Write 6 Done Done Sent false
8532 3595 Acked Write 2 Done Done Sent false
8533 3596 Acked Flush 0 Done Done Sent false
8534 3597 Acked Write 10 Done Done Sent false
8535 3598 Acked Write 3 Done Done Sent false
8536 3599 Acked Flush 0 Done Done Sent false
8537 3600 Acked Write 5 Done Done Sent false
8538 3601 Acked Flush 0 Done Done Sent false
8539 3602 Acked Write 5 Done Done Sent false
8540 3603 Acked Flush 0 Done Done Sent false
8541 3604 Acked Read 1 Done Done Sent false
8542 3605 Acked Read 10 Done Done Sent false
8543 3606 Acked Write 7 Done Done Sent false
8544 3607 Acked Read 2 Done Done Sent false
8545 3608 Acked Write 6 Done Done Sent false
8546 3609 Acked Read 9 Done Done Sent false
8547 3610 Acked Read 3 Done Done Sent false
8548 3611 Acked Read 10 Done Done Sent false
8549 3612 Acked Write 7 Done Done Sent false
8550 3613 Acked Read 7 Done Done Sent false
8551 3614 Acked Read 10 Done Done Sent false
8552 3615 Acked Write 9 Done Done Sent false
8553 3616 Acked Read 8 Done Done Sent false
8554 3617 Acked Flush 0 Done Done Sent false
8555 3618 Acked Read 7 Done Done Sent false
8556 3619 Acked Read 6 Done Done Sent false
8557 3620 Acked Write 9 Done Done Sent false
8558 3621 Acked Read 8 Done Done Sent false
8559 3622 Acked Write 2 Done Done Sent false
8560 3623 Acked Flush 0 Done Done Sent false
8561 3624 Acked Write 7 Done Done Sent false
8562 3625 Acked Read 7 Done Done Sent false
8563 3626 Acked Read 8 Done Done Sent false
8564 3627 Acked Write 6 Done Done Sent false
8565 3628 Acked Read 5 Done Done Sent false
8566 3629 Acked Flush 0 Done Done Sent false
8567 3630 Acked Read 10 Done Done Sent false
8568 3631 Acked Read 8 Done Done Sent false
8569 3632 Acked Write 4 Done Done Sent false
8570 3633 Acked Flush 0 Done Done Sent false
8571 3634 Acked Read 3 Done Done Sent false
8572 3635 Acked Read 4 Done Done Sent false
8573 3636 Acked Read 8 Done Done Sent false
8574 3637 Acked Write 7 Done Done Sent false
8575 3638 Acked Read 4 Done Done Sent false
8576 3639 Acked Read 5 Done Done Sent false
8577 3640 Acked Write 10 Done Done Sent false
8578 3641 Acked Read 2 Done Done Sent false
8579 3642 Acked Read 3 Done Done Sent false
8580 3643 Acked Write 3 Done Done Sent false
8581 3644 Acked Write 3 Done Done Sent false
8582 3645 Acked Write 8 Done Done Sent false
8583 3646 Acked Write 3 Done Done Sent false
8584 3647 Acked Read 8 Done Done Sent false
8585 3648 Acked Read 10 Done Done Sent false
8586 3649 Acked Write 2 Done Done Sent false
8587 3650 Acked Flush 0 Done Done Sent false
8588 3651 Acked Write 8 Done Done Sent false
8589 3652 Acked Read 3 Done Done Sent false
8590 3653 Acked Read 3 Done Done Sent false
8591 3654 Acked Read 5 Done Done Sent false
8592 3655 Acked Read 2 Done Done Sent false
8593 3656 Acked Read 2 Done Done Sent false
8594 3657 Acked Read 10 Done Done Sent false
8595 3658 Acked Flush 0 Done Done Sent false
8596 3659 Acked Read 1 Done Done Sent false
8597 3660 Acked Write 6 Done Done Sent false
8598 3661 Acked Read 1 Done Done Sent false
8599 3662 Acked Read 5 Done Done Sent false
8600 3663 Acked Read 4 Done Done Sent false
8601 3664 Acked Write 3 Done Done Sent false
8602 3665 Acked Write 1 Done Done Sent false
8603 3666 Acked Write 6 Done Done Sent false
8604 3667 Acked Read 3 Done Done Sent false
8605 3668 Acked Flush 0 Done Done Sent false
8606 3669 Acked Write 3 Done Done Sent false
8607 3670 Acked Read 3 Done Done Sent false
8608 3671 Acked Flush 0 Done Done Sent false
8609 3672 Acked Read 8 Done Done Sent false
8610 3673 Acked Write 2 Done Done Sent false
8611 3674 Acked Read 4 Done Done Sent false
8612 3675 Acked Read 10 Done Done Sent false
8613 3676 Acked Flush 0 Done Done Sent false
8614 3677 Acked Read 8 Done Done Sent false
8615 3678 Acked Write 10 Done Done Sent false
8616 3679 Acked Flush 0 Done Done Sent false
8617 3680 Acked Read 3 Done Done Sent false
8618 3681 Acked Write 4 Done Done Sent false
8619 3682 Acked Write 4 Done Done Sent false
8620 3683 Acked Read 7 Done Done Sent false
8621 3684 Acked Read 9 Done Done Sent false
8622 3685 Acked Flush 0 Done Done Sent false
8623 3686 Acked Read 1 Done Done Sent false
8624 3687 Acked Read 1 Done Done Sent false
8625 3688 Acked Read 1 Done Done Sent false
8626 3689 Acked Read 4 Done Done Sent false
8627 3690 Acked Write 4 Done Done Sent false
8628 3691 Acked Write 2 Done Done Sent false
8629 3692 Acked Write 9 Done Done Sent false
8630 3693 Acked Write 4 Done Done Sent false
8631 3694 Acked Read 7 Done Done Sent false
8632 3695 Acked Read 5 Done Done Sent false
8633 3696 Acked Read 2 Done Done Sent false
8634 3697 Acked Write 1 Done Done Sent false
8635 3698 Acked Read 2 Done Done Sent false
8636 3699 Acked Read 4 Done Done Sent false
8637 3700 Acked Read 10 Done Done Sent false
8638 3701 Acked Read 9 Done Done Sent false
8639 3702 Acked Read 6 Done Done Sent false
8640 3703 Acked Read 10 Done Done Sent false
8641 3704 Acked Read 2 Done Done Sent false
8642 3705 Acked Write 6 Done Done Sent false
8643 3706 Acked Read 5 Done Done Sent false
8644 3707 Acked Read 1 Done Done Sent false
8645 3708 Acked Read 10 Done Done Sent false
8646 3709 Acked Write 8 Done Done Sent false
8647 3710 Acked Write 7 Done Done Sent false
8648 3711 Acked Flush 0 Done Done Sent false
8649 3712 Acked Read 6 Done Done Sent false
8650 3713 Acked Write 3 Done Done Sent false
8651 3714 Acked Write 4 Done Done Sent false
8652 3715 Acked Read 5 Done Done Sent false
8653 3716 Acked Write 2 Done Done Sent false
8654 STATES DS:0 DS:1 DS:2 TOTAL
8655 Sent 0 0 301 301
8656 Done 301 301 0 602
8657 Skipped 0 0 0 0
8658 Error 0 0 0 0
8659 Last Flush: 3711 3711 3415
8660 Downstairs last five completed: 3415 3414 3413 3412 3411
8661 Upstairs last five completed: 3716 3715 3714 3713 3712
8662 CLIENT: Up:0 ds:301 act:2
8663 ----------------------------------------------------------------
8664 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
8665 States: Active Active Offline
8666 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8667 3416 Acked Flush 0 Done Done Sent false
8668 3417 Acked Write 8 Done Done Sent false
8669 3418 Acked Write 8 Done Done Sent false
8670 3419 Acked Read 1 Done Done Sent false
8671 3420 Acked Read 3 Done Done Sent false
8672 3421 Acked Write 7 Done Done Sent false
8673 3422 Acked Write 1 Done Done Sent false
8674 3423 Acked Write 5 Done Done Sent false
8675 3424 Acked Write 2 Done Done Sent false
8676 3425 Acked Read 5 Done Done Sent false
8677 3426 Acked Write 3 Done Done Sent false
8678 3427 Acked Read 3 Done Done Sent false
8679 3428 Acked Read 10 Done Done Sent false
8680 3429 Acked Read 3 Done Done Sent false
8681 3430 Acked Write 9 Done Done Sent false
8682 3431 Acked Write 7 Done Done Sent false
8683 3432 Acked Read 5 Done Done Sent false
8684 3433 Acked Read 3 Done Done Sent false
8685 3434 Acked Read 10 Done Done Sent false
8686 3435 Acked Read 10 Done Done Sent false
8687 3436 Acked Write 5 Done Done Sent false
8688 3437 Acked Read 2 Done Done Sent false
8689 3438 Acked Flush 0 Done Done Sent false
8690 3439 Acked Read 5 Done Done Sent false
8691 3440 Acked Write 7 Done Done Sent false
8692 3441 Acked Flush 0 Done Done Sent false
8693 3442 Acked Write 1 Done Done Sent false
8694 3443 Acked Read 1 Done Done Sent false
8695 3444 Acked Write 5 Done Done Sent false
8696 3445 Acked Flush 0 Done Done Sent false
8697 3446 Acked Write 4 Done Done Sent false
8698 3447 Acked Read 5 Done Done Sent false
8699 3448 Acked Read 9 Done Done Sent false
8700 3449 Acked Read 4 Done Done Sent false
8701 3450 Acked Flush 0 Done Done Sent false
8702 3451 Acked Read 4 Done Done Sent false
8703 3452 Acked Write 9 Done Done Sent false
8704 3453 Acked Read 3 Done Done Sent false
8705 3454 Acked Read 8 Done Done Sent false
8706 3455 Acked Write 7 Done Done Sent false
8707 3456 Acked Read 4 Done Done Sent false
8708 3457 Acked Flush 0 Done Done Sent false
8709 3458 Acked Read 10 Done Done Sent false
8710 3459 Acked Read 4 Done Done Sent false
8711 3460 Acked Read 7 Done Done Sent false
8712 3461 Acked Write 5 Done Done Sent false
8713 3462 Acked Read 8 Done Done Sent false
8714 3463 Acked Read 7 Done Done Sent false
8715 3464 Acked Read 9 Done Done Sent false
8716 3465 Acked Write 6 Done Done Sent false
8717 3466 Acked Write 2 Done Done Sent false
8718 3467 Acked Read 6 Done Done Sent false
8719 3468 Acked Write 8 Done Done Sent false
8720 3469 Acked Read 5 Done Done Sent false
8721 3470 Acked Read 5 Done Done Sent false
8722 3471 Acked Read 2 Done Done Sent false
8723 3472 Acked Write 7 Done Done Sent false
8724 3473 Acked Read 7 Done Done Sent false
8725 3474 Acked Write 6 Done Done Sent false
8726 3475 Acked Read 2 Done Done Sent false
8727 3476 Acked Read 2 Done Done Sent false
8728 3477 Acked Read 10 Done Done Sent false
8729 3478 Acked Read 8 Done Done Sent false
8730 3479 Acked Read 10 Done Done Sent false
8731 3480 Acked Write 6 Done Done Sent false
8732 3481 Acked Write 3 Done Done Sent false
8733 3482 Acked Read 4 Done Done Sent false
8734 3483 Acked Read 9 Done Done Sent false
8735 3484 Acked Flush 0 Done Done Sent false
8736 3485 Acked Read 1 Done Done Sent false
8737 3486 Acked Read 1 Done Done Sent false
8738 3487 Acked Read 7 Done Done Sent false
8739 3488 Acked Write 6 Done Done Sent false
8740 3489 Acked Write 1 Done Done Sent false
8741 3490 Acked Read 2 Done Done Sent false
8742 3491 Acked Write 8 Done Done Sent false
8743 3492 Acked Flush 0 Done Done Sent false
8744 3493 Acked Write 7 Done Done Sent false
8745 3494 Acked Write 4 Done Done Sent false
8746 3495 Acked Read 10 Done Done Sent false
8747 3496 Acked Read 6 Done Done Sent false
8748 3497 Acked Flush 0 Done Done Sent false
8749 3498 Acked Write 1 Done Done Sent false
8750 3499 Acked Read 7 Done Done Sent false
8751 3500 Acked Read 6 Done Done Sent false
8752 3501 Acked Read 1 Done Done Sent false
8753 3502 Acked Write 3 Done Done Sent false
8754 3503 Acked Write 7 Done Done Sent false
8755 3504 Acked Read 3 Done Done Sent false
8756 3505 Acked Read 4 Done Done Sent false
8757 3506 Acked Read 9 Done Done Sent false
8758 3507 Acked Write 1 Done Done Sent false
8759 3508 Acked Flush 0 Done Done Sent false
8760 3509 Acked Read 4 Done Done Sent false
8761 3510 Acked Read 10 Done Done Sent false
8762 3511 Acked Read 7 Done Done Sent false
8763 3512 Acked Read 3 Done Done Sent false
8764 3513 Acked Read 7 Done Done Sent false
8765 3514 Acked Read 4 Done Done Sent false
8766 3515 Acked Write 7 Done Done Sent false
8767 3516 Acked Read 1 Done Done Sent false
8768 3517 Acked Read 10 Done Done Sent false
8769 3518 Acked Read 7 Done Done Sent false
8770 3519 Acked Write 3 Done Done Sent false
8771 3520 Acked Read 2 Done Done Sent false
8772 3521 Acked Write 5 Done Done Sent false
8773 3522 Acked Read 6 Done Done Sent false
8774 3523 Acked Write 10 Done Done Sent false
8775 3524 Acked Write 10 Done Done Sent false
8776 3525 Acked Read 9 Done Done Sent false
8777 3526 Acked Read 10 Done Done Sent false
8778 3527 Acked Read 7 Done Done Sent false
8779 3528 Acked Write 4 Done Done Sent false
8780 3529 Acked Read 10 Done Done Sent false
8781 3530 Acked Write 5 Done Done Sent false
8782 3531 Acked Flush 0 Done Done Sent false
8783 3532 Acked Read 1 Done Done Sent false
8784 3533 Acked Write 9 Done Done Sent false
8785 3534 Acked Flush 0 Done Done Sent false
8786 3535 Acked Read 2 Done Done Sent false
8787 3536 Acked Write 1 Done Done Sent false
8788 3537 Acked Write 3 Done Done Sent false
8789 3538 Acked Flush 0 Done Done Sent false
8790 3539 Acked Write 1 Done Done Sent false
8791 3540 Acked Read 8 Done Done Sent false
8792 3541 Acked Read 8 Done Done Sent false
8793 3542 Acked Read 5 Done Done Sent false
8794 3543 Acked Write 10 Done Done Sent false
8795 3544 Acked Write 6 Done Done Sent false
8796 3545 Acked Write 2 Done Done Sent false
8797 3546 Acked Read 8 Done Done Sent false
8798 3547 Acked Read 9 Done Done Sent false
8799 3548 Acked Write 10 Done Done Sent false
8800 3549 Acked Flush 0 Done Done Sent false
8801 3550 Acked Write 10 Done Done Sent false
8802 3551 Acked Read 2 Done Done Sent false
8803 3552 Acked Write 7 Done Done Sent false
8804 3553 Acked Write 5 Done Done Sent false
8805 3554 Acked Write 8 Done Done Sent false
8806 3555 Acked Write 7 Done Done Sent false
8807 3556 Acked Flush 0 Done Done Sent false
8808 3557 Acked Write 5 Done Done Sent false
8809 3558 Acked Read 7 Done Done Sent false
8810 3559 Acked Read 7 Done Done Sent false
8811 3560 Acked Read 1 Done Done Sent false
8812 3561 Acked Write 6 Done Done Sent false
8813 3562 Acked Read 1 Done Done Sent false
8814 3563 Acked Write 1 Done Done Sent false
8815 3564 Acked Write 6 Done Done Sent false
8816 3565 Acked Write 7 Done Done Sent false
8817 3566 Acked Write 5 Done Done Sent false
8818 3567 Acked Read 10 Done Done Sent false
8819 3568 Acked Read 9 Done Done Sent false
8820 3569 Acked Write 3 Done Done Sent false
8821 3570 Acked Read 1 Done Done Sent false
8822 3571 Acked Flush 0 Done Done Sent false
8823 3572 Acked Flush 0 Done Done Sent false
8824 3573 Acked Write 4 Done Done Sent false
8825 3574 Acked Write 9 Done Done Sent false
8826 3575 Acked Write 8 Done Done Sent false
8827 3576 Acked Read 2 Done Done Sent false
8828 3577 Acked Read 9 Done Done Sent false
8829 3578 Acked Read 8 Done Done Sent false
8830 3579 Acked Write 3 Done Done Sent false
8831 3580 Acked Read 6 Done Done Sent false
8832 3581 Acked Flush 0 Done Done Sent false
8833 3582 Acked Read 7 Done Done Sent false
8834 3583 Acked Read 8 Done Done Sent false
8835 3584 Acked Read 10 Done Done Sent false
8836 3585 Acked Write 5 Done Done Sent false
8837 3586 Acked Read 7 Done Done Sent false
8838 3587 Acked Write 10 Done Done Sent false
8839 3588 Acked Read 8 Done Done Sent false
8840 3589 Acked Write 10 Done Done Sent false
8841 3590 Acked Flush 0 Done Done Sent false
8842 3591 Acked Read 9 Done Done Sent false
8843 3592 Acked Write 1 Done Done Sent false
8844 3593 Acked Write 2 Done Done Sent false
8845 3594 Acked Write 6 Done Done Sent false
8846 3595 Acked Write 2 Done Done Sent false
8847 3596 Acked Flush 0 Done Done Sent false
8848 3597 Acked Write 10 Done Done Sent false
8849 3598 Acked Write 3 Done Done Sent false
8850 3599 Acked Flush 0 Done Done Sent false
8851 3600 Acked Write 5 Done Done Sent false
8852 3601 Acked Flush 0 Done Done Sent false
8853 3602 Acked Write 5 Done Done Sent false
8854 3603 Acked Flush 0 Done Done Sent false
8855 3604 Acked Read 1 Done Done Sent false
8856 3605 Acked Read 10 Done Done Sent false
8857 3606 Acked Write 7 Done Done Sent false
8858 3607 Acked Read 2 Done Done Sent false
8859 3608 Acked Write 6 Done Done Sent false
8860 3609 Acked Read 9 Done Done Sent false
8861 3610 Acked Read 3 Done Done Sent false
8862 3611 Acked Read 10 Done Done Sent false
8863 3612 Acked Write 7 Done Done Sent false
8864 3613 Acked Read 7 Done Done Sent false
8865 3614 Acked Read 10 Done Done Sent false
8866 3615 Acked Write 9 Done Done Sent false
8867 3616 Acked Read 8 Done Done Sent false
8868 3617 Acked Flush 0 Done Done Sent false
8869 3618 Acked Read 7 Done Done Sent false
8870 3619 Acked Read 6 Done Done Sent false
8871 3620 Acked Write 9 Done Done Sent false
8872 3621 Acked Read 8 Done Done Sent false
8873 3622 Acked Write 2 Done Done Sent false
8874 3623 Acked Flush 0 Done Done Sent false
8875 3624 Acked Write 7 Done Done Sent false
8876 3625 Acked Read 7 Done Done Sent false
8877 3626 Acked Read 8 Done Done Sent false
8878 3627 Acked Write 6 Done Done Sent false
8879 3628 Acked Read 5 Done Done Sent false
8880 3629 Acked Flush 0 Done Done Sent false
8881 3630 Acked Read 10 Done Done Sent false
8882 3631 Acked Read 8 Done Done Sent false
8883 3632 Acked Write 4 Done Done Sent false
8884 3633 Acked Flush 0 Done Done Sent false
8885 3634 Acked Read 3 Done Done Sent false
8886 3635 Acked Read 4 Done Done Sent false
8887 3636 Acked Read 8 Done Done Sent false
8888 3637 Acked Write 7 Done Done Sent false
8889 3638 Acked Read 4 Done Done Sent false
8890 3639 Acked Read 5 Done Done Sent false
8891 3640 Acked Write 10 Done Done Sent false
8892 3641 Acked Read 2 Done Done Sent false
8893 3642 Acked Read 3 Done Done Sent false
8894 3643 Acked Write 3 Done Done Sent false
8895 3644 Acked Write 3 Done Done Sent false
8896 3645 Acked Write 8 Done Done Sent false
8897 3646 Acked Write 3 Done Done Sent false
8898 3647 Acked Read 8 Done Done Sent false
8899 3648 Acked Read 10 Done Done Sent false
8900 3649 Acked Write 2 Done Done Sent false
8901 3650 Acked Flush 0 Done Done Sent false
8902 3651 Acked Write 8 Done Done Sent false
8903 3652 Acked Read 3 Done Done Sent false
8904 3653 Acked Read 3 Done Done Sent false
8905 3654 Acked Read 5 Done Done Sent false
8906 3655 Acked Read 2 Done Done Sent false
8907 3656 Acked Read 2 Done Done Sent false
8908 3657 Acked Read 10 Done Done Sent false
8909 3658 Acked Flush 0 Done Done Sent false
8910 3659 Acked Read 1 Done Done Sent false
8911 3660 Acked Write 6 Done Done Sent false
8912 3661 Acked Read 1 Done Done Sent false
8913 3662 Acked Read 5 Done Done Sent false
8914 3663 Acked Read 4 Done Done Sent false
8915 3664 Acked Write 3 Done Done Sent false
8916 3665 Acked Write 1 Done Done Sent false
8917 3666 Acked Write 6 Done Done Sent false
8918 3667 Acked Read 3 Done Done Sent false
8919 3668 Acked Flush 0 Done Done Sent false
8920 3669 Acked Write 3 Done Done Sent false
8921 3670 Acked Read 3 Done Done Sent false
8922 3671 Acked Flush 0 Done Done Sent false
8923 3672 Acked Read 8 Done Done Sent false
8924 3673 Acked Write 2 Done Done Sent false
8925 3674 Acked Read 4 Done Done Sent false
8926 3675 Acked Read 10 Done Done Sent false
8927 3676 Acked Flush 0 Done Done Sent false
8928 3677 Acked Read 8 Done Done Sent false
8929 3678 Acked Write 10 Done Done Sent false
8930 3679 Acked Flush 0 Done Done Sent false
8931 3680 Acked Read 3 Done Done Sent false
8932 3681 Acked Write 4 Done Done Sent false
8933 3682 Acked Write 4 Done Done Sent false
8934 3683 Acked Read 7 Done Done Sent false
8935 3684 Acked Read 9 Done Done Sent false
8936 3685 Acked Flush 0 Done Done Sent false
8937 3686 Acked Read 1 Done Done Sent false
8938 3687 Acked Read 1 Done Done Sent false
8939 3688 Acked Read 1 Done Done Sent false
8940 3689 Acked Read 4 Done Done Sent false
8941 3690 Acked Write 4 Done Done Sent false
8942 3691 Acked Write 2 Done Done Sent false
8943 3692 Acked Write 9 Done Done Sent false
8944 3693 Acked Write 4 Done Done Sent false
8945 3694 Acked Read 7 Done Done Sent false
8946 3695 Acked Read 5 Done Done Sent false
8947 3696 Acked Read 2 Done Done Sent false
8948 3697 Acked Write 1 Done Done Sent false
8949 3698 Acked Read 2 Done Done Sent false
8950 3699 Acked Read 4 Done Done Sent false
8951 3700 Acked Read 10 Done Done Sent false
8952 3701 Acked Read 9 Done Done Sent false
8953 3702 Acked Read 6 Done Done Sent false
8954 3703 Acked Read 10 Done Done Sent false
8955 3704 Acked Read 2 Done Done Sent false
8956 3705 Acked Write 6 Done Done Sent false
8957 3706 Acked Read 5 Done Done Sent false
8958 3707 Acked Read 1 Done Done Sent false
8959 3708 Acked Read 10 Done Done Sent false
8960 3709 Acked Write 8 Done Done Sent false
8961 3710 Acked Write 7 Done Done Sent false
8962 3711 Acked Flush 0 Done Done Sent false
8963 3712 Acked Read 6 Done Done Sent false
8964 3713 Acked Write 3 Done Done Sent false
8965 3714 Acked Write 4 Done Done Sent false
8966 3715 Acked Read 5 Done Done Sent false
8967 3716 Acked Write 2 Done Done Sent false
8968 3717 Acked Flush 0 Done Done Sent false
8969 STATES DS:0 DS:1 DS:2 TOTAL
8970 Sent 0 0 302 302
8971 Done 302 302 0 604
8972 Skipped 0 0 0 0
8973 Error 0 0 0 0
8974 Last Flush: 3717 3717 3415
8975 Downstairs last five completed: 3415 3414 3413 3412 3411
8976 Upstairs last five completed: 3717 3716 3715 3714 3713
8977 CLIENT: Up:0 ds:302 act:2
89782024-12-09T15:09:43.071ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
89792024-12-09T15:09:43.072ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
89802024-12-09T15:09:43.072ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
89812024-12-09T15:09:43.072ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
89822024-12-09T15:09:43.072ZINFOcrucible: client task is exiting = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
8983 ----------------------------------------------------------------
8984 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
8985 States: Active Active Offline
8986 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8987 3416 Acked Flush 0 Done Done Sent false
8988 3417 Acked Write 8 Done Done Sent false
8989 3418 Acked Write 8 Done Done Sent false
8990 3419 Acked Read 1 Done Done Sent false
8991 3420 Acked Read 3 Done Done Sent false
8992 3421 Acked Write 7 Done Done Sent false
8993 3422 Acked Write 1 Done Done Sent false
8994 3423 Acked Write 5 Done Done Sent false
8995 3424 Acked Write 2 Done Done Sent false
8996 3425 Acked Read 5 Done Done Sent false
8997 3426 Acked Write 3 Done Done Sent false
8998 3427 Acked Read 3 Done Done Sent false
8999 3428 Acked Read 10 Done Done Sent false
9000 3429 Acked Read 3 Done Done Sent false
9001 3430 Acked Write 9 Done Done Sent false
9002 3431 Acked Write 7 Done Done Sent false
9003 3432 Acked Read 5 Done Done Sent false
9004 3433 Acked Read 3 Done Done Sent false
9005 3434 Acked Read 10 Done Done Sent false
9006 3435 Acked Read 10 Done Done Sent false
9007 3436 Acked Write 5 Done Done Sent false
9008 3437 Acked Read 2 Done Done Sent false
9009 3438 Acked Flush 0 Done Done Sent false
9010 3439 Acked Read 5 Done Done Sent false
9011 3440 Acked Write 7 Done Done Sent false
9012 3441 Acked Flush 0 Done Done Sent false
9013 3442 Acked Write 1 Done Done Sent false
9014 3443 Acked Read 1 Done Done Sent false
9015 3444 Acked Write 5 Done Done Sent false
9016 3445 Acked Flush 0 Done Done Sent false
9017 3446 Acked Write 4 Done Done Sent false
9018 3447 Acked Read 5 Done Done Sent false
9019 3448 Acked Read 9 Done Done Sent false
9020 3449 Acked Read 4 Done Done Sent false
9021 3450 Acked Flush 0 Done Done Sent false
9022 3451 Acked Read 4 Done Done Sent false
9023 3452 Acked Write 9 Done Done Sent false
9024 3453 Acked Read 3 Done Done Sent false
9025 3454 Acked Read 8 Done Done Sent false
9026 3455 Acked Write 7 Done Done Sent false
9027 3456 Acked Read 4 Done Done Sent false
9028 3457 Acked Flush 0 Done Done Sent false
9029 3458 Acked Read 10 Done Done Sent false
9030 3459 Acked Read 4 Done Done Sent false
9031 3460 Acked Read 7 Done Done Sent false
9032 3461 Acked Write 5 Done Done Sent false
9033 3462 Acked Read 8 Done Done Sent false
9034 3463 Acked Read 7 Done Done Sent false
9035 3464 Acked Read 9 Done Done Sent false
9036 3465 Acked Write 6 Done Done Sent false
9037 3466 Acked Write 2 Done Done Sent false
9038 3467 Acked Read 6 Done Done Sent false
9039 3468 Acked Write 8 Done Done Sent false
9040 3469 Acked Read 5 Done Done Sent false
9041 3470 Acked Read 5 Done Done Sent false
9042 3471 Acked Read 2 Done Done Sent false
9043 3472 Acked Write 7 Done Done Sent false
9044 3473 Acked Read 7 Done Done Sent false
9045 3474 Acked Write 6 Done Done Sent false
9046 3475 Acked Read 2 Done Done Sent false
9047 3476 Acked Read 2 Done Done Sent false
9048 3477 Acked Read 10 Done Done Sent false
9049 3478 Acked Read 8 Done Done Sent false
9050 3479 Acked Read 10 Done Done Sent false
9051 3480 Acked Write 6 Done Done Sent false
9052 3481 Acked Write 3 Done Done Sent false
9053 3482 Acked Read 4 Done Done Sent false
9054 3483 Acked Read 9 Done Done Sent false
9055 3484 Acked Flush 0 Done Done Sent false
9056 3485 Acked Read 1 Done Done Sent false
9057 3486 Acked Read 1 Done Done Sent false
9058 3487 Acked Read 7 Done Done Sent false
9059 3488 Acked Write 6 Done Done Sent false
9060 3489 Acked Write 1 Done Done Sent false
9061 3490 Acked Read 2 Done Done Sent false
9062 3491 Acked Write 8 Done Done Sent false
9063 3492 Acked Flush 0 Done Done Sent false
9064 3493 Acked Write 7 Done Done Sent false
9065 3494 Acked Write 4 Done Done Sent false
9066 3495 Acked Read 10 Done Done Sent false
9067 3496 Acked Read 6 Done Done Sent false
9068 3497 Acked Flush 0 Done Done Sent false
9069 3498 Acked Write 1 Done Done Sent false
9070 3499 Acked Read 7 Done Done Sent false
9071 3500 Acked Read 6 Done Done Sent false
9072 3501 Acked Read 1 Done Done Sent false
9073 3502 Acked Write 3 Done Done Sent false
9074 3503 Acked Write 7 Done Done Sent false
9075 3504 Acked Read 3 Done Done Sent false
9076 3505 Acked Read 4 Done Done Sent false
9077 3506 Acked Read 9 Done Done Sent false
9078 3507 Acked Write 1 Done Done Sent false
9079 3508 Acked Flush 0 Done Done Sent false
9080 3509 Acked Read 4 Done Done Sent false
9081 3510 Acked Read 10 Done Done Sent false
9082 3511 Acked Read 7 Done Done Sent false
9083 3512 Acked Read 3 Done Done Sent false
9084 3513 Acked Read 7 Done Done Sent false
9085 3514 Acked Read 4 Done Done Sent false
9086 3515 Acked Write 7 Done Done Sent false
9087 3516 Acked Read 1 Done Done Sent false
9088 3517 Acked Read 10 Done Done Sent false
9089 3518 Acked Read 7 Done Done Sent false
9090 3519 Acked Write 3 Done Done Sent false
9091 3520 Acked Read 2 Done Done Sent false
9092 3521 Acked Write 5 Done Done Sent false
9093 3522 Acked Read 6 Done Done Sent false
9094 3523 Acked Write 10 Done Done Sent false
9095 3524 Acked Write 10 Done Done Sent false
9096 3525 Acked Read 9 Done Done Sent false
9097 3526 Acked Read 10 Done Done Sent false
9098 3527 Acked Read 7 Done Done Sent false
9099 3528 Acked Write 4 Done Done Sent false
9100 3529 Acked Read 10 Done Done Sent false
9101 3530 Acked Write 5 Done Done Sent false
9102 3531 Acked Flush 0 Done Done Sent false
9103 3532 Acked Read 1 Done Done Sent false
9104 3533 Acked Write 9 Done Done Sent false
9105 3534 Acked Flush 0 Done Done Sent false
9106 3535 Acked Read 2 Done Done Sent false
9107 3536 Acked Write 1 Done Done Sent false
9108 3537 Acked Write 3 Done Done Sent false
9109 3538 Acked Flush 0 Done Done Sent false
9110 3539 Acked Write 1 Done Done Sent false
9111 3540 Acked Read 8 Done Done Sent false
9112 3541 Acked Read 8 Done Done Sent false
9113 3542 Acked Read 5 Done Done Sent false
9114 3543 Acked Write 10 Done Done Sent false
9115 3544 Acked Write 6 Done Done Sent false
9116 3545 Acked Write 2 Done Done Sent false
9117 3546 Acked Read 8 Done Done Sent false
9118 3547 Acked Read 9 Done Done Sent false
9119 3548 Acked Write 10 Done Done Sent false
9120 3549 Acked Flush 0 Done Done Sent false
9121 3550 Acked Write 10 Done Done Sent false
9122 3551 Acked Read 2 Done Done Sent false
9123 3552 Acked Write 7 Done Done Sent false
9124 3553 Acked Write 5 Done Done Sent false
9125 3554 Acked Write 8 Done Done Sent false
9126 3555 Acked Write 7 Done Done Sent false
9127 3556 Acked Flush 0 Done Done Sent false
9128 3557 Acked Write 5 Done Done Sent false
9129 3558 Acked Read 7 Done Done Sent false
9130 3559 Acked Read 7 Done Done Sent false
9131 3560 Acked Read 1 Done Done Sent false
9132 3561 Acked Write 6 Done Done Sent false
9133 3562 Acked Read 1 Done Done Sent false
9134 3563 Acked Write 1 Done Done Sent false
9135 3564 Acked Write 6 Done Done Sent false
9136 3565 Acked Write 7 Done Done Sent false
9137 3566 Acked Write 5 Done Done Sent false
9138 3567 Acked Read 10 Done Done Sent false
9139 3568 Acked Read 9 Done Done Sent false
9140 3569 Acked Write 3 Done Done Sent false
9141 3570 Acked Read 1 Done Done Sent false
9142 3571 Acked Flush 0 Done Done Sent false
9143 3572 Acked Flush 0 Done Done Sent false
9144 3573 Acked Write 4 Done Done Sent false
9145 3574 Acked Write 9 Done Done Sent false
9146 3575 Acked Write 8 Done Done Sent false
9147 3576 Acked Read 2 Done Done Sent false
9148 3577 Acked Read 9 Done Done Sent false
9149 3578 Acked Read 8 Done Done Sent false
9150 3579 Acked Write 3 Done Done Sent false
9151 3580 Acked Read 6 Done Done Sent false
9152 3581 Acked Flush 0 Done Done Sent false
9153 3582 Acked Read 7 Done Done Sent false
9154 3583 Acked Read 8 Done Done Sent false
9155 3584 Acked Read 10 Done Done Sent false
9156 3585 Acked Write 5 Done Done Sent false
9157 3586 Acked Read 7 Done Done Sent false
9158 3587 Acked Write 10 Done Done Sent false
9159 3588 Acked Read 8 Done Done Sent false
9160 3589 Acked Write 10 Done Done Sent false
9161 3590 Acked Flush 0 Done Done Sent false
9162 3591 Acked Read 9 Done Done Sent false
9163 3592 Acked Write 1 Done Done Sent false
9164 3593 Acked Write 2 Done Done Sent false
9165 3594 Acked Write 6 Done Done Sent false
9166 3595 Acked Write 2 Done Done Sent false
9167 3596 Acked Flush 0 Done Done Sent false
9168 3597 Acked Write 10 Done Done Sent false
9169 3598 Acked Write 3 Done Done Sent false
9170 3599 Acked Flush 0 Done Done Sent false
9171 3600 Acked Write 5 Done Done Sent false
9172 3601 Acked Flush 0 Done Done Sent false
9173 3602 Acked Write 5 Done Done Sent false
9174 3603 Acked Flush 0 Done Done Sent false
9175 3604 Acked Read 1 Done Done Sent false
9176 3605 Acked Read 10 Done Done Sent false
9177 3606 Acked Write 7 Done Done Sent false
9178 3607 Acked Read 2 Done Done Sent false
9179 3608 Acked Write 6 Done Done Sent false
9180 3609 Acked Read 9 Done Done Sent false
9181 3610 Acked Read 3 Done Done Sent false
9182 3611 Acked Read 10 Done Done Sent false
9183 3612 Acked Write 7 Done Done Sent false
9184 3613 Acked Read 7 Done Done Sent false
9185 3614 Acked Read 10 Done Done Sent false
9186 3615 Acked Write 9 Done Done Sent false
9187 3616 Acked Read 8 Done Done Sent false
9188 3617 Acked Flush 0 Done Done Sent false
9189 3618 Acked Read 7 Done Done Sent false
9190 3619 Acked Read 6 Done Done Sent false
9191 3620 Acked Write 9 Done Done Sent false
9192 3621 Acked Read 8 Done Done Sent false
9193 3622 Acked Write 2 Done Done Sent false
9194 3623 Acked Flush 0 Done Done Sent false
9195 3624 Acked Write 7 Done Done Sent false
9196 3625 Acked Read 7 Done Done Sent false
9197 3626 Acked Read 8 Done Done Sent false
9198 3627 Acked Write 6 Done Done Sent false
9199 3628 Acked Read 5 Done Done Sent false
9200 3629 Acked Flush 0 Done Done Sent false
9201 3630 Acked Read 10 Done Done Sent false
9202 3631 Acked Read 8 Done Done Sent false
9203 3632 Acked Write 4 Done Done Sent false
9204 3633 Acked Flush 0 Done Done Sent false
9205 3634 Acked Read 3 Done Done Sent false
9206 3635 Acked Read 4 Done Done Sent false
9207 3636 Acked Read 8 Done Done Sent false
9208 3637 Acked Write 7 Done Done Sent false
9209 3638 Acked Read 4 Done Done Sent false
9210 3639 Acked Read 5 Done Done Sent false
9211 3640 Acked Write 10 Done Done Sent false
9212 3641 Acked Read 2 Done Done Sent false
9213 3642 Acked Read 3 Done Done Sent false
9214 3643 Acked Write 3 Done Done Sent false
9215 3644 Acked Write 3 Done Done Sent false
9216 3645 Acked Write 8 Done Done Sent false
9217 3646 Acked Write 3 Done Done Sent false
9218 3647 Acked Read 8 Done Done Sent false
9219 3648 Acked Read 10 Done Done Sent false
9220 3649 Acked Write 2 Done Done Sent false
9221 3650 Acked Flush 0 Done Done Sent false
9222 3651 Acked Write 8 Done Done Sent false
9223 3652 Acked Read 3 Done Done Sent false
9224 3653 Acked Read 3 Done Done Sent false
9225 3654 Acked Read 5 Done Done Sent false
9226 3655 Acked Read 2 Done Done Sent false
9227 3656 Acked Read 2 Done Done Sent false
9228 3657 Acked Read 10 Done Done Sent false
9229 3658 Acked Flush 0 Done Done Sent false
9230 3659 Acked Read 1 Done Done Sent false
9231 3660 Acked Write 6 Done Done Sent false
9232 3661 Acked Read 1 Done Done Sent false
9233 3662 Acked Read 5 Done Done Sent false
9234 3663 Acked Read 4 Done Done Sent false
9235 3664 Acked Write 3 Done Done Sent false
9236 3665 Acked Write 1 Done Done Sent false
9237 3666 Acked Write 6 Done Done Sent false
9238 3667 Acked Read 3 Done Done Sent false
9239 3668 Acked Flush 0 Done Done Sent false
9240 3669 Acked Write 3 Done Done Sent false
9241 3670 Acked Read 3 Done Done Sent false
9242 3671 Acked Flush 0 Done Done Sent false
9243 3672 Acked Read 8 Done Done Sent false
9244 3673 Acked Write 2 Done Done Sent false
9245 3674 Acked Read 4 Done Done Sent false
9246 3675 Acked Read 10 Done Done Sent false
9247 3676 Acked Flush 0 Done Done Sent false
9248 3677 Acked Read 8 Done Done Sent false
9249 3678 Acked Write 10 Done Done Sent false
9250 3679 Acked Flush 0 Done Done Sent false
9251 3680 Acked Read 3 Done Done Sent false
9252 3681 Acked Write 4 Done Done Sent false
9253 3682 Acked Write 4 Done Done Sent false
9254 3683 Acked Read 7 Done Done Sent false
9255 3684 Acked Read 9 Done Done Sent false
9256 3685 Acked Flush 0 Done Done Sent false
9257 3686 Acked Read 1 Done Done Sent false
9258 3687 Acked Read 1 Done Done Sent false
9259 3688 Acked Read 1 Done Done Sent false
9260 3689 Acked Read 4 Done Done Sent false
9261 3690 Acked Write 4 Done Done Sent false
9262 3691 Acked Write 2 Done Done Sent false
9263 3692 Acked Write 9 Done Done Sent false
9264 3693 Acked Write 4 Done Done Sent false
9265 3694 Acked Read 7 Done Done Sent false
9266 3695 Acked Read 5 Done Done Sent false
9267 3696 Acked Read 2 Done Done Sent false
9268 3697 Acked Write 1 Done Done Sent false
9269 3698 Acked Read 2 Done Done Sent false
9270 3699 Acked Read 4 Done Done Sent false
9271 3700 Acked Read 10 Done Done Sent false
9272 3701 Acked Read 9 Done Done Sent false
9273 3702 Acked Read 6 Done Done Sent false
9274 3703 Acked Read 10 Done Done Sent false
9275 3704 Acked Read 2 Done Done Sent false
9276 3705 Acked Write 6 Done Done Sent false
9277 3706 Acked Read 5 Done Done Sent false
9278 3707 Acked Read 1 Done Done Sent false
9279 3708 Acked Read 10 Done Done Sent false
9280 3709 Acked Write 8 Done Done Sent false
9281 3710 Acked Write 7 Done Done Sent false
9282 3711 Acked Flush 0 Done Done Sent false
9283 3712 Acked Read 6 Done Done Sent false
9284 3713 Acked Write 3 Done Done Sent false
9285 3714 Acked Write 4 Done Done Sent false
9286 3715 Acked Read 5 Done Done Sent false
9287 3716 Acked Write 2 Done Done Sent false
9288 3717 Acked Flush 0 Done Done Sent false
9289 STATES DS:0 DS:1 DS:2 TOTAL
9290 Sent 0 0 302 302
9291 Done 302 302 0 604
9292 Skipped 0 0 0 0
9293 Error 0 0 0 0
9294 Last Flush: 3717 3717 3415
9295 Downstairs last five completed: 3415 3414 3413 3412 3411
9296 Upstairs last five completed: 3717 3716 3715 3714 3713
9297 CLIENT: Up:0 ds:302 act:2
9298 [8830] got start action
9299 [8830] I am going to restart
9300 Starting downstairs at port 8830
9301 Make output file at "/tmp/dsc/downstairs-8830.txt"
9302 [8830][2] reports Starting
9303 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1210)
9304 [8830][2] reports Running
9305 ----------------------------------------------------------------
9306 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
9307 States: Active Active Offline
9308 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9309 3416 Acked Flush 0 Done Done Sent false
9310 3417 Acked Write 8 Done Done Sent false
9311 3418 Acked Write 8 Done Done Sent false
9312 3419 Acked Read 1 Done Done Sent false
9313 3420 Acked Read 3 Done Done Sent false
9314 3421 Acked Write 7 Done Done Sent false
9315 3422 Acked Write 1 Done Done Sent false
9316 3423 Acked Write 5 Done Done Sent false
9317 3424 Acked Write 2 Done Done Sent false
9318 3425 Acked Read 5 Done Done Sent false
9319 3426 Acked Write 3 Done Done Sent false
9320 3427 Acked Read 3 Done Done Sent false
9321 3428 Acked Read 10 Done Done Sent false
9322 3429 Acked Read 3 Done Done Sent false
9323 3430 Acked Write 9 Done Done Sent false
9324 3431 Acked Write 7 Done Done Sent false
9325 3432 Acked Read 5 Done Done Sent false
9326 3433 Acked Read 3 Done Done Sent false
9327 3434 Acked Read 10 Done Done Sent false
9328 3435 Acked Read 10 Done Done Sent false
9329 3436 Acked Write 5 Done Done Sent false
9330 3437 Acked Read 2 Done Done Sent false
9331 3438 Acked Flush 0 Done Done Sent false
9332 3439 Acked Read 5 Done Done Sent false
9333 3440 Acked Write 7 Done Done Sent false
9334 3441 Acked Flush 0 Done Done Sent false
9335 3442 Acked Write 1 Done Done Sent false
9336 3443 Acked Read 1 Done Done Sent false
9337 3444 Acked Write 5 Done Done Sent false
9338 3445 Acked Flush 0 Done Done Sent false
9339 3446 Acked Write 4 Done Done Sent false
9340 3447 Acked Read 5 Done Done Sent false
9341 3448 Acked Read 9 Done Done Sent false
9342 3449 Acked Read 4 Done Done Sent false
9343 3450 Acked Flush 0 Done Done Sent false
9344 3451 Acked Read 4 Done Done Sent false
9345 3452 Acked Write 9 Done Done Sent false
9346 3453 Acked Read 3 Done Done Sent false
9347 3454 Acked Read 8 Done Done Sent false
9348 3455 Acked Write 7 Done Done Sent false
9349 3456 Acked Read 4 Done Done Sent false
9350 3457 Acked Flush 0 Done Done Sent false
9351 3458 Acked Read 10 Done Done Sent false
9352 3459 Acked Read 4 Done Done Sent false
9353 3460 Acked Read 7 Done Done Sent false
9354 3461 Acked Write 5 Done Done Sent false
9355 3462 Acked Read 8 Done Done Sent false
9356 3463 Acked Read 7 Done Done Sent false
9357 3464 Acked Read 9 Done Done Sent false
9358 3465 Acked Write 6 Done Done Sent false
9359 3466 Acked Write 2 Done Done Sent false
9360 3467 Acked Read 6 Done Done Sent false
9361 3468 Acked Write 8 Done Done Sent false
9362 3469 Acked Read 5 Done Done Sent false
9363 3470 Acked Read 5 Done Done Sent false
9364 3471 Acked Read 2 Done Done Sent false
9365 3472 Acked Write 7 Done Done Sent false
9366 3473 Acked Read 7 Done Done Sent false
9367 3474 Acked Write 6 Done Done Sent false
9368 3475 Acked Read 2 Done Done Sent false
9369 3476 Acked Read 2 Done Done Sent false
9370 3477 Acked Read 10 Done Done Sent false
9371 3478 Acked Read 8 Done Done Sent false
9372 3479 Acked Read 10 Done Done Sent false
9373 3480 Acked Write 6 Done Done Sent false
9374 3481 Acked Write 3 Done Done Sent false
9375 3482 Acked Read 4 Done Done Sent false
9376 3483 Acked Read 9 Done Done Sent false
9377 3484 Acked Flush 0 Done Done Sent false
9378 3485 Acked Read 1 Done Done Sent false
9379 3486 Acked Read 1 Done Done Sent false
9380 3487 Acked Read 7 Done Done Sent false
9381 3488 Acked Write 6 Done Done Sent false
9382 3489 Acked Write 1 Done Done Sent false
9383 3490 Acked Read 2 Done Done Sent false
9384 3491 Acked Write 8 Done Done Sent false
9385 3492 Acked Flush 0 Done Done Sent false
9386 3493 Acked Write 7 Done Done Sent false
9387 3494 Acked Write 4 Done Done Sent false
9388 3495 Acked Read 10 Done Done Sent false
9389 3496 Acked Read 6 Done Done Sent false
9390 3497 Acked Flush 0 Done Done Sent false
9391 3498 Acked Write 1 Done Done Sent false
9392 3499 Acked Read 7 Done Done Sent false
9393 3500 Acked Read 6 Done Done Sent false
9394 3501 Acked Read 1 Done Done Sent false
9395 3502 Acked Write 3 Done Done Sent false
9396 3503 Acked Write 7 Done Done Sent false
9397 3504 Acked Read 3 Done Done Sent false
9398 3505 Acked Read 4 Done Done Sent false
9399 3506 Acked Read 9 Done Done Sent false
9400 3507 Acked Write 1 Done Done Sent false
9401 3508 Acked Flush 0 Done Done Sent false
9402 3509 Acked Read 4 Done Done Sent false
9403 3510 Acked Read 10 Done Done Sent false
9404 3511 Acked Read 7 Done Done Sent false
9405 3512 Acked Read 3 Done Done Sent false
9406 3513 Acked Read 7 Done Done Sent false
9407 3514 Acked Read 4 Done Done Sent false
9408 3515 Acked Write 7 Done Done Sent false
9409 3516 Acked Read 1 Done Done Sent false
9410 3517 Acked Read 10 Done Done Sent false
9411 3518 Acked Read 7 Done Done Sent false
9412 3519 Acked Write 3 Done Done Sent false
9413 3520 Acked Read 2 Done Done Sent false
9414 3521 Acked Write 5 Done Done Sent false
9415 3522 Acked Read 6 Done Done Sent false
9416 3523 Acked Write 10 Done Done Sent false
9417 3524 Acked Write 10 Done Done Sent false
9418 3525 Acked Read 9 Done Done Sent false
9419 3526 Acked Read 10 Done Done Sent false
9420 3527 Acked Read 7 Done Done Sent false
9421 3528 Acked Write 4 Done Done Sent false
9422 3529 Acked Read 10 Done Done Sent false
9423 3530 Acked Write 5 Done Done Sent false
9424 3531 Acked Flush 0 Done Done Sent false
9425 3532 Acked Read 1 Done Done Sent false
9426 3533 Acked Write 9 Done Done Sent false
9427 3534 Acked Flush 0 Done Done Sent false
9428 3535 Acked Read 2 Done Done Sent false
9429 3536 Acked Write 1 Done Done Sent false
9430 3537 Acked Write 3 Done Done Sent false
9431 3538 Acked Flush 0 Done Done Sent false
9432 3539 Acked Write 1 Done Done Sent false
9433 3540 Acked Read 8 Done Done Sent false
9434 3541 Acked Read 8 Done Done Sent false
9435 3542 Acked Read 5 Done Done Sent false
9436 3543 Acked Write 10 Done Done Sent false
9437 3544 Acked Write 6 Done Done Sent false
9438 3545 Acked Write 2 Done Done Sent false
9439 3546 Acked Read 8 Done Done Sent false
9440 3547 Acked Read 9 Done Done Sent false
9441 3548 Acked Write 10 Done Done Sent false
9442 3549 Acked Flush 0 Done Done Sent false
9443 3550 Acked Write 10 Done Done Sent false
9444 3551 Acked Read 2 Done Done Sent false
9445 3552 Acked Write 7 Done Done Sent false
9446 3553 Acked Write 5 Done Done Sent false
9447 3554 Acked Write 8 Done Done Sent false
9448 3555 Acked Write 7 Done Done Sent false
9449 3556 Acked Flush 0 Done Done Sent false
9450 3557 Acked Write 5 Done Done Sent false
9451 3558 Acked Read 7 Done Done Sent false
9452 3559 Acked Read 7 Done Done Sent false
9453 3560 Acked Read 1 Done Done Sent false
9454 3561 Acked Write 6 Done Done Sent false
9455 3562 Acked Read 1 Done Done Sent false
9456 3563 Acked Write 1 Done Done Sent false
9457 3564 Acked Write 6 Done Done Sent false
9458 3565 Acked Write 7 Done Done Sent false
9459 3566 Acked Write 5 Done Done Sent false
9460 3567 Acked Read 10 Done Done Sent false
9461 3568 Acked Read 9 Done Done Sent false
9462 3569 Acked Write 3 Done Done Sent false
9463 3570 Acked Read 1 Done Done Sent false
9464 3571 Acked Flush 0 Done Done Sent false
9465 3572 Acked Flush 0 Done Done Sent false
9466 3573 Acked Write 4 Done Done Sent false
9467 3574 Acked Write 9 Done Done Sent false
9468 3575 Acked Write 8 Done Done Sent false
9469 3576 Acked Read 2 Done Done Sent false
9470 3577 Acked Read 9 Done Done Sent false
9471 3578 Acked Read 8 Done Done Sent false
9472 3579 Acked Write 3 Done Done Sent false
9473 3580 Acked Read 6 Done Done Sent false
9474 3581 Acked Flush 0 Done Done Sent false
9475 3582 Acked Read 7 Done Done Sent false
9476 3583 Acked Read 8 Done Done Sent false
9477 3584 Acked Read 10 Done Done Sent false
9478 3585 Acked Write 5 Done Done Sent false
9479 3586 Acked Read 7 Done Done Sent false
9480 3587 Acked Write 10 Done Done Sent false
9481 3588 Acked Read 8 Done Done Sent false
9482 3589 Acked Write 10 Done Done Sent false
9483 3590 Acked Flush 0 Done Done Sent false
9484 3591 Acked Read 9 Done Done Sent false
9485 3592 Acked Write 1 Done Done Sent false
9486 3593 Acked Write 2 Done Done Sent false
9487 3594 Acked Write 6 Done Done Sent false
9488 3595 Acked Write 2 Done Done Sent false
9489 3596 Acked Flush 0 Done Done Sent false
9490 3597 Acked Write 10 Done Done Sent false
9491 3598 Acked Write 3 Done Done Sent false
9492 3599 Acked Flush 0 Done Done Sent false
9493 3600 Acked Write 5 Done Done Sent false
9494 3601 Acked Flush 0 Done Done Sent false
9495 3602 Acked Write 5 Done Done Sent false
9496 3603 Acked Flush 0 Done Done Sent false
9497 3604 Acked Read 1 Done Done Sent false
9498 3605 Acked Read 10 Done Done Sent false
9499 3606 Acked Write 7 Done Done Sent false
9500 3607 Acked Read 2 Done Done Sent false
9501 3608 Acked Write 6 Done Done Sent false
9502 3609 Acked Read 9 Done Done Sent false
9503 3610 Acked Read 3 Done Done Sent false
9504 3611 Acked Read 10 Done Done Sent false
9505 3612 Acked Write 7 Done Done Sent false
9506 3613 Acked Read 7 Done Done Sent false
9507 3614 Acked Read 10 Done Done Sent false
9508 3615 Acked Write 9 Done Done Sent false
9509 3616 Acked Read 8 Done Done Sent false
9510 3617 Acked Flush 0 Done Done Sent false
9511 3618 Acked Read 7 Done Done Sent false
9512 3619 Acked Read 6 Done Done Sent false
9513 3620 Acked Write 9 Done Done Sent false
9514 3621 Acked Read 8 Done Done Sent false
9515 3622 Acked Write 2 Done Done Sent false
9516 3623 Acked Flush 0 Done Done Sent false
9517 3624 Acked Write 7 Done Done Sent false
9518 3625 Acked Read 7 Done Done Sent false
9519 3626 Acked Read 8 Done Done Sent false
9520 3627 Acked Write 6 Done Done Sent false
9521 3628 Acked Read 5 Done Done Sent false
9522 3629 Acked Flush 0 Done Done Sent false
9523 3630 Acked Read 10 Done Done Sent false
9524 3631 Acked Read 8 Done Done Sent false
9525 3632 Acked Write 4 Done Done Sent false
9526 3633 Acked Flush 0 Done Done Sent false
9527 3634 Acked Read 3 Done Done Sent false
9528 3635 Acked Read 4 Done Done Sent false
9529 3636 Acked Read 8 Done Done Sent false
9530 3637 Acked Write 7 Done Done Sent false
9531 3638 Acked Read 4 Done Done Sent false
9532 3639 Acked Read 5 Done Done Sent false
9533 3640 Acked Write 10 Done Done Sent false
9534 3641 Acked Read 2 Done Done Sent false
9535 3642 Acked Read 3 Done Done Sent false
9536 3643 Acked Write 3 Done Done Sent false
9537 3644 Acked Write 3 Done Done Sent false
9538 3645 Acked Write 8 Done Done Sent false
9539 3646 Acked Write 3 Done Done Sent false
9540 3647 Acked Read 8 Done Done Sent false
9541 3648 Acked Read 10 Done Done Sent false
9542 3649 Acked Write 2 Done Done Sent false
9543 3650 Acked Flush 0 Done Done Sent false
9544 3651 Acked Write 8 Done Done Sent false
9545 3652 Acked Read 3 Done Done Sent false
9546 3653 Acked Read 3 Done Done Sent false
9547 3654 Acked Read 5 Done Done Sent false
9548 3655 Acked Read 2 Done Done Sent false
9549 3656 Acked Read 2 Done Done Sent false
9550 3657 Acked Read 10 Done Done Sent false
9551 3658 Acked Flush 0 Done Done Sent false
9552 3659 Acked Read 1 Done Done Sent false
9553 3660 Acked Write 6 Done Done Sent false
9554 3661 Acked Read 1 Done Done Sent false
9555 3662 Acked Read 5 Done Done Sent false
9556 3663 Acked Read 4 Done Done Sent false
9557 3664 Acked Write 3 Done Done Sent false
9558 3665 Acked Write 1 Done Done Sent false
9559 3666 Acked Write 6 Done Done Sent false
9560 3667 Acked Read 3 Done Done Sent false
9561 3668 Acked Flush 0 Done Done Sent false
9562 3669 Acked Write 3 Done Done Sent false
9563 3670 Acked Read 3 Done Done Sent false
9564 3671 Acked Flush 0 Done Done Sent false
9565 3672 Acked Read 8 Done Done Sent false
9566 3673 Acked Write 2 Done Done Sent false
9567 3674 Acked Read 4 Done Done Sent false
9568 3675 Acked Read 10 Done Done Sent false
9569 3676 Acked Flush 0 Done Done Sent false
9570 3677 Acked Read 8 Done Done Sent false
9571 3678 Acked Write 10 Done Done Sent false
9572 3679 Acked Flush 0 Done Done Sent false
9573 3680 Acked Read 3 Done Done Sent false
9574 3681 Acked Write 4 Done Done Sent false
9575 3682 Acked Write 4 Done Done Sent false
9576 3683 Acked Read 7 Done Done Sent false
9577 3684 Acked Read 9 Done Done Sent false
9578 3685 Acked Flush 0 Done Done Sent false
9579 3686 Acked Read 1 Done Done Sent false
9580 3687 Acked Read 1 Done Done Sent false
9581 3688 Acked Read 1 Done Done Sent false
9582 3689 Acked Read 4 Done Done Sent false
9583 3690 Acked Write 4 Done Done Sent false
9584 3691 Acked Write 2 Done Done Sent false
9585 3692 Acked Write 9 Done Done Sent false
9586 3693 Acked Write 4 Done Done Sent false
9587 3694 Acked Read 7 Done Done Sent false
9588 3695 Acked Read 5 Done Done Sent false
9589 3696 Acked Read 2 Done Done Sent false
9590 3697 Acked Write 1 Done Done Sent false
9591 3698 Acked Read 2 Done Done Sent false
9592 3699 Acked Read 4 Done Done Sent false
9593 3700 Acked Read 10 Done Done Sent false
9594 3701 Acked Read 9 Done Done Sent false
9595 3702 Acked Read 6 Done Done Sent false
9596 3703 Acked Read 10 Done Done Sent false
9597 3704 Acked Read 2 Done Done Sent false
9598 3705 Acked Write 6 Done Done Sent false
9599 3706 Acked Read 5 Done Done Sent false
9600 3707 Acked Read 1 Done Done Sent false
9601 3708 Acked Read 10 Done Done Sent false
9602 3709 Acked Write 8 Done Done Sent false
9603 3710 Acked Write 7 Done Done Sent false
9604 3711 Acked Flush 0 Done Done Sent false
9605 3712 Acked Read 6 Done Done Sent false
9606 3713 Acked Write 3 Done Done Sent false
9607 3714 Acked Write 4 Done Done Sent false
9608 3715 Acked Read 5 Done Done Sent false
9609 3716 Acked Write 2 Done Done Sent false
9610 3717 Acked Flush 0 Done Done Sent false
9611 STATES DS:0 DS:1 DS:2 TOTAL
9612 Sent 0 0 302 302
9613 Done 302 302 0 604
9614 Skipped 0 0 0 0
9615 Error 0 0 0 0
9616 Last Flush: 3717 3717 3415
9617 Downstairs last five completed: 3415 3414 3413 3412 3411
9618 Upstairs last five completed: 3717 3716 3715 3714 3713
9619 CLIENT: Up:0 ds:302 act:2
96202024-12-09T15:09:53.073ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96212024-12-09T15:09:53.073ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53833) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96222024-12-09T15:09:53.074ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96232024-12-09T15:09:53.075ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96242024-12-09T15:09:53.075ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96252024-12-09T15:09:53.075ZINFOcrucible: send last flush ID to this DS: 3415 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96262024-12-09T15:09:53.075ZINFOcrucible: Replied this last flush ID: 3415 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96272024-12-09T15:09:53.075ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96282024-12-09T15:09:53.075ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96292024-12-09T15:09:53.075ZINFOcrucible: [2] client re-new 302 jobs since flush 3415 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96302024-12-09T15:09:53.076ZINFOcrucible: [2] Replayed 302 jobs since flush: 3415 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
9631 ----------------------------------------------------------------
9632 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9633 Downstairs last five completed: 3717 3716 3715 3714 3713
9634 Upstairs last five completed: 3717 3716 3715 3714 3713
9635 CLIENT: Up:0 ds:0 act:3
9636 Replay: All jobs finished
9637 Main task has work to do, go find it
9638 got dsc Stop(2)
9639 stop 2
9640 Dec 09 15:09:53.325 INFO request completed, latency_us: [8830] Got stop action so:false kr:false
9641 337, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 00b9cb93-9b34-4482-bf44-b521e485fe20, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
96422024-12-09T15:09:53.326ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96432024-12-09T15:09:53.326ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96442024-12-09T15:09:53.326ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96452024-12-09T15:09:53.326ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
96462024-12-09T15:09:53.326ZINFOcrucible: [2] Transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
9647 Dec 09 15:09:53.326 INFO request completed, latency_us: 162, response_code: 200, uri: /state/cid/2{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:09:53.326843983Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
9648 , method: GET, req_id: 6fa51716-73e1-4dad-a67d-258e0cf6b692, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
9649 [8830] Exited with: ExitStatus(unix_wait_status(9))
9650 [8830][2] reports Exit
9651 Dec 09 15:09:57.328 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5bd115cd-ac59-489a-b9e3-d0e19f444b6e, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
9652 001/300 Write block 877 len 24576 data: 3 3 3 3 4 4
9653 002/300 Read block 2821 len 16384
9654 003/300 Write block 2434 len 8192 data: 3 3
9655 004/300 Write block 4825 len 4096 data: 3
9656 005/300 Write block 1108 len 40960 data: 4 4 3 3 4 4 5 4 3 2
9657 006/300 Read block 4068 len 24576
9658 007/300 Write block 4296 len 24576 data: 2 2 3 3 3 2
9659 008/300 Write block 3006 len 32768 data: 4 3 3 3 3 3 4 4
9660 009/300 Read block 629 len 40960
9661 010/300 Read block 749 len 32768
9662 011/300 Write block 1425 len 8192 data: 2 2
9663 012/300 Read block 4024 len 4096
9664 013/300 Read block 797 len 20480
9665 014/300 Write block 1486 len 24576 data: 3 3 2 2 2 2
9666 015/300 Read block 2188 len 8192
9667 016/300 Read block 1047 len 4096
9668 017/300 Write block 1634 len 24576 data: 3 3 2 2 2 2
9669 018/300 Write block 37 len 20480 data: 3 3 3 2 2
9670 019/300 Read block 926 len 28672
9671 020/300 Write block 489 len 4096 data: 2
9672 021/300 Write block 750 len 12288 data: 3 3 3
9673 022/300 Write block 3553 len 28672 data: 4 5 5 5 5 6 5
9674 023/300 Write block 3836 len 4096 data: 3
9675 024/300 Flush
9676 025/300 Write block 1374 len 36864 data: 6 5 4 5 4 4 2 2 3
9677 026/300 Write block 4148 len 8192 data: 2 2
9678 027/300 Read block 2783 len 28672
9679 028/300 Read block 1816 len 8192
9680 029/300 Read block 503 len 16384
9681 030/300 Write block 3515 len 32768 data: 2 2 4 5 5 6 5 4
9682 031/300 Write block 3972 len 32768 data: 4 3 4 4 4 4 4 5
9683 032/300 Write block 3614 len 32768 data: 5 4 4 4 4 4 4 4
9684 033/300 Write block 4421 len 32768 data: 2 2 2 2 3 4 4 4
9685 034/300 Flush
9686 035/300 Write block 4547 len 4096 data: 2
9687 036/300 Read block 2412 len 24576
9688 037/300 Read block 3241 len 16384
9689 038/300 Read block 3374 len 8192
9690 039/300 Read block 123 len 8192
9691 040/300 Write block 142 len 36864 data: 5 3 3 4 4 3 3 6 6
9692 041/300 Read block 2569 len 20480
9693 042/300 Write block 338 len 40960 data: 3 2 2 2 2 2 2 2 2 2
9694 043/300 Write block 1268 len 8192 data: 2 2
9695 044/300 Write block 1795 len 24576 data: 4 4 3 3 2 2
9696 045/300 Write block 4791 len 40960 data: 2 2 2 2 2 3 3 3 4 5
9697 046/300 Write block 871 len 4096 data: 2
9698 047/300 Write block 4492 len 8192 data: 2 3
9699 048/300 Write block 105 len 12288 data: 3 3 3
9700 049/300 Write block 1660 len 32768 data: 3 2 3 3 4 4 4 3
9701 050/300 Read block 2201 len 16384
9702 051/300 Write block 112 len 12288 data: 2 3 3
9703 052/300 Read block 2958 len 36864
9704 053/300 Read block 2029 len 16384
9705 054/300 Write block 1612 len 4096 data: 4
9706 055/300 Write block 2120 len 40960 data: 3 3 3 3 3 3 2 2 4 4
9707 056/300 Read block 3788 len 20480
9708 057/300 Write block 2630 len 16384 data: 2 2 2 2
9709 058/300 Write block 3525 len 16384 data: 4 3 3 3
9710 059/300 Read block 2502 len 28672
9711 060/300 Write block 4578 len 40960 data: 3 3 3 3 4 5 3 3 3 3
9712 061/300 Flush
9713 062/300 Read block 3360 len 16384
9714 063/300 Write block 2336 len 24576 data: 7 7 6 6 6 6
9715 064/300 Write block 2999 len 36864 data: 3 3 3 4 4 4 4 5 4
9716 065/300 Read block 4135 len 36864
9717 066/300 Read block 3169 len 12288
9718 067/300 Write block 4861 len 4096 data: 3
9719 068/300 Read block 4870 len 36864
9720 069/300 Read block 2194 len 20480
9721 070/300 Write block 796 len 12288 data: 3 3 4
9722 071/300 Read block 216 len 24576
9723 072/300 Write block 4006 len 20480 data: 3 3 3 3 4
9724 073/300 Write block 4295 len 8192 data: 2 3
9725 074/300 Read block 3426 len 20480
9726 075/300 Read block 4696 len 24576
9727 076/300 Write block 3876 len 20480 data: 3 3 2 2 4
9728 077/300 Read block 619 len 16384
9729 078/300 Read block 1118 len 12288
9730 079/300 Write block 818 len 4096 data: 3
9731 080/300 Write block 4223 len 40960 data: 4 4 4 4 3 4 3 3 2 2
9732 081/300 Write block 3221 len 16384 data: 2 3 3 3
9733 082/300 Write block 2799 len 32768 data: 4 3 3 2 2 2 2 2
9734 083/300 Read block 4944 len 4096
9735 084/300 Read block 3282 len 16384
9736 085/300 Read block 4624 len 4096
9737 086/300 Write block 1094 len 16384 data: 4 4 3 2
9738 087/300 Write block 4591 len 20480 data: 2 3 4 4 4
9739 088/300 Write block 4095 len 16384 data: 4 5 5 5
9740 089/300 Read block 2544 len 8192
9741 090/300 Read block 1241 len 32768
9742 091/300 Read block 2412 len 20480
9743 092/300 Read block 4628 len 20480
9744 093/300 Write block 22 len 20480 data: 3 3 3 3 3
9745 094/300 Read block 2262 len 4096
9746 095/300 Read block 2654 len 40960
9747 096/300 Flush
9748 097/300 Read block 2252 len 24576
9749 098/300 Read block 2124 len 24576
9750 099/300 Write block 841 len 28672 data: 3 3 2 2 3 2 2
9751 100/300 Write block 4647 len 40960 data: 4 4 4 5 4 4 3 4 3 3
9752 101/300 Write block 1828 len 36864 data: 5 4 3 3 3 2 3 4 3
9753 102/300 Read block 3183 len 36864
9754 103/300 Read block 1212 len 40960
9755 104/300 Read block 809 len 24576
9756 105/300 Read block 3411 len 40960
9757 106/300 Read block 1673 len 24576
9758 107/300 Write block 1408 len 36864 data: 4 4 4 4 4 4 4 4 5
9759 108/300 Read block 778 len 12288
9760 109/300 Read block 2430 len 24576
9761 110/300 Read block 3312 len 36864
9762 111/300 Read block 1045 len 16384
9763 112/300 Write block 1650 len 4096 data: 3
9764 113/300 Write block 4887 len 32768 data: 2 2 3 2 2 2 2 2
9765 114/300 Read block 1118 len 24576
9766 115/300 Read block 3879 len 12288
9767 116/300 Read block 538 len 20480
9768 117/300 Write block 2415 len 4096 data: 2
9769 118/300 Read block 4006 len 12288
9770 119/300 Read block 3642 len 20480
9771 120/300 Write block 3394 len 16384 data: 4 4 4 4
9772 121/300 Read block 2917 len 4096
9773 122/300 Write block 3023 len 4096 data: 3
9774 123/300 Read block 285 len 36864
9775 124/300 Write block 3151 len 16384 data: 6 5 4 4
9776 125/300 Read block 744 len 40960
9777 126/300 Write block 4582 len 36864 data: 5 6 4 4 4 4 3 2 2
9778 127/300 Read block 4196 len 4096
9779 128/300 Read block 2570 len 36864
9780 129/300 Write block 4171 len 16384 data: 5 6 5 4
9781 130/300 Read block 4443 len 28672
9782 131/300 Read block 744 len 24576
9783 132/300 Read block 198 len 16384
9784 133/300 Read block 103 len 4096
9785 134/300 Write block 4158 len 24576 data: 5 5 3 4 3 3
9786 135/300 Write block 4506 len 24576 data: 5 5 6 5 5 5
9787 136/300 Read block 2140 len 16384
9788 137/300 Read block 2478 len 40960
9789 138/300 Write block 570 len 36864 data: 2 3 3 4 4 5 5 5 6
9790 139/300 Read block 2874 len 32768
9791 140/300 Write block 1775 len 32768 data: 4 4 3 3 5 5 5 5
9792 141/300 Read block 2296 len 16384
9793 142/300 Write block 1608 len 32768 data: 3 4 4 4 5 4 4 4
9794 143/300 Write block 3242 len 20480 data: 3 5 5 5 5
9795 144/300 Read block 2317 len 20480
9796 145/300 Write block 738 len 40960 data: 2 2 3 3 3 2 2 2 2 2
9797 146/300 Write block 3870 len 28672 data: 3 3 3 3 3 3 4
9798 147/300 Flush
9799 148/300 Write block 1518 len 12288 data: 4 3 3
9800 149/300 Read block 1526 len 32768
9801 150/300 Read block 1433 len 20480
9802 151/300 Read block 4121 len 16384
9803 152/300 Read block 251 len 12288
9804 153/300 Write block 2089 len 12288 data: 3 3 3
9805 154/300 Write block 1740 len 8192 data: 4 3
9806 155/300 Read block 3998 len 16384
9807 156/300 Read block 1544 len 24576
9808 157/300 Read block 540 len 24576
9809 158/300 Read block 103 len 4096
9810 159/300 Read block 1594 len 16384
9811 160/300 Write block 2253 len 32768 data: 3 4 4 4 4 4 3 3
9812 161/300 Flush
9813 162/300 Write block 4650 len 36864 data: 6 5 5 4 5 4 4 3 3
9814 163/300 Flush
9815 164/300 Read block 3915 len 12288
9816 165/300 Read block 3064 len 36864
9817 166/300 Read block 316 len 28672
9818 167/300 Read block 4256 len 40960
9819 168/300 Flush
9820 169/300 Write block 3144 len 12288 data: 6 6 6
9821 170/300 Read block 2255 len 8192
9822 171/300 Write block 2061 len 24576 data: 3 2 3 3 3 3
9823 172/300 Write block 4319 len 36864 data: 2 2 3 3 3 3 3 3 3
9824 173/300 Flush
9825 174/300 Write block 269 len 16384 data: 3 4 4 4
9826 175/300 Read block 4389 len 8192
9827 176/300 Read block 2513 len 32768
9828 177/300 Read block 1841 len 36864
9829 178/300 Read block 2619 len 8192
9830 179/300 Read block 2797 len 40960
9831 180/300 Flush
9832 181/300 Read block 2651 len 36864
9833 182/300 Write block 301 len 4096 data: 3
9834 183/300 Write block 3553 len 28672 data: 5 6 6 6 6 7 6
9835 184/300 Read block 118 len 36864
9836 185/300 Write block 4502 len 32768 data: 4 4 4 4 6 6 7 6
9837 186/300 Read block 3941 len 4096
9838 187/300 Read block 3983 len 12288
9839 188/300 Read block 2171 len 8192
9840 189/300 Read block 3277 len 32768
9841 190/300 Write block 4435 len 8192 data: 6 4
9842 191/300 Flush
9843 192/300 Read block 929 len 12288
9844 193/300 Write block 3531 len 28672 data: 6 4 4 4 5 5 4
9845 194/300 Read block 2178 len 24576
9846 195/300 Write block 2439 len 28672 data: 4 3 4 4 5 5 4
9847 196/300 Flush
9848 197/300 Write block 2516 len 20480 data: 4 4 4 3 3
9849 198/300 Flush
9850 199/300 Read block 1314 len 16384
9851 200/300 Read block 453 len 12288
9852 201/300 Flush
9853 202/300 Write block 297 len 40960 data: 2 2 2 3 4 3 3 4 4 4
9854 203/300 Read block 2710 len 40960
9855 204/300 Write block 2558 len 40960 data: 4 5 4 3 3 2 2 2 2 2
9856 205/300 Write block 3455 len 40960 data: 4 3 3 3 3 3 3 3 4 5
9857 206/300 Read block 4610 len 20480
9858 207/300 Flush
9859 208/300 Write block 4087 len 16384 data: 5 5 6 7
9860 209/300 Read block 1396 len 12288
9861 210/300 Read block 1104 len 24576
9862 211/300 Read block 1163 len 8192
9863 212/300 Read block 4529 len 16384
9864 213/300 Flush
9865 214/300 Read block 4450 len 36864
9866 215/300 Read block 2828 len 8192
9867 216/300 Read block 4040 len 36864
9868 217/300 Read block 542 len 40960
9869 218/300 Read block 4067 len 4096
9870 219/300 Write block 2581 len 24576 data: 3 2 3 3 3 2
9871 220/300 Read block 3286 len 12288
9872 221/300 Read block 3805 len 24576
9873 222/300 Write block 4888 len 4096 data: 3
9874 223/300 Write block 739 len 40960 data: 3 4 4 4 3 3 3 3 3 2
9875 224/300 Write block 1676 len 24576 data: 2 2 2 2 2 2
9876 225/300 Read block 3595 len 20480
9877 226/300 Write block 3029 len 8192 data: 3 2
9878 227/300 Read block 1266 len 8192
9879 228/300 Read block 449 len 24576
9880 229/300 Read block 1564 len 36864
9881 230/300 Read block 1925 len 24576
9882 231/300 Read block 3288 len 28672
9883 232/300 Read block 3461 len 24576
9884 233/300 Read block 3487 len 40960
9885 234/300 Read block 4150 len 12288
9886 235/300 Flush
9887 236/300 Write block 2985 len 32768 data: 5 6 7 6 5 4 4 4
9888 237/300 Read block 2125 len 28672
9889 238/300 Flush
9890 239/300 Write block 4188 len 24576 data: 5 4 4 3 3 2
9891 240/300 Write block 4081 len 8192 data: 3 4
9892 241/300 Read block 952 len 28672
9893 242/300 Write block 245 len 40960 data: 4 5 4 4 3 3 3 3 3 2
9894 243/300 Read block 3587 len 24576
9895 244/300 Write block 3371 len 28672 data: 2 2 2 2 3 2 2
9896 245/300 Write block 2644 len 28672 data: 5 5 4 2 3 4 4
9897 246/300 Read block 2438 len 16384
9898 247/300 Read block 2383 len 12288
9899 248/300 Write block 2532 len 40960 data: 6 6 6 6 6 5 4 4 4 4
9900 249/300 Write block 1451 len 20480 data: 4 4 4 3 2
9901 250/300 Flush
9902 251/300 Read block 3001 len 8192
9903 252/300 Flush
9904 253/300 Read block 231 len 40960
9905 254/300 Read block 1959 len 20480
9906 255/300 Read block 898 len 8192
9907 256/300 Read block 3477 len 8192
9908 257/300 Read block 2277 len 32768
9909 258/300 Read block 3341 len 32768
9910 259/300 Read block 1145 len 4096
9911 260/300 Read block 977 len 36864
9912 261/300 Read block 3242 len 36864
9913 262/300 Write block 546 len 32768 data: 5 7 6 5 5 5 4 3
9914 263/300 Write block 2855 len 12288 data: 3 3 3
9915 264/300 Write block 3775 len 28672 data: 2 2 3 3 3 3 4
9916 265/300 Write block 1291 len 28672 data: 4 4 4 4 4 3 4
9917 266/300 Read block 3364 len 36864
9918 267/300 Write block 2625 len 4096 data: 4
9919 268/300 Write block 1533 len 24576 data: 4 5 4 4 4 3
9920 269/300 Write block 1671 len 12288 data: 3 2 2
9921 270/300 Write block 2494 len 40960 data: 5 5 4 4 4 4 4 4 3 4
9922 271/300 Read block 1081 len 40960
9923 272/300 Write block 2246 len 20480 data: 3 3 3 4 4
9924 273/300 Read block 2516 len 16384
9925 274/300 Read block 1814 len 20480
9926 275/300 Read block 3457 len 36864
9927 276/300 Write block 4701 len 24576 data: 5 4 4 4 4 4
9928 277/300 Write block 2903 len 4096 data: 2
9929 278/300 Read block 3226 len 8192
9930 279/300 Read block 1119 len 28672
9931 280/300 Read block 4582 len 16384
9932 281/300 Read block 4496 len 4096
9933 282/300 Write block 2928 len 36864 data: 4 3 4 4 4 4 2 2 2
9934 283/300 Read block 3601 len 32768
9935 284/300 Read block 581 len 24576
9936 285/300 Flush
9937 286/300 Flush
9938 287/300 Write block 3880 len 28672 data: 5 4 4 4 3 4 6
9939 288/300 Read block 2063 len 24576
9940 289/300 Read block 1456 len 16384
9941 290/300 Read block 727 len 12288
9942 291/300 Write block 4662 len 12288 data: 4 4 4
9943 292/300 Read block 1678 len 8192
9944 293/300 Flush
9945 294/300 Write block 957 len 8192 data: 3 3
9946 295/300 Read block 585 len 16384
9947 296/300 Read block 4893 len 20480
9948 297/300 Flush
9949 298/300 Read block 1713 len 36864
9950 299/300 Read block 4234 len 12288
9951 300/300 Read block 4507 len 4096
9952 301/300 Read block 4512 len 16384
9953 Main task has work to do, go find it
9954 got dsc Start(2)
9955 start 2
9956 Dec 09 15:09:57.572 INFO request completed, latency_us: 250, response_code: 204, uri: /start/cid/2, method: POST, req_id: 5fa294b8-1047-4d76-b4ff-420f838f924b, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
9957 Replay: started 2, returned:Ok(())
9958 ----------------------------------------------------------------
9959 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9960 States: Active Active Offline
9961 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9962 3718 Acked Write 6 Done Done Sent false
9963 3719 Acked Read 4 Done Done Sent false
9964 3720 Acked Write 2 Done Done Sent false
9965 3721 Acked Write 1 Done Done Sent false
9966 3722 Acked Write 10 Done Done Sent false
9967 3723 Acked Read 6 Done Done Sent false
9968 3724 Acked Write 6 Done Done Sent false
9969 3725 Acked Write 8 Done Done Sent false
9970 3726 Acked Read 10 Done Done Sent false
9971 3727 Acked Read 8 Done Done Sent false
9972 3728 Acked Write 2 Done Done Sent false
9973 3729 Acked Read 1 Done Done Sent false
9974 3730 Acked Read 5 Done Done Sent false
9975 3731 Acked Write 6 Done Done Sent false
9976 3732 Acked Read 2 Done Done Sent false
9977 3733 Acked Read 1 Done Done Sent false
9978 3734 Acked Write 6 Done Done Sent false
9979 3735 Acked Write 5 Done Done Sent false
9980 3736 Acked Read 7 Done Done Sent false
9981 3737 Acked Write 1 Done Done Sent false
9982 3738 Acked Write 3 Done Done Sent false
9983 3739 Acked Write 7 Done Done Sent false
9984 3740 Acked Write 1 Done Done Sent false
9985 3741 Acked Flush 0 Done Done Sent false
9986 3742 Acked Write 9 Done Done Sent false
9987 3743 Acked Write 2 Done Done Sent false
9988 3744 Acked Read 7 Done Done Sent false
9989 3745 Acked Read 2 Done Done Sent false
9990 3746 Acked Read 4 Done Done Sent false
9991 3747 Acked Write 8 Done Done Sent false
9992 3748 Acked Write 8 Done Done Sent false
9993 3749 Acked Write 8 Done Done Sent false
9994 3750 Acked Write 8 Done Done Sent false
9995 3751 Acked Flush 0 Done Done Sent false
9996 3752 Acked Write 1 Done Done Sent false
9997 3753 Acked Read 6 Done Done Sent false
9998 3754 Acked Read 4 Done Done Sent false
9999 3755 Acked Read 2 Done Done Sent false
10000 3756 Acked Read 2 Done Done Sent false
10001 3757 Acked Write 9 Done Done Sent false
10002 3758 Acked Read 5 Done Done Sent false
10003 3759 Acked Write 10 Done Done Sent false
10004 3760 Acked Write 2 Done Done Sent false
10005 3761 Acked Write 6 Done Done Sent false
10006 3762 Acked Write 10 Done Done Sent false
10007 3763 Acked Write 1 Done Done Sent false
10008 3764 Acked Write 2 Done Done Sent false
10009 3765 Acked Write 3 Done Done Sent false
10010 3766 Acked Write 8 Done Done Sent false
10011 3767 Acked Read 4 Done Done Sent false
10012 3768 Acked Write 3 Done Done Sent false
10013 3769 Acked Read 9 Done Done Sent false
10014 3770 Acked Read 4 Done Done Sent false
10015 3771 Acked Write 1 Done Done Sent false
10016 3772 Acked Write 10 Done Done Sent false
10017 3773 Acked Read 5 Done Done Sent false
10018 3774 Acked Write 4 Done Done Sent false
10019 3775 Acked Write 4 Done Done Sent false
10020 3776 Acked Read 7 Done Done Sent false
10021 3777 Acked Write 10 Done Done Sent false
10022 3778 Acked Flush 0 Done Done Sent false
10023 3779 Acked Read 4 Done Done Sent false
10024 3780 Acked Write 6 Done Done Sent false
10025 3781 Acked Write 9 Done Done Sent false
10026 3782 Acked Read 9 Done Done Sent false
10027 3783 Acked Read 3 Done Done Sent false
10028 3784 Acked Write 1 Done Done Sent false
10029 3785 Acked Read 9 Done Done Sent false
10030 3786 Acked Read 5 Done Done Sent false
10031 3787 Acked Write 3 Done Done Sent false
10032 3788 Acked Read 6 Done Done Sent false
10033 3789 Acked Write 5 Done Done Sent false
10034 3790 Acked Write 2 Done Done Sent false
10035 3791 Acked Read 5 Done Done Sent false
10036 3792 Acked Read 6 Done Done Sent false
10037 3793 Acked Write 5 Done Done Sent false
10038 3794 Acked Read 4 Done Done Sent false
10039 3795 Acked Read 3 Done Done Sent false
10040 3796 Acked Write 1 Done Done Sent false
10041 3797 Acked Write 10 Done Done Sent false
10042 3798 Acked Write 4 Done Done Sent false
10043 3799 Acked Write 8 Done Done Sent false
10044 3800 Acked Read 1 Done Done Sent false
10045 3801 Acked Read 4 Done Done Sent false
10046 3802 Acked Read 1 Done Done Sent false
10047 3803 Acked Write 4 Done Done Sent false
10048 3804 Acked Write 5 Done Done Sent false
10049 3805 Acked Write 4 Done Done Sent false
10050 3806 Acked Read 2 Done Done Sent false
10051 3807 Acked Read 8 Done Done Sent false
10052 3808 Acked Read 5 Done Done Sent false
10053 3809 Acked Read 5 Done Done Sent false
10054 3810 Acked Write 5 Done Done Sent false
10055 3811 Acked Read 1 Done Done Sent false
10056 3812 Acked Read 10 Done Done Sent false
10057 3813 Acked Flush 0 Done Done Sent false
10058 3814 Acked Read 6 Done Done Sent false
10059 3815 Acked Read 6 Done Done Sent false
10060 3816 Acked Write 7 Done Done Sent false
10061 3817 Acked Write 10 Done Done Sent false
10062 3818 Acked Write 9 Done Done Sent false
10063 3819 Acked Read 9 Done Done Sent false
10064 3820 Acked Read 10 Done Done Sent false
10065 3821 Acked Read 6 Done Done Sent false
10066 3822 Acked Read 10 Done Done Sent false
10067 3823 Acked Read 6 Done Done Sent false
10068 3824 Acked Write 9 Done Done Sent false
10069 3825 Acked Read 3 Done Done Sent false
10070 3826 Acked Read 6 Done Done Sent false
10071 3827 Acked Read 9 Done Done Sent false
10072 3828 Acked Read 4 Done Done Sent false
10073 3829 Acked Write 1 Done Done Sent false
10074 3830 Acked Write 8 Done Done Sent false
10075 3831 Acked Read 6 Done Done Sent false
10076 3832 Acked Read 3 Done Done Sent false
10077 3833 Acked Read 5 Done Done Sent false
10078 3834 Acked Write 1 Done Done Sent false
10079 3835 Acked Read 3 Done Done Sent false
10080 3836 Acked Read 5 Done Done Sent false
10081 3837 Acked Write 4 Done Done Sent false
10082 3838 Acked Read 1 Done Done Sent false
10083 3839 Acked Write 1 Done Done Sent false
10084 3840 Acked Read 9 Done Done Sent false
10085 3841 Acked Write 4 Done Done Sent false
10086 3842 Acked Read 10 Done Done Sent false
10087 3843 Acked Write 9 Done Done Sent false
10088 3844 Acked Read 1 Done Done Sent false
10089 3845 Acked Read 9 Done Done Sent false
10090 3846 Acked Write 4 Done Done Sent false
10091 3847 Acked Read 7 Done Done Sent false
10092 3848 Acked Read 6 Done Done Sent false
10093 3849 Acked Read 4 Done Done Sent false
10094 3850 Acked Read 1 Done Done Sent false
10095 3851 Acked Write 6 Done Done Sent false
10096 3852 Acked Write 6 Done Done Sent false
10097 3853 Acked Read 4 Done Done Sent false
10098 3854 Acked Read 10 Done Done Sent false
10099 3855 Acked Write 9 Done Done Sent false
10100 3856 Acked Read 8 Done Done Sent false
10101 3857 Acked Write 8 Done Done Sent false
10102 3858 Acked Read 4 Done Done Sent false
10103 3859 Acked Write 8 Done Done Sent false
10104 3860 Acked Write 5 Done Done Sent false
10105 3861 Acked Read 5 Done Done Sent false
10106 3862 Acked Write 10 Done Done Sent false
10107 3863 Acked Write 7 Done Done Sent false
10108 3864 Acked Flush 0 Done Done Sent false
10109 3865 Acked Write 3 Done Done Sent false
10110 3866 Acked Read 8 Done Done Sent false
10111 3867 Acked Read 5 Done Done Sent false
10112 3868 Acked Read 4 Done Done Sent false
10113 3869 Acked Read 3 Done Done Sent false
10114 3870 Acked Write 3 Done Done Sent false
10115 3871 Acked Write 2 Done Done Sent false
10116 3872 Acked Read 4 Done Done Sent false
10117 3873 Acked Read 6 Done Done Sent false
10118 3874 Acked Read 6 Done Done Sent false
10119 3875 Acked Read 1 Done Done Sent false
10120 3876 Acked Read 4 Done Done Sent false
10121 3877 Acked Write 8 Done Done Sent false
10122 3878 Acked Flush 0 Done Done Sent false
10123 3879 Acked Write 9 Done Done Sent false
10124 3880 Acked Flush 0 Done Done Sent false
10125 3881 Acked Read 3 Done Done Sent false
10126 3882 Acked Read 9 Done Done Sent false
10127 3883 Acked Read 7 Done Done Sent false
10128 3884 Acked Read 10 Done Done Sent false
10129 3885 Acked Flush 0 Done Done Sent false
10130 3886 Acked Write 3 Done Done Sent false
10131 3887 Acked Read 2 Done Done Sent false
10132 3888 Acked Write 6 Done Done Sent false
10133 3889 Acked Write 9 Done Done Sent false
10134 3890 Acked Flush 0 Done Done Sent false
10135 3891 Acked Write 4 Done Done Sent false
10136 3892 Acked Read 2 Done Done Sent false
10137 3893 Acked Read 8 Done Done Sent false
10138 3894 Acked Read 9 Done Done Sent false
10139 3895 Acked Read 2 Done Done Sent false
10140 3896 Acked Read 10 Done Done Sent false
10141 3897 Acked Flush 0 Done Done Sent false
10142 3898 Acked Read 9 Done Done Sent false
10143 3899 Acked Write 1 Done Done Sent false
10144 3900 Acked Write 7 Done Done Sent false
10145 3901 Acked Read 9 Done Done Sent false
10146 3902 Acked Write 8 Done Done Sent false
10147 3903 Acked Read 1 Done Done Sent false
10148 3904 Acked Read 3 Done Done Sent false
10149 3905 Acked Read 2 Done Done Sent false
10150 3906 Acked Read 8 Done Done Sent false
10151 3907 Acked Write 2 Done Done Sent false
10152 3908 Acked Flush 0 Done Done Sent false
10153 3909 Acked Read 3 Done Done Sent false
10154 3910 Acked Write 7 Done Done Sent false
10155 3911 Acked Read 6 Done Done Sent false
10156 3912 Acked Write 7 Done Done Sent false
10157 3913 Acked Flush 0 Done Done Sent false
10158 3914 Acked Write 5 Done Done Sent false
10159 3915 Acked Flush 0 Done Done Sent false
10160 3916 Acked Read 4 Done Done Sent false
10161 3917 Acked Read 3 Done Done Sent false
10162 3918 Acked Flush 0 Done Done Sent false
10163 3919 Acked Write 10 Done Done Sent false
10164 3920 Acked Read 10 Done Done Sent false
10165 3921 Acked Write 10 Done Done Sent false
10166 3922 Acked Write 10 Done Done Sent false
10167 3923 Acked Read 5 Done Done Sent false
10168 3924 Acked Flush 0 Done Done Sent false
10169 3925 Acked Write 4 Done Done Sent false
10170 3926 Acked Read 3 Done Done Sent false
10171 3927 Acked Read 6 Done Done Sent false
10172 3928 Acked Read 2 Done Done Sent false
10173 3929 Acked Read 4 Done Done Sent false
10174 3930 Acked Flush 0 Done Done Sent false
10175 3931 Acked Read 9 Done Done Sent false
10176 3932 Acked Read 2 Done Done Sent false
10177 3933 Acked Read 9 Done Done Sent false
10178 3934 Acked Read 10 Done Done Sent false
10179 3935 Acked Read 1 Done Done Sent false
10180 3936 Acked Write 6 Done Done Sent false
10181 3937 Acked Read 3 Done Done Sent false
10182 3938 Acked Read 6 Done Done Sent false
10183 3939 Acked Write 1 Done Done Sent false
10184 3940 Acked Write 10 Done Done Sent false
10185 3941 Acked Write 6 Done Done Sent false
10186 3942 Acked Read 5 Done Done Sent false
10187 3943 Acked Write 2 Done Done Sent false
10188 3944 Acked Read 2 Done Done Sent false
10189 3945 Acked Read 6 Done Done Sent false
10190 3946 Acked Read 9 Done Done Sent false
10191 3947 Acked Read 6 Done Done Sent false
10192 3948 Acked Read 7 Done Done Sent false
10193 3949 Acked Read 6 Done Done Sent false
10194 3950 Acked Read 10 Done Done Sent false
10195 3951 Acked Read 3 Done Done Sent false
10196 3952 Acked Flush 0 Done Done Sent false
10197 3953 Acked Write 8 Done Done Sent false
10198 3954 Acked Read 7 Done Done Sent false
10199 3955 Acked Flush 0 Done Done Sent false
10200 3956 Acked Write 6 Done Done Sent false
10201 3957 Acked Write 2 Done Done Sent false
10202 3958 Acked Read 7 Done Done Sent false
10203 3959 Acked Write 10 Done Done Sent false
10204 3960 Acked Read 6 Done Done Sent false
10205 3961 Acked Write 7 Done Done Sent false
10206 3962 Acked Write 7 Done Done Sent false
10207 3963 Acked Read 4 Done Done Sent false
10208 3964 Acked Read 3 Done Done Sent false
10209 3965 Acked Write 10 Done Done Sent false
10210 3966 Acked Write 5 Done Done Sent false
10211 3967 Acked Flush 0 Done Done Sent false
10212 3968 Acked Read 2 Done Done Sent false
10213 3969 Acked Flush 0 Done Done Sent false
10214 3970 Acked Read 10 Done Done Sent false
10215 3971 Acked Read 5 Done Done Sent false
10216 3972 Acked Read 2 Done Done Sent false
10217 3973 Acked Read 2 Done Done Sent false
10218 3974 Acked Read 8 Done Done Sent false
10219 3975 Acked Read 8 Done Done Sent false
10220 3976 Acked Read 1 Done Done Sent false
10221 3977 Acked Read 9 Done Done Sent false
10222 3978 Acked Read 9 Done Done Sent false
10223 3979 Acked Write 8 Done Done Sent false
10224 3980 Acked Write 3 Done Done Sent false
10225 3981 Acked Write 7 Done Done Sent false
10226 3982 Acked Write 7 Done Done Sent false
10227 3983 Acked Read 9 Done Done Sent false
10228 3984 Acked Write 1 Done Done Sent false
10229 3985 Acked Write 6 Done Done Sent false
10230 3986 Acked Write 3 Done Done Sent false
10231 3987 Acked Write 10 Done Done Sent false
10232 3988 Acked Read 10 Done Done Sent false
10233 3989 Acked Write 5 Done Done Sent false
10234 3990 Acked Read 4 Done Done Sent false
10235 3991 Acked Read 5 Done Done Sent false
10236 3992 Acked Read 9 Done Done Sent false
10237 3993 Acked Write 6 Done Done Sent false
10238 3994 Acked Write 1 Done Done Sent false
10239 3995 Acked Read 2 Done Done Sent false
10240 3996 Acked Read 7 Done Done Sent false
10241 3997 Acked Read 4 Done Done Sent false
10242 3998 Acked Read 1 Done Done Sent false
10243 3999 Acked Write 9 Done Done Sent false
10244 4000 Acked Read 8 Done Done Sent false
10245 4001 Acked Read 6 Done Done Sent false
10246 4002 Acked Flush 0 Done Done Sent false
10247 4003 Acked Flush 0 Done Done Sent false
10248 4004 Acked Write 7 Done Done Sent false
10249 4005 Acked Read 6 Done Done Sent false
10250 4006 Acked Read 4 Done Done Sent false
10251 4007 Acked Read 3 Done Done Sent false
10252 4008 Acked Write 3 Done Done Sent false
10253 4009 Acked Read 2 Done Done Sent false
10254 4010 Acked Flush 0 Done Done Sent false
10255 4011 Acked Write 2 Done Done Sent false
10256 4012 Acked Read 4 Done Done Sent false
10257 4013 Acked Read 5 Done Done Sent false
10258 4014 Acked Flush 0 Done Done Sent false
10259 4015 Acked Read 9 Done Done Sent false
10260 4016 Acked Read 3 Done Done Sent false
10261 4017 Acked Read 1 Done Done Sent false
10262 4018 Acked Read 4 Done Done Sent false
10263 STATES DS:0 DS:1 DS:2 TOTAL
10264 Sent 0 0 301 301
10265 Done 301 301 0 602
10266 Skipped 0 0 0 0
10267 Error 0 0 0 0
10268 Last Flush: 4014 4014 3717
10269 Downstairs last five completed: 3717 3716 3715 3714 3713
10270 Upstairs last five completed: 4018 4017 4016 4015 4014
10271 CLIENT: Up:0 ds:301 act:2
10272 [8830] got start action
10273 [8830] I am going to restart
10274 Starting downstairs at port 8830
10275 Make output file at "/tmp/dsc/downstairs-8830.txt"
10276 [8830][2] reports Starting
10277 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1211)
10278 [8830][2] reports Running
10279 ----------------------------------------------------------------
10280 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
10281 States: Active Active Offline
10282 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10283 3718 Acked Write 6 Done Done Sent false
10284 3719 Acked Read 4 Done Done Sent false
10285 3720 Acked Write 2 Done Done Sent false
10286 3721 Acked Write 1 Done Done Sent false
10287 3722 Acked Write 10 Done Done Sent false
10288 3723 Acked Read 6 Done Done Sent false
10289 3724 Acked Write 6 Done Done Sent false
10290 3725 Acked Write 8 Done Done Sent false
10291 3726 Acked Read 10 Done Done Sent false
10292 3727 Acked Read 8 Done Done Sent false
10293 3728 Acked Write 2 Done Done Sent false
10294 3729 Acked Read 1 Done Done Sent false
10295 3730 Acked Read 5 Done Done Sent false
10296 3731 Acked Write 6 Done Done Sent false
10297 3732 Acked Read 2 Done Done Sent false
10298 3733 Acked Read 1 Done Done Sent false
10299 3734 Acked Write 6 Done Done Sent false
10300 3735 Acked Write 5 Done Done Sent false
10301 3736 Acked Read 7 Done Done Sent false
10302 3737 Acked Write 1 Done Done Sent false
10303 3738 Acked Write 3 Done Done Sent false
10304 3739 Acked Write 7 Done Done Sent false
10305 3740 Acked Write 1 Done Done Sent false
10306 3741 Acked Flush 0 Done Done Sent false
10307 3742 Acked Write 9 Done Done Sent false
10308 3743 Acked Write 2 Done Done Sent false
10309 3744 Acked Read 7 Done Done Sent false
10310 3745 Acked Read 2 Done Done Sent false
10311 3746 Acked Read 4 Done Done Sent false
10312 3747 Acked Write 8 Done Done Sent false
10313 3748 Acked Write 8 Done Done Sent false
10314 3749 Acked Write 8 Done Done Sent false
10315 3750 Acked Write 8 Done Done Sent false
10316 3751 Acked Flush 0 Done Done Sent false
10317 3752 Acked Write 1 Done Done Sent false
10318 3753 Acked Read 6 Done Done Sent false
10319 3754 Acked Read 4 Done Done Sent false
10320 3755 Acked Read 2 Done Done Sent false
10321 3756 Acked Read 2 Done Done Sent false
10322 3757 Acked Write 9 Done Done Sent false
10323 3758 Acked Read 5 Done Done Sent false
10324 3759 Acked Write 10 Done Done Sent false
10325 3760 Acked Write 2 Done Done Sent false
10326 3761 Acked Write 6 Done Done Sent false
10327 3762 Acked Write 10 Done Done Sent false
10328 3763 Acked Write 1 Done Done Sent false
10329 3764 Acked Write 2 Done Done Sent false
10330 3765 Acked Write 3 Done Done Sent false
10331 3766 Acked Write 8 Done Done Sent false
10332 3767 Acked Read 4 Done Done Sent false
10333 3768 Acked Write 3 Done Done Sent false
10334 3769 Acked Read 9 Done Done Sent false
10335 3770 Acked Read 4 Done Done Sent false
10336 3771 Acked Write 1 Done Done Sent false
10337 3772 Acked Write 10 Done Done Sent false
10338 3773 Acked Read 5 Done Done Sent false
10339 3774 Acked Write 4 Done Done Sent false
10340 3775 Acked Write 4 Done Done Sent false
10341 3776 Acked Read 7 Done Done Sent false
10342 3777 Acked Write 10 Done Done Sent false
10343 3778 Acked Flush 0 Done Done Sent false
10344 3779 Acked Read 4 Done Done Sent false
10345 3780 Acked Write 6 Done Done Sent false
10346 3781 Acked Write 9 Done Done Sent false
10347 3782 Acked Read 9 Done Done Sent false
10348 3783 Acked Read 3 Done Done Sent false
10349 3784 Acked Write 1 Done Done Sent false
10350 3785 Acked Read 9 Done Done Sent false
10351 3786 Acked Read 5 Done Done Sent false
10352 3787 Acked Write 3 Done Done Sent false
10353 3788 Acked Read 6 Done Done Sent false
10354 3789 Acked Write 5 Done Done Sent false
10355 3790 Acked Write 2 Done Done Sent false
10356 3791 Acked Read 5 Done Done Sent false
10357 3792 Acked Read 6 Done Done Sent false
10358 3793 Acked Write 5 Done Done Sent false
10359 3794 Acked Read 4 Done Done Sent false
10360 3795 Acked Read 3 Done Done Sent false
10361 3796 Acked Write 1 Done Done Sent false
10362 3797 Acked Write 10 Done Done Sent false
10363 3798 Acked Write 4 Done Done Sent false
10364 3799 Acked Write 8 Done Done Sent false
10365 3800 Acked Read 1 Done Done Sent false
10366 3801 Acked Read 4 Done Done Sent false
10367 3802 Acked Read 1 Done Done Sent false
10368 3803 Acked Write 4 Done Done Sent false
10369 3804 Acked Write 5 Done Done Sent false
10370 3805 Acked Write 4 Done Done Sent false
10371 3806 Acked Read 2 Done Done Sent false
10372 3807 Acked Read 8 Done Done Sent false
10373 3808 Acked Read 5 Done Done Sent false
10374 3809 Acked Read 5 Done Done Sent false
10375 3810 Acked Write 5 Done Done Sent false
10376 3811 Acked Read 1 Done Done Sent false
10377 3812 Acked Read 10 Done Done Sent false
10378 3813 Acked Flush 0 Done Done Sent false
10379 3814 Acked Read 6 Done Done Sent false
10380 3815 Acked Read 6 Done Done Sent false
10381 3816 Acked Write 7 Done Done Sent false
10382 3817 Acked Write 10 Done Done Sent false
10383 3818 Acked Write 9 Done Done Sent false
10384 3819 Acked Read 9 Done Done Sent false
10385 3820 Acked Read 10 Done Done Sent false
10386 3821 Acked Read 6 Done Done Sent false
10387 3822 Acked Read 10 Done Done Sent false
10388 3823 Acked Read 6 Done Done Sent false
10389 3824 Acked Write 9 Done Done Sent false
10390 3825 Acked Read 3 Done Done Sent false
10391 3826 Acked Read 6 Done Done Sent false
10392 3827 Acked Read 9 Done Done Sent false
10393 3828 Acked Read 4 Done Done Sent false
10394 3829 Acked Write 1 Done Done Sent false
10395 3830 Acked Write 8 Done Done Sent false
10396 3831 Acked Read 6 Done Done Sent false
10397 3832 Acked Read 3 Done Done Sent false
10398 3833 Acked Read 5 Done Done Sent false
10399 3834 Acked Write 1 Done Done Sent false
10400 3835 Acked Read 3 Done Done Sent false
10401 3836 Acked Read 5 Done Done Sent false
10402 3837 Acked Write 4 Done Done Sent false
10403 3838 Acked Read 1 Done Done Sent false
10404 3839 Acked Write 1 Done Done Sent false
10405 3840 Acked Read 9 Done Done Sent false
10406 3841 Acked Write 4 Done Done Sent false
10407 3842 Acked Read 10 Done Done Sent false
10408 3843 Acked Write 9 Done Done Sent false
10409 3844 Acked Read 1 Done Done Sent false
10410 3845 Acked Read 9 Done Done Sent false
10411 3846 Acked Write 4 Done Done Sent false
10412 3847 Acked Read 7 Done Done Sent false
10413 3848 Acked Read 6 Done Done Sent false
10414 3849 Acked Read 4 Done Done Sent false
10415 3850 Acked Read 1 Done Done Sent false
10416 3851 Acked Write 6 Done Done Sent false
10417 3852 Acked Write 6 Done Done Sent false
10418 3853 Acked Read 4 Done Done Sent false
10419 3854 Acked Read 10 Done Done Sent false
10420 3855 Acked Write 9 Done Done Sent false
10421 3856 Acked Read 8 Done Done Sent false
10422 3857 Acked Write 8 Done Done Sent false
10423 3858 Acked Read 4 Done Done Sent false
10424 3859 Acked Write 8 Done Done Sent false
10425 3860 Acked Write 5 Done Done Sent false
10426 3861 Acked Read 5 Done Done Sent false
10427 3862 Acked Write 10 Done Done Sent false
10428 3863 Acked Write 7 Done Done Sent false
10429 3864 Acked Flush 0 Done Done Sent false
10430 3865 Acked Write 3 Done Done Sent false
10431 3866 Acked Read 8 Done Done Sent false
10432 3867 Acked Read 5 Done Done Sent false
10433 3868 Acked Read 4 Done Done Sent false
10434 3869 Acked Read 3 Done Done Sent false
10435 3870 Acked Write 3 Done Done Sent false
10436 3871 Acked Write 2 Done Done Sent false
10437 3872 Acked Read 4 Done Done Sent false
10438 3873 Acked Read 6 Done Done Sent false
10439 3874 Acked Read 6 Done Done Sent false
10440 3875 Acked Read 1 Done Done Sent false
10441 3876 Acked Read 4 Done Done Sent false
10442 3877 Acked Write 8 Done Done Sent false
10443 3878 Acked Flush 0 Done Done Sent false
10444 3879 Acked Write 9 Done Done Sent false
10445 3880 Acked Flush 0 Done Done Sent false
10446 3881 Acked Read 3 Done Done Sent false
10447 3882 Acked Read 9 Done Done Sent false
10448 3883 Acked Read 7 Done Done Sent false
10449 3884 Acked Read 10 Done Done Sent false
10450 3885 Acked Flush 0 Done Done Sent false
10451 3886 Acked Write 3 Done Done Sent false
10452 3887 Acked Read 2 Done Done Sent false
10453 3888 Acked Write 6 Done Done Sent false
10454 3889 Acked Write 9 Done Done Sent false
10455 3890 Acked Flush 0 Done Done Sent false
10456 3891 Acked Write 4 Done Done Sent false
10457 3892 Acked Read 2 Done Done Sent false
10458 3893 Acked Read 8 Done Done Sent false
10459 3894 Acked Read 9 Done Done Sent false
10460 3895 Acked Read 2 Done Done Sent false
10461 3896 Acked Read 10 Done Done Sent false
10462 3897 Acked Flush 0 Done Done Sent false
10463 3898 Acked Read 9 Done Done Sent false
10464 3899 Acked Write 1 Done Done Sent false
10465 3900 Acked Write 7 Done Done Sent false
10466 3901 Acked Read 9 Done Done Sent false
10467 3902 Acked Write 8 Done Done Sent false
10468 3903 Acked Read 1 Done Done Sent false
10469 3904 Acked Read 3 Done Done Sent false
10470 3905 Acked Read 2 Done Done Sent false
10471 3906 Acked Read 8 Done Done Sent false
10472 3907 Acked Write 2 Done Done Sent false
10473 3908 Acked Flush 0 Done Done Sent false
10474 3909 Acked Read 3 Done Done Sent false
10475 3910 Acked Write 7 Done Done Sent false
10476 3911 Acked Read 6 Done Done Sent false
10477 3912 Acked Write 7 Done Done Sent false
10478 3913 Acked Flush 0 Done Done Sent false
10479 3914 Acked Write 5 Done Done Sent false
10480 3915 Acked Flush 0 Done Done Sent false
10481 3916 Acked Read 4 Done Done Sent false
10482 3917 Acked Read 3 Done Done Sent false
10483 3918 Acked Flush 0 Done Done Sent false
10484 3919 Acked Write 10 Done Done Sent false
10485 3920 Acked Read 10 Done Done Sent false
10486 3921 Acked Write 10 Done Done Sent false
10487 3922 Acked Write 10 Done Done Sent false
10488 3923 Acked Read 5 Done Done Sent false
10489 3924 Acked Flush 0 Done Done Sent false
10490 3925 Acked Write 4 Done Done Sent false
10491 3926 Acked Read 3 Done Done Sent false
10492 3927 Acked Read 6 Done Done Sent false
10493 3928 Acked Read 2 Done Done Sent false
10494 3929 Acked Read 4 Done Done Sent false
10495 3930 Acked Flush 0 Done Done Sent false
10496 3931 Acked Read 9 Done Done Sent false
10497 3932 Acked Read 2 Done Done Sent false
10498 3933 Acked Read 9 Done Done Sent false
10499 3934 Acked Read 10 Done Done Sent false
10500 3935 Acked Read 1 Done Done Sent false
10501 3936 Acked Write 6 Done Done Sent false
10502 3937 Acked Read 3 Done Done Sent false
10503 3938 Acked Read 6 Done Done Sent false
10504 3939 Acked Write 1 Done Done Sent false
10505 3940 Acked Write 10 Done Done Sent false
10506 3941 Acked Write 6 Done Done Sent false
10507 3942 Acked Read 5 Done Done Sent false
10508 3943 Acked Write 2 Done Done Sent false
10509 3944 Acked Read 2 Done Done Sent false
10510 3945 Acked Read 6 Done Done Sent false
10511 3946 Acked Read 9 Done Done Sent false
10512 3947 Acked Read 6 Done Done Sent false
10513 3948 Acked Read 7 Done Done Sent false
10514 3949 Acked Read 6 Done Done Sent false
10515 3950 Acked Read 10 Done Done Sent false
10516 3951 Acked Read 3 Done Done Sent false
10517 3952 Acked Flush 0 Done Done Sent false
10518 3953 Acked Write 8 Done Done Sent false
10519 3954 Acked Read 7 Done Done Sent false
10520 3955 Acked Flush 0 Done Done Sent false
10521 3956 Acked Write 6 Done Done Sent false
10522 3957 Acked Write 2 Done Done Sent false
10523 3958 Acked Read 7 Done Done Sent false
10524 3959 Acked Write 10 Done Done Sent false
10525 3960 Acked Read 6 Done Done Sent false
10526 3961 Acked Write 7 Done Done Sent false
10527 3962 Acked Write 7 Done Done Sent false
10528 3963 Acked Read 4 Done Done Sent false
10529 3964 Acked Read 3 Done Done Sent false
10530 3965 Acked Write 10 Done Done Sent false
10531 3966 Acked Write 5 Done Done Sent false
10532 3967 Acked Flush 0 Done Done Sent false
10533 3968 Acked Read 2 Done Done Sent false
10534 3969 Acked Flush 0 Done Done Sent false
10535 3970 Acked Read 10 Done Done Sent false
10536 3971 Acked Read 5 Done Done Sent false
10537 3972 Acked Read 2 Done Done Sent false
10538 3973 Acked Read 2 Done Done Sent false
10539 3974 Acked Read 8 Done Done Sent false
10540 3975 Acked Read 8 Done Done Sent false
10541 3976 Acked Read 1 Done Done Sent false
10542 3977 Acked Read 9 Done Done Sent false
10543 3978 Acked Read 9 Done Done Sent false
10544 3979 Acked Write 8 Done Done Sent false
10545 3980 Acked Write 3 Done Done Sent false
10546 3981 Acked Write 7 Done Done Sent false
10547 3982 Acked Write 7 Done Done Sent false
10548 3983 Acked Read 9 Done Done Sent false
10549 3984 Acked Write 1 Done Done Sent false
10550 3985 Acked Write 6 Done Done Sent false
10551 3986 Acked Write 3 Done Done Sent false
10552 3987 Acked Write 10 Done Done Sent false
10553 3988 Acked Read 10 Done Done Sent false
10554 3989 Acked Write 5 Done Done Sent false
10555 3990 Acked Read 4 Done Done Sent false
10556 3991 Acked Read 5 Done Done Sent false
10557 3992 Acked Read 9 Done Done Sent false
10558 3993 Acked Write 6 Done Done Sent false
10559 3994 Acked Write 1 Done Done Sent false
10560 3995 Acked Read 2 Done Done Sent false
10561 3996 Acked Read 7 Done Done Sent false
10562 3997 Acked Read 4 Done Done Sent false
10563 3998 Acked Read 1 Done Done Sent false
10564 3999 Acked Write 9 Done Done Sent false
10565 4000 Acked Read 8 Done Done Sent false
10566 4001 Acked Read 6 Done Done Sent false
10567 4002 Acked Flush 0 Done Done Sent false
10568 4003 Acked Flush 0 Done Done Sent false
10569 4004 Acked Write 7 Done Done Sent false
10570 4005 Acked Read 6 Done Done Sent false
10571 4006 Acked Read 4 Done Done Sent false
10572 4007 Acked Read 3 Done Done Sent false
10573 4008 Acked Write 3 Done Done Sent false
10574 4009 Acked Read 2 Done Done Sent false
10575 4010 Acked Flush 0 Done Done Sent false
10576 4011 Acked Write 2 Done Done Sent false
10577 4012 Acked Read 4 Done Done Sent false
10578 4013 Acked Read 5 Done Done Sent false
10579 4014 Acked Flush 0 Done Done Sent false
10580 4015 Acked Read 9 Done Done Sent false
10581 4016 Acked Read 3 Done Done Sent false
10582 4017 Acked Read 1 Done Done Sent false
10583 4018 Acked Read 4 Done Done Sent false
10584 4019 Acked Flush 0 Done Done Sent false
10585 STATES DS:0 DS:1 DS:2 TOTAL
10586 Sent 0 0 302 302
10587 Done 302 302 0 604
10588 Skipped 0 0 0 0
10589 Error 0 0 0 0
10590 Last Flush: 4019 4019 3717
10591 Downstairs last five completed: 3717 3716 3715 3714 3713
10592 Upstairs last five completed: 4019 4018 4017 4016 4015
10593 CLIENT: Up:0 ds:302 act:2
105942024-12-09T15:10:03.327ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
105952024-12-09T15:10:03.327ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60970) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
105962024-12-09T15:10:03.328ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
105972024-12-09T15:10:03.329ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
105982024-12-09T15:10:03.329ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
105992024-12-09T15:10:03.329ZINFOcrucible: send last flush ID to this DS: 3717 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
106002024-12-09T15:10:03.329ZINFOcrucible: Replied this last flush ID: 3717 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
106012024-12-09T15:10:03.329ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
106022024-12-09T15:10:03.329ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
106032024-12-09T15:10:03.329ZINFOcrucible: [2] client re-new 302 jobs since flush 3717 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
106042024-12-09T15:10:03.329ZINFOcrucible: [2] Replayed 302 jobs since flush: 3717 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
10605 ----------------------------------------------------------------
10606 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10607 Downstairs last five completed: 4019 4018 4017 4016 4015
10608 Upstairs last five completed: 4019 4018 4017 4016 4015
10609 CLIENT: Up:0 ds:0 act:3
10610 Replay: All jobs finished
10611 Main task has work to do, go find it
10612 got dsc Stop(0)
10613 stop 0
10614 [8810] Got stop action so:false kr:false
10615 Dec 09 15:10:05.577 INFO request completed, latency_us: 340, response_code: 204, uri: /stop/cid/0, method: POST, req_id: c0960252-ac8b-4134-9200-c873e1fabba3, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
10616 Dec 09 15:10:05.578 INFO request completed, latency_us: 111, response_code: 200, uri: /state/cid/0, method: GET, req_id: f1be8ac7-fae3-431f-b42c-d47bf7154972, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
106172024-12-09T15:10:05.578ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
106182024-12-09T15:10:05.578ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
106192024-12-09T15:10:05.578ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
106202024-12-09T15:10:05.578ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
106212024-12-09T15:10:05.578ZINFOcrucible: [0] Transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
106222024-12-09T15:10:05.578ZINFOcrucible: client task is exiting = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
10623 [8810] Exited with: ExitStatus(unix_wait_status(9))
10624 [8810][0] reports Exit
10625 Dec 09 15:10:09.581 INFO request completed, latency_us: 230, response_code: 200, uri: /state/cid/0, method: GET, req_id: 90043153-aa12-496a-92cb-3f3a0e727cac, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
10626 001/300 Read block 3292 len 16384
10627 002/300 Write block 2052 len 32768 data: 2 2 2 2 2 2 3 3
10628 003/300 Read block 60 len 20480
10629 004/300 Read block 1392 len 20480
10630 005/300 Write block 3339 len 36864 data: 5 5 6 6 6 6 3 4 4
10631 006/300 Flush
10632 007/300 Read block 529 len 16384
10633 008/300 Flush
10634 009/300 Write block 4192 len 4096 data: 4
10635 010/300 Read block 2835 len 12288
10636 011/300 Read block 4426 len 36864
10637 012/300 Read block 859 len 8192
10638 013/300 Write block 4781 len 40960 data: 4 4 4 5 4 3 3 2 2 2
10639 014/300 Write block 4707 len 12288 data: 4 4 4
10640 015/300 Read block 1094 len 20480
10641 016/300 Write block 4046 len 32768 data: 3 3 3 3 3 4 4 3
10642 017/300 Write block 3394 len 4096 data: 5
10643 018/300 Read block 1155 len 12288
10644 019/300 Read block 4907 len 16384
10645 020/300 Read block 887 len 16384
10646 021/300 Read block 184 len 8192
10647 022/300 Read block 4100 len 12288
10648 023/300 Read block 3689 len 8192
10649 024/300 Read block 2972 len 36864
10650 025/300 Read block 4351 len 28672
10651 026/300 Read block 2242 len 28672
10652 027/300 Read block 3035 len 40960
10653 028/300 Write block 4785 len 16384 data: 5 4 4 3
10654 029/300 Write block 1506 len 24576 data: 2 2 4 4 3 3
10655 030/300 Read block 4828 len 8192
10656 031/300 Read block 2814 len 40960
10657 032/300 Flush
10658 033/300 Read block 2685 len 32768
10659 034/300 Write block 1560 len 24576 data: 6 8 8 8 7 5
10660 035/300 Read block 3997 len 8192
10661 036/300 Write block 813 len 36864 data: 5 4 4 4 4 4 4 4 4
10662 037/300 Flush
10663 038/300 Write block 1126 len 8192 data: 3 3
10664 039/300 Write block 2419 len 36864 data: 3 3 3 3 3 3 3 2 2
10665 040/300 Write block 2440 len 40960 data: 4 5 5 6 6 5 4 4 4 3
10666 041/300 Flush
10667 042/300 Write block 2409 len 32768 data: 2 2 2 2 2 2 3 3
10668 043/300 Read block 2112 len 32768
10669 044/300 Write block 2045 len 36864 data: 5 3 3 2 2 2 2 3 3
10670 045/300 Flush
10671 046/300 Write block 1605 len 32768 data: 2 2 3 4 5 5 5 6
10672 047/300 Flush
10673 048/300 Write block 2672 len 40960 data: 4 3 3 2 2 2 2 2 2 2
10674 049/300 Read block 649 len 8192
10675 050/300 Flush
10676 051/300 Write block 1125 len 8192 data: 2 4
10677 052/300 Flush
10678 053/300 Flush
10679 054/300 Read block 4991 len 28672
10680 055/300 Flush
10681 056/300 Write block 3726 len 20480 data: 3 3 3 3 2
10682 057/300 Write block 3917 len 8192 data: 2 2
10683 058/300 Write block 2984 len 40960 data: 4 6 7 8 7 6 5 5 5 4
10684 059/300 Write block 2654 len 32768 data: 5 5 5 3 3 3 3 3
10685 060/300 Write block 2388 len 28672 data: 4 4 4 3 3 4 4
10686 061/300 Read block 748 len 16384
10687 062/300 Write block 1123 len 28672 data: 2 2 3 5 4 3 3
10688 063/300 Read block 2836 len 36864
10689 064/300 Read block 2417 len 40960
10690 065/300 Write block 1838 len 36864 data: 3 3 6 6 6 4 2 2 3
10691 066/300 Read block 1342 len 16384
10692 067/300 Read block 4409 len 36864
10693 068/300 Write block 1290 len 16384 data: 3 5 5 5
10694 069/300 Read block 1876 len 32768
10695 070/300 Write block 618 len 40960 data: 2 2 2 2 2 2 2 2 2 4
10696 071/300 Write block 4838 len 24576 data: 4 5 6 6 5 5
10697 072/300 Read block 2273 len 4096
10698 073/300 Write block 4807 len 20480 data: 2 2 2 2 3
10699 074/300 Read block 1629 len 32768
10700 075/300 Write block 2544 len 24576 data: 2 2 2 2 4 4
10701 076/300 Write block 2347 len 36864 data: 3 3 3 3 3 3 4 4 4
10702 077/300 Read block 4871 len 4096
10703 078/300 Read block 648 len 28672
10704 079/300 Write block 1929 len 32768 data: 5 5 4 4 4 3 2 2
10705 080/300 Read block 2425 len 4096
10706 081/300 Write block 4874 len 16384 data: 3 2 2 2
10707 082/300 Write block 1716 len 28672 data: 4 3 3 2 2 2 4
10708 083/300 Read block 1668 len 40960
10709 084/300 Write block 2623 len 24576 data: 5 4 5 3 3 3
10710 085/300 Write block 4914 len 4096 data: 5
10711 086/300 Write block 2309 len 16384 data: 2 2 2 3
10712 087/300 Read block 121 len 28672
10713 088/300 Write block 346 len 32768 data: 3 3 2 2 2 2 2 2
10714 089/300 Read block 1885 len 20480
10715 090/300 Write block 227 len 4096 data: 4
10716 091/300 Write block 4549 len 36864 data: 3 4 4 4 4 4 4 5 4
10717 092/300 Read block 4477 len 24576
10718 093/300 Write block 98 len 12288 data: 4 3 4
10719 094/300 Write block 1408 len 32768 data: 5 5 5 5 5 5 5 5
10720 095/300 Read block 4082 len 16384
10721 096/300 Read block 330 len 12288
10722 097/300 Flush
10723 098/300 Read block 2717 len 8192
10724 099/300 Read block 2901 len 32768
10725 100/300 Write block 1311 len 20480 data: 4 5 5 5 5
10726 101/300 Flush
10727 102/300 Write block 132 len 20480 data: 4 6 5 5 5
10728 103/300 Read block 4005 len 20480
10729 104/300 Write block 4310 len 20480 data: 2 2 3 3 3
10730 105/300 Read block 1681 len 4096
10731 106/300 Flush
10732 107/300 Read block 3972 len 12288
10733 108/300 Read block 1166 len 24576
10734 109/300 Flush
10735 110/300 Flush
10736 111/300 Read block 2598 len 24576
10737 112/300 Write block 156 len 24576 data: 6 5 4 4 3 3
10738 113/300 Write block 2391 len 24576 data: 4 4 5 5 4 4
10739 114/300 Flush
10740 115/300 Read block 503 len 28672
10741 116/300 Read block 2109 len 28672
10742 117/300 Read block 2589 len 20480
10743 118/300 Write block 1524 len 12288 data: 2 2 3
10744 119/300 Write block 680 len 24576 data: 3 3 2 4 4 4
10745 120/300 Write block 1264 len 40960 data: 2 2 2 2 3 3 4 4 4 4
10746 121/300 Read block 163 len 8192
10747 122/300 Write block 1864 len 12288 data: 2 4 5
10748 123/300 Write block 3378 len 20480 data: 2 2 2 2 3
10749 124/300 Write block 2773 len 32768 data: 6 6 5 4 4 4 4 4
10750 125/300 Write block 2655 len 36864 data: 6 6 4 4 4 4 4 2 2
10751 126/300 Write block 1417 len 8192 data: 5 4
10752 127/300 Read block 1783 len 12288
10753 128/300 Read block 1557 len 4096
10754 129/300 Write block 3277 len 32768 data: 5 4 3 3 3 3 3 3
10755 130/300 Flush
10756 131/300 Read block 4404 len 8192
10757 132/300 Flush
10758 133/300 Write block 3295 len 32768 data: 2 2 2 2 2 2 3 3
10759 134/300 Read block 3209 len 40960
10760 135/300 Read block 2568 len 12288
10761 136/300 Write block 3565 len 40960 data: 5 5 5 5 3 2 2 2 2 2
10762 137/300 Flush
10763 138/300 Write block 1981 len 20480 data: 5 5 5 5 5
10764 139/300 Write block 3756 len 36864 data: 2 2 3 3 3 3 4 4 4
10765 140/300 Write block 2396 len 40960 data: 5 4 4 4 4 5 4 2 2 2
10766 141/300 Flush
10767 142/300 Read block 3075 len 36864
10768 143/300 Write block 1207 len 28672 data: 4 3 3 3 4 4 4
10769 144/300 Read block 3108 len 40960
10770 145/300 Read block 3773 len 16384
10771 146/300 Read block 4014 len 20480
10772 147/300 Write block 4783 len 12288 data: 5 6 6
10773 148/300 Read block 2054 len 20480
10774 149/300 Write block 1207 len 4096 data: 5
10775 150/300 Read block 4673 len 24576
10776 151/300 Read block 3220 len 20480
10777 152/300 Read block 2584 len 40960
10778 153/300 Write block 3524 len 12288 data: 4 5 4
10779 154/300 Write block 3808 len 32768 data: 4 3 3 2 2 2 2 2
10780 155/300 Read block 3475 len 36864
10781 156/300 Read block 3071 len 36864
10782 157/300 Write block 4486 len 36864 data: 3 3 3 3 3 3 3 4 3
10783 158/300 Write block 3453 len 16384 data: 5 5 5 4
10784 159/300 Read block 4842 len 20480
10785 160/300 Flush
10786 161/300 Write block 4383 len 20480 data: 4 3 3 3 4
10787 162/300 Read block 4598 len 28672
10788 163/300 Write block 2396 len 4096 data: 6
10789 164/300 Write block 4748 len 36864 data: 4 4 4 4 3 3 3 3 3
10790 165/300 Flush
10791 166/300 Write block 1194 len 40960 data: 2 2 2 3 5 5 5 5 4 4
10792 167/300 Write block 4866 len 4096 data: 4
10793 168/300 Write block 3822 len 32768 data: 4 5 5 4 4 4 4 5
10794 169/300 Flush
10795 170/300 Read block 2249 len 32768
10796 171/300 Write block 2011 len 24576 data: 4 3 4 5 5 6
10797 172/300 Read block 2114 len 12288
10798 173/300 Read block 3139 len 32768
10799 174/300 Read block 1051 len 28672
10800 175/300 Write block 1820 len 12288 data: 3 3 4
10801 176/300 Read block 3828 len 32768
10802 177/300 Write block 4832 len 28672 data: 3 2 3 4 4 4 5
10803 178/300 Read block 4380 len 12288
10804 179/300 Read block 3433 len 8192
10805 180/300 Read block 3870 len 20480
10806 181/300 Flush
10807 182/300 Read block 1437 len 28672
10808 183/300 Write block 889 len 8192 data: 2 2
10809 184/300 Write block 1252 len 40960 data: 2 2 2 2 2 2 2 2 2 2
10810 185/300 Read block 4345 len 16384
10811 186/300 Write block 3823 len 24576 data: 6 6 5 5 5 5
10812 187/300 Write block 4893 len 12288 data: 3 3 2
10813 188/300 Read block 4946 len 8192
10814 189/300 Read block 1142 len 4096
10815 190/300 Write block 6 len 32768 data: 2 2 2 2 3 3 2 2
10816 191/300 Read block 461 len 8192
10817 192/300 Read block 2629 len 28672
10818 193/300 Write block 4096 len 40960 data: 6 6 6 5 3 3 2 2 2 2
10819 194/300 Read block 601 len 28672
10820 195/300 Flush
10821 196/300 Read block 3376 len 4096
10822 197/300 Read block 1761 len 36864
10823 198/300 Write block 3290 len 24576 data: 5 4 4 3 3 3
10824 199/300 Read block 3448 len 28672
10825 200/300 Write block 2866 len 24576 data: 4 6 6 6 5 5
10826 201/300 Write block 4370 len 20480 data: 2 2 2 2 3
10827 202/300 Flush
10828 203/300 Write block 4944 len 32768 data: 2 3 4 3 3 4 4 3
10829 204/300 Flush
10830 205/300 Read block 3969 len 20480
10831 206/300 Read block 3592 len 36864
10832 207/300 Read block 4624 len 20480
10833 208/300 Flush
10834 209/300 Flush
10835 210/300 Read block 1615 len 16384
10836 211/300 Read block 3908 len 20480
10837 212/300 Read block 3275 len 28672
10838 213/300 Read block 4487 len 12288
10839 214/300 Read block 2055 len 24576
10840 215/300 Read block 184 len 12288
10841 216/300 Read block 2909 len 24576
10842 217/300 Write block 1005 len 8192 data: 2 3
10843 218/300 Write block 2148 len 16384 data: 4 3 3 3
10844 219/300 Write block 4246 len 12288 data: 4 4 2
10845 220/300 Read block 1746 len 28672
10846 221/300 Read block 1771 len 16384
10847 222/300 Read block 674 len 36864
10848 223/300 Write block 3525 len 8192 data: 6 5
10849 224/300 Write block 1094 len 4096 data: 5
10850 225/300 Read block 67 len 28672
10851 226/300 Write block 2625 len 8192 data: 6 4
10852 227/300 Read block 2045 len 40960
10853 228/300 Flush
10854 229/300 Read block 3603 len 32768
10855 230/300 Read block 201 len 40960
10856 231/300 Read block 2140 len 16384
10857 232/300 Write block 4524 len 20480 data: 3 3 3 4 4
10858 233/300 Read block 3046 len 24576
10859 234/300 Write block 3179 len 28672 data: 4 4 4 4 3 3 2
10860 235/300 Write block 1855 len 32768 data: 5 4 3 3 3 3 3 3
10861 236/300 Write block 1709 len 36864 data: 3 4 4 3 4 4 4 5 4
10862 237/300 Read block 3716 len 24576
10863 238/300 Read block 2836 len 16384
10864 239/300 Read block 4315 len 36864
10865 240/300 Flush
10866 241/300 Write block 356 len 32768 data: 3 3 3 3 4 4 4 4
10867 242/300 Write block 4731 len 8192 data: 2 2
10868 243/300 Write block 4307 len 28672 data: 2 2 2 3 3 4 4
10869 244/300 Write block 401 len 40960 data: 3 3 3 4 4 4 3 3 3 5
10870 245/300 Read block 1592 len 8192
10871 246/300 Read block 1564 len 28672
10872 247/300 Write block 3272 len 16384 data: 3 4 5 5
10873 248/300 Write block 4545 len 20480 data: 2 2 3 2 4
10874 249/300 Read block 131 len 20480
10875 250/300 Flush
10876 251/300 Read block 4777 len 36864
10877 252/300 Read block 2668 len 24576
10878 253/300 Read block 3211 len 40960
10879 254/300 Write block 1060 len 12288 data: 3 3 3
10880 255/300 Read block 4108 len 32768
10881 256/300 Write block 4808 len 12288 data: 3 3 3
10882 257/300 Read block 2854 len 40960
10883 258/300 Read block 919 len 16384
10884 259/300 Read block 1746 len 12288
10885 260/300 Read block 2467 len 32768
10886 261/300 Read block 1000 len 4096
10887 262/300 Flush
10888 263/300 Write block 3974 len 16384 data: 5 5 5 5
10889 264/300 Flush
10890 265/300 Read block 2151 len 36864
10891 266/300 Write block 4813 len 8192 data: 4 5
10892 267/300 Read block 345 len 4096
10893 268/300 Read block 2169 len 16384
10894 269/300 Read block 4941 len 32768
10895 270/300 Write block 213 len 24576 data: 6 6 5 5 5 5
10896 271/300 Read block 3176 len 16384
10897 272/300 Write block 1213 len 12288 data: 5 4 3
10898 273/300 Write block 501 len 40960 data: 2 2 2 2 2 2 3 3 3 3
10899 274/300 Write block 4646 len 20480 data: 4 5 5 5 7
10900 275/300 Read block 1786 len 24576
10901 276/300 Write block 865 len 40960 data: 3 3 3 3 3 3 3 2 2 2
10902 277/300 Read block 142 len 8192
10903 278/300 Write block 903 len 36864 data: 5 5 3 3 2 2 3 2 2
10904 279/300 Write block 3868 len 4096 data: 2
10905 280/300 Read block 2515 len 4096
10906 281/300 Write block 357 len 24576 data: 4 4 4 5 5 5
10907 282/300 Write block 1073 len 28672 data: 2 2 2 4 4 4 5
10908 283/300 Read block 2258 len 12288
10909 284/300 Write block 2774 len 40960 data: 7 6 5 5 5 5 5 3 3 3
10910 285/300 Read block 159 len 8192
10911 286/300 Read block 398 len 28672
10912 287/300 Read block 2779 len 36864
10913 288/300 Write block 3019 len 32768 data: 2 2 2 3 4 5 6 5
10914 289/300 Write block 1795 len 12288 data: 5 5 4
10915 290/300 Read block 2193 len 40960
10916 291/300 Write block 2541 len 8192 data: 5 3
10917 292/300 Read block 4797 len 32768
10918 293/300 Write block 3619 len 16384 data: 5 5 5 4
10919 294/300 Flush
10920 295/300 Write block 2709 len 20480 data: 2 2 5 5 5
10921 296/300 Write block 2696 len 20480 data: 2 2 3 3 3
10922 297/300 Flush
10923 298/300 Read block 4719 len 12288
10924 299/300 Read block 3593 len 4096
10925 300/300 Read block 4394 len 16384
10926 301/300 Read block 837 len 40960
10927 Main task has work to do, go find it
10928 got dsc Start(0)
10929 start 0
10930 Dec 09 15:10:09.819 INFO request completed, latency_us: 229, response_code: 204, uri: /start/cid/0, method: POST, req_id: 6607bc64-b206-4778-b8a6-9a549ec533a9, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
10931 Replay: started 0, returned:Ok(())
10932 ----------------------------------------------------------------
10933 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10934 States: Offline Active Active
10935 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10936 4020 Acked Read 4 Sent Done Done false
10937 4021 Acked Write 8 Sent Done Done false
10938 4022 Acked Read 5 Sent Done Done false
10939 4023 Acked Read 5 Sent Done Done false
10940 4024 Acked Write 9 Sent Done Done false
10941 4025 Acked Flush 0 Sent Done Done false
10942 4026 Acked Read 4 Sent Done Done false
10943 4027 Acked Flush 0 Sent Done Done false
10944 4028 Acked Write 1 Sent Done Done false
10945 4029 Acked Read 3 Sent Done Done false
10946 4030 Acked Read 9 Sent Done Done false
10947 4031 Acked Read 2 Sent Done Done false
10948 4032 Acked Write 10 Sent Done Done false
10949 4033 Acked Write 3 Sent Done Done false
10950 4034 Acked Read 5 Sent Done Done false
10951 4035 Acked Write 8 Sent Done Done false
10952 4036 Acked Write 1 Sent Done Done false
10953 4037 Acked Read 3 Sent Done Done false
10954 4038 Acked Read 4 Sent Done Done false
10955 4039 Acked Read 4 Sent Done Done false
10956 4040 Acked Read 2 Sent Done Done false
10957 4041 Acked Read 3 Sent Done Done false
10958 4042 Acked Read 2 Sent Done Done false
10959 4043 Acked Read 9 Sent Done Done false
10960 4044 Acked Read 7 Sent Done Done false
10961 4045 Acked Read 7 Sent Done Done false
10962 4046 Acked Read 10 Sent Done Done false
10963 4047 Acked Write 4 Sent Done Done false
10964 4048 Acked Write 6 Sent Done Done false
10965 4049 Acked Read 2 Sent Done Done false
10966 4050 Acked Read 10 Sent Done Done false
10967 4051 Acked Flush 0 Sent Done Done false
10968 4052 Acked Read 8 Sent Done Done false
10969 4053 Acked Write 6 Sent Done Done false
10970 4054 Acked Read 2 Sent Done Done false
10971 4055 Acked Write 9 Sent Done Done false
10972 4056 Acked Flush 0 Sent Done Done false
10973 4057 Acked Write 2 Sent Done Done false
10974 4058 Acked Write 9 Sent Done Done false
10975 4059 Acked Write 10 Sent Done Done false
10976 4060 Acked Flush 0 Sent Done Done false
10977 4061 Acked Write 8 Sent Done Done false
10978 4062 Acked Read 8 Sent Done Done false
10979 4063 Acked Write 9 Sent Done Done false
10980 4064 Acked Flush 0 Sent Done Done false
10981 4065 Acked Write 8 Sent Done Done false
10982 4066 Acked Flush 0 Sent Done Done false
10983 4067 Acked Write 10 Sent Done Done false
10984 4068 Acked Read 2 Sent Done Done false
10985 4069 Acked Flush 0 Sent Done Done false
10986 4070 Acked Write 2 Sent Done Done false
10987 4071 Acked Flush 0 Sent Done Done false
10988 4072 Acked Flush 0 Sent Done Done false
10989 4073 Acked Read 7 Sent Done Done false
10990 4074 Acked Flush 0 Sent Done Done false
10991 4075 Acked Write 5 Sent Done Done false
10992 4076 Acked Write 2 Sent Done Done false
10993 4077 Acked Write 10 Sent Done Done false
10994 4078 Acked Write 8 Sent Done Done false
10995 4079 Acked Write 7 Sent Done Done false
10996 4080 Acked Read 4 Sent Done Done false
10997 4081 Acked Write 7 Sent Done Done false
10998 4082 Acked Read 9 Sent Done Done false
10999 4083 Acked Read 10 Sent Done Done false
11000 4084 Acked Write 9 Sent Done Done false
11001 4085 Acked Read 4 Sent Done Done false
11002 4086 Acked Read 9 Sent Done Done false
11003 4087 Acked Write 4 Sent Done Done false
11004 4088 Acked Read 8 Sent Done Done false
11005 4089 Acked Write 10 Sent Done Done false
11006 4090 Acked Write 6 Sent Done Done false
11007 4091 Acked Read 1 Sent Done Done false
11008 4092 Acked Write 5 Sent Done Done false
11009 4093 Acked Read 8 Sent Done Done false
11010 4094 Acked Write 6 Sent Done Done false
11011 4095 Acked Write 9 Sent Done Done false
11012 4096 Acked Read 1 Sent Done Done false
11013 4097 Acked Read 7 Sent Done Done false
11014 4098 Acked Write 8 Sent Done Done false
11015 4099 Acked Read 1 Sent Done Done false
11016 4100 Acked Write 4 Sent Done Done false
11017 4101 Acked Write 7 Sent Done Done false
11018 4102 Acked Read 10 Sent Done Done false
11019 4103 Acked Write 6 Sent Done Done false
11020 4104 Acked Write 1 Sent Done Done false
11021 4105 Acked Write 4 Sent Done Done false
11022 4106 Acked Read 7 Sent Done Done false
11023 4107 Acked Write 8 Sent Done Done false
11024 4108 Acked Read 5 Sent Done Done false
11025 4109 Acked Write 1 Sent Done Done false
11026 4110 Acked Write 9 Sent Done Done false
11027 4111 Acked Read 6 Sent Done Done false
11028 4112 Acked Write 3 Sent Done Done false
11029 4113 Acked Write 8 Sent Done Done false
11030 4114 Acked Read 4 Sent Done Done false
11031 4115 Acked Read 3 Sent Done Done false
11032 4116 Acked Flush 0 Sent Done Done false
11033 4117 Acked Read 2 Sent Done Done false
11034 4118 Acked Read 8 Sent Done Done false
11035 4119 Acked Write 5 Sent Done Done false
11036 4120 Acked Flush 0 Sent Done Done false
11037 4121 Acked Write 5 Sent Done Done false
11038 4122 Acked Read 5 Sent Done Done false
11039 4123 Acked Write 5 Sent Done Done false
11040 4124 Acked Read 1 Sent Done Done false
11041 4125 Acked Flush 0 Sent Done Done false
11042 4126 Acked Read 3 Sent Done Done false
11043 4127 Acked Read 6 Sent Done Done false
11044 4128 Acked Flush 0 Sent Done Done false
11045 4129 Acked Flush 0 Sent Done Done false
11046 4130 Acked Read 6 Sent Done Done false
11047 4131 Acked Write 6 Sent Done Done false
11048 4132 Acked Write 6 Sent Done Done false
11049 4133 Acked Flush 0 Sent Done Done false
11050 4134 Acked Read 7 Sent Done Done false
11051 4135 Acked Read 7 Sent Done Done false
11052 4136 Acked Read 5 Sent Done Done false
11053 4137 Acked Write 3 Sent Done Done false
11054 4138 Acked Write 6 Sent Done Done false
11055 4139 Acked Write 10 Sent Done Done false
11056 4140 Acked Read 2 Sent Done Done false
11057 4141 Acked Write 3 Sent Done Done false
11058 4142 Acked Write 5 Sent Done Done false
11059 4143 Acked Write 8 Sent Done Done false
11060 4144 Acked Write 9 Sent Done Done false
11061 4145 Acked Write 2 Sent Done Done false
11062 4146 Acked Read 3 Sent Done Done false
11063 4147 Acked Read 1 Sent Done Done false
11064 4148 Acked Write 8 Sent Done Done false
11065 4149 Acked Flush 0 Sent Done Done false
11066 4150 Acked Read 2 Sent Done Done false
11067 4151 Acked Flush 0 Sent Done Done false
11068 4152 Acked Write 8 Sent Done Done false
11069 4153 Acked Read 10 Sent Done Done false
11070 4154 Acked Read 3 Sent Done Done false
11071 4155 Acked Write 10 Sent Done Done false
11072 4156 Acked Flush 0 Sent Done Done false
11073 4157 Acked Write 5 Sent Done Done false
11074 4158 Acked Write 9 Sent Done Done false
11075 4159 Acked Write 10 Sent Done Done false
11076 4160 Acked Flush 0 Sent Done Done false
11077 4161 Acked Read 9 Sent Done Done false
11078 4162 Acked Write 7 Sent Done Done false
11079 4163 Acked Read 10 Sent Done Done false
11080 4164 Acked Read 4 Sent Done Done false
11081 4165 Acked Read 5 Sent Done Done false
11082 4166 Acked Write 3 Sent Done Done false
11083 4167 Acked Read 5 Sent Done Done false
11084 4168 Acked Write 1 Sent Done Done false
11085 4169 Acked Read 6 Sent Done Done false
11086 4170 Acked Read 5 Sent Done Done false
11087 4171 Acked Read 10 Sent Done Done false
11088 4172 Acked Write 3 Sent Done Done false
11089 4173 Acked Write 8 Sent Done Done false
11090 4174 Acked Read 9 Sent Done Done false
11091 4175 Acked Read 9 Sent Done Done false
11092 4176 Acked Write 9 Sent Done Done false
11093 4177 Acked Write 4 Sent Done Done false
11094 4178 Acked Read 5 Sent Done Done false
11095 4179 Acked Flush 0 Sent Done Done false
11096 4180 Acked Write 5 Sent Done Done false
11097 4181 Acked Read 7 Sent Done Done false
11098 4182 Acked Write 1 Sent Done Done false
11099 4183 Acked Write 9 Sent Done Done false
11100 4184 Acked Flush 0 Sent Done Done false
11101 4185 Acked Write 10 Sent Done Done false
11102 4186 Acked Write 1 Sent Done Done false
11103 4187 Acked Write 8 Sent Done Done false
11104 4188 Acked Flush 0 Sent Done Done false
11105 4189 Acked Read 8 Sent Done Done false
11106 4190 Acked Write 6 Sent Done Done false
11107 4191 Acked Read 3 Sent Done Done false
11108 4192 Acked Read 8 Sent Done Done false
11109 4193 Acked Read 7 Sent Done Done false
11110 4194 Acked Write 3 Sent Done Done false
11111 4195 Acked Read 8 Sent Done Done false
11112 4196 Acked Write 7 Sent Done Done false
11113 4197 Acked Read 3 Sent Done Done false
11114 4198 Acked Read 2 Sent Done Done false
11115 4199 Acked Read 5 Sent Done Done false
11116 4200 Acked Flush 0 Sent Done Done false
11117 4201 Acked Read 7 Sent Done Done false
11118 4202 Acked Write 2 Sent Done Done false
11119 4203 Acked Write 10 Sent Done Done false
11120 4204 Acked Read 4 Sent Done Done false
11121 4205 Acked Write 6 Sent Done Done false
11122 4206 Acked Write 3 Sent Done Done false
11123 4207 Acked Read 2 Sent Done Done false
11124 4208 Acked Read 1 Sent Done Done false
11125 4209 Acked Write 8 Sent Done Done false
11126 4210 Acked Read 2 Sent Done Done false
11127 4211 Acked Read 7 Sent Done Done false
11128 4212 Acked Write 10 Sent Done Done false
11129 4213 Acked Read 7 Sent Done Done false
11130 4214 Acked Flush 0 Sent Done Done false
11131 4215 Acked Read 1 Sent Done Done false
11132 4216 Acked Read 9 Sent Done Done false
11133 4217 Acked Write 6 Sent Done Done false
11134 4218 Acked Read 7 Sent Done Done false
11135 4219 Acked Write 6 Sent Done Done false
11136 4220 Acked Write 5 Sent Done Done false
11137 4221 Acked Flush 0 Sent Done Done false
11138 4222 Acked Write 8 Sent Done Done false
11139 4223 Acked Flush 0 Sent Done Done false
11140 4224 Acked Read 5 Sent Done Done false
11141 4225 Acked Read 9 Sent Done Done false
11142 4226 Acked Read 5 Sent Done Done false
11143 4227 Acked Flush 0 Sent Done Done false
11144 4228 Acked Flush 0 Sent Done Done false
11145 4229 Acked Read 4 Sent Done Done false
11146 4230 Acked Read 5 Sent Done Done false
11147 4231 Acked Read 7 Sent Done Done false
11148 4232 Acked Read 3 Sent Done Done false
11149 4233 Acked Read 6 Sent Done Done false
11150 4234 Acked Read 3 Sent Done Done false
11151 4235 Acked Read 6 Sent Done Done false
11152 4236 Acked Write 2 Sent Done Done false
11153 4237 Acked Write 4 Sent Done Done false
11154 4238 Acked Write 3 Sent Done Done false
11155 4239 Acked Read 7 Sent Done Done false
11156 4240 Acked Read 4 Sent Done Done false
11157 4241 Acked Read 9 Sent Done Done false
11158 4242 Acked Write 2 Sent Done Done false
11159 4243 Acked Write 1 Sent Done Done false
11160 4244 Acked Read 7 Sent Done Done false
11161 4245 Acked Write 2 Sent Done Done false
11162 4246 Acked Read 10 Sent Done Done false
11163 4247 Acked Flush 0 Sent Done Done false
11164 4248 Acked Read 8 Sent Done Done false
11165 4249 Acked Read 10 Sent Done Done false
11166 4250 Acked Read 4 Sent Done Done false
11167 4251 Acked Write 5 Sent Done Done false
11168 4252 Acked Read 6 Sent Done Done false
11169 4253 Acked Write 7 Sent Done Done false
11170 4254 Acked Write 8 Sent Done Done false
11171 4255 Acked Write 9 Sent Done Done false
11172 4256 Acked Read 6 Sent Done Done false
11173 4257 Acked Read 4 Sent Done Done false
11174 4258 Acked Read 9 Sent Done Done false
11175 4259 Acked Flush 0 Sent Done Done false
11176 4260 Acked Write 8 Sent Done Done false
11177 4261 Acked Write 2 Sent Done Done false
11178 4262 Acked Write 7 Sent Done Done false
11179 4263 Acked Write 10 Sent Done Done false
11180 4264 Acked Read 2 Sent Done Done false
11181 4265 Acked Read 7 Sent Done Done false
11182 4266 Acked Write 4 Sent Done Done false
11183 4267 Acked Write 5 Sent Done Done false
11184 4268 Acked Read 5 Sent Done Done false
11185 4269 Acked Flush 0 Sent Done Done false
11186 4270 Acked Read 9 Sent Done Done false
11187 4271 Acked Read 6 Sent Done Done false
11188 4272 Acked Read 10 Sent Done Done false
11189 4273 Acked Write 3 Sent Done Done false
11190 4274 Acked Read 8 Sent Done Done false
11191 4275 Acked Write 3 Sent Done Done false
11192 4276 Acked Read 10 Sent Done Done false
11193 4277 Acked Read 4 Sent Done Done false
11194 4278 Acked Read 3 Sent Done Done false
11195 4279 Acked Read 8 Sent Done Done false
11196 4280 Acked Read 1 Sent Done Done false
11197 4281 Acked Flush 0 Sent Done Done false
11198 4282 Acked Write 4 Sent Done Done false
11199 4283 Acked Flush 0 Sent Done Done false
11200 4284 Acked Read 9 Sent Done Done false
11201 4285 Acked Write 2 Sent Done Done false
11202 4286 Acked Read 1 Sent Done Done false
11203 4287 Acked Read 4 Sent Done Done false
11204 4288 Acked Read 8 Sent Done Done false
11205 4289 Acked Write 6 Sent Done Done false
11206 4290 Acked Read 4 Sent Done Done false
11207 4291 Acked Write 3 Sent Done Done false
11208 4292 Acked Write 10 Sent Done Done false
11209 4293 Acked Write 5 Sent Done Done false
11210 4294 Acked Read 6 Sent Done Done false
11211 4295 Acked Write 10 Sent Done Done false
11212 4296 Acked Read 2 Sent Done Done false
11213 4297 Acked Write 9 Sent Done Done false
11214 4298 Acked Write 1 Sent Done Done false
11215 4299 Acked Read 1 Sent Done Done false
11216 4300 Acked Write 6 Sent Done Done false
11217 4301 Acked Write 7 Sent Done Done false
11218 4302 Acked Read 3 Sent Done Done false
11219 4303 Acked Write 10 Sent Done Done false
11220 4304 Acked Read 2 Sent Done Done false
11221 4305 Acked Read 7 Sent Done Done false
11222 4306 Acked Read 9 Sent Done Done false
11223 4307 Acked Write 8 Sent Done Done false
11224 4308 Acked Write 3 Sent Done Done false
11225 4309 Acked Read 10 Sent Done Done false
11226 4310 Acked Write 2 Sent Done Done false
11227 4311 Acked Read 8 Sent Done Done false
11228 4312 Acked Write 4 Sent Done Done false
11229 4313 Acked Flush 0 Sent Done Done false
11230 4314 Acked Write 5 Sent Done Done false
11231 4315 Acked Write 5 Sent Done Done false
11232 4316 Acked Flush 0 Sent Done Done false
11233 4317 Acked Read 3 Sent Done Done false
11234 4318 Acked Read 1 Sent Done Done false
11235 4319 Acked Read 4 Sent Done Done false
11236 4320 Acked Read 10 Sent Done Done false
11237 STATES DS:0 DS:1 DS:2 TOTAL
11238 Sent 301 0 0 301
11239 Done 0 301 301 602
11240 Skipped 0 0 0 0
11241 Error 0 0 0 0
11242 Last Flush: 4019 4316 4316
11243 Downstairs last five completed: 4019 4018 4017 4016 4015
11244 Upstairs last five completed: 4320 4319 4318 4317 4316
11245 CLIENT: Up:0 ds:301 act:2
11246 [8810] got start action
11247 [8810] I am going to restart
11248 Starting downstairs at port 8810
11249 Make output file at "/tmp/dsc/downstairs-8810.txt"
11250 [8810][0] reports Starting
11251 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1212)
11252 [8810][0] reports Running
11253 ----------------------------------------------------------------
11254 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
11255 States: Offline Active Active
11256 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11257 4020 Acked Read 4 Sent Done Done false
11258 4021 Acked Write 8 Sent Done Done false
11259 4022 Acked Read 5 Sent Done Done false
11260 4023 Acked Read 5 Sent Done Done false
11261 4024 Acked Write 9 Sent Done Done false
11262 4025 Acked Flush 0 Sent Done Done false
11263 4026 Acked Read 4 Sent Done Done false
11264 4027 Acked Flush 0 Sent Done Done false
11265 4028 Acked Write 1 Sent Done Done false
11266 4029 Acked Read 3 Sent Done Done false
11267 4030 Acked Read 9 Sent Done Done false
11268 4031 Acked Read 2 Sent Done Done false
11269 4032 Acked Write 10 Sent Done Done false
11270 4033 Acked Write 3 Sent Done Done false
11271 4034 Acked Read 5 Sent Done Done false
11272 4035 Acked Write 8 Sent Done Done false
11273 4036 Acked Write 1 Sent Done Done false
11274 4037 Acked Read 3 Sent Done Done false
11275 4038 Acked Read 4 Sent Done Done false
11276 4039 Acked Read 4 Sent Done Done false
11277 4040 Acked Read 2 Sent Done Done false
11278 4041 Acked Read 3 Sent Done Done false
11279 4042 Acked Read 2 Sent Done Done false
11280 4043 Acked Read 9 Sent Done Done false
11281 4044 Acked Read 7 Sent Done Done false
11282 4045 Acked Read 7 Sent Done Done false
11283 4046 Acked Read 10 Sent Done Done false
11284 4047 Acked Write 4 Sent Done Done false
11285 4048 Acked Write 6 Sent Done Done false
11286 4049 Acked Read 2 Sent Done Done false
11287 4050 Acked Read 10 Sent Done Done false
11288 4051 Acked Flush 0 Sent Done Done false
11289 4052 Acked Read 8 Sent Done Done false
11290 4053 Acked Write 6 Sent Done Done false
11291 4054 Acked Read 2 Sent Done Done false
11292 4055 Acked Write 9 Sent Done Done false
11293 4056 Acked Flush 0 Sent Done Done false
11294 4057 Acked Write 2 Sent Done Done false
11295 4058 Acked Write 9 Sent Done Done false
11296 4059 Acked Write 10 Sent Done Done false
11297 4060 Acked Flush 0 Sent Done Done false
11298 4061 Acked Write 8 Sent Done Done false
11299 4062 Acked Read 8 Sent Done Done false
11300 4063 Acked Write 9 Sent Done Done false
11301 4064 Acked Flush 0 Sent Done Done false
11302 4065 Acked Write 8 Sent Done Done false
11303 4066 Acked Flush 0 Sent Done Done false
11304 4067 Acked Write 10 Sent Done Done false
11305 4068 Acked Read 2 Sent Done Done false
11306 4069 Acked Flush 0 Sent Done Done false
11307 4070 Acked Write 2 Sent Done Done false
11308 4071 Acked Flush 0 Sent Done Done false
11309 4072 Acked Flush 0 Sent Done Done false
11310 4073 Acked Read 7 Sent Done Done false
11311 4074 Acked Flush 0 Sent Done Done false
11312 4075 Acked Write 5 Sent Done Done false
11313 4076 Acked Write 2 Sent Done Done false
11314 4077 Acked Write 10 Sent Done Done false
11315 4078 Acked Write 8 Sent Done Done false
11316 4079 Acked Write 7 Sent Done Done false
11317 4080 Acked Read 4 Sent Done Done false
11318 4081 Acked Write 7 Sent Done Done false
11319 4082 Acked Read 9 Sent Done Done false
11320 4083 Acked Read 10 Sent Done Done false
11321 4084 Acked Write 9 Sent Done Done false
11322 4085 Acked Read 4 Sent Done Done false
11323 4086 Acked Read 9 Sent Done Done false
11324 4087 Acked Write 4 Sent Done Done false
11325 4088 Acked Read 8 Sent Done Done false
11326 4089 Acked Write 10 Sent Done Done false
11327 4090 Acked Write 6 Sent Done Done false
11328 4091 Acked Read 1 Sent Done Done false
11329 4092 Acked Write 5 Sent Done Done false
11330 4093 Acked Read 8 Sent Done Done false
11331 4094 Acked Write 6 Sent Done Done false
11332 4095 Acked Write 9 Sent Done Done false
11333 4096 Acked Read 1 Sent Done Done false
11334 4097 Acked Read 7 Sent Done Done false
11335 4098 Acked Write 8 Sent Done Done false
11336 4099 Acked Read 1 Sent Done Done false
11337 4100 Acked Write 4 Sent Done Done false
11338 4101 Acked Write 7 Sent Done Done false
11339 4102 Acked Read 10 Sent Done Done false
11340 4103 Acked Write 6 Sent Done Done false
11341 4104 Acked Write 1 Sent Done Done false
11342 4105 Acked Write 4 Sent Done Done false
11343 4106 Acked Read 7 Sent Done Done false
11344 4107 Acked Write 8 Sent Done Done false
11345 4108 Acked Read 5 Sent Done Done false
11346 4109 Acked Write 1 Sent Done Done false
11347 4110 Acked Write 9 Sent Done Done false
11348 4111 Acked Read 6 Sent Done Done false
11349 4112 Acked Write 3 Sent Done Done false
11350 4113 Acked Write 8 Sent Done Done false
11351 4114 Acked Read 4 Sent Done Done false
11352 4115 Acked Read 3 Sent Done Done false
11353 4116 Acked Flush 0 Sent Done Done false
11354 4117 Acked Read 2 Sent Done Done false
11355 4118 Acked Read 8 Sent Done Done false
11356 4119 Acked Write 5 Sent Done Done false
11357 4120 Acked Flush 0 Sent Done Done false
11358 4121 Acked Write 5 Sent Done Done false
11359 4122 Acked Read 5 Sent Done Done false
11360 4123 Acked Write 5 Sent Done Done false
11361 4124 Acked Read 1 Sent Done Done false
11362 4125 Acked Flush 0 Sent Done Done false
11363 4126 Acked Read 3 Sent Done Done false
11364 4127 Acked Read 6 Sent Done Done false
11365 4128 Acked Flush 0 Sent Done Done false
11366 4129 Acked Flush 0 Sent Done Done false
11367 4130 Acked Read 6 Sent Done Done false
11368 4131 Acked Write 6 Sent Done Done false
11369 4132 Acked Write 6 Sent Done Done false
11370 4133 Acked Flush 0 Sent Done Done false
11371 4134 Acked Read 7 Sent Done Done false
11372 4135 Acked Read 7 Sent Done Done false
11373 4136 Acked Read 5 Sent Done Done false
11374 4137 Acked Write 3 Sent Done Done false
11375 4138 Acked Write 6 Sent Done Done false
11376 4139 Acked Write 10 Sent Done Done false
11377 4140 Acked Read 2 Sent Done Done false
11378 4141 Acked Write 3 Sent Done Done false
11379 4142 Acked Write 5 Sent Done Done false
11380 4143 Acked Write 8 Sent Done Done false
11381 4144 Acked Write 9 Sent Done Done false
11382 4145 Acked Write 2 Sent Done Done false
11383 4146 Acked Read 3 Sent Done Done false
11384 4147 Acked Read 1 Sent Done Done false
11385 4148 Acked Write 8 Sent Done Done false
11386 4149 Acked Flush 0 Sent Done Done false
11387 4150 Acked Read 2 Sent Done Done false
11388 4151 Acked Flush 0 Sent Done Done false
11389 4152 Acked Write 8 Sent Done Done false
11390 4153 Acked Read 10 Sent Done Done false
11391 4154 Acked Read 3 Sent Done Done false
11392 4155 Acked Write 10 Sent Done Done false
11393 4156 Acked Flush 0 Sent Done Done false
11394 4157 Acked Write 5 Sent Done Done false
11395 4158 Acked Write 9 Sent Done Done false
11396 4159 Acked Write 10 Sent Done Done false
11397 4160 Acked Flush 0 Sent Done Done false
11398 4161 Acked Read 9 Sent Done Done false
11399 4162 Acked Write 7 Sent Done Done false
11400 4163 Acked Read 10 Sent Done Done false
11401 4164 Acked Read 4 Sent Done Done false
11402 4165 Acked Read 5 Sent Done Done false
11403 4166 Acked Write 3 Sent Done Done false
11404 4167 Acked Read 5 Sent Done Done false
11405 4168 Acked Write 1 Sent Done Done false
11406 4169 Acked Read 6 Sent Done Done false
11407 4170 Acked Read 5 Sent Done Done false
11408 4171 Acked Read 10 Sent Done Done false
11409 4172 Acked Write 3 Sent Done Done false
11410 4173 Acked Write 8 Sent Done Done false
11411 4174 Acked Read 9 Sent Done Done false
11412 4175 Acked Read 9 Sent Done Done false
11413 4176 Acked Write 9 Sent Done Done false
11414 4177 Acked Write 4 Sent Done Done false
11415 4178 Acked Read 5 Sent Done Done false
11416 4179 Acked Flush 0 Sent Done Done false
11417 4180 Acked Write 5 Sent Done Done false
11418 4181 Acked Read 7 Sent Done Done false
11419 4182 Acked Write 1 Sent Done Done false
11420 4183 Acked Write 9 Sent Done Done false
11421 4184 Acked Flush 0 Sent Done Done false
11422 4185 Acked Write 10 Sent Done Done false
11423 4186 Acked Write 1 Sent Done Done false
11424 4187 Acked Write 8 Sent Done Done false
11425 4188 Acked Flush 0 Sent Done Done false
11426 4189 Acked Read 8 Sent Done Done false
11427 4190 Acked Write 6 Sent Done Done false
11428 4191 Acked Read 3 Sent Done Done false
11429 4192 Acked Read 8 Sent Done Done false
11430 4193 Acked Read 7 Sent Done Done false
11431 4194 Acked Write 3 Sent Done Done false
11432 4195 Acked Read 8 Sent Done Done false
11433 4196 Acked Write 7 Sent Done Done false
11434 4197 Acked Read 3 Sent Done Done false
11435 4198 Acked Read 2 Sent Done Done false
11436 4199 Acked Read 5 Sent Done Done false
11437 4200 Acked Flush 0 Sent Done Done false
11438 4201 Acked Read 7 Sent Done Done false
11439 4202 Acked Write 2 Sent Done Done false
11440 4203 Acked Write 10 Sent Done Done false
11441 4204 Acked Read 4 Sent Done Done false
11442 4205 Acked Write 6 Sent Done Done false
11443 4206 Acked Write 3 Sent Done Done false
11444 4207 Acked Read 2 Sent Done Done false
11445 4208 Acked Read 1 Sent Done Done false
11446 4209 Acked Write 8 Sent Done Done false
11447 4210 Acked Read 2 Sent Done Done false
11448 4211 Acked Read 7 Sent Done Done false
11449 4212 Acked Write 10 Sent Done Done false
11450 4213 Acked Read 7 Sent Done Done false
11451 4214 Acked Flush 0 Sent Done Done false
11452 4215 Acked Read 1 Sent Done Done false
11453 4216 Acked Read 9 Sent Done Done false
11454 4217 Acked Write 6 Sent Done Done false
11455 4218 Acked Read 7 Sent Done Done false
11456 4219 Acked Write 6 Sent Done Done false
11457 4220 Acked Write 5 Sent Done Done false
11458 4221 Acked Flush 0 Sent Done Done false
11459 4222 Acked Write 8 Sent Done Done false
11460 4223 Acked Flush 0 Sent Done Done false
11461 4224 Acked Read 5 Sent Done Done false
11462 4225 Acked Read 9 Sent Done Done false
11463 4226 Acked Read 5 Sent Done Done false
11464 4227 Acked Flush 0 Sent Done Done false
11465 4228 Acked Flush 0 Sent Done Done false
11466 4229 Acked Read 4 Sent Done Done false
11467 4230 Acked Read 5 Sent Done Done false
11468 4231 Acked Read 7 Sent Done Done false
11469 4232 Acked Read 3 Sent Done Done false
11470 4233 Acked Read 6 Sent Done Done false
11471 4234 Acked Read 3 Sent Done Done false
11472 4235 Acked Read 6 Sent Done Done false
11473 4236 Acked Write 2 Sent Done Done false
11474 4237 Acked Write 4 Sent Done Done false
11475 4238 Acked Write 3 Sent Done Done false
11476 4239 Acked Read 7 Sent Done Done false
11477 4240 Acked Read 4 Sent Done Done false
11478 4241 Acked Read 9 Sent Done Done false
11479 4242 Acked Write 2 Sent Done Done false
11480 4243 Acked Write 1 Sent Done Done false
11481 4244 Acked Read 7 Sent Done Done false
11482 4245 Acked Write 2 Sent Done Done false
11483 4246 Acked Read 10 Sent Done Done false
11484 4247 Acked Flush 0 Sent Done Done false
11485 4248 Acked Read 8 Sent Done Done false
11486 4249 Acked Read 10 Sent Done Done false
11487 4250 Acked Read 4 Sent Done Done false
11488 4251 Acked Write 5 Sent Done Done false
11489 4252 Acked Read 6 Sent Done Done false
11490 4253 Acked Write 7 Sent Done Done false
11491 4254 Acked Write 8 Sent Done Done false
11492 4255 Acked Write 9 Sent Done Done false
11493 4256 Acked Read 6 Sent Done Done false
11494 4257 Acked Read 4 Sent Done Done false
11495 4258 Acked Read 9 Sent Done Done false
11496 4259 Acked Flush 0 Sent Done Done false
11497 4260 Acked Write 8 Sent Done Done false
11498 4261 Acked Write 2 Sent Done Done false
11499 4262 Acked Write 7 Sent Done Done false
11500 4263 Acked Write 10 Sent Done Done false
11501 4264 Acked Read 2 Sent Done Done false
11502 4265 Acked Read 7 Sent Done Done false
11503 4266 Acked Write 4 Sent Done Done false
11504 4267 Acked Write 5 Sent Done Done false
11505 4268 Acked Read 5 Sent Done Done false
11506 4269 Acked Flush 0 Sent Done Done false
11507 4270 Acked Read 9 Sent Done Done false
11508 4271 Acked Read 6 Sent Done Done false
11509 4272 Acked Read 10 Sent Done Done false
11510 4273 Acked Write 3 Sent Done Done false
11511 4274 Acked Read 8 Sent Done Done false
11512 4275 Acked Write 3 Sent Done Done false
11513 4276 Acked Read 10 Sent Done Done false
11514 4277 Acked Read 4 Sent Done Done false
11515 4278 Acked Read 3 Sent Done Done false
11516 4279 Acked Read 8 Sent Done Done false
11517 4280 Acked Read 1 Sent Done Done false
11518 4281 Acked Flush 0 Sent Done Done false
11519 4282 Acked Write 4 Sent Done Done false
11520 4283 Acked Flush 0 Sent Done Done false
11521 4284 Acked Read 9 Sent Done Done false
11522 4285 Acked Write 2 Sent Done Done false
11523 4286 Acked Read 1 Sent Done Done false
11524 4287 Acked Read 4 Sent Done Done false
11525 4288 Acked Read 8 Sent Done Done false
11526 4289 Acked Write 6 Sent Done Done false
11527 4290 Acked Read 4 Sent Done Done false
11528 4291 Acked Write 3 Sent Done Done false
11529 4292 Acked Write 10 Sent Done Done false
11530 4293 Acked Write 5 Sent Done Done false
11531 4294 Acked Read 6 Sent Done Done false
11532 4295 Acked Write 10 Sent Done Done false
11533 4296 Acked Read 2 Sent Done Done false
11534 4297 Acked Write 9 Sent Done Done false
11535 4298 Acked Write 1 Sent Done Done false
11536 4299 Acked Read 1 Sent Done Done false
11537 4300 Acked Write 6 Sent Done Done false
11538 4301 Acked Write 7 Sent Done Done false
11539 4302 Acked Read 3 Sent Done Done false
11540 4303 Acked Write 10 Sent Done Done false
11541 4304 Acked Read 2 Sent Done Done false
11542 4305 Acked Read 7 Sent Done Done false
11543 4306 Acked Read 9 Sent Done Done false
11544 4307 Acked Write 8 Sent Done Done false
11545 4308 Acked Write 3 Sent Done Done false
11546 4309 Acked Read 10 Sent Done Done false
11547 4310 Acked Write 2 Sent Done Done false
11548 4311 Acked Read 8 Sent Done Done false
11549 4312 Acked Write 4 Sent Done Done false
11550 4313 Acked Flush 0 Sent Done Done false
11551 4314 Acked Write 5 Sent Done Done false
11552 4315 Acked Write 5 Sent Done Done false
11553 4316 Acked Flush 0 Sent Done Done false
11554 4317 Acked Read 3 Sent Done Done false
11555 4318 Acked Read 1 Sent Done Done false
11556 4319 Acked Read 4 Sent Done Done false
11557 4320 Acked Read 10 Sent Done Done false
11558 4321 Acked Flush 0 Sent Done Done false
11559 STATES DS:0 DS:1 DS:2 TOTAL
11560 Sent 302 0 0 302
11561 Done 0 302 302 604
11562 Skipped 0 0 0 0
11563 Error 0 0 0 0
11564 Last Flush: 4019 4321 4321
11565 Downstairs last five completed: 4019 4018 4017 4016 4015
11566 Upstairs last five completed: 4321 4320 4319 4318 4317
11567 CLIENT: Up:0 ds:302 act:2
115682024-12-09T15:10:15.579ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115692024-12-09T15:10:15.579ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46869) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115702024-12-09T15:10:15.580ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115712024-12-09T15:10:15.581ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115722024-12-09T15:10:15.581ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115732024-12-09T15:10:15.581ZINFOcrucible: send last flush ID to this DS: 4019 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115742024-12-09T15:10:15.581ZINFOcrucible: Replied this last flush ID: 4019 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115752024-12-09T15:10:15.581ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115762024-12-09T15:10:15.581ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115772024-12-09T15:10:15.581ZINFOcrucible: [0] client re-new 302 jobs since flush 4019 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115782024-12-09T15:10:15.582ZINFOcrucible: [0] Replayed 302 jobs since flush: 4019 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
11579 ----------------------------------------------------------------
11580 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11581 Downstairs last five completed: 4321 4320 4319 4318 4317
11582 Upstairs last five completed: 4321 4320 4319 4318 4317
11583 CLIENT: Up:0 ds:0 act:3
11584 Replay: All jobs finished
11585 Main task has work to do, go find it
11586 got dsc Stop(0)
11587 stop 0
11588 Dec 09 15:10:17.825 INFO request completed, latency_us: 359, response_code: 204, uri:[8810] Got stop action so:false kr:false
11589 /stop/cid/0, method: POST, req_id: 8ec14c12-7a71-4343-8b98-92462e58488d, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
115902024-12-09T15:10:17.825ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115912024-12-09T15:10:17.825ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115922024-12-09T15:10:17.825ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115932024-12-09T15:10:17.825ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
115942024-12-09T15:10:17.825ZINFOcrucible: [0] Transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
11595 Dec 09 15:10:17.825 INFO request completed, latency_us: 124, response_code: 200, uri: /state/cid/0, method: GET, req_id: ee9cee5d-66ed-405d-95fa-58997c05cc38, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
115962024-12-09T15:10:17.826ZINFOcrucible: client task is exiting = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
11597 [8810] Exited with: ExitStatus(unix_wait_status(9))
11598 [8810][0] reports Exit
11599 Dec 09 15:10:21.828 INFO request completed, latency_us: 222, response_code: 200, uri: /state/cid/0, method: GET, req_id: cffbcc0d-d0d7-4834-8928-d374b5a467b7, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
11600 001/300 Write block 557 len 8192 data: 2 2
11601 002/300 Write block 758 len 36864 data: 5 5 4 4 4 4 3 3 3
11602 003/300 Read block 4049 len 8192
11603 004/300 Read block 2988 len 20480
11604 005/300 Read block 2993 len 16384
11605 006/300 Read block 4552 len 24576
11606 007/300 Write block 3066 len 8192 data: 3 2
11607 008/300 Read block 3806 len 28672
11608 009/300 Write block 4018 len 16384 data: 5 4 5 5
11609 010/300 Read block 4763 len 20480
11610 011/300 Read block 1451 len 4096
11611 012/300 Write block 3100 len 8192 data: 3 3
11612 013/300 Read block 2973 len 36864
11613 014/300 Read block 3638 len 36864
11614 015/300 Read block 1808 len 8192
11615 016/300 Write block 795 len 36864 data: 3 4 4 5 4 3 6 6 6
11616 017/300 Read block 4736 len 8192
11617 018/300 Write block 4314 len 16384 data: 4 3 3 3
11618 019/300 Write block 4799 len 16384 data: 5 6 5 4
11619 020/300 Write block 1779 len 8192 data: 6 6
11620 021/300 Flush
11621 022/300 Write block 495 len 8192 data: 5 4
11622 023/300 Read block 3895 len 40960
11623 024/300 Read block 1647 len 24576
11624 025/300 Read block 3216 len 4096
11625 026/300 Write block 4800 len 16384 data: 7 6 5 4
11626 027/300 Flush
11627 028/300 Write block 4796 len 4096 data: 4
11628 029/300 Read block 3539 len 4096
11629 030/300 Write block 103 len 24576 data: 3 3 4 4 4 3
11630 031/300 Flush
11631 032/300 Flush
11632 033/300 Flush
11633 034/300 Read block 2673 len 24576
11634 035/300 Read block 2144 len 12288
11635 036/300 Read block 3100 len 8192
11636 037/300 Write block 4683 len 20480 data: 3 3 3 2 2
11637 038/300 Write block 1186 len 28672 data: 2 2 2 2 2 2 2
11638 039/300 Read block 2436 len 4096
11639 040/300 Write block 4243 len 40960 data: 4 4 4 5 5 3 2 2 2 3
11640 041/300 Read block 3507 len 32768
11641 042/300 Read block 2298 len 12288
11642 043/300 Write block 860 len 4096 data: 4
11643 044/300 Write block 2662 len 8192 data: 3 3
11644 045/300 Write block 4932 len 4096 data: 3
11645 046/300 Read block 1166 len 12288
11646 047/300 Read block 1633 len 12288
11647 048/300 Read block 1137 len 12288
11648 049/300 Read block 2343 len 4096
11649 050/300 Write block 1909 len 12288 data: 4 4 4
11650 051/300 Write block 867 len 40960 data: 4 4 4 4 4 3 3 3 2 3
11651 052/300 Write block 2051 len 20480 data: 3 4 4 3 3
11652 053/300 Read block 4135 len 8192
11653 054/300 Flush
11654 055/300 Flush
11655 056/300 Write block 1335 len 28672 data: 4 4 4 4 4 3 3
11656 057/300 Read block 336 len 4096
11657 058/300 Write block 4742 len 16384 data: 2 2 3 3
11658 059/300 Write block 1991 len 28672 data: 2 2 2 2 2 3 4
11659 060/300 Write block 3360 len 16384 data: 3 4 4 4
11660 061/300 Read block 4501 len 40960
11661 062/300 Read block 99 len 12288
11662 063/300 Write block 218 len 40960 data: 6 5 3 3 3 3 3 3 3 5
11663 064/300 Read block 1420 len 32768
11664 065/300 Write block 2924 len 32768 data: 3 2 2 4 5 4 5 5
11665 066/300 Write block 1335 len 20480 data: 5 5 5 5 5
11666 067/300 Flush
11667 068/300 Write block 549 len 16384 data: 6 6 6 5
11668 069/300 Write block 3049 len 8192 data: 2 2
11669 070/300 Flush
11670 071/300 Flush
11671 072/300 Read block 3932 len 16384
11672 073/300 Write block 4169 len 4096 data: 4
11673 074/300 Read block 1279 len 16384
11674 075/300 Write block 3755 len 40960 data: 3 3 3 4 4 4 4 5 5 5
11675 076/300 Read block 2438 len 28672
11676 077/300 Read block 2985 len 12288
11677 078/300 Write block 2970 len 40960 data: 3 3 3 3 2 2 2 2 2 2
11678 079/300 Read block 4948 len 20480
11679 080/300 Read block 4595 len 20480
11680 081/300 Flush
11681 082/300 Read block 2075 len 20480
11682 083/300 Write block 47 len 28672 data: 3 3 3 3 2 2 3
11683 084/300 Write block 2525 len 8192 data: 5 5
11684 085/300 Write block 606 len 28672 data: 3 3 4 3 2 2 2
11685 086/300 Read block 3548 len 12288
11686 087/300 Read block 3158 len 8192
11687 088/300 Write block 2069 len 20480 data: 2 2 2 2 3
11688 089/300 Write block 4655 len 32768 data: 5 5 4 4 3 3 3 5
11689 090/300 Write block 1617 len 36864 data: 2 3 3 3 3 3 3 3 3
11690 091/300 Read block 4280 len 32768
11691 092/300 Flush
11692 093/300 Write block 4555 len 20480 data: 5 6 5 4 3
11693 094/300 Read block 1097 len 28672
11694 095/300 Read block 221 len 16384
11695 096/300 Flush
11696 097/300 Write block 2761 len 32768 data: 3 2 2 2 2 2 2 4
11697 098/300 Write block 1321 len 8192 data: 4 4
11698 099/300 Read block 2580 len 12288
11699 100/300 Read block 1735 len 16384
11700 101/300 Write block 315 len 16384 data: 3 4 4 5
11701 102/300 Write block 1336 len 12288 data: 6 6 6
11702 103/300 Write block 1762 len 20480 data: 4 4 4 4 3
11703 104/300 Flush
11704 105/300 Write block 547 len 40960 data: 8 7 7 7 7 6 4 3 3 3
11705 106/300 Read block 2266 len 36864
11706 107/300 Read block 362 len 12288
11707 108/300 Write block 1817 len 16384 data: 3 3 4 4
11708 109/300 Read block 4251 len 36864
11709 110/300 Write block 1754 len 40960 data: 5 5 6 6 6 5 4 3 5 5
11710 111/300 Flush
11711 112/300 Write block 1326 len 16384 data: 2 2 2 2
11712 113/300 Flush
11713 114/300 Write block 4612 len 20480 data: 2 2 2 2 3
11714 115/300 Write block 4343 len 20480 data: 4 3 3 4 5
11715 116/300 Write block 199 len 12288 data: 3 3 2
11716 117/300 Write block 367 len 24576 data: 4 3 3 3 3 3
11717 118/300 Read block 287 len 28672
11718 119/300 Read block 4309 len 16384
11719 120/300 Read block 2942 len 28672
11720 121/300 Write block 1117 len 36864 data: 3 2 2 2 2 2 3 3 4
11721 122/300 Write block 2599 len 4096 data: 2
11722 123/300 Write block 3773 len 40960 data: 2 2 3 3 4 4 4 4 5 4
11723 124/300 Write block 233 len 28672 data: 3 3 3 3 3 3 3
11724 125/300 Read block 4838 len 40960
11725 126/300 Write block 3376 len 32768 data: 3 3 3 3 3 3 4 5
11726 127/300 Write block 145 len 28672 data: 5 5 4 4 7 7 6
11727 128/300 Flush
11728 129/300 Write block 766 len 16384 data: 4 3 2 3
11729 130/300 Read block 4523 len 28672
11730 131/300 Read block 3013 len 32768
11731 132/300 Read block 2109 len 36864
11732 133/300 Write block 667 len 28672 data: 4 4 2 2 4 4 4
11733 134/300 Write block 4420 len 12288 data: 2 3 3
11734 135/300 Read block 3463 len 12288
11735 136/300 Read block 3890 len 12288
11736 137/300 Read block 4014 len 4096
11737 138/300 Read block 2788 len 40960
11738 139/300 Read block 3006 len 32768
11739 140/300 Write block 1881 len 20480 data: 2 2 2 2 2
11740 141/300 Read block 2753 len 32768
11741 142/300 Flush
11742 143/300 Write block 2645 len 16384 data: 6 5 3 4
11743 144/300 Read block 3734 len 28672
11744 145/300 Write block 2392 len 20480 data: 5 6 6 5 7
11745 146/300 Write block 3824 len 12288 data: 7 6 6
11746 147/300 Write block 1767 len 4096 data: 2
11747 148/300 Read block 3285 len 20480
11748 149/300 Flush
11749 150/300 Read block 514 len 32768
11750 151/300 Write block 548 len 16384 data: 8 8 8 8
11751 152/300 Write block 686 len 8192 data: 4 4
11752 153/300 Write block 1712 len 8192 data: 4 5
11753 154/300 Read block 615 len 32768
11754 155/300 Read block 1700 len 28672
11755 156/300 Flush
11756 157/300 Write block 223 len 8192 data: 4 4
11757 158/300 Read block 3828 len 36864
11758 159/300 Write block 22 len 20480 data: 4 4 4 4 4
11759 160/300 Read block 1403 len 36864
11760 161/300 Write block 168 len 4096 data: 3
11761 162/300 Write block 4746 len 12288 data: 3 3 5
11762 163/300 Write block 1231 len 4096 data: 4
11763 164/300 Read block 2861 len 20480
11764 165/300 Flush
11765 166/300 Read block 4896 len 8192
11766 167/300 Write block 1791 len 36864 data: 4 4 5 5 6 6 5 4 3
11767 168/300 Read block 2588 len 40960
11768 169/300 Write block 2900 len 24576 data: 4 4 3 3 2 2
11769 170/300 Read block 2405 len 24576
11770 171/300 Read block 3425 len 24576
11771 172/300 Write block 4462 len 16384 data: 3 4 4 5
11772 173/300 Write block 2713 len 24576 data: 6 4 4 4 4 4
11773 174/300 Flush
11774 175/300 Read block 4077 len 24576
11775 176/300 Flush
11776 177/300 Read block 862 len 24576
11777 178/300 Flush
11778 179/300 Read block 4371 len 8192
11779 180/300 Read block 3297 len 16384
11780 181/300 Read block 3422 len 8192
11781 182/300 Read block 2969 len 12288
11782 183/300 Read block 3066 len 28672
11783 184/300 Read block 3878 len 20480
11784 185/300 Write block 3814 len 20480 data: 3 3 3 3 3
11785 186/300 Flush
11786 187/300 Read block 2191 len 24576
11787 188/300 Read block 24 len 24576
11788 189/300 Write block 701 len 20480 data: 3 3 3 3 3
11789 190/300 Write block 360 len 36864 data: 6 6 6 5 5 4 4 5 4
11790 191/300 Read block 3962 len 32768
11791 192/300 Write block 4811 len 20480 data: 4 4 5 6 3
11792 193/300 Write block 4355 len 24576 data: 3 4 3 3 3 4
11793 194/300 Write block 45 len 20480 data: 3 3 4 4 4
11794 195/300 Read block 3277 len 36864
11795 196/300 Read block 2741 len 12288
11796 197/300 Read block 682 len 24576
11797 198/300 Write block 1388 len 12288 data: 3 3 3
11798 199/300 Read block 2217 len 28672
11799 200/300 Write block 819 len 28672 data: 5 5 5 4 4 4 4
11800 201/300 Read block 2395 len 40960
11801 202/300 Read block 4650 len 28672
11802 203/300 Read block 3315 len 8192
11803 204/300 Flush
11804 205/300 Read block 2201 len 36864
11805 206/300 Write block 909 len 32768 data: 4 3 3 3 3 3 4 4
11806 207/300 Write block 2590 len 16384 data: 2 2 2 2
11807 208/300 Write block 3750 len 40960 data: 4 4 4 5 5 4 4 4 5 5
11808 209/300 Flush
11809 210/300 Read block 2035 len 12288
11810 211/300 Read block 24 len 20480
11811 212/300 Write block 2609 len 4096 data: 4
11812 213/300 Flush
11813 214/300 Flush
11814 215/300 Write block 4499 len 36864 data: 4 4 4 5 5 5 5 7 7
11815 216/300 Read block 2435 len 40960
11816 217/300 Write block 4708 len 40960 data: 5 5 3 3 3 4 4 4 4 4
11817 218/300 Write block 3499 len 12288 data: 2 3 3
11818 219/300 Flush
11819 220/300 Read block 4199 len 20480
11820 221/300 Write block 1017 len 24576 data: 2 2 2 2 2 2
11821 222/300 Read block 4184 len 40960
11822 223/300 Read block 3672 len 28672
11823 224/300 Read block 2334 len 24576
11824 225/300 Read block 4167 len 16384
11825 226/300 Read block 4870 len 24576
11826 227/300 Flush
11827 228/300 Flush
11828 229/300 Read block 4520 len 40960
11829 230/300 Write block 3327 len 12288 data: 3 3 3
11830 231/300 Read block 2831 len 36864
11831 232/300 Flush
11832 233/300 Write block 4472 len 20480 data: 3 3 3 2 2
11833 234/300 Read block 4304 len 32768
11834 235/300 Write block 1547 len 8192 data: 4 3
11835 236/300 Write block 4424 len 28672 data: 3 4 5 5 5 5 4
11836 237/300 Read block 4939 len 12288
11837 238/300 Write block 2971 len 20480 data: 4 4 4 3 3
11838 239/300 Read block 4090 len 12288
11839 240/300 Write block 2136 len 16384 data: 7 7 7 5
11840 241/300 Write block 1069 len 32768 data: 2 2 2 2 3 3 3 5
11841 242/300 Read block 465 len 16384
11842 243/300 Read block 1520 len 16384
11843 244/300 Read block 3265 len 40960
11844 245/300 Read block 2390 len 24576
11845 246/300 Read block 742 len 40960
11846 247/300 Read block 3178 len 28672
11847 248/300 Read block 1351 len 12288
11848 249/300 Write block 4103 len 28672 data: 3 3 3 2 2 2 2
11849 250/300 Write block 3616 len 24576 data: 5 5 5 6 6 6
11850 251/300 Write block 1142 len 36864 data: 4 3 3 4 3 4 5 6 6
11851 252/300 Read block 1743 len 32768
11852 253/300 Read block 1297 len 8192
11853 254/300 Read block 4710 len 8192
11854 255/300 Write block 1132 len 32768 data: 4 4 4 3 2 2 2 2
11855 256/300 Read block 698 len 20480
11856 257/300 Flush
11857 258/300 Write block 3755 len 16384 data: 5 5 5 6
11858 259/300 Flush
11859 260/300 Write block 661 len 40960 data: 3 3 4 4 4 4 5 5 3 3
11860 261/300 Read block 3659 len 4096
11861 262/300 Write block 1884 len 16384 data: 3 3 4 4
11862 263/300 Write block 4721 len 40960 data: 4 3 3 3 3 2 2 2 2 2
11863 264/300 Read block 1827 len 36864
11864 265/300 Write block 1971 len 12288 data: 5 4 3
11865 266/300 Read block 2434 len 36864
11866 267/300 Read block 636 len 32768
11867 268/300 Write block 1156 len 12288 data: 2 2 2
11868 269/300 Read block 1176 len 28672
11869 270/300 Write block 2290 len 16384 data: 4 3 3 3
11870 271/300 Write block 2711 len 28672 data: 6 6 7 5 5 5 5
11871 272/300 Flush
11872 273/300 Write block 219 len 20480 data: 6 4 4 4 5
11873 274/300 Flush
11874 275/300 Read block 2682 len 16384
11875 276/300 Write block 732 len 12288 data: 3 3 2
11876 277/300 Read block 3985 len 8192
11877 278/300 Write block 18 len 28672 data: 3 3 3 3 5 5 5
11878 279/300 Read block 175 len 8192
11879 280/300 Read block 3547 len 24576
11880 281/300 Read block 25 len 12288
11881 282/300 Read block 2314 len 32768
11882 283/300 Read block 2279 len 12288
11883 284/300 Read block 4949 len 12288
11884 285/300 Write block 4159 len 4096 data: 6
11885 286/300 Read block 424 len 8192
11886 287/300 Read block 1147 len 32768
11887 288/300 Read block 2686 len 28672
11888 289/300 Read block 4349 len 32768
11889 290/300 Write block 4101 len 36864 data: 4 3 4 4 4 3 3 3 3
11890 291/300 Read block 2355 len 40960
11891 292/300 Read block 4502 len 36864
11892 293/300 Read block 2731 len 32768
11893 294/300 Read block 2508 len 8192
11894 295/300 Write block 2813 len 28672 data: 3 3 3 3 4 4 4
11895 296/300 Write block 4097 len 40960 data: 7 7 6 4 5 4 5 5 5 4
11896 297/300 Write block 1404 len 8192 data: 3 3
11897 298/300 Read block 1589 len 8192
11898 299/300 Read block 2491 len 8192
11899 300/300 Write block 3159 len 12288 data: 6 6 7
11900 301/300 Read block 909 len 28672
11901 Main task has work to do, go find it
11902 got dsc Start(0)
11903 start 0
11904 Dec 09 15:10:22.059 INFO request completed, latency_us: 225, response_code: 204, uri: /start/cid/0, method: POST, req_id: 843f6a5d-e6b8-429e-8027-b0d160d8197f, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
11905 Replay: started 0, returned:Ok(())
11906 ----------------------------------------------------------------
11907 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11908 States: Offline Active Active
11909 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11910 4322 Acked Write 2 Sent Done Done false
11911 4323 Acked Write 9 Sent Done Done false
11912 4324 Acked Read 2 Sent Done Done false
11913 4325 Acked Read 5 Sent Done Done false
11914 4326 Acked Read 4 Sent Done Done false
11915 4327 Acked Read 6 Sent Done Done false
11916 4328 Acked Write 2 Sent Done Done false
11917 4329 Acked Read 7 Sent Done Done false
11918 4330 Acked Write 4 Sent Done Done false
11919 4331 Acked Read 5 Sent Done Done false
11920 4332 Acked Read 1 Sent Done Done false
11921 4333 Acked Write 2 Sent Done Done false
11922 4334 Acked Read 9 Sent Done Done false
11923 4335 Acked Read 9 Sent Done Done false
11924 4336 Acked Read 2 Sent Done Done false
11925 4337 Acked Write 9 Sent Done Done false
11926 4338 Acked Read 2 Sent Done Done false
11927 4339 Acked Write 4 Sent Done Done false
11928 4340 Acked Write 4 Sent Done Done false
11929 4341 Acked Write 2 Sent Done Done false
11930 4342 Acked Flush 0 Sent Done Done false
11931 4343 Acked Write 2 Sent Done Done false
11932 4344 Acked Read 10 Sent Done Done false
11933 4345 Acked Read 6 Sent Done Done false
11934 4346 Acked Read 1 Sent Done Done false
11935 4347 Acked Write 4 Sent Done Done false
11936 4348 Acked Flush 0 Sent Done Done false
11937 4349 Acked Write 1 Sent Done Done false
11938 4350 Acked Read 1 Sent Done Done false
11939 4351 Acked Write 6 Sent Done Done false
11940 4352 Acked Flush 0 Sent Done Done false
11941 4353 Acked Flush 0 Sent Done Done false
11942 4354 Acked Flush 0 Sent Done Done false
11943 4355 Acked Read 6 Sent Done Done false
11944 4356 Acked Read 3 Sent Done Done false
11945 4357 Acked Read 2 Sent Done Done false
11946 4358 Acked Write 5 Sent Done Done false
11947 4359 Acked Write 7 Sent Done Done false
11948 4360 Acked Read 1 Sent Done Done false
11949 4361 Acked Write 10 Sent Done Done false
11950 4362 Acked Read 8 Sent Done Done false
11951 4363 Acked Read 3 Sent Done Done false
11952 4364 Acked Write 1 Sent Done Done false
11953 4365 Acked Write 2 Sent Done Done false
11954 4366 Acked Write 1 Sent Done Done false
11955 4367 Acked Read 3 Sent Done Done false
11956 4368 Acked Read 3 Sent Done Done false
11957 4369 Acked Read 3 Sent Done Done false
11958 4370 Acked Read 1 Sent Done Done false
11959 4371 Acked Write 3 Sent Done Done false
11960 4372 Acked Write 10 Sent Done Done false
11961 4373 Acked Write 5 Sent Done Done false
11962 4374 Acked Read 2 Sent Done Done false
11963 4375 Acked Flush 0 Sent Done Done false
11964 4376 Acked Flush 0 Sent Done Done false
11965 4377 Acked Write 7 Sent Done Done false
11966 4378 Acked Read 1 Sent Done Done false
11967 4379 Acked Write 4 Sent Done Done false
11968 4380 Acked Write 7 Sent Done Done false
11969 4381 Acked Write 4 Sent Done Done false
11970 4382 Acked Read 10 Sent Done Done false
11971 4383 Acked Read 3 Sent Done Done false
11972 4384 Acked Write 10 Sent Done Done false
11973 4385 Acked Read 8 Sent Done Done false
11974 4386 Acked Write 8 Sent Done Done false
11975 4387 Acked Write 5 Sent Done Done false
11976 4388 Acked Flush 0 Sent Done Done false
11977 4389 Acked Write 4 Sent Done Done false
11978 4390 Acked Write 2 Sent Done Done false
11979 4391 Acked Flush 0 Sent Done Done false
11980 4392 Acked Flush 0 Sent Done Done false
11981 4393 Acked Read 4 Sent Done Done false
11982 4394 Acked Write 1 Sent Done Done false
11983 4395 Acked Read 4 Sent Done Done false
11984 4396 Acked Write 10 Sent Done Done false
11985 4397 Acked Read 7 Sent Done Done false
11986 4398 Acked Read 3 Sent Done Done false
11987 4399 Acked Write 10 Sent Done Done false
11988 4400 Acked Read 5 Sent Done Done false
11989 4401 Acked Read 5 Sent Done Done false
11990 4402 Acked Flush 0 Sent Done Done false
11991 4403 Acked Read 5 Sent Done Done false
11992 4404 Acked Write 7 Sent Done Done false
11993 4405 Acked Write 2 Sent Done Done false
11994 4406 Acked Write 7 Sent Done Done false
11995 4407 Acked Read 3 Sent Done Done false
11996 4408 Acked Read 2 Sent Done Done false
11997 4409 Acked Write 5 Sent Done Done false
11998 4410 Acked Write 8 Sent Done Done false
11999 4411 Acked Write 9 Sent Done Done false
12000 4412 Acked Read 8 Sent Done Done false
12001 4413 Acked Flush 0 Sent Done Done false
12002 4414 Acked Write 5 Sent Done Done false
12003 4415 Acked Read 7 Sent Done Done false
12004 4416 Acked Read 4 Sent Done Done false
12005 4417 Acked Flush 0 Sent Done Done false
12006 4418 Acked Write 8 Sent Done Done false
12007 4419 Acked Write 2 Sent Done Done false
12008 4420 Acked Read 3 Sent Done Done false
12009 4421 Acked Read 4 Sent Done Done false
12010 4422 Acked Write 4 Sent Done Done false
12011 4423 Acked Write 3 Sent Done Done false
12012 4424 Acked Write 5 Sent Done Done false
12013 4425 Acked Flush 0 Sent Done Done false
12014 4426 Acked Write 10 Sent Done Done false
12015 4427 Acked Read 9 Sent Done Done false
12016 4428 Acked Read 3 Sent Done Done false
12017 4429 Acked Write 4 Sent Done Done false
12018 4430 Acked Read 9 Sent Done Done false
12019 4431 Acked Write 10 Sent Done Done false
12020 4432 Acked Flush 0 Sent Done Done false
12021 4433 Acked Write 4 Sent Done Done false
12022 4434 Acked Flush 0 Sent Done Done false
12023 4435 Acked Write 5 Sent Done Done false
12024 4436 Acked Write 5 Sent Done Done false
12025 4437 Acked Write 3 Sent Done Done false
12026 4438 Acked Write 6 Sent Done Done false
12027 4439 Acked Read 7 Sent Done Done false
12028 4440 Acked Read 4 Sent Done Done false
12029 4441 Acked Read 7 Sent Done Done false
12030 4442 Acked Write 9 Sent Done Done false
12031 4443 Acked Write 1 Sent Done Done false
12032 4444 Acked Write 10 Sent Done Done false
12033 4445 Acked Write 7 Sent Done Done false
12034 4446 Acked Read 10 Sent Done Done false
12035 4447 Acked Write 8 Sent Done Done false
12036 4448 Acked Write 7 Sent Done Done false
12037 4449 Acked Flush 0 Sent Done Done false
12038 4450 Acked Write 4 Sent Done Done false
12039 4451 Acked Read 7 Sent Done Done false
12040 4452 Acked Read 8 Sent Done Done false
12041 4453 Acked Read 9 Sent Done Done false
12042 4454 Acked Write 7 Sent Done Done false
12043 4455 Acked Write 3 Sent Done Done false
12044 4456 Acked Read 3 Sent Done Done false
12045 4457 Acked Read 3 Sent Done Done false
12046 4458 Acked Read 1 Sent Done Done false
12047 4459 Acked Read 10 Sent Done Done false
12048 4460 Acked Read 8 Sent Done Done false
12049 4461 Acked Write 5 Sent Done Done false
12050 4462 Acked Read 8 Sent Done Done false
12051 4463 Acked Flush 0 Sent Done Done false
12052 4464 Acked Write 4 Sent Done Done false
12053 4465 Acked Read 7 Sent Done Done false
12054 4466 Acked Write 5 Sent Done Done false
12055 4467 Acked Write 3 Sent Done Done false
12056 4468 Acked Write 1 Sent Done Done false
12057 4469 Acked Read 5 Sent Done Done false
12058 4470 Acked Flush 0 Sent Done Done false
12059 4471 Acked Read 8 Sent Done Done false
12060 4472 Acked Write 4 Sent Done Done false
12061 4473 Acked Write 2 Sent Done Done false
12062 4474 Acked Write 2 Sent Done Done false
12063 4475 Acked Read 8 Sent Done Done false
12064 4476 Acked Read 7 Sent Done Done false
12065 4477 Acked Flush 0 Sent Done Done false
12066 4478 Acked Write 2 Sent Done Done false
12067 4479 Acked Read 9 Sent Done Done false
12068 4480 Acked Write 5 Sent Done Done false
12069 4481 Acked Read 9 Sent Done Done false
12070 4482 Acked Write 1 Sent Done Done false
12071 4483 Acked Write 3 Sent Done Done false
12072 4484 Acked Write 1 Sent Done Done false
12073 4485 Acked Read 5 Sent Done Done false
12074 4486 Acked Flush 0 Sent Done Done false
12075 4487 Acked Read 2 Sent Done Done false
12076 4488 Acked Write 9 Sent Done Done false
12077 4489 Acked Read 10 Sent Done Done false
12078 4490 Acked Write 6 Sent Done Done false
12079 4491 Acked Read 6 Sent Done Done false
12080 4492 Acked Read 6 Sent Done Done false
12081 4493 Acked Write 4 Sent Done Done false
12082 4494 Acked Write 6 Sent Done Done false
12083 4495 Acked Flush 0 Sent Done Done false
12084 4496 Acked Read 6 Sent Done Done false
12085 4497 Acked Flush 0 Sent Done Done false
12086 4498 Acked Read 6 Sent Done Done false
12087 4499 Acked Flush 0 Sent Done Done false
12088 4500 Acked Read 2 Sent Done Done false
12089 4501 Acked Read 4 Sent Done Done false
12090 4502 Acked Read 2 Sent Done Done false
12091 4503 Acked Read 3 Sent Done Done false
12092 4504 Acked Read 7 Sent Done Done false
12093 4505 Acked Read 5 Sent Done Done false
12094 4506 Acked Write 5 Sent Done Done false
12095 4507 Acked Flush 0 Sent Done Done false
12096 4508 Acked Read 6 Sent Done Done false
12097 4509 Acked Read 6 Sent Done Done false
12098 4510 Acked Write 5 Sent Done Done false
12099 4511 Acked Write 9 Sent Done Done false
12100 4512 Acked Read 8 Sent Done Done false
12101 4513 Acked Write 5 Sent Done Done false
12102 4514 Acked Write 6 Sent Done Done false
12103 4515 Acked Write 5 Sent Done Done false
12104 4516 Acked Read 9 Sent Done Done false
12105 4517 Acked Read 3 Sent Done Done false
12106 4518 Acked Read 6 Sent Done Done false
12107 4519 Acked Write 3 Sent Done Done false
12108 4520 Acked Read 7 Sent Done Done false
12109 4521 Acked Write 7 Sent Done Done false
12110 4522 Acked Read 10 Sent Done Done false
12111 4523 Acked Read 7 Sent Done Done false
12112 4524 Acked Read 2 Sent Done Done false
12113 4525 Acked Flush 0 Sent Done Done false
12114 4526 Acked Read 9 Sent Done Done false
12115 4527 Acked Write 8 Sent Done Done false
12116 4528 Acked Write 4 Sent Done Done false
12117 4529 Acked Write 10 Sent Done Done false
12118 4530 Acked Flush 0 Sent Done Done false
12119 4531 Acked Read 3 Sent Done Done false
12120 4532 Acked Read 5 Sent Done Done false
12121 4533 Acked Write 1 Sent Done Done false
12122 4534 Acked Flush 0 Sent Done Done false
12123 4535 Acked Flush 0 Sent Done Done false
12124 4536 Acked Write 9 Sent Done Done false
12125 4537 Acked Read 10 Sent Done Done false
12126 4538 Acked Write 10 Sent Done Done false
12127 4539 Acked Write 3 Sent Done Done false
12128 4540 Acked Flush 0 Sent Done Done false
12129 4541 Acked Read 5 Sent Done Done false
12130 4542 Acked Write 6 Sent Done Done false
12131 4543 Acked Read 10 Sent Done Done false
12132 4544 Acked Read 7 Sent Done Done false
12133 4545 Acked Read 6 Sent Done Done false
12134 4546 Acked Read 4 Sent Done Done false
12135 4547 Acked Read 6 Sent Done Done false
12136 4548 Acked Flush 0 Sent Done Done false
12137 4549 Acked Flush 0 Sent Done Done false
12138 4550 Acked Read 10 Sent Done Done false
12139 4551 Acked Write 3 Sent Done Done false
12140 4552 Acked Read 9 Sent Done Done false
12141 4553 Acked Flush 0 Sent Done Done false
12142 4554 Acked Write 5 Sent Done Done false
12143 4555 Acked Read 8 Sent Done Done false
12144 4556 Acked Write 2 Sent Done Done false
12145 4557 Acked Write 7 Sent Done Done false
12146 4558 Acked Read 3 Sent Done Done false
12147 4559 Acked Write 5 Sent Done Done false
12148 4560 Acked Read 3 Sent Done Done false
12149 4561 Acked Write 4 Sent Done Done false
12150 4562 Acked Write 8 Sent Done Done false
12151 4563 Acked Read 4 Sent Done Done false
12152 4564 Acked Read 4 Sent Done Done false
12153 4565 Acked Read 10 Sent Done Done false
12154 4566 Acked Read 6 Sent Done Done false
12155 4567 Acked Read 10 Sent Done Done false
12156 4568 Acked Read 7 Sent Done Done false
12157 4569 Acked Read 3 Sent Done Done false
12158 4570 Acked Write 7 Sent Done Done false
12159 4571 Acked Write 6 Sent Done Done false
12160 4572 Acked Write 9 Sent Done Done false
12161 4573 Acked Read 8 Sent Done Done false
12162 4574 Acked Read 2 Sent Done Done false
12163 4575 Acked Read 2 Sent Done Done false
12164 4576 Acked Write 8 Sent Done Done false
12165 4577 Acked Read 5 Sent Done Done false
12166 4578 Acked Flush 0 Sent Done Done false
12167 4579 Acked Write 4 Sent Done Done false
12168 4580 Acked Flush 0 Sent Done Done false
12169 4581 Acked Write 10 Sent Done Done false
12170 4582 Acked Read 1 Sent Done Done false
12171 4583 Acked Write 4 Sent Done Done false
12172 4584 Acked Write 10 Sent Done Done false
12173 4585 Acked Read 9 Sent Done Done false
12174 4586 Acked Write 3 Sent Done Done false
12175 4587 Acked Read 9 Sent Done Done false
12176 4588 Acked Read 8 Sent Done Done false
12177 4589 Acked Write 3 Sent Done Done false
12178 4590 Acked Read 7 Sent Done Done false
12179 4591 Acked Write 4 Sent Done Done false
12180 4592 Acked Write 7 Sent Done Done false
12181 4593 Acked Flush 0 Sent Done Done false
12182 4594 Acked Write 5 Sent Done Done false
12183 4595 Acked Flush 0 Sent Done Done false
12184 4596 Acked Read 4 Sent Done Done false
12185 4597 Acked Write 3 Sent Done Done false
12186 4598 Acked Read 2 Sent Done Done false
12187 4599 Acked Write 7 Sent Done Done false
12188 4600 Acked Read 2 Sent Done Done false
12189 4601 Acked Read 6 Sent Done Done false
12190 4602 Acked Read 3 Sent Done Done false
12191 4603 Acked Read 8 Sent Done Done false
12192 4604 Acked Read 3 Sent Done Done false
12193 4605 Acked Read 3 Sent Done Done false
12194 4606 Acked Write 1 Sent Done Done false
12195 4607 Acked Read 2 Sent Done Done false
12196 4608 Acked Read 8 Sent Done Done false
12197 4609 Acked Read 7 Sent Done Done false
12198 4610 Acked Read 8 Sent Done Done false
12199 4611 Acked Write 9 Sent Done Done false
12200 4612 Acked Read 10 Sent Done Done false
12201 4613 Acked Read 9 Sent Done Done false
12202 4614 Acked Read 8 Sent Done Done false
12203 4615 Acked Read 2 Sent Done Done false
12204 4616 Acked Write 7 Sent Done Done false
12205 4617 Acked Write 10 Sent Done Done false
12206 4618 Acked Write 2 Sent Done Done false
12207 4619 Acked Read 2 Sent Done Done false
12208 4620 Acked Read 2 Sent Done Done false
12209 4621 Acked Write 3 Sent Done Done false
12210 4622 Acked Read 7 Sent Done Done false
12211 STATES DS:0 DS:1 DS:2 TOTAL
12212 Sent 301 0 0 301
12213 Done 0 301 301 602
12214 Skipped 0 0 0 0
12215 Error 0 0 0 0
12216 Last Flush: 4321 4595 4595
12217 Downstairs last five completed: 4321 4320 4319 4318 4317
12218 Upstairs last five completed: 4622 4621 4620 4619 4618
12219 CLIENT: Up:0 ds:301 act:2
12220 [8810] got start action
12221 [8810] I am going to restart
12222 Starting downstairs at port 8810
12223 Make output file at "/tmp/dsc/downstairs-8810.txt"
12224 [8810][0] reports Starting
12225 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1213)
12226 [8810][0] reports Running
12227 ----------------------------------------------------------------
12228 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
12229 States: Offline Active Active
12230 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12231 4322 Acked Write 2 Sent Done Done false
12232 4323 Acked Write 9 Sent Done Done false
12233 4324 Acked Read 2 Sent Done Done false
12234 4325 Acked Read 5 Sent Done Done false
12235 4326 Acked Read 4 Sent Done Done false
12236 4327 Acked Read 6 Sent Done Done false
12237 4328 Acked Write 2 Sent Done Done false
12238 4329 Acked Read 7 Sent Done Done false
12239 4330 Acked Write 4 Sent Done Done false
12240 4331 Acked Read 5 Sent Done Done false
12241 4332 Acked Read 1 Sent Done Done false
12242 4333 Acked Write 2 Sent Done Done false
12243 4334 Acked Read 9 Sent Done Done false
12244 4335 Acked Read 9 Sent Done Done false
12245 4336 Acked Read 2 Sent Done Done false
12246 4337 Acked Write 9 Sent Done Done false
12247 4338 Acked Read 2 Sent Done Done false
12248 4339 Acked Write 4 Sent Done Done false
12249 4340 Acked Write 4 Sent Done Done false
12250 4341 Acked Write 2 Sent Done Done false
12251 4342 Acked Flush 0 Sent Done Done false
12252 4343 Acked Write 2 Sent Done Done false
12253 4344 Acked Read 10 Sent Done Done false
12254 4345 Acked Read 6 Sent Done Done false
12255 4346 Acked Read 1 Sent Done Done false
12256 4347 Acked Write 4 Sent Done Done false
12257 4348 Acked Flush 0 Sent Done Done false
12258 4349 Acked Write 1 Sent Done Done false
12259 4350 Acked Read 1 Sent Done Done false
12260 4351 Acked Write 6 Sent Done Done false
12261 4352 Acked Flush 0 Sent Done Done false
12262 4353 Acked Flush 0 Sent Done Done false
12263 4354 Acked Flush 0 Sent Done Done false
12264 4355 Acked Read 6 Sent Done Done false
12265 4356 Acked Read 3 Sent Done Done false
12266 4357 Acked Read 2 Sent Done Done false
12267 4358 Acked Write 5 Sent Done Done false
12268 4359 Acked Write 7 Sent Done Done false
12269 4360 Acked Read 1 Sent Done Done false
12270 4361 Acked Write 10 Sent Done Done false
12271 4362 Acked Read 8 Sent Done Done false
12272 4363 Acked Read 3 Sent Done Done false
12273 4364 Acked Write 1 Sent Done Done false
12274 4365 Acked Write 2 Sent Done Done false
12275 4366 Acked Write 1 Sent Done Done false
12276 4367 Acked Read 3 Sent Done Done false
12277 4368 Acked Read 3 Sent Done Done false
12278 4369 Acked Read 3 Sent Done Done false
12279 4370 Acked Read 1 Sent Done Done false
12280 4371 Acked Write 3 Sent Done Done false
12281 4372 Acked Write 10 Sent Done Done false
12282 4373 Acked Write 5 Sent Done Done false
12283 4374 Acked Read 2 Sent Done Done false
12284 4375 Acked Flush 0 Sent Done Done false
12285 4376 Acked Flush 0 Sent Done Done false
12286 4377 Acked Write 7 Sent Done Done false
12287 4378 Acked Read 1 Sent Done Done false
12288 4379 Acked Write 4 Sent Done Done false
12289 4380 Acked Write 7 Sent Done Done false
12290 4381 Acked Write 4 Sent Done Done false
12291 4382 Acked Read 10 Sent Done Done false
12292 4383 Acked Read 3 Sent Done Done false
12293 4384 Acked Write 10 Sent Done Done false
12294 4385 Acked Read 8 Sent Done Done false
12295 4386 Acked Write 8 Sent Done Done false
12296 4387 Acked Write 5 Sent Done Done false
12297 4388 Acked Flush 0 Sent Done Done false
12298 4389 Acked Write 4 Sent Done Done false
12299 4390 Acked Write 2 Sent Done Done false
12300 4391 Acked Flush 0 Sent Done Done false
12301 4392 Acked Flush 0 Sent Done Done false
12302 4393 Acked Read 4 Sent Done Done false
12303 4394 Acked Write 1 Sent Done Done false
12304 4395 Acked Read 4 Sent Done Done false
12305 4396 Acked Write 10 Sent Done Done false
12306 4397 Acked Read 7 Sent Done Done false
12307 4398 Acked Read 3 Sent Done Done false
12308 4399 Acked Write 10 Sent Done Done false
12309 4400 Acked Read 5 Sent Done Done false
12310 4401 Acked Read 5 Sent Done Done false
12311 4402 Acked Flush 0 Sent Done Done false
12312 4403 Acked Read 5 Sent Done Done false
12313 4404 Acked Write 7 Sent Done Done false
12314 4405 Acked Write 2 Sent Done Done false
12315 4406 Acked Write 7 Sent Done Done false
12316 4407 Acked Read 3 Sent Done Done false
12317 4408 Acked Read 2 Sent Done Done false
12318 4409 Acked Write 5 Sent Done Done false
12319 4410 Acked Write 8 Sent Done Done false
12320 4411 Acked Write 9 Sent Done Done false
12321 4412 Acked Read 8 Sent Done Done false
12322 4413 Acked Flush 0 Sent Done Done false
12323 4414 Acked Write 5 Sent Done Done false
12324 4415 Acked Read 7 Sent Done Done false
12325 4416 Acked Read 4 Sent Done Done false
12326 4417 Acked Flush 0 Sent Done Done false
12327 4418 Acked Write 8 Sent Done Done false
12328 4419 Acked Write 2 Sent Done Done false
12329 4420 Acked Read 3 Sent Done Done false
12330 4421 Acked Read 4 Sent Done Done false
12331 4422 Acked Write 4 Sent Done Done false
12332 4423 Acked Write 3 Sent Done Done false
12333 4424 Acked Write 5 Sent Done Done false
12334 4425 Acked Flush 0 Sent Done Done false
12335 4426 Acked Write 10 Sent Done Done false
12336 4427 Acked Read 9 Sent Done Done false
12337 4428 Acked Read 3 Sent Done Done false
12338 4429 Acked Write 4 Sent Done Done false
12339 4430 Acked Read 9 Sent Done Done false
12340 4431 Acked Write 10 Sent Done Done false
12341 4432 Acked Flush 0 Sent Done Done false
12342 4433 Acked Write 4 Sent Done Done false
12343 4434 Acked Flush 0 Sent Done Done false
12344 4435 Acked Write 5 Sent Done Done false
12345 4436 Acked Write 5 Sent Done Done false
12346 4437 Acked Write 3 Sent Done Done false
12347 4438 Acked Write 6 Sent Done Done false
12348 4439 Acked Read 7 Sent Done Done false
12349 4440 Acked Read 4 Sent Done Done false
12350 4441 Acked Read 7 Sent Done Done false
12351 4442 Acked Write 9 Sent Done Done false
12352 4443 Acked Write 1 Sent Done Done false
12353 4444 Acked Write 10 Sent Done Done false
12354 4445 Acked Write 7 Sent Done Done false
12355 4446 Acked Read 10 Sent Done Done false
12356 4447 Acked Write 8 Sent Done Done false
12357 4448 Acked Write 7 Sent Done Done false
12358 4449 Acked Flush 0 Sent Done Done false
12359 4450 Acked Write 4 Sent Done Done false
12360 4451 Acked Read 7 Sent Done Done false
12361 4452 Acked Read 8 Sent Done Done false
12362 4453 Acked Read 9 Sent Done Done false
12363 4454 Acked Write 7 Sent Done Done false
12364 4455 Acked Write 3 Sent Done Done false
12365 4456 Acked Read 3 Sent Done Done false
12366 4457 Acked Read 3 Sent Done Done false
12367 4458 Acked Read 1 Sent Done Done false
12368 4459 Acked Read 10 Sent Done Done false
12369 4460 Acked Read 8 Sent Done Done false
12370 4461 Acked Write 5 Sent Done Done false
12371 4462 Acked Read 8 Sent Done Done false
12372 4463 Acked Flush 0 Sent Done Done false
12373 4464 Acked Write 4 Sent Done Done false
12374 4465 Acked Read 7 Sent Done Done false
12375 4466 Acked Write 5 Sent Done Done false
12376 4467 Acked Write 3 Sent Done Done false
12377 4468 Acked Write 1 Sent Done Done false
12378 4469 Acked Read 5 Sent Done Done false
12379 4470 Acked Flush 0 Sent Done Done false
12380 4471 Acked Read 8 Sent Done Done false
12381 4472 Acked Write 4 Sent Done Done false
12382 4473 Acked Write 2 Sent Done Done false
12383 4474 Acked Write 2 Sent Done Done false
12384 4475 Acked Read 8 Sent Done Done false
12385 4476 Acked Read 7 Sent Done Done false
12386 4477 Acked Flush 0 Sent Done Done false
12387 4478 Acked Write 2 Sent Done Done false
12388 4479 Acked Read 9 Sent Done Done false
12389 4480 Acked Write 5 Sent Done Done false
12390 4481 Acked Read 9 Sent Done Done false
12391 4482 Acked Write 1 Sent Done Done false
12392 4483 Acked Write 3 Sent Done Done false
12393 4484 Acked Write 1 Sent Done Done false
12394 4485 Acked Read 5 Sent Done Done false
12395 4486 Acked Flush 0 Sent Done Done false
12396 4487 Acked Read 2 Sent Done Done false
12397 4488 Acked Write 9 Sent Done Done false
12398 4489 Acked Read 10 Sent Done Done false
12399 4490 Acked Write 6 Sent Done Done false
12400 4491 Acked Read 6 Sent Done Done false
12401 4492 Acked Read 6 Sent Done Done false
12402 4493 Acked Write 4 Sent Done Done false
12403 4494 Acked Write 6 Sent Done Done false
12404 4495 Acked Flush 0 Sent Done Done false
12405 4496 Acked Read 6 Sent Done Done false
12406 4497 Acked Flush 0 Sent Done Done false
12407 4498 Acked Read 6 Sent Done Done false
12408 4499 Acked Flush 0 Sent Done Done false
12409 4500 Acked Read 2 Sent Done Done false
12410 4501 Acked Read 4 Sent Done Done false
12411 4502 Acked Read 2 Sent Done Done false
12412 4503 Acked Read 3 Sent Done Done false
12413 4504 Acked Read 7 Sent Done Done false
12414 4505 Acked Read 5 Sent Done Done false
12415 4506 Acked Write 5 Sent Done Done false
12416 4507 Acked Flush 0 Sent Done Done false
12417 4508 Acked Read 6 Sent Done Done false
12418 4509 Acked Read 6 Sent Done Done false
12419 4510 Acked Write 5 Sent Done Done false
12420 4511 Acked Write 9 Sent Done Done false
12421 4512 Acked Read 8 Sent Done Done false
12422 4513 Acked Write 5 Sent Done Done false
12423 4514 Acked Write 6 Sent Done Done false
12424 4515 Acked Write 5 Sent Done Done false
12425 4516 Acked Read 9 Sent Done Done false
12426 4517 Acked Read 3 Sent Done Done false
12427 4518 Acked Read 6 Sent Done Done false
12428 4519 Acked Write 3 Sent Done Done false
12429 4520 Acked Read 7 Sent Done Done false
12430 4521 Acked Write 7 Sent Done Done false
12431 4522 Acked Read 10 Sent Done Done false
12432 4523 Acked Read 7 Sent Done Done false
12433 4524 Acked Read 2 Sent Done Done false
12434 4525 Acked Flush 0 Sent Done Done false
12435 4526 Acked Read 9 Sent Done Done false
12436 4527 Acked Write 8 Sent Done Done false
12437 4528 Acked Write 4 Sent Done Done false
12438 4529 Acked Write 10 Sent Done Done false
12439 4530 Acked Flush 0 Sent Done Done false
12440 4531 Acked Read 3 Sent Done Done false
12441 4532 Acked Read 5 Sent Done Done false
12442 4533 Acked Write 1 Sent Done Done false
12443 4534 Acked Flush 0 Sent Done Done false
12444 4535 Acked Flush 0 Sent Done Done false
12445 4536 Acked Write 9 Sent Done Done false
12446 4537 Acked Read 10 Sent Done Done false
12447 4538 Acked Write 10 Sent Done Done false
12448 4539 Acked Write 3 Sent Done Done false
12449 4540 Acked Flush 0 Sent Done Done false
12450 4541 Acked Read 5 Sent Done Done false
12451 4542 Acked Write 6 Sent Done Done false
12452 4543 Acked Read 10 Sent Done Done false
12453 4544 Acked Read 7 Sent Done Done false
12454 4545 Acked Read 6 Sent Done Done false
12455 4546 Acked Read 4 Sent Done Done false
12456 4547 Acked Read 6 Sent Done Done false
12457 4548 Acked Flush 0 Sent Done Done false
12458 4549 Acked Flush 0 Sent Done Done false
12459 4550 Acked Read 10 Sent Done Done false
12460 4551 Acked Write 3 Sent Done Done false
12461 4552 Acked Read 9 Sent Done Done false
12462 4553 Acked Flush 0 Sent Done Done false
12463 4554 Acked Write 5 Sent Done Done false
12464 4555 Acked Read 8 Sent Done Done false
12465 4556 Acked Write 2 Sent Done Done false
12466 4557 Acked Write 7 Sent Done Done false
12467 4558 Acked Read 3 Sent Done Done false
12468 4559 Acked Write 5 Sent Done Done false
12469 4560 Acked Read 3 Sent Done Done false
12470 4561 Acked Write 4 Sent Done Done false
12471 4562 Acked Write 8 Sent Done Done false
12472 4563 Acked Read 4 Sent Done Done false
12473 4564 Acked Read 4 Sent Done Done false
12474 4565 Acked Read 10 Sent Done Done false
12475 4566 Acked Read 6 Sent Done Done false
12476 4567 Acked Read 10 Sent Done Done false
12477 4568 Acked Read 7 Sent Done Done false
12478 4569 Acked Read 3 Sent Done Done false
12479 4570 Acked Write 7 Sent Done Done false
12480 4571 Acked Write 6 Sent Done Done false
12481 4572 Acked Write 9 Sent Done Done false
12482 4573 Acked Read 8 Sent Done Done false
12483 4574 Acked Read 2 Sent Done Done false
12484 4575 Acked Read 2 Sent Done Done false
12485 4576 Acked Write 8 Sent Done Done false
12486 4577 Acked Read 5 Sent Done Done false
12487 4578 Acked Flush 0 Sent Done Done false
12488 4579 Acked Write 4 Sent Done Done false
12489 4580 Acked Flush 0 Sent Done Done false
12490 4581 Acked Write 10 Sent Done Done false
12491 4582 Acked Read 1 Sent Done Done false
12492 4583 Acked Write 4 Sent Done Done false
12493 4584 Acked Write 10 Sent Done Done false
12494 4585 Acked Read 9 Sent Done Done false
12495 4586 Acked Write 3 Sent Done Done false
12496 4587 Acked Read 9 Sent Done Done false
12497 4588 Acked Read 8 Sent Done Done false
12498 4589 Acked Write 3 Sent Done Done false
12499 4590 Acked Read 7 Sent Done Done false
12500 4591 Acked Write 4 Sent Done Done false
12501 4592 Acked Write 7 Sent Done Done false
12502 4593 Acked Flush 0 Sent Done Done false
12503 4594 Acked Write 5 Sent Done Done false
12504 4595 Acked Flush 0 Sent Done Done false
12505 4596 Acked Read 4 Sent Done Done false
12506 4597 Acked Write 3 Sent Done Done false
12507 4598 Acked Read 2 Sent Done Done false
12508 4599 Acked Write 7 Sent Done Done false
12509 4600 Acked Read 2 Sent Done Done false
12510 4601 Acked Read 6 Sent Done Done false
12511 4602 Acked Read 3 Sent Done Done false
12512 4603 Acked Read 8 Sent Done Done false
12513 4604 Acked Read 3 Sent Done Done false
12514 4605 Acked Read 3 Sent Done Done false
12515 4606 Acked Write 1 Sent Done Done false
12516 4607 Acked Read 2 Sent Done Done false
12517 4608 Acked Read 8 Sent Done Done false
12518 4609 Acked Read 7 Sent Done Done false
12519 4610 Acked Read 8 Sent Done Done false
12520 4611 Acked Write 9 Sent Done Done false
12521 4612 Acked Read 10 Sent Done Done false
12522 4613 Acked Read 9 Sent Done Done false
12523 4614 Acked Read 8 Sent Done Done false
12524 4615 Acked Read 2 Sent Done Done false
12525 4616 Acked Write 7 Sent Done Done false
12526 4617 Acked Write 10 Sent Done Done false
12527 4618 Acked Write 2 Sent Done Done false
12528 4619 Acked Read 2 Sent Done Done false
12529 4620 Acked Read 2 Sent Done Done false
12530 4621 Acked Write 3 Sent Done Done false
12531 4622 Acked Read 7 Sent Done Done false
12532 4623 Acked Flush 0 Sent Done Done false
12533 STATES DS:0 DS:1 DS:2 TOTAL
12534 Sent 302 0 0 302
12535 Done 0 302 302 604
12536 Skipped 0 0 0 0
12537 Error 0 0 0 0
12538 Last Flush: 4321 4623 4623
12539 Downstairs last five completed: 4321 4320 4319 4318 4317
12540 Upstairs last five completed: 4623 4622 4621 4620 4619
12541 CLIENT: Up:0 ds:302 act:2
125422024-12-09T15:10:27.827ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
125432024-12-09T15:10:27.827ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33759) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
125442024-12-09T15:10:27.828ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
125452024-12-09T15:10:27.828ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
125462024-12-09T15:10:27.828ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
125472024-12-09T15:10:27.828ZINFOcrucible: send last flush ID to this DS: 4321 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
125482024-12-09T15:10:27.829ZINFOcrucible: Replied this last flush ID: 4321 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
125492024-12-09T15:10:27.829ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
125502024-12-09T15:10:27.829ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
125512024-12-09T15:10:27.829ZINFOcrucible: [0] client re-new 302 jobs since flush 4321 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
125522024-12-09T15:10:27.829ZINFOcrucible: [0] Replayed 302 jobs since flush: 4321 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
12553 ----------------------------------------------------------------
12554 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12555 Downstairs last five completed: 4623 4622 4621 4620 4619
12556 Upstairs last five completed: 4623 4622 4621 4620 4619
12557 CLIENT: Up:0 ds:0 act:3
12558 Replay: All jobs finished
12559 Main task has work to do, go find it
12560 got dsc Stop(0)
12561 stop 0
12562 [8810] Got stop action so:false kr:false
12563 Dec 09 15:10:30.064 INFO request completed, latency_us: 362, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 20359c3b-d1ea-4c59-a9ce-53dac3f17899, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
125642024-12-09T15:10:30.065ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
12565 Dec 09 15:10:30.065 INFO request completed, latency_us: 114, response_code: 200, uri: /state/cid/0, method: GET, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:10:30.065718446Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"0","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
12566 req_id: 168bb3b3-15a7-46ba-a6a0-48f6bdd834e7, remote_addr: 127.0.0.1:47510, local_addr: 127{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:10:30.065750741Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
12567 .0.0.1:9998
125682024-12-09T15:10:30.065ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
125692024-12-09T15:10:30.065ZINFOcrucible: [0] Transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
125702024-12-09T15:10:30.065ZINFOcrucible: client task is exiting = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
12571 [8810] Exited with: ExitStatus(unix_wait_status(9))
12572 [8810][0] reports Exit
12573 Dec 09 15:10:34.067 INFO request completed, latency_us: 231, response_code: 200, uri: /state/cid/0, method: GET, req_id: ce685ed5-efe8-45ab-8a2b-b0794b0c3741, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
12574 001/300 Flush
12575 002/300 Read block 725 len 20480
12576 003/300 Read block 4660 len 12288
12577 004/300 Write block 4620 len 36864 data: 3 3 4 5 4 5 6 5 5
12578 005/300 Read block 4264 len 40960
12579 006/300 Write block 1161 len 32768 data: 3 3 3 3 3 3 3 2
12580 007/300 Write block 2480 len 32768 data: 4 3 3 3 3 3 3 2
12581 008/300 Write block 924 len 12288 data: 4 5 6
12582 009/300 Write block 3136 len 4096 data: 3
12583 010/300 Write block 1382 len 24576 data: 4 3 3 3 4 4
12584 011/300 Read block 4193 len 4096
12585 012/300 Flush
12586 013/300 Read block 2910 len 20480
12587 014/300 Write block 1028 len 32768 data: 3 3 3 3 3 2 2 3
12588 015/300 Read block 298 len 12288
12589 016/300 Read block 4066 len 16384
12590 017/300 Read block 830 len 32768
12591 018/300 Write block 2762 len 20480 data: 3 3 3 3 3
12592 019/300 Read block 3333 len 36864
12593 020/300 Read block 3349 len 12288
12594 021/300 Flush
12595 022/300 Write block 192 len 4096 data: 5
12596 023/300 Read block 3522 len 4096
12597 024/300 Read block 733 len 28672
12598 025/300 Read block 4941 len 32768
12599 026/300 Write block 4135 len 20480 data: 2 2 2 3 3
12600 027/300 Write block 2000 len 24576 data: 4 4 4 3 4 4
12601 028/300 Write block 3869 len 8192 data: 2 4
12602 029/300 Write block 4212 len 36864 data: 2 2 2 2 2 2 3 3 3
12603 030/300 Read block 3593 len 36864
12604 031/300 Write block 1564 len 16384 data: 8 6 5 4
12605 032/300 Read block 475 len 4096
12606 033/300 Read block 2923 len 36864
12607 034/300 Read block 4886 len 16384
12608 035/300 Read block 3783 len 8192
12609 036/300 Write block 4349 len 24576 data: 5 4 4 3 3 3
12610 037/300 Write block 1413 len 4096 data: 6
12611 038/300 Read block 1804 len 4096
12612 039/300 Write block 800 len 40960 data: 4 7 7 7 6 4 4 3 4 4
12613 040/300 Read block 3141 len 20480
12614 041/300 Read block 1157 len 8192
12615 042/300 Write block 4453 len 8192 data: 3 3
12616 043/300 Read block 2760 len 24576
12617 044/300 Write block 512 len 32768 data: 3 4 4 5 4 5 4 3
12618 045/300 Flush
12619 046/300 Read block 3967 len 28672
12620 047/300 Read block 4366 len 28672
12621 048/300 Read block 1968 len 8192
12622 049/300 Read block 1432 len 16384
12623 050/300 Read block 1813 len 20480
12624 051/300 Read block 2394 len 8192
12625 052/300 Flush
12626 053/300 Write block 4964 len 32768 data: 5 3 2 2 2 2 2 2
12627 054/300 Write block 860 len 8192 data: 5 4
12628 055/300 Read block 2529 len 36864
12629 056/300 Write block 3930 len 36864 data: 4 3 4 3 3 3 3 3 4
12630 057/300 Read block 1811 len 16384
12631 058/300 Write block 2196 len 32768 data: 4 3 2 3 3 4 5 5
12632 059/300 Read block 289 len 12288
12633 060/300 Read block 2296 len 16384
12634 061/300 Write block 4842 len 36864 data: 6 6 4 3 2 2 2 2 2
12635 062/300 Read block 3640 len 16384
12636 063/300 Read block 15 len 20480
12637 064/300 Read block 1779 len 24576
12638 065/300 Write block 180 len 40960 data: 4 3 3 3 3 4 4 4 4 4
12639 066/300 Read block 4314 len 20480
12640 067/300 Read block 3994 len 20480
12641 068/300 Flush
12642 069/300 Read block 1540 len 16384
12643 070/300 Write block 1291 len 28672 data: 6 6 6 5 5 4 5
12644 071/300 Flush
12645 072/300 Write block 126 len 12288 data: 3 3 3
12646 073/300 Read block 2438 len 32768
12647 074/300 Write block 1737 len 16384 data: 3 3 3 5
12648 075/300 Read block 875 len 36864
12649 076/300 Read block 3549 len 8192
12650 077/300 Read block 4287 len 8192
12651 078/300 Write block 1147 len 16384 data: 5 6 7 7
12652 079/300 Flush
12653 080/300 Read block 790 len 28672
12654 081/300 Read block 4919 len 24576
12655 082/300 Read block 3674 len 36864
12656 083/300 Read block 2716 len 24576
12657 084/300 Read block 2599 len 8192
12658 085/300 Flush
12659 086/300 Read block 768 len 24576
12660 087/300 Read block 2088 len 4096
12661 088/300 Read block 246 len 36864
12662 089/300 Write block 1652 len 40960 data: 4 4 4 4 4 3 3 3 4 3
12663 090/300 Write block 2740 len 40960 data: 4 3 3 3 3 3 3 2 2 3
12664 091/300 Read block 1986 len 16384
12665 092/300 Read block 2087 len 36864
12666 093/300 Read block 1068 len 20480
12667 094/300 Read block 2741 len 28672
12668 095/300 Read block 3175 len 28672
12669 096/300 Read block 1934 len 32768
12670 097/300 Read block 502 len 8192
12671 098/300 Read block 3072 len 36864
12672 099/300 Write block 4520 len 28672 data: 2 2 2 2 4 4 4
12673 100/300 Write block 477 len 32768 data: 3 4 4 4 4 6 5 5
12674 101/300 Write block 1323 len 20480 data: 3 3 2 3 3
12675 102/300 Write block 3792 len 8192 data: 3 2
12676 103/300 Read block 1717 len 36864
12677 104/300 Write block 1247 len 4096 data: 5
12678 105/300 Write block 3151 len 24576 data: 7 6 5 5 5 5
12679 106/300 Read block 1245 len 36864
12680 107/300 Read block 817 len 40960
12681 108/300 Flush
12682 109/300 Write block 1709 len 36864 data: 4 5 5 5 6 5 5 6 5
12683 110/300 Write block 1102 len 40960 data: 2 2 2 3 4 4 5 5 4 4
12684 111/300 Flush
12685 112/300 Write block 178 len 8192 data: 4 4
12686 113/300 Write block 1817 len 16384 data: 4 4 5 5
12687 114/300 Read block 1563 len 12288
12688 115/300 Read block 3444 len 28672
12689 116/300 Read block 2648 len 28672
12690 117/300 Flush
12691 118/300 Read block 1009 len 32768
12692 119/300 Read block 4895 len 8192
12693 120/300 Write block 988 len 16384 data: 5 6 6 5
12694 121/300 Write block 1213 len 36864 data: 6 5 4 3 4 4 4 4 4
12695 122/300 Write block 2126 len 24576 data: 3 3 5 5 5 7
12696 123/300 Read block 2124 len 8192
12697 124/300 Write block 4830 len 4096 data: 2
12698 125/300 Read block 68 len 20480
12699 126/300 Read block 2260 len 36864
12700 127/300 Read block 3563 len 12288
12701 128/300 Read block 2633 len 8192
12702 129/300 Read block 1171 len 4096
12703 130/300 Write block 4888 len 32768 data: 4 4 3 3 3 4 4 3
12704 131/300 Read block 3669 len 40960
12705 132/300 Write block 4787 len 12288 data: 5 4 3
12706 133/300 Flush
12707 134/300 Write block 947 len 32768 data: 4 5 5 5 5 4 5 5
12708 135/300 Read block 4422 len 4096
12709 136/300 Write block 4381 len 40960 data: 7 4 5 4 4 4 5 4 3 3
12710 137/300 Write block 3081 len 8192 data: 3 2
12711 138/300 Write block 4641 len 36864 data: 4 4 4 4 4 5 6 6 6
12712 139/300 Read block 1921 len 40960
12713 140/300 Write block 862 len 4096 data: 4
12714 141/300 Read block 525 len 4096
12715 142/300 Read block 4211 len 20480
12716 143/300 Read block 3792 len 4096
12717 144/300 Write block 4223 len 24576 data: 5 5 5 5 4 5
12718 145/300 Read block 4835 len 24576
12719 146/300 Write block 3772 len 32768 data: 2 3 3 4 4 5 5 5
12720 147/300 Read block 2576 len 12288
12721 148/300 Write block 4096 len 24576 data: 7 8 8 7 5 6
12722 149/300 Write block 131 len 40960 data: 3 5 7 6 6 6 6 6 6 6
12723 150/300 Write block 4845 len 16384 data: 4 3 3 3
12724 151/300 Read block 4669 len 36864
12725 152/300 Write block 2483 len 20480 data: 4 4 4 4 3
12726 153/300 Read block 3417 len 32768
12727 154/300 Flush
12728 155/300 Write block 800 len 4096 data: 5
12729 156/300 Write block 2608 len 4096 data: 3
12730 157/300 Read block 3496 len 40960
12731 158/300 Write block 4938 len 24576 data: 3 3 3 3 3 3
12732 159/300 Flush
12733 160/300 Write block 2316 len 16384 data: 3 4 4 4
12734 161/300 Read block 3637 len 24576
12735 162/300 Read block 4559 len 20480
12736 163/300 Flush
12737 164/300 Read block 4216 len 12288
12738 165/300 Write block 1512 len 32768 data: 3 4 4 3 4 4 5 4
12739 166/300 Read block 1345 len 12288
12740 167/300 Write block 79 len 32768 data: 3 3 3 4 4 4 3 3
12741 168/300 Write block 3876 len 8192 data: 5 4
12742 169/300 Read block 4641 len 8192
12743 170/300 Read block 4605 len 8192
12744 171/300 Read block 2250 len 28672
12745 172/300 Read block 2121 len 16384
12746 173/300 Read block 3766 len 20480
12747 174/300 Write block 4083 len 16384 data: 6 5 5 5
12748 175/300 Read block 1039 len 20480
12749 176/300 Read block 3586 len 36864
12750 177/300 Flush
12751 178/300 Read block 4265 len 8192
12752 179/300 Write block 4493 len 32768 data: 5 4 3 3 3 3 5 5
12753 180/300 Write block 1337 len 4096 data: 7
12754 181/300 Write block 504 len 28672 data: 3 3 3 4 4 4 4
12755 182/300 Write block 931 len 8192 data: 3 4
12756 183/300 Write block 3226 len 32768 data: 3 3 3 2 2 2 3 3
12757 184/300 Flush
12758 185/300 Write block 2931 len 36864 data: 6 5 5 3 3 3 2 3 4
12759 186/300 Read block 3542 len 40960
12760 187/300 Write block 287 len 28672 data: 2 3 3 2 2 2 2
12761 188/300 Read block 1638 len 8192
12762 189/300 Write block 2772 len 36864 data: 6 7 8 7 6 6 6 6 6
12763 190/300 Write block 4179 len 24576 data: 4 4 4 5 5 5
12764 191/300 Read block 4032 len 32768
12765 192/300 Flush
12766 193/300 Read block 1279 len 16384
12767 194/300 Read block 3523 len 36864
12768 195/300 Flush
12769 196/300 Flush
12770 197/300 Write block 1356 len 4096 data: 6
12771 198/300 Write block 2149 len 4096 data: 4
12772 199/300 Write block 268 len 32768 data: 3 4 5 5 5 3 3 2
12773 200/300 Flush
12774 201/300 Read block 220 len 32768
12775 202/300 Write block 1467 len 16384 data: 2 3 3 3
12776 203/300 Read block 2060 len 40960
12777 204/300 Read block 1362 len 12288
12778 205/300 Read block 313 len 40960
12779 206/300 Write block 3980 len 28672 data: 4 3 4 3 3 4 4
12780 207/300 Write block 2079 len 20480 data: 3 5 4 4 3
12781 208/300 Write block 4519 len 4096 data: 4
12782 209/300 Write block 2717 len 40960 data: 6 5 4 3 2 2 2 2 2 2
12783 210/300 Write block 4011 len 16384 data: 3 3 3 3
12784 211/300 Read block 505 len 36864
12785 212/300 Read block 3837 len 16384
12786 213/300 Write block 4318 len 32768 data: 3 3 3 4 4 4 4 4
12787 214/300 Write block 3372 len 12288 data: 3 3 3
12788 215/300 Write block 3798 len 24576 data: 3 4 5 5 5 4
12789 216/300 Write block 2872 len 28672 data: 4 4 3 2 2 2 2
12790 217/300 Write block 2211 len 28672 data: 3 3 3 3 3 3 4
12791 218/300 Read block 4577 len 36864
12792 219/300 Write block 854 len 24576 data: 4 4 3 3 3 3
12793 220/300 Read block 2349 len 20480
12794 221/300 Read block 561 len 28672
12795 222/300 Write block 3348 len 40960 data: 4 3 3 3 3 2 2 3 3 3
12796 223/300 Read block 2910 len 20480
12797 224/300 Write block 1702 len 4096 data: 2
12798 225/300 Write block 774 len 20480 data: 4 4 4 3 4
12799 226/300 Read block 4815 len 16384
12800 227/300 Read block 2262 len 4096
12801 228/300 Write block 996 len 40960 data: 2 2 2 2 2 2 3 3 3 3
12802 229/300 Read block 4324 len 20480
12803 230/300 Read block 4848 len 24576
12804 231/300 Read block 2206 len 20480
12805 232/300 Read block 995 len 28672
12806 233/300 Write block 3923 len 36864 data: 3 3 2 2 3 3 4 5 4
12807 234/300 Write block 1189 len 24576 data: 3 3 3 3 2 3
12808 235/300 Flush
12809 236/300 Read block 1566 len 4096
12810 237/300 Write block 3899 len 36864 data: 2 2 2 2 2 3 3 3 3
12811 238/300 Write block 4223 len 20480 data: 6 6 6 6 5
12812 239/300 Write block 2298 len 16384 data: 3 3 2 2
12813 240/300 Read block 1785 len 28672
12814 241/300 Write block 1442 len 8192 data: 4 5
12815 242/300 Read block 3895 len 36864
12816 243/300 Write block 378 len 4096 data: 3
12817 244/300 Read block 4604 len 28672
12818 245/300 Write block 793 len 20480 data: 3 3 4 5 5
12819 246/300 Write block 3364 len 20480 data: 4 4 4 4 4
12820 247/300 Write block 1037 len 28672 data: 4 2 4 3 4 4 3
12821 248/300 Write block 2341 len 36864 data: 7 6 4 4 4 4 4 4 4
12822 249/300 Read block 4987 len 24576
12823 250/300 Write block 4550 len 32768 data: 5 5 5 5 5 6 7 6
12824 251/300 Read block 1044 len 28672
12825 252/300 Write block 2846 len 16384 data: 4 3 4 4
12826 253/300 Write block 1741 len 20480 data: 4 4 5 5 4
12827 254/300 Flush
12828 255/300 Read block 3183 len 36864
12829 256/300 Read block 3848 len 36864
12830 257/300 Write block 4322 len 8192 data: 5 5
12831 258/300 Write block 1127 len 24576 data: 5 4 4 3 3 5
12832 259/300 Read block 691 len 4096
12833 260/300 Write block 1601 len 24576 data: 3 4 4 2 3 3
12834 261/300 Read block 4219 len 12288
12835 262/300 Read block 3547 len 36864
12836 263/300 Read block 4765 len 28672
12837 264/300 Write block 4493 len 16384 data: 6 5 4 4
12838 265/300 Read block 1013 len 4096
12839 266/300 Read block 1050 len 24576
12840 267/300 Write block 2591 len 4096 data: 3
12841 268/300 Write block 4098 len 20480 data: 9 8 6 7 5
12842 269/300 Write block 2774 len 28672 data: 9 8 7 7 7 7 7
12843 270/300 Write block 413 len 16384 data: 3 3 4 4
12844 271/300 Write block 2105 len 36864 data: 5 5 6 5 4 2 2 3 3
12845 272/300 Read block 3118 len 40960
12846 273/300 Write block 1450 len 16384 data: 4 5 5 5
12847 274/300 Read block 2658 len 28672
12848 275/300 Read block 1663 len 16384
12849 276/300 Write block 1702 len 16384 data: 3 2 2 2
12850 277/300 Read block 3997 len 32768
12851 278/300 Write block 4948 len 36864 data: 4 5 5 4 2 2 2 2 2
12852 279/300 Read block 3058 len 20480
12853 280/300 Write block 3314 len 4096 data: 3
12854 281/300 Write block 2715 len 24576 data: 6 6 7 6 5 4
12855 282/300 Read block 2456 len 36864
12856 283/300 Read block 2792 len 20480
12857 284/300 Read block 3061 len 16384
12858 285/300 Read block 230 len 16384
12859 286/300 Read block 1435 len 8192
12860 287/300 Write block 1030 len 40960 data: 4 4 4 3 3 4 3 5 3 5
12861 288/300 Write block 630 len 32768 data: 5 5 4 4 3 4 4 5
12862 289/300 Read block 3753 len 16384
12863 290/300 Write block 2282 len 4096 data: 3
12864 291/300 Read block 44 len 40960
12865 292/300 Read block 4673 len 12288
12866 293/300 Write block 2452 len 32768 data: 2 2 3 3 4 4 5 5
12867 294/300 Read block 1084 len 24576
12868 295/300 Read block 4672 len 36864
12869 296/300 Write block 3720 len 36864 data: 3 4 3 3 3 3 4 4 4
12870 297/300 Write block 4924 len 8192 data: 2 2
12871 298/300 Read block 4706 len 40960
12872 299/300 Write block 357 len 32768 data: 5 5 5 7 7 7 6 6
12873 300/300 Read block 2443 len 36864
12874 301/300 Write block 169 len 20480 data: 4 4 4 3 2
12875 Main task has work to do, go find it
12876 got dsc Start(0)
12877 start 0
12878 Dec 09 15:10:34.304 INFO request completed, latency_us: 265, response_code: 204, uri: /start/cid/0, method: POST, req_id: 59554802-01a7-45fe-b044-2dd2a0408e6b, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
12879 Replay: started 0, returned:Ok(())
12880 ----------------------------------------------------------------
12881 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12882 States: Offline Active Active
12883 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12884 4624 Acked Flush 0 Sent Done Done false
12885 4625 Acked Read 5 Sent Done Done false
12886 4626 Acked Read 3 Sent Done Done false
12887 4627 Acked Write 9 Sent Done Done false
12888 4628 Acked Read 10 Sent Done Done false
12889 4629 Acked Write 8 Sent Done Done false
12890 4630 Acked Write 8 Sent Done Done false
12891 4631 Acked Write 3 Sent Done Done false
12892 4632 Acked Write 1 Sent Done Done false
12893 4633 Acked Write 6 Sent Done Done false
12894 4634 Acked Read 1 Sent Done Done false
12895 4635 Acked Flush 0 Sent Done Done false
12896 4636 Acked Read 5 Sent Done Done false
12897 4637 Acked Write 8 Sent Done Done false
12898 4638 Acked Read 3 Sent Done Done false
12899 4639 Acked Read 4 Sent Done Done false
12900 4640 Acked Read 8 Sent Done Done false
12901 4641 Acked Write 5 Sent Done Done false
12902 4642 Acked Read 9 Sent Done Done false
12903 4643 Acked Read 3 Sent Done Done false
12904 4644 Acked Flush 0 Sent Done Done false
12905 4645 Acked Write 1 Sent Done Done false
12906 4646 Acked Read 1 Sent Done Done false
12907 4647 Acked Read 7 Sent Done Done false
12908 4648 Acked Read 8 Sent Done Done false
12909 4649 Acked Write 5 Sent Done Done false
12910 4650 Acked Write 6 Sent Done Done false
12911 4651 Acked Write 2 Sent Done Done false
12912 4652 Acked Write 9 Sent Done Done false
12913 4653 Acked Read 9 Sent Done Done false
12914 4654 Acked Write 4 Sent Done Done false
12915 4655 Acked Read 1 Sent Done Done false
12916 4656 Acked Read 9 Sent Done Done false
12917 4657 Acked Read 4 Sent Done Done false
12918 4658 Acked Read 2 Sent Done Done false
12919 4659 Acked Write 6 Sent Done Done false
12920 4660 Acked Write 1 Sent Done Done false
12921 4661 Acked Read 1 Sent Done Done false
12922 4662 Acked Write 10 Sent Done Done false
12923 4663 Acked Read 5 Sent Done Done false
12924 4664 Acked Read 2 Sent Done Done false
12925 4665 Acked Write 2 Sent Done Done false
12926 4666 Acked Read 6 Sent Done Done false
12927 4667 Acked Write 8 Sent Done Done false
12928 4668 Acked Flush 0 Sent Done Done false
12929 4669 Acked Read 7 Sent Done Done false
12930 4670 Acked Read 7 Sent Done Done false
12931 4671 Acked Read 2 Sent Done Done false
12932 4672 Acked Read 4 Sent Done Done false
12933 4673 Acked Read 5 Sent Done Done false
12934 4674 Acked Read 2 Sent Done Done false
12935 4675 Acked Flush 0 Sent Done Done false
12936 4676 Acked Write 8 Sent Done Done false
12937 4677 Acked Write 2 Sent Done Done false
12938 4678 Acked Read 9 Sent Done Done false
12939 4679 Acked Write 9 Sent Done Done false
12940 4680 Acked Read 4 Sent Done Done false
12941 4681 Acked Write 8 Sent Done Done false
12942 4682 Acked Read 3 Sent Done Done false
12943 4683 Acked Read 4 Sent Done Done false
12944 4684 Acked Write 9 Sent Done Done false
12945 4685 Acked Read 4 Sent Done Done false
12946 4686 Acked Read 5 Sent Done Done false
12947 4687 Acked Read 6 Sent Done Done false
12948 4688 Acked Write 10 Sent Done Done false
12949 4689 Acked Read 5 Sent Done Done false
12950 4690 Acked Read 5 Sent Done Done false
12951 4691 Acked Flush 0 Sent Done Done false
12952 4692 Acked Read 4 Sent Done Done false
12953 4693 Acked Write 7 Sent Done Done false
12954 4694 Acked Flush 0 Sent Done Done false
12955 4695 Acked Write 3 Sent Done Done false
12956 4696 Acked Read 8 Sent Done Done false
12957 4697 Acked Write 4 Sent Done Done false
12958 4698 Acked Read 9 Sent Done Done false
12959 4699 Acked Read 2 Sent Done Done false
12960 4700 Acked Read 2 Sent Done Done false
12961 4701 Acked Write 4 Sent Done Done false
12962 4702 Acked Flush 0 Sent Done Done false
12963 4703 Acked Read 7 Sent Done Done false
12964 4704 Acked Read 6 Sent Done Done false
12965 4705 Acked Read 9 Sent Done Done false
12966 4706 Acked Read 6 Sent Done Done false
12967 4707 Acked Read 2 Sent Done Done false
12968 4708 Acked Flush 0 Sent Done Done false
12969 4709 Acked Read 6 Sent Done Done false
12970 4710 Acked Read 1 Sent Done Done false
12971 4711 Acked Read 9 Sent Done Done false
12972 4712 Acked Write 10 Sent Done Done false
12973 4713 Acked Write 10 Sent Done Done false
12974 4714 Acked Read 4 Sent Done Done false
12975 4715 Acked Read 9 Sent Done Done false
12976 4716 Acked Read 5 Sent Done Done false
12977 4717 Acked Read 7 Sent Done Done false
12978 4718 Acked Read 7 Sent Done Done false
12979 4719 Acked Read 8 Sent Done Done false
12980 4720 Acked Read 2 Sent Done Done false
12981 4721 Acked Read 9 Sent Done Done false
12982 4722 Acked Write 7 Sent Done Done false
12983 4723 Acked Write 8 Sent Done Done false
12984 4724 Acked Write 5 Sent Done Done false
12985 4725 Acked Write 2 Sent Done Done false
12986 4726 Acked Read 9 Sent Done Done false
12987 4727 Acked Write 1 Sent Done Done false
12988 4728 Acked Write 6 Sent Done Done false
12989 4729 Acked Read 9 Sent Done Done false
12990 4730 Acked Read 10 Sent Done Done false
12991 4731 Acked Flush 0 Sent Done Done false
12992 4732 Acked Write 9 Sent Done Done false
12993 4733 Acked Write 10 Sent Done Done false
12994 4734 Acked Flush 0 Sent Done Done false
12995 4735 Acked Write 2 Sent Done Done false
12996 4736 Acked Write 4 Sent Done Done false
12997 4737 Acked Read 3 Sent Done Done false
12998 4738 Acked Read 7 Sent Done Done false
12999 4739 Acked Read 7 Sent Done Done false
13000 4740 Acked Flush 0 Sent Done Done false
13001 4741 Acked Read 8 Sent Done Done false
13002 4742 Acked Read 2 Sent Done Done false
13003 4743 Acked Write 4 Sent Done Done false
13004 4744 Acked Write 9 Sent Done Done false
13005 4745 Acked Write 6 Sent Done Done false
13006 4746 Acked Read 2 Sent Done Done false
13007 4747 Acked Write 1 Sent Done Done false
13008 4748 Acked Read 5 Sent Done Done false
13009 4749 Acked Read 9 Sent Done Done false
13010 4750 Acked Read 3 Sent Done Done false
13011 4751 Acked Read 2 Sent Done Done false
13012 4752 Acked Read 1 Sent Done Done false
13013 4753 Acked Write 8 Sent Done Done false
13014 4754 Acked Read 10 Sent Done Done false
13015 4755 Acked Write 3 Sent Done Done false
13016 4756 Acked Flush 0 Sent Done Done false
13017 4757 Acked Write 8 Sent Done Done false
13018 4758 Acked Read 1 Sent Done Done false
13019 4759 Acked Write 10 Sent Done Done false
13020 4760 Acked Write 2 Sent Done Done false
13021 4761 Acked Write 9 Sent Done Done false
13022 4762 Acked Read 10 Sent Done Done false
13023 4763 Acked Write 1 Sent Done Done false
13024 4764 Acked Read 1 Sent Done Done false
13025 4765 Acked Read 5 Sent Done Done false
13026 4766 Acked Read 1 Sent Done Done false
13027 4767 Acked Write 6 Sent Done Done false
13028 4768 Acked Read 6 Sent Done Done false
13029 4769 Acked Write 8 Sent Done Done false
13030 4770 Acked Read 3 Sent Done Done false
13031 4771 Acked Write 6 Sent Done Done false
13032 4772 Acked Write 10 Sent Done Done false
13033 4773 Acked Write 4 Sent Done Done false
13034 4774 Acked Read 9 Sent Done Done false
13035 4775 Acked Write 5 Sent Done Done false
13036 4776 Acked Read 8 Sent Done Done false
13037 4777 Acked Flush 0 Sent Done Done false
13038 4778 Acked Write 1 Sent Done Done false
13039 4779 Acked Write 1 Sent Done Done false
13040 4780 Acked Read 10 Sent Done Done false
13041 4781 Acked Write 6 Sent Done Done false
13042 4782 Acked Flush 0 Sent Done Done false
13043 4783 Acked Write 4 Sent Done Done false
13044 4784 Acked Read 6 Sent Done Done false
13045 4785 Acked Read 5 Sent Done Done false
13046 4786 Acked Flush 0 Sent Done Done false
13047 4787 Acked Read 3 Sent Done Done false
13048 4788 Acked Write 8 Sent Done Done false
13049 4789 Acked Read 3 Sent Done Done false
13050 4790 Acked Write 8 Sent Done Done false
13051 4791 Acked Write 2 Sent Done Done false
13052 4792 Acked Read 2 Sent Done Done false
13053 4793 Acked Read 2 Sent Done Done false
13054 4794 Acked Read 7 Sent Done Done false
13055 4795 Acked Read 4 Sent Done Done false
13056 4796 Acked Read 5 Sent Done Done false
13057 4797 Acked Write 4 Sent Done Done false
13058 4798 Acked Read 5 Sent Done Done false
13059 4799 Acked Read 9 Sent Done Done false
13060 4800 Acked Flush 0 Sent Done Done false
13061 4801 Acked Read 2 Sent Done Done false
13062 4802 Acked Write 8 Sent Done Done false
13063 4803 Acked Write 1 Sent Done Done false
13064 4804 Acked Write 7 Sent Done Done false
13065 4805 Acked Write 2 Sent Done Done false
13066 4806 Acked Write 8 Sent Done Done false
13067 4807 Acked Flush 0 Sent Done Done false
13068 4808 Acked Write 9 Sent Done Done false
13069 4809 Acked Read 10 Sent Done Done false
13070 4810 Acked Write 7 Sent Done Done false
13071 4811 Acked Read 2 Sent Done Done false
13072 4812 Acked Write 9 Sent Done Done false
13073 4813 Acked Write 6 Sent Done Done false
13074 4814 Acked Read 8 Sent Done Done false
13075 4815 Acked Flush 0 Sent Done Done false
13076 4816 Acked Read 4 Sent Done Done false
13077 4817 Acked Read 9 Sent Done Done false
13078 4818 Acked Flush 0 Sent Done Done false
13079 4819 Acked Flush 0 Sent Done Done false
13080 4820 Acked Write 1 Sent Done Done false
13081 4821 Acked Write 1 Sent Done Done false
13082 4822 Acked Write 8 Sent Done Done false
13083 4823 Acked Flush 0 Sent Done Done false
13084 4824 Acked Read 8 Sent Done Done false
13085 4825 Acked Write 4 Sent Done Done false
13086 4826 Acked Read 10 Sent Done Done false
13087 4827 Acked Read 3 Sent Done Done false
13088 4828 Acked Read 10 Sent Done Done false
13089 4829 Acked Write 7 Sent Done Done false
13090 4830 Acked Write 5 Sent Done Done false
13091 4831 Acked Write 1 Sent Done Done false
13092 4832 Acked Write 10 Sent Done Done false
13093 4833 Acked Write 4 Sent Done Done false
13094 4834 Acked Read 9 Sent Done Done false
13095 4835 Acked Read 4 Sent Done Done false
13096 4836 Acked Write 8 Sent Done Done false
13097 4837 Acked Write 3 Sent Done Done false
13098 4838 Acked Write 6 Sent Done Done false
13099 4839 Acked Write 7 Sent Done Done false
13100 4840 Acked Write 7 Sent Done Done false
13101 4841 Acked Read 9 Sent Done Done false
13102 4842 Acked Write 6 Sent Done Done false
13103 4843 Acked Read 5 Sent Done Done false
13104 4844 Acked Read 7 Sent Done Done false
13105 4845 Acked Write 10 Sent Done Done false
13106 4846 Acked Read 5 Sent Done Done false
13107 4847 Acked Write 1 Sent Done Done false
13108 4848 Acked Write 5 Sent Done Done false
13109 4849 Acked Read 4 Sent Done Done false
13110 4850 Acked Read 1 Sent Done Done false
13111 4851 Acked Write 10 Sent Done Done false
13112 4852 Acked Read 5 Sent Done Done false
13113 4853 Acked Read 6 Sent Done Done false
13114 4854 Acked Read 5 Sent Done Done false
13115 4855 Acked Read 7 Sent Done Done false
13116 4856 Acked Write 9 Sent Done Done false
13117 4857 Acked Write 6 Sent Done Done false
13118 4858 Acked Flush 0 Sent Done Done false
13119 4859 Acked Read 1 Sent Done Done false
13120 4860 Acked Write 9 Sent Done Done false
13121 4861 Acked Write 5 Sent Done Done false
13122 4862 Acked Write 4 Sent Done Done false
13123 4863 Acked Read 7 Sent Done Done false
13124 4864 Acked Write 2 Sent Done Done false
13125 4865 Acked Read 9 Sent Done Done false
13126 4866 Acked Write 1 Sent Done Done false
13127 4867 Acked Read 7 Sent Done Done false
13128 4868 Acked Write 5 Sent Done Done false
13129 4869 Acked Write 5 Sent Done Done false
13130 4870 Acked Write 7 Sent Done Done false
13131 4871 Acked Write 9 Sent Done Done false
13132 4872 Acked Read 6 Sent Done Done false
13133 4873 Acked Write 8 Sent Done Done false
13134 4874 Acked Read 7 Sent Done Done false
13135 4875 Acked Write 4 Sent Done Done false
13136 4876 Acked Write 5 Sent Done Done false
13137 4877 Acked Flush 0 Sent Done Done false
13138 4878 Acked Read 9 Sent Done Done false
13139 4879 Acked Read 9 Sent Done Done false
13140 4880 Acked Write 2 Sent Done Done false
13141 4881 Acked Write 6 Sent Done Done false
13142 4882 Acked Read 1 Sent Done Done false
13143 4883 Acked Write 6 Sent Done Done false
13144 4884 Acked Read 3 Sent Done Done false
13145 4885 Acked Read 9 Sent Done Done false
13146 4886 Acked Read 7 Sent Done Done false
13147 4887 Acked Write 4 Sent Done Done false
13148 4888 Acked Read 1 Sent Done Done false
13149 4889 Acked Read 6 Sent Done Done false
13150 4890 Acked Write 1 Sent Done Done false
13151 4891 Acked Write 5 Sent Done Done false
13152 4892 Acked Write 7 Sent Done Done false
13153 4893 Acked Write 4 Sent Done Done false
13154 4894 Acked Write 9 Sent Done Done false
13155 4895 Acked Read 10 Sent Done Done false
13156 4896 Acked Write 4 Sent Done Done false
13157 4897 Acked Read 7 Sent Done Done false
13158 4898 Acked Read 4 Sent Done Done false
13159 4899 Acked Write 4 Sent Done Done false
13160 4900 Acked Read 8 Sent Done Done false
13161 4901 Acked Write 9 Sent Done Done false
13162 4902 Acked Read 5 Sent Done Done false
13163 4903 Acked Write 1 Sent Done Done false
13164 4904 Acked Write 6 Sent Done Done false
13165 4905 Acked Read 9 Sent Done Done false
13166 4906 Acked Read 5 Sent Done Done false
13167 4907 Acked Read 4 Sent Done Done false
13168 4908 Acked Read 4 Sent Done Done false
13169 4909 Acked Read 2 Sent Done Done false
13170 4910 Acked Write 10 Sent Done Done false
13171 4911 Acked Write 8 Sent Done Done false
13172 4912 Acked Read 4 Sent Done Done false
13173 4913 Acked Write 1 Sent Done Done false
13174 4914 Acked Read 10 Sent Done Done false
13175 4915 Acked Read 3 Sent Done Done false
13176 4916 Acked Write 8 Sent Done Done false
13177 4917 Acked Read 6 Sent Done Done false
13178 4918 Acked Read 9 Sent Done Done false
13179 4919 Acked Write 9 Sent Done Done false
13180 4920 Acked Write 2 Sent Done Done false
13181 4921 Acked Read 10 Sent Done Done false
13182 4922 Acked Write 8 Sent Done Done false
13183 4923 Acked Read 9 Sent Done Done false
13184 4924 Acked Write 5 Sent Done Done false
13185 STATES DS:0 DS:1 DS:2 TOTAL
13186 Sent 301 0 0 301
13187 Done 0 301 301 602
13188 Skipped 0 0 0 0
13189 Error 0 0 0 0
13190 Last Flush: 4623 4877 4877
13191 Downstairs last five completed: 4623 4622 4621 4620 4619
13192 Upstairs last five completed: 4924 4923 4922 4921 4920
13193 CLIENT: Up:0 ds:301 act:2
13194 [8810] got start action
13195 [8810] I am going to restart
13196 Starting downstairs at port 8810
13197 Make output file at "/tmp/dsc/downstairs-8810.txt"
13198 [8810][0] reports Starting
13199 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1214)
13200 [8810][0] reports Running
13201 ----------------------------------------------------------------
13202 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
13203 States: Offline Active Active
13204 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13205 4624 Acked Flush 0 Sent Done Done false
13206 4625 Acked Read 5 Sent Done Done false
13207 4626 Acked Read 3 Sent Done Done false
13208 4627 Acked Write 9 Sent Done Done false
13209 4628 Acked Read 10 Sent Done Done false
13210 4629 Acked Write 8 Sent Done Done false
13211 4630 Acked Write 8 Sent Done Done false
13212 4631 Acked Write 3 Sent Done Done false
13213 4632 Acked Write 1 Sent Done Done false
13214 4633 Acked Write 6 Sent Done Done false
13215 4634 Acked Read 1 Sent Done Done false
13216 4635 Acked Flush 0 Sent Done Done false
13217 4636 Acked Read 5 Sent Done Done false
13218 4637 Acked Write 8 Sent Done Done false
13219 4638 Acked Read 3 Sent Done Done false
13220 4639 Acked Read 4 Sent Done Done false
13221 4640 Acked Read 8 Sent Done Done false
13222 4641 Acked Write 5 Sent Done Done false
13223 4642 Acked Read 9 Sent Done Done false
13224 4643 Acked Read 3 Sent Done Done false
13225 4644 Acked Flush 0 Sent Done Done false
13226 4645 Acked Write 1 Sent Done Done false
13227 4646 Acked Read 1 Sent Done Done false
13228 4647 Acked Read 7 Sent Done Done false
13229 4648 Acked Read 8 Sent Done Done false
13230 4649 Acked Write 5 Sent Done Done false
13231 4650 Acked Write 6 Sent Done Done false
13232 4651 Acked Write 2 Sent Done Done false
13233 4652 Acked Write 9 Sent Done Done false
13234 4653 Acked Read 9 Sent Done Done false
13235 4654 Acked Write 4 Sent Done Done false
13236 4655 Acked Read 1 Sent Done Done false
13237 4656 Acked Read 9 Sent Done Done false
13238 4657 Acked Read 4 Sent Done Done false
13239 4658 Acked Read 2 Sent Done Done false
13240 4659 Acked Write 6 Sent Done Done false
13241 4660 Acked Write 1 Sent Done Done false
13242 4661 Acked Read 1 Sent Done Done false
13243 4662 Acked Write 10 Sent Done Done false
13244 4663 Acked Read 5 Sent Done Done false
13245 4664 Acked Read 2 Sent Done Done false
13246 4665 Acked Write 2 Sent Done Done false
13247 4666 Acked Read 6 Sent Done Done false
13248 4667 Acked Write 8 Sent Done Done false
13249 4668 Acked Flush 0 Sent Done Done false
13250 4669 Acked Read 7 Sent Done Done false
13251 4670 Acked Read 7 Sent Done Done false
13252 4671 Acked Read 2 Sent Done Done false
13253 4672 Acked Read 4 Sent Done Done false
13254 4673 Acked Read 5 Sent Done Done false
13255 4674 Acked Read 2 Sent Done Done false
13256 4675 Acked Flush 0 Sent Done Done false
13257 4676 Acked Write 8 Sent Done Done false
13258 4677 Acked Write 2 Sent Done Done false
13259 4678 Acked Read 9 Sent Done Done false
13260 4679 Acked Write 9 Sent Done Done false
13261 4680 Acked Read 4 Sent Done Done false
13262 4681 Acked Write 8 Sent Done Done false
13263 4682 Acked Read 3 Sent Done Done false
13264 4683 Acked Read 4 Sent Done Done false
13265 4684 Acked Write 9 Sent Done Done false
13266 4685 Acked Read 4 Sent Done Done false
13267 4686 Acked Read 5 Sent Done Done false
13268 4687 Acked Read 6 Sent Done Done false
13269 4688 Acked Write 10 Sent Done Done false
13270 4689 Acked Read 5 Sent Done Done false
13271 4690 Acked Read 5 Sent Done Done false
13272 4691 Acked Flush 0 Sent Done Done false
13273 4692 Acked Read 4 Sent Done Done false
13274 4693 Acked Write 7 Sent Done Done false
13275 4694 Acked Flush 0 Sent Done Done false
13276 4695 Acked Write 3 Sent Done Done false
13277 4696 Acked Read 8 Sent Done Done false
13278 4697 Acked Write 4 Sent Done Done false
13279 4698 Acked Read 9 Sent Done Done false
13280 4699 Acked Read 2 Sent Done Done false
13281 4700 Acked Read 2 Sent Done Done false
13282 4701 Acked Write 4 Sent Done Done false
13283 4702 Acked Flush 0 Sent Done Done false
13284 4703 Acked Read 7 Sent Done Done false
13285 4704 Acked Read 6 Sent Done Done false
13286 4705 Acked Read 9 Sent Done Done false
13287 4706 Acked Read 6 Sent Done Done false
13288 4707 Acked Read 2 Sent Done Done false
13289 4708 Acked Flush 0 Sent Done Done false
13290 4709 Acked Read 6 Sent Done Done false
13291 4710 Acked Read 1 Sent Done Done false
13292 4711 Acked Read 9 Sent Done Done false
13293 4712 Acked Write 10 Sent Done Done false
13294 4713 Acked Write 10 Sent Done Done false
13295 4714 Acked Read 4 Sent Done Done false
13296 4715 Acked Read 9 Sent Done Done false
13297 4716 Acked Read 5 Sent Done Done false
13298 4717 Acked Read 7 Sent Done Done false
13299 4718 Acked Read 7 Sent Done Done false
13300 4719 Acked Read 8 Sent Done Done false
13301 4720 Acked Read 2 Sent Done Done false
13302 4721 Acked Read 9 Sent Done Done false
13303 4722 Acked Write 7 Sent Done Done false
13304 4723 Acked Write 8 Sent Done Done false
13305 4724 Acked Write 5 Sent Done Done false
13306 4725 Acked Write 2 Sent Done Done false
13307 4726 Acked Read 9 Sent Done Done false
13308 4727 Acked Write 1 Sent Done Done false
13309 4728 Acked Write 6 Sent Done Done false
13310 4729 Acked Read 9 Sent Done Done false
13311 4730 Acked Read 10 Sent Done Done false
13312 4731 Acked Flush 0 Sent Done Done false
13313 4732 Acked Write 9 Sent Done Done false
13314 4733 Acked Write 10 Sent Done Done false
13315 4734 Acked Flush 0 Sent Done Done false
13316 4735 Acked Write 2 Sent Done Done false
13317 4736 Acked Write 4 Sent Done Done false
13318 4737 Acked Read 3 Sent Done Done false
13319 4738 Acked Read 7 Sent Done Done false
13320 4739 Acked Read 7 Sent Done Done false
13321 4740 Acked Flush 0 Sent Done Done false
13322 4741 Acked Read 8 Sent Done Done false
13323 4742 Acked Read 2 Sent Done Done false
13324 4743 Acked Write 4 Sent Done Done false
13325 4744 Acked Write 9 Sent Done Done false
13326 4745 Acked Write 6 Sent Done Done false
13327 4746 Acked Read 2 Sent Done Done false
13328 4747 Acked Write 1 Sent Done Done false
13329 4748 Acked Read 5 Sent Done Done false
13330 4749 Acked Read 9 Sent Done Done false
13331 4750 Acked Read 3 Sent Done Done false
13332 4751 Acked Read 2 Sent Done Done false
13333 4752 Acked Read 1 Sent Done Done false
13334 4753 Acked Write 8 Sent Done Done false
13335 4754 Acked Read 10 Sent Done Done false
13336 4755 Acked Write 3 Sent Done Done false
13337 4756 Acked Flush 0 Sent Done Done false
13338 4757 Acked Write 8 Sent Done Done false
13339 4758 Acked Read 1 Sent Done Done false
13340 4759 Acked Write 10 Sent Done Done false
13341 4760 Acked Write 2 Sent Done Done false
13342 4761 Acked Write 9 Sent Done Done false
13343 4762 Acked Read 10 Sent Done Done false
13344 4763 Acked Write 1 Sent Done Done false
13345 4764 Acked Read 1 Sent Done Done false
13346 4765 Acked Read 5 Sent Done Done false
13347 4766 Acked Read 1 Sent Done Done false
13348 4767 Acked Write 6 Sent Done Done false
13349 4768 Acked Read 6 Sent Done Done false
13350 4769 Acked Write 8 Sent Done Done false
13351 4770 Acked Read 3 Sent Done Done false
13352 4771 Acked Write 6 Sent Done Done false
13353 4772 Acked Write 10 Sent Done Done false
13354 4773 Acked Write 4 Sent Done Done false
13355 4774 Acked Read 9 Sent Done Done false
13356 4775 Acked Write 5 Sent Done Done false
13357 4776 Acked Read 8 Sent Done Done false
13358 4777 Acked Flush 0 Sent Done Done false
13359 4778 Acked Write 1 Sent Done Done false
13360 4779 Acked Write 1 Sent Done Done false
13361 4780 Acked Read 10 Sent Done Done false
13362 4781 Acked Write 6 Sent Done Done false
13363 4782 Acked Flush 0 Sent Done Done false
13364 4783 Acked Write 4 Sent Done Done false
13365 4784 Acked Read 6 Sent Done Done false
13366 4785 Acked Read 5 Sent Done Done false
13367 4786 Acked Flush 0 Sent Done Done false
13368 4787 Acked Read 3 Sent Done Done false
13369 4788 Acked Write 8 Sent Done Done false
13370 4789 Acked Read 3 Sent Done Done false
13371 4790 Acked Write 8 Sent Done Done false
13372 4791 Acked Write 2 Sent Done Done false
13373 4792 Acked Read 2 Sent Done Done false
13374 4793 Acked Read 2 Sent Done Done false
13375 4794 Acked Read 7 Sent Done Done false
13376 4795 Acked Read 4 Sent Done Done false
13377 4796 Acked Read 5 Sent Done Done false
13378 4797 Acked Write 4 Sent Done Done false
13379 4798 Acked Read 5 Sent Done Done false
13380 4799 Acked Read 9 Sent Done Done false
13381 4800 Acked Flush 0 Sent Done Done false
13382 4801 Acked Read 2 Sent Done Done false
13383 4802 Acked Write 8 Sent Done Done false
13384 4803 Acked Write 1 Sent Done Done false
13385 4804 Acked Write 7 Sent Done Done false
13386 4805 Acked Write 2 Sent Done Done false
13387 4806 Acked Write 8 Sent Done Done false
13388 4807 Acked Flush 0 Sent Done Done false
13389 4808 Acked Write 9 Sent Done Done false
13390 4809 Acked Read 10 Sent Done Done false
13391 4810 Acked Write 7 Sent Done Done false
13392 4811 Acked Read 2 Sent Done Done false
13393 4812 Acked Write 9 Sent Done Done false
13394 4813 Acked Write 6 Sent Done Done false
13395 4814 Acked Read 8 Sent Done Done false
13396 4815 Acked Flush 0 Sent Done Done false
13397 4816 Acked Read 4 Sent Done Done false
13398 4817 Acked Read 9 Sent Done Done false
13399 4818 Acked Flush 0 Sent Done Done false
13400 4819 Acked Flush 0 Sent Done Done false
13401 4820 Acked Write 1 Sent Done Done false
13402 4821 Acked Write 1 Sent Done Done false
13403 4822 Acked Write 8 Sent Done Done false
13404 4823 Acked Flush 0 Sent Done Done false
13405 4824 Acked Read 8 Sent Done Done false
13406 4825 Acked Write 4 Sent Done Done false
13407 4826 Acked Read 10 Sent Done Done false
13408 4827 Acked Read 3 Sent Done Done false
13409 4828 Acked Read 10 Sent Done Done false
13410 4829 Acked Write 7 Sent Done Done false
13411 4830 Acked Write 5 Sent Done Done false
13412 4831 Acked Write 1 Sent Done Done false
13413 4832 Acked Write 10 Sent Done Done false
13414 4833 Acked Write 4 Sent Done Done false
13415 4834 Acked Read 9 Sent Done Done false
13416 4835 Acked Read 4 Sent Done Done false
13417 4836 Acked Write 8 Sent Done Done false
13418 4837 Acked Write 3 Sent Done Done false
13419 4838 Acked Write 6 Sent Done Done false
13420 4839 Acked Write 7 Sent Done Done false
13421 4840 Acked Write 7 Sent Done Done false
13422 4841 Acked Read 9 Sent Done Done false
13423 4842 Acked Write 6 Sent Done Done false
13424 4843 Acked Read 5 Sent Done Done false
13425 4844 Acked Read 7 Sent Done Done false
13426 4845 Acked Write 10 Sent Done Done false
13427 4846 Acked Read 5 Sent Done Done false
13428 4847 Acked Write 1 Sent Done Done false
13429 4848 Acked Write 5 Sent Done Done false
13430 4849 Acked Read 4 Sent Done Done false
13431 4850 Acked Read 1 Sent Done Done false
13432 4851 Acked Write 10 Sent Done Done false
13433 4852 Acked Read 5 Sent Done Done false
13434 4853 Acked Read 6 Sent Done Done false
13435 4854 Acked Read 5 Sent Done Done false
13436 4855 Acked Read 7 Sent Done Done false
13437 4856 Acked Write 9 Sent Done Done false
13438 4857 Acked Write 6 Sent Done Done false
13439 4858 Acked Flush 0 Sent Done Done false
13440 4859 Acked Read 1 Sent Done Done false
13441 4860 Acked Write 9 Sent Done Done false
13442 4861 Acked Write 5 Sent Done Done false
13443 4862 Acked Write 4 Sent Done Done false
13444 4863 Acked Read 7 Sent Done Done false
13445 4864 Acked Write 2 Sent Done Done false
13446 4865 Acked Read 9 Sent Done Done false
13447 4866 Acked Write 1 Sent Done Done false
13448 4867 Acked Read 7 Sent Done Done false
13449 4868 Acked Write 5 Sent Done Done false
13450 4869 Acked Write 5 Sent Done Done false
13451 4870 Acked Write 7 Sent Done Done false
13452 4871 Acked Write 9 Sent Done Done false
13453 4872 Acked Read 6 Sent Done Done false
13454 4873 Acked Write 8 Sent Done Done false
13455 4874 Acked Read 7 Sent Done Done false
13456 4875 Acked Write 4 Sent Done Done false
13457 4876 Acked Write 5 Sent Done Done false
13458 4877 Acked Flush 0 Sent Done Done false
13459 4878 Acked Read 9 Sent Done Done false
13460 4879 Acked Read 9 Sent Done Done false
13461 4880 Acked Write 2 Sent Done Done false
13462 4881 Acked Write 6 Sent Done Done false
13463 4882 Acked Read 1 Sent Done Done false
13464 4883 Acked Write 6 Sent Done Done false
13465 4884 Acked Read 3 Sent Done Done false
13466 4885 Acked Read 9 Sent Done Done false
13467 4886 Acked Read 7 Sent Done Done false
13468 4887 Acked Write 4 Sent Done Done false
13469 4888 Acked Read 1 Sent Done Done false
13470 4889 Acked Read 6 Sent Done Done false
13471 4890 Acked Write 1 Sent Done Done false
13472 4891 Acked Write 5 Sent Done Done false
13473 4892 Acked Write 7 Sent Done Done false
13474 4893 Acked Write 4 Sent Done Done false
13475 4894 Acked Write 9 Sent Done Done false
13476 4895 Acked Read 10 Sent Done Done false
13477 4896 Acked Write 4 Sent Done Done false
13478 4897 Acked Read 7 Sent Done Done false
13479 4898 Acked Read 4 Sent Done Done false
13480 4899 Acked Write 4 Sent Done Done false
13481 4900 Acked Read 8 Sent Done Done false
13482 4901 Acked Write 9 Sent Done Done false
13483 4902 Acked Read 5 Sent Done Done false
13484 4903 Acked Write 1 Sent Done Done false
13485 4904 Acked Write 6 Sent Done Done false
13486 4905 Acked Read 9 Sent Done Done false
13487 4906 Acked Read 5 Sent Done Done false
13488 4907 Acked Read 4 Sent Done Done false
13489 4908 Acked Read 4 Sent Done Done false
13490 4909 Acked Read 2 Sent Done Done false
13491 4910 Acked Write 10 Sent Done Done false
13492 4911 Acked Write 8 Sent Done Done false
13493 4912 Acked Read 4 Sent Done Done false
13494 4913 Acked Write 1 Sent Done Done false
13495 4914 Acked Read 10 Sent Done Done false
13496 4915 Acked Read 3 Sent Done Done false
13497 4916 Acked Write 8 Sent Done Done false
13498 4917 Acked Read 6 Sent Done Done false
13499 4918 Acked Read 9 Sent Done Done false
13500 4919 Acked Write 9 Sent Done Done false
13501 4920 Acked Write 2 Sent Done Done false
13502 4921 Acked Read 10 Sent Done Done false
13503 4922 Acked Write 8 Sent Done Done false
13504 4923 Acked Read 9 Sent Done Done false
13505 4924 Acked Write 5 Sent Done Done false
13506 4925 Acked Flush 0 Sent Done Done false
13507 STATES DS:0 DS:1 DS:2 TOTAL
13508 Sent 302 0 0 302
13509 Done 0 302 302 604
13510 Skipped 0 0 0 0
13511 Error 0 0 0 0
13512 Last Flush: 4623 4925 4925
13513 Downstairs last five completed: 4623 4622 4621 4620 4619
13514 Upstairs last five completed: 4925 4924 4923 4922 4921
13515 CLIENT: Up:0 ds:302 act:2
135162024-12-09T15:10:40.067ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135172024-12-09T15:10:40.067ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34485) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135182024-12-09T15:10:40.068ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135192024-12-09T15:10:40.068ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135202024-12-09T15:10:40.068ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135212024-12-09T15:10:40.068ZINFOcrucible: send last flush ID to this DS: 4623 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135222024-12-09T15:10:40.068ZINFOcrucible: Replied this last flush ID: 4623 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135232024-12-09T15:10:40.069ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135242024-12-09T15:10:40.069ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135252024-12-09T15:10:40.069ZINFOcrucible: [0] client re-new 302 jobs since flush 4623 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135262024-12-09T15:10:40.069ZINFOcrucible: [0] Replayed 302 jobs since flush: 4623 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
13527 ----------------------------------------------------------------
13528 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13529 Downstairs last five completed: 4925 4924 4923 4922 4921
13530 Upstairs last five completed: 4925 4924 4923 4922 4921
13531 CLIENT: Up:0 ds:0 act:3
13532 Replay: All jobs finished
13533 Main task has work to do, go find it
13534 got dsc Stop(1)
13535 stop 1
13536 Dec 09 15:10:42.309 INFO request completed, latency_us: 325, response_code: 204, uri[8820] Got stop action so:false kr:false
13537 : /stop/cid/1, method: POST, req_id: 228baa8c-c868-4a4d-a583-13e163e1c358, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
135382024-12-09T15:10:42.310ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135392024-12-09T15:10:42.310ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135402024-12-09T15:10:42.310ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135412024-12-09T15:10:42.310ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
135422024-12-09T15:10:42.310ZINFOcrucible: [1] Transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
13543 Dec 09 15:10:42.310 INFO request completed, latency_us: 138, response_code: 200, uri: /state/cid/1, method: GET, req_id: 97a93a4b-f51f-4678-874b-787fc39b71b0, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
135442024-12-09T15:10:42.310ZINFOcrucible: client task is exiting = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
13545 [8820] Exited with: ExitStatus(unix_wait_status(9))
13546 [8820][1] reports Exit
13547 Dec 09 15:10:46.312 INFO request completed, latency_us: 223, response_code: 200, uri: /state/cid/1, method: GET, req_id: 22d1e3b2-cbff-4f59-ad17-a95c4ce904e6, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
13548 001/300 Read block 2801 len 16384
13549 002/300 Write block 3120 len 16384 data: 4 4 4 4
13550 003/300 Flush
13551 004/300 Read block 4973 len 8192
13552 005/300 Write block 2540 len 16384 data: 5 6 4 3
13553 006/300 Read block 140 len 20480
13554 007/300 Flush
13555 008/300 Read block 3441 len 12288
13556 009/300 Read block 1080 len 40960
13557 010/300 Write block 1 len 40960 data: 2 3 3 2 2 3 3 3 3 4
13558 011/300 Read block 4467 len 16384
13559 012/300 Read block 4470 len 8192
13560 013/300 Read block 4695 len 4096
13561 014/300 Write block 2922 len 24576 data: 4 3 4 3 3 5
13562 015/300 Read block 3665 len 40960
13563 016/300 Read block 757 len 12288
13564 017/300 Read block 511 len 28672
13565 018/300 Read block 1219 len 4096
13566 019/300 Read block 4218 len 8192
13567 020/300 Read block 2807 len 28672
13568 021/300 Write block 3017 len 12288 data: 3 3 3
13569 022/300 Read block 245 len 16384
13570 023/300 Read block 2759 len 4096
13571 024/300 Write block 1426 len 20480 data: 3 2 3 3 2
13572 025/300 Write block 2406 len 32768 data: 2 2 2 3 3 3 3 3
13573 026/300 Read block 2921 len 4096
13574 027/300 Read block 3594 len 12288
13575 028/300 Write block 1075 len 20480 data: 4 6 5 5 6
13576 029/300 Read block 1638 len 28672
13577 030/300 Write block 3394 len 36864 data: 6 5 5 5 6 6 4 4 4
13578 031/300 Write block 1266 len 36864 data: 3 3 4 4 5 5 5 5 5
13579 032/300 Read block 1623 len 4096
13580 033/300 Read block 1956 len 36864
13581 034/300 Write block 4657 len 24576 data: 5 5 4 4 4 6
13582 035/300 Read block 442 len 8192
13583 036/300 Read block 149 len 24576
13584 037/300 Read block 1470 len 28672
13585 038/300 Flush
13586 039/300 Read block 2268 len 16384
13587 040/300 Write block 2187 len 32768 data: 2 3 3 3 3 3 4 4
13588 041/300 Write block 2661 len 40960 data: 5 4 4 2 2 2 4 4 4 4
13589 042/300 Write block 1468 len 20480 data: 4 4 4 3 3
13590 043/300 Read block 1533 len 8192
13591 044/300 Read block 2386 len 28672
13592 045/300 Write block 109 len 32768 data: 2 2 2 3 4 4 3 3
13593 046/300 Read block 1859 len 16384
13594 047/300 Read block 2051 len 28672
13595 048/300 Read block 3870 len 16384
13596 049/300 Write block 688 len 32768 data: 4 3 2 2 2 2 2 2
13597 050/300 Write block 4635 len 40960 data: 3 3 2 2 3 4 5 5 5 5
13598 051/300 Write block 604 len 32768 data: 3 3 4 4 5 4 3 3
13599 052/300 Read block 1120 len 36864
13600 053/300 Write block 2598 len 40960 data: 2 3 2 4 4 4 4 4 4 2
13601 054/300 Flush
13602 055/300 Read block 474 len 20480
13603 056/300 Write block 227 len 36864 data: 6 4 3 3 4 2 4 4 4
13604 057/300 Flush
13605 058/300 Read block 4767 len 32768
13606 059/300 Read block 1810 len 12288
13607 060/300 Flush
13608 061/300 Write block 605 len 4096 data: 4
13609 062/300 Read block 1983 len 28672
13610 063/300 Write block 3842 len 32768 data: 2 4 6 6 6 5 5 5
13611 064/300 Write block 1369 len 16384 data: 5 7 7 7
13612 065/300 Flush
13613 066/300 Flush
13614 067/300 Write block 1760 len 40960 data: 5 4 6 6 5 5 4 3 3 4
13615 068/300 Read block 4851 len 20480
13616 069/300 Write block 4766 len 16384 data: 3 3 3 2
13617 070/300 Write block 4319 len 12288 data: 4 4 5
13618 071/300 Flush
13619 072/300 Flush
13620 073/300 Read block 695 len 16384
13621 074/300 Flush
13622 075/300 Read block 1917 len 8192
13623 076/300 Read block 3606 len 4096
13624 077/300 Write block 1664 len 8192 data: 5 5
13625 078/300 Read block 345 len 8192
13626 079/300 Read block 4470 len 12288
13627 080/300 Write block 4408 len 16384 data: 4 4 4 5
13628 081/300 Write block 541 len 8192 data: 4 4
13629 082/300 Write block 1492 len 12288 data: 2 2 2
13630 083/300 Flush
13631 084/300 Read block 1888 len 4096
13632 085/300 Read block 3862 len 8192
13633 086/300 Read block 1651 len 20480
13634 087/300 Read block 1561 len 36864
13635 088/300 Read block 3468 len 32768
13636 089/300 Write block 4419 len 32768 data: 3 3 4 4 3 4 5 6
13637 090/300 Write block 3852 len 24576 data: 3 3 3 4 4 4
13638 091/300 Read block 383 len 16384
13639 092/300 Flush
13640 093/300 Write block 4565 len 36864 data: 2 2 2 2 2 3 3 4 4
13641 094/300 Read block 2625 len 24576
13642 095/300 Read block 2518 len 8192
13643 096/300 Read block 4646 len 40960
13644 097/300 Read block 4563 len 12288
13645 098/300 Read block 2638 len 20480
13646 099/300 Read block 3771 len 12288
13647 100/300 Write block 1699 len 28672 data: 2 2 2 4 3 3 3
13648 101/300 Read block 766 len 16384
13649 102/300 Read block 3931 len 16384
13650 103/300 Read block 551 len 20480
13651 104/300 Write block 1090 len 40960 data: 5 4 4 4 6 5 4 3 2 2
13652 105/300 Read block 132 len 4096
13653 106/300 Write block 1915 len 20480 data: 2 2 3 3 3
13654 107/300 Read block 4273 len 12288
13655 108/300 Write block 3101 len 24576 data: 4 3 3 4 4 3
13656 109/300 Write block 4324 len 8192 data: 5 5
13657 110/300 Write block 3218 len 40960 data: 2 2 2 3 4 4 4 3 4 4
13658 111/300 Read block 663 len 28672
13659 112/300 Read block 4767 len 40960
13660 113/300 Read block 3276 len 24576
13661 114/300 Read block 3483 len 24576
13662 115/300 Write block 607 len 28672 data: 5 6 5 4 4 3 2
13663 116/300 Read block 69 len 40960
13664 117/300 Read block 4327 len 32768
13665 118/300 Read block 3162 len 4096
13666 119/300 Read block 3200 len 24576
13667 120/300 Write block 647 len 36864 data: 2 3 3 3 3 3 3 3 4
13668 121/300 Read block 1320 len 8192
13669 122/300 Write block 3936 len 16384 data: 4 4 5 3
13670 123/300 Flush
13671 124/300 Read block 2316 len 24576
13672 125/300 Read block 4535 len 36864
13673 126/300 Write block 3138 len 28672 data: 3 3 4 5 4 4 7
13674 127/300 Write block 4630 len 32768 data: 3 3 3 3 3 4 4 3
13675 128/300 Read block 1887 len 36864
13676 129/300 Flush
13677 130/300 Write block 2017 len 8192 data: 5 5
13678 131/300 Read block 158 len 4096
13679 132/300 Read block 1192 len 40960
13680 133/300 Write block 769 len 16384 data: 4 3 4 4
13681 134/300 Write block 3042 len 12288 data: 3 3 4
13682 135/300 Write block 1341 len 4096 data: 4
13683 136/300 Read block 1374 len 24576
13684 137/300 Read block 2250 len 4096
13685 138/300 Write block 1434 len 24576 data: 3 3 3 3 4 3
13686 139/300 Read block 903 len 4096
13687 140/300 Flush
13688 141/300 Flush
13689 142/300 Flush
13690 143/300 Write block 3531 len 20480 data: 7 5 5 5 6
13691 144/300 Read block 3353 len 24576
13692 145/300 Read block 229 len 8192
13693 146/300 Read block 563 len 28672
13694 147/300 Write block 500 len 40960 data: 2 3 3 3 4 4 4 5 5 5
13695 148/300 Read block 4442 len 4096
13696 149/300 Flush
13697 150/300 Flush
13698 151/300 Write block 2058 len 12288 data: 4 4 3
13699 152/300 Read block 4491 len 24576
13700 153/300 Write block 886 len 4096 data: 4
13701 154/300 Write block 935 len 12288 data: 2 2 2
13702 155/300 Flush
13703 156/300 Write block 4152 len 16384 data: 2 3 4 3
13704 157/300 Write block 1980 len 28672 data: 4 6 6 6 6 6 4
13705 158/300 Read block 1675 len 8192
13706 159/300 Read block 484 len 4096
13707 160/300 Read block 2930 len 40960
13708 161/300 Flush
13709 162/300 Write block 3290 len 20480 data: 6 5 5 4 4
13710 163/300 Write block 1307 len 28672 data: 2 2 2 2 5 6 6
13711 164/300 Write block 1901 len 8192 data: 2 2
13712 165/300 Write block 1246 len 28672 data: 5 6 5 4 4 2 3
13713 166/300 Read block 1634 len 40960
13714 167/300 Read block 1261 len 28672
13715 168/300 Read block 2887 len 32768
13716 169/300 Write block 1181 len 20480 data: 2 2 2 2 2
13717 170/300 Read block 3673 len 12288
13718 171/300 Read block 2496 len 24576
13719 172/300 Write block 3230 len 20480 data: 3 3 4 4 3
13720 173/300 Write block 2648 len 12288 data: 5 5 5
13721 174/300 Write block 3515 len 16384 data: 3 3 5 6
13722 175/300 Write block 1569 len 8192 data: 6 6
13723 176/300 Read block 4322 len 12288
13724 177/300 Read block 405 len 28672
13725 178/300 Write block 2228 len 32768 data: 3 4 4 3 4 4 5 5
13726 179/300 Read block 2637 len 20480
13727 180/300 Read block 4669 len 8192
13728 181/300 Read block 4753 len 32768
13729 182/300 Read block 2645 len 28672
13730 183/300 Write block 1432 len 24576 data: 3 3 4 4 4 4
13731 184/300 Read block 4094 len 32768
13732 185/300 Flush
13733 186/300 Read block 2632 len 32768
13734 187/300 Write block 1282 len 4096 data: 4
13735 188/300 Read block 4297 len 24576
13736 189/300 Read block 4993 len 4096
13737 190/300 Write block 2253 len 12288 data: 4 5 5
13738 191/300 Read block 290 len 32768
13739 192/300 Write block 741 len 32768 data: 5 5 4 4 4 4 4 3
13740 193/300 Read block 4649 len 20480
13741 194/300 Write block 4614 len 32768 data: 3 3 4 3 3 4 4 4
13742 195/300 Read block 4969 len 32768
13743 196/300 Write block 2970 len 16384 data: 4 5 5 5
13744 197/300 Write block 2167 len 12288 data: 4 3 3
13745 198/300 Read block 294 len 40960
13746 199/300 Read block 4425 len 8192
13747 200/300 Write block 3755 len 24576 data: 6 6 6 7 6 5
13748 201/300 Read block 3539 len 40960
13749 202/300 Read block 4962 len 4096
13750 203/300 Read block 3687 len 40960
13751 204/300 Write block 1994 len 16384 data: 3 3 4 5
13752 205/300 Write block 4422 len 40960 data: 5 4 5 6 7 6 6 6 5 4
13753 206/300 Write block 1171 len 16384 data: 2 2 2 3
13754 207/300 Read block 3311 len 24576
13755 208/300 Write block 1528 len 16384 data: 4 4 4 4
13756 209/300 Write block 3261 len 12288 data: 3 2 3
13757 210/300 Write block 1037 len 36864 data: 6 4 6 4 5 5 4 3 4
13758 211/300 Read block 3458 len 28672
13759 212/300 Read block 4692 len 28672
13760 213/300 Write block 4646 len 12288 data: 6 7 7
13761 214/300 Read block 2578 len 32768
13762 215/300 Read block 588 len 36864
13763 216/300 Flush
13764 217/300 Read block 2520 len 28672
13765 218/300 Read block 1574 len 28672
13766 219/300 Write block 4692 len 8192 data: 2 2
13767 220/300 Flush
13768 221/300 Read block 4240 len 20480
13769 222/300 Write block 3697 len 36864 data: 2 2 2 2 2 2 2 2 2
13770 223/300 Read block 3331 len 4096
13771 224/300 Write block 4239 len 20480 data: 3 3 3 3 5
13772 225/300 Write block 4406 len 16384 data: 4 4 5 5
13773 226/300 Read block 1451 len 4096
13774 227/300 Read block 1720 len 40960
13775 228/300 Write block 3792 len 16384 data: 4 3 2 2
13776 229/300 Write block 4792 len 12288 data: 3 3 3
13777 230/300 Read block 2769 len 8192
13778 231/300 Read block 2845 len 8192
13779 232/300 Write block 1978 len 16384 data: 4 4 5 7
13780 233/300 Write block 3985 len 8192 data: 5 5
13781 234/300 Read block 2937 len 24576
13782 235/300 Read block 2416 len 28672
13783 236/300 Read block 3561 len 16384
13784 237/300 Write block 505 len 24576 data: 5 5 6 6 6 5
13785 238/300 Read block 1846 len 12288
13786 239/300 Write block 3211 len 40960 data: 3 3 3 3 4 4 4 3 3 3
13787 240/300 Read block 2828 len 24576
13788 241/300 Write block 1394 len 12288 data: 4 4 4
13789 242/300 Read block 2091 len 4096
13790 243/300 Write block 4766 len 32768 data: 4 4 4 3 2 3 2 2
13791 244/300 Read block 2685 len 16384
13792 245/300 Read block 3954 len 24576
13793 246/300 Read block 4599 len 16384
13794 247/300 Write block 858 len 20480 data: 4 4 6 5 5
13795 248/300 Flush
13796 249/300 Read block 687 len 12288
13797 250/300 Read block 1141 len 4096
13798 251/300 Read block 4450 len 20480
13799 252/300 Write block 1961 len 40960 data: 4 4 5 4 4 4 5 4 5 5
13800 253/300 Write block 3798 len 20480 data: 4 5 6 6 6
13801 254/300 Read block 4967 len 16384
13802 255/300 Write block 357 len 12288 data: 6 6 6
13803 256/300 Read block 413 len 16384
13804 257/300 Read block 3267 len 28672
13805 258/300 Read block 1056 len 28672
13806 259/300 Read block 2635 len 36864
13807 260/300 Read block 1971 len 36864
13808 261/300 Read block 4315 len 12288
13809 262/300 Flush
13810 263/300 Flush
13811 264/300 Write block 4436 len 12288 data: 5 4 4
13812 265/300 Write block 3835 len 36864 data: 4 4 3 2 2 2 2 3 5
13813 266/300 Write block 3698 len 20480 data: 3 3 3 3 3
13814 267/300 Write block 2798 len 4096 data: 4
13815 268/300 Read block 4007 len 12288
13816 269/300 Write block 2785 len 36864 data: 3 2 2 3 3 3 4 4 5
13817 270/300 Write block 2336 len 24576 data: 8 8 7 7 7 8
13818 271/300 Read block 4157 len 16384
13819 272/300 Read block 4899 len 28672
13820 273/300 Write block 403 len 12288 data: 4 5 5
13821 274/300 Read block 3825 len 36864
13822 275/300 Write block 2075 len 12288 data: 4 4 4
13823 276/300 Write block 2423 len 8192 data: 4 4
13824 277/300 Write block 1795 len 4096 data: 7
13825 278/300 Write block 4333 len 36864 data: 2 2 2 2 3 3 3 3 3
13826 279/300 Read block 3632 len 24576
13827 280/300 Write block 4501 len 36864 data: 5 6 6 6 6 8 8 8 7
13828 281/300 Read block 1506 len 36864
13829 282/300 Write block 867 len 28672 data: 5 5 5 5 5 4 4
13830 283/300 Read block 3136 len 32768
13831 284/300 Read block 3305 len 12288
13832 285/300 Flush
13833 286/300 Read block 3566 len 36864
13834 287/300 Read block 3782 len 32768
13835 288/300 Write block 900 len 40960 data: 5 6 6 6 6 4 4 3 3 5
13836 289/300 Read block 316 len 4096
13837 290/300 Read block 3437 len 32768
13838 291/300 Read block 1988 len 36864
13839 292/300 Flush
13840 293/300 Read block 1403 len 28672
13841 294/300 Write block 164 len 8192 data: 3 3
13842 295/300 Write block 2733 len 24576 data: 4 4 5 4 4 4
13843 296/300 Read block 4657 len 28672
13844 297/300 Write block 3066 len 40960 data: 4 3 2 2 2 2 3 3 4 5
13845 298/300 Write block 2195 len 32768 data: 4 5 4 3 4 4 5 6
13846 299/300 Read block 165 len 32768
13847 300/300 Read block 3230 len 12288
13848 301/300 Read block 567 len 24576
13849 Main task has work to do, go find it
13850 got dsc Start(1)
13851 start 1
13852 Dec 09 15:10:46.552 INFO request completed, latency_us: 253, response_code: 204, uri: /start/cid/1, method: POST, req_id: f7321658-adfd-4fda-830f-cf6ebc617124, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
13853 Replay: started 1, returned:Ok(())
13854 ----------------------------------------------------------------
13855 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13856 States: Active Offline Active
13857 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13858 4926 Acked Read 4 Done Sent Done false
13859 4927 Acked Write 4 Done Sent Done false
13860 4928 Acked Flush 0 Done Sent Done false
13861 4929 Acked Read 2 Done Sent Done false
13862 4930 Acked Write 4 Done Sent Done false
13863 4931 Acked Read 5 Done Sent Done false
13864 4932 Acked Flush 0 Done Sent Done false
13865 4933 Acked Read 3 Done Sent Done false
13866 4934 Acked Read 10 Done Sent Done false
13867 4935 Acked Write 10 Done Sent Done false
13868 4936 Acked Read 4 Done Sent Done false
13869 4937 Acked Read 2 Done Sent Done false
13870 4938 Acked Read 1 Done Sent Done false
13871 4939 Acked Write 6 Done Sent Done false
13872 4940 Acked Read 10 Done Sent Done false
13873 4941 Acked Read 3 Done Sent Done false
13874 4942 Acked Read 7 Done Sent Done false
13875 4943 Acked Read 1 Done Sent Done false
13876 4944 Acked Read 2 Done Sent Done false
13877 4945 Acked Read 7 Done Sent Done false
13878 4946 Acked Write 3 Done Sent Done false
13879 4947 Acked Read 4 Done Sent Done false
13880 4948 Acked Read 1 Done Sent Done false
13881 4949 Acked Write 5 Done Sent Done false
13882 4950 Acked Write 8 Done Sent Done false
13883 4951 Acked Read 1 Done Sent Done false
13884 4952 Acked Read 3 Done Sent Done false
13885 4953 Acked Write 5 Done Sent Done false
13886 4954 Acked Read 7 Done Sent Done false
13887 4955 Acked Write 9 Done Sent Done false
13888 4956 Acked Write 9 Done Sent Done false
13889 4957 Acked Read 1 Done Sent Done false
13890 4958 Acked Read 9 Done Sent Done false
13891 4959 Acked Write 6 Done Sent Done false
13892 4960 Acked Read 2 Done Sent Done false
13893 4961 Acked Read 6 Done Sent Done false
13894 4962 Acked Read 7 Done Sent Done false
13895 4963 Acked Flush 0 Done Sent Done false
13896 4964 Acked Read 4 Done Sent Done false
13897 4965 Acked Write 8 Done Sent Done false
13898 4966 Acked Write 10 Done Sent Done false
13899 4967 Acked Write 5 Done Sent Done false
13900 4968 Acked Read 2 Done Sent Done false
13901 4969 Acked Read 7 Done Sent Done false
13902 4970 Acked Write 8 Done Sent Done false
13903 4971 Acked Read 4 Done Sent Done false
13904 4972 Acked Read 7 Done Sent Done false
13905 4973 Acked Read 4 Done Sent Done false
13906 4974 Acked Write 8 Done Sent Done false
13907 4975 Acked Write 10 Done Sent Done false
13908 4976 Acked Write 8 Done Sent Done false
13909 4977 Acked Read 9 Done Sent Done false
13910 4978 Acked Write 10 Done Sent Done false
13911 4979 Acked Flush 0 Done Sent Done false
13912 4980 Acked Read 5 Done Sent Done false
13913 4981 Acked Write 9 Done Sent Done false
13914 4982 Acked Flush 0 Done Sent Done false
13915 4983 Acked Read 8 Done Sent Done false
13916 4984 Acked Read 3 Done Sent Done false
13917 4985 Acked Flush 0 Done Sent Done false
13918 4986 Acked Write 1 Done Sent Done false
13919 4987 Acked Read 7 Done Sent Done false
13920 4988 Acked Write 8 Done Sent Done false
13921 4989 Acked Write 4 Done Sent Done false
13922 4990 Acked Flush 0 Done Sent Done false
13923 4991 Acked Flush 0 Done Sent Done false
13924 4992 Acked Write 10 Done Sent Done false
13925 4993 Acked Read 5 Done Sent Done false
13926 4994 Acked Write 4 Done Sent Done false
13927 4995 Acked Write 3 Done Sent Done false
13928 4996 Acked Flush 0 Done Sent Done false
13929 4997 Acked Flush 0 Done Sent Done false
13930 4998 Acked Read 4 Done Sent Done false
13931 4999 Acked Flush 0 Done Sent Done false
13932 5000 Acked Read 2 Done Sent Done false
13933 5001 Acked Read 1 Done Sent Done false
13934 5002 Acked Write 2 Done Sent Done false
13935 5003 Acked Read 2 Done Sent Done false
13936 5004 Acked Read 3 Done Sent Done false
13937 5005 Acked Write 4 Done Sent Done false
13938 5006 Acked Write 2 Done Sent Done false
13939 5007 Acked Write 3 Done Sent Done false
13940 5008 Acked Flush 0 Done Sent Done false
13941 5009 Acked Read 1 Done Sent Done false
13942 5010 Acked Read 2 Done Sent Done false
13943 5011 Acked Read 5 Done Sent Done false
13944 5012 Acked Read 9 Done Sent Done false
13945 5013 Acked Read 8 Done Sent Done false
13946 5014 Acked Write 8 Done Sent Done false
13947 5015 Acked Write 6 Done Sent Done false
13948 5016 Acked Read 4 Done Sent Done false
13949 5017 Acked Flush 0 Done Sent Done false
13950 5018 Acked Write 9 Done Sent Done false
13951 5019 Acked Read 6 Done Sent Done false
13952 5020 Acked Read 2 Done Sent Done false
13953 5021 Acked Read 10 Done Sent Done false
13954 5022 Acked Read 3 Done Sent Done false
13955 5023 Acked Read 5 Done Sent Done false
13956 5024 Acked Read 3 Done Sent Done false
13957 5025 Acked Write 7 Done Sent Done false
13958 5026 Acked Read 4 Done Sent Done false
13959 5027 Acked Read 4 Done Sent Done false
13960 5028 Acked Read 5 Done Sent Done false
13961 5029 Acked Write 10 Done Sent Done false
13962 5030 Acked Read 1 Done Sent Done false
13963 5031 Acked Write 5 Done Sent Done false
13964 5032 Acked Read 3 Done Sent Done false
13965 5033 Acked Write 6 Done Sent Done false
13966 5034 Acked Write 2 Done Sent Done false
13967 5035 Acked Write 10 Done Sent Done false
13968 5036 Acked Read 7 Done Sent Done false
13969 5037 Acked Read 10 Done Sent Done false
13970 5038 Acked Read 6 Done Sent Done false
13971 5039 Acked Read 6 Done Sent Done false
13972 5040 Acked Write 7 Done Sent Done false
13973 5041 Acked Read 10 Done Sent Done false
13974 5042 Acked Read 8 Done Sent Done false
13975 5043 Acked Read 1 Done Sent Done false
13976 5044 Acked Read 6 Done Sent Done false
13977 5045 Acked Write 9 Done Sent Done false
13978 5046 Acked Read 2 Done Sent Done false
13979 5047 Acked Write 4 Done Sent Done false
13980 5048 Acked Flush 0 Done Sent Done false
13981 5049 Acked Read 6 Done Sent Done false
13982 5050 Acked Read 9 Done Sent Done false
13983 5051 Acked Write 7 Done Sent Done false
13984 5052 Acked Write 8 Done Sent Done false
13985 5053 Acked Read 9 Done Sent Done false
13986 5054 Acked Flush 0 Done Sent Done false
13987 5055 Acked Write 2 Done Sent Done false
13988 5056 Acked Read 1 Done Sent Done false
13989 5057 Acked Read 10 Done Sent Done false
13990 5058 Acked Write 4 Done Sent Done false
13991 5059 Acked Write 3 Done Sent Done false
13992 5060 Acked Write 1 Done Sent Done false
13993 5061 Acked Read 6 Done Sent Done false
13994 5062 Acked Read 1 Done Sent Done false
13995 5063 Acked Write 6 Done Sent Done false
13996 5064 Acked Read 1 Done Sent Done false
13997 5065 Acked Flush 0 Done Sent Done false
13998 5066 Acked Flush 0 Done Sent Done false
13999 5067 Acked Flush 0 Done Sent Done false
14000 5068 Acked Write 5 Done Sent Done false
14001 5069 Acked Read 6 Done Sent Done false
14002 5070 Acked Read 2 Done Sent Done false
14003 5071 Acked Read 7 Done Sent Done false
14004 5072 Acked Write 10 Done Sent Done false
14005 5073 Acked Read 1 Done Sent Done false
14006 5074 Acked Flush 0 Done Sent Done false
14007 5075 Acked Flush 0 Done Sent Done false
14008 5076 Acked Write 3 Done Sent Done false
14009 5077 Acked Read 6 Done Sent Done false
14010 5078 Acked Write 1 Done Sent Done false
14011 5079 Acked Write 3 Done Sent Done false
14012 5080 Acked Flush 0 Done Sent Done false
14013 5081 Acked Write 4 Done Sent Done false
14014 5082 Acked Write 7 Done Sent Done false
14015 5083 Acked Read 2 Done Sent Done false
14016 5084 Acked Read 1 Done Sent Done false
14017 5085 Acked Read 10 Done Sent Done false
14018 5086 Acked Flush 0 Done Sent Done false
14019 5087 Acked Write 5 Done Sent Done false
14020 5088 Acked Write 7 Done Sent Done false
14021 5089 Acked Write 2 Done Sent Done false
14022 5090 Acked Write 7 Done Sent Done false
14023 5091 Acked Read 10 Done Sent Done false
14024 5092 Acked Read 7 Done Sent Done false
14025 5093 Acked Read 8 Done Sent Done false
14026 5094 Acked Write 5 Done Sent Done false
14027 5095 Acked Read 3 Done Sent Done false
14028 5096 Acked Read 6 Done Sent Done false
14029 5097 Acked Write 5 Done Sent Done false
14030 5098 Acked Write 3 Done Sent Done false
14031 5099 Acked Write 4 Done Sent Done false
14032 5100 Acked Write 2 Done Sent Done false
14033 5101 Acked Read 3 Done Sent Done false
14034 5102 Acked Read 7 Done Sent Done false
14035 5103 Acked Write 8 Done Sent Done false
14036 5104 Acked Read 5 Done Sent Done false
14037 5105 Acked Read 2 Done Sent Done false
14038 5106 Acked Read 8 Done Sent Done false
14039 5107 Acked Read 7 Done Sent Done false
14040 5108 Acked Write 6 Done Sent Done false
14041 5109 Acked Read 8 Done Sent Done false
14042 5110 Acked Flush 0 Done Sent Done false
14043 5111 Acked Read 8 Done Sent Done false
14044 5112 Acked Write 1 Done Sent Done false
14045 5113 Acked Read 6 Done Sent Done false
14046 5114 Acked Read 1 Done Sent Done false
14047 5115 Acked Write 3 Done Sent Done false
14048 5116 Acked Read 8 Done Sent Done false
14049 5117 Acked Write 8 Done Sent Done false
14050 5118 Acked Read 5 Done Sent Done false
14051 5119 Acked Write 8 Done Sent Done false
14052 5120 Acked Read 8 Done Sent Done false
14053 5121 Acked Write 4 Done Sent Done false
14054 5122 Acked Write 3 Done Sent Done false
14055 5123 Acked Read 10 Done Sent Done false
14056 5124 Acked Read 2 Done Sent Done false
14057 5125 Acked Write 6 Done Sent Done false
14058 5126 Acked Read 10 Done Sent Done false
14059 5127 Acked Read 1 Done Sent Done false
14060 5128 Acked Read 10 Done Sent Done false
14061 5129 Acked Write 4 Done Sent Done false
14062 5130 Acked Write 10 Done Sent Done false
14063 5131 Acked Write 4 Done Sent Done false
14064 5132 Acked Read 6 Done Sent Done false
14065 5133 Acked Write 4 Done Sent Done false
14066 5134 Acked Write 3 Done Sent Done false
14067 5135 Acked Write 9 Done Sent Done false
14068 5136 Acked Read 7 Done Sent Done false
14069 5137 Acked Read 7 Done Sent Done false
14070 5138 Acked Write 3 Done Sent Done false
14071 5139 Acked Read 8 Done Sent Done false
14072 5140 Acked Read 9 Done Sent Done false
14073 5141 Acked Flush 0 Done Sent Done false
14074 5142 Acked Read 7 Done Sent Done false
14075 5143 Acked Read 7 Done Sent Done false
14076 5144 Acked Write 2 Done Sent Done false
14077 5145 Acked Flush 0 Done Sent Done false
14078 5146 Acked Read 5 Done Sent Done false
14079 5147 Acked Write 9 Done Sent Done false
14080 5148 Acked Read 1 Done Sent Done false
14081 5149 Acked Write 5 Done Sent Done false
14082 5150 Acked Write 4 Done Sent Done false
14083 5151 Acked Read 1 Done Sent Done false
14084 5152 Acked Read 10 Done Sent Done false
14085 5153 Acked Write 4 Done Sent Done false
14086 5154 Acked Write 3 Done Sent Done false
14087 5155 Acked Read 2 Done Sent Done false
14088 5156 Acked Read 2 Done Sent Done false
14089 5157 Acked Write 4 Done Sent Done false
14090 5158 Acked Write 2 Done Sent Done false
14091 5159 Acked Read 6 Done Sent Done false
14092 5160 Acked Read 7 Done Sent Done false
14093 5161 Acked Read 4 Done Sent Done false
14094 5162 Acked Write 6 Done Sent Done false
14095 5163 Acked Read 3 Done Sent Done false
14096 5164 Acked Write 10 Done Sent Done false
14097 5165 Acked Read 6 Done Sent Done false
14098 5166 Acked Write 3 Done Sent Done false
14099 5167 Acked Read 1 Done Sent Done false
14100 5168 Acked Write 8 Done Sent Done false
14101 5169 Acked Read 4 Done Sent Done false
14102 5170 Acked Read 6 Done Sent Done false
14103 5171 Acked Read 4 Done Sent Done false
14104 5172 Acked Write 5 Done Sent Done false
14105 5173 Acked Flush 0 Done Sent Done false
14106 5174 Acked Read 3 Done Sent Done false
14107 5175 Acked Read 1 Done Sent Done false
14108 5176 Acked Read 5 Done Sent Done false
14109 5177 Acked Write 10 Done Sent Done false
14110 5178 Acked Write 5 Done Sent Done false
14111 5179 Acked Read 4 Done Sent Done false
14112 5180 Acked Write 3 Done Sent Done false
14113 5181 Acked Read 4 Done Sent Done false
14114 5182 Acked Read 7 Done Sent Done false
14115 5183 Acked Read 7 Done Sent Done false
14116 5184 Acked Read 9 Done Sent Done false
14117 5185 Acked Read 9 Done Sent Done false
14118 5186 Acked Read 3 Done Sent Done false
14119 5187 Acked Flush 0 Done Sent Done false
14120 5188 Acked Flush 0 Done Sent Done false
14121 5189 Acked Write 3 Done Sent Done false
14122 5190 Acked Write 9 Done Sent Done false
14123 5191 Acked Write 5 Done Sent Done false
14124 5192 Acked Write 1 Done Sent Done false
14125 5193 Acked Read 3 Done Sent Done false
14126 5194 Acked Write 9 Done Sent Done false
14127 5195 Acked Write 6 Done Sent Done false
14128 5196 Acked Read 4 Done Sent Done false
14129 5197 Acked Read 7 Done Sent Done false
14130 5198 Acked Write 3 Done Sent Done false
14131 5199 Acked Read 9 Done Sent Done false
14132 5200 Acked Write 3 Done Sent Done false
14133 5201 Acked Write 2 Done Sent Done false
14134 5202 Acked Write 1 Done Sent Done false
14135 5203 Acked Write 9 Done Sent Done false
14136 5204 Acked Read 6 Done Sent Done false
14137 5205 Acked Write 9 Done Sent Done false
14138 5206 Acked Read 9 Done Sent Done false
14139 5207 Acked Write 7 Done Sent Done false
14140 5208 Acked Read 8 Done Sent Done false
14141 5209 Acked Read 3 Done Sent Done false
14142 5210 Acked Flush 0 Done Sent Done false
14143 5211 Acked Read 9 Done Sent Done false
14144 5212 Acked Read 8 Done Sent Done false
14145 5213 Acked Write 10 Done Sent Done false
14146 5214 Acked Read 1 Done Sent Done false
14147 5215 Acked Read 8 Done Sent Done false
14148 5216 Acked Read 9 Done Sent Done false
14149 5217 Acked Flush 0 Done Sent Done false
14150 5218 Acked Read 7 Done Sent Done false
14151 5219 Acked Write 2 Done Sent Done false
14152 5220 Acked Write 6 Done Sent Done false
14153 5221 Acked Read 7 Done Sent Done false
14154 5222 Acked Write 10 Done Sent Done false
14155 5223 Acked Write 8 Done Sent Done false
14156 5224 Acked Read 8 Done Sent Done false
14157 5225 Acked Read 3 Done Sent Done false
14158 5226 Acked Read 6 Done Sent Done false
14159 STATES DS:0 DS:1 DS:2 TOTAL
14160 Sent 0 301 0 301
14161 Done 301 0 301 602
14162 Skipped 0 0 0 0
14163 Error 0 0 0 0
14164 Last Flush: 5217 4925 5217
14165 Downstairs last five completed: 4925 4924 4923 4922 4921
14166 Upstairs last five completed: 5226 5225 5224 5223 5222
14167 CLIENT: Up:0 ds:301 act:2
14168 [8820] got start action
14169 [8820] I am going to restart
14170 Starting downstairs at port 8820
14171 Make output file at "/tmp/dsc/downstairs-8820.txt"
14172 [8820][1] reports Starting
14173 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1215)
14174 [8820][1] reports Running
14175 ----------------------------------------------------------------
14176 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
14177 States: Active Offline Active
14178 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14179 4926 Acked Read 4 Done Sent Done false
14180 4927 Acked Write 4 Done Sent Done false
14181 4928 Acked Flush 0 Done Sent Done false
14182 4929 Acked Read 2 Done Sent Done false
14183 4930 Acked Write 4 Done Sent Done false
14184 4931 Acked Read 5 Done Sent Done false
14185 4932 Acked Flush 0 Done Sent Done false
14186 4933 Acked Read 3 Done Sent Done false
14187 4934 Acked Read 10 Done Sent Done false
14188 4935 Acked Write 10 Done Sent Done false
14189 4936 Acked Read 4 Done Sent Done false
14190 4937 Acked Read 2 Done Sent Done false
14191 4938 Acked Read 1 Done Sent Done false
14192 4939 Acked Write 6 Done Sent Done false
14193 4940 Acked Read 10 Done Sent Done false
14194 4941 Acked Read 3 Done Sent Done false
14195 4942 Acked Read 7 Done Sent Done false
14196 4943 Acked Read 1 Done Sent Done false
14197 4944 Acked Read 2 Done Sent Done false
14198 4945 Acked Read 7 Done Sent Done false
14199 4946 Acked Write 3 Done Sent Done false
14200 4947 Acked Read 4 Done Sent Done false
14201 4948 Acked Read 1 Done Sent Done false
14202 4949 Acked Write 5 Done Sent Done false
14203 4950 Acked Write 8 Done Sent Done false
14204 4951 Acked Read 1 Done Sent Done false
14205 4952 Acked Read 3 Done Sent Done false
14206 4953 Acked Write 5 Done Sent Done false
14207 4954 Acked Read 7 Done Sent Done false
14208 4955 Acked Write 9 Done Sent Done false
14209 4956 Acked Write 9 Done Sent Done false
14210 4957 Acked Read 1 Done Sent Done false
14211 4958 Acked Read 9 Done Sent Done false
14212 4959 Acked Write 6 Done Sent Done false
14213 4960 Acked Read 2 Done Sent Done false
14214 4961 Acked Read 6 Done Sent Done false
14215 4962 Acked Read 7 Done Sent Done false
14216 4963 Acked Flush 0 Done Sent Done false
14217 4964 Acked Read 4 Done Sent Done false
14218 4965 Acked Write 8 Done Sent Done false
14219 4966 Acked Write 10 Done Sent Done false
14220 4967 Acked Write 5 Done Sent Done false
14221 4968 Acked Read 2 Done Sent Done false
14222 4969 Acked Read 7 Done Sent Done false
14223 4970 Acked Write 8 Done Sent Done false
14224 4971 Acked Read 4 Done Sent Done false
14225 4972 Acked Read 7 Done Sent Done false
14226 4973 Acked Read 4 Done Sent Done false
14227 4974 Acked Write 8 Done Sent Done false
14228 4975 Acked Write 10 Done Sent Done false
14229 4976 Acked Write 8 Done Sent Done false
14230 4977 Acked Read 9 Done Sent Done false
14231 4978 Acked Write 10 Done Sent Done false
14232 4979 Acked Flush 0 Done Sent Done false
14233 4980 Acked Read 5 Done Sent Done false
14234 4981 Acked Write 9 Done Sent Done false
14235 4982 Acked Flush 0 Done Sent Done false
14236 4983 Acked Read 8 Done Sent Done false
14237 4984 Acked Read 3 Done Sent Done false
14238 4985 Acked Flush 0 Done Sent Done false
14239 4986 Acked Write 1 Done Sent Done false
14240 4987 Acked Read 7 Done Sent Done false
14241 4988 Acked Write 8 Done Sent Done false
14242 4989 Acked Write 4 Done Sent Done false
14243 4990 Acked Flush 0 Done Sent Done false
14244 4991 Acked Flush 0 Done Sent Done false
14245 4992 Acked Write 10 Done Sent Done false
14246 4993 Acked Read 5 Done Sent Done false
14247 4994 Acked Write 4 Done Sent Done false
14248 4995 Acked Write 3 Done Sent Done false
14249 4996 Acked Flush 0 Done Sent Done false
14250 4997 Acked Flush 0 Done Sent Done false
14251 4998 Acked Read 4 Done Sent Done false
14252 4999 Acked Flush 0 Done Sent Done false
14253 5000 Acked Read 2 Done Sent Done false
14254 5001 Acked Read 1 Done Sent Done false
14255 5002 Acked Write 2 Done Sent Done false
14256 5003 Acked Read 2 Done Sent Done false
14257 5004 Acked Read 3 Done Sent Done false
14258 5005 Acked Write 4 Done Sent Done false
14259 5006 Acked Write 2 Done Sent Done false
14260 5007 Acked Write 3 Done Sent Done false
14261 5008 Acked Flush 0 Done Sent Done false
14262 5009 Acked Read 1 Done Sent Done false
14263 5010 Acked Read 2 Done Sent Done false
14264 5011 Acked Read 5 Done Sent Done false
14265 5012 Acked Read 9 Done Sent Done false
14266 5013 Acked Read 8 Done Sent Done false
14267 5014 Acked Write 8 Done Sent Done false
14268 5015 Acked Write 6 Done Sent Done false
14269 5016 Acked Read 4 Done Sent Done false
14270 5017 Acked Flush 0 Done Sent Done false
14271 5018 Acked Write 9 Done Sent Done false
14272 5019 Acked Read 6 Done Sent Done false
14273 5020 Acked Read 2 Done Sent Done false
14274 5021 Acked Read 10 Done Sent Done false
14275 5022 Acked Read 3 Done Sent Done false
14276 5023 Acked Read 5 Done Sent Done false
14277 5024 Acked Read 3 Done Sent Done false
14278 5025 Acked Write 7 Done Sent Done false
14279 5026 Acked Read 4 Done Sent Done false
14280 5027 Acked Read 4 Done Sent Done false
14281 5028 Acked Read 5 Done Sent Done false
14282 5029 Acked Write 10 Done Sent Done false
14283 5030 Acked Read 1 Done Sent Done false
14284 5031 Acked Write 5 Done Sent Done false
14285 5032 Acked Read 3 Done Sent Done false
14286 5033 Acked Write 6 Done Sent Done false
14287 5034 Acked Write 2 Done Sent Done false
14288 5035 Acked Write 10 Done Sent Done false
14289 5036 Acked Read 7 Done Sent Done false
14290 5037 Acked Read 10 Done Sent Done false
14291 5038 Acked Read 6 Done Sent Done false
14292 5039 Acked Read 6 Done Sent Done false
14293 5040 Acked Write 7 Done Sent Done false
14294 5041 Acked Read 10 Done Sent Done false
14295 5042 Acked Read 8 Done Sent Done false
14296 5043 Acked Read 1 Done Sent Done false
14297 5044 Acked Read 6 Done Sent Done false
14298 5045 Acked Write 9 Done Sent Done false
14299 5046 Acked Read 2 Done Sent Done false
14300 5047 Acked Write 4 Done Sent Done false
14301 5048 Acked Flush 0 Done Sent Done false
14302 5049 Acked Read 6 Done Sent Done false
14303 5050 Acked Read 9 Done Sent Done false
14304 5051 Acked Write 7 Done Sent Done false
14305 5052 Acked Write 8 Done Sent Done false
14306 5053 Acked Read 9 Done Sent Done false
14307 5054 Acked Flush 0 Done Sent Done false
14308 5055 Acked Write 2 Done Sent Done false
14309 5056 Acked Read 1 Done Sent Done false
14310 5057 Acked Read 10 Done Sent Done false
14311 5058 Acked Write 4 Done Sent Done false
14312 5059 Acked Write 3 Done Sent Done false
14313 5060 Acked Write 1 Done Sent Done false
14314 5061 Acked Read 6 Done Sent Done false
14315 5062 Acked Read 1 Done Sent Done false
14316 5063 Acked Write 6 Done Sent Done false
14317 5064 Acked Read 1 Done Sent Done false
14318 5065 Acked Flush 0 Done Sent Done false
14319 5066 Acked Flush 0 Done Sent Done false
14320 5067 Acked Flush 0 Done Sent Done false
14321 5068 Acked Write 5 Done Sent Done false
14322 5069 Acked Read 6 Done Sent Done false
14323 5070 Acked Read 2 Done Sent Done false
14324 5071 Acked Read 7 Done Sent Done false
14325 5072 Acked Write 10 Done Sent Done false
14326 5073 Acked Read 1 Done Sent Done false
14327 5074 Acked Flush 0 Done Sent Done false
14328 5075 Acked Flush 0 Done Sent Done false
14329 5076 Acked Write 3 Done Sent Done false
14330 5077 Acked Read 6 Done Sent Done false
14331 5078 Acked Write 1 Done Sent Done false
14332 5079 Acked Write 3 Done Sent Done false
14333 5080 Acked Flush 0 Done Sent Done false
14334 5081 Acked Write 4 Done Sent Done false
14335 5082 Acked Write 7 Done Sent Done false
14336 5083 Acked Read 2 Done Sent Done false
14337 5084 Acked Read 1 Done Sent Done false
14338 5085 Acked Read 10 Done Sent Done false
14339 5086 Acked Flush 0 Done Sent Done false
14340 5087 Acked Write 5 Done Sent Done false
14341 5088 Acked Write 7 Done Sent Done false
14342 5089 Acked Write 2 Done Sent Done false
14343 5090 Acked Write 7 Done Sent Done false
14344 5091 Acked Read 10 Done Sent Done false
14345 5092 Acked Read 7 Done Sent Done false
14346 5093 Acked Read 8 Done Sent Done false
14347 5094 Acked Write 5 Done Sent Done false
14348 5095 Acked Read 3 Done Sent Done false
14349 5096 Acked Read 6 Done Sent Done false
14350 5097 Acked Write 5 Done Sent Done false
14351 5098 Acked Write 3 Done Sent Done false
14352 5099 Acked Write 4 Done Sent Done false
14353 5100 Acked Write 2 Done Sent Done false
14354 5101 Acked Read 3 Done Sent Done false
14355 5102 Acked Read 7 Done Sent Done false
14356 5103 Acked Write 8 Done Sent Done false
14357 5104 Acked Read 5 Done Sent Done false
14358 5105 Acked Read 2 Done Sent Done false
14359 5106 Acked Read 8 Done Sent Done false
14360 5107 Acked Read 7 Done Sent Done false
14361 5108 Acked Write 6 Done Sent Done false
14362 5109 Acked Read 8 Done Sent Done false
14363 5110 Acked Flush 0 Done Sent Done false
14364 5111 Acked Read 8 Done Sent Done false
14365 5112 Acked Write 1 Done Sent Done false
14366 5113 Acked Read 6 Done Sent Done false
14367 5114 Acked Read 1 Done Sent Done false
14368 5115 Acked Write 3 Done Sent Done false
14369 5116 Acked Read 8 Done Sent Done false
14370 5117 Acked Write 8 Done Sent Done false
14371 5118 Acked Read 5 Done Sent Done false
14372 5119 Acked Write 8 Done Sent Done false
14373 5120 Acked Read 8 Done Sent Done false
14374 5121 Acked Write 4 Done Sent Done false
14375 5122 Acked Write 3 Done Sent Done false
14376 5123 Acked Read 10 Done Sent Done false
14377 5124 Acked Read 2 Done Sent Done false
14378 5125 Acked Write 6 Done Sent Done false
14379 5126 Acked Read 10 Done Sent Done false
14380 5127 Acked Read 1 Done Sent Done false
14381 5128 Acked Read 10 Done Sent Done false
14382 5129 Acked Write 4 Done Sent Done false
14383 5130 Acked Write 10 Done Sent Done false
14384 5131 Acked Write 4 Done Sent Done false
14385 5132 Acked Read 6 Done Sent Done false
14386 5133 Acked Write 4 Done Sent Done false
14387 5134 Acked Write 3 Done Sent Done false
14388 5135 Acked Write 9 Done Sent Done false
14389 5136 Acked Read 7 Done Sent Done false
14390 5137 Acked Read 7 Done Sent Done false
14391 5138 Acked Write 3 Done Sent Done false
14392 5139 Acked Read 8 Done Sent Done false
14393 5140 Acked Read 9 Done Sent Done false
14394 5141 Acked Flush 0 Done Sent Done false
14395 5142 Acked Read 7 Done Sent Done false
14396 5143 Acked Read 7 Done Sent Done false
14397 5144 Acked Write 2 Done Sent Done false
14398 5145 Acked Flush 0 Done Sent Done false
14399 5146 Acked Read 5 Done Sent Done false
14400 5147 Acked Write 9 Done Sent Done false
14401 5148 Acked Read 1 Done Sent Done false
14402 5149 Acked Write 5 Done Sent Done false
14403 5150 Acked Write 4 Done Sent Done false
14404 5151 Acked Read 1 Done Sent Done false
14405 5152 Acked Read 10 Done Sent Done false
14406 5153 Acked Write 4 Done Sent Done false
14407 5154 Acked Write 3 Done Sent Done false
14408 5155 Acked Read 2 Done Sent Done false
14409 5156 Acked Read 2 Done Sent Done false
14410 5157 Acked Write 4 Done Sent Done false
14411 5158 Acked Write 2 Done Sent Done false
14412 5159 Acked Read 6 Done Sent Done false
14413 5160 Acked Read 7 Done Sent Done false
14414 5161 Acked Read 4 Done Sent Done false
14415 5162 Acked Write 6 Done Sent Done false
14416 5163 Acked Read 3 Done Sent Done false
14417 5164 Acked Write 10 Done Sent Done false
14418 5165 Acked Read 6 Done Sent Done false
14419 5166 Acked Write 3 Done Sent Done false
14420 5167 Acked Read 1 Done Sent Done false
14421 5168 Acked Write 8 Done Sent Done false
14422 5169 Acked Read 4 Done Sent Done false
14423 5170 Acked Read 6 Done Sent Done false
14424 5171 Acked Read 4 Done Sent Done false
14425 5172 Acked Write 5 Done Sent Done false
14426 5173 Acked Flush 0 Done Sent Done false
14427 5174 Acked Read 3 Done Sent Done false
14428 5175 Acked Read 1 Done Sent Done false
14429 5176 Acked Read 5 Done Sent Done false
14430 5177 Acked Write 10 Done Sent Done false
14431 5178 Acked Write 5 Done Sent Done false
14432 5179 Acked Read 4 Done Sent Done false
14433 5180 Acked Write 3 Done Sent Done false
14434 5181 Acked Read 4 Done Sent Done false
14435 5182 Acked Read 7 Done Sent Done false
14436 5183 Acked Read 7 Done Sent Done false
14437 5184 Acked Read 9 Done Sent Done false
14438 5185 Acked Read 9 Done Sent Done false
14439 5186 Acked Read 3 Done Sent Done false
14440 5187 Acked Flush 0 Done Sent Done false
14441 5188 Acked Flush 0 Done Sent Done false
14442 5189 Acked Write 3 Done Sent Done false
14443 5190 Acked Write 9 Done Sent Done false
14444 5191 Acked Write 5 Done Sent Done false
14445 5192 Acked Write 1 Done Sent Done false
14446 5193 Acked Read 3 Done Sent Done false
14447 5194 Acked Write 9 Done Sent Done false
14448 5195 Acked Write 6 Done Sent Done false
14449 5196 Acked Read 4 Done Sent Done false
14450 5197 Acked Read 7 Done Sent Done false
14451 5198 Acked Write 3 Done Sent Done false
14452 5199 Acked Read 9 Done Sent Done false
14453 5200 Acked Write 3 Done Sent Done false
14454 5201 Acked Write 2 Done Sent Done false
14455 5202 Acked Write 1 Done Sent Done false
14456 5203 Acked Write 9 Done Sent Done false
14457 5204 Acked Read 6 Done Sent Done false
14458 5205 Acked Write 9 Done Sent Done false
14459 5206 Acked Read 9 Done Sent Done false
14460 5207 Acked Write 7 Done Sent Done false
14461 5208 Acked Read 8 Done Sent Done false
14462 5209 Acked Read 3 Done Sent Done false
14463 5210 Acked Flush 0 Done Sent Done false
14464 5211 Acked Read 9 Done Sent Done false
14465 5212 Acked Read 8 Done Sent Done false
14466 5213 Acked Write 10 Done Sent Done false
14467 5214 Acked Read 1 Done Sent Done false
14468 5215 Acked Read 8 Done Sent Done false
14469 5216 Acked Read 9 Done Sent Done false
14470 5217 Acked Flush 0 Done Sent Done false
14471 5218 Acked Read 7 Done Sent Done false
14472 5219 Acked Write 2 Done Sent Done false
14473 5220 Acked Write 6 Done Sent Done false
14474 5221 Acked Read 7 Done Sent Done false
14475 5222 Acked Write 10 Done Sent Done false
14476 5223 Acked Write 8 Done Sent Done false
14477 5224 Acked Read 8 Done Sent Done false
14478 5225 Acked Read 3 Done Sent Done false
14479 5226 Acked Read 6 Done Sent Done false
14480 5227 Acked Flush 0 Done Sent Done false
14481 STATES DS:0 DS:1 DS:2 TOTAL
14482 Sent 0 302 0 302
14483 Done 302 0 302 604
14484 Skipped 0 0 0 0
14485 Error 0 0 0 0
14486 Last Flush: 5227 4925 5227
14487 Downstairs last five completed: 4925 4924 4923 4922 4921
14488 Upstairs last five completed: 5227 5226 5225 5224 5223
14489 CLIENT: Up:0 ds:302 act:2
144902024-12-09T15:10:52.311ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
144912024-12-09T15:10:52.312ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55637) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
144922024-12-09T15:10:52.313ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
144932024-12-09T15:10:52.313ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
144942024-12-09T15:10:52.313ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
144952024-12-09T15:10:52.313ZINFOcrucible: send last flush ID to this DS: 4925 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
144962024-12-09T15:10:52.313ZINFOcrucible: Replied this last flush ID: 4925 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
144972024-12-09T15:10:52.314ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
144982024-12-09T15:10:52.314ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
144992024-12-09T15:10:52.314ZINFOcrucible: [1] client re-new 302 jobs since flush 4925 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
145002024-12-09T15:10:52.314ZINFOcrucible: [1] Replayed 302 jobs since flush: 4925 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
14501 ----------------------------------------------------------------
14502 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14503 Downstairs last five completed: 5227 5226 5225 5224 5223
14504 Upstairs last five completed: 5227 5226 5225 5224 5223
14505 CLIENT: Up:0 ds:0 act:3
14506 Replay: All jobs finished
14507 Main task has work to do, go find it
14508 got dsc Stop(1)
14509 stop 1
14510 Dec 09 15:10:54.557 INFO request completed[8820] Got stop action so:false kr:false
14511 , latency_us: 359, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 0b23e079-4cf7-4bb5-8a26-e061899cfab3, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
14512 Dec 09 15:10:54.558 INFO request completed, latency_us: 120, response_code: 200, uri: /state/cid/1, method: GET, req_id: f7358062-65fd-4aa8-86ed-b08d935152bb, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:10:54.558651714Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"1","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
14513 1:9998
145142024-12-09T15:10:54.558ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
145152024-12-09T15:10:54.558ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
145162024-12-09T15:10:54.558ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
145172024-12-09T15:10:54.558ZINFOcrucible: [1] Transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
145182024-12-09T15:10:54.558ZINFOcrucible: client task is exiting = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
14519 [8820] Exited with: ExitStatus(unix_wait_status(9))
14520 [8820][1] reports Exit
14521 Dec 09 15:10:58.560 INFO request completed, latency_us: 233, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2e66e6c4-4f51-4884-bb2d-fbda075f299d, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
14522 001/300 Write block 212 len 24576 data: 5 7 7 6 6 6
14523 002/300 Write block 1533 len 4096 data: 5
14524 003/300 Read block 3863 len 28672
14525 004/300 Write block 367 len 8192 data: 6 5
14526 005/300 Read block 4304 len 36864
14527 006/300 Read block 3564 len 8192
14528 007/300 Write block 3746 len 4096 data: 4
14529 008/300 Read block 25 len 32768
14530 009/300 Read block 3887 len 32768
14531 010/300 Read block 4657 len 16384
14532 011/300 Write block 165 len 12288 data: 4 3 3
14533 012/300 Flush
14534 013/300 Write block 1102 len 40960 data: 3 3 3 4 5 5 6 6 5 5
14535 014/300 Read block 2226 len 32768
14536 015/300 Read block 3446 len 40960
14537 016/300 Write block 2479 len 36864 data: 5 5 4 4 5 5 5 5 4
14538 017/300 Write block 3003 len 40960 data: 5 5 5 6 5 4 4 4 4 5
14539 018/300 Read block 457 len 8192
14540 019/300 Read block 2548 len 28672
14541 020/300 Flush
14542 021/300 Flush
14543 022/300 Read block 1398 len 40960
14544 023/300 Write block 2138 len 40960 data: 8 6 5 6 5 6 5 4 4 5
14545 024/300 Flush
14546 025/300 Write block 4149 len 32768 data: 3 2 2 3 4 5 4 4
14547 026/300 Read block 2864 len 16384
14548 027/300 Write block 3598 len 40960 data: 5 5 2 2 2 2 2 2 2 4
14549 028/300 Read block 1710 len 32768
14550 029/300 Flush
14551 030/300 Write block 1193 len 16384 data: 3 4 3 3
14552 031/300 Flush
14553 032/300 Read block 1653 len 40960
14554 033/300 Read block 714 len 40960
14555 034/300 Read block 3008 len 4096
14556 035/300 Write block 454 len 36864 data: 3 2 2 2 2 2 3 3 3
14557 036/300 Flush
14558 037/300 Flush
14559 038/300 Read block 4484 len 24576
14560 039/300 Flush
14561 040/300 Write block 833 len 20480 data: 3 3 3 3 3
14562 041/300 Read block 3170 len 4096
14563 042/300 Read block 2033 len 28672
14564 043/300 Read block 1846 len 28672
14565 044/300 Write block 2734 len 36864 data: 5 6 5 5 5 3 5 4 4
14566 045/300 Write block 2185 len 24576 data: 2 2 3 4 4 4
14567 046/300 Read block 4030 len 40960
14568 047/300 Read block 2091 len 20480
14569 048/300 Write block 4066 len 28672 data: 2 2 3 3 3 3 2
14570 049/300 Read block 3769 len 12288
14571 050/300 Write block 2366 len 20480 data: 2 2 2 3 2
14572 051/300 Write block 2581 len 32768 data: 4 3 4 4 4 3 2 2
14573 052/300 Write block 1727 len 40960 data: 3 3 3 3 3 3 4 4 3 3
14574 053/300 Write block 4259 len 12288 data: 5 4 4
14575 054/300 Write block 3442 len 24576 data: 4 3 3 4 3 4
14576 055/300 Write block 4423 len 12288 data: 5 6 7
14577 056/300 Write block 389 len 36864 data: 2 2 2 2 2 2 3 3 3
14578 057/300 Write block 61 len 16384 data: 4 2 2 2
14579 058/300 Write block 850 len 8192 data: 2 2
14580 059/300 Read block 1762 len 8192
14581 060/300 Write block 3061 len 20480 data: 3 3 3 2 2
14582 061/300 Write block 4540 len 16384 data: 3 2 2 2
14583 062/300 Read block 1931 len 16384
14584 063/300 Flush
14585 064/300 Write block 2296 len 4096 data: 4
14586 065/300 Read block 3932 len 20480
14587 066/300 Read block 1024 len 20480
14588 067/300 Write block 4707 len 32768 data: 5 6 6 4 4 4 5 5
14589 068/300 Write block 2645 len 12288 data: 7 6 4
14590 069/300 Write block 638 len 36864 data: 4 4 4 3 3 2 2 2 2
14591 070/300 Read block 4421 len 12288
14592 071/300 Read block 1027 len 28672
14593 072/300 Write block 670 len 12288 data: 4 5 5
14594 073/300 Read block 4145 len 12288
14595 074/300 Read block 4086 len 28672
14596 075/300 Read block 2745 len 40960
14597 076/300 Read block 2366 len 32768
14598 077/300 Read block 1610 len 36864
14599 078/300 Write block 643 len 24576 data: 3 3 3 3 3 4
14600 079/300 Write block 1335 len 40960 data: 6 7 8 7 6 4 5 3 3 3
14601 080/300 Flush
14602 081/300 Write block 4838 len 12288 data: 6 6 7
14603 082/300 Flush
14604 083/300 Read block 3751 len 24576
14605 084/300 Write block 1661 len 8192 data: 4 4
14606 085/300 Read block 2507 len 40960
14607 086/300 Write block 2022 len 20480 data: 5 4 2 2 3
14608 087/300 Write block 1052 len 40960 data: 2 2 2 2 2 3 3 3 4 4
14609 088/300 Read block 265 len 24576
14610 089/300 Read block 4686 len 4096
14611 090/300 Write block 686 len 36864 data: 5 5 5 4 3 3 3 3 3
14612 091/300 Write block 1040 len 36864 data: 5 6 6 5 4 5 3 2 2
14613 092/300 Write block 2783 len 16384 data: 4 3 4 3
14614 093/300 Flush
14615 094/300 Read block 2395 len 36864
14616 095/300 Read block 2113 len 40960
14617 096/300 Read block 3005 len 8192
14618 097/300 Write block 2313 len 8192 data: 3 3
14619 098/300 Read block 2149 len 24576
14620 099/300 Flush
14621 100/300 Write block 480 len 28672 data: 5 5 7 6 6 4 3
14622 101/300 Write block 3093 len 24576 data: 3 3 2 2 2 2
14623 102/300 Read block 3767 len 20480
14624 103/300 Read block 4570 len 4096
14625 104/300 Write block 4913 len 4096 data: 5
14626 105/300 Read block 4510 len 24576
14627 106/300 Write block 328 len 8192 data: 4 3
14628 107/300 Flush
14629 108/300 Read block 1005 len 28672
14630 109/300 Read block 117 len 20480
14631 110/300 Read block 3046 len 20480
14632 111/300 Read block 3938 len 36864
14633 112/300 Write block 876 len 12288 data: 4 4 4
14634 113/300 Write block 4645 len 28672 data: 5 7 8 8 7 8 6
14635 114/300 Read block 4228 len 4096
14636 115/300 Read block 1226 len 32768
14637 116/300 Write block 3649 len 4096 data: 2
14638 117/300 Read block 4412 len 40960
14639 118/300 Write block 1176 len 40960 data: 3 3 3 3 2 3 3 3 3 3
14640 119/300 Flush
14641 120/300 Read block 193 len 8192
14642 121/300 Write block 3272 len 8192 data: 4 5
14643 122/300 Write block 1996 len 12288 data: 5 6 4
14644 123/300 Read block 3994 len 16384
14645 124/300 Write block 361 len 16384 data: 8 8 7 7
14646 125/300 Write block 4191 len 28672 data: 4 5 3 2 2 3 3
14647 126/300 Read block 1659 len 28672
14648 127/300 Write block 1084 len 32768 data: 4 4 4 4 4 5 6 5
14649 128/300 Read block 2831 len 40960
14650 129/300 Write block 626 len 4096 data: 3
14651 130/300 Read block 2836 len 28672
14652 131/300 Flush
14653 132/300 Read block 903 len 16384
14654 133/300 Write block 2704 len 16384 data: 4 4 3 2
14655 134/300 Read block 1003 len 4096
14656 135/300 Read block 119 len 32768
14657 136/300 Write block 948 len 40960 data: 6 6 6 6 5 6 6 4 3 4
14658 137/300 Flush
14659 138/300 Write block 1196 len 4096 data: 4
14660 139/300 Read block 4745 len 40960
14661 140/300 Write block 3092 len 4096 data: 3
14662 141/300 Flush
14663 142/300 Read block 2875 len 4096
14664 143/300 Write block 3928 len 36864 data: 4 5 6 5 5 4 4 4 5
14665 144/300 Read block 615 len 36864
14666 145/300 Write block 3574 len 4096 data: 3
14667 146/300 Read block 765 len 12288
14668 147/300 Read block 4503 len 20480
14669 148/300 Read block 3789 len 36864
14670 149/300 Read block 3101 len 16384
14671 150/300 Write block 2299 len 32768 data: 4 3 3 2 2 2 2 2
14672 151/300 Read block 199 len 4096
14673 152/300 Read block 4237 len 16384
14674 153/300 Write block 2771 len 36864 data: 6 7 8 10 9 8 8 8 8
14675 154/300 Read block 1893 len 36864
14676 155/300 Read block 498 len 20480
14677 156/300 Read block 2381 len 4096
14678 157/300 Read block 4703 len 8192
14679 158/300 Write block 898 len 28672 data: 3 4 6 7 7 7 7
14680 159/300 Write block 1162 len 36864 data: 4 4 4 4 4 4 3 2 2
14681 160/300 Write block 4804 len 32768 data: 3 2 2 3 4 4 4 5
14682 161/300 Read block 3112 len 24576
14683 162/300 Write block 3865 len 36864 data: 2 2 2 3 3 5 4 4 4
14684 163/300 Read block 2825 len 24576
14685 164/300 Read block 1913 len 24576
14686 165/300 Read block 3575 len 8192
14687 166/300 Read block 4636 len 12288
14688 167/300 Read block 3884 len 16384
14689 168/300 Read block 2485 len 20480
14690 169/300 Read block 1339 len 40960
14691 170/300 Write block 2517 len 36864 data: 5 5 4 4 4 3 3 3 6
14692 171/300 Write block 1322 len 16384 data: 5 4 4 3
14693 172/300 Write block 1953 len 16384 data: 3 2 2 3
14694 173/300 Write block 4603 len 32768 data: 3 3 3 3 3 3 3 3
14695 174/300 Write block 3210 len 12288 data: 2 4 4
14696 175/300 Write block 4700 len 12288 data: 5 6 5
14697 176/300 Write block 3578 len 28672 data: 2 2 3 4 4 5 6
14698 177/300 Write block 1691 len 8192 data: 4 4
14699 178/300 Write block 635 len 28672 data: 5 5 6 5 5 5 4
14700 179/300 Write block 2038 len 8192 data: 3 3
14701 180/300 Read block 1381 len 16384
14702 181/300 Write block 4015 len 28672 data: 4 5 5 6 5 6 6
14703 182/300 Read block 3637 len 32768
14704 183/300 Read block 2492 len 40960
14705 184/300 Read block 3604 len 36864
14706 185/300 Write block 1249 len 12288 data: 5 5 3
14707 186/300 Read block 4457 len 32768
14708 187/300 Read block 2002 len 16384
14709 188/300 Read block 483 len 16384
14710 189/300 Write block 1567 len 8192 data: 5 6
14711 190/300 Read block 2414 len 8192
14712 191/300 Read block 875 len 36864
14713 192/300 Read block 4644 len 28672
14714 193/300 Flush
14715 194/300 Write block 2829 len 24576 data: 3 3 4 4 3 3
14716 195/300 Read block 4086 len 28672
14717 196/300 Read block 1913 len 16384
14718 197/300 Read block 354 len 40960
14719 198/300 Read block 1009 len 36864
14720 199/300 Read block 647 len 12288
14721 200/300 Read block 252 len 4096
14722 201/300 Write block 2693 len 36864 data: 2 2 2 3 3 4 4 4 4
14723 202/300 Read block 3054 len 28672
14724 203/300 Write block 424 len 36864 data: 4 5 5 3 3 4 4 4 4
14725 204/300 Read block 3837 len 32768
14726 205/300 Write block 503 len 36864 data: 4 5 6 6 7 7 7 6 3
14727 206/300 Flush
14728 207/300 Write block 4741 len 8192 data: 2 3
14729 208/300 Read block 3127 len 36864
14730 209/300 Read block 4236 len 4096
14731 210/300 Write block 480 len 12288 data: 6 6 8
14732 211/300 Read block 3298 len 12288
14733 212/300 Read block 556 len 24576
14734 213/300 Write block 1929 len 40960 data: 6 6 5 5 5 4 3 3 2 2
14735 214/300 Write block 4930 len 28672 data: 3 2 4 3 4 4 3
14736 215/300 Flush
14737 216/300 Read block 1371 len 4096
14738 217/300 Read block 1905 len 8192
14739 218/300 Read block 1306 len 40960
14740 219/300 Flush
14741 220/300 Write block 2689 len 24576 data: 2 2 2 2 3 3
14742 221/300 Flush
14743 222/300 Write block 4452 len 20480 data: 3 4 4 3 3
14744 223/300 Write block 754 len 4096 data: 3
14745 224/300 Write block 4189 len 8192 data: 5 5
14746 225/300 Read block 728 len 24576
14747 226/300 Write block 297 len 16384 data: 3 3 3 4
14748 227/300 Write block 1911 len 8192 data: 5 4
14749 228/300 Read block 414 len 24576
14750 229/300 Flush
14751 230/300 Write block 3320 len 32768 data: 4 4 5 4 4 4 4 4
14752 231/300 Read block 4426 len 36864
14753 232/300 Write block 234 len 32768 data: 5 5 4 4 4 4 3 3
14754 233/300 Read block 411 len 36864
14755 234/300 Read block 2508 len 28672
14756 235/300 Write block 2924 len 36864 data: 5 4 4 6 6 5 6 7 6
14757 236/300 Flush
14758 237/300 Flush
14759 238/300 Write block 3523 len 36864 data: 4 5 7 6 4 4 4 5 8
14760 239/300 Write block 3787 len 12288 data: 3 3 3
14761 240/300 Read block 3777 len 4096
14762 241/300 Write block 109 len 36864 data: 3 3 3 4 5 5 4 4 3
14763 242/300 Flush
14764 243/300 Read block 3589 len 16384
14765 244/300 Read block 4027 len 28672
14766 245/300 Write block 4226 len 12288 data: 7 6 6
14767 246/300 Write block 3779 len 40960 data: 6 5 6 5 4 3 2 2 4 4
14768 247/300 Write block 1539 len 12288 data: 3 2 2
14769 248/300 Write block 3061 len 12288 data: 4 4 4
14770 249/300 Read block 2491 len 12288
14771 250/300 Read block 188 len 36864
14772 251/300 Read block 4898 len 16384
14773 252/300 Write block 2885 len 4096 data: 2
14774 253/300 Read block 2063 len 28672
14775 254/300 Read block 4868 len 36864
14776 255/300 Read block 787 len 16384
14777 256/300 Write block 4361 len 12288 data: 4 4 4
14778 257/300 Read block 3082 len 12288
14779 258/300 Read block 3433 len 12288
14780 259/300 Read block 4292 len 16384
14781 260/300 Write block 4825 len 4096 data: 4
14782 261/300 Read block 2417 len 32768
14783 262/300 Write block 661 len 16384 data: 4 4 5 5
14784 263/300 Write block 4538 len 32768 data: 3 3 4 3 3 3 2 3
14785 264/300 Read block 3416 len 40960
14786 265/300 Read block 1275 len 4096
14787 266/300 Read block 2031 len 16384
14788 267/300 Read block 523 len 32768
14789 268/300 Read block 1176 len 36864
14790 269/300 Read block 4266 len 12288
14791 270/300 Write block 4958 len 24576 data: 3 2 2 2 2 4
14792 271/300 Read block 1875 len 24576
14793 272/300 Read block 644 len 4096
14794 273/300 Read block 284 len 36864
14795 274/300 Read block 626 len 32768
14796 275/300 Read block 3267 len 40960
14797 276/300 Write block 3679 len 28672 data: 4 4 4 4 4 4 5
14798 277/300 Flush
14799 278/300 Write block 2079 len 8192 data: 4 6
14800 279/300 Flush
14801 280/300 Read block 2594 len 16384
14802 281/300 Read block 2728 len 32768
14803 282/300 Read block 4258 len 40960
14804 283/300 Read block 574 len 16384
14805 284/300 Flush
14806 285/300 Read block 2389 len 32768
14807 286/300 Read block 3059 len 32768
14808 287/300 Read block 3353 len 36864
14809 288/300 Flush
14810 289/300 Write block 248 len 16384 data: 5 4 4 4
14811 290/300 Write block 3 len 4096 data: 4
14812 291/300 Write block 448 len 8192 data: 3 3
14813 292/300 Read block 1373 len 4096
14814 293/300 Write block 719 len 24576 data: 3 3 3 2 2 3
14815 294/300 Flush
14816 295/300 Write block 4286 len 24576 data: 4 5 4 4 4 2
14817 296/300 Flush
14818 297/300 Flush
14819 298/300 Write block 2408 len 8192 data: 3 4
14820 299/300 Read block 3183 len 36864
14821 300/300 Write block 1892 len 16384 data: 2 2 3 3
14822 301/300 Read block 467 len 20480
14823 Main task has work to do, go find it
14824 got dsc Start(1)
14825 start 1
14826 Dec 09 15:10:58.802 INFO request completed, latency_us: 256, response_code: 204, uri: /start/cid/1, method: POST, req_id: e1c1ae32-2419-4175-94e5-b6b331500df1, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
14827 Replay: started 1, returned:Ok(())
14828 ----------------------------------------------------------------
14829 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14830 States: Active Offline Active
14831 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14832 5228 Acked Write 6 Done Sent Done false
14833 5229 Acked Write 1 Done Sent Done false
14834 5230 Acked Read 7 Done Sent Done false
14835 5231 Acked Write 2 Done Sent Done false
14836 5232 Acked Read 9 Done Sent Done false
14837 5233 Acked Read 2 Done Sent Done false
14838 5234 Acked Write 1 Done Sent Done false
14839 5235 Acked Read 8 Done Sent Done false
14840 5236 Acked Read 8 Done Sent Done false
14841 5237 Acked Read 4 Done Sent Done false
14842 5238 Acked Write 3 Done Sent Done false
14843 5239 Acked Flush 0 Done Sent Done false
14844 5240 Acked Write 10 Done Sent Done false
14845 5241 Acked Read 8 Done Sent Done false
14846 5242 Acked Read 10 Done Sent Done false
14847 5243 Acked Write 9 Done Sent Done false
14848 5244 Acked Write 10 Done Sent Done false
14849 5245 Acked Read 2 Done Sent Done false
14850 5246 Acked Read 7 Done Sent Done false
14851 5247 Acked Flush 0 Done Sent Done false
14852 5248 Acked Flush 0 Done Sent Done false
14853 5249 Acked Read 10 Done Sent Done false
14854 5250 Acked Write 10 Done Sent Done false
14855 5251 Acked Flush 0 Done Sent Done false
14856 5252 Acked Write 8 Done Sent Done false
14857 5253 Acked Read 4 Done Sent Done false
14858 5254 Acked Write 10 Done Sent Done false
14859 5255 Acked Read 8 Done Sent Done false
14860 5256 Acked Flush 0 Done Sent Done false
14861 5257 Acked Write 4 Done Sent Done false
14862 5258 Acked Flush 0 Done Sent Done false
14863 5259 Acked Read 10 Done Sent Done false
14864 5260 Acked Read 10 Done Sent Done false
14865 5261 Acked Read 1 Done Sent Done false
14866 5262 Acked Write 9 Done Sent Done false
14867 5263 Acked Flush 0 Done Sent Done false
14868 5264 Acked Flush 0 Done Sent Done false
14869 5265 Acked Read 6 Done Sent Done false
14870 5266 Acked Flush 0 Done Sent Done false
14871 5267 Acked Write 5 Done Sent Done false
14872 5268 Acked Read 1 Done Sent Done false
14873 5269 Acked Read 7 Done Sent Done false
14874 5270 Acked Read 7 Done Sent Done false
14875 5271 Acked Write 9 Done Sent Done false
14876 5272 Acked Write 6 Done Sent Done false
14877 5273 Acked Read 10 Done Sent Done false
14878 5274 Acked Read 5 Done Sent Done false
14879 5275 Acked Write 7 Done Sent Done false
14880 5276 Acked Read 3 Done Sent Done false
14881 5277 Acked Write 5 Done Sent Done false
14882 5278 Acked Write 8 Done Sent Done false
14883 5279 Acked Write 10 Done Sent Done false
14884 5280 Acked Write 3 Done Sent Done false
14885 5281 Acked Write 6 Done Sent Done false
14886 5282 Acked Write 3 Done Sent Done false
14887 5283 Acked Write 9 Done Sent Done false
14888 5284 Acked Write 4 Done Sent Done false
14889 5285 Acked Write 2 Done Sent Done false
14890 5286 Acked Read 2 Done Sent Done false
14891 5287 Acked Write 5 Done Sent Done false
14892 5288 Acked Write 4 Done Sent Done false
14893 5289 Acked Read 4 Done Sent Done false
14894 5290 Acked Flush 0 Done Sent Done false
14895 5291 Acked Write 1 Done Sent Done false
14896 5292 Acked Read 5 Done Sent Done false
14897 5293 Acked Read 5 Done Sent Done false
14898 5294 Acked Write 8 Done Sent Done false
14899 5295 Acked Write 3 Done Sent Done false
14900 5296 Acked Write 9 Done Sent Done false
14901 5297 Acked Read 3 Done Sent Done false
14902 5298 Acked Read 7 Done Sent Done false
14903 5299 Acked Write 3 Done Sent Done false
14904 5300 Acked Read 3 Done Sent Done false
14905 5301 Acked Read 7 Done Sent Done false
14906 5302 Acked Read 10 Done Sent Done false
14907 5303 Acked Read 8 Done Sent Done false
14908 5304 Acked Read 9 Done Sent Done false
14909 5305 Acked Write 6 Done Sent Done false
14910 5306 Acked Write 10 Done Sent Done false
14911 5307 Acked Flush 0 Done Sent Done false
14912 5308 Acked Write 3 Done Sent Done false
14913 5309 Acked Flush 0 Done Sent Done false
14914 5310 Acked Read 6 Done Sent Done false
14915 5311 Acked Write 2 Done Sent Done false
14916 5312 Acked Read 10 Done Sent Done false
14917 5313 Acked Write 5 Done Sent Done false
14918 5314 Acked Write 10 Done Sent Done false
14919 5315 Acked Read 6 Done Sent Done false
14920 5316 Acked Read 1 Done Sent Done false
14921 5317 Acked Write 9 Done Sent Done false
14922 5318 Acked Write 9 Done Sent Done false
14923 5319 Acked Write 4 Done Sent Done false
14924 5320 Acked Flush 0 Done Sent Done false
14925 5321 Acked Read 9 Done Sent Done false
14926 5322 Acked Read 10 Done Sent Done false
14927 5323 Acked Read 2 Done Sent Done false
14928 5324 Acked Write 2 Done Sent Done false
14929 5325 Acked Read 6 Done Sent Done false
14930 5326 Acked Flush 0 Done Sent Done false
14931 5327 Acked Write 7 Done Sent Done false
14932 5328 Acked Write 6 Done Sent Done false
14933 5329 Acked Read 5 Done Sent Done false
14934 5330 Acked Read 1 Done Sent Done false
14935 5331 Acked Write 1 Done Sent Done false
14936 5332 Acked Read 6 Done Sent Done false
14937 5333 Acked Write 2 Done Sent Done false
14938 5334 Acked Flush 0 Done Sent Done false
14939 5335 Acked Read 7 Done Sent Done false
14940 5336 Acked Read 5 Done Sent Done false
14941 5337 Acked Read 5 Done Sent Done false
14942 5338 Acked Read 9 Done Sent Done false
14943 5339 Acked Write 3 Done Sent Done false
14944 5340 Acked Write 7 Done Sent Done false
14945 5341 Acked Read 1 Done Sent Done false
14946 5342 Acked Read 8 Done Sent Done false
14947 5343 Acked Write 1 Done Sent Done false
14948 5344 Acked Read 10 Done Sent Done false
14949 5345 Acked Write 10 Done Sent Done false
14950 5346 Acked Flush 0 Done Sent Done false
14951 5347 Acked Read 2 Done Sent Done false
14952 5348 Acked Write 2 Done Sent Done false
14953 5349 Acked Write 3 Done Sent Done false
14954 5350 Acked Read 4 Done Sent Done false
14955 5351 Acked Write 4 Done Sent Done false
14956 5352 Acked Write 7 Done Sent Done false
14957 5353 Acked Read 7 Done Sent Done false
14958 5354 Acked Write 8 Done Sent Done false
14959 5355 Acked Read 10 Done Sent Done false
14960 5356 Acked Write 1 Done Sent Done false
14961 5357 Acked Read 7 Done Sent Done false
14962 5358 Acked Flush 0 Done Sent Done false
14963 5359 Acked Read 4 Done Sent Done false
14964 5360 Acked Write 4 Done Sent Done false
14965 5361 Acked Read 1 Done Sent Done false
14966 5362 Acked Read 8 Done Sent Done false
14967 5363 Acked Write 10 Done Sent Done false
14968 5364 Acked Flush 0 Done Sent Done false
14969 5365 Acked Write 1 Done Sent Done false
14970 5366 Acked Read 10 Done Sent Done false
14971 5367 Acked Write 1 Done Sent Done false
14972 5368 Acked Flush 0 Done Sent Done false
14973 5369 Acked Read 1 Done Sent Done false
14974 5370 Acked Write 9 Done Sent Done false
14975 5371 Acked Read 9 Done Sent Done false
14976 5372 Acked Write 1 Done Sent Done false
14977 5373 Acked Read 3 Done Sent Done false
14978 5374 Acked Read 5 Done Sent Done false
14979 5375 Acked Read 9 Done Sent Done false
14980 5376 Acked Read 4 Done Sent Done false
14981 5377 Acked Write 8 Done Sent Done false
14982 5378 Acked Read 1 Done Sent Done false
14983 5379 Acked Read 4 Done Sent Done false
14984 5380 Acked Write 9 Done Sent Done false
14985 5381 Acked Read 9 Done Sent Done false
14986 5382 Acked Read 5 Done Sent Done false
14987 5383 Acked Read 1 Done Sent Done false
14988 5384 Acked Read 2 Done Sent Done false
14989 5385 Acked Write 7 Done Sent Done false
14990 5386 Acked Write 9 Done Sent Done false
14991 5387 Acked Write 8 Done Sent Done false
14992 5388 Acked Read 6 Done Sent Done false
14993 5389 Acked Write 9 Done Sent Done false
14994 5390 Acked Read 6 Done Sent Done false
14995 5391 Acked Read 6 Done Sent Done false
14996 5392 Acked Read 2 Done Sent Done false
14997 5393 Acked Read 3 Done Sent Done false
14998 5394 Acked Read 4 Done Sent Done false
14999 5395 Acked Read 5 Done Sent Done false
15000 5396 Acked Read 10 Done Sent Done false
15001 5397 Acked Write 9 Done Sent Done false
15002 5398 Acked Write 4 Done Sent Done false
15003 5399 Acked Write 4 Done Sent Done false
15004 5400 Acked Write 8 Done Sent Done false
15005 5401 Acked Write 3 Done Sent Done false
15006 5402 Acked Write 3 Done Sent Done false
15007 5403 Acked Write 7 Done Sent Done false
15008 5404 Acked Write 2 Done Sent Done false
15009 5405 Acked Write 7 Done Sent Done false
15010 5406 Acked Write 2 Done Sent Done false
15011 5407 Acked Read 4 Done Sent Done false
15012 5408 Acked Write 7 Done Sent Done false
15013 5409 Acked Read 8 Done Sent Done false
15014 5410 Acked Read 10 Done Sent Done false
15015 5411 Acked Read 9 Done Sent Done false
15016 5412 Acked Write 3 Done Sent Done false
15017 5413 Acked Read 8 Done Sent Done false
15018 5414 Acked Read 4 Done Sent Done false
15019 5415 Acked Read 4 Done Sent Done false
15020 5416 Acked Write 2 Done Sent Done false
15021 5417 Acked Read 2 Done Sent Done false
15022 5418 Acked Read 9 Done Sent Done false
15023 5419 Acked Read 7 Done Sent Done false
15024 5420 Acked Flush 0 Done Sent Done false
15025 5421 Acked Write 6 Done Sent Done false
15026 5422 Acked Read 7 Done Sent Done false
15027 5423 Acked Read 4 Done Sent Done false
15028 5424 Acked Read 10 Done Sent Done false
15029 5425 Acked Read 9 Done Sent Done false
15030 5426 Acked Read 3 Done Sent Done false
15031 5427 Acked Read 1 Done Sent Done false
15032 5428 Acked Write 9 Done Sent Done false
15033 5429 Acked Read 7 Done Sent Done false
15034 5430 Acked Write 9 Done Sent Done false
15035 5431 Acked Read 8 Done Sent Done false
15036 5432 Acked Write 9 Done Sent Done false
15037 5433 Acked Flush 0 Done Sent Done false
15038 5434 Acked Write 2 Done Sent Done false
15039 5435 Acked Read 9 Done Sent Done false
15040 5436 Acked Read 1 Done Sent Done false
15041 5437 Acked Write 3 Done Sent Done false
15042 5438 Acked Read 3 Done Sent Done false
15043 5439 Acked Read 6 Done Sent Done false
15044 5440 Acked Write 10 Done Sent Done false
15045 5441 Acked Write 7 Done Sent Done false
15046 5442 Acked Flush 0 Done Sent Done false
15047 5443 Acked Read 1 Done Sent Done false
15048 5444 Acked Read 2 Done Sent Done false
15049 5445 Acked Read 10 Done Sent Done false
15050 5446 Acked Flush 0 Done Sent Done false
15051 5447 Acked Write 6 Done Sent Done false
15052 5448 Acked Flush 0 Done Sent Done false
15053 5449 Acked Write 5 Done Sent Done false
15054 5450 Acked Write 1 Done Sent Done false
15055 5451 Acked Write 2 Done Sent Done false
15056 5452 Acked Read 6 Done Sent Done false
15057 5453 Acked Write 4 Done Sent Done false
15058 5454 Acked Write 2 Done Sent Done false
15059 5455 Acked Read 6 Done Sent Done false
15060 5456 Acked Flush 0 Done Sent Done false
15061 5457 Acked Write 8 Done Sent Done false
15062 5458 Acked Read 9 Done Sent Done false
15063 5459 Acked Write 8 Done Sent Done false
15064 5460 Acked Read 9 Done Sent Done false
15065 5461 Acked Read 7 Done Sent Done false
15066 5462 Acked Write 9 Done Sent Done false
15067 5463 Acked Flush 0 Done Sent Done false
15068 5464 Acked Flush 0 Done Sent Done false
15069 5465 Acked Write 9 Done Sent Done false
15070 5466 Acked Write 3 Done Sent Done false
15071 5467 Acked Read 1 Done Sent Done false
15072 5468 Acked Write 9 Done Sent Done false
15073 5469 Acked Flush 0 Done Sent Done false
15074 5470 Acked Read 4 Done Sent Done false
15075 5471 Acked Read 7 Done Sent Done false
15076 5472 Acked Write 3 Done Sent Done false
15077 5473 Acked Write 10 Done Sent Done false
15078 5474 Acked Write 3 Done Sent Done false
15079 5475 Acked Write 3 Done Sent Done false
15080 5476 Acked Read 3 Done Sent Done false
15081 5477 Acked Read 9 Done Sent Done false
15082 5478 Acked Read 4 Done Sent Done false
15083 5479 Acked Write 1 Done Sent Done false
15084 5480 Acked Read 7 Done Sent Done false
15085 5481 Acked Read 9 Done Sent Done false
15086 5482 Acked Read 4 Done Sent Done false
15087 5483 Acked Write 3 Done Sent Done false
15088 5484 Acked Read 3 Done Sent Done false
15089 5485 Acked Read 3 Done Sent Done false
15090 5486 Acked Read 4 Done Sent Done false
15091 5487 Acked Write 1 Done Sent Done false
15092 5488 Acked Read 8 Done Sent Done false
15093 5489 Acked Write 4 Done Sent Done false
15094 5490 Acked Write 8 Done Sent Done false
15095 5491 Acked Read 10 Done Sent Done false
15096 5492 Acked Read 1 Done Sent Done false
15097 5493 Acked Read 4 Done Sent Done false
15098 5494 Acked Read 8 Done Sent Done false
15099 5495 Acked Read 9 Done Sent Done false
15100 5496 Acked Read 3 Done Sent Done false
15101 5497 Acked Write 6 Done Sent Done false
15102 5498 Acked Read 6 Done Sent Done false
15103 5499 Acked Read 1 Done Sent Done false
15104 5500 Acked Read 9 Done Sent Done false
15105 5501 Acked Read 8 Done Sent Done false
15106 5502 Acked Read 10 Done Sent Done false
15107 5503 Acked Write 7 Done Sent Done false
15108 5504 Acked Flush 0 Done Sent Done false
15109 5505 Acked Write 2 Done Sent Done false
15110 5506 Acked Flush 0 Done Sent Done false
15111 5507 Acked Read 4 Done Sent Done false
15112 5508 Acked Read 8 Done Sent Done false
15113 5509 Acked Read 10 Done Sent Done false
15114 5510 Acked Read 4 Done Sent Done false
15115 5511 Acked Flush 0 Done Sent Done false
15116 5512 Acked Read 8 Done Sent Done false
15117 5513 Acked Read 8 Done Sent Done false
15118 5514 Acked Read 9 Done Sent Done false
15119 5515 Acked Flush 0 Done Sent Done false
15120 5516 Acked Write 4 Done Sent Done false
15121 5517 Acked Write 1 Done Sent Done false
15122 5518 Acked Write 2 Done Sent Done false
15123 5519 Acked Read 1 Done Sent Done false
15124 5520 Acked Write 6 Done Sent Done false
15125 5521 Acked Flush 0 Done Sent Done false
15126 5522 Acked Write 6 Done Sent Done false
15127 5523 Acked Flush 0 Done Sent Done false
15128 5524 Acked Flush 0 Done Sent Done false
15129 5525 Acked Write 2 Done Sent Done false
15130 5526 Acked Read 9 Done Sent Done false
15131 5527 Acked Write 4 Done Sent Done false
15132 5528 Acked Read 5 Done Sent Done false
15133 STATES DS:0 DS:1 DS:2 TOTAL
15134 Sent 0 301 0 301
15135 Done 301 0 301 602
15136 Skipped 0 0 0 0
15137 Error 0 0 0 0
15138 Last Flush: 5524 5227 5524
15139 Downstairs last five completed: 5227 5226 5225 5224 5223
15140 Upstairs last five completed: 5528 5527 5526 5525 5524
15141 CLIENT: Up:0 ds:301 act:2
15142 [8820] got start action
15143 [8820] I am going to restart
15144 Starting downstairs at port 8820
15145 Make output file at "/tmp/dsc/downstairs-8820.txt"
15146 [8820][1] reports Starting
15147 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1216)
15148 [8820][1] reports Running
15149 ----------------------------------------------------------------
15150 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
15151 States: Active Offline Active
15152 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15153 5228 Acked Write 6 Done Sent Done false
15154 5229 Acked Write 1 Done Sent Done false
15155 5230 Acked Read 7 Done Sent Done false
15156 5231 Acked Write 2 Done Sent Done false
15157 5232 Acked Read 9 Done Sent Done false
15158 5233 Acked Read 2 Done Sent Done false
15159 5234 Acked Write 1 Done Sent Done false
15160 5235 Acked Read 8 Done Sent Done false
15161 5236 Acked Read 8 Done Sent Done false
15162 5237 Acked Read 4 Done Sent Done false
15163 5238 Acked Write 3 Done Sent Done false
15164 5239 Acked Flush 0 Done Sent Done false
15165 5240 Acked Write 10 Done Sent Done false
15166 5241 Acked Read 8 Done Sent Done false
15167 5242 Acked Read 10 Done Sent Done false
15168 5243 Acked Write 9 Done Sent Done false
15169 5244 Acked Write 10 Done Sent Done false
15170 5245 Acked Read 2 Done Sent Done false
15171 5246 Acked Read 7 Done Sent Done false
15172 5247 Acked Flush 0 Done Sent Done false
15173 5248 Acked Flush 0 Done Sent Done false
15174 5249 Acked Read 10 Done Sent Done false
15175 5250 Acked Write 10 Done Sent Done false
15176 5251 Acked Flush 0 Done Sent Done false
15177 5252 Acked Write 8 Done Sent Done false
15178 5253 Acked Read 4 Done Sent Done false
15179 5254 Acked Write 10 Done Sent Done false
15180 5255 Acked Read 8 Done Sent Done false
15181 5256 Acked Flush 0 Done Sent Done false
15182 5257 Acked Write 4 Done Sent Done false
15183 5258 Acked Flush 0 Done Sent Done false
15184 5259 Acked Read 10 Done Sent Done false
15185 5260 Acked Read 10 Done Sent Done false
15186 5261 Acked Read 1 Done Sent Done false
15187 5262 Acked Write 9 Done Sent Done false
15188 5263 Acked Flush 0 Done Sent Done false
15189 5264 Acked Flush 0 Done Sent Done false
15190 5265 Acked Read 6 Done Sent Done false
15191 5266 Acked Flush 0 Done Sent Done false
15192 5267 Acked Write 5 Done Sent Done false
15193 5268 Acked Read 1 Done Sent Done false
15194 5269 Acked Read 7 Done Sent Done false
15195 5270 Acked Read 7 Done Sent Done false
15196 5271 Acked Write 9 Done Sent Done false
15197 5272 Acked Write 6 Done Sent Done false
15198 5273 Acked Read 10 Done Sent Done false
15199 5274 Acked Read 5 Done Sent Done false
15200 5275 Acked Write 7 Done Sent Done false
15201 5276 Acked Read 3 Done Sent Done false
15202 5277 Acked Write 5 Done Sent Done false
15203 5278 Acked Write 8 Done Sent Done false
15204 5279 Acked Write 10 Done Sent Done false
15205 5280 Acked Write 3 Done Sent Done false
15206 5281 Acked Write 6 Done Sent Done false
15207 5282 Acked Write 3 Done Sent Done false
15208 5283 Acked Write 9 Done Sent Done false
15209 5284 Acked Write 4 Done Sent Done false
15210 5285 Acked Write 2 Done Sent Done false
15211 5286 Acked Read 2 Done Sent Done false
15212 5287 Acked Write 5 Done Sent Done false
15213 5288 Acked Write 4 Done Sent Done false
15214 5289 Acked Read 4 Done Sent Done false
15215 5290 Acked Flush 0 Done Sent Done false
15216 5291 Acked Write 1 Done Sent Done false
15217 5292 Acked Read 5 Done Sent Done false
15218 5293 Acked Read 5 Done Sent Done false
15219 5294 Acked Write 8 Done Sent Done false
15220 5295 Acked Write 3 Done Sent Done false
15221 5296 Acked Write 9 Done Sent Done false
15222 5297 Acked Read 3 Done Sent Done false
15223 5298 Acked Read 7 Done Sent Done false
15224 5299 Acked Write 3 Done Sent Done false
15225 5300 Acked Read 3 Done Sent Done false
15226 5301 Acked Read 7 Done Sent Done false
15227 5302 Acked Read 10 Done Sent Done false
15228 5303 Acked Read 8 Done Sent Done false
15229 5304 Acked Read 9 Done Sent Done false
15230 5305 Acked Write 6 Done Sent Done false
15231 5306 Acked Write 10 Done Sent Done false
15232 5307 Acked Flush 0 Done Sent Done false
15233 5308 Acked Write 3 Done Sent Done false
15234 5309 Acked Flush 0 Done Sent Done false
15235 5310 Acked Read 6 Done Sent Done false
15236 5311 Acked Write 2 Done Sent Done false
15237 5312 Acked Read 10 Done Sent Done false
15238 5313 Acked Write 5 Done Sent Done false
15239 5314 Acked Write 10 Done Sent Done false
15240 5315 Acked Read 6 Done Sent Done false
15241 5316 Acked Read 1 Done Sent Done false
15242 5317 Acked Write 9 Done Sent Done false
15243 5318 Acked Write 9 Done Sent Done false
15244 5319 Acked Write 4 Done Sent Done false
15245 5320 Acked Flush 0 Done Sent Done false
15246 5321 Acked Read 9 Done Sent Done false
15247 5322 Acked Read 10 Done Sent Done false
15248 5323 Acked Read 2 Done Sent Done false
15249 5324 Acked Write 2 Done Sent Done false
15250 5325 Acked Read 6 Done Sent Done false
15251 5326 Acked Flush 0 Done Sent Done false
15252 5327 Acked Write 7 Done Sent Done false
15253 5328 Acked Write 6 Done Sent Done false
15254 5329 Acked Read 5 Done Sent Done false
15255 5330 Acked Read 1 Done Sent Done false
15256 5331 Acked Write 1 Done Sent Done false
15257 5332 Acked Read 6 Done Sent Done false
15258 5333 Acked Write 2 Done Sent Done false
15259 5334 Acked Flush 0 Done Sent Done false
15260 5335 Acked Read 7 Done Sent Done false
15261 5336 Acked Read 5 Done Sent Done false
15262 5337 Acked Read 5 Done Sent Done false
15263 5338 Acked Read 9 Done Sent Done false
15264 5339 Acked Write 3 Done Sent Done false
15265 5340 Acked Write 7 Done Sent Done false
15266 5341 Acked Read 1 Done Sent Done false
15267 5342 Acked Read 8 Done Sent Done false
15268 5343 Acked Write 1 Done Sent Done false
15269 5344 Acked Read 10 Done Sent Done false
15270 5345 Acked Write 10 Done Sent Done false
15271 5346 Acked Flush 0 Done Sent Done false
15272 5347 Acked Read 2 Done Sent Done false
15273 5348 Acked Write 2 Done Sent Done false
15274 5349 Acked Write 3 Done Sent Done false
15275 5350 Acked Read 4 Done Sent Done false
15276 5351 Acked Write 4 Done Sent Done false
15277 5352 Acked Write 7 Done Sent Done false
15278 5353 Acked Read 7 Done Sent Done false
15279 5354 Acked Write 8 Done Sent Done false
15280 5355 Acked Read 10 Done Sent Done false
15281 5356 Acked Write 1 Done Sent Done false
15282 5357 Acked Read 7 Done Sent Done false
15283 5358 Acked Flush 0 Done Sent Done false
15284 5359 Acked Read 4 Done Sent Done false
15285 5360 Acked Write 4 Done Sent Done false
15286 5361 Acked Read 1 Done Sent Done false
15287 5362 Acked Read 8 Done Sent Done false
15288 5363 Acked Write 10 Done Sent Done false
15289 5364 Acked Flush 0 Done Sent Done false
15290 5365 Acked Write 1 Done Sent Done false
15291 5366 Acked Read 10 Done Sent Done false
15292 5367 Acked Write 1 Done Sent Done false
15293 5368 Acked Flush 0 Done Sent Done false
15294 5369 Acked Read 1 Done Sent Done false
15295 5370 Acked Write 9 Done Sent Done false
15296 5371 Acked Read 9 Done Sent Done false
15297 5372 Acked Write 1 Done Sent Done false
15298 5373 Acked Read 3 Done Sent Done false
15299 5374 Acked Read 5 Done Sent Done false
15300 5375 Acked Read 9 Done Sent Done false
15301 5376 Acked Read 4 Done Sent Done false
15302 5377 Acked Write 8 Done Sent Done false
15303 5378 Acked Read 1 Done Sent Done false
15304 5379 Acked Read 4 Done Sent Done false
15305 5380 Acked Write 9 Done Sent Done false
15306 5381 Acked Read 9 Done Sent Done false
15307 5382 Acked Read 5 Done Sent Done false
15308 5383 Acked Read 1 Done Sent Done false
15309 5384 Acked Read 2 Done Sent Done false
15310 5385 Acked Write 7 Done Sent Done false
15311 5386 Acked Write 9 Done Sent Done false
15312 5387 Acked Write 8 Done Sent Done false
15313 5388 Acked Read 6 Done Sent Done false
15314 5389 Acked Write 9 Done Sent Done false
15315 5390 Acked Read 6 Done Sent Done false
15316 5391 Acked Read 6 Done Sent Done false
15317 5392 Acked Read 2 Done Sent Done false
15318 5393 Acked Read 3 Done Sent Done false
15319 5394 Acked Read 4 Done Sent Done false
15320 5395 Acked Read 5 Done Sent Done false
15321 5396 Acked Read 10 Done Sent Done false
15322 5397 Acked Write 9 Done Sent Done false
15323 5398 Acked Write 4 Done Sent Done false
15324 5399 Acked Write 4 Done Sent Done false
15325 5400 Acked Write 8 Done Sent Done false
15326 5401 Acked Write 3 Done Sent Done false
15327 5402 Acked Write 3 Done Sent Done false
15328 5403 Acked Write 7 Done Sent Done false
15329 5404 Acked Write 2 Done Sent Done false
15330 5405 Acked Write 7 Done Sent Done false
15331 5406 Acked Write 2 Done Sent Done false
15332 5407 Acked Read 4 Done Sent Done false
15333 5408 Acked Write 7 Done Sent Done false
15334 5409 Acked Read 8 Done Sent Done false
15335 5410 Acked Read 10 Done Sent Done false
15336 5411 Acked Read 9 Done Sent Done false
15337 5412 Acked Write 3 Done Sent Done false
15338 5413 Acked Read 8 Done Sent Done false
15339 5414 Acked Read 4 Done Sent Done false
15340 5415 Acked Read 4 Done Sent Done false
15341 5416 Acked Write 2 Done Sent Done false
15342 5417 Acked Read 2 Done Sent Done false
15343 5418 Acked Read 9 Done Sent Done false
15344 5419 Acked Read 7 Done Sent Done false
15345 5420 Acked Flush 0 Done Sent Done false
15346 5421 Acked Write 6 Done Sent Done false
15347 5422 Acked Read 7 Done Sent Done false
15348 5423 Acked Read 4 Done Sent Done false
15349 5424 Acked Read 10 Done Sent Done false
15350 5425 Acked Read 9 Done Sent Done false
15351 5426 Acked Read 3 Done Sent Done false
15352 5427 Acked Read 1 Done Sent Done false
15353 5428 Acked Write 9 Done Sent Done false
15354 5429 Acked Read 7 Done Sent Done false
15355 5430 Acked Write 9 Done Sent Done false
15356 5431 Acked Read 8 Done Sent Done false
15357 5432 Acked Write 9 Done Sent Done false
15358 5433 Acked Flush 0 Done Sent Done false
15359 5434 Acked Write 2 Done Sent Done false
15360 5435 Acked Read 9 Done Sent Done false
15361 5436 Acked Read 1 Done Sent Done false
15362 5437 Acked Write 3 Done Sent Done false
15363 5438 Acked Read 3 Done Sent Done false
15364 5439 Acked Read 6 Done Sent Done false
15365 5440 Acked Write 10 Done Sent Done false
15366 5441 Acked Write 7 Done Sent Done false
15367 5442 Acked Flush 0 Done Sent Done false
15368 5443 Acked Read 1 Done Sent Done false
15369 5444 Acked Read 2 Done Sent Done false
15370 5445 Acked Read 10 Done Sent Done false
15371 5446 Acked Flush 0 Done Sent Done false
15372 5447 Acked Write 6 Done Sent Done false
15373 5448 Acked Flush 0 Done Sent Done false
15374 5449 Acked Write 5 Done Sent Done false
15375 5450 Acked Write 1 Done Sent Done false
15376 5451 Acked Write 2 Done Sent Done false
15377 5452 Acked Read 6 Done Sent Done false
15378 5453 Acked Write 4 Done Sent Done false
15379 5454 Acked Write 2 Done Sent Done false
15380 5455 Acked Read 6 Done Sent Done false
15381 5456 Acked Flush 0 Done Sent Done false
15382 5457 Acked Write 8 Done Sent Done false
15383 5458 Acked Read 9 Done Sent Done false
15384 5459 Acked Write 8 Done Sent Done false
15385 5460 Acked Read 9 Done Sent Done false
15386 5461 Acked Read 7 Done Sent Done false
15387 5462 Acked Write 9 Done Sent Done false
15388 5463 Acked Flush 0 Done Sent Done false
15389 5464 Acked Flush 0 Done Sent Done false
15390 5465 Acked Write 9 Done Sent Done false
15391 5466 Acked Write 3 Done Sent Done false
15392 5467 Acked Read 1 Done Sent Done false
15393 5468 Acked Write 9 Done Sent Done false
15394 5469 Acked Flush 0 Done Sent Done false
15395 5470 Acked Read 4 Done Sent Done false
15396 5471 Acked Read 7 Done Sent Done false
15397 5472 Acked Write 3 Done Sent Done false
15398 5473 Acked Write 10 Done Sent Done false
15399 5474 Acked Write 3 Done Sent Done false
15400 5475 Acked Write 3 Done Sent Done false
15401 5476 Acked Read 3 Done Sent Done false
15402 5477 Acked Read 9 Done Sent Done false
15403 5478 Acked Read 4 Done Sent Done false
15404 5479 Acked Write 1 Done Sent Done false
15405 5480 Acked Read 7 Done Sent Done false
15406 5481 Acked Read 9 Done Sent Done false
15407 5482 Acked Read 4 Done Sent Done false
15408 5483 Acked Write 3 Done Sent Done false
15409 5484 Acked Read 3 Done Sent Done false
15410 5485 Acked Read 3 Done Sent Done false
15411 5486 Acked Read 4 Done Sent Done false
15412 5487 Acked Write 1 Done Sent Done false
15413 5488 Acked Read 8 Done Sent Done false
15414 5489 Acked Write 4 Done Sent Done false
15415 5490 Acked Write 8 Done Sent Done false
15416 5491 Acked Read 10 Done Sent Done false
15417 5492 Acked Read 1 Done Sent Done false
15418 5493 Acked Read 4 Done Sent Done false
15419 5494 Acked Read 8 Done Sent Done false
15420 5495 Acked Read 9 Done Sent Done false
15421 5496 Acked Read 3 Done Sent Done false
15422 5497 Acked Write 6 Done Sent Done false
15423 5498 Acked Read 6 Done Sent Done false
15424 5499 Acked Read 1 Done Sent Done false
15425 5500 Acked Read 9 Done Sent Done false
15426 5501 Acked Read 8 Done Sent Done false
15427 5502 Acked Read 10 Done Sent Done false
15428 5503 Acked Write 7 Done Sent Done false
15429 5504 Acked Flush 0 Done Sent Done false
15430 5505 Acked Write 2 Done Sent Done false
15431 5506 Acked Flush 0 Done Sent Done false
15432 5507 Acked Read 4 Done Sent Done false
15433 5508 Acked Read 8 Done Sent Done false
15434 5509 Acked Read 10 Done Sent Done false
15435 5510 Acked Read 4 Done Sent Done false
15436 5511 Acked Flush 0 Done Sent Done false
15437 5512 Acked Read 8 Done Sent Done false
15438 5513 Acked Read 8 Done Sent Done false
15439 5514 Acked Read 9 Done Sent Done false
15440 5515 Acked Flush 0 Done Sent Done false
15441 5516 Acked Write 4 Done Sent Done false
15442 5517 Acked Write 1 Done Sent Done false
15443 5518 Acked Write 2 Done Sent Done false
15444 5519 Acked Read 1 Done Sent Done false
15445 5520 Acked Write 6 Done Sent Done false
15446 5521 Acked Flush 0 Done Sent Done false
15447 5522 Acked Write 6 Done Sent Done false
15448 5523 Acked Flush 0 Done Sent Done false
15449 5524 Acked Flush 0 Done Sent Done false
15450 5525 Acked Write 2 Done Sent Done false
15451 5526 Acked Read 9 Done Sent Done false
15452 5527 Acked Write 4 Done Sent Done false
15453 5528 Acked Read 5 Done Sent Done false
15454 5529 Acked Flush 0 Done Sent Done false
15455 STATES DS:0 DS:1 DS:2 TOTAL
15456 Sent 0 302 0 302
15457 Done 302 0 302 604
15458 Skipped 0 0 0 0
15459 Error 0 0 0 0
15460 Last Flush: 5529 5227 5529
15461 Downstairs last five completed: 5227 5226 5225 5224 5223
15462 Upstairs last five completed: 5529 5528 5527 5526 5525
15463 CLIENT: Up:0 ds:302 act:2
154642024-12-09T15:11:04.559ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154652024-12-09T15:11:04.559ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50571) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154662024-12-09T15:11:04.560ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154672024-12-09T15:11:04.561ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154682024-12-09T15:11:04.561ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154692024-12-09T15:11:04.561ZINFOcrucible: send last flush ID to this DS: 5227 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154702024-12-09T15:11:04.561ZINFOcrucible: Replied this last flush ID: 5227 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154712024-12-09T15:11:04.561ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154722024-12-09T15:11:04.561ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154732024-12-09T15:11:04.561ZINFOcrucible: [1] client re-new 302 jobs since flush 5227 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154742024-12-09T15:11:04.562ZINFOcrucible: [1] Replayed 302 jobs since flush: 5227 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
15475 ----------------------------------------------------------------
15476 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15477 Downstairs last five completed: 5529 5528 5527 5526 5525
15478 Upstairs last five completed: 5529 5528 5527 5526 5525
15479 CLIENT: Up:0 ds:0 act:3
15480 Replay: All jobs finished
15481 Main task has work to do, go find it
15482 got dsc Stop(2)
15483 stop 2
15484 Dec 09 15:11:06.807[8830] Got stop action so:false kr:false
15485 INFO request completed, latency_us: 357, response_code: 204, uri: /stop/cid/2, method: POST, req_id: a4b5b65e-783f-439c-8461-3e65ddff369e, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
154862024-12-09T15:11:06.807ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154872024-12-09T15:11:06.807ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154882024-12-09T15:11:06.807ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154892024-12-09T15:11:06.808ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154902024-12-09T15:11:06.808ZINFOcrucible: [2] Transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
154912024-12-09T15:11:06.808ZINFOcrucible: client task is exiting = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
15492 Dec 09 15:11:06.808 INFO request completed, latency_us: 159, response_code: 200, uri: /state/cid/2, method: GET, req_id: bf9b60d6-614c-4f28-870d-6d841bca729a, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
15493 [8830] Exited with: ExitStatus(unix_wait_status(9))
15494 [8830][2] reports Exit
15495 Dec 09 15:11:10.810 INFO request completed, latency_us: 235, response_code: 200, uri: /state/cid/2, method: GET, req_id: a4237b7b-2a35-42b3-8549-97f930b9db06, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
15496 001/300 Write block 75 len 40960 data: 3 3 3 3 4 4 4 5 5 5
15497 002/300 Flush
15498 003/300 Read block 3692 len 36864
15499 004/300 Flush
15500 005/300 Read block 3299 len 12288
15501 006/300 Write block 32 len 32768 data: 4 3 3 3 2 4 4 4
15502 007/300 Read block 4770 len 40960
15503 008/300 Read block 4646 len 16384
15504 009/300 Read block 1431 len 12288
15505 010/300 Read block 1395 len 28672
15506 011/300 Read block 2573 len 28672
15507 012/300 Read block 2668 len 24576
15508 013/300 Read block 1215 len 20480
15509 014/300 Write block 472 len 32768 data: 3 3 2 3 3 4 5 5
15510 015/300 Write block 1454 len 28672 data: 4 3 3 4 4 4 4
15511 016/300 Read block 3605 len 32768
15512 017/300 Read block 2255 len 20480
15513 018/300 Read block 4135 len 16384
15514 019/300 Write block 1470 len 32768 data: 5 4 4 3 4 4 4 2
15515 020/300 Flush
15516 021/300 Read block 2615 len 4096
15517 022/300 Write block 1825 len 28672 data: 4 4 5 6 5 4 4
15518 023/300 Read block 1557 len 12288
15519 024/300 Write block 1159 len 8192 data: 3 3
15520 025/300 Read block 862 len 32768
15521 026/300 Flush
15522 027/300 Read block 4520 len 16384
15523 028/300 Write block 2537 len 24576 data: 6 5 5 6 7 5
15524 029/300 Write block 2016 len 12288 data: 7 6 6
15525 030/300 Read block 4944 len 32768
15526 031/300 Flush
15527 032/300 Write block 3150 len 20480 data: 7 8 7 6 6
15528 033/300 Write block 2241 len 20480 data: 3 3 3 3 3
15529 034/300 Read block 4881 len 12288
15530 035/300 Read block 1581 len 8192
15531 036/300 Write block 726 len 24576 data: 3 3 3 3 3 3
15532 037/300 Read block 1682 len 12288
15533 038/300 Read block 2045 len 20480
15534 039/300 Read block 3578 len 12288
15535 040/300 Read block 386 len 24576
15536 041/300 Read block 1293 len 36864
15537 042/300 Write block 3616 len 20480 data: 6 6 6 7 7
15538 043/300 Write block 1727 len 32768 data: 4 4 4 4 4 4 5 5
15539 044/300 Read block 2474 len 32768
15540 045/300 Read block 4456 len 40960
15541 046/300 Write block 3119 len 20480 data: 4 5 5 5 5
15542 047/300 Write block 3459 len 24576 data: 4 4 4 4 5 6
15543 048/300 Read block 2360 len 20480
15544 049/300 Read block 1783 len 16384
15545 050/300 Write block 1789 len 40960 data: 4 4 5 5 6 6 8 7 6 5
15546 051/300 Read block 1624 len 4096
15547 052/300 Write block 1043 len 16384 data: 6 5 6 4
15548 053/300 Read block 1824 len 24576
15549 054/300 Write block 2746 len 20480 data: 4 3 3 4 3
15550 055/300 Write block 2571 len 20480 data: 2 2 2 2 2
15551 056/300 Read block 3489 len 36864
15552 057/300 Write block 3256 len 36864 data: 4 4 4 4 3 4 3 4 3
15553 058/300 Read block 1421 len 12288
15554 059/300 Flush
15555 060/300 Write block 4156 len 36864 data: 5 4 6 7 4 5 4 4 3
15556 061/300 Write block 1896 len 36864 data: 2 2 2 3 2 3 3 2 2
15557 062/300 Read block 3626 len 12288
15558 063/300 Read block 1122 len 24576
15559 064/300 Write block 2678 len 4096 data: 3
15560 065/300 Write block 187 len 4096 data: 5
15561 066/300 Read block 685 len 16384
15562 067/300 Read block 3213 len 4096
15563 068/300 Read block 3824 len 4096
15564 069/300 Write block 3160 len 12288 data: 7 8 8
15565 070/300 Flush
15566 071/300 Read block 1270 len 12288
15567 072/300 Write block 2572 len 40960 data: 3 3 3 3 2 2 2 3 3 5
15568 073/300 Read block 288 len 24576
15569 074/300 Write block 4393 len 24576 data: 2 2 3 3 3 3
15570 075/300 Read block 2006 len 16384
15571 076/300 Read block 159 len 40960
15572 077/300 Write block 4071 len 24576 data: 4 3 2 2 2 2
15573 078/300 Read block 4532 len 4096
15574 079/300 Read block 1028 len 28672
15575 080/300 Write block 3934 len 12288 data: 5 5 6
15576 081/300 Write block 3582 len 40960 data: 5 6 7 5 5 5 4 2 3 3
15577 082/300 Read block 2254 len 28672
15578 083/300 Write block 4052 len 4096 data: 5
15579 084/300 Write block 3556 len 4096 data: 7
15580 085/300 Write block 2981 len 20480 data: 3 3 4 5 7
15581 086/300 Write block 3653 len 20480 data: 3 3 3 3 3
15582 087/300 Write block 4200 len 8192 data: 3 4
15583 088/300 Read block 2197 len 36864
15584 089/300 Read block 1204 len 4096
15585 090/300 Write block 2172 len 36864 data: 3 4 4 4 4 3 3 4 4
15586 091/300 Write block 3013 len 24576 data: 5 3 3 3 4 4
15587 092/300 Write block 3826 len 32768 data: 7 6 6 6 3 3 4 4
15588 093/300 Write block 3264 len 40960 data: 4 3 2 2 2 3 3 3 5 6
15589 094/300 Read block 4392 len 36864
15590 095/300 Write block 3939 len 40960 data: 4 3 3 3 3 3 3 3 2 2
15591 096/300 Read block 839 len 20480
15592 097/300 Read block 1542 len 8192
15593 098/300 Read block 4276 len 36864
15594 099/300 Flush
15595 100/300 Read block 2524 len 12288
15596 101/300 Read block 2437 len 36864
15597 102/300 Read block 1460 len 24576
15598 103/300 Read block 4308 len 8192
15599 104/300 Write block 2986 len 28672 data: 8 9 8 7 6 6 6
15600 105/300 Flush
15601 106/300 Read block 1495 len 32768
15602 107/300 Read block 1895 len 36864
15603 108/300 Write block 1885 len 36864 data: 4 5 5 3 3 3 3 3 3
15604 109/300 Write block 1426 len 20480 data: 4 3 4 4 3
15605 110/300 Write block 836 len 12288 data: 4 4 4
15606 111/300 Write block 3259 len 40960 data: 5 4 5 4 5 5 4 3 3 3
15607 112/300 Read block 133 len 40960
15608 113/300 Read block 1939 len 28672
15609 114/300 Write block 3311 len 8192 data: 4 4
15610 115/300 Write block 3802 len 28672 data: 7 5 4 3 3 4 5
15611 116/300 Write block 3425 len 12288 data: 4 3 3
15612 117/300 Read block 1471 len 40960
15613 118/300 Read block 1471 len 28672
15614 119/300 Read block 4877 len 8192
15615 120/300 Write block 505 len 12288 data: 7 7 8
15616 121/300 Flush
15617 122/300 Write block 3284 len 36864 data: 4 4 5 5 5 5 7 6 6
15618 123/300 Write block 830 len 24576 data: 3 3 3 4 4 4
15619 124/300 Write block 2003 len 16384 data: 4 5 5 4
15620 125/300 Read block 1703 len 36864
15621 126/300 Write block 565 len 24576 data: 2 2 2 2 2 3
15622 127/300 Write block 189 len 36864 data: 5 4 4 6 3 3 3 3 3
15623 128/300 Write block 1905 len 40960 data: 2 3 3 3 5 5 6 5 4 4
15624 129/300 Write block 2803 len 20480 data: 3 3 3 3 2
15625 130/300 Read block 763 len 4096
15626 131/300 Write block 3475 len 36864 data: 5 5 5 5 4 3 2 2 4
15627 132/300 Write block 2428 len 36864 data: 3 3 3 3 3 3 4 4 4
15628 133/300 Flush
15629 134/300 Write block 2110 len 20480 data: 3 3 4 4 3
15630 135/300 Read block 3347 len 36864
15631 136/300 Write block 2297 len 40960 data: 3 4 5 4 4 3 3 3 3 3
15632 137/300 Write block 1539 len 20480 data: 4 3 3 2 3
15633 138/300 Read block 1392 len 12288
15634 139/300 Read block 3823 len 36864
15635 140/300 Read block 4297 len 40960
15636 141/300 Read block 988 len 8192
15637 142/300 Write block 4056 len 4096 data: 2
15638 143/300 Read block 4554 len 32768
15639 144/300 Read block 4118 len 20480
15640 145/300 Read block 2041 len 20480
15641 146/300 Write block 2480 len 32768 data: 6 5 5 6 6 6 6 5
15642 147/300 Write block 1886 len 32768 data: 6 6 4 4 4 4 4 4
15643 148/300 Read block 3929 len 4096
15644 149/300 Read block 4355 len 12288
15645 150/300 Read block 331 len 36864
15646 151/300 Read block 975 len 4096
15647 152/300 Read block 4187 len 40960
15648 153/300 Read block 949 len 20480
15649 154/300 Write block 1400 len 24576 data: 3 2 2 2 4 4
15650 155/300 Write block 1084 len 40960 data: 5 5 5 5 5 6 7 6 5 5
15651 156/300 Read block 3638 len 32768
15652 157/300 Read block 3466 len 40960
15653 158/300 Read block 2133 len 20480
15654 159/300 Write block 4490 len 24576 data: 4 4 4 7 6 5
15655 160/300 Read block 4027 len 32768
15656 161/300 Write block 1169 len 36864 data: 3 3 3 3 3 4 3 4 4
15657 162/300 Read block 3132 len 40960
15658 163/300 Read block 276 len 40960
15659 164/300 Read block 4245 len 28672
15660 165/300 Read block 95 len 36864
15661 166/300 Read block 1612 len 32768
15662 167/300 Read block 2915 len 8192
15663 168/300 Read block 3377 len 4096
15664 169/300 Write block 3242 len 8192 data: 4 6
15665 170/300 Read block 1966 len 16384
15666 171/300 Write block 1923 len 28672 data: 3 3 3 5 5 4 7
15667 172/300 Read block 1160 len 28672
15668 173/300 Read block 1489 len 8192
15669 174/300 Read block 1464 len 4096
15670 175/300 Write block 4167 len 28672 data: 2 3 5 4 6 7 6
15671 176/300 Read block 912 len 28672
15672 177/300 Write block 3223 len 4096 data: 5
15673 178/300 Write block 128 len 16384 data: 4 2 2 4
15674 179/300 Write block 2384 len 32768 data: 3 3 3 4 5 5 5 5
15675 180/300 Write block 1441 len 28672 data: 3 5 6 5 5 5 5
15676 181/300 Write block 4750 len 20480 data: 5 5 4 4 4
15677 182/300 Write block 3369 len 4096 data: 2
15678 183/300 Write block 1271 len 40960 data: 6 6 6 6 5 5 5 4 5 3
15679 184/300 Read block 4505 len 12288
15680 185/300 Write block 4509 len 24576 data: 8 6 6 6 5 5
15681 186/300 Read block 478 len 40960
15682 187/300 Read block 3836 len 4096
15683 188/300 Write block 895 len 8192 data: 3 2
15684 189/300 Read block 1161 len 36864
15685 190/300 Write block 1336 len 32768 data: 8 9 8 7 5 6 4 4
15686 191/300 Write block 4241 len 28672 data: 4 4 6 5 5 6 6
15687 192/300 Read block 1880 len 28672
15688 193/300 Flush
15689 194/300 Read block 2649 len 28672
15690 195/300 Write block 3132 len 16384 data: 4 4 3 4
15691 196/300 Read block 3648 len 12288
15692 197/300 Write block 3198 len 24576 data: 4 3 3 3 3 3
15693 198/300 Read block 1882 len 28672
15694 199/300 Write block 4233 len 36864 data: 2 2 2 2 3 3 4 4 5
15695 200/300 Read block 398 len 36864
15696 201/300 Read block 3642 len 36864
15697 202/300 Read block 4530 len 12288
15698 203/300 Write block 251 len 20480 data: 5 4 4 3 2
15699 204/300 Read block 4199 len 32768
15700 205/300 Write block 1216 len 28672 data: 4 5 5 5 5 5 4
15701 206/300 Read block 4178 len 40960
15702 207/300 Flush
15703 208/300 Read block 4164 len 12288
15704 209/300 Write block 4068 len 16384 data: 4 4 4 5
15705 210/300 Flush
15706 211/300 Read block 1869 len 32768
15707 212/300 Write block 2903 len 8192 data: 4 3
15708 213/300 Read block 2975 len 36864
15709 214/300 Read block 3397 len 32768
15710 215/300 Flush
15711 216/300 Write block 4286 len 28672 data: 5 6 5 5 5 3 2
15712 217/300 Read block 3688 len 8192
15713 218/300 Read block 4304 len 40960
15714 219/300 Write block 4985 len 32768 data: 6 6 7 7 8 6 8 7
15715 220/300 Read block 2531 len 24576
15716 221/300 Write block 716 len 4096 data: 3
15717 222/300 Write block 906 len 36864 data: 5 4 4 6 4 4 4 4 4
15718 223/300 Write block 3471 len 8192 data: 5 4
15719 224/300 Write block 4910 len 8192 data: 2 2
15720 225/300 Read block 4325 len 36864
15721 226/300 Write block 4676 len 36864 data: 2 2 4 4 3 3 3 4 4
15722 227/300 Read block 3344 len 24576
15723 228/300 Read block 3554 len 28672
15724 229/300 Write block 2180 len 20480 data: 5 4 5 5 4
15725 230/300 Write block 4282 len 32768 data: 3 3 3 4 6 7 6 6
15726 231/300 Flush
15727 232/300 Read block 202 len 4096
15728 233/300 Read block 4157 len 12288
15729 234/300 Read block 1485 len 12288
15730 235/300 Read block 1369 len 16384
15731 236/300 Read block 3211 len 4096
15732 237/300 Write block 2282 len 16384 data: 4 3 3 3
15733 238/300 Write block 1984 len 4096 data: 7
15734 239/300 Write block 900 len 8192 data: 7 8
15735 240/300 Flush
15736 241/300 Write block 1653 len 8192 data: 5 5
15737 242/300 Write block 738 len 20480 data: 3 4 5 6 6
15738 243/300 Write block 4012 len 20480 data: 4 4 4 5 6
15739 244/300 Flush
15740 245/300 Flush
15741 246/300 Read block 2725 len 24576
15742 247/300 Flush
15743 248/300 Read block 104 len 24576
15744 249/300 Read block 3281 len 36864
15745 250/300 Write block 172 len 8192 data: 4 3
15746 251/300 Read block 4600 len 28672
15747 252/300 Read block 955 len 4096
15748 253/300 Read block 4552 len 16384
15749 254/300 Write block 4342 len 40960 data: 3 5 4 4 5 6 5 6 5 5
15750 255/300 Write block 2903 len 24576 data: 5 4 3 2 2 2
15751 256/300 Read block 2464 len 4096
15752 257/300 Write block 2125 len 40960 data: 4 4 4 6 6 6 8 7 6 7
15753 258/300 Read block 2878 len 36864
15754 259/300 Flush
15755 260/300 Read block 2736 len 4096
15756 261/300 Read block 3660 len 32768
15757 262/300 Read block 2471 len 24576
15758 263/300 Write block 595 len 12288 data: 3 3 3
15759 264/300 Write block 255 len 16384 data: 3 3 3 4
15760 265/300 Read block 3274 len 36864
15761 266/300 Write block 230 len 8192 data: 4 5
15762 267/300 Read block 3540 len 28672
15763 268/300 Write block 2265 len 4096 data: 2
15764 269/300 Read block 2893 len 4096
15765 270/300 Read block 3291 len 12288
15766 271/300 Read block 4256 len 20480
15767 272/300 Read block 196 len 40960
15768 273/300 Flush
15769 274/300 Write block 4529 len 4096 data: 4
15770 275/300 Flush
15771 276/300 Read block 1191 len 32768
15772 277/300 Write block 2432 len 8192 data: 4 4
15773 278/300 Read block 3241 len 4096
15774 279/300 Write block 2904 len 28672 data: 5 4 3 3 3 2 2
15775 280/300 Read block 4170 len 28672
15776 281/300 Write block 3824 len 40960 data: 8 7 8 7 7 7 4 4 5 5
15777 282/300 Write block 3467 len 40960 data: 6 6 5 5 6 5 4 5 6 6
15778 283/300 Write block 1723 len 20480 data: 4 4 4 4 5
15779 284/300 Read block 2461 len 28672
15780 285/300 Read block 2903 len 4096
15781 286/300 Read block 782 len 40960
15782 287/300 Write block 3098 len 4096 data: 3
15783 288/300 Flush
15784 289/300 Read block 4736 len 8192
15785 290/300 Write block 283 len 28672 data: 3 2 2 2 3 4 4
15786 291/300 Write block 1226 len 16384 data: 5 4 3 4
15787 292/300 Read block 3932 len 32768
15788 293/300 Read block 2591 len 24576
15789 294/300 Write block 4891 len 4096 data: 4
15790 295/300 Write block 685 len 24576 data: 5 6 6 6 5 4
15791 296/300 Read block 2051 len 40960
15792 297/300 Write block 518 len 16384 data: 5 4 3 3
15793 298/300 Read block 1182 len 28672
15794 299/300 Write block 3451 len 8192 data: 4 4
15795 300/300 Read block 4165 len 8192
15796 301/300 Read block 156 len 20480
15797 Main task has work to do, go find it
15798 got dsc Start(2)
15799 start 2
15800 Dec 09 15:11:11.055 INFO request completed, latency_us: 253, response_code: 204, uri: /start/cid/2, method: POST, req_id: 642ce7be-9372-4263-9446-b90f96252b23, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
15801 Replay: started 2, returned:Ok(())
15802 ----------------------------------------------------------------
15803 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15804 States: Active Active Offline
15805 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15806 5530 Acked Write 10 Done Done Sent false
15807 5531 Acked Flush 0 Done Done Sent false
15808 5532 Acked Read 9 Done Done Sent false
15809 5533 Acked Flush 0 Done Done Sent false
15810 5534 Acked Read 3 Done Done Sent false
15811 5535 Acked Write 8 Done Done Sent false
15812 5536 Acked Read 10 Done Done Sent false
15813 5537 Acked Read 4 Done Done Sent false
15814 5538 Acked Read 3 Done Done Sent false
15815 5539 Acked Read 7 Done Done Sent false
15816 5540 Acked Read 7 Done Done Sent false
15817 5541 Acked Read 6 Done Done Sent false
15818 5542 Acked Read 5 Done Done Sent false
15819 5543 Acked Write 8 Done Done Sent false
15820 5544 Acked Write 7 Done Done Sent false
15821 5545 Acked Read 8 Done Done Sent false
15822 5546 Acked Read 5 Done Done Sent false
15823 5547 Acked Read 4 Done Done Sent false
15824 5548 Acked Write 8 Done Done Sent false
15825 5549 Acked Flush 0 Done Done Sent false
15826 5550 Acked Read 1 Done Done Sent false
15827 5551 Acked Write 7 Done Done Sent false
15828 5552 Acked Read 3 Done Done Sent false
15829 5553 Acked Write 2 Done Done Sent false
15830 5554 Acked Read 8 Done Done Sent false
15831 5555 Acked Flush 0 Done Done Sent false
15832 5556 Acked Read 4 Done Done Sent false
15833 5557 Acked Write 6 Done Done Sent false
15834 5558 Acked Write 3 Done Done Sent false
15835 5559 Acked Read 8 Done Done Sent false
15836 5560 Acked Flush 0 Done Done Sent false
15837 5561 Acked Write 5 Done Done Sent false
15838 5562 Acked Write 5 Done Done Sent false
15839 5563 Acked Read 3 Done Done Sent false
15840 5564 Acked Read 2 Done Done Sent false
15841 5565 Acked Write 6 Done Done Sent false
15842 5566 Acked Read 3 Done Done Sent false
15843 5567 Acked Read 5 Done Done Sent false
15844 5568 Acked Read 3 Done Done Sent false
15845 5569 Acked Read 6 Done Done Sent false
15846 5570 Acked Read 9 Done Done Sent false
15847 5571 Acked Write 5 Done Done Sent false
15848 5572 Acked Write 8 Done Done Sent false
15849 5573 Acked Read 8 Done Done Sent false
15850 5574 Acked Read 10 Done Done Sent false
15851 5575 Acked Write 5 Done Done Sent false
15852 5576 Acked Write 6 Done Done Sent false
15853 5577 Acked Read 5 Done Done Sent false
15854 5578 Acked Read 4 Done Done Sent false
15855 5579 Acked Write 10 Done Done Sent false
15856 5580 Acked Read 1 Done Done Sent false
15857 5581 Acked Write 4 Done Done Sent false
15858 5582 Acked Read 6 Done Done Sent false
15859 5583 Acked Write 5 Done Done Sent false
15860 5584 Acked Write 5 Done Done Sent false
15861 5585 Acked Read 9 Done Done Sent false
15862 5586 Acked Write 9 Done Done Sent false
15863 5587 Acked Read 3 Done Done Sent false
15864 5588 Acked Flush 0 Done Done Sent false
15865 5589 Acked Write 9 Done Done Sent false
15866 5590 Acked Write 9 Done Done Sent false
15867 5591 Acked Read 3 Done Done Sent false
15868 5592 Acked Read 6 Done Done Sent false
15869 5593 Acked Write 1 Done Done Sent false
15870 5594 Acked Write 1 Done Done Sent false
15871 5595 Acked Read 4 Done Done Sent false
15872 5596 Acked Read 1 Done Done Sent false
15873 5597 Acked Read 1 Done Done Sent false
15874 5598 Acked Write 3 Done Done Sent false
15875 5599 Acked Flush 0 Done Done Sent false
15876 5600 Acked Read 3 Done Done Sent false
15877 5601 Acked Write 10 Done Done Sent false
15878 5602 Acked Read 6 Done Done Sent false
15879 5603 Acked Write 6 Done Done Sent false
15880 5604 Acked Read 4 Done Done Sent false
15881 5605 Acked Read 10 Done Done Sent false
15882 5606 Acked Write 6 Done Done Sent false
15883 5607 Acked Read 1 Done Done Sent false
15884 5608 Acked Read 7 Done Done Sent false
15885 5609 Acked Write 3 Done Done Sent false
15886 5610 Acked Write 10 Done Done Sent false
15887 5611 Acked Read 7 Done Done Sent false
15888 5612 Acked Write 1 Done Done Sent false
15889 5613 Acked Write 1 Done Done Sent false
15890 5614 Acked Write 5 Done Done Sent false
15891 5615 Acked Write 5 Done Done Sent false
15892 5616 Acked Write 2 Done Done Sent false
15893 5617 Acked Read 9 Done Done Sent false
15894 5618 Acked Read 1 Done Done Sent false
15895 5619 Acked Write 9 Done Done Sent false
15896 5620 Acked Write 6 Done Done Sent false
15897 5621 Acked Write 8 Done Done Sent false
15898 5622 Acked Write 10 Done Done Sent false
15899 5623 Acked Read 9 Done Done Sent false
15900 5624 Acked Write 10 Done Done Sent false
15901 5625 Acked Read 5 Done Done Sent false
15902 5626 Acked Read 2 Done Done Sent false
15903 5627 Acked Read 9 Done Done Sent false
15904 5628 Acked Flush 0 Done Done Sent false
15905 5629 Acked Read 3 Done Done Sent false
15906 5630 Acked Read 9 Done Done Sent false
15907 5631 Acked Read 6 Done Done Sent false
15908 5632 Acked Read 2 Done Done Sent false
15909 5633 Acked Write 7 Done Done Sent false
15910 5634 Acked Flush 0 Done Done Sent false
15911 5635 Acked Read 8 Done Done Sent false
15912 5636 Acked Read 9 Done Done Sent false
15913 5637 Acked Write 9 Done Done Sent false
15914 5638 Acked Write 5 Done Done Sent false
15915 5639 Acked Write 3 Done Done Sent false
15916 5640 Acked Write 10 Done Done Sent false
15917 5641 Acked Read 10 Done Done Sent false
15918 5642 Acked Read 7 Done Done Sent false
15919 5643 Acked Write 2 Done Done Sent false
15920 5644 Acked Write 7 Done Done Sent false
15921 5645 Acked Write 3 Done Done Sent false
15922 5646 Acked Read 10 Done Done Sent false
15923 5647 Acked Read 7 Done Done Sent false
15924 5648 Acked Read 2 Done Done Sent false
15925 5649 Acked Write 3 Done Done Sent false
15926 5650 Acked Flush 0 Done Done Sent false
15927 5651 Acked Write 9 Done Done Sent false
15928 5652 Acked Write 6 Done Done Sent false
15929 5653 Acked Write 4 Done Done Sent false
15930 5654 Acked Read 9 Done Done Sent false
15931 5655 Acked Write 6 Done Done Sent false
15932 5656 Acked Write 9 Done Done Sent false
15933 5657 Acked Write 10 Done Done Sent false
15934 5658 Acked Write 5 Done Done Sent false
15935 5659 Acked Read 1 Done Done Sent false
15936 5660 Acked Write 9 Done Done Sent false
15937 5661 Acked Write 9 Done Done Sent false
15938 5662 Acked Flush 0 Done Done Sent false
15939 5663 Acked Write 5 Done Done Sent false
15940 5664 Acked Read 9 Done Done Sent false
15941 5665 Acked Write 10 Done Done Sent false
15942 5666 Acked Write 5 Done Done Sent false
15943 5667 Acked Read 3 Done Done Sent false
15944 5668 Acked Read 9 Done Done Sent false
15945 5669 Acked Read 10 Done Done Sent false
15946 5670 Acked Read 2 Done Done Sent false
15947 5671 Acked Write 1 Done Done Sent false
15948 5672 Acked Read 8 Done Done Sent false
15949 5673 Acked Read 5 Done Done Sent false
15950 5674 Acked Read 5 Done Done Sent false
15951 5675 Acked Write 8 Done Done Sent false
15952 5676 Acked Write 8 Done Done Sent false
15953 5677 Acked Read 1 Done Done Sent false
15954 5678 Acked Read 3 Done Done Sent false
15955 5679 Acked Read 9 Done Done Sent false
15956 5680 Acked Read 1 Done Done Sent false
15957 5681 Acked Read 10 Done Done Sent false
15958 5682 Acked Read 5 Done Done Sent false
15959 5683 Acked Write 6 Done Done Sent false
15960 5684 Acked Write 10 Done Done Sent false
15961 5685 Acked Read 8 Done Done Sent false
15962 5686 Acked Read 10 Done Done Sent false
15963 5687 Acked Read 5 Done Done Sent false
15964 5688 Acked Write 6 Done Done Sent false
15965 5689 Acked Read 8 Done Done Sent false
15966 5690 Acked Write 9 Done Done Sent false
15967 5691 Acked Read 10 Done Done Sent false
15968 5692 Acked Read 10 Done Done Sent false
15969 5693 Acked Read 7 Done Done Sent false
15970 5694 Acked Read 9 Done Done Sent false
15971 5695 Acked Read 8 Done Done Sent false
15972 5696 Acked Read 2 Done Done Sent false
15973 5697 Acked Read 1 Done Done Sent false
15974 5698 Acked Write 2 Done Done Sent false
15975 5699 Acked Read 4 Done Done Sent false
15976 5700 Acked Write 7 Done Done Sent false
15977 5701 Acked Read 7 Done Done Sent false
15978 5702 Acked Read 2 Done Done Sent false
15979 5703 Acked Read 1 Done Done Sent false
15980 5704 Acked Write 7 Done Done Sent false
15981 5705 Acked Read 7 Done Done Sent false
15982 5706 Acked Write 1 Done Done Sent false
15983 5707 Acked Write 4 Done Done Sent false
15984 5708 Acked Write 8 Done Done Sent false
15985 5709 Acked Write 7 Done Done Sent false
15986 5710 Acked Write 5 Done Done Sent false
15987 5711 Acked Write 1 Done Done Sent false
15988 5712 Acked Write 10 Done Done Sent false
15989 5713 Acked Read 3 Done Done Sent false
15990 5714 Acked Write 6 Done Done Sent false
15991 5715 Acked Read 10 Done Done Sent false
15992 5716 Acked Read 1 Done Done Sent false
15993 5717 Acked Write 2 Done Done Sent false
15994 5718 Acked Read 9 Done Done Sent false
15995 5719 Acked Write 8 Done Done Sent false
15996 5720 Acked Write 7 Done Done Sent false
15997 5721 Acked Read 7 Done Done Sent false
15998 5722 Acked Flush 0 Done Done Sent false
15999 5723 Acked Read 7 Done Done Sent false
16000 5724 Acked Write 4 Done Done Sent false
16001 5725 Acked Read 3 Done Done Sent false
16002 5726 Acked Write 6 Done Done Sent false
16003 5727 Acked Read 7 Done Done Sent false
16004 5728 Acked Write 9 Done Done Sent false
16005 5729 Acked Read 9 Done Done Sent false
16006 5730 Acked Read 9 Done Done Sent false
16007 5731 Acked Read 3 Done Done Sent false
16008 5732 Acked Write 5 Done Done Sent false
16009 5733 Acked Read 8 Done Done Sent false
16010 5734 Acked Write 7 Done Done Sent false
16011 5735 Acked Read 10 Done Done Sent false
16012 5736 Acked Flush 0 Done Done Sent false
16013 5737 Acked Read 3 Done Done Sent false
16014 5738 Acked Write 4 Done Done Sent false
16015 5739 Acked Flush 0 Done Done Sent false
16016 5740 Acked Read 8 Done Done Sent false
16017 5741 Acked Write 2 Done Done Sent false
16018 5742 Acked Read 9 Done Done Sent false
16019 5743 Acked Read 8 Done Done Sent false
16020 5744 Acked Flush 0 Done Done Sent false
16021 5745 Acked Write 7 Done Done Sent false
16022 5746 Acked Read 2 Done Done Sent false
16023 5747 Acked Read 10 Done Done Sent false
16024 5748 Acked Write 8 Done Done Sent false
16025 5749 Acked Read 6 Done Done Sent false
16026 5750 Acked Write 1 Done Done Sent false
16027 5751 Acked Write 9 Done Done Sent false
16028 5752 Acked Write 2 Done Done Sent false
16029 5753 Acked Write 2 Done Done Sent false
16030 5754 Acked Read 9 Done Done Sent false
16031 5755 Acked Write 9 Done Done Sent false
16032 5756 Acked Read 6 Done Done Sent false
16033 5757 Acked Read 7 Done Done Sent false
16034 5758 Acked Write 5 Done Done Sent false
16035 5759 Acked Write 8 Done Done Sent false
16036 5760 Acked Flush 0 Done Done Sent false
16037 5761 Acked Read 1 Done Done Sent false
16038 5762 Acked Read 3 Done Done Sent false
16039 5763 Acked Read 3 Done Done Sent false
16040 5764 Acked Read 4 Done Done Sent false
16041 5765 Acked Read 1 Done Done Sent false
16042 5766 Acked Write 4 Done Done Sent false
16043 5767 Acked Write 1 Done Done Sent false
16044 5768 Acked Write 2 Done Done Sent false
16045 5769 Acked Flush 0 Done Done Sent false
16046 5770 Acked Write 2 Done Done Sent false
16047 5771 Acked Write 5 Done Done Sent false
16048 5772 Acked Write 5 Done Done Sent false
16049 5773 Acked Flush 0 Done Done Sent false
16050 5774 Acked Flush 0 Done Done Sent false
16051 5775 Acked Read 6 Done Done Sent false
16052 5776 Acked Flush 0 Done Done Sent false
16053 5777 Acked Read 6 Done Done Sent false
16054 5778 Acked Read 9 Done Done Sent false
16055 5779 Acked Write 2 Done Done Sent false
16056 5780 Acked Read 7 Done Done Sent false
16057 5781 Acked Read 1 Done Done Sent false
16058 5782 Acked Read 4 Done Done Sent false
16059 5783 Acked Write 10 Done Done Sent false
16060 5784 Acked Write 6 Done Done Sent false
16061 5785 Acked Read 1 Done Done Sent false
16062 5786 Acked Write 10 Done Done Sent false
16063 5787 Acked Read 9 Done Done Sent false
16064 5788 Acked Flush 0 Done Done Sent false
16065 5789 Acked Read 1 Done Done Sent false
16066 5790 Acked Read 8 Done Done Sent false
16067 5791 Acked Read 6 Done Done Sent false
16068 5792 Acked Write 3 Done Done Sent false
16069 5793 Acked Write 4 Done Done Sent false
16070 5794 Acked Read 9 Done Done Sent false
16071 5795 Acked Write 2 Done Done Sent false
16072 5796 Acked Read 7 Done Done Sent false
16073 5797 Acked Write 1 Done Done Sent false
16074 5798 Acked Read 1 Done Done Sent false
16075 5799 Acked Read 3 Done Done Sent false
16076 5800 Acked Read 5 Done Done Sent false
16077 5801 Acked Read 10 Done Done Sent false
16078 5802 Acked Flush 0 Done Done Sent false
16079 5803 Acked Write 1 Done Done Sent false
16080 5804 Acked Flush 0 Done Done Sent false
16081 5805 Acked Read 8 Done Done Sent false
16082 5806 Acked Write 2 Done Done Sent false
16083 5807 Acked Read 1 Done Done Sent false
16084 5808 Acked Write 7 Done Done Sent false
16085 5809 Acked Read 7 Done Done Sent false
16086 5810 Acked Write 10 Done Done Sent false
16087 5811 Acked Write 10 Done Done Sent false
16088 5812 Acked Write 5 Done Done Sent false
16089 5813 Acked Read 7 Done Done Sent false
16090 5814 Acked Read 1 Done Done Sent false
16091 5815 Acked Read 10 Done Done Sent false
16092 5816 Acked Write 1 Done Done Sent false
16093 5817 Acked Flush 0 Done Done Sent false
16094 5818 Acked Read 2 Done Done Sent false
16095 5819 Acked Write 7 Done Done Sent false
16096 5820 Acked Write 4 Done Done Sent false
16097 5821 Acked Read 8 Done Done Sent false
16098 5822 Acked Read 6 Done Done Sent false
16099 5823 Acked Write 1 Done Done Sent false
16100 5824 Acked Write 6 Done Done Sent false
16101 5825 Acked Read 10 Done Done Sent false
16102 5826 Acked Write 4 Done Done Sent false
16103 5827 Acked Read 7 Done Done Sent false
16104 5828 Acked Write 2 Done Done Sent false
16105 5829 Acked Read 2 Done Done Sent false
16106 5830 Acked Read 5 Done Done Sent false
16107 STATES DS:0 DS:1 DS:2 TOTAL
16108 Sent 0 0 301 301
16109 Done 301 301 0 602
16110 Skipped 0 0 0 0
16111 Error 0 0 0 0
16112 Last Flush: 5817 5817 5529
16113 Downstairs last five completed: 5529 5528 5527 5526 5525
16114 Upstairs last five completed: 5830 5829 5828 5827 5826
16115 CLIENT: Up:0 ds:301 act:2
16116 ----------------------------------------------------------------
16117 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
16118 States: Active Active Offline
16119 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16120 5530 Acked Write 10 Done Done Sent false
16121 5531 Acked Flush 0 Done Done Sent false
16122 5532 Acked Read 9 Done Done Sent false
16123 5533 Acked Flush 0 Done Done Sent false
16124 5534 Acked Read 3 Done Done Sent false
16125 5535 Acked Write 8 Done Done Sent false
16126 5536 Acked Read 10 Done Done Sent false
16127 5537 Acked Read 4 Done Done Sent false
16128 5538 Acked Read 3 Done Done Sent false
16129 5539 Acked Read 7 Done Done Sent false
16130 5540 Acked Read 7 Done Done Sent false
16131 5541 Acked Read 6 Done Done Sent false
16132 5542 Acked Read 5 Done Done Sent false
16133 5543 Acked Write 8 Done Done Sent false
16134 5544 Acked Write 7 Done Done Sent false
16135 5545 Acked Read 8 Done Done Sent false
16136 5546 Acked Read 5 Done Done Sent false
16137 5547 Acked Read 4 Done Done Sent false
16138 5548 Acked Write 8 Done Done Sent false
16139 5549 Acked Flush 0 Done Done Sent false
16140 5550 Acked Read 1 Done Done Sent false
16141 5551 Acked Write 7 Done Done Sent false
16142 5552 Acked Read 3 Done Done Sent false
16143 5553 Acked Write 2 Done Done Sent false
16144 5554 Acked Read 8 Done Done Sent false
16145 5555 Acked Flush 0 Done Done Sent false
16146 5556 Acked Read 4 Done Done Sent false
16147 5557 Acked Write 6 Done Done Sent false
16148 5558 Acked Write 3 Done Done Sent false
16149 5559 Acked Read 8 Done Done Sent false
16150 5560 Acked Flush 0 Done Done Sent false
16151 5561 Acked Write 5 Done Done Sent false
16152 5562 Acked Write 5 Done Done Sent false
16153 5563 Acked Read 3 Done Done Sent false
16154 5564 Acked Read 2 Done Done Sent false
16155 5565 Acked Write 6 Done Done Sent false
16156 5566 Acked Read 3 Done Done Sent false
16157 5567 Acked Read 5 Done Done Sent false
16158 5568 Acked Read 3 Done Done Sent false
16159 5569 Acked Read 6 Done Done Sent false
16160 5570 Acked Read 9 Done Done Sent false
16161 5571 Acked Write 5 Done Done Sent false
16162 5572 Acked Write 8 Done Done Sent false
16163 5573 Acked Read 8 Done Done Sent false
16164 5574 Acked Read 10 Done Done Sent false
16165 5575 Acked Write 5 Done Done Sent false
16166 5576 Acked Write 6 Done Done Sent false
16167 5577 Acked Read 5 Done Done Sent false
16168 5578 Acked Read 4 Done Done Sent false
16169 5579 Acked Write 10 Done Done Sent false
16170 5580 Acked Read 1 Done Done Sent false
16171 5581 Acked Write 4 Done Done Sent false
16172 5582 Acked Read 6 Done Done Sent false
16173 5583 Acked Write 5 Done Done Sent false
16174 5584 Acked Write 5 Done Done Sent false
16175 5585 Acked Read 9 Done Done Sent false
16176 5586 Acked Write 9 Done Done Sent false
16177 5587 Acked Read 3 Done Done Sent false
16178 5588 Acked Flush 0 Done Done Sent false
16179 5589 Acked Write 9 Done Done Sent false
16180 5590 Acked Write 9 Done Done Sent false
16181 5591 Acked Read 3 Done Done Sent false
16182 5592 Acked Read 6 Done Done Sent false
16183 5593 Acked Write 1 Done Done Sent false
16184 5594 Acked Write 1 Done Done Sent false
16185 5595 Acked Read 4 Done Done Sent false
16186 5596 Acked Read 1 Done Done Sent false
16187 5597 Acked Read 1 Done Done Sent false
16188 5598 Acked Write 3 Done Done Sent false
16189 5599 Acked Flush 0 Done Done Sent false
16190 5600 Acked Read 3 Done Done Sent false
16191 5601 Acked Write 10 Done Done Sent false
16192 5602 Acked Read 6 Done Done Sent false
16193 5603 Acked Write 6 Done Done Sent false
16194 5604 Acked Read 4 Done Done Sent false
16195 5605 Acked Read 10 Done Done Sent false
16196 5606 Acked Write 6 Done Done Sent false
16197 5607 Acked Read 1 Done Done Sent false
16198 5608 Acked Read 7 Done Done Sent false
16199 5609 Acked Write 3 Done Done Sent false
16200 5610 Acked Write 10 Done Done Sent false
16201 5611 Acked Read 7 Done Done Sent false
16202 5612 Acked Write 1 Done Done Sent false
16203 5613 Acked Write 1 Done Done Sent false
16204 5614 Acked Write 5 Done Done Sent false
16205 5615 Acked Write 5 Done Done Sent false
16206 5616 Acked Write 2 Done Done Sent false
16207 5617 Acked Read 9 Done Done Sent false
16208 5618 Acked Read 1 Done Done Sent false
16209 5619 Acked Write 9 Done Done Sent false
16210 5620 Acked Write 6 Done Done Sent false
16211 5621 Acked Write 8 Done Done Sent false
16212 5622 Acked Write 10 Done Done Sent false
16213 5623 Acked Read 9 Done Done Sent false
16214 5624 Acked Write 10 Done Done Sent false
16215 5625 Acked Read 5 Done Done Sent false
16216 5626 Acked Read 2 Done Done Sent false
16217 5627 Acked Read 9 Done Done Sent false
16218 5628 Acked Flush 0 Done Done Sent false
16219 5629 Acked Read 3 Done Done Sent false
16220 5630 Acked Read 9 Done Done Sent false
16221 5631 Acked Read 6 Done Done Sent false
16222 5632 Acked Read 2 Done Done Sent false
16223 5633 Acked Write 7 Done Done Sent false
16224 5634 Acked Flush 0 Done Done Sent false
16225 5635 Acked Read 8 Done Done Sent false
16226 5636 Acked Read 9 Done Done Sent false
16227 5637 Acked Write 9 Done Done Sent false
16228 5638 Acked Write 5 Done Done Sent false
16229 5639 Acked Write 3 Done Done Sent false
16230 5640 Acked Write 10 Done Done Sent false
16231 5641 Acked Read 10 Done Done Sent false
16232 5642 Acked Read 7 Done Done Sent false
16233 5643 Acked Write 2 Done Done Sent false
16234 5644 Acked Write 7 Done Done Sent false
16235 5645 Acked Write 3 Done Done Sent false
16236 5646 Acked Read 10 Done Done Sent false
16237 5647 Acked Read 7 Done Done Sent false
16238 5648 Acked Read 2 Done Done Sent false
16239 5649 Acked Write 3 Done Done Sent false
16240 5650 Acked Flush 0 Done Done Sent false
16241 5651 Acked Write 9 Done Done Sent false
16242 5652 Acked Write 6 Done Done Sent false
16243 5653 Acked Write 4 Done Done Sent false
16244 5654 Acked Read 9 Done Done Sent false
16245 5655 Acked Write 6 Done Done Sent false
16246 5656 Acked Write 9 Done Done Sent false
16247 5657 Acked Write 10 Done Done Sent false
16248 5658 Acked Write 5 Done Done Sent false
16249 5659 Acked Read 1 Done Done Sent false
16250 5660 Acked Write 9 Done Done Sent false
16251 5661 Acked Write 9 Done Done Sent false
16252 5662 Acked Flush 0 Done Done Sent false
16253 5663 Acked Write 5 Done Done Sent false
16254 5664 Acked Read 9 Done Done Sent false
16255 5665 Acked Write 10 Done Done Sent false
16256 5666 Acked Write 5 Done Done Sent false
16257 5667 Acked Read 3 Done Done Sent false
16258 5668 Acked Read 9 Done Done Sent false
16259 5669 Acked Read 10 Done Done Sent false
16260 5670 Acked Read 2 Done Done Sent false
16261 5671 Acked Write 1 Done Done Sent false
16262 5672 Acked Read 8 Done Done Sent false
16263 5673 Acked Read 5 Done Done Sent false
16264 5674 Acked Read 5 Done Done Sent false
16265 5675 Acked Write 8 Done Done Sent false
16266 5676 Acked Write 8 Done Done Sent false
16267 5677 Acked Read 1 Done Done Sent false
16268 5678 Acked Read 3 Done Done Sent false
16269 5679 Acked Read 9 Done Done Sent false
16270 5680 Acked Read 1 Done Done Sent false
16271 5681 Acked Read 10 Done Done Sent false
16272 5682 Acked Read 5 Done Done Sent false
16273 5683 Acked Write 6 Done Done Sent false
16274 5684 Acked Write 10 Done Done Sent false
16275 5685 Acked Read 8 Done Done Sent false
16276 5686 Acked Read 10 Done Done Sent false
16277 5687 Acked Read 5 Done Done Sent false
16278 5688 Acked Write 6 Done Done Sent false
16279 5689 Acked Read 8 Done Done Sent false
16280 5690 Acked Write 9 Done Done Sent false
16281 5691 Acked Read 10 Done Done Sent false
16282 5692 Acked Read 10 Done Done Sent false
16283 5693 Acked Read 7 Done Done Sent false
16284 5694 Acked Read 9 Done Done Sent false
16285 5695 Acked Read 8 Done Done Sent false
16286 5696 Acked Read 2 Done Done Sent false
16287 5697 Acked Read 1 Done Done Sent false
16288 5698 Acked Write 2 Done Done Sent false
16289 5699 Acked Read 4 Done Done Sent false
16290 5700 Acked Write 7 Done Done Sent false
16291 5701 Acked Read 7 Done Done Sent false
16292 5702 Acked Read 2 Done Done Sent false
16293 5703 Acked Read 1 Done Done Sent false
16294 5704 Acked Write 7 Done Done Sent false
16295 5705 Acked Read 7 Done Done Sent false
16296 5706 Acked Write 1 Done Done Sent false
16297 5707 Acked Write 4 Done Done Sent false
16298 5708 Acked Write 8 Done Done Sent false
16299 5709 Acked Write 7 Done Done Sent false
16300 5710 Acked Write 5 Done Done Sent false
16301 5711 Acked Write 1 Done Done Sent false
16302 5712 Acked Write 10 Done Done Sent false
16303 5713 Acked Read 3 Done Done Sent false
16304 5714 Acked Write 6 Done Done Sent false
16305 5715 Acked Read 10 Done Done Sent false
16306 5716 Acked Read 1 Done Done Sent false
16307 5717 Acked Write 2 Done Done Sent false
16308 5718 Acked Read 9 Done Done Sent false
16309 5719 Acked Write 8 Done Done Sent false
16310 5720 Acked Write 7 Done Done Sent false
16311 5721 Acked Read 7 Done Done Sent false
16312 5722 Acked Flush 0 Done Done Sent false
16313 5723 Acked Read 7 Done Done Sent false
16314 5724 Acked Write 4 Done Done Sent false
16315 5725 Acked Read 3 Done Done Sent false
16316 5726 Acked Write 6 Done Done Sent false
16317 5727 Acked Read 7 Done Done Sent false
16318 5728 Acked Write 9 Done Done Sent false
16319 5729 Acked Read 9 Done Done Sent false
16320 5730 Acked Read 9 Done Done Sent false
16321 5731 Acked Read 3 Done Done Sent false
16322 5732 Acked Write 5 Done Done Sent false
16323 5733 Acked Read 8 Done Done Sent false
16324 5734 Acked Write 7 Done Done Sent false
16325 5735 Acked Read 10 Done Done Sent false
16326 5736 Acked Flush 0 Done Done Sent false
16327 5737 Acked Read 3 Done Done Sent false
16328 5738 Acked Write 4 Done Done Sent false
16329 5739 Acked Flush 0 Done Done Sent false
16330 5740 Acked Read 8 Done Done Sent false
16331 5741 Acked Write 2 Done Done Sent false
16332 5742 Acked Read 9 Done Done Sent false
16333 5743 Acked Read 8 Done Done Sent false
16334 5744 Acked Flush 0 Done Done Sent false
16335 5745 Acked Write 7 Done Done Sent false
16336 5746 Acked Read 2 Done Done Sent false
16337 5747 Acked Read 10 Done Done Sent false
16338 5748 Acked Write 8 Done Done Sent false
16339 5749 Acked Read 6 Done Done Sent false
16340 5750 Acked Write 1 Done Done Sent false
16341 5751 Acked Write 9 Done Done Sent false
16342 5752 Acked Write 2 Done Done Sent false
16343 5753 Acked Write 2 Done Done Sent false
16344 5754 Acked Read 9 Done Done Sent false
16345 5755 Acked Write 9 Done Done Sent false
16346 5756 Acked Read 6 Done Done Sent false
16347 5757 Acked Read 7 Done Done Sent false
16348 5758 Acked Write 5 Done Done Sent false
16349 5759 Acked Write 8 Done Done Sent false
16350 5760 Acked Flush 0 Done Done Sent false
16351 5761 Acked Read 1 Done Done Sent false
16352 5762 Acked Read 3 Done Done Sent false
16353 5763 Acked Read 3 Done Done Sent false
16354 5764 Acked Read 4 Done Done Sent false
16355 5765 Acked Read 1 Done Done Sent false
16356 5766 Acked Write 4 Done Done Sent false
16357 5767 Acked Write 1 Done Done Sent false
16358 5768 Acked Write 2 Done Done Sent false
16359 5769 Acked Flush 0 Done Done Sent false
16360 5770 Acked Write 2 Done Done Sent false
16361 5771 Acked Write 5 Done Done Sent false
16362 5772 Acked Write 5 Done Done Sent false
16363 5773 Acked Flush 0 Done Done Sent false
16364 5774 Acked Flush 0 Done Done Sent false
16365 5775 Acked Read 6 Done Done Sent false
16366 5776 Acked Flush 0 Done Done Sent false
16367 5777 Acked Read 6 Done Done Sent false
16368 5778 Acked Read 9 Done Done Sent false
16369 5779 Acked Write 2 Done Done Sent false
16370 5780 Acked Read 7 Done Done Sent false
16371 5781 Acked Read 1 Done Done Sent false
16372 5782 Acked Read 4 Done Done Sent false
16373 5783 Acked Write 10 Done Done Sent false
16374 5784 Acked Write 6 Done Done Sent false
16375 5785 Acked Read 1 Done Done Sent false
16376 5786 Acked Write 10 Done Done Sent false
16377 5787 Acked Read 9 Done Done Sent false
16378 5788 Acked Flush 0 Done Done Sent false
16379 5789 Acked Read 1 Done Done Sent false
16380 5790 Acked Read 8 Done Done Sent false
16381 5791 Acked Read 6 Done Done Sent false
16382 5792 Acked Write 3 Done Done Sent false
16383 5793 Acked Write 4 Done Done Sent false
16384 5794 Acked Read 9 Done Done Sent false
16385 5795 Acked Write 2 Done Done Sent false
16386 5796 Acked Read 7 Done Done Sent false
16387 5797 Acked Write 1 Done Done Sent false
16388 5798 Acked Read 1 Done Done Sent false
16389 5799 Acked Read 3 Done Done Sent false
16390 5800 Acked Read 5 Done Done Sent false
16391 5801 Acked Read 10 Done Done Sent false
16392 5802 Acked Flush 0 Done Done Sent false
16393 5803 Acked Write 1 Done Done Sent false
16394 5804 Acked Flush 0 Done Done Sent false
16395 5805 Acked Read 8 Done Done Sent false
16396 5806 Acked Write 2 Done Done Sent false
16397 5807 Acked Read 1 Done Done Sent false
16398 5808 Acked Write 7 Done Done Sent false
16399 5809 Acked Read 7 Done Done Sent false
16400 5810 Acked Write 10 Done Done Sent false
16401 5811 Acked Write 10 Done Done Sent false
16402 5812 Acked Write 5 Done Done Sent false
16403 5813 Acked Read 7 Done Done Sent false
16404 5814 Acked Read 1 Done Done Sent false
16405 5815 Acked Read 10 Done Done Sent false
16406 5816 Acked Write 1 Done Done Sent false
16407 5817 Acked Flush 0 Done Done Sent false
16408 5818 Acked Read 2 Done Done Sent false
16409 5819 Acked Write 7 Done Done Sent false
16410 5820 Acked Write 4 Done Done Sent false
16411 5821 Acked Read 8 Done Done Sent false
16412 5822 Acked Read 6 Done Done Sent false
16413 5823 Acked Write 1 Done Done Sent false
16414 5824 Acked Write 6 Done Done Sent false
16415 5825 Acked Read 10 Done Done Sent false
16416 5826 Acked Write 4 Done Done Sent false
16417 5827 Acked Read 7 Done Done Sent false
16418 5828 Acked Write 2 Done Done Sent false
16419 5829 Acked Read 2 Done Done Sent false
16420 5830 Acked Read 5 Done Done Sent false
16421 5831 Acked Flush 0 Done Done Sent false
16422 STATES DS:0 DS:1 DS:2 TOTAL
16423 Sent 0 0 302 302
16424 Done 302 302 0 604
16425 Skipped 0 0 0 0
16426 Error 0 0 0 0
16427 Last Flush: 5831 5831 5529
16428 Downstairs last five completed: 5529 5528 5527 5526 5525
16429 Upstairs last five completed: 5831 5830 5829 5828 5827
16430 CLIENT: Up:0 ds:302 act:2
164312024-12-09T15:11:16.809ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
164322024-12-09T15:11:16.809ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
164332024-12-09T15:11:16.809ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
164342024-12-09T15:11:16.809ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
164352024-12-09T15:11:16.809ZINFOcrucible: client task is exiting = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
16436 [8830] got start action
16437 [8830] I am going to restart
16438 Starting downstairs at port 8830
16439 Make output file at "/tmp/dsc/downstairs-8830.txt"
16440 [8830][2] reports Starting
16441 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1217)
16442 [8830][2] reports Running
16443 ----------------------------------------------------------------
16444 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
16445 States: Active Active Offline
16446 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16447 5530 Acked Write 10 Done Done Sent false
16448 5531 Acked Flush 0 Done Done Sent false
16449 5532 Acked Read 9 Done Done Sent false
16450 5533 Acked Flush 0 Done Done Sent false
16451 5534 Acked Read 3 Done Done Sent false
16452 5535 Acked Write 8 Done Done Sent false
16453 5536 Acked Read 10 Done Done Sent false
16454 5537 Acked Read 4 Done Done Sent false
16455 5538 Acked Read 3 Done Done Sent false
16456 5539 Acked Read 7 Done Done Sent false
16457 5540 Acked Read 7 Done Done Sent false
16458 5541 Acked Read 6 Done Done Sent false
16459 5542 Acked Read 5 Done Done Sent false
16460 5543 Acked Write 8 Done Done Sent false
16461 5544 Acked Write 7 Done Done Sent false
16462 5545 Acked Read 8 Done Done Sent false
16463 5546 Acked Read 5 Done Done Sent false
16464 5547 Acked Read 4 Done Done Sent false
16465 5548 Acked Write 8 Done Done Sent false
16466 5549 Acked Flush 0 Done Done Sent false
16467 5550 Acked Read 1 Done Done Sent false
16468 5551 Acked Write 7 Done Done Sent false
16469 5552 Acked Read 3 Done Done Sent false
16470 5553 Acked Write 2 Done Done Sent false
16471 5554 Acked Read 8 Done Done Sent false
16472 5555 Acked Flush 0 Done Done Sent false
16473 5556 Acked Read 4 Done Done Sent false
16474 5557 Acked Write 6 Done Done Sent false
16475 5558 Acked Write 3 Done Done Sent false
16476 5559 Acked Read 8 Done Done Sent false
16477 5560 Acked Flush 0 Done Done Sent false
16478 5561 Acked Write 5 Done Done Sent false
16479 5562 Acked Write 5 Done Done Sent false
16480 5563 Acked Read 3 Done Done Sent false
16481 5564 Acked Read 2 Done Done Sent false
16482 5565 Acked Write 6 Done Done Sent false
16483 5566 Acked Read 3 Done Done Sent false
16484 5567 Acked Read 5 Done Done Sent false
16485 5568 Acked Read 3 Done Done Sent false
16486 5569 Acked Read 6 Done Done Sent false
16487 5570 Acked Read 9 Done Done Sent false
16488 5571 Acked Write 5 Done Done Sent false
16489 5572 Acked Write 8 Done Done Sent false
16490 5573 Acked Read 8 Done Done Sent false
16491 5574 Acked Read 10 Done Done Sent false
16492 5575 Acked Write 5 Done Done Sent false
16493 5576 Acked Write 6 Done Done Sent false
16494 5577 Acked Read 5 Done Done Sent false
16495 5578 Acked Read 4 Done Done Sent false
16496 5579 Acked Write 10 Done Done Sent false
16497 5580 Acked Read 1 Done Done Sent false
16498 5581 Acked Write 4 Done Done Sent false
16499 5582 Acked Read 6 Done Done Sent false
16500 5583 Acked Write 5 Done Done Sent false
16501 5584 Acked Write 5 Done Done Sent false
16502 5585 Acked Read 9 Done Done Sent false
16503 5586 Acked Write 9 Done Done Sent false
16504 5587 Acked Read 3 Done Done Sent false
16505 5588 Acked Flush 0 Done Done Sent false
16506 5589 Acked Write 9 Done Done Sent false
16507 5590 Acked Write 9 Done Done Sent false
16508 5591 Acked Read 3 Done Done Sent false
16509 5592 Acked Read 6 Done Done Sent false
16510 5593 Acked Write 1 Done Done Sent false
16511 5594 Acked Write 1 Done Done Sent false
16512 5595 Acked Read 4 Done Done Sent false
16513 5596 Acked Read 1 Done Done Sent false
16514 5597 Acked Read 1 Done Done Sent false
16515 5598 Acked Write 3 Done Done Sent false
16516 5599 Acked Flush 0 Done Done Sent false
16517 5600 Acked Read 3 Done Done Sent false
16518 5601 Acked Write 10 Done Done Sent false
16519 5602 Acked Read 6 Done Done Sent false
16520 5603 Acked Write 6 Done Done Sent false
16521 5604 Acked Read 4 Done Done Sent false
16522 5605 Acked Read 10 Done Done Sent false
16523 5606 Acked Write 6 Done Done Sent false
16524 5607 Acked Read 1 Done Done Sent false
16525 5608 Acked Read 7 Done Done Sent false
16526 5609 Acked Write 3 Done Done Sent false
16527 5610 Acked Write 10 Done Done Sent false
16528 5611 Acked Read 7 Done Done Sent false
16529 5612 Acked Write 1 Done Done Sent false
16530 5613 Acked Write 1 Done Done Sent false
16531 5614 Acked Write 5 Done Done Sent false
16532 5615 Acked Write 5 Done Done Sent false
16533 5616 Acked Write 2 Done Done Sent false
16534 5617 Acked Read 9 Done Done Sent false
16535 5618 Acked Read 1 Done Done Sent false
16536 5619 Acked Write 9 Done Done Sent false
16537 5620 Acked Write 6 Done Done Sent false
16538 5621 Acked Write 8 Done Done Sent false
16539 5622 Acked Write 10 Done Done Sent false
16540 5623 Acked Read 9 Done Done Sent false
16541 5624 Acked Write 10 Done Done Sent false
16542 5625 Acked Read 5 Done Done Sent false
16543 5626 Acked Read 2 Done Done Sent false
16544 5627 Acked Read 9 Done Done Sent false
16545 5628 Acked Flush 0 Done Done Sent false
16546 5629 Acked Read 3 Done Done Sent false
16547 5630 Acked Read 9 Done Done Sent false
16548 5631 Acked Read 6 Done Done Sent false
16549 5632 Acked Read 2 Done Done Sent false
16550 5633 Acked Write 7 Done Done Sent false
16551 5634 Acked Flush 0 Done Done Sent false
16552 5635 Acked Read 8 Done Done Sent false
16553 5636 Acked Read 9 Done Done Sent false
16554 5637 Acked Write 9 Done Done Sent false
16555 5638 Acked Write 5 Done Done Sent false
16556 5639 Acked Write 3 Done Done Sent false
16557 5640 Acked Write 10 Done Done Sent false
16558 5641 Acked Read 10 Done Done Sent false
16559 5642 Acked Read 7 Done Done Sent false
16560 5643 Acked Write 2 Done Done Sent false
16561 5644 Acked Write 7 Done Done Sent false
16562 5645 Acked Write 3 Done Done Sent false
16563 5646 Acked Read 10 Done Done Sent false
16564 5647 Acked Read 7 Done Done Sent false
16565 5648 Acked Read 2 Done Done Sent false
16566 5649 Acked Write 3 Done Done Sent false
16567 5650 Acked Flush 0 Done Done Sent false
16568 5651 Acked Write 9 Done Done Sent false
16569 5652 Acked Write 6 Done Done Sent false
16570 5653 Acked Write 4 Done Done Sent false
16571 5654 Acked Read 9 Done Done Sent false
16572 5655 Acked Write 6 Done Done Sent false
16573 5656 Acked Write 9 Done Done Sent false
16574 5657 Acked Write 10 Done Done Sent false
16575 5658 Acked Write 5 Done Done Sent false
16576 5659 Acked Read 1 Done Done Sent false
16577 5660 Acked Write 9 Done Done Sent false
16578 5661 Acked Write 9 Done Done Sent false
16579 5662 Acked Flush 0 Done Done Sent false
16580 5663 Acked Write 5 Done Done Sent false
16581 5664 Acked Read 9 Done Done Sent false
16582 5665 Acked Write 10 Done Done Sent false
16583 5666 Acked Write 5 Done Done Sent false
16584 5667 Acked Read 3 Done Done Sent false
16585 5668 Acked Read 9 Done Done Sent false
16586 5669 Acked Read 10 Done Done Sent false
16587 5670 Acked Read 2 Done Done Sent false
16588 5671 Acked Write 1 Done Done Sent false
16589 5672 Acked Read 8 Done Done Sent false
16590 5673 Acked Read 5 Done Done Sent false
16591 5674 Acked Read 5 Done Done Sent false
16592 5675 Acked Write 8 Done Done Sent false
16593 5676 Acked Write 8 Done Done Sent false
16594 5677 Acked Read 1 Done Done Sent false
16595 5678 Acked Read 3 Done Done Sent false
16596 5679 Acked Read 9 Done Done Sent false
16597 5680 Acked Read 1 Done Done Sent false
16598 5681 Acked Read 10 Done Done Sent false
16599 5682 Acked Read 5 Done Done Sent false
16600 5683 Acked Write 6 Done Done Sent false
16601 5684 Acked Write 10 Done Done Sent false
16602 5685 Acked Read 8 Done Done Sent false
16603 5686 Acked Read 10 Done Done Sent false
16604 5687 Acked Read 5 Done Done Sent false
16605 5688 Acked Write 6 Done Done Sent false
16606 5689 Acked Read 8 Done Done Sent false
16607 5690 Acked Write 9 Done Done Sent false
16608 5691 Acked Read 10 Done Done Sent false
16609 5692 Acked Read 10 Done Done Sent false
16610 5693 Acked Read 7 Done Done Sent false
16611 5694 Acked Read 9 Done Done Sent false
16612 5695 Acked Read 8 Done Done Sent false
16613 5696 Acked Read 2 Done Done Sent false
16614 5697 Acked Read 1 Done Done Sent false
16615 5698 Acked Write 2 Done Done Sent false
16616 5699 Acked Read 4 Done Done Sent false
16617 5700 Acked Write 7 Done Done Sent false
16618 5701 Acked Read 7 Done Done Sent false
16619 5702 Acked Read 2 Done Done Sent false
16620 5703 Acked Read 1 Done Done Sent false
16621 5704 Acked Write 7 Done Done Sent false
16622 5705 Acked Read 7 Done Done Sent false
16623 5706 Acked Write 1 Done Done Sent false
16624 5707 Acked Write 4 Done Done Sent false
16625 5708 Acked Write 8 Done Done Sent false
16626 5709 Acked Write 7 Done Done Sent false
16627 5710 Acked Write 5 Done Done Sent false
16628 5711 Acked Write 1 Done Done Sent false
16629 5712 Acked Write 10 Done Done Sent false
16630 5713 Acked Read 3 Done Done Sent false
16631 5714 Acked Write 6 Done Done Sent false
16632 5715 Acked Read 10 Done Done Sent false
16633 5716 Acked Read 1 Done Done Sent false
16634 5717 Acked Write 2 Done Done Sent false
16635 5718 Acked Read 9 Done Done Sent false
16636 5719 Acked Write 8 Done Done Sent false
16637 5720 Acked Write 7 Done Done Sent false
16638 5721 Acked Read 7 Done Done Sent false
16639 5722 Acked Flush 0 Done Done Sent false
16640 5723 Acked Read 7 Done Done Sent false
16641 5724 Acked Write 4 Done Done Sent false
16642 5725 Acked Read 3 Done Done Sent false
16643 5726 Acked Write 6 Done Done Sent false
16644 5727 Acked Read 7 Done Done Sent false
16645 5728 Acked Write 9 Done Done Sent false
16646 5729 Acked Read 9 Done Done Sent false
16647 5730 Acked Read 9 Done Done Sent false
16648 5731 Acked Read 3 Done Done Sent false
16649 5732 Acked Write 5 Done Done Sent false
16650 5733 Acked Read 8 Done Done Sent false
16651 5734 Acked Write 7 Done Done Sent false
16652 5735 Acked Read 10 Done Done Sent false
16653 5736 Acked Flush 0 Done Done Sent false
16654 5737 Acked Read 3 Done Done Sent false
16655 5738 Acked Write 4 Done Done Sent false
16656 5739 Acked Flush 0 Done Done Sent false
16657 5740 Acked Read 8 Done Done Sent false
16658 5741 Acked Write 2 Done Done Sent false
16659 5742 Acked Read 9 Done Done Sent false
16660 5743 Acked Read 8 Done Done Sent false
16661 5744 Acked Flush 0 Done Done Sent false
16662 5745 Acked Write 7 Done Done Sent false
16663 5746 Acked Read 2 Done Done Sent false
16664 5747 Acked Read 10 Done Done Sent false
16665 5748 Acked Write 8 Done Done Sent false
16666 5749 Acked Read 6 Done Done Sent false
16667 5750 Acked Write 1 Done Done Sent false
16668 5751 Acked Write 9 Done Done Sent false
16669 5752 Acked Write 2 Done Done Sent false
16670 5753 Acked Write 2 Done Done Sent false
16671 5754 Acked Read 9 Done Done Sent false
16672 5755 Acked Write 9 Done Done Sent false
16673 5756 Acked Read 6 Done Done Sent false
16674 5757 Acked Read 7 Done Done Sent false
16675 5758 Acked Write 5 Done Done Sent false
16676 5759 Acked Write 8 Done Done Sent false
16677 5760 Acked Flush 0 Done Done Sent false
16678 5761 Acked Read 1 Done Done Sent false
16679 5762 Acked Read 3 Done Done Sent false
16680 5763 Acked Read 3 Done Done Sent false
16681 5764 Acked Read 4 Done Done Sent false
16682 5765 Acked Read 1 Done Done Sent false
16683 5766 Acked Write 4 Done Done Sent false
16684 5767 Acked Write 1 Done Done Sent false
16685 5768 Acked Write 2 Done Done Sent false
16686 5769 Acked Flush 0 Done Done Sent false
16687 5770 Acked Write 2 Done Done Sent false
16688 5771 Acked Write 5 Done Done Sent false
16689 5772 Acked Write 5 Done Done Sent false
16690 5773 Acked Flush 0 Done Done Sent false
16691 5774 Acked Flush 0 Done Done Sent false
16692 5775 Acked Read 6 Done Done Sent false
16693 5776 Acked Flush 0 Done Done Sent false
16694 5777 Acked Read 6 Done Done Sent false
16695 5778 Acked Read 9 Done Done Sent false
16696 5779 Acked Write 2 Done Done Sent false
16697 5780 Acked Read 7 Done Done Sent false
16698 5781 Acked Read 1 Done Done Sent false
16699 5782 Acked Read 4 Done Done Sent false
16700 5783 Acked Write 10 Done Done Sent false
16701 5784 Acked Write 6 Done Done Sent false
16702 5785 Acked Read 1 Done Done Sent false
16703 5786 Acked Write 10 Done Done Sent false
16704 5787 Acked Read 9 Done Done Sent false
16705 5788 Acked Flush 0 Done Done Sent false
16706 5789 Acked Read 1 Done Done Sent false
16707 5790 Acked Read 8 Done Done Sent false
16708 5791 Acked Read 6 Done Done Sent false
16709 5792 Acked Write 3 Done Done Sent false
16710 5793 Acked Write 4 Done Done Sent false
16711 5794 Acked Read 9 Done Done Sent false
16712 5795 Acked Write 2 Done Done Sent false
16713 5796 Acked Read 7 Done Done Sent false
16714 5797 Acked Write 1 Done Done Sent false
16715 5798 Acked Read 1 Done Done Sent false
16716 5799 Acked Read 3 Done Done Sent false
16717 5800 Acked Read 5 Done Done Sent false
16718 5801 Acked Read 10 Done Done Sent false
16719 5802 Acked Flush 0 Done Done Sent false
16720 5803 Acked Write 1 Done Done Sent false
16721 5804 Acked Flush 0 Done Done Sent false
16722 5805 Acked Read 8 Done Done Sent false
16723 5806 Acked Write 2 Done Done Sent false
16724 5807 Acked Read 1 Done Done Sent false
16725 5808 Acked Write 7 Done Done Sent false
16726 5809 Acked Read 7 Done Done Sent false
16727 5810 Acked Write 10 Done Done Sent false
16728 5811 Acked Write 10 Done Done Sent false
16729 5812 Acked Write 5 Done Done Sent false
16730 5813 Acked Read 7 Done Done Sent false
16731 5814 Acked Read 1 Done Done Sent false
16732 5815 Acked Read 10 Done Done Sent false
16733 5816 Acked Write 1 Done Done Sent false
16734 5817 Acked Flush 0 Done Done Sent false
16735 5818 Acked Read 2 Done Done Sent false
16736 5819 Acked Write 7 Done Done Sent false
16737 5820 Acked Write 4 Done Done Sent false
16738 5821 Acked Read 8 Done Done Sent false
16739 5822 Acked Read 6 Done Done Sent false
16740 5823 Acked Write 1 Done Done Sent false
16741 5824 Acked Write 6 Done Done Sent false
16742 5825 Acked Read 10 Done Done Sent false
16743 5826 Acked Write 4 Done Done Sent false
16744 5827 Acked Read 7 Done Done Sent false
16745 5828 Acked Write 2 Done Done Sent false
16746 5829 Acked Read 2 Done Done Sent false
16747 5830 Acked Read 5 Done Done Sent false
16748 5831 Acked Flush 0 Done Done Sent false
16749 STATES DS:0 DS:1 DS:2 TOTAL
16750 Sent 0 0 302 302
16751 Done 302 302 0 604
16752 Skipped 0 0 0 0
16753 Error 0 0 0 0
16754 Last Flush: 5831 5831 5529
16755 Downstairs last five completed: 5529 5528 5527 5526 5525
16756 Upstairs last five completed: 5831 5830 5829 5828 5827
16757 CLIENT: Up:0 ds:302 act:2
16758 ----------------------------------------------------------------
16759 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
16760 States: Active Active Offline
16761 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16762 5530 Acked Write 10 Done Done Sent false
16763 5531 Acked Flush 0 Done Done Sent false
16764 5532 Acked Read 9 Done Done Sent false
16765 5533 Acked Flush 0 Done Done Sent false
16766 5534 Acked Read 3 Done Done Sent false
16767 5535 Acked Write 8 Done Done Sent false
16768 5536 Acked Read 10 Done Done Sent false
16769 5537 Acked Read 4 Done Done Sent false
16770 5538 Acked Read 3 Done Done Sent false
16771 5539 Acked Read 7 Done Done Sent false
16772 5540 Acked Read 7 Done Done Sent false
16773 5541 Acked Read 6 Done Done Sent false
16774 5542 Acked Read 5 Done Done Sent false
16775 5543 Acked Write 8 Done Done Sent false
16776 5544 Acked Write 7 Done Done Sent false
16777 5545 Acked Read 8 Done Done Sent false
16778 5546 Acked Read 5 Done Done Sent false
16779 5547 Acked Read 4 Done Done Sent false
16780 5548 Acked Write 8 Done Done Sent false
16781 5549 Acked Flush 0 Done Done Sent false
16782 5550 Acked Read 1 Done Done Sent false
16783 5551 Acked Write 7 Done Done Sent false
16784 5552 Acked Read 3 Done Done Sent false
16785 5553 Acked Write 2 Done Done Sent false
16786 5554 Acked Read 8 Done Done Sent false
16787 5555 Acked Flush 0 Done Done Sent false
16788 5556 Acked Read 4 Done Done Sent false
16789 5557 Acked Write 6 Done Done Sent false
16790 5558 Acked Write 3 Done Done Sent false
16791 5559 Acked Read 8 Done Done Sent false
16792 5560 Acked Flush 0 Done Done Sent false
16793 5561 Acked Write 5 Done Done Sent false
16794 5562 Acked Write 5 Done Done Sent false
16795 5563 Acked Read 3 Done Done Sent false
16796 5564 Acked Read 2 Done Done Sent false
16797 5565 Acked Write 6 Done Done Sent false
16798 5566 Acked Read 3 Done Done Sent false
16799 5567 Acked Read 5 Done Done Sent false
16800 5568 Acked Read 3 Done Done Sent false
16801 5569 Acked Read 6 Done Done Sent false
16802 5570 Acked Read 9 Done Done Sent false
16803 5571 Acked Write 5 Done Done Sent false
16804 5572 Acked Write 8 Done Done Sent false
16805 5573 Acked Read 8 Done Done Sent false
16806 5574 Acked Read 10 Done Done Sent false
16807 5575 Acked Write 5 Done Done Sent false
16808 5576 Acked Write 6 Done Done Sent false
16809 5577 Acked Read 5 Done Done Sent false
16810 5578 Acked Read 4 Done Done Sent false
16811 5579 Acked Write 10 Done Done Sent false
16812 5580 Acked Read 1 Done Done Sent false
16813 5581 Acked Write 4 Done Done Sent false
16814 5582 Acked Read 6 Done Done Sent false
16815 5583 Acked Write 5 Done Done Sent false
16816 5584 Acked Write 5 Done Done Sent false
16817 5585 Acked Read 9 Done Done Sent false
16818 5586 Acked Write 9 Done Done Sent false
16819 5587 Acked Read 3 Done Done Sent false
16820 5588 Acked Flush 0 Done Done Sent false
16821 5589 Acked Write 9 Done Done Sent false
16822 5590 Acked Write 9 Done Done Sent false
16823 5591 Acked Read 3 Done Done Sent false
16824 5592 Acked Read 6 Done Done Sent false
16825 5593 Acked Write 1 Done Done Sent false
16826 5594 Acked Write 1 Done Done Sent false
16827 5595 Acked Read 4 Done Done Sent false
16828 5596 Acked Read 1 Done Done Sent false
16829 5597 Acked Read 1 Done Done Sent false
16830 5598 Acked Write 3 Done Done Sent false
16831 5599 Acked Flush 0 Done Done Sent false
16832 5600 Acked Read 3 Done Done Sent false
16833 5601 Acked Write 10 Done Done Sent false
16834 5602 Acked Read 6 Done Done Sent false
16835 5603 Acked Write 6 Done Done Sent false
16836 5604 Acked Read 4 Done Done Sent false
16837 5605 Acked Read 10 Done Done Sent false
16838 5606 Acked Write 6 Done Done Sent false
16839 5607 Acked Read 1 Done Done Sent false
16840 5608 Acked Read 7 Done Done Sent false
16841 5609 Acked Write 3 Done Done Sent false
16842 5610 Acked Write 10 Done Done Sent false
16843 5611 Acked Read 7 Done Done Sent false
16844 5612 Acked Write 1 Done Done Sent false
16845 5613 Acked Write 1 Done Done Sent false
16846 5614 Acked Write 5 Done Done Sent false
16847 5615 Acked Write 5 Done Done Sent false
16848 5616 Acked Write 2 Done Done Sent false
16849 5617 Acked Read 9 Done Done Sent false
16850 5618 Acked Read 1 Done Done Sent false
16851 5619 Acked Write 9 Done Done Sent false
16852 5620 Acked Write 6 Done Done Sent false
16853 5621 Acked Write 8 Done Done Sent false
16854 5622 Acked Write 10 Done Done Sent false
16855 5623 Acked Read 9 Done Done Sent false
16856 5624 Acked Write 10 Done Done Sent false
16857 5625 Acked Read 5 Done Done Sent false
16858 5626 Acked Read 2 Done Done Sent false
16859 5627 Acked Read 9 Done Done Sent false
16860 5628 Acked Flush 0 Done Done Sent false
16861 5629 Acked Read 3 Done Done Sent false
16862 5630 Acked Read 9 Done Done Sent false
16863 5631 Acked Read 6 Done Done Sent false
16864 5632 Acked Read 2 Done Done Sent false
16865 5633 Acked Write 7 Done Done Sent false
16866 5634 Acked Flush 0 Done Done Sent false
16867 5635 Acked Read 8 Done Done Sent false
16868 5636 Acked Read 9 Done Done Sent false
16869 5637 Acked Write 9 Done Done Sent false
16870 5638 Acked Write 5 Done Done Sent false
16871 5639 Acked Write 3 Done Done Sent false
16872 5640 Acked Write 10 Done Done Sent false
16873 5641 Acked Read 10 Done Done Sent false
16874 5642 Acked Read 7 Done Done Sent false
16875 5643 Acked Write 2 Done Done Sent false
16876 5644 Acked Write 7 Done Done Sent false
16877 5645 Acked Write 3 Done Done Sent false
16878 5646 Acked Read 10 Done Done Sent false
16879 5647 Acked Read 7 Done Done Sent false
16880 5648 Acked Read 2 Done Done Sent false
16881 5649 Acked Write 3 Done Done Sent false
16882 5650 Acked Flush 0 Done Done Sent false
16883 5651 Acked Write 9 Done Done Sent false
16884 5652 Acked Write 6 Done Done Sent false
16885 5653 Acked Write 4 Done Done Sent false
16886 5654 Acked Read 9 Done Done Sent false
16887 5655 Acked Write 6 Done Done Sent false
16888 5656 Acked Write 9 Done Done Sent false
16889 5657 Acked Write 10 Done Done Sent false
16890 5658 Acked Write 5 Done Done Sent false
16891 5659 Acked Read 1 Done Done Sent false
16892 5660 Acked Write 9 Done Done Sent false
16893 5661 Acked Write 9 Done Done Sent false
16894 5662 Acked Flush 0 Done Done Sent false
16895 5663 Acked Write 5 Done Done Sent false
16896 5664 Acked Read 9 Done Done Sent false
16897 5665 Acked Write 10 Done Done Sent false
16898 5666 Acked Write 5 Done Done Sent false
16899 5667 Acked Read 3 Done Done Sent false
16900 5668 Acked Read 9 Done Done Sent false
16901 5669 Acked Read 10 Done Done Sent false
16902 5670 Acked Read 2 Done Done Sent false
16903 5671 Acked Write 1 Done Done Sent false
16904 5672 Acked Read 8 Done Done Sent false
16905 5673 Acked Read 5 Done Done Sent false
16906 5674 Acked Read 5 Done Done Sent false
16907 5675 Acked Write 8 Done Done Sent false
16908 5676 Acked Write 8 Done Done Sent false
16909 5677 Acked Read 1 Done Done Sent false
16910 5678 Acked Read 3 Done Done Sent false
16911 5679 Acked Read 9 Done Done Sent false
16912 5680 Acked Read 1 Done Done Sent false
16913 5681 Acked Read 10 Done Done Sent false
16914 5682 Acked Read 5 Done Done Sent false
16915 5683 Acked Write 6 Done Done Sent false
16916 5684 Acked Write 10 Done Done Sent false
16917 5685 Acked Read 8 Done Done Sent false
16918 5686 Acked Read 10 Done Done Sent false
16919 5687 Acked Read 5 Done Done Sent false
16920 5688 Acked Write 6 Done Done Sent false
16921 5689 Acked Read 8 Done Done Sent false
16922 5690 Acked Write 9 Done Done Sent false
16923 5691 Acked Read 10 Done Done Sent false
16924 5692 Acked Read 10 Done Done Sent false
16925 5693 Acked Read 7 Done Done Sent false
16926 5694 Acked Read 9 Done Done Sent false
16927 5695 Acked Read 8 Done Done Sent false
16928 5696 Acked Read 2 Done Done Sent false
16929 5697 Acked Read 1 Done Done Sent false
16930 5698 Acked Write 2 Done Done Sent false
16931 5699 Acked Read 4 Done Done Sent false
16932 5700 Acked Write 7 Done Done Sent false
16933 5701 Acked Read 7 Done Done Sent false
16934 5702 Acked Read 2 Done Done Sent false
16935 5703 Acked Read 1 Done Done Sent false
16936 5704 Acked Write 7 Done Done Sent false
16937 5705 Acked Read 7 Done Done Sent false
16938 5706 Acked Write 1 Done Done Sent false
16939 5707 Acked Write 4 Done Done Sent false
16940 5708 Acked Write 8 Done Done Sent false
16941 5709 Acked Write 7 Done Done Sent false
16942 5710 Acked Write 5 Done Done Sent false
16943 5711 Acked Write 1 Done Done Sent false
16944 5712 Acked Write 10 Done Done Sent false
16945 5713 Acked Read 3 Done Done Sent false
16946 5714 Acked Write 6 Done Done Sent false
16947 5715 Acked Read 10 Done Done Sent false
16948 5716 Acked Read 1 Done Done Sent false
16949 5717 Acked Write 2 Done Done Sent false
16950 5718 Acked Read 9 Done Done Sent false
16951 5719 Acked Write 8 Done Done Sent false
16952 5720 Acked Write 7 Done Done Sent false
16953 5721 Acked Read 7 Done Done Sent false
16954 5722 Acked Flush 0 Done Done Sent false
16955 5723 Acked Read 7 Done Done Sent false
16956 5724 Acked Write 4 Done Done Sent false
16957 5725 Acked Read 3 Done Done Sent false
16958 5726 Acked Write 6 Done Done Sent false
16959 5727 Acked Read 7 Done Done Sent false
16960 5728 Acked Write 9 Done Done Sent false
16961 5729 Acked Read 9 Done Done Sent false
16962 5730 Acked Read 9 Done Done Sent false
16963 5731 Acked Read 3 Done Done Sent false
16964 5732 Acked Write 5 Done Done Sent false
16965 5733 Acked Read 8 Done Done Sent false
16966 5734 Acked Write 7 Done Done Sent false
16967 5735 Acked Read 10 Done Done Sent false
16968 5736 Acked Flush 0 Done Done Sent false
16969 5737 Acked Read 3 Done Done Sent false
16970 5738 Acked Write 4 Done Done Sent false
16971 5739 Acked Flush 0 Done Done Sent false
16972 5740 Acked Read 8 Done Done Sent false
16973 5741 Acked Write 2 Done Done Sent false
16974 5742 Acked Read 9 Done Done Sent false
16975 5743 Acked Read 8 Done Done Sent false
16976 5744 Acked Flush 0 Done Done Sent false
16977 5745 Acked Write 7 Done Done Sent false
16978 5746 Acked Read 2 Done Done Sent false
16979 5747 Acked Read 10 Done Done Sent false
16980 5748 Acked Write 8 Done Done Sent false
16981 5749 Acked Read 6 Done Done Sent false
16982 5750 Acked Write 1 Done Done Sent false
16983 5751 Acked Write 9 Done Done Sent false
16984 5752 Acked Write 2 Done Done Sent false
16985 5753 Acked Write 2 Done Done Sent false
16986 5754 Acked Read 9 Done Done Sent false
16987 5755 Acked Write 9 Done Done Sent false
16988 5756 Acked Read 6 Done Done Sent false
16989 5757 Acked Read 7 Done Done Sent false
16990 5758 Acked Write 5 Done Done Sent false
16991 5759 Acked Write 8 Done Done Sent false
16992 5760 Acked Flush 0 Done Done Sent false
16993 5761 Acked Read 1 Done Done Sent false
16994 5762 Acked Read 3 Done Done Sent false
16995 5763 Acked Read 3 Done Done Sent false
16996 5764 Acked Read 4 Done Done Sent false
16997 5765 Acked Read 1 Done Done Sent false
16998 5766 Acked Write 4 Done Done Sent false
16999 5767 Acked Write 1 Done Done Sent false
17000 5768 Acked Write 2 Done Done Sent false
17001 5769 Acked Flush 0 Done Done Sent false
17002 5770 Acked Write 2 Done Done Sent false
17003 5771 Acked Write 5 Done Done Sent false
17004 5772 Acked Write 5 Done Done Sent false
17005 5773 Acked Flush 0 Done Done Sent false
17006 5774 Acked Flush 0 Done Done Sent false
17007 5775 Acked Read 6 Done Done Sent false
17008 5776 Acked Flush 0 Done Done Sent false
17009 5777 Acked Read 6 Done Done Sent false
17010 5778 Acked Read 9 Done Done Sent false
17011 5779 Acked Write 2 Done Done Sent false
17012 5780 Acked Read 7 Done Done Sent false
17013 5781 Acked Read 1 Done Done Sent false
17014 5782 Acked Read 4 Done Done Sent false
17015 5783 Acked Write 10 Done Done Sent false
17016 5784 Acked Write 6 Done Done Sent false
17017 5785 Acked Read 1 Done Done Sent false
17018 5786 Acked Write 10 Done Done Sent false
17019 5787 Acked Read 9 Done Done Sent false
17020 5788 Acked Flush 0 Done Done Sent false
17021 5789 Acked Read 1 Done Done Sent false
17022 5790 Acked Read 8 Done Done Sent false
17023 5791 Acked Read 6 Done Done Sent false
17024 5792 Acked Write 3 Done Done Sent false
17025 5793 Acked Write 4 Done Done Sent false
17026 5794 Acked Read 9 Done Done Sent false
17027 5795 Acked Write 2 Done Done Sent false
17028 5796 Acked Read 7 Done Done Sent false
17029 5797 Acked Write 1 Done Done Sent false
17030 5798 Acked Read 1 Done Done Sent false
17031 5799 Acked Read 3 Done Done Sent false
17032 5800 Acked Read 5 Done Done Sent false
17033 5801 Acked Read 10 Done Done Sent false
17034 5802 Acked Flush 0 Done Done Sent false
17035 5803 Acked Write 1 Done Done Sent false
17036 5804 Acked Flush 0 Done Done Sent false
17037 5805 Acked Read 8 Done Done Sent false
17038 5806 Acked Write 2 Done Done Sent false
17039 5807 Acked Read 1 Done Done Sent false
17040 5808 Acked Write 7 Done Done Sent false
17041 5809 Acked Read 7 Done Done Sent false
17042 5810 Acked Write 10 Done Done Sent false
17043 5811 Acked Write 10 Done Done Sent false
17044 5812 Acked Write 5 Done Done Sent false
17045 5813 Acked Read 7 Done Done Sent false
17046 5814 Acked Read 1 Done Done Sent false
17047 5815 Acked Read 10 Done Done Sent false
17048 5816 Acked Write 1 Done Done Sent false
17049 5817 Acked Flush 0 Done Done Sent false
17050 5818 Acked Read 2 Done Done Sent false
17051 5819 Acked Write 7 Done Done Sent false
17052 5820 Acked Write 4 Done Done Sent false
17053 5821 Acked Read 8 Done Done Sent false
17054 5822 Acked Read 6 Done Done Sent false
17055 5823 Acked Write 1 Done Done Sent false
17056 5824 Acked Write 6 Done Done Sent false
17057 5825 Acked Read 10 Done Done Sent false
17058 5826 Acked Write 4 Done Done Sent false
17059 5827 Acked Read 7 Done Done Sent false
17060 5828 Acked Write 2 Done Done Sent false
17061 5829 Acked Read 2 Done Done Sent false
17062 5830 Acked Read 5 Done Done Sent false
17063 5831 Acked Flush 0 Done Done Sent false
17064 STATES DS:0 DS:1 DS:2 TOTAL
17065 Sent 0 0 302 302
17066 Done 302 302 0 604
17067 Skipped 0 0 0 0
17068 Error 0 0 0 0
17069 Last Flush: 5831 5831 5529
17070 Downstairs last five completed: 5529 5528 5527 5526 5525
17071 Upstairs last five completed: 5831 5830 5829 5828 5827
17072 CLIENT: Up:0 ds:302 act:2
170732024-12-09T15:11:26.810ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
170742024-12-09T15:11:26.811ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39674) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
170752024-12-09T15:11:26.812ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
170762024-12-09T15:11:26.812ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
170772024-12-09T15:11:26.812ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
170782024-12-09T15:11:26.812ZINFOcrucible: send last flush ID to this DS: 5529 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
170792024-12-09T15:11:26.813ZINFOcrucible: Replied this last flush ID: 5529 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
170802024-12-09T15:11:26.813ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
170812024-12-09T15:11:26.813ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
170822024-12-09T15:11:26.813ZINFOcrucible: [2] client re-new 302 jobs since flush 5529 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
170832024-12-09T15:11:26.813ZINFOcrucible: [2] Replayed 302 jobs since flush: 5529 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
17084 ----------------------------------------------------------------
17085 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17086 Downstairs last five completed: 5831 5830 5829 5828 5827
17087 Upstairs last five completed: 5831 5830 5829 5828 5827
17088 CLIENT: Up:0 ds:0 act:3
17089 Replay: All jobs finished
17090 Main task has work to do, go find it
17091 got dsc Stop(2)
17092 stop 2
17093 Dec 09 15:11:27.064 INFO request completed[8830] Got stop action so:false kr:false
17094 , latency_us: 363, response_code: 204, uri: /stop/cid/2, method: POST, req_id: e802b920-8b8e-40b1-a4bf-d71fda8b265b, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
17095 Dec 09 15:11:27.065 INFO request completed{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:11:27.065739707Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
17096 , latency_us: 126, response_code: 200, uri: /state/cid/2{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:11:27.065811124Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
17097 , method: GET, req_id: 734a4bc0-8bd5-4b26-a9d5-7383be447ef9, remote_addr: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:11:27.065855986Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
17098 127.0.0{"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:11:27.065893619Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
17099 .{"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:11:27.065922274Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
17100 1:47510, local_addr: 127.0.0.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:11:27.065958867Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
17101 1:9998
17102 [8830] Exited with: ExitStatus(unix_wait_status(9))
17103 [8830][2] reports Exit
17104 Dec 09 15:11:31.068 INFO request completed, latency_us: 225, response_code: 200, uri: /state/cid/2, method: GET, req_id: 6c2f284d-4b57-4b3d-a214-9f92ce97528f, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
17105 001/300 Read block 557 len 4096
17106 002/300 Read block 1699 len 32768
17107 003/300 Write block 1682 len 28672 data: 2 3 2 2 2 2 3
17108 004/300 Read block 3781 len 24576
17109 005/300 Flush
17110 006/300 Read block 1164 len 4096
17111 007/300 Write block 2061 len 4096 data: 4
17112 008/300 Write block 763 len 28672 data: 5 4 4 5 4 3 5
17113 009/300 Read block 4984 len 8192
17114 010/300 Write block 4103 len 20480 data: 6 6 6 5 4
17115 011/300 Read block 4657 len 16384
17116 012/300 Write block 2000 len 36864 data: 5 5 5 5 6 6 5 4 5
17117 013/300 Read block 4717 len 20480
17118 014/300 Write block 722 len 8192 data: 3 3
17119 015/300 Read block 3214 len 28672
17120 016/300 Write block 3428 len 16384 data: 3 3 3 4
17121 017/300 Flush
17122 018/300 Read block 4813 len 8192
17123 019/300 Read block 1242 len 4096
17124 020/300 Read block 283 len 12288
17125 021/300 Write block 4035 len 32768 data: 3 3 3 3 2 2 2 2
17126 022/300 Read block 2320 len 32768
17127 023/300 Write block 2673 len 16384 data: 4 4 3 3
17128 024/300 Read block 3766 len 32768
17129 025/300 Write block 4116 len 4096 data: 4
17130 026/300 Read block 4862 len 12288
17131 027/300 Write block 2509 len 20480 data: 3 3 3 3 4
17132 028/300 Write block 1231 len 16384 data: 5 4 4 4
17133 029/300 Write block 3852 len 12288 data: 4 4 4
17134 030/300 Write block 2026 len 28672 data: 4 3 4 3 3 3 3
17135 031/300 Read block 3304 len 24576
17136 032/300 Write block 2453 len 8192 data: 3 4
17137 033/300 Read block 308 len 8192
17138 034/300 Write block 2697 len 28672 data: 4 5 5 5 5 5 4
17139 035/300 Write block 1906 len 16384 data: 4 4 4 6
17140 036/300 Read block 1369 len 32768
17141 037/300 Read block 2821 len 20480
17142 038/300 Read block 2508 len 16384
17143 039/300 Read block 333 len 40960
17144 040/300 Write block 733 len 12288 data: 4 3 2
17145 041/300 Read block 1478 len 28672
17146 042/300 Read block 2215 len 16384
17147 043/300 Read block 3294 len 20480
17148 044/300 Write block 3237 len 16384 data: 3 3 3 3
17149 045/300 Flush
17150 046/300 Read block 1989 len 4096
17151 047/300 Write block 2574 len 36864 data: 4 4 3 3 3 4 4 6 4
17152 048/300 Read block 1757 len 32768
17153 049/300 Read block 1145 len 16384
17154 050/300 Write block 3533 len 20480 data: 6 6 7 6 5
17155 051/300 Read block 413 len 8192
17156 052/300 Flush
17157 053/300 Write block 1685 len 4096 data: 3
17158 054/300 Read block 2999 len 4096
17159 055/300 Write block 3744 len 32768 data: 3 4 5 4 4 4 5 5
17160 056/300 Write block 4896 len 40960 data: 2 2 2 2 2 2 2 2 2 2
17161 057/300 Read block 1286 len 12288
17162 058/300 Read block 4324 len 40960
17163 059/300 Write block 3644 len 40960 data: 2 2 2 2 2 3 2 3 3 4
17164 060/300 Write block 1474 len 28672 data: 5 5 5 3 2 3 3
17165 061/300 Write block 550 len 12288 data: 9 9 7
17166 062/300 Read block 4104 len 32768
17167 063/300 Read block 111 len 28672
17168 064/300 Write block 2928 len 32768 data: 7 6 7 8 7 6 4 4
17169 065/300 Flush
17170 066/300 Read block 2598 len 20480
17171 067/300 Read block 95 len 4096
17172 068/300 Read block 3418 len 40960
17173 069/300 Read block 3030 len 28672
17174 070/300 Read block 2020 len 40960
17175 071/300 Read block 3498 len 32768
17176 072/300 Read block 3414 len 8192
17177 073/300 Write block 2600 len 12288 data: 3 5 5
17178 074/300 Read block 4171 len 40960
17179 075/300 Read block 1374 len 32768
17180 076/300 Write block 2933 len 40960 data: 7 5 5 4 3 4 5 3 4 4
17181 077/300 Write block 4720 len 12288 data: 4 5 4
17182 078/300 Read block 518 len 28672
17183 079/300 Write block 4247 len 40960 data: 7 4 3 3 3 4 3 3 3 4
17184 080/300 Read block 352 len 16384
17185 081/300 Write block 3313 len 32768 data: 3 4 3 3 2 2 2 5
17186 082/300 Read block 2737 len 32768
17187 083/300 Read block 3208 len 4096
17188 084/300 Read block 2584 len 24576
17189 085/300 Write block 431 len 8192 data: 5 5
17190 086/300 Flush
17191 087/300 Write block 1332 len 4096 data: 3
17192 088/300 Read block 1586 len 40960
17193 089/300 Read block 641 len 8192
17194 090/300 Write block 1341 len 4096 data: 7
17195 091/300 Write block 2456 len 16384 data: 5 5 6 6
17196 092/300 Flush
17197 093/300 Write block 2409 len 20480 data: 5 4 4 4 4
17198 094/300 Read block 1210 len 36864
17199 095/300 Write block 32 len 36864 data: 5 4 4 4 3 5 5 5 3
17200 096/300 Read block 4985 len 28672
17201 097/300 Read block 3559 len 16384
17202 098/300 Write block 3925 len 8192 data: 3 3
17203 099/300 Flush
17204 100/300 Flush
17205 101/300 Read block 2140 len 28672
17206 102/300 Write block 885 len 24576 data: 4 5 3 2 3 3
17207 103/300 Flush
17208 104/300 Write block 2248 len 32768 data: 4 5 5 3 3 5 6 6
17209 105/300 Read block 4863 len 12288
17210 106/300 Write block 2913 len 12288 data: 2 2 2
17211 107/300 Read block 866 len 20480
17212 108/300 Write block 1869 len 28672 data: 6 6 6 3 3 2 2
17213 109/300 Read block 3017 len 40960
17214 110/300 Read block 4650 len 4096
17215 111/300 Write block 1011 len 4096 data: 3
17216 112/300 Read block 2905 len 28672
17217 113/300 Read block 1544 len 36864
17218 114/300 Write block 3874 len 32768 data: 4 4 6 5 3 3 6 5
17219 115/300 Read block 2598 len 28672
17220 116/300 Read block 2670 len 24576
17221 117/300 Write block 1713 len 8192 data: 7 6
17222 118/300 Flush
17223 119/300 Read block 3542 len 8192
17224 120/300 Read block 3557 len 4096
17225 121/300 Read block 3290 len 4096
17226 122/300 Read block 276 len 24576
17227 123/300 Read block 815 len 40960
17228 124/300 Read block 1438 len 32768
17229 125/300 Write block 4009 len 20480 data: 4 5 4 5 5
17230 126/300 Read block 4487 len 4096
17231 127/300 Write block 90 len 32768 data: 2 2 2 3 3 3 3 3
17232 128/300 Write block 3689 len 8192 data: 4 4
17233 129/300 Write block 3195 len 12288 data: 3 3 4
17234 130/300 Read block 363 len 12288
17235 131/300 Write block 2504 len 36864 data: 3 3 3 3 3 4 4 4 4
17236 132/300 Write block 2103 len 8192 data: 4 5
17237 133/300 Read block 2180 len 32768
17238 134/300 Write block 676 len 16384 data: 4 4 4 3
17239 135/300 Read block 835 len 40960
17240 136/300 Read block 3286 len 28672
17241 137/300 Read block 4588 len 8192
17242 138/300 Read block 1411 len 20480
17243 139/300 Read block 3826 len 8192
17244 140/300 Read block 4306 len 20480
17245 141/300 Read block 4578 len 24576
17246 142/300 Read block 3585 len 12288
17247 143/300 Write block 1375 len 16384 data: 6 5 6 5
17248 144/300 Write block 1006 len 24576 data: 4 3 4 2 3 4
17249 145/300 Write block 3347 len 8192 data: 5 5
17250 146/300 Write block 1227 len 20480 data: 5 4 5 4 6
17251 147/300 Read block 4891 len 20480
17252 148/300 Read block 3919 len 28672
17253 149/300 Read block 4417 len 16384
17254 150/300 Read block 655 len 24576
17255 151/300 Read block 1141 len 16384
17256 152/300 Read block 3225 len 32768
17257 153/300 Write block 4357 len 24576 data: 4 4 4 5 5 5
17258 154/300 Read block 3457 len 20480
17259 155/300 Write block 2465 len 28672 data: 4 4 4 3 3 3 3
17260 156/300 Write block 611 len 4096 data: 5
17261 157/300 Write block 1887 len 12288 data: 7 5 5
17262 158/300 Write block 1537 len 20480 data: 5 4 5 4 4
17263 159/300 Flush
17264 160/300 Read block 1978 len 28672
17265 161/300 Read block 1093 len 16384
17266 162/300 Read block 2711 len 36864
17267 163/300 Read block 4951 len 12288
17268 164/300 Read block 3852 len 36864
17269 165/300 Write block 3171 len 28672 data: 3 2 2 3 4 4 4
17270 166/300 Flush
17271 167/300 Write block 2807 len 20480 data: 3 2 2 3 3
17272 168/300 Flush
17273 169/300 Write block 2776 len 40960 data: 9 9 9 9 8 4 4 5 4 5
17274 170/300 Read block 755 len 32768
17275 171/300 Read block 4110 len 32768
17276 172/300 Flush
17277 173/300 Write block 4927 len 40960 data: 2 2 2 4 3 5 4 5 5 4
17278 174/300 Read block 4087 len 4096
17279 175/300 Read block 2186 len 28672
17280 176/300 Write block 2399 len 16384 data: 5 5 6 5
17281 177/300 Write block 708 len 8192 data: 3 3
17282 178/300 Read block 3654 len 8192
17283 179/300 Write block 3183 len 36864 data: 4 4 3 2 2 2 2 2 2
17284 180/300 Write block 344 len 36864 data: 3 3 4 4 3 3 3 3 3
17285 181/300 Read block 1705 len 36864
17286 182/300 Read block 4133 len 4096
17287 183/300 Read block 3847 len 4096
17288 184/300 Write block 938 len 20480 data: 2 3 3 3 3
17289 185/300 Read block 1230 len 28672
17290 186/300 Read block 1115 len 12288
17291 187/300 Read block 2561 len 8192
17292 188/300 Write block 1441 len 36864 data: 4 6 7 6 6 6 6 3 3
17293 189/300 Read block 3193 len 36864
17294 190/300 Read block 2021 len 4096
17295 191/300 Read block 66 len 8192
17296 192/300 Read block 2190 len 12288
17297 193/300 Write block 3414 len 12288 data: 4 3 5
17298 194/300 Read block 4224 len 20480
17299 195/300 Write block 3727 len 20480 data: 5 5 4 3 2
17300 196/300 Write block 2170 len 40960 data: 2 2 4 5 5 5 5 4 4 5
17301 197/300 Flush
17302 198/300 Write block 3222 len 40960 data: 5 6 5 4 5 5 4 3 4 4
17303 199/300 Read block 4573 len 28672
17304 200/300 Read block 3873 len 28672
17305 201/300 Write block 3999 len 36864 data: 2 2 3 3 3 3 3 4 4
17306 202/300 Read block 323 len 8192
17307 203/300 Write block 1406 len 24576 data: 3 4 6 6 6 6
17308 204/300 Read block 2581 len 12288
17309 205/300 Flush
17310 206/300 Read block 1803 len 28672
17311 207/300 Write block 2937 len 40960 data: 4 5 6 4 5 5 4 3 3 4
17312 208/300 Read block 571 len 4096
17313 209/300 Read block 1365 len 40960
17314 210/300 Read block 3058 len 4096
17315 211/300 Write block 676 len 24576 data: 5 5 5 4 4 4
17316 212/300 Read block 851 len 16384
17317 213/300 Write block 4486 len 40960 data: 4 4 4 4 5 5 5 8 7 6
17318 214/300 Write block 3146 len 12288 data: 7 6 5
17319 215/300 Read block 4371 len 16384
17320 216/300 Read block 2992 len 28672
17321 217/300 Read block 2281 len 16384
17322 218/300 Read block 4010 len 20480
17323 219/300 Write block 4399 len 36864 data: 3 3 3 2 2 2 3 5 5
17324 220/300 Read block 3467 len 36864
17325 221/300 Read block 1127 len 16384
17326 222/300 Flush
17327 223/300 Write block 1040 len 28672 data: 6 7 7 7 6 7 5
17328 224/300 Write block 3783 len 32768 data: 5 4 3 3 5 5 4 3
17329 225/300 Read block 674 len 24576
17330 226/300 Write block 3014 len 32768 data: 4 4 4 5 5 4 3 3
17331 227/300 Read block 2931 len 32768
17332 228/300 Read block 3360 len 32768
17333 229/300 Read block 2096 len 32768
17334 230/300 Write block 2257 len 32768 data: 5 5 4 4 4 3 3 3
17335 231/300 Write block 3421 len 24576 data: 4 5 5 4 5 4
17336 232/300 Read block 4076 len 8192
17337 233/300 Read block 3769 len 4096
17338 234/300 Write block 4321 len 36864 data: 6 6 6 6 6 4 4 3 3
17339 235/300 Read block 2081 len 40960
17340 236/300 Write block 1877 len 8192 data: 2 2
17341 237/300 Read block 3935 len 4096
17342 238/300 Read block 2228 len 28672
17343 239/300 Write block 2879 len 16384 data: 2 2 2 2
17344 240/300 Read block 369 len 20480
17345 241/300 Write block 2005 len 40960 data: 7 6 5 6 5 4 5 4 5 6
17346 242/300 Write block 1336 len 8192 data: 9 10
17347 243/300 Read block 2836 len 12288
17348 244/300 Write block 4003 len 24576 data: 4 4 4 5 5 4
17349 245/300 Write block 2433 len 4096 data: 5
17350 246/300 Read block 2971 len 12288
17351 247/300 Write block 4788 len 16384 data: 5 4 3 3
17352 248/300 Read block 1980 len 20480
17353 249/300 Write block 1068 len 32768 data: 2 3 3 3 3 4 4 5
17354 250/300 Read block 4461 len 28672
17355 251/300 Read block 4899 len 12288
17356 252/300 Flush
17357 253/300 Read block 2522 len 8192
17358 254/300 Write block 4556 len 32768 data: 8 7 5 4 3 4 3 3
17359 255/300 Write block 4068 len 16384 data: 5 5 5 6
17360 256/300 Read block 839 len 4096
17361 257/300 Read block 1995 len 20480
17362 258/300 Write block 584 len 16384 data: 3 3 2 2
17363 259/300 Flush
17364 260/300 Write block 740 len 8192 data: 6 7
17365 261/300 Flush
17366 262/300 Write block 2788 len 32768 data: 4 4 4 5 5 6 5 5
17367 263/300 Read block 108 len 12288
17368 264/300 Read block 4783 len 40960
17369 265/300 Write block 157 len 32768 data: 6 5 5 4 4 3 3 4
17370 266/300 Read block 2581 len 20480
17371 267/300 Read block 320 len 36864
17372 268/300 Read block 4212 len 24576
17373 269/300 Read block 4557 len 36864
17374 270/300 Write block 4469 len 28672 data: 4 4 4 4 4 4 3
17375 271/300 Read block 1827 len 16384
17376 272/300 Read block 2051 len 40960
17377 273/300 Read block 2863 len 36864
17378 274/300 Read block 3879 len 32768
17379 275/300 Write block 4230 len 40960 data: 4 3 3 3 3 3 3 4 4 5
17380 276/300 Write block 2378 len 32768 data: 3 3 2 3 3 3 4 4
17381 277/300 Write block 2199 len 16384 data: 5 5 6 7
17382 278/300 Read block 3738 len 16384
17383 279/300 Read block 4856 len 36864
17384 280/300 Read block 2643 len 4096
17385 281/300 Flush
17386 282/300 Read block 602 len 24576
17387 283/300 Flush
17388 284/300 Read block 1992 len 4096
17389 285/300 Read block 2846 len 20480
17390 286/300 Write block 357 len 12288 data: 7 7 7
17391 287/300 Flush
17392 288/300 Write block 154 len 36864 data: 5 5 7 7 6 6 5 5 4
17393 289/300 Read block 4105 len 8192
17394 290/300 Read block 419 len 20480
17395 291/300 Read block 2311 len 20480
17396 292/300 Write block 30 len 28672 data: 4 4 6 5 5 5 4
17397 293/300 Read block 3352 len 36864
17398 294/300 Write block 1262 len 8192 data: 2 2
17399 295/300 Write block 3876 len 20480 data: 7 6 4 4 7
17400 296/300 Read block 3940 len 16384
17401 297/300 Read block 2176 len 20480
17402 298/300 Read block 2745 len 8192
17403 299/300 Write block 1370 len 16384 data: 8 8 8 6
17404 300/300 Write block 2120 len 24576 data: 4 4 4 4 4 5
17405 301/300 Read block 170 len 20480
17406 Main task has work to do, go find it
17407 got dsc Start(2)
17408 start 2
17409 Dec 09 15:11:31.310 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/2, method: POST, req_id: a57af8d9-673f-4721-8158-7286bf2ee753, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
17410 Replay: started 2, returned:Ok(())
17411 ----------------------------------------------------------------
17412 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17413 States: Active Active Offline
17414 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17415 5832 Acked Read 1 Done Done Sent false
17416 5833 Acked Read 8 Done Done Sent false
17417 5834 Acked Write 7 Done Done Sent false
17418 5835 Acked Read 6 Done Done Sent false
17419 5836 Acked Flush 0 Done Done Sent false
17420 5837 Acked Read 1 Done Done Sent false
17421 5838 Acked Write 1 Done Done Sent false
17422 5839 Acked Write 7 Done Done Sent false
17423 5840 Acked Read 2 Done Done Sent false
17424 5841 Acked Write 5 Done Done Sent false
17425 5842 Acked Read 4 Done Done Sent false
17426 5843 Acked Write 9 Done Done Sent false
17427 5844 Acked Read 5 Done Done Sent false
17428 5845 Acked Write 2 Done Done Sent false
17429 5846 Acked Read 7 Done Done Sent false
17430 5847 Acked Write 4 Done Done Sent false
17431 5848 Acked Flush 0 Done Done Sent false
17432 5849 Acked Read 2 Done Done Sent false
17433 5850 Acked Read 1 Done Done Sent false
17434 5851 Acked Read 3 Done Done Sent false
17435 5852 Acked Write 8 Done Done Sent false
17436 5853 Acked Read 8 Done Done Sent false
17437 5854 Acked Write 4 Done Done Sent false
17438 5855 Acked Read 8 Done Done Sent false
17439 5856 Acked Write 1 Done Done Sent false
17440 5857 Acked Read 3 Done Done Sent false
17441 5858 Acked Write 5 Done Done Sent false
17442 5859 Acked Write 4 Done Done Sent false
17443 5860 Acked Write 3 Done Done Sent false
17444 5861 Acked Write 7 Done Done Sent false
17445 5862 Acked Read 6 Done Done Sent false
17446 5863 Acked Write 2 Done Done Sent false
17447 5864 Acked Read 2 Done Done Sent false
17448 5865 Acked Write 7 Done Done Sent false
17449 5866 Acked Write 4 Done Done Sent false
17450 5867 Acked Read 8 Done Done Sent false
17451 5868 Acked Read 5 Done Done Sent false
17452 5869 Acked Read 4 Done Done Sent false
17453 5870 Acked Read 10 Done Done Sent false
17454 5871 Acked Write 3 Done Done Sent false
17455 5872 Acked Read 7 Done Done Sent false
17456 5873 Acked Read 4 Done Done Sent false
17457 5874 Acked Read 5 Done Done Sent false
17458 5875 Acked Write 4 Done Done Sent false
17459 5876 Acked Flush 0 Done Done Sent false
17460 5877 Acked Read 1 Done Done Sent false
17461 5878 Acked Write 9 Done Done Sent false
17462 5879 Acked Read 8 Done Done Sent false
17463 5880 Acked Read 4 Done Done Sent false
17464 5881 Acked Write 5 Done Done Sent false
17465 5882 Acked Read 2 Done Done Sent false
17466 5883 Acked Flush 0 Done Done Sent false
17467 5884 Acked Write 1 Done Done Sent false
17468 5885 Acked Read 1 Done Done Sent false
17469 5886 Acked Write 8 Done Done Sent false
17470 5887 Acked Write 10 Done Done Sent false
17471 5888 Acked Read 3 Done Done Sent false
17472 5889 Acked Read 10 Done Done Sent false
17473 5890 Acked Write 10 Done Done Sent false
17474 5891 Acked Write 7 Done Done Sent false
17475 5892 Acked Write 3 Done Done Sent false
17476 5893 Acked Read 8 Done Done Sent false
17477 5894 Acked Read 7 Done Done Sent false
17478 5895 Acked Write 8 Done Done Sent false
17479 5896 Acked Flush 0 Done Done Sent false
17480 5897 Acked Read 5 Done Done Sent false
17481 5898 Acked Read 1 Done Done Sent false
17482 5899 Acked Read 10 Done Done Sent false
17483 5900 Acked Read 7 Done Done Sent false
17484 5901 Acked Read 10 Done Done Sent false
17485 5902 Acked Read 8 Done Done Sent false
17486 5903 Acked Read 2 Done Done Sent false
17487 5904 Acked Write 3 Done Done Sent false
17488 5905 Acked Read 10 Done Done Sent false
17489 5906 Acked Read 8 Done Done Sent false
17490 5907 Acked Write 10 Done Done Sent false
17491 5908 Acked Write 3 Done Done Sent false
17492 5909 Acked Read 7 Done Done Sent false
17493 5910 Acked Write 10 Done Done Sent false
17494 5911 Acked Read 4 Done Done Sent false
17495 5912 Acked Write 8 Done Done Sent false
17496 5913 Acked Read 8 Done Done Sent false
17497 5914 Acked Read 1 Done Done Sent false
17498 5915 Acked Read 6 Done Done Sent false
17499 5916 Acked Write 2 Done Done Sent false
17500 5917 Acked Flush 0 Done Done Sent false
17501 5918 Acked Write 1 Done Done Sent false
17502 5919 Acked Read 10 Done Done Sent false
17503 5920 Acked Read 2 Done Done Sent false
17504 5921 Acked Write 1 Done Done Sent false
17505 5922 Acked Write 4 Done Done Sent false
17506 5923 Acked Flush 0 Done Done Sent false
17507 5924 Acked Write 5 Done Done Sent false
17508 5925 Acked Read 9 Done Done Sent false
17509 5926 Acked Write 9 Done Done Sent false
17510 5927 Acked Read 7 Done Done Sent false
17511 5928 Acked Read 4 Done Done Sent false
17512 5929 Acked Write 2 Done Done Sent false
17513 5930 Acked Flush 0 Done Done Sent false
17514 5931 Acked Flush 0 Done Done Sent false
17515 5932 Acked Read 7 Done Done Sent false
17516 5933 Acked Write 6 Done Done Sent false
17517 5934 Acked Flush 0 Done Done Sent false
17518 5935 Acked Write 8 Done Done Sent false
17519 5936 Acked Read 3 Done Done Sent false
17520 5937 Acked Write 3 Done Done Sent false
17521 5938 Acked Read 5 Done Done Sent false
17522 5939 Acked Write 7 Done Done Sent false
17523 5940 Acked Read 10 Done Done Sent false
17524 5941 Acked Read 1 Done Done Sent false
17525 5942 Acked Write 1 Done Done Sent false
17526 5943 Acked Read 7 Done Done Sent false
17527 5944 Acked Read 9 Done Done Sent false
17528 5945 Acked Write 8 Done Done Sent false
17529 5946 Acked Read 7 Done Done Sent false
17530 5947 Acked Read 6 Done Done Sent false
17531 5948 Acked Write 2 Done Done Sent false
17532 5949 Acked Flush 0 Done Done Sent false
17533 5950 Acked Read 2 Done Done Sent false
17534 5951 Acked Read 1 Done Done Sent false
17535 5952 Acked Read 1 Done Done Sent false
17536 5953 Acked Read 6 Done Done Sent false
17537 5954 Acked Read 10 Done Done Sent false
17538 5955 Acked Read 8 Done Done Sent false
17539 5956 Acked Write 5 Done Done Sent false
17540 5957 Acked Read 1 Done Done Sent false
17541 5958 Acked Write 8 Done Done Sent false
17542 5959 Acked Write 2 Done Done Sent false
17543 5960 Acked Write 3 Done Done Sent false
17544 5961 Acked Read 3 Done Done Sent false
17545 5962 Acked Write 9 Done Done Sent false
17546 5963 Acked Write 2 Done Done Sent false
17547 5964 Acked Read 8 Done Done Sent false
17548 5965 Acked Write 4 Done Done Sent false
17549 5966 Acked Read 10 Done Done Sent false
17550 5967 Acked Read 7 Done Done Sent false
17551 5968 Acked Read 2 Done Done Sent false
17552 5969 Acked Read 5 Done Done Sent false
17553 5970 Acked Read 2 Done Done Sent false
17554 5971 Acked Read 5 Done Done Sent false
17555 5972 Acked Read 6 Done Done Sent false
17556 5973 Acked Read 3 Done Done Sent false
17557 5974 Acked Write 4 Done Done Sent false
17558 5975 Acked Write 6 Done Done Sent false
17559 5976 Acked Write 2 Done Done Sent false
17560 5977 Acked Write 5 Done Done Sent false
17561 5978 Acked Read 5 Done Done Sent false
17562 5979 Acked Read 7 Done Done Sent false
17563 5980 Acked Read 4 Done Done Sent false
17564 5981 Acked Read 6 Done Done Sent false
17565 5982 Acked Read 4 Done Done Sent false
17566 5983 Acked Read 8 Done Done Sent false
17567 5984 Acked Write 6 Done Done Sent false
17568 5985 Acked Read 5 Done Done Sent false
17569 5986 Acked Write 7 Done Done Sent false
17570 5987 Acked Write 1 Done Done Sent false
17571 5988 Acked Write 3 Done Done Sent false
17572 5989 Acked Write 5 Done Done Sent false
17573 5990 Acked Flush 0 Done Done Sent false
17574 5991 Acked Read 7 Done Done Sent false
17575 5992 Acked Read 4 Done Done Sent false
17576 5993 Acked Read 9 Done Done Sent false
17577 5994 Acked Read 3 Done Done Sent false
17578 5995 Acked Read 9 Done Done Sent false
17579 5996 Acked Write 7 Done Done Sent false
17580 5997 Acked Flush 0 Done Done Sent false
17581 5998 Acked Write 5 Done Done Sent false
17582 5999 Acked Flush 0 Done Done Sent false
17583 6000 Acked Write 10 Done Done Sent false
17584 6001 Acked Read 8 Done Done Sent false
17585 6002 Acked Read 8 Done Done Sent false
17586 6003 Acked Flush 0 Done Done Sent false
17587 6004 Acked Write 10 Done Done Sent false
17588 6005 Acked Read 1 Done Done Sent false
17589 6006 Acked Read 7 Done Done Sent false
17590 6007 Acked Write 4 Done Done Sent false
17591 6008 Acked Write 2 Done Done Sent false
17592 6009 Acked Read 2 Done Done Sent false
17593 6010 Acked Write 9 Done Done Sent false
17594 6011 Acked Write 9 Done Done Sent false
17595 6012 Acked Read 9 Done Done Sent false
17596 6013 Acked Read 1 Done Done Sent false
17597 6014 Acked Read 1 Done Done Sent false
17598 6015 Acked Write 5 Done Done Sent false
17599 6016 Acked Read 7 Done Done Sent false
17600 6017 Acked Read 3 Done Done Sent false
17601 6018 Acked Read 2 Done Done Sent false
17602 6019 Acked Write 9 Done Done Sent false
17603 6020 Acked Read 9 Done Done Sent false
17604 6021 Acked Read 1 Done Done Sent false
17605 6022 Acked Read 2 Done Done Sent false
17606 6023 Acked Read 3 Done Done Sent false
17607 6024 Acked Write 3 Done Done Sent false
17608 6025 Acked Read 5 Done Done Sent false
17609 6026 Acked Write 5 Done Done Sent false
17610 6027 Acked Write 10 Done Done Sent false
17611 6028 Acked Flush 0 Done Done Sent false
17612 6029 Acked Write 10 Done Done Sent false
17613 6030 Acked Read 7 Done Done Sent false
17614 6031 Acked Read 7 Done Done Sent false
17615 6032 Acked Write 9 Done Done Sent false
17616 6033 Acked Read 2 Done Done Sent false
17617 6034 Acked Write 6 Done Done Sent false
17618 6035 Acked Read 3 Done Done Sent false
17619 6036 Acked Flush 0 Done Done Sent false
17620 6037 Acked Read 7 Done Done Sent false
17621 6038 Acked Write 10 Done Done Sent false
17622 6039 Acked Read 1 Done Done Sent false
17623 6040 Acked Read 10 Done Done Sent false
17624 6041 Acked Read 1 Done Done Sent false
17625 6042 Acked Write 6 Done Done Sent false
17626 6043 Acked Read 4 Done Done Sent false
17627 6044 Acked Write 10 Done Done Sent false
17628 6045 Acked Write 3 Done Done Sent false
17629 6046 Acked Read 4 Done Done Sent false
17630 6047 Acked Read 7 Done Done Sent false
17631 6048 Acked Read 4 Done Done Sent false
17632 6049 Acked Read 5 Done Done Sent false
17633 6050 Acked Write 9 Done Done Sent false
17634 6051 Acked Read 9 Done Done Sent false
17635 6052 Acked Read 4 Done Done Sent false
17636 6053 Acked Flush 0 Done Done Sent false
17637 6054 Acked Write 7 Done Done Sent false
17638 6055 Acked Write 8 Done Done Sent false
17639 6056 Acked Read 6 Done Done Sent false
17640 6057 Acked Write 8 Done Done Sent false
17641 6058 Acked Read 8 Done Done Sent false
17642 6059 Acked Read 8 Done Done Sent false
17643 6060 Acked Read 8 Done Done Sent false
17644 6061 Acked Write 8 Done Done Sent false
17645 6062 Acked Write 6 Done Done Sent false
17646 6063 Acked Read 2 Done Done Sent false
17647 6064 Acked Read 1 Done Done Sent false
17648 6065 Acked Write 9 Done Done Sent false
17649 6066 Acked Read 10 Done Done Sent false
17650 6067 Acked Write 2 Done Done Sent false
17651 6068 Acked Read 1 Done Done Sent false
17652 6069 Acked Read 7 Done Done Sent false
17653 6070 Acked Write 4 Done Done Sent false
17654 6071 Acked Read 5 Done Done Sent false
17655 6072 Acked Write 10 Done Done Sent false
17656 6073 Acked Write 2 Done Done Sent false
17657 6074 Acked Read 3 Done Done Sent false
17658 6075 Acked Write 6 Done Done Sent false
17659 6076 Acked Write 1 Done Done Sent false
17660 6077 Acked Read 3 Done Done Sent false
17661 6078 Acked Write 4 Done Done Sent false
17662 6079 Acked Read 5 Done Done Sent false
17663 6080 Acked Write 8 Done Done Sent false
17664 6081 Acked Read 7 Done Done Sent false
17665 6082 Acked Read 3 Done Done Sent false
17666 6083 Acked Flush 0 Done Done Sent false
17667 6084 Acked Read 2 Done Done Sent false
17668 6085 Acked Write 8 Done Done Sent false
17669 6086 Acked Write 4 Done Done Sent false
17670 6087 Acked Read 1 Done Done Sent false
17671 6088 Acked Read 5 Done Done Sent false
17672 6089 Acked Write 4 Done Done Sent false
17673 6090 Acked Flush 0 Done Done Sent false
17674 6091 Acked Write 2 Done Done Sent false
17675 6092 Acked Flush 0 Done Done Sent false
17676 6093 Acked Write 8 Done Done Sent false
17677 6094 Acked Read 3 Done Done Sent false
17678 6095 Acked Read 10 Done Done Sent false
17679 6096 Acked Write 8 Done Done Sent false
17680 6097 Acked Read 5 Done Done Sent false
17681 6098 Acked Read 9 Done Done Sent false
17682 6099 Acked Read 6 Done Done Sent false
17683 6100 Acked Read 9 Done Done Sent false
17684 6101 Acked Write 7 Done Done Sent false
17685 6102 Acked Read 4 Done Done Sent false
17686 6103 Acked Read 10 Done Done Sent false
17687 6104 Acked Read 9 Done Done Sent false
17688 6105 Acked Read 8 Done Done Sent false
17689 6106 Acked Write 10 Done Done Sent false
17690 6107 Acked Write 8 Done Done Sent false
17691 6108 Acked Write 4 Done Done Sent false
17692 6109 Acked Read 4 Done Done Sent false
17693 6110 Acked Read 9 Done Done Sent false
17694 6111 Acked Read 1 Done Done Sent false
17695 6112 Acked Flush 0 Done Done Sent false
17696 6113 Acked Read 6 Done Done Sent false
17697 6114 Acked Flush 0 Done Done Sent false
17698 6115 Acked Read 1 Done Done Sent false
17699 6116 Acked Read 5 Done Done Sent false
17700 6117 Acked Write 3 Done Done Sent false
17701 6118 Acked Flush 0 Done Done Sent false
17702 6119 Acked Write 9 Done Done Sent false
17703 6120 Acked Read 2 Done Done Sent false
17704 6121 Acked Read 5 Done Done Sent false
17705 6122 Acked Read 5 Done Done Sent false
17706 6123 Acked Write 7 Done Done Sent false
17707 6124 Acked Read 9 Done Done Sent false
17708 6125 Acked Write 2 Done Done Sent false
17709 6126 Acked Write 5 Done Done Sent false
17710 6127 Acked Read 4 Done Done Sent false
17711 6128 Acked Read 5 Done Done Sent false
17712 6129 Acked Read 2 Done Done Sent false
17713 6130 Acked Write 4 Done Done Sent false
17714 6131 Acked Write 6 Done Done Sent false
17715 6132 Acked Read 5 Done Done Sent false
17716 STATES DS:0 DS:1 DS:2 TOTAL
17717 Sent 0 0 301 301
17718 Done 301 301 0 602
17719 Skipped 0 0 0 0
17720 Error 0 0 0 0
17721 Last Flush: 6118 6118 5831
17722 Downstairs last five completed: 5831 5830 5829 5828 5827
17723 Upstairs last five completed: 6132 6131 6130 6129 6128
17724 CLIENT: Up:0 ds:301 act:2
17725 ----------------------------------------------------------------
17726 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
17727 States: Active Active Offline
17728 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17729 5832 Acked Read 1 Done Done Sent false
17730 5833 Acked Read 8 Done Done Sent false
17731 5834 Acked Write 7 Done Done Sent false
17732 5835 Acked Read 6 Done Done Sent false
17733 5836 Acked Flush 0 Done Done Sent false
17734 5837 Acked Read 1 Done Done Sent false
17735 5838 Acked Write 1 Done Done Sent false
17736 5839 Acked Write 7 Done Done Sent false
17737 5840 Acked Read 2 Done Done Sent false
17738 5841 Acked Write 5 Done Done Sent false
17739 5842 Acked Read 4 Done Done Sent false
17740 5843 Acked Write 9 Done Done Sent false
17741 5844 Acked Read 5 Done Done Sent false
17742 5845 Acked Write 2 Done Done Sent false
17743 5846 Acked Read 7 Done Done Sent false
17744 5847 Acked Write 4 Done Done Sent false
17745 5848 Acked Flush 0 Done Done Sent false
17746 5849 Acked Read 2 Done Done Sent false
17747 5850 Acked Read 1 Done Done Sent false
17748 5851 Acked Read 3 Done Done Sent false
17749 5852 Acked Write 8 Done Done Sent false
17750 5853 Acked Read 8 Done Done Sent false
17751 5854 Acked Write 4 Done Done Sent false
17752 5855 Acked Read 8 Done Done Sent false
17753 5856 Acked Write 1 Done Done Sent false
17754 5857 Acked Read 3 Done Done Sent false
17755 5858 Acked Write 5 Done Done Sent false
17756 5859 Acked Write 4 Done Done Sent false
17757 5860 Acked Write 3 Done Done Sent false
17758 5861 Acked Write 7 Done Done Sent false
17759 5862 Acked Read 6 Done Done Sent false
17760 5863 Acked Write 2 Done Done Sent false
17761 5864 Acked Read 2 Done Done Sent false
17762 5865 Acked Write 7 Done Done Sent false
17763 5866 Acked Write 4 Done Done Sent false
17764 5867 Acked Read 8 Done Done Sent false
17765 5868 Acked Read 5 Done Done Sent false
17766 5869 Acked Read 4 Done Done Sent false
17767 5870 Acked Read 10 Done Done Sent false
17768 5871 Acked Write 3 Done Done Sent false
17769 5872 Acked Read 7 Done Done Sent false
17770 5873 Acked Read 4 Done Done Sent false
17771 5874 Acked Read 5 Done Done Sent false
17772 5875 Acked Write 4 Done Done Sent false
17773 5876 Acked Flush 0 Done Done Sent false
17774 5877 Acked Read 1 Done Done Sent false
17775 5878 Acked Write 9 Done Done Sent false
17776 5879 Acked Read 8 Done Done Sent false
17777 5880 Acked Read 4 Done Done Sent false
17778 5881 Acked Write 5 Done Done Sent false
17779 5882 Acked Read 2 Done Done Sent false
17780 5883 Acked Flush 0 Done Done Sent false
17781 5884 Acked Write 1 Done Done Sent false
17782 5885 Acked Read 1 Done Done Sent false
17783 5886 Acked Write 8 Done Done Sent false
17784 5887 Acked Write 10 Done Done Sent false
17785 5888 Acked Read 3 Done Done Sent false
17786 5889 Acked Read 10 Done Done Sent false
17787 5890 Acked Write 10 Done Done Sent false
17788 5891 Acked Write 7 Done Done Sent false
17789 5892 Acked Write 3 Done Done Sent false
17790 5893 Acked Read 8 Done Done Sent false
17791 5894 Acked Read 7 Done Done Sent false
17792 5895 Acked Write 8 Done Done Sent false
17793 5896 Acked Flush 0 Done Done Sent false
17794 5897 Acked Read 5 Done Done Sent false
17795 5898 Acked Read 1 Done Done Sent false
17796 5899 Acked Read 10 Done Done Sent false
17797 5900 Acked Read 7 Done Done Sent false
17798 5901 Acked Read 10 Done Done Sent false
17799 5902 Acked Read 8 Done Done Sent false
17800 5903 Acked Read 2 Done Done Sent false
17801 5904 Acked Write 3 Done Done Sent false
17802 5905 Acked Read 10 Done Done Sent false
17803 5906 Acked Read 8 Done Done Sent false
17804 5907 Acked Write 10 Done Done Sent false
17805 5908 Acked Write 3 Done Done Sent false
17806 5909 Acked Read 7 Done Done Sent false
17807 5910 Acked Write 10 Done Done Sent false
17808 5911 Acked Read 4 Done Done Sent false
17809 5912 Acked Write 8 Done Done Sent false
17810 5913 Acked Read 8 Done Done Sent false
17811 5914 Acked Read 1 Done Done Sent false
17812 5915 Acked Read 6 Done Done Sent false
17813 5916 Acked Write 2 Done Done Sent false
17814 5917 Acked Flush 0 Done Done Sent false
17815 5918 Acked Write 1 Done Done Sent false
17816 5919 Acked Read 10 Done Done Sent false
17817 5920 Acked Read 2 Done Done Sent false
17818 5921 Acked Write 1 Done Done Sent false
17819 5922 Acked Write 4 Done Done Sent false
17820 5923 Acked Flush 0 Done Done Sent false
17821 5924 Acked Write 5 Done Done Sent false
17822 5925 Acked Read 9 Done Done Sent false
17823 5926 Acked Write 9 Done Done Sent false
17824 5927 Acked Read 7 Done Done Sent false
17825 5928 Acked Read 4 Done Done Sent false
17826 5929 Acked Write 2 Done Done Sent false
17827 5930 Acked Flush 0 Done Done Sent false
17828 5931 Acked Flush 0 Done Done Sent false
17829 5932 Acked Read 7 Done Done Sent false
17830 5933 Acked Write 6 Done Done Sent false
17831 5934 Acked Flush 0 Done Done Sent false
17832 5935 Acked Write 8 Done Done Sent false
17833 5936 Acked Read 3 Done Done Sent false
17834 5937 Acked Write 3 Done Done Sent false
17835 5938 Acked Read 5 Done Done Sent false
17836 5939 Acked Write 7 Done Done Sent false
17837 5940 Acked Read 10 Done Done Sent false
17838 5941 Acked Read 1 Done Done Sent false
17839 5942 Acked Write 1 Done Done Sent false
17840 5943 Acked Read 7 Done Done Sent false
17841 5944 Acked Read 9 Done Done Sent false
17842 5945 Acked Write 8 Done Done Sent false
17843 5946 Acked Read 7 Done Done Sent false
17844 5947 Acked Read 6 Done Done Sent false
17845 5948 Acked Write 2 Done Done Sent false
17846 5949 Acked Flush 0 Done Done Sent false
17847 5950 Acked Read 2 Done Done Sent false
17848 5951 Acked Read 1 Done Done Sent false
17849 5952 Acked Read 1 Done Done Sent false
17850 5953 Acked Read 6 Done Done Sent false
17851 5954 Acked Read 10 Done Done Sent false
17852 5955 Acked Read 8 Done Done Sent false
17853 5956 Acked Write 5 Done Done Sent false
17854 5957 Acked Read 1 Done Done Sent false
17855 5958 Acked Write 8 Done Done Sent false
17856 5959 Acked Write 2 Done Done Sent false
17857 5960 Acked Write 3 Done Done Sent false
17858 5961 Acked Read 3 Done Done Sent false
17859 5962 Acked Write 9 Done Done Sent false
17860 5963 Acked Write 2 Done Done Sent false
17861 5964 Acked Read 8 Done Done Sent false
17862 5965 Acked Write 4 Done Done Sent false
17863 5966 Acked Read 10 Done Done Sent false
17864 5967 Acked Read 7 Done Done Sent false
17865 5968 Acked Read 2 Done Done Sent false
17866 5969 Acked Read 5 Done Done Sent false
17867 5970 Acked Read 2 Done Done Sent false
17868 5971 Acked Read 5 Done Done Sent false
17869 5972 Acked Read 6 Done Done Sent false
17870 5973 Acked Read 3 Done Done Sent false
17871 5974 Acked Write 4 Done Done Sent false
17872 5975 Acked Write 6 Done Done Sent false
17873 5976 Acked Write 2 Done Done Sent false
17874 5977 Acked Write 5 Done Done Sent false
17875 5978 Acked Read 5 Done Done Sent false
17876 5979 Acked Read 7 Done Done Sent false
17877 5980 Acked Read 4 Done Done Sent false
17878 5981 Acked Read 6 Done Done Sent false
17879 5982 Acked Read 4 Done Done Sent false
17880 5983 Acked Read 8 Done Done Sent false
17881 5984 Acked Write 6 Done Done Sent false
17882 5985 Acked Read 5 Done Done Sent false
17883 5986 Acked Write 7 Done Done Sent false
17884 5987 Acked Write 1 Done Done Sent false
17885 5988 Acked Write 3 Done Done Sent false
17886 5989 Acked Write 5 Done Done Sent false
17887 5990 Acked Flush 0 Done Done Sent false
17888 5991 Acked Read 7 Done Done Sent false
17889 5992 Acked Read 4 Done Done Sent false
17890 5993 Acked Read 9 Done Done Sent false
17891 5994 Acked Read 3 Done Done Sent false
17892 5995 Acked Read 9 Done Done Sent false
17893 5996 Acked Write 7 Done Done Sent false
17894 5997 Acked Flush 0 Done Done Sent false
17895 5998 Acked Write 5 Done Done Sent false
17896 5999 Acked Flush 0 Done Done Sent false
17897 6000 Acked Write 10 Done Done Sent false
17898 6001 Acked Read 8 Done Done Sent false
17899 6002 Acked Read 8 Done Done Sent false
17900 6003 Acked Flush 0 Done Done Sent false
17901 6004 Acked Write 10 Done Done Sent false
17902 6005 Acked Read 1 Done Done Sent false
17903 6006 Acked Read 7 Done Done Sent false
17904 6007 Acked Write 4 Done Done Sent false
17905 6008 Acked Write 2 Done Done Sent false
17906 6009 Acked Read 2 Done Done Sent false
17907 6010 Acked Write 9 Done Done Sent false
17908 6011 Acked Write 9 Done Done Sent false
17909 6012 Acked Read 9 Done Done Sent false
17910 6013 Acked Read 1 Done Done Sent false
17911 6014 Acked Read 1 Done Done Sent false
17912 6015 Acked Write 5 Done Done Sent false
17913 6016 Acked Read 7 Done Done Sent false
17914 6017 Acked Read 3 Done Done Sent false
17915 6018 Acked Read 2 Done Done Sent false
17916 6019 Acked Write 9 Done Done Sent false
17917 6020 Acked Read 9 Done Done Sent false
17918 6021 Acked Read 1 Done Done Sent false
17919 6022 Acked Read 2 Done Done Sent false
17920 6023 Acked Read 3 Done Done Sent false
17921 6024 Acked Write 3 Done Done Sent false
17922 6025 Acked Read 5 Done Done Sent false
17923 6026 Acked Write 5 Done Done Sent false
17924 6027 Acked Write 10 Done Done Sent false
17925 6028 Acked Flush 0 Done Done Sent false
17926 6029 Acked Write 10 Done Done Sent false
17927 6030 Acked Read 7 Done Done Sent false
17928 6031 Acked Read 7 Done Done Sent false
17929 6032 Acked Write 9 Done Done Sent false
17930 6033 Acked Read 2 Done Done Sent false
17931 6034 Acked Write 6 Done Done Sent false
17932 6035 Acked Read 3 Done Done Sent false
17933 6036 Acked Flush 0 Done Done Sent false
17934 6037 Acked Read 7 Done Done Sent false
17935 6038 Acked Write 10 Done Done Sent false
17936 6039 Acked Read 1 Done Done Sent false
17937 6040 Acked Read 10 Done Done Sent false
17938 6041 Acked Read 1 Done Done Sent false
17939 6042 Acked Write 6 Done Done Sent false
17940 6043 Acked Read 4 Done Done Sent false
17941 6044 Acked Write 10 Done Done Sent false
17942 6045 Acked Write 3 Done Done Sent false
17943 6046 Acked Read 4 Done Done Sent false
17944 6047 Acked Read 7 Done Done Sent false
17945 6048 Acked Read 4 Done Done Sent false
17946 6049 Acked Read 5 Done Done Sent false
17947 6050 Acked Write 9 Done Done Sent false
17948 6051 Acked Read 9 Done Done Sent false
17949 6052 Acked Read 4 Done Done Sent false
17950 6053 Acked Flush 0 Done Done Sent false
17951 6054 Acked Write 7 Done Done Sent false
17952 6055 Acked Write 8 Done Done Sent false
17953 6056 Acked Read 6 Done Done Sent false
17954 6057 Acked Write 8 Done Done Sent false
17955 6058 Acked Read 8 Done Done Sent false
17956 6059 Acked Read 8 Done Done Sent false
17957 6060 Acked Read 8 Done Done Sent false
17958 6061 Acked Write 8 Done Done Sent false
17959 6062 Acked Write 6 Done Done Sent false
17960 6063 Acked Read 2 Done Done Sent false
17961 6064 Acked Read 1 Done Done Sent false
17962 6065 Acked Write 9 Done Done Sent false
17963 6066 Acked Read 10 Done Done Sent false
17964 6067 Acked Write 2 Done Done Sent false
17965 6068 Acked Read 1 Done Done Sent false
17966 6069 Acked Read 7 Done Done Sent false
17967 6070 Acked Write 4 Done Done Sent false
17968 6071 Acked Read 5 Done Done Sent false
17969 6072 Acked Write 10 Done Done Sent false
17970 6073 Acked Write 2 Done Done Sent false
17971 6074 Acked Read 3 Done Done Sent false
17972 6075 Acked Write 6 Done Done Sent false
17973 6076 Acked Write 1 Done Done Sent false
17974 6077 Acked Read 3 Done Done Sent false
17975 6078 Acked Write 4 Done Done Sent false
17976 6079 Acked Read 5 Done Done Sent false
17977 6080 Acked Write 8 Done Done Sent false
17978 6081 Acked Read 7 Done Done Sent false
17979 6082 Acked Read 3 Done Done Sent false
17980 6083 Acked Flush 0 Done Done Sent false
17981 6084 Acked Read 2 Done Done Sent false
17982 6085 Acked Write 8 Done Done Sent false
17983 6086 Acked Write 4 Done Done Sent false
17984 6087 Acked Read 1 Done Done Sent false
17985 6088 Acked Read 5 Done Done Sent false
17986 6089 Acked Write 4 Done Done Sent false
17987 6090 Acked Flush 0 Done Done Sent false
17988 6091 Acked Write 2 Done Done Sent false
17989 6092 Acked Flush 0 Done Done Sent false
17990 6093 Acked Write 8 Done Done Sent false
17991 6094 Acked Read 3 Done Done Sent false
17992 6095 Acked Read 10 Done Done Sent false
17993 6096 Acked Write 8 Done Done Sent false
17994 6097 Acked Read 5 Done Done Sent false
17995 6098 Acked Read 9 Done Done Sent false
17996 6099 Acked Read 6 Done Done Sent false
17997 6100 Acked Read 9 Done Done Sent false
17998 6101 Acked Write 7 Done Done Sent false
17999 6102 Acked Read 4 Done Done Sent false
18000 6103 Acked Read 10 Done Done Sent false
18001 6104 Acked Read 9 Done Done Sent false
18002 6105 Acked Read 8 Done Done Sent false
18003 6106 Acked Write 10 Done Done Sent false
18004 6107 Acked Write 8 Done Done Sent false
18005 6108 Acked Write 4 Done Done Sent false
18006 6109 Acked Read 4 Done Done Sent false
18007 6110 Acked Read 9 Done Done Sent false
18008 6111 Acked Read 1 Done Done Sent false
18009 6112 Acked Flush 0 Done Done Sent false
18010 6113 Acked Read 6 Done Done Sent false
18011 6114 Acked Flush 0 Done Done Sent false
18012 6115 Acked Read 1 Done Done Sent false
18013 6116 Acked Read 5 Done Done Sent false
18014 6117 Acked Write 3 Done Done Sent false
18015 6118 Acked Flush 0 Done Done Sent false
18016 6119 Acked Write 9 Done Done Sent false
18017 6120 Acked Read 2 Done Done Sent false
18018 6121 Acked Read 5 Done Done Sent false
18019 6122 Acked Read 5 Done Done Sent false
18020 6123 Acked Write 7 Done Done Sent false
18021 6124 Acked Read 9 Done Done Sent false
18022 6125 Acked Write 2 Done Done Sent false
18023 6126 Acked Write 5 Done Done Sent false
18024 6127 Acked Read 4 Done Done Sent false
18025 6128 Acked Read 5 Done Done Sent false
18026 6129 Acked Read 2 Done Done Sent false
18027 6130 Acked Write 4 Done Done Sent false
18028 6131 Acked Write 6 Done Done Sent false
18029 6132 Acked Read 5 Done Done Sent false
18030 6133 Acked Flush 0 Done Done Sent false
18031 STATES DS:0 DS:1 DS:2 TOTAL
18032 Sent 0 0 302 302
18033 Done 302 302 0 604
18034 Skipped 0 0 0 0
18035 Error 0 0 0 0
18036 Last Flush: 6133 6133 5831
18037 Downstairs last five completed: 5831 5830 5829 5828 5827
18038 Upstairs last five completed: 6133 6132 6131 6130 6129
18039 CLIENT: Up:0 ds:302 act:2
180402024-12-09T15:11:37.066ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
180412024-12-09T15:11:37.066ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
180422024-12-09T15:11:37.066ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
180432024-12-09T15:11:37.066ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
180442024-12-09T15:11:37.066ZINFOcrucible: client task is exiting = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
18045 [8830] got start action
18046 [8830] I am going to restart
18047 Starting downstairs at port 8830
18048 Make output file at "/tmp/dsc/downstairs-8830.txt"
18049 [8830][2] reports Starting
18050 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1218)
18051 [8830][2] reports Running
18052 ----------------------------------------------------------------
18053 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
18054 States: Active Active Offline
18055 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18056 5832 Acked Read 1 Done Done Sent false
18057 5833 Acked Read 8 Done Done Sent false
18058 5834 Acked Write 7 Done Done Sent false
18059 5835 Acked Read 6 Done Done Sent false
18060 5836 Acked Flush 0 Done Done Sent false
18061 5837 Acked Read 1 Done Done Sent false
18062 5838 Acked Write 1 Done Done Sent false
18063 5839 Acked Write 7 Done Done Sent false
18064 5840 Acked Read 2 Done Done Sent false
18065 5841 Acked Write 5 Done Done Sent false
18066 5842 Acked Read 4 Done Done Sent false
18067 5843 Acked Write 9 Done Done Sent false
18068 5844 Acked Read 5 Done Done Sent false
18069 5845 Acked Write 2 Done Done Sent false
18070 5846 Acked Read 7 Done Done Sent false
18071 5847 Acked Write 4 Done Done Sent false
18072 5848 Acked Flush 0 Done Done Sent false
18073 5849 Acked Read 2 Done Done Sent false
18074 5850 Acked Read 1 Done Done Sent false
18075 5851 Acked Read 3 Done Done Sent false
18076 5852 Acked Write 8 Done Done Sent false
18077 5853 Acked Read 8 Done Done Sent false
18078 5854 Acked Write 4 Done Done Sent false
18079 5855 Acked Read 8 Done Done Sent false
18080 5856 Acked Write 1 Done Done Sent false
18081 5857 Acked Read 3 Done Done Sent false
18082 5858 Acked Write 5 Done Done Sent false
18083 5859 Acked Write 4 Done Done Sent false
18084 5860 Acked Write 3 Done Done Sent false
18085 5861 Acked Write 7 Done Done Sent false
18086 5862 Acked Read 6 Done Done Sent false
18087 5863 Acked Write 2 Done Done Sent false
18088 5864 Acked Read 2 Done Done Sent false
18089 5865 Acked Write 7 Done Done Sent false
18090 5866 Acked Write 4 Done Done Sent false
18091 5867 Acked Read 8 Done Done Sent false
18092 5868 Acked Read 5 Done Done Sent false
18093 5869 Acked Read 4 Done Done Sent false
18094 5870 Acked Read 10 Done Done Sent false
18095 5871 Acked Write 3 Done Done Sent false
18096 5872 Acked Read 7 Done Done Sent false
18097 5873 Acked Read 4 Done Done Sent false
18098 5874 Acked Read 5 Done Done Sent false
18099 5875 Acked Write 4 Done Done Sent false
18100 5876 Acked Flush 0 Done Done Sent false
18101 5877 Acked Read 1 Done Done Sent false
18102 5878 Acked Write 9 Done Done Sent false
18103 5879 Acked Read 8 Done Done Sent false
18104 5880 Acked Read 4 Done Done Sent false
18105 5881 Acked Write 5 Done Done Sent false
18106 5882 Acked Read 2 Done Done Sent false
18107 5883 Acked Flush 0 Done Done Sent false
18108 5884 Acked Write 1 Done Done Sent false
18109 5885 Acked Read 1 Done Done Sent false
18110 5886 Acked Write 8 Done Done Sent false
18111 5887 Acked Write 10 Done Done Sent false
18112 5888 Acked Read 3 Done Done Sent false
18113 5889 Acked Read 10 Done Done Sent false
18114 5890 Acked Write 10 Done Done Sent false
18115 5891 Acked Write 7 Done Done Sent false
18116 5892 Acked Write 3 Done Done Sent false
18117 5893 Acked Read 8 Done Done Sent false
18118 5894 Acked Read 7 Done Done Sent false
18119 5895 Acked Write 8 Done Done Sent false
18120 5896 Acked Flush 0 Done Done Sent false
18121 5897 Acked Read 5 Done Done Sent false
18122 5898 Acked Read 1 Done Done Sent false
18123 5899 Acked Read 10 Done Done Sent false
18124 5900 Acked Read 7 Done Done Sent false
18125 5901 Acked Read 10 Done Done Sent false
18126 5902 Acked Read 8 Done Done Sent false
18127 5903 Acked Read 2 Done Done Sent false
18128 5904 Acked Write 3 Done Done Sent false
18129 5905 Acked Read 10 Done Done Sent false
18130 5906 Acked Read 8 Done Done Sent false
18131 5907 Acked Write 10 Done Done Sent false
18132 5908 Acked Write 3 Done Done Sent false
18133 5909 Acked Read 7 Done Done Sent false
18134 5910 Acked Write 10 Done Done Sent false
18135 5911 Acked Read 4 Done Done Sent false
18136 5912 Acked Write 8 Done Done Sent false
18137 5913 Acked Read 8 Done Done Sent false
18138 5914 Acked Read 1 Done Done Sent false
18139 5915 Acked Read 6 Done Done Sent false
18140 5916 Acked Write 2 Done Done Sent false
18141 5917 Acked Flush 0 Done Done Sent false
18142 5918 Acked Write 1 Done Done Sent false
18143 5919 Acked Read 10 Done Done Sent false
18144 5920 Acked Read 2 Done Done Sent false
18145 5921 Acked Write 1 Done Done Sent false
18146 5922 Acked Write 4 Done Done Sent false
18147 5923 Acked Flush 0 Done Done Sent false
18148 5924 Acked Write 5 Done Done Sent false
18149 5925 Acked Read 9 Done Done Sent false
18150 5926 Acked Write 9 Done Done Sent false
18151 5927 Acked Read 7 Done Done Sent false
18152 5928 Acked Read 4 Done Done Sent false
18153 5929 Acked Write 2 Done Done Sent false
18154 5930 Acked Flush 0 Done Done Sent false
18155 5931 Acked Flush 0 Done Done Sent false
18156 5932 Acked Read 7 Done Done Sent false
18157 5933 Acked Write 6 Done Done Sent false
18158 5934 Acked Flush 0 Done Done Sent false
18159 5935 Acked Write 8 Done Done Sent false
18160 5936 Acked Read 3 Done Done Sent false
18161 5937 Acked Write 3 Done Done Sent false
18162 5938 Acked Read 5 Done Done Sent false
18163 5939 Acked Write 7 Done Done Sent false
18164 5940 Acked Read 10 Done Done Sent false
18165 5941 Acked Read 1 Done Done Sent false
18166 5942 Acked Write 1 Done Done Sent false
18167 5943 Acked Read 7 Done Done Sent false
18168 5944 Acked Read 9 Done Done Sent false
18169 5945 Acked Write 8 Done Done Sent false
18170 5946 Acked Read 7 Done Done Sent false
18171 5947 Acked Read 6 Done Done Sent false
18172 5948 Acked Write 2 Done Done Sent false
18173 5949 Acked Flush 0 Done Done Sent false
18174 5950 Acked Read 2 Done Done Sent false
18175 5951 Acked Read 1 Done Done Sent false
18176 5952 Acked Read 1 Done Done Sent false
18177 5953 Acked Read 6 Done Done Sent false
18178 5954 Acked Read 10 Done Done Sent false
18179 5955 Acked Read 8 Done Done Sent false
18180 5956 Acked Write 5 Done Done Sent false
18181 5957 Acked Read 1 Done Done Sent false
18182 5958 Acked Write 8 Done Done Sent false
18183 5959 Acked Write 2 Done Done Sent false
18184 5960 Acked Write 3 Done Done Sent false
18185 5961 Acked Read 3 Done Done Sent false
18186 5962 Acked Write 9 Done Done Sent false
18187 5963 Acked Write 2 Done Done Sent false
18188 5964 Acked Read 8 Done Done Sent false
18189 5965 Acked Write 4 Done Done Sent false
18190 5966 Acked Read 10 Done Done Sent false
18191 5967 Acked Read 7 Done Done Sent false
18192 5968 Acked Read 2 Done Done Sent false
18193 5969 Acked Read 5 Done Done Sent false
18194 5970 Acked Read 2 Done Done Sent false
18195 5971 Acked Read 5 Done Done Sent false
18196 5972 Acked Read 6 Done Done Sent false
18197 5973 Acked Read 3 Done Done Sent false
18198 5974 Acked Write 4 Done Done Sent false
18199 5975 Acked Write 6 Done Done Sent false
18200 5976 Acked Write 2 Done Done Sent false
18201 5977 Acked Write 5 Done Done Sent false
18202 5978 Acked Read 5 Done Done Sent false
18203 5979 Acked Read 7 Done Done Sent false
18204 5980 Acked Read 4 Done Done Sent false
18205 5981 Acked Read 6 Done Done Sent false
18206 5982 Acked Read 4 Done Done Sent false
18207 5983 Acked Read 8 Done Done Sent false
18208 5984 Acked Write 6 Done Done Sent false
18209 5985 Acked Read 5 Done Done Sent false
18210 5986 Acked Write 7 Done Done Sent false
18211 5987 Acked Write 1 Done Done Sent false
18212 5988 Acked Write 3 Done Done Sent false
18213 5989 Acked Write 5 Done Done Sent false
18214 5990 Acked Flush 0 Done Done Sent false
18215 5991 Acked Read 7 Done Done Sent false
18216 5992 Acked Read 4 Done Done Sent false
18217 5993 Acked Read 9 Done Done Sent false
18218 5994 Acked Read 3 Done Done Sent false
18219 5995 Acked Read 9 Done Done Sent false
18220 5996 Acked Write 7 Done Done Sent false
18221 5997 Acked Flush 0 Done Done Sent false
18222 5998 Acked Write 5 Done Done Sent false
18223 5999 Acked Flush 0 Done Done Sent false
18224 6000 Acked Write 10 Done Done Sent false
18225 6001 Acked Read 8 Done Done Sent false
18226 6002 Acked Read 8 Done Done Sent false
18227 6003 Acked Flush 0 Done Done Sent false
18228 6004 Acked Write 10 Done Done Sent false
18229 6005 Acked Read 1 Done Done Sent false
18230 6006 Acked Read 7 Done Done Sent false
18231 6007 Acked Write 4 Done Done Sent false
18232 6008 Acked Write 2 Done Done Sent false
18233 6009 Acked Read 2 Done Done Sent false
18234 6010 Acked Write 9 Done Done Sent false
18235 6011 Acked Write 9 Done Done Sent false
18236 6012 Acked Read 9 Done Done Sent false
18237 6013 Acked Read 1 Done Done Sent false
18238 6014 Acked Read 1 Done Done Sent false
18239 6015 Acked Write 5 Done Done Sent false
18240 6016 Acked Read 7 Done Done Sent false
18241 6017 Acked Read 3 Done Done Sent false
18242 6018 Acked Read 2 Done Done Sent false
18243 6019 Acked Write 9 Done Done Sent false
18244 6020 Acked Read 9 Done Done Sent false
18245 6021 Acked Read 1 Done Done Sent false
18246 6022 Acked Read 2 Done Done Sent false
18247 6023 Acked Read 3 Done Done Sent false
18248 6024 Acked Write 3 Done Done Sent false
18249 6025 Acked Read 5 Done Done Sent false
18250 6026 Acked Write 5 Done Done Sent false
18251 6027 Acked Write 10 Done Done Sent false
18252 6028 Acked Flush 0 Done Done Sent false
18253 6029 Acked Write 10 Done Done Sent false
18254 6030 Acked Read 7 Done Done Sent false
18255 6031 Acked Read 7 Done Done Sent false
18256 6032 Acked Write 9 Done Done Sent false
18257 6033 Acked Read 2 Done Done Sent false
18258 6034 Acked Write 6 Done Done Sent false
18259 6035 Acked Read 3 Done Done Sent false
18260 6036 Acked Flush 0 Done Done Sent false
18261 6037 Acked Read 7 Done Done Sent false
18262 6038 Acked Write 10 Done Done Sent false
18263 6039 Acked Read 1 Done Done Sent false
18264 6040 Acked Read 10 Done Done Sent false
18265 6041 Acked Read 1 Done Done Sent false
18266 6042 Acked Write 6 Done Done Sent false
18267 6043 Acked Read 4 Done Done Sent false
18268 6044 Acked Write 10 Done Done Sent false
18269 6045 Acked Write 3 Done Done Sent false
18270 6046 Acked Read 4 Done Done Sent false
18271 6047 Acked Read 7 Done Done Sent false
18272 6048 Acked Read 4 Done Done Sent false
18273 6049 Acked Read 5 Done Done Sent false
18274 6050 Acked Write 9 Done Done Sent false
18275 6051 Acked Read 9 Done Done Sent false
18276 6052 Acked Read 4 Done Done Sent false
18277 6053 Acked Flush 0 Done Done Sent false
18278 6054 Acked Write 7 Done Done Sent false
18279 6055 Acked Write 8 Done Done Sent false
18280 6056 Acked Read 6 Done Done Sent false
18281 6057 Acked Write 8 Done Done Sent false
18282 6058 Acked Read 8 Done Done Sent false
18283 6059 Acked Read 8 Done Done Sent false
18284 6060 Acked Read 8 Done Done Sent false
18285 6061 Acked Write 8 Done Done Sent false
18286 6062 Acked Write 6 Done Done Sent false
18287 6063 Acked Read 2 Done Done Sent false
18288 6064 Acked Read 1 Done Done Sent false
18289 6065 Acked Write 9 Done Done Sent false
18290 6066 Acked Read 10 Done Done Sent false
18291 6067 Acked Write 2 Done Done Sent false
18292 6068 Acked Read 1 Done Done Sent false
18293 6069 Acked Read 7 Done Done Sent false
18294 6070 Acked Write 4 Done Done Sent false
18295 6071 Acked Read 5 Done Done Sent false
18296 6072 Acked Write 10 Done Done Sent false
18297 6073 Acked Write 2 Done Done Sent false
18298 6074 Acked Read 3 Done Done Sent false
18299 6075 Acked Write 6 Done Done Sent false
18300 6076 Acked Write 1 Done Done Sent false
18301 6077 Acked Read 3 Done Done Sent false
18302 6078 Acked Write 4 Done Done Sent false
18303 6079 Acked Read 5 Done Done Sent false
18304 6080 Acked Write 8 Done Done Sent false
18305 6081 Acked Read 7 Done Done Sent false
18306 6082 Acked Read 3 Done Done Sent false
18307 6083 Acked Flush 0 Done Done Sent false
18308 6084 Acked Read 2 Done Done Sent false
18309 6085 Acked Write 8 Done Done Sent false
18310 6086 Acked Write 4 Done Done Sent false
18311 6087 Acked Read 1 Done Done Sent false
18312 6088 Acked Read 5 Done Done Sent false
18313 6089 Acked Write 4 Done Done Sent false
18314 6090 Acked Flush 0 Done Done Sent false
18315 6091 Acked Write 2 Done Done Sent false
18316 6092 Acked Flush 0 Done Done Sent false
18317 6093 Acked Write 8 Done Done Sent false
18318 6094 Acked Read 3 Done Done Sent false
18319 6095 Acked Read 10 Done Done Sent false
18320 6096 Acked Write 8 Done Done Sent false
18321 6097 Acked Read 5 Done Done Sent false
18322 6098 Acked Read 9 Done Done Sent false
18323 6099 Acked Read 6 Done Done Sent false
18324 6100 Acked Read 9 Done Done Sent false
18325 6101 Acked Write 7 Done Done Sent false
18326 6102 Acked Read 4 Done Done Sent false
18327 6103 Acked Read 10 Done Done Sent false
18328 6104 Acked Read 9 Done Done Sent false
18329 6105 Acked Read 8 Done Done Sent false
18330 6106 Acked Write 10 Done Done Sent false
18331 6107 Acked Write 8 Done Done Sent false
18332 6108 Acked Write 4 Done Done Sent false
18333 6109 Acked Read 4 Done Done Sent false
18334 6110 Acked Read 9 Done Done Sent false
18335 6111 Acked Read 1 Done Done Sent false
18336 6112 Acked Flush 0 Done Done Sent false
18337 6113 Acked Read 6 Done Done Sent false
18338 6114 Acked Flush 0 Done Done Sent false
18339 6115 Acked Read 1 Done Done Sent false
18340 6116 Acked Read 5 Done Done Sent false
18341 6117 Acked Write 3 Done Done Sent false
18342 6118 Acked Flush 0 Done Done Sent false
18343 6119 Acked Write 9 Done Done Sent false
18344 6120 Acked Read 2 Done Done Sent false
18345 6121 Acked Read 5 Done Done Sent false
18346 6122 Acked Read 5 Done Done Sent false
18347 6123 Acked Write 7 Done Done Sent false
18348 6124 Acked Read 9 Done Done Sent false
18349 6125 Acked Write 2 Done Done Sent false
18350 6126 Acked Write 5 Done Done Sent false
18351 6127 Acked Read 4 Done Done Sent false
18352 6128 Acked Read 5 Done Done Sent false
18353 6129 Acked Read 2 Done Done Sent false
18354 6130 Acked Write 4 Done Done Sent false
18355 6131 Acked Write 6 Done Done Sent false
18356 6132 Acked Read 5 Done Done Sent false
18357 6133 Acked Flush 0 Done Done Sent false
18358 STATES DS:0 DS:1 DS:2 TOTAL
18359 Sent 0 0 302 302
18360 Done 302 302 0 604
18361 Skipped 0 0 0 0
18362 Error 0 0 0 0
18363 Last Flush: 6133 6133 5831
18364 Downstairs last five completed: 5831 5830 5829 5828 5827
18365 Upstairs last five completed: 6133 6132 6131 6130 6129
18366 CLIENT: Up:0 ds:302 act:2
18367 ----------------------------------------------------------------
18368 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
18369 States: Active Active Offline
18370 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18371 5832 Acked Read 1 Done Done Sent false
18372 5833 Acked Read 8 Done Done Sent false
18373 5834 Acked Write 7 Done Done Sent false
18374 5835 Acked Read 6 Done Done Sent false
18375 5836 Acked Flush 0 Done Done Sent false
18376 5837 Acked Read 1 Done Done Sent false
18377 5838 Acked Write 1 Done Done Sent false
18378 5839 Acked Write 7 Done Done Sent false
18379 5840 Acked Read 2 Done Done Sent false
18380 5841 Acked Write 5 Done Done Sent false
18381 5842 Acked Read 4 Done Done Sent false
18382 5843 Acked Write 9 Done Done Sent false
18383 5844 Acked Read 5 Done Done Sent false
18384 5845 Acked Write 2 Done Done Sent false
18385 5846 Acked Read 7 Done Done Sent false
18386 5847 Acked Write 4 Done Done Sent false
18387 5848 Acked Flush 0 Done Done Sent false
18388 5849 Acked Read 2 Done Done Sent false
18389 5850 Acked Read 1 Done Done Sent false
18390 5851 Acked Read 3 Done Done Sent false
18391 5852 Acked Write 8 Done Done Sent false
18392 5853 Acked Read 8 Done Done Sent false
18393 5854 Acked Write 4 Done Done Sent false
18394 5855 Acked Read 8 Done Done Sent false
18395 5856 Acked Write 1 Done Done Sent false
18396 5857 Acked Read 3 Done Done Sent false
18397 5858 Acked Write 5 Done Done Sent false
18398 5859 Acked Write 4 Done Done Sent false
18399 5860 Acked Write 3 Done Done Sent false
18400 5861 Acked Write 7 Done Done Sent false
18401 5862 Acked Read 6 Done Done Sent false
18402 5863 Acked Write 2 Done Done Sent false
18403 5864 Acked Read 2 Done Done Sent false
18404 5865 Acked Write 7 Done Done Sent false
18405 5866 Acked Write 4 Done Done Sent false
18406 5867 Acked Read 8 Done Done Sent false
18407 5868 Acked Read 5 Done Done Sent false
18408 5869 Acked Read 4 Done Done Sent false
18409 5870 Acked Read 10 Done Done Sent false
18410 5871 Acked Write 3 Done Done Sent false
18411 5872 Acked Read 7 Done Done Sent false
18412 5873 Acked Read 4 Done Done Sent false
18413 5874 Acked Read 5 Done Done Sent false
18414 5875 Acked Write 4 Done Done Sent false
18415 5876 Acked Flush 0 Done Done Sent false
18416 5877 Acked Read 1 Done Done Sent false
18417 5878 Acked Write 9 Done Done Sent false
18418 5879 Acked Read 8 Done Done Sent false
18419 5880 Acked Read 4 Done Done Sent false
18420 5881 Acked Write 5 Done Done Sent false
18421 5882 Acked Read 2 Done Done Sent false
18422 5883 Acked Flush 0 Done Done Sent false
18423 5884 Acked Write 1 Done Done Sent false
18424 5885 Acked Read 1 Done Done Sent false
18425 5886 Acked Write 8 Done Done Sent false
18426 5887 Acked Write 10 Done Done Sent false
18427 5888 Acked Read 3 Done Done Sent false
18428 5889 Acked Read 10 Done Done Sent false
18429 5890 Acked Write 10 Done Done Sent false
18430 5891 Acked Write 7 Done Done Sent false
18431 5892 Acked Write 3 Done Done Sent false
18432 5893 Acked Read 8 Done Done Sent false
18433 5894 Acked Read 7 Done Done Sent false
18434 5895 Acked Write 8 Done Done Sent false
18435 5896 Acked Flush 0 Done Done Sent false
18436 5897 Acked Read 5 Done Done Sent false
18437 5898 Acked Read 1 Done Done Sent false
18438 5899 Acked Read 10 Done Done Sent false
18439 5900 Acked Read 7 Done Done Sent false
18440 5901 Acked Read 10 Done Done Sent false
18441 5902 Acked Read 8 Done Done Sent false
18442 5903 Acked Read 2 Done Done Sent false
18443 5904 Acked Write 3 Done Done Sent false
18444 5905 Acked Read 10 Done Done Sent false
18445 5906 Acked Read 8 Done Done Sent false
18446 5907 Acked Write 10 Done Done Sent false
18447 5908 Acked Write 3 Done Done Sent false
18448 5909 Acked Read 7 Done Done Sent false
18449 5910 Acked Write 10 Done Done Sent false
18450 5911 Acked Read 4 Done Done Sent false
18451 5912 Acked Write 8 Done Done Sent false
18452 5913 Acked Read 8 Done Done Sent false
18453 5914 Acked Read 1 Done Done Sent false
18454 5915 Acked Read 6 Done Done Sent false
18455 5916 Acked Write 2 Done Done Sent false
18456 5917 Acked Flush 0 Done Done Sent false
18457 5918 Acked Write 1 Done Done Sent false
18458 5919 Acked Read 10 Done Done Sent false
18459 5920 Acked Read 2 Done Done Sent false
18460 5921 Acked Write 1 Done Done Sent false
18461 5922 Acked Write 4 Done Done Sent false
18462 5923 Acked Flush 0 Done Done Sent false
18463 5924 Acked Write 5 Done Done Sent false
18464 5925 Acked Read 9 Done Done Sent false
18465 5926 Acked Write 9 Done Done Sent false
18466 5927 Acked Read 7 Done Done Sent false
18467 5928 Acked Read 4 Done Done Sent false
18468 5929 Acked Write 2 Done Done Sent false
18469 5930 Acked Flush 0 Done Done Sent false
18470 5931 Acked Flush 0 Done Done Sent false
18471 5932 Acked Read 7 Done Done Sent false
18472 5933 Acked Write 6 Done Done Sent false
18473 5934 Acked Flush 0 Done Done Sent false
18474 5935 Acked Write 8 Done Done Sent false
18475 5936 Acked Read 3 Done Done Sent false
18476 5937 Acked Write 3 Done Done Sent false
18477 5938 Acked Read 5 Done Done Sent false
18478 5939 Acked Write 7 Done Done Sent false
18479 5940 Acked Read 10 Done Done Sent false
18480 5941 Acked Read 1 Done Done Sent false
18481 5942 Acked Write 1 Done Done Sent false
18482 5943 Acked Read 7 Done Done Sent false
18483 5944 Acked Read 9 Done Done Sent false
18484 5945 Acked Write 8 Done Done Sent false
18485 5946 Acked Read 7 Done Done Sent false
18486 5947 Acked Read 6 Done Done Sent false
18487 5948 Acked Write 2 Done Done Sent false
18488 5949 Acked Flush 0 Done Done Sent false
18489 5950 Acked Read 2 Done Done Sent false
18490 5951 Acked Read 1 Done Done Sent false
18491 5952 Acked Read 1 Done Done Sent false
18492 5953 Acked Read 6 Done Done Sent false
18493 5954 Acked Read 10 Done Done Sent false
18494 5955 Acked Read 8 Done Done Sent false
18495 5956 Acked Write 5 Done Done Sent false
18496 5957 Acked Read 1 Done Done Sent false
18497 5958 Acked Write 8 Done Done Sent false
18498 5959 Acked Write 2 Done Done Sent false
18499 5960 Acked Write 3 Done Done Sent false
18500 5961 Acked Read 3 Done Done Sent false
18501 5962 Acked Write 9 Done Done Sent false
18502 5963 Acked Write 2 Done Done Sent false
18503 5964 Acked Read 8 Done Done Sent false
18504 5965 Acked Write 4 Done Done Sent false
18505 5966 Acked Read 10 Done Done Sent false
18506 5967 Acked Read 7 Done Done Sent false
18507 5968 Acked Read 2 Done Done Sent false
18508 5969 Acked Read 5 Done Done Sent false
18509 5970 Acked Read 2 Done Done Sent false
18510 5971 Acked Read 5 Done Done Sent false
18511 5972 Acked Read 6 Done Done Sent false
18512 5973 Acked Read 3 Done Done Sent false
18513 5974 Acked Write 4 Done Done Sent false
18514 5975 Acked Write 6 Done Done Sent false
18515 5976 Acked Write 2 Done Done Sent false
18516 5977 Acked Write 5 Done Done Sent false
18517 5978 Acked Read 5 Done Done Sent false
18518 5979 Acked Read 7 Done Done Sent false
18519 5980 Acked Read 4 Done Done Sent false
18520 5981 Acked Read 6 Done Done Sent false
18521 5982 Acked Read 4 Done Done Sent false
18522 5983 Acked Read 8 Done Done Sent false
18523 5984 Acked Write 6 Done Done Sent false
18524 5985 Acked Read 5 Done Done Sent false
18525 5986 Acked Write 7 Done Done Sent false
18526 5987 Acked Write 1 Done Done Sent false
18527 5988 Acked Write 3 Done Done Sent false
18528 5989 Acked Write 5 Done Done Sent false
18529 5990 Acked Flush 0 Done Done Sent false
18530 5991 Acked Read 7 Done Done Sent false
18531 5992 Acked Read 4 Done Done Sent false
18532 5993 Acked Read 9 Done Done Sent false
18533 5994 Acked Read 3 Done Done Sent false
18534 5995 Acked Read 9 Done Done Sent false
18535 5996 Acked Write 7 Done Done Sent false
18536 5997 Acked Flush 0 Done Done Sent false
18537 5998 Acked Write 5 Done Done Sent false
18538 5999 Acked Flush 0 Done Done Sent false
18539 6000 Acked Write 10 Done Done Sent false
18540 6001 Acked Read 8 Done Done Sent false
18541 6002 Acked Read 8 Done Done Sent false
18542 6003 Acked Flush 0 Done Done Sent false
18543 6004 Acked Write 10 Done Done Sent false
18544 6005 Acked Read 1 Done Done Sent false
18545 6006 Acked Read 7 Done Done Sent false
18546 6007 Acked Write 4 Done Done Sent false
18547 6008 Acked Write 2 Done Done Sent false
18548 6009 Acked Read 2 Done Done Sent false
18549 6010 Acked Write 9 Done Done Sent false
18550 6011 Acked Write 9 Done Done Sent false
18551 6012 Acked Read 9 Done Done Sent false
18552 6013 Acked Read 1 Done Done Sent false
18553 6014 Acked Read 1 Done Done Sent false
18554 6015 Acked Write 5 Done Done Sent false
18555 6016 Acked Read 7 Done Done Sent false
18556 6017 Acked Read 3 Done Done Sent false
18557 6018 Acked Read 2 Done Done Sent false
18558 6019 Acked Write 9 Done Done Sent false
18559 6020 Acked Read 9 Done Done Sent false
18560 6021 Acked Read 1 Done Done Sent false
18561 6022 Acked Read 2 Done Done Sent false
18562 6023 Acked Read 3 Done Done Sent false
18563 6024 Acked Write 3 Done Done Sent false
18564 6025 Acked Read 5 Done Done Sent false
18565 6026 Acked Write 5 Done Done Sent false
18566 6027 Acked Write 10 Done Done Sent false
18567 6028 Acked Flush 0 Done Done Sent false
18568 6029 Acked Write 10 Done Done Sent false
18569 6030 Acked Read 7 Done Done Sent false
18570 6031 Acked Read 7 Done Done Sent false
18571 6032 Acked Write 9 Done Done Sent false
18572 6033 Acked Read 2 Done Done Sent false
18573 6034 Acked Write 6 Done Done Sent false
18574 6035 Acked Read 3 Done Done Sent false
18575 6036 Acked Flush 0 Done Done Sent false
18576 6037 Acked Read 7 Done Done Sent false
18577 6038 Acked Write 10 Done Done Sent false
18578 6039 Acked Read 1 Done Done Sent false
18579 6040 Acked Read 10 Done Done Sent false
18580 6041 Acked Read 1 Done Done Sent false
18581 6042 Acked Write 6 Done Done Sent false
18582 6043 Acked Read 4 Done Done Sent false
18583 6044 Acked Write 10 Done Done Sent false
18584 6045 Acked Write 3 Done Done Sent false
18585 6046 Acked Read 4 Done Done Sent false
18586 6047 Acked Read 7 Done Done Sent false
18587 6048 Acked Read 4 Done Done Sent false
18588 6049 Acked Read 5 Done Done Sent false
18589 6050 Acked Write 9 Done Done Sent false
18590 6051 Acked Read 9 Done Done Sent false
18591 6052 Acked Read 4 Done Done Sent false
18592 6053 Acked Flush 0 Done Done Sent false
18593 6054 Acked Write 7 Done Done Sent false
18594 6055 Acked Write 8 Done Done Sent false
18595 6056 Acked Read 6 Done Done Sent false
18596 6057 Acked Write 8 Done Done Sent false
18597 6058 Acked Read 8 Done Done Sent false
18598 6059 Acked Read 8 Done Done Sent false
18599 6060 Acked Read 8 Done Done Sent false
18600 6061 Acked Write 8 Done Done Sent false
18601 6062 Acked Write 6 Done Done Sent false
18602 6063 Acked Read 2 Done Done Sent false
18603 6064 Acked Read 1 Done Done Sent false
18604 6065 Acked Write 9 Done Done Sent false
18605 6066 Acked Read 10 Done Done Sent false
18606 6067 Acked Write 2 Done Done Sent false
18607 6068 Acked Read 1 Done Done Sent false
18608 6069 Acked Read 7 Done Done Sent false
18609 6070 Acked Write 4 Done Done Sent false
18610 6071 Acked Read 5 Done Done Sent false
18611 6072 Acked Write 10 Done Done Sent false
18612 6073 Acked Write 2 Done Done Sent false
18613 6074 Acked Read 3 Done Done Sent false
18614 6075 Acked Write 6 Done Done Sent false
18615 6076 Acked Write 1 Done Done Sent false
18616 6077 Acked Read 3 Done Done Sent false
18617 6078 Acked Write 4 Done Done Sent false
18618 6079 Acked Read 5 Done Done Sent false
18619 6080 Acked Write 8 Done Done Sent false
18620 6081 Acked Read 7 Done Done Sent false
18621 6082 Acked Read 3 Done Done Sent false
18622 6083 Acked Flush 0 Done Done Sent false
18623 6084 Acked Read 2 Done Done Sent false
18624 6085 Acked Write 8 Done Done Sent false
18625 6086 Acked Write 4 Done Done Sent false
18626 6087 Acked Read 1 Done Done Sent false
18627 6088 Acked Read 5 Done Done Sent false
18628 6089 Acked Write 4 Done Done Sent false
18629 6090 Acked Flush 0 Done Done Sent false
18630 6091 Acked Write 2 Done Done Sent false
18631 6092 Acked Flush 0 Done Done Sent false
18632 6093 Acked Write 8 Done Done Sent false
18633 6094 Acked Read 3 Done Done Sent false
18634 6095 Acked Read 10 Done Done Sent false
18635 6096 Acked Write 8 Done Done Sent false
18636 6097 Acked Read 5 Done Done Sent false
18637 6098 Acked Read 9 Done Done Sent false
18638 6099 Acked Read 6 Done Done Sent false
18639 6100 Acked Read 9 Done Done Sent false
18640 6101 Acked Write 7 Done Done Sent false
18641 6102 Acked Read 4 Done Done Sent false
18642 6103 Acked Read 10 Done Done Sent false
18643 6104 Acked Read 9 Done Done Sent false
18644 6105 Acked Read 8 Done Done Sent false
18645 6106 Acked Write 10 Done Done Sent false
18646 6107 Acked Write 8 Done Done Sent false
18647 6108 Acked Write 4 Done Done Sent false
18648 6109 Acked Read 4 Done Done Sent false
18649 6110 Acked Read 9 Done Done Sent false
18650 6111 Acked Read 1 Done Done Sent false
18651 6112 Acked Flush 0 Done Done Sent false
18652 6113 Acked Read 6 Done Done Sent false
18653 6114 Acked Flush 0 Done Done Sent false
18654 6115 Acked Read 1 Done Done Sent false
18655 6116 Acked Read 5 Done Done Sent false
18656 6117 Acked Write 3 Done Done Sent false
18657 6118 Acked Flush 0 Done Done Sent false
18658 6119 Acked Write 9 Done Done Sent false
18659 6120 Acked Read 2 Done Done Sent false
18660 6121 Acked Read 5 Done Done Sent false
18661 6122 Acked Read 5 Done Done Sent false
18662 6123 Acked Write 7 Done Done Sent false
18663 6124 Acked Read 9 Done Done Sent false
18664 6125 Acked Write 2 Done Done Sent false
18665 6126 Acked Write 5 Done Done Sent false
18666 6127 Acked Read 4 Done Done Sent false
18667 6128 Acked Read 5 Done Done Sent false
18668 6129 Acked Read 2 Done Done Sent false
18669 6130 Acked Write 4 Done Done Sent false
18670 6131 Acked Write 6 Done Done Sent false
18671 6132 Acked Read 5 Done Done Sent false
18672 6133 Acked Flush 0 Done Done Sent false
18673 STATES DS:0 DS:1 DS:2 TOTAL
18674 Sent 0 0 302 302
18675 Done 302 302 0 604
18676 Skipped 0 0 0 0
18677 Error 0 0 0 0
18678 Last Flush: 6133 6133 5831
18679 Downstairs last five completed: 5831 5830 5829 5828 5827
18680 Upstairs last five completed: 6133 6132 6131 6130 6129
18681 CLIENT: Up:0 ds:302 act:2
186822024-12-09T15:11:47.067ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
186832024-12-09T15:11:47.067ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64500) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
186842024-12-09T15:11:47.068ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
186852024-12-09T15:11:47.068ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
186862024-12-09T15:11:47.068ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
186872024-12-09T15:11:47.068ZINFOcrucible: send last flush ID to this DS: 5831 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
186882024-12-09T15:11:47.069ZINFOcrucible: Replied this last flush ID: 5831 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
186892024-12-09T15:11:47.069ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
186902024-12-09T15:11:47.069ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
186912024-12-09T15:11:47.069ZINFOcrucible: [2] client re-new 302 jobs since flush 5831 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
186922024-12-09T15:11:47.069ZINFOcrucible: [2] Replayed 302 jobs since flush: 5831 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
18693 ----------------------------------------------------------------
18694 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18695 Downstairs last five completed: 6133 6132 6131 6130 6129
18696 Upstairs last five completed: 6133 6132 6131 6130 6129
18697 CLIENT: Up:0 ds:0 act:3
18698 Replay: All jobs finished
18699 Main task has work to do, go find it
18700 got dsc Stop(0)
18701 stop 0
18702 Dec 09 15:11:47.320 [8810] Got stop action so:false kr:false
18703 INFO request completed, latency_us: 350, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 7d20a2df-d71e-4656-a65f-4957090e95c1, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
187042024-12-09T15:11:47.321ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
187052024-12-09T15:11:47.321ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
18706 Dec 09 15:11:47.321 INFO request completed, latency_us: 195, response_code: 200, uri: /state/cid/0, method:{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:11:47.321316361Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
18707 GET, req_id: 4ad4ff43-4125-4570-914f-2afcb401c807, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
187082024-12-09T15:11:47.321ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
187092024-12-09T15:11:47.321ZINFOcrucible: [0] Transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
187102024-12-09T15:11:47.321ZINFOcrucible: client task is exiting = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
18711 [8810] Exited with: ExitStatus(unix_wait_status(9))
18712 [8810][0] reports Exit
18713 Dec 09 15:11:51.323 INFO request completed, latency_us: 230, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1c2be288-87d9-4ede-9670-8d5f9a9e1601, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
18714 001/300 Write block 3963 len 24576 data: 3 3 4 4 3 3
18715 002/300 Write block 3924 len 28672 data: 4 4 4 4 5 6 7
18716 003/300 Read block 2597 len 24576
18717 004/300 Read block 890 len 20480
18718 005/300 Read block 2647 len 16384
18719 006/300 Write block 4965 len 20480 data: 4 3 3 3 3
18720 007/300 Write block 2706 len 40960 data: 4 3 2 3 3 7 7 8 6 7
18721 008/300 Read block 4618 len 36864
18722 009/300 Read block 931 len 16384
18723 010/300 Write block 4128 len 32768 data: 3 3 3 4 3 4 2 3
18724 011/300 Read block 4295 len 4096
18725 012/300 Write block 390 len 4096 data: 3
18726 013/300 Read block 414 len 32768
18727 014/300 Read block 4647 len 24576
18728 015/300 Write block 3864 len 40960 data: 2 3 3 3 4 4 6 5 5 5
18729 016/300 Read block 487 len 28672
18730 017/300 Read block 2537 len 20480
18731 018/300 Write block 4045 len 40960 data: 2 4 4 4 4 4 5 6 4 3
18732 019/300 Flush
18733 020/300 Write block 2175 len 32768 data: 6 6 5 5 6 6 5 6
18734 021/300 Flush
18735 022/300 Write block 4814 len 36864 data: 7 4 3 3 3 2 2 2 2
18736 023/300 Read block 2534 len 32768
18737 024/300 Read block 1728 len 12288
18738 025/300 Write block 1989 len 16384 data: 3 3 3 3
18739 026/300 Read block 2170 len 40960
18740 027/300 Write block 4813 len 28672 data: 6 8 5 4 4 4 3
18741 028/300 Write block 891 len 4096 data: 2
18742 029/300 Read block 3046 len 8192
18743 030/300 Read block 1731 len 8192
18744 031/300 Read block 4496 len 16384
18745 032/300 Read block 2539 len 24576
18746 033/300 Write block 1228 len 8192 data: 5 6
18747 034/300 Read block 2398 len 20480
18748 035/300 Read block 2761 len 20480
18749 036/300 Write block 2926 len 40960 data: 5 7 8 7 8 9 8 8 6 6
18750 037/300 Read block 1004 len 36864
18751 038/300 Read block 4846 len 24576
18752 039/300 Read block 1072 len 28672
18753 040/300 Write block 609 len 4096 data: 6
18754 041/300 Read block 112 len 32768
18755 042/300 Read block 118 len 24576
18756 043/300 Write block 959 len 24576 data: 3 4 4 3 3 3
18757 044/300 Flush
18758 045/300 Read block 4169 len 4096
18759 046/300 Write block 1936 len 4096 data: 4
18760 047/300 Read block 1928 len 40960
18761 048/300 Write block 3794 len 40960 data: 3 3 2 3 5 6 7 7 8 6
18762 049/300 Flush
18763 050/300 Read block 2577 len 20480
18764 051/300 Read block 3162 len 12288
18765 052/300 Read block 1313 len 32768
18766 053/300 Read block 3227 len 8192
18767 054/300 Write block 4271 len 12288 data: 2 2 2
18768 055/300 Write block 4180 len 28672 data: 5 5 6 6 6 5 5
18769 056/300 Read block 3175 len 12288
18770 057/300 Write block 3142 len 12288 data: 5 5 8
18771 058/300 Write block 501 len 16384 data: 4 4 5 6
18772 059/300 Read block 4522 len 36864
18773 060/300 Write block 3121 len 24576 data: 6 6 6 4 3 2
18774 061/300 Write block 963 len 4096 data: 4
18775 062/300 Read block 1644 len 4096
18776 063/300 Read block 4961 len 20480
18777 064/300 Write block 1047 len 8192 data: 3 3
18778 065/300 Write block 4296 len 8192 data: 4 3
18779 066/300 Write block 1543 len 36864 data: 4 4 4 4 5 4 2 2 3
18780 067/300 Flush
18781 068/300 Write block 1286 len 32768 data: 3 3 2 3 4 7 7 7
18782 069/300 Read block 1871 len 8192
18783 070/300 Read block 1785 len 16384
18784 071/300 Flush
18785 072/300 Flush
18786 073/300 Write block 3916 len 20480 data: 3 3 3 2 2
18787 074/300 Read block 3592 len 4096
18788 075/300 Read block 425 len 24576
18789 076/300 Flush
18790 077/300 Read block 3627 len 4096
18791 078/300 Read block 726 len 40960
18792 079/300 Write block 717 len 20480 data: 3 3 4 4 4
18793 080/300 Read block 1552 len 24576
18794 081/300 Write block 1644 len 28672 data: 3 3 3 3 4 3 4
18795 082/300 Read block 4855 len 40960
18796 083/300 Read block 2190 len 20480
18797 084/300 Read block 3013 len 32768
18798 085/300 Read block 2804 len 12288
18799 086/300 Write block 1130 len 4096 data: 4
18800 087/300 Write block 4929 len 40960 data: 3 5 4 6 5 6 6 5 3 4
18801 088/300 Read block 3864 len 8192
18802 089/300 Write block 3410 len 28672 data: 4 5 4 4 5 4 6
18803 090/300 Write block 2611 len 8192 data: 4 5
18804 091/300 Write block 3017 len 8192 data: 6 6
18805 092/300 Write block 2623 len 8192 data: 6 5
18806 093/300 Read block 4015 len 4096
18807 094/300 Read block 1464 len 8192
18808 095/300 Write block 3180 len 40960 data: 5 5 5 5 5 4 3 3 3 3
18809 096/300 Read block 115 len 36864
18810 097/300 Read block 2826 len 28672
18811 098/300 Read block 4212 len 20480
18812 099/300 Write block 313 len 40960 data: 2 3 4 5 5 6 5 6 5 5
18813 100/300 Read block 4473 len 16384
18814 101/300 Write block 3591 len 4096 data: 4
18815 102/300 Write block 688 len 16384 data: 7 6 5 4
18816 103/300 Read block 979 len 32768
18817 104/300 Flush
18818 105/300 Write block 202 len 4096 data: 2
18819 106/300 Read block 1859 len 24576
18820 107/300 Write block 2963 len 24576 data: 3 3 2 2 2 2
18821 108/300 Flush
18822 109/300 Read block 4673 len 24576
18823 110/300 Write block 4818 len 8192 data: 5 4
18824 111/300 Write block 1511 len 32768 data: 4 4 5 5 4 5 5 6
18825 112/300 Read block 3286 len 40960
18826 113/300 Read block 577 len 20480
18827 114/300 Read block 1140 len 28672
18828 115/300 Read block 3438 len 40960
18829 116/300 Flush
18830 117/300 Write block 2863 len 28672 data: 3 3 4 5 7 7 7
18831 118/300 Read block 4126 len 16384
18832 119/300 Write block 1944 len 20480 data: 2 2 2 2 2
18833 120/300 Write block 4044 len 32768 data: 2 3 5 5 5 5 5 6
18834 121/300 Write block 2526 len 20480 data: 6 4 4 5 6
18835 122/300 Write block 556 len 40960 data: 4 3 3 2 2 2 2 2 2 3
18836 123/300 Read block 1821 len 36864
18837 124/300 Read block 1094 len 24576
18838 125/300 Read block 2457 len 16384
18839 126/300 Read block 4759 len 8192
18840 127/300 Read block 3612 len 16384
18841 128/300 Write block 213 len 32768 data: 8 8 7 7 7 7 7 5
18842 129/300 Flush
18843 130/300 Read block 3279 len 4096
18844 131/300 Write block 4920 len 16384 data: 3 3 3 2
18845 132/300 Read block 494 len 36864
18846 133/300 Read block 112 len 24576
18847 134/300 Read block 777 len 16384
18848 135/300 Read block 1951 len 16384
18849 136/300 Flush
18850 137/300 Write block 1093 len 8192 data: 6 7
18851 138/300 Write block 2800 len 12288 data: 4 4 3
18852 139/300 Write block 3235 len 28672 data: 3 3 4 4 4 4 2
18853 140/300 Write block 265 len 40960 data: 2 2 3 4 5 6 6 6 4 4
18854 141/300 Read block 2235 len 4096
18855 142/300 Read block 2407 len 20480
18856 143/300 Write block 1067 len 28672 data: 2 3 4 4 4 4 5
18857 144/300 Flush
18858 145/300 Write block 1188 len 36864 data: 3 4 4 4 4 4 5 4 5
18859 146/300 Read block 4270 len 36864
18860 147/300 Write block 4924 len 40960 data: 3 3 2 3 3 4 6 5 7 6
18861 148/300 Write block 4261 len 28672 data: 5 4 4 4 3 3 3
18862 149/300 Read block 3443 len 40960
18863 150/300 Read block 3804 len 24576
18864 151/300 Write block 3416 len 40960 data: 7 5 5 5 5 5 6 6 5 6
18865 152/300 Read block 3676 len 12288
18866 153/300 Write block 3152 len 16384 data: 8 7 7 6
18867 154/300 Read block 3455 len 28672
18868 155/300 Read block 3245 len 8192
18869 156/300 Read block 4459 len 36864
18870 157/300 Write block 3662 len 32768 data: 3 3 3 3 3 3 3 3
18871 158/300 Write block 1661 len 40960 data: 5 5 4 6 6 5 4 4 4 3
18872 159/300 Read block 4021 len 32768
18873 160/300 Read block 3287 len 28672
18874 161/300 Write block 2367 len 28672 data: 3 3 4 3 2 2 3
18875 162/300 Read block 669 len 36864
18876 163/300 Read block 2351 len 40960
18877 164/300 Read block 1899 len 16384
18878 165/300 Read block 3787 len 12288
18879 166/300 Write block 1542 len 16384 data: 3 5 5 5
18880 167/300 Write block 3396 len 12288 data: 6 6 7
18881 168/300 Write block 4403 len 28672 data: 3 3 4 6 6 6 6
18882 169/300 Flush
18883 170/300 Read block 3189 len 4096
18884 171/300 Read block 4452 len 4096
18885 172/300 Read block 3212 len 16384
18886 173/300 Flush
18887 174/300 Read block 4730 len 16384
18888 175/300 Write block 4105 len 8192 data: 7 6
18889 176/300 Read block 171 len 36864
18890 177/300 Write block 851 len 12288 data: 3 3 4
18891 178/300 Read block 1452 len 20480
18892 179/300 Write block 2654 len 12288 data: 6 7 7
18893 180/300 Read block 3280 len 8192
18894 181/300 Read block 3755 len 36864
18895 182/300 Read block 2623 len 8192
18896 183/300 Read block 2723 len 40960
18897 184/300 Flush
18898 185/300 Read block 1683 len 16384
18899 186/300 Flush
18900 187/300 Flush
18901 188/300 Flush
18902 189/300 Write block 3583 len 16384 data: 7 8 6 6
18903 190/300 Flush
18904 191/300 Read block 267 len 40960
18905 192/300 Read block 4256 len 40960
18906 193/300 Read block 4519 len 24576
18907 194/300 Write block 2327 len 36864 data: 2 2 2 2 3 3 5 6 7
18908 195/300 Write block 2630 len 36864 data: 3 3 3 3 2 3 3 2 2
18909 196/300 Read block 4553 len 8192
18910 197/300 Write block 1566 len 4096 data: 6
18911 198/300 Flush
18912 199/300 Read block 1801 len 40960
18913 200/300 Read block 3022 len 20480
18914 201/300 Read block 4624 len 36864
18915 202/300 Write block 1664 len 24576 data: 7 7 6 5 5 5
18916 203/300 Read block 3921 len 20480
18917 204/300 Write block 1106 len 28672 data: 6 6 7 7 6 6 5
18918 205/300 Flush
18919 206/300 Read block 4779 len 36864
18920 207/300 Write block 788 len 12288 data: 3 3 3
18921 208/300 Write block 4107 len 40960 data: 5 4 4 2 3 3 3 3 3 5
18922 209/300 Read block 4972 len 28672
18923 210/300 Read block 4844 len 28672
18924 211/300 Read block 4684 len 20480
18925 212/300 Write block 572 len 32768 data: 4 5 5 6 6 6 7 6
18926 213/300 Write block 475 len 12288 data: 4 4 5
18927 214/300 Read block 1285 len 20480
18928 215/300 Write block 2768 len 28672 data: 5 5 5 7 8 9 11
18929 216/300 Flush
18930 217/300 Read block 57 len 8192
18931 218/300 Read block 334 len 32768
18932 219/300 Write block 2434 len 12288 data: 5 5 5
18933 220/300 Write block 2112 len 20480 data: 5 5 4 3 4
18934 221/300 Read block 4016 len 40960
18935 222/300 Read block 1782 len 24576
18936 223/300 Flush
18937 224/300 Write block 4468 len 12288 data: 4 5 5
18938 225/300 Write block 3889 len 40960 data: 4 2 2 2 2 3 3 3 3 3
18939 226/300 Read block 1630 len 20480
18940 227/300 Flush
18941 228/300 Read block 3771 len 4096
18942 229/300 Flush
18943 230/300 Read block 923 len 8192
18944 231/300 Write block 3830 len 8192 data: 5 5
18945 232/300 Read block 3711 len 4096
18946 233/300 Write block 549 len 8192 data: 9 10
18947 234/300 Read block 4309 len 28672
18948 235/300 Write block 2611 len 12288 data: 5 6 5
18949 236/300 Read block 2451 len 24576
18950 237/300 Read block 4276 len 28672
18951 238/300 Write block 1399 len 8192 data: 3 4
18952 239/300 Write block 4757 len 28672 data: 2 2 4 4 4 4 4
18953 240/300 Write block 1077 len 16384 data: 6 6 7 5
18954 241/300 Read block 2902 len 20480
18955 242/300 Write block 2509 len 16384 data: 5 5 5 5
18956 243/300 Flush
18957 244/300 Write block 911 len 32768 data: 5 5 5 5 5 5 4 4
18958 245/300 Read block 2140 len 20480
18959 246/300 Write block 3839 len 28672 data: 3 3 3 4 6 7 7
18960 247/300 Read block 1771 len 24576
18961 248/300 Read block 3161 len 20480
18962 249/300 Write block 733 len 20480 data: 5 4 3 2 2
18963 250/300 Write block 3444 len 4096 data: 4
18964 251/300 Read block 1501 len 36864
18965 252/300 Read block 986 len 40960
18966 253/300 Flush
18967 254/300 Write block 3586 len 16384 data: 7 6 5 3
18968 255/300 Read block 1687 len 12288
18969 256/300 Flush
18970 257/300 Write block 3634 len 20480 data: 3 2 2 2 2
18971 258/300 Read block 3252 len 8192
18972 259/300 Read block 4229 len 12288
18973 260/300 Read block 3311 len 40960
18974 261/300 Read block 4900 len 8192
18975 262/300 Write block 1421 len 4096 data: 3
18976 263/300 Write block 608 len 20480 data: 7 7 5 6 4
18977 264/300 Write block 3361 len 12288 data: 5 5 5
18978 265/300 Write block 4190 len 12288 data: 6 5 6
18979 266/300 Read block 4802 len 16384
18980 267/300 Write block 1998 len 32768 data: 5 4 6 6 6 6 7 8
18981 268/300 Read block 2234 len 4096
18982 269/300 Flush
18983 270/300 Flush
18984 271/300 Write block 1631 len 12288 data: 3 3 3
18985 272/300 Write block 2081 len 8192 data: 5 5
18986 273/300 Read block 1014 len 12288
18987 274/300 Write block 102 len 24576 data: 3 4 4 5 5 5
18988 275/300 Read block 1201 len 40960
18989 276/300 Read block 4126 len 12288
18990 277/300 Read block 692 len 4096
18991 278/300 Read block 3959 len 16384
18992 279/300 Read block 3037 len 32768
18993 280/300 Read block 2122 len 36864
18994 281/300 Write block 763 len 24576 data: 6 5 5 6 5 4
18995 282/300 Write block 220 len 4096 data: 6
18996 283/300 Write block 1693 len 32768 data: 5 4 4 4 3 2 3 3
18997 284/300 Read block 758 len 20480
18998 285/300 Read block 2946 len 4096
18999 286/300 Flush
19000 287/300 Write block 1058 len 32768 data: 4 4 5 5 4 3 2 2
19001 288/300 Read block 425 len 28672
19002 289/300 Read block 150 len 20480
19003 290/300 Read block 1737 len 4096
19004 291/300 Write block 4577 len 24576 data: 3 4 4 4 4 6
19005 292/300 Write block 1275 len 16384 data: 6 6 6 5
19006 293/300 Write block 3827 len 4096 data: 8
19007 294/300 Read block 152 len 40960
19008 295/300 Read block 3393 len 28672
19009 296/300 Flush
19010 297/300 Flush
19011 298/300 Flush
19012 299/300 Read block 4883 len 28672
19013 300/300 Read block 2561 len 8192
19014 301/300 Write block 1191 len 16384 data: 5 5 5 6
19015 Main task has work to do, go find it
19016 got dsc Start(0)
19017 start 0
19018 Dec 09 15:11:51.561 INFO request completed, latency_us: 276, response_code: 204, uri: /start/cid/0, method: POST, req_id: 540c91ea-e821-44d5-a7b8-ccc3d2d9989e, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
19019 Replay: started 0, returned:Ok(())
19020 ----------------------------------------------------------------
19021 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19022 States: Offline Active Active
19023 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19024 6134 Acked Write 6 Sent Done Done false
19025 6135 Acked Write 7 Sent Done Done false
19026 6136 Acked Read 6 Sent Done Done false
19027 6137 Acked Read 5 Sent Done Done false
19028 6138 Acked Read 4 Sent Done Done false
19029 6139 Acked Write 5 Sent Done Done false
19030 6140 Acked Write 10 Sent Done Done false
19031 6141 Acked Read 9 Sent Done Done false
19032 6142 Acked Read 4 Sent Done Done false
19033 6143 Acked Write 8 Sent Done Done false
19034 6144 Acked Read 1 Sent Done Done false
19035 6145 Acked Write 1 Sent Done Done false
19036 6146 Acked Read 8 Sent Done Done false
19037 6147 Acked Read 6 Sent Done Done false
19038 6148 Acked Write 10 Sent Done Done false
19039 6149 Acked Read 7 Sent Done Done false
19040 6150 Acked Read 5 Sent Done Done false
19041 6151 Acked Write 10 Sent Done Done false
19042 6152 Acked Flush 0 Sent Done Done false
19043 6153 Acked Write 8 Sent Done Done false
19044 6154 Acked Flush 0 Sent Done Done false
19045 6155 Acked Write 9 Sent Done Done false
19046 6156 Acked Read 8 Sent Done Done false
19047 6157 Acked Read 3 Sent Done Done false
19048 6158 Acked Write 4 Sent Done Done false
19049 6159 Acked Read 10 Sent Done Done false
19050 6160 Acked Write 7 Sent Done Done false
19051 6161 Acked Write 1 Sent Done Done false
19052 6162 Acked Read 2 Sent Done Done false
19053 6163 Acked Read 2 Sent Done Done false
19054 6164 Acked Read 4 Sent Done Done false
19055 6165 Acked Read 6 Sent Done Done false
19056 6166 Acked Write 2 Sent Done Done false
19057 6167 Acked Read 5 Sent Done Done false
19058 6168 Acked Read 5 Sent Done Done false
19059 6169 Acked Write 10 Sent Done Done false
19060 6170 Acked Read 9 Sent Done Done false
19061 6171 Acked Read 6 Sent Done Done false
19062 6172 Acked Read 7 Sent Done Done false
19063 6173 Acked Write 1 Sent Done Done false
19064 6174 Acked Read 8 Sent Done Done false
19065 6175 Acked Read 6 Sent Done Done false
19066 6176 Acked Write 6 Sent Done Done false
19067 6177 Acked Flush 0 Sent Done Done false
19068 6178 Acked Read 1 Sent Done Done false
19069 6179 Acked Write 1 Sent Done Done false
19070 6180 Acked Read 10 Sent Done Done false
19071 6181 Acked Write 10 Sent Done Done false
19072 6182 Acked Flush 0 Sent Done Done false
19073 6183 Acked Read 5 Sent Done Done false
19074 6184 Acked Read 3 Sent Done Done false
19075 6185 Acked Read 8 Sent Done Done false
19076 6186 Acked Read 2 Sent Done Done false
19077 6187 Acked Write 3 Sent Done Done false
19078 6188 Acked Write 7 Sent Done Done false
19079 6189 Acked Read 3 Sent Done Done false
19080 6190 Acked Write 3 Sent Done Done false
19081 6191 Acked Write 4 Sent Done Done false
19082 6192 Acked Read 9 Sent Done Done false
19083 6193 Acked Write 6 Sent Done Done false
19084 6194 Acked Write 1 Sent Done Done false
19085 6195 Acked Read 1 Sent Done Done false
19086 6196 Acked Read 5 Sent Done Done false
19087 6197 Acked Write 2 Sent Done Done false
19088 6198 Acked Write 2 Sent Done Done false
19089 6199 Acked Write 9 Sent Done Done false
19090 6200 Acked Flush 0 Sent Done Done false
19091 6201 Acked Write 8 Sent Done Done false
19092 6202 Acked Read 2 Sent Done Done false
19093 6203 Acked Read 4 Sent Done Done false
19094 6204 Acked Flush 0 Sent Done Done false
19095 6205 Acked Flush 0 Sent Done Done false
19096 6206 Acked Write 5 Sent Done Done false
19097 6207 Acked Read 1 Sent Done Done false
19098 6208 Acked Read 6 Sent Done Done false
19099 6209 Acked Flush 0 Sent Done Done false
19100 6210 Acked Read 1 Sent Done Done false
19101 6211 Acked Read 10 Sent Done Done false
19102 6212 Acked Write 5 Sent Done Done false
19103 6213 Acked Read 6 Sent Done Done false
19104 6214 Acked Write 7 Sent Done Done false
19105 6215 Acked Read 10 Sent Done Done false
19106 6216 Acked Read 5 Sent Done Done false
19107 6217 Acked Read 8 Sent Done Done false
19108 6218 Acked Read 3 Sent Done Done false
19109 6219 Acked Write 1 Sent Done Done false
19110 6220 Acked Write 10 Sent Done Done false
19111 6221 Acked Read 2 Sent Done Done false
19112 6222 Acked Write 7 Sent Done Done false
19113 6223 Acked Write 2 Sent Done Done false
19114 6224 Acked Write 2 Sent Done Done false
19115 6225 Acked Write 2 Sent Done Done false
19116 6226 Acked Read 1 Sent Done Done false
19117 6227 Acked Read 2 Sent Done Done false
19118 6228 Acked Write 10 Sent Done Done false
19119 6229 Acked Read 9 Sent Done Done false
19120 6230 Acked Read 7 Sent Done Done false
19121 6231 Acked Read 5 Sent Done Done false
19122 6232 Acked Write 10 Sent Done Done false
19123 6233 Acked Read 4 Sent Done Done false
19124 6234 Acked Write 1 Sent Done Done false
19125 6235 Acked Write 4 Sent Done Done false
19126 6236 Acked Read 8 Sent Done Done false
19127 6237 Acked Flush 0 Sent Done Done false
19128 6238 Acked Write 1 Sent Done Done false
19129 6239 Acked Read 6 Sent Done Done false
19130 6240 Acked Write 6 Sent Done Done false
19131 6241 Acked Flush 0 Sent Done Done false
19132 6242 Acked Read 6 Sent Done Done false
19133 6243 Acked Write 2 Sent Done Done false
19134 6244 Acked Write 8 Sent Done Done false
19135 6245 Acked Read 10 Sent Done Done false
19136 6246 Acked Read 5 Sent Done Done false
19137 6247 Acked Read 7 Sent Done Done false
19138 6248 Acked Read 10 Sent Done Done false
19139 6249 Acked Flush 0 Sent Done Done false
19140 6250 Acked Write 7 Sent Done Done false
19141 6251 Acked Read 4 Sent Done Done false
19142 6252 Acked Write 5 Sent Done Done false
19143 6253 Acked Write 8 Sent Done Done false
19144 6254 Acked Write 5 Sent Done Done false
19145 6255 Acked Write 10 Sent Done Done false
19146 6256 Acked Read 9 Sent Done Done false
19147 6257 Acked Read 6 Sent Done Done false
19148 6258 Acked Read 4 Sent Done Done false
19149 6259 Acked Read 2 Sent Done Done false
19150 6260 Acked Read 4 Sent Done Done false
19151 6261 Acked Write 8 Sent Done Done false
19152 6262 Acked Flush 0 Sent Done Done false
19153 6263 Acked Read 1 Sent Done Done false
19154 6264 Acked Write 4 Sent Done Done false
19155 6265 Acked Read 9 Sent Done Done false
19156 6266 Acked Read 6 Sent Done Done false
19157 6267 Acked Read 4 Sent Done Done false
19158 6268 Acked Read 4 Sent Done Done false
19159 6269 Acked Flush 0 Sent Done Done false
19160 6270 Acked Write 2 Sent Done Done false
19161 6271 Acked Write 3 Sent Done Done false
19162 6272 Acked Write 7 Sent Done Done false
19163 6273 Acked Write 10 Sent Done Done false
19164 6274 Acked Read 1 Sent Done Done false
19165 6275 Acked Read 5 Sent Done Done false
19166 6276 Acked Write 7 Sent Done Done false
19167 6277 Acked Flush 0 Sent Done Done false
19168 6278 Acked Write 9 Sent Done Done false
19169 6279 Acked Read 9 Sent Done Done false
19170 6280 Acked Write 10 Sent Done Done false
19171 6281 Acked Write 7 Sent Done Done false
19172 6282 Acked Read 10 Sent Done Done false
19173 6283 Acked Read 6 Sent Done Done false
19174 6284 Acked Write 10 Sent Done Done false
19175 6285 Acked Read 3 Sent Done Done false
19176 6286 Acked Write 4 Sent Done Done false
19177 6287 Acked Read 7 Sent Done Done false
19178 6288 Acked Read 2 Sent Done Done false
19179 6289 Acked Read 9 Sent Done Done false
19180 6290 Acked Write 8 Sent Done Done false
19181 6291 Acked Write 10 Sent Done Done false
19182 6292 Acked Read 8 Sent Done Done false
19183 6293 Acked Read 7 Sent Done Done false
19184 6294 Acked Write 7 Sent Done Done false
19185 6295 Acked Read 9 Sent Done Done false
19186 6296 Acked Read 10 Sent Done Done false
19187 6297 Acked Read 4 Sent Done Done false
19188 6298 Acked Read 3 Sent Done Done false
19189 6299 Acked Write 4 Sent Done Done false
19190 6300 Acked Write 3 Sent Done Done false
19191 6301 Acked Write 7 Sent Done Done false
19192 6302 Acked Flush 0 Sent Done Done false
19193 6303 Acked Read 1 Sent Done Done false
19194 6304 Acked Read 1 Sent Done Done false
19195 6305 Acked Read 4 Sent Done Done false
19196 6306 Acked Flush 0 Sent Done Done false
19197 6307 Acked Read 4 Sent Done Done false
19198 6308 Acked Write 2 Sent Done Done false
19199 6309 Acked Read 9 Sent Done Done false
19200 6310 Acked Write 3 Sent Done Done false
19201 6311 Acked Read 5 Sent Done Done false
19202 6312 Acked Write 3 Sent Done Done false
19203 6313 Acked Read 2 Sent Done Done false
19204 6314 Acked Read 9 Sent Done Done false
19205 6315 Acked Read 2 Sent Done Done false
19206 6316 Acked Read 10 Sent Done Done false
19207 6317 Acked Flush 0 Sent Done Done false
19208 6318 Acked Read 4 Sent Done Done false
19209 6319 Acked Flush 0 Sent Done Done false
19210 6320 Acked Flush 0 Sent Done Done false
19211 6321 Acked Flush 0 Sent Done Done false
19212 6322 Acked Write 4 Sent Done Done false
19213 6323 Acked Flush 0 Sent Done Done false
19214 6324 Acked Read 10 Sent Done Done false
19215 6325 Acked Read 10 Sent Done Done false
19216 6326 Acked Read 6 Sent Done Done false
19217 6327 Acked Write 9 Sent Done Done false
19218 6328 Acked Write 9 Sent Done Done false
19219 6329 Acked Read 2 Sent Done Done false
19220 6330 Acked Write 1 Sent Done Done false
19221 6331 Acked Flush 0 Sent Done Done false
19222 6332 Acked Read 10 Sent Done Done false
19223 6333 Acked Read 5 Sent Done Done false
19224 6334 Acked Read 9 Sent Done Done false
19225 6335 Acked Write 6 Sent Done Done false
19226 6336 Acked Read 5 Sent Done Done false
19227 6337 Acked Write 7 Sent Done Done false
19228 6338 Acked Flush 0 Sent Done Done false
19229 6339 Acked Read 9 Sent Done Done false
19230 6340 Acked Write 3 Sent Done Done false
19231 6341 Acked Write 10 Sent Done Done false
19232 6342 Acked Read 7 Sent Done Done false
19233 6343 Acked Read 7 Sent Done Done false
19234 6344 Acked Read 5 Sent Done Done false
19235 6345 Acked Write 8 Sent Done Done false
19236 6346 Acked Write 3 Sent Done Done false
19237 6347 Acked Read 5 Sent Done Done false
19238 6348 Acked Write 7 Sent Done Done false
19239 6349 Acked Flush 0 Sent Done Done false
19240 6350 Acked Read 2 Sent Done Done false
19241 6351 Acked Read 8 Sent Done Done false
19242 6352 Acked Write 3 Sent Done Done false
19243 6353 Acked Write 5 Sent Done Done false
19244 6354 Acked Read 10 Sent Done Done false
19245 6355 Acked Read 6 Sent Done Done false
19246 6356 Acked Flush 0 Sent Done Done false
19247 6357 Acked Write 3 Sent Done Done false
19248 6358 Acked Write 10 Sent Done Done false
19249 6359 Acked Read 5 Sent Done Done false
19250 6360 Acked Flush 0 Sent Done Done false
19251 6361 Acked Read 1 Sent Done Done false
19252 6362 Acked Flush 0 Sent Done Done false
19253 6363 Acked Read 2 Sent Done Done false
19254 6364 Acked Write 2 Sent Done Done false
19255 6365 Acked Read 1 Sent Done Done false
19256 6366 Acked Write 2 Sent Done Done false
19257 6367 Acked Read 7 Sent Done Done false
19258 6368 Acked Write 3 Sent Done Done false
19259 6369 Acked Read 6 Sent Done Done false
19260 6370 Acked Read 7 Sent Done Done false
19261 6371 Acked Write 2 Sent Done Done false
19262 6372 Acked Write 7 Sent Done Done false
19263 6373 Acked Write 4 Sent Done Done false
19264 6374 Acked Read 5 Sent Done Done false
19265 6375 Acked Write 4 Sent Done Done false
19266 6376 Acked Flush 0 Sent Done Done false
19267 6377 Acked Write 8 Sent Done Done false
19268 6378 Acked Read 5 Sent Done Done false
19269 6379 Acked Write 7 Sent Done Done false
19270 6380 Acked Read 6 Sent Done Done false
19271 6381 Acked Read 5 Sent Done Done false
19272 6382 Acked Write 5 Sent Done Done false
19273 6383 Acked Write 1 Sent Done Done false
19274 6384 Acked Read 9 Sent Done Done false
19275 6385 Acked Read 10 Sent Done Done false
19276 6386 Acked Flush 0 Sent Done Done false
19277 6387 Acked Write 4 Sent Done Done false
19278 6388 Acked Read 3 Sent Done Done false
19279 6389 Acked Flush 0 Sent Done Done false
19280 6390 Acked Write 5 Sent Done Done false
19281 6391 Acked Read 2 Sent Done Done false
19282 6392 Acked Read 3 Sent Done Done false
19283 6393 Acked Read 10 Sent Done Done false
19284 6394 Acked Read 2 Sent Done Done false
19285 6395 Acked Write 1 Sent Done Done false
19286 6396 Acked Write 5 Sent Done Done false
19287 6397 Acked Write 3 Sent Done Done false
19288 6398 Acked Write 3 Sent Done Done false
19289 6399 Acked Read 4 Sent Done Done false
19290 6400 Acked Write 8 Sent Done Done false
19291 6401 Acked Read 1 Sent Done Done false
19292 6402 Acked Flush 0 Sent Done Done false
19293 6403 Acked Flush 0 Sent Done Done false
19294 6404 Acked Write 3 Sent Done Done false
19295 6405 Acked Write 2 Sent Done Done false
19296 6406 Acked Read 3 Sent Done Done false
19297 6407 Acked Write 6 Sent Done Done false
19298 6408 Acked Read 10 Sent Done Done false
19299 6409 Acked Read 3 Sent Done Done false
19300 6410 Acked Read 1 Sent Done Done false
19301 6411 Acked Read 4 Sent Done Done false
19302 6412 Acked Read 8 Sent Done Done false
19303 6413 Acked Read 9 Sent Done Done false
19304 6414 Acked Write 6 Sent Done Done false
19305 6415 Acked Write 1 Sent Done Done false
19306 6416 Acked Write 8 Sent Done Done false
19307 6417 Acked Read 5 Sent Done Done false
19308 6418 Acked Read 1 Sent Done Done false
19309 6419 Acked Flush 0 Sent Done Done false
19310 6420 Acked Write 8 Sent Done Done false
19311 6421 Acked Read 7 Sent Done Done false
19312 6422 Acked Read 5 Sent Done Done false
19313 6423 Acked Read 1 Sent Done Done false
19314 6424 Acked Write 6 Sent Done Done false
19315 6425 Acked Write 4 Sent Done Done false
19316 6426 Acked Write 1 Sent Done Done false
19317 6427 Acked Read 10 Sent Done Done false
19318 6428 Acked Read 7 Sent Done Done false
19319 6429 Acked Flush 0 Sent Done Done false
19320 6430 Acked Flush 0 Sent Done Done false
19321 6431 Acked Flush 0 Sent Done Done false
19322 6432 Acked Read 7 Sent Done Done false
19323 6433 Acked Read 2 Sent Done Done false
19324 6434 Acked Write 4 Sent Done Done false
19325 STATES DS:0 DS:1 DS:2 TOTAL
19326 Sent 301 0 0 301
19327 Done 0 301 301 602
19328 Skipped 0 0 0 0
19329 Error 0 0 0 0
19330 Last Flush: 6133 6431 6431
19331 Downstairs last five completed: 6133 6132 6131 6130 6129
19332 Upstairs last five completed: 6434 6433 6432 6431 6430
19333 CLIENT: Up:0 ds:301 act:2
19334 [8810] got start action
19335 [8810] I am going to restart
19336 Starting downstairs at port 8810
19337 Make output file at "/tmp/dsc/downstairs-8810.txt"
19338 [8810][0] reports Starting
19339 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1219)
19340 [8810][0] reports Running
19341 ----------------------------------------------------------------
19342 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
19343 States: Offline Active Active
19344 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19345 6134 Acked Write 6 Sent Done Done false
19346 6135 Acked Write 7 Sent Done Done false
19347 6136 Acked Read 6 Sent Done Done false
19348 6137 Acked Read 5 Sent Done Done false
19349 6138 Acked Read 4 Sent Done Done false
19350 6139 Acked Write 5 Sent Done Done false
19351 6140 Acked Write 10 Sent Done Done false
19352 6141 Acked Read 9 Sent Done Done false
19353 6142 Acked Read 4 Sent Done Done false
19354 6143 Acked Write 8 Sent Done Done false
19355 6144 Acked Read 1 Sent Done Done false
19356 6145 Acked Write 1 Sent Done Done false
19357 6146 Acked Read 8 Sent Done Done false
19358 6147 Acked Read 6 Sent Done Done false
19359 6148 Acked Write 10 Sent Done Done false
19360 6149 Acked Read 7 Sent Done Done false
19361 6150 Acked Read 5 Sent Done Done false
19362 6151 Acked Write 10 Sent Done Done false
19363 6152 Acked Flush 0 Sent Done Done false
19364 6153 Acked Write 8 Sent Done Done false
19365 6154 Acked Flush 0 Sent Done Done false
19366 6155 Acked Write 9 Sent Done Done false
19367 6156 Acked Read 8 Sent Done Done false
19368 6157 Acked Read 3 Sent Done Done false
19369 6158 Acked Write 4 Sent Done Done false
19370 6159 Acked Read 10 Sent Done Done false
19371 6160 Acked Write 7 Sent Done Done false
19372 6161 Acked Write 1 Sent Done Done false
19373 6162 Acked Read 2 Sent Done Done false
19374 6163 Acked Read 2 Sent Done Done false
19375 6164 Acked Read 4 Sent Done Done false
19376 6165 Acked Read 6 Sent Done Done false
19377 6166 Acked Write 2 Sent Done Done false
19378 6167 Acked Read 5 Sent Done Done false
19379 6168 Acked Read 5 Sent Done Done false
19380 6169 Acked Write 10 Sent Done Done false
19381 6170 Acked Read 9 Sent Done Done false
19382 6171 Acked Read 6 Sent Done Done false
19383 6172 Acked Read 7 Sent Done Done false
19384 6173 Acked Write 1 Sent Done Done false
19385 6174 Acked Read 8 Sent Done Done false
19386 6175 Acked Read 6 Sent Done Done false
19387 6176 Acked Write 6 Sent Done Done false
19388 6177 Acked Flush 0 Sent Done Done false
19389 6178 Acked Read 1 Sent Done Done false
19390 6179 Acked Write 1 Sent Done Done false
19391 6180 Acked Read 10 Sent Done Done false
19392 6181 Acked Write 10 Sent Done Done false
19393 6182 Acked Flush 0 Sent Done Done false
19394 6183 Acked Read 5 Sent Done Done false
19395 6184 Acked Read 3 Sent Done Done false
19396 6185 Acked Read 8 Sent Done Done false
19397 6186 Acked Read 2 Sent Done Done false
19398 6187 Acked Write 3 Sent Done Done false
19399 6188 Acked Write 7 Sent Done Done false
19400 6189 Acked Read 3 Sent Done Done false
19401 6190 Acked Write 3 Sent Done Done false
19402 6191 Acked Write 4 Sent Done Done false
19403 6192 Acked Read 9 Sent Done Done false
19404 6193 Acked Write 6 Sent Done Done false
19405 6194 Acked Write 1 Sent Done Done false
19406 6195 Acked Read 1 Sent Done Done false
19407 6196 Acked Read 5 Sent Done Done false
19408 6197 Acked Write 2 Sent Done Done false
19409 6198 Acked Write 2 Sent Done Done false
19410 6199 Acked Write 9 Sent Done Done false
19411 6200 Acked Flush 0 Sent Done Done false
19412 6201 Acked Write 8 Sent Done Done false
19413 6202 Acked Read 2 Sent Done Done false
19414 6203 Acked Read 4 Sent Done Done false
19415 6204 Acked Flush 0 Sent Done Done false
19416 6205 Acked Flush 0 Sent Done Done false
19417 6206 Acked Write 5 Sent Done Done false
19418 6207 Acked Read 1 Sent Done Done false
19419 6208 Acked Read 6 Sent Done Done false
19420 6209 Acked Flush 0 Sent Done Done false
19421 6210 Acked Read 1 Sent Done Done false
19422 6211 Acked Read 10 Sent Done Done false
19423 6212 Acked Write 5 Sent Done Done false
19424 6213 Acked Read 6 Sent Done Done false
19425 6214 Acked Write 7 Sent Done Done false
19426 6215 Acked Read 10 Sent Done Done false
19427 6216 Acked Read 5 Sent Done Done false
19428 6217 Acked Read 8 Sent Done Done false
19429 6218 Acked Read 3 Sent Done Done false
19430 6219 Acked Write 1 Sent Done Done false
19431 6220 Acked Write 10 Sent Done Done false
19432 6221 Acked Read 2 Sent Done Done false
19433 6222 Acked Write 7 Sent Done Done false
19434 6223 Acked Write 2 Sent Done Done false
19435 6224 Acked Write 2 Sent Done Done false
19436 6225 Acked Write 2 Sent Done Done false
19437 6226 Acked Read 1 Sent Done Done false
19438 6227 Acked Read 2 Sent Done Done false
19439 6228 Acked Write 10 Sent Done Done false
19440 6229 Acked Read 9 Sent Done Done false
19441 6230 Acked Read 7 Sent Done Done false
19442 6231 Acked Read 5 Sent Done Done false
19443 6232 Acked Write 10 Sent Done Done false
19444 6233 Acked Read 4 Sent Done Done false
19445 6234 Acked Write 1 Sent Done Done false
19446 6235 Acked Write 4 Sent Done Done false
19447 6236 Acked Read 8 Sent Done Done false
19448 6237 Acked Flush 0 Sent Done Done false
19449 6238 Acked Write 1 Sent Done Done false
19450 6239 Acked Read 6 Sent Done Done false
19451 6240 Acked Write 6 Sent Done Done false
19452 6241 Acked Flush 0 Sent Done Done false
19453 6242 Acked Read 6 Sent Done Done false
19454 6243 Acked Write 2 Sent Done Done false
19455 6244 Acked Write 8 Sent Done Done false
19456 6245 Acked Read 10 Sent Done Done false
19457 6246 Acked Read 5 Sent Done Done false
19458 6247 Acked Read 7 Sent Done Done false
19459 6248 Acked Read 10 Sent Done Done false
19460 6249 Acked Flush 0 Sent Done Done false
19461 6250 Acked Write 7 Sent Done Done false
19462 6251 Acked Read 4 Sent Done Done false
19463 6252 Acked Write 5 Sent Done Done false
19464 6253 Acked Write 8 Sent Done Done false
19465 6254 Acked Write 5 Sent Done Done false
19466 6255 Acked Write 10 Sent Done Done false
19467 6256 Acked Read 9 Sent Done Done false
19468 6257 Acked Read 6 Sent Done Done false
19469 6258 Acked Read 4 Sent Done Done false
19470 6259 Acked Read 2 Sent Done Done false
19471 6260 Acked Read 4 Sent Done Done false
19472 6261 Acked Write 8 Sent Done Done false
19473 6262 Acked Flush 0 Sent Done Done false
19474 6263 Acked Read 1 Sent Done Done false
19475 6264 Acked Write 4 Sent Done Done false
19476 6265 Acked Read 9 Sent Done Done false
19477 6266 Acked Read 6 Sent Done Done false
19478 6267 Acked Read 4 Sent Done Done false
19479 6268 Acked Read 4 Sent Done Done false
19480 6269 Acked Flush 0 Sent Done Done false
19481 6270 Acked Write 2 Sent Done Done false
19482 6271 Acked Write 3 Sent Done Done false
19483 6272 Acked Write 7 Sent Done Done false
19484 6273 Acked Write 10 Sent Done Done false
19485 6274 Acked Read 1 Sent Done Done false
19486 6275 Acked Read 5 Sent Done Done false
19487 6276 Acked Write 7 Sent Done Done false
19488 6277 Acked Flush 0 Sent Done Done false
19489 6278 Acked Write 9 Sent Done Done false
19490 6279 Acked Read 9 Sent Done Done false
19491 6280 Acked Write 10 Sent Done Done false
19492 6281 Acked Write 7 Sent Done Done false
19493 6282 Acked Read 10 Sent Done Done false
19494 6283 Acked Read 6 Sent Done Done false
19495 6284 Acked Write 10 Sent Done Done false
19496 6285 Acked Read 3 Sent Done Done false
19497 6286 Acked Write 4 Sent Done Done false
19498 6287 Acked Read 7 Sent Done Done false
19499 6288 Acked Read 2 Sent Done Done false
19500 6289 Acked Read 9 Sent Done Done false
19501 6290 Acked Write 8 Sent Done Done false
19502 6291 Acked Write 10 Sent Done Done false
19503 6292 Acked Read 8 Sent Done Done false
19504 6293 Acked Read 7 Sent Done Done false
19505 6294 Acked Write 7 Sent Done Done false
19506 6295 Acked Read 9 Sent Done Done false
19507 6296 Acked Read 10 Sent Done Done false
19508 6297 Acked Read 4 Sent Done Done false
19509 6298 Acked Read 3 Sent Done Done false
19510 6299 Acked Write 4 Sent Done Done false
19511 6300 Acked Write 3 Sent Done Done false
19512 6301 Acked Write 7 Sent Done Done false
19513 6302 Acked Flush 0 Sent Done Done false
19514 6303 Acked Read 1 Sent Done Done false
19515 6304 Acked Read 1 Sent Done Done false
19516 6305 Acked Read 4 Sent Done Done false
19517 6306 Acked Flush 0 Sent Done Done false
19518 6307 Acked Read 4 Sent Done Done false
19519 6308 Acked Write 2 Sent Done Done false
19520 6309 Acked Read 9 Sent Done Done false
19521 6310 Acked Write 3 Sent Done Done false
19522 6311 Acked Read 5 Sent Done Done false
19523 6312 Acked Write 3 Sent Done Done false
19524 6313 Acked Read 2 Sent Done Done false
19525 6314 Acked Read 9 Sent Done Done false
19526 6315 Acked Read 2 Sent Done Done false
19527 6316 Acked Read 10 Sent Done Done false
19528 6317 Acked Flush 0 Sent Done Done false
19529 6318 Acked Read 4 Sent Done Done false
19530 6319 Acked Flush 0 Sent Done Done false
19531 6320 Acked Flush 0 Sent Done Done false
19532 6321 Acked Flush 0 Sent Done Done false
19533 6322 Acked Write 4 Sent Done Done false
19534 6323 Acked Flush 0 Sent Done Done false
19535 6324 Acked Read 10 Sent Done Done false
19536 6325 Acked Read 10 Sent Done Done false
19537 6326 Acked Read 6 Sent Done Done false
19538 6327 Acked Write 9 Sent Done Done false
19539 6328 Acked Write 9 Sent Done Done false
19540 6329 Acked Read 2 Sent Done Done false
19541 6330 Acked Write 1 Sent Done Done false
19542 6331 Acked Flush 0 Sent Done Done false
19543 6332 Acked Read 10 Sent Done Done false
19544 6333 Acked Read 5 Sent Done Done false
19545 6334 Acked Read 9 Sent Done Done false
19546 6335 Acked Write 6 Sent Done Done false
19547 6336 Acked Read 5 Sent Done Done false
19548 6337 Acked Write 7 Sent Done Done false
19549 6338 Acked Flush 0 Sent Done Done false
19550 6339 Acked Read 9 Sent Done Done false
19551 6340 Acked Write 3 Sent Done Done false
19552 6341 Acked Write 10 Sent Done Done false
19553 6342 Acked Read 7 Sent Done Done false
19554 6343 Acked Read 7 Sent Done Done false
19555 6344 Acked Read 5 Sent Done Done false
19556 6345 Acked Write 8 Sent Done Done false
19557 6346 Acked Write 3 Sent Done Done false
19558 6347 Acked Read 5 Sent Done Done false
19559 6348 Acked Write 7 Sent Done Done false
19560 6349 Acked Flush 0 Sent Done Done false
19561 6350 Acked Read 2 Sent Done Done false
19562 6351 Acked Read 8 Sent Done Done false
19563 6352 Acked Write 3 Sent Done Done false
19564 6353 Acked Write 5 Sent Done Done false
19565 6354 Acked Read 10 Sent Done Done false
19566 6355 Acked Read 6 Sent Done Done false
19567 6356 Acked Flush 0 Sent Done Done false
19568 6357 Acked Write 3 Sent Done Done false
19569 6358 Acked Write 10 Sent Done Done false
19570 6359 Acked Read 5 Sent Done Done false
19571 6360 Acked Flush 0 Sent Done Done false
19572 6361 Acked Read 1 Sent Done Done false
19573 6362 Acked Flush 0 Sent Done Done false
19574 6363 Acked Read 2 Sent Done Done false
19575 6364 Acked Write 2 Sent Done Done false
19576 6365 Acked Read 1 Sent Done Done false
19577 6366 Acked Write 2 Sent Done Done false
19578 6367 Acked Read 7 Sent Done Done false
19579 6368 Acked Write 3 Sent Done Done false
19580 6369 Acked Read 6 Sent Done Done false
19581 6370 Acked Read 7 Sent Done Done false
19582 6371 Acked Write 2 Sent Done Done false
19583 6372 Acked Write 7 Sent Done Done false
19584 6373 Acked Write 4 Sent Done Done false
19585 6374 Acked Read 5 Sent Done Done false
19586 6375 Acked Write 4 Sent Done Done false
19587 6376 Acked Flush 0 Sent Done Done false
19588 6377 Acked Write 8 Sent Done Done false
19589 6378 Acked Read 5 Sent Done Done false
19590 6379 Acked Write 7 Sent Done Done false
19591 6380 Acked Read 6 Sent Done Done false
19592 6381 Acked Read 5 Sent Done Done false
19593 6382 Acked Write 5 Sent Done Done false
19594 6383 Acked Write 1 Sent Done Done false
19595 6384 Acked Read 9 Sent Done Done false
19596 6385 Acked Read 10 Sent Done Done false
19597 6386 Acked Flush 0 Sent Done Done false
19598 6387 Acked Write 4 Sent Done Done false
19599 6388 Acked Read 3 Sent Done Done false
19600 6389 Acked Flush 0 Sent Done Done false
19601 6390 Acked Write 5 Sent Done Done false
19602 6391 Acked Read 2 Sent Done Done false
19603 6392 Acked Read 3 Sent Done Done false
19604 6393 Acked Read 10 Sent Done Done false
19605 6394 Acked Read 2 Sent Done Done false
19606 6395 Acked Write 1 Sent Done Done false
19607 6396 Acked Write 5 Sent Done Done false
19608 6397 Acked Write 3 Sent Done Done false
19609 6398 Acked Write 3 Sent Done Done false
19610 6399 Acked Read 4 Sent Done Done false
19611 6400 Acked Write 8 Sent Done Done false
19612 6401 Acked Read 1 Sent Done Done false
19613 6402 Acked Flush 0 Sent Done Done false
19614 6403 Acked Flush 0 Sent Done Done false
19615 6404 Acked Write 3 Sent Done Done false
19616 6405 Acked Write 2 Sent Done Done false
19617 6406 Acked Read 3 Sent Done Done false
19618 6407 Acked Write 6 Sent Done Done false
19619 6408 Acked Read 10 Sent Done Done false
19620 6409 Acked Read 3 Sent Done Done false
19621 6410 Acked Read 1 Sent Done Done false
19622 6411 Acked Read 4 Sent Done Done false
19623 6412 Acked Read 8 Sent Done Done false
19624 6413 Acked Read 9 Sent Done Done false
19625 6414 Acked Write 6 Sent Done Done false
19626 6415 Acked Write 1 Sent Done Done false
19627 6416 Acked Write 8 Sent Done Done false
19628 6417 Acked Read 5 Sent Done Done false
19629 6418 Acked Read 1 Sent Done Done false
19630 6419 Acked Flush 0 Sent Done Done false
19631 6420 Acked Write 8 Sent Done Done false
19632 6421 Acked Read 7 Sent Done Done false
19633 6422 Acked Read 5 Sent Done Done false
19634 6423 Acked Read 1 Sent Done Done false
19635 6424 Acked Write 6 Sent Done Done false
19636 6425 Acked Write 4 Sent Done Done false
19637 6426 Acked Write 1 Sent Done Done false
19638 6427 Acked Read 10 Sent Done Done false
19639 6428 Acked Read 7 Sent Done Done false
19640 6429 Acked Flush 0 Sent Done Done false
19641 6430 Acked Flush 0 Sent Done Done false
19642 6431 Acked Flush 0 Sent Done Done false
19643 6432 Acked Read 7 Sent Done Done false
19644 6433 Acked Read 2 Sent Done Done false
19645 6434 Acked Write 4 Sent Done Done false
19646 6435 Acked Flush 0 Sent Done Done false
19647 STATES DS:0 DS:1 DS:2 TOTAL
19648 Sent 302 0 0 302
19649 Done 0 302 302 604
19650 Skipped 0 0 0 0
19651 Error 0 0 0 0
19652 Last Flush: 6133 6435 6435
19653 Downstairs last five completed: 6133 6132 6131 6130 6129
19654 Upstairs last five completed: 6435 6434 6433 6432 6431
19655 CLIENT: Up:0 ds:302 act:2
196562024-12-09T15:11:57.322ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
196572024-12-09T15:11:57.322ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49759) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
196582024-12-09T15:11:57.324ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
196592024-12-09T15:11:57.324ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
196602024-12-09T15:11:57.324ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
196612024-12-09T15:11:57.324ZINFOcrucible: send last flush ID to this DS: 6133 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
196622024-12-09T15:11:57.324ZINFOcrucible: Replied this last flush ID: 6133 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
196632024-12-09T15:11:57.324ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
196642024-12-09T15:11:57.324ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
196652024-12-09T15:11:57.324ZINFOcrucible: [0] client re-new 302 jobs since flush 6133 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
196662024-12-09T15:11:57.325ZINFOcrucible: [0] Replayed 302 jobs since flush: 6133 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
19667 ----------------------------------------------------------------
19668 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19669 Downstairs last five completed: 6435 6434 6433 6432 6431
19670 Upstairs last five completed: 6435 6434 6433 6432 6431
19671 CLIENT: Up:0 ds:0 act:3
19672 Replay: All jobs finished
19673 Main task has work to do, go find it
19674 got dsc Stop(0)
19675 stop 0
19676 Dec 09 15:11:59.566 INFO request completed, [8810] Got stop action so:false kr:false
19677 latency_us: 375, response_code: 204, uri: /stop/cid/0, method: POST, req_id: d7f49a96-261d-464a-b48a-1b77b49279ef, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
196782024-12-09T15:11:59.566ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
196792024-12-09T15:11:59.567ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
196802024-12-09T15:11:59.567ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
19681 Dec 09 15:11:59.567 INFO request completed, latency_us: 162, response_code: 200, uri{"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:11:59.567099627Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"0","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
19682 : /state/cid/0, method: GET, req_id: f377c734-a0bf-4dbe-be77-942d9894f585, remote_addr: 127.0.0.1:47510, {"msg":"[0] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:11:59.567134021Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"0","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
19683 local_addr: 127.0.0.1:9998
196842024-12-09T15:11:59.567ZINFOcrucible: client task is exiting = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
19685 [8810] Exited with: ExitStatus(unix_wait_status(9))
19686 [8810][0] reports Exit
19687 Dec 09 15:12:03.569 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4d58bd03-194d-4db4-b011-296ee699f03e, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
19688 001/300 Read block 517 len 36864
19689 002/300 Read block 2822 len 20480
19690 003/300 Read block 2677 len 12288
19691 004/300 Write block 3645 len 8192 data: 3 3
19692 005/300 Write block 2619 len 28672 data: 6 6 5 5 7 6 7
19693 006/300 Read block 4234 len 28672
19694 007/300 Write block 4070 len 16384 data: 6 7 4 3
19695 008/300 Write block 1348 len 8192 data: 3 3
19696 009/300 Flush
19697 010/300 Read block 306 len 12288
19698 011/300 Read block 4559 len 4096
19699 012/300 Read block 1105 len 32768
19700 013/300 Read block 4978 len 24576
19701 014/300 Read block 910 len 32768
19702 015/300 Flush
19703 016/300 Flush
19704 017/300 Write block 4649 len 36864 data: 8 9 7 6 5 6 6 6 6
19705 018/300 Write block 3841 len 20480 data: 4 5 7 8 8
19706 019/300 Read block 4722 len 28672
19707 020/300 Read block 1934 len 28672
19708 021/300 Read block 1235 len 24576
19709 022/300 Read block 4969 len 28672
19710 023/300 Read block 1799 len 36864
19711 024/300 Write block 213 len 32768 data: 9 9 8 8 8 8 8 7
19712 025/300 Write block 3443 len 4096 data: 4
19713 026/300 Flush
19714 027/300 Flush
19715 028/300 Flush
19716 029/300 Read block 1316 len 16384
19717 030/300 Flush
19718 031/300 Read block 520 len 4096
19719 032/300 Write block 436 len 24576 data: 3 3 2 2 2 2
19720 033/300 Write block 2005 len 24576 data: 9 7 6 7 6 5
19721 034/300 Write block 2893 len 20480 data: 5 5 5 4 4
19722 035/300 Read block 4424 len 40960
19723 036/300 Read block 1302 len 20480
19724 037/300 Read block 619 len 28672
19725 038/300 Write block 2883 len 8192 data: 3 3
19726 039/300 Read block 4625 len 36864
19727 040/300 Write block 1776 len 8192 data: 5 4
19728 041/300 Read block 178 len 12288
19729 042/300 Read block 4220 len 28672
19730 043/300 Read block 2157 len 4096
19731 044/300 Flush
19732 045/300 Read block 3200 len 8192
19733 046/300 Write block 1003 len 16384 data: 4 4 4 5
19734 047/300 Read block 4385 len 40960
19735 048/300 Read block 2106 len 16384
19736 049/300 Read block 3269 len 16384
19737 050/300 Read block 3328 len 16384
19738 051/300 Read block 1599 len 24576
19739 052/300 Flush
19740 053/300 Read block 4729 len 28672
19741 054/300 Write block 4774 len 24576 data: 2 2 3 3 3 3
19742 055/300 Read block 64 len 32768
19743 056/300 Read block 888 len 28672
19744 057/300 Read block 3440 len 36864
19745 058/300 Read block 1296 len 32768
19746 059/300 Write block 4309 len 4096 data: 3
19747 060/300 Read block 462 len 36864
19748 061/300 Read block 748 len 32768
19749 062/300 Read block 2059 len 12288
19750 063/300 Write block 636 len 40960 data: 6 7 6 6 6 5 4 4 4 4
19751 064/300 Write block 3078 len 40960 data: 4 4 3 4 3 2 2 2 2 2
19752 065/300 Read block 4488 len 24576
19753 066/300 Read block 3984 len 32768
19754 067/300 Read block 4155 len 24576
19755 068/300 Read block 1572 len 20480
19756 069/300 Write block 457 len 28672 data: 3 3 3 4 4 4 3
19757 070/300 Write block 3609 len 40960 data: 3 3 3 4 4 6 5 7 7 7
19758 071/300 Write block 4293 len 4096 data: 2
19759 072/300 Write block 2154 len 16384 data: 3 2 2 2
19760 073/300 Write block 3856 len 24576 data: 5 5 3 3 2 2
19761 074/300 Flush
19762 075/300 Read block 4929 len 4096
19763 076/300 Read block 2898 len 40960
19764 077/300 Read block 3264 len 24576
19765 078/300 Write block 1963 len 8192 data: 6 5
19766 079/300 Write block 1117 len 20480 data: 4 3 3 3 3
19767 080/300 Write block 617 len 16384 data: 2 3 3 3
19768 081/300 Write block 4987 len 36864 data: 8 8 9 7 9 8 6 5 4
19769 082/300 Flush
19770 083/300 Flush
19771 084/300 Read block 4383 len 32768
19772 085/300 Read block 2256 len 16384
19773 086/300 Read block 4305 len 24576
19774 087/300 Read block 990 len 24576
19775 088/300 Write block 1070 len 12288 data: 5 5 5
19776 089/300 Write block 4095 len 8192 data: 5 8
19777 090/300 Flush
19778 091/300 Read block 1052 len 32768
19779 092/300 Read block 1441 len 36864
19780 093/300 Flush
19781 094/300 Read block 135 len 16384
19782 095/300 Write block 4566 len 36864 data: 3 3 3 3 4 4 5 5 5
19783 096/300 Read block 468 len 36864
19784 097/300 Write block 4349 len 28672 data: 7 6 6 4 4 4 4
19785 098/300 Flush
19786 099/300 Write block 1584 len 8192 data: 2 2
19787 100/300 Read block 3846 len 40960
19788 101/300 Write block 3230 len 40960 data: 5 5 5 5 4 4 4 5 5 5
19789 102/300 Write block 2406 len 8192 data: 3 3
19790 103/300 Read block 3707 len 20480
19791 104/300 Write block 3311 len 32768 data: 5 5 4 5 4 4 3 3
19792 105/300 Read block 3152 len 40960
19793 106/300 Read block 3261 len 4096
19794 107/300 Read block 2348 len 16384
19795 108/300 Write block 918 len 36864 data: 5 3 3 4 4 4 5 6 7
19796 109/300 Write block 1818 len 28672 data: 5 6 6 4 5 4 3
19797 110/300 Read block 1122 len 36864
19798 111/300 Read block 4177 len 16384
19799 112/300 Read block 747 len 20480
19800 113/300 Read block 857 len 28672
19801 114/300 Write block 3669 len 24576 data: 4 2 2 2 3 3
19802 115/300 Write block 204 len 12288 data: 2 2 3
19803 116/300 Write block 4183 len 20480 data: 7 7 6 6 5
19804 117/300 Read block 383 len 12288
19805 118/300 Read block 872 len 4096
19806 119/300 Write block 3400 len 12288 data: 5 5 5
19807 120/300 Read block 4334 len 24576
19808 121/300 Read block 746 len 24576
19809 122/300 Read block 3880 len 4096
19810 123/300 Write block 3593 len 20480 data: 4 4 4 6 6
19811 124/300 Write block 3055 len 12288 data: 2 2 2
19812 125/300 Read block 3544 len 4096
19813 126/300 Write block 3978 len 40960 data: 5 6 5 4 5 4 4 6 6 4
19814 127/300 Read block 824 len 8192
19815 128/300 Write block 4468 len 4096 data: 5
19816 129/300 Write block 3948 len 12288 data: 3 2 2
19817 130/300 Read block 3517 len 4096
19818 131/300 Write block 686 len 36864 data: 7 7 8 7 6 5 4 4 4
19819 132/300 Write block 623 len 16384 data: 3 3 3 4
19820 133/300 Read block 940 len 32768
19821 134/300 Read block 928 len 4096
19822 135/300 Read block 4885 len 16384
19823 136/300 Write block 125 len 32768 data: 3 4 4 5 3 3 5 6
19824 137/300 Write block 4443 len 8192 data: 5 5
19825 138/300 Flush
19826 139/300 Write block 2018 len 24576 data: 7 4 4 4 6 5
19827 140/300 Write block 4100 len 28672 data: 7 8 6 7 7 8 7
19828 141/300 Write block 3573 len 4096 data: 3
19829 142/300 Read block 490 len 8192
19830 143/300 Flush
19831 144/300 Read block 689 len 36864
19832 145/300 Write block 1106 len 4096 data: 7
19833 146/300 Read block 2260 len 24576
19834 147/300 Read block 140 len 4096
19835 148/300 Read block 218 len 36864
19836 149/300 Write block 1553 len 20480 data: 4 2 2 4 4
19837 150/300 Read block 4647 len 12288
19838 151/300 Read block 4875 len 40960
19839 152/300 Write block 753 len 12288 data: 2 4 3
19840 153/300 Write block 799 len 16384 data: 5 6 8 8
19841 154/300 Read block 888 len 20480
19842 155/300 Write block 2240 len 20480 data: 2 4 4 4 4
19843 156/300 Read block 4908 len 12288
19844 157/300 Write block 2082 len 20480 data: 6 4 2 2 2
19845 158/300 Write block 4268 len 4096 data: 2
19846 159/300 Write block 1969 len 20480 data: 6 6 6 5 4
19847 160/300 Read block 3293 len 32768
19848 161/300 Read block 1017 len 40960
19849 162/300 Write block 222 len 16384 data: 5 6 5 4
19850 163/300 Write block 467 len 28672 data: 5 3 3 3 3 4 4
19851 164/300 Read block 2118 len 32768
19852 165/300 Read block 4772 len 16384
19853 166/300 Read block 3734 len 4096
19854 167/300 Flush
19855 168/300 Read block 4295 len 28672
19856 169/300 Write block 2651 len 20480 data: 4 4 5 7 8
19857 170/300 Write block 3251 len 24576 data: 3 3 3 3 3 5
19858 171/300 Read block 797 len 16384
19859 172/300 Write block 1954 len 8192 data: 3 3
19860 173/300 Read block 2977 len 8192
19861 174/300 Read block 3029 len 8192
19862 175/300 Read block 1146 len 16384
19863 176/300 Read block 1999 len 36864
19864 177/300 Flush
19865 178/300 Read block 1140 len 32768
19866 179/300 Flush
19867 180/300 Flush
19868 181/300 Write block 4894 len 24576 data: 5 4 3 3 3 3
19869 182/300 Write block 1753 len 36864 data: 6 6 6 7 7 7 6 6 5
19870 183/300 Read block 3948 len 24576
19871 184/300 Read block 2082 len 28672
19872 185/300 Read block 3814 len 36864
19873 186/300 Write block 1280 len 24576 data: 4 3 5 4 4 4
19874 187/300 Write block 3388 len 24576 data: 3 3 2 2 2 3
19875 188/300 Write block 1 len 28672 data: 3 4 5 3 3 4 4
19876 189/300 Write block 3118 len 20480 data: 3 5 6 7 7
19877 190/300 Read block 3406 len 36864
19878 191/300 Write block 4445 len 12288 data: 4 4 5
19879 192/300 Write block 174 len 4096 data: 2
19880 193/300 Read block 3667 len 12288
19881 194/300 Write block 3382 len 28672 data: 5 6 5 5 5 3 4
19882 195/300 Read block 4410 len 24576
19883 196/300 Read block 3424 len 8192
19884 197/300 Read block 2289 len 4096
19885 198/300 Write block 2903 len 28672 data: 6 6 5 4 4 4 3
19886 199/300 Read block 3518 len 24576
19887 200/300 Read block 3852 len 12288
19888 201/300 Write block 3013 len 40960 data: 6 5 5 5 7 7 5 4 4 4
19889 202/300 Flush
19890 203/300 Write block 518 len 4096 data: 6
19891 204/300 Read block 3513 len 20480
19892 205/300 Flush
19893 206/300 Read block 3858 len 12288
19894 207/300 Flush
19895 208/300 Read block 1346 len 40960
19896 209/300 Write block 3509 len 32768 data: 2 2 2 2 2 2 4 4
19897 210/300 Write block 248 len 36864 data: 6 5 5 6 5 5 4 4 4
19898 211/300 Read block 2498 len 20480
19899 212/300 Flush
19900 213/300 Read block 2458 len 20480
19901 214/300 Read block 816 len 36864
19902 215/300 Read block 2165 len 32768
19903 216/300 Read block 4926 len 40960
19904 217/300 Read block 1788 len 40960
19905 218/300 Read block 3289 len 24576
19906 219/300 Write block 4013 len 20480 data: 6 5 6 7 6
19907 220/300 Flush
19908 221/300 Read block 4803 len 24576
19909 222/300 Write block 3465 len 28672 data: 5 6 7 7 6 6 7
19910 223/300 Write block 3517 len 40960 data: 6 7 6 7 6 5 5 6 8 7
19911 224/300 Read block 4349 len 12288
19912 225/300 Write block 4321 len 32768 data: 7 7 7 7 7 5 5 4
19913 226/300 Read block 4517 len 4096
19914 227/300 Read block 3190 len 4096
19915 228/300 Flush
19916 229/300 Read block 3526 len 36864
19917 230/300 Read block 4761 len 16384
19918 231/300 Read block 1525 len 8192
19919 232/300 Flush
19920 233/300 Read block 4313 len 4096
19921 234/300 Read block 2642 len 4096
19922 235/300 Read block 3493 len 36864
19923 236/300 Write block 1497 len 20480 data: 4 4 4 4 5
19924 237/300 Write block 2046 len 40960 data: 4 4 3 3 3 4 5 5 4 4
19925 238/300 Flush
19926 239/300 Read block 969 len 20480
19927 240/300 Write block 2115 len 12288 data: 4 5 4
19928 241/300 Write block 3283 len 16384 data: 4 5 5 6
19929 242/300 Flush
19930 243/300 Read block 3021 len 32768
19931 244/300 Flush
19932 245/300 Read block 3173 len 24576
19933 246/300 Read block 2847 len 4096
19934 247/300 Flush
19935 248/300 Write block 4111 len 12288 data: 4 4 4
19936 249/300 Write block 3376 len 16384 data: 4 4 4 4
19937 250/300 Read block 3109 len 28672
19938 251/300 Read block 3358 len 28672
19939 252/300 Read block 1324 len 24576
19940 253/300 Write block 326 len 28672 data: 4 4 5 4 3 3 2
19941 254/300 Write block 321 len 28672 data: 6 6 5 5 3 5 5
19942 255/300 Read block 890 len 12288
19943 256/300 Flush
19944 257/300 Write block 737 len 28672 data: 3 4 5 7 8 7 5
19945 258/300 Read block 4327 len 28672
19946 259/300 Read block 589 len 8192
19947 260/300 Read block 3378 len 28672
19948 261/300 Read block 2156 len 4096
19949 262/300 Read block 4590 len 28672
19950 263/300 Read block 2545 len 32768
19951 264/300 Read block 2771 len 40960
19952 265/300 Write block 1884 len 36864 data: 4 5 7 8 6 6 5 5 5
19953 266/300 Write block 3671 len 32768 data: 3 3 4 4 3 4 4 3
19954 267/300 Flush
19955 268/300 Write block 1948 len 32768 data: 3 2 2 2 3 4 4 4
19956 269/300 Flush
19957 270/300 Flush
19958 271/300 Read block 1877 len 24576
19959 272/300 Read block 3648 len 4096
19960 273/300 Write block 1362 len 36864 data: 3 5 5 4 4 4 5 6 9
19961 274/300 Write block 3006 len 28672 data: 7 6 5 5 5 5 6
19962 275/300 Write block 4575 len 24576 data: 4 4 4 5 5 5
19963 276/300 Read block 2299 len 24576
19964 277/300 Write block 1617 len 4096 data: 3
19965 278/300 Write block 1161 len 20480 data: 4 5 5 5 5
19966 279/300 Write block 554 len 8192 data: 4 4
19967 280/300 Read block 4511 len 8192
19968 281/300 Read block 4974 len 24576
19969 282/300 Write block 250 len 16384 data: 6 7 6 6
19970 283/300 Read block 3259 len 16384
19971 284/300 Write block 2891 len 40960 data: 5 5 6 6 6 5 5 4 5 5
19972 285/300 Write block 2808 len 28672 data: 3 3 4 4 3 4 4
19973 286/300 Read block 3485 len 28672
19974 287/300 Write block 720 len 8192 data: 5 5
19975 288/300 Write block 1268 len 4096 data: 5
19976 289/300 Write block 1452 len 40960 data: 6 6 5 4 4 5 5 5 5 4
19977 290/300 Flush
19978 291/300 Read block 2830 len 16384
19979 292/300 Read block 347 len 20480
19980 293/300 Write block 3955 len 4096 data: 3
19981 294/300 Write block 209 len 4096 data: 5
19982 295/300 Read block 974 len 40960
19983 296/300 Flush
19984 297/300 Write block 2154 len 36864 data: 4 3 3 3 2 2 2 3 3
19985 298/300 Read block 2191 len 4096
19986 299/300 Write block 2880 len 4096 data: 3
19987 300/300 Read block 2119 len 16384
19988 301/300 Write block 4530 len 36864 data: 3 3 3 3 3 4 3 2 4
19989 Main task has work to do, go find it
19990 got dsc Start(0)
19991 start 0
19992 Dec 09 15:12:03.805 INFO request completed, latency_us: 246, response_code: 204, uri: /start/cid/0, method: POST, req_id: 89cb6a07-80c2-44c6-9cc5-caf5a98c10ba, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
19993 Replay: started 0, returned:Ok(())
19994 ----------------------------------------------------------------
19995 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19996 States: Offline Active Active
19997 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19998 6436 Acked Read 9 Sent Done Done false
19999 6437 Acked Read 5 Sent Done Done false
20000 6438 Acked Read 3 Sent Done Done false
20001 6439 Acked Write 2 Sent Done Done false
20002 6440 Acked Write 7 Sent Done Done false
20003 6441 Acked Read 7 Sent Done Done false
20004 6442 Acked Write 4 Sent Done Done false
20005 6443 Acked Write 2 Sent Done Done false
20006 6444 Acked Flush 0 Sent Done Done false
20007 6445 Acked Read 3 Sent Done Done false
20008 6446 Acked Read 1 Sent Done Done false
20009 6447 Acked Read 8 Sent Done Done false
20010 6448 Acked Read 6 Sent Done Done false
20011 6449 Acked Read 8 Sent Done Done false
20012 6450 Acked Flush 0 Sent Done Done false
20013 6451 Acked Flush 0 Sent Done Done false
20014 6452 Acked Write 9 Sent Done Done false
20015 6453 Acked Write 5 Sent Done Done false
20016 6454 Acked Read 7 Sent Done Done false
20017 6455 Acked Read 7 Sent Done Done false
20018 6456 Acked Read 6 Sent Done Done false
20019 6457 Acked Read 7 Sent Done Done false
20020 6458 Acked Read 9 Sent Done Done false
20021 6459 Acked Write 8 Sent Done Done false
20022 6460 Acked Write 1 Sent Done Done false
20023 6461 Acked Flush 0 Sent Done Done false
20024 6462 Acked Flush 0 Sent Done Done false
20025 6463 Acked Flush 0 Sent Done Done false
20026 6464 Acked Read 4 Sent Done Done false
20027 6465 Acked Flush 0 Sent Done Done false
20028 6466 Acked Read 1 Sent Done Done false
20029 6467 Acked Write 6 Sent Done Done false
20030 6468 Acked Write 6 Sent Done Done false
20031 6469 Acked Write 5 Sent Done Done false
20032 6470 Acked Read 10 Sent Done Done false
20033 6471 Acked Read 5 Sent Done Done false
20034 6472 Acked Read 7 Sent Done Done false
20035 6473 Acked Write 2 Sent Done Done false
20036 6474 Acked Read 9 Sent Done Done false
20037 6475 Acked Write 2 Sent Done Done false
20038 6476 Acked Read 3 Sent Done Done false
20039 6477 Acked Read 7 Sent Done Done false
20040 6478 Acked Read 1 Sent Done Done false
20041 6479 Acked Flush 0 Sent Done Done false
20042 6480 Acked Read 2 Sent Done Done false
20043 6481 Acked Write 4 Sent Done Done false
20044 6482 Acked Read 10 Sent Done Done false
20045 6483 Acked Read 4 Sent Done Done false
20046 6484 Acked Read 4 Sent Done Done false
20047 6485 Acked Read 4 Sent Done Done false
20048 6486 Acked Read 6 Sent Done Done false
20049 6487 Acked Flush 0 Sent Done Done false
20050 6488 Acked Read 7 Sent Done Done false
20051 6489 Acked Write 6 Sent Done Done false
20052 6490 Acked Read 8 Sent Done Done false
20053 6491 Acked Read 7 Sent Done Done false
20054 6492 Acked Read 9 Sent Done Done false
20055 6493 Acked Read 8 Sent Done Done false
20056 6494 Acked Write 1 Sent Done Done false
20057 6495 Acked Read 9 Sent Done Done false
20058 6496 Acked Read 8 Sent Done Done false
20059 6497 Acked Read 3 Sent Done Done false
20060 6498 Acked Write 10 Sent Done Done false
20061 6499 Acked Write 10 Sent Done Done false
20062 6500 Acked Read 6 Sent Done Done false
20063 6501 Acked Read 8 Sent Done Done false
20064 6502 Acked Read 6 Sent Done Done false
20065 6503 Acked Read 5 Sent Done Done false
20066 6504 Acked Write 7 Sent Done Done false
20067 6505 Acked Write 10 Sent Done Done false
20068 6506 Acked Write 1 Sent Done Done false
20069 6507 Acked Write 4 Sent Done Done false
20070 6508 Acked Write 6 Sent Done Done false
20071 6509 Acked Flush 0 Sent Done Done false
20072 6510 Acked Read 1 Sent Done Done false
20073 6511 Acked Read 10 Sent Done Done false
20074 6512 Acked Read 6 Sent Done Done false
20075 6513 Acked Write 2 Sent Done Done false
20076 6514 Acked Write 5 Sent Done Done false
20077 6515 Acked Write 4 Sent Done Done false
20078 6516 Acked Write 9 Sent Done Done false
20079 6517 Acked Flush 0 Sent Done Done false
20080 6518 Acked Flush 0 Sent Done Done false
20081 6519 Acked Read 8 Sent Done Done false
20082 6520 Acked Read 4 Sent Done Done false
20083 6521 Acked Read 6 Sent Done Done false
20084 6522 Acked Read 6 Sent Done Done false
20085 6523 Acked Write 3 Sent Done Done false
20086 6524 Acked Write 2 Sent Done Done false
20087 6525 Acked Flush 0 Sent Done Done false
20088 6526 Acked Read 8 Sent Done Done false
20089 6527 Acked Read 9 Sent Done Done false
20090 6528 Acked Flush 0 Sent Done Done false
20091 6529 Acked Read 4 Sent Done Done false
20092 6530 Acked Write 9 Sent Done Done false
20093 6531 Acked Read 9 Sent Done Done false
20094 6532 Acked Write 7 Sent Done Done false
20095 6533 Acked Flush 0 Sent Done Done false
20096 6534 Acked Write 2 Sent Done Done false
20097 6535 Acked Read 10 Sent Done Done false
20098 6536 Acked Write 10 Sent Done Done false
20099 6537 Acked Write 2 Sent Done Done false
20100 6538 Acked Read 5 Sent Done Done false
20101 6539 Acked Write 8 Sent Done Done false
20102 6540 Acked Read 10 Sent Done Done false
20103 6541 Acked Read 1 Sent Done Done false
20104 6542 Acked Read 4 Sent Done Done false
20105 6543 Acked Write 9 Sent Done Done false
20106 6544 Acked Write 7 Sent Done Done false
20107 6545 Acked Read 9 Sent Done Done false
20108 6546 Acked Read 4 Sent Done Done false
20109 6547 Acked Read 5 Sent Done Done false
20110 6548 Acked Read 7 Sent Done Done false
20111 6549 Acked Write 6 Sent Done Done false
20112 6550 Acked Write 3 Sent Done Done false
20113 6551 Acked Write 5 Sent Done Done false
20114 6552 Acked Read 3 Sent Done Done false
20115 6553 Acked Read 1 Sent Done Done false
20116 6554 Acked Write 3 Sent Done Done false
20117 6555 Acked Read 6 Sent Done Done false
20118 6556 Acked Read 6 Sent Done Done false
20119 6557 Acked Read 1 Sent Done Done false
20120 6558 Acked Write 5 Sent Done Done false
20121 6559 Acked Write 3 Sent Done Done false
20122 6560 Acked Read 1 Sent Done Done false
20123 6561 Acked Write 10 Sent Done Done false
20124 6562 Acked Read 2 Sent Done Done false
20125 6563 Acked Write 1 Sent Done Done false
20126 6564 Acked Write 3 Sent Done Done false
20127 6565 Acked Read 1 Sent Done Done false
20128 6566 Acked Write 9 Sent Done Done false
20129 6567 Acked Write 4 Sent Done Done false
20130 6568 Acked Read 8 Sent Done Done false
20131 6569 Acked Read 1 Sent Done Done false
20132 6570 Acked Read 4 Sent Done Done false
20133 6571 Acked Write 8 Sent Done Done false
20134 6572 Acked Write 2 Sent Done Done false
20135 6573 Acked Flush 0 Sent Done Done false
20136 6574 Acked Write 6 Sent Done Done false
20137 6575 Acked Write 7 Sent Done Done false
20138 6576 Acked Write 1 Sent Done Done false
20139 6577 Acked Read 2 Sent Done Done false
20140 6578 Acked Flush 0 Sent Done Done false
20141 6579 Acked Read 9 Sent Done Done false
20142 6580 Acked Write 1 Sent Done Done false
20143 6581 Acked Read 6 Sent Done Done false
20144 6582 Acked Read 1 Sent Done Done false
20145 6583 Acked Read 9 Sent Done Done false
20146 6584 Acked Write 5 Sent Done Done false
20147 6585 Acked Read 3 Sent Done Done false
20148 6586 Acked Read 10 Sent Done Done false
20149 6587 Acked Write 3 Sent Done Done false
20150 6588 Acked Write 4 Sent Done Done false
20151 6589 Acked Read 5 Sent Done Done false
20152 6590 Acked Write 5 Sent Done Done false
20153 6591 Acked Read 3 Sent Done Done false
20154 6592 Acked Write 5 Sent Done Done false
20155 6593 Acked Write 1 Sent Done Done false
20156 6594 Acked Write 5 Sent Done Done false
20157 6595 Acked Read 8 Sent Done Done false
20158 6596 Acked Read 10 Sent Done Done false
20159 6597 Acked Write 4 Sent Done Done false
20160 6598 Acked Write 7 Sent Done Done false
20161 6599 Acked Read 8 Sent Done Done false
20162 6600 Acked Read 4 Sent Done Done false
20163 6601 Acked Read 1 Sent Done Done false
20164 6602 Acked Flush 0 Sent Done Done false
20165 6603 Acked Read 7 Sent Done Done false
20166 6604 Acked Write 5 Sent Done Done false
20167 6605 Acked Write 6 Sent Done Done false
20168 6606 Acked Read 4 Sent Done Done false
20169 6607 Acked Write 2 Sent Done Done false
20170 6608 Acked Read 2 Sent Done Done false
20171 6609 Acked Read 2 Sent Done Done false
20172 6610 Acked Read 4 Sent Done Done false
20173 6611 Acked Read 9 Sent Done Done false
20174 6612 Acked Flush 0 Sent Done Done false
20175 6613 Acked Read 8 Sent Done Done false
20176 6614 Acked Flush 0 Sent Done Done false
20177 6615 Acked Flush 0 Sent Done Done false
20178 6616 Acked Write 6 Sent Done Done false
20179 6617 Acked Write 9 Sent Done Done false
20180 6618 Acked Read 6 Sent Done Done false
20181 6619 Acked Read 7 Sent Done Done false
20182 6620 Acked Read 9 Sent Done Done false
20183 6621 Acked Write 6 Sent Done Done false
20184 6622 Acked Write 6 Sent Done Done false
20185 6623 Acked Write 7 Sent Done Done false
20186 6624 Acked Write 5 Sent Done Done false
20187 6625 Acked Read 9 Sent Done Done false
20188 6626 Acked Write 3 Sent Done Done false
20189 6627 Acked Write 1 Sent Done Done false
20190 6628 Acked Read 3 Sent Done Done false
20191 6629 Acked Write 7 Sent Done Done false
20192 6630 Acked Read 6 Sent Done Done false
20193 6631 Acked Read 2 Sent Done Done false
20194 6632 Acked Read 1 Sent Done Done false
20195 6633 Acked Write 7 Sent Done Done false
20196 6634 Acked Read 6 Sent Done Done false
20197 6635 Acked Read 3 Sent Done Done false
20198 6636 Acked Write 10 Sent Done Done false
20199 6637 Acked Flush 0 Sent Done Done false
20200 6638 Acked Write 1 Sent Done Done false
20201 6639 Acked Read 5 Sent Done Done false
20202 6640 Acked Flush 0 Sent Done Done false
20203 6641 Acked Read 3 Sent Done Done false
20204 6642 Acked Flush 0 Sent Done Done false
20205 6643 Acked Read 10 Sent Done Done false
20206 6644 Acked Write 8 Sent Done Done false
20207 6645 Acked Write 9 Sent Done Done false
20208 6646 Acked Read 5 Sent Done Done false
20209 6647 Acked Flush 0 Sent Done Done false
20210 6648 Acked Read 5 Sent Done Done false
20211 6649 Acked Read 9 Sent Done Done false
20212 6650 Acked Read 8 Sent Done Done false
20213 6651 Acked Read 10 Sent Done Done false
20214 6652 Acked Read 10 Sent Done Done false
20215 6653 Acked Read 6 Sent Done Done false
20216 6654 Acked Write 5 Sent Done Done false
20217 6655 Acked Flush 0 Sent Done Done false
20218 6656 Acked Read 6 Sent Done Done false
20219 6657 Acked Write 7 Sent Done Done false
20220 6658 Acked Write 10 Sent Done Done false
20221 6659 Acked Read 3 Sent Done Done false
20222 6660 Acked Write 8 Sent Done Done false
20223 6661 Acked Read 1 Sent Done Done false
20224 6662 Acked Read 1 Sent Done Done false
20225 6663 Acked Flush 0 Sent Done Done false
20226 6664 Acked Read 9 Sent Done Done false
20227 6665 Acked Read 4 Sent Done Done false
20228 6666 Acked Read 2 Sent Done Done false
20229 6667 Acked Flush 0 Sent Done Done false
20230 6668 Acked Read 1 Sent Done Done false
20231 6669 Acked Read 1 Sent Done Done false
20232 6670 Acked Read 9 Sent Done Done false
20233 6671 Acked Write 5 Sent Done Done false
20234 6672 Acked Write 10 Sent Done Done false
20235 6673 Acked Flush 0 Sent Done Done false
20236 6674 Acked Read 5 Sent Done Done false
20237 6675 Acked Write 3 Sent Done Done false
20238 6676 Acked Write 4 Sent Done Done false
20239 6677 Acked Flush 0 Sent Done Done false
20240 6678 Acked Read 8 Sent Done Done false
20241 6679 Acked Flush 0 Sent Done Done false
20242 6680 Acked Read 6 Sent Done Done false
20243 6681 Acked Read 1 Sent Done Done false
20244 6682 Acked Flush 0 Sent Done Done false
20245 6683 Acked Write 3 Sent Done Done false
20246 6684 Acked Write 4 Sent Done Done false
20247 6685 Acked Read 7 Sent Done Done false
20248 6686 Acked Read 7 Sent Done Done false
20249 6687 Acked Read 6 Sent Done Done false
20250 6688 Acked Write 7 Sent Done Done false
20251 6689 Acked Write 7 Sent Done Done false
20252 6690 Acked Read 3 Sent Done Done false
20253 6691 Acked Flush 0 Sent Done Done false
20254 6692 Acked Write 7 Sent Done Done false
20255 6693 Acked Read 7 Sent Done Done false
20256 6694 Acked Read 2 Sent Done Done false
20257 6695 Acked Read 7 Sent Done Done false
20258 6696 Acked Read 1 Sent Done Done false
20259 6697 Acked Read 7 Sent Done Done false
20260 6698 Acked Read 8 Sent Done Done false
20261 6699 Acked Read 10 Sent Done Done false
20262 6700 Acked Write 9 Sent Done Done false
20263 6701 Acked Write 8 Sent Done Done false
20264 6702 Acked Flush 0 Sent Done Done false
20265 6703 Acked Write 8 Sent Done Done false
20266 6704 Acked Flush 0 Sent Done Done false
20267 6705 Acked Flush 0 Sent Done Done false
20268 6706 Acked Read 6 Sent Done Done false
20269 6707 Acked Read 1 Sent Done Done false
20270 6708 Acked Write 9 Sent Done Done false
20271 6709 Acked Write 7 Sent Done Done false
20272 6710 Acked Write 6 Sent Done Done false
20273 6711 Acked Read 6 Sent Done Done false
20274 6712 Acked Write 1 Sent Done Done false
20275 6713 Acked Write 5 Sent Done Done false
20276 6714 Acked Write 2 Sent Done Done false
20277 6715 Acked Read 2 Sent Done Done false
20278 6716 Acked Read 6 Sent Done Done false
20279 6717 Acked Write 4 Sent Done Done false
20280 6718 Acked Read 4 Sent Done Done false
20281 6719 Acked Write 10 Sent Done Done false
20282 6720 Acked Write 7 Sent Done Done false
20283 6721 Acked Read 7 Sent Done Done false
20284 6722 Acked Write 2 Sent Done Done false
20285 6723 Acked Write 1 Sent Done Done false
20286 6724 Acked Write 10 Sent Done Done false
20287 6725 Acked Flush 0 Sent Done Done false
20288 6726 Acked Read 4 Sent Done Done false
20289 6727 Acked Read 5 Sent Done Done false
20290 6728 Acked Write 1 Sent Done Done false
20291 6729 Acked Write 1 Sent Done Done false
20292 6730 Acked Read 10 Sent Done Done false
20293 6731 Acked Flush 0 Sent Done Done false
20294 6732 Acked Write 9 Sent Done Done false
20295 6733 Acked Read 1 Sent Done Done false
20296 6734 Acked Write 1 Sent Done Done false
20297 6735 Acked Read 4 Sent Done Done false
20298 6736 Acked Write 9 Sent Done Done false
20299 STATES DS:0 DS:1 DS:2 TOTAL
20300 Sent 301 0 0 301
20301 Done 0 301 301 602
20302 Skipped 0 0 0 0
20303 Error 0 0 0 0
20304 Last Flush: 6435 6731 6731
20305 Downstairs last five completed: 6435 6434 6433 6432 6431
20306 Upstairs last five completed: 6736 6735 6734 6733 6732
20307 CLIENT: Up:0 ds:301 act:2
20308 [8810] got start action
20309 [8810] I am going to restart
20310 Starting downstairs at port 8810
20311 Make output file at "/tmp/dsc/downstairs-8810.txt"
20312 [8810][0] reports Starting
20313 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1220)
20314 [8810][0] reports Running
20315 ----------------------------------------------------------------
20316 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
20317 States: Offline Active Active
20318 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20319 6436 Acked Read 9 Sent Done Done false
20320 6437 Acked Read 5 Sent Done Done false
20321 6438 Acked Read 3 Sent Done Done false
20322 6439 Acked Write 2 Sent Done Done false
20323 6440 Acked Write 7 Sent Done Done false
20324 6441 Acked Read 7 Sent Done Done false
20325 6442 Acked Write 4 Sent Done Done false
20326 6443 Acked Write 2 Sent Done Done false
20327 6444 Acked Flush 0 Sent Done Done false
20328 6445 Acked Read 3 Sent Done Done false
20329 6446 Acked Read 1 Sent Done Done false
20330 6447 Acked Read 8 Sent Done Done false
20331 6448 Acked Read 6 Sent Done Done false
20332 6449 Acked Read 8 Sent Done Done false
20333 6450 Acked Flush 0 Sent Done Done false
20334 6451 Acked Flush 0 Sent Done Done false
20335 6452 Acked Write 9 Sent Done Done false
20336 6453 Acked Write 5 Sent Done Done false
20337 6454 Acked Read 7 Sent Done Done false
20338 6455 Acked Read 7 Sent Done Done false
20339 6456 Acked Read 6 Sent Done Done false
20340 6457 Acked Read 7 Sent Done Done false
20341 6458 Acked Read 9 Sent Done Done false
20342 6459 Acked Write 8 Sent Done Done false
20343 6460 Acked Write 1 Sent Done Done false
20344 6461 Acked Flush 0 Sent Done Done false
20345 6462 Acked Flush 0 Sent Done Done false
20346 6463 Acked Flush 0 Sent Done Done false
20347 6464 Acked Read 4 Sent Done Done false
20348 6465 Acked Flush 0 Sent Done Done false
20349 6466 Acked Read 1 Sent Done Done false
20350 6467 Acked Write 6 Sent Done Done false
20351 6468 Acked Write 6 Sent Done Done false
20352 6469 Acked Write 5 Sent Done Done false
20353 6470 Acked Read 10 Sent Done Done false
20354 6471 Acked Read 5 Sent Done Done false
20355 6472 Acked Read 7 Sent Done Done false
20356 6473 Acked Write 2 Sent Done Done false
20357 6474 Acked Read 9 Sent Done Done false
20358 6475 Acked Write 2 Sent Done Done false
20359 6476 Acked Read 3 Sent Done Done false
20360 6477 Acked Read 7 Sent Done Done false
20361 6478 Acked Read 1 Sent Done Done false
20362 6479 Acked Flush 0 Sent Done Done false
20363 6480 Acked Read 2 Sent Done Done false
20364 6481 Acked Write 4 Sent Done Done false
20365 6482 Acked Read 10 Sent Done Done false
20366 6483 Acked Read 4 Sent Done Done false
20367 6484 Acked Read 4 Sent Done Done false
20368 6485 Acked Read 4 Sent Done Done false
20369 6486 Acked Read 6 Sent Done Done false
20370 6487 Acked Flush 0 Sent Done Done false
20371 6488 Acked Read 7 Sent Done Done false
20372 6489 Acked Write 6 Sent Done Done false
20373 6490 Acked Read 8 Sent Done Done false
20374 6491 Acked Read 7 Sent Done Done false
20375 6492 Acked Read 9 Sent Done Done false
20376 6493 Acked Read 8 Sent Done Done false
20377 6494 Acked Write 1 Sent Done Done false
20378 6495 Acked Read 9 Sent Done Done false
20379 6496 Acked Read 8 Sent Done Done false
20380 6497 Acked Read 3 Sent Done Done false
20381 6498 Acked Write 10 Sent Done Done false
20382 6499 Acked Write 10 Sent Done Done false
20383 6500 Acked Read 6 Sent Done Done false
20384 6501 Acked Read 8 Sent Done Done false
20385 6502 Acked Read 6 Sent Done Done false
20386 6503 Acked Read 5 Sent Done Done false
20387 6504 Acked Write 7 Sent Done Done false
20388 6505 Acked Write 10 Sent Done Done false
20389 6506 Acked Write 1 Sent Done Done false
20390 6507 Acked Write 4 Sent Done Done false
20391 6508 Acked Write 6 Sent Done Done false
20392 6509 Acked Flush 0 Sent Done Done false
20393 6510 Acked Read 1 Sent Done Done false
20394 6511 Acked Read 10 Sent Done Done false
20395 6512 Acked Read 6 Sent Done Done false
20396 6513 Acked Write 2 Sent Done Done false
20397 6514 Acked Write 5 Sent Done Done false
20398 6515 Acked Write 4 Sent Done Done false
20399 6516 Acked Write 9 Sent Done Done false
20400 6517 Acked Flush 0 Sent Done Done false
20401 6518 Acked Flush 0 Sent Done Done false
20402 6519 Acked Read 8 Sent Done Done false
20403 6520 Acked Read 4 Sent Done Done false
20404 6521 Acked Read 6 Sent Done Done false
20405 6522 Acked Read 6 Sent Done Done false
20406 6523 Acked Write 3 Sent Done Done false
20407 6524 Acked Write 2 Sent Done Done false
20408 6525 Acked Flush 0 Sent Done Done false
20409 6526 Acked Read 8 Sent Done Done false
20410 6527 Acked Read 9 Sent Done Done false
20411 6528 Acked Flush 0 Sent Done Done false
20412 6529 Acked Read 4 Sent Done Done false
20413 6530 Acked Write 9 Sent Done Done false
20414 6531 Acked Read 9 Sent Done Done false
20415 6532 Acked Write 7 Sent Done Done false
20416 6533 Acked Flush 0 Sent Done Done false
20417 6534 Acked Write 2 Sent Done Done false
20418 6535 Acked Read 10 Sent Done Done false
20419 6536 Acked Write 10 Sent Done Done false
20420 6537 Acked Write 2 Sent Done Done false
20421 6538 Acked Read 5 Sent Done Done false
20422 6539 Acked Write 8 Sent Done Done false
20423 6540 Acked Read 10 Sent Done Done false
20424 6541 Acked Read 1 Sent Done Done false
20425 6542 Acked Read 4 Sent Done Done false
20426 6543 Acked Write 9 Sent Done Done false
20427 6544 Acked Write 7 Sent Done Done false
20428 6545 Acked Read 9 Sent Done Done false
20429 6546 Acked Read 4 Sent Done Done false
20430 6547 Acked Read 5 Sent Done Done false
20431 6548 Acked Read 7 Sent Done Done false
20432 6549 Acked Write 6 Sent Done Done false
20433 6550 Acked Write 3 Sent Done Done false
20434 6551 Acked Write 5 Sent Done Done false
20435 6552 Acked Read 3 Sent Done Done false
20436 6553 Acked Read 1 Sent Done Done false
20437 6554 Acked Write 3 Sent Done Done false
20438 6555 Acked Read 6 Sent Done Done false
20439 6556 Acked Read 6 Sent Done Done false
20440 6557 Acked Read 1 Sent Done Done false
20441 6558 Acked Write 5 Sent Done Done false
20442 6559 Acked Write 3 Sent Done Done false
20443 6560 Acked Read 1 Sent Done Done false
20444 6561 Acked Write 10 Sent Done Done false
20445 6562 Acked Read 2 Sent Done Done false
20446 6563 Acked Write 1 Sent Done Done false
20447 6564 Acked Write 3 Sent Done Done false
20448 6565 Acked Read 1 Sent Done Done false
20449 6566 Acked Write 9 Sent Done Done false
20450 6567 Acked Write 4 Sent Done Done false
20451 6568 Acked Read 8 Sent Done Done false
20452 6569 Acked Read 1 Sent Done Done false
20453 6570 Acked Read 4 Sent Done Done false
20454 6571 Acked Write 8 Sent Done Done false
20455 6572 Acked Write 2 Sent Done Done false
20456 6573 Acked Flush 0 Sent Done Done false
20457 6574 Acked Write 6 Sent Done Done false
20458 6575 Acked Write 7 Sent Done Done false
20459 6576 Acked Write 1 Sent Done Done false
20460 6577 Acked Read 2 Sent Done Done false
20461 6578 Acked Flush 0 Sent Done Done false
20462 6579 Acked Read 9 Sent Done Done false
20463 6580 Acked Write 1 Sent Done Done false
20464 6581 Acked Read 6 Sent Done Done false
20465 6582 Acked Read 1 Sent Done Done false
20466 6583 Acked Read 9 Sent Done Done false
20467 6584 Acked Write 5 Sent Done Done false
20468 6585 Acked Read 3 Sent Done Done false
20469 6586 Acked Read 10 Sent Done Done false
20470 6587 Acked Write 3 Sent Done Done false
20471 6588 Acked Write 4 Sent Done Done false
20472 6589 Acked Read 5 Sent Done Done false
20473 6590 Acked Write 5 Sent Done Done false
20474 6591 Acked Read 3 Sent Done Done false
20475 6592 Acked Write 5 Sent Done Done false
20476 6593 Acked Write 1 Sent Done Done false
20477 6594 Acked Write 5 Sent Done Done false
20478 6595 Acked Read 8 Sent Done Done false
20479 6596 Acked Read 10 Sent Done Done false
20480 6597 Acked Write 4 Sent Done Done false
20481 6598 Acked Write 7 Sent Done Done false
20482 6599 Acked Read 8 Sent Done Done false
20483 6600 Acked Read 4 Sent Done Done false
20484 6601 Acked Read 1 Sent Done Done false
20485 6602 Acked Flush 0 Sent Done Done false
20486 6603 Acked Read 7 Sent Done Done false
20487 6604 Acked Write 5 Sent Done Done false
20488 6605 Acked Write 6 Sent Done Done false
20489 6606 Acked Read 4 Sent Done Done false
20490 6607 Acked Write 2 Sent Done Done false
20491 6608 Acked Read 2 Sent Done Done false
20492 6609 Acked Read 2 Sent Done Done false
20493 6610 Acked Read 4 Sent Done Done false
20494 6611 Acked Read 9 Sent Done Done false
20495 6612 Acked Flush 0 Sent Done Done false
20496 6613 Acked Read 8 Sent Done Done false
20497 6614 Acked Flush 0 Sent Done Done false
20498 6615 Acked Flush 0 Sent Done Done false
20499 6616 Acked Write 6 Sent Done Done false
20500 6617 Acked Write 9 Sent Done Done false
20501 6618 Acked Read 6 Sent Done Done false
20502 6619 Acked Read 7 Sent Done Done false
20503 6620 Acked Read 9 Sent Done Done false
20504 6621 Acked Write 6 Sent Done Done false
20505 6622 Acked Write 6 Sent Done Done false
20506 6623 Acked Write 7 Sent Done Done false
20507 6624 Acked Write 5 Sent Done Done false
20508 6625 Acked Read 9 Sent Done Done false
20509 6626 Acked Write 3 Sent Done Done false
20510 6627 Acked Write 1 Sent Done Done false
20511 6628 Acked Read 3 Sent Done Done false
20512 6629 Acked Write 7 Sent Done Done false
20513 6630 Acked Read 6 Sent Done Done false
20514 6631 Acked Read 2 Sent Done Done false
20515 6632 Acked Read 1 Sent Done Done false
20516 6633 Acked Write 7 Sent Done Done false
20517 6634 Acked Read 6 Sent Done Done false
20518 6635 Acked Read 3 Sent Done Done false
20519 6636 Acked Write 10 Sent Done Done false
20520 6637 Acked Flush 0 Sent Done Done false
20521 6638 Acked Write 1 Sent Done Done false
20522 6639 Acked Read 5 Sent Done Done false
20523 6640 Acked Flush 0 Sent Done Done false
20524 6641 Acked Read 3 Sent Done Done false
20525 6642 Acked Flush 0 Sent Done Done false
20526 6643 Acked Read 10 Sent Done Done false
20527 6644 Acked Write 8 Sent Done Done false
20528 6645 Acked Write 9 Sent Done Done false
20529 6646 Acked Read 5 Sent Done Done false
20530 6647 Acked Flush 0 Sent Done Done false
20531 6648 Acked Read 5 Sent Done Done false
20532 6649 Acked Read 9 Sent Done Done false
20533 6650 Acked Read 8 Sent Done Done false
20534 6651 Acked Read 10 Sent Done Done false
20535 6652 Acked Read 10 Sent Done Done false
20536 6653 Acked Read 6 Sent Done Done false
20537 6654 Acked Write 5 Sent Done Done false
20538 6655 Acked Flush 0 Sent Done Done false
20539 6656 Acked Read 6 Sent Done Done false
20540 6657 Acked Write 7 Sent Done Done false
20541 6658 Acked Write 10 Sent Done Done false
20542 6659 Acked Read 3 Sent Done Done false
20543 6660 Acked Write 8 Sent Done Done false
20544 6661 Acked Read 1 Sent Done Done false
20545 6662 Acked Read 1 Sent Done Done false
20546 6663 Acked Flush 0 Sent Done Done false
20547 6664 Acked Read 9 Sent Done Done false
20548 6665 Acked Read 4 Sent Done Done false
20549 6666 Acked Read 2 Sent Done Done false
20550 6667 Acked Flush 0 Sent Done Done false
20551 6668 Acked Read 1 Sent Done Done false
20552 6669 Acked Read 1 Sent Done Done false
20553 6670 Acked Read 9 Sent Done Done false
20554 6671 Acked Write 5 Sent Done Done false
20555 6672 Acked Write 10 Sent Done Done false
20556 6673 Acked Flush 0 Sent Done Done false
20557 6674 Acked Read 5 Sent Done Done false
20558 6675 Acked Write 3 Sent Done Done false
20559 6676 Acked Write 4 Sent Done Done false
20560 6677 Acked Flush 0 Sent Done Done false
20561 6678 Acked Read 8 Sent Done Done false
20562 6679 Acked Flush 0 Sent Done Done false
20563 6680 Acked Read 6 Sent Done Done false
20564 6681 Acked Read 1 Sent Done Done false
20565 6682 Acked Flush 0 Sent Done Done false
20566 6683 Acked Write 3 Sent Done Done false
20567 6684 Acked Write 4 Sent Done Done false
20568 6685 Acked Read 7 Sent Done Done false
20569 6686 Acked Read 7 Sent Done Done false
20570 6687 Acked Read 6 Sent Done Done false
20571 6688 Acked Write 7 Sent Done Done false
20572 6689 Acked Write 7 Sent Done Done false
20573 6690 Acked Read 3 Sent Done Done false
20574 6691 Acked Flush 0 Sent Done Done false
20575 6692 Acked Write 7 Sent Done Done false
20576 6693 Acked Read 7 Sent Done Done false
20577 6694 Acked Read 2 Sent Done Done false
20578 6695 Acked Read 7 Sent Done Done false
20579 6696 Acked Read 1 Sent Done Done false
20580 6697 Acked Read 7 Sent Done Done false
20581 6698 Acked Read 8 Sent Done Done false
20582 6699 Acked Read 10 Sent Done Done false
20583 6700 Acked Write 9 Sent Done Done false
20584 6701 Acked Write 8 Sent Done Done false
20585 6702 Acked Flush 0 Sent Done Done false
20586 6703 Acked Write 8 Sent Done Done false
20587 6704 Acked Flush 0 Sent Done Done false
20588 6705 Acked Flush 0 Sent Done Done false
20589 6706 Acked Read 6 Sent Done Done false
20590 6707 Acked Read 1 Sent Done Done false
20591 6708 Acked Write 9 Sent Done Done false
20592 6709 Acked Write 7 Sent Done Done false
20593 6710 Acked Write 6 Sent Done Done false
20594 6711 Acked Read 6 Sent Done Done false
20595 6712 Acked Write 1 Sent Done Done false
20596 6713 Acked Write 5 Sent Done Done false
20597 6714 Acked Write 2 Sent Done Done false
20598 6715 Acked Read 2 Sent Done Done false
20599 6716 Acked Read 6 Sent Done Done false
20600 6717 Acked Write 4 Sent Done Done false
20601 6718 Acked Read 4 Sent Done Done false
20602 6719 Acked Write 10 Sent Done Done false
20603 6720 Acked Write 7 Sent Done Done false
20604 6721 Acked Read 7 Sent Done Done false
20605 6722 Acked Write 2 Sent Done Done false
20606 6723 Acked Write 1 Sent Done Done false
20607 6724 Acked Write 10 Sent Done Done false
20608 6725 Acked Flush 0 Sent Done Done false
20609 6726 Acked Read 4 Sent Done Done false
20610 6727 Acked Read 5 Sent Done Done false
20611 6728 Acked Write 1 Sent Done Done false
20612 6729 Acked Write 1 Sent Done Done false
20613 6730 Acked Read 10 Sent Done Done false
20614 6731 Acked Flush 0 Sent Done Done false
20615 6732 Acked Write 9 Sent Done Done false
20616 6733 Acked Read 1 Sent Done Done false
20617 6734 Acked Write 1 Sent Done Done false
20618 6735 Acked Read 4 Sent Done Done false
20619 6736 Acked Write 9 Sent Done Done false
20620 6737 Acked Flush 0 Sent Done Done false
20621 STATES DS:0 DS:1 DS:2 TOTAL
20622 Sent 302 0 0 302
20623 Done 0 302 302 604
20624 Skipped 0 0 0 0
20625 Error 0 0 0 0
20626 Last Flush: 6435 6737 6737
20627 Downstairs last five completed: 6435 6434 6433 6432 6431
20628 Upstairs last five completed: 6737 6736 6735 6734 6733
20629 CLIENT: Up:0 ds:302 act:2
206302024-12-09T15:12:09.568ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206312024-12-09T15:12:09.568ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47156) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206322024-12-09T15:12:09.570ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206332024-12-09T15:12:09.570ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206342024-12-09T15:12:09.570ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206352024-12-09T15:12:09.570ZINFOcrucible: send last flush ID to this DS: 6435 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206362024-12-09T15:12:09.570ZINFOcrucible: Replied this last flush ID: 6435 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206372024-12-09T15:12:09.570ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206382024-12-09T15:12:09.570ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206392024-12-09T15:12:09.570ZINFOcrucible: [0] client re-new 302 jobs since flush 6435 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206402024-12-09T15:12:09.571ZINFOcrucible: [0] Replayed 302 jobs since flush: 6435 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
20641 ----------------------------------------------------------------
20642 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20643 Downstairs last five completed: 6737 6736 6735 6734 6733
20644 Upstairs last five completed: 6737 6736 6735 6734 6733
20645 CLIENT: Up:0 ds:0 act:3
20646 Replay: All jobs finished
20647 Main task has work to do, go find it
20648 got dsc Stop(1)
20649 stop 1
20650 Dec 09 15:12:11.810 [8820] Got stop action so:false kr:false
20651 INFO request completed, latency_us: 346, response_code: 204, uri: /stop/cid/1, method: POST, req_id: a57df59a-3a63-4d4b-95fe-045bd5ae0a1e, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
206522024-12-09T15:12:11.811ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206532024-12-09T15:12:11.811ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206542024-12-09T15:12:11.811ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206552024-12-09T15:12:11.811ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
206562024-12-09T15:12:11.811ZINFOcrucible: [1] Transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
20657 Dec 09 15:12:11.811 INFO request completed, latency_us: 139, response_code: 200, uri: /state/cid/1, method: GET, req_id: 01045053-0733-4176-b258-59be93054a9a, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
206582024-12-09T15:12:11.811ZINFOcrucible: client task is exiting = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
20659 [8820] Exited with: ExitStatus(unix_wait_status(9))
20660 [8820][1] reports Exit
20661 Dec 09 15:12:15.814 INFO request completed, latency_us: 244, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4f815c55-c2b1-4e9c-8ffb-6bb4fd27348c, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
20662 001/300 Read block 2738 len 36864
20663 002/300 Write block 3846 len 32768 data: 7 6 6 6 5 4 5 5
20664 003/300 Flush
20665 004/300 Read block 4034 len 20480
20666 005/300 Write block 3798 len 20480 data: 6 7 8 8 9
20667 006/300 Write block 2187 len 40960 data: 4 5 5 5 4 4 5 5 5 6
20668 007/300 Flush
20669 008/300 Write block 1475 len 36864 data: 6 6 4 3 4 4 3 3 2
20670 009/300 Write block 3316 len 20480 data: 5 4 4 3 6
20671 010/300 Write block 829 len 12288 data: 3 4 4
20672 011/300 Read block 4004 len 28672
20673 012/300 Write block 1829 len 24576 data: 6 5 5 4 3 4
20674 013/300 Read block 2949 len 12288
20675 014/300 Write block 3193 len 20480 data: 2 2 4 4 5
20676 015/300 Write block 3903 len 40960 data: 3 4 4 4 4 3 2 2 2 3
20677 016/300 Write block 2409 len 16384 data: 6 5 5 5
20678 017/300 Read block 3585 len 32768
20679 018/300 Read block 4650 len 16384
20680 019/300 Read block 2783 len 32768
20681 020/300 Read block 1678 len 16384
20682 021/300 Write block 1144 len 4096 data: 4
20683 022/300 Write block 653 len 16384 data: 4 4 5 3
20684 023/300 Read block 840 len 12288
20685 024/300 Write block 3873 len 40960 data: 6 5 5 8 7 5 5 8 6 5
20686 025/300 Write block 2367 len 40960 data: 4 4 5 4 3 3 4 3 3 3
20687 026/300 Write block 1797 len 32768 data: 7 6 4 3 2 3 4 5
20688 027/300 Read block 3336 len 24576
20689 028/300 Write block 4725 len 4096 data: 4
20690 029/300 Flush
20691 030/300 Write block 3001 len 36864 data: 4 5 6 6 6 8 7 6 6
20692 031/300 Write block 2492 len 16384 data: 3 4 6 6
20693 032/300 Read block 4786 len 28672
20694 033/300 Read block 2562 len 32768
20695 034/300 Read block 1542 len 40960
20696 035/300 Read block 3975 len 24576
20697 036/300 Write block 2678 len 12288 data: 4 3 3
20698 037/300 Flush
20699 038/300 Read block 2693 len 8192
20700 039/300 Read block 2138 len 32768
20701 040/300 Read block 772 len 8192
20702 041/300 Read block 2138 len 16384
20703 042/300 Read block 4830 len 36864
20704 043/300 Read block 1595 len 16384
20705 044/300 Write block 1839 len 36864 data: 4 7 7 7 5 3 3 4 3
20706 045/300 Write block 4374 len 4096 data: 4
20707 046/300 Read block 3857 len 24576
20708 047/300 Write block 4021 len 20480 data: 7 4 4 4 4
20709 048/300 Read block 4113 len 20480
20710 049/300 Write block 4962 len 4096 data: 3
20711 050/300 Write block 2502 len 8192 data: 4 5
20712 051/300 Read block 539 len 20480
20713 052/300 Read block 1480 len 20480
20714 053/300 Read block 1360 len 36864
20715 054/300 Write block 4221 len 20480 data: 3 4 7 7 7
20716 055/300 Read block 3661 len 4096
20717 056/300 Read block 2370 len 12288
20718 057/300 Write block 1624 len 12288 data: 4 4 2
20719 058/300 Write block 532 len 28672 data: 3 3 3 2 4 4 5
20720 059/300 Flush
20721 060/300 Read block 1190 len 12288
20722 061/300 Write block 222 len 40960 data: 6 7 6 5 4 7 5 4 5 6
20723 062/300 Flush
20724 063/300 Write block 1069 len 4096 data: 5
20725 064/300 Write block 2240 len 24576 data: 3 5 5 5 5 4
20726 065/300 Flush
20727 066/300 Write block 3874 len 28672 data: 6 6 9 8 6 6 9
20728 067/300 Write block 3177 len 8192 data: 5 4
20729 068/300 Write block 1264 len 12288 data: 3 3 4
20730 069/300 Write block 2079 len 24576 data: 5 7 6 7 5 3
20731 070/300 Read block 4925 len 40960
20732 071/300 Read block 748 len 28672
20733 072/300 Write block 3460 len 20480 data: 5 5 5 6 7
20734 073/300 Flush
20735 074/300 Write block 3298 len 28672 data: 3 3 3 4 4 3 3
20736 075/300 Read block 4827 len 40960
20737 076/300 Read block 1975 len 28672
20738 077/300 Read block 1656 len 28672
20739 078/300 Write block 1308 len 28672 data: 3 3 3 6 7 7 6
20740 079/300 Read block 942 len 4096
20741 080/300 Write block 2386 len 16384 data: 4 5 6 6
20742 081/300 Read block 1564 len 4096
20743 082/300 Write block 2477 len 36864 data: 4 5 6 7 6 6 7 7 7
20744 083/300 Read block 4474 len 40960
20745 084/300 Write block 3577 len 40960 data: 2 3 3 4 5 6 8 9 7 8
20746 085/300 Write block 4841 len 4096 data: 7
20747 086/300 Write block 2618 len 40960 data: 6 7 7 6 6 8 7 8 5 4
20748 087/300 Write block 1858 len 16384 data: 4 4 4 4
20749 088/300 Write block 1132 len 4096 data: 6
20750 089/300 Read block 2887 len 4096
20751 090/300 Write block 3005 len 24576 data: 7 9 8 7 7 6
20752 091/300 Write block 1955 len 24576 data: 5 4 4 4 4 4
20753 092/300 Read block 1071 len 40960
20754 093/300 Read block 2817 len 4096
20755 094/300 Write block 3991 len 16384 data: 3 4 4 4
20756 095/300 Write block 688 len 24576 data: 9 8 7 6 5 5
20757 096/300 Read block 4979 len 8192
20758 097/300 Read block 2629 len 28672
20759 098/300 Read block 197 len 32768
20760 099/300 Write block 2295 len 36864 data: 3 5 4 5 6 5 5 4 4
20761 100/300 Read block 1374 len 28672
20762 101/300 Write block 1863 len 36864 data: 2 3 5 6 5 5 7 7 7
20763 102/300 Read block 2366 len 36864
20764 103/300 Read block 238 len 40960
20765 104/300 Write block 1193 len 40960 data: 6 7 5 6 4 6 6 6 6 5
20766 105/300 Write block 4828 len 36864 data: 3 2 3 2 4 3 4 5 5
20767 106/300 Write block 2921 len 12288 data: 4 5 4
20768 107/300 Write block 4346 len 20480 data: 6 7 6 8 7
20769 108/300 Write block 3309 len 24576 data: 5 4 6 6 5 6
20770 109/300 Read block 4764 len 36864
20771 110/300 Write block 1488 len 20480 data: 3 3 3 3 3
20772 111/300 Read block 2533 len 28672
20773 112/300 Read block 2790 len 32768
20774 113/300 Read block 774 len 8192
20775 114/300 Flush
20776 115/300 Flush
20777 116/300 Write block 1399 len 36864 data: 4 5 3 3 3 5 5 4 5
20778 117/300 Write block 2565 len 40960 data: 3 3 3 2 2 2 3 4 4 5
20779 118/300 Write block 3720 len 20480 data: 4 5 4 4 4
20780 119/300 Write block 4091 len 24576 data: 5 5 3 4 6 9
20781 120/300 Read block 2435 len 8192
20782 121/300 Write block 555 len 12288 data: 5 5 4
20783 122/300 Read block 2501 len 32768
20784 123/300 Read block 1019 len 40960
20785 124/300 Read block 1939 len 4096
20786 125/300 Flush
20787 126/300 Read block 2959 len 40960
20788 127/300 Write block 3101 len 40960 data: 5 4 4 5 5 4 3 3 3 3
20789 128/300 Write block 2757 len 16384 data: 5 4 3 4
20790 129/300 Read block 4830 len 32768
20791 130/300 Write block 4291 len 4096 data: 4
20792 131/300 Read block 2229 len 20480
20793 132/300 Write block 2089 len 32768 data: 4 4 4 2 2 3 4 4
20794 133/300 Write block 1552 len 20480 data: 3 5 3 3 5
20795 134/300 Read block 3269 len 20480
20796 135/300 Write block 1907 len 8192 data: 5 5
20797 136/300 Write block 664 len 36864 data: 6 5 5 6 6 4 5 6 6
20798 137/300 Read block 1774 len 20480
20799 138/300 Write block 3545 len 36864 data: 4 3 3 4 5 5 5 4 6
20800 139/300 Read block 4288 len 40960
20801 140/300 Write block 218 len 16384 data: 9 9 8 5
20802 141/300 Read block 1009 len 28672
20803 142/300 Write block 1746 len 16384 data: 5 4 4 4
20804 143/300 Write block 2931 len 32768 data: 10 9 9 7 7 5 5 6
20805 144/300 Read block 3436 len 4096
20806 145/300 Write block 999 len 40960 data: 3 3 3 4 5 5 5 6 4 5
20807 146/300 Write block 1693 len 16384 data: 6 5 5 5
20808 147/300 Read block 2173 len 8192
20809 148/300 Write block 2085 len 4096 data: 3
20810 149/300 Write block 598 len 24576 data: 2 2 2 2 2 3
20811 150/300 Write block 970 len 12288 data: 2 2 3
20812 151/300 Read block 1549 len 32768
20813 152/300 Read block 1992 len 4096
20814 153/300 Write block 3234 len 16384 data: 5 5 5 6
20815 154/300 Read block 3688 len 20480
20816 155/300 Read block 4484 len 28672
20817 156/300 Read block 3458 len 20480
20818 157/300 Read block 3742 len 16384
20819 158/300 Write block 3818 len 32768 data: 4 3 3 4 5 7 9 8
20820 159/300 Write block 2679 len 16384 data: 4 4 3 2
20821 160/300 Read block 134 len 40960
20822 161/300 Read block 1783 len 40960
20823 162/300 Write block 3209 len 40960 data: 3 3 5 5 4 4 5 5 5 4
20824 163/300 Write block 2465 len 40960 data: 5 5 5 4 4 4 4 3 3 3
20825 164/300 Write block 1307 len 28672 data: 3 4 4 4 7 8 8
20826 165/300 Write block 661 len 24576 data: 5 5 6 7 6 6
20827 166/300 Read block 187 len 12288
20828 167/300 Read block 3654 len 16384
20829 168/300 Write block 4026 len 32768 data: 4 4 4 5 4 4 5 4
20830 169/300 Read block 1917 len 8192
20831 170/300 Write block 3992 len 24576 data: 5 5 5 4 2 2
20832 171/300 Write block 2907 len 40960 data: 5 5 4 3 2 2 3 3 3 3
20833 172/300 Flush
20834 173/300 Read block 460 len 36864
20835 174/300 Read block 4263 len 24576
20836 175/300 Write block 184 len 24576 data: 4 5 5 6 5 6
20837 176/300 Write block 3021 len 32768 data: 5 5 5 6 7 6 4 3
20838 177/300 Read block 4738 len 20480
20839 178/300 Read block 416 len 40960
20840 179/300 Read block 3373 len 20480
20841 180/300 Read block 133 len 16384
20842 181/300 Read block 513 len 36864
20843 182/300 Flush
20844 183/300 Read block 82 len 16384
20845 184/300 Write block 4860 len 20480 data: 3 4 3 3 3
20846 185/300 Read block 686 len 24576
20847 186/300 Flush
20848 187/300 Write block 4496 len 32768 data: 5 4 4 6 6 6 7 7
20849 188/300 Flush
20850 189/300 Read block 3103 len 4096
20851 190/300 Read block 1299 len 28672
20852 191/300 Write block 2238 len 32768 data: 3 2 4 6 6 6 6 5
20853 192/300 Write block 397 len 40960 data: 4 3 3 3 4 4 5 6 6 5
20854 193/300 Write block 4448 len 16384 data: 5 5 4 4
20855 194/300 Read block 2968 len 24576
20856 195/300 Read block 2029 len 36864
20857 196/300 Write block 4706 len 36864 data: 5 6 7 7 5 5 5 6 6
20858 197/300 Write block 1174 len 36864 data: 5 4 5 5 4 4 3 4 4
20859 198/300 Read block 4134 len 16384
20860 199/300 Read block 2556 len 28672
20861 200/300 Read block 1729 len 12288
20862 201/300 Read block 2302 len 24576
20863 202/300 Write block 3999 len 40960 data: 3 3 4 4 5 5 5 6 6 5
20864 203/300 Flush
20865 204/300 Read block 2360 len 8192
20866 205/300 Read block 2812 len 20480
20867 206/300 Write block 2564 len 12288 data: 3 4 4
20868 207/300 Write block 1272 len 24576 data: 7 7 7 7 7 7
20869 208/300 Read block 2582 len 16384
20870 209/300 Write block 3621 len 36864 data: 7 5 4 3 2 2 2 2 3
20871 210/300 Read block 4982 len 16384
20872 211/300 Read block 296 len 40960
20873 212/300 Read block 977 len 28672
20874 213/300 Write block 4683 len 16384 data: 5 5 4 3
20875 214/300 Write block 4186 len 36864 data: 7 6 6 6 7 6 7 4 3
20876 215/300 Write block 3197 len 4096 data: 6
20877 216/300 Write block 3154 len 40960 data: 8 7 6 6 6 7 8 9 9 7
20878 217/300 Write block 2504 len 16384 data: 4 4 4 4
20879 218/300 Read block 2456 len 20480
20880 219/300 Write block 3650 len 24576 data: 3 4 4 5 4 4
20881 220/300 Read block 4605 len 16384
20882 221/300 Read block 4221 len 28672
20883 222/300 Write block 340 len 28672 data: 3 3 3 3 4 4 5
20884 223/300 Write block 1845 len 8192 data: 4 5
20885 224/300 Read block 2220 len 36864
20886 225/300 Read block 3412 len 32768
20887 226/300 Read block 3430 len 12288
20888 227/300 Read block 2058 len 40960
20889 228/300 Read block 4731 len 8192
20890 229/300 Read block 1555 len 16384
20891 230/300 Read block 3759 len 40960
20892 231/300 Read block 3839 len 28672
20893 232/300 Read block 4720 len 4096
20894 233/300 Write block 3254 len 28672 data: 4 4 6 5 5 6 5
20895 234/300 Write block 3062 len 16384 data: 5 5 3 3
20896 235/300 Read block 436 len 36864
20897 236/300 Read block 745 len 24576
20898 237/300 Write block 2733 len 16384 data: 5 6 7 6
20899 238/300 Flush
20900 239/300 Flush
20901 240/300 Write block 1237 len 28672 data: 3 2 3 3 3 4 4
20902 241/300 Write block 1501 len 16384 data: 6 5 5 3
20903 242/300 Write block 367 len 40960 data: 7 6 4 4 4 4 3 3 3 3
20904 243/300 Read block 1082 len 32768
20905 244/300 Write block 723 len 16384 data: 4 4 3 4
20906 245/300 Read block 1505 len 28672
20907 246/300 Flush
20908 247/300 Read block 4793 len 12288
20909 248/300 Flush
20910 249/300 Write block 3463 len 32768 data: 7 8 6 7 8 8 7 7
20911 250/300 Flush
20912 251/300 Read block 2935 len 24576
20913 252/300 Read block 4223 len 20480
20914 253/300 Write block 3911 len 32768 data: 3 4 3 3 3 4 4 4
20915 254/300 Read block 1326 len 40960
20916 255/300 Read block 3721 len 12288
20917 256/300 Write block 1504 len 24576 data: 4 3 3 3 5 5
20918 257/300 Write block 686 len 40960 data: 8 8 10 9 8 7 6 6 5 3
20919 258/300 Read block 131 len 20480
20920 259/300 Flush
20921 260/300 Write block 1592 len 28672 data: 3 2 2 3 3 4 4
20922 261/300 Write block 1330 len 16384 data: 2 2 4 3
20923 262/300 Read block 2990 len 40960
20924 263/300 Write block 4359 len 36864 data: 5 6 6 6 5 3 2 2 2
20925 264/300 Read block 4361 len 40960
20926 265/300 Flush
20927 266/300 Read block 1433 len 24576
20928 267/300 Write block 3688 len 4096 data: 4
20929 268/300 Read block 1164 len 28672
20930 269/300 Flush
20931 270/300 Write block 941 len 4096 data: 4
20932 271/300 Write block 2618 len 4096 data: 7
20933 272/300 Write block 322 len 4096 data: 7
20934 273/300 Read block 2747 len 40960
20935 274/300 Write block 4653 len 24576 data: 6 7 7 7 7 6
20936 275/300 Write block 2238 len 16384 data: 4 3 5 7
20937 276/300 Flush
20938 277/300 Read block 4989 len 12288
20939 278/300 Read block 1550 len 16384
20940 279/300 Write block 2617 len 40960 data: 6 8 8 8 7 7 9 8 9 6
20941 280/300 Read block 804 len 4096
20942 281/300 Write block 482 len 36864 data: 9 7 7 5 4 3 3 3 2
20943 282/300 Flush
20944 283/300 Write block 3552 len 20480 data: 5 7 7 7 8
20945 284/300 Write block 4105 len 20480 data: 9 8 6 5 5
20946 285/300 Read block 4302 len 20480
20947 286/300 Write block 2007 len 8192 data: 7 8
20948 287/300 Read block 1461 len 16384
20949 288/300 Write block 455 len 28672 data: 3 3 4 4 4 5 5
20950 289/300 Write block 1976 len 36864 data: 2 3 5 5 6 8 7 7 8
20951 290/300 Write block 2017 len 40960 data: 7 8 5 5 5 7 6 3 3 5
20952 291/300 Read block 2781 len 20480
20953 292/300 Write block 1731 len 8192 data: 5 5
20954 293/300 Flush
20955 294/300 Write block 1708 len 4096 data: 2
20956 295/300 Write block 3708 len 36864 data: 3 4 4 5 5 5 4 5 5
20957 296/300 Read block 4710 len 4096
20958 297/300 Read block 2465 len 20480
20959 298/300 Write block 527 len 20480 data: 5 5 4 4 3
20960 299/300 Write block 1932 len 36864 data: 6 6 5 4 5 3 3 2 2
20961 300/300 Write block 2046 len 16384 data: 5 5 4 4
20962 301/300 Read block 1367 len 32768
20963 Main task has work to do, go find it
20964 got dsc Start(1)
20965 start 1
20966 Dec 09 15:12:16.054 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/1, method: POST, req_id: 6866eee8-13a6-4254-ad8b-bf8203c1fd73, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
20967 Replay: started 1, returned:Ok(())
20968 ----------------------------------------------------------------
20969 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20970 States: Active Offline Active
20971 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20972 6738 Acked Read 9 Done Sent Done false
20973 6739 Acked Write 8 Done Sent Done false
20974 6740 Acked Flush 0 Done Sent Done false
20975 6741 Acked Read 5 Done Sent Done false
20976 6742 Acked Write 5 Done Sent Done false
20977 6743 Acked Write 10 Done Sent Done false
20978 6744 Acked Flush 0 Done Sent Done false
20979 6745 Acked Write 9 Done Sent Done false
20980 6746 Acked Write 5 Done Sent Done false
20981 6747 Acked Write 3 Done Sent Done false
20982 6748 Acked Read 7 Done Sent Done false
20983 6749 Acked Write 6 Done Sent Done false
20984 6750 Acked Read 3 Done Sent Done false
20985 6751 Acked Write 5 Done Sent Done false
20986 6752 Acked Write 10 Done Sent Done false
20987 6753 Acked Write 4 Done Sent Done false
20988 6754 Acked Read 8 Done Sent Done false
20989 6755 Acked Read 4 Done Sent Done false
20990 6756 Acked Read 8 Done Sent Done false
20991 6757 Acked Read 4 Done Sent Done false
20992 6758 Acked Write 1 Done Sent Done false
20993 6759 Acked Write 4 Done Sent Done false
20994 6760 Acked Read 3 Done Sent Done false
20995 6761 Acked Write 10 Done Sent Done false
20996 6762 Acked Write 10 Done Sent Done false
20997 6763 Acked Write 8 Done Sent Done false
20998 6764 Acked Read 6 Done Sent Done false
20999 6765 Acked Write 1 Done Sent Done false
21000 6766 Acked Flush 0 Done Sent Done false
21001 6767 Acked Write 9 Done Sent Done false
21002 6768 Acked Write 4 Done Sent Done false
21003 6769 Acked Read 7 Done Sent Done false
21004 6770 Acked Read 8 Done Sent Done false
21005 6771 Acked Read 10 Done Sent Done false
21006 6772 Acked Read 6 Done Sent Done false
21007 6773 Acked Write 3 Done Sent Done false
21008 6774 Acked Flush 0 Done Sent Done false
21009 6775 Acked Read 2 Done Sent Done false
21010 6776 Acked Read 8 Done Sent Done false
21011 6777 Acked Read 2 Done Sent Done false
21012 6778 Acked Read 4 Done Sent Done false
21013 6779 Acked Read 9 Done Sent Done false
21014 6780 Acked Read 4 Done Sent Done false
21015 6781 Acked Write 9 Done Sent Done false
21016 6782 Acked Write 1 Done Sent Done false
21017 6783 Acked Read 6 Done Sent Done false
21018 6784 Acked Write 5 Done Sent Done false
21019 6785 Acked Read 5 Done Sent Done false
21020 6786 Acked Write 1 Done Sent Done false
21021 6787 Acked Write 2 Done Sent Done false
21022 6788 Acked Read 5 Done Sent Done false
21023 6789 Acked Read 5 Done Sent Done false
21024 6790 Acked Read 9 Done Sent Done false
21025 6791 Acked Write 5 Done Sent Done false
21026 6792 Acked Read 1 Done Sent Done false
21027 6793 Acked Read 3 Done Sent Done false
21028 6794 Acked Write 3 Done Sent Done false
21029 6795 Acked Write 7 Done Sent Done false
21030 6796 Acked Flush 0 Done Sent Done false
21031 6797 Acked Read 3 Done Sent Done false
21032 6798 Acked Write 10 Done Sent Done false
21033 6799 Acked Flush 0 Done Sent Done false
21034 6800 Acked Write 1 Done Sent Done false
21035 6801 Acked Write 6 Done Sent Done false
21036 6802 Acked Flush 0 Done Sent Done false
21037 6803 Acked Write 7 Done Sent Done false
21038 6804 Acked Write 2 Done Sent Done false
21039 6805 Acked Write 3 Done Sent Done false
21040 6806 Acked Write 6 Done Sent Done false
21041 6807 Acked Read 10 Done Sent Done false
21042 6808 Acked Read 7 Done Sent Done false
21043 6809 Acked Write 5 Done Sent Done false
21044 6810 Acked Flush 0 Done Sent Done false
21045 6811 Acked Write 7 Done Sent Done false
21046 6812 Acked Read 10 Done Sent Done false
21047 6813 Acked Read 7 Done Sent Done false
21048 6814 Acked Read 7 Done Sent Done false
21049 6815 Acked Write 7 Done Sent Done false
21050 6816 Acked Read 1 Done Sent Done false
21051 6817 Acked Write 4 Done Sent Done false
21052 6818 Acked Read 1 Done Sent Done false
21053 6819 Acked Write 9 Done Sent Done false
21054 6820 Acked Read 10 Done Sent Done false
21055 6821 Acked Write 10 Done Sent Done false
21056 6822 Acked Write 1 Done Sent Done false
21057 6823 Acked Write 10 Done Sent Done false
21058 6824 Acked Write 4 Done Sent Done false
21059 6825 Acked Write 1 Done Sent Done false
21060 6826 Acked Read 1 Done Sent Done false
21061 6827 Acked Write 6 Done Sent Done false
21062 6828 Acked Write 6 Done Sent Done false
21063 6829 Acked Read 10 Done Sent Done false
21064 6830 Acked Read 1 Done Sent Done false
21065 6831 Acked Write 4 Done Sent Done false
21066 6832 Acked Write 6 Done Sent Done false
21067 6833 Acked Read 2 Done Sent Done false
21068 6834 Acked Read 7 Done Sent Done false
21069 6835 Acked Read 8 Done Sent Done false
21070 6836 Acked Write 9 Done Sent Done false
21071 6837 Acked Read 7 Done Sent Done false
21072 6838 Acked Write 9 Done Sent Done false
21073 6839 Acked Read 9 Done Sent Done false
21074 6840 Acked Read 10 Done Sent Done false
21075 6841 Acked Write 10 Done Sent Done false
21076 6842 Acked Write 9 Done Sent Done false
21077 6843 Acked Write 3 Done Sent Done false
21078 6844 Acked Write 5 Done Sent Done false
21079 6845 Acked Write 6 Done Sent Done false
21080 6846 Acked Read 9 Done Sent Done false
21081 6847 Acked Write 5 Done Sent Done false
21082 6848 Acked Read 7 Done Sent Done false
21083 6849 Acked Read 8 Done Sent Done false
21084 6850 Acked Read 2 Done Sent Done false
21085 6851 Acked Flush 0 Done Sent Done false
21086 6852 Acked Flush 0 Done Sent Done false
21087 6853 Acked Write 9 Done Sent Done false
21088 6854 Acked Write 10 Done Sent Done false
21089 6855 Acked Write 5 Done Sent Done false
21090 6856 Acked Write 6 Done Sent Done false
21091 6857 Acked Read 2 Done Sent Done false
21092 6858 Acked Write 3 Done Sent Done false
21093 6859 Acked Read 8 Done Sent Done false
21094 6860 Acked Read 10 Done Sent Done false
21095 6861 Acked Read 1 Done Sent Done false
21096 6862 Acked Flush 0 Done Sent Done false
21097 6863 Acked Read 10 Done Sent Done false
21098 6864 Acked Write 10 Done Sent Done false
21099 6865 Acked Write 4 Done Sent Done false
21100 6866 Acked Read 8 Done Sent Done false
21101 6867 Acked Write 1 Done Sent Done false
21102 6868 Acked Read 5 Done Sent Done false
21103 6869 Acked Write 8 Done Sent Done false
21104 6870 Acked Write 5 Done Sent Done false
21105 6871 Acked Read 5 Done Sent Done false
21106 6872 Acked Write 2 Done Sent Done false
21107 6873 Acked Write 9 Done Sent Done false
21108 6874 Acked Read 5 Done Sent Done false
21109 6875 Acked Write 9 Done Sent Done false
21110 6876 Acked Read 10 Done Sent Done false
21111 6877 Acked Write 4 Done Sent Done false
21112 6878 Acked Read 7 Done Sent Done false
21113 6879 Acked Write 4 Done Sent Done false
21114 6880 Acked Write 8 Done Sent Done false
21115 6881 Acked Read 1 Done Sent Done false
21116 6882 Acked Write 10 Done Sent Done false
21117 6883 Acked Write 4 Done Sent Done false
21118 6884 Acked Read 2 Done Sent Done false
21119 6885 Acked Write 1 Done Sent Done false
21120 6886 Acked Write 6 Done Sent Done false
21121 6887 Acked Write 3 Done Sent Done false
21122 6888 Acked Read 8 Done Sent Done false
21123 6889 Acked Read 1 Done Sent Done false
21124 6890 Acked Write 4 Done Sent Done false
21125 6891 Acked Read 5 Done Sent Done false
21126 6892 Acked Read 7 Done Sent Done false
21127 6893 Acked Read 5 Done Sent Done false
21128 6894 Acked Read 4 Done Sent Done false
21129 6895 Acked Write 8 Done Sent Done false
21130 6896 Acked Write 4 Done Sent Done false
21131 6897 Acked Read 10 Done Sent Done false
21132 6898 Acked Read 10 Done Sent Done false
21133 6899 Acked Write 10 Done Sent Done false
21134 6900 Acked Write 10 Done Sent Done false
21135 6901 Acked Write 7 Done Sent Done false
21136 6902 Acked Write 6 Done Sent Done false
21137 6903 Acked Read 3 Done Sent Done false
21138 6904 Acked Read 4 Done Sent Done false
21139 6905 Acked Write 8 Done Sent Done false
21140 6906 Acked Read 2 Done Sent Done false
21141 6907 Acked Write 6 Done Sent Done false
21142 6908 Acked Write 10 Done Sent Done false
21143 6909 Acked Flush 0 Done Sent Done false
21144 6910 Acked Read 9 Done Sent Done false
21145 6911 Acked Read 6 Done Sent Done false
21146 6912 Acked Write 6 Done Sent Done false
21147 6913 Acked Write 8 Done Sent Done false
21148 6914 Acked Read 5 Done Sent Done false
21149 6915 Acked Read 10 Done Sent Done false
21150 6916 Acked Read 5 Done Sent Done false
21151 6917 Acked Read 4 Done Sent Done false
21152 6918 Acked Read 9 Done Sent Done false
21153 6919 Acked Flush 0 Done Sent Done false
21154 6920 Acked Read 4 Done Sent Done false
21155 6921 Acked Write 5 Done Sent Done false
21156 6922 Acked Read 6 Done Sent Done false
21157 6923 Acked Flush 0 Done Sent Done false
21158 6924 Acked Write 8 Done Sent Done false
21159 6925 Acked Flush 0 Done Sent Done false
21160 6926 Acked Read 1 Done Sent Done false
21161 6927 Acked Read 7 Done Sent Done false
21162 6928 Acked Write 8 Done Sent Done false
21163 6929 Acked Write 10 Done Sent Done false
21164 6930 Acked Write 4 Done Sent Done false
21165 6931 Acked Read 6 Done Sent Done false
21166 6932 Acked Read 9 Done Sent Done false
21167 6933 Acked Write 9 Done Sent Done false
21168 6934 Acked Write 9 Done Sent Done false
21169 6935 Acked Read 4 Done Sent Done false
21170 6936 Acked Read 7 Done Sent Done false
21171 6937 Acked Read 3 Done Sent Done false
21172 6938 Acked Read 6 Done Sent Done false
21173 6939 Acked Write 10 Done Sent Done false
21174 6940 Acked Flush 0 Done Sent Done false
21175 6941 Acked Read 2 Done Sent Done false
21176 6942 Acked Read 5 Done Sent Done false
21177 6943 Acked Write 3 Done Sent Done false
21178 6944 Acked Write 6 Done Sent Done false
21179 6945 Acked Read 4 Done Sent Done false
21180 6946 Acked Write 9 Done Sent Done false
21181 6947 Acked Read 4 Done Sent Done false
21182 6948 Acked Read 10 Done Sent Done false
21183 6949 Acked Read 7 Done Sent Done false
21184 6950 Acked Write 4 Done Sent Done false
21185 6951 Acked Write 9 Done Sent Done false
21186 6952 Acked Write 1 Done Sent Done false
21187 6953 Acked Write 10 Done Sent Done false
21188 6954 Acked Write 4 Done Sent Done false
21189 6955 Acked Read 5 Done Sent Done false
21190 6956 Acked Write 6 Done Sent Done false
21191 6957 Acked Read 4 Done Sent Done false
21192 6958 Acked Read 7 Done Sent Done false
21193 6959 Acked Write 7 Done Sent Done false
21194 6960 Acked Write 2 Done Sent Done false
21195 6961 Acked Read 9 Done Sent Done false
21196 6962 Acked Read 8 Done Sent Done false
21197 6963 Acked Read 3 Done Sent Done false
21198 6964 Acked Read 10 Done Sent Done false
21199 6965 Acked Read 2 Done Sent Done false
21200 6966 Acked Read 4 Done Sent Done false
21201 6967 Acked Read 10 Done Sent Done false
21202 6968 Acked Read 7 Done Sent Done false
21203 6969 Acked Read 1 Done Sent Done false
21204 6970 Acked Write 7 Done Sent Done false
21205 6971 Acked Write 4 Done Sent Done false
21206 6972 Acked Read 9 Done Sent Done false
21207 6973 Acked Read 6 Done Sent Done false
21208 6974 Acked Write 4 Done Sent Done false
21209 6975 Acked Flush 0 Done Sent Done false
21210 6976 Acked Flush 0 Done Sent Done false
21211 6977 Acked Write 7 Done Sent Done false
21212 6978 Acked Write 4 Done Sent Done false
21213 6979 Acked Write 10 Done Sent Done false
21214 6980 Acked Read 8 Done Sent Done false
21215 6981 Acked Write 4 Done Sent Done false
21216 6982 Acked Read 7 Done Sent Done false
21217 6983 Acked Flush 0 Done Sent Done false
21218 6984 Acked Read 3 Done Sent Done false
21219 6985 Acked Flush 0 Done Sent Done false
21220 6986 Acked Write 8 Done Sent Done false
21221 6987 Acked Flush 0 Done Sent Done false
21222 6988 Acked Read 6 Done Sent Done false
21223 6989 Acked Read 5 Done Sent Done false
21224 6990 Acked Write 8 Done Sent Done false
21225 6991 Acked Read 10 Done Sent Done false
21226 6992 Acked Read 3 Done Sent Done false
21227 6993 Acked Write 6 Done Sent Done false
21228 6994 Acked Write 10 Done Sent Done false
21229 6995 Acked Read 5 Done Sent Done false
21230 6996 Acked Flush 0 Done Sent Done false
21231 6997 Acked Write 7 Done Sent Done false
21232 6998 Acked Write 4 Done Sent Done false
21233 6999 Acked Read 10 Done Sent Done false
21234 7000 Acked Write 9 Done Sent Done false
21235 7001 Acked Read 10 Done Sent Done false
21236 7002 Acked Flush 0 Done Sent Done false
21237 7003 Acked Read 6 Done Sent Done false
21238 7004 Acked Write 1 Done Sent Done false
21239 7005 Acked Read 7 Done Sent Done false
21240 7006 Acked Flush 0 Done Sent Done false
21241 7007 Acked Write 1 Done Sent Done false
21242 7008 Acked Write 1 Done Sent Done false
21243 7009 Acked Write 1 Done Sent Done false
21244 7010 Acked Read 10 Done Sent Done false
21245 7011 Acked Write 6 Done Sent Done false
21246 7012 Acked Write 4 Done Sent Done false
21247 7013 Acked Flush 0 Done Sent Done false
21248 7014 Acked Read 3 Done Sent Done false
21249 7015 Acked Read 4 Done Sent Done false
21250 7016 Acked Write 10 Done Sent Done false
21251 7017 Acked Read 1 Done Sent Done false
21252 7018 Acked Write 9 Done Sent Done false
21253 7019 Acked Flush 0 Done Sent Done false
21254 7020 Acked Write 5 Done Sent Done false
21255 7021 Acked Write 5 Done Sent Done false
21256 7022 Acked Read 5 Done Sent Done false
21257 7023 Acked Write 2 Done Sent Done false
21258 7024 Acked Read 4 Done Sent Done false
21259 7025 Acked Write 7 Done Sent Done false
21260 7026 Acked Write 9 Done Sent Done false
21261 7027 Acked Write 10 Done Sent Done false
21262 7028 Acked Read 5 Done Sent Done false
21263 7029 Acked Write 2 Done Sent Done false
21264 7030 Acked Flush 0 Done Sent Done false
21265 7031 Acked Write 1 Done Sent Done false
21266 7032 Acked Write 9 Done Sent Done false
21267 7033 Acked Read 1 Done Sent Done false
21268 7034 Acked Read 5 Done Sent Done false
21269 7035 Acked Write 5 Done Sent Done false
21270 7036 Acked Write 9 Done Sent Done false
21271 7037 Acked Write 4 Done Sent Done false
21272 7038 Acked Read 8 Done Sent Done false
21273 STATES DS:0 DS:1 DS:2 TOTAL
21274 Sent 0 301 0 301
21275 Done 301 0 301 602
21276 Skipped 0 0 0 0
21277 Error 0 0 0 0
21278 Last Flush: 7030 6737 7030
21279 Downstairs last five completed: 6737 6736 6735 6734 6733
21280 Upstairs last five completed: 7038 7037 7036 7035 7034
21281 CLIENT: Up:0 ds:301 act:2
21282 ----------------------------------------------------------------
21283 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
21284 States: Active Offline Active
21285 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21286 6738 Acked Read 9 Done Sent Done false
21287 6739 Acked Write 8 Done Sent Done false
21288 6740 Acked Flush 0 Done Sent Done false
21289 6741 Acked Read 5 Done Sent Done false
21290 6742 Acked Write 5 Done Sent Done false
21291 6743 Acked Write 10 Done Sent Done false
21292 6744 Acked Flush 0 Done Sent Done false
21293 6745 Acked Write 9 Done Sent Done false
21294 6746 Acked Write 5 Done Sent Done false
21295 6747 Acked Write 3 Done Sent Done false
21296 6748 Acked Read 7 Done Sent Done false
21297 6749 Acked Write 6 Done Sent Done false
21298 6750 Acked Read 3 Done Sent Done false
21299 6751 Acked Write 5 Done Sent Done false
21300 6752 Acked Write 10 Done Sent Done false
21301 6753 Acked Write 4 Done Sent Done false
21302 6754 Acked Read 8 Done Sent Done false
21303 6755 Acked Read 4 Done Sent Done false
21304 6756 Acked Read 8 Done Sent Done false
21305 6757 Acked Read 4 Done Sent Done false
21306 6758 Acked Write 1 Done Sent Done false
21307 6759 Acked Write 4 Done Sent Done false
21308 6760 Acked Read 3 Done Sent Done false
21309 6761 Acked Write 10 Done Sent Done false
21310 6762 Acked Write 10 Done Sent Done false
21311 6763 Acked Write 8 Done Sent Done false
21312 6764 Acked Read 6 Done Sent Done false
21313 6765 Acked Write 1 Done Sent Done false
21314 6766 Acked Flush 0 Done Sent Done false
21315 6767 Acked Write 9 Done Sent Done false
21316 6768 Acked Write 4 Done Sent Done false
21317 6769 Acked Read 7 Done Sent Done false
21318 6770 Acked Read 8 Done Sent Done false
21319 6771 Acked Read 10 Done Sent Done false
21320 6772 Acked Read 6 Done Sent Done false
21321 6773 Acked Write 3 Done Sent Done false
21322 6774 Acked Flush 0 Done Sent Done false
21323 6775 Acked Read 2 Done Sent Done false
21324 6776 Acked Read 8 Done Sent Done false
21325 6777 Acked Read 2 Done Sent Done false
21326 6778 Acked Read 4 Done Sent Done false
21327 6779 Acked Read 9 Done Sent Done false
21328 6780 Acked Read 4 Done Sent Done false
21329 6781 Acked Write 9 Done Sent Done false
21330 6782 Acked Write 1 Done Sent Done false
21331 6783 Acked Read 6 Done Sent Done false
21332 6784 Acked Write 5 Done Sent Done false
21333 6785 Acked Read 5 Done Sent Done false
21334 6786 Acked Write 1 Done Sent Done false
21335 6787 Acked Write 2 Done Sent Done false
21336 6788 Acked Read 5 Done Sent Done false
21337 6789 Acked Read 5 Done Sent Done false
21338 6790 Acked Read 9 Done Sent Done false
21339 6791 Acked Write 5 Done Sent Done false
21340 6792 Acked Read 1 Done Sent Done false
21341 6793 Acked Read 3 Done Sent Done false
21342 6794 Acked Write 3 Done Sent Done false
21343 6795 Acked Write 7 Done Sent Done false
21344 6796 Acked Flush 0 Done Sent Done false
21345 6797 Acked Read 3 Done Sent Done false
21346 6798 Acked Write 10 Done Sent Done false
21347 6799 Acked Flush 0 Done Sent Done false
21348 6800 Acked Write 1 Done Sent Done false
21349 6801 Acked Write 6 Done Sent Done false
21350 6802 Acked Flush 0 Done Sent Done false
21351 6803 Acked Write 7 Done Sent Done false
21352 6804 Acked Write 2 Done Sent Done false
21353 6805 Acked Write 3 Done Sent Done false
21354 6806 Acked Write 6 Done Sent Done false
21355 6807 Acked Read 10 Done Sent Done false
21356 6808 Acked Read 7 Done Sent Done false
21357 6809 Acked Write 5 Done Sent Done false
21358 6810 Acked Flush 0 Done Sent Done false
21359 6811 Acked Write 7 Done Sent Done false
21360 6812 Acked Read 10 Done Sent Done false
21361 6813 Acked Read 7 Done Sent Done false
21362 6814 Acked Read 7 Done Sent Done false
21363 6815 Acked Write 7 Done Sent Done false
21364 6816 Acked Read 1 Done Sent Done false
21365 6817 Acked Write 4 Done Sent Done false
21366 6818 Acked Read 1 Done Sent Done false
21367 6819 Acked Write 9 Done Sent Done false
21368 6820 Acked Read 10 Done Sent Done false
21369 6821 Acked Write 10 Done Sent Done false
21370 6822 Acked Write 1 Done Sent Done false
21371 6823 Acked Write 10 Done Sent Done false
21372 6824 Acked Write 4 Done Sent Done false
21373 6825 Acked Write 1 Done Sent Done false
21374 6826 Acked Read 1 Done Sent Done false
21375 6827 Acked Write 6 Done Sent Done false
21376 6828 Acked Write 6 Done Sent Done false
21377 6829 Acked Read 10 Done Sent Done false
21378 6830 Acked Read 1 Done Sent Done false
21379 6831 Acked Write 4 Done Sent Done false
21380 6832 Acked Write 6 Done Sent Done false
21381 6833 Acked Read 2 Done Sent Done false
21382 6834 Acked Read 7 Done Sent Done false
21383 6835 Acked Read 8 Done Sent Done false
21384 6836 Acked Write 9 Done Sent Done false
21385 6837 Acked Read 7 Done Sent Done false
21386 6838 Acked Write 9 Done Sent Done false
21387 6839 Acked Read 9 Done Sent Done false
21388 6840 Acked Read 10 Done Sent Done false
21389 6841 Acked Write 10 Done Sent Done false
21390 6842 Acked Write 9 Done Sent Done false
21391 6843 Acked Write 3 Done Sent Done false
21392 6844 Acked Write 5 Done Sent Done false
21393 6845 Acked Write 6 Done Sent Done false
21394 6846 Acked Read 9 Done Sent Done false
21395 6847 Acked Write 5 Done Sent Done false
21396 6848 Acked Read 7 Done Sent Done false
21397 6849 Acked Read 8 Done Sent Done false
21398 6850 Acked Read 2 Done Sent Done false
21399 6851 Acked Flush 0 Done Sent Done false
21400 6852 Acked Flush 0 Done Sent Done false
21401 6853 Acked Write 9 Done Sent Done false
21402 6854 Acked Write 10 Done Sent Done false
21403 6855 Acked Write 5 Done Sent Done false
21404 6856 Acked Write 6 Done Sent Done false
21405 6857 Acked Read 2 Done Sent Done false
21406 6858 Acked Write 3 Done Sent Done false
21407 6859 Acked Read 8 Done Sent Done false
21408 6860 Acked Read 10 Done Sent Done false
21409 6861 Acked Read 1 Done Sent Done false
21410 6862 Acked Flush 0 Done Sent Done false
21411 6863 Acked Read 10 Done Sent Done false
21412 6864 Acked Write 10 Done Sent Done false
21413 6865 Acked Write 4 Done Sent Done false
21414 6866 Acked Read 8 Done Sent Done false
21415 6867 Acked Write 1 Done Sent Done false
21416 6868 Acked Read 5 Done Sent Done false
21417 6869 Acked Write 8 Done Sent Done false
21418 6870 Acked Write 5 Done Sent Done false
21419 6871 Acked Read 5 Done Sent Done false
21420 6872 Acked Write 2 Done Sent Done false
21421 6873 Acked Write 9 Done Sent Done false
21422 6874 Acked Read 5 Done Sent Done false
21423 6875 Acked Write 9 Done Sent Done false
21424 6876 Acked Read 10 Done Sent Done false
21425 6877 Acked Write 4 Done Sent Done false
21426 6878 Acked Read 7 Done Sent Done false
21427 6879 Acked Write 4 Done Sent Done false
21428 6880 Acked Write 8 Done Sent Done false
21429 6881 Acked Read 1 Done Sent Done false
21430 6882 Acked Write 10 Done Sent Done false
21431 6883 Acked Write 4 Done Sent Done false
21432 6884 Acked Read 2 Done Sent Done false
21433 6885 Acked Write 1 Done Sent Done false
21434 6886 Acked Write 6 Done Sent Done false
21435 6887 Acked Write 3 Done Sent Done false
21436 6888 Acked Read 8 Done Sent Done false
21437 6889 Acked Read 1 Done Sent Done false
21438 6890 Acked Write 4 Done Sent Done false
21439 6891 Acked Read 5 Done Sent Done false
21440 6892 Acked Read 7 Done Sent Done false
21441 6893 Acked Read 5 Done Sent Done false
21442 6894 Acked Read 4 Done Sent Done false
21443 6895 Acked Write 8 Done Sent Done false
21444 6896 Acked Write 4 Done Sent Done false
21445 6897 Acked Read 10 Done Sent Done false
21446 6898 Acked Read 10 Done Sent Done false
21447 6899 Acked Write 10 Done Sent Done false
21448 6900 Acked Write 10 Done Sent Done false
21449 6901 Acked Write 7 Done Sent Done false
21450 6902 Acked Write 6 Done Sent Done false
21451 6903 Acked Read 3 Done Sent Done false
21452 6904 Acked Read 4 Done Sent Done false
21453 6905 Acked Write 8 Done Sent Done false
21454 6906 Acked Read 2 Done Sent Done false
21455 6907 Acked Write 6 Done Sent Done false
21456 6908 Acked Write 10 Done Sent Done false
21457 6909 Acked Flush 0 Done Sent Done false
21458 6910 Acked Read 9 Done Sent Done false
21459 6911 Acked Read 6 Done Sent Done false
21460 6912 Acked Write 6 Done Sent Done false
21461 6913 Acked Write 8 Done Sent Done false
21462 6914 Acked Read 5 Done Sent Done false
21463 6915 Acked Read 10 Done Sent Done false
21464 6916 Acked Read 5 Done Sent Done false
21465 6917 Acked Read 4 Done Sent Done false
21466 6918 Acked Read 9 Done Sent Done false
21467 6919 Acked Flush 0 Done Sent Done false
21468 6920 Acked Read 4 Done Sent Done false
21469 6921 Acked Write 5 Done Sent Done false
21470 6922 Acked Read 6 Done Sent Done false
21471 6923 Acked Flush 0 Done Sent Done false
21472 6924 Acked Write 8 Done Sent Done false
21473 6925 Acked Flush 0 Done Sent Done false
21474 6926 Acked Read 1 Done Sent Done false
21475 6927 Acked Read 7 Done Sent Done false
21476 6928 Acked Write 8 Done Sent Done false
21477 6929 Acked Write 10 Done Sent Done false
21478 6930 Acked Write 4 Done Sent Done false
21479 6931 Acked Read 6 Done Sent Done false
21480 6932 Acked Read 9 Done Sent Done false
21481 6933 Acked Write 9 Done Sent Done false
21482 6934 Acked Write 9 Done Sent Done false
21483 6935 Acked Read 4 Done Sent Done false
21484 6936 Acked Read 7 Done Sent Done false
21485 6937 Acked Read 3 Done Sent Done false
21486 6938 Acked Read 6 Done Sent Done false
21487 6939 Acked Write 10 Done Sent Done false
21488 6940 Acked Flush 0 Done Sent Done false
21489 6941 Acked Read 2 Done Sent Done false
21490 6942 Acked Read 5 Done Sent Done false
21491 6943 Acked Write 3 Done Sent Done false
21492 6944 Acked Write 6 Done Sent Done false
21493 6945 Acked Read 4 Done Sent Done false
21494 6946 Acked Write 9 Done Sent Done false
21495 6947 Acked Read 4 Done Sent Done false
21496 6948 Acked Read 10 Done Sent Done false
21497 6949 Acked Read 7 Done Sent Done false
21498 6950 Acked Write 4 Done Sent Done false
21499 6951 Acked Write 9 Done Sent Done false
21500 6952 Acked Write 1 Done Sent Done false
21501 6953 Acked Write 10 Done Sent Done false
21502 6954 Acked Write 4 Done Sent Done false
21503 6955 Acked Read 5 Done Sent Done false
21504 6956 Acked Write 6 Done Sent Done false
21505 6957 Acked Read 4 Done Sent Done false
21506 6958 Acked Read 7 Done Sent Done false
21507 6959 Acked Write 7 Done Sent Done false
21508 6960 Acked Write 2 Done Sent Done false
21509 6961 Acked Read 9 Done Sent Done false
21510 6962 Acked Read 8 Done Sent Done false
21511 6963 Acked Read 3 Done Sent Done false
21512 6964 Acked Read 10 Done Sent Done false
21513 6965 Acked Read 2 Done Sent Done false
21514 6966 Acked Read 4 Done Sent Done false
21515 6967 Acked Read 10 Done Sent Done false
21516 6968 Acked Read 7 Done Sent Done false
21517 6969 Acked Read 1 Done Sent Done false
21518 6970 Acked Write 7 Done Sent Done false
21519 6971 Acked Write 4 Done Sent Done false
21520 6972 Acked Read 9 Done Sent Done false
21521 6973 Acked Read 6 Done Sent Done false
21522 6974 Acked Write 4 Done Sent Done false
21523 6975 Acked Flush 0 Done Sent Done false
21524 6976 Acked Flush 0 Done Sent Done false
21525 6977 Acked Write 7 Done Sent Done false
21526 6978 Acked Write 4 Done Sent Done false
21527 6979 Acked Write 10 Done Sent Done false
21528 6980 Acked Read 8 Done Sent Done false
21529 6981 Acked Write 4 Done Sent Done false
21530 6982 Acked Read 7 Done Sent Done false
21531 6983 Acked Flush 0 Done Sent Done false
21532 6984 Acked Read 3 Done Sent Done false
21533 6985 Acked Flush 0 Done Sent Done false
21534 6986 Acked Write 8 Done Sent Done false
21535 6987 Acked Flush 0 Done Sent Done false
21536 6988 Acked Read 6 Done Sent Done false
21537 6989 Acked Read 5 Done Sent Done false
21538 6990 Acked Write 8 Done Sent Done false
21539 6991 Acked Read 10 Done Sent Done false
21540 6992 Acked Read 3 Done Sent Done false
21541 6993 Acked Write 6 Done Sent Done false
21542 6994 Acked Write 10 Done Sent Done false
21543 6995 Acked Read 5 Done Sent Done false
21544 6996 Acked Flush 0 Done Sent Done false
21545 6997 Acked Write 7 Done Sent Done false
21546 6998 Acked Write 4 Done Sent Done false
21547 6999 Acked Read 10 Done Sent Done false
21548 7000 Acked Write 9 Done Sent Done false
21549 7001 Acked Read 10 Done Sent Done false
21550 7002 Acked Flush 0 Done Sent Done false
21551 7003 Acked Read 6 Done Sent Done false
21552 7004 Acked Write 1 Done Sent Done false
21553 7005 Acked Read 7 Done Sent Done false
21554 7006 Acked Flush 0 Done Sent Done false
21555 7007 Acked Write 1 Done Sent Done false
21556 7008 Acked Write 1 Done Sent Done false
21557 7009 Acked Write 1 Done Sent Done false
21558 7010 Acked Read 10 Done Sent Done false
21559 7011 Acked Write 6 Done Sent Done false
21560 7012 Acked Write 4 Done Sent Done false
21561 7013 Acked Flush 0 Done Sent Done false
21562 7014 Acked Read 3 Done Sent Done false
21563 7015 Acked Read 4 Done Sent Done false
21564 7016 Acked Write 10 Done Sent Done false
21565 7017 Acked Read 1 Done Sent Done false
21566 7018 Acked Write 9 Done Sent Done false
21567 7019 Acked Flush 0 Done Sent Done false
21568 7020 Acked Write 5 Done Sent Done false
21569 7021 Acked Write 5 Done Sent Done false
21570 7022 Acked Read 5 Done Sent Done false
21571 7023 Acked Write 2 Done Sent Done false
21572 7024 Acked Read 4 Done Sent Done false
21573 7025 Acked Write 7 Done Sent Done false
21574 7026 Acked Write 9 Done Sent Done false
21575 7027 Acked Write 10 Done Sent Done false
21576 7028 Acked Read 5 Done Sent Done false
21577 7029 Acked Write 2 Done Sent Done false
21578 7030 Acked Flush 0 Done Sent Done false
21579 7031 Acked Write 1 Done Sent Done false
21580 7032 Acked Write 9 Done Sent Done false
21581 7033 Acked Read 1 Done Sent Done false
21582 7034 Acked Read 5 Done Sent Done false
21583 7035 Acked Write 5 Done Sent Done false
21584 7036 Acked Write 9 Done Sent Done false
21585 7037 Acked Write 4 Done Sent Done false
21586 7038 Acked Read 8 Done Sent Done false
21587 7039 Acked Flush 0 Done Sent Done false
21588 STATES DS:0 DS:1 DS:2 TOTAL
21589 Sent 0 302 0 302
21590 Done 302 0 302 604
21591 Skipped 0 0 0 0
21592 Error 0 0 0 0
21593 Last Flush: 7039 6737 7039
21594 Downstairs last five completed: 6737 6736 6735 6734 6733
21595 Upstairs last five completed: 7039 7038 7037 7036 7035
21596 CLIENT: Up:0 ds:302 act:2
215972024-12-09T15:12:21.812ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
215982024-12-09T15:12:21.812ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
215992024-12-09T15:12:21.812ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
216002024-12-09T15:12:21.812ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
216012024-12-09T15:12:21.812ZINFOcrucible: client task is exiting = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
21602 [8820] got start action
21603 [8820] I am going to restart
21604 Starting downstairs at port 8820
21605 Make output file at "/tmp/dsc/downstairs-8820.txt"
21606 [8820][1] reports Starting
21607 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1221)
21608 [8820][1] reports Running
21609 ----------------------------------------------------------------
21610 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
21611 States: Active Offline Active
21612 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21613 6738 Acked Read 9 Done Sent Done false
21614 6739 Acked Write 8 Done Sent Done false
21615 6740 Acked Flush 0 Done Sent Done false
21616 6741 Acked Read 5 Done Sent Done false
21617 6742 Acked Write 5 Done Sent Done false
21618 6743 Acked Write 10 Done Sent Done false
21619 6744 Acked Flush 0 Done Sent Done false
21620 6745 Acked Write 9 Done Sent Done false
21621 6746 Acked Write 5 Done Sent Done false
21622 6747 Acked Write 3 Done Sent Done false
21623 6748 Acked Read 7 Done Sent Done false
21624 6749 Acked Write 6 Done Sent Done false
21625 6750 Acked Read 3 Done Sent Done false
21626 6751 Acked Write 5 Done Sent Done false
21627 6752 Acked Write 10 Done Sent Done false
21628 6753 Acked Write 4 Done Sent Done false
21629 6754 Acked Read 8 Done Sent Done false
21630 6755 Acked Read 4 Done Sent Done false
21631 6756 Acked Read 8 Done Sent Done false
21632 6757 Acked Read 4 Done Sent Done false
21633 6758 Acked Write 1 Done Sent Done false
21634 6759 Acked Write 4 Done Sent Done false
21635 6760 Acked Read 3 Done Sent Done false
21636 6761 Acked Write 10 Done Sent Done false
21637 6762 Acked Write 10 Done Sent Done false
21638 6763 Acked Write 8 Done Sent Done false
21639 6764 Acked Read 6 Done Sent Done false
21640 6765 Acked Write 1 Done Sent Done false
21641 6766 Acked Flush 0 Done Sent Done false
21642 6767 Acked Write 9 Done Sent Done false
21643 6768 Acked Write 4 Done Sent Done false
21644 6769 Acked Read 7 Done Sent Done false
21645 6770 Acked Read 8 Done Sent Done false
21646 6771 Acked Read 10 Done Sent Done false
21647 6772 Acked Read 6 Done Sent Done false
21648 6773 Acked Write 3 Done Sent Done false
21649 6774 Acked Flush 0 Done Sent Done false
21650 6775 Acked Read 2 Done Sent Done false
21651 6776 Acked Read 8 Done Sent Done false
21652 6777 Acked Read 2 Done Sent Done false
21653 6778 Acked Read 4 Done Sent Done false
21654 6779 Acked Read 9 Done Sent Done false
21655 6780 Acked Read 4 Done Sent Done false
21656 6781 Acked Write 9 Done Sent Done false
21657 6782 Acked Write 1 Done Sent Done false
21658 6783 Acked Read 6 Done Sent Done false
21659 6784 Acked Write 5 Done Sent Done false
21660 6785 Acked Read 5 Done Sent Done false
21661 6786 Acked Write 1 Done Sent Done false
21662 6787 Acked Write 2 Done Sent Done false
21663 6788 Acked Read 5 Done Sent Done false
21664 6789 Acked Read 5 Done Sent Done false
21665 6790 Acked Read 9 Done Sent Done false
21666 6791 Acked Write 5 Done Sent Done false
21667 6792 Acked Read 1 Done Sent Done false
21668 6793 Acked Read 3 Done Sent Done false
21669 6794 Acked Write 3 Done Sent Done false
21670 6795 Acked Write 7 Done Sent Done false
21671 6796 Acked Flush 0 Done Sent Done false
21672 6797 Acked Read 3 Done Sent Done false
21673 6798 Acked Write 10 Done Sent Done false
21674 6799 Acked Flush 0 Done Sent Done false
21675 6800 Acked Write 1 Done Sent Done false
21676 6801 Acked Write 6 Done Sent Done false
21677 6802 Acked Flush 0 Done Sent Done false
21678 6803 Acked Write 7 Done Sent Done false
21679 6804 Acked Write 2 Done Sent Done false
21680 6805 Acked Write 3 Done Sent Done false
21681 6806 Acked Write 6 Done Sent Done false
21682 6807 Acked Read 10 Done Sent Done false
21683 6808 Acked Read 7 Done Sent Done false
21684 6809 Acked Write 5 Done Sent Done false
21685 6810 Acked Flush 0 Done Sent Done false
21686 6811 Acked Write 7 Done Sent Done false
21687 6812 Acked Read 10 Done Sent Done false
21688 6813 Acked Read 7 Done Sent Done false
21689 6814 Acked Read 7 Done Sent Done false
21690 6815 Acked Write 7 Done Sent Done false
21691 6816 Acked Read 1 Done Sent Done false
21692 6817 Acked Write 4 Done Sent Done false
21693 6818 Acked Read 1 Done Sent Done false
21694 6819 Acked Write 9 Done Sent Done false
21695 6820 Acked Read 10 Done Sent Done false
21696 6821 Acked Write 10 Done Sent Done false
21697 6822 Acked Write 1 Done Sent Done false
21698 6823 Acked Write 10 Done Sent Done false
21699 6824 Acked Write 4 Done Sent Done false
21700 6825 Acked Write 1 Done Sent Done false
21701 6826 Acked Read 1 Done Sent Done false
21702 6827 Acked Write 6 Done Sent Done false
21703 6828 Acked Write 6 Done Sent Done false
21704 6829 Acked Read 10 Done Sent Done false
21705 6830 Acked Read 1 Done Sent Done false
21706 6831 Acked Write 4 Done Sent Done false
21707 6832 Acked Write 6 Done Sent Done false
21708 6833 Acked Read 2 Done Sent Done false
21709 6834 Acked Read 7 Done Sent Done false
21710 6835 Acked Read 8 Done Sent Done false
21711 6836 Acked Write 9 Done Sent Done false
21712 6837 Acked Read 7 Done Sent Done false
21713 6838 Acked Write 9 Done Sent Done false
21714 6839 Acked Read 9 Done Sent Done false
21715 6840 Acked Read 10 Done Sent Done false
21716 6841 Acked Write 10 Done Sent Done false
21717 6842 Acked Write 9 Done Sent Done false
21718 6843 Acked Write 3 Done Sent Done false
21719 6844 Acked Write 5 Done Sent Done false
21720 6845 Acked Write 6 Done Sent Done false
21721 6846 Acked Read 9 Done Sent Done false
21722 6847 Acked Write 5 Done Sent Done false
21723 6848 Acked Read 7 Done Sent Done false
21724 6849 Acked Read 8 Done Sent Done false
21725 6850 Acked Read 2 Done Sent Done false
21726 6851 Acked Flush 0 Done Sent Done false
21727 6852 Acked Flush 0 Done Sent Done false
21728 6853 Acked Write 9 Done Sent Done false
21729 6854 Acked Write 10 Done Sent Done false
21730 6855 Acked Write 5 Done Sent Done false
21731 6856 Acked Write 6 Done Sent Done false
21732 6857 Acked Read 2 Done Sent Done false
21733 6858 Acked Write 3 Done Sent Done false
21734 6859 Acked Read 8 Done Sent Done false
21735 6860 Acked Read 10 Done Sent Done false
21736 6861 Acked Read 1 Done Sent Done false
21737 6862 Acked Flush 0 Done Sent Done false
21738 6863 Acked Read 10 Done Sent Done false
21739 6864 Acked Write 10 Done Sent Done false
21740 6865 Acked Write 4 Done Sent Done false
21741 6866 Acked Read 8 Done Sent Done false
21742 6867 Acked Write 1 Done Sent Done false
21743 6868 Acked Read 5 Done Sent Done false
21744 6869 Acked Write 8 Done Sent Done false
21745 6870 Acked Write 5 Done Sent Done false
21746 6871 Acked Read 5 Done Sent Done false
21747 6872 Acked Write 2 Done Sent Done false
21748 6873 Acked Write 9 Done Sent Done false
21749 6874 Acked Read 5 Done Sent Done false
21750 6875 Acked Write 9 Done Sent Done false
21751 6876 Acked Read 10 Done Sent Done false
21752 6877 Acked Write 4 Done Sent Done false
21753 6878 Acked Read 7 Done Sent Done false
21754 6879 Acked Write 4 Done Sent Done false
21755 6880 Acked Write 8 Done Sent Done false
21756 6881 Acked Read 1 Done Sent Done false
21757 6882 Acked Write 10 Done Sent Done false
21758 6883 Acked Write 4 Done Sent Done false
21759 6884 Acked Read 2 Done Sent Done false
21760 6885 Acked Write 1 Done Sent Done false
21761 6886 Acked Write 6 Done Sent Done false
21762 6887 Acked Write 3 Done Sent Done false
21763 6888 Acked Read 8 Done Sent Done false
21764 6889 Acked Read 1 Done Sent Done false
21765 6890 Acked Write 4 Done Sent Done false
21766 6891 Acked Read 5 Done Sent Done false
21767 6892 Acked Read 7 Done Sent Done false
21768 6893 Acked Read 5 Done Sent Done false
21769 6894 Acked Read 4 Done Sent Done false
21770 6895 Acked Write 8 Done Sent Done false
21771 6896 Acked Write 4 Done Sent Done false
21772 6897 Acked Read 10 Done Sent Done false
21773 6898 Acked Read 10 Done Sent Done false
21774 6899 Acked Write 10 Done Sent Done false
21775 6900 Acked Write 10 Done Sent Done false
21776 6901 Acked Write 7 Done Sent Done false
21777 6902 Acked Write 6 Done Sent Done false
21778 6903 Acked Read 3 Done Sent Done false
21779 6904 Acked Read 4 Done Sent Done false
21780 6905 Acked Write 8 Done Sent Done false
21781 6906 Acked Read 2 Done Sent Done false
21782 6907 Acked Write 6 Done Sent Done false
21783 6908 Acked Write 10 Done Sent Done false
21784 6909 Acked Flush 0 Done Sent Done false
21785 6910 Acked Read 9 Done Sent Done false
21786 6911 Acked Read 6 Done Sent Done false
21787 6912 Acked Write 6 Done Sent Done false
21788 6913 Acked Write 8 Done Sent Done false
21789 6914 Acked Read 5 Done Sent Done false
21790 6915 Acked Read 10 Done Sent Done false
21791 6916 Acked Read 5 Done Sent Done false
21792 6917 Acked Read 4 Done Sent Done false
21793 6918 Acked Read 9 Done Sent Done false
21794 6919 Acked Flush 0 Done Sent Done false
21795 6920 Acked Read 4 Done Sent Done false
21796 6921 Acked Write 5 Done Sent Done false
21797 6922 Acked Read 6 Done Sent Done false
21798 6923 Acked Flush 0 Done Sent Done false
21799 6924 Acked Write 8 Done Sent Done false
21800 6925 Acked Flush 0 Done Sent Done false
21801 6926 Acked Read 1 Done Sent Done false
21802 6927 Acked Read 7 Done Sent Done false
21803 6928 Acked Write 8 Done Sent Done false
21804 6929 Acked Write 10 Done Sent Done false
21805 6930 Acked Write 4 Done Sent Done false
21806 6931 Acked Read 6 Done Sent Done false
21807 6932 Acked Read 9 Done Sent Done false
21808 6933 Acked Write 9 Done Sent Done false
21809 6934 Acked Write 9 Done Sent Done false
21810 6935 Acked Read 4 Done Sent Done false
21811 6936 Acked Read 7 Done Sent Done false
21812 6937 Acked Read 3 Done Sent Done false
21813 6938 Acked Read 6 Done Sent Done false
21814 6939 Acked Write 10 Done Sent Done false
21815 6940 Acked Flush 0 Done Sent Done false
21816 6941 Acked Read 2 Done Sent Done false
21817 6942 Acked Read 5 Done Sent Done false
21818 6943 Acked Write 3 Done Sent Done false
21819 6944 Acked Write 6 Done Sent Done false
21820 6945 Acked Read 4 Done Sent Done false
21821 6946 Acked Write 9 Done Sent Done false
21822 6947 Acked Read 4 Done Sent Done false
21823 6948 Acked Read 10 Done Sent Done false
21824 6949 Acked Read 7 Done Sent Done false
21825 6950 Acked Write 4 Done Sent Done false
21826 6951 Acked Write 9 Done Sent Done false
21827 6952 Acked Write 1 Done Sent Done false
21828 6953 Acked Write 10 Done Sent Done false
21829 6954 Acked Write 4 Done Sent Done false
21830 6955 Acked Read 5 Done Sent Done false
21831 6956 Acked Write 6 Done Sent Done false
21832 6957 Acked Read 4 Done Sent Done false
21833 6958 Acked Read 7 Done Sent Done false
21834 6959 Acked Write 7 Done Sent Done false
21835 6960 Acked Write 2 Done Sent Done false
21836 6961 Acked Read 9 Done Sent Done false
21837 6962 Acked Read 8 Done Sent Done false
21838 6963 Acked Read 3 Done Sent Done false
21839 6964 Acked Read 10 Done Sent Done false
21840 6965 Acked Read 2 Done Sent Done false
21841 6966 Acked Read 4 Done Sent Done false
21842 6967 Acked Read 10 Done Sent Done false
21843 6968 Acked Read 7 Done Sent Done false
21844 6969 Acked Read 1 Done Sent Done false
21845 6970 Acked Write 7 Done Sent Done false
21846 6971 Acked Write 4 Done Sent Done false
21847 6972 Acked Read 9 Done Sent Done false
21848 6973 Acked Read 6 Done Sent Done false
21849 6974 Acked Write 4 Done Sent Done false
21850 6975 Acked Flush 0 Done Sent Done false
21851 6976 Acked Flush 0 Done Sent Done false
21852 6977 Acked Write 7 Done Sent Done false
21853 6978 Acked Write 4 Done Sent Done false
21854 6979 Acked Write 10 Done Sent Done false
21855 6980 Acked Read 8 Done Sent Done false
21856 6981 Acked Write 4 Done Sent Done false
21857 6982 Acked Read 7 Done Sent Done false
21858 6983 Acked Flush 0 Done Sent Done false
21859 6984 Acked Read 3 Done Sent Done false
21860 6985 Acked Flush 0 Done Sent Done false
21861 6986 Acked Write 8 Done Sent Done false
21862 6987 Acked Flush 0 Done Sent Done false
21863 6988 Acked Read 6 Done Sent Done false
21864 6989 Acked Read 5 Done Sent Done false
21865 6990 Acked Write 8 Done Sent Done false
21866 6991 Acked Read 10 Done Sent Done false
21867 6992 Acked Read 3 Done Sent Done false
21868 6993 Acked Write 6 Done Sent Done false
21869 6994 Acked Write 10 Done Sent Done false
21870 6995 Acked Read 5 Done Sent Done false
21871 6996 Acked Flush 0 Done Sent Done false
21872 6997 Acked Write 7 Done Sent Done false
21873 6998 Acked Write 4 Done Sent Done false
21874 6999 Acked Read 10 Done Sent Done false
21875 7000 Acked Write 9 Done Sent Done false
21876 7001 Acked Read 10 Done Sent Done false
21877 7002 Acked Flush 0 Done Sent Done false
21878 7003 Acked Read 6 Done Sent Done false
21879 7004 Acked Write 1 Done Sent Done false
21880 7005 Acked Read 7 Done Sent Done false
21881 7006 Acked Flush 0 Done Sent Done false
21882 7007 Acked Write 1 Done Sent Done false
21883 7008 Acked Write 1 Done Sent Done false
21884 7009 Acked Write 1 Done Sent Done false
21885 7010 Acked Read 10 Done Sent Done false
21886 7011 Acked Write 6 Done Sent Done false
21887 7012 Acked Write 4 Done Sent Done false
21888 7013 Acked Flush 0 Done Sent Done false
21889 7014 Acked Read 3 Done Sent Done false
21890 7015 Acked Read 4 Done Sent Done false
21891 7016 Acked Write 10 Done Sent Done false
21892 7017 Acked Read 1 Done Sent Done false
21893 7018 Acked Write 9 Done Sent Done false
21894 7019 Acked Flush 0 Done Sent Done false
21895 7020 Acked Write 5 Done Sent Done false
21896 7021 Acked Write 5 Done Sent Done false
21897 7022 Acked Read 5 Done Sent Done false
21898 7023 Acked Write 2 Done Sent Done false
21899 7024 Acked Read 4 Done Sent Done false
21900 7025 Acked Write 7 Done Sent Done false
21901 7026 Acked Write 9 Done Sent Done false
21902 7027 Acked Write 10 Done Sent Done false
21903 7028 Acked Read 5 Done Sent Done false
21904 7029 Acked Write 2 Done Sent Done false
21905 7030 Acked Flush 0 Done Sent Done false
21906 7031 Acked Write 1 Done Sent Done false
21907 7032 Acked Write 9 Done Sent Done false
21908 7033 Acked Read 1 Done Sent Done false
21909 7034 Acked Read 5 Done Sent Done false
21910 7035 Acked Write 5 Done Sent Done false
21911 7036 Acked Write 9 Done Sent Done false
21912 7037 Acked Write 4 Done Sent Done false
21913 7038 Acked Read 8 Done Sent Done false
21914 7039 Acked Flush 0 Done Sent Done false
21915 STATES DS:0 DS:1 DS:2 TOTAL
21916 Sent 0 302 0 302
21917 Done 302 0 302 604
21918 Skipped 0 0 0 0
21919 Error 0 0 0 0
21920 Last Flush: 7039 6737 7039
21921 Downstairs last five completed: 6737 6736 6735 6734 6733
21922 Upstairs last five completed: 7039 7038 7037 7036 7035
21923 CLIENT: Up:0 ds:302 act:2
21924 ----------------------------------------------------------------
21925 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
21926 States: Active Offline Active
21927 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21928 6738 Acked Read 9 Done Sent Done false
21929 6739 Acked Write 8 Done Sent Done false
21930 6740 Acked Flush 0 Done Sent Done false
21931 6741 Acked Read 5 Done Sent Done false
21932 6742 Acked Write 5 Done Sent Done false
21933 6743 Acked Write 10 Done Sent Done false
21934 6744 Acked Flush 0 Done Sent Done false
21935 6745 Acked Write 9 Done Sent Done false
21936 6746 Acked Write 5 Done Sent Done false
21937 6747 Acked Write 3 Done Sent Done false
21938 6748 Acked Read 7 Done Sent Done false
21939 6749 Acked Write 6 Done Sent Done false
21940 6750 Acked Read 3 Done Sent Done false
21941 6751 Acked Write 5 Done Sent Done false
21942 6752 Acked Write 10 Done Sent Done false
21943 6753 Acked Write 4 Done Sent Done false
21944 6754 Acked Read 8 Done Sent Done false
21945 6755 Acked Read 4 Done Sent Done false
21946 6756 Acked Read 8 Done Sent Done false
21947 6757 Acked Read 4 Done Sent Done false
21948 6758 Acked Write 1 Done Sent Done false
21949 6759 Acked Write 4 Done Sent Done false
21950 6760 Acked Read 3 Done Sent Done false
21951 6761 Acked Write 10 Done Sent Done false
21952 6762 Acked Write 10 Done Sent Done false
21953 6763 Acked Write 8 Done Sent Done false
21954 6764 Acked Read 6 Done Sent Done false
21955 6765 Acked Write 1 Done Sent Done false
21956 6766 Acked Flush 0 Done Sent Done false
21957 6767 Acked Write 9 Done Sent Done false
21958 6768 Acked Write 4 Done Sent Done false
21959 6769 Acked Read 7 Done Sent Done false
21960 6770 Acked Read 8 Done Sent Done false
21961 6771 Acked Read 10 Done Sent Done false
21962 6772 Acked Read 6 Done Sent Done false
21963 6773 Acked Write 3 Done Sent Done false
21964 6774 Acked Flush 0 Done Sent Done false
21965 6775 Acked Read 2 Done Sent Done false
21966 6776 Acked Read 8 Done Sent Done false
21967 6777 Acked Read 2 Done Sent Done false
21968 6778 Acked Read 4 Done Sent Done false
21969 6779 Acked Read 9 Done Sent Done false
21970 6780 Acked Read 4 Done Sent Done false
21971 6781 Acked Write 9 Done Sent Done false
21972 6782 Acked Write 1 Done Sent Done false
21973 6783 Acked Read 6 Done Sent Done false
21974 6784 Acked Write 5 Done Sent Done false
21975 6785 Acked Read 5 Done Sent Done false
21976 6786 Acked Write 1 Done Sent Done false
21977 6787 Acked Write 2 Done Sent Done false
21978 6788 Acked Read 5 Done Sent Done false
21979 6789 Acked Read 5 Done Sent Done false
21980 6790 Acked Read 9 Done Sent Done false
21981 6791 Acked Write 5 Done Sent Done false
21982 6792 Acked Read 1 Done Sent Done false
21983 6793 Acked Read 3 Done Sent Done false
21984 6794 Acked Write 3 Done Sent Done false
21985 6795 Acked Write 7 Done Sent Done false
21986 6796 Acked Flush 0 Done Sent Done false
21987 6797 Acked Read 3 Done Sent Done false
21988 6798 Acked Write 10 Done Sent Done false
21989 6799 Acked Flush 0 Done Sent Done false
21990 6800 Acked Write 1 Done Sent Done false
21991 6801 Acked Write 6 Done Sent Done false
21992 6802 Acked Flush 0 Done Sent Done false
21993 6803 Acked Write 7 Done Sent Done false
21994 6804 Acked Write 2 Done Sent Done false
21995 6805 Acked Write 3 Done Sent Done false
21996 6806 Acked Write 6 Done Sent Done false
21997 6807 Acked Read 10 Done Sent Done false
21998 6808 Acked Read 7 Done Sent Done false
21999 6809 Acked Write 5 Done Sent Done false
22000 6810 Acked Flush 0 Done Sent Done false
22001 6811 Acked Write 7 Done Sent Done false
22002 6812 Acked Read 10 Done Sent Done false
22003 6813 Acked Read 7 Done Sent Done false
22004 6814 Acked Read 7 Done Sent Done false
22005 6815 Acked Write 7 Done Sent Done false
22006 6816 Acked Read 1 Done Sent Done false
22007 6817 Acked Write 4 Done Sent Done false
22008 6818 Acked Read 1 Done Sent Done false
22009 6819 Acked Write 9 Done Sent Done false
22010 6820 Acked Read 10 Done Sent Done false
22011 6821 Acked Write 10 Done Sent Done false
22012 6822 Acked Write 1 Done Sent Done false
22013 6823 Acked Write 10 Done Sent Done false
22014 6824 Acked Write 4 Done Sent Done false
22015 6825 Acked Write 1 Done Sent Done false
22016 6826 Acked Read 1 Done Sent Done false
22017 6827 Acked Write 6 Done Sent Done false
22018 6828 Acked Write 6 Done Sent Done false
22019 6829 Acked Read 10 Done Sent Done false
22020 6830 Acked Read 1 Done Sent Done false
22021 6831 Acked Write 4 Done Sent Done false
22022 6832 Acked Write 6 Done Sent Done false
22023 6833 Acked Read 2 Done Sent Done false
22024 6834 Acked Read 7 Done Sent Done false
22025 6835 Acked Read 8 Done Sent Done false
22026 6836 Acked Write 9 Done Sent Done false
22027 6837 Acked Read 7 Done Sent Done false
22028 6838 Acked Write 9 Done Sent Done false
22029 6839 Acked Read 9 Done Sent Done false
22030 6840 Acked Read 10 Done Sent Done false
22031 6841 Acked Write 10 Done Sent Done false
22032 6842 Acked Write 9 Done Sent Done false
22033 6843 Acked Write 3 Done Sent Done false
22034 6844 Acked Write 5 Done Sent Done false
22035 6845 Acked Write 6 Done Sent Done false
22036 6846 Acked Read 9 Done Sent Done false
22037 6847 Acked Write 5 Done Sent Done false
22038 6848 Acked Read 7 Done Sent Done false
22039 6849 Acked Read 8 Done Sent Done false
22040 6850 Acked Read 2 Done Sent Done false
22041 6851 Acked Flush 0 Done Sent Done false
22042 6852 Acked Flush 0 Done Sent Done false
22043 6853 Acked Write 9 Done Sent Done false
22044 6854 Acked Write 10 Done Sent Done false
22045 6855 Acked Write 5 Done Sent Done false
22046 6856 Acked Write 6 Done Sent Done false
22047 6857 Acked Read 2 Done Sent Done false
22048 6858 Acked Write 3 Done Sent Done false
22049 6859 Acked Read 8 Done Sent Done false
22050 6860 Acked Read 10 Done Sent Done false
22051 6861 Acked Read 1 Done Sent Done false
22052 6862 Acked Flush 0 Done Sent Done false
22053 6863 Acked Read 10 Done Sent Done false
22054 6864 Acked Write 10 Done Sent Done false
22055 6865 Acked Write 4 Done Sent Done false
22056 6866 Acked Read 8 Done Sent Done false
22057 6867 Acked Write 1 Done Sent Done false
22058 6868 Acked Read 5 Done Sent Done false
22059 6869 Acked Write 8 Done Sent Done false
22060 6870 Acked Write 5 Done Sent Done false
22061 6871 Acked Read 5 Done Sent Done false
22062 6872 Acked Write 2 Done Sent Done false
22063 6873 Acked Write 9 Done Sent Done false
22064 6874 Acked Read 5 Done Sent Done false
22065 6875 Acked Write 9 Done Sent Done false
22066 6876 Acked Read 10 Done Sent Done false
22067 6877 Acked Write 4 Done Sent Done false
22068 6878 Acked Read 7 Done Sent Done false
22069 6879 Acked Write 4 Done Sent Done false
22070 6880 Acked Write 8 Done Sent Done false
22071 6881 Acked Read 1 Done Sent Done false
22072 6882 Acked Write 10 Done Sent Done false
22073 6883 Acked Write 4 Done Sent Done false
22074 6884 Acked Read 2 Done Sent Done false
22075 6885 Acked Write 1 Done Sent Done false
22076 6886 Acked Write 6 Done Sent Done false
22077 6887 Acked Write 3 Done Sent Done false
22078 6888 Acked Read 8 Done Sent Done false
22079 6889 Acked Read 1 Done Sent Done false
22080 6890 Acked Write 4 Done Sent Done false
22081 6891 Acked Read 5 Done Sent Done false
22082 6892 Acked Read 7 Done Sent Done false
22083 6893 Acked Read 5 Done Sent Done false
22084 6894 Acked Read 4 Done Sent Done false
22085 6895 Acked Write 8 Done Sent Done false
22086 6896 Acked Write 4 Done Sent Done false
22087 6897 Acked Read 10 Done Sent Done false
22088 6898 Acked Read 10 Done Sent Done false
22089 6899 Acked Write 10 Done Sent Done false
22090 6900 Acked Write 10 Done Sent Done false
22091 6901 Acked Write 7 Done Sent Done false
22092 6902 Acked Write 6 Done Sent Done false
22093 6903 Acked Read 3 Done Sent Done false
22094 6904 Acked Read 4 Done Sent Done false
22095 6905 Acked Write 8 Done Sent Done false
22096 6906 Acked Read 2 Done Sent Done false
22097 6907 Acked Write 6 Done Sent Done false
22098 6908 Acked Write 10 Done Sent Done false
22099 6909 Acked Flush 0 Done Sent Done false
22100 6910 Acked Read 9 Done Sent Done false
22101 6911 Acked Read 6 Done Sent Done false
22102 6912 Acked Write 6 Done Sent Done false
22103 6913 Acked Write 8 Done Sent Done false
22104 6914 Acked Read 5 Done Sent Done false
22105 6915 Acked Read 10 Done Sent Done false
22106 6916 Acked Read 5 Done Sent Done false
22107 6917 Acked Read 4 Done Sent Done false
22108 6918 Acked Read 9 Done Sent Done false
22109 6919 Acked Flush 0 Done Sent Done false
22110 6920 Acked Read 4 Done Sent Done false
22111 6921 Acked Write 5 Done Sent Done false
22112 6922 Acked Read 6 Done Sent Done false
22113 6923 Acked Flush 0 Done Sent Done false
22114 6924 Acked Write 8 Done Sent Done false
22115 6925 Acked Flush 0 Done Sent Done false
22116 6926 Acked Read 1 Done Sent Done false
22117 6927 Acked Read 7 Done Sent Done false
22118 6928 Acked Write 8 Done Sent Done false
22119 6929 Acked Write 10 Done Sent Done false
22120 6930 Acked Write 4 Done Sent Done false
22121 6931 Acked Read 6 Done Sent Done false
22122 6932 Acked Read 9 Done Sent Done false
22123 6933 Acked Write 9 Done Sent Done false
22124 6934 Acked Write 9 Done Sent Done false
22125 6935 Acked Read 4 Done Sent Done false
22126 6936 Acked Read 7 Done Sent Done false
22127 6937 Acked Read 3 Done Sent Done false
22128 6938 Acked Read 6 Done Sent Done false
22129 6939 Acked Write 10 Done Sent Done false
22130 6940 Acked Flush 0 Done Sent Done false
22131 6941 Acked Read 2 Done Sent Done false
22132 6942 Acked Read 5 Done Sent Done false
22133 6943 Acked Write 3 Done Sent Done false
22134 6944 Acked Write 6 Done Sent Done false
22135 6945 Acked Read 4 Done Sent Done false
22136 6946 Acked Write 9 Done Sent Done false
22137 6947 Acked Read 4 Done Sent Done false
22138 6948 Acked Read 10 Done Sent Done false
22139 6949 Acked Read 7 Done Sent Done false
22140 6950 Acked Write 4 Done Sent Done false
22141 6951 Acked Write 9 Done Sent Done false
22142 6952 Acked Write 1 Done Sent Done false
22143 6953 Acked Write 10 Done Sent Done false
22144 6954 Acked Write 4 Done Sent Done false
22145 6955 Acked Read 5 Done Sent Done false
22146 6956 Acked Write 6 Done Sent Done false
22147 6957 Acked Read 4 Done Sent Done false
22148 6958 Acked Read 7 Done Sent Done false
22149 6959 Acked Write 7 Done Sent Done false
22150 6960 Acked Write 2 Done Sent Done false
22151 6961 Acked Read 9 Done Sent Done false
22152 6962 Acked Read 8 Done Sent Done false
22153 6963 Acked Read 3 Done Sent Done false
22154 6964 Acked Read 10 Done Sent Done false
22155 6965 Acked Read 2 Done Sent Done false
22156 6966 Acked Read 4 Done Sent Done false
22157 6967 Acked Read 10 Done Sent Done false
22158 6968 Acked Read 7 Done Sent Done false
22159 6969 Acked Read 1 Done Sent Done false
22160 6970 Acked Write 7 Done Sent Done false
22161 6971 Acked Write 4 Done Sent Done false
22162 6972 Acked Read 9 Done Sent Done false
22163 6973 Acked Read 6 Done Sent Done false
22164 6974 Acked Write 4 Done Sent Done false
22165 6975 Acked Flush 0 Done Sent Done false
22166 6976 Acked Flush 0 Done Sent Done false
22167 6977 Acked Write 7 Done Sent Done false
22168 6978 Acked Write 4 Done Sent Done false
22169 6979 Acked Write 10 Done Sent Done false
22170 6980 Acked Read 8 Done Sent Done false
22171 6981 Acked Write 4 Done Sent Done false
22172 6982 Acked Read 7 Done Sent Done false
22173 6983 Acked Flush 0 Done Sent Done false
22174 6984 Acked Read 3 Done Sent Done false
22175 6985 Acked Flush 0 Done Sent Done false
22176 6986 Acked Write 8 Done Sent Done false
22177 6987 Acked Flush 0 Done Sent Done false
22178 6988 Acked Read 6 Done Sent Done false
22179 6989 Acked Read 5 Done Sent Done false
22180 6990 Acked Write 8 Done Sent Done false
22181 6991 Acked Read 10 Done Sent Done false
22182 6992 Acked Read 3 Done Sent Done false
22183 6993 Acked Write 6 Done Sent Done false
22184 6994 Acked Write 10 Done Sent Done false
22185 6995 Acked Read 5 Done Sent Done false
22186 6996 Acked Flush 0 Done Sent Done false
22187 6997 Acked Write 7 Done Sent Done false
22188 6998 Acked Write 4 Done Sent Done false
22189 6999 Acked Read 10 Done Sent Done false
22190 7000 Acked Write 9 Done Sent Done false
22191 7001 Acked Read 10 Done Sent Done false
22192 7002 Acked Flush 0 Done Sent Done false
22193 7003 Acked Read 6 Done Sent Done false
22194 7004 Acked Write 1 Done Sent Done false
22195 7005 Acked Read 7 Done Sent Done false
22196 7006 Acked Flush 0 Done Sent Done false
22197 7007 Acked Write 1 Done Sent Done false
22198 7008 Acked Write 1 Done Sent Done false
22199 7009 Acked Write 1 Done Sent Done false
22200 7010 Acked Read 10 Done Sent Done false
22201 7011 Acked Write 6 Done Sent Done false
22202 7012 Acked Write 4 Done Sent Done false
22203 7013 Acked Flush 0 Done Sent Done false
22204 7014 Acked Read 3 Done Sent Done false
22205 7015 Acked Read 4 Done Sent Done false
22206 7016 Acked Write 10 Done Sent Done false
22207 7017 Acked Read 1 Done Sent Done false
22208 7018 Acked Write 9 Done Sent Done false
22209 7019 Acked Flush 0 Done Sent Done false
22210 7020 Acked Write 5 Done Sent Done false
22211 7021 Acked Write 5 Done Sent Done false
22212 7022 Acked Read 5 Done Sent Done false
22213 7023 Acked Write 2 Done Sent Done false
22214 7024 Acked Read 4 Done Sent Done false
22215 7025 Acked Write 7 Done Sent Done false
22216 7026 Acked Write 9 Done Sent Done false
22217 7027 Acked Write 10 Done Sent Done false
22218 7028 Acked Read 5 Done Sent Done false
22219 7029 Acked Write 2 Done Sent Done false
22220 7030 Acked Flush 0 Done Sent Done false
22221 7031 Acked Write 1 Done Sent Done false
22222 7032 Acked Write 9 Done Sent Done false
22223 7033 Acked Read 1 Done Sent Done false
22224 7034 Acked Read 5 Done Sent Done false
22225 7035 Acked Write 5 Done Sent Done false
22226 7036 Acked Write 9 Done Sent Done false
22227 7037 Acked Write 4 Done Sent Done false
22228 7038 Acked Read 8 Done Sent Done false
22229 7039 Acked Flush 0 Done Sent Done false
22230 STATES DS:0 DS:1 DS:2 TOTAL
22231 Sent 0 302 0 302
22232 Done 302 0 302 604
22233 Skipped 0 0 0 0
22234 Error 0 0 0 0
22235 Last Flush: 7039 6737 7039
22236 Downstairs last five completed: 6737 6736 6735 6734 6733
22237 Upstairs last five completed: 7039 7038 7037 7036 7035
22238 CLIENT: Up:0 ds:302 act:2
222392024-12-09T15:12:31.813ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222402024-12-09T15:12:31.813ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64388) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222412024-12-09T15:12:31.815ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222422024-12-09T15:12:31.815ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222432024-12-09T15:12:31.815ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222442024-12-09T15:12:31.815ZINFOcrucible: send last flush ID to this DS: 6737 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222452024-12-09T15:12:31.815ZINFOcrucible: Replied this last flush ID: 6737 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222462024-12-09T15:12:31.815ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222472024-12-09T15:12:31.815ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222482024-12-09T15:12:31.815ZINFOcrucible: [1] client re-new 302 jobs since flush 6737 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222492024-12-09T15:12:31.816ZINFOcrucible: [1] Replayed 302 jobs since flush: 6737 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
22250 ----------------------------------------------------------------
22251 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22252 Downstairs last five completed: 7039 7038 7037 7036 7035
22253 Upstairs last five completed: 7039 7038 7037 7036 7035
22254 CLIENT: Up:0 ds:0 act:3
22255 Replay: All jobs finished
22256 Main task has work to do, go find it
22257 got dsc Stop(2)
22258 stop 2
22259 Dec 09 15:12:32.063 INFO [8830] Got stop action so:false kr:false
22260 request completed, latency_us: 341, response_code: 204, uri: /stop/cid/2, method: POST, req_id: c784963f-9267-4ed9-b118-83db94276784, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
222612024-12-09T15:12:32.064ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222622024-12-09T15:12:32.064ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222632024-12-09T15:12:32.064ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222642024-12-09T15:12:32.064ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
222652024-12-09T15:12:32.064ZINFOcrucible: [2] Transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
22266 Dec 09 15:12:32.064 INFO request completed, latency_us: 157, response_code: 200, uri: /state/cid/2, method: GET, req_id: 57c2741c-c986-494c-b824-889f96fad375, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
222672024-12-09T15:12:32.064ZINFOcrucible: client task is exiting = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
22268 [8830] Exited with: ExitStatus(unix_wait_status(9))
22269 [8830][2] reports Exit
22270 Dec 09 15:12:36.066 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/2, method: GET, req_id: c8dd1a07-8bb1-42ad-a72d-7b8123aae781, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
22271 001/300 Read block 1858 len 28672
22272 002/300 Write block 4774 len 16384 data: 3 3 4 4
22273 003/300 Read block 2573 len 12288
22274 004/300 Flush
22275 005/300 Read block 707 len 32768
22276 006/300 Write block 4704 len 20480 data: 5 5 6 7 8
22277 007/300 Read block 814 len 28672
22278 008/300 Read block 338 len 4096
22279 009/300 Write block 1793 len 40960 data: 7 7 9 8 8 7 5 4 3 4
22280 010/300 Write block 3141 len 12288 data: 6 6 6
22281 011/300 Read block 4515 len 8192
22282 012/300 Write block 2104 len 40960 data: 6 6 6 7 6 5 4 4 6 6
22283 013/300 Write block 3167 len 16384 data: 4 4 4 4
22284 014/300 Flush
22285 015/300 Read block 1072 len 8192
22286 016/300 Read block 1934 len 40960
22287 017/300 Read block 1810 len 16384
22288 018/300 Read block 2241 len 4096
22289 019/300 Write block 152 len 16384 data: 6 6 6 6
22290 020/300 Write block 2850 len 12288 data: 3 3 4
22291 021/300 Read block 4759 len 16384
22292 022/300 Write block 2615 len 8192 data: 4 6
22293 023/300 Write block 2431 len 36864 data: 4 5 6 6 6 6 4 4 5
22294 024/300 Read block 2384 len 32768
22295 025/300 Read block 4733 len 4096
22296 026/300 Read block 3721 len 20480
22297 027/300 Flush
22298 028/300 Read block 2636 len 40960
22299 029/300 Flush
22300 030/300 Write block 4582 len 36864 data: 7 7 5 5 5 5 4 3 3
22301 031/300 Write block 660 len 12288 data: 2 6 6
22302 032/300 Read block 975 len 36864
22303 033/300 Read block 2244 len 16384
22304 034/300 Read block 3178 len 36864
22305 035/300 Read block 1013 len 40960
22306 036/300 Read block 4922 len 4096
22307 037/300 Write block 3229 len 36864 data: 4 6 6 6 6 6 6 6 7
22308 038/300 Read block 798 len 40960
22309 039/300 Write block 3323 len 16384 data: 5 5 5 5
22310 040/300 Write block 1892 len 36864 data: 6 5 4 4 3 3 3 4 3
22311 041/300 Read block 2967 len 8192
22312 042/300 Write block 3069 len 16384 data: 3 3 3 4
22313 043/300 Read block 1961 len 20480
22314 044/300 Flush
22315 045/300 Read block 920 len 32768
22316 046/300 Write block 3908 len 12288 data: 4 3 3
22317 047/300 Read block 2133 len 8192
22318 048/300 Write block 2861 len 36864 data: 2 2 4 4 5 6 8 8 8
22319 049/300 Write block 35 len 4096 data: 6
22320 050/300 Write block 4409 len 36864 data: 7 5 6 5 5 3 3 3 3
22321 051/300 Flush
22322 052/300 Read block 3783 len 8192
22323 053/300 Write block 2556 len 20480 data: 4 4 5 6 5
22324 054/300 Write block 506 len 36864 data: 8 9 8 8 7 4 4 5 5
22325 055/300 Read block 2957 len 12288
22326 056/300 Write block 2969 len 28672 data: 2 5 6 6 6 4 4
22327 057/300 Read block 979 len 8192
22328 058/300 Write block 2030 len 28672 data: 4 4 4 3 2 2 4
22329 059/300 Read block 4638 len 20480
22330 060/300 Write block 2452 len 8192 data: 3 4
22331 061/300 Read block 1595 len 20480
22332 062/300 Write block 3474 len 40960 data: 6 7 7 6 6 5 4 3 3 5
22333 063/300 Read block 2199 len 16384
22334 064/300 Read block 2003 len 24576
22335 065/300 Read block 4782 len 36864
22336 066/300 Write block 4752 len 20480 data: 5 5 5 4 4
22337 067/300 Read block 1508 len 24576
22338 068/300 Read block 403 len 24576
22339 069/300 Flush
22340 070/300 Read block 208 len 16384
22341 071/300 Read block 4488 len 16384
22342 072/300 Write block 1026 len 36864 data: 4 4 4 4 5 5 5 4 4
22343 073/300 Read block 1822 len 24576
22344 074/300 Write block 201 len 12288 data: 3 3 2
22345 075/300 Read block 512 len 40960
22346 076/300 Read block 864 len 32768
22347 077/300 Read block 700 len 32768
22348 078/300 Write block 4566 len 28672 data: 4 4 4 4 5 5 6
22349 079/300 Write block 3558 len 16384 data: 8 7 4 6
22350 080/300 Read block 862 len 36864
22351 081/300 Read block 3708 len 36864
22352 082/300 Read block 3596 len 8192
22353 083/300 Write block 3613 len 16384 data: 5 7 6 8
22354 084/300 Read block 1482 len 4096
22355 085/300 Read block 1554 len 24576
22356 086/300 Read block 813 len 28672
22357 087/300 Read block 603 len 32768
22358 088/300 Flush
22359 089/300 Write block 166 len 20480 data: 4 4 4 5 5
22360 090/300 Read block 4411 len 24576
22361 091/300 Read block 687 len 4096
22362 092/300 Write block 1206 len 32768 data: 3 6 4 4 4 5 5 7
22363 093/300 Read block 3344 len 40960
22364 094/300 Write block 1511 len 36864 data: 5 5 6 6 5 6 6 7 5
22365 095/300 Read block 3858 len 20480
22366 096/300 Write block 2775 len 16384 data: 10 10 10 10
22367 097/300 Flush
22368 098/300 Read block 3545 len 20480
22369 099/300 Read block 629 len 28672
22370 100/300 Write block 4506 len 12288 data: 9 9 9
22371 101/300 Write block 4260 len 16384 data: 5 6 5 5
22372 102/300 Read block 2771 len 24576
22373 103/300 Read block 3699 len 36864
22374 104/300 Flush
22375 105/300 Read block 2402 len 16384
22376 106/300 Read block 3783 len 32768
22377 107/300 Flush
22378 108/300 Write block 685 len 20480 data: 6 9 9 11 10
22379 109/300 Read block 2128 len 12288
22380 110/300 Flush
22381 111/300 Read block 524 len 32768
22382 112/300 Read block 125 len 12288
22383 113/300 Write block 724 len 12288 data: 5 4 5
22384 114/300 Write block 3641 len 8192 data: 3 2
22385 115/300 Write block 3433 len 36864 data: 5 6 5 3 3 3 4 4 4
22386 116/300 Write block 1730 len 16384 data: 5 6 6 6
22387 117/300 Read block 1568 len 20480
22388 118/300 Write block 3376 len 16384 data: 5 5 5 5
22389 119/300 Write block 872 len 20480 data: 5 5 4 3 5
22390 120/300 Write block 4231 len 28672 data: 4 4 4 4 4 4 5
22391 121/300 Read block 101 len 24576
22392 122/300 Flush
22393 123/300 Write block 3175 len 40960 data: 5 5 6 5 5 6 6 6 6 6
22394 124/300 Read block 3463 len 20480
22395 125/300 Read block 2637 len 28672
22396 126/300 Read block 494 len 8192
22397 127/300 Read block 494 len 20480
22398 128/300 Read block 4915 len 8192
22399 129/300 Read block 1894 len 16384
22400 130/300 Read block 4551 len 40960
22401 131/300 Write block 4332 len 28672 data: 2 3 3 3 3 4 4
22402 132/300 Read block 2736 len 24576
22403 133/300 Write block 307 len 36864 data: 3 3 3 2 2 2 3 4 5
22404 134/300 Read block 4636 len 4096
22405 135/300 Read block 1337 len 20480
22406 136/300 Read block 4513 len 36864
22407 137/300 Write block 1715 len 4096 data: 6
22408 138/300 Write block 1861 len 28672 data: 5 4 3 4 6 7 6
22409 139/300 Write block 2020 len 8192 data: 6 6
22410 140/300 Write block 2724 len 4096 data: 3
22411 141/300 Read block 2219 len 28672
22412 142/300 Read block 944 len 8192
22413 143/300 Read block 3016 len 36864
22414 144/300 Read block 1847 len 20480
22415 145/300 Write block 3262 len 36864 data: 5 6 6 5 4 4 4 4 4
22416 146/300 Write block 1697 len 28672 data: 4 3 4 4 3 5 4
22417 147/300 Write block 2203 len 4096 data: 6
22418 148/300 Write block 1202 len 24576 data: 6 5 4 4 4 7
22419 149/300 Read block 1818 len 4096
22420 150/300 Read block 3130 len 12288
22421 151/300 Read block 310 len 36864
22422 152/300 Read block 3687 len 36864
22423 153/300 Write block 2445 len 8192 data: 6 5
22424 154/300 Read block 2357 len 8192
22425 155/300 Write block 2068 len 40960 data: 3 3 3 3 3 4 4 5 5 5
22426 156/300 Read block 2594 len 12288
22427 157/300 Read block 4464 len 36864
22428 158/300 Flush
22429 159/300 Read block 2845 len 8192
22430 160/300 Read block 1013 len 8192
22431 161/300 Write block 3776 len 32768 data: 5 6 6 7 6 7 6 6
22432 162/300 Read block 4237 len 32768
22433 163/300 Write block 2738 len 8192 data: 6 4
22434 164/300 Write block 3755 len 32768 data: 7 7 7 8 7 6 5 6
22435 165/300 Read block 1986 len 20480
22436 166/300 Write block 925 len 28672 data: 7 8 5 5 4 3 4
22437 167/300 Flush
22438 168/300 Write block 3401 len 40960 data: 6 6 3 2 2 2 3 3 4 5
22439 169/300 Read block 4685 len 40960
22440 170/300 Read block 1652 len 40960
22441 171/300 Write block 4526 len 40960 data: 5 5 5 5 4 4 4 4 4 5
22442 172/300 Read block 2218 len 24576
22443 173/300 Write block 3097 len 20480 data: 3 4 2 4 6
22444 174/300 Write block 2258 len 40960 data: 6 5 5 5 4 4 4 3 2 2
22445 175/300 Write block 921 len 12288 data: 5 5 5
22446 176/300 Read block 234 len 12288
22447 177/300 Flush
22448 178/300 Read block 2379 len 12288
22449 179/300 Read block 4113 len 40960
22450 180/300 Read block 4764 len 12288
22451 181/300 Write block 4255 len 8192 data: 4 5
22452 182/300 Flush
22453 183/300 Write block 2780 len 40960 data: 9 5 5 6 5 6 4 3 5 5
22454 184/300 Read block 4453 len 40960
22455 185/300 Read block 4450 len 16384
22456 186/300 Write block 4510 len 40960 data: 7 7 7 6 6 5 4 4 4 5
22457 187/300 Read block 4554 len 12288
22458 188/300 Write block 4971 len 32768 data: 3 2 2 2 2 2 2 3
22459 189/300 Read block 4941 len 12288
22460 190/300 Write block 4470 len 28672 data: 6 5 5 5 5 4 3
22461 191/300 Write block 2054 len 8192 data: 5 5
22462 192/300 Read block 2874 len 36864
22463 193/300 Write block 3662 len 32768 data: 4 4 4 4 4 4 4 5
22464 194/300 Read block 1721 len 20480
22465 195/300 Read block 2632 len 40960
22466 196/300 Read block 1856 len 24576
22467 197/300 Write block 3541 len 40960 data: 3 3 4 4 5 4 4 5 6 6
22468 198/300 Flush
22469 199/300 Write block 3602 len 40960 data: 3 3 3 3 3 5 3 4 4 4
22470 200/300 Write block 1952 len 36864 data: 4 5 5 6 5 5 5 5 5
22471 201/300 Write block 3524 len 20480 data: 7 9 8 5 5
22472 202/300 Read block 4973 len 28672
22473 203/300 Read block 1195 len 28672
22474 204/300 Read block 2226 len 16384
22475 205/300 Read block 3632 len 8192
22476 206/300 Read block 3032 len 12288
22477 207/300 Write block 3753 len 16384 data: 6 6 8 8
22478 208/300 Read block 3557 len 28672
22479 209/300 Read block 1102 len 36864
22480 210/300 Read block 4446 len 40960
22481 211/300 Write block 1996 len 32768 data: 6 7 6 5 7 7 7 7
22482 212/300 Write block 2765 len 28672 data: 4 4 3 6 6 6 8
22483 213/300 Write block 298 len 12288 data: 4 4 5
22484 214/300 Write block 3299 len 40960 data: 4 4 5 5 4 4 4 4 4 4
22485 215/300 Write block 3170 len 8192 data: 5 4
22486 216/300 Write block 3726 len 20480 data: 5 6 6 5 4
22487 217/300 Write block 2915 len 32768 data: 4 4 4 4 4 4 5 6
22488 218/300 Read block 1554 len 32768
22489 219/300 Read block 1110 len 16384
22490 220/300 Read block 1210 len 40960
22491 221/300 Read block 4725 len 12288
22492 222/300 Read block 2602 len 20480
22493 223/300 Flush
22494 224/300 Read block 4696 len 24576
22495 225/300 Read block 2572 len 8192
22496 226/300 Write block 3799 len 12288 data: 8 9 9
22497 227/300 Read block 3928 len 40960
22498 228/300 Write block 2748 len 4096 data: 4
22499 229/300 Write block 4723 len 36864 data: 4 4 5 3 3 3 3 3 3
22500 230/300 Write block 623 len 12288 data: 4 4 4
22501 231/300 Write block 4460 len 24576 data: 2 3 4 5 5 6
22502 232/300 Read block 1353 len 36864
22503 233/300 Flush
22504 234/300 Write block 2168 len 40960 data: 4 4 3 3 5 6 6 7 7 6
22505 235/300 Read block 3692 len 24576
22506 236/300 Flush
22507 237/300 Read block 2633 len 20480
22508 238/300 Read block 2652 len 24576
22509 239/300 Read block 3395 len 36864
22510 240/300 Write block 4864 len 12288 data: 4 4 5
22511 241/300 Read block 2833 len 32768
22512 242/300 Write block 3213 len 8192 data: 5 5
22513 243/300 Read block 1266 len 20480
22514 244/300 Read block 3871 len 32768
22515 245/300 Read block 3187 len 12288
22516 246/300 Read block 244 len 28672
22517 247/300 Flush
22518 248/300 Write block 1495 len 36864 data: 2 2 5 5 5 5 7 6 6
22519 249/300 Read block 454 len 8192
22520 250/300 Read block 935 len 24576
22521 251/300 Read block 490 len 12288
22522 252/300 Read block 4635 len 32768
22523 253/300 Write block 2349 len 40960 data: 5 4 4 4 5 5 5 4 3 3
22524 254/300 Read block 3224 len 16384
22525 255/300 Read block 1589 len 28672
22526 256/300 Read block 280 len 20480
22527 257/300 Read block 152 len 4096
22528 258/300 Read block 1593 len 40960
22529 259/300 Flush
22530 260/300 Read block 4722 len 4096
22531 261/300 Write block 311 len 24576 data: 3 3 4 5 6 6
22532 262/300 Read block 827 len 28672
22533 263/300 Write block 402 len 32768 data: 5 6 7 7 6 4 4 4
22534 264/300 Read block 489 len 24576
22535 265/300 Write block 1203 len 12288 data: 6 5 5
22536 266/300 Read block 3373 len 32768
22537 267/300 Write block 3754 len 16384 data: 7 9 9 8
22538 268/300 Read block 2066 len 40960
22539 269/300 Read block 4707 len 32768
22540 270/300 Read block 2940 len 28672
22541 271/300 Read block 2694 len 4096
22542 272/300 Flush
22543 273/300 Read block 1314 len 20480
22544 274/300 Flush
22545 275/300 Read block 3353 len 8192
22546 276/300 Read block 1644 len 8192
22547 277/300 Read block 4420 len 32768
22548 278/300 Write block 2990 len 32768 data: 7 7 7 5 4 4 4 3
22549 279/300 Write block 3474 len 36864 data: 7 8 8 7 7 6 5 4 4
22550 280/300 Read block 4799 len 12288
22551 281/300 Read block 4623 len 36864
22552 282/300 Write block 819 len 16384 data: 6 6 6 5
22553 283/300 Write block 4274 len 24576 data: 3 3 2 2 2 2
22554 284/300 Write block 2556 len 40960 data: 5 5 6 7 6 4 4 3 4 5
22555 285/300 Read block 2925 len 4096
22556 286/300 Flush
22557 287/300 Write block 4762 len 20480 data: 5 5 4 4 5
22558 288/300 Flush
22559 289/300 Flush
22560 290/300 Write block 213 len 4096 data: 10
22561 291/300 Flush
22562 292/300 Write block 4811 len 40960 data: 6 5 7 9 6 5 5 6 5 3
22563 293/300 Write block 359 len 4096 data: 8
22564 294/300 Flush
22565 295/300 Write block 1999 len 4096 data: 6
22566 296/300 Write block 198 len 16384 data: 3 4 4 4
22567 297/300 Flush
22568 298/300 Read block 2116 len 8192
22569 299/300 Read block 4423 len 4096
22570 300/300 Write block 741 len 8192 data: 9 8
22571 301/300 Read block 255 len 28672
22572 Main task has work to do, go find it
22573 got dsc Start(2)
22574 start 2
22575 Dec 09 15:12:36.313 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/2, method: POST, req_id: 619700f4-81c4-4aeb-bb3d-8202ffdb7b88, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
22576 Replay: started 2, returned:Ok(())
22577 ----------------------------------------------------------------
22578 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22579 States: Active Active Offline
22580 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22581 7040 Acked Read 7 Done Done Sent false
22582 7041 Acked Write 4 Done Done Sent false
22583 7042 Acked Read 3 Done Done Sent false
22584 7043 Acked Flush 0 Done Done Sent false
22585 7044 Acked Read 8 Done Done Sent false
22586 7045 Acked Write 5 Done Done Sent false
22587 7046 Acked Read 7 Done Done Sent false
22588 7047 Acked Read 1 Done Done Sent false
22589 7048 Acked Write 10 Done Done Sent false
22590 7049 Acked Write 3 Done Done Sent false
22591 7050 Acked Read 2 Done Done Sent false
22592 7051 Acked Write 10 Done Done Sent false
22593 7052 Acked Write 4 Done Done Sent false
22594 7053 Acked Flush 0 Done Done Sent false
22595 7054 Acked Read 2 Done Done Sent false
22596 7055 Acked Read 10 Done Done Sent false
22597 7056 Acked Read 4 Done Done Sent false
22598 7057 Acked Read 1 Done Done Sent false
22599 7058 Acked Write 4 Done Done Sent false
22600 7059 Acked Write 3 Done Done Sent false
22601 7060 Acked Read 4 Done Done Sent false
22602 7061 Acked Write 2 Done Done Sent false
22603 7062 Acked Write 9 Done Done Sent false
22604 7063 Acked Read 8 Done Done Sent false
22605 7064 Acked Read 1 Done Done Sent false
22606 7065 Acked Read 5 Done Done Sent false
22607 7066 Acked Flush 0 Done Done Sent false
22608 7067 Acked Read 10 Done Done Sent false
22609 7068 Acked Flush 0 Done Done Sent false
22610 7069 Acked Write 9 Done Done Sent false
22611 7070 Acked Write 3 Done Done Sent false
22612 7071 Acked Read 9 Done Done Sent false
22613 7072 Acked Read 4 Done Done Sent false
22614 7073 Acked Read 9 Done Done Sent false
22615 7074 Acked Read 10 Done Done Sent false
22616 7075 Acked Read 1 Done Done Sent false
22617 7076 Acked Write 9 Done Done Sent false
22618 7077 Acked Read 10 Done Done Sent false
22619 7078 Acked Write 4 Done Done Sent false
22620 7079 Acked Write 9 Done Done Sent false
22621 7080 Acked Read 2 Done Done Sent false
22622 7081 Acked Write 4 Done Done Sent false
22623 7082 Acked Read 5 Done Done Sent false
22624 7083 Acked Flush 0 Done Done Sent false
22625 7084 Acked Read 8 Done Done Sent false
22626 7085 Acked Write 3 Done Done Sent false
22627 7086 Acked Read 2 Done Done Sent false
22628 7087 Acked Write 9 Done Done Sent false
22629 7088 Acked Write 1 Done Done Sent false
22630 7089 Acked Write 9 Done Done Sent false
22631 7090 Acked Flush 0 Done Done Sent false
22632 7091 Acked Read 2 Done Done Sent false
22633 7092 Acked Write 5 Done Done Sent false
22634 7093 Acked Write 9 Done Done Sent false
22635 7094 Acked Read 3 Done Done Sent false
22636 7095 Acked Write 7 Done Done Sent false
22637 7096 Acked Read 2 Done Done Sent false
22638 7097 Acked Write 7 Done Done Sent false
22639 7098 Acked Read 5 Done Done Sent false
22640 7099 Acked Write 2 Done Done Sent false
22641 7100 Acked Read 5 Done Done Sent false
22642 7101 Acked Write 10 Done Done Sent false
22643 7102 Acked Read 4 Done Done Sent false
22644 7103 Acked Read 6 Done Done Sent false
22645 7104 Acked Read 9 Done Done Sent false
22646 7105 Acked Write 5 Done Done Sent false
22647 7106 Acked Read 6 Done Done Sent false
22648 7107 Acked Read 6 Done Done Sent false
22649 7108 Acked Flush 0 Done Done Sent false
22650 7109 Acked Read 4 Done Done Sent false
22651 7110 Acked Read 4 Done Done Sent false
22652 7111 Acked Write 9 Done Done Sent false
22653 7112 Acked Read 6 Done Done Sent false
22654 7113 Acked Write 3 Done Done Sent false
22655 7114 Acked Read 10 Done Done Sent false
22656 7115 Acked Read 8 Done Done Sent false
22657 7116 Acked Read 8 Done Done Sent false
22658 7117 Acked Write 7 Done Done Sent false
22659 7118 Acked Write 4 Done Done Sent false
22660 7119 Acked Read 9 Done Done Sent false
22661 7120 Acked Read 9 Done Done Sent false
22662 7121 Acked Read 2 Done Done Sent false
22663 7122 Acked Write 4 Done Done Sent false
22664 7123 Acked Read 1 Done Done Sent false
22665 7124 Acked Read 6 Done Done Sent false
22666 7125 Acked Read 7 Done Done Sent false
22667 7126 Acked Read 8 Done Done Sent false
22668 7127 Acked Flush 0 Done Done Sent false
22669 7128 Acked Write 5 Done Done Sent false
22670 7129 Acked Read 6 Done Done Sent false
22671 7130 Acked Read 1 Done Done Sent false
22672 7131 Acked Write 8 Done Done Sent false
22673 7132 Acked Read 10 Done Done Sent false
22674 7133 Acked Write 9 Done Done Sent false
22675 7134 Acked Read 5 Done Done Sent false
22676 7135 Acked Write 4 Done Done Sent false
22677 7136 Acked Flush 0 Done Done Sent false
22678 7137 Acked Read 5 Done Done Sent false
22679 7138 Acked Read 7 Done Done Sent false
22680 7139 Acked Write 3 Done Done Sent false
22681 7140 Acked Write 4 Done Done Sent false
22682 7141 Acked Read 6 Done Done Sent false
22683 7142 Acked Read 9 Done Done Sent false
22684 7143 Acked Flush 0 Done Done Sent false
22685 7144 Acked Read 4 Done Done Sent false
22686 7145 Acked Read 8 Done Done Sent false
22687 7146 Acked Flush 0 Done Done Sent false
22688 7147 Acked Write 5 Done Done Sent false
22689 7148 Acked Read 3 Done Done Sent false
22690 7149 Acked Flush 0 Done Done Sent false
22691 7150 Acked Read 8 Done Done Sent false
22692 7151 Acked Read 3 Done Done Sent false
22693 7152 Acked Write 3 Done Done Sent false
22694 7153 Acked Write 2 Done Done Sent false
22695 7154 Acked Write 9 Done Done Sent false
22696 7155 Acked Write 4 Done Done Sent false
22697 7156 Acked Read 5 Done Done Sent false
22698 7157 Acked Write 4 Done Done Sent false
22699 7158 Acked Write 5 Done Done Sent false
22700 7159 Acked Write 7 Done Done Sent false
22701 7160 Acked Read 6 Done Done Sent false
22702 7161 Acked Flush 0 Done Done Sent false
22703 7162 Acked Write 10 Done Done Sent false
22704 7163 Acked Read 5 Done Done Sent false
22705 7164 Acked Read 7 Done Done Sent false
22706 7165 Acked Read 2 Done Done Sent false
22707 7166 Acked Read 5 Done Done Sent false
22708 7167 Acked Read 2 Done Done Sent false
22709 7168 Acked Read 4 Done Done Sent false
22710 7169 Acked Read 10 Done Done Sent false
22711 7170 Acked Write 7 Done Done Sent false
22712 7171 Acked Read 6 Done Done Sent false
22713 7172 Acked Write 9 Done Done Sent false
22714 7173 Acked Read 1 Done Done Sent false
22715 7174 Acked Read 5 Done Done Sent false
22716 7175 Acked Read 9 Done Done Sent false
22717 7176 Acked Write 1 Done Done Sent false
22718 7177 Acked Write 7 Done Done Sent false
22719 7178 Acked Write 2 Done Done Sent false
22720 7179 Acked Write 1 Done Done Sent false
22721 7180 Acked Read 7 Done Done Sent false
22722 7181 Acked Read 2 Done Done Sent false
22723 7182 Acked Read 9 Done Done Sent false
22724 7183 Acked Read 5 Done Done Sent false
22725 7184 Acked Write 9 Done Done Sent false
22726 7185 Acked Write 7 Done Done Sent false
22727 7186 Acked Write 1 Done Done Sent false
22728 7187 Acked Write 6 Done Done Sent false
22729 7188 Acked Read 1 Done Done Sent false
22730 7189 Acked Read 3 Done Done Sent false
22731 7190 Acked Read 9 Done Done Sent false
22732 7191 Acked Read 9 Done Done Sent false
22733 7192 Acked Write 2 Done Done Sent false
22734 7193 Acked Read 2 Done Done Sent false
22735 7194 Acked Write 10 Done Done Sent false
22736 7195 Acked Read 3 Done Done Sent false
22737 7196 Acked Read 9 Done Done Sent false
22738 7197 Acked Flush 0 Done Done Sent false
22739 7198 Acked Read 2 Done Done Sent false
22740 7199 Acked Read 2 Done Done Sent false
22741 7200 Acked Write 8 Done Done Sent false
22742 7201 Acked Read 8 Done Done Sent false
22743 7202 Acked Write 2 Done Done Sent false
22744 7203 Acked Write 8 Done Done Sent false
22745 7204 Acked Read 5 Done Done Sent false
22746 7205 Acked Write 7 Done Done Sent false
22747 7206 Acked Flush 0 Done Done Sent false
22748 7207 Acked Write 10 Done Done Sent false
22749 7208 Acked Read 10 Done Done Sent false
22750 7209 Acked Read 10 Done Done Sent false
22751 7210 Acked Write 10 Done Done Sent false
22752 7211 Acked Read 6 Done Done Sent false
22753 7212 Acked Write 5 Done Done Sent false
22754 7213 Acked Write 10 Done Done Sent false
22755 7214 Acked Write 3 Done Done Sent false
22756 7215 Acked Read 3 Done Done Sent false
22757 7216 Acked Flush 0 Done Done Sent false
22758 7217 Acked Read 3 Done Done Sent false
22759 7218 Acked Read 10 Done Done Sent false
22760 7219 Acked Read 3 Done Done Sent false
22761 7220 Acked Write 2 Done Done Sent false
22762 7221 Acked Flush 0 Done Done Sent false
22763 7222 Acked Write 10 Done Done Sent false
22764 7223 Acked Read 10 Done Done Sent false
22765 7224 Acked Read 4 Done Done Sent false
22766 7225 Acked Write 10 Done Done Sent false
22767 7226 Acked Read 3 Done Done Sent false
22768 7227 Acked Write 8 Done Done Sent false
22769 7228 Acked Read 3 Done Done Sent false
22770 7229 Acked Write 7 Done Done Sent false
22771 7230 Acked Write 2 Done Done Sent false
22772 7231 Acked Read 9 Done Done Sent false
22773 7232 Acked Write 8 Done Done Sent false
22774 7233 Acked Read 5 Done Done Sent false
22775 7234 Acked Read 10 Done Done Sent false
22776 7235 Acked Read 6 Done Done Sent false
22777 7236 Acked Write 10 Done Done Sent false
22778 7237 Acked Flush 0 Done Done Sent false
22779 7238 Acked Write 10 Done Done Sent false
22780 7239 Acked Write 9 Done Done Sent false
22781 7240 Acked Write 5 Done Done Sent false
22782 7241 Acked Read 7 Done Done Sent false
22783 7242 Acked Read 7 Done Done Sent false
22784 7243 Acked Read 4 Done Done Sent false
22785 7244 Acked Read 2 Done Done Sent false
22786 7245 Acked Read 3 Done Done Sent false
22787 7246 Acked Write 4 Done Done Sent false
22788 7247 Acked Read 7 Done Done Sent false
22789 7248 Acked Read 9 Done Done Sent false
22790 7249 Acked Read 10 Done Done Sent false
22791 7250 Acked Write 8 Done Done Sent false
22792 7251 Acked Write 7 Done Done Sent false
22793 7252 Acked Write 3 Done Done Sent false
22794 7253 Acked Write 10 Done Done Sent false
22795 7254 Acked Write 2 Done Done Sent false
22796 7255 Acked Write 5 Done Done Sent false
22797 7256 Acked Write 8 Done Done Sent false
22798 7257 Acked Read 8 Done Done Sent false
22799 7258 Acked Read 4 Done Done Sent false
22800 7259 Acked Read 10 Done Done Sent false
22801 7260 Acked Read 3 Done Done Sent false
22802 7261 Acked Read 5 Done Done Sent false
22803 7262 Acked Flush 0 Done Done Sent false
22804 7263 Acked Read 6 Done Done Sent false
22805 7264 Acked Read 2 Done Done Sent false
22806 7265 Acked Write 3 Done Done Sent false
22807 7266 Acked Read 10 Done Done Sent false
22808 7267 Acked Write 1 Done Done Sent false
22809 7268 Acked Write 9 Done Done Sent false
22810 7269 Acked Write 3 Done Done Sent false
22811 7270 Acked Write 6 Done Done Sent false
22812 7271 Acked Read 9 Done Done Sent false
22813 7272 Acked Flush 0 Done Done Sent false
22814 7273 Acked Write 10 Done Done Sent false
22815 7274 Acked Read 6 Done Done Sent false
22816 7275 Acked Flush 0 Done Done Sent false
22817 7276 Acked Read 5 Done Done Sent false
22818 7277 Acked Read 6 Done Done Sent false
22819 7278 Acked Read 9 Done Done Sent false
22820 7279 Acked Write 3 Done Done Sent false
22821 7280 Acked Read 8 Done Done Sent false
22822 7281 Acked Write 2 Done Done Sent false
22823 7282 Acked Read 5 Done Done Sent false
22824 7283 Acked Read 8 Done Done Sent false
22825 7284 Acked Read 3 Done Done Sent false
22826 7285 Acked Read 7 Done Done Sent false
22827 7286 Acked Flush 0 Done Done Sent false
22828 7287 Acked Write 9 Done Done Sent false
22829 7288 Acked Read 2 Done Done Sent false
22830 7289 Acked Read 6 Done Done Sent false
22831 7290 Acked Read 3 Done Done Sent false
22832 7291 Acked Read 8 Done Done Sent false
22833 7292 Acked Write 10 Done Done Sent false
22834 7293 Acked Read 4 Done Done Sent false
22835 7294 Acked Read 7 Done Done Sent false
22836 7295 Acked Read 5 Done Done Sent false
22837 7296 Acked Read 1 Done Done Sent false
22838 7297 Acked Read 10 Done Done Sent false
22839 7298 Acked Flush 0 Done Done Sent false
22840 7299 Acked Read 1 Done Done Sent false
22841 7300 Acked Write 6 Done Done Sent false
22842 7301 Acked Read 7 Done Done Sent false
22843 7302 Acked Write 8 Done Done Sent false
22844 7303 Acked Read 6 Done Done Sent false
22845 7304 Acked Write 3 Done Done Sent false
22846 7305 Acked Read 8 Done Done Sent false
22847 7306 Acked Write 4 Done Done Sent false
22848 7307 Acked Read 10 Done Done Sent false
22849 7308 Acked Read 8 Done Done Sent false
22850 7309 Acked Read 7 Done Done Sent false
22851 7310 Acked Read 1 Done Done Sent false
22852 7311 Acked Flush 0 Done Done Sent false
22853 7312 Acked Read 5 Done Done Sent false
22854 7313 Acked Flush 0 Done Done Sent false
22855 7314 Acked Read 2 Done Done Sent false
22856 7315 Acked Read 2 Done Done Sent false
22857 7316 Acked Read 8 Done Done Sent false
22858 7317 Acked Write 8 Done Done Sent false
22859 7318 Acked Write 9 Done Done Sent false
22860 7319 Acked Read 3 Done Done Sent false
22861 7320 Acked Read 9 Done Done Sent false
22862 7321 Acked Write 4 Done Done Sent false
22863 7322 Acked Write 6 Done Done Sent false
22864 7323 Acked Write 10 Done Done Sent false
22865 7324 Acked Read 1 Done Done Sent false
22866 7325 Acked Flush 0 Done Done Sent false
22867 7326 Acked Write 5 Done Done Sent false
22868 7327 Acked Flush 0 Done Done Sent false
22869 7328 Acked Flush 0 Done Done Sent false
22870 7329 Acked Write 1 Done Done Sent false
22871 7330 Acked Flush 0 Done Done Sent false
22872 7331 Acked Write 10 Done Done Sent false
22873 7332 Acked Write 1 Done Done Sent false
22874 7333 Acked Flush 0 Done Done Sent false
22875 7334 Acked Write 1 Done Done Sent false
22876 7335 Acked Write 4 Done Done Sent false
22877 7336 Acked Flush 0 Done Done Sent false
22878 7337 Acked Read 2 Done Done Sent false
22879 7338 Acked Read 1 Done Done Sent false
22880 7339 Acked Write 2 Done Done Sent false
22881 7340 Acked Read 7 Done Done Sent false
22882 STATES DS:0 DS:1 DS:2 TOTAL
22883 Sent 0 0 301 301
22884 Done 301 301 0 602
22885 Skipped 0 0 0 0
22886 Error 0 0 0 0
22887 Last Flush: 7336 7336 7039
22888 Downstairs last five completed: 7039 7038 7037 7036 7035
22889 Upstairs last five completed: 7340 7339 7338 7337 7336
22890 CLIENT: Up:0 ds:301 act:2
22891 ----------------------------------------------------------------
22892 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
22893 States: Active Active Offline
22894 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22895 7040 Acked Read 7 Done Done Sent false
22896 7041 Acked Write 4 Done Done Sent false
22897 7042 Acked Read 3 Done Done Sent false
22898 7043 Acked Flush 0 Done Done Sent false
22899 7044 Acked Read 8 Done Done Sent false
22900 7045 Acked Write 5 Done Done Sent false
22901 7046 Acked Read 7 Done Done Sent false
22902 7047 Acked Read 1 Done Done Sent false
22903 7048 Acked Write 10 Done Done Sent false
22904 7049 Acked Write 3 Done Done Sent false
22905 7050 Acked Read 2 Done Done Sent false
22906 7051 Acked Write 10 Done Done Sent false
22907 7052 Acked Write 4 Done Done Sent false
22908 7053 Acked Flush 0 Done Done Sent false
22909 7054 Acked Read 2 Done Done Sent false
22910 7055 Acked Read 10 Done Done Sent false
22911 7056 Acked Read 4 Done Done Sent false
22912 7057 Acked Read 1 Done Done Sent false
22913 7058 Acked Write 4 Done Done Sent false
22914 7059 Acked Write 3 Done Done Sent false
22915 7060 Acked Read 4 Done Done Sent false
22916 7061 Acked Write 2 Done Done Sent false
22917 7062 Acked Write 9 Done Done Sent false
22918 7063 Acked Read 8 Done Done Sent false
22919 7064 Acked Read 1 Done Done Sent false
22920 7065 Acked Read 5 Done Done Sent false
22921 7066 Acked Flush 0 Done Done Sent false
22922 7067 Acked Read 10 Done Done Sent false
22923 7068 Acked Flush 0 Done Done Sent false
22924 7069 Acked Write 9 Done Done Sent false
22925 7070 Acked Write 3 Done Done Sent false
22926 7071 Acked Read 9 Done Done Sent false
22927 7072 Acked Read 4 Done Done Sent false
22928 7073 Acked Read 9 Done Done Sent false
22929 7074 Acked Read 10 Done Done Sent false
22930 7075 Acked Read 1 Done Done Sent false
22931 7076 Acked Write 9 Done Done Sent false
22932 7077 Acked Read 10 Done Done Sent false
22933 7078 Acked Write 4 Done Done Sent false
22934 7079 Acked Write 9 Done Done Sent false
22935 7080 Acked Read 2 Done Done Sent false
22936 7081 Acked Write 4 Done Done Sent false
22937 7082 Acked Read 5 Done Done Sent false
22938 7083 Acked Flush 0 Done Done Sent false
22939 7084 Acked Read 8 Done Done Sent false
22940 7085 Acked Write 3 Done Done Sent false
22941 7086 Acked Read 2 Done Done Sent false
22942 7087 Acked Write 9 Done Done Sent false
22943 7088 Acked Write 1 Done Done Sent false
22944 7089 Acked Write 9 Done Done Sent false
22945 7090 Acked Flush 0 Done Done Sent false
22946 7091 Acked Read 2 Done Done Sent false
22947 7092 Acked Write 5 Done Done Sent false
22948 7093 Acked Write 9 Done Done Sent false
22949 7094 Acked Read 3 Done Done Sent false
22950 7095 Acked Write 7 Done Done Sent false
22951 7096 Acked Read 2 Done Done Sent false
22952 7097 Acked Write 7 Done Done Sent false
22953 7098 Acked Read 5 Done Done Sent false
22954 7099 Acked Write 2 Done Done Sent false
22955 7100 Acked Read 5 Done Done Sent false
22956 7101 Acked Write 10 Done Done Sent false
22957 7102 Acked Read 4 Done Done Sent false
22958 7103 Acked Read 6 Done Done Sent false
22959 7104 Acked Read 9 Done Done Sent false
22960 7105 Acked Write 5 Done Done Sent false
22961 7106 Acked Read 6 Done Done Sent false
22962 7107 Acked Read 6 Done Done Sent false
22963 7108 Acked Flush 0 Done Done Sent false
22964 7109 Acked Read 4 Done Done Sent false
22965 7110 Acked Read 4 Done Done Sent false
22966 7111 Acked Write 9 Done Done Sent false
22967 7112 Acked Read 6 Done Done Sent false
22968 7113 Acked Write 3 Done Done Sent false
22969 7114 Acked Read 10 Done Done Sent false
22970 7115 Acked Read 8 Done Done Sent false
22971 7116 Acked Read 8 Done Done Sent false
22972 7117 Acked Write 7 Done Done Sent false
22973 7118 Acked Write 4 Done Done Sent false
22974 7119 Acked Read 9 Done Done Sent false
22975 7120 Acked Read 9 Done Done Sent false
22976 7121 Acked Read 2 Done Done Sent false
22977 7122 Acked Write 4 Done Done Sent false
22978 7123 Acked Read 1 Done Done Sent false
22979 7124 Acked Read 6 Done Done Sent false
22980 7125 Acked Read 7 Done Done Sent false
22981 7126 Acked Read 8 Done Done Sent false
22982 7127 Acked Flush 0 Done Done Sent false
22983 7128 Acked Write 5 Done Done Sent false
22984 7129 Acked Read 6 Done Done Sent false
22985 7130 Acked Read 1 Done Done Sent false
22986 7131 Acked Write 8 Done Done Sent false
22987 7132 Acked Read 10 Done Done Sent false
22988 7133 Acked Write 9 Done Done Sent false
22989 7134 Acked Read 5 Done Done Sent false
22990 7135 Acked Write 4 Done Done Sent false
22991 7136 Acked Flush 0 Done Done Sent false
22992 7137 Acked Read 5 Done Done Sent false
22993 7138 Acked Read 7 Done Done Sent false
22994 7139 Acked Write 3 Done Done Sent false
22995 7140 Acked Write 4 Done Done Sent false
22996 7141 Acked Read 6 Done Done Sent false
22997 7142 Acked Read 9 Done Done Sent false
22998 7143 Acked Flush 0 Done Done Sent false
22999 7144 Acked Read 4 Done Done Sent false
23000 7145 Acked Read 8 Done Done Sent false
23001 7146 Acked Flush 0 Done Done Sent false
23002 7147 Acked Write 5 Done Done Sent false
23003 7148 Acked Read 3 Done Done Sent false
23004 7149 Acked Flush 0 Done Done Sent false
23005 7150 Acked Read 8 Done Done Sent false
23006 7151 Acked Read 3 Done Done Sent false
23007 7152 Acked Write 3 Done Done Sent false
23008 7153 Acked Write 2 Done Done Sent false
23009 7154 Acked Write 9 Done Done Sent false
23010 7155 Acked Write 4 Done Done Sent false
23011 7156 Acked Read 5 Done Done Sent false
23012 7157 Acked Write 4 Done Done Sent false
23013 7158 Acked Write 5 Done Done Sent false
23014 7159 Acked Write 7 Done Done Sent false
23015 7160 Acked Read 6 Done Done Sent false
23016 7161 Acked Flush 0 Done Done Sent false
23017 7162 Acked Write 10 Done Done Sent false
23018 7163 Acked Read 5 Done Done Sent false
23019 7164 Acked Read 7 Done Done Sent false
23020 7165 Acked Read 2 Done Done Sent false
23021 7166 Acked Read 5 Done Done Sent false
23022 7167 Acked Read 2 Done Done Sent false
23023 7168 Acked Read 4 Done Done Sent false
23024 7169 Acked Read 10 Done Done Sent false
23025 7170 Acked Write 7 Done Done Sent false
23026 7171 Acked Read 6 Done Done Sent false
23027 7172 Acked Write 9 Done Done Sent false
23028 7173 Acked Read 1 Done Done Sent false
23029 7174 Acked Read 5 Done Done Sent false
23030 7175 Acked Read 9 Done Done Sent false
23031 7176 Acked Write 1 Done Done Sent false
23032 7177 Acked Write 7 Done Done Sent false
23033 7178 Acked Write 2 Done Done Sent false
23034 7179 Acked Write 1 Done Done Sent false
23035 7180 Acked Read 7 Done Done Sent false
23036 7181 Acked Read 2 Done Done Sent false
23037 7182 Acked Read 9 Done Done Sent false
23038 7183 Acked Read 5 Done Done Sent false
23039 7184 Acked Write 9 Done Done Sent false
23040 7185 Acked Write 7 Done Done Sent false
23041 7186 Acked Write 1 Done Done Sent false
23042 7187 Acked Write 6 Done Done Sent false
23043 7188 Acked Read 1 Done Done Sent false
23044 7189 Acked Read 3 Done Done Sent false
23045 7190 Acked Read 9 Done Done Sent false
23046 7191 Acked Read 9 Done Done Sent false
23047 7192 Acked Write 2 Done Done Sent false
23048 7193 Acked Read 2 Done Done Sent false
23049 7194 Acked Write 10 Done Done Sent false
23050 7195 Acked Read 3 Done Done Sent false
23051 7196 Acked Read 9 Done Done Sent false
23052 7197 Acked Flush 0 Done Done Sent false
23053 7198 Acked Read 2 Done Done Sent false
23054 7199 Acked Read 2 Done Done Sent false
23055 7200 Acked Write 8 Done Done Sent false
23056 7201 Acked Read 8 Done Done Sent false
23057 7202 Acked Write 2 Done Done Sent false
23058 7203 Acked Write 8 Done Done Sent false
23059 7204 Acked Read 5 Done Done Sent false
23060 7205 Acked Write 7 Done Done Sent false
23061 7206 Acked Flush 0 Done Done Sent false
23062 7207 Acked Write 10 Done Done Sent false
23063 7208 Acked Read 10 Done Done Sent false
23064 7209 Acked Read 10 Done Done Sent false
23065 7210 Acked Write 10 Done Done Sent false
23066 7211 Acked Read 6 Done Done Sent false
23067 7212 Acked Write 5 Done Done Sent false
23068 7213 Acked Write 10 Done Done Sent false
23069 7214 Acked Write 3 Done Done Sent false
23070 7215 Acked Read 3 Done Done Sent false
23071 7216 Acked Flush 0 Done Done Sent false
23072 7217 Acked Read 3 Done Done Sent false
23073 7218 Acked Read 10 Done Done Sent false
23074 7219 Acked Read 3 Done Done Sent false
23075 7220 Acked Write 2 Done Done Sent false
23076 7221 Acked Flush 0 Done Done Sent false
23077 7222 Acked Write 10 Done Done Sent false
23078 7223 Acked Read 10 Done Done Sent false
23079 7224 Acked Read 4 Done Done Sent false
23080 7225 Acked Write 10 Done Done Sent false
23081 7226 Acked Read 3 Done Done Sent false
23082 7227 Acked Write 8 Done Done Sent false
23083 7228 Acked Read 3 Done Done Sent false
23084 7229 Acked Write 7 Done Done Sent false
23085 7230 Acked Write 2 Done Done Sent false
23086 7231 Acked Read 9 Done Done Sent false
23087 7232 Acked Write 8 Done Done Sent false
23088 7233 Acked Read 5 Done Done Sent false
23089 7234 Acked Read 10 Done Done Sent false
23090 7235 Acked Read 6 Done Done Sent false
23091 7236 Acked Write 10 Done Done Sent false
23092 7237 Acked Flush 0 Done Done Sent false
23093 7238 Acked Write 10 Done Done Sent false
23094 7239 Acked Write 9 Done Done Sent false
23095 7240 Acked Write 5 Done Done Sent false
23096 7241 Acked Read 7 Done Done Sent false
23097 7242 Acked Read 7 Done Done Sent false
23098 7243 Acked Read 4 Done Done Sent false
23099 7244 Acked Read 2 Done Done Sent false
23100 7245 Acked Read 3 Done Done Sent false
23101 7246 Acked Write 4 Done Done Sent false
23102 7247 Acked Read 7 Done Done Sent false
23103 7248 Acked Read 9 Done Done Sent false
23104 7249 Acked Read 10 Done Done Sent false
23105 7250 Acked Write 8 Done Done Sent false
23106 7251 Acked Write 7 Done Done Sent false
23107 7252 Acked Write 3 Done Done Sent false
23108 7253 Acked Write 10 Done Done Sent false
23109 7254 Acked Write 2 Done Done Sent false
23110 7255 Acked Write 5 Done Done Sent false
23111 7256 Acked Write 8 Done Done Sent false
23112 7257 Acked Read 8 Done Done Sent false
23113 7258 Acked Read 4 Done Done Sent false
23114 7259 Acked Read 10 Done Done Sent false
23115 7260 Acked Read 3 Done Done Sent false
23116 7261 Acked Read 5 Done Done Sent false
23117 7262 Acked Flush 0 Done Done Sent false
23118 7263 Acked Read 6 Done Done Sent false
23119 7264 Acked Read 2 Done Done Sent false
23120 7265 Acked Write 3 Done Done Sent false
23121 7266 Acked Read 10 Done Done Sent false
23122 7267 Acked Write 1 Done Done Sent false
23123 7268 Acked Write 9 Done Done Sent false
23124 7269 Acked Write 3 Done Done Sent false
23125 7270 Acked Write 6 Done Done Sent false
23126 7271 Acked Read 9 Done Done Sent false
23127 7272 Acked Flush 0 Done Done Sent false
23128 7273 Acked Write 10 Done Done Sent false
23129 7274 Acked Read 6 Done Done Sent false
23130 7275 Acked Flush 0 Done Done Sent false
23131 7276 Acked Read 5 Done Done Sent false
23132 7277 Acked Read 6 Done Done Sent false
23133 7278 Acked Read 9 Done Done Sent false
23134 7279 Acked Write 3 Done Done Sent false
23135 7280 Acked Read 8 Done Done Sent false
23136 7281 Acked Write 2 Done Done Sent false
23137 7282 Acked Read 5 Done Done Sent false
23138 7283 Acked Read 8 Done Done Sent false
23139 7284 Acked Read 3 Done Done Sent false
23140 7285 Acked Read 7 Done Done Sent false
23141 7286 Acked Flush 0 Done Done Sent false
23142 7287 Acked Write 9 Done Done Sent false
23143 7288 Acked Read 2 Done Done Sent false
23144 7289 Acked Read 6 Done Done Sent false
23145 7290 Acked Read 3 Done Done Sent false
23146 7291 Acked Read 8 Done Done Sent false
23147 7292 Acked Write 10 Done Done Sent false
23148 7293 Acked Read 4 Done Done Sent false
23149 7294 Acked Read 7 Done Done Sent false
23150 7295 Acked Read 5 Done Done Sent false
23151 7296 Acked Read 1 Done Done Sent false
23152 7297 Acked Read 10 Done Done Sent false
23153 7298 Acked Flush 0 Done Done Sent false
23154 7299 Acked Read 1 Done Done Sent false
23155 7300 Acked Write 6 Done Done Sent false
23156 7301 Acked Read 7 Done Done Sent false
23157 7302 Acked Write 8 Done Done Sent false
23158 7303 Acked Read 6 Done Done Sent false
23159 7304 Acked Write 3 Done Done Sent false
23160 7305 Acked Read 8 Done Done Sent false
23161 7306 Acked Write 4 Done Done Sent false
23162 7307 Acked Read 10 Done Done Sent false
23163 7308 Acked Read 8 Done Done Sent false
23164 7309 Acked Read 7 Done Done Sent false
23165 7310 Acked Read 1 Done Done Sent false
23166 7311 Acked Flush 0 Done Done Sent false
23167 7312 Acked Read 5 Done Done Sent false
23168 7313 Acked Flush 0 Done Done Sent false
23169 7314 Acked Read 2 Done Done Sent false
23170 7315 Acked Read 2 Done Done Sent false
23171 7316 Acked Read 8 Done Done Sent false
23172 7317 Acked Write 8 Done Done Sent false
23173 7318 Acked Write 9 Done Done Sent false
23174 7319 Acked Read 3 Done Done Sent false
23175 7320 Acked Read 9 Done Done Sent false
23176 7321 Acked Write 4 Done Done Sent false
23177 7322 Acked Write 6 Done Done Sent false
23178 7323 Acked Write 10 Done Done Sent false
23179 7324 Acked Read 1 Done Done Sent false
23180 7325 Acked Flush 0 Done Done Sent false
23181 7326 Acked Write 5 Done Done Sent false
23182 7327 Acked Flush 0 Done Done Sent false
23183 7328 Acked Flush 0 Done Done Sent false
23184 7329 Acked Write 1 Done Done Sent false
23185 7330 Acked Flush 0 Done Done Sent false
23186 7331 Acked Write 10 Done Done Sent false
23187 7332 Acked Write 1 Done Done Sent false
23188 7333 Acked Flush 0 Done Done Sent false
23189 7334 Acked Write 1 Done Done Sent false
23190 7335 Acked Write 4 Done Done Sent false
23191 7336 Acked Flush 0 Done Done Sent false
23192 7337 Acked Read 2 Done Done Sent false
23193 7338 Acked Read 1 Done Done Sent false
23194 7339 Acked Write 2 Done Done Sent false
23195 7340 Acked Read 7 Done Done Sent false
23196 7341 Acked Flush 0 Done Done Sent false
23197 STATES DS:0 DS:1 DS:2 TOTAL
23198 Sent 0 0 302 302
23199 Done 302 302 0 604
23200 Skipped 0 0 0 0
23201 Error 0 0 0 0
23202 Last Flush: 7341 7341 7039
23203 Downstairs last five completed: 7039 7038 7037 7036 7035
23204 Upstairs last five completed: 7341 7340 7339 7338 7337
23205 CLIENT: Up:0 ds:302 act:2
232062024-12-09T15:12:42.065ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
232072024-12-09T15:12:42.066ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
232082024-12-09T15:12:42.066ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
232092024-12-09T15:12:42.066ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
232102024-12-09T15:12:42.066ZINFOcrucible: client task is exiting = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
23211 [8830] got start action
23212 [8830] I am going to restart
23213 Starting downstairs at port 8830
23214 Make output file at "/tmp/dsc/downstairs-8830.txt"
23215 [8830][2] reports Starting
23216 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1222)
23217 [8830][2] reports Running
23218 ----------------------------------------------------------------
23219 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
23220 States: Active Active Offline
23221 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23222 7040 Acked Read 7 Done Done Sent false
23223 7041 Acked Write 4 Done Done Sent false
23224 7042 Acked Read 3 Done Done Sent false
23225 7043 Acked Flush 0 Done Done Sent false
23226 7044 Acked Read 8 Done Done Sent false
23227 7045 Acked Write 5 Done Done Sent false
23228 7046 Acked Read 7 Done Done Sent false
23229 7047 Acked Read 1 Done Done Sent false
23230 7048 Acked Write 10 Done Done Sent false
23231 7049 Acked Write 3 Done Done Sent false
23232 7050 Acked Read 2 Done Done Sent false
23233 7051 Acked Write 10 Done Done Sent false
23234 7052 Acked Write 4 Done Done Sent false
23235 7053 Acked Flush 0 Done Done Sent false
23236 7054 Acked Read 2 Done Done Sent false
23237 7055 Acked Read 10 Done Done Sent false
23238 7056 Acked Read 4 Done Done Sent false
23239 7057 Acked Read 1 Done Done Sent false
23240 7058 Acked Write 4 Done Done Sent false
23241 7059 Acked Write 3 Done Done Sent false
23242 7060 Acked Read 4 Done Done Sent false
23243 7061 Acked Write 2 Done Done Sent false
23244 7062 Acked Write 9 Done Done Sent false
23245 7063 Acked Read 8 Done Done Sent false
23246 7064 Acked Read 1 Done Done Sent false
23247 7065 Acked Read 5 Done Done Sent false
23248 7066 Acked Flush 0 Done Done Sent false
23249 7067 Acked Read 10 Done Done Sent false
23250 7068 Acked Flush 0 Done Done Sent false
23251 7069 Acked Write 9 Done Done Sent false
23252 7070 Acked Write 3 Done Done Sent false
23253 7071 Acked Read 9 Done Done Sent false
23254 7072 Acked Read 4 Done Done Sent false
23255 7073 Acked Read 9 Done Done Sent false
23256 7074 Acked Read 10 Done Done Sent false
23257 7075 Acked Read 1 Done Done Sent false
23258 7076 Acked Write 9 Done Done Sent false
23259 7077 Acked Read 10 Done Done Sent false
23260 7078 Acked Write 4 Done Done Sent false
23261 7079 Acked Write 9 Done Done Sent false
23262 7080 Acked Read 2 Done Done Sent false
23263 7081 Acked Write 4 Done Done Sent false
23264 7082 Acked Read 5 Done Done Sent false
23265 7083 Acked Flush 0 Done Done Sent false
23266 7084 Acked Read 8 Done Done Sent false
23267 7085 Acked Write 3 Done Done Sent false
23268 7086 Acked Read 2 Done Done Sent false
23269 7087 Acked Write 9 Done Done Sent false
23270 7088 Acked Write 1 Done Done Sent false
23271 7089 Acked Write 9 Done Done Sent false
23272 7090 Acked Flush 0 Done Done Sent false
23273 7091 Acked Read 2 Done Done Sent false
23274 7092 Acked Write 5 Done Done Sent false
23275 7093 Acked Write 9 Done Done Sent false
23276 7094 Acked Read 3 Done Done Sent false
23277 7095 Acked Write 7 Done Done Sent false
23278 7096 Acked Read 2 Done Done Sent false
23279 7097 Acked Write 7 Done Done Sent false
23280 7098 Acked Read 5 Done Done Sent false
23281 7099 Acked Write 2 Done Done Sent false
23282 7100 Acked Read 5 Done Done Sent false
23283 7101 Acked Write 10 Done Done Sent false
23284 7102 Acked Read 4 Done Done Sent false
23285 7103 Acked Read 6 Done Done Sent false
23286 7104 Acked Read 9 Done Done Sent false
23287 7105 Acked Write 5 Done Done Sent false
23288 7106 Acked Read 6 Done Done Sent false
23289 7107 Acked Read 6 Done Done Sent false
23290 7108 Acked Flush 0 Done Done Sent false
23291 7109 Acked Read 4 Done Done Sent false
23292 7110 Acked Read 4 Done Done Sent false
23293 7111 Acked Write 9 Done Done Sent false
23294 7112 Acked Read 6 Done Done Sent false
23295 7113 Acked Write 3 Done Done Sent false
23296 7114 Acked Read 10 Done Done Sent false
23297 7115 Acked Read 8 Done Done Sent false
23298 7116 Acked Read 8 Done Done Sent false
23299 7117 Acked Write 7 Done Done Sent false
23300 7118 Acked Write 4 Done Done Sent false
23301 7119 Acked Read 9 Done Done Sent false
23302 7120 Acked Read 9 Done Done Sent false
23303 7121 Acked Read 2 Done Done Sent false
23304 7122 Acked Write 4 Done Done Sent false
23305 7123 Acked Read 1 Done Done Sent false
23306 7124 Acked Read 6 Done Done Sent false
23307 7125 Acked Read 7 Done Done Sent false
23308 7126 Acked Read 8 Done Done Sent false
23309 7127 Acked Flush 0 Done Done Sent false
23310 7128 Acked Write 5 Done Done Sent false
23311 7129 Acked Read 6 Done Done Sent false
23312 7130 Acked Read 1 Done Done Sent false
23313 7131 Acked Write 8 Done Done Sent false
23314 7132 Acked Read 10 Done Done Sent false
23315 7133 Acked Write 9 Done Done Sent false
23316 7134 Acked Read 5 Done Done Sent false
23317 7135 Acked Write 4 Done Done Sent false
23318 7136 Acked Flush 0 Done Done Sent false
23319 7137 Acked Read 5 Done Done Sent false
23320 7138 Acked Read 7 Done Done Sent false
23321 7139 Acked Write 3 Done Done Sent false
23322 7140 Acked Write 4 Done Done Sent false
23323 7141 Acked Read 6 Done Done Sent false
23324 7142 Acked Read 9 Done Done Sent false
23325 7143 Acked Flush 0 Done Done Sent false
23326 7144 Acked Read 4 Done Done Sent false
23327 7145 Acked Read 8 Done Done Sent false
23328 7146 Acked Flush 0 Done Done Sent false
23329 7147 Acked Write 5 Done Done Sent false
23330 7148 Acked Read 3 Done Done Sent false
23331 7149 Acked Flush 0 Done Done Sent false
23332 7150 Acked Read 8 Done Done Sent false
23333 7151 Acked Read 3 Done Done Sent false
23334 7152 Acked Write 3 Done Done Sent false
23335 7153 Acked Write 2 Done Done Sent false
23336 7154 Acked Write 9 Done Done Sent false
23337 7155 Acked Write 4 Done Done Sent false
23338 7156 Acked Read 5 Done Done Sent false
23339 7157 Acked Write 4 Done Done Sent false
23340 7158 Acked Write 5 Done Done Sent false
23341 7159 Acked Write 7 Done Done Sent false
23342 7160 Acked Read 6 Done Done Sent false
23343 7161 Acked Flush 0 Done Done Sent false
23344 7162 Acked Write 10 Done Done Sent false
23345 7163 Acked Read 5 Done Done Sent false
23346 7164 Acked Read 7 Done Done Sent false
23347 7165 Acked Read 2 Done Done Sent false
23348 7166 Acked Read 5 Done Done Sent false
23349 7167 Acked Read 2 Done Done Sent false
23350 7168 Acked Read 4 Done Done Sent false
23351 7169 Acked Read 10 Done Done Sent false
23352 7170 Acked Write 7 Done Done Sent false
23353 7171 Acked Read 6 Done Done Sent false
23354 7172 Acked Write 9 Done Done Sent false
23355 7173 Acked Read 1 Done Done Sent false
23356 7174 Acked Read 5 Done Done Sent false
23357 7175 Acked Read 9 Done Done Sent false
23358 7176 Acked Write 1 Done Done Sent false
23359 7177 Acked Write 7 Done Done Sent false
23360 7178 Acked Write 2 Done Done Sent false
23361 7179 Acked Write 1 Done Done Sent false
23362 7180 Acked Read 7 Done Done Sent false
23363 7181 Acked Read 2 Done Done Sent false
23364 7182 Acked Read 9 Done Done Sent false
23365 7183 Acked Read 5 Done Done Sent false
23366 7184 Acked Write 9 Done Done Sent false
23367 7185 Acked Write 7 Done Done Sent false
23368 7186 Acked Write 1 Done Done Sent false
23369 7187 Acked Write 6 Done Done Sent false
23370 7188 Acked Read 1 Done Done Sent false
23371 7189 Acked Read 3 Done Done Sent false
23372 7190 Acked Read 9 Done Done Sent false
23373 7191 Acked Read 9 Done Done Sent false
23374 7192 Acked Write 2 Done Done Sent false
23375 7193 Acked Read 2 Done Done Sent false
23376 7194 Acked Write 10 Done Done Sent false
23377 7195 Acked Read 3 Done Done Sent false
23378 7196 Acked Read 9 Done Done Sent false
23379 7197 Acked Flush 0 Done Done Sent false
23380 7198 Acked Read 2 Done Done Sent false
23381 7199 Acked Read 2 Done Done Sent false
23382 7200 Acked Write 8 Done Done Sent false
23383 7201 Acked Read 8 Done Done Sent false
23384 7202 Acked Write 2 Done Done Sent false
23385 7203 Acked Write 8 Done Done Sent false
23386 7204 Acked Read 5 Done Done Sent false
23387 7205 Acked Write 7 Done Done Sent false
23388 7206 Acked Flush 0 Done Done Sent false
23389 7207 Acked Write 10 Done Done Sent false
23390 7208 Acked Read 10 Done Done Sent false
23391 7209 Acked Read 10 Done Done Sent false
23392 7210 Acked Write 10 Done Done Sent false
23393 7211 Acked Read 6 Done Done Sent false
23394 7212 Acked Write 5 Done Done Sent false
23395 7213 Acked Write 10 Done Done Sent false
23396 7214 Acked Write 3 Done Done Sent false
23397 7215 Acked Read 3 Done Done Sent false
23398 7216 Acked Flush 0 Done Done Sent false
23399 7217 Acked Read 3 Done Done Sent false
23400 7218 Acked Read 10 Done Done Sent false
23401 7219 Acked Read 3 Done Done Sent false
23402 7220 Acked Write 2 Done Done Sent false
23403 7221 Acked Flush 0 Done Done Sent false
23404 7222 Acked Write 10 Done Done Sent false
23405 7223 Acked Read 10 Done Done Sent false
23406 7224 Acked Read 4 Done Done Sent false
23407 7225 Acked Write 10 Done Done Sent false
23408 7226 Acked Read 3 Done Done Sent false
23409 7227 Acked Write 8 Done Done Sent false
23410 7228 Acked Read 3 Done Done Sent false
23411 7229 Acked Write 7 Done Done Sent false
23412 7230 Acked Write 2 Done Done Sent false
23413 7231 Acked Read 9 Done Done Sent false
23414 7232 Acked Write 8 Done Done Sent false
23415 7233 Acked Read 5 Done Done Sent false
23416 7234 Acked Read 10 Done Done Sent false
23417 7235 Acked Read 6 Done Done Sent false
23418 7236 Acked Write 10 Done Done Sent false
23419 7237 Acked Flush 0 Done Done Sent false
23420 7238 Acked Write 10 Done Done Sent false
23421 7239 Acked Write 9 Done Done Sent false
23422 7240 Acked Write 5 Done Done Sent false
23423 7241 Acked Read 7 Done Done Sent false
23424 7242 Acked Read 7 Done Done Sent false
23425 7243 Acked Read 4 Done Done Sent false
23426 7244 Acked Read 2 Done Done Sent false
23427 7245 Acked Read 3 Done Done Sent false
23428 7246 Acked Write 4 Done Done Sent false
23429 7247 Acked Read 7 Done Done Sent false
23430 7248 Acked Read 9 Done Done Sent false
23431 7249 Acked Read 10 Done Done Sent false
23432 7250 Acked Write 8 Done Done Sent false
23433 7251 Acked Write 7 Done Done Sent false
23434 7252 Acked Write 3 Done Done Sent false
23435 7253 Acked Write 10 Done Done Sent false
23436 7254 Acked Write 2 Done Done Sent false
23437 7255 Acked Write 5 Done Done Sent false
23438 7256 Acked Write 8 Done Done Sent false
23439 7257 Acked Read 8 Done Done Sent false
23440 7258 Acked Read 4 Done Done Sent false
23441 7259 Acked Read 10 Done Done Sent false
23442 7260 Acked Read 3 Done Done Sent false
23443 7261 Acked Read 5 Done Done Sent false
23444 7262 Acked Flush 0 Done Done Sent false
23445 7263 Acked Read 6 Done Done Sent false
23446 7264 Acked Read 2 Done Done Sent false
23447 7265 Acked Write 3 Done Done Sent false
23448 7266 Acked Read 10 Done Done Sent false
23449 7267 Acked Write 1 Done Done Sent false
23450 7268 Acked Write 9 Done Done Sent false
23451 7269 Acked Write 3 Done Done Sent false
23452 7270 Acked Write 6 Done Done Sent false
23453 7271 Acked Read 9 Done Done Sent false
23454 7272 Acked Flush 0 Done Done Sent false
23455 7273 Acked Write 10 Done Done Sent false
23456 7274 Acked Read 6 Done Done Sent false
23457 7275 Acked Flush 0 Done Done Sent false
23458 7276 Acked Read 5 Done Done Sent false
23459 7277 Acked Read 6 Done Done Sent false
23460 7278 Acked Read 9 Done Done Sent false
23461 7279 Acked Write 3 Done Done Sent false
23462 7280 Acked Read 8 Done Done Sent false
23463 7281 Acked Write 2 Done Done Sent false
23464 7282 Acked Read 5 Done Done Sent false
23465 7283 Acked Read 8 Done Done Sent false
23466 7284 Acked Read 3 Done Done Sent false
23467 7285 Acked Read 7 Done Done Sent false
23468 7286 Acked Flush 0 Done Done Sent false
23469 7287 Acked Write 9 Done Done Sent false
23470 7288 Acked Read 2 Done Done Sent false
23471 7289 Acked Read 6 Done Done Sent false
23472 7290 Acked Read 3 Done Done Sent false
23473 7291 Acked Read 8 Done Done Sent false
23474 7292 Acked Write 10 Done Done Sent false
23475 7293 Acked Read 4 Done Done Sent false
23476 7294 Acked Read 7 Done Done Sent false
23477 7295 Acked Read 5 Done Done Sent false
23478 7296 Acked Read 1 Done Done Sent false
23479 7297 Acked Read 10 Done Done Sent false
23480 7298 Acked Flush 0 Done Done Sent false
23481 7299 Acked Read 1 Done Done Sent false
23482 7300 Acked Write 6 Done Done Sent false
23483 7301 Acked Read 7 Done Done Sent false
23484 7302 Acked Write 8 Done Done Sent false
23485 7303 Acked Read 6 Done Done Sent false
23486 7304 Acked Write 3 Done Done Sent false
23487 7305 Acked Read 8 Done Done Sent false
23488 7306 Acked Write 4 Done Done Sent false
23489 7307 Acked Read 10 Done Done Sent false
23490 7308 Acked Read 8 Done Done Sent false
23491 7309 Acked Read 7 Done Done Sent false
23492 7310 Acked Read 1 Done Done Sent false
23493 7311 Acked Flush 0 Done Done Sent false
23494 7312 Acked Read 5 Done Done Sent false
23495 7313 Acked Flush 0 Done Done Sent false
23496 7314 Acked Read 2 Done Done Sent false
23497 7315 Acked Read 2 Done Done Sent false
23498 7316 Acked Read 8 Done Done Sent false
23499 7317 Acked Write 8 Done Done Sent false
23500 7318 Acked Write 9 Done Done Sent false
23501 7319 Acked Read 3 Done Done Sent false
23502 7320 Acked Read 9 Done Done Sent false
23503 7321 Acked Write 4 Done Done Sent false
23504 7322 Acked Write 6 Done Done Sent false
23505 7323 Acked Write 10 Done Done Sent false
23506 7324 Acked Read 1 Done Done Sent false
23507 7325 Acked Flush 0 Done Done Sent false
23508 7326 Acked Write 5 Done Done Sent false
23509 7327 Acked Flush 0 Done Done Sent false
23510 7328 Acked Flush 0 Done Done Sent false
23511 7329 Acked Write 1 Done Done Sent false
23512 7330 Acked Flush 0 Done Done Sent false
23513 7331 Acked Write 10 Done Done Sent false
23514 7332 Acked Write 1 Done Done Sent false
23515 7333 Acked Flush 0 Done Done Sent false
23516 7334 Acked Write 1 Done Done Sent false
23517 7335 Acked Write 4 Done Done Sent false
23518 7336 Acked Flush 0 Done Done Sent false
23519 7337 Acked Read 2 Done Done Sent false
23520 7338 Acked Read 1 Done Done Sent false
23521 7339 Acked Write 2 Done Done Sent false
23522 7340 Acked Read 7 Done Done Sent false
23523 7341 Acked Flush 0 Done Done Sent false
23524 STATES DS:0 DS:1 DS:2 TOTAL
23525 Sent 0 0 302 302
23526 Done 302 302 0 604
23527 Skipped 0 0 0 0
23528 Error 0 0 0 0
23529 Last Flush: 7341 7341 7039
23530 Downstairs last five completed: 7039 7038 7037 7036 7035
23531 Upstairs last five completed: 7341 7340 7339 7338 7337
23532 CLIENT: Up:0 ds:302 act:2
23533 ----------------------------------------------------------------
23534 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
23535 States: Active Active Offline
23536 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23537 7040 Acked Read 7 Done Done Sent false
23538 7041 Acked Write 4 Done Done Sent false
23539 7042 Acked Read 3 Done Done Sent false
23540 7043 Acked Flush 0 Done Done Sent false
23541 7044 Acked Read 8 Done Done Sent false
23542 7045 Acked Write 5 Done Done Sent false
23543 7046 Acked Read 7 Done Done Sent false
23544 7047 Acked Read 1 Done Done Sent false
23545 7048 Acked Write 10 Done Done Sent false
23546 7049 Acked Write 3 Done Done Sent false
23547 7050 Acked Read 2 Done Done Sent false
23548 7051 Acked Write 10 Done Done Sent false
23549 7052 Acked Write 4 Done Done Sent false
23550 7053 Acked Flush 0 Done Done Sent false
23551 7054 Acked Read 2 Done Done Sent false
23552 7055 Acked Read 10 Done Done Sent false
23553 7056 Acked Read 4 Done Done Sent false
23554 7057 Acked Read 1 Done Done Sent false
23555 7058 Acked Write 4 Done Done Sent false
23556 7059 Acked Write 3 Done Done Sent false
23557 7060 Acked Read 4 Done Done Sent false
23558 7061 Acked Write 2 Done Done Sent false
23559 7062 Acked Write 9 Done Done Sent false
23560 7063 Acked Read 8 Done Done Sent false
23561 7064 Acked Read 1 Done Done Sent false
23562 7065 Acked Read 5 Done Done Sent false
23563 7066 Acked Flush 0 Done Done Sent false
23564 7067 Acked Read 10 Done Done Sent false
23565 7068 Acked Flush 0 Done Done Sent false
23566 7069 Acked Write 9 Done Done Sent false
23567 7070 Acked Write 3 Done Done Sent false
23568 7071 Acked Read 9 Done Done Sent false
23569 7072 Acked Read 4 Done Done Sent false
23570 7073 Acked Read 9 Done Done Sent false
23571 7074 Acked Read 10 Done Done Sent false
23572 7075 Acked Read 1 Done Done Sent false
23573 7076 Acked Write 9 Done Done Sent false
23574 7077 Acked Read 10 Done Done Sent false
23575 7078 Acked Write 4 Done Done Sent false
23576 7079 Acked Write 9 Done Done Sent false
23577 7080 Acked Read 2 Done Done Sent false
23578 7081 Acked Write 4 Done Done Sent false
23579 7082 Acked Read 5 Done Done Sent false
23580 7083 Acked Flush 0 Done Done Sent false
23581 7084 Acked Read 8 Done Done Sent false
23582 7085 Acked Write 3 Done Done Sent false
23583 7086 Acked Read 2 Done Done Sent false
23584 7087 Acked Write 9 Done Done Sent false
23585 7088 Acked Write 1 Done Done Sent false
23586 7089 Acked Write 9 Done Done Sent false
23587 7090 Acked Flush 0 Done Done Sent false
23588 7091 Acked Read 2 Done Done Sent false
23589 7092 Acked Write 5 Done Done Sent false
23590 7093 Acked Write 9 Done Done Sent false
23591 7094 Acked Read 3 Done Done Sent false
23592 7095 Acked Write 7 Done Done Sent false
23593 7096 Acked Read 2 Done Done Sent false
23594 7097 Acked Write 7 Done Done Sent false
23595 7098 Acked Read 5 Done Done Sent false
23596 7099 Acked Write 2 Done Done Sent false
23597 7100 Acked Read 5 Done Done Sent false
23598 7101 Acked Write 10 Done Done Sent false
23599 7102 Acked Read 4 Done Done Sent false
23600 7103 Acked Read 6 Done Done Sent false
23601 7104 Acked Read 9 Done Done Sent false
23602 7105 Acked Write 5 Done Done Sent false
23603 7106 Acked Read 6 Done Done Sent false
23604 7107 Acked Read 6 Done Done Sent false
23605 7108 Acked Flush 0 Done Done Sent false
23606 7109 Acked Read 4 Done Done Sent false
23607 7110 Acked Read 4 Done Done Sent false
23608 7111 Acked Write 9 Done Done Sent false
23609 7112 Acked Read 6 Done Done Sent false
23610 7113 Acked Write 3 Done Done Sent false
23611 7114 Acked Read 10 Done Done Sent false
23612 7115 Acked Read 8 Done Done Sent false
23613 7116 Acked Read 8 Done Done Sent false
23614 7117 Acked Write 7 Done Done Sent false
23615 7118 Acked Write 4 Done Done Sent false
23616 7119 Acked Read 9 Done Done Sent false
23617 7120 Acked Read 9 Done Done Sent false
23618 7121 Acked Read 2 Done Done Sent false
23619 7122 Acked Write 4 Done Done Sent false
23620 7123 Acked Read 1 Done Done Sent false
23621 7124 Acked Read 6 Done Done Sent false
23622 7125 Acked Read 7 Done Done Sent false
23623 7126 Acked Read 8 Done Done Sent false
23624 7127 Acked Flush 0 Done Done Sent false
23625 7128 Acked Write 5 Done Done Sent false
23626 7129 Acked Read 6 Done Done Sent false
23627 7130 Acked Read 1 Done Done Sent false
23628 7131 Acked Write 8 Done Done Sent false
23629 7132 Acked Read 10 Done Done Sent false
23630 7133 Acked Write 9 Done Done Sent false
23631 7134 Acked Read 5 Done Done Sent false
23632 7135 Acked Write 4 Done Done Sent false
23633 7136 Acked Flush 0 Done Done Sent false
23634 7137 Acked Read 5 Done Done Sent false
23635 7138 Acked Read 7 Done Done Sent false
23636 7139 Acked Write 3 Done Done Sent false
23637 7140 Acked Write 4 Done Done Sent false
23638 7141 Acked Read 6 Done Done Sent false
23639 7142 Acked Read 9 Done Done Sent false
23640 7143 Acked Flush 0 Done Done Sent false
23641 7144 Acked Read 4 Done Done Sent false
23642 7145 Acked Read 8 Done Done Sent false
23643 7146 Acked Flush 0 Done Done Sent false
23644 7147 Acked Write 5 Done Done Sent false
23645 7148 Acked Read 3 Done Done Sent false
23646 7149 Acked Flush 0 Done Done Sent false
23647 7150 Acked Read 8 Done Done Sent false
23648 7151 Acked Read 3 Done Done Sent false
23649 7152 Acked Write 3 Done Done Sent false
23650 7153 Acked Write 2 Done Done Sent false
23651 7154 Acked Write 9 Done Done Sent false
23652 7155 Acked Write 4 Done Done Sent false
23653 7156 Acked Read 5 Done Done Sent false
23654 7157 Acked Write 4 Done Done Sent false
23655 7158 Acked Write 5 Done Done Sent false
23656 7159 Acked Write 7 Done Done Sent false
23657 7160 Acked Read 6 Done Done Sent false
23658 7161 Acked Flush 0 Done Done Sent false
23659 7162 Acked Write 10 Done Done Sent false
23660 7163 Acked Read 5 Done Done Sent false
23661 7164 Acked Read 7 Done Done Sent false
23662 7165 Acked Read 2 Done Done Sent false
23663 7166 Acked Read 5 Done Done Sent false
23664 7167 Acked Read 2 Done Done Sent false
23665 7168 Acked Read 4 Done Done Sent false
23666 7169 Acked Read 10 Done Done Sent false
23667 7170 Acked Write 7 Done Done Sent false
23668 7171 Acked Read 6 Done Done Sent false
23669 7172 Acked Write 9 Done Done Sent false
23670 7173 Acked Read 1 Done Done Sent false
23671 7174 Acked Read 5 Done Done Sent false
23672 7175 Acked Read 9 Done Done Sent false
23673 7176 Acked Write 1 Done Done Sent false
23674 7177 Acked Write 7 Done Done Sent false
23675 7178 Acked Write 2 Done Done Sent false
23676 7179 Acked Write 1 Done Done Sent false
23677 7180 Acked Read 7 Done Done Sent false
23678 7181 Acked Read 2 Done Done Sent false
23679 7182 Acked Read 9 Done Done Sent false
23680 7183 Acked Read 5 Done Done Sent false
23681 7184 Acked Write 9 Done Done Sent false
23682 7185 Acked Write 7 Done Done Sent false
23683 7186 Acked Write 1 Done Done Sent false
23684 7187 Acked Write 6 Done Done Sent false
23685 7188 Acked Read 1 Done Done Sent false
23686 7189 Acked Read 3 Done Done Sent false
23687 7190 Acked Read 9 Done Done Sent false
23688 7191 Acked Read 9 Done Done Sent false
23689 7192 Acked Write 2 Done Done Sent false
23690 7193 Acked Read 2 Done Done Sent false
23691 7194 Acked Write 10 Done Done Sent false
23692 7195 Acked Read 3 Done Done Sent false
23693 7196 Acked Read 9 Done Done Sent false
23694 7197 Acked Flush 0 Done Done Sent false
23695 7198 Acked Read 2 Done Done Sent false
23696 7199 Acked Read 2 Done Done Sent false
23697 7200 Acked Write 8 Done Done Sent false
23698 7201 Acked Read 8 Done Done Sent false
23699 7202 Acked Write 2 Done Done Sent false
23700 7203 Acked Write 8 Done Done Sent false
23701 7204 Acked Read 5 Done Done Sent false
23702 7205 Acked Write 7 Done Done Sent false
23703 7206 Acked Flush 0 Done Done Sent false
23704 7207 Acked Write 10 Done Done Sent false
23705 7208 Acked Read 10 Done Done Sent false
23706 7209 Acked Read 10 Done Done Sent false
23707 7210 Acked Write 10 Done Done Sent false
23708 7211 Acked Read 6 Done Done Sent false
23709 7212 Acked Write 5 Done Done Sent false
23710 7213 Acked Write 10 Done Done Sent false
23711 7214 Acked Write 3 Done Done Sent false
23712 7215 Acked Read 3 Done Done Sent false
23713 7216 Acked Flush 0 Done Done Sent false
23714 7217 Acked Read 3 Done Done Sent false
23715 7218 Acked Read 10 Done Done Sent false
23716 7219 Acked Read 3 Done Done Sent false
23717 7220 Acked Write 2 Done Done Sent false
23718 7221 Acked Flush 0 Done Done Sent false
23719 7222 Acked Write 10 Done Done Sent false
23720 7223 Acked Read 10 Done Done Sent false
23721 7224 Acked Read 4 Done Done Sent false
23722 7225 Acked Write 10 Done Done Sent false
23723 7226 Acked Read 3 Done Done Sent false
23724 7227 Acked Write 8 Done Done Sent false
23725 7228 Acked Read 3 Done Done Sent false
23726 7229 Acked Write 7 Done Done Sent false
23727 7230 Acked Write 2 Done Done Sent false
23728 7231 Acked Read 9 Done Done Sent false
23729 7232 Acked Write 8 Done Done Sent false
23730 7233 Acked Read 5 Done Done Sent false
23731 7234 Acked Read 10 Done Done Sent false
23732 7235 Acked Read 6 Done Done Sent false
23733 7236 Acked Write 10 Done Done Sent false
23734 7237 Acked Flush 0 Done Done Sent false
23735 7238 Acked Write 10 Done Done Sent false
23736 7239 Acked Write 9 Done Done Sent false
23737 7240 Acked Write 5 Done Done Sent false
23738 7241 Acked Read 7 Done Done Sent false
23739 7242 Acked Read 7 Done Done Sent false
23740 7243 Acked Read 4 Done Done Sent false
23741 7244 Acked Read 2 Done Done Sent false
23742 7245 Acked Read 3 Done Done Sent false
23743 7246 Acked Write 4 Done Done Sent false
23744 7247 Acked Read 7 Done Done Sent false
23745 7248 Acked Read 9 Done Done Sent false
23746 7249 Acked Read 10 Done Done Sent false
23747 7250 Acked Write 8 Done Done Sent false
23748 7251 Acked Write 7 Done Done Sent false
23749 7252 Acked Write 3 Done Done Sent false
23750 7253 Acked Write 10 Done Done Sent false
23751 7254 Acked Write 2 Done Done Sent false
23752 7255 Acked Write 5 Done Done Sent false
23753 7256 Acked Write 8 Done Done Sent false
23754 7257 Acked Read 8 Done Done Sent false
23755 7258 Acked Read 4 Done Done Sent false
23756 7259 Acked Read 10 Done Done Sent false
23757 7260 Acked Read 3 Done Done Sent false
23758 7261 Acked Read 5 Done Done Sent false
23759 7262 Acked Flush 0 Done Done Sent false
23760 7263 Acked Read 6 Done Done Sent false
23761 7264 Acked Read 2 Done Done Sent false
23762 7265 Acked Write 3 Done Done Sent false
23763 7266 Acked Read 10 Done Done Sent false
23764 7267 Acked Write 1 Done Done Sent false
23765 7268 Acked Write 9 Done Done Sent false
23766 7269 Acked Write 3 Done Done Sent false
23767 7270 Acked Write 6 Done Done Sent false
23768 7271 Acked Read 9 Done Done Sent false
23769 7272 Acked Flush 0 Done Done Sent false
23770 7273 Acked Write 10 Done Done Sent false
23771 7274 Acked Read 6 Done Done Sent false
23772 7275 Acked Flush 0 Done Done Sent false
23773 7276 Acked Read 5 Done Done Sent false
23774 7277 Acked Read 6 Done Done Sent false
23775 7278 Acked Read 9 Done Done Sent false
23776 7279 Acked Write 3 Done Done Sent false
23777 7280 Acked Read 8 Done Done Sent false
23778 7281 Acked Write 2 Done Done Sent false
23779 7282 Acked Read 5 Done Done Sent false
23780 7283 Acked Read 8 Done Done Sent false
23781 7284 Acked Read 3 Done Done Sent false
23782 7285 Acked Read 7 Done Done Sent false
23783 7286 Acked Flush 0 Done Done Sent false
23784 7287 Acked Write 9 Done Done Sent false
23785 7288 Acked Read 2 Done Done Sent false
23786 7289 Acked Read 6 Done Done Sent false
23787 7290 Acked Read 3 Done Done Sent false
23788 7291 Acked Read 8 Done Done Sent false
23789 7292 Acked Write 10 Done Done Sent false
23790 7293 Acked Read 4 Done Done Sent false
23791 7294 Acked Read 7 Done Done Sent false
23792 7295 Acked Read 5 Done Done Sent false
23793 7296 Acked Read 1 Done Done Sent false
23794 7297 Acked Read 10 Done Done Sent false
23795 7298 Acked Flush 0 Done Done Sent false
23796 7299 Acked Read 1 Done Done Sent false
23797 7300 Acked Write 6 Done Done Sent false
23798 7301 Acked Read 7 Done Done Sent false
23799 7302 Acked Write 8 Done Done Sent false
23800 7303 Acked Read 6 Done Done Sent false
23801 7304 Acked Write 3 Done Done Sent false
23802 7305 Acked Read 8 Done Done Sent false
23803 7306 Acked Write 4 Done Done Sent false
23804 7307 Acked Read 10 Done Done Sent false
23805 7308 Acked Read 8 Done Done Sent false
23806 7309 Acked Read 7 Done Done Sent false
23807 7310 Acked Read 1 Done Done Sent false
23808 7311 Acked Flush 0 Done Done Sent false
23809 7312 Acked Read 5 Done Done Sent false
23810 7313 Acked Flush 0 Done Done Sent false
23811 7314 Acked Read 2 Done Done Sent false
23812 7315 Acked Read 2 Done Done Sent false
23813 7316 Acked Read 8 Done Done Sent false
23814 7317 Acked Write 8 Done Done Sent false
23815 7318 Acked Write 9 Done Done Sent false
23816 7319 Acked Read 3 Done Done Sent false
23817 7320 Acked Read 9 Done Done Sent false
23818 7321 Acked Write 4 Done Done Sent false
23819 7322 Acked Write 6 Done Done Sent false
23820 7323 Acked Write 10 Done Done Sent false
23821 7324 Acked Read 1 Done Done Sent false
23822 7325 Acked Flush 0 Done Done Sent false
23823 7326 Acked Write 5 Done Done Sent false
23824 7327 Acked Flush 0 Done Done Sent false
23825 7328 Acked Flush 0 Done Done Sent false
23826 7329 Acked Write 1 Done Done Sent false
23827 7330 Acked Flush 0 Done Done Sent false
23828 7331 Acked Write 10 Done Done Sent false
23829 7332 Acked Write 1 Done Done Sent false
23830 7333 Acked Flush 0 Done Done Sent false
23831 7334 Acked Write 1 Done Done Sent false
23832 7335 Acked Write 4 Done Done Sent false
23833 7336 Acked Flush 0 Done Done Sent false
23834 7337 Acked Read 2 Done Done Sent false
23835 7338 Acked Read 1 Done Done Sent false
23836 7339 Acked Write 2 Done Done Sent false
23837 7340 Acked Read 7 Done Done Sent false
23838 7341 Acked Flush 0 Done Done Sent false
23839 STATES DS:0 DS:1 DS:2 TOTAL
23840 Sent 0 0 302 302
23841 Done 302 302 0 604
23842 Skipped 0 0 0 0
23843 Error 0 0 0 0
23844 Last Flush: 7341 7341 7039
23845 Downstairs last five completed: 7039 7038 7037 7036 7035
23846 Upstairs last five completed: 7341 7340 7339 7338 7337
23847 CLIENT: Up:0 ds:302 act:2
238482024-12-09T15:12:52.067ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238492024-12-09T15:12:52.067ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33910) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238502024-12-09T15:12:52.068ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238512024-12-09T15:12:52.069ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238522024-12-09T15:12:52.069ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238532024-12-09T15:12:52.069ZINFOcrucible: send last flush ID to this DS: 7039 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238542024-12-09T15:12:52.069ZINFOcrucible: Replied this last flush ID: 7039 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238552024-12-09T15:12:52.069ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238562024-12-09T15:12:52.069ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238572024-12-09T15:12:52.069ZINFOcrucible: [2] client re-new 302 jobs since flush 7039 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238582024-12-09T15:12:52.070ZINFOcrucible: [2] Replayed 302 jobs since flush: 7039 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
23859 ----------------------------------------------------------------
23860 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23861 Downstairs last five completed: 7341 7340 7339 7338 7337
23862 Upstairs last five completed: 7341 7340 7339 7338 7337
23863 CLIENT: Up:0 ds:0 act:3
23864 Replay: All jobs finished
23865 Main task has work to do, go find it
23866 got dsc Stop(1)
23867 stop 1
23868 Dec 09 15:12:52.322[8820] Got stop action so:false kr:false
23869 INFO request completed, latency_us: 336, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 105fc0a3-3331-4b19-8d24-81c4e3b1cede, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
238702024-12-09T15:12:52.322ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238712024-12-09T15:12:52.323ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238722024-12-09T15:12:52.323ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238732024-12-09T15:12:52.323ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238742024-12-09T15:12:52.323ZINFOcrucible: [1] Transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
238752024-12-09T15:12:52.323ZINFOcrucible: client task is exiting = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
23876 Dec 09 15:12:52.323 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/1, method: GET, req_id: 677d3e36-1b58-43ed-a9cc-9caee14195e9, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
23877 [8820] Exited with: ExitStatus(unix_wait_status(9))
23878 [8820][1] reports Exit
23879 Dec 09 15:12:56.325 INFO request completed, latency_us: 212, response_code: 200, uri: /state/cid/1, method: GET, req_id: 08e2b7bf-a56f-43ad-99fe-33cd4626a32c, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
23880 001/300 Write block 3517 len 12288 data: 7 8 7
23881 002/300 Flush
23882 003/300 Read block 3317 len 12288
23883 004/300 Write block 2531 len 36864 data: 5 7 7 7 7 7 7 6 6
23884 005/300 Write block 1597 len 20480 data: 5 5 4 3 4
23885 006/300 Flush
23886 007/300 Write block 4663 len 32768 data: 5 5 4 4 3 3 3 2
23887 008/300 Write block 2995 len 40960 data: 5 5 4 2 4 4 5 6 7 7
23888 009/300 Write block 1141 len 32768 data: 3 5 4 5 5 4 6 7
23889 010/300 Read block 775 len 16384
23890 011/300 Write block 2616 len 16384 data: 7 7 9 9
23891 012/300 Write block 2356 len 16384 data: 5 4 4 3
23892 013/300 Read block 2093 len 12288
23893 014/300 Read block 3558 len 28672
23894 015/300 Read block 2606 len 8192
23895 016/300 Write block 1783 len 12288 data: 4 4 5
23896 017/300 Write block 824 len 12288 data: 5 5 4
23897 018/300 Read block 267 len 24576
23898 019/300 Write block 2190 len 12288 data: 6 5 5
23899 020/300 Read block 469 len 4096
23900 021/300 Read block 2770 len 16384
23901 022/300 Read block 3017 len 4096
23902 023/300 Read block 2086 len 12288
23903 024/300 Read block 155 len 16384
23904 025/300 Read block 1830 len 32768
23905 026/300 Flush
23906 027/300 Read block 4139 len 12288
23907 028/300 Flush
23908 029/300 Read block 1725 len 12288
23909 030/300 Flush
23910 031/300 Read block 12 len 16384
23911 032/300 Read block 4504 len 16384
23912 033/300 Read block 2822 len 32768
23913 034/300 Read block 2547 len 36864
23914 035/300 Write block 2894 len 16384 data: 7 7 6 6
23915 036/300 Write block 1331 len 24576 data: 3 5 4 2 7 10
23916 037/300 Flush
23917 038/300 Read block 1472 len 36864
23918 039/300 Read block 4327 len 32768
23919 040/300 Read block 793 len 24576
23920 041/300 Write block 4735 len 24576 data: 2 2 2 2 2 2
23921 042/300 Read block 3891 len 28672
23922 043/300 Read block 1276 len 4096
23923 044/300 Flush
23924 045/300 Read block 2106 len 8192
23925 046/300 Read block 3503 len 12288
23926 047/300 Write block 222 len 36864 data: 7 8 7 6 5 8 6 5 6
23927 048/300 Read block 1130 len 40960
23928 049/300 Write block 2031 len 24576 data: 5 5 4 3 3 5
23929 050/300 Flush
23930 051/300 Flush
23931 052/300 Flush
23932 053/300 Write block 3720 len 12288 data: 5 6 5
23933 054/300 Write block 3579 len 12288 data: 4 5 6
23934 055/300 Read block 2732 len 28672
23935 056/300 Write block 3210 len 8192 data: 4 6
23936 057/300 Flush
23937 058/300 Write block 1410 len 8192 data: 7 7
23938 059/300 Write block 651 len 4096 data: 4
23939 060/300 Write block 2394 len 16384 data: 7 6 8 5
23940 061/300 Flush
23941 062/300 Write block 2803 len 16384 data: 4 4 4 4
23942 063/300 Read block 1736 len 32768
23943 064/300 Write block 1466 len 40960 data: 2 3 5 5 6 5 5 4 6 7
23944 065/300 Flush
23945 066/300 Write block 1387 len 12288 data: 5 4 4
23946 067/300 Flush
23947 068/300 Flush
23948 069/300 Write block 3093 len 40960 data: 4 4 3 3 4 5 3 5 7 5
23949 070/300 Read block 3952 len 4096
23950 071/300 Write block 974 len 12288 data: 3 3 3
23951 072/300 Read block 3355 len 4096
23952 073/300 Write block 2200 len 8192 data: 6 7
23953 074/300 Flush
23954 075/300 Write block 190 len 4096 data: 5
23955 076/300 Read block 1698 len 20480
23956 077/300 Write block 1661 len 12288 data: 6 6 5
23957 078/300 Write block 4314 len 28672 data: 5 4 4 4 4 5 5
23958 079/300 Flush
23959 080/300 Read block 3751 len 20480
23960 081/300 Write block 1508 len 28672 data: 6 6 4 6 6 7 7
23961 082/300 Flush
23962 083/300 Write block 247 len 16384 data: 5 7 6 7
23963 084/300 Write block 4382 len 12288 data: 5 6 5
23964 085/300 Read block 3651 len 12288
23965 086/300 Write block 2914 len 36864 data: 4 5 5 5 5 5 5 6 7
23966 087/300 Read block 3278 len 12288
23967 088/300 Flush
23968 089/300 Write block 2719 len 16384 data: 6 5 3 3
23969 090/300 Flush
23970 091/300 Write block 488 len 28672 data: 4 4 3 2 3 4 4
23971 092/300 Write block 850 len 32768 data: 3 4 4 5 5 5 4 4
23972 093/300 Write block 2598 len 4096 data: 3
23973 094/300 Write block 3860 len 24576 data: 3 3 2 2 3 4
23974 095/300 Write block 4394 len 20480 data: 3 4 4 4 4
23975 096/300 Write block 12 len 36864 data: 3 3 2 2 2 2 4 4 4
23976 097/300 Read block 4099 len 28672
23977 098/300 Read block 3128 len 32768
23978 099/300 Read block 4539 len 28672
23979 100/300 Write block 2186 len 36864 data: 3 5 6 6 7 6 6 6 6
23980 101/300 Write block 1099 len 24576 data: 3 2 2 4 4 4
23981 102/300 Read block 2322 len 32768
23982 103/300 Read block 4493 len 36864
23983 104/300 Read block 4335 len 40960
23984 105/300 Read block 1063 len 36864
23985 106/300 Write block 93 len 20480 data: 4 4 4 4 4
23986 107/300 Write block 2820 len 36864 data: 3 3 3 3 3 3 3 2 2
23987 108/300 Read block 818 len 16384
23988 109/300 Read block 191 len 32768
23989 110/300 Read block 3675 len 36864
23990 111/300 Read block 133 len 8192
23991 112/300 Write block 4590 len 36864 data: 4 3 4 5 5 5 4 5 5
23992 113/300 Read block 748 len 36864
23993 114/300 Read block 3926 len 12288
23994 115/300 Write block 4015 len 28672 data: 7 8 7 7 6 7 8
23995 116/300 Read block 2467 len 12288
23996 117/300 Write block 534 len 16384 data: 4 3 5 5
23997 118/300 Read block 968 len 4096
23998 119/300 Read block 2218 len 12288
23999 120/300 Write block 3057 len 32768 data: 3 2 2 2 5 6 6 4
24000 121/300 Write block 4898 len 12288 data: 4 4 3
24001 122/300 Read block 2308 len 36864
24002 123/300 Write block 1521 len 32768 data: 3 2 2 3 3 4 3 5
24003 124/300 Write block 3297 len 20480 data: 3 4 5 5 6
24004 125/300 Read block 4090 len 16384
24005 126/300 Write block 4120 len 40960 data: 4 4 4 4 5 4 5 4 4 4
24006 127/300 Read block 1773 len 24576
24007 128/300 Read block 3122 len 32768
24008 129/300 Write block 4136 len 36864 data: 3 3 4 4 3 3 3 3 3
24009 130/300 Read block 4606 len 8192
24010 131/300 Write block 1767 len 12288 data: 4 4 5
24011 132/300 Write block 1023 len 32768 data: 3 3 3 5 5 5 5 6
24012 133/300 Read block 2426 len 28672
24013 134/300 Read block 673 len 8192
24014 135/300 Write block 3683 len 40960 data: 5 5 6 4 4 5 5 5 4 3
24015 136/300 Write block 2621 len 12288 data: 8 8 10
24016 137/300 Write block 1439 len 32768 data: 4 3 5 7 8 7 7 7
24017 138/300 Write block 1516 len 4096 data: 7
24018 139/300 Write block 2706 len 40960 data: 5 4 3 4 4 8 8 9 7 8
24019 140/300 Write block 2168 len 32768 data: 5 5 4 4 6 7 7 8
24020 141/300 Read block 657 len 28672
24021 142/300 Write block 2317 len 20480 data: 5 5 5 4 4
24022 143/300 Read block 454 len 32768
24023 144/300 Read block 2161 len 16384
24024 145/300 Write block 1979 len 28672 data: 6 7 9 8 8 9 7
24025 146/300 Write block 2176 len 24576 data: 8 7 6 7 7 6
24026 147/300 Write block 2970 len 16384 data: 6 7 7 7
24027 148/300 Write block 368 len 8192 data: 7 5
24028 149/300 Read block 400 len 4096
24029 150/300 Read block 4140 len 20480
24030 151/300 Flush
24031 152/300 Write block 1367 len 20480 data: 5 6 7 10 9
24032 153/300 Read block 2326 len 8192
24033 154/300 Write block 735 len 28672 data: 4 3 4 5 6 8 10
24034 155/300 Read block 3658 len 12288
24035 156/300 Read block 187 len 4096
24036 157/300 Read block 2461 len 12288
24037 158/300 Read block 3589 len 20480
24038 159/300 Read block 4806 len 28672
24039 160/300 Flush
24040 161/300 Write block 4236 len 16384 data: 5 6 5 6
24041 162/300 Read block 3083 len 40960
24042 163/300 Read block 3975 len 24576
24043 164/300 Write block 2624 len 16384 data: 9 10 7 5
24044 165/300 Write block 322 len 20480 data: 8 6 6 4 6
24045 166/300 Write block 264 len 36864 data: 2 3 3 4 5 6 7 7 7
24046 167/300 Read block 1814 len 12288
24047 168/300 Read block 1885 len 12288
24048 169/300 Read block 1816 len 40960
24049 170/300 Write block 2082 len 28672 data: 8 6 4 4 3 2 3
24050 171/300 Write block 2357 len 8192 data: 5 5
24051 172/300 Read block 1167 len 24576
24052 173/300 Read block 1626 len 8192
24053 174/300 Read block 3113 len 24576
24054 175/300 Write block 2732 len 8192 data: 4 6
24055 176/300 Read block 2838 len 4096
24056 177/300 Flush
24057 178/300 Read block 2737 len 8192
24058 179/300 Read block 295 len 20480
24059 180/300 Read block 2454 len 32768
24060 181/300 Read block 2299 len 32768
24061 182/300 Write block 1082 len 8192 data: 4 4
24062 183/300 Write block 4654 len 12288 data: 8 8 8
24063 184/300 Read block 2935 len 40960
24064 185/300 Write block 2337 len 24576 data: 9 8 8 8 9 7
24065 186/300 Read block 2273 len 4096
24066 187/300 Read block 1543 len 12288
24067 188/300 Read block 299 len 12288
24068 189/300 Read block 2432 len 16384
24069 190/300 Write block 2940 len 16384 data: 5 6 6 5
24070 191/300 Read block 4400 len 32768
24071 192/300 Write block 2223 len 32768 data: 5 5 5 5 4 4 5 5
24072 193/300 Read block 1917 len 20480
24073 194/300 Write block 249 len 28672 data: 7 8 8 7 7 5 5
24074 195/300 Write block 1035 len 20480 data: 5 4 7 5 7
24075 196/300 Read block 2480 len 20480
24076 197/300 Read block 1962 len 24576
24077 198/300 Flush
24078 199/300 Read block 2894 len 16384
24079 200/300 Read block 3158 len 4096
24080 201/300 Write block 3518 len 40960 data: 9 8 8 7 6 6 8 10 9 6
24081 202/300 Read block 1468 len 40960
24082 203/300 Read block 663 len 4096
24083 204/300 Read block 689 len 36864
24084 205/300 Read block 3415 len 40960
24085 206/300 Read block 4612 len 20480
24086 207/300 Write block 2807 len 20480 data: 4 4 4 5 5
24087 208/300 Read block 1393 len 16384
24088 209/300 Read block 4868 len 36864
24089 210/300 Write block 704 len 40960 data: 4 4 2 2 4 4 3 3 2 3
24090 211/300 Read block 2969 len 36864
24091 212/300 Write block 3288 len 40960 data: 6 6 8 7 7 5 5 4 3 4
24092 213/300 Read block 4174 len 12288
24093 214/300 Flush
24094 215/300 Write block 4955 len 20480 data: 3 3 2 4 3
24095 216/300 Read block 1672 len 32768
24096 217/300 Write block 1866 len 24576 data: 8 7 6 8 8 8
24097 218/300 Read block 3356 len 20480
24098 219/300 Read block 1933 len 40960
24099 220/300 Read block 252 len 36864
24100 221/300 Read block 2410 len 8192
24101 222/300 Write block 539 len 4096 data: 4
24102 223/300 Read block 3279 len 32768
24103 224/300 Flush
24104 225/300 Read block 2022 len 4096
24105 226/300 Read block 991 len 32768
24106 227/300 Write block 4963 len 16384 data: 5 6 5 4
24107 228/300 Read block 3152 len 20480
24108 229/300 Write block 2025 len 12288 data: 4 6 4
24109 230/300 Write block 551 len 28672 data: 10 8 5 5 6 6 5
24110 231/300 Flush
24111 232/300 Read block 4091 len 12288
24112 233/300 Read block 3899 len 36864
24113 234/300 Read block 1246 len 36864
24114 235/300 Read block 3193 len 12288
24115 236/300 Write block 4728 len 12288 data: 4 4 4
24116 237/300 Flush
24117 238/300 Flush
24118 239/300 Flush
24119 240/300 Flush
24120 241/300 Write block 1047 len 16384 data: 4 4 2 2
24121 242/300 Read block 529 len 40960
24122 243/300 Write block 2187 len 12288 data: 6 7 7
24123 244/300 Write block 1315 len 36864 data: 6 6 5 6 4 4 5 6 5
24124 245/300 Write block 3132 len 16384 data: 5 5 4 5
24125 246/300 Write block 4092 len 32768 data: 6 4 5 7 10 9 10 9
24126 247/300 Read block 695 len 8192
24127 248/300 Write block 1482 len 32768 data: 4 3 3 3 4 4 4 4
24128 249/300 Write block 2826 len 24576 data: 4 3 3 4 4 5
24129 250/300 Read block 965 len 16384
24130 251/300 Write block 749 len 4096 data: 2
24131 252/300 Read block 2503 len 40960
24132 253/300 Read block 6 len 4096
24133 254/300 Read block 4192 len 16384
24134 255/300 Flush
24135 256/300 Read block 4799 len 28672
24136 257/300 Read block 2142 len 36864
24137 258/300 Read block 606 len 36864
24138 259/300 Read block 2109 len 32768
24139 260/300 Read block 4569 len 40960
24140 261/300 Read block 3831 len 28672
24141 262/300 Read block 4094 len 20480
24142 263/300 Write block 3808 len 28672 data: 6 4 4 3 3 3 4
24143 264/300 Read block 2190 len 8192
24144 265/300 Read block 680 len 20480
24145 266/300 Read block 3099 len 4096
24146 267/300 Write block 1614 len 36864 data: 5 5 3 4 4 4 4 4 4
24147 268/300 Write block 1599 len 20480 data: 5 4 5 5 5
24148 269/300 Read block 1092 len 36864
24149 270/300 Read block 1253 len 32768
24150 271/300 Read block 2836 len 16384
24151 272/300 Write block 491 len 12288 data: 3 4 5
24152 273/300 Read block 1606 len 40960
24153 274/300 Write block 1391 len 36864 data: 3 3 3 5 5 5 4 3 5
24154 275/300 Write block 1577 len 40960 data: 5 5 5 5 5 3 3 3 3 2
24155 276/300 Write block 657 len 40960 data: 2 2 2 3 7 7 7 8 7 7
24156 277/300 Read block 3119 len 12288
24157 278/300 Write block 1732 len 24576 data: 7 7 6 4 4 4
24158 279/300 Read block 1902 len 28672
24159 280/300 Read block 4387 len 24576
24160 281/300 Write block 4010 len 4096 data: 6
24161 282/300 Read block 417 len 36864
24162 283/300 Read block 4404 len 20480
24163 284/300 Write block 38 len 4096 data: 6
24164 285/300 Write block 2030 len 36864 data: 5 6 6 5 4 4 6 3 4
24165 286/300 Write block 1426 len 24576 data: 5 4 5 5 4 3
24166 287/300 Read block 1419 len 32768
24167 288/300 Flush
24168 289/300 Read block 4903 len 8192
24169 290/300 Flush
24170 291/300 Read block 1203 len 12288
24171 292/300 Read block 2906 len 4096
24172 293/300 Read block 1925 len 28672
24173 294/300 Read block 132 len 36864
24174 295/300 Read block 1341 len 20480
24175 296/300 Write block 3305 len 40960 data: 5 5 5 5 6 5 7 7 6 7
24176 297/300 Read block 4532 len 28672
24177 298/300 Flush
24178 299/300 Read block 4339 len 16384
24179 300/300 Read block 204 len 24576
24180 301/300 Read block 4990 len 16384
24181 Main task has work to do, go find it
24182 got dsc Start(1)
24183 start 1
24184 Dec 09 15:12:56.563 INFO request completed, latency_us: 220, response_code: 204, uri: /start/cid/1, method: POST, req_id: 7719fa88-8bdc-43f6-b1f5-3fdf30428662, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
24185 Replay: started 1, returned:Ok(())
24186 ----------------------------------------------------------------
24187 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24188 States: Active Offline Active
24189 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24190 7342 Acked Write 3 Done Sent Done false
24191 7343 Acked Flush 0 Done Sent Done false
24192 7344 Acked Read 3 Done Sent Done false
24193 7345 Acked Write 9 Done Sent Done false
24194 7346 Acked Write 5 Done Sent Done false
24195 7347 Acked Flush 0 Done Sent Done false
24196 7348 Acked Write 8 Done Sent Done false
24197 7349 Acked Write 10 Done Sent Done false
24198 7350 Acked Write 8 Done Sent Done false
24199 7351 Acked Read 4 Done Sent Done false
24200 7352 Acked Write 4 Done Sent Done false
24201 7353 Acked Write 4 Done Sent Done false
24202 7354 Acked Read 3 Done Sent Done false
24203 7355 Acked Read 7 Done Sent Done false
24204 7356 Acked Read 2 Done Sent Done false
24205 7357 Acked Write 3 Done Sent Done false
24206 7358 Acked Write 3 Done Sent Done false
24207 7359 Acked Read 6 Done Sent Done false
24208 7360 Acked Write 3 Done Sent Done false
24209 7361 Acked Read 1 Done Sent Done false
24210 7362 Acked Read 4 Done Sent Done false
24211 7363 Acked Read 1 Done Sent Done false
24212 7364 Acked Read 3 Done Sent Done false
24213 7365 Acked Read 4 Done Sent Done false
24214 7366 Acked Read 8 Done Sent Done false
24215 7367 Acked Flush 0 Done Sent Done false
24216 7368 Acked Read 3 Done Sent Done false
24217 7369 Acked Flush 0 Done Sent Done false
24218 7370 Acked Read 3 Done Sent Done false
24219 7371 Acked Flush 0 Done Sent Done false
24220 7372 Acked Read 4 Done Sent Done false
24221 7373 Acked Read 4 Done Sent Done false
24222 7374 Acked Read 8 Done Sent Done false
24223 7375 Acked Read 9 Done Sent Done false
24224 7376 Acked Write 4 Done Sent Done false
24225 7377 Acked Write 6 Done Sent Done false
24226 7378 Acked Flush 0 Done Sent Done false
24227 7379 Acked Read 9 Done Sent Done false
24228 7380 Acked Read 8 Done Sent Done false
24229 7381 Acked Read 6 Done Sent Done false
24230 7382 Acked Write 6 Done Sent Done false
24231 7383 Acked Read 7 Done Sent Done false
24232 7384 Acked Read 1 Done Sent Done false
24233 7385 Acked Flush 0 Done Sent Done false
24234 7386 Acked Read 2 Done Sent Done false
24235 7387 Acked Read 3 Done Sent Done false
24236 7388 Acked Write 9 Done Sent Done false
24237 7389 Acked Read 10 Done Sent Done false
24238 7390 Acked Write 6 Done Sent Done false
24239 7391 Acked Flush 0 Done Sent Done false
24240 7392 Acked Flush 0 Done Sent Done false
24241 7393 Acked Flush 0 Done Sent Done false
24242 7394 Acked Write 3 Done Sent Done false
24243 7395 Acked Write 3 Done Sent Done false
24244 7396 Acked Read 7 Done Sent Done false
24245 7397 Acked Write 2 Done Sent Done false
24246 7398 Acked Flush 0 Done Sent Done false
24247 7399 Acked Write 2 Done Sent Done false
24248 7400 Acked Write 1 Done Sent Done false
24249 7401 Acked Write 4 Done Sent Done false
24250 7402 Acked Flush 0 Done Sent Done false
24251 7403 Acked Write 4 Done Sent Done false
24252 7404 Acked Read 8 Done Sent Done false
24253 7405 Acked Write 10 Done Sent Done false
24254 7406 Acked Flush 0 Done Sent Done false
24255 7407 Acked Write 3 Done Sent Done false
24256 7408 Acked Flush 0 Done Sent Done false
24257 7409 Acked Flush 0 Done Sent Done false
24258 7410 Acked Write 10 Done Sent Done false
24259 7411 Acked Read 1 Done Sent Done false
24260 7412 Acked Write 3 Done Sent Done false
24261 7413 Acked Read 1 Done Sent Done false
24262 7414 Acked Write 2 Done Sent Done false
24263 7415 Acked Flush 0 Done Sent Done false
24264 7416 Acked Write 1 Done Sent Done false
24265 7417 Acked Read 5 Done Sent Done false
24266 7418 Acked Write 3 Done Sent Done false
24267 7419 Acked Write 7 Done Sent Done false
24268 7420 Acked Flush 0 Done Sent Done false
24269 7421 Acked Read 5 Done Sent Done false
24270 7422 Acked Write 7 Done Sent Done false
24271 7423 Acked Flush 0 Done Sent Done false
24272 7424 Acked Write 4 Done Sent Done false
24273 7425 Acked Write 3 Done Sent Done false
24274 7426 Acked Read 3 Done Sent Done false
24275 7427 Acked Write 9 Done Sent Done false
24276 7428 Acked Read 3 Done Sent Done false
24277 7429 Acked Flush 0 Done Sent Done false
24278 7430 Acked Write 4 Done Sent Done false
24279 7431 Acked Flush 0 Done Sent Done false
24280 7432 Acked Write 7 Done Sent Done false
24281 7433 Acked Write 8 Done Sent Done false
24282 7434 Acked Write 1 Done Sent Done false
24283 7435 Acked Write 6 Done Sent Done false
24284 7436 Acked Write 5 Done Sent Done false
24285 7437 Acked Write 9 Done Sent Done false
24286 7438 Acked Read 7 Done Sent Done false
24287 7439 Acked Read 8 Done Sent Done false
24288 7440 Acked Read 7 Done Sent Done false
24289 7441 Acked Write 9 Done Sent Done false
24290 7442 Acked Write 6 Done Sent Done false
24291 7443 Acked Read 8 Done Sent Done false
24292 7444 Acked Read 9 Done Sent Done false
24293 7445 Acked Read 10 Done Sent Done false
24294 7446 Acked Read 9 Done Sent Done false
24295 7447 Acked Write 5 Done Sent Done false
24296 7448 Acked Write 9 Done Sent Done false
24297 7449 Acked Read 4 Done Sent Done false
24298 7450 Acked Read 8 Done Sent Done false
24299 7451 Acked Read 9 Done Sent Done false
24300 7452 Acked Read 2 Done Sent Done false
24301 7453 Acked Write 9 Done Sent Done false
24302 7454 Acked Read 9 Done Sent Done false
24303 7455 Acked Read 3 Done Sent Done false
24304 7456 Acked Write 7 Done Sent Done false
24305 7457 Acked Read 3 Done Sent Done false
24306 7458 Acked Write 4 Done Sent Done false
24307 7459 Acked Read 1 Done Sent Done false
24308 7460 Acked Read 3 Done Sent Done false
24309 7461 Acked Write 8 Done Sent Done false
24310 7462 Acked Write 3 Done Sent Done false
24311 7463 Acked Read 9 Done Sent Done false
24312 7464 Acked Write 8 Done Sent Done false
24313 7465 Acked Write 5 Done Sent Done false
24314 7466 Acked Read 4 Done Sent Done false
24315 7467 Acked Write 10 Done Sent Done false
24316 7468 Acked Read 6 Done Sent Done false
24317 7469 Acked Read 8 Done Sent Done false
24318 7470 Acked Write 9 Done Sent Done false
24319 7471 Acked Read 2 Done Sent Done false
24320 7472 Acked Write 3 Done Sent Done false
24321 7473 Acked Write 8 Done Sent Done false
24322 7474 Acked Read 7 Done Sent Done false
24323 7475 Acked Read 2 Done Sent Done false
24324 7476 Acked Write 10 Done Sent Done false
24325 7477 Acked Write 3 Done Sent Done false
24326 7478 Acked Write 8 Done Sent Done false
24327 7479 Acked Write 1 Done Sent Done false
24328 7480 Acked Write 10 Done Sent Done false
24329 7481 Acked Write 8 Done Sent Done false
24330 7482 Acked Read 7 Done Sent Done false
24331 7483 Acked Write 5 Done Sent Done false
24332 7484 Acked Read 8 Done Sent Done false
24333 7485 Acked Read 4 Done Sent Done false
24334 7486 Acked Write 7 Done Sent Done false
24335 7487 Acked Write 6 Done Sent Done false
24336 7488 Acked Write 4 Done Sent Done false
24337 7489 Acked Write 2 Done Sent Done false
24338 7490 Acked Read 1 Done Sent Done false
24339 7491 Acked Read 5 Done Sent Done false
24340 7492 Acked Flush 0 Done Sent Done false
24341 7493 Acked Write 5 Done Sent Done false
24342 7494 Acked Read 2 Done Sent Done false
24343 7495 Acked Write 7 Done Sent Done false
24344 7496 Acked Read 3 Done Sent Done false
24345 7497 Acked Read 1 Done Sent Done false
24346 7498 Acked Read 3 Done Sent Done false
24347 7499 Acked Read 5 Done Sent Done false
24348 7500 Acked Read 7 Done Sent Done false
24349 7501 Acked Flush 0 Done Sent Done false
24350 7502 Acked Write 4 Done Sent Done false
24351 7503 Acked Read 10 Done Sent Done false
24352 7504 Acked Read 6 Done Sent Done false
24353 7505 Acked Write 4 Done Sent Done false
24354 7506 Acked Write 5 Done Sent Done false
24355 7507 Acked Write 9 Done Sent Done false
24356 7508 Acked Read 3 Done Sent Done false
24357 7509 Acked Read 3 Done Sent Done false
24358 7510 Acked Read 10 Done Sent Done false
24359 7511 Acked Write 7 Done Sent Done false
24360 7512 Acked Write 2 Done Sent Done false
24361 7513 Acked Read 6 Done Sent Done false
24362 7514 Acked Read 2 Done Sent Done false
24363 7515 Acked Read 6 Done Sent Done false
24364 7516 Acked Write 2 Done Sent Done false
24365 7517 Acked Read 1 Done Sent Done false
24366 7518 Acked Flush 0 Done Sent Done false
24367 7519 Acked Read 2 Done Sent Done false
24368 7520 Acked Read 5 Done Sent Done false
24369 7521 Acked Read 8 Done Sent Done false
24370 7522 Acked Read 8 Done Sent Done false
24371 7523 Acked Write 2 Done Sent Done false
24372 7524 Acked Write 3 Done Sent Done false
24373 7525 Acked Read 10 Done Sent Done false
24374 7526 Acked Write 6 Done Sent Done false
24375 7527 Acked Read 1 Done Sent Done false
24376 7528 Acked Read 3 Done Sent Done false
24377 7529 Acked Read 3 Done Sent Done false
24378 7530 Acked Read 4 Done Sent Done false
24379 7531 Acked Write 4 Done Sent Done false
24380 7532 Acked Read 8 Done Sent Done false
24381 7533 Acked Write 8 Done Sent Done false
24382 7534 Acked Read 5 Done Sent Done false
24383 7535 Acked Write 7 Done Sent Done false
24384 7536 Acked Write 5 Done Sent Done false
24385 7537 Acked Read 5 Done Sent Done false
24386 7538 Acked Read 6 Done Sent Done false
24387 7539 Acked Flush 0 Done Sent Done false
24388 7540 Acked Read 4 Done Sent Done false
24389 7541 Acked Read 1 Done Sent Done false
24390 7542 Acked Write 10 Done Sent Done false
24391 7543 Acked Read 10 Done Sent Done false
24392 7544 Acked Read 1 Done Sent Done false
24393 7545 Acked Read 9 Done Sent Done false
24394 7546 Acked Read 10 Done Sent Done false
24395 7547 Acked Read 5 Done Sent Done false
24396 7548 Acked Write 5 Done Sent Done false
24397 7549 Acked Read 4 Done Sent Done false
24398 7550 Acked Read 9 Done Sent Done false
24399 7551 Acked Write 10 Done Sent Done false
24400 7552 Acked Read 9 Done Sent Done false
24401 7553 Acked Write 10 Done Sent Done false
24402 7554 Acked Read 3 Done Sent Done false
24403 7555 Acked Flush 0 Done Sent Done false
24404 7556 Acked Write 5 Done Sent Done false
24405 7557 Acked Read 8 Done Sent Done false
24406 7558 Acked Write 6 Done Sent Done false
24407 7559 Acked Read 5 Done Sent Done false
24408 7560 Acked Read 10 Done Sent Done false
24409 7561 Acked Read 9 Done Sent Done false
24410 7562 Acked Read 2 Done Sent Done false
24411 7563 Acked Write 1 Done Sent Done false
24412 7564 Acked Read 8 Done Sent Done false
24413 7565 Acked Flush 0 Done Sent Done false
24414 7566 Acked Read 1 Done Sent Done false
24415 7567 Acked Read 8 Done Sent Done false
24416 7568 Acked Write 4 Done Sent Done false
24417 7569 Acked Read 5 Done Sent Done false
24418 7570 Acked Write 3 Done Sent Done false
24419 7571 Acked Write 7 Done Sent Done false
24420 7572 Acked Flush 0 Done Sent Done false
24421 7573 Acked Read 3 Done Sent Done false
24422 7574 Acked Read 9 Done Sent Done false
24423 7575 Acked Read 9 Done Sent Done false
24424 7576 Acked Read 3 Done Sent Done false
24425 7577 Acked Write 3 Done Sent Done false
24426 7578 Acked Flush 0 Done Sent Done false
24427 7579 Acked Flush 0 Done Sent Done false
24428 7580 Acked Flush 0 Done Sent Done false
24429 7581 Acked Flush 0 Done Sent Done false
24430 7582 Acked Write 4 Done Sent Done false
24431 7583 Acked Read 10 Done Sent Done false
24432 7584 Acked Write 3 Done Sent Done false
24433 7585 Acked Write 9 Done Sent Done false
24434 7586 Acked Write 4 Done Sent Done false
24435 7587 Acked Write 8 Done Sent Done false
24436 7588 Acked Read 2 Done Sent Done false
24437 7589 Acked Write 8 Done Sent Done false
24438 7590 Acked Write 6 Done Sent Done false
24439 7591 Acked Read 4 Done Sent Done false
24440 7592 Acked Write 1 Done Sent Done false
24441 7593 Acked Read 10 Done Sent Done false
24442 7594 Acked Read 1 Done Sent Done false
24443 7595 Acked Read 4 Done Sent Done false
24444 7596 Acked Flush 0 Done Sent Done false
24445 7597 Acked Read 7 Done Sent Done false
24446 7598 Acked Read 9 Done Sent Done false
24447 7599 Acked Read 9 Done Sent Done false
24448 7600 Acked Read 8 Done Sent Done false
24449 7601 Acked Read 10 Done Sent Done false
24450 7602 Acked Read 7 Done Sent Done false
24451 7603 Acked Read 5 Done Sent Done false
24452 7604 Acked Write 7 Done Sent Done false
24453 7605 Acked Read 2 Done Sent Done false
24454 7606 Acked Read 5 Done Sent Done false
24455 7607 Acked Read 1 Done Sent Done false
24456 7608 Acked Write 9 Done Sent Done false
24457 7609 Acked Write 5 Done Sent Done false
24458 7610 Acked Read 9 Done Sent Done false
24459 7611 Acked Read 8 Done Sent Done false
24460 7612 Acked Read 4 Done Sent Done false
24461 7613 Acked Write 3 Done Sent Done false
24462 7614 Acked Read 10 Done Sent Done false
24463 7615 Acked Write 9 Done Sent Done false
24464 7616 Acked Write 10 Done Sent Done false
24465 7617 Acked Write 10 Done Sent Done false
24466 7618 Acked Read 3 Done Sent Done false
24467 7619 Acked Write 6 Done Sent Done false
24468 7620 Acked Read 7 Done Sent Done false
24469 7621 Acked Read 6 Done Sent Done false
24470 7622 Acked Write 1 Done Sent Done false
24471 7623 Acked Read 9 Done Sent Done false
24472 7624 Acked Read 5 Done Sent Done false
24473 7625 Acked Write 1 Done Sent Done false
24474 7626 Acked Write 9 Done Sent Done false
24475 7627 Acked Write 6 Done Sent Done false
24476 7628 Acked Read 8 Done Sent Done false
24477 7629 Acked Flush 0 Done Sent Done false
24478 7630 Acked Read 2 Done Sent Done false
24479 7631 Acked Flush 0 Done Sent Done false
24480 7632 Acked Read 3 Done Sent Done false
24481 7633 Acked Read 1 Done Sent Done false
24482 7634 Acked Read 7 Done Sent Done false
24483 7635 Acked Read 9 Done Sent Done false
24484 7636 Acked Read 5 Done Sent Done false
24485 7637 Acked Write 10 Done Sent Done false
24486 7638 Acked Read 7 Done Sent Done false
24487 7639 Acked Flush 0 Done Sent Done false
24488 7640 Acked Read 4 Done Sent Done false
24489 7641 Acked Read 6 Done Sent Done false
24490 7642 Acked Read 4 Done Sent Done false
24491 STATES DS:0 DS:1 DS:2 TOTAL
24492 Sent 0 301 0 301
24493 Done 301 0 301 602
24494 Skipped 0 0 0 0
24495 Error 0 0 0 0
24496 Last Flush: 7639 7341 7639
24497 Downstairs last five completed: 7341 7340 7339 7338 7337
24498 Upstairs last five completed: 7642 7641 7640 7639 7638
24499 CLIENT: Up:0 ds:301 act:2
24500 [8820] got start action
24501 [8820] I am going to restart
24502 Starting downstairs at port 8820
24503 Make output file at "/tmp/dsc/downstairs-8820.txt"
24504 [8820][1] reports Starting
24505 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1223)
24506 [8820][1] reports Running
24507 ----------------------------------------------------------------
24508 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24509 States: Active Offline Active
24510 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24511 7342 Acked Write 3 Done Sent Done false
24512 7343 Acked Flush 0 Done Sent Done false
24513 7344 Acked Read 3 Done Sent Done false
24514 7345 Acked Write 9 Done Sent Done false
24515 7346 Acked Write 5 Done Sent Done false
24516 7347 Acked Flush 0 Done Sent Done false
24517 7348 Acked Write 8 Done Sent Done false
24518 7349 Acked Write 10 Done Sent Done false
24519 7350 Acked Write 8 Done Sent Done false
24520 7351 Acked Read 4 Done Sent Done false
24521 7352 Acked Write 4 Done Sent Done false
24522 7353 Acked Write 4 Done Sent Done false
24523 7354 Acked Read 3 Done Sent Done false
24524 7355 Acked Read 7 Done Sent Done false
24525 7356 Acked Read 2 Done Sent Done false
24526 7357 Acked Write 3 Done Sent Done false
24527 7358 Acked Write 3 Done Sent Done false
24528 7359 Acked Read 6 Done Sent Done false
24529 7360 Acked Write 3 Done Sent Done false
24530 7361 Acked Read 1 Done Sent Done false
24531 7362 Acked Read 4 Done Sent Done false
24532 7363 Acked Read 1 Done Sent Done false
24533 7364 Acked Read 3 Done Sent Done false
24534 7365 Acked Read 4 Done Sent Done false
24535 7366 Acked Read 8 Done Sent Done false
24536 7367 Acked Flush 0 Done Sent Done false
24537 7368 Acked Read 3 Done Sent Done false
24538 7369 Acked Flush 0 Done Sent Done false
24539 7370 Acked Read 3 Done Sent Done false
24540 7371 Acked Flush 0 Done Sent Done false
24541 7372 Acked Read 4 Done Sent Done false
24542 7373 Acked Read 4 Done Sent Done false
24543 7374 Acked Read 8 Done Sent Done false
24544 7375 Acked Read 9 Done Sent Done false
24545 7376 Acked Write 4 Done Sent Done false
24546 7377 Acked Write 6 Done Sent Done false
24547 7378 Acked Flush 0 Done Sent Done false
24548 7379 Acked Read 9 Done Sent Done false
24549 7380 Acked Read 8 Done Sent Done false
24550 7381 Acked Read 6 Done Sent Done false
24551 7382 Acked Write 6 Done Sent Done false
24552 7383 Acked Read 7 Done Sent Done false
24553 7384 Acked Read 1 Done Sent Done false
24554 7385 Acked Flush 0 Done Sent Done false
24555 7386 Acked Read 2 Done Sent Done false
24556 7387 Acked Read 3 Done Sent Done false
24557 7388 Acked Write 9 Done Sent Done false
24558 7389 Acked Read 10 Done Sent Done false
24559 7390 Acked Write 6 Done Sent Done false
24560 7391 Acked Flush 0 Done Sent Done false
24561 7392 Acked Flush 0 Done Sent Done false
24562 7393 Acked Flush 0 Done Sent Done false
24563 7394 Acked Write 3 Done Sent Done false
24564 7395 Acked Write 3 Done Sent Done false
24565 7396 Acked Read 7 Done Sent Done false
24566 7397 Acked Write 2 Done Sent Done false
24567 7398 Acked Flush 0 Done Sent Done false
24568 7399 Acked Write 2 Done Sent Done false
24569 7400 Acked Write 1 Done Sent Done false
24570 7401 Acked Write 4 Done Sent Done false
24571 7402 Acked Flush 0 Done Sent Done false
24572 7403 Acked Write 4 Done Sent Done false
24573 7404 Acked Read 8 Done Sent Done false
24574 7405 Acked Write 10 Done Sent Done false
24575 7406 Acked Flush 0 Done Sent Done false
24576 7407 Acked Write 3 Done Sent Done false
24577 7408 Acked Flush 0 Done Sent Done false
24578 7409 Acked Flush 0 Done Sent Done false
24579 7410 Acked Write 10 Done Sent Done false
24580 7411 Acked Read 1 Done Sent Done false
24581 7412 Acked Write 3 Done Sent Done false
24582 7413 Acked Read 1 Done Sent Done false
24583 7414 Acked Write 2 Done Sent Done false
24584 7415 Acked Flush 0 Done Sent Done false
24585 7416 Acked Write 1 Done Sent Done false
24586 7417 Acked Read 5 Done Sent Done false
24587 7418 Acked Write 3 Done Sent Done false
24588 7419 Acked Write 7 Done Sent Done false
24589 7420 Acked Flush 0 Done Sent Done false
24590 7421 Acked Read 5 Done Sent Done false
24591 7422 Acked Write 7 Done Sent Done false
24592 7423 Acked Flush 0 Done Sent Done false
24593 7424 Acked Write 4 Done Sent Done false
24594 7425 Acked Write 3 Done Sent Done false
24595 7426 Acked Read 3 Done Sent Done false
24596 7427 Acked Write 9 Done Sent Done false
24597 7428 Acked Read 3 Done Sent Done false
24598 7429 Acked Flush 0 Done Sent Done false
24599 7430 Acked Write 4 Done Sent Done false
24600 7431 Acked Flush 0 Done Sent Done false
24601 7432 Acked Write 7 Done Sent Done false
24602 7433 Acked Write 8 Done Sent Done false
24603 7434 Acked Write 1 Done Sent Done false
24604 7435 Acked Write 6 Done Sent Done false
24605 7436 Acked Write 5 Done Sent Done false
24606 7437 Acked Write 9 Done Sent Done false
24607 7438 Acked Read 7 Done Sent Done false
24608 7439 Acked Read 8 Done Sent Done false
24609 7440 Acked Read 7 Done Sent Done false
24610 7441 Acked Write 9 Done Sent Done false
24611 7442 Acked Write 6 Done Sent Done false
24612 7443 Acked Read 8 Done Sent Done false
24613 7444 Acked Read 9 Done Sent Done false
24614 7445 Acked Read 10 Done Sent Done false
24615 7446 Acked Read 9 Done Sent Done false
24616 7447 Acked Write 5 Done Sent Done false
24617 7448 Acked Write 9 Done Sent Done false
24618 7449 Acked Read 4 Done Sent Done false
24619 7450 Acked Read 8 Done Sent Done false
24620 7451 Acked Read 9 Done Sent Done false
24621 7452 Acked Read 2 Done Sent Done false
24622 7453 Acked Write 9 Done Sent Done false
24623 7454 Acked Read 9 Done Sent Done false
24624 7455 Acked Read 3 Done Sent Done false
24625 7456 Acked Write 7 Done Sent Done false
24626 7457 Acked Read 3 Done Sent Done false
24627 7458 Acked Write 4 Done Sent Done false
24628 7459 Acked Read 1 Done Sent Done false
24629 7460 Acked Read 3 Done Sent Done false
24630 7461 Acked Write 8 Done Sent Done false
24631 7462 Acked Write 3 Done Sent Done false
24632 7463 Acked Read 9 Done Sent Done false
24633 7464 Acked Write 8 Done Sent Done false
24634 7465 Acked Write 5 Done Sent Done false
24635 7466 Acked Read 4 Done Sent Done false
24636 7467 Acked Write 10 Done Sent Done false
24637 7468 Acked Read 6 Done Sent Done false
24638 7469 Acked Read 8 Done Sent Done false
24639 7470 Acked Write 9 Done Sent Done false
24640 7471 Acked Read 2 Done Sent Done false
24641 7472 Acked Write 3 Done Sent Done false
24642 7473 Acked Write 8 Done Sent Done false
24643 7474 Acked Read 7 Done Sent Done false
24644 7475 Acked Read 2 Done Sent Done false
24645 7476 Acked Write 10 Done Sent Done false
24646 7477 Acked Write 3 Done Sent Done false
24647 7478 Acked Write 8 Done Sent Done false
24648 7479 Acked Write 1 Done Sent Done false
24649 7480 Acked Write 10 Done Sent Done false
24650 7481 Acked Write 8 Done Sent Done false
24651 7482 Acked Read 7 Done Sent Done false
24652 7483 Acked Write 5 Done Sent Done false
24653 7484 Acked Read 8 Done Sent Done false
24654 7485 Acked Read 4 Done Sent Done false
24655 7486 Acked Write 7 Done Sent Done false
24656 7487 Acked Write 6 Done Sent Done false
24657 7488 Acked Write 4 Done Sent Done false
24658 7489 Acked Write 2 Done Sent Done false
24659 7490 Acked Read 1 Done Sent Done false
24660 7491 Acked Read 5 Done Sent Done false
24661 7492 Acked Flush 0 Done Sent Done false
24662 7493 Acked Write 5 Done Sent Done false
24663 7494 Acked Read 2 Done Sent Done false
24664 7495 Acked Write 7 Done Sent Done false
24665 7496 Acked Read 3 Done Sent Done false
24666 7497 Acked Read 1 Done Sent Done false
24667 7498 Acked Read 3 Done Sent Done false
24668 7499 Acked Read 5 Done Sent Done false
24669 7500 Acked Read 7 Done Sent Done false
24670 7501 Acked Flush 0 Done Sent Done false
24671 7502 Acked Write 4 Done Sent Done false
24672 7503 Acked Read 10 Done Sent Done false
24673 7504 Acked Read 6 Done Sent Done false
24674 7505 Acked Write 4 Done Sent Done false
24675 7506 Acked Write 5 Done Sent Done false
24676 7507 Acked Write 9 Done Sent Done false
24677 7508 Acked Read 3 Done Sent Done false
24678 7509 Acked Read 3 Done Sent Done false
24679 7510 Acked Read 10 Done Sent Done false
24680 7511 Acked Write 7 Done Sent Done false
24681 7512 Acked Write 2 Done Sent Done false
24682 7513 Acked Read 6 Done Sent Done false
24683 7514 Acked Read 2 Done Sent Done false
24684 7515 Acked Read 6 Done Sent Done false
24685 7516 Acked Write 2 Done Sent Done false
24686 7517 Acked Read 1 Done Sent Done false
24687 7518 Acked Flush 0 Done Sent Done false
24688 7519 Acked Read 2 Done Sent Done false
24689 7520 Acked Read 5 Done Sent Done false
24690 7521 Acked Read 8 Done Sent Done false
24691 7522 Acked Read 8 Done Sent Done false
24692 7523 Acked Write 2 Done Sent Done false
24693 7524 Acked Write 3 Done Sent Done false
24694 7525 Acked Read 10 Done Sent Done false
24695 7526 Acked Write 6 Done Sent Done false
24696 7527 Acked Read 1 Done Sent Done false
24697 7528 Acked Read 3 Done Sent Done false
24698 7529 Acked Read 3 Done Sent Done false
24699 7530 Acked Read 4 Done Sent Done false
24700 7531 Acked Write 4 Done Sent Done false
24701 7532 Acked Read 8 Done Sent Done false
24702 7533 Acked Write 8 Done Sent Done false
24703 7534 Acked Read 5 Done Sent Done false
24704 7535 Acked Write 7 Done Sent Done false
24705 7536 Acked Write 5 Done Sent Done false
24706 7537 Acked Read 5 Done Sent Done false
24707 7538 Acked Read 6 Done Sent Done false
24708 7539 Acked Flush 0 Done Sent Done false
24709 7540 Acked Read 4 Done Sent Done false
24710 7541 Acked Read 1 Done Sent Done false
24711 7542 Acked Write 10 Done Sent Done false
24712 7543 Acked Read 10 Done Sent Done false
24713 7544 Acked Read 1 Done Sent Done false
24714 7545 Acked Read 9 Done Sent Done false
24715 7546 Acked Read 10 Done Sent Done false
24716 7547 Acked Read 5 Done Sent Done false
24717 7548 Acked Write 5 Done Sent Done false
24718 7549 Acked Read 4 Done Sent Done false
24719 7550 Acked Read 9 Done Sent Done false
24720 7551 Acked Write 10 Done Sent Done false
24721 7552 Acked Read 9 Done Sent Done false
24722 7553 Acked Write 10 Done Sent Done false
24723 7554 Acked Read 3 Done Sent Done false
24724 7555 Acked Flush 0 Done Sent Done false
24725 7556 Acked Write 5 Done Sent Done false
24726 7557 Acked Read 8 Done Sent Done false
24727 7558 Acked Write 6 Done Sent Done false
24728 7559 Acked Read 5 Done Sent Done false
24729 7560 Acked Read 10 Done Sent Done false
24730 7561 Acked Read 9 Done Sent Done false
24731 7562 Acked Read 2 Done Sent Done false
24732 7563 Acked Write 1 Done Sent Done false
24733 7564 Acked Read 8 Done Sent Done false
24734 7565 Acked Flush 0 Done Sent Done false
24735 7566 Acked Read 1 Done Sent Done false
24736 7567 Acked Read 8 Done Sent Done false
24737 7568 Acked Write 4 Done Sent Done false
24738 7569 Acked Read 5 Done Sent Done false
24739 7570 Acked Write 3 Done Sent Done false
24740 7571 Acked Write 7 Done Sent Done false
24741 7572 Acked Flush 0 Done Sent Done false
24742 7573 Acked Read 3 Done Sent Done false
24743 7574 Acked Read 9 Done Sent Done false
24744 7575 Acked Read 9 Done Sent Done false
24745 7576 Acked Read 3 Done Sent Done false
24746 7577 Acked Write 3 Done Sent Done false
24747 7578 Acked Flush 0 Done Sent Done false
24748 7579 Acked Flush 0 Done Sent Done false
24749 7580 Acked Flush 0 Done Sent Done false
24750 7581 Acked Flush 0 Done Sent Done false
24751 7582 Acked Write 4 Done Sent Done false
24752 7583 Acked Read 10 Done Sent Done false
24753 7584 Acked Write 3 Done Sent Done false
24754 7585 Acked Write 9 Done Sent Done false
24755 7586 Acked Write 4 Done Sent Done false
24756 7587 Acked Write 8 Done Sent Done false
24757 7588 Acked Read 2 Done Sent Done false
24758 7589 Acked Write 8 Done Sent Done false
24759 7590 Acked Write 6 Done Sent Done false
24760 7591 Acked Read 4 Done Sent Done false
24761 7592 Acked Write 1 Done Sent Done false
24762 7593 Acked Read 10 Done Sent Done false
24763 7594 Acked Read 1 Done Sent Done false
24764 7595 Acked Read 4 Done Sent Done false
24765 7596 Acked Flush 0 Done Sent Done false
24766 7597 Acked Read 7 Done Sent Done false
24767 7598 Acked Read 9 Done Sent Done false
24768 7599 Acked Read 9 Done Sent Done false
24769 7600 Acked Read 8 Done Sent Done false
24770 7601 Acked Read 10 Done Sent Done false
24771 7602 Acked Read 7 Done Sent Done false
24772 7603 Acked Read 5 Done Sent Done false
24773 7604 Acked Write 7 Done Sent Done false
24774 7605 Acked Read 2 Done Sent Done false
24775 7606 Acked Read 5 Done Sent Done false
24776 7607 Acked Read 1 Done Sent Done false
24777 7608 Acked Write 9 Done Sent Done false
24778 7609 Acked Write 5 Done Sent Done false
24779 7610 Acked Read 9 Done Sent Done false
24780 7611 Acked Read 8 Done Sent Done false
24781 7612 Acked Read 4 Done Sent Done false
24782 7613 Acked Write 3 Done Sent Done false
24783 7614 Acked Read 10 Done Sent Done false
24784 7615 Acked Write 9 Done Sent Done false
24785 7616 Acked Write 10 Done Sent Done false
24786 7617 Acked Write 10 Done Sent Done false
24787 7618 Acked Read 3 Done Sent Done false
24788 7619 Acked Write 6 Done Sent Done false
24789 7620 Acked Read 7 Done Sent Done false
24790 7621 Acked Read 6 Done Sent Done false
24791 7622 Acked Write 1 Done Sent Done false
24792 7623 Acked Read 9 Done Sent Done false
24793 7624 Acked Read 5 Done Sent Done false
24794 7625 Acked Write 1 Done Sent Done false
24795 7626 Acked Write 9 Done Sent Done false
24796 7627 Acked Write 6 Done Sent Done false
24797 7628 Acked Read 8 Done Sent Done false
24798 7629 Acked Flush 0 Done Sent Done false
24799 7630 Acked Read 2 Done Sent Done false
24800 7631 Acked Flush 0 Done Sent Done false
24801 7632 Acked Read 3 Done Sent Done false
24802 7633 Acked Read 1 Done Sent Done false
24803 7634 Acked Read 7 Done Sent Done false
24804 7635 Acked Read 9 Done Sent Done false
24805 7636 Acked Read 5 Done Sent Done false
24806 7637 Acked Write 10 Done Sent Done false
24807 7638 Acked Read 7 Done Sent Done false
24808 7639 Acked Flush 0 Done Sent Done false
24809 7640 Acked Read 4 Done Sent Done false
24810 7641 Acked Read 6 Done Sent Done false
24811 7642 Acked Read 4 Done Sent Done false
24812 7643 Acked Flush 0 Done Sent Done false
24813 STATES DS:0 DS:1 DS:2 TOTAL
24814 Sent 0 302 0 302
24815 Done 302 0 302 604
24816 Skipped 0 0 0 0
24817 Error 0 0 0 0
24818 Last Flush: 7643 7341 7643
24819 Downstairs last five completed: 7341 7340 7339 7338 7337
24820 Upstairs last five completed: 7643 7642 7641 7640 7639
24821 CLIENT: Up:0 ds:302 act:2
248222024-12-09T15:13:02.325ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
248232024-12-09T15:13:02.325ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48102) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
248242024-12-09T15:13:02.326ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
248252024-12-09T15:13:02.326ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
248262024-12-09T15:13:02.326ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
248272024-12-09T15:13:02.326ZINFOcrucible: send last flush ID to this DS: 7341 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
248282024-12-09T15:13:02.327ZINFOcrucible: Replied this last flush ID: 7341 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
248292024-12-09T15:13:02.327ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
248302024-12-09T15:13:02.327ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
248312024-12-09T15:13:02.327ZINFOcrucible: [1] client re-new 302 jobs since flush 7341 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
248322024-12-09T15:13:02.327ZINFOcrucible: [1] Replayed 302 jobs since flush: 7341 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
24833 ----------------------------------------------------------------
24834 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24835 Downstairs last five completed: 7643 7642 7641 7640 7639
24836 Upstairs last five completed: 7643 7642 7641 7640 7639
24837 CLIENT: Up:0 ds:0 act:3
24838 Replay: All jobs finished
24839 Main task has work to do, go find it
24840 got dsc Stop(0)
24841 stop 0
24842 Dec 09 15:13:04.568[8810] Got stop action so:false kr:false
24843 INFO request completed, latency_us: 332, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 73e67ab4-18ac-4d3c-94c3-319c1eb55752, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
248442024-12-09T15:13:04.569ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
248452024-12-09T15:13:04.569ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
248462024-12-09T15:13:04.569ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
24847 Dec 09 15:13:04.569 INFO request completed{"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:13:04.56984809Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"0","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
24848 , latency_us: 169, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1e1ca861-77e0-4e2f-963f-7a5289c72f46, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.{"msg":"[0] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:13:04.569876995Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"0","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
24849 1:9998
248502024-12-09T15:13:04.569ZINFOcrucible: client task is exiting = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
24851 [8810] Exited with: ExitStatus(unix_wait_status(9))
24852 [8810][0] reports Exit
24853 Dec 09 15:13:08.572 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/0, method: GET, req_id: d0e7ad4f-9590-4dbc-a546-d1fe69eb75a1, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
24854 001/300 Read block 3711 len 4096
24855 002/300 Write block 4578 len 8192 data: 6 6
24856 003/300 Read block 4144 len 4096
24857 004/300 Read block 1245 len 40960
24858 005/300 Read block 2524 len 16384
24859 006/300 Write block 4502 len 28672 data: 8 8 7 7 10 10 10
24860 007/300 Read block 395 len 16384
24861 008/300 Write block 1260 len 24576 data: 3 3 3 3 4 4
24862 009/300 Flush
24863 010/300 Write block 2344 len 16384 data: 5 5 5 5
24864 011/300 Read block 1540 len 36864
24865 012/300 Write block 4927 len 8192 data: 4 4
24866 013/300 Write block 295 len 40960 data: 2 2 4 5 5 6 5 4 4 5
24867 014/300 Read block 966 len 4096
24868 015/300 Write block 1947 len 12288 data: 3 4 3
24869 016/300 Write block 4938 len 8192 data: 5 4
24870 017/300 Flush
24871 018/300 Read block 1030 len 16384
24872 019/300 Read block 4613 len 40960
24873 020/300 Read block 3482 len 32768
24874 021/300 Read block 364 len 32768
24875 022/300 Read block 1958 len 32768
24876 023/300 Write block 3239 len 28672 data: 6 5 3 5 7 6 6
24877 024/300 Write block 3871 len 28672 data: 6 6 7 7 7 10 9
24878 025/300 Read block 1558 len 32768
24879 026/300 Read block 4393 len 20480
24880 027/300 Read block 2775 len 8192
24881 028/300 Read block 1019 len 28672
24882 029/300 Write block 3089 len 40960 data: 4 4 4 4 5 5 4 4 5 6
24883 030/300 Read block 1676 len 24576
24884 031/300 Flush
24885 032/300 Flush
24886 033/300 Read block 3715 len 16384
24887 034/300 Write block 2734 len 32768 data: 7 8 7 6 7 5 6 5
24888 035/300 Write block 820 len 24576 data: 7 7 6 5 6 6
24889 036/300 Write block 3120 len 28672 data: 7 8 8 7 5 4 3
24890 037/300 Flush
24891 038/300 Flush
24892 039/300 Read block 1002 len 4096
24893 040/300 Read block 4581 len 36864
24894 041/300 Read block 2805 len 40960
24895 042/300 Read block 758 len 8192
24896 043/300 Flush
24897 044/300 Write block 749 len 36864 data: 3 4 4 4 3 5 4 4 4
24898 045/300 Flush
24899 046/300 Read block 1852 len 36864
24900 047/300 Read block 1809 len 28672
24901 048/300 Read block 1442 len 36864
24902 049/300 Write block 3970 len 28672 data: 4 4 5 4 6 6 6
24903 050/300 Write block 265 len 36864 data: 4 4 5 6 7 8 8 8 5
24904 051/300 Write block 1734 len 32768 data: 7 5 5 5 4 4 6 5
24905 052/300 Read block 4959 len 12288
24906 053/300 Read block 3079 len 4096
24907 054/300 Read block 4472 len 4096
24908 055/300 Write block 1097 len 8192 data: 4 3
24909 056/300 Read block 4652 len 24576
24910 057/300 Read block 2564 len 16384
24911 058/300 Flush
24912 059/300 Read block 1894 len 8192
24913 060/300 Read block 985 len 16384
24914 061/300 Write block 4296 len 28672 data: 5 4 4 4 4 3 2
24915 062/300 Read block 2174 len 12288
24916 063/300 Read block 3495 len 24576
24917 064/300 Read block 3389 len 8192
24918 065/300 Write block 717 len 28672 data: 4 4 5 6 6 4 5
24919 066/300 Read block 3163 len 16384
24920 067/300 Write block 1408 len 24576 data: 7 7 8 8 6 7
24921 068/300 Flush
24922 069/300 Read block 3000 len 20480
24923 070/300 Flush
24924 071/300 Read block 1683 len 36864
24925 072/300 Read block 2560 len 36864
24926 073/300 Write block 2547 len 8192 data: 3 5
24927 074/300 Write block 3176 len 40960 data: 6 7 6 6 7 7 7 7 7 5
24928 075/300 Write block 2895 len 24576 data: 8 7 7 5 6 6
24929 076/300 Write block 2103 len 4096 data: 5
24930 077/300 Write block 3655 len 40960 data: 5 4 4 3 3 3 3 5 5 5
24931 078/300 Write block 2254 len 24576 data: 7 7 5 6 7 6
24932 079/300 Write block 3192 len 20480 data: 2 3 3 5 5
24933 080/300 Flush
24934 081/300 Write block 4423 len 8192 data: 6 7
24935 082/300 Write block 3393 len 20480 data: 4 7 6 7 7
24936 083/300 Write block 2869 len 20480 data: 9 6 6 5 5
24937 084/300 Write block 1967 len 40960 data: 6 5 7 7 7 6 5 3 3 3
24938 085/300 Read block 2976 len 16384
24939 086/300 Write block 4052 len 16384 data: 7 5 4 2
24940 087/300 Write block 1773 len 20480 data: 3 3 5 6 5
24941 088/300 Read block 4167 len 4096
24942 089/300 Read block 586 len 12288
24943 090/300 Read block 1766 len 24576
24944 091/300 Read block 2440 len 32768
24945 092/300 Flush
24946 093/300 Flush
24947 094/300 Read block 2733 len 36864
24948 095/300 Read block 2235 len 4096
24949 096/300 Read block 2566 len 16384
24950 097/300 Write block 4645 len 4096 data: 6
24951 098/300 Read block 3641 len 28672
24952 099/300 Write block 483 len 8192 data: 8 8
24953 100/300 Read block 1915 len 28672
24954 101/300 Flush
24955 102/300 Write block 3055 len 16384 data: 3 3 4 3
24956 103/300 Read block 1531 len 4096
24957 104/300 Flush
24958 105/300 Read block 1078 len 28672
24959 106/300 Flush
24960 107/300 Write block 2723 len 28672 data: 3 4 3 3 3 4 5
24961 108/300 Write block 4949 len 24576 data: 6 6 5 3 3 3
24962 109/300 Read block 1421 len 40960
24963 110/300 Write block 121 len 20480 data: 3 4 3 4 4
24964 111/300 Read block 273 len 4096
24965 112/300 Read block 1204 len 24576
24966 113/300 Write block 3052 len 40960 data: 2 2 2 4 4 5 4 3 3 6
24967 114/300 Read block 2777 len 12288
24968 115/300 Read block 4470 len 20480
24969 116/300 Write block 2190 len 8192 data: 8 7
24970 117/300 Flush
24971 118/300 Flush
24972 119/300 Write block 3723 len 8192 data: 5 5
24973 120/300 Read block 4325 len 8192
24974 121/300 Read block 1879 len 16384
24975 122/300 Read block 2621 len 12288
24976 123/300 Read block 1442 len 8192
24977 124/300 Write block 1047 len 32768 data: 5 5 3 3 2 3 3 3
24978 125/300 Write block 3761 len 12288 data: 6 7 6
24979 126/300 Read block 3587 len 28672
24980 127/300 Read block 2077 len 32768
24981 128/300 Read block 1740 len 4096
24982 129/300 Read block 4597 len 12288
24983 130/300 Read block 2034 len 16384
24984 131/300 Read block 3232 len 40960
24985 132/300 Write block 1542 len 36864 data: 4 6 6 6 5 6 5 3 3
24986 133/300 Write block 2281 len 20480 data: 2 5 4 4 4
24987 134/300 Read block 3695 len 28672
24988 135/300 Write block 4738 len 28672 data: 3 3 3 3 4 3 4
24989 136/300 Read block 2370 len 24576
24990 137/300 Write block 2241 len 4096 data: 8
24991 138/300 Write block 3227 len 16384 data: 6 5 5 7
24992 139/300 Read block 3434 len 28672
24993 140/300 Flush
24994 141/300 Write block 2179 len 40960 data: 8 8 7 7 6 5 3 4 7 8
24995 142/300 Read block 1703 len 40960
24996 143/300 Write block 2874 len 12288 data: 4 3 3
24997 144/300 Write block 2230 len 24576 data: 6 4 5 5 6 6
24998 145/300 Read block 4951 len 24576
24999 146/300 Read block 418 len 20480
25000 147/300 Write block 3171 len 40960 data: 5 3 3 4 6 7 8 7 7 8
25001 148/300 Write block 1846 len 8192 data: 6 4
25002 149/300 Read block 1134 len 4096
25003 150/300 Write block 769 len 12288 data: 6 4 5
25004 151/300 Read block 3014 len 28672
25005 152/300 Read block 1712 len 4096
25006 153/300 Read block 2387 len 4096
25007 154/300 Write block 3813 len 36864 data: 4 5 4 4 4 5 4 4 5
25008 155/300 Read block 989 len 36864
25009 156/300 Write block 1470 len 20480 data: 7 6 6 5 7
25010 157/300 Write block 3029 len 32768 data: 4 3 2 2 2 2 2 2
25011 158/300 Write block 2129 len 24576 data: 7 7 9 8 7 8
25012 159/300 Read block 457 len 36864
25013 160/300 Read block 1941 len 28672
25014 161/300 Write block 4901 len 24576 data: 3 3 3 3 3 2
25015 162/300 Write block 2693 len 20480 data: 4 4 3 4 5
25016 163/300 Read block 2708 len 40960
25017 164/300 Read block 4055 len 40960
25018 165/300 Flush
25019 166/300 Write block 867 len 12288 data: 6 6 6
25020 167/300 Read block 738 len 12288
25021 168/300 Read block 4158 len 20480
25022 169/300 Read block 362 len 28672
25023 170/300 Write block 4346 len 36864 data: 7 8 7 9 8 7 5 5 5
25024 171/300 Read block 107 len 4096
25025 172/300 Flush
25026 173/300 Write block 806 len 4096 data: 5
25027 174/300 Read block 4127 len 32768
25028 175/300 Flush
25029 176/300 Write block 4553 len 4096 data: 6
25030 177/300 Write block 2401 len 4096 data: 7
25031 178/300 Write block 923 len 20480 data: 6 6 8 9 6
25032 179/300 Read block 1039 len 8192
25033 180/300 Write block 4405 len 36864 data: 5 7 7 7 8 6 7 6 6
25034 181/300 Write block 3353 len 20480 data: 3 3 4 4 4
25035 182/300 Write block 2802 len 24576 data: 4 5 5 5 5 5
25036 183/300 Flush
25037 184/300 Read block 4678 len 40960
25038 185/300 Write block 2679 len 32768 data: 5 5 4 3 2 4 4 4
25039 186/300 Flush
25040 187/300 Read block 2135 len 32768
25041 188/300 Write block 2076 len 24576 data: 6 6 3 6 8 7
25042 189/300 Write block 2422 len 20480 data: 4 5 5 4 3
25043 190/300 Write block 190 len 12288 data: 6 5 7
25044 191/300 Read block 3134 len 12288
25045 192/300 Write block 4049 len 32768 data: 6 6 7 8 6 5 3 3
25046 193/300 Write block 4284 len 32768 data: 4 5 7 8 7 7 6 5
25047 194/300 Read block 4520 len 28672
25048 195/300 Flush
25049 196/300 Write block 206 len 12288 data: 4 4 4
25050 197/300 Read block 3221 len 36864
25051 198/300 Read block 1023 len 24576
25052 199/300 Flush
25053 200/300 Write block 2346 len 24576 data: 6 6 5 6 5 5
25054 201/300 Read block 4612 len 24576
25055 202/300 Read block 395 len 36864
25056 203/300 Write block 2874 len 20480 data: 5 4 4 3 3
25057 204/300 Read block 224 len 24576
25058 205/300 Read block 3854 len 4096
25059 206/300 Read block 1274 len 28672
25060 207/300 Write block 3206 len 20480 data: 2 2 2 4 5
25061 208/300 Read block 3407 len 16384
25062 209/300 Read block 1940 len 8192
25063 210/300 Read block 24 len 8192
25064 211/300 Read block 381 len 12288
25065 212/300 Flush
25066 213/300 Read block 3035 len 32768
25067 214/300 Flush
25068 215/300 Write block 296 len 20480 data: 3 5 6 6 7
25069 216/300 Flush
25070 217/300 Read block 4252 len 4096
25071 218/300 Read block 4269 len 40960
25072 219/300 Flush
25073 220/300 Flush
25074 221/300 Read block 4689 len 16384
25075 222/300 Write block 1404 len 16384 data: 6 6 5 6
25076 223/300 Read block 106 len 12288
25077 224/300 Read block 1049 len 8192
25078 225/300 Read block 3657 len 28672
25079 226/300 Write block 2552 len 40960 data: 4 3 3 4 6 6 7 8 7 5
25080 227/300 Flush
25081 228/300 Flush
25082 229/300 Write block 1893 len 12288 data: 6 5 5
25083 230/300 Read block 243 len 28672
25084 231/300 Read block 4157 len 28672
25085 232/300 Write block 1986 len 12288 data: 5 3 3
25086 233/300 Flush
25087 234/300 Write block 3562 len 36864 data: 4 3 4 6 6 6 6 4 3
25088 235/300 Write block 4601 len 12288 data: 6 5 4
25089 236/300 Read block 4911 len 4096
25090 237/300 Read block 1089 len 28672
25091 238/300 Write block 4344 len 16384 data: 5 5 8 9
25092 239/300 Read block 2621 len 24576
25093 240/300 Read block 4930 len 16384
25094 241/300 Read block 1104 len 40960
25095 242/300 Read block 1387 len 20480
25096 243/300 Write block 4901 len 24576 data: 4 4 4 4 4 3
25097 244/300 Flush
25098 245/300 Read block 424 len 28672
25099 246/300 Read block 4148 len 16384
25100 247/300 Read block 553 len 16384
25101 248/300 Flush
25102 249/300 Read block 827 len 4096
25103 250/300 Write block 2649 len 4096 data: 6
25104 251/300 Read block 2490 len 32768
25105 252/300 Read block 146 len 20480
25106 253/300 Write block 1764 len 4096 data: 6
25107 254/300 Read block 1864 len 32768
25108 255/300 Read block 209 len 12288
25109 256/300 Write block 4253 len 40960 data: 4 4 5 6 4 4 6 6 7 6
25110 257/300 Read block 4643 len 12288
25111 258/300 Read block 2211 len 8192
25112 259/300 Write block 3772 len 24576 data: 3 4 4 5 6 7
25113 260/300 Write block 128 len 20480 data: 6 4 4 6 7
25114 261/300 Write block 3693 len 4096 data: 3
25115 262/300 Read block 136 len 16384
25116 263/300 Read block 3077 len 16384
25117 264/300 Read block 2168 len 4096
25118 265/300 Write block 39 len 16384 data: 6 4 3 2
25119 266/300 Read block 1855 len 20480
25120 267/300 Read block 3914 len 24576
25121 268/300 Read block 384 len 16384
25122 269/300 Write block 3830 len 32768 data: 6 6 6 6 4 5 5 4
25123 270/300 Write block 475 len 16384 data: 5 5 6 6
25124 271/300 Write block 4840 len 32768 data: 8 8 7 7 5 5 4 4
25125 272/300 Read block 3258 len 40960
25126 273/300 Read block 2538 len 28672
25127 274/300 Write block 1101 len 20480 data: 3 5 5 5 5
25128 275/300 Read block 188 len 28672
25129 276/300 Flush
25130 277/300 Read block 4115 len 8192
25131 278/300 Write block 4556 len 16384 data: 9 8 6 5
25132 279/300 Read block 3341 len 32768
25133 280/300 Write block 849 len 40960 data: 2 4 5 5 6 6 6 5 5 5
25134 281/300 Read block 182 len 40960
25135 282/300 Write block 2783 len 40960 data: 7 6 7 5 4 6 6 5 6 6
25136 283/300 Write block 3538 len 32768 data: 3 3 4 4 4 5 5 6
25137 284/300 Read block 4131 len 28672
25138 285/300 Read block 1658 len 20480
25139 286/300 Read block 4478 len 28672
25140 287/300 Write block 3744 len 28672 data: 4 5 6 5 5 5 6
25141 288/300 Write block 1524 len 28672 data: 4 4 5 4 6 5 5
25142 289/300 Write block 2034 len 32768 data: 5 5 7 4 5 4 4 5
25143 290/300 Write block 582 len 12288 data: 6 5 4
25144 291/300 Write block 4813 len 4096 data: 8
25145 292/300 Read block 3355 len 12288
25146 293/300 Write block 4468 len 36864 data: 6 6 7 6 6 6 6 5 4
25147 294/300 Write block 2241 len 32768 data: 9 7 7 7 6 4 4 5
25148 295/300 Read block 3191 len 24576
25149 296/300 Read block 1650 len 8192
25150 297/300 Read block 3192 len 8192
25151 298/300 Write block 2790 len 40960 data: 6 7 7 7 6 6 4 5 5 5
25152 299/300 Read block 489 len 40960
25153 300/300 Write block 1139 len 24576 data: 3 2 4 6 5 6
25154 301/300 Read block 4325 len 40960
25155 Main task has work to do, go find it
25156 got dsc Start(0)
25157 start 0
25158 Dec 09 15:13:08.807 INFO request completed, latency_us: 231, response_code: 204, uri: /start/cid/0, method: POST, req_id: eb4f0bbf-453f-466d-8173-8e8019f6ae82, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
25159 Replay: started 0, returned:Ok(())
25160 ----------------------------------------------------------------
25161 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25162 States: Offline Active Active
25163 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25164 7644 Acked Read 1 Sent Done Done false
25165 7645 Acked Write 2 Sent Done Done false
25166 7646 Acked Read 1 Sent Done Done false
25167 7647 Acked Read 10 Sent Done Done false
25168 7648 Acked Read 4 Sent Done Done false
25169 7649 Acked Write 7 Sent Done Done false
25170 7650 Acked Read 4 Sent Done Done false
25171 7651 Acked Write 6 Sent Done Done false
25172 7652 Acked Flush 0 Sent Done Done false
25173 7653 Acked Write 4 Sent Done Done false
25174 7654 Acked Read 9 Sent Done Done false
25175 7655 Acked Write 2 Sent Done Done false
25176 7656 Acked Write 10 Sent Done Done false
25177 7657 Acked Read 1 Sent Done Done false
25178 7658 Acked Write 3 Sent Done Done false
25179 7659 Acked Write 2 Sent Done Done false
25180 7660 Acked Flush 0 Sent Done Done false
25181 7661 Acked Read 4 Sent Done Done false
25182 7662 Acked Read 10 Sent Done Done false
25183 7663 Acked Read 8 Sent Done Done false
25184 7664 Acked Read 8 Sent Done Done false
25185 7665 Acked Read 8 Sent Done Done false
25186 7666 Acked Write 7 Sent Done Done false
25187 7667 Acked Write 7 Sent Done Done false
25188 7668 Acked Read 8 Sent Done Done false
25189 7669 Acked Read 5 Sent Done Done false
25190 7670 Acked Read 2 Sent Done Done false
25191 7671 Acked Read 7 Sent Done Done false
25192 7672 Acked Write 10 Sent Done Done false
25193 7673 Acked Read 6 Sent Done Done false
25194 7674 Acked Flush 0 Sent Done Done false
25195 7675 Acked Flush 0 Sent Done Done false
25196 7676 Acked Read 4 Sent Done Done false
25197 7677 Acked Write 8 Sent Done Done false
25198 7678 Acked Write 6 Sent Done Done false
25199 7679 Acked Write 7 Sent Done Done false
25200 7680 Acked Flush 0 Sent Done Done false
25201 7681 Acked Flush 0 Sent Done Done false
25202 7682 Acked Read 1 Sent Done Done false
25203 7683 Acked Read 9 Sent Done Done false
25204 7684 Acked Read 10 Sent Done Done false
25205 7685 Acked Read 2 Sent Done Done false
25206 7686 Acked Flush 0 Sent Done Done false
25207 7687 Acked Write 9 Sent Done Done false
25208 7688 Acked Flush 0 Sent Done Done false
25209 7689 Acked Read 9 Sent Done Done false
25210 7690 Acked Read 7 Sent Done Done false
25211 7691 Acked Read 9 Sent Done Done false
25212 7692 Acked Write 7 Sent Done Done false
25213 7693 Acked Write 9 Sent Done Done false
25214 7694 Acked Write 8 Sent Done Done false
25215 7695 Acked Read 3 Sent Done Done false
25216 7696 Acked Read 1 Sent Done Done false
25217 7697 Acked Read 1 Sent Done Done false
25218 7698 Acked Write 2 Sent Done Done false
25219 7699 Acked Read 6 Sent Done Done false
25220 7700 Acked Read 4 Sent Done Done false
25221 7701 Acked Flush 0 Sent Done Done false
25222 7702 Acked Read 2 Sent Done Done false
25223 7703 Acked Read 4 Sent Done Done false
25224 7704 Acked Write 7 Sent Done Done false
25225 7705 Acked Read 3 Sent Done Done false
25226 7706 Acked Read 6 Sent Done Done false
25227 7707 Acked Read 2 Sent Done Done false
25228 7708 Acked Write 7 Sent Done Done false
25229 7709 Acked Read 4 Sent Done Done false
25230 7710 Acked Write 6 Sent Done Done false
25231 7711 Acked Flush 0 Sent Done Done false
25232 7712 Acked Read 5 Sent Done Done false
25233 7713 Acked Flush 0 Sent Done Done false
25234 7714 Acked Read 9 Sent Done Done false
25235 7715 Acked Read 9 Sent Done Done false
25236 7716 Acked Write 2 Sent Done Done false
25237 7717 Acked Write 10 Sent Done Done false
25238 7718 Acked Write 6 Sent Done Done false
25239 7719 Acked Write 1 Sent Done Done false
25240 7720 Acked Write 10 Sent Done Done false
25241 7721 Acked Write 6 Sent Done Done false
25242 7722 Acked Write 5 Sent Done Done false
25243 7723 Acked Flush 0 Sent Done Done false
25244 7724 Acked Write 2 Sent Done Done false
25245 7725 Acked Write 5 Sent Done Done false
25246 7726 Acked Write 5 Sent Done Done false
25247 7727 Acked Write 10 Sent Done Done false
25248 7728 Acked Read 4 Sent Done Done false
25249 7729 Acked Write 4 Sent Done Done false
25250 7730 Acked Write 5 Sent Done Done false
25251 7731 Acked Read 1 Sent Done Done false
25252 7732 Acked Read 3 Sent Done Done false
25253 7733 Acked Read 6 Sent Done Done false
25254 7734 Acked Read 8 Sent Done Done false
25255 7735 Acked Flush 0 Sent Done Done false
25256 7736 Acked Flush 0 Sent Done Done false
25257 7737 Acked Read 9 Sent Done Done false
25258 7738 Acked Read 1 Sent Done Done false
25259 7739 Acked Read 4 Sent Done Done false
25260 7740 Acked Write 1 Sent Done Done false
25261 7741 Acked Read 7 Sent Done Done false
25262 7742 Acked Write 2 Sent Done Done false
25263 7743 Acked Read 7 Sent Done Done false
25264 7744 Acked Flush 0 Sent Done Done false
25265 7745 Acked Write 4 Sent Done Done false
25266 7746 Acked Read 1 Sent Done Done false
25267 7747 Acked Flush 0 Sent Done Done false
25268 7748 Acked Read 7 Sent Done Done false
25269 7749 Acked Flush 0 Sent Done Done false
25270 7750 Acked Write 7 Sent Done Done false
25271 7751 Acked Write 6 Sent Done Done false
25272 7752 Acked Read 10 Sent Done Done false
25273 7753 Acked Write 5 Sent Done Done false
25274 7754 Acked Read 1 Sent Done Done false
25275 7755 Acked Read 6 Sent Done Done false
25276 7756 Acked Write 10 Sent Done Done false
25277 7757 Acked Read 3 Sent Done Done false
25278 7758 Acked Read 5 Sent Done Done false
25279 7759 Acked Write 2 Sent Done Done false
25280 7760 Acked Flush 0 Sent Done Done false
25281 7761 Acked Flush 0 Sent Done Done false
25282 7762 Acked Write 2 Sent Done Done false
25283 7763 Acked Read 2 Sent Done Done false
25284 7764 Acked Read 4 Sent Done Done false
25285 7765 Acked Read 3 Sent Done Done false
25286 7766 Acked Read 2 Sent Done Done false
25287 7767 Acked Write 8 Sent Done Done false
25288 7768 Acked Write 3 Sent Done Done false
25289 7769 Acked Read 7 Sent Done Done false
25290 7770 Acked Read 8 Sent Done Done false
25291 7771 Acked Read 1 Sent Done Done false
25292 7772 Acked Read 3 Sent Done Done false
25293 7773 Acked Read 4 Sent Done Done false
25294 7774 Acked Read 10 Sent Done Done false
25295 7775 Acked Write 9 Sent Done Done false
25296 7776 Acked Write 5 Sent Done Done false
25297 7777 Acked Read 7 Sent Done Done false
25298 7778 Acked Write 7 Sent Done Done false
25299 7779 Acked Read 6 Sent Done Done false
25300 7780 Acked Write 1 Sent Done Done false
25301 7781 Acked Write 4 Sent Done Done false
25302 7782 Acked Read 7 Sent Done Done false
25303 7783 Acked Flush 0 Sent Done Done false
25304 7784 Acked Write 10 Sent Done Done false
25305 7785 Acked Read 10 Sent Done Done false
25306 7786 Acked Write 3 Sent Done Done false
25307 7787 Acked Write 6 Sent Done Done false
25308 7788 Acked Read 6 Sent Done Done false
25309 7789 Acked Read 5 Sent Done Done false
25310 7790 Acked Write 10 Sent Done Done false
25311 7791 Acked Write 2 Sent Done Done false
25312 7792 Acked Read 1 Sent Done Done false
25313 7793 Acked Write 3 Sent Done Done false
25314 7794 Acked Read 7 Sent Done Done false
25315 7795 Acked Read 1 Sent Done Done false
25316 7796 Acked Read 1 Sent Done Done false
25317 7797 Acked Write 9 Sent Done Done false
25318 7798 Acked Read 9 Sent Done Done false
25319 7799 Acked Write 5 Sent Done Done false
25320 7800 Acked Write 8 Sent Done Done false
25321 7801 Acked Write 6 Sent Done Done false
25322 7802 Acked Read 9 Sent Done Done false
25323 7803 Acked Read 7 Sent Done Done false
25324 7804 Acked Write 6 Sent Done Done false
25325 7805 Acked Write 5 Sent Done Done false
25326 7806 Acked Read 10 Sent Done Done false
25327 7807 Acked Read 10 Sent Done Done false
25328 7808 Acked Flush 0 Sent Done Done false
25329 7809 Acked Write 3 Sent Done Done false
25330 7810 Acked Read 3 Sent Done Done false
25331 7811 Acked Read 5 Sent Done Done false
25332 7812 Acked Read 7 Sent Done Done false
25333 7813 Acked Write 9 Sent Done Done false
25334 7814 Acked Read 1 Sent Done Done false
25335 7815 Acked Flush 0 Sent Done Done false
25336 7816 Acked Write 1 Sent Done Done false
25337 7817 Acked Read 8 Sent Done Done false
25338 7818 Acked Flush 0 Sent Done Done false
25339 7819 Acked Write 1 Sent Done Done false
25340 7820 Acked Write 1 Sent Done Done false
25341 7821 Acked Write 5 Sent Done Done false
25342 7822 Acked Read 2 Sent Done Done false
25343 7823 Acked Write 9 Sent Done Done false
25344 7824 Acked Write 5 Sent Done Done false
25345 7825 Acked Write 6 Sent Done Done false
25346 7826 Acked Flush 0 Sent Done Done false
25347 7827 Acked Read 10 Sent Done Done false
25348 7828 Acked Write 8 Sent Done Done false
25349 7829 Acked Flush 0 Sent Done Done false
25350 7830 Acked Read 8 Sent Done Done false
25351 7831 Acked Write 6 Sent Done Done false
25352 7832 Acked Write 5 Sent Done Done false
25353 7833 Acked Write 3 Sent Done Done false
25354 7834 Acked Read 3 Sent Done Done false
25355 7835 Acked Write 8 Sent Done Done false
25356 7836 Acked Write 8 Sent Done Done false
25357 7837 Acked Read 7 Sent Done Done false
25358 7838 Acked Flush 0 Sent Done Done false
25359 7839 Acked Write 3 Sent Done Done false
25360 7840 Acked Read 9 Sent Done Done false
25361 7841 Acked Read 6 Sent Done Done false
25362 7842 Acked Flush 0 Sent Done Done false
25363 7843 Acked Write 6 Sent Done Done false
25364 7844 Acked Read 6 Sent Done Done false
25365 7845 Acked Read 9 Sent Done Done false
25366 7846 Acked Write 5 Sent Done Done false
25367 7847 Acked Read 6 Sent Done Done false
25368 7848 Acked Read 1 Sent Done Done false
25369 7849 Acked Read 7 Sent Done Done false
25370 7850 Acked Write 5 Sent Done Done false
25371 7851 Acked Read 4 Sent Done Done false
25372 7852 Acked Read 2 Sent Done Done false
25373 7853 Acked Read 2 Sent Done Done false
25374 7854 Acked Read 3 Sent Done Done false
25375 7855 Acked Flush 0 Sent Done Done false
25376 7856 Acked Read 8 Sent Done Done false
25377 7857 Acked Flush 0 Sent Done Done false
25378 7858 Acked Write 5 Sent Done Done false
25379 7859 Acked Flush 0 Sent Done Done false
25380 7860 Acked Read 1 Sent Done Done false
25381 7861 Acked Read 10 Sent Done Done false
25382 7862 Acked Flush 0 Sent Done Done false
25383 7863 Acked Flush 0 Sent Done Done false
25384 7864 Acked Read 4 Sent Done Done false
25385 7865 Acked Write 4 Sent Done Done false
25386 7866 Acked Read 3 Sent Done Done false
25387 7867 Acked Read 2 Sent Done Done false
25388 7868 Acked Read 7 Sent Done Done false
25389 7869 Acked Write 10 Sent Done Done false
25390 7870 Acked Flush 0 Sent Done Done false
25391 7871 Acked Flush 0 Sent Done Done false
25392 7872 Acked Write 3 Sent Done Done false
25393 7873 Acked Read 7 Sent Done Done false
25394 7874 Acked Read 7 Sent Done Done false
25395 7875 Acked Write 3 Sent Done Done false
25396 7876 Acked Flush 0 Sent Done Done false
25397 7877 Acked Write 9 Sent Done Done false
25398 7878 Acked Write 3 Sent Done Done false
25399 7879 Acked Read 1 Sent Done Done false
25400 7880 Acked Read 7 Sent Done Done false
25401 7881 Acked Write 4 Sent Done Done false
25402 7882 Acked Read 6 Sent Done Done false
25403 7883 Acked Read 4 Sent Done Done false
25404 7884 Acked Read 10 Sent Done Done false
25405 7885 Acked Read 5 Sent Done Done false
25406 7886 Acked Write 6 Sent Done Done false
25407 7887 Acked Flush 0 Sent Done Done false
25408 7888 Acked Read 7 Sent Done Done false
25409 7889 Acked Read 4 Sent Done Done false
25410 7890 Acked Read 4 Sent Done Done false
25411 7891 Acked Flush 0 Sent Done Done false
25412 7892 Acked Read 1 Sent Done Done false
25413 7893 Acked Write 1 Sent Done Done false
25414 7894 Acked Read 8 Sent Done Done false
25415 7895 Acked Read 5 Sent Done Done false
25416 7896 Acked Write 1 Sent Done Done false
25417 7897 Acked Read 8 Sent Done Done false
25418 7898 Acked Read 3 Sent Done Done false
25419 7899 Acked Write 10 Sent Done Done false
25420 7900 Acked Read 3 Sent Done Done false
25421 7901 Acked Read 2 Sent Done Done false
25422 7902 Acked Write 6 Sent Done Done false
25423 7903 Acked Write 5 Sent Done Done false
25424 7904 Acked Write 1 Sent Done Done false
25425 7905 Acked Read 4 Sent Done Done false
25426 7906 Acked Read 4 Sent Done Done false
25427 7907 Acked Read 1 Sent Done Done false
25428 7908 Acked Write 4 Sent Done Done false
25429 7909 Acked Read 5 Sent Done Done false
25430 7910 Acked Read 6 Sent Done Done false
25431 7911 Acked Read 4 Sent Done Done false
25432 7912 Acked Write 8 Sent Done Done false
25433 7913 Acked Write 4 Sent Done Done false
25434 7914 Acked Write 8 Sent Done Done false
25435 7915 Acked Read 10 Sent Done Done false
25436 7916 Acked Read 7 Sent Done Done false
25437 7917 Acked Write 5 Sent Done Done false
25438 7918 Acked Read 7 Sent Done Done false
25439 7919 Acked Flush 0 Sent Done Done false
25440 7920 Acked Read 2 Sent Done Done false
25441 7921 Acked Write 4 Sent Done Done false
25442 7922 Acked Read 8 Sent Done Done false
25443 7923 Acked Write 10 Sent Done Done false
25444 7924 Acked Read 10 Sent Done Done false
25445 7925 Acked Write 10 Sent Done Done false
25446 7926 Acked Write 8 Sent Done Done false
25447 7927 Acked Read 7 Sent Done Done false
25448 7928 Acked Read 5 Sent Done Done false
25449 7929 Acked Read 7 Sent Done Done false
25450 7930 Acked Write 7 Sent Done Done false
25451 7931 Acked Write 7 Sent Done Done false
25452 7932 Acked Write 8 Sent Done Done false
25453 7933 Acked Write 3 Sent Done Done false
25454 7934 Acked Write 1 Sent Done Done false
25455 7935 Acked Read 3 Sent Done Done false
25456 7936 Acked Write 9 Sent Done Done false
25457 7937 Acked Write 8 Sent Done Done false
25458 7938 Acked Read 6 Sent Done Done false
25459 7939 Acked Read 2 Sent Done Done false
25460 7940 Acked Read 2 Sent Done Done false
25461 7941 Acked Write 10 Sent Done Done false
25462 7942 Acked Read 10 Sent Done Done false
25463 7943 Acked Write 6 Sent Done Done false
25464 7944 Acked Read 10 Sent Done Done false
25465 STATES DS:0 DS:1 DS:2 TOTAL
25466 Sent 301 0 0 301
25467 Done 0 301 301 602
25468 Skipped 0 0 0 0
25469 Error 0 0 0 0
25470 Last Flush: 7643 7919 7919
25471 Downstairs last five completed: 7643 7642 7641 7640 7639
25472 Upstairs last five completed: 7944 7943 7942 7941 7940
25473 CLIENT: Up:0 ds:301 act:2
25474 [8810] got start action
25475 [8810] I am going to restart
25476 Starting downstairs at port 8810
25477 Make output file at "/tmp/dsc/downstairs-8810.txt"
25478 [8810][0] reports Starting
25479 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1224)
25480 [8810][0] reports Running
25481 ----------------------------------------------------------------
25482 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
25483 States: Offline Active Active
25484 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25485 7644 Acked Read 1 Sent Done Done false
25486 7645 Acked Write 2 Sent Done Done false
25487 7646 Acked Read 1 Sent Done Done false
25488 7647 Acked Read 10 Sent Done Done false
25489 7648 Acked Read 4 Sent Done Done false
25490 7649 Acked Write 7 Sent Done Done false
25491 7650 Acked Read 4 Sent Done Done false
25492 7651 Acked Write 6 Sent Done Done false
25493 7652 Acked Flush 0 Sent Done Done false
25494 7653 Acked Write 4 Sent Done Done false
25495 7654 Acked Read 9 Sent Done Done false
25496 7655 Acked Write 2 Sent Done Done false
25497 7656 Acked Write 10 Sent Done Done false
25498 7657 Acked Read 1 Sent Done Done false
25499 7658 Acked Write 3 Sent Done Done false
25500 7659 Acked Write 2 Sent Done Done false
25501 7660 Acked Flush 0 Sent Done Done false
25502 7661 Acked Read 4 Sent Done Done false
25503 7662 Acked Read 10 Sent Done Done false
25504 7663 Acked Read 8 Sent Done Done false
25505 7664 Acked Read 8 Sent Done Done false
25506 7665 Acked Read 8 Sent Done Done false
25507 7666 Acked Write 7 Sent Done Done false
25508 7667 Acked Write 7 Sent Done Done false
25509 7668 Acked Read 8 Sent Done Done false
25510 7669 Acked Read 5 Sent Done Done false
25511 7670 Acked Read 2 Sent Done Done false
25512 7671 Acked Read 7 Sent Done Done false
25513 7672 Acked Write 10 Sent Done Done false
25514 7673 Acked Read 6 Sent Done Done false
25515 7674 Acked Flush 0 Sent Done Done false
25516 7675 Acked Flush 0 Sent Done Done false
25517 7676 Acked Read 4 Sent Done Done false
25518 7677 Acked Write 8 Sent Done Done false
25519 7678 Acked Write 6 Sent Done Done false
25520 7679 Acked Write 7 Sent Done Done false
25521 7680 Acked Flush 0 Sent Done Done false
25522 7681 Acked Flush 0 Sent Done Done false
25523 7682 Acked Read 1 Sent Done Done false
25524 7683 Acked Read 9 Sent Done Done false
25525 7684 Acked Read 10 Sent Done Done false
25526 7685 Acked Read 2 Sent Done Done false
25527 7686 Acked Flush 0 Sent Done Done false
25528 7687 Acked Write 9 Sent Done Done false
25529 7688 Acked Flush 0 Sent Done Done false
25530 7689 Acked Read 9 Sent Done Done false
25531 7690 Acked Read 7 Sent Done Done false
25532 7691 Acked Read 9 Sent Done Done false
25533 7692 Acked Write 7 Sent Done Done false
25534 7693 Acked Write 9 Sent Done Done false
25535 7694 Acked Write 8 Sent Done Done false
25536 7695 Acked Read 3 Sent Done Done false
25537 7696 Acked Read 1 Sent Done Done false
25538 7697 Acked Read 1 Sent Done Done false
25539 7698 Acked Write 2 Sent Done Done false
25540 7699 Acked Read 6 Sent Done Done false
25541 7700 Acked Read 4 Sent Done Done false
25542 7701 Acked Flush 0 Sent Done Done false
25543 7702 Acked Read 2 Sent Done Done false
25544 7703 Acked Read 4 Sent Done Done false
25545 7704 Acked Write 7 Sent Done Done false
25546 7705 Acked Read 3 Sent Done Done false
25547 7706 Acked Read 6 Sent Done Done false
25548 7707 Acked Read 2 Sent Done Done false
25549 7708 Acked Write 7 Sent Done Done false
25550 7709 Acked Read 4 Sent Done Done false
25551 7710 Acked Write 6 Sent Done Done false
25552 7711 Acked Flush 0 Sent Done Done false
25553 7712 Acked Read 5 Sent Done Done false
25554 7713 Acked Flush 0 Sent Done Done false
25555 7714 Acked Read 9 Sent Done Done false
25556 7715 Acked Read 9 Sent Done Done false
25557 7716 Acked Write 2 Sent Done Done false
25558 7717 Acked Write 10 Sent Done Done false
25559 7718 Acked Write 6 Sent Done Done false
25560 7719 Acked Write 1 Sent Done Done false
25561 7720 Acked Write 10 Sent Done Done false
25562 7721 Acked Write 6 Sent Done Done false
25563 7722 Acked Write 5 Sent Done Done false
25564 7723 Acked Flush 0 Sent Done Done false
25565 7724 Acked Write 2 Sent Done Done false
25566 7725 Acked Write 5 Sent Done Done false
25567 7726 Acked Write 5 Sent Done Done false
25568 7727 Acked Write 10 Sent Done Done false
25569 7728 Acked Read 4 Sent Done Done false
25570 7729 Acked Write 4 Sent Done Done false
25571 7730 Acked Write 5 Sent Done Done false
25572 7731 Acked Read 1 Sent Done Done false
25573 7732 Acked Read 3 Sent Done Done false
25574 7733 Acked Read 6 Sent Done Done false
25575 7734 Acked Read 8 Sent Done Done false
25576 7735 Acked Flush 0 Sent Done Done false
25577 7736 Acked Flush 0 Sent Done Done false
25578 7737 Acked Read 9 Sent Done Done false
25579 7738 Acked Read 1 Sent Done Done false
25580 7739 Acked Read 4 Sent Done Done false
25581 7740 Acked Write 1 Sent Done Done false
25582 7741 Acked Read 7 Sent Done Done false
25583 7742 Acked Write 2 Sent Done Done false
25584 7743 Acked Read 7 Sent Done Done false
25585 7744 Acked Flush 0 Sent Done Done false
25586 7745 Acked Write 4 Sent Done Done false
25587 7746 Acked Read 1 Sent Done Done false
25588 7747 Acked Flush 0 Sent Done Done false
25589 7748 Acked Read 7 Sent Done Done false
25590 7749 Acked Flush 0 Sent Done Done false
25591 7750 Acked Write 7 Sent Done Done false
25592 7751 Acked Write 6 Sent Done Done false
25593 7752 Acked Read 10 Sent Done Done false
25594 7753 Acked Write 5 Sent Done Done false
25595 7754 Acked Read 1 Sent Done Done false
25596 7755 Acked Read 6 Sent Done Done false
25597 7756 Acked Write 10 Sent Done Done false
25598 7757 Acked Read 3 Sent Done Done false
25599 7758 Acked Read 5 Sent Done Done false
25600 7759 Acked Write 2 Sent Done Done false
25601 7760 Acked Flush 0 Sent Done Done false
25602 7761 Acked Flush 0 Sent Done Done false
25603 7762 Acked Write 2 Sent Done Done false
25604 7763 Acked Read 2 Sent Done Done false
25605 7764 Acked Read 4 Sent Done Done false
25606 7765 Acked Read 3 Sent Done Done false
25607 7766 Acked Read 2 Sent Done Done false
25608 7767 Acked Write 8 Sent Done Done false
25609 7768 Acked Write 3 Sent Done Done false
25610 7769 Acked Read 7 Sent Done Done false
25611 7770 Acked Read 8 Sent Done Done false
25612 7771 Acked Read 1 Sent Done Done false
25613 7772 Acked Read 3 Sent Done Done false
25614 7773 Acked Read 4 Sent Done Done false
25615 7774 Acked Read 10 Sent Done Done false
25616 7775 Acked Write 9 Sent Done Done false
25617 7776 Acked Write 5 Sent Done Done false
25618 7777 Acked Read 7 Sent Done Done false
25619 7778 Acked Write 7 Sent Done Done false
25620 7779 Acked Read 6 Sent Done Done false
25621 7780 Acked Write 1 Sent Done Done false
25622 7781 Acked Write 4 Sent Done Done false
25623 7782 Acked Read 7 Sent Done Done false
25624 7783 Acked Flush 0 Sent Done Done false
25625 7784 Acked Write 10 Sent Done Done false
25626 7785 Acked Read 10 Sent Done Done false
25627 7786 Acked Write 3 Sent Done Done false
25628 7787 Acked Write 6 Sent Done Done false
25629 7788 Acked Read 6 Sent Done Done false
25630 7789 Acked Read 5 Sent Done Done false
25631 7790 Acked Write 10 Sent Done Done false
25632 7791 Acked Write 2 Sent Done Done false
25633 7792 Acked Read 1 Sent Done Done false
25634 7793 Acked Write 3 Sent Done Done false
25635 7794 Acked Read 7 Sent Done Done false
25636 7795 Acked Read 1 Sent Done Done false
25637 7796 Acked Read 1 Sent Done Done false
25638 7797 Acked Write 9 Sent Done Done false
25639 7798 Acked Read 9 Sent Done Done false
25640 7799 Acked Write 5 Sent Done Done false
25641 7800 Acked Write 8 Sent Done Done false
25642 7801 Acked Write 6 Sent Done Done false
25643 7802 Acked Read 9 Sent Done Done false
25644 7803 Acked Read 7 Sent Done Done false
25645 7804 Acked Write 6 Sent Done Done false
25646 7805 Acked Write 5 Sent Done Done false
25647 7806 Acked Read 10 Sent Done Done false
25648 7807 Acked Read 10 Sent Done Done false
25649 7808 Acked Flush 0 Sent Done Done false
25650 7809 Acked Write 3 Sent Done Done false
25651 7810 Acked Read 3 Sent Done Done false
25652 7811 Acked Read 5 Sent Done Done false
25653 7812 Acked Read 7 Sent Done Done false
25654 7813 Acked Write 9 Sent Done Done false
25655 7814 Acked Read 1 Sent Done Done false
25656 7815 Acked Flush 0 Sent Done Done false
25657 7816 Acked Write 1 Sent Done Done false
25658 7817 Acked Read 8 Sent Done Done false
25659 7818 Acked Flush 0 Sent Done Done false
25660 7819 Acked Write 1 Sent Done Done false
25661 7820 Acked Write 1 Sent Done Done false
25662 7821 Acked Write 5 Sent Done Done false
25663 7822 Acked Read 2 Sent Done Done false
25664 7823 Acked Write 9 Sent Done Done false
25665 7824 Acked Write 5 Sent Done Done false
25666 7825 Acked Write 6 Sent Done Done false
25667 7826 Acked Flush 0 Sent Done Done false
25668 7827 Acked Read 10 Sent Done Done false
25669 7828 Acked Write 8 Sent Done Done false
25670 7829 Acked Flush 0 Sent Done Done false
25671 7830 Acked Read 8 Sent Done Done false
25672 7831 Acked Write 6 Sent Done Done false
25673 7832 Acked Write 5 Sent Done Done false
25674 7833 Acked Write 3 Sent Done Done false
25675 7834 Acked Read 3 Sent Done Done false
25676 7835 Acked Write 8 Sent Done Done false
25677 7836 Acked Write 8 Sent Done Done false
25678 7837 Acked Read 7 Sent Done Done false
25679 7838 Acked Flush 0 Sent Done Done false
25680 7839 Acked Write 3 Sent Done Done false
25681 7840 Acked Read 9 Sent Done Done false
25682 7841 Acked Read 6 Sent Done Done false
25683 7842 Acked Flush 0 Sent Done Done false
25684 7843 Acked Write 6 Sent Done Done false
25685 7844 Acked Read 6 Sent Done Done false
25686 7845 Acked Read 9 Sent Done Done false
25687 7846 Acked Write 5 Sent Done Done false
25688 7847 Acked Read 6 Sent Done Done false
25689 7848 Acked Read 1 Sent Done Done false
25690 7849 Acked Read 7 Sent Done Done false
25691 7850 Acked Write 5 Sent Done Done false
25692 7851 Acked Read 4 Sent Done Done false
25693 7852 Acked Read 2 Sent Done Done false
25694 7853 Acked Read 2 Sent Done Done false
25695 7854 Acked Read 3 Sent Done Done false
25696 7855 Acked Flush 0 Sent Done Done false
25697 7856 Acked Read 8 Sent Done Done false
25698 7857 Acked Flush 0 Sent Done Done false
25699 7858 Acked Write 5 Sent Done Done false
25700 7859 Acked Flush 0 Sent Done Done false
25701 7860 Acked Read 1 Sent Done Done false
25702 7861 Acked Read 10 Sent Done Done false
25703 7862 Acked Flush 0 Sent Done Done false
25704 7863 Acked Flush 0 Sent Done Done false
25705 7864 Acked Read 4 Sent Done Done false
25706 7865 Acked Write 4 Sent Done Done false
25707 7866 Acked Read 3 Sent Done Done false
25708 7867 Acked Read 2 Sent Done Done false
25709 7868 Acked Read 7 Sent Done Done false
25710 7869 Acked Write 10 Sent Done Done false
25711 7870 Acked Flush 0 Sent Done Done false
25712 7871 Acked Flush 0 Sent Done Done false
25713 7872 Acked Write 3 Sent Done Done false
25714 7873 Acked Read 7 Sent Done Done false
25715 7874 Acked Read 7 Sent Done Done false
25716 7875 Acked Write 3 Sent Done Done false
25717 7876 Acked Flush 0 Sent Done Done false
25718 7877 Acked Write 9 Sent Done Done false
25719 7878 Acked Write 3 Sent Done Done false
25720 7879 Acked Read 1 Sent Done Done false
25721 7880 Acked Read 7 Sent Done Done false
25722 7881 Acked Write 4 Sent Done Done false
25723 7882 Acked Read 6 Sent Done Done false
25724 7883 Acked Read 4 Sent Done Done false
25725 7884 Acked Read 10 Sent Done Done false
25726 7885 Acked Read 5 Sent Done Done false
25727 7886 Acked Write 6 Sent Done Done false
25728 7887 Acked Flush 0 Sent Done Done false
25729 7888 Acked Read 7 Sent Done Done false
25730 7889 Acked Read 4 Sent Done Done false
25731 7890 Acked Read 4 Sent Done Done false
25732 7891 Acked Flush 0 Sent Done Done false
25733 7892 Acked Read 1 Sent Done Done false
25734 7893 Acked Write 1 Sent Done Done false
25735 7894 Acked Read 8 Sent Done Done false
25736 7895 Acked Read 5 Sent Done Done false
25737 7896 Acked Write 1 Sent Done Done false
25738 7897 Acked Read 8 Sent Done Done false
25739 7898 Acked Read 3 Sent Done Done false
25740 7899 Acked Write 10 Sent Done Done false
25741 7900 Acked Read 3 Sent Done Done false
25742 7901 Acked Read 2 Sent Done Done false
25743 7902 Acked Write 6 Sent Done Done false
25744 7903 Acked Write 5 Sent Done Done false
25745 7904 Acked Write 1 Sent Done Done false
25746 7905 Acked Read 4 Sent Done Done false
25747 7906 Acked Read 4 Sent Done Done false
25748 7907 Acked Read 1 Sent Done Done false
25749 7908 Acked Write 4 Sent Done Done false
25750 7909 Acked Read 5 Sent Done Done false
25751 7910 Acked Read 6 Sent Done Done false
25752 7911 Acked Read 4 Sent Done Done false
25753 7912 Acked Write 8 Sent Done Done false
25754 7913 Acked Write 4 Sent Done Done false
25755 7914 Acked Write 8 Sent Done Done false
25756 7915 Acked Read 10 Sent Done Done false
25757 7916 Acked Read 7 Sent Done Done false
25758 7917 Acked Write 5 Sent Done Done false
25759 7918 Acked Read 7 Sent Done Done false
25760 7919 Acked Flush 0 Sent Done Done false
25761 7920 Acked Read 2 Sent Done Done false
25762 7921 Acked Write 4 Sent Done Done false
25763 7922 Acked Read 8 Sent Done Done false
25764 7923 Acked Write 10 Sent Done Done false
25765 7924 Acked Read 10 Sent Done Done false
25766 7925 Acked Write 10 Sent Done Done false
25767 7926 Acked Write 8 Sent Done Done false
25768 7927 Acked Read 7 Sent Done Done false
25769 7928 Acked Read 5 Sent Done Done false
25770 7929 Acked Read 7 Sent Done Done false
25771 7930 Acked Write 7 Sent Done Done false
25772 7931 Acked Write 7 Sent Done Done false
25773 7932 Acked Write 8 Sent Done Done false
25774 7933 Acked Write 3 Sent Done Done false
25775 7934 Acked Write 1 Sent Done Done false
25776 7935 Acked Read 3 Sent Done Done false
25777 7936 Acked Write 9 Sent Done Done false
25778 7937 Acked Write 8 Sent Done Done false
25779 7938 Acked Read 6 Sent Done Done false
25780 7939 Acked Read 2 Sent Done Done false
25781 7940 Acked Read 2 Sent Done Done false
25782 7941 Acked Write 10 Sent Done Done false
25783 7942 Acked Read 10 Sent Done Done false
25784 7943 Acked Write 6 Sent Done Done false
25785 7944 Acked Read 10 Sent Done Done false
25786 7945 Acked Flush 0 Sent Done Done false
25787 STATES DS:0 DS:1 DS:2 TOTAL
25788 Sent 302 0 0 302
25789 Done 0 302 302 604
25790 Skipped 0 0 0 0
25791 Error 0 0 0 0
25792 Last Flush: 7643 7945 7945
25793 Downstairs last five completed: 7643 7642 7641 7640 7639
25794 Upstairs last five completed: 7945 7944 7943 7942 7941
25795 CLIENT: Up:0 ds:302 act:2
257962024-12-09T15:13:14.570ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
257972024-12-09T15:13:14.570ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63068) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
257982024-12-09T15:13:14.571ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
257992024-12-09T15:13:14.572ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
258002024-12-09T15:13:14.572ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
258012024-12-09T15:13:14.572ZINFOcrucible: send last flush ID to this DS: 7643 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
258022024-12-09T15:13:14.572ZINFOcrucible: Replied this last flush ID: 7643 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
258032024-12-09T15:13:14.572ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
258042024-12-09T15:13:14.572ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
258052024-12-09T15:13:14.572ZINFOcrucible: [0] client re-new 302 jobs since flush 7643 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
258062024-12-09T15:13:14.573ZINFOcrucible: [0] Replayed 302 jobs since flush: 7643 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
25807 ----------------------------------------------------------------
25808 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25809 Downstairs last five completed: 7945 7944 7943 7942 7941
25810 Upstairs last five completed: 7945 7944 7943 7942 7941
25811 CLIENT: Up:0 ds:0 act:3
25812 Replay: All jobs finished
25813 Main task has work to do, go find it
25814 got dsc Stop(2)
25815 stop 2
25816 [8830] Got stop action so:false kr:false
25817 Dec 09 15:13:16.813 INFO request completed, latency_us: 354, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 560ea301-cb8b-481b-b098-56d3a4d5fd52, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
25818 Dec 09 15:13:16.813{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:13:16.813929808Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
25819 INFO request completed, latency_us: 208{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:13:16.814001625Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
25820 , response_code: 200, uri: /state/cid/2, method: GET, req_id: cb286dce-2fe6-468f-9a82-74e8ec5b191a, remote_addr: 127.{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:13:16.814071723Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
25821 0.0.1:47510, local_addr: 127.0.0.1:{"msg":"ds_transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:13:16.814107016Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
25822 9998
258232024-12-09T15:13:16.814ZINFOcrucible: [2] Transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
258242024-12-09T15:13:16.814ZINFOcrucible: client task is exiting = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
25825 [8830] Exited with: ExitStatus(unix_wait_status(9))
25826 [8830][2] reports Exit
25827 Dec 09 15:13:20.816 INFO request completed, latency_us: 221, response_code: 200, uri: /state/cid/2, method: GET, req_id: 875d784d-8a83-467c-8999-aef015d8d1de, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
25828 001/300 Write block 1560 len 8192 data: 7 9
25829 002/300 Read block 608 len 36864
25830 003/300 Read block 1904 len 12288
25831 004/300 Write block 3175 len 20480 data: 7 8 9 8 8
25832 005/300 Read block 1412 len 32768
25833 006/300 Flush
25834 007/300 Write block 794 len 12288 data: 4 5 6
25835 008/300 Flush
25836 009/300 Write block 1544 len 24576 data: 7 7 6 7 6 4
25837 010/300 Write block 2650 len 32768 data: 6 5 5 6 8 9 8 5
25838 011/300 Read block 4712 len 20480
25839 012/300 Write block 2687 len 32768 data: 4 4 3 3 3 3 5 5
25840 013/300 Read block 4522 len 12288
25841 014/300 Flush
25842 015/300 Write block 2880 len 16384 data: 4 3 3 4
25843 016/300 Read block 1693 len 36864
25844 017/300 Read block 4165 len 40960
25845 018/300 Read block 3736 len 32768
25846 019/300 Read block 4096 len 36864
25847 020/300 Read block 1770 len 28672
25848 021/300 Flush
25849 022/300 Read block 3138 len 36864
25850 023/300 Write block 3245 len 8192 data: 7 6
25851 024/300 Write block 2157 len 16384 data: 4 3 3 3
25852 025/300 Read block 2881 len 20480
25853 026/300 Read block 2152 len 24576
25854 027/300 Read block 2499 len 32768
25855 028/300 Read block 569 len 36864
25856 029/300 Read block 4596 len 4096
25857 030/300 Read block 427 len 20480
25858 031/300 Read block 1664 len 12288
25859 032/300 Read block 3202 len 12288
25860 033/300 Flush
25861 034/300 Read block 3796 len 24576
25862 035/300 Read block 312 len 8192
25863 036/300 Write block 1394 len 40960 data: 6 6 6 5 4 6 6 4 4 4
25864 037/300 Write block 3818 len 20480 data: 6 5 5 6 6
25865 038/300 Read block 1986 len 24576
25866 039/300 Write block 3474 len 32768 data: 8 9 9 8 8 7 6 5
25867 040/300 Write block 4596 len 32768 data: 5 6 6 4 5 7 6 5
25868 041/300 Read block 2641 len 4096
25869 042/300 Flush
25870 043/300 Read block 3934 len 20480
25871 044/300 Write block 1795 len 12288 data: 10 9 9
25872 045/300 Read block 1070 len 12288
25873 046/300 Write block 975 len 16384 data: 4 4 3 3
25874 047/300 Write block 659 len 28672 data: 3 4 8 8 8 9 8
25875 048/300 Write block 1463 len 40960 data: 4 3 2 3 4 6 6 8 7 7
25876 049/300 Flush
25877 050/300 Read block 877 len 4096
25878 051/300 Read block 1513 len 24576
25879 052/300 Write block 4126 len 36864 data: 6 5 5 5 4 5 4 5 3
25880 053/300 Read block 4790 len 8192
25881 054/300 Read block 848 len 36864
25882 055/300 Read block 3275 len 40960
25883 056/300 Flush
25884 057/300 Read block 1086 len 20480
25885 058/300 Read block 1078 len 36864
25886 059/300 Read block 4675 len 32768
25887 060/300 Read block 2582 len 4096
25888 061/300 Flush
25889 062/300 Read block 3012 len 20480
25890 063/300 Read block 3514 len 40960
25891 064/300 Read block 3900 len 20480
25892 065/300 Flush
25893 066/300 Write block 2199 len 32768 data: 6 7 8 8 7 3 3 3
25894 067/300 Write block 1375 len 28672 data: 7 6 7 6 5 3 3
25895 068/300 Write block 1514 len 12288 data: 8 6 8
25896 069/300 Write block 4546 len 32768 data: 3 4 3 5 6 6 6 7
25897 070/300 Read block 1410 len 24576
25898 071/300 Read block 861 len 40960
25899 072/300 Read block 4204 len 32768
25900 073/300 Write block 4139 len 8192 data: 5 4
25901 074/300 Read block 3346 len 20480
25902 075/300 Write block 2913 len 20480 data: 4 5 6 6 6
25903 076/300 Write block 470 len 12288 data: 4 4 5
25904 077/300 Read block 1127 len 28672
25905 078/300 Flush
25906 079/300 Read block 2582 len 40960
25907 080/300 Write block 1850 len 36864 data: 4 4 5 5 5 6 5 4 5
25908 081/300 Write block 1239 len 4096 data: 4
25909 082/300 Read block 143 len 36864
25910 083/300 Write block 4299 len 36864 data: 5 5 4 3 3 3 3 2 3
25911 084/300 Read block 4307 len 20480
25912 085/300 Write block 1737 len 4096 data: 6
25913 086/300 Read block 4230 len 36864
25914 087/300 Read block 194 len 20480
25915 088/300 Flush
25916 089/300 Read block 1269 len 40960
25917 090/300 Write block 877 len 4096 data: 5
25918 091/300 Write block 4015 len 8192 data: 8 9
25919 092/300 Write block 2553 len 12288 data: 4 4 5
25920 093/300 Read block 4682 len 8192
25921 094/300 Read block 3986 len 20480
25922 095/300 Read block 667 len 8192
25923 096/300 Write block 1143 len 8192 data: 6 7
25924 097/300 Read block 1277 len 16384
25925 098/300 Write block 2004 len 12288 data: 8 10 8
25926 099/300 Read block 1840 len 20480
25927 100/300 Read block 3031 len 40960
25928 101/300 Write block 121 len 8192 data: 4 5
25929 102/300 Read block 621 len 40960
25930 103/300 Write block 4644 len 20480 data: 6 7 8 9 9
25931 104/300 Read block 3850 len 20480
25932 105/300 Read block 3919 len 20480
25933 106/300 Read block 1656 len 16384
25934 107/300 Read block 1951 len 12288
25935 108/300 Write block 463 len 28672 data: 4 3 3 4 6 4 4
25936 109/300 Read block 1853 len 4096
25937 110/300 Read block 1436 len 28672
25938 111/300 Read block 2182 len 16384
25939 112/300 Read block 681 len 16384
25940 113/300 Read block 3003 len 20480
25941 114/300 Read block 2643 len 40960
25942 115/300 Write block 4766 len 36864 data: 6 5 5 4 3 4 3 3 4
25943 116/300 Read block 2768 len 32768
25944 117/300 Read block 599 len 8192
25945 118/300 Write block 1637 len 20480 data: 3 3 3 3 3
25946 119/300 Write block 134 len 24576 data: 7 7 7 7 7 7
25947 120/300 Write block 4312 len 12288 data: 5 5 6
25948 121/300 Flush
25949 122/300 Write block 1151 len 16384 data: 6 4 3 3
25950 123/300 Write block 4465 len 40960 data: 7 6 4 7 7 8 7 7 7 7
25951 124/300 Read block 2184 len 20480
25952 125/300 Read block 4824 len 20480
25953 126/300 Write block 2054 len 8192 data: 6 6
25954 127/300 Write block 4775 len 16384 data: 4 5 5 4
25955 128/300 Read block 4385 len 24576
25956 129/300 Read block 1738 len 28672
25957 130/300 Write block 2533 len 12288 data: 8 8 8
25958 131/300 Read block 4881 len 20480
25959 132/300 Read block 1276 len 24576
25960 133/300 Write block 1004 len 16384 data: 6 6 7 5
25961 134/300 Flush
25962 135/300 Write block 1058 len 36864 data: 5 5 6 6 5 4 3 3 2
25963 136/300 Write block 4038 len 4096 data: 4
25964 137/300 Read block 2192 len 40960
25965 138/300 Write block 3742 len 12288 data: 3 3 5
25966 139/300 Read block 4165 len 40960
25967 140/300 Write block 4145 len 8192 data: 2 2
25968 141/300 Read block 643 len 40960
25969 142/300 Write block 4043 len 16384 data: 2 3 4 6
25970 143/300 Write block 1328 len 36864 data: 3 3 3 4 6 5 3 8 11
25971 144/300 Write block 582 len 36864 data: 7 6 5 4 3 3 3 3 3
25972 145/300 Write block 4876 len 40960 data: 3 3 2 3 3 3 2 2 2 2
25973 146/300 Read block 4631 len 36864
25974 147/300 Write block 4198 len 4096 data: 3
25975 148/300 Read block 2933 len 12288
25976 149/300 Read block 3830 len 8192
25977 150/300 Read block 444 len 20480
25978 151/300 Write block 3271 len 20480 data: 4 6 7 6 6
25979 152/300 Read block 1220 len 8192
25980 153/300 Write block 2105 len 32768 data: 7 7 8 7 6 5 5 7
25981 154/300 Read block 1089 len 4096
25982 155/300 Read block 4343 len 8192
25983 156/300 Read block 3761 len 8192
25984 157/300 Read block 3709 len 36864
25985 158/300 Write block 3981 len 36864 data: 5 6 5 5 7 7 5 3 3
25986 159/300 Write block 2311 len 8192 data: 3 4
25987 160/300 Read block 655 len 16384
25988 161/300 Flush
25989 162/300 Read block 451 len 36864
25990 163/300 Write block 3630 len 40960 data: 3 3 3 3 4 3 3 3 3 2
25991 164/300 Read block 1930 len 4096
25992 165/300 Write block 838 len 40960 data: 5 3 3 4 4 3 3 4 3 3
25993 166/300 Read block 692 len 32768
25994 167/300 Flush
25995 168/300 Read block 3073 len 16384
25996 169/300 Read block 3108 len 36864
25997 170/300 Read block 2574 len 4096
25998 171/300 Read block 117 len 16384
25999 172/300 Write block 3049 len 32768 data: 3 3 2 3 3 3 5 5
26000 173/300 Write block 2375 len 4096 data: 4
26001 174/300 Write block 2392 len 4096 data: 6
26002 175/300 Read block 4799 len 24576
26003 176/300 Read block 4573 len 8192
26004 177/300 Write block 1104 len 16384 data: 6 6 8 7
26005 178/300 Read block 1419 len 16384
26006 179/300 Read block 1431 len 20480
26007 180/300 Write block 456 len 20480 data: 4 5 5 5 6
26008 181/300 Flush
26009 182/300 Write block 3632 len 36864 data: 4 4 5 4 4 4 4 3 3
26010 183/300 Read block 3367 len 20480
26011 184/300 Write block 3818 len 8192 data: 7 6
26012 185/300 Read block 1785 len 20480
26013 186/300 Write block 2037 len 12288 data: 5 6 5
26014 187/300 Read block 2389 len 4096
26015 188/300 Read block 383 len 40960
26016 189/300 Write block 3357 len 32768 data: 5 3 3 4 6 6 6 5
26017 190/300 Read block 4613 len 20480
26018 191/300 Write block 148 len 12288 data: 5 8 8
26019 192/300 Read block 1911 len 40960
26020 193/300 Write block 4092 len 16384 data: 7 5 6 8
26021 194/300 Read block 3595 len 12288
26022 195/300 Flush
26023 196/300 Read block 1787 len 12288
26024 197/300 Read block 4506 len 28672
26025 198/300 Read block 4433 len 20480
26026 199/300 Write block 2325 len 24576 data: 3 3 3 3 3 3
26027 200/300 Read block 4129 len 4096
26028 201/300 Read block 3877 len 8192
26029 202/300 Read block 4357 len 16384
26030 203/300 Read block 391 len 36864
26031 204/300 Write block 4544 len 36864 data: 3 4 4 5 4 6 7 7 7
26032 205/300 Flush
26033 206/300 Write block 913 len 40960 data: 6 6 6 6 5 6 4 4 6 6
26034 207/300 Read block 3666 len 24576
26035 208/300 Write block 4267 len 12288 data: 4 3 2
26036 209/300 Write block 2127 len 32768 data: 5 7 8 8 10 9 8 9
26037 210/300 Write block 3596 len 12288 data: 7 7 6
26038 211/300 Write block 3657 len 24576 data: 5 4 4 4 4 6
26039 212/300 Write block 4436 len 4096 data: 6
26040 213/300 Read block 3831 len 16384
26041 214/300 Read block 4699 len 8192
26042 215/300 Read block 3105 len 40960
26043 216/300 Write block 1246 len 40960 data: 6 7 6 6 6 4 4 3 3 3
26044 217/300 Write block 1606 len 8192 data: 4 4
26045 218/300 Read block 1093 len 8192
26046 219/300 Write block 3665 len 28672 data: 5 5 5 5 6 3 4
26047 220/300 Write block 3862 len 12288 data: 3 3 4
26048 221/300 Read block 3824 len 8192
26049 222/300 Flush
26050 223/300 Read block 1419 len 4096
26051 224/300 Write block 4010 len 4096 data: 7
26052 225/300 Write block 1600 len 28672 data: 5 6 6 6 3 4 5
26053 226/300 Read block 1066 len 36864
26054 227/300 Read block 4188 len 12288
26055 228/300 Read block 3676 len 28672
26056 229/300 Read block 779 len 12288
26057 230/300 Write block 4367 len 16384 data: 3 2 2 3
26058 231/300 Read block 1645 len 8192
26059 232/300 Write block 4297 len 28672 data: 5 5 6 6 5 4 4
26060 233/300 Read block 2755 len 20480
26061 234/300 Read block 4338 len 24576
26062 235/300 Read block 79 len 36864
26063 236/300 Write block 28 len 12288 data: 4 4 5
26064 237/300 Write block 2032 len 16384 data: 7 6 6 6
26065 238/300 Write block 3187 len 16384 data: 4 4 4 3
26066 239/300 Write block 4654 len 4096 data: 9
26067 240/300 Read block 2378 len 36864
26068 241/300 Read block 3602 len 32768
26069 242/300 Write block 1625 len 4096 data: 5
26070 243/300 Read block 94 len 24576
26071 244/300 Read block 4946 len 28672
26072 245/300 Read block 4343 len 24576
26073 246/300 Write block 2129 len 16384 data: 9 9 11 10
26074 247/300 Read block 8 len 16384
26075 248/300 Write block 1572 len 16384 data: 4 3 4 4
26076 249/300 Read block 2821 len 4096
26077 250/300 Read block 2475 len 12288
26078 251/300 Write block 2555 len 4096 data: 6
26079 252/300 Flush
26080 253/300 Write block 3724 len 4096 data: 6
26081 254/300 Read block 4154 len 16384
26082 255/300 Read block 3501 len 4096
26083 256/300 Read block 4726 len 28672
26084 257/300 Read block 1980 len 4096
26085 258/300 Write block 485 len 4096 data: 6
26086 259/300 Write block 571 len 16384 data: 4 5 6 6
26087 260/300 Write block 2682 len 24576 data: 4 3 5 5 5 5
26088 261/300 Read block 2585 len 28672
26089 262/300 Write block 4653 len 12288 data: 7 10 9
26090 263/300 Read block 1922 len 12288
26091 264/300 Read block 1454 len 12288
26092 265/300 Read block 3175 len 20480
26093 266/300 Read block 4269 len 4096
26094 267/300 Read block 4078 len 4096
26095 268/300 Write block 4704 len 32768 data: 6 6 7 8 9 8 6 6
26096 269/300 Read block 4610 len 32768
26097 270/300 Read block 2800 len 16384
26098 271/300 Write block 1797 len 24576 data: 10 8 6 5 4 5
26099 272/300 Read block 391 len 8192
26100 273/300 Write block 4497 len 24576 data: 5 5 7 7 7 9
26101 274/300 Write block 4626 len 4096 data: 7
26102 275/300 Write block 971 len 24576 data: 3 4 3 4 5 5
26103 276/300 Write block 3289 len 4096 data: 7
26104 277/300 Read block 4249 len 20480
26105 278/300 Read block 3782 len 36864
26106 279/300 Read block 4990 len 8192
26107 280/300 Read block 36 len 4096
26108 281/300 Write block 2016 len 12288 data: 8 8 9
26109 282/300 Read block 1973 len 40960
26110 283/300 Read block 2193 len 8192
26111 284/300 Read block 2812 len 16384
26112 285/300 Read block 1674 len 16384
26113 286/300 Read block 3479 len 28672
26114 287/300 Read block 595 len 20480
26115 288/300 Read block 1644 len 36864
26116 289/300 Read block 446 len 4096
26117 290/300 Read block 1576 len 12288
26118 291/300 Read block 1084 len 4096
26119 292/300 Write block 1341 len 16384 data: 8 5 5 4
26120 293/300 Read block 1652 len 20480
26121 294/300 Flush
26122 295/300 Write block 1191 len 20480 data: 6 6 7 8 6
26123 296/300 Read block 3764 len 40960
26124 297/300 Read block 4269 len 4096
26125 298/300 Read block 4843 len 8192
26126 299/300 Read block 4763 len 24576
26127 300/300 Read block 3427 len 16384
26128 301/300 Flush
26129 Main task has work to do, go find it
26130 got dsc Start(2)
26131 start 2
26132 Dec 09 15:13:21.058 INFO request completed, latency_us: 236, response_code: 204, uri: /start/cid/2, method: POST, req_id: c3422b42-2c65-4d1f-b1c4-27d179a4635b, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
26133 Replay: started 2, returned:Ok(())
26134 ----------------------------------------------------------------
26135 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26136 States: Active Active Offline
26137 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26138 7946 Acked Write 2 Done Done Sent false
26139 7947 Acked Read 9 Done Done Sent false
26140 7948 Acked Read 3 Done Done Sent false
26141 7949 Acked Write 5 Done Done Sent false
26142 7950 Acked Read 8 Done Done Sent false
26143 7951 Acked Flush 0 Done Done Sent false
26144 7952 Acked Write 3 Done Done Sent false
26145 7953 Acked Flush 0 Done Done Sent false
26146 7954 Acked Write 6 Done Done Sent false
26147 7955 Acked Write 8 Done Done Sent false
26148 7956 Acked Read 5 Done Done Sent false
26149 7957 Acked Write 8 Done Done Sent false
26150 7958 Acked Read 3 Done Done Sent false
26151 7959 Acked Flush 0 Done Done Sent false
26152 7960 Acked Write 4 Done Done Sent false
26153 7961 Acked Read 9 Done Done Sent false
26154 7962 Acked Read 10 Done Done Sent false
26155 7963 Acked Read 8 Done Done Sent false
26156 7964 Acked Read 9 Done Done Sent false
26157 7965 Acked Read 7 Done Done Sent false
26158 7966 Acked Flush 0 Done Done Sent false
26159 7967 Acked Read 9 Done Done Sent false
26160 7968 Acked Write 2 Done Done Sent false
26161 7969 Acked Write 4 Done Done Sent false
26162 7970 Acked Read 5 Done Done Sent false
26163 7971 Acked Read 6 Done Done Sent false
26164 7972 Acked Read 8 Done Done Sent false
26165 7973 Acked Read 9 Done Done Sent false
26166 7974 Acked Read 1 Done Done Sent false
26167 7975 Acked Read 5 Done Done Sent false
26168 7976 Acked Read 3 Done Done Sent false
26169 7977 Acked Read 3 Done Done Sent false
26170 7978 Acked Flush 0 Done Done Sent false
26171 7979 Acked Read 6 Done Done Sent false
26172 7980 Acked Read 2 Done Done Sent false
26173 7981 Acked Write 10 Done Done Sent false
26174 7982 Acked Write 5 Done Done Sent false
26175 7983 Acked Read 6 Done Done Sent false
26176 7984 Acked Write 8 Done Done Sent false
26177 7985 Acked Write 8 Done Done Sent false
26178 7986 Acked Read 1 Done Done Sent false
26179 7987 Acked Flush 0 Done Done Sent false
26180 7988 Acked Read 5 Done Done Sent false
26181 7989 Acked Write 3 Done Done Sent false
26182 7990 Acked Read 3 Done Done Sent false
26183 7991 Acked Write 4 Done Done Sent false
26184 7992 Acked Write 7 Done Done Sent false
26185 7993 Acked Write 10 Done Done Sent false
26186 7994 Acked Flush 0 Done Done Sent false
26187 7995 Acked Read 1 Done Done Sent false
26188 7996 Acked Read 6 Done Done Sent false
26189 7997 Acked Write 9 Done Done Sent false
26190 7998 Acked Read 2 Done Done Sent false
26191 7999 Acked Read 9 Done Done Sent false
26192 8000 Acked Read 10 Done Done Sent false
26193 8001 Acked Flush 0 Done Done Sent false
26194 8002 Acked Read 5 Done Done Sent false
26195 8003 Acked Read 9 Done Done Sent false
26196 8004 Acked Read 8 Done Done Sent false
26197 8005 Acked Read 1 Done Done Sent false
26198 8006 Acked Flush 0 Done Done Sent false
26199 8007 Acked Read 5 Done Done Sent false
26200 8008 Acked Read 10 Done Done Sent false
26201 8009 Acked Read 5 Done Done Sent false
26202 8010 Acked Flush 0 Done Done Sent false
26203 8011 Acked Write 8 Done Done Sent false
26204 8012 Acked Write 7 Done Done Sent false
26205 8013 Acked Write 3 Done Done Sent false
26206 8014 Acked Write 8 Done Done Sent false
26207 8015 Acked Read 6 Done Done Sent false
26208 8016 Acked Read 10 Done Done Sent false
26209 8017 Acked Read 8 Done Done Sent false
26210 8018 Acked Write 2 Done Done Sent false
26211 8019 Acked Read 5 Done Done Sent false
26212 8020 Acked Write 5 Done Done Sent false
26213 8021 Acked Write 3 Done Done Sent false
26214 8022 Acked Read 7 Done Done Sent false
26215 8023 Acked Flush 0 Done Done Sent false
26216 8024 Acked Read 10 Done Done Sent false
26217 8025 Acked Write 9 Done Done Sent false
26218 8026 Acked Write 1 Done Done Sent false
26219 8027 Acked Read 9 Done Done Sent false
26220 8028 Acked Write 9 Done Done Sent false
26221 8029 Acked Read 5 Done Done Sent false
26222 8030 Acked Write 1 Done Done Sent false
26223 8031 Acked Read 9 Done Done Sent false
26224 8032 Acked Read 5 Done Done Sent false
26225 8033 Acked Flush 0 Done Done Sent false
26226 8034 Acked Read 10 Done Done Sent false
26227 8035 Acked Write 1 Done Done Sent false
26228 8036 Acked Write 2 Done Done Sent false
26229 8037 Acked Write 3 Done Done Sent false
26230 8038 Acked Read 2 Done Done Sent false
26231 8039 Acked Read 5 Done Done Sent false
26232 8040 Acked Read 2 Done Done Sent false
26233 8041 Acked Write 2 Done Done Sent false
26234 8042 Acked Read 4 Done Done Sent false
26235 8043 Acked Write 3 Done Done Sent false
26236 8044 Acked Read 5 Done Done Sent false
26237 8045 Acked Read 10 Done Done Sent false
26238 8046 Acked Write 2 Done Done Sent false
26239 8047 Acked Read 10 Done Done Sent false
26240 8048 Acked Write 5 Done Done Sent false
26241 8049 Acked Read 5 Done Done Sent false
26242 8050 Acked Read 5 Done Done Sent false
26243 8051 Acked Read 4 Done Done Sent false
26244 8052 Acked Read 3 Done Done Sent false
26245 8053 Acked Write 7 Done Done Sent false
26246 8054 Acked Read 1 Done Done Sent false
26247 8055 Acked Read 7 Done Done Sent false
26248 8056 Acked Read 4 Done Done Sent false
26249 8057 Acked Read 4 Done Done Sent false
26250 8058 Acked Read 5 Done Done Sent false
26251 8059 Acked Read 10 Done Done Sent false
26252 8060 Acked Write 9 Done Done Sent false
26253 8061 Acked Read 8 Done Done Sent false
26254 8062 Acked Read 2 Done Done Sent false
26255 8063 Acked Write 5 Done Done Sent false
26256 8064 Acked Write 6 Done Done Sent false
26257 8065 Acked Write 3 Done Done Sent false
26258 8066 Acked Flush 0 Done Done Sent false
26259 8067 Acked Write 4 Done Done Sent false
26260 8068 Acked Write 10 Done Done Sent false
26261 8069 Acked Read 5 Done Done Sent false
26262 8070 Acked Read 5 Done Done Sent false
26263 8071 Acked Write 2 Done Done Sent false
26264 8072 Acked Write 4 Done Done Sent false
26265 8073 Acked Read 6 Done Done Sent false
26266 8074 Acked Read 7 Done Done Sent false
26267 8075 Acked Write 3 Done Done Sent false
26268 8076 Acked Read 5 Done Done Sent false
26269 8077 Acked Read 6 Done Done Sent false
26270 8078 Acked Write 4 Done Done Sent false
26271 8079 Acked Flush 0 Done Done Sent false
26272 8080 Acked Write 9 Done Done Sent false
26273 8081 Acked Write 1 Done Done Sent false
26274 8082 Acked Read 10 Done Done Sent false
26275 8083 Acked Write 3 Done Done Sent false
26276 8084 Acked Read 10 Done Done Sent false
26277 8085 Acked Write 2 Done Done Sent false
26278 8086 Acked Read 10 Done Done Sent false
26279 8087 Acked Write 4 Done Done Sent false
26280 8088 Acked Write 9 Done Done Sent false
26281 8089 Acked Write 9 Done Done Sent false
26282 8090 Acked Write 10 Done Done Sent false
26283 8091 Acked Read 9 Done Done Sent false
26284 8092 Acked Write 1 Done Done Sent false
26285 8093 Acked Read 3 Done Done Sent false
26286 8094 Acked Read 2 Done Done Sent false
26287 8095 Acked Read 5 Done Done Sent false
26288 8096 Acked Write 5 Done Done Sent false
26289 8097 Acked Read 2 Done Done Sent false
26290 8098 Acked Write 8 Done Done Sent false
26291 8099 Acked Read 1 Done Done Sent false
26292 8100 Acked Read 2 Done Done Sent false
26293 8101 Acked Read 2 Done Done Sent false
26294 8102 Acked Read 9 Done Done Sent false
26295 8103 Acked Write 9 Done Done Sent false
26296 8104 Acked Write 2 Done Done Sent false
26297 8105 Acked Read 4 Done Done Sent false
26298 8106 Acked Flush 0 Done Done Sent false
26299 8107 Acked Read 9 Done Done Sent false
26300 8108 Acked Write 10 Done Done Sent false
26301 8109 Acked Read 1 Done Done Sent false
26302 8110 Acked Write 10 Done Done Sent false
26303 8111 Acked Read 8 Done Done Sent false
26304 8112 Acked Flush 0 Done Done Sent false
26305 8113 Acked Read 4 Done Done Sent false
26306 8114 Acked Read 9 Done Done Sent false
26307 8115 Acked Read 1 Done Done Sent false
26308 8116 Acked Read 4 Done Done Sent false
26309 8117 Acked Write 8 Done Done Sent false
26310 8118 Acked Write 1 Done Done Sent false
26311 8119 Acked Write 1 Done Done Sent false
26312 8120 Acked Read 6 Done Done Sent false
26313 8121 Acked Read 2 Done Done Sent false
26314 8122 Acked Write 4 Done Done Sent false
26315 8123 Acked Read 4 Done Done Sent false
26316 8124 Acked Read 5 Done Done Sent false
26317 8125 Acked Write 5 Done Done Sent false
26318 8126 Acked Flush 0 Done Done Sent false
26319 8127 Acked Write 9 Done Done Sent false
26320 8128 Acked Read 5 Done Done Sent false
26321 8129 Acked Write 2 Done Done Sent false
26322 8130 Acked Read 5 Done Done Sent false
26323 8131 Acked Write 3 Done Done Sent false
26324 8132 Acked Read 1 Done Done Sent false
26325 8133 Acked Read 10 Done Done Sent false
26326 8134 Acked Write 8 Done Done Sent false
26327 8135 Acked Read 5 Done Done Sent false
26328 8136 Acked Write 3 Done Done Sent false
26329 8137 Acked Read 10 Done Done Sent false
26330 8138 Acked Write 4 Done Done Sent false
26331 8139 Acked Read 3 Done Done Sent false
26332 8140 Acked Flush 0 Done Done Sent false
26333 8141 Acked Read 3 Done Done Sent false
26334 8142 Acked Read 7 Done Done Sent false
26335 8143 Acked Read 5 Done Done Sent false
26336 8144 Acked Write 6 Done Done Sent false
26337 8145 Acked Read 1 Done Done Sent false
26338 8146 Acked Read 2 Done Done Sent false
26339 8147 Acked Read 4 Done Done Sent false
26340 8148 Acked Read 9 Done Done Sent false
26341 8149 Acked Write 9 Done Done Sent false
26342 8150 Acked Flush 0 Done Done Sent false
26343 8151 Acked Write 10 Done Done Sent false
26344 8152 Acked Read 6 Done Done Sent false
26345 8153 Acked Write 3 Done Done Sent false
26346 8154 Acked Write 8 Done Done Sent false
26347 8155 Acked Write 3 Done Done Sent false
26348 8156 Acked Write 6 Done Done Sent false
26349 8157 Acked Write 1 Done Done Sent false
26350 8158 Acked Read 4 Done Done Sent false
26351 8159 Acked Read 2 Done Done Sent false
26352 8160 Acked Read 10 Done Done Sent false
26353 8161 Acked Write 10 Done Done Sent false
26354 8162 Acked Write 2 Done Done Sent false
26355 8163 Acked Read 2 Done Done Sent false
26356 8164 Acked Write 7 Done Done Sent false
26357 8165 Acked Write 3 Done Done Sent false
26358 8166 Acked Read 2 Done Done Sent false
26359 8167 Acked Flush 0 Done Done Sent false
26360 8168 Acked Read 1 Done Done Sent false
26361 8169 Acked Write 1 Done Done Sent false
26362 8170 Acked Write 7 Done Done Sent false
26363 8171 Acked Read 9 Done Done Sent false
26364 8172 Acked Read 3 Done Done Sent false
26365 8173 Acked Read 7 Done Done Sent false
26366 8174 Acked Read 3 Done Done Sent false
26367 8175 Acked Write 4 Done Done Sent false
26368 8176 Acked Read 2 Done Done Sent false
26369 8177 Acked Write 7 Done Done Sent false
26370 8178 Acked Read 5 Done Done Sent false
26371 8179 Acked Read 6 Done Done Sent false
26372 8180 Acked Read 9 Done Done Sent false
26373 8181 Acked Write 3 Done Done Sent false
26374 8182 Acked Write 4 Done Done Sent false
26375 8183 Acked Write 4 Done Done Sent false
26376 8184 Acked Write 1 Done Done Sent false
26377 8185 Acked Read 9 Done Done Sent false
26378 8186 Acked Read 8 Done Done Sent false
26379 8187 Acked Write 1 Done Done Sent false
26380 8188 Acked Read 6 Done Done Sent false
26381 8189 Acked Read 7 Done Done Sent false
26382 8190 Acked Read 6 Done Done Sent false
26383 8191 Acked Write 4 Done Done Sent false
26384 8192 Acked Read 4 Done Done Sent false
26385 8193 Acked Write 4 Done Done Sent false
26386 8194 Acked Read 1 Done Done Sent false
26387 8195 Acked Read 3 Done Done Sent false
26388 8196 Acked Write 1 Done Done Sent false
26389 8197 Acked Flush 0 Done Done Sent false
26390 8198 Acked Write 1 Done Done Sent false
26391 8199 Acked Read 4 Done Done Sent false
26392 8200 Acked Read 1 Done Done Sent false
26393 8201 Acked Read 7 Done Done Sent false
26394 8202 Acked Read 1 Done Done Sent false
26395 8203 Acked Write 1 Done Done Sent false
26396 8204 Acked Write 4 Done Done Sent false
26397 8205 Acked Write 6 Done Done Sent false
26398 8206 Acked Read 7 Done Done Sent false
26399 8207 Acked Write 3 Done Done Sent false
26400 8208 Acked Read 3 Done Done Sent false
26401 8209 Acked Read 3 Done Done Sent false
26402 8210 Acked Read 5 Done Done Sent false
26403 8211 Acked Read 1 Done Done Sent false
26404 8212 Acked Read 1 Done Done Sent false
26405 8213 Acked Write 8 Done Done Sent false
26406 8214 Acked Read 8 Done Done Sent false
26407 8215 Acked Read 4 Done Done Sent false
26408 8216 Acked Write 6 Done Done Sent false
26409 8217 Acked Read 2 Done Done Sent false
26410 8218 Acked Write 6 Done Done Sent false
26411 8219 Acked Write 1 Done Done Sent false
26412 8220 Acked Write 6 Done Done Sent false
26413 8221 Acked Write 1 Done Done Sent false
26414 8222 Acked Read 5 Done Done Sent false
26415 8223 Acked Read 9 Done Done Sent false
26416 8224 Acked Read 2 Done Done Sent false
26417 8225 Acked Read 1 Done Done Sent false
26418 8226 Acked Write 3 Done Done Sent false
26419 8227 Acked Read 10 Done Done Sent false
26420 8228 Acked Read 2 Done Done Sent false
26421 8229 Acked Read 4 Done Done Sent false
26422 8230 Acked Read 4 Done Done Sent false
26423 8231 Acked Read 7 Done Done Sent false
26424 8232 Acked Read 5 Done Done Sent false
26425 8233 Acked Read 9 Done Done Sent false
26426 8234 Acked Read 1 Done Done Sent false
26427 8235 Acked Read 3 Done Done Sent false
26428 8236 Acked Read 1 Done Done Sent false
26429 8237 Acked Write 4 Done Done Sent false
26430 8238 Acked Read 5 Done Done Sent false
26431 8239 Acked Flush 0 Done Done Sent false
26432 8240 Acked Write 5 Done Done Sent false
26433 8241 Acked Read 10 Done Done Sent false
26434 8242 Acked Read 1 Done Done Sent false
26435 8243 Acked Read 2 Done Done Sent false
26436 8244 Acked Read 6 Done Done Sent false
26437 8245 Acked Read 4 Done Done Sent false
26438 8246 Acked Flush 0 Done Done Sent false
26439 STATES DS:0 DS:1 DS:2 TOTAL
26440 Sent 0 0 301 301
26441 Done 301 301 0 602
26442 Skipped 0 0 0 0
26443 Error 0 0 0 0
26444 Last Flush: 8246 8246 7945
26445 Downstairs last five completed: 7945 7944 7943 7942 7941
26446 Upstairs last five completed: 8246 8245 8244 8243 8242
26447 CLIENT: Up:0 ds:301 act:2
26448 [8830] got start action
26449 [8830] I am going to restart
26450 Starting downstairs at port 8830
26451 Make output file at "/tmp/dsc/downstairs-8830.txt"
26452 [8830][2] reports Starting
26453 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1225)
26454 [8830][2] reports Running
26455 ----------------------------------------------------------------
26456 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26457 States: Active Active Offline
26458 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26459 7946 Acked Write 2 Done Done Sent false
26460 7947 Acked Read 9 Done Done Sent false
26461 7948 Acked Read 3 Done Done Sent false
26462 7949 Acked Write 5 Done Done Sent false
26463 7950 Acked Read 8 Done Done Sent false
26464 7951 Acked Flush 0 Done Done Sent false
26465 7952 Acked Write 3 Done Done Sent false
26466 7953 Acked Flush 0 Done Done Sent false
26467 7954 Acked Write 6 Done Done Sent false
26468 7955 Acked Write 8 Done Done Sent false
26469 7956 Acked Read 5 Done Done Sent false
26470 7957 Acked Write 8 Done Done Sent false
26471 7958 Acked Read 3 Done Done Sent false
26472 7959 Acked Flush 0 Done Done Sent false
26473 7960 Acked Write 4 Done Done Sent false
26474 7961 Acked Read 9 Done Done Sent false
26475 7962 Acked Read 10 Done Done Sent false
26476 7963 Acked Read 8 Done Done Sent false
26477 7964 Acked Read 9 Done Done Sent false
26478 7965 Acked Read 7 Done Done Sent false
26479 7966 Acked Flush 0 Done Done Sent false
26480 7967 Acked Read 9 Done Done Sent false
26481 7968 Acked Write 2 Done Done Sent false
26482 7969 Acked Write 4 Done Done Sent false
26483 7970 Acked Read 5 Done Done Sent false
26484 7971 Acked Read 6 Done Done Sent false
26485 7972 Acked Read 8 Done Done Sent false
26486 7973 Acked Read 9 Done Done Sent false
26487 7974 Acked Read 1 Done Done Sent false
26488 7975 Acked Read 5 Done Done Sent false
26489 7976 Acked Read 3 Done Done Sent false
26490 7977 Acked Read 3 Done Done Sent false
26491 7978 Acked Flush 0 Done Done Sent false
26492 7979 Acked Read 6 Done Done Sent false
26493 7980 Acked Read 2 Done Done Sent false
26494 7981 Acked Write 10 Done Done Sent false
26495 7982 Acked Write 5 Done Done Sent false
26496 7983 Acked Read 6 Done Done Sent false
26497 7984 Acked Write 8 Done Done Sent false
26498 7985 Acked Write 8 Done Done Sent false
26499 7986 Acked Read 1 Done Done Sent false
26500 7987 Acked Flush 0 Done Done Sent false
26501 7988 Acked Read 5 Done Done Sent false
26502 7989 Acked Write 3 Done Done Sent false
26503 7990 Acked Read 3 Done Done Sent false
26504 7991 Acked Write 4 Done Done Sent false
26505 7992 Acked Write 7 Done Done Sent false
26506 7993 Acked Write 10 Done Done Sent false
26507 7994 Acked Flush 0 Done Done Sent false
26508 7995 Acked Read 1 Done Done Sent false
26509 7996 Acked Read 6 Done Done Sent false
26510 7997 Acked Write 9 Done Done Sent false
26511 7998 Acked Read 2 Done Done Sent false
26512 7999 Acked Read 9 Done Done Sent false
26513 8000 Acked Read 10 Done Done Sent false
26514 8001 Acked Flush 0 Done Done Sent false
26515 8002 Acked Read 5 Done Done Sent false
26516 8003 Acked Read 9 Done Done Sent false
26517 8004 Acked Read 8 Done Done Sent false
26518 8005 Acked Read 1 Done Done Sent false
26519 8006 Acked Flush 0 Done Done Sent false
26520 8007 Acked Read 5 Done Done Sent false
26521 8008 Acked Read 10 Done Done Sent false
26522 8009 Acked Read 5 Done Done Sent false
26523 8010 Acked Flush 0 Done Done Sent false
26524 8011 Acked Write 8 Done Done Sent false
26525 8012 Acked Write 7 Done Done Sent false
26526 8013 Acked Write 3 Done Done Sent false
26527 8014 Acked Write 8 Done Done Sent false
26528 8015 Acked Read 6 Done Done Sent false
26529 8016 Acked Read 10 Done Done Sent false
26530 8017 Acked Read 8 Done Done Sent false
26531 8018 Acked Write 2 Done Done Sent false
26532 8019 Acked Read 5 Done Done Sent false
26533 8020 Acked Write 5 Done Done Sent false
26534 8021 Acked Write 3 Done Done Sent false
26535 8022 Acked Read 7 Done Done Sent false
26536 8023 Acked Flush 0 Done Done Sent false
26537 8024 Acked Read 10 Done Done Sent false
26538 8025 Acked Write 9 Done Done Sent false
26539 8026 Acked Write 1 Done Done Sent false
26540 8027 Acked Read 9 Done Done Sent false
26541 8028 Acked Write 9 Done Done Sent false
26542 8029 Acked Read 5 Done Done Sent false
26543 8030 Acked Write 1 Done Done Sent false
26544 8031 Acked Read 9 Done Done Sent false
26545 8032 Acked Read 5 Done Done Sent false
26546 8033 Acked Flush 0 Done Done Sent false
26547 8034 Acked Read 10 Done Done Sent false
26548 8035 Acked Write 1 Done Done Sent false
26549 8036 Acked Write 2 Done Done Sent false
26550 8037 Acked Write 3 Done Done Sent false
26551 8038 Acked Read 2 Done Done Sent false
26552 8039 Acked Read 5 Done Done Sent false
26553 8040 Acked Read 2 Done Done Sent false
26554 8041 Acked Write 2 Done Done Sent false
26555 8042 Acked Read 4 Done Done Sent false
26556 8043 Acked Write 3 Done Done Sent false
26557 8044 Acked Read 5 Done Done Sent false
26558 8045 Acked Read 10 Done Done Sent false
26559 8046 Acked Write 2 Done Done Sent false
26560 8047 Acked Read 10 Done Done Sent false
26561 8048 Acked Write 5 Done Done Sent false
26562 8049 Acked Read 5 Done Done Sent false
26563 8050 Acked Read 5 Done Done Sent false
26564 8051 Acked Read 4 Done Done Sent false
26565 8052 Acked Read 3 Done Done Sent false
26566 8053 Acked Write 7 Done Done Sent false
26567 8054 Acked Read 1 Done Done Sent false
26568 8055 Acked Read 7 Done Done Sent false
26569 8056 Acked Read 4 Done Done Sent false
26570 8057 Acked Read 4 Done Done Sent false
26571 8058 Acked Read 5 Done Done Sent false
26572 8059 Acked Read 10 Done Done Sent false
26573 8060 Acked Write 9 Done Done Sent false
26574 8061 Acked Read 8 Done Done Sent false
26575 8062 Acked Read 2 Done Done Sent false
26576 8063 Acked Write 5 Done Done Sent false
26577 8064 Acked Write 6 Done Done Sent false
26578 8065 Acked Write 3 Done Done Sent false
26579 8066 Acked Flush 0 Done Done Sent false
26580 8067 Acked Write 4 Done Done Sent false
26581 8068 Acked Write 10 Done Done Sent false
26582 8069 Acked Read 5 Done Done Sent false
26583 8070 Acked Read 5 Done Done Sent false
26584 8071 Acked Write 2 Done Done Sent false
26585 8072 Acked Write 4 Done Done Sent false
26586 8073 Acked Read 6 Done Done Sent false
26587 8074 Acked Read 7 Done Done Sent false
26588 8075 Acked Write 3 Done Done Sent false
26589 8076 Acked Read 5 Done Done Sent false
26590 8077 Acked Read 6 Done Done Sent false
26591 8078 Acked Write 4 Done Done Sent false
26592 8079 Acked Flush 0 Done Done Sent false
26593 8080 Acked Write 9 Done Done Sent false
26594 8081 Acked Write 1 Done Done Sent false
26595 8082 Acked Read 10 Done Done Sent false
26596 8083 Acked Write 3 Done Done Sent false
26597 8084 Acked Read 10 Done Done Sent false
26598 8085 Acked Write 2 Done Done Sent false
26599 8086 Acked Read 10 Done Done Sent false
26600 8087 Acked Write 4 Done Done Sent false
26601 8088 Acked Write 9 Done Done Sent false
26602 8089 Acked Write 9 Done Done Sent false
26603 8090 Acked Write 10 Done Done Sent false
26604 8091 Acked Read 9 Done Done Sent false
26605 8092 Acked Write 1 Done Done Sent false
26606 8093 Acked Read 3 Done Done Sent false
26607 8094 Acked Read 2 Done Done Sent false
26608 8095 Acked Read 5 Done Done Sent false
26609 8096 Acked Write 5 Done Done Sent false
26610 8097 Acked Read 2 Done Done Sent false
26611 8098 Acked Write 8 Done Done Sent false
26612 8099 Acked Read 1 Done Done Sent false
26613 8100 Acked Read 2 Done Done Sent false
26614 8101 Acked Read 2 Done Done Sent false
26615 8102 Acked Read 9 Done Done Sent false
26616 8103 Acked Write 9 Done Done Sent false
26617 8104 Acked Write 2 Done Done Sent false
26618 8105 Acked Read 4 Done Done Sent false
26619 8106 Acked Flush 0 Done Done Sent false
26620 8107 Acked Read 9 Done Done Sent false
26621 8108 Acked Write 10 Done Done Sent false
26622 8109 Acked Read 1 Done Done Sent false
26623 8110 Acked Write 10 Done Done Sent false
26624 8111 Acked Read 8 Done Done Sent false
26625 8112 Acked Flush 0 Done Done Sent false
26626 8113 Acked Read 4 Done Done Sent false
26627 8114 Acked Read 9 Done Done Sent false
26628 8115 Acked Read 1 Done Done Sent false
26629 8116 Acked Read 4 Done Done Sent false
26630 8117 Acked Write 8 Done Done Sent false
26631 8118 Acked Write 1 Done Done Sent false
26632 8119 Acked Write 1 Done Done Sent false
26633 8120 Acked Read 6 Done Done Sent false
26634 8121 Acked Read 2 Done Done Sent false
26635 8122 Acked Write 4 Done Done Sent false
26636 8123 Acked Read 4 Done Done Sent false
26637 8124 Acked Read 5 Done Done Sent false
26638 8125 Acked Write 5 Done Done Sent false
26639 8126 Acked Flush 0 Done Done Sent false
26640 8127 Acked Write 9 Done Done Sent false
26641 8128 Acked Read 5 Done Done Sent false
26642 8129 Acked Write 2 Done Done Sent false
26643 8130 Acked Read 5 Done Done Sent false
26644 8131 Acked Write 3 Done Done Sent false
26645 8132 Acked Read 1 Done Done Sent false
26646 8133 Acked Read 10 Done Done Sent false
26647 8134 Acked Write 8 Done Done Sent false
26648 8135 Acked Read 5 Done Done Sent false
26649 8136 Acked Write 3 Done Done Sent false
26650 8137 Acked Read 10 Done Done Sent false
26651 8138 Acked Write 4 Done Done Sent false
26652 8139 Acked Read 3 Done Done Sent false
26653 8140 Acked Flush 0 Done Done Sent false
26654 8141 Acked Read 3 Done Done Sent false
26655 8142 Acked Read 7 Done Done Sent false
26656 8143 Acked Read 5 Done Done Sent false
26657 8144 Acked Write 6 Done Done Sent false
26658 8145 Acked Read 1 Done Done Sent false
26659 8146 Acked Read 2 Done Done Sent false
26660 8147 Acked Read 4 Done Done Sent false
26661 8148 Acked Read 9 Done Done Sent false
26662 8149 Acked Write 9 Done Done Sent false
26663 8150 Acked Flush 0 Done Done Sent false
26664 8151 Acked Write 10 Done Done Sent false
26665 8152 Acked Read 6 Done Done Sent false
26666 8153 Acked Write 3 Done Done Sent false
26667 8154 Acked Write 8 Done Done Sent false
26668 8155 Acked Write 3 Done Done Sent false
26669 8156 Acked Write 6 Done Done Sent false
26670 8157 Acked Write 1 Done Done Sent false
26671 8158 Acked Read 4 Done Done Sent false
26672 8159 Acked Read 2 Done Done Sent false
26673 8160 Acked Read 10 Done Done Sent false
26674 8161 Acked Write 10 Done Done Sent false
26675 8162 Acked Write 2 Done Done Sent false
26676 8163 Acked Read 2 Done Done Sent false
26677 8164 Acked Write 7 Done Done Sent false
26678 8165 Acked Write 3 Done Done Sent false
26679 8166 Acked Read 2 Done Done Sent false
26680 8167 Acked Flush 0 Done Done Sent false
26681 8168 Acked Read 1 Done Done Sent false
26682 8169 Acked Write 1 Done Done Sent false
26683 8170 Acked Write 7 Done Done Sent false
26684 8171 Acked Read 9 Done Done Sent false
26685 8172 Acked Read 3 Done Done Sent false
26686 8173 Acked Read 7 Done Done Sent false
26687 8174 Acked Read 3 Done Done Sent false
26688 8175 Acked Write 4 Done Done Sent false
26689 8176 Acked Read 2 Done Done Sent false
26690 8177 Acked Write 7 Done Done Sent false
26691 8178 Acked Read 5 Done Done Sent false
26692 8179 Acked Read 6 Done Done Sent false
26693 8180 Acked Read 9 Done Done Sent false
26694 8181 Acked Write 3 Done Done Sent false
26695 8182 Acked Write 4 Done Done Sent false
26696 8183 Acked Write 4 Done Done Sent false
26697 8184 Acked Write 1 Done Done Sent false
26698 8185 Acked Read 9 Done Done Sent false
26699 8186 Acked Read 8 Done Done Sent false
26700 8187 Acked Write 1 Done Done Sent false
26701 8188 Acked Read 6 Done Done Sent false
26702 8189 Acked Read 7 Done Done Sent false
26703 8190 Acked Read 6 Done Done Sent false
26704 8191 Acked Write 4 Done Done Sent false
26705 8192 Acked Read 4 Done Done Sent false
26706 8193 Acked Write 4 Done Done Sent false
26707 8194 Acked Read 1 Done Done Sent false
26708 8195 Acked Read 3 Done Done Sent false
26709 8196 Acked Write 1 Done Done Sent false
26710 8197 Acked Flush 0 Done Done Sent false
26711 8198 Acked Write 1 Done Done Sent false
26712 8199 Acked Read 4 Done Done Sent false
26713 8200 Acked Read 1 Done Done Sent false
26714 8201 Acked Read 7 Done Done Sent false
26715 8202 Acked Read 1 Done Done Sent false
26716 8203 Acked Write 1 Done Done Sent false
26717 8204 Acked Write 4 Done Done Sent false
26718 8205 Acked Write 6 Done Done Sent false
26719 8206 Acked Read 7 Done Done Sent false
26720 8207 Acked Write 3 Done Done Sent false
26721 8208 Acked Read 3 Done Done Sent false
26722 8209 Acked Read 3 Done Done Sent false
26723 8210 Acked Read 5 Done Done Sent false
26724 8211 Acked Read 1 Done Done Sent false
26725 8212 Acked Read 1 Done Done Sent false
26726 8213 Acked Write 8 Done Done Sent false
26727 8214 Acked Read 8 Done Done Sent false
26728 8215 Acked Read 4 Done Done Sent false
26729 8216 Acked Write 6 Done Done Sent false
26730 8217 Acked Read 2 Done Done Sent false
26731 8218 Acked Write 6 Done Done Sent false
26732 8219 Acked Write 1 Done Done Sent false
26733 8220 Acked Write 6 Done Done Sent false
26734 8221 Acked Write 1 Done Done Sent false
26735 8222 Acked Read 5 Done Done Sent false
26736 8223 Acked Read 9 Done Done Sent false
26737 8224 Acked Read 2 Done Done Sent false
26738 8225 Acked Read 1 Done Done Sent false
26739 8226 Acked Write 3 Done Done Sent false
26740 8227 Acked Read 10 Done Done Sent false
26741 8228 Acked Read 2 Done Done Sent false
26742 8229 Acked Read 4 Done Done Sent false
26743 8230 Acked Read 4 Done Done Sent false
26744 8231 Acked Read 7 Done Done Sent false
26745 8232 Acked Read 5 Done Done Sent false
26746 8233 Acked Read 9 Done Done Sent false
26747 8234 Acked Read 1 Done Done Sent false
26748 8235 Acked Read 3 Done Done Sent false
26749 8236 Acked Read 1 Done Done Sent false
26750 8237 Acked Write 4 Done Done Sent false
26751 8238 Acked Read 5 Done Done Sent false
26752 8239 Acked Flush 0 Done Done Sent false
26753 8240 Acked Write 5 Done Done Sent false
26754 8241 Acked Read 10 Done Done Sent false
26755 8242 Acked Read 1 Done Done Sent false
26756 8243 Acked Read 2 Done Done Sent false
26757 8244 Acked Read 6 Done Done Sent false
26758 8245 Acked Read 4 Done Done Sent false
26759 8246 Acked Flush 0 Done Done Sent false
26760 STATES DS:0 DS:1 DS:2 TOTAL
26761 Sent 0 0 301 301
26762 Done 301 301 0 602
26763 Skipped 0 0 0 0
26764 Error 0 0 0 0
26765 Last Flush: 8246 8246 7945
26766 Downstairs last five completed: 7945 7944 7943 7942 7941
26767 Upstairs last five completed: 8246 8245 8244 8243 8242
26768 CLIENT: Up:0 ds:301 act:2
267692024-12-09T15:13:26.815ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
267702024-12-09T15:13:26.816ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36427) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
267712024-12-09T15:13:26.817ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
267722024-12-09T15:13:26.817ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
267732024-12-09T15:13:26.817ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
267742024-12-09T15:13:26.817ZINFOcrucible: send last flush ID to this DS: 7945 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
267752024-12-09T15:13:26.818ZINFOcrucible: Replied this last flush ID: 7945 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
267762024-12-09T15:13:26.818ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
267772024-12-09T15:13:26.818ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
267782024-12-09T15:13:26.818ZINFOcrucible: [2] client re-new 301 jobs since flush 7945 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
267792024-12-09T15:13:26.818ZINFOcrucible: [2] Replayed 301 jobs since flush: 7945 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
26780 ----------------------------------------------------------------
26781 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26782 Downstairs last five completed: 8246 8245 8244 8243 8242
26783 Upstairs last five completed: 8246 8245 8244 8243 8242
26784 CLIENT: Up:0 ds:0 act:3
26785 Replay: All jobs finished
26786 Main task has work to do, go find it
26787 got dsc Stop(2)
26788 stop 2
26789 [8830] Got stop action so:false kr:false
26790 Dec 09 15:13:29.063 INFO request completed, latency_us: 378, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 0dd858f0-adb6-4c54-99ed-560adfa6137e, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
267912024-12-09T15:13:29.064ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
26792 Dec 09 15:13:29.064 INFO request completed, latency_us: 151, response_code: 200, uri: /state/cid/2, method{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:13:29.064607358Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
26793 : GET, req_id: 147052cf-63f5-46b2-bcc8-97db12a2576a, remote_addr: 127.0.0.{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:13:29.06465279Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
26794 1:47510, local_addr: 127.0.0.1:9998
267952024-12-09T15:13:29.064ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
267962024-12-09T15:13:29.064ZINFOcrucible: [2] Transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
267972024-12-09T15:13:29.064ZINFOcrucible: client task is exiting = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
26798 [8830] Exited with: ExitStatus(unix_wait_status(9))
26799 [8830][2] reports Exit
26800 Dec 09 15:13:33.066 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/2, method: GET, req_id: e3155d53-8514-4b3d-8468-4d6ee1285252, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
26801 001/300 Read block 884 len 24576
26802 002/300 Flush
26803 003/300 Write block 1396 len 28672 data: 7 6 5 7 7 5 5
26804 004/300 Read block 2415 len 40960
26805 005/300 Write block 625 len 40960 data: 5 5 5 4 5 6 6 5 5 4
26806 006/300 Read block 353 len 28672
26807 007/300 Read block 520 len 8192
26808 008/300 Read block 887 len 8192
26809 009/300 Read block 903 len 20480
26810 010/300 Read block 2451 len 8192
26811 011/300 Read block 2166 len 40960
26812 012/300 Write block 4122 len 24576 data: 5 5 6 5 7 6
26813 013/300 Write block 3133 len 24576 data: 6 5 6 4 3 4
26814 014/300 Write block 2017 len 36864 data: 9 10 6 7 7 8 7 4 5
26815 015/300 Write block 1332 len 40960 data: 7 6 4 9 12 11 9 8 6 9
26816 016/300 Read block 3122 len 20480
26817 017/300 Write block 159 len 24576 data: 7 6 6 5 4 5
26818 018/300 Write block 4425 len 4096 data: 8
26819 019/300 Read block 1688 len 16384
26820 020/300 Read block 4942 len 12288
26821 021/300 Flush
26822 022/300 Read block 866 len 8192
26823 023/300 Write block 4009 len 4096 data: 5
26824 024/300 Write block 1174 len 8192 data: 6 5
26825 025/300 Write block 1807 len 32768 data: 6 5 4 3 3 3 3 3
26826 026/300 Read block 2114 len 40960
26827 027/300 Write block 4111 len 40960 data: 5 5 5 4 4 6 4 3 4 5
26828 028/300 Write block 4509 len 32768 data: 9 8 8 8 7 7 6 5
26829 029/300 Read block 242 len 36864
26830 030/300 Read block 845 len 8192
26831 031/300 Read block 4191 len 20480
26832 032/300 Read block 4960 len 4096
26833 033/300 Flush
26834 034/300 Read block 4111 len 20480
26835 035/300 Write block 1376 len 16384 data: 7 8 7 6
26836 036/300 Read block 2785 len 16384
26837 037/300 Write block 3868 len 20480 data: 5 5 7 7 7
26838 038/300 Read block 3825 len 24576
26839 039/300 Flush
26840 040/300 Write block 2470 len 24576 data: 5 5 4 4 4 4
26841 041/300 Write block 682 len 24576 data: 3 5 5 7 10 10
26842 042/300 Write block 2297 len 4096 data: 5
26843 043/300 Read block 1970 len 20480
26844 044/300 Write block 322 len 36864 data: 9 7 7 5 7 6 6 5 4
26845 045/300 Read block 1928 len 8192
26846 046/300 Read block 2091 len 28672
26847 047/300 Read block 764 len 40960
26848 048/300 Write block 4299 len 8192 data: 7 7
26849 049/300 Write block 4837 len 8192 data: 5 7
26850 050/300 Write block 3647 len 24576 data: 3 3 4 4 5 5
26851 051/300 Flush
26852 052/300 Read block 2032 len 40960
26853 053/300 Read block 4888 len 28672
26854 054/300 Read block 3913 len 32768
26855 055/300 Write block 1432 len 12288 data: 4 4 5
26856 056/300 Write block 4385 len 16384 data: 5 5 6 5
26857 057/300 Read block 572 len 32768
26858 058/300 Read block 4790 len 28672
26859 059/300 Read block 2470 len 4096
26860 060/300 Write block 193 len 24576 data: 4 4 4 4 4 4
26861 061/300 Read block 1512 len 40960
26862 062/300 Read block 2001 len 32768
26863 063/300 Write block 3515 len 40960 data: 5 5 8 10 9 9 8 7 7 9
26864 064/300 Flush
26865 065/300 Read block 3901 len 20480
26866 066/300 Read block 4641 len 20480
26867 067/300 Read block 266 len 8192
26868 068/300 Flush
26869 069/300 Write block 3139 len 28672 data: 4 5 7 7 7 9 7
26870 070/300 Read block 1105 len 4096
26871 071/300 Write block 3929 len 20480 data: 7 8 6 6 5
26872 072/300 Read block 3522 len 36864
26873 073/300 Read block 3051 len 28672
26874 074/300 Write block 1571 len 40960 data: 5 5 4 5 5 4 6 6 6 6
26875 075/300 Read block 2321 len 12288
26876 076/300 Read block 1720 len 28672
26877 077/300 Flush
26878 078/300 Read block 550 len 24576
26879 079/300 Write block 1762 len 4096 data: 7
26880 080/300 Read block 2981 len 36864
26881 081/300 Write block 1288 len 36864 data: 3 4 5 8 8 8 6 6 5
26882 082/300 Read block 3421 len 12288
26883 083/300 Write block 3177 len 32768 data: 10 9 9 9 8 8 8 8
26884 084/300 Write block 3711 len 20480 data: 6 6 6 5 6
26885 085/300 Read block 1251 len 28672
26886 086/300 Write block 3723 len 28672 data: 6 7 4 6 7 7 6
26887 087/300 Read block 3940 len 4096
26888 088/300 Write block 4140 len 28672 data: 5 4 4 4 4 3 3
26889 089/300 Flush
26890 090/300 Read block 4430 len 28672
26891 091/300 Read block 1697 len 28672
26892 092/300 Write block 1750 len 28672 data: 2 4 5 7 7 7 8
26893 093/300 Write block 2451 len 24576 data: 2 4 5 5 4 6
26894 094/300 Read block 2727 len 36864
26895 095/300 Write block 3837 len 28672 data: 5 3 4 4 5 6 8
26896 096/300 Read block 1573 len 12288
26897 097/300 Read block 2648 len 8192
26898 098/300 Write block 2832 len 4096 data: 5
26899 099/300 Write block 4824 len 28672 data: 3 5 3 3 4 3 4
26900 100/300 Read block 3098 len 12288
26901 101/300 Read block 1038 len 40960
26902 102/300 Read block 308 len 40960
26903 103/300 Read block 1445 len 24576
26904 104/300 Read block 1974 len 8192
26905 105/300 Read block 650 len 40960
26906 106/300 Write block 2136 len 28672 data: 8 8 9 7 6 7 6
26907 107/300 Write block 2224 len 4096 data: 6
26908 108/300 Read block 379 len 20480
26909 109/300 Read block 1727 len 36864
26910 110/300 Read block 3004 len 32768
26911 111/300 Write block 4774 len 16384 data: 5 5 6 6
26912 112/300 Write block 3578 len 28672 data: 4 5 6 7 7 9 10
26913 113/300 Read block 4064 len 32768
26914 114/300 Write block 2831 len 24576 data: 6 6 4 4 3 3
26915 115/300 Write block 3153 len 40960 data: 8 9 8 7 7 7 8 9 10 10
26916 116/300 Read block 2136 len 24576
26917 117/300 Read block 1339 len 24576
26918 118/300 Write block 4715 len 16384 data: 5 5 5 4
26919 119/300 Read block 4623 len 12288
26920 120/300 Write block 1578 len 36864 data: 7 7 7 6 4 4 4 4 3
26921 121/300 Read block 1887 len 24576
26922 122/300 Read block 611 len 36864
26923 123/300 Write block 2706 len 36864 data: 6 5 4 5 5 9 9 10 8
26924 124/300 Write block 2938 len 12288 data: 7 7 6
26925 125/300 Read block 1193 len 12288
26926 126/300 Read block 1867 len 12288
26927 127/300 Write block 288 len 24576 data: 5 5 3 3 3 3
26928 128/300 Write block 1111 len 24576 data: 7 6 5 6 5 4
26929 129/300 Write block 1995 len 12288 data: 4 7 8
26930 130/300 Write block 2909 len 32768 data: 5 4 3 3 5 6 7 7
26931 131/300 Write block 3134 len 8192 data: 6 7
26932 132/300 Read block 4246 len 4096
26933 133/300 Read block 81 len 8192
26934 134/300 Flush
26935 135/300 Write block 4026 len 32768 data: 5 5 5 6 5 5 6 5
26936 136/300 Read block 3222 len 36864
26937 137/300 Read block 926 len 16384
26938 138/300 Write block 1409 len 16384 data: 8 9 9 7
26939 139/300 Read block 4629 len 32768
26940 140/300 Read block 2966 len 12288
26941 141/300 Read block 758 len 32768
26942 142/300 Read block 690 len 12288
26943 143/300 Read block 4035 len 36864
26944 144/300 Read block 759 len 28672
26945 145/300 Read block 2830 len 4096
26946 146/300 Read block 4710 len 36864
26947 147/300 Read block 2429 len 12288
26948 148/300 Read block 3185 len 28672
26949 149/300 Write block 4720 len 32768 data: 5 6 5 5 5 6 4 4
26950 150/300 Read block 4886 len 4096
26951 151/300 Write block 3683 len 12288 data: 6 6 7
26952 152/300 Write block 305 len 12288 data: 5 5 4
26953 153/300 Flush
26954 154/300 Write block 1048 len 20480 data: 6 4 4 3 4
26955 155/300 Read block 4717 len 24576
26956 156/300 Write block 612 len 8192 data: 5 3
26957 157/300 Read block 1067 len 16384
26958 158/300 Read block 2606 len 16384
26959 159/300 Write block 1848 len 40960 data: 4 4 5 5 6 6 6 7 6 5
26960 160/300 Read block 2620 len 24576
26961 161/300 Flush
26962 162/300 Read block 4444 len 4096
26963 163/300 Read block 1913 len 12288
26964 164/300 Write block 4861 len 24576 data: 5 4 4 5 5 6
26965 165/300 Write block 4559 len 32768 data: 6 4 5 4 4 2 3 5
26966 166/300 Write block 3163 len 40960 data: 8 5 4 4 5 5 5 6 6 4
26967 167/300 Read block 3400 len 24576
26968 168/300 Write block 3105 len 12288 data: 6 5 4
26969 169/300 Read block 1045 len 16384
26970 170/300 Write block 3537 len 8192 data: 6 4
26971 171/300 Write block 2817 len 24576 data: 5 5 5 4 4 4
26972 172/300 Read block 1795 len 24576
26973 173/300 Read block 2845 len 28672
26974 174/300 Write block 1560 len 28672 data: 8 10 9 9 9 7 7
26975 175/300 Read block 426 len 28672
26976 176/300 Flush
26977 177/300 Write block 657 len 8192 data: 3 3
26978 178/300 Read block 2936 len 24576
26979 179/300 Flush
26980 180/300 Write block 4579 len 40960 data: 7 6 5 8 8 6 6 6 6 5
26981 181/300 Write block 2915 len 8192 data: 8 8
26982 182/300 Read block 2383 len 12288
26983 183/300 Read block 4533 len 4096
26984 184/300 Write block 104 len 24576 data: 5 6 6 6 4 4
26985 185/300 Flush
26986 186/300 Flush
26987 187/300 Read block 1368 len 40960
26988 188/300 Read block 295 len 8192
26989 189/300 Write block 1105 len 20480 data: 7 9 8 8 8
26990 190/300 Read block 4586 len 4096
26991 191/300 Write block 3070 len 4096 data: 4
26992 192/300 Write block 4737 len 4096 data: 3
26993 193/300 Flush
26994 194/300 Write block 2684 len 20480 data: 6 6 6 6 5
26995 195/300 Write block 4397 len 16384 data: 5 5 4 4
26996 196/300 Read block 695 len 32768
26997 197/300 Read block 764 len 12288
26998 198/300 Read block 676 len 16384
26999 199/300 Read block 4872 len 12288
27000 200/300 Write block 2710 len 32768 data: 6 10 10 11 9 9 7 8
27001 201/300 Read block 4583 len 4096
27002 202/300 Read block 4028 len 16384
27003 203/300 Read block 2030 len 40960
27004 204/300 Read block 0 len 20480
27005 205/300 Write block 2692 len 36864 data: 4 6 6 4 5 6 6 6 6
27006 206/300 Read block 1242 len 36864
27007 207/300 Read block 1045 len 36864
27008 208/300 Write block 940 len 12288 data: 4 5 4
27009 209/300 Read block 1958 len 28672
27010 210/300 Read block 4038 len 40960
27011 211/300 Write block 1250 len 16384 data: 7 5 5 4
27012 212/300 Flush
27013 213/300 Write block 1832 len 24576 data: 5 4 5 5 4 3
27014 214/300 Write block 359 len 8192 data: 9 8
27015 215/300 Read block 2300 len 36864
27016 216/300 Write block 1418 len 4096 data: 5
27017 217/300 Write block 1573 len 20480 data: 5 6 6 5 7
27018 218/300 Write block 822 len 40960 data: 7 6 7 7 5 4 3 4 5 5
27019 219/300 Read block 3624 len 12288
27020 220/300 Read block 2759 len 36864
27021 221/300 Read block 3397 len 36864
27022 222/300 Read block 1244 len 28672
27023 223/300 Write block 1496 len 12288 data: 3 6 6
27024 224/300 Write block 2364 len 20480 data: 2 2 3 5 5
27025 225/300 Flush
27026 226/300 Write block 4328 len 28672 data: 5 4 3 2 3 4 4
27027 227/300 Read block 3732 len 12288
27028 228/300 Flush
27029 229/300 Write block 55 len 32768 data: 4 4 4 4 4 4 5 3
27030 230/300 Read block 409 len 16384
27031 231/300 Write block 512 len 28672 data: 5 6 6 6 5 6 7
27032 232/300 Read block 3827 len 16384
27033 233/300 Flush
27034 234/300 Read block 3360 len 8192
27035 235/300 Write block 3391 len 28672 data: 3 3 5 8 7 8 8
27036 236/300 Read block 2090 len 40960
27037 237/300 Read block 2700 len 4096
27038 238/300 Write block 0 len 32768 data: 2 4 5 6 4 4 5 5
27039 239/300 Write block 1434 len 28672 data: 6 5 5 5 5 5 4
27040 240/300 Read block 144 len 8192
27041 241/300 Write block 1633 len 12288 data: 4 4 4
27042 242/300 Write block 714 len 24576 data: 3 3 4 5 5 6
27043 243/300 Write block 4990 len 32768 data: 8 10 9 7 6 5 4 3
27044 244/300 Read block 1815 len 8192
27045 245/300 Read block 4159 len 36864
27046 246/300 Read block 1257 len 24576
27047 247/300 Write block 1961 len 36864 data: 5 5 7 6 5 5 7 6 8
27048 248/300 Read block 1554 len 8192
27049 249/300 Read block 4902 len 40960
27050 250/300 Write block 4481 len 16384 data: 2 2 2 2
27051 251/300 Write block 1715 len 12288 data: 7 7 6
27052 252/300 Write block 800 len 16384 data: 7 9 9 8
27053 253/300 Write block 1223 len 20480 data: 4 4 5 6 6
27054 254/300 Write block 1990 len 12288 data: 4 4 4
27055 255/300 Write block 441 len 32768 data: 3 2 2 2 2 3 3 4
27056 256/300 Write block 22 len 32768 data: 6 6 6 5 5 4 5 5
27057 257/300 Flush
27058 258/300 Write block 1931 len 28672 data: 6 7 7 6 5 6 4
27059 259/300 Write block 4628 len 4096 data: 6
27060 260/300 Read block 3460 len 32768
27061 261/300 Read block 1997 len 28672
27062 262/300 Write block 1816 len 36864 data: 3 5 6 7 7 5 6 5 4
27063 263/300 Write block 1802 len 36864 data: 6 5 6 5 6 7 6 5 4
27064 264/300 Write block 2235 len 12288 data: 7 4 3
27065 265/300 Read block 4143 len 16384
27066 266/300 Read block 1250 len 40960
27067 267/300 Read block 3597 len 32768
27068 268/300 Read block 3644 len 20480
27069 269/300 Write block 1507 len 28672 data: 4 7 7 5 7 7 8
27070 270/300 Write block 2934 len 16384 data: 8 8 6 6
27071 271/300 Read block 626 len 24576
27072 272/300 Read block 1116 len 4096
27073 273/300 Read block 2148 len 28672
27074 274/300 Write block 1617 len 32768 data: 5 5 5 5 5 5 4 5
27075 275/300 Read block 3245 len 8192
27076 276/300 Read block 2689 len 28672
27077 277/300 Read block 3987 len 40960
27078 278/300 Write block 447 len 24576 data: 4 5 4 3 3 3
27079 279/300 Read block 846 len 28672
27080 280/300 Read block 2438 len 32768
27081 281/300 Flush
27082 282/300 Write block 191 len 4096 data: 6
27083 283/300 Flush
27084 284/300 Read block 2552 len 20480
27085 285/300 Read block 592 len 32768
27086 286/300 Flush
27087 287/300 Read block 3965 len 28672
27088 288/300 Flush
27089 289/300 Read block 3405 len 24576
27090 290/300 Write block 3782 len 8192 data: 7 7
27091 291/300 Write block 736 len 32768 data: 4 5 6 7 9 11 9 6
27092 292/300 Read block 4471 len 28672
27093 293/300 Write block 2864 len 4096 data: 5
27094 294/300 Read block 709 len 32768
27095 295/300 Read block 185 len 28672
27096 296/300 Read block 1939 len 12288
27097 297/300 Read block 2809 len 8192
27098 298/300 Read block 1508 len 32768
27099 299/300 Read block 728 len 28672
27100 300/300 Read block 3382 len 36864
27101 301/300 Read block 2740 len 12288
27102 Main task has work to do, go find it
27103 got dsc Start(2)
27104 start 2
27105 Dec 09 15:13:33.311 INFO request completed, latency_us: 227, response_code: 204, uri: /start/cid/2, method: POST, req_id: f2649783-f9df-4a9f-8b02-00ff45c42ba3, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
27106 Replay: started 2, returned:Ok(())
27107 ----------------------------------------------------------------
27108 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27109 States: Active Active Offline
27110 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27111 8247 Acked Read 6 Done Done Sent false
27112 8248 Acked Flush 0 Done Done Sent false
27113 8249 Acked Write 7 Done Done Sent false
27114 8250 Acked Read 10 Done Done Sent false
27115 8251 Acked Write 10 Done Done Sent false
27116 8252 Acked Read 7 Done Done Sent false
27117 8253 Acked Read 2 Done Done Sent false
27118 8254 Acked Read 2 Done Done Sent false
27119 8255 Acked Read 5 Done Done Sent false
27120 8256 Acked Read 2 Done Done Sent false
27121 8257 Acked Read 10 Done Done Sent false
27122 8258 Acked Write 6 Done Done Sent false
27123 8259 Acked Write 6 Done Done Sent false
27124 8260 Acked Write 9 Done Done Sent false
27125 8261 Acked Write 10 Done Done Sent false
27126 8262 Acked Read 5 Done Done Sent false
27127 8263 Acked Write 6 Done Done Sent false
27128 8264 Acked Write 1 Done Done Sent false
27129 8265 Acked Read 4 Done Done Sent false
27130 8266 Acked Read 3 Done Done Sent false
27131 8267 Acked Flush 0 Done Done Sent false
27132 8268 Acked Read 2 Done Done Sent false
27133 8269 Acked Write 1 Done Done Sent false
27134 8270 Acked Write 2 Done Done Sent false
27135 8271 Acked Write 8 Done Done Sent false
27136 8272 Acked Read 10 Done Done Sent false
27137 8273 Acked Write 10 Done Done Sent false
27138 8274 Acked Write 8 Done Done Sent false
27139 8275 Acked Read 9 Done Done Sent false
27140 8276 Acked Read 2 Done Done Sent false
27141 8277 Acked Read 5 Done Done Sent false
27142 8278 Acked Read 1 Done Done Sent false
27143 8279 Acked Flush 0 Done Done Sent false
27144 8280 Acked Read 5 Done Done Sent false
27145 8281 Acked Write 4 Done Done Sent false
27146 8282 Acked Read 4 Done Done Sent false
27147 8283 Acked Write 5 Done Done Sent false
27148 8284 Acked Read 6 Done Done Sent false
27149 8285 Acked Flush 0 Done Done Sent false
27150 8286 Acked Write 6 Done Done Sent false
27151 8287 Acked Write 6 Done Done Sent false
27152 8288 Acked Write 1 Done Done Sent false
27153 8289 Acked Read 5 Done Done Sent false
27154 8290 Acked Write 9 Done Done Sent false
27155 8291 Acked Read 2 Done Done Sent false
27156 8292 Acked Read 7 Done Done Sent false
27157 8293 Acked Read 10 Done Done Sent false
27158 8294 Acked Write 2 Done Done Sent false
27159 8295 Acked Write 2 Done Done Sent false
27160 8296 Acked Write 6 Done Done Sent false
27161 8297 Acked Flush 0 Done Done Sent false
27162 8298 Acked Read 10 Done Done Sent false
27163 8299 Acked Read 7 Done Done Sent false
27164 8300 Acked Read 8 Done Done Sent false
27165 8301 Acked Write 3 Done Done Sent false
27166 8302 Acked Write 4 Done Done Sent false
27167 8303 Acked Read 8 Done Done Sent false
27168 8304 Acked Read 7 Done Done Sent false
27169 8305 Acked Read 1 Done Done Sent false
27170 8306 Acked Write 6 Done Done Sent false
27171 8307 Acked Read 10 Done Done Sent false
27172 8308 Acked Read 8 Done Done Sent false
27173 8309 Acked Write 10 Done Done Sent false
27174 8310 Acked Flush 0 Done Done Sent false
27175 8311 Acked Read 5 Done Done Sent false
27176 8312 Acked Read 5 Done Done Sent false
27177 8313 Acked Read 2 Done Done Sent false
27178 8314 Acked Flush 0 Done Done Sent false
27179 8315 Acked Write 7 Done Done Sent false
27180 8316 Acked Read 1 Done Done Sent false
27181 8317 Acked Write 5 Done Done Sent false
27182 8318 Acked Read 9 Done Done Sent false
27183 8319 Acked Read 7 Done Done Sent false
27184 8320 Acked Write 10 Done Done Sent false
27185 8321 Acked Read 3 Done Done Sent false
27186 8322 Acked Read 7 Done Done Sent false
27187 8323 Acked Flush 0 Done Done Sent false
27188 8324 Acked Read 6 Done Done Sent false
27189 8325 Acked Write 1 Done Done Sent false
27190 8326 Acked Read 9 Done Done Sent false
27191 8327 Acked Write 9 Done Done Sent false
27192 8328 Acked Read 3 Done Done Sent false
27193 8329 Acked Write 8 Done Done Sent false
27194 8330 Acked Write 5 Done Done Sent false
27195 8331 Acked Read 7 Done Done Sent false
27196 8332 Acked Write 7 Done Done Sent false
27197 8333 Acked Read 1 Done Done Sent false
27198 8334 Acked Write 7 Done Done Sent false
27199 8335 Acked Flush 0 Done Done Sent false
27200 8336 Acked Read 7 Done Done Sent false
27201 8337 Acked Read 7 Done Done Sent false
27202 8338 Acked Write 7 Done Done Sent false
27203 8339 Acked Write 6 Done Done Sent false
27204 8340 Acked Read 9 Done Done Sent false
27205 8341 Acked Write 7 Done Done Sent false
27206 8342 Acked Read 3 Done Done Sent false
27207 8343 Acked Read 2 Done Done Sent false
27208 8344 Acked Write 1 Done Done Sent false
27209 8345 Acked Write 7 Done Done Sent false
27210 8346 Acked Read 3 Done Done Sent false
27211 8347 Acked Read 10 Done Done Sent false
27212 8348 Acked Read 10 Done Done Sent false
27213 8349 Acked Read 6 Done Done Sent false
27214 8350 Acked Read 2 Done Done Sent false
27215 8351 Acked Read 10 Done Done Sent false
27216 8352 Acked Write 7 Done Done Sent false
27217 8353 Acked Write 1 Done Done Sent false
27218 8354 Acked Read 5 Done Done Sent false
27219 8355 Acked Read 9 Done Done Sent false
27220 8356 Acked Read 8 Done Done Sent false
27221 8357 Acked Write 4 Done Done Sent false
27222 8358 Acked Write 7 Done Done Sent false
27223 8359 Acked Read 8 Done Done Sent false
27224 8360 Acked Write 6 Done Done Sent false
27225 8361 Acked Write 10 Done Done Sent false
27226 8362 Acked Read 6 Done Done Sent false
27227 8363 Acked Read 6 Done Done Sent false
27228 8364 Acked Write 4 Done Done Sent false
27229 8365 Acked Read 3 Done Done Sent false
27230 8366 Acked Write 9 Done Done Sent false
27231 8367 Acked Read 6 Done Done Sent false
27232 8368 Acked Read 9 Done Done Sent false
27233 8369 Acked Write 9 Done Done Sent false
27234 8370 Acked Write 3 Done Done Sent false
27235 8371 Acked Read 3 Done Done Sent false
27236 8372 Acked Read 3 Done Done Sent false
27237 8373 Acked Write 6 Done Done Sent false
27238 8374 Acked Write 6 Done Done Sent false
27239 8375 Acked Write 3 Done Done Sent false
27240 8376 Acked Write 8 Done Done Sent false
27241 8377 Acked Write 2 Done Done Sent false
27242 8378 Acked Read 1 Done Done Sent false
27243 8379 Acked Read 2 Done Done Sent false
27244 8380 Acked Flush 0 Done Done Sent false
27245 8381 Acked Write 8 Done Done Sent false
27246 8382 Acked Read 9 Done Done Sent false
27247 8383 Acked Read 4 Done Done Sent false
27248 8384 Acked Write 4 Done Done Sent false
27249 8385 Acked Read 8 Done Done Sent false
27250 8386 Acked Read 3 Done Done Sent false
27251 8387 Acked Read 8 Done Done Sent false
27252 8388 Acked Read 3 Done Done Sent false
27253 8389 Acked Read 9 Done Done Sent false
27254 8390 Acked Read 7 Done Done Sent false
27255 8391 Acked Read 1 Done Done Sent false
27256 8392 Acked Read 9 Done Done Sent false
27257 8393 Acked Read 3 Done Done Sent false
27258 8394 Acked Read 7 Done Done Sent false
27259 8395 Acked Write 8 Done Done Sent false
27260 8396 Acked Read 1 Done Done Sent false
27261 8397 Acked Write 3 Done Done Sent false
27262 8398 Acked Write 3 Done Done Sent false
27263 8399 Acked Flush 0 Done Done Sent false
27264 8400 Acked Write 5 Done Done Sent false
27265 8401 Acked Read 6 Done Done Sent false
27266 8402 Acked Write 2 Done Done Sent false
27267 8403 Acked Read 4 Done Done Sent false
27268 8404 Acked Read 4 Done Done Sent false
27269 8405 Acked Write 10 Done Done Sent false
27270 8406 Acked Read 6 Done Done Sent false
27271 8407 Acked Flush 0 Done Done Sent false
27272 8408 Acked Read 1 Done Done Sent false
27273 8409 Acked Read 3 Done Done Sent false
27274 8410 Acked Write 6 Done Done Sent false
27275 8411 Acked Write 8 Done Done Sent false
27276 8412 Acked Write 10 Done Done Sent false
27277 8413 Acked Read 6 Done Done Sent false
27278 8414 Acked Write 3 Done Done Sent false
27279 8415 Acked Read 4 Done Done Sent false
27280 8416 Acked Write 2 Done Done Sent false
27281 8417 Acked Write 6 Done Done Sent false
27282 8418 Acked Read 6 Done Done Sent false
27283 8419 Acked Read 7 Done Done Sent false
27284 8420 Acked Write 7 Done Done Sent false
27285 8421 Acked Read 7 Done Done Sent false
27286 8422 Acked Flush 0 Done Done Sent false
27287 8423 Acked Write 2 Done Done Sent false
27288 8424 Acked Read 6 Done Done Sent false
27289 8425 Acked Flush 0 Done Done Sent false
27290 8426 Acked Write 10 Done Done Sent false
27291 8427 Acked Write 2 Done Done Sent false
27292 8428 Acked Read 3 Done Done Sent false
27293 8429 Acked Read 1 Done Done Sent false
27294 8430 Acked Write 6 Done Done Sent false
27295 8431 Acked Flush 0 Done Done Sent false
27296 8432 Acked Flush 0 Done Done Sent false
27297 8433 Acked Read 10 Done Done Sent false
27298 8434 Acked Read 2 Done Done Sent false
27299 8435 Acked Write 5 Done Done Sent false
27300 8436 Acked Read 1 Done Done Sent false
27301 8437 Acked Write 1 Done Done Sent false
27302 8438 Acked Write 1 Done Done Sent false
27303 8439 Acked Flush 0 Done Done Sent false
27304 8440 Acked Write 5 Done Done Sent false
27305 8441 Acked Write 4 Done Done Sent false
27306 8442 Acked Read 8 Done Done Sent false
27307 8443 Acked Read 3 Done Done Sent false
27308 8444 Acked Read 4 Done Done Sent false
27309 8445 Acked Read 3 Done Done Sent false
27310 8446 Acked Write 8 Done Done Sent false
27311 8447 Acked Read 1 Done Done Sent false
27312 8448 Acked Read 4 Done Done Sent false
27313 8449 Acked Read 10 Done Done Sent false
27314 8450 Acked Read 5 Done Done Sent false
27315 8451 Acked Write 9 Done Done Sent false
27316 8452 Acked Read 9 Done Done Sent false
27317 8453 Acked Read 9 Done Done Sent false
27318 8454 Acked Write 3 Done Done Sent false
27319 8455 Acked Read 7 Done Done Sent false
27320 8456 Acked Read 10 Done Done Sent false
27321 8457 Acked Write 4 Done Done Sent false
27322 8458 Acked Flush 0 Done Done Sent false
27323 8459 Acked Write 6 Done Done Sent false
27324 8460 Acked Write 2 Done Done Sent false
27325 8461 Acked Read 9 Done Done Sent false
27326 8462 Acked Write 1 Done Done Sent false
27327 8463 Acked Write 5 Done Done Sent false
27328 8464 Acked Write 10 Done Done Sent false
27329 8465 Acked Read 3 Done Done Sent false
27330 8466 Acked Read 9 Done Done Sent false
27331 8467 Acked Read 9 Done Done Sent false
27332 8468 Acked Read 7 Done Done Sent false
27333 8469 Acked Write 3 Done Done Sent false
27334 8470 Acked Write 5 Done Done Sent false
27335 8471 Acked Flush 0 Done Done Sent false
27336 8472 Acked Write 7 Done Done Sent false
27337 8473 Acked Read 3 Done Done Sent false
27338 8474 Acked Flush 0 Done Done Sent false
27339 8475 Acked Write 8 Done Done Sent false
27340 8476 Acked Read 4 Done Done Sent false
27341 8477 Acked Write 7 Done Done Sent false
27342 8478 Acked Read 4 Done Done Sent false
27343 8479 Acked Flush 0 Done Done Sent false
27344 8480 Acked Read 2 Done Done Sent false
27345 8481 Acked Write 7 Done Done Sent false
27346 8482 Acked Read 10 Done Done Sent false
27347 8483 Acked Read 1 Done Done Sent false
27348 8484 Acked Write 8 Done Done Sent false
27349 8485 Acked Write 7 Done Done Sent false
27350 8486 Acked Read 2 Done Done Sent false
27351 8487 Acked Write 3 Done Done Sent false
27352 8488 Acked Write 6 Done Done Sent false
27353 8489 Acked Write 8 Done Done Sent false
27354 8490 Acked Read 2 Done Done Sent false
27355 8491 Acked Read 9 Done Done Sent false
27356 8492 Acked Read 6 Done Done Sent false
27357 8493 Acked Write 9 Done Done Sent false
27358 8494 Acked Read 2 Done Done Sent false
27359 8495 Acked Read 10 Done Done Sent false
27360 8496 Acked Write 4 Done Done Sent false
27361 8497 Acked Write 3 Done Done Sent false
27362 8498 Acked Write 4 Done Done Sent false
27363 8499 Acked Write 5 Done Done Sent false
27364 8500 Acked Write 3 Done Done Sent false
27365 8501 Acked Write 8 Done Done Sent false
27366 8502 Acked Write 8 Done Done Sent false
27367 8503 Acked Flush 0 Done Done Sent false
27368 8504 Acked Write 7 Done Done Sent false
27369 8505 Acked Write 1 Done Done Sent false
27370 8506 Acked Read 8 Done Done Sent false
27371 8507 Acked Read 7 Done Done Sent false
27372 8508 Acked Write 9 Done Done Sent false
27373 8509 Acked Write 9 Done Done Sent false
27374 8510 Acked Write 3 Done Done Sent false
27375 8511 Acked Read 4 Done Done Sent false
27376 8512 Acked Read 10 Done Done Sent false
27377 8513 Acked Read 8 Done Done Sent false
27378 8514 Acked Read 5 Done Done Sent false
27379 8515 Acked Write 7 Done Done Sent false
27380 8516 Acked Write 4 Done Done Sent false
27381 8517 Acked Read 6 Done Done Sent false
27382 8518 Acked Read 1 Done Done Sent false
27383 8519 Acked Read 7 Done Done Sent false
27384 8520 Acked Write 8 Done Done Sent false
27385 8521 Acked Read 2 Done Done Sent false
27386 8522 Acked Read 7 Done Done Sent false
27387 8523 Acked Read 10 Done Done Sent false
27388 8524 Acked Write 6 Done Done Sent false
27389 8525 Acked Read 7 Done Done Sent false
27390 8526 Acked Read 8 Done Done Sent false
27391 8527 Acked Flush 0 Done Done Sent false
27392 8528 Acked Write 1 Done Done Sent false
27393 8529 Acked Flush 0 Done Done Sent false
27394 8530 Acked Read 5 Done Done Sent false
27395 8531 Acked Read 8 Done Done Sent false
27396 8532 Acked Flush 0 Done Done Sent false
27397 8533 Acked Read 7 Done Done Sent false
27398 8534 Acked Flush 0 Done Done Sent false
27399 8535 Acked Read 6 Done Done Sent false
27400 8536 Acked Write 2 Done Done Sent false
27401 8537 Acked Write 8 Done Done Sent false
27402 8538 Acked Read 7 Done Done Sent false
27403 8539 Acked Write 1 Done Done Sent false
27404 8540 Acked Read 8 Done Done Sent false
27405 8541 Acked Read 7 Done Done Sent false
27406 8542 Acked Read 3 Done Done Sent false
27407 8543 Acked Read 2 Done Done Sent false
27408 8544 Acked Read 8 Done Done Sent false
27409 8545 Acked Read 7 Done Done Sent false
27410 8546 Acked Read 9 Done Done Sent false
27411 8547 Acked Read 3 Done Done Sent false
27412 STATES DS:0 DS:1 DS:2 TOTAL
27413 Sent 0 0 301 301
27414 Done 301 301 0 602
27415 Skipped 0 0 0 0
27416 Error 0 0 0 0
27417 Last Flush: 8534 8534 8246
27418 Downstairs last five completed: 8246 8245 8244 8243 8242
27419 Upstairs last five completed: 8547 8546 8545 8544 8543
27420 CLIENT: Up:0 ds:301 act:2
27421 [8830] got start action
27422 [8830] I am going to restart
27423 Starting downstairs at port 8830
27424 Make output file at "/tmp/dsc/downstairs-8830.txt"
27425 [8830][2] reports Starting
27426 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1226)
27427 [8830][2] reports Running
27428 ----------------------------------------------------------------
27429 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
27430 States: Active Active Offline
27431 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27432 8247 Acked Read 6 Done Done Sent false
27433 8248 Acked Flush 0 Done Done Sent false
27434 8249 Acked Write 7 Done Done Sent false
27435 8250 Acked Read 10 Done Done Sent false
27436 8251 Acked Write 10 Done Done Sent false
27437 8252 Acked Read 7 Done Done Sent false
27438 8253 Acked Read 2 Done Done Sent false
27439 8254 Acked Read 2 Done Done Sent false
27440 8255 Acked Read 5 Done Done Sent false
27441 8256 Acked Read 2 Done Done Sent false
27442 8257 Acked Read 10 Done Done Sent false
27443 8258 Acked Write 6 Done Done Sent false
27444 8259 Acked Write 6 Done Done Sent false
27445 8260 Acked Write 9 Done Done Sent false
27446 8261 Acked Write 10 Done Done Sent false
27447 8262 Acked Read 5 Done Done Sent false
27448 8263 Acked Write 6 Done Done Sent false
27449 8264 Acked Write 1 Done Done Sent false
27450 8265 Acked Read 4 Done Done Sent false
27451 8266 Acked Read 3 Done Done Sent false
27452 8267 Acked Flush 0 Done Done Sent false
27453 8268 Acked Read 2 Done Done Sent false
27454 8269 Acked Write 1 Done Done Sent false
27455 8270 Acked Write 2 Done Done Sent false
27456 8271 Acked Write 8 Done Done Sent false
27457 8272 Acked Read 10 Done Done Sent false
27458 8273 Acked Write 10 Done Done Sent false
27459 8274 Acked Write 8 Done Done Sent false
27460 8275 Acked Read 9 Done Done Sent false
27461 8276 Acked Read 2 Done Done Sent false
27462 8277 Acked Read 5 Done Done Sent false
27463 8278 Acked Read 1 Done Done Sent false
27464 8279 Acked Flush 0 Done Done Sent false
27465 8280 Acked Read 5 Done Done Sent false
27466 8281 Acked Write 4 Done Done Sent false
27467 8282 Acked Read 4 Done Done Sent false
27468 8283 Acked Write 5 Done Done Sent false
27469 8284 Acked Read 6 Done Done Sent false
27470 8285 Acked Flush 0 Done Done Sent false
27471 8286 Acked Write 6 Done Done Sent false
27472 8287 Acked Write 6 Done Done Sent false
27473 8288 Acked Write 1 Done Done Sent false
27474 8289 Acked Read 5 Done Done Sent false
27475 8290 Acked Write 9 Done Done Sent false
27476 8291 Acked Read 2 Done Done Sent false
27477 8292 Acked Read 7 Done Done Sent false
27478 8293 Acked Read 10 Done Done Sent false
27479 8294 Acked Write 2 Done Done Sent false
27480 8295 Acked Write 2 Done Done Sent false
27481 8296 Acked Write 6 Done Done Sent false
27482 8297 Acked Flush 0 Done Done Sent false
27483 8298 Acked Read 10 Done Done Sent false
27484 8299 Acked Read 7 Done Done Sent false
27485 8300 Acked Read 8 Done Done Sent false
27486 8301 Acked Write 3 Done Done Sent false
27487 8302 Acked Write 4 Done Done Sent false
27488 8303 Acked Read 8 Done Done Sent false
27489 8304 Acked Read 7 Done Done Sent false
27490 8305 Acked Read 1 Done Done Sent false
27491 8306 Acked Write 6 Done Done Sent false
27492 8307 Acked Read 10 Done Done Sent false
27493 8308 Acked Read 8 Done Done Sent false
27494 8309 Acked Write 10 Done Done Sent false
27495 8310 Acked Flush 0 Done Done Sent false
27496 8311 Acked Read 5 Done Done Sent false
27497 8312 Acked Read 5 Done Done Sent false
27498 8313 Acked Read 2 Done Done Sent false
27499 8314 Acked Flush 0 Done Done Sent false
27500 8315 Acked Write 7 Done Done Sent false
27501 8316 Acked Read 1 Done Done Sent false
27502 8317 Acked Write 5 Done Done Sent false
27503 8318 Acked Read 9 Done Done Sent false
27504 8319 Acked Read 7 Done Done Sent false
27505 8320 Acked Write 10 Done Done Sent false
27506 8321 Acked Read 3 Done Done Sent false
27507 8322 Acked Read 7 Done Done Sent false
27508 8323 Acked Flush 0 Done Done Sent false
27509 8324 Acked Read 6 Done Done Sent false
27510 8325 Acked Write 1 Done Done Sent false
27511 8326 Acked Read 9 Done Done Sent false
27512 8327 Acked Write 9 Done Done Sent false
27513 8328 Acked Read 3 Done Done Sent false
27514 8329 Acked Write 8 Done Done Sent false
27515 8330 Acked Write 5 Done Done Sent false
27516 8331 Acked Read 7 Done Done Sent false
27517 8332 Acked Write 7 Done Done Sent false
27518 8333 Acked Read 1 Done Done Sent false
27519 8334 Acked Write 7 Done Done Sent false
27520 8335 Acked Flush 0 Done Done Sent false
27521 8336 Acked Read 7 Done Done Sent false
27522 8337 Acked Read 7 Done Done Sent false
27523 8338 Acked Write 7 Done Done Sent false
27524 8339 Acked Write 6 Done Done Sent false
27525 8340 Acked Read 9 Done Done Sent false
27526 8341 Acked Write 7 Done Done Sent false
27527 8342 Acked Read 3 Done Done Sent false
27528 8343 Acked Read 2 Done Done Sent false
27529 8344 Acked Write 1 Done Done Sent false
27530 8345 Acked Write 7 Done Done Sent false
27531 8346 Acked Read 3 Done Done Sent false
27532 8347 Acked Read 10 Done Done Sent false
27533 8348 Acked Read 10 Done Done Sent false
27534 8349 Acked Read 6 Done Done Sent false
27535 8350 Acked Read 2 Done Done Sent false
27536 8351 Acked Read 10 Done Done Sent false
27537 8352 Acked Write 7 Done Done Sent false
27538 8353 Acked Write 1 Done Done Sent false
27539 8354 Acked Read 5 Done Done Sent false
27540 8355 Acked Read 9 Done Done Sent false
27541 8356 Acked Read 8 Done Done Sent false
27542 8357 Acked Write 4 Done Done Sent false
27543 8358 Acked Write 7 Done Done Sent false
27544 8359 Acked Read 8 Done Done Sent false
27545 8360 Acked Write 6 Done Done Sent false
27546 8361 Acked Write 10 Done Done Sent false
27547 8362 Acked Read 6 Done Done Sent false
27548 8363 Acked Read 6 Done Done Sent false
27549 8364 Acked Write 4 Done Done Sent false
27550 8365 Acked Read 3 Done Done Sent false
27551 8366 Acked Write 9 Done Done Sent false
27552 8367 Acked Read 6 Done Done Sent false
27553 8368 Acked Read 9 Done Done Sent false
27554 8369 Acked Write 9 Done Done Sent false
27555 8370 Acked Write 3 Done Done Sent false
27556 8371 Acked Read 3 Done Done Sent false
27557 8372 Acked Read 3 Done Done Sent false
27558 8373 Acked Write 6 Done Done Sent false
27559 8374 Acked Write 6 Done Done Sent false
27560 8375 Acked Write 3 Done Done Sent false
27561 8376 Acked Write 8 Done Done Sent false
27562 8377 Acked Write 2 Done Done Sent false
27563 8378 Acked Read 1 Done Done Sent false
27564 8379 Acked Read 2 Done Done Sent false
27565 8380 Acked Flush 0 Done Done Sent false
27566 8381 Acked Write 8 Done Done Sent false
27567 8382 Acked Read 9 Done Done Sent false
27568 8383 Acked Read 4 Done Done Sent false
27569 8384 Acked Write 4 Done Done Sent false
27570 8385 Acked Read 8 Done Done Sent false
27571 8386 Acked Read 3 Done Done Sent false
27572 8387 Acked Read 8 Done Done Sent false
27573 8388 Acked Read 3 Done Done Sent false
27574 8389 Acked Read 9 Done Done Sent false
27575 8390 Acked Read 7 Done Done Sent false
27576 8391 Acked Read 1 Done Done Sent false
27577 8392 Acked Read 9 Done Done Sent false
27578 8393 Acked Read 3 Done Done Sent false
27579 8394 Acked Read 7 Done Done Sent false
27580 8395 Acked Write 8 Done Done Sent false
27581 8396 Acked Read 1 Done Done Sent false
27582 8397 Acked Write 3 Done Done Sent false
27583 8398 Acked Write 3 Done Done Sent false
27584 8399 Acked Flush 0 Done Done Sent false
27585 8400 Acked Write 5 Done Done Sent false
27586 8401 Acked Read 6 Done Done Sent false
27587 8402 Acked Write 2 Done Done Sent false
27588 8403 Acked Read 4 Done Done Sent false
27589 8404 Acked Read 4 Done Done Sent false
27590 8405 Acked Write 10 Done Done Sent false
27591 8406 Acked Read 6 Done Done Sent false
27592 8407 Acked Flush 0 Done Done Sent false
27593 8408 Acked Read 1 Done Done Sent false
27594 8409 Acked Read 3 Done Done Sent false
27595 8410 Acked Write 6 Done Done Sent false
27596 8411 Acked Write 8 Done Done Sent false
27597 8412 Acked Write 10 Done Done Sent false
27598 8413 Acked Read 6 Done Done Sent false
27599 8414 Acked Write 3 Done Done Sent false
27600 8415 Acked Read 4 Done Done Sent false
27601 8416 Acked Write 2 Done Done Sent false
27602 8417 Acked Write 6 Done Done Sent false
27603 8418 Acked Read 6 Done Done Sent false
27604 8419 Acked Read 7 Done Done Sent false
27605 8420 Acked Write 7 Done Done Sent false
27606 8421 Acked Read 7 Done Done Sent false
27607 8422 Acked Flush 0 Done Done Sent false
27608 8423 Acked Write 2 Done Done Sent false
27609 8424 Acked Read 6 Done Done Sent false
27610 8425 Acked Flush 0 Done Done Sent false
27611 8426 Acked Write 10 Done Done Sent false
27612 8427 Acked Write 2 Done Done Sent false
27613 8428 Acked Read 3 Done Done Sent false
27614 8429 Acked Read 1 Done Done Sent false
27615 8430 Acked Write 6 Done Done Sent false
27616 8431 Acked Flush 0 Done Done Sent false
27617 8432 Acked Flush 0 Done Done Sent false
27618 8433 Acked Read 10 Done Done Sent false
27619 8434 Acked Read 2 Done Done Sent false
27620 8435 Acked Write 5 Done Done Sent false
27621 8436 Acked Read 1 Done Done Sent false
27622 8437 Acked Write 1 Done Done Sent false
27623 8438 Acked Write 1 Done Done Sent false
27624 8439 Acked Flush 0 Done Done Sent false
27625 8440 Acked Write 5 Done Done Sent false
27626 8441 Acked Write 4 Done Done Sent false
27627 8442 Acked Read 8 Done Done Sent false
27628 8443 Acked Read 3 Done Done Sent false
27629 8444 Acked Read 4 Done Done Sent false
27630 8445 Acked Read 3 Done Done Sent false
27631 8446 Acked Write 8 Done Done Sent false
27632 8447 Acked Read 1 Done Done Sent false
27633 8448 Acked Read 4 Done Done Sent false
27634 8449 Acked Read 10 Done Done Sent false
27635 8450 Acked Read 5 Done Done Sent false
27636 8451 Acked Write 9 Done Done Sent false
27637 8452 Acked Read 9 Done Done Sent false
27638 8453 Acked Read 9 Done Done Sent false
27639 8454 Acked Write 3 Done Done Sent false
27640 8455 Acked Read 7 Done Done Sent false
27641 8456 Acked Read 10 Done Done Sent false
27642 8457 Acked Write 4 Done Done Sent false
27643 8458 Acked Flush 0 Done Done Sent false
27644 8459 Acked Write 6 Done Done Sent false
27645 8460 Acked Write 2 Done Done Sent false
27646 8461 Acked Read 9 Done Done Sent false
27647 8462 Acked Write 1 Done Done Sent false
27648 8463 Acked Write 5 Done Done Sent false
27649 8464 Acked Write 10 Done Done Sent false
27650 8465 Acked Read 3 Done Done Sent false
27651 8466 Acked Read 9 Done Done Sent false
27652 8467 Acked Read 9 Done Done Sent false
27653 8468 Acked Read 7 Done Done Sent false
27654 8469 Acked Write 3 Done Done Sent false
27655 8470 Acked Write 5 Done Done Sent false
27656 8471 Acked Flush 0 Done Done Sent false
27657 8472 Acked Write 7 Done Done Sent false
27658 8473 Acked Read 3 Done Done Sent false
27659 8474 Acked Flush 0 Done Done Sent false
27660 8475 Acked Write 8 Done Done Sent false
27661 8476 Acked Read 4 Done Done Sent false
27662 8477 Acked Write 7 Done Done Sent false
27663 8478 Acked Read 4 Done Done Sent false
27664 8479 Acked Flush 0 Done Done Sent false
27665 8480 Acked Read 2 Done Done Sent false
27666 8481 Acked Write 7 Done Done Sent false
27667 8482 Acked Read 10 Done Done Sent false
27668 8483 Acked Read 1 Done Done Sent false
27669 8484 Acked Write 8 Done Done Sent false
27670 8485 Acked Write 7 Done Done Sent false
27671 8486 Acked Read 2 Done Done Sent false
27672 8487 Acked Write 3 Done Done Sent false
27673 8488 Acked Write 6 Done Done Sent false
27674 8489 Acked Write 8 Done Done Sent false
27675 8490 Acked Read 2 Done Done Sent false
27676 8491 Acked Read 9 Done Done Sent false
27677 8492 Acked Read 6 Done Done Sent false
27678 8493 Acked Write 9 Done Done Sent false
27679 8494 Acked Read 2 Done Done Sent false
27680 8495 Acked Read 10 Done Done Sent false
27681 8496 Acked Write 4 Done Done Sent false
27682 8497 Acked Write 3 Done Done Sent false
27683 8498 Acked Write 4 Done Done Sent false
27684 8499 Acked Write 5 Done Done Sent false
27685 8500 Acked Write 3 Done Done Sent false
27686 8501 Acked Write 8 Done Done Sent false
27687 8502 Acked Write 8 Done Done Sent false
27688 8503 Acked Flush 0 Done Done Sent false
27689 8504 Acked Write 7 Done Done Sent false
27690 8505 Acked Write 1 Done Done Sent false
27691 8506 Acked Read 8 Done Done Sent false
27692 8507 Acked Read 7 Done Done Sent false
27693 8508 Acked Write 9 Done Done Sent false
27694 8509 Acked Write 9 Done Done Sent false
27695 8510 Acked Write 3 Done Done Sent false
27696 8511 Acked Read 4 Done Done Sent false
27697 8512 Acked Read 10 Done Done Sent false
27698 8513 Acked Read 8 Done Done Sent false
27699 8514 Acked Read 5 Done Done Sent false
27700 8515 Acked Write 7 Done Done Sent false
27701 8516 Acked Write 4 Done Done Sent false
27702 8517 Acked Read 6 Done Done Sent false
27703 8518 Acked Read 1 Done Done Sent false
27704 8519 Acked Read 7 Done Done Sent false
27705 8520 Acked Write 8 Done Done Sent false
27706 8521 Acked Read 2 Done Done Sent false
27707 8522 Acked Read 7 Done Done Sent false
27708 8523 Acked Read 10 Done Done Sent false
27709 8524 Acked Write 6 Done Done Sent false
27710 8525 Acked Read 7 Done Done Sent false
27711 8526 Acked Read 8 Done Done Sent false
27712 8527 Acked Flush 0 Done Done Sent false
27713 8528 Acked Write 1 Done Done Sent false
27714 8529 Acked Flush 0 Done Done Sent false
27715 8530 Acked Read 5 Done Done Sent false
27716 8531 Acked Read 8 Done Done Sent false
27717 8532 Acked Flush 0 Done Done Sent false
27718 8533 Acked Read 7 Done Done Sent false
27719 8534 Acked Flush 0 Done Done Sent false
27720 8535 Acked Read 6 Done Done Sent false
27721 8536 Acked Write 2 Done Done Sent false
27722 8537 Acked Write 8 Done Done Sent false
27723 8538 Acked Read 7 Done Done Sent false
27724 8539 Acked Write 1 Done Done Sent false
27725 8540 Acked Read 8 Done Done Sent false
27726 8541 Acked Read 7 Done Done Sent false
27727 8542 Acked Read 3 Done Done Sent false
27728 8543 Acked Read 2 Done Done Sent false
27729 8544 Acked Read 8 Done Done Sent false
27730 8545 Acked Read 7 Done Done Sent false
27731 8546 Acked Read 9 Done Done Sent false
27732 8547 Acked Read 3 Done Done Sent false
27733 8548 Acked Flush 0 Done Done Sent false
27734 STATES DS:0 DS:1 DS:2 TOTAL
27735 Sent 0 0 302 302
27736 Done 302 302 0 604
27737 Skipped 0 0 0 0
27738 Error 0 0 0 0
27739 Last Flush: 8548 8548 8246
27740 Downstairs last five completed: 8246 8245 8244 8243 8242
27741 Upstairs last five completed: 8548 8547 8546 8545 8544
27742 CLIENT: Up:0 ds:302 act:2
277432024-12-09T15:13:39.066ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277442024-12-09T15:13:39.066ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41653) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277452024-12-09T15:13:39.067ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277462024-12-09T15:13:39.067ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277472024-12-09T15:13:39.067ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277482024-12-09T15:13:39.067ZINFOcrucible: send last flush ID to this DS: 8246 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277492024-12-09T15:13:39.068ZINFOcrucible: Replied this last flush ID: 8246 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277502024-12-09T15:13:39.068ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277512024-12-09T15:13:39.068ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277522024-12-09T15:13:39.068ZINFOcrucible: [2] client re-new 302 jobs since flush 8246 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277532024-12-09T15:13:39.068ZINFOcrucible: [2] Replayed 302 jobs since flush: 8246 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
27754 ----------------------------------------------------------------
27755 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
27756 Downstairs last five completed: 8548 8547 8546 8545 8544
27757 Upstairs last five completed: 8548 8547 8546 8545 8544
27758 CLIENT: Up:0 ds:0 act:3
27759 Replay: All jobs finished
27760 Main task has work to do, go find it
27761 got dsc Stop(1)
27762 stop 1
27763 Dec 09 15:13:41.316[8820] Got stop action so:false kr:false
27764 INFO request completed, latency_us: 357, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 1318a370-5c3f-4e98-82e5-4e8b30b2f8c1, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
27765 Dec 09 15:13:41.317 INFO request completed, latency_us: 119, response_code: 200, uri: /state/cid/1, method: GET, req_id: bc9c3f09-2511-4f42-8459-d073a200a48d, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
277662024-12-09T15:13:41.317ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277672024-12-09T15:13:41.317ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277682024-12-09T15:13:41.317ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277692024-12-09T15:13:41.317ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277702024-12-09T15:13:41.317ZINFOcrucible: [1] Transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
277712024-12-09T15:13:41.317ZINFOcrucible: client task is exiting = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
27772 [8820] Exited with: ExitStatus(unix_wait_status(9))
27773 [8820][1] reports Exit
27774 Dec 09 15:13:45.319 INFO request completed, latency_us: 231, response_code: 200, uri: /state/cid/1, method: GET, req_id: 96b6d138-ea87-4720-9d97-aa3a609a8d21, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
27775 001/300 Write block 1743 len 36864 data: 6 6 5 6 5 5 5 3 5
27776 002/300 Write block 121 len 12288 data: 5 6 4
27777 003/300 Read block 3454 len 32768
27778 004/300 Read block 4257 len 20480
27779 005/300 Read block 4006 len 4096
27780 006/300 Read block 1589 len 32768
27781 007/300 Write block 3531 len 4096 data: 9
27782 008/300 Read block 2052 len 4096
27783 009/300 Write block 2450 len 16384 data: 2 3 5 6
27784 010/300 Read block 4428 len 8192
27785 011/300 Read block 3937 len 28672
27786 012/300 Read block 3638 len 20480
27787 013/300 Read block 3779 len 12288
27788 014/300 Flush
27789 015/300 Read block 1753 len 32768
27790 016/300 Write block 3600 len 24576 data: 3 3 4 4 4 4
27791 017/300 Read block 2150 len 32768
27792 018/300 Read block 777 len 16384
27793 019/300 Read block 3735 len 4096
27794 020/300 Read block 595 len 40960
27795 021/300 Read block 3174 len 16384
27796 022/300 Write block 3104 len 20480 data: 6 7 6 5 4
27797 023/300 Read block 1799 len 24576
27798 024/300 Write block 642 len 20480 data: 5 5 5 5 4
27799 025/300 Read block 1127 len 8192
27800 026/300 Write block 4780 len 12288 data: 4 5 5
27801 027/300 Read block 2870 len 32768
27802 028/300 Read block 4460 len 16384
27803 029/300 Write block 521 len 12288 data: 4 5 5
27804 030/300 Write block 322 len 16384 data: 10 8 8 6
27805 031/300 Read block 1116 len 28672
27806 032/300 Flush
27807 033/300 Read block 1140 len 12288
27808 034/300 Write block 1116 len 28672 data: 5 5 4 4 4 4 3
27809 035/300 Read block 2657 len 12288
27810 036/300 Write block 1462 len 12288 data: 3 5 4
27811 037/300 Read block 3542 len 16384
27812 038/300 Read block 1577 len 28672
27813 039/300 Read block 1698 len 28672
27814 040/300 Read block 2995 len 24576
27815 041/300 Flush
27816 042/300 Write block 3088 len 16384 data: 3 5 5 5
27817 043/300 Write block 1643 len 20480 data: 3 4 4 4 4
27818 044/300 Write block 526 len 32768 data: 5 6 6 5 5 4 4 4
27819 045/300 Write block 3547 len 16384 data: 5 6 7 7
27820 046/300 Flush
27821 047/300 Write block 2640 len 16384 data: 4 5 5 5
27822 048/300 Read block 575 len 28672
27823 049/300 Read block 3926 len 24576
27824 050/300 Write block 2086 len 32768 data: 4 3 4 5 5 5 3 3
27825 051/300 Read block 4745 len 24576
27826 052/300 Write block 1713 len 8192 data: 8 7
27827 053/300 Write block 402 len 16384 data: 6 7 8 8
27828 054/300 Write block 2335 len 4096 data: 8
27829 055/300 Read block 3599 len 32768
27830 056/300 Read block 3952 len 28672
27831 057/300 Read block 1614 len 20480
27832 058/300 Write block 4701 len 4096 data: 7
27833 059/300 Write block 3036 len 16384 data: 3 2 2 3
27834 060/300 Write block 443 len 24576 data: 3 3 3 4 5 6
27835 061/300 Flush
27836 062/300 Flush
27837 063/300 Read block 4632 len 24576
27838 064/300 Read block 1363 len 40960
27839 065/300 Write block 533 len 28672 data: 5 5 4 6 6 6 5
27840 066/300 Read block 452 len 20480
27841 067/300 Read block 3360 len 28672
27842 068/300 Read block 821 len 20480
27843 069/300 Read block 3071 len 8192
27844 070/300 Write block 399 len 16384 data: 4 4 5 7
27845 071/300 Write block 2802 len 32768 data: 5 6 6 6 6 6 5 5
27846 072/300 Write block 3444 len 20480 data: 5 5 4 5 4
27847 073/300 Read block 3624 len 4096
27848 074/300 Read block 775 len 16384
27849 075/300 Write block 2442 len 4096 data: 6
27850 076/300 Read block 436 len 20480
27851 077/300 Write block 3829 len 28672 data: 8 7 7 7 7 5 6
27852 078/300 Write block 3903 len 24576 data: 4 5 5 5 5 5
27853 079/300 Read block 3422 len 16384
27854 080/300 Write block 4202 len 24576 data: 4 3 4 3 3 3
27855 081/300 Write block 2867 len 4096 data: 9
27856 082/300 Read block 1017 len 40960
27857 083/300 Read block 1359 len 36864
27858 084/300 Write block 935 len 32768 data: 3 3 3 3 4 5 6 5
27859 085/300 Write block 1123 len 28672 data: 4 4 5 6 6 5 5
27860 086/300 Read block 2172 len 8192
27861 087/300 Write block 3372 len 40960 data: 4 4 4 4 6 6 6 6 4 4
27862 088/300 Write block 1088 len 40960 data: 6 7 8 7 6 7 8 6 5 5
27863 089/300 Read block 3928 len 28672
27864 090/300 Read block 4210 len 32768
27865 091/300 Read block 2687 len 16384
27866 092/300 Write block 1734 len 8192 data: 8 6
27867 093/300 Write block 775 len 4096 data: 5
27868 094/300 Write block 1243 len 16384 data: 5 4 5 7
27869 095/300 Write block 985 len 36864 data: 4 3 4 6 7 7 6 4 3
27870 096/300 Write block 1000 len 24576 data: 4 4 5 6 7 7
27871 097/300 Read block 1814 len 20480
27872 098/300 Flush
27873 099/300 Write block 4528 len 36864 data: 6 6 5 5 5 5 5 6 4
27874 100/300 Write block 2864 len 36864 data: 6 6 7 10 9 10 7 7 6
27875 101/300 Read block 1213 len 16384
27876 102/300 Flush
27877 103/300 Flush
27878 104/300 Read block 4436 len 32768
27879 105/300 Write block 1389 len 4096 data: 5
27880 106/300 Read block 602 len 28672
27881 107/300 Write block 4620 len 20480 data: 5 5 5 6 5
27882 108/300 Read block 2565 len 8192
27883 109/300 Read block 3456 len 32768
27884 110/300 Write block 4965 len 40960 data: 6 5 4 4 4 3 4 3 3 3
27885 111/300 Write block 1557 len 28672 data: 5 4 3 9 11 10 10
27886 112/300 Read block 535 len 8192
27887 113/300 Write block 4470 len 4096 data: 9
27888 114/300 Read block 4260 len 8192
27889 115/300 Write block 3763 len 20480 data: 7 6 4 4 3
27890 116/300 Read block 2527 len 28672
27891 117/300 Write block 3854 len 4096 data: 5
27892 118/300 Write block 4993 len 4096 data: 8
27893 119/300 Read block 2218 len 12288
27894 120/300 Read block 4527 len 36864
27895 121/300 Read block 1977 len 36864
27896 122/300 Write block 2976 len 28672 data: 3 3 3 3 3 4 4
27897 123/300 Flush
27898 124/300 Read block 2940 len 36864
27899 125/300 Read block 2652 len 16384
27900 126/300 Write block 4302 len 20480 data: 5 5 4 4 3
27901 127/300 Read block 3441 len 24576
27902 128/300 Read block 2098 len 16384
27903 129/300 Read block 3142 len 24576
27904 130/300 Read block 2514 len 36864
27905 131/300 Read block 3416 len 16384
27906 132/300 Read block 599 len 28672
27907 133/300 Write block 703 len 20480 data: 4 5 5 3 3
27908 134/300 Write block 1114 len 40960 data: 7 6 6 6 5 5 5 5 4 5
27909 135/300 Read block 2454 len 8192
27910 136/300 Read block 835 len 20480
27911 137/300 Read block 3576 len 4096
27912 138/300 Read block 543 len 40960
27913 139/300 Read block 2422 len 20480
27914 140/300 Write block 1984 len 12288 data: 10 8 6
27915 141/300 Flush
27916 142/300 Write block 960 len 20480 data: 5 5 4 5 4
27917 143/300 Write block 3705 len 24576 data: 3 2 2 4 5 5
27918 144/300 Read block 2236 len 32768
27919 145/300 Read block 2528 len 16384
27920 146/300 Read block 2581 len 40960
27921 147/300 Read block 2100 len 24576
27922 148/300 Write block 4003 len 12288 data: 6 6 6
27923 149/300 Flush
27924 150/300 Read block 677 len 8192
27925 151/300 Read block 1812 len 36864
27926 152/300 Read block 1738 len 8192
27927 153/300 Read block 4159 len 36864
27928 154/300 Write block 2072 len 4096 data: 4
27929 155/300 Read block 3421 len 4096
27930 156/300 Read block 636 len 16384
27931 157/300 Flush
27932 158/300 Write block 528 len 32768 data: 7 6 6 5 5 6 6 5
27933 159/300 Read block 4533 len 4096
27934 160/300 Write block 4302 len 40960 data: 6 6 5 5 4 4 3 4 4 4
27935 161/300 Read block 2374 len 28672
27936 162/300 Write block 3074 len 32768 data: 5 6 4 4 5 5 4 5
27937 163/300 Write block 1344 len 28672 data: 5 3 3 3 4 4 4
27938 164/300 Write block 458 len 16384 data: 6 6 7 6
27939 165/300 Read block 3681 len 40960
27940 166/300 Write block 4833 len 20480 data: 4 5 6 6 6
27941 167/300 Write block 2804 len 16384 data: 7 7 7 7
27942 168/300 Read block 2509 len 12288
27943 169/300 Read block 1935 len 32768
27944 170/300 Write block 4304 len 36864 data: 6 6 5 5 4 5 5 5 6
27945 171/300 Write block 3083 len 40960 data: 3 3 3 3 3 4 6 6 6 5
27946 172/300 Write block 466 len 4096 data: 5
27947 173/300 Read block 1826 len 4096
27948 174/300 Read block 831 len 28672
27949 175/300 Read block 1720 len 12288
27950 176/300 Write block 3560 len 24576 data: 5 7 5 4 5 7
27951 177/300 Flush
27952 178/300 Read block 3746 len 32768
27953 179/300 Write block 3385 len 8192 data: 6 6
27954 180/300 Write block 557 len 32768 data: 6 4 3 3 3 3 3 3
27955 181/300 Read block 37 len 36864
27956 182/300 Write block 2652 len 36864 data: 6 7 9 10 9 6 5 5 5
27957 183/300 Read block 3771 len 28672
27958 184/300 Write block 4642 len 28672 data: 6 6 7 8 9 10 10
27959 185/300 Read block 3006 len 16384
27960 186/300 Read block 2977 len 28672
27961 187/300 Read block 4567 len 40960
27962 188/300 Write block 3213 len 12288 data: 6 6 6
27963 189/300 Read block 1600 len 24576
27964 190/300 Read block 3805 len 36864
27965 191/300 Read block 3369 len 40960
27966 192/300 Read block 4895 len 36864
27967 193/300 Write block 2068 len 24576 data: 4 4 4 4 5 5
27968 194/300 Write block 1570 len 36864 data: 7 6 6 6 7 7 6 8 8
27969 195/300 Read block 4317 len 4096
27970 196/300 Write block 3900 len 32768 data: 3 3 3 5 6 6 6 6
27971 197/300 Flush
27972 198/300 Write block 4541 len 36864 data: 4 4 4 4 5 5 6 5 7
27973 199/300 Read block 215 len 20480
27974 200/300 Flush
27975 201/300 Read block 4905 len 8192
27976 202/300 Write block 3157 len 24576 data: 8 8 9 10 11 11
27977 203/300 Write block 1664 len 32768 data: 8 8 7 6 6 6 4 4
27978 204/300 Read block 4763 len 20480
27979 205/300 Write block 4565 len 8192 data: 4 6
27980 206/300 Write block 3783 len 8192 data: 8 5
27981 207/300 Read block 1819 len 8192
27982 208/300 Read block 1159 len 28672
27983 209/300 Write block 4614 len 40960 data: 4 4 5 4 4 5 6 6 6 7
27984 210/300 Read block 224 len 12288
27985 211/300 Write block 3884 len 28672 data: 4 5 7 5 5 5 3
27986 212/300 Read block 3192 len 32768
27987 213/300 Read block 3513 len 8192
27988 214/300 Read block 2362 len 20480
27989 215/300 Flush
27990 216/300 Write block 946 len 40960 data: 2 5 7 7 7 7 6 7 7 5
27991 217/300 Write block 1167 len 40960 data: 5 4 4 4 4 4 4 7 6 6
27992 218/300 Write block 3252 len 32768 data: 4 4 5 5 7 6 6 7
27993 219/300 Read block 2899 len 40960
27994 220/300 Write block 2448 len 28672 data: 5 4 3 4 6 7 6
27995 221/300 Read block 346 len 36864
27996 222/300 Write block 1316 len 24576 data: 7 6 7 5 5 6
27997 223/300 Flush
27998 224/300 Flush
27999 225/300 Write block 312 len 8192 data: 4 5
28000 226/300 Write block 4824 len 40960 data: 4 6 4 4 5 4 5 3 5 5
28001 227/300 Write block 3561 len 4096 data: 8
28002 228/300 Flush
28003 229/300 Read block 2360 len 36864
28004 230/300 Write block 3865 len 20480 data: 5 4 4 6 6
28005 231/300 Write block 3487 len 36864 data: 4 5 5 5 4 4 4 4 4
28006 232/300 Read block 1076 len 40960
28007 233/300 Read block 3958 len 28672
28008 234/300 Read block 3484 len 24576
28009 235/300 Write block 3543 len 20480 data: 6 6 7 5 6
28010 236/300 Read block 1374 len 12288
28011 237/300 Flush
28012 238/300 Write block 2210 len 4096 data: 3
28013 239/300 Write block 2630 len 36864 data: 4 4 4 4 3 4 4 3 3
28014 240/300 Read block 4747 len 40960
28015 241/300 Read block 1938 len 8192
28016 242/300 Read block 2163 len 32768
28017 243/300 Read block 569 len 4096
28018 244/300 Read block 586 len 16384
28019 245/300 Write block 3657 len 32768 data: 6 5 5 5 5 7 6 6
28020 246/300 Read block 745 len 12288
28021 247/300 Read block 3949 len 12288
28022 248/300 Read block 1173 len 36864
28023 249/300 Flush
28024 250/300 Read block 1855 len 20480
28025 251/300 Write block 809 len 40960 data: 5 5 5 5 6 5 5 5 5 5
28026 252/300 Read block 4453 len 16384
28027 253/300 Write block 3010 len 20480 data: 7 6 7 7 6
28028 254/300 Read block 2028 len 4096
28029 255/300 Write block 2455 len 24576 data: 5 7 6 7 7 5
28030 256/300 Write block 2283 len 16384 data: 5 5 5 3
28031 257/300 Read block 3823 len 4096
28032 258/300 Write block 383 len 32768 data: 3 2 2 2 2 2 3 4
28033 259/300 Read block 2331 len 24576
28034 260/300 Write block 4453 len 32768 data: 5 5 4 4 2 2 2 3
28035 261/300 Read block 720 len 16384
28036 262/300 Write block 1337 len 36864 data: 12 10 9 7 10 6 6 6 4
28037 263/300 Write block 3067 len 8192 data: 4 3
28038 264/300 Read block 395 len 36864
28039 265/300 Flush
28040 266/300 Write block 1678 len 32768 data: 3 3 3 3 3 4 3 4
28041 267/300 Write block 151 len 16384 data: 7 7 7 7
28042 268/300 Write block 3759 len 20480 data: 8 7 7 8 8
28043 269/300 Read block 1381 len 40960
28044 270/300 Write block 1638 len 28672 data: 4 4 4 4 3 4 5
28045 271/300 Write block 3357 len 4096 data: 6
28046 272/300 Write block 4340 len 20480 data: 4 4 4 6 6
28047 273/300 Read block 1026 len 36864
28048 274/300 Read block 3284 len 4096
28049 275/300 Write block 1479 len 32768 data: 5 5 4 5 4 4 4 5
28050 276/300 Write block 3670 len 20480 data: 4 5 4 5 5
28051 277/300 Write block 252 len 40960 data: 8 8 6 6 5 4 5 4 4 4
28052 278/300 Write block 1873 len 4096 data: 4
28053 279/300 Read block 3408 len 12288
28054 280/300 Read block 3582 len 36864
28055 281/300 Write block 2679 len 36864 data: 6 6 5 5 4 7 7 7 7
28056 282/300 Read block 620 len 4096
28057 283/300 Read block 2348 len 4096
28058 284/300 Write block 535 len 16384 data: 6 7 7 7
28059 285/300 Write block 1894 len 32768 data: 6 6 4 4 4 5 4 4
28060 286/300 Read block 4703 len 4096
28061 287/300 Write block 1975 len 32768 data: 4 4 4 6 7 8 10 9
28062 288/300 Write block 2935 len 36864 data: 9 7 7 8 8 7 7 7 6
28063 289/300 Write block 3789 len 16384 data: 5 4 3 5
28064 290/300 Read block 4894 len 36864
28065 291/300 Read block 2672 len 16384
28066 292/300 Read block 24 len 32768
28067 293/300 Read block 1928 len 12288
28068 294/300 Read block 3789 len 12288
28069 295/300 Write block 3594 len 12288 data: 5 5 8
28070 296/300 Read block 3647 len 20480
28071 297/300 Write block 3984 len 28672 data: 6 8 8 6 4 4 3
28072 298/300 Read block 3570 len 12288
28073 299/300 Read block 3026 len 4096
28074 300/300 Read block 1809 len 28672
28075 301/300 Write block 1271 len 16384 data: 7 8 8 8
28076 Main task has work to do, go find it
28077 got dsc Start(1)
28078 start 1
28079 Dec 09 15:13:45.557 INFO request completed, latency_us: 299, response_code: 204, uri: /start/cid/1, method: POST, req_id: 5313ea3c-2193-40ab-bd5c-eaab91a2504c, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
28080 Replay: started 1, returned:Ok(())
28081 ----------------------------------------------------------------
28082 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28083 States: Active Offline Active
28084 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28085 8549 Acked Write 9 Done Sent Done false
28086 8550 Acked Write 3 Done Sent Done false
28087 8551 Acked Read 8 Done Sent Done false
28088 8552 Acked Read 5 Done Sent Done false
28089 8553 Acked Read 1 Done Sent Done false
28090 8554 Acked Read 8 Done Sent Done false
28091 8555 Acked Write 1 Done Sent Done false
28092 8556 Acked Read 1 Done Sent Done false
28093 8557 Acked Write 4 Done Sent Done false
28094 8558 Acked Read 2 Done Sent Done false
28095 8559 Acked Read 7 Done Sent Done false
28096 8560 Acked Read 5 Done Sent Done false
28097 8561 Acked Read 3 Done Sent Done false
28098 8562 Acked Flush 0 Done Sent Done false
28099 8563 Acked Read 8 Done Sent Done false
28100 8564 Acked Write 6 Done Sent Done false
28101 8565 Acked Read 8 Done Sent Done false
28102 8566 Acked Read 4 Done Sent Done false
28103 8567 Acked Read 1 Done Sent Done false
28104 8568 Acked Read 10 Done Sent Done false
28105 8569 Acked Read 4 Done Sent Done false
28106 8570 Acked Write 5 Done Sent Done false
28107 8571 Acked Read 6 Done Sent Done false
28108 8572 Acked Write 5 Done Sent Done false
28109 8573 Acked Read 2 Done Sent Done false
28110 8574 Acked Write 3 Done Sent Done false
28111 8575 Acked Read 8 Done Sent Done false
28112 8576 Acked Read 4 Done Sent Done false
28113 8577 Acked Write 3 Done Sent Done false
28114 8578 Acked Write 4 Done Sent Done false
28115 8579 Acked Read 7 Done Sent Done false
28116 8580 Acked Flush 0 Done Sent Done false
28117 8581 Acked Read 3 Done Sent Done false
28118 8582 Acked Write 7 Done Sent Done false
28119 8583 Acked Read 3 Done Sent Done false
28120 8584 Acked Write 3 Done Sent Done false
28121 8585 Acked Read 4 Done Sent Done false
28122 8586 Acked Read 7 Done Sent Done false
28123 8587 Acked Read 7 Done Sent Done false
28124 8588 Acked Read 6 Done Sent Done false
28125 8589 Acked Flush 0 Done Sent Done false
28126 8590 Acked Write 4 Done Sent Done false
28127 8591 Acked Write 5 Done Sent Done false
28128 8592 Acked Write 8 Done Sent Done false
28129 8593 Acked Write 4 Done Sent Done false
28130 8594 Acked Flush 0 Done Sent Done false
28131 8595 Acked Write 4 Done Sent Done false
28132 8596 Acked Read 7 Done Sent Done false
28133 8597 Acked Read 6 Done Sent Done false
28134 8598 Acked Write 8 Done Sent Done false
28135 8599 Acked Read 6 Done Sent Done false
28136 8600 Acked Write 2 Done Sent Done false
28137 8601 Acked Write 4 Done Sent Done false
28138 8602 Acked Write 1 Done Sent Done false
28139 8603 Acked Read 8 Done Sent Done false
28140 8604 Acked Read 7 Done Sent Done false
28141 8605 Acked Read 5 Done Sent Done false
28142 8606 Acked Write 1 Done Sent Done false
28143 8607 Acked Write 4 Done Sent Done false
28144 8608 Acked Write 6 Done Sent Done false
28145 8609 Acked Flush 0 Done Sent Done false
28146 8610 Acked Flush 0 Done Sent Done false
28147 8611 Acked Read 6 Done Sent Done false
28148 8612 Acked Read 10 Done Sent Done false
28149 8613 Acked Write 7 Done Sent Done false
28150 8614 Acked Read 5 Done Sent Done false
28151 8615 Acked Read 7 Done Sent Done false
28152 8616 Acked Read 5 Done Sent Done false
28153 8617 Acked Read 2 Done Sent Done false
28154 8618 Acked Write 4 Done Sent Done false
28155 8619 Acked Write 8 Done Sent Done false
28156 8620 Acked Write 5 Done Sent Done false
28157 8621 Acked Read 1 Done Sent Done false
28158 8622 Acked Read 4 Done Sent Done false
28159 8623 Acked Write 1 Done Sent Done false
28160 8624 Acked Read 5 Done Sent Done false
28161 8625 Acked Write 7 Done Sent Done false
28162 8626 Acked Write 6 Done Sent Done false
28163 8627 Acked Read 4 Done Sent Done false
28164 8628 Acked Write 6 Done Sent Done false
28165 8629 Acked Write 1 Done Sent Done false
28166 8630 Acked Read 10 Done Sent Done false
28167 8631 Acked Read 9 Done Sent Done false
28168 8632 Acked Write 8 Done Sent Done false
28169 8633 Acked Write 7 Done Sent Done false
28170 8634 Acked Read 2 Done Sent Done false
28171 8635 Acked Write 10 Done Sent Done false
28172 8636 Acked Write 10 Done Sent Done false
28173 8637 Acked Read 7 Done Sent Done false
28174 8638 Acked Read 8 Done Sent Done false
28175 8639 Acked Read 4 Done Sent Done false
28176 8640 Acked Write 2 Done Sent Done false
28177 8641 Acked Write 1 Done Sent Done false
28178 8642 Acked Write 4 Done Sent Done false
28179 8643 Acked Write 9 Done Sent Done false
28180 8644 Acked Write 6 Done Sent Done false
28181 8645 Acked Read 5 Done Sent Done false
28182 8646 Acked Flush 0 Done Sent Done false
28183 8647 Acked Write 9 Done Sent Done false
28184 8648 Acked Write 9 Done Sent Done false
28185 8649 Acked Read 4 Done Sent Done false
28186 8650 Acked Flush 0 Done Sent Done false
28187 8651 Acked Flush 0 Done Sent Done false
28188 8652 Acked Read 8 Done Sent Done false
28189 8653 Acked Write 1 Done Sent Done false
28190 8654 Acked Read 7 Done Sent Done false
28191 8655 Acked Write 5 Done Sent Done false
28192 8656 Acked Read 2 Done Sent Done false
28193 8657 Acked Read 8 Done Sent Done false
28194 8658 Acked Write 10 Done Sent Done false
28195 8659 Acked Write 7 Done Sent Done false
28196 8660 Acked Read 2 Done Sent Done false
28197 8661 Acked Write 1 Done Sent Done false
28198 8662 Acked Read 2 Done Sent Done false
28199 8663 Acked Write 5 Done Sent Done false
28200 8664 Acked Read 7 Done Sent Done false
28201 8665 Acked Write 1 Done Sent Done false
28202 8666 Acked Write 1 Done Sent Done false
28203 8667 Acked Read 3 Done Sent Done false
28204 8668 Acked Read 9 Done Sent Done false
28205 8669 Acked Read 9 Done Sent Done false
28206 8670 Acked Write 7 Done Sent Done false
28207 8671 Acked Flush 0 Done Sent Done false
28208 8672 Acked Read 9 Done Sent Done false
28209 8673 Acked Read 4 Done Sent Done false
28210 8674 Acked Write 5 Done Sent Done false
28211 8675 Acked Read 6 Done Sent Done false
28212 8676 Acked Read 4 Done Sent Done false
28213 8677 Acked Read 6 Done Sent Done false
28214 8678 Acked Read 9 Done Sent Done false
28215 8679 Acked Read 4 Done Sent Done false
28216 8680 Acked Read 7 Done Sent Done false
28217 8681 Acked Write 5 Done Sent Done false
28218 8682 Acked Write 10 Done Sent Done false
28219 8683 Acked Read 2 Done Sent Done false
28220 8684 Acked Read 5 Done Sent Done false
28221 8685 Acked Read 1 Done Sent Done false
28222 8686 Acked Read 10 Done Sent Done false
28223 8687 Acked Read 5 Done Sent Done false
28224 8688 Acked Write 3 Done Sent Done false
28225 8689 Acked Flush 0 Done Sent Done false
28226 8690 Acked Write 5 Done Sent Done false
28227 8691 Acked Write 6 Done Sent Done false
28228 8692 Acked Read 8 Done Sent Done false
28229 8693 Acked Read 4 Done Sent Done false
28230 8694 Acked Read 10 Done Sent Done false
28231 8695 Acked Read 6 Done Sent Done false
28232 8696 Acked Write 3 Done Sent Done false
28233 8697 Acked Flush 0 Done Sent Done false
28234 8698 Acked Read 2 Done Sent Done false
28235 8699 Acked Read 9 Done Sent Done false
28236 8700 Acked Read 2 Done Sent Done false
28237 8701 Acked Read 9 Done Sent Done false
28238 8702 Acked Write 1 Done Sent Done false
28239 8703 Acked Read 1 Done Sent Done false
28240 8704 Acked Read 4 Done Sent Done false
28241 8705 Acked Flush 0 Done Sent Done false
28242 8706 Acked Write 8 Done Sent Done false
28243 8707 Acked Read 1 Done Sent Done false
28244 8708 Acked Write 10 Done Sent Done false
28245 8709 Acked Read 7 Done Sent Done false
28246 8710 Acked Write 8 Done Sent Done false
28247 8711 Acked Write 7 Done Sent Done false
28248 8712 Acked Write 4 Done Sent Done false
28249 8713 Acked Read 10 Done Sent Done false
28250 8714 Acked Write 5 Done Sent Done false
28251 8715 Acked Write 4 Done Sent Done false
28252 8716 Acked Read 3 Done Sent Done false
28253 8717 Acked Read 8 Done Sent Done false
28254 8718 Acked Write 9 Done Sent Done false
28255 8719 Acked Write 10 Done Sent Done false
28256 8720 Acked Write 1 Done Sent Done false
28257 8721 Acked Read 1 Done Sent Done false
28258 8722 Acked Read 7 Done Sent Done false
28259 8723 Acked Read 3 Done Sent Done false
28260 8724 Acked Write 6 Done Sent Done false
28261 8725 Acked Flush 0 Done Sent Done false
28262 8726 Acked Read 8 Done Sent Done false
28263 8727 Acked Write 2 Done Sent Done false
28264 8728 Acked Write 8 Done Sent Done false
28265 8729 Acked Read 9 Done Sent Done false
28266 8730 Acked Write 9 Done Sent Done false
28267 8731 Acked Read 7 Done Sent Done false
28268 8732 Acked Write 7 Done Sent Done false
28269 8733 Acked Read 4 Done Sent Done false
28270 8734 Acked Read 7 Done Sent Done false
28271 8735 Acked Read 10 Done Sent Done false
28272 8736 Acked Write 3 Done Sent Done false
28273 8737 Acked Read 6 Done Sent Done false
28274 8738 Acked Read 9 Done Sent Done false
28275 8739 Acked Read 10 Done Sent Done false
28276 8740 Acked Read 9 Done Sent Done false
28277 8741 Acked Write 6 Done Sent Done false
28278 8742 Acked Write 9 Done Sent Done false
28279 8743 Acked Read 1 Done Sent Done false
28280 8744 Acked Write 8 Done Sent Done false
28281 8745 Acked Flush 0 Done Sent Done false
28282 8746 Acked Write 9 Done Sent Done false
28283 8747 Acked Read 5 Done Sent Done false
28284 8748 Acked Flush 0 Done Sent Done false
28285 8749 Acked Read 2 Done Sent Done false
28286 8750 Acked Write 6 Done Sent Done false
28287 8751 Acked Write 8 Done Sent Done false
28288 8752 Acked Read 5 Done Sent Done false
28289 8753 Acked Write 2 Done Sent Done false
28290 8754 Acked Write 2 Done Sent Done false
28291 8755 Acked Read 2 Done Sent Done false
28292 8756 Acked Read 7 Done Sent Done false
28293 8757 Acked Write 10 Done Sent Done false
28294 8758 Acked Read 3 Done Sent Done false
28295 8759 Acked Write 7 Done Sent Done false
28296 8760 Acked Read 8 Done Sent Done false
28297 8761 Acked Read 2 Done Sent Done false
28298 8762 Acked Read 5 Done Sent Done false
28299 8763 Acked Flush 0 Done Sent Done false
28300 8764 Acked Write 10 Done Sent Done false
28301 8765 Acked Write 10 Done Sent Done false
28302 8766 Acked Write 8 Done Sent Done false
28303 8767 Acked Read 10 Done Sent Done false
28304 8768 Acked Write 7 Done Sent Done false
28305 8769 Acked Read 9 Done Sent Done false
28306 8770 Acked Write 6 Done Sent Done false
28307 8771 Acked Flush 0 Done Sent Done false
28308 8772 Acked Flush 0 Done Sent Done false
28309 8773 Acked Write 2 Done Sent Done false
28310 8774 Acked Write 10 Done Sent Done false
28311 8775 Acked Write 1 Done Sent Done false
28312 8776 Acked Flush 0 Done Sent Done false
28313 8777 Acked Read 9 Done Sent Done false
28314 8778 Acked Write 5 Done Sent Done false
28315 8779 Acked Write 9 Done Sent Done false
28316 8780 Acked Read 10 Done Sent Done false
28317 8781 Acked Read 7 Done Sent Done false
28318 8782 Acked Read 6 Done Sent Done false
28319 8783 Acked Write 5 Done Sent Done false
28320 8784 Acked Read 3 Done Sent Done false
28321 8785 Acked Flush 0 Done Sent Done false
28322 8786 Acked Write 1 Done Sent Done false
28323 8787 Acked Write 9 Done Sent Done false
28324 8788 Acked Read 10 Done Sent Done false
28325 8789 Acked Read 2 Done Sent Done false
28326 8790 Acked Read 8 Done Sent Done false
28327 8791 Acked Read 1 Done Sent Done false
28328 8792 Acked Read 4 Done Sent Done false
28329 8793 Acked Write 8 Done Sent Done false
28330 8794 Acked Read 3 Done Sent Done false
28331 8795 Acked Read 3 Done Sent Done false
28332 8796 Acked Read 9 Done Sent Done false
28333 8797 Acked Flush 0 Done Sent Done false
28334 8798 Acked Read 5 Done Sent Done false
28335 8799 Acked Write 10 Done Sent Done false
28336 8800 Acked Read 4 Done Sent Done false
28337 8801 Acked Write 5 Done Sent Done false
28338 8802 Acked Read 1 Done Sent Done false
28339 8803 Acked Write 6 Done Sent Done false
28340 8804 Acked Write 4 Done Sent Done false
28341 8805 Acked Read 1 Done Sent Done false
28342 8806 Acked Write 8 Done Sent Done false
28343 8807 Acked Read 6 Done Sent Done false
28344 8808 Acked Write 8 Done Sent Done false
28345 8809 Acked Read 4 Done Sent Done false
28346 8810 Acked Write 9 Done Sent Done false
28347 8811 Acked Write 2 Done Sent Done false
28348 8812 Acked Read 9 Done Sent Done false
28349 8813 Acked Flush 0 Done Sent Done false
28350 8814 Acked Write 8 Done Sent Done false
28351 8815 Acked Write 4 Done Sent Done false
28352 8816 Acked Write 5 Done Sent Done false
28353 8817 Acked Read 10 Done Sent Done false
28354 8818 Acked Write 7 Done Sent Done false
28355 8819 Acked Write 1 Done Sent Done false
28356 8820 Acked Write 5 Done Sent Done false
28357 8821 Acked Read 9 Done Sent Done false
28358 8822 Acked Read 1 Done Sent Done false
28359 8823 Acked Write 8 Done Sent Done false
28360 8824 Acked Write 5 Done Sent Done false
28361 8825 Acked Write 10 Done Sent Done false
28362 8826 Acked Write 1 Done Sent Done false
28363 8827 Acked Read 3 Done Sent Done false
28364 8828 Acked Read 9 Done Sent Done false
28365 8829 Acked Write 9 Done Sent Done false
28366 8830 Acked Read 1 Done Sent Done false
28367 8831 Acked Read 1 Done Sent Done false
28368 8832 Acked Write 4 Done Sent Done false
28369 8833 Acked Write 8 Done Sent Done false
28370 8834 Acked Read 1 Done Sent Done false
28371 8835 Acked Write 8 Done Sent Done false
28372 8836 Acked Write 9 Done Sent Done false
28373 8837 Acked Write 4 Done Sent Done false
28374 8838 Acked Read 9 Done Sent Done false
28375 8839 Acked Read 4 Done Sent Done false
28376 8840 Acked Read 8 Done Sent Done false
28377 8841 Acked Read 3 Done Sent Done false
28378 8842 Acked Read 3 Done Sent Done false
28379 8843 Acked Write 3 Done Sent Done false
28380 8844 Acked Read 5 Done Sent Done false
28381 8845 Acked Write 7 Done Sent Done false
28382 8846 Acked Read 3 Done Sent Done false
28383 8847 Acked Read 1 Done Sent Done false
28384 8848 Acked Read 7 Done Sent Done false
28385 8849 Acked Write 4 Done Sent Done false
28386 STATES DS:0 DS:1 DS:2 TOTAL
28387 Sent 0 301 0 301
28388 Done 301 0 301 602
28389 Skipped 0 0 0 0
28390 Error 0 0 0 0
28391 Last Flush: 8813 8548 8813
28392 Downstairs last five completed: 8548 8547 8546 8545 8544
28393 Upstairs last five completed: 8849 8848 8847 8846 8845
28394 CLIENT: Up:0 ds:301 act:2
28395 ----------------------------------------------------------------
28396 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
28397 States: Active Offline Active
28398 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28399 8549 Acked Write 9 Done Sent Done false
28400 8550 Acked Write 3 Done Sent Done false
28401 8551 Acked Read 8 Done Sent Done false
28402 8552 Acked Read 5 Done Sent Done false
28403 8553 Acked Read 1 Done Sent Done false
28404 8554 Acked Read 8 Done Sent Done false
28405 8555 Acked Write 1 Done Sent Done false
28406 8556 Acked Read 1 Done Sent Done false
28407 8557 Acked Write 4 Done Sent Done false
28408 8558 Acked Read 2 Done Sent Done false
28409 8559 Acked Read 7 Done Sent Done false
28410 8560 Acked Read 5 Done Sent Done false
28411 8561 Acked Read 3 Done Sent Done false
28412 8562 Acked Flush 0 Done Sent Done false
28413 8563 Acked Read 8 Done Sent Done false
28414 8564 Acked Write 6 Done Sent Done false
28415 8565 Acked Read 8 Done Sent Done false
28416 8566 Acked Read 4 Done Sent Done false
28417 8567 Acked Read 1 Done Sent Done false
28418 8568 Acked Read 10 Done Sent Done false
28419 8569 Acked Read 4 Done Sent Done false
28420 8570 Acked Write 5 Done Sent Done false
28421 8571 Acked Read 6 Done Sent Done false
28422 8572 Acked Write 5 Done Sent Done false
28423 8573 Acked Read 2 Done Sent Done false
28424 8574 Acked Write 3 Done Sent Done false
28425 8575 Acked Read 8 Done Sent Done false
28426 8576 Acked Read 4 Done Sent Done false
28427 8577 Acked Write 3 Done Sent Done false
28428 8578 Acked Write 4 Done Sent Done false
28429 8579 Acked Read 7 Done Sent Done false
28430 8580 Acked Flush 0 Done Sent Done false
28431 8581 Acked Read 3 Done Sent Done false
28432 8582 Acked Write 7 Done Sent Done false
28433 8583 Acked Read 3 Done Sent Done false
28434 8584 Acked Write 3 Done Sent Done false
28435 8585 Acked Read 4 Done Sent Done false
28436 8586 Acked Read 7 Done Sent Done false
28437 8587 Acked Read 7 Done Sent Done false
28438 8588 Acked Read 6 Done Sent Done false
28439 8589 Acked Flush 0 Done Sent Done false
28440 8590 Acked Write 4 Done Sent Done false
28441 8591 Acked Write 5 Done Sent Done false
28442 8592 Acked Write 8 Done Sent Done false
28443 8593 Acked Write 4 Done Sent Done false
28444 8594 Acked Flush 0 Done Sent Done false
28445 8595 Acked Write 4 Done Sent Done false
28446 8596 Acked Read 7 Done Sent Done false
28447 8597 Acked Read 6 Done Sent Done false
28448 8598 Acked Write 8 Done Sent Done false
28449 8599 Acked Read 6 Done Sent Done false
28450 8600 Acked Write 2 Done Sent Done false
28451 8601 Acked Write 4 Done Sent Done false
28452 8602 Acked Write 1 Done Sent Done false
28453 8603 Acked Read 8 Done Sent Done false
28454 8604 Acked Read 7 Done Sent Done false
28455 8605 Acked Read 5 Done Sent Done false
28456 8606 Acked Write 1 Done Sent Done false
28457 8607 Acked Write 4 Done Sent Done false
28458 8608 Acked Write 6 Done Sent Done false
28459 8609 Acked Flush 0 Done Sent Done false
28460 8610 Acked Flush 0 Done Sent Done false
28461 8611 Acked Read 6 Done Sent Done false
28462 8612 Acked Read 10 Done Sent Done false
28463 8613 Acked Write 7 Done Sent Done false
28464 8614 Acked Read 5 Done Sent Done false
28465 8615 Acked Read 7 Done Sent Done false
28466 8616 Acked Read 5 Done Sent Done false
28467 8617 Acked Read 2 Done Sent Done false
28468 8618 Acked Write 4 Done Sent Done false
28469 8619 Acked Write 8 Done Sent Done false
28470 8620 Acked Write 5 Done Sent Done false
28471 8621 Acked Read 1 Done Sent Done false
28472 8622 Acked Read 4 Done Sent Done false
28473 8623 Acked Write 1 Done Sent Done false
28474 8624 Acked Read 5 Done Sent Done false
28475 8625 Acked Write 7 Done Sent Done false
28476 8626 Acked Write 6 Done Sent Done false
28477 8627 Acked Read 4 Done Sent Done false
28478 8628 Acked Write 6 Done Sent Done false
28479 8629 Acked Write 1 Done Sent Done false
28480 8630 Acked Read 10 Done Sent Done false
28481 8631 Acked Read 9 Done Sent Done false
28482 8632 Acked Write 8 Done Sent Done false
28483 8633 Acked Write 7 Done Sent Done false
28484 8634 Acked Read 2 Done Sent Done false
28485 8635 Acked Write 10 Done Sent Done false
28486 8636 Acked Write 10 Done Sent Done false
28487 8637 Acked Read 7 Done Sent Done false
28488 8638 Acked Read 8 Done Sent Done false
28489 8639 Acked Read 4 Done Sent Done false
28490 8640 Acked Write 2 Done Sent Done false
28491 8641 Acked Write 1 Done Sent Done false
28492 8642 Acked Write 4 Done Sent Done false
28493 8643 Acked Write 9 Done Sent Done false
28494 8644 Acked Write 6 Done Sent Done false
28495 8645 Acked Read 5 Done Sent Done false
28496 8646 Acked Flush 0 Done Sent Done false
28497 8647 Acked Write 9 Done Sent Done false
28498 8648 Acked Write 9 Done Sent Done false
28499 8649 Acked Read 4 Done Sent Done false
28500 8650 Acked Flush 0 Done Sent Done false
28501 8651 Acked Flush 0 Done Sent Done false
28502 8652 Acked Read 8 Done Sent Done false
28503 8653 Acked Write 1 Done Sent Done false
28504 8654 Acked Read 7 Done Sent Done false
28505 8655 Acked Write 5 Done Sent Done false
28506 8656 Acked Read 2 Done Sent Done false
28507 8657 Acked Read 8 Done Sent Done false
28508 8658 Acked Write 10 Done Sent Done false
28509 8659 Acked Write 7 Done Sent Done false
28510 8660 Acked Read 2 Done Sent Done false
28511 8661 Acked Write 1 Done Sent Done false
28512 8662 Acked Read 2 Done Sent Done false
28513 8663 Acked Write 5 Done Sent Done false
28514 8664 Acked Read 7 Done Sent Done false
28515 8665 Acked Write 1 Done Sent Done false
28516 8666 Acked Write 1 Done Sent Done false
28517 8667 Acked Read 3 Done Sent Done false
28518 8668 Acked Read 9 Done Sent Done false
28519 8669 Acked Read 9 Done Sent Done false
28520 8670 Acked Write 7 Done Sent Done false
28521 8671 Acked Flush 0 Done Sent Done false
28522 8672 Acked Read 9 Done Sent Done false
28523 8673 Acked Read 4 Done Sent Done false
28524 8674 Acked Write 5 Done Sent Done false
28525 8675 Acked Read 6 Done Sent Done false
28526 8676 Acked Read 4 Done Sent Done false
28527 8677 Acked Read 6 Done Sent Done false
28528 8678 Acked Read 9 Done Sent Done false
28529 8679 Acked Read 4 Done Sent Done false
28530 8680 Acked Read 7 Done Sent Done false
28531 8681 Acked Write 5 Done Sent Done false
28532 8682 Acked Write 10 Done Sent Done false
28533 8683 Acked Read 2 Done Sent Done false
28534 8684 Acked Read 5 Done Sent Done false
28535 8685 Acked Read 1 Done Sent Done false
28536 8686 Acked Read 10 Done Sent Done false
28537 8687 Acked Read 5 Done Sent Done false
28538 8688 Acked Write 3 Done Sent Done false
28539 8689 Acked Flush 0 Done Sent Done false
28540 8690 Acked Write 5 Done Sent Done false
28541 8691 Acked Write 6 Done Sent Done false
28542 8692 Acked Read 8 Done Sent Done false
28543 8693 Acked Read 4 Done Sent Done false
28544 8694 Acked Read 10 Done Sent Done false
28545 8695 Acked Read 6 Done Sent Done false
28546 8696 Acked Write 3 Done Sent Done false
28547 8697 Acked Flush 0 Done Sent Done false
28548 8698 Acked Read 2 Done Sent Done false
28549 8699 Acked Read 9 Done Sent Done false
28550 8700 Acked Read 2 Done Sent Done false
28551 8701 Acked Read 9 Done Sent Done false
28552 8702 Acked Write 1 Done Sent Done false
28553 8703 Acked Read 1 Done Sent Done false
28554 8704 Acked Read 4 Done Sent Done false
28555 8705 Acked Flush 0 Done Sent Done false
28556 8706 Acked Write 8 Done Sent Done false
28557 8707 Acked Read 1 Done Sent Done false
28558 8708 Acked Write 10 Done Sent Done false
28559 8709 Acked Read 7 Done Sent Done false
28560 8710 Acked Write 8 Done Sent Done false
28561 8711 Acked Write 7 Done Sent Done false
28562 8712 Acked Write 4 Done Sent Done false
28563 8713 Acked Read 10 Done Sent Done false
28564 8714 Acked Write 5 Done Sent Done false
28565 8715 Acked Write 4 Done Sent Done false
28566 8716 Acked Read 3 Done Sent Done false
28567 8717 Acked Read 8 Done Sent Done false
28568 8718 Acked Write 9 Done Sent Done false
28569 8719 Acked Write 10 Done Sent Done false
28570 8720 Acked Write 1 Done Sent Done false
28571 8721 Acked Read 1 Done Sent Done false
28572 8722 Acked Read 7 Done Sent Done false
28573 8723 Acked Read 3 Done Sent Done false
28574 8724 Acked Write 6 Done Sent Done false
28575 8725 Acked Flush 0 Done Sent Done false
28576 8726 Acked Read 8 Done Sent Done false
28577 8727 Acked Write 2 Done Sent Done false
28578 8728 Acked Write 8 Done Sent Done false
28579 8729 Acked Read 9 Done Sent Done false
28580 8730 Acked Write 9 Done Sent Done false
28581 8731 Acked Read 7 Done Sent Done false
28582 8732 Acked Write 7 Done Sent Done false
28583 8733 Acked Read 4 Done Sent Done false
28584 8734 Acked Read 7 Done Sent Done false
28585 8735 Acked Read 10 Done Sent Done false
28586 8736 Acked Write 3 Done Sent Done false
28587 8737 Acked Read 6 Done Sent Done false
28588 8738 Acked Read 9 Done Sent Done false
28589 8739 Acked Read 10 Done Sent Done false
28590 8740 Acked Read 9 Done Sent Done false
28591 8741 Acked Write 6 Done Sent Done false
28592 8742 Acked Write 9 Done Sent Done false
28593 8743 Acked Read 1 Done Sent Done false
28594 8744 Acked Write 8 Done Sent Done false
28595 8745 Acked Flush 0 Done Sent Done false
28596 8746 Acked Write 9 Done Sent Done false
28597 8747 Acked Read 5 Done Sent Done false
28598 8748 Acked Flush 0 Done Sent Done false
28599 8749 Acked Read 2 Done Sent Done false
28600 8750 Acked Write 6 Done Sent Done false
28601 8751 Acked Write 8 Done Sent Done false
28602 8752 Acked Read 5 Done Sent Done false
28603 8753 Acked Write 2 Done Sent Done false
28604 8754 Acked Write 2 Done Sent Done false
28605 8755 Acked Read 2 Done Sent Done false
28606 8756 Acked Read 7 Done Sent Done false
28607 8757 Acked Write 10 Done Sent Done false
28608 8758 Acked Read 3 Done Sent Done false
28609 8759 Acked Write 7 Done Sent Done false
28610 8760 Acked Read 8 Done Sent Done false
28611 8761 Acked Read 2 Done Sent Done false
28612 8762 Acked Read 5 Done Sent Done false
28613 8763 Acked Flush 0 Done Sent Done false
28614 8764 Acked Write 10 Done Sent Done false
28615 8765 Acked Write 10 Done Sent Done false
28616 8766 Acked Write 8 Done Sent Done false
28617 8767 Acked Read 10 Done Sent Done false
28618 8768 Acked Write 7 Done Sent Done false
28619 8769 Acked Read 9 Done Sent Done false
28620 8770 Acked Write 6 Done Sent Done false
28621 8771 Acked Flush 0 Done Sent Done false
28622 8772 Acked Flush 0 Done Sent Done false
28623 8773 Acked Write 2 Done Sent Done false
28624 8774 Acked Write 10 Done Sent Done false
28625 8775 Acked Write 1 Done Sent Done false
28626 8776 Acked Flush 0 Done Sent Done false
28627 8777 Acked Read 9 Done Sent Done false
28628 8778 Acked Write 5 Done Sent Done false
28629 8779 Acked Write 9 Done Sent Done false
28630 8780 Acked Read 10 Done Sent Done false
28631 8781 Acked Read 7 Done Sent Done false
28632 8782 Acked Read 6 Done Sent Done false
28633 8783 Acked Write 5 Done Sent Done false
28634 8784 Acked Read 3 Done Sent Done false
28635 8785 Acked Flush 0 Done Sent Done false
28636 8786 Acked Write 1 Done Sent Done false
28637 8787 Acked Write 9 Done Sent Done false
28638 8788 Acked Read 10 Done Sent Done false
28639 8789 Acked Read 2 Done Sent Done false
28640 8790 Acked Read 8 Done Sent Done false
28641 8791 Acked Read 1 Done Sent Done false
28642 8792 Acked Read 4 Done Sent Done false
28643 8793 Acked Write 8 Done Sent Done false
28644 8794 Acked Read 3 Done Sent Done false
28645 8795 Acked Read 3 Done Sent Done false
28646 8796 Acked Read 9 Done Sent Done false
28647 8797 Acked Flush 0 Done Sent Done false
28648 8798 Acked Read 5 Done Sent Done false
28649 8799 Acked Write 10 Done Sent Done false
28650 8800 Acked Read 4 Done Sent Done false
28651 8801 Acked Write 5 Done Sent Done false
28652 8802 Acked Read 1 Done Sent Done false
28653 8803 Acked Write 6 Done Sent Done false
28654 8804 Acked Write 4 Done Sent Done false
28655 8805 Acked Read 1 Done Sent Done false
28656 8806 Acked Write 8 Done Sent Done false
28657 8807 Acked Read 6 Done Sent Done false
28658 8808 Acked Write 8 Done Sent Done false
28659 8809 Acked Read 4 Done Sent Done false
28660 8810 Acked Write 9 Done Sent Done false
28661 8811 Acked Write 2 Done Sent Done false
28662 8812 Acked Read 9 Done Sent Done false
28663 8813 Acked Flush 0 Done Sent Done false
28664 8814 Acked Write 8 Done Sent Done false
28665 8815 Acked Write 4 Done Sent Done false
28666 8816 Acked Write 5 Done Sent Done false
28667 8817 Acked Read 10 Done Sent Done false
28668 8818 Acked Write 7 Done Sent Done false
28669 8819 Acked Write 1 Done Sent Done false
28670 8820 Acked Write 5 Done Sent Done false
28671 8821 Acked Read 9 Done Sent Done false
28672 8822 Acked Read 1 Done Sent Done false
28673 8823 Acked Write 8 Done Sent Done false
28674 8824 Acked Write 5 Done Sent Done false
28675 8825 Acked Write 10 Done Sent Done false
28676 8826 Acked Write 1 Done Sent Done false
28677 8827 Acked Read 3 Done Sent Done false
28678 8828 Acked Read 9 Done Sent Done false
28679 8829 Acked Write 9 Done Sent Done false
28680 8830 Acked Read 1 Done Sent Done false
28681 8831 Acked Read 1 Done Sent Done false
28682 8832 Acked Write 4 Done Sent Done false
28683 8833 Acked Write 8 Done Sent Done false
28684 8834 Acked Read 1 Done Sent Done false
28685 8835 Acked Write 8 Done Sent Done false
28686 8836 Acked Write 9 Done Sent Done false
28687 8837 Acked Write 4 Done Sent Done false
28688 8838 Acked Read 9 Done Sent Done false
28689 8839 Acked Read 4 Done Sent Done false
28690 8840 Acked Read 8 Done Sent Done false
28691 8841 Acked Read 3 Done Sent Done false
28692 8842 Acked Read 3 Done Sent Done false
28693 8843 Acked Write 3 Done Sent Done false
28694 8844 Acked Read 5 Done Sent Done false
28695 8845 Acked Write 7 Done Sent Done false
28696 8846 Acked Read 3 Done Sent Done false
28697 8847 Acked Read 1 Done Sent Done false
28698 8848 Acked Read 7 Done Sent Done false
28699 8849 Acked Write 4 Done Sent Done false
28700 8850 Acked Flush 0 Done Sent Done false
28701 STATES DS:0 DS:1 DS:2 TOTAL
28702 Sent 0 302 0 302
28703 Done 302 0 302 604
28704 Skipped 0 0 0 0
28705 Error 0 0 0 0
28706 Last Flush: 8850 8548 8850
28707 Downstairs last five completed: 8548 8547 8546 8545 8544
28708 Upstairs last five completed: 8850 8849 8848 8847 8846
28709 CLIENT: Up:0 ds:302 act:2
287102024-12-09T15:13:51.318ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
287112024-12-09T15:13:51.318ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
287122024-12-09T15:13:51.318ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
287132024-12-09T15:13:51.318ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
287142024-12-09T15:13:51.318ZINFOcrucible: client task is exiting = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
28715 ----------------------------------------------------------------
28716 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
28717 States: Active Offline Active
28718 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28719 8549 Acked Write 9 Done Sent Done false
28720 8550 Acked Write 3 Done Sent Done false
28721 8551 Acked Read 8 Done Sent Done false
28722 8552 Acked Read 5 Done Sent Done false
28723 8553 Acked Read 1 Done Sent Done false
28724 8554 Acked Read 8 Done Sent Done false
28725 8555 Acked Write 1 Done Sent Done false
28726 8556 Acked Read 1 Done Sent Done false
28727 8557 Acked Write 4 Done Sent Done false
28728 8558 Acked Read 2 Done Sent Done false
28729 8559 Acked Read 7 Done Sent Done false
28730 8560 Acked Read 5 Done Sent Done false
28731 8561 Acked Read 3 Done Sent Done false
28732 8562 Acked Flush 0 Done Sent Done false
28733 8563 Acked Read 8 Done Sent Done false
28734 8564 Acked Write 6 Done Sent Done false
28735 8565 Acked Read 8 Done Sent Done false
28736 8566 Acked Read 4 Done Sent Done false
28737 8567 Acked Read 1 Done Sent Done false
28738 8568 Acked Read 10 Done Sent Done false
28739 8569 Acked Read 4 Done Sent Done false
28740 8570 Acked Write 5 Done Sent Done false
28741 8571 Acked Read 6 Done Sent Done false
28742 8572 Acked Write 5 Done Sent Done false
28743 8573 Acked Read 2 Done Sent Done false
28744 8574 Acked Write 3 Done Sent Done false
28745 8575 Acked Read 8 Done Sent Done false
28746 8576 Acked Read 4 Done Sent Done false
28747 8577 Acked Write 3 Done Sent Done false
28748 8578 Acked Write 4 Done Sent Done false
28749 8579 Acked Read 7 Done Sent Done false
28750 8580 Acked Flush 0 Done Sent Done false
28751 8581 Acked Read 3 Done Sent Done false
28752 8582 Acked Write 7 Done Sent Done false
28753 8583 Acked Read 3 Done Sent Done false
28754 8584 Acked Write 3 Done Sent Done false
28755 8585 Acked Read 4 Done Sent Done false
28756 8586 Acked Read 7 Done Sent Done false
28757 8587 Acked Read 7 Done Sent Done false
28758 8588 Acked Read 6 Done Sent Done false
28759 8589 Acked Flush 0 Done Sent Done false
28760 8590 Acked Write 4 Done Sent Done false
28761 8591 Acked Write 5 Done Sent Done false
28762 8592 Acked Write 8 Done Sent Done false
28763 8593 Acked Write 4 Done Sent Done false
28764 8594 Acked Flush 0 Done Sent Done false
28765 8595 Acked Write 4 Done Sent Done false
28766 8596 Acked Read 7 Done Sent Done false
28767 8597 Acked Read 6 Done Sent Done false
28768 8598 Acked Write 8 Done Sent Done false
28769 8599 Acked Read 6 Done Sent Done false
28770 8600 Acked Write 2 Done Sent Done false
28771 8601 Acked Write 4 Done Sent Done false
28772 8602 Acked Write 1 Done Sent Done false
28773 8603 Acked Read 8 Done Sent Done false
28774 8604 Acked Read 7 Done Sent Done false
28775 8605 Acked Read 5 Done Sent Done false
28776 8606 Acked Write 1 Done Sent Done false
28777 8607 Acked Write 4 Done Sent Done false
28778 8608 Acked Write 6 Done Sent Done false
28779 8609 Acked Flush 0 Done Sent Done false
28780 8610 Acked Flush 0 Done Sent Done false
28781 8611 Acked Read 6 Done Sent Done false
28782 8612 Acked Read 10 Done Sent Done false
28783 8613 Acked Write 7 Done Sent Done false
28784 8614 Acked Read 5 Done Sent Done false
28785 8615 Acked Read 7 Done Sent Done false
28786 8616 Acked Read 5 Done Sent Done false
28787 8617 Acked Read 2 Done Sent Done false
28788 8618 Acked Write 4 Done Sent Done false
28789 8619 Acked Write 8 Done Sent Done false
28790 8620 Acked Write 5 Done Sent Done false
28791 8621 Acked Read 1 Done Sent Done false
28792 8622 Acked Read 4 Done Sent Done false
28793 8623 Acked Write 1 Done Sent Done false
28794 8624 Acked Read 5 Done Sent Done false
28795 8625 Acked Write 7 Done Sent Done false
28796 8626 Acked Write 6 Done Sent Done false
28797 8627 Acked Read 4 Done Sent Done false
28798 8628 Acked Write 6 Done Sent Done false
28799 8629 Acked Write 1 Done Sent Done false
28800 8630 Acked Read 10 Done Sent Done false
28801 8631 Acked Read 9 Done Sent Done false
28802 8632 Acked Write 8 Done Sent Done false
28803 8633 Acked Write 7 Done Sent Done false
28804 8634 Acked Read 2 Done Sent Done false
28805 8635 Acked Write 10 Done Sent Done false
28806 8636 Acked Write 10 Done Sent Done false
28807 8637 Acked Read 7 Done Sent Done false
28808 8638 Acked Read 8 Done Sent Done false
28809 8639 Acked Read 4 Done Sent Done false
28810 8640 Acked Write 2 Done Sent Done false
28811 8641 Acked Write 1 Done Sent Done false
28812 8642 Acked Write 4 Done Sent Done false
28813 8643 Acked Write 9 Done Sent Done false
28814 8644 Acked Write 6 Done Sent Done false
28815 8645 Acked Read 5 Done Sent Done false
28816 8646 Acked Flush 0 Done Sent Done false
28817 8647 Acked Write 9 Done Sent Done false
28818 8648 Acked Write 9 Done Sent Done false
28819 8649 Acked Read 4 Done Sent Done false
28820 8650 Acked Flush 0 Done Sent Done false
28821 8651 Acked Flush 0 Done Sent Done false
28822 8652 Acked Read 8 Done Sent Done false
28823 8653 Acked Write 1 Done Sent Done false
28824 8654 Acked Read 7 Done Sent Done false
28825 8655 Acked Write 5 Done Sent Done false
28826 8656 Acked Read 2 Done Sent Done false
28827 8657 Acked Read 8 Done Sent Done false
28828 8658 Acked Write 10 Done Sent Done false
28829 8659 Acked Write 7 Done Sent Done false
28830 8660 Acked Read 2 Done Sent Done false
28831 8661 Acked Write 1 Done Sent Done false
28832 8662 Acked Read 2 Done Sent Done false
28833 8663 Acked Write 5 Done Sent Done false
28834 8664 Acked Read 7 Done Sent Done false
28835 8665 Acked Write 1 Done Sent Done false
28836 8666 Acked Write 1 Done Sent Done false
28837 8667 Acked Read 3 Done Sent Done false
28838 8668 Acked Read 9 Done Sent Done false
28839 8669 Acked Read 9 Done Sent Done false
28840 8670 Acked Write 7 Done Sent Done false
28841 8671 Acked Flush 0 Done Sent Done false
28842 8672 Acked Read 9 Done Sent Done false
28843 8673 Acked Read 4 Done Sent Done false
28844 8674 Acked Write 5 Done Sent Done false
28845 8675 Acked Read 6 Done Sent Done false
28846 8676 Acked Read 4 Done Sent Done false
28847 8677 Acked Read 6 Done Sent Done false
28848 8678 Acked Read 9 Done Sent Done false
28849 8679 Acked Read 4 Done Sent Done false
28850 8680 Acked Read 7 Done Sent Done false
28851 8681 Acked Write 5 Done Sent Done false
28852 8682 Acked Write 10 Done Sent Done false
28853 8683 Acked Read 2 Done Sent Done false
28854 8684 Acked Read 5 Done Sent Done false
28855 8685 Acked Read 1 Done Sent Done false
28856 8686 Acked Read 10 Done Sent Done false
28857 8687 Acked Read 5 Done Sent Done false
28858 8688 Acked Write 3 Done Sent Done false
28859 8689 Acked Flush 0 Done Sent Done false
28860 8690 Acked Write 5 Done Sent Done false
28861 8691 Acked Write 6 Done Sent Done false
28862 8692 Acked Read 8 Done Sent Done false
28863 8693 Acked Read 4 Done Sent Done false
28864 8694 Acked Read 10 Done Sent Done false
28865 8695 Acked Read 6 Done Sent Done false
28866 8696 Acked Write 3 Done Sent Done false
28867 8697 Acked Flush 0 Done Sent Done false
28868 8698 Acked Read 2 Done Sent Done false
28869 8699 Acked Read 9 Done Sent Done false
28870 8700 Acked Read 2 Done Sent Done false
28871 8701 Acked Read 9 Done Sent Done false
28872 8702 Acked Write 1 Done Sent Done false
28873 8703 Acked Read 1 Done Sent Done false
28874 8704 Acked Read 4 Done Sent Done false
28875 8705 Acked Flush 0 Done Sent Done false
28876 8706 Acked Write 8 Done Sent Done false
28877 8707 Acked Read 1 Done Sent Done false
28878 8708 Acked Write 10 Done Sent Done false
28879 8709 Acked Read 7 Done Sent Done false
28880 8710 Acked Write 8 Done Sent Done false
28881 8711 Acked Write 7 Done Sent Done false
28882 8712 Acked Write 4 Done Sent Done false
28883 8713 Acked Read 10 Done Sent Done false
28884 8714 Acked Write 5 Done Sent Done false
28885 8715 Acked Write 4 Done Sent Done false
28886 8716 Acked Read 3 Done Sent Done false
28887 8717 Acked Read 8 Done Sent Done false
28888 8718 Acked Write 9 Done Sent Done false
28889 8719 Acked Write 10 Done Sent Done false
28890 8720 Acked Write 1 Done Sent Done false
28891 8721 Acked Read 1 Done Sent Done false
28892 8722 Acked Read 7 Done Sent Done false
28893 8723 Acked Read 3 Done Sent Done false
28894 8724 Acked Write 6 Done Sent Done false
28895 8725 Acked Flush 0 Done Sent Done false
28896 8726 Acked Read 8 Done Sent Done false
28897 8727 Acked Write 2 Done Sent Done false
28898 8728 Acked Write 8 Done Sent Done false
28899 8729 Acked Read 9 Done Sent Done false
28900 8730 Acked Write 9 Done Sent Done false
28901 8731 Acked Read 7 Done Sent Done false
28902 8732 Acked Write 7 Done Sent Done false
28903 8733 Acked Read 4 Done Sent Done false
28904 8734 Acked Read 7 Done Sent Done false
28905 8735 Acked Read 10 Done Sent Done false
28906 8736 Acked Write 3 Done Sent Done false
28907 8737 Acked Read 6 Done Sent Done false
28908 8738 Acked Read 9 Done Sent Done false
28909 8739 Acked Read 10 Done Sent Done false
28910 8740 Acked Read 9 Done Sent Done false
28911 8741 Acked Write 6 Done Sent Done false
28912 8742 Acked Write 9 Done Sent Done false
28913 8743 Acked Read 1 Done Sent Done false
28914 8744 Acked Write 8 Done Sent Done false
28915 8745 Acked Flush 0 Done Sent Done false
28916 8746 Acked Write 9 Done Sent Done false
28917 8747 Acked Read 5 Done Sent Done false
28918 8748 Acked Flush 0 Done Sent Done false
28919 8749 Acked Read 2 Done Sent Done false
28920 8750 Acked Write 6 Done Sent Done false
28921 8751 Acked Write 8 Done Sent Done false
28922 8752 Acked Read 5 Done Sent Done false
28923 8753 Acked Write 2 Done Sent Done false
28924 8754 Acked Write 2 Done Sent Done false
28925 8755 Acked Read 2 Done Sent Done false
28926 8756 Acked Read 7 Done Sent Done false
28927 8757 Acked Write 10 Done Sent Done false
28928 8758 Acked Read 3 Done Sent Done false
28929 8759 Acked Write 7 Done Sent Done false
28930 8760 Acked Read 8 Done Sent Done false
28931 8761 Acked Read 2 Done Sent Done false
28932 8762 Acked Read 5 Done Sent Done false
28933 8763 Acked Flush 0 Done Sent Done false
28934 8764 Acked Write 10 Done Sent Done false
28935 8765 Acked Write 10 Done Sent Done false
28936 8766 Acked Write 8 Done Sent Done false
28937 8767 Acked Read 10 Done Sent Done false
28938 8768 Acked Write 7 Done Sent Done false
28939 8769 Acked Read 9 Done Sent Done false
28940 8770 Acked Write 6 Done Sent Done false
28941 8771 Acked Flush 0 Done Sent Done false
28942 8772 Acked Flush 0 Done Sent Done false
28943 8773 Acked Write 2 Done Sent Done false
28944 8774 Acked Write 10 Done Sent Done false
28945 8775 Acked Write 1 Done Sent Done false
28946 8776 Acked Flush 0 Done Sent Done false
28947 8777 Acked Read 9 Done Sent Done false
28948 8778 Acked Write 5 Done Sent Done false
28949 8779 Acked Write 9 Done Sent Done false
28950 8780 Acked Read 10 Done Sent Done false
28951 8781 Acked Read 7 Done Sent Done false
28952 8782 Acked Read 6 Done Sent Done false
28953 8783 Acked Write 5 Done Sent Done false
28954 8784 Acked Read 3 Done Sent Done false
28955 8785 Acked Flush 0 Done Sent Done false
28956 8786 Acked Write 1 Done Sent Done false
28957 8787 Acked Write 9 Done Sent Done false
28958 8788 Acked Read 10 Done Sent Done false
28959 8789 Acked Read 2 Done Sent Done false
28960 8790 Acked Read 8 Done Sent Done false
28961 8791 Acked Read 1 Done Sent Done false
28962 8792 Acked Read 4 Done Sent Done false
28963 8793 Acked Write 8 Done Sent Done false
28964 8794 Acked Read 3 Done Sent Done false
28965 8795 Acked Read 3 Done Sent Done false
28966 8796 Acked Read 9 Done Sent Done false
28967 8797 Acked Flush 0 Done Sent Done false
28968 8798 Acked Read 5 Done Sent Done false
28969 8799 Acked Write 10 Done Sent Done false
28970 8800 Acked Read 4 Done Sent Done false
28971 8801 Acked Write 5 Done Sent Done false
28972 8802 Acked Read 1 Done Sent Done false
28973 8803 Acked Write 6 Done Sent Done false
28974 8804 Acked Write 4 Done Sent Done false
28975 8805 Acked Read 1 Done Sent Done false
28976 8806 Acked Write 8 Done Sent Done false
28977 8807 Acked Read 6 Done Sent Done false
28978 8808 Acked Write 8 Done Sent Done false
28979 8809 Acked Read 4 Done Sent Done false
28980 8810 Acked Write 9 Done Sent Done false
28981 8811 Acked Write 2 Done Sent Done false
28982 8812 Acked Read 9 Done Sent Done false
28983 8813 Acked Flush 0 Done Sent Done false
28984 8814 Acked Write 8 Done Sent Done false
28985 8815 Acked Write 4 Done Sent Done false
28986 8816 Acked Write 5 Done Sent Done false
28987 8817 Acked Read 10 Done Sent Done false
28988 8818 Acked Write 7 Done Sent Done false
28989 8819 Acked Write 1 Done Sent Done false
28990 8820 Acked Write 5 Done Sent Done false
28991 8821 Acked Read 9 Done Sent Done false
28992 8822 Acked Read 1 Done Sent Done false
28993 8823 Acked Write 8 Done Sent Done false
28994 8824 Acked Write 5 Done Sent Done false
28995 8825 Acked Write 10 Done Sent Done false
28996 8826 Acked Write 1 Done Sent Done false
28997 8827 Acked Read 3 Done Sent Done false
28998 8828 Acked Read 9 Done Sent Done false
28999 8829 Acked Write 9 Done Sent Done false
29000 8830 Acked Read 1 Done Sent Done false
29001 8831 Acked Read 1 Done Sent Done false
29002 8832 Acked Write 4 Done Sent Done false
29003 8833 Acked Write 8 Done Sent Done false
29004 8834 Acked Read 1 Done Sent Done false
29005 8835 Acked Write 8 Done Sent Done false
29006 8836 Acked Write 9 Done Sent Done false
29007 8837 Acked Write 4 Done Sent Done false
29008 8838 Acked Read 9 Done Sent Done false
29009 8839 Acked Read 4 Done Sent Done false
29010 8840 Acked Read 8 Done Sent Done false
29011 8841 Acked Read 3 Done Sent Done false
29012 8842 Acked Read 3 Done Sent Done false
29013 8843 Acked Write 3 Done Sent Done false
29014 8844 Acked Read 5 Done Sent Done false
29015 8845 Acked Write 7 Done Sent Done false
29016 8846 Acked Read 3 Done Sent Done false
29017 8847 Acked Read 1 Done Sent Done false
29018 8848 Acked Read 7 Done Sent Done false
29019 8849 Acked Write 4 Done Sent Done false
29020 8850 Acked Flush 0 Done Sent Done false
29021 STATES DS:0 DS:1 DS:2 TOTAL
29022 Sent 0 302 0 302
29023 Done 302 0 302 604
29024 Skipped 0 0 0 0
29025 Error 0 0 0 0
29026 Last Flush: 8850 8548 8850
29027 Downstairs last five completed: 8548 8547 8546 8545 8544
29028 Upstairs last five completed: 8850 8849 8848 8847 8846
29029 CLIENT: Up:0 ds:302 act:2
29030 [8820] got start action
29031 [8820] I am going to restart
29032 Starting downstairs at port 8820
29033 Make output file at "/tmp/dsc/downstairs-8820.txt"
29034 [8820][1] reports Starting
29035 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1227)
29036 [8820][1] reports Running
29037 ----------------------------------------------------------------
29038 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
29039 States: Active Offline Active
29040 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29041 8549 Acked Write 9 Done Sent Done false
29042 8550 Acked Write 3 Done Sent Done false
29043 8551 Acked Read 8 Done Sent Done false
29044 8552 Acked Read 5 Done Sent Done false
29045 8553 Acked Read 1 Done Sent Done false
29046 8554 Acked Read 8 Done Sent Done false
29047 8555 Acked Write 1 Done Sent Done false
29048 8556 Acked Read 1 Done Sent Done false
29049 8557 Acked Write 4 Done Sent Done false
29050 8558 Acked Read 2 Done Sent Done false
29051 8559 Acked Read 7 Done Sent Done false
29052 8560 Acked Read 5 Done Sent Done false
29053 8561 Acked Read 3 Done Sent Done false
29054 8562 Acked Flush 0 Done Sent Done false
29055 8563 Acked Read 8 Done Sent Done false
29056 8564 Acked Write 6 Done Sent Done false
29057 8565 Acked Read 8 Done Sent Done false
29058 8566 Acked Read 4 Done Sent Done false
29059 8567 Acked Read 1 Done Sent Done false
29060 8568 Acked Read 10 Done Sent Done false
29061 8569 Acked Read 4 Done Sent Done false
29062 8570 Acked Write 5 Done Sent Done false
29063 8571 Acked Read 6 Done Sent Done false
29064 8572 Acked Write 5 Done Sent Done false
29065 8573 Acked Read 2 Done Sent Done false
29066 8574 Acked Write 3 Done Sent Done false
29067 8575 Acked Read 8 Done Sent Done false
29068 8576 Acked Read 4 Done Sent Done false
29069 8577 Acked Write 3 Done Sent Done false
29070 8578 Acked Write 4 Done Sent Done false
29071 8579 Acked Read 7 Done Sent Done false
29072 8580 Acked Flush 0 Done Sent Done false
29073 8581 Acked Read 3 Done Sent Done false
29074 8582 Acked Write 7 Done Sent Done false
29075 8583 Acked Read 3 Done Sent Done false
29076 8584 Acked Write 3 Done Sent Done false
29077 8585 Acked Read 4 Done Sent Done false
29078 8586 Acked Read 7 Done Sent Done false
29079 8587 Acked Read 7 Done Sent Done false
29080 8588 Acked Read 6 Done Sent Done false
29081 8589 Acked Flush 0 Done Sent Done false
29082 8590 Acked Write 4 Done Sent Done false
29083 8591 Acked Write 5 Done Sent Done false
29084 8592 Acked Write 8 Done Sent Done false
29085 8593 Acked Write 4 Done Sent Done false
29086 8594 Acked Flush 0 Done Sent Done false
29087 8595 Acked Write 4 Done Sent Done false
29088 8596 Acked Read 7 Done Sent Done false
29089 8597 Acked Read 6 Done Sent Done false
29090 8598 Acked Write 8 Done Sent Done false
29091 8599 Acked Read 6 Done Sent Done false
29092 8600 Acked Write 2 Done Sent Done false
29093 8601 Acked Write 4 Done Sent Done false
29094 8602 Acked Write 1 Done Sent Done false
29095 8603 Acked Read 8 Done Sent Done false
29096 8604 Acked Read 7 Done Sent Done false
29097 8605 Acked Read 5 Done Sent Done false
29098 8606 Acked Write 1 Done Sent Done false
29099 8607 Acked Write 4 Done Sent Done false
29100 8608 Acked Write 6 Done Sent Done false
29101 8609 Acked Flush 0 Done Sent Done false
29102 8610 Acked Flush 0 Done Sent Done false
29103 8611 Acked Read 6 Done Sent Done false
29104 8612 Acked Read 10 Done Sent Done false
29105 8613 Acked Write 7 Done Sent Done false
29106 8614 Acked Read 5 Done Sent Done false
29107 8615 Acked Read 7 Done Sent Done false
29108 8616 Acked Read 5 Done Sent Done false
29109 8617 Acked Read 2 Done Sent Done false
29110 8618 Acked Write 4 Done Sent Done false
29111 8619 Acked Write 8 Done Sent Done false
29112 8620 Acked Write 5 Done Sent Done false
29113 8621 Acked Read 1 Done Sent Done false
29114 8622 Acked Read 4 Done Sent Done false
29115 8623 Acked Write 1 Done Sent Done false
29116 8624 Acked Read 5 Done Sent Done false
29117 8625 Acked Write 7 Done Sent Done false
29118 8626 Acked Write 6 Done Sent Done false
29119 8627 Acked Read 4 Done Sent Done false
29120 8628 Acked Write 6 Done Sent Done false
29121 8629 Acked Write 1 Done Sent Done false
29122 8630 Acked Read 10 Done Sent Done false
29123 8631 Acked Read 9 Done Sent Done false
29124 8632 Acked Write 8 Done Sent Done false
29125 8633 Acked Write 7 Done Sent Done false
29126 8634 Acked Read 2 Done Sent Done false
29127 8635 Acked Write 10 Done Sent Done false
29128 8636 Acked Write 10 Done Sent Done false
29129 8637 Acked Read 7 Done Sent Done false
29130 8638 Acked Read 8 Done Sent Done false
29131 8639 Acked Read 4 Done Sent Done false
29132 8640 Acked Write 2 Done Sent Done false
29133 8641 Acked Write 1 Done Sent Done false
29134 8642 Acked Write 4 Done Sent Done false
29135 8643 Acked Write 9 Done Sent Done false
29136 8644 Acked Write 6 Done Sent Done false
29137 8645 Acked Read 5 Done Sent Done false
29138 8646 Acked Flush 0 Done Sent Done false
29139 8647 Acked Write 9 Done Sent Done false
29140 8648 Acked Write 9 Done Sent Done false
29141 8649 Acked Read 4 Done Sent Done false
29142 8650 Acked Flush 0 Done Sent Done false
29143 8651 Acked Flush 0 Done Sent Done false
29144 8652 Acked Read 8 Done Sent Done false
29145 8653 Acked Write 1 Done Sent Done false
29146 8654 Acked Read 7 Done Sent Done false
29147 8655 Acked Write 5 Done Sent Done false
29148 8656 Acked Read 2 Done Sent Done false
29149 8657 Acked Read 8 Done Sent Done false
29150 8658 Acked Write 10 Done Sent Done false
29151 8659 Acked Write 7 Done Sent Done false
29152 8660 Acked Read 2 Done Sent Done false
29153 8661 Acked Write 1 Done Sent Done false
29154 8662 Acked Read 2 Done Sent Done false
29155 8663 Acked Write 5 Done Sent Done false
29156 8664 Acked Read 7 Done Sent Done false
29157 8665 Acked Write 1 Done Sent Done false
29158 8666 Acked Write 1 Done Sent Done false
29159 8667 Acked Read 3 Done Sent Done false
29160 8668 Acked Read 9 Done Sent Done false
29161 8669 Acked Read 9 Done Sent Done false
29162 8670 Acked Write 7 Done Sent Done false
29163 8671 Acked Flush 0 Done Sent Done false
29164 8672 Acked Read 9 Done Sent Done false
29165 8673 Acked Read 4 Done Sent Done false
29166 8674 Acked Write 5 Done Sent Done false
29167 8675 Acked Read 6 Done Sent Done false
29168 8676 Acked Read 4 Done Sent Done false
29169 8677 Acked Read 6 Done Sent Done false
29170 8678 Acked Read 9 Done Sent Done false
29171 8679 Acked Read 4 Done Sent Done false
29172 8680 Acked Read 7 Done Sent Done false
29173 8681 Acked Write 5 Done Sent Done false
29174 8682 Acked Write 10 Done Sent Done false
29175 8683 Acked Read 2 Done Sent Done false
29176 8684 Acked Read 5 Done Sent Done false
29177 8685 Acked Read 1 Done Sent Done false
29178 8686 Acked Read 10 Done Sent Done false
29179 8687 Acked Read 5 Done Sent Done false
29180 8688 Acked Write 3 Done Sent Done false
29181 8689 Acked Flush 0 Done Sent Done false
29182 8690 Acked Write 5 Done Sent Done false
29183 8691 Acked Write 6 Done Sent Done false
29184 8692 Acked Read 8 Done Sent Done false
29185 8693 Acked Read 4 Done Sent Done false
29186 8694 Acked Read 10 Done Sent Done false
29187 8695 Acked Read 6 Done Sent Done false
29188 8696 Acked Write 3 Done Sent Done false
29189 8697 Acked Flush 0 Done Sent Done false
29190 8698 Acked Read 2 Done Sent Done false
29191 8699 Acked Read 9 Done Sent Done false
29192 8700 Acked Read 2 Done Sent Done false
29193 8701 Acked Read 9 Done Sent Done false
29194 8702 Acked Write 1 Done Sent Done false
29195 8703 Acked Read 1 Done Sent Done false
29196 8704 Acked Read 4 Done Sent Done false
29197 8705 Acked Flush 0 Done Sent Done false
29198 8706 Acked Write 8 Done Sent Done false
29199 8707 Acked Read 1 Done Sent Done false
29200 8708 Acked Write 10 Done Sent Done false
29201 8709 Acked Read 7 Done Sent Done false
29202 8710 Acked Write 8 Done Sent Done false
29203 8711 Acked Write 7 Done Sent Done false
29204 8712 Acked Write 4 Done Sent Done false
29205 8713 Acked Read 10 Done Sent Done false
29206 8714 Acked Write 5 Done Sent Done false
29207 8715 Acked Write 4 Done Sent Done false
29208 8716 Acked Read 3 Done Sent Done false
29209 8717 Acked Read 8 Done Sent Done false
29210 8718 Acked Write 9 Done Sent Done false
29211 8719 Acked Write 10 Done Sent Done false
29212 8720 Acked Write 1 Done Sent Done false
29213 8721 Acked Read 1 Done Sent Done false
29214 8722 Acked Read 7 Done Sent Done false
29215 8723 Acked Read 3 Done Sent Done false
29216 8724 Acked Write 6 Done Sent Done false
29217 8725 Acked Flush 0 Done Sent Done false
29218 8726 Acked Read 8 Done Sent Done false
29219 8727 Acked Write 2 Done Sent Done false
29220 8728 Acked Write 8 Done Sent Done false
29221 8729 Acked Read 9 Done Sent Done false
29222 8730 Acked Write 9 Done Sent Done false
29223 8731 Acked Read 7 Done Sent Done false
29224 8732 Acked Write 7 Done Sent Done false
29225 8733 Acked Read 4 Done Sent Done false
29226 8734 Acked Read 7 Done Sent Done false
29227 8735 Acked Read 10 Done Sent Done false
29228 8736 Acked Write 3 Done Sent Done false
29229 8737 Acked Read 6 Done Sent Done false
29230 8738 Acked Read 9 Done Sent Done false
29231 8739 Acked Read 10 Done Sent Done false
29232 8740 Acked Read 9 Done Sent Done false
29233 8741 Acked Write 6 Done Sent Done false
29234 8742 Acked Write 9 Done Sent Done false
29235 8743 Acked Read 1 Done Sent Done false
29236 8744 Acked Write 8 Done Sent Done false
29237 8745 Acked Flush 0 Done Sent Done false
29238 8746 Acked Write 9 Done Sent Done false
29239 8747 Acked Read 5 Done Sent Done false
29240 8748 Acked Flush 0 Done Sent Done false
29241 8749 Acked Read 2 Done Sent Done false
29242 8750 Acked Write 6 Done Sent Done false
29243 8751 Acked Write 8 Done Sent Done false
29244 8752 Acked Read 5 Done Sent Done false
29245 8753 Acked Write 2 Done Sent Done false
29246 8754 Acked Write 2 Done Sent Done false
29247 8755 Acked Read 2 Done Sent Done false
29248 8756 Acked Read 7 Done Sent Done false
29249 8757 Acked Write 10 Done Sent Done false
29250 8758 Acked Read 3 Done Sent Done false
29251 8759 Acked Write 7 Done Sent Done false
29252 8760 Acked Read 8 Done Sent Done false
29253 8761 Acked Read 2 Done Sent Done false
29254 8762 Acked Read 5 Done Sent Done false
29255 8763 Acked Flush 0 Done Sent Done false
29256 8764 Acked Write 10 Done Sent Done false
29257 8765 Acked Write 10 Done Sent Done false
29258 8766 Acked Write 8 Done Sent Done false
29259 8767 Acked Read 10 Done Sent Done false
29260 8768 Acked Write 7 Done Sent Done false
29261 8769 Acked Read 9 Done Sent Done false
29262 8770 Acked Write 6 Done Sent Done false
29263 8771 Acked Flush 0 Done Sent Done false
29264 8772 Acked Flush 0 Done Sent Done false
29265 8773 Acked Write 2 Done Sent Done false
29266 8774 Acked Write 10 Done Sent Done false
29267 8775 Acked Write 1 Done Sent Done false
29268 8776 Acked Flush 0 Done Sent Done false
29269 8777 Acked Read 9 Done Sent Done false
29270 8778 Acked Write 5 Done Sent Done false
29271 8779 Acked Write 9 Done Sent Done false
29272 8780 Acked Read 10 Done Sent Done false
29273 8781 Acked Read 7 Done Sent Done false
29274 8782 Acked Read 6 Done Sent Done false
29275 8783 Acked Write 5 Done Sent Done false
29276 8784 Acked Read 3 Done Sent Done false
29277 8785 Acked Flush 0 Done Sent Done false
29278 8786 Acked Write 1 Done Sent Done false
29279 8787 Acked Write 9 Done Sent Done false
29280 8788 Acked Read 10 Done Sent Done false
29281 8789 Acked Read 2 Done Sent Done false
29282 8790 Acked Read 8 Done Sent Done false
29283 8791 Acked Read 1 Done Sent Done false
29284 8792 Acked Read 4 Done Sent Done false
29285 8793 Acked Write 8 Done Sent Done false
29286 8794 Acked Read 3 Done Sent Done false
29287 8795 Acked Read 3 Done Sent Done false
29288 8796 Acked Read 9 Done Sent Done false
29289 8797 Acked Flush 0 Done Sent Done false
29290 8798 Acked Read 5 Done Sent Done false
29291 8799 Acked Write 10 Done Sent Done false
29292 8800 Acked Read 4 Done Sent Done false
29293 8801 Acked Write 5 Done Sent Done false
29294 8802 Acked Read 1 Done Sent Done false
29295 8803 Acked Write 6 Done Sent Done false
29296 8804 Acked Write 4 Done Sent Done false
29297 8805 Acked Read 1 Done Sent Done false
29298 8806 Acked Write 8 Done Sent Done false
29299 8807 Acked Read 6 Done Sent Done false
29300 8808 Acked Write 8 Done Sent Done false
29301 8809 Acked Read 4 Done Sent Done false
29302 8810 Acked Write 9 Done Sent Done false
29303 8811 Acked Write 2 Done Sent Done false
29304 8812 Acked Read 9 Done Sent Done false
29305 8813 Acked Flush 0 Done Sent Done false
29306 8814 Acked Write 8 Done Sent Done false
29307 8815 Acked Write 4 Done Sent Done false
29308 8816 Acked Write 5 Done Sent Done false
29309 8817 Acked Read 10 Done Sent Done false
29310 8818 Acked Write 7 Done Sent Done false
29311 8819 Acked Write 1 Done Sent Done false
29312 8820 Acked Write 5 Done Sent Done false
29313 8821 Acked Read 9 Done Sent Done false
29314 8822 Acked Read 1 Done Sent Done false
29315 8823 Acked Write 8 Done Sent Done false
29316 8824 Acked Write 5 Done Sent Done false
29317 8825 Acked Write 10 Done Sent Done false
29318 8826 Acked Write 1 Done Sent Done false
29319 8827 Acked Read 3 Done Sent Done false
29320 8828 Acked Read 9 Done Sent Done false
29321 8829 Acked Write 9 Done Sent Done false
29322 8830 Acked Read 1 Done Sent Done false
29323 8831 Acked Read 1 Done Sent Done false
29324 8832 Acked Write 4 Done Sent Done false
29325 8833 Acked Write 8 Done Sent Done false
29326 8834 Acked Read 1 Done Sent Done false
29327 8835 Acked Write 8 Done Sent Done false
29328 8836 Acked Write 9 Done Sent Done false
29329 8837 Acked Write 4 Done Sent Done false
29330 8838 Acked Read 9 Done Sent Done false
29331 8839 Acked Read 4 Done Sent Done false
29332 8840 Acked Read 8 Done Sent Done false
29333 8841 Acked Read 3 Done Sent Done false
29334 8842 Acked Read 3 Done Sent Done false
29335 8843 Acked Write 3 Done Sent Done false
29336 8844 Acked Read 5 Done Sent Done false
29337 8845 Acked Write 7 Done Sent Done false
29338 8846 Acked Read 3 Done Sent Done false
29339 8847 Acked Read 1 Done Sent Done false
29340 8848 Acked Read 7 Done Sent Done false
29341 8849 Acked Write 4 Done Sent Done false
29342 8850 Acked Flush 0 Done Sent Done false
29343 STATES DS:0 DS:1 DS:2 TOTAL
29344 Sent 0 302 0 302
29345 Done 302 0 302 604
29346 Skipped 0 0 0 0
29347 Error 0 0 0 0
29348 Last Flush: 8850 8548 8850
29349 Downstairs last five completed: 8548 8547 8546 8545 8544
29350 Upstairs last five completed: 8850 8849 8848 8847 8846
29351 CLIENT: Up:0 ds:302 act:2
293522024-12-09T15:14:01.319ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
293532024-12-09T15:14:01.320ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:35777) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
293542024-12-09T15:14:01.321ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
293552024-12-09T15:14:01.321ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
293562024-12-09T15:14:01.321ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
293572024-12-09T15:14:01.321ZINFOcrucible: send last flush ID to this DS: 8548 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
293582024-12-09T15:14:01.321ZINFOcrucible: Replied this last flush ID: 8548 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
293592024-12-09T15:14:01.321ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
293602024-12-09T15:14:01.322ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
293612024-12-09T15:14:01.322ZINFOcrucible: [1] client re-new 302 jobs since flush 8548 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
293622024-12-09T15:14:01.322ZINFOcrucible: [1] Replayed 302 jobs since flush: 8548 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
29363 ----------------------------------------------------------------
29364 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
29365 Downstairs last five completed: 8850 8849 8848 8847 8846
29366 Upstairs last five completed: 8850 8849 8848 8847 8846
29367 CLIENT: Up:0 ds:0 act:3
29368 Replay: All jobs finished
29369 Main task has work to do, go find it
29370 got dsc Stop(2)
29371 stop 2
29372 Dec 09 15:14:01.567 INFO [8830] Got stop action so:false kr:false
29373 request completed, latency_us: 322, response_code: 204, uri: /stop/cid/2, method: POST, req_id: edffa5b1-008e-4f95-9c49-71822befb9f7, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
293742024-12-09T15:14:01.568ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
293752024-12-09T15:14:01.568ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
293762024-12-09T15:14:01.568ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
293772024-12-09T15:14:01.568ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
29378 Dec 09 15:14:01.568{"msg":"[2] Transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:14:01.568665765Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
29379 INFO request completed, latency_us: 152, response_code: 200, uri: /state/cid/2, method: GET, req_id: bb0eca2a-d190-496d-b840-b3762a3d8e23, remote_addr: 127.0.0.1:47510, local_addr: 127.0{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:14:01.56874886Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
29380 .0.1:9998
29381 [8830] Exited with: ExitStatus(unix_wait_status(9))
29382 [8830][2] reports Exit
29383 Dec 09 15:14:05.570 INFO request completed, latency_us: 245, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7da81989-9dae-4ec4-9350-d7bfe94f3fb5, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
29384 001/300 Write block 3904 len 36864 data: 7 7 7 7 6 4 4 4 5
29385 002/300 Write block 1439 len 32768 data: 6 5 6 8 9 8 8 8
29386 003/300 Write block 3147 len 32768 data: 7 6 6 8 9 9 9 10
29387 004/300 Read block 3480 len 24576
29388 005/300 Write block 4233 len 12288 data: 5 5 5
29389 006/300 Read block 4544 len 28672
29390 007/300 Read block 4535 len 32768
29391 008/300 Write block 2417 len 24576 data: 3 3 4 4 4 5
29392 009/300 Write block 4880 len 16384 data: 4 4 3 3
29393 010/300 Write block 113 len 28672 data: 6 6 5 5 4 3 3
29394 011/300 Read block 484 len 4096
29395 012/300 Write block 673 len 12288 data: 5 4 4
29396 013/300 Write block 4494 len 20480 data: 8 7 6 6 6
29397 014/300 Write block 1089 len 32768 data: 8 9 8 7 8 9 7 6
29398 015/300 Write block 4085 len 4096 data: 6
29399 016/300 Write block 4940 len 36864 data: 4 4 4 4 3 4 5 4 5
29400 017/300 Write block 127 len 36864 data: 5 7 5 5 7 8 8 8 8
29401 018/300 Read block 4962 len 4096
29402 019/300 Read block 1713 len 28672
29403 020/300 Read block 4244 len 4096
29404 021/300 Read block 1773 len 28672
29405 022/300 Read block 2132 len 12288
29406 023/300 Read block 708 len 32768
29407 024/300 Read block 3475 len 16384
29408 025/300 Read block 1294 len 40960
29409 026/300 Read block 2689 len 4096
29410 027/300 Write block 2733 len 36864 data: 7 8 9 8 7 8 6 7 6
29411 028/300 Write block 4256 len 36864 data: 7 5 5 7 7 8 7 6 5
29412 029/300 Flush
29413 030/300 Write block 1566 len 32768 data: 8 6 7 7 8 7 7 7
29414 031/300 Read block 1748 len 16384
29415 032/300 Write block 537 len 40960 data: 8 8 6 4 5 5 5 5 4 6
29416 033/300 Write block 1458 len 12288 data: 6 6 6
29417 034/300 Flush
29418 035/300 Flush
29419 036/300 Flush
29420 037/300 Read block 1380 len 16384
29421 038/300 Read block 4262 len 36864
29422 039/300 Read block 4270 len 12288
29423 040/300 Read block 4027 len 12288
29424 041/300 Read block 8 len 24576
29425 042/300 Write block 313 len 40960 data: 6 6 7 7 6 7 6 7 7 11
29426 043/300 Read block 2196 len 24576
29427 044/300 Read block 376 len 36864
29428 045/300 Read block 921 len 16384
29429 046/300 Read block 893 len 24576
29430 047/300 Write block 3053 len 16384 data: 4 4 6 6
29431 048/300 Write block 963 len 32768 data: 6 5 3 3 3 3 3 3
29432 049/300 Write block 1041 len 16384 data: 8 8 8 7
29433 050/300 Write block 1308 len 8192 data: 5 5
29434 051/300 Write block 46 len 16384 data: 4 5 5 5
29435 052/300 Read block 2156 len 4096
29436 053/300 Write block 1268 len 8192 data: 6 5
29437 054/300 Read block 4157 len 12288
29438 055/300 Write block 4265 len 12288 data: 4 4 5
29439 056/300 Write block 3152 len 4096 data: 10
29440 057/300 Read block 3873 len 28672
29441 058/300 Read block 2842 len 28672
29442 059/300 Write block 4953 len 24576 data: 4 4 4 4 3 5
29443 060/300 Read block 4326 len 8192
29444 061/300 Read block 19 len 20480
29445 062/300 Read block 1297 len 40960
29446 063/300 Write block 3844 len 40960 data: 9 9 8 7 7 7 6 5 6 6
29447 064/300 Read block 2091 len 28672
29448 065/300 Write block 3962 len 20480 data: 3 4 4 5 5
29449 066/300 Read block 522 len 32768
29450 067/300 Flush
29451 068/300 Read block 4139 len 28672
29452 069/300 Write block 462 len 12288 data: 5 5 4
29453 070/300 Read block 3544 len 28672
29454 071/300 Read block 183 len 12288
29455 072/300 Read block 1681 len 16384
29456 073/300 Read block 3649 len 28672
29457 074/300 Read block 1672 len 36864
29458 075/300 Read block 1893 len 20480
29459 076/300 Read block 150 len 16384
29460 077/300 Write block 168 len 12288 data: 5 6 6
29461 078/300 Read block 2982 len 16384
29462 079/300 Write block 3086 len 16384 data: 4 4 5 7
29463 080/300 Read block 4822 len 16384
29464 081/300 Write block 1960 len 20480 data: 6 6 6 8 7
29465 082/300 Write block 3394 len 40960 data: 9 8 9 9 8 7 6 7 7 4
29466 083/300 Read block 2842 len 28672
29467 084/300 Write block 594 len 8192 data: 3 4
29468 085/300 Write block 1597 len 32768 data: 6 6 6 6 7 7 7 4
29469 086/300 Read block 2474 len 20480
29470 087/300 Read block 73 len 12288
29471 088/300 Flush
29472 089/300 Write block 2695 len 4096 data: 5
29473 090/300 Write block 651 len 28672 data: 5 4 5 5 6 4 4
29474 091/300 Read block 2463 len 12288
29475 092/300 Read block 702 len 16384
29476 093/300 Read block 2104 len 16384
29477 094/300 Flush
29478 095/300 Read block 4275 len 16384
29479 096/300 Read block 3736 len 8192
29480 097/300 Read block 2844 len 12288
29481 098/300 Read block 848 len 36864
29482 099/300 Write block 490 len 12288 data: 4 4 5
29483 100/300 Write block 1218 len 24576 data: 6 6 6 6 5 5
29484 101/300 Read block 2733 len 36864
29485 102/300 Read block 3308 len 16384
29486 103/300 Flush
29487 104/300 Read block 3599 len 24576
29488 105/300 Read block 374 len 36864
29489 106/300 Read block 996 len 24576
29490 107/300 Read block 2390 len 8192
29491 108/300 Read block 984 len 36864
29492 109/300 Flush
29493 110/300 Read block 2286 len 4096
29494 111/300 Read block 3347 len 20480
29495 112/300 Write block 3806 len 20480 data: 4 5 7 5 5
29496 113/300 Flush
29497 114/300 Flush
29498 115/300 Write block 478 len 32768 data: 7 6 7 7 10 9 9 7
29499 116/300 Flush
29500 117/300 Write block 3069 len 20480 data: 4 5 4 5 4
29501 118/300 Write block 2194 len 20480 data: 7 6 7 5 4
29502 119/300 Read block 729 len 40960
29503 120/300 Read block 1804 len 28672
29504 121/300 Read block 1278 len 20480
29505 122/300 Flush
29506 123/300 Read block 1788 len 36864
29507 124/300 Read block 1730 len 8192
29508 125/300 Read block 4252 len 40960
29509 126/300 Write block 4187 len 16384 data: 7 7 7 8
29510 127/300 Write block 4304 len 8192 data: 7 7
29511 128/300 Write block 2380 len 24576 data: 3 4 4 4 5 5
29512 129/300 Write block 905 len 28672 data: 5 6 5 5 7 5 6
29513 130/300 Read block 2140 len 28672
29514 131/300 Read block 1432 len 4096
29515 132/300 Read block 4293 len 8192
29516 133/300 Read block 1287 len 24576
29517 134/300 Read block 693 len 32768
29518 135/300 Read block 3324 len 36864
29519 136/300 Write block 2342 len 40960 data: 8 5 6 6 7 7 6 7 6 6
29520 137/300 Read block 4795 len 16384
29521 138/300 Write block 4632 len 28672 data: 4 4 4 5 5 4 3
29522 139/300 Flush
29523 140/300 Read block 4002 len 40960
29524 141/300 Write block 1741 len 36864 data: 6 5 7 7 6 7 6 6 6
29525 142/300 Read block 3154 len 40960
29526 143/300 Read block 1918 len 16384
29527 144/300 Read block 607 len 36864
29528 145/300 Write block 3899 len 40960 data: 3 4 4 4 6 8 8 8 8 7
29529 146/300 Flush
29530 147/300 Write block 4915 len 12288 data: 5 4 3
29531 148/300 Read block 4294 len 20480
29532 149/300 Read block 2511 len 20480
29533 150/300 Read block 782 len 28672
29534 151/300 Write block 3894 len 24576 data: 4 4 4 4 4 4
29535 152/300 Flush
29536 153/300 Flush
29537 154/300 Read block 1606 len 32768
29538 155/300 Read block 1555 len 16384
29539 156/300 Write block 1427 len 8192 data: 5 6
29540 157/300 Write block 844 len 32768 data: 4 5 4 4 3 3 5 6
29541 158/300 Write block 3148 len 28672 data: 7 7 9 10 11 10 11
29542 159/300 Read block 2821 len 32768
29543 160/300 Read block 988 len 40960
29544 161/300 Read block 3626 len 16384
29545 162/300 Read block 2783 len 24576
29546 163/300 Read block 2854 len 20480
29547 164/300 Write block 2723 len 32768 data: 4 5 4 4 4 5 6 4
29548 165/300 Read block 4764 len 32768
29549 166/300 Write block 3765 len 12288 data: 5 5 4
29550 167/300 Read block 3531 len 24576
29551 168/300 Write block 2934 len 40960 data: 9 10 8 8 9 9 8 8 8 7
29552 169/300 Read block 4181 len 8192
29553 170/300 Write block 1528 len 8192 data: 7 6
29554 171/300 Write block 4949 len 4096 data: 7
29555 172/300 Write block 1917 len 20480 data: 4 4 4 3 3
29556 173/300 Flush
29557 174/300 Write block 2897 len 12288 data: 8 6 7
29558 175/300 Write block 4398 len 4096 data: 6
29559 176/300 Read block 3248 len 12288
29560 177/300 Write block 1220 len 40960 data: 7 7 6 6 5 6 7 7 6 7
29561 178/300 Read block 3191 len 24576
29562 179/300 Read block 44 len 12288
29563 180/300 Write block 3577 len 24576 data: 3 5 6 7 8 8
29564 181/300 Write block 2205 len 4096 data: 4
29565 182/300 Read block 4715 len 12288
29566 183/300 Read block 3602 len 24576
29567 184/300 Read block 3008 len 4096
29568 185/300 Flush
29569 186/300 Write block 2483 len 36864 data: 8 8 8 7 6 2 2 2 2
29570 187/300 Write block 4318 len 20480 data: 5 6 6 8 8
29571 188/300 Write block 3804 len 32768 data: 5 4 5 6 8 6 6 4
29572 189/300 Read block 4185 len 28672
29573 190/300 Read block 4992 len 20480
29574 191/300 Read block 1853 len 16384
29575 192/300 Write block 2730 len 8192 data: 5 4
29576 193/300 Write block 1898 len 4096 data: 5
29577 194/300 Read block 3497 len 4096
29578 195/300 Write block 1492 len 32768 data: 4 3 3 3 4 7 7 6
29579 196/300 Read block 4485 len 20480
29580 197/300 Read block 854 len 4096
29581 198/300 Read block 3540 len 24576
29582 199/300 Read block 3162 len 36864
29583 200/300 Read block 4948 len 16384
29584 201/300 Read block 3946 len 24576
29585 202/300 Write block 4611 len 28672 data: 2 3 3 5 5 6 5
29586 203/300 Read block 728 len 20480
29587 204/300 Write block 4336 len 20480 data: 4 5 5 4 5
29588 205/300 Read block 4247 len 12288
29589 206/300 Read block 4731 len 8192
29590 207/300 Write block 1778 len 40960 data: 4 7 7 6 6 5 5 6 7 7
29591 208/300 Read block 1015 len 20480
29592 209/300 Read block 2363 len 20480
29593 210/300 Read block 3480 len 32768
29594 211/300 Read block 1911 len 4096
29595 212/300 Read block 4700 len 4096
29596 213/300 Write block 2323 len 40960 data: 3 3 4 4 4 4 4 4 4 4
29597 214/300 Write block 4489 len 12288 data: 5 6 6
29598 215/300 Flush
29599 216/300 Flush
29600 217/300 Read block 2210 len 32768
29601 218/300 Flush
29602 219/300 Read block 2907 len 16384
29603 220/300 Read block 2084 len 12288
29604 221/300 Read block 455 len 40960
29605 222/300 Write block 957 len 4096 data: 5
29606 223/300 Read block 3363 len 40960
29607 224/300 Write block 2664 len 20480 data: 3 3 3 5 5
29608 225/300 Read block 3467 len 36864
29609 226/300 Write block 4672 len 12288 data: 2 2 2
29610 227/300 Read block 745 len 4096
29611 228/300 Write block 4029 len 8192 data: 7 6
29612 229/300 Read block 4580 len 32768
29613 230/300 Write block 1994 len 16384 data: 4 5 8 9
29614 231/300 Read block 2742 len 12288
29615 232/300 Write block 132 len 4096 data: 9
29616 233/300 Read block 3049 len 4096
29617 234/300 Read block 3184 len 4096
29618 235/300 Read block 2485 len 36864
29619 236/300 Read block 4312 len 8192
29620 237/300 Write block 4186 len 16384 data: 8 8 8 8
29621 238/300 Write block 4378 len 28672 data: 7 7 7 8 6 7 6
29622 239/300 Write block 99 len 12288 data: 4 5 3
29623 240/300 Read block 2377 len 40960
29624 241/300 Read block 1783 len 28672
29625 242/300 Write block 1673 len 4096 data: 3
29626 243/300 Flush
29627 244/300 Read block 2084 len 12288
29628 245/300 Read block 2631 len 28672
29629 246/300 Read block 3345 len 32768
29630 247/300 Write block 2469 len 32768 data: 5 6 6 5 5 5 5 4
29631 248/300 Write block 2299 len 40960 data: 7 6 6 5 5 4 4 4 2 2
29632 249/300 Write block 1110 len 40960 data: 7 8 7 6 8 7 7 7 6 6
29633 250/300 Write block 4908 len 36864 data: 2 2 3 3 5 6 6 6 5
29634 251/300 Flush
29635 252/300 Flush
29636 253/300 Read block 1796 len 28672
29637 254/300 Write block 3607 len 16384 data: 6 4 5 5
29638 255/300 Read block 2867 len 20480
29639 256/300 Write block 63 len 12288 data: 3 3 2
29640 257/300 Write block 2114 len 32768 data: 5 5 6 5 4 3 5 5
29641 258/300 Read block 1212 len 40960
29642 259/300 Read block 893 len 40960
29643 260/300 Write block 2006 len 24576 data: 9 8 9 7 6 6
29644 261/300 Read block 1067 len 32768
29645 262/300 Read block 2744 len 4096
29646 263/300 Read block 168 len 4096
29647 264/300 Write block 1078 len 28672 data: 7 8 6 4 5 5 6
29648 265/300 Read block 2217 len 28672
29649 266/300 Read block 4905 len 4096
29650 267/300 Write block 2064 len 4096 data: 4
29651 268/300 Read block 3276 len 32768
29652 269/300 Read block 1673 len 8192
29653 270/300 Write block 2750 len 24576 data: 4 5 6 6 5 5
29654 271/300 Write block 1462 len 8192 data: 4 6
29655 272/300 Flush
29656 273/300 Read block 3873 len 16384
29657 274/300 Write block 3571 len 32768 data: 3 3 4 4 2 2 4 6
29658 275/300 Write block 250 len 28672 data: 9 9 9 9 7 7 6
29659 276/300 Write block 872 len 20480 data: 6 6 5 4 6
29660 277/300 Read block 2013 len 12288
29661 278/300 Write block 4904 len 28672 data: 5 5 4 2 3 3 4
29662 279/300 Write block 2268 len 28672 data: 2 2 2 2 2 2 2
29663 280/300 Read block 4276 len 20480
29664 281/300 Read block 3763 len 4096
29665 282/300 Read block 2811 len 16384
29666 283/300 Read block 982 len 40960
29667 284/300 Write block 4987 len 20480 data: 9 9 10 9 11
29668 285/300 Write block 168 len 20480 data: 6 7 7 5 5
29669 286/300 Write block 3446 len 16384 data: 5 6 5 4
29670 287/300 Read block 1158 len 36864
29671 288/300 Read block 2899 len 16384
29672 289/300 Read block 1093 len 16384
29673 290/300 Read block 1213 len 12288
29674 291/300 Read block 2283 len 28672
29675 292/300 Write block 2069 len 12288 data: 5 5 5
29676 293/300 Write block 4761 len 36864 data: 5 6 6 5 5 7 6 6 5
29677 294/300 Write block 1359 len 24576 data: 4 4 2 4 6 6
29678 295/300 Flush
29679 296/300 Read block 192 len 12288
29680 297/300 Flush
29681 298/300 Write block 2785 len 4096 data: 8
29682 299/300 Write block 4048 len 4096 data: 6
29683 300/300 Read block 545 len 32768
29684 301/300 Read block 3044 len 32768
29685 Main task has work to do, go find it
29686 got dsc Start(2)
29687 start 2
29688 Dec 09 15:14:05.814 INFO request completed, latency_us: 224, response_code: 204, uri: /start/cid/2, method: POST, req_id: 63c81763-1c81-488a-abcc-ef6da2593362, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
29689 Replay: started 2, returned:Ok(())
29690 ----------------------------------------------------------------
29691 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29692 States: Active Active Offline
29693 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29694 8851 Acked Write 9 Done Done Sent false
29695 8852 Acked Write 8 Done Done Sent false
29696 8853 Acked Write 8 Done Done Sent false
29697 8854 Acked Read 6 Done Done Sent false
29698 8855 Acked Write 3 Done Done Sent false
29699 8856 Acked Read 7 Done Done Sent false
29700 8857 Acked Read 8 Done Done Sent false
29701 8858 Acked Write 6 Done Done Sent false
29702 8859 Acked Write 4 Done Done Sent false
29703 8860 Acked Write 7 Done Done Sent false
29704 8861 Acked Read 1 Done Done Sent false
29705 8862 Acked Write 3 Done Done Sent false
29706 8863 Acked Write 5 Done Done Sent false
29707 8864 Acked Write 8 Done Done Sent false
29708 8865 Acked Write 1 Done Done Sent false
29709 8866 Acked Write 9 Done Done Sent false
29710 8867 Acked Write 9 Done Done Sent false
29711 8868 Acked Read 1 Done Done Sent false
29712 8869 Acked Read 7 Done Done Sent false
29713 8870 Acked Read 1 Done Done Sent false
29714 8871 Acked Read 7 Done Done Sent false
29715 8872 Acked Read 3 Done Done Sent false
29716 8873 Acked Read 8 Done Done Sent false
29717 8874 Acked Read 4 Done Done Sent false
29718 8875 Acked Read 10 Done Done Sent false
29719 8876 Acked Read 1 Done Done Sent false
29720 8877 Acked Write 9 Done Done Sent false
29721 8878 Acked Write 9 Done Done Sent false
29722 8879 Acked Flush 0 Done Done Sent false
29723 8880 Acked Write 8 Done Done Sent false
29724 8881 Acked Read 4 Done Done Sent false
29725 8882 Acked Write 10 Done Done Sent false
29726 8883 Acked Write 3 Done Done Sent false
29727 8884 Acked Flush 0 Done Done Sent false
29728 8885 Acked Flush 0 Done Done Sent false
29729 8886 Acked Flush 0 Done Done Sent false
29730 8887 Acked Read 4 Done Done Sent false
29731 8888 Acked Read 9 Done Done Sent false
29732 8889 Acked Read 3 Done Done Sent false
29733 8890 Acked Read 3 Done Done Sent false
29734 8891 Acked Read 6 Done Done Sent false
29735 8892 Acked Write 10 Done Done Sent false
29736 8893 Acked Read 6 Done Done Sent false
29737 8894 Acked Read 9 Done Done Sent false
29738 8895 Acked Read 4 Done Done Sent false
29739 8896 Acked Read 6 Done Done Sent false
29740 8897 Acked Write 4 Done Done Sent false
29741 8898 Acked Write 8 Done Done Sent false
29742 8899 Acked Write 4 Done Done Sent false
29743 8900 Acked Write 2 Done Done Sent false
29744 8901 Acked Write 4 Done Done Sent false
29745 8902 Acked Read 1 Done Done Sent false
29746 8903 Acked Write 2 Done Done Sent false
29747 8904 Acked Read 3 Done Done Sent false
29748 8905 Acked Write 3 Done Done Sent false
29749 8906 Acked Write 1 Done Done Sent false
29750 8907 Acked Read 7 Done Done Sent false
29751 8908 Acked Read 7 Done Done Sent false
29752 8909 Acked Write 6 Done Done Sent false
29753 8910 Acked Read 2 Done Done Sent false
29754 8911 Acked Read 5 Done Done Sent false
29755 8912 Acked Read 10 Done Done Sent false
29756 8913 Acked Write 10 Done Done Sent false
29757 8914 Acked Read 7 Done Done Sent false
29758 8915 Acked Write 5 Done Done Sent false
29759 8916 Acked Read 8 Done Done Sent false
29760 8917 Acked Flush 0 Done Done Sent false
29761 8918 Acked Read 7 Done Done Sent false
29762 8919 Acked Write 3 Done Done Sent false
29763 8920 Acked Read 7 Done Done Sent false
29764 8921 Acked Read 3 Done Done Sent false
29765 8922 Acked Read 4 Done Done Sent false
29766 8923 Acked Read 7 Done Done Sent false
29767 8924 Acked Read 9 Done Done Sent false
29768 8925 Acked Read 5 Done Done Sent false
29769 8926 Acked Read 4 Done Done Sent false
29770 8927 Acked Write 3 Done Done Sent false
29771 8928 Acked Read 4 Done Done Sent false
29772 8929 Acked Write 4 Done Done Sent false
29773 8930 Acked Read 4 Done Done Sent false
29774 8931 Acked Write 5 Done Done Sent false
29775 8932 Acked Write 10 Done Done Sent false
29776 8933 Acked Read 7 Done Done Sent false
29777 8934 Acked Write 2 Done Done Sent false
29778 8935 Acked Write 8 Done Done Sent false
29779 8936 Acked Read 5 Done Done Sent false
29780 8937 Acked Read 3 Done Done Sent false
29781 8938 Acked Flush 0 Done Done Sent false
29782 8939 Acked Write 1 Done Done Sent false
29783 8940 Acked Write 7 Done Done Sent false
29784 8941 Acked Read 3 Done Done Sent false
29785 8942 Acked Read 4 Done Done Sent false
29786 8943 Acked Read 4 Done Done Sent false
29787 8944 Acked Flush 0 Done Done Sent false
29788 8945 Acked Read 4 Done Done Sent false
29789 8946 Acked Read 2 Done Done Sent false
29790 8947 Acked Read 3 Done Done Sent false
29791 8948 Acked Read 9 Done Done Sent false
29792 8949 Acked Write 3 Done Done Sent false
29793 8950 Acked Write 6 Done Done Sent false
29794 8951 Acked Read 9 Done Done Sent false
29795 8952 Acked Read 4 Done Done Sent false
29796 8953 Acked Flush 0 Done Done Sent false
29797 8954 Acked Read 6 Done Done Sent false
29798 8955 Acked Read 9 Done Done Sent false
29799 8956 Acked Read 6 Done Done Sent false
29800 8957 Acked Read 2 Done Done Sent false
29801 8958 Acked Read 9 Done Done Sent false
29802 8959 Acked Flush 0 Done Done Sent false
29803 8960 Acked Read 1 Done Done Sent false
29804 8961 Acked Read 5 Done Done Sent false
29805 8962 Acked Write 5 Done Done Sent false
29806 8963 Acked Flush 0 Done Done Sent false
29807 8964 Acked Flush 0 Done Done Sent false
29808 8965 Acked Write 8 Done Done Sent false
29809 8966 Acked Flush 0 Done Done Sent false
29810 8967 Acked Write 5 Done Done Sent false
29811 8968 Acked Write 5 Done Done Sent false
29812 8969 Acked Read 10 Done Done Sent false
29813 8970 Acked Read 7 Done Done Sent false
29814 8971 Acked Read 5 Done Done Sent false
29815 8972 Acked Flush 0 Done Done Sent false
29816 8973 Acked Read 9 Done Done Sent false
29817 8974 Acked Read 2 Done Done Sent false
29818 8975 Acked Read 10 Done Done Sent false
29819 8976 Acked Write 4 Done Done Sent false
29820 8977 Acked Write 2 Done Done Sent false
29821 8978 Acked Write 6 Done Done Sent false
29822 8979 Acked Write 7 Done Done Sent false
29823 8980 Acked Read 7 Done Done Sent false
29824 8981 Acked Read 1 Done Done Sent false
29825 8982 Acked Read 2 Done Done Sent false
29826 8983 Acked Read 6 Done Done Sent false
29827 8984 Acked Read 8 Done Done Sent false
29828 8985 Acked Read 9 Done Done Sent false
29829 8986 Acked Write 10 Done Done Sent false
29830 8987 Acked Read 4 Done Done Sent false
29831 8988 Acked Write 7 Done Done Sent false
29832 8989 Acked Flush 0 Done Done Sent false
29833 8990 Acked Read 10 Done Done Sent false
29834 8991 Acked Write 9 Done Done Sent false
29835 8992 Acked Read 10 Done Done Sent false
29836 8993 Acked Read 4 Done Done Sent false
29837 8994 Acked Read 9 Done Done Sent false
29838 8995 Acked Write 10 Done Done Sent false
29839 8996 Acked Flush 0 Done Done Sent false
29840 8997 Acked Write 3 Done Done Sent false
29841 8998 Acked Read 5 Done Done Sent false
29842 8999 Acked Read 5 Done Done Sent false
29843 9000 Acked Read 7 Done Done Sent false
29844 9001 Acked Write 6 Done Done Sent false
29845 9002 Acked Flush 0 Done Done Sent false
29846 9003 Acked Flush 0 Done Done Sent false
29847 9004 Acked Read 8 Done Done Sent false
29848 9005 Acked Read 4 Done Done Sent false
29849 9006 Acked Write 2 Done Done Sent false
29850 9007 Acked Write 8 Done Done Sent false
29851 9008 Acked Write 7 Done Done Sent false
29852 9009 Acked Read 8 Done Done Sent false
29853 9010 Acked Read 10 Done Done Sent false
29854 9011 Acked Read 4 Done Done Sent false
29855 9012 Acked Read 6 Done Done Sent false
29856 9013 Acked Read 5 Done Done Sent false
29857 9014 Acked Write 8 Done Done Sent false
29858 9015 Acked Read 8 Done Done Sent false
29859 9016 Acked Write 3 Done Done Sent false
29860 9017 Acked Read 6 Done Done Sent false
29861 9018 Acked Write 10 Done Done Sent false
29862 9019 Acked Read 2 Done Done Sent false
29863 9020 Acked Write 2 Done Done Sent false
29864 9021 Acked Write 1 Done Done Sent false
29865 9022 Acked Write 5 Done Done Sent false
29866 9023 Acked Flush 0 Done Done Sent false
29867 9024 Acked Write 3 Done Done Sent false
29868 9025 Acked Write 1 Done Done Sent false
29869 9026 Acked Read 3 Done Done Sent false
29870 9027 Acked Write 10 Done Done Sent false
29871 9028 Acked Read 6 Done Done Sent false
29872 9029 Acked Read 3 Done Done Sent false
29873 9030 Acked Write 6 Done Done Sent false
29874 9031 Acked Write 1 Done Done Sent false
29875 9032 Acked Read 3 Done Done Sent false
29876 9033 Acked Read 6 Done Done Sent false
29877 9034 Acked Read 1 Done Done Sent false
29878 9035 Acked Flush 0 Done Done Sent false
29879 9036 Acked Write 9 Done Done Sent false
29880 9037 Acked Write 5 Done Done Sent false
29881 9038 Acked Write 8 Done Done Sent false
29882 9039 Acked Read 7 Done Done Sent false
29883 9040 Acked Read 5 Done Done Sent false
29884 9041 Acked Read 4 Done Done Sent false
29885 9042 Acked Write 2 Done Done Sent false
29886 9043 Acked Write 1 Done Done Sent false
29887 9044 Acked Read 1 Done Done Sent false
29888 9045 Acked Write 8 Done Done Sent false
29889 9046 Acked Read 5 Done Done Sent false
29890 9047 Acked Read 1 Done Done Sent false
29891 9048 Acked Read 6 Done Done Sent false
29892 9049 Acked Read 9 Done Done Sent false
29893 9050 Acked Read 4 Done Done Sent false
29894 9051 Acked Read 6 Done Done Sent false
29895 9052 Acked Write 7 Done Done Sent false
29896 9053 Acked Read 5 Done Done Sent false
29897 9054 Acked Write 5 Done Done Sent false
29898 9055 Acked Read 3 Done Done Sent false
29899 9056 Acked Read 2 Done Done Sent false
29900 9057 Acked Write 10 Done Done Sent false
29901 9058 Acked Read 5 Done Done Sent false
29902 9059 Acked Read 5 Done Done Sent false
29903 9060 Acked Read 8 Done Done Sent false
29904 9061 Acked Read 1 Done Done Sent false
29905 9062 Acked Read 1 Done Done Sent false
29906 9063 Acked Write 10 Done Done Sent false
29907 9064 Acked Write 3 Done Done Sent false
29908 9065 Acked Flush 0 Done Done Sent false
29909 9066 Acked Flush 0 Done Done Sent false
29910 9067 Acked Read 8 Done Done Sent false
29911 9068 Acked Flush 0 Done Done Sent false
29912 9069 Acked Read 4 Done Done Sent false
29913 9070 Acked Read 3 Done Done Sent false
29914 9071 Acked Read 10 Done Done Sent false
29915 9072 Acked Write 1 Done Done Sent false
29916 9073 Acked Read 10 Done Done Sent false
29917 9074 Acked Write 5 Done Done Sent false
29918 9075 Acked Read 9 Done Done Sent false
29919 9076 Acked Write 3 Done Done Sent false
29920 9077 Acked Read 1 Done Done Sent false
29921 9078 Acked Write 2 Done Done Sent false
29922 9079 Acked Read 8 Done Done Sent false
29923 9080 Acked Write 4 Done Done Sent false
29924 9081 Acked Read 3 Done Done Sent false
29925 9082 Acked Write 1 Done Done Sent false
29926 9083 Acked Read 1 Done Done Sent false
29927 9084 Acked Read 1 Done Done Sent false
29928 9085 Acked Read 9 Done Done Sent false
29929 9086 Acked Read 2 Done Done Sent false
29930 9087 Acked Write 4 Done Done Sent false
29931 9088 Acked Write 7 Done Done Sent false
29932 9089 Acked Write 3 Done Done Sent false
29933 9090 Acked Read 10 Done Done Sent false
29934 9091 Acked Read 7 Done Done Sent false
29935 9092 Acked Write 1 Done Done Sent false
29936 9093 Acked Flush 0 Done Done Sent false
29937 9094 Acked Read 3 Done Done Sent false
29938 9095 Acked Read 7 Done Done Sent false
29939 9096 Acked Read 8 Done Done Sent false
29940 9097 Acked Write 8 Done Done Sent false
29941 9098 Acked Write 10 Done Done Sent false
29942 9099 Acked Write 10 Done Done Sent false
29943 9100 Acked Write 9 Done Done Sent false
29944 9101 Acked Flush 0 Done Done Sent false
29945 9102 Acked Flush 0 Done Done Sent false
29946 9103 Acked Read 7 Done Done Sent false
29947 9104 Acked Write 4 Done Done Sent false
29948 9105 Acked Read 5 Done Done Sent false
29949 9106 Acked Write 3 Done Done Sent false
29950 9107 Acked Write 8 Done Done Sent false
29951 9108 Acked Read 10 Done Done Sent false
29952 9109 Acked Read 10 Done Done Sent false
29953 9110 Acked Write 6 Done Done Sent false
29954 9111 Acked Read 8 Done Done Sent false
29955 9112 Acked Read 1 Done Done Sent false
29956 9113 Acked Read 1 Done Done Sent false
29957 9114 Acked Write 7 Done Done Sent false
29958 9115 Acked Read 7 Done Done Sent false
29959 9116 Acked Read 1 Done Done Sent false
29960 9117 Acked Write 1 Done Done Sent false
29961 9118 Acked Read 8 Done Done Sent false
29962 9119 Acked Read 2 Done Done Sent false
29963 9120 Acked Write 6 Done Done Sent false
29964 9121 Acked Write 2 Done Done Sent false
29965 9122 Acked Flush 0 Done Done Sent false
29966 9123 Acked Read 4 Done Done Sent false
29967 9124 Acked Write 8 Done Done Sent false
29968 9125 Acked Write 7 Done Done Sent false
29969 9126 Acked Write 5 Done Done Sent false
29970 9127 Acked Read 3 Done Done Sent false
29971 9128 Acked Write 7 Done Done Sent false
29972 9129 Acked Write 7 Done Done Sent false
29973 9130 Acked Read 5 Done Done Sent false
29974 9131 Acked Read 1 Done Done Sent false
29975 9132 Acked Read 4 Done Done Sent false
29976 9133 Acked Read 10 Done Done Sent false
29977 9134 Acked Write 5 Done Done Sent false
29978 9135 Acked Write 5 Done Done Sent false
29979 9136 Acked Write 4 Done Done Sent false
29980 9137 Acked Read 9 Done Done Sent false
29981 9138 Acked Read 4 Done Done Sent false
29982 9139 Acked Read 4 Done Done Sent false
29983 9140 Acked Read 3 Done Done Sent false
29984 9141 Acked Read 7 Done Done Sent false
29985 9142 Acked Write 3 Done Done Sent false
29986 9143 Acked Write 9 Done Done Sent false
29987 9144 Acked Write 6 Done Done Sent false
29988 9145 Acked Flush 0 Done Done Sent false
29989 9146 Acked Read 3 Done Done Sent false
29990 9147 Acked Flush 0 Done Done Sent false
29991 9148 Acked Write 1 Done Done Sent false
29992 9149 Acked Write 1 Done Done Sent false
29993 9150 Acked Read 8 Done Done Sent false
29994 9151 Acked Read 8 Done Done Sent false
29995 STATES DS:0 DS:1 DS:2 TOTAL
29996 Sent 0 0 301 301
29997 Done 301 301 0 602
29998 Skipped 0 0 0 0
29999 Error 0 0 0 0
30000 Last Flush: 9147 9147 8850
30001 Downstairs last five completed: 8850 8849 8848 8847 8846
30002 Upstairs last five completed: 9151 9150 9149 9148 9147
30003 CLIENT: Up:0 ds:301 act:2
30004 [8830] got start action
30005 [8830] I am going to restart
30006 Starting downstairs at port 8830
30007 Make output file at "/tmp/dsc/downstairs-8830.txt"
30008 [8830][2] reports Starting
30009 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1228)
30010 [8830][2] reports Running
30011 ----------------------------------------------------------------
30012 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
30013 States: Active Active Offline
30014 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30015 8851 Acked Write 9 Done Done Sent false
30016 8852 Acked Write 8 Done Done Sent false
30017 8853 Acked Write 8 Done Done Sent false
30018 8854 Acked Read 6 Done Done Sent false
30019 8855 Acked Write 3 Done Done Sent false
30020 8856 Acked Read 7 Done Done Sent false
30021 8857 Acked Read 8 Done Done Sent false
30022 8858 Acked Write 6 Done Done Sent false
30023 8859 Acked Write 4 Done Done Sent false
30024 8860 Acked Write 7 Done Done Sent false
30025 8861 Acked Read 1 Done Done Sent false
30026 8862 Acked Write 3 Done Done Sent false
30027 8863 Acked Write 5 Done Done Sent false
30028 8864 Acked Write 8 Done Done Sent false
30029 8865 Acked Write 1 Done Done Sent false
30030 8866 Acked Write 9 Done Done Sent false
30031 8867 Acked Write 9 Done Done Sent false
30032 8868 Acked Read 1 Done Done Sent false
30033 8869 Acked Read 7 Done Done Sent false
30034 8870 Acked Read 1 Done Done Sent false
30035 8871 Acked Read 7 Done Done Sent false
30036 8872 Acked Read 3 Done Done Sent false
30037 8873 Acked Read 8 Done Done Sent false
30038 8874 Acked Read 4 Done Done Sent false
30039 8875 Acked Read 10 Done Done Sent false
30040 8876 Acked Read 1 Done Done Sent false
30041 8877 Acked Write 9 Done Done Sent false
30042 8878 Acked Write 9 Done Done Sent false
30043 8879 Acked Flush 0 Done Done Sent false
30044 8880 Acked Write 8 Done Done Sent false
30045 8881 Acked Read 4 Done Done Sent false
30046 8882 Acked Write 10 Done Done Sent false
30047 8883 Acked Write 3 Done Done Sent false
30048 8884 Acked Flush 0 Done Done Sent false
30049 8885 Acked Flush 0 Done Done Sent false
30050 8886 Acked Flush 0 Done Done Sent false
30051 8887 Acked Read 4 Done Done Sent false
30052 8888 Acked Read 9 Done Done Sent false
30053 8889 Acked Read 3 Done Done Sent false
30054 8890 Acked Read 3 Done Done Sent false
30055 8891 Acked Read 6 Done Done Sent false
30056 8892 Acked Write 10 Done Done Sent false
30057 8893 Acked Read 6 Done Done Sent false
30058 8894 Acked Read 9 Done Done Sent false
30059 8895 Acked Read 4 Done Done Sent false
30060 8896 Acked Read 6 Done Done Sent false
30061 8897 Acked Write 4 Done Done Sent false
30062 8898 Acked Write 8 Done Done Sent false
30063 8899 Acked Write 4 Done Done Sent false
30064 8900 Acked Write 2 Done Done Sent false
30065 8901 Acked Write 4 Done Done Sent false
30066 8902 Acked Read 1 Done Done Sent false
30067 8903 Acked Write 2 Done Done Sent false
30068 8904 Acked Read 3 Done Done Sent false
30069 8905 Acked Write 3 Done Done Sent false
30070 8906 Acked Write 1 Done Done Sent false
30071 8907 Acked Read 7 Done Done Sent false
30072 8908 Acked Read 7 Done Done Sent false
30073 8909 Acked Write 6 Done Done Sent false
30074 8910 Acked Read 2 Done Done Sent false
30075 8911 Acked Read 5 Done Done Sent false
30076 8912 Acked Read 10 Done Done Sent false
30077 8913 Acked Write 10 Done Done Sent false
30078 8914 Acked Read 7 Done Done Sent false
30079 8915 Acked Write 5 Done Done Sent false
30080 8916 Acked Read 8 Done Done Sent false
30081 8917 Acked Flush 0 Done Done Sent false
30082 8918 Acked Read 7 Done Done Sent false
30083 8919 Acked Write 3 Done Done Sent false
30084 8920 Acked Read 7 Done Done Sent false
30085 8921 Acked Read 3 Done Done Sent false
30086 8922 Acked Read 4 Done Done Sent false
30087 8923 Acked Read 7 Done Done Sent false
30088 8924 Acked Read 9 Done Done Sent false
30089 8925 Acked Read 5 Done Done Sent false
30090 8926 Acked Read 4 Done Done Sent false
30091 8927 Acked Write 3 Done Done Sent false
30092 8928 Acked Read 4 Done Done Sent false
30093 8929 Acked Write 4 Done Done Sent false
30094 8930 Acked Read 4 Done Done Sent false
30095 8931 Acked Write 5 Done Done Sent false
30096 8932 Acked Write 10 Done Done Sent false
30097 8933 Acked Read 7 Done Done Sent false
30098 8934 Acked Write 2 Done Done Sent false
30099 8935 Acked Write 8 Done Done Sent false
30100 8936 Acked Read 5 Done Done Sent false
30101 8937 Acked Read 3 Done Done Sent false
30102 8938 Acked Flush 0 Done Done Sent false
30103 8939 Acked Write 1 Done Done Sent false
30104 8940 Acked Write 7 Done Done Sent false
30105 8941 Acked Read 3 Done Done Sent false
30106 8942 Acked Read 4 Done Done Sent false
30107 8943 Acked Read 4 Done Done Sent false
30108 8944 Acked Flush 0 Done Done Sent false
30109 8945 Acked Read 4 Done Done Sent false
30110 8946 Acked Read 2 Done Done Sent false
30111 8947 Acked Read 3 Done Done Sent false
30112 8948 Acked Read 9 Done Done Sent false
30113 8949 Acked Write 3 Done Done Sent false
30114 8950 Acked Write 6 Done Done Sent false
30115 8951 Acked Read 9 Done Done Sent false
30116 8952 Acked Read 4 Done Done Sent false
30117 8953 Acked Flush 0 Done Done Sent false
30118 8954 Acked Read 6 Done Done Sent false
30119 8955 Acked Read 9 Done Done Sent false
30120 8956 Acked Read 6 Done Done Sent false
30121 8957 Acked Read 2 Done Done Sent false
30122 8958 Acked Read 9 Done Done Sent false
30123 8959 Acked Flush 0 Done Done Sent false
30124 8960 Acked Read 1 Done Done Sent false
30125 8961 Acked Read 5 Done Done Sent false
30126 8962 Acked Write 5 Done Done Sent false
30127 8963 Acked Flush 0 Done Done Sent false
30128 8964 Acked Flush 0 Done Done Sent false
30129 8965 Acked Write 8 Done Done Sent false
30130 8966 Acked Flush 0 Done Done Sent false
30131 8967 Acked Write 5 Done Done Sent false
30132 8968 Acked Write 5 Done Done Sent false
30133 8969 Acked Read 10 Done Done Sent false
30134 8970 Acked Read 7 Done Done Sent false
30135 8971 Acked Read 5 Done Done Sent false
30136 8972 Acked Flush 0 Done Done Sent false
30137 8973 Acked Read 9 Done Done Sent false
30138 8974 Acked Read 2 Done Done Sent false
30139 8975 Acked Read 10 Done Done Sent false
30140 8976 Acked Write 4 Done Done Sent false
30141 8977 Acked Write 2 Done Done Sent false
30142 8978 Acked Write 6 Done Done Sent false
30143 8979 Acked Write 7 Done Done Sent false
30144 8980 Acked Read 7 Done Done Sent false
30145 8981 Acked Read 1 Done Done Sent false
30146 8982 Acked Read 2 Done Done Sent false
30147 8983 Acked Read 6 Done Done Sent false
30148 8984 Acked Read 8 Done Done Sent false
30149 8985 Acked Read 9 Done Done Sent false
30150 8986 Acked Write 10 Done Done Sent false
30151 8987 Acked Read 4 Done Done Sent false
30152 8988 Acked Write 7 Done Done Sent false
30153 8989 Acked Flush 0 Done Done Sent false
30154 8990 Acked Read 10 Done Done Sent false
30155 8991 Acked Write 9 Done Done Sent false
30156 8992 Acked Read 10 Done Done Sent false
30157 8993 Acked Read 4 Done Done Sent false
30158 8994 Acked Read 9 Done Done Sent false
30159 8995 Acked Write 10 Done Done Sent false
30160 8996 Acked Flush 0 Done Done Sent false
30161 8997 Acked Write 3 Done Done Sent false
30162 8998 Acked Read 5 Done Done Sent false
30163 8999 Acked Read 5 Done Done Sent false
30164 9000 Acked Read 7 Done Done Sent false
30165 9001 Acked Write 6 Done Done Sent false
30166 9002 Acked Flush 0 Done Done Sent false
30167 9003 Acked Flush 0 Done Done Sent false
30168 9004 Acked Read 8 Done Done Sent false
30169 9005 Acked Read 4 Done Done Sent false
30170 9006 Acked Write 2 Done Done Sent false
30171 9007 Acked Write 8 Done Done Sent false
30172 9008 Acked Write 7 Done Done Sent false
30173 9009 Acked Read 8 Done Done Sent false
30174 9010 Acked Read 10 Done Done Sent false
30175 9011 Acked Read 4 Done Done Sent false
30176 9012 Acked Read 6 Done Done Sent false
30177 9013 Acked Read 5 Done Done Sent false
30178 9014 Acked Write 8 Done Done Sent false
30179 9015 Acked Read 8 Done Done Sent false
30180 9016 Acked Write 3 Done Done Sent false
30181 9017 Acked Read 6 Done Done Sent false
30182 9018 Acked Write 10 Done Done Sent false
30183 9019 Acked Read 2 Done Done Sent false
30184 9020 Acked Write 2 Done Done Sent false
30185 9021 Acked Write 1 Done Done Sent false
30186 9022 Acked Write 5 Done Done Sent false
30187 9023 Acked Flush 0 Done Done Sent false
30188 9024 Acked Write 3 Done Done Sent false
30189 9025 Acked Write 1 Done Done Sent false
30190 9026 Acked Read 3 Done Done Sent false
30191 9027 Acked Write 10 Done Done Sent false
30192 9028 Acked Read 6 Done Done Sent false
30193 9029 Acked Read 3 Done Done Sent false
30194 9030 Acked Write 6 Done Done Sent false
30195 9031 Acked Write 1 Done Done Sent false
30196 9032 Acked Read 3 Done Done Sent false
30197 9033 Acked Read 6 Done Done Sent false
30198 9034 Acked Read 1 Done Done Sent false
30199 9035 Acked Flush 0 Done Done Sent false
30200 9036 Acked Write 9 Done Done Sent false
30201 9037 Acked Write 5 Done Done Sent false
30202 9038 Acked Write 8 Done Done Sent false
30203 9039 Acked Read 7 Done Done Sent false
30204 9040 Acked Read 5 Done Done Sent false
30205 9041 Acked Read 4 Done Done Sent false
30206 9042 Acked Write 2 Done Done Sent false
30207 9043 Acked Write 1 Done Done Sent false
30208 9044 Acked Read 1 Done Done Sent false
30209 9045 Acked Write 8 Done Done Sent false
30210 9046 Acked Read 5 Done Done Sent false
30211 9047 Acked Read 1 Done Done Sent false
30212 9048 Acked Read 6 Done Done Sent false
30213 9049 Acked Read 9 Done Done Sent false
30214 9050 Acked Read 4 Done Done Sent false
30215 9051 Acked Read 6 Done Done Sent false
30216 9052 Acked Write 7 Done Done Sent false
30217 9053 Acked Read 5 Done Done Sent false
30218 9054 Acked Write 5 Done Done Sent false
30219 9055 Acked Read 3 Done Done Sent false
30220 9056 Acked Read 2 Done Done Sent false
30221 9057 Acked Write 10 Done Done Sent false
30222 9058 Acked Read 5 Done Done Sent false
30223 9059 Acked Read 5 Done Done Sent false
30224 9060 Acked Read 8 Done Done Sent false
30225 9061 Acked Read 1 Done Done Sent false
30226 9062 Acked Read 1 Done Done Sent false
30227 9063 Acked Write 10 Done Done Sent false
30228 9064 Acked Write 3 Done Done Sent false
30229 9065 Acked Flush 0 Done Done Sent false
30230 9066 Acked Flush 0 Done Done Sent false
30231 9067 Acked Read 8 Done Done Sent false
30232 9068 Acked Flush 0 Done Done Sent false
30233 9069 Acked Read 4 Done Done Sent false
30234 9070 Acked Read 3 Done Done Sent false
30235 9071 Acked Read 10 Done Done Sent false
30236 9072 Acked Write 1 Done Done Sent false
30237 9073 Acked Read 10 Done Done Sent false
30238 9074 Acked Write 5 Done Done Sent false
30239 9075 Acked Read 9 Done Done Sent false
30240 9076 Acked Write 3 Done Done Sent false
30241 9077 Acked Read 1 Done Done Sent false
30242 9078 Acked Write 2 Done Done Sent false
30243 9079 Acked Read 8 Done Done Sent false
30244 9080 Acked Write 4 Done Done Sent false
30245 9081 Acked Read 3 Done Done Sent false
30246 9082 Acked Write 1 Done Done Sent false
30247 9083 Acked Read 1 Done Done Sent false
30248 9084 Acked Read 1 Done Done Sent false
30249 9085 Acked Read 9 Done Done Sent false
30250 9086 Acked Read 2 Done Done Sent false
30251 9087 Acked Write 4 Done Done Sent false
30252 9088 Acked Write 7 Done Done Sent false
30253 9089 Acked Write 3 Done Done Sent false
30254 9090 Acked Read 10 Done Done Sent false
30255 9091 Acked Read 7 Done Done Sent false
30256 9092 Acked Write 1 Done Done Sent false
30257 9093 Acked Flush 0 Done Done Sent false
30258 9094 Acked Read 3 Done Done Sent false
30259 9095 Acked Read 7 Done Done Sent false
30260 9096 Acked Read 8 Done Done Sent false
30261 9097 Acked Write 8 Done Done Sent false
30262 9098 Acked Write 10 Done Done Sent false
30263 9099 Acked Write 10 Done Done Sent false
30264 9100 Acked Write 9 Done Done Sent false
30265 9101 Acked Flush 0 Done Done Sent false
30266 9102 Acked Flush 0 Done Done Sent false
30267 9103 Acked Read 7 Done Done Sent false
30268 9104 Acked Write 4 Done Done Sent false
30269 9105 Acked Read 5 Done Done Sent false
30270 9106 Acked Write 3 Done Done Sent false
30271 9107 Acked Write 8 Done Done Sent false
30272 9108 Acked Read 10 Done Done Sent false
30273 9109 Acked Read 10 Done Done Sent false
30274 9110 Acked Write 6 Done Done Sent false
30275 9111 Acked Read 8 Done Done Sent false
30276 9112 Acked Read 1 Done Done Sent false
30277 9113 Acked Read 1 Done Done Sent false
30278 9114 Acked Write 7 Done Done Sent false
30279 9115 Acked Read 7 Done Done Sent false
30280 9116 Acked Read 1 Done Done Sent false
30281 9117 Acked Write 1 Done Done Sent false
30282 9118 Acked Read 8 Done Done Sent false
30283 9119 Acked Read 2 Done Done Sent false
30284 9120 Acked Write 6 Done Done Sent false
30285 9121 Acked Write 2 Done Done Sent false
30286 9122 Acked Flush 0 Done Done Sent false
30287 9123 Acked Read 4 Done Done Sent false
30288 9124 Acked Write 8 Done Done Sent false
30289 9125 Acked Write 7 Done Done Sent false
30290 9126 Acked Write 5 Done Done Sent false
30291 9127 Acked Read 3 Done Done Sent false
30292 9128 Acked Write 7 Done Done Sent false
30293 9129 Acked Write 7 Done Done Sent false
30294 9130 Acked Read 5 Done Done Sent false
30295 9131 Acked Read 1 Done Done Sent false
30296 9132 Acked Read 4 Done Done Sent false
30297 9133 Acked Read 10 Done Done Sent false
30298 9134 Acked Write 5 Done Done Sent false
30299 9135 Acked Write 5 Done Done Sent false
30300 9136 Acked Write 4 Done Done Sent false
30301 9137 Acked Read 9 Done Done Sent false
30302 9138 Acked Read 4 Done Done Sent false
30303 9139 Acked Read 4 Done Done Sent false
30304 9140 Acked Read 3 Done Done Sent false
30305 9141 Acked Read 7 Done Done Sent false
30306 9142 Acked Write 3 Done Done Sent false
30307 9143 Acked Write 9 Done Done Sent false
30308 9144 Acked Write 6 Done Done Sent false
30309 9145 Acked Flush 0 Done Done Sent false
30310 9146 Acked Read 3 Done Done Sent false
30311 9147 Acked Flush 0 Done Done Sent false
30312 9148 Acked Write 1 Done Done Sent false
30313 9149 Acked Write 1 Done Done Sent false
30314 9150 Acked Read 8 Done Done Sent false
30315 9151 Acked Read 8 Done Done Sent false
30316 9152 Acked Flush 0 Done Done Sent false
30317 STATES DS:0 DS:1 DS:2 TOTAL
30318 Sent 0 0 302 302
30319 Done 302 302 0 604
30320 Skipped 0 0 0 0
30321 Error 0 0 0 0
30322 Last Flush: 9152 9152 8850
30323 Downstairs last five completed: 8850 8849 8848 8847 8846
30324 Upstairs last five completed: 9152 9151 9150 9149 9148
30325 CLIENT: Up:0 ds:302 act:2
303262024-12-09T15:14:11.569ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
303272024-12-09T15:14:11.569ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50265) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
303282024-12-09T15:14:11.570ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
303292024-12-09T15:14:11.571ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
303302024-12-09T15:14:11.571ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
303312024-12-09T15:14:11.571ZINFOcrucible: send last flush ID to this DS: 8850 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
303322024-12-09T15:14:11.571ZINFOcrucible: Replied this last flush ID: 8850 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
303332024-12-09T15:14:11.571ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
303342024-12-09T15:14:11.571ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
303352024-12-09T15:14:11.571ZINFOcrucible: [2] client re-new 302 jobs since flush 8850 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
303362024-12-09T15:14:11.572ZINFOcrucible: [2] Replayed 302 jobs since flush: 8850 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
30337 ----------------------------------------------------------------
30338 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30339 Downstairs last five completed: 9152 9151 9150 9149 9148
30340 Upstairs last five completed: 9152 9151 9150 9149 9148
30341 CLIENT: Up:0 ds:0 act:3
30342 Replay: All jobs finished
30343 Main task has work to do, go find it
30344 got dsc Stop(0)
30345 stop 0
30346 Dec 09 15:14:13.819 INFO request completed, latency_us[8810] Got stop action so:false kr:false
30347 : 422, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 931cb219-24b9-43be-9d9d-b59924bfa4ba, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
30348 Dec 09 15:14:13.820 INFO request completed, latency_us: 143, response_code: 200, uri: /state/cid/0, method: GET, req_id: f3afd8c2-f913-4fcf-a052-73b9a8feb51e, remote_addr: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:14:13.820087312Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"0","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
30349 127.0.0.1:47510, local_addr: 127.0.0.1:9998{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-12-09T15:14:13.820164069Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"0","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
30350 
303512024-12-09T15:14:13.820ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
303522024-12-09T15:14:13.820ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
303532024-12-09T15:14:13.820ZINFOcrucible: [0] Transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
303542024-12-09T15:14:13.820ZINFOcrucible: client task is exiting = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
30355 [8810] Exited with: ExitStatus(unix_wait_status(9))
30356 [8810][0] reports Exit
30357 Dec 09 15:14:17.822 INFO request completed, latency_us: 227, response_code: 200, uri: /state/cid/0, method: GET, req_id: 8df444b4-61fc-4814-aed1-cf919ff43845, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
30358 001/300 Read block 662 len 40960
30359 002/300 Write block 2814 len 32768 data: 5 4 4 6 6 6 5 5
30360 003/300 Write block 368 len 12288 data: 8 6 5
30361 004/300 Read block 1130 len 4096
30362 005/300 Read block 3181 len 36864
30363 006/300 Flush
30364 007/300 Read block 3086 len 24576
30365 008/300 Write block 242 len 32768 data: 3 3 3 5 6 6 8 8
30366 009/300 Read block 2425 len 20480
30367 010/300 Read block 2836 len 4096
30368 011/300 Read block 904 len 8192
30369 012/300 Write block 2348 len 20480 data: 7 8 7 7 5
30370 013/300 Write block 829 len 12288 data: 5 6 6
30371 014/300 Read block 102 len 12288
30372 015/300 Read block 4602 len 20480
30373 016/300 Write block 4167 len 8192 data: 3 4
30374 017/300 Write block 3747 len 8192 data: 6 6
30375 018/300 Write block 1513 len 36864 data: 9 9 7 9 7 8 6 4 4
30376 019/300 Read block 1821 len 36864
30377 020/300 Write block 953 len 24576 data: 8 8 6 4 6 4
30378 021/300 Read block 671 len 24576
30379 022/300 Read block 3757 len 16384
30380 023/300 Write block 1713 len 16384 data: 9 8 8 8
30381 024/300 Read block 3425 len 12288
30382 025/300 Flush
30383 026/300 Flush
30384 027/300 Read block 3379 len 36864
30385 028/300 Read block 4578 len 16384
30386 029/300 Write block 3112 len 8192 data: 2 2
30387 030/300 Read block 3348 len 28672
30388 031/300 Write block 4677 len 16384 data: 3 5 5 4
30389 032/300 Write block 2460 len 20480 data: 6 4 4 4 4
30390 033/300 Read block 2881 len 36864
30391 034/300 Read block 2367 len 32768
30392 035/300 Read block 1318 len 12288
30393 036/300 Read block 4427 len 16384
30394 037/300 Write block 3348 len 4096 data: 6
30395 038/300 Read block 1010 len 12288
30396 039/300 Write block 2186 len 8192 data: 5 8
30397 040/300 Read block 3331 len 28672
30398 041/300 Write block 1082 len 20480 data: 6 6 7 6 6
30399 042/300 Read block 411 len 36864
30400 043/300 Write block 212 len 24576 data: 6 11 10 9 9 9
30401 044/300 Write block 1690 len 40960 data: 3 5 5 7 6 6 6 5 4 5
30402 045/300 Write block 4138 len 32768 data: 5 6 6 5 5 5 5 4
30403 046/300 Read block 4001 len 40960
30404 047/300 Flush
30405 048/300 Read block 159 len 24576
30406 049/300 Flush
30407 050/300 Read block 380 len 8192
30408 051/300 Write block 3842 len 24576 data: 7 9 10 10 9 8
30409 052/300 Flush
30410 053/300 Read block 2146 len 12288
30411 054/300 Read block 4056 len 36864
30412 055/300 Read block 558 len 4096
30413 056/300 Read block 3048 len 24576
30414 057/300 Read block 2102 len 24576
30415 058/300 Write block 4299 len 28672 data: 8 8 6 7 7 8 8
30416 059/300 Read block 208 len 12288
30417 060/300 Flush
30418 061/300 Write block 3761 len 4096 data: 8
30419 062/300 Read block 364 len 40960
30420 063/300 Read block 2995 len 40960
30421 064/300 Write block 1904 len 4096 data: 3
30422 065/300 Read block 4745 len 32768
30423 066/300 Write block 1131 len 28672 data: 4 7 5 5 4 3 3
30424 067/300 Write block 19 len 16384 data: 5 5 4 7
30425 068/300 Write block 2770 len 8192 data: 7 9
30426 069/300 Write block 3839 len 36864 data: 5 5 6 8 10 11 11 10 9
30427 070/300 Flush
30428 071/300 Write block 2556 len 40960 data: 7 7 8 9 8 6 5 4 5 6
30429 072/300 Write block 1916 len 32768 data: 3 5 5 5 4 4 3 4
30430 073/300 Write block 897 len 24576 data: 2 4 5 8 9 8
30431 074/300 Read block 4586 len 40960
30432 075/300 Write block 476 len 20480 data: 6 7 8 7 8
30433 076/300 Read block 2404 len 32768
30434 077/300 Read block 3371 len 40960
30435 078/300 Flush
30436 079/300 Read block 950 len 12288
30437 080/300 Read block 2860 len 12288
30438 081/300 Read block 2225 len 16384
30439 082/300 Write block 3475 len 28672 data: 10 10 9 9 8 7 6
30440 083/300 Read block 4058 len 20480
30441 084/300 Read block 4361 len 16384
30442 085/300 Read block 2844 len 40960
30443 086/300 Flush
30444 087/300 Read block 4348 len 16384
30445 088/300 Write block 1485 len 36864 data: 5 6 5 5 5 4 4 5 4
30446 089/300 Write block 2773 len 36864 data: 10 12 11 11 11 11 10 10 6
30447 090/300 Write block 3333 len 28672 data: 3 3 3 4 5 4 6
30448 091/300 Flush
30449 092/300 Read block 4923 len 28672
30450 093/300 Read block 1631 len 32768
30451 094/300 Write block 3619 len 8192 data: 8 8
30452 095/300 Read block 730 len 4096
30453 096/300 Flush
30454 097/300 Write block 1880 len 24576 data: 2 3 3 3 5 6
30455 098/300 Read block 304 len 32768
30456 099/300 Flush
30457 100/300 Write block 2382 len 36864 data: 5 5 6 6 5 6 7 7 6
30458 101/300 Read block 911 len 36864
30459 102/300 Write block 2437 len 16384 data: 5 5 6 5
30460 103/300 Write block 1641 len 4096 data: 5
30461 104/300 Read block 770 len 8192
30462 105/300 Read block 2267 len 12288
30463 106/300 Write block 2131 len 16384 data: 12 11 9 10
30464 107/300 Read block 849 len 40960
30465 108/300 Read block 1804 len 32768
30466 109/300 Write block 2609 len 28672 data: 5 4 6 7 6 5 5
30467 110/300 Write block 4890 len 40960 data: 4 5 4 5 6 5 4 4 5 5
30468 111/300 Read block 3623 len 4096
30469 112/300 Read block 2486 len 40960
30470 113/300 Read block 4706 len 8192
30471 114/300 Read block 42 len 8192
30472 115/300 Flush
30473 116/300 Read block 63 len 20480
30474 117/300 Write block 111 len 32768 data: 4 5 7 7 6 6 5 4
30475 118/300 Write block 980 len 28672 data: 4 3 3 3 3 5 4
30476 119/300 Write block 1803 len 12288 data: 6 7 6
30477 120/300 Read block 4698 len 12288
30478 121/300 Write block 1523 len 40960 data: 3 5 5 6 5 8 7 6 5 5
30479 122/300 Read block 608 len 8192
30480 123/300 Write block 1721 len 8192 data: 3 5
30481 124/300 Read block 1282 len 28672
30482 125/300 Read block 2105 len 28672
30483 126/300 Flush
30484 127/300 Read block 4232 len 4096
30485 128/300 Read block 3653 len 40960
30486 129/300 Read block 1000 len 8192
30487 130/300 Write block 1293 len 20480 data: 9 7 7 6 6
30488 131/300 Write block 2841 len 20480 data: 3 3 3 4 4
30489 132/300 Write block 2687 len 28672 data: 8 6 4 4 4 5 7
30490 133/300 Read block 674 len 16384
30491 134/300 Read block 3183 len 16384
30492 135/300 Read block 621 len 16384
30493 136/300 Flush
30494 137/300 Read block 4037 len 4096
30495 138/300 Write block 109 len 40960 data: 5 4 5 6 8 8 7 7 6 5
30496 139/300 Write block 4739 len 12288 data: 4 4 4
30497 140/300 Read block 1233 len 20480
30498 141/300 Write block 11 len 36864 data: 4 4 4 3 3 3 3 5 6
30499 142/300 Read block 3517 len 20480
30500 143/300 Read block 1331 len 20480
30501 144/300 Write block 4819 len 40960 data: 6 4 3 3 3 5 7 5 5 6
30502 145/300 Write block 3164 len 12288 data: 6 5 5
30503 146/300 Write block 68 len 12288 data: 2 2 2
30504 147/300 Read block 4994 len 24576
30505 148/300 Write block 2383 len 40960 data: 6 7 7 6 7 8 8 7 6 7
30506 149/300 Write block 154 len 24576 data: 8 7 8 8 7 8
30507 150/300 Write block 978 len 28672 data: 4 4 5 4 4 4 4
30508 151/300 Read block 2709 len 8192
30509 152/300 Write block 937 len 28672 data: 4 4 5 6 7 6 3
30510 153/300 Write block 131 len 28672 data: 8 10 9 9 9 8 8
30511 154/300 Read block 483 len 12288
30512 155/300 Read block 4752 len 4096
30513 156/300 Write block 4897 len 32768 data: 5 6 6 4 5 5 5 6
30514 157/300 Read block 612 len 24576
30515 158/300 Read block 370 len 16384
30516 159/300 Write block 3566 len 8192 data: 7 7
30517 160/300 Write block 3222 len 36864 data: 6 7 6 5 6 7 6 6 8
30518 161/300 Read block 1193 len 28672
30519 162/300 Read block 2704 len 8192
30520 163/300 Write block 3440 len 36864 data: 5 5 5 5 6 6 6 7 6
30521 164/300 Read block 4265 len 40960
30522 165/300 Write block 3440 len 20480 data: 6 6 6 6 7
30523 166/300 Read block 638 len 16384
30524 167/300 Flush
30525 168/300 Write block 1104 len 40960 data: 7 8 10 9 9 9 8 9 8 7
30526 169/300 Write block 899 len 12288 data: 6 9 10
30527 170/300 Read block 4303 len 4096
30528 171/300 Read block 2800 len 32768
30529 172/300 Read block 1503 len 16384
30530 173/300 Flush
30531 174/300 Read block 3481 len 8192
30532 175/300 Read block 132 len 28672
30533 176/300 Flush
30534 177/300 Read block 1381 len 24576
30535 178/300 Flush
30536 179/300 Read block 790 len 8192
30537 180/300 Write block 3084 len 36864 data: 4 4 5 5 6 8 7 7 6
30538 181/300 Flush
30539 182/300 Write block 1788 len 4096 data: 5
30540 183/300 Write block 2354 len 20480 data: 6 6 6 6 6
30541 184/300 Read block 1060 len 8192
30542 185/300 Read block 2828 len 36864
30543 186/300 Write block 3398 len 40960 data: 9 8 7 8 8 5 3 3 3 4
30544 187/300 Read block 2777 len 32768
30545 188/300 Flush
30546 189/300 Read block 3214 len 16384
30547 190/300 Read block 3468 len 20480
30548 191/300 Write block 4261 len 36864 data: 9 8 7 6 5 5 6 4 3
30549 192/300 Read block 3860 len 28672
30550 193/300 Flush
30551 194/300 Write block 402 len 24576 data: 8 8 9 9 7 5
30552 195/300 Read block 1133 len 28672
30553 196/300 Write block 2197 len 20480 data: 6 5 7 8 9
30554 197/300 Read block 1557 len 16384
30555 198/300 Write block 4729 len 8192 data: 5 5
30556 199/300 Read block 236 len 12288
30557 200/300 Write block 302 len 8192 data: 5 5
30558 201/300 Read block 217 len 4096
30559 202/300 Read block 1791 len 20480
30560 203/300 Read block 3571 len 28672
30561 204/300 Read block 2064 len 40960
30562 205/300 Read block 3933 len 16384
30563 206/300 Read block 2470 len 28672
30564 207/300 Write block 3330 len 36864 data: 2 3 3 4 4 4 5 6 5
30565 208/300 Write block 1630 len 28672 data: 3 4 4 5 5 5 3
30566 209/300 Read block 3630 len 16384
30567 210/300 Write block 869 len 28672 data: 7 6 6 7 7 6 5
30568 211/300 Write block 153 len 20480 data: 8 9 8 9 9
30569 212/300 Write block 4830 len 28672 data: 6 4 6 6 6 7 7
30570 213/300 Write block 4750 len 40960 data: 6 6 6 6 6 5 5 3 3 5
30571 214/300 Write block 2289 len 12288 data: 4 5 4
30572 215/300 Read block 991 len 24576
30573 216/300 Write block 1822 len 16384 data: 7 6 5 5
30574 217/300 Write block 3214 len 16384 data: 7 7 6 6
30575 218/300 Flush
30576 219/300 Read block 2496 len 28672
30577 220/300 Write block 27 len 36864 data: 5 6 6 6 5 7 6 6 7
30578 221/300 Read block 4811 len 40960
30579 222/300 Read block 2749 len 32768
30580 223/300 Read block 3609 len 20480
30581 224/300 Write block 3067 len 8192 data: 5 4
30582 225/300 Read block 4247 len 24576
30583 226/300 Read block 1711 len 28672
30584 227/300 Write block 4250 len 12288 data: 4 4 5
30585 228/300 Read block 1882 len 4096
30586 229/300 Write block 4455 len 40960 data: 5 5 3 3 3 4 4 5 6 6
30587 230/300 Write block 1438 len 4096 data: 6
30588 231/300 Read block 1200 len 32768
30589 232/300 Read block 4182 len 24576
30590 233/300 Read block 1775 len 20480
30591 234/300 Flush
30592 235/300 Read block 4065 len 20480
30593 236/300 Write block 2411 len 28672 data: 6 6 5 3 4 4 4
30594 237/300 Read block 126 len 8192
30595 238/300 Read block 69 len 16384
30596 239/300 Write block 3510 len 12288 data: 3 3 3
30597 240/300 Write block 1031 len 40960 data: 6 6 5 5 6 5 8 6 8 7
30598 241/300 Read block 2287 len 20480
30599 242/300 Read block 439 len 20480
30600 243/300 Write block 4671 len 20480 data: 2 3 3 3 2
30601 244/300 Write block 919 len 40960 data: 5 5 7 7 7 7 9 10 7 6
30602 245/300 Read block 1248 len 32768
30603 246/300 Read block 817 len 40960
30604 247/300 Write block 3143 len 4096 data: 8
30605 248/300 Write block 3807 len 36864 data: 7 9 7 7 5 4 5 6 5
30606 249/300 Read block 1374 len 20480
30607 250/300 Read block 2066 len 36864
30608 251/300 Read block 2758 len 4096
30609 252/300 Write block 4809 len 16384 data: 5 5 7 6
30610 253/300 Write block 177 len 28672 data: 2 5 5 5 4 4 4
30611 254/300 Write block 3995 len 24576 data: 5 3 3 2 4 4
30612 255/300 Read block 3943 len 20480
30613 256/300 Write block 4854 len 28672 data: 2 2 3 3 3 3 4
30614 257/300 Write block 3829 len 32768 data: 9 8 8 8 8 6 7 6
30615 258/300 Read block 4547 len 32768
30616 259/300 Write block 3169 len 12288 data: 6 7 7
30617 260/300 Read block 2306 len 28672
30618 261/300 Write block 3928 len 12288 data: 6 8 9
30619 262/300 Read block 2588 len 20480
30620 263/300 Read block 1721 len 12288
30621 264/300 Write block 4205 len 32768 data: 4 4 4 2 2 2 2 3
30622 265/300 Write block 2146 len 20480 data: 5 6 5 5 4
30623 266/300 Read block 827 len 36864
30624 267/300 Write block 4332 len 20480 data: 4 5 5 4 5
30625 268/300 Read block 3965 len 16384
30626 269/300 Write block 4569 len 36864 data: 5 6 6 7 6 6 5 5 5
30627 270/300 Flush
30628 271/300 Read block 157 len 40960
30629 272/300 Read block 4769 len 20480
30630 273/300 Read block 981 len 16384
30631 274/300 Write block 640 len 16384 data: 7 6 6 6
30632 275/300 Write block 1189 len 4096 data: 5
30633 276/300 Flush
30634 277/300 Write block 4438 len 16384 data: 5 4 4 3
30635 278/300 Read block 3748 len 40960
30636 279/300 Write block 3954 len 36864 data: 2 4 3 3 3 3 3 4 4
30637 280/300 Flush
30638 281/300 Read block 2632 len 4096
30639 282/300 Read block 2795 len 16384
30640 283/300 Read block 222 len 24576
30641 284/300 Flush
30642 285/300 Write block 59 len 4096 data: 5
30643 286/300 Flush
30644 287/300 Read block 4207 len 40960
30645 288/300 Write block 76 len 32768 data: 4 4 4 5 5 5 6 6
30646 289/300 Read block 1614 len 40960
30647 290/300 Write block 1134 len 36864 data: 6 5 4 4 3 4 3 5 7
30648 291/300 Flush
30649 292/300 Read block 604 len 20480
30650 293/300 Flush
30651 294/300 Write block 942 len 16384 data: 7 4 3 2
30652 295/300 Read block 2788 len 20480
30653 296/300 Read block 2212 len 8192
30654 297/300 Write block 3753 len 32768 data: 7 8 10 10 9 9 9 8
30655 298/300 Read block 4264 len 28672
30656 299/300 Read block 1660 len 4096
30657 300/300 Write block 1404 len 16384 data: 7 7 6 7
30658 301/300 Write block 788 len 24576 data: 4 4 4 3 3 4
30659 Main task has work to do, go find it
30660 got dsc Start(0)
30661 start 0
30662 Dec 09 15:14:18.059 INFO request completed, latency_us: 239, response_code: 204, uri: /start/cid/0, method: POST, req_id: 48655126-b0df-4880-b690-c590008856a1, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
30663 Replay: started 0, returned:Ok(())
30664 ----------------------------------------------------------------
30665 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30666 States: Offline Active Active
30667 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30668 9153 Acked Read 10 Sent Done Done false
30669 9154 Acked Write 8 Sent Done Done false
30670 9155 Acked Write 3 Sent Done Done false
30671 9156 Acked Read 1 Sent Done Done false
30672 9157 Acked Read 9 Sent Done Done false
30673 9158 Acked Flush 0 Sent Done Done false
30674 9159 Acked Read 6 Sent Done Done false
30675 9160 Acked Write 8 Sent Done Done false
30676 9161 Acked Read 5 Sent Done Done false
30677 9162 Acked Read 1 Sent Done Done false
30678 9163 Acked Read 2 Sent Done Done false
30679 9164 Acked Write 5 Sent Done Done false
30680 9165 Acked Write 3 Sent Done Done false
30681 9166 Acked Read 3 Sent Done Done false
30682 9167 Acked Read 5 Sent Done Done false
30683 9168 Acked Write 2 Sent Done Done false
30684 9169 Acked Write 2 Sent Done Done false
30685 9170 Acked Write 9 Sent Done Done false
30686 9171 Acked Read 9 Sent Done Done false
30687 9172 Acked Write 6 Sent Done Done false
30688 9173 Acked Read 6 Sent Done Done false
30689 9174 Acked Read 4 Sent Done Done false
30690 9175 Acked Write 4 Sent Done Done false
30691 9176 Acked Read 3 Sent Done Done false
30692 9177 Acked Flush 0 Sent Done Done false
30693 9178 Acked Flush 0 Sent Done Done false
30694 9179 Acked Read 9 Sent Done Done false
30695 9180 Acked Read 4 Sent Done Done false
30696 9181 Acked Write 2 Sent Done Done false
30697 9182 Acked Read 7 Sent Done Done false
30698 9183 Acked Write 4 Sent Done Done false
30699 9184 Acked Write 5 Sent Done Done false
30700 9185 Acked Read 9 Sent Done Done false
30701 9186 Acked Read 8 Sent Done Done false
30702 9187 Acked Read 3 Sent Done Done false
30703 9188 Acked Read 4 Sent Done Done false
30704 9189 Acked Write 1 Sent Done Done false
30705 9190 Acked Read 3 Sent Done Done false
30706 9191 Acked Write 2 Sent Done Done false
30707 9192 Acked Read 7 Sent Done Done false
30708 9193 Acked Write 5 Sent Done Done false
30709 9194 Acked Read 9 Sent Done Done false
30710 9195 Acked Write 6 Sent Done Done false
30711 9196 Acked Write 10 Sent Done Done false
30712 9197 Acked Write 8 Sent Done Done false
30713 9198 Acked Read 10 Sent Done Done false
30714 9199 Acked Flush 0 Sent Done Done false
30715 9200 Acked Read 6 Sent Done Done false
30716 9201 Acked Flush 0 Sent Done Done false
30717 9202 Acked Read 2 Sent Done Done false
30718 9203 Acked Write 6 Sent Done Done false
30719 9204 Acked Flush 0 Sent Done Done false
30720 9205 Acked Read 3 Sent Done Done false
30721 9206 Acked Read 9 Sent Done Done false
30722 9207 Acked Read 1 Sent Done Done false
30723 9208 Acked Read 6 Sent Done Done false
30724 9209 Acked Read 6 Sent Done Done false
30725 9210 Acked Write 7 Sent Done Done false
30726 9211 Acked Read 3 Sent Done Done false
30727 9212 Acked Flush 0 Sent Done Done false
30728 9213 Acked Write 1 Sent Done Done false
30729 9214 Acked Read 10 Sent Done Done false
30730 9215 Acked Read 10 Sent Done Done false
30731 9216 Acked Write 1 Sent Done Done false
30732 9217 Acked Read 8 Sent Done Done false
30733 9218 Acked Write 7 Sent Done Done false
30734 9219 Acked Write 4 Sent Done Done false
30735 9220 Acked Write 2 Sent Done Done false
30736 9221 Acked Write 9 Sent Done Done false
30737 9222 Acked Flush 0 Sent Done Done false
30738 9223 Acked Write 10 Sent Done Done false
30739 9224 Acked Write 8 Sent Done Done false
30740 9225 Acked Write 6 Sent Done Done false
30741 9226 Acked Read 10 Sent Done Done false
30742 9227 Acked Write 5 Sent Done Done false
30743 9228 Acked Read 8 Sent Done Done false
30744 9229 Acked Read 10 Sent Done Done false
30745 9230 Acked Flush 0 Sent Done Done false
30746 9231 Acked Read 3 Sent Done Done false
30747 9232 Acked Read 3 Sent Done Done false
30748 9233 Acked Read 4 Sent Done Done false
30749 9234 Acked Write 7 Sent Done Done false
30750 9235 Acked Read 5 Sent Done Done false
30751 9236 Acked Read 4 Sent Done Done false
30752 9237 Acked Read 10 Sent Done Done false
30753 9238 Acked Flush 0 Sent Done Done false
30754 9239 Acked Read 4 Sent Done Done false
30755 9240 Acked Write 9 Sent Done Done false
30756 9241 Acked Write 9 Sent Done Done false
30757 9242 Acked Write 7 Sent Done Done false
30758 9243 Acked Flush 0 Sent Done Done false
30759 9244 Acked Read 7 Sent Done Done false
30760 9245 Acked Read 8 Sent Done Done false
30761 9246 Acked Write 2 Sent Done Done false
30762 9247 Acked Read 1 Sent Done Done false
30763 9248 Acked Flush 0 Sent Done Done false
30764 9249 Acked Write 6 Sent Done Done false
30765 9250 Acked Read 8 Sent Done Done false
30766 9251 Acked Flush 0 Sent Done Done false
30767 9252 Acked Write 9 Sent Done Done false
30768 9253 Acked Read 9 Sent Done Done false
30769 9254 Acked Write 4 Sent Done Done false
30770 9255 Acked Write 1 Sent Done Done false
30771 9256 Acked Read 2 Sent Done Done false
30772 9257 Acked Read 3 Sent Done Done false
30773 9258 Acked Write 4 Sent Done Done false
30774 9259 Acked Read 10 Sent Done Done false
30775 9260 Acked Read 8 Sent Done Done false
30776 9261 Acked Write 7 Sent Done Done false
30777 9262 Acked Write 10 Sent Done Done false
30778 9263 Acked Read 1 Sent Done Done false
30779 9264 Acked Read 10 Sent Done Done false
30780 9265 Acked Read 2 Sent Done Done false
30781 9266 Acked Read 2 Sent Done Done false
30782 9267 Acked Flush 0 Sent Done Done false
30783 9268 Acked Read 5 Sent Done Done false
30784 9269 Acked Write 8 Sent Done Done false
30785 9270 Acked Write 7 Sent Done Done false
30786 9271 Acked Write 3 Sent Done Done false
30787 9272 Acked Read 3 Sent Done Done false
30788 9273 Acked Write 10 Sent Done Done false
30789 9274 Acked Read 2 Sent Done Done false
30790 9275 Acked Write 2 Sent Done Done false
30791 9276 Acked Read 7 Sent Done Done false
30792 9277 Acked Read 7 Sent Done Done false
30793 9278 Acked Flush 0 Sent Done Done false
30794 9279 Acked Read 1 Sent Done Done false
30795 9280 Acked Read 10 Sent Done Done false
30796 9281 Acked Read 2 Sent Done Done false
30797 9282 Acked Write 5 Sent Done Done false
30798 9283 Acked Write 5 Sent Done Done false
30799 9284 Acked Write 7 Sent Done Done false
30800 9285 Acked Read 4 Sent Done Done false
30801 9286 Acked Read 4 Sent Done Done false
30802 9287 Acked Read 4 Sent Done Done false
30803 9288 Acked Flush 0 Sent Done Done false
30804 9289 Acked Read 1 Sent Done Done false
30805 9290 Acked Write 10 Sent Done Done false
30806 9291 Acked Write 3 Sent Done Done false
30807 9292 Acked Read 5 Sent Done Done false
30808 9293 Acked Write 9 Sent Done Done false
30809 9294 Acked Read 5 Sent Done Done false
30810 9295 Acked Read 5 Sent Done Done false
30811 9296 Acked Write 10 Sent Done Done false
30812 9297 Acked Write 3 Sent Done Done false
30813 9298 Acked Write 3 Sent Done Done false
30814 9299 Acked Read 6 Sent Done Done false
30815 9300 Acked Write 10 Sent Done Done false
30816 9301 Acked Write 6 Sent Done Done false
30817 9302 Acked Write 7 Sent Done Done false
30818 9303 Acked Read 2 Sent Done Done false
30819 9304 Acked Write 7 Sent Done Done false
30820 9305 Acked Write 7 Sent Done Done false
30821 9306 Acked Read 3 Sent Done Done false
30822 9307 Acked Read 1 Sent Done Done false
30823 9308 Acked Write 8 Sent Done Done false
30824 9309 Acked Read 6 Sent Done Done false
30825 9310 Acked Read 4 Sent Done Done false
30826 9311 Acked Write 2 Sent Done Done false
30827 9312 Acked Write 9 Sent Done Done false
30828 9313 Acked Read 7 Sent Done Done false
30829 9314 Acked Read 2 Sent Done Done false
30830 9315 Acked Write 9 Sent Done Done false
30831 9316 Acked Read 10 Sent Done Done false
30832 9317 Acked Write 5 Sent Done Done false
30833 9318 Acked Read 4 Sent Done Done false
30834 9319 Acked Flush 0 Sent Done Done false
30835 9320 Acked Write 10 Sent Done Done false
30836 9321 Acked Write 3 Sent Done Done false
30837 9322 Acked Read 1 Sent Done Done false
30838 9323 Acked Read 8 Sent Done Done false
30839 9324 Acked Read 4 Sent Done Done false
30840 9325 Acked Flush 0 Sent Done Done false
30841 9326 Acked Read 2 Sent Done Done false
30842 9327 Acked Read 7 Sent Done Done false
30843 9328 Acked Flush 0 Sent Done Done false
30844 9329 Acked Read 6 Sent Done Done false
30845 9330 Acked Flush 0 Sent Done Done false
30846 9331 Acked Read 2 Sent Done Done false
30847 9332 Acked Write 9 Sent Done Done false
30848 9333 Acked Flush 0 Sent Done Done false
30849 9334 Acked Write 1 Sent Done Done false
30850 9335 Acked Write 5 Sent Done Done false
30851 9336 Acked Read 2 Sent Done Done false
30852 9337 Acked Read 9 Sent Done Done false
30853 9338 Acked Write 10 Sent Done Done false
30854 9339 Acked Read 8 Sent Done Done false
30855 9340 Acked Flush 0 Sent Done Done false
30856 9341 Acked Read 4 Sent Done Done false
30857 9342 Acked Read 5 Sent Done Done false
30858 9343 Acked Write 9 Sent Done Done false
30859 9344 Acked Read 7 Sent Done Done false
30860 9345 Acked Flush 0 Sent Done Done false
30861 9346 Acked Write 6 Sent Done Done false
30862 9347 Acked Read 7 Sent Done Done false
30863 9348 Acked Write 5 Sent Done Done false
30864 9349 Acked Read 4 Sent Done Done false
30865 9350 Acked Write 2 Sent Done Done false
30866 9351 Acked Read 3 Sent Done Done false
30867 9352 Acked Write 2 Sent Done Done false
30868 9353 Acked Read 1 Sent Done Done false
30869 9354 Acked Read 5 Sent Done Done false
30870 9355 Acked Read 7 Sent Done Done false
30871 9356 Acked Read 10 Sent Done Done false
30872 9357 Acked Read 4 Sent Done Done false
30873 9358 Acked Read 7 Sent Done Done false
30874 9359 Acked Write 9 Sent Done Done false
30875 9360 Acked Write 7 Sent Done Done false
30876 9361 Acked Read 4 Sent Done Done false
30877 9362 Acked Write 7 Sent Done Done false
30878 9363 Acked Write 5 Sent Done Done false
30879 9364 Acked Write 7 Sent Done Done false
30880 9365 Acked Write 10 Sent Done Done false
30881 9366 Acked Write 3 Sent Done Done false
30882 9367 Acked Read 6 Sent Done Done false
30883 9368 Acked Write 4 Sent Done Done false
30884 9369 Acked Write 4 Sent Done Done false
30885 9370 Acked Flush 0 Sent Done Done false
30886 9371 Acked Read 7 Sent Done Done false
30887 9372 Acked Write 9 Sent Done Done false
30888 9373 Acked Read 10 Sent Done Done false
30889 9374 Acked Read 8 Sent Done Done false
30890 9375 Acked Read 5 Sent Done Done false
30891 9376 Acked Write 2 Sent Done Done false
30892 9377 Acked Read 6 Sent Done Done false
30893 9378 Acked Read 7 Sent Done Done false
30894 9379 Acked Write 3 Sent Done Done false
30895 9380 Acked Read 1 Sent Done Done false
30896 9381 Acked Write 10 Sent Done Done false
30897 9382 Acked Write 1 Sent Done Done false
30898 9383 Acked Read 8 Sent Done Done false
30899 9384 Acked Read 6 Sent Done Done false
30900 9385 Acked Read 5 Sent Done Done false
30901 9386 Acked Flush 0 Sent Done Done false
30902 9387 Acked Read 5 Sent Done Done false
30903 9388 Acked Write 7 Sent Done Done false
30904 9389 Acked Read 2 Sent Done Done false
30905 9390 Acked Read 4 Sent Done Done false
30906 9391 Acked Write 3 Sent Done Done false
30907 9392 Acked Write 10 Sent Done Done false
30908 9393 Acked Read 5 Sent Done Done false
30909 9394 Acked Read 5 Sent Done Done false
30910 9395 Acked Write 5 Sent Done Done false
30911 9396 Acked Write 10 Sent Done Done false
30912 9397 Acked Read 8 Sent Done Done false
30913 9398 Acked Read 10 Sent Done Done false
30914 9399 Acked Write 1 Sent Done Done false
30915 9400 Acked Write 9 Sent Done Done false
30916 9401 Acked Read 5 Sent Done Done false
30917 9402 Acked Read 9 Sent Done Done false
30918 9403 Acked Read 1 Sent Done Done false
30919 9404 Acked Write 4 Sent Done Done false
30920 9405 Acked Write 7 Sent Done Done false
30921 9406 Acked Write 6 Sent Done Done false
30922 9407 Acked Read 5 Sent Done Done false
30923 9408 Acked Write 7 Sent Done Done false
30924 9409 Acked Write 8 Sent Done Done false
30925 9410 Acked Read 8 Sent Done Done false
30926 9411 Acked Write 3 Sent Done Done false
30927 9412 Acked Read 7 Sent Done Done false
30928 9413 Acked Write 3 Sent Done Done false
30929 9414 Acked Read 5 Sent Done Done false
30930 9415 Acked Read 3 Sent Done Done false
30931 9416 Acked Write 8 Sent Done Done false
30932 9417 Acked Write 5 Sent Done Done false
30933 9418 Acked Read 9 Sent Done Done false
30934 9419 Acked Write 5 Sent Done Done false
30935 9420 Acked Read 4 Sent Done Done false
30936 9421 Acked Write 9 Sent Done Done false
30937 9422 Acked Flush 0 Sent Done Done false
30938 9423 Acked Read 10 Sent Done Done false
30939 9424 Acked Read 5 Sent Done Done false
30940 9425 Acked Read 4 Sent Done Done false
30941 9426 Acked Write 4 Sent Done Done false
30942 9427 Acked Write 1 Sent Done Done false
30943 9428 Acked Flush 0 Sent Done Done false
30944 9429 Acked Write 4 Sent Done Done false
30945 9430 Acked Read 10 Sent Done Done false
30946 9431 Acked Write 9 Sent Done Done false
30947 9432 Acked Flush 0 Sent Done Done false
30948 9433 Acked Read 1 Sent Done Done false
30949 9434 Acked Read 4 Sent Done Done false
30950 9435 Acked Read 6 Sent Done Done false
30951 9436 Acked Flush 0 Sent Done Done false
30952 9437 Acked Write 1 Sent Done Done false
30953 9438 Acked Flush 0 Sent Done Done false
30954 9439 Acked Read 10 Sent Done Done false
30955 9440 Acked Write 8 Sent Done Done false
30956 9441 Acked Read 10 Sent Done Done false
30957 9442 Acked Write 9 Sent Done Done false
30958 9443 Acked Flush 0 Sent Done Done false
30959 9444 Acked Read 5 Sent Done Done false
30960 9445 Acked Flush 0 Sent Done Done false
30961 9446 Acked Write 4 Sent Done Done false
30962 9447 Acked Read 5 Sent Done Done false
30963 9448 Acked Read 2 Sent Done Done false
30964 9449 Acked Write 8 Sent Done Done false
30965 9450 Acked Read 7 Sent Done Done false
30966 9451 Acked Read 1 Sent Done Done false
30967 9452 Acked Write 4 Sent Done Done false
30968 9453 Acked Write 6 Sent Done Done false
30969 STATES DS:0 DS:1 DS:2 TOTAL
30970 Sent 301 0 0 301
30971 Done 0 301 301 602
30972 Skipped 0 0 0 0
30973 Error 0 0 0 0
30974 Last Flush: 9152 9445 9445
30975 Downstairs last five completed: 9152 9151 9150 9149 9148
30976 Upstairs last five completed: 9453 9452 9451 9450 9449
30977 CLIENT: Up:0 ds:301 act:2
30978 ----------------------------------------------------------------
30979 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
30980 States: Offline Active Active
30981 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30982 9153 Acked Read 10 Sent Done Done false
30983 9154 Acked Write 8 Sent Done Done false
30984 9155 Acked Write 3 Sent Done Done false
30985 9156 Acked Read 1 Sent Done Done false
30986 9157 Acked Read 9 Sent Done Done false
30987 9158 Acked Flush 0 Sent Done Done false
30988 9159 Acked Read 6 Sent Done Done false
30989 9160 Acked Write 8 Sent Done Done false
30990 9161 Acked Read 5 Sent Done Done false
30991 9162 Acked Read 1 Sent Done Done false
30992 9163 Acked Read 2 Sent Done Done false
30993 9164 Acked Write 5 Sent Done Done false
30994 9165 Acked Write 3 Sent Done Done false
30995 9166 Acked Read 3 Sent Done Done false
30996 9167 Acked Read 5 Sent Done Done false
30997 9168 Acked Write 2 Sent Done Done false
30998 9169 Acked Write 2 Sent Done Done false
30999 9170 Acked Write 9 Sent Done Done false
31000 9171 Acked Read 9 Sent Done Done false
31001 9172 Acked Write 6 Sent Done Done false
31002 9173 Acked Read 6 Sent Done Done false
31003 9174 Acked Read 4 Sent Done Done false
31004 9175 Acked Write 4 Sent Done Done false
31005 9176 Acked Read 3 Sent Done Done false
31006 9177 Acked Flush 0 Sent Done Done false
31007 9178 Acked Flush 0 Sent Done Done false
31008 9179 Acked Read 9 Sent Done Done false
31009 9180 Acked Read 4 Sent Done Done false
31010 9181 Acked Write 2 Sent Done Done false
31011 9182 Acked Read 7 Sent Done Done false
31012 9183 Acked Write 4 Sent Done Done false
31013 9184 Acked Write 5 Sent Done Done false
31014 9185 Acked Read 9 Sent Done Done false
31015 9186 Acked Read 8 Sent Done Done false
31016 9187 Acked Read 3 Sent Done Done false
31017 9188 Acked Read 4 Sent Done Done false
31018 9189 Acked Write 1 Sent Done Done false
31019 9190 Acked Read 3 Sent Done Done false
31020 9191 Acked Write 2 Sent Done Done false
31021 9192 Acked Read 7 Sent Done Done false
31022 9193 Acked Write 5 Sent Done Done false
31023 9194 Acked Read 9 Sent Done Done false
31024 9195 Acked Write 6 Sent Done Done false
31025 9196 Acked Write 10 Sent Done Done false
31026 9197 Acked Write 8 Sent Done Done false
31027 9198 Acked Read 10 Sent Done Done false
31028 9199 Acked Flush 0 Sent Done Done false
31029 9200 Acked Read 6 Sent Done Done false
31030 9201 Acked Flush 0 Sent Done Done false
31031 9202 Acked Read 2 Sent Done Done false
31032 9203 Acked Write 6 Sent Done Done false
31033 9204 Acked Flush 0 Sent Done Done false
31034 9205 Acked Read 3 Sent Done Done false
31035 9206 Acked Read 9 Sent Done Done false
31036 9207 Acked Read 1 Sent Done Done false
31037 9208 Acked Read 6 Sent Done Done false
31038 9209 Acked Read 6 Sent Done Done false
31039 9210 Acked Write 7 Sent Done Done false
31040 9211 Acked Read 3 Sent Done Done false
31041 9212 Acked Flush 0 Sent Done Done false
31042 9213 Acked Write 1 Sent Done Done false
31043 9214 Acked Read 10 Sent Done Done false
31044 9215 Acked Read 10 Sent Done Done false
31045 9216 Acked Write 1 Sent Done Done false
31046 9217 Acked Read 8 Sent Done Done false
31047 9218 Acked Write 7 Sent Done Done false
31048 9219 Acked Write 4 Sent Done Done false
31049 9220 Acked Write 2 Sent Done Done false
31050 9221 Acked Write 9 Sent Done Done false
31051 9222 Acked Flush 0 Sent Done Done false
31052 9223 Acked Write 10 Sent Done Done false
31053 9224 Acked Write 8 Sent Done Done false
31054 9225 Acked Write 6 Sent Done Done false
31055 9226 Acked Read 10 Sent Done Done false
31056 9227 Acked Write 5 Sent Done Done false
31057 9228 Acked Read 8 Sent Done Done false
31058 9229 Acked Read 10 Sent Done Done false
31059 9230 Acked Flush 0 Sent Done Done false
31060 9231 Acked Read 3 Sent Done Done false
31061 9232 Acked Read 3 Sent Done Done false
31062 9233 Acked Read 4 Sent Done Done false
31063 9234 Acked Write 7 Sent Done Done false
31064 9235 Acked Read 5 Sent Done Done false
31065 9236 Acked Read 4 Sent Done Done false
31066 9237 Acked Read 10 Sent Done Done false
31067 9238 Acked Flush 0 Sent Done Done false
31068 9239 Acked Read 4 Sent Done Done false
31069 9240 Acked Write 9 Sent Done Done false
31070 9241 Acked Write 9 Sent Done Done false
31071 9242 Acked Write 7 Sent Done Done false
31072 9243 Acked Flush 0 Sent Done Done false
31073 9244 Acked Read 7 Sent Done Done false
31074 9245 Acked Read 8 Sent Done Done false
31075 9246 Acked Write 2 Sent Done Done false
31076 9247 Acked Read 1 Sent Done Done false
31077 9248 Acked Flush 0 Sent Done Done false
31078 9249 Acked Write 6 Sent Done Done false
31079 9250 Acked Read 8 Sent Done Done false
31080 9251 Acked Flush 0 Sent Done Done false
31081 9252 Acked Write 9 Sent Done Done false
31082 9253 Acked Read 9 Sent Done Done false
31083 9254 Acked Write 4 Sent Done Done false
31084 9255 Acked Write 1 Sent Done Done false
31085 9256 Acked Read 2 Sent Done Done false
31086 9257 Acked Read 3 Sent Done Done false
31087 9258 Acked Write 4 Sent Done Done false
31088 9259 Acked Read 10 Sent Done Done false
31089 9260 Acked Read 8 Sent Done Done false
31090 9261 Acked Write 7 Sent Done Done false
31091 9262 Acked Write 10 Sent Done Done false
31092 9263 Acked Read 1 Sent Done Done false
31093 9264 Acked Read 10 Sent Done Done false
31094 9265 Acked Read 2 Sent Done Done false
31095 9266 Acked Read 2 Sent Done Done false
31096 9267 Acked Flush 0 Sent Done Done false
31097 9268 Acked Read 5 Sent Done Done false
31098 9269 Acked Write 8 Sent Done Done false
31099 9270 Acked Write 7 Sent Done Done false
31100 9271 Acked Write 3 Sent Done Done false
31101 9272 Acked Read 3 Sent Done Done false
31102 9273 Acked Write 10 Sent Done Done false
31103 9274 Acked Read 2 Sent Done Done false
31104 9275 Acked Write 2 Sent Done Done false
31105 9276 Acked Read 7 Sent Done Done false
31106 9277 Acked Read 7 Sent Done Done false
31107 9278 Acked Flush 0 Sent Done Done false
31108 9279 Acked Read 1 Sent Done Done false
31109 9280 Acked Read 10 Sent Done Done false
31110 9281 Acked Read 2 Sent Done Done false
31111 9282 Acked Write 5 Sent Done Done false
31112 9283 Acked Write 5 Sent Done Done false
31113 9284 Acked Write 7 Sent Done Done false
31114 9285 Acked Read 4 Sent Done Done false
31115 9286 Acked Read 4 Sent Done Done false
31116 9287 Acked Read 4 Sent Done Done false
31117 9288 Acked Flush 0 Sent Done Done false
31118 9289 Acked Read 1 Sent Done Done false
31119 9290 Acked Write 10 Sent Done Done false
31120 9291 Acked Write 3 Sent Done Done false
31121 9292 Acked Read 5 Sent Done Done false
31122 9293 Acked Write 9 Sent Done Done false
31123 9294 Acked Read 5 Sent Done Done false
31124 9295 Acked Read 5 Sent Done Done false
31125 9296 Acked Write 10 Sent Done Done false
31126 9297 Acked Write 3 Sent Done Done false
31127 9298 Acked Write 3 Sent Done Done false
31128 9299 Acked Read 6 Sent Done Done false
31129 9300 Acked Write 10 Sent Done Done false
31130 9301 Acked Write 6 Sent Done Done false
31131 9302 Acked Write 7 Sent Done Done false
31132 9303 Acked Read 2 Sent Done Done false
31133 9304 Acked Write 7 Sent Done Done false
31134 9305 Acked Write 7 Sent Done Done false
31135 9306 Acked Read 3 Sent Done Done false
31136 9307 Acked Read 1 Sent Done Done false
31137 9308 Acked Write 8 Sent Done Done false
31138 9309 Acked Read 6 Sent Done Done false
31139 9310 Acked Read 4 Sent Done Done false
31140 9311 Acked Write 2 Sent Done Done false
31141 9312 Acked Write 9 Sent Done Done false
31142 9313 Acked Read 7 Sent Done Done false
31143 9314 Acked Read 2 Sent Done Done false
31144 9315 Acked Write 9 Sent Done Done false
31145 9316 Acked Read 10 Sent Done Done false
31146 9317 Acked Write 5 Sent Done Done false
31147 9318 Acked Read 4 Sent Done Done false
31148 9319 Acked Flush 0 Sent Done Done false
31149 9320 Acked Write 10 Sent Done Done false
31150 9321 Acked Write 3 Sent Done Done false
31151 9322 Acked Read 1 Sent Done Done false
31152 9323 Acked Read 8 Sent Done Done false
31153 9324 Acked Read 4 Sent Done Done false
31154 9325 Acked Flush 0 Sent Done Done false
31155 9326 Acked Read 2 Sent Done Done false
31156 9327 Acked Read 7 Sent Done Done false
31157 9328 Acked Flush 0 Sent Done Done false
31158 9329 Acked Read 6 Sent Done Done false
31159 9330 Acked Flush 0 Sent Done Done false
31160 9331 Acked Read 2 Sent Done Done false
31161 9332 Acked Write 9 Sent Done Done false
31162 9333 Acked Flush 0 Sent Done Done false
31163 9334 Acked Write 1 Sent Done Done false
31164 9335 Acked Write 5 Sent Done Done false
31165 9336 Acked Read 2 Sent Done Done false
31166 9337 Acked Read 9 Sent Done Done false
31167 9338 Acked Write 10 Sent Done Done false
31168 9339 Acked Read 8 Sent Done Done false
31169 9340 Acked Flush 0 Sent Done Done false
31170 9341 Acked Read 4 Sent Done Done false
31171 9342 Acked Read 5 Sent Done Done false
31172 9343 Acked Write 9 Sent Done Done false
31173 9344 Acked Read 7 Sent Done Done false
31174 9345 Acked Flush 0 Sent Done Done false
31175 9346 Acked Write 6 Sent Done Done false
31176 9347 Acked Read 7 Sent Done Done false
31177 9348 Acked Write 5 Sent Done Done false
31178 9349 Acked Read 4 Sent Done Done false
31179 9350 Acked Write 2 Sent Done Done false
31180 9351 Acked Read 3 Sent Done Done false
31181 9352 Acked Write 2 Sent Done Done false
31182 9353 Acked Read 1 Sent Done Done false
31183 9354 Acked Read 5 Sent Done Done false
31184 9355 Acked Read 7 Sent Done Done false
31185 9356 Acked Read 10 Sent Done Done false
31186 9357 Acked Read 4 Sent Done Done false
31187 9358 Acked Read 7 Sent Done Done false
31188 9359 Acked Write 9 Sent Done Done false
31189 9360 Acked Write 7 Sent Done Done false
31190 9361 Acked Read 4 Sent Done Done false
31191 9362 Acked Write 7 Sent Done Done false
31192 9363 Acked Write 5 Sent Done Done false
31193 9364 Acked Write 7 Sent Done Done false
31194 9365 Acked Write 10 Sent Done Done false
31195 9366 Acked Write 3 Sent Done Done false
31196 9367 Acked Read 6 Sent Done Done false
31197 9368 Acked Write 4 Sent Done Done false
31198 9369 Acked Write 4 Sent Done Done false
31199 9370 Acked Flush 0 Sent Done Done false
31200 9371 Acked Read 7 Sent Done Done false
31201 9372 Acked Write 9 Sent Done Done false
31202 9373 Acked Read 10 Sent Done Done false
31203 9374 Acked Read 8 Sent Done Done false
31204 9375 Acked Read 5 Sent Done Done false
31205 9376 Acked Write 2 Sent Done Done false
31206 9377 Acked Read 6 Sent Done Done false
31207 9378 Acked Read 7 Sent Done Done false
31208 9379 Acked Write 3 Sent Done Done false
31209 9380 Acked Read 1 Sent Done Done false
31210 9381 Acked Write 10 Sent Done Done false
31211 9382 Acked Write 1 Sent Done Done false
31212 9383 Acked Read 8 Sent Done Done false
31213 9384 Acked Read 6 Sent Done Done false
31214 9385 Acked Read 5 Sent Done Done false
31215 9386 Acked Flush 0 Sent Done Done false
31216 9387 Acked Read 5 Sent Done Done false
31217 9388 Acked Write 7 Sent Done Done false
31218 9389 Acked Read 2 Sent Done Done false
31219 9390 Acked Read 4 Sent Done Done false
31220 9391 Acked Write 3 Sent Done Done false
31221 9392 Acked Write 10 Sent Done Done false
31222 9393 Acked Read 5 Sent Done Done false
31223 9394 Acked Read 5 Sent Done Done false
31224 9395 Acked Write 5 Sent Done Done false
31225 9396 Acked Write 10 Sent Done Done false
31226 9397 Acked Read 8 Sent Done Done false
31227 9398 Acked Read 10 Sent Done Done false
31228 9399 Acked Write 1 Sent Done Done false
31229 9400 Acked Write 9 Sent Done Done false
31230 9401 Acked Read 5 Sent Done Done false
31231 9402 Acked Read 9 Sent Done Done false
31232 9403 Acked Read 1 Sent Done Done false
31233 9404 Acked Write 4 Sent Done Done false
31234 9405 Acked Write 7 Sent Done Done false
31235 9406 Acked Write 6 Sent Done Done false
31236 9407 Acked Read 5 Sent Done Done false
31237 9408 Acked Write 7 Sent Done Done false
31238 9409 Acked Write 8 Sent Done Done false
31239 9410 Acked Read 8 Sent Done Done false
31240 9411 Acked Write 3 Sent Done Done false
31241 9412 Acked Read 7 Sent Done Done false
31242 9413 Acked Write 3 Sent Done Done false
31243 9414 Acked Read 5 Sent Done Done false
31244 9415 Acked Read 3 Sent Done Done false
31245 9416 Acked Write 8 Sent Done Done false
31246 9417 Acked Write 5 Sent Done Done false
31247 9418 Acked Read 9 Sent Done Done false
31248 9419 Acked Write 5 Sent Done Done false
31249 9420 Acked Read 4 Sent Done Done false
31250 9421 Acked Write 9 Sent Done Done false
31251 9422 Acked Flush 0 Sent Done Done false
31252 9423 Acked Read 10 Sent Done Done false
31253 9424 Acked Read 5 Sent Done Done false
31254 9425 Acked Read 4 Sent Done Done false
31255 9426 Acked Write 4 Sent Done Done false
31256 9427 Acked Write 1 Sent Done Done false
31257 9428 Acked Flush 0 Sent Done Done false
31258 9429 Acked Write 4 Sent Done Done false
31259 9430 Acked Read 10 Sent Done Done false
31260 9431 Acked Write 9 Sent Done Done false
31261 9432 Acked Flush 0 Sent Done Done false
31262 9433 Acked Read 1 Sent Done Done false
31263 9434 Acked Read 4 Sent Done Done false
31264 9435 Acked Read 6 Sent Done Done false
31265 9436 Acked Flush 0 Sent Done Done false
31266 9437 Acked Write 1 Sent Done Done false
31267 9438 Acked Flush 0 Sent Done Done false
31268 9439 Acked Read 10 Sent Done Done false
31269 9440 Acked Write 8 Sent Done Done false
31270 9441 Acked Read 10 Sent Done Done false
31271 9442 Acked Write 9 Sent Done Done false
31272 9443 Acked Flush 0 Sent Done Done false
31273 9444 Acked Read 5 Sent Done Done false
31274 9445 Acked Flush 0 Sent Done Done false
31275 9446 Acked Write 4 Sent Done Done false
31276 9447 Acked Read 5 Sent Done Done false
31277 9448 Acked Read 2 Sent Done Done false
31278 9449 Acked Write 8 Sent Done Done false
31279 9450 Acked Read 7 Sent Done Done false
31280 9451 Acked Read 1 Sent Done Done false
31281 9452 Acked Write 4 Sent Done Done false
31282 9453 Acked Write 6 Sent Done Done false
31283 9454 Acked Flush 0 Sent Done Done false
31284 STATES DS:0 DS:1 DS:2 TOTAL
31285 Sent 302 0 0 302
31286 Done 0 302 302 604
31287 Skipped 0 0 0 0
31288 Error 0 0 0 0
31289 Last Flush: 9152 9454 9454
31290 Downstairs last five completed: 9152 9151 9150 9149 9148
31291 Upstairs last five completed: 9454 9453 9452 9451 9450
31292 CLIENT: Up:0 ds:302 act:2
312932024-12-09T15:14:23.821ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
312942024-12-09T15:14:23.821ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
312952024-12-09T15:14:23.821ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
312962024-12-09T15:14:23.821ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
312972024-12-09T15:14:23.821ZINFOcrucible: client task is exiting = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31298 [8810] got start action
31299 [8810] I am going to restart
31300 Starting downstairs at port 8810
31301 Make output file at "/tmp/dsc/downstairs-8810.txt"
31302 [8810][0] reports Starting
31303 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1229)
31304 [8810][0] reports Running
31305 ----------------------------------------------------------------
31306 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
31307 States: Offline Active Active
31308 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31309 9153 Acked Read 10 Sent Done Done false
31310 9154 Acked Write 8 Sent Done Done false
31311 9155 Acked Write 3 Sent Done Done false
31312 9156 Acked Read 1 Sent Done Done false
31313 9157 Acked Read 9 Sent Done Done false
31314 9158 Acked Flush 0 Sent Done Done false
31315 9159 Acked Read 6 Sent Done Done false
31316 9160 Acked Write 8 Sent Done Done false
31317 9161 Acked Read 5 Sent Done Done false
31318 9162 Acked Read 1 Sent Done Done false
31319 9163 Acked Read 2 Sent Done Done false
31320 9164 Acked Write 5 Sent Done Done false
31321 9165 Acked Write 3 Sent Done Done false
31322 9166 Acked Read 3 Sent Done Done false
31323 9167 Acked Read 5 Sent Done Done false
31324 9168 Acked Write 2 Sent Done Done false
31325 9169 Acked Write 2 Sent Done Done false
31326 9170 Acked Write 9 Sent Done Done false
31327 9171 Acked Read 9 Sent Done Done false
31328 9172 Acked Write 6 Sent Done Done false
31329 9173 Acked Read 6 Sent Done Done false
31330 9174 Acked Read 4 Sent Done Done false
31331 9175 Acked Write 4 Sent Done Done false
31332 9176 Acked Read 3 Sent Done Done false
31333 9177 Acked Flush 0 Sent Done Done false
31334 9178 Acked Flush 0 Sent Done Done false
31335 9179 Acked Read 9 Sent Done Done false
31336 9180 Acked Read 4 Sent Done Done false
31337 9181 Acked Write 2 Sent Done Done false
31338 9182 Acked Read 7 Sent Done Done false
31339 9183 Acked Write 4 Sent Done Done false
31340 9184 Acked Write 5 Sent Done Done false
31341 9185 Acked Read 9 Sent Done Done false
31342 9186 Acked Read 8 Sent Done Done false
31343 9187 Acked Read 3 Sent Done Done false
31344 9188 Acked Read 4 Sent Done Done false
31345 9189 Acked Write 1 Sent Done Done false
31346 9190 Acked Read 3 Sent Done Done false
31347 9191 Acked Write 2 Sent Done Done false
31348 9192 Acked Read 7 Sent Done Done false
31349 9193 Acked Write 5 Sent Done Done false
31350 9194 Acked Read 9 Sent Done Done false
31351 9195 Acked Write 6 Sent Done Done false
31352 9196 Acked Write 10 Sent Done Done false
31353 9197 Acked Write 8 Sent Done Done false
31354 9198 Acked Read 10 Sent Done Done false
31355 9199 Acked Flush 0 Sent Done Done false
31356 9200 Acked Read 6 Sent Done Done false
31357 9201 Acked Flush 0 Sent Done Done false
31358 9202 Acked Read 2 Sent Done Done false
31359 9203 Acked Write 6 Sent Done Done false
31360 9204 Acked Flush 0 Sent Done Done false
31361 9205 Acked Read 3 Sent Done Done false
31362 9206 Acked Read 9 Sent Done Done false
31363 9207 Acked Read 1 Sent Done Done false
31364 9208 Acked Read 6 Sent Done Done false
31365 9209 Acked Read 6 Sent Done Done false
31366 9210 Acked Write 7 Sent Done Done false
31367 9211 Acked Read 3 Sent Done Done false
31368 9212 Acked Flush 0 Sent Done Done false
31369 9213 Acked Write 1 Sent Done Done false
31370 9214 Acked Read 10 Sent Done Done false
31371 9215 Acked Read 10 Sent Done Done false
31372 9216 Acked Write 1 Sent Done Done false
31373 9217 Acked Read 8 Sent Done Done false
31374 9218 Acked Write 7 Sent Done Done false
31375 9219 Acked Write 4 Sent Done Done false
31376 9220 Acked Write 2 Sent Done Done false
31377 9221 Acked Write 9 Sent Done Done false
31378 9222 Acked Flush 0 Sent Done Done false
31379 9223 Acked Write 10 Sent Done Done false
31380 9224 Acked Write 8 Sent Done Done false
31381 9225 Acked Write 6 Sent Done Done false
31382 9226 Acked Read 10 Sent Done Done false
31383 9227 Acked Write 5 Sent Done Done false
31384 9228 Acked Read 8 Sent Done Done false
31385 9229 Acked Read 10 Sent Done Done false
31386 9230 Acked Flush 0 Sent Done Done false
31387 9231 Acked Read 3 Sent Done Done false
31388 9232 Acked Read 3 Sent Done Done false
31389 9233 Acked Read 4 Sent Done Done false
31390 9234 Acked Write 7 Sent Done Done false
31391 9235 Acked Read 5 Sent Done Done false
31392 9236 Acked Read 4 Sent Done Done false
31393 9237 Acked Read 10 Sent Done Done false
31394 9238 Acked Flush 0 Sent Done Done false
31395 9239 Acked Read 4 Sent Done Done false
31396 9240 Acked Write 9 Sent Done Done false
31397 9241 Acked Write 9 Sent Done Done false
31398 9242 Acked Write 7 Sent Done Done false
31399 9243 Acked Flush 0 Sent Done Done false
31400 9244 Acked Read 7 Sent Done Done false
31401 9245 Acked Read 8 Sent Done Done false
31402 9246 Acked Write 2 Sent Done Done false
31403 9247 Acked Read 1 Sent Done Done false
31404 9248 Acked Flush 0 Sent Done Done false
31405 9249 Acked Write 6 Sent Done Done false
31406 9250 Acked Read 8 Sent Done Done false
31407 9251 Acked Flush 0 Sent Done Done false
31408 9252 Acked Write 9 Sent Done Done false
31409 9253 Acked Read 9 Sent Done Done false
31410 9254 Acked Write 4 Sent Done Done false
31411 9255 Acked Write 1 Sent Done Done false
31412 9256 Acked Read 2 Sent Done Done false
31413 9257 Acked Read 3 Sent Done Done false
31414 9258 Acked Write 4 Sent Done Done false
31415 9259 Acked Read 10 Sent Done Done false
31416 9260 Acked Read 8 Sent Done Done false
31417 9261 Acked Write 7 Sent Done Done false
31418 9262 Acked Write 10 Sent Done Done false
31419 9263 Acked Read 1 Sent Done Done false
31420 9264 Acked Read 10 Sent Done Done false
31421 9265 Acked Read 2 Sent Done Done false
31422 9266 Acked Read 2 Sent Done Done false
31423 9267 Acked Flush 0 Sent Done Done false
31424 9268 Acked Read 5 Sent Done Done false
31425 9269 Acked Write 8 Sent Done Done false
31426 9270 Acked Write 7 Sent Done Done false
31427 9271 Acked Write 3 Sent Done Done false
31428 9272 Acked Read 3 Sent Done Done false
31429 9273 Acked Write 10 Sent Done Done false
31430 9274 Acked Read 2 Sent Done Done false
31431 9275 Acked Write 2 Sent Done Done false
31432 9276 Acked Read 7 Sent Done Done false
31433 9277 Acked Read 7 Sent Done Done false
31434 9278 Acked Flush 0 Sent Done Done false
31435 9279 Acked Read 1 Sent Done Done false
31436 9280 Acked Read 10 Sent Done Done false
31437 9281 Acked Read 2 Sent Done Done false
31438 9282 Acked Write 5 Sent Done Done false
31439 9283 Acked Write 5 Sent Done Done false
31440 9284 Acked Write 7 Sent Done Done false
31441 9285 Acked Read 4 Sent Done Done false
31442 9286 Acked Read 4 Sent Done Done false
31443 9287 Acked Read 4 Sent Done Done false
31444 9288 Acked Flush 0 Sent Done Done false
31445 9289 Acked Read 1 Sent Done Done false
31446 9290 Acked Write 10 Sent Done Done false
31447 9291 Acked Write 3 Sent Done Done false
31448 9292 Acked Read 5 Sent Done Done false
31449 9293 Acked Write 9 Sent Done Done false
31450 9294 Acked Read 5 Sent Done Done false
31451 9295 Acked Read 5 Sent Done Done false
31452 9296 Acked Write 10 Sent Done Done false
31453 9297 Acked Write 3 Sent Done Done false
31454 9298 Acked Write 3 Sent Done Done false
31455 9299 Acked Read 6 Sent Done Done false
31456 9300 Acked Write 10 Sent Done Done false
31457 9301 Acked Write 6 Sent Done Done false
31458 9302 Acked Write 7 Sent Done Done false
31459 9303 Acked Read 2 Sent Done Done false
31460 9304 Acked Write 7 Sent Done Done false
31461 9305 Acked Write 7 Sent Done Done false
31462 9306 Acked Read 3 Sent Done Done false
31463 9307 Acked Read 1 Sent Done Done false
31464 9308 Acked Write 8 Sent Done Done false
31465 9309 Acked Read 6 Sent Done Done false
31466 9310 Acked Read 4 Sent Done Done false
31467 9311 Acked Write 2 Sent Done Done false
31468 9312 Acked Write 9 Sent Done Done false
31469 9313 Acked Read 7 Sent Done Done false
31470 9314 Acked Read 2 Sent Done Done false
31471 9315 Acked Write 9 Sent Done Done false
31472 9316 Acked Read 10 Sent Done Done false
31473 9317 Acked Write 5 Sent Done Done false
31474 9318 Acked Read 4 Sent Done Done false
31475 9319 Acked Flush 0 Sent Done Done false
31476 9320 Acked Write 10 Sent Done Done false
31477 9321 Acked Write 3 Sent Done Done false
31478 9322 Acked Read 1 Sent Done Done false
31479 9323 Acked Read 8 Sent Done Done false
31480 9324 Acked Read 4 Sent Done Done false
31481 9325 Acked Flush 0 Sent Done Done false
31482 9326 Acked Read 2 Sent Done Done false
31483 9327 Acked Read 7 Sent Done Done false
31484 9328 Acked Flush 0 Sent Done Done false
31485 9329 Acked Read 6 Sent Done Done false
31486 9330 Acked Flush 0 Sent Done Done false
31487 9331 Acked Read 2 Sent Done Done false
31488 9332 Acked Write 9 Sent Done Done false
31489 9333 Acked Flush 0 Sent Done Done false
31490 9334 Acked Write 1 Sent Done Done false
31491 9335 Acked Write 5 Sent Done Done false
31492 9336 Acked Read 2 Sent Done Done false
31493 9337 Acked Read 9 Sent Done Done false
31494 9338 Acked Write 10 Sent Done Done false
31495 9339 Acked Read 8 Sent Done Done false
31496 9340 Acked Flush 0 Sent Done Done false
31497 9341 Acked Read 4 Sent Done Done false
31498 9342 Acked Read 5 Sent Done Done false
31499 9343 Acked Write 9 Sent Done Done false
31500 9344 Acked Read 7 Sent Done Done false
31501 9345 Acked Flush 0 Sent Done Done false
31502 9346 Acked Write 6 Sent Done Done false
31503 9347 Acked Read 7 Sent Done Done false
31504 9348 Acked Write 5 Sent Done Done false
31505 9349 Acked Read 4 Sent Done Done false
31506 9350 Acked Write 2 Sent Done Done false
31507 9351 Acked Read 3 Sent Done Done false
31508 9352 Acked Write 2 Sent Done Done false
31509 9353 Acked Read 1 Sent Done Done false
31510 9354 Acked Read 5 Sent Done Done false
31511 9355 Acked Read 7 Sent Done Done false
31512 9356 Acked Read 10 Sent Done Done false
31513 9357 Acked Read 4 Sent Done Done false
31514 9358 Acked Read 7 Sent Done Done false
31515 9359 Acked Write 9 Sent Done Done false
31516 9360 Acked Write 7 Sent Done Done false
31517 9361 Acked Read 4 Sent Done Done false
31518 9362 Acked Write 7 Sent Done Done false
31519 9363 Acked Write 5 Sent Done Done false
31520 9364 Acked Write 7 Sent Done Done false
31521 9365 Acked Write 10 Sent Done Done false
31522 9366 Acked Write 3 Sent Done Done false
31523 9367 Acked Read 6 Sent Done Done false
31524 9368 Acked Write 4 Sent Done Done false
31525 9369 Acked Write 4 Sent Done Done false
31526 9370 Acked Flush 0 Sent Done Done false
31527 9371 Acked Read 7 Sent Done Done false
31528 9372 Acked Write 9 Sent Done Done false
31529 9373 Acked Read 10 Sent Done Done false
31530 9374 Acked Read 8 Sent Done Done false
31531 9375 Acked Read 5 Sent Done Done false
31532 9376 Acked Write 2 Sent Done Done false
31533 9377 Acked Read 6 Sent Done Done false
31534 9378 Acked Read 7 Sent Done Done false
31535 9379 Acked Write 3 Sent Done Done false
31536 9380 Acked Read 1 Sent Done Done false
31537 9381 Acked Write 10 Sent Done Done false
31538 9382 Acked Write 1 Sent Done Done false
31539 9383 Acked Read 8 Sent Done Done false
31540 9384 Acked Read 6 Sent Done Done false
31541 9385 Acked Read 5 Sent Done Done false
31542 9386 Acked Flush 0 Sent Done Done false
31543 9387 Acked Read 5 Sent Done Done false
31544 9388 Acked Write 7 Sent Done Done false
31545 9389 Acked Read 2 Sent Done Done false
31546 9390 Acked Read 4 Sent Done Done false
31547 9391 Acked Write 3 Sent Done Done false
31548 9392 Acked Write 10 Sent Done Done false
31549 9393 Acked Read 5 Sent Done Done false
31550 9394 Acked Read 5 Sent Done Done false
31551 9395 Acked Write 5 Sent Done Done false
31552 9396 Acked Write 10 Sent Done Done false
31553 9397 Acked Read 8 Sent Done Done false
31554 9398 Acked Read 10 Sent Done Done false
31555 9399 Acked Write 1 Sent Done Done false
31556 9400 Acked Write 9 Sent Done Done false
31557 9401 Acked Read 5 Sent Done Done false
31558 9402 Acked Read 9 Sent Done Done false
31559 9403 Acked Read 1 Sent Done Done false
31560 9404 Acked Write 4 Sent Done Done false
31561 9405 Acked Write 7 Sent Done Done false
31562 9406 Acked Write 6 Sent Done Done false
31563 9407 Acked Read 5 Sent Done Done false
31564 9408 Acked Write 7 Sent Done Done false
31565 9409 Acked Write 8 Sent Done Done false
31566 9410 Acked Read 8 Sent Done Done false
31567 9411 Acked Write 3 Sent Done Done false
31568 9412 Acked Read 7 Sent Done Done false
31569 9413 Acked Write 3 Sent Done Done false
31570 9414 Acked Read 5 Sent Done Done false
31571 9415 Acked Read 3 Sent Done Done false
31572 9416 Acked Write 8 Sent Done Done false
31573 9417 Acked Write 5 Sent Done Done false
31574 9418 Acked Read 9 Sent Done Done false
31575 9419 Acked Write 5 Sent Done Done false
31576 9420 Acked Read 4 Sent Done Done false
31577 9421 Acked Write 9 Sent Done Done false
31578 9422 Acked Flush 0 Sent Done Done false
31579 9423 Acked Read 10 Sent Done Done false
31580 9424 Acked Read 5 Sent Done Done false
31581 9425 Acked Read 4 Sent Done Done false
31582 9426 Acked Write 4 Sent Done Done false
31583 9427 Acked Write 1 Sent Done Done false
31584 9428 Acked Flush 0 Sent Done Done false
31585 9429 Acked Write 4 Sent Done Done false
31586 9430 Acked Read 10 Sent Done Done false
31587 9431 Acked Write 9 Sent Done Done false
31588 9432 Acked Flush 0 Sent Done Done false
31589 9433 Acked Read 1 Sent Done Done false
31590 9434 Acked Read 4 Sent Done Done false
31591 9435 Acked Read 6 Sent Done Done false
31592 9436 Acked Flush 0 Sent Done Done false
31593 9437 Acked Write 1 Sent Done Done false
31594 9438 Acked Flush 0 Sent Done Done false
31595 9439 Acked Read 10 Sent Done Done false
31596 9440 Acked Write 8 Sent Done Done false
31597 9441 Acked Read 10 Sent Done Done false
31598 9442 Acked Write 9 Sent Done Done false
31599 9443 Acked Flush 0 Sent Done Done false
31600 9444 Acked Read 5 Sent Done Done false
31601 9445 Acked Flush 0 Sent Done Done false
31602 9446 Acked Write 4 Sent Done Done false
31603 9447 Acked Read 5 Sent Done Done false
31604 9448 Acked Read 2 Sent Done Done false
31605 9449 Acked Write 8 Sent Done Done false
31606 9450 Acked Read 7 Sent Done Done false
31607 9451 Acked Read 1 Sent Done Done false
31608 9452 Acked Write 4 Sent Done Done false
31609 9453 Acked Write 6 Sent Done Done false
31610 9454 Acked Flush 0 Sent Done Done false
31611 STATES DS:0 DS:1 DS:2 TOTAL
31612 Sent 302 0 0 302
31613 Done 0 302 302 604
31614 Skipped 0 0 0 0
31615 Error 0 0 0 0
31616 Last Flush: 9152 9454 9454
31617 Downstairs last five completed: 9152 9151 9150 9149 9148
31618 Upstairs last five completed: 9454 9453 9452 9451 9450
31619 CLIENT: Up:0 ds:302 act:2
31620 ----------------------------------------------------------------
31621 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
31622 States: Offline Active Active
31623 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31624 9153 Acked Read 10 Sent Done Done false
31625 9154 Acked Write 8 Sent Done Done false
31626 9155 Acked Write 3 Sent Done Done false
31627 9156 Acked Read 1 Sent Done Done false
31628 9157 Acked Read 9 Sent Done Done false
31629 9158 Acked Flush 0 Sent Done Done false
31630 9159 Acked Read 6 Sent Done Done false
31631 9160 Acked Write 8 Sent Done Done false
31632 9161 Acked Read 5 Sent Done Done false
31633 9162 Acked Read 1 Sent Done Done false
31634 9163 Acked Read 2 Sent Done Done false
31635 9164 Acked Write 5 Sent Done Done false
31636 9165 Acked Write 3 Sent Done Done false
31637 9166 Acked Read 3 Sent Done Done false
31638 9167 Acked Read 5 Sent Done Done false
31639 9168 Acked Write 2 Sent Done Done false
31640 9169 Acked Write 2 Sent Done Done false
31641 9170 Acked Write 9 Sent Done Done false
31642 9171 Acked Read 9 Sent Done Done false
31643 9172 Acked Write 6 Sent Done Done false
31644 9173 Acked Read 6 Sent Done Done false
31645 9174 Acked Read 4 Sent Done Done false
31646 9175 Acked Write 4 Sent Done Done false
31647 9176 Acked Read 3 Sent Done Done false
31648 9177 Acked Flush 0 Sent Done Done false
31649 9178 Acked Flush 0 Sent Done Done false
31650 9179 Acked Read 9 Sent Done Done false
31651 9180 Acked Read 4 Sent Done Done false
31652 9181 Acked Write 2 Sent Done Done false
31653 9182 Acked Read 7 Sent Done Done false
31654 9183 Acked Write 4 Sent Done Done false
31655 9184 Acked Write 5 Sent Done Done false
31656 9185 Acked Read 9 Sent Done Done false
31657 9186 Acked Read 8 Sent Done Done false
31658 9187 Acked Read 3 Sent Done Done false
31659 9188 Acked Read 4 Sent Done Done false
31660 9189 Acked Write 1 Sent Done Done false
31661 9190 Acked Read 3 Sent Done Done false
31662 9191 Acked Write 2 Sent Done Done false
31663 9192 Acked Read 7 Sent Done Done false
31664 9193 Acked Write 5 Sent Done Done false
31665 9194 Acked Read 9 Sent Done Done false
31666 9195 Acked Write 6 Sent Done Done false
31667 9196 Acked Write 10 Sent Done Done false
31668 9197 Acked Write 8 Sent Done Done false
31669 9198 Acked Read 10 Sent Done Done false
31670 9199 Acked Flush 0 Sent Done Done false
31671 9200 Acked Read 6 Sent Done Done false
31672 9201 Acked Flush 0 Sent Done Done false
31673 9202 Acked Read 2 Sent Done Done false
31674 9203 Acked Write 6 Sent Done Done false
31675 9204 Acked Flush 0 Sent Done Done false
31676 9205 Acked Read 3 Sent Done Done false
31677 9206 Acked Read 9 Sent Done Done false
31678 9207 Acked Read 1 Sent Done Done false
31679 9208 Acked Read 6 Sent Done Done false
31680 9209 Acked Read 6 Sent Done Done false
31681 9210 Acked Write 7 Sent Done Done false
31682 9211 Acked Read 3 Sent Done Done false
31683 9212 Acked Flush 0 Sent Done Done false
31684 9213 Acked Write 1 Sent Done Done false
31685 9214 Acked Read 10 Sent Done Done false
31686 9215 Acked Read 10 Sent Done Done false
31687 9216 Acked Write 1 Sent Done Done false
31688 9217 Acked Read 8 Sent Done Done false
31689 9218 Acked Write 7 Sent Done Done false
31690 9219 Acked Write 4 Sent Done Done false
31691 9220 Acked Write 2 Sent Done Done false
31692 9221 Acked Write 9 Sent Done Done false
31693 9222 Acked Flush 0 Sent Done Done false
31694 9223 Acked Write 10 Sent Done Done false
31695 9224 Acked Write 8 Sent Done Done false
31696 9225 Acked Write 6 Sent Done Done false
31697 9226 Acked Read 10 Sent Done Done false
31698 9227 Acked Write 5 Sent Done Done false
31699 9228 Acked Read 8 Sent Done Done false
31700 9229 Acked Read 10 Sent Done Done false
31701 9230 Acked Flush 0 Sent Done Done false
31702 9231 Acked Read 3 Sent Done Done false
31703 9232 Acked Read 3 Sent Done Done false
31704 9233 Acked Read 4 Sent Done Done false
31705 9234 Acked Write 7 Sent Done Done false
31706 9235 Acked Read 5 Sent Done Done false
31707 9236 Acked Read 4 Sent Done Done false
31708 9237 Acked Read 10 Sent Done Done false
31709 9238 Acked Flush 0 Sent Done Done false
31710 9239 Acked Read 4 Sent Done Done false
31711 9240 Acked Write 9 Sent Done Done false
31712 9241 Acked Write 9 Sent Done Done false
31713 9242 Acked Write 7 Sent Done Done false
31714 9243 Acked Flush 0 Sent Done Done false
31715 9244 Acked Read 7 Sent Done Done false
31716 9245 Acked Read 8 Sent Done Done false
31717 9246 Acked Write 2 Sent Done Done false
31718 9247 Acked Read 1 Sent Done Done false
31719 9248 Acked Flush 0 Sent Done Done false
31720 9249 Acked Write 6 Sent Done Done false
31721 9250 Acked Read 8 Sent Done Done false
31722 9251 Acked Flush 0 Sent Done Done false
31723 9252 Acked Write 9 Sent Done Done false
31724 9253 Acked Read 9 Sent Done Done false
31725 9254 Acked Write 4 Sent Done Done false
31726 9255 Acked Write 1 Sent Done Done false
31727 9256 Acked Read 2 Sent Done Done false
31728 9257 Acked Read 3 Sent Done Done false
31729 9258 Acked Write 4 Sent Done Done false
31730 9259 Acked Read 10 Sent Done Done false
31731 9260 Acked Read 8 Sent Done Done false
31732 9261 Acked Write 7 Sent Done Done false
31733 9262 Acked Write 10 Sent Done Done false
31734 9263 Acked Read 1 Sent Done Done false
31735 9264 Acked Read 10 Sent Done Done false
31736 9265 Acked Read 2 Sent Done Done false
31737 9266 Acked Read 2 Sent Done Done false
31738 9267 Acked Flush 0 Sent Done Done false
31739 9268 Acked Read 5 Sent Done Done false
31740 9269 Acked Write 8 Sent Done Done false
31741 9270 Acked Write 7 Sent Done Done false
31742 9271 Acked Write 3 Sent Done Done false
31743 9272 Acked Read 3 Sent Done Done false
31744 9273 Acked Write 10 Sent Done Done false
31745 9274 Acked Read 2 Sent Done Done false
31746 9275 Acked Write 2 Sent Done Done false
31747 9276 Acked Read 7 Sent Done Done false
31748 9277 Acked Read 7 Sent Done Done false
31749 9278 Acked Flush 0 Sent Done Done false
31750 9279 Acked Read 1 Sent Done Done false
31751 9280 Acked Read 10 Sent Done Done false
31752 9281 Acked Read 2 Sent Done Done false
31753 9282 Acked Write 5 Sent Done Done false
31754 9283 Acked Write 5 Sent Done Done false
31755 9284 Acked Write 7 Sent Done Done false
31756 9285 Acked Read 4 Sent Done Done false
31757 9286 Acked Read 4 Sent Done Done false
31758 9287 Acked Read 4 Sent Done Done false
31759 9288 Acked Flush 0 Sent Done Done false
31760 9289 Acked Read 1 Sent Done Done false
31761 9290 Acked Write 10 Sent Done Done false
31762 9291 Acked Write 3 Sent Done Done false
31763 9292 Acked Read 5 Sent Done Done false
31764 9293 Acked Write 9 Sent Done Done false
31765 9294 Acked Read 5 Sent Done Done false
31766 9295 Acked Read 5 Sent Done Done false
31767 9296 Acked Write 10 Sent Done Done false
31768 9297 Acked Write 3 Sent Done Done false
31769 9298 Acked Write 3 Sent Done Done false
31770 9299 Acked Read 6 Sent Done Done false
31771 9300 Acked Write 10 Sent Done Done false
31772 9301 Acked Write 6 Sent Done Done false
31773 9302 Acked Write 7 Sent Done Done false
31774 9303 Acked Read 2 Sent Done Done false
31775 9304 Acked Write 7 Sent Done Done false
31776 9305 Acked Write 7 Sent Done Done false
31777 9306 Acked Read 3 Sent Done Done false
31778 9307 Acked Read 1 Sent Done Done false
31779 9308 Acked Write 8 Sent Done Done false
31780 9309 Acked Read 6 Sent Done Done false
31781 9310 Acked Read 4 Sent Done Done false
31782 9311 Acked Write 2 Sent Done Done false
31783 9312 Acked Write 9 Sent Done Done false
31784 9313 Acked Read 7 Sent Done Done false
31785 9314 Acked Read 2 Sent Done Done false
31786 9315 Acked Write 9 Sent Done Done false
31787 9316 Acked Read 10 Sent Done Done false
31788 9317 Acked Write 5 Sent Done Done false
31789 9318 Acked Read 4 Sent Done Done false
31790 9319 Acked Flush 0 Sent Done Done false
31791 9320 Acked Write 10 Sent Done Done false
31792 9321 Acked Write 3 Sent Done Done false
31793 9322 Acked Read 1 Sent Done Done false
31794 9323 Acked Read 8 Sent Done Done false
31795 9324 Acked Read 4 Sent Done Done false
31796 9325 Acked Flush 0 Sent Done Done false
31797 9326 Acked Read 2 Sent Done Done false
31798 9327 Acked Read 7 Sent Done Done false
31799 9328 Acked Flush 0 Sent Done Done false
31800 9329 Acked Read 6 Sent Done Done false
31801 9330 Acked Flush 0 Sent Done Done false
31802 9331 Acked Read 2 Sent Done Done false
31803 9332 Acked Write 9 Sent Done Done false
31804 9333 Acked Flush 0 Sent Done Done false
31805 9334 Acked Write 1 Sent Done Done false
31806 9335 Acked Write 5 Sent Done Done false
31807 9336 Acked Read 2 Sent Done Done false
31808 9337 Acked Read 9 Sent Done Done false
31809 9338 Acked Write 10 Sent Done Done false
31810 9339 Acked Read 8 Sent Done Done false
31811 9340 Acked Flush 0 Sent Done Done false
31812 9341 Acked Read 4 Sent Done Done false
31813 9342 Acked Read 5 Sent Done Done false
31814 9343 Acked Write 9 Sent Done Done false
31815 9344 Acked Read 7 Sent Done Done false
31816 9345 Acked Flush 0 Sent Done Done false
31817 9346 Acked Write 6 Sent Done Done false
31818 9347 Acked Read 7 Sent Done Done false
31819 9348 Acked Write 5 Sent Done Done false
31820 9349 Acked Read 4 Sent Done Done false
31821 9350 Acked Write 2 Sent Done Done false
31822 9351 Acked Read 3 Sent Done Done false
31823 9352 Acked Write 2 Sent Done Done false
31824 9353 Acked Read 1 Sent Done Done false
31825 9354 Acked Read 5 Sent Done Done false
31826 9355 Acked Read 7 Sent Done Done false
31827 9356 Acked Read 10 Sent Done Done false
31828 9357 Acked Read 4 Sent Done Done false
31829 9358 Acked Read 7 Sent Done Done false
31830 9359 Acked Write 9 Sent Done Done false
31831 9360 Acked Write 7 Sent Done Done false
31832 9361 Acked Read 4 Sent Done Done false
31833 9362 Acked Write 7 Sent Done Done false
31834 9363 Acked Write 5 Sent Done Done false
31835 9364 Acked Write 7 Sent Done Done false
31836 9365 Acked Write 10 Sent Done Done false
31837 9366 Acked Write 3 Sent Done Done false
31838 9367 Acked Read 6 Sent Done Done false
31839 9368 Acked Write 4 Sent Done Done false
31840 9369 Acked Write 4 Sent Done Done false
31841 9370 Acked Flush 0 Sent Done Done false
31842 9371 Acked Read 7 Sent Done Done false
31843 9372 Acked Write 9 Sent Done Done false
31844 9373 Acked Read 10 Sent Done Done false
31845 9374 Acked Read 8 Sent Done Done false
31846 9375 Acked Read 5 Sent Done Done false
31847 9376 Acked Write 2 Sent Done Done false
31848 9377 Acked Read 6 Sent Done Done false
31849 9378 Acked Read 7 Sent Done Done false
31850 9379 Acked Write 3 Sent Done Done false
31851 9380 Acked Read 1 Sent Done Done false
31852 9381 Acked Write 10 Sent Done Done false
31853 9382 Acked Write 1 Sent Done Done false
31854 9383 Acked Read 8 Sent Done Done false
31855 9384 Acked Read 6 Sent Done Done false
31856 9385 Acked Read 5 Sent Done Done false
31857 9386 Acked Flush 0 Sent Done Done false
31858 9387 Acked Read 5 Sent Done Done false
31859 9388 Acked Write 7 Sent Done Done false
31860 9389 Acked Read 2 Sent Done Done false
31861 9390 Acked Read 4 Sent Done Done false
31862 9391 Acked Write 3 Sent Done Done false
31863 9392 Acked Write 10 Sent Done Done false
31864 9393 Acked Read 5 Sent Done Done false
31865 9394 Acked Read 5 Sent Done Done false
31866 9395 Acked Write 5 Sent Done Done false
31867 9396 Acked Write 10 Sent Done Done false
31868 9397 Acked Read 8 Sent Done Done false
31869 9398 Acked Read 10 Sent Done Done false
31870 9399 Acked Write 1 Sent Done Done false
31871 9400 Acked Write 9 Sent Done Done false
31872 9401 Acked Read 5 Sent Done Done false
31873 9402 Acked Read 9 Sent Done Done false
31874 9403 Acked Read 1 Sent Done Done false
31875 9404 Acked Write 4 Sent Done Done false
31876 9405 Acked Write 7 Sent Done Done false
31877 9406 Acked Write 6 Sent Done Done false
31878 9407 Acked Read 5 Sent Done Done false
31879 9408 Acked Write 7 Sent Done Done false
31880 9409 Acked Write 8 Sent Done Done false
31881 9410 Acked Read 8 Sent Done Done false
31882 9411 Acked Write 3 Sent Done Done false
31883 9412 Acked Read 7 Sent Done Done false
31884 9413 Acked Write 3 Sent Done Done false
31885 9414 Acked Read 5 Sent Done Done false
31886 9415 Acked Read 3 Sent Done Done false
31887 9416 Acked Write 8 Sent Done Done false
31888 9417 Acked Write 5 Sent Done Done false
31889 9418 Acked Read 9 Sent Done Done false
31890 9419 Acked Write 5 Sent Done Done false
31891 9420 Acked Read 4 Sent Done Done false
31892 9421 Acked Write 9 Sent Done Done false
31893 9422 Acked Flush 0 Sent Done Done false
31894 9423 Acked Read 10 Sent Done Done false
31895 9424 Acked Read 5 Sent Done Done false
31896 9425 Acked Read 4 Sent Done Done false
31897 9426 Acked Write 4 Sent Done Done false
31898 9427 Acked Write 1 Sent Done Done false
31899 9428 Acked Flush 0 Sent Done Done false
31900 9429 Acked Write 4 Sent Done Done false
31901 9430 Acked Read 10 Sent Done Done false
31902 9431 Acked Write 9 Sent Done Done false
31903 9432 Acked Flush 0 Sent Done Done false
31904 9433 Acked Read 1 Sent Done Done false
31905 9434 Acked Read 4 Sent Done Done false
31906 9435 Acked Read 6 Sent Done Done false
31907 9436 Acked Flush 0 Sent Done Done false
31908 9437 Acked Write 1 Sent Done Done false
31909 9438 Acked Flush 0 Sent Done Done false
31910 9439 Acked Read 10 Sent Done Done false
31911 9440 Acked Write 8 Sent Done Done false
31912 9441 Acked Read 10 Sent Done Done false
31913 9442 Acked Write 9 Sent Done Done false
31914 9443 Acked Flush 0 Sent Done Done false
31915 9444 Acked Read 5 Sent Done Done false
31916 9445 Acked Flush 0 Sent Done Done false
31917 9446 Acked Write 4 Sent Done Done false
31918 9447 Acked Read 5 Sent Done Done false
31919 9448 Acked Read 2 Sent Done Done false
31920 9449 Acked Write 8 Sent Done Done false
31921 9450 Acked Read 7 Sent Done Done false
31922 9451 Acked Read 1 Sent Done Done false
31923 9452 Acked Write 4 Sent Done Done false
31924 9453 Acked Write 6 Sent Done Done false
31925 9454 Acked Flush 0 Sent Done Done false
31926 STATES DS:0 DS:1 DS:2 TOTAL
31927 Sent 302 0 0 302
31928 Done 0 302 302 604
31929 Skipped 0 0 0 0
31930 Error 0 0 0 0
31931 Last Flush: 9152 9454 9454
31932 Downstairs last five completed: 9152 9151 9150 9149 9148
31933 Upstairs last five completed: 9454 9453 9452 9451 9450
31934 CLIENT: Up:0 ds:302 act:2
319352024-12-09T15:14:33.823ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319362024-12-09T15:14:33.823ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53086) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319372024-12-09T15:14:33.824ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319382024-12-09T15:14:33.824ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319392024-12-09T15:14:33.824ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319402024-12-09T15:14:33.824ZINFOcrucible: send last flush ID to this DS: 9152 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319412024-12-09T15:14:33.825ZINFOcrucible: Replied this last flush ID: 9152 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319422024-12-09T15:14:33.825ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319432024-12-09T15:14:33.825ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319442024-12-09T15:14:33.825ZINFOcrucible: [0] client re-new 302 jobs since flush 9152 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319452024-12-09T15:14:33.825ZINFOcrucible: [0] Replayed 302 jobs since flush: 9152 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31946 ----------------------------------------------------------------
31947 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
31948 Downstairs last five completed: 9454 9453 9452 9451 9450
31949 Upstairs last five completed: 9454 9453 9452 9451 9450
31950 CLIENT: Up:0 ds:0 act:3
31951 Replay: All jobs finished
31952 Main task has work to do, go find it
31953 got dsc Stop(1)
31954 stop 1
31955 Dec 09 15:14:34.068[8820] Got stop action so:false kr:false
31956 INFO request completed, latency_us: 383, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 43a7a621-9317-4209-aed5-7b2805f91a20, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
319572024-12-09T15:14:34.069ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319582024-12-09T15:14:34.069ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319592024-12-09T15:14:34.069ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319602024-12-09T15:14:34.069ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
319612024-12-09T15:14:34.069ZINFOcrucible: [1] Transition from Active to Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31962 Dec 09 15:14:34.069 INFO request completed, latency_us: 126, response_code: 200, uri: /state/cid/1, method: GET, req_id: 0bdb7b36-f9b3-4502-b694-13b2e3772bfc, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
319632024-12-09T15:14:34.069ZINFOcrucible: client task is exiting = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
31964 [8820] Exited with: ExitStatus(unix_wait_status(9))
31965 [8820][1] reports Exit
31966 Dec 09 15:14:38.072 INFO request completed, latency_us: 229, response_code: 200, uri: /state/cid/1, method: GET, req_id: 96f32778-b7e0-40b3-9d3b-4d6a1973153f, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
31967 001/300 Read block 365 len 8192
31968 002/300 Read block 2104 len 16384
31969 003/300 Read block 235 len 16384
31970 004/300 Read block 2257 len 40960
31971 005/300 Read block 1391 len 28672
31972 006/300 Flush
31973 007/300 Write block 1171 len 12288 data: 5 5 5
31974 008/300 Write block 3296 len 28672 data: 4 5 5 6 6 7 6
31975 009/300 Write block 337 len 40960 data: 3 4 3 4 4 4 4 5 5 6
31976 010/300 Write block 2984 len 20480 data: 6 8 9 10 9
31977 011/300 Write block 3688 len 36864 data: 6 6 6 5 4 4 2 2 2
31978 012/300 Write block 2150 len 32768 data: 5 4 3 3 5 4 4 5
31979 013/300 Read block 3303 len 28672
31980 014/300 Flush
31981 015/300 Write block 4390 len 40960 data: 4 2 2 3 4 5 5 6 7 5
31982 016/300 Read block 4540 len 40960
31983 017/300 Write block 1311 len 16384 data: 8 9 9 7
31984 018/300 Flush
31985 019/300 Write block 1603 len 36864 data: 8 5 5 6 5 5 6 6 6
31986 020/300 Write block 4880 len 40960 data: 5 5 4 4 3 3 2 3 5 5
31987 021/300 Write block 1117 len 32768 data: 8 7 7 6 6 5 6 5
31988 022/300 Read block 1499 len 8192
31989 023/300 Read block 2417 len 12288
31990 024/300 Flush
31991 025/300 Write block 4094 len 28672 data: 7 9 11 10 11 10 8
31992 026/300 Read block 3555 len 12288
31993 027/300 Write block 4839 len 16384 data: 7 9 9 8
31994 028/300 Read block 4981 len 20480
31995 029/300 Write block 1666 len 16384 data: 8 7 7 7
31996 030/300 Write block 187 len 32768 data: 7 6 7 7 7 8 5 5
31997 031/300 Write block 1409 len 16384 data: 9 10 10 8
31998 032/300 Write block 502 len 40960 data: 5 6 7 8 9 10 9 9 8 5
31999 033/300 Read block 1445 len 28672
32000 034/300 Read block 4786 len 40960
32001 035/300 Write block 4246 len 4096 data: 7
32002 036/300 Read block 1325 len 36864
32003 037/300 Flush
32004 038/300 Read block 3802 len 4096
32005 039/300 Write block 3111 len 24576 data: 2 3 3 2 2 3
32006 040/300 Write block 2296 len 8192 data: 6 6
32007 041/300 Write block 2930 len 36864 data: 9 11 10 10 10 11 9 9 10
32008 042/300 Write block 4813 len 32768 data: 9 10 7 6 6 7 7 5
32009 043/300 Write block 1183 len 12288 data: 4 4 4
32010 044/300 Read block 2694 len 40960
32011 045/300 Read block 2828 len 24576
32012 046/300 Write block 2060 len 12288 data: 4 5 3
32013 047/300 Write block 2202 len 36864 data: 9 8 4 5 4 3 2 2 4
32014 048/300 Write block 2095 len 12288 data: 5 5 4
32015 049/300 Flush
32016 050/300 Read block 1074 len 28672
32017 051/300 Read block 3901 len 40960
32018 052/300 Flush
32019 053/300 Read block 3335 len 24576
32020 054/300 Read block 4211 len 12288
32021 055/300 Write block 3356 len 12288 data: 5 7 4
32022 056/300 Read block 3969 len 24576
32023 057/300 Read block 2478 len 32768
32024 058/300 Read block 1979 len 20480
32025 059/300 Read block 2996 len 28672
32026 060/300 Read block 657 len 4096
32027 061/300 Write block 1083 len 16384 data: 7 8 7 7
32028 062/300 Write block 2849 len 32768 data: 5 4 4 5 4 4 4 4
32029 063/300 Write block 1276 len 16384 data: 8 8 6 6
32030 064/300 Write block 4473 len 8192 data: 8 8
32031 065/300 Read block 4623 len 40960
32032 066/300 Write block 351 len 16384 data: 4 4 3 3
32033 067/300 Write block 1934 len 16384 data: 7 6 7 5
32034 068/300 Read block 4324 len 16384
32035 069/300 Write block 4892 len 36864 data: 5 6 7 6 5 6 7 7 5
32036 070/300 Read block 3504 len 4096
32037 071/300 Flush
32038 072/300 Read block 997 len 12288
32039 073/300 Read block 4697 len 8192
32040 074/300 Read block 2522 len 40960
32041 075/300 Read block 3761 len 36864
32042 076/300 Write block 3031 len 20480 data: 3 3 3 3 3
32043 077/300 Write block 3583 len 4096 data: 10
32044 078/300 Write block 1630 len 20480 data: 4 5 5 6 6
32045 079/300 Write block 4413 len 4096 data: 7
32046 080/300 Write block 800 len 12288 data: 8 10 10
32047 081/300 Read block 2143 len 4096
32048 082/300 Read block 1544 len 28672
32049 083/300 Read block 2224 len 32768
32050 084/300 Flush
32051 085/300 Read block 2138 len 28672
32052 086/300 Write block 985 len 20480 data: 6 5 5 7 8
32053 087/300 Write block 4026 len 20480 data: 6 6 6 8 7
32054 088/300 Write block 3723 len 36864 data: 7 8 5 7 8 8 7 5 3
32055 089/300 Read block 370 len 20480
32056 090/300 Write block 2720 len 36864 data: 6 4 4 5 6 5 5 5 6
32057 091/300 Flush
32058 092/300 Write block 4521 len 36864 data: 3 3 3 5 5 6 6 7 7
32059 093/300 Write block 4097 len 4096 data: 11
32060 094/300 Read block 4980 len 4096
32061 095/300 Read block 1014 len 20480
32062 096/300 Write block 403 len 8192 data: 9 10
32063 097/300 Write block 750 len 8192 data: 5 5
32064 098/300 Write block 2586 len 16384 data: 4 3 3 2
32065 099/300 Read block 3640 len 32768
32066 100/300 Write block 4352 len 32768 data: 6 6 6 5 5 5 5 6
32067 101/300 Read block 1838 len 24576
32068 102/300 Read block 2265 len 4096
32069 103/300 Write block 4620 len 12288 data: 7 7 7
32070 104/300 Write block 1884 len 24576 data: 6 7 8 9 7 7
32071 105/300 Read block 1104 len 28672
32072 106/300 Write block 1609 len 20480 data: 7 7 7 7 5
32073 107/300 Read block 3763 len 24576
32074 108/300 Read block 589 len 24576
32075 109/300 Write block 4987 len 32768 data: 10 10 11 10 12 10 9 7
32076 110/300 Read block 3514 len 28672
32077 111/300 Write block 292 len 8192 data: 4 4
32078 112/300 Write block 1098 len 12288 data: 4 4 3
32079 113/300 Write block 268 len 16384 data: 7 8 9 9
32080 114/300 Read block 853 len 4096
32081 115/300 Read block 2578 len 32768
32082 116/300 Read block 2485 len 36864
32083 117/300 Read block 1648 len 28672
32084 118/300 Read block 3105 len 40960
32085 119/300 Read block 1203 len 16384
32086 120/300 Read block 0 len 24576
32087 121/300 Read block 2644 len 36864
32088 122/300 Flush
32089 123/300 Flush
32090 124/300 Write block 4124 len 12288 data: 7 6 8
32091 125/300 Read block 1033 len 40960
32092 126/300 Read block 267 len 24576
32093 127/300 Read block 885 len 20480
32094 128/300 Read block 4343 len 8192
32095 129/300 Write block 3981 len 12288 data: 6 7 6
32096 130/300 Read block 393 len 8192
32097 131/300 Read block 757 len 8192
32098 132/300 Write block 1626 len 32768 data: 3 2 2 3 5 6 6 7
32099 133/300 Write block 3209 len 28672 data: 5 6 7 6 7 8 8
32100 134/300 Read block 4789 len 28672
32101 135/300 Read block 3159 len 8192
32102 136/300 Read block 2271 len 4096
32103 137/300 Flush
32104 138/300 Read block 1246 len 4096
32105 139/300 Read block 2802 len 4096
32106 140/300 Read block 2229 len 20480
32107 141/300 Write block 2683 len 20480 data: 5 8 8 8 9
32108 142/300 Read block 3352 len 16384
32109 143/300 Flush
32110 144/300 Read block 2705 len 40960
32111 145/300 Write block 493 len 36864 data: 6 5 6 5 3 4 3 3 5
32112 146/300 Read block 4910 len 4096
32113 147/300 Read block 1687 len 24576
32114 148/300 Write block 2268 len 28672 data: 3 3 3 3 3 3 3
32115 149/300 Read block 528 len 16384
32116 150/300 Read block 834 len 36864
32117 151/300 Read block 4758 len 16384
32118 152/300 Flush
32119 153/300 Write block 3906 len 36864 data: 9 9 8 5 5 5 6 4 4
32120 154/300 Read block 2104 len 24576
32121 155/300 Read block 2179 len 16384
32122 156/300 Write block 3443 len 4096 data: 7
32123 157/300 Read block 233 len 4096
32124 158/300 Read block 1252 len 36864
32125 159/300 Read block 2568 len 32768
32126 160/300 Write block 1686 len 36864 data: 3 3 4 3 4 6 6 8 7
32127 161/300 Write block 2147 len 12288 data: 7 6 6
32128 162/300 Flush
32129 163/300 Read block 3563 len 32768
32130 164/300 Read block 4954 len 8192
32131 165/300 Write block 394 len 28672 data: 3 4 4 5 4 5 5
32132 166/300 Write block 2431 len 16384 data: 5 6 7 7
32133 167/300 Write block 3726 len 28672 data: 8 9 9 8 6 4 2
32134 168/300 Read block 2079 len 24576
32135 169/300 Write block 3124 len 20480 data: 6 5 4 2 3
32136 170/300 Write block 1668 len 28672 data: 8 8 5 5 3 4 2
32137 171/300 Write block 2222 len 32768 data: 5 6 7 6 6 5 5 6
32138 172/300 Read block 2947 len 16384
32139 173/300 Read block 71 len 8192
32140 174/300 Write block 856 len 16384 data: 6 6 6 5
32141 175/300 Read block 3343 len 40960
32142 176/300 Read block 4659 len 4096
32143 177/300 Read block 1343 len 32768
32144 178/300 Read block 1205 len 20480
32145 179/300 Read block 2518 len 36864
32146 180/300 Write block 872 len 12288 data: 8 8 7
32147 181/300 Write block 3091 len 8192 data: 8 7
32148 182/300 Write block 4342 len 16384 data: 5 7 7 6
32149 183/300 Read block 814 len 4096
32150 184/300 Read block 3540 len 36864
32151 185/300 Read block 4867 len 8192
32152 186/300 Read block 2836 len 8192
32153 187/300 Read block 4344 len 8192
32154 188/300 Read block 275 len 16384
32155 189/300 Flush
32156 190/300 Write block 3247 len 36864 data: 5 4 4 4 4 5 5 6 6
32157 191/300 Write block 2668 len 20480 data: 6 5 5 4 5
32158 192/300 Write block 1182 len 8192 data: 5 5
32159 193/300 Flush
32160 194/300 Read block 777 len 8192
32161 195/300 Write block 3569 len 24576 data: 5 4 4 4 5 5
32162 196/300 Read block 3043 len 40960
32163 197/300 Flush
32164 198/300 Read block 3692 len 12288
32165 199/300 Write block 3168 len 20480 data: 6 7 8 8 5
32166 200/300 Read block 2375 len 4096
32167 201/300 Read block 4894 len 8192
32168 202/300 Read block 2721 len 24576
32169 203/300 Flush
32170 204/300 Read block 1516 len 28672
32171 205/300 Flush
32172 206/300 Read block 984 len 32768
32173 207/300 Write block 1700 len 40960 data: 5 4 6 5 4 4 2 2 3 5
32174 208/300 Read block 2503 len 8192
32175 209/300 Read block 551 len 8192
32176 210/300 Read block 1379 len 36864
32177 211/300 Write block 2252 len 4096 data: 4
32178 212/300 Read block 3465 len 8192
32179 213/300 Write block 3448 len 24576 data: 7 5 4 5 5 6
32180 214/300 Read block 4140 len 28672
32181 215/300 Read block 1693 len 20480
32182 216/300 Read block 2379 len 24576
32183 217/300 Write block 3742 len 28672 data: 4 4 6 6 7 7 7
32184 218/300 Write block 174 len 4096 data: 3
32185 219/300 Flush
32186 220/300 Flush
32187 221/300 Flush
32188 222/300 Read block 4143 len 4096
32189 223/300 Write block 1141 len 40960 data: 6 8 7 8 6 5 7 8 8 8
32190 224/300 Write block 962 len 20480 data: 5 7 6 4 4
32191 225/300 Flush
32192 226/300 Read block 3799 len 40960
32193 227/300 Flush
32194 228/300 Flush
32195 229/300 Read block 4222 len 4096
32196 230/300 Write block 2743 len 28672 data: 4 4 4 5 4 5 5
32197 231/300 Write block 178 len 8192 data: 6 6
32198 232/300 Flush
32199 233/300 Read block 484 len 20480
32200 234/300 Read block 2954 len 8192
32201 235/300 Write block 2174 len 12288 data: 8 9 9
32202 236/300 Write block 4267 len 28672 data: 7 5 4 2 3 3 3
32203 237/300 Read block 1489 len 24576
32204 238/300 Flush
32205 239/300 Read block 766 len 16384
32206 240/300 Write block 2932 len 32768 data: 11 11 11 12 10 10 11 10
32207 241/300 Write block 3041 len 16384 data: 3 4 4 5
32208 242/300 Write block 969 len 28672 data: 4 4 4 5 4 5 6
32209 243/300 Read block 2011 len 20480
32210 244/300 Read block 445 len 36864
32211 245/300 Write block 3676 len 40960 data: 5 5 4 5 5 5 5 7 7 8
32212 246/300 Write block 2018 len 32768 data: 11 7 8 8 9 8 5 6
32213 247/300 Flush
32214 248/300 Read block 267 len 36864
32215 249/300 Flush
32216 250/300 Read block 2344 len 40960
32217 251/300 Write block 1010 len 28672 data: 4 5 3 3 3 3 3
32218 252/300 Read block 1464 len 24576
32219 253/300 Write block 1152 len 4096 data: 5
32220 254/300 Read block 345 len 8192
32221 255/300 Write block 3130 len 40960 data: 3 3 6 7 7 8 5 4 5 5
32222 256/300 Flush
32223 257/300 Read block 2354 len 12288
32224 258/300 Read block 615 len 24576
32225 259/300 Write block 3794 len 24576 data: 4 4 3 4 7 9
32226 260/300 Read block 1823 len 32768
32227 261/300 Read block 820 len 16384
32228 262/300 Read block 289 len 4096
32229 263/300 Read block 1654 len 8192
32230 264/300 Read block 848 len 4096
32231 265/300 Write block 372 len 32768 data: 5 4 4 4 4 3 4 2
32232 266/300 Write block 319 len 4096 data: 7
32233 267/300 Read block 4374 len 40960
32234 268/300 Write block 1685 len 36864 data: 5 4 4 5 4 5 7 7 9
32235 269/300 Write block 797 len 28672 data: 6 6 6 9 11 11 9
32236 270/300 Read block 1568 len 16384
32237 271/300 Write block 1764 len 4096 data: 7
32238 272/300 Write block 2542 len 8192 data: 6 4
32239 273/300 Flush
32240 274/300 Read block 633 len 16384
32241 275/300 Flush
32242 276/300 Write block 2331 len 36864 data: 5 5 6 7 9 9 10 9 9
32243 277/300 Read block 2950 len 4096
32244 278/300 Read block 720 len 40960
32245 279/300 Flush
32246 280/300 Flush
32247 281/300 Write block 1852 len 40960 data: 7 7 7 8 7 6 6 5 5 6
32248 282/300 Write block 3551 len 8192 data: 6 6
32249 283/300 Write block 676 len 20480 data: 6 6 6 5 5
32250 284/300 Read block 2228 len 20480
32251 285/300 Write block 4398 len 4096 data: 8
32252 286/300 Write block 2765 len 20480 data: 5 5 4 7 7
32253 287/300 Write block 260 len 28672 data: 5 5 4 3 3 5 5
32254 288/300 Write block 1915 len 8192 data: 3 4
32255 289/300 Write block 1698 len 4096 data: 5
32256 290/300 Read block 1574 len 32768
32257 291/300 Read block 4503 len 40960
32258 292/300 Read block 3 len 12288
32259 293/300 Write block 3673 len 4096 data: 6
32260 294/300 Write block 3269 len 36864 data: 5 5 5 7 8 7 7 5 6
32261 295/300 Write block 4581 len 36864 data: 6 9 9 7 7 7 7 6 4
32262 296/300 Write block 1559 len 4096 data: 4
32263 297/300 Write block 380 len 12288 data: 2 3 3
32264 298/300 Write block 3839 len 32768 data: 6 6 7 9 11 12 12 11
32265 299/300 Read block 3884 len 16384
32266 300/300 Read block 2074 len 40960
32267 301/300 Write block 1821 len 16384 data: 6 8 7 6
32268 Main task has work to do, go find it
32269 got dsc Start(1)
32270 start 1
32271 Dec 09 15:14:38.297 INFO request completed, latency_us: 225, response_code: 204, uri: /start/cid/1, method: POST, req_id: 7f0622e6-93aa-4118-8757-f61f57cd6682, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
32272 Replay: started 1, returned:Ok(())
32273 ----------------------------------------------------------------
32274 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32275 States: Active Offline Active
32276 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32277 9455 Acked Read 2 Done Sent Done false
32278 9456 Acked Read 4 Done Sent Done false
32279 9457 Acked Read 4 Done Sent Done false
32280 9458 Acked Read 10 Done Sent Done false
32281 9459 Acked Read 7 Done Sent Done false
32282 9460 Acked Flush 0 Done Sent Done false
32283 9461 Acked Write 3 Done Sent Done false
32284 9462 Acked Write 7 Done Sent Done false
32285 9463 Acked Write 10 Done Sent Done false
32286 9464 Acked Write 5 Done Sent Done false
32287 9465 Acked Write 9 Done Sent Done false
32288 9466 Acked Write 8 Done Sent Done false
32289 9467 Acked Read 7 Done Sent Done false
32290 9468 Acked Flush 0 Done Sent Done false
32291 9469 Acked Write 10 Done Sent Done false
32292 9470 Acked Read 10 Done Sent Done false
32293 9471 Acked Write 4 Done Sent Done false
32294 9472 Acked Flush 0 Done Sent Done false
32295 9473 Acked Write 9 Done Sent Done false
32296 9474 Acked Write 10 Done Sent Done false
32297 9475 Acked Write 8 Done Sent Done false
32298 9476 Acked Read 2 Done Sent Done false
32299 9477 Acked Read 3 Done Sent Done false
32300 9478 Acked Flush 0 Done Sent Done false
32301 9479 Acked Write 7 Done Sent Done false
32302 9480 Acked Read 3 Done Sent Done false
32303 9481 Acked Write 4 Done Sent Done false
32304 9482 Acked Read 5 Done Sent Done false
32305 9483 Acked Write 4 Done Sent Done false
32306 9484 Acked Write 8 Done Sent Done false
32307 9485 Acked Write 4 Done Sent Done false
32308 9486 Acked Write 10 Done Sent Done false
32309 9487 Acked Read 7 Done Sent Done false
32310 9488 Acked Read 10 Done Sent Done false
32311 9489 Acked Write 1 Done Sent Done false
32312 9490 Acked Read 9 Done Sent Done false
32313 9491 Acked Flush 0 Done Sent Done false
32314 9492 Acked Read 1 Done Sent Done false
32315 9493 Acked Write 6 Done Sent Done false
32316 9494 Acked Write 2 Done Sent Done false
32317 9495 Acked Write 9 Done Sent Done false
32318 9496 Acked Write 8 Done Sent Done false
32319 9497 Acked Write 3 Done Sent Done false
32320 9498 Acked Read 10 Done Sent Done false
32321 9499 Acked Read 6 Done Sent Done false
32322 9500 Acked Write 3 Done Sent Done false
32323 9501 Acked Write 9 Done Sent Done false
32324 9502 Acked Write 3 Done Sent Done false
32325 9503 Acked Flush 0 Done Sent Done false
32326 9504 Acked Read 7 Done Sent Done false
32327 9505 Acked Read 10 Done Sent Done false
32328 9506 Acked Flush 0 Done Sent Done false
32329 9507 Acked Read 6 Done Sent Done false
32330 9508 Acked Read 3 Done Sent Done false
32331 9509 Acked Write 3 Done Sent Done false
32332 9510 Acked Read 6 Done Sent Done false
32333 9511 Acked Read 8 Done Sent Done false
32334 9512 Acked Read 5 Done Sent Done false
32335 9513 Acked Read 7 Done Sent Done false
32336 9514 Acked Read 1 Done Sent Done false
32337 9515 Acked Write 4 Done Sent Done false
32338 9516 Acked Write 8 Done Sent Done false
32339 9517 Acked Write 4 Done Sent Done false
32340 9518 Acked Write 2 Done Sent Done false
32341 9519 Acked Read 10 Done Sent Done false
32342 9520 Acked Write 4 Done Sent Done false
32343 9521 Acked Write 4 Done Sent Done false
32344 9522 Acked Read 4 Done Sent Done false
32345 9523 Acked Write 9 Done Sent Done false
32346 9524 Acked Read 1 Done Sent Done false
32347 9525 Acked Flush 0 Done Sent Done false
32348 9526 Acked Read 3 Done Sent Done false
32349 9527 Acked Read 2 Done Sent Done false
32350 9528 Acked Read 10 Done Sent Done false
32351 9529 Acked Read 9 Done Sent Done false
32352 9530 Acked Write 5 Done Sent Done false
32353 9531 Acked Write 1 Done Sent Done false
32354 9532 Acked Write 5 Done Sent Done false
32355 9533 Acked Write 1 Done Sent Done false
32356 9534 Acked Write 3 Done Sent Done false
32357 9535 Acked Read 1 Done Sent Done false
32358 9536 Acked Read 7 Done Sent Done false
32359 9537 Acked Read 8 Done Sent Done false
32360 9538 Acked Flush 0 Done Sent Done false
32361 9539 Acked Read 7 Done Sent Done false
32362 9540 Acked Write 5 Done Sent Done false
32363 9541 Acked Write 5 Done Sent Done false
32364 9542 Acked Write 9 Done Sent Done false
32365 9543 Acked Read 5 Done Sent Done false
32366 9544 Acked Write 9 Done Sent Done false
32367 9545 Acked Flush 0 Done Sent Done false
32368 9546 Acked Write 9 Done Sent Done false
32369 9547 Acked Write 1 Done Sent Done false
32370 9548 Acked Read 1 Done Sent Done false
32371 9549 Acked Read 5 Done Sent Done false
32372 9550 Acked Write 2 Done Sent Done false
32373 9551 Acked Write 2 Done Sent Done false
32374 9552 Acked Write 4 Done Sent Done false
32375 9553 Acked Read 8 Done Sent Done false
32376 9554 Acked Write 8 Done Sent Done false
32377 9555 Acked Read 6 Done Sent Done false
32378 9556 Acked Read 1 Done Sent Done false
32379 9557 Acked Write 3 Done Sent Done false
32380 9558 Acked Write 6 Done Sent Done false
32381 9559 Acked Read 7 Done Sent Done false
32382 9560 Acked Write 5 Done Sent Done false
32383 9561 Acked Read 6 Done Sent Done false
32384 9562 Acked Read 6 Done Sent Done false
32385 9563 Acked Write 8 Done Sent Done false
32386 9564 Acked Read 7 Done Sent Done false
32387 9565 Acked Write 2 Done Sent Done false
32388 9566 Acked Write 3 Done Sent Done false
32389 9567 Acked Write 4 Done Sent Done false
32390 9568 Acked Read 1 Done Sent Done false
32391 9569 Acked Read 8 Done Sent Done false
32392 9570 Acked Read 9 Done Sent Done false
32393 9571 Acked Read 7 Done Sent Done false
32394 9572 Acked Read 10 Done Sent Done false
32395 9573 Acked Read 4 Done Sent Done false
32396 9574 Acked Read 6 Done Sent Done false
32397 9575 Acked Read 9 Done Sent Done false
32398 9576 Acked Flush 0 Done Sent Done false
32399 9577 Acked Flush 0 Done Sent Done false
32400 9578 Acked Write 3 Done Sent Done false
32401 9579 Acked Read 10 Done Sent Done false
32402 9580 Acked Read 6 Done Sent Done false
32403 9581 Acked Read 5 Done Sent Done false
32404 9582 Acked Read 2 Done Sent Done false
32405 9583 Acked Write 3 Done Sent Done false
32406 9584 Acked Read 2 Done Sent Done false
32407 9585 Acked Read 2 Done Sent Done false
32408 9586 Acked Write 8 Done Sent Done false
32409 9587 Acked Write 7 Done Sent Done false
32410 9588 Acked Read 7 Done Sent Done false
32411 9589 Acked Read 2 Done Sent Done false
32412 9590 Acked Read 1 Done Sent Done false
32413 9591 Acked Flush 0 Done Sent Done false
32414 9592 Acked Read 1 Done Sent Done false
32415 9593 Acked Read 1 Done Sent Done false
32416 9594 Acked Read 5 Done Sent Done false
32417 9595 Acked Write 5 Done Sent Done false
32418 9596 Acked Read 4 Done Sent Done false
32419 9597 Acked Flush 0 Done Sent Done false
32420 9598 Acked Read 10 Done Sent Done false
32421 9599 Acked Write 9 Done Sent Done false
32422 9600 Acked Read 1 Done Sent Done false
32423 9601 Acked Read 6 Done Sent Done false
32424 9602 Acked Write 7 Done Sent Done false
32425 9603 Acked Read 4 Done Sent Done false
32426 9604 Acked Read 9 Done Sent Done false
32427 9605 Acked Read 4 Done Sent Done false
32428 9606 Acked Flush 0 Done Sent Done false
32429 9607 Acked Write 9 Done Sent Done false
32430 9608 Acked Read 6 Done Sent Done false
32431 9609 Acked Read 4 Done Sent Done false
32432 9610 Acked Write 1 Done Sent Done false
32433 9611 Acked Read 1 Done Sent Done false
32434 9612 Acked Read 9 Done Sent Done false
32435 9613 Acked Read 8 Done Sent Done false
32436 9614 Acked Write 9 Done Sent Done false
32437 9615 Acked Write 3 Done Sent Done false
32438 9616 Acked Flush 0 Done Sent Done false
32439 9617 Acked Read 8 Done Sent Done false
32440 9618 Acked Read 2 Done Sent Done false
32441 9619 Acked Write 7 Done Sent Done false
32442 9620 Acked Write 4 Done Sent Done false
32443 9621 Acked Write 7 Done Sent Done false
32444 9622 Acked Read 6 Done Sent Done false
32445 9623 Acked Write 5 Done Sent Done false
32446 9624 Acked Write 7 Done Sent Done false
32447 9625 Acked Write 8 Done Sent Done false
32448 9626 Acked Read 4 Done Sent Done false
32449 9627 Acked Read 2 Done Sent Done false
32450 9628 Acked Write 4 Done Sent Done false
32451 9629 Acked Read 10 Done Sent Done false
32452 9630 Acked Read 1 Done Sent Done false
32453 9631 Acked Read 8 Done Sent Done false
32454 9632 Acked Read 5 Done Sent Done false
32455 9633 Acked Read 9 Done Sent Done false
32456 9634 Acked Write 3 Done Sent Done false
32457 9635 Acked Write 2 Done Sent Done false
32458 9636 Acked Write 4 Done Sent Done false
32459 9637 Acked Read 1 Done Sent Done false
32460 9638 Acked Read 9 Done Sent Done false
32461 9639 Acked Read 2 Done Sent Done false
32462 9640 Acked Read 2 Done Sent Done false
32463 9641 Acked Read 2 Done Sent Done false
32464 9642 Acked Read 4 Done Sent Done false
32465 9643 Acked Flush 0 Done Sent Done false
32466 9644 Acked Write 9 Done Sent Done false
32467 9645 Acked Write 5 Done Sent Done false
32468 9646 Acked Write 2 Done Sent Done false
32469 9647 Acked Flush 0 Done Sent Done false
32470 9648 Acked Read 2 Done Sent Done false
32471 9649 Acked Write 6 Done Sent Done false
32472 9650 Acked Read 10 Done Sent Done false
32473 9651 Acked Flush 0 Done Sent Done false
32474 9652 Acked Read 3 Done Sent Done false
32475 9653 Acked Write 5 Done Sent Done false
32476 9654 Acked Read 1 Done Sent Done false
32477 9655 Acked Read 2 Done Sent Done false
32478 9656 Acked Read 6 Done Sent Done false
32479 9657 Acked Flush 0 Done Sent Done false
32480 9658 Acked Read 7 Done Sent Done false
32481 9659 Acked Flush 0 Done Sent Done false
32482 9660 Acked Read 8 Done Sent Done false
32483 9661 Acked Write 10 Done Sent Done false
32484 9662 Acked Read 2 Done Sent Done false
32485 9663 Acked Read 2 Done Sent Done false
32486 9664 Acked Read 9 Done Sent Done false
32487 9665 Acked Write 1 Done Sent Done false
32488 9666 Acked Read 2 Done Sent Done false
32489 9667 Acked Write 6 Done Sent Done false
32490 9668 Acked Read 7 Done Sent Done false
32491 9669 Acked Read 5 Done Sent Done false
32492 9670 Acked Read 6 Done Sent Done false
32493 9671 Acked Write 7 Done Sent Done false
32494 9672 Acked Write 1 Done Sent Done false
32495 9673 Acked Flush 0 Done Sent Done false
32496 9674 Acked Flush 0 Done Sent Done false
32497 9675 Acked Flush 0 Done Sent Done false
32498 9676 Acked Read 1 Done Sent Done false
32499 9677 Acked Write 10 Done Sent Done false
32500 9678 Acked Write 5 Done Sent Done false
32501 9679 Acked Flush 0 Done Sent Done false
32502 9680 Acked Read 10 Done Sent Done false
32503 9681 Acked Flush 0 Done Sent Done false
32504 9682 Acked Flush 0 Done Sent Done false
32505 9683 Acked Read 1 Done Sent Done false
32506 9684 Acked Write 7 Done Sent Done false
32507 9685 Acked Write 2 Done Sent Done false
32508 9686 Acked Flush 0 Done Sent Done false
32509 9687 Acked Read 5 Done Sent Done false
32510 9688 Acked Read 2 Done Sent Done false
32511 9689 Acked Write 3 Done Sent Done false
32512 9690 Acked Write 7 Done Sent Done false
32513 9691 Acked Read 6 Done Sent Done false
32514 9692 Acked Flush 0 Done Sent Done false
32515 9693 Acked Read 4 Done Sent Done false
32516 9694 Acked Write 8 Done Sent Done false
32517 9695 Acked Write 4 Done Sent Done false
32518 9696 Acked Write 7 Done Sent Done false
32519 9697 Acked Read 5 Done Sent Done false
32520 9698 Acked Read 9 Done Sent Done false
32521 9699 Acked Write 10 Done Sent Done false
32522 9700 Acked Write 8 Done Sent Done false
32523 9701 Acked Flush 0 Done Sent Done false
32524 9702 Acked Read 9 Done Sent Done false
32525 9703 Acked Flush 0 Done Sent Done false
32526 9704 Acked Read 10 Done Sent Done false
32527 9705 Acked Write 7 Done Sent Done false
32528 9706 Acked Read 6 Done Sent Done false
32529 9707 Acked Write 1 Done Sent Done false
32530 9708 Acked Read 2 Done Sent Done false
32531 9709 Acked Write 10 Done Sent Done false
32532 9710 Acked Flush 0 Done Sent Done false
32533 9711 Acked Read 3 Done Sent Done false
32534 9712 Acked Read 6 Done Sent Done false
32535 9713 Acked Write 6 Done Sent Done false
32536 9714 Acked Read 8 Done Sent Done false
32537 9715 Acked Read 4 Done Sent Done false
32538 9716 Acked Read 1 Done Sent Done false
32539 9717 Acked Read 2 Done Sent Done false
32540 9718 Acked Read 1 Done Sent Done false
32541 9719 Acked Write 8 Done Sent Done false
32542 9720 Acked Write 1 Done Sent Done false
32543 9721 Acked Read 10 Done Sent Done false
32544 9722 Acked Write 9 Done Sent Done false
32545 9723 Acked Write 7 Done Sent Done false
32546 9724 Acked Read 4 Done Sent Done false
32547 9725 Acked Write 1 Done Sent Done false
32548 9726 Acked Write 2 Done Sent Done false
32549 9727 Acked Flush 0 Done Sent Done false
32550 9728 Acked Read 4 Done Sent Done false
32551 9729 Acked Flush 0 Done Sent Done false
32552 9730 Acked Write 9 Done Sent Done false
32553 9731 Acked Read 1 Done Sent Done false
32554 9732 Acked Read 10 Done Sent Done false
32555 9733 Acked Flush 0 Done Sent Done false
32556 9734 Acked Flush 0 Done Sent Done false
32557 9735 Acked Write 10 Done Sent Done false
32558 9736 Acked Write 2 Done Sent Done false
32559 9737 Acked Write 5 Done Sent Done false
32560 9738 Acked Read 5 Done Sent Done false
32561 9739 Acked Write 1 Done Sent Done false
32562 9740 Acked Write 5 Done Sent Done false
32563 9741 Acked Write 7 Done Sent Done false
32564 9742 Acked Write 2 Done Sent Done false
32565 9743 Acked Write 1 Done Sent Done false
32566 9744 Acked Read 8 Done Sent Done false
32567 9745 Acked Read 10 Done Sent Done false
32568 9746 Acked Read 3 Done Sent Done false
32569 9747 Acked Write 1 Done Sent Done false
32570 9748 Acked Write 9 Done Sent Done false
32571 9749 Acked Write 9 Done Sent Done false
32572 9750 Acked Write 1 Done Sent Done false
32573 9751 Acked Write 3 Done Sent Done false
32574 9752 Acked Write 8 Done Sent Done false
32575 9753 Acked Read 4 Done Sent Done false
32576 9754 Acked Read 10 Done Sent Done false
32577 9755 Acked Write 4 Done Sent Done false
32578 STATES DS:0 DS:1 DS:2 TOTAL
32579 Sent 0 301 0 301
32580 Done 301 0 301 602
32581 Skipped 0 0 0 0
32582 Error 0 0 0 0
32583 Last Flush: 9734 9454 9734
32584 Downstairs last five completed: 9454 9453 9452 9451 9450
32585 Upstairs last five completed: 9755 9754 9753 9752 9751
32586 CLIENT: Up:0 ds:301 act:2
32587 [8820] got start action
32588 [8820] I am going to restart
32589 Starting downstairs at port 8820
32590 Make output file at "/tmp/dsc/downstairs-8820.txt"
32591 [8820][1] reports Starting
32592 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1230)
32593 [8820][1] reports Running
32594 ----------------------------------------------------------------
32595 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
32596 States: Active Offline Active
32597 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32598 9455 Acked Read 2 Done Sent Done false
32599 9456 Acked Read 4 Done Sent Done false
32600 9457 Acked Read 4 Done Sent Done false
32601 9458 Acked Read 10 Done Sent Done false
32602 9459 Acked Read 7 Done Sent Done false
32603 9460 Acked Flush 0 Done Sent Done false
32604 9461 Acked Write 3 Done Sent Done false
32605 9462 Acked Write 7 Done Sent Done false
32606 9463 Acked Write 10 Done Sent Done false
32607 9464 Acked Write 5 Done Sent Done false
32608 9465 Acked Write 9 Done Sent Done false
32609 9466 Acked Write 8 Done Sent Done false
32610 9467 Acked Read 7 Done Sent Done false
32611 9468 Acked Flush 0 Done Sent Done false
32612 9469 Acked Write 10 Done Sent Done false
32613 9470 Acked Read 10 Done Sent Done false
32614 9471 Acked Write 4 Done Sent Done false
32615 9472 Acked Flush 0 Done Sent Done false
32616 9473 Acked Write 9 Done Sent Done false
32617 9474 Acked Write 10 Done Sent Done false
32618 9475 Acked Write 8 Done Sent Done false
32619 9476 Acked Read 2 Done Sent Done false
32620 9477 Acked Read 3 Done Sent Done false
32621 9478 Acked Flush 0 Done Sent Done false
32622 9479 Acked Write 7 Done Sent Done false
32623 9480 Acked Read 3 Done Sent Done false
32624 9481 Acked Write 4 Done Sent Done false
32625 9482 Acked Read 5 Done Sent Done false
32626 9483 Acked Write 4 Done Sent Done false
32627 9484 Acked Write 8 Done Sent Done false
32628 9485 Acked Write 4 Done Sent Done false
32629 9486 Acked Write 10 Done Sent Done false
32630 9487 Acked Read 7 Done Sent Done false
32631 9488 Acked Read 10 Done Sent Done false
32632 9489 Acked Write 1 Done Sent Done false
32633 9490 Acked Read 9 Done Sent Done false
32634 9491 Acked Flush 0 Done Sent Done false
32635 9492 Acked Read 1 Done Sent Done false
32636 9493 Acked Write 6 Done Sent Done false
32637 9494 Acked Write 2 Done Sent Done false
32638 9495 Acked Write 9 Done Sent Done false
32639 9496 Acked Write 8 Done Sent Done false
32640 9497 Acked Write 3 Done Sent Done false
32641 9498 Acked Read 10 Done Sent Done false
32642 9499 Acked Read 6 Done Sent Done false
32643 9500 Acked Write 3 Done Sent Done false
32644 9501 Acked Write 9 Done Sent Done false
32645 9502 Acked Write 3 Done Sent Done false
32646 9503 Acked Flush 0 Done Sent Done false
32647 9504 Acked Read 7 Done Sent Done false
32648 9505 Acked Read 10 Done Sent Done false
32649 9506 Acked Flush 0 Done Sent Done false
32650 9507 Acked Read 6 Done Sent Done false
32651 9508 Acked Read 3 Done Sent Done false
32652 9509 Acked Write 3 Done Sent Done false
32653 9510 Acked Read 6 Done Sent Done false
32654 9511 Acked Read 8 Done Sent Done false
32655 9512 Acked Read 5 Done Sent Done false
32656 9513 Acked Read 7 Done Sent Done false
32657 9514 Acked Read 1 Done Sent Done false
32658 9515 Acked Write 4 Done Sent Done false
32659 9516 Acked Write 8 Done Sent Done false
32660 9517 Acked Write 4 Done Sent Done false
32661 9518 Acked Write 2 Done Sent Done false
32662 9519 Acked Read 10 Done Sent Done false
32663 9520 Acked Write 4 Done Sent Done false
32664 9521 Acked Write 4 Done Sent Done false
32665 9522 Acked Read 4 Done Sent Done false
32666 9523 Acked Write 9 Done Sent Done false
32667 9524 Acked Read 1 Done Sent Done false
32668 9525 Acked Flush 0 Done Sent Done false
32669 9526 Acked Read 3 Done Sent Done false
32670 9527 Acked Read 2 Done Sent Done false
32671 9528 Acked Read 10 Done Sent Done false
32672 9529 Acked Read 9 Done Sent Done false
32673 9530 Acked Write 5 Done Sent Done false
32674 9531 Acked Write 1 Done Sent Done false
32675 9532 Acked Write 5 Done Sent Done false
32676 9533 Acked Write 1 Done Sent Done false
32677 9534 Acked Write 3 Done Sent Done false
32678 9535 Acked Read 1 Done Sent Done false
32679 9536 Acked Read 7 Done Sent Done false
32680 9537 Acked Read 8 Done Sent Done false
32681 9538 Acked Flush 0 Done Sent Done false
32682 9539 Acked Read 7 Done Sent Done false
32683 9540 Acked Write 5 Done Sent Done false
32684 9541 Acked Write 5 Done Sent Done false
32685 9542 Acked Write 9 Done Sent Done false
32686 9543 Acked Read 5 Done Sent Done false
32687 9544 Acked Write 9 Done Sent Done false
32688 9545 Acked Flush 0 Done Sent Done false
32689 9546 Acked Write 9 Done Sent Done false
32690 9547 Acked Write 1 Done Sent Done false
32691 9548 Acked Read 1 Done Sent Done false
32692 9549 Acked Read 5 Done Sent Done false
32693 9550 Acked Write 2 Done Sent Done false
32694 9551 Acked Write 2 Done Sent Done false
32695 9552 Acked Write 4 Done Sent Done false
32696 9553 Acked Read 8 Done Sent Done false
32697 9554 Acked Write 8 Done Sent Done false
32698 9555 Acked Read 6 Done Sent Done false
32699 9556 Acked Read 1 Done Sent Done false
32700 9557 Acked Write 3 Done Sent Done false
32701 9558 Acked Write 6 Done Sent Done false
32702 9559 Acked Read 7 Done Sent Done false
32703 9560 Acked Write 5 Done Sent Done false
32704 9561 Acked Read 6 Done Sent Done false
32705 9562 Acked Read 6 Done Sent Done false
32706 9563 Acked Write 8 Done Sent Done false
32707 9564 Acked Read 7 Done Sent Done false
32708 9565 Acked Write 2 Done Sent Done false
32709 9566 Acked Write 3 Done Sent Done false
32710 9567 Acked Write 4 Done Sent Done false
32711 9568 Acked Read 1 Done Sent Done false
32712 9569 Acked Read 8 Done Sent Done false
32713 9570 Acked Read 9 Done Sent Done false
32714 9571 Acked Read 7 Done Sent Done false
32715 9572 Acked Read 10 Done Sent Done false
32716 9573 Acked Read 4 Done Sent Done false
32717 9574 Acked Read 6 Done Sent Done false
32718 9575 Acked Read 9 Done Sent Done false
32719 9576 Acked Flush 0 Done Sent Done false
32720 9577 Acked Flush 0 Done Sent Done false
32721 9578 Acked Write 3 Done Sent Done false
32722 9579 Acked Read 10 Done Sent Done false
32723 9580 Acked Read 6 Done Sent Done false
32724 9581 Acked Read 5 Done Sent Done false
32725 9582 Acked Read 2 Done Sent Done false
32726 9583 Acked Write 3 Done Sent Done false
32727 9584 Acked Read 2 Done Sent Done false
32728 9585 Acked Read 2 Done Sent Done false
32729 9586 Acked Write 8 Done Sent Done false
32730 9587 Acked Write 7 Done Sent Done false
32731 9588 Acked Read 7 Done Sent Done false
32732 9589 Acked Read 2 Done Sent Done false
32733 9590 Acked Read 1 Done Sent Done false
32734 9591 Acked Flush 0 Done Sent Done false
32735 9592 Acked Read 1 Done Sent Done false
32736 9593 Acked Read 1 Done Sent Done false
32737 9594 Acked Read 5 Done Sent Done false
32738 9595 Acked Write 5 Done Sent Done false
32739 9596 Acked Read 4 Done Sent Done false
32740 9597 Acked Flush 0 Done Sent Done false
32741 9598 Acked Read 10 Done Sent Done false
32742 9599 Acked Write 9 Done Sent Done false
32743 9600 Acked Read 1 Done Sent Done false
32744 9601 Acked Read 6 Done Sent Done false
32745 9602 Acked Write 7 Done Sent Done false
32746 9603 Acked Read 4 Done Sent Done false
32747 9604 Acked Read 9 Done Sent Done false
32748 9605 Acked Read 4 Done Sent Done false
32749 9606 Acked Flush 0 Done Sent Done false
32750 9607 Acked Write 9 Done Sent Done false
32751 9608 Acked Read 6 Done Sent Done false
32752 9609 Acked Read 4 Done Sent Done false
32753 9610 Acked Write 1 Done Sent Done false
32754 9611 Acked Read 1 Done Sent Done false
32755 9612 Acked Read 9 Done Sent Done false
32756 9613 Acked Read 8 Done Sent Done false
32757 9614 Acked Write 9 Done Sent Done false
32758 9615 Acked Write 3 Done Sent Done false
32759 9616 Acked Flush 0 Done Sent Done false
32760 9617 Acked Read 8 Done Sent Done false
32761 9618 Acked Read 2 Done Sent Done false
32762 9619 Acked Write 7 Done Sent Done false
32763 9620 Acked Write 4 Done Sent Done false
32764 9621 Acked Write 7 Done Sent Done false
32765 9622 Acked Read 6 Done Sent Done false
32766 9623 Acked Write 5 Done Sent Done false
32767 9624 Acked Write 7 Done Sent Done false
32768 9625 Acked Write 8 Done Sent Done false
32769 9626 Acked Read 4 Done Sent Done false
32770 9627 Acked Read 2 Done Sent Done false
32771 9628 Acked Write 4 Done Sent Done false
32772 9629 Acked Read 10 Done Sent Done false
32773 9630 Acked Read 1 Done Sent Done false
32774 9631 Acked Read 8 Done Sent Done false
32775 9632 Acked Read 5 Done Sent Done false
32776 9633 Acked Read 9 Done Sent Done false
32777 9634 Acked Write 3 Done Sent Done false
32778 9635 Acked Write 2 Done Sent Done false
32779 9636 Acked Write 4 Done Sent Done false
32780 9637 Acked Read 1 Done Sent Done false
32781 9638 Acked Read 9 Done Sent Done false
32782 9639 Acked Read 2 Done Sent Done false
32783 9640 Acked Read 2 Done Sent Done false
32784 9641 Acked Read 2 Done Sent Done false
32785 9642 Acked Read 4 Done Sent Done false
32786 9643 Acked Flush 0 Done Sent Done false
32787 9644 Acked Write 9 Done Sent Done false
32788 9645 Acked Write 5 Done Sent Done false
32789 9646 Acked Write 2 Done Sent Done false
32790 9647 Acked Flush 0 Done Sent Done false
32791 9648 Acked Read 2 Done Sent Done false
32792 9649 Acked Write 6 Done Sent Done false
32793 9650 Acked Read 10 Done Sent Done false
32794 9651 Acked Flush 0 Done Sent Done false
32795 9652 Acked Read 3 Done Sent Done false
32796 9653 Acked Write 5 Done Sent Done false
32797 9654 Acked Read 1 Done Sent Done false
32798 9655 Acked Read 2 Done Sent Done false
32799 9656 Acked Read 6 Done Sent Done false
32800 9657 Acked Flush 0 Done Sent Done false
32801 9658 Acked Read 7 Done Sent Done false
32802 9659 Acked Flush 0 Done Sent Done false
32803 9660 Acked Read 8 Done Sent Done false
32804 9661 Acked Write 10 Done Sent Done false
32805 9662 Acked Read 2 Done Sent Done false
32806 9663 Acked Read 2 Done Sent Done false
32807 9664 Acked Read 9 Done Sent Done false
32808 9665 Acked Write 1 Done Sent Done false
32809 9666 Acked Read 2 Done Sent Done false
32810 9667 Acked Write 6 Done Sent Done false
32811 9668 Acked Read 7 Done Sent Done false
32812 9669 Acked Read 5 Done Sent Done false
32813 9670 Acked Read 6 Done Sent Done false
32814 9671 Acked Write 7 Done Sent Done false
32815 9672 Acked Write 1 Done Sent Done false
32816 9673 Acked Flush 0 Done Sent Done false
32817 9674 Acked Flush 0 Done Sent Done false
32818 9675 Acked Flush 0 Done Sent Done false
32819 9676 Acked Read 1 Done Sent Done false
32820 9677 Acked Write 10 Done Sent Done false
32821 9678 Acked Write 5 Done Sent Done false
32822 9679 Acked Flush 0 Done Sent Done false
32823 9680 Acked Read 10 Done Sent Done false
32824 9681 Acked Flush 0 Done Sent Done false
32825 9682 Acked Flush 0 Done Sent Done false
32826 9683 Acked Read 1 Done Sent Done false
32827 9684 Acked Write 7 Done Sent Done false
32828 9685 Acked Write 2 Done Sent Done false
32829 9686 Acked Flush 0 Done Sent Done false
32830 9687 Acked Read 5 Done Sent Done false
32831 9688 Acked Read 2 Done Sent Done false
32832 9689 Acked Write 3 Done Sent Done false
32833 9690 Acked Write 7 Done Sent Done false
32834 9691 Acked Read 6 Done Sent Done false
32835 9692 Acked Flush 0 Done Sent Done false
32836 9693 Acked Read 4 Done Sent Done false
32837 9694 Acked Write 8 Done Sent Done false
32838 9695 Acked Write 4 Done Sent Done false
32839 9696 Acked Write 7 Done Sent Done false
32840 9697 Acked Read 5 Done Sent Done false
32841 9698 Acked Read 9 Done Sent Done false
32842 9699 Acked Write 10 Done Sent Done false
32843 9700 Acked Write 8 Done Sent Done false
32844 9701 Acked Flush 0 Done Sent Done false
32845 9702 Acked Read 9 Done Sent Done false
32846 9703 Acked Flush 0 Done Sent Done false
32847 9704 Acked Read 10 Done Sent Done false
32848 9705 Acked Write 7 Done Sent Done false
32849 9706 Acked Read 6 Done Sent Done false
32850 9707 Acked Write 1 Done Sent Done false
32851 9708 Acked Read 2 Done Sent Done false
32852 9709 Acked Write 10 Done Sent Done false
32853 9710 Acked Flush 0 Done Sent Done false
32854 9711 Acked Read 3 Done Sent Done false
32855 9712 Acked Read 6 Done Sent Done false
32856 9713 Acked Write 6 Done Sent Done false
32857 9714 Acked Read 8 Done Sent Done false
32858 9715 Acked Read 4 Done Sent Done false
32859 9716 Acked Read 1 Done Sent Done false
32860 9717 Acked Read 2 Done Sent Done false
32861 9718 Acked Read 1 Done Sent Done false
32862 9719 Acked Write 8 Done Sent Done false
32863 9720 Acked Write 1 Done Sent Done false
32864 9721 Acked Read 10 Done Sent Done false
32865 9722 Acked Write 9 Done Sent Done false
32866 9723 Acked Write 7 Done Sent Done false
32867 9724 Acked Read 4 Done Sent Done false
32868 9725 Acked Write 1 Done Sent Done false
32869 9726 Acked Write 2 Done Sent Done false
32870 9727 Acked Flush 0 Done Sent Done false
32871 9728 Acked Read 4 Done Sent Done false
32872 9729 Acked Flush 0 Done Sent Done false
32873 9730 Acked Write 9 Done Sent Done false
32874 9731 Acked Read 1 Done Sent Done false
32875 9732 Acked Read 10 Done Sent Done false
32876 9733 Acked Flush 0 Done Sent Done false
32877 9734 Acked Flush 0 Done Sent Done false
32878 9735 Acked Write 10 Done Sent Done false
32879 9736 Acked Write 2 Done Sent Done false
32880 9737 Acked Write 5 Done Sent Done false
32881 9738 Acked Read 5 Done Sent Done false
32882 9739 Acked Write 1 Done Sent Done false
32883 9740 Acked Write 5 Done Sent Done false
32884 9741 Acked Write 7 Done Sent Done false
32885 9742 Acked Write 2 Done Sent Done false
32886 9743 Acked Write 1 Done Sent Done false
32887 9744 Acked Read 8 Done Sent Done false
32888 9745 Acked Read 10 Done Sent Done false
32889 9746 Acked Read 3 Done Sent Done false
32890 9747 Acked Write 1 Done Sent Done false
32891 9748 Acked Write 9 Done Sent Done false
32892 9749 Acked Write 9 Done Sent Done false
32893 9750 Acked Write 1 Done Sent Done false
32894 9751 Acked Write 3 Done Sent Done false
32895 9752 Acked Write 8 Done Sent Done false
32896 9753 Acked Read 4 Done Sent Done false
32897 9754 Acked Read 10 Done Sent Done false
32898 9755 Acked Write 4 Done Sent Done false
32899 9756 Acked Flush 0 Done Sent Done false
32900 STATES DS:0 DS:1 DS:2 TOTAL
32901 Sent 0 302 0 302
32902 Done 302 0 302 604
32903 Skipped 0 0 0 0
32904 Error 0 0 0 0
32905 Last Flush: 9756 9454 9756
32906 Downstairs last five completed: 9454 9453 9452 9451 9450
32907 Upstairs last five completed: 9756 9755 9754 9753 9752
32908 CLIENT: Up:0 ds:302 act:2
329092024-12-09T15:14:44.070ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329102024-12-09T15:14:44.070ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37650) = io task client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329112024-12-09T15:14:44.071ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329122024-12-09T15:14:44.072ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329132024-12-09T15:14:44.072ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329142024-12-09T15:14:44.072ZINFOcrucible: send last flush ID to this DS: 9454 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329152024-12-09T15:14:44.072ZINFOcrucible: Replied this last flush ID: 9454 = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329162024-12-09T15:14:44.072ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329172024-12-09T15:14:44.072ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329182024-12-09T15:14:44.072ZINFOcrucible: [1] client re-new 302 jobs since flush 9454 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329192024-12-09T15:14:44.072ZINFOcrucible: [1] Replayed 302 jobs since flush: 9454 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
32920 ----------------------------------------------------------------
32921 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
32922 Downstairs last five completed: 9756 9755 9754 9753 9752
32923 Upstairs last five completed: 9756 9755 9754 9753 9752
32924 CLIENT: Up:0 ds:0 act:3
32925 Replay: All jobs finished
32926 Main task has work to do, go find it
32927 got dsc Stop(0)
32928 stop 0
32929 Dec 09 15:14:46.303 INFO request completed[8810] Got stop action so:false kr:false
32930 , latency_us: 365, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 62918be0-ab22-43ac-87fc-76c83923600f, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
329312024-12-09T15:14:46.303ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329322024-12-09T15:14:46.303ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329332024-12-09T15:14:46.303ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329342024-12-09T15:14:46.303ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329352024-12-09T15:14:46.303ZINFOcrucible: [0] Transition from Active to Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
329362024-12-09T15:14:46.303ZINFOcrucible: client task is exiting = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
32937 Dec 09 15:14:46.304 INFO request completed, latency_us: 154, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2d7f234f-2fdc-4581-a0ec-3f47c341b998, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
32938 [8810] Exited with: ExitStatus(unix_wait_status(9))
32939 [8810][0] reports Exit
32940 Dec 09 15:14:50.306 INFO request completed, latency_us: 257, response_code: 200, uri: /state/cid/0, method: GET, req_id: cc3012b8-1cfb-41c8-a0bc-e8a379442a09, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
32941 001/300 Read block 917 len 16384
32942 002/300 Read block 4299 len 4096
32943 003/300 Read block 1140 len 12288
32944 004/300 Read block 4566 len 24576
32945 005/300 Read block 3387 len 16384
32946 006/300 Read block 2787 len 36864
32947 007/300 Write block 3704 len 8192 data: 3 4
32948 008/300 Read block 359 len 12288
32949 009/300 Write block 1140 len 28672 data: 4 7 9 8 9 7 6
32950 010/300 Write block 508 len 40960 data: 10 10 9 6 6 7 7 7 6 7
32951 011/300 Write block 4607 len 8192 data: 4 4
32952 012/300 Read block 3961 len 28672
32953 013/300 Read block 2678 len 28672
32954 014/300 Write block 3216 len 24576 data: 7 7 5 4 4 4
32955 015/300 Read block 4382 len 32768
32956 016/300 Write block 1322 len 32768 data: 7 6 5 4 4 4 4 4
32957 017/300 Read block 397 len 28672
32958 018/300 Read block 127 len 32768
32959 019/300 Write block 1583 len 20480 data: 5 5 5 4 3
32960 020/300 Read block 461 len 28672
32961 021/300 Write block 2752 len 12288 data: 7 7 6
32962 022/300 Write block 3734 len 8192 data: 2 4
32963 023/300 Flush
32964 024/300 Read block 3839 len 20480
32965 025/300 Flush
32966 026/300 Read block 1998 len 4096
32967 027/300 Write block 2680 len 20480 data: 7 6 6 6 9
32968 028/300 Read block 2587 len 16384
32969 029/300 Write block 2061 len 16384 data: 6 4 4 5
32970 030/300 Flush
32971 031/300 Read block 1970 len 32768
32972 032/300 Write block 1537 len 20480 data: 6 5 6 5 5
32973 033/300 Read block 1598 len 32768
32974 034/300 Write block 4027 len 8192 data: 7 7
32975 035/300 Flush
32976 036/300 Read block 2126 len 4096
32977 037/300 Read block 2098 len 24576
32978 038/300 Read block 1779 len 24576
32979 039/300 Write block 3122 len 20480 data: 9 8 7 6 5
32980 040/300 Read block 639 len 8192
32981 041/300 Read block 826 len 16384
32982 042/300 Flush
32983 043/300 Read block 2075 len 12288
32984 044/300 Read block 3852 len 20480
32985 045/300 Read block 2705 len 4096
32986 046/300 Write block 2732 len 4096 data: 5
32987 047/300 Write block 3747 len 12288 data: 8 8 6
32988 048/300 Read block 3714 len 36864
32989 049/300 Read block 4791 len 36864
32990 050/300 Read block 3713 len 32768
32991 051/300 Read block 4020 len 12288
32992 052/300 Write block 216 len 4096 data: 10
32993 053/300 Read block 4811 len 28672
32994 054/300 Flush
32995 055/300 Write block 3537 len 16384 data: 7 5 4 5
32996 056/300 Read block 3491 len 40960
32997 057/300 Write block 4411 len 36864 data: 8 7 8 4 4 4 4 3 4
32998 058/300 Read block 132 len 16384
32999 059/300 Write block 4271 len 40960 data: 4 4 4 4 4 3 3 3 3 3
33000 060/300 Write block 2578 len 36864 data: 4 5 5 7 5 5 5 5 5
33001 061/300 Write block 1668 len 32768 data: 9 9 6 6 4 5 3 2
33002 062/300 Read block 1297 len 4096
33003 063/300 Read block 2637 len 12288
33004 064/300 Read block 2218 len 28672
33005 065/300 Read block 1112 len 20480
33006 066/300 Read block 424 len 20480
33007 067/300 Write block 636 len 32768 data: 7 8 7 7 8 7 7 7
33008 068/300 Flush
33009 069/300 Read block 2237 len 20480
33010 070/300 Read block 4434 len 40960
33011 071/300 Read block 2655 len 28672
33012 072/300 Write block 4575 len 36864 data: 6 6 6 7 8 7 7 10 10
33013 073/300 Read block 3340 len 8192
33014 074/300 Write block 4439 len 4096 data: 5
33015 075/300 Read block 3339 len 28672
33016 076/300 Write block 2025 len 4096 data: 7
33017 077/300 Read block 2715 len 8192
33018 078/300 Write block 3805 len 20480 data: 5 6 8 10 8
33019 079/300 Read block 4766 len 4096
33020 080/300 Read block 65 len 8192
33021 081/300 Write block 3892 len 4096 data: 3
33022 082/300 Read block 3068 len 12288
33023 083/300 Read block 959 len 16384
33024 084/300 Read block 1204 len 20480
33025 085/300 Read block 1390 len 12288
33026 086/300 Write block 4610 len 4096 data: 4
33027 087/300 Write block 4815 len 8192 data: 8 7
33028 088/300 Read block 2278 len 16384
33029 089/300 Read block 3830 len 32768
33030 090/300 Read block 4479 len 12288
33031 091/300 Flush
33032 092/300 Read block 3071 len 36864
33033 093/300 Flush
33034 094/300 Write block 4191 len 40960 data: 7 8 5 4 3 4 4 4 5 4
33035 095/300 Write block 4101 len 36864 data: 9 7 8 8 10 9 7 6 6
33036 096/300 Write block 3681 len 24576 data: 6 6 8 8 9 5
33037 097/300 Write block 4358 len 16384 data: 6 7 7 7
33038 098/300 Read block 1362 len 32768
33039 099/300 Write block 2548 len 24576 data: 6 5 4 4 5 5
33040 100/300 Write block 4176 len 24576 data: 4 4 3 5 6 6
33041 101/300 Read block 4859 len 40960
33042 102/300 Read block 4897 len 36864
33043 103/300 Read block 1568 len 28672
33044 104/300 Read block 4489 len 12288
33045 105/300 Write block 4058 len 8192 data: 2 2
33046 106/300 Read block 4080 len 4096
33047 107/300 Write block 1630 len 20480 data: 6 7 7 8 7
33048 108/300 Write block 1838 len 20480 data: 4 5 8 8 8
33049 109/300 Write block 4348 len 32768 data: 8 10 9 8 7 7 7 6
33050 110/300 Read block 2478 len 24576
33051 111/300 Read block 4025 len 20480
33052 112/300 Read block 4551 len 20480
33053 113/300 Read block 4330 len 28672
33054 114/300 Flush
33055 115/300 Write block 2350 len 8192 data: 8 8
33056 116/300 Write block 4857 len 40960 data: 4 4 4 5 6 5 5 6 6 7
33057 117/300 Flush
33058 118/300 Write block 3493 len 28672 data: 5 5 5 3 3 2 3
33059 119/300 Read block 938 len 4096
33060 120/300 Read block 3230 len 16384
33061 121/300 Read block 855 len 12288
33062 122/300 Read block 4235 len 36864
33063 123/300 Read block 2686 len 12288
33064 124/300 Write block 3812 len 28672 data: 5 6 7 6 5 5 8
33065 125/300 Write block 314 len 4096 data: 7
33066 126/300 Read block 2762 len 28672
33067 127/300 Write block 67 len 20480 data: 2 3 3 3 2
33068 128/300 Write block 465 len 20480 data: 4 6 7 5 5
33069 129/300 Write block 1639 len 4096 data: 5
33070 130/300 Write block 41 len 28672 data: 4 3 2 3 4 5 6
33071 131/300 Write block 568 len 16384 data: 3 3 4 5
33072 132/300 Write block 4537 len 36864 data: 3 5 4 5 5 5 5 5 6
33073 133/300 Flush
33074 134/300 Write block 4753 len 28672 data: 7 7 6 6 4 4 6
33075 135/300 Read block 1047 len 16384
33076 136/300 Read block 27 len 36864
33077 137/300 Read block 2216 len 28672
33078 138/300 Read block 2857 len 16384
33079 139/300 Read block 708 len 32768
33080 140/300 Read block 4031 len 4096
33081 141/300 Write block 1932 len 40960 data: 8 8 8 7 8 6 4 3 3 3
33082 142/300 Read block 4830 len 32768
33083 143/300 Read block 4604 len 8192
33084 144/300 Flush
33085 145/300 Read block 719 len 32768
33086 146/300 Read block 1316 len 40960
33087 147/300 Flush
33088 148/300 Write block 4479 len 36864 data: 2 3 3 3 3 3 3 5 5
33089 149/300 Write block 1995 len 32768 data: 6 9 10 7 7 8 8 8
33090 150/300 Read block 896 len 16384
33091 151/300 Write block 2513 len 4096 data: 5
33092 152/300 Read block 123 len 32768
33093 153/300 Read block 4253 len 4096
33094 154/300 Read block 993 len 40960
33095 155/300 Read block 2854 len 20480
33096 156/300 Read block 4445 len 28672
33097 157/300 Flush
33098 158/300 Read block 3895 len 12288
33099 159/300 Read block 170 len 32768
33100 160/300 Read block 1083 len 4096
33101 161/300 Flush
33102 162/300 Read block 2136 len 4096
33103 163/300 Write block 4696 len 20480 data: 3 4 5 5 6
33104 164/300 Write block 4354 len 20480 data: 8 7 6 6 7
33105 165/300 Read block 1451 len 40960
33106 166/300 Read block 702 len 8192
33107 167/300 Write block 4544 len 4096 data: 6
33108 168/300 Write block 4419 len 40960 data: 5 4 5 6 7 8 9 8 7 7
33109 169/300 Read block 2347 len 16384
33110 170/300 Write block 4599 len 20480 data: 5 6 8 7 6
33111 171/300 Read block 3803 len 36864
33112 172/300 Read block 928 len 4096
33113 173/300 Read block 2548 len 24576
33114 174/300 Flush
33115 175/300 Write block 3974 len 32768 data: 7 7 7 6 6 7 6 7
33116 176/300 Flush
33117 177/300 Write block 3915 len 36864 data: 4 5 5 5 3 3 3 3 4
33118 178/300 Write block 1363 len 12288 data: 7 7 5
33119 179/300 Write block 2357 len 16384 data: 7 7 4 3
33120 180/300 Read block 1676 len 36864
33121 181/300 Flush
33122 182/300 Read block 3060 len 16384
33123 183/300 Read block 3174 len 4096
33124 184/300 Read block 2505 len 24576
33125 185/300 Read block 940 len 4096
33126 186/300 Read block 1881 len 40960
33127 187/300 Read block 1642 len 16384
33128 188/300 Read block 2934 len 12288
33129 189/300 Read block 13 len 8192
33130 190/300 Flush
33131 191/300 Read block 4068 len 24576
33132 192/300 Write block 4732 len 8192 data: 3 2
33133 193/300 Read block 3705 len 28672
33134 194/300 Read block 1525 len 40960
33135 195/300 Write block 1461 len 8192 data: 5 5
33136 196/300 Read block 2379 len 20480
33137 197/300 Write block 3287 len 40960 data: 6 7 8 9 8 8 6 6 5 5
33138 198/300 Write block 2476 len 36864 data: 5 5 6 7 8 7 7 9 9
33139 199/300 Read block 1605 len 36864
33140 200/300 Write block 3581 len 40960 data: 9 9 11 11 8 9 7 6 4 4
33141 201/300 Read block 3022 len 28672
33142 202/300 Read block 4777 len 8192
33143 203/300 Read block 2312 len 24576
33144 204/300 Write block 331 len 36864 data: 4 3 2 2 2 3 4 5 4
33145 205/300 Write block 430 len 40960 data: 5 6 6 4 3 3 4 4 3 3
33146 206/300 Write block 347 len 32768 data: 5 4 4 4 5 5 4 4
33147 207/300 Write block 4425 len 12288 data: 10 9 8
33148 208/300 Read block 1875 len 36864
33149 209/300 Read block 4579 len 40960
33150 210/300 Read block 1518 len 28672
33151 211/300 Write block 1502 len 24576 data: 7 7 5 4 4 5
33152 212/300 Write block 30 len 32768 data: 7 6 8 7 7 8 5 6
33153 213/300 Write block 1586 len 4096 data: 5
33154 214/300 Read block 1674 len 28672
33155 215/300 Read block 1010 len 8192
33156 216/300 Write block 1248 len 4096 data: 7
33157 217/300 Flush
33158 218/300 Read block 866 len 28672
33159 219/300 Read block 2270 len 20480
33160 220/300 Read block 4371 len 40960
33161 221/300 Flush
33162 222/300 Read block 1673 len 36864
33163 223/300 Read block 1344 len 16384
33164 224/300 Read block 1586 len 12288
33165 225/300 Read block 3596 len 4096
33166 226/300 Read block 562 len 8192
33167 227/300 Write block 2791 len 32768 data: 8 8 8 7 7 5 6 6
33168 228/300 Read block 2990 len 4096
33169 229/300 Write block 4630 len 28672 data: 4 4 5 5 5 6 6
33170 230/300 Write block 3622 len 32768 data: 6 5 4 3 3 3 3 4
33171 231/300 Read block 3252 len 28672
33172 232/300 Read block 4562 len 40960
33173 233/300 Read block 2135 len 12288
33174 234/300 Write block 27 len 32768 data: 6 7 7 8 7 9 8 8
33175 235/300 Read block 4487 len 36864
33176 236/300 Read block 1512 len 8192
33177 237/300 Write block 2213 len 8192 data: 4 4
33178 238/300 Read block 1708 len 8192
33179 239/300 Read block 871 len 32768
33180 240/300 Read block 542 len 28672
33181 241/300 Read block 4709 len 40960
33182 242/300 Write block 2434 len 40960 data: 8 7 7 6 6 7 6 6 7 7
33183 243/300 Write block 3669 len 20480 data: 7 5 6 5 7
33184 244/300 Flush
33185 245/300 Write block 4029 len 12288 data: 9 8 6
33186 246/300 Flush
33187 247/300 Write block 2269 len 36864 data: 4 4 4 4 4 4 2 3 3
33188 248/300 Write block 1590 len 20480 data: 3 3 4 3 3
33189 249/300 Write block 839 len 40960 data: 4 4 5 5 4 5 6 5 5 4
33190 250/300 Write block 4078 len 28672 data: 2 3 3 4 5 7 6
33191 251/300 Write block 1087 len 36864 data: 6 7 9 10 9 8 9 10 8
33192 252/300 Read block 1037 len 40960
33193 253/300 Write block 4251 len 12288 data: 5 6 5
33194 254/300 Write block 1536 len 16384 data: 5 7 6 7
33195 255/300 Write block 3978 len 24576 data: 7 8 7 8 8 7
33196 256/300 Read block 4561 len 12288
33197 257/300 Read block 2491 len 12288
33198 258/300 Read block 1263 len 28672
33199 259/300 Flush
33200 260/300 Read block 4462 len 32768
33201 261/300 Write block 2133 len 24576 data: 10 11 9 9 9 10
33202 262/300 Read block 4255 len 8192
33203 263/300 Flush
33204 264/300 Write block 2344 len 32768 data: 7 7 8 8 8 9 9 9
33205 265/300 Read block 3745 len 40960
33206 266/300 Write block 2782 len 20480 data: 6 8 7 9 6
33207 267/300 Write block 4556 len 40960 data: 10 9 7 7 5 6 5 5 3 5
33208 268/300 Write block 1420 len 16384 data: 3 4 3 2
33209 269/300 Write block 207 len 20480 data: 5 5 6 5 5
33210 270/300 Read block 810 len 40960
33211 271/300 Write block 504 len 28672 data: 8 9 10 11 11 11 10
33212 272/300 Read block 2672 len 32768
33213 273/300 Write block 2430 len 16384 data: 4 6 7 8
33214 274/300 Read block 1457 len 28672
33215 275/300 Write block 2574 len 24576 data: 6 5 4 4 5 6
33216 276/300 Read block 4507 len 4096
33217 277/300 Write block 2292 len 16384 data: 4 4 3 4
33218 278/300 Read block 3246 len 32768
33219 279/300 Read block 2843 len 4096
33220 280/300 Write block 4791 len 12288 data: 4 4 4
33221 281/300 Write block 389 len 28672 data: 4 5 3 3 3 4 5
33222 282/300 Read block 3992 len 24576
33223 283/300 Read block 3896 len 4096
33224 284/300 Read block 1363 len 8192
33225 285/300 Write block 4366 len 4096 data: 3
33226 286/300 Read block 76 len 4096
33227 287/300 Read block 1791 len 16384
33228 288/300 Read block 1940 len 40960
33229 289/300 Write block 1902 len 24576 data: 4 3 4 3 5 6
33230 290/300 Read block 2945 len 12288
33231 291/300 Write block 531 len 12288 data: 6 6 7
33232 292/300 Read block 4105 len 16384
33233 293/300 Write block 3931 len 28672 data: 7 7 6 6 6 7 5
33234 294/300 Flush
33235 295/300 Write block 1424 len 8192 data: 2 3
33236 296/300 Read block 4086 len 40960
33237 297/300 Flush
33238 298/300 Flush
33239 299/300 Read block 3074 len 16384
33240 300/300 Read block 1386 len 4096
33241 301/300 Write block 3775 len 32768 data: 6 7 8 7 8 7 8 8
33242 Main task has work to do, go find it
33243 got dsc Start(0)
33244 start 0
33245 Dec 09 15:14:50.547 INFO request completed, latency_us: 259, response_code: 204, uri: /start/cid/0, method: POST, req_id: 0f30a199-4162-4ee2-9c9e-c02f119e035b, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
33246 Replay: started 0, returned:Ok(())
33247 ----------------------------------------------------------------
33248 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33249 States: Offline Active Active
33250 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33251 9757 Acked Read 4 Sent Done Done false
33252 9758 Acked Read 1 Sent Done Done false
33253 9759 Acked Read 3 Sent Done Done false
33254 9760 Acked Read 6 Sent Done Done false
33255 9761 Acked Read 4 Sent Done Done false
33256 9762 Acked Read 9 Sent Done Done false
33257 9763 Acked Write 2 Sent Done Done false
33258 9764 Acked Read 3 Sent Done Done false
33259 9765 Acked Write 7 Sent Done Done false
33260 9766 Acked Write 10 Sent Done Done false
33261 9767 Acked Write 2 Sent Done Done false
33262 9768 Acked Read 7 Sent Done Done false
33263 9769 Acked Read 7 Sent Done Done false
33264 9770 Acked Write 6 Sent Done Done false
33265 9771 Acked Read 8 Sent Done Done false
33266 9772 Acked Write 8 Sent Done Done false
33267 9773 Acked Read 7 Sent Done Done false
33268 9774 Acked Read 8 Sent Done Done false
33269 9775 Acked Write 5 Sent Done Done false
33270 9776 Acked Read 7 Sent Done Done false
33271 9777 Acked Write 3 Sent Done Done false
33272 9778 Acked Write 2 Sent Done Done false
33273 9779 Acked Flush 0 Sent Done Done false
33274 9780 Acked Read 5 Sent Done Done false
33275 9781 Acked Flush 0 Sent Done Done false
33276 9782 Acked Read 1 Sent Done Done false
33277 9783 Acked Write 5 Sent Done Done false
33278 9784 Acked Read 4 Sent Done Done false
33279 9785 Acked Write 4 Sent Done Done false
33280 9786 Acked Flush 0 Sent Done Done false
33281 9787 Acked Read 8 Sent Done Done false
33282 9788 Acked Write 5 Sent Done Done false
33283 9789 Acked Read 8 Sent Done Done false
33284 9790 Acked Write 2 Sent Done Done false
33285 9791 Acked Flush 0 Sent Done Done false
33286 9792 Acked Read 1 Sent Done Done false
33287 9793 Acked Read 6 Sent Done Done false
33288 9794 Acked Read 6 Sent Done Done false
33289 9795 Acked Write 5 Sent Done Done false
33290 9796 Acked Read 2 Sent Done Done false
33291 9797 Acked Read 4 Sent Done Done false
33292 9798 Acked Flush 0 Sent Done Done false
33293 9799 Acked Read 3 Sent Done Done false
33294 9800 Acked Read 5 Sent Done Done false
33295 9801 Acked Read 1 Sent Done Done false
33296 9802 Acked Write 1 Sent Done Done false
33297 9803 Acked Write 3 Sent Done Done false
33298 9804 Acked Read 9 Sent Done Done false
33299 9805 Acked Read 9 Sent Done Done false
33300 9806 Acked Read 8 Sent Done Done false
33301 9807 Acked Read 3 Sent Done Done false
33302 9808 Acked Write 1 Sent Done Done false
33303 9809 Acked Read 7 Sent Done Done false
33304 9810 Acked Flush 0 Sent Done Done false
33305 9811 Acked Write 4 Sent Done Done false
33306 9812 Acked Read 10 Sent Done Done false
33307 9813 Acked Write 9 Sent Done Done false
33308 9814 Acked Read 4 Sent Done Done false
33309 9815 Acked Write 10 Sent Done Done false
33310 9816 Acked Write 9 Sent Done Done false
33311 9817 Acked Write 8 Sent Done Done false
33312 9818 Acked Read 1 Sent Done Done false
33313 9819 Acked Read 3 Sent Done Done false
33314 9820 Acked Read 7 Sent Done Done false
33315 9821 Acked Read 5 Sent Done Done false
33316 9822 Acked Read 5 Sent Done Done false
33317 9823 Acked Write 8 Sent Done Done false
33318 9824 Acked Flush 0 Sent Done Done false
33319 9825 Acked Read 5 Sent Done Done false
33320 9826 Acked Read 10 Sent Done Done false
33321 9827 Acked Read 7 Sent Done Done false
33322 9828 Acked Write 9 Sent Done Done false
33323 9829 Acked Read 2 Sent Done Done false
33324 9830 Acked Write 1 Sent Done Done false
33325 9831 Acked Read 7 Sent Done Done false
33326 9832 Acked Write 1 Sent Done Done false
33327 9833 Acked Read 2 Sent Done Done false
33328 9834 Acked Write 5 Sent Done Done false
33329 9835 Acked Read 1 Sent Done Done false
33330 9836 Acked Read 2 Sent Done Done false
33331 9837 Acked Write 1 Sent Done Done false
33332 9838 Acked Read 3 Sent Done Done false
33333 9839 Acked Read 4 Sent Done Done false
33334 9840 Acked Read 5 Sent Done Done false
33335 9841 Acked Read 3 Sent Done Done false
33336 9842 Acked Write 1 Sent Done Done false
33337 9843 Acked Write 2 Sent Done Done false
33338 9844 Acked Read 4 Sent Done Done false
33339 9845 Acked Read 8 Sent Done Done false
33340 9846 Acked Read 3 Sent Done Done false
33341 9847 Acked Flush 0 Sent Done Done false
33342 9848 Acked Read 9 Sent Done Done false
33343 9849 Acked Flush 0 Sent Done Done false
33344 9850 Acked Write 10 Sent Done Done false
33345 9851 Acked Write 9 Sent Done Done false
33346 9852 Acked Write 6 Sent Done Done false
33347 9853 Acked Write 4 Sent Done Done false
33348 9854 Acked Read 8 Sent Done Done false
33349 9855 Acked Write 6 Sent Done Done false
33350 9856 Acked Write 6 Sent Done Done false
33351 9857 Acked Read 10 Sent Done Done false
33352 9858 Acked Read 9 Sent Done Done false
33353 9859 Acked Read 7 Sent Done Done false
33354 9860 Acked Read 3 Sent Done Done false
33355 9861 Acked Write 2 Sent Done Done false
33356 9862 Acked Read 1 Sent Done Done false
33357 9863 Acked Write 5 Sent Done Done false
33358 9864 Acked Write 5 Sent Done Done false
33359 9865 Acked Write 8 Sent Done Done false
33360 9866 Acked Read 6 Sent Done Done false
33361 9867 Acked Read 5 Sent Done Done false
33362 9868 Acked Read 5 Sent Done Done false
33363 9869 Acked Read 7 Sent Done Done false
33364 9870 Acked Flush 0 Sent Done Done false
33365 9871 Acked Write 2 Sent Done Done false
33366 9872 Acked Write 10 Sent Done Done false
33367 9873 Acked Flush 0 Sent Done Done false
33368 9874 Acked Write 7 Sent Done Done false
33369 9875 Acked Read 1 Sent Done Done false
33370 9876 Acked Read 4 Sent Done Done false
33371 9877 Acked Read 3 Sent Done Done false
33372 9878 Acked Read 9 Sent Done Done false
33373 9879 Acked Read 3 Sent Done Done false
33374 9880 Acked Write 7 Sent Done Done false
33375 9881 Acked Write 1 Sent Done Done false
33376 9882 Acked Read 7 Sent Done Done false
33377 9883 Acked Write 5 Sent Done Done false
33378 9884 Acked Write 5 Sent Done Done false
33379 9885 Acked Write 1 Sent Done Done false
33380 9886 Acked Write 7 Sent Done Done false
33381 9887 Acked Write 4 Sent Done Done false
33382 9888 Acked Write 9 Sent Done Done false
33383 9889 Acked Flush 0 Sent Done Done false
33384 9890 Acked Write 7 Sent Done Done false
33385 9891 Acked Read 4 Sent Done Done false
33386 9892 Acked Read 9 Sent Done Done false
33387 9893 Acked Read 7 Sent Done Done false
33388 9894 Acked Read 4 Sent Done Done false
33389 9895 Acked Read 8 Sent Done Done false
33390 9896 Acked Read 1 Sent Done Done false
33391 9897 Acked Write 10 Sent Done Done false
33392 9898 Acked Read 8 Sent Done Done false
33393 9899 Acked Read 2 Sent Done Done false
33394 9900 Acked Flush 0 Sent Done Done false
33395 9901 Acked Read 8 Sent Done Done false
33396 9902 Acked Read 10 Sent Done Done false
33397 9903 Acked Flush 0 Sent Done Done false
33398 9904 Acked Write 9 Sent Done Done false
33399 9905 Acked Write 8 Sent Done Done false
33400 9906 Acked Read 4 Sent Done Done false
33401 9907 Acked Write 1 Sent Done Done false
33402 9908 Acked Read 8 Sent Done Done false
33403 9909 Acked Read 1 Sent Done Done false
33404 9910 Acked Read 10 Sent Done Done false
33405 9911 Acked Read 5 Sent Done Done false
33406 9912 Acked Read 7 Sent Done Done false
33407 9913 Acked Flush 0 Sent Done Done false
33408 9914 Acked Read 3 Sent Done Done false
33409 9915 Acked Read 8 Sent Done Done false
33410 9916 Acked Read 1 Sent Done Done false
33411 9917 Acked Flush 0 Sent Done Done false
33412 9918 Acked Read 1 Sent Done Done false
33413 9919 Acked Write 5 Sent Done Done false
33414 9920 Acked Write 5 Sent Done Done false
33415 9921 Acked Read 10 Sent Done Done false
33416 9922 Acked Read 2 Sent Done Done false
33417 9923 Acked Write 1 Sent Done Done false
33418 9924 Acked Write 10 Sent Done Done false
33419 9925 Acked Read 4 Sent Done Done false
33420 9926 Acked Write 5 Sent Done Done false
33421 9927 Acked Read 9 Sent Done Done false
33422 9928 Acked Read 1 Sent Done Done false
33423 9929 Acked Read 6 Sent Done Done false
33424 9930 Acked Flush 0 Sent Done Done false
33425 9931 Acked Write 8 Sent Done Done false
33426 9932 Acked Flush 0 Sent Done Done false
33427 9933 Acked Write 9 Sent Done Done false
33428 9934 Acked Write 3 Sent Done Done false
33429 9935 Acked Write 4 Sent Done Done false
33430 9936 Acked Read 9 Sent Done Done false
33431 9937 Acked Flush 0 Sent Done Done false
33432 9938 Acked Read 4 Sent Done Done false
33433 9939 Acked Read 1 Sent Done Done false
33434 9940 Acked Read 6 Sent Done Done false
33435 9941 Acked Read 1 Sent Done Done false
33436 9942 Acked Read 10 Sent Done Done false
33437 9943 Acked Read 4 Sent Done Done false
33438 9944 Acked Read 3 Sent Done Done false
33439 9945 Acked Read 2 Sent Done Done false
33440 9946 Acked Flush 0 Sent Done Done false
33441 9947 Acked Read 6 Sent Done Done false
33442 9948 Acked Write 2 Sent Done Done false
33443 9949 Acked Read 7 Sent Done Done false
33444 9950 Acked Read 10 Sent Done Done false
33445 9951 Acked Write 2 Sent Done Done false
33446 9952 Acked Read 5 Sent Done Done false
33447 9953 Acked Write 10 Sent Done Done false
33448 9954 Acked Write 9 Sent Done Done false
33449 9955 Acked Read 9 Sent Done Done false
33450 9956 Acked Write 10 Sent Done Done false
33451 9957 Acked Read 7 Sent Done Done false
33452 9958 Acked Read 2 Sent Done Done false
33453 9959 Acked Read 6 Sent Done Done false
33454 9960 Acked Write 9 Sent Done Done false
33455 9961 Acked Write 10 Sent Done Done false
33456 9962 Acked Write 8 Sent Done Done false
33457 9963 Acked Write 3 Sent Done Done false
33458 9964 Acked Read 9 Sent Done Done false
33459 9965 Acked Read 10 Sent Done Done false
33460 9966 Acked Read 7 Sent Done Done false
33461 9967 Acked Write 6 Sent Done Done false
33462 9968 Acked Write 8 Sent Done Done false
33463 9969 Acked Write 1 Sent Done Done false
33464 9970 Acked Read 7 Sent Done Done false
33465 9971 Acked Read 2 Sent Done Done false
33466 9972 Acked Write 1 Sent Done Done false
33467 9973 Acked Flush 0 Sent Done Done false
33468 9974 Acked Read 7 Sent Done Done false
33469 9975 Acked Read 5 Sent Done Done false
33470 9976 Acked Read 10 Sent Done Done false
33471 9977 Acked Flush 0 Sent Done Done false
33472 9978 Acked Read 9 Sent Done Done false
33473 9979 Acked Read 4 Sent Done Done false
33474 9980 Acked Read 3 Sent Done Done false
33475 9981 Acked Read 1 Sent Done Done false
33476 9982 Acked Read 2 Sent Done Done false
33477 9983 Acked Write 8 Sent Done Done false
33478 9984 Acked Read 1 Sent Done Done false
33479 9985 Acked Write 7 Sent Done Done false
33480 9986 Acked Write 8 Sent Done Done false
33481 9987 Acked Read 7 Sent Done Done false
33482 9988 Acked Read 10 Sent Done Done false
33483 9989 Acked Read 3 Sent Done Done false
33484 9990 Acked Write 8 Sent Done Done false
33485 9991 Acked Read 9 Sent Done Done false
33486 9992 Acked Read 2 Sent Done Done false
33487 9993 Acked Write 2 Sent Done Done false
33488 9994 Acked Read 2 Sent Done Done false
33489 9995 Acked Read 8 Sent Done Done false
33490 9996 Acked Read 7 Sent Done Done false
33491 9997 Acked Read 10 Sent Done Done false
33492 9998 Acked Write 10 Sent Done Done false
33493 9999 Acked Write 5 Sent Done Done false
33494 10000 Acked Flush 0 Sent Done Done false
33495 10001 Acked Write 3 Sent Done Done false
33496 10002 Acked Flush 0 Sent Done Done false
33497 10003 Acked Write 9 Sent Done Done false
33498 10004 Acked Write 5 Sent Done Done false
33499 10005 Acked Write 10 Sent Done Done false
33500 10006 Acked Write 7 Sent Done Done false
33501 10007 Acked Write 9 Sent Done Done false
33502 10008 Acked Read 10 Sent Done Done false
33503 10009 Acked Write 3 Sent Done Done false
33504 10010 Acked Write 4 Sent Done Done false
33505 10011 Acked Write 6 Sent Done Done false
33506 10012 Acked Read 3 Sent Done Done false
33507 10013 Acked Read 3 Sent Done Done false
33508 10014 Acked Read 7 Sent Done Done false
33509 10015 Acked Flush 0 Sent Done Done false
33510 10016 Acked Read 8 Sent Done Done false
33511 10017 Acked Write 6 Sent Done Done false
33512 10018 Acked Read 2 Sent Done Done false
33513 10019 Acked Flush 0 Sent Done Done false
33514 10020 Acked Write 8 Sent Done Done false
33515 10021 Acked Read 10 Sent Done Done false
33516 10022 Acked Write 5 Sent Done Done false
33517 10023 Acked Write 10 Sent Done Done false
33518 10024 Acked Write 4 Sent Done Done false
33519 10025 Acked Write 5 Sent Done Done false
33520 10026 Acked Read 10 Sent Done Done false
33521 10027 Acked Write 7 Sent Done Done false
33522 10028 Acked Read 8 Sent Done Done false
33523 10029 Acked Write 4 Sent Done Done false
33524 10030 Acked Read 7 Sent Done Done false
33525 10031 Acked Write 6 Sent Done Done false
33526 10032 Acked Read 1 Sent Done Done false
33527 10033 Acked Write 4 Sent Done Done false
33528 10034 Acked Read 8 Sent Done Done false
33529 10035 Acked Read 1 Sent Done Done false
33530 10036 Acked Write 3 Sent Done Done false
33531 10037 Acked Write 7 Sent Done Done false
33532 10038 Acked Read 6 Sent Done Done false
33533 10039 Acked Read 1 Sent Done Done false
33534 10040 Acked Read 2 Sent Done Done false
33535 10041 Acked Write 1 Sent Done Done false
33536 10042 Acked Read 1 Sent Done Done false
33537 10043 Acked Read 4 Sent Done Done false
33538 10044 Acked Read 10 Sent Done Done false
33539 10045 Acked Write 6 Sent Done Done false
33540 10046 Acked Read 3 Sent Done Done false
33541 10047 Acked Write 3 Sent Done Done false
33542 10048 Acked Read 4 Sent Done Done false
33543 10049 Acked Write 7 Sent Done Done false
33544 10050 Acked Flush 0 Sent Done Done false
33545 10051 Acked Write 2 Sent Done Done false
33546 10052 Acked Read 10 Sent Done Done false
33547 10053 Acked Flush 0 Sent Done Done false
33548 10054 Acked Flush 0 Sent Done Done false
33549 10055 Acked Read 4 Sent Done Done false
33550 10056 Acked Read 1 Sent Done Done false
33551 10057 Acked Write 8 Sent Done Done false
33552 STATES DS:0 DS:1 DS:2 TOTAL
33553 Sent 301 0 0 301
33554 Done 0 301 301 602
33555 Skipped 0 0 0 0
33556 Error 0 0 0 0
33557 Last Flush: 9756 10054 10054
33558 Downstairs last five completed: 9756 9755 9754 9753 9752
33559 Upstairs last five completed: 10057 10056 10055 10054 10053
33560 CLIENT: Up:0 ds:301 act:2
33561 [8810] got start action
33562 [8810] I am going to restart
33563 Starting downstairs at port 8810
33564 Make output file at "/tmp/dsc/downstairs-8810.txt"
33565 [8810][0] reports Starting
33566 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1231)
33567 [8810][0] reports Running
33568 ----------------------------------------------------------------
33569 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
33570 States: Offline Active Active
33571 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33572 9757 Acked Read 4 Sent Done Done false
33573 9758 Acked Read 1 Sent Done Done false
33574 9759 Acked Read 3 Sent Done Done false
33575 9760 Acked Read 6 Sent Done Done false
33576 9761 Acked Read 4 Sent Done Done false
33577 9762 Acked Read 9 Sent Done Done false
33578 9763 Acked Write 2 Sent Done Done false
33579 9764 Acked Read 3 Sent Done Done false
33580 9765 Acked Write 7 Sent Done Done false
33581 9766 Acked Write 10 Sent Done Done false
33582 9767 Acked Write 2 Sent Done Done false
33583 9768 Acked Read 7 Sent Done Done false
33584 9769 Acked Read 7 Sent Done Done false
33585 9770 Acked Write 6 Sent Done Done false
33586 9771 Acked Read 8 Sent Done Done false
33587 9772 Acked Write 8 Sent Done Done false
33588 9773 Acked Read 7 Sent Done Done false
33589 9774 Acked Read 8 Sent Done Done false
33590 9775 Acked Write 5 Sent Done Done false
33591 9776 Acked Read 7 Sent Done Done false
33592 9777 Acked Write 3 Sent Done Done false
33593 9778 Acked Write 2 Sent Done Done false
33594 9779 Acked Flush 0 Sent Done Done false
33595 9780 Acked Read 5 Sent Done Done false
33596 9781 Acked Flush 0 Sent Done Done false
33597 9782 Acked Read 1 Sent Done Done false
33598 9783 Acked Write 5 Sent Done Done false
33599 9784 Acked Read 4 Sent Done Done false
33600 9785 Acked Write 4 Sent Done Done false
33601 9786 Acked Flush 0 Sent Done Done false
33602 9787 Acked Read 8 Sent Done Done false
33603 9788 Acked Write 5 Sent Done Done false
33604 9789 Acked Read 8 Sent Done Done false
33605 9790 Acked Write 2 Sent Done Done false
33606 9791 Acked Flush 0 Sent Done Done false
33607 9792 Acked Read 1 Sent Done Done false
33608 9793 Acked Read 6 Sent Done Done false
33609 9794 Acked Read 6 Sent Done Done false
33610 9795 Acked Write 5 Sent Done Done false
33611 9796 Acked Read 2 Sent Done Done false
33612 9797 Acked Read 4 Sent Done Done false
33613 9798 Acked Flush 0 Sent Done Done false
33614 9799 Acked Read 3 Sent Done Done false
33615 9800 Acked Read 5 Sent Done Done false
33616 9801 Acked Read 1 Sent Done Done false
33617 9802 Acked Write 1 Sent Done Done false
33618 9803 Acked Write 3 Sent Done Done false
33619 9804 Acked Read 9 Sent Done Done false
33620 9805 Acked Read 9 Sent Done Done false
33621 9806 Acked Read 8 Sent Done Done false
33622 9807 Acked Read 3 Sent Done Done false
33623 9808 Acked Write 1 Sent Done Done false
33624 9809 Acked Read 7 Sent Done Done false
33625 9810 Acked Flush 0 Sent Done Done false
33626 9811 Acked Write 4 Sent Done Done false
33627 9812 Acked Read 10 Sent Done Done false
33628 9813 Acked Write 9 Sent Done Done false
33629 9814 Acked Read 4 Sent Done Done false
33630 9815 Acked Write 10 Sent Done Done false
33631 9816 Acked Write 9 Sent Done Done false
33632 9817 Acked Write 8 Sent Done Done false
33633 9818 Acked Read 1 Sent Done Done false
33634 9819 Acked Read 3 Sent Done Done false
33635 9820 Acked Read 7 Sent Done Done false
33636 9821 Acked Read 5 Sent Done Done false
33637 9822 Acked Read 5 Sent Done Done false
33638 9823 Acked Write 8 Sent Done Done false
33639 9824 Acked Flush 0 Sent Done Done false
33640 9825 Acked Read 5 Sent Done Done false
33641 9826 Acked Read 10 Sent Done Done false
33642 9827 Acked Read 7 Sent Done Done false
33643 9828 Acked Write 9 Sent Done Done false
33644 9829 Acked Read 2 Sent Done Done false
33645 9830 Acked Write 1 Sent Done Done false
33646 9831 Acked Read 7 Sent Done Done false
33647 9832 Acked Write 1 Sent Done Done false
33648 9833 Acked Read 2 Sent Done Done false
33649 9834 Acked Write 5 Sent Done Done false
33650 9835 Acked Read 1 Sent Done Done false
33651 9836 Acked Read 2 Sent Done Done false
33652 9837 Acked Write 1 Sent Done Done false
33653 9838 Acked Read 3 Sent Done Done false
33654 9839 Acked Read 4 Sent Done Done false
33655 9840 Acked Read 5 Sent Done Done false
33656 9841 Acked Read 3 Sent Done Done false
33657 9842 Acked Write 1 Sent Done Done false
33658 9843 Acked Write 2 Sent Done Done false
33659 9844 Acked Read 4 Sent Done Done false
33660 9845 Acked Read 8 Sent Done Done false
33661 9846 Acked Read 3 Sent Done Done false
33662 9847 Acked Flush 0 Sent Done Done false
33663 9848 Acked Read 9 Sent Done Done false
33664 9849 Acked Flush 0 Sent Done Done false
33665 9850 Acked Write 10 Sent Done Done false
33666 9851 Acked Write 9 Sent Done Done false
33667 9852 Acked Write 6 Sent Done Done false
33668 9853 Acked Write 4 Sent Done Done false
33669 9854 Acked Read 8 Sent Done Done false
33670 9855 Acked Write 6 Sent Done Done false
33671 9856 Acked Write 6 Sent Done Done false
33672 9857 Acked Read 10 Sent Done Done false
33673 9858 Acked Read 9 Sent Done Done false
33674 9859 Acked Read 7 Sent Done Done false
33675 9860 Acked Read 3 Sent Done Done false
33676 9861 Acked Write 2 Sent Done Done false
33677 9862 Acked Read 1 Sent Done Done false
33678 9863 Acked Write 5 Sent Done Done false
33679 9864 Acked Write 5 Sent Done Done false
33680 9865 Acked Write 8 Sent Done Done false
33681 9866 Acked Read 6 Sent Done Done false
33682 9867 Acked Read 5 Sent Done Done false
33683 9868 Acked Read 5 Sent Done Done false
33684 9869 Acked Read 7 Sent Done Done false
33685 9870 Acked Flush 0 Sent Done Done false
33686 9871 Acked Write 2 Sent Done Done false
33687 9872 Acked Write 10 Sent Done Done false
33688 9873 Acked Flush 0 Sent Done Done false
33689 9874 Acked Write 7 Sent Done Done false
33690 9875 Acked Read 1 Sent Done Done false
33691 9876 Acked Read 4 Sent Done Done false
33692 9877 Acked Read 3 Sent Done Done false
33693 9878 Acked Read 9 Sent Done Done false
33694 9879 Acked Read 3 Sent Done Done false
33695 9880 Acked Write 7 Sent Done Done false
33696 9881 Acked Write 1 Sent Done Done false
33697 9882 Acked Read 7 Sent Done Done false
33698 9883 Acked Write 5 Sent Done Done false
33699 9884 Acked Write 5 Sent Done Done false
33700 9885 Acked Write 1 Sent Done Done false
33701 9886 Acked Write 7 Sent Done Done false
33702 9887 Acked Write 4 Sent Done Done false
33703 9888 Acked Write 9 Sent Done Done false
33704 9889 Acked Flush 0 Sent Done Done false
33705 9890 Acked Write 7 Sent Done Done false
33706 9891 Acked Read 4 Sent Done Done false
33707 9892 Acked Read 9 Sent Done Done false
33708 9893 Acked Read 7 Sent Done Done false
33709 9894 Acked Read 4 Sent Done Done false
33710 9895 Acked Read 8 Sent Done Done false
33711 9896 Acked Read 1 Sent Done Done false
33712 9897 Acked Write 10 Sent Done Done false
33713 9898 Acked Read 8 Sent Done Done false
33714 9899 Acked Read 2 Sent Done Done false
33715 9900 Acked Flush 0 Sent Done Done false
33716 9901 Acked Read 8 Sent Done Done false
33717 9902 Acked Read 10 Sent Done Done false
33718 9903 Acked Flush 0 Sent Done Done false
33719 9904 Acked Write 9 Sent Done Done false
33720 9905 Acked Write 8 Sent Done Done false
33721 9906 Acked Read 4 Sent Done Done false
33722 9907 Acked Write 1 Sent Done Done false
33723 9908 Acked Read 8 Sent Done Done false
33724 9909 Acked Read 1 Sent Done Done false
33725 9910 Acked Read 10 Sent Done Done false
33726 9911 Acked Read 5 Sent Done Done false
33727 9912 Acked Read 7 Sent Done Done false
33728 9913 Acked Flush 0 Sent Done Done false
33729 9914 Acked Read 3 Sent Done Done false
33730 9915 Acked Read 8 Sent Done Done false
33731 9916 Acked Read 1 Sent Done Done false
33732 9917 Acked Flush 0 Sent Done Done false
33733 9918 Acked Read 1 Sent Done Done false
33734 9919 Acked Write 5 Sent Done Done false
33735 9920 Acked Write 5 Sent Done Done false
33736 9921 Acked Read 10 Sent Done Done false
33737 9922 Acked Read 2 Sent Done Done false
33738 9923 Acked Write 1 Sent Done Done false
33739 9924 Acked Write 10 Sent Done Done false
33740 9925 Acked Read 4 Sent Done Done false
33741 9926 Acked Write 5 Sent Done Done false
33742 9927 Acked Read 9 Sent Done Done false
33743 9928 Acked Read 1 Sent Done Done false
33744 9929 Acked Read 6 Sent Done Done false
33745 9930 Acked Flush 0 Sent Done Done false
33746 9931 Acked Write 8 Sent Done Done false
33747 9932 Acked Flush 0 Sent Done Done false
33748 9933 Acked Write 9 Sent Done Done false
33749 9934 Acked Write 3 Sent Done Done false
33750 9935 Acked Write 4 Sent Done Done false
33751 9936 Acked Read 9 Sent Done Done false
33752 9937 Acked Flush 0 Sent Done Done false
33753 9938 Acked Read 4 Sent Done Done false
33754 9939 Acked Read 1 Sent Done Done false
33755 9940 Acked Read 6 Sent Done Done false
33756 9941 Acked Read 1 Sent Done Done false
33757 9942 Acked Read 10 Sent Done Done false
33758 9943 Acked Read 4 Sent Done Done false
33759 9944 Acked Read 3 Sent Done Done false
33760 9945 Acked Read 2 Sent Done Done false
33761 9946 Acked Flush 0 Sent Done Done false
33762 9947 Acked Read 6 Sent Done Done false
33763 9948 Acked Write 2 Sent Done Done false
33764 9949 Acked Read 7 Sent Done Done false
33765 9950 Acked Read 10 Sent Done Done false
33766 9951 Acked Write 2 Sent Done Done false
33767 9952 Acked Read 5 Sent Done Done false
33768 9953 Acked Write 10 Sent Done Done false
33769 9954 Acked Write 9 Sent Done Done false
33770 9955 Acked Read 9 Sent Done Done false
33771 9956 Acked Write 10 Sent Done Done false
33772 9957 Acked Read 7 Sent Done Done false
33773 9958 Acked Read 2 Sent Done Done false
33774 9959 Acked Read 6 Sent Done Done false
33775 9960 Acked Write 9 Sent Done Done false
33776 9961 Acked Write 10 Sent Done Done false
33777 9962 Acked Write 8 Sent Done Done false
33778 9963 Acked Write 3 Sent Done Done false
33779 9964 Acked Read 9 Sent Done Done false
33780 9965 Acked Read 10 Sent Done Done false
33781 9966 Acked Read 7 Sent Done Done false
33782 9967 Acked Write 6 Sent Done Done false
33783 9968 Acked Write 8 Sent Done Done false
33784 9969 Acked Write 1 Sent Done Done false
33785 9970 Acked Read 7 Sent Done Done false
33786 9971 Acked Read 2 Sent Done Done false
33787 9972 Acked Write 1 Sent Done Done false
33788 9973 Acked Flush 0 Sent Done Done false
33789 9974 Acked Read 7 Sent Done Done false
33790 9975 Acked Read 5 Sent Done Done false
33791 9976 Acked Read 10 Sent Done Done false
33792 9977 Acked Flush 0 Sent Done Done false
33793 9978 Acked Read 9 Sent Done Done false
33794 9979 Acked Read 4 Sent Done Done false
33795 9980 Acked Read 3 Sent Done Done false
33796 9981 Acked Read 1 Sent Done Done false
33797 9982 Acked Read 2 Sent Done Done false
33798 9983 Acked Write 8 Sent Done Done false
33799 9984 Acked Read 1 Sent Done Done false
33800 9985 Acked Write 7 Sent Done Done false
33801 9986 Acked Write 8 Sent Done Done false
33802 9987 Acked Read 7 Sent Done Done false
33803 9988 Acked Read 10 Sent Done Done false
33804 9989 Acked Read 3 Sent Done Done false
33805 9990 Acked Write 8 Sent Done Done false
33806 9991 Acked Read 9 Sent Done Done false
33807 9992 Acked Read 2 Sent Done Done false
33808 9993 Acked Write 2 Sent Done Done false
33809 9994 Acked Read 2 Sent Done Done false
33810 9995 Acked Read 8 Sent Done Done false
33811 9996 Acked Read 7 Sent Done Done false
33812 9997 Acked Read 10 Sent Done Done false
33813 9998 Acked Write 10 Sent Done Done false
33814 9999 Acked Write 5 Sent Done Done false
33815 10000 Acked Flush 0 Sent Done Done false
33816 10001 Acked Write 3 Sent Done Done false
33817 10002 Acked Flush 0 Sent Done Done false
33818 10003 Acked Write 9 Sent Done Done false
33819 10004 Acked Write 5 Sent Done Done false
33820 10005 Acked Write 10 Sent Done Done false
33821 10006 Acked Write 7 Sent Done Done false
33822 10007 Acked Write 9 Sent Done Done false
33823 10008 Acked Read 10 Sent Done Done false
33824 10009 Acked Write 3 Sent Done Done false
33825 10010 Acked Write 4 Sent Done Done false
33826 10011 Acked Write 6 Sent Done Done false
33827 10012 Acked Read 3 Sent Done Done false
33828 10013 Acked Read 3 Sent Done Done false
33829 10014 Acked Read 7 Sent Done Done false
33830 10015 Acked Flush 0 Sent Done Done false
33831 10016 Acked Read 8 Sent Done Done false
33832 10017 Acked Write 6 Sent Done Done false
33833 10018 Acked Read 2 Sent Done Done false
33834 10019 Acked Flush 0 Sent Done Done false
33835 10020 Acked Write 8 Sent Done Done false
33836 10021 Acked Read 10 Sent Done Done false
33837 10022 Acked Write 5 Sent Done Done false
33838 10023 Acked Write 10 Sent Done Done false
33839 10024 Acked Write 4 Sent Done Done false
33840 10025 Acked Write 5 Sent Done Done false
33841 10026 Acked Read 10 Sent Done Done false
33842 10027 Acked Write 7 Sent Done Done false
33843 10028 Acked Read 8 Sent Done Done false
33844 10029 Acked Write 4 Sent Done Done false
33845 10030 Acked Read 7 Sent Done Done false
33846 10031 Acked Write 6 Sent Done Done false
33847 10032 Acked Read 1 Sent Done Done false
33848 10033 Acked Write 4 Sent Done Done false
33849 10034 Acked Read 8 Sent Done Done false
33850 10035 Acked Read 1 Sent Done Done false
33851 10036 Acked Write 3 Sent Done Done false
33852 10037 Acked Write 7 Sent Done Done false
33853 10038 Acked Read 6 Sent Done Done false
33854 10039 Acked Read 1 Sent Done Done false
33855 10040 Acked Read 2 Sent Done Done false
33856 10041 Acked Write 1 Sent Done Done false
33857 10042 Acked Read 1 Sent Done Done false
33858 10043 Acked Read 4 Sent Done Done false
33859 10044 Acked Read 10 Sent Done Done false
33860 10045 Acked Write 6 Sent Done Done false
33861 10046 Acked Read 3 Sent Done Done false
33862 10047 Acked Write 3 Sent Done Done false
33863 10048 Acked Read 4 Sent Done Done false
33864 10049 Acked Write 7 Sent Done Done false
33865 10050 Acked Flush 0 Sent Done Done false
33866 10051 Acked Write 2 Sent Done Done false
33867 10052 Acked Read 10 Sent Done Done false
33868 10053 Acked Flush 0 Sent Done Done false
33869 10054 Acked Flush 0 Sent Done Done false
33870 10055 Acked Read 4 Sent Done Done false
33871 10056 Acked Read 1 Sent Done Done false
33872 10057 Acked Write 8 Sent Done Done false
33873 10058 Acked Flush 0 Sent Done Done false
33874 STATES DS:0 DS:1 DS:2 TOTAL
33875 Sent 302 0 0 302
33876 Done 0 302 302 604
33877 Skipped 0 0 0 0
33878 Error 0 0 0 0
33879 Last Flush: 9756 10058 10058
33880 Downstairs last five completed: 9756 9755 9754 9753 9752
33881 Upstairs last five completed: 10058 10057 10056 10055 10054
33882 CLIENT: Up:0 ds:302 act:2
338832024-12-09T15:14:56.304ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
338842024-12-09T15:14:56.304ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59251) = io task client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
338852024-12-09T15:14:56.305ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
338862024-12-09T15:14:56.306ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
338872024-12-09T15:14:56.306ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
338882024-12-09T15:14:56.306ZINFOcrucible: send last flush ID to this DS: 9756 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
338892024-12-09T15:14:56.306ZINFOcrucible: Replied this last flush ID: 9756 = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
338902024-12-09T15:14:56.306ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
338912024-12-09T15:14:56.306ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
338922024-12-09T15:14:56.306ZINFOcrucible: [0] client re-new 302 jobs since flush 9756 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
338932024-12-09T15:14:56.306ZINFOcrucible: [0] Replayed 302 jobs since flush: 9756 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
33894 ----------------------------------------------------------------
33895 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33896 Downstairs last five completed: 10058 10057 10056 10055 10054
33897 Upstairs last five completed: 10058 10057 10056 10055 10054
33898 CLIENT: Up:0 ds:0 act:3
33899 Replay: All jobs finished
33900 Main task has work to do, go find it
33901 got dsc Stop(2)
33902 stop 2
33903 [8830] Got stop action so:false kr:false
33904 Dec 09 15:14:58.552 INFO request completed, latency_us: 374, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 8ed61bf0-a16d-4898-a19b-e318fe4aef4c, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
339052024-12-09T15:14:58.553ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
339062024-12-09T15:14:58.553ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
339072024-12-09T15:14:58.553ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
339082024-12-09T15:14:58.553ZINFOcrucible: ds_transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
339092024-12-09T15:14:58.553ZINFOcrucible: [2] Transition from Active to Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
33910 Dec 09 15:14:58.553 INFO request completed, latency_us: 122, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3c2663c9-da6b-4f2b-ad1d-6a52cbc58699, remote_addr:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-12-09T15:14:58.553723592Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"585553f3-c0fa-4eaa-8619-2b4b00d00492"}
33911 127.0.0.1:47510, local_addr: 127.0.0.1:9998
33912 [8830] Exited with: ExitStatus(unix_wait_status(9))
33913 [8830][2] reports Exit
33914 Dec 09 15:15:02.555 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/2, method: GET, req_id: dd32b15f-e237-4eff-bb38-fb09676022e3, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
33915 001/300 Write block 1655 len 28672 data: 5 5 4 4 4 5 7
33916 002/300 Read block 2479 len 40960
33917 003/300 Write block 258 len 8192 data: 6 5
33918 004/300 Read block 4313 len 16384
33919 005/300 Read block 3910 len 20480
33920 006/300 Write block 946 len 24576 data: 3 6 8 8 8 8
33921 007/300 Write block 318 len 12288 data: 8 8 8
33922 008/300 Read block 1516 len 36864
33923 009/300 Write block 2928 len 40960 data: 9 8 10 12 12 12 12 13 11 11
33924 010/300 Read block 3911 len 32768
33925 011/300 Write block 3738 len 12288 data: 4 4 3
33926 012/300 Write block 1767 len 20480 data: 5 5 6 3 3
33927 013/300 Write block 4926 len 24576 data: 3 5 5 5 7 6
33928 014/300 Flush
33929 015/300 Flush
33930 016/300 Read block 2977 len 16384
33931 017/300 Flush
33932 018/300 Write block 2831 len 12288 data: 7 7 5
33933 019/300 Write block 4410 len 36864 data: 7 9 8 9 5 5 5 5 4
33934 020/300 Read block 1354 len 16384
33935 021/300 Read block 1276 len 8192
33936 022/300 Read block 1498 len 4096
33937 023/300 Write block 2136 len 24576 data: 10 10 11 8 7 8
33938 024/300 Write block 3719 len 28672 data: 3 6 7 6 8 9 6
33939 025/300 Read block 4287 len 32768
33940 026/300 Read block 2193 len 36864
33941 027/300 Flush
33942 028/300 Write block 4643 len 24576 data: 7 8 9 10 11 11
33943 029/300 Read block 801 len 8192
33944 030/300 Read block 366 len 4096
33945 031/300 Read block 1570 len 20480
33946 032/300 Write block 1676 len 32768 data: 3 3 4 4 4 4 4 5
33947 033/300 Read block 482 len 28672
33948 034/300 Flush
33949 035/300 Flush
33950 036/300 Read block 4508 len 12288
33951 037/300 Read block 4841 len 36864
33952 038/300 Read block 3626 len 40960
33953 039/300 Read block 2950 len 32768
33954 040/300 Read block 4058 len 32768
33955 041/300 Read block 3764 len 24576
33956 042/300 Read block 2120 len 8192
33957 043/300 Write block 79 len 28672 data: 6 6 6 7 7 6 4
33958 044/300 Write block 395 len 8192 data: 6 5
33959 045/300 Write block 915 len 8192 data: 7 7
33960 046/300 Read block 490 len 16384
33961 047/300 Read block 1800 len 20480
33962 048/300 Write block 3497 len 24576 data: 4 3 4 4 4 3
33963 049/300 Write block 3550 len 8192 data: 8 7
33964 050/300 Read block 104 len 24576
33965 051/300 Read block 189 len 8192
33966 052/300 Read block 1859 len 32768
33967 053/300 Write block 4705 len 40960 data: 7 8 9 10 9 7 7 6 7 7
33968 054/300 Write block 1020 len 12288 data: 3 3 3
33969 055/300 Write block 1742 len 8192 data: 6 8
33970 056/300 Write block 3819 len 32768 data: 7 6 7 7 8 10 9 9
33971 057/300 Read block 1353 len 4096
33972 058/300 Read block 110 len 12288
33973 059/300 Write block 2822 len 20480 data: 5 4 4 4 5
33974 060/300 Read block 479 len 28672
33975 061/300 Write block 2630 len 28672 data: 5 5 5 5 4 5 5
33976 062/300 Flush
33977 063/300 Read block 3954 len 40960
33978 064/300 Write block 4840 len 12288 data: 10 10 9
33979 065/300 Write block 2235 len 40960 data: 8 5 4 5 4 6 10 8 8 8
33980 066/300 Write block 1053 len 40960 data: 4 4 3 3 4 6 6 7 7 6
33981 067/300 Read block 4671 len 36864
33982 068/300 Read block 1250 len 4096
33983 069/300 Read block 3361 len 8192
33984 070/300 Read block 2497 len 12288
33985 071/300 Read block 1621 len 16384
33986 072/300 Write block 2323 len 4096 data: 4
33987 073/300 Flush
33988 074/300 Write block 1786 len 28672 data: 8 8 6 5 5 6 6
33989 075/300 Read block 410 len 36864
33990 076/300 Write block 2594 len 24576 data: 2 2 2 2 4 4
33991 077/300 Read block 341 len 16384
33992 078/300 Flush
33993 079/300 Write block 2063 len 12288 data: 5 6 4
33994 080/300 Flush
33995 081/300 Read block 2362 len 4096
33996 082/300 Read block 4740 len 20480
33997 083/300 Read block 3193 len 4096
33998 084/300 Read block 2747 len 8192
33999 085/300 Read block 579 len 28672
34000 086/300 Write block 4191 len 16384 data: 8 9 6 5
34001 087/300 Flush
34002 088/300 Write block 2577 len 20480 data: 5 6 7 6 8
34003 089/300 Flush
34004 090/300 Read block 79 len 12288
34005 091/300 Read block 3473 len 24576
34006 092/300 Read block 703 len 12288
34007 093/300 Read block 3659 len 16384
34008 094/300 Write block 2860 len 40960 data: 2 3 3 5 7 7 8 11 10 11
34009 095/300 Write block 1521 len 16384 data: 5 3 4 6
34010 096/300 Read block 2192 len 20480
34011 097/300 Write block 519 len 32768 data: 5 4 5 6 6 5 5 6
34012 098/300 Write block 2893 len 16384 data: 7 8 9 8
34013 099/300 Write block 221 len 24576 data: 6 8 9 8 7 6
34014 100/300 Read block 660 len 32768
34015 101/300 Read block 2911 len 32768
34016 102/300 Read block 2468 len 20480
34017 103/300 Read block 3283 len 40960
34018 104/300 Write block 1466 len 20480 data: 4 5 7 7 9
34019 105/300 Write block 299 len 40960 data: 7 8 6 6 6 6 6 6 5 4
34020 106/300 Read block 2050 len 16384
34021 107/300 Read block 191 len 4096
34022 108/300 Read block 2592 len 24576
34023 109/300 Write block 1329 len 12288 data: 5 4 5
34024 110/300 Write block 2716 len 32768 data: 8 9 7 7 7 5 5 6
34025 111/300 Write block 321 len 16384 data: 8 12 9 9
34026 112/300 Flush
34027 113/300 Write block 3554 len 32768 data: 8 8 9 7 9 8 6 9
34028 114/300 Write block 4327 len 20480 data: 6 6 5 4 3
34029 115/300 Read block 576 len 16384
34030 116/300 Write block 4299 len 40960 data: 9 9 7 8 8 9 9 6 6 5
34031 117/300 Read block 1726 len 16384
34032 118/300 Read block 4254 len 16384
34033 119/300 Read block 2858 len 40960
34034 120/300 Write block 1065 len 36864 data: 4 3 3 4 6 6 6 6 6
34035 121/300 Read block 3649 len 24576
34036 122/300 Flush
34037 123/300 Read block 2081 len 8192
34038 124/300 Read block 4222 len 40960
34039 125/300 Flush
34040 126/300 Write block 3458 len 36864 data: 4 5 6 6 6 8 9 7 8
34041 127/300 Write block 3399 len 8192 data: 9 8
34042 128/300 Read block 314 len 4096
34043 129/300 Write block 1968 len 8192 data: 7 9
34044 130/300 Flush
34045 131/300 Write block 752 len 4096 data: 5
34046 132/300 Read block 33 len 36864
34047 133/300 Read block 4121 len 20480
34048 134/300 Read block 3833 len 8192
34049 135/300 Flush
34050 136/300 Flush
34051 137/300 Read block 2393 len 32768
34052 138/300 Write block 2353 len 28672 data: 6 7 7 7 8 8 5
34053 139/300 Read block 2213 len 20480
34054 140/300 Read block 3081 len 24576
34055 141/300 Read block 104 len 24576
34056 142/300 Write block 2820 len 32768 data: 6 6 6 5 5 5 6 4
34057 143/300 Flush
34058 144/300 Write block 3291 len 16384 data: 9 9 7 7
34059 145/300 Flush
34060 146/300 Read block 56 len 24576
34061 147/300 Flush
34062 148/300 Read block 2326 len 32768
34063 149/300 Read block 1472 len 8192
34064 150/300 Read block 2957 len 8192
34065 151/300 Flush
34066 152/300 Read block 564 len 8192
34067 153/300 Read block 565 len 4096
34068 154/300 Read block 3031 len 28672
34069 155/300 Write block 3037 len 4096 data: 3
34070 156/300 Read block 2727 len 16384
34071 157/300 Flush
34072 158/300 Read block 389 len 16384
34073 159/300 Flush
34074 160/300 Write block 3911 len 4096 data: 6
34075 161/300 Read block 4600 len 32768
34076 162/300 Write block 711 len 36864 data: 4 3 4 4 4 5 6 6 7
34077 163/300 Read block 3656 len 32768
34078 164/300 Read block 4071 len 4096
34079 165/300 Write block 4208 len 8192 data: 3 3
34080 166/300 Write block 3444 len 24576 data: 8 7 7 8 8 6
34081 167/300 Read block 3984 len 28672
34082 168/300 Write block 2820 len 20480 data: 7 7 7 6 6
34083 169/300 Read block 99 len 16384
34084 170/300 Flush
34085 171/300 Flush
34086 172/300 Write block 3581 len 36864 data: 10 10 12 12 9 10 8 7 5
34087 173/300 Read block 2147 len 24576
34088 174/300 Read block 30 len 32768
34089 175/300 Read block 4367 len 24576
34090 176/300 Read block 1785 len 40960
34091 177/300 Write block 1269 len 24576 data: 6 6 8 9 9 9
34092 178/300 Read block 1837 len 20480
34093 179/300 Flush
34094 180/300 Write block 974 len 12288 data: 6 7 6
34095 181/300 Read block 3719 len 36864
34096 182/300 Write block 388 len 16384 data: 3 5 6 4
34097 183/300 Write block 4115 len 40960 data: 5 7 5 4 5 6 5 6 6 8
34098 184/300 Read block 4895 len 32768
34099 185/300 Write block 4381 len 28672 data: 9 7 8 7 6 6 7
34100 186/300 Read block 278 len 40960
34101 187/300 Read block 1083 len 20480
34102 188/300 Read block 189 len 40960
34103 189/300 Write block 4093 len 8192 data: 6 8
34104 190/300 Read block 3819 len 4096
34105 191/300 Read block 1653 len 4096
34106 192/300 Read block 992 len 8192
34107 193/300 Read block 2255 len 20480
34108 194/300 Write block 4611 len 32768 data: 3 4 4 6 6 7 6 5
34109 195/300 Read block 3772 len 40960
34110 196/300 Read block 3253 len 24576
34111 197/300 Read block 978 len 24576
34112 198/300 Read block 2992 len 16384
34113 199/300 Read block 873 len 24576
34114 200/300 Write block 3434 len 40960 data: 7 6 4 4 4 5 7 7 7 8
34115 201/300 Read block 2694 len 16384
34116 202/300 Write block 267 len 32768 data: 6 8 9 10 10 9 6 5
34117 203/300 Write block 1154 len 32768 data: 4 2 3 3 3 4 4 5
34118 204/300 Read block 158 len 36864
34119 205/300 Read block 1815 len 8192
34120 206/300 Read block 2551 len 28672
34121 207/300 Write block 4063 len 24576 data: 2 2 2 3 3 6
34122 208/300 Flush
34123 209/300 Read block 4793 len 40960
34124 210/300 Write block 353 len 24576 data: 5 5 3 4 8 8
34125 211/300 Read block 3266 len 12288
34126 212/300 Write block 404 len 32768 data: 11 10 8 6 5 5 6 3
34127 213/300 Flush
34128 214/300 Read block 91 len 8192
34129 215/300 Read block 830 len 12288
34130 216/300 Flush
34131 217/300 Flush
34132 218/300 Read block 4044 len 40960
34133 219/300 Read block 3942 len 32768
34134 220/300 Read block 4594 len 12288
34135 221/300 Read block 434 len 4096
34136 222/300 Write block 1794 len 32768 data: 8 11 10 11 9 7 6 5
34137 223/300 Write block 3624 len 12288 data: 5 4 4
34138 224/300 Write block 3533 len 32768 data: 7 7 8 7 8 6 5 6
34139 225/300 Read block 2108 len 32768
34140 226/300 Read block 1152 len 40960
34141 227/300 Write block 4986 len 36864 data: 7 11 11 12 11 13 11 10 8
34142 228/300 Write block 3994 len 24576 data: 6 6 4 4 3 5
34143 229/300 Write block 2077 len 28672 data: 7 4 7 9 8 9 7
34144 230/300 Write block 2499 len 20480 data: 5 5 5 5 6
34145 231/300 Read block 1160 len 28672
34146 232/300 Write block 2603 len 40960 data: 5 5 5 5 3 4 6 5 7 8
34147 233/300 Read block 2020 len 40960
34148 234/300 Read block 4451 len 28672
34149 235/300 Write block 3714 len 32768 data: 6 7 6 4 4 4 7 8
34150 236/300 Read block 699 len 40960
34151 237/300 Read block 4194 len 4096
34152 238/300 Read block 3567 len 32768
34153 239/300 Read block 1468 len 8192
34154 240/300 Read block 1247 len 20480
34155 241/300 Write block 4709 len 8192 data: 10 8
34156 242/300 Write block 4071 len 28672 data: 8 5 4 3 3 3 2
34157 243/300 Read block 3795 len 12288
34158 244/300 Flush
34159 245/300 Write block 3925 len 16384 data: 5 5 5 7
34160 246/300 Flush
34161 247/300 Write block 1587 len 28672 data: 4 3 3 4 4 5 4
34162 248/300 Read block 1249 len 40960
34163 249/300 Write block 1846 len 4096 data: 7
34164 250/300 Read block 4689 len 8192
34165 251/300 Read block 4456 len 8192
34166 252/300 Read block 4648 len 8192
34167 253/300 Read block 4605 len 4096
34168 254/300 Read block 929 len 20480
34169 255/300 Write block 259 len 36864 data: 6 6 6 5 4 4 6 6 7
34170 256/300 Write block 4970 len 32768 data: 4 5 4 4 4 3 3 3
34171 257/300 Read block 3142 len 8192
34172 258/300 Write block 1769 len 8192 data: 7 4
34173 259/300 Flush
34174 260/300 Write block 1821 len 12288 data: 7 9 8
34175 261/300 Read block 2218 len 32768
34176 262/300 Write block 4461 len 40960 data: 5 6 7 7 8 7 5 8 8 10
34177 263/300 Write block 2217 len 8192 data: 5 5
34178 264/300 Write block 3234 len 4096 data: 7
34179 265/300 Read block 823 len 8192
34180 266/300 Read block 206 len 4096
34181 267/300 Read block 119 len 24576
34182 268/300 Read block 622 len 8192
34183 269/300 Write block 2186 len 20480 data: 6 9 9 8 9
34184 270/300 Read block 3043 len 8192
34185 271/300 Read block 1105 len 20480
34186 272/300 Read block 3810 len 12288
34187 273/300 Write block 4173 len 8192 data: 7 5
34188 274/300 Write block 3754 len 20480 data: 9 11 11 10 10
34189 275/300 Read block 3598 len 28672
34190 276/300 Write block 745 len 12288 data: 5 5 5
34191 277/300 Write block 962 len 8192 data: 6 8
34192 278/300 Read block 4192 len 36864
34193 279/300 Write block 1134 len 8192 data: 7 6
34194 280/300 Read block 2701 len 20480
34195 281/300 Flush
34196 282/300 Read block 2222 len 20480
34197 283/300 Read block 1267 len 16384
34198 284/300 Read block 2936 len 32768
34199 285/300 Read block 762 len 40960
34200 286/300 Flush
34201 287/300 Write block 4143 len 4096 data: 6
34202 288/300 Flush
34203 289/300 Read block 4592 len 20480
34204 290/300 Read block 36 len 24576
34205 291/300 Flush
34206 292/300 Write block 1724 len 32768 data: 5 5 5 6 5 5 6 7
34207 293/300 Read block 3003 len 28672
34208 294/300 Read block 2246 len 16384
34209 295/300 Read block 1264 len 40960
34210 296/300 Write block 4173 len 40960 data: 8 6 4 5 5 4 6 7 7 7
34211 297/300 Read block 4983 len 36864
34212 298/300 Read block 3309 len 40960
34213 299/300 Write block 2580 len 28672 data: 7 9 6 6 6 6 6
34214 300/300 Flush
34215 301/300 Write block 1427 len 4096 data: 6
34216 Main task has work to do, go find it
34217 got dsc Start(2)
34218 start 2
34219 Dec 09 15:15:02.798 INFO request completed, latency_us: 252, response_code: 204, uri: /start/cid/2, method: POST, req_id: ce8611e5-ddd1-4ae0-bdd5-a5fe94c4a392, remote_addr: 127.0.0.1:47510, local_addr: 127.0.0.1:9998
34220 Replay: started 2, returned:Ok(())
34221 ----------------------------------------------------------------
34222 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34223 States: Active Active Offline
34224 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34225 10059 Acked Write 7 Done Done Sent false
34226 10060 Acked Read 10 Done Done Sent false
34227 10061 Acked Write 2 Done Done Sent false
34228 10062 Acked Read 4 Done Done Sent false
34229 10063 Acked Read 5 Done Done Sent false
34230 10064 Acked Write 6 Done Done Sent false
34231 10065 Acked Write 3 Done Done Sent false
34232 10066 Acked Read 9 Done Done Sent false
34233 10067 Acked Write 10 Done Done Sent false
34234 10068 Acked Read 8 Done Done Sent false
34235 10069 Acked Write 3 Done Done Sent false
34236 10070 Acked Write 5 Done Done Sent false
34237 10071 Acked Write 6 Done Done Sent false
34238 10072 Acked Flush 0 Done Done Sent false
34239 10073 Acked Flush 0 Done Done Sent false
34240 10074 Acked Read 4 Done Done Sent false
34241 10075 Acked Flush 0 Done Done Sent false
34242 10076 Acked Write 3 Done Done Sent false
34243 10077 Acked Write 9 Done Done Sent false
34244 10078 Acked Read 4 Done Done Sent false
34245 10079 Acked Read 2 Done Done Sent false
34246 10080 Acked Read 1 Done Done Sent false
34247 10081 Acked Write 6 Done Done Sent false
34248 10082 Acked Write 7 Done Done Sent false
34249 10083 Acked Read 8 Done Done Sent false
34250 10084 Acked Read 9 Done Done Sent false
34251 10085 Acked Flush 0 Done Done Sent false
34252 10086 Acked Write 6 Done Done Sent false
34253 10087 Acked Read 2 Done Done Sent false
34254 10088 Acked Read 1 Done Done Sent false
34255 10089 Acked Read 5 Done Done Sent false
34256 10090 Acked Write 8 Done Done Sent false
34257 10091 Acked Read 7 Done Done Sent false
34258 10092 Acked Flush 0 Done Done Sent false
34259 10093 Acked Flush 0 Done Done Sent false
34260 10094 Acked Read 3 Done Done Sent false
34261 10095 Acked Read 9 Done Done Sent false
34262 10096 Acked Read 10 Done Done Sent false
34263 10097 Acked Read 8 Done Done Sent false
34264 10098 Acked Read 8 Done Done Sent false
34265 10099 Acked Read 6 Done Done Sent false
34266 10100 Acked Read 2 Done Done Sent false
34267 10101 Acked Write 7 Done Done Sent false
34268 10102 Acked Write 2 Done Done Sent false
34269 10103 Acked Write 2 Done Done Sent false
34270 10104 Acked Read 4 Done Done Sent false
34271 10105 Acked Read 5 Done Done Sent false
34272 10106 Acked Write 6 Done Done Sent false
34273 10107 Acked Write 2 Done Done Sent false
34274 10108 Acked Read 6 Done Done Sent false
34275 10109 Acked Read 2 Done Done Sent false
34276 10110 Acked Read 8 Done Done Sent false
34277 10111 Acked Write 10 Done Done Sent false
34278 10112 Acked Write 3 Done Done Sent false
34279 10113 Acked Write 2 Done Done Sent false
34280 10114 Acked Write 8 Done Done Sent false
34281 10115 Acked Read 1 Done Done Sent false
34282 10116 Acked Read 3 Done Done Sent false
34283 10117 Acked Write 5 Done Done Sent false
34284 10118 Acked Read 7 Done Done Sent false
34285 10119 Acked Write 7 Done Done Sent false
34286 10120 Acked Flush 0 Done Done Sent false
34287 10121 Acked Read 10 Done Done Sent false
34288 10122 Acked Write 3 Done Done Sent false
34289 10123 Acked Write 10 Done Done Sent false
34290 10124 Acked Write 10 Done Done Sent false
34291 10125 Acked Read 9 Done Done Sent false
34292 10126 Acked Read 1 Done Done Sent false
34293 10127 Acked Read 2 Done Done Sent false
34294 10128 Acked Read 3 Done Done Sent false
34295 10129 Acked Read 4 Done Done Sent false
34296 10130 Acked Write 1 Done Done Sent false
34297 10131 Acked Flush 0 Done Done Sent false
34298 10132 Acked Write 7 Done Done Sent false
34299 10133 Acked Read 9 Done Done Sent false
34300 10134 Acked Write 6 Done Done Sent false
34301 10135 Acked Read 4 Done Done Sent false
34302 10136 Acked Flush 0 Done Done Sent false
34303 10137 Acked Write 3 Done Done Sent false
34304 10138 Acked Flush 0 Done Done Sent false
34305 10139 Acked Read 1 Done Done Sent false
34306 10140 Acked Read 5 Done Done Sent false
34307 10141 Acked Read 1 Done Done Sent false
34308 10142 Acked Read 2 Done Done Sent false
34309 10143 Acked Read 7 Done Done Sent false
34310 10144 Acked Write 4 Done Done Sent false
34311 10145 Acked Flush 0 Done Done Sent false
34312 10146 Acked Write 5 Done Done Sent false
34313 10147 Acked Flush 0 Done Done Sent false
34314 10148 Acked Read 3 Done Done Sent false
34315 10149 Acked Read 6 Done Done Sent false
34316 10150 Acked Read 3 Done Done Sent false
34317 10151 Acked Read 4 Done Done Sent false
34318 10152 Acked Write 10 Done Done Sent false
34319 10153 Acked Write 4 Done Done Sent false
34320 10154 Acked Read 5 Done Done Sent false
34321 10155 Acked Write 8 Done Done Sent false
34322 10156 Acked Write 4 Done Done Sent false
34323 10157 Acked Write 6 Done Done Sent false
34324 10158 Acked Read 8 Done Done Sent false
34325 10159 Acked Read 8 Done Done Sent false
34326 10160 Acked Read 5 Done Done Sent false
34327 10161 Acked Read 10 Done Done Sent false
34328 10162 Acked Write 5 Done Done Sent false
34329 10163 Acked Write 10 Done Done Sent false
34330 10164 Acked Read 4 Done Done Sent false
34331 10165 Acked Read 1 Done Done Sent false
34332 10166 Acked Read 6 Done Done Sent false
34333 10167 Acked Write 3 Done Done Sent false
34334 10168 Acked Write 8 Done Done Sent false
34335 10169 Acked Write 4 Done Done Sent false
34336 10170 Acked Flush 0 Done Done Sent false
34337 10171 Acked Write 8 Done Done Sent false
34338 10172 Acked Write 5 Done Done Sent false
34339 10173 Acked Read 4 Done Done Sent false
34340 10174 Acked Write 10 Done Done Sent false
34341 10175 Acked Read 4 Done Done Sent false
34342 10176 Acked Read 4 Done Done Sent false
34343 10177 Acked Read 10 Done Done Sent false
34344 10178 Acked Write 9 Done Done Sent false
34345 10179 Acked Read 6 Done Done Sent false
34346 10180 Acked Flush 0 Done Done Sent false
34347 10181 Acked Read 2 Done Done Sent false
34348 10182 Acked Read 10 Done Done Sent false
34349 10183 Acked Flush 0 Done Done Sent false
34350 10184 Acked Write 9 Done Done Sent false
34351 10185 Acked Write 2 Done Done Sent false
34352 10186 Acked Read 1 Done Done Sent false
34353 10187 Acked Write 2 Done Done Sent false
34354 10188 Acked Flush 0 Done Done Sent false
34355 10189 Acked Write 1 Done Done Sent false
34356 10190 Acked Read 9 Done Done Sent false
34357 10191 Acked Read 5 Done Done Sent false
34358 10192 Acked Read 2 Done Done Sent false
34359 10193 Acked Flush 0 Done Done Sent false
34360 10194 Acked Flush 0 Done Done Sent false
34361 10195 Acked Read 8 Done Done Sent false
34362 10196 Acked Write 7 Done Done Sent false
34363 10197 Acked Read 5 Done Done Sent false
34364 10198 Acked Read 6 Done Done Sent false
34365 10199 Acked Read 6 Done Done Sent false
34366 10200 Acked Write 8 Done Done Sent false
34367 10201 Acked Flush 0 Done Done Sent false
34368 10202 Acked Write 4 Done Done Sent false
34369 10203 Acked Flush 0 Done Done Sent false
34370 10204 Acked Read 6 Done Done Sent false
34371 10205 Acked Flush 0 Done Done Sent false
34372 10206 Acked Read 8 Done Done Sent false
34373 10207 Acked Read 2 Done Done Sent false
34374 10208 Acked Read 2 Done Done Sent false
34375 10209 Acked Flush 0 Done Done Sent false
34376 10210 Acked Read 2 Done Done Sent false
34377 10211 Acked Read 1 Done Done Sent false
34378 10212 Acked Read 7 Done Done Sent false
34379 10213 Acked Write 1 Done Done Sent false
34380 10214 Acked Read 4 Done Done Sent false
34381 10215 Acked Flush 0 Done Done Sent false
34382 10216 Acked Read 4 Done Done Sent false
34383 10217 Acked Flush 0 Done Done Sent false
34384 10218 Acked Write 1 Done Done Sent false
34385 10219 Acked Read 8 Done Done Sent false
34386 10220 Acked Write 9 Done Done Sent false
34387 10221 Acked Read 8 Done Done Sent false
34388 10222 Acked Read 1 Done Done Sent false
34389 10223 Acked Write 2 Done Done Sent false
34390 10224 Acked Write 6 Done Done Sent false
34391 10225 Acked Read 7 Done Done Sent false
34392 10226 Acked Write 5 Done Done Sent false
34393 10227 Acked Read 4 Done Done Sent false
34394 10228 Acked Flush 0 Done Done Sent false
34395 10229 Acked Flush 0 Done Done Sent false
34396 10230 Acked Write 9 Done Done Sent false
34397 10231 Acked Read 6 Done Done Sent false
34398 10232 Acked Read 8 Done Done Sent false
34399 10233 Acked Read 6 Done Done Sent false
34400 10234 Acked Read 10 Done Done Sent false
34401 10235 Acked Write 6 Done Done Sent false
34402 10236 Acked Read 5 Done Done Sent false
34403 10237 Acked Flush 0 Done Done Sent false
34404 10238 Acked Write 3 Done Done Sent false
34405 10239 Acked Read 9 Done Done Sent false
34406 10240 Acked Write 4 Done Done Sent false
34407 10241 Acked Write 10 Done Done Sent false
34408 10242 Acked Read 8 Done Done Sent false
34409 10243 Acked Write 7 Done Done Sent false
34410 10244 Acked Read 10 Done Done Sent false
34411 10245 Acked Read 5 Done Done Sent false
34412 10246 Acked Read 10 Done Done Sent false
34413 10247 Acked Write 2 Done Done Sent false
34414 10248 Acked Read 1 Done Done Sent false
34415 10249 Acked Read 1 Done Done Sent false
34416 10250 Acked Read 2 Done Done Sent false
34417 10251 Acked Read 5 Done Done Sent false
34418 10252 Acked Write 8 Done Done Sent false
34419 10253 Acked Read 10 Done Done Sent false
34420 10254 Acked Read 6 Done Done Sent false
34421 10255 Acked Read 6 Done Done Sent false
34422 10256 Acked Read 4 Done Done Sent false
34423 10257 Acked Read 6 Done Done Sent false
34424 10258 Acked Write 10 Done Done Sent false
34425 10259 Acked Read 4 Done Done Sent false
34426 10260 Acked Write 8 Done Done Sent false
34427 10261 Acked Write 8 Done Done Sent false
34428 10262 Acked Read 9 Done Done Sent false
34429 10263 Acked Read 2 Done Done Sent false
34430 10264 Acked Read 7 Done Done Sent false
34431 10265 Acked Write 6 Done Done Sent false
34432 10266 Acked Flush 0 Done Done Sent false
34433 10267 Acked Read 10 Done Done Sent false
34434 10268 Acked Write 6 Done Done Sent false
34435 10269 Acked Read 3 Done Done Sent false
34436 10270 Acked Write 8 Done Done Sent false
34437 10271 Acked Flush 0 Done Done Sent false
34438 10272 Acked Read 2 Done Done Sent false
34439 10273 Acked Read 3 Done Done Sent false
34440 10274 Acked Flush 0 Done Done Sent false
34441 10275 Acked Flush 0 Done Done Sent false
34442 10276 Acked Read 10 Done Done Sent false
34443 10277 Acked Read 8 Done Done Sent false
34444 10278 Acked Read 3 Done Done Sent false
34445 10279 Acked Read 1 Done Done Sent false
34446 10280 Acked Write 8 Done Done Sent false
34447 10281 Acked Write 3 Done Done Sent false
34448 10282 Acked Write 8 Done Done Sent false
34449 10283 Acked Read 8 Done Done Sent false
34450 10284 Acked Read 10 Done Done Sent false
34451 10285 Acked Write 9 Done Done Sent false
34452 10286 Acked Write 6 Done Done Sent false
34453 10287 Acked Write 7 Done Done Sent false
34454 10288 Acked Write 5 Done Done Sent false
34455 10289 Acked Read 7 Done Done Sent false
34456 10290 Acked Write 10 Done Done Sent false
34457 10291 Acked Read 10 Done Done Sent false
34458 10292 Acked Read 7 Done Done Sent false
34459 10293 Acked Write 8 Done Done Sent false
34460 10294 Acked Read 10 Done Done Sent false
34461 10295 Acked Read 1 Done Done Sent false
34462 10296 Acked Read 8 Done Done Sent false
34463 10297 Acked Read 2 Done Done Sent false
34464 10298 Acked Read 5 Done Done Sent false
34465 10299 Acked Write 2 Done Done Sent false
34466 10300 Acked Write 7 Done Done Sent false
34467 10301 Acked Read 3 Done Done Sent false
34468 10302 Acked Flush 0 Done Done Sent false
34469 10303 Acked Write 4 Done Done Sent false
34470 10304 Acked Flush 0 Done Done Sent false
34471 10305 Acked Write 7 Done Done Sent false
34472 10306 Acked Read 10 Done Done Sent false
34473 10307 Acked Write 1 Done Done Sent false
34474 10308 Acked Read 2 Done Done Sent false
34475 10309 Acked Read 2 Done Done Sent false
34476 10310 Acked Read 2 Done Done Sent false
34477 10311 Acked Read 1 Done Done Sent false
34478 10312 Acked Read 5 Done Done Sent false
34479 10313 Acked Write 9 Done Done Sent false
34480 10314 Acked Write 8 Done Done Sent false
34481 10315 Acked Read 2 Done Done Sent false
34482 10316 Acked Write 2 Done Done Sent false
34483 10317 Acked Flush 0 Done Done Sent false
34484 10318 Acked Write 3 Done Done Sent false
34485 10319 Acked Read 8 Done Done Sent false
34486 10320 Acked Write 10 Done Done Sent false
34487 10321 Acked Write 2 Done Done Sent false
34488 10322 Acked Write 1 Done Done Sent false
34489 10323 Acked Read 2 Done Done Sent false
34490 10324 Acked Read 1 Done Done Sent false
34491 10325 Acked Read 6 Done Done Sent false
34492 10326 Acked Read 2 Done Done Sent false
34493 10327 Acked Write 5 Done Done Sent false
34494 10328 Acked Read 2 Done Done Sent false
34495 10329 Acked Read 5 Done Done Sent false
34496 10330 Acked Read 3 Done Done Sent false
34497 10331 Acked Write 2 Done Done Sent false
34498 10332 Acked Write 5 Done Done Sent false
34499 10333 Acked Read 7 Done Done Sent false
34500 10334 Acked Write 3 Done Done Sent false
34501 10335 Acked Write 2 Done Done Sent false
34502 10336 Acked Read 9 Done Done Sent false
34503 10337 Acked Write 2 Done Done Sent false
34504 10338 Acked Read 5 Done Done Sent false
34505 10339 Acked Flush 0 Done Done Sent false
34506 10340 Acked Read 5 Done Done Sent false
34507 10341 Acked Read 4 Done Done Sent false
34508 10342 Acked Read 8 Done Done Sent false
34509 10343 Acked Read 10 Done Done Sent false
34510 10344 Acked Flush 0 Done Done Sent false
34511 10345 Acked Write 1 Done Done Sent false
34512 10346 Acked Flush 0 Done Done Sent false
34513 10347 Acked Read 5 Done Done Sent false
34514 10348 Acked Read 6 Done Done Sent false
34515 10349 Acked Flush 0 Done Done Sent false
34516 10350 Acked Write 8 Done Done Sent false
34517 10351 Acked Read 7 Done Done Sent false
34518 10352 Acked Read 4 Done Done Sent false
34519 10353 Acked Read 10 Done Done Sent false
34520 10354 Acked Write 10 Done Done Sent false
34521 10355 Acked Read 9 Done Done Sent false
34522 10356 Acked Read 10 Done Done Sent false
34523 10357 Acked Write 7 Done Done Sent false
34524 10358 Acked Flush 0 Done Done Sent false
34525 10359 Acked Write 1 Done Done Sent false
34526 STATES DS:0 DS:1 DS:2 TOTAL
34527 Sent 0 0 301 301
34528 Done 301 301 0 602
34529 Skipped 0 0 0 0
34530 Error 0 0 0 0
34531 Last Flush: 10358 10358 10058
34532 Downstairs last five completed: 10058 10057 10056 10055 10054
34533 Upstairs last five completed: 10359 10358 10357 10356 10355
34534 CLIENT: Up:0 ds:301 act:2
34535 [8830] got start action
34536 [8830] I am going to restart
34537 Starting downstairs at port 8830
34538 Make output file at "/tmp/dsc/downstairs-8830.txt"
34539 [8830][2] reports Starting
34540 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1232)
34541 [8830][2] reports Running
34542 ----------------------------------------------------------------
34543 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
34544 States: Active Active Offline
34545 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34546 10059 Acked Write 7 Done Done Sent false
34547 10060 Acked Read 10 Done Done Sent false
34548 10061 Acked Write 2 Done Done Sent false
34549 10062 Acked Read 4 Done Done Sent false
34550 10063 Acked Read 5 Done Done Sent false
34551 10064 Acked Write 6 Done Done Sent false
34552 10065 Acked Write 3 Done Done Sent false
34553 10066 Acked Read 9 Done Done Sent false
34554 10067 Acked Write 10 Done Done Sent false
34555 10068 Acked Read 8 Done Done Sent false
34556 10069 Acked Write 3 Done Done Sent false
34557 10070 Acked Write 5 Done Done Sent false
34558 10071 Acked Write 6 Done Done Sent false
34559 10072 Acked Flush 0 Done Done Sent false
34560 10073 Acked Flush 0 Done Done Sent false
34561 10074 Acked Read 4 Done Done Sent false
34562 10075 Acked Flush 0 Done Done Sent false
34563 10076 Acked Write 3 Done Done Sent false
34564 10077 Acked Write 9 Done Done Sent false
34565 10078 Acked Read 4 Done Done Sent false
34566 10079 Acked Read 2 Done Done Sent false
34567 10080 Acked Read 1 Done Done Sent false
34568 10081 Acked Write 6 Done Done Sent false
34569 10082 Acked Write 7 Done Done Sent false
34570 10083 Acked Read 8 Done Done Sent false
34571 10084 Acked Read 9 Done Done Sent false
34572 10085 Acked Flush 0 Done Done Sent false
34573 10086 Acked Write 6 Done Done Sent false
34574 10087 Acked Read 2 Done Done Sent false
34575 10088 Acked Read 1 Done Done Sent false
34576 10089 Acked Read 5 Done Done Sent false
34577 10090 Acked Write 8 Done Done Sent false
34578 10091 Acked Read 7 Done Done Sent false
34579 10092 Acked Flush 0 Done Done Sent false
34580 10093 Acked Flush 0 Done Done Sent false
34581 10094 Acked Read 3 Done Done Sent false
34582 10095 Acked Read 9 Done Done Sent false
34583 10096 Acked Read 10 Done Done Sent false
34584 10097 Acked Read 8 Done Done Sent false
34585 10098 Acked Read 8 Done Done Sent false
34586 10099 Acked Read 6 Done Done Sent false
34587 10100 Acked Read 2 Done Done Sent false
34588 10101 Acked Write 7 Done Done Sent false
34589 10102 Acked Write 2 Done Done Sent false
34590 10103 Acked Write 2 Done Done Sent false
34591 10104 Acked Read 4 Done Done Sent false
34592 10105 Acked Read 5 Done Done Sent false
34593 10106 Acked Write 6 Done Done Sent false
34594 10107 Acked Write 2 Done Done Sent false
34595 10108 Acked Read 6 Done Done Sent false
34596 10109 Acked Read 2 Done Done Sent false
34597 10110 Acked Read 8 Done Done Sent false
34598 10111 Acked Write 10 Done Done Sent false
34599 10112 Acked Write 3 Done Done Sent false
34600 10113 Acked Write 2 Done Done Sent false
34601 10114 Acked Write 8 Done Done Sent false
34602 10115 Acked Read 1 Done Done Sent false
34603 10116 Acked Read 3 Done Done Sent false
34604 10117 Acked Write 5 Done Done Sent false
34605 10118 Acked Read 7 Done Done Sent false
34606 10119 Acked Write 7 Done Done Sent false
34607 10120 Acked Flush 0 Done Done Sent false
34608 10121 Acked Read 10 Done Done Sent false
34609 10122 Acked Write 3 Done Done Sent false
34610 10123 Acked Write 10 Done Done Sent false
34611 10124 Acked Write 10 Done Done Sent false
34612 10125 Acked Read 9 Done Done Sent false
34613 10126 Acked Read 1 Done Done Sent false
34614 10127 Acked Read 2 Done Done Sent false
34615 10128 Acked Read 3 Done Done Sent false
34616 10129 Acked Read 4 Done Done Sent false
34617 10130 Acked Write 1 Done Done Sent false
34618 10131 Acked Flush 0 Done Done Sent false
34619 10132 Acked Write 7 Done Done Sent false
34620 10133 Acked Read 9 Done Done Sent false
34621 10134 Acked Write 6 Done Done Sent false
34622 10135 Acked Read 4 Done Done Sent false
34623 10136 Acked Flush 0 Done Done Sent false
34624 10137 Acked Write 3 Done Done Sent false
34625 10138 Acked Flush 0 Done Done Sent false
34626 10139 Acked Read 1 Done Done Sent false
34627 10140 Acked Read 5 Done Done Sent false
34628 10141 Acked Read 1 Done Done Sent false
34629 10142 Acked Read 2 Done Done Sent false
34630 10143 Acked Read 7 Done Done Sent false
34631 10144 Acked Write 4 Done Done Sent false
34632 10145 Acked Flush 0 Done Done Sent false
34633 10146 Acked Write 5 Done Done Sent false
34634 10147 Acked Flush 0 Done Done Sent false
34635 10148 Acked Read 3 Done Done Sent false
34636 10149 Acked Read 6 Done Done Sent false
34637 10150 Acked Read 3 Done Done Sent false
34638 10151 Acked Read 4 Done Done Sent false
34639 10152 Acked Write 10 Done Done Sent false
34640 10153 Acked Write 4 Done Done Sent false
34641 10154 Acked Read 5 Done Done Sent false
34642 10155 Acked Write 8 Done Done Sent false
34643 10156 Acked Write 4 Done Done Sent false
34644 10157 Acked Write 6 Done Done Sent false
34645 10158 Acked Read 8 Done Done Sent false
34646 10159 Acked Read 8 Done Done Sent false
34647 10160 Acked Read 5 Done Done Sent false
34648 10161 Acked Read 10 Done Done Sent false
34649 10162 Acked Write 5 Done Done Sent false
34650 10163 Acked Write 10 Done Done Sent false
34651 10164 Acked Read 4 Done Done Sent false
34652 10165 Acked Read 1 Done Done Sent false
34653 10166 Acked Read 6 Done Done Sent false
34654 10167 Acked Write 3 Done Done Sent false
34655 10168 Acked Write 8 Done Done Sent false
34656 10169 Acked Write 4 Done Done Sent false
34657 10170 Acked Flush 0 Done Done Sent false
34658 10171 Acked Write 8 Done Done Sent false
34659 10172 Acked Write 5 Done Done Sent false
34660 10173 Acked Read 4 Done Done Sent false
34661 10174 Acked Write 10 Done Done Sent false
34662 10175 Acked Read 4 Done Done Sent false
34663 10176 Acked Read 4 Done Done Sent false
34664 10177 Acked Read 10 Done Done Sent false
34665 10178 Acked Write 9 Done Done Sent false
34666 10179 Acked Read 6 Done Done Sent false
34667 10180 Acked Flush 0 Done Done Sent false
34668 10181 Acked Read 2 Done Done Sent false
34669 10182 Acked Read 10 Done Done Sent false
34670 10183 Acked Flush 0 Done Done Sent false
34671 10184 Acked Write 9 Done Done Sent false
34672 10185 Acked Write 2 Done Done Sent false
34673 10186 Acked Read 1 Done Done Sent false
34674 10187 Acked Write 2 Done Done Sent false
34675 10188 Acked Flush 0 Done Done Sent false
34676 10189 Acked Write 1 Done Done Sent false
34677 10190 Acked Read 9 Done Done Sent false
34678 10191 Acked Read 5 Done Done Sent false
34679 10192 Acked Read 2 Done Done Sent false
34680 10193 Acked Flush 0 Done Done Sent false
34681 10194 Acked Flush 0 Done Done Sent false
34682 10195 Acked Read 8 Done Done Sent false
34683 10196 Acked Write 7 Done Done Sent false
34684 10197 Acked Read 5 Done Done Sent false
34685 10198 Acked Read 6 Done Done Sent false
34686 10199 Acked Read 6 Done Done Sent false
34687 10200 Acked Write 8 Done Done Sent false
34688 10201 Acked Flush 0 Done Done Sent false
34689 10202 Acked Write 4 Done Done Sent false
34690 10203 Acked Flush 0 Done Done Sent false
34691 10204 Acked Read 6 Done Done Sent false
34692 10205 Acked Flush 0 Done Done Sent false
34693 10206 Acked Read 8 Done Done Sent false
34694 10207 Acked Read 2 Done Done Sent false
34695 10208 Acked Read 2 Done Done Sent false
34696 10209 Acked Flush 0 Done Done Sent false
34697 10210 Acked Read 2 Done Done Sent false
34698 10211 Acked Read 1 Done Done Sent false
34699 10212 Acked Read 7 Done Done Sent false
34700 10213 Acked Write 1 Done Done Sent false
34701 10214 Acked Read 4 Done Done Sent false
34702 10215 Acked Flush 0 Done Done Sent false
34703 10216 Acked Read 4 Done Done Sent false
34704 10217 Acked Flush 0 Done Done Sent false
34705 10218 Acked Write 1 Done Done Sent false
34706 10219 Acked Read 8 Done Done Sent false
34707 10220 Acked Write 9 Done Done Sent false
34708 10221 Acked Read 8 Done Done Sent false
34709 10222 Acked Read 1 Done Done Sent false
34710 10223 Acked Write 2 Done Done Sent false
34711 10224 Acked Write 6 Done Done Sent false
34712 10225 Acked Read 7 Done Done Sent false
34713 10226 Acked Write 5 Done Done Sent false
34714 10227 Acked Read 4 Done Done Sent false
34715 10228 Acked Flush 0 Done Done Sent false
34716 10229 Acked Flush 0 Done Done Sent false
34717 10230 Acked Write 9 Done Done Sent false
34718 10231 Acked Read 6 Done Done Sent false
34719 10232 Acked Read 8 Done Done Sent false
34720 10233 Acked Read 6 Done Done Sent false
34721 10234 Acked Read 10 Done Done Sent false
34722 10235 Acked Write 6 Done Done Sent false
34723 10236 Acked Read 5 Done Done Sent false
34724 10237 Acked Flush 0 Done Done Sent false
34725 10238 Acked Write 3 Done Done Sent false
34726 10239 Acked Read 9 Done Done Sent false
34727 10240 Acked Write 4 Done Done Sent false
34728 10241 Acked Write 10 Done Done Sent false
34729 10242 Acked Read 8 Done Done Sent false
34730 10243 Acked Write 7 Done Done Sent false
34731 10244 Acked Read 10 Done Done Sent false
34732 10245 Acked Read 5 Done Done Sent false
34733 10246 Acked Read 10 Done Done Sent false
34734 10247 Acked Write 2 Done Done Sent false
34735 10248 Acked Read 1 Done Done Sent false
34736 10249 Acked Read 1 Done Done Sent false
34737 10250 Acked Read 2 Done Done Sent false
34738 10251 Acked Read 5 Done Done Sent false
34739 10252 Acked Write 8 Done Done Sent false
34740 10253 Acked Read 10 Done Done Sent false
34741 10254 Acked Read 6 Done Done Sent false
34742 10255 Acked Read 6 Done Done Sent false
34743 10256 Acked Read 4 Done Done Sent false
34744 10257 Acked Read 6 Done Done Sent false
34745 10258 Acked Write 10 Done Done Sent false
34746 10259 Acked Read 4 Done Done Sent false
34747 10260 Acked Write 8 Done Done Sent false
34748 10261 Acked Write 8 Done Done Sent false
34749 10262 Acked Read 9 Done Done Sent false
34750 10263 Acked Read 2 Done Done Sent false
34751 10264 Acked Read 7 Done Done Sent false
34752 10265 Acked Write 6 Done Done Sent false
34753 10266 Acked Flush 0 Done Done Sent false
34754 10267 Acked Read 10 Done Done Sent false
34755 10268 Acked Write 6 Done Done Sent false
34756 10269 Acked Read 3 Done Done Sent false
34757 10270 Acked Write 8 Done Done Sent false
34758 10271 Acked Flush 0 Done Done Sent false
34759 10272 Acked Read 2 Done Done Sent false
34760 10273 Acked Read 3 Done Done Sent false
34761 10274 Acked Flush 0 Done Done Sent false
34762 10275 Acked Flush 0 Done Done Sent false
34763 10276 Acked Read 10 Done Done Sent false
34764 10277 Acked Read 8 Done Done Sent false
34765 10278 Acked Read 3 Done Done Sent false
34766 10279 Acked Read 1 Done Done Sent false
34767 10280 Acked Write 8 Done Done Sent false
34768 10281 Acked Write 3 Done Done Sent false
34769 10282 Acked Write 8 Done Done Sent false
34770 10283 Acked Read 8 Done Done Sent false
34771 10284 Acked Read 10 Done Done Sent false
34772 10285 Acked Write 9 Done Done Sent false
34773 10286 Acked Write 6 Done Done Sent false
34774 10287 Acked Write 7 Done Done Sent false
34775 10288 Acked Write 5 Done Done Sent false
34776 10289 Acked Read 7 Done Done Sent false
34777 10290 Acked Write 10 Done Done Sent false
34778 10291 Acked Read 10 Done Done Sent false
34779 10292 Acked Read 7 Done Done Sent false
34780 10293 Acked Write 8 Done Done Sent false
34781 10294 Acked Read 10 Done Done Sent false
34782 10295 Acked Read 1 Done Done Sent false
34783 10296 Acked Read 8 Done Done Sent false
34784 10297 Acked Read 2 Done Done Sent false
34785 10298 Acked Read 5 Done Done Sent false
34786 10299 Acked Write 2 Done Done Sent false
34787 10300 Acked Write 7 Done Done Sent false
34788 10301 Acked Read 3 Done Done Sent false
34789 10302 Acked Flush 0 Done Done Sent false
34790 10303 Acked Write 4 Done Done Sent false
34791 10304 Acked Flush 0 Done Done Sent false
34792 10305 Acked Write 7 Done Done Sent false
34793 10306 Acked Read 10 Done Done Sent false
34794 10307 Acked Write 1 Done Done Sent false
34795 10308 Acked Read 2 Done Done Sent false
34796 10309 Acked Read 2 Done Done Sent false
34797 10310 Acked Read 2 Done Done Sent false
34798 10311 Acked Read 1 Done Done Sent false
34799 10312 Acked Read 5 Done Done Sent false
34800 10313 Acked Write 9 Done Done Sent false
34801 10314 Acked Write 8 Done Done Sent false
34802 10315 Acked Read 2 Done Done Sent false
34803 10316 Acked Write 2 Done Done Sent false
34804 10317 Acked Flush 0 Done Done Sent false
34805 10318 Acked Write 3 Done Done Sent false
34806 10319 Acked Read 8 Done Done Sent false
34807 10320 Acked Write 10 Done Done Sent false
34808 10321 Acked Write 2 Done Done Sent false
34809 10322 Acked Write 1 Done Done Sent false
34810 10323 Acked Read 2 Done Done Sent false
34811 10324 Acked Read 1 Done Done Sent false
34812 10325 Acked Read 6 Done Done Sent false
34813 10326 Acked Read 2 Done Done Sent false
34814 10327 Acked Write 5 Done Done Sent false
34815 10328 Acked Read 2 Done Done Sent false
34816 10329 Acked Read 5 Done Done Sent false
34817 10330 Acked Read 3 Done Done Sent false
34818 10331 Acked Write 2 Done Done Sent false
34819 10332 Acked Write 5 Done Done Sent false
34820 10333 Acked Read 7 Done Done Sent false
34821 10334 Acked Write 3 Done Done Sent false
34822 10335 Acked Write 2 Done Done Sent false
34823 10336 Acked Read 9 Done Done Sent false
34824 10337 Acked Write 2 Done Done Sent false
34825 10338 Acked Read 5 Done Done Sent false
34826 10339 Acked Flush 0 Done Done Sent false
34827 10340 Acked Read 5 Done Done Sent false
34828 10341 Acked Read 4 Done Done Sent false
34829 10342 Acked Read 8 Done Done Sent false
34830 10343 Acked Read 10 Done Done Sent false
34831 10344 Acked Flush 0 Done Done Sent false
34832 10345 Acked Write 1 Done Done Sent false
34833 10346 Acked Flush 0 Done Done Sent false
34834 10347 Acked Read 5 Done Done Sent false
34835 10348 Acked Read 6 Done Done Sent false
34836 10349 Acked Flush 0 Done Done Sent false
34837 10350 Acked Write 8 Done Done Sent false
34838 10351 Acked Read 7 Done Done Sent false
34839 10352 Acked Read 4 Done Done Sent false
34840 10353 Acked Read 10 Done Done Sent false
34841 10354 Acked Write 10 Done Done Sent false
34842 10355 Acked Read 9 Done Done Sent false
34843 10356 Acked Read 10 Done Done Sent false
34844 10357 Acked Write 7 Done Done Sent false
34845 10358 Acked Flush 0 Done Done Sent false
34846 10359 Acked Write 1 Done Done Sent false
34847 10360 Acked Flush 0 Done Done Sent false
34848 STATES DS:0 DS:1 DS:2 TOTAL
34849 Sent 0 0 302 302
34850 Done 302 302 0 604
34851 Skipped 0 0 0 0
34852 Error 0 0 0 0
34853 Last Flush: 10360 10360 10058
34854 Downstairs last five completed: 10058 10057 10056 10055 10054
34855 Upstairs last five completed: 10360 10359 10358 10357 10356
34856 CLIENT: Up:0 ds:302 act:2
348572024-12-09T15:15:08.555ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
348582024-12-09T15:15:08.555ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49206) = io task client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
348592024-12-09T15:15:08.556ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
348602024-12-09T15:15:08.556ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
348612024-12-09T15:15:08.556ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
348622024-12-09T15:15:08.556ZINFOcrucible: send last flush ID to this DS: 10058 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
348632024-12-09T15:15:08.557ZINFOcrucible: Replied this last flush ID: 10058 = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
348642024-12-09T15:15:08.557ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
348652024-12-09T15:15:08.557ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
348662024-12-09T15:15:08.557ZINFOcrucible: [2] client re-new 302 jobs since flush 10058 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
348672024-12-09T15:15:08.557ZINFOcrucible: [2] Replayed 302 jobs since flush: 10058 = downstairs session_id = 585553f3-c0fa-4eaa-8619-2b4b00d00492
34868 ----------------------------------------------------------------
34869 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
34870 Downstairs last five completed: 10360 10359 10358 10357 10356
34871 Upstairs last five completed: 10360 10359 10358 10357 10356
34872 CLIENT: Up:0 ds:0 act:3
34873 Replay: All jobs finished
34874 Test replay has completed
34875 Wrote out file "/tmp/test_replay_verify.log"
34876 CLIENT: Tests done. All submitted work has been ACK'd
34877 ----------------------------------------------------------------
34878 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
34879 Downstairs last five completed: 10360 10359 10358 10357 10356
34880 Upstairs last five completed: 10360 10359 10358 10357 10356
34881 CLIENT: Up:0 ds:0 act:3
34882 CLIENT: All jobs finished, all DS active.
348832024-12-09T15:15:10.806ZWARNcrucible: Guest handle has been dropped
34884 Replays:30 time: 7:16
34885 Do final verify
34886 Dec 09 15:15:10.881 INFO accepted connection, remote_addr: 127.0.0.1:54122, local_addr: 127.0.0.1:9998
34887 Dec 09 15:15:10.882 INFO request completed, latency_us: 206, response_code: 200, uri: /regioninfo, method: GET, req_id: 7f9ad0bb-2ec7-4255-871b-9b800a41f533, remote_addr: 127.0.0.1:54122, local_addr: 127.0.0.1:9998
348882024-12-09T15:15:10.882ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
34889 Dec 09 15:15:10.883 INFO request completed, latency_us: 99, response_code: 200, uri: /regioncount, method: GET, req_id: 754465b3-e2a2-4a30-99b6-8869a7de4f22, remote_addr: 127.0.0.1:54122, local_addr: 127.0.0.1:9998
348902024-12-09T15:15:10.883ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
34891 Dec 09 15:15:10.883 INFO request completed, latency_us: 118, response_code: 200, uri: /port/cid/0, method: GET, req_id: 5d968f30-dd3f-44cd-bf40-7c593b9a7290, remote_addr: 127.0.0.1:54122, local_addr: 127.0.0.1:9998
34892 Dec 09 15:15:10.884 INFO request completed, latency_us: 92, response_code: 200, uri: /port/cid/1, method: GET, req_id: 5b9d9224-a250-435d-b51f-926d198c01fe, remote_addr: 127.0.0.1:54122, local_addr: 127.0.0.1:9998
34893 Dec 09 15:15:10.884 INFO request completed, latency_us: 93, response_code: 200, uri: /port/cid/2, method: GET, req_id: 6fc41a3a-ba20-47ab-8ff6-2f328f32ada5, remote_addr: 127.0.0.1:54122, local_addr: 127.0.0.1:9998
348942024-12-09T15:15:10.885ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
348952024-12-09T15:15:10.885ZINFOcrucible: Upstairs starts
348962024-12-09T15:15:10.885ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "2978a42101da810efa0e51ea3f370f1f90c0069b", git_commit_timestamp: "2024-12-09T14:48:58.000000000Z", git_branch: "mkeeter/simplify-faults", rustc_semver: "1.82.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "f6e511eec7342f59a25f7c0534f1dbea00d01b14", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
348972024-12-09T15:15:10.885ZINFOcrucible: Upstairs <-> Downstairs Message Version: 12
348982024-12-09T15:15:10.885ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
348992024-12-09T15:15:10.885ZINFOcrucible: Crucible 4f78a850-f0ac-42f8-990b-e69c039696bd has session id: c9bae798-09f5-454b-9f2b-bcfd915eb174 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349002024-12-09T15:15:10.886ZINFOcrucible: Upstairs opts: Upstairs UUID: 4f78a850-f0ac-42f8-990b-e69c039696bd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349012024-12-09T15:15:10.886ZINFOcrucible: Crucible stats registered with UUID: 4f78a850-f0ac-42f8-990b-e69c039696bd session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349022024-12-09T15:15:10.920ZINFOcrucible: Downstairs in volume = 3 task = crutest
349032024-12-09T15:15:10.920ZINFOcrucible: The guest has requested activation with gen:3
349042024-12-09T15:15:10.920ZINFOcrucible: 4f78a850-f0ac-42f8-990b-e69c039696bd active request set session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349052024-12-09T15:15:10.920ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349062024-12-09T15:15:10.920ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349072024-12-09T15:15:10.920ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349082024-12-09T15:15:10.920ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349092024-12-09T15:15:10.920ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349102024-12-09T15:15:10.920ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349112024-12-09T15:15:10.920ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349122024-12-09T15:15:10.920ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349132024-12-09T15:15:10.920ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349142024-12-09T15:15:10.920ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38294) = io task client = 0 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349152024-12-09T15:15:10.921ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:40678) = io task client = 2 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349162024-12-09T15:15:10.921ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34459) = io task client = 1 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349172024-12-09T15:15:10.921ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349182024-12-09T15:15:10.921ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349192024-12-09T15:15:10.921ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349202024-12-09T15:15:10.921ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349212024-12-09T15:15:10.921ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349222024-12-09T15:15:10.921ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349232024-12-09T15:15:10.922ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349242024-12-09T15:15:10.922ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349252024-12-09T15:15:10.922ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349262024-12-09T15:15:10.923ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349272024-12-09T15:15:10.923ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349282024-12-09T15:15:10.923ZINFOcrucible: Waiting for more clients to be ready session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349292024-12-09T15:15:10.923ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349302024-12-09T15:15:10.923ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349312024-12-09T15:15:10.923ZINFOcrucible: Waiting for more clients to be ready session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349322024-12-09T15:15:10.923ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349332024-12-09T15:15:10.923ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349342024-12-09T15:15:10.923ZINFOcrucible: [0]R flush_numbers[0..12]: [917, 871, 944, 939, 939, 923, 887, 945, 905, 945, 924, 945] = downstairs session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349352024-12-09T15:15:10.923ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349362024-12-09T15:15:10.923ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349372024-12-09T15:15:10.923ZINFOcrucible: [1]R flush_numbers[0..12]: [917, 871, 944, 939, 939, 923, 887, 945, 905, 945, 924, 945] = downstairs session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349382024-12-09T15:15:10.923ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349392024-12-09T15:15:10.923ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349402024-12-09T15:15:10.923ZINFOcrucible: [2]R flush_numbers[0..12]: [917, 871, 944, 939, 939, 923, 887, 945, 905, 945, 924, 945] = downstairs session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349412024-12-09T15:15:10.923ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349422024-12-09T15:15:10.923ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349432024-12-09T15:15:10.923ZINFOcrucible: Max found gen is 3 = downstairs session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349442024-12-09T15:15:10.923ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349452024-12-09T15:15:10.923ZINFOcrucible: Next flush: 951 = downstairs session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349462024-12-09T15:15:10.923ZINFOcrucible: All extents match = downstairs session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349472024-12-09T15:15:10.923ZINFOcrucible: No downstairs reconciliation required session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
34948 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
34949 ,"time":"2024-12-09T15:15:10.923750171Z","hostname":"w-01JENZABS6YWW6KJY3YSEMQBFD","pid":1237,"client":"0","":"downstairs","session_id":"c9bae798-09f5-454b-9f2b-bcfd915eb174"}
349502024-12-09T15:15:10.923ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349512024-12-09T15:15:10.923ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349522024-12-09T15:15:10.923ZINFOcrucible: All required reconciliation work is completed session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349532024-12-09T15:15:10.923ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349542024-12-09T15:15:10.923ZINFOcrucible: 4f78a850-f0ac-42f8-990b-e69c039696bd is now active with session: c9bae798-09f5-454b-9f2b-bcfd915eb174 session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349552024-12-09T15:15:10.923ZINFOcrucible: Set Active after no reconciliation session_id = c9bae798-09f5-454b-9f2b-bcfd915eb174
349562024-12-09T15:15:10.923ZINFOcrucible: The guest has finished waiting for activation with:3
34957 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
34958 Loading write count information from file "/tmp/test_replay_verify.log"
34959 Read and Verify all blocks (0..5000 range:false)
34960 Wrote out file "/tmp/test_replay_verify.log"
34961 Verify test completed
34962 Wrote out file "/tmp/test_replay_verify.log"
34963 CLIENT: Tests done. All submitted work has been ACK'd
34964 ----------------------------------------------------------------
34965 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
34966 States: Active Active Active
34967 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34968 1000 Acked Read 256 Done Done Done false
34969 1001 Acked Read 256 Done Done Done false
34970 1002 Acked Read 256 Done Done Done false
34971 1003 Acked Read 256 Done Done Done false
34972 1004 Acked Read 256 Done Done Done false
34973 1005 Acked Read 256 Done Done Done false
34974 1006 Acked Read 256 Done Done Done false
34975 1007 Acked Read 256 Done Done Done false
34976 1008 Acked Read 256 Done Done Done false
34977 1009 Acked Read 256 Done Done Done false
34978 1010 Acked Read 256 Done Done Done false
34979 1011 Acked Read 256 Done Done Done false
34980 1012 Acked Read 256 Done Done Done false
34981 1013 Acked Read 256 Done Done Done false
34982 1014 Acked Read 256 Done Done Done false
34983 1015 Acked Read 256 Done Done Done false
34984 1016 Acked Read 256 Done Done Done false
34985 1017 Acked Read 136 Done Done Done false
34986 1018 Acked Read 256 Done Done Done false
34987 1019 Acked Read 256 Done Done Done false
34988 STATES DS:0 DS:1 DS:2 TOTAL
34989 Sent 0 0 0 0
34990 Done 20 20 20 60
34991 Skipped 0 0 0 0
34992 Error 0 0 0 0
34993 Last Flush: 0 0 0
34994 Downstairs last five completed:
34995 Upstairs last five completed: 1019 1018 1017 1016 1015
34996 CLIENT: Up:0 ds:20 act:3
34997 ----------------------------------------------------------------
34998 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
34999 Downstairs last five completed: 1020 1019 1018 1017 1016
35000 Upstairs last five completed: 1020 1019 1018 1017 1016
35001 CLIENT: Up:0 ds:0 act:3
35002 CLIENT: All crucible jobs finished, exiting program
350032024-12-09T15:15:15.329ZWARNcrucible: Guest handle has been dropped
35004 Dec 09 15:15:15.396 INFO accepted connection, remote_addr: 127.0.0.1:52913, local_addr: 127.0.0.1:9998
35005 Main task has work to do, go find it
35006 got dsc Shutdown
35007 Shutdown
35008 Dec 09 15:15:15.397 INFO request completed, latency_us: 161, response_code: 204Shut it down
35009 , uri: /shutdown, method: POST, req_id: c41e4511-ec4f-41f5-83b4-ccd964339302, remote_addr:[8810] Disable keep_running
35010 127.0.0.1:52913, local_addr: 127[8810] Got stop action so:false kr:false
35011 .0.0.1:9998
35012 [8820] Disable keep_running
35013 [8820] Got stop action so:false kr:false
35014 [8830] Disable keep_running
35015 [8830] Got stop action so:false kr:false
35016 [8830] Exited with: ExitStatus(unix_wait_status(9))
35017 [8810] Exited with: ExitStatus(unix_wait_status(9))
35018 [8830][2] reports Exit
35019 [8820] Exited with: ExitStatus(unix_wait_status(9))
35020 [8810][0] reports Exit
35021 [8820][1] reports Exit
35022 December 9, 2024 at 03:15:20 PM UTC Test ends with 0