Max level shown:
1 starting January 23, 2025 at 06:47:46 PM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.26073593
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.022510732
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.020137012
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start ds: 8820
14 start ds: 8830
15 start access at:127.0.0.1:9998
16 Starting downstairs at port 8810
17 Starting downstairs at port 8830
18 Starting downstairs at port 8820
19 Make output file at "/tmp/dsc/downstairs-8810.txt"
20 Make output file at "/tmp/dsc/downstairs-8830.txt"
21 Make output file at "/tmp/dsc/downstairs-8820.txt"
22 [8810][0] initial start wait reports Starting
23 [8830][2] initial start wait reports Starting
24 [8820][1] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1190)
26 [8810][0] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1191)
28 [8820][1] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1192)
30 [8830][2] initial start wait reports Running
31 All downstairs are running
32 Jan 23 18:47:47.157 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Jan 23 18:47:52.248 INFO accepted connection, remote_addr: 127.0.0.1:33914, local_addr: 127.0.0.1:9998
36 Jan 23 18:47:52.250 INFO request completed, latency_us: 409, response_code: 200, uri: /regioninfo, method: GET, req_id: afdba09f-f151-44bf-84d9-9454b67ece5f, remote_addr: 127.0.0.1:33914, local_addr: 127.0.0.1:9998
37 Jan 23 18:47:52.253 INFO request completed, latency_us: 112, response_code: 200, uri: /regioncount, method: GET, req_id: 6576c53d-3a34-474e-ac96-cad211acff59, remote_addr: 127.0.0.1:33914, local_addr: 127.0.0.1:9998
382025-01-23T18:47:52.252ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392025-01-23T18:47:52.253ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Jan 23 18:47:52.254 INFO request completed, latency_us: 124, response_code: 200, uri: /port/cid/0, method: GET, req_id: 39d72448-e181-46f8-9a32-0b323083522d, remote_addr: 127.0.0.1:33914, local_addr: 127.0.0.1:9998
41 Jan 23 18:47:52.254 INFO request completed, latency_us: 94, response_code: 200, uri: /port/cid/1, method: GET, req_id: 0b474d85-00b6-4361-81b9-feb0066f5f30, remote_addr: 127.0.0.1:33914, local_addr: 127.0.0.1:9998
42 Jan 23 18:47:52.255 INFO request completed, latency_us: 84, response_code: 200, uri: /port/cid/2, method: GET, req_id: 7bd4eaa6-b59f-4333-9e9e-9aaa3f6b5dd0, remote_addr: 127.0.0.1:33914, local_addr: 127.0.0.1:9998
432025-01-23T18:47:52.255ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442025-01-23T18:47:52.256ZINFOcrucible: Upstairs starts
452025-01-23T18:47:52.256ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "dd1cf1330bf5c5b1551a0ad961a7aa7c6207feae", git_commit_timestamp: "2025-01-23T18:31:41.000000000Z", git_branch: "mkeeter/early-io-rejection", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462025-01-23T18:47:52.256ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
472025-01-23T18:47:52.257ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482025-01-23T18:47:52.258ZINFOcrucible: Crucible 97ccc19a-a701-4961-b381-9cbf39ad27a2 has session id: d95af84a-10c5-4af1-a3e8-b0f60269fb0c session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
492025-01-23T18:47:52.258ZINFOcrucible: Upstairs opts: Upstairs UUID: 97ccc19a-a701-4961-b381-9cbf39ad27a2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
502025-01-23T18:47:52.258ZINFOcrucible: Crucible stats registered with UUID: 97ccc19a-a701-4961-b381-9cbf39ad27a2 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
512025-01-23T18:47:52.258ZWARNcrucible: could not find Downstairs address for Nexus session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
522025-01-23T18:47:52.260ZINFOcrucible: Downstairs in volume = 3 task = crutest
532025-01-23T18:47:52.260ZINFOcrucible: The guest has requested activation with gen:1
542025-01-23T18:47:52.260ZINFOcrucible: 97ccc19a-a701-4961-b381-9cbf39ad27a2 active request set session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
552025-01-23T18:47:52.260ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
562025-01-23T18:47:52.260ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
572025-01-23T18:47:52.260ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
582025-01-23T18:47:52.260ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
592025-01-23T18:47:52.260ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
602025-01-23T18:47:52.260ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
612025-01-23T18:47:52.260ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
622025-01-23T18:47:52.260ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
632025-01-23T18:47:52.260ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
642025-01-23T18:47:52.260ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43570) = io task client = 2 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
652025-01-23T18:47:52.261ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53017) = io task client = 1 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
662025-01-23T18:47:52.261ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63397) = io task client = 0 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
672025-01-23T18:47:52.262ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
682025-01-23T18:47:52.262ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
692025-01-23T18:47:52.262ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
702025-01-23T18:47:52.263ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
712025-01-23T18:47:52.263ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
722025-01-23T18:47:52.263ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
732025-01-23T18:47:52.265ZINFOcrucible: Waiting for more clients to be ready session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
742025-01-23T18:47:52.265ZINFOcrucible: Waiting for more clients to be ready session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
752025-01-23T18:47:52.265ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
762025-01-23T18:47:52.265ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
772025-01-23T18:47:52.265ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
782025-01-23T18:47:52.265ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
792025-01-23T18:47:52.265ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
802025-01-23T18:47:52.265ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
812025-01-23T18:47:52.265ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
822025-01-23T18:47:52.265ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
832025-01-23T18:47:52.265ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
842025-01-23T18:47:52.265ZINFOcrucible: Max found gen is 1 = downstairs session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
852025-01-23T18:47:52.265ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
862025-01-23T18:47:52.265ZINFOcrucible: Next flush: 1 = downstairs session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
872025-01-23T18:47:52.265ZINFOcrucible: All extents match = downstairs session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
882025-01-23T18:47:52.265ZINFOcrucible: No downstairs reconciliation required session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
892025-01-23T18:47:52.265ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
902025-01-23T18:47:52.265ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
912025-01-23T18:47:52.265ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
922025-01-23T18:47:52.265ZINFOcrucible: All required reconciliation work is completed session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
932025-01-23T18:47:52.265ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
942025-01-23T18:47:52.265ZINFOcrucible: 97ccc19a-a701-4961-b381-9cbf39ad27a2 is now active with session: d95af84a-10c5-4af1-a3e8-b0f60269fb0c session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
952025-01-23T18:47:52.265ZINFOcrucible: The guest has finished waiting for activation with:1
962025-01-23T18:47:52.265ZINFOcrucible: Set Active after no reconciliation session_id = d95af84a-10c5-4af1-a3e8-b0f60269fb0c
97 Activate successful
98 Wait for a query_work_queue command to finish before sending IO
99 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
100 Fill test
101 Read and Verify all blocks (0..5000 range:false)
102 Wrote out file "/tmp/test_replay_verify.log"
103 CLIENT: Tests done. All submitted work has been ACK'd
104 ----------------------------------------------------------------
105 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
106 States: Active Active Active
107 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
108 1021 Acked Read 256 Done Done Done false
109 1022 Acked Read 256 Done Done Done false
110 1023 Acked Read 256 Done Done Done false
111 1024 Acked Read 256 Done Done Done false
112 1025 Acked Read 256 Done Done Done false
113 1026 Acked Read 256 Done Done Done false
114 1027 Acked Read 256 Done Done Done false
115 1028 Acked Read 256 Done Done Done false
116 1029 Acked Read 256 Done Done Done false
117 1030 Acked Read 256 Done Done Done false
118 1031 Acked Read 256 Done Done Done false
119 1032 Acked Read 256 Done Done Done false
120 1033 Acked Read 256 Done Done Done false
121 1034 Acked Read 256 Done Done Done false
122 1035 Acked Read 256 Done Done Done false
123 1036 Acked Read 256 Done Done Done false
124 1037 Acked Read 256 Done Done Done false
125 1038 Acked Read 136 Done Done Done false
126 1039 Acked Read 256 Done Done Done false
127 1040 Acked Read 256 Done Done Done false
128 STATES DS:0 DS:1 DS:2 TOTAL
129 Sent 0 0 0 0
130 Done 20 20 20 60
131 Skipped 0 0 0 0
132 Error 0 0 0 0
133 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
134 Downstairs last five completed: 1020 1019 1018 1017 1016
135 Upstairs last five completed: 1040 1039 1038 1037 1036
136 CLIENT: Up:0 ds:20 act:3
137 ----------------------------------------------------------------
138 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
139 Downstairs last five completed: 1041 1040 1039 1038 1037
140 Upstairs last five completed: 1041 1040 1039 1038 1037
141 CLIENT: Up:0 ds:0 act:3
142 CLIENT: All crucible jobs finished, exiting program
1432025-01-23T18:47:56.854ZWARNcrucible: Guest handle has been dropped
144 Replay loop starts now January 23, 2025 at 06:47:56 PM UTC
145 Jan 23 18:47:56.925 INFO accepted connection, remote_addr: 127.0.0.1:54682, local_addr: 127.0.0.1:9998
146 Jan 23 18:47:56.925 INFO request completed, latency_us: 143, response_code: 200, uri: /regioninfo, method: GET, req_id: 7ab95282-8737-4830-8907-93f9e774e93e, remote_addr: 127.0.0.1:54682, local_addr: 127.0.0.1:9998
1472025-01-23T18:47:56.926ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
148 Jan 23 18:47:56.926 INFO request completed, latency_us: 96, response_code: 200, uri: /regioncount, method: GET, req_id: b1baae99-81da-47b8-ae19-473428ad5399, remote_addr: 127.0.0.1:54682, local_addr: 127.0.0.1:9998
1492025-01-23T18:47:56.927ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
150 Jan 23 18:47:56.927 INFO request completed, latency_us: 121, response_code: 200, uri: /port/cid/0, method: GET, req_id: eaf2eb28-c38f-44e1-a45d-7d63d7c77be4, remote_addr: 127.0.0.1:54682, local_addr: 127.0.0.1:9998
151 Jan 23 18:47:56.927 INFO request completed, latency_us: 87, response_code: 200, uri: /port/cid/1, method: GET, req_id: 95eff5b1-dfd8-4cd6-9f4b-855a9ff3db2d, remote_addr: 127.0.0.1:54682, local_addr: 127.0.0.1:9998
152 Jan 23 18:47:56.928 INFO request completed, latency_us: 87, response_code: 200, uri: /port/cid/2, method: GET, req_id: 36abf526-dd57-486c-a618-8ebc9036936b, remote_addr: 127.0.0.1:54682, local_addr: 127.0.0.1:9998
1532025-01-23T18:47:56.928ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1542025-01-23T18:47:56.928ZINFOcrucible: Upstairs starts
1552025-01-23T18:47:56.928ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "dd1cf1330bf5c5b1551a0ad961a7aa7c6207feae", git_commit_timestamp: "2025-01-23T18:31:41.000000000Z", git_branch: "mkeeter/early-io-rejection", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1562025-01-23T18:47:56.928ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
1572025-01-23T18:47:56.928ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1582025-01-23T18:47:56.929ZINFOcrucible: Crucible 7f73de2f-d7b3-49a6-86aa-d4636c873d60 has session id: bb6faa00-ad53-4274-84ae-5c5874c7a798 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1592025-01-23T18:47:56.929ZINFOcrucible: Upstairs opts: Upstairs UUID: 7f73de2f-d7b3-49a6-86aa-d4636c873d60, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1602025-01-23T18:47:56.929ZINFOcrucible: Crucible stats registered with UUID: 7f73de2f-d7b3-49a6-86aa-d4636c873d60 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1612025-01-23T18:47:56.929ZWARNcrucible: could not find Downstairs address for Nexus session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1622025-01-23T18:47:56.930ZINFOcrucible: Downstairs in volume = 3 task = crutest
1632025-01-23T18:47:56.930ZINFOcrucible: The guest has requested activation with gen:2
1642025-01-23T18:47:56.930ZINFOcrucible: 7f73de2f-d7b3-49a6-86aa-d4636c873d60 active request set session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1652025-01-23T18:47:56.930ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1662025-01-23T18:47:56.930ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1672025-01-23T18:47:56.930ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1682025-01-23T18:47:56.930ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1692025-01-23T18:47:56.930ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1702025-01-23T18:47:56.930ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1712025-01-23T18:47:56.930ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1722025-01-23T18:47:56.930ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1732025-01-23T18:47:56.930ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1742025-01-23T18:47:56.930ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33959) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1752025-01-23T18:47:56.930ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34549) = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1762025-01-23T18:47:56.930ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46777) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1772025-01-23T18:47:56.931ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1782025-01-23T18:47:56.931ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1792025-01-23T18:47:56.931ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1802025-01-23T18:47:56.931ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1812025-01-23T18:47:56.932ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1822025-01-23T18:47:56.932ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1832025-01-23T18:47:56.932ZINFOcrucible: Waiting for more clients to be ready session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1842025-01-23T18:47:56.932ZINFOcrucible: Waiting for more clients to be ready session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1852025-01-23T18:47:56.932ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1862025-01-23T18:47:56.932ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1872025-01-23T18:47:56.932ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1882025-01-23T18:47:56.932ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1892025-01-23T18:47:56.932ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1902025-01-23T18:47:56.932ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1912025-01-23T18:47:56.932ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1922025-01-23T18:47:56.932ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1932025-01-23T18:47:56.932ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
194 {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30Activate successful
195 Wait for a query_work_queue command to finish before sending IO
196 ,"time":"2025-01-23T18:47:56.932944745Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"downstairs","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
1972025-01-23T18:47:56.932ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1982025-01-23T18:47:56.932ZINFOcrucible: Next flush: 2 = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1992025-01-23T18:47:56.933ZINFOcrucible: All extents match = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2002025-01-23T18:47:56.933ZINFOcrucible: No downstairs reconciliation required session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2012025-01-23T18:47:56.933ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2022025-01-23T18:47:56.933ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2032025-01-23T18:47:56.933ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2042025-01-23T18:47:56.933ZINFOcrucible: All required reconciliation work is completed session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2052025-01-23T18:47:56.933ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2062025-01-23T18:47:56.933ZINFOcrucible: 7f73de2f-d7b3-49a6-86aa-d4636c873d60 is now active with session: bb6faa00-ad53-4274-84ae-5c5874c7a798 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2072025-01-23T18:47:56.933ZINFOcrucible: Set Active after no reconciliation session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2082025-01-23T18:47:56.933ZINFOcrucible: The guest has finished waiting for activation with:2
209 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
210 Loading write count information from file "/tmp/test_replay_verify.log"
211 Jan 23 18:47:56.966 INFO accepted connection, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
212 Main task has work to do, go find it
213 got dsc Stop(0)
214 stop 0
215 Jan 23 18:47:56.967 INFO request completed, latency_us: 159, [8810] Got stop action so:false kr:false
216 response_code: 204, uri: /stop/cid/0, method: POST, req_id: b65a44a2-02d3-473c-b8b3-627460372ca8, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
2172025-01-23T18:47:56.967ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2182025-01-23T18:47:56.967ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2192025-01-23T18:47:56.968ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2202025-01-23T18:47:56.968ZINFOcrucible: client task is exiting = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
221 Jan 23 18:47:56.968 INFO request completed, latency_us: 332, response_code: 200, uri: /state/cid/0, method: GET, req_id: 668be2ac-4e6b-4411-ad7d-3927056ae970, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
222 [8810] Exited with: ExitStatus(unix_wait_status(9))
223 [8810][0] reports Exit
224 Jan 23 18:48:00.970 INFO request completed, latency_us: 181, response_code: 200, uri: /state/cid/0, method: GET, req_id: 653ca6d1-601c-4309-b051-21e29bdbc8cb, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
225 001/300 Read block 3366 len 4096
226 002/300 Read block 958 len 8192
227 003/300 Write block 1569 len 8192 data: 2 2
228 004/300 Write block 3309 len 20480 data: 2 2 2 2 2
229 005/300 Write block 4296 len 20480 data: 2 2 2 2 2
230 006/300 Read block 1975 len 28672
231 007/300 Write block 3510 len 16384 data: 2 2 2 2
232 008/300 Write block 73 len 28672 data: 2 2 2 2 2 2 2
233 009/300 Read block 2226 len 32768
234 010/300 Write block 1778 len 24576 data: 2 2 2 2 2 2
235 011/300 Read block 3387 len 40960
236 012/300 Read block 1923 len 40960
237 013/300 Read block 4653 len 20480
238 014/300 Write block 347 len 24576 data: 2 2 2 2 2 2
239 015/300 Read block 4695 len 4096
240 016/300 Write block 4448 len 4096 data: 2
241 017/300 Write block 2831 len 24576 data: 2 2 2 2 2 2
242 018/300 Read block 3380 len 32768
243 019/300 Read block 3726 len 32768
244 020/300 Write block 4147 len 16384 data: 2 2 2 2
245 021/300 Read block 254 len 32768
246 022/300 Write block 3422 len 40960 data: 2 2 2 2 2 2 2 2 2 2
247 023/300 Write block 4090 len 40960 data: 2 2 2 2 2 2 2 2 2 2
248 024/300 Flush
249 025/300 Write block 4081 len 8192 data: 2 2
250 026/300 Write block 295 len 4096 data: 2
251 027/300 Write block 3001 len 24576 data: 2 2 2 2 2 2
252 028/300 Read block 2577 len 20480
253 029/300 Read block 3385 len 40960
254 030/300 Write block 490 len 36864 data: 2 2 2 2 2 2 2 2 2
255 031/300 Write block 4494 len 36864 data: 2 2 2 2 2 2 2 2 2
256 032/300 Write block 4320 len 32768 data: 2 2 2 2 2 2 2 2
257 033/300 Read block 2750 len 8192
258 034/300 Read block 1257 len 4096
259 035/300 Read block 2071 len 4096
260 036/300 Read block 2472 len 8192
261 037/300 Write block 1008 len 24576 data: 2 2 2 2 2 2
262 038/300 Read block 1699 len 8192
263 039/300 Read block 1000 len 36864
264 040/300 Read block 3844 len 12288
265 041/300 Write block 3357 len 4096 data: 2
266 042/300 Read block 957 len 28672
267 043/300 Flush
268 044/300 Write block 3030 len 32768 data: 2 2 2 2 2 2 2 2
269 045/300 Read block 627 len 4096
270 046/300 Write block 4504 len 8192 data: 2 2
271 047/300 Write block 4901 len 24576 data: 2 2 2 2 2 2
272 048/300 Write block 710 len 36864 data: 2 2 2 2 2 2 2 2 2
273 049/300 Read block 1402 len 16384
274 050/300 Write block 1895 len 40960 data: 2 2 2 2 2 2 2 2 2 2
275 051/300 Read block 2755 len 12288
276 052/300 Write block 1095 len 40960 data: 2 2 2 2 2 2 2 2 2 2
277 053/300 Write block 90 len 32768 data: 2 2 2 2 2 2 2 2
278 054/300 Read block 1104 len 4096
279 055/300 Write block 2523 len 8192 data: 2 2
280 056/300 Write block 462 len 8192 data: 2 2
281 057/300 Read block 600 len 4096
282 058/300 Write block 2128 len 12288 data: 2 2 2
283 059/300 Write block 2471 len 20480 data: 2 2 2 2 2
284 060/300 Write block 2764 len 4096 data: 2
285 061/300 Read block 1609 len 32768
286 062/300 Write block 826 len 20480 data: 2 2 2 2 2
287 063/300 Read block 3271 len 24576
288 064/300 Read block 832 len 28672
289 065/300 Read block 3515 len 28672
290 066/300 Write block 2882 len 4096 data: 2
291 067/300 Read block 3754 len 4096
292 068/300 Read block 3170 len 32768
293 069/300 Read block 644 len 24576
294 070/300 Read block 160 len 8192
295 071/300 Read block 4578 len 20480
296 072/300 Write block 2050 len 8192 data: 2 2
297 073/300 Read block 3690 len 24576
298 074/300 Read block 3620 len 16384
299 075/300 Read block 4394 len 24576
300 076/300 Write block 1675 len 20480 data: 2 2 2 2 2
301 077/300 Read block 3526 len 24576
302 078/300 Read block 1927 len 20480
303 079/300 Write block 30 len 4096 data: 2
304 080/300 Write block 122 len 20480 data: 2 2 2 2 2
305 081/300 Write block 450 len 40960 data: 2 2 2 2 2 2 2 2 2 2
306 082/300 Read block 3441 len 12288
307 083/300 Read block 752 len 12288
308 084/300 Read block 320 len 8192
309 085/300 Read block 1221 len 40960
310 086/300 Write block 2071 len 4096 data: 2
311 087/300 Write block 3676 len 28672 data: 2 2 2 2 2 2 2
312 088/300 Write block 4064 len 20480 data: 2 2 2 2 2
313 089/300 Read block 1066 len 32768
314 090/300 Write block 3048 len 16384 data: 2 2 2 2
315 091/300 Read block 4241 len 36864
316 092/300 Write block 178 len 8192 data: 2 2
317 093/300 Write block 1484 len 32768 data: 2 2 2 2 2 2 2 2
318 094/300 Read block 868 len 32768
319 095/300 Write block 2540 len 28672 data: 2 2 2 2 2 2 2
320 096/300 Read block 4285 len 4096
321 097/300 Read block 1478 len 40960
322 098/300 Read block 3585 len 36864
323 099/300 Write block 783 len 8192 data: 2 2
324 100/300 Write block 4915 len 28672 data: 2 2 2 2 2 2 2
325 101/300 Write block 4924 len 20480 data: 2 2 2 2 2
326 102/300 Read block 2637 len 4096
327 103/300 Flush
328 104/300 Write block 4460 len 24576 data: 2 2 2 2 2 2
329 105/300 Read block 4331 len 40960
330 106/300 Read block 1851 len 20480
331 107/300 Read block 3063 len 36864
332 108/300 Flush
333 109/300 Write block 1809 len 4096 data: 2
334 110/300 Read block 3699 len 40960
335 111/300 Read block 3189 len 24576
336 112/300 Read block 4637 len 20480
337 113/300 Write block 2070 len 40960 data: 2 3 2 2 2 2 2 2 2 2
338 114/300 Read block 3041 len 28672
339 115/300 Write block 78 len 24576 data: 3 3 2 2 2 2
340 116/300 Read block 3424 len 20480
341 117/300 Read block 1384 len 32768
342 118/300 Read block 929 len 16384
343 119/300 Read block 1553 len 40960
344 120/300 Write block 478 len 16384 data: 2 2 2 2
345 121/300 Write block 2850 len 20480 data: 2 2 2 2 2
346 122/300 Write block 3293 len 20480 data: 2 2 2 2 2
347 123/300 Read block 1614 len 28672
348 124/300 Write block 3913 len 12288 data: 2 2 2
349 125/300 Write block 3349 len 4096 data: 2
350 126/300 Write block 3911 len 40960 data: 2 2 3 3 3 2 2 2 2 2
351 127/300 Read block 3186 len 4096
352 128/300 Read block 4713 len 12288
353 129/300 Write block 2602 len 40960 data: 2 2 2 2 2 2 2 2 2 2
354 130/300 Read block 2390 len 24576
355 131/300 Write block 4135 len 32768 data: 2 2 2 2 2 2 2 2
356 132/300 Read block 2774 len 12288
357 133/300 Flush
358 134/300 Write block 4806 len 28672 data: 2 2 2 2 2 2 2
359 135/300 Flush
360 136/300 Write block 3653 len 4096 data: 2
361 137/300 Read block 4660 len 8192
362 138/300 Read block 1277 len 16384
363 139/300 Write block 461 len 36864 data: 2 3 3 2 2 2 2 2 2
364 140/300 Read block 3813 len 16384
365 141/300 Flush
366 142/300 Write block 2886 len 40960 data: 2 2 2 2 2 2 2 2 2 2
367 143/300 Write block 3857 len 16384 data: 2 2 2 2
368 144/300 Read block 2246 len 12288
369 145/300 Read block 32 len 12288
370 146/300 Read block 3201 len 28672
371 147/300 Read block 4668 len 4096
372 148/300 Read block 948 len 40960
373 149/300 Write block 775 len 32768 data: 2 2 2 2 2 2 2 2
374 150/300 Read block 2748 len 12288
375 151/300 Write block 3592 len 28672 data: 2 2 2 2 2 2 2
376 152/300 Write block 931 len 36864 data: 2 2 2 2 2 2 2 2 2
377 153/300 Read block 1981 len 28672
378 154/300 Flush
379 155/300 Write block 1373 len 12288 data: 2 2 2
380 156/300 Write block 3358 len 36864 data: 2 2 2 2 2 2 2 2 2
381 157/300 Read block 383 len 28672
382 158/300 Write block 2993 len 40960 data: 2 2 2 2 2 2 2 2 3 3
383 159/300 Write block 354 len 20480 data: 2 2 2 2 2
384 160/300 Write block 3039 len 8192 data: 2 2
385 161/300 Write block 527 len 28672 data: 2 2 2 2 2 2 2
386 162/300 Write block 454 len 40960 data: 3 3 3 3 3 3 2 3 4 4
387 163/300 Read block 3543 len 12288
388 164/300 Read block 274 len 24576
389 165/300 Read block 3050 len 16384
390 166/300 Write block 1061 len 40960 data: 2 2 2 2 2 2 2 2 2 2
391 167/300 Write block 3283 len 4096 data: 2
392 168/300 Read block 3200 len 28672
393 169/300 Read block 2648 len 28672
394 170/300 Write block 2614 len 24576 data: 2 2 2 2 2 2
395 171/300 Read block 852 len 8192
396 172/300 Read block 4279 len 40960
397 173/300 Read block 1422 len 28672
398 174/300 Read block 1560 len 20480
399 175/300 Read block 4264 len 32768
400 176/300 Write block 4685 len 20480 data: 2 2 2 2 2
401 177/300 Read block 1034 len 12288
402 178/300 Write block 1985 len 16384 data: 2 2 2 2
403 179/300 Read block 3923 len 8192
404 180/300 Read block 624 len 32768
405 181/300 Read block 1083 len 32768
406 182/300 Write block 939 len 28672 data: 3 2 2 2 2 2 2
407 183/300 Flush
408 184/300 Write block 4369 len 32768 data: 2 2 2 2 2 2 2 2
409 185/300 Read block 61 len 32768
410 186/300 Write block 3922 len 32768 data: 2 2 2 2 2 2 2 2
411 187/300 Write block 1341 len 20480 data: 2 2 2 2 2
412 188/300 Write block 2937 len 24576 data: 2 2 2 2 2 2
413 189/300 Flush
414 190/300 Write block 1608 len 8192 data: 2 2
415 191/300 Write block 654 len 12288 data: 2 2 2
416 192/300 Write block 4928 len 32768 data: 3 2 2 2 2 2 2 2
417 193/300 Write block 4446 len 4096 data: 2
418 194/300 Write block 3464 len 32768 data: 2 2 2 2 2 2 2 2
419 195/300 Write block 4286 len 32768 data: 2 2 2 2 2 2 2 2
420 196/300 Read block 3808 len 16384
421 197/300 Write block 1106 len 4096 data: 2
422 198/300 Read block 4443 len 20480
423 199/300 Read block 3627 len 36864
424 200/300 Flush
425 201/300 Read block 719 len 32768
426 202/300 Read block 2964 len 40960
427 203/300 Write block 919 len 8192 data: 2 2
428 204/300 Read block 2664 len 20480
429 205/300 Read block 1005 len 4096
430 206/300 Read block 3582 len 32768
431 207/300 Read block 14 len 36864
432 208/300 Write block 4575 len 32768 data: 2 2 2 2 2 2 2 2
433 209/300 Flush
434 210/300 Write block 1908 len 8192 data: 2 2
435 211/300 Read block 335 len 40960
436 212/300 Write block 4146 len 32768 data: 2 3 3 3 3 2 2 2
437 213/300 Read block 2775 len 36864
438 214/300 Write block 760 len 20480 data: 2 2 2 2 2
439 215/300 Read block 1692 len 12288
440 216/300 Write block 2378 len 8192 data: 2 2
441 217/300 Write block 4658 len 28672 data: 2 2 2 2 2 2 2
442 218/300 Write block 1047 len 12288 data: 2 2 2
443 219/300 Read block 4514 len 32768
444 220/300 Write block 3944 len 32768 data: 2 2 2 2 2 2 2 2
445 221/300 Write block 2620 len 20480 data: 2 2 2 2 2
446 222/300 Read block 1901 len 12288
447 223/300 Write block 2382 len 16384 data: 2 2 2 2
448 224/300 Read block 1408 len 16384
449 225/300 Write block 3236 len 40960 data: 2 2 2 2 2 2 2 2 2 2
450 226/300 Read block 44 len 40960
451 227/300 Read block 3862 len 8192
452 228/300 Write block 4643 len 20480 data: 2 2 2 2 2
453 229/300 Write block 3294 len 4096 data: 3
454 230/300 Read block 1737 len 24576
455 231/300 Read block 1750 len 32768
456 232/300 Read block 750 len 16384
457 233/300 Read block 4983 len 24576
458 234/300 Read block 1115 len 16384
459 235/300 Write block 3499 len 4096 data: 2
460 236/300 Read block 3787 len 4096
461 237/300 Read block 1655 len 20480
462 238/300 Read block 4792 len 28672
463 239/300 Read block 4747 len 20480
464 240/300 Read block 384 len 16384
465 241/300 Read block 1807 len 24576
466 242/300 Write block 1491 len 8192 data: 3 2
467 243/300 Read block 3574 len 28672
468 244/300 Read block 2125 len 4096
469 245/300 Read block 2052 len 24576
470 246/300 Write block 2676 len 28672 data: 2 2 2 2 2 2 2
471 247/300 Read block 2940 len 32768
472 248/300 Read block 167 len 32768
473 249/300 Read block 3789 len 24576
474 250/300 Read block 4099 len 28672
475 251/300 Read block 262 len 36864
476 252/300 Write block 3295 len 36864 data: 3 3 3 2 2 2 2 2 2
477 253/300 Read block 3371 len 28672
478 254/300 Read block 4893 len 16384
479 255/300 Flush
480 256/300 Read block 3495 len 20480
481 257/300 Write block 270 len 36864 data: 2 2 2 2 2 2 2 2 2
482 258/300 Write block 1337 len 16384 data: 2 2 2 2
483 259/300 Read block 2907 len 24576
484 260/300 Write block 3693 len 24576 data: 2 2 2 2 2 2
485 261/300 Write block 562 len 32768 data: 2 2 2 2 2 2 2 2
486 262/300 Write block 1440 len 20480 data: 2 2 2 2 2
487 263/300 Read block 1888 len 16384
488 264/300 Read block 1605 len 20480
489 265/300 Read block 4185 len 4096
490 266/300 Read block 2695 len 4096
491 267/300 Read block 228 len 20480
492 268/300 Read block 4817 len 32768
493 269/300 Read block 2074 len 28672
494 270/300 Read block 652 len 24576
495 271/300 Write block 1688 len 36864 data: 2 2 2 2 2 2 2 2 2
496 272/300 Read block 3361 len 20480
497 273/300 Read block 1526 len 12288
498 274/300 Flush
499 275/300 Write block 786 len 12288 data: 2 2 2
500 276/300 Write block 2403 len 36864 data: 2 2 2 2 2 2 2 2 2
501 277/300 Write block 2741 len 8192 data: 2 2
502 278/300 Write block 165 len 24576 data: 2 2 2 2 2 2
503 279/300 Write block 2448 len 20480 data: 2 2 2 2 2
504 280/300 Write block 911 len 36864 data: 2 2 2 2 2 2 2 2 3
505 281/300 Read block 4481 len 4096
506 282/300 Flush
507 283/300 Read block 3572 len 36864
508 284/300 Write block 2266 len 40960 data: 2 2 2 2 2 2 2 2 2 2
509 285/300 Write block 2096 len 8192 data: 2 2
510 286/300 Read block 1798 len 20480
511 287/300 Write block 4777 len 28672 data: 2 2 2 2 2 2 2
512 288/300 Read block 3417 len 40960
513 289/300 Read block 4855 len 40960
514 290/300 Read block 468 len 24576
515 291/300 Write block 3953 len 32768 data: 2 2 2 2 2 2 2 2
516 292/300 Write block 2330 len 12288 data: 2 2 2
517 293/300 Write block 2403 len 28672 data: 3 3 3 3 3 3 3
518 294/300 Read block 3739 len 20480
519 295/300 Read block 3258 len 12288
520 296/300 Read block 4538 len 36864
521 297/300 Flush
522 298/300 Read block 1756 len 4096
523 299/300 Read block 3258 len 20480
524 300/300 Read block 2845 len 4096
525 301/300 Write block 2313 len 12288 data: 2 2 2
526 Main task has work to do, go find it
527 got dsc Start(0)
528 start 0
529 Jan 23 18:48:01.214 INFO request completed, latency_us: 205, response_code: 204, uri: /start/cid/0, method: POST, req_id: 37daaaf5-fc03-4fc8-babd-c2010f94f797, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
530 Replay: started 0, returned:Ok(())
531 ----------------------------------------------------------------
532 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
533 States: Offline Active Active
534 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
535 1000 Acked Read 1 Sent Done Done false
536 1001 Acked Read 2 Sent Done Done false
537 1002 Acked Write 2 Sent Done Done false
538 1003 Acked Write 5 Sent Done Done false
539 1004 Acked Write 5 Sent Done Done false
540 1005 Acked Read 7 Sent Done Done false
541 1006 Acked Write 4 Sent Done Done false
542 1007 Acked Write 7 Sent Done Done false
543 1008 Acked Read 8 Sent Done Done false
544 1009 Acked Write 6 Sent Done Done false
545 1010 Acked Read 10 Sent Done Done false
546 1011 Acked Read 10 Sent Done Done false
547 1012 Acked Read 5 Sent Done Done false
548 1013 Acked Write 6 Sent Done Done false
549 1014 Acked Read 1 Sent Done Done false
550 1015 Acked Write 1 Sent Done Done false
551 1016 Acked Write 6 Sent Done Done false
552 1017 Acked Read 8 Sent Done Done false
553 1018 Acked Read 8 Sent Done Done false
554 1019 Acked Write 4 Sent Done Done false
555 1020 Acked Read 8 Sent Done Done false
556 1021 Acked Write 10 Sent Done Done false
557 1022 Acked Write 10 Sent Done Done false
558 1023 Acked Flush 0 Sent Done Done false
559 1024 Acked Write 2 Sent Done Done false
560 1025 Acked Write 1 Sent Done Done false
561 1026 Acked Write 6 Sent Done Done false
562 1027 Acked Read 5 Sent Done Done false
563 1028 Acked Read 10 Sent Done Done false
564 1029 Acked Write 9 Sent Done Done false
565 1030 Acked Write 9 Sent Done Done false
566 1031 Acked Write 8 Sent Done Done false
567 1032 Acked Read 2 Sent Done Done false
568 1033 Acked Read 1 Sent Done Done false
569 1034 Acked Read 1 Sent Done Done false
570 1035 Acked Read 2 Sent Done Done false
571 1036 Acked Write 6 Sent Done Done false
572 1037 Acked Read 2 Sent Done Done false
573 1038 Acked Read 9 Sent Done Done false
574 1039 Acked Read 3 Sent Done Done false
575 1040 Acked Write 1 Sent Done Done false
576 1041 Acked Read 7 Sent Done Done false
577 1042 Acked Flush 0 Sent Done Done false
578 1043 Acked Write 8 Sent Done Done false
579 1044 Acked Read 1 Sent Done Done false
580 1045 Acked Write 2 Sent Done Done false
581 1046 Acked Write 6 Sent Done Done false
582 1047 Acked Write 9 Sent Done Done false
583 1048 Acked Read 4 Sent Done Done false
584 1049 Acked Write 10 Sent Done Done false
585 1050 Acked Read 3 Sent Done Done false
586 1051 Acked Write 10 Sent Done Done false
587 1052 Acked Write 8 Sent Done Done false
588 1053 Acked Read 1 Sent Done Done false
589 1054 Acked Write 2 Sent Done Done false
590 1055 Acked Write 2 Sent Done Done false
591 1056 Acked Read 1 Sent Done Done false
592 1057 Acked Write 3 Sent Done Done false
593 1058 Acked Write 5 Sent Done Done false
594 1059 Acked Write 1 Sent Done Done false
595 1060 Acked Read 8 Sent Done Done false
596 1061 Acked Write 5 Sent Done Done false
597 1062 Acked Read 6 Sent Done Done false
598 1063 Acked Read 7 Sent Done Done false
599 1064 Acked Read 7 Sent Done Done false
600 1065 Acked Write 1 Sent Done Done false
601 1066 Acked Read 1 Sent Done Done false
602 1067 Acked Read 8 Sent Done Done false
603 1068 Acked Read 6 Sent Done Done false
604 1069 Acked Read 2 Sent Done Done false
605 1070 Acked Read 5 Sent Done Done false
606 1071 Acked Write 2 Sent Done Done false
607 1072 Acked Read 6 Sent Done Done false
608 1073 Acked Read 4 Sent Done Done false
609 1074 Acked Read 6 Sent Done Done false
610 1075 Acked Write 5 Sent Done Done false
611 1076 Acked Read 6 Sent Done Done false
612 1077 Acked Read 5 Sent Done Done false
613 1078 Acked Write 1 Sent Done Done false
614 1079 Acked Write 5 Sent Done Done false
615 1080 Acked Write 10 Sent Done Done false
616 1081 Acked Read 3 Sent Done Done false
617 1082 Acked Read 3 Sent Done Done false
618 1083 Acked Read 2 Sent Done Done false
619 1084 Acked Read 10 Sent Done Done false
620 1085 Acked Write 1 Sent Done Done false
621 1086 Acked Write 7 Sent Done Done false
622 1087 Acked Write 5 Sent Done Done false
623 1088 Acked Read 8 Sent Done Done false
624 1089 Acked Write 4 Sent Done Done false
625 1090 Acked Read 9 Sent Done Done false
626 1091 Acked Write 2 Sent Done Done false
627 1092 Acked Write 8 Sent Done Done false
628 1093 Acked Read 8 Sent Done Done false
629 1094 Acked Write 7 Sent Done Done false
630 1095 Acked Read 1 Sent Done Done false
631 1096 Acked Read 10 Sent Done Done false
632 1097 Acked Read 9 Sent Done Done false
633 1098 Acked Write 2 Sent Done Done false
634 1099 Acked Write 7 Sent Done Done false
635 1100 Acked Write 5 Sent Done Done false
636 1101 Acked Read 1 Sent Done Done false
637 1102 Acked Flush 0 Sent Done Done false
638 1103 Acked Write 6 Sent Done Done false
639 1104 Acked Read 10 Sent Done Done false
640 1105 Acked Read 5 Sent Done Done false
641 1106 Acked Read 9 Sent Done Done false
642 1107 Acked Flush 0 Sent Done Done false
643 1108 Acked Write 1 Sent Done Done false
644 1109 Acked Read 10 Sent Done Done false
645 1110 Acked Read 6 Sent Done Done false
646 1111 Acked Read 5 Sent Done Done false
647 1112 Acked Write 10 Sent Done Done false
648 1113 Acked Read 7 Sent Done Done false
649 1114 Acked Write 6 Sent Done Done false
650 1115 Acked Read 5 Sent Done Done false
651 1116 Acked Read 8 Sent Done Done false
652 1117 Acked Read 4 Sent Done Done false
653 1118 Acked Read 10 Sent Done Done false
654 1119 Acked Write 4 Sent Done Done false
655 1120 Acked Write 5 Sent Done Done false
656 1121 Acked Write 5 Sent Done Done false
657 1122 Acked Read 7 Sent Done Done false
658 1123 Acked Write 3 Sent Done Done false
659 1124 Acked Write 1 Sent Done Done false
660 1125 Acked Write 10 Sent Done Done false
661 1126 Acked Read 1 Sent Done Done false
662 1127 Acked Read 3 Sent Done Done false
663 1128 Acked Write 10 Sent Done Done false
664 1129 Acked Read 6 Sent Done Done false
665 1130 Acked Write 8 Sent Done Done false
666 1131 Acked Read 3 Sent Done Done false
667 1132 Acked Flush 0 Sent Done Done false
668 1133 Acked Write 7 Sent Done Done false
669 1134 Acked Flush 0 Sent Done Done false
670 1135 Acked Write 1 Sent Done Done false
671 1136 Acked Read 2 Sent Done Done false
672 1137 Acked Read 4 Sent Done Done false
673 1138 Acked Write 9 Sent Done Done false
674 1139 Acked Read 4 Sent Done Done false
675 1140 Acked Flush 0 Sent Done Done false
676 1141 Acked Write 10 Sent Done Done false
677 1142 Acked Write 4 Sent Done Done false
678 1143 Acked Read 3 Sent Done Done false
679 1144 Acked Read 3 Sent Done Done false
680 1145 Acked Read 7 Sent Done Done false
681 1146 Acked Read 1 Sent Done Done false
682 1147 Acked Read 10 Sent Done Done false
683 1148 Acked Write 8 Sent Done Done false
684 1149 Acked Read 3 Sent Done Done false
685 1150 Acked Write 7 Sent Done Done false
686 1151 Acked Write 9 Sent Done Done false
687 1152 Acked Read 7 Sent Done Done false
688 1153 Acked Flush 0 Sent Done Done false
689 1154 Acked Write 3 Sent Done Done false
690 1155 Acked Write 9 Sent Done Done false
691 1156 Acked Read 7 Sent Done Done false
692 1157 Acked Write 10 Sent Done Done false
693 1158 Acked Write 5 Sent Done Done false
694 1159 Acked Write 2 Sent Done Done false
695 1160 Acked Write 7 Sent Done Done false
696 1161 Acked Write 10 Sent Done Done false
697 1162 Acked Read 3 Sent Done Done false
698 1163 Acked Read 6 Sent Done Done false
699 1164 Acked Read 4 Sent Done Done false
700 1165 Acked Write 10 Sent Done Done false
701 1166 Acked Write 1 Sent Done Done false
702 1167 Acked Read 7 Sent Done Done false
703 1168 Acked Read 7 Sent Done Done false
704 1169 Acked Write 6 Sent Done Done false
705 1170 Acked Read 2 Sent Done Done false
706 1171 Acked Read 10 Sent Done Done false
707 1172 Acked Read 7 Sent Done Done false
708 1173 Acked Read 5 Sent Done Done false
709 1174 Acked Read 8 Sent Done Done false
710 1175 Acked Write 5 Sent Done Done false
711 1176 Acked Read 3 Sent Done Done false
712 1177 Acked Write 4 Sent Done Done false
713 1178 Acked Read 2 Sent Done Done false
714 1179 Acked Read 8 Sent Done Done false
715 1180 Acked Read 8 Sent Done Done false
716 1181 Acked Write 7 Sent Done Done false
717 1182 Acked Flush 0 Sent Done Done false
718 1183 Acked Write 8 Sent Done Done false
719 1184 Acked Read 8 Sent Done Done false
720 1185 Acked Write 8 Sent Done Done false
721 1186 Acked Write 5 Sent Done Done false
722 1187 Acked Write 6 Sent Done Done false
723 1188 Acked Flush 0 Sent Done Done false
724 1189 Acked Write 2 Sent Done Done false
725 1190 Acked Write 3 Sent Done Done false
726 1191 Acked Write 8 Sent Done Done false
727 1192 Acked Write 1 Sent Done Done false
728 1193 Acked Write 8 Sent Done Done false
729 1194 Acked Write 8 Sent Done Done false
730 1195 Acked Read 4 Sent Done Done false
731 1196 Acked Write 1 Sent Done Done false
732 1197 Acked Read 5 Sent Done Done false
733 1198 Acked Read 9 Sent Done Done false
734 1199 Acked Flush 0 Sent Done Done false
735 1200 Acked Read 8 Sent Done Done false
736 1201 Acked Read 10 Sent Done Done false
737 1202 Acked Write 2 Sent Done Done false
738 1203 Acked Read 5 Sent Done Done false
739 1204 Acked Read 1 Sent Done Done false
740 1205 Acked Read 8 Sent Done Done false
741 1206 Acked Read 9 Sent Done Done false
742 1207 Acked Write 8 Sent Done Done false
743 1208 Acked Flush 0 Sent Done Done false
744 1209 Acked Write 2 Sent Done Done false
745 1210 Acked Read 10 Sent Done Done false
746 1211 Acked Write 8 Sent Done Done false
747 1212 Acked Read 9 Sent Done Done false
748 1213 Acked Write 5 Sent Done Done false
749 1214 Acked Read 3 Sent Done Done false
750 1215 Acked Write 2 Sent Done Done false
751 1216 Acked Write 7 Sent Done Done false
752 1217 Acked Write 3 Sent Done Done false
753 1218 Acked Read 8 Sent Done Done false
754 1219 Acked Write 8 Sent Done Done false
755 1220 Acked Write 5 Sent Done Done false
756 1221 Acked Read 3 Sent Done Done false
757 1222 Acked Write 4 Sent Done Done false
758 1223 Acked Read 4 Sent Done Done false
759 1224 Acked Write 10 Sent Done Done false
760 1225 Acked Read 10 Sent Done Done false
761 1226 Acked Read 2 Sent Done Done false
762 1227 Acked Write 5 Sent Done Done false
763 1228 Acked Write 1 Sent Done Done false
764 1229 Acked Read 6 Sent Done Done false
765 1230 Acked Read 8 Sent Done Done false
766 1231 Acked Read 4 Sent Done Done false
767 1232 Acked Read 6 Sent Done Done false
768 1233 Acked Read 4 Sent Done Done false
769 1234 Acked Write 1 Sent Done Done false
770 1235 Acked Read 1 Sent Done Done false
771 1236 Acked Read 5 Sent Done Done false
772 1237 Acked Read 7 Sent Done Done false
773 1238 Acked Read 5 Sent Done Done false
774 1239 Acked Read 4 Sent Done Done false
775 1240 Acked Read 6 Sent Done Done false
776 1241 Acked Write 2 Sent Done Done false
777 1242 Acked Read 7 Sent Done Done false
778 1243 Acked Read 1 Sent Done Done false
779 1244 Acked Read 6 Sent Done Done false
780 1245 Acked Write 7 Sent Done Done false
781 1246 Acked Read 8 Sent Done Done false
782 1247 Acked Read 8 Sent Done Done false
783 1248 Acked Read 6 Sent Done Done false
784 1249 Acked Read 7 Sent Done Done false
785 1250 Acked Read 9 Sent Done Done false
786 1251 Acked Write 9 Sent Done Done false
787 1252 Acked Read 7 Sent Done Done false
788 1253 Acked Read 4 Sent Done Done false
789 1254 Acked Flush 0 Sent Done Done false
790 1255 Acked Read 5 Sent Done Done false
791 1256 Acked Write 9 Sent Done Done false
792 1257 Acked Write 4 Sent Done Done false
793 1258 Acked Read 6 Sent Done Done false
794 1259 Acked Write 6 Sent Done Done false
795 1260 Acked Write 8 Sent Done Done false
796 1261 Acked Write 5 Sent Done Done false
797 1262 Acked Read 4 Sent Done Done false
798 1263 Acked Read 5 Sent Done Done false
799 1264 Acked Read 1 Sent Done Done false
800 1265 Acked Read 1 Sent Done Done false
801 1266 Acked Read 5 Sent Done Done false
802 1267 Acked Read 8 Sent Done Done false
803 1268 Acked Read 7 Sent Done Done false
804 1269 Acked Read 6 Sent Done Done false
805 1270 Acked Write 9 Sent Done Done false
806 1271 Acked Read 5 Sent Done Done false
807 1272 Acked Read 3 Sent Done Done false
808 1273 Acked Flush 0 Sent Done Done false
809 1274 Acked Write 3 Sent Done Done false
810 1275 Acked Write 9 Sent Done Done false
811 1276 Acked Write 2 Sent Done Done false
812 1277 Acked Write 6 Sent Done Done false
813 1278 Acked Write 5 Sent Done Done false
814 1279 Acked Write 9 Sent Done Done false
815 1280 Acked Read 1 Sent Done Done false
816 1281 Acked Flush 0 Sent Done Done false
817 1282 Acked Read 9 Sent Done Done false
818 1283 Acked Write 10 Sent Done Done false
819 1284 Acked Write 2 Sent Done Done false
820 1285 Acked Read 5 Sent Done Done false
821 1286 Acked Write 7 Sent Done Done false
822 1287 Acked Read 10 Sent Done Done false
823 1288 Acked Read 10 Sent Done Done false
824 1289 Acked Read 6 Sent Done Done false
825 1290 Acked Write 8 Sent Done Done false
826 1291 Acked Write 3 Sent Done Done false
827 1292 Acked Write 7 Sent Done Done false
828 1293 Acked Read 5 Sent Done Done false
829 1294 Acked Read 3 Sent Done Done false
830 1295 Acked Read 9 Sent Done Done false
831 1296 Acked Flush 0 Sent Done Done false
832 1297 Acked Read 1 Sent Done Done false
833 1298 Acked Read 5 Sent Done Done false
834 1299 Acked Read 1 Sent Done Done false
835 1300 Acked Write 3 Sent Done Done false
836 STATES DS:0 DS:1 DS:2 TOTAL
837 Sent 301 0 0 301
838 Done 0 301 301 602
839 Skipped 0 0 0 0
840 Error 0 0 0 0
841 Last Flush: None Some(JobId(1296)) Some(JobId(1296))
842 Downstairs last five completed:
843 Upstairs last five completed: 1300 1299 1298 1297 1296
844 CLIENT: Up:0 ds:301 act:2
845 ----------------------------------------------------------------
846 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
847 States: Offline Active Active
848 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
849 1000 Acked Read 1 Sent Done Done false
850 1001 Acked Read 2 Sent Done Done false
851 1002 Acked Write 2 Sent Done Done false
852 1003 Acked Write 5 Sent Done Done false
853 1004 Acked Write 5 Sent Done Done false
854 1005 Acked Read 7 Sent Done Done false
855 1006 Acked Write 4 Sent Done Done false
856 1007 Acked Write 7 Sent Done Done false
857 1008 Acked Read 8 Sent Done Done false
858 1009 Acked Write 6 Sent Done Done false
859 1010 Acked Read 10 Sent Done Done false
860 1011 Acked Read 10 Sent Done Done false
861 1012 Acked Read 5 Sent Done Done false
862 1013 Acked Write 6 Sent Done Done false
863 1014 Acked Read 1 Sent Done Done false
864 1015 Acked Write 1 Sent Done Done false
865 1016 Acked Write 6 Sent Done Done false
866 1017 Acked Read 8 Sent Done Done false
867 1018 Acked Read 8 Sent Done Done false
868 1019 Acked Write 4 Sent Done Done false
869 1020 Acked Read 8 Sent Done Done false
870 1021 Acked Write 10 Sent Done Done false
871 1022 Acked Write 10 Sent Done Done false
872 1023 Acked Flush 0 Sent Done Done false
873 1024 Acked Write 2 Sent Done Done false
874 1025 Acked Write 1 Sent Done Done false
875 1026 Acked Write 6 Sent Done Done false
876 1027 Acked Read 5 Sent Done Done false
877 1028 Acked Read 10 Sent Done Done false
878 1029 Acked Write 9 Sent Done Done false
879 1030 Acked Write 9 Sent Done Done false
880 1031 Acked Write 8 Sent Done Done false
881 1032 Acked Read 2 Sent Done Done false
882 1033 Acked Read 1 Sent Done Done false
883 1034 Acked Read 1 Sent Done Done false
884 1035 Acked Read 2 Sent Done Done false
885 1036 Acked Write 6 Sent Done Done false
886 1037 Acked Read 2 Sent Done Done false
887 1038 Acked Read 9 Sent Done Done false
888 1039 Acked Read 3 Sent Done Done false
889 1040 Acked Write 1 Sent Done Done false
890 1041 Acked Read 7 Sent Done Done false
891 1042 Acked Flush 0 Sent Done Done false
892 1043 Acked Write 8 Sent Done Done false
893 1044 Acked Read 1 Sent Done Done false
894 1045 Acked Write 2 Sent Done Done false
895 1046 Acked Write 6 Sent Done Done false
896 1047 Acked Write 9 Sent Done Done false
897 1048 Acked Read 4 Sent Done Done false
898 1049 Acked Write 10 Sent Done Done false
899 1050 Acked Read 3 Sent Done Done false
900 1051 Acked Write 10 Sent Done Done false
901 1052 Acked Write 8 Sent Done Done false
902 1053 Acked Read 1 Sent Done Done false
903 1054 Acked Write 2 Sent Done Done false
904 1055 Acked Write 2 Sent Done Done false
905 1056 Acked Read 1 Sent Done Done false
906 1057 Acked Write 3 Sent Done Done false
907 1058 Acked Write 5 Sent Done Done false
908 1059 Acked Write 1 Sent Done Done false
909 1060 Acked Read 8 Sent Done Done false
910 1061 Acked Write 5 Sent Done Done false
911 1062 Acked Read 6 Sent Done Done false
912 1063 Acked Read 7 Sent Done Done false
913 1064 Acked Read 7 Sent Done Done false
914 1065 Acked Write 1 Sent Done Done false
915 1066 Acked Read 1 Sent Done Done false
916 1067 Acked Read 8 Sent Done Done false
917 1068 Acked Read 6 Sent Done Done false
918 1069 Acked Read 2 Sent Done Done false
919 1070 Acked Read 5 Sent Done Done false
920 1071 Acked Write 2 Sent Done Done false
921 1072 Acked Read 6 Sent Done Done false
922 1073 Acked Read 4 Sent Done Done false
923 1074 Acked Read 6 Sent Done Done false
924 1075 Acked Write 5 Sent Done Done false
925 1076 Acked Read 6 Sent Done Done false
926 1077 Acked Read 5 Sent Done Done false
927 1078 Acked Write 1 Sent Done Done false
928 1079 Acked Write 5 Sent Done Done false
929 1080 Acked Write 10 Sent Done Done false
930 1081 Acked Read 3 Sent Done Done false
931 1082 Acked Read 3 Sent Done Done false
932 1083 Acked Read 2 Sent Done Done false
933 1084 Acked Read 10 Sent Done Done false
934 1085 Acked Write 1 Sent Done Done false
935 1086 Acked Write 7 Sent Done Done false
936 1087 Acked Write 5 Sent Done Done false
937 1088 Acked Read 8 Sent Done Done false
938 1089 Acked Write 4 Sent Done Done false
939 1090 Acked Read 9 Sent Done Done false
940 1091 Acked Write 2 Sent Done Done false
941 1092 Acked Write 8 Sent Done Done false
942 1093 Acked Read 8 Sent Done Done false
943 1094 Acked Write 7 Sent Done Done false
944 1095 Acked Read 1 Sent Done Done false
945 1096 Acked Read 10 Sent Done Done false
946 1097 Acked Read 9 Sent Done Done false
947 1098 Acked Write 2 Sent Done Done false
948 1099 Acked Write 7 Sent Done Done false
949 1100 Acked Write 5 Sent Done Done false
950 1101 Acked Read 1 Sent Done Done false
951 1102 Acked Flush 0 Sent Done Done false
952 1103 Acked Write 6 Sent Done Done false
953 1104 Acked Read 10 Sent Done Done false
954 1105 Acked Read 5 Sent Done Done false
955 1106 Acked Read 9 Sent Done Done false
956 1107 Acked Flush 0 Sent Done Done false
957 1108 Acked Write 1 Sent Done Done false
958 1109 Acked Read 10 Sent Done Done false
959 1110 Acked Read 6 Sent Done Done false
960 1111 Acked Read 5 Sent Done Done false
961 1112 Acked Write 10 Sent Done Done false
962 1113 Acked Read 7 Sent Done Done false
963 1114 Acked Write 6 Sent Done Done false
964 1115 Acked Read 5 Sent Done Done false
965 1116 Acked Read 8 Sent Done Done false
966 1117 Acked Read 4 Sent Done Done false
967 1118 Acked Read 10 Sent Done Done false
968 1119 Acked Write 4 Sent Done Done false
969 1120 Acked Write 5 Sent Done Done false
970 1121 Acked Write 5 Sent Done Done false
971 1122 Acked Read 7 Sent Done Done false
972 1123 Acked Write 3 Sent Done Done false
973 1124 Acked Write 1 Sent Done Done false
974 1125 Acked Write 10 Sent Done Done false
975 1126 Acked Read 1 Sent Done Done false
976 1127 Acked Read 3 Sent Done Done false
977 1128 Acked Write 10 Sent Done Done false
978 1129 Acked Read 6 Sent Done Done false
979 1130 Acked Write 8 Sent Done Done false
980 1131 Acked Read 3 Sent Done Done false
981 1132 Acked Flush 0 Sent Done Done false
982 1133 Acked Write 7 Sent Done Done false
983 1134 Acked Flush 0 Sent Done Done false
984 1135 Acked Write 1 Sent Done Done false
985 1136 Acked Read 2 Sent Done Done false
986 1137 Acked Read 4 Sent Done Done false
987 1138 Acked Write 9 Sent Done Done false
988 1139 Acked Read 4 Sent Done Done false
989 1140 Acked Flush 0 Sent Done Done false
990 1141 Acked Write 10 Sent Done Done false
991 1142 Acked Write 4 Sent Done Done false
992 1143 Acked Read 3 Sent Done Done false
993 1144 Acked Read 3 Sent Done Done false
994 1145 Acked Read 7 Sent Done Done false
995 1146 Acked Read 1 Sent Done Done false
996 1147 Acked Read 10 Sent Done Done false
997 1148 Acked Write 8 Sent Done Done false
998 1149 Acked Read 3 Sent Done Done false
999 1150 Acked Write 7 Sent Done Done false
1000 1151 Acked Write 9 Sent Done Done false
1001 1152 Acked Read 7 Sent Done Done false
1002 1153 Acked Flush 0 Sent Done Done false
1003 1154 Acked Write 3 Sent Done Done false
1004 1155 Acked Write 9 Sent Done Done false
1005 1156 Acked Read 7 Sent Done Done false
1006 1157 Acked Write 10 Sent Done Done false
1007 1158 Acked Write 5 Sent Done Done false
1008 1159 Acked Write 2 Sent Done Done false
1009 1160 Acked Write 7 Sent Done Done false
1010 1161 Acked Write 10 Sent Done Done false
1011 1162 Acked Read 3 Sent Done Done false
1012 1163 Acked Read 6 Sent Done Done false
1013 1164 Acked Read 4 Sent Done Done false
1014 1165 Acked Write 10 Sent Done Done false
1015 1166 Acked Write 1 Sent Done Done false
1016 1167 Acked Read 7 Sent Done Done false
1017 1168 Acked Read 7 Sent Done Done false
1018 1169 Acked Write 6 Sent Done Done false
1019 1170 Acked Read 2 Sent Done Done false
1020 1171 Acked Read 10 Sent Done Done false
1021 1172 Acked Read 7 Sent Done Done false
1022 1173 Acked Read 5 Sent Done Done false
1023 1174 Acked Read 8 Sent Done Done false
1024 1175 Acked Write 5 Sent Done Done false
1025 1176 Acked Read 3 Sent Done Done false
1026 1177 Acked Write 4 Sent Done Done false
1027 1178 Acked Read 2 Sent Done Done false
1028 1179 Acked Read 8 Sent Done Done false
1029 1180 Acked Read 8 Sent Done Done false
1030 1181 Acked Write 7 Sent Done Done false
1031 1182 Acked Flush 0 Sent Done Done false
1032 1183 Acked Write 8 Sent Done Done false
1033 1184 Acked Read 8 Sent Done Done false
1034 1185 Acked Write 8 Sent Done Done false
1035 1186 Acked Write 5 Sent Done Done false
1036 1187 Acked Write 6 Sent Done Done false
1037 1188 Acked Flush 0 Sent Done Done false
1038 1189 Acked Write 2 Sent Done Done false
1039 1190 Acked Write 3 Sent Done Done false
1040 1191 Acked Write 8 Sent Done Done false
1041 1192 Acked Write 1 Sent Done Done false
1042 1193 Acked Write 8 Sent Done Done false
1043 1194 Acked Write 8 Sent Done Done false
1044 1195 Acked Read 4 Sent Done Done false
1045 1196 Acked Write 1 Sent Done Done false
1046 1197 Acked Read 5 Sent Done Done false
1047 1198 Acked Read 9 Sent Done Done false
1048 1199 Acked Flush 0 Sent Done Done false
1049 1200 Acked Read 8 Sent Done Done false
1050 1201 Acked Read 10 Sent Done Done false
1051 1202 Acked Write 2 Sent Done Done false
1052 1203 Acked Read 5 Sent Done Done false
1053 1204 Acked Read 1 Sent Done Done false
1054 1205 Acked Read 8 Sent Done Done false
1055 1206 Acked Read 9 Sent Done Done false
1056 1207 Acked Write 8 Sent Done Done false
1057 1208 Acked Flush 0 Sent Done Done false
1058 1209 Acked Write 2 Sent Done Done false
1059 1210 Acked Read 10 Sent Done Done false
1060 1211 Acked Write 8 Sent Done Done false
1061 1212 Acked Read 9 Sent Done Done false
1062 1213 Acked Write 5 Sent Done Done false
1063 1214 Acked Read 3 Sent Done Done false
1064 1215 Acked Write 2 Sent Done Done false
1065 1216 Acked Write 7 Sent Done Done false
1066 1217 Acked Write 3 Sent Done Done false
1067 1218 Acked Read 8 Sent Done Done false
1068 1219 Acked Write 8 Sent Done Done false
1069 1220 Acked Write 5 Sent Done Done false
1070 1221 Acked Read 3 Sent Done Done false
1071 1222 Acked Write 4 Sent Done Done false
1072 1223 Acked Read 4 Sent Done Done false
1073 1224 Acked Write 10 Sent Done Done false
1074 1225 Acked Read 10 Sent Done Done false
1075 1226 Acked Read 2 Sent Done Done false
1076 1227 Acked Write 5 Sent Done Done false
1077 1228 Acked Write 1 Sent Done Done false
1078 1229 Acked Read 6 Sent Done Done false
1079 1230 Acked Read 8 Sent Done Done false
1080 1231 Acked Read 4 Sent Done Done false
1081 1232 Acked Read 6 Sent Done Done false
1082 1233 Acked Read 4 Sent Done Done false
1083 1234 Acked Write 1 Sent Done Done false
1084 1235 Acked Read 1 Sent Done Done false
1085 1236 Acked Read 5 Sent Done Done false
1086 1237 Acked Read 7 Sent Done Done false
1087 1238 Acked Read 5 Sent Done Done false
1088 1239 Acked Read 4 Sent Done Done false
1089 1240 Acked Read 6 Sent Done Done false
1090 1241 Acked Write 2 Sent Done Done false
1091 1242 Acked Read 7 Sent Done Done false
1092 1243 Acked Read 1 Sent Done Done false
1093 1244 Acked Read 6 Sent Done Done false
1094 1245 Acked Write 7 Sent Done Done false
1095 1246 Acked Read 8 Sent Done Done false
1096 1247 Acked Read 8 Sent Done Done false
1097 1248 Acked Read 6 Sent Done Done false
1098 1249 Acked Read 7 Sent Done Done false
1099 1250 Acked Read 9 Sent Done Done false
1100 1251 Acked Write 9 Sent Done Done false
1101 1252 Acked Read 7 Sent Done Done false
1102 1253 Acked Read 4 Sent Done Done false
1103 1254 Acked Flush 0 Sent Done Done false
1104 1255 Acked Read 5 Sent Done Done false
1105 1256 Acked Write 9 Sent Done Done false
1106 1257 Acked Write 4 Sent Done Done false
1107 1258 Acked Read 6 Sent Done Done false
1108 1259 Acked Write 6 Sent Done Done false
1109 1260 Acked Write 8 Sent Done Done false
1110 1261 Acked Write 5 Sent Done Done false
1111 1262 Acked Read 4 Sent Done Done false
1112 1263 Acked Read 5 Sent Done Done false
1113 1264 Acked Read 1 Sent Done Done false
1114 1265 Acked Read 1 Sent Done Done false
1115 1266 Acked Read 5 Sent Done Done false
1116 1267 Acked Read 8 Sent Done Done false
1117 1268 Acked Read 7 Sent Done Done false
1118 1269 Acked Read 6 Sent Done Done false
1119 1270 Acked Write 9 Sent Done Done false
1120 1271 Acked Read 5 Sent Done Done false
1121 1272 Acked Read 3 Sent Done Done false
1122 1273 Acked Flush 0 Sent Done Done false
1123 1274 Acked Write 3 Sent Done Done false
1124 1275 Acked Write 9 Sent Done Done false
1125 1276 Acked Write 2 Sent Done Done false
1126 1277 Acked Write 6 Sent Done Done false
1127 1278 Acked Write 5 Sent Done Done false
1128 1279 Acked Write 9 Sent Done Done false
1129 1280 Acked Read 1 Sent Done Done false
1130 1281 Acked Flush 0 Sent Done Done false
1131 1282 Acked Read 9 Sent Done Done false
1132 1283 Acked Write 10 Sent Done Done false
1133 1284 Acked Write 2 Sent Done Done false
1134 1285 Acked Read 5 Sent Done Done false
1135 1286 Acked Write 7 Sent Done Done false
1136 1287 Acked Read 10 Sent Done Done false
1137 1288 Acked Read 10 Sent Done Done false
1138 1289 Acked Read 6 Sent Done Done false
1139 1290 Acked Write 8 Sent Done Done false
1140 1291 Acked Write 3 Sent Done Done false
1141 1292 Acked Write 7 Sent Done Done false
1142 1293 Acked Read 5 Sent Done Done false
1143 1294 Acked Read 3 Sent Done Done false
1144 1295 Acked Read 9 Sent Done Done false
1145 1296 Acked Flush 0 Sent Done Done false
1146 1297 Acked Read 1 Sent Done Done false
1147 1298 Acked Read 5 Sent Done Done false
1148 1299 Acked Read 1 Sent Done Done false
1149 1300 Acked Write 3 Sent Done Done false
1150 1301 Acked Flush 0 Sent Done Done false
1151 STATES DS:0 DS:1 DS:2 TOTAL
1152 Sent 302 0 0 302
1153 Done 0 302 302 604
1154 Skipped 0 0 0 0
1155 Error 0 0 0 0
1156 Last Flush: None Some(JobId(1301)) Some(JobId(1301))
1157 Downstairs last five completed:
1158 Upstairs last five completed: 1301 1300 1299 1298 1297
1159 CLIENT: Up:0 ds:302 act:2
11602025-01-23T18:48:06.969ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
11612025-01-23T18:48:06.970ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
11622025-01-23T18:48:06.970ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
11632025-01-23T18:48:06.970ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
11642025-01-23T18:48:06.970ZINFOcrucible: client task is exiting = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1165 [8810] got start action
1166 [8810] I am going to restart
1167 Starting downstairs at port 8810
1168 Make output file at "/tmp/dsc/downstairs-8810.txt"
1169 [8810][0] reports Starting
1170 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1201)
1171 [8810][0] reports Running
1172 ----------------------------------------------------------------
1173 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
1174 States: Offline Active Active
1175 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1176 1000 Acked Read 1 Sent Done Done false
1177 1001 Acked Read 2 Sent Done Done false
1178 1002 Acked Write 2 Sent Done Done false
1179 1003 Acked Write 5 Sent Done Done false
1180 1004 Acked Write 5 Sent Done Done false
1181 1005 Acked Read 7 Sent Done Done false
1182 1006 Acked Write 4 Sent Done Done false
1183 1007 Acked Write 7 Sent Done Done false
1184 1008 Acked Read 8 Sent Done Done false
1185 1009 Acked Write 6 Sent Done Done false
1186 1010 Acked Read 10 Sent Done Done false
1187 1011 Acked Read 10 Sent Done Done false
1188 1012 Acked Read 5 Sent Done Done false
1189 1013 Acked Write 6 Sent Done Done false
1190 1014 Acked Read 1 Sent Done Done false
1191 1015 Acked Write 1 Sent Done Done false
1192 1016 Acked Write 6 Sent Done Done false
1193 1017 Acked Read 8 Sent Done Done false
1194 1018 Acked Read 8 Sent Done Done false
1195 1019 Acked Write 4 Sent Done Done false
1196 1020 Acked Read 8 Sent Done Done false
1197 1021 Acked Write 10 Sent Done Done false
1198 1022 Acked Write 10 Sent Done Done false
1199 1023 Acked Flush 0 Sent Done Done false
1200 1024 Acked Write 2 Sent Done Done false
1201 1025 Acked Write 1 Sent Done Done false
1202 1026 Acked Write 6 Sent Done Done false
1203 1027 Acked Read 5 Sent Done Done false
1204 1028 Acked Read 10 Sent Done Done false
1205 1029 Acked Write 9 Sent Done Done false
1206 1030 Acked Write 9 Sent Done Done false
1207 1031 Acked Write 8 Sent Done Done false
1208 1032 Acked Read 2 Sent Done Done false
1209 1033 Acked Read 1 Sent Done Done false
1210 1034 Acked Read 1 Sent Done Done false
1211 1035 Acked Read 2 Sent Done Done false
1212 1036 Acked Write 6 Sent Done Done false
1213 1037 Acked Read 2 Sent Done Done false
1214 1038 Acked Read 9 Sent Done Done false
1215 1039 Acked Read 3 Sent Done Done false
1216 1040 Acked Write 1 Sent Done Done false
1217 1041 Acked Read 7 Sent Done Done false
1218 1042 Acked Flush 0 Sent Done Done false
1219 1043 Acked Write 8 Sent Done Done false
1220 1044 Acked Read 1 Sent Done Done false
1221 1045 Acked Write 2 Sent Done Done false
1222 1046 Acked Write 6 Sent Done Done false
1223 1047 Acked Write 9 Sent Done Done false
1224 1048 Acked Read 4 Sent Done Done false
1225 1049 Acked Write 10 Sent Done Done false
1226 1050 Acked Read 3 Sent Done Done false
1227 1051 Acked Write 10 Sent Done Done false
1228 1052 Acked Write 8 Sent Done Done false
1229 1053 Acked Read 1 Sent Done Done false
1230 1054 Acked Write 2 Sent Done Done false
1231 1055 Acked Write 2 Sent Done Done false
1232 1056 Acked Read 1 Sent Done Done false
1233 1057 Acked Write 3 Sent Done Done false
1234 1058 Acked Write 5 Sent Done Done false
1235 1059 Acked Write 1 Sent Done Done false
1236 1060 Acked Read 8 Sent Done Done false
1237 1061 Acked Write 5 Sent Done Done false
1238 1062 Acked Read 6 Sent Done Done false
1239 1063 Acked Read 7 Sent Done Done false
1240 1064 Acked Read 7 Sent Done Done false
1241 1065 Acked Write 1 Sent Done Done false
1242 1066 Acked Read 1 Sent Done Done false
1243 1067 Acked Read 8 Sent Done Done false
1244 1068 Acked Read 6 Sent Done Done false
1245 1069 Acked Read 2 Sent Done Done false
1246 1070 Acked Read 5 Sent Done Done false
1247 1071 Acked Write 2 Sent Done Done false
1248 1072 Acked Read 6 Sent Done Done false
1249 1073 Acked Read 4 Sent Done Done false
1250 1074 Acked Read 6 Sent Done Done false
1251 1075 Acked Write 5 Sent Done Done false
1252 1076 Acked Read 6 Sent Done Done false
1253 1077 Acked Read 5 Sent Done Done false
1254 1078 Acked Write 1 Sent Done Done false
1255 1079 Acked Write 5 Sent Done Done false
1256 1080 Acked Write 10 Sent Done Done false
1257 1081 Acked Read 3 Sent Done Done false
1258 1082 Acked Read 3 Sent Done Done false
1259 1083 Acked Read 2 Sent Done Done false
1260 1084 Acked Read 10 Sent Done Done false
1261 1085 Acked Write 1 Sent Done Done false
1262 1086 Acked Write 7 Sent Done Done false
1263 1087 Acked Write 5 Sent Done Done false
1264 1088 Acked Read 8 Sent Done Done false
1265 1089 Acked Write 4 Sent Done Done false
1266 1090 Acked Read 9 Sent Done Done false
1267 1091 Acked Write 2 Sent Done Done false
1268 1092 Acked Write 8 Sent Done Done false
1269 1093 Acked Read 8 Sent Done Done false
1270 1094 Acked Write 7 Sent Done Done false
1271 1095 Acked Read 1 Sent Done Done false
1272 1096 Acked Read 10 Sent Done Done false
1273 1097 Acked Read 9 Sent Done Done false
1274 1098 Acked Write 2 Sent Done Done false
1275 1099 Acked Write 7 Sent Done Done false
1276 1100 Acked Write 5 Sent Done Done false
1277 1101 Acked Read 1 Sent Done Done false
1278 1102 Acked Flush 0 Sent Done Done false
1279 1103 Acked Write 6 Sent Done Done false
1280 1104 Acked Read 10 Sent Done Done false
1281 1105 Acked Read 5 Sent Done Done false
1282 1106 Acked Read 9 Sent Done Done false
1283 1107 Acked Flush 0 Sent Done Done false
1284 1108 Acked Write 1 Sent Done Done false
1285 1109 Acked Read 10 Sent Done Done false
1286 1110 Acked Read 6 Sent Done Done false
1287 1111 Acked Read 5 Sent Done Done false
1288 1112 Acked Write 10 Sent Done Done false
1289 1113 Acked Read 7 Sent Done Done false
1290 1114 Acked Write 6 Sent Done Done false
1291 1115 Acked Read 5 Sent Done Done false
1292 1116 Acked Read 8 Sent Done Done false
1293 1117 Acked Read 4 Sent Done Done false
1294 1118 Acked Read 10 Sent Done Done false
1295 1119 Acked Write 4 Sent Done Done false
1296 1120 Acked Write 5 Sent Done Done false
1297 1121 Acked Write 5 Sent Done Done false
1298 1122 Acked Read 7 Sent Done Done false
1299 1123 Acked Write 3 Sent Done Done false
1300 1124 Acked Write 1 Sent Done Done false
1301 1125 Acked Write 10 Sent Done Done false
1302 1126 Acked Read 1 Sent Done Done false
1303 1127 Acked Read 3 Sent Done Done false
1304 1128 Acked Write 10 Sent Done Done false
1305 1129 Acked Read 6 Sent Done Done false
1306 1130 Acked Write 8 Sent Done Done false
1307 1131 Acked Read 3 Sent Done Done false
1308 1132 Acked Flush 0 Sent Done Done false
1309 1133 Acked Write 7 Sent Done Done false
1310 1134 Acked Flush 0 Sent Done Done false
1311 1135 Acked Write 1 Sent Done Done false
1312 1136 Acked Read 2 Sent Done Done false
1313 1137 Acked Read 4 Sent Done Done false
1314 1138 Acked Write 9 Sent Done Done false
1315 1139 Acked Read 4 Sent Done Done false
1316 1140 Acked Flush 0 Sent Done Done false
1317 1141 Acked Write 10 Sent Done Done false
1318 1142 Acked Write 4 Sent Done Done false
1319 1143 Acked Read 3 Sent Done Done false
1320 1144 Acked Read 3 Sent Done Done false
1321 1145 Acked Read 7 Sent Done Done false
1322 1146 Acked Read 1 Sent Done Done false
1323 1147 Acked Read 10 Sent Done Done false
1324 1148 Acked Write 8 Sent Done Done false
1325 1149 Acked Read 3 Sent Done Done false
1326 1150 Acked Write 7 Sent Done Done false
1327 1151 Acked Write 9 Sent Done Done false
1328 1152 Acked Read 7 Sent Done Done false
1329 1153 Acked Flush 0 Sent Done Done false
1330 1154 Acked Write 3 Sent Done Done false
1331 1155 Acked Write 9 Sent Done Done false
1332 1156 Acked Read 7 Sent Done Done false
1333 1157 Acked Write 10 Sent Done Done false
1334 1158 Acked Write 5 Sent Done Done false
1335 1159 Acked Write 2 Sent Done Done false
1336 1160 Acked Write 7 Sent Done Done false
1337 1161 Acked Write 10 Sent Done Done false
1338 1162 Acked Read 3 Sent Done Done false
1339 1163 Acked Read 6 Sent Done Done false
1340 1164 Acked Read 4 Sent Done Done false
1341 1165 Acked Write 10 Sent Done Done false
1342 1166 Acked Write 1 Sent Done Done false
1343 1167 Acked Read 7 Sent Done Done false
1344 1168 Acked Read 7 Sent Done Done false
1345 1169 Acked Write 6 Sent Done Done false
1346 1170 Acked Read 2 Sent Done Done false
1347 1171 Acked Read 10 Sent Done Done false
1348 1172 Acked Read 7 Sent Done Done false
1349 1173 Acked Read 5 Sent Done Done false
1350 1174 Acked Read 8 Sent Done Done false
1351 1175 Acked Write 5 Sent Done Done false
1352 1176 Acked Read 3 Sent Done Done false
1353 1177 Acked Write 4 Sent Done Done false
1354 1178 Acked Read 2 Sent Done Done false
1355 1179 Acked Read 8 Sent Done Done false
1356 1180 Acked Read 8 Sent Done Done false
1357 1181 Acked Write 7 Sent Done Done false
1358 1182 Acked Flush 0 Sent Done Done false
1359 1183 Acked Write 8 Sent Done Done false
1360 1184 Acked Read 8 Sent Done Done false
1361 1185 Acked Write 8 Sent Done Done false
1362 1186 Acked Write 5 Sent Done Done false
1363 1187 Acked Write 6 Sent Done Done false
1364 1188 Acked Flush 0 Sent Done Done false
1365 1189 Acked Write 2 Sent Done Done false
1366 1190 Acked Write 3 Sent Done Done false
1367 1191 Acked Write 8 Sent Done Done false
1368 1192 Acked Write 1 Sent Done Done false
1369 1193 Acked Write 8 Sent Done Done false
1370 1194 Acked Write 8 Sent Done Done false
1371 1195 Acked Read 4 Sent Done Done false
1372 1196 Acked Write 1 Sent Done Done false
1373 1197 Acked Read 5 Sent Done Done false
1374 1198 Acked Read 9 Sent Done Done false
1375 1199 Acked Flush 0 Sent Done Done false
1376 1200 Acked Read 8 Sent Done Done false
1377 1201 Acked Read 10 Sent Done Done false
1378 1202 Acked Write 2 Sent Done Done false
1379 1203 Acked Read 5 Sent Done Done false
1380 1204 Acked Read 1 Sent Done Done false
1381 1205 Acked Read 8 Sent Done Done false
1382 1206 Acked Read 9 Sent Done Done false
1383 1207 Acked Write 8 Sent Done Done false
1384 1208 Acked Flush 0 Sent Done Done false
1385 1209 Acked Write 2 Sent Done Done false
1386 1210 Acked Read 10 Sent Done Done false
1387 1211 Acked Write 8 Sent Done Done false
1388 1212 Acked Read 9 Sent Done Done false
1389 1213 Acked Write 5 Sent Done Done false
1390 1214 Acked Read 3 Sent Done Done false
1391 1215 Acked Write 2 Sent Done Done false
1392 1216 Acked Write 7 Sent Done Done false
1393 1217 Acked Write 3 Sent Done Done false
1394 1218 Acked Read 8 Sent Done Done false
1395 1219 Acked Write 8 Sent Done Done false
1396 1220 Acked Write 5 Sent Done Done false
1397 1221 Acked Read 3 Sent Done Done false
1398 1222 Acked Write 4 Sent Done Done false
1399 1223 Acked Read 4 Sent Done Done false
1400 1224 Acked Write 10 Sent Done Done false
1401 1225 Acked Read 10 Sent Done Done false
1402 1226 Acked Read 2 Sent Done Done false
1403 1227 Acked Write 5 Sent Done Done false
1404 1228 Acked Write 1 Sent Done Done false
1405 1229 Acked Read 6 Sent Done Done false
1406 1230 Acked Read 8 Sent Done Done false
1407 1231 Acked Read 4 Sent Done Done false
1408 1232 Acked Read 6 Sent Done Done false
1409 1233 Acked Read 4 Sent Done Done false
1410 1234 Acked Write 1 Sent Done Done false
1411 1235 Acked Read 1 Sent Done Done false
1412 1236 Acked Read 5 Sent Done Done false
1413 1237 Acked Read 7 Sent Done Done false
1414 1238 Acked Read 5 Sent Done Done false
1415 1239 Acked Read 4 Sent Done Done false
1416 1240 Acked Read 6 Sent Done Done false
1417 1241 Acked Write 2 Sent Done Done false
1418 1242 Acked Read 7 Sent Done Done false
1419 1243 Acked Read 1 Sent Done Done false
1420 1244 Acked Read 6 Sent Done Done false
1421 1245 Acked Write 7 Sent Done Done false
1422 1246 Acked Read 8 Sent Done Done false
1423 1247 Acked Read 8 Sent Done Done false
1424 1248 Acked Read 6 Sent Done Done false
1425 1249 Acked Read 7 Sent Done Done false
1426 1250 Acked Read 9 Sent Done Done false
1427 1251 Acked Write 9 Sent Done Done false
1428 1252 Acked Read 7 Sent Done Done false
1429 1253 Acked Read 4 Sent Done Done false
1430 1254 Acked Flush 0 Sent Done Done false
1431 1255 Acked Read 5 Sent Done Done false
1432 1256 Acked Write 9 Sent Done Done false
1433 1257 Acked Write 4 Sent Done Done false
1434 1258 Acked Read 6 Sent Done Done false
1435 1259 Acked Write 6 Sent Done Done false
1436 1260 Acked Write 8 Sent Done Done false
1437 1261 Acked Write 5 Sent Done Done false
1438 1262 Acked Read 4 Sent Done Done false
1439 1263 Acked Read 5 Sent Done Done false
1440 1264 Acked Read 1 Sent Done Done false
1441 1265 Acked Read 1 Sent Done Done false
1442 1266 Acked Read 5 Sent Done Done false
1443 1267 Acked Read 8 Sent Done Done false
1444 1268 Acked Read 7 Sent Done Done false
1445 1269 Acked Read 6 Sent Done Done false
1446 1270 Acked Write 9 Sent Done Done false
1447 1271 Acked Read 5 Sent Done Done false
1448 1272 Acked Read 3 Sent Done Done false
1449 1273 Acked Flush 0 Sent Done Done false
1450 1274 Acked Write 3 Sent Done Done false
1451 1275 Acked Write 9 Sent Done Done false
1452 1276 Acked Write 2 Sent Done Done false
1453 1277 Acked Write 6 Sent Done Done false
1454 1278 Acked Write 5 Sent Done Done false
1455 1279 Acked Write 9 Sent Done Done false
1456 1280 Acked Read 1 Sent Done Done false
1457 1281 Acked Flush 0 Sent Done Done false
1458 1282 Acked Read 9 Sent Done Done false
1459 1283 Acked Write 10 Sent Done Done false
1460 1284 Acked Write 2 Sent Done Done false
1461 1285 Acked Read 5 Sent Done Done false
1462 1286 Acked Write 7 Sent Done Done false
1463 1287 Acked Read 10 Sent Done Done false
1464 1288 Acked Read 10 Sent Done Done false
1465 1289 Acked Read 6 Sent Done Done false
1466 1290 Acked Write 8 Sent Done Done false
1467 1291 Acked Write 3 Sent Done Done false
1468 1292 Acked Write 7 Sent Done Done false
1469 1293 Acked Read 5 Sent Done Done false
1470 1294 Acked Read 3 Sent Done Done false
1471 1295 Acked Read 9 Sent Done Done false
1472 1296 Acked Flush 0 Sent Done Done false
1473 1297 Acked Read 1 Sent Done Done false
1474 1298 Acked Read 5 Sent Done Done false
1475 1299 Acked Read 1 Sent Done Done false
1476 1300 Acked Write 3 Sent Done Done false
1477 1301 Acked Flush 0 Sent Done Done false
1478 STATES DS:0 DS:1 DS:2 TOTAL
1479 Sent 302 0 0 302
1480 Done 0 302 302 604
1481 Skipped 0 0 0 0
1482 Error 0 0 0 0
1483 Last Flush: None Some(JobId(1301)) Some(JobId(1301))
1484 Downstairs last five completed:
1485 Upstairs last five completed: 1301 1300 1299 1298 1297
1486 CLIENT: Up:0 ds:302 act:2
1487 ----------------------------------------------------------------
1488 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
1489 States: Offline Active Active
1490 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1491 1000 Acked Read 1 Sent Done Done false
1492 1001 Acked Read 2 Sent Done Done false
1493 1002 Acked Write 2 Sent Done Done false
1494 1003 Acked Write 5 Sent Done Done false
1495 1004 Acked Write 5 Sent Done Done false
1496 1005 Acked Read 7 Sent Done Done false
1497 1006 Acked Write 4 Sent Done Done false
1498 1007 Acked Write 7 Sent Done Done false
1499 1008 Acked Read 8 Sent Done Done false
1500 1009 Acked Write 6 Sent Done Done false
1501 1010 Acked Read 10 Sent Done Done false
1502 1011 Acked Read 10 Sent Done Done false
1503 1012 Acked Read 5 Sent Done Done false
1504 1013 Acked Write 6 Sent Done Done false
1505 1014 Acked Read 1 Sent Done Done false
1506 1015 Acked Write 1 Sent Done Done false
1507 1016 Acked Write 6 Sent Done Done false
1508 1017 Acked Read 8 Sent Done Done false
1509 1018 Acked Read 8 Sent Done Done false
1510 1019 Acked Write 4 Sent Done Done false
1511 1020 Acked Read 8 Sent Done Done false
1512 1021 Acked Write 10 Sent Done Done false
1513 1022 Acked Write 10 Sent Done Done false
1514 1023 Acked Flush 0 Sent Done Done false
1515 1024 Acked Write 2 Sent Done Done false
1516 1025 Acked Write 1 Sent Done Done false
1517 1026 Acked Write 6 Sent Done Done false
1518 1027 Acked Read 5 Sent Done Done false
1519 1028 Acked Read 10 Sent Done Done false
1520 1029 Acked Write 9 Sent Done Done false
1521 1030 Acked Write 9 Sent Done Done false
1522 1031 Acked Write 8 Sent Done Done false
1523 1032 Acked Read 2 Sent Done Done false
1524 1033 Acked Read 1 Sent Done Done false
1525 1034 Acked Read 1 Sent Done Done false
1526 1035 Acked Read 2 Sent Done Done false
1527 1036 Acked Write 6 Sent Done Done false
1528 1037 Acked Read 2 Sent Done Done false
1529 1038 Acked Read 9 Sent Done Done false
1530 1039 Acked Read 3 Sent Done Done false
1531 1040 Acked Write 1 Sent Done Done false
1532 1041 Acked Read 7 Sent Done Done false
1533 1042 Acked Flush 0 Sent Done Done false
1534 1043 Acked Write 8 Sent Done Done false
1535 1044 Acked Read 1 Sent Done Done false
1536 1045 Acked Write 2 Sent Done Done false
1537 1046 Acked Write 6 Sent Done Done false
1538 1047 Acked Write 9 Sent Done Done false
1539 1048 Acked Read 4 Sent Done Done false
1540 1049 Acked Write 10 Sent Done Done false
1541 1050 Acked Read 3 Sent Done Done false
1542 1051 Acked Write 10 Sent Done Done false
1543 1052 Acked Write 8 Sent Done Done false
1544 1053 Acked Read 1 Sent Done Done false
1545 1054 Acked Write 2 Sent Done Done false
1546 1055 Acked Write 2 Sent Done Done false
1547 1056 Acked Read 1 Sent Done Done false
1548 1057 Acked Write 3 Sent Done Done false
1549 1058 Acked Write 5 Sent Done Done false
1550 1059 Acked Write 1 Sent Done Done false
1551 1060 Acked Read 8 Sent Done Done false
1552 1061 Acked Write 5 Sent Done Done false
1553 1062 Acked Read 6 Sent Done Done false
1554 1063 Acked Read 7 Sent Done Done false
1555 1064 Acked Read 7 Sent Done Done false
1556 1065 Acked Write 1 Sent Done Done false
1557 1066 Acked Read 1 Sent Done Done false
1558 1067 Acked Read 8 Sent Done Done false
1559 1068 Acked Read 6 Sent Done Done false
1560 1069 Acked Read 2 Sent Done Done false
1561 1070 Acked Read 5 Sent Done Done false
1562 1071 Acked Write 2 Sent Done Done false
1563 1072 Acked Read 6 Sent Done Done false
1564 1073 Acked Read 4 Sent Done Done false
1565 1074 Acked Read 6 Sent Done Done false
1566 1075 Acked Write 5 Sent Done Done false
1567 1076 Acked Read 6 Sent Done Done false
1568 1077 Acked Read 5 Sent Done Done false
1569 1078 Acked Write 1 Sent Done Done false
1570 1079 Acked Write 5 Sent Done Done false
1571 1080 Acked Write 10 Sent Done Done false
1572 1081 Acked Read 3 Sent Done Done false
1573 1082 Acked Read 3 Sent Done Done false
1574 1083 Acked Read 2 Sent Done Done false
1575 1084 Acked Read 10 Sent Done Done false
1576 1085 Acked Write 1 Sent Done Done false
1577 1086 Acked Write 7 Sent Done Done false
1578 1087 Acked Write 5 Sent Done Done false
1579 1088 Acked Read 8 Sent Done Done false
1580 1089 Acked Write 4 Sent Done Done false
1581 1090 Acked Read 9 Sent Done Done false
1582 1091 Acked Write 2 Sent Done Done false
1583 1092 Acked Write 8 Sent Done Done false
1584 1093 Acked Read 8 Sent Done Done false
1585 1094 Acked Write 7 Sent Done Done false
1586 1095 Acked Read 1 Sent Done Done false
1587 1096 Acked Read 10 Sent Done Done false
1588 1097 Acked Read 9 Sent Done Done false
1589 1098 Acked Write 2 Sent Done Done false
1590 1099 Acked Write 7 Sent Done Done false
1591 1100 Acked Write 5 Sent Done Done false
1592 1101 Acked Read 1 Sent Done Done false
1593 1102 Acked Flush 0 Sent Done Done false
1594 1103 Acked Write 6 Sent Done Done false
1595 1104 Acked Read 10 Sent Done Done false
1596 1105 Acked Read 5 Sent Done Done false
1597 1106 Acked Read 9 Sent Done Done false
1598 1107 Acked Flush 0 Sent Done Done false
1599 1108 Acked Write 1 Sent Done Done false
1600 1109 Acked Read 10 Sent Done Done false
1601 1110 Acked Read 6 Sent Done Done false
1602 1111 Acked Read 5 Sent Done Done false
1603 1112 Acked Write 10 Sent Done Done false
1604 1113 Acked Read 7 Sent Done Done false
1605 1114 Acked Write 6 Sent Done Done false
1606 1115 Acked Read 5 Sent Done Done false
1607 1116 Acked Read 8 Sent Done Done false
1608 1117 Acked Read 4 Sent Done Done false
1609 1118 Acked Read 10 Sent Done Done false
1610 1119 Acked Write 4 Sent Done Done false
1611 1120 Acked Write 5 Sent Done Done false
1612 1121 Acked Write 5 Sent Done Done false
1613 1122 Acked Read 7 Sent Done Done false
1614 1123 Acked Write 3 Sent Done Done false
1615 1124 Acked Write 1 Sent Done Done false
1616 1125 Acked Write 10 Sent Done Done false
1617 1126 Acked Read 1 Sent Done Done false
1618 1127 Acked Read 3 Sent Done Done false
1619 1128 Acked Write 10 Sent Done Done false
1620 1129 Acked Read 6 Sent Done Done false
1621 1130 Acked Write 8 Sent Done Done false
1622 1131 Acked Read 3 Sent Done Done false
1623 1132 Acked Flush 0 Sent Done Done false
1624 1133 Acked Write 7 Sent Done Done false
1625 1134 Acked Flush 0 Sent Done Done false
1626 1135 Acked Write 1 Sent Done Done false
1627 1136 Acked Read 2 Sent Done Done false
1628 1137 Acked Read 4 Sent Done Done false
1629 1138 Acked Write 9 Sent Done Done false
1630 1139 Acked Read 4 Sent Done Done false
1631 1140 Acked Flush 0 Sent Done Done false
1632 1141 Acked Write 10 Sent Done Done false
1633 1142 Acked Write 4 Sent Done Done false
1634 1143 Acked Read 3 Sent Done Done false
1635 1144 Acked Read 3 Sent Done Done false
1636 1145 Acked Read 7 Sent Done Done false
1637 1146 Acked Read 1 Sent Done Done false
1638 1147 Acked Read 10 Sent Done Done false
1639 1148 Acked Write 8 Sent Done Done false
1640 1149 Acked Read 3 Sent Done Done false
1641 1150 Acked Write 7 Sent Done Done false
1642 1151 Acked Write 9 Sent Done Done false
1643 1152 Acked Read 7 Sent Done Done false
1644 1153 Acked Flush 0 Sent Done Done false
1645 1154 Acked Write 3 Sent Done Done false
1646 1155 Acked Write 9 Sent Done Done false
1647 1156 Acked Read 7 Sent Done Done false
1648 1157 Acked Write 10 Sent Done Done false
1649 1158 Acked Write 5 Sent Done Done false
1650 1159 Acked Write 2 Sent Done Done false
1651 1160 Acked Write 7 Sent Done Done false
1652 1161 Acked Write 10 Sent Done Done false
1653 1162 Acked Read 3 Sent Done Done false
1654 1163 Acked Read 6 Sent Done Done false
1655 1164 Acked Read 4 Sent Done Done false
1656 1165 Acked Write 10 Sent Done Done false
1657 1166 Acked Write 1 Sent Done Done false
1658 1167 Acked Read 7 Sent Done Done false
1659 1168 Acked Read 7 Sent Done Done false
1660 1169 Acked Write 6 Sent Done Done false
1661 1170 Acked Read 2 Sent Done Done false
1662 1171 Acked Read 10 Sent Done Done false
1663 1172 Acked Read 7 Sent Done Done false
1664 1173 Acked Read 5 Sent Done Done false
1665 1174 Acked Read 8 Sent Done Done false
1666 1175 Acked Write 5 Sent Done Done false
1667 1176 Acked Read 3 Sent Done Done false
1668 1177 Acked Write 4 Sent Done Done false
1669 1178 Acked Read 2 Sent Done Done false
1670 1179 Acked Read 8 Sent Done Done false
1671 1180 Acked Read 8 Sent Done Done false
1672 1181 Acked Write 7 Sent Done Done false
1673 1182 Acked Flush 0 Sent Done Done false
1674 1183 Acked Write 8 Sent Done Done false
1675 1184 Acked Read 8 Sent Done Done false
1676 1185 Acked Write 8 Sent Done Done false
1677 1186 Acked Write 5 Sent Done Done false
1678 1187 Acked Write 6 Sent Done Done false
1679 1188 Acked Flush 0 Sent Done Done false
1680 1189 Acked Write 2 Sent Done Done false
1681 1190 Acked Write 3 Sent Done Done false
1682 1191 Acked Write 8 Sent Done Done false
1683 1192 Acked Write 1 Sent Done Done false
1684 1193 Acked Write 8 Sent Done Done false
1685 1194 Acked Write 8 Sent Done Done false
1686 1195 Acked Read 4 Sent Done Done false
1687 1196 Acked Write 1 Sent Done Done false
1688 1197 Acked Read 5 Sent Done Done false
1689 1198 Acked Read 9 Sent Done Done false
1690 1199 Acked Flush 0 Sent Done Done false
1691 1200 Acked Read 8 Sent Done Done false
1692 1201 Acked Read 10 Sent Done Done false
1693 1202 Acked Write 2 Sent Done Done false
1694 1203 Acked Read 5 Sent Done Done false
1695 1204 Acked Read 1 Sent Done Done false
1696 1205 Acked Read 8 Sent Done Done false
1697 1206 Acked Read 9 Sent Done Done false
1698 1207 Acked Write 8 Sent Done Done false
1699 1208 Acked Flush 0 Sent Done Done false
1700 1209 Acked Write 2 Sent Done Done false
1701 1210 Acked Read 10 Sent Done Done false
1702 1211 Acked Write 8 Sent Done Done false
1703 1212 Acked Read 9 Sent Done Done false
1704 1213 Acked Write 5 Sent Done Done false
1705 1214 Acked Read 3 Sent Done Done false
1706 1215 Acked Write 2 Sent Done Done false
1707 1216 Acked Write 7 Sent Done Done false
1708 1217 Acked Write 3 Sent Done Done false
1709 1218 Acked Read 8 Sent Done Done false
1710 1219 Acked Write 8 Sent Done Done false
1711 1220 Acked Write 5 Sent Done Done false
1712 1221 Acked Read 3 Sent Done Done false
1713 1222 Acked Write 4 Sent Done Done false
1714 1223 Acked Read 4 Sent Done Done false
1715 1224 Acked Write 10 Sent Done Done false
1716 1225 Acked Read 10 Sent Done Done false
1717 1226 Acked Read 2 Sent Done Done false
1718 1227 Acked Write 5 Sent Done Done false
1719 1228 Acked Write 1 Sent Done Done false
1720 1229 Acked Read 6 Sent Done Done false
1721 1230 Acked Read 8 Sent Done Done false
1722 1231 Acked Read 4 Sent Done Done false
1723 1232 Acked Read 6 Sent Done Done false
1724 1233 Acked Read 4 Sent Done Done false
1725 1234 Acked Write 1 Sent Done Done false
1726 1235 Acked Read 1 Sent Done Done false
1727 1236 Acked Read 5 Sent Done Done false
1728 1237 Acked Read 7 Sent Done Done false
1729 1238 Acked Read 5 Sent Done Done false
1730 1239 Acked Read 4 Sent Done Done false
1731 1240 Acked Read 6 Sent Done Done false
1732 1241 Acked Write 2 Sent Done Done false
1733 1242 Acked Read 7 Sent Done Done false
1734 1243 Acked Read 1 Sent Done Done false
1735 1244 Acked Read 6 Sent Done Done false
1736 1245 Acked Write 7 Sent Done Done false
1737 1246 Acked Read 8 Sent Done Done false
1738 1247 Acked Read 8 Sent Done Done false
1739 1248 Acked Read 6 Sent Done Done false
1740 1249 Acked Read 7 Sent Done Done false
1741 1250 Acked Read 9 Sent Done Done false
1742 1251 Acked Write 9 Sent Done Done false
1743 1252 Acked Read 7 Sent Done Done false
1744 1253 Acked Read 4 Sent Done Done false
1745 1254 Acked Flush 0 Sent Done Done false
1746 1255 Acked Read 5 Sent Done Done false
1747 1256 Acked Write 9 Sent Done Done false
1748 1257 Acked Write 4 Sent Done Done false
1749 1258 Acked Read 6 Sent Done Done false
1750 1259 Acked Write 6 Sent Done Done false
1751 1260 Acked Write 8 Sent Done Done false
1752 1261 Acked Write 5 Sent Done Done false
1753 1262 Acked Read 4 Sent Done Done false
1754 1263 Acked Read 5 Sent Done Done false
1755 1264 Acked Read 1 Sent Done Done false
1756 1265 Acked Read 1 Sent Done Done false
1757 1266 Acked Read 5 Sent Done Done false
1758 1267 Acked Read 8 Sent Done Done false
1759 1268 Acked Read 7 Sent Done Done false
1760 1269 Acked Read 6 Sent Done Done false
1761 1270 Acked Write 9 Sent Done Done false
1762 1271 Acked Read 5 Sent Done Done false
1763 1272 Acked Read 3 Sent Done Done false
1764 1273 Acked Flush 0 Sent Done Done false
1765 1274 Acked Write 3 Sent Done Done false
1766 1275 Acked Write 9 Sent Done Done false
1767 1276 Acked Write 2 Sent Done Done false
1768 1277 Acked Write 6 Sent Done Done false
1769 1278 Acked Write 5 Sent Done Done false
1770 1279 Acked Write 9 Sent Done Done false
1771 1280 Acked Read 1 Sent Done Done false
1772 1281 Acked Flush 0 Sent Done Done false
1773 1282 Acked Read 9 Sent Done Done false
1774 1283 Acked Write 10 Sent Done Done false
1775 1284 Acked Write 2 Sent Done Done false
1776 1285 Acked Read 5 Sent Done Done false
1777 1286 Acked Write 7 Sent Done Done false
1778 1287 Acked Read 10 Sent Done Done false
1779 1288 Acked Read 10 Sent Done Done false
1780 1289 Acked Read 6 Sent Done Done false
1781 1290 Acked Write 8 Sent Done Done false
1782 1291 Acked Write 3 Sent Done Done false
1783 1292 Acked Write 7 Sent Done Done false
1784 1293 Acked Read 5 Sent Done Done false
1785 1294 Acked Read 3 Sent Done Done false
1786 1295 Acked Read 9 Sent Done Done false
1787 1296 Acked Flush 0 Sent Done Done false
1788 1297 Acked Read 1 Sent Done Done false
1789 1298 Acked Read 5 Sent Done Done false
1790 1299 Acked Read 1 Sent Done Done false
1791 1300 Acked Write 3 Sent Done Done false
1792 1301 Acked Flush 0 Sent Done Done false
1793 STATES DS:0 DS:1 DS:2 TOTAL
1794 Sent 302 0 0 302
1795 Done 0 302 302 604
1796 Skipped 0 0 0 0
1797 Error 0 0 0 0
1798 Last Flush: None Some(JobId(1301)) Some(JobId(1301))
1799 Downstairs last five completed:
1800 Upstairs last five completed: 1301 1300 1299 1298 1297
1801 CLIENT: Up:0 ds:302 act:2
18022025-01-23T18:48:16.971ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
18032025-01-23T18:48:16.971ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34886) = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
18042025-01-23T18:48:16.972ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
18052025-01-23T18:48:16.973ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
18062025-01-23T18:48:16.973ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
18072025-01-23T18:48:16.973ZINFOcrucible: send last flush ID to this DS: None = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
18082025-01-23T18:48:16.973ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
18092025-01-23T18:48:16.973ZINFOcrucible: [0] client re-new 302 jobs since flush None = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
18102025-01-23T18:48:16.973ZINFOcrucible: [0] Replayed 302 jobs since flush: None = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1811 ----------------------------------------------------------------
1812 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1813 Downstairs last five completed: 1301 1300 1299 1298 1297
1814 Upstairs last five completed: 1301 1300 1299 1298 1297
1815 CLIENT: Up:0 ds:0 act:3
1816 Replay: All jobs finished
1817 Main task has work to do, go find it
1818 got dsc Stop(2)
1819 stop 2
1820 Jan 23 18:48:17.223 INFO request completed, latency_us: 339, response_code: 204, uri[8830] Got stop action so:false kr:false
1821 : /stop/cid/2, method: POST, req_id: d4ba356e-fd93-4baa-a137-544e599884e6, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
18222025-01-23T18:48:17.224ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
18232025-01-23T18:48:17.224ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
18242025-01-23T18:48:17.224ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1825 Jan 23 18:48:17.224 INFO request completed, latency_us: 150, response_code: 200, uri: /state/cid/2, method: GET, req_id: afe7bf61-5392-4aae-907e-afaf29fd7c95, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
18262025-01-23T18:48:17.224ZINFOcrucible: client task is exiting = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
1827 [8830] Exited with: ExitStatus(unix_wait_status(9))
1828 [8830][2] reports Exit
1829 Jan 23 18:48:21.226 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/2, method: GET, req_id: 6cf4de0d-b403-4117-86f5-10417249c565, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
1830 001/300 Read block 1950 len 8192
1831 002/300 Read block 2451 len 28672
1832 003/300 Write block 628 len 32768 data: 2 2 2 2 2 2 2 2
1833 004/300 Flush
1834 005/300 Write block 572 len 28672 data: 2 2 2 2 2 2 2
1835 006/300 Read block 4515 len 8192
1836 007/300 Write block 2671 len 24576 data: 2 2 2 2 2 3
1837 008/300 Read block 4871 len 28672
1838 009/300 Read block 1333 len 8192
1839 010/300 Read block 3240 len 24576
1840 011/300 Write block 2776 len 8192 data: 2 2
1841 012/300 Write block 4373 len 20480 data: 3 3 3 3 2
1842 013/300 Read block 4591 len 32768
1843 014/300 Read block 1883 len 16384
1844 015/300 Read block 4814 len 4096
1845 016/300 Flush
1846 017/300 Write block 3245 len 28672 data: 3 2 2 2 2 2 2
1847 018/300 Write block 1158 len 20480 data: 2 2 2 2 2
1848 019/300 Read block 4596 len 20480
1849 020/300 Read block 2465 len 24576
1850 021/300 Write block 4076 len 36864 data: 2 2 2 2 2 3 3 2 2
1851 022/300 Write block 3860 len 12288 data: 3 2 2
1852 023/300 Read block 91 len 24576
1853 024/300 Read block 381 len 24576
1854 025/300 Read block 1091 len 36864
1855 026/300 Read block 1477 len 24576
1856 027/300 Read block 4562 len 40960
1857 028/300 Write block 140 len 4096 data: 2
1858 029/300 Read block 573 len 12288
1859 030/300 Read block 4054 len 12288
1860 031/300 Write block 3616 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1861 032/300 Read block 2171 len 40960
1862 033/300 Write block 1220 len 16384 data: 2 2 2 2
1863 034/300 Write block 4228 len 28672 data: 2 2 2 2 2 2 2
1864 035/300 Write block 404 len 32768 data: 2 2 2 2 2 2 2 2
1865 036/300 Flush
1866 037/300 Write block 1145 len 16384 data: 2 2 2 2
1867 038/300 Read block 722 len 28672
1868 039/300 Read block 2685 len 36864
1869 040/300 Write block 3204 len 4096 data: 2
1870 041/300 Write block 2053 len 36864 data: 2 2 2 2 2 2 2 2 2
1871 042/300 Flush
1872 043/300 Write block 782 len 12288 data: 3 3 3
1873 044/300 Read block 3012 len 32768
1874 045/300 Read block 1462 len 24576
1875 046/300 Read block 829 len 40960
1876 047/300 Read block 551 len 40960
1877 048/300 Read block 4265 len 28672
1878 049/300 Read block 4215 len 24576
1879 050/300 Write block 761 len 8192 data: 3 3
1880 051/300 Flush
1881 052/300 Read block 2188 len 8192
1882 053/300 Flush
1883 054/300 Read block 3732 len 16384
1884 055/300 Flush
1885 056/300 Write block 4784 len 20480 data: 2 2 2 2 2
1886 057/300 Write block 2056 len 16384 data: 3 3 3 3
1887 058/300 Read block 1145 len 28672
1888 059/300 Read block 1928 len 4096
1889 060/300 Read block 243 len 8192
1890 061/300 Read block 301 len 28672
1891 062/300 Read block 696 len 4096
1892 063/300 Write block 2648 len 8192 data: 2 2
1893 064/300 Read block 1515 len 8192
1894 065/300 Write block 3625 len 20480 data: 3 2 2 2 2
1895 066/300 Write block 423 len 36864 data: 2 2 2 2 2 2 2 2 2
1896 067/300 Read block 1763 len 8192
1897 068/300 Flush
1898 069/300 Write block 3622 len 40960 data: 3 3 3 4 3 3 3 3 2 2
1899 070/300 Read block 4610 len 40960
1900 071/300 Read block 3107 len 24576
1901 072/300 Read block 3106 len 12288
1902 073/300 Write block 213 len 24576 data: 2 2 2 2 2 2
1903 074/300 Flush
1904 075/300 Read block 186 len 28672
1905 076/300 Flush
1906 077/300 Write block 2133 len 24576 data: 2 2 2 2 2 2
1907 078/300 Read block 1027 len 12288
1908 079/300 Read block 3247 len 20480
1909 080/300 Read block 4933 len 12288
1910 081/300 Write block 577 len 32768 data: 3 3 2 2 2 2 2 2
1911 082/300 Flush
1912 083/300 Flush
1913 084/300 Read block 4452 len 8192
1914 085/300 Write block 3726 len 28672 data: 2 2 2 2 2 2 2
1915 086/300 Write block 2628 len 32768 data: 2 2 2 2 2 2 2 2
1916 087/300 Read block 328 len 40960
1917 088/300 Write block 4467 len 36864 data: 2 2 2 2 2 2 2 2 2
1918 089/300 Read block 2726 len 12288
1919 090/300 Read block 378 len 32768
1920 091/300 Read block 686 len 12288
1921 092/300 Read block 4163 len 24576
1922 093/300 Read block 159 len 36864
1923 094/300 Write block 2117 len 32768 data: 2 2 2 2 2 2 2 2
1924 095/300 Read block 32 len 40960
1925 096/300 Read block 3000 len 28672
1926 097/300 Read block 1295 len 24576
1927 098/300 Read block 52 len 8192
1928 099/300 Write block 4344 len 4096 data: 2
1929 100/300 Read block 4379 len 8192
1930 101/300 Read block 4457 len 8192
1931 102/300 Read block 21 len 4096
1932 103/300 Flush
1933 104/300 Read block 4783 len 24576
1934 105/300 Write block 1152 len 32768 data: 2 2 2 2 2 2 3 3
1935 106/300 Write block 3137 len 16384 data: 2 2 2 2
1936 107/300 Write block 626 len 4096 data: 2
1937 108/300 Read block 4933 len 32768
1938 109/300 Read block 755 len 32768
1939 110/300 Write block 2652 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1940 111/300 Write block 4677 len 28672 data: 2 2 2 2 2 2 2
1941 112/300 Read block 537 len 40960
1942 113/300 Read block 4909 len 16384
1943 114/300 Write block 1886 len 28672 data: 2 2 2 2 2 2 2
1944 115/300 Write block 4207 len 12288 data: 2 2 2
1945 116/300 Read block 2113 len 28672
1946 117/300 Flush
1947 118/300 Write block 4148 len 28672 data: 4 4 4 3 3 3 2
1948 119/300 Read block 762 len 32768
1949 120/300 Write block 2788 len 32768 data: 2 2 2 2 2 2 2 2
1950 121/300 Read block 4149 len 28672
1951 122/300 Read block 4646 len 28672
1952 123/300 Flush
1953 124/300 Read block 3200 len 8192
1954 125/300 Read block 3948 len 40960
1955 126/300 Write block 3287 len 28672 data: 2 2 2 2 2 2 3
1956 127/300 Read block 1141 len 20480
1957 128/300 Write block 3614 len 24576 data: 2 2 3 3 3 3
1958 129/300 Read block 1849 len 4096
1959 130/300 Write block 4808 len 32768 data: 3 3 3 3 3 2 2 2
1960 131/300 Flush
1961 132/300 Write block 2283 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1962 133/300 Flush
1963 134/300 Read block 4864 len 36864
1964 135/300 Read block 2758 len 12288
1965 136/300 Read block 3438 len 36864
1966 137/300 Write block 3001 len 28672 data: 4 4 3 3 3 3 2
1967 138/300 Read block 2208 len 4096
1968 139/300 Read block 3993 len 28672
1969 140/300 Read block 4558 len 20480
1970 141/300 Write block 3350 len 20480 data: 2 2 2 2 2
1971 142/300 Flush
1972 143/300 Read block 2590 len 4096
1973 144/300 Flush
1974 145/300 Read block 1037 len 28672
1975 146/300 Read block 4706 len 20480
1976 147/300 Write block 3997 len 28672 data: 2 2 2 2 2 2 2
1977 148/300 Read block 584 len 28672
1978 149/300 Write block 1244 len 12288 data: 2 2 2
1979 150/300 Read block 4359 len 20480
1980 151/300 Write block 1257 len 20480 data: 2 2 2 2 2
1981 152/300 Write block 115 len 4096 data: 2
1982 153/300 Write block 265 len 32768 data: 2 2 2 2 2 3 3 3
1983 154/300 Write block 4949 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1984 155/300 Read block 510 len 4096
1985 156/300 Write block 4253 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1986 157/300 Read block 2480 len 40960
1987 158/300 Write block 365 len 12288 data: 2 2 2
1988 159/300 Write block 1981 len 4096 data: 2
1989 160/300 Read block 3506 len 16384
1990 161/300 Write block 567 len 8192 data: 3 3
1991 162/300 Read block 602 len 4096
1992 163/300 Read block 1761 len 20480
1993 164/300 Read block 1901 len 36864
1994 165/300 Write block 1098 len 24576 data: 3 3 3 3 3 3
1995 166/300 Flush
1996 167/300 Write block 546 len 28672 data: 2 2 2 2 2 2 2
1997 168/300 Write block 849 len 4096 data: 2
1998 169/300 Write block 4353 len 4096 data: 2
1999 170/300 Read block 3662 len 16384
2000 171/300 Read block 3019 len 24576
2001 172/300 Read block 2535 len 28672
2002 173/300 Read block 651 len 4096
2003 174/300 Read block 4193 len 40960
2004 175/300 Flush
2005 176/300 Write block 4752 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2006 177/300 Read block 1462 len 4096
2007 178/300 Read block 2684 len 4096
2008 179/300 Read block 3403 len 32768
2009 180/300 Read block 3755 len 32768
2010 181/300 Write block 1731 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2011 182/300 Flush
2012 183/300 Write block 3627 len 4096 data: 4
2013 184/300 Read block 2856 len 32768
2014 185/300 Read block 573 len 36864
2015 186/300 Write block 1549 len 12288 data: 2 2 2
2016 187/300 Read block 3878 len 28672
2017 188/300 Write block 2164 len 24576 data: 2 2 2 2 2 2
2018 189/300 Write block 557 len 20480 data: 2 2 2 2 2
2019 190/300 Write block 4847 len 12288 data: 2 2 2
2020 191/300 Read block 1 len 8192
2021 192/300 Write block 102 len 32768 data: 2 2 2 2 2 2 2 2
2022 193/300 Read block 4874 len 40960
2023 194/300 Write block 960 len 36864 data: 2 2 2 2 2 2 2 2 2
2024 195/300 Read block 1461 len 28672
2025 196/300 Write block 1976 len 24576 data: 2 2 2 2 2 3
2026 197/300 Read block 1122 len 32768
2027 198/300 Write block 1015 len 28672 data: 2 2 2 2 2 2 2
2028 199/300 Read block 3376 len 4096
2029 200/300 Read block 4028 len 36864
2030 201/300 Read block 2749 len 24576
2031 202/300 Write block 1621 len 12288 data: 2 2 2
2032 203/300 Read block 4709 len 24576
2033 204/300 Read block 113 len 36864
2034 205/300 Write block 2784 len 12288 data: 2 2 2
2035 206/300 Write block 3185 len 24576 data: 2 2 2 2 2 2
2036 207/300 Write block 3250 len 16384 data: 3 3 2 2
2037 208/300 Write block 4167 len 36864 data: 2 2 2 2 2 2 2 2 2
2038 209/300 Read block 4790 len 16384
2039 210/300 Read block 126 len 28672
2040 211/300 Read block 947 len 16384
2041 212/300 Read block 1052 len 32768
2042 213/300 Read block 4142 len 20480
2043 214/300 Read block 1419 len 32768
2044 215/300 Write block 1313 len 16384 data: 2 2 2 2
2045 216/300 Write block 4104 len 12288 data: 2 2 2
2046 217/300 Read block 1148 len 4096
2047 218/300 Read block 1587 len 20480
2048 219/300 Write block 4708 len 32768 data: 2 2 2 2 2 2 2 2
2049 220/300 Read block 3530 len 4096
2050 221/300 Read block 1707 len 20480
2051 222/300 Read block 4240 len 8192
2052 223/300 Write block 1794 len 24576 data: 2 2 2 2 2 2
2053 224/300 Write block 4066 len 36864 data: 3 3 3 2 2 2 2 2 2
2054 225/300 Flush
2055 226/300 Read block 17 len 28672
2056 227/300 Read block 4755 len 4096
2057 228/300 Flush
2058 229/300 Write block 4321 len 40960 data: 3 3 3 3 3 3 3 2 2 2
2059 230/300 Read block 4655 len 16384
2060 231/300 Write block 527 len 12288 data: 3 3 3
2061 232/300 Write block 1803 len 36864 data: 2 2 2 2 2 2 3 2 2
2062 233/300 Write block 4025 len 28672 data: 2 2 2 2 2 2 2
2063 234/300 Flush
2064 235/300 Read block 4425 len 20480
2065 236/300 Read block 437 len 24576
2066 237/300 Read block 2331 len 12288
2067 238/300 Read block 4111 len 40960
2068 239/300 Read block 1314 len 32768
2069 240/300 Write block 4662 len 20480 data: 3 3 3 2 2
2070 241/300 Read block 729 len 4096
2071 242/300 Flush
2072 243/300 Write block 1586 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2073 244/300 Read block 4772 len 4096
2074 245/300 Write block 3995 len 36864 data: 2 2 3 3 3 3 3 3 3
2075 246/300 Read block 4707 len 16384
2076 247/300 Read block 2694 len 20480
2077 248/300 Read block 169 len 20480
2078 249/300 Write block 1820 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2079 250/300 Read block 1132 len 28672
2080 251/300 Read block 2328 len 8192
2081 252/300 Read block 3271 len 16384
2082 253/300 Read block 3225 len 28672
2083 254/300 Read block 3677 len 28672
2084 255/300 Read block 4298 len 16384
2085 256/300 Read block 3414 len 36864
2086 257/300 Write block 1019 len 12288 data: 3 3 3
2087 258/300 Write block 1865 len 36864 data: 2 2 2 2 2 2 2 2 2
2088 259/300 Write block 1012 len 12288 data: 3 3 2
2089 260/300 Flush
2090 261/300 Write block 2595 len 12288 data: 2 2 2
2091 262/300 Read block 339 len 16384
2092 263/300 Write block 2615 len 36864 data: 3 3 3 3 3 3 3 3 3
2093 264/300 Read block 3492 len 20480
2094 265/300 Read block 2217 len 32768
2095 266/300 Write block 1966 len 20480 data: 2 2 2 2 2
2096 267/300 Write block 2853 len 32768 data: 3 3 2 2 2 2 2 2
2097 268/300 Read block 3638 len 16384
2098 269/300 Write block 247 len 24576 data: 2 2 2 2 2 2
2099 270/300 Read block 3262 len 28672
2100 271/300 Write block 277 len 40960 data: 3 3 2 2 2 2 2 2 2 2
2101 272/300 Write block 3103 len 32768 data: 2 2 2 2 2 2 2 2
2102 273/300 Write block 1465 len 16384 data: 2 2 2 2
2103 274/300 Write block 675 len 20480 data: 2 2 2 2 2
2104 275/300 Write block 3711 len 28672 data: 2 2 2 2 2 2 2
2105 276/300 Read block 387 len 12288
2106 277/300 Read block 4750 len 12288
2107 278/300 Read block 3847 len 32768
2108 279/300 Read block 2388 len 32768
2109 280/300 Flush
2110 281/300 Read block 910 len 24576
2111 282/300 Write block 1603 len 4096 data: 2
2112 283/300 Write block 2663 len 8192 data: 2 2
2113 284/300 Read block 631 len 28672
2114 285/300 Write block 1076 len 32768 data: 2 2 2 2 2 2 2 2
2115 286/300 Read block 2406 len 36864
2116 287/300 Write block 874 len 12288 data: 2 2 2
2117 288/300 Read block 3118 len 32768
2118 289/300 Write block 995 len 16384 data: 2 2 2 2
2119 290/300 Read block 943 len 20480
2120 291/300 Read block 3122 len 8192
2121 292/300 Read block 2230 len 28672
2122 293/300 Read block 4775 len 28672
2123 294/300 Write block 3863 len 36864 data: 2 2 2 2 2 2 2 2 2
2124 295/300 Read block 3176 len 16384
2125 296/300 Read block 4333 len 40960
2126 297/300 Read block 4261 len 16384
2127 298/300 Write block 2334 len 28672 data: 2 2 2 2 2 2 2
2128 299/300 Read block 3316 len 16384
2129 300/300 Flush
2130 301/300 Write block 4572 len 36864 data: 2 2 2 3 3 3 3 3 3
2131 Main task has work to do, go find it
2132 got dsc Start(2)
2133 start 2
2134 Jan 23 18:48:21.473 INFO request completed, latency_us: 202, response_code: 204, uri: /start/cid/2, method: POST, req_id: b75b3d80-1edc-4872-bc04-eb103f05c161, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
2135 Replay: started 2, returned:Ok(())
2136 ----------------------------------------------------------------
2137 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2138 States: Active Active Offline
2139 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2140 1302 Acked Read 2 Done Done Sent false
2141 1303 Acked Read 7 Done Done Sent false
2142 1304 Acked Write 8 Done Done Sent false
2143 1305 Acked Flush 0 Done Done Sent false
2144 1306 Acked Write 7 Done Done Sent false
2145 1307 Acked Read 2 Done Done Sent false
2146 1308 Acked Write 6 Done Done Sent false
2147 1309 Acked Read 7 Done Done Sent false
2148 1310 Acked Read 2 Done Done Sent false
2149 1311 Acked Read 6 Done Done Sent false
2150 1312 Acked Write 2 Done Done Sent false
2151 1313 Acked Write 5 Done Done Sent false
2152 1314 Acked Read 8 Done Done Sent false
2153 1315 Acked Read 4 Done Done Sent false
2154 1316 Acked Read 1 Done Done Sent false
2155 1317 Acked Flush 0 Done Done Sent false
2156 1318 Acked Write 7 Done Done Sent false
2157 1319 Acked Write 5 Done Done Sent false
2158 1320 Acked Read 5 Done Done Sent false
2159 1321 Acked Read 6 Done Done Sent false
2160 1322 Acked Write 9 Done Done Sent false
2161 1323 Acked Write 3 Done Done Sent false
2162 1324 Acked Read 6 Done Done Sent false
2163 1325 Acked Read 6 Done Done Sent false
2164 1326 Acked Read 9 Done Done Sent false
2165 1327 Acked Read 6 Done Done Sent false
2166 1328 Acked Read 10 Done Done Sent false
2167 1329 Acked Write 1 Done Done Sent false
2168 1330 Acked Read 3 Done Done Sent false
2169 1331 Acked Read 3 Done Done Sent false
2170 1332 Acked Write 10 Done Done Sent false
2171 1333 Acked Read 10 Done Done Sent false
2172 1334 Acked Write 4 Done Done Sent false
2173 1335 Acked Write 7 Done Done Sent false
2174 1336 Acked Write 8 Done Done Sent false
2175 1337 Acked Flush 0 Done Done Sent false
2176 1338 Acked Write 4 Done Done Sent false
2177 1339 Acked Read 7 Done Done Sent false
2178 1340 Acked Read 9 Done Done Sent false
2179 1341 Acked Write 1 Done Done Sent false
2180 1342 Acked Write 9 Done Done Sent false
2181 1343 Acked Flush 0 Done Done Sent false
2182 1344 Acked Write 3 Done Done Sent false
2183 1345 Acked Read 8 Done Done Sent false
2184 1346 Acked Read 6 Done Done Sent false
2185 1347 Acked Read 10 Done Done Sent false
2186 1348 Acked Read 10 Done Done Sent false
2187 1349 Acked Read 7 Done Done Sent false
2188 1350 Acked Read 6 Done Done Sent false
2189 1351 Acked Write 2 Done Done Sent false
2190 1352 Acked Flush 0 Done Done Sent false
2191 1353 Acked Read 2 Done Done Sent false
2192 1354 Acked Flush 0 Done Done Sent false
2193 1355 Acked Read 4 Done Done Sent false
2194 1356 Acked Flush 0 Done Done Sent false
2195 1357 Acked Write 5 Done Done Sent false
2196 1358 Acked Write 4 Done Done Sent false
2197 1359 Acked Read 7 Done Done Sent false
2198 1360 Acked Read 1 Done Done Sent false
2199 1361 Acked Read 2 Done Done Sent false
2200 1362 Acked Read 7 Done Done Sent false
2201 1363 Acked Read 1 Done Done Sent false
2202 1364 Acked Write 2 Done Done Sent false
2203 1365 Acked Read 2 Done Done Sent false
2204 1366 Acked Write 5 Done Done Sent false
2205 1367 Acked Write 9 Done Done Sent false
2206 1368 Acked Read 2 Done Done Sent false
2207 1369 Acked Flush 0 Done Done Sent false
2208 1370 Acked Write 10 Done Done Sent false
2209 1371 Acked Read 10 Done Done Sent false
2210 1372 Acked Read 6 Done Done Sent false
2211 1373 Acked Read 3 Done Done Sent false
2212 1374 Acked Write 6 Done Done Sent false
2213 1375 Acked Flush 0 Done Done Sent false
2214 1376 Acked Read 7 Done Done Sent false
2215 1377 Acked Flush 0 Done Done Sent false
2216 1378 Acked Write 6 Done Done Sent false
2217 1379 Acked Read 3 Done Done Sent false
2218 1380 Acked Read 5 Done Done Sent false
2219 1381 Acked Read 3 Done Done Sent false
2220 1382 Acked Write 8 Done Done Sent false
2221 1383 Acked Flush 0 Done Done Sent false
2222 1384 Acked Flush 0 Done Done Sent false
2223 1385 Acked Read 2 Done Done Sent false
2224 1386 Acked Write 7 Done Done Sent false
2225 1387 Acked Write 8 Done Done Sent false
2226 1388 Acked Read 10 Done Done Sent false
2227 1389 Acked Write 9 Done Done Sent false
2228 1390 Acked Read 3 Done Done Sent false
2229 1391 Acked Read 8 Done Done Sent false
2230 1392 Acked Read 3 Done Done Sent false
2231 1393 Acked Read 6 Done Done Sent false
2232 1394 Acked Read 9 Done Done Sent false
2233 1395 Acked Write 8 Done Done Sent false
2234 1396 Acked Read 10 Done Done Sent false
2235 1397 Acked Read 7 Done Done Sent false
2236 1398 Acked Read 6 Done Done Sent false
2237 1399 Acked Read 2 Done Done Sent false
2238 1400 Acked Write 1 Done Done Sent false
2239 1401 Acked Read 2 Done Done Sent false
2240 1402 Acked Read 2 Done Done Sent false
2241 1403 Acked Read 1 Done Done Sent false
2242 1404 Acked Flush 0 Done Done Sent false
2243 1405 Acked Read 6 Done Done Sent false
2244 1406 Acked Write 8 Done Done Sent false
2245 1407 Acked Write 4 Done Done Sent false
2246 1408 Acked Write 1 Done Done Sent false
2247 1409 Acked Read 8 Done Done Sent false
2248 1410 Acked Read 8 Done Done Sent false
2249 1411 Acked Write 10 Done Done Sent false
2250 1412 Acked Write 7 Done Done Sent false
2251 1413 Acked Read 10 Done Done Sent false
2252 1414 Acked Read 4 Done Done Sent false
2253 1415 Acked Write 7 Done Done Sent false
2254 1416 Acked Write 3 Done Done Sent false
2255 1417 Acked Read 7 Done Done Sent false
2256 1418 Acked Flush 0 Done Done Sent false
2257 1419 Acked Write 7 Done Done Sent false
2258 1420 Acked Read 8 Done Done Sent false
2259 1421 Acked Write 8 Done Done Sent false
2260 1422 Acked Read 7 Done Done Sent false
2261 1423 Acked Read 7 Done Done Sent false
2262 1424 Acked Flush 0 Done Done Sent false
2263 1425 Acked Read 2 Done Done Sent false
2264 1426 Acked Read 10 Done Done Sent false
2265 1427 Acked Write 7 Done Done Sent false
2266 1428 Acked Read 5 Done Done Sent false
2267 1429 Acked Write 6 Done Done Sent false
2268 1430 Acked Read 1 Done Done Sent false
2269 1431 Acked Write 8 Done Done Sent false
2270 1432 Acked Flush 0 Done Done Sent false
2271 1433 Acked Write 10 Done Done Sent false
2272 1434 Acked Flush 0 Done Done Sent false
2273 1435 Acked Read 9 Done Done Sent false
2274 1436 Acked Read 3 Done Done Sent false
2275 1437 Acked Read 9 Done Done Sent false
2276 1438 Acked Write 7 Done Done Sent false
2277 1439 Acked Read 1 Done Done Sent false
2278 1440 Acked Read 7 Done Done Sent false
2279 1441 Acked Read 5 Done Done Sent false
2280 1442 Acked Write 5 Done Done Sent false
2281 1443 Acked Flush 0 Done Done Sent false
2282 1444 Acked Read 1 Done Done Sent false
2283 1445 Acked Flush 0 Done Done Sent false
2284 1446 Acked Read 7 Done Done Sent false
2285 1447 Acked Read 5 Done Done Sent false
2286 1448 Acked Write 7 Done Done Sent false
2287 1449 Acked Read 7 Done Done Sent false
2288 1450 Acked Write 3 Done Done Sent false
2289 1451 Acked Read 5 Done Done Sent false
2290 1452 Acked Write 5 Done Done Sent false
2291 1453 Acked Write 1 Done Done Sent false
2292 1454 Acked Write 8 Done Done Sent false
2293 1455 Acked Write 10 Done Done Sent false
2294 1456 Acked Read 1 Done Done Sent false
2295 1457 Acked Write 10 Done Done Sent false
2296 1458 Acked Read 10 Done Done Sent false
2297 1459 Acked Write 3 Done Done Sent false
2298 1460 Acked Write 1 Done Done Sent false
2299 1461 Acked Read 4 Done Done Sent false
2300 1462 Acked Write 2 Done Done Sent false
2301 1463 Acked Read 1 Done Done Sent false
2302 1464 Acked Read 5 Done Done Sent false
2303 1465 Acked Read 9 Done Done Sent false
2304 1466 Acked Write 6 Done Done Sent false
2305 1467 Acked Flush 0 Done Done Sent false
2306 1468 Acked Write 7 Done Done Sent false
2307 1469 Acked Write 1 Done Done Sent false
2308 1470 Acked Write 1 Done Done Sent false
2309 1471 Acked Read 4 Done Done Sent false
2310 1472 Acked Read 6 Done Done Sent false
2311 1473 Acked Read 7 Done Done Sent false
2312 1474 Acked Read 1 Done Done Sent false
2313 1475 Acked Read 10 Done Done Sent false
2314 1476 Acked Flush 0 Done Done Sent false
2315 1477 Acked Write 10 Done Done Sent false
2316 1478 Acked Read 1 Done Done Sent false
2317 1479 Acked Read 1 Done Done Sent false
2318 1480 Acked Read 8 Done Done Sent false
2319 1481 Acked Read 8 Done Done Sent false
2320 1482 Acked Write 10 Done Done Sent false
2321 1483 Acked Flush 0 Done Done Sent false
2322 1484 Acked Write 1 Done Done Sent false
2323 1485 Acked Read 8 Done Done Sent false
2324 1486 Acked Read 9 Done Done Sent false
2325 1487 Acked Write 3 Done Done Sent false
2326 1488 Acked Read 7 Done Done Sent false
2327 1489 Acked Write 6 Done Done Sent false
2328 1490 Acked Write 5 Done Done Sent false
2329 1491 Acked Write 3 Done Done Sent false
2330 1492 Acked Read 2 Done Done Sent false
2331 1493 Acked Write 8 Done Done Sent false
2332 1494 Acked Read 10 Done Done Sent false
2333 1495 Acked Write 9 Done Done Sent false
2334 1496 Acked Read 7 Done Done Sent false
2335 1497 Acked Write 6 Done Done Sent false
2336 1498 Acked Read 8 Done Done Sent false
2337 1499 Acked Write 7 Done Done Sent false
2338 1500 Acked Read 1 Done Done Sent false
2339 1501 Acked Read 9 Done Done Sent false
2340 1502 Acked Read 6 Done Done Sent false
2341 1503 Acked Write 3 Done Done Sent false
2342 1504 Acked Read 6 Done Done Sent false
2343 1505 Acked Read 9 Done Done Sent false
2344 1506 Acked Write 3 Done Done Sent false
2345 1507 Acked Write 6 Done Done Sent false
2346 1508 Acked Write 4 Done Done Sent false
2347 1509 Acked Write 9 Done Done Sent false
2348 1510 Acked Read 4 Done Done Sent false
2349 1511 Acked Read 7 Done Done Sent false
2350 1512 Acked Read 4 Done Done Sent false
2351 1513 Acked Read 8 Done Done Sent false
2352 1514 Acked Read 5 Done Done Sent false
2353 1515 Acked Read 8 Done Done Sent false
2354 1516 Acked Write 4 Done Done Sent false
2355 1517 Acked Write 3 Done Done Sent false
2356 1518 Acked Read 1 Done Done Sent false
2357 1519 Acked Read 5 Done Done Sent false
2358 1520 Acked Write 8 Done Done Sent false
2359 1521 Acked Read 1 Done Done Sent false
2360 1522 Acked Read 5 Done Done Sent false
2361 1523 Acked Read 2 Done Done Sent false
2362 1524 Acked Write 6 Done Done Sent false
2363 1525 Acked Write 9 Done Done Sent false
2364 1526 Acked Flush 0 Done Done Sent false
2365 1527 Acked Read 7 Done Done Sent false
2366 1528 Acked Read 1 Done Done Sent false
2367 1529 Acked Flush 0 Done Done Sent false
2368 1530 Acked Write 10 Done Done Sent false
2369 1531 Acked Read 4 Done Done Sent false
2370 1532 Acked Write 3 Done Done Sent false
2371 1533 Acked Write 9 Done Done Sent false
2372 1534 Acked Write 7 Done Done Sent false
2373 1535 Acked Flush 0 Done Done Sent false
2374 1536 Acked Read 5 Done Done Sent false
2375 1537 Acked Read 6 Done Done Sent false
2376 1538 Acked Read 3 Done Done Sent false
2377 1539 Acked Read 10 Done Done Sent false
2378 1540 Acked Read 8 Done Done Sent false
2379 1541 Acked Write 5 Done Done Sent false
2380 1542 Acked Read 1 Done Done Sent false
2381 1543 Acked Flush 0 Done Done Sent false
2382 1544 Acked Write 10 Done Done Sent false
2383 1545 Acked Read 1 Done Done Sent false
2384 1546 Acked Write 9 Done Done Sent false
2385 1547 Acked Read 4 Done Done Sent false
2386 1548 Acked Read 5 Done Done Sent false
2387 1549 Acked Read 5 Done Done Sent false
2388 1550 Acked Write 10 Done Done Sent false
2389 1551 Acked Read 7 Done Done Sent false
2390 1552 Acked Read 2 Done Done Sent false
2391 1553 Acked Read 4 Done Done Sent false
2392 1554 Acked Read 7 Done Done Sent false
2393 1555 Acked Read 7 Done Done Sent false
2394 1556 Acked Read 4 Done Done Sent false
2395 1557 Acked Read 9 Done Done Sent false
2396 1558 Acked Write 3 Done Done Sent false
2397 1559 Acked Write 9 Done Done Sent false
2398 1560 Acked Write 3 Done Done Sent false
2399 1561 Acked Flush 0 Done Done Sent false
2400 1562 Acked Write 3 Done Done Sent false
2401 1563 Acked Read 4 Done Done Sent false
2402 1564 Acked Write 9 Done Done Sent false
2403 1565 Acked Read 5 Done Done Sent false
2404 1566 Acked Read 8 Done Done Sent false
2405 1567 Acked Write 5 Done Done Sent false
2406 1568 Acked Write 8 Done Done Sent false
2407 1569 Acked Read 4 Done Done Sent false
2408 1570 Acked Write 6 Done Done Sent false
2409 1571 Acked Read 7 Done Done Sent false
2410 1572 Acked Write 10 Done Done Sent false
2411 1573 Acked Write 8 Done Done Sent false
2412 1574 Acked Write 4 Done Done Sent false
2413 1575 Acked Write 5 Done Done Sent false
2414 1576 Acked Write 7 Done Done Sent false
2415 1577 Acked Read 3 Done Done Sent false
2416 1578 Acked Read 3 Done Done Sent false
2417 1579 Acked Read 8 Done Done Sent false
2418 1580 Acked Read 8 Done Done Sent false
2419 1581 Acked Flush 0 Done Done Sent false
2420 1582 Acked Read 6 Done Done Sent false
2421 1583 Acked Write 1 Done Done Sent false
2422 1584 Acked Write 2 Done Done Sent false
2423 1585 Acked Read 7 Done Done Sent false
2424 1586 Acked Write 8 Done Done Sent false
2425 1587 Acked Read 9 Done Done Sent false
2426 1588 Acked Write 3 Done Done Sent false
2427 1589 Acked Read 8 Done Done Sent false
2428 1590 Acked Write 4 Done Done Sent false
2429 1591 Acked Read 5 Done Done Sent false
2430 1592 Acked Read 2 Done Done Sent false
2431 1593 Acked Read 7 Done Done Sent false
2432 1594 Acked Read 7 Done Done Sent false
2433 1595 Acked Write 9 Done Done Sent false
2434 1596 Acked Read 4 Done Done Sent false
2435 1597 Acked Read 10 Done Done Sent false
2436 1598 Acked Read 4 Done Done Sent false
2437 1599 Acked Write 7 Done Done Sent false
2438 1600 Acked Read 4 Done Done Sent false
2439 1601 Acked Flush 0 Done Done Sent false
2440 1602 Acked Write 9 Done Done Sent false
2441 STATES DS:0 DS:1 DS:2 TOTAL
2442 Sent 0 0 301 301
2443 Done 301 301 0 602
2444 Skipped 0 0 0 0
2445 Error 0 0 0 0
2446 Last Flush: Some(JobId(1601)) Some(JobId(1601)) Some(JobId(1301))
2447 Downstairs last five completed: 1301 1300 1299 1298 1297
2448 Upstairs last five completed: 1602 1601 1600 1599 1598
2449 CLIENT: Up:0 ds:301 act:2
2450 [8830] got start action
2451 [8830] I am going to restart
2452 Starting downstairs at port 8830
2453 Make output file at "/tmp/dsc/downstairs-8830.txt"
2454 [8830][2] reports Starting
2455 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1202)
2456 [8830][2] reports Running
2457 ----------------------------------------------------------------
2458 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
2459 States: Active Active Offline
2460 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2461 1302 Acked Read 2 Done Done Sent false
2462 1303 Acked Read 7 Done Done Sent false
2463 1304 Acked Write 8 Done Done Sent false
2464 1305 Acked Flush 0 Done Done Sent false
2465 1306 Acked Write 7 Done Done Sent false
2466 1307 Acked Read 2 Done Done Sent false
2467 1308 Acked Write 6 Done Done Sent false
2468 1309 Acked Read 7 Done Done Sent false
2469 1310 Acked Read 2 Done Done Sent false
2470 1311 Acked Read 6 Done Done Sent false
2471 1312 Acked Write 2 Done Done Sent false
2472 1313 Acked Write 5 Done Done Sent false
2473 1314 Acked Read 8 Done Done Sent false
2474 1315 Acked Read 4 Done Done Sent false
2475 1316 Acked Read 1 Done Done Sent false
2476 1317 Acked Flush 0 Done Done Sent false
2477 1318 Acked Write 7 Done Done Sent false
2478 1319 Acked Write 5 Done Done Sent false
2479 1320 Acked Read 5 Done Done Sent false
2480 1321 Acked Read 6 Done Done Sent false
2481 1322 Acked Write 9 Done Done Sent false
2482 1323 Acked Write 3 Done Done Sent false
2483 1324 Acked Read 6 Done Done Sent false
2484 1325 Acked Read 6 Done Done Sent false
2485 1326 Acked Read 9 Done Done Sent false
2486 1327 Acked Read 6 Done Done Sent false
2487 1328 Acked Read 10 Done Done Sent false
2488 1329 Acked Write 1 Done Done Sent false
2489 1330 Acked Read 3 Done Done Sent false
2490 1331 Acked Read 3 Done Done Sent false
2491 1332 Acked Write 10 Done Done Sent false
2492 1333 Acked Read 10 Done Done Sent false
2493 1334 Acked Write 4 Done Done Sent false
2494 1335 Acked Write 7 Done Done Sent false
2495 1336 Acked Write 8 Done Done Sent false
2496 1337 Acked Flush 0 Done Done Sent false
2497 1338 Acked Write 4 Done Done Sent false
2498 1339 Acked Read 7 Done Done Sent false
2499 1340 Acked Read 9 Done Done Sent false
2500 1341 Acked Write 1 Done Done Sent false
2501 1342 Acked Write 9 Done Done Sent false
2502 1343 Acked Flush 0 Done Done Sent false
2503 1344 Acked Write 3 Done Done Sent false
2504 1345 Acked Read 8 Done Done Sent false
2505 1346 Acked Read 6 Done Done Sent false
2506 1347 Acked Read 10 Done Done Sent false
2507 1348 Acked Read 10 Done Done Sent false
2508 1349 Acked Read 7 Done Done Sent false
2509 1350 Acked Read 6 Done Done Sent false
2510 1351 Acked Write 2 Done Done Sent false
2511 1352 Acked Flush 0 Done Done Sent false
2512 1353 Acked Read 2 Done Done Sent false
2513 1354 Acked Flush 0 Done Done Sent false
2514 1355 Acked Read 4 Done Done Sent false
2515 1356 Acked Flush 0 Done Done Sent false
2516 1357 Acked Write 5 Done Done Sent false
2517 1358 Acked Write 4 Done Done Sent false
2518 1359 Acked Read 7 Done Done Sent false
2519 1360 Acked Read 1 Done Done Sent false
2520 1361 Acked Read 2 Done Done Sent false
2521 1362 Acked Read 7 Done Done Sent false
2522 1363 Acked Read 1 Done Done Sent false
2523 1364 Acked Write 2 Done Done Sent false
2524 1365 Acked Read 2 Done Done Sent false
2525 1366 Acked Write 5 Done Done Sent false
2526 1367 Acked Write 9 Done Done Sent false
2527 1368 Acked Read 2 Done Done Sent false
2528 1369 Acked Flush 0 Done Done Sent false
2529 1370 Acked Write 10 Done Done Sent false
2530 1371 Acked Read 10 Done Done Sent false
2531 1372 Acked Read 6 Done Done Sent false
2532 1373 Acked Read 3 Done Done Sent false
2533 1374 Acked Write 6 Done Done Sent false
2534 1375 Acked Flush 0 Done Done Sent false
2535 1376 Acked Read 7 Done Done Sent false
2536 1377 Acked Flush 0 Done Done Sent false
2537 1378 Acked Write 6 Done Done Sent false
2538 1379 Acked Read 3 Done Done Sent false
2539 1380 Acked Read 5 Done Done Sent false
2540 1381 Acked Read 3 Done Done Sent false
2541 1382 Acked Write 8 Done Done Sent false
2542 1383 Acked Flush 0 Done Done Sent false
2543 1384 Acked Flush 0 Done Done Sent false
2544 1385 Acked Read 2 Done Done Sent false
2545 1386 Acked Write 7 Done Done Sent false
2546 1387 Acked Write 8 Done Done Sent false
2547 1388 Acked Read 10 Done Done Sent false
2548 1389 Acked Write 9 Done Done Sent false
2549 1390 Acked Read 3 Done Done Sent false
2550 1391 Acked Read 8 Done Done Sent false
2551 1392 Acked Read 3 Done Done Sent false
2552 1393 Acked Read 6 Done Done Sent false
2553 1394 Acked Read 9 Done Done Sent false
2554 1395 Acked Write 8 Done Done Sent false
2555 1396 Acked Read 10 Done Done Sent false
2556 1397 Acked Read 7 Done Done Sent false
2557 1398 Acked Read 6 Done Done Sent false
2558 1399 Acked Read 2 Done Done Sent false
2559 1400 Acked Write 1 Done Done Sent false
2560 1401 Acked Read 2 Done Done Sent false
2561 1402 Acked Read 2 Done Done Sent false
2562 1403 Acked Read 1 Done Done Sent false
2563 1404 Acked Flush 0 Done Done Sent false
2564 1405 Acked Read 6 Done Done Sent false
2565 1406 Acked Write 8 Done Done Sent false
2566 1407 Acked Write 4 Done Done Sent false
2567 1408 Acked Write 1 Done Done Sent false
2568 1409 Acked Read 8 Done Done Sent false
2569 1410 Acked Read 8 Done Done Sent false
2570 1411 Acked Write 10 Done Done Sent false
2571 1412 Acked Write 7 Done Done Sent false
2572 1413 Acked Read 10 Done Done Sent false
2573 1414 Acked Read 4 Done Done Sent false
2574 1415 Acked Write 7 Done Done Sent false
2575 1416 Acked Write 3 Done Done Sent false
2576 1417 Acked Read 7 Done Done Sent false
2577 1418 Acked Flush 0 Done Done Sent false
2578 1419 Acked Write 7 Done Done Sent false
2579 1420 Acked Read 8 Done Done Sent false
2580 1421 Acked Write 8 Done Done Sent false
2581 1422 Acked Read 7 Done Done Sent false
2582 1423 Acked Read 7 Done Done Sent false
2583 1424 Acked Flush 0 Done Done Sent false
2584 1425 Acked Read 2 Done Done Sent false
2585 1426 Acked Read 10 Done Done Sent false
2586 1427 Acked Write 7 Done Done Sent false
2587 1428 Acked Read 5 Done Done Sent false
2588 1429 Acked Write 6 Done Done Sent false
2589 1430 Acked Read 1 Done Done Sent false
2590 1431 Acked Write 8 Done Done Sent false
2591 1432 Acked Flush 0 Done Done Sent false
2592 1433 Acked Write 10 Done Done Sent false
2593 1434 Acked Flush 0 Done Done Sent false
2594 1435 Acked Read 9 Done Done Sent false
2595 1436 Acked Read 3 Done Done Sent false
2596 1437 Acked Read 9 Done Done Sent false
2597 1438 Acked Write 7 Done Done Sent false
2598 1439 Acked Read 1 Done Done Sent false
2599 1440 Acked Read 7 Done Done Sent false
2600 1441 Acked Read 5 Done Done Sent false
2601 1442 Acked Write 5 Done Done Sent false
2602 1443 Acked Flush 0 Done Done Sent false
2603 1444 Acked Read 1 Done Done Sent false
2604 1445 Acked Flush 0 Done Done Sent false
2605 1446 Acked Read 7 Done Done Sent false
2606 1447 Acked Read 5 Done Done Sent false
2607 1448 Acked Write 7 Done Done Sent false
2608 1449 Acked Read 7 Done Done Sent false
2609 1450 Acked Write 3 Done Done Sent false
2610 1451 Acked Read 5 Done Done Sent false
2611 1452 Acked Write 5 Done Done Sent false
2612 1453 Acked Write 1 Done Done Sent false
2613 1454 Acked Write 8 Done Done Sent false
2614 1455 Acked Write 10 Done Done Sent false
2615 1456 Acked Read 1 Done Done Sent false
2616 1457 Acked Write 10 Done Done Sent false
2617 1458 Acked Read 10 Done Done Sent false
2618 1459 Acked Write 3 Done Done Sent false
2619 1460 Acked Write 1 Done Done Sent false
2620 1461 Acked Read 4 Done Done Sent false
2621 1462 Acked Write 2 Done Done Sent false
2622 1463 Acked Read 1 Done Done Sent false
2623 1464 Acked Read 5 Done Done Sent false
2624 1465 Acked Read 9 Done Done Sent false
2625 1466 Acked Write 6 Done Done Sent false
2626 1467 Acked Flush 0 Done Done Sent false
2627 1468 Acked Write 7 Done Done Sent false
2628 1469 Acked Write 1 Done Done Sent false
2629 1470 Acked Write 1 Done Done Sent false
2630 1471 Acked Read 4 Done Done Sent false
2631 1472 Acked Read 6 Done Done Sent false
2632 1473 Acked Read 7 Done Done Sent false
2633 1474 Acked Read 1 Done Done Sent false
2634 1475 Acked Read 10 Done Done Sent false
2635 1476 Acked Flush 0 Done Done Sent false
2636 1477 Acked Write 10 Done Done Sent false
2637 1478 Acked Read 1 Done Done Sent false
2638 1479 Acked Read 1 Done Done Sent false
2639 1480 Acked Read 8 Done Done Sent false
2640 1481 Acked Read 8 Done Done Sent false
2641 1482 Acked Write 10 Done Done Sent false
2642 1483 Acked Flush 0 Done Done Sent false
2643 1484 Acked Write 1 Done Done Sent false
2644 1485 Acked Read 8 Done Done Sent false
2645 1486 Acked Read 9 Done Done Sent false
2646 1487 Acked Write 3 Done Done Sent false
2647 1488 Acked Read 7 Done Done Sent false
2648 1489 Acked Write 6 Done Done Sent false
2649 1490 Acked Write 5 Done Done Sent false
2650 1491 Acked Write 3 Done Done Sent false
2651 1492 Acked Read 2 Done Done Sent false
2652 1493 Acked Write 8 Done Done Sent false
2653 1494 Acked Read 10 Done Done Sent false
2654 1495 Acked Write 9 Done Done Sent false
2655 1496 Acked Read 7 Done Done Sent false
2656 1497 Acked Write 6 Done Done Sent false
2657 1498 Acked Read 8 Done Done Sent false
2658 1499 Acked Write 7 Done Done Sent false
2659 1500 Acked Read 1 Done Done Sent false
2660 1501 Acked Read 9 Done Done Sent false
2661 1502 Acked Read 6 Done Done Sent false
2662 1503 Acked Write 3 Done Done Sent false
2663 1504 Acked Read 6 Done Done Sent false
2664 1505 Acked Read 9 Done Done Sent false
2665 1506 Acked Write 3 Done Done Sent false
2666 1507 Acked Write 6 Done Done Sent false
2667 1508 Acked Write 4 Done Done Sent false
2668 1509 Acked Write 9 Done Done Sent false
2669 1510 Acked Read 4 Done Done Sent false
2670 1511 Acked Read 7 Done Done Sent false
2671 1512 Acked Read 4 Done Done Sent false
2672 1513 Acked Read 8 Done Done Sent false
2673 1514 Acked Read 5 Done Done Sent false
2674 1515 Acked Read 8 Done Done Sent false
2675 1516 Acked Write 4 Done Done Sent false
2676 1517 Acked Write 3 Done Done Sent false
2677 1518 Acked Read 1 Done Done Sent false
2678 1519 Acked Read 5 Done Done Sent false
2679 1520 Acked Write 8 Done Done Sent false
2680 1521 Acked Read 1 Done Done Sent false
2681 1522 Acked Read 5 Done Done Sent false
2682 1523 Acked Read 2 Done Done Sent false
2683 1524 Acked Write 6 Done Done Sent false
2684 1525 Acked Write 9 Done Done Sent false
2685 1526 Acked Flush 0 Done Done Sent false
2686 1527 Acked Read 7 Done Done Sent false
2687 1528 Acked Read 1 Done Done Sent false
2688 1529 Acked Flush 0 Done Done Sent false
2689 1530 Acked Write 10 Done Done Sent false
2690 1531 Acked Read 4 Done Done Sent false
2691 1532 Acked Write 3 Done Done Sent false
2692 1533 Acked Write 9 Done Done Sent false
2693 1534 Acked Write 7 Done Done Sent false
2694 1535 Acked Flush 0 Done Done Sent false
2695 1536 Acked Read 5 Done Done Sent false
2696 1537 Acked Read 6 Done Done Sent false
2697 1538 Acked Read 3 Done Done Sent false
2698 1539 Acked Read 10 Done Done Sent false
2699 1540 Acked Read 8 Done Done Sent false
2700 1541 Acked Write 5 Done Done Sent false
2701 1542 Acked Read 1 Done Done Sent false
2702 1543 Acked Flush 0 Done Done Sent false
2703 1544 Acked Write 10 Done Done Sent false
2704 1545 Acked Read 1 Done Done Sent false
2705 1546 Acked Write 9 Done Done Sent false
2706 1547 Acked Read 4 Done Done Sent false
2707 1548 Acked Read 5 Done Done Sent false
2708 1549 Acked Read 5 Done Done Sent false
2709 1550 Acked Write 10 Done Done Sent false
2710 1551 Acked Read 7 Done Done Sent false
2711 1552 Acked Read 2 Done Done Sent false
2712 1553 Acked Read 4 Done Done Sent false
2713 1554 Acked Read 7 Done Done Sent false
2714 1555 Acked Read 7 Done Done Sent false
2715 1556 Acked Read 4 Done Done Sent false
2716 1557 Acked Read 9 Done Done Sent false
2717 1558 Acked Write 3 Done Done Sent false
2718 1559 Acked Write 9 Done Done Sent false
2719 1560 Acked Write 3 Done Done Sent false
2720 1561 Acked Flush 0 Done Done Sent false
2721 1562 Acked Write 3 Done Done Sent false
2722 1563 Acked Read 4 Done Done Sent false
2723 1564 Acked Write 9 Done Done Sent false
2724 1565 Acked Read 5 Done Done Sent false
2725 1566 Acked Read 8 Done Done Sent false
2726 1567 Acked Write 5 Done Done Sent false
2727 1568 Acked Write 8 Done Done Sent false
2728 1569 Acked Read 4 Done Done Sent false
2729 1570 Acked Write 6 Done Done Sent false
2730 1571 Acked Read 7 Done Done Sent false
2731 1572 Acked Write 10 Done Done Sent false
2732 1573 Acked Write 8 Done Done Sent false
2733 1574 Acked Write 4 Done Done Sent false
2734 1575 Acked Write 5 Done Done Sent false
2735 1576 Acked Write 7 Done Done Sent false
2736 1577 Acked Read 3 Done Done Sent false
2737 1578 Acked Read 3 Done Done Sent false
2738 1579 Acked Read 8 Done Done Sent false
2739 1580 Acked Read 8 Done Done Sent false
2740 1581 Acked Flush 0 Done Done Sent false
2741 1582 Acked Read 6 Done Done Sent false
2742 1583 Acked Write 1 Done Done Sent false
2743 1584 Acked Write 2 Done Done Sent false
2744 1585 Acked Read 7 Done Done Sent false
2745 1586 Acked Write 8 Done Done Sent false
2746 1587 Acked Read 9 Done Done Sent false
2747 1588 Acked Write 3 Done Done Sent false
2748 1589 Acked Read 8 Done Done Sent false
2749 1590 Acked Write 4 Done Done Sent false
2750 1591 Acked Read 5 Done Done Sent false
2751 1592 Acked Read 2 Done Done Sent false
2752 1593 Acked Read 7 Done Done Sent false
2753 1594 Acked Read 7 Done Done Sent false
2754 1595 Acked Write 9 Done Done Sent false
2755 1596 Acked Read 4 Done Done Sent false
2756 1597 Acked Read 10 Done Done Sent false
2757 1598 Acked Read 4 Done Done Sent false
2758 1599 Acked Write 7 Done Done Sent false
2759 1600 Acked Read 4 Done Done Sent false
2760 1601 Acked Flush 0 Done Done Sent false
2761 1602 Acked Write 9 Done Done Sent false
2762 1603 Acked Flush 0 Done Done Sent false
2763 STATES DS:0 DS:1 DS:2 TOTAL
2764 Sent 0 0 302 302
2765 Done 302 302 0 604
2766 Skipped 0 0 0 0
2767 Error 0 0 0 0
2768 Last Flush: Some(JobId(1603)) Some(JobId(1603)) Some(JobId(1301))
2769 Downstairs last five completed: 1301 1300 1299 1298 1297
2770 Upstairs last five completed: 1603 1602 1601 1600 1599
2771 CLIENT: Up:0 ds:302 act:2
27722025-01-23T18:48:27.226ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
27732025-01-23T18:48:27.226ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47570) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
27742025-01-23T18:48:27.226ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
27752025-01-23T18:48:27.227ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
27762025-01-23T18:48:27.227ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
27772025-01-23T18:48:27.227ZINFOcrucible: send last flush ID to this DS: Some(JobId(1301)) = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
27782025-01-23T18:48:27.227ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
27792025-01-23T18:48:27.227ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(1301)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
27802025-01-23T18:48:27.227ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(1301)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2781 ----------------------------------------------------------------
2782 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2783 Downstairs last five completed: 1603 1602 1601 1600 1599
2784 Upstairs last five completed: 1603 1602 1601 1600 1599
2785 CLIENT: Up:0 ds:0 act:3
2786 Replay: All jobs finished
2787 Main task has work to do, go find it
2788 got dsc Stop(1)
2789 stop 1
2790 Jan 23 18:48:29.479 INFO request completed, latency_us: 336, response_code: 204, uri: /stop/cid/1, method: POST, req_id: ce10c387-2c06-49a2-94ff-f251d5165577, remote_addr:[8820] Got stop action so:false kr:false
2791 127.0.0.1:59295, local_addr: 127.0.0.1:9998
27922025-01-23T18:48:29.479ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2793 Jan 23 18:48:29.479 INFO {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:48:29.479812247Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
2794 request completed, latency_us: 114, response_code: 200, uri: /state/cid/1, method: GET, req_id: fbfe66f4-7672-4a73-bad9-db16fe080bc8, {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:48:29.479843473Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
2795 remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
27962025-01-23T18:48:29.479ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
2797 [8820] Exited with: ExitStatus(unix_wait_status(9))
2798 [8820][1] reports Exit
2799 Jan 23 18:48:33.482 INFO request completed, latency_us: 244, response_code: 200, uri: /state/cid/1, method: GET, req_id: 34948835-64a5-4696-bdf7-2c7231ad805e, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
2800 001/300 Write block 4932 len 4096 data: 3
2801 002/300 Flush
2802 003/300 Read block 1661 len 24576
2803 004/300 Write block 4733 len 8192 data: 2 2
2804 005/300 Write block 148 len 16384 data: 2 2 2 2
2805 006/300 Read block 2695 len 4096
2806 007/300 Write block 1393 len 16384 data: 2 2 2 2
2807 008/300 Read block 1919 len 12288
2808 009/300 Flush
2809 010/300 Write block 2206 len 28672 data: 2 2 2 2 2 2 2
2810 011/300 Read block 424 len 12288
2811 012/300 Read block 1399 len 20480
2812 013/300 Read block 3695 len 40960
2813 014/300 Write block 1402 len 28672 data: 2 2 2 2 2 2 2
2814 015/300 Flush
2815 016/300 Read block 4729 len 36864
2816 017/300 Write block 178 len 40960 data: 3 3 2 2 2 2 2 2 2 2
2817 018/300 Read block 120 len 8192
2818 019/300 Write block 3944 len 16384 data: 3 3 3 3
2819 020/300 Flush
2820 021/300 Write block 275 len 12288 data: 3 3 4
2821 022/300 Flush
2822 023/300 Read block 1337 len 20480
2823 024/300 Read block 3699 len 4096
2824 025/300 Write block 3079 len 32768 data: 2 2 2 2 2 2 2 2
2825 026/300 Read block 3275 len 20480
2826 027/300 Read block 1995 len 24576
2827 028/300 Write block 1111 len 36864 data: 2 2 2 2 2 2 2 2 2
2828 029/300 Flush
2829 030/300 Flush
2830 031/300 Read block 1760 len 4096
2831 032/300 Read block 3235 len 4096
2832 033/300 Write block 4964 len 28672 data: 2 2 2 2 2 2 2
2833 034/300 Flush
2834 035/300 Write block 406 len 28672 data: 3 3 3 3 3 3 2
2835 036/300 Write block 4015 len 8192 data: 2 2
2836 037/300 Write block 3057 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2837 038/300 Read block 1078 len 24576
2838 039/300 Read block 4027 len 20480
2839 040/300 Read block 4134 len 20480
2840 041/300 Write block 2677 len 32768 data: 3 3 3 3 3 3 2 2
2841 042/300 Write block 3021 len 24576 data: 2 2 2 2 2 2
2842 043/300 Read block 1646 len 4096
2843 044/300 Write block 4537 len 32768 data: 2 2 2 2 2 2 2 2
2844 045/300 Write block 887 len 8192 data: 2 2
2845 046/300 Write block 3644 len 40960 data: 2 2 2 2 2 2 2 2 2 3
2846 047/300 Write block 1424 len 32768 data: 2 2 2 2 2 2 2 2
2847 048/300 Write block 602 len 4096 data: 2
2848 049/300 Read block 2687 len 4096
2849 050/300 Write block 1275 len 16384 data: 2 2 2 2
2850 051/300 Write block 3361 len 20480 data: 3 3 3 3 3
2851 052/300 Write block 4340 len 40960 data: 2 2 2 2 3 2 2 2 2 2
2852 053/300 Read block 3888 len 4096
2853 054/300 Write block 2466 len 24576 data: 2 2 2 2 2 3
2854 055/300 Write block 4728 len 16384 data: 2 2 2 2
2855 056/300 Flush
2856 057/300 Read block 1282 len 32768
2857 058/300 Read block 1327 len 4096
2858 059/300 Write block 4326 len 28672 data: 4 4 3 3 3 2 2
2859 060/300 Write block 372 len 24576 data: 2 2 2 2 2 2
2860 061/300 Flush
2861 062/300 Read block 1655 len 12288
2862 063/300 Write block 735 len 8192 data: 2 2
2863 064/300 Write block 428 len 28672 data: 3 3 3 3 2 2 2
2864 065/300 Read block 2290 len 28672
2865 066/300 Read block 3390 len 20480
2866 067/300 Read block 1010 len 40960
2867 068/300 Write block 372 len 24576 data: 3 3 3 3 3 3
2868 069/300 Write block 2003 len 28672 data: 2 2 2 2 2 2 2
2869 070/300 Write block 2324 len 36864 data: 2 2 2 2 2 2 3 3 3
2870 071/300 Read block 1860 len 8192
2871 072/300 Read block 1966 len 40960
2872 073/300 Read block 37 len 16384
2873 074/300 Write block 4846 len 12288 data: 2 3 3
2874 075/300 Write block 584 len 4096 data: 3
2875 076/300 Read block 2155 len 32768
2876 077/300 Read block 4916 len 24576
2877 078/300 Write block 4523 len 20480 data: 2 2 2 2 2
2878 079/300 Write block 2995 len 4096 data: 3
2879 080/300 Read block 4983 len 36864
2880 081/300 Read block 2406 len 20480
2881 082/300 Read block 821 len 12288
2882 083/300 Read block 414 len 36864
2883 084/300 Flush
2884 085/300 Read block 2222 len 28672
2885 086/300 Read block 1972 len 40960
2886 087/300 Write block 1807 len 36864 data: 3 3 4 3 3 2 2 2 2
2887 088/300 Write block 1279 len 12288 data: 2 2 2
2888 089/300 Read block 129 len 32768
2889 090/300 Write block 2433 len 36864 data: 2 2 2 2 2 2 2 2 2
2890 091/300 Write block 3666 len 24576 data: 2 2 2 2 2 2
2891 092/300 Read block 385 len 40960
2892 093/300 Write block 4046 len 16384 data: 2 2 2 2
2893 094/300 Write block 3996 len 4096 data: 3
2894 095/300 Read block 2758 len 12288
2895 096/300 Write block 4545 len 8192 data: 2 2
2896 097/300 Write block 319 len 32768 data: 2 2 2 2 2 2 2 2
2897 098/300 Read block 2561 len 12288
2898 099/300 Write block 1390 len 8192 data: 2 2
2899 100/300 Read block 3512 len 16384
2900 101/300 Write block 666 len 20480 data: 2 2 2 2 2
2901 102/300 Read block 2468 len 4096
2902 103/300 Write block 2498 len 16384 data: 2 2 2 2
2903 104/300 Read block 1980 len 16384
2904 105/300 Write block 3046 len 20480 data: 2 2 3 3 3
2905 106/300 Write block 763 len 8192 data: 3 3
2906 107/300 Read block 2469 len 4096
2907 108/300 Read block 2356 len 40960
2908 109/300 Read block 1700 len 8192
2909 110/300 Read block 2241 len 20480
2910 111/300 Read block 2051 len 32768
2911 112/300 Write block 833 len 24576 data: 2 2 2 2 2 2
2912 113/300 Read block 23 len 8192
2913 114/300 Write block 953 len 28672 data: 2 2 2 2 2 2 2
2914 115/300 Write block 542 len 40960 data: 2 2 2 2 3 3 3 3 3 3
2915 116/300 Read block 4040 len 40960
2916 117/300 Write block 644 len 28672 data: 2 2 2 2 2 2 2
2917 118/300 Read block 4376 len 40960
2918 119/300 Flush
2919 120/300 Write block 3442 len 12288 data: 2 2 2
2920 121/300 Read block 2380 len 8192
2921 122/300 Read block 2696 len 40960
2922 123/300 Read block 2858 len 16384
2923 124/300 Flush
2924 125/300 Read block 346 len 24576
2925 126/300 Read block 1473 len 12288
2926 127/300 Read block 2462 len 20480
2927 128/300 Write block 4007 len 24576 data: 2 2 2 2 2 2
2928 129/300 Read block 371 len 40960
2929 130/300 Write block 1307 len 4096 data: 2
2930 131/300 Read block 4845 len 20480
2931 132/300 Write block 1665 len 24576 data: 2 2 2 2 2 2
2932 133/300 Read block 3702 len 36864
2933 134/300 Read block 2173 len 36864
2934 135/300 Read block 4707 len 28672
2935 136/300 Write block 2769 len 32768 data: 2 2 2 2 2 2 2 3
2936 137/300 Write block 3152 len 12288 data: 2 2 2
2937 138/300 Read block 4546 len 20480
2938 139/300 Write block 2695 len 36864 data: 2 2 2 2 2 2 2 2 2
2939 140/300 Read block 2551 len 20480
2940 141/300 Write block 3040 len 24576 data: 3 2 2 2 2 2
2941 142/300 Read block 4583 len 8192
2942 143/300 Read block 3127 len 40960
2943 144/300 Write block 36 len 8192 data: 2 2
2944 145/300 Read block 3645 len 36864
2945 146/300 Read block 2168 len 4096
2946 147/300 Read block 4742 len 4096
2947 148/300 Write block 1602 len 28672 data: 2 3 2 2 2 2 3
2948 149/300 Read block 4857 len 32768
2949 150/300 Write block 2603 len 24576 data: 3 3 3 3 3 3
2950 151/300 Read block 4624 len 4096
2951 152/300 Write block 1968 len 8192 data: 3 3
2952 153/300 Read block 3111 len 36864
2953 154/300 Flush
2954 155/300 Write block 3504 len 40960 data: 2 2 2 2 2 2 3 3 3 3
2955 156/300 Read block 1491 len 4096
2956 157/300 Read block 3749 len 20480
2957 158/300 Read block 3925 len 36864
2958 159/300 Write block 2122 len 24576 data: 3 3 3 2 2 2
2959 160/300 Write block 2156 len 4096 data: 2
2960 161/300 Write block 1489 len 36864 data: 3 3 4 3 2 2 2 2 2
2961 162/300 Write block 3801 len 16384 data: 2 2 2 2
2962 163/300 Read block 1140 len 16384
2963 164/300 Read block 2037 len 12288
2964 165/300 Read block 1182 len 36864
2965 166/300 Flush
2966 167/300 Write block 1125 len 4096 data: 2
2967 168/300 Write block 413 len 20480 data: 2 2 2 2 2
2968 169/300 Flush
2969 170/300 Write block 2597 len 40960 data: 3 2 2 2 2 3 4 4 4 4
2970 171/300 Write block 2436 len 32768 data: 3 3 3 3 3 3 2 2
2971 172/300 Write block 113 len 24576 data: 2 2 3 2 2 2
2972 173/300 Read block 4798 len 8192
2973 174/300 Write block 4274 len 32768 data: 2 2 2 2 2 2 2 2
2974 175/300 Write block 2315 len 24576 data: 3 2 2 2 2 2
2975 176/300 Write block 2128 len 28672 data: 3 3 3 2 2 3 3
2976 177/300 Read block 2549 len 24576
2977 178/300 Write block 776 len 32768 data: 3 3 3 3 3 3 4 4
2978 179/300 Write block 3715 len 36864 data: 3 3 3 2 2 2 2 2 2
2979 180/300 Read block 1001 len 40960
2980 181/300 Read block 4674 len 24576
2981 182/300 Read block 2534 len 36864
2982 183/300 Write block 4237 len 8192 data: 2 2
2983 184/300 Write block 4929 len 28672 data: 3 3 3 4 3 3 3
2984 185/300 Read block 1414 len 16384
2985 186/300 Read block 1621 len 28672
2986 187/300 Write block 259 len 40960 data: 2 2 2 2 2 2 3 3 3 3
2987 188/300 Read block 621 len 40960
2988 189/300 Write block 247 len 12288 data: 3 3 3
2989 190/300 Write block 4484 len 8192 data: 2 2
2990 191/300 Write block 2943 len 36864 data: 2 2 2 2 2 2 2 2 2
2991 192/300 Write block 3310 len 20480 data: 3 3 3 3 2
2992 193/300 Write block 1416 len 32768 data: 2 2 2 2 2 2 2 2
2993 194/300 Write block 4257 len 8192 data: 3 3
2994 195/300 Flush
2995 196/300 Read block 3591 len 16384
2996 197/300 Write block 213 len 16384 data: 3 3 3 3
2997 198/300 Read block 3051 len 36864
2998 199/300 Read block 4454 len 8192
2999 200/300 Read block 2146 len 4096
3000 201/300 Read block 2748 len 12288
3001 202/300 Write block 2516 len 32768 data: 2 2 2 2 2 2 2 3
3002 203/300 Write block 2714 len 28672 data: 2 2 2 2 2 2 2
3003 204/300 Read block 3030 len 4096
3004 205/300 Read block 1206 len 4096
3005 206/300 Write block 3114 len 32768 data: 2 2 2 2 2 2 2 2
3006 207/300 Write block 642 len 24576 data: 2 2 3 3 3 3
3007 208/300 Read block 3430 len 20480
3008 209/300 Read block 2396 len 8192
3009 210/300 Flush
3010 211/300 Flush
3011 212/300 Write block 469 len 20480 data: 3 2 2 2 2
3012 213/300 Write block 4146 len 20480 data: 3 4 5 5 5
3013 214/300 Write block 4219 len 24576 data: 2 2 2 2 2 2
3014 215/300 Write block 534 len 12288 data: 2 2 2
3015 216/300 Write block 3120 len 40960 data: 3 3 2 2 2 2 2 2 2 2
3016 217/300 Read block 1616 len 40960
3017 218/300 Read block 141 len 4096
3018 219/300 Write block 131 len 4096 data: 2
3019 220/300 Read block 647 len 40960
3020 221/300 Write block 780 len 8192 data: 4 4
3021 222/300 Flush
3022 223/300 Write block 83 len 16384 data: 3 2 2 2
3023 224/300 Read block 3250 len 4096
3024 225/300 Write block 1312 len 40960 data: 2 3 3 3 3 2 2 2 2 2
3025 226/300 Read block 4488 len 40960
3026 227/300 Write block 75 len 16384 data: 3 3 3 4
3027 228/300 Read block 4827 len 20480
3028 229/300 Read block 3807 len 8192
3029 230/300 Write block 4486 len 40960 data: 2 2 2 2 2 2 2 2 3 3
3030 231/300 Read block 906 len 12288
3031 232/300 Read block 1160 len 20480
3032 233/300 Read block 1138 len 4096
3033 234/300 Write block 2409 len 8192 data: 4 3
3034 235/300 Write block 3343 len 28672 data: 2 2 2 2 2 2 3
3035 236/300 Read block 551 len 4096
3036 237/300 Read block 1442 len 12288
3037 238/300 Read block 2358 len 8192
3038 239/300 Read block 2378 len 20480
3039 240/300 Write block 483 len 36864 data: 2 2 2 2 2 2 2 3 3
3040 241/300 Read block 1461 len 24576
3041 242/300 Write block 3268 len 28672 data: 2 2 2 2 2 2 2
3042 243/300 Read block 2967 len 24576
3043 244/300 Write block 2867 len 24576 data: 2 2 2 2 2 2
3044 245/300 Flush
3045 246/300 Flush
3046 247/300 Read block 2263 len 24576
3047 248/300 Read block 871 len 40960
3048 249/300 Write block 3652 len 12288 data: 3 4 2
3049 250/300 Write block 670 len 32768 data: 3 2 2 2 2 3 3 3
3050 251/300 Write block 2154 len 20480 data: 2 2 3 2 2
3051 252/300 Read block 3598 len 28672
3052 253/300 Write block 137 len 20480 data: 2 2 2 3 2
3053 254/300 Read block 1597 len 8192
3054 255/300 Write block 3369 len 4096 data: 2
3055 256/300 Read block 4598 len 16384
3056 257/300 Read block 1936 len 8192
3057 258/300 Read block 3888 len 24576
3058 259/300 Write block 4159 len 24576 data: 2 2 2 2 2 2
3059 260/300 Write block 363 len 12288 data: 2 2 3
3060 261/300 Write block 2594 len 12288 data: 2 3 3
3061 262/300 Write block 3085 len 12288 data: 3 3 2
3062 263/300 Read block 1558 len 24576
3063 264/300 Write block 2568 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3064 265/300 Write block 854 len 24576 data: 2 2 2 2 2 2
3065 266/300 Read block 1437 len 12288
3066 267/300 Read block 3694 len 28672
3067 268/300 Read block 4399 len 28672
3068 269/300 Read block 1544 len 20480
3069 270/300 Read block 1774 len 24576
3070 271/300 Read block 4712 len 36864
3071 272/300 Read block 3206 len 36864
3072 273/300 Flush
3073 274/300 Flush
3074 275/300 Read block 3544 len 40960
3075 276/300 Read block 4463 len 20480
3076 277/300 Write block 3290 len 16384 data: 3 3 3 4
3077 278/300 Write block 3638 len 28672 data: 2 2 2 2 2 2 3
3078 279/300 Flush
3079 280/300 Write block 730 len 8192 data: 2 2
3080 281/300 Flush
3081 282/300 Write block 1985 len 24576 data: 3 3 3 3 2 2
3082 283/300 Write block 1980 len 12288 data: 3 4 2
3083 284/300 Write block 2798 len 4096 data: 2
3084 285/300 Write block 1712 len 12288 data: 2 2 2
3085 286/300 Write block 1278 len 8192 data: 3 3
3086 287/300 Read block 787 len 36864
3087 288/300 Write block 1678 len 8192 data: 3 3
3088 289/300 Read block 279 len 4096
3089 290/300 Read block 489 len 20480
3090 291/300 Read block 4552 len 32768
3091 292/300 Read block 763 len 12288
3092 293/300 Write block 1191 len 4096 data: 2
3093 294/300 Read block 4377 len 16384
3094 295/300 Read block 494 len 12288
3095 296/300 Write block 2395 len 12288 data: 2 2 2
3096 297/300 Flush
3097 298/300 Read block 2982 len 40960
3098 299/300 Flush
3099 300/300 Flush
3100 301/300 Read block 609 len 28672
3101 Main task has work to do, go find it
3102 got dsc Start(1)
3103 start 1
3104 Jan 23 18:48:33.705 INFO request completed, latency_us: 178, response_code: 204, uri: /start/cid/1, method: POST, req_id: a364661d-a783-4080-9dfe-d3e47cc840eb, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
3105 Replay: started 1, returned:Ok(())
3106 ----------------------------------------------------------------
3107 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3108 States: Active Offline Active
3109 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3110 1604 Acked Write 1 Done Sent Done false
3111 1605 Acked Flush 0 Done Sent Done false
3112 1606 Acked Read 6 Done Sent Done false
3113 1607 Acked Write 2 Done Sent Done false
3114 1608 Acked Write 4 Done Sent Done false
3115 1609 Acked Read 1 Done Sent Done false
3116 1610 Acked Write 4 Done Sent Done false
3117 1611 Acked Read 3 Done Sent Done false
3118 1612 Acked Flush 0 Done Sent Done false
3119 1613 Acked Write 7 Done Sent Done false
3120 1614 Acked Read 3 Done Sent Done false
3121 1615 Acked Read 5 Done Sent Done false
3122 1616 Acked Read 10 Done Sent Done false
3123 1617 Acked Write 7 Done Sent Done false
3124 1618 Acked Flush 0 Done Sent Done false
3125 1619 Acked Read 9 Done Sent Done false
3126 1620 Acked Write 10 Done Sent Done false
3127 1621 Acked Read 2 Done Sent Done false
3128 1622 Acked Write 4 Done Sent Done false
3129 1623 Acked Flush 0 Done Sent Done false
3130 1624 Acked Write 3 Done Sent Done false
3131 1625 Acked Flush 0 Done Sent Done false
3132 1626 Acked Read 5 Done Sent Done false
3133 1627 Acked Read 1 Done Sent Done false
3134 1628 Acked Write 8 Done Sent Done false
3135 1629 Acked Read 5 Done Sent Done false
3136 1630 Acked Read 6 Done Sent Done false
3137 1631 Acked Write 9 Done Sent Done false
3138 1632 Acked Flush 0 Done Sent Done false
3139 1633 Acked Flush 0 Done Sent Done false
3140 1634 Acked Read 1 Done Sent Done false
3141 1635 Acked Read 1 Done Sent Done false
3142 1636 Acked Write 7 Done Sent Done false
3143 1637 Acked Flush 0 Done Sent Done false
3144 1638 Acked Write 7 Done Sent Done false
3145 1639 Acked Write 2 Done Sent Done false
3146 1640 Acked Write 10 Done Sent Done false
3147 1641 Acked Read 6 Done Sent Done false
3148 1642 Acked Read 5 Done Sent Done false
3149 1643 Acked Read 5 Done Sent Done false
3150 1644 Acked Write 8 Done Sent Done false
3151 1645 Acked Write 6 Done Sent Done false
3152 1646 Acked Read 1 Done Sent Done false
3153 1647 Acked Write 8 Done Sent Done false
3154 1648 Acked Write 2 Done Sent Done false
3155 1649 Acked Write 10 Done Sent Done false
3156 1650 Acked Write 8 Done Sent Done false
3157 1651 Acked Write 1 Done Sent Done false
3158 1652 Acked Read 1 Done Sent Done false
3159 1653 Acked Write 4 Done Sent Done false
3160 1654 Acked Write 5 Done Sent Done false
3161 1655 Acked Write 10 Done Sent Done false
3162 1656 Acked Read 1 Done Sent Done false
3163 1657 Acked Write 6 Done Sent Done false
3164 1658 Acked Write 4 Done Sent Done false
3165 1659 Acked Flush 0 Done Sent Done false
3166 1660 Acked Read 8 Done Sent Done false
3167 1661 Acked Read 1 Done Sent Done false
3168 1662 Acked Write 7 Done Sent Done false
3169 1663 Acked Write 6 Done Sent Done false
3170 1664 Acked Flush 0 Done Sent Done false
3171 1665 Acked Read 3 Done Sent Done false
3172 1666 Acked Write 2 Done Sent Done false
3173 1667 Acked Write 7 Done Sent Done false
3174 1668 Acked Read 7 Done Sent Done false
3175 1669 Acked Read 5 Done Sent Done false
3176 1670 Acked Read 10 Done Sent Done false
3177 1671 Acked Write 6 Done Sent Done false
3178 1672 Acked Write 7 Done Sent Done false
3179 1673 Acked Write 9 Done Sent Done false
3180 1674 Acked Read 2 Done Sent Done false
3181 1675 Acked Read 10 Done Sent Done false
3182 1676 Acked Read 4 Done Sent Done false
3183 1677 Acked Write 3 Done Sent Done false
3184 1678 Acked Write 1 Done Sent Done false
3185 1679 Acked Read 8 Done Sent Done false
3186 1680 Acked Read 6 Done Sent Done false
3187 1681 Acked Write 5 Done Sent Done false
3188 1682 Acked Write 1 Done Sent Done false
3189 1683 Acked Read 9 Done Sent Done false
3190 1684 Acked Read 5 Done Sent Done false
3191 1685 Acked Read 3 Done Sent Done false
3192 1686 Acked Read 9 Done Sent Done false
3193 1687 Acked Flush 0 Done Sent Done false
3194 1688 Acked Read 7 Done Sent Done false
3195 1689 Acked Read 10 Done Sent Done false
3196 1690 Acked Write 9 Done Sent Done false
3197 1691 Acked Write 3 Done Sent Done false
3198 1692 Acked Read 8 Done Sent Done false
3199 1693 Acked Write 9 Done Sent Done false
3200 1694 Acked Write 6 Done Sent Done false
3201 1695 Acked Read 10 Done Sent Done false
3202 1696 Acked Write 4 Done Sent Done false
3203 1697 Acked Write 1 Done Sent Done false
3204 1698 Acked Read 3 Done Sent Done false
3205 1699 Acked Write 2 Done Sent Done false
3206 1700 Acked Write 8 Done Sent Done false
3207 1701 Acked Read 3 Done Sent Done false
3208 1702 Acked Write 2 Done Sent Done false
3209 1703 Acked Read 4 Done Sent Done false
3210 1704 Acked Write 5 Done Sent Done false
3211 1705 Acked Read 1 Done Sent Done false
3212 1706 Acked Write 4 Done Sent Done false
3213 1707 Acked Read 4 Done Sent Done false
3214 1708 Acked Write 5 Done Sent Done false
3215 1709 Acked Write 2 Done Sent Done false
3216 1710 Acked Read 1 Done Sent Done false
3217 1711 Acked Read 10 Done Sent Done false
3218 1712 Acked Read 2 Done Sent Done false
3219 1713 Acked Read 5 Done Sent Done false
3220 1714 Acked Read 8 Done Sent Done false
3221 1715 Acked Write 6 Done Sent Done false
3222 1716 Acked Read 2 Done Sent Done false
3223 1717 Acked Write 7 Done Sent Done false
3224 1718 Acked Write 10 Done Sent Done false
3225 1719 Acked Read 10 Done Sent Done false
3226 1720 Acked Write 7 Done Sent Done false
3227 1721 Acked Read 10 Done Sent Done false
3228 1722 Acked Flush 0 Done Sent Done false
3229 1723 Acked Write 3 Done Sent Done false
3230 1724 Acked Read 2 Done Sent Done false
3231 1725 Acked Read 10 Done Sent Done false
3232 1726 Acked Read 4 Done Sent Done false
3233 1727 Acked Flush 0 Done Sent Done false
3234 1728 Acked Read 6 Done Sent Done false
3235 1729 Acked Read 3 Done Sent Done false
3236 1730 Acked Read 5 Done Sent Done false
3237 1731 Acked Write 6 Done Sent Done false
3238 1732 Acked Read 10 Done Sent Done false
3239 1733 Acked Write 1 Done Sent Done false
3240 1734 Acked Read 5 Done Sent Done false
3241 1735 Acked Write 6 Done Sent Done false
3242 1736 Acked Read 9 Done Sent Done false
3243 1737 Acked Read 9 Done Sent Done false
3244 1738 Acked Read 7 Done Sent Done false
3245 1739 Acked Write 8 Done Sent Done false
3246 1740 Acked Write 3 Done Sent Done false
3247 1741 Acked Read 5 Done Sent Done false
3248 1742 Acked Write 9 Done Sent Done false
3249 1743 Acked Read 5 Done Sent Done false
3250 1744 Acked Write 6 Done Sent Done false
3251 1745 Acked Read 2 Done Sent Done false
3252 1746 Acked Read 10 Done Sent Done false
3253 1747 Acked Write 2 Done Sent Done false
3254 1748 Acked Read 9 Done Sent Done false
3255 1749 Acked Read 1 Done Sent Done false
3256 1750 Acked Read 1 Done Sent Done false
3257 1751 Acked Write 7 Done Sent Done false
3258 1752 Acked Read 8 Done Sent Done false
3259 1753 Acked Write 6 Done Sent Done false
3260 1754 Acked Read 1 Done Sent Done false
3261 1755 Acked Write 2 Done Sent Done false
3262 1756 Acked Read 9 Done Sent Done false
3263 1757 Acked Flush 0 Done Sent Done false
3264 1758 Acked Write 10 Done Sent Done false
3265 1759 Acked Read 1 Done Sent Done false
3266 1760 Acked Read 5 Done Sent Done false
3267 1761 Acked Read 9 Done Sent Done false
3268 1762 Acked Write 6 Done Sent Done false
3269 1763 Acked Write 1 Done Sent Done false
3270 1764 Acked Write 9 Done Sent Done false
3271 1765 Acked Write 4 Done Sent Done false
3272 1766 Acked Read 4 Done Sent Done false
3273 1767 Acked Read 3 Done Sent Done false
3274 1768 Acked Read 9 Done Sent Done false
3275 1769 Acked Flush 0 Done Sent Done false
3276 1770 Acked Write 1 Done Sent Done false
3277 1771 Acked Write 5 Done Sent Done false
3278 1772 Acked Flush 0 Done Sent Done false
3279 1773 Acked Write 10 Done Sent Done false
3280 1774 Acked Write 8 Done Sent Done false
3281 1775 Acked Write 6 Done Sent Done false
3282 1776 Acked Read 2 Done Sent Done false
3283 1777 Acked Write 8 Done Sent Done false
3284 1778 Acked Write 6 Done Sent Done false
3285 1779 Acked Write 7 Done Sent Done false
3286 1780 Acked Read 6 Done Sent Done false
3287 1781 Acked Write 8 Done Sent Done false
3288 1782 Acked Write 9 Done Sent Done false
3289 1783 Acked Read 10 Done Sent Done false
3290 1784 Acked Read 6 Done Sent Done false
3291 1785 Acked Read 9 Done Sent Done false
3292 1786 Acked Write 2 Done Sent Done false
3293 1787 Acked Write 7 Done Sent Done false
3294 1788 Acked Read 4 Done Sent Done false
3295 1789 Acked Read 7 Done Sent Done false
3296 1790 Acked Write 10 Done Sent Done false
3297 1791 Acked Read 10 Done Sent Done false
3298 1792 Acked Write 3 Done Sent Done false
3299 1793 Acked Write 2 Done Sent Done false
3300 1794 Acked Write 9 Done Sent Done false
3301 1795 Acked Write 5 Done Sent Done false
3302 1796 Acked Write 8 Done Sent Done false
3303 1797 Acked Write 2 Done Sent Done false
3304 1798 Acked Flush 0 Done Sent Done false
3305 1799 Acked Read 4 Done Sent Done false
3306 1800 Acked Write 4 Done Sent Done false
3307 1801 Acked Read 9 Done Sent Done false
3308 1802 Acked Read 2 Done Sent Done false
3309 1803 Acked Read 1 Done Sent Done false
3310 1804 Acked Read 3 Done Sent Done false
3311 1805 Acked Write 8 Done Sent Done false
3312 1806 Acked Write 7 Done Sent Done false
3313 1807 Acked Read 1 Done Sent Done false
3314 1808 Acked Read 1 Done Sent Done false
3315 1809 Acked Write 8 Done Sent Done false
3316 1810 Acked Write 6 Done Sent Done false
3317 1811 Acked Read 5 Done Sent Done false
3318 1812 Acked Read 2 Done Sent Done false
3319 1813 Acked Flush 0 Done Sent Done false
3320 1814 Acked Flush 0 Done Sent Done false
3321 1815 Acked Write 5 Done Sent Done false
3322 1816 Acked Write 5 Done Sent Done false
3323 1817 Acked Write 6 Done Sent Done false
3324 1818 Acked Write 3 Done Sent Done false
3325 1819 Acked Write 10 Done Sent Done false
3326 1820 Acked Read 10 Done Sent Done false
3327 1821 Acked Read 1 Done Sent Done false
3328 1822 Acked Write 1 Done Sent Done false
3329 1823 Acked Read 10 Done Sent Done false
3330 1824 Acked Write 2 Done Sent Done false
3331 1825 Acked Flush 0 Done Sent Done false
3332 1826 Acked Write 4 Done Sent Done false
3333 1827 Acked Read 1 Done Sent Done false
3334 1828 Acked Write 10 Done Sent Done false
3335 1829 Acked Read 10 Done Sent Done false
3336 1830 Acked Write 4 Done Sent Done false
3337 1831 Acked Read 5 Done Sent Done false
3338 1832 Acked Read 2 Done Sent Done false
3339 1833 Acked Write 10 Done Sent Done false
3340 1834 Acked Read 3 Done Sent Done false
3341 1835 Acked Read 5 Done Sent Done false
3342 1836 Acked Read 1 Done Sent Done false
3343 1837 Acked Write 2 Done Sent Done false
3344 1838 Acked Write 7 Done Sent Done false
3345 1839 Acked Read 1 Done Sent Done false
3346 1840 Acked Read 3 Done Sent Done false
3347 1841 Acked Read 2 Done Sent Done false
3348 1842 Acked Read 5 Done Sent Done false
3349 1843 Acked Write 9 Done Sent Done false
3350 1844 Acked Read 6 Done Sent Done false
3351 1845 Acked Write 7 Done Sent Done false
3352 1846 Acked Read 6 Done Sent Done false
3353 1847 Acked Write 6 Done Sent Done false
3354 1848 Acked Flush 0 Done Sent Done false
3355 1849 Acked Flush 0 Done Sent Done false
3356 1850 Acked Read 6 Done Sent Done false
3357 1851 Acked Read 10 Done Sent Done false
3358 1852 Acked Write 3 Done Sent Done false
3359 1853 Acked Write 8 Done Sent Done false
3360 1854 Acked Write 5 Done Sent Done false
3361 1855 Acked Read 7 Done Sent Done false
3362 1856 Acked Write 5 Done Sent Done false
3363 1857 Acked Read 2 Done Sent Done false
3364 1858 Acked Write 1 Done Sent Done false
3365 1859 Acked Read 4 Done Sent Done false
3366 1860 Acked Read 2 Done Sent Done false
3367 1861 Acked Read 6 Done Sent Done false
3368 1862 Acked Write 6 Done Sent Done false
3369 1863 Acked Write 3 Done Sent Done false
3370 1864 Acked Write 3 Done Sent Done false
3371 1865 Acked Write 3 Done Sent Done false
3372 1866 Acked Read 6 Done Sent Done false
3373 1867 Acked Write 10 Done Sent Done false
3374 1868 Acked Write 6 Done Sent Done false
3375 1869 Acked Read 3 Done Sent Done false
3376 1870 Acked Read 7 Done Sent Done false
3377 1871 Acked Read 7 Done Sent Done false
3378 1872 Acked Read 5 Done Sent Done false
3379 1873 Acked Read 6 Done Sent Done false
3380 1874 Acked Read 9 Done Sent Done false
3381 1875 Acked Read 9 Done Sent Done false
3382 1876 Acked Flush 0 Done Sent Done false
3383 1877 Acked Flush 0 Done Sent Done false
3384 1878 Acked Read 10 Done Sent Done false
3385 1879 Acked Read 5 Done Sent Done false
3386 1880 Acked Write 4 Done Sent Done false
3387 1881 Acked Write 7 Done Sent Done false
3388 1882 Acked Flush 0 Done Sent Done false
3389 1883 Acked Write 2 Done Sent Done false
3390 1884 Acked Flush 0 Done Sent Done false
3391 1885 Acked Write 6 Done Sent Done false
3392 1886 Acked Write 3 Done Sent Done false
3393 1887 Acked Write 1 Done Sent Done false
3394 1888 Acked Write 3 Done Sent Done false
3395 1889 Acked Write 2 Done Sent Done false
3396 1890 Acked Read 9 Done Sent Done false
3397 1891 Acked Write 2 Done Sent Done false
3398 1892 Acked Read 1 Done Sent Done false
3399 1893 Acked Read 5 Done Sent Done false
3400 1894 Acked Read 8 Done Sent Done false
3401 1895 Acked Read 3 Done Sent Done false
3402 1896 Acked Write 1 Done Sent Done false
3403 1897 Acked Read 4 Done Sent Done false
3404 1898 Acked Read 3 Done Sent Done false
3405 1899 Acked Write 3 Done Sent Done false
3406 1900 Acked Flush 0 Done Sent Done false
3407 1901 Acked Read 10 Done Sent Done false
3408 1902 Acked Flush 0 Done Sent Done false
3409 1903 Acked Flush 0 Done Sent Done false
3410 1904 Acked Read 7 Done Sent Done false
3411 STATES DS:0 DS:1 DS:2 TOTAL
3412 Sent 0 301 0 301
3413 Done 301 0 301 602
3414 Skipped 0 0 0 0
3415 Error 0 0 0 0
3416 Last Flush: Some(JobId(1903)) Some(JobId(1603)) Some(JobId(1903))
3417 Downstairs last five completed: 1603 1602 1601 1600 1599
3418 Upstairs last five completed: 1904 1903 1902 1901 1900
3419 CLIENT: Up:0 ds:301 act:2
3420 [8820] got start action
3421 [8820] I am going to restart
3422 Starting downstairs at port 8820
3423 Make output file at "/tmp/dsc/downstairs-8820.txt"
3424 [8820][1] reports Starting
3425 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1203)
3426 [8820][1] reports Running
3427 ----------------------------------------------------------------
3428 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
3429 States: Active Offline Active
3430 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3431 1604 Acked Write 1 Done Sent Done false
3432 1605 Acked Flush 0 Done Sent Done false
3433 1606 Acked Read 6 Done Sent Done false
3434 1607 Acked Write 2 Done Sent Done false
3435 1608 Acked Write 4 Done Sent Done false
3436 1609 Acked Read 1 Done Sent Done false
3437 1610 Acked Write 4 Done Sent Done false
3438 1611 Acked Read 3 Done Sent Done false
3439 1612 Acked Flush 0 Done Sent Done false
3440 1613 Acked Write 7 Done Sent Done false
3441 1614 Acked Read 3 Done Sent Done false
3442 1615 Acked Read 5 Done Sent Done false
3443 1616 Acked Read 10 Done Sent Done false
3444 1617 Acked Write 7 Done Sent Done false
3445 1618 Acked Flush 0 Done Sent Done false
3446 1619 Acked Read 9 Done Sent Done false
3447 1620 Acked Write 10 Done Sent Done false
3448 1621 Acked Read 2 Done Sent Done false
3449 1622 Acked Write 4 Done Sent Done false
3450 1623 Acked Flush 0 Done Sent Done false
3451 1624 Acked Write 3 Done Sent Done false
3452 1625 Acked Flush 0 Done Sent Done false
3453 1626 Acked Read 5 Done Sent Done false
3454 1627 Acked Read 1 Done Sent Done false
3455 1628 Acked Write 8 Done Sent Done false
3456 1629 Acked Read 5 Done Sent Done false
3457 1630 Acked Read 6 Done Sent Done false
3458 1631 Acked Write 9 Done Sent Done false
3459 1632 Acked Flush 0 Done Sent Done false
3460 1633 Acked Flush 0 Done Sent Done false
3461 1634 Acked Read 1 Done Sent Done false
3462 1635 Acked Read 1 Done Sent Done false
3463 1636 Acked Write 7 Done Sent Done false
3464 1637 Acked Flush 0 Done Sent Done false
3465 1638 Acked Write 7 Done Sent Done false
3466 1639 Acked Write 2 Done Sent Done false
3467 1640 Acked Write 10 Done Sent Done false
3468 1641 Acked Read 6 Done Sent Done false
3469 1642 Acked Read 5 Done Sent Done false
3470 1643 Acked Read 5 Done Sent Done false
3471 1644 Acked Write 8 Done Sent Done false
3472 1645 Acked Write 6 Done Sent Done false
3473 1646 Acked Read 1 Done Sent Done false
3474 1647 Acked Write 8 Done Sent Done false
3475 1648 Acked Write 2 Done Sent Done false
3476 1649 Acked Write 10 Done Sent Done false
3477 1650 Acked Write 8 Done Sent Done false
3478 1651 Acked Write 1 Done Sent Done false
3479 1652 Acked Read 1 Done Sent Done false
3480 1653 Acked Write 4 Done Sent Done false
3481 1654 Acked Write 5 Done Sent Done false
3482 1655 Acked Write 10 Done Sent Done false
3483 1656 Acked Read 1 Done Sent Done false
3484 1657 Acked Write 6 Done Sent Done false
3485 1658 Acked Write 4 Done Sent Done false
3486 1659 Acked Flush 0 Done Sent Done false
3487 1660 Acked Read 8 Done Sent Done false
3488 1661 Acked Read 1 Done Sent Done false
3489 1662 Acked Write 7 Done Sent Done false
3490 1663 Acked Write 6 Done Sent Done false
3491 1664 Acked Flush 0 Done Sent Done false
3492 1665 Acked Read 3 Done Sent Done false
3493 1666 Acked Write 2 Done Sent Done false
3494 1667 Acked Write 7 Done Sent Done false
3495 1668 Acked Read 7 Done Sent Done false
3496 1669 Acked Read 5 Done Sent Done false
3497 1670 Acked Read 10 Done Sent Done false
3498 1671 Acked Write 6 Done Sent Done false
3499 1672 Acked Write 7 Done Sent Done false
3500 1673 Acked Write 9 Done Sent Done false
3501 1674 Acked Read 2 Done Sent Done false
3502 1675 Acked Read 10 Done Sent Done false
3503 1676 Acked Read 4 Done Sent Done false
3504 1677 Acked Write 3 Done Sent Done false
3505 1678 Acked Write 1 Done Sent Done false
3506 1679 Acked Read 8 Done Sent Done false
3507 1680 Acked Read 6 Done Sent Done false
3508 1681 Acked Write 5 Done Sent Done false
3509 1682 Acked Write 1 Done Sent Done false
3510 1683 Acked Read 9 Done Sent Done false
3511 1684 Acked Read 5 Done Sent Done false
3512 1685 Acked Read 3 Done Sent Done false
3513 1686 Acked Read 9 Done Sent Done false
3514 1687 Acked Flush 0 Done Sent Done false
3515 1688 Acked Read 7 Done Sent Done false
3516 1689 Acked Read 10 Done Sent Done false
3517 1690 Acked Write 9 Done Sent Done false
3518 1691 Acked Write 3 Done Sent Done false
3519 1692 Acked Read 8 Done Sent Done false
3520 1693 Acked Write 9 Done Sent Done false
3521 1694 Acked Write 6 Done Sent Done false
3522 1695 Acked Read 10 Done Sent Done false
3523 1696 Acked Write 4 Done Sent Done false
3524 1697 Acked Write 1 Done Sent Done false
3525 1698 Acked Read 3 Done Sent Done false
3526 1699 Acked Write 2 Done Sent Done false
3527 1700 Acked Write 8 Done Sent Done false
3528 1701 Acked Read 3 Done Sent Done false
3529 1702 Acked Write 2 Done Sent Done false
3530 1703 Acked Read 4 Done Sent Done false
3531 1704 Acked Write 5 Done Sent Done false
3532 1705 Acked Read 1 Done Sent Done false
3533 1706 Acked Write 4 Done Sent Done false
3534 1707 Acked Read 4 Done Sent Done false
3535 1708 Acked Write 5 Done Sent Done false
3536 1709 Acked Write 2 Done Sent Done false
3537 1710 Acked Read 1 Done Sent Done false
3538 1711 Acked Read 10 Done Sent Done false
3539 1712 Acked Read 2 Done Sent Done false
3540 1713 Acked Read 5 Done Sent Done false
3541 1714 Acked Read 8 Done Sent Done false
3542 1715 Acked Write 6 Done Sent Done false
3543 1716 Acked Read 2 Done Sent Done false
3544 1717 Acked Write 7 Done Sent Done false
3545 1718 Acked Write 10 Done Sent Done false
3546 1719 Acked Read 10 Done Sent Done false
3547 1720 Acked Write 7 Done Sent Done false
3548 1721 Acked Read 10 Done Sent Done false
3549 1722 Acked Flush 0 Done Sent Done false
3550 1723 Acked Write 3 Done Sent Done false
3551 1724 Acked Read 2 Done Sent Done false
3552 1725 Acked Read 10 Done Sent Done false
3553 1726 Acked Read 4 Done Sent Done false
3554 1727 Acked Flush 0 Done Sent Done false
3555 1728 Acked Read 6 Done Sent Done false
3556 1729 Acked Read 3 Done Sent Done false
3557 1730 Acked Read 5 Done Sent Done false
3558 1731 Acked Write 6 Done Sent Done false
3559 1732 Acked Read 10 Done Sent Done false
3560 1733 Acked Write 1 Done Sent Done false
3561 1734 Acked Read 5 Done Sent Done false
3562 1735 Acked Write 6 Done Sent Done false
3563 1736 Acked Read 9 Done Sent Done false
3564 1737 Acked Read 9 Done Sent Done false
3565 1738 Acked Read 7 Done Sent Done false
3566 1739 Acked Write 8 Done Sent Done false
3567 1740 Acked Write 3 Done Sent Done false
3568 1741 Acked Read 5 Done Sent Done false
3569 1742 Acked Write 9 Done Sent Done false
3570 1743 Acked Read 5 Done Sent Done false
3571 1744 Acked Write 6 Done Sent Done false
3572 1745 Acked Read 2 Done Sent Done false
3573 1746 Acked Read 10 Done Sent Done false
3574 1747 Acked Write 2 Done Sent Done false
3575 1748 Acked Read 9 Done Sent Done false
3576 1749 Acked Read 1 Done Sent Done false
3577 1750 Acked Read 1 Done Sent Done false
3578 1751 Acked Write 7 Done Sent Done false
3579 1752 Acked Read 8 Done Sent Done false
3580 1753 Acked Write 6 Done Sent Done false
3581 1754 Acked Read 1 Done Sent Done false
3582 1755 Acked Write 2 Done Sent Done false
3583 1756 Acked Read 9 Done Sent Done false
3584 1757 Acked Flush 0 Done Sent Done false
3585 1758 Acked Write 10 Done Sent Done false
3586 1759 Acked Read 1 Done Sent Done false
3587 1760 Acked Read 5 Done Sent Done false
3588 1761 Acked Read 9 Done Sent Done false
3589 1762 Acked Write 6 Done Sent Done false
3590 1763 Acked Write 1 Done Sent Done false
3591 1764 Acked Write 9 Done Sent Done false
3592 1765 Acked Write 4 Done Sent Done false
3593 1766 Acked Read 4 Done Sent Done false
3594 1767 Acked Read 3 Done Sent Done false
3595 1768 Acked Read 9 Done Sent Done false
3596 1769 Acked Flush 0 Done Sent Done false
3597 1770 Acked Write 1 Done Sent Done false
3598 1771 Acked Write 5 Done Sent Done false
3599 1772 Acked Flush 0 Done Sent Done false
3600 1773 Acked Write 10 Done Sent Done false
3601 1774 Acked Write 8 Done Sent Done false
3602 1775 Acked Write 6 Done Sent Done false
3603 1776 Acked Read 2 Done Sent Done false
3604 1777 Acked Write 8 Done Sent Done false
3605 1778 Acked Write 6 Done Sent Done false
3606 1779 Acked Write 7 Done Sent Done false
3607 1780 Acked Read 6 Done Sent Done false
3608 1781 Acked Write 8 Done Sent Done false
3609 1782 Acked Write 9 Done Sent Done false
3610 1783 Acked Read 10 Done Sent Done false
3611 1784 Acked Read 6 Done Sent Done false
3612 1785 Acked Read 9 Done Sent Done false
3613 1786 Acked Write 2 Done Sent Done false
3614 1787 Acked Write 7 Done Sent Done false
3615 1788 Acked Read 4 Done Sent Done false
3616 1789 Acked Read 7 Done Sent Done false
3617 1790 Acked Write 10 Done Sent Done false
3618 1791 Acked Read 10 Done Sent Done false
3619 1792 Acked Write 3 Done Sent Done false
3620 1793 Acked Write 2 Done Sent Done false
3621 1794 Acked Write 9 Done Sent Done false
3622 1795 Acked Write 5 Done Sent Done false
3623 1796 Acked Write 8 Done Sent Done false
3624 1797 Acked Write 2 Done Sent Done false
3625 1798 Acked Flush 0 Done Sent Done false
3626 1799 Acked Read 4 Done Sent Done false
3627 1800 Acked Write 4 Done Sent Done false
3628 1801 Acked Read 9 Done Sent Done false
3629 1802 Acked Read 2 Done Sent Done false
3630 1803 Acked Read 1 Done Sent Done false
3631 1804 Acked Read 3 Done Sent Done false
3632 1805 Acked Write 8 Done Sent Done false
3633 1806 Acked Write 7 Done Sent Done false
3634 1807 Acked Read 1 Done Sent Done false
3635 1808 Acked Read 1 Done Sent Done false
3636 1809 Acked Write 8 Done Sent Done false
3637 1810 Acked Write 6 Done Sent Done false
3638 1811 Acked Read 5 Done Sent Done false
3639 1812 Acked Read 2 Done Sent Done false
3640 1813 Acked Flush 0 Done Sent Done false
3641 1814 Acked Flush 0 Done Sent Done false
3642 1815 Acked Write 5 Done Sent Done false
3643 1816 Acked Write 5 Done Sent Done false
3644 1817 Acked Write 6 Done Sent Done false
3645 1818 Acked Write 3 Done Sent Done false
3646 1819 Acked Write 10 Done Sent Done false
3647 1820 Acked Read 10 Done Sent Done false
3648 1821 Acked Read 1 Done Sent Done false
3649 1822 Acked Write 1 Done Sent Done false
3650 1823 Acked Read 10 Done Sent Done false
3651 1824 Acked Write 2 Done Sent Done false
3652 1825 Acked Flush 0 Done Sent Done false
3653 1826 Acked Write 4 Done Sent Done false
3654 1827 Acked Read 1 Done Sent Done false
3655 1828 Acked Write 10 Done Sent Done false
3656 1829 Acked Read 10 Done Sent Done false
3657 1830 Acked Write 4 Done Sent Done false
3658 1831 Acked Read 5 Done Sent Done false
3659 1832 Acked Read 2 Done Sent Done false
3660 1833 Acked Write 10 Done Sent Done false
3661 1834 Acked Read 3 Done Sent Done false
3662 1835 Acked Read 5 Done Sent Done false
3663 1836 Acked Read 1 Done Sent Done false
3664 1837 Acked Write 2 Done Sent Done false
3665 1838 Acked Write 7 Done Sent Done false
3666 1839 Acked Read 1 Done Sent Done false
3667 1840 Acked Read 3 Done Sent Done false
3668 1841 Acked Read 2 Done Sent Done false
3669 1842 Acked Read 5 Done Sent Done false
3670 1843 Acked Write 9 Done Sent Done false
3671 1844 Acked Read 6 Done Sent Done false
3672 1845 Acked Write 7 Done Sent Done false
3673 1846 Acked Read 6 Done Sent Done false
3674 1847 Acked Write 6 Done Sent Done false
3675 1848 Acked Flush 0 Done Sent Done false
3676 1849 Acked Flush 0 Done Sent Done false
3677 1850 Acked Read 6 Done Sent Done false
3678 1851 Acked Read 10 Done Sent Done false
3679 1852 Acked Write 3 Done Sent Done false
3680 1853 Acked Write 8 Done Sent Done false
3681 1854 Acked Write 5 Done Sent Done false
3682 1855 Acked Read 7 Done Sent Done false
3683 1856 Acked Write 5 Done Sent Done false
3684 1857 Acked Read 2 Done Sent Done false
3685 1858 Acked Write 1 Done Sent Done false
3686 1859 Acked Read 4 Done Sent Done false
3687 1860 Acked Read 2 Done Sent Done false
3688 1861 Acked Read 6 Done Sent Done false
3689 1862 Acked Write 6 Done Sent Done false
3690 1863 Acked Write 3 Done Sent Done false
3691 1864 Acked Write 3 Done Sent Done false
3692 1865 Acked Write 3 Done Sent Done false
3693 1866 Acked Read 6 Done Sent Done false
3694 1867 Acked Write 10 Done Sent Done false
3695 1868 Acked Write 6 Done Sent Done false
3696 1869 Acked Read 3 Done Sent Done false
3697 1870 Acked Read 7 Done Sent Done false
3698 1871 Acked Read 7 Done Sent Done false
3699 1872 Acked Read 5 Done Sent Done false
3700 1873 Acked Read 6 Done Sent Done false
3701 1874 Acked Read 9 Done Sent Done false
3702 1875 Acked Read 9 Done Sent Done false
3703 1876 Acked Flush 0 Done Sent Done false
3704 1877 Acked Flush 0 Done Sent Done false
3705 1878 Acked Read 10 Done Sent Done false
3706 1879 Acked Read 5 Done Sent Done false
3707 1880 Acked Write 4 Done Sent Done false
3708 1881 Acked Write 7 Done Sent Done false
3709 1882 Acked Flush 0 Done Sent Done false
3710 1883 Acked Write 2 Done Sent Done false
3711 1884 Acked Flush 0 Done Sent Done false
3712 1885 Acked Write 6 Done Sent Done false
3713 1886 Acked Write 3 Done Sent Done false
3714 1887 Acked Write 1 Done Sent Done false
3715 1888 Acked Write 3 Done Sent Done false
3716 1889 Acked Write 2 Done Sent Done false
3717 1890 Acked Read 9 Done Sent Done false
3718 1891 Acked Write 2 Done Sent Done false
3719 1892 Acked Read 1 Done Sent Done false
3720 1893 Acked Read 5 Done Sent Done false
3721 1894 Acked Read 8 Done Sent Done false
3722 1895 Acked Read 3 Done Sent Done false
3723 1896 Acked Write 1 Done Sent Done false
3724 1897 Acked Read 4 Done Sent Done false
3725 1898 Acked Read 3 Done Sent Done false
3726 1899 Acked Write 3 Done Sent Done false
3727 1900 Acked Flush 0 Done Sent Done false
3728 1901 Acked Read 10 Done Sent Done false
3729 1902 Acked Flush 0 Done Sent Done false
3730 1903 Acked Flush 0 Done Sent Done false
3731 1904 Acked Read 7 Done Sent Done false
3732 1905 Acked Flush 0 Done Sent Done false
3733 STATES DS:0 DS:1 DS:2 TOTAL
3734 Sent 0 302 0 302
3735 Done 302 0 302 604
3736 Skipped 0 0 0 0
3737 Error 0 0 0 0
3738 Last Flush: Some(JobId(1905)) Some(JobId(1603)) Some(JobId(1905))
3739 Downstairs last five completed: 1603 1602 1601 1600 1599
3740 Upstairs last five completed: 1905 1904 1903 1902 1901
3741 CLIENT: Up:0 ds:302 act:2
37422025-01-23T18:48:39.480ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
37432025-01-23T18:48:39.480ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38209) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
37442025-01-23T18:48:39.481ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
37452025-01-23T18:48:39.482ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
37462025-01-23T18:48:39.482ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
37472025-01-23T18:48:39.482ZINFOcrucible: send last flush ID to this DS: Some(JobId(1603)) = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
37482025-01-23T18:48:39.482ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
37492025-01-23T18:48:39.482ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(1603)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
37502025-01-23T18:48:39.482ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(1603)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
3751 ----------------------------------------------------------------
3752 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3753 Downstairs last five completed: 1905 1904 1903 1902 1901
3754 Upstairs last five completed: 1905 1904 1903 1902 1901
3755 CLIENT: Up:0 ds:0 act:3
3756 Replay: All jobs finished
3757 Main task has work to do, go find it
3758 got dsc Stop(1)
3759 stop 1
3760 Jan 23 18:48:41.710 INFO request completed[8820] Got stop action so:false kr:false
3761 , latency_us: 307, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 2e27ad6e-1213-492d-8ebb-0f1ac7f28a16, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
37622025-01-23T18:48:41.711ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
3763 Jan 23 18:48:41.711 INFO{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:48:41.711666245Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
37642025-01-23T18:48:41.711ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
3765 request completed, latency_us: 111, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2ad12493-1bb7-4da5-bd48-22b378a9a34e, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T18:48:41.711777232Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
3766 9998
3767 [8820] Exited with: ExitStatus(unix_wait_status(9))
3768 [8820][1] reports Exit
3769 Jan 23 18:48:45.713 INFO request completed, latency_us: 255, response_code: 200, uri: /state/cid/1, method: GET, req_id: d2c0dfd9-82be-4025-8ed2-5deaa87280ce, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
3770 001/300 Read block 4749 len 28672
3771 002/300 Read block 2764 len 36864
3772 003/300 Write block 1737 len 8192 data: 3 3
3773 004/300 Read block 1682 len 4096
3774 005/300 Read block 1823 len 20480
3775 006/300 Write block 492 len 12288 data: 3 3 3
3776 007/300 Read block 942 len 20480
3777 008/300 Write block 3019 len 36864 data: 2 2 3 3 3 3 3 3 2
3778 009/300 Read block 1375 len 28672
3779 010/300 Read block 2543 len 40960
3780 011/300 Read block 4190 len 24576
3781 012/300 Read block 4435 len 20480
3782 013/300 Write block 1151 len 12288 data: 2 3 3
3783 014/300 Read block 63 len 32768
3784 015/300 Read block 524 len 28672
3785 016/300 Read block 624 len 32768
3786 017/300 Read block 806 len 4096
3787 018/300 Write block 2585 len 28672 data: 2 2 2 2 2 2 2
3788 019/300 Write block 4918 len 4096 data: 3
3789 020/300 Write block 4328 len 12288 data: 4 4 4
3790 021/300 Write block 3645 len 16384 data: 3 3 3 3
3791 022/300 Read block 1400 len 36864
3792 023/300 Read block 1028 len 12288
3793 024/300 Read block 1098 len 24576
3794 025/300 Write block 2586 len 4096 data: 3
3795 026/300 Write block 1805 len 4096 data: 3
3796 027/300 Write block 3812 len 36864 data: 2 2 2 2 2 2 2 2 2
3797 028/300 Write block 4761 len 28672 data: 3 2 2 2 2 2 2
3798 029/300 Read block 1562 len 36864
3799 030/300 Write block 2919 len 4096 data: 2
3800 031/300 Write block 4750 len 8192 data: 2 2
3801 032/300 Read block 4492 len 40960
3802 033/300 Read block 3962 len 4096
3803 034/300 Write block 3230 len 28672 data: 2 2 2 2 2 2 3
3804 035/300 Read block 4496 len 12288
3805 036/300 Write block 2287 len 4096 data: 3
3806 037/300 Read block 4659 len 28672
3807 038/300 Write block 32 len 36864 data: 2 2 2 2 3 3 2 2 2
3808 039/300 Read block 2445 len 12288
3809 040/300 Read block 1210 len 4096
3810 041/300 Write block 3252 len 40960 data: 3 3 2 2 2 2 2 2 2 2
3811 042/300 Write block 2765 len 16384 data: 2 2 2 2
3812 043/300 Read block 2550 len 4096
3813 044/300 Read block 3870 len 12288
3814 045/300 Write block 3633 len 8192 data: 2 2
3815 046/300 Read block 1038 len 16384
3816 047/300 Flush
3817 048/300 Flush
3818 049/300 Write block 74 len 28672 data: 3 4 4 4 5 4 3
3819 050/300 Read block 1691 len 40960
3820 051/300 Write block 667 len 20480 data: 3 3 3 4 3
3821 052/300 Write block 3500 len 8192 data: 2 2
3822 053/300 Write block 4304 len 36864 data: 2 2 2 2 2 2 2 2 2
3823 054/300 Write block 4306 len 40960 data: 3 3 3 3 3 3 3 2 2 2
3824 055/300 Read block 3662 len 8192
3825 056/300 Read block 996 len 24576
3826 057/300 Read block 4685 len 36864
3827 058/300 Write block 1903 len 12288 data: 3 3 2
3828 059/300 Write block 758 len 28672 data: 2 2 3 4 4 4 4
3829 060/300 Flush
3830 061/300 Read block 3730 len 24576
3831 062/300 Write block 3028 len 24576 data: 2 2 3 3 3 3
3832 063/300 Write block 64 len 32768 data: 2 2 2 2 2 2 2 2
3833 064/300 Write block 3620 len 28672 data: 3 3 4 4 4 5 4
3834 065/300 Read block 4644 len 40960
3835 066/300 Write block 815 len 36864 data: 2 2 2 2 2 2 2 2 2
3836 067/300 Write block 534 len 4096 data: 3
3837 068/300 Read block 2309 len 20480
3838 069/300 Read block 1126 len 12288
3839 070/300 Write block 4904 len 16384 data: 3 3 3 2
3840 071/300 Flush
3841 072/300 Write block 2252 len 20480 data: 2 2 2 2 2
3842 073/300 Write block 931 len 32768 data: 3 3 3 3 3 3 3 3
3843 074/300 Read block 2282 len 8192
3844 075/300 Write block 2741 len 8192 data: 3 3
3845 076/300 Read block 2609 len 32768
3846 077/300 Read block 2044 len 8192
3847 078/300 Write block 3053 len 24576 data: 2 2 2 2 3 3
3848 079/300 Write block 960 len 36864 data: 3 3 3 3 3 3 3 3 3
3849 080/300 Read block 1439 len 8192
3850 081/300 Write block 89 len 36864 data: 2 3 3 3 3 3 3 3 3
3851 082/300 Read block 313 len 4096
3852 083/300 Read block 2325 len 40960
3853 084/300 Read block 826 len 40960
3854 085/300 Read block 2260 len 4096
3855 086/300 Read block 2911 len 36864
3856 087/300 Flush
3857 088/300 Read block 3470 len 8192
3858 089/300 Write block 536 len 8192 data: 3 2
3859 090/300 Read block 4747 len 12288
3860 091/300 Write block 2010 len 28672 data: 2 2 2 2 2 2 2
3861 092/300 Read block 3501 len 32768
3862 093/300 Write block 626 len 40960 data: 3 2 3 3 3 3 3 3 3 3
3863 094/300 Read block 2060 len 16384
3864 095/300 Write block 2899 len 4096 data: 2
3865 096/300 Flush
3866 097/300 Write block 4976 len 24576 data: 2 2 2 2 2 2
3867 098/300 Write block 4568 len 20480 data: 2 2 2 2 3
3868 099/300 Flush
3869 100/300 Read block 1603 len 8192
3870 101/300 Write block 2868 len 16384 data: 3 3 3 3
3871 102/300 Write block 4579 len 24576 data: 4 4 3 3 2 2
3872 103/300 Write block 12 len 36864 data: 2 2 2 2 2 2 2 2 2
3873 104/300 Read block 96 len 20480
3874 105/300 Write block 851 len 4096 data: 2
3875 106/300 Write block 798 len 16384 data: 2 2 2 2
3876 107/300 Read block 4143 len 24576
3877 108/300 Read block 2964 len 12288
3878 109/300 Write block 2580 len 24576 data: 2 2 2 2 2 3
3879 110/300 Read block 2503 len 36864
3880 111/300 Read block 479 len 16384
3881 112/300 Read block 4563 len 40960
3882 113/300 Flush
3883 114/300 Read block 2681 len 32768
3884 115/300 Read block 262 len 32768
3885 116/300 Read block 3921 len 4096
3886 117/300 Write block 2588 len 24576 data: 3 3 3 3 2 2
3887 118/300 Read block 4047 len 40960
3888 119/300 Write block 138 len 40960 data: 3 3 4 3 2 2 2 2 2 2
3889 120/300 Read block 1563 len 4096
3890 121/300 Read block 2209 len 28672
3891 122/300 Read block 1242 len 32768
3892 123/300 Read block 1297 len 36864
3893 124/300 Read block 306 len 16384
3894 125/300 Write block 1132 len 32768 data: 2 2 2 2 2 2 2 2
3895 126/300 Read block 3204 len 16384
3896 127/300 Flush
3897 128/300 Read block 4398 len 36864
3898 129/300 Write block 131 len 24576 data: 3 2 2 2 2 2
3899 130/300 Flush
3900 131/300 Flush
3901 132/300 Write block 1395 len 12288 data: 3 3 2
3902 133/300 Read block 4773 len 12288
3903 134/300 Flush
3904 135/300 Flush
3905 136/300 Read block 3263 len 40960
3906 137/300 Write block 1262 len 16384 data: 2 2 2 2
3907 138/300 Read block 4110 len 28672
3908 139/300 Read block 3694 len 36864
3909 140/300 Write block 2135 len 24576 data: 3 3 3 3 2 2
3910 141/300 Read block 3311 len 28672
3911 142/300 Write block 1016 len 12288 data: 3 3 3
3912 143/300 Write block 2391 len 8192 data: 2 2
3913 144/300 Read block 3127 len 4096
3914 145/300 Read block 401 len 20480
3915 146/300 Flush
3916 147/300 Read block 254 len 40960
3917 148/300 Write block 309 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3918 149/300 Read block 452 len 32768
3919 150/300 Read block 445 len 24576
3920 151/300 Write block 3798 len 20480 data: 2 2 2 3 3
3921 152/300 Write block 2257 len 24576 data: 2 2 2 2 2 2
3922 153/300 Read block 1826 len 32768
3923 154/300 Flush
3924 155/300 Write block 1199 len 16384 data: 2 2 2 2
3925 156/300 Read block 871 len 20480
3926 157/300 Write block 3617 len 40960 data: 4 4 4 4 4 5 5 5 6 5
3927 158/300 Write block 2307 len 28672 data: 2 2 2 2 2 2 3
3928 159/300 Write block 1699 len 32768 data: 2 2 2 2 2 2 2 2
3929 160/300 Read block 3455 len 12288
3930 161/300 Read block 2524 len 8192
3931 162/300 Read block 2939 len 32768
3932 163/300 Read block 537 len 32768
3933 164/300 Write block 3762 len 16384 data: 2 2 2 2
3934 165/300 Write block 2456 len 24576 data: 2 2 2 2 2 2
3935 166/300 Read block 1478 len 28672
3936 167/300 Write block 4752 len 24576 data: 3 3 3 3 3 3
3937 168/300 Read block 2894 len 4096
3938 169/300 Write block 3072 len 24576 data: 2 2 2 2 2 2
3939 170/300 Read block 4565 len 28672
3940 171/300 Write block 1583 len 20480 data: 2 2 2 3 3
3941 172/300 Write block 2206 len 32768 data: 3 3 3 3 3 3 3 2
3942 173/300 Read block 4937 len 20480
3943 174/300 Write block 2220 len 24576 data: 2 2 2 2 2 2
3944 175/300 Write block 1792 len 32768 data: 2 2 3 3 3 3 3 3
3945 176/300 Flush
3946 177/300 Write block 2026 len 4096 data: 2
3947 178/300 Write block 2635 len 4096 data: 3
3948 179/300 Read block 736 len 4096
3949 180/300 Write block 2044 len 40960 data: 2 2 2 2 2 2 3 3 2 3
3950 181/300 Read block 4766 len 20480
3951 182/300 Read block 264 len 36864
3952 183/300 Read block 858 len 28672
3953 184/300 Write block 322 len 12288 data: 3 3 3
3954 185/300 Read block 486 len 36864
3955 186/300 Read block 4051 len 12288
3956 187/300 Read block 3904 len 36864
3957 188/300 Read block 3039 len 12288
3958 189/300 Write block 741 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3959 190/300 Write block 1939 len 28672 data: 2 2 2 2 2 2 2
3960 191/300 Read block 1161 len 32768
3961 192/300 Read block 1888 len 28672
3962 193/300 Read block 982 len 8192
3963 194/300 Read block 3454 len 20480
3964 195/300 Write block 2909 len 12288 data: 2 2 2
3965 196/300 Write block 4921 len 28672 data: 3 2 2 3 3 3 3
3966 197/300 Write block 4500 len 32768 data: 3 3 3 2 3 3 2 2
3967 198/300 Write block 1139 len 36864 data: 3 2 2 2 2 2 3 3 3
3968 199/300 Write block 3622 len 24576 data: 6 6 6 7 6 5
3969 200/300 Write block 1361 len 4096 data: 2
3970 201/300 Read block 2148 len 32768
3971 202/300 Flush
3972 203/300 Write block 4868 len 36864 data: 2 2 2 2 2 2 2 2 2
3973 204/300 Write block 785 len 28672 data: 2 3 3 3 2 2 2
3974 205/300 Flush
3975 206/300 Read block 3495 len 12288
3976 207/300 Flush
3977 208/300 Flush
3978 209/300 Write block 3290 len 16384 data: 4 4 4 5
3979 210/300 Write block 438 len 32768 data: 2 2 2 2 2 2 2 2
3980 211/300 Read block 4148 len 28672
3981 212/300 Read block 1839 len 40960
3982 213/300 Read block 984 len 12288
3983 214/300 Write block 1353 len 16384 data: 2 2 2 2
3984 215/300 Read block 2708 len 32768
3985 216/300 Read block 3016 len 24576
3986 217/300 Read block 603 len 4096
3987 218/300 Read block 3007 len 20480
3988 219/300 Read block 532 len 36864
3989 220/300 Write block 103 len 4096 data: 3
3990 221/300 Read block 2586 len 24576
3991 222/300 Read block 2489 len 24576
3992 223/300 Write block 1359 len 4096 data: 2
3993 224/300 Read block 3177 len 4096
3994 225/300 Read block 1721 len 20480
3995 226/300 Write block 3045 len 24576 data: 3 3 3 4 4 4
3996 227/300 Read block 4502 len 28672
3997 228/300 Write block 3045 len 28672 data: 4 4 4 5 5 5 3
3998 229/300 Read block 4480 len 40960
3999 230/300 Read block 1485 len 24576
4000 231/300 Read block 792 len 4096
4001 232/300 Write block 3638 len 16384 data: 3 3 3 3
4002 233/300 Read block 2525 len 4096
4003 234/300 Write block 382 len 8192 data: 2 2
4004 235/300 Read block 2393 len 8192
4005 236/300 Read block 2242 len 20480
4006 237/300 Write block 2659 len 36864 data: 3 3 3 2 3 3 2 2 2
4007 238/300 Write block 427 len 32768 data: 3 4 4 4 4 3 3 3
4008 239/300 Flush
4009 240/300 Read block 1702 len 20480
4010 241/300 Read block 1449 len 16384
4011 242/300 Write block 3014 len 12288 data: 2 2 2
4012 243/300 Flush
4013 244/300 Write block 1172 len 24576 data: 2 2 2 2 2 2
4014 245/300 Write block 4822 len 12288 data: 2 2 2
4015 246/300 Read block 324 len 32768
4016 247/300 Read block 1369 len 16384
4017 248/300 Write block 265 len 4096 data: 4
4018 249/300 Write block 3487 len 8192 data: 2 2
4019 250/300 Read block 4554 len 12288
4020 251/300 Read block 3009 len 8192
4021 252/300 Read block 549 len 40960
4022 253/300 Write block 3423 len 8192 data: 3 3
4023 254/300 Read block 2542 len 16384
4024 255/300 Read block 4821 len 28672
4025 256/300 Write block 1663 len 4096 data: 2
4026 257/300 Write block 1933 len 16384 data: 2 2 2 2
4027 258/300 Write block 1260 len 24576 data: 3 3 3 3 3 3
4028 259/300 Write block 820 len 40960 data: 3 3 3 3 2 2 3 3 3 3
4029 260/300 Flush
4030 261/300 Read block 1039 len 32768
4031 262/300 Write block 2628 len 4096 data: 3
4032 263/300 Read block 1404 len 16384
4033 264/300 Read block 1273 len 20480
4034 265/300 Flush
4035 266/300 Flush
4036 267/300 Read block 3346 len 20480
4037 268/300 Read block 1549 len 28672
4038 269/300 Write block 3215 len 16384 data: 2 2 2 2
4039 270/300 Write block 2704 len 12288 data: 2 2 2
4040 271/300 Read block 4099 len 28672
4041 272/300 Write block 3754 len 28672 data: 2 2 2 2 2 2 2
4042 273/300 Read block 3064 len 32768
4043 274/300 Write block 4428 len 24576 data: 2 2 2 2 2 2
4044 275/300 Read block 3244 len 28672
4045 276/300 Write block 4658 len 36864 data: 3 3 3 3 4 4 4 3 3
4046 277/300 Write block 2403 len 20480 data: 4 4 4 4 4
4047 278/300 Write block 1234 len 36864 data: 2 2 2 2 2 2 2 2 2
4048 279/300 Write block 2143 len 32768 data: 2 2 2 2 2 2 2 2
4049 280/300 Write block 279 len 8192 data: 3 3
4050 281/300 Write block 3307 len 16384 data: 2 2 3 4
4051 282/300 Write block 2494 len 32768 data: 2 2 2 2 3 3 3 3
4052 283/300 Write block 2999 len 24576 data: 3 3 5 5 4 4
4053 284/300 Write block 340 len 12288 data: 2 2 2
4054 285/300 Write block 3702 len 12288 data: 2 2 2
4055 286/300 Read block 2977 len 16384
4056 287/300 Write block 4882 len 16384 data: 2 2 2 2
4057 288/300 Read block 3969 len 40960
4058 289/300 Write block 254 len 24576 data: 2 2 2 2 2 3
4059 290/300 Read block 2944 len 8192
4060 291/300 Read block 1155 len 36864
4061 292/300 Read block 1855 len 16384
4062 293/300 Read block 2853 len 24576
4063 294/300 Write block 1672 len 40960 data: 2 2 2 3 3 3 4 4 2 2
4064 295/300 Flush
4065 296/300 Write block 2120 len 8192 data: 3 3
4066 297/300 Read block 3516 len 32768
4067 298/300 Write block 4871 len 24576 data: 3 3 3 3 3 3
4068 299/300 Write block 4435 len 4096 data: 2
4069 300/300 Write block 650 len 16384 data: 3 2 2 2
4070 301/300 Read block 649 len 32768
4071 Main task has work to do, go find it
4072 got dsc Start(1)
4073 start 1
4074 Jan 23 18:48:45.947 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/1, method: POST, req_id: d264a3ba-eb02-41b7-9004-382b21b4bccb, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
4075 Replay: started 1, returned:Ok(())
4076 ----------------------------------------------------------------
4077 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4078 States: Active Offline Active
4079 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4080 1906 Acked Read 7 Done Sent Done false
4081 1907 Acked Read 9 Done Sent Done false
4082 1908 Acked Write 2 Done Sent Done false
4083 1909 Acked Read 1 Done Sent Done false
4084 1910 Acked Read 5 Done Sent Done false
4085 1911 Acked Write 3 Done Sent Done false
4086 1912 Acked Read 5 Done Sent Done false
4087 1913 Acked Write 9 Done Sent Done false
4088 1914 Acked Read 7 Done Sent Done false
4089 1915 Acked Read 10 Done Sent Done false
4090 1916 Acked Read 6 Done Sent Done false
4091 1917 Acked Read 5 Done Sent Done false
4092 1918 Acked Write 3 Done Sent Done false
4093 1919 Acked Read 8 Done Sent Done false
4094 1920 Acked Read 7 Done Sent Done false
4095 1921 Acked Read 8 Done Sent Done false
4096 1922 Acked Read 1 Done Sent Done false
4097 1923 Acked Write 7 Done Sent Done false
4098 1924 Acked Write 1 Done Sent Done false
4099 1925 Acked Write 3 Done Sent Done false
4100 1926 Acked Write 4 Done Sent Done false
4101 1927 Acked Read 9 Done Sent Done false
4102 1928 Acked Read 3 Done Sent Done false
4103 1929 Acked Read 6 Done Sent Done false
4104 1930 Acked Write 1 Done Sent Done false
4105 1931 Acked Write 1 Done Sent Done false
4106 1932 Acked Write 9 Done Sent Done false
4107 1933 Acked Write 7 Done Sent Done false
4108 1934 Acked Read 9 Done Sent Done false
4109 1935 Acked Write 1 Done Sent Done false
4110 1936 Acked Write 2 Done Sent Done false
4111 1937 Acked Read 10 Done Sent Done false
4112 1938 Acked Read 1 Done Sent Done false
4113 1939 Acked Write 7 Done Sent Done false
4114 1940 Acked Read 3 Done Sent Done false
4115 1941 Acked Write 1 Done Sent Done false
4116 1942 Acked Read 7 Done Sent Done false
4117 1943 Acked Write 9 Done Sent Done false
4118 1944 Acked Read 3 Done Sent Done false
4119 1945 Acked Read 1 Done Sent Done false
4120 1946 Acked Write 10 Done Sent Done false
4121 1947 Acked Write 4 Done Sent Done false
4122 1948 Acked Read 1 Done Sent Done false
4123 1949 Acked Read 3 Done Sent Done false
4124 1950 Acked Write 2 Done Sent Done false
4125 1951 Acked Read 4 Done Sent Done false
4126 1952 Acked Flush 0 Done Sent Done false
4127 1953 Acked Flush 0 Done Sent Done false
4128 1954 Acked Write 7 Done Sent Done false
4129 1955 Acked Read 10 Done Sent Done false
4130 1956 Acked Write 5 Done Sent Done false
4131 1957 Acked Write 2 Done Sent Done false
4132 1958 Acked Write 9 Done Sent Done false
4133 1959 Acked Write 10 Done Sent Done false
4134 1960 Acked Read 2 Done Sent Done false
4135 1961 Acked Read 6 Done Sent Done false
4136 1962 Acked Read 9 Done Sent Done false
4137 1963 Acked Write 3 Done Sent Done false
4138 1964 Acked Write 7 Done Sent Done false
4139 1965 Acked Flush 0 Done Sent Done false
4140 1966 Acked Read 6 Done Sent Done false
4141 1967 Acked Write 6 Done Sent Done false
4142 1968 Acked Write 8 Done Sent Done false
4143 1969 Acked Write 7 Done Sent Done false
4144 1970 Acked Read 10 Done Sent Done false
4145 1971 Acked Write 9 Done Sent Done false
4146 1972 Acked Write 1 Done Sent Done false
4147 1973 Acked Read 5 Done Sent Done false
4148 1974 Acked Read 3 Done Sent Done false
4149 1975 Acked Write 4 Done Sent Done false
4150 1976 Acked Flush 0 Done Sent Done false
4151 1977 Acked Write 5 Done Sent Done false
4152 1978 Acked Write 8 Done Sent Done false
4153 1979 Acked Read 2 Done Sent Done false
4154 1980 Acked Write 2 Done Sent Done false
4155 1981 Acked Read 8 Done Sent Done false
4156 1982 Acked Read 2 Done Sent Done false
4157 1983 Acked Write 6 Done Sent Done false
4158 1984 Acked Write 9 Done Sent Done false
4159 1985 Acked Read 2 Done Sent Done false
4160 1986 Acked Write 9 Done Sent Done false
4161 1987 Acked Read 1 Done Sent Done false
4162 1988 Acked Read 10 Done Sent Done false
4163 1989 Acked Read 10 Done Sent Done false
4164 1990 Acked Read 1 Done Sent Done false
4165 1991 Acked Read 9 Done Sent Done false
4166 1992 Acked Flush 0 Done Sent Done false
4167 1993 Acked Read 2 Done Sent Done false
4168 1994 Acked Write 2 Done Sent Done false
4169 1995 Acked Read 3 Done Sent Done false
4170 1996 Acked Write 7 Done Sent Done false
4171 1997 Acked Read 8 Done Sent Done false
4172 1998 Acked Write 10 Done Sent Done false
4173 1999 Acked Read 4 Done Sent Done false
4174 2000 Acked Write 1 Done Sent Done false
4175 2001 Acked Flush 0 Done Sent Done false
4176 2002 Acked Write 6 Done Sent Done false
4177 2003 Acked Write 5 Done Sent Done false
4178 2004 Acked Flush 0 Done Sent Done false
4179 2005 Acked Read 2 Done Sent Done false
4180 2006 Acked Write 4 Done Sent Done false
4181 2007 Acked Write 6 Done Sent Done false
4182 2008 Acked Write 9 Done Sent Done false
4183 2009 Acked Read 5 Done Sent Done false
4184 2010 Acked Write 1 Done Sent Done false
4185 2011 Acked Write 4 Done Sent Done false
4186 2012 Acked Read 6 Done Sent Done false
4187 2013 Acked Read 3 Done Sent Done false
4188 2014 Acked Write 6 Done Sent Done false
4189 2015 Acked Read 9 Done Sent Done false
4190 2016 Acked Read 4 Done Sent Done false
4191 2017 Acked Read 10 Done Sent Done false
4192 2018 Acked Flush 0 Done Sent Done false
4193 2019 Acked Read 8 Done Sent Done false
4194 2020 Acked Read 8 Done Sent Done false
4195 2021 Acked Read 1 Done Sent Done false
4196 2022 Acked Write 6 Done Sent Done false
4197 2023 Acked Read 10 Done Sent Done false
4198 2024 Acked Write 10 Done Sent Done false
4199 2025 Acked Read 1 Done Sent Done false
4200 2026 Acked Read 7 Done Sent Done false
4201 2027 Acked Read 8 Done Sent Done false
4202 2028 Acked Read 9 Done Sent Done false
4203 2029 Acked Read 4 Done Sent Done false
4204 2030 Acked Write 8 Done Sent Done false
4205 2031 Acked Read 4 Done Sent Done false
4206 2032 Acked Flush 0 Done Sent Done false
4207 2033 Acked Read 9 Done Sent Done false
4208 2034 Acked Write 6 Done Sent Done false
4209 2035 Acked Flush 0 Done Sent Done false
4210 2036 Acked Flush 0 Done Sent Done false
4211 2037 Acked Write 3 Done Sent Done false
4212 2038 Acked Read 3 Done Sent Done false
4213 2039 Acked Flush 0 Done Sent Done false
4214 2040 Acked Flush 0 Done Sent Done false
4215 2041 Acked Read 10 Done Sent Done false
4216 2042 Acked Write 4 Done Sent Done false
4217 2043 Acked Read 7 Done Sent Done false
4218 2044 Acked Read 9 Done Sent Done false
4219 2045 Acked Write 6 Done Sent Done false
4220 2046 Acked Read 7 Done Sent Done false
4221 2047 Acked Write 3 Done Sent Done false
4222 2048 Acked Write 2 Done Sent Done false
4223 2049 Acked Read 1 Done Sent Done false
4224 2050 Acked Read 5 Done Sent Done false
4225 2051 Acked Flush 0 Done Sent Done false
4226 2052 Acked Read 10 Done Sent Done false
4227 2053 Acked Write 10 Done Sent Done false
4228 2054 Acked Read 8 Done Sent Done false
4229 2055 Acked Read 6 Done Sent Done false
4230 2056 Acked Write 5 Done Sent Done false
4231 2057 Acked Write 6 Done Sent Done false
4232 2058 Acked Read 8 Done Sent Done false
4233 2059 Acked Flush 0 Done Sent Done false
4234 2060 Acked Write 4 Done Sent Done false
4235 2061 Acked Read 5 Done Sent Done false
4236 2062 Acked Write 10 Done Sent Done false
4237 2063 Acked Write 7 Done Sent Done false
4238 2064 Acked Write 8 Done Sent Done false
4239 2065 Acked Read 3 Done Sent Done false
4240 2066 Acked Read 2 Done Sent Done false
4241 2067 Acked Read 8 Done Sent Done false
4242 2068 Acked Read 8 Done Sent Done false
4243 2069 Acked Write 4 Done Sent Done false
4244 2070 Acked Write 6 Done Sent Done false
4245 2071 Acked Read 7 Done Sent Done false
4246 2072 Acked Write 6 Done Sent Done false
4247 2073 Acked Read 1 Done Sent Done false
4248 2074 Acked Write 6 Done Sent Done false
4249 2075 Acked Read 7 Done Sent Done false
4250 2076 Acked Write 5 Done Sent Done false
4251 2077 Acked Write 8 Done Sent Done false
4252 2078 Acked Read 5 Done Sent Done false
4253 2079 Acked Write 6 Done Sent Done false
4254 2080 Acked Write 8 Done Sent Done false
4255 2081 Acked Flush 0 Done Sent Done false
4256 2082 Acked Write 1 Done Sent Done false
4257 2083 Acked Write 1 Done Sent Done false
4258 2084 Acked Read 1 Done Sent Done false
4259 2085 Acked Write 10 Done Sent Done false
4260 2086 Acked Read 5 Done Sent Done false
4261 2087 Acked Read 9 Done Sent Done false
4262 2088 Acked Read 7 Done Sent Done false
4263 2089 Acked Write 3 Done Sent Done false
4264 2090 Acked Read 9 Done Sent Done false
4265 2091 Acked Read 3 Done Sent Done false
4266 2092 Acked Read 9 Done Sent Done false
4267 2093 Acked Read 3 Done Sent Done false
4268 2094 Acked Write 10 Done Sent Done false
4269 2095 Acked Write 7 Done Sent Done false
4270 2096 Acked Read 8 Done Sent Done false
4271 2097 Acked Read 7 Done Sent Done false
4272 2098 Acked Read 2 Done Sent Done false
4273 2099 Acked Read 5 Done Sent Done false
4274 2100 Acked Write 3 Done Sent Done false
4275 2101 Acked Write 7 Done Sent Done false
4276 2102 Acked Write 8 Done Sent Done false
4277 2103 Acked Write 9 Done Sent Done false
4278 2104 Acked Write 6 Done Sent Done false
4279 2105 Acked Write 1 Done Sent Done false
4280 2106 Acked Read 8 Done Sent Done false
4281 2107 Acked Flush 0 Done Sent Done false
4282 2108 Acked Write 9 Done Sent Done false
4283 2109 Acked Write 7 Done Sent Done false
4284 2110 Acked Flush 0 Done Sent Done false
4285 2111 Acked Read 3 Done Sent Done false
4286 2112 Acked Flush 0 Done Sent Done false
4287 2113 Acked Flush 0 Done Sent Done false
4288 2114 Acked Write 4 Done Sent Done false
4289 2115 Acked Write 8 Done Sent Done false
4290 2116 Acked Read 7 Done Sent Done false
4291 2117 Acked Read 10 Done Sent Done false
4292 2118 Acked Read 3 Done Sent Done false
4293 2119 Acked Write 4 Done Sent Done false
4294 2120 Acked Read 8 Done Sent Done false
4295 2121 Acked Read 6 Done Sent Done false
4296 2122 Acked Read 1 Done Sent Done false
4297 2123 Acked Read 5 Done Sent Done false
4298 2124 Acked Read 9 Done Sent Done false
4299 2125 Acked Write 1 Done Sent Done false
4300 2126 Acked Read 6 Done Sent Done false
4301 2127 Acked Read 6 Done Sent Done false
4302 2128 Acked Write 1 Done Sent Done false
4303 2129 Acked Read 1 Done Sent Done false
4304 2130 Acked Read 5 Done Sent Done false
4305 2131 Acked Write 6 Done Sent Done false
4306 2132 Acked Read 7 Done Sent Done false
4307 2133 Acked Write 7 Done Sent Done false
4308 2134 Acked Read 10 Done Sent Done false
4309 2135 Acked Read 6 Done Sent Done false
4310 2136 Acked Read 1 Done Sent Done false
4311 2137 Acked Write 4 Done Sent Done false
4312 2138 Acked Read 1 Done Sent Done false
4313 2139 Acked Write 2 Done Sent Done false
4314 2140 Acked Read 2 Done Sent Done false
4315 2141 Acked Read 5 Done Sent Done false
4316 2142 Acked Write 9 Done Sent Done false
4317 2143 Acked Write 8 Done Sent Done false
4318 2144 Acked Flush 0 Done Sent Done false
4319 2145 Acked Read 5 Done Sent Done false
4320 2146 Acked Read 4 Done Sent Done false
4321 2147 Acked Write 3 Done Sent Done false
4322 2148 Acked Flush 0 Done Sent Done false
4323 2149 Acked Write 6 Done Sent Done false
4324 2150 Acked Write 3 Done Sent Done false
4325 2151 Acked Read 8 Done Sent Done false
4326 2152 Acked Read 4 Done Sent Done false
4327 2153 Acked Write 1 Done Sent Done false
4328 2154 Acked Write 2 Done Sent Done false
4329 2155 Acked Read 3 Done Sent Done false
4330 2156 Acked Read 2 Done Sent Done false
4331 2157 Acked Read 10 Done Sent Done false
4332 2158 Acked Write 2 Done Sent Done false
4333 2159 Acked Read 4 Done Sent Done false
4334 2160 Acked Read 7 Done Sent Done false
4335 2161 Acked Write 1 Done Sent Done false
4336 2162 Acked Write 4 Done Sent Done false
4337 2163 Acked Write 6 Done Sent Done false
4338 2164 Acked Write 10 Done Sent Done false
4339 2165 Acked Flush 0 Done Sent Done false
4340 2166 Acked Read 8 Done Sent Done false
4341 2167 Acked Write 1 Done Sent Done false
4342 2168 Acked Read 4 Done Sent Done false
4343 2169 Acked Read 5 Done Sent Done false
4344 2170 Acked Flush 0 Done Sent Done false
4345 2171 Acked Flush 0 Done Sent Done false
4346 2172 Acked Read 5 Done Sent Done false
4347 2173 Acked Read 7 Done Sent Done false
4348 2174 Acked Write 4 Done Sent Done false
4349 2175 Acked Write 3 Done Sent Done false
4350 2176 Acked Read 7 Done Sent Done false
4351 2177 Acked Write 7 Done Sent Done false
4352 2178 Acked Read 8 Done Sent Done false
4353 2179 Acked Write 6 Done Sent Done false
4354 2180 Acked Read 7 Done Sent Done false
4355 2181 Acked Write 9 Done Sent Done false
4356 2182 Acked Write 5 Done Sent Done false
4357 2183 Acked Write 9 Done Sent Done false
4358 2184 Acked Write 8 Done Sent Done false
4359 2185 Acked Write 2 Done Sent Done false
4360 2186 Acked Write 4 Done Sent Done false
4361 2187 Acked Write 8 Done Sent Done false
4362 2188 Acked Write 6 Done Sent Done false
4363 2189 Acked Write 3 Done Sent Done false
4364 2190 Acked Write 3 Done Sent Done false
4365 2191 Acked Read 4 Done Sent Done false
4366 2192 Acked Write 4 Done Sent Done false
4367 2193 Acked Read 10 Done Sent Done false
4368 2194 Acked Write 6 Done Sent Done false
4369 2195 Acked Read 2 Done Sent Done false
4370 2196 Acked Read 9 Done Sent Done false
4371 2197 Acked Read 4 Done Sent Done false
4372 2198 Acked Read 6 Done Sent Done false
4373 2199 Acked Write 10 Done Sent Done false
4374 2200 Acked Flush 0 Done Sent Done false
4375 2201 Acked Write 2 Done Sent Done false
4376 2202 Acked Read 8 Done Sent Done false
4377 2203 Acked Write 6 Done Sent Done false
4378 2204 Acked Write 1 Done Sent Done false
4379 2205 Acked Write 4 Done Sent Done false
4380 2206 Acked Read 8 Done Sent Done false
4381 STATES DS:0 DS:1 DS:2 TOTAL
4382 Sent 0 301 0 301
4383 Done 301 0 301 602
4384 Skipped 0 0 0 0
4385 Error 0 0 0 0
4386 Last Flush: Some(JobId(2200)) Some(JobId(1905)) Some(JobId(2200))
4387 Downstairs last five completed: 1905 1904 1903 1902 1901
4388 Upstairs last five completed: 2206 2205 2204 2203 2202
4389 CLIENT: Up:0 ds:301 act:2
4390 ----------------------------------------------------------------
4391 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
4392 States: Active Offline Active
4393 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4394 1906 Acked Read 7 Done Sent Done false
4395 1907 Acked Read 9 Done Sent Done false
4396 1908 Acked Write 2 Done Sent Done false
4397 1909 Acked Read 1 Done Sent Done false
4398 1910 Acked Read 5 Done Sent Done false
4399 1911 Acked Write 3 Done Sent Done false
4400 1912 Acked Read 5 Done Sent Done false
4401 1913 Acked Write 9 Done Sent Done false
4402 1914 Acked Read 7 Done Sent Done false
4403 1915 Acked Read 10 Done Sent Done false
4404 1916 Acked Read 6 Done Sent Done false
4405 1917 Acked Read 5 Done Sent Done false
4406 1918 Acked Write 3 Done Sent Done false
4407 1919 Acked Read 8 Done Sent Done false
4408 1920 Acked Read 7 Done Sent Done false
4409 1921 Acked Read 8 Done Sent Done false
4410 1922 Acked Read 1 Done Sent Done false
4411 1923 Acked Write 7 Done Sent Done false
4412 1924 Acked Write 1 Done Sent Done false
4413 1925 Acked Write 3 Done Sent Done false
4414 1926 Acked Write 4 Done Sent Done false
4415 1927 Acked Read 9 Done Sent Done false
4416 1928 Acked Read 3 Done Sent Done false
4417 1929 Acked Read 6 Done Sent Done false
4418 1930 Acked Write 1 Done Sent Done false
4419 1931 Acked Write 1 Done Sent Done false
4420 1932 Acked Write 9 Done Sent Done false
4421 1933 Acked Write 7 Done Sent Done false
4422 1934 Acked Read 9 Done Sent Done false
4423 1935 Acked Write 1 Done Sent Done false
4424 1936 Acked Write 2 Done Sent Done false
4425 1937 Acked Read 10 Done Sent Done false
4426 1938 Acked Read 1 Done Sent Done false
4427 1939 Acked Write 7 Done Sent Done false
4428 1940 Acked Read 3 Done Sent Done false
4429 1941 Acked Write 1 Done Sent Done false
4430 1942 Acked Read 7 Done Sent Done false
4431 1943 Acked Write 9 Done Sent Done false
4432 1944 Acked Read 3 Done Sent Done false
4433 1945 Acked Read 1 Done Sent Done false
4434 1946 Acked Write 10 Done Sent Done false
4435 1947 Acked Write 4 Done Sent Done false
4436 1948 Acked Read 1 Done Sent Done false
4437 1949 Acked Read 3 Done Sent Done false
4438 1950 Acked Write 2 Done Sent Done false
4439 1951 Acked Read 4 Done Sent Done false
4440 1952 Acked Flush 0 Done Sent Done false
4441 1953 Acked Flush 0 Done Sent Done false
4442 1954 Acked Write 7 Done Sent Done false
4443 1955 Acked Read 10 Done Sent Done false
4444 1956 Acked Write 5 Done Sent Done false
4445 1957 Acked Write 2 Done Sent Done false
4446 1958 Acked Write 9 Done Sent Done false
4447 1959 Acked Write 10 Done Sent Done false
4448 1960 Acked Read 2 Done Sent Done false
4449 1961 Acked Read 6 Done Sent Done false
4450 1962 Acked Read 9 Done Sent Done false
4451 1963 Acked Write 3 Done Sent Done false
4452 1964 Acked Write 7 Done Sent Done false
4453 1965 Acked Flush 0 Done Sent Done false
4454 1966 Acked Read 6 Done Sent Done false
4455 1967 Acked Write 6 Done Sent Done false
4456 1968 Acked Write 8 Done Sent Done false
4457 1969 Acked Write 7 Done Sent Done false
4458 1970 Acked Read 10 Done Sent Done false
4459 1971 Acked Write 9 Done Sent Done false
4460 1972 Acked Write 1 Done Sent Done false
4461 1973 Acked Read 5 Done Sent Done false
4462 1974 Acked Read 3 Done Sent Done false
4463 1975 Acked Write 4 Done Sent Done false
4464 1976 Acked Flush 0 Done Sent Done false
4465 1977 Acked Write 5 Done Sent Done false
4466 1978 Acked Write 8 Done Sent Done false
4467 1979 Acked Read 2 Done Sent Done false
4468 1980 Acked Write 2 Done Sent Done false
4469 1981 Acked Read 8 Done Sent Done false
4470 1982 Acked Read 2 Done Sent Done false
4471 1983 Acked Write 6 Done Sent Done false
4472 1984 Acked Write 9 Done Sent Done false
4473 1985 Acked Read 2 Done Sent Done false
4474 1986 Acked Write 9 Done Sent Done false
4475 1987 Acked Read 1 Done Sent Done false
4476 1988 Acked Read 10 Done Sent Done false
4477 1989 Acked Read 10 Done Sent Done false
4478 1990 Acked Read 1 Done Sent Done false
4479 1991 Acked Read 9 Done Sent Done false
4480 1992 Acked Flush 0 Done Sent Done false
4481 1993 Acked Read 2 Done Sent Done false
4482 1994 Acked Write 2 Done Sent Done false
4483 1995 Acked Read 3 Done Sent Done false
4484 1996 Acked Write 7 Done Sent Done false
4485 1997 Acked Read 8 Done Sent Done false
4486 1998 Acked Write 10 Done Sent Done false
4487 1999 Acked Read 4 Done Sent Done false
4488 2000 Acked Write 1 Done Sent Done false
4489 2001 Acked Flush 0 Done Sent Done false
4490 2002 Acked Write 6 Done Sent Done false
4491 2003 Acked Write 5 Done Sent Done false
4492 2004 Acked Flush 0 Done Sent Done false
4493 2005 Acked Read 2 Done Sent Done false
4494 2006 Acked Write 4 Done Sent Done false
4495 2007 Acked Write 6 Done Sent Done false
4496 2008 Acked Write 9 Done Sent Done false
4497 2009 Acked Read 5 Done Sent Done false
4498 2010 Acked Write 1 Done Sent Done false
4499 2011 Acked Write 4 Done Sent Done false
4500 2012 Acked Read 6 Done Sent Done false
4501 2013 Acked Read 3 Done Sent Done false
4502 2014 Acked Write 6 Done Sent Done false
4503 2015 Acked Read 9 Done Sent Done false
4504 2016 Acked Read 4 Done Sent Done false
4505 2017 Acked Read 10 Done Sent Done false
4506 2018 Acked Flush 0 Done Sent Done false
4507 2019 Acked Read 8 Done Sent Done false
4508 2020 Acked Read 8 Done Sent Done false
4509 2021 Acked Read 1 Done Sent Done false
4510 2022 Acked Write 6 Done Sent Done false
4511 2023 Acked Read 10 Done Sent Done false
4512 2024 Acked Write 10 Done Sent Done false
4513 2025 Acked Read 1 Done Sent Done false
4514 2026 Acked Read 7 Done Sent Done false
4515 2027 Acked Read 8 Done Sent Done false
4516 2028 Acked Read 9 Done Sent Done false
4517 2029 Acked Read 4 Done Sent Done false
4518 2030 Acked Write 8 Done Sent Done false
4519 2031 Acked Read 4 Done Sent Done false
4520 2032 Acked Flush 0 Done Sent Done false
4521 2033 Acked Read 9 Done Sent Done false
4522 2034 Acked Write 6 Done Sent Done false
4523 2035 Acked Flush 0 Done Sent Done false
4524 2036 Acked Flush 0 Done Sent Done false
4525 2037 Acked Write 3 Done Sent Done false
4526 2038 Acked Read 3 Done Sent Done false
4527 2039 Acked Flush 0 Done Sent Done false
4528 2040 Acked Flush 0 Done Sent Done false
4529 2041 Acked Read 10 Done Sent Done false
4530 2042 Acked Write 4 Done Sent Done false
4531 2043 Acked Read 7 Done Sent Done false
4532 2044 Acked Read 9 Done Sent Done false
4533 2045 Acked Write 6 Done Sent Done false
4534 2046 Acked Read 7 Done Sent Done false
4535 2047 Acked Write 3 Done Sent Done false
4536 2048 Acked Write 2 Done Sent Done false
4537 2049 Acked Read 1 Done Sent Done false
4538 2050 Acked Read 5 Done Sent Done false
4539 2051 Acked Flush 0 Done Sent Done false
4540 2052 Acked Read 10 Done Sent Done false
4541 2053 Acked Write 10 Done Sent Done false
4542 2054 Acked Read 8 Done Sent Done false
4543 2055 Acked Read 6 Done Sent Done false
4544 2056 Acked Write 5 Done Sent Done false
4545 2057 Acked Write 6 Done Sent Done false
4546 2058 Acked Read 8 Done Sent Done false
4547 2059 Acked Flush 0 Done Sent Done false
4548 2060 Acked Write 4 Done Sent Done false
4549 2061 Acked Read 5 Done Sent Done false
4550 2062 Acked Write 10 Done Sent Done false
4551 2063 Acked Write 7 Done Sent Done false
4552 2064 Acked Write 8 Done Sent Done false
4553 2065 Acked Read 3 Done Sent Done false
4554 2066 Acked Read 2 Done Sent Done false
4555 2067 Acked Read 8 Done Sent Done false
4556 2068 Acked Read 8 Done Sent Done false
4557 2069 Acked Write 4 Done Sent Done false
4558 2070 Acked Write 6 Done Sent Done false
4559 2071 Acked Read 7 Done Sent Done false
4560 2072 Acked Write 6 Done Sent Done false
4561 2073 Acked Read 1 Done Sent Done false
4562 2074 Acked Write 6 Done Sent Done false
4563 2075 Acked Read 7 Done Sent Done false
4564 2076 Acked Write 5 Done Sent Done false
4565 2077 Acked Write 8 Done Sent Done false
4566 2078 Acked Read 5 Done Sent Done false
4567 2079 Acked Write 6 Done Sent Done false
4568 2080 Acked Write 8 Done Sent Done false
4569 2081 Acked Flush 0 Done Sent Done false
4570 2082 Acked Write 1 Done Sent Done false
4571 2083 Acked Write 1 Done Sent Done false
4572 2084 Acked Read 1 Done Sent Done false
4573 2085 Acked Write 10 Done Sent Done false
4574 2086 Acked Read 5 Done Sent Done false
4575 2087 Acked Read 9 Done Sent Done false
4576 2088 Acked Read 7 Done Sent Done false
4577 2089 Acked Write 3 Done Sent Done false
4578 2090 Acked Read 9 Done Sent Done false
4579 2091 Acked Read 3 Done Sent Done false
4580 2092 Acked Read 9 Done Sent Done false
4581 2093 Acked Read 3 Done Sent Done false
4582 2094 Acked Write 10 Done Sent Done false
4583 2095 Acked Write 7 Done Sent Done false
4584 2096 Acked Read 8 Done Sent Done false
4585 2097 Acked Read 7 Done Sent Done false
4586 2098 Acked Read 2 Done Sent Done false
4587 2099 Acked Read 5 Done Sent Done false
4588 2100 Acked Write 3 Done Sent Done false
4589 2101 Acked Write 7 Done Sent Done false
4590 2102 Acked Write 8 Done Sent Done false
4591 2103 Acked Write 9 Done Sent Done false
4592 2104 Acked Write 6 Done Sent Done false
4593 2105 Acked Write 1 Done Sent Done false
4594 2106 Acked Read 8 Done Sent Done false
4595 2107 Acked Flush 0 Done Sent Done false
4596 2108 Acked Write 9 Done Sent Done false
4597 2109 Acked Write 7 Done Sent Done false
4598 2110 Acked Flush 0 Done Sent Done false
4599 2111 Acked Read 3 Done Sent Done false
4600 2112 Acked Flush 0 Done Sent Done false
4601 2113 Acked Flush 0 Done Sent Done false
4602 2114 Acked Write 4 Done Sent Done false
4603 2115 Acked Write 8 Done Sent Done false
4604 2116 Acked Read 7 Done Sent Done false
4605 2117 Acked Read 10 Done Sent Done false
4606 2118 Acked Read 3 Done Sent Done false
4607 2119 Acked Write 4 Done Sent Done false
4608 2120 Acked Read 8 Done Sent Done false
4609 2121 Acked Read 6 Done Sent Done false
4610 2122 Acked Read 1 Done Sent Done false
4611 2123 Acked Read 5 Done Sent Done false
4612 2124 Acked Read 9 Done Sent Done false
4613 2125 Acked Write 1 Done Sent Done false
4614 2126 Acked Read 6 Done Sent Done false
4615 2127 Acked Read 6 Done Sent Done false
4616 2128 Acked Write 1 Done Sent Done false
4617 2129 Acked Read 1 Done Sent Done false
4618 2130 Acked Read 5 Done Sent Done false
4619 2131 Acked Write 6 Done Sent Done false
4620 2132 Acked Read 7 Done Sent Done false
4621 2133 Acked Write 7 Done Sent Done false
4622 2134 Acked Read 10 Done Sent Done false
4623 2135 Acked Read 6 Done Sent Done false
4624 2136 Acked Read 1 Done Sent Done false
4625 2137 Acked Write 4 Done Sent Done false
4626 2138 Acked Read 1 Done Sent Done false
4627 2139 Acked Write 2 Done Sent Done false
4628 2140 Acked Read 2 Done Sent Done false
4629 2141 Acked Read 5 Done Sent Done false
4630 2142 Acked Write 9 Done Sent Done false
4631 2143 Acked Write 8 Done Sent Done false
4632 2144 Acked Flush 0 Done Sent Done false
4633 2145 Acked Read 5 Done Sent Done false
4634 2146 Acked Read 4 Done Sent Done false
4635 2147 Acked Write 3 Done Sent Done false
4636 2148 Acked Flush 0 Done Sent Done false
4637 2149 Acked Write 6 Done Sent Done false
4638 2150 Acked Write 3 Done Sent Done false
4639 2151 Acked Read 8 Done Sent Done false
4640 2152 Acked Read 4 Done Sent Done false
4641 2153 Acked Write 1 Done Sent Done false
4642 2154 Acked Write 2 Done Sent Done false
4643 2155 Acked Read 3 Done Sent Done false
4644 2156 Acked Read 2 Done Sent Done false
4645 2157 Acked Read 10 Done Sent Done false
4646 2158 Acked Write 2 Done Sent Done false
4647 2159 Acked Read 4 Done Sent Done false
4648 2160 Acked Read 7 Done Sent Done false
4649 2161 Acked Write 1 Done Sent Done false
4650 2162 Acked Write 4 Done Sent Done false
4651 2163 Acked Write 6 Done Sent Done false
4652 2164 Acked Write 10 Done Sent Done false
4653 2165 Acked Flush 0 Done Sent Done false
4654 2166 Acked Read 8 Done Sent Done false
4655 2167 Acked Write 1 Done Sent Done false
4656 2168 Acked Read 4 Done Sent Done false
4657 2169 Acked Read 5 Done Sent Done false
4658 2170 Acked Flush 0 Done Sent Done false
4659 2171 Acked Flush 0 Done Sent Done false
4660 2172 Acked Read 5 Done Sent Done false
4661 2173 Acked Read 7 Done Sent Done false
4662 2174 Acked Write 4 Done Sent Done false
4663 2175 Acked Write 3 Done Sent Done false
4664 2176 Acked Read 7 Done Sent Done false
4665 2177 Acked Write 7 Done Sent Done false
4666 2178 Acked Read 8 Done Sent Done false
4667 2179 Acked Write 6 Done Sent Done false
4668 2180 Acked Read 7 Done Sent Done false
4669 2181 Acked Write 9 Done Sent Done false
4670 2182 Acked Write 5 Done Sent Done false
4671 2183 Acked Write 9 Done Sent Done false
4672 2184 Acked Write 8 Done Sent Done false
4673 2185 Acked Write 2 Done Sent Done false
4674 2186 Acked Write 4 Done Sent Done false
4675 2187 Acked Write 8 Done Sent Done false
4676 2188 Acked Write 6 Done Sent Done false
4677 2189 Acked Write 3 Done Sent Done false
4678 2190 Acked Write 3 Done Sent Done false
4679 2191 Acked Read 4 Done Sent Done false
4680 2192 Acked Write 4 Done Sent Done false
4681 2193 Acked Read 10 Done Sent Done false
4682 2194 Acked Write 6 Done Sent Done false
4683 2195 Acked Read 2 Done Sent Done false
4684 2196 Acked Read 9 Done Sent Done false
4685 2197 Acked Read 4 Done Sent Done false
4686 2198 Acked Read 6 Done Sent Done false
4687 2199 Acked Write 10 Done Sent Done false
4688 2200 Acked Flush 0 Done Sent Done false
4689 2201 Acked Write 2 Done Sent Done false
4690 2202 Acked Read 8 Done Sent Done false
4691 2203 Acked Write 6 Done Sent Done false
4692 2204 Acked Write 1 Done Sent Done false
4693 2205 Acked Write 4 Done Sent Done false
4694 2206 Acked Read 8 Done Sent Done false
4695 2207 Acked Flush 0 Done Sent Done false
4696 STATES DS:0 DS:1 DS:2 TOTAL
4697 Sent 0 302 0 302
4698 Done 302 0 302 604
4699 Skipped 0 0 0 0
4700 Error 0 0 0 0
4701 Last Flush: Some(JobId(2207)) Some(JobId(1905)) Some(JobId(2207))
4702 Downstairs last five completed: 1905 1904 1903 1902 1901
4703 Upstairs last five completed: 2207 2206 2205 2204 2203
4704 CLIENT: Up:0 ds:302 act:2
47052025-01-23T18:48:51.712ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
47062025-01-23T18:48:51.712ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
47072025-01-23T18:48:51.712ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
47082025-01-23T18:48:51.712ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
47092025-01-23T18:48:51.712ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
4710 [8820] got start action
4711 [8820] I am going to restart
4712 Starting downstairs at port 8820
4713 Make output file at "/tmp/dsc/downstairs-8820.txt"
4714 [8820][1] reports Starting
4715 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1204)
4716 [8820][1] reports Running
4717 ----------------------------------------------------------------
4718 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
4719 States: Active Offline Active
4720 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4721 1906 Acked Read 7 Done Sent Done false
4722 1907 Acked Read 9 Done Sent Done false
4723 1908 Acked Write 2 Done Sent Done false
4724 1909 Acked Read 1 Done Sent Done false
4725 1910 Acked Read 5 Done Sent Done false
4726 1911 Acked Write 3 Done Sent Done false
4727 1912 Acked Read 5 Done Sent Done false
4728 1913 Acked Write 9 Done Sent Done false
4729 1914 Acked Read 7 Done Sent Done false
4730 1915 Acked Read 10 Done Sent Done false
4731 1916 Acked Read 6 Done Sent Done false
4732 1917 Acked Read 5 Done Sent Done false
4733 1918 Acked Write 3 Done Sent Done false
4734 1919 Acked Read 8 Done Sent Done false
4735 1920 Acked Read 7 Done Sent Done false
4736 1921 Acked Read 8 Done Sent Done false
4737 1922 Acked Read 1 Done Sent Done false
4738 1923 Acked Write 7 Done Sent Done false
4739 1924 Acked Write 1 Done Sent Done false
4740 1925 Acked Write 3 Done Sent Done false
4741 1926 Acked Write 4 Done Sent Done false
4742 1927 Acked Read 9 Done Sent Done false
4743 1928 Acked Read 3 Done Sent Done false
4744 1929 Acked Read 6 Done Sent Done false
4745 1930 Acked Write 1 Done Sent Done false
4746 1931 Acked Write 1 Done Sent Done false
4747 1932 Acked Write 9 Done Sent Done false
4748 1933 Acked Write 7 Done Sent Done false
4749 1934 Acked Read 9 Done Sent Done false
4750 1935 Acked Write 1 Done Sent Done false
4751 1936 Acked Write 2 Done Sent Done false
4752 1937 Acked Read 10 Done Sent Done false
4753 1938 Acked Read 1 Done Sent Done false
4754 1939 Acked Write 7 Done Sent Done false
4755 1940 Acked Read 3 Done Sent Done false
4756 1941 Acked Write 1 Done Sent Done false
4757 1942 Acked Read 7 Done Sent Done false
4758 1943 Acked Write 9 Done Sent Done false
4759 1944 Acked Read 3 Done Sent Done false
4760 1945 Acked Read 1 Done Sent Done false
4761 1946 Acked Write 10 Done Sent Done false
4762 1947 Acked Write 4 Done Sent Done false
4763 1948 Acked Read 1 Done Sent Done false
4764 1949 Acked Read 3 Done Sent Done false
4765 1950 Acked Write 2 Done Sent Done false
4766 1951 Acked Read 4 Done Sent Done false
4767 1952 Acked Flush 0 Done Sent Done false
4768 1953 Acked Flush 0 Done Sent Done false
4769 1954 Acked Write 7 Done Sent Done false
4770 1955 Acked Read 10 Done Sent Done false
4771 1956 Acked Write 5 Done Sent Done false
4772 1957 Acked Write 2 Done Sent Done false
4773 1958 Acked Write 9 Done Sent Done false
4774 1959 Acked Write 10 Done Sent Done false
4775 1960 Acked Read 2 Done Sent Done false
4776 1961 Acked Read 6 Done Sent Done false
4777 1962 Acked Read 9 Done Sent Done false
4778 1963 Acked Write 3 Done Sent Done false
4779 1964 Acked Write 7 Done Sent Done false
4780 1965 Acked Flush 0 Done Sent Done false
4781 1966 Acked Read 6 Done Sent Done false
4782 1967 Acked Write 6 Done Sent Done false
4783 1968 Acked Write 8 Done Sent Done false
4784 1969 Acked Write 7 Done Sent Done false
4785 1970 Acked Read 10 Done Sent Done false
4786 1971 Acked Write 9 Done Sent Done false
4787 1972 Acked Write 1 Done Sent Done false
4788 1973 Acked Read 5 Done Sent Done false
4789 1974 Acked Read 3 Done Sent Done false
4790 1975 Acked Write 4 Done Sent Done false
4791 1976 Acked Flush 0 Done Sent Done false
4792 1977 Acked Write 5 Done Sent Done false
4793 1978 Acked Write 8 Done Sent Done false
4794 1979 Acked Read 2 Done Sent Done false
4795 1980 Acked Write 2 Done Sent Done false
4796 1981 Acked Read 8 Done Sent Done false
4797 1982 Acked Read 2 Done Sent Done false
4798 1983 Acked Write 6 Done Sent Done false
4799 1984 Acked Write 9 Done Sent Done false
4800 1985 Acked Read 2 Done Sent Done false
4801 1986 Acked Write 9 Done Sent Done false
4802 1987 Acked Read 1 Done Sent Done false
4803 1988 Acked Read 10 Done Sent Done false
4804 1989 Acked Read 10 Done Sent Done false
4805 1990 Acked Read 1 Done Sent Done false
4806 1991 Acked Read 9 Done Sent Done false
4807 1992 Acked Flush 0 Done Sent Done false
4808 1993 Acked Read 2 Done Sent Done false
4809 1994 Acked Write 2 Done Sent Done false
4810 1995 Acked Read 3 Done Sent Done false
4811 1996 Acked Write 7 Done Sent Done false
4812 1997 Acked Read 8 Done Sent Done false
4813 1998 Acked Write 10 Done Sent Done false
4814 1999 Acked Read 4 Done Sent Done false
4815 2000 Acked Write 1 Done Sent Done false
4816 2001 Acked Flush 0 Done Sent Done false
4817 2002 Acked Write 6 Done Sent Done false
4818 2003 Acked Write 5 Done Sent Done false
4819 2004 Acked Flush 0 Done Sent Done false
4820 2005 Acked Read 2 Done Sent Done false
4821 2006 Acked Write 4 Done Sent Done false
4822 2007 Acked Write 6 Done Sent Done false
4823 2008 Acked Write 9 Done Sent Done false
4824 2009 Acked Read 5 Done Sent Done false
4825 2010 Acked Write 1 Done Sent Done false
4826 2011 Acked Write 4 Done Sent Done false
4827 2012 Acked Read 6 Done Sent Done false
4828 2013 Acked Read 3 Done Sent Done false
4829 2014 Acked Write 6 Done Sent Done false
4830 2015 Acked Read 9 Done Sent Done false
4831 2016 Acked Read 4 Done Sent Done false
4832 2017 Acked Read 10 Done Sent Done false
4833 2018 Acked Flush 0 Done Sent Done false
4834 2019 Acked Read 8 Done Sent Done false
4835 2020 Acked Read 8 Done Sent Done false
4836 2021 Acked Read 1 Done Sent Done false
4837 2022 Acked Write 6 Done Sent Done false
4838 2023 Acked Read 10 Done Sent Done false
4839 2024 Acked Write 10 Done Sent Done false
4840 2025 Acked Read 1 Done Sent Done false
4841 2026 Acked Read 7 Done Sent Done false
4842 2027 Acked Read 8 Done Sent Done false
4843 2028 Acked Read 9 Done Sent Done false
4844 2029 Acked Read 4 Done Sent Done false
4845 2030 Acked Write 8 Done Sent Done false
4846 2031 Acked Read 4 Done Sent Done false
4847 2032 Acked Flush 0 Done Sent Done false
4848 2033 Acked Read 9 Done Sent Done false
4849 2034 Acked Write 6 Done Sent Done false
4850 2035 Acked Flush 0 Done Sent Done false
4851 2036 Acked Flush 0 Done Sent Done false
4852 2037 Acked Write 3 Done Sent Done false
4853 2038 Acked Read 3 Done Sent Done false
4854 2039 Acked Flush 0 Done Sent Done false
4855 2040 Acked Flush 0 Done Sent Done false
4856 2041 Acked Read 10 Done Sent Done false
4857 2042 Acked Write 4 Done Sent Done false
4858 2043 Acked Read 7 Done Sent Done false
4859 2044 Acked Read 9 Done Sent Done false
4860 2045 Acked Write 6 Done Sent Done false
4861 2046 Acked Read 7 Done Sent Done false
4862 2047 Acked Write 3 Done Sent Done false
4863 2048 Acked Write 2 Done Sent Done false
4864 2049 Acked Read 1 Done Sent Done false
4865 2050 Acked Read 5 Done Sent Done false
4866 2051 Acked Flush 0 Done Sent Done false
4867 2052 Acked Read 10 Done Sent Done false
4868 2053 Acked Write 10 Done Sent Done false
4869 2054 Acked Read 8 Done Sent Done false
4870 2055 Acked Read 6 Done Sent Done false
4871 2056 Acked Write 5 Done Sent Done false
4872 2057 Acked Write 6 Done Sent Done false
4873 2058 Acked Read 8 Done Sent Done false
4874 2059 Acked Flush 0 Done Sent Done false
4875 2060 Acked Write 4 Done Sent Done false
4876 2061 Acked Read 5 Done Sent Done false
4877 2062 Acked Write 10 Done Sent Done false
4878 2063 Acked Write 7 Done Sent Done false
4879 2064 Acked Write 8 Done Sent Done false
4880 2065 Acked Read 3 Done Sent Done false
4881 2066 Acked Read 2 Done Sent Done false
4882 2067 Acked Read 8 Done Sent Done false
4883 2068 Acked Read 8 Done Sent Done false
4884 2069 Acked Write 4 Done Sent Done false
4885 2070 Acked Write 6 Done Sent Done false
4886 2071 Acked Read 7 Done Sent Done false
4887 2072 Acked Write 6 Done Sent Done false
4888 2073 Acked Read 1 Done Sent Done false
4889 2074 Acked Write 6 Done Sent Done false
4890 2075 Acked Read 7 Done Sent Done false
4891 2076 Acked Write 5 Done Sent Done false
4892 2077 Acked Write 8 Done Sent Done false
4893 2078 Acked Read 5 Done Sent Done false
4894 2079 Acked Write 6 Done Sent Done false
4895 2080 Acked Write 8 Done Sent Done false
4896 2081 Acked Flush 0 Done Sent Done false
4897 2082 Acked Write 1 Done Sent Done false
4898 2083 Acked Write 1 Done Sent Done false
4899 2084 Acked Read 1 Done Sent Done false
4900 2085 Acked Write 10 Done Sent Done false
4901 2086 Acked Read 5 Done Sent Done false
4902 2087 Acked Read 9 Done Sent Done false
4903 2088 Acked Read 7 Done Sent Done false
4904 2089 Acked Write 3 Done Sent Done false
4905 2090 Acked Read 9 Done Sent Done false
4906 2091 Acked Read 3 Done Sent Done false
4907 2092 Acked Read 9 Done Sent Done false
4908 2093 Acked Read 3 Done Sent Done false
4909 2094 Acked Write 10 Done Sent Done false
4910 2095 Acked Write 7 Done Sent Done false
4911 2096 Acked Read 8 Done Sent Done false
4912 2097 Acked Read 7 Done Sent Done false
4913 2098 Acked Read 2 Done Sent Done false
4914 2099 Acked Read 5 Done Sent Done false
4915 2100 Acked Write 3 Done Sent Done false
4916 2101 Acked Write 7 Done Sent Done false
4917 2102 Acked Write 8 Done Sent Done false
4918 2103 Acked Write 9 Done Sent Done false
4919 2104 Acked Write 6 Done Sent Done false
4920 2105 Acked Write 1 Done Sent Done false
4921 2106 Acked Read 8 Done Sent Done false
4922 2107 Acked Flush 0 Done Sent Done false
4923 2108 Acked Write 9 Done Sent Done false
4924 2109 Acked Write 7 Done Sent Done false
4925 2110 Acked Flush 0 Done Sent Done false
4926 2111 Acked Read 3 Done Sent Done false
4927 2112 Acked Flush 0 Done Sent Done false
4928 2113 Acked Flush 0 Done Sent Done false
4929 2114 Acked Write 4 Done Sent Done false
4930 2115 Acked Write 8 Done Sent Done false
4931 2116 Acked Read 7 Done Sent Done false
4932 2117 Acked Read 10 Done Sent Done false
4933 2118 Acked Read 3 Done Sent Done false
4934 2119 Acked Write 4 Done Sent Done false
4935 2120 Acked Read 8 Done Sent Done false
4936 2121 Acked Read 6 Done Sent Done false
4937 2122 Acked Read 1 Done Sent Done false
4938 2123 Acked Read 5 Done Sent Done false
4939 2124 Acked Read 9 Done Sent Done false
4940 2125 Acked Write 1 Done Sent Done false
4941 2126 Acked Read 6 Done Sent Done false
4942 2127 Acked Read 6 Done Sent Done false
4943 2128 Acked Write 1 Done Sent Done false
4944 2129 Acked Read 1 Done Sent Done false
4945 2130 Acked Read 5 Done Sent Done false
4946 2131 Acked Write 6 Done Sent Done false
4947 2132 Acked Read 7 Done Sent Done false
4948 2133 Acked Write 7 Done Sent Done false
4949 2134 Acked Read 10 Done Sent Done false
4950 2135 Acked Read 6 Done Sent Done false
4951 2136 Acked Read 1 Done Sent Done false
4952 2137 Acked Write 4 Done Sent Done false
4953 2138 Acked Read 1 Done Sent Done false
4954 2139 Acked Write 2 Done Sent Done false
4955 2140 Acked Read 2 Done Sent Done false
4956 2141 Acked Read 5 Done Sent Done false
4957 2142 Acked Write 9 Done Sent Done false
4958 2143 Acked Write 8 Done Sent Done false
4959 2144 Acked Flush 0 Done Sent Done false
4960 2145 Acked Read 5 Done Sent Done false
4961 2146 Acked Read 4 Done Sent Done false
4962 2147 Acked Write 3 Done Sent Done false
4963 2148 Acked Flush 0 Done Sent Done false
4964 2149 Acked Write 6 Done Sent Done false
4965 2150 Acked Write 3 Done Sent Done false
4966 2151 Acked Read 8 Done Sent Done false
4967 2152 Acked Read 4 Done Sent Done false
4968 2153 Acked Write 1 Done Sent Done false
4969 2154 Acked Write 2 Done Sent Done false
4970 2155 Acked Read 3 Done Sent Done false
4971 2156 Acked Read 2 Done Sent Done false
4972 2157 Acked Read 10 Done Sent Done false
4973 2158 Acked Write 2 Done Sent Done false
4974 2159 Acked Read 4 Done Sent Done false
4975 2160 Acked Read 7 Done Sent Done false
4976 2161 Acked Write 1 Done Sent Done false
4977 2162 Acked Write 4 Done Sent Done false
4978 2163 Acked Write 6 Done Sent Done false
4979 2164 Acked Write 10 Done Sent Done false
4980 2165 Acked Flush 0 Done Sent Done false
4981 2166 Acked Read 8 Done Sent Done false
4982 2167 Acked Write 1 Done Sent Done false
4983 2168 Acked Read 4 Done Sent Done false
4984 2169 Acked Read 5 Done Sent Done false
4985 2170 Acked Flush 0 Done Sent Done false
4986 2171 Acked Flush 0 Done Sent Done false
4987 2172 Acked Read 5 Done Sent Done false
4988 2173 Acked Read 7 Done Sent Done false
4989 2174 Acked Write 4 Done Sent Done false
4990 2175 Acked Write 3 Done Sent Done false
4991 2176 Acked Read 7 Done Sent Done false
4992 2177 Acked Write 7 Done Sent Done false
4993 2178 Acked Read 8 Done Sent Done false
4994 2179 Acked Write 6 Done Sent Done false
4995 2180 Acked Read 7 Done Sent Done false
4996 2181 Acked Write 9 Done Sent Done false
4997 2182 Acked Write 5 Done Sent Done false
4998 2183 Acked Write 9 Done Sent Done false
4999 2184 Acked Write 8 Done Sent Done false
5000 2185 Acked Write 2 Done Sent Done false
5001 2186 Acked Write 4 Done Sent Done false
5002 2187 Acked Write 8 Done Sent Done false
5003 2188 Acked Write 6 Done Sent Done false
5004 2189 Acked Write 3 Done Sent Done false
5005 2190 Acked Write 3 Done Sent Done false
5006 2191 Acked Read 4 Done Sent Done false
5007 2192 Acked Write 4 Done Sent Done false
5008 2193 Acked Read 10 Done Sent Done false
5009 2194 Acked Write 6 Done Sent Done false
5010 2195 Acked Read 2 Done Sent Done false
5011 2196 Acked Read 9 Done Sent Done false
5012 2197 Acked Read 4 Done Sent Done false
5013 2198 Acked Read 6 Done Sent Done false
5014 2199 Acked Write 10 Done Sent Done false
5015 2200 Acked Flush 0 Done Sent Done false
5016 2201 Acked Write 2 Done Sent Done false
5017 2202 Acked Read 8 Done Sent Done false
5018 2203 Acked Write 6 Done Sent Done false
5019 2204 Acked Write 1 Done Sent Done false
5020 2205 Acked Write 4 Done Sent Done false
5021 2206 Acked Read 8 Done Sent Done false
5022 2207 Acked Flush 0 Done Sent Done false
5023 STATES DS:0 DS:1 DS:2 TOTAL
5024 Sent 0 302 0 302
5025 Done 302 0 302 604
5026 Skipped 0 0 0 0
5027 Error 0 0 0 0
5028 Last Flush: Some(JobId(2207)) Some(JobId(1905)) Some(JobId(2207))
5029 Downstairs last five completed: 1905 1904 1903 1902 1901
5030 Upstairs last five completed: 2207 2206 2205 2204 2203
5031 CLIENT: Up:0 ds:302 act:2
5032 ----------------------------------------------------------------
5033 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
5034 States: Active Offline Active
5035 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5036 1906 Acked Read 7 Done Sent Done false
5037 1907 Acked Read 9 Done Sent Done false
5038 1908 Acked Write 2 Done Sent Done false
5039 1909 Acked Read 1 Done Sent Done false
5040 1910 Acked Read 5 Done Sent Done false
5041 1911 Acked Write 3 Done Sent Done false
5042 1912 Acked Read 5 Done Sent Done false
5043 1913 Acked Write 9 Done Sent Done false
5044 1914 Acked Read 7 Done Sent Done false
5045 1915 Acked Read 10 Done Sent Done false
5046 1916 Acked Read 6 Done Sent Done false
5047 1917 Acked Read 5 Done Sent Done false
5048 1918 Acked Write 3 Done Sent Done false
5049 1919 Acked Read 8 Done Sent Done false
5050 1920 Acked Read 7 Done Sent Done false
5051 1921 Acked Read 8 Done Sent Done false
5052 1922 Acked Read 1 Done Sent Done false
5053 1923 Acked Write 7 Done Sent Done false
5054 1924 Acked Write 1 Done Sent Done false
5055 1925 Acked Write 3 Done Sent Done false
5056 1926 Acked Write 4 Done Sent Done false
5057 1927 Acked Read 9 Done Sent Done false
5058 1928 Acked Read 3 Done Sent Done false
5059 1929 Acked Read 6 Done Sent Done false
5060 1930 Acked Write 1 Done Sent Done false
5061 1931 Acked Write 1 Done Sent Done false
5062 1932 Acked Write 9 Done Sent Done false
5063 1933 Acked Write 7 Done Sent Done false
5064 1934 Acked Read 9 Done Sent Done false
5065 1935 Acked Write 1 Done Sent Done false
5066 1936 Acked Write 2 Done Sent Done false
5067 1937 Acked Read 10 Done Sent Done false
5068 1938 Acked Read 1 Done Sent Done false
5069 1939 Acked Write 7 Done Sent Done false
5070 1940 Acked Read 3 Done Sent Done false
5071 1941 Acked Write 1 Done Sent Done false
5072 1942 Acked Read 7 Done Sent Done false
5073 1943 Acked Write 9 Done Sent Done false
5074 1944 Acked Read 3 Done Sent Done false
5075 1945 Acked Read 1 Done Sent Done false
5076 1946 Acked Write 10 Done Sent Done false
5077 1947 Acked Write 4 Done Sent Done false
5078 1948 Acked Read 1 Done Sent Done false
5079 1949 Acked Read 3 Done Sent Done false
5080 1950 Acked Write 2 Done Sent Done false
5081 1951 Acked Read 4 Done Sent Done false
5082 1952 Acked Flush 0 Done Sent Done false
5083 1953 Acked Flush 0 Done Sent Done false
5084 1954 Acked Write 7 Done Sent Done false
5085 1955 Acked Read 10 Done Sent Done false
5086 1956 Acked Write 5 Done Sent Done false
5087 1957 Acked Write 2 Done Sent Done false
5088 1958 Acked Write 9 Done Sent Done false
5089 1959 Acked Write 10 Done Sent Done false
5090 1960 Acked Read 2 Done Sent Done false
5091 1961 Acked Read 6 Done Sent Done false
5092 1962 Acked Read 9 Done Sent Done false
5093 1963 Acked Write 3 Done Sent Done false
5094 1964 Acked Write 7 Done Sent Done false
5095 1965 Acked Flush 0 Done Sent Done false
5096 1966 Acked Read 6 Done Sent Done false
5097 1967 Acked Write 6 Done Sent Done false
5098 1968 Acked Write 8 Done Sent Done false
5099 1969 Acked Write 7 Done Sent Done false
5100 1970 Acked Read 10 Done Sent Done false
5101 1971 Acked Write 9 Done Sent Done false
5102 1972 Acked Write 1 Done Sent Done false
5103 1973 Acked Read 5 Done Sent Done false
5104 1974 Acked Read 3 Done Sent Done false
5105 1975 Acked Write 4 Done Sent Done false
5106 1976 Acked Flush 0 Done Sent Done false
5107 1977 Acked Write 5 Done Sent Done false
5108 1978 Acked Write 8 Done Sent Done false
5109 1979 Acked Read 2 Done Sent Done false
5110 1980 Acked Write 2 Done Sent Done false
5111 1981 Acked Read 8 Done Sent Done false
5112 1982 Acked Read 2 Done Sent Done false
5113 1983 Acked Write 6 Done Sent Done false
5114 1984 Acked Write 9 Done Sent Done false
5115 1985 Acked Read 2 Done Sent Done false
5116 1986 Acked Write 9 Done Sent Done false
5117 1987 Acked Read 1 Done Sent Done false
5118 1988 Acked Read 10 Done Sent Done false
5119 1989 Acked Read 10 Done Sent Done false
5120 1990 Acked Read 1 Done Sent Done false
5121 1991 Acked Read 9 Done Sent Done false
5122 1992 Acked Flush 0 Done Sent Done false
5123 1993 Acked Read 2 Done Sent Done false
5124 1994 Acked Write 2 Done Sent Done false
5125 1995 Acked Read 3 Done Sent Done false
5126 1996 Acked Write 7 Done Sent Done false
5127 1997 Acked Read 8 Done Sent Done false
5128 1998 Acked Write 10 Done Sent Done false
5129 1999 Acked Read 4 Done Sent Done false
5130 2000 Acked Write 1 Done Sent Done false
5131 2001 Acked Flush 0 Done Sent Done false
5132 2002 Acked Write 6 Done Sent Done false
5133 2003 Acked Write 5 Done Sent Done false
5134 2004 Acked Flush 0 Done Sent Done false
5135 2005 Acked Read 2 Done Sent Done false
5136 2006 Acked Write 4 Done Sent Done false
5137 2007 Acked Write 6 Done Sent Done false
5138 2008 Acked Write 9 Done Sent Done false
5139 2009 Acked Read 5 Done Sent Done false
5140 2010 Acked Write 1 Done Sent Done false
5141 2011 Acked Write 4 Done Sent Done false
5142 2012 Acked Read 6 Done Sent Done false
5143 2013 Acked Read 3 Done Sent Done false
5144 2014 Acked Write 6 Done Sent Done false
5145 2015 Acked Read 9 Done Sent Done false
5146 2016 Acked Read 4 Done Sent Done false
5147 2017 Acked Read 10 Done Sent Done false
5148 2018 Acked Flush 0 Done Sent Done false
5149 2019 Acked Read 8 Done Sent Done false
5150 2020 Acked Read 8 Done Sent Done false
5151 2021 Acked Read 1 Done Sent Done false
5152 2022 Acked Write 6 Done Sent Done false
5153 2023 Acked Read 10 Done Sent Done false
5154 2024 Acked Write 10 Done Sent Done false
5155 2025 Acked Read 1 Done Sent Done false
5156 2026 Acked Read 7 Done Sent Done false
5157 2027 Acked Read 8 Done Sent Done false
5158 2028 Acked Read 9 Done Sent Done false
5159 2029 Acked Read 4 Done Sent Done false
5160 2030 Acked Write 8 Done Sent Done false
5161 2031 Acked Read 4 Done Sent Done false
5162 2032 Acked Flush 0 Done Sent Done false
5163 2033 Acked Read 9 Done Sent Done false
5164 2034 Acked Write 6 Done Sent Done false
5165 2035 Acked Flush 0 Done Sent Done false
5166 2036 Acked Flush 0 Done Sent Done false
5167 2037 Acked Write 3 Done Sent Done false
5168 2038 Acked Read 3 Done Sent Done false
5169 2039 Acked Flush 0 Done Sent Done false
5170 2040 Acked Flush 0 Done Sent Done false
5171 2041 Acked Read 10 Done Sent Done false
5172 2042 Acked Write 4 Done Sent Done false
5173 2043 Acked Read 7 Done Sent Done false
5174 2044 Acked Read 9 Done Sent Done false
5175 2045 Acked Write 6 Done Sent Done false
5176 2046 Acked Read 7 Done Sent Done false
5177 2047 Acked Write 3 Done Sent Done false
5178 2048 Acked Write 2 Done Sent Done false
5179 2049 Acked Read 1 Done Sent Done false
5180 2050 Acked Read 5 Done Sent Done false
5181 2051 Acked Flush 0 Done Sent Done false
5182 2052 Acked Read 10 Done Sent Done false
5183 2053 Acked Write 10 Done Sent Done false
5184 2054 Acked Read 8 Done Sent Done false
5185 2055 Acked Read 6 Done Sent Done false
5186 2056 Acked Write 5 Done Sent Done false
5187 2057 Acked Write 6 Done Sent Done false
5188 2058 Acked Read 8 Done Sent Done false
5189 2059 Acked Flush 0 Done Sent Done false
5190 2060 Acked Write 4 Done Sent Done false
5191 2061 Acked Read 5 Done Sent Done false
5192 2062 Acked Write 10 Done Sent Done false
5193 2063 Acked Write 7 Done Sent Done false
5194 2064 Acked Write 8 Done Sent Done false
5195 2065 Acked Read 3 Done Sent Done false
5196 2066 Acked Read 2 Done Sent Done false
5197 2067 Acked Read 8 Done Sent Done false
5198 2068 Acked Read 8 Done Sent Done false
5199 2069 Acked Write 4 Done Sent Done false
5200 2070 Acked Write 6 Done Sent Done false
5201 2071 Acked Read 7 Done Sent Done false
5202 2072 Acked Write 6 Done Sent Done false
5203 2073 Acked Read 1 Done Sent Done false
5204 2074 Acked Write 6 Done Sent Done false
5205 2075 Acked Read 7 Done Sent Done false
5206 2076 Acked Write 5 Done Sent Done false
5207 2077 Acked Write 8 Done Sent Done false
5208 2078 Acked Read 5 Done Sent Done false
5209 2079 Acked Write 6 Done Sent Done false
5210 2080 Acked Write 8 Done Sent Done false
5211 2081 Acked Flush 0 Done Sent Done false
5212 2082 Acked Write 1 Done Sent Done false
5213 2083 Acked Write 1 Done Sent Done false
5214 2084 Acked Read 1 Done Sent Done false
5215 2085 Acked Write 10 Done Sent Done false
5216 2086 Acked Read 5 Done Sent Done false
5217 2087 Acked Read 9 Done Sent Done false
5218 2088 Acked Read 7 Done Sent Done false
5219 2089 Acked Write 3 Done Sent Done false
5220 2090 Acked Read 9 Done Sent Done false
5221 2091 Acked Read 3 Done Sent Done false
5222 2092 Acked Read 9 Done Sent Done false
5223 2093 Acked Read 3 Done Sent Done false
5224 2094 Acked Write 10 Done Sent Done false
5225 2095 Acked Write 7 Done Sent Done false
5226 2096 Acked Read 8 Done Sent Done false
5227 2097 Acked Read 7 Done Sent Done false
5228 2098 Acked Read 2 Done Sent Done false
5229 2099 Acked Read 5 Done Sent Done false
5230 2100 Acked Write 3 Done Sent Done false
5231 2101 Acked Write 7 Done Sent Done false
5232 2102 Acked Write 8 Done Sent Done false
5233 2103 Acked Write 9 Done Sent Done false
5234 2104 Acked Write 6 Done Sent Done false
5235 2105 Acked Write 1 Done Sent Done false
5236 2106 Acked Read 8 Done Sent Done false
5237 2107 Acked Flush 0 Done Sent Done false
5238 2108 Acked Write 9 Done Sent Done false
5239 2109 Acked Write 7 Done Sent Done false
5240 2110 Acked Flush 0 Done Sent Done false
5241 2111 Acked Read 3 Done Sent Done false
5242 2112 Acked Flush 0 Done Sent Done false
5243 2113 Acked Flush 0 Done Sent Done false
5244 2114 Acked Write 4 Done Sent Done false
5245 2115 Acked Write 8 Done Sent Done false
5246 2116 Acked Read 7 Done Sent Done false
5247 2117 Acked Read 10 Done Sent Done false
5248 2118 Acked Read 3 Done Sent Done false
5249 2119 Acked Write 4 Done Sent Done false
5250 2120 Acked Read 8 Done Sent Done false
5251 2121 Acked Read 6 Done Sent Done false
5252 2122 Acked Read 1 Done Sent Done false
5253 2123 Acked Read 5 Done Sent Done false
5254 2124 Acked Read 9 Done Sent Done false
5255 2125 Acked Write 1 Done Sent Done false
5256 2126 Acked Read 6 Done Sent Done false
5257 2127 Acked Read 6 Done Sent Done false
5258 2128 Acked Write 1 Done Sent Done false
5259 2129 Acked Read 1 Done Sent Done false
5260 2130 Acked Read 5 Done Sent Done false
5261 2131 Acked Write 6 Done Sent Done false
5262 2132 Acked Read 7 Done Sent Done false
5263 2133 Acked Write 7 Done Sent Done false
5264 2134 Acked Read 10 Done Sent Done false
5265 2135 Acked Read 6 Done Sent Done false
5266 2136 Acked Read 1 Done Sent Done false
5267 2137 Acked Write 4 Done Sent Done false
5268 2138 Acked Read 1 Done Sent Done false
5269 2139 Acked Write 2 Done Sent Done false
5270 2140 Acked Read 2 Done Sent Done false
5271 2141 Acked Read 5 Done Sent Done false
5272 2142 Acked Write 9 Done Sent Done false
5273 2143 Acked Write 8 Done Sent Done false
5274 2144 Acked Flush 0 Done Sent Done false
5275 2145 Acked Read 5 Done Sent Done false
5276 2146 Acked Read 4 Done Sent Done false
5277 2147 Acked Write 3 Done Sent Done false
5278 2148 Acked Flush 0 Done Sent Done false
5279 2149 Acked Write 6 Done Sent Done false
5280 2150 Acked Write 3 Done Sent Done false
5281 2151 Acked Read 8 Done Sent Done false
5282 2152 Acked Read 4 Done Sent Done false
5283 2153 Acked Write 1 Done Sent Done false
5284 2154 Acked Write 2 Done Sent Done false
5285 2155 Acked Read 3 Done Sent Done false
5286 2156 Acked Read 2 Done Sent Done false
5287 2157 Acked Read 10 Done Sent Done false
5288 2158 Acked Write 2 Done Sent Done false
5289 2159 Acked Read 4 Done Sent Done false
5290 2160 Acked Read 7 Done Sent Done false
5291 2161 Acked Write 1 Done Sent Done false
5292 2162 Acked Write 4 Done Sent Done false
5293 2163 Acked Write 6 Done Sent Done false
5294 2164 Acked Write 10 Done Sent Done false
5295 2165 Acked Flush 0 Done Sent Done false
5296 2166 Acked Read 8 Done Sent Done false
5297 2167 Acked Write 1 Done Sent Done false
5298 2168 Acked Read 4 Done Sent Done false
5299 2169 Acked Read 5 Done Sent Done false
5300 2170 Acked Flush 0 Done Sent Done false
5301 2171 Acked Flush 0 Done Sent Done false
5302 2172 Acked Read 5 Done Sent Done false
5303 2173 Acked Read 7 Done Sent Done false
5304 2174 Acked Write 4 Done Sent Done false
5305 2175 Acked Write 3 Done Sent Done false
5306 2176 Acked Read 7 Done Sent Done false
5307 2177 Acked Write 7 Done Sent Done false
5308 2178 Acked Read 8 Done Sent Done false
5309 2179 Acked Write 6 Done Sent Done false
5310 2180 Acked Read 7 Done Sent Done false
5311 2181 Acked Write 9 Done Sent Done false
5312 2182 Acked Write 5 Done Sent Done false
5313 2183 Acked Write 9 Done Sent Done false
5314 2184 Acked Write 8 Done Sent Done false
5315 2185 Acked Write 2 Done Sent Done false
5316 2186 Acked Write 4 Done Sent Done false
5317 2187 Acked Write 8 Done Sent Done false
5318 2188 Acked Write 6 Done Sent Done false
5319 2189 Acked Write 3 Done Sent Done false
5320 2190 Acked Write 3 Done Sent Done false
5321 2191 Acked Read 4 Done Sent Done false
5322 2192 Acked Write 4 Done Sent Done false
5323 2193 Acked Read 10 Done Sent Done false
5324 2194 Acked Write 6 Done Sent Done false
5325 2195 Acked Read 2 Done Sent Done false
5326 2196 Acked Read 9 Done Sent Done false
5327 2197 Acked Read 4 Done Sent Done false
5328 2198 Acked Read 6 Done Sent Done false
5329 2199 Acked Write 10 Done Sent Done false
5330 2200 Acked Flush 0 Done Sent Done false
5331 2201 Acked Write 2 Done Sent Done false
5332 2202 Acked Read 8 Done Sent Done false
5333 2203 Acked Write 6 Done Sent Done false
5334 2204 Acked Write 1 Done Sent Done false
5335 2205 Acked Write 4 Done Sent Done false
5336 2206 Acked Read 8 Done Sent Done false
5337 2207 Acked Flush 0 Done Sent Done false
5338 STATES DS:0 DS:1 DS:2 TOTAL
5339 Sent 0 302 0 302
5340 Done 302 0 302 604
5341 Skipped 0 0 0 0
5342 Error 0 0 0 0
5343 Last Flush: Some(JobId(2207)) Some(JobId(1905)) Some(JobId(2207))
5344 Downstairs last five completed: 1905 1904 1903 1902 1901
5345 Upstairs last five completed: 2207 2206 2205 2204 2203
5346 CLIENT: Up:0 ds:302 act:2
53472025-01-23T18:49:01.713ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
53482025-01-23T18:49:01.713ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36700) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
53492025-01-23T18:49:01.714ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
53502025-01-23T18:49:01.715ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
53512025-01-23T18:49:01.715ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
53522025-01-23T18:49:01.715ZINFOcrucible: send last flush ID to this DS: Some(JobId(1905)) = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
53532025-01-23T18:49:01.715ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
53542025-01-23T18:49:01.715ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(1905)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
53552025-01-23T18:49:01.715ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(1905)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
5356 ----------------------------------------------------------------
5357 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5358 Downstairs last five completed: 2207 2206 2205 2204 2203
5359 Upstairs last five completed: 2207 2206 2205 2204 2203
5360 CLIENT: Up:0 ds:0 act:3
5361 Replay: All jobs finished
5362 Main task has work to do, go find it
5363 got dsc Stop(2)
5364 stop 2
5365 Jan 23 18:49:01.955[8830] Got stop action so:false kr:false
5366 INFO request completed, latency_us: 314, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 985b967e-1151-44b9-894e-f0a4c77f4764, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
53672025-01-23T18:49:01.956ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
53682025-01-23T18:49:01.956ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
53692025-01-23T18:49:01.956ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
5370 Jan 23 18:49:01.956 INFO request completed, latency_us: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T18:49:01.95650851Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
5371 134, response_code: 200, uri: /state/cid/2, method: GET, req_id: 91db9674-2737-4c63-85af-44495c57ae42, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
5372 [8830] Exited with: ExitStatus(unix_wait_status(9))
5373 [8830][2] reports Exit
5374 Jan 23 18:49:05.959 INFO request completed, latency_us: 338, response_code: 200, uri: /state/cid/2, method: GET, req_id: 71ac0b7e-1ace-473f-b569-03b0125fb2f6, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
5375 001/300 Read block 4867 len 24576
5376 002/300 Write block 3876 len 24576 data: 2 2 2 2 2 2
5377 003/300 Write block 2258 len 8192 data: 3 3
5378 004/300 Write block 2466 len 8192 data: 3 3
5379 005/300 Read block 719 len 20480
5380 006/300 Write block 3197 len 4096 data: 2
5381 007/300 Write block 3902 len 20480 data: 2 2 2 2 2
5382 008/300 Write block 1290 len 20480 data: 2 2 2 2 2
5383 009/300 Read block 22 len 28672
5384 010/300 Write block 2930 len 24576 data: 2 2 2 2 2 2
5385 011/300 Read block 3630 len 28672
5386 012/300 Read block 1824 len 40960
5387 013/300 Flush
5388 014/300 Flush
5389 015/300 Flush
5390 016/300 Write block 3379 len 8192 data: 2 2
5391 017/300 Read block 3084 len 20480
5392 018/300 Write block 3560 len 4096 data: 2
5393 019/300 Write block 90 len 4096 data: 4
5394 020/300 Write block 3167 len 12288 data: 2 2 2
5395 021/300 Read block 3643 len 4096
5396 022/300 Read block 2766 len 40960
5397 023/300 Write block 3735 len 36864 data: 2 2 2 2 2 2 2 2 2
5398 024/300 Write block 4485 len 4096 data: 3
5399 025/300 Write block 1191 len 20480 data: 3 2 2 2 2
5400 026/300 Flush
5401 027/300 Read block 1677 len 8192
5402 028/300 Write block 3875 len 36864 data: 2 3 3 3 3 3 3 2 2
5403 029/300 Write block 3639 len 8192 data: 4 4
5404 030/300 Write block 2293 len 28672 data: 2 2 2 2 2 2 2
5405 031/300 Read block 3989 len 40960
5406 032/300 Write block 4533 len 12288 data: 2 2 2
5407 033/300 Read block 3881 len 4096
5408 034/300 Write block 2752 len 4096 data: 2
5409 035/300 Write block 4720 len 32768 data: 2 2 2 2 2 2 2 2
5410 036/300 Read block 1232 len 32768
5411 037/300 Read block 349 len 8192
5412 038/300 Read block 4442 len 20480
5413 039/300 Read block 4163 len 8192
5414 040/300 Write block 1132 len 40960 data: 3 3 3 3 3 3 3 4 3 3
5415 041/300 Write block 150 len 20480 data: 3 3 2 2 2
5416 042/300 Write block 4028 len 24576 data: 3 3 3 3 2 2
5417 043/300 Write block 4289 len 12288 data: 3 3 3
5418 044/300 Read block 2524 len 28672
5419 045/300 Read block 4934 len 16384
5420 046/300 Write block 4689 len 40960 data: 3 2 2 2 2 2 2 2 2 2
5421 047/300 Write block 1283 len 36864 data: 2 2 2 2 2 2 2 3 3
5422 048/300 Read block 104 len 4096
5423 049/300 Read block 2394 len 20480
5424 050/300 Write block 1733 len 4096 data: 3
5425 051/300 Read block 4160 len 20480
5426 052/300 Read block 4667 len 12288
5427 053/300 Read block 2488 len 32768
5428 054/300 Write block 3354 len 4096 data: 3
5429 055/300 Read block 1256 len 32768
5430 056/300 Write block 4883 len 4096 data: 3
5431 057/300 Flush
5432 058/300 Read block 845 len 8192
5433 059/300 Read block 3023 len 4096
5434 060/300 Flush
5435 061/300 Write block 2858 len 4096 data: 3
5436 062/300 Write block 396 len 24576 data: 2 2 2 2 2 2
5437 063/300 Write block 2099 len 4096 data: 2
5438 064/300 Read block 1520 len 16384
5439 065/300 Read block 1850 len 40960
5440 066/300 Read block 1373 len 4096
5441 067/300 Write block 4830 len 32768 data: 2 2 2 2 2 2 2 2
5442 068/300 Read block 3333 len 12288
5443 069/300 Flush
5444 070/300 Read block 3346 len 36864
5445 071/300 Write block 2886 len 28672 data: 3 3 3 3 3 3 3
5446 072/300 Write block 347 len 28672 data: 3 3 3 3 3 3 2
5447 073/300 Read block 3164 len 8192
5448 074/300 Read block 4608 len 4096
5449 075/300 Read block 1917 len 16384
5450 076/300 Write block 3162 len 24576 data: 2 2 2 2 2 3
5451 077/300 Write block 2504 len 28672 data: 2 2 2 2 2 2 2
5452 078/300 Write block 602 len 36864 data: 3 2 2 2 2 2 2 2 2
5453 079/300 Read block 4276 len 24576
5454 080/300 Read block 44 len 8192
5455 081/300 Read block 2202 len 20480
5456 082/300 Read block 584 len 40960
5457 083/300 Read block 2402 len 20480
5458 084/300 Read block 910 len 4096
5459 085/300 Write block 863 len 24576 data: 2 2 2 2 2 2
5460 086/300 Read block 3830 len 40960
5461 087/300 Write block 2792 len 32768 data: 3 3 3 3 2 2 3 2
5462 088/300 Read block 1240 len 4096
5463 089/300 Write block 1356 len 4096 data: 3
5464 090/300 Flush
5465 091/300 Write block 1974 len 24576 data: 2 2 3 3 3 3
5466 092/300 Read block 2978 len 24576
5467 093/300 Read block 3448 len 40960
5468 094/300 Read block 3003 len 36864
5469 095/300 Read block 3819 len 36864
5470 096/300 Flush
5471 097/300 Write block 306 len 32768 data: 2 2 2 3 3 3 3 3
5472 098/300 Flush
5473 099/300 Write block 268 len 4096 data: 4
5474 100/300 Flush
5475 101/300 Write block 2069 len 4096 data: 2
5476 102/300 Read block 474 len 8192
5477 103/300 Read block 3417 len 36864
5478 104/300 Write block 2532 len 36864 data: 2 2 2 2 2 2 2 2 3
5479 105/300 Read block 4575 len 36864
5480 106/300 Write block 1613 len 20480 data: 2 2 2 2 2
5481 107/300 Read block 2174 len 32768
5482 108/300 Read block 24 len 40960
5483 109/300 Write block 4592 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5484 110/300 Flush
5485 111/300 Flush
5486 112/300 Write block 4342 len 32768 data: 3 3 4 3 3 3 3 3
5487 113/300 Read block 1440 len 8192
5488 114/300 Write block 4394 len 4096 data: 2
5489 115/300 Write block 2753 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5490 116/300 Write block 3556 len 20480 data: 2 2 2 2 3
5491 117/300 Write block 3518 len 24576 data: 2 2 2 2 2 2
5492 118/300 Read block 4580 len 4096
5493 119/300 Read block 3107 len 8192
5494 120/300 Read block 375 len 12288
5495 121/300 Write block 729 len 24576 data: 2 3 3 2 2 2
5496 122/300 Flush
5497 123/300 Write block 525 len 28672 data: 2 2 4 4 4 3 3
5498 124/300 Read block 233 len 40960
5499 125/300 Read block 3639 len 12288
5500 126/300 Write block 3295 len 16384 data: 4 4 4 3
5501 127/300 Read block 4301 len 40960
5502 128/300 Read block 224 len 40960
5503 129/300 Flush
5504 130/300 Flush
5505 131/300 Write block 2518 len 28672 data: 3 3 3 3 3 4 3
5506 132/300 Read block 79 len 20480
5507 133/300 Write block 4421 len 4096 data: 2
5508 134/300 Read block 2632 len 24576
5509 135/300 Read block 2286 len 16384
5510 136/300 Read block 4238 len 4096
5511 137/300 Read block 3548 len 4096
5512 138/300 Write block 1996 len 24576 data: 2 2 2 2 2 2
5513 139/300 Read block 3370 len 28672
5514 140/300 Read block 3099 len 16384
5515 141/300 Write block 3555 len 8192 data: 2 3
5516 142/300 Read block 3445 len 24576
5517 143/300 Read block 2230 len 4096
5518 144/300 Read block 1672 len 8192
5519 145/300 Flush
5520 146/300 Write block 1941 len 16384 data: 3 3 3 3
5521 147/300 Read block 1210 len 28672
5522 148/300 Write block 4786 len 20480 data: 3 3 3 2 2
5523 149/300 Read block 4069 len 36864
5524 150/300 Read block 1265 len 32768
5525 151/300 Flush
5526 152/300 Read block 3058 len 12288
5527 153/300 Read block 3034 len 28672
5528 154/300 Write block 384 len 8192 data: 2 2
5529 155/300 Write block 3425 len 16384 data: 3 3 3 3
5530 156/300 Read block 714 len 20480
5531 157/300 Write block 4312 len 36864 data: 4 3 3 3 2 2 2 2 3
5532 158/300 Write block 532 len 32768 data: 3 3 4 3 4 3 2 2
5533 159/300 Read block 1598 len 20480
5534 160/300 Flush
5535 161/300 Write block 2926 len 20480 data: 2 2 2 2 3
5536 162/300 Write block 367 len 8192 data: 3 2
5537 163/300 Read block 1576 len 40960
5538 164/300 Write block 3769 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5539 165/300 Write block 3988 len 12288 data: 2 2 2
5540 166/300 Read block 4758 len 24576
5541 167/300 Read block 3144 len 36864
5542 168/300 Write block 3586 len 12288 data: 2 2 2
5543 169/300 Write block 3812 len 40960 data: 3 3 3 3 3 3 3 3 3 2
5544 170/300 Flush
5545 171/300 Write block 4139 len 32768 data: 3 3 3 3 2 2 2 4
5546 172/300 Read block 2094 len 12288
5547 173/300 Write block 252 len 16384 data: 3 2 3 3
5548 174/300 Write block 108 len 8192 data: 3 3
5549 175/300 Write block 597 len 24576 data: 2 2 2 2 2 4
5550 176/300 Write block 999 len 16384 data: 2 2 2 2
5551 177/300 Read block 1190 len 32768
5552 178/300 Read block 2127 len 12288
5553 179/300 Write block 2473 len 28672 data: 3 3 3 2 2 2 2
5554 180/300 Read block 2710 len 24576
5555 181/300 Write block 4092 len 8192 data: 3 3
5556 182/300 Write block 4368 len 32768 data: 2 3 3 3 3 4 4 4
5557 183/300 Write block 1810 len 36864 data: 4 4 3 3 3 3 2 2 2
5558 184/300 Write block 4758 len 36864 data: 3 3 3 4 3 3 3 3 3
5559 185/300 Write block 4159 len 36864 data: 3 3 3 3 3 3 2 2 3
5560 186/300 Flush
5561 187/300 Write block 2264 len 32768 data: 2 2 3 3 3 3 3 3
5562 188/300 Read block 1900 len 40960
5563 189/300 Read block 2502 len 12288
5564 190/300 Flush
5565 191/300 Read block 4215 len 16384
5566 192/300 Read block 3598 len 12288
5567 193/300 Read block 4949 len 16384
5568 194/300 Read block 3310 len 16384
5569 195/300 Write block 602 len 24576 data: 5 3 3 3 3 3
5570 196/300 Flush
5571 197/300 Write block 990 len 16384 data: 2 2 2 2
5572 198/300 Read block 1214 len 36864
5573 199/300 Read block 4403 len 32768
5574 200/300 Read block 2152 len 8192
5575 201/300 Read block 3149 len 20480
5576 202/300 Read block 1918 len 32768
5577 203/300 Write block 2685 len 32768 data: 2 2 2 2 2 2 2 2
5578 204/300 Read block 90 len 32768
5579 205/300 Read block 2431 len 36864
5580 206/300 Read block 817 len 20480
5581 207/300 Read block 2046 len 16384
5582 208/300 Write block 842 len 24576 data: 2 2 2 2 2 2
5583 209/300 Write block 125 len 20480 data: 3 3 2 2 2
5584 210/300 Write block 3736 len 12288 data: 3 3 3
5585 211/300 Read block 3778 len 12288
5586 212/300 Read block 3221 len 24576
5587 213/300 Write block 3077 len 40960 data: 3 2 3 3 3 3 3 3 4 4
5588 214/300 Read block 421 len 8192
5589 215/300 Read block 2591 len 32768
5590 216/300 Write block 2956 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5591 217/300 Read block 2431 len 4096
5592 218/300 Read block 892 len 8192
5593 219/300 Write block 4985 len 16384 data: 2 2 2 2
5594 220/300 Write block 1620 len 24576 data: 2 3 3 3 2 2
5595 221/300 Read block 4975 len 28672
5596 222/300 Read block 280 len 24576
5597 223/300 Write block 1521 len 32768 data: 2 2 2 2 2 2 2 2
5598 224/300 Write block 2422 len 12288 data: 2 2 2
5599 225/300 Write block 1300 len 24576 data: 2 2 2 2 2 2
5600 226/300 Read block 4162 len 16384
5601 227/300 Read block 2172 len 36864
5602 228/300 Read block 4354 len 40960
5603 229/300 Read block 3055 len 24576
5604 230/300 Write block 1211 len 36864 data: 2 2 2 2 2 2 2 2 2
5605 231/300 Read block 4209 len 36864
5606 232/300 Write block 3574 len 16384 data: 2 2 2 2
5607 233/300 Read block 467 len 36864
5608 234/300 Write block 725 len 12288 data: 2 2 2
5609 235/300 Read block 2870 len 36864
5610 236/300 Read block 3527 len 32768
5611 237/300 Read block 2319 len 32768
5612 238/300 Read block 3260 len 20480
5613 239/300 Read block 216 len 4096
5614 240/300 Read block 3388 len 16384
5615 241/300 Read block 716 len 36864
5616 242/300 Write block 4930 len 28672 data: 4 4 5 4 4 4 2
5617 243/300 Read block 3560 len 4096
5618 244/300 Read block 111 len 40960
5619 245/300 Write block 832 len 24576 data: 2 3 3 3 3 3
5620 246/300 Read block 1086 len 32768
5621 247/300 Read block 3949 len 16384
5622 248/300 Flush
5623 249/300 Read block 141 len 16384
5624 250/300 Write block 1944 len 28672 data: 4 3 2 2 2 2 2
5625 251/300 Flush
5626 252/300 Flush
5627 253/300 Flush
5628 254/300 Read block 4947 len 24576
5629 255/300 Flush
5630 256/300 Flush
5631 257/300 Read block 3200 len 24576
5632 258/300 Flush
5633 259/300 Write block 2314 len 8192 data: 3 4
5634 260/300 Write block 895 len 28672 data: 2 2 2 2 2 2 2
5635 261/300 Write block 966 len 8192 data: 4 4
5636 262/300 Write block 1026 len 28672 data: 2 2 2 2 2 2 2
5637 263/300 Write block 1717 len 20480 data: 2 2 2 2 2
5638 264/300 Read block 153 len 20480
5639 265/300 Read block 4814 len 12288
5640 266/300 Read block 4149 len 36864
5641 267/300 Read block 4027 len 32768
5642 268/300 Read block 1057 len 32768
5643 269/300 Write block 4962 len 16384 data: 2 2 3 3
5644 270/300 Write block 803 len 28672 data: 2 2 2 2 2 2 2
5645 271/300 Write block 3235 len 20480 data: 3 4 3 3 3
5646 272/300 Write block 2802 len 20480 data: 2 2 2 2 2
5647 273/300 Write block 1526 len 36864 data: 3 3 3 2 2 2 2 2 2
5648 274/300 Read block 2935 len 4096
5649 275/300 Write block 14 len 36864 data: 3 3 3 3 3 3 3 2 2
5650 276/300 Write block 1696 len 20480 data: 3 2 2 3 3
5651 277/300 Write block 3251 len 28672 data: 4 4 4 3 3 3 3
5652 278/300 Write block 899 len 36864 data: 3 3 3 2 2 2 2 2 2
5653 279/300 Write block 2871 len 20480 data: 4 3 2 2 2
5654 280/300 Flush
5655 281/300 Write block 3401 len 12288 data: 2 2 2
5656 282/300 Read block 1834 len 12288
5657 283/300 Read block 670 len 4096
5658 284/300 Write block 4872 len 24576 data: 4 4 4 4 4 2
5659 285/300 Write block 323 len 4096 data: 4
5660 286/300 Write block 2893 len 32768 data: 3 3 3 2 2 2 3 2
5661 287/300 Read block 4442 len 20480
5662 288/300 Write block 1688 len 32768 data: 3 3 3 3 3 3 3 3
5663 289/300 Write block 4753 len 4096 data: 4
5664 290/300 Read block 3664 len 8192
5665 291/300 Read block 4666 len 32768
5666 292/300 Write block 923 len 4096 data: 2
5667 293/300 Read block 4610 len 24576
5668 294/300 Flush
5669 295/300 Read block 1315 len 8192
5670 296/300 Read block 3508 len 20480
5671 297/300 Read block 1928 len 16384
5672 298/300 Read block 98 len 4096
5673 299/300 Read block 275 len 24576
5674 300/300 Read block 1094 len 40960
5675 301/300 Write block 4131 len 12288 data: 2 2 2
5676 Main task has work to do, go find it
5677 got dsc Start(2)
5678 start 2
5679 Jan 23 18:49:06.190 INFO request completed, latency_us: 191, response_code: 204, uri: /start/cid/2, method: POST, req_id: 04f4804d-824d-40eb-a160-775daa97d3b3, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
5680 Replay: started 2, returned:Ok(())
5681 ----------------------------------------------------------------
5682 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5683 States: Active Active Offline
5684 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5685 2208 Acked Read 6 Done Done Sent false
5686 2209 Acked Write 6 Done Done Sent false
5687 2210 Acked Write 2 Done Done Sent false
5688 2211 Acked Write 2 Done Done Sent false
5689 2212 Acked Read 5 Done Done Sent false
5690 2213 Acked Write 1 Done Done Sent false
5691 2214 Acked Write 5 Done Done Sent false
5692 2215 Acked Write 5 Done Done Sent false
5693 2216 Acked Read 7 Done Done Sent false
5694 2217 Acked Write 6 Done Done Sent false
5695 2218 Acked Read 7 Done Done Sent false
5696 2219 Acked Read 10 Done Done Sent false
5697 2220 Acked Flush 0 Done Done Sent false
5698 2221 Acked Flush 0 Done Done Sent false
5699 2222 Acked Flush 0 Done Done Sent false
5700 2223 Acked Write 2 Done Done Sent false
5701 2224 Acked Read 5 Done Done Sent false
5702 2225 Acked Write 1 Done Done Sent false
5703 2226 Acked Write 1 Done Done Sent false
5704 2227 Acked Write 3 Done Done Sent false
5705 2228 Acked Read 1 Done Done Sent false
5706 2229 Acked Read 10 Done Done Sent false
5707 2230 Acked Write 9 Done Done Sent false
5708 2231 Acked Write 1 Done Done Sent false
5709 2232 Acked Write 5 Done Done Sent false
5710 2233 Acked Flush 0 Done Done Sent false
5711 2234 Acked Read 2 Done Done Sent false
5712 2235 Acked Write 9 Done Done Sent false
5713 2236 Acked Write 2 Done Done Sent false
5714 2237 Acked Write 7 Done Done Sent false
5715 2238 Acked Read 10 Done Done Sent false
5716 2239 Acked Write 3 Done Done Sent false
5717 2240 Acked Read 1 Done Done Sent false
5718 2241 Acked Write 1 Done Done Sent false
5719 2242 Acked Write 8 Done Done Sent false
5720 2243 Acked Read 8 Done Done Sent false
5721 2244 Acked Read 2 Done Done Sent false
5722 2245 Acked Read 5 Done Done Sent false
5723 2246 Acked Read 2 Done Done Sent false
5724 2247 Acked Write 10 Done Done Sent false
5725 2248 Acked Write 5 Done Done Sent false
5726 2249 Acked Write 6 Done Done Sent false
5727 2250 Acked Write 3 Done Done Sent false
5728 2251 Acked Read 7 Done Done Sent false
5729 2252 Acked Read 4 Done Done Sent false
5730 2253 Acked Write 10 Done Done Sent false
5731 2254 Acked Write 9 Done Done Sent false
5732 2255 Acked Read 1 Done Done Sent false
5733 2256 Acked Read 5 Done Done Sent false
5734 2257 Acked Write 1 Done Done Sent false
5735 2258 Acked Read 5 Done Done Sent false
5736 2259 Acked Read 3 Done Done Sent false
5737 2260 Acked Read 8 Done Done Sent false
5738 2261 Acked Write 1 Done Done Sent false
5739 2262 Acked Read 8 Done Done Sent false
5740 2263 Acked Write 1 Done Done Sent false
5741 2264 Acked Flush 0 Done Done Sent false
5742 2265 Acked Read 2 Done Done Sent false
5743 2266 Acked Read 1 Done Done Sent false
5744 2267 Acked Flush 0 Done Done Sent false
5745 2268 Acked Write 1 Done Done Sent false
5746 2269 Acked Write 6 Done Done Sent false
5747 2270 Acked Write 1 Done Done Sent false
5748 2271 Acked Read 4 Done Done Sent false
5749 2272 Acked Read 10 Done Done Sent false
5750 2273 Acked Read 1 Done Done Sent false
5751 2274 Acked Write 8 Done Done Sent false
5752 2275 Acked Read 3 Done Done Sent false
5753 2276 Acked Flush 0 Done Done Sent false
5754 2277 Acked Read 9 Done Done Sent false
5755 2278 Acked Write 7 Done Done Sent false
5756 2279 Acked Write 7 Done Done Sent false
5757 2280 Acked Read 2 Done Done Sent false
5758 2281 Acked Read 1 Done Done Sent false
5759 2282 Acked Read 4 Done Done Sent false
5760 2283 Acked Write 6 Done Done Sent false
5761 2284 Acked Write 7 Done Done Sent false
5762 2285 Acked Write 9 Done Done Sent false
5763 2286 Acked Read 6 Done Done Sent false
5764 2287 Acked Read 2 Done Done Sent false
5765 2288 Acked Read 5 Done Done Sent false
5766 2289 Acked Read 10 Done Done Sent false
5767 2290 Acked Read 5 Done Done Sent false
5768 2291 Acked Read 1 Done Done Sent false
5769 2292 Acked Write 6 Done Done Sent false
5770 2293 Acked Read 10 Done Done Sent false
5771 2294 Acked Write 8 Done Done Sent false
5772 2295 Acked Read 1 Done Done Sent false
5773 2296 Acked Write 1 Done Done Sent false
5774 2297 Acked Flush 0 Done Done Sent false
5775 2298 Acked Write 6 Done Done Sent false
5776 2299 Acked Read 6 Done Done Sent false
5777 2300 Acked Read 10 Done Done Sent false
5778 2301 Acked Read 9 Done Done Sent false
5779 2302 Acked Read 9 Done Done Sent false
5780 2303 Acked Flush 0 Done Done Sent false
5781 2304 Acked Write 8 Done Done Sent false
5782 2305 Acked Flush 0 Done Done Sent false
5783 2306 Acked Write 1 Done Done Sent false
5784 2307 Acked Flush 0 Done Done Sent false
5785 2308 Acked Write 1 Done Done Sent false
5786 2309 Acked Read 2 Done Done Sent false
5787 2310 Acked Read 9 Done Done Sent false
5788 2311 Acked Write 9 Done Done Sent false
5789 2312 Acked Read 9 Done Done Sent false
5790 2313 Acked Write 5 Done Done Sent false
5791 2314 Acked Read 8 Done Done Sent false
5792 2315 Acked Read 10 Done Done Sent false
5793 2316 Acked Write 10 Done Done Sent false
5794 2317 Acked Flush 0 Done Done Sent false
5795 2318 Acked Flush 0 Done Done Sent false
5796 2319 Acked Write 8 Done Done Sent false
5797 2320 Acked Read 2 Done Done Sent false
5798 2321 Acked Write 1 Done Done Sent false
5799 2322 Acked Write 10 Done Done Sent false
5800 2323 Acked Write 5 Done Done Sent false
5801 2324 Acked Write 6 Done Done Sent false
5802 2325 Acked Read 1 Done Done Sent false
5803 2326 Acked Read 2 Done Done Sent false
5804 2327 Acked Read 3 Done Done Sent false
5805 2328 Acked Write 6 Done Done Sent false
5806 2329 Acked Flush 0 Done Done Sent false
5807 2330 Acked Write 7 Done Done Sent false
5808 2331 Acked Read 10 Done Done Sent false
5809 2332 Acked Read 3 Done Done Sent false
5810 2333 Acked Write 4 Done Done Sent false
5811 2334 Acked Read 10 Done Done Sent false
5812 2335 Acked Read 10 Done Done Sent false
5813 2336 Acked Flush 0 Done Done Sent false
5814 2337 Acked Flush 0 Done Done Sent false
5815 2338 Acked Write 7 Done Done Sent false
5816 2339 Acked Read 5 Done Done Sent false
5817 2340 Acked Write 1 Done Done Sent false
5818 2341 Acked Read 6 Done Done Sent false
5819 2342 Acked Read 4 Done Done Sent false
5820 2343 Acked Read 1 Done Done Sent false
5821 2344 Acked Read 1 Done Done Sent false
5822 2345 Acked Write 6 Done Done Sent false
5823 2346 Acked Read 7 Done Done Sent false
5824 2347 Acked Read 4 Done Done Sent false
5825 2348 Acked Write 2 Done Done Sent false
5826 2349 Acked Read 6 Done Done Sent false
5827 2350 Acked Read 1 Done Done Sent false
5828 2351 Acked Read 2 Done Done Sent false
5829 2352 Acked Flush 0 Done Done Sent false
5830 2353 Acked Write 4 Done Done Sent false
5831 2354 Acked Read 7 Done Done Sent false
5832 2355 Acked Write 5 Done Done Sent false
5833 2356 Acked Read 9 Done Done Sent false
5834 2357 Acked Read 8 Done Done Sent false
5835 2358 Acked Flush 0 Done Done Sent false
5836 2359 Acked Read 3 Done Done Sent false
5837 2360 Acked Read 7 Done Done Sent false
5838 2361 Acked Write 2 Done Done Sent false
5839 2362 Acked Write 4 Done Done Sent false
5840 2363 Acked Read 5 Done Done Sent false
5841 2364 Acked Write 9 Done Done Sent false
5842 2365 Acked Write 8 Done Done Sent false
5843 2366 Acked Read 5 Done Done Sent false
5844 2367 Acked Flush 0 Done Done Sent false
5845 2368 Acked Write 5 Done Done Sent false
5846 2369 Acked Write 2 Done Done Sent false
5847 2370 Acked Read 10 Done Done Sent false
5848 2371 Acked Write 10 Done Done Sent false
5849 2372 Acked Write 3 Done Done Sent false
5850 2373 Acked Read 6 Done Done Sent false
5851 2374 Acked Read 9 Done Done Sent false
5852 2375 Acked Write 3 Done Done Sent false
5853 2376 Acked Write 10 Done Done Sent false
5854 2377 Acked Flush 0 Done Done Sent false
5855 2378 Acked Write 8 Done Done Sent false
5856 2379 Acked Read 3 Done Done Sent false
5857 2380 Acked Write 4 Done Done Sent false
5858 2381 Acked Write 2 Done Done Sent false
5859 2382 Acked Write 6 Done Done Sent false
5860 2383 Acked Write 4 Done Done Sent false
5861 2384 Acked Read 8 Done Done Sent false
5862 2385 Acked Read 3 Done Done Sent false
5863 2386 Acked Write 7 Done Done Sent false
5864 2387 Acked Read 6 Done Done Sent false
5865 2388 Acked Write 2 Done Done Sent false
5866 2389 Acked Write 8 Done Done Sent false
5867 2390 Acked Write 9 Done Done Sent false
5868 2391 Acked Write 9 Done Done Sent false
5869 2392 Acked Write 9 Done Done Sent false
5870 2393 Acked Flush 0 Done Done Sent false
5871 2394 Acked Write 8 Done Done Sent false
5872 2395 Acked Read 10 Done Done Sent false
5873 2396 Acked Read 3 Done Done Sent false
5874 2397 Acked Flush 0 Done Done Sent false
5875 2398 Acked Read 4 Done Done Sent false
5876 2399 Acked Read 3 Done Done Sent false
5877 2400 Acked Read 4 Done Done Sent false
5878 2401 Acked Read 4 Done Done Sent false
5879 2402 Acked Write 6 Done Done Sent false
5880 2403 Acked Flush 0 Done Done Sent false
5881 2404 Acked Write 4 Done Done Sent false
5882 2405 Acked Read 9 Done Done Sent false
5883 2406 Acked Read 8 Done Done Sent false
5884 2407 Acked Read 2 Done Done Sent false
5885 2408 Acked Read 5 Done Done Sent false
5886 2409 Acked Read 8 Done Done Sent false
5887 2410 Acked Write 8 Done Done Sent false
5888 2411 Acked Read 8 Done Done Sent false
5889 2412 Acked Read 9 Done Done Sent false
5890 2413 Acked Read 5 Done Done Sent false
5891 2414 Acked Read 4 Done Done Sent false
5892 2415 Acked Write 6 Done Done Sent false
5893 2416 Acked Write 5 Done Done Sent false
5894 2417 Acked Write 3 Done Done Sent false
5895 2418 Acked Read 3 Done Done Sent false
5896 2419 Acked Read 6 Done Done Sent false
5897 2420 Acked Write 10 Done Done Sent false
5898 2421 Acked Read 2 Done Done Sent false
5899 2422 Acked Read 8 Done Done Sent false
5900 2423 Acked Write 10 Done Done Sent false
5901 2424 Acked Read 1 Done Done Sent false
5902 2425 Acked Read 2 Done Done Sent false
5903 2426 Acked Write 4 Done Done Sent false
5904 2427 Acked Write 6 Done Done Sent false
5905 2428 Acked Read 7 Done Done Sent false
5906 2429 Acked Read 6 Done Done Sent false
5907 2430 Acked Write 8 Done Done Sent false
5908 2431 Acked Write 3 Done Done Sent false
5909 2432 Acked Write 6 Done Done Sent false
5910 2433 Acked Read 4 Done Done Sent false
5911 2434 Acked Read 9 Done Done Sent false
5912 2435 Acked Read 10 Done Done Sent false
5913 2436 Acked Read 6 Done Done Sent false
5914 2437 Acked Write 9 Done Done Sent false
5915 2438 Acked Read 9 Done Done Sent false
5916 2439 Acked Write 4 Done Done Sent false
5917 2440 Acked Read 9 Done Done Sent false
5918 2441 Acked Write 3 Done Done Sent false
5919 2442 Acked Read 9 Done Done Sent false
5920 2443 Acked Read 8 Done Done Sent false
5921 2444 Acked Read 8 Done Done Sent false
5922 2445 Acked Read 5 Done Done Sent false
5923 2446 Acked Read 1 Done Done Sent false
5924 2447 Acked Read 4 Done Done Sent false
5925 2448 Acked Read 9 Done Done Sent false
5926 2449 Acked Write 7 Done Done Sent false
5927 2450 Acked Read 1 Done Done Sent false
5928 2451 Acked Read 10 Done Done Sent false
5929 2452 Acked Write 6 Done Done Sent false
5930 2453 Acked Read 8 Done Done Sent false
5931 2454 Acked Read 4 Done Done Sent false
5932 2455 Acked Flush 0 Done Done Sent false
5933 2456 Acked Read 4 Done Done Sent false
5934 2457 Acked Write 7 Done Done Sent false
5935 2458 Acked Flush 0 Done Done Sent false
5936 2459 Acked Flush 0 Done Done Sent false
5937 2460 Acked Flush 0 Done Done Sent false
5938 2461 Acked Read 6 Done Done Sent false
5939 2462 Acked Flush 0 Done Done Sent false
5940 2463 Acked Flush 0 Done Done Sent false
5941 2464 Acked Read 6 Done Done Sent false
5942 2465 Acked Flush 0 Done Done Sent false
5943 2466 Acked Write 2 Done Done Sent false
5944 2467 Acked Write 7 Done Done Sent false
5945 2468 Acked Write 2 Done Done Sent false
5946 2469 Acked Write 7 Done Done Sent false
5947 2470 Acked Write 5 Done Done Sent false
5948 2471 Acked Read 5 Done Done Sent false
5949 2472 Acked Read 3 Done Done Sent false
5950 2473 Acked Read 9 Done Done Sent false
5951 2474 Acked Read 8 Done Done Sent false
5952 2475 Acked Read 8 Done Done Sent false
5953 2476 Acked Write 4 Done Done Sent false
5954 2477 Acked Write 7 Done Done Sent false
5955 2478 Acked Write 5 Done Done Sent false
5956 2479 Acked Write 5 Done Done Sent false
5957 2480 Acked Write 9 Done Done Sent false
5958 2481 Acked Read 1 Done Done Sent false
5959 2482 Acked Write 9 Done Done Sent false
5960 2483 Acked Write 5 Done Done Sent false
5961 2484 Acked Write 7 Done Done Sent false
5962 2485 Acked Write 9 Done Done Sent false
5963 2486 Acked Write 5 Done Done Sent false
5964 2487 Acked Flush 0 Done Done Sent false
5965 2488 Acked Write 3 Done Done Sent false
5966 2489 Acked Read 3 Done Done Sent false
5967 2490 Acked Read 1 Done Done Sent false
5968 2491 Acked Write 6 Done Done Sent false
5969 2492 Acked Write 1 Done Done Sent false
5970 2493 Acked Write 8 Done Done Sent false
5971 2494 Acked Read 5 Done Done Sent false
5972 2495 Acked Write 8 Done Done Sent false
5973 2496 Acked Write 1 Done Done Sent false
5974 2497 Acked Read 2 Done Done Sent false
5975 2498 Acked Read 8 Done Done Sent false
5976 2499 Acked Write 1 Done Done Sent false
5977 2500 Acked Read 6 Done Done Sent false
5978 2501 Acked Flush 0 Done Done Sent false
5979 2502 Acked Read 2 Done Done Sent false
5980 2503 Acked Read 5 Done Done Sent false
5981 2504 Acked Read 4 Done Done Sent false
5982 2505 Acked Read 1 Done Done Sent false
5983 2506 Acked Read 6 Done Done Sent false
5984 2507 Acked Read 10 Done Done Sent false
5985 2508 Acked Write 3 Done Done Sent false
5986 STATES DS:0 DS:1 DS:2 TOTAL
5987 Sent 0 0 301 301
5988 Done 301 301 0 602
5989 Skipped 0 0 0 0
5990 Error 0 0 0 0
5991 Last Flush: Some(JobId(2501)) Some(JobId(2501)) Some(JobId(2207))
5992 Downstairs last five completed: 2207 2206 2205 2204 2203
5993 Upstairs last five completed: 2508 2507 2506 2505 2504
5994 CLIENT: Up:0 ds:301 act:2
5995 [8830] got start action
5996 [8830] I am going to restart
5997 Starting downstairs at port 8830
5998 Make output file at "/tmp/dsc/downstairs-8830.txt"
5999 [8830][2] reports Starting
6000 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1205)
6001 [8830][2] reports Running
6002 ----------------------------------------------------------------
6003 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
6004 States: Active Active Offline
6005 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6006 2208 Acked Read 6 Done Done Sent false
6007 2209 Acked Write 6 Done Done Sent false
6008 2210 Acked Write 2 Done Done Sent false
6009 2211 Acked Write 2 Done Done Sent false
6010 2212 Acked Read 5 Done Done Sent false
6011 2213 Acked Write 1 Done Done Sent false
6012 2214 Acked Write 5 Done Done Sent false
6013 2215 Acked Write 5 Done Done Sent false
6014 2216 Acked Read 7 Done Done Sent false
6015 2217 Acked Write 6 Done Done Sent false
6016 2218 Acked Read 7 Done Done Sent false
6017 2219 Acked Read 10 Done Done Sent false
6018 2220 Acked Flush 0 Done Done Sent false
6019 2221 Acked Flush 0 Done Done Sent false
6020 2222 Acked Flush 0 Done Done Sent false
6021 2223 Acked Write 2 Done Done Sent false
6022 2224 Acked Read 5 Done Done Sent false
6023 2225 Acked Write 1 Done Done Sent false
6024 2226 Acked Write 1 Done Done Sent false
6025 2227 Acked Write 3 Done Done Sent false
6026 2228 Acked Read 1 Done Done Sent false
6027 2229 Acked Read 10 Done Done Sent false
6028 2230 Acked Write 9 Done Done Sent false
6029 2231 Acked Write 1 Done Done Sent false
6030 2232 Acked Write 5 Done Done Sent false
6031 2233 Acked Flush 0 Done Done Sent false
6032 2234 Acked Read 2 Done Done Sent false
6033 2235 Acked Write 9 Done Done Sent false
6034 2236 Acked Write 2 Done Done Sent false
6035 2237 Acked Write 7 Done Done Sent false
6036 2238 Acked Read 10 Done Done Sent false
6037 2239 Acked Write 3 Done Done Sent false
6038 2240 Acked Read 1 Done Done Sent false
6039 2241 Acked Write 1 Done Done Sent false
6040 2242 Acked Write 8 Done Done Sent false
6041 2243 Acked Read 8 Done Done Sent false
6042 2244 Acked Read 2 Done Done Sent false
6043 2245 Acked Read 5 Done Done Sent false
6044 2246 Acked Read 2 Done Done Sent false
6045 2247 Acked Write 10 Done Done Sent false
6046 2248 Acked Write 5 Done Done Sent false
6047 2249 Acked Write 6 Done Done Sent false
6048 2250 Acked Write 3 Done Done Sent false
6049 2251 Acked Read 7 Done Done Sent false
6050 2252 Acked Read 4 Done Done Sent false
6051 2253 Acked Write 10 Done Done Sent false
6052 2254 Acked Write 9 Done Done Sent false
6053 2255 Acked Read 1 Done Done Sent false
6054 2256 Acked Read 5 Done Done Sent false
6055 2257 Acked Write 1 Done Done Sent false
6056 2258 Acked Read 5 Done Done Sent false
6057 2259 Acked Read 3 Done Done Sent false
6058 2260 Acked Read 8 Done Done Sent false
6059 2261 Acked Write 1 Done Done Sent false
6060 2262 Acked Read 8 Done Done Sent false
6061 2263 Acked Write 1 Done Done Sent false
6062 2264 Acked Flush 0 Done Done Sent false
6063 2265 Acked Read 2 Done Done Sent false
6064 2266 Acked Read 1 Done Done Sent false
6065 2267 Acked Flush 0 Done Done Sent false
6066 2268 Acked Write 1 Done Done Sent false
6067 2269 Acked Write 6 Done Done Sent false
6068 2270 Acked Write 1 Done Done Sent false
6069 2271 Acked Read 4 Done Done Sent false
6070 2272 Acked Read 10 Done Done Sent false
6071 2273 Acked Read 1 Done Done Sent false
6072 2274 Acked Write 8 Done Done Sent false
6073 2275 Acked Read 3 Done Done Sent false
6074 2276 Acked Flush 0 Done Done Sent false
6075 2277 Acked Read 9 Done Done Sent false
6076 2278 Acked Write 7 Done Done Sent false
6077 2279 Acked Write 7 Done Done Sent false
6078 2280 Acked Read 2 Done Done Sent false
6079 2281 Acked Read 1 Done Done Sent false
6080 2282 Acked Read 4 Done Done Sent false
6081 2283 Acked Write 6 Done Done Sent false
6082 2284 Acked Write 7 Done Done Sent false
6083 2285 Acked Write 9 Done Done Sent false
6084 2286 Acked Read 6 Done Done Sent false
6085 2287 Acked Read 2 Done Done Sent false
6086 2288 Acked Read 5 Done Done Sent false
6087 2289 Acked Read 10 Done Done Sent false
6088 2290 Acked Read 5 Done Done Sent false
6089 2291 Acked Read 1 Done Done Sent false
6090 2292 Acked Write 6 Done Done Sent false
6091 2293 Acked Read 10 Done Done Sent false
6092 2294 Acked Write 8 Done Done Sent false
6093 2295 Acked Read 1 Done Done Sent false
6094 2296 Acked Write 1 Done Done Sent false
6095 2297 Acked Flush 0 Done Done Sent false
6096 2298 Acked Write 6 Done Done Sent false
6097 2299 Acked Read 6 Done Done Sent false
6098 2300 Acked Read 10 Done Done Sent false
6099 2301 Acked Read 9 Done Done Sent false
6100 2302 Acked Read 9 Done Done Sent false
6101 2303 Acked Flush 0 Done Done Sent false
6102 2304 Acked Write 8 Done Done Sent false
6103 2305 Acked Flush 0 Done Done Sent false
6104 2306 Acked Write 1 Done Done Sent false
6105 2307 Acked Flush 0 Done Done Sent false
6106 2308 Acked Write 1 Done Done Sent false
6107 2309 Acked Read 2 Done Done Sent false
6108 2310 Acked Read 9 Done Done Sent false
6109 2311 Acked Write 9 Done Done Sent false
6110 2312 Acked Read 9 Done Done Sent false
6111 2313 Acked Write 5 Done Done Sent false
6112 2314 Acked Read 8 Done Done Sent false
6113 2315 Acked Read 10 Done Done Sent false
6114 2316 Acked Write 10 Done Done Sent false
6115 2317 Acked Flush 0 Done Done Sent false
6116 2318 Acked Flush 0 Done Done Sent false
6117 2319 Acked Write 8 Done Done Sent false
6118 2320 Acked Read 2 Done Done Sent false
6119 2321 Acked Write 1 Done Done Sent false
6120 2322 Acked Write 10 Done Done Sent false
6121 2323 Acked Write 5 Done Done Sent false
6122 2324 Acked Write 6 Done Done Sent false
6123 2325 Acked Read 1 Done Done Sent false
6124 2326 Acked Read 2 Done Done Sent false
6125 2327 Acked Read 3 Done Done Sent false
6126 2328 Acked Write 6 Done Done Sent false
6127 2329 Acked Flush 0 Done Done Sent false
6128 2330 Acked Write 7 Done Done Sent false
6129 2331 Acked Read 10 Done Done Sent false
6130 2332 Acked Read 3 Done Done Sent false
6131 2333 Acked Write 4 Done Done Sent false
6132 2334 Acked Read 10 Done Done Sent false
6133 2335 Acked Read 10 Done Done Sent false
6134 2336 Acked Flush 0 Done Done Sent false
6135 2337 Acked Flush 0 Done Done Sent false
6136 2338 Acked Write 7 Done Done Sent false
6137 2339 Acked Read 5 Done Done Sent false
6138 2340 Acked Write 1 Done Done Sent false
6139 2341 Acked Read 6 Done Done Sent false
6140 2342 Acked Read 4 Done Done Sent false
6141 2343 Acked Read 1 Done Done Sent false
6142 2344 Acked Read 1 Done Done Sent false
6143 2345 Acked Write 6 Done Done Sent false
6144 2346 Acked Read 7 Done Done Sent false
6145 2347 Acked Read 4 Done Done Sent false
6146 2348 Acked Write 2 Done Done Sent false
6147 2349 Acked Read 6 Done Done Sent false
6148 2350 Acked Read 1 Done Done Sent false
6149 2351 Acked Read 2 Done Done Sent false
6150 2352 Acked Flush 0 Done Done Sent false
6151 2353 Acked Write 4 Done Done Sent false
6152 2354 Acked Read 7 Done Done Sent false
6153 2355 Acked Write 5 Done Done Sent false
6154 2356 Acked Read 9 Done Done Sent false
6155 2357 Acked Read 8 Done Done Sent false
6156 2358 Acked Flush 0 Done Done Sent false
6157 2359 Acked Read 3 Done Done Sent false
6158 2360 Acked Read 7 Done Done Sent false
6159 2361 Acked Write 2 Done Done Sent false
6160 2362 Acked Write 4 Done Done Sent false
6161 2363 Acked Read 5 Done Done Sent false
6162 2364 Acked Write 9 Done Done Sent false
6163 2365 Acked Write 8 Done Done Sent false
6164 2366 Acked Read 5 Done Done Sent false
6165 2367 Acked Flush 0 Done Done Sent false
6166 2368 Acked Write 5 Done Done Sent false
6167 2369 Acked Write 2 Done Done Sent false
6168 2370 Acked Read 10 Done Done Sent false
6169 2371 Acked Write 10 Done Done Sent false
6170 2372 Acked Write 3 Done Done Sent false
6171 2373 Acked Read 6 Done Done Sent false
6172 2374 Acked Read 9 Done Done Sent false
6173 2375 Acked Write 3 Done Done Sent false
6174 2376 Acked Write 10 Done Done Sent false
6175 2377 Acked Flush 0 Done Done Sent false
6176 2378 Acked Write 8 Done Done Sent false
6177 2379 Acked Read 3 Done Done Sent false
6178 2380 Acked Write 4 Done Done Sent false
6179 2381 Acked Write 2 Done Done Sent false
6180 2382 Acked Write 6 Done Done Sent false
6181 2383 Acked Write 4 Done Done Sent false
6182 2384 Acked Read 8 Done Done Sent false
6183 2385 Acked Read 3 Done Done Sent false
6184 2386 Acked Write 7 Done Done Sent false
6185 2387 Acked Read 6 Done Done Sent false
6186 2388 Acked Write 2 Done Done Sent false
6187 2389 Acked Write 8 Done Done Sent false
6188 2390 Acked Write 9 Done Done Sent false
6189 2391 Acked Write 9 Done Done Sent false
6190 2392 Acked Write 9 Done Done Sent false
6191 2393 Acked Flush 0 Done Done Sent false
6192 2394 Acked Write 8 Done Done Sent false
6193 2395 Acked Read 10 Done Done Sent false
6194 2396 Acked Read 3 Done Done Sent false
6195 2397 Acked Flush 0 Done Done Sent false
6196 2398 Acked Read 4 Done Done Sent false
6197 2399 Acked Read 3 Done Done Sent false
6198 2400 Acked Read 4 Done Done Sent false
6199 2401 Acked Read 4 Done Done Sent false
6200 2402 Acked Write 6 Done Done Sent false
6201 2403 Acked Flush 0 Done Done Sent false
6202 2404 Acked Write 4 Done Done Sent false
6203 2405 Acked Read 9 Done Done Sent false
6204 2406 Acked Read 8 Done Done Sent false
6205 2407 Acked Read 2 Done Done Sent false
6206 2408 Acked Read 5 Done Done Sent false
6207 2409 Acked Read 8 Done Done Sent false
6208 2410 Acked Write 8 Done Done Sent false
6209 2411 Acked Read 8 Done Done Sent false
6210 2412 Acked Read 9 Done Done Sent false
6211 2413 Acked Read 5 Done Done Sent false
6212 2414 Acked Read 4 Done Done Sent false
6213 2415 Acked Write 6 Done Done Sent false
6214 2416 Acked Write 5 Done Done Sent false
6215 2417 Acked Write 3 Done Done Sent false
6216 2418 Acked Read 3 Done Done Sent false
6217 2419 Acked Read 6 Done Done Sent false
6218 2420 Acked Write 10 Done Done Sent false
6219 2421 Acked Read 2 Done Done Sent false
6220 2422 Acked Read 8 Done Done Sent false
6221 2423 Acked Write 10 Done Done Sent false
6222 2424 Acked Read 1 Done Done Sent false
6223 2425 Acked Read 2 Done Done Sent false
6224 2426 Acked Write 4 Done Done Sent false
6225 2427 Acked Write 6 Done Done Sent false
6226 2428 Acked Read 7 Done Done Sent false
6227 2429 Acked Read 6 Done Done Sent false
6228 2430 Acked Write 8 Done Done Sent false
6229 2431 Acked Write 3 Done Done Sent false
6230 2432 Acked Write 6 Done Done Sent false
6231 2433 Acked Read 4 Done Done Sent false
6232 2434 Acked Read 9 Done Done Sent false
6233 2435 Acked Read 10 Done Done Sent false
6234 2436 Acked Read 6 Done Done Sent false
6235 2437 Acked Write 9 Done Done Sent false
6236 2438 Acked Read 9 Done Done Sent false
6237 2439 Acked Write 4 Done Done Sent false
6238 2440 Acked Read 9 Done Done Sent false
6239 2441 Acked Write 3 Done Done Sent false
6240 2442 Acked Read 9 Done Done Sent false
6241 2443 Acked Read 8 Done Done Sent false
6242 2444 Acked Read 8 Done Done Sent false
6243 2445 Acked Read 5 Done Done Sent false
6244 2446 Acked Read 1 Done Done Sent false
6245 2447 Acked Read 4 Done Done Sent false
6246 2448 Acked Read 9 Done Done Sent false
6247 2449 Acked Write 7 Done Done Sent false
6248 2450 Acked Read 1 Done Done Sent false
6249 2451 Acked Read 10 Done Done Sent false
6250 2452 Acked Write 6 Done Done Sent false
6251 2453 Acked Read 8 Done Done Sent false
6252 2454 Acked Read 4 Done Done Sent false
6253 2455 Acked Flush 0 Done Done Sent false
6254 2456 Acked Read 4 Done Done Sent false
6255 2457 Acked Write 7 Done Done Sent false
6256 2458 Acked Flush 0 Done Done Sent false
6257 2459 Acked Flush 0 Done Done Sent false
6258 2460 Acked Flush 0 Done Done Sent false
6259 2461 Acked Read 6 Done Done Sent false
6260 2462 Acked Flush 0 Done Done Sent false
6261 2463 Acked Flush 0 Done Done Sent false
6262 2464 Acked Read 6 Done Done Sent false
6263 2465 Acked Flush 0 Done Done Sent false
6264 2466 Acked Write 2 Done Done Sent false
6265 2467 Acked Write 7 Done Done Sent false
6266 2468 Acked Write 2 Done Done Sent false
6267 2469 Acked Write 7 Done Done Sent false
6268 2470 Acked Write 5 Done Done Sent false
6269 2471 Acked Read 5 Done Done Sent false
6270 2472 Acked Read 3 Done Done Sent false
6271 2473 Acked Read 9 Done Done Sent false
6272 2474 Acked Read 8 Done Done Sent false
6273 2475 Acked Read 8 Done Done Sent false
6274 2476 Acked Write 4 Done Done Sent false
6275 2477 Acked Write 7 Done Done Sent false
6276 2478 Acked Write 5 Done Done Sent false
6277 2479 Acked Write 5 Done Done Sent false
6278 2480 Acked Write 9 Done Done Sent false
6279 2481 Acked Read 1 Done Done Sent false
6280 2482 Acked Write 9 Done Done Sent false
6281 2483 Acked Write 5 Done Done Sent false
6282 2484 Acked Write 7 Done Done Sent false
6283 2485 Acked Write 9 Done Done Sent false
6284 2486 Acked Write 5 Done Done Sent false
6285 2487 Acked Flush 0 Done Done Sent false
6286 2488 Acked Write 3 Done Done Sent false
6287 2489 Acked Read 3 Done Done Sent false
6288 2490 Acked Read 1 Done Done Sent false
6289 2491 Acked Write 6 Done Done Sent false
6290 2492 Acked Write 1 Done Done Sent false
6291 2493 Acked Write 8 Done Done Sent false
6292 2494 Acked Read 5 Done Done Sent false
6293 2495 Acked Write 8 Done Done Sent false
6294 2496 Acked Write 1 Done Done Sent false
6295 2497 Acked Read 2 Done Done Sent false
6296 2498 Acked Read 8 Done Done Sent false
6297 2499 Acked Write 1 Done Done Sent false
6298 2500 Acked Read 6 Done Done Sent false
6299 2501 Acked Flush 0 Done Done Sent false
6300 2502 Acked Read 2 Done Done Sent false
6301 2503 Acked Read 5 Done Done Sent false
6302 2504 Acked Read 4 Done Done Sent false
6303 2505 Acked Read 1 Done Done Sent false
6304 2506 Acked Read 6 Done Done Sent false
6305 2507 Acked Read 10 Done Done Sent false
6306 2508 Acked Write 3 Done Done Sent false
6307 2509 Acked Flush 0 Done Done Sent false
6308 STATES DS:0 DS:1 DS:2 TOTAL
6309 Sent 0 0 302 302
6310 Done 302 302 0 604
6311 Skipped 0 0 0 0
6312 Error 0 0 0 0
6313 Last Flush: Some(JobId(2509)) Some(JobId(2509)) Some(JobId(2207))
6314 Downstairs last five completed: 2207 2206 2205 2204 2203
6315 Upstairs last five completed: 2509 2508 2507 2506 2505
6316 CLIENT: Up:0 ds:302 act:2
63172025-01-23T18:49:11.957ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
63182025-01-23T18:49:11.957ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53561) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
63192025-01-23T18:49:11.958ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
63202025-01-23T18:49:11.959ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
63212025-01-23T18:49:11.959ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
63222025-01-23T18:49:11.959ZINFOcrucible: send last flush ID to this DS: Some(JobId(2207)) = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
63232025-01-23T18:49:11.959ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
63242025-01-23T18:49:11.959ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(2207)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
63252025-01-23T18:49:11.959ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(2207)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
6326 ----------------------------------------------------------------
6327 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6328 Downstairs last five completed: 2509 2508 2507 2506 2505
6329 Upstairs last five completed: 2509 2508 2507 2506 2505
6330 CLIENT: Up:0 ds:0 act:3
6331 Replay: All jobs finished
6332 Main task has work to do, go find it
6333 got dsc Stop(1)
6334 Jan 23 18:49:14.194 INFO stop 1
6335 request completed, latency_us: 357, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 12b599eb-a6ec-4ee7-b664-1d1feafc4a50, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
6336 [8820] Got stop action so:false kr:false
6337 Jan 23 18:49:14.195 INFO request completed, latency_us: 107, response_code: 200, uri: /state/cid/1, method: GET, req_id: 26af301a-a106-4246-b41a-9f48be4b35f5, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
63382025-01-23T18:49:14.195ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
63392025-01-23T18:49:14.195ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
63402025-01-23T18:49:14.195ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
63412025-01-23T18:49:14.195ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
6342 [8820] Exited with: ExitStatus(unix_wait_status(9))
6343 [8820][1] reports Exit
6344 Jan 23 18:49:18.197 INFO request completed, latency_us: 191, response_code: 200, uri: /state/cid/1, method: GET, req_id: d3aea598-5322-4761-96b1-9b9584d633e5, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
6345 001/300 Write block 410 len 24576 data: 4 4 3 3 3 3
6346 002/300 Write block 1993 len 32768 data: 2 2 2 3 3 3 3 3
6347 003/300 Flush
6348 004/300 Write block 3997 len 4096 data: 4
6349 005/300 Read block 2833 len 8192
6350 006/300 Read block 4353 len 8192
6351 007/300 Read block 4274 len 36864
6352 008/300 Read block 885 len 24576
6353 009/300 Read block 917 len 16384
6354 010/300 Read block 1914 len 16384
6355 011/300 Write block 4977 len 32768 data: 3 3 3 3 3 2 2 2
6356 012/300 Read block 279 len 24576
6357 013/300 Read block 1331 len 4096
6358 014/300 Write block 1898 len 40960 data: 3 3 3 3 3 4 4 3 2 2
6359 015/300 Flush
6360 016/300 Read block 3481 len 8192
6361 017/300 Write block 2312 len 24576 data: 3 4 4 5 3 3
6362 018/300 Write block 3534 len 20480 data: 2 2 2 2 2
6363 019/300 Write block 2360 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6364 020/300 Write block 2448 len 36864 data: 3 3 3 3 3 2 2 2 3
6365 021/300 Read block 4641 len 36864
6366 022/300 Write block 4164 len 36864 data: 4 3 3 4 3 3 3 3 3
6367 023/300 Write block 4833 len 16384 data: 3 3 3 3
6368 024/300 Write block 3477 len 24576 data: 2 2 2 2 2 2
6369 025/300 Write block 3359 len 28672 data: 3 3 4 4 4 4 4
6370 026/300 Read block 3152 len 20480
6371 027/300 Write block 2035 len 16384 data: 2 2 2 2
6372 028/300 Flush
6373 029/300 Read block 1293 len 16384
6374 030/300 Write block 1661 len 16384 data: 2 2 3 2
6375 031/300 Read block 3040 len 4096
6376 032/300 Write block 71 len 16384 data: 3 2 3 4
6377 033/300 Write block 4632 len 36864 data: 2 2 2 2 2 2 2 2 2
6378 034/300 Write block 835 len 4096 data: 4
6379 035/300 Flush
6380 036/300 Flush
6381 037/300 Read block 4019 len 40960
6382 038/300 Read block 4933 len 8192
6383 039/300 Read block 4156 len 28672
6384 040/300 Write block 3534 len 32768 data: 3 3 3 3 3 2 2 2
6385 041/300 Read block 1413 len 28672
6386 042/300 Read block 4262 len 16384
6387 043/300 Read block 474 len 20480
6388 044/300 Read block 99 len 12288
6389 045/300 Write block 3436 len 16384 data: 2 2 2 2
6390 046/300 Flush
6391 047/300 Read block 4612 len 32768
6392 048/300 Flush
6393 049/300 Write block 487 len 32768 data: 3 3 3 4 4 4 4 4
6394 050/300 Read block 4836 len 16384
6395 051/300 Write block 2703 len 12288 data: 3 3 3
6396 052/300 Read block 2654 len 4096
6397 053/300 Write block 4341 len 16384 data: 3 4 4 5
6398 054/300 Flush
6399 055/300 Write block 3172 len 20480 data: 2 2 2 2 2
6400 056/300 Write block 1589 len 24576 data: 3 3 3 3 3 3
6401 057/300 Write block 650 len 36864 data: 4 3 3 3 3 3 3 2 2
6402 058/300 Write block 4807 len 40960 data: 3 4 4 4 4 4 3 3 3 2
6403 059/300 Read block 1756 len 36864
6404 060/300 Write block 4693 len 16384 data: 3 3 3 3
6405 061/300 Write block 4582 len 24576 data: 4 3 3 2 2 2
6406 062/300 Read block 1690 len 16384
6407 063/300 Write block 2314 len 8192 data: 5 6
6408 064/300 Read block 2127 len 4096
6409 065/300 Read block 2928 len 28672
6410 066/300 Read block 3673 len 12288
6411 067/300 Read block 4990 len 20480
6412 068/300 Write block 173 len 24576 data: 2 2 2 2 2 4
6413 069/300 Read block 3589 len 32768
6414 070/300 Read block 996 len 32768
6415 071/300 Write block 1476 len 16384 data: 2 2 2 2
6416 072/300 Flush
6417 073/300 Write block 2440 len 20480 data: 4 4 3 3 2
6418 074/300 Write block 3850 len 16384 data: 2 2 2 2
6419 075/300 Read block 1528 len 28672
6420 076/300 Read block 2849 len 28672
6421 077/300 Write block 4398 len 4096 data: 2
6422 078/300 Read block 3109 len 16384
6423 079/300 Read block 3295 len 28672
6424 080/300 Write block 2788 len 28672 data: 3 3 3 3 4 4 4
6425 081/300 Read block 1312 len 28672
6426 082/300 Write block 2984 len 12288 data: 2 2 2
6427 083/300 Write block 2921 len 24576 data: 2 2 2 2 2 3
6428 084/300 Write block 2585 len 32768 data: 4 4 3 4 4 4 4 3
6429 085/300 Write block 1418 len 40960 data: 3 3 3 3 3 3 3 3 3 3
6430 086/300 Flush
6431 087/300 Write block 896 len 36864 data: 3 3 3 4 4 4 3 3 3
6432 088/300 Write block 3255 len 32768 data: 4 4 4 3 3 3 3 2
6433 089/300 Read block 4798 len 28672
6434 090/300 Write block 3013 len 24576 data: 2 3 3 3 2 2
6435 091/300 Read block 33 len 36864
6436 092/300 Write block 715 len 24576 data: 3 3 3 3 2 2
6437 093/300 Write block 1461 len 24576 data: 2 2 2 2 3 3
6438 094/300 Read block 1928 len 24576
6439 095/300 Write block 3463 len 16384 data: 2 3 3 3
6440 096/300 Flush
6441 097/300 Write block 1839 len 4096 data: 2
6442 098/300 Read block 678 len 16384
6443 099/300 Read block 2261 len 4096
6444 100/300 Flush
6445 101/300 Flush
6446 102/300 Write block 4168 len 32768 data: 4 4 4 4 4 3 3 3
6447 103/300 Write block 2759 len 20480 data: 3 3 3 3 2
6448 104/300 Read block 3387 len 8192
6449 105/300 Read block 2421 len 24576
6450 106/300 Flush
6451 107/300 Read block 2465 len 20480
6452 108/300 Read block 2388 len 16384
6453 109/300 Flush
6454 110/300 Read block 1560 len 24576
6455 111/300 Write block 902 len 40960 data: 4 4 4 3 3 3 2 2 2 3
6456 112/300 Read block 3069 len 36864
6457 113/300 Read block 4916 len 40960
6458 114/300 Read block 4632 len 12288
6459 115/300 Write block 975 len 4096 data: 2
6460 116/300 Read block 2743 len 36864
6461 117/300 Read block 2438 len 4096
6462 118/300 Read block 3588 len 24576
6463 119/300 Read block 330 len 24576
6464 120/300 Read block 218 len 12288
6465 121/300 Read block 2640 len 16384
6466 122/300 Write block 3823 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6467 123/300 Read block 4404 len 4096
6468 124/300 Write block 2497 len 24576 data: 3 4 4 4 4 2
6469 125/300 Read block 2895 len 40960
6470 126/300 Read block 1805 len 24576
6471 127/300 Write block 332 len 36864 data: 2 2 2 2 2 2 2 2 3
6472 128/300 Write block 1678 len 32768 data: 5 5 3 3 2 2 2 2
6473 129/300 Read block 277 len 32768
6474 130/300 Read block 504 len 8192
6475 131/300 Read block 4002 len 8192
6476 132/300 Write block 4506 len 40960 data: 3 3 2 2 2 2 2 2 2 2
6477 133/300 Read block 14 len 24576
6478 134/300 Flush
6479 135/300 Read block 1607 len 24576
6480 136/300 Write block 2534 len 28672 data: 3 3 3 3 3 3 4
6481 137/300 Write block 2655 len 20480 data: 3 3 3 3 4
6482 138/300 Write block 762 len 32768 data: 5 5 5 2 2 2 2 2
6483 139/300 Read block 3065 len 12288
6484 140/300 Read block 3089 len 16384
6485 141/300 Read block 811 len 8192
6486 142/300 Write block 2555 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6487 143/300 Read block 1259 len 8192
6488 144/300 Read block 37 len 32768
6489 145/300 Read block 2262 len 8192
6490 146/300 Write block 3495 len 20480 data: 2 2 2 2 3
6491 147/300 Read block 503 len 8192
6492 148/300 Read block 3583 len 16384
6493 149/300 Read block 2726 len 16384
6494 150/300 Flush
6495 151/300 Write block 3787 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6496 152/300 Write block 2962 len 36864 data: 3 3 3 3 2 2 2 2 2
6497 153/300 Read block 287 len 20480
6498 154/300 Write block 4714 len 8192 data: 3 3
6499 155/300 Read block 555 len 24576
6500 156/300 Write block 2456 len 28672 data: 4 3 3 3 3 3 2
6501 157/300 Write block 505 len 28672 data: 2 2 2 2 2 2 2
6502 158/300 Read block 1517 len 8192
6503 159/300 Write block 4446 len 24576 data: 3 2 3 2 2 2
6504 160/300 Write block 4479 len 36864 data: 2 2 2 2 2 3 4 3 3
6505 161/300 Read block 3723 len 32768
6506 162/300 Read block 2507 len 16384
6507 163/300 Write block 1067 len 40960 data: 3 3 3 3 2 2 2 2 2 3
6508 164/300 Write block 4806 len 20480 data: 3 4 5 5 5
6509 165/300 Write block 3100 len 32768 data: 2 2 2 3 3 3 3 3
6510 166/300 Read block 3670 len 24576
6511 167/300 Write block 2998 len 32768 data: 3 4 4 6 6 5 5 4
6512 168/300 Write block 2415 len 20480 data: 2 2 2 2 2
6513 169/300 Flush
6514 170/300 Write block 1847 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6515 171/300 Read block 1946 len 24576
6516 172/300 Write block 4020 len 20480 data: 2 2 2 2 2
6517 173/300 Write block 483 len 24576 data: 3 3 3 3 4 4
6518 174/300 Flush
6519 175/300 Read block 3729 len 24576
6520 176/300 Write block 2203 len 4096 data: 2
6521 177/300 Write block 3657 len 24576 data: 2 2 2 2 2 2
6522 178/300 Write block 2704 len 32768 data: 4 4 3 2 2 2 2 2
6523 179/300 Write block 1497 len 20480 data: 3 2 2 2 2
6524 180/300 Read block 1807 len 24576
6525 181/300 Flush
6526 182/300 Read block 945 len 4096
6527 183/300 Read block 2653 len 24576
6528 184/300 Read block 652 len 28672
6529 185/300 Write block 2625 len 8192 data: 2 2
6530 186/300 Flush
6531 187/300 Write block 1570 len 12288 data: 3 2 2
6532 188/300 Read block 2008 len 40960
6533 189/300 Write block 277 len 4096 data: 5
6534 190/300 Read block 3075 len 20480
6535 191/300 Read block 1325 len 20480
6536 192/300 Write block 3139 len 16384 data: 3 3 2 2
6537 193/300 Read block 2861 len 12288
6538 194/300 Read block 667 len 32768
6539 195/300 Read block 2189 len 12288
6540 196/300 Write block 3263 len 12288 data: 2 2 2
6541 197/300 Read block 3941 len 24576
6542 198/300 Read block 2339 len 36864
6543 199/300 Write block 907 len 36864 data: 4 3 3 3 4 3 3 3 3
6544 200/300 Read block 4117 len 32768
6545 201/300 Write block 2330 len 4096 data: 4
6546 202/300 Read block 2515 len 36864
6547 203/300 Write block 2603 len 24576 data: 5 5 5 5 4 4
6548 204/300 Read block 3997 len 36864
6549 205/300 Read block 1431 len 24576
6550 206/300 Write block 4577 len 4096 data: 4
6551 207/300 Read block 2055 len 32768
6552 208/300 Read block 3393 len 28672
6553 209/300 Read block 1314 len 20480
6554 210/300 Write block 2271 len 28672 data: 4 3 3 3 3 2 2
6555 211/300 Write block 3821 len 36864 data: 3 2 3 3 3 3 3 3 3
6556 212/300 Write block 4603 len 28672 data: 2 2 2 2 2 2 2
6557 213/300 Read block 4072 len 36864
6558 214/300 Write block 3237 len 8192 data: 4 4
6559 215/300 Read block 2319 len 8192
6560 216/300 Read block 3425 len 28672
6561 217/300 Write block 4215 len 32768 data: 2 2 2 2 3 3 3 3
6562 218/300 Write block 2011 len 36864 data: 3 3 3 3 3 3 2 2 2
6563 219/300 Write block 2699 len 32768 data: 3 3 3 3 4 5 5 4
6564 220/300 Read block 2693 len 28672
6565 221/300 Write block 4788 len 4096 data: 4
6566 222/300 Read block 3307 len 36864
6567 223/300 Write block 1281 len 36864 data: 3 2 3 3 3 3 3 3 3
6568 224/300 Flush
6569 225/300 Read block 4305 len 20480
6570 226/300 Read block 899 len 8192
6571 227/300 Write block 2605 len 16384 data: 6 6 5 5
6572 228/300 Read block 169 len 12288
6573 229/300 Flush
6574 230/300 Write block 3542 len 36864 data: 2 2 2 2 2 2 2 2 2
6575 231/300 Flush
6576 232/300 Write block 4078 len 20480 data: 3 3 3 4 4
6577 233/300 Read block 3867 len 36864
6578 234/300 Read block 218 len 16384
6579 235/300 Flush
6580 236/300 Read block 4015 len 36864
6581 237/300 Write block 3554 len 20480 data: 2 3 4 3 3
6582 238/300 Write block 2331 len 24576 data: 4 4 2 3 3 3
6583 239/300 Read block 977 len 28672
6584 240/300 Read block 4064 len 8192
6585 241/300 Write block 3688 len 24576 data: 2 2 2 2 2 3
6586 242/300 Read block 3481 len 40960
6587 243/300 Write block 2673 len 24576 data: 3 3 3 4 4 4
6588 244/300 Write block 4191 len 4096 data: 2
6589 245/300 Read block 2214 len 8192
6590 246/300 Read block 840 len 12288
6591 247/300 Read block 3630 len 40960
6592 248/300 Write block 1559 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6593 249/300 Flush
6594 250/300 Write block 3896 len 12288 data: 2 2 2
6595 251/300 Write block 2964 len 32768 data: 4 4 3 3 3 3 3 2
6596 252/300 Read block 783 len 24576
6597 253/300 Write block 4823 len 28672 data: 3 3 2 2 2 2 2
6598 254/300 Read block 4037 len 12288
6599 255/300 Read block 2894 len 20480
6600 256/300 Read block 2049 len 36864
6601 257/300 Flush
6602 258/300 Write block 915 len 24576 data: 4 3 3 3 4 3
6603 259/300 Read block 3319 len 12288
6604 260/300 Write block 738 len 20480 data: 2 2 2 3 3
6605 261/300 Write block 777 len 16384 data: 4 4 4 5
6606 262/300 Read block 205 len 36864
6607 263/300 Write block 2377 len 4096 data: 2
6608 264/300 Write block 3573 len 12288 data: 2 3 3
6609 265/300 Flush
6610 266/300 Write block 3994 len 28672 data: 2 3 4 5 4 4 4
6611 267/300 Read block 3939 len 8192
6612 268/300 Read block 2286 len 32768
6613 269/300 Read block 4757 len 32768
6614 270/300 Read block 4554 len 16384
6615 271/300 Write block 1062 len 4096 data: 3
6616 272/300 Read block 4525 len 12288
6617 273/300 Read block 3388 len 32768
6618 274/300 Read block 4987 len 20480
6619 275/300 Flush
6620 276/300 Read block 1245 len 12288
6621 277/300 Read block 4353 len 16384
6622 278/300 Read block 2439 len 16384
6623 279/300 Write block 1364 len 16384 data: 2 2 2 2
6624 280/300 Write block 2607 len 20480 data: 6 6 3 3 3
6625 281/300 Write block 2018 len 32768 data: 3 3 2 2 2 2 2 2
6626 282/300 Write block 3029 len 20480 data: 3 4 4 4 4
6627 283/300 Write block 2559 len 28672 data: 3 3 3 3 3 3 2
6628 284/300 Flush
6629 285/300 Write block 2921 len 16384 data: 3 3 3 3
6630 286/300 Read block 925 len 36864
6631 287/300 Write block 3232 len 40960 data: 3 3 3 4 5 5 5 4 3 3
6632 288/300 Write block 21 len 28672 data: 3 3 2 2 2 2 2
6633 289/300 Read block 1400 len 4096
6634 290/300 Write block 4908 len 40960 data: 2 2 2 2 2 2 2 3 3 3
6635 291/300 Read block 890 len 8192
6636 292/300 Write block 1220 len 28672 data: 3 3 3 3 2 2 2
6637 293/300 Write block 639 len 12288 data: 2 2 2
6638 294/300 Read block 44 len 4096
6639 295/300 Read block 1864 len 4096
6640 296/300 Write block 1016 len 32768 data: 4 4 4 4 4 4 2 2
6641 297/300 Read block 4251 len 28672
6642 298/300 Write block 2427 len 16384 data: 2 2 2 2
6643 299/300 Write block 518 len 4096 data: 2
6644 300/300 Write block 3294 len 24576 data: 4 5 5 5 4 3
6645 301/300 Flush
6646 Main task has work to do, go find it
6647 got dsc Start(1)
6648 start 1
6649 Jan 23 18:49:18.429 INFO request completed, latency_us: 203, response_code: 204, uri: /start/cid/1, method: POST, req_id: 98e067ad-928b-45d1-a456-a34ad3d9695a, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
6650 Replay: started 1, returned:Ok(())
6651 ----------------------------------------------------------------
6652 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6653 States: Active Offline Active
6654 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6655 2510 Acked Write 6 Done Sent Done false
6656 2511 Acked Write 8 Done Sent Done false
6657 2512 Acked Flush 0 Done Sent Done false
6658 2513 Acked Write 1 Done Sent Done false
6659 2514 Acked Read 2 Done Sent Done false
6660 2515 Acked Read 2 Done Sent Done false
6661 2516 Acked Read 9 Done Sent Done false
6662 2517 Acked Read 6 Done Sent Done false
6663 2518 Acked Read 4 Done Sent Done false
6664 2519 Acked Read 4 Done Sent Done false
6665 2520 Acked Write 8 Done Sent Done false
6666 2521 Acked Read 6 Done Sent Done false
6667 2522 Acked Read 1 Done Sent Done false
6668 2523 Acked Write 10 Done Sent Done false
6669 2524 Acked Flush 0 Done Sent Done false
6670 2525 Acked Read 2 Done Sent Done false
6671 2526 Acked Write 6 Done Sent Done false
6672 2527 Acked Write 5 Done Sent Done false
6673 2528 Acked Write 10 Done Sent Done false
6674 2529 Acked Write 9 Done Sent Done false
6675 2530 Acked Read 9 Done Sent Done false
6676 2531 Acked Write 9 Done Sent Done false
6677 2532 Acked Write 4 Done Sent Done false
6678 2533 Acked Write 6 Done Sent Done false
6679 2534 Acked Write 7 Done Sent Done false
6680 2535 Acked Read 5 Done Sent Done false
6681 2536 Acked Write 4 Done Sent Done false
6682 2537 Acked Flush 0 Done Sent Done false
6683 2538 Acked Read 4 Done Sent Done false
6684 2539 Acked Write 4 Done Sent Done false
6685 2540 Acked Read 1 Done Sent Done false
6686 2541 Acked Write 4 Done Sent Done false
6687 2542 Acked Write 9 Done Sent Done false
6688 2543 Acked Write 1 Done Sent Done false
6689 2544 Acked Flush 0 Done Sent Done false
6690 2545 Acked Flush 0 Done Sent Done false
6691 2546 Acked Read 10 Done Sent Done false
6692 2547 Acked Read 2 Done Sent Done false
6693 2548 Acked Read 7 Done Sent Done false
6694 2549 Acked Write 8 Done Sent Done false
6695 2550 Acked Read 7 Done Sent Done false
6696 2551 Acked Read 4 Done Sent Done false
6697 2552 Acked Read 5 Done Sent Done false
6698 2553 Acked Read 3 Done Sent Done false
6699 2554 Acked Write 4 Done Sent Done false
6700 2555 Acked Flush 0 Done Sent Done false
6701 2556 Acked Read 8 Done Sent Done false
6702 2557 Acked Flush 0 Done Sent Done false
6703 2558 Acked Write 8 Done Sent Done false
6704 2559 Acked Read 4 Done Sent Done false
6705 2560 Acked Write 3 Done Sent Done false
6706 2561 Acked Read 1 Done Sent Done false
6707 2562 Acked Write 4 Done Sent Done false
6708 2563 Acked Flush 0 Done Sent Done false
6709 2564 Acked Write 5 Done Sent Done false
6710 2565 Acked Write 6 Done Sent Done false
6711 2566 Acked Write 9 Done Sent Done false
6712 2567 Acked Write 10 Done Sent Done false
6713 2568 Acked Read 9 Done Sent Done false
6714 2569 Acked Write 4 Done Sent Done false
6715 2570 Acked Write 6 Done Sent Done false
6716 2571 Acked Read 4 Done Sent Done false
6717 2572 Acked Write 2 Done Sent Done false
6718 2573 Acked Read 1 Done Sent Done false
6719 2574 Acked Read 7 Done Sent Done false
6720 2575 Acked Read 3 Done Sent Done false
6721 2576 Acked Read 5 Done Sent Done false
6722 2577 Acked Write 6 Done Sent Done false
6723 2578 Acked Read 8 Done Sent Done false
6724 2579 Acked Read 8 Done Sent Done false
6725 2580 Acked Write 4 Done Sent Done false
6726 2581 Acked Flush 0 Done Sent Done false
6727 2582 Acked Write 5 Done Sent Done false
6728 2583 Acked Write 4 Done Sent Done false
6729 2584 Acked Read 7 Done Sent Done false
6730 2585 Acked Read 7 Done Sent Done false
6731 2586 Acked Write 1 Done Sent Done false
6732 2587 Acked Read 4 Done Sent Done false
6733 2588 Acked Read 7 Done Sent Done false
6734 2589 Acked Write 7 Done Sent Done false
6735 2590 Acked Read 7 Done Sent Done false
6736 2591 Acked Write 3 Done Sent Done false
6737 2592 Acked Write 6 Done Sent Done false
6738 2593 Acked Write 8 Done Sent Done false
6739 2594 Acked Write 10 Done Sent Done false
6740 2595 Acked Flush 0 Done Sent Done false
6741 2596 Acked Write 9 Done Sent Done false
6742 2597 Acked Write 8 Done Sent Done false
6743 2598 Acked Read 7 Done Sent Done false
6744 2599 Acked Write 6 Done Sent Done false
6745 2600 Acked Read 9 Done Sent Done false
6746 2601 Acked Write 6 Done Sent Done false
6747 2602 Acked Write 6 Done Sent Done false
6748 2603 Acked Read 6 Done Sent Done false
6749 2604 Acked Write 4 Done Sent Done false
6750 2605 Acked Flush 0 Done Sent Done false
6751 2606 Acked Write 1 Done Sent Done false
6752 2607 Acked Read 4 Done Sent Done false
6753 2608 Acked Read 1 Done Sent Done false
6754 2609 Acked Flush 0 Done Sent Done false
6755 2610 Acked Flush 0 Done Sent Done false
6756 2611 Acked Write 8 Done Sent Done false
6757 2612 Acked Write 5 Done Sent Done false
6758 2613 Acked Read 2 Done Sent Done false
6759 2614 Acked Read 6 Done Sent Done false
6760 2615 Acked Flush 0 Done Sent Done false
6761 2616 Acked Read 5 Done Sent Done false
6762 2617 Acked Read 4 Done Sent Done false
6763 2618 Acked Flush 0 Done Sent Done false
6764 2619 Acked Read 6 Done Sent Done false
6765 2620 Acked Write 10 Done Sent Done false
6766 2621 Acked Read 9 Done Sent Done false
6767 2622 Acked Read 10 Done Sent Done false
6768 2623 Acked Read 3 Done Sent Done false
6769 2624 Acked Write 1 Done Sent Done false
6770 2625 Acked Read 9 Done Sent Done false
6771 2626 Acked Read 1 Done Sent Done false
6772 2627 Acked Read 6 Done Sent Done false
6773 2628 Acked Read 6 Done Sent Done false
6774 2629 Acked Read 3 Done Sent Done false
6775 2630 Acked Read 4 Done Sent Done false
6776 2631 Acked Write 10 Done Sent Done false
6777 2632 Acked Read 1 Done Sent Done false
6778 2633 Acked Write 6 Done Sent Done false
6779 2634 Acked Read 10 Done Sent Done false
6780 2635 Acked Read 6 Done Sent Done false
6781 2636 Acked Write 9 Done Sent Done false
6782 2637 Acked Write 8 Done Sent Done false
6783 2638 Acked Read 8 Done Sent Done false
6784 2639 Acked Read 2 Done Sent Done false
6785 2640 Acked Read 2 Done Sent Done false
6786 2641 Acked Write 10 Done Sent Done false
6787 2642 Acked Read 6 Done Sent Done false
6788 2643 Acked Flush 0 Done Sent Done false
6789 2644 Acked Read 6 Done Sent Done false
6790 2645 Acked Write 7 Done Sent Done false
6791 2646 Acked Write 5 Done Sent Done false
6792 2647 Acked Write 8 Done Sent Done false
6793 2648 Acked Read 3 Done Sent Done false
6794 2649 Acked Read 4 Done Sent Done false
6795 2650 Acked Read 2 Done Sent Done false
6796 2651 Acked Write 10 Done Sent Done false
6797 2652 Acked Read 2 Done Sent Done false
6798 2653 Acked Read 8 Done Sent Done false
6799 2654 Acked Read 2 Done Sent Done false
6800 2655 Acked Write 5 Done Sent Done false
6801 2656 Acked Read 2 Done Sent Done false
6802 2657 Acked Read 4 Done Sent Done false
6803 2658 Acked Read 4 Done Sent Done false
6804 2659 Acked Flush 0 Done Sent Done false
6805 2660 Acked Write 10 Done Sent Done false
6806 2661 Acked Write 9 Done Sent Done false
6807 2662 Acked Read 5 Done Sent Done false
6808 2663 Acked Write 2 Done Sent Done false
6809 2664 Acked Read 6 Done Sent Done false
6810 2665 Acked Write 7 Done Sent Done false
6811 2666 Acked Write 7 Done Sent Done false
6812 2667 Acked Read 2 Done Sent Done false
6813 2668 Acked Write 6 Done Sent Done false
6814 2669 Acked Write 9 Done Sent Done false
6815 2670 Acked Read 8 Done Sent Done false
6816 2671 Acked Read 4 Done Sent Done false
6817 2672 Acked Write 10 Done Sent Done false
6818 2673 Acked Write 5 Done Sent Done false
6819 2674 Acked Write 8 Done Sent Done false
6820 2675 Acked Read 6 Done Sent Done false
6821 2676 Acked Write 8 Done Sent Done false
6822 2677 Acked Write 5 Done Sent Done false
6823 2678 Acked Flush 0 Done Sent Done false
6824 2679 Acked Write 10 Done Sent Done false
6825 2680 Acked Read 6 Done Sent Done false
6826 2681 Acked Write 5 Done Sent Done false
6827 2682 Acked Write 6 Done Sent Done false
6828 2683 Acked Flush 0 Done Sent Done false
6829 2684 Acked Read 6 Done Sent Done false
6830 2685 Acked Write 1 Done Sent Done false
6831 2686 Acked Write 6 Done Sent Done false
6832 2687 Acked Write 8 Done Sent Done false
6833 2688 Acked Write 5 Done Sent Done false
6834 2689 Acked Read 6 Done Sent Done false
6835 2690 Acked Flush 0 Done Sent Done false
6836 2691 Acked Read 1 Done Sent Done false
6837 2692 Acked Read 6 Done Sent Done false
6838 2693 Acked Read 7 Done Sent Done false
6839 2694 Acked Write 2 Done Sent Done false
6840 2695 Acked Flush 0 Done Sent Done false
6841 2696 Acked Write 3 Done Sent Done false
6842 2697 Acked Read 10 Done Sent Done false
6843 2698 Acked Write 1 Done Sent Done false
6844 2699 Acked Read 5 Done Sent Done false
6845 2700 Acked Read 5 Done Sent Done false
6846 2701 Acked Write 4 Done Sent Done false
6847 2702 Acked Read 3 Done Sent Done false
6848 2703 Acked Read 8 Done Sent Done false
6849 2704 Acked Read 3 Done Sent Done false
6850 2705 Acked Write 3 Done Sent Done false
6851 2706 Acked Read 6 Done Sent Done false
6852 2707 Acked Read 9 Done Sent Done false
6853 2708 Acked Write 9 Done Sent Done false
6854 2709 Acked Read 8 Done Sent Done false
6855 2710 Acked Write 1 Done Sent Done false
6856 2711 Acked Read 9 Done Sent Done false
6857 2712 Acked Write 6 Done Sent Done false
6858 2713 Acked Read 9 Done Sent Done false
6859 2714 Acked Read 6 Done Sent Done false
6860 2715 Acked Write 1 Done Sent Done false
6861 2716 Acked Read 8 Done Sent Done false
6862 2717 Acked Read 7 Done Sent Done false
6863 2718 Acked Read 5 Done Sent Done false
6864 2719 Acked Write 7 Done Sent Done false
6865 2720 Acked Write 9 Done Sent Done false
6866 2721 Acked Write 7 Done Sent Done false
6867 2722 Acked Read 9 Done Sent Done false
6868 2723 Acked Write 2 Done Sent Done false
6869 2724 Acked Read 2 Done Sent Done false
6870 2725 Acked Read 7 Done Sent Done false
6871 2726 Acked Write 8 Done Sent Done false
6872 2727 Acked Write 9 Done Sent Done false
6873 2728 Acked Write 8 Done Sent Done false
6874 2729 Acked Read 7 Done Sent Done false
6875 2730 Acked Write 1 Done Sent Done false
6876 2731 Acked Read 9 Done Sent Done false
6877 2732 Acked Write 9 Done Sent Done false
6878 2733 Acked Flush 0 Done Sent Done false
6879 2734 Acked Read 5 Done Sent Done false
6880 2735 Acked Read 2 Done Sent Done false
6881 2736 Acked Write 4 Done Sent Done false
6882 2737 Acked Read 3 Done Sent Done false
6883 2738 Acked Flush 0 Done Sent Done false
6884 2739 Acked Write 9 Done Sent Done false
6885 2740 Acked Flush 0 Done Sent Done false
6886 2741 Acked Write 5 Done Sent Done false
6887 2742 Acked Read 9 Done Sent Done false
6888 2743 Acked Read 4 Done Sent Done false
6889 2744 Acked Flush 0 Done Sent Done false
6890 2745 Acked Read 9 Done Sent Done false
6891 2746 Acked Write 5 Done Sent Done false
6892 2747 Acked Write 6 Done Sent Done false
6893 2748 Acked Read 7 Done Sent Done false
6894 2749 Acked Read 2 Done Sent Done false
6895 2750 Acked Write 6 Done Sent Done false
6896 2751 Acked Read 10 Done Sent Done false
6897 2752 Acked Write 6 Done Sent Done false
6898 2753 Acked Write 1 Done Sent Done false
6899 2754 Acked Read 2 Done Sent Done false
6900 2755 Acked Read 3 Done Sent Done false
6901 2756 Acked Read 10 Done Sent Done false
6902 2757 Acked Write 10 Done Sent Done false
6903 2758 Acked Flush 0 Done Sent Done false
6904 2759 Acked Write 3 Done Sent Done false
6905 2760 Acked Write 8 Done Sent Done false
6906 2761 Acked Read 6 Done Sent Done false
6907 2762 Acked Write 7 Done Sent Done false
6908 2763 Acked Read 3 Done Sent Done false
6909 2764 Acked Read 5 Done Sent Done false
6910 2765 Acked Read 9 Done Sent Done false
6911 2766 Acked Flush 0 Done Sent Done false
6912 2767 Acked Write 6 Done Sent Done false
6913 2768 Acked Read 3 Done Sent Done false
6914 2769 Acked Write 5 Done Sent Done false
6915 2770 Acked Write 4 Done Sent Done false
6916 2771 Acked Read 9 Done Sent Done false
6917 2772 Acked Write 1 Done Sent Done false
6918 2773 Acked Write 3 Done Sent Done false
6919 2774 Acked Flush 0 Done Sent Done false
6920 2775 Acked Write 7 Done Sent Done false
6921 2776 Acked Read 2 Done Sent Done false
6922 2777 Acked Read 8 Done Sent Done false
6923 2778 Acked Read 8 Done Sent Done false
6924 2779 Acked Read 4 Done Sent Done false
6925 2780 Acked Write 1 Done Sent Done false
6926 2781 Acked Read 3 Done Sent Done false
6927 2782 Acked Read 8 Done Sent Done false
6928 2783 Acked Read 5 Done Sent Done false
6929 2784 Acked Flush 0 Done Sent Done false
6930 2785 Acked Read 3 Done Sent Done false
6931 2786 Acked Read 4 Done Sent Done false
6932 2787 Acked Read 4 Done Sent Done false
6933 2788 Acked Write 4 Done Sent Done false
6934 2789 Acked Write 5 Done Sent Done false
6935 2790 Acked Write 8 Done Sent Done false
6936 2791 Acked Write 5 Done Sent Done false
6937 2792 Acked Write 7 Done Sent Done false
6938 2793 Acked Flush 0 Done Sent Done false
6939 2794 Acked Write 4 Done Sent Done false
6940 2795 Acked Read 9 Done Sent Done false
6941 2796 Acked Write 10 Done Sent Done false
6942 2797 Acked Write 7 Done Sent Done false
6943 2798 Acked Read 1 Done Sent Done false
6944 2799 Acked Write 10 Done Sent Done false
6945 2800 Acked Read 2 Done Sent Done false
6946 2801 Acked Write 7 Done Sent Done false
6947 2802 Acked Write 3 Done Sent Done false
6948 2803 Acked Read 1 Done Sent Done false
6949 2804 Acked Read 1 Done Sent Done false
6950 2805 Acked Write 8 Done Sent Done false
6951 2806 Acked Read 7 Done Sent Done false
6952 2807 Acked Write 4 Done Sent Done false
6953 2808 Acked Write 1 Done Sent Done false
6954 2809 Acked Write 6 Done Sent Done false
6955 2810 Acked Flush 0 Done Sent Done false
6956 STATES DS:0 DS:1 DS:2 TOTAL
6957 Sent 0 301 0 301
6958 Done 301 0 301 602
6959 Skipped 0 0 0 0
6960 Error 0 0 0 0
6961 Last Flush: Some(JobId(2810)) Some(JobId(2509)) Some(JobId(2810))
6962 Downstairs last five completed: 2509 2508 2507 2506 2505
6963 Upstairs last five completed: 2810 2809 2808 2807 2806
6964 CLIENT: Up:0 ds:301 act:2
6965 [8820] got start action
6966 [8820] I am going to restart
6967 Starting downstairs at port 8820
6968 Make output file at "/tmp/dsc/downstairs-8820.txt"
6969 [8820][1] reports Starting
6970 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1206)
6971 [8820][1] reports Running
6972 ----------------------------------------------------------------
6973 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6974 States: Active Offline Active
6975 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6976 2510 Acked Write 6 Done Sent Done false
6977 2511 Acked Write 8 Done Sent Done false
6978 2512 Acked Flush 0 Done Sent Done false
6979 2513 Acked Write 1 Done Sent Done false
6980 2514 Acked Read 2 Done Sent Done false
6981 2515 Acked Read 2 Done Sent Done false
6982 2516 Acked Read 9 Done Sent Done false
6983 2517 Acked Read 6 Done Sent Done false
6984 2518 Acked Read 4 Done Sent Done false
6985 2519 Acked Read 4 Done Sent Done false
6986 2520 Acked Write 8 Done Sent Done false
6987 2521 Acked Read 6 Done Sent Done false
6988 2522 Acked Read 1 Done Sent Done false
6989 2523 Acked Write 10 Done Sent Done false
6990 2524 Acked Flush 0 Done Sent Done false
6991 2525 Acked Read 2 Done Sent Done false
6992 2526 Acked Write 6 Done Sent Done false
6993 2527 Acked Write 5 Done Sent Done false
6994 2528 Acked Write 10 Done Sent Done false
6995 2529 Acked Write 9 Done Sent Done false
6996 2530 Acked Read 9 Done Sent Done false
6997 2531 Acked Write 9 Done Sent Done false
6998 2532 Acked Write 4 Done Sent Done false
6999 2533 Acked Write 6 Done Sent Done false
7000 2534 Acked Write 7 Done Sent Done false
7001 2535 Acked Read 5 Done Sent Done false
7002 2536 Acked Write 4 Done Sent Done false
7003 2537 Acked Flush 0 Done Sent Done false
7004 2538 Acked Read 4 Done Sent Done false
7005 2539 Acked Write 4 Done Sent Done false
7006 2540 Acked Read 1 Done Sent Done false
7007 2541 Acked Write 4 Done Sent Done false
7008 2542 Acked Write 9 Done Sent Done false
7009 2543 Acked Write 1 Done Sent Done false
7010 2544 Acked Flush 0 Done Sent Done false
7011 2545 Acked Flush 0 Done Sent Done false
7012 2546 Acked Read 10 Done Sent Done false
7013 2547 Acked Read 2 Done Sent Done false
7014 2548 Acked Read 7 Done Sent Done false
7015 2549 Acked Write 8 Done Sent Done false
7016 2550 Acked Read 7 Done Sent Done false
7017 2551 Acked Read 4 Done Sent Done false
7018 2552 Acked Read 5 Done Sent Done false
7019 2553 Acked Read 3 Done Sent Done false
7020 2554 Acked Write 4 Done Sent Done false
7021 2555 Acked Flush 0 Done Sent Done false
7022 2556 Acked Read 8 Done Sent Done false
7023 2557 Acked Flush 0 Done Sent Done false
7024 2558 Acked Write 8 Done Sent Done false
7025 2559 Acked Read 4 Done Sent Done false
7026 2560 Acked Write 3 Done Sent Done false
7027 2561 Acked Read 1 Done Sent Done false
7028 2562 Acked Write 4 Done Sent Done false
7029 2563 Acked Flush 0 Done Sent Done false
7030 2564 Acked Write 5 Done Sent Done false
7031 2565 Acked Write 6 Done Sent Done false
7032 2566 Acked Write 9 Done Sent Done false
7033 2567 Acked Write 10 Done Sent Done false
7034 2568 Acked Read 9 Done Sent Done false
7035 2569 Acked Write 4 Done Sent Done false
7036 2570 Acked Write 6 Done Sent Done false
7037 2571 Acked Read 4 Done Sent Done false
7038 2572 Acked Write 2 Done Sent Done false
7039 2573 Acked Read 1 Done Sent Done false
7040 2574 Acked Read 7 Done Sent Done false
7041 2575 Acked Read 3 Done Sent Done false
7042 2576 Acked Read 5 Done Sent Done false
7043 2577 Acked Write 6 Done Sent Done false
7044 2578 Acked Read 8 Done Sent Done false
7045 2579 Acked Read 8 Done Sent Done false
7046 2580 Acked Write 4 Done Sent Done false
7047 2581 Acked Flush 0 Done Sent Done false
7048 2582 Acked Write 5 Done Sent Done false
7049 2583 Acked Write 4 Done Sent Done false
7050 2584 Acked Read 7 Done Sent Done false
7051 2585 Acked Read 7 Done Sent Done false
7052 2586 Acked Write 1 Done Sent Done false
7053 2587 Acked Read 4 Done Sent Done false
7054 2588 Acked Read 7 Done Sent Done false
7055 2589 Acked Write 7 Done Sent Done false
7056 2590 Acked Read 7 Done Sent Done false
7057 2591 Acked Write 3 Done Sent Done false
7058 2592 Acked Write 6 Done Sent Done false
7059 2593 Acked Write 8 Done Sent Done false
7060 2594 Acked Write 10 Done Sent Done false
7061 2595 Acked Flush 0 Done Sent Done false
7062 2596 Acked Write 9 Done Sent Done false
7063 2597 Acked Write 8 Done Sent Done false
7064 2598 Acked Read 7 Done Sent Done false
7065 2599 Acked Write 6 Done Sent Done false
7066 2600 Acked Read 9 Done Sent Done false
7067 2601 Acked Write 6 Done Sent Done false
7068 2602 Acked Write 6 Done Sent Done false
7069 2603 Acked Read 6 Done Sent Done false
7070 2604 Acked Write 4 Done Sent Done false
7071 2605 Acked Flush 0 Done Sent Done false
7072 2606 Acked Write 1 Done Sent Done false
7073 2607 Acked Read 4 Done Sent Done false
7074 2608 Acked Read 1 Done Sent Done false
7075 2609 Acked Flush 0 Done Sent Done false
7076 2610 Acked Flush 0 Done Sent Done false
7077 2611 Acked Write 8 Done Sent Done false
7078 2612 Acked Write 5 Done Sent Done false
7079 2613 Acked Read 2 Done Sent Done false
7080 2614 Acked Read 6 Done Sent Done false
7081 2615 Acked Flush 0 Done Sent Done false
7082 2616 Acked Read 5 Done Sent Done false
7083 2617 Acked Read 4 Done Sent Done false
7084 2618 Acked Flush 0 Done Sent Done false
7085 2619 Acked Read 6 Done Sent Done false
7086 2620 Acked Write 10 Done Sent Done false
7087 2621 Acked Read 9 Done Sent Done false
7088 2622 Acked Read 10 Done Sent Done false
7089 2623 Acked Read 3 Done Sent Done false
7090 2624 Acked Write 1 Done Sent Done false
7091 2625 Acked Read 9 Done Sent Done false
7092 2626 Acked Read 1 Done Sent Done false
7093 2627 Acked Read 6 Done Sent Done false
7094 2628 Acked Read 6 Done Sent Done false
7095 2629 Acked Read 3 Done Sent Done false
7096 2630 Acked Read 4 Done Sent Done false
7097 2631 Acked Write 10 Done Sent Done false
7098 2632 Acked Read 1 Done Sent Done false
7099 2633 Acked Write 6 Done Sent Done false
7100 2634 Acked Read 10 Done Sent Done false
7101 2635 Acked Read 6 Done Sent Done false
7102 2636 Acked Write 9 Done Sent Done false
7103 2637 Acked Write 8 Done Sent Done false
7104 2638 Acked Read 8 Done Sent Done false
7105 2639 Acked Read 2 Done Sent Done false
7106 2640 Acked Read 2 Done Sent Done false
7107 2641 Acked Write 10 Done Sent Done false
7108 2642 Acked Read 6 Done Sent Done false
7109 2643 Acked Flush 0 Done Sent Done false
7110 2644 Acked Read 6 Done Sent Done false
7111 2645 Acked Write 7 Done Sent Done false
7112 2646 Acked Write 5 Done Sent Done false
7113 2647 Acked Write 8 Done Sent Done false
7114 2648 Acked Read 3 Done Sent Done false
7115 2649 Acked Read 4 Done Sent Done false
7116 2650 Acked Read 2 Done Sent Done false
7117 2651 Acked Write 10 Done Sent Done false
7118 2652 Acked Read 2 Done Sent Done false
7119 2653 Acked Read 8 Done Sent Done false
7120 2654 Acked Read 2 Done Sent Done false
7121 2655 Acked Write 5 Done Sent Done false
7122 2656 Acked Read 2 Done Sent Done false
7123 2657 Acked Read 4 Done Sent Done false
7124 2658 Acked Read 4 Done Sent Done false
7125 2659 Acked Flush 0 Done Sent Done false
7126 2660 Acked Write 10 Done Sent Done false
7127 2661 Acked Write 9 Done Sent Done false
7128 2662 Acked Read 5 Done Sent Done false
7129 2663 Acked Write 2 Done Sent Done false
7130 2664 Acked Read 6 Done Sent Done false
7131 2665 Acked Write 7 Done Sent Done false
7132 2666 Acked Write 7 Done Sent Done false
7133 2667 Acked Read 2 Done Sent Done false
7134 2668 Acked Write 6 Done Sent Done false
7135 2669 Acked Write 9 Done Sent Done false
7136 2670 Acked Read 8 Done Sent Done false
7137 2671 Acked Read 4 Done Sent Done false
7138 2672 Acked Write 10 Done Sent Done false
7139 2673 Acked Write 5 Done Sent Done false
7140 2674 Acked Write 8 Done Sent Done false
7141 2675 Acked Read 6 Done Sent Done false
7142 2676 Acked Write 8 Done Sent Done false
7143 2677 Acked Write 5 Done Sent Done false
7144 2678 Acked Flush 0 Done Sent Done false
7145 2679 Acked Write 10 Done Sent Done false
7146 2680 Acked Read 6 Done Sent Done false
7147 2681 Acked Write 5 Done Sent Done false
7148 2682 Acked Write 6 Done Sent Done false
7149 2683 Acked Flush 0 Done Sent Done false
7150 2684 Acked Read 6 Done Sent Done false
7151 2685 Acked Write 1 Done Sent Done false
7152 2686 Acked Write 6 Done Sent Done false
7153 2687 Acked Write 8 Done Sent Done false
7154 2688 Acked Write 5 Done Sent Done false
7155 2689 Acked Read 6 Done Sent Done false
7156 2690 Acked Flush 0 Done Sent Done false
7157 2691 Acked Read 1 Done Sent Done false
7158 2692 Acked Read 6 Done Sent Done false
7159 2693 Acked Read 7 Done Sent Done false
7160 2694 Acked Write 2 Done Sent Done false
7161 2695 Acked Flush 0 Done Sent Done false
7162 2696 Acked Write 3 Done Sent Done false
7163 2697 Acked Read 10 Done Sent Done false
7164 2698 Acked Write 1 Done Sent Done false
7165 2699 Acked Read 5 Done Sent Done false
7166 2700 Acked Read 5 Done Sent Done false
7167 2701 Acked Write 4 Done Sent Done false
7168 2702 Acked Read 3 Done Sent Done false
7169 2703 Acked Read 8 Done Sent Done false
7170 2704 Acked Read 3 Done Sent Done false
7171 2705 Acked Write 3 Done Sent Done false
7172 2706 Acked Read 6 Done Sent Done false
7173 2707 Acked Read 9 Done Sent Done false
7174 2708 Acked Write 9 Done Sent Done false
7175 2709 Acked Read 8 Done Sent Done false
7176 2710 Acked Write 1 Done Sent Done false
7177 2711 Acked Read 9 Done Sent Done false
7178 2712 Acked Write 6 Done Sent Done false
7179 2713 Acked Read 9 Done Sent Done false
7180 2714 Acked Read 6 Done Sent Done false
7181 2715 Acked Write 1 Done Sent Done false
7182 2716 Acked Read 8 Done Sent Done false
7183 2717 Acked Read 7 Done Sent Done false
7184 2718 Acked Read 5 Done Sent Done false
7185 2719 Acked Write 7 Done Sent Done false
7186 2720 Acked Write 9 Done Sent Done false
7187 2721 Acked Write 7 Done Sent Done false
7188 2722 Acked Read 9 Done Sent Done false
7189 2723 Acked Write 2 Done Sent Done false
7190 2724 Acked Read 2 Done Sent Done false
7191 2725 Acked Read 7 Done Sent Done false
7192 2726 Acked Write 8 Done Sent Done false
7193 2727 Acked Write 9 Done Sent Done false
7194 2728 Acked Write 8 Done Sent Done false
7195 2729 Acked Read 7 Done Sent Done false
7196 2730 Acked Write 1 Done Sent Done false
7197 2731 Acked Read 9 Done Sent Done false
7198 2732 Acked Write 9 Done Sent Done false
7199 2733 Acked Flush 0 Done Sent Done false
7200 2734 Acked Read 5 Done Sent Done false
7201 2735 Acked Read 2 Done Sent Done false
7202 2736 Acked Write 4 Done Sent Done false
7203 2737 Acked Read 3 Done Sent Done false
7204 2738 Acked Flush 0 Done Sent Done false
7205 2739 Acked Write 9 Done Sent Done false
7206 2740 Acked Flush 0 Done Sent Done false
7207 2741 Acked Write 5 Done Sent Done false
7208 2742 Acked Read 9 Done Sent Done false
7209 2743 Acked Read 4 Done Sent Done false
7210 2744 Acked Flush 0 Done Sent Done false
7211 2745 Acked Read 9 Done Sent Done false
7212 2746 Acked Write 5 Done Sent Done false
7213 2747 Acked Write 6 Done Sent Done false
7214 2748 Acked Read 7 Done Sent Done false
7215 2749 Acked Read 2 Done Sent Done false
7216 2750 Acked Write 6 Done Sent Done false
7217 2751 Acked Read 10 Done Sent Done false
7218 2752 Acked Write 6 Done Sent Done false
7219 2753 Acked Write 1 Done Sent Done false
7220 2754 Acked Read 2 Done Sent Done false
7221 2755 Acked Read 3 Done Sent Done false
7222 2756 Acked Read 10 Done Sent Done false
7223 2757 Acked Write 10 Done Sent Done false
7224 2758 Acked Flush 0 Done Sent Done false
7225 2759 Acked Write 3 Done Sent Done false
7226 2760 Acked Write 8 Done Sent Done false
7227 2761 Acked Read 6 Done Sent Done false
7228 2762 Acked Write 7 Done Sent Done false
7229 2763 Acked Read 3 Done Sent Done false
7230 2764 Acked Read 5 Done Sent Done false
7231 2765 Acked Read 9 Done Sent Done false
7232 2766 Acked Flush 0 Done Sent Done false
7233 2767 Acked Write 6 Done Sent Done false
7234 2768 Acked Read 3 Done Sent Done false
7235 2769 Acked Write 5 Done Sent Done false
7236 2770 Acked Write 4 Done Sent Done false
7237 2771 Acked Read 9 Done Sent Done false
7238 2772 Acked Write 1 Done Sent Done false
7239 2773 Acked Write 3 Done Sent Done false
7240 2774 Acked Flush 0 Done Sent Done false
7241 2775 Acked Write 7 Done Sent Done false
7242 2776 Acked Read 2 Done Sent Done false
7243 2777 Acked Read 8 Done Sent Done false
7244 2778 Acked Read 8 Done Sent Done false
7245 2779 Acked Read 4 Done Sent Done false
7246 2780 Acked Write 1 Done Sent Done false
7247 2781 Acked Read 3 Done Sent Done false
7248 2782 Acked Read 8 Done Sent Done false
7249 2783 Acked Read 5 Done Sent Done false
7250 2784 Acked Flush 0 Done Sent Done false
7251 2785 Acked Read 3 Done Sent Done false
7252 2786 Acked Read 4 Done Sent Done false
7253 2787 Acked Read 4 Done Sent Done false
7254 2788 Acked Write 4 Done Sent Done false
7255 2789 Acked Write 5 Done Sent Done false
7256 2790 Acked Write 8 Done Sent Done false
7257 2791 Acked Write 5 Done Sent Done false
7258 2792 Acked Write 7 Done Sent Done false
7259 2793 Acked Flush 0 Done Sent Done false
7260 2794 Acked Write 4 Done Sent Done false
7261 2795 Acked Read 9 Done Sent Done false
7262 2796 Acked Write 10 Done Sent Done false
7263 2797 Acked Write 7 Done Sent Done false
7264 2798 Acked Read 1 Done Sent Done false
7265 2799 Acked Write 10 Done Sent Done false
7266 2800 Acked Read 2 Done Sent Done false
7267 2801 Acked Write 7 Done Sent Done false
7268 2802 Acked Write 3 Done Sent Done false
7269 2803 Acked Read 1 Done Sent Done false
7270 2804 Acked Read 1 Done Sent Done false
7271 2805 Acked Write 8 Done Sent Done false
7272 2806 Acked Read 7 Done Sent Done false
7273 2807 Acked Write 4 Done Sent Done false
7274 2808 Acked Write 1 Done Sent Done false
7275 2809 Acked Write 6 Done Sent Done false
7276 2810 Acked Flush 0 Done Sent Done false
7277 STATES DS:0 DS:1 DS:2 TOTAL
7278 Sent 0 301 0 301
7279 Done 301 0 301 602
7280 Skipped 0 0 0 0
7281 Error 0 0 0 0
7282 Last Flush: Some(JobId(2810)) Some(JobId(2509)) Some(JobId(2810))
7283 Downstairs last five completed: 2509 2508 2507 2506 2505
7284 Upstairs last five completed: 2810 2809 2808 2807 2806
7285 CLIENT: Up:0 ds:301 act:2
72862025-01-23T18:49:24.196ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
72872025-01-23T18:49:24.197ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58761) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
72882025-01-23T18:49:24.198ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
72892025-01-23T18:49:24.198ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
72902025-01-23T18:49:24.198ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
72912025-01-23T18:49:24.198ZINFOcrucible: send last flush ID to this DS: Some(JobId(2509)) = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
72922025-01-23T18:49:24.198ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
72932025-01-23T18:49:24.198ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2509)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
72942025-01-23T18:49:24.198ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2509)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
7295 ----------------------------------------------------------------
7296 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7297 Downstairs last five completed: 2810 2809 2808 2807 2806
7298 Upstairs last five completed: 2810 2809 2808 2807 2806
7299 CLIENT: Up:0 ds:0 act:3
7300 Replay: All jobs finished
7301 Main task has work to do, go find it
7302 got dsc Stop(2)
7303 stop 2
7304 Jan 23 18:49:26.433 INFO request completed, latency_us: 351, response_code[8830] Got stop action so:false kr:false
7305 : 204, uri: /stop/cid/2, method: POST, req_id: e3a054a1-84be-4454-8eca-de26b9e1832a, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
73062025-01-23T18:49:26.434ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
73072025-01-23T18:49:26.434ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
73082025-01-23T18:49:26.434ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
7309 Jan 23 18:49:26.434 INFO request completed, latency_us: 93, response_code: 200, uri: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T18:49:26.434518314Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
7310 /state/cid/2, method: GET, req_id: 0fc8646e-9dda-4dce-9f85-d96becd52f5b, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
7311 [8830] Exited with: ExitStatus(unix_wait_status(9))
7312 [8830][2] reports Exit
7313 Jan 23 18:49:30.436 INFO request completed, latency_us: 205, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2a706421-7c02-4448-a696-43f759ea049a, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
7314 001/300 Write block 4457 len 24576 data: 2 2 2 3 3 3
7315 002/300 Read block 42 len 40960
7316 003/300 Read block 4655 len 4096
7317 004/300 Read block 2874 len 36864
7318 005/300 Read block 1094 len 12288
7319 006/300 Read block 4612 len 36864
7320 007/300 Write block 547 len 32768 data: 4 4 4 4 4 3 2 2
7321 008/300 Read block 2738 len 12288
7322 009/300 Read block 379 len 28672
7323 010/300 Read block 2650 len 32768
7324 011/300 Read block 3707 len 20480
7325 012/300 Read block 4667 len 28672
7326 013/300 Read block 1549 len 24576
7327 014/300 Write block 2238 len 36864 data: 2 2 2 2 2 2 2 2 2
7328 015/300 Write block 2925 len 40960 data: 3 4 3 3 3 4 3 3 3 3
7329 016/300 Flush
7330 017/300 Flush
7331 018/300 Write block 276 len 4096 data: 4
7332 019/300 Write block 384 len 36864 data: 3 3 2 2 2 2 2 2 2
7333 020/300 Flush
7334 021/300 Read block 950 len 12288
7335 022/300 Write block 1758 len 32768 data: 2 2 2 2 2 2 2 2
7336 023/300 Flush
7337 024/300 Read block 3801 len 4096
7338 025/300 Read block 2238 len 28672
7339 026/300 Read block 3467 len 12288
7340 027/300 Write block 4006 len 24576 data: 2 3 3 3 3 3
7341 028/300 Read block 1179 len 40960
7342 029/300 Write block 4549 len 12288 data: 2 2 2
7343 030/300 Read block 1786 len 4096
7344 031/300 Flush
7345 032/300 Write block 2121 len 16384 data: 4 4 4 4
7346 033/300 Flush
7347 034/300 Read block 1084 len 12288
7348 035/300 Read block 211 len 16384
7349 036/300 Write block 4572 len 16384 data: 4 3 3 4
7350 037/300 Read block 2726 len 24576
7351 038/300 Read block 922 len 16384
7352 039/300 Read block 3138 len 8192
7353 040/300 Read block 4896 len 40960
7354 041/300 Read block 1280 len 8192
7355 042/300 Read block 670 len 24576
7356 043/300 Write block 2716 len 4096 data: 3
7357 044/300 Read block 1150 len 20480
7358 045/300 Read block 4311 len 40960
7359 046/300 Read block 4330 len 12288
7360 047/300 Write block 2146 len 20480 data: 3 3 3 3 3
7361 048/300 Write block 219 len 16384 data: 2 2 2 2
7362 049/300 Read block 649 len 12288
7363 050/300 Write block 4915 len 16384 data: 4 4 4 4
7364 051/300 Flush
7365 052/300 Read block 2675 len 32768
7366 053/300 Flush
7367 054/300 Read block 3578 len 36864
7368 055/300 Read block 2635 len 28672
7369 056/300 Write block 3308 len 16384 data: 3 4 5 4
7370 057/300 Read block 4690 len 28672
7371 058/300 Write block 1108 len 4096 data: 2
7372 059/300 Read block 1254 len 24576
7373 060/300 Write block 1608 len 32768 data: 4 3 2 2 2 3 3 3
7374 061/300 Write block 3349 len 36864 data: 4 3 3 3 3 4 2 2 3
7375 062/300 Read block 1353 len 32768
7376 063/300 Write block 3134 len 20480 data: 2 2 2 3 3
7377 064/300 Read block 730 len 20480
7378 065/300 Read block 1099 len 32768
7379 066/300 Read block 4014 len 36864
7380 067/300 Flush
7381 068/300 Read block 3241 len 4096
7382 069/300 Flush
7383 070/300 Write block 2820 len 20480 data: 2 2 2 2 2
7384 071/300 Write block 2518 len 28672 data: 4 4 4 4 4 5 4
7385 072/300 Read block 4279 len 32768
7386 073/300 Read block 1341 len 28672
7387 074/300 Write block 3970 len 20480 data: 2 2 2 2 2
7388 075/300 Write block 4424 len 32768 data: 2 2 2 2 3 3 3 3
7389 076/300 Write block 2319 len 36864 data: 3 3 2 2 2 3 3 3 3
7390 077/300 Read block 1355 len 8192
7391 078/300 Read block 3487 len 40960
7392 079/300 Flush
7393 080/300 Flush
7394 081/300 Read block 1838 len 32768
7395 082/300 Write block 3644 len 4096 data: 4
7396 083/300 Write block 4489 len 20480 data: 3 3 3 3 3
7397 084/300 Flush
7398 085/300 Write block 2403 len 36864 data: 5 5 5 5 5 4 5 4 3
7399 086/300 Read block 1937 len 36864
7400 087/300 Read block 3850 len 20480
7401 088/300 Read block 2930 len 8192
7402 089/300 Read block 4889 len 12288
7403 090/300 Write block 3716 len 36864 data: 4 4 3 3 3 3 3 3 2
7404 091/300 Read block 674 len 36864
7405 092/300 Read block 2228 len 16384
7406 093/300 Read block 1293 len 24576
7407 094/300 Read block 2656 len 16384
7408 095/300 Flush
7409 096/300 Write block 4348 len 40960 data: 4 4 2 2 2 3 2 2 2 2
7410 097/300 Read block 4756 len 32768
7411 098/300 Read block 2199 len 28672
7412 099/300 Flush
7413 100/300 Read block 1779 len 8192
7414 101/300 Read block 3808 len 12288
7415 102/300 Read block 2174 len 20480
7416 103/300 Read block 238 len 16384
7417 104/300 Flush
7418 105/300 Read block 1049 len 20480
7419 106/300 Read block 1070 len 32768
7420 107/300 Flush
7421 108/300 Read block 2073 len 40960
7422 109/300 Write block 3597 len 24576 data: 3 3 2 2 2 2
7423 110/300 Read block 3525 len 36864
7424 111/300 Read block 3245 len 32768
7425 112/300 Read block 4925 len 40960
7426 113/300 Read block 2344 len 36864
7427 114/300 Write block 428 len 16384 data: 5 5 5 5
7428 115/300 Read block 3382 len 12288
7429 116/300 Write block 3157 len 28672 data: 2 2 2 2 2 3 3
7430 117/300 Write block 4235 len 24576 data: 2 2 3 3 2 2
7431 118/300 Write block 4263 len 12288 data: 2 2 2
7432 119/300 Write block 4933 len 20480 data: 5 5 5 3 2
7433 120/300 Read block 3207 len 16384
7434 121/300 Write block 1846 len 24576 data: 2 3 3 3 3 3
7435 122/300 Write block 1782 len 12288 data: 3 3 2
7436 123/300 Flush
7437 124/300 Flush
7438 125/300 Write block 4143 len 20480 data: 3 3 3 5 5
7439 126/300 Read block 27 len 36864
7440 127/300 Flush
7441 128/300 Write block 657 len 4096 data: 3
7442 129/300 Flush
7443 130/300 Read block 3141 len 4096
7444 131/300 Read block 284 len 32768
7445 132/300 Read block 1882 len 36864
7446 133/300 Read block 4470 len 24576
7447 134/300 Read block 2767 len 28672
7448 135/300 Read block 215 len 20480
7449 136/300 Write block 1503 len 36864 data: 2 2 2 2 2 2 2 2 2
7450 137/300 Read block 4908 len 16384
7451 138/300 Flush
7452 139/300 Write block 4969 len 16384 data: 3 3 2 2
7453 140/300 Read block 4122 len 32768
7454 141/300 Read block 548 len 36864
7455 142/300 Read block 1025 len 28672
7456 143/300 Write block 251 len 16384 data: 3 4 3 4
7457 144/300 Read block 1142 len 16384
7458 145/300 Write block 2323 len 32768 data: 3 4 4 4 4 3 3 5
7459 146/300 Read block 3214 len 36864
7460 147/300 Read block 775 len 4096
7461 148/300 Read block 1371 len 32768
7462 149/300 Read block 888 len 8192
7463 150/300 Read block 4172 len 16384
7464 151/300 Read block 4355 len 32768
7465 152/300 Read block 2854 len 28672
7466 153/300 Write block 3038 len 36864 data: 2 3 4 3 3 3 3 5 5
7467 154/300 Write block 4151 len 12288 data: 4 4 4
7468 155/300 Read block 4026 len 28672
7469 156/300 Write block 4988 len 20480 data: 3 2 2 2 2
7470 157/300 Flush
7471 158/300 Write block 1936 len 28672 data: 3 2 2 3 3 4 4
7472 159/300 Read block 1990 len 28672
7473 160/300 Write block 2539 len 28672 data: 4 5 3 3 3 3 3
7474 161/300 Read block 2111 len 24576
7475 162/300 Write block 456 len 36864 data: 4 4 4 4 3 4 5 5 3
7476 163/300 Read block 1699 len 4096
7477 164/300 Read block 1736 len 40960
7478 165/300 Read block 3048 len 12288
7479 166/300 Read block 4439 len 32768
7480 167/300 Read block 1111 len 4096
7481 168/300 Write block 3286 len 20480 data: 2 3 3 3 5
7482 169/300 Write block 1418 len 24576 data: 4 4 4 4 4 4
7483 170/300 Write block 1552 len 40960 data: 2 2 2 2 2 2 2 3 3 3
7484 171/300 Read block 2813 len 4096
7485 172/300 Read block 2679 len 8192
7486 173/300 Read block 2379 len 12288
7487 174/300 Read block 1043 len 36864
7488 175/300 Read block 1354 len 32768
7489 176/300 Read block 2735 len 36864
7490 177/300 Read block 455 len 32768
7491 178/300 Write block 4702 len 12288 data: 2 2 2
7492 179/300 Write block 481 len 8192 data: 3 2
7493 180/300 Read block 668 len 40960
7494 181/300 Write block 4449 len 16384 data: 3 3 3 2
7495 182/300 Write block 4563 len 24576 data: 2 2 2 2 2 3
7496 183/300 Read block 1055 len 40960
7497 184/300 Read block 2407 len 24576
7498 185/300 Write block 4422 len 16384 data: 2 2 3 3
7499 186/300 Flush
7500 187/300 Read block 106 len 4096
7501 188/300 Read block 1552 len 4096
7502 189/300 Read block 3349 len 16384
7503 190/300 Write block 4778 len 32768 data: 3 3 3 3 3 3 3 3
7504 191/300 Read block 3535 len 12288
7505 192/300 Read block 1786 len 40960
7506 193/300 Write block 598 len 20480 data: 3 3 3 3 6
7507 194/300 Read block 1950 len 12288
7508 195/300 Write block 2174 len 24576 data: 2 2 2 2 2 2
7509 196/300 Write block 3163 len 40960 data: 4 3 3 3 4 3 3 2 2 3
7510 197/300 Flush
7511 198/300 Read block 3240 len 36864
7512 199/300 Flush
7513 200/300 Read block 660 len 28672
7514 201/300 Read block 1008 len 20480
7515 202/300 Write block 106 len 36864 data: 3 3 4 4 2 2 2 3 3
7516 203/300 Write block 3504 len 16384 data: 3 3 3 3
7517 204/300 Read block 3361 len 12288
7518 205/300 Write block 1464 len 16384 data: 3 4 4 3
7519 206/300 Read block 1383 len 16384
7520 207/300 Read block 4485 len 40960
7521 208/300 Write block 2295 len 4096 data: 3
7522 209/300 Write block 1092 len 32768 data: 2 2 2 3 3 3 4 4
7523 210/300 Write block 3545 len 20480 data: 3 3 3 3 3
7524 211/300 Write block 3753 len 36864 data: 2 3 3 3 3 3 3 3 2
7525 212/300 Write block 4153 len 32768 data: 5 3 2 2 2 2 4 4
7526 213/300 Read block 1158 len 8192
7527 214/300 Write block 4175 len 16384 data: 4 2 2 2
7528 215/300 Read block 1612 len 32768
7529 216/300 Write block 2205 len 4096 data: 2
7530 217/300 Write block 665 len 8192 data: 2 3
7531 218/300 Read block 2503 len 20480
7532 219/300 Read block 1285 len 24576
7533 220/300 Flush
7534 221/300 Write block 3309 len 36864 data: 5 6 5 4 4 3 2 2 2
7535 222/300 Write block 2632 len 8192 data: 3 3
7536 223/300 Write block 2104 len 28672 data: 2 2 2 2 2 2 2
7537 224/300 Write block 1469 len 8192 data: 2 2
7538 225/300 Write block 2448 len 12288 data: 4 4 4
7539 226/300 Write block 4072 len 28672 data: 3 3 3 2 3 3 4
7540 227/300 Write block 782 len 32768 data: 5 5 4 3 4 4 4 3
7541 228/300 Read block 3266 len 16384
7542 229/300 Read block 4573 len 32768
7543 230/300 Write block 3830 len 36864 data: 3 3 3 2 2 2 2 2 2
7544 231/300 Write block 457 len 16384 data: 5 5 5 4
7545 232/300 Write block 3193 len 36864 data: 2 2 2 2 3 2 2 2 2
7546 233/300 Read block 1901 len 28672
7547 234/300 Write block 1527 len 32768 data: 4 4 3 3 3 3 3 3
7548 235/300 Write block 2723 len 24576 data: 2 2 2 2 2 2
7549 236/300 Read block 3646 len 40960
7550 237/300 Read block 3859 len 12288
7551 238/300 Write block 2837 len 16384 data: 2 2 2 2
7552 239/300 Read block 3331 len 8192
7553 240/300 Read block 1807 len 20480
7554 241/300 Read block 1935 len 16384
7555 242/300 Read block 4010 len 36864
7556 243/300 Read block 726 len 8192
7557 244/300 Write block 3038 len 24576 data: 3 4 5 4 4 4
7558 245/300 Write block 1095 len 8192 data: 4 4
7559 246/300 Write block 2835 len 4096 data: 3
7560 247/300 Read block 3034 len 32768
7561 248/300 Flush
7562 249/300 Write block 4805 len 4096 data: 2
7563 250/300 Flush
7564 251/300 Read block 2914 len 28672
7565 252/300 Read block 1255 len 24576
7566 253/300 Read block 2926 len 8192
7567 254/300 Read block 1369 len 40960
7568 255/300 Read block 1271 len 32768
7569 256/300 Flush
7570 257/300 Read block 3123 len 36864
7571 258/300 Read block 3411 len 4096
7572 259/300 Read block 2826 len 28672
7573 260/300 Read block 2522 len 28672
7574 261/300 Read block 4674 len 36864
7575 262/300 Write block 1782 len 8192 data: 4 4
7576 263/300 Read block 1135 len 20480
7577 264/300 Read block 2661 len 4096
7578 265/300 Write block 2908 len 4096 data: 2
7579 266/300 Write block 3558 len 16384 data: 4 3 4 2
7580 267/300 Read block 3960 len 40960
7581 268/300 Read block 1324 len 32768
7582 269/300 Write block 2797 len 24576 data: 3 4 3 2 2 3
7583 270/300 Write block 4424 len 40960 data: 4 4 3 3 4 4 4 4 3 3
7584 271/300 Write block 1308 len 32768 data: 2 2 2 2 3 4 4 4
7585 272/300 Flush
7586 273/300 Write block 3680 len 32768 data: 3 3 3 2 2 2 2 2
7587 274/300 Read block 3987 len 20480
7588 275/300 Write block 4065 len 40960 data: 3 4 4 4 3 3 3 4 4 4
7589 276/300 Write block 3948 len 40960 data: 3 3 3 3 2 3 3 3 3 3
7590 277/300 Write block 415 len 20480 data: 4 3 3 2 2
7591 278/300 Read block 4959 len 16384
7592 279/300 Read block 1592 len 4096
7593 280/300 Read block 4008 len 8192
7594 281/300 Read block 4310 len 16384
7595 282/300 Flush
7596 283/300 Write block 94 len 16384 data: 4 4 4 4
7597 284/300 Read block 899 len 32768
7598 285/300 Write block 4267 len 28672 data: 2 2 2 2 2 2 2
7599 286/300 Read block 1902 len 12288
7600 287/300 Write block 4473 len 20480 data: 3 3 3 2 2
7601 288/300 Write block 1782 len 40960 data: 5 5 3 2 2 2 2 2 2 2
7602 289/300 Read block 1669 len 16384
7603 290/300 Write block 2148 len 16384 data: 4 4 4 2
7604 291/300 Read block 3072 len 32768
7605 292/300 Read block 2282 len 12288
7606 293/300 Write block 236 len 32768 data: 2 2 2 2 2 2 2 2
7607 294/300 Write block 2277 len 28672 data: 3 2 2 2 2 2 3
7608 295/300 Read block 64 len 32768
7609 296/300 Read block 2080 len 20480
7610 297/300 Write block 2426 len 40960 data: 2 3 3 3 3 2 2 3 3 3
7611 298/300 Write block 1216 len 28672 data: 3 3 3 3 4 4 4
7612 299/300 Read block 246 len 20480
7613 300/300 Read block 1432 len 8192
7614 301/300 Read block 3103 len 40960
7615 Main task has work to do, go find it
7616 got dsc Start(2)
7617 start 2
7618 Jan 23 18:49:30.685 INFO request completed, latency_us: 202, response_code: 204, uri: /start/cid/2, method: POST, req_id: 6d859d14-4d19-4b36-bd7b-63aec932d405, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
7619 Replay: started 2, returned:Ok(())
7620 ----------------------------------------------------------------
7621 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7622 States: Active Active Offline
7623 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7624 2811 Acked Write 6 Done Done Sent false
7625 2812 Acked Read 10 Done Done Sent false
7626 2813 Acked Read 1 Done Done Sent false
7627 2814 Acked Read 9 Done Done Sent false
7628 2815 Acked Read 3 Done Done Sent false
7629 2816 Acked Read 9 Done Done Sent false
7630 2817 Acked Write 8 Done Done Sent false
7631 2818 Acked Read 3 Done Done Sent false
7632 2819 Acked Read 7 Done Done Sent false
7633 2820 Acked Read 8 Done Done Sent false
7634 2821 Acked Read 5 Done Done Sent false
7635 2822 Acked Read 7 Done Done Sent false
7636 2823 Acked Read 6 Done Done Sent false
7637 2824 Acked Write 9 Done Done Sent false
7638 2825 Acked Write 10 Done Done Sent false
7639 2826 Acked Flush 0 Done Done Sent false
7640 2827 Acked Flush 0 Done Done Sent false
7641 2828 Acked Write 1 Done Done Sent false
7642 2829 Acked Write 9 Done Done Sent false
7643 2830 Acked Flush 0 Done Done Sent false
7644 2831 Acked Read 3 Done Done Sent false
7645 2832 Acked Write 8 Done Done Sent false
7646 2833 Acked Flush 0 Done Done Sent false
7647 2834 Acked Read 1 Done Done Sent false
7648 2835 Acked Read 7 Done Done Sent false
7649 2836 Acked Read 3 Done Done Sent false
7650 2837 Acked Write 6 Done Done Sent false
7651 2838 Acked Read 10 Done Done Sent false
7652 2839 Acked Write 3 Done Done Sent false
7653 2840 Acked Read 1 Done Done Sent false
7654 2841 Acked Flush 0 Done Done Sent false
7655 2842 Acked Write 4 Done Done Sent false
7656 2843 Acked Flush 0 Done Done Sent false
7657 2844 Acked Read 3 Done Done Sent false
7658 2845 Acked Read 4 Done Done Sent false
7659 2846 Acked Write 4 Done Done Sent false
7660 2847 Acked Read 6 Done Done Sent false
7661 2848 Acked Read 4 Done Done Sent false
7662 2849 Acked Read 2 Done Done Sent false
7663 2850 Acked Read 10 Done Done Sent false
7664 2851 Acked Read 2 Done Done Sent false
7665 2852 Acked Read 6 Done Done Sent false
7666 2853 Acked Write 1 Done Done Sent false
7667 2854 Acked Read 5 Done Done Sent false
7668 2855 Acked Read 10 Done Done Sent false
7669 2856 Acked Read 3 Done Done Sent false
7670 2857 Acked Write 5 Done Done Sent false
7671 2858 Acked Write 4 Done Done Sent false
7672 2859 Acked Read 3 Done Done Sent false
7673 2860 Acked Write 4 Done Done Sent false
7674 2861 Acked Flush 0 Done Done Sent false
7675 2862 Acked Read 8 Done Done Sent false
7676 2863 Acked Flush 0 Done Done Sent false
7677 2864 Acked Read 9 Done Done Sent false
7678 2865 Acked Read 7 Done Done Sent false
7679 2866 Acked Write 4 Done Done Sent false
7680 2867 Acked Read 7 Done Done Sent false
7681 2868 Acked Write 1 Done Done Sent false
7682 2869 Acked Read 6 Done Done Sent false
7683 2870 Acked Write 8 Done Done Sent false
7684 2871 Acked Write 9 Done Done Sent false
7685 2872 Acked Read 8 Done Done Sent false
7686 2873 Acked Write 5 Done Done Sent false
7687 2874 Acked Read 5 Done Done Sent false
7688 2875 Acked Read 8 Done Done Sent false
7689 2876 Acked Read 9 Done Done Sent false
7690 2877 Acked Flush 0 Done Done Sent false
7691 2878 Acked Read 1 Done Done Sent false
7692 2879 Acked Flush 0 Done Done Sent false
7693 2880 Acked Write 5 Done Done Sent false
7694 2881 Acked Write 7 Done Done Sent false
7695 2882 Acked Read 8 Done Done Sent false
7696 2883 Acked Read 7 Done Done Sent false
7697 2884 Acked Write 5 Done Done Sent false
7698 2885 Acked Write 8 Done Done Sent false
7699 2886 Acked Write 9 Done Done Sent false
7700 2887 Acked Read 2 Done Done Sent false
7701 2888 Acked Read 10 Done Done Sent false
7702 2889 Acked Flush 0 Done Done Sent false
7703 2890 Acked Flush 0 Done Done Sent false
7704 2891 Acked Read 8 Done Done Sent false
7705 2892 Acked Write 1 Done Done Sent false
7706 2893 Acked Write 5 Done Done Sent false
7707 2894 Acked Flush 0 Done Done Sent false
7708 2895 Acked Write 9 Done Done Sent false
7709 2896 Acked Read 9 Done Done Sent false
7710 2897 Acked Read 5 Done Done Sent false
7711 2898 Acked Read 2 Done Done Sent false
7712 2899 Acked Read 3 Done Done Sent false
7713 2900 Acked Write 9 Done Done Sent false
7714 2901 Acked Read 9 Done Done Sent false
7715 2902 Acked Read 4 Done Done Sent false
7716 2903 Acked Read 6 Done Done Sent false
7717 2904 Acked Read 4 Done Done Sent false
7718 2905 Acked Flush 0 Done Done Sent false
7719 2906 Acked Write 10 Done Done Sent false
7720 2907 Acked Read 8 Done Done Sent false
7721 2908 Acked Read 7 Done Done Sent false
7722 2909 Acked Flush 0 Done Done Sent false
7723 2910 Acked Read 2 Done Done Sent false
7724 2911 Acked Read 3 Done Done Sent false
7725 2912 Acked Read 5 Done Done Sent false
7726 2913 Acked Read 4 Done Done Sent false
7727 2914 Acked Flush 0 Done Done Sent false
7728 2915 Acked Read 5 Done Done Sent false
7729 2916 Acked Read 8 Done Done Sent false
7730 2917 Acked Flush 0 Done Done Sent false
7731 2918 Acked Read 10 Done Done Sent false
7732 2919 Acked Write 6 Done Done Sent false
7733 2920 Acked Read 9 Done Done Sent false
7734 2921 Acked Read 8 Done Done Sent false
7735 2922 Acked Read 10 Done Done Sent false
7736 2923 Acked Read 9 Done Done Sent false
7737 2924 Acked Write 4 Done Done Sent false
7738 2925 Acked Read 3 Done Done Sent false
7739 2926 Acked Write 7 Done Done Sent false
7740 2927 Acked Write 6 Done Done Sent false
7741 2928 Acked Write 3 Done Done Sent false
7742 2929 Acked Write 5 Done Done Sent false
7743 2930 Acked Read 4 Done Done Sent false
7744 2931 Acked Write 6 Done Done Sent false
7745 2932 Acked Write 3 Done Done Sent false
7746 2933 Acked Flush 0 Done Done Sent false
7747 2934 Acked Flush 0 Done Done Sent false
7748 2935 Acked Write 5 Done Done Sent false
7749 2936 Acked Read 9 Done Done Sent false
7750 2937 Acked Flush 0 Done Done Sent false
7751 2938 Acked Write 1 Done Done Sent false
7752 2939 Acked Flush 0 Done Done Sent false
7753 2940 Acked Read 1 Done Done Sent false
7754 2941 Acked Read 8 Done Done Sent false
7755 2942 Acked Read 9 Done Done Sent false
7756 2943 Acked Read 6 Done Done Sent false
7757 2944 Acked Read 7 Done Done Sent false
7758 2945 Acked Read 5 Done Done Sent false
7759 2946 Acked Write 9 Done Done Sent false
7760 2947 Acked Read 4 Done Done Sent false
7761 2948 Acked Flush 0 Done Done Sent false
7762 2949 Acked Write 4 Done Done Sent false
7763 2950 Acked Read 8 Done Done Sent false
7764 2951 Acked Read 9 Done Done Sent false
7765 2952 Acked Read 7 Done Done Sent false
7766 2953 Acked Write 4 Done Done Sent false
7767 2954 Acked Read 4 Done Done Sent false
7768 2955 Acked Write 8 Done Done Sent false
7769 2956 Acked Read 9 Done Done Sent false
7770 2957 Acked Read 1 Done Done Sent false
7771 2958 Acked Read 8 Done Done Sent false
7772 2959 Acked Read 2 Done Done Sent false
7773 2960 Acked Read 4 Done Done Sent false
7774 2961 Acked Read 8 Done Done Sent false
7775 2962 Acked Read 7 Done Done Sent false
7776 2963 Acked Write 9 Done Done Sent false
7777 2964 Acked Write 3 Done Done Sent false
7778 2965 Acked Read 7 Done Done Sent false
7779 2966 Acked Write 5 Done Done Sent false
7780 2967 Acked Flush 0 Done Done Sent false
7781 2968 Acked Write 7 Done Done Sent false
7782 2969 Acked Read 7 Done Done Sent false
7783 2970 Acked Write 7 Done Done Sent false
7784 2971 Acked Read 6 Done Done Sent false
7785 2972 Acked Write 9 Done Done Sent false
7786 2973 Acked Read 1 Done Done Sent false
7787 2974 Acked Read 10 Done Done Sent false
7788 2975 Acked Read 3 Done Done Sent false
7789 2976 Acked Read 8 Done Done Sent false
7790 2977 Acked Read 1 Done Done Sent false
7791 2978 Acked Write 5 Done Done Sent false
7792 2979 Acked Write 6 Done Done Sent false
7793 2980 Acked Write 10 Done Done Sent false
7794 2981 Acked Read 1 Done Done Sent false
7795 2982 Acked Read 2 Done Done Sent false
7796 2983 Acked Read 3 Done Done Sent false
7797 2984 Acked Read 9 Done Done Sent false
7798 2985 Acked Read 8 Done Done Sent false
7799 2986 Acked Read 9 Done Done Sent false
7800 2987 Acked Read 8 Done Done Sent false
7801 2988 Acked Write 3 Done Done Sent false
7802 2989 Acked Write 2 Done Done Sent false
7803 2990 Acked Read 10 Done Done Sent false
7804 2991 Acked Write 4 Done Done Sent false
7805 2992 Acked Write 6 Done Done Sent false
7806 2993 Acked Read 10 Done Done Sent false
7807 2994 Acked Read 6 Done Done Sent false
7808 2995 Acked Write 4 Done Done Sent false
7809 2996 Acked Flush 0 Done Done Sent false
7810 2997 Acked Read 1 Done Done Sent false
7811 2998 Acked Read 1 Done Done Sent false
7812 2999 Acked Read 4 Done Done Sent false
7813 3000 Acked Write 8 Done Done Sent false
7814 3001 Acked Read 3 Done Done Sent false
7815 3002 Acked Read 10 Done Done Sent false
7816 3003 Acked Write 5 Done Done Sent false
7817 3004 Acked Read 3 Done Done Sent false
7818 3005 Acked Write 6 Done Done Sent false
7819 3006 Acked Write 10 Done Done Sent false
7820 3007 Acked Flush 0 Done Done Sent false
7821 3008 Acked Read 9 Done Done Sent false
7822 3009 Acked Flush 0 Done Done Sent false
7823 3010 Acked Read 7 Done Done Sent false
7824 3011 Acked Read 5 Done Done Sent false
7825 3012 Acked Write 9 Done Done Sent false
7826 3013 Acked Write 4 Done Done Sent false
7827 3014 Acked Read 3 Done Done Sent false
7828 3015 Acked Write 4 Done Done Sent false
7829 3016 Acked Read 4 Done Done Sent false
7830 3017 Acked Read 10 Done Done Sent false
7831 3018 Acked Write 1 Done Done Sent false
7832 3019 Acked Write 8 Done Done Sent false
7833 3020 Acked Write 5 Done Done Sent false
7834 3021 Acked Write 9 Done Done Sent false
7835 3022 Acked Write 8 Done Done Sent false
7836 3023 Acked Read 2 Done Done Sent false
7837 3024 Acked Write 4 Done Done Sent false
7838 3025 Acked Read 8 Done Done Sent false
7839 3026 Acked Write 1 Done Done Sent false
7840 3027 Acked Write 2 Done Done Sent false
7841 3028 Acked Read 5 Done Done Sent false
7842 3029 Acked Read 6 Done Done Sent false
7843 3030 Acked Flush 0 Done Done Sent false
7844 3031 Acked Write 9 Done Done Sent false
7845 3032 Acked Write 2 Done Done Sent false
7846 3033 Acked Write 7 Done Done Sent false
7847 3034 Acked Write 2 Done Done Sent false
7848 3035 Acked Write 3 Done Done Sent false
7849 3036 Acked Write 7 Done Done Sent false
7850 3037 Acked Write 8 Done Done Sent false
7851 3038 Acked Read 4 Done Done Sent false
7852 3039 Acked Read 8 Done Done Sent false
7853 3040 Acked Write 9 Done Done Sent false
7854 3041 Acked Write 4 Done Done Sent false
7855 3042 Acked Write 9 Done Done Sent false
7856 3043 Acked Read 7 Done Done Sent false
7857 3044 Acked Write 8 Done Done Sent false
7858 3045 Acked Write 6 Done Done Sent false
7859 3046 Acked Read 10 Done Done Sent false
7860 3047 Acked Read 3 Done Done Sent false
7861 3048 Acked Write 4 Done Done Sent false
7862 3049 Acked Read 2 Done Done Sent false
7863 3050 Acked Read 5 Done Done Sent false
7864 3051 Acked Read 4 Done Done Sent false
7865 3052 Acked Read 9 Done Done Sent false
7866 3053 Acked Read 2 Done Done Sent false
7867 3054 Acked Write 6 Done Done Sent false
7868 3055 Acked Write 2 Done Done Sent false
7869 3056 Acked Write 1 Done Done Sent false
7870 3057 Acked Read 8 Done Done Sent false
7871 3058 Acked Flush 0 Done Done Sent false
7872 3059 Acked Write 1 Done Done Sent false
7873 3060 Acked Flush 0 Done Done Sent false
7874 3061 Acked Read 7 Done Done Sent false
7875 3062 Acked Read 6 Done Done Sent false
7876 3063 Acked Read 2 Done Done Sent false
7877 3064 Acked Read 10 Done Done Sent false
7878 3065 Acked Read 8 Done Done Sent false
7879 3066 Acked Flush 0 Done Done Sent false
7880 3067 Acked Read 9 Done Done Sent false
7881 3068 Acked Read 1 Done Done Sent false
7882 3069 Acked Read 7 Done Done Sent false
7883 3070 Acked Read 7 Done Done Sent false
7884 3071 Acked Read 9 Done Done Sent false
7885 3072 Acked Write 2 Done Done Sent false
7886 3073 Acked Read 5 Done Done Sent false
7887 3074 Acked Read 1 Done Done Sent false
7888 3075 Acked Write 1 Done Done Sent false
7889 3076 Acked Write 4 Done Done Sent false
7890 3077 Acked Read 10 Done Done Sent false
7891 3078 Acked Read 8 Done Done Sent false
7892 3079 Acked Write 6 Done Done Sent false
7893 3080 Acked Write 10 Done Done Sent false
7894 3081 Acked Write 8 Done Done Sent false
7895 3082 Acked Flush 0 Done Done Sent false
7896 3083 Acked Write 8 Done Done Sent false
7897 3084 Acked Read 5 Done Done Sent false
7898 3085 Acked Write 10 Done Done Sent false
7899 3086 Acked Write 10 Done Done Sent false
7900 3087 Acked Write 5 Done Done Sent false
7901 3088 Acked Read 4 Done Done Sent false
7902 3089 Acked Read 1 Done Done Sent false
7903 3090 Acked Read 2 Done Done Sent false
7904 3091 Acked Read 4 Done Done Sent false
7905 3092 Acked Flush 0 Done Done Sent false
7906 3093 Acked Write 4 Done Done Sent false
7907 3094 Acked Read 8 Done Done Sent false
7908 3095 Acked Write 7 Done Done Sent false
7909 3096 Acked Read 3 Done Done Sent false
7910 3097 Acked Write 5 Done Done Sent false
7911 3098 Acked Write 10 Done Done Sent false
7912 3099 Acked Read 4 Done Done Sent false
7913 3100 Acked Write 4 Done Done Sent false
7914 3101 Acked Read 8 Done Done Sent false
7915 3102 Acked Read 3 Done Done Sent false
7916 3103 Acked Write 8 Done Done Sent false
7917 3104 Acked Write 7 Done Done Sent false
7918 3105 Acked Read 8 Done Done Sent false
7919 3106 Acked Read 5 Done Done Sent false
7920 3107 Acked Write 10 Done Done Sent false
7921 3108 Acked Write 7 Done Done Sent false
7922 3109 Acked Read 5 Done Done Sent false
7923 3110 Acked Read 2 Done Done Sent false
7924 3111 Acked Read 10 Done Done Sent false
7925 STATES DS:0 DS:1 DS:2 TOTAL
7926 Sent 0 0 301 301
7927 Done 301 301 0 602
7928 Skipped 0 0 0 0
7929 Error 0 0 0 0
7930 Last Flush: Some(JobId(3092)) Some(JobId(3092)) Some(JobId(2810))
7931 Downstairs last five completed: 2810 2809 2808 2807 2806
7932 Upstairs last five completed: 3111 3110 3109 3108 3107
7933 CLIENT: Up:0 ds:301 act:2
7934 [8830] got start action
7935 [8830] I am going to restart
7936 Starting downstairs at port 8830
7937 Make output file at "/tmp/dsc/downstairs-8830.txt"
7938 [8830][2] reports Starting
7939 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1207)
7940 [8830][2] reports Running
7941 ----------------------------------------------------------------
7942 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
7943 States: Active Active Offline
7944 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7945 2811 Acked Write 6 Done Done Sent false
7946 2812 Acked Read 10 Done Done Sent false
7947 2813 Acked Read 1 Done Done Sent false
7948 2814 Acked Read 9 Done Done Sent false
7949 2815 Acked Read 3 Done Done Sent false
7950 2816 Acked Read 9 Done Done Sent false
7951 2817 Acked Write 8 Done Done Sent false
7952 2818 Acked Read 3 Done Done Sent false
7953 2819 Acked Read 7 Done Done Sent false
7954 2820 Acked Read 8 Done Done Sent false
7955 2821 Acked Read 5 Done Done Sent false
7956 2822 Acked Read 7 Done Done Sent false
7957 2823 Acked Read 6 Done Done Sent false
7958 2824 Acked Write 9 Done Done Sent false
7959 2825 Acked Write 10 Done Done Sent false
7960 2826 Acked Flush 0 Done Done Sent false
7961 2827 Acked Flush 0 Done Done Sent false
7962 2828 Acked Write 1 Done Done Sent false
7963 2829 Acked Write 9 Done Done Sent false
7964 2830 Acked Flush 0 Done Done Sent false
7965 2831 Acked Read 3 Done Done Sent false
7966 2832 Acked Write 8 Done Done Sent false
7967 2833 Acked Flush 0 Done Done Sent false
7968 2834 Acked Read 1 Done Done Sent false
7969 2835 Acked Read 7 Done Done Sent false
7970 2836 Acked Read 3 Done Done Sent false
7971 2837 Acked Write 6 Done Done Sent false
7972 2838 Acked Read 10 Done Done Sent false
7973 2839 Acked Write 3 Done Done Sent false
7974 2840 Acked Read 1 Done Done Sent false
7975 2841 Acked Flush 0 Done Done Sent false
7976 2842 Acked Write 4 Done Done Sent false
7977 2843 Acked Flush 0 Done Done Sent false
7978 2844 Acked Read 3 Done Done Sent false
7979 2845 Acked Read 4 Done Done Sent false
7980 2846 Acked Write 4 Done Done Sent false
7981 2847 Acked Read 6 Done Done Sent false
7982 2848 Acked Read 4 Done Done Sent false
7983 2849 Acked Read 2 Done Done Sent false
7984 2850 Acked Read 10 Done Done Sent false
7985 2851 Acked Read 2 Done Done Sent false
7986 2852 Acked Read 6 Done Done Sent false
7987 2853 Acked Write 1 Done Done Sent false
7988 2854 Acked Read 5 Done Done Sent false
7989 2855 Acked Read 10 Done Done Sent false
7990 2856 Acked Read 3 Done Done Sent false
7991 2857 Acked Write 5 Done Done Sent false
7992 2858 Acked Write 4 Done Done Sent false
7993 2859 Acked Read 3 Done Done Sent false
7994 2860 Acked Write 4 Done Done Sent false
7995 2861 Acked Flush 0 Done Done Sent false
7996 2862 Acked Read 8 Done Done Sent false
7997 2863 Acked Flush 0 Done Done Sent false
7998 2864 Acked Read 9 Done Done Sent false
7999 2865 Acked Read 7 Done Done Sent false
8000 2866 Acked Write 4 Done Done Sent false
8001 2867 Acked Read 7 Done Done Sent false
8002 2868 Acked Write 1 Done Done Sent false
8003 2869 Acked Read 6 Done Done Sent false
8004 2870 Acked Write 8 Done Done Sent false
8005 2871 Acked Write 9 Done Done Sent false
8006 2872 Acked Read 8 Done Done Sent false
8007 2873 Acked Write 5 Done Done Sent false
8008 2874 Acked Read 5 Done Done Sent false
8009 2875 Acked Read 8 Done Done Sent false
8010 2876 Acked Read 9 Done Done Sent false
8011 2877 Acked Flush 0 Done Done Sent false
8012 2878 Acked Read 1 Done Done Sent false
8013 2879 Acked Flush 0 Done Done Sent false
8014 2880 Acked Write 5 Done Done Sent false
8015 2881 Acked Write 7 Done Done Sent false
8016 2882 Acked Read 8 Done Done Sent false
8017 2883 Acked Read 7 Done Done Sent false
8018 2884 Acked Write 5 Done Done Sent false
8019 2885 Acked Write 8 Done Done Sent false
8020 2886 Acked Write 9 Done Done Sent false
8021 2887 Acked Read 2 Done Done Sent false
8022 2888 Acked Read 10 Done Done Sent false
8023 2889 Acked Flush 0 Done Done Sent false
8024 2890 Acked Flush 0 Done Done Sent false
8025 2891 Acked Read 8 Done Done Sent false
8026 2892 Acked Write 1 Done Done Sent false
8027 2893 Acked Write 5 Done Done Sent false
8028 2894 Acked Flush 0 Done Done Sent false
8029 2895 Acked Write 9 Done Done Sent false
8030 2896 Acked Read 9 Done Done Sent false
8031 2897 Acked Read 5 Done Done Sent false
8032 2898 Acked Read 2 Done Done Sent false
8033 2899 Acked Read 3 Done Done Sent false
8034 2900 Acked Write 9 Done Done Sent false
8035 2901 Acked Read 9 Done Done Sent false
8036 2902 Acked Read 4 Done Done Sent false
8037 2903 Acked Read 6 Done Done Sent false
8038 2904 Acked Read 4 Done Done Sent false
8039 2905 Acked Flush 0 Done Done Sent false
8040 2906 Acked Write 10 Done Done Sent false
8041 2907 Acked Read 8 Done Done Sent false
8042 2908 Acked Read 7 Done Done Sent false
8043 2909 Acked Flush 0 Done Done Sent false
8044 2910 Acked Read 2 Done Done Sent false
8045 2911 Acked Read 3 Done Done Sent false
8046 2912 Acked Read 5 Done Done Sent false
8047 2913 Acked Read 4 Done Done Sent false
8048 2914 Acked Flush 0 Done Done Sent false
8049 2915 Acked Read 5 Done Done Sent false
8050 2916 Acked Read 8 Done Done Sent false
8051 2917 Acked Flush 0 Done Done Sent false
8052 2918 Acked Read 10 Done Done Sent false
8053 2919 Acked Write 6 Done Done Sent false
8054 2920 Acked Read 9 Done Done Sent false
8055 2921 Acked Read 8 Done Done Sent false
8056 2922 Acked Read 10 Done Done Sent false
8057 2923 Acked Read 9 Done Done Sent false
8058 2924 Acked Write 4 Done Done Sent false
8059 2925 Acked Read 3 Done Done Sent false
8060 2926 Acked Write 7 Done Done Sent false
8061 2927 Acked Write 6 Done Done Sent false
8062 2928 Acked Write 3 Done Done Sent false
8063 2929 Acked Write 5 Done Done Sent false
8064 2930 Acked Read 4 Done Done Sent false
8065 2931 Acked Write 6 Done Done Sent false
8066 2932 Acked Write 3 Done Done Sent false
8067 2933 Acked Flush 0 Done Done Sent false
8068 2934 Acked Flush 0 Done Done Sent false
8069 2935 Acked Write 5 Done Done Sent false
8070 2936 Acked Read 9 Done Done Sent false
8071 2937 Acked Flush 0 Done Done Sent false
8072 2938 Acked Write 1 Done Done Sent false
8073 2939 Acked Flush 0 Done Done Sent false
8074 2940 Acked Read 1 Done Done Sent false
8075 2941 Acked Read 8 Done Done Sent false
8076 2942 Acked Read 9 Done Done Sent false
8077 2943 Acked Read 6 Done Done Sent false
8078 2944 Acked Read 7 Done Done Sent false
8079 2945 Acked Read 5 Done Done Sent false
8080 2946 Acked Write 9 Done Done Sent false
8081 2947 Acked Read 4 Done Done Sent false
8082 2948 Acked Flush 0 Done Done Sent false
8083 2949 Acked Write 4 Done Done Sent false
8084 2950 Acked Read 8 Done Done Sent false
8085 2951 Acked Read 9 Done Done Sent false
8086 2952 Acked Read 7 Done Done Sent false
8087 2953 Acked Write 4 Done Done Sent false
8088 2954 Acked Read 4 Done Done Sent false
8089 2955 Acked Write 8 Done Done Sent false
8090 2956 Acked Read 9 Done Done Sent false
8091 2957 Acked Read 1 Done Done Sent false
8092 2958 Acked Read 8 Done Done Sent false
8093 2959 Acked Read 2 Done Done Sent false
8094 2960 Acked Read 4 Done Done Sent false
8095 2961 Acked Read 8 Done Done Sent false
8096 2962 Acked Read 7 Done Done Sent false
8097 2963 Acked Write 9 Done Done Sent false
8098 2964 Acked Write 3 Done Done Sent false
8099 2965 Acked Read 7 Done Done Sent false
8100 2966 Acked Write 5 Done Done Sent false
8101 2967 Acked Flush 0 Done Done Sent false
8102 2968 Acked Write 7 Done Done Sent false
8103 2969 Acked Read 7 Done Done Sent false
8104 2970 Acked Write 7 Done Done Sent false
8105 2971 Acked Read 6 Done Done Sent false
8106 2972 Acked Write 9 Done Done Sent false
8107 2973 Acked Read 1 Done Done Sent false
8108 2974 Acked Read 10 Done Done Sent false
8109 2975 Acked Read 3 Done Done Sent false
8110 2976 Acked Read 8 Done Done Sent false
8111 2977 Acked Read 1 Done Done Sent false
8112 2978 Acked Write 5 Done Done Sent false
8113 2979 Acked Write 6 Done Done Sent false
8114 2980 Acked Write 10 Done Done Sent false
8115 2981 Acked Read 1 Done Done Sent false
8116 2982 Acked Read 2 Done Done Sent false
8117 2983 Acked Read 3 Done Done Sent false
8118 2984 Acked Read 9 Done Done Sent false
8119 2985 Acked Read 8 Done Done Sent false
8120 2986 Acked Read 9 Done Done Sent false
8121 2987 Acked Read 8 Done Done Sent false
8122 2988 Acked Write 3 Done Done Sent false
8123 2989 Acked Write 2 Done Done Sent false
8124 2990 Acked Read 10 Done Done Sent false
8125 2991 Acked Write 4 Done Done Sent false
8126 2992 Acked Write 6 Done Done Sent false
8127 2993 Acked Read 10 Done Done Sent false
8128 2994 Acked Read 6 Done Done Sent false
8129 2995 Acked Write 4 Done Done Sent false
8130 2996 Acked Flush 0 Done Done Sent false
8131 2997 Acked Read 1 Done Done Sent false
8132 2998 Acked Read 1 Done Done Sent false
8133 2999 Acked Read 4 Done Done Sent false
8134 3000 Acked Write 8 Done Done Sent false
8135 3001 Acked Read 3 Done Done Sent false
8136 3002 Acked Read 10 Done Done Sent false
8137 3003 Acked Write 5 Done Done Sent false
8138 3004 Acked Read 3 Done Done Sent false
8139 3005 Acked Write 6 Done Done Sent false
8140 3006 Acked Write 10 Done Done Sent false
8141 3007 Acked Flush 0 Done Done Sent false
8142 3008 Acked Read 9 Done Done Sent false
8143 3009 Acked Flush 0 Done Done Sent false
8144 3010 Acked Read 7 Done Done Sent false
8145 3011 Acked Read 5 Done Done Sent false
8146 3012 Acked Write 9 Done Done Sent false
8147 3013 Acked Write 4 Done Done Sent false
8148 3014 Acked Read 3 Done Done Sent false
8149 3015 Acked Write 4 Done Done Sent false
8150 3016 Acked Read 4 Done Done Sent false
8151 3017 Acked Read 10 Done Done Sent false
8152 3018 Acked Write 1 Done Done Sent false
8153 3019 Acked Write 8 Done Done Sent false
8154 3020 Acked Write 5 Done Done Sent false
8155 3021 Acked Write 9 Done Done Sent false
8156 3022 Acked Write 8 Done Done Sent false
8157 3023 Acked Read 2 Done Done Sent false
8158 3024 Acked Write 4 Done Done Sent false
8159 3025 Acked Read 8 Done Done Sent false
8160 3026 Acked Write 1 Done Done Sent false
8161 3027 Acked Write 2 Done Done Sent false
8162 3028 Acked Read 5 Done Done Sent false
8163 3029 Acked Read 6 Done Done Sent false
8164 3030 Acked Flush 0 Done Done Sent false
8165 3031 Acked Write 9 Done Done Sent false
8166 3032 Acked Write 2 Done Done Sent false
8167 3033 Acked Write 7 Done Done Sent false
8168 3034 Acked Write 2 Done Done Sent false
8169 3035 Acked Write 3 Done Done Sent false
8170 3036 Acked Write 7 Done Done Sent false
8171 3037 Acked Write 8 Done Done Sent false
8172 3038 Acked Read 4 Done Done Sent false
8173 3039 Acked Read 8 Done Done Sent false
8174 3040 Acked Write 9 Done Done Sent false
8175 3041 Acked Write 4 Done Done Sent false
8176 3042 Acked Write 9 Done Done Sent false
8177 3043 Acked Read 7 Done Done Sent false
8178 3044 Acked Write 8 Done Done Sent false
8179 3045 Acked Write 6 Done Done Sent false
8180 3046 Acked Read 10 Done Done Sent false
8181 3047 Acked Read 3 Done Done Sent false
8182 3048 Acked Write 4 Done Done Sent false
8183 3049 Acked Read 2 Done Done Sent false
8184 3050 Acked Read 5 Done Done Sent false
8185 3051 Acked Read 4 Done Done Sent false
8186 3052 Acked Read 9 Done Done Sent false
8187 3053 Acked Read 2 Done Done Sent false
8188 3054 Acked Write 6 Done Done Sent false
8189 3055 Acked Write 2 Done Done Sent false
8190 3056 Acked Write 1 Done Done Sent false
8191 3057 Acked Read 8 Done Done Sent false
8192 3058 Acked Flush 0 Done Done Sent false
8193 3059 Acked Write 1 Done Done Sent false
8194 3060 Acked Flush 0 Done Done Sent false
8195 3061 Acked Read 7 Done Done Sent false
8196 3062 Acked Read 6 Done Done Sent false
8197 3063 Acked Read 2 Done Done Sent false
8198 3064 Acked Read 10 Done Done Sent false
8199 3065 Acked Read 8 Done Done Sent false
8200 3066 Acked Flush 0 Done Done Sent false
8201 3067 Acked Read 9 Done Done Sent false
8202 3068 Acked Read 1 Done Done Sent false
8203 3069 Acked Read 7 Done Done Sent false
8204 3070 Acked Read 7 Done Done Sent false
8205 3071 Acked Read 9 Done Done Sent false
8206 3072 Acked Write 2 Done Done Sent false
8207 3073 Acked Read 5 Done Done Sent false
8208 3074 Acked Read 1 Done Done Sent false
8209 3075 Acked Write 1 Done Done Sent false
8210 3076 Acked Write 4 Done Done Sent false
8211 3077 Acked Read 10 Done Done Sent false
8212 3078 Acked Read 8 Done Done Sent false
8213 3079 Acked Write 6 Done Done Sent false
8214 3080 Acked Write 10 Done Done Sent false
8215 3081 Acked Write 8 Done Done Sent false
8216 3082 Acked Flush 0 Done Done Sent false
8217 3083 Acked Write 8 Done Done Sent false
8218 3084 Acked Read 5 Done Done Sent false
8219 3085 Acked Write 10 Done Done Sent false
8220 3086 Acked Write 10 Done Done Sent false
8221 3087 Acked Write 5 Done Done Sent false
8222 3088 Acked Read 4 Done Done Sent false
8223 3089 Acked Read 1 Done Done Sent false
8224 3090 Acked Read 2 Done Done Sent false
8225 3091 Acked Read 4 Done Done Sent false
8226 3092 Acked Flush 0 Done Done Sent false
8227 3093 Acked Write 4 Done Done Sent false
8228 3094 Acked Read 8 Done Done Sent false
8229 3095 Acked Write 7 Done Done Sent false
8230 3096 Acked Read 3 Done Done Sent false
8231 3097 Acked Write 5 Done Done Sent false
8232 3098 Acked Write 10 Done Done Sent false
8233 3099 Acked Read 4 Done Done Sent false
8234 3100 Acked Write 4 Done Done Sent false
8235 3101 Acked Read 8 Done Done Sent false
8236 3102 Acked Read 3 Done Done Sent false
8237 3103 Acked Write 8 Done Done Sent false
8238 3104 Acked Write 7 Done Done Sent false
8239 3105 Acked Read 8 Done Done Sent false
8240 3106 Acked Read 5 Done Done Sent false
8241 3107 Acked Write 10 Done Done Sent false
8242 3108 Acked Write 7 Done Done Sent false
8243 3109 Acked Read 5 Done Done Sent false
8244 3110 Acked Read 2 Done Done Sent false
8245 3111 Acked Read 10 Done Done Sent false
8246 3112 Acked Flush 0 Done Done Sent false
8247 STATES DS:0 DS:1 DS:2 TOTAL
8248 Sent 0 0 302 302
8249 Done 302 302 0 604
8250 Skipped 0 0 0 0
8251 Error 0 0 0 0
8252 Last Flush: Some(JobId(3112)) Some(JobId(3112)) Some(JobId(2810))
8253 Downstairs last five completed: 2810 2809 2808 2807 2806
8254 Upstairs last five completed: 3112 3111 3110 3109 3108
8255 CLIENT: Up:0 ds:302 act:2
82562025-01-23T18:49:36.435ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
82572025-01-23T18:49:36.435ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43331) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
82582025-01-23T18:49:36.436ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
82592025-01-23T18:49:36.437ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
82602025-01-23T18:49:36.437ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
82612025-01-23T18:49:36.437ZINFOcrucible: send last flush ID to this DS: Some(JobId(2810)) = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
82622025-01-23T18:49:36.437ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
82632025-01-23T18:49:36.437ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(2810)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
82642025-01-23T18:49:36.437ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(2810)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
8265 ----------------------------------------------------------------
8266 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8267 Downstairs last five completed: 3112 3111 3110 3109 3108
8268 Upstairs last five completed: 3112 3111 3110 3109 3108
8269 CLIENT: Up:0 ds:0 act:3
8270 Replay: All jobs finished
8271 Main task has work to do, go find it
8272 got dsc Stop(1)
8273 stop 1
8274 [8820] Got stop action so:false kr:false
8275 Jan 23 18:49:38.690 INFO request completed, latency_us: 297, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 1d8264a1-4dc2-4f17-ae79-976c1cea2950, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
82762025-01-23T18:49:38.691ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
8277 Jan 23 18:49:38.691 INFO request completed{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:49:38.691681534Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
8278 , latency_us: 119, response_code: 200, uri:{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:49:38.691702512Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
8279 /state/cid/1, method: GET, req_id: ff682994-d527-49dd-a02f-f849c8e67760, remote_addr: 127.0.0.1:59295, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T18:49:38.691733448Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
8280 local_addr: 127.0.0.1:9998
8281 [8820] Exited with: ExitStatus(unix_wait_status(9))
8282 [8820][1] reports Exit
8283 Jan 23 18:49:42.693 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4712adf3-a277-4469-a0cb-91ef75847c6d, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
8284 001/300 Write block 1864 len 40960 data: 2 3 3 3 3 3 3 3 3 3
8285 002/300 Write block 1134 len 12288 data: 4 4 4
8286 003/300 Read block 1288 len 4096
8287 004/300 Write block 2888 len 4096 data: 4
8288 005/300 Write block 2593 len 8192 data: 3 3
8289 006/300 Write block 3057 len 20480 data: 4 4 3 3 3
8290 007/300 Write block 3235 len 16384 data: 5 6 6 6
8291 008/300 Read block 249 len 12288
8292 009/300 Read block 4656 len 16384
8293 010/300 Write block 2274 len 40960 data: 4 4 3 4 3 3 3 3 3 4
8294 011/300 Read block 3105 len 12288
8295 012/300 Read block 3798 len 40960
8296 013/300 Read block 2065 len 4096
8297 014/300 Read block 1814 len 16384
8298 015/300 Write block 1770 len 12288 data: 2 2 2
8299 016/300 Read block 3685 len 32768
8300 017/300 Read block 869 len 40960
8301 018/300 Write block 3859 len 8192 data: 3 4
8302 019/300 Flush
8303 020/300 Read block 1012 len 8192
8304 021/300 Write block 2885 len 4096 data: 2
8305 022/300 Write block 4982 len 28672 data: 3 3 3 3 3 3 4
8306 023/300 Read block 246 len 12288
8307 024/300 Read block 3255 len 12288
8308 025/300 Flush
8309 026/300 Read block 2944 len 32768
8310 027/300 Read block 867 len 40960
8311 028/300 Read block 3829 len 16384
8312 029/300 Read block 4545 len 20480
8313 030/300 Read block 1039 len 28672
8314 031/300 Write block 2394 len 4096 data: 2
8315 032/300 Write block 3603 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8316 033/300 Read block 1439 len 32768
8317 034/300 Write block 4653 len 28672 data: 2 2 2 2 2 4 4
8318 035/300 Read block 2805 len 20480
8319 036/300 Read block 2332 len 4096
8320 037/300 Write block 2844 len 8192 data: 2 2
8321 038/300 Read block 4653 len 20480
8322 039/300 Read block 163 len 28672
8323 040/300 Write block 707 len 40960 data: 2 2 2 3 3 3 3 3 4 4
8324 041/300 Write block 343 len 36864 data: 2 2 2 2 4 4 4 4 4
8325 042/300 Read block 2531 len 24576
8326 043/300 Read block 2395 len 4096
8327 044/300 Read block 4914 len 36864
8328 045/300 Read block 3190 len 16384
8329 046/300 Read block 646 len 36864
8330 047/300 Flush
8331 048/300 Write block 4279 len 12288 data: 3 3 3
8332 049/300 Read block 2248 len 24576
8333 050/300 Write block 1088 len 16384 data: 2 2 2 2
8334 051/300 Read block 4210 len 8192
8335 052/300 Write block 2974 len 8192 data: 2 2
8336 053/300 Read block 3006 len 4096
8337 054/300 Read block 2464 len 28672
8338 055/300 Read block 2034 len 40960
8339 056/300 Write block 1139 len 32768 data: 5 4 4 3 3 3 4 4
8340 057/300 Read block 395 len 40960
8341 058/300 Read block 4818 len 12288
8342 059/300 Write block 3895 len 32768 data: 2 3 3 3 2 2 2 3
8343 060/300 Read block 4284 len 12288
8344 061/300 Read block 1264 len 24576
8345 062/300 Read block 2568 len 36864
8346 063/300 Write block 3370 len 40960 data: 2 2 2 2 2 2 2 2 2 3
8347 064/300 Write block 4456 len 36864 data: 2 3 3 3 4 4 4 3 3
8348 065/300 Write block 1149 len 24576 data: 2 2 3 4 4 3
8349 066/300 Read block 2003 len 20480
8350 067/300 Write block 3893 len 36864 data: 2 2 3 4 4 4 3 3 3
8351 068/300 Write block 953 len 40960 data: 3 3 3 3 3 3 3 4 4 4
8352 069/300 Write block 2980 len 8192 data: 2 2
8353 070/300 Read block 2278 len 12288
8354 071/300 Read block 4036 len 32768
8355 072/300 Read block 3571 len 16384
8356 073/300 Read block 459 len 20480
8357 074/300 Flush
8358 075/300 Write block 2612 len 8192 data: 2 2
8359 076/300 Read block 4480 len 32768
8360 077/300 Write block 3251 len 36864 data: 5 5 5 4 5 5 5 4 4
8361 078/300 Read block 4659 len 24576
8362 079/300 Write block 4125 len 4096 data: 2
8363 080/300 Read block 2441 len 32768
8364 081/300 Write block 514 len 8192 data: 2 2
8365 082/300 Read block 3544 len 8192
8366 083/300 Write block 4514 len 4096 data: 3
8367 084/300 Write block 2006 len 40960 data: 3 3 3 3 3 4 4 4 4 4
8368 085/300 Read block 597 len 36864
8369 086/300 Write block 2026 len 32768 data: 3 2 2 2 2 2 2 2
8370 087/300 Write block 1754 len 4096 data: 2
8371 088/300 Flush
8372 089/300 Flush
8373 090/300 Write block 2320 len 24576 data: 4 3 3 4 5 5
8374 091/300 Read block 1065 len 16384
8375 092/300 Read block 3234 len 24576
8376 093/300 Read block 4926 len 40960
8377 094/300 Write block 815 len 36864 data: 3 3 3 3 3 4 4 4 4
8378 095/300 Read block 4146 len 32768
8379 096/300 Read block 49 len 24576
8380 097/300 Write block 3385 len 16384 data: 2 2 2 2
8381 098/300 Write block 1140 len 40960 data: 5 5 4 4 4 5 5 4 3 3
8382 099/300 Read block 1008 len 12288
8383 100/300 Flush
8384 101/300 Read block 3597 len 20480
8385 102/300 Write block 563 len 32768 data: 3 3 3 3 4 4 3 2
8386 103/300 Flush
8387 104/300 Write block 3720 len 28672 data: 4 4 4 4 3 2 3
8388 105/300 Write block 886 len 24576 data: 2 3 3 2 2 2
8389 106/300 Read block 438 len 40960
8390 107/300 Write block 2053 len 4096 data: 4
8391 108/300 Read block 4296 len 20480
8392 109/300 Flush
8393 110/300 Write block 4111 len 24576 data: 2 2 2 2 2 2
8394 111/300 Read block 30 len 28672
8395 112/300 Flush
8396 113/300 Read block 1006 len 32768
8397 114/300 Read block 1978 len 28672
8398 115/300 Read block 1364 len 4096
8399 116/300 Read block 4508 len 28672
8400 117/300 Write block 3035 len 28672 data: 3 3 3 4 5 6 5
8401 118/300 Write block 1505 len 40960 data: 3 3 3 3 3 3 3 2 2 2
8402 119/300 Write block 43 len 32768 data: 2 2 2 2 2 2 2 2
8403 120/300 Write block 2295 len 36864 data: 4 3 3 3 3 2 2 2 2
8404 121/300 Write block 4622 len 36864 data: 2 2 2 2 2 2 2 2 2
8405 122/300 Read block 2852 len 32768
8406 123/300 Read block 2617 len 28672
8407 124/300 Read block 616 len 8192
8408 125/300 Read block 1265 len 32768
8409 126/300 Write block 1200 len 20480 data: 3 3 3 2 2
8410 127/300 Read block 1087 len 4096
8411 128/300 Write block 1193 len 20480 data: 3 3 3 2 2
8412 129/300 Write block 4755 len 8192 data: 4 4
8413 130/300 Read block 4398 len 12288
8414 131/300 Write block 917 len 4096 data: 4
8415 132/300 Flush
8416 133/300 Flush
8417 134/300 Read block 3250 len 24576
8418 135/300 Write block 1020 len 20480 data: 5 5 3 3 2
8419 136/300 Read block 2579 len 28672
8420 137/300 Write block 1131 len 40960 data: 2 4 4 5 5 5 4 4 6 6
8421 138/300 Write block 3231 len 40960 data: 3 4 4 4 6 7 7 7 5 4
8422 139/300 Write block 788 len 40960 data: 5 4 3 3 2 2 2 2 2 2
8423 140/300 Read block 4783 len 20480
8424 141/300 Read block 4092 len 16384
8425 142/300 Write block 1644 len 36864 data: 2 2 2 2 2 2 2 2 2
8426 143/300 Read block 1298 len 32768
8427 144/300 Read block 210 len 4096
8428 145/300 Read block 2757 len 12288
8429 146/300 Flush
8430 147/300 Write block 547 len 28672 data: 5 5 5 5 5 4 3
8431 148/300 Flush
8432 149/300 Read block 2442 len 12288
8433 150/300 Write block 2095 len 36864 data: 2 3 3 2 3 2 2 2 2
8434 151/300 Read block 2433 len 24576
8435 152/300 Flush
8436 153/300 Read block 2625 len 24576
8437 154/300 Write block 2533 len 20480 data: 3 4 4 4 4
8438 155/300 Write block 4355 len 40960 data: 3 3 3 2 2 2 2 2 2 2
8439 156/300 Read block 1465 len 16384
8440 157/300 Read block 3236 len 24576
8441 158/300 Flush
8442 159/300 Read block 806 len 40960
8443 160/300 Read block 4502 len 36864
8444 161/300 Flush
8445 162/300 Write block 1811 len 20480 data: 5 4 4 4 4
8446 163/300 Read block 4012 len 32768
8447 164/300 Write block 245 len 16384 data: 2 2 4 4
8448 165/300 Read block 2299 len 16384
8449 166/300 Read block 106 len 16384
8450 167/300 Write block 3404 len 4096 data: 2
8451 168/300 Write block 4416 len 36864 data: 2 2 2 2 2 3 3 3 5
8452 169/300 Write block 1794 len 36864 data: 4 4 4 4 4 4 2 2 2
8453 170/300 Write block 4499 len 4096 data: 3
8454 171/300 Write block 2426 len 8192 data: 3 4
8455 172/300 Write block 1148 len 32768 data: 4 4 3 4 5 5 4 3
8456 173/300 Read block 3733 len 16384
8457 174/300 Read block 3474 len 28672
8458 175/300 Read block 2468 len 4096
8459 176/300 Flush
8460 177/300 Read block 1142 len 40960
8461 178/300 Read block 1179 len 32768
8462 179/300 Read block 2849 len 20480
8463 180/300 Write block 1567 len 32768 data: 3 3 3 4 3 3 2 2
8464 181/300 Read block 3603 len 12288
8465 182/300 Read block 1910 len 8192
8466 183/300 Write block 1514 len 4096 data: 3
8467 184/300 Read block 2990 len 8192
8468 185/300 Read block 3 len 12288
8469 186/300 Write block 4147 len 20480 data: 6 6 6 6 5
8470 187/300 Read block 4725 len 24576
8471 188/300 Write block 3880 len 28672 data: 4 4 3 3 2 2 2
8472 189/300 Read block 2767 len 24576
8473 190/300 Write block 4813 len 4096 data: 4
8474 191/300 Write block 536 len 28672 data: 5 4 3 3 2 2 3
8475 192/300 Read block 1306 len 20480
8476 193/300 Write block 339 len 8192 data: 3 4
8477 194/300 Read block 2016 len 12288
8478 195/300 Flush
8479 196/300 Flush
8480 197/300 Write block 950 len 4096 data: 2
8481 198/300 Read block 974 len 8192
8482 199/300 Flush
8483 200/300 Read block 4152 len 32768
8484 201/300 Write block 1790 len 36864 data: 3 3 3 3 5 5 5 5 5
8485 202/300 Read block 1505 len 24576
8486 203/300 Write block 3838 len 20480 data: 3 2 2 2 2
8487 204/300 Write block 3120 len 16384 data: 4 4 3 3
8488 205/300 Write block 2804 len 4096 data: 3
8489 206/300 Write block 3417 len 28672 data: 2 2 2 2 2 3 4
8490 207/300 Read block 1135 len 32768
8491 208/300 Write block 3245 len 8192 data: 4 3
8492 209/300 Read block 4167 len 36864
8493 210/300 Write block 3566 len 40960 data: 2 2 2 2 2 2 2 3 4 4
8494 211/300 Write block 429 len 28672 data: 6 6 6 4 4 4 2
8495 212/300 Write block 1811 len 32768 data: 6 5 5 5 5 3 3 3
8496 213/300 Read block 2694 len 24576
8497 214/300 Write block 1064 len 32768 data: 3 3 3 4 4 4 4 3
8498 215/300 Flush
8499 216/300 Write block 1534 len 8192 data: 4 2
8500 217/300 Write block 3464 len 8192 data: 4 4
8501 218/300 Read block 3696 len 12288
8502 219/300 Write block 3316 len 36864 data: 3 3 2 2 2 2 2 2 2
8503 220/300 Write block 4198 len 32768 data: 2 2 2 2 2 2 2 2
8504 221/300 Write block 1069 len 20480 data: 5 5 4 3 3
8505 222/300 Read block 551 len 20480
8506 223/300 Write block 1861 len 28672 data: 2 2 2 3 4 4 4
8507 224/300 Write block 2125 len 20480 data: 3 3 3 4 4
8508 225/300 Read block 3448 len 36864
8509 226/300 Read block 2909 len 40960
8510 227/300 Read block 1776 len 12288
8511 228/300 Write block 4534 len 16384 data: 3 3 2 3
8512 229/300 Flush
8513 230/300 Read block 2095 len 20480
8514 231/300 Read block 3993 len 32768
8515 232/300 Write block 426 len 16384 data: 3 4 6 7
8516 233/300 Write block 3441 len 40960 data: 2 3 3 3 2 2 2 2 2 2
8517 234/300 Read block 3606 len 4096
8518 235/300 Write block 3559 len 4096 data: 4
8519 236/300 Write block 1133 len 8192 data: 5 6
8520 237/300 Flush
8521 238/300 Write block 2370 len 4096 data: 2
8522 239/300 Write block 1304 len 28672 data: 3 3 2 3 3 3 3
8523 240/300 Read block 2112 len 20480
8524 241/300 Read block 1803 len 8192
8525 242/300 Read block 3502 len 28672
8526 243/300 Write block 4027 len 20480 data: 3 4 4 4 4
8527 244/300 Write block 2846 len 32768 data: 2 2 2 2 3 3 3 4
8528 245/300 Write block 1009 len 16384 data: 3 3 3 4
8529 246/300 Write block 4423 len 8192 data: 4 6
8530 247/300 Read block 2638 len 32768
8531 248/300 Read block 3551 len 12288
8532 249/300 Read block 3961 len 28672
8533 250/300 Write block 4847 len 28672 data: 4 4 3 2 2 2 2
8534 251/300 Write block 3229 len 36864 data: 2 3 4 5 5 5 7 8 8
8535 252/300 Flush
8536 253/300 Read block 1581 len 40960
8537 254/300 Read block 3742 len 20480
8538 255/300 Write block 1412 len 32768 data: 2 2 2 2 3 3 5 5
8539 256/300 Read block 4658 len 12288
8540 257/300 Write block 400 len 24576 data: 3 3 2 2 3 3
8541 258/300 Flush
8542 259/300 Write block 3857 len 36864 data: 3 3 4 5 3 3 3 3 3
8543 260/300 Write block 743 len 24576 data: 3 3 3 3 3 3
8544 261/300 Write block 2265 len 36864 data: 3 4 4 4 4 4 5 4 4
8545 262/300 Write block 2390 len 36864 data: 2 3 3 2 3 3 3 3 2
8546 263/300 Read block 2246 len 28672
8547 264/300 Write block 1461 len 28672 data: 3 3 3 4 5 5 4
8548 265/300 Flush
8549 266/300 Read block 2 len 28672
8550 267/300 Read block 3075 len 36864
8551 268/300 Read block 3481 len 32768
8552 269/300 Read block 418 len 20480
8553 270/300 Read block 52 len 16384
8554 271/300 Flush
8555 272/300 Read block 254 len 36864
8556 273/300 Write block 981 len 28672 data: 2 2 2 2 2 2 2
8557 274/300 Write block 2640 len 12288 data: 2 2 2
8558 275/300 Write block 1745 len 8192 data: 2 2
8559 276/300 Read block 3407 len 36864
8560 277/300 Read block 4462 len 32768
8561 278/300 Write block 3572 len 4096 data: 3
8562 279/300 Read block 1826 len 16384
8563 280/300 Write block 562 len 36864 data: 3 4 4 4 4 5 5 4 3
8564 281/300 Write block 2861 len 24576 data: 2 2 2 2 2 2
8565 282/300 Write block 3201 len 16384 data: 3 2 2 3
8566 283/300 Read block 3431 len 40960
8567 284/300 Write block 376 len 40960 data: 4 4 2 2 2 2 3 3 4 4
8568 285/300 Write block 176 len 36864 data: 3 3 5 4 3 3 3 3 3
8569 286/300 Write block 2101 len 24576 data: 3 3 3 3 3 3
8570 287/300 Write block 2270 len 24576 data: 5 6 5 5 5 5
8571 288/300 Flush
8572 289/300 Read block 689 len 12288
8573 290/300 Write block 2203 len 40960 data: 3 2 3 4 4 4 4 4 4 4
8574 291/300 Write block 1045 len 32768 data: 2 2 3 3 3 2 2 2
8575 292/300 Write block 3369 len 4096 data: 3
8576 293/300 Write block 140 len 32768 data: 5 4 3 3 3 3 3 3
8577 294/300 Read block 4874 len 28672
8578 295/300 Write block 1248 len 24576 data: 2 2 2 2 2 2
8579 296/300 Read block 1629 len 20480
8580 297/300 Write block 130 len 36864 data: 2 4 3 3 3 3 3 3 4
8581 298/300 Read block 3681 len 16384
8582 299/300 Read block 4091 len 20480
8583 300/300 Write block 477 len 8192 data: 2 3
8584 301/300 Write block 3416 len 12288 data: 2 3 3
8585 Main task has work to do, go find it
8586 got dsc Start(1)
8587 start 1
8588 Jan 23 18:49:42.931 INFO request completed, latency_us: 234, response_code: 204, uri: /start/cid/1, method: POST, req_id: ae452a16-9e7d-44da-8cfd-e565a1b6e310, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
8589 Replay: started 1, returned:Ok(())
8590 ----------------------------------------------------------------
8591 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8592 States: Active Offline Active
8593 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8594 3113 Acked Write 10 Done Sent Done false
8595 3114 Acked Write 3 Done Sent Done false
8596 3115 Acked Read 1 Done Sent Done false
8597 3116 Acked Write 1 Done Sent Done false
8598 3117 Acked Write 2 Done Sent Done false
8599 3118 Acked Write 5 Done Sent Done false
8600 3119 Acked Write 4 Done Sent Done false
8601 3120 Acked Read 3 Done Sent Done false
8602 3121 Acked Read 4 Done Sent Done false
8603 3122 Acked Write 10 Done Sent Done false
8604 3123 Acked Read 3 Done Sent Done false
8605 3124 Acked Read 10 Done Sent Done false
8606 3125 Acked Read 1 Done Sent Done false
8607 3126 Acked Read 4 Done Sent Done false
8608 3127 Acked Write 3 Done Sent Done false
8609 3128 Acked Read 8 Done Sent Done false
8610 3129 Acked Read 10 Done Sent Done false
8611 3130 Acked Write 2 Done Sent Done false
8612 3131 Acked Flush 0 Done Sent Done false
8613 3132 Acked Read 2 Done Sent Done false
8614 3133 Acked Write 1 Done Sent Done false
8615 3134 Acked Write 7 Done Sent Done false
8616 3135 Acked Read 3 Done Sent Done false
8617 3136 Acked Read 3 Done Sent Done false
8618 3137 Acked Flush 0 Done Sent Done false
8619 3138 Acked Read 8 Done Sent Done false
8620 3139 Acked Read 10 Done Sent Done false
8621 3140 Acked Read 4 Done Sent Done false
8622 3141 Acked Read 5 Done Sent Done false
8623 3142 Acked Read 7 Done Sent Done false
8624 3143 Acked Write 1 Done Sent Done false
8625 3144 Acked Write 10 Done Sent Done false
8626 3145 Acked Read 8 Done Sent Done false
8627 3146 Acked Write 7 Done Sent Done false
8628 3147 Acked Read 5 Done Sent Done false
8629 3148 Acked Read 1 Done Sent Done false
8630 3149 Acked Write 2 Done Sent Done false
8631 3150 Acked Read 5 Done Sent Done false
8632 3151 Acked Read 7 Done Sent Done false
8633 3152 Acked Write 10 Done Sent Done false
8634 3153 Acked Write 9 Done Sent Done false
8635 3154 Acked Read 6 Done Sent Done false
8636 3155 Acked Read 1 Done Sent Done false
8637 3156 Acked Read 9 Done Sent Done false
8638 3157 Acked Read 4 Done Sent Done false
8639 3158 Acked Read 9 Done Sent Done false
8640 3159 Acked Flush 0 Done Sent Done false
8641 3160 Acked Write 3 Done Sent Done false
8642 3161 Acked Read 6 Done Sent Done false
8643 3162 Acked Write 4 Done Sent Done false
8644 3163 Acked Read 2 Done Sent Done false
8645 3164 Acked Write 2 Done Sent Done false
8646 3165 Acked Read 1 Done Sent Done false
8647 3166 Acked Read 7 Done Sent Done false
8648 3167 Acked Read 10 Done Sent Done false
8649 3168 Acked Write 8 Done Sent Done false
8650 3169 Acked Read 10 Done Sent Done false
8651 3170 Acked Read 3 Done Sent Done false
8652 3171 Acked Write 8 Done Sent Done false
8653 3172 Acked Read 3 Done Sent Done false
8654 3173 Acked Read 6 Done Sent Done false
8655 3174 Acked Read 9 Done Sent Done false
8656 3175 Acked Write 10 Done Sent Done false
8657 3176 Acked Write 9 Done Sent Done false
8658 3177 Acked Write 6 Done Sent Done false
8659 3178 Acked Read 5 Done Sent Done false
8660 3179 Acked Write 9 Done Sent Done false
8661 3180 Acked Write 10 Done Sent Done false
8662 3181 Acked Write 2 Done Sent Done false
8663 3182 Acked Read 3 Done Sent Done false
8664 3183 Acked Read 8 Done Sent Done false
8665 3184 Acked Read 4 Done Sent Done false
8666 3185 Acked Read 5 Done Sent Done false
8667 3186 Acked Flush 0 Done Sent Done false
8668 3187 Acked Write 2 Done Sent Done false
8669 3188 Acked Read 8 Done Sent Done false
8670 3189 Acked Write 9 Done Sent Done false
8671 3190 Acked Read 6 Done Sent Done false
8672 3191 Acked Write 1 Done Sent Done false
8673 3192 Acked Read 8 Done Sent Done false
8674 3193 Acked Write 2 Done Sent Done false
8675 3194 Acked Read 2 Done Sent Done false
8676 3195 Acked Write 1 Done Sent Done false
8677 3196 Acked Write 10 Done Sent Done false
8678 3197 Acked Read 9 Done Sent Done false
8679 3198 Acked Write 8 Done Sent Done false
8680 3199 Acked Write 1 Done Sent Done false
8681 3200 Acked Flush 0 Done Sent Done false
8682 3201 Acked Flush 0 Done Sent Done false
8683 3202 Acked Write 6 Done Sent Done false
8684 3203 Acked Read 4 Done Sent Done false
8685 3204 Acked Read 6 Done Sent Done false
8686 3205 Acked Read 10 Done Sent Done false
8687 3206 Acked Write 9 Done Sent Done false
8688 3207 Acked Read 8 Done Sent Done false
8689 3208 Acked Read 6 Done Sent Done false
8690 3209 Acked Write 4 Done Sent Done false
8691 3210 Acked Write 10 Done Sent Done false
8692 3211 Acked Read 3 Done Sent Done false
8693 3212 Acked Flush 0 Done Sent Done false
8694 3213 Acked Read 5 Done Sent Done false
8695 3214 Acked Write 8 Done Sent Done false
8696 3215 Acked Flush 0 Done Sent Done false
8697 3216 Acked Write 7 Done Sent Done false
8698 3217 Acked Write 6 Done Sent Done false
8699 3218 Acked Read 10 Done Sent Done false
8700 3219 Acked Write 1 Done Sent Done false
8701 3220 Acked Read 5 Done Sent Done false
8702 3221 Acked Flush 0 Done Sent Done false
8703 3222 Acked Write 6 Done Sent Done false
8704 3223 Acked Read 7 Done Sent Done false
8705 3224 Acked Flush 0 Done Sent Done false
8706 3225 Acked Read 8 Done Sent Done false
8707 3226 Acked Read 7 Done Sent Done false
8708 3227 Acked Read 1 Done Sent Done false
8709 3228 Acked Read 7 Done Sent Done false
8710 3229 Acked Write 7 Done Sent Done false
8711 3230 Acked Write 10 Done Sent Done false
8712 3231 Acked Write 8 Done Sent Done false
8713 3232 Acked Write 9 Done Sent Done false
8714 3233 Acked Write 9 Done Sent Done false
8715 3234 Acked Read 8 Done Sent Done false
8716 3235 Acked Read 7 Done Sent Done false
8717 3236 Acked Read 2 Done Sent Done false
8718 3237 Acked Read 8 Done Sent Done false
8719 3238 Acked Write 5 Done Sent Done false
8720 3239 Acked Read 1 Done Sent Done false
8721 3240 Acked Write 5 Done Sent Done false
8722 3241 Acked Write 2 Done Sent Done false
8723 3242 Acked Read 3 Done Sent Done false
8724 3243 Acked Write 1 Done Sent Done false
8725 3244 Acked Flush 0 Done Sent Done false
8726 3245 Acked Flush 0 Done Sent Done false
8727 3246 Acked Read 6 Done Sent Done false
8728 3247 Acked Write 5 Done Sent Done false
8729 3248 Acked Read 7 Done Sent Done false
8730 3249 Acked Write 10 Done Sent Done false
8731 3250 Acked Write 10 Done Sent Done false
8732 3251 Acked Write 10 Done Sent Done false
8733 3252 Acked Read 5 Done Sent Done false
8734 3253 Acked Read 4 Done Sent Done false
8735 3254 Acked Write 9 Done Sent Done false
8736 3255 Acked Read 8 Done Sent Done false
8737 3256 Acked Read 1 Done Sent Done false
8738 3257 Acked Read 3 Done Sent Done false
8739 3258 Acked Flush 0 Done Sent Done false
8740 3259 Acked Write 7 Done Sent Done false
8741 3260 Acked Flush 0 Done Sent Done false
8742 3261 Acked Read 3 Done Sent Done false
8743 3262 Acked Write 9 Done Sent Done false
8744 3263 Acked Read 6 Done Sent Done false
8745 3264 Acked Flush 0 Done Sent Done false
8746 3265 Acked Read 6 Done Sent Done false
8747 3266 Acked Write 5 Done Sent Done false
8748 3267 Acked Write 10 Done Sent Done false
8749 3268 Acked Read 4 Done Sent Done false
8750 3269 Acked Read 6 Done Sent Done false
8751 3270 Acked Flush 0 Done Sent Done false
8752 3271 Acked Read 10 Done Sent Done false
8753 3272 Acked Read 9 Done Sent Done false
8754 3273 Acked Flush 0 Done Sent Done false
8755 3274 Acked Write 5 Done Sent Done false
8756 3275 Acked Read 8 Done Sent Done false
8757 3276 Acked Write 4 Done Sent Done false
8758 3277 Acked Read 4 Done Sent Done false
8759 3278 Acked Read 4 Done Sent Done false
8760 3279 Acked Write 1 Done Sent Done false
8761 3280 Acked Write 9 Done Sent Done false
8762 3281 Acked Write 9 Done Sent Done false
8763 3282 Acked Write 1 Done Sent Done false
8764 3283 Acked Write 2 Done Sent Done false
8765 3284 Acked Write 8 Done Sent Done false
8766 3285 Acked Read 4 Done Sent Done false
8767 3286 Acked Read 7 Done Sent Done false
8768 3287 Acked Read 1 Done Sent Done false
8769 3288 Acked Flush 0 Done Sent Done false
8770 3289 Acked Read 10 Done Sent Done false
8771 3290 Acked Read 8 Done Sent Done false
8772 3291 Acked Read 5 Done Sent Done false
8773 3292 Acked Write 8 Done Sent Done false
8774 3293 Acked Read 3 Done Sent Done false
8775 3294 Acked Read 2 Done Sent Done false
8776 3295 Acked Write 1 Done Sent Done false
8777 3296 Acked Read 2 Done Sent Done false
8778 3297 Acked Read 3 Done Sent Done false
8779 3298 Acked Write 5 Done Sent Done false
8780 3299 Acked Read 6 Done Sent Done false
8781 3300 Acked Write 7 Done Sent Done false
8782 3301 Acked Read 6 Done Sent Done false
8783 3302 Acked Write 1 Done Sent Done false
8784 3303 Acked Write 7 Done Sent Done false
8785 3304 Acked Read 5 Done Sent Done false
8786 3305 Acked Write 2 Done Sent Done false
8787 3306 Acked Read 3 Done Sent Done false
8788 3307 Acked Flush 0 Done Sent Done false
8789 3308 Acked Flush 0 Done Sent Done false
8790 3309 Acked Write 1 Done Sent Done false
8791 3310 Acked Read 2 Done Sent Done false
8792 3311 Acked Flush 0 Done Sent Done false
8793 3312 Acked Read 8 Done Sent Done false
8794 3313 Acked Write 9 Done Sent Done false
8795 3314 Acked Read 6 Done Sent Done false
8796 3315 Acked Write 5 Done Sent Done false
8797 3316 Acked Write 4 Done Sent Done false
8798 3317 Acked Write 1 Done Sent Done false
8799 3318 Acked Write 7 Done Sent Done false
8800 3319 Acked Read 8 Done Sent Done false
8801 3320 Acked Write 2 Done Sent Done false
8802 3321 Acked Read 9 Done Sent Done false
8803 3322 Acked Write 10 Done Sent Done false
8804 3323 Acked Write 7 Done Sent Done false
8805 3324 Acked Write 8 Done Sent Done false
8806 3325 Acked Read 6 Done Sent Done false
8807 3326 Acked Write 8 Done Sent Done false
8808 3327 Acked Flush 0 Done Sent Done false
8809 3328 Acked Write 2 Done Sent Done false
8810 3329 Acked Write 2 Done Sent Done false
8811 3330 Acked Read 3 Done Sent Done false
8812 3331 Acked Write 9 Done Sent Done false
8813 3332 Acked Write 8 Done Sent Done false
8814 3333 Acked Write 5 Done Sent Done false
8815 3334 Acked Read 5 Done Sent Done false
8816 3335 Acked Write 7 Done Sent Done false
8817 3336 Acked Write 5 Done Sent Done false
8818 3337 Acked Read 9 Done Sent Done false
8819 3338 Acked Read 10 Done Sent Done false
8820 3339 Acked Read 3 Done Sent Done false
8821 3340 Acked Write 4 Done Sent Done false
8822 3341 Acked Flush 0 Done Sent Done false
8823 3342 Acked Read 5 Done Sent Done false
8824 3343 Acked Read 8 Done Sent Done false
8825 3344 Acked Write 4 Done Sent Done false
8826 3345 Acked Write 10 Done Sent Done false
8827 3346 Acked Read 1 Done Sent Done false
8828 3347 Acked Write 1 Done Sent Done false
8829 3348 Acked Write 2 Done Sent Done false
8830 3349 Acked Flush 0 Done Sent Done false
8831 3350 Acked Write 1 Done Sent Done false
8832 3351 Acked Write 7 Done Sent Done false
8833 3352 Acked Read 5 Done Sent Done false
8834 3353 Acked Read 2 Done Sent Done false
8835 3354 Acked Read 7 Done Sent Done false
8836 3355 Acked Write 5 Done Sent Done false
8837 3356 Acked Write 8 Done Sent Done false
8838 3357 Acked Write 4 Done Sent Done false
8839 3358 Acked Write 2 Done Sent Done false
8840 3359 Acked Read 8 Done Sent Done false
8841 3360 Acked Read 3 Done Sent Done false
8842 3361 Acked Read 7 Done Sent Done false
8843 3362 Acked Write 7 Done Sent Done false
8844 3363 Acked Write 9 Done Sent Done false
8845 3364 Acked Flush 0 Done Sent Done false
8846 3365 Acked Read 10 Done Sent Done false
8847 3366 Acked Read 5 Done Sent Done false
8848 3367 Acked Write 8 Done Sent Done false
8849 3368 Acked Read 3 Done Sent Done false
8850 3369 Acked Write 6 Done Sent Done false
8851 3370 Acked Flush 0 Done Sent Done false
8852 3371 Acked Write 9 Done Sent Done false
8853 3372 Acked Write 6 Done Sent Done false
8854 3373 Acked Write 9 Done Sent Done false
8855 3374 Acked Write 9 Done Sent Done false
8856 3375 Acked Read 7 Done Sent Done false
8857 3376 Acked Write 7 Done Sent Done false
8858 3377 Acked Flush 0 Done Sent Done false
8859 3378 Acked Read 7 Done Sent Done false
8860 3379 Acked Read 9 Done Sent Done false
8861 3380 Acked Read 8 Done Sent Done false
8862 3381 Acked Read 5 Done Sent Done false
8863 3382 Acked Read 4 Done Sent Done false
8864 3383 Acked Flush 0 Done Sent Done false
8865 3384 Acked Read 9 Done Sent Done false
8866 3385 Acked Write 7 Done Sent Done false
8867 3386 Acked Write 3 Done Sent Done false
8868 3387 Acked Write 2 Done Sent Done false
8869 3388 Acked Read 9 Done Sent Done false
8870 3389 Acked Read 8 Done Sent Done false
8871 3390 Acked Write 1 Done Sent Done false
8872 3391 Acked Read 4 Done Sent Done false
8873 3392 Acked Write 9 Done Sent Done false
8874 3393 Acked Write 6 Done Sent Done false
8875 3394 Acked Write 4 Done Sent Done false
8876 3395 Acked Read 10 Done Sent Done false
8877 3396 Acked Write 10 Done Sent Done false
8878 3397 Acked Write 9 Done Sent Done false
8879 3398 Acked Write 6 Done Sent Done false
8880 3399 Acked Write 6 Done Sent Done false
8881 3400 Acked Flush 0 Done Sent Done false
8882 3401 Acked Read 3 Done Sent Done false
8883 3402 Acked Write 10 Done Sent Done false
8884 3403 Acked Write 8 Done Sent Done false
8885 3404 Acked Write 1 Done Sent Done false
8886 3405 Acked Write 8 Done Sent Done false
8887 3406 Acked Read 7 Done Sent Done false
8888 3407 Acked Write 6 Done Sent Done false
8889 3408 Acked Read 5 Done Sent Done false
8890 3409 Acked Write 9 Done Sent Done false
8891 3410 Acked Read 4 Done Sent Done false
8892 3411 Acked Read 5 Done Sent Done false
8893 3412 Acked Write 2 Done Sent Done false
8894 3413 Acked Write 3 Done Sent Done false
8895 STATES DS:0 DS:1 DS:2 TOTAL
8896 Sent 0 301 0 301
8897 Done 301 0 301 602
8898 Skipped 0 0 0 0
8899 Error 0 0 0 0
8900 Last Flush: Some(JobId(3400)) Some(JobId(3112)) Some(JobId(3400))
8901 Downstairs last five completed: 3112 3111 3110 3109 3108
8902 Upstairs last five completed: 3413 3412 3411 3410 3409
8903 CLIENT: Up:0 ds:301 act:2
8904 [8820] got start action
8905 [8820] I am going to restart
8906 Starting downstairs at port 8820
8907 Make output file at "/tmp/dsc/downstairs-8820.txt"
8908 [8820][1] reports Starting
8909 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1208)
8910 [8820][1] reports Running
8911 ----------------------------------------------------------------
8912 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
8913 States: Active Offline Active
8914 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8915 3113 Acked Write 10 Done Sent Done false
8916 3114 Acked Write 3 Done Sent Done false
8917 3115 Acked Read 1 Done Sent Done false
8918 3116 Acked Write 1 Done Sent Done false
8919 3117 Acked Write 2 Done Sent Done false
8920 3118 Acked Write 5 Done Sent Done false
8921 3119 Acked Write 4 Done Sent Done false
8922 3120 Acked Read 3 Done Sent Done false
8923 3121 Acked Read 4 Done Sent Done false
8924 3122 Acked Write 10 Done Sent Done false
8925 3123 Acked Read 3 Done Sent Done false
8926 3124 Acked Read 10 Done Sent Done false
8927 3125 Acked Read 1 Done Sent Done false
8928 3126 Acked Read 4 Done Sent Done false
8929 3127 Acked Write 3 Done Sent Done false
8930 3128 Acked Read 8 Done Sent Done false
8931 3129 Acked Read 10 Done Sent Done false
8932 3130 Acked Write 2 Done Sent Done false
8933 3131 Acked Flush 0 Done Sent Done false
8934 3132 Acked Read 2 Done Sent Done false
8935 3133 Acked Write 1 Done Sent Done false
8936 3134 Acked Write 7 Done Sent Done false
8937 3135 Acked Read 3 Done Sent Done false
8938 3136 Acked Read 3 Done Sent Done false
8939 3137 Acked Flush 0 Done Sent Done false
8940 3138 Acked Read 8 Done Sent Done false
8941 3139 Acked Read 10 Done Sent Done false
8942 3140 Acked Read 4 Done Sent Done false
8943 3141 Acked Read 5 Done Sent Done false
8944 3142 Acked Read 7 Done Sent Done false
8945 3143 Acked Write 1 Done Sent Done false
8946 3144 Acked Write 10 Done Sent Done false
8947 3145 Acked Read 8 Done Sent Done false
8948 3146 Acked Write 7 Done Sent Done false
8949 3147 Acked Read 5 Done Sent Done false
8950 3148 Acked Read 1 Done Sent Done false
8951 3149 Acked Write 2 Done Sent Done false
8952 3150 Acked Read 5 Done Sent Done false
8953 3151 Acked Read 7 Done Sent Done false
8954 3152 Acked Write 10 Done Sent Done false
8955 3153 Acked Write 9 Done Sent Done false
8956 3154 Acked Read 6 Done Sent Done false
8957 3155 Acked Read 1 Done Sent Done false
8958 3156 Acked Read 9 Done Sent Done false
8959 3157 Acked Read 4 Done Sent Done false
8960 3158 Acked Read 9 Done Sent Done false
8961 3159 Acked Flush 0 Done Sent Done false
8962 3160 Acked Write 3 Done Sent Done false
8963 3161 Acked Read 6 Done Sent Done false
8964 3162 Acked Write 4 Done Sent Done false
8965 3163 Acked Read 2 Done Sent Done false
8966 3164 Acked Write 2 Done Sent Done false
8967 3165 Acked Read 1 Done Sent Done false
8968 3166 Acked Read 7 Done Sent Done false
8969 3167 Acked Read 10 Done Sent Done false
8970 3168 Acked Write 8 Done Sent Done false
8971 3169 Acked Read 10 Done Sent Done false
8972 3170 Acked Read 3 Done Sent Done false
8973 3171 Acked Write 8 Done Sent Done false
8974 3172 Acked Read 3 Done Sent Done false
8975 3173 Acked Read 6 Done Sent Done false
8976 3174 Acked Read 9 Done Sent Done false
8977 3175 Acked Write 10 Done Sent Done false
8978 3176 Acked Write 9 Done Sent Done false
8979 3177 Acked Write 6 Done Sent Done false
8980 3178 Acked Read 5 Done Sent Done false
8981 3179 Acked Write 9 Done Sent Done false
8982 3180 Acked Write 10 Done Sent Done false
8983 3181 Acked Write 2 Done Sent Done false
8984 3182 Acked Read 3 Done Sent Done false
8985 3183 Acked Read 8 Done Sent Done false
8986 3184 Acked Read 4 Done Sent Done false
8987 3185 Acked Read 5 Done Sent Done false
8988 3186 Acked Flush 0 Done Sent Done false
8989 3187 Acked Write 2 Done Sent Done false
8990 3188 Acked Read 8 Done Sent Done false
8991 3189 Acked Write 9 Done Sent Done false
8992 3190 Acked Read 6 Done Sent Done false
8993 3191 Acked Write 1 Done Sent Done false
8994 3192 Acked Read 8 Done Sent Done false
8995 3193 Acked Write 2 Done Sent Done false
8996 3194 Acked Read 2 Done Sent Done false
8997 3195 Acked Write 1 Done Sent Done false
8998 3196 Acked Write 10 Done Sent Done false
8999 3197 Acked Read 9 Done Sent Done false
9000 3198 Acked Write 8 Done Sent Done false
9001 3199 Acked Write 1 Done Sent Done false
9002 3200 Acked Flush 0 Done Sent Done false
9003 3201 Acked Flush 0 Done Sent Done false
9004 3202 Acked Write 6 Done Sent Done false
9005 3203 Acked Read 4 Done Sent Done false
9006 3204 Acked Read 6 Done Sent Done false
9007 3205 Acked Read 10 Done Sent Done false
9008 3206 Acked Write 9 Done Sent Done false
9009 3207 Acked Read 8 Done Sent Done false
9010 3208 Acked Read 6 Done Sent Done false
9011 3209 Acked Write 4 Done Sent Done false
9012 3210 Acked Write 10 Done Sent Done false
9013 3211 Acked Read 3 Done Sent Done false
9014 3212 Acked Flush 0 Done Sent Done false
9015 3213 Acked Read 5 Done Sent Done false
9016 3214 Acked Write 8 Done Sent Done false
9017 3215 Acked Flush 0 Done Sent Done false
9018 3216 Acked Write 7 Done Sent Done false
9019 3217 Acked Write 6 Done Sent Done false
9020 3218 Acked Read 10 Done Sent Done false
9021 3219 Acked Write 1 Done Sent Done false
9022 3220 Acked Read 5 Done Sent Done false
9023 3221 Acked Flush 0 Done Sent Done false
9024 3222 Acked Write 6 Done Sent Done false
9025 3223 Acked Read 7 Done Sent Done false
9026 3224 Acked Flush 0 Done Sent Done false
9027 3225 Acked Read 8 Done Sent Done false
9028 3226 Acked Read 7 Done Sent Done false
9029 3227 Acked Read 1 Done Sent Done false
9030 3228 Acked Read 7 Done Sent Done false
9031 3229 Acked Write 7 Done Sent Done false
9032 3230 Acked Write 10 Done Sent Done false
9033 3231 Acked Write 8 Done Sent Done false
9034 3232 Acked Write 9 Done Sent Done false
9035 3233 Acked Write 9 Done Sent Done false
9036 3234 Acked Read 8 Done Sent Done false
9037 3235 Acked Read 7 Done Sent Done false
9038 3236 Acked Read 2 Done Sent Done false
9039 3237 Acked Read 8 Done Sent Done false
9040 3238 Acked Write 5 Done Sent Done false
9041 3239 Acked Read 1 Done Sent Done false
9042 3240 Acked Write 5 Done Sent Done false
9043 3241 Acked Write 2 Done Sent Done false
9044 3242 Acked Read 3 Done Sent Done false
9045 3243 Acked Write 1 Done Sent Done false
9046 3244 Acked Flush 0 Done Sent Done false
9047 3245 Acked Flush 0 Done Sent Done false
9048 3246 Acked Read 6 Done Sent Done false
9049 3247 Acked Write 5 Done Sent Done false
9050 3248 Acked Read 7 Done Sent Done false
9051 3249 Acked Write 10 Done Sent Done false
9052 3250 Acked Write 10 Done Sent Done false
9053 3251 Acked Write 10 Done Sent Done false
9054 3252 Acked Read 5 Done Sent Done false
9055 3253 Acked Read 4 Done Sent Done false
9056 3254 Acked Write 9 Done Sent Done false
9057 3255 Acked Read 8 Done Sent Done false
9058 3256 Acked Read 1 Done Sent Done false
9059 3257 Acked Read 3 Done Sent Done false
9060 3258 Acked Flush 0 Done Sent Done false
9061 3259 Acked Write 7 Done Sent Done false
9062 3260 Acked Flush 0 Done Sent Done false
9063 3261 Acked Read 3 Done Sent Done false
9064 3262 Acked Write 9 Done Sent Done false
9065 3263 Acked Read 6 Done Sent Done false
9066 3264 Acked Flush 0 Done Sent Done false
9067 3265 Acked Read 6 Done Sent Done false
9068 3266 Acked Write 5 Done Sent Done false
9069 3267 Acked Write 10 Done Sent Done false
9070 3268 Acked Read 4 Done Sent Done false
9071 3269 Acked Read 6 Done Sent Done false
9072 3270 Acked Flush 0 Done Sent Done false
9073 3271 Acked Read 10 Done Sent Done false
9074 3272 Acked Read 9 Done Sent Done false
9075 3273 Acked Flush 0 Done Sent Done false
9076 3274 Acked Write 5 Done Sent Done false
9077 3275 Acked Read 8 Done Sent Done false
9078 3276 Acked Write 4 Done Sent Done false
9079 3277 Acked Read 4 Done Sent Done false
9080 3278 Acked Read 4 Done Sent Done false
9081 3279 Acked Write 1 Done Sent Done false
9082 3280 Acked Write 9 Done Sent Done false
9083 3281 Acked Write 9 Done Sent Done false
9084 3282 Acked Write 1 Done Sent Done false
9085 3283 Acked Write 2 Done Sent Done false
9086 3284 Acked Write 8 Done Sent Done false
9087 3285 Acked Read 4 Done Sent Done false
9088 3286 Acked Read 7 Done Sent Done false
9089 3287 Acked Read 1 Done Sent Done false
9090 3288 Acked Flush 0 Done Sent Done false
9091 3289 Acked Read 10 Done Sent Done false
9092 3290 Acked Read 8 Done Sent Done false
9093 3291 Acked Read 5 Done Sent Done false
9094 3292 Acked Write 8 Done Sent Done false
9095 3293 Acked Read 3 Done Sent Done false
9096 3294 Acked Read 2 Done Sent Done false
9097 3295 Acked Write 1 Done Sent Done false
9098 3296 Acked Read 2 Done Sent Done false
9099 3297 Acked Read 3 Done Sent Done false
9100 3298 Acked Write 5 Done Sent Done false
9101 3299 Acked Read 6 Done Sent Done false
9102 3300 Acked Write 7 Done Sent Done false
9103 3301 Acked Read 6 Done Sent Done false
9104 3302 Acked Write 1 Done Sent Done false
9105 3303 Acked Write 7 Done Sent Done false
9106 3304 Acked Read 5 Done Sent Done false
9107 3305 Acked Write 2 Done Sent Done false
9108 3306 Acked Read 3 Done Sent Done false
9109 3307 Acked Flush 0 Done Sent Done false
9110 3308 Acked Flush 0 Done Sent Done false
9111 3309 Acked Write 1 Done Sent Done false
9112 3310 Acked Read 2 Done Sent Done false
9113 3311 Acked Flush 0 Done Sent Done false
9114 3312 Acked Read 8 Done Sent Done false
9115 3313 Acked Write 9 Done Sent Done false
9116 3314 Acked Read 6 Done Sent Done false
9117 3315 Acked Write 5 Done Sent Done false
9118 3316 Acked Write 4 Done Sent Done false
9119 3317 Acked Write 1 Done Sent Done false
9120 3318 Acked Write 7 Done Sent Done false
9121 3319 Acked Read 8 Done Sent Done false
9122 3320 Acked Write 2 Done Sent Done false
9123 3321 Acked Read 9 Done Sent Done false
9124 3322 Acked Write 10 Done Sent Done false
9125 3323 Acked Write 7 Done Sent Done false
9126 3324 Acked Write 8 Done Sent Done false
9127 3325 Acked Read 6 Done Sent Done false
9128 3326 Acked Write 8 Done Sent Done false
9129 3327 Acked Flush 0 Done Sent Done false
9130 3328 Acked Write 2 Done Sent Done false
9131 3329 Acked Write 2 Done Sent Done false
9132 3330 Acked Read 3 Done Sent Done false
9133 3331 Acked Write 9 Done Sent Done false
9134 3332 Acked Write 8 Done Sent Done false
9135 3333 Acked Write 5 Done Sent Done false
9136 3334 Acked Read 5 Done Sent Done false
9137 3335 Acked Write 7 Done Sent Done false
9138 3336 Acked Write 5 Done Sent Done false
9139 3337 Acked Read 9 Done Sent Done false
9140 3338 Acked Read 10 Done Sent Done false
9141 3339 Acked Read 3 Done Sent Done false
9142 3340 Acked Write 4 Done Sent Done false
9143 3341 Acked Flush 0 Done Sent Done false
9144 3342 Acked Read 5 Done Sent Done false
9145 3343 Acked Read 8 Done Sent Done false
9146 3344 Acked Write 4 Done Sent Done false
9147 3345 Acked Write 10 Done Sent Done false
9148 3346 Acked Read 1 Done Sent Done false
9149 3347 Acked Write 1 Done Sent Done false
9150 3348 Acked Write 2 Done Sent Done false
9151 3349 Acked Flush 0 Done Sent Done false
9152 3350 Acked Write 1 Done Sent Done false
9153 3351 Acked Write 7 Done Sent Done false
9154 3352 Acked Read 5 Done Sent Done false
9155 3353 Acked Read 2 Done Sent Done false
9156 3354 Acked Read 7 Done Sent Done false
9157 3355 Acked Write 5 Done Sent Done false
9158 3356 Acked Write 8 Done Sent Done false
9159 3357 Acked Write 4 Done Sent Done false
9160 3358 Acked Write 2 Done Sent Done false
9161 3359 Acked Read 8 Done Sent Done false
9162 3360 Acked Read 3 Done Sent Done false
9163 3361 Acked Read 7 Done Sent Done false
9164 3362 Acked Write 7 Done Sent Done false
9165 3363 Acked Write 9 Done Sent Done false
9166 3364 Acked Flush 0 Done Sent Done false
9167 3365 Acked Read 10 Done Sent Done false
9168 3366 Acked Read 5 Done Sent Done false
9169 3367 Acked Write 8 Done Sent Done false
9170 3368 Acked Read 3 Done Sent Done false
9171 3369 Acked Write 6 Done Sent Done false
9172 3370 Acked Flush 0 Done Sent Done false
9173 3371 Acked Write 9 Done Sent Done false
9174 3372 Acked Write 6 Done Sent Done false
9175 3373 Acked Write 9 Done Sent Done false
9176 3374 Acked Write 9 Done Sent Done false
9177 3375 Acked Read 7 Done Sent Done false
9178 3376 Acked Write 7 Done Sent Done false
9179 3377 Acked Flush 0 Done Sent Done false
9180 3378 Acked Read 7 Done Sent Done false
9181 3379 Acked Read 9 Done Sent Done false
9182 3380 Acked Read 8 Done Sent Done false
9183 3381 Acked Read 5 Done Sent Done false
9184 3382 Acked Read 4 Done Sent Done false
9185 3383 Acked Flush 0 Done Sent Done false
9186 3384 Acked Read 9 Done Sent Done false
9187 3385 Acked Write 7 Done Sent Done false
9188 3386 Acked Write 3 Done Sent Done false
9189 3387 Acked Write 2 Done Sent Done false
9190 3388 Acked Read 9 Done Sent Done false
9191 3389 Acked Read 8 Done Sent Done false
9192 3390 Acked Write 1 Done Sent Done false
9193 3391 Acked Read 4 Done Sent Done false
9194 3392 Acked Write 9 Done Sent Done false
9195 3393 Acked Write 6 Done Sent Done false
9196 3394 Acked Write 4 Done Sent Done false
9197 3395 Acked Read 10 Done Sent Done false
9198 3396 Acked Write 10 Done Sent Done false
9199 3397 Acked Write 9 Done Sent Done false
9200 3398 Acked Write 6 Done Sent Done false
9201 3399 Acked Write 6 Done Sent Done false
9202 3400 Acked Flush 0 Done Sent Done false
9203 3401 Acked Read 3 Done Sent Done false
9204 3402 Acked Write 10 Done Sent Done false
9205 3403 Acked Write 8 Done Sent Done false
9206 3404 Acked Write 1 Done Sent Done false
9207 3405 Acked Write 8 Done Sent Done false
9208 3406 Acked Read 7 Done Sent Done false
9209 3407 Acked Write 6 Done Sent Done false
9210 3408 Acked Read 5 Done Sent Done false
9211 3409 Acked Write 9 Done Sent Done false
9212 3410 Acked Read 4 Done Sent Done false
9213 3411 Acked Read 5 Done Sent Done false
9214 3412 Acked Write 2 Done Sent Done false
9215 3413 Acked Write 3 Done Sent Done false
9216 3414 Acked Flush 0 Done Sent Done false
9217 STATES DS:0 DS:1 DS:2 TOTAL
9218 Sent 0 302 0 302
9219 Done 302 0 302 604
9220 Skipped 0 0 0 0
9221 Error 0 0 0 0
9222 Last Flush: Some(JobId(3414)) Some(JobId(3112)) Some(JobId(3414))
9223 Downstairs last five completed: 3112 3111 3110 3109 3108
9224 Upstairs last five completed: 3414 3413 3412 3411 3410
9225 CLIENT: Up:0 ds:302 act:2
92262025-01-23T18:49:48.693ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
92272025-01-23T18:49:48.693ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48867) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
92282025-01-23T18:49:48.694ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
92292025-01-23T18:49:48.694ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
92302025-01-23T18:49:48.694ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
92312025-01-23T18:49:48.694ZINFOcrucible: send last flush ID to this DS: Some(JobId(3112)) = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
92322025-01-23T18:49:48.694ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
92332025-01-23T18:49:48.694ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(3112)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
92342025-01-23T18:49:48.694ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(3112)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
9235 ----------------------------------------------------------------
9236 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9237 Downstairs last five completed: 3414 3413 3412 3411 3410
9238 Upstairs last five completed: 3414 3413 3412 3411 3410
9239 CLIENT: Up:0 ds:0 act:3
9240 Replay: All jobs finished
9241 Main task has work to do, go find it
9242 got dsc Stop(0)
9243 stop 0
9244 [8810] Got stop action so:false kr:false
9245 Jan 23 18:49:50.937 INFO request completed, latency_us: 319, response_code: 204, uri: /stop/cid/0, method: POST, req_id: aebdf642-91c0-4582-aecc-88d250aaf385, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
92462025-01-23T18:49:50.937ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
92472025-01-23T18:49:50.937ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
92482025-01-23T18:49:50.938ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
9249 Jan 23 18:49:50.938 INFO request completed, latency_us: 122, response_code: 200, uri: /state/cid/0, method: GET, req_id: febd6890-05c1-449c-8487-ba63f686b560, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
92502025-01-23T18:49:50.938ZINFOcrucible: client task is exiting = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
9251 [8810] Exited with: ExitStatus(unix_wait_status(9))
9252 [8810][0] reports Exit
9253 Jan 23 18:49:54.940 INFO request completed, latency_us: 202, response_code: 200, uri: /state/cid/0, method: GET, req_id: 57164fad-14f1-4c59-820b-d665b946a574, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
9254 001/300 Read block 2102 len 20480
9255 002/300 Write block 2562 len 4096 data: 4
9256 003/300 Read block 3013 len 4096
9257 004/300 Write block 942 len 20480 data: 3 3 3 3 2
9258 005/300 Read block 4180 len 4096
9259 006/300 Write block 4644 len 40960 data: 3 3 3 3 2 2 2 2 2 3
9260 007/300 Read block 3751 len 8192
9261 008/300 Flush
9262 009/300 Read block 2497 len 32768
9263 010/300 Write block 4081 len 36864 data: 5 5 3 3 2 2 2 2 2
9264 011/300 Flush
9265 012/300 Write block 4182 len 12288 data: 2 2 2
9266 013/300 Write block 4447 len 40960 data: 3 4 4 4 4 3 2 2 2 3
9267 014/300 Write block 4375 len 4096 data: 5
9268 015/300 Read block 2572 len 28672
9269 016/300 Read block 4177 len 24576
9270 017/300 Write block 4322 len 40960 data: 4 4 4 4 5 5 5 5 5 3
9271 018/300 Write block 3233 len 8192 data: 6 6
9272 019/300 Write block 4286 len 40960 data: 3 3 3 4 4 4 3 3 2 2
9273 020/300 Write block 541 len 12288 data: 3 4 3
9274 021/300 Read block 235 len 8192
9275 022/300 Read block 1469 len 4096
9276 023/300 Read block 3250 len 12288
9277 024/300 Read block 2126 len 16384
9278 025/300 Write block 2290 len 16384 data: 3 3 3 3
9279 026/300 Flush
9280 027/300 Write block 3337 len 36864 data: 2 2 2 2 2 2 3 3 3
9281 028/300 Write block 1339 len 12288 data: 3 3 3
9282 029/300 Write block 1641 len 40960 data: 2 2 2 3 3 3 3 3 3 3
9283 030/300 Flush
9284 031/300 Write block 2491 len 28672 data: 2 2 2 3 3 3 4
9285 032/300 Write block 1455 len 4096 data: 2
9286 033/300 Write block 1251 len 16384 data: 3 3 3 2
9287 034/300 Read block 92 len 24576
9288 035/300 Write block 1129 len 32768 data: 2 2 3 5 6 7 6 6
9289 036/300 Read block 3251 len 36864
9290 037/300 Write block 2988 len 4096 data: 2
9291 038/300 Read block 1196 len 20480
9292 039/300 Read block 1834 len 4096
9293 040/300 Read block 3396 len 40960
9294 041/300 Flush
9295 042/300 Read block 2611 len 4096
9296 043/300 Write block 3347 len 24576 data: 3 3 5 4 4 4
9297 044/300 Read block 1788 len 32768
9298 045/300 Write block 4148 len 16384 data: 7 7 7 6
9299 046/300 Read block 3912 len 4096
9300 047/300 Write block 1102 len 32768 data: 4 4 3 2 3 2 3 2
9301 048/300 Write block 615 len 32768 data: 2 2 2 2 2 2 2 2
9302 049/300 Read block 4704 len 20480
9303 050/300 Write block 4222 len 4096 data: 4
9304 051/300 Write block 4198 len 8192 data: 3 3
9305 052/300 Flush
9306 053/300 Write block 4744 len 28672 data: 2 2 2 2 2 2 3
9307 054/300 Write block 3219 len 28672 data: 2 2 2 2 2 2 2
9308 055/300 Write block 48 len 4096 data: 3
9309 056/300 Read block 4786 len 16384
9310 057/300 Read block 1780 len 8192
9311 058/300 Write block 2156 len 32768 data: 4 3 3 2 2 2 2 2
9312 059/300 Read block 2732 len 28672
9313 060/300 Write block 4740 len 36864 data: 2 2 2 2 3 3 3 3 3
9314 061/300 Write block 3778 len 32768 data: 3 2 2 2 2 2 2 2
9315 062/300 Read block 2952 len 40960
9316 063/300 Write block 673 len 4096 data: 3
9317 064/300 Read block 4072 len 40960
9318 065/300 Write block 1036 len 24576 data: 2 2 2 2 2 2
9319 066/300 Read block 3251 len 36864
9320 067/300 Write block 2303 len 16384 data: 3 2 2 2
9321 068/300 Read block 3148 len 28672
9322 069/300 Read block 2458 len 28672
9323 070/300 Read block 3181 len 32768
9324 071/300 Read block 3319 len 36864
9325 072/300 Read block 4320 len 32768
9326 073/300 Read block 1611 len 40960
9327 074/300 Read block 3405 len 12288
9328 075/300 Read block 4924 len 20480
9329 076/300 Read block 3548 len 32768
9330 077/300 Read block 850 len 36864
9331 078/300 Write block 3691 len 12288 data: 3 3 4
9332 079/300 Write block 3532 len 36864 data: 2 2 4 4 4 4 4 3 3
9333 080/300 Read block 2741 len 8192
9334 081/300 Read block 4230 len 20480
9335 082/300 Write block 4704 len 40960 data: 3 2 2 2 3 3 3 3 3 3
9336 083/300 Write block 3915 len 28672 data: 4 3 3 3 3 3 2
9337 084/300 Read block 4120 len 36864
9338 085/300 Write block 2855 len 32768 data: 3 3 3 4 3 3 3 3
9339 086/300 Write block 2066 len 16384 data: 2 2 2 3
9340 087/300 Read block 2329 len 8192
9341 088/300 Read block 3335 len 4096
9342 089/300 Write block 998 len 32768 data: 3 3 3 3 3 2 2 2
9343 090/300 Flush
9344 091/300 Read block 4245 len 8192
9345 092/300 Write block 1378 len 16384 data: 2 2 2 2
9346 093/300 Read block 3098 len 36864
9347 094/300 Flush
9348 095/300 Read block 3421 len 4096
9349 096/300 Read block 3096 len 40960
9350 097/300 Read block 2708 len 28672
9351 098/300 Flush
9352 099/300 Read block 2681 len 28672
9353 100/300 Write block 2389 len 24576 data: 2 3 4 4 3 4
9354 101/300 Write block 1736 len 20480 data: 3 4 4 3 3
9355 102/300 Read block 4171 len 8192
9356 103/300 Read block 2814 len 24576
9357 104/300 Flush
9358 105/300 Read block 3098 len 40960
9359 106/300 Flush
9360 107/300 Read block 3495 len 32768
9361 108/300 Write block 3494 len 12288 data: 2 3 3
9362 109/300 Read block 2207 len 28672
9363 110/300 Read block 917 len 20480
9364 111/300 Read block 462 len 40960
9365 112/300 Read block 1357 len 4096
9366 113/300 Read block 517 len 24576
9367 114/300 Write block 155 len 4096 data: 2
9368 115/300 Write block 1468 len 8192 data: 3 3
9369 116/300 Read block 3186 len 16384
9370 117/300 Read block 2306 len 24576
9371 118/300 Read block 1966 len 8192
9372 119/300 Read block 2808 len 20480
9373 120/300 Read block 3386 len 8192
9374 121/300 Write block 1934 len 36864 data: 3 3 4 3 3 4 4 5 5
9375 122/300 Read block 3801 len 20480
9376 123/300 Write block 1931 len 36864 data: 2 2 3 4 4 5 4 4 5
9377 124/300 Read block 1442 len 28672
9378 125/300 Write block 4229 len 24576 data: 3 3 3 3 3 3
9379 126/300 Read block 803 len 16384
9380 127/300 Write block 3161 len 40960 data: 3 4 5 4 4 4 5 4 4 3
9381 128/300 Read block 2515 len 32768
9382 129/300 Write block 2486 len 28672 data: 2 2 2 2 2 3 3
9383 130/300 Read block 131 len 40960
9384 131/300 Read block 374 len 40960
9385 132/300 Write block 782 len 28672 data: 6 6 5 4 5 5 6
9386 133/300 Write block 1621 len 36864 data: 4 4 4 3 3 2 2 2 2
9387 134/300 Write block 305 len 40960 data: 2 3 3 3 4 4 4 4 4 3
9388 135/300 Write block 3610 len 24576 data: 3 3 3 2 3 3
9389 136/300 Flush
9390 137/300 Write block 709 len 4096 data: 3
9391 138/300 Read block 4814 len 24576
9392 139/300 Flush
9393 140/300 Read block 2455 len 40960
9394 141/300 Write block 3744 len 40960 data: 2 2 2 2 2 2 2 2 2 3
9395 142/300 Write block 876 len 40960 data: 3 2 2 2 2 2 2 2 2 2
9396 143/300 Write block 3518 len 4096 data: 3
9397 144/300 Read block 4574 len 16384
9398 145/300 Write block 3501 len 8192 data: 3 2
9399 146/300 Read block 1286 len 28672
9400 147/300 Read block 1565 len 32768
9401 148/300 Read block 973 len 28672
9402 149/300 Read block 4736 len 40960
9403 150/300 Read block 514 len 32768
9404 151/300 Read block 1721 len 28672
9405 152/300 Write block 2199 len 40960 data: 2 2 2 2 4 3 4 5 5 5
9406 153/300 Flush
9407 154/300 Read block 4168 len 20480
9408 155/300 Read block 4284 len 24576
9409 156/300 Read block 163 len 36864
9410 157/300 Read block 2043 len 20480
9411 158/300 Read block 187 len 16384
9412 159/300 Write block 3950 len 36864 data: 4 4 3 4 4 4 4 4 3
9413 160/300 Read block 3149 len 20480
9414 161/300 Write block 4071 len 28672 data: 4 5 5 5 3 4 4
9415 162/300 Write block 2891 len 28672 data: 4 4 4 4 4 3 3
9416 163/300 Read block 640 len 40960
9417 164/300 Read block 2791 len 20480
9418 165/300 Flush
9419 166/300 Read block 344 len 32768
9420 167/300 Read block 4605 len 12288
9421 168/300 Read block 3719 len 24576
9422 169/300 Read block 1884 len 12288
9423 170/300 Read block 1394 len 36864
9424 171/300 Write block 1384 len 4096 data: 2
9425 172/300 Read block 440 len 40960
9426 173/300 Read block 824 len 16384
9427 174/300 Read block 3035 len 16384
9428 175/300 Write block 2932 len 36864 data: 4 4 4 3 2 3 3 3 3
9429 176/300 Write block 822 len 12288 data: 5 5 3
9430 177/300 Read block 566 len 36864
9431 178/300 Write block 1021 len 28672 data: 6 4 4 3 2 3 3
9432 179/300 Read block 1322 len 36864
9433 180/300 Write block 1009 len 4096 data: 4
9434 181/300 Read block 4902 len 16384
9435 182/300 Read block 1685 len 40960
9436 183/300 Read block 4903 len 16384
9437 184/300 Write block 3793 len 28672 data: 3 3 3 3 2 3 3
9438 185/300 Read block 3163 len 24576
9439 186/300 Read block 508 len 12288
9440 187/300 Flush
9441 188/300 Read block 224 len 20480
9442 189/300 Flush
9443 190/300 Read block 1352 len 12288
9444 191/300 Read block 1625 len 28672
9445 192/300 Write block 2731 len 40960 data: 2 2 2 2 2 2 2 2 2 2
9446 193/300 Write block 3985 len 4096 data: 2
9447 194/300 Read block 1470 len 16384
9448 195/300 Read block 1917 len 40960
9449 196/300 Write block 4831 len 40960 data: 3 3 4 4 4 4 3 2 2 2
9450 197/300 Read block 1808 len 40960
9451 198/300 Flush
9452 199/300 Read block 3518 len 40960
9453 200/300 Read block 1065 len 28672
9454 201/300 Write block 3149 len 8192 data: 2 2
9455 202/300 Read block 321 len 36864
9456 203/300 Read block 1818 len 20480
9457 204/300 Write block 688 len 40960 data: 2 2 2 2 2 2 2 2 2 2
9458 205/300 Write block 2846 len 20480 data: 3 3 3 3 4
9459 206/300 Read block 1345 len 24576
9460 207/300 Read block 842 len 8192
9461 208/300 Flush
9462 209/300 Write block 4870 len 24576 data: 3 4 5 5 5 5
9463 210/300 Write block 4046 len 28672 data: 3 3 3 3 2 2 2
9464 211/300 Read block 907 len 20480
9465 212/300 Read block 2156 len 4096
9466 213/300 Read block 812 len 8192
9467 214/300 Write block 4594 len 24576 data: 3 3 3 3 3 3
9468 215/300 Read block 2517 len 4096
9469 216/300 Write block 1929 len 24576 data: 2 2 3 3 4 5
9470 217/300 Write block 1214 len 36864 data: 3 3 4 4 4 4 5 5 5
9471 218/300 Flush
9472 219/300 Write block 2196 len 28672 data: 2 2 2 3 3 3 3
9473 220/300 Read block 3765 len 28672
9474 221/300 Write block 185 len 28672 data: 3 3 3 2 2 2 2
9475 222/300 Write block 467 len 40960 data: 3 3 4 3 3 3 3 2 2 2
9476 223/300 Read block 4656 len 16384
9477 224/300 Write block 3806 len 4096 data: 2
9478 225/300 Write block 1062 len 4096 data: 4
9479 226/300 Read block 2264 len 36864
9480 227/300 Read block 2704 len 28672
9481 228/300 Read block 1748 len 32768
9482 229/300 Write block 3986 len 16384 data: 2 2 3 3
9483 230/300 Read block 1167 len 24576
9484 231/300 Read block 4146 len 4096
9485 232/300 Write block 1036 len 12288 data: 3 3 3
9486 233/300 Read block 1581 len 32768
9487 234/300 Write block 112 len 32768 data: 3 4 4 4 3 3 3 2
9488 235/300 Write block 2618 len 36864 data: 4 4 4 4 4 4 3 3 3
9489 236/300 Flush
9490 237/300 Flush
9491 238/300 Read block 2579 len 12288
9492 239/300 Write block 3491 len 40960 data: 2 2 2 3 4 4 3 3 4 3
9493 240/300 Write block 1200 len 28672 data: 4 4 4 3 3 2 2
9494 241/300 Write block 149 len 4096 data: 3
9495 242/300 Read block 1681 len 28672
9496 243/300 Read block 4799 len 8192
9497 244/300 Read block 909 len 32768
9498 245/300 Read block 1866 len 20480
9499 246/300 Read block 2359 len 32768
9500 247/300 Write block 1238 len 8192 data: 3 3
9501 248/300 Write block 393 len 28672 data: 2 2 2 3 3 3 3
9502 249/300 Write block 4664 len 36864 data: 5 4 4 2 2 2 2 2 2
9503 250/300 Read block 2631 len 12288
9504 251/300 Write block 778 len 16384 data: 5 5 6 5
9505 252/300 Read block 858 len 16384
9506 253/300 Read block 3378 len 8192
9507 254/300 Write block 1395 len 28672 data: 4 4 3 2 2 2 2
9508 255/300 Write block 3763 len 8192 data: 3 3
9509 256/300 Write block 3130 len 36864 data: 2 2 2 2 3 3 3 4 4
9510 257/300 Read block 716 len 24576
9511 258/300 Read block 2685 len 24576
9512 259/300 Read block 2711 len 4096
9513 260/300 Write block 3418 len 32768 data: 4 3 3 3 4 5 4 4
9514 261/300 Write block 4711 len 16384 data: 4 4 4 4
9515 262/300 Read block 534 len 36864
9516 263/300 Read block 4041 len 12288
9517 264/300 Read block 1448 len 24576
9518 265/300 Write block 1523 len 28672 data: 3 3 3 4 5 5 4
9519 266/300 Read block 404 len 32768
9520 267/300 Read block 1928 len 20480
9521 268/300 Write block 2490 len 8192 data: 3 4
9522 269/300 Write block 2849 len 36864 data: 4 5 4 4 5 4 4 4 4
9523 270/300 Read block 3649 len 8192
9524 271/300 Write block 4721 len 16384 data: 3 3 3 3
9525 272/300 Write block 3421 len 20480 data: 4 5 6 5 5
9526 273/300 Read block 3736 len 32768
9527 274/300 Write block 558 len 12288 data: 3 3 3
9528 275/300 Write block 1888 len 8192 data: 3 3
9529 276/300 Read block 201 len 16384
9530 277/300 Read block 3388 len 24576
9531 278/300 Read block 4634 len 40960
9532 279/300 Read block 1071 len 20480
9533 280/300 Read block 2130 len 36864
9534 281/300 Read block 2234 len 20480
9535 282/300 Write block 1964 len 20480 data: 2 2 3 3 4
9536 283/300 Read block 1699 len 8192
9537 284/300 Read block 1439 len 12288
9538 285/300 Write block 4355 len 32768 data: 4 4 4 3 3 3 3 3
9539 286/300 Read block 913 len 4096
9540 287/300 Read block 1771 len 20480
9541 288/300 Write block 519 len 16384 data: 2 2 2 2
9542 289/300 Write block 2545 len 24576 data: 4 3 2 2 2 2
9543 290/300 Flush
9544 291/300 Write block 2017 len 8192 data: 3 4
9545 292/300 Read block 4940 len 40960
9546 293/300 Read block 4692 len 12288
9547 294/300 Read block 1721 len 8192
9548 295/300 Read block 1224 len 40960
9549 296/300 Write block 2927 len 36864 data: 4 4 4 5 4 5 5 5 4
9550 297/300 Write block 1895 len 16384 data: 3 3 3 4
9551 298/300 Read block 2130 len 12288
9552 299/300 Read block 4414 len 24576
9553 300/300 Read block 2075 len 36864
9554 301/300 Write block 3105 len 12288 data: 4 4 4
9555 Main task has work to do, go find it
9556 got dsc Start(0)
9557 start 0
9558 Jan 23 18:49:55.192 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/0, method: POST, req_id: f6d3f396-2f3b-41e8-a097-fb017dc3bd65, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
9559 Replay: started 0, returned:Ok(())
9560 ----------------------------------------------------------------
9561 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9562 States: Offline Active Active
9563 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9564 3415 Acked Read 5 Sent Done Done false
9565 3416 Acked Write 1 Sent Done Done false
9566 3417 Acked Read 1 Sent Done Done false
9567 3418 Acked Write 5 Sent Done Done false
9568 3419 Acked Read 1 Sent Done Done false
9569 3420 Acked Write 10 Sent Done Done false
9570 3421 Acked Read 2 Sent Done Done false
9571 3422 Acked Flush 0 Sent Done Done false
9572 3423 Acked Read 8 Sent Done Done false
9573 3424 Acked Write 9 Sent Done Done false
9574 3425 Acked Flush 0 Sent Done Done false
9575 3426 Acked Write 3 Sent Done Done false
9576 3427 Acked Write 10 Sent Done Done false
9577 3428 Acked Write 1 Sent Done Done false
9578 3429 Acked Read 7 Sent Done Done false
9579 3430 Acked Read 6 Sent Done Done false
9580 3431 Acked Write 10 Sent Done Done false
9581 3432 Acked Write 2 Sent Done Done false
9582 3433 Acked Write 10 Sent Done Done false
9583 3434 Acked Write 3 Sent Done Done false
9584 3435 Acked Read 2 Sent Done Done false
9585 3436 Acked Read 1 Sent Done Done false
9586 3437 Acked Read 3 Sent Done Done false
9587 3438 Acked Read 4 Sent Done Done false
9588 3439 Acked Write 4 Sent Done Done false
9589 3440 Acked Flush 0 Sent Done Done false
9590 3441 Acked Write 9 Sent Done Done false
9591 3442 Acked Write 3 Sent Done Done false
9592 3443 Acked Write 10 Sent Done Done false
9593 3444 Acked Flush 0 Sent Done Done false
9594 3445 Acked Write 7 Sent Done Done false
9595 3446 Acked Write 1 Sent Done Done false
9596 3447 Acked Write 4 Sent Done Done false
9597 3448 Acked Read 6 Sent Done Done false
9598 3449 Acked Write 8 Sent Done Done false
9599 3450 Acked Read 9 Sent Done Done false
9600 3451 Acked Write 1 Sent Done Done false
9601 3452 Acked Read 5 Sent Done Done false
9602 3453 Acked Read 1 Sent Done Done false
9603 3454 Acked Read 10 Sent Done Done false
9604 3455 Acked Flush 0 Sent Done Done false
9605 3456 Acked Read 1 Sent Done Done false
9606 3457 Acked Write 6 Sent Done Done false
9607 3458 Acked Read 8 Sent Done Done false
9608 3459 Acked Write 4 Sent Done Done false
9609 3460 Acked Read 1 Sent Done Done false
9610 3461 Acked Write 8 Sent Done Done false
9611 3462 Acked Write 8 Sent Done Done false
9612 3463 Acked Read 5 Sent Done Done false
9613 3464 Acked Write 1 Sent Done Done false
9614 3465 Acked Write 2 Sent Done Done false
9615 3466 Acked Flush 0 Sent Done Done false
9616 3467 Acked Write 7 Sent Done Done false
9617 3468 Acked Write 7 Sent Done Done false
9618 3469 Acked Write 1 Sent Done Done false
9619 3470 Acked Read 4 Sent Done Done false
9620 3471 Acked Read 2 Sent Done Done false
9621 3472 Acked Write 8 Sent Done Done false
9622 3473 Acked Read 7 Sent Done Done false
9623 3474 Acked Write 9 Sent Done Done false
9624 3475 Acked Write 8 Sent Done Done false
9625 3476 Acked Read 10 Sent Done Done false
9626 3477 Acked Write 1 Sent Done Done false
9627 3478 Acked Read 10 Sent Done Done false
9628 3479 Acked Write 6 Sent Done Done false
9629 3480 Acked Read 9 Sent Done Done false
9630 3481 Acked Write 4 Sent Done Done false
9631 3482 Acked Read 7 Sent Done Done false
9632 3483 Acked Read 7 Sent Done Done false
9633 3484 Acked Read 8 Sent Done Done false
9634 3485 Acked Read 9 Sent Done Done false
9635 3486 Acked Read 8 Sent Done Done false
9636 3487 Acked Read 10 Sent Done Done false
9637 3488 Acked Read 3 Sent Done Done false
9638 3489 Acked Read 5 Sent Done Done false
9639 3490 Acked Read 8 Sent Done Done false
9640 3491 Acked Read 9 Sent Done Done false
9641 3492 Acked Write 3 Sent Done Done false
9642 3493 Acked Write 9 Sent Done Done false
9643 3494 Acked Read 2 Sent Done Done false
9644 3495 Acked Read 5 Sent Done Done false
9645 3496 Acked Write 10 Sent Done Done false
9646 3497 Acked Write 7 Sent Done Done false
9647 3498 Acked Read 9 Sent Done Done false
9648 3499 Acked Write 8 Sent Done Done false
9649 3500 Acked Write 4 Sent Done Done false
9650 3501 Acked Read 2 Sent Done Done false
9651 3502 Acked Read 1 Sent Done Done false
9652 3503 Acked Write 8 Sent Done Done false
9653 3504 Acked Flush 0 Sent Done Done false
9654 3505 Acked Read 2 Sent Done Done false
9655 3506 Acked Write 4 Sent Done Done false
9656 3507 Acked Read 9 Sent Done Done false
9657 3508 Acked Flush 0 Sent Done Done false
9658 3509 Acked Read 1 Sent Done Done false
9659 3510 Acked Read 10 Sent Done Done false
9660 3511 Acked Read 7 Sent Done Done false
9661 3512 Acked Flush 0 Sent Done Done false
9662 3513 Acked Read 7 Sent Done Done false
9663 3514 Acked Write 6 Sent Done Done false
9664 3515 Acked Write 5 Sent Done Done false
9665 3516 Acked Read 2 Sent Done Done false
9666 3517 Acked Read 6 Sent Done Done false
9667 3518 Acked Flush 0 Sent Done Done false
9668 3519 Acked Read 10 Sent Done Done false
9669 3520 Acked Flush 0 Sent Done Done false
9670 3521 Acked Read 8 Sent Done Done false
9671 3522 Acked Write 3 Sent Done Done false
9672 3523 Acked Read 7 Sent Done Done false
9673 3524 Acked Read 5 Sent Done Done false
9674 3525 Acked Read 10 Sent Done Done false
9675 3526 Acked Read 1 Sent Done Done false
9676 3527 Acked Read 6 Sent Done Done false
9677 3528 Acked Write 1 Sent Done Done false
9678 3529 Acked Write 2 Sent Done Done false
9679 3530 Acked Read 4 Sent Done Done false
9680 3531 Acked Read 6 Sent Done Done false
9681 3532 Acked Read 2 Sent Done Done false
9682 3533 Acked Read 5 Sent Done Done false
9683 3534 Acked Read 2 Sent Done Done false
9684 3535 Acked Write 9 Sent Done Done false
9685 3536 Acked Read 5 Sent Done Done false
9686 3537 Acked Write 9 Sent Done Done false
9687 3538 Acked Read 7 Sent Done Done false
9688 3539 Acked Write 6 Sent Done Done false
9689 3540 Acked Read 4 Sent Done Done false
9690 3541 Acked Write 10 Sent Done Done false
9691 3542 Acked Read 8 Sent Done Done false
9692 3543 Acked Write 7 Sent Done Done false
9693 3544 Acked Read 10 Sent Done Done false
9694 3545 Acked Read 10 Sent Done Done false
9695 3546 Acked Write 7 Sent Done Done false
9696 3547 Acked Write 9 Sent Done Done false
9697 3548 Acked Write 10 Sent Done Done false
9698 3549 Acked Write 6 Sent Done Done false
9699 3550 Acked Flush 0 Sent Done Done false
9700 3551 Acked Write 1 Sent Done Done false
9701 3552 Acked Read 6 Sent Done Done false
9702 3553 Acked Flush 0 Sent Done Done false
9703 3554 Acked Read 10 Sent Done Done false
9704 3555 Acked Write 10 Sent Done Done false
9705 3556 Acked Write 10 Sent Done Done false
9706 3557 Acked Write 1 Sent Done Done false
9707 3558 Acked Read 4 Sent Done Done false
9708 3559 Acked Write 2 Sent Done Done false
9709 3560 Acked Read 7 Sent Done Done false
9710 3561 Acked Read 8 Sent Done Done false
9711 3562 Acked Read 7 Sent Done Done false
9712 3563 Acked Read 10 Sent Done Done false
9713 3564 Acked Read 8 Sent Done Done false
9714 3565 Acked Read 7 Sent Done Done false
9715 3566 Acked Write 10 Sent Done Done false
9716 3567 Acked Flush 0 Sent Done Done false
9717 3568 Acked Read 5 Sent Done Done false
9718 3569 Acked Read 6 Sent Done Done false
9719 3570 Acked Read 9 Sent Done Done false
9720 3571 Acked Read 5 Sent Done Done false
9721 3572 Acked Read 4 Sent Done Done false
9722 3573 Acked Write 9 Sent Done Done false
9723 3574 Acked Read 5 Sent Done Done false
9724 3575 Acked Write 7 Sent Done Done false
9725 3576 Acked Write 7 Sent Done Done false
9726 3577 Acked Read 10 Sent Done Done false
9727 3578 Acked Read 5 Sent Done Done false
9728 3579 Acked Flush 0 Sent Done Done false
9729 3580 Acked Read 8 Sent Done Done false
9730 3581 Acked Read 3 Sent Done Done false
9731 3582 Acked Read 6 Sent Done Done false
9732 3583 Acked Read 3 Sent Done Done false
9733 3584 Acked Read 9 Sent Done Done false
9734 3585 Acked Write 1 Sent Done Done false
9735 3586 Acked Read 10 Sent Done Done false
9736 3587 Acked Read 4 Sent Done Done false
9737 3588 Acked Read 4 Sent Done Done false
9738 3589 Acked Write 9 Sent Done Done false
9739 3590 Acked Write 3 Sent Done Done false
9740 3591 Acked Read 9 Sent Done Done false
9741 3592 Acked Write 7 Sent Done Done false
9742 3593 Acked Read 9 Sent Done Done false
9743 3594 Acked Write 1 Sent Done Done false
9744 3595 Acked Read 4 Sent Done Done false
9745 3596 Acked Read 10 Sent Done Done false
9746 3597 Acked Read 4 Sent Done Done false
9747 3598 Acked Write 7 Sent Done Done false
9748 3599 Acked Read 6 Sent Done Done false
9749 3600 Acked Read 3 Sent Done Done false
9750 3601 Acked Flush 0 Sent Done Done false
9751 3602 Acked Read 5 Sent Done Done false
9752 3603 Acked Flush 0 Sent Done Done false
9753 3604 Acked Read 3 Sent Done Done false
9754 3605 Acked Read 7 Sent Done Done false
9755 3606 Acked Write 10 Sent Done Done false
9756 3607 Acked Write 1 Sent Done Done false
9757 3608 Acked Read 4 Sent Done Done false
9758 3609 Acked Read 10 Sent Done Done false
9759 3610 Acked Write 10 Sent Done Done false
9760 3611 Acked Read 10 Sent Done Done false
9761 3612 Acked Flush 0 Sent Done Done false
9762 3613 Acked Read 10 Sent Done Done false
9763 3614 Acked Read 7 Sent Done Done false
9764 3615 Acked Write 2 Sent Done Done false
9765 3616 Acked Read 9 Sent Done Done false
9766 3617 Acked Read 5 Sent Done Done false
9767 3618 Acked Write 10 Sent Done Done false
9768 3619 Acked Write 5 Sent Done Done false
9769 3620 Acked Read 6 Sent Done Done false
9770 3621 Acked Read 2 Sent Done Done false
9771 3622 Acked Flush 0 Sent Done Done false
9772 3623 Acked Write 6 Sent Done Done false
9773 3624 Acked Write 7 Sent Done Done false
9774 3625 Acked Read 5 Sent Done Done false
9775 3626 Acked Read 1 Sent Done Done false
9776 3627 Acked Read 2 Sent Done Done false
9777 3628 Acked Write 6 Sent Done Done false
9778 3629 Acked Read 1 Sent Done Done false
9779 3630 Acked Write 6 Sent Done Done false
9780 3631 Acked Write 9 Sent Done Done false
9781 3632 Acked Flush 0 Sent Done Done false
9782 3633 Acked Write 7 Sent Done Done false
9783 3634 Acked Read 7 Sent Done Done false
9784 3635 Acked Write 7 Sent Done Done false
9785 3636 Acked Write 10 Sent Done Done false
9786 3637 Acked Read 4 Sent Done Done false
9787 3638 Acked Write 1 Sent Done Done false
9788 3639 Acked Write 1 Sent Done Done false
9789 3640 Acked Read 9 Sent Done Done false
9790 3641 Acked Read 7 Sent Done Done false
9791 3642 Acked Read 8 Sent Done Done false
9792 3643 Acked Write 4 Sent Done Done false
9793 3644 Acked Read 6 Sent Done Done false
9794 3645 Acked Read 1 Sent Done Done false
9795 3646 Acked Write 3 Sent Done Done false
9796 3647 Acked Read 8 Sent Done Done false
9797 3648 Acked Write 8 Sent Done Done false
9798 3649 Acked Write 9 Sent Done Done false
9799 3650 Acked Flush 0 Sent Done Done false
9800 3651 Acked Flush 0 Sent Done Done false
9801 3652 Acked Read 3 Sent Done Done false
9802 3653 Acked Write 10 Sent Done Done false
9803 3654 Acked Write 7 Sent Done Done false
9804 3655 Acked Write 1 Sent Done Done false
9805 3656 Acked Read 7 Sent Done Done false
9806 3657 Acked Read 2 Sent Done Done false
9807 3658 Acked Read 8 Sent Done Done false
9808 3659 Acked Read 5 Sent Done Done false
9809 3660 Acked Read 8 Sent Done Done false
9810 3661 Acked Write 2 Sent Done Done false
9811 3662 Acked Write 7 Sent Done Done false
9812 3663 Acked Write 9 Sent Done Done false
9813 3664 Acked Read 3 Sent Done Done false
9814 3665 Acked Write 4 Sent Done Done false
9815 3666 Acked Read 4 Sent Done Done false
9816 3667 Acked Read 2 Sent Done Done false
9817 3668 Acked Write 7 Sent Done Done false
9818 3669 Acked Write 2 Sent Done Done false
9819 3670 Acked Write 9 Sent Done Done false
9820 3671 Acked Read 6 Sent Done Done false
9821 3672 Acked Read 6 Sent Done Done false
9822 3673 Acked Read 1 Sent Done Done false
9823 3674 Acked Write 8 Sent Done Done false
9824 3675 Acked Write 4 Sent Done Done false
9825 3676 Acked Read 9 Sent Done Done false
9826 3677 Acked Read 3 Sent Done Done false
9827 3678 Acked Read 6 Sent Done Done false
9828 3679 Acked Write 7 Sent Done Done false
9829 3680 Acked Read 8 Sent Done Done false
9830 3681 Acked Read 5 Sent Done Done false
9831 3682 Acked Write 2 Sent Done Done false
9832 3683 Acked Write 9 Sent Done Done false
9833 3684 Acked Read 2 Sent Done Done false
9834 3685 Acked Write 4 Sent Done Done false
9835 3686 Acked Write 5 Sent Done Done false
9836 3687 Acked Read 8 Sent Done Done false
9837 3688 Acked Write 3 Sent Done Done false
9838 3689 Acked Write 2 Sent Done Done false
9839 3690 Acked Read 4 Sent Done Done false
9840 3691 Acked Read 6 Sent Done Done false
9841 3692 Acked Read 10 Sent Done Done false
9842 3693 Acked Read 5 Sent Done Done false
9843 3694 Acked Read 9 Sent Done Done false
9844 3695 Acked Read 5 Sent Done Done false
9845 3696 Acked Write 5 Sent Done Done false
9846 3697 Acked Read 2 Sent Done Done false
9847 3698 Acked Read 3 Sent Done Done false
9848 3699 Acked Write 8 Sent Done Done false
9849 3700 Acked Read 1 Sent Done Done false
9850 3701 Acked Read 5 Sent Done Done false
9851 3702 Acked Write 4 Sent Done Done false
9852 3703 Acked Write 6 Sent Done Done false
9853 3704 Acked Flush 0 Sent Done Done false
9854 3705 Acked Write 2 Sent Done Done false
9855 3706 Acked Read 10 Sent Done Done false
9856 3707 Acked Read 3 Sent Done Done false
9857 3708 Acked Read 2 Sent Done Done false
9858 3709 Acked Read 10 Sent Done Done false
9859 3710 Acked Write 9 Sent Done Done false
9860 3711 Acked Write 4 Sent Done Done false
9861 3712 Acked Read 3 Sent Done Done false
9862 3713 Acked Read 6 Sent Done Done false
9863 3714 Acked Read 9 Sent Done Done false
9864 3715 Acked Write 3 Sent Done Done false
9865 STATES DS:0 DS:1 DS:2 TOTAL
9866 Sent 301 0 0 301
9867 Done 0 301 301 602
9868 Skipped 0 0 0 0
9869 Error 0 0 0 0
9870 Last Flush: Some(JobId(3414)) Some(JobId(3704)) Some(JobId(3704))
9871 Downstairs last five completed: 3414 3413 3412 3411 3410
9872 Upstairs last five completed: 3715 3714 3713 3712 3711
9873 CLIENT: Up:0 ds:301 act:2
9874 [8810] got start action
9875 [8810] I am going to restart
9876 Starting downstairs at port 8810
9877 Make output file at "/tmp/dsc/downstairs-8810.txt"
9878 [8810][0] reports Starting
9879 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1209)
9880 [8810][0] reports Running
9881 ----------------------------------------------------------------
9882 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
9883 States: Offline Active Active
9884 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9885 3415 Acked Read 5 Sent Done Done false
9886 3416 Acked Write 1 Sent Done Done false
9887 3417 Acked Read 1 Sent Done Done false
9888 3418 Acked Write 5 Sent Done Done false
9889 3419 Acked Read 1 Sent Done Done false
9890 3420 Acked Write 10 Sent Done Done false
9891 3421 Acked Read 2 Sent Done Done false
9892 3422 Acked Flush 0 Sent Done Done false
9893 3423 Acked Read 8 Sent Done Done false
9894 3424 Acked Write 9 Sent Done Done false
9895 3425 Acked Flush 0 Sent Done Done false
9896 3426 Acked Write 3 Sent Done Done false
9897 3427 Acked Write 10 Sent Done Done false
9898 3428 Acked Write 1 Sent Done Done false
9899 3429 Acked Read 7 Sent Done Done false
9900 3430 Acked Read 6 Sent Done Done false
9901 3431 Acked Write 10 Sent Done Done false
9902 3432 Acked Write 2 Sent Done Done false
9903 3433 Acked Write 10 Sent Done Done false
9904 3434 Acked Write 3 Sent Done Done false
9905 3435 Acked Read 2 Sent Done Done false
9906 3436 Acked Read 1 Sent Done Done false
9907 3437 Acked Read 3 Sent Done Done false
9908 3438 Acked Read 4 Sent Done Done false
9909 3439 Acked Write 4 Sent Done Done false
9910 3440 Acked Flush 0 Sent Done Done false
9911 3441 Acked Write 9 Sent Done Done false
9912 3442 Acked Write 3 Sent Done Done false
9913 3443 Acked Write 10 Sent Done Done false
9914 3444 Acked Flush 0 Sent Done Done false
9915 3445 Acked Write 7 Sent Done Done false
9916 3446 Acked Write 1 Sent Done Done false
9917 3447 Acked Write 4 Sent Done Done false
9918 3448 Acked Read 6 Sent Done Done false
9919 3449 Acked Write 8 Sent Done Done false
9920 3450 Acked Read 9 Sent Done Done false
9921 3451 Acked Write 1 Sent Done Done false
9922 3452 Acked Read 5 Sent Done Done false
9923 3453 Acked Read 1 Sent Done Done false
9924 3454 Acked Read 10 Sent Done Done false
9925 3455 Acked Flush 0 Sent Done Done false
9926 3456 Acked Read 1 Sent Done Done false
9927 3457 Acked Write 6 Sent Done Done false
9928 3458 Acked Read 8 Sent Done Done false
9929 3459 Acked Write 4 Sent Done Done false
9930 3460 Acked Read 1 Sent Done Done false
9931 3461 Acked Write 8 Sent Done Done false
9932 3462 Acked Write 8 Sent Done Done false
9933 3463 Acked Read 5 Sent Done Done false
9934 3464 Acked Write 1 Sent Done Done false
9935 3465 Acked Write 2 Sent Done Done false
9936 3466 Acked Flush 0 Sent Done Done false
9937 3467 Acked Write 7 Sent Done Done false
9938 3468 Acked Write 7 Sent Done Done false
9939 3469 Acked Write 1 Sent Done Done false
9940 3470 Acked Read 4 Sent Done Done false
9941 3471 Acked Read 2 Sent Done Done false
9942 3472 Acked Write 8 Sent Done Done false
9943 3473 Acked Read 7 Sent Done Done false
9944 3474 Acked Write 9 Sent Done Done false
9945 3475 Acked Write 8 Sent Done Done false
9946 3476 Acked Read 10 Sent Done Done false
9947 3477 Acked Write 1 Sent Done Done false
9948 3478 Acked Read 10 Sent Done Done false
9949 3479 Acked Write 6 Sent Done Done false
9950 3480 Acked Read 9 Sent Done Done false
9951 3481 Acked Write 4 Sent Done Done false
9952 3482 Acked Read 7 Sent Done Done false
9953 3483 Acked Read 7 Sent Done Done false
9954 3484 Acked Read 8 Sent Done Done false
9955 3485 Acked Read 9 Sent Done Done false
9956 3486 Acked Read 8 Sent Done Done false
9957 3487 Acked Read 10 Sent Done Done false
9958 3488 Acked Read 3 Sent Done Done false
9959 3489 Acked Read 5 Sent Done Done false
9960 3490 Acked Read 8 Sent Done Done false
9961 3491 Acked Read 9 Sent Done Done false
9962 3492 Acked Write 3 Sent Done Done false
9963 3493 Acked Write 9 Sent Done Done false
9964 3494 Acked Read 2 Sent Done Done false
9965 3495 Acked Read 5 Sent Done Done false
9966 3496 Acked Write 10 Sent Done Done false
9967 3497 Acked Write 7 Sent Done Done false
9968 3498 Acked Read 9 Sent Done Done false
9969 3499 Acked Write 8 Sent Done Done false
9970 3500 Acked Write 4 Sent Done Done false
9971 3501 Acked Read 2 Sent Done Done false
9972 3502 Acked Read 1 Sent Done Done false
9973 3503 Acked Write 8 Sent Done Done false
9974 3504 Acked Flush 0 Sent Done Done false
9975 3505 Acked Read 2 Sent Done Done false
9976 3506 Acked Write 4 Sent Done Done false
9977 3507 Acked Read 9 Sent Done Done false
9978 3508 Acked Flush 0 Sent Done Done false
9979 3509 Acked Read 1 Sent Done Done false
9980 3510 Acked Read 10 Sent Done Done false
9981 3511 Acked Read 7 Sent Done Done false
9982 3512 Acked Flush 0 Sent Done Done false
9983 3513 Acked Read 7 Sent Done Done false
9984 3514 Acked Write 6 Sent Done Done false
9985 3515 Acked Write 5 Sent Done Done false
9986 3516 Acked Read 2 Sent Done Done false
9987 3517 Acked Read 6 Sent Done Done false
9988 3518 Acked Flush 0 Sent Done Done false
9989 3519 Acked Read 10 Sent Done Done false
9990 3520 Acked Flush 0 Sent Done Done false
9991 3521 Acked Read 8 Sent Done Done false
9992 3522 Acked Write 3 Sent Done Done false
9993 3523 Acked Read 7 Sent Done Done false
9994 3524 Acked Read 5 Sent Done Done false
9995 3525 Acked Read 10 Sent Done Done false
9996 3526 Acked Read 1 Sent Done Done false
9997 3527 Acked Read 6 Sent Done Done false
9998 3528 Acked Write 1 Sent Done Done false
9999 3529 Acked Write 2 Sent Done Done false
10000 3530 Acked Read 4 Sent Done Done false
10001 3531 Acked Read 6 Sent Done Done false
10002 3532 Acked Read 2 Sent Done Done false
10003 3533 Acked Read 5 Sent Done Done false
10004 3534 Acked Read 2 Sent Done Done false
10005 3535 Acked Write 9 Sent Done Done false
10006 3536 Acked Read 5 Sent Done Done false
10007 3537 Acked Write 9 Sent Done Done false
10008 3538 Acked Read 7 Sent Done Done false
10009 3539 Acked Write 6 Sent Done Done false
10010 3540 Acked Read 4 Sent Done Done false
10011 3541 Acked Write 10 Sent Done Done false
10012 3542 Acked Read 8 Sent Done Done false
10013 3543 Acked Write 7 Sent Done Done false
10014 3544 Acked Read 10 Sent Done Done false
10015 3545 Acked Read 10 Sent Done Done false
10016 3546 Acked Write 7 Sent Done Done false
10017 3547 Acked Write 9 Sent Done Done false
10018 3548 Acked Write 10 Sent Done Done false
10019 3549 Acked Write 6 Sent Done Done false
10020 3550 Acked Flush 0 Sent Done Done false
10021 3551 Acked Write 1 Sent Done Done false
10022 3552 Acked Read 6 Sent Done Done false
10023 3553 Acked Flush 0 Sent Done Done false
10024 3554 Acked Read 10 Sent Done Done false
10025 3555 Acked Write 10 Sent Done Done false
10026 3556 Acked Write 10 Sent Done Done false
10027 3557 Acked Write 1 Sent Done Done false
10028 3558 Acked Read 4 Sent Done Done false
10029 3559 Acked Write 2 Sent Done Done false
10030 3560 Acked Read 7 Sent Done Done false
10031 3561 Acked Read 8 Sent Done Done false
10032 3562 Acked Read 7 Sent Done Done false
10033 3563 Acked Read 10 Sent Done Done false
10034 3564 Acked Read 8 Sent Done Done false
10035 3565 Acked Read 7 Sent Done Done false
10036 3566 Acked Write 10 Sent Done Done false
10037 3567 Acked Flush 0 Sent Done Done false
10038 3568 Acked Read 5 Sent Done Done false
10039 3569 Acked Read 6 Sent Done Done false
10040 3570 Acked Read 9 Sent Done Done false
10041 3571 Acked Read 5 Sent Done Done false
10042 3572 Acked Read 4 Sent Done Done false
10043 3573 Acked Write 9 Sent Done Done false
10044 3574 Acked Read 5 Sent Done Done false
10045 3575 Acked Write 7 Sent Done Done false
10046 3576 Acked Write 7 Sent Done Done false
10047 3577 Acked Read 10 Sent Done Done false
10048 3578 Acked Read 5 Sent Done Done false
10049 3579 Acked Flush 0 Sent Done Done false
10050 3580 Acked Read 8 Sent Done Done false
10051 3581 Acked Read 3 Sent Done Done false
10052 3582 Acked Read 6 Sent Done Done false
10053 3583 Acked Read 3 Sent Done Done false
10054 3584 Acked Read 9 Sent Done Done false
10055 3585 Acked Write 1 Sent Done Done false
10056 3586 Acked Read 10 Sent Done Done false
10057 3587 Acked Read 4 Sent Done Done false
10058 3588 Acked Read 4 Sent Done Done false
10059 3589 Acked Write 9 Sent Done Done false
10060 3590 Acked Write 3 Sent Done Done false
10061 3591 Acked Read 9 Sent Done Done false
10062 3592 Acked Write 7 Sent Done Done false
10063 3593 Acked Read 9 Sent Done Done false
10064 3594 Acked Write 1 Sent Done Done false
10065 3595 Acked Read 4 Sent Done Done false
10066 3596 Acked Read 10 Sent Done Done false
10067 3597 Acked Read 4 Sent Done Done false
10068 3598 Acked Write 7 Sent Done Done false
10069 3599 Acked Read 6 Sent Done Done false
10070 3600 Acked Read 3 Sent Done Done false
10071 3601 Acked Flush 0 Sent Done Done false
10072 3602 Acked Read 5 Sent Done Done false
10073 3603 Acked Flush 0 Sent Done Done false
10074 3604 Acked Read 3 Sent Done Done false
10075 3605 Acked Read 7 Sent Done Done false
10076 3606 Acked Write 10 Sent Done Done false
10077 3607 Acked Write 1 Sent Done Done false
10078 3608 Acked Read 4 Sent Done Done false
10079 3609 Acked Read 10 Sent Done Done false
10080 3610 Acked Write 10 Sent Done Done false
10081 3611 Acked Read 10 Sent Done Done false
10082 3612 Acked Flush 0 Sent Done Done false
10083 3613 Acked Read 10 Sent Done Done false
10084 3614 Acked Read 7 Sent Done Done false
10085 3615 Acked Write 2 Sent Done Done false
10086 3616 Acked Read 9 Sent Done Done false
10087 3617 Acked Read 5 Sent Done Done false
10088 3618 Acked Write 10 Sent Done Done false
10089 3619 Acked Write 5 Sent Done Done false
10090 3620 Acked Read 6 Sent Done Done false
10091 3621 Acked Read 2 Sent Done Done false
10092 3622 Acked Flush 0 Sent Done Done false
10093 3623 Acked Write 6 Sent Done Done false
10094 3624 Acked Write 7 Sent Done Done false
10095 3625 Acked Read 5 Sent Done Done false
10096 3626 Acked Read 1 Sent Done Done false
10097 3627 Acked Read 2 Sent Done Done false
10098 3628 Acked Write 6 Sent Done Done false
10099 3629 Acked Read 1 Sent Done Done false
10100 3630 Acked Write 6 Sent Done Done false
10101 3631 Acked Write 9 Sent Done Done false
10102 3632 Acked Flush 0 Sent Done Done false
10103 3633 Acked Write 7 Sent Done Done false
10104 3634 Acked Read 7 Sent Done Done false
10105 3635 Acked Write 7 Sent Done Done false
10106 3636 Acked Write 10 Sent Done Done false
10107 3637 Acked Read 4 Sent Done Done false
10108 3638 Acked Write 1 Sent Done Done false
10109 3639 Acked Write 1 Sent Done Done false
10110 3640 Acked Read 9 Sent Done Done false
10111 3641 Acked Read 7 Sent Done Done false
10112 3642 Acked Read 8 Sent Done Done false
10113 3643 Acked Write 4 Sent Done Done false
10114 3644 Acked Read 6 Sent Done Done false
10115 3645 Acked Read 1 Sent Done Done false
10116 3646 Acked Write 3 Sent Done Done false
10117 3647 Acked Read 8 Sent Done Done false
10118 3648 Acked Write 8 Sent Done Done false
10119 3649 Acked Write 9 Sent Done Done false
10120 3650 Acked Flush 0 Sent Done Done false
10121 3651 Acked Flush 0 Sent Done Done false
10122 3652 Acked Read 3 Sent Done Done false
10123 3653 Acked Write 10 Sent Done Done false
10124 3654 Acked Write 7 Sent Done Done false
10125 3655 Acked Write 1 Sent Done Done false
10126 3656 Acked Read 7 Sent Done Done false
10127 3657 Acked Read 2 Sent Done Done false
10128 3658 Acked Read 8 Sent Done Done false
10129 3659 Acked Read 5 Sent Done Done false
10130 3660 Acked Read 8 Sent Done Done false
10131 3661 Acked Write 2 Sent Done Done false
10132 3662 Acked Write 7 Sent Done Done false
10133 3663 Acked Write 9 Sent Done Done false
10134 3664 Acked Read 3 Sent Done Done false
10135 3665 Acked Write 4 Sent Done Done false
10136 3666 Acked Read 4 Sent Done Done false
10137 3667 Acked Read 2 Sent Done Done false
10138 3668 Acked Write 7 Sent Done Done false
10139 3669 Acked Write 2 Sent Done Done false
10140 3670 Acked Write 9 Sent Done Done false
10141 3671 Acked Read 6 Sent Done Done false
10142 3672 Acked Read 6 Sent Done Done false
10143 3673 Acked Read 1 Sent Done Done false
10144 3674 Acked Write 8 Sent Done Done false
10145 3675 Acked Write 4 Sent Done Done false
10146 3676 Acked Read 9 Sent Done Done false
10147 3677 Acked Read 3 Sent Done Done false
10148 3678 Acked Read 6 Sent Done Done false
10149 3679 Acked Write 7 Sent Done Done false
10150 3680 Acked Read 8 Sent Done Done false
10151 3681 Acked Read 5 Sent Done Done false
10152 3682 Acked Write 2 Sent Done Done false
10153 3683 Acked Write 9 Sent Done Done false
10154 3684 Acked Read 2 Sent Done Done false
10155 3685 Acked Write 4 Sent Done Done false
10156 3686 Acked Write 5 Sent Done Done false
10157 3687 Acked Read 8 Sent Done Done false
10158 3688 Acked Write 3 Sent Done Done false
10159 3689 Acked Write 2 Sent Done Done false
10160 3690 Acked Read 4 Sent Done Done false
10161 3691 Acked Read 6 Sent Done Done false
10162 3692 Acked Read 10 Sent Done Done false
10163 3693 Acked Read 5 Sent Done Done false
10164 3694 Acked Read 9 Sent Done Done false
10165 3695 Acked Read 5 Sent Done Done false
10166 3696 Acked Write 5 Sent Done Done false
10167 3697 Acked Read 2 Sent Done Done false
10168 3698 Acked Read 3 Sent Done Done false
10169 3699 Acked Write 8 Sent Done Done false
10170 3700 Acked Read 1 Sent Done Done false
10171 3701 Acked Read 5 Sent Done Done false
10172 3702 Acked Write 4 Sent Done Done false
10173 3703 Acked Write 6 Sent Done Done false
10174 3704 Acked Flush 0 Sent Done Done false
10175 3705 Acked Write 2 Sent Done Done false
10176 3706 Acked Read 10 Sent Done Done false
10177 3707 Acked Read 3 Sent Done Done false
10178 3708 Acked Read 2 Sent Done Done false
10179 3709 Acked Read 10 Sent Done Done false
10180 3710 Acked Write 9 Sent Done Done false
10181 3711 Acked Write 4 Sent Done Done false
10182 3712 Acked Read 3 Sent Done Done false
10183 3713 Acked Read 6 Sent Done Done false
10184 3714 Acked Read 9 Sent Done Done false
10185 3715 Acked Write 3 Sent Done Done false
10186 3716 Acked Flush 0 Sent Done Done false
10187 STATES DS:0 DS:1 DS:2 TOTAL
10188 Sent 302 0 0 302
10189 Done 0 302 302 604
10190 Skipped 0 0 0 0
10191 Error 0 0 0 0
10192 Last Flush: Some(JobId(3414)) Some(JobId(3716)) Some(JobId(3716))
10193 Downstairs last five completed: 3414 3413 3412 3411 3410
10194 Upstairs last five completed: 3716 3715 3714 3713 3712
10195 CLIENT: Up:0 ds:302 act:2
101962025-01-23T18:50:00.940ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
101972025-01-23T18:50:00.940ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61157) = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
101982025-01-23T18:50:00.941ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
101992025-01-23T18:50:00.941ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
102002025-01-23T18:50:00.941ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
102012025-01-23T18:50:00.941ZINFOcrucible: send last flush ID to this DS: Some(JobId(3414)) = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
102022025-01-23T18:50:00.941ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
102032025-01-23T18:50:00.941ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(3414)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
102042025-01-23T18:50:00.942ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(3414)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
10205 ----------------------------------------------------------------
10206 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10207 Downstairs last five completed: 3716 3715 3714 3713 3712
10208 Upstairs last five completed: 3716 3715 3714 3713 3712
10209 CLIENT: Up:0 ds:0 act:3
10210 Replay: All jobs finished
10211 Main task has work to do, go find it
10212 got dsc Stop(1)
10213 stop 1
10214 [8820] Got stop action so:false kr:false
10215 Jan 23 18:50:03.198 INFO request completed, latency_us: 298, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 64bf49e9-0c22-4158-b2c2-f17da3c53c48, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
102162025-01-23T18:50:03.198ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
102172025-01-23T18:50:03.198ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
102182025-01-23T18:50:03.198ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
10219 Jan 23 18:50:03.198 INFO request completed, latency_us: 117, response_code: 200, uri: /state/cid/1, method: GET, req_id: 9f731a1c-b965-4ab7-ab82-d71a495330bd, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
102202025-01-23T18:50:03.198ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
10221 [8820] Exited with: ExitStatus(unix_wait_status(9))
10222 [8820][1] reports Exit
10223 Jan 23 18:50:07.201 INFO request completed, latency_us: 188, response_code: 200, uri: /state/cid/1, method: GET, req_id: 7ec66ec7-7fd2-4519-bf44-2528f55f2351, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
10224 001/300 Write block 4542 len 12288 data: 3 3 3
10225 002/300 Read block 234 len 40960
10226 003/300 Flush
10227 004/300 Read block 489 len 20480
10228 005/300 Read block 2029 len 36864
10229 006/300 Read block 2098 len 16384
10230 007/300 Write block 3224 len 24576 data: 3 3 2 2 2 3
10231 008/300 Write block 3690 len 8192 data: 3 4
10232 009/300 Read block 1348 len 16384
10233 010/300 Write block 413 len 24576 data: 4 4 5 4 4 3
10234 011/300 Write block 3515 len 12288 data: 2 2 2
10235 012/300 Read block 1413 len 40960
10236 013/300 Read block 630 len 24576
10237 014/300 Write block 542 len 20480 data: 5 4 3 3 4
10238 015/300 Write block 852 len 20480 data: 2 2 3 3 3
10239 016/300 Read block 3209 len 36864
10240 017/300 Write block 1003 len 20480 data: 3 3 3 2 2
10241 018/300 Write block 4266 len 36864 data: 2 3 3 3 3 3 3 3 3
10242 019/300 Write block 2748 len 20480 data: 2 2 2 2 3
10243 020/300 Read block 593 len 36864
10244 021/300 Flush
10245 022/300 Read block 4407 len 12288
10246 023/300 Write block 4321 len 20480 data: 4 5 5 5 5
10247 024/300 Write block 2408 len 20480 data: 5 6 5 4 2
10248 025/300 Write block 869 len 24576 data: 2 2 2 2 2 3
10249 026/300 Write block 4558 len 20480 data: 2 2 2 2 2
10250 027/300 Write block 1245 len 36864 data: 3 3 2 3 3 3 4 4 4
10251 028/300 Flush
10252 029/300 Write block 4695 len 8192 data: 4 4
10253 030/300 Read block 1769 len 40960
10254 031/300 Read block 1619 len 12288
10255 032/300 Read block 1259 len 12288
10256 033/300 Read block 2520 len 32768
10257 034/300 Flush
10258 035/300 Read block 3457 len 24576
10259 036/300 Read block 2339 len 40960
10260 037/300 Write block 4042 len 20480 data: 2 2 2 2 4
10261 038/300 Read block 3278 len 40960
10262 039/300 Read block 3849 len 12288
10263 040/300 Write block 2982 len 12288 data: 2 2 3
10264 041/300 Read block 4744 len 36864
10265 042/300 Read block 2636 len 4096
10266 043/300 Read block 4133 len 16384
10267 044/300 Flush
10268 045/300 Read block 4332 len 32768
10269 046/300 Read block 2592 len 8192
10270 047/300 Read block 2583 len 4096
10271 048/300 Write block 6 len 28672 data: 2 2 2 2 2 2 3
10272 049/300 Write block 4088 len 40960 data: 3 3 3 3 4 4 3 3 3 3
10273 050/300 Read block 97 len 4096
10274 051/300 Write block 1644 len 28672 data: 4 4 4 4 4 4 4
10275 052/300 Write block 4352 len 4096 data: 3
10276 053/300 Write block 1344 len 8192 data: 3 3
10277 054/300 Read block 3956 len 8192
10278 055/300 Write block 921 len 16384 data: 2 2 3 2
10279 056/300 Read block 3446 len 24576
10280 057/300 Read block 365 len 36864
10281 058/300 Read block 984 len 4096
10282 059/300 Read block 3828 len 12288
10283 060/300 Write block 3991 len 16384 data: 2 2 2 3
10284 061/300 Write block 2995 len 32768 data: 4 3 3 4 5 5 7 7
10285 062/300 Write block 4660 len 36864 data: 4 4 5 5 6 5 5 3 3
10286 063/300 Read block 1622 len 4096
10287 064/300 Read block 4626 len 12288
10288 065/300 Write block 2598 len 12288 data: 3 3 3
10289 066/300 Read block 796 len 36864
10290 067/300 Read block 4141 len 24576
10291 068/300 Read block 1806 len 32768
10292 069/300 Write block 2039 len 32768 data: 2 2 2 2 2 3 3 3
10293 070/300 Write block 2331 len 4096 data: 5
10294 071/300 Write block 2841 len 32768 data: 2 2 2 3 3 4 4 4
10295 072/300 Read block 3853 len 4096
10296 073/300 Write block 3012 len 40960 data: 2 3 4 4 4 3 3 3 3 4
10297 074/300 Read block 4584 len 28672
10298 075/300 Flush
10299 076/300 Write block 3839 len 40960 data: 3 3 3 3 2 2 2 2 2 2
10300 077/300 Write block 2197 len 20480 data: 3 3 4 4 4
10301 078/300 Read block 559 len 16384
10302 079/300 Write block 1888 len 12288 data: 4 4 3
10303 080/300 Write block 3712 len 12288 data: 3 3 3
10304 081/300 Read block 3796 len 24576
10305 082/300 Read block 680 len 20480
10306 083/300 Read block 1867 len 40960
10307 084/300 Read block 1987 len 16384
10308 085/300 Read block 593 len 40960
10309 086/300 Flush
10310 087/300 Flush
10311 088/300 Write block 1660 len 20480 data: 2 3 3 4 3
10312 089/300 Write block 3734 len 32768 data: 2 3 4 4 4 3 3 3
10313 090/300 Write block 60 len 24576 data: 2 2 2 2 3 3
10314 091/300 Read block 4091 len 12288
10315 092/300 Write block 1462 len 28672 data: 4 4 5 6 6 5 4
10316 093/300 Read block 1460 len 32768
10317 094/300 Write block 3698 len 12288 data: 3 2 2
10318 095/300 Flush
10319 096/300 Read block 2793 len 4096
10320 097/300 Write block 2661 len 8192 data: 4 3
10321 098/300 Read block 2765 len 12288
10322 099/300 Read block 3946 len 40960
10323 100/300 Flush
10324 101/300 Read block 2308 len 36864
10325 102/300 Read block 991 len 24576
10326 103/300 Write block 3361 len 12288 data: 5 5 5
10327 104/300 Write block 2077 len 12288 data: 3 3 3
10328 105/300 Read block 3464 len 40960
10329 106/300 Read block 576 len 16384
10330 107/300 Read block 241 len 4096
10331 108/300 Read block 2346 len 24576
10332 109/300 Flush
10333 110/300 Write block 1658 len 16384 data: 2 2 3 4
10334 111/300 Read block 3599 len 28672
10335 112/300 Read block 737 len 12288
10336 113/300 Write block 3241 len 36864 data: 4 3 3 3 5 4 3 3 3
10337 114/300 Read block 1852 len 16384
10338 115/300 Flush
10339 116/300 Read block 2227 len 8192
10340 117/300 Read block 1693 len 20480
10341 118/300 Read block 486 len 36864
10342 119/300 Read block 2251 len 4096
10343 120/300 Write block 3745 len 20480 data: 3 3 3 3 3
10344 121/300 Write block 2636 len 28672 data: 2 2 2 2 3 3 3
10345 122/300 Write block 111 len 20480 data: 3 4 5 5 5
10346 123/300 Read block 4440 len 24576
10347 124/300 Write block 3987 len 4096 data: 3
10348 125/300 Read block 3872 len 16384
10349 126/300 Read block 63 len 24576
10350 127/300 Flush
10351 128/300 Write block 2618 len 16384 data: 5 5 5 5
10352 129/300 Read block 11 len 4096
10353 130/300 Read block 3760 len 8192
10354 131/300 Read block 3022 len 20480
10355 132/300 Write block 1784 len 28672 data: 4 3 3 3 3 3 4
10356 133/300 Read block 1663 len 24576
10357 134/300 Write block 1691 len 32768 data: 4 4 4 4 4 4 3 3
10358 135/300 Write block 4670 len 16384 data: 3 3 3 2
10359 136/300 Write block 3380 len 32768 data: 3 2 2 2 2 3 3 3
10360 137/300 Read block 4974 len 20480
10361 138/300 Write block 896 len 16384 data: 4 4 4 5
10362 139/300 Read block 2951 len 8192
10363 140/300 Read block 3472 len 4096
10364 141/300 Read block 3051 len 16384
10365 142/300 Read block 4022 len 40960
10366 143/300 Read block 3137 len 12288
10367 144/300 Write block 654 len 12288 data: 4 4 4
10368 145/300 Write block 2682 len 8192 data: 4 3
10369 146/300 Read block 3545 len 12288
10370 147/300 Read block 3572 len 4096
10371 148/300 Read block 3185 len 8192
10372 149/300 Read block 4056 len 12288
10373 150/300 Read block 2780 len 12288
10374 151/300 Write block 514 len 16384 data: 3 3 2 2
10375 152/300 Write block 1859 len 24576 data: 2 2 3 3 3 4
10376 153/300 Write block 413 len 24576 data: 5 5 6 5 5 4
10377 154/300 Read block 3283 len 32768
10378 155/300 Write block 1104 len 8192 data: 4 3
10379 156/300 Read block 3127 len 16384
10380 157/300 Read block 577 len 36864
10381 158/300 Read block 2692 len 36864
10382 159/300 Flush
10383 160/300 Read block 1778 len 32768
10384 161/300 Read block 185 len 40960
10385 162/300 Write block 1652 len 20480 data: 3 2 2 2 2
10386 163/300 Read block 608 len 12288
10387 164/300 Write block 4944 len 4096 data: 2
10388 165/300 Write block 817 len 40960 data: 4 4 4 5 5 6 6 4 3 4
10389 166/300 Write block 3507 len 12288 data: 4 3 3
10390 167/300 Read block 287 len 28672
10391 168/300 Write block 2972 len 4096 data: 2
10392 169/300 Read block 1162 len 4096
10393 170/300 Read block 4413 len 16384
10394 171/300 Write block 1692 len 32768 data: 5 5 5 5 5 4 4 4
10395 172/300 Flush
10396 173/300 Read block 4383 len 4096
10397 174/300 Read block 203 len 28672
10398 175/300 Write block 739 len 28672 data: 3 3 4 4 4 4 4
10399 176/300 Read block 1601 len 4096
10400 177/300 Read block 1051 len 24576
10401 178/300 Read block 3941 len 32768
10402 179/300 Write block 3477 len 40960 data: 3 3 3 3 3 3 2 2 2 2
10403 180/300 Write block 3733 len 40960 data: 2 3 4 5 5 5 4 4 4 3
10404 181/300 Write block 1158 len 32768 data: 4 4 3 3 3 2 2 2
10405 182/300 Flush
10406 183/300 Read block 1845 len 12288
10407 184/300 Write block 4267 len 24576 data: 4 4 4 4 4 4
10408 185/300 Write block 404 len 32768 data: 4 4 4 4 4 4 5 5
10409 186/300 Write block 415 len 36864 data: 7 6 6 5 3 2 2 2 3
10410 187/300 Read block 4546 len 12288
10411 188/300 Read block 4970 len 20480
10412 189/300 Flush
10413 190/300 Write block 1758 len 12288 data: 3 3 3
10414 191/300 Read block 1663 len 28672
10415 192/300 Flush
10416 193/300 Write block 2137 len 16384 data: 4 4 3 3
10417 194/300 Read block 1301 len 40960
10418 195/300 Write block 3533 len 20480 data: 3 5 5 5 5
10419 196/300 Write block 1726 len 28672 data: 2 2 2 2 2 3 3
10420 197/300 Write block 2731 len 24576 data: 3 3 3 3 3 3
10421 198/300 Flush
10422 199/300 Flush
10423 200/300 Flush
10424 201/300 Read block 2931 len 20480
10425 202/300 Write block 2416 len 12288 data: 3 3 3
10426 203/300 Read block 4823 len 28672
10427 204/300 Write block 4344 len 28672 data: 6 4 4 4 5 5 3
10428 205/300 Read block 804 len 16384
10429 206/300 Read block 2180 len 24576
10430 207/300 Write block 2301 len 16384 data: 3 3 4 3
10431 208/300 Read block 4854 len 32768
10432 209/300 Read block 1121 len 12288
10433 210/300 Flush
10434 211/300 Read block 2828 len 28672
10435 212/300 Write block 4279 len 40960 data: 4 4 4 2 2 2 2 4 4 4
10436 213/300 Write block 1517 len 20480 data: 2 2 2 2 3
10437 214/300 Write block 4542 len 28672 data: 4 4 4 3 3 2 2
10438 215/300 Flush
10439 216/300 Read block 966 len 4096
10440 217/300 Write block 4670 len 8192 data: 4 4
10441 218/300 Write block 4732 len 24576 data: 2 3 3 2 2 2
10442 219/300 Write block 1042 len 32768 data: 2 2 2 3 3 4 4 4
10443 220/300 Write block 1113 len 20480 data: 3 3 3 3 3
10444 221/300 Read block 42 len 8192
10445 222/300 Read block 4911 len 20480
10446 223/300 Write block 4087 len 16384 data: 3 4 4 4
10447 224/300 Read block 3011 len 8192
10448 225/300 Flush
10449 226/300 Read block 3503 len 40960
10450 227/300 Flush
10451 228/300 Read block 3526 len 8192
10452 229/300 Write block 1358 len 40960 data: 2 3 2 3 2 2 3 3 3 3
10453 230/300 Read block 2045 len 16384
10454 231/300 Write block 2776 len 8192 data: 4 3
10455 232/300 Write block 1213 len 12288 data: 3 4 4
10456 233/300 Write block 3306 len 36864 data: 2 3 4 6 7 6 5 5 4
10457 234/300 Write block 2668 len 20480 data: 2 2 2 3 3
10458 235/300 Write block 2460 len 36864 data: 4 4 3 2 2 2 4 4 3
10459 236/300 Write block 4534 len 32768 data: 4 4 3 4 3 3 3 3
10460 237/300 Write block 3883 len 24576 data: 4 3 3 3 2 2
10461 238/300 Write block 3567 len 12288 data: 3 3 3
10462 239/300 Write block 1508 len 4096 data: 4
10463 240/300 Flush
10464 241/300 Read block 1693 len 24576
10465 242/300 Read block 1777 len 36864
10466 243/300 Read block 4841 len 36864
10467 244/300 Write block 187 len 36864 data: 4 3 3 3 3 2 2 2 2
10468 245/300 Write block 65 len 4096 data: 4
10469 246/300 Read block 4007 len 8192
10470 247/300 Write block 3584 len 8192 data: 2 2
10471 248/300 Flush
10472 249/300 Write block 1669 len 36864 data: 3 3 2 3 3 3 4 4 4
10473 250/300 Write block 4596 len 28672 data: 4 4 4 4 3 3 2
10474 251/300 Write block 428 len 16384 data: 7 8 7 7
10475 252/300 Read block 3177 len 16384
10476 253/300 Read block 1417 len 4096
10477 254/300 Read block 557 len 20480
10478 255/300 Flush
10479 256/300 Read block 3141 len 12288
10480 257/300 Write block 2836 len 24576 data: 3 3 3 3 3 3
10481 258/300 Read block 724 len 20480
10482 259/300 Write block 1830 len 28672 data: 2 2 2 2 2 2 2
10483 260/300 Write block 4340 len 28672 data: 3 4 5 5 7 5 5
10484 261/300 Read block 537 len 28672
10485 262/300 Read block 3859 len 12288
10486 263/300 Write block 91 len 16384 data: 4 4 4 5
10487 264/300 Read block 3141 len 40960
10488 265/300 Write block 2009 len 24576 data: 4 4 5 5 5 5
10489 266/300 Write block 4217 len 24576 data: 3 3 4 4 4 5
10490 267/300 Read block 4164 len 32768
10491 268/300 Read block 1465 len 16384
10492 269/300 Read block 4106 len 16384
10493 270/300 Write block 3665 len 28672 data: 2 3 3 3 3 3 3
10494 271/300 Read block 4440 len 32768
10495 272/300 Write block 3408 len 32768 data: 2 2 2 2 2 2 2 2
10496 273/300 Read block 4547 len 36864
10497 274/300 Read block 4834 len 4096
10498 275/300 Flush
10499 276/300 Write block 4092 len 36864 data: 5 5 4 4 4 4 3 3 2
10500 277/300 Write block 3805 len 20480 data: 2 3 2 2 2
10501 278/300 Write block 4302 len 20480 data: 2 2 3 3 4
10502 279/300 Write block 2154 len 40960 data: 3 3 5 4 4 3 3 3 3 3
10503 280/300 Flush
10504 281/300 Flush
10505 282/300 Read block 985 len 12288
10506 283/300 Read block 1606 len 8192
10507 284/300 Read block 537 len 36864
10508 285/300 Read block 4701 len 8192
10509 286/300 Read block 4365 len 40960
10510 287/300 Write block 2905 len 32768 data: 2 2 2 3 3 3 3 2
10511 288/300 Write block 3722 len 4096 data: 5
10512 289/300 Write block 1828 len 20480 data: 3 3 3 3 3
10513 290/300 Flush
10514 291/300 Read block 2190 len 8192
10515 292/300 Flush
10516 293/300 Read block 1056 len 36864
10517 294/300 Write block 1505 len 40960 data: 4 4 4 5 4 4 4 3 3 4
10518 295/300 Read block 3465 len 12288
10519 296/300 Read block 747 len 20480
10520 297/300 Read block 3429 len 24576
10521 298/300 Write block 509 len 40960 data: 3 3 3 2 2 4 4 3 3 3
10522 299/300 Write block 1061 len 24576 data: 3 5 3 4 4 4
10523 300/300 Read block 943 len 16384
10524 301/300 Flush
10525 Main task has work to do, go find it
10526 got dsc Start(1)
10527 start 1
10528 Jan 23 18:50:07.429 INFO request completed, latency_us: 179, response_code: 204, uri: /start/cid/1, method: POST, req_id: ca5e269a-d0e4-4ac4-af92-26999bf926a6, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
10529 Replay: started 1, returned:Ok(())
10530 ----------------------------------------------------------------
10531 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10532 States: Active Offline Active
10533 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10534 3717 Acked Write 3 Done Sent Done false
10535 3718 Acked Read 10 Done Sent Done false
10536 3719 Acked Flush 0 Done Sent Done false
10537 3720 Acked Read 5 Done Sent Done false
10538 3721 Acked Read 9 Done Sent Done false
10539 3722 Acked Read 4 Done Sent Done false
10540 3723 Acked Write 6 Done Sent Done false
10541 3724 Acked Write 2 Done Sent Done false
10542 3725 Acked Read 4 Done Sent Done false
10543 3726 Acked Write 6 Done Sent Done false
10544 3727 Acked Write 3 Done Sent Done false
10545 3728 Acked Read 10 Done Sent Done false
10546 3729 Acked Read 6 Done Sent Done false
10547 3730 Acked Write 5 Done Sent Done false
10548 3731 Acked Write 5 Done Sent Done false
10549 3732 Acked Read 9 Done Sent Done false
10550 3733 Acked Write 5 Done Sent Done false
10551 3734 Acked Write 9 Done Sent Done false
10552 3735 Acked Write 5 Done Sent Done false
10553 3736 Acked Read 9 Done Sent Done false
10554 3737 Acked Flush 0 Done Sent Done false
10555 3738 Acked Read 3 Done Sent Done false
10556 3739 Acked Write 5 Done Sent Done false
10557 3740 Acked Write 5 Done Sent Done false
10558 3741 Acked Write 6 Done Sent Done false
10559 3742 Acked Write 5 Done Sent Done false
10560 3743 Acked Write 9 Done Sent Done false
10561 3744 Acked Flush 0 Done Sent Done false
10562 3745 Acked Write 2 Done Sent Done false
10563 3746 Acked Read 10 Done Sent Done false
10564 3747 Acked Read 3 Done Sent Done false
10565 3748 Acked Read 3 Done Sent Done false
10566 3749 Acked Read 8 Done Sent Done false
10567 3750 Acked Flush 0 Done Sent Done false
10568 3751 Acked Read 6 Done Sent Done false
10569 3752 Acked Read 10 Done Sent Done false
10570 3753 Acked Write 5 Done Sent Done false
10571 3754 Acked Read 10 Done Sent Done false
10572 3755 Acked Read 3 Done Sent Done false
10573 3756 Acked Write 3 Done Sent Done false
10574 3757 Acked Read 9 Done Sent Done false
10575 3758 Acked Read 1 Done Sent Done false
10576 3759 Acked Read 4 Done Sent Done false
10577 3760 Acked Flush 0 Done Sent Done false
10578 3761 Acked Read 8 Done Sent Done false
10579 3762 Acked Read 2 Done Sent Done false
10580 3763 Acked Read 1 Done Sent Done false
10581 3764 Acked Write 7 Done Sent Done false
10582 3765 Acked Write 10 Done Sent Done false
10583 3766 Acked Read 1 Done Sent Done false
10584 3767 Acked Write 7 Done Sent Done false
10585 3768 Acked Write 1 Done Sent Done false
10586 3769 Acked Write 2 Done Sent Done false
10587 3770 Acked Read 2 Done Sent Done false
10588 3771 Acked Write 4 Done Sent Done false
10589 3772 Acked Read 6 Done Sent Done false
10590 3773 Acked Read 9 Done Sent Done false
10591 3774 Acked Read 1 Done Sent Done false
10592 3775 Acked Read 3 Done Sent Done false
10593 3776 Acked Write 4 Done Sent Done false
10594 3777 Acked Write 8 Done Sent Done false
10595 3778 Acked Write 9 Done Sent Done false
10596 3779 Acked Read 1 Done Sent Done false
10597 3780 Acked Read 3 Done Sent Done false
10598 3781 Acked Write 3 Done Sent Done false
10599 3782 Acked Read 9 Done Sent Done false
10600 3783 Acked Read 6 Done Sent Done false
10601 3784 Acked Read 8 Done Sent Done false
10602 3785 Acked Write 8 Done Sent Done false
10603 3786 Acked Write 1 Done Sent Done false
10604 3787 Acked Write 8 Done Sent Done false
10605 3788 Acked Read 1 Done Sent Done false
10606 3789 Acked Write 10 Done Sent Done false
10607 3790 Acked Read 7 Done Sent Done false
10608 3791 Acked Flush 0 Done Sent Done false
10609 3792 Acked Write 10 Done Sent Done false
10610 3793 Acked Write 5 Done Sent Done false
10611 3794 Acked Read 4 Done Sent Done false
10612 3795 Acked Write 3 Done Sent Done false
10613 3796 Acked Write 3 Done Sent Done false
10614 3797 Acked Read 6 Done Sent Done false
10615 3798 Acked Read 5 Done Sent Done false
10616 3799 Acked Read 10 Done Sent Done false
10617 3800 Acked Read 4 Done Sent Done false
10618 3801 Acked Read 10 Done Sent Done false
10619 3802 Acked Flush 0 Done Sent Done false
10620 3803 Acked Flush 0 Done Sent Done false
10621 3804 Acked Write 5 Done Sent Done false
10622 3805 Acked Write 8 Done Sent Done false
10623 3806 Acked Write 6 Done Sent Done false
10624 3807 Acked Read 3 Done Sent Done false
10625 3808 Acked Write 7 Done Sent Done false
10626 3809 Acked Read 8 Done Sent Done false
10627 3810 Acked Write 3 Done Sent Done false
10628 3811 Acked Flush 0 Done Sent Done false
10629 3812 Acked Read 1 Done Sent Done false
10630 3813 Acked Write 2 Done Sent Done false
10631 3814 Acked Read 3 Done Sent Done false
10632 3815 Acked Read 10 Done Sent Done false
10633 3816 Acked Flush 0 Done Sent Done false
10634 3817 Acked Read 9 Done Sent Done false
10635 3818 Acked Read 6 Done Sent Done false
10636 3819 Acked Write 3 Done Sent Done false
10637 3820 Acked Write 3 Done Sent Done false
10638 3821 Acked Read 10 Done Sent Done false
10639 3822 Acked Read 4 Done Sent Done false
10640 3823 Acked Read 1 Done Sent Done false
10641 3824 Acked Read 6 Done Sent Done false
10642 3825 Acked Flush 0 Done Sent Done false
10643 3826 Acked Write 4 Done Sent Done false
10644 3827 Acked Read 7 Done Sent Done false
10645 3828 Acked Read 3 Done Sent Done false
10646 3829 Acked Write 9 Done Sent Done false
10647 3830 Acked Read 4 Done Sent Done false
10648 3831 Acked Flush 0 Done Sent Done false
10649 3832 Acked Read 2 Done Sent Done false
10650 3833 Acked Read 5 Done Sent Done false
10651 3834 Acked Read 9 Done Sent Done false
10652 3835 Acked Read 1 Done Sent Done false
10653 3836 Acked Write 5 Done Sent Done false
10654 3837 Acked Write 7 Done Sent Done false
10655 3838 Acked Write 5 Done Sent Done false
10656 3839 Acked Read 6 Done Sent Done false
10657 3840 Acked Write 1 Done Sent Done false
10658 3841 Acked Read 4 Done Sent Done false
10659 3842 Acked Read 6 Done Sent Done false
10660 3843 Acked Flush 0 Done Sent Done false
10661 3844 Acked Write 4 Done Sent Done false
10662 3845 Acked Read 1 Done Sent Done false
10663 3846 Acked Read 2 Done Sent Done false
10664 3847 Acked Read 5 Done Sent Done false
10665 3848 Acked Write 7 Done Sent Done false
10666 3849 Acked Read 6 Done Sent Done false
10667 3850 Acked Write 8 Done Sent Done false
10668 3851 Acked Write 4 Done Sent Done false
10669 3852 Acked Write 8 Done Sent Done false
10670 3853 Acked Read 5 Done Sent Done false
10671 3854 Acked Write 4 Done Sent Done false
10672 3855 Acked Read 2 Done Sent Done false
10673 3856 Acked Read 1 Done Sent Done false
10674 3857 Acked Read 4 Done Sent Done false
10675 3858 Acked Read 10 Done Sent Done false
10676 3859 Acked Read 3 Done Sent Done false
10677 3860 Acked Write 3 Done Sent Done false
10678 3861 Acked Write 2 Done Sent Done false
10679 3862 Acked Read 3 Done Sent Done false
10680 3863 Acked Read 1 Done Sent Done false
10681 3864 Acked Read 2 Done Sent Done false
10682 3865 Acked Read 3 Done Sent Done false
10683 3866 Acked Read 3 Done Sent Done false
10684 3867 Acked Write 4 Done Sent Done false
10685 3868 Acked Write 6 Done Sent Done false
10686 3869 Acked Write 6 Done Sent Done false
10687 3870 Acked Read 8 Done Sent Done false
10688 3871 Acked Write 2 Done Sent Done false
10689 3872 Acked Read 4 Done Sent Done false
10690 3873 Acked Read 9 Done Sent Done false
10691 3874 Acked Read 9 Done Sent Done false
10692 3875 Acked Flush 0 Done Sent Done false
10693 3876 Acked Read 8 Done Sent Done false
10694 3877 Acked Read 10 Done Sent Done false
10695 3878 Acked Write 5 Done Sent Done false
10696 3879 Acked Read 3 Done Sent Done false
10697 3880 Acked Write 1 Done Sent Done false
10698 3881 Acked Write 10 Done Sent Done false
10699 3882 Acked Write 3 Done Sent Done false
10700 3883 Acked Read 7 Done Sent Done false
10701 3884 Acked Write 1 Done Sent Done false
10702 3885 Acked Read 1 Done Sent Done false
10703 3886 Acked Read 4 Done Sent Done false
10704 3887 Acked Write 8 Done Sent Done false
10705 3888 Acked Flush 0 Done Sent Done false
10706 3889 Acked Read 1 Done Sent Done false
10707 3890 Acked Read 7 Done Sent Done false
10708 3891 Acked Write 7 Done Sent Done false
10709 3892 Acked Read 1 Done Sent Done false
10710 3893 Acked Read 6 Done Sent Done false
10711 3894 Acked Read 8 Done Sent Done false
10712 3895 Acked Write 10 Done Sent Done false
10713 3896 Acked Write 10 Done Sent Done false
10714 3897 Acked Write 8 Done Sent Done false
10715 3898 Acked Flush 0 Done Sent Done false
10716 3899 Acked Read 3 Done Sent Done false
10717 3900 Acked Write 6 Done Sent Done false
10718 3901 Acked Write 8 Done Sent Done false
10719 3902 Acked Write 9 Done Sent Done false
10720 3903 Acked Read 3 Done Sent Done false
10721 3904 Acked Read 5 Done Sent Done false
10722 3905 Acked Flush 0 Done Sent Done false
10723 3906 Acked Write 3 Done Sent Done false
10724 3907 Acked Read 7 Done Sent Done false
10725 3908 Acked Flush 0 Done Sent Done false
10726 3909 Acked Write 4 Done Sent Done false
10727 3910 Acked Read 10 Done Sent Done false
10728 3911 Acked Write 5 Done Sent Done false
10729 3912 Acked Write 7 Done Sent Done false
10730 3913 Acked Write 6 Done Sent Done false
10731 3914 Acked Flush 0 Done Sent Done false
10732 3915 Acked Flush 0 Done Sent Done false
10733 3916 Acked Flush 0 Done Sent Done false
10734 3917 Acked Read 5 Done Sent Done false
10735 3918 Acked Write 3 Done Sent Done false
10736 3919 Acked Read 7 Done Sent Done false
10737 3920 Acked Write 7 Done Sent Done false
10738 3921 Acked Read 4 Done Sent Done false
10739 3922 Acked Read 6 Done Sent Done false
10740 3923 Acked Write 4 Done Sent Done false
10741 3924 Acked Read 8 Done Sent Done false
10742 3925 Acked Read 3 Done Sent Done false
10743 3926 Acked Flush 0 Done Sent Done false
10744 3927 Acked Read 7 Done Sent Done false
10745 3928 Acked Write 10 Done Sent Done false
10746 3929 Acked Write 5 Done Sent Done false
10747 3930 Acked Write 7 Done Sent Done false
10748 3931 Acked Flush 0 Done Sent Done false
10749 3932 Acked Read 1 Done Sent Done false
10750 3933 Acked Write 2 Done Sent Done false
10751 3934 Acked Write 6 Done Sent Done false
10752 3935 Acked Write 8 Done Sent Done false
10753 3936 Acked Write 5 Done Sent Done false
10754 3937 Acked Read 2 Done Sent Done false
10755 3938 Acked Read 5 Done Sent Done false
10756 3939 Acked Write 4 Done Sent Done false
10757 3940 Acked Read 2 Done Sent Done false
10758 3941 Acked Flush 0 Done Sent Done false
10759 3942 Acked Read 10 Done Sent Done false
10760 3943 Acked Flush 0 Done Sent Done false
10761 3944 Acked Read 2 Done Sent Done false
10762 3945 Acked Write 10 Done Sent Done false
10763 3946 Acked Read 4 Done Sent Done false
10764 3947 Acked Write 2 Done Sent Done false
10765 3948 Acked Write 3 Done Sent Done false
10766 3949 Acked Write 9 Done Sent Done false
10767 3950 Acked Write 5 Done Sent Done false
10768 3951 Acked Write 9 Done Sent Done false
10769 3952 Acked Write 8 Done Sent Done false
10770 3953 Acked Write 6 Done Sent Done false
10771 3954 Acked Write 3 Done Sent Done false
10772 3955 Acked Write 1 Done Sent Done false
10773 3956 Acked Flush 0 Done Sent Done false
10774 3957 Acked Read 6 Done Sent Done false
10775 3958 Acked Read 9 Done Sent Done false
10776 3959 Acked Read 9 Done Sent Done false
10777 3960 Acked Write 9 Done Sent Done false
10778 3961 Acked Write 1 Done Sent Done false
10779 3962 Acked Read 2 Done Sent Done false
10780 3963 Acked Write 2 Done Sent Done false
10781 3964 Acked Flush 0 Done Sent Done false
10782 3965 Acked Write 9 Done Sent Done false
10783 3966 Acked Write 7 Done Sent Done false
10784 3967 Acked Write 4 Done Sent Done false
10785 3968 Acked Read 4 Done Sent Done false
10786 3969 Acked Read 1 Done Sent Done false
10787 3970 Acked Read 5 Done Sent Done false
10788 3971 Acked Flush 0 Done Sent Done false
10789 3972 Acked Read 3 Done Sent Done false
10790 3973 Acked Write 6 Done Sent Done false
10791 3974 Acked Read 5 Done Sent Done false
10792 3975 Acked Write 7 Done Sent Done false
10793 3976 Acked Write 7 Done Sent Done false
10794 3977 Acked Read 7 Done Sent Done false
10795 3978 Acked Read 3 Done Sent Done false
10796 3979 Acked Write 4 Done Sent Done false
10797 3980 Acked Read 10 Done Sent Done false
10798 3981 Acked Write 6 Done Sent Done false
10799 3982 Acked Write 6 Done Sent Done false
10800 3983 Acked Read 8 Done Sent Done false
10801 3984 Acked Read 4 Done Sent Done false
10802 3985 Acked Read 4 Done Sent Done false
10803 3986 Acked Write 7 Done Sent Done false
10804 3987 Acked Read 8 Done Sent Done false
10805 3988 Acked Write 8 Done Sent Done false
10806 3989 Acked Read 9 Done Sent Done false
10807 3990 Acked Read 1 Done Sent Done false
10808 3991 Acked Flush 0 Done Sent Done false
10809 3992 Acked Write 9 Done Sent Done false
10810 3993 Acked Write 5 Done Sent Done false
10811 3994 Acked Write 5 Done Sent Done false
10812 3995 Acked Write 10 Done Sent Done false
10813 3996 Acked Flush 0 Done Sent Done false
10814 3997 Acked Flush 0 Done Sent Done false
10815 3998 Acked Read 3 Done Sent Done false
10816 3999 Acked Read 2 Done Sent Done false
10817 4000 Acked Read 9 Done Sent Done false
10818 4001 Acked Read 2 Done Sent Done false
10819 4002 Acked Read 10 Done Sent Done false
10820 4003 Acked Write 8 Done Sent Done false
10821 4004 Acked Write 1 Done Sent Done false
10822 4005 Acked Write 5 Done Sent Done false
10823 4006 Acked Flush 0 Done Sent Done false
10824 4007 Acked Read 2 Done Sent Done false
10825 4008 Acked Flush 0 Done Sent Done false
10826 4009 Acked Read 9 Done Sent Done false
10827 4010 Acked Write 10 Done Sent Done false
10828 4011 Acked Read 3 Done Sent Done false
10829 4012 Acked Read 5 Done Sent Done false
10830 4013 Acked Read 6 Done Sent Done false
10831 4014 Acked Write 10 Done Sent Done false
10832 4015 Acked Write 6 Done Sent Done false
10833 4016 Acked Read 4 Done Sent Done false
10834 4017 Acked Flush 0 Done Sent Done false
10835 STATES DS:0 DS:1 DS:2 TOTAL
10836 Sent 0 301 0 301
10837 Done 301 0 301 602
10838 Skipped 0 0 0 0
10839 Error 0 0 0 0
10840 Last Flush: Some(JobId(4017)) Some(JobId(3716)) Some(JobId(4017))
10841 Downstairs last five completed: 3716 3715 3714 3713 3712
10842 Upstairs last five completed: 4017 4016 4015 4014 4013
10843 CLIENT: Up:0 ds:301 act:2
10844 ----------------------------------------------------------------
10845 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10846 States: Active Offline Active
10847 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10848 3717 Acked Write 3 Done Sent Done false
10849 3718 Acked Read 10 Done Sent Done false
10850 3719 Acked Flush 0 Done Sent Done false
10851 3720 Acked Read 5 Done Sent Done false
10852 3721 Acked Read 9 Done Sent Done false
10853 3722 Acked Read 4 Done Sent Done false
10854 3723 Acked Write 6 Done Sent Done false
10855 3724 Acked Write 2 Done Sent Done false
10856 3725 Acked Read 4 Done Sent Done false
10857 3726 Acked Write 6 Done Sent Done false
10858 3727 Acked Write 3 Done Sent Done false
10859 3728 Acked Read 10 Done Sent Done false
10860 3729 Acked Read 6 Done Sent Done false
10861 3730 Acked Write 5 Done Sent Done false
10862 3731 Acked Write 5 Done Sent Done false
10863 3732 Acked Read 9 Done Sent Done false
10864 3733 Acked Write 5 Done Sent Done false
10865 3734 Acked Write 9 Done Sent Done false
10866 3735 Acked Write 5 Done Sent Done false
10867 3736 Acked Read 9 Done Sent Done false
10868 3737 Acked Flush 0 Done Sent Done false
10869 3738 Acked Read 3 Done Sent Done false
10870 3739 Acked Write 5 Done Sent Done false
10871 3740 Acked Write 5 Done Sent Done false
10872 3741 Acked Write 6 Done Sent Done false
10873 3742 Acked Write 5 Done Sent Done false
10874 3743 Acked Write 9 Done Sent Done false
10875 3744 Acked Flush 0 Done Sent Done false
10876 3745 Acked Write 2 Done Sent Done false
10877 3746 Acked Read 10 Done Sent Done false
10878 3747 Acked Read 3 Done Sent Done false
10879 3748 Acked Read 3 Done Sent Done false
10880 3749 Acked Read 8 Done Sent Done false
10881 3750 Acked Flush 0 Done Sent Done false
10882 3751 Acked Read 6 Done Sent Done false
10883 3752 Acked Read 10 Done Sent Done false
10884 3753 Acked Write 5 Done Sent Done false
10885 3754 Acked Read 10 Done Sent Done false
10886 3755 Acked Read 3 Done Sent Done false
10887 3756 Acked Write 3 Done Sent Done false
10888 3757 Acked Read 9 Done Sent Done false
10889 3758 Acked Read 1 Done Sent Done false
10890 3759 Acked Read 4 Done Sent Done false
10891 3760 Acked Flush 0 Done Sent Done false
10892 3761 Acked Read 8 Done Sent Done false
10893 3762 Acked Read 2 Done Sent Done false
10894 3763 Acked Read 1 Done Sent Done false
10895 3764 Acked Write 7 Done Sent Done false
10896 3765 Acked Write 10 Done Sent Done false
10897 3766 Acked Read 1 Done Sent Done false
10898 3767 Acked Write 7 Done Sent Done false
10899 3768 Acked Write 1 Done Sent Done false
10900 3769 Acked Write 2 Done Sent Done false
10901 3770 Acked Read 2 Done Sent Done false
10902 3771 Acked Write 4 Done Sent Done false
10903 3772 Acked Read 6 Done Sent Done false
10904 3773 Acked Read 9 Done Sent Done false
10905 3774 Acked Read 1 Done Sent Done false
10906 3775 Acked Read 3 Done Sent Done false
10907 3776 Acked Write 4 Done Sent Done false
10908 3777 Acked Write 8 Done Sent Done false
10909 3778 Acked Write 9 Done Sent Done false
10910 3779 Acked Read 1 Done Sent Done false
10911 3780 Acked Read 3 Done Sent Done false
10912 3781 Acked Write 3 Done Sent Done false
10913 3782 Acked Read 9 Done Sent Done false
10914 3783 Acked Read 6 Done Sent Done false
10915 3784 Acked Read 8 Done Sent Done false
10916 3785 Acked Write 8 Done Sent Done false
10917 3786 Acked Write 1 Done Sent Done false
10918 3787 Acked Write 8 Done Sent Done false
10919 3788 Acked Read 1 Done Sent Done false
10920 3789 Acked Write 10 Done Sent Done false
10921 3790 Acked Read 7 Done Sent Done false
10922 3791 Acked Flush 0 Done Sent Done false
10923 3792 Acked Write 10 Done Sent Done false
10924 3793 Acked Write 5 Done Sent Done false
10925 3794 Acked Read 4 Done Sent Done false
10926 3795 Acked Write 3 Done Sent Done false
10927 3796 Acked Write 3 Done Sent Done false
10928 3797 Acked Read 6 Done Sent Done false
10929 3798 Acked Read 5 Done Sent Done false
10930 3799 Acked Read 10 Done Sent Done false
10931 3800 Acked Read 4 Done Sent Done false
10932 3801 Acked Read 10 Done Sent Done false
10933 3802 Acked Flush 0 Done Sent Done false
10934 3803 Acked Flush 0 Done Sent Done false
10935 3804 Acked Write 5 Done Sent Done false
10936 3805 Acked Write 8 Done Sent Done false
10937 3806 Acked Write 6 Done Sent Done false
10938 3807 Acked Read 3 Done Sent Done false
10939 3808 Acked Write 7 Done Sent Done false
10940 3809 Acked Read 8 Done Sent Done false
10941 3810 Acked Write 3 Done Sent Done false
10942 3811 Acked Flush 0 Done Sent Done false
10943 3812 Acked Read 1 Done Sent Done false
10944 3813 Acked Write 2 Done Sent Done false
10945 3814 Acked Read 3 Done Sent Done false
10946 3815 Acked Read 10 Done Sent Done false
10947 3816 Acked Flush 0 Done Sent Done false
10948 3817 Acked Read 9 Done Sent Done false
10949 3818 Acked Read 6 Done Sent Done false
10950 3819 Acked Write 3 Done Sent Done false
10951 3820 Acked Write 3 Done Sent Done false
10952 3821 Acked Read 10 Done Sent Done false
10953 3822 Acked Read 4 Done Sent Done false
10954 3823 Acked Read 1 Done Sent Done false
10955 3824 Acked Read 6 Done Sent Done false
10956 3825 Acked Flush 0 Done Sent Done false
10957 3826 Acked Write 4 Done Sent Done false
10958 3827 Acked Read 7 Done Sent Done false
10959 3828 Acked Read 3 Done Sent Done false
10960 3829 Acked Write 9 Done Sent Done false
10961 3830 Acked Read 4 Done Sent Done false
10962 3831 Acked Flush 0 Done Sent Done false
10963 3832 Acked Read 2 Done Sent Done false
10964 3833 Acked Read 5 Done Sent Done false
10965 3834 Acked Read 9 Done Sent Done false
10966 3835 Acked Read 1 Done Sent Done false
10967 3836 Acked Write 5 Done Sent Done false
10968 3837 Acked Write 7 Done Sent Done false
10969 3838 Acked Write 5 Done Sent Done false
10970 3839 Acked Read 6 Done Sent Done false
10971 3840 Acked Write 1 Done Sent Done false
10972 3841 Acked Read 4 Done Sent Done false
10973 3842 Acked Read 6 Done Sent Done false
10974 3843 Acked Flush 0 Done Sent Done false
10975 3844 Acked Write 4 Done Sent Done false
10976 3845 Acked Read 1 Done Sent Done false
10977 3846 Acked Read 2 Done Sent Done false
10978 3847 Acked Read 5 Done Sent Done false
10979 3848 Acked Write 7 Done Sent Done false
10980 3849 Acked Read 6 Done Sent Done false
10981 3850 Acked Write 8 Done Sent Done false
10982 3851 Acked Write 4 Done Sent Done false
10983 3852 Acked Write 8 Done Sent Done false
10984 3853 Acked Read 5 Done Sent Done false
10985 3854 Acked Write 4 Done Sent Done false
10986 3855 Acked Read 2 Done Sent Done false
10987 3856 Acked Read 1 Done Sent Done false
10988 3857 Acked Read 4 Done Sent Done false
10989 3858 Acked Read 10 Done Sent Done false
10990 3859 Acked Read 3 Done Sent Done false
10991 3860 Acked Write 3 Done Sent Done false
10992 3861 Acked Write 2 Done Sent Done false
10993 3862 Acked Read 3 Done Sent Done false
10994 3863 Acked Read 1 Done Sent Done false
10995 3864 Acked Read 2 Done Sent Done false
10996 3865 Acked Read 3 Done Sent Done false
10997 3866 Acked Read 3 Done Sent Done false
10998 3867 Acked Write 4 Done Sent Done false
10999 3868 Acked Write 6 Done Sent Done false
11000 3869 Acked Write 6 Done Sent Done false
11001 3870 Acked Read 8 Done Sent Done false
11002 3871 Acked Write 2 Done Sent Done false
11003 3872 Acked Read 4 Done Sent Done false
11004 3873 Acked Read 9 Done Sent Done false
11005 3874 Acked Read 9 Done Sent Done false
11006 3875 Acked Flush 0 Done Sent Done false
11007 3876 Acked Read 8 Done Sent Done false
11008 3877 Acked Read 10 Done Sent Done false
11009 3878 Acked Write 5 Done Sent Done false
11010 3879 Acked Read 3 Done Sent Done false
11011 3880 Acked Write 1 Done Sent Done false
11012 3881 Acked Write 10 Done Sent Done false
11013 3882 Acked Write 3 Done Sent Done false
11014 3883 Acked Read 7 Done Sent Done false
11015 3884 Acked Write 1 Done Sent Done false
11016 3885 Acked Read 1 Done Sent Done false
11017 3886 Acked Read 4 Done Sent Done false
11018 3887 Acked Write 8 Done Sent Done false
11019 3888 Acked Flush 0 Done Sent Done false
11020 3889 Acked Read 1 Done Sent Done false
11021 3890 Acked Read 7 Done Sent Done false
11022 3891 Acked Write 7 Done Sent Done false
11023 3892 Acked Read 1 Done Sent Done false
11024 3893 Acked Read 6 Done Sent Done false
11025 3894 Acked Read 8 Done Sent Done false
11026 3895 Acked Write 10 Done Sent Done false
11027 3896 Acked Write 10 Done Sent Done false
11028 3897 Acked Write 8 Done Sent Done false
11029 3898 Acked Flush 0 Done Sent Done false
11030 3899 Acked Read 3 Done Sent Done false
11031 3900 Acked Write 6 Done Sent Done false
11032 3901 Acked Write 8 Done Sent Done false
11033 3902 Acked Write 9 Done Sent Done false
11034 3903 Acked Read 3 Done Sent Done false
11035 3904 Acked Read 5 Done Sent Done false
11036 3905 Acked Flush 0 Done Sent Done false
11037 3906 Acked Write 3 Done Sent Done false
11038 3907 Acked Read 7 Done Sent Done false
11039 3908 Acked Flush 0 Done Sent Done false
11040 3909 Acked Write 4 Done Sent Done false
11041 3910 Acked Read 10 Done Sent Done false
11042 3911 Acked Write 5 Done Sent Done false
11043 3912 Acked Write 7 Done Sent Done false
11044 3913 Acked Write 6 Done Sent Done false
11045 3914 Acked Flush 0 Done Sent Done false
11046 3915 Acked Flush 0 Done Sent Done false
11047 3916 Acked Flush 0 Done Sent Done false
11048 3917 Acked Read 5 Done Sent Done false
11049 3918 Acked Write 3 Done Sent Done false
11050 3919 Acked Read 7 Done Sent Done false
11051 3920 Acked Write 7 Done Sent Done false
11052 3921 Acked Read 4 Done Sent Done false
11053 3922 Acked Read 6 Done Sent Done false
11054 3923 Acked Write 4 Done Sent Done false
11055 3924 Acked Read 8 Done Sent Done false
11056 3925 Acked Read 3 Done Sent Done false
11057 3926 Acked Flush 0 Done Sent Done false
11058 3927 Acked Read 7 Done Sent Done false
11059 3928 Acked Write 10 Done Sent Done false
11060 3929 Acked Write 5 Done Sent Done false
11061 3930 Acked Write 7 Done Sent Done false
11062 3931 Acked Flush 0 Done Sent Done false
11063 3932 Acked Read 1 Done Sent Done false
11064 3933 Acked Write 2 Done Sent Done false
11065 3934 Acked Write 6 Done Sent Done false
11066 3935 Acked Write 8 Done Sent Done false
11067 3936 Acked Write 5 Done Sent Done false
11068 3937 Acked Read 2 Done Sent Done false
11069 3938 Acked Read 5 Done Sent Done false
11070 3939 Acked Write 4 Done Sent Done false
11071 3940 Acked Read 2 Done Sent Done false
11072 3941 Acked Flush 0 Done Sent Done false
11073 3942 Acked Read 10 Done Sent Done false
11074 3943 Acked Flush 0 Done Sent Done false
11075 3944 Acked Read 2 Done Sent Done false
11076 3945 Acked Write 10 Done Sent Done false
11077 3946 Acked Read 4 Done Sent Done false
11078 3947 Acked Write 2 Done Sent Done false
11079 3948 Acked Write 3 Done Sent Done false
11080 3949 Acked Write 9 Done Sent Done false
11081 3950 Acked Write 5 Done Sent Done false
11082 3951 Acked Write 9 Done Sent Done false
11083 3952 Acked Write 8 Done Sent Done false
11084 3953 Acked Write 6 Done Sent Done false
11085 3954 Acked Write 3 Done Sent Done false
11086 3955 Acked Write 1 Done Sent Done false
11087 3956 Acked Flush 0 Done Sent Done false
11088 3957 Acked Read 6 Done Sent Done false
11089 3958 Acked Read 9 Done Sent Done false
11090 3959 Acked Read 9 Done Sent Done false
11091 3960 Acked Write 9 Done Sent Done false
11092 3961 Acked Write 1 Done Sent Done false
11093 3962 Acked Read 2 Done Sent Done false
11094 3963 Acked Write 2 Done Sent Done false
11095 3964 Acked Flush 0 Done Sent Done false
11096 3965 Acked Write 9 Done Sent Done false
11097 3966 Acked Write 7 Done Sent Done false
11098 3967 Acked Write 4 Done Sent Done false
11099 3968 Acked Read 4 Done Sent Done false
11100 3969 Acked Read 1 Done Sent Done false
11101 3970 Acked Read 5 Done Sent Done false
11102 3971 Acked Flush 0 Done Sent Done false
11103 3972 Acked Read 3 Done Sent Done false
11104 3973 Acked Write 6 Done Sent Done false
11105 3974 Acked Read 5 Done Sent Done false
11106 3975 Acked Write 7 Done Sent Done false
11107 3976 Acked Write 7 Done Sent Done false
11108 3977 Acked Read 7 Done Sent Done false
11109 3978 Acked Read 3 Done Sent Done false
11110 3979 Acked Write 4 Done Sent Done false
11111 3980 Acked Read 10 Done Sent Done false
11112 3981 Acked Write 6 Done Sent Done false
11113 3982 Acked Write 6 Done Sent Done false
11114 3983 Acked Read 8 Done Sent Done false
11115 3984 Acked Read 4 Done Sent Done false
11116 3985 Acked Read 4 Done Sent Done false
11117 3986 Acked Write 7 Done Sent Done false
11118 3987 Acked Read 8 Done Sent Done false
11119 3988 Acked Write 8 Done Sent Done false
11120 3989 Acked Read 9 Done Sent Done false
11121 3990 Acked Read 1 Done Sent Done false
11122 3991 Acked Flush 0 Done Sent Done false
11123 3992 Acked Write 9 Done Sent Done false
11124 3993 Acked Write 5 Done Sent Done false
11125 3994 Acked Write 5 Done Sent Done false
11126 3995 Acked Write 10 Done Sent Done false
11127 3996 Acked Flush 0 Done Sent Done false
11128 3997 Acked Flush 0 Done Sent Done false
11129 3998 Acked Read 3 Done Sent Done false
11130 3999 Acked Read 2 Done Sent Done false
11131 4000 Acked Read 9 Done Sent Done false
11132 4001 Acked Read 2 Done Sent Done false
11133 4002 Acked Read 10 Done Sent Done false
11134 4003 Acked Write 8 Done Sent Done false
11135 4004 Acked Write 1 Done Sent Done false
11136 4005 Acked Write 5 Done Sent Done false
11137 4006 Acked Flush 0 Done Sent Done false
11138 4007 Acked Read 2 Done Sent Done false
11139 4008 Acked Flush 0 Done Sent Done false
11140 4009 Acked Read 9 Done Sent Done false
11141 4010 Acked Write 10 Done Sent Done false
11142 4011 Acked Read 3 Done Sent Done false
11143 4012 Acked Read 5 Done Sent Done false
11144 4013 Acked Read 6 Done Sent Done false
11145 4014 Acked Write 10 Done Sent Done false
11146 4015 Acked Write 6 Done Sent Done false
11147 4016 Acked Read 4 Done Sent Done false
11148 4017 Acked Flush 0 Done Sent Done false
11149 STATES DS:0 DS:1 DS:2 TOTAL
11150 Sent 0 301 0 301
11151 Done 301 0 301 602
11152 Skipped 0 0 0 0
11153 Error 0 0 0 0
11154 Last Flush: Some(JobId(4017)) Some(JobId(3716)) Some(JobId(4017))
11155 Downstairs last five completed: 3716 3715 3714 3713 3712
11156 Upstairs last five completed: 4017 4016 4015 4014 4013
11157 CLIENT: Up:0 ds:301 act:2
111582025-01-23T18:50:13.199ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
111592025-01-23T18:50:13.199ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
111602025-01-23T18:50:13.199ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
111612025-01-23T18:50:13.199ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
111622025-01-23T18:50:13.199ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
11163 [8820] got start action
11164 [8820] I am going to restart
11165 Starting downstairs at port 8820
11166 Make output file at "/tmp/dsc/downstairs-8820.txt"
11167 [8820][1] reports Starting
11168 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1210)
11169 [8820][1] reports Running
11170 ----------------------------------------------------------------
11171 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11172 States: Active Offline Active
11173 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11174 3717 Acked Write 3 Done Sent Done false
11175 3718 Acked Read 10 Done Sent Done false
11176 3719 Acked Flush 0 Done Sent Done false
11177 3720 Acked Read 5 Done Sent Done false
11178 3721 Acked Read 9 Done Sent Done false
11179 3722 Acked Read 4 Done Sent Done false
11180 3723 Acked Write 6 Done Sent Done false
11181 3724 Acked Write 2 Done Sent Done false
11182 3725 Acked Read 4 Done Sent Done false
11183 3726 Acked Write 6 Done Sent Done false
11184 3727 Acked Write 3 Done Sent Done false
11185 3728 Acked Read 10 Done Sent Done false
11186 3729 Acked Read 6 Done Sent Done false
11187 3730 Acked Write 5 Done Sent Done false
11188 3731 Acked Write 5 Done Sent Done false
11189 3732 Acked Read 9 Done Sent Done false
11190 3733 Acked Write 5 Done Sent Done false
11191 3734 Acked Write 9 Done Sent Done false
11192 3735 Acked Write 5 Done Sent Done false
11193 3736 Acked Read 9 Done Sent Done false
11194 3737 Acked Flush 0 Done Sent Done false
11195 3738 Acked Read 3 Done Sent Done false
11196 3739 Acked Write 5 Done Sent Done false
11197 3740 Acked Write 5 Done Sent Done false
11198 3741 Acked Write 6 Done Sent Done false
11199 3742 Acked Write 5 Done Sent Done false
11200 3743 Acked Write 9 Done Sent Done false
11201 3744 Acked Flush 0 Done Sent Done false
11202 3745 Acked Write 2 Done Sent Done false
11203 3746 Acked Read 10 Done Sent Done false
11204 3747 Acked Read 3 Done Sent Done false
11205 3748 Acked Read 3 Done Sent Done false
11206 3749 Acked Read 8 Done Sent Done false
11207 3750 Acked Flush 0 Done Sent Done false
11208 3751 Acked Read 6 Done Sent Done false
11209 3752 Acked Read 10 Done Sent Done false
11210 3753 Acked Write 5 Done Sent Done false
11211 3754 Acked Read 10 Done Sent Done false
11212 3755 Acked Read 3 Done Sent Done false
11213 3756 Acked Write 3 Done Sent Done false
11214 3757 Acked Read 9 Done Sent Done false
11215 3758 Acked Read 1 Done Sent Done false
11216 3759 Acked Read 4 Done Sent Done false
11217 3760 Acked Flush 0 Done Sent Done false
11218 3761 Acked Read 8 Done Sent Done false
11219 3762 Acked Read 2 Done Sent Done false
11220 3763 Acked Read 1 Done Sent Done false
11221 3764 Acked Write 7 Done Sent Done false
11222 3765 Acked Write 10 Done Sent Done false
11223 3766 Acked Read 1 Done Sent Done false
11224 3767 Acked Write 7 Done Sent Done false
11225 3768 Acked Write 1 Done Sent Done false
11226 3769 Acked Write 2 Done Sent Done false
11227 3770 Acked Read 2 Done Sent Done false
11228 3771 Acked Write 4 Done Sent Done false
11229 3772 Acked Read 6 Done Sent Done false
11230 3773 Acked Read 9 Done Sent Done false
11231 3774 Acked Read 1 Done Sent Done false
11232 3775 Acked Read 3 Done Sent Done false
11233 3776 Acked Write 4 Done Sent Done false
11234 3777 Acked Write 8 Done Sent Done false
11235 3778 Acked Write 9 Done Sent Done false
11236 3779 Acked Read 1 Done Sent Done false
11237 3780 Acked Read 3 Done Sent Done false
11238 3781 Acked Write 3 Done Sent Done false
11239 3782 Acked Read 9 Done Sent Done false
11240 3783 Acked Read 6 Done Sent Done false
11241 3784 Acked Read 8 Done Sent Done false
11242 3785 Acked Write 8 Done Sent Done false
11243 3786 Acked Write 1 Done Sent Done false
11244 3787 Acked Write 8 Done Sent Done false
11245 3788 Acked Read 1 Done Sent Done false
11246 3789 Acked Write 10 Done Sent Done false
11247 3790 Acked Read 7 Done Sent Done false
11248 3791 Acked Flush 0 Done Sent Done false
11249 3792 Acked Write 10 Done Sent Done false
11250 3793 Acked Write 5 Done Sent Done false
11251 3794 Acked Read 4 Done Sent Done false
11252 3795 Acked Write 3 Done Sent Done false
11253 3796 Acked Write 3 Done Sent Done false
11254 3797 Acked Read 6 Done Sent Done false
11255 3798 Acked Read 5 Done Sent Done false
11256 3799 Acked Read 10 Done Sent Done false
11257 3800 Acked Read 4 Done Sent Done false
11258 3801 Acked Read 10 Done Sent Done false
11259 3802 Acked Flush 0 Done Sent Done false
11260 3803 Acked Flush 0 Done Sent Done false
11261 3804 Acked Write 5 Done Sent Done false
11262 3805 Acked Write 8 Done Sent Done false
11263 3806 Acked Write 6 Done Sent Done false
11264 3807 Acked Read 3 Done Sent Done false
11265 3808 Acked Write 7 Done Sent Done false
11266 3809 Acked Read 8 Done Sent Done false
11267 3810 Acked Write 3 Done Sent Done false
11268 3811 Acked Flush 0 Done Sent Done false
11269 3812 Acked Read 1 Done Sent Done false
11270 3813 Acked Write 2 Done Sent Done false
11271 3814 Acked Read 3 Done Sent Done false
11272 3815 Acked Read 10 Done Sent Done false
11273 3816 Acked Flush 0 Done Sent Done false
11274 3817 Acked Read 9 Done Sent Done false
11275 3818 Acked Read 6 Done Sent Done false
11276 3819 Acked Write 3 Done Sent Done false
11277 3820 Acked Write 3 Done Sent Done false
11278 3821 Acked Read 10 Done Sent Done false
11279 3822 Acked Read 4 Done Sent Done false
11280 3823 Acked Read 1 Done Sent Done false
11281 3824 Acked Read 6 Done Sent Done false
11282 3825 Acked Flush 0 Done Sent Done false
11283 3826 Acked Write 4 Done Sent Done false
11284 3827 Acked Read 7 Done Sent Done false
11285 3828 Acked Read 3 Done Sent Done false
11286 3829 Acked Write 9 Done Sent Done false
11287 3830 Acked Read 4 Done Sent Done false
11288 3831 Acked Flush 0 Done Sent Done false
11289 3832 Acked Read 2 Done Sent Done false
11290 3833 Acked Read 5 Done Sent Done false
11291 3834 Acked Read 9 Done Sent Done false
11292 3835 Acked Read 1 Done Sent Done false
11293 3836 Acked Write 5 Done Sent Done false
11294 3837 Acked Write 7 Done Sent Done false
11295 3838 Acked Write 5 Done Sent Done false
11296 3839 Acked Read 6 Done Sent Done false
11297 3840 Acked Write 1 Done Sent Done false
11298 3841 Acked Read 4 Done Sent Done false
11299 3842 Acked Read 6 Done Sent Done false
11300 3843 Acked Flush 0 Done Sent Done false
11301 3844 Acked Write 4 Done Sent Done false
11302 3845 Acked Read 1 Done Sent Done false
11303 3846 Acked Read 2 Done Sent Done false
11304 3847 Acked Read 5 Done Sent Done false
11305 3848 Acked Write 7 Done Sent Done false
11306 3849 Acked Read 6 Done Sent Done false
11307 3850 Acked Write 8 Done Sent Done false
11308 3851 Acked Write 4 Done Sent Done false
11309 3852 Acked Write 8 Done Sent Done false
11310 3853 Acked Read 5 Done Sent Done false
11311 3854 Acked Write 4 Done Sent Done false
11312 3855 Acked Read 2 Done Sent Done false
11313 3856 Acked Read 1 Done Sent Done false
11314 3857 Acked Read 4 Done Sent Done false
11315 3858 Acked Read 10 Done Sent Done false
11316 3859 Acked Read 3 Done Sent Done false
11317 3860 Acked Write 3 Done Sent Done false
11318 3861 Acked Write 2 Done Sent Done false
11319 3862 Acked Read 3 Done Sent Done false
11320 3863 Acked Read 1 Done Sent Done false
11321 3864 Acked Read 2 Done Sent Done false
11322 3865 Acked Read 3 Done Sent Done false
11323 3866 Acked Read 3 Done Sent Done false
11324 3867 Acked Write 4 Done Sent Done false
11325 3868 Acked Write 6 Done Sent Done false
11326 3869 Acked Write 6 Done Sent Done false
11327 3870 Acked Read 8 Done Sent Done false
11328 3871 Acked Write 2 Done Sent Done false
11329 3872 Acked Read 4 Done Sent Done false
11330 3873 Acked Read 9 Done Sent Done false
11331 3874 Acked Read 9 Done Sent Done false
11332 3875 Acked Flush 0 Done Sent Done false
11333 3876 Acked Read 8 Done Sent Done false
11334 3877 Acked Read 10 Done Sent Done false
11335 3878 Acked Write 5 Done Sent Done false
11336 3879 Acked Read 3 Done Sent Done false
11337 3880 Acked Write 1 Done Sent Done false
11338 3881 Acked Write 10 Done Sent Done false
11339 3882 Acked Write 3 Done Sent Done false
11340 3883 Acked Read 7 Done Sent Done false
11341 3884 Acked Write 1 Done Sent Done false
11342 3885 Acked Read 1 Done Sent Done false
11343 3886 Acked Read 4 Done Sent Done false
11344 3887 Acked Write 8 Done Sent Done false
11345 3888 Acked Flush 0 Done Sent Done false
11346 3889 Acked Read 1 Done Sent Done false
11347 3890 Acked Read 7 Done Sent Done false
11348 3891 Acked Write 7 Done Sent Done false
11349 3892 Acked Read 1 Done Sent Done false
11350 3893 Acked Read 6 Done Sent Done false
11351 3894 Acked Read 8 Done Sent Done false
11352 3895 Acked Write 10 Done Sent Done false
11353 3896 Acked Write 10 Done Sent Done false
11354 3897 Acked Write 8 Done Sent Done false
11355 3898 Acked Flush 0 Done Sent Done false
11356 3899 Acked Read 3 Done Sent Done false
11357 3900 Acked Write 6 Done Sent Done false
11358 3901 Acked Write 8 Done Sent Done false
11359 3902 Acked Write 9 Done Sent Done false
11360 3903 Acked Read 3 Done Sent Done false
11361 3904 Acked Read 5 Done Sent Done false
11362 3905 Acked Flush 0 Done Sent Done false
11363 3906 Acked Write 3 Done Sent Done false
11364 3907 Acked Read 7 Done Sent Done false
11365 3908 Acked Flush 0 Done Sent Done false
11366 3909 Acked Write 4 Done Sent Done false
11367 3910 Acked Read 10 Done Sent Done false
11368 3911 Acked Write 5 Done Sent Done false
11369 3912 Acked Write 7 Done Sent Done false
11370 3913 Acked Write 6 Done Sent Done false
11371 3914 Acked Flush 0 Done Sent Done false
11372 3915 Acked Flush 0 Done Sent Done false
11373 3916 Acked Flush 0 Done Sent Done false
11374 3917 Acked Read 5 Done Sent Done false
11375 3918 Acked Write 3 Done Sent Done false
11376 3919 Acked Read 7 Done Sent Done false
11377 3920 Acked Write 7 Done Sent Done false
11378 3921 Acked Read 4 Done Sent Done false
11379 3922 Acked Read 6 Done Sent Done false
11380 3923 Acked Write 4 Done Sent Done false
11381 3924 Acked Read 8 Done Sent Done false
11382 3925 Acked Read 3 Done Sent Done false
11383 3926 Acked Flush 0 Done Sent Done false
11384 3927 Acked Read 7 Done Sent Done false
11385 3928 Acked Write 10 Done Sent Done false
11386 3929 Acked Write 5 Done Sent Done false
11387 3930 Acked Write 7 Done Sent Done false
11388 3931 Acked Flush 0 Done Sent Done false
11389 3932 Acked Read 1 Done Sent Done false
11390 3933 Acked Write 2 Done Sent Done false
11391 3934 Acked Write 6 Done Sent Done false
11392 3935 Acked Write 8 Done Sent Done false
11393 3936 Acked Write 5 Done Sent Done false
11394 3937 Acked Read 2 Done Sent Done false
11395 3938 Acked Read 5 Done Sent Done false
11396 3939 Acked Write 4 Done Sent Done false
11397 3940 Acked Read 2 Done Sent Done false
11398 3941 Acked Flush 0 Done Sent Done false
11399 3942 Acked Read 10 Done Sent Done false
11400 3943 Acked Flush 0 Done Sent Done false
11401 3944 Acked Read 2 Done Sent Done false
11402 3945 Acked Write 10 Done Sent Done false
11403 3946 Acked Read 4 Done Sent Done false
11404 3947 Acked Write 2 Done Sent Done false
11405 3948 Acked Write 3 Done Sent Done false
11406 3949 Acked Write 9 Done Sent Done false
11407 3950 Acked Write 5 Done Sent Done false
11408 3951 Acked Write 9 Done Sent Done false
11409 3952 Acked Write 8 Done Sent Done false
11410 3953 Acked Write 6 Done Sent Done false
11411 3954 Acked Write 3 Done Sent Done false
11412 3955 Acked Write 1 Done Sent Done false
11413 3956 Acked Flush 0 Done Sent Done false
11414 3957 Acked Read 6 Done Sent Done false
11415 3958 Acked Read 9 Done Sent Done false
11416 3959 Acked Read 9 Done Sent Done false
11417 3960 Acked Write 9 Done Sent Done false
11418 3961 Acked Write 1 Done Sent Done false
11419 3962 Acked Read 2 Done Sent Done false
11420 3963 Acked Write 2 Done Sent Done false
11421 3964 Acked Flush 0 Done Sent Done false
11422 3965 Acked Write 9 Done Sent Done false
11423 3966 Acked Write 7 Done Sent Done false
11424 3967 Acked Write 4 Done Sent Done false
11425 3968 Acked Read 4 Done Sent Done false
11426 3969 Acked Read 1 Done Sent Done false
11427 3970 Acked Read 5 Done Sent Done false
11428 3971 Acked Flush 0 Done Sent Done false
11429 3972 Acked Read 3 Done Sent Done false
11430 3973 Acked Write 6 Done Sent Done false
11431 3974 Acked Read 5 Done Sent Done false
11432 3975 Acked Write 7 Done Sent Done false
11433 3976 Acked Write 7 Done Sent Done false
11434 3977 Acked Read 7 Done Sent Done false
11435 3978 Acked Read 3 Done Sent Done false
11436 3979 Acked Write 4 Done Sent Done false
11437 3980 Acked Read 10 Done Sent Done false
11438 3981 Acked Write 6 Done Sent Done false
11439 3982 Acked Write 6 Done Sent Done false
11440 3983 Acked Read 8 Done Sent Done false
11441 3984 Acked Read 4 Done Sent Done false
11442 3985 Acked Read 4 Done Sent Done false
11443 3986 Acked Write 7 Done Sent Done false
11444 3987 Acked Read 8 Done Sent Done false
11445 3988 Acked Write 8 Done Sent Done false
11446 3989 Acked Read 9 Done Sent Done false
11447 3990 Acked Read 1 Done Sent Done false
11448 3991 Acked Flush 0 Done Sent Done false
11449 3992 Acked Write 9 Done Sent Done false
11450 3993 Acked Write 5 Done Sent Done false
11451 3994 Acked Write 5 Done Sent Done false
11452 3995 Acked Write 10 Done Sent Done false
11453 3996 Acked Flush 0 Done Sent Done false
11454 3997 Acked Flush 0 Done Sent Done false
11455 3998 Acked Read 3 Done Sent Done false
11456 3999 Acked Read 2 Done Sent Done false
11457 4000 Acked Read 9 Done Sent Done false
11458 4001 Acked Read 2 Done Sent Done false
11459 4002 Acked Read 10 Done Sent Done false
11460 4003 Acked Write 8 Done Sent Done false
11461 4004 Acked Write 1 Done Sent Done false
11462 4005 Acked Write 5 Done Sent Done false
11463 4006 Acked Flush 0 Done Sent Done false
11464 4007 Acked Read 2 Done Sent Done false
11465 4008 Acked Flush 0 Done Sent Done false
11466 4009 Acked Read 9 Done Sent Done false
11467 4010 Acked Write 10 Done Sent Done false
11468 4011 Acked Read 3 Done Sent Done false
11469 4012 Acked Read 5 Done Sent Done false
11470 4013 Acked Read 6 Done Sent Done false
11471 4014 Acked Write 10 Done Sent Done false
11472 4015 Acked Write 6 Done Sent Done false
11473 4016 Acked Read 4 Done Sent Done false
11474 4017 Acked Flush 0 Done Sent Done false
11475 STATES DS:0 DS:1 DS:2 TOTAL
11476 Sent 0 301 0 301
11477 Done 301 0 301 602
11478 Skipped 0 0 0 0
11479 Error 0 0 0 0
11480 Last Flush: Some(JobId(4017)) Some(JobId(3716)) Some(JobId(4017))
11481 Downstairs last five completed: 3716 3715 3714 3713 3712
11482 Upstairs last five completed: 4017 4016 4015 4014 4013
11483 CLIENT: Up:0 ds:301 act:2
11484 ----------------------------------------------------------------
11485 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11486 States: Active Offline Active
11487 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11488 3717 Acked Write 3 Done Sent Done false
11489 3718 Acked Read 10 Done Sent Done false
11490 3719 Acked Flush 0 Done Sent Done false
11491 3720 Acked Read 5 Done Sent Done false
11492 3721 Acked Read 9 Done Sent Done false
11493 3722 Acked Read 4 Done Sent Done false
11494 3723 Acked Write 6 Done Sent Done false
11495 3724 Acked Write 2 Done Sent Done false
11496 3725 Acked Read 4 Done Sent Done false
11497 3726 Acked Write 6 Done Sent Done false
11498 3727 Acked Write 3 Done Sent Done false
11499 3728 Acked Read 10 Done Sent Done false
11500 3729 Acked Read 6 Done Sent Done false
11501 3730 Acked Write 5 Done Sent Done false
11502 3731 Acked Write 5 Done Sent Done false
11503 3732 Acked Read 9 Done Sent Done false
11504 3733 Acked Write 5 Done Sent Done false
11505 3734 Acked Write 9 Done Sent Done false
11506 3735 Acked Write 5 Done Sent Done false
11507 3736 Acked Read 9 Done Sent Done false
11508 3737 Acked Flush 0 Done Sent Done false
11509 3738 Acked Read 3 Done Sent Done false
11510 3739 Acked Write 5 Done Sent Done false
11511 3740 Acked Write 5 Done Sent Done false
11512 3741 Acked Write 6 Done Sent Done false
11513 3742 Acked Write 5 Done Sent Done false
11514 3743 Acked Write 9 Done Sent Done false
11515 3744 Acked Flush 0 Done Sent Done false
11516 3745 Acked Write 2 Done Sent Done false
11517 3746 Acked Read 10 Done Sent Done false
11518 3747 Acked Read 3 Done Sent Done false
11519 3748 Acked Read 3 Done Sent Done false
11520 3749 Acked Read 8 Done Sent Done false
11521 3750 Acked Flush 0 Done Sent Done false
11522 3751 Acked Read 6 Done Sent Done false
11523 3752 Acked Read 10 Done Sent Done false
11524 3753 Acked Write 5 Done Sent Done false
11525 3754 Acked Read 10 Done Sent Done false
11526 3755 Acked Read 3 Done Sent Done false
11527 3756 Acked Write 3 Done Sent Done false
11528 3757 Acked Read 9 Done Sent Done false
11529 3758 Acked Read 1 Done Sent Done false
11530 3759 Acked Read 4 Done Sent Done false
11531 3760 Acked Flush 0 Done Sent Done false
11532 3761 Acked Read 8 Done Sent Done false
11533 3762 Acked Read 2 Done Sent Done false
11534 3763 Acked Read 1 Done Sent Done false
11535 3764 Acked Write 7 Done Sent Done false
11536 3765 Acked Write 10 Done Sent Done false
11537 3766 Acked Read 1 Done Sent Done false
11538 3767 Acked Write 7 Done Sent Done false
11539 3768 Acked Write 1 Done Sent Done false
11540 3769 Acked Write 2 Done Sent Done false
11541 3770 Acked Read 2 Done Sent Done false
11542 3771 Acked Write 4 Done Sent Done false
11543 3772 Acked Read 6 Done Sent Done false
11544 3773 Acked Read 9 Done Sent Done false
11545 3774 Acked Read 1 Done Sent Done false
11546 3775 Acked Read 3 Done Sent Done false
11547 3776 Acked Write 4 Done Sent Done false
11548 3777 Acked Write 8 Done Sent Done false
11549 3778 Acked Write 9 Done Sent Done false
11550 3779 Acked Read 1 Done Sent Done false
11551 3780 Acked Read 3 Done Sent Done false
11552 3781 Acked Write 3 Done Sent Done false
11553 3782 Acked Read 9 Done Sent Done false
11554 3783 Acked Read 6 Done Sent Done false
11555 3784 Acked Read 8 Done Sent Done false
11556 3785 Acked Write 8 Done Sent Done false
11557 3786 Acked Write 1 Done Sent Done false
11558 3787 Acked Write 8 Done Sent Done false
11559 3788 Acked Read 1 Done Sent Done false
11560 3789 Acked Write 10 Done Sent Done false
11561 3790 Acked Read 7 Done Sent Done false
11562 3791 Acked Flush 0 Done Sent Done false
11563 3792 Acked Write 10 Done Sent Done false
11564 3793 Acked Write 5 Done Sent Done false
11565 3794 Acked Read 4 Done Sent Done false
11566 3795 Acked Write 3 Done Sent Done false
11567 3796 Acked Write 3 Done Sent Done false
11568 3797 Acked Read 6 Done Sent Done false
11569 3798 Acked Read 5 Done Sent Done false
11570 3799 Acked Read 10 Done Sent Done false
11571 3800 Acked Read 4 Done Sent Done false
11572 3801 Acked Read 10 Done Sent Done false
11573 3802 Acked Flush 0 Done Sent Done false
11574 3803 Acked Flush 0 Done Sent Done false
11575 3804 Acked Write 5 Done Sent Done false
11576 3805 Acked Write 8 Done Sent Done false
11577 3806 Acked Write 6 Done Sent Done false
11578 3807 Acked Read 3 Done Sent Done false
11579 3808 Acked Write 7 Done Sent Done false
11580 3809 Acked Read 8 Done Sent Done false
11581 3810 Acked Write 3 Done Sent Done false
11582 3811 Acked Flush 0 Done Sent Done false
11583 3812 Acked Read 1 Done Sent Done false
11584 3813 Acked Write 2 Done Sent Done false
11585 3814 Acked Read 3 Done Sent Done false
11586 3815 Acked Read 10 Done Sent Done false
11587 3816 Acked Flush 0 Done Sent Done false
11588 3817 Acked Read 9 Done Sent Done false
11589 3818 Acked Read 6 Done Sent Done false
11590 3819 Acked Write 3 Done Sent Done false
11591 3820 Acked Write 3 Done Sent Done false
11592 3821 Acked Read 10 Done Sent Done false
11593 3822 Acked Read 4 Done Sent Done false
11594 3823 Acked Read 1 Done Sent Done false
11595 3824 Acked Read 6 Done Sent Done false
11596 3825 Acked Flush 0 Done Sent Done false
11597 3826 Acked Write 4 Done Sent Done false
11598 3827 Acked Read 7 Done Sent Done false
11599 3828 Acked Read 3 Done Sent Done false
11600 3829 Acked Write 9 Done Sent Done false
11601 3830 Acked Read 4 Done Sent Done false
11602 3831 Acked Flush 0 Done Sent Done false
11603 3832 Acked Read 2 Done Sent Done false
11604 3833 Acked Read 5 Done Sent Done false
11605 3834 Acked Read 9 Done Sent Done false
11606 3835 Acked Read 1 Done Sent Done false
11607 3836 Acked Write 5 Done Sent Done false
11608 3837 Acked Write 7 Done Sent Done false
11609 3838 Acked Write 5 Done Sent Done false
11610 3839 Acked Read 6 Done Sent Done false
11611 3840 Acked Write 1 Done Sent Done false
11612 3841 Acked Read 4 Done Sent Done false
11613 3842 Acked Read 6 Done Sent Done false
11614 3843 Acked Flush 0 Done Sent Done false
11615 3844 Acked Write 4 Done Sent Done false
11616 3845 Acked Read 1 Done Sent Done false
11617 3846 Acked Read 2 Done Sent Done false
11618 3847 Acked Read 5 Done Sent Done false
11619 3848 Acked Write 7 Done Sent Done false
11620 3849 Acked Read 6 Done Sent Done false
11621 3850 Acked Write 8 Done Sent Done false
11622 3851 Acked Write 4 Done Sent Done false
11623 3852 Acked Write 8 Done Sent Done false
11624 3853 Acked Read 5 Done Sent Done false
11625 3854 Acked Write 4 Done Sent Done false
11626 3855 Acked Read 2 Done Sent Done false
11627 3856 Acked Read 1 Done Sent Done false
11628 3857 Acked Read 4 Done Sent Done false
11629 3858 Acked Read 10 Done Sent Done false
11630 3859 Acked Read 3 Done Sent Done false
11631 3860 Acked Write 3 Done Sent Done false
11632 3861 Acked Write 2 Done Sent Done false
11633 3862 Acked Read 3 Done Sent Done false
11634 3863 Acked Read 1 Done Sent Done false
11635 3864 Acked Read 2 Done Sent Done false
11636 3865 Acked Read 3 Done Sent Done false
11637 3866 Acked Read 3 Done Sent Done false
11638 3867 Acked Write 4 Done Sent Done false
11639 3868 Acked Write 6 Done Sent Done false
11640 3869 Acked Write 6 Done Sent Done false
11641 3870 Acked Read 8 Done Sent Done false
11642 3871 Acked Write 2 Done Sent Done false
11643 3872 Acked Read 4 Done Sent Done false
11644 3873 Acked Read 9 Done Sent Done false
11645 3874 Acked Read 9 Done Sent Done false
11646 3875 Acked Flush 0 Done Sent Done false
11647 3876 Acked Read 8 Done Sent Done false
11648 3877 Acked Read 10 Done Sent Done false
11649 3878 Acked Write 5 Done Sent Done false
11650 3879 Acked Read 3 Done Sent Done false
11651 3880 Acked Write 1 Done Sent Done false
11652 3881 Acked Write 10 Done Sent Done false
11653 3882 Acked Write 3 Done Sent Done false
11654 3883 Acked Read 7 Done Sent Done false
11655 3884 Acked Write 1 Done Sent Done false
11656 3885 Acked Read 1 Done Sent Done false
11657 3886 Acked Read 4 Done Sent Done false
11658 3887 Acked Write 8 Done Sent Done false
11659 3888 Acked Flush 0 Done Sent Done false
11660 3889 Acked Read 1 Done Sent Done false
11661 3890 Acked Read 7 Done Sent Done false
11662 3891 Acked Write 7 Done Sent Done false
11663 3892 Acked Read 1 Done Sent Done false
11664 3893 Acked Read 6 Done Sent Done false
11665 3894 Acked Read 8 Done Sent Done false
11666 3895 Acked Write 10 Done Sent Done false
11667 3896 Acked Write 10 Done Sent Done false
11668 3897 Acked Write 8 Done Sent Done false
11669 3898 Acked Flush 0 Done Sent Done false
11670 3899 Acked Read 3 Done Sent Done false
11671 3900 Acked Write 6 Done Sent Done false
11672 3901 Acked Write 8 Done Sent Done false
11673 3902 Acked Write 9 Done Sent Done false
11674 3903 Acked Read 3 Done Sent Done false
11675 3904 Acked Read 5 Done Sent Done false
11676 3905 Acked Flush 0 Done Sent Done false
11677 3906 Acked Write 3 Done Sent Done false
11678 3907 Acked Read 7 Done Sent Done false
11679 3908 Acked Flush 0 Done Sent Done false
11680 3909 Acked Write 4 Done Sent Done false
11681 3910 Acked Read 10 Done Sent Done false
11682 3911 Acked Write 5 Done Sent Done false
11683 3912 Acked Write 7 Done Sent Done false
11684 3913 Acked Write 6 Done Sent Done false
11685 3914 Acked Flush 0 Done Sent Done false
11686 3915 Acked Flush 0 Done Sent Done false
11687 3916 Acked Flush 0 Done Sent Done false
11688 3917 Acked Read 5 Done Sent Done false
11689 3918 Acked Write 3 Done Sent Done false
11690 3919 Acked Read 7 Done Sent Done false
11691 3920 Acked Write 7 Done Sent Done false
11692 3921 Acked Read 4 Done Sent Done false
11693 3922 Acked Read 6 Done Sent Done false
11694 3923 Acked Write 4 Done Sent Done false
11695 3924 Acked Read 8 Done Sent Done false
11696 3925 Acked Read 3 Done Sent Done false
11697 3926 Acked Flush 0 Done Sent Done false
11698 3927 Acked Read 7 Done Sent Done false
11699 3928 Acked Write 10 Done Sent Done false
11700 3929 Acked Write 5 Done Sent Done false
11701 3930 Acked Write 7 Done Sent Done false
11702 3931 Acked Flush 0 Done Sent Done false
11703 3932 Acked Read 1 Done Sent Done false
11704 3933 Acked Write 2 Done Sent Done false
11705 3934 Acked Write 6 Done Sent Done false
11706 3935 Acked Write 8 Done Sent Done false
11707 3936 Acked Write 5 Done Sent Done false
11708 3937 Acked Read 2 Done Sent Done false
11709 3938 Acked Read 5 Done Sent Done false
11710 3939 Acked Write 4 Done Sent Done false
11711 3940 Acked Read 2 Done Sent Done false
11712 3941 Acked Flush 0 Done Sent Done false
11713 3942 Acked Read 10 Done Sent Done false
11714 3943 Acked Flush 0 Done Sent Done false
11715 3944 Acked Read 2 Done Sent Done false
11716 3945 Acked Write 10 Done Sent Done false
11717 3946 Acked Read 4 Done Sent Done false
11718 3947 Acked Write 2 Done Sent Done false
11719 3948 Acked Write 3 Done Sent Done false
11720 3949 Acked Write 9 Done Sent Done false
11721 3950 Acked Write 5 Done Sent Done false
11722 3951 Acked Write 9 Done Sent Done false
11723 3952 Acked Write 8 Done Sent Done false
11724 3953 Acked Write 6 Done Sent Done false
11725 3954 Acked Write 3 Done Sent Done false
11726 3955 Acked Write 1 Done Sent Done false
11727 3956 Acked Flush 0 Done Sent Done false
11728 3957 Acked Read 6 Done Sent Done false
11729 3958 Acked Read 9 Done Sent Done false
11730 3959 Acked Read 9 Done Sent Done false
11731 3960 Acked Write 9 Done Sent Done false
11732 3961 Acked Write 1 Done Sent Done false
11733 3962 Acked Read 2 Done Sent Done false
11734 3963 Acked Write 2 Done Sent Done false
11735 3964 Acked Flush 0 Done Sent Done false
11736 3965 Acked Write 9 Done Sent Done false
11737 3966 Acked Write 7 Done Sent Done false
11738 3967 Acked Write 4 Done Sent Done false
11739 3968 Acked Read 4 Done Sent Done false
11740 3969 Acked Read 1 Done Sent Done false
11741 3970 Acked Read 5 Done Sent Done false
11742 3971 Acked Flush 0 Done Sent Done false
11743 3972 Acked Read 3 Done Sent Done false
11744 3973 Acked Write 6 Done Sent Done false
11745 3974 Acked Read 5 Done Sent Done false
11746 3975 Acked Write 7 Done Sent Done false
11747 3976 Acked Write 7 Done Sent Done false
11748 3977 Acked Read 7 Done Sent Done false
11749 3978 Acked Read 3 Done Sent Done false
11750 3979 Acked Write 4 Done Sent Done false
11751 3980 Acked Read 10 Done Sent Done false
11752 3981 Acked Write 6 Done Sent Done false
11753 3982 Acked Write 6 Done Sent Done false
11754 3983 Acked Read 8 Done Sent Done false
11755 3984 Acked Read 4 Done Sent Done false
11756 3985 Acked Read 4 Done Sent Done false
11757 3986 Acked Write 7 Done Sent Done false
11758 3987 Acked Read 8 Done Sent Done false
11759 3988 Acked Write 8 Done Sent Done false
11760 3989 Acked Read 9 Done Sent Done false
11761 3990 Acked Read 1 Done Sent Done false
11762 3991 Acked Flush 0 Done Sent Done false
11763 3992 Acked Write 9 Done Sent Done false
11764 3993 Acked Write 5 Done Sent Done false
11765 3994 Acked Write 5 Done Sent Done false
11766 3995 Acked Write 10 Done Sent Done false
11767 3996 Acked Flush 0 Done Sent Done false
11768 3997 Acked Flush 0 Done Sent Done false
11769 3998 Acked Read 3 Done Sent Done false
11770 3999 Acked Read 2 Done Sent Done false
11771 4000 Acked Read 9 Done Sent Done false
11772 4001 Acked Read 2 Done Sent Done false
11773 4002 Acked Read 10 Done Sent Done false
11774 4003 Acked Write 8 Done Sent Done false
11775 4004 Acked Write 1 Done Sent Done false
11776 4005 Acked Write 5 Done Sent Done false
11777 4006 Acked Flush 0 Done Sent Done false
11778 4007 Acked Read 2 Done Sent Done false
11779 4008 Acked Flush 0 Done Sent Done false
11780 4009 Acked Read 9 Done Sent Done false
11781 4010 Acked Write 10 Done Sent Done false
11782 4011 Acked Read 3 Done Sent Done false
11783 4012 Acked Read 5 Done Sent Done false
11784 4013 Acked Read 6 Done Sent Done false
11785 4014 Acked Write 10 Done Sent Done false
11786 4015 Acked Write 6 Done Sent Done false
11787 4016 Acked Read 4 Done Sent Done false
11788 4017 Acked Flush 0 Done Sent Done false
11789 STATES DS:0 DS:1 DS:2 TOTAL
11790 Sent 0 301 0 301
11791 Done 301 0 301 602
11792 Skipped 0 0 0 0
11793 Error 0 0 0 0
11794 Last Flush: Some(JobId(4017)) Some(JobId(3716)) Some(JobId(4017))
11795 Downstairs last five completed: 3716 3715 3714 3713 3712
11796 Upstairs last five completed: 4017 4016 4015 4014 4013
11797 CLIENT: Up:0 ds:301 act:2
117982025-01-23T18:50:23.200ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
117992025-01-23T18:50:23.200ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56497) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
118002025-01-23T18:50:23.201ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
118012025-01-23T18:50:23.202ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
118022025-01-23T18:50:23.202ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
118032025-01-23T18:50:23.202ZINFOcrucible: send last flush ID to this DS: Some(JobId(3716)) = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
118042025-01-23T18:50:23.202ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
118052025-01-23T18:50:23.202ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(3716)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
118062025-01-23T18:50:23.202ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(3716)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
11807 ----------------------------------------------------------------
11808 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11809 Downstairs last five completed: 4017 4016 4015 4014 4013
11810 Upstairs last five completed: 4017 4016 4015 4014 4013
11811 CLIENT: Up:0 ds:0 act:3
11812 Replay: All jobs finished
11813 Main task has work to do, go find it
11814 got dsc Stop(2)
11815 stop 2
11816 [8830] Got stop action so:false kr:false
11817 Jan 23 18:50:23.439 INFO request completed, latency_us: 306, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 5a039a55-6f31-4217-a46b-6aade95bf241, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
118182025-01-23T18:50:23.439ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
118192025-01-23T18:50:23.439ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
118202025-01-23T18:50:23.439ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
118212025-01-23T18:50:23.439ZINFOcrucible: client task is exiting = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
11822 Jan 23 18:50:23.439 INFO request completed, latency_us: 125, response_code: 200, uri: /state/cid/2, method: GET, req_id: db69b79c-1d9b-4e67-8f24-c3fe1f74c91f, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
11823 [8830] Exited with: ExitStatus(unix_wait_status(9))
11824 [8830][2] reports Exit
11825 Jan 23 18:50:27.441 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/2, method: GET, req_id: e3728538-f604-4012-8576-d2161d321aad, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
11826 001/300 Read block 226 len 16384
11827 002/300 Write block 618 len 36864 data: 3 3 3 3 3 2 2 2 4
11828 003/300 Read block 1270 len 8192
11829 004/300 Read block 1232 len 32768
11830 005/300 Write block 176 len 28672 data: 4 4 6 5 4 4 4
11831 006/300 Write block 4250 len 40960 data: 2 2 2 3 3 3 3 4 4 3
11832 007/300 Flush
11833 008/300 Read block 2175 len 28672
11834 009/300 Read block 1857 len 12288
11835 010/300 Flush
11836 011/300 Flush
11837 012/300 Write block 1487 len 28672 data: 3 3 4 4 5 4 3
11838 013/300 Write block 3108 len 40960 data: 3 3 3 2 2 2 3 3 3 3
11839 014/300 Write block 3268 len 40960 data: 3 3 3 3 3 3 3 2 2 2
11840 015/300 Write block 3652 len 16384 data: 4 5 3 2
11841 016/300 Write block 1852 len 4096 data: 3
11842 017/300 Read block 1271 len 32768
11843 018/300 Write block 1830 len 32768 data: 4 4 4 3 3 3 3 2
11844 019/300 Write block 4642 len 40960 data: 2 3 4 4 4 4 3 3 3 3
11845 020/300 Write block 1091 len 20480 data: 3 3 3 3 5
11846 021/300 Write block 4643 len 4096 data: 4
11847 022/300 Write block 1951 len 36864 data: 2 2 2 2 2 2 2 2 2
11848 023/300 Write block 4149 len 24576 data: 8 8 7 5 6 4
11849 024/300 Read block 2868 len 8192
11850 025/300 Read block 1077 len 24576
11851 026/300 Read block 2603 len 8192
11852 027/300 Read block 4089 len 24576
11853 028/300 Write block 3503 len 40960 data: 2 4 4 4 5 4 4 4 4 4
11854 029/300 Read block 227 len 20480
11855 030/300 Write block 3385 len 24576 data: 4 4 4 3 2 2
11856 031/300 Read block 1 len 12288
11857 032/300 Read block 1038 len 40960
11858 033/300 Flush
11859 034/300 Write block 4383 len 4096 data: 2
11860 035/300 Flush
11861 036/300 Read block 2305 len 24576
11862 037/300 Read block 3899 len 20480
11863 038/300 Flush
11864 039/300 Write block 3509 len 40960 data: 5 5 5 5 4 2 3 3 3 4
11865 040/300 Read block 2877 len 8192
11866 041/300 Write block 2737 len 12288 data: 3 3 3
11867 042/300 Read block 1736 len 12288
11868 043/300 Read block 2311 len 4096
11869 044/300 Write block 854 len 24576 data: 4 4 4 3 3 3
11870 045/300 Write block 2394 len 4096 data: 5
11871 046/300 Read block 450 len 32768
11872 047/300 Write block 1085 len 28672 data: 2 2 2 3 3 3 4
11873 048/300 Read block 3600 len 36864
11874 049/300 Flush
11875 050/300 Read block 830 len 16384
11876 051/300 Read block 3722 len 36864
11877 052/300 Write block 1873 len 16384 data: 4 2 2 2
11878 053/300 Write block 649 len 4096 data: 3
11879 054/300 Write block 4449 len 40960 data: 5 5 5 4 3 3 3 4 4 4
11880 055/300 Write block 2711 len 12288 data: 3 2 2
11881 056/300 Flush
11882 057/300 Read block 4913 len 32768
11883 058/300 Write block 4756 len 20480 data: 5 4 4 4 4
11884 059/300 Read block 3873 len 16384
11885 060/300 Flush
11886 061/300 Read block 3992 len 40960
11887 062/300 Read block 2200 len 8192
11888 063/300 Read block 914 len 24576
11889 064/300 Read block 4761 len 12288
11890 065/300 Flush
11891 066/300 Flush
11892 067/300 Read block 1093 len 32768
11893 068/300 Write block 4884 len 28672 data: 3 3 2 2 2 2 2
11894 069/300 Write block 1030 len 12288 data: 3 3 3
11895 070/300 Read block 2173 len 32768
11896 071/300 Write block 4557 len 32768 data: 2 3 3 3 3 3 3 3
11897 072/300 Write block 3539 len 28672 data: 4 4 3 3 3 3 4
11898 073/300 Write block 985 len 36864 data: 3 3 3 2 2 3 3 3 3
11899 074/300 Write block 894 len 12288 data: 2 3 5
11900 075/300 Read block 2309 len 40960
11901 076/300 Write block 4974 len 4096 data: 2
11902 077/300 Read block 3221 len 20480
11903 078/300 Read block 4987 len 28672
11904 079/300 Write block 2670 len 28672 data: 3 4 4 4 4 4 5
11905 080/300 Read block 625 len 28672
11906 081/300 Read block 1525 len 28672
11907 082/300 Read block 3357 len 8192
11908 083/300 Read block 2404 len 20480
11909 084/300 Read block 3327 len 16384
11910 085/300 Write block 452 len 36864 data: 3 3 4 4 5 6 6 6 5
11911 086/300 Read block 2631 len 16384
11912 087/300 Read block 3343 len 24576
11913 088/300 Read block 4724 len 36864
11914 089/300 Read block 1302 len 36864
11915 090/300 Read block 2527 len 24576
11916 091/300 Write block 3385 len 32768 data: 5 5 5 4 3 3 2 2
11917 092/300 Read block 3301 len 16384
11918 093/300 Read block 3094 len 8192
11919 094/300 Write block 4852 len 20480 data: 3 3 2 2 2
11920 095/300 Read block 2780 len 40960
11921 096/300 Read block 4618 len 32768
11922 097/300 Write block 118 len 36864 data: 4 3 2 2 3 3 3 4 4
11923 098/300 Write block 3656 len 4096 data: 2
11924 099/300 Read block 2817 len 8192
11925 100/300 Write block 1370 len 20480 data: 2 2 2 3 3
11926 101/300 Write block 4447 len 28672 data: 4 5 6 6 6 5 4
11927 102/300 Read block 3916 len 16384
11928 103/300 Flush
11929 104/300 Read block 2875 len 16384
11930 105/300 Read block 739 len 28672
11931 106/300 Write block 496 len 4096 data: 3
11932 107/300 Write block 3180 len 28672 data: 2 2 2 2 2 3 3
11933 108/300 Read block 3591 len 32768
11934 109/300 Write block 2100 len 4096 data: 3
11935 110/300 Read block 1399 len 24576
11936 111/300 Write block 4706 len 4096 data: 3
11937 112/300 Read block 2575 len 8192
11938 113/300 Flush
11939 114/300 Read block 1510 len 40960
11940 115/300 Flush
11941 116/300 Read block 3730 len 32768
11942 117/300 Read block 1313 len 20480
11943 118/300 Flush
11944 119/300 Write block 862 len 32768 data: 2 3 3 3 3 3 3 3
11945 120/300 Read block 2 len 28672
11946 121/300 Write block 4494 len 36864 data: 4 4 3 3 3 4 4 4 4
11947 122/300 Write block 2540 len 20480 data: 6 4 4 4 4
11948 123/300 Write block 3102 len 24576 data: 3 4 4 5 5 5
11949 124/300 Write block 776 len 32768 data: 4 5 6 6 7 6 7 7
11950 125/300 Write block 3849 len 28672 data: 2 3 3 3 3 2 2
11951 126/300 Read block 1230 len 20480
11952 127/300 Write block 2746 len 24576 data: 2 2 3 3 3 3
11953 128/300 Read block 1049 len 8192
11954 129/300 Read block 4371 len 32768
11955 130/300 Read block 2227 len 24576
11956 131/300 Read block 4120 len 20480
11957 132/300 Write block 1475 len 8192 data: 2 3
11958 133/300 Read block 1022 len 32768
11959 134/300 Read block 414 len 28672
11960 135/300 Write block 219 len 36864 data: 3 3 3 3 2 2 2 2 2
11961 136/300 Read block 3028 len 36864
11962 137/300 Read block 1518 len 4096
11963 138/300 Write block 2102 len 32768 data: 4 4 4 4 4 3 3 3
11964 139/300 Write block 3798 len 28672 data: 4 4 3 4 4 3 3
11965 140/300 Write block 3687 len 40960 data: 3 3 3 4 5 4 5 3 3 3
11966 141/300 Read block 2930 len 20480
11967 142/300 Read block 3892 len 40960
11968 143/300 Read block 131 len 20480
11969 144/300 Read block 461 len 20480
11970 145/300 Write block 4625 len 16384 data: 3 3 3 3
11971 146/300 Write block 2559 len 16384 data: 4 4 4 5
11972 147/300 Read block 961 len 24576
11973 148/300 Read block 2977 len 24576
11974 149/300 Read block 856 len 24576
11975 150/300 Read block 120 len 40960
11976 151/300 Read block 2650 len 4096
11977 152/300 Write block 770 len 32768 data: 2 2 2 2 2 3 5 6
11978 153/300 Write block 4047 len 20480 data: 4 4 4 3 3
11979 154/300 Read block 4169 len 12288
11980 155/300 Read block 1878 len 36864
11981 156/300 Read block 4297 len 36864
11982 157/300 Read block 1607 len 36864
11983 158/300 Flush
11984 159/300 Write block 1420 len 32768 data: 5 5 5 5 4 4 4 4
11985 160/300 Read block 4750 len 8192
11986 161/300 Read block 3585 len 16384
11987 162/300 Read block 156 len 4096
11988 163/300 Read block 2225 len 36864
11989 164/300 Read block 1631 len 28672
11990 165/300 Read block 4723 len 32768
11991 166/300 Read block 1616 len 16384
11992 167/300 Read block 4685 len 32768
11993 168/300 Write block 405 len 28672 data: 5 5 5 5 5 6 6
11994 169/300 Write block 3632 len 20480 data: 2 3 3 2 2
11995 170/300 Write block 2621 len 36864 data: 6 5 5 4 4 4 2 4 3
11996 171/300 Read block 848 len 8192
11997 172/300 Write block 2183 len 12288 data: 2 2 2
11998 173/300 Read block 847 len 28672
11999 174/300 Flush
12000 175/300 Read block 3594 len 28672
12001 176/300 Write block 3099 len 40960 data: 2 3 3 4 5 5 6 6 6 4
12002 177/300 Read block 3623 len 28672
12003 178/300 Read block 4415 len 20480
12004 179/300 Read block 3957 len 12288
12005 180/300 Read block 1030 len 20480
12006 181/300 Read block 2068 len 28672
12007 182/300 Read block 2206 len 32768
12008 183/300 Read block 2943 len 4096
12009 184/300 Read block 58 len 4096
12010 185/300 Flush
12011 186/300 Flush
12012 187/300 Read block 1443 len 4096
12013 188/300 Write block 1911 len 20480 data: 2 2 2 2 2
12014 189/300 Read block 4112 len 20480
12015 190/300 Read block 4797 len 40960
12016 191/300 Write block 4350 len 40960 data: 4 3 4 4 3 5 5 5 4 4
12017 192/300 Read block 1459 len 24576
12018 193/300 Write block 2523 len 28672 data: 6 5 2 2 2 2 2
12019 194/300 Write block 3456 len 4096 data: 2
12020 195/300 Read block 3568 len 8192
12021 196/300 Write block 671 len 40960 data: 4 3 4 3 4 4 4 3 3 2
12022 197/300 Read block 288 len 16384
12023 198/300 Write block 1557 len 12288 data: 3 3 4
12024 199/300 Read block 1740 len 36864
12025 200/300 Read block 3290 len 32768
12026 201/300 Write block 939 len 12288 data: 4 3 3
12027 202/300 Write block 3736 len 20480 data: 6 6 6 5 5
12028 203/300 Read block 3083 len 4096
12029 204/300 Read block 591 len 40960
12030 205/300 Read block 3687 len 8192
12031 206/300 Read block 2593 len 36864
12032 207/300 Read block 1233 len 8192
12033 208/300 Write block 270 len 40960 data: 4 4 4 3 3 4 5 6 4 4
12034 209/300 Write block 4951 len 40960 data: 3 3 3 3 3 3 3 3 2 2
12035 210/300 Read block 3523 len 12288
12036 211/300 Write block 4683 len 4096 data: 3
12037 212/300 Write block 4961 len 32768 data: 2 3 3 4 4 3 3 3
12038 213/300 Read block 4345 len 8192
12039 214/300 Flush
12040 215/300 Write block 76 len 16384 data: 5 5 6 5
12041 216/300 Read block 695 len 4096
12042 217/300 Write block 1284 len 40960 data: 4 4 4 4 4 4 4 4 3 3
12043 218/300 Read block 1889 len 40960
12044 219/300 Read block 2173 len 4096
12045 220/300 Read block 1946 len 12288
12046 221/300 Read block 1433 len 8192
12047 222/300 Write block 4142 len 24576 data: 4 4 4 4 6 7
12048 223/300 Flush
12049 224/300 Read block 2460 len 16384
12050 225/300 Write block 1703 len 28672 data: 3 3 3 3 2 2 2
12051 226/300 Read block 3650 len 32768
12052 227/300 Write block 4052 len 40960 data: 3 2 2 2 2 2 2 2 2 2
12053 228/300 Write block 1499 len 32768 data: 3 3 3 2 3 3 5 5
12054 229/300 Write block 1730 len 20480 data: 3 4 4 4 3
12055 230/300 Read block 2445 len 8192
12056 231/300 Write block 3230 len 20480 data: 4 5 6 7 7
12057 232/300 Read block 1898 len 12288
12058 233/300 Write block 3562 len 36864 data: 2 2 2 2 3 4 4 4 3
12059 234/300 Write block 1089 len 36864 data: 4 4 5 4 4 4 6 5 4
12060 235/300 Write block 1304 len 36864 data: 4 4 3 4 4 4 4 3 4
12061 236/300 Write block 944 len 8192 data: 4 4
12062 237/300 Write block 962 len 36864 data: 5 4 4 4 5 5 4 2 2
12063 238/300 Write block 2130 len 12288 data: 4 3 3
12064 239/300 Write block 3878 len 32768 data: 4 4 5 5 4 5 4 4
12065 240/300 Flush
12066 241/300 Write block 3619 len 16384 data: 5 5 5 7
12067 242/300 Read block 585 len 32768
12068 243/300 Read block 2976 len 4096
12069 244/300 Read block 4183 len 4096
12070 245/300 Write block 1041 len 20480 data: 3 3 3 3 4
12071 246/300 Read block 1498 len 16384
12072 247/300 Write block 2359 len 12288 data: 2 3 3
12073 248/300 Read block 2111 len 28672
12074 249/300 Write block 2652 len 20480 data: 3 3 3 4 4
12075 250/300 Flush
12076 251/300 Read block 4003 len 28672
12077 252/300 Flush
12078 253/300 Read block 1383 len 32768
12079 254/300 Read block 3515 len 40960
12080 255/300 Read block 4188 len 4096
12081 256/300 Write block 4561 len 32768 data: 4 4 4 4 3 3 3 4
12082 257/300 Read block 2451 len 28672
12083 258/300 Write block 3045 len 24576 data: 6 6 5 6 6 6
12084 259/300 Write block 3034 len 32768 data: 3 4 4 4 5 6 7 6
12085 260/300 Write block 4016 len 4096 data: 3
12086 261/300 Write block 724 len 36864 data: 2 3 3 3 2 3 4 4 3
12087 262/300 Read block 2385 len 32768
12088 263/300 Write block 2098 len 32768 data: 3 4 4 4 5 5 5 5
12089 264/300 Write block 2884 len 32768 data: 2 3 4 4 5 4 4 5
12090 265/300 Read block 1394 len 40960
12091 266/300 Write block 2675 len 4096 data: 5
12092 267/300 Write block 3785 len 12288 data: 3 2 3
12093 268/300 Write block 3734 len 12288 data: 4 5 7
12094 269/300 Read block 3962 len 12288
12095 270/300 Read block 1362 len 4096
12096 271/300 Read block 4672 len 24576
12097 272/300 Write block 2313 len 40960 data: 5 6 7 4 4 3 4 5 4 4
12098 273/300 Write block 3155 len 4096 data: 2
12099 274/300 Write block 3280 len 28672 data: 2 2 2 3 2 2 3
12100 275/300 Read block 2091 len 32768
12101 276/300 Read block 478 len 4096
12102 277/300 Read block 2038 len 4096
12103 278/300 Write block 1246 len 32768 data: 4 3 4 4 4 5 5 5
12104 279/300 Write block 404 len 8192 data: 5 6
12105 280/300 Read block 1791 len 12288
12106 281/300 Read block 3606 len 20480
12107 282/300 Write block 4639 len 36864 data: 3 3 2 3 5 5 5 5 5
12108 283/300 Read block 2635 len 40960
12109 284/300 Write block 433 len 4096 data: 5
12110 285/300 Write block 4637 len 4096 data: 3
12111 286/300 Read block 3586 len 4096
12112 287/300 Write block 3108 len 4096 data: 5
12113 288/300 Write block 1658 len 12288 data: 3 3 4
12114 289/300 Write block 2459 len 24576 data: 4 5 5 4 3 3
12115 290/300 Read block 1786 len 28672
12116 291/300 Read block 4497 len 8192
12117 292/300 Write block 4993 len 8192 data: 2 2
12118 293/300 Read block 3085 len 8192
12119 294/300 Flush
12120 295/300 Write block 158 len 12288 data: 2 2 2
12121 296/300 Read block 4523 len 32768
12122 297/300 Read block 2489 len 24576
12123 298/300 Read block 923 len 8192
12124 299/300 Write block 3067 len 12288 data: 2 2 2
12125 300/300 Write block 2348 len 8192 data: 2 2
12126 301/300 Read block 301 len 24576
12127 Main task has work to do, go find it
12128 got dsc Start(2)
12129 start 2
12130 Jan 23 18:50:27.680 INFO request completed, latency_us: 194, response_code: 204, uri: /start/cid/2, method: POST, req_id: 2a696cb5-9287-44cc-8b98-e532df55a88e, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
12131 Replay: started 2, returned:Ok(())
12132 ----------------------------------------------------------------
12133 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12134 States: Active Active Offline
12135 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12136 4018 Acked Read 4 Done Done Sent false
12137 4019 Acked Write 9 Done Done Sent false
12138 4020 Acked Read 2 Done Done Sent false
12139 4021 Acked Read 8 Done Done Sent false
12140 4022 Acked Write 7 Done Done Sent false
12141 4023 Acked Write 10 Done Done Sent false
12142 4024 Acked Flush 0 Done Done Sent false
12143 4025 Acked Read 7 Done Done Sent false
12144 4026 Acked Read 3 Done Done Sent false
12145 4027 Acked Flush 0 Done Done Sent false
12146 4028 Acked Flush 0 Done Done Sent false
12147 4029 Acked Write 7 Done Done Sent false
12148 4030 Acked Write 10 Done Done Sent false
12149 4031 Acked Write 10 Done Done Sent false
12150 4032 Acked Write 4 Done Done Sent false
12151 4033 Acked Write 1 Done Done Sent false
12152 4034 Acked Read 8 Done Done Sent false
12153 4035 Acked Write 8 Done Done Sent false
12154 4036 Acked Write 10 Done Done Sent false
12155 4037 Acked Write 5 Done Done Sent false
12156 4038 Acked Write 1 Done Done Sent false
12157 4039 Acked Write 9 Done Done Sent false
12158 4040 Acked Write 6 Done Done Sent false
12159 4041 Acked Read 2 Done Done Sent false
12160 4042 Acked Read 6 Done Done Sent false
12161 4043 Acked Read 2 Done Done Sent false
12162 4044 Acked Read 6 Done Done Sent false
12163 4045 Acked Write 10 Done Done Sent false
12164 4046 Acked Read 5 Done Done Sent false
12165 4047 Acked Write 6 Done Done Sent false
12166 4048 Acked Read 3 Done Done Sent false
12167 4049 Acked Read 10 Done Done Sent false
12168 4050 Acked Flush 0 Done Done Sent false
12169 4051 Acked Write 1 Done Done Sent false
12170 4052 Acked Flush 0 Done Done Sent false
12171 4053 Acked Read 6 Done Done Sent false
12172 4054 Acked Read 5 Done Done Sent false
12173 4055 Acked Flush 0 Done Done Sent false
12174 4056 Acked Write 10 Done Done Sent false
12175 4057 Acked Read 2 Done Done Sent false
12176 4058 Acked Write 3 Done Done Sent false
12177 4059 Acked Read 3 Done Done Sent false
12178 4060 Acked Read 1 Done Done Sent false
12179 4061 Acked Write 6 Done Done Sent false
12180 4062 Acked Write 1 Done Done Sent false
12181 4063 Acked Read 8 Done Done Sent false
12182 4064 Acked Write 7 Done Done Sent false
12183 4065 Acked Read 9 Done Done Sent false
12184 4066 Acked Flush 0 Done Done Sent false
12185 4067 Acked Read 4 Done Done Sent false
12186 4068 Acked Read 9 Done Done Sent false
12187 4069 Acked Write 4 Done Done Sent false
12188 4070 Acked Write 1 Done Done Sent false
12189 4071 Acked Write 10 Done Done Sent false
12190 4072 Acked Write 3 Done Done Sent false
12191 4073 Acked Flush 0 Done Done Sent false
12192 4074 Acked Read 8 Done Done Sent false
12193 4075 Acked Write 5 Done Done Sent false
12194 4076 Acked Read 4 Done Done Sent false
12195 4077 Acked Flush 0 Done Done Sent false
12196 4078 Acked Read 10 Done Done Sent false
12197 4079 Acked Read 2 Done Done Sent false
12198 4080 Acked Read 6 Done Done Sent false
12199 4081 Acked Read 3 Done Done Sent false
12200 4082 Acked Flush 0 Done Done Sent false
12201 4083 Acked Flush 0 Done Done Sent false
12202 4084 Acked Read 8 Done Done Sent false
12203 4085 Acked Write 7 Done Done Sent false
12204 4086 Acked Write 3 Done Done Sent false
12205 4087 Acked Read 8 Done Done Sent false
12206 4088 Acked Write 8 Done Done Sent false
12207 4089 Acked Write 7 Done Done Sent false
12208 4090 Acked Write 9 Done Done Sent false
12209 4091 Acked Write 3 Done Done Sent false
12210 4092 Acked Read 10 Done Done Sent false
12211 4093 Acked Write 1 Done Done Sent false
12212 4094 Acked Read 5 Done Done Sent false
12213 4095 Acked Read 7 Done Done Sent false
12214 4096 Acked Write 7 Done Done Sent false
12215 4097 Acked Read 7 Done Done Sent false
12216 4098 Acked Read 7 Done Done Sent false
12217 4099 Acked Read 2 Done Done Sent false
12218 4100 Acked Read 5 Done Done Sent false
12219 4101 Acked Read 4 Done Done Sent false
12220 4102 Acked Write 9 Done Done Sent false
12221 4103 Acked Read 4 Done Done Sent false
12222 4104 Acked Read 6 Done Done Sent false
12223 4105 Acked Read 9 Done Done Sent false
12224 4106 Acked Read 9 Done Done Sent false
12225 4107 Acked Read 6 Done Done Sent false
12226 4108 Acked Write 8 Done Done Sent false
12227 4109 Acked Read 4 Done Done Sent false
12228 4110 Acked Read 2 Done Done Sent false
12229 4111 Acked Write 5 Done Done Sent false
12230 4112 Acked Read 10 Done Done Sent false
12231 4113 Acked Read 8 Done Done Sent false
12232 4114 Acked Write 9 Done Done Sent false
12233 4115 Acked Write 1 Done Done Sent false
12234 4116 Acked Read 2 Done Done Sent false
12235 4117 Acked Write 5 Done Done Sent false
12236 4118 Acked Write 7 Done Done Sent false
12237 4119 Acked Read 4 Done Done Sent false
12238 4120 Acked Flush 0 Done Done Sent false
12239 4121 Acked Read 4 Done Done Sent false
12240 4122 Acked Read 7 Done Done Sent false
12241 4123 Acked Write 1 Done Done Sent false
12242 4124 Acked Write 7 Done Done Sent false
12243 4125 Acked Read 8 Done Done Sent false
12244 4126 Acked Write 1 Done Done Sent false
12245 4127 Acked Read 6 Done Done Sent false
12246 4128 Acked Write 1 Done Done Sent false
12247 4129 Acked Read 2 Done Done Sent false
12248 4130 Acked Flush 0 Done Done Sent false
12249 4131 Acked Read 10 Done Done Sent false
12250 4132 Acked Flush 0 Done Done Sent false
12251 4133 Acked Read 8 Done Done Sent false
12252 4134 Acked Read 5 Done Done Sent false
12253 4135 Acked Flush 0 Done Done Sent false
12254 4136 Acked Write 8 Done Done Sent false
12255 4137 Acked Read 7 Done Done Sent false
12256 4138 Acked Write 9 Done Done Sent false
12257 4139 Acked Write 5 Done Done Sent false
12258 4140 Acked Write 6 Done Done Sent false
12259 4141 Acked Write 8 Done Done Sent false
12260 4142 Acked Write 7 Done Done Sent false
12261 4143 Acked Read 5 Done Done Sent false
12262 4144 Acked Write 6 Done Done Sent false
12263 4145 Acked Read 2 Done Done Sent false
12264 4146 Acked Read 8 Done Done Sent false
12265 4147 Acked Read 6 Done Done Sent false
12266 4148 Acked Read 5 Done Done Sent false
12267 4149 Acked Write 2 Done Done Sent false
12268 4150 Acked Read 8 Done Done Sent false
12269 4151 Acked Read 7 Done Done Sent false
12270 4152 Acked Write 9 Done Done Sent false
12271 4153 Acked Read 9 Done Done Sent false
12272 4154 Acked Read 1 Done Done Sent false
12273 4155 Acked Write 8 Done Done Sent false
12274 4156 Acked Write 7 Done Done Sent false
12275 4157 Acked Write 10 Done Done Sent false
12276 4158 Acked Read 5 Done Done Sent false
12277 4159 Acked Read 10 Done Done Sent false
12278 4160 Acked Read 5 Done Done Sent false
12279 4161 Acked Read 5 Done Done Sent false
12280 4162 Acked Write 4 Done Done Sent false
12281 4163 Acked Write 4 Done Done Sent false
12282 4164 Acked Read 6 Done Done Sent false
12283 4165 Acked Read 6 Done Done Sent false
12284 4166 Acked Read 6 Done Done Sent false
12285 4167 Acked Read 10 Done Done Sent false
12286 4168 Acked Read 1 Done Done Sent false
12287 4169 Acked Write 8 Done Done Sent false
12288 4170 Acked Write 5 Done Done Sent false
12289 4171 Acked Read 3 Done Done Sent false
12290 4172 Acked Read 9 Done Done Sent false
12291 4173 Acked Read 9 Done Done Sent false
12292 4174 Acked Read 9 Done Done Sent false
12293 4175 Acked Flush 0 Done Done Sent false
12294 4176 Acked Write 8 Done Done Sent false
12295 4177 Acked Read 2 Done Done Sent false
12296 4178 Acked Read 4 Done Done Sent false
12297 4179 Acked Read 1 Done Done Sent false
12298 4180 Acked Read 9 Done Done Sent false
12299 4181 Acked Read 7 Done Done Sent false
12300 4182 Acked Read 8 Done Done Sent false
12301 4183 Acked Read 4 Done Done Sent false
12302 4184 Acked Read 8 Done Done Sent false
12303 4185 Acked Write 7 Done Done Sent false
12304 4186 Acked Write 5 Done Done Sent false
12305 4187 Acked Write 9 Done Done Sent false
12306 4188 Acked Read 2 Done Done Sent false
12307 4189 Acked Write 3 Done Done Sent false
12308 4190 Acked Read 7 Done Done Sent false
12309 4191 Acked Flush 0 Done Done Sent false
12310 4192 Acked Read 7 Done Done Sent false
12311 4193 Acked Write 10 Done Done Sent false
12312 4194 Acked Read 7 Done Done Sent false
12313 4195 Acked Read 5 Done Done Sent false
12314 4196 Acked Read 3 Done Done Sent false
12315 4197 Acked Read 5 Done Done Sent false
12316 4198 Acked Read 7 Done Done Sent false
12317 4199 Acked Read 8 Done Done Sent false
12318 4200 Acked Read 1 Done Done Sent false
12319 4201 Acked Read 1 Done Done Sent false
12320 4202 Acked Flush 0 Done Done Sent false
12321 4203 Acked Flush 0 Done Done Sent false
12322 4204 Acked Read 1 Done Done Sent false
12323 4205 Acked Write 5 Done Done Sent false
12324 4206 Acked Read 5 Done Done Sent false
12325 4207 Acked Read 10 Done Done Sent false
12326 4208 Acked Write 10 Done Done Sent false
12327 4209 Acked Read 6 Done Done Sent false
12328 4210 Acked Write 7 Done Done Sent false
12329 4211 Acked Write 1 Done Done Sent false
12330 4212 Acked Read 2 Done Done Sent false
12331 4213 Acked Write 10 Done Done Sent false
12332 4214 Acked Read 4 Done Done Sent false
12333 4215 Acked Write 3 Done Done Sent false
12334 4216 Acked Read 9 Done Done Sent false
12335 4217 Acked Read 8 Done Done Sent false
12336 4218 Acked Write 3 Done Done Sent false
12337 4219 Acked Write 5 Done Done Sent false
12338 4220 Acked Read 1 Done Done Sent false
12339 4221 Acked Read 10 Done Done Sent false
12340 4222 Acked Read 2 Done Done Sent false
12341 4223 Acked Read 9 Done Done Sent false
12342 4224 Acked Read 2 Done Done Sent false
12343 4225 Acked Write 10 Done Done Sent false
12344 4226 Acked Write 10 Done Done Sent false
12345 4227 Acked Read 3 Done Done Sent false
12346 4228 Acked Write 1 Done Done Sent false
12347 4229 Acked Write 8 Done Done Sent false
12348 4230 Acked Read 2 Done Done Sent false
12349 4231 Acked Flush 0 Done Done Sent false
12350 4232 Acked Write 4 Done Done Sent false
12351 4233 Acked Read 1 Done Done Sent false
12352 4234 Acked Write 10 Done Done Sent false
12353 4235 Acked Read 10 Done Done Sent false
12354 4236 Acked Read 1 Done Done Sent false
12355 4237 Acked Read 3 Done Done Sent false
12356 4238 Acked Read 2 Done Done Sent false
12357 4239 Acked Write 6 Done Done Sent false
12358 4240 Acked Flush 0 Done Done Sent false
12359 4241 Acked Read 4 Done Done Sent false
12360 4242 Acked Write 7 Done Done Sent false
12361 4243 Acked Read 8 Done Done Sent false
12362 4244 Acked Write 10 Done Done Sent false
12363 4245 Acked Write 8 Done Done Sent false
12364 4246 Acked Write 5 Done Done Sent false
12365 4247 Acked Read 2 Done Done Sent false
12366 4248 Acked Write 5 Done Done Sent false
12367 4249 Acked Read 3 Done Done Sent false
12368 4250 Acked Write 9 Done Done Sent false
12369 4251 Acked Write 9 Done Done Sent false
12370 4252 Acked Write 9 Done Done Sent false
12371 4253 Acked Write 2 Done Done Sent false
12372 4254 Acked Write 9 Done Done Sent false
12373 4255 Acked Write 3 Done Done Sent false
12374 4256 Acked Write 8 Done Done Sent false
12375 4257 Acked Flush 0 Done Done Sent false
12376 4258 Acked Write 4 Done Done Sent false
12377 4259 Acked Read 8 Done Done Sent false
12378 4260 Acked Read 1 Done Done Sent false
12379 4261 Acked Read 1 Done Done Sent false
12380 4262 Acked Write 5 Done Done Sent false
12381 4263 Acked Read 4 Done Done Sent false
12382 4264 Acked Write 3 Done Done Sent false
12383 4265 Acked Read 7 Done Done Sent false
12384 4266 Acked Write 5 Done Done Sent false
12385 4267 Acked Flush 0 Done Done Sent false
12386 4268 Acked Read 7 Done Done Sent false
12387 4269 Acked Flush 0 Done Done Sent false
12388 4270 Acked Read 8 Done Done Sent false
12389 4271 Acked Read 10 Done Done Sent false
12390 4272 Acked Read 1 Done Done Sent false
12391 4273 Acked Write 8 Done Done Sent false
12392 4274 Acked Read 7 Done Done Sent false
12393 4275 Acked Write 6 Done Done Sent false
12394 4276 Acked Write 8 Done Done Sent false
12395 4277 Acked Write 1 Done Done Sent false
12396 4278 Acked Write 9 Done Done Sent false
12397 4279 Acked Read 8 Done Done Sent false
12398 4280 Acked Write 8 Done Done Sent false
12399 4281 Acked Write 8 Done Done Sent false
12400 4282 Acked Read 10 Done Done Sent false
12401 4283 Acked Write 1 Done Done Sent false
12402 4284 Acked Write 3 Done Done Sent false
12403 4285 Acked Write 3 Done Done Sent false
12404 4286 Acked Read 3 Done Done Sent false
12405 4287 Acked Read 1 Done Done Sent false
12406 4288 Acked Read 6 Done Done Sent false
12407 4289 Acked Write 10 Done Done Sent false
12408 4290 Acked Write 1 Done Done Sent false
12409 4291 Acked Write 7 Done Done Sent false
12410 4292 Acked Read 8 Done Done Sent false
12411 4293 Acked Read 1 Done Done Sent false
12412 4294 Acked Read 1 Done Done Sent false
12413 4295 Acked Write 8 Done Done Sent false
12414 4296 Acked Write 2 Done Done Sent false
12415 4297 Acked Read 3 Done Done Sent false
12416 4298 Acked Read 5 Done Done Sent false
12417 4299 Acked Write 9 Done Done Sent false
12418 4300 Acked Read 10 Done Done Sent false
12419 4301 Acked Write 1 Done Done Sent false
12420 4302 Acked Write 1 Done Done Sent false
12421 4303 Acked Read 1 Done Done Sent false
12422 4304 Acked Write 1 Done Done Sent false
12423 4305 Acked Write 3 Done Done Sent false
12424 4306 Acked Write 6 Done Done Sent false
12425 4307 Acked Read 7 Done Done Sent false
12426 4308 Acked Read 2 Done Done Sent false
12427 4309 Acked Write 2 Done Done Sent false
12428 4310 Acked Read 2 Done Done Sent false
12429 4311 Acked Flush 0 Done Done Sent false
12430 4312 Acked Write 3 Done Done Sent false
12431 4313 Acked Read 8 Done Done Sent false
12432 4314 Acked Read 6 Done Done Sent false
12433 4315 Acked Read 2 Done Done Sent false
12434 4316 Acked Write 3 Done Done Sent false
12435 4317 Acked Write 2 Done Done Sent false
12436 4318 Acked Read 6 Done Done Sent false
12437 STATES DS:0 DS:1 DS:2 TOTAL
12438 Sent 0 0 301 301
12439 Done 301 301 0 602
12440 Skipped 0 0 0 0
12441 Error 0 0 0 0
12442 Last Flush: Some(JobId(4311)) Some(JobId(4311)) Some(JobId(4017))
12443 Downstairs last five completed: 4017 4016 4015 4014 4013
12444 Upstairs last five completed: 4318 4317 4316 4315 4314
12445 CLIENT: Up:0 ds:301 act:2
12446 [8830] got start action
12447 [8830] I am going to restart
12448 Starting downstairs at port 8830
12449 Make output file at "/tmp/dsc/downstairs-8830.txt"
12450 [8830][2] reports Starting
12451 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1211)
12452 [8830][2] reports Running
12453 ----------------------------------------------------------------
12454 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
12455 States: Active Active Offline
12456 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12457 4018 Acked Read 4 Done Done Sent false
12458 4019 Acked Write 9 Done Done Sent false
12459 4020 Acked Read 2 Done Done Sent false
12460 4021 Acked Read 8 Done Done Sent false
12461 4022 Acked Write 7 Done Done Sent false
12462 4023 Acked Write 10 Done Done Sent false
12463 4024 Acked Flush 0 Done Done Sent false
12464 4025 Acked Read 7 Done Done Sent false
12465 4026 Acked Read 3 Done Done Sent false
12466 4027 Acked Flush 0 Done Done Sent false
12467 4028 Acked Flush 0 Done Done Sent false
12468 4029 Acked Write 7 Done Done Sent false
12469 4030 Acked Write 10 Done Done Sent false
12470 4031 Acked Write 10 Done Done Sent false
12471 4032 Acked Write 4 Done Done Sent false
12472 4033 Acked Write 1 Done Done Sent false
12473 4034 Acked Read 8 Done Done Sent false
12474 4035 Acked Write 8 Done Done Sent false
12475 4036 Acked Write 10 Done Done Sent false
12476 4037 Acked Write 5 Done Done Sent false
12477 4038 Acked Write 1 Done Done Sent false
12478 4039 Acked Write 9 Done Done Sent false
12479 4040 Acked Write 6 Done Done Sent false
12480 4041 Acked Read 2 Done Done Sent false
12481 4042 Acked Read 6 Done Done Sent false
12482 4043 Acked Read 2 Done Done Sent false
12483 4044 Acked Read 6 Done Done Sent false
12484 4045 Acked Write 10 Done Done Sent false
12485 4046 Acked Read 5 Done Done Sent false
12486 4047 Acked Write 6 Done Done Sent false
12487 4048 Acked Read 3 Done Done Sent false
12488 4049 Acked Read 10 Done Done Sent false
12489 4050 Acked Flush 0 Done Done Sent false
12490 4051 Acked Write 1 Done Done Sent false
12491 4052 Acked Flush 0 Done Done Sent false
12492 4053 Acked Read 6 Done Done Sent false
12493 4054 Acked Read 5 Done Done Sent false
12494 4055 Acked Flush 0 Done Done Sent false
12495 4056 Acked Write 10 Done Done Sent false
12496 4057 Acked Read 2 Done Done Sent false
12497 4058 Acked Write 3 Done Done Sent false
12498 4059 Acked Read 3 Done Done Sent false
12499 4060 Acked Read 1 Done Done Sent false
12500 4061 Acked Write 6 Done Done Sent false
12501 4062 Acked Write 1 Done Done Sent false
12502 4063 Acked Read 8 Done Done Sent false
12503 4064 Acked Write 7 Done Done Sent false
12504 4065 Acked Read 9 Done Done Sent false
12505 4066 Acked Flush 0 Done Done Sent false
12506 4067 Acked Read 4 Done Done Sent false
12507 4068 Acked Read 9 Done Done Sent false
12508 4069 Acked Write 4 Done Done Sent false
12509 4070 Acked Write 1 Done Done Sent false
12510 4071 Acked Write 10 Done Done Sent false
12511 4072 Acked Write 3 Done Done Sent false
12512 4073 Acked Flush 0 Done Done Sent false
12513 4074 Acked Read 8 Done Done Sent false
12514 4075 Acked Write 5 Done Done Sent false
12515 4076 Acked Read 4 Done Done Sent false
12516 4077 Acked Flush 0 Done Done Sent false
12517 4078 Acked Read 10 Done Done Sent false
12518 4079 Acked Read 2 Done Done Sent false
12519 4080 Acked Read 6 Done Done Sent false
12520 4081 Acked Read 3 Done Done Sent false
12521 4082 Acked Flush 0 Done Done Sent false
12522 4083 Acked Flush 0 Done Done Sent false
12523 4084 Acked Read 8 Done Done Sent false
12524 4085 Acked Write 7 Done Done Sent false
12525 4086 Acked Write 3 Done Done Sent false
12526 4087 Acked Read 8 Done Done Sent false
12527 4088 Acked Write 8 Done Done Sent false
12528 4089 Acked Write 7 Done Done Sent false
12529 4090 Acked Write 9 Done Done Sent false
12530 4091 Acked Write 3 Done Done Sent false
12531 4092 Acked Read 10 Done Done Sent false
12532 4093 Acked Write 1 Done Done Sent false
12533 4094 Acked Read 5 Done Done Sent false
12534 4095 Acked Read 7 Done Done Sent false
12535 4096 Acked Write 7 Done Done Sent false
12536 4097 Acked Read 7 Done Done Sent false
12537 4098 Acked Read 7 Done Done Sent false
12538 4099 Acked Read 2 Done Done Sent false
12539 4100 Acked Read 5 Done Done Sent false
12540 4101 Acked Read 4 Done Done Sent false
12541 4102 Acked Write 9 Done Done Sent false
12542 4103 Acked Read 4 Done Done Sent false
12543 4104 Acked Read 6 Done Done Sent false
12544 4105 Acked Read 9 Done Done Sent false
12545 4106 Acked Read 9 Done Done Sent false
12546 4107 Acked Read 6 Done Done Sent false
12547 4108 Acked Write 8 Done Done Sent false
12548 4109 Acked Read 4 Done Done Sent false
12549 4110 Acked Read 2 Done Done Sent false
12550 4111 Acked Write 5 Done Done Sent false
12551 4112 Acked Read 10 Done Done Sent false
12552 4113 Acked Read 8 Done Done Sent false
12553 4114 Acked Write 9 Done Done Sent false
12554 4115 Acked Write 1 Done Done Sent false
12555 4116 Acked Read 2 Done Done Sent false
12556 4117 Acked Write 5 Done Done Sent false
12557 4118 Acked Write 7 Done Done Sent false
12558 4119 Acked Read 4 Done Done Sent false
12559 4120 Acked Flush 0 Done Done Sent false
12560 4121 Acked Read 4 Done Done Sent false
12561 4122 Acked Read 7 Done Done Sent false
12562 4123 Acked Write 1 Done Done Sent false
12563 4124 Acked Write 7 Done Done Sent false
12564 4125 Acked Read 8 Done Done Sent false
12565 4126 Acked Write 1 Done Done Sent false
12566 4127 Acked Read 6 Done Done Sent false
12567 4128 Acked Write 1 Done Done Sent false
12568 4129 Acked Read 2 Done Done Sent false
12569 4130 Acked Flush 0 Done Done Sent false
12570 4131 Acked Read 10 Done Done Sent false
12571 4132 Acked Flush 0 Done Done Sent false
12572 4133 Acked Read 8 Done Done Sent false
12573 4134 Acked Read 5 Done Done Sent false
12574 4135 Acked Flush 0 Done Done Sent false
12575 4136 Acked Write 8 Done Done Sent false
12576 4137 Acked Read 7 Done Done Sent false
12577 4138 Acked Write 9 Done Done Sent false
12578 4139 Acked Write 5 Done Done Sent false
12579 4140 Acked Write 6 Done Done Sent false
12580 4141 Acked Write 8 Done Done Sent false
12581 4142 Acked Write 7 Done Done Sent false
12582 4143 Acked Read 5 Done Done Sent false
12583 4144 Acked Write 6 Done Done Sent false
12584 4145 Acked Read 2 Done Done Sent false
12585 4146 Acked Read 8 Done Done Sent false
12586 4147 Acked Read 6 Done Done Sent false
12587 4148 Acked Read 5 Done Done Sent false
12588 4149 Acked Write 2 Done Done Sent false
12589 4150 Acked Read 8 Done Done Sent false
12590 4151 Acked Read 7 Done Done Sent false
12591 4152 Acked Write 9 Done Done Sent false
12592 4153 Acked Read 9 Done Done Sent false
12593 4154 Acked Read 1 Done Done Sent false
12594 4155 Acked Write 8 Done Done Sent false
12595 4156 Acked Write 7 Done Done Sent false
12596 4157 Acked Write 10 Done Done Sent false
12597 4158 Acked Read 5 Done Done Sent false
12598 4159 Acked Read 10 Done Done Sent false
12599 4160 Acked Read 5 Done Done Sent false
12600 4161 Acked Read 5 Done Done Sent false
12601 4162 Acked Write 4 Done Done Sent false
12602 4163 Acked Write 4 Done Done Sent false
12603 4164 Acked Read 6 Done Done Sent false
12604 4165 Acked Read 6 Done Done Sent false
12605 4166 Acked Read 6 Done Done Sent false
12606 4167 Acked Read 10 Done Done Sent false
12607 4168 Acked Read 1 Done Done Sent false
12608 4169 Acked Write 8 Done Done Sent false
12609 4170 Acked Write 5 Done Done Sent false
12610 4171 Acked Read 3 Done Done Sent false
12611 4172 Acked Read 9 Done Done Sent false
12612 4173 Acked Read 9 Done Done Sent false
12613 4174 Acked Read 9 Done Done Sent false
12614 4175 Acked Flush 0 Done Done Sent false
12615 4176 Acked Write 8 Done Done Sent false
12616 4177 Acked Read 2 Done Done Sent false
12617 4178 Acked Read 4 Done Done Sent false
12618 4179 Acked Read 1 Done Done Sent false
12619 4180 Acked Read 9 Done Done Sent false
12620 4181 Acked Read 7 Done Done Sent false
12621 4182 Acked Read 8 Done Done Sent false
12622 4183 Acked Read 4 Done Done Sent false
12623 4184 Acked Read 8 Done Done Sent false
12624 4185 Acked Write 7 Done Done Sent false
12625 4186 Acked Write 5 Done Done Sent false
12626 4187 Acked Write 9 Done Done Sent false
12627 4188 Acked Read 2 Done Done Sent false
12628 4189 Acked Write 3 Done Done Sent false
12629 4190 Acked Read 7 Done Done Sent false
12630 4191 Acked Flush 0 Done Done Sent false
12631 4192 Acked Read 7 Done Done Sent false
12632 4193 Acked Write 10 Done Done Sent false
12633 4194 Acked Read 7 Done Done Sent false
12634 4195 Acked Read 5 Done Done Sent false
12635 4196 Acked Read 3 Done Done Sent false
12636 4197 Acked Read 5 Done Done Sent false
12637 4198 Acked Read 7 Done Done Sent false
12638 4199 Acked Read 8 Done Done Sent false
12639 4200 Acked Read 1 Done Done Sent false
12640 4201 Acked Read 1 Done Done Sent false
12641 4202 Acked Flush 0 Done Done Sent false
12642 4203 Acked Flush 0 Done Done Sent false
12643 4204 Acked Read 1 Done Done Sent false
12644 4205 Acked Write 5 Done Done Sent false
12645 4206 Acked Read 5 Done Done Sent false
12646 4207 Acked Read 10 Done Done Sent false
12647 4208 Acked Write 10 Done Done Sent false
12648 4209 Acked Read 6 Done Done Sent false
12649 4210 Acked Write 7 Done Done Sent false
12650 4211 Acked Write 1 Done Done Sent false
12651 4212 Acked Read 2 Done Done Sent false
12652 4213 Acked Write 10 Done Done Sent false
12653 4214 Acked Read 4 Done Done Sent false
12654 4215 Acked Write 3 Done Done Sent false
12655 4216 Acked Read 9 Done Done Sent false
12656 4217 Acked Read 8 Done Done Sent false
12657 4218 Acked Write 3 Done Done Sent false
12658 4219 Acked Write 5 Done Done Sent false
12659 4220 Acked Read 1 Done Done Sent false
12660 4221 Acked Read 10 Done Done Sent false
12661 4222 Acked Read 2 Done Done Sent false
12662 4223 Acked Read 9 Done Done Sent false
12663 4224 Acked Read 2 Done Done Sent false
12664 4225 Acked Write 10 Done Done Sent false
12665 4226 Acked Write 10 Done Done Sent false
12666 4227 Acked Read 3 Done Done Sent false
12667 4228 Acked Write 1 Done Done Sent false
12668 4229 Acked Write 8 Done Done Sent false
12669 4230 Acked Read 2 Done Done Sent false
12670 4231 Acked Flush 0 Done Done Sent false
12671 4232 Acked Write 4 Done Done Sent false
12672 4233 Acked Read 1 Done Done Sent false
12673 4234 Acked Write 10 Done Done Sent false
12674 4235 Acked Read 10 Done Done Sent false
12675 4236 Acked Read 1 Done Done Sent false
12676 4237 Acked Read 3 Done Done Sent false
12677 4238 Acked Read 2 Done Done Sent false
12678 4239 Acked Write 6 Done Done Sent false
12679 4240 Acked Flush 0 Done Done Sent false
12680 4241 Acked Read 4 Done Done Sent false
12681 4242 Acked Write 7 Done Done Sent false
12682 4243 Acked Read 8 Done Done Sent false
12683 4244 Acked Write 10 Done Done Sent false
12684 4245 Acked Write 8 Done Done Sent false
12685 4246 Acked Write 5 Done Done Sent false
12686 4247 Acked Read 2 Done Done Sent false
12687 4248 Acked Write 5 Done Done Sent false
12688 4249 Acked Read 3 Done Done Sent false
12689 4250 Acked Write 9 Done Done Sent false
12690 4251 Acked Write 9 Done Done Sent false
12691 4252 Acked Write 9 Done Done Sent false
12692 4253 Acked Write 2 Done Done Sent false
12693 4254 Acked Write 9 Done Done Sent false
12694 4255 Acked Write 3 Done Done Sent false
12695 4256 Acked Write 8 Done Done Sent false
12696 4257 Acked Flush 0 Done Done Sent false
12697 4258 Acked Write 4 Done Done Sent false
12698 4259 Acked Read 8 Done Done Sent false
12699 4260 Acked Read 1 Done Done Sent false
12700 4261 Acked Read 1 Done Done Sent false
12701 4262 Acked Write 5 Done Done Sent false
12702 4263 Acked Read 4 Done Done Sent false
12703 4264 Acked Write 3 Done Done Sent false
12704 4265 Acked Read 7 Done Done Sent false
12705 4266 Acked Write 5 Done Done Sent false
12706 4267 Acked Flush 0 Done Done Sent false
12707 4268 Acked Read 7 Done Done Sent false
12708 4269 Acked Flush 0 Done Done Sent false
12709 4270 Acked Read 8 Done Done Sent false
12710 4271 Acked Read 10 Done Done Sent false
12711 4272 Acked Read 1 Done Done Sent false
12712 4273 Acked Write 8 Done Done Sent false
12713 4274 Acked Read 7 Done Done Sent false
12714 4275 Acked Write 6 Done Done Sent false
12715 4276 Acked Write 8 Done Done Sent false
12716 4277 Acked Write 1 Done Done Sent false
12717 4278 Acked Write 9 Done Done Sent false
12718 4279 Acked Read 8 Done Done Sent false
12719 4280 Acked Write 8 Done Done Sent false
12720 4281 Acked Write 8 Done Done Sent false
12721 4282 Acked Read 10 Done Done Sent false
12722 4283 Acked Write 1 Done Done Sent false
12723 4284 Acked Write 3 Done Done Sent false
12724 4285 Acked Write 3 Done Done Sent false
12725 4286 Acked Read 3 Done Done Sent false
12726 4287 Acked Read 1 Done Done Sent false
12727 4288 Acked Read 6 Done Done Sent false
12728 4289 Acked Write 10 Done Done Sent false
12729 4290 Acked Write 1 Done Done Sent false
12730 4291 Acked Write 7 Done Done Sent false
12731 4292 Acked Read 8 Done Done Sent false
12732 4293 Acked Read 1 Done Done Sent false
12733 4294 Acked Read 1 Done Done Sent false
12734 4295 Acked Write 8 Done Done Sent false
12735 4296 Acked Write 2 Done Done Sent false
12736 4297 Acked Read 3 Done Done Sent false
12737 4298 Acked Read 5 Done Done Sent false
12738 4299 Acked Write 9 Done Done Sent false
12739 4300 Acked Read 10 Done Done Sent false
12740 4301 Acked Write 1 Done Done Sent false
12741 4302 Acked Write 1 Done Done Sent false
12742 4303 Acked Read 1 Done Done Sent false
12743 4304 Acked Write 1 Done Done Sent false
12744 4305 Acked Write 3 Done Done Sent false
12745 4306 Acked Write 6 Done Done Sent false
12746 4307 Acked Read 7 Done Done Sent false
12747 4308 Acked Read 2 Done Done Sent false
12748 4309 Acked Write 2 Done Done Sent false
12749 4310 Acked Read 2 Done Done Sent false
12750 4311 Acked Flush 0 Done Done Sent false
12751 4312 Acked Write 3 Done Done Sent false
12752 4313 Acked Read 8 Done Done Sent false
12753 4314 Acked Read 6 Done Done Sent false
12754 4315 Acked Read 2 Done Done Sent false
12755 4316 Acked Write 3 Done Done Sent false
12756 4317 Acked Write 2 Done Done Sent false
12757 4318 Acked Read 6 Done Done Sent false
12758 4319 Acked Flush 0 Done Done Sent false
12759 STATES DS:0 DS:1 DS:2 TOTAL
12760 Sent 0 0 302 302
12761 Done 302 302 0 604
12762 Skipped 0 0 0 0
12763 Error 0 0 0 0
12764 Last Flush: Some(JobId(4319)) Some(JobId(4319)) Some(JobId(4017))
12765 Downstairs last five completed: 4017 4016 4015 4014 4013
12766 Upstairs last five completed: 4319 4318 4317 4316 4315
12767 CLIENT: Up:0 ds:302 act:2
127682025-01-23T18:50:33.440ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
127692025-01-23T18:50:33.441ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33863) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
127702025-01-23T18:50:33.441ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
127712025-01-23T18:50:33.442ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
127722025-01-23T18:50:33.442ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
127732025-01-23T18:50:33.442ZINFOcrucible: send last flush ID to this DS: Some(JobId(4017)) = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
127742025-01-23T18:50:33.442ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
127752025-01-23T18:50:33.442ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(4017)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
127762025-01-23T18:50:33.442ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(4017)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
12777 ----------------------------------------------------------------
12778 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12779 Downstairs last five completed: 4319 4318 4317 4316 4315
12780 Upstairs last five completed: 4319 4318 4317 4316 4315
12781 CLIENT: Up:0 ds:0 act:3
12782 Replay: All jobs finished
12783 Main task has work to do, go find it
12784 got dsc Stop(2)
12785 stop 2
12786 [8830] Got stop action so:false kr:false
12787 Jan 23 18:50:35.686 INFO request completed, latency_us: 342, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 705817b6-8b12-4c00-b2bd-e2621569671b, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
127882025-01-23T18:50:35.686ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
127892025-01-23T18:50:35.686ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
12790 Jan 23 18:50:35.686 INFO request completed, latency_us{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:50:35.686804147Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
12791 : 126, response_code: 200, uri: /state/cid/2, method: GET, req_id: e950158d-fee0-4392-9456-2b8d182bf957, remote_addr: 127.0.0.1:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T18:50:35.686854431Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
12792 59295, local_addr: 127.0.0.1:9998
12793 [8830] Exited with: ExitStatus(unix_wait_status(9))
12794 [8830][2] reports Exit
12795 Jan 23 18:50:39.688 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0dd51240-911b-45ed-9ca3-08e04edfddd8, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
12796 001/300 Write block 592 len 20480 data: 2 2 2 2 2
12797 002/300 Write block 1858 len 28672 data: 2 3 3 4 4 4 5
12798 003/300 Read block 3234 len 16384
12799 004/300 Read block 1798 len 4096
12800 005/300 Read block 1803 len 32768
12801 006/300 Write block 1966 len 40960 data: 4 4 5 4 3 2 2 2 3 3
12802 007/300 Read block 4500 len 32768
12803 008/300 Read block 2780 len 36864
12804 009/300 Write block 4398 len 4096 data: 3
12805 010/300 Write block 1765 len 40960 data: 3 2 2 2 2 3 3 3 2 2
12806 011/300 Write block 734 len 4096 data: 3
12807 012/300 Write block 1382 len 40960 data: 2 2 3 2 2 2 2 2 3 3
12808 013/300 Read block 4657 len 28672
12809 014/300 Read block 654 len 28672
12810 015/300 Write block 4947 len 24576 data: 2 2 3 3 4 4
12811 016/300 Write block 3925 len 28672 data: 3 3 3 3 3 2 2
12812 017/300 Read block 1671 len 4096
12813 018/300 Read block 1072 len 16384
12814 019/300 Read block 675 len 32768
12815 020/300 Flush
12816 021/300 Write block 808 len 28672 data: 3 3 2 2 2 2 2
12817 022/300 Read block 4669 len 16384
12818 023/300 Read block 971 len 20480
12819 024/300 Write block 729 len 36864 data: 4 5 5 4 3 4 3 3 2
12820 025/300 Read block 3867 len 40960
12821 026/300 Read block 3381 len 4096
12822 027/300 Read block 67 len 20480
12823 028/300 Write block 4102 len 8192 data: 2 2
12824 029/300 Read block 1648 len 24576
12825 030/300 Read block 991 len 16384
12826 031/300 Read block 390 len 28672
12827 032/300 Read block 714 len 28672
12828 033/300 Read block 4503 len 4096
12829 034/300 Read block 227 len 24576
12830 035/300 Flush
12831 036/300 Read block 1070 len 32768
12832 037/300 Read block 3675 len 16384
12833 038/300 Read block 4807 len 40960
12834 039/300 Read block 1747 len 40960
12835 040/300 Flush
12836 041/300 Flush
12837 042/300 Write block 3969 len 36864 data: 2 3 3 3 3 3 2 2 2
12838 043/300 Read block 2926 len 16384
12839 044/300 Read block 1406 len 4096
12840 045/300 Flush
12841 046/300 Write block 2586 len 4096 data: 5
12842 047/300 Write block 3345 len 12288 data: 4 3 4
12843 048/300 Write block 3534 len 24576 data: 6 6 6 6 5 5
12844 049/300 Read block 2199 len 24576
12845 050/300 Read block 1049 len 36864
12846 051/300 Flush
12847 052/300 Write block 2140 len 20480 data: 4 2 2 3 3
12848 053/300 Write block 416 len 32768 data: 7 7 6 4 3 3 3 4
12849 054/300 Read block 2056 len 4096
12850 055/300 Read block 2489 len 12288
12851 056/300 Write block 1042 len 16384 data: 4 4 4 5
12852 057/300 Read block 4380 len 36864
12853 058/300 Flush
12854 059/300 Write block 1920 len 16384 data: 2 2 2 2
12855 060/300 Read block 3504 len 32768
12856 061/300 Write block 737 len 8192 data: 3 3
12857 062/300 Write block 1387 len 4096 data: 3
12858 063/300 Read block 432 len 36864
12859 064/300 Write block 2331 len 40960 data: 6 5 3 4 4 4 3 3 3 3
12860 065/300 Write block 2340 len 20480 data: 4 2 2 2 2
12861 066/300 Read block 4544 len 36864
12862 067/300 Read block 2175 len 40960
12863 068/300 Read block 4293 len 12288
12864 069/300 Write block 1826 len 12288 data: 3 3 4
12865 070/300 Write block 1683 len 32768 data: 3 3 3 2 2 4 4 4
12866 071/300 Read block 2814 len 36864
12867 072/300 Write block 3894 len 4096 data: 3
12868 073/300 Read block 484 len 32768
12869 074/300 Write block 4084 len 32768 data: 4 3 3 4 5 5 5 4
12870 075/300 Read block 274 len 4096
12871 076/300 Read block 3453 len 12288
12872 077/300 Write block 4555 len 4096 data: 2
12873 078/300 Read block 1042 len 40960
12874 079/300 Write block 1324 len 8192 data: 2 2
12875 080/300 Write block 4691 len 32768 data: 3 3 4 4 5 5 3 3
12876 081/300 Read block 2491 len 36864
12877 082/300 Write block 3936 len 16384 data: 2 2 2 2
12878 083/300 Write block 2918 len 8192 data: 2 3
12879 084/300 Read block 3478 len 40960
12880 085/300 Read block 1642 len 16384
12881 086/300 Read block 1438 len 8192
12882 087/300 Write block 4822 len 40960 data: 3 4 4 3 3 3 3 3 3 4
12883 088/300 Read block 2231 len 24576
12884 089/300 Write block 3899 len 32768 data: 4 4 4 4 3 3 3 3
12885 090/300 Read block 250 len 20480
12886 091/300 Write block 3926 len 8192 data: 4 4
12887 092/300 Read block 937 len 28672
12888 093/300 Write block 4520 len 24576 data: 2 2 2 3 3 3
12889 094/300 Write block 1262 len 36864 data: 4 4 4 4 2 2 2 2 2
12890 095/300 Write block 1240 len 32768 data: 3 3 3 2 3 4 5 4
12891 096/300 Read block 1843 len 12288
12892 097/300 Read block 2298 len 8192
12893 098/300 Write block 3064 len 40960 data: 3 3 3 3 3 3 2 2 3 3
12894 099/300 Read block 1510 len 28672
12895 100/300 Read block 2574 len 24576
12896 101/300 Write block 2157 len 16384 data: 5 5 4 4
12897 102/300 Read block 4353 len 36864
12898 103/300 Write block 1081 len 36864 data: 3 3 3 2 3 3 3 4 5
12899 104/300 Write block 1417 len 40960 data: 4 6 6 6 6 6 6 5 5 5
12900 105/300 Read block 3834 len 8192
12901 106/300 Write block 4258 len 24576 data: 5 4 3 3 3 3
12902 107/300 Read block 1520 len 28672
12903 108/300 Read block 3595 len 8192
12904 109/300 Read block 818 len 20480
12905 110/300 Write block 2080 len 16384 data: 2 2 2 2
12906 111/300 Flush
12907 112/300 Read block 3940 len 32768
12908 113/300 Flush
12909 114/300 Read block 3224 len 4096
12910 115/300 Read block 4594 len 20480
12911 116/300 Read block 4778 len 24576
12912 117/300 Read block 1726 len 8192
12913 118/300 Read block 3558 len 24576
12914 119/300 Write block 2707 len 20480 data: 3 3 3 3 4
12915 120/300 Write block 3005 len 24576 data: 5 4 3 2 2 2
12916 121/300 Write block 4749 len 28672 data: 3 4 3 4 5 4 5
12917 122/300 Read block 4114 len 16384
12918 123/300 Write block 1012 len 40960 data: 5 4 3 3 5 5 5 5 6 7
12919 124/300 Read block 3658 len 20480
12920 125/300 Write block 2020 len 32768 data: 3 3 3 3 3 3 4 3
12921 126/300 Write block 3048 len 32768 data: 7 7 7 4 2 3 3 3
12922 127/300 Write block 2151 len 28672 data: 3 2 2 4 4 6 6
12923 128/300 Write block 2781 len 16384 data: 2 2 2 3
12924 129/300 Write block 1626 len 40960 data: 3 3 3 3 2 2 2 2 2 2
12925 130/300 Read block 2270 len 36864
12926 131/300 Read block 476 len 24576
12927 132/300 Read block 2354 len 40960
12928 133/300 Write block 3141 len 20480 data: 3 3 2 2 2
12929 134/300 Read block 4570 len 4096
12930 135/300 Read block 3139 len 40960
12931 136/300 Read block 419 len 36864
12932 137/300 Write block 3741 len 20480 data: 5 4 3 3 4
12933 138/300 Flush
12934 139/300 Read block 4388 len 4096
12935 140/300 Read block 3959 len 36864
12936 141/300 Flush
12937 142/300 Read block 308 len 16384
12938 143/300 Read block 3743 len 28672
12939 144/300 Write block 148 len 36864 data: 3 4 4 4 3 3 3 3 2
12940 145/300 Write block 737 len 32768 data: 4 4 4 4 5 5 5 5
12941 146/300 Write block 4378 len 4096 data: 2
12942 147/300 Write block 3417 len 32768 data: 4 5 4 4 5 6 7 6
12943 148/300 Write block 4403 len 12288 data: 2 2 2
12944 149/300 Write block 2302 len 24576 data: 4 5 4 3 3 3
12945 150/300 Write block 1286 len 36864 data: 5 5 5 5 5 5 4 4 3
12946 151/300 Read block 2811 len 12288
12947 152/300 Read block 1614 len 40960
12948 153/300 Read block 1474 len 4096
12949 154/300 Write block 1709 len 36864 data: 3 2 2 3 3 3 2 2 3
12950 155/300 Write block 3885 len 40960 data: 5 4 3 3 2 2 2 2 3 4
12951 156/300 Read block 2549 len 12288
12952 157/300 Read block 4538 len 24576
12953 158/300 Read block 112 len 40960
12954 159/300 Write block 955 len 20480 data: 4 4 4 4 4
12955 160/300 Write block 3711 len 32768 data: 3 4 4 4 4 5 5 4
12956 161/300 Write block 2685 len 24576 data: 3 3 3 3 3 3
12957 162/300 Write block 3425 len 28672 data: 6 4 4 4 3 3 3
12958 163/300 Read block 3069 len 20480
12959 164/300 Flush
12960 165/300 Read block 4801 len 4096
12961 166/300 Write block 1240 len 28672 data: 4 4 4 3 4 5 6
12962 167/300 Write block 526 len 4096 data: 3
12963 168/300 Write block 577 len 28672 data: 4 4 3 3 3 3 3
12964 169/300 Read block 2841 len 16384
12965 170/300 Write block 599 len 24576 data: 4 4 4 7 4 4
12966 171/300 Read block 1768 len 28672
12967 172/300 Write block 2497 len 4096 data: 5
12968 173/300 Read block 4287 len 4096
12969 174/300 Write block 289 len 12288 data: 2 2 2
12970 175/300 Write block 2870 len 40960 data: 4 5 4 3 3 3 2 2 2 2
12971 176/300 Read block 3160 len 8192
12972 177/300 Write block 2288 len 20480 data: 3 3 4 4 4
12973 178/300 Flush
12974 179/300 Read block 4664 len 16384
12975 180/300 Read block 1049 len 36864
12976 181/300 Write block 1826 len 24576 data: 4 4 5 4 5 5
12977 182/300 Read block 1368 len 16384
12978 183/300 Read block 2170 len 24576
12979 184/300 Write block 1941 len 8192 data: 6 6
12980 185/300 Flush
12981 186/300 Read block 4657 len 24576
12982 187/300 Read block 780 len 20480
12983 188/300 Write block 2423 len 12288 data: 3 3 2
12984 189/300 Read block 3002 len 40960
12985 190/300 Write block 3192 len 16384 data: 2 3 3 3
12986 191/300 Write block 1543 len 40960 data: 2 2 2 2 2 2 3 3 3 3
12987 192/300 Read block 4579 len 28672
12988 193/300 Read block 1479 len 24576
12989 194/300 Read block 1343 len 8192
12990 195/300 Read block 2837 len 20480
12991 196/300 Read block 1445 len 40960
12992 197/300 Write block 2228 len 20480 data: 2 2 2 2 2
12993 198/300 Read block 4626 len 24576
12994 199/300 Write block 4690 len 32768 data: 3 4 4 5 5 6 6 4
12995 200/300 Flush
12996 201/300 Write block 2544 len 4096 data: 5
12997 202/300 Write block 4869 len 8192 data: 3 4
12998 203/300 Flush
12999 204/300 Read block 1624 len 12288
13000 205/300 Write block 1321 len 16384 data: 3 2 2 3
13001 206/300 Read block 1624 len 16384
13002 207/300 Read block 3949 len 8192
13003 208/300 Write block 2497 len 32768 data: 6 5 5 5 5 3 2 3
13004 209/300 Write block 2994 len 40960 data: 3 5 4 4 5 6 6 8 8 6
13005 210/300 Write block 2624 len 28672 data: 5 5 5 3 5 4 3
13006 211/300 Write block 4634 len 8192 data: 3 3
13007 212/300 Flush
13008 213/300 Flush
13009 214/300 Read block 1351 len 20480
13010 215/300 Write block 4379 len 4096 data: 2
13011 216/300 Write block 2574 len 8192 data: 3 3
13012 217/300 Read block 1530 len 4096
13013 218/300 Read block 971 len 20480
13014 219/300 Write block 2736 len 4096 data: 4
13015 220/300 Flush
13016 221/300 Read block 1889 len 40960
13017 222/300 Read block 2758 len 8192
13018 223/300 Write block 1970 len 24576 data: 4 3 3 3 4 4
13019 224/300 Flush
13020 225/300 Write block 3531 len 36864 data: 2 3 4 7 7 7 7 6 6
13021 226/300 Write block 3317 len 16384 data: 4 3 3 3
13022 227/300 Read block 1032 len 8192
13023 228/300 Read block 3982 len 12288
13024 229/300 Write block 3336 len 4096 data: 2
13025 230/300 Flush
13026 231/300 Flush
13027 232/300 Read block 2647 len 24576
13028 233/300 Flush
13029 234/300 Read block 1284 len 4096
13030 235/300 Read block 4571 len 24576
13031 236/300 Write block 3486 len 4096 data: 3
13032 237/300 Flush
13033 238/300 Write block 3884 len 36864 data: 5 6 5 4 4 3 3 3 3
13034 239/300 Write block 3673 len 36864 data: 2 2 2 3 3 3 3 4 4
13035 240/300 Write block 1231 len 40960 data: 2 2 2 3 3 3 3 4 4 5
13036 241/300 Read block 194 len 20480
13037 242/300 Read block 2845 len 8192
13038 243/300 Write block 779 len 8192 data: 7 8
13039 244/300 Flush
13040 245/300 Write block 85 len 32768 data: 3 3 2 2 3 5 5 5
13041 246/300 Write block 958 len 8192 data: 5 5
13042 247/300 Read block 4270 len 8192
13043 248/300 Read block 2462 len 36864
13044 249/300 Write block 471 len 20480 data: 4 4 4 3 3
13045 250/300 Read block 3462 len 32768
13046 251/300 Write block 109 len 32768 data: 5 3 4 5 6 6 6 4
13047 252/300 Flush
13048 253/300 Read block 3400 len 24576
13049 254/300 Read block 4531 len 4096
13050 255/300 Write block 4458 len 36864 data: 5 4 5 5 5 4 4 3 2
13051 256/300 Read block 718 len 20480
13052 257/300 Read block 4758 len 24576
13053 258/300 Write block 2079 len 12288 data: 4 3 3
13054 259/300 Flush
13055 260/300 Read block 3100 len 36864
13056 261/300 Write block 1639 len 32768 data: 2 2 3 3 3 5 5 5
13057 262/300 Read block 3394 len 32768
13058 263/300 Read block 2921 len 12288
13059 264/300 Read block 4933 len 32768
13060 265/300 Read block 4108 len 8192
13061 266/300 Flush
13062 267/300 Write block 3130 len 4096 data: 3
13063 268/300 Read block 715 len 4096
13064 269/300 Read block 4574 len 36864
13065 270/300 Write block 1774 len 32768 data: 3 2 2 2 3 3 3 3
13066 271/300 Write block 2830 len 40960 data: 2 3 3 3 3 4 4 4 4 4
13067 272/300 Read block 3143 len 40960
13068 273/300 Write block 4452 len 20480 data: 6 5 4 4 5
13069 274/300 Write block 4940 len 8192 data: 2 2
13070 275/300 Write block 830 len 8192 data: 3 2
13071 276/300 Read block 3474 len 36864
13072 277/300 Read block 1069 len 12288
13073 278/300 Read block 2290 len 8192
13074 279/300 Read block 3299 len 16384
13075 280/300 Read block 2535 len 20480
13076 281/300 Read block 1424 len 12288
13077 282/300 Read block 3930 len 32768
13078 283/300 Read block 2355 len 20480
13079 284/300 Read block 1784 len 28672
13080 285/300 Read block 2983 len 24576
13081 286/300 Write block 1773 len 20480 data: 3 4 3 3 3
13082 287/300 Read block 4225 len 4096
13083 288/300 Read block 1262 len 4096
13084 289/300 Read block 523 len 8192
13085 290/300 Flush
13086 291/300 Read block 587 len 28672
13087 292/300 Read block 1056 len 8192
13088 293/300 Read block 3717 len 36864
13089 294/300 Read block 3767 len 16384
13090 295/300 Read block 3950 len 4096
13091 296/300 Read block 418 len 4096
13092 297/300 Write block 2408 len 4096 data: 6
13093 298/300 Read block 2833 len 20480
13094 299/300 Read block 3737 len 4096
13095 300/300 Read block 3171 len 32768
13096 301/300 Write block 4175 len 20480 data: 5 3 3 3 2
13097 Main task has work to do, go find it
13098 got dsc Start(2)
13099 start 2
13100 Jan 23 18:50:39.926 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/2, method: POST, req_id: 6fea1223-7373-428b-a145-f50e55e42417, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
13101 Replay: started 2, returned:Ok(())
13102 ----------------------------------------------------------------
13103 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13104 States: Active Active Offline
13105 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13106 4320 Acked Write 5 Done Done Sent false
13107 4321 Acked Write 7 Done Done Sent false
13108 4322 Acked Read 4 Done Done Sent false
13109 4323 Acked Read 1 Done Done Sent false
13110 4324 Acked Read 8 Done Done Sent false
13111 4325 Acked Write 10 Done Done Sent false
13112 4326 Acked Read 8 Done Done Sent false
13113 4327 Acked Read 9 Done Done Sent false
13114 4328 Acked Write 1 Done Done Sent false
13115 4329 Acked Write 10 Done Done Sent false
13116 4330 Acked Write 1 Done Done Sent false
13117 4331 Acked Write 10 Done Done Sent false
13118 4332 Acked Read 7 Done Done Sent false
13119 4333 Acked Read 7 Done Done Sent false
13120 4334 Acked Write 6 Done Done Sent false
13121 4335 Acked Write 7 Done Done Sent false
13122 4336 Acked Read 1 Done Done Sent false
13123 4337 Acked Read 4 Done Done Sent false
13124 4338 Acked Read 8 Done Done Sent false
13125 4339 Acked Flush 0 Done Done Sent false
13126 4340 Acked Write 7 Done Done Sent false
13127 4341 Acked Read 4 Done Done Sent false
13128 4342 Acked Read 5 Done Done Sent false
13129 4343 Acked Write 9 Done Done Sent false
13130 4344 Acked Read 10 Done Done Sent false
13131 4345 Acked Read 1 Done Done Sent false
13132 4346 Acked Read 5 Done Done Sent false
13133 4347 Acked Write 2 Done Done Sent false
13134 4348 Acked Read 6 Done Done Sent false
13135 4349 Acked Read 4 Done Done Sent false
13136 4350 Acked Read 7 Done Done Sent false
13137 4351 Acked Read 7 Done Done Sent false
13138 4352 Acked Read 1 Done Done Sent false
13139 4353 Acked Read 6 Done Done Sent false
13140 4354 Acked Flush 0 Done Done Sent false
13141 4355 Acked Read 8 Done Done Sent false
13142 4356 Acked Read 4 Done Done Sent false
13143 4357 Acked Read 10 Done Done Sent false
13144 4358 Acked Read 10 Done Done Sent false
13145 4359 Acked Flush 0 Done Done Sent false
13146 4360 Acked Flush 0 Done Done Sent false
13147 4361 Acked Write 9 Done Done Sent false
13148 4362 Acked Read 4 Done Done Sent false
13149 4363 Acked Read 1 Done Done Sent false
13150 4364 Acked Flush 0 Done Done Sent false
13151 4365 Acked Write 1 Done Done Sent false
13152 4366 Acked Write 3 Done Done Sent false
13153 4367 Acked Write 6 Done Done Sent false
13154 4368 Acked Read 6 Done Done Sent false
13155 4369 Acked Read 9 Done Done Sent false
13156 4370 Acked Flush 0 Done Done Sent false
13157 4371 Acked Write 5 Done Done Sent false
13158 4372 Acked Write 8 Done Done Sent false
13159 4373 Acked Read 1 Done Done Sent false
13160 4374 Acked Read 3 Done Done Sent false
13161 4375 Acked Write 4 Done Done Sent false
13162 4376 Acked Read 9 Done Done Sent false
13163 4377 Acked Flush 0 Done Done Sent false
13164 4378 Acked Write 4 Done Done Sent false
13165 4379 Acked Read 8 Done Done Sent false
13166 4380 Acked Write 2 Done Done Sent false
13167 4381 Acked Write 1 Done Done Sent false
13168 4382 Acked Read 9 Done Done Sent false
13169 4383 Acked Write 10 Done Done Sent false
13170 4384 Acked Write 5 Done Done Sent false
13171 4385 Acked Read 9 Done Done Sent false
13172 4386 Acked Read 10 Done Done Sent false
13173 4387 Acked Read 3 Done Done Sent false
13174 4388 Acked Write 3 Done Done Sent false
13175 4389 Acked Write 8 Done Done Sent false
13176 4390 Acked Read 9 Done Done Sent false
13177 4391 Acked Write 1 Done Done Sent false
13178 4392 Acked Read 8 Done Done Sent false
13179 4393 Acked Write 8 Done Done Sent false
13180 4394 Acked Read 1 Done Done Sent false
13181 4395 Acked Read 3 Done Done Sent false
13182 4396 Acked Write 1 Done Done Sent false
13183 4397 Acked Read 10 Done Done Sent false
13184 4398 Acked Write 2 Done Done Sent false
13185 4399 Acked Write 8 Done Done Sent false
13186 4400 Acked Read 9 Done Done Sent false
13187 4401 Acked Write 4 Done Done Sent false
13188 4402 Acked Write 2 Done Done Sent false
13189 4403 Acked Read 10 Done Done Sent false
13190 4404 Acked Read 4 Done Done Sent false
13191 4405 Acked Read 2 Done Done Sent false
13192 4406 Acked Write 10 Done Done Sent false
13193 4407 Acked Read 6 Done Done Sent false
13194 4408 Acked Write 8 Done Done Sent false
13195 4409 Acked Read 5 Done Done Sent false
13196 4410 Acked Write 2 Done Done Sent false
13197 4411 Acked Read 7 Done Done Sent false
13198 4412 Acked Write 6 Done Done Sent false
13199 4413 Acked Write 9 Done Done Sent false
13200 4414 Acked Write 8 Done Done Sent false
13201 4415 Acked Read 3 Done Done Sent false
13202 4416 Acked Read 2 Done Done Sent false
13203 4417 Acked Write 10 Done Done Sent false
13204 4418 Acked Read 7 Done Done Sent false
13205 4419 Acked Read 6 Done Done Sent false
13206 4420 Acked Write 4 Done Done Sent false
13207 4421 Acked Read 9 Done Done Sent false
13208 4422 Acked Write 9 Done Done Sent false
13209 4423 Acked Write 10 Done Done Sent false
13210 4424 Acked Read 2 Done Done Sent false
13211 4425 Acked Write 6 Done Done Sent false
13212 4426 Acked Read 7 Done Done Sent false
13213 4427 Acked Read 2 Done Done Sent false
13214 4428 Acked Read 5 Done Done Sent false
13215 4429 Acked Write 4 Done Done Sent false
13216 4430 Acked Flush 0 Done Done Sent false
13217 4431 Acked Read 8 Done Done Sent false
13218 4432 Acked Flush 0 Done Done Sent false
13219 4433 Acked Read 1 Done Done Sent false
13220 4434 Acked Read 5 Done Done Sent false
13221 4435 Acked Read 6 Done Done Sent false
13222 4436 Acked Read 2 Done Done Sent false
13223 4437 Acked Read 6 Done Done Sent false
13224 4438 Acked Write 5 Done Done Sent false
13225 4439 Acked Write 6 Done Done Sent false
13226 4440 Acked Write 7 Done Done Sent false
13227 4441 Acked Read 4 Done Done Sent false
13228 4442 Acked Write 10 Done Done Sent false
13229 4443 Acked Read 5 Done Done Sent false
13230 4444 Acked Write 8 Done Done Sent false
13231 4445 Acked Write 8 Done Done Sent false
13232 4446 Acked Write 7 Done Done Sent false
13233 4447 Acked Write 4 Done Done Sent false
13234 4448 Acked Write 10 Done Done Sent false
13235 4449 Acked Read 9 Done Done Sent false
13236 4450 Acked Read 6 Done Done Sent false
13237 4451 Acked Read 10 Done Done Sent false
13238 4452 Acked Write 5 Done Done Sent false
13239 4453 Acked Read 1 Done Done Sent false
13240 4454 Acked Read 10 Done Done Sent false
13241 4455 Acked Read 9 Done Done Sent false
13242 4456 Acked Write 5 Done Done Sent false
13243 4457 Acked Flush 0 Done Done Sent false
13244 4458 Acked Read 1 Done Done Sent false
13245 4459 Acked Read 9 Done Done Sent false
13246 4460 Acked Flush 0 Done Done Sent false
13247 4461 Acked Read 4 Done Done Sent false
13248 4462 Acked Read 7 Done Done Sent false
13249 4463 Acked Write 9 Done Done Sent false
13250 4464 Acked Write 8 Done Done Sent false
13251 4465 Acked Write 1 Done Done Sent false
13252 4466 Acked Write 8 Done Done Sent false
13253 4467 Acked Write 3 Done Done Sent false
13254 4468 Acked Write 6 Done Done Sent false
13255 4469 Acked Write 9 Done Done Sent false
13256 4470 Acked Read 3 Done Done Sent false
13257 4471 Acked Read 10 Done Done Sent false
13258 4472 Acked Read 1 Done Done Sent false
13259 4473 Acked Write 9 Done Done Sent false
13260 4474 Acked Write 10 Done Done Sent false
13261 4475 Acked Read 3 Done Done Sent false
13262 4476 Acked Read 6 Done Done Sent false
13263 4477 Acked Read 10 Done Done Sent false
13264 4478 Acked Write 5 Done Done Sent false
13265 4479 Acked Write 8 Done Done Sent false
13266 4480 Acked Write 6 Done Done Sent false
13267 4481 Acked Write 7 Done Done Sent false
13268 4482 Acked Read 5 Done Done Sent false
13269 4483 Acked Flush 0 Done Done Sent false
13270 4484 Acked Read 1 Done Done Sent false
13271 4485 Acked Write 7 Done Done Sent false
13272 4486 Acked Write 1 Done Done Sent false
13273 4487 Acked Write 7 Done Done Sent false
13274 4488 Acked Read 4 Done Done Sent false
13275 4489 Acked Write 6 Done Done Sent false
13276 4490 Acked Read 7 Done Done Sent false
13277 4491 Acked Write 1 Done Done Sent false
13278 4492 Acked Read 1 Done Done Sent false
13279 4493 Acked Write 3 Done Done Sent false
13280 4494 Acked Write 10 Done Done Sent false
13281 4495 Acked Read 2 Done Done Sent false
13282 4496 Acked Write 5 Done Done Sent false
13283 4497 Acked Flush 0 Done Done Sent false
13284 4498 Acked Read 4 Done Done Sent false
13285 4499 Acked Read 9 Done Done Sent false
13286 4500 Acked Write 6 Done Done Sent false
13287 4501 Acked Read 4 Done Done Sent false
13288 4502 Acked Read 6 Done Done Sent false
13289 4503 Acked Write 2 Done Done Sent false
13290 4504 Acked Flush 0 Done Done Sent false
13291 4505 Acked Read 6 Done Done Sent false
13292 4506 Acked Read 5 Done Done Sent false
13293 4507 Acked Write 3 Done Done Sent false
13294 4508 Acked Read 10 Done Done Sent false
13295 4509 Acked Write 4 Done Done Sent false
13296 4510 Acked Write 10 Done Done Sent false
13297 4511 Acked Read 7 Done Done Sent false
13298 4512 Acked Read 6 Done Done Sent false
13299 4513 Acked Read 2 Done Done Sent false
13300 4514 Acked Read 5 Done Done Sent false
13301 4515 Acked Read 10 Done Done Sent false
13302 4516 Acked Write 5 Done Done Sent false
13303 4517 Acked Read 6 Done Done Sent false
13304 4518 Acked Write 8 Done Done Sent false
13305 4519 Acked Flush 0 Done Done Sent false
13306 4520 Acked Write 1 Done Done Sent false
13307 4521 Acked Write 2 Done Done Sent false
13308 4522 Acked Flush 0 Done Done Sent false
13309 4523 Acked Read 3 Done Done Sent false
13310 4524 Acked Write 4 Done Done Sent false
13311 4525 Acked Read 4 Done Done Sent false
13312 4526 Acked Read 2 Done Done Sent false
13313 4527 Acked Write 8 Done Done Sent false
13314 4528 Acked Write 10 Done Done Sent false
13315 4529 Acked Write 7 Done Done Sent false
13316 4530 Acked Write 2 Done Done Sent false
13317 4531 Acked Flush 0 Done Done Sent false
13318 4532 Acked Flush 0 Done Done Sent false
13319 4533 Acked Read 5 Done Done Sent false
13320 4534 Acked Write 1 Done Done Sent false
13321 4535 Acked Write 2 Done Done Sent false
13322 4536 Acked Read 1 Done Done Sent false
13323 4537 Acked Read 5 Done Done Sent false
13324 4538 Acked Write 1 Done Done Sent false
13325 4539 Acked Flush 0 Done Done Sent false
13326 4540 Acked Read 10 Done Done Sent false
13327 4541 Acked Read 2 Done Done Sent false
13328 4542 Acked Write 6 Done Done Sent false
13329 4543 Acked Flush 0 Done Done Sent false
13330 4544 Acked Write 9 Done Done Sent false
13331 4545 Acked Write 4 Done Done Sent false
13332 4546 Acked Read 2 Done Done Sent false
13333 4547 Acked Read 3 Done Done Sent false
13334 4548 Acked Write 1 Done Done Sent false
13335 4549 Acked Flush 0 Done Done Sent false
13336 4550 Acked Flush 0 Done Done Sent false
13337 4551 Acked Read 6 Done Done Sent false
13338 4552 Acked Flush 0 Done Done Sent false
13339 4553 Acked Read 1 Done Done Sent false
13340 4554 Acked Read 6 Done Done Sent false
13341 4555 Acked Write 1 Done Done Sent false
13342 4556 Acked Flush 0 Done Done Sent false
13343 4557 Acked Write 9 Done Done Sent false
13344 4558 Acked Write 9 Done Done Sent false
13345 4559 Acked Write 10 Done Done Sent false
13346 4560 Acked Read 5 Done Done Sent false
13347 4561 Acked Read 2 Done Done Sent false
13348 4562 Acked Write 2 Done Done Sent false
13349 4563 Acked Flush 0 Done Done Sent false
13350 4564 Acked Write 8 Done Done Sent false
13351 4565 Acked Write 2 Done Done Sent false
13352 4566 Acked Read 2 Done Done Sent false
13353 4567 Acked Read 9 Done Done Sent false
13354 4568 Acked Write 5 Done Done Sent false
13355 4569 Acked Read 8 Done Done Sent false
13356 4570 Acked Write 8 Done Done Sent false
13357 4571 Acked Flush 0 Done Done Sent false
13358 4572 Acked Read 6 Done Done Sent false
13359 4573 Acked Read 1 Done Done Sent false
13360 4574 Acked Write 9 Done Done Sent false
13361 4575 Acked Read 5 Done Done Sent false
13362 4576 Acked Read 6 Done Done Sent false
13363 4577 Acked Write 3 Done Done Sent false
13364 4578 Acked Flush 0 Done Done Sent false
13365 4579 Acked Read 9 Done Done Sent false
13366 4580 Acked Write 8 Done Done Sent false
13367 4581 Acked Read 8 Done Done Sent false
13368 4582 Acked Read 3 Done Done Sent false
13369 4583 Acked Read 8 Done Done Sent false
13370 4584 Acked Read 2 Done Done Sent false
13371 4585 Acked Flush 0 Done Done Sent false
13372 4586 Acked Write 1 Done Done Sent false
13373 4587 Acked Read 1 Done Done Sent false
13374 4588 Acked Read 9 Done Done Sent false
13375 4589 Acked Write 8 Done Done Sent false
13376 4590 Acked Write 10 Done Done Sent false
13377 4591 Acked Read 10 Done Done Sent false
13378 4592 Acked Write 5 Done Done Sent false
13379 4593 Acked Write 2 Done Done Sent false
13380 4594 Acked Write 2 Done Done Sent false
13381 4595 Acked Read 9 Done Done Sent false
13382 4596 Acked Read 3 Done Done Sent false
13383 4597 Acked Read 2 Done Done Sent false
13384 4598 Acked Read 4 Done Done Sent false
13385 4599 Acked Read 5 Done Done Sent false
13386 4600 Acked Read 3 Done Done Sent false
13387 4601 Acked Read 8 Done Done Sent false
13388 4602 Acked Read 5 Done Done Sent false
13389 4603 Acked Read 7 Done Done Sent false
13390 4604 Acked Read 6 Done Done Sent false
13391 4605 Acked Write 5 Done Done Sent false
13392 4606 Acked Read 1 Done Done Sent false
13393 4607 Acked Read 1 Done Done Sent false
13394 4608 Acked Read 2 Done Done Sent false
13395 4609 Acked Flush 0 Done Done Sent false
13396 4610 Acked Read 7 Done Done Sent false
13397 4611 Acked Read 2 Done Done Sent false
13398 4612 Acked Read 9 Done Done Sent false
13399 4613 Acked Read 4 Done Done Sent false
13400 4614 Acked Read 1 Done Done Sent false
13401 4615 Acked Read 1 Done Done Sent false
13402 4616 Acked Write 1 Done Done Sent false
13403 4617 Acked Read 5 Done Done Sent false
13404 4618 Acked Read 1 Done Done Sent false
13405 4619 Acked Read 8 Done Done Sent false
13406 4620 Acked Write 5 Done Done Sent false
13407 STATES DS:0 DS:1 DS:2 TOTAL
13408 Sent 0 0 301 301
13409 Done 301 301 0 602
13410 Skipped 0 0 0 0
13411 Error 0 0 0 0
13412 Last Flush: Some(JobId(4609)) Some(JobId(4609)) Some(JobId(4319))
13413 Downstairs last five completed: 4319 4318 4317 4316 4315
13414 Upstairs last five completed: 4620 4619 4618 4617 4616
13415 CLIENT: Up:0 ds:301 act:2
13416 [8830] got start action
13417 [8830] I am going to restart
13418 Starting downstairs at port 8830
13419 Make output file at "/tmp/dsc/downstairs-8830.txt"
13420 [8830][2] reports Starting
13421 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1212)
13422 [8830][2] reports Running
13423 ----------------------------------------------------------------
13424 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
13425 States: Active Active Offline
13426 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13427 4320 Acked Write 5 Done Done Sent false
13428 4321 Acked Write 7 Done Done Sent false
13429 4322 Acked Read 4 Done Done Sent false
13430 4323 Acked Read 1 Done Done Sent false
13431 4324 Acked Read 8 Done Done Sent false
13432 4325 Acked Write 10 Done Done Sent false
13433 4326 Acked Read 8 Done Done Sent false
13434 4327 Acked Read 9 Done Done Sent false
13435 4328 Acked Write 1 Done Done Sent false
13436 4329 Acked Write 10 Done Done Sent false
13437 4330 Acked Write 1 Done Done Sent false
13438 4331 Acked Write 10 Done Done Sent false
13439 4332 Acked Read 7 Done Done Sent false
13440 4333 Acked Read 7 Done Done Sent false
13441 4334 Acked Write 6 Done Done Sent false
13442 4335 Acked Write 7 Done Done Sent false
13443 4336 Acked Read 1 Done Done Sent false
13444 4337 Acked Read 4 Done Done Sent false
13445 4338 Acked Read 8 Done Done Sent false
13446 4339 Acked Flush 0 Done Done Sent false
13447 4340 Acked Write 7 Done Done Sent false
13448 4341 Acked Read 4 Done Done Sent false
13449 4342 Acked Read 5 Done Done Sent false
13450 4343 Acked Write 9 Done Done Sent false
13451 4344 Acked Read 10 Done Done Sent false
13452 4345 Acked Read 1 Done Done Sent false
13453 4346 Acked Read 5 Done Done Sent false
13454 4347 Acked Write 2 Done Done Sent false
13455 4348 Acked Read 6 Done Done Sent false
13456 4349 Acked Read 4 Done Done Sent false
13457 4350 Acked Read 7 Done Done Sent false
13458 4351 Acked Read 7 Done Done Sent false
13459 4352 Acked Read 1 Done Done Sent false
13460 4353 Acked Read 6 Done Done Sent false
13461 4354 Acked Flush 0 Done Done Sent false
13462 4355 Acked Read 8 Done Done Sent false
13463 4356 Acked Read 4 Done Done Sent false
13464 4357 Acked Read 10 Done Done Sent false
13465 4358 Acked Read 10 Done Done Sent false
13466 4359 Acked Flush 0 Done Done Sent false
13467 4360 Acked Flush 0 Done Done Sent false
13468 4361 Acked Write 9 Done Done Sent false
13469 4362 Acked Read 4 Done Done Sent false
13470 4363 Acked Read 1 Done Done Sent false
13471 4364 Acked Flush 0 Done Done Sent false
13472 4365 Acked Write 1 Done Done Sent false
13473 4366 Acked Write 3 Done Done Sent false
13474 4367 Acked Write 6 Done Done Sent false
13475 4368 Acked Read 6 Done Done Sent false
13476 4369 Acked Read 9 Done Done Sent false
13477 4370 Acked Flush 0 Done Done Sent false
13478 4371 Acked Write 5 Done Done Sent false
13479 4372 Acked Write 8 Done Done Sent false
13480 4373 Acked Read 1 Done Done Sent false
13481 4374 Acked Read 3 Done Done Sent false
13482 4375 Acked Write 4 Done Done Sent false
13483 4376 Acked Read 9 Done Done Sent false
13484 4377 Acked Flush 0 Done Done Sent false
13485 4378 Acked Write 4 Done Done Sent false
13486 4379 Acked Read 8 Done Done Sent false
13487 4380 Acked Write 2 Done Done Sent false
13488 4381 Acked Write 1 Done Done Sent false
13489 4382 Acked Read 9 Done Done Sent false
13490 4383 Acked Write 10 Done Done Sent false
13491 4384 Acked Write 5 Done Done Sent false
13492 4385 Acked Read 9 Done Done Sent false
13493 4386 Acked Read 10 Done Done Sent false
13494 4387 Acked Read 3 Done Done Sent false
13495 4388 Acked Write 3 Done Done Sent false
13496 4389 Acked Write 8 Done Done Sent false
13497 4390 Acked Read 9 Done Done Sent false
13498 4391 Acked Write 1 Done Done Sent false
13499 4392 Acked Read 8 Done Done Sent false
13500 4393 Acked Write 8 Done Done Sent false
13501 4394 Acked Read 1 Done Done Sent false
13502 4395 Acked Read 3 Done Done Sent false
13503 4396 Acked Write 1 Done Done Sent false
13504 4397 Acked Read 10 Done Done Sent false
13505 4398 Acked Write 2 Done Done Sent false
13506 4399 Acked Write 8 Done Done Sent false
13507 4400 Acked Read 9 Done Done Sent false
13508 4401 Acked Write 4 Done Done Sent false
13509 4402 Acked Write 2 Done Done Sent false
13510 4403 Acked Read 10 Done Done Sent false
13511 4404 Acked Read 4 Done Done Sent false
13512 4405 Acked Read 2 Done Done Sent false
13513 4406 Acked Write 10 Done Done Sent false
13514 4407 Acked Read 6 Done Done Sent false
13515 4408 Acked Write 8 Done Done Sent false
13516 4409 Acked Read 5 Done Done Sent false
13517 4410 Acked Write 2 Done Done Sent false
13518 4411 Acked Read 7 Done Done Sent false
13519 4412 Acked Write 6 Done Done Sent false
13520 4413 Acked Write 9 Done Done Sent false
13521 4414 Acked Write 8 Done Done Sent false
13522 4415 Acked Read 3 Done Done Sent false
13523 4416 Acked Read 2 Done Done Sent false
13524 4417 Acked Write 10 Done Done Sent false
13525 4418 Acked Read 7 Done Done Sent false
13526 4419 Acked Read 6 Done Done Sent false
13527 4420 Acked Write 4 Done Done Sent false
13528 4421 Acked Read 9 Done Done Sent false
13529 4422 Acked Write 9 Done Done Sent false
13530 4423 Acked Write 10 Done Done Sent false
13531 4424 Acked Read 2 Done Done Sent false
13532 4425 Acked Write 6 Done Done Sent false
13533 4426 Acked Read 7 Done Done Sent false
13534 4427 Acked Read 2 Done Done Sent false
13535 4428 Acked Read 5 Done Done Sent false
13536 4429 Acked Write 4 Done Done Sent false
13537 4430 Acked Flush 0 Done Done Sent false
13538 4431 Acked Read 8 Done Done Sent false
13539 4432 Acked Flush 0 Done Done Sent false
13540 4433 Acked Read 1 Done Done Sent false
13541 4434 Acked Read 5 Done Done Sent false
13542 4435 Acked Read 6 Done Done Sent false
13543 4436 Acked Read 2 Done Done Sent false
13544 4437 Acked Read 6 Done Done Sent false
13545 4438 Acked Write 5 Done Done Sent false
13546 4439 Acked Write 6 Done Done Sent false
13547 4440 Acked Write 7 Done Done Sent false
13548 4441 Acked Read 4 Done Done Sent false
13549 4442 Acked Write 10 Done Done Sent false
13550 4443 Acked Read 5 Done Done Sent false
13551 4444 Acked Write 8 Done Done Sent false
13552 4445 Acked Write 8 Done Done Sent false
13553 4446 Acked Write 7 Done Done Sent false
13554 4447 Acked Write 4 Done Done Sent false
13555 4448 Acked Write 10 Done Done Sent false
13556 4449 Acked Read 9 Done Done Sent false
13557 4450 Acked Read 6 Done Done Sent false
13558 4451 Acked Read 10 Done Done Sent false
13559 4452 Acked Write 5 Done Done Sent false
13560 4453 Acked Read 1 Done Done Sent false
13561 4454 Acked Read 10 Done Done Sent false
13562 4455 Acked Read 9 Done Done Sent false
13563 4456 Acked Write 5 Done Done Sent false
13564 4457 Acked Flush 0 Done Done Sent false
13565 4458 Acked Read 1 Done Done Sent false
13566 4459 Acked Read 9 Done Done Sent false
13567 4460 Acked Flush 0 Done Done Sent false
13568 4461 Acked Read 4 Done Done Sent false
13569 4462 Acked Read 7 Done Done Sent false
13570 4463 Acked Write 9 Done Done Sent false
13571 4464 Acked Write 8 Done Done Sent false
13572 4465 Acked Write 1 Done Done Sent false
13573 4466 Acked Write 8 Done Done Sent false
13574 4467 Acked Write 3 Done Done Sent false
13575 4468 Acked Write 6 Done Done Sent false
13576 4469 Acked Write 9 Done Done Sent false
13577 4470 Acked Read 3 Done Done Sent false
13578 4471 Acked Read 10 Done Done Sent false
13579 4472 Acked Read 1 Done Done Sent false
13580 4473 Acked Write 9 Done Done Sent false
13581 4474 Acked Write 10 Done Done Sent false
13582 4475 Acked Read 3 Done Done Sent false
13583 4476 Acked Read 6 Done Done Sent false
13584 4477 Acked Read 10 Done Done Sent false
13585 4478 Acked Write 5 Done Done Sent false
13586 4479 Acked Write 8 Done Done Sent false
13587 4480 Acked Write 6 Done Done Sent false
13588 4481 Acked Write 7 Done Done Sent false
13589 4482 Acked Read 5 Done Done Sent false
13590 4483 Acked Flush 0 Done Done Sent false
13591 4484 Acked Read 1 Done Done Sent false
13592 4485 Acked Write 7 Done Done Sent false
13593 4486 Acked Write 1 Done Done Sent false
13594 4487 Acked Write 7 Done Done Sent false
13595 4488 Acked Read 4 Done Done Sent false
13596 4489 Acked Write 6 Done Done Sent false
13597 4490 Acked Read 7 Done Done Sent false
13598 4491 Acked Write 1 Done Done Sent false
13599 4492 Acked Read 1 Done Done Sent false
13600 4493 Acked Write 3 Done Done Sent false
13601 4494 Acked Write 10 Done Done Sent false
13602 4495 Acked Read 2 Done Done Sent false
13603 4496 Acked Write 5 Done Done Sent false
13604 4497 Acked Flush 0 Done Done Sent false
13605 4498 Acked Read 4 Done Done Sent false
13606 4499 Acked Read 9 Done Done Sent false
13607 4500 Acked Write 6 Done Done Sent false
13608 4501 Acked Read 4 Done Done Sent false
13609 4502 Acked Read 6 Done Done Sent false
13610 4503 Acked Write 2 Done Done Sent false
13611 4504 Acked Flush 0 Done Done Sent false
13612 4505 Acked Read 6 Done Done Sent false
13613 4506 Acked Read 5 Done Done Sent false
13614 4507 Acked Write 3 Done Done Sent false
13615 4508 Acked Read 10 Done Done Sent false
13616 4509 Acked Write 4 Done Done Sent false
13617 4510 Acked Write 10 Done Done Sent false
13618 4511 Acked Read 7 Done Done Sent false
13619 4512 Acked Read 6 Done Done Sent false
13620 4513 Acked Read 2 Done Done Sent false
13621 4514 Acked Read 5 Done Done Sent false
13622 4515 Acked Read 10 Done Done Sent false
13623 4516 Acked Write 5 Done Done Sent false
13624 4517 Acked Read 6 Done Done Sent false
13625 4518 Acked Write 8 Done Done Sent false
13626 4519 Acked Flush 0 Done Done Sent false
13627 4520 Acked Write 1 Done Done Sent false
13628 4521 Acked Write 2 Done Done Sent false
13629 4522 Acked Flush 0 Done Done Sent false
13630 4523 Acked Read 3 Done Done Sent false
13631 4524 Acked Write 4 Done Done Sent false
13632 4525 Acked Read 4 Done Done Sent false
13633 4526 Acked Read 2 Done Done Sent false
13634 4527 Acked Write 8 Done Done Sent false
13635 4528 Acked Write 10 Done Done Sent false
13636 4529 Acked Write 7 Done Done Sent false
13637 4530 Acked Write 2 Done Done Sent false
13638 4531 Acked Flush 0 Done Done Sent false
13639 4532 Acked Flush 0 Done Done Sent false
13640 4533 Acked Read 5 Done Done Sent false
13641 4534 Acked Write 1 Done Done Sent false
13642 4535 Acked Write 2 Done Done Sent false
13643 4536 Acked Read 1 Done Done Sent false
13644 4537 Acked Read 5 Done Done Sent false
13645 4538 Acked Write 1 Done Done Sent false
13646 4539 Acked Flush 0 Done Done Sent false
13647 4540 Acked Read 10 Done Done Sent false
13648 4541 Acked Read 2 Done Done Sent false
13649 4542 Acked Write 6 Done Done Sent false
13650 4543 Acked Flush 0 Done Done Sent false
13651 4544 Acked Write 9 Done Done Sent false
13652 4545 Acked Write 4 Done Done Sent false
13653 4546 Acked Read 2 Done Done Sent false
13654 4547 Acked Read 3 Done Done Sent false
13655 4548 Acked Write 1 Done Done Sent false
13656 4549 Acked Flush 0 Done Done Sent false
13657 4550 Acked Flush 0 Done Done Sent false
13658 4551 Acked Read 6 Done Done Sent false
13659 4552 Acked Flush 0 Done Done Sent false
13660 4553 Acked Read 1 Done Done Sent false
13661 4554 Acked Read 6 Done Done Sent false
13662 4555 Acked Write 1 Done Done Sent false
13663 4556 Acked Flush 0 Done Done Sent false
13664 4557 Acked Write 9 Done Done Sent false
13665 4558 Acked Write 9 Done Done Sent false
13666 4559 Acked Write 10 Done Done Sent false
13667 4560 Acked Read 5 Done Done Sent false
13668 4561 Acked Read 2 Done Done Sent false
13669 4562 Acked Write 2 Done Done Sent false
13670 4563 Acked Flush 0 Done Done Sent false
13671 4564 Acked Write 8 Done Done Sent false
13672 4565 Acked Write 2 Done Done Sent false
13673 4566 Acked Read 2 Done Done Sent false
13674 4567 Acked Read 9 Done Done Sent false
13675 4568 Acked Write 5 Done Done Sent false
13676 4569 Acked Read 8 Done Done Sent false
13677 4570 Acked Write 8 Done Done Sent false
13678 4571 Acked Flush 0 Done Done Sent false
13679 4572 Acked Read 6 Done Done Sent false
13680 4573 Acked Read 1 Done Done Sent false
13681 4574 Acked Write 9 Done Done Sent false
13682 4575 Acked Read 5 Done Done Sent false
13683 4576 Acked Read 6 Done Done Sent false
13684 4577 Acked Write 3 Done Done Sent false
13685 4578 Acked Flush 0 Done Done Sent false
13686 4579 Acked Read 9 Done Done Sent false
13687 4580 Acked Write 8 Done Done Sent false
13688 4581 Acked Read 8 Done Done Sent false
13689 4582 Acked Read 3 Done Done Sent false
13690 4583 Acked Read 8 Done Done Sent false
13691 4584 Acked Read 2 Done Done Sent false
13692 4585 Acked Flush 0 Done Done Sent false
13693 4586 Acked Write 1 Done Done Sent false
13694 4587 Acked Read 1 Done Done Sent false
13695 4588 Acked Read 9 Done Done Sent false
13696 4589 Acked Write 8 Done Done Sent false
13697 4590 Acked Write 10 Done Done Sent false
13698 4591 Acked Read 10 Done Done Sent false
13699 4592 Acked Write 5 Done Done Sent false
13700 4593 Acked Write 2 Done Done Sent false
13701 4594 Acked Write 2 Done Done Sent false
13702 4595 Acked Read 9 Done Done Sent false
13703 4596 Acked Read 3 Done Done Sent false
13704 4597 Acked Read 2 Done Done Sent false
13705 4598 Acked Read 4 Done Done Sent false
13706 4599 Acked Read 5 Done Done Sent false
13707 4600 Acked Read 3 Done Done Sent false
13708 4601 Acked Read 8 Done Done Sent false
13709 4602 Acked Read 5 Done Done Sent false
13710 4603 Acked Read 7 Done Done Sent false
13711 4604 Acked Read 6 Done Done Sent false
13712 4605 Acked Write 5 Done Done Sent false
13713 4606 Acked Read 1 Done Done Sent false
13714 4607 Acked Read 1 Done Done Sent false
13715 4608 Acked Read 2 Done Done Sent false
13716 4609 Acked Flush 0 Done Done Sent false
13717 4610 Acked Read 7 Done Done Sent false
13718 4611 Acked Read 2 Done Done Sent false
13719 4612 Acked Read 9 Done Done Sent false
13720 4613 Acked Read 4 Done Done Sent false
13721 4614 Acked Read 1 Done Done Sent false
13722 4615 Acked Read 1 Done Done Sent false
13723 4616 Acked Write 1 Done Done Sent false
13724 4617 Acked Read 5 Done Done Sent false
13725 4618 Acked Read 1 Done Done Sent false
13726 4619 Acked Read 8 Done Done Sent false
13727 4620 Acked Write 5 Done Done Sent false
13728 4621 Acked Flush 0 Done Done Sent false
13729 STATES DS:0 DS:1 DS:2 TOTAL
13730 Sent 0 0 302 302
13731 Done 302 302 0 604
13732 Skipped 0 0 0 0
13733 Error 0 0 0 0
13734 Last Flush: Some(JobId(4621)) Some(JobId(4621)) Some(JobId(4319))
13735 Downstairs last five completed: 4319 4318 4317 4316 4315
13736 Upstairs last five completed: 4621 4620 4619 4618 4617
13737 CLIENT: Up:0 ds:302 act:2
137382025-01-23T18:50:45.687ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
137392025-01-23T18:50:45.687ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37899) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
137402025-01-23T18:50:45.688ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
137412025-01-23T18:50:45.689ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
137422025-01-23T18:50:45.689ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
137432025-01-23T18:50:45.689ZINFOcrucible: send last flush ID to this DS: Some(JobId(4319)) = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
137442025-01-23T18:50:45.689ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
137452025-01-23T18:50:45.689ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(4319)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
137462025-01-23T18:50:45.689ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(4319)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
13747 ----------------------------------------------------------------
13748 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13749 Downstairs last five completed: 4621 4620 4619 4618 4617
13750 Upstairs last five completed: 4621 4620 4619 4618 4617
13751 CLIENT: Up:0 ds:0 act:3
13752 Replay: All jobs finished
13753 Main task has work to do, go find it
13754 got dsc Stop(1)
13755 stop 1
13756 Jan 23 18:50:47.931 INFO request completed, latency_us: 340, response_code: 204, [8820] Got stop action so:false kr:false
13757 uri: /stop/cid/1, method: POST, req_id: f1dfffaf-b804-4c61-9280-ef662952dc95, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
137582025-01-23T18:50:47.932ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
13759 Jan 23 18:50:47.932 INFO request completed, latency_us: 128, response_code: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:50:47.932200374Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
13760 200, uri: /state/cid/1, method: GET, req_id: 249eda1c-5a3b-4a25-9a55-2e22c2a12ac0{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:50:47.932256188Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
13761 , remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
137622025-01-23T18:50:47.932ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
13763 [8820] Exited with: ExitStatus(unix_wait_status(9))
13764 [8820][1] reports Exit
13765 Jan 23 18:50:51.934 INFO request completed, latency_us: 184, response_code: 200, uri: /state/cid/1, method: GET, req_id: bae57e88-d746-4451-9178-8cac0c44721e, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
13766 001/300 Read block 2129 len 16384
13767 002/300 Read block 3736 len 36864
13768 003/300 Write block 2112 len 4096 data: 2
13769 004/300 Read block 2948 len 24576
13770 005/300 Read block 417 len 24576
13771 006/300 Write block 726 len 28672 data: 4 4 3 5 6 6 5
13772 007/300 Read block 4120 len 36864
13773 008/300 Read block 1237 len 28672
13774 009/300 Read block 4360 len 24576
13775 010/300 Read block 2679 len 24576
13776 011/300 Read block 819 len 16384
13777 012/300 Write block 1786 len 40960 data: 4 4 4 4 5 4 4 4 6 6
13778 013/300 Read block 1681 len 4096
13779 014/300 Read block 1697 len 32768
13780 015/300 Read block 1365 len 40960
13781 016/300 Read block 2402 len 20480
13782 017/300 Write block 526 len 20480 data: 4 5 5 5 4
13783 018/300 Write block 3346 len 8192 data: 4 5
13784 019/300 Flush
13785 020/300 Read block 3695 len 16384
13786 021/300 Read block 2696 len 24576
13787 022/300 Write block 2616 len 20480 data: 4 4 6 6 6
13788 023/300 Read block 2642 len 36864
13789 024/300 Read block 434 len 4096
13790 025/300 Read block 2723 len 16384
13791 026/300 Write block 2276 len 8192 data: 4 5
13792 027/300 Write block 3356 len 16384 data: 3 4 3 4
13793 028/300 Write block 3498 len 16384 data: 4 5 4 4
13794 029/300 Write block 3322 len 8192 data: 3 3
13795 030/300 Flush
13796 031/300 Read block 4564 len 12288
13797 032/300 Read block 1815 len 32768
13798 033/300 Read block 1834 len 20480
13799 034/300 Read block 1453 len 32768
13800 035/300 Read block 819 len 36864
13801 036/300 Write block 2963 len 24576 data: 4 5 5 4 4 4
13802 037/300 Read block 4402 len 36864
13803 038/300 Read block 3962 len 36864
13804 039/300 Read block 3393 len 28672
13805 040/300 Write block 2669 len 36864 data: 3 4 5 5 5 5 6 6 5
13806 041/300 Read block 4454 len 40960
13807 042/300 Read block 2193 len 16384
13808 043/300 Read block 604 len 32768
13809 044/300 Read block 397 len 20480
13810 045/300 Read block 3872 len 32768
13811 046/300 Read block 2881 len 24576
13812 047/300 Write block 3973 len 32768 data: 4 4 3 3 3 2 2 2
13813 048/300 Read block 3518 len 12288
13814 049/300 Read block 1212 len 36864
13815 050/300 Write block 3756 len 4096 data: 4
13816 051/300 Read block 4131 len 24576
13817 052/300 Write block 4778 len 12288 data: 4 4 4
13818 053/300 Write block 2912 len 32768 data: 3 2 2 2 2 2 3 4
13819 054/300 Read block 3535 len 4096
13820 055/300 Read block 1633 len 4096
13821 056/300 Read block 4781 len 28672
13822 057/300 Read block 2765 len 32768
13823 058/300 Read block 1656 len 36864
13824 059/300 Read block 4670 len 4096
13825 060/300 Read block 1359 len 20480
13826 061/300 Read block 3393 len 16384
13827 062/300 Write block 1635 len 32768 data: 3 2 2 2 3 3 4 4
13828 063/300 Read block 473 len 8192
13829 064/300 Read block 161 len 24576
13830 065/300 Write block 1994 len 4096 data: 3
13831 066/300 Write block 3386 len 16384 data: 6 6 5 4
13832 067/300 Flush
13833 068/300 Flush
13834 069/300 Read block 2082 len 24576
13835 070/300 Read block 3901 len 8192
13836 071/300 Read block 3328 len 24576
13837 072/300 Read block 4432 len 16384
13838 073/300 Read block 269 len 36864
13839 074/300 Write block 4363 len 32768 data: 3 3 2 2 2 3 4 4
13840 075/300 Write block 4243 len 36864 data: 2 2 2 2 2 2 2 3 3
13841 076/300 Read block 4167 len 24576
13842 077/300 Write block 4021 len 20480 data: 3 3 3 3 3
13843 078/300 Write block 3791 len 28672 data: 3 3 4 4 4 4 3
13844 079/300 Write block 4171 len 28672 data: 5 5 4 4 6 4 4
13845 080/300 Write block 3153 len 20480 data: 3 3 3 2 3
13846 081/300 Read block 222 len 32768
13847 082/300 Read block 4775 len 12288
13848 083/300 Read block 4463 len 36864
13849 084/300 Write block 1596 len 8192 data: 2 2
13850 085/300 Write block 4647 len 8192 data: 6 4
13851 086/300 Write block 935 len 28672 data: 4 4 4 4 5 4 4
13852 087/300 Write block 1862 len 24576 data: 5 5 6 5 5 5
13853 088/300 Flush
13854 089/300 Read block 313 len 32768
13855 090/300 Flush
13856 091/300 Write block 2492 len 32768 data: 4 3 4 4 4 7 6 6
13857 092/300 Read block 1205 len 12288
13858 093/300 Write block 3098 len 16384 data: 2 3 4 4
13859 094/300 Read block 3667 len 32768
13860 095/300 Write block 4609 len 20480 data: 3 2 2 2 2
13861 096/300 Read block 4381 len 32768
13862 097/300 Read block 3648 len 4096
13863 098/300 Read block 51 len 24576
13864 099/300 Read block 1632 len 40960
13865 100/300 Read block 3591 len 8192
13866 101/300 Write block 1912 len 40960 data: 3 3 3 3 2 2 2 2 3 3
13867 102/300 Flush
13868 103/300 Flush
13869 104/300 Read block 3102 len 8192
13870 105/300 Flush
13871 106/300 Read block 3965 len 28672
13872 107/300 Read block 4567 len 32768
13873 108/300 Read block 2011 len 12288
13874 109/300 Read block 4451 len 12288
13875 110/300 Read block 976 len 8192
13876 111/300 Write block 552 len 32768 data: 5 4 3 2 2 3 4 4
13877 112/300 Read block 424 len 12288
13878 113/300 Write block 1622 len 32768 data: 5 5 4 4 4 4 4 4
13879 114/300 Read block 3494 len 36864
13880 115/300 Read block 4081 len 20480
13881 116/300 Read block 4807 len 4096
13882 117/300 Write block 4444 len 36864 data: 2 2 4 5 6 7 7 7 7
13883 118/300 Read block 860 len 16384
13884 119/300 Write block 2642 len 4096 data: 4
13885 120/300 Flush
13886 121/300 Read block 2584 len 32768
13887 122/300 Read block 1982 len 40960
13888 123/300 Read block 2370 len 4096
13889 124/300 Write block 2199 len 24576 data: 5 5 5 4 5 4
13890 125/300 Write block 4669 len 4096 data: 3
13891 126/300 Read block 1589 len 28672
13892 127/300 Read block 1746 len 28672
13893 128/300 Write block 2306 len 40960 data: 4 4 3 3 3 3 4 6 7 8
13894 129/300 Read block 4359 len 28672
13895 130/300 Write block 495 len 32768 data: 3 4 3 3 2 2 2 2
13896 131/300 Write block 727 len 28672 data: 5 4 6 7 7 6 4
13897 132/300 Read block 619 len 28672
13898 133/300 Read block 4474 len 24576
13899 134/300 Write block 4216 len 24576 data: 3 4 4 5 5 5
13900 135/300 Read block 3823 len 28672
13901 136/300 Read block 2048 len 36864
13902 137/300 Read block 1896 len 40960
13903 138/300 Read block 2123 len 28672
13904 139/300 Read block 3616 len 4096
13905 140/300 Read block 732 len 32768
13906 141/300 Read block 2884 len 28672
13907 142/300 Write block 38 len 12288 data: 3 3 3
13908 143/300 Write block 2079 len 32768 data: 5 4 4 3 3 2 2 2
13909 144/300 Write block 4516 len 4096 data: 2
13910 145/300 Flush
13911 146/300 Read block 3987 len 12288
13912 147/300 Read block 3810 len 4096
13913 148/300 Read block 1226 len 16384
13914 149/300 Read block 1597 len 28672
13915 150/300 Read block 772 len 4096
13916 151/300 Write block 3464 len 20480 data: 5 5 4 3 3
13917 152/300 Read block 1789 len 4096
13918 153/300 Write block 1922 len 16384 data: 3 3 2 2
13919 154/300 Read block 535 len 40960
13920 155/300 Read block 2191 len 32768
13921 156/300 Write block 4826 len 12288 data: 4 4 4
13922 157/300 Read block 4411 len 12288
13923 158/300 Read block 3748 len 36864
13924 159/300 Write block 3042 len 24576 data: 5 5 4 7 7 6
13925 160/300 Write block 1902 len 8192 data: 4 5
13926 161/300 Read block 1959 len 24576
13927 162/300 Write block 716 len 16384 data: 5 4 4 3
13928 163/300 Write block 2429 len 20480 data: 4 4 3 3 4
13929 164/300 Read block 1321 len 40960
13930 165/300 Read block 1050 len 24576
13931 166/300 Read block 3705 len 24576
13932 167/300 Flush
13933 168/300 Read block 57 len 32768
13934 169/300 Write block 4397 len 28672 data: 2 4 2 2 2 2 3
13935 170/300 Read block 1360 len 8192
13936 171/300 Write block 4187 len 40960 data: 2 2 2 2 3 2 2 2 2 2
13937 172/300 Write block 2670 len 16384 data: 5 6 6 6
13938 173/300 Read block 1624 len 4096
13939 174/300 Write block 3713 len 8192 data: 5 5
13940 175/300 Write block 3038 len 28672 data: 6 7 8 7 6 6 5
13941 176/300 Write block 214 len 24576 data: 4 4 4 3 3 4
13942 177/300 Write block 33 len 40960 data: 3 3 3 4 4 4 4 4 2 2
13943 178/300 Read block 2027 len 24576
13944 179/300 Write block 4157 len 4096 data: 3
13945 180/300 Read block 549 len 32768
13946 181/300 Flush
13947 182/300 Read block 896 len 36864
13948 183/300 Read block 840 len 24576
13949 184/300 Read block 941 len 12288
13950 185/300 Read block 2905 len 4096
13951 186/300 Flush
13952 187/300 Write block 2438 len 36864 data: 4 4 5 5 4 4 3 2 2
13953 188/300 Read block 761 len 32768
13954 189/300 Read block 3239 len 20480
13955 190/300 Write block 2849 len 20480 data: 5 6 5 5 6
13956 191/300 Flush
13957 192/300 Flush
13958 193/300 Write block 1738 len 20480 data: 5 4 4 2 2
13959 194/300 Read block 2708 len 40960
13960 195/300 Read block 3581 len 4096
13961 196/300 Write block 391 len 36864 data: 3 3 3 3 3 4 4 4 4
13962 197/300 Write block 1045 len 16384 data: 6 4 5 5
13963 198/300 Read block 3973 len 12288
13964 199/300 Read block 1120 len 28672
13965 200/300 Write block 4896 len 16384 data: 2 2 2 2
13966 201/300 Write block 4080 len 24576 data: 4 6 6 4 5 4
13967 202/300 Read block 2480 len 16384
13968 203/300 Flush
13969 204/300 Read block 4645 len 8192
13970 205/300 Read block 2452 len 32768
13971 206/300 Write block 68 len 32768 data: 3 3 3 4 3 4 5 5
13972 207/300 Write block 4344 len 20480 data: 8 6 6 5 6
13973 208/300 Read block 3973 len 12288
13974 209/300 Read block 3567 len 36864
13975 210/300 Flush
13976 211/300 Write block 957 len 36864 data: 5 6 6 5 5 6 5 5 5
13977 212/300 Write block 2609 len 28672 data: 4 4 4 3 3 3 4
13978 213/300 Read block 1068 len 8192
13979 214/300 Read block 3225 len 8192
13980 215/300 Write block 2811 len 8192 data: 2 2
13981 216/300 Read block 2747 len 28672
13982 217/300 Write block 4079 len 20480 data: 4 5 7 7 5
13983 218/300 Write block 2069 len 40960 data: 4 3 4 3 3 3 3 3 4 4
13984 219/300 Read block 2801 len 28672
13985 220/300 Read block 4824 len 4096
13986 221/300 Read block 3064 len 32768
13987 222/300 Read block 360 len 40960
13988 223/300 Write block 602 len 16384 data: 8 5 5 4
13989 224/300 Write block 4267 len 20480 data: 5 5 5 5 5
13990 225/300 Read block 4895 len 20480
13991 226/300 Read block 2015 len 4096
13992 227/300 Write block 4160 len 24576 data: 5 4 4 4 5 4
13993 228/300 Write block 3297 len 4096 data: 6
13994 229/300 Read block 1210 len 8192
13995 230/300 Read block 3798 len 28672
13996 231/300 Write block 4895 len 8192 data: 2 3
13997 232/300 Read block 3889 len 32768
13998 233/300 Write block 3678 len 4096 data: 4
13999 234/300 Read block 54 len 16384
14000 235/300 Read block 4875 len 28672
14001 236/300 Read block 1440 len 24576
14002 237/300 Read block 1006 len 20480
14003 238/300 Read block 2836 len 16384
14004 239/300 Read block 2021 len 8192
14005 240/300 Read block 3901 len 24576
14006 241/300 Read block 4041 len 8192
14007 242/300 Write block 2437 len 8192 data: 4 5
14008 243/300 Flush
14009 244/300 Read block 2119 len 20480
14010 245/300 Read block 1654 len 16384
14011 246/300 Write block 33 len 12288 data: 4 4 4
14012 247/300 Flush
14013 248/300 Write block 3112 len 16384 data: 3 3 4 4
14014 249/300 Read block 33 len 40960
14015 250/300 Write block 2960 len 12288 data: 3 3 4
14016 251/300 Write block 1296 len 36864 data: 2 2 2 2 3 3 3 3 5
14017 252/300 Read block 651 len 20480
14018 253/300 Read block 1888 len 28672
14019 254/300 Write block 4141 len 28672 data: 4 5 5 5 5 7 8
14020 255/300 Read block 3905 len 32768
14021 256/300 Read block 1907 len 40960
14022 257/300 Write block 1418 len 24576 data: 7 7 7 7 7 7
14023 258/300 Read block 3897 len 8192
14024 259/300 Write block 3684 len 24576 data: 3 3 3 4 4 4
14025 260/300 Read block 2841 len 4096
14026 261/300 Read block 2830 len 32768
14027 262/300 Write block 2649 len 28672 data: 3 2 2 4 4 4 5
14028 263/300 Write block 3700 len 40960 data: 3 2 3 3 3 2 2 2 2 2
14029 264/300 Write block 2535 len 28672 data: 5 5 5 4 5 7 5
14030 265/300 Read block 4470 len 24576
14031 266/300 Read block 4037 len 8192
14032 267/300 Write block 1719 len 20480 data: 3 3 3 2 2
14033 268/300 Write block 632 len 20480 data: 4 4 4 4 2
14034 269/300 Write block 3000 len 32768 data: 7 9 9 7 6 6 5 4
14035 270/300 Write block 901 len 4096 data: 5
14036 271/300 Read block 2707 len 28672
14037 272/300 Read block 854 len 28672
14038 273/300 Write block 4778 len 8192 data: 5 5
14039 274/300 Write block 2415 len 8192 data: 3 4
14040 275/300 Read block 4402 len 8192
14041 276/300 Write block 2015 len 32768 data: 5 4 4 5 4 4 4 4
14042 277/300 Flush
14043 278/300 Read block 4584 len 20480
14044 279/300 Read block 1494 len 4096
14045 280/300 Write block 1340 len 16384 data: 4 4 3 3
14046 281/300 Flush
14047 282/300 Read block 46 len 16384
14048 283/300 Read block 2472 len 24576
14049 284/300 Read block 1107 len 28672
14050 285/300 Read block 4116 len 4096
14051 286/300 Flush
14052 287/300 Flush
14053 288/300 Write block 3457 len 32768 data: 2 2 2 2 2 2 3 6
14054 289/300 Write block 128 len 36864 data: 3 3 3 5 4 4 4 4 4
14055 290/300 Read block 859 len 28672
14056 291/300 Read block 2103 len 8192
14057 292/300 Read block 4200 len 40960
14058 293/300 Read block 2454 len 24576
14059 294/300 Write block 168 len 36864 data: 3 3 3 2 2 3 3 3 5
14060 295/300 Write block 3431 len 8192 data: 4 2
14061 296/300 Write block 305 len 4096 data: 3
14062 297/300 Write block 2443 len 40960 data: 5 4 3 3 2 5 5 5 4 4
14063 298/300 Read block 1769 len 24576
14064 299/300 Write block 3920 len 40960 data: 4 3 3 3 3 4 5 5 4 4
14065 300/300 Flush
14066 301/300 Write block 4960 len 20480 data: 3 3 4 4 5
14067 Main task has work to do, go find it
14068 got dsc Start(1)
14069 start 1
14070 Jan 23 18:50:52.188 INFO request completed, latency_us: 188, response_code: 204, uri: /start/cid/1, method: POST, req_id: 101d1a16-3325-49c7-93eb-f5c3cc3fdf04, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
14071 Replay: started 1, returned:Ok(())
14072 ----------------------------------------------------------------
14073 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14074 States: Active Offline Active
14075 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14076 4622 Acked Read 4 Done Sent Done false
14077 4623 Acked Read 9 Done Sent Done false
14078 4624 Acked Write 1 Done Sent Done false
14079 4625 Acked Read 6 Done Sent Done false
14080 4626 Acked Read 6 Done Sent Done false
14081 4627 Acked Write 7 Done Sent Done false
14082 4628 Acked Read 9 Done Sent Done false
14083 4629 Acked Read 7 Done Sent Done false
14084 4630 Acked Read 6 Done Sent Done false
14085 4631 Acked Read 6 Done Sent Done false
14086 4632 Acked Read 4 Done Sent Done false
14087 4633 Acked Write 10 Done Sent Done false
14088 4634 Acked Read 1 Done Sent Done false
14089 4635 Acked Read 8 Done Sent Done false
14090 4636 Acked Read 10 Done Sent Done false
14091 4637 Acked Read 5 Done Sent Done false
14092 4638 Acked Write 5 Done Sent Done false
14093 4639 Acked Write 2 Done Sent Done false
14094 4640 Acked Flush 0 Done Sent Done false
14095 4641 Acked Read 4 Done Sent Done false
14096 4642 Acked Read 6 Done Sent Done false
14097 4643 Acked Write 5 Done Sent Done false
14098 4644 Acked Read 9 Done Sent Done false
14099 4645 Acked Read 1 Done Sent Done false
14100 4646 Acked Read 4 Done Sent Done false
14101 4647 Acked Write 2 Done Sent Done false
14102 4648 Acked Write 4 Done Sent Done false
14103 4649 Acked Write 4 Done Sent Done false
14104 4650 Acked Write 2 Done Sent Done false
14105 4651 Acked Flush 0 Done Sent Done false
14106 4652 Acked Read 3 Done Sent Done false
14107 4653 Acked Read 8 Done Sent Done false
14108 4654 Acked Read 5 Done Sent Done false
14109 4655 Acked Read 8 Done Sent Done false
14110 4656 Acked Read 9 Done Sent Done false
14111 4657 Acked Write 6 Done Sent Done false
14112 4658 Acked Read 9 Done Sent Done false
14113 4659 Acked Read 9 Done Sent Done false
14114 4660 Acked Read 7 Done Sent Done false
14115 4661 Acked Write 9 Done Sent Done false
14116 4662 Acked Read 10 Done Sent Done false
14117 4663 Acked Read 4 Done Sent Done false
14118 4664 Acked Read 8 Done Sent Done false
14119 4665 Acked Read 5 Done Sent Done false
14120 4666 Acked Read 8 Done Sent Done false
14121 4667 Acked Read 6 Done Sent Done false
14122 4668 Acked Write 8 Done Sent Done false
14123 4669 Acked Read 3 Done Sent Done false
14124 4670 Acked Read 9 Done Sent Done false
14125 4671 Acked Write 1 Done Sent Done false
14126 4672 Acked Read 6 Done Sent Done false
14127 4673 Acked Write 3 Done Sent Done false
14128 4674 Acked Write 8 Done Sent Done false
14129 4675 Acked Read 1 Done Sent Done false
14130 4676 Acked Read 1 Done Sent Done false
14131 4677 Acked Read 7 Done Sent Done false
14132 4678 Acked Read 8 Done Sent Done false
14133 4679 Acked Read 9 Done Sent Done false
14134 4680 Acked Read 1 Done Sent Done false
14135 4681 Acked Read 5 Done Sent Done false
14136 4682 Acked Read 4 Done Sent Done false
14137 4683 Acked Write 8 Done Sent Done false
14138 4684 Acked Read 2 Done Sent Done false
14139 4685 Acked Read 6 Done Sent Done false
14140 4686 Acked Write 1 Done Sent Done false
14141 4687 Acked Write 4 Done Sent Done false
14142 4688 Acked Flush 0 Done Sent Done false
14143 4689 Acked Flush 0 Done Sent Done false
14144 4690 Acked Read 6 Done Sent Done false
14145 4691 Acked Read 2 Done Sent Done false
14146 4692 Acked Read 6 Done Sent Done false
14147 4693 Acked Read 4 Done Sent Done false
14148 4694 Acked Read 9 Done Sent Done false
14149 4695 Acked Write 8 Done Sent Done false
14150 4696 Acked Write 9 Done Sent Done false
14151 4697 Acked Read 6 Done Sent Done false
14152 4698 Acked Write 5 Done Sent Done false
14153 4699 Acked Write 7 Done Sent Done false
14154 4700 Acked Write 7 Done Sent Done false
14155 4701 Acked Write 5 Done Sent Done false
14156 4702 Acked Read 8 Done Sent Done false
14157 4703 Acked Read 3 Done Sent Done false
14158 4704 Acked Read 9 Done Sent Done false
14159 4705 Acked Write 2 Done Sent Done false
14160 4706 Acked Write 2 Done Sent Done false
14161 4707 Acked Write 7 Done Sent Done false
14162 4708 Acked Write 6 Done Sent Done false
14163 4709 Acked Flush 0 Done Sent Done false
14164 4710 Acked Read 8 Done Sent Done false
14165 4711 Acked Flush 0 Done Sent Done false
14166 4712 Acked Write 8 Done Sent Done false
14167 4713 Acked Read 3 Done Sent Done false
14168 4714 Acked Write 4 Done Sent Done false
14169 4715 Acked Read 8 Done Sent Done false
14170 4716 Acked Write 5 Done Sent Done false
14171 4717 Acked Read 8 Done Sent Done false
14172 4718 Acked Read 1 Done Sent Done false
14173 4719 Acked Read 6 Done Sent Done false
14174 4720 Acked Read 10 Done Sent Done false
14175 4721 Acked Read 2 Done Sent Done false
14176 4722 Acked Write 10 Done Sent Done false
14177 4723 Acked Flush 0 Done Sent Done false
14178 4724 Acked Flush 0 Done Sent Done false
14179 4725 Acked Read 2 Done Sent Done false
14180 4726 Acked Flush 0 Done Sent Done false
14181 4727 Acked Read 7 Done Sent Done false
14182 4728 Acked Read 8 Done Sent Done false
14183 4729 Acked Read 3 Done Sent Done false
14184 4730 Acked Read 3 Done Sent Done false
14185 4731 Acked Read 2 Done Sent Done false
14186 4732 Acked Write 8 Done Sent Done false
14187 4733 Acked Read 3 Done Sent Done false
14188 4734 Acked Write 8 Done Sent Done false
14189 4735 Acked Read 9 Done Sent Done false
14190 4736 Acked Read 5 Done Sent Done false
14191 4737 Acked Read 1 Done Sent Done false
14192 4738 Acked Write 9 Done Sent Done false
14193 4739 Acked Read 4 Done Sent Done false
14194 4740 Acked Write 1 Done Sent Done false
14195 4741 Acked Flush 0 Done Sent Done false
14196 4742 Acked Read 8 Done Sent Done false
14197 4743 Acked Read 10 Done Sent Done false
14198 4744 Acked Read 1 Done Sent Done false
14199 4745 Acked Write 6 Done Sent Done false
14200 4746 Acked Write 1 Done Sent Done false
14201 4747 Acked Read 7 Done Sent Done false
14202 4748 Acked Read 7 Done Sent Done false
14203 4749 Acked Write 10 Done Sent Done false
14204 4750 Acked Read 7 Done Sent Done false
14205 4751 Acked Write 8 Done Sent Done false
14206 4752 Acked Write 7 Done Sent Done false
14207 4753 Acked Read 7 Done Sent Done false
14208 4754 Acked Read 6 Done Sent Done false
14209 4755 Acked Write 6 Done Sent Done false
14210 4756 Acked Read 7 Done Sent Done false
14211 4757 Acked Read 9 Done Sent Done false
14212 4758 Acked Read 10 Done Sent Done false
14213 4759 Acked Read 7 Done Sent Done false
14214 4760 Acked Read 1 Done Sent Done false
14215 4761 Acked Read 8 Done Sent Done false
14216 4762 Acked Read 7 Done Sent Done false
14217 4763 Acked Write 3 Done Sent Done false
14218 4764 Acked Write 8 Done Sent Done false
14219 4765 Acked Write 1 Done Sent Done false
14220 4766 Acked Flush 0 Done Sent Done false
14221 4767 Acked Read 3 Done Sent Done false
14222 4768 Acked Read 1 Done Sent Done false
14223 4769 Acked Read 4 Done Sent Done false
14224 4770 Acked Read 7 Done Sent Done false
14225 4771 Acked Read 1 Done Sent Done false
14226 4772 Acked Write 5 Done Sent Done false
14227 4773 Acked Read 1 Done Sent Done false
14228 4774 Acked Write 4 Done Sent Done false
14229 4775 Acked Read 10 Done Sent Done false
14230 4776 Acked Read 8 Done Sent Done false
14231 4777 Acked Write 3 Done Sent Done false
14232 4778 Acked Read 3 Done Sent Done false
14233 4779 Acked Read 9 Done Sent Done false
14234 4780 Acked Write 6 Done Sent Done false
14235 4781 Acked Write 2 Done Sent Done false
14236 4782 Acked Read 6 Done Sent Done false
14237 4783 Acked Write 4 Done Sent Done false
14238 4784 Acked Write 5 Done Sent Done false
14239 4785 Acked Read 10 Done Sent Done false
14240 4786 Acked Read 6 Done Sent Done false
14241 4787 Acked Read 6 Done Sent Done false
14242 4788 Acked Flush 0 Done Sent Done false
14243 4789 Acked Read 8 Done Sent Done false
14244 4790 Acked Write 7 Done Sent Done false
14245 4791 Acked Read 2 Done Sent Done false
14246 4792 Acked Write 10 Done Sent Done false
14247 4793 Acked Write 4 Done Sent Done false
14248 4794 Acked Read 1 Done Sent Done false
14249 4795 Acked Write 2 Done Sent Done false
14250 4796 Acked Write 7 Done Sent Done false
14251 4797 Acked Write 6 Done Sent Done false
14252 4798 Acked Write 10 Done Sent Done false
14253 4799 Acked Read 6 Done Sent Done false
14254 4800 Acked Write 1 Done Sent Done false
14255 4801 Acked Read 8 Done Sent Done false
14256 4802 Acked Flush 0 Done Sent Done false
14257 4803 Acked Read 9 Done Sent Done false
14258 4804 Acked Read 6 Done Sent Done false
14259 4805 Acked Read 3 Done Sent Done false
14260 4806 Acked Read 1 Done Sent Done false
14261 4807 Acked Flush 0 Done Sent Done false
14262 4808 Acked Write 9 Done Sent Done false
14263 4809 Acked Read 8 Done Sent Done false
14264 4810 Acked Read 5 Done Sent Done false
14265 4811 Acked Write 5 Done Sent Done false
14266 4812 Acked Flush 0 Done Sent Done false
14267 4813 Acked Flush 0 Done Sent Done false
14268 4814 Acked Write 5 Done Sent Done false
14269 4815 Acked Read 10 Done Sent Done false
14270 4816 Acked Read 1 Done Sent Done false
14271 4817 Acked Write 9 Done Sent Done false
14272 4818 Acked Write 4 Done Sent Done false
14273 4819 Acked Read 3 Done Sent Done false
14274 4820 Acked Read 7 Done Sent Done false
14275 4821 Acked Write 4 Done Sent Done false
14276 4822 Acked Write 6 Done Sent Done false
14277 4823 Acked Read 4 Done Sent Done false
14278 4824 Acked Flush 0 Done Sent Done false
14279 4825 Acked Read 2 Done Sent Done false
14280 4826 Acked Read 8 Done Sent Done false
14281 4827 Acked Write 8 Done Sent Done false
14282 4828 Acked Write 5 Done Sent Done false
14283 4829 Acked Read 3 Done Sent Done false
14284 4830 Acked Read 9 Done Sent Done false
14285 4831 Acked Flush 0 Done Sent Done false
14286 4832 Acked Write 9 Done Sent Done false
14287 4833 Acked Write 7 Done Sent Done false
14288 4834 Acked Read 2 Done Sent Done false
14289 4835 Acked Read 2 Done Sent Done false
14290 4836 Acked Write 2 Done Sent Done false
14291 4837 Acked Read 7 Done Sent Done false
14292 4838 Acked Write 5 Done Sent Done false
14293 4839 Acked Write 10 Done Sent Done false
14294 4840 Acked Read 7 Done Sent Done false
14295 4841 Acked Read 1 Done Sent Done false
14296 4842 Acked Read 8 Done Sent Done false
14297 4843 Acked Read 10 Done Sent Done false
14298 4844 Acked Write 4 Done Sent Done false
14299 4845 Acked Write 5 Done Sent Done false
14300 4846 Acked Read 5 Done Sent Done false
14301 4847 Acked Read 1 Done Sent Done false
14302 4848 Acked Write 6 Done Sent Done false
14303 4849 Acked Write 1 Done Sent Done false
14304 4850 Acked Read 2 Done Sent Done false
14305 4851 Acked Read 7 Done Sent Done false
14306 4852 Acked Write 2 Done Sent Done false
14307 4853 Acked Read 8 Done Sent Done false
14308 4854 Acked Write 1 Done Sent Done false
14309 4855 Acked Read 4 Done Sent Done false
14310 4856 Acked Read 7 Done Sent Done false
14311 4857 Acked Read 6 Done Sent Done false
14312 4858 Acked Read 5 Done Sent Done false
14313 4859 Acked Read 4 Done Sent Done false
14314 4860 Acked Read 2 Done Sent Done false
14315 4861 Acked Read 6 Done Sent Done false
14316 4862 Acked Read 2 Done Sent Done false
14317 4863 Acked Write 2 Done Sent Done false
14318 4864 Acked Flush 0 Done Sent Done false
14319 4865 Acked Read 5 Done Sent Done false
14320 4866 Acked Read 4 Done Sent Done false
14321 4867 Acked Write 3 Done Sent Done false
14322 4868 Acked Flush 0 Done Sent Done false
14323 4869 Acked Write 4 Done Sent Done false
14324 4870 Acked Read 10 Done Sent Done false
14325 4871 Acked Write 3 Done Sent Done false
14326 4872 Acked Write 9 Done Sent Done false
14327 4873 Acked Read 5 Done Sent Done false
14328 4874 Acked Read 7 Done Sent Done false
14329 4875 Acked Write 7 Done Sent Done false
14330 4876 Acked Read 8 Done Sent Done false
14331 4877 Acked Read 10 Done Sent Done false
14332 4878 Acked Write 6 Done Sent Done false
14333 4879 Acked Read 2 Done Sent Done false
14334 4880 Acked Write 6 Done Sent Done false
14335 4881 Acked Read 1 Done Sent Done false
14336 4882 Acked Read 8 Done Sent Done false
14337 4883 Acked Write 7 Done Sent Done false
14338 4884 Acked Write 10 Done Sent Done false
14339 4885 Acked Write 7 Done Sent Done false
14340 4886 Acked Read 6 Done Sent Done false
14341 4887 Acked Read 2 Done Sent Done false
14342 4888 Acked Write 5 Done Sent Done false
14343 4889 Acked Write 5 Done Sent Done false
14344 4890 Acked Write 8 Done Sent Done false
14345 4891 Acked Write 1 Done Sent Done false
14346 4892 Acked Read 7 Done Sent Done false
14347 4893 Acked Read 7 Done Sent Done false
14348 4894 Acked Write 2 Done Sent Done false
14349 4895 Acked Write 2 Done Sent Done false
14350 4896 Acked Read 2 Done Sent Done false
14351 4897 Acked Write 8 Done Sent Done false
14352 4898 Acked Flush 0 Done Sent Done false
14353 4899 Acked Read 5 Done Sent Done false
14354 4900 Acked Read 1 Done Sent Done false
14355 4901 Acked Write 4 Done Sent Done false
14356 4902 Acked Flush 0 Done Sent Done false
14357 4903 Acked Read 4 Done Sent Done false
14358 4904 Acked Read 6 Done Sent Done false
14359 4905 Acked Read 7 Done Sent Done false
14360 4906 Acked Read 1 Done Sent Done false
14361 4907 Acked Flush 0 Done Sent Done false
14362 4908 Acked Flush 0 Done Sent Done false
14363 4909 Acked Write 8 Done Sent Done false
14364 4910 Acked Write 9 Done Sent Done false
14365 4911 Acked Read 7 Done Sent Done false
14366 4912 Acked Read 2 Done Sent Done false
14367 4913 Acked Read 10 Done Sent Done false
14368 4914 Acked Read 6 Done Sent Done false
14369 4915 Acked Write 9 Done Sent Done false
14370 4916 Acked Write 2 Done Sent Done false
14371 4917 Acked Write 1 Done Sent Done false
14372 4918 Acked Write 10 Done Sent Done false
14373 4919 Acked Read 6 Done Sent Done false
14374 4920 Acked Write 10 Done Sent Done false
14375 4921 Acked Flush 0 Done Sent Done false
14376 4922 Acked Write 5 Done Sent Done false
14377 STATES DS:0 DS:1 DS:2 TOTAL
14378 Sent 0 301 0 301
14379 Done 301 0 301 602
14380 Skipped 0 0 0 0
14381 Error 0 0 0 0
14382 Last Flush: Some(JobId(4921)) Some(JobId(4621)) Some(JobId(4921))
14383 Downstairs last five completed: 4621 4620 4619 4618 4617
14384 Upstairs last five completed: 4922 4921 4920 4919 4918
14385 CLIENT: Up:0 ds:301 act:2
14386 [8820] got start action
14387 [8820] I am going to restart
14388 Starting downstairs at port 8820
14389 Make output file at "/tmp/dsc/downstairs-8820.txt"
14390 [8820][1] reports Starting
14391 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1213)
14392 [8820][1] reports Running
14393 ----------------------------------------------------------------
14394 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
14395 States: Active Offline Active
14396 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14397 4622 Acked Read 4 Done Sent Done false
14398 4623 Acked Read 9 Done Sent Done false
14399 4624 Acked Write 1 Done Sent Done false
14400 4625 Acked Read 6 Done Sent Done false
14401 4626 Acked Read 6 Done Sent Done false
14402 4627 Acked Write 7 Done Sent Done false
14403 4628 Acked Read 9 Done Sent Done false
14404 4629 Acked Read 7 Done Sent Done false
14405 4630 Acked Read 6 Done Sent Done false
14406 4631 Acked Read 6 Done Sent Done false
14407 4632 Acked Read 4 Done Sent Done false
14408 4633 Acked Write 10 Done Sent Done false
14409 4634 Acked Read 1 Done Sent Done false
14410 4635 Acked Read 8 Done Sent Done false
14411 4636 Acked Read 10 Done Sent Done false
14412 4637 Acked Read 5 Done Sent Done false
14413 4638 Acked Write 5 Done Sent Done false
14414 4639 Acked Write 2 Done Sent Done false
14415 4640 Acked Flush 0 Done Sent Done false
14416 4641 Acked Read 4 Done Sent Done false
14417 4642 Acked Read 6 Done Sent Done false
14418 4643 Acked Write 5 Done Sent Done false
14419 4644 Acked Read 9 Done Sent Done false
14420 4645 Acked Read 1 Done Sent Done false
14421 4646 Acked Read 4 Done Sent Done false
14422 4647 Acked Write 2 Done Sent Done false
14423 4648 Acked Write 4 Done Sent Done false
14424 4649 Acked Write 4 Done Sent Done false
14425 4650 Acked Write 2 Done Sent Done false
14426 4651 Acked Flush 0 Done Sent Done false
14427 4652 Acked Read 3 Done Sent Done false
14428 4653 Acked Read 8 Done Sent Done false
14429 4654 Acked Read 5 Done Sent Done false
14430 4655 Acked Read 8 Done Sent Done false
14431 4656 Acked Read 9 Done Sent Done false
14432 4657 Acked Write 6 Done Sent Done false
14433 4658 Acked Read 9 Done Sent Done false
14434 4659 Acked Read 9 Done Sent Done false
14435 4660 Acked Read 7 Done Sent Done false
14436 4661 Acked Write 9 Done Sent Done false
14437 4662 Acked Read 10 Done Sent Done false
14438 4663 Acked Read 4 Done Sent Done false
14439 4664 Acked Read 8 Done Sent Done false
14440 4665 Acked Read 5 Done Sent Done false
14441 4666 Acked Read 8 Done Sent Done false
14442 4667 Acked Read 6 Done Sent Done false
14443 4668 Acked Write 8 Done Sent Done false
14444 4669 Acked Read 3 Done Sent Done false
14445 4670 Acked Read 9 Done Sent Done false
14446 4671 Acked Write 1 Done Sent Done false
14447 4672 Acked Read 6 Done Sent Done false
14448 4673 Acked Write 3 Done Sent Done false
14449 4674 Acked Write 8 Done Sent Done false
14450 4675 Acked Read 1 Done Sent Done false
14451 4676 Acked Read 1 Done Sent Done false
14452 4677 Acked Read 7 Done Sent Done false
14453 4678 Acked Read 8 Done Sent Done false
14454 4679 Acked Read 9 Done Sent Done false
14455 4680 Acked Read 1 Done Sent Done false
14456 4681 Acked Read 5 Done Sent Done false
14457 4682 Acked Read 4 Done Sent Done false
14458 4683 Acked Write 8 Done Sent Done false
14459 4684 Acked Read 2 Done Sent Done false
14460 4685 Acked Read 6 Done Sent Done false
14461 4686 Acked Write 1 Done Sent Done false
14462 4687 Acked Write 4 Done Sent Done false
14463 4688 Acked Flush 0 Done Sent Done false
14464 4689 Acked Flush 0 Done Sent Done false
14465 4690 Acked Read 6 Done Sent Done false
14466 4691 Acked Read 2 Done Sent Done false
14467 4692 Acked Read 6 Done Sent Done false
14468 4693 Acked Read 4 Done Sent Done false
14469 4694 Acked Read 9 Done Sent Done false
14470 4695 Acked Write 8 Done Sent Done false
14471 4696 Acked Write 9 Done Sent Done false
14472 4697 Acked Read 6 Done Sent Done false
14473 4698 Acked Write 5 Done Sent Done false
14474 4699 Acked Write 7 Done Sent Done false
14475 4700 Acked Write 7 Done Sent Done false
14476 4701 Acked Write 5 Done Sent Done false
14477 4702 Acked Read 8 Done Sent Done false
14478 4703 Acked Read 3 Done Sent Done false
14479 4704 Acked Read 9 Done Sent Done false
14480 4705 Acked Write 2 Done Sent Done false
14481 4706 Acked Write 2 Done Sent Done false
14482 4707 Acked Write 7 Done Sent Done false
14483 4708 Acked Write 6 Done Sent Done false
14484 4709 Acked Flush 0 Done Sent Done false
14485 4710 Acked Read 8 Done Sent Done false
14486 4711 Acked Flush 0 Done Sent Done false
14487 4712 Acked Write 8 Done Sent Done false
14488 4713 Acked Read 3 Done Sent Done false
14489 4714 Acked Write 4 Done Sent Done false
14490 4715 Acked Read 8 Done Sent Done false
14491 4716 Acked Write 5 Done Sent Done false
14492 4717 Acked Read 8 Done Sent Done false
14493 4718 Acked Read 1 Done Sent Done false
14494 4719 Acked Read 6 Done Sent Done false
14495 4720 Acked Read 10 Done Sent Done false
14496 4721 Acked Read 2 Done Sent Done false
14497 4722 Acked Write 10 Done Sent Done false
14498 4723 Acked Flush 0 Done Sent Done false
14499 4724 Acked Flush 0 Done Sent Done false
14500 4725 Acked Read 2 Done Sent Done false
14501 4726 Acked Flush 0 Done Sent Done false
14502 4727 Acked Read 7 Done Sent Done false
14503 4728 Acked Read 8 Done Sent Done false
14504 4729 Acked Read 3 Done Sent Done false
14505 4730 Acked Read 3 Done Sent Done false
14506 4731 Acked Read 2 Done Sent Done false
14507 4732 Acked Write 8 Done Sent Done false
14508 4733 Acked Read 3 Done Sent Done false
14509 4734 Acked Write 8 Done Sent Done false
14510 4735 Acked Read 9 Done Sent Done false
14511 4736 Acked Read 5 Done Sent Done false
14512 4737 Acked Read 1 Done Sent Done false
14513 4738 Acked Write 9 Done Sent Done false
14514 4739 Acked Read 4 Done Sent Done false
14515 4740 Acked Write 1 Done Sent Done false
14516 4741 Acked Flush 0 Done Sent Done false
14517 4742 Acked Read 8 Done Sent Done false
14518 4743 Acked Read 10 Done Sent Done false
14519 4744 Acked Read 1 Done Sent Done false
14520 4745 Acked Write 6 Done Sent Done false
14521 4746 Acked Write 1 Done Sent Done false
14522 4747 Acked Read 7 Done Sent Done false
14523 4748 Acked Read 7 Done Sent Done false
14524 4749 Acked Write 10 Done Sent Done false
14525 4750 Acked Read 7 Done Sent Done false
14526 4751 Acked Write 8 Done Sent Done false
14527 4752 Acked Write 7 Done Sent Done false
14528 4753 Acked Read 7 Done Sent Done false
14529 4754 Acked Read 6 Done Sent Done false
14530 4755 Acked Write 6 Done Sent Done false
14531 4756 Acked Read 7 Done Sent Done false
14532 4757 Acked Read 9 Done Sent Done false
14533 4758 Acked Read 10 Done Sent Done false
14534 4759 Acked Read 7 Done Sent Done false
14535 4760 Acked Read 1 Done Sent Done false
14536 4761 Acked Read 8 Done Sent Done false
14537 4762 Acked Read 7 Done Sent Done false
14538 4763 Acked Write 3 Done Sent Done false
14539 4764 Acked Write 8 Done Sent Done false
14540 4765 Acked Write 1 Done Sent Done false
14541 4766 Acked Flush 0 Done Sent Done false
14542 4767 Acked Read 3 Done Sent Done false
14543 4768 Acked Read 1 Done Sent Done false
14544 4769 Acked Read 4 Done Sent Done false
14545 4770 Acked Read 7 Done Sent Done false
14546 4771 Acked Read 1 Done Sent Done false
14547 4772 Acked Write 5 Done Sent Done false
14548 4773 Acked Read 1 Done Sent Done false
14549 4774 Acked Write 4 Done Sent Done false
14550 4775 Acked Read 10 Done Sent Done false
14551 4776 Acked Read 8 Done Sent Done false
14552 4777 Acked Write 3 Done Sent Done false
14553 4778 Acked Read 3 Done Sent Done false
14554 4779 Acked Read 9 Done Sent Done false
14555 4780 Acked Write 6 Done Sent Done false
14556 4781 Acked Write 2 Done Sent Done false
14557 4782 Acked Read 6 Done Sent Done false
14558 4783 Acked Write 4 Done Sent Done false
14559 4784 Acked Write 5 Done Sent Done false
14560 4785 Acked Read 10 Done Sent Done false
14561 4786 Acked Read 6 Done Sent Done false
14562 4787 Acked Read 6 Done Sent Done false
14563 4788 Acked Flush 0 Done Sent Done false
14564 4789 Acked Read 8 Done Sent Done false
14565 4790 Acked Write 7 Done Sent Done false
14566 4791 Acked Read 2 Done Sent Done false
14567 4792 Acked Write 10 Done Sent Done false
14568 4793 Acked Write 4 Done Sent Done false
14569 4794 Acked Read 1 Done Sent Done false
14570 4795 Acked Write 2 Done Sent Done false
14571 4796 Acked Write 7 Done Sent Done false
14572 4797 Acked Write 6 Done Sent Done false
14573 4798 Acked Write 10 Done Sent Done false
14574 4799 Acked Read 6 Done Sent Done false
14575 4800 Acked Write 1 Done Sent Done false
14576 4801 Acked Read 8 Done Sent Done false
14577 4802 Acked Flush 0 Done Sent Done false
14578 4803 Acked Read 9 Done Sent Done false
14579 4804 Acked Read 6 Done Sent Done false
14580 4805 Acked Read 3 Done Sent Done false
14581 4806 Acked Read 1 Done Sent Done false
14582 4807 Acked Flush 0 Done Sent Done false
14583 4808 Acked Write 9 Done Sent Done false
14584 4809 Acked Read 8 Done Sent Done false
14585 4810 Acked Read 5 Done Sent Done false
14586 4811 Acked Write 5 Done Sent Done false
14587 4812 Acked Flush 0 Done Sent Done false
14588 4813 Acked Flush 0 Done Sent Done false
14589 4814 Acked Write 5 Done Sent Done false
14590 4815 Acked Read 10 Done Sent Done false
14591 4816 Acked Read 1 Done Sent Done false
14592 4817 Acked Write 9 Done Sent Done false
14593 4818 Acked Write 4 Done Sent Done false
14594 4819 Acked Read 3 Done Sent Done false
14595 4820 Acked Read 7 Done Sent Done false
14596 4821 Acked Write 4 Done Sent Done false
14597 4822 Acked Write 6 Done Sent Done false
14598 4823 Acked Read 4 Done Sent Done false
14599 4824 Acked Flush 0 Done Sent Done false
14600 4825 Acked Read 2 Done Sent Done false
14601 4826 Acked Read 8 Done Sent Done false
14602 4827 Acked Write 8 Done Sent Done false
14603 4828 Acked Write 5 Done Sent Done false
14604 4829 Acked Read 3 Done Sent Done false
14605 4830 Acked Read 9 Done Sent Done false
14606 4831 Acked Flush 0 Done Sent Done false
14607 4832 Acked Write 9 Done Sent Done false
14608 4833 Acked Write 7 Done Sent Done false
14609 4834 Acked Read 2 Done Sent Done false
14610 4835 Acked Read 2 Done Sent Done false
14611 4836 Acked Write 2 Done Sent Done false
14612 4837 Acked Read 7 Done Sent Done false
14613 4838 Acked Write 5 Done Sent Done false
14614 4839 Acked Write 10 Done Sent Done false
14615 4840 Acked Read 7 Done Sent Done false
14616 4841 Acked Read 1 Done Sent Done false
14617 4842 Acked Read 8 Done Sent Done false
14618 4843 Acked Read 10 Done Sent Done false
14619 4844 Acked Write 4 Done Sent Done false
14620 4845 Acked Write 5 Done Sent Done false
14621 4846 Acked Read 5 Done Sent Done false
14622 4847 Acked Read 1 Done Sent Done false
14623 4848 Acked Write 6 Done Sent Done false
14624 4849 Acked Write 1 Done Sent Done false
14625 4850 Acked Read 2 Done Sent Done false
14626 4851 Acked Read 7 Done Sent Done false
14627 4852 Acked Write 2 Done Sent Done false
14628 4853 Acked Read 8 Done Sent Done false
14629 4854 Acked Write 1 Done Sent Done false
14630 4855 Acked Read 4 Done Sent Done false
14631 4856 Acked Read 7 Done Sent Done false
14632 4857 Acked Read 6 Done Sent Done false
14633 4858 Acked Read 5 Done Sent Done false
14634 4859 Acked Read 4 Done Sent Done false
14635 4860 Acked Read 2 Done Sent Done false
14636 4861 Acked Read 6 Done Sent Done false
14637 4862 Acked Read 2 Done Sent Done false
14638 4863 Acked Write 2 Done Sent Done false
14639 4864 Acked Flush 0 Done Sent Done false
14640 4865 Acked Read 5 Done Sent Done false
14641 4866 Acked Read 4 Done Sent Done false
14642 4867 Acked Write 3 Done Sent Done false
14643 4868 Acked Flush 0 Done Sent Done false
14644 4869 Acked Write 4 Done Sent Done false
14645 4870 Acked Read 10 Done Sent Done false
14646 4871 Acked Write 3 Done Sent Done false
14647 4872 Acked Write 9 Done Sent Done false
14648 4873 Acked Read 5 Done Sent Done false
14649 4874 Acked Read 7 Done Sent Done false
14650 4875 Acked Write 7 Done Sent Done false
14651 4876 Acked Read 8 Done Sent Done false
14652 4877 Acked Read 10 Done Sent Done false
14653 4878 Acked Write 6 Done Sent Done false
14654 4879 Acked Read 2 Done Sent Done false
14655 4880 Acked Write 6 Done Sent Done false
14656 4881 Acked Read 1 Done Sent Done false
14657 4882 Acked Read 8 Done Sent Done false
14658 4883 Acked Write 7 Done Sent Done false
14659 4884 Acked Write 10 Done Sent Done false
14660 4885 Acked Write 7 Done Sent Done false
14661 4886 Acked Read 6 Done Sent Done false
14662 4887 Acked Read 2 Done Sent Done false
14663 4888 Acked Write 5 Done Sent Done false
14664 4889 Acked Write 5 Done Sent Done false
14665 4890 Acked Write 8 Done Sent Done false
14666 4891 Acked Write 1 Done Sent Done false
14667 4892 Acked Read 7 Done Sent Done false
14668 4893 Acked Read 7 Done Sent Done false
14669 4894 Acked Write 2 Done Sent Done false
14670 4895 Acked Write 2 Done Sent Done false
14671 4896 Acked Read 2 Done Sent Done false
14672 4897 Acked Write 8 Done Sent Done false
14673 4898 Acked Flush 0 Done Sent Done false
14674 4899 Acked Read 5 Done Sent Done false
14675 4900 Acked Read 1 Done Sent Done false
14676 4901 Acked Write 4 Done Sent Done false
14677 4902 Acked Flush 0 Done Sent Done false
14678 4903 Acked Read 4 Done Sent Done false
14679 4904 Acked Read 6 Done Sent Done false
14680 4905 Acked Read 7 Done Sent Done false
14681 4906 Acked Read 1 Done Sent Done false
14682 4907 Acked Flush 0 Done Sent Done false
14683 4908 Acked Flush 0 Done Sent Done false
14684 4909 Acked Write 8 Done Sent Done false
14685 4910 Acked Write 9 Done Sent Done false
14686 4911 Acked Read 7 Done Sent Done false
14687 4912 Acked Read 2 Done Sent Done false
14688 4913 Acked Read 10 Done Sent Done false
14689 4914 Acked Read 6 Done Sent Done false
14690 4915 Acked Write 9 Done Sent Done false
14691 4916 Acked Write 2 Done Sent Done false
14692 4917 Acked Write 1 Done Sent Done false
14693 4918 Acked Write 10 Done Sent Done false
14694 4919 Acked Read 6 Done Sent Done false
14695 4920 Acked Write 10 Done Sent Done false
14696 4921 Acked Flush 0 Done Sent Done false
14697 4922 Acked Write 5 Done Sent Done false
14698 4923 Acked Flush 0 Done Sent Done false
14699 STATES DS:0 DS:1 DS:2 TOTAL
14700 Sent 0 302 0 302
14701 Done 302 0 302 604
14702 Skipped 0 0 0 0
14703 Error 0 0 0 0
14704 Last Flush: Some(JobId(4923)) Some(JobId(4621)) Some(JobId(4923))
14705 Downstairs last five completed: 4621 4620 4619 4618 4617
14706 Upstairs last five completed: 4923 4922 4921 4920 4919
14707 CLIENT: Up:0 ds:302 act:2
147082025-01-23T18:50:57.934ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
147092025-01-23T18:50:57.934ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56885) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
147102025-01-23T18:50:57.935ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
147112025-01-23T18:50:57.935ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
147122025-01-23T18:50:57.935ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
147132025-01-23T18:50:57.935ZINFOcrucible: send last flush ID to this DS: Some(JobId(4621)) = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
147142025-01-23T18:50:57.935ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
147152025-01-23T18:50:57.935ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(4621)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
147162025-01-23T18:50:57.936ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(4621)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
14717 ----------------------------------------------------------------
14718 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14719 Downstairs last five completed: 4923 4922 4921 4920 4919
14720 Upstairs last five completed: 4923 4922 4921 4920 4919
14721 CLIENT: Up:0 ds:0 act:3
14722 Replay: All jobs finished
14723 Main task has work to do, go find it
14724 got dsc Stop(1)
14725 stop 1
14726 Jan 23 18:51:00.193 INFO request completed, latency_us: 304, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 6087e436-f189-4fa9-9170-ee2f5e31a1ed, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
14727 [8820] Got stop action so:false kr:false
14728 Jan 23 18:51:00.194 INFO request completed, latency_us: 108, response_code: 200, uri: /state/cid/1, method: GET, req_id: 828f8421-461f-47a4-b97e-48ce089e5d4b, remote_addr: 127.0.0.1:59295, local_addr:{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:51:00.194636759Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
14729 127.0.0.1:9998
147302025-01-23T18:51:00.194ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
147312025-01-23T18:51:00.194ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
147322025-01-23T18:51:00.194ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
14733 [8820] Exited with: ExitStatus(unix_wait_status(9))
14734 [8820][1] reports Exit
14735 Jan 23 18:51:04.196 INFO request completed, latency_us: 202, response_code: 200, uri: /state/cid/1, method: GET, req_id: 920b693e-b62a-4525-8735-79bb41111970, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
14736 001/300 Read block 4874 len 16384
14737 002/300 Write block 247 len 28672 data: 5 5 4 3 4 5 4
14738 003/300 Read block 3561 len 32768
14739 004/300 Write block 1261 len 28672 data: 4 5 5 5 5 3 3
14740 005/300 Write block 3799 len 40960 data: 5 4 5 5 4 4 3 4 3 3
14741 006/300 Read block 1569 len 4096
14742 007/300 Read block 1711 len 28672
14743 008/300 Read block 3821 len 16384
14744 009/300 Read block 726 len 12288
14745 010/300 Read block 656 len 12288
14746 011/300 Read block 1350 len 24576
14747 012/300 Read block 3495 len 28672
14748 013/300 Read block 2241 len 24576
14749 014/300 Flush
14750 015/300 Flush
14751 016/300 Write block 1596 len 16384 data: 3 3 2 2
14752 017/300 Write block 1135 len 4096 data: 7
14753 018/300 Write block 3842 len 12288 data: 4 3 3
14754 019/300 Read block 2949 len 40960
14755 020/300 Read block 2342 len 24576
14756 021/300 Read block 3840 len 40960
14757 022/300 Write block 487 len 28672 data: 5 5 4 5 5 5 5
14758 023/300 Write block 3817 len 32768 data: 4 4 4 4 4 3 4 4
14759 024/300 Read block 3394 len 36864
14760 025/300 Read block 407 len 32768
14761 026/300 Read block 241 len 40960
14762 027/300 Read block 1386 len 4096
14763 028/300 Write block 439 len 16384 data: 3 3 3 3
14764 029/300 Write block 1963 len 12288 data: 2 3 3
14765 030/300 Read block 1049 len 36864
14766 031/300 Flush
14767 032/300 Read block 1155 len 20480
14768 033/300 Read block 2241 len 8192
14769 034/300 Write block 467 len 4096 data: 4
14770 035/300 Read block 4726 len 28672
14771 036/300 Write block 1807 len 40960 data: 4 4 5 5 7 6 6 6 6 4
14772 037/300 Write block 3919 len 36864 data: 4 5 4 4 4 4 5 6 6
14773 038/300 Write block 3961 len 8192 data: 2 2
14774 039/300 Read block 3211 len 16384
14775 040/300 Flush
14776 041/300 Write block 11 len 8192 data: 3 4
14777 042/300 Write block 3219 len 24576 data: 3 3 3 3 3 4
14778 043/300 Read block 1325 len 16384
14779 044/300 Write block 1563 len 32768 data: 3 3 3 3 4 4 4 5
14780 045/300 Write block 1030 len 40960 data: 4 4 4 2 2 2 4 4 4 3
14781 046/300 Read block 4347 len 8192
14782 047/300 Read block 4879 len 24576
14783 048/300 Read block 3950 len 32768
14784 049/300 Write block 4845 len 20480 data: 2 3 5 5 4
14785 050/300 Flush
14786 051/300 Read block 1013 len 12288
14787 052/300 Read block 1405 len 8192
14788 053/300 Read block 3969 len 32768
14789 054/300 Write block 4029 len 24576 data: 5 5 5 3 3 2
14790 055/300 Read block 3376 len 24576
14791 056/300 Write block 334 len 28672 data: 3 3 3 3 3 4 5
14792 057/300 Write block 3660 len 40960 data: 3 3 3 2 2 3 4 4 4 4
14793 058/300 Read block 2751 len 28672
14794 059/300 Read block 2862 len 4096
14795 060/300 Write block 129 len 40960 data: 4 4 6 5 5 5 5 5 4 5
14796 061/300 Write block 2363 len 12288 data: 3 3 3
14797 062/300 Write block 868 len 32768 data: 4 4 3 3 3 3 4 3
14798 063/300 Read block 2314 len 32768
14799 064/300 Flush
14800 065/300 Write block 3386 len 20480 data: 7 7 6 5 4
14801 066/300 Flush
14802 067/300 Read block 1927 len 24576
14803 068/300 Write block 70 len 8192 data: 4 5
14804 069/300 Write block 1234 len 28672 data: 4 4 4 4 5 5 6
14805 070/300 Flush
14806 071/300 Write block 2108 len 16384 data: 4 4 3 2
14807 072/300 Write block 1689 len 36864 data: 5 5 5 6 6 6 6 6 5
14808 073/300 Read block 938 len 20480
14809 074/300 Read block 2428 len 12288
14810 075/300 Write block 4120 len 40960 data: 2 2 2 2 2 3 2 2 2 2
14811 076/300 Read block 2575 len 36864
14812 077/300 Read block 262 len 24576
14813 078/300 Write block 4704 len 24576 data: 4 3 4 3 4 4
14814 079/300 Read block 249 len 32768
14815 080/300 Read block 1135 len 28672
14816 081/300 Read block 4021 len 4096
14817 082/300 Read block 930 len 36864
14818 083/300 Flush
14819 084/300 Read block 3336 len 40960
14820 085/300 Read block 1292 len 4096
14821 086/300 Write block 3885 len 20480 data: 7 6 5 5 4
14822 087/300 Write block 3457 len 32768 data: 3 3 3 3 3 3 4 7
14823 088/300 Read block 3572 len 36864
14824 089/300 Write block 3909 len 12288 data: 2 2 3
14825 090/300 Flush
14826 091/300 Write block 4002 len 28672 data: 4 4 2 2 3 4 4
14827 092/300 Write block 289 len 12288 data: 3 3 3
14828 093/300 Write block 3457 len 36864 data: 4 4 4 4 4 4 5 8 6
14829 094/300 Read block 2434 len 8192
14830 095/300 Read block 4656 len 40960
14831 096/300 Write block 2069 len 36864 data: 5 4 5 4 4 4 4 4 5
14832 097/300 Write block 2546 len 12288 data: 4 3 3
14833 098/300 Write block 1867 len 16384 data: 6 4 4 4
14834 099/300 Read block 3282 len 28672
14835 100/300 Read block 4656 len 24576
14836 101/300 Read block 2168 len 4096
14837 102/300 Read block 699 len 36864
14838 103/300 Read block 2506 len 16384
14839 104/300 Write block 291 len 8192 data: 4 2
14840 105/300 Write block 4581 len 28672 data: 4 5 4 4 3 3 3
14841 106/300 Flush
14842 107/300 Write block 3040 len 36864 data: 9 8 7 7 6 8 8 7 8
14843 108/300 Read block 4621 len 16384
14844 109/300 Write block 3088 len 40960 data: 2 2 2 2 2 2 2 2 2 2
14845 110/300 Read block 4440 len 20480
14846 111/300 Read block 2220 len 28672
14847 112/300 Read block 264 len 36864
14848 113/300 Write block 2846 len 4096 data: 5
14849 114/300 Write block 3514 len 12288 data: 3 4 4
14850 115/300 Read block 899 len 24576
14851 116/300 Write block 4619 len 20480 data: 2 2 2 3 3
14852 117/300 Read block 1159 len 32768
14853 118/300 Read block 571 len 40960
14854 119/300 Flush
14855 120/300 Write block 3575 len 24576 data: 5 3 3 2 2 2
14856 121/300 Read block 1951 len 32768
14857 122/300 Read block 8 len 12288
14858 123/300 Read block 554 len 28672
14859 124/300 Flush
14860 125/300 Write block 2401 len 16384 data: 2 2 6 6
14861 126/300 Read block 114 len 36864
14862 127/300 Write block 592 len 32768 data: 3 3 3 3 3 3 4 5
14863 128/300 Read block 3194 len 40960
14864 129/300 Flush
14865 130/300 Read block 19 len 8192
14866 131/300 Write block 2299 len 20480 data: 4 3 4 5 6
14867 132/300 Write block 2233 len 12288 data: 2 2 2
14868 133/300 Flush
14869 134/300 Flush
14870 135/300 Write block 92 len 32768 data: 6 5 6 5 5 5 2 2
14871 136/300 Flush
14872 137/300 Read block 2128 len 4096
14873 138/300 Read block 4813 len 28672
14874 139/300 Write block 937 len 4096 data: 5
14875 140/300 Write block 3584 len 32768 data: 3 3 3 3 3 2 2 2
14876 141/300 Write block 4060 len 32768 data: 3 3 2 2 3 4 5 5
14877 142/300 Write block 333 len 28672 data: 3 4 4 4 4 4 5
14878 143/300 Read block 1863 len 24576
14879 144/300 Write block 4818 len 28672 data: 2 2 2 2 4 5 5
14880 145/300 Write block 2683 len 32768 data: 4 3 4 4 4 4 4 4
14881 146/300 Read block 886 len 8192
14882 147/300 Flush
14883 148/300 Read block 2073 len 32768
14884 149/300 Read block 470 len 36864
14885 150/300 Write block 977 len 40960 data: 2 2 2 2 3 3 3 3 4 4
14886 151/300 Read block 4075 len 24576
14887 152/300 Write block 597 len 20480 data: 4 5 6 5 5
14888 153/300 Read block 4216 len 20480
14889 154/300 Read block 1532 len 40960
14890 155/300 Write block 3332 len 36864 data: 2 2 2 2 3 3 3 3 3
14891 156/300 Write block 1099 len 24576 data: 5 4 4 5 5 5
14892 157/300 Read block 4560 len 16384
14893 158/300 Read block 3503 len 24576
14894 159/300 Read block 4994 len 16384
14895 160/300 Write block 1868 len 20480 data: 5 5 5 4 4
14896 161/300 Read block 572 len 24576
14897 162/300 Read block 3117 len 4096
14898 163/300 Write block 3736 len 20480 data: 8 7 7 6 6
14899 164/300 Write block 609 len 16384 data: 3 3 2 2
14900 165/300 Write block 2077 len 36864 data: 6 5 6 5 5 4 4 3 3
14901 166/300 Read block 14 len 8192
14902 167/300 Read block 2924 len 36864
14903 168/300 Write block 2432 len 20480 data: 4 5 4 4 4
14904 169/300 Read block 4651 len 24576
14905 170/300 Read block 4812 len 28672
14906 171/300 Read block 1076 len 36864
14907 172/300 Read block 4342 len 24576
14908 173/300 Write block 1772 len 28672 data: 4 4 5 4 4 4 4
14909 174/300 Write block 3121 len 28672 data: 5 4 4 3 3 3 3
14910 175/300 Read block 112 len 16384
14911 176/300 Flush
14912 177/300 Write block 642 len 4096 data: 3
14913 178/300 Write block 3631 len 36864 data: 3 3 4 4 3 3 2 4 5
14914 179/300 Write block 4368 len 32768 data: 4 5 5 4 4 5 5 6
14915 180/300 Write block 1387 len 40960 data: 4 3 3 4 4 2 3 3 5 5
14916 181/300 Read block 478 len 8192
14917 182/300 Write block 114 len 40960 data: 7 7 5 4 5 4 3 3 4 4
14918 183/300 Read block 2353 len 24576
14919 184/300 Read block 287 len 28672
14920 185/300 Read block 905 len 40960
14921 186/300 Read block 3037 len 4096
14922 187/300 Read block 1316 len 24576
14923 188/300 Read block 4410 len 4096
14924 189/300 Write block 3225 len 32768 data: 4 3 3 3 4 5 6 7
14925 190/300 Write block 3275 len 28672 data: 3 3 3 2 2 3 3
14926 191/300 Write block 1630 len 36864 data: 3 3 3 3 3 4 3 3 3
14927 192/300 Read block 4711 len 32768
14928 193/300 Read block 3993 len 40960
14929 194/300 Write block 3007 len 36864 data: 5 3 3 3 2 3 4 5 5
14930 195/300 Write block 1098 len 12288 data: 5 6 5
14931 196/300 Write block 1812 len 40960 data: 7 7 7 7 5 4 4 2 3 3
14932 197/300 Read block 3277 len 40960
14933 198/300 Write block 1465 len 12288 data: 7 7 6
14934 199/300 Read block 621 len 8192
14935 200/300 Read block 1700 len 32768
14936 201/300 Read block 1909 len 36864
14937 202/300 Read block 2713 len 32768
14938 203/300 Read block 2467 len 32768
14939 204/300 Read block 3320 len 20480
14940 205/300 Write block 1286 len 4096 data: 6
14941 206/300 Write block 3009 len 36864 data: 4 4 3 4 5 6 6 5 4
14942 207/300 Write block 4143 len 32768 data: 6 6 6 8 9 8 9 9
14943 208/300 Write block 4768 len 4096 data: 2
14944 209/300 Write block 201 len 28672 data: 2 2 2 2 2 2 2
14945 210/300 Read block 148 len 24576
14946 211/300 Read block 3872 len 40960
14947 212/300 Write block 4139 len 28672 data: 4 4 5 6 7 7 7
14948 213/300 Flush
14949 214/300 Read block 4760 len 16384
14950 215/300 Read block 3497 len 12288
14951 216/300 Read block 980 len 12288
14952 217/300 Write block 2921 len 4096 data: 4
14953 218/300 Read block 2818 len 32768
14954 219/300 Write block 3985 len 16384 data: 3 3 4 4
14955 220/300 Write block 4043 len 28672 data: 3 3 3 5 5 5 5
14956 221/300 Read block 1139 len 12288
14957 222/300 Read block 2510 len 12288
14958 223/300 Flush
14959 224/300 Read block 2280 len 32768
14960 225/300 Read block 739 len 28672
14961 226/300 Read block 1269 len 16384
14962 227/300 Flush
14963 228/300 Flush
14964 229/300 Write block 1384 len 28672 data: 4 3 3 5 4 4 5
14965 230/300 Write block 758 len 4096 data: 3
14966 231/300 Write block 4143 len 24576 data: 8 8 8 9 10 9
14967 232/300 Read block 1565 len 16384
14968 233/300 Read block 4928 len 24576
14969 234/300 Read block 2653 len 16384
14970 235/300 Write block 2868 len 24576 data: 4 4 5 6 5 4
14971 236/300 Flush
14972 237/300 Write block 1145 len 24576 data: 6 6 5 5 5 4
14973 238/300 Read block 4396 len 40960
14974 239/300 Read block 1225 len 20480
14975 240/300 Read block 4344 len 32768
14976 241/300 Write block 1911 len 32768 data: 3 4 4 4 4 3 3 3
14977 242/300 Read block 993 len 20480
14978 243/300 Write block 39 len 8192 data: 5 5
14979 244/300 Read block 119 len 20480
14980 245/300 Read block 2026 len 32768
14981 246/300 Read block 1361 len 36864
14982 247/300 Read block 2951 len 20480
14983 248/300 Read block 367 len 36864
14984 249/300 Read block 1462 len 24576
14985 250/300 Write block 462 len 24576 data: 6 6 4 3 3 5
14986 251/300 Read block 77 len 24576
14987 252/300 Read block 963 len 24576
14988 253/300 Write block 893 len 20480 data: 2 3 4 6 5
14989 254/300 Read block 4464 len 16384
14990 255/300 Read block 3996 len 16384
14991 256/300 Read block 3666 len 20480
14992 257/300 Read block 3438 len 32768
14993 258/300 Flush
14994 259/300 Write block 3339 len 32768 data: 4 4 3 3 4 4 5 5
14995 260/300 Write block 2055 len 36864 data: 3 4 4 4 4 3 3 2 2
14996 261/300 Read block 3894 len 16384
14997 262/300 Write block 3975 len 36864 data: 4 4 4 3 3 3 2 2 2
14998 263/300 Read block 3407 len 32768
14999 264/300 Read block 779 len 4096
15000 265/300 Read block 1515 len 36864
15001 266/300 Read block 766 len 40960
15002 267/300 Write block 2132 len 20480 data: 4 4 4 4 4
15003 268/300 Read block 4169 len 16384
15004 269/300 Read block 4591 len 20480
15005 270/300 Read block 2815 len 28672
15006 271/300 Read block 2532 len 32768
15007 272/300 Write block 1191 len 40960 data: 4 3 4 4 4 3 3 2 3 5
15008 273/300 Read block 2742 len 20480
15009 274/300 Read block 3250 len 24576
15010 275/300 Write block 4473 len 32768 data: 4 4 4 3 3 2 3 3
15011 276/300 Write block 123 len 40960 data: 5 4 5 5 3 4 5 5 7 6
15012 277/300 Flush
15013 278/300 Write block 3568 len 16384 data: 5 5 4 3
15014 279/300 Write block 1672 len 4096 data: 4
15015 280/300 Write block 3420 len 20480 data: 5 6 7 8 7
15016 281/300 Flush
15017 282/300 Read block 4100 len 20480
15018 283/300 Read block 1104 len 40960
15019 284/300 Read block 665 len 28672
15020 285/300 Write block 2948 len 36864 data: 3 3 3 3 2 2 2 2 3
15021 286/300 Write block 3782 len 36864 data: 3 3 3 4 3 4 3 3 3
15022 287/300 Flush
15023 288/300 Write block 2904 len 28672 data: 2 3 3 3 4 4 4
15024 289/300 Write block 3359 len 24576 data: 5 4 6 6 6 5
15025 290/300 Write block 1350 len 24576 data: 2 2 2 3 3 3
15026 291/300 Read block 1950 len 20480
15027 292/300 Read block 1902 len 12288
15028 293/300 Write block 1351 len 8192 data: 3 3
15029 294/300 Read block 1085 len 20480
15030 295/300 Read block 440 len 36864
15031 296/300 Read block 3978 len 36864
15032 297/300 Write block 2342 len 32768 data: 3 3 3 2 2 2 3 3
15033 298/300 Read block 4907 len 36864
15034 299/300 Read block 4590 len 40960
15035 300/300 Write block 2772 len 28672 data: 3 3 3 3 5 4 2
15036 301/300 Read block 926 len 28672
15037 Main task has work to do, go find it
15038 got dsc Start(1)
15039 start 1
15040 Jan 23 18:51:04.452 INFO request completed, latency_us: 182, response_code: 204, uri: /start/cid/1, method: POST, req_id: ac2944e2-95d7-41ad-9492-62b933a5e35e, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
15041 Replay: started 1, returned:Ok(())
15042 ----------------------------------------------------------------
15043 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15044 States: Active Offline Active
15045 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15046 4924 Acked Read 4 Done Sent Done false
15047 4925 Acked Write 7 Done Sent Done false
15048 4926 Acked Read 8 Done Sent Done false
15049 4927 Acked Write 7 Done Sent Done false
15050 4928 Acked Write 10 Done Sent Done false
15051 4929 Acked Read 1 Done Sent Done false
15052 4930 Acked Read 7 Done Sent Done false
15053 4931 Acked Read 4 Done Sent Done false
15054 4932 Acked Read 3 Done Sent Done false
15055 4933 Acked Read 3 Done Sent Done false
15056 4934 Acked Read 6 Done Sent Done false
15057 4935 Acked Read 7 Done Sent Done false
15058 4936 Acked Read 6 Done Sent Done false
15059 4937 Acked Flush 0 Done Sent Done false
15060 4938 Acked Flush 0 Done Sent Done false
15061 4939 Acked Write 4 Done Sent Done false
15062 4940 Acked Write 1 Done Sent Done false
15063 4941 Acked Write 3 Done Sent Done false
15064 4942 Acked Read 10 Done Sent Done false
15065 4943 Acked Read 6 Done Sent Done false
15066 4944 Acked Read 10 Done Sent Done false
15067 4945 Acked Write 7 Done Sent Done false
15068 4946 Acked Write 8 Done Sent Done false
15069 4947 Acked Read 9 Done Sent Done false
15070 4948 Acked Read 8 Done Sent Done false
15071 4949 Acked Read 10 Done Sent Done false
15072 4950 Acked Read 1 Done Sent Done false
15073 4951 Acked Write 4 Done Sent Done false
15074 4952 Acked Write 3 Done Sent Done false
15075 4953 Acked Read 9 Done Sent Done false
15076 4954 Acked Flush 0 Done Sent Done false
15077 4955 Acked Read 5 Done Sent Done false
15078 4956 Acked Read 2 Done Sent Done false
15079 4957 Acked Write 1 Done Sent Done false
15080 4958 Acked Read 7 Done Sent Done false
15081 4959 Acked Write 10 Done Sent Done false
15082 4960 Acked Write 9 Done Sent Done false
15083 4961 Acked Write 2 Done Sent Done false
15084 4962 Acked Read 4 Done Sent Done false
15085 4963 Acked Flush 0 Done Sent Done false
15086 4964 Acked Write 2 Done Sent Done false
15087 4965 Acked Write 6 Done Sent Done false
15088 4966 Acked Read 4 Done Sent Done false
15089 4967 Acked Write 8 Done Sent Done false
15090 4968 Acked Write 10 Done Sent Done false
15091 4969 Acked Read 2 Done Sent Done false
15092 4970 Acked Read 6 Done Sent Done false
15093 4971 Acked Read 8 Done Sent Done false
15094 4972 Acked Write 5 Done Sent Done false
15095 4973 Acked Flush 0 Done Sent Done false
15096 4974 Acked Read 3 Done Sent Done false
15097 4975 Acked Read 2 Done Sent Done false
15098 4976 Acked Read 8 Done Sent Done false
15099 4977 Acked Write 6 Done Sent Done false
15100 4978 Acked Read 6 Done Sent Done false
15101 4979 Acked Write 7 Done Sent Done false
15102 4980 Acked Write 10 Done Sent Done false
15103 4981 Acked Read 7 Done Sent Done false
15104 4982 Acked Read 1 Done Sent Done false
15105 4983 Acked Write 10 Done Sent Done false
15106 4984 Acked Write 3 Done Sent Done false
15107 4985 Acked Write 8 Done Sent Done false
15108 4986 Acked Read 8 Done Sent Done false
15109 4987 Acked Flush 0 Done Sent Done false
15110 4988 Acked Write 5 Done Sent Done false
15111 4989 Acked Flush 0 Done Sent Done false
15112 4990 Acked Read 6 Done Sent Done false
15113 4991 Acked Write 2 Done Sent Done false
15114 4992 Acked Write 7 Done Sent Done false
15115 4993 Acked Flush 0 Done Sent Done false
15116 4994 Acked Write 4 Done Sent Done false
15117 4995 Acked Write 9 Done Sent Done false
15118 4996 Acked Read 5 Done Sent Done false
15119 4997 Acked Read 3 Done Sent Done false
15120 4998 Acked Write 10 Done Sent Done false
15121 4999 Acked Read 9 Done Sent Done false
15122 5000 Acked Read 6 Done Sent Done false
15123 5001 Acked Write 6 Done Sent Done false
15124 5002 Acked Read 8 Done Sent Done false
15125 5003 Acked Read 7 Done Sent Done false
15126 5004 Acked Read 1 Done Sent Done false
15127 5005 Acked Read 9 Done Sent Done false
15128 5006 Acked Flush 0 Done Sent Done false
15129 5007 Acked Read 10 Done Sent Done false
15130 5008 Acked Read 1 Done Sent Done false
15131 5009 Acked Write 5 Done Sent Done false
15132 5010 Acked Write 8 Done Sent Done false
15133 5011 Acked Read 9 Done Sent Done false
15134 5012 Acked Write 3 Done Sent Done false
15135 5013 Acked Flush 0 Done Sent Done false
15136 5014 Acked Write 7 Done Sent Done false
15137 5015 Acked Write 3 Done Sent Done false
15138 5016 Acked Write 9 Done Sent Done false
15139 5017 Acked Read 2 Done Sent Done false
15140 5018 Acked Read 10 Done Sent Done false
15141 5019 Acked Write 9 Done Sent Done false
15142 5020 Acked Write 3 Done Sent Done false
15143 5021 Acked Write 4 Done Sent Done false
15144 5022 Acked Read 7 Done Sent Done false
15145 5023 Acked Read 6 Done Sent Done false
15146 5024 Acked Read 1 Done Sent Done false
15147 5025 Acked Read 9 Done Sent Done false
15148 5026 Acked Read 4 Done Sent Done false
15149 5027 Acked Write 2 Done Sent Done false
15150 5028 Acked Write 7 Done Sent Done false
15151 5029 Acked Flush 0 Done Sent Done false
15152 5030 Acked Write 9 Done Sent Done false
15153 5031 Acked Read 4 Done Sent Done false
15154 5032 Acked Write 10 Done Sent Done false
15155 5033 Acked Read 5 Done Sent Done false
15156 5034 Acked Read 7 Done Sent Done false
15157 5035 Acked Read 9 Done Sent Done false
15158 5036 Acked Write 1 Done Sent Done false
15159 5037 Acked Write 3 Done Sent Done false
15160 5038 Acked Read 6 Done Sent Done false
15161 5039 Acked Write 5 Done Sent Done false
15162 5040 Acked Read 8 Done Sent Done false
15163 5041 Acked Read 10 Done Sent Done false
15164 5042 Acked Flush 0 Done Sent Done false
15165 5043 Acked Write 6 Done Sent Done false
15166 5044 Acked Read 8 Done Sent Done false
15167 5045 Acked Read 3 Done Sent Done false
15168 5046 Acked Read 7 Done Sent Done false
15169 5047 Acked Flush 0 Done Sent Done false
15170 5048 Acked Write 4 Done Sent Done false
15171 5049 Acked Read 9 Done Sent Done false
15172 5050 Acked Write 8 Done Sent Done false
15173 5051 Acked Read 10 Done Sent Done false
15174 5052 Acked Flush 0 Done Sent Done false
15175 5053 Acked Read 2 Done Sent Done false
15176 5054 Acked Write 5 Done Sent Done false
15177 5055 Acked Write 3 Done Sent Done false
15178 5056 Acked Flush 0 Done Sent Done false
15179 5057 Acked Flush 0 Done Sent Done false
15180 5058 Acked Write 8 Done Sent Done false
15181 5059 Acked Flush 0 Done Sent Done false
15182 5060 Acked Read 1 Done Sent Done false
15183 5061 Acked Read 7 Done Sent Done false
15184 5062 Acked Write 1 Done Sent Done false
15185 5063 Acked Write 8 Done Sent Done false
15186 5064 Acked Write 8 Done Sent Done false
15187 5065 Acked Write 7 Done Sent Done false
15188 5066 Acked Read 6 Done Sent Done false
15189 5067 Acked Write 7 Done Sent Done false
15190 5068 Acked Write 8 Done Sent Done false
15191 5069 Acked Read 2 Done Sent Done false
15192 5070 Acked Flush 0 Done Sent Done false
15193 5071 Acked Read 8 Done Sent Done false
15194 5072 Acked Read 9 Done Sent Done false
15195 5073 Acked Write 10 Done Sent Done false
15196 5074 Acked Read 6 Done Sent Done false
15197 5075 Acked Write 5 Done Sent Done false
15198 5076 Acked Read 5 Done Sent Done false
15199 5077 Acked Read 10 Done Sent Done false
15200 5078 Acked Write 9 Done Sent Done false
15201 5079 Acked Write 6 Done Sent Done false
15202 5080 Acked Read 4 Done Sent Done false
15203 5081 Acked Read 6 Done Sent Done false
15204 5082 Acked Read 4 Done Sent Done false
15205 5083 Acked Write 5 Done Sent Done false
15206 5084 Acked Read 6 Done Sent Done false
15207 5085 Acked Read 1 Done Sent Done false
15208 5086 Acked Write 5 Done Sent Done false
15209 5087 Acked Write 4 Done Sent Done false
15210 5088 Acked Write 9 Done Sent Done false
15211 5089 Acked Read 2 Done Sent Done false
15212 5090 Acked Read 9 Done Sent Done false
15213 5091 Acked Write 5 Done Sent Done false
15214 5092 Acked Read 6 Done Sent Done false
15215 5093 Acked Read 7 Done Sent Done false
15216 5094 Acked Read 9 Done Sent Done false
15217 5095 Acked Read 6 Done Sent Done false
15218 5096 Acked Write 7 Done Sent Done false
15219 5097 Acked Write 7 Done Sent Done false
15220 5098 Acked Read 4 Done Sent Done false
15221 5099 Acked Flush 0 Done Sent Done false
15222 5100 Acked Write 1 Done Sent Done false
15223 5101 Acked Write 9 Done Sent Done false
15224 5102 Acked Write 8 Done Sent Done false
15225 5103 Acked Write 10 Done Sent Done false
15226 5104 Acked Read 2 Done Sent Done false
15227 5105 Acked Write 10 Done Sent Done false
15228 5106 Acked Read 6 Done Sent Done false
15229 5107 Acked Read 7 Done Sent Done false
15230 5108 Acked Read 10 Done Sent Done false
15231 5109 Acked Read 1 Done Sent Done false
15232 5110 Acked Read 6 Done Sent Done false
15233 5111 Acked Read 1 Done Sent Done false
15234 5112 Acked Write 8 Done Sent Done false
15235 5113 Acked Write 7 Done Sent Done false
15236 5114 Acked Write 9 Done Sent Done false
15237 5115 Acked Read 8 Done Sent Done false
15238 5116 Acked Read 10 Done Sent Done false
15239 5117 Acked Write 9 Done Sent Done false
15240 5118 Acked Write 3 Done Sent Done false
15241 5119 Acked Write 10 Done Sent Done false
15242 5120 Acked Read 10 Done Sent Done false
15243 5121 Acked Write 3 Done Sent Done false
15244 5122 Acked Read 2 Done Sent Done false
15245 5123 Acked Read 8 Done Sent Done false
15246 5124 Acked Read 9 Done Sent Done false
15247 5125 Acked Read 8 Done Sent Done false
15248 5126 Acked Read 8 Done Sent Done false
15249 5127 Acked Read 5 Done Sent Done false
15250 5128 Acked Write 1 Done Sent Done false
15251 5129 Acked Write 9 Done Sent Done false
15252 5130 Acked Write 8 Done Sent Done false
15253 5131 Acked Write 1 Done Sent Done false
15254 5132 Acked Write 7 Done Sent Done false
15255 5133 Acked Read 6 Done Sent Done false
15256 5134 Acked Read 10 Done Sent Done false
15257 5135 Acked Write 7 Done Sent Done false
15258 5136 Acked Flush 0 Done Sent Done false
15259 5137 Acked Read 4 Done Sent Done false
15260 5138 Acked Read 3 Done Sent Done false
15261 5139 Acked Read 3 Done Sent Done false
15262 5140 Acked Write 1 Done Sent Done false
15263 5141 Acked Read 8 Done Sent Done false
15264 5142 Acked Write 4 Done Sent Done false
15265 5143 Acked Write 7 Done Sent Done false
15266 5144 Acked Read 3 Done Sent Done false
15267 5145 Acked Read 3 Done Sent Done false
15268 5146 Acked Flush 0 Done Sent Done false
15269 5147 Acked Read 8 Done Sent Done false
15270 5148 Acked Read 7 Done Sent Done false
15271 5149 Acked Read 4 Done Sent Done false
15272 5150 Acked Flush 0 Done Sent Done false
15273 5151 Acked Flush 0 Done Sent Done false
15274 5152 Acked Write 7 Done Sent Done false
15275 5153 Acked Write 1 Done Sent Done false
15276 5154 Acked Write 6 Done Sent Done false
15277 5155 Acked Read 4 Done Sent Done false
15278 5156 Acked Read 6 Done Sent Done false
15279 5157 Acked Read 4 Done Sent Done false
15280 5158 Acked Write 6 Done Sent Done false
15281 5159 Acked Flush 0 Done Sent Done false
15282 5160 Acked Write 6 Done Sent Done false
15283 5161 Acked Read 10 Done Sent Done false
15284 5162 Acked Read 5 Done Sent Done false
15285 5163 Acked Read 8 Done Sent Done false
15286 5164 Acked Write 8 Done Sent Done false
15287 5165 Acked Read 5 Done Sent Done false
15288 5166 Acked Write 2 Done Sent Done false
15289 5167 Acked Read 5 Done Sent Done false
15290 5168 Acked Read 8 Done Sent Done false
15291 5169 Acked Read 9 Done Sent Done false
15292 5170 Acked Read 5 Done Sent Done false
15293 5171 Acked Read 9 Done Sent Done false
15294 5172 Acked Read 6 Done Sent Done false
15295 5173 Acked Write 6 Done Sent Done false
15296 5174 Acked Read 6 Done Sent Done false
15297 5175 Acked Read 6 Done Sent Done false
15298 5176 Acked Write 5 Done Sent Done false
15299 5177 Acked Read 4 Done Sent Done false
15300 5178 Acked Read 4 Done Sent Done false
15301 5179 Acked Read 5 Done Sent Done false
15302 5180 Acked Read 8 Done Sent Done false
15303 5181 Acked Flush 0 Done Sent Done false
15304 5182 Acked Write 8 Done Sent Done false
15305 5183 Acked Write 9 Done Sent Done false
15306 5184 Acked Read 4 Done Sent Done false
15307 5185 Acked Write 9 Done Sent Done false
15308 5186 Acked Read 8 Done Sent Done false
15309 5187 Acked Read 1 Done Sent Done false
15310 5188 Acked Read 9 Done Sent Done false
15311 5189 Acked Read 10 Done Sent Done false
15312 5190 Acked Write 5 Done Sent Done false
15313 5191 Acked Read 4 Done Sent Done false
15314 5192 Acked Read 5 Done Sent Done false
15315 5193 Acked Read 7 Done Sent Done false
15316 5194 Acked Read 8 Done Sent Done false
15317 5195 Acked Write 10 Done Sent Done false
15318 5196 Acked Read 5 Done Sent Done false
15319 5197 Acked Read 6 Done Sent Done false
15320 5198 Acked Write 8 Done Sent Done false
15321 5199 Acked Write 10 Done Sent Done false
15322 5200 Acked Flush 0 Done Sent Done false
15323 5201 Acked Write 4 Done Sent Done false
15324 5202 Acked Write 1 Done Sent Done false
15325 5203 Acked Write 5 Done Sent Done false
15326 5204 Acked Flush 0 Done Sent Done false
15327 5205 Acked Read 5 Done Sent Done false
15328 5206 Acked Read 10 Done Sent Done false
15329 5207 Acked Read 7 Done Sent Done false
15330 5208 Acked Write 9 Done Sent Done false
15331 5209 Acked Write 9 Done Sent Done false
15332 5210 Acked Flush 0 Done Sent Done false
15333 5211 Acked Write 7 Done Sent Done false
15334 5212 Acked Write 6 Done Sent Done false
15335 5213 Acked Write 6 Done Sent Done false
15336 5214 Acked Read 5 Done Sent Done false
15337 5215 Acked Read 3 Done Sent Done false
15338 5216 Acked Write 2 Done Sent Done false
15339 5217 Acked Read 5 Done Sent Done false
15340 5218 Acked Read 9 Done Sent Done false
15341 5219 Acked Read 9 Done Sent Done false
15342 5220 Acked Write 8 Done Sent Done false
15343 5221 Acked Read 9 Done Sent Done false
15344 5222 Acked Read 10 Done Sent Done false
15345 5223 Acked Write 7 Done Sent Done false
15346 5224 Acked Read 7 Done Sent Done false
15347 STATES DS:0 DS:1 DS:2 TOTAL
15348 Sent 0 301 0 301
15349 Done 301 0 301 602
15350 Skipped 0 0 0 0
15351 Error 0 0 0 0
15352 Last Flush: Some(JobId(5210)) Some(JobId(4923)) Some(JobId(5210))
15353 Downstairs last five completed: 4923 4922 4921 4920 4919
15354 Upstairs last five completed: 5224 5223 5222 5221 5220
15355 CLIENT: Up:0 ds:301 act:2
15356 [8820] got start action
15357 [8820] I am going to restart
15358 Starting downstairs at port 8820
15359 Make output file at "/tmp/dsc/downstairs-8820.txt"
15360 [8820][1] reports Starting
15361 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1214)
15362 [8820][1] reports Running
15363 ----------------------------------------------------------------
15364 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
15365 States: Active Offline Active
15366 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15367 4924 Acked Read 4 Done Sent Done false
15368 4925 Acked Write 7 Done Sent Done false
15369 4926 Acked Read 8 Done Sent Done false
15370 4927 Acked Write 7 Done Sent Done false
15371 4928 Acked Write 10 Done Sent Done false
15372 4929 Acked Read 1 Done Sent Done false
15373 4930 Acked Read 7 Done Sent Done false
15374 4931 Acked Read 4 Done Sent Done false
15375 4932 Acked Read 3 Done Sent Done false
15376 4933 Acked Read 3 Done Sent Done false
15377 4934 Acked Read 6 Done Sent Done false
15378 4935 Acked Read 7 Done Sent Done false
15379 4936 Acked Read 6 Done Sent Done false
15380 4937 Acked Flush 0 Done Sent Done false
15381 4938 Acked Flush 0 Done Sent Done false
15382 4939 Acked Write 4 Done Sent Done false
15383 4940 Acked Write 1 Done Sent Done false
15384 4941 Acked Write 3 Done Sent Done false
15385 4942 Acked Read 10 Done Sent Done false
15386 4943 Acked Read 6 Done Sent Done false
15387 4944 Acked Read 10 Done Sent Done false
15388 4945 Acked Write 7 Done Sent Done false
15389 4946 Acked Write 8 Done Sent Done false
15390 4947 Acked Read 9 Done Sent Done false
15391 4948 Acked Read 8 Done Sent Done false
15392 4949 Acked Read 10 Done Sent Done false
15393 4950 Acked Read 1 Done Sent Done false
15394 4951 Acked Write 4 Done Sent Done false
15395 4952 Acked Write 3 Done Sent Done false
15396 4953 Acked Read 9 Done Sent Done false
15397 4954 Acked Flush 0 Done Sent Done false
15398 4955 Acked Read 5 Done Sent Done false
15399 4956 Acked Read 2 Done Sent Done false
15400 4957 Acked Write 1 Done Sent Done false
15401 4958 Acked Read 7 Done Sent Done false
15402 4959 Acked Write 10 Done Sent Done false
15403 4960 Acked Write 9 Done Sent Done false
15404 4961 Acked Write 2 Done Sent Done false
15405 4962 Acked Read 4 Done Sent Done false
15406 4963 Acked Flush 0 Done Sent Done false
15407 4964 Acked Write 2 Done Sent Done false
15408 4965 Acked Write 6 Done Sent Done false
15409 4966 Acked Read 4 Done Sent Done false
15410 4967 Acked Write 8 Done Sent Done false
15411 4968 Acked Write 10 Done Sent Done false
15412 4969 Acked Read 2 Done Sent Done false
15413 4970 Acked Read 6 Done Sent Done false
15414 4971 Acked Read 8 Done Sent Done false
15415 4972 Acked Write 5 Done Sent Done false
15416 4973 Acked Flush 0 Done Sent Done false
15417 4974 Acked Read 3 Done Sent Done false
15418 4975 Acked Read 2 Done Sent Done false
15419 4976 Acked Read 8 Done Sent Done false
15420 4977 Acked Write 6 Done Sent Done false
15421 4978 Acked Read 6 Done Sent Done false
15422 4979 Acked Write 7 Done Sent Done false
15423 4980 Acked Write 10 Done Sent Done false
15424 4981 Acked Read 7 Done Sent Done false
15425 4982 Acked Read 1 Done Sent Done false
15426 4983 Acked Write 10 Done Sent Done false
15427 4984 Acked Write 3 Done Sent Done false
15428 4985 Acked Write 8 Done Sent Done false
15429 4986 Acked Read 8 Done Sent Done false
15430 4987 Acked Flush 0 Done Sent Done false
15431 4988 Acked Write 5 Done Sent Done false
15432 4989 Acked Flush 0 Done Sent Done false
15433 4990 Acked Read 6 Done Sent Done false
15434 4991 Acked Write 2 Done Sent Done false
15435 4992 Acked Write 7 Done Sent Done false
15436 4993 Acked Flush 0 Done Sent Done false
15437 4994 Acked Write 4 Done Sent Done false
15438 4995 Acked Write 9 Done Sent Done false
15439 4996 Acked Read 5 Done Sent Done false
15440 4997 Acked Read 3 Done Sent Done false
15441 4998 Acked Write 10 Done Sent Done false
15442 4999 Acked Read 9 Done Sent Done false
15443 5000 Acked Read 6 Done Sent Done false
15444 5001 Acked Write 6 Done Sent Done false
15445 5002 Acked Read 8 Done Sent Done false
15446 5003 Acked Read 7 Done Sent Done false
15447 5004 Acked Read 1 Done Sent Done false
15448 5005 Acked Read 9 Done Sent Done false
15449 5006 Acked Flush 0 Done Sent Done false
15450 5007 Acked Read 10 Done Sent Done false
15451 5008 Acked Read 1 Done Sent Done false
15452 5009 Acked Write 5 Done Sent Done false
15453 5010 Acked Write 8 Done Sent Done false
15454 5011 Acked Read 9 Done Sent Done false
15455 5012 Acked Write 3 Done Sent Done false
15456 5013 Acked Flush 0 Done Sent Done false
15457 5014 Acked Write 7 Done Sent Done false
15458 5015 Acked Write 3 Done Sent Done false
15459 5016 Acked Write 9 Done Sent Done false
15460 5017 Acked Read 2 Done Sent Done false
15461 5018 Acked Read 10 Done Sent Done false
15462 5019 Acked Write 9 Done Sent Done false
15463 5020 Acked Write 3 Done Sent Done false
15464 5021 Acked Write 4 Done Sent Done false
15465 5022 Acked Read 7 Done Sent Done false
15466 5023 Acked Read 6 Done Sent Done false
15467 5024 Acked Read 1 Done Sent Done false
15468 5025 Acked Read 9 Done Sent Done false
15469 5026 Acked Read 4 Done Sent Done false
15470 5027 Acked Write 2 Done Sent Done false
15471 5028 Acked Write 7 Done Sent Done false
15472 5029 Acked Flush 0 Done Sent Done false
15473 5030 Acked Write 9 Done Sent Done false
15474 5031 Acked Read 4 Done Sent Done false
15475 5032 Acked Write 10 Done Sent Done false
15476 5033 Acked Read 5 Done Sent Done false
15477 5034 Acked Read 7 Done Sent Done false
15478 5035 Acked Read 9 Done Sent Done false
15479 5036 Acked Write 1 Done Sent Done false
15480 5037 Acked Write 3 Done Sent Done false
15481 5038 Acked Read 6 Done Sent Done false
15482 5039 Acked Write 5 Done Sent Done false
15483 5040 Acked Read 8 Done Sent Done false
15484 5041 Acked Read 10 Done Sent Done false
15485 5042 Acked Flush 0 Done Sent Done false
15486 5043 Acked Write 6 Done Sent Done false
15487 5044 Acked Read 8 Done Sent Done false
15488 5045 Acked Read 3 Done Sent Done false
15489 5046 Acked Read 7 Done Sent Done false
15490 5047 Acked Flush 0 Done Sent Done false
15491 5048 Acked Write 4 Done Sent Done false
15492 5049 Acked Read 9 Done Sent Done false
15493 5050 Acked Write 8 Done Sent Done false
15494 5051 Acked Read 10 Done Sent Done false
15495 5052 Acked Flush 0 Done Sent Done false
15496 5053 Acked Read 2 Done Sent Done false
15497 5054 Acked Write 5 Done Sent Done false
15498 5055 Acked Write 3 Done Sent Done false
15499 5056 Acked Flush 0 Done Sent Done false
15500 5057 Acked Flush 0 Done Sent Done false
15501 5058 Acked Write 8 Done Sent Done false
15502 5059 Acked Flush 0 Done Sent Done false
15503 5060 Acked Read 1 Done Sent Done false
15504 5061 Acked Read 7 Done Sent Done false
15505 5062 Acked Write 1 Done Sent Done false
15506 5063 Acked Write 8 Done Sent Done false
15507 5064 Acked Write 8 Done Sent Done false
15508 5065 Acked Write 7 Done Sent Done false
15509 5066 Acked Read 6 Done Sent Done false
15510 5067 Acked Write 7 Done Sent Done false
15511 5068 Acked Write 8 Done Sent Done false
15512 5069 Acked Read 2 Done Sent Done false
15513 5070 Acked Flush 0 Done Sent Done false
15514 5071 Acked Read 8 Done Sent Done false
15515 5072 Acked Read 9 Done Sent Done false
15516 5073 Acked Write 10 Done Sent Done false
15517 5074 Acked Read 6 Done Sent Done false
15518 5075 Acked Write 5 Done Sent Done false
15519 5076 Acked Read 5 Done Sent Done false
15520 5077 Acked Read 10 Done Sent Done false
15521 5078 Acked Write 9 Done Sent Done false
15522 5079 Acked Write 6 Done Sent Done false
15523 5080 Acked Read 4 Done Sent Done false
15524 5081 Acked Read 6 Done Sent Done false
15525 5082 Acked Read 4 Done Sent Done false
15526 5083 Acked Write 5 Done Sent Done false
15527 5084 Acked Read 6 Done Sent Done false
15528 5085 Acked Read 1 Done Sent Done false
15529 5086 Acked Write 5 Done Sent Done false
15530 5087 Acked Write 4 Done Sent Done false
15531 5088 Acked Write 9 Done Sent Done false
15532 5089 Acked Read 2 Done Sent Done false
15533 5090 Acked Read 9 Done Sent Done false
15534 5091 Acked Write 5 Done Sent Done false
15535 5092 Acked Read 6 Done Sent Done false
15536 5093 Acked Read 7 Done Sent Done false
15537 5094 Acked Read 9 Done Sent Done false
15538 5095 Acked Read 6 Done Sent Done false
15539 5096 Acked Write 7 Done Sent Done false
15540 5097 Acked Write 7 Done Sent Done false
15541 5098 Acked Read 4 Done Sent Done false
15542 5099 Acked Flush 0 Done Sent Done false
15543 5100 Acked Write 1 Done Sent Done false
15544 5101 Acked Write 9 Done Sent Done false
15545 5102 Acked Write 8 Done Sent Done false
15546 5103 Acked Write 10 Done Sent Done false
15547 5104 Acked Read 2 Done Sent Done false
15548 5105 Acked Write 10 Done Sent Done false
15549 5106 Acked Read 6 Done Sent Done false
15550 5107 Acked Read 7 Done Sent Done false
15551 5108 Acked Read 10 Done Sent Done false
15552 5109 Acked Read 1 Done Sent Done false
15553 5110 Acked Read 6 Done Sent Done false
15554 5111 Acked Read 1 Done Sent Done false
15555 5112 Acked Write 8 Done Sent Done false
15556 5113 Acked Write 7 Done Sent Done false
15557 5114 Acked Write 9 Done Sent Done false
15558 5115 Acked Read 8 Done Sent Done false
15559 5116 Acked Read 10 Done Sent Done false
15560 5117 Acked Write 9 Done Sent Done false
15561 5118 Acked Write 3 Done Sent Done false
15562 5119 Acked Write 10 Done Sent Done false
15563 5120 Acked Read 10 Done Sent Done false
15564 5121 Acked Write 3 Done Sent Done false
15565 5122 Acked Read 2 Done Sent Done false
15566 5123 Acked Read 8 Done Sent Done false
15567 5124 Acked Read 9 Done Sent Done false
15568 5125 Acked Read 8 Done Sent Done false
15569 5126 Acked Read 8 Done Sent Done false
15570 5127 Acked Read 5 Done Sent Done false
15571 5128 Acked Write 1 Done Sent Done false
15572 5129 Acked Write 9 Done Sent Done false
15573 5130 Acked Write 8 Done Sent Done false
15574 5131 Acked Write 1 Done Sent Done false
15575 5132 Acked Write 7 Done Sent Done false
15576 5133 Acked Read 6 Done Sent Done false
15577 5134 Acked Read 10 Done Sent Done false
15578 5135 Acked Write 7 Done Sent Done false
15579 5136 Acked Flush 0 Done Sent Done false
15580 5137 Acked Read 4 Done Sent Done false
15581 5138 Acked Read 3 Done Sent Done false
15582 5139 Acked Read 3 Done Sent Done false
15583 5140 Acked Write 1 Done Sent Done false
15584 5141 Acked Read 8 Done Sent Done false
15585 5142 Acked Write 4 Done Sent Done false
15586 5143 Acked Write 7 Done Sent Done false
15587 5144 Acked Read 3 Done Sent Done false
15588 5145 Acked Read 3 Done Sent Done false
15589 5146 Acked Flush 0 Done Sent Done false
15590 5147 Acked Read 8 Done Sent Done false
15591 5148 Acked Read 7 Done Sent Done false
15592 5149 Acked Read 4 Done Sent Done false
15593 5150 Acked Flush 0 Done Sent Done false
15594 5151 Acked Flush 0 Done Sent Done false
15595 5152 Acked Write 7 Done Sent Done false
15596 5153 Acked Write 1 Done Sent Done false
15597 5154 Acked Write 6 Done Sent Done false
15598 5155 Acked Read 4 Done Sent Done false
15599 5156 Acked Read 6 Done Sent Done false
15600 5157 Acked Read 4 Done Sent Done false
15601 5158 Acked Write 6 Done Sent Done false
15602 5159 Acked Flush 0 Done Sent Done false
15603 5160 Acked Write 6 Done Sent Done false
15604 5161 Acked Read 10 Done Sent Done false
15605 5162 Acked Read 5 Done Sent Done false
15606 5163 Acked Read 8 Done Sent Done false
15607 5164 Acked Write 8 Done Sent Done false
15608 5165 Acked Read 5 Done Sent Done false
15609 5166 Acked Write 2 Done Sent Done false
15610 5167 Acked Read 5 Done Sent Done false
15611 5168 Acked Read 8 Done Sent Done false
15612 5169 Acked Read 9 Done Sent Done false
15613 5170 Acked Read 5 Done Sent Done false
15614 5171 Acked Read 9 Done Sent Done false
15615 5172 Acked Read 6 Done Sent Done false
15616 5173 Acked Write 6 Done Sent Done false
15617 5174 Acked Read 6 Done Sent Done false
15618 5175 Acked Read 6 Done Sent Done false
15619 5176 Acked Write 5 Done Sent Done false
15620 5177 Acked Read 4 Done Sent Done false
15621 5178 Acked Read 4 Done Sent Done false
15622 5179 Acked Read 5 Done Sent Done false
15623 5180 Acked Read 8 Done Sent Done false
15624 5181 Acked Flush 0 Done Sent Done false
15625 5182 Acked Write 8 Done Sent Done false
15626 5183 Acked Write 9 Done Sent Done false
15627 5184 Acked Read 4 Done Sent Done false
15628 5185 Acked Write 9 Done Sent Done false
15629 5186 Acked Read 8 Done Sent Done false
15630 5187 Acked Read 1 Done Sent Done false
15631 5188 Acked Read 9 Done Sent Done false
15632 5189 Acked Read 10 Done Sent Done false
15633 5190 Acked Write 5 Done Sent Done false
15634 5191 Acked Read 4 Done Sent Done false
15635 5192 Acked Read 5 Done Sent Done false
15636 5193 Acked Read 7 Done Sent Done false
15637 5194 Acked Read 8 Done Sent Done false
15638 5195 Acked Write 10 Done Sent Done false
15639 5196 Acked Read 5 Done Sent Done false
15640 5197 Acked Read 6 Done Sent Done false
15641 5198 Acked Write 8 Done Sent Done false
15642 5199 Acked Write 10 Done Sent Done false
15643 5200 Acked Flush 0 Done Sent Done false
15644 5201 Acked Write 4 Done Sent Done false
15645 5202 Acked Write 1 Done Sent Done false
15646 5203 Acked Write 5 Done Sent Done false
15647 5204 Acked Flush 0 Done Sent Done false
15648 5205 Acked Read 5 Done Sent Done false
15649 5206 Acked Read 10 Done Sent Done false
15650 5207 Acked Read 7 Done Sent Done false
15651 5208 Acked Write 9 Done Sent Done false
15652 5209 Acked Write 9 Done Sent Done false
15653 5210 Acked Flush 0 Done Sent Done false
15654 5211 Acked Write 7 Done Sent Done false
15655 5212 Acked Write 6 Done Sent Done false
15656 5213 Acked Write 6 Done Sent Done false
15657 5214 Acked Read 5 Done Sent Done false
15658 5215 Acked Read 3 Done Sent Done false
15659 5216 Acked Write 2 Done Sent Done false
15660 5217 Acked Read 5 Done Sent Done false
15661 5218 Acked Read 9 Done Sent Done false
15662 5219 Acked Read 9 Done Sent Done false
15663 5220 Acked Write 8 Done Sent Done false
15664 5221 Acked Read 9 Done Sent Done false
15665 5222 Acked Read 10 Done Sent Done false
15666 5223 Acked Write 7 Done Sent Done false
15667 5224 Acked Read 7 Done Sent Done false
15668 5225 Acked Flush 0 Done Sent Done false
15669 STATES DS:0 DS:1 DS:2 TOTAL
15670 Sent 0 302 0 302
15671 Done 302 0 302 604
15672 Skipped 0 0 0 0
15673 Error 0 0 0 0
15674 Last Flush: Some(JobId(5225)) Some(JobId(4923)) Some(JobId(5225))
15675 Downstairs last five completed: 4923 4922 4921 4920 4919
15676 Upstairs last five completed: 5225 5224 5223 5222 5221
15677 CLIENT: Up:0 ds:302 act:2
156782025-01-23T18:51:10.195ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
156792025-01-23T18:51:10.195ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36059) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
156802025-01-23T18:51:10.196ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
156812025-01-23T18:51:10.197ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
156822025-01-23T18:51:10.197ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
156832025-01-23T18:51:10.197ZINFOcrucible: send last flush ID to this DS: Some(JobId(4923)) = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
156842025-01-23T18:51:10.197ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
156852025-01-23T18:51:10.197ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(4923)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
156862025-01-23T18:51:10.197ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(4923)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
15687 ----------------------------------------------------------------
15688 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15689 Downstairs last five completed: 5225 5224 5223 5222 5221
15690 Upstairs last five completed: 5225 5224 5223 5222 5221
15691 CLIENT: Up:0 ds:0 act:3
15692 Replay: All jobs finished
15693 Main task has work to do, go find it
15694 got dsc Stop(0)
15695 stop 0
15696 Jan 23 18:51:12.457 INFO request completed, latency_us: 311, response_code: 204, uri[8810] Got stop action so:false kr:false
15697 : /stop/cid/0, method: POST, req_id: b8ddf1d1-64d2-4378-b7f6-25710563a1f7, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
156982025-01-23T18:51:12.458ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
15699 Jan 23 18:51:12.458 INFO request completed, latency_us: 127, response_code: 200{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:51:12.458124027Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
15700 , uri: /state/cid/0, method: GET, req_id: 9454518e-55aa-4013-8ef2-a7ce3941f9ed, remote_addr: 127{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:51:12.458151564Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
15701 .0.0.1:59295, local_addr: 127.0.0.1:9998
157022025-01-23T18:51:12.458ZINFOcrucible: client task is exiting = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
15703 [8810] Exited with: ExitStatus(unix_wait_status(9))
15704 [8810][0] reports Exit
15705 Jan 23 18:51:16.460 INFO request completed, latency_us: 201, response_code: 200, uri: /state/cid/0, method: GET, req_id: 19c1d54b-fd74-42b3-acfe-e9264be38cb8, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
15706 001/300 Read block 4829 len 28672
15707 002/300 Write block 1045 len 20480 data: 7 5 6 6 5
15708 003/300 Read block 3572 len 28672
15709 004/300 Write block 4383 len 28672 data: 3 2 2 2 2 2 2
15710 005/300 Flush
15711 006/300 Write block 3413 len 4096 data: 3
15712 007/300 Write block 1848 len 40960 data: 4 4 4 4 4 3 3 3 3 2
15713 008/300 Read block 1874 len 32768
15714 009/300 Read block 471 len 36864
15715 010/300 Flush
15716 011/300 Flush
15717 012/300 Write block 1252 len 36864 data: 6 6 3 2 2 3 3 3 4
15718 013/300 Flush
15719 014/300 Read block 3551 len 4096
15720 015/300 Flush
15721 016/300 Write block 2774 len 20480 data: 4 4 6 5 3
15722 017/300 Read block 888 len 40960
15723 018/300 Write block 2912 len 8192 data: 4 3
15724 019/300 Read block 49 len 20480
15725 020/300 Read block 888 len 40960
15726 021/300 Read block 1873 len 28672
15727 022/300 Flush
15728 023/300 Read block 641 len 16384
15729 024/300 Write block 529 len 28672 data: 6 5 4 4 4 5 4
15730 025/300 Read block 2958 len 16384
15731 026/300 Flush
15732 027/300 Read block 876 len 28672
15733 028/300 Read block 4936 len 32768
15734 029/300 Write block 1771 len 28672 data: 4 5 5 6 5 5 5
15735 030/300 Read block 3334 len 20480
15736 031/300 Flush
15737 032/300 Read block 1415 len 24576
15738 033/300 Read block 3793 len 36864
15739 034/300 Read block 886 len 20480
15740 035/300 Read block 573 len 40960
15741 036/300 Read block 1217 len 36864
15742 037/300 Write block 892 len 20480 data: 2 3 4 5 7
15743 038/300 Read block 192 len 16384
15744 039/300 Read block 272 len 4096
15745 040/300 Write block 475 len 32768 data: 4 3 3 4 3 3 4 3
15746 041/300 Read block 4422 len 8192
15747 042/300 Flush
15748 043/300 Read block 4561 len 8192
15749 044/300 Write block 1104 len 36864 data: 6 4 4 3 4 3 2 3 3
15750 045/300 Write block 1441 len 20480 data: 3 3 3 3 2
15751 046/300 Read block 3613 len 36864
15752 047/300 Write block 3061 len 24576 data: 4 3 3 4 4 4
15753 048/300 Read block 3752 len 36864
15754 049/300 Write block 2575 len 12288 data: 4 3 3
15755 050/300 Read block 4085 len 16384
15756 051/300 Read block 1084 len 8192
15757 052/300 Read block 2421 len 24576
15758 053/300 Read block 1806 len 20480
15759 054/300 Read block 4309 len 32768
15760 055/300 Read block 3520 len 12288
15761 056/300 Flush
15762 057/300 Read block 1498 len 4096
15763 058/300 Read block 1353 len 36864
15764 059/300 Write block 692 len 36864 data: 3 3 3 3 3 3 2 2 2
15765 060/300 Flush
15766 061/300 Write block 3091 len 20480 data: 3 3 3 3 3
15767 062/300 Read block 352 len 20480
15768 063/300 Flush
15769 064/300 Write block 3013 len 16384 data: 6 7 7 6
15770 065/300 Write block 3016 len 16384 data: 7 5 4 4
15771 066/300 Read block 4806 len 40960
15772 067/300 Flush
15773 068/300 Write block 4170 len 36864 data: 5 6 6 5 5 7 5 5 4
15774 069/300 Write block 1397 len 4096 data: 4
15775 070/300 Write block 2177 len 8192 data: 3 3
15776 071/300 Write block 4413 len 40960 data: 2 2 2 3 3 3 3 3 4 4
15777 072/300 Write block 470 len 8192 data: 4 5
15778 073/300 Read block 3611 len 32768
15779 074/300 Read block 3864 len 16384
15780 075/300 Read block 2763 len 4096
15781 076/300 Read block 2427 len 32768
15782 077/300 Write block 2183 len 16384 data: 3 3 3 2
15783 078/300 Write block 1817 len 36864 data: 5 5 3 4 4 3 3 3 3
15784 079/300 Flush
15785 080/300 Write block 461 len 8192 data: 5 7
15786 081/300 Write block 1801 len 20480 data: 3 3 3 3 4
15787 082/300 Read block 4404 len 36864
15788 083/300 Write block 4275 len 20480 data: 3 3 3 3 5
15789 084/300 Read block 2181 len 16384
15790 085/300 Read block 1315 len 40960
15791 086/300 Read block 2044 len 24576
15792 087/300 Read block 3405 len 24576
15793 088/300 Read block 4975 len 28672
15794 089/300 Read block 4137 len 16384
15795 090/300 Read block 2899 len 8192
15796 091/300 Read block 4292 len 36864
15797 092/300 Read block 315 len 4096
15798 093/300 Read block 1153 len 36864
15799 094/300 Read block 523 len 16384
15800 095/300 Read block 4839 len 16384
15801 096/300 Read block 1366 len 32768
15802 097/300 Read block 121 len 20480
15803 098/300 Read block 1492 len 4096
15804 099/300 Read block 765 len 24576
15805 100/300 Read block 1841 len 32768
15806 101/300 Write block 575 len 4096 data: 3
15807 102/300 Flush
15808 103/300 Write block 3708 len 20480 data: 3 3 2 4 5
15809 104/300 Read block 1103 len 28672
15810 105/300 Read block 3960 len 24576
15811 106/300 Flush
15812 107/300 Write block 2137 len 28672 data: 5 5 4 5 3 3 4
15813 108/300 Write block 2482 len 36864 data: 2 2 2 2 3 3 3 3 4
15814 109/300 Flush
15815 110/300 Write block 2987 len 20480 data: 2 3 2 2 2
15816 111/300 Read block 3758 len 20480
15817 112/300 Write block 178 len 8192 data: 7 6
15818 113/300 Read block 1660 len 36864
15819 114/300 Write block 1312 len 8192 data: 5 5
15820 115/300 Write block 3531 len 4096 data: 3
15821 116/300 Write block 4702 len 4096 data: 3
15822 117/300 Read block 3688 len 8192
15823 118/300 Write block 3752 len 16384 data: 3 4 4 4
15824 119/300 Read block 2465 len 20480
15825 120/300 Read block 4358 len 20480
15826 121/300 Write block 1594 len 20480 data: 4 3 4 4 3
15827 122/300 Read block 1648 len 16384
15828 123/300 Read block 3940 len 4096
15829 124/300 Read block 2692 len 16384
15830 125/300 Write block 1383 len 12288 data: 3 5 4
15831 126/300 Write block 4911 len 4096 data: 3
15832 127/300 Read block 4782 len 28672
15833 128/300 Write block 1911 len 40960 data: 4 5 5 5 5 4 4 4 3 4
15834 129/300 Read block 3931 len 32768
15835 130/300 Read block 4599 len 36864
15836 131/300 Write block 1100 len 20480 data: 6 5 6 6 7
15837 132/300 Read block 3727 len 28672
15838 133/300 Read block 1293 len 20480
15839 134/300 Read block 636 len 16384
15840 135/300 Read block 87 len 4096
15841 136/300 Write block 4936 len 40960 data: 4 3 2 2 3 3 2 2 3 2
15842 137/300 Flush
15843 138/300 Write block 942 len 32768 data: 4 4 5 5 3 2 2 2
15844 139/300 Read block 1738 len 36864
15845 140/300 Read block 2937 len 20480
15846 141/300 Read block 2628 len 16384
15847 142/300 Read block 2923 len 8192
15848 143/300 Read block 345 len 8192
15849 144/300 Read block 1592 len 32768
15850 145/300 Read block 2473 len 20480
15851 146/300 Read block 633 len 4096
15852 147/300 Read block 3506 len 36864
15853 148/300 Read block 83 len 24576
15854 149/300 Flush
15855 150/300 Read block 1936 len 20480
15856 151/300 Read block 1119 len 40960
15857 152/300 Write block 1084 len 28672 data: 3 4 4 4 5 6 5
15858 153/300 Write block 1724 len 40960 data: 2 2 3 3 3 3 4 5 5 5
15859 154/300 Write block 3853 len 20480 data: 4 3 3 2 4
15860 155/300 Read block 1497 len 20480
15861 156/300 Flush
15862 157/300 Read block 2138 len 20480
15863 158/300 Read block 2587 len 8192
15864 159/300 Write block 1666 len 16384 data: 3 3 3 4
15865 160/300 Read block 373 len 8192
15866 161/300 Write block 2600 len 12288 data: 4 3 4
15867 162/300 Read block 229 len 8192
15868 163/300 Read block 2111 len 12288
15869 164/300 Read block 1171 len 24576
15870 165/300 Read block 899 len 4096
15871 166/300 Read block 3796 len 8192
15872 167/300 Read block 1389 len 28672
15873 168/300 Read block 2886 len 12288
15874 169/300 Flush
15875 170/300 Read block 4739 len 8192
15876 171/300 Read block 1354 len 20480
15877 172/300 Read block 4679 len 32768
15878 173/300 Read block 3352 len 36864
15879 174/300 Read block 1056 len 4096
15880 175/300 Read block 3105 len 8192
15881 176/300 Write block 2821 len 20480 data: 3 3 3 3 2
15882 177/300 Write block 3964 len 28672 data: 2 2 2 2 2 3 4
15883 178/300 Flush
15884 179/300 Write block 2737 len 8192 data: 4 4
15885 180/300 Write block 2673 len 4096 data: 7
15886 181/300 Read block 3004 len 36864
15887 182/300 Write block 4288 len 36864 data: 5 5 5 5 4 4 3 3 3
15888 183/300 Read block 2215 len 16384
15889 184/300 Read block 4432 len 16384
15890 185/300 Read block 2094 len 16384
15891 186/300 Write block 4056 len 8192 data: 3 3
15892 187/300 Read block 3124 len 20480
15893 188/300 Read block 3504 len 12288
15894 189/300 Read block 4128 len 32768
15895 190/300 Read block 932 len 20480
15896 191/300 Read block 4028 len 40960
15897 192/300 Flush
15898 193/300 Write block 1888 len 4096 data: 5
15899 194/300 Read block 3516 len 16384
15900 195/300 Read block 4322 len 4096
15901 196/300 Write block 4158 len 28672 data: 3 5 6 5 5 5 6
15902 197/300 Write block 1789 len 36864 data: 5 6 5 5 5 7 7 6 6
15903 198/300 Read block 2458 len 36864
15904 199/300 Read block 1938 len 40960
15905 200/300 Read block 3490 len 40960
15906 201/300 Flush
15907 202/300 Write block 3010 len 16384 data: 5 4 5 7
15908 203/300 Flush
15909 204/300 Write block 291 len 8192 data: 5 3
15910 205/300 Read block 1540 len 24576
15911 206/300 Write block 4076 len 32768 data: 5 5 5 5 6 8 8 6
15912 207/300 Write block 1596 len 8192 data: 5 5
15913 208/300 Write block 1229 len 28672 data: 2 2 3 3 3 5 5
15914 209/300 Write block 4161 len 32768 data: 6 6 6 7 5 4 5 5
15915 210/300 Write block 1010 len 16384 data: 4 4 6 5
15916 211/300 Read block 835 len 4096
15917 212/300 Read block 432 len 24576
15918 213/300 Write block 877 len 40960 data: 3 3 3 3 3 3 3 3 3 3
15919 214/300 Read block 3923 len 36864
15920 215/300 Read block 3077 len 28672
15921 216/300 Flush
15922 217/300 Read block 3274 len 40960
15923 218/300 Read block 2590 len 16384
15924 219/300 Read block 255 len 4096
15925 220/300 Write block 4445 len 20480 data: 3 5 6 7 8
15926 221/300 Write block 2520 len 20480 data: 5 5 5 7 6
15927 222/300 Read block 3144 len 12288
15928 223/300 Flush
15929 224/300 Read block 2831 len 20480
15930 225/300 Write block 199 len 40960 data: 2 2 3 3 3 3 3 3 3 2
15931 226/300 Flush
15932 227/300 Read block 4432 len 28672
15933 228/300 Read block 4884 len 40960
15934 229/300 Read block 2642 len 24576
15935 230/300 Read block 389 len 12288
15936 231/300 Write block 3871 len 40960 data: 3 2 2 2 3 4 4 5 5 6
15937 232/300 Write block 524 len 20480 data: 2 3 5 6 6
15938 233/300 Write block 2157 len 36864 data: 7 6 5 5 4 4 4 3 3
15939 234/300 Read block 3499 len 40960
15940 235/300 Read block 2698 len 40960
15941 236/300 Read block 2356 len 36864
15942 237/300 Write block 4978 len 16384 data: 4 4 4 4
15943 238/300 Write block 4015 len 8192 data: 3 4
15944 239/300 Write block 4397 len 4096 data: 3
15945 240/300 Write block 1057 len 8192 data: 2 2
15946 241/300 Write block 1904 len 12288 data: 5 4 3
15947 242/300 Write block 2350 len 12288 data: 2 2 2
15948 243/300 Read block 3323 len 20480
15949 244/300 Read block 3031 len 40960
15950 245/300 Read block 4003 len 8192
15951 246/300 Write block 2194 len 20480 data: 2 2 3 4 4
15952 247/300 Read block 4505 len 40960
15953 248/300 Read block 278 len 36864
15954 249/300 Flush
15955 250/300 Read block 3472 len 40960
15956 251/300 Read block 683 len 16384
15957 252/300 Write block 4336 len 24576 data: 2 2 2 2 4 5
15958 253/300 Read block 211 len 20480
15959 254/300 Read block 232 len 20480
15960 255/300 Read block 1296 len 36864
15961 256/300 Flush
15962 257/300 Read block 44 len 8192
15963 258/300 Write block 418 len 32768 data: 7 5 4 4 4 5 3 3
15964 259/300 Write block 1557 len 32768 data: 4 4 5 4 4 3 4 4
15965 260/300 Write block 2888 len 36864 data: 6 5 5 6 5 5 5 5 4
15966 261/300 Write block 339 len 12288 data: 6 6 3
15967 262/300 Read block 2989 len 32768
15968 263/300 Read block 1426 len 12288
15969 264/300 Write block 3414 len 24576 data: 3 3 3 5 6 5
15970 265/300 Read block 3176 len 40960
15971 266/300 Read block 2007 len 28672
15972 267/300 Write block 1932 len 8192 data: 4 5
15973 268/300 Read block 3149 len 4096
15974 269/300 Read block 4955 len 12288
15975 270/300 Write block 3024 len 20480 data: 4 4 4 3 3
15976 271/300 Read block 628 len 36864
15977 272/300 Flush
15978 273/300 Write block 3773 len 36864 data: 3 3 3 3 3 4 3 3 3
15979 274/300 Write block 139 len 40960 data: 4 6 5 4 4 4 4 4 4 4
15980 275/300 Read block 1519 len 32768
15981 276/300 Write block 2444 len 24576 data: 5 4 4 3 6 6
15982 277/300 Read block 496 len 24576
15983 278/300 Write block 405 len 4096 data: 7
15984 279/300 Write block 3030 len 8192 data: 5 5
15985 280/300 Flush
15986 281/300 Write block 4707 len 8192 data: 4 5
15987 282/300 Write block 4516 len 4096 data: 3
15988 283/300 Read block 4450 len 8192
15989 284/300 Write block 3644 len 8192 data: 5 4
15990 285/300 Read block 1422 len 4096
15991 286/300 Write block 2865 len 12288 data: 3 3 3
15992 287/300 Read block 4525 len 8192
15993 288/300 Read block 4669 len 24576
15994 289/300 Read block 4 len 16384
15995 290/300 Write block 2838 len 4096 data: 5
15996 291/300 Read block 4078 len 8192
15997 292/300 Write block 373 len 8192 data: 4 4
15998 293/300 Write block 1150 len 20480 data: 5 5 6 6 5
15999 294/300 Read block 358 len 28672
16000 295/300 Write block 1609 len 28672 data: 4 3 3 3 4 4 4
16001 296/300 Read block 2886 len 8192
16002 297/300 Write block 1970 len 4096 data: 5
16003 298/300 Read block 4167 len 40960
16004 299/300 Write block 2757 len 32768 data: 3 3 4 4 4 4 3 3
16005 300/300 Write block 2538 len 20480 data: 5 6 8 6 5
16006 301/300 Read block 905 len 24576
16007 Main task has work to do, go find it
16008 got dsc Start(0)
16009 start 0
16010 Jan 23 18:51:16.706 INFO request completed, latency_us: 205, response_code: 204, uri: /start/cid/0, method: POST, req_id: 107ac23e-f66c-4fc6-9a37-e8832742327e, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
16011 Replay: started 0, returned:Ok(())
16012 ----------------------------------------------------------------
16013 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16014 States: Offline Active Active
16015 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16016 5226 Acked Read 7 Sent Done Done false
16017 5227 Acked Write 5 Sent Done Done false
16018 5228 Acked Read 7 Sent Done Done false
16019 5229 Acked Write 7 Sent Done Done false
16020 5230 Acked Flush 0 Sent Done Done false
16021 5231 Acked Write 1 Sent Done Done false
16022 5232 Acked Write 10 Sent Done Done false
16023 5233 Acked Read 8 Sent Done Done false
16024 5234 Acked Read 9 Sent Done Done false
16025 5235 Acked Flush 0 Sent Done Done false
16026 5236 Acked Flush 0 Sent Done Done false
16027 5237 Acked Write 9 Sent Done Done false
16028 5238 Acked Flush 0 Sent Done Done false
16029 5239 Acked Read 1 Sent Done Done false
16030 5240 Acked Flush 0 Sent Done Done false
16031 5241 Acked Write 5 Sent Done Done false
16032 5242 Acked Read 10 Sent Done Done false
16033 5243 Acked Write 2 Sent Done Done false
16034 5244 Acked Read 5 Sent Done Done false
16035 5245 Acked Read 10 Sent Done Done false
16036 5246 Acked Read 7 Sent Done Done false
16037 5247 Acked Flush 0 Sent Done Done false
16038 5248 Acked Read 4 Sent Done Done false
16039 5249 Acked Write 7 Sent Done Done false
16040 5250 Acked Read 4 Sent Done Done false
16041 5251 Acked Flush 0 Sent Done Done false
16042 5252 Acked Read 7 Sent Done Done false
16043 5253 Acked Read 8 Sent Done Done false
16044 5254 Acked Write 7 Sent Done Done false
16045 5255 Acked Read 5 Sent Done Done false
16046 5256 Acked Flush 0 Sent Done Done false
16047 5257 Acked Read 6 Sent Done Done false
16048 5258 Acked Read 9 Sent Done Done false
16049 5259 Acked Read 5 Sent Done Done false
16050 5260 Acked Read 10 Sent Done Done false
16051 5261 Acked Read 9 Sent Done Done false
16052 5262 Acked Write 5 Sent Done Done false
16053 5263 Acked Read 4 Sent Done Done false
16054 5264 Acked Read 1 Sent Done Done false
16055 5265 Acked Write 8 Sent Done Done false
16056 5266 Acked Read 2 Sent Done Done false
16057 5267 Acked Flush 0 Sent Done Done false
16058 5268 Acked Read 2 Sent Done Done false
16059 5269 Acked Write 9 Sent Done Done false
16060 5270 Acked Write 5 Sent Done Done false
16061 5271 Acked Read 9 Sent Done Done false
16062 5272 Acked Write 6 Sent Done Done false
16063 5273 Acked Read 9 Sent Done Done false
16064 5274 Acked Write 3 Sent Done Done false
16065 5275 Acked Read 4 Sent Done Done false
16066 5276 Acked Read 2 Sent Done Done false
16067 5277 Acked Read 6 Sent Done Done false
16068 5278 Acked Read 5 Sent Done Done false
16069 5279 Acked Read 8 Sent Done Done false
16070 5280 Acked Read 3 Sent Done Done false
16071 5281 Acked Flush 0 Sent Done Done false
16072 5282 Acked Read 1 Sent Done Done false
16073 5283 Acked Read 9 Sent Done Done false
16074 5284 Acked Write 9 Sent Done Done false
16075 5285 Acked Flush 0 Sent Done Done false
16076 5286 Acked Write 5 Sent Done Done false
16077 5287 Acked Read 5 Sent Done Done false
16078 5288 Acked Flush 0 Sent Done Done false
16079 5289 Acked Write 4 Sent Done Done false
16080 5290 Acked Write 4 Sent Done Done false
16081 5291 Acked Read 10 Sent Done Done false
16082 5292 Acked Flush 0 Sent Done Done false
16083 5293 Acked Write 9 Sent Done Done false
16084 5294 Acked Write 1 Sent Done Done false
16085 5295 Acked Write 2 Sent Done Done false
16086 5296 Acked Write 10 Sent Done Done false
16087 5297 Acked Write 2 Sent Done Done false
16088 5298 Acked Read 8 Sent Done Done false
16089 5299 Acked Read 4 Sent Done Done false
16090 5300 Acked Read 1 Sent Done Done false
16091 5301 Acked Read 8 Sent Done Done false
16092 5302 Acked Write 4 Sent Done Done false
16093 5303 Acked Write 9 Sent Done Done false
16094 5304 Acked Flush 0 Sent Done Done false
16095 5305 Acked Write 2 Sent Done Done false
16096 5306 Acked Write 5 Sent Done Done false
16097 5307 Acked Read 9 Sent Done Done false
16098 5308 Acked Write 5 Sent Done Done false
16099 5309 Acked Read 4 Sent Done Done false
16100 5310 Acked Read 10 Sent Done Done false
16101 5311 Acked Read 6 Sent Done Done false
16102 5312 Acked Read 6 Sent Done Done false
16103 5313 Acked Read 7 Sent Done Done false
16104 5314 Acked Read 4 Sent Done Done false
16105 5315 Acked Read 2 Sent Done Done false
16106 5316 Acked Read 9 Sent Done Done false
16107 5317 Acked Read 1 Sent Done Done false
16108 5318 Acked Read 9 Sent Done Done false
16109 5319 Acked Read 4 Sent Done Done false
16110 5320 Acked Read 4 Sent Done Done false
16111 5321 Acked Read 8 Sent Done Done false
16112 5322 Acked Read 5 Sent Done Done false
16113 5323 Acked Read 1 Sent Done Done false
16114 5324 Acked Read 6 Sent Done Done false
16115 5325 Acked Read 8 Sent Done Done false
16116 5326 Acked Write 1 Sent Done Done false
16117 5327 Acked Flush 0 Sent Done Done false
16118 5328 Acked Write 5 Sent Done Done false
16119 5329 Acked Read 7 Sent Done Done false
16120 5330 Acked Read 6 Sent Done Done false
16121 5331 Acked Flush 0 Sent Done Done false
16122 5332 Acked Write 7 Sent Done Done false
16123 5333 Acked Write 9 Sent Done Done false
16124 5334 Acked Flush 0 Sent Done Done false
16125 5335 Acked Write 5 Sent Done Done false
16126 5336 Acked Read 5 Sent Done Done false
16127 5337 Acked Write 2 Sent Done Done false
16128 5338 Acked Read 9 Sent Done Done false
16129 5339 Acked Write 2 Sent Done Done false
16130 5340 Acked Write 1 Sent Done Done false
16131 5341 Acked Write 1 Sent Done Done false
16132 5342 Acked Read 2 Sent Done Done false
16133 5343 Acked Write 4 Sent Done Done false
16134 5344 Acked Read 5 Sent Done Done false
16135 5345 Acked Read 5 Sent Done Done false
16136 5346 Acked Write 5 Sent Done Done false
16137 5347 Acked Read 4 Sent Done Done false
16138 5348 Acked Read 1 Sent Done Done false
16139 5349 Acked Read 4 Sent Done Done false
16140 5350 Acked Write 3 Sent Done Done false
16141 5351 Acked Write 1 Sent Done Done false
16142 5352 Acked Read 7 Sent Done Done false
16143 5353 Acked Write 10 Sent Done Done false
16144 5354 Acked Read 8 Sent Done Done false
16145 5355 Acked Read 9 Sent Done Done false
16146 5356 Acked Write 5 Sent Done Done false
16147 5357 Acked Read 7 Sent Done Done false
16148 5358 Acked Read 5 Sent Done Done false
16149 5359 Acked Read 4 Sent Done Done false
16150 5360 Acked Read 1 Sent Done Done false
16151 5361 Acked Write 10 Sent Done Done false
16152 5362 Acked Flush 0 Sent Done Done false
16153 5363 Acked Write 8 Sent Done Done false
16154 5364 Acked Read 9 Sent Done Done false
16155 5365 Acked Read 5 Sent Done Done false
16156 5366 Acked Read 4 Sent Done Done false
16157 5367 Acked Read 2 Sent Done Done false
16158 5368 Acked Read 2 Sent Done Done false
16159 5369 Acked Read 8 Sent Done Done false
16160 5370 Acked Read 5 Sent Done Done false
16161 5371 Acked Read 1 Sent Done Done false
16162 5372 Acked Read 9 Sent Done Done false
16163 5373 Acked Read 6 Sent Done Done false
16164 5374 Acked Flush 0 Sent Done Done false
16165 5375 Acked Read 5 Sent Done Done false
16166 5376 Acked Read 10 Sent Done Done false
16167 5377 Acked Write 7 Sent Done Done false
16168 5378 Acked Write 10 Sent Done Done false
16169 5379 Acked Write 5 Sent Done Done false
16170 5380 Acked Read 5 Sent Done Done false
16171 5381 Acked Flush 0 Sent Done Done false
16172 5382 Acked Read 5 Sent Done Done false
16173 5383 Acked Read 2 Sent Done Done false
16174 5384 Acked Write 4 Sent Done Done false
16175 5385 Acked Read 2 Sent Done Done false
16176 5386 Acked Write 3 Sent Done Done false
16177 5387 Acked Read 2 Sent Done Done false
16178 5388 Acked Read 3 Sent Done Done false
16179 5389 Acked Read 6 Sent Done Done false
16180 5390 Acked Read 1 Sent Done Done false
16181 5391 Acked Read 2 Sent Done Done false
16182 5392 Acked Read 7 Sent Done Done false
16183 5393 Acked Read 3 Sent Done Done false
16184 5394 Acked Flush 0 Sent Done Done false
16185 5395 Acked Read 2 Sent Done Done false
16186 5396 Acked Read 5 Sent Done Done false
16187 5397 Acked Read 8 Sent Done Done false
16188 5398 Acked Read 9 Sent Done Done false
16189 5399 Acked Read 1 Sent Done Done false
16190 5400 Acked Read 2 Sent Done Done false
16191 5401 Acked Write 5 Sent Done Done false
16192 5402 Acked Write 7 Sent Done Done false
16193 5403 Acked Flush 0 Sent Done Done false
16194 5404 Acked Write 2 Sent Done Done false
16195 5405 Acked Write 1 Sent Done Done false
16196 5406 Acked Read 9 Sent Done Done false
16197 5407 Acked Write 9 Sent Done Done false
16198 5408 Acked Read 4 Sent Done Done false
16199 5409 Acked Read 4 Sent Done Done false
16200 5410 Acked Read 4 Sent Done Done false
16201 5411 Acked Write 2 Sent Done Done false
16202 5412 Acked Read 5 Sent Done Done false
16203 5413 Acked Read 3 Sent Done Done false
16204 5414 Acked Read 8 Sent Done Done false
16205 5415 Acked Read 5 Sent Done Done false
16206 5416 Acked Read 10 Sent Done Done false
16207 5417 Acked Flush 0 Sent Done Done false
16208 5418 Acked Write 1 Sent Done Done false
16209 5419 Acked Read 4 Sent Done Done false
16210 5420 Acked Read 1 Sent Done Done false
16211 5421 Acked Write 7 Sent Done Done false
16212 5422 Acked Write 9 Sent Done Done false
16213 5423 Acked Read 9 Sent Done Done false
16214 5424 Acked Read 10 Sent Done Done false
16215 5425 Acked Read 10 Sent Done Done false
16216 5426 Acked Flush 0 Sent Done Done false
16217 5427 Acked Write 4 Sent Done Done false
16218 5428 Acked Flush 0 Sent Done Done false
16219 5429 Acked Write 2 Sent Done Done false
16220 5430 Acked Read 6 Sent Done Done false
16221 5431 Acked Write 8 Sent Done Done false
16222 5432 Acked Write 2 Sent Done Done false
16223 5433 Acked Write 7 Sent Done Done false
16224 5434 Acked Write 8 Sent Done Done false
16225 5435 Acked Write 4 Sent Done Done false
16226 5436 Acked Read 1 Sent Done Done false
16227 5437 Acked Read 6 Sent Done Done false
16228 5438 Acked Write 10 Sent Done Done false
16229 5439 Acked Read 9 Sent Done Done false
16230 5440 Acked Read 7 Sent Done Done false
16231 5441 Acked Flush 0 Sent Done Done false
16232 5442 Acked Read 10 Sent Done Done false
16233 5443 Acked Read 4 Sent Done Done false
16234 5444 Acked Read 1 Sent Done Done false
16235 5445 Acked Write 5 Sent Done Done false
16236 5446 Acked Write 5 Sent Done Done false
16237 5447 Acked Read 3 Sent Done Done false
16238 5448 Acked Flush 0 Sent Done Done false
16239 5449 Acked Read 5 Sent Done Done false
16240 5450 Acked Write 10 Sent Done Done false
16241 5451 Acked Flush 0 Sent Done Done false
16242 5452 Acked Read 7 Sent Done Done false
16243 5453 Acked Read 10 Sent Done Done false
16244 5454 Acked Read 6 Sent Done Done false
16245 5455 Acked Read 3 Sent Done Done false
16246 5456 Acked Write 10 Sent Done Done false
16247 5457 Acked Write 5 Sent Done Done false
16248 5458 Acked Write 9 Sent Done Done false
16249 5459 Acked Read 10 Sent Done Done false
16250 5460 Acked Read 10 Sent Done Done false
16251 5461 Acked Read 9 Sent Done Done false
16252 5462 Acked Write 4 Sent Done Done false
16253 5463 Acked Write 2 Sent Done Done false
16254 5464 Acked Write 1 Sent Done Done false
16255 5465 Acked Write 2 Sent Done Done false
16256 5466 Acked Write 3 Sent Done Done false
16257 5467 Acked Write 3 Sent Done Done false
16258 5468 Acked Read 5 Sent Done Done false
16259 5469 Acked Read 10 Sent Done Done false
16260 5470 Acked Read 2 Sent Done Done false
16261 5471 Acked Write 5 Sent Done Done false
16262 5472 Acked Read 10 Sent Done Done false
16263 5473 Acked Read 9 Sent Done Done false
16264 5474 Acked Flush 0 Sent Done Done false
16265 5475 Acked Read 10 Sent Done Done false
16266 5476 Acked Read 4 Sent Done Done false
16267 5477 Acked Write 6 Sent Done Done false
16268 5478 Acked Read 5 Sent Done Done false
16269 5479 Acked Read 5 Sent Done Done false
16270 5480 Acked Read 9 Sent Done Done false
16271 5481 Acked Flush 0 Sent Done Done false
16272 5482 Acked Read 2 Sent Done Done false
16273 5483 Acked Write 8 Sent Done Done false
16274 5484 Acked Write 8 Sent Done Done false
16275 5485 Acked Write 9 Sent Done Done false
16276 5486 Acked Write 3 Sent Done Done false
16277 5487 Acked Read 8 Sent Done Done false
16278 5488 Acked Read 3 Sent Done Done false
16279 5489 Acked Write 6 Sent Done Done false
16280 5490 Acked Read 10 Sent Done Done false
16281 5491 Acked Read 7 Sent Done Done false
16282 5492 Acked Write 2 Sent Done Done false
16283 5493 Acked Read 1 Sent Done Done false
16284 5494 Acked Read 3 Sent Done Done false
16285 5495 Acked Write 5 Sent Done Done false
16286 5496 Acked Read 9 Sent Done Done false
16287 5497 Acked Flush 0 Sent Done Done false
16288 5498 Acked Write 9 Sent Done Done false
16289 5499 Acked Write 10 Sent Done Done false
16290 5500 Acked Read 8 Sent Done Done false
16291 5501 Acked Write 6 Sent Done Done false
16292 5502 Acked Read 6 Sent Done Done false
16293 5503 Acked Write 1 Sent Done Done false
16294 5504 Acked Write 2 Sent Done Done false
16295 5505 Acked Flush 0 Sent Done Done false
16296 5506 Acked Write 2 Sent Done Done false
16297 5507 Acked Write 1 Sent Done Done false
16298 5508 Acked Read 2 Sent Done Done false
16299 5509 Acked Write 2 Sent Done Done false
16300 5510 Acked Read 1 Sent Done Done false
16301 5511 Acked Write 3 Sent Done Done false
16302 5512 Acked Read 2 Sent Done Done false
16303 5513 Acked Read 6 Sent Done Done false
16304 5514 Acked Read 4 Sent Done Done false
16305 5515 Acked Write 1 Sent Done Done false
16306 5516 Acked Read 2 Sent Done Done false
16307 5517 Acked Write 2 Sent Done Done false
16308 5518 Acked Write 5 Sent Done Done false
16309 5519 Acked Read 7 Sent Done Done false
16310 5520 Acked Write 7 Sent Done Done false
16311 5521 Acked Read 2 Sent Done Done false
16312 5522 Acked Write 1 Sent Done Done false
16313 5523 Acked Read 10 Sent Done Done false
16314 5524 Acked Write 8 Sent Done Done false
16315 5525 Acked Write 5 Sent Done Done false
16316 5526 Acked Read 6 Sent Done Done false
16317 STATES DS:0 DS:1 DS:2 TOTAL
16318 Sent 301 0 0 301
16319 Done 0 301 301 602
16320 Skipped 0 0 0 0
16321 Error 0 0 0 0
16322 Last Flush: Some(JobId(5225)) Some(JobId(5505)) Some(JobId(5505))
16323 Downstairs last five completed: 5225 5224 5223 5222 5221
16324 Upstairs last five completed: 5526 5525 5524 5523 5522
16325 CLIENT: Up:0 ds:301 act:2
16326 [8810] got start action
16327 [8810] I am going to restart
16328 Starting downstairs at port 8810
16329 Make output file at "/tmp/dsc/downstairs-8810.txt"
16330 [8810][0] reports Starting
16331 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1215)
16332 [8810][0] reports Running
16333 ----------------------------------------------------------------
16334 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
16335 States: Offline Active Active
16336 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16337 5226 Acked Read 7 Sent Done Done false
16338 5227 Acked Write 5 Sent Done Done false
16339 5228 Acked Read 7 Sent Done Done false
16340 5229 Acked Write 7 Sent Done Done false
16341 5230 Acked Flush 0 Sent Done Done false
16342 5231 Acked Write 1 Sent Done Done false
16343 5232 Acked Write 10 Sent Done Done false
16344 5233 Acked Read 8 Sent Done Done false
16345 5234 Acked Read 9 Sent Done Done false
16346 5235 Acked Flush 0 Sent Done Done false
16347 5236 Acked Flush 0 Sent Done Done false
16348 5237 Acked Write 9 Sent Done Done false
16349 5238 Acked Flush 0 Sent Done Done false
16350 5239 Acked Read 1 Sent Done Done false
16351 5240 Acked Flush 0 Sent Done Done false
16352 5241 Acked Write 5 Sent Done Done false
16353 5242 Acked Read 10 Sent Done Done false
16354 5243 Acked Write 2 Sent Done Done false
16355 5244 Acked Read 5 Sent Done Done false
16356 5245 Acked Read 10 Sent Done Done false
16357 5246 Acked Read 7 Sent Done Done false
16358 5247 Acked Flush 0 Sent Done Done false
16359 5248 Acked Read 4 Sent Done Done false
16360 5249 Acked Write 7 Sent Done Done false
16361 5250 Acked Read 4 Sent Done Done false
16362 5251 Acked Flush 0 Sent Done Done false
16363 5252 Acked Read 7 Sent Done Done false
16364 5253 Acked Read 8 Sent Done Done false
16365 5254 Acked Write 7 Sent Done Done false
16366 5255 Acked Read 5 Sent Done Done false
16367 5256 Acked Flush 0 Sent Done Done false
16368 5257 Acked Read 6 Sent Done Done false
16369 5258 Acked Read 9 Sent Done Done false
16370 5259 Acked Read 5 Sent Done Done false
16371 5260 Acked Read 10 Sent Done Done false
16372 5261 Acked Read 9 Sent Done Done false
16373 5262 Acked Write 5 Sent Done Done false
16374 5263 Acked Read 4 Sent Done Done false
16375 5264 Acked Read 1 Sent Done Done false
16376 5265 Acked Write 8 Sent Done Done false
16377 5266 Acked Read 2 Sent Done Done false
16378 5267 Acked Flush 0 Sent Done Done false
16379 5268 Acked Read 2 Sent Done Done false
16380 5269 Acked Write 9 Sent Done Done false
16381 5270 Acked Write 5 Sent Done Done false
16382 5271 Acked Read 9 Sent Done Done false
16383 5272 Acked Write 6 Sent Done Done false
16384 5273 Acked Read 9 Sent Done Done false
16385 5274 Acked Write 3 Sent Done Done false
16386 5275 Acked Read 4 Sent Done Done false
16387 5276 Acked Read 2 Sent Done Done false
16388 5277 Acked Read 6 Sent Done Done false
16389 5278 Acked Read 5 Sent Done Done false
16390 5279 Acked Read 8 Sent Done Done false
16391 5280 Acked Read 3 Sent Done Done false
16392 5281 Acked Flush 0 Sent Done Done false
16393 5282 Acked Read 1 Sent Done Done false
16394 5283 Acked Read 9 Sent Done Done false
16395 5284 Acked Write 9 Sent Done Done false
16396 5285 Acked Flush 0 Sent Done Done false
16397 5286 Acked Write 5 Sent Done Done false
16398 5287 Acked Read 5 Sent Done Done false
16399 5288 Acked Flush 0 Sent Done Done false
16400 5289 Acked Write 4 Sent Done Done false
16401 5290 Acked Write 4 Sent Done Done false
16402 5291 Acked Read 10 Sent Done Done false
16403 5292 Acked Flush 0 Sent Done Done false
16404 5293 Acked Write 9 Sent Done Done false
16405 5294 Acked Write 1 Sent Done Done false
16406 5295 Acked Write 2 Sent Done Done false
16407 5296 Acked Write 10 Sent Done Done false
16408 5297 Acked Write 2 Sent Done Done false
16409 5298 Acked Read 8 Sent Done Done false
16410 5299 Acked Read 4 Sent Done Done false
16411 5300 Acked Read 1 Sent Done Done false
16412 5301 Acked Read 8 Sent Done Done false
16413 5302 Acked Write 4 Sent Done Done false
16414 5303 Acked Write 9 Sent Done Done false
16415 5304 Acked Flush 0 Sent Done Done false
16416 5305 Acked Write 2 Sent Done Done false
16417 5306 Acked Write 5 Sent Done Done false
16418 5307 Acked Read 9 Sent Done Done false
16419 5308 Acked Write 5 Sent Done Done false
16420 5309 Acked Read 4 Sent Done Done false
16421 5310 Acked Read 10 Sent Done Done false
16422 5311 Acked Read 6 Sent Done Done false
16423 5312 Acked Read 6 Sent Done Done false
16424 5313 Acked Read 7 Sent Done Done false
16425 5314 Acked Read 4 Sent Done Done false
16426 5315 Acked Read 2 Sent Done Done false
16427 5316 Acked Read 9 Sent Done Done false
16428 5317 Acked Read 1 Sent Done Done false
16429 5318 Acked Read 9 Sent Done Done false
16430 5319 Acked Read 4 Sent Done Done false
16431 5320 Acked Read 4 Sent Done Done false
16432 5321 Acked Read 8 Sent Done Done false
16433 5322 Acked Read 5 Sent Done Done false
16434 5323 Acked Read 1 Sent Done Done false
16435 5324 Acked Read 6 Sent Done Done false
16436 5325 Acked Read 8 Sent Done Done false
16437 5326 Acked Write 1 Sent Done Done false
16438 5327 Acked Flush 0 Sent Done Done false
16439 5328 Acked Write 5 Sent Done Done false
16440 5329 Acked Read 7 Sent Done Done false
16441 5330 Acked Read 6 Sent Done Done false
16442 5331 Acked Flush 0 Sent Done Done false
16443 5332 Acked Write 7 Sent Done Done false
16444 5333 Acked Write 9 Sent Done Done false
16445 5334 Acked Flush 0 Sent Done Done false
16446 5335 Acked Write 5 Sent Done Done false
16447 5336 Acked Read 5 Sent Done Done false
16448 5337 Acked Write 2 Sent Done Done false
16449 5338 Acked Read 9 Sent Done Done false
16450 5339 Acked Write 2 Sent Done Done false
16451 5340 Acked Write 1 Sent Done Done false
16452 5341 Acked Write 1 Sent Done Done false
16453 5342 Acked Read 2 Sent Done Done false
16454 5343 Acked Write 4 Sent Done Done false
16455 5344 Acked Read 5 Sent Done Done false
16456 5345 Acked Read 5 Sent Done Done false
16457 5346 Acked Write 5 Sent Done Done false
16458 5347 Acked Read 4 Sent Done Done false
16459 5348 Acked Read 1 Sent Done Done false
16460 5349 Acked Read 4 Sent Done Done false
16461 5350 Acked Write 3 Sent Done Done false
16462 5351 Acked Write 1 Sent Done Done false
16463 5352 Acked Read 7 Sent Done Done false
16464 5353 Acked Write 10 Sent Done Done false
16465 5354 Acked Read 8 Sent Done Done false
16466 5355 Acked Read 9 Sent Done Done false
16467 5356 Acked Write 5 Sent Done Done false
16468 5357 Acked Read 7 Sent Done Done false
16469 5358 Acked Read 5 Sent Done Done false
16470 5359 Acked Read 4 Sent Done Done false
16471 5360 Acked Read 1 Sent Done Done false
16472 5361 Acked Write 10 Sent Done Done false
16473 5362 Acked Flush 0 Sent Done Done false
16474 5363 Acked Write 8 Sent Done Done false
16475 5364 Acked Read 9 Sent Done Done false
16476 5365 Acked Read 5 Sent Done Done false
16477 5366 Acked Read 4 Sent Done Done false
16478 5367 Acked Read 2 Sent Done Done false
16479 5368 Acked Read 2 Sent Done Done false
16480 5369 Acked Read 8 Sent Done Done false
16481 5370 Acked Read 5 Sent Done Done false
16482 5371 Acked Read 1 Sent Done Done false
16483 5372 Acked Read 9 Sent Done Done false
16484 5373 Acked Read 6 Sent Done Done false
16485 5374 Acked Flush 0 Sent Done Done false
16486 5375 Acked Read 5 Sent Done Done false
16487 5376 Acked Read 10 Sent Done Done false
16488 5377 Acked Write 7 Sent Done Done false
16489 5378 Acked Write 10 Sent Done Done false
16490 5379 Acked Write 5 Sent Done Done false
16491 5380 Acked Read 5 Sent Done Done false
16492 5381 Acked Flush 0 Sent Done Done false
16493 5382 Acked Read 5 Sent Done Done false
16494 5383 Acked Read 2 Sent Done Done false
16495 5384 Acked Write 4 Sent Done Done false
16496 5385 Acked Read 2 Sent Done Done false
16497 5386 Acked Write 3 Sent Done Done false
16498 5387 Acked Read 2 Sent Done Done false
16499 5388 Acked Read 3 Sent Done Done false
16500 5389 Acked Read 6 Sent Done Done false
16501 5390 Acked Read 1 Sent Done Done false
16502 5391 Acked Read 2 Sent Done Done false
16503 5392 Acked Read 7 Sent Done Done false
16504 5393 Acked Read 3 Sent Done Done false
16505 5394 Acked Flush 0 Sent Done Done false
16506 5395 Acked Read 2 Sent Done Done false
16507 5396 Acked Read 5 Sent Done Done false
16508 5397 Acked Read 8 Sent Done Done false
16509 5398 Acked Read 9 Sent Done Done false
16510 5399 Acked Read 1 Sent Done Done false
16511 5400 Acked Read 2 Sent Done Done false
16512 5401 Acked Write 5 Sent Done Done false
16513 5402 Acked Write 7 Sent Done Done false
16514 5403 Acked Flush 0 Sent Done Done false
16515 5404 Acked Write 2 Sent Done Done false
16516 5405 Acked Write 1 Sent Done Done false
16517 5406 Acked Read 9 Sent Done Done false
16518 5407 Acked Write 9 Sent Done Done false
16519 5408 Acked Read 4 Sent Done Done false
16520 5409 Acked Read 4 Sent Done Done false
16521 5410 Acked Read 4 Sent Done Done false
16522 5411 Acked Write 2 Sent Done Done false
16523 5412 Acked Read 5 Sent Done Done false
16524 5413 Acked Read 3 Sent Done Done false
16525 5414 Acked Read 8 Sent Done Done false
16526 5415 Acked Read 5 Sent Done Done false
16527 5416 Acked Read 10 Sent Done Done false
16528 5417 Acked Flush 0 Sent Done Done false
16529 5418 Acked Write 1 Sent Done Done false
16530 5419 Acked Read 4 Sent Done Done false
16531 5420 Acked Read 1 Sent Done Done false
16532 5421 Acked Write 7 Sent Done Done false
16533 5422 Acked Write 9 Sent Done Done false
16534 5423 Acked Read 9 Sent Done Done false
16535 5424 Acked Read 10 Sent Done Done false
16536 5425 Acked Read 10 Sent Done Done false
16537 5426 Acked Flush 0 Sent Done Done false
16538 5427 Acked Write 4 Sent Done Done false
16539 5428 Acked Flush 0 Sent Done Done false
16540 5429 Acked Write 2 Sent Done Done false
16541 5430 Acked Read 6 Sent Done Done false
16542 5431 Acked Write 8 Sent Done Done false
16543 5432 Acked Write 2 Sent Done Done false
16544 5433 Acked Write 7 Sent Done Done false
16545 5434 Acked Write 8 Sent Done Done false
16546 5435 Acked Write 4 Sent Done Done false
16547 5436 Acked Read 1 Sent Done Done false
16548 5437 Acked Read 6 Sent Done Done false
16549 5438 Acked Write 10 Sent Done Done false
16550 5439 Acked Read 9 Sent Done Done false
16551 5440 Acked Read 7 Sent Done Done false
16552 5441 Acked Flush 0 Sent Done Done false
16553 5442 Acked Read 10 Sent Done Done false
16554 5443 Acked Read 4 Sent Done Done false
16555 5444 Acked Read 1 Sent Done Done false
16556 5445 Acked Write 5 Sent Done Done false
16557 5446 Acked Write 5 Sent Done Done false
16558 5447 Acked Read 3 Sent Done Done false
16559 5448 Acked Flush 0 Sent Done Done false
16560 5449 Acked Read 5 Sent Done Done false
16561 5450 Acked Write 10 Sent Done Done false
16562 5451 Acked Flush 0 Sent Done Done false
16563 5452 Acked Read 7 Sent Done Done false
16564 5453 Acked Read 10 Sent Done Done false
16565 5454 Acked Read 6 Sent Done Done false
16566 5455 Acked Read 3 Sent Done Done false
16567 5456 Acked Write 10 Sent Done Done false
16568 5457 Acked Write 5 Sent Done Done false
16569 5458 Acked Write 9 Sent Done Done false
16570 5459 Acked Read 10 Sent Done Done false
16571 5460 Acked Read 10 Sent Done Done false
16572 5461 Acked Read 9 Sent Done Done false
16573 5462 Acked Write 4 Sent Done Done false
16574 5463 Acked Write 2 Sent Done Done false
16575 5464 Acked Write 1 Sent Done Done false
16576 5465 Acked Write 2 Sent Done Done false
16577 5466 Acked Write 3 Sent Done Done false
16578 5467 Acked Write 3 Sent Done Done false
16579 5468 Acked Read 5 Sent Done Done false
16580 5469 Acked Read 10 Sent Done Done false
16581 5470 Acked Read 2 Sent Done Done false
16582 5471 Acked Write 5 Sent Done Done false
16583 5472 Acked Read 10 Sent Done Done false
16584 5473 Acked Read 9 Sent Done Done false
16585 5474 Acked Flush 0 Sent Done Done false
16586 5475 Acked Read 10 Sent Done Done false
16587 5476 Acked Read 4 Sent Done Done false
16588 5477 Acked Write 6 Sent Done Done false
16589 5478 Acked Read 5 Sent Done Done false
16590 5479 Acked Read 5 Sent Done Done false
16591 5480 Acked Read 9 Sent Done Done false
16592 5481 Acked Flush 0 Sent Done Done false
16593 5482 Acked Read 2 Sent Done Done false
16594 5483 Acked Write 8 Sent Done Done false
16595 5484 Acked Write 8 Sent Done Done false
16596 5485 Acked Write 9 Sent Done Done false
16597 5486 Acked Write 3 Sent Done Done false
16598 5487 Acked Read 8 Sent Done Done false
16599 5488 Acked Read 3 Sent Done Done false
16600 5489 Acked Write 6 Sent Done Done false
16601 5490 Acked Read 10 Sent Done Done false
16602 5491 Acked Read 7 Sent Done Done false
16603 5492 Acked Write 2 Sent Done Done false
16604 5493 Acked Read 1 Sent Done Done false
16605 5494 Acked Read 3 Sent Done Done false
16606 5495 Acked Write 5 Sent Done Done false
16607 5496 Acked Read 9 Sent Done Done false
16608 5497 Acked Flush 0 Sent Done Done false
16609 5498 Acked Write 9 Sent Done Done false
16610 5499 Acked Write 10 Sent Done Done false
16611 5500 Acked Read 8 Sent Done Done false
16612 5501 Acked Write 6 Sent Done Done false
16613 5502 Acked Read 6 Sent Done Done false
16614 5503 Acked Write 1 Sent Done Done false
16615 5504 Acked Write 2 Sent Done Done false
16616 5505 Acked Flush 0 Sent Done Done false
16617 5506 Acked Write 2 Sent Done Done false
16618 5507 Acked Write 1 Sent Done Done false
16619 5508 Acked Read 2 Sent Done Done false
16620 5509 Acked Write 2 Sent Done Done false
16621 5510 Acked Read 1 Sent Done Done false
16622 5511 Acked Write 3 Sent Done Done false
16623 5512 Acked Read 2 Sent Done Done false
16624 5513 Acked Read 6 Sent Done Done false
16625 5514 Acked Read 4 Sent Done Done false
16626 5515 Acked Write 1 Sent Done Done false
16627 5516 Acked Read 2 Sent Done Done false
16628 5517 Acked Write 2 Sent Done Done false
16629 5518 Acked Write 5 Sent Done Done false
16630 5519 Acked Read 7 Sent Done Done false
16631 5520 Acked Write 7 Sent Done Done false
16632 5521 Acked Read 2 Sent Done Done false
16633 5522 Acked Write 1 Sent Done Done false
16634 5523 Acked Read 10 Sent Done Done false
16635 5524 Acked Write 8 Sent Done Done false
16636 5525 Acked Write 5 Sent Done Done false
16637 5526 Acked Read 6 Sent Done Done false
16638 5527 Acked Flush 0 Sent Done Done false
16639 STATES DS:0 DS:1 DS:2 TOTAL
16640 Sent 302 0 0 302
16641 Done 0 302 302 604
16642 Skipped 0 0 0 0
16643 Error 0 0 0 0
16644 Last Flush: Some(JobId(5225)) Some(JobId(5527)) Some(JobId(5527))
16645 Downstairs last five completed: 5225 5224 5223 5222 5221
16646 Upstairs last five completed: 5527 5526 5525 5524 5523
16647 CLIENT: Up:0 ds:302 act:2
166482025-01-23T18:51:22.460ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
166492025-01-23T18:51:22.460ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64095) = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
166502025-01-23T18:51:22.461ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
166512025-01-23T18:51:22.461ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
166522025-01-23T18:51:22.461ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
166532025-01-23T18:51:22.461ZINFOcrucible: send last flush ID to this DS: Some(JobId(5225)) = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
166542025-01-23T18:51:22.461ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
166552025-01-23T18:51:22.461ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(5225)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
166562025-01-23T18:51:22.461ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(5225)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
16657 ----------------------------------------------------------------
16658 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16659 Downstairs last five completed: 5527 5526 5525 5524 5523
16660 Upstairs last five completed: 5527 5526 5525 5524 5523
16661 CLIENT: Up:0 ds:0 act:3
16662 Replay: All jobs finished
16663 Main task has work to do, go find it
16664 got dsc Stop(2)
16665 stop 2
16666 [8830] Got stop action so:false kr:false
16667 Jan 23 18:51:24.712 INFO request completed, latency_us: 362, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 0b36894a-5d2b-41f1-b9ed-90a10eb5d0ea, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
166682025-01-23T18:51:24.712ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
166692025-01-23T18:51:24.712ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
166702025-01-23T18:51:24.712ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
16671 Jan 23 18:51:24.713 INFO request completed, latency_us: 117, response_code: 200, uri: /state/cid/2, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T18:51:24.713011006Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
16672 method: GET, req_id: 8bceb9d5-d582-4955-85a4-8271453822e4, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
16673 [8830] Exited with: ExitStatus(unix_wait_status(9))
16674 [8830][2] reports Exit
16675 Jan 23 18:51:28.715 INFO request completed, latency_us: 179, response_code: 200, uri: /state/cid/2, method: GET, req_id: f0ba7f39-9911-45ff-b552-c4c470ad16b4, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
16676 001/300 Flush
16677 002/300 Write block 3795 len 12288 data: 5 5 4
16678 003/300 Read block 2604 len 12288
16679 004/300 Write block 2927 len 12288 data: 5 5 5
16680 005/300 Write block 1338 len 4096 data: 3
16681 006/300 Write block 4515 len 4096 data: 3
16682 007/300 Read block 4235 len 32768
16683 008/300 Write block 3240 len 32768 data: 5 5 4 4 4 6 5 4
16684 009/300 Read block 2430 len 24576
16685 010/300 Write block 552 len 36864 data: 6 5 4 3 3 4 5 5 4
16686 011/300 Write block 1762 len 20480 data: 3 3 3 4 3
16687 012/300 Write block 849 len 8192 data: 3 2
16688 013/300 Write block 587 len 4096 data: 2
16689 014/300 Write block 3924 len 28672 data: 5 6 7 7 5 5 3
16690 015/300 Write block 2635 len 16384 data: 4 3 3 3
16691 016/300 Write block 984 len 12288 data: 4 5 5
16692 017/300 Read block 3786 len 32768
16693 018/300 Flush
16694 019/300 Read block 2928 len 16384
16695 020/300 Read block 988 len 4096
16696 021/300 Write block 1576 len 36864 data: 2 2 2 2 2 2 2 3 3
16697 022/300 Write block 2698 len 12288 data: 3 4 4
16698 023/300 Write block 285 len 40960 data: 3 3 2 2 4 4 6 4 2 2
16699 024/300 Read block 3645 len 24576
16700 025/300 Write block 3453 len 8192 data: 2 2
16701 026/300 Read block 1006 len 4096
16702 027/300 Write block 4211 len 28672 data: 2 2 2 2 3 4 5
16703 028/300 Write block 2670 len 8192 data: 6 7
16704 029/300 Write block 109 len 40960 data: 6 4 5 6 7 8 8 6 5 6
16705 030/300 Write block 3279 len 8192 data: 3 4
16706 031/300 Write block 1311 len 16384 data: 4 6 6 5
16707 032/300 Read block 2600 len 16384
16708 033/300 Write block 1630 len 40960 data: 4 4 4 4 4 5 4 4 4 4
16709 034/300 Read block 4371 len 36864
16710 035/300 Write block 3715 len 8192 data: 5 6
16711 036/300 Write block 323 len 40960 data: 5 4 3 3 2 2 2 2 2 3
16712 037/300 Write block 2453 len 20480 data: 3 3 3 5 4
16713 038/300 Read block 1252 len 16384
16714 039/300 Read block 3426 len 32768
16715 040/300 Read block 2870 len 8192
16716 041/300 Write block 3549 len 32768 data: 4 3 2 2 2 3 4 5
16717 042/300 Write block 1488 len 8192 data: 4 5
16718 043/300 Read block 1869 len 28672
16719 044/300 Flush
16720 045/300 Read block 644 len 4096
16721 046/300 Write block 1044 len 32768 data: 5 8 6 7 7 6 3 3
16722 047/300 Write block 838 len 40960 data: 3 2 2 2 3 3 3 3 3 3
16723 048/300 Read block 234 len 20480
16724 049/300 Read block 3603 len 40960
16725 050/300 Write block 3460 len 8192 data: 5 5
16726 051/300 Write block 4336 len 20480 data: 3 3 3 3 5
16727 052/300 Read block 1603 len 28672
16728 053/300 Flush
16729 054/300 Read block 2152 len 36864
16730 055/300 Read block 1685 len 24576
16731 056/300 Read block 3009 len 24576
16732 057/300 Write block 4431 len 20480 data: 5 4 4 2 3
16733 058/300 Write block 289 len 8192 data: 5 5
16734 059/300 Flush
16735 060/300 Write block 3297 len 40960 data: 7 5 4 3 3 3 3 2 2 3
16736 061/300 Read block 3575 len 12288
16737 062/300 Write block 2763 len 28672 data: 4 4 3 3 3 3 3
16738 063/300 Write block 3052 len 4096 data: 3
16739 064/300 Read block 407 len 4096
16740 065/300 Read block 3437 len 4096
16741 066/300 Flush
16742 067/300 Read block 4871 len 4096
16743 068/300 Read block 78 len 36864
16744 069/300 Write block 3280 len 36864 data: 5 4 3 4 3 3 4 4 4
16745 070/300 Write block 3457 len 36864 data: 5 5 5 6 6 5 6 9 7
16746 071/300 Read block 4188 len 12288
16747 072/300 Read block 2791 len 32768
16748 073/300 Read block 4047 len 16384
16749 074/300 Write block 1910 len 4096 data: 2
16750 075/300 Read block 3698 len 28672
16751 076/300 Flush
16752 077/300 Write block 1388 len 4096 data: 5
16753 078/300 Write block 1170 len 36864 data: 2 2 3 3 3 3 3 3 2
16754 079/300 Write block 2053 len 16384 data: 5 3 4 5
16755 080/300 Write block 2921 len 20480 data: 5 4 4 4 4
16756 081/300 Write block 2782 len 4096 data: 3
16757 082/300 Write block 3191 len 16384 data: 2 3 4 4
16758 083/300 Write block 3043 len 36864 data: 8 7 9 9 8 9 8 8 5
16759 084/300 Read block 3054 len 16384
16760 085/300 Write block 2803 len 4096 data: 3
16761 086/300 Flush
16762 087/300 Read block 2990 len 20480
16763 088/300 Write block 3806 len 20480 data: 5 4 4 3 2
16764 089/300 Read block 910 len 20480
16765 090/300 Write block 2632 len 16384 data: 4 4 3 5
16766 091/300 Read block 1022 len 4096
16767 092/300 Read block 253 len 24576
16768 093/300 Read block 1087 len 16384
16769 094/300 Write block 3631 len 40960 data: 4 4 5 5 4 4 3 5 6 5
16770 095/300 Write block 1703 len 16384 data: 4 4 4 4
16771 096/300 Write block 3589 len 28672 data: 3 3 3 3 3 3 3
16772 097/300 Read block 3172 len 8192
16773 098/300 Flush
16774 099/300 Read block 4469 len 8192
16775 100/300 Write block 4755 len 32768 data: 6 6 5 5 5 5 5 4
16776 101/300 Flush
16777 102/300 Read block 2411 len 28672
16778 103/300 Read block 3620 len 16384
16779 104/300 Read block 3577 len 12288
16780 105/300 Read block 239 len 32768
16781 106/300 Read block 995 len 12288
16782 107/300 Read block 2202 len 16384
16783 108/300 Write block 504 len 20480 data: 2 3 3 3 3
16784 109/300 Read block 4314 len 4096
16785 110/300 Read block 80 len 28672
16786 111/300 Write block 4255 len 20480 data: 4 4 5 6 5
16787 112/300 Write block 150 len 40960 data: 5 5 4 4 4 4 3 2 3 3
16788 113/300 Read block 2261 len 16384
16789 114/300 Read block 2651 len 20480
16790 115/300 Write block 2917 len 4096 data: 3
16791 116/300 Read block 4445 len 32768
16792 117/300 Flush
16793 118/300 Read block 4554 len 8192
16794 119/300 Read block 73 len 20480
16795 120/300 Read block 1048 len 40960
16796 121/300 Read block 1603 len 20480
16797 122/300 Read block 4446 len 28672
16798 123/300 Read block 1436 len 36864
16799 124/300 Write block 371 len 8192 data: 2 4
16800 125/300 Read block 4010 len 28672
16801 126/300 Read block 4188 len 36864
16802 127/300 Read block 2891 len 4096
16803 128/300 Write block 4868 len 28672 data: 3 4 5 5 6 6 6
16804 129/300 Read block 393 len 24576
16805 130/300 Write block 3436 len 36864 data: 3 3 3 3 2 3 4 4 4
16806 131/300 Read block 3041 len 12288
16807 132/300 Write block 4019 len 12288 data: 2 3 4
16808 133/300 Flush
16809 134/300 Read block 1188 len 8192
16810 135/300 Read block 539 len 24576
16811 136/300 Read block 1246 len 24576
16812 137/300 Read block 4440 len 40960
16813 138/300 Write block 1083 len 32768 data: 4 4 5 5 5 6 7 6
16814 139/300 Write block 1416 len 28672 data: 4 5 8 8 8 8 8
16815 140/300 Write block 473 len 36864 data: 5 4 5 4 4 5 4 4 5
16816 141/300 Write block 2742 len 12288 data: 4 2 2
16817 142/300 Flush
16818 143/300 Read block 4704 len 20480
16819 144/300 Read block 4484 len 24576
16820 145/300 Read block 4076 len 8192
16821 146/300 Read block 1554 len 8192
16822 147/300 Read block 4151 len 24576
16823 148/300 Read block 3169 len 16384
16824 149/300 Read block 4280 len 24576
16825 150/300 Read block 2881 len 40960
16826 151/300 Write block 1509 len 32768 data: 5 5 5 4 4 5 2 2
16827 152/300 Write block 690 len 28672 data: 3 3 4 4 4 4 4
16828 153/300 Write block 2413 len 40960 data: 2 2 4 5 4 4 3 2 2 3
16829 154/300 Read block 467 len 36864
16830 155/300 Write block 1773 len 20480 data: 6 7 6 6 6
16831 156/300 Write block 2930 len 20480 data: 6 5 6 6 6
16832 157/300 Read block 4323 len 12288
16833 158/300 Read block 3846 len 24576
16834 159/300 Flush
16835 160/300 Write block 4911 len 24576 data: 4 3 3 3 5 5
16836 161/300 Read block 1603 len 12288
16837 162/300 Write block 485 len 32768 data: 4 4 6 6 5 6 6 6
16838 163/300 Read block 2409 len 36864
16839 164/300 Write block 4627 len 20480 data: 4 4 3 3 2
16840 165/300 Read block 2962 len 12288
16841 166/300 Write block 4251 len 36864 data: 4 3 4 4 5 5 6 7 6
16842 167/300 Read block 2966 len 20480
16843 168/300 Read block 3979 len 4096
16844 169/300 Write block 1243 len 4096 data: 4
16845 170/300 Read block 2728 len 28672
16846 171/300 Write block 2855 len 40960 data: 5 5 5 5 4 4 4 4 3 3
16847 172/300 Flush
16848 173/300 Write block 2948 len 40960 data: 4 4 4 4 3 3 3 3 4 3
16849 174/300 Write block 1575 len 20480 data: 2 3 3 3 3
16850 175/300 Write block 3897 len 36864 data: 5 5 5 5 5 5 4 4 4
16851 176/300 Read block 4109 len 8192
16852 177/300 Write block 3521 len 8192 data: 3 3
16853 178/300 Write block 939 len 40960 data: 6 5 5 5 5 6 6 4 3 3
16854 179/300 Write block 2678 len 32768 data: 5 4 4 4 5 5 4 5
16855 180/300 Read block 2921 len 40960
16856 181/300 Write block 175 len 16384 data: 4 6 5 8
16857 182/300 Write block 3242 len 8192 data: 5 5
16858 183/300 Write block 1231 len 16384 data: 4 4 4 6
16859 184/300 Read block 1223 len 16384
16860 185/300 Read block 3437 len 4096
16861 186/300 Write block 821 len 12288 data: 6 7 7
16862 187/300 Write block 355 len 8192 data: 3 3
16863 188/300 Read block 1629 len 40960
16864 189/300 Write block 1819 len 12288 data: 4 5 5
16865 190/300 Read block 899 len 36864
16866 191/300 Read block 2937 len 12288
16867 192/300 Read block 3727 len 16384
16868 193/300 Write block 4474 len 32768 data: 5 5 4 4 3 4 4 3
16869 194/300 Read block 1333 len 20480
16870 195/300 Read block 2808 len 36864
16871 196/300 Read block 1152 len 24576
16872 197/300 Write block 2379 len 12288 data: 3 2 2
16873 198/300 Flush
16874 199/300 Read block 4617 len 12288
16875 200/300 Read block 878 len 32768
16876 201/300 Read block 3731 len 24576
16877 202/300 Read block 2336 len 32768
16878 203/300 Write block 3727 len 16384 data: 3 3 3 3
16879 204/300 Write block 1577 len 8192 data: 4 4
16880 205/300 Flush
16881 206/300 Write block 118 len 12288 data: 7 5 4
16882 207/300 Flush
16883 208/300 Read block 4930 len 40960
16884 209/300 Write block 936 len 28672 data: 5 6 5 7 6 6 6
16885 210/300 Flush
16886 211/300 Write block 795 len 4096 data: 3
16887 212/300 Write block 2689 len 36864 data: 5 5 3 3 2 2 3 3 3
16888 213/300 Read block 3276 len 8192
16889 214/300 Write block 1064 len 28672 data: 5 5 5 5 5 6 6
16890 215/300 Write block 979 len 40960 data: 3 3 4 4 4 5 6 6 4 3
16891 216/300 Flush
16892 217/300 Read block 4547 len 40960
16893 218/300 Write block 2316 len 12288 data: 5 5 4
16894 219/300 Read block 3704 len 28672
16895 220/300 Read block 260 len 16384
16896 221/300 Read block 1417 len 24576
16897 222/300 Flush
16898 223/300 Write block 967 len 40960 data: 6 5 3 3 2 2 2 2 3 2
16899 224/300 Read block 4943 len 4096
16900 225/300 Write block 542 len 12288 data: 6 5 4
16901 226/300 Write block 36 len 24576 data: 5 5 5 6 6 3
16902 227/300 Read block 1526 len 32768
16903 228/300 Write block 3353 len 12288 data: 4 5 3
16904 229/300 Write block 4519 len 24576 data: 2 3 3 3 4 4
16905 230/300 Write block 4938 len 32768 data: 3 3 4 4 3 3 4 3
16906 231/300 Read block 618 len 40960
16907 232/300 Write block 668 len 36864 data: 4 4 5 5 4 5 4 5 5
16908 233/300 Write block 4482 len 40960 data: 3 3 4 5 4 4 3 4 4 4
16909 234/300 Read block 134 len 36864
16910 235/300 Read block 3889 len 28672
16911 236/300 Write block 3444 len 36864 data: 5 3 3 3 3 3 3 2 2
16912 237/300 Read block 4613 len 12288
16913 238/300 Write block 3247 len 40960 data: 5 4 4 4 6 6 6 5 6 6
16914 239/300 Write block 519 len 40960 data: 3 3 3 3 2 3 4 6 7 7
16915 240/300 Write block 1073 len 40960 data: 4 3 3 4 3 3 3 3 4 4
16916 241/300 Write block 174 len 12288 data: 4 5 7
16917 242/300 Read block 1543 len 12288
16918 243/300 Read block 3695 len 8192
16919 244/300 Read block 3750 len 8192
16920 245/300 Read block 2218 len 28672
16921 246/300 Flush
16922 247/300 Read block 4389 len 28672
16923 248/300 Flush
16924 249/300 Write block 1923 len 40960 data: 4 3 3 2 2 2 3 3 4 5
16925 250/300 Read block 2150 len 36864
16926 251/300 Read block 2310 len 36864
16927 252/300 Write block 3577 len 12288 data: 4 3 3
16928 253/300 Read block 1630 len 12288
16929 254/300 Write block 4884 len 20480 data: 4 4 3 3 3
16930 255/300 Write block 4050 len 28672 data: 4 4 4 3 3 3 4
16931 256/300 Read block 2643 len 24576
16932 257/300 Read block 423 len 16384
16933 258/300 Write block 1947 len 36864 data: 3 3 3 3 3 3 3 3 3
16934 259/300 Write block 4028 len 40960 data: 5 6 6 6 4 4 3 2 2 2
16935 260/300 Flush
16936 261/300 Read block 2422 len 12288
16937 262/300 Flush
16938 263/300 Read block 1057 len 32768
16939 264/300 Read block 2998 len 12288
16940 265/300 Read block 1305 len 12288
16941 266/300 Read block 4475 len 36864
16942 267/300 Read block 327 len 4096
16943 268/300 Read block 1929 len 40960
16944 269/300 Write block 1575 len 40960 data: 3 4 5 5 4 3 3 3 4 4
16945 270/300 Read block 1070 len 24576
16946 271/300 Write block 2100 len 36864 data: 5 5 6 6 6 6 5 4 5
16947 272/300 Write block 2302 len 40960 data: 6 7 5 4 5 5 4 4 4 4
16948 273/300 Flush
16949 274/300 Read block 1788 len 28672
16950 275/300 Write block 1073 len 36864 data: 5 4 4 5 4 4 4 4 5
16951 276/300 Read block 2022 len 20480
16952 277/300 Read block 853 len 40960
16953 278/300 Read block 439 len 16384
16954 279/300 Flush
16955 280/300 Write block 1424 len 24576 data: 6 6 6 5 3 3
16956 281/300 Read block 4586 len 8192
16957 282/300 Write block 1312 len 12288 data: 7 7 6
16958 283/300 Read block 4591 len 16384
16959 284/300 Write block 1693 len 16384 data: 7 7 7 7
16960 285/300 Write block 305 len 40960 data: 4 4 4 4 5 5 5 5 5 4
16961 286/300 Read block 598 len 32768
16962 287/300 Read block 3886 len 12288
16963 288/300 Read block 4840 len 40960
16964 289/300 Write block 3456 len 28672 data: 3 6 6 6 7 7 6
16965 290/300 Read block 1680 len 8192
16966 291/300 Write block 2292 len 4096 data: 5
16967 292/300 Read block 3132 len 36864
16968 293/300 Read block 3019 len 20480
16969 294/300 Read block 4165 len 16384
16970 295/300 Write block 2577 len 24576 data: 4 2 2 3 3 3
16971 296/300 Read block 1835 len 8192
16972 297/300 Read block 2985 len 40960
16973 298/300 Read block 1720 len 20480
16974 299/300 Flush
16975 300/300 Read block 4694 len 4096
16976 301/300 Write block 3408 len 12288 data: 3 3 3
16977 Main task has work to do, go find it
16978 got dsc Start(2)
16979 start 2
16980 Jan 23 18:51:28.950 INFO request completed, latency_us: 251, response_code: 204, uri: /start/cid/2, method: POST, req_id: aab6b8e7-319d-4404-a560-fe0479ff4c73, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
16981 Replay: started 2, returned:Ok(())
16982 ----------------------------------------------------------------
16983 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16984 States: Active Active Offline
16985 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16986 5528 Acked Flush 0 Done Done Sent false
16987 5529 Acked Write 3 Done Done Sent false
16988 5530 Acked Read 3 Done Done Sent false
16989 5531 Acked Write 3 Done Done Sent false
16990 5532 Acked Write 1 Done Done Sent false
16991 5533 Acked Write 1 Done Done Sent false
16992 5534 Acked Read 8 Done Done Sent false
16993 5535 Acked Write 8 Done Done Sent false
16994 5536 Acked Read 6 Done Done Sent false
16995 5537 Acked Write 9 Done Done Sent false
16996 5538 Acked Write 5 Done Done Sent false
16997 5539 Acked Write 2 Done Done Sent false
16998 5540 Acked Write 1 Done Done Sent false
16999 5541 Acked Write 7 Done Done Sent false
17000 5542 Acked Write 4 Done Done Sent false
17001 5543 Acked Write 3 Done Done Sent false
17002 5544 Acked Read 8 Done Done Sent false
17003 5545 Acked Flush 0 Done Done Sent false
17004 5546 Acked Read 4 Done Done Sent false
17005 5547 Acked Read 1 Done Done Sent false
17006 5548 Acked Write 9 Done Done Sent false
17007 5549 Acked Write 3 Done Done Sent false
17008 5550 Acked Write 10 Done Done Sent false
17009 5551 Acked Read 6 Done Done Sent false
17010 5552 Acked Write 2 Done Done Sent false
17011 5553 Acked Read 1 Done Done Sent false
17012 5554 Acked Write 7 Done Done Sent false
17013 5555 Acked Write 2 Done Done Sent false
17014 5556 Acked Write 10 Done Done Sent false
17015 5557 Acked Write 2 Done Done Sent false
17016 5558 Acked Write 4 Done Done Sent false
17017 5559 Acked Read 4 Done Done Sent false
17018 5560 Acked Write 10 Done Done Sent false
17019 5561 Acked Read 9 Done Done Sent false
17020 5562 Acked Write 2 Done Done Sent false
17021 5563 Acked Write 10 Done Done Sent false
17022 5564 Acked Write 5 Done Done Sent false
17023 5565 Acked Read 4 Done Done Sent false
17024 5566 Acked Read 8 Done Done Sent false
17025 5567 Acked Read 2 Done Done Sent false
17026 5568 Acked Write 8 Done Done Sent false
17027 5569 Acked Write 2 Done Done Sent false
17028 5570 Acked Read 7 Done Done Sent false
17029 5571 Acked Flush 0 Done Done Sent false
17030 5572 Acked Read 1 Done Done Sent false
17031 5573 Acked Write 8 Done Done Sent false
17032 5574 Acked Write 10 Done Done Sent false
17033 5575 Acked Read 5 Done Done Sent false
17034 5576 Acked Read 10 Done Done Sent false
17035 5577 Acked Write 2 Done Done Sent false
17036 5578 Acked Write 5 Done Done Sent false
17037 5579 Acked Read 7 Done Done Sent false
17038 5580 Acked Flush 0 Done Done Sent false
17039 5581 Acked Read 9 Done Done Sent false
17040 5582 Acked Read 6 Done Done Sent false
17041 5583 Acked Read 6 Done Done Sent false
17042 5584 Acked Write 5 Done Done Sent false
17043 5585 Acked Write 2 Done Done Sent false
17044 5586 Acked Flush 0 Done Done Sent false
17045 5587 Acked Write 10 Done Done Sent false
17046 5588 Acked Read 3 Done Done Sent false
17047 5589 Acked Write 7 Done Done Sent false
17048 5590 Acked Write 1 Done Done Sent false
17049 5591 Acked Read 1 Done Done Sent false
17050 5592 Acked Read 1 Done Done Sent false
17051 5593 Acked Flush 0 Done Done Sent false
17052 5594 Acked Read 1 Done Done Sent false
17053 5595 Acked Read 9 Done Done Sent false
17054 5596 Acked Write 9 Done Done Sent false
17055 5597 Acked Write 9 Done Done Sent false
17056 5598 Acked Read 3 Done Done Sent false
17057 5599 Acked Read 8 Done Done Sent false
17058 5600 Acked Read 4 Done Done Sent false
17059 5601 Acked Write 1 Done Done Sent false
17060 5602 Acked Read 7 Done Done Sent false
17061 5603 Acked Flush 0 Done Done Sent false
17062 5604 Acked Write 1 Done Done Sent false
17063 5605 Acked Write 9 Done Done Sent false
17064 5606 Acked Write 4 Done Done Sent false
17065 5607 Acked Write 5 Done Done Sent false
17066 5608 Acked Write 1 Done Done Sent false
17067 5609 Acked Write 4 Done Done Sent false
17068 5610 Acked Write 9 Done Done Sent false
17069 5611 Acked Read 4 Done Done Sent false
17070 5612 Acked Write 1 Done Done Sent false
17071 5613 Acked Flush 0 Done Done Sent false
17072 5614 Acked Read 5 Done Done Sent false
17073 5615 Acked Write 5 Done Done Sent false
17074 5616 Acked Read 5 Done Done Sent false
17075 5617 Acked Write 4 Done Done Sent false
17076 5618 Acked Read 1 Done Done Sent false
17077 5619 Acked Read 6 Done Done Sent false
17078 5620 Acked Read 4 Done Done Sent false
17079 5621 Acked Write 10 Done Done Sent false
17080 5622 Acked Write 4 Done Done Sent false
17081 5623 Acked Write 7 Done Done Sent false
17082 5624 Acked Read 2 Done Done Sent false
17083 5625 Acked Flush 0 Done Done Sent false
17084 5626 Acked Read 2 Done Done Sent false
17085 5627 Acked Write 8 Done Done Sent false
17086 5628 Acked Flush 0 Done Done Sent false
17087 5629 Acked Read 7 Done Done Sent false
17088 5630 Acked Read 4 Done Done Sent false
17089 5631 Acked Read 3 Done Done Sent false
17090 5632 Acked Read 8 Done Done Sent false
17091 5633 Acked Read 3 Done Done Sent false
17092 5634 Acked Read 4 Done Done Sent false
17093 5635 Acked Write 5 Done Done Sent false
17094 5636 Acked Read 1 Done Done Sent false
17095 5637 Acked Read 7 Done Done Sent false
17096 5638 Acked Write 5 Done Done Sent false
17097 5639 Acked Write 10 Done Done Sent false
17098 5640 Acked Read 4 Done Done Sent false
17099 5641 Acked Read 5 Done Done Sent false
17100 5642 Acked Write 1 Done Done Sent false
17101 5643 Acked Read 8 Done Done Sent false
17102 5644 Acked Flush 0 Done Done Sent false
17103 5645 Acked Read 2 Done Done Sent false
17104 5646 Acked Read 5 Done Done Sent false
17105 5647 Acked Read 10 Done Done Sent false
17106 5648 Acked Read 5 Done Done Sent false
17107 5649 Acked Read 7 Done Done Sent false
17108 5650 Acked Read 9 Done Done Sent false
17109 5651 Acked Write 2 Done Done Sent false
17110 5652 Acked Read 7 Done Done Sent false
17111 5653 Acked Read 9 Done Done Sent false
17112 5654 Acked Read 1 Done Done Sent false
17113 5655 Acked Write 7 Done Done Sent false
17114 5656 Acked Read 6 Done Done Sent false
17115 5657 Acked Write 9 Done Done Sent false
17116 5658 Acked Read 3 Done Done Sent false
17117 5659 Acked Write 3 Done Done Sent false
17118 5660 Acked Flush 0 Done Done Sent false
17119 5661 Acked Read 2 Done Done Sent false
17120 5662 Acked Read 6 Done Done Sent false
17121 5663 Acked Read 6 Done Done Sent false
17122 5664 Acked Read 10 Done Done Sent false
17123 5665 Acked Write 8 Done Done Sent false
17124 5666 Acked Write 7 Done Done Sent false
17125 5667 Acked Write 9 Done Done Sent false
17126 5668 Acked Write 3 Done Done Sent false
17127 5669 Acked Flush 0 Done Done Sent false
17128 5670 Acked Read 5 Done Done Sent false
17129 5671 Acked Read 6 Done Done Sent false
17130 5672 Acked Read 2 Done Done Sent false
17131 5673 Acked Read 2 Done Done Sent false
17132 5674 Acked Read 6 Done Done Sent false
17133 5675 Acked Read 4 Done Done Sent false
17134 5676 Acked Read 6 Done Done Sent false
17135 5677 Acked Read 10 Done Done Sent false
17136 5678 Acked Write 8 Done Done Sent false
17137 5679 Acked Write 7 Done Done Sent false
17138 5680 Acked Write 10 Done Done Sent false
17139 5681 Acked Read 9 Done Done Sent false
17140 5682 Acked Write 5 Done Done Sent false
17141 5683 Acked Write 5 Done Done Sent false
17142 5684 Acked Read 3 Done Done Sent false
17143 5685 Acked Read 6 Done Done Sent false
17144 5686 Acked Flush 0 Done Done Sent false
17145 5687 Acked Write 6 Done Done Sent false
17146 5688 Acked Read 3 Done Done Sent false
17147 5689 Acked Write 8 Done Done Sent false
17148 5690 Acked Read 9 Done Done Sent false
17149 5691 Acked Write 5 Done Done Sent false
17150 5692 Acked Read 3 Done Done Sent false
17151 5693 Acked Write 9 Done Done Sent false
17152 5694 Acked Read 5 Done Done Sent false
17153 5695 Acked Read 1 Done Done Sent false
17154 5696 Acked Write 1 Done Done Sent false
17155 5697 Acked Read 7 Done Done Sent false
17156 5698 Acked Write 10 Done Done Sent false
17157 5699 Acked Flush 0 Done Done Sent false
17158 5700 Acked Write 10 Done Done Sent false
17159 5701 Acked Write 5 Done Done Sent false
17160 5702 Acked Write 9 Done Done Sent false
17161 5703 Acked Read 2 Done Done Sent false
17162 5704 Acked Write 2 Done Done Sent false
17163 5705 Acked Write 10 Done Done Sent false
17164 5706 Acked Write 8 Done Done Sent false
17165 5707 Acked Read 10 Done Done Sent false
17166 5708 Acked Write 4 Done Done Sent false
17167 5709 Acked Write 2 Done Done Sent false
17168 5710 Acked Write 4 Done Done Sent false
17169 5711 Acked Read 4 Done Done Sent false
17170 5712 Acked Read 1 Done Done Sent false
17171 5713 Acked Write 3 Done Done Sent false
17172 5714 Acked Write 2 Done Done Sent false
17173 5715 Acked Read 10 Done Done Sent false
17174 5716 Acked Write 3 Done Done Sent false
17175 5717 Acked Read 9 Done Done Sent false
17176 5718 Acked Read 3 Done Done Sent false
17177 5719 Acked Read 4 Done Done Sent false
17178 5720 Acked Write 8 Done Done Sent false
17179 5721 Acked Read 5 Done Done Sent false
17180 5722 Acked Read 9 Done Done Sent false
17181 5723 Acked Read 6 Done Done Sent false
17182 5724 Acked Write 3 Done Done Sent false
17183 5725 Acked Flush 0 Done Done Sent false
17184 5726 Acked Read 3 Done Done Sent false
17185 5727 Acked Read 8 Done Done Sent false
17186 5728 Acked Read 6 Done Done Sent false
17187 5729 Acked Read 8 Done Done Sent false
17188 5730 Acked Write 4 Done Done Sent false
17189 5731 Acked Write 2 Done Done Sent false
17190 5732 Acked Flush 0 Done Done Sent false
17191 5733 Acked Write 3 Done Done Sent false
17192 5734 Acked Flush 0 Done Done Sent false
17193 5735 Acked Read 10 Done Done Sent false
17194 5736 Acked Write 7 Done Done Sent false
17195 5737 Acked Flush 0 Done Done Sent false
17196 5738 Acked Write 1 Done Done Sent false
17197 5739 Acked Write 9 Done Done Sent false
17198 5740 Acked Read 2 Done Done Sent false
17199 5741 Acked Write 7 Done Done Sent false
17200 5742 Acked Write 10 Done Done Sent false
17201 5743 Acked Flush 0 Done Done Sent false
17202 5744 Acked Read 10 Done Done Sent false
17203 5745 Acked Write 3 Done Done Sent false
17204 5746 Acked Read 7 Done Done Sent false
17205 5747 Acked Read 4 Done Done Sent false
17206 5748 Acked Read 6 Done Done Sent false
17207 5749 Acked Flush 0 Done Done Sent false
17208 5750 Acked Write 10 Done Done Sent false
17209 5751 Acked Read 1 Done Done Sent false
17210 5752 Acked Write 3 Done Done Sent false
17211 5753 Acked Write 6 Done Done Sent false
17212 5754 Acked Read 8 Done Done Sent false
17213 5755 Acked Write 3 Done Done Sent false
17214 5756 Acked Write 6 Done Done Sent false
17215 5757 Acked Write 8 Done Done Sent false
17216 5758 Acked Read 10 Done Done Sent false
17217 5759 Acked Write 9 Done Done Sent false
17218 5760 Acked Write 10 Done Done Sent false
17219 5761 Acked Read 9 Done Done Sent false
17220 5762 Acked Read 7 Done Done Sent false
17221 5763 Acked Write 9 Done Done Sent false
17222 5764 Acked Read 3 Done Done Sent false
17223 5765 Acked Write 10 Done Done Sent false
17224 5766 Acked Write 10 Done Done Sent false
17225 5767 Acked Write 10 Done Done Sent false
17226 5768 Acked Write 3 Done Done Sent false
17227 5769 Acked Read 3 Done Done Sent false
17228 5770 Acked Read 2 Done Done Sent false
17229 5771 Acked Read 2 Done Done Sent false
17230 5772 Acked Read 7 Done Done Sent false
17231 5773 Acked Flush 0 Done Done Sent false
17232 5774 Acked Read 7 Done Done Sent false
17233 5775 Acked Flush 0 Done Done Sent false
17234 5776 Acked Write 10 Done Done Sent false
17235 5777 Acked Read 9 Done Done Sent false
17236 5778 Acked Read 9 Done Done Sent false
17237 5779 Acked Write 3 Done Done Sent false
17238 5780 Acked Read 3 Done Done Sent false
17239 5781 Acked Write 5 Done Done Sent false
17240 5782 Acked Write 7 Done Done Sent false
17241 5783 Acked Read 6 Done Done Sent false
17242 5784 Acked Read 4 Done Done Sent false
17243 5785 Acked Write 9 Done Done Sent false
17244 5786 Acked Write 10 Done Done Sent false
17245 5787 Acked Flush 0 Done Done Sent false
17246 5788 Acked Read 3 Done Done Sent false
17247 5789 Acked Flush 0 Done Done Sent false
17248 5790 Acked Read 8 Done Done Sent false
17249 5791 Acked Read 3 Done Done Sent false
17250 5792 Acked Read 3 Done Done Sent false
17251 5793 Acked Read 9 Done Done Sent false
17252 5794 Acked Read 1 Done Done Sent false
17253 5795 Acked Read 10 Done Done Sent false
17254 5796 Acked Write 10 Done Done Sent false
17255 5797 Acked Read 6 Done Done Sent false
17256 5798 Acked Write 9 Done Done Sent false
17257 5799 Acked Write 10 Done Done Sent false
17258 5800 Acked Flush 0 Done Done Sent false
17259 5801 Acked Read 7 Done Done Sent false
17260 5802 Acked Write 9 Done Done Sent false
17261 5803 Acked Read 5 Done Done Sent false
17262 5804 Acked Read 10 Done Done Sent false
17263 5805 Acked Read 4 Done Done Sent false
17264 5806 Acked Flush 0 Done Done Sent false
17265 5807 Acked Write 6 Done Done Sent false
17266 5808 Acked Read 2 Done Done Sent false
17267 5809 Acked Write 3 Done Done Sent false
17268 5810 Acked Read 4 Done Done Sent false
17269 5811 Acked Write 4 Done Done Sent false
17270 5812 Acked Write 10 Done Done Sent false
17271 5813 Acked Read 8 Done Done Sent false
17272 5814 Acked Read 3 Done Done Sent false
17273 5815 Acked Read 10 Done Done Sent false
17274 5816 Acked Write 7 Done Done Sent false
17275 5817 Acked Read 2 Done Done Sent false
17276 5818 Acked Write 1 Done Done Sent false
17277 5819 Acked Read 9 Done Done Sent false
17278 5820 Acked Read 5 Done Done Sent false
17279 5821 Acked Read 4 Done Done Sent false
17280 5822 Acked Write 6 Done Done Sent false
17281 5823 Acked Read 2 Done Done Sent false
17282 5824 Acked Read 10 Done Done Sent false
17283 5825 Acked Read 5 Done Done Sent false
17284 5826 Acked Flush 0 Done Done Sent false
17285 5827 Acked Read 1 Done Done Sent false
17286 5828 Acked Write 3 Done Done Sent false
17287 STATES DS:0 DS:1 DS:2 TOTAL
17288 Sent 0 0 301 301
17289 Done 301 301 0 602
17290 Skipped 0 0 0 0
17291 Error 0 0 0 0
17292 Last Flush: Some(JobId(5826)) Some(JobId(5826)) Some(JobId(5527))
17293 Downstairs last five completed: 5527 5526 5525 5524 5523
17294 Upstairs last five completed: 5828 5827 5826 5825 5824
17295 CLIENT: Up:0 ds:301 act:2
17296 [8830] got start action
17297 [8830] I am going to restart
17298 Starting downstairs at port 8830
17299 Make output file at "/tmp/dsc/downstairs-8830.txt"
17300 [8830][2] reports Starting
17301 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1216)
17302 [8830][2] reports Running
17303 ----------------------------------------------------------------
17304 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
17305 States: Active Active Offline
17306 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17307 5528 Acked Flush 0 Done Done Sent false
17308 5529 Acked Write 3 Done Done Sent false
17309 5530 Acked Read 3 Done Done Sent false
17310 5531 Acked Write 3 Done Done Sent false
17311 5532 Acked Write 1 Done Done Sent false
17312 5533 Acked Write 1 Done Done Sent false
17313 5534 Acked Read 8 Done Done Sent false
17314 5535 Acked Write 8 Done Done Sent false
17315 5536 Acked Read 6 Done Done Sent false
17316 5537 Acked Write 9 Done Done Sent false
17317 5538 Acked Write 5 Done Done Sent false
17318 5539 Acked Write 2 Done Done Sent false
17319 5540 Acked Write 1 Done Done Sent false
17320 5541 Acked Write 7 Done Done Sent false
17321 5542 Acked Write 4 Done Done Sent false
17322 5543 Acked Write 3 Done Done Sent false
17323 5544 Acked Read 8 Done Done Sent false
17324 5545 Acked Flush 0 Done Done Sent false
17325 5546 Acked Read 4 Done Done Sent false
17326 5547 Acked Read 1 Done Done Sent false
17327 5548 Acked Write 9 Done Done Sent false
17328 5549 Acked Write 3 Done Done Sent false
17329 5550 Acked Write 10 Done Done Sent false
17330 5551 Acked Read 6 Done Done Sent false
17331 5552 Acked Write 2 Done Done Sent false
17332 5553 Acked Read 1 Done Done Sent false
17333 5554 Acked Write 7 Done Done Sent false
17334 5555 Acked Write 2 Done Done Sent false
17335 5556 Acked Write 10 Done Done Sent false
17336 5557 Acked Write 2 Done Done Sent false
17337 5558 Acked Write 4 Done Done Sent false
17338 5559 Acked Read 4 Done Done Sent false
17339 5560 Acked Write 10 Done Done Sent false
17340 5561 Acked Read 9 Done Done Sent false
17341 5562 Acked Write 2 Done Done Sent false
17342 5563 Acked Write 10 Done Done Sent false
17343 5564 Acked Write 5 Done Done Sent false
17344 5565 Acked Read 4 Done Done Sent false
17345 5566 Acked Read 8 Done Done Sent false
17346 5567 Acked Read 2 Done Done Sent false
17347 5568 Acked Write 8 Done Done Sent false
17348 5569 Acked Write 2 Done Done Sent false
17349 5570 Acked Read 7 Done Done Sent false
17350 5571 Acked Flush 0 Done Done Sent false
17351 5572 Acked Read 1 Done Done Sent false
17352 5573 Acked Write 8 Done Done Sent false
17353 5574 Acked Write 10 Done Done Sent false
17354 5575 Acked Read 5 Done Done Sent false
17355 5576 Acked Read 10 Done Done Sent false
17356 5577 Acked Write 2 Done Done Sent false
17357 5578 Acked Write 5 Done Done Sent false
17358 5579 Acked Read 7 Done Done Sent false
17359 5580 Acked Flush 0 Done Done Sent false
17360 5581 Acked Read 9 Done Done Sent false
17361 5582 Acked Read 6 Done Done Sent false
17362 5583 Acked Read 6 Done Done Sent false
17363 5584 Acked Write 5 Done Done Sent false
17364 5585 Acked Write 2 Done Done Sent false
17365 5586 Acked Flush 0 Done Done Sent false
17366 5587 Acked Write 10 Done Done Sent false
17367 5588 Acked Read 3 Done Done Sent false
17368 5589 Acked Write 7 Done Done Sent false
17369 5590 Acked Write 1 Done Done Sent false
17370 5591 Acked Read 1 Done Done Sent false
17371 5592 Acked Read 1 Done Done Sent false
17372 5593 Acked Flush 0 Done Done Sent false
17373 5594 Acked Read 1 Done Done Sent false
17374 5595 Acked Read 9 Done Done Sent false
17375 5596 Acked Write 9 Done Done Sent false
17376 5597 Acked Write 9 Done Done Sent false
17377 5598 Acked Read 3 Done Done Sent false
17378 5599 Acked Read 8 Done Done Sent false
17379 5600 Acked Read 4 Done Done Sent false
17380 5601 Acked Write 1 Done Done Sent false
17381 5602 Acked Read 7 Done Done Sent false
17382 5603 Acked Flush 0 Done Done Sent false
17383 5604 Acked Write 1 Done Done Sent false
17384 5605 Acked Write 9 Done Done Sent false
17385 5606 Acked Write 4 Done Done Sent false
17386 5607 Acked Write 5 Done Done Sent false
17387 5608 Acked Write 1 Done Done Sent false
17388 5609 Acked Write 4 Done Done Sent false
17389 5610 Acked Write 9 Done Done Sent false
17390 5611 Acked Read 4 Done Done Sent false
17391 5612 Acked Write 1 Done Done Sent false
17392 5613 Acked Flush 0 Done Done Sent false
17393 5614 Acked Read 5 Done Done Sent false
17394 5615 Acked Write 5 Done Done Sent false
17395 5616 Acked Read 5 Done Done Sent false
17396 5617 Acked Write 4 Done Done Sent false
17397 5618 Acked Read 1 Done Done Sent false
17398 5619 Acked Read 6 Done Done Sent false
17399 5620 Acked Read 4 Done Done Sent false
17400 5621 Acked Write 10 Done Done Sent false
17401 5622 Acked Write 4 Done Done Sent false
17402 5623 Acked Write 7 Done Done Sent false
17403 5624 Acked Read 2 Done Done Sent false
17404 5625 Acked Flush 0 Done Done Sent false
17405 5626 Acked Read 2 Done Done Sent false
17406 5627 Acked Write 8 Done Done Sent false
17407 5628 Acked Flush 0 Done Done Sent false
17408 5629 Acked Read 7 Done Done Sent false
17409 5630 Acked Read 4 Done Done Sent false
17410 5631 Acked Read 3 Done Done Sent false
17411 5632 Acked Read 8 Done Done Sent false
17412 5633 Acked Read 3 Done Done Sent false
17413 5634 Acked Read 4 Done Done Sent false
17414 5635 Acked Write 5 Done Done Sent false
17415 5636 Acked Read 1 Done Done Sent false
17416 5637 Acked Read 7 Done Done Sent false
17417 5638 Acked Write 5 Done Done Sent false
17418 5639 Acked Write 10 Done Done Sent false
17419 5640 Acked Read 4 Done Done Sent false
17420 5641 Acked Read 5 Done Done Sent false
17421 5642 Acked Write 1 Done Done Sent false
17422 5643 Acked Read 8 Done Done Sent false
17423 5644 Acked Flush 0 Done Done Sent false
17424 5645 Acked Read 2 Done Done Sent false
17425 5646 Acked Read 5 Done Done Sent false
17426 5647 Acked Read 10 Done Done Sent false
17427 5648 Acked Read 5 Done Done Sent false
17428 5649 Acked Read 7 Done Done Sent false
17429 5650 Acked Read 9 Done Done Sent false
17430 5651 Acked Write 2 Done Done Sent false
17431 5652 Acked Read 7 Done Done Sent false
17432 5653 Acked Read 9 Done Done Sent false
17433 5654 Acked Read 1 Done Done Sent false
17434 5655 Acked Write 7 Done Done Sent false
17435 5656 Acked Read 6 Done Done Sent false
17436 5657 Acked Write 9 Done Done Sent false
17437 5658 Acked Read 3 Done Done Sent false
17438 5659 Acked Write 3 Done Done Sent false
17439 5660 Acked Flush 0 Done Done Sent false
17440 5661 Acked Read 2 Done Done Sent false
17441 5662 Acked Read 6 Done Done Sent false
17442 5663 Acked Read 6 Done Done Sent false
17443 5664 Acked Read 10 Done Done Sent false
17444 5665 Acked Write 8 Done Done Sent false
17445 5666 Acked Write 7 Done Done Sent false
17446 5667 Acked Write 9 Done Done Sent false
17447 5668 Acked Write 3 Done Done Sent false
17448 5669 Acked Flush 0 Done Done Sent false
17449 5670 Acked Read 5 Done Done Sent false
17450 5671 Acked Read 6 Done Done Sent false
17451 5672 Acked Read 2 Done Done Sent false
17452 5673 Acked Read 2 Done Done Sent false
17453 5674 Acked Read 6 Done Done Sent false
17454 5675 Acked Read 4 Done Done Sent false
17455 5676 Acked Read 6 Done Done Sent false
17456 5677 Acked Read 10 Done Done Sent false
17457 5678 Acked Write 8 Done Done Sent false
17458 5679 Acked Write 7 Done Done Sent false
17459 5680 Acked Write 10 Done Done Sent false
17460 5681 Acked Read 9 Done Done Sent false
17461 5682 Acked Write 5 Done Done Sent false
17462 5683 Acked Write 5 Done Done Sent false
17463 5684 Acked Read 3 Done Done Sent false
17464 5685 Acked Read 6 Done Done Sent false
17465 5686 Acked Flush 0 Done Done Sent false
17466 5687 Acked Write 6 Done Done Sent false
17467 5688 Acked Read 3 Done Done Sent false
17468 5689 Acked Write 8 Done Done Sent false
17469 5690 Acked Read 9 Done Done Sent false
17470 5691 Acked Write 5 Done Done Sent false
17471 5692 Acked Read 3 Done Done Sent false
17472 5693 Acked Write 9 Done Done Sent false
17473 5694 Acked Read 5 Done Done Sent false
17474 5695 Acked Read 1 Done Done Sent false
17475 5696 Acked Write 1 Done Done Sent false
17476 5697 Acked Read 7 Done Done Sent false
17477 5698 Acked Write 10 Done Done Sent false
17478 5699 Acked Flush 0 Done Done Sent false
17479 5700 Acked Write 10 Done Done Sent false
17480 5701 Acked Write 5 Done Done Sent false
17481 5702 Acked Write 9 Done Done Sent false
17482 5703 Acked Read 2 Done Done Sent false
17483 5704 Acked Write 2 Done Done Sent false
17484 5705 Acked Write 10 Done Done Sent false
17485 5706 Acked Write 8 Done Done Sent false
17486 5707 Acked Read 10 Done Done Sent false
17487 5708 Acked Write 4 Done Done Sent false
17488 5709 Acked Write 2 Done Done Sent false
17489 5710 Acked Write 4 Done Done Sent false
17490 5711 Acked Read 4 Done Done Sent false
17491 5712 Acked Read 1 Done Done Sent false
17492 5713 Acked Write 3 Done Done Sent false
17493 5714 Acked Write 2 Done Done Sent false
17494 5715 Acked Read 10 Done Done Sent false
17495 5716 Acked Write 3 Done Done Sent false
17496 5717 Acked Read 9 Done Done Sent false
17497 5718 Acked Read 3 Done Done Sent false
17498 5719 Acked Read 4 Done Done Sent false
17499 5720 Acked Write 8 Done Done Sent false
17500 5721 Acked Read 5 Done Done Sent false
17501 5722 Acked Read 9 Done Done Sent false
17502 5723 Acked Read 6 Done Done Sent false
17503 5724 Acked Write 3 Done Done Sent false
17504 5725 Acked Flush 0 Done Done Sent false
17505 5726 Acked Read 3 Done Done Sent false
17506 5727 Acked Read 8 Done Done Sent false
17507 5728 Acked Read 6 Done Done Sent false
17508 5729 Acked Read 8 Done Done Sent false
17509 5730 Acked Write 4 Done Done Sent false
17510 5731 Acked Write 2 Done Done Sent false
17511 5732 Acked Flush 0 Done Done Sent false
17512 5733 Acked Write 3 Done Done Sent false
17513 5734 Acked Flush 0 Done Done Sent false
17514 5735 Acked Read 10 Done Done Sent false
17515 5736 Acked Write 7 Done Done Sent false
17516 5737 Acked Flush 0 Done Done Sent false
17517 5738 Acked Write 1 Done Done Sent false
17518 5739 Acked Write 9 Done Done Sent false
17519 5740 Acked Read 2 Done Done Sent false
17520 5741 Acked Write 7 Done Done Sent false
17521 5742 Acked Write 10 Done Done Sent false
17522 5743 Acked Flush 0 Done Done Sent false
17523 5744 Acked Read 10 Done Done Sent false
17524 5745 Acked Write 3 Done Done Sent false
17525 5746 Acked Read 7 Done Done Sent false
17526 5747 Acked Read 4 Done Done Sent false
17527 5748 Acked Read 6 Done Done Sent false
17528 5749 Acked Flush 0 Done Done Sent false
17529 5750 Acked Write 10 Done Done Sent false
17530 5751 Acked Read 1 Done Done Sent false
17531 5752 Acked Write 3 Done Done Sent false
17532 5753 Acked Write 6 Done Done Sent false
17533 5754 Acked Read 8 Done Done Sent false
17534 5755 Acked Write 3 Done Done Sent false
17535 5756 Acked Write 6 Done Done Sent false
17536 5757 Acked Write 8 Done Done Sent false
17537 5758 Acked Read 10 Done Done Sent false
17538 5759 Acked Write 9 Done Done Sent false
17539 5760 Acked Write 10 Done Done Sent false
17540 5761 Acked Read 9 Done Done Sent false
17541 5762 Acked Read 7 Done Done Sent false
17542 5763 Acked Write 9 Done Done Sent false
17543 5764 Acked Read 3 Done Done Sent false
17544 5765 Acked Write 10 Done Done Sent false
17545 5766 Acked Write 10 Done Done Sent false
17546 5767 Acked Write 10 Done Done Sent false
17547 5768 Acked Write 3 Done Done Sent false
17548 5769 Acked Read 3 Done Done Sent false
17549 5770 Acked Read 2 Done Done Sent false
17550 5771 Acked Read 2 Done Done Sent false
17551 5772 Acked Read 7 Done Done Sent false
17552 5773 Acked Flush 0 Done Done Sent false
17553 5774 Acked Read 7 Done Done Sent false
17554 5775 Acked Flush 0 Done Done Sent false
17555 5776 Acked Write 10 Done Done Sent false
17556 5777 Acked Read 9 Done Done Sent false
17557 5778 Acked Read 9 Done Done Sent false
17558 5779 Acked Write 3 Done Done Sent false
17559 5780 Acked Read 3 Done Done Sent false
17560 5781 Acked Write 5 Done Done Sent false
17561 5782 Acked Write 7 Done Done Sent false
17562 5783 Acked Read 6 Done Done Sent false
17563 5784 Acked Read 4 Done Done Sent false
17564 5785 Acked Write 9 Done Done Sent false
17565 5786 Acked Write 10 Done Done Sent false
17566 5787 Acked Flush 0 Done Done Sent false
17567 5788 Acked Read 3 Done Done Sent false
17568 5789 Acked Flush 0 Done Done Sent false
17569 5790 Acked Read 8 Done Done Sent false
17570 5791 Acked Read 3 Done Done Sent false
17571 5792 Acked Read 3 Done Done Sent false
17572 5793 Acked Read 9 Done Done Sent false
17573 5794 Acked Read 1 Done Done Sent false
17574 5795 Acked Read 10 Done Done Sent false
17575 5796 Acked Write 10 Done Done Sent false
17576 5797 Acked Read 6 Done Done Sent false
17577 5798 Acked Write 9 Done Done Sent false
17578 5799 Acked Write 10 Done Done Sent false
17579 5800 Acked Flush 0 Done Done Sent false
17580 5801 Acked Read 7 Done Done Sent false
17581 5802 Acked Write 9 Done Done Sent false
17582 5803 Acked Read 5 Done Done Sent false
17583 5804 Acked Read 10 Done Done Sent false
17584 5805 Acked Read 4 Done Done Sent false
17585 5806 Acked Flush 0 Done Done Sent false
17586 5807 Acked Write 6 Done Done Sent false
17587 5808 Acked Read 2 Done Done Sent false
17588 5809 Acked Write 3 Done Done Sent false
17589 5810 Acked Read 4 Done Done Sent false
17590 5811 Acked Write 4 Done Done Sent false
17591 5812 Acked Write 10 Done Done Sent false
17592 5813 Acked Read 8 Done Done Sent false
17593 5814 Acked Read 3 Done Done Sent false
17594 5815 Acked Read 10 Done Done Sent false
17595 5816 Acked Write 7 Done Done Sent false
17596 5817 Acked Read 2 Done Done Sent false
17597 5818 Acked Write 1 Done Done Sent false
17598 5819 Acked Read 9 Done Done Sent false
17599 5820 Acked Read 5 Done Done Sent false
17600 5821 Acked Read 4 Done Done Sent false
17601 5822 Acked Write 6 Done Done Sent false
17602 5823 Acked Read 2 Done Done Sent false
17603 5824 Acked Read 10 Done Done Sent false
17604 5825 Acked Read 5 Done Done Sent false
17605 5826 Acked Flush 0 Done Done Sent false
17606 5827 Acked Read 1 Done Done Sent false
17607 5828 Acked Write 3 Done Done Sent false
17608 5829 Acked Flush 0 Done Done Sent false
17609 STATES DS:0 DS:1 DS:2 TOTAL
17610 Sent 0 0 302 302
17611 Done 302 302 0 604
17612 Skipped 0 0 0 0
17613 Error 0 0 0 0
17614 Last Flush: Some(JobId(5829)) Some(JobId(5829)) Some(JobId(5527))
17615 Downstairs last five completed: 5527 5526 5525 5524 5523
17616 Upstairs last five completed: 5829 5828 5827 5826 5825
17617 CLIENT: Up:0 ds:302 act:2
176182025-01-23T18:51:34.713ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
176192025-01-23T18:51:34.713ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62975) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
176202025-01-23T18:51:34.714ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
176212025-01-23T18:51:34.715ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
176222025-01-23T18:51:34.715ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
176232025-01-23T18:51:34.715ZINFOcrucible: send last flush ID to this DS: Some(JobId(5527)) = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
176242025-01-23T18:51:34.715ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
176252025-01-23T18:51:34.715ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(5527)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
176262025-01-23T18:51:34.715ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(5527)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
17627 ----------------------------------------------------------------
17628 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17629 Downstairs last five completed: 5829 5828 5827 5826 5825
17630 Upstairs last five completed: 5829 5828 5827 5826 5825
17631 CLIENT: Up:0 ds:0 act:3
17632 Replay: All jobs finished
17633 Main task has work to do, go find it
17634 got dsc Stop(1)
17635 stop 1
17636 Jan 23 18:51:36.955 INFO request completed, latency_us[8820] Got stop action so:false kr:false
17637 : 311, response_code: 204, uri: /stop/cid/1, method: POST, req_id: d49797d0-c057-4206-b344-11fac0ef466d, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
176382025-01-23T18:51:36.956ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
17639 Jan 23 18:51:36.956 INFO request completed, latency_us: 130, response_code: 200, uri: /state/cid/1, method: GET, req_id: 11d2c9b1-8464-4261-a90a-1aa40b5fce71, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
176402025-01-23T18:51:36.956ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
176412025-01-23T18:51:36.956ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
176422025-01-23T18:51:36.956ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
17643 [8820] Exited with: ExitStatus(unix_wait_status(9))
17644 [8820][1] reports Exit
17645 Jan 23 18:51:40.958 INFO request completed, latency_us: 223, response_code: 200, uri: /state/cid/1, method: GET, req_id: d3510a8f-5f1c-43dc-ab3b-c086051c6673, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
17646 001/300 Write block 3371 len 12288 data: 3 3 3
17647 002/300 Read block 2340 len 32768
17648 003/300 Write block 17 len 4096 data: 4
17649 004/300 Write block 1164 len 4096 data: 3
17650 005/300 Read block 3060 len 4096
17651 006/300 Write block 2687 len 40960 data: 5 5 6 6 4 4 3 3 4 4
17652 007/300 Write block 1242 len 24576 data: 5 5 5 6 7 5
17653 008/300 Write block 4859 len 8192 data: 2 2
17654 009/300 Read block 3959 len 28672
17655 010/300 Write block 776 len 8192 data: 6 7
17656 011/300 Flush
17657 012/300 Read block 4224 len 4096
17658 013/300 Write block 4953 len 40960 data: 4 4 4 4 4 4 3 4 4 5
17659 014/300 Flush
17660 015/300 Write block 2418 len 24576 data: 5 4 3 3 4 4
17661 016/300 Flush
17662 017/300 Write block 4911 len 32768 data: 5 4 4 4 6 6 5 5
17663 018/300 Write block 3786 len 24576 data: 4 5 4 4 4 4
17664 019/300 Flush
17665 020/300 Write block 3872 len 16384 data: 3 3 3 4
17666 021/300 Write block 4495 len 16384 data: 5 4 4 4
17667 022/300 Flush
17668 023/300 Flush
17669 024/300 Write block 603 len 32768 data: 6 6 5 4 4 3 4 4
17670 025/300 Read block 185 len 36864
17671 026/300 Flush
17672 027/300 Read block 2067 len 16384
17673 028/300 Write block 2388 len 28672 data: 2 3 4 5 5 4 6
17674 029/300 Write block 823 len 28672 data: 8 5 4 5 4 4 4
17675 030/300 Write block 14 len 12288 data: 4 4 4
17676 031/300 Flush
17677 032/300 Write block 755 len 36864 data: 2 2 2 4 3 4 5 6 6
17678 033/300 Write block 4222 len 40960 data: 6 3 3 2 2 2 3 4 4 4
17679 034/300 Read block 2993 len 36864
17680 035/300 Read block 251 len 28672
17681 036/300 Read block 4436 len 8192
17682 037/300 Write block 2577 len 4096 data: 5
17683 038/300 Write block 1059 len 20480 data: 2 2 4 6 4
17684 039/300 Write block 2056 len 4096 data: 6
17685 040/300 Write block 540 len 8192 data: 3 4
17686 041/300 Read block 1129 len 8192
17687 042/300 Write block 1593 len 24576 data: 4 5 4 6 6 4
17688 043/300 Write block 3317 len 4096 data: 5
17689 044/300 Write block 1018 len 40960 data: 6 6 7 8 5 5 4 3 4 4
17690 045/300 Read block 4404 len 32768
17691 046/300 Read block 981 len 4096
17692 047/300 Read block 2806 len 20480
17693 048/300 Read block 3885 len 20480
17694 049/300 Write block 4628 len 16384 data: 5 4 4 3
17695 050/300 Read block 2504 len 8192
17696 051/300 Write block 649 len 36864 data: 4 5 4 4 4 5 5 5 4
17697 052/300 Read block 2697 len 36864
17698 053/300 Read block 3935 len 20480
17699 054/300 Read block 4560 len 4096
17700 055/300 Write block 3920 len 24576 data: 6 5 5 5 6 7
17701 056/300 Write block 2517 len 32768 data: 3 5 5 6 6 6 8 7
17702 057/300 Write block 4474 len 32768 data: 6 6 5 5 4 5 5 4
17703 058/300 Write block 4423 len 24576 data: 5 7 5 4 4 5
17704 059/300 Read block 1206 len 16384
17705 060/300 Read block 295 len 20480
17706 061/300 Read block 3734 len 16384
17707 062/300 Read block 1083 len 4096
17708 063/300 Read block 3284 len 40960
17709 064/300 Write block 890 len 20480 data: 3 3 3 4 5
17710 065/300 Write block 3690 len 28672 data: 5 6 5 6 4 4 4
17711 066/300 Read block 2564 len 24576
17712 067/300 Write block 138 len 16384 data: 6 5 7 6
17713 068/300 Flush
17714 069/300 Write block 1300 len 8192 data: 4 4
17715 070/300 Read block 2780 len 12288
17716 071/300 Write block 88 len 32768 data: 3 4 6 6 7 6 7 6
17717 072/300 Write block 1250 len 4096 data: 5
17718 073/300 Write block 4882 len 20480 data: 3 4 5 5 4
17719 074/300 Read block 4457 len 12288
17720 075/300 Write block 2123 len 36864 data: 5 5 4 4 4 5 5 5 4
17721 076/300 Write block 2201 len 4096 data: 6
17722 077/300 Flush
17723 078/300 Read block 707 len 40960
17724 079/300 Read block 556 len 36864
17725 080/300 Flush
17726 081/300 Read block 4116 len 8192
17727 082/300 Read block 4005 len 4096
17728 083/300 Write block 3912 len 12288 data: 3 4 4
17729 084/300 Read block 4208 len 32768
17730 085/300 Flush
17731 086/300 Read block 4435 len 32768
17732 087/300 Read block 4687 len 40960
17733 088/300 Flush
17734 089/300 Read block 3043 len 20480
17735 090/300 Write block 999 len 24576 data: 4 4 4 4 4 4
17736 091/300 Write block 2989 len 32768 data: 3 3 3 2 3 4 6 5
17737 092/300 Write block 1955 len 36864 data: 4 3 3 3 3 2 2 2 3
17738 093/300 Read block 4824 len 8192
17739 094/300 Write block 575 len 4096 data: 4
17740 095/300 Write block 3070 len 16384 data: 3 3 4 4
17741 096/300 Write block 61 len 12288 data: 3 3 3
17742 097/300 Flush
17743 098/300 Write block 882 len 4096 data: 4
17744 099/300 Write block 4166 len 28672 data: 5 6 6 5 6 7 7
17745 100/300 Read block 840 len 8192
17746 101/300 Read block 2931 len 36864
17747 102/300 Write block 4577 len 28672 data: 5 4 5 5 5 6 5
17748 103/300 Read block 4716 len 16384
17749 104/300 Read block 4818 len 36864
17750 105/300 Write block 245 len 40960 data: 3 3 6 6 5 4 5 6 5 5
17751 106/300 Write block 624 len 12288 data: 3 3 5
17752 107/300 Read block 2899 len 28672
17753 108/300 Write block 1737 len 16384 data: 5 6 5 5
17754 109/300 Write block 4492 len 24576 data: 4 4 5 6 5 5
17755 110/300 Write block 3589 len 28672 data: 4 4 4 4 4 4 4
17756 111/300 Read block 567 len 4096
17757 112/300 Flush
17758 113/300 Flush
17759 114/300 Read block 2277 len 32768
17760 115/300 Read block 648 len 36864
17761 116/300 Read block 4027 len 32768
17762 117/300 Write block 1477 len 16384 data: 3 3 3 2
17763 118/300 Write block 148 len 24576 data: 5 5 6 6 5 5
17764 119/300 Write block 3961 len 20480 data: 3 3 2 3 3
17765 120/300 Write block 2553 len 4096 data: 2
17766 121/300 Read block 3589 len 12288
17767 122/300 Flush
17768 123/300 Write block 3327 len 36864 data: 2 2 2 2 2 3 3 3 3
17769 124/300 Read block 4323 len 12288
17770 125/300 Write block 2224 len 8192 data: 3 3
17771 126/300 Read block 1458 len 4096
17772 127/300 Write block 1817 len 16384 data: 6 6 5 6
17773 128/300 Write block 1050 len 12288 data: 4 4 3
17774 129/300 Write block 3917 len 40960 data: 4 4 5 7 6 6 6 7 8 8
17775 130/300 Write block 4959 len 32768 data: 4 5 5 6 5 6 5 4
17776 131/300 Write block 2613 len 20480 data: 4 4 5 5 5
17777 132/300 Flush
17778 133/300 Read block 639 len 36864
17779 134/300 Read block 3229 len 36864
17780 135/300 Flush
17781 136/300 Write block 2795 len 28672 data: 4 3 4 5 4 3 3
17782 137/300 Read block 1675 len 8192
17783 138/300 Write block 1548 len 12288 data: 3 4 4
17784 139/300 Write block 2799 len 4096 data: 5
17785 140/300 Read block 4448 len 4096
17786 141/300 Write block 2086 len 12288 data: 3 2 2
17787 142/300 Write block 4891 len 40960 data: 2 2 2 2 3 4 3 3 3 2
17788 143/300 Read block 420 len 40960
17789 144/300 Write block 2106 len 4096 data: 6
17790 145/300 Read block 1044 len 4096
17791 146/300 Write block 2744 len 20480 data: 3 2 3 3 4
17792 147/300 Read block 3899 len 8192
17793 148/300 Read block 2978 len 40960
17794 149/300 Read block 4771 len 8192
17795 150/300 Write block 1460 len 8192 data: 2 4
17796 151/300 Write block 2918 len 40960 data: 4 5 2 6 5 5 5 5 5 6
17797 152/300 Read block 3245 len 24576
17798 153/300 Read block 2323 len 8192
17799 154/300 Write block 4631 len 40960 data: 4 3 3 4 4 3 4 3 4 4
17800 155/300 Read block 2167 len 16384
17801 156/300 Write block 4788 len 28672 data: 5 3 3 2 2 2 2
17802 157/300 Write block 4319 len 24576 data: 3 4 5 6 6 6
17803 158/300 Write block 1186 len 8192 data: 2 2
17804 159/300 Read block 3657 len 16384
17805 160/300 Read block 1337 len 24576
17806 161/300 Read block 888 len 16384
17807 162/300 Read block 1445 len 4096
17808 163/300 Read block 2121 len 4096
17809 164/300 Read block 2003 len 28672
17810 165/300 Read block 1956 len 12288
17811 166/300 Read block 4140 len 12288
17812 167/300 Read block 446 len 32768
17813 168/300 Read block 4885 len 40960
17814 169/300 Write block 3854 len 4096 data: 4
17815 170/300 Write block 2655 len 28672 data: 6 5 4 4 5 4 5
17816 171/300 Write block 3670 len 24576 data: 4 4 2 3 3 3
17817 172/300 Write block 111 len 4096 data: 6
17818 173/300 Flush
17819 174/300 Read block 827 len 16384
17820 175/300 Flush
17821 176/300 Read block 1111 len 28672
17822 177/300 Read block 1795 len 12288
17823 178/300 Flush
17824 179/300 Write block 95 len 40960 data: 7 6 6 3 3 2 2 3 4 3
17825 180/300 Read block 4457 len 20480
17826 181/300 Write block 1064 len 16384 data: 6 6 6 6
17827 182/300 Write block 308 len 32768 data: 5 6 6 6 6 6 5 3
17828 183/300 Write block 1007 len 20480 data: 3 3 5 5 5
17829 184/300 Read block 4549 len 8192
17830 185/300 Read block 1862 len 36864
17831 186/300 Write block 283 len 4096 data: 3
17832 187/300 Read block 3550 len 28672
17833 188/300 Write block 1617 len 28672 data: 3 2 2 3 5 6 6
17834 189/300 Write block 301 len 12288 data: 2 2 2
17835 190/300 Read block 956 len 12288
17836 191/300 Write block 721 len 4096 data: 2
17837 192/300 Read block 3722 len 20480
17838 193/300 Read block 4487 len 28672
17839 194/300 Write block 3890 len 16384 data: 4 4 4 4
17840 195/300 Write block 4092 len 36864 data: 6 6 5 5 5 5 4 4 3
17841 196/300 Read block 3244 len 4096
17842 197/300 Read block 2441 len 40960
17843 198/300 Write block 1566 len 12288 data: 4 5 5
17844 199/300 Flush
17845 200/300 Write block 3309 len 20480 data: 7 8 7 6 6
17846 201/300 Write block 292 len 28672 data: 5 3 3 3 2 2 2
17847 202/300 Write block 4795 len 12288 data: 2 2 2
17848 203/300 Write block 162 len 36864 data: 2 2 2 3 3 3 4 4 4
17849 204/300 Read block 3089 len 4096
17850 205/300 Read block 478 len 36864
17851 206/300 Read block 4601 len 8192
17852 207/300 Flush
17853 208/300 Write block 3772 len 36864 data: 3 4 4 4 4 4 5 4 4
17854 209/300 Read block 4176 len 28672
17855 210/300 Write block 4609 len 4096 data: 4
17856 211/300 Read block 1519 len 40960
17857 212/300 Read block 3264 len 16384
17858 213/300 Read block 1991 len 24576
17859 214/300 Read block 3151 len 24576
17860 215/300 Read block 1761 len 4096
17861 216/300 Write block 1702 len 32768 data: 3 5 5 5 5 3 3 4
17862 217/300 Write block 4110 len 28672 data: 2 3 3 3 3 3 3
17863 218/300 Read block 1850 len 16384
17864 219/300 Flush
17865 220/300 Write block 3873 len 40960 data: 4 4 5 5 5 6 6 7 6 5
17866 221/300 Write block 875 len 8192 data: 4 4
17867 222/300 Read block 918 len 12288
17868 223/300 Write block 3088 len 32768 data: 3 3 3 4 4 4 4 4
17869 224/300 Read block 3451 len 8192
17870 225/300 Write block 708 len 20480 data: 3 4 4 4 4
17871 226/300 Write block 887 len 4096 data: 4
17872 227/300 Write block 1175 len 24576 data: 4 4 4 3 2 2
17873 228/300 Write block 2013 len 20480 data: 6 6 6 5 5
17874 229/300 Write block 1219 len 4096 data: 5
17875 230/300 Read block 4013 len 12288
17876 231/300 Write block 4242 len 32768 data: 2 3 3 3 3 3 3 3
17877 232/300 Read block 1479 len 36864
17878 233/300 Read block 4393 len 20480
17879 234/300 Write block 3480 len 16384 data: 4 4 4 3
17880 235/300 Read block 1744 len 20480
17881 236/300 Read block 2861 len 36864
17882 237/300 Read block 4108 len 4096
17883 238/300 Read block 4616 len 32768
17884 239/300 Write block 2070 len 16384 data: 5 6 5 5
17885 240/300 Read block 1749 len 4096
17886 241/300 Read block 555 len 12288
17887 242/300 Read block 1340 len 40960
17888 243/300 Read block 3609 len 8192
17889 244/300 Write block 2601 len 12288 data: 4 5 6
17890 245/300 Write block 4254 len 28672 data: 5 6 6 7 8 7 4
17891 246/300 Write block 3329 len 12288 data: 3 3 3
17892 247/300 Write block 927 len 4096 data: 2
17893 248/300 Read block 773 len 32768
17894 249/300 Write block 1002 len 20480 data: 5 5 5 4 3
17895 250/300 Write block 2815 len 8192 data: 2 2
17896 251/300 Write block 1869 len 4096 data: 6
17897 252/300 Write block 1854 len 40960 data: 4 4 4 3 3 4 4 5 6 6
17898 253/300 Read block 2947 len 16384
17899 254/300 Read block 3530 len 28672
17900 255/300 Read block 2381 len 28672
17901 256/300 Read block 2266 len 20480
17902 257/300 Write block 4215 len 36864 data: 4 5 6 5 6 6 6 7 4
17903 258/300 Write block 2174 len 12288 data: 3 3 3
17904 259/300 Read block 4186 len 28672
17905 260/300 Write block 2241 len 28672 data: 3 3 3 3 3 3 2
17906 261/300 Flush
17907 262/300 Read block 4001 len 40960
17908 263/300 Write block 4747 len 4096 data: 4
17909 264/300 Read block 2289 len 16384
17910 265/300 Read block 3197 len 24576
17911 266/300 Write block 218 len 28672 data: 4 5 4 4 4 3 3
17912 267/300 Read block 130 len 12288
17913 268/300 Read block 3856 len 20480
17914 269/300 Read block 2501 len 4096
17915 270/300 Write block 2909 len 40960 data: 5 5 4 5 4 3 3 3 4 5
17916 271/300 Read block 213 len 16384
17917 272/300 Read block 3827 len 32768
17918 273/300 Read block 3497 len 16384
17919 274/300 Read block 4808 len 24576
17920 275/300 Write block 2755 len 24576 data: 3 3 4 4 5 5
17921 276/300 Read block 1867 len 20480
17922 277/300 Write block 4195 len 40960 data: 3 3 2 4 4 3 3 3 3 3
17923 278/300 Flush
17924 279/300 Write block 2840 len 32768 data: 4 4 3 3 4 4 6 5
17925 280/300 Write block 2714 len 4096 data: 3
17926 281/300 Write block 2374 len 4096 data: 2
17927 282/300 Read block 3441 len 8192
17928 283/300 Read block 977 len 36864
17929 284/300 Flush
17930 285/300 Write block 3375 len 40960 data: 3 3 3 3 4 4 3 3 3 3
17931 286/300 Write block 1245 len 12288 data: 7 8 6
17932 287/300 Read block 4786 len 4096
17933 288/300 Read block 759 len 8192
17934 289/300 Read block 1218 len 16384
17935 290/300 Read block 4031 len 4096
17936 291/300 Read block 3488 len 12288
17937 292/300 Write block 2012 len 12288 data: 6 7 7
17938 293/300 Write block 935 len 16384 data: 5 6 7 6
17939 294/300 Flush
17940 295/300 Write block 174 len 28672 data: 5 6 8 6 9 7 5
17941 296/300 Read block 1144 len 36864
17942 297/300 Write block 2232 len 12288 data: 3 3 3
17943 298/300 Read block 3741 len 28672
17944 299/300 Read block 2659 len 28672
17945 300/300 Write block 4454 len 8192 data: 5 5
17946 301/300 Write block 2828 len 20480 data: 2 2 3 4 4
17947 Main task has work to do, go find it
17948 got dsc Start(1)
17949 start 1
17950 Jan 23 18:51:41.179 INFO request completed, latency_us: 249, response_code: 204, uri: /start/cid/1, method: POST, req_id: 468f735b-7463-42cd-9e55-e3a2642a05df, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
17951 Replay: started 1, returned:Ok(())
17952 ----------------------------------------------------------------
17953 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17954 States: Active Offline Active
17955 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17956 5830 Acked Write 3 Done Sent Done false
17957 5831 Acked Read 8 Done Sent Done false
17958 5832 Acked Write 1 Done Sent Done false
17959 5833 Acked Write 1 Done Sent Done false
17960 5834 Acked Read 1 Done Sent Done false
17961 5835 Acked Write 10 Done Sent Done false
17962 5836 Acked Write 6 Done Sent Done false
17963 5837 Acked Write 2 Done Sent Done false
17964 5838 Acked Read 7 Done Sent Done false
17965 5839 Acked Write 2 Done Sent Done false
17966 5840 Acked Flush 0 Done Sent Done false
17967 5841 Acked Read 1 Done Sent Done false
17968 5842 Acked Write 10 Done Sent Done false
17969 5843 Acked Flush 0 Done Sent Done false
17970 5844 Acked Write 6 Done Sent Done false
17971 5845 Acked Flush 0 Done Sent Done false
17972 5846 Acked Write 8 Done Sent Done false
17973 5847 Acked Write 6 Done Sent Done false
17974 5848 Acked Flush 0 Done Sent Done false
17975 5849 Acked Write 4 Done Sent Done false
17976 5850 Acked Write 4 Done Sent Done false
17977 5851 Acked Flush 0 Done Sent Done false
17978 5852 Acked Flush 0 Done Sent Done false
17979 5853 Acked Write 8 Done Sent Done false
17980 5854 Acked Read 9 Done Sent Done false
17981 5855 Acked Flush 0 Done Sent Done false
17982 5856 Acked Read 4 Done Sent Done false
17983 5857 Acked Write 7 Done Sent Done false
17984 5858 Acked Write 7 Done Sent Done false
17985 5859 Acked Write 3 Done Sent Done false
17986 5860 Acked Flush 0 Done Sent Done false
17987 5861 Acked Write 9 Done Sent Done false
17988 5862 Acked Write 10 Done Sent Done false
17989 5863 Acked Read 9 Done Sent Done false
17990 5864 Acked Read 7 Done Sent Done false
17991 5865 Acked Read 2 Done Sent Done false
17992 5866 Acked Write 1 Done Sent Done false
17993 5867 Acked Write 5 Done Sent Done false
17994 5868 Acked Write 1 Done Sent Done false
17995 5869 Acked Write 2 Done Sent Done false
17996 5870 Acked Read 2 Done Sent Done false
17997 5871 Acked Write 6 Done Sent Done false
17998 5872 Acked Write 1 Done Sent Done false
17999 5873 Acked Write 10 Done Sent Done false
18000 5874 Acked Read 8 Done Sent Done false
18001 5875 Acked Read 1 Done Sent Done false
18002 5876 Acked Read 5 Done Sent Done false
18003 5877 Acked Read 5 Done Sent Done false
18004 5878 Acked Write 4 Done Sent Done false
18005 5879 Acked Read 2 Done Sent Done false
18006 5880 Acked Write 9 Done Sent Done false
18007 5881 Acked Read 9 Done Sent Done false
18008 5882 Acked Read 5 Done Sent Done false
18009 5883 Acked Read 1 Done Sent Done false
18010 5884 Acked Write 6 Done Sent Done false
18011 5885 Acked Write 8 Done Sent Done false
18012 5886 Acked Write 8 Done Sent Done false
18013 5887 Acked Write 6 Done Sent Done false
18014 5888 Acked Read 4 Done Sent Done false
18015 5889 Acked Read 5 Done Sent Done false
18016 5890 Acked Read 4 Done Sent Done false
18017 5891 Acked Read 1 Done Sent Done false
18018 5892 Acked Read 10 Done Sent Done false
18019 5893 Acked Write 5 Done Sent Done false
18020 5894 Acked Write 7 Done Sent Done false
18021 5895 Acked Read 6 Done Sent Done false
18022 5896 Acked Write 4 Done Sent Done false
18023 5897 Acked Flush 0 Done Sent Done false
18024 5898 Acked Write 2 Done Sent Done false
18025 5899 Acked Read 3 Done Sent Done false
18026 5900 Acked Write 8 Done Sent Done false
18027 5901 Acked Write 1 Done Sent Done false
18028 5902 Acked Write 5 Done Sent Done false
18029 5903 Acked Read 3 Done Sent Done false
18030 5904 Acked Write 9 Done Sent Done false
18031 5905 Acked Write 1 Done Sent Done false
18032 5906 Acked Flush 0 Done Sent Done false
18033 5907 Acked Read 10 Done Sent Done false
18034 5908 Acked Read 9 Done Sent Done false
18035 5909 Acked Flush 0 Done Sent Done false
18036 5910 Acked Read 2 Done Sent Done false
18037 5911 Acked Read 1 Done Sent Done false
18038 5912 Acked Write 3 Done Sent Done false
18039 5913 Acked Read 8 Done Sent Done false
18040 5914 Acked Flush 0 Done Sent Done false
18041 5915 Acked Read 8 Done Sent Done false
18042 5916 Acked Read 10 Done Sent Done false
18043 5917 Acked Flush 0 Done Sent Done false
18044 5918 Acked Read 5 Done Sent Done false
18045 5919 Acked Write 6 Done Sent Done false
18046 5920 Acked Write 8 Done Sent Done false
18047 5921 Acked Write 9 Done Sent Done false
18048 5922 Acked Read 2 Done Sent Done false
18049 5923 Acked Write 1 Done Sent Done false
18050 5924 Acked Write 4 Done Sent Done false
18051 5925 Acked Write 3 Done Sent Done false
18052 5926 Acked Flush 0 Done Sent Done false
18053 5927 Acked Write 1 Done Sent Done false
18054 5928 Acked Write 7 Done Sent Done false
18055 5929 Acked Read 2 Done Sent Done false
18056 5930 Acked Read 9 Done Sent Done false
18057 5931 Acked Write 7 Done Sent Done false
18058 5932 Acked Read 4 Done Sent Done false
18059 5933 Acked Read 9 Done Sent Done false
18060 5934 Acked Write 10 Done Sent Done false
18061 5935 Acked Write 3 Done Sent Done false
18062 5936 Acked Read 7 Done Sent Done false
18063 5937 Acked Write 4 Done Sent Done false
18064 5938 Acked Write 6 Done Sent Done false
18065 5939 Acked Write 7 Done Sent Done false
18066 5940 Acked Read 1 Done Sent Done false
18067 5941 Acked Flush 0 Done Sent Done false
18068 5942 Acked Flush 0 Done Sent Done false
18069 5943 Acked Read 8 Done Sent Done false
18070 5944 Acked Read 9 Done Sent Done false
18071 5945 Acked Read 8 Done Sent Done false
18072 5946 Acked Write 4 Done Sent Done false
18073 5947 Acked Write 6 Done Sent Done false
18074 5948 Acked Write 5 Done Sent Done false
18075 5949 Acked Write 1 Done Sent Done false
18076 5950 Acked Read 3 Done Sent Done false
18077 5951 Acked Flush 0 Done Sent Done false
18078 5952 Acked Write 9 Done Sent Done false
18079 5953 Acked Read 3 Done Sent Done false
18080 5954 Acked Write 2 Done Sent Done false
18081 5955 Acked Read 1 Done Sent Done false
18082 5956 Acked Write 4 Done Sent Done false
18083 5957 Acked Write 3 Done Sent Done false
18084 5958 Acked Write 10 Done Sent Done false
18085 5959 Acked Write 8 Done Sent Done false
18086 5960 Acked Write 5 Done Sent Done false
18087 5961 Acked Flush 0 Done Sent Done false
18088 5962 Acked Read 9 Done Sent Done false
18089 5963 Acked Read 9 Done Sent Done false
18090 5964 Acked Flush 0 Done Sent Done false
18091 5965 Acked Write 7 Done Sent Done false
18092 5966 Acked Read 2 Done Sent Done false
18093 5967 Acked Write 3 Done Sent Done false
18094 5968 Acked Write 1 Done Sent Done false
18095 5969 Acked Read 1 Done Sent Done false
18096 5970 Acked Write 3 Done Sent Done false
18097 5971 Acked Write 10 Done Sent Done false
18098 5972 Acked Read 10 Done Sent Done false
18099 5973 Acked Write 1 Done Sent Done false
18100 5974 Acked Read 1 Done Sent Done false
18101 5975 Acked Write 5 Done Sent Done false
18102 5976 Acked Read 2 Done Sent Done false
18103 5977 Acked Read 10 Done Sent Done false
18104 5978 Acked Read 2 Done Sent Done false
18105 5979 Acked Write 2 Done Sent Done false
18106 5980 Acked Write 10 Done Sent Done false
18107 5981 Acked Read 6 Done Sent Done false
18108 5982 Acked Read 2 Done Sent Done false
18109 5983 Acked Write 10 Done Sent Done false
18110 5984 Acked Read 4 Done Sent Done false
18111 5985 Acked Write 7 Done Sent Done false
18112 5986 Acked Write 6 Done Sent Done false
18113 5987 Acked Write 2 Done Sent Done false
18114 5988 Acked Read 4 Done Sent Done false
18115 5989 Acked Read 6 Done Sent Done false
18116 5990 Acked Read 4 Done Sent Done false
18117 5991 Acked Read 1 Done Sent Done false
18118 5992 Acked Read 1 Done Sent Done false
18119 5993 Acked Read 7 Done Sent Done false
18120 5994 Acked Read 3 Done Sent Done false
18121 5995 Acked Read 3 Done Sent Done false
18122 5996 Acked Read 8 Done Sent Done false
18123 5997 Acked Read 10 Done Sent Done false
18124 5998 Acked Write 1 Done Sent Done false
18125 5999 Acked Write 7 Done Sent Done false
18126 6000 Acked Write 6 Done Sent Done false
18127 6001 Acked Write 1 Done Sent Done false
18128 6002 Acked Flush 0 Done Sent Done false
18129 6003 Acked Read 4 Done Sent Done false
18130 6004 Acked Flush 0 Done Sent Done false
18131 6005 Acked Read 7 Done Sent Done false
18132 6006 Acked Read 3 Done Sent Done false
18133 6007 Acked Flush 0 Done Sent Done false
18134 6008 Acked Write 10 Done Sent Done false
18135 6009 Acked Read 5 Done Sent Done false
18136 6010 Acked Write 4 Done Sent Done false
18137 6011 Acked Write 8 Done Sent Done false
18138 6012 Acked Write 5 Done Sent Done false
18139 6013 Acked Read 2 Done Sent Done false
18140 6014 Acked Read 9 Done Sent Done false
18141 6015 Acked Write 1 Done Sent Done false
18142 6016 Acked Read 7 Done Sent Done false
18143 6017 Acked Write 7 Done Sent Done false
18144 6018 Acked Write 3 Done Sent Done false
18145 6019 Acked Read 3 Done Sent Done false
18146 6020 Acked Write 1 Done Sent Done false
18147 6021 Acked Read 5 Done Sent Done false
18148 6022 Acked Read 7 Done Sent Done false
18149 6023 Acked Write 4 Done Sent Done false
18150 6024 Acked Write 9 Done Sent Done false
18151 6025 Acked Read 1 Done Sent Done false
18152 6026 Acked Read 10 Done Sent Done false
18153 6027 Acked Write 3 Done Sent Done false
18154 6028 Acked Flush 0 Done Sent Done false
18155 6029 Acked Write 5 Done Sent Done false
18156 6030 Acked Write 7 Done Sent Done false
18157 6031 Acked Write 3 Done Sent Done false
18158 6032 Acked Write 9 Done Sent Done false
18159 6033 Acked Read 1 Done Sent Done false
18160 6034 Acked Read 9 Done Sent Done false
18161 6035 Acked Read 2 Done Sent Done false
18162 6036 Acked Flush 0 Done Sent Done false
18163 6037 Acked Write 9 Done Sent Done false
18164 6038 Acked Read 7 Done Sent Done false
18165 6039 Acked Write 1 Done Sent Done false
18166 6040 Acked Read 10 Done Sent Done false
18167 6041 Acked Read 4 Done Sent Done false
18168 6042 Acked Read 6 Done Sent Done false
18169 6043 Acked Read 6 Done Sent Done false
18170 6044 Acked Read 1 Done Sent Done false
18171 6045 Acked Write 8 Done Sent Done false
18172 6046 Acked Write 7 Done Sent Done false
18173 6047 Acked Read 4 Done Sent Done false
18174 6048 Acked Flush 0 Done Sent Done false
18175 6049 Acked Write 10 Done Sent Done false
18176 6050 Acked Write 2 Done Sent Done false
18177 6051 Acked Read 3 Done Sent Done false
18178 6052 Acked Write 8 Done Sent Done false
18179 6053 Acked Read 2 Done Sent Done false
18180 6054 Acked Write 5 Done Sent Done false
18181 6055 Acked Write 1 Done Sent Done false
18182 6056 Acked Write 6 Done Sent Done false
18183 6057 Acked Write 5 Done Sent Done false
18184 6058 Acked Write 1 Done Sent Done false
18185 6059 Acked Read 3 Done Sent Done false
18186 6060 Acked Write 8 Done Sent Done false
18187 6061 Acked Read 9 Done Sent Done false
18188 6062 Acked Read 5 Done Sent Done false
18189 6063 Acked Write 4 Done Sent Done false
18190 6064 Acked Read 5 Done Sent Done false
18191 6065 Acked Read 9 Done Sent Done false
18192 6066 Acked Read 1 Done Sent Done false
18193 6067 Acked Read 8 Done Sent Done false
18194 6068 Acked Write 4 Done Sent Done false
18195 6069 Acked Read 1 Done Sent Done false
18196 6070 Acked Read 3 Done Sent Done false
18197 6071 Acked Read 10 Done Sent Done false
18198 6072 Acked Read 2 Done Sent Done false
18199 6073 Acked Write 3 Done Sent Done false
18200 6074 Acked Write 7 Done Sent Done false
18201 6075 Acked Write 3 Done Sent Done false
18202 6076 Acked Write 1 Done Sent Done false
18203 6077 Acked Read 8 Done Sent Done false
18204 6078 Acked Write 5 Done Sent Done false
18205 6079 Acked Write 2 Done Sent Done false
18206 6080 Acked Write 1 Done Sent Done false
18207 6081 Acked Write 10 Done Sent Done false
18208 6082 Acked Read 4 Done Sent Done false
18209 6083 Acked Read 7 Done Sent Done false
18210 6084 Acked Read 7 Done Sent Done false
18211 6085 Acked Read 5 Done Sent Done false
18212 6086 Acked Write 9 Done Sent Done false
18213 6087 Acked Write 3 Done Sent Done false
18214 6088 Acked Read 7 Done Sent Done false
18215 6089 Acked Write 7 Done Sent Done false
18216 6090 Acked Flush 0 Done Sent Done false
18217 6091 Acked Read 10 Done Sent Done false
18218 6092 Acked Write 1 Done Sent Done false
18219 6093 Acked Read 4 Done Sent Done false
18220 6094 Acked Read 6 Done Sent Done false
18221 6095 Acked Write 7 Done Sent Done false
18222 6096 Acked Read 3 Done Sent Done false
18223 6097 Acked Read 5 Done Sent Done false
18224 6098 Acked Read 1 Done Sent Done false
18225 6099 Acked Write 10 Done Sent Done false
18226 6100 Acked Read 4 Done Sent Done false
18227 6101 Acked Read 8 Done Sent Done false
18228 6102 Acked Read 4 Done Sent Done false
18229 6103 Acked Read 6 Done Sent Done false
18230 6104 Acked Write 6 Done Sent Done false
18231 6105 Acked Read 5 Done Sent Done false
18232 6106 Acked Write 10 Done Sent Done false
18233 6107 Acked Flush 0 Done Sent Done false
18234 6108 Acked Write 8 Done Sent Done false
18235 6109 Acked Write 1 Done Sent Done false
18236 6110 Acked Write 1 Done Sent Done false
18237 6111 Acked Read 2 Done Sent Done false
18238 6112 Acked Read 9 Done Sent Done false
18239 6113 Acked Flush 0 Done Sent Done false
18240 6114 Acked Write 10 Done Sent Done false
18241 6115 Acked Write 3 Done Sent Done false
18242 6116 Acked Read 1 Done Sent Done false
18243 6117 Acked Read 2 Done Sent Done false
18244 6118 Acked Read 4 Done Sent Done false
18245 6119 Acked Read 1 Done Sent Done false
18246 6120 Acked Read 3 Done Sent Done false
18247 6121 Acked Write 3 Done Sent Done false
18248 6122 Acked Write 4 Done Sent Done false
18249 6123 Acked Flush 0 Done Sent Done false
18250 6124 Acked Write 7 Done Sent Done false
18251 6125 Acked Read 9 Done Sent Done false
18252 6126 Acked Write 3 Done Sent Done false
18253 6127 Acked Read 7 Done Sent Done false
18254 6128 Acked Read 7 Done Sent Done false
18255 6129 Acked Write 2 Done Sent Done false
18256 6130 Acked Write 5 Done Sent Done false
18257 STATES DS:0 DS:1 DS:2 TOTAL
18258 Sent 0 301 0 301
18259 Done 301 0 301 602
18260 Skipped 0 0 0 0
18261 Error 0 0 0 0
18262 Last Flush: Some(JobId(6123)) Some(JobId(5829)) Some(JobId(6123))
18263 Downstairs last five completed: 5829 5828 5827 5826 5825
18264 Upstairs last five completed: 6130 6129 6128 6127 6126
18265 CLIENT: Up:0 ds:301 act:2
18266 [8820] got start action
18267 [8820] I am going to restart
18268 Starting downstairs at port 8820
18269 Make output file at "/tmp/dsc/downstairs-8820.txt"
18270 [8820][1] reports Starting
18271 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1217)
18272 [8820][1] reports Running
18273 ----------------------------------------------------------------
18274 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
18275 States: Active Offline Active
18276 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18277 5830 Acked Write 3 Done Sent Done false
18278 5831 Acked Read 8 Done Sent Done false
18279 5832 Acked Write 1 Done Sent Done false
18280 5833 Acked Write 1 Done Sent Done false
18281 5834 Acked Read 1 Done Sent Done false
18282 5835 Acked Write 10 Done Sent Done false
18283 5836 Acked Write 6 Done Sent Done false
18284 5837 Acked Write 2 Done Sent Done false
18285 5838 Acked Read 7 Done Sent Done false
18286 5839 Acked Write 2 Done Sent Done false
18287 5840 Acked Flush 0 Done Sent Done false
18288 5841 Acked Read 1 Done Sent Done false
18289 5842 Acked Write 10 Done Sent Done false
18290 5843 Acked Flush 0 Done Sent Done false
18291 5844 Acked Write 6 Done Sent Done false
18292 5845 Acked Flush 0 Done Sent Done false
18293 5846 Acked Write 8 Done Sent Done false
18294 5847 Acked Write 6 Done Sent Done false
18295 5848 Acked Flush 0 Done Sent Done false
18296 5849 Acked Write 4 Done Sent Done false
18297 5850 Acked Write 4 Done Sent Done false
18298 5851 Acked Flush 0 Done Sent Done false
18299 5852 Acked Flush 0 Done Sent Done false
18300 5853 Acked Write 8 Done Sent Done false
18301 5854 Acked Read 9 Done Sent Done false
18302 5855 Acked Flush 0 Done Sent Done false
18303 5856 Acked Read 4 Done Sent Done false
18304 5857 Acked Write 7 Done Sent Done false
18305 5858 Acked Write 7 Done Sent Done false
18306 5859 Acked Write 3 Done Sent Done false
18307 5860 Acked Flush 0 Done Sent Done false
18308 5861 Acked Write 9 Done Sent Done false
18309 5862 Acked Write 10 Done Sent Done false
18310 5863 Acked Read 9 Done Sent Done false
18311 5864 Acked Read 7 Done Sent Done false
18312 5865 Acked Read 2 Done Sent Done false
18313 5866 Acked Write 1 Done Sent Done false
18314 5867 Acked Write 5 Done Sent Done false
18315 5868 Acked Write 1 Done Sent Done false
18316 5869 Acked Write 2 Done Sent Done false
18317 5870 Acked Read 2 Done Sent Done false
18318 5871 Acked Write 6 Done Sent Done false
18319 5872 Acked Write 1 Done Sent Done false
18320 5873 Acked Write 10 Done Sent Done false
18321 5874 Acked Read 8 Done Sent Done false
18322 5875 Acked Read 1 Done Sent Done false
18323 5876 Acked Read 5 Done Sent Done false
18324 5877 Acked Read 5 Done Sent Done false
18325 5878 Acked Write 4 Done Sent Done false
18326 5879 Acked Read 2 Done Sent Done false
18327 5880 Acked Write 9 Done Sent Done false
18328 5881 Acked Read 9 Done Sent Done false
18329 5882 Acked Read 5 Done Sent Done false
18330 5883 Acked Read 1 Done Sent Done false
18331 5884 Acked Write 6 Done Sent Done false
18332 5885 Acked Write 8 Done Sent Done false
18333 5886 Acked Write 8 Done Sent Done false
18334 5887 Acked Write 6 Done Sent Done false
18335 5888 Acked Read 4 Done Sent Done false
18336 5889 Acked Read 5 Done Sent Done false
18337 5890 Acked Read 4 Done Sent Done false
18338 5891 Acked Read 1 Done Sent Done false
18339 5892 Acked Read 10 Done Sent Done false
18340 5893 Acked Write 5 Done Sent Done false
18341 5894 Acked Write 7 Done Sent Done false
18342 5895 Acked Read 6 Done Sent Done false
18343 5896 Acked Write 4 Done Sent Done false
18344 5897 Acked Flush 0 Done Sent Done false
18345 5898 Acked Write 2 Done Sent Done false
18346 5899 Acked Read 3 Done Sent Done false
18347 5900 Acked Write 8 Done Sent Done false
18348 5901 Acked Write 1 Done Sent Done false
18349 5902 Acked Write 5 Done Sent Done false
18350 5903 Acked Read 3 Done Sent Done false
18351 5904 Acked Write 9 Done Sent Done false
18352 5905 Acked Write 1 Done Sent Done false
18353 5906 Acked Flush 0 Done Sent Done false
18354 5907 Acked Read 10 Done Sent Done false
18355 5908 Acked Read 9 Done Sent Done false
18356 5909 Acked Flush 0 Done Sent Done false
18357 5910 Acked Read 2 Done Sent Done false
18358 5911 Acked Read 1 Done Sent Done false
18359 5912 Acked Write 3 Done Sent Done false
18360 5913 Acked Read 8 Done Sent Done false
18361 5914 Acked Flush 0 Done Sent Done false
18362 5915 Acked Read 8 Done Sent Done false
18363 5916 Acked Read 10 Done Sent Done false
18364 5917 Acked Flush 0 Done Sent Done false
18365 5918 Acked Read 5 Done Sent Done false
18366 5919 Acked Write 6 Done Sent Done false
18367 5920 Acked Write 8 Done Sent Done false
18368 5921 Acked Write 9 Done Sent Done false
18369 5922 Acked Read 2 Done Sent Done false
18370 5923 Acked Write 1 Done Sent Done false
18371 5924 Acked Write 4 Done Sent Done false
18372 5925 Acked Write 3 Done Sent Done false
18373 5926 Acked Flush 0 Done Sent Done false
18374 5927 Acked Write 1 Done Sent Done false
18375 5928 Acked Write 7 Done Sent Done false
18376 5929 Acked Read 2 Done Sent Done false
18377 5930 Acked Read 9 Done Sent Done false
18378 5931 Acked Write 7 Done Sent Done false
18379 5932 Acked Read 4 Done Sent Done false
18380 5933 Acked Read 9 Done Sent Done false
18381 5934 Acked Write 10 Done Sent Done false
18382 5935 Acked Write 3 Done Sent Done false
18383 5936 Acked Read 7 Done Sent Done false
18384 5937 Acked Write 4 Done Sent Done false
18385 5938 Acked Write 6 Done Sent Done false
18386 5939 Acked Write 7 Done Sent Done false
18387 5940 Acked Read 1 Done Sent Done false
18388 5941 Acked Flush 0 Done Sent Done false
18389 5942 Acked Flush 0 Done Sent Done false
18390 5943 Acked Read 8 Done Sent Done false
18391 5944 Acked Read 9 Done Sent Done false
18392 5945 Acked Read 8 Done Sent Done false
18393 5946 Acked Write 4 Done Sent Done false
18394 5947 Acked Write 6 Done Sent Done false
18395 5948 Acked Write 5 Done Sent Done false
18396 5949 Acked Write 1 Done Sent Done false
18397 5950 Acked Read 3 Done Sent Done false
18398 5951 Acked Flush 0 Done Sent Done false
18399 5952 Acked Write 9 Done Sent Done false
18400 5953 Acked Read 3 Done Sent Done false
18401 5954 Acked Write 2 Done Sent Done false
18402 5955 Acked Read 1 Done Sent Done false
18403 5956 Acked Write 4 Done Sent Done false
18404 5957 Acked Write 3 Done Sent Done false
18405 5958 Acked Write 10 Done Sent Done false
18406 5959 Acked Write 8 Done Sent Done false
18407 5960 Acked Write 5 Done Sent Done false
18408 5961 Acked Flush 0 Done Sent Done false
18409 5962 Acked Read 9 Done Sent Done false
18410 5963 Acked Read 9 Done Sent Done false
18411 5964 Acked Flush 0 Done Sent Done false
18412 5965 Acked Write 7 Done Sent Done false
18413 5966 Acked Read 2 Done Sent Done false
18414 5967 Acked Write 3 Done Sent Done false
18415 5968 Acked Write 1 Done Sent Done false
18416 5969 Acked Read 1 Done Sent Done false
18417 5970 Acked Write 3 Done Sent Done false
18418 5971 Acked Write 10 Done Sent Done false
18419 5972 Acked Read 10 Done Sent Done false
18420 5973 Acked Write 1 Done Sent Done false
18421 5974 Acked Read 1 Done Sent Done false
18422 5975 Acked Write 5 Done Sent Done false
18423 5976 Acked Read 2 Done Sent Done false
18424 5977 Acked Read 10 Done Sent Done false
18425 5978 Acked Read 2 Done Sent Done false
18426 5979 Acked Write 2 Done Sent Done false
18427 5980 Acked Write 10 Done Sent Done false
18428 5981 Acked Read 6 Done Sent Done false
18429 5982 Acked Read 2 Done Sent Done false
18430 5983 Acked Write 10 Done Sent Done false
18431 5984 Acked Read 4 Done Sent Done false
18432 5985 Acked Write 7 Done Sent Done false
18433 5986 Acked Write 6 Done Sent Done false
18434 5987 Acked Write 2 Done Sent Done false
18435 5988 Acked Read 4 Done Sent Done false
18436 5989 Acked Read 6 Done Sent Done false
18437 5990 Acked Read 4 Done Sent Done false
18438 5991 Acked Read 1 Done Sent Done false
18439 5992 Acked Read 1 Done Sent Done false
18440 5993 Acked Read 7 Done Sent Done false
18441 5994 Acked Read 3 Done Sent Done false
18442 5995 Acked Read 3 Done Sent Done false
18443 5996 Acked Read 8 Done Sent Done false
18444 5997 Acked Read 10 Done Sent Done false
18445 5998 Acked Write 1 Done Sent Done false
18446 5999 Acked Write 7 Done Sent Done false
18447 6000 Acked Write 6 Done Sent Done false
18448 6001 Acked Write 1 Done Sent Done false
18449 6002 Acked Flush 0 Done Sent Done false
18450 6003 Acked Read 4 Done Sent Done false
18451 6004 Acked Flush 0 Done Sent Done false
18452 6005 Acked Read 7 Done Sent Done false
18453 6006 Acked Read 3 Done Sent Done false
18454 6007 Acked Flush 0 Done Sent Done false
18455 6008 Acked Write 10 Done Sent Done false
18456 6009 Acked Read 5 Done Sent Done false
18457 6010 Acked Write 4 Done Sent Done false
18458 6011 Acked Write 8 Done Sent Done false
18459 6012 Acked Write 5 Done Sent Done false
18460 6013 Acked Read 2 Done Sent Done false
18461 6014 Acked Read 9 Done Sent Done false
18462 6015 Acked Write 1 Done Sent Done false
18463 6016 Acked Read 7 Done Sent Done false
18464 6017 Acked Write 7 Done Sent Done false
18465 6018 Acked Write 3 Done Sent Done false
18466 6019 Acked Read 3 Done Sent Done false
18467 6020 Acked Write 1 Done Sent Done false
18468 6021 Acked Read 5 Done Sent Done false
18469 6022 Acked Read 7 Done Sent Done false
18470 6023 Acked Write 4 Done Sent Done false
18471 6024 Acked Write 9 Done Sent Done false
18472 6025 Acked Read 1 Done Sent Done false
18473 6026 Acked Read 10 Done Sent Done false
18474 6027 Acked Write 3 Done Sent Done false
18475 6028 Acked Flush 0 Done Sent Done false
18476 6029 Acked Write 5 Done Sent Done false
18477 6030 Acked Write 7 Done Sent Done false
18478 6031 Acked Write 3 Done Sent Done false
18479 6032 Acked Write 9 Done Sent Done false
18480 6033 Acked Read 1 Done Sent Done false
18481 6034 Acked Read 9 Done Sent Done false
18482 6035 Acked Read 2 Done Sent Done false
18483 6036 Acked Flush 0 Done Sent Done false
18484 6037 Acked Write 9 Done Sent Done false
18485 6038 Acked Read 7 Done Sent Done false
18486 6039 Acked Write 1 Done Sent Done false
18487 6040 Acked Read 10 Done Sent Done false
18488 6041 Acked Read 4 Done Sent Done false
18489 6042 Acked Read 6 Done Sent Done false
18490 6043 Acked Read 6 Done Sent Done false
18491 6044 Acked Read 1 Done Sent Done false
18492 6045 Acked Write 8 Done Sent Done false
18493 6046 Acked Write 7 Done Sent Done false
18494 6047 Acked Read 4 Done Sent Done false
18495 6048 Acked Flush 0 Done Sent Done false
18496 6049 Acked Write 10 Done Sent Done false
18497 6050 Acked Write 2 Done Sent Done false
18498 6051 Acked Read 3 Done Sent Done false
18499 6052 Acked Write 8 Done Sent Done false
18500 6053 Acked Read 2 Done Sent Done false
18501 6054 Acked Write 5 Done Sent Done false
18502 6055 Acked Write 1 Done Sent Done false
18503 6056 Acked Write 6 Done Sent Done false
18504 6057 Acked Write 5 Done Sent Done false
18505 6058 Acked Write 1 Done Sent Done false
18506 6059 Acked Read 3 Done Sent Done false
18507 6060 Acked Write 8 Done Sent Done false
18508 6061 Acked Read 9 Done Sent Done false
18509 6062 Acked Read 5 Done Sent Done false
18510 6063 Acked Write 4 Done Sent Done false
18511 6064 Acked Read 5 Done Sent Done false
18512 6065 Acked Read 9 Done Sent Done false
18513 6066 Acked Read 1 Done Sent Done false
18514 6067 Acked Read 8 Done Sent Done false
18515 6068 Acked Write 4 Done Sent Done false
18516 6069 Acked Read 1 Done Sent Done false
18517 6070 Acked Read 3 Done Sent Done false
18518 6071 Acked Read 10 Done Sent Done false
18519 6072 Acked Read 2 Done Sent Done false
18520 6073 Acked Write 3 Done Sent Done false
18521 6074 Acked Write 7 Done Sent Done false
18522 6075 Acked Write 3 Done Sent Done false
18523 6076 Acked Write 1 Done Sent Done false
18524 6077 Acked Read 8 Done Sent Done false
18525 6078 Acked Write 5 Done Sent Done false
18526 6079 Acked Write 2 Done Sent Done false
18527 6080 Acked Write 1 Done Sent Done false
18528 6081 Acked Write 10 Done Sent Done false
18529 6082 Acked Read 4 Done Sent Done false
18530 6083 Acked Read 7 Done Sent Done false
18531 6084 Acked Read 7 Done Sent Done false
18532 6085 Acked Read 5 Done Sent Done false
18533 6086 Acked Write 9 Done Sent Done false
18534 6087 Acked Write 3 Done Sent Done false
18535 6088 Acked Read 7 Done Sent Done false
18536 6089 Acked Write 7 Done Sent Done false
18537 6090 Acked Flush 0 Done Sent Done false
18538 6091 Acked Read 10 Done Sent Done false
18539 6092 Acked Write 1 Done Sent Done false
18540 6093 Acked Read 4 Done Sent Done false
18541 6094 Acked Read 6 Done Sent Done false
18542 6095 Acked Write 7 Done Sent Done false
18543 6096 Acked Read 3 Done Sent Done false
18544 6097 Acked Read 5 Done Sent Done false
18545 6098 Acked Read 1 Done Sent Done false
18546 6099 Acked Write 10 Done Sent Done false
18547 6100 Acked Read 4 Done Sent Done false
18548 6101 Acked Read 8 Done Sent Done false
18549 6102 Acked Read 4 Done Sent Done false
18550 6103 Acked Read 6 Done Sent Done false
18551 6104 Acked Write 6 Done Sent Done false
18552 6105 Acked Read 5 Done Sent Done false
18553 6106 Acked Write 10 Done Sent Done false
18554 6107 Acked Flush 0 Done Sent Done false
18555 6108 Acked Write 8 Done Sent Done false
18556 6109 Acked Write 1 Done Sent Done false
18557 6110 Acked Write 1 Done Sent Done false
18558 6111 Acked Read 2 Done Sent Done false
18559 6112 Acked Read 9 Done Sent Done false
18560 6113 Acked Flush 0 Done Sent Done false
18561 6114 Acked Write 10 Done Sent Done false
18562 6115 Acked Write 3 Done Sent Done false
18563 6116 Acked Read 1 Done Sent Done false
18564 6117 Acked Read 2 Done Sent Done false
18565 6118 Acked Read 4 Done Sent Done false
18566 6119 Acked Read 1 Done Sent Done false
18567 6120 Acked Read 3 Done Sent Done false
18568 6121 Acked Write 3 Done Sent Done false
18569 6122 Acked Write 4 Done Sent Done false
18570 6123 Acked Flush 0 Done Sent Done false
18571 6124 Acked Write 7 Done Sent Done false
18572 6125 Acked Read 9 Done Sent Done false
18573 6126 Acked Write 3 Done Sent Done false
18574 6127 Acked Read 7 Done Sent Done false
18575 6128 Acked Read 7 Done Sent Done false
18576 6129 Acked Write 2 Done Sent Done false
18577 6130 Acked Write 5 Done Sent Done false
18578 6131 Acked Flush 0 Done Sent Done false
18579 STATES DS:0 DS:1 DS:2 TOTAL
18580 Sent 0 302 0 302
18581 Done 302 0 302 604
18582 Skipped 0 0 0 0
18583 Error 0 0 0 0
18584 Last Flush: Some(JobId(6131)) Some(JobId(5829)) Some(JobId(6131))
18585 Downstairs last five completed: 5829 5828 5827 5826 5825
18586 Upstairs last five completed: 6131 6130 6129 6128 6127
18587 CLIENT: Up:0 ds:302 act:2
185882025-01-23T18:51:46.957ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
185892025-01-23T18:51:46.957ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49585) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
185902025-01-23T18:51:46.958ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
185912025-01-23T18:51:46.959ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
185922025-01-23T18:51:46.959ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
185932025-01-23T18:51:46.959ZINFOcrucible: send last flush ID to this DS: Some(JobId(5829)) = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
185942025-01-23T18:51:46.959ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
185952025-01-23T18:51:46.959ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(5829)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
185962025-01-23T18:51:46.959ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(5829)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
18597 ----------------------------------------------------------------
18598 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18599 Downstairs last five completed: 6131 6130 6129 6128 6127
18600 Upstairs last five completed: 6131 6130 6129 6128 6127
18601 CLIENT: Up:0 ds:0 act:3
18602 Replay: All jobs finished
18603 Main task has work to do, go find it
18604 got dsc Stop(2)
18605 stop 2
18606 Jan 23 18:51:49.185 INFO request completed, latency_us: 320, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 41c93a75-1d53-4493-ac25-7b622938d699, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
18607 [8830] Got stop action so:false kr:false
18608 Jan 23 18:51:49.185 INFO request completed, latency_us: 151, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2821d868-2b0d-43c2-a35a-11f4677cd49b, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
186092025-01-23T18:51:49.185ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
186102025-01-23T18:51:49.185ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
186112025-01-23T18:51:49.185ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
186122025-01-23T18:51:49.185ZINFOcrucible: client task is exiting = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
18613 [8830] Exited with: ExitStatus(unix_wait_status(9))
18614 [8830][2] reports Exit
18615 Jan 23 18:51:53.188 INFO request completed, latency_us: 199, response_code: 200, uri: /state/cid/2, method: GET, req_id: d761e33a-1256-4d5e-ae12-d8235e8e8663, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
18616 001/300 Write block 57 len 32768 data: 2 2 2 3 4 4 4 4
18617 002/300 Write block 2236 len 24576 data: 2 2 3 3 3 4
18618 003/300 Write block 2087 len 8192 data: 3 3
18619 004/300 Write block 1887 len 4096 data: 3
18620 005/300 Write block 4562 len 28672 data: 5 5 5 4 4 4 5
18621 006/300 Write block 2119 len 36864 data: 3 4 5 5 6 6 5 5 5
18622 007/300 Read block 1947 len 28672
18623 008/300 Write block 3323 len 40960 data: 4 3 2 2 3 3 4 4 4 4
18624 009/300 Write block 4016 len 16384 data: 5 2 2 3
18625 010/300 Read block 714 len 36864
18626 011/300 Read block 880 len 36864
18627 012/300 Read block 4615 len 36864
18628 013/300 Read block 336 len 36864
18629 014/300 Read block 4836 len 12288
18630 015/300 Read block 4625 len 40960
18631 016/300 Read block 3024 len 8192
18632 017/300 Read block 3941 len 32768
18633 018/300 Write block 762 len 4096 data: 7
18634 019/300 Flush
18635 020/300 Read block 1291 len 4096
18636 021/300 Write block 1922 len 36864 data: 4 5 4 4 3 3 3 4 4
18637 022/300 Write block 2337 len 32768 data: 4 4 4 5 3 4 4 4
18638 023/300 Read block 1712 len 20480
18639 024/300 Write block 1054 len 20480 data: 2 2 2 3 3
18640 025/300 Read block 2814 len 24576
18641 026/300 Read block 2113 len 4096
18642 027/300 Read block 154 len 12288
18643 028/300 Read block 2373 len 24576
18644 029/300 Read block 468 len 32768
18645 030/300 Read block 4962 len 24576
18646 031/300 Read block 4066 len 24576
18647 032/300 Read block 40 len 40960
18648 033/300 Read block 1448 len 20480
18649 034/300 Write block 3747 len 36864 data: 4 4 4 3 3 4 5 5 5
18650 035/300 Write block 3454 len 4096 data: 3
18651 036/300 Read block 3096 len 32768
18652 037/300 Write block 4334 len 36864 data: 2 2 4 4 4 4 6 6 6
18653 038/300 Read block 4249 len 16384
18654 039/300 Write block 736 len 12288 data: 4 5 5
18655 040/300 Write block 2677 len 20480 data: 6 6 5 5 5
18656 041/300 Read block 4336 len 24576
18657 042/300 Write block 2602 len 20480 data: 6 7 6 7 7
18658 043/300 Read block 4612 len 28672
18659 044/300 Read block 2120 len 32768
18660 045/300 Write block 365 len 32768 data: 4 3 4 3 2 2 3 5
18661 046/300 Write block 2749 len 36864 data: 4 4 4 4 3 3 4 4 5
18662 047/300 Write block 1417 len 32768 data: 6 9 9 9 9 9 8 7
18663 048/300 Write block 2795 len 20480 data: 5 4 5 6 6
18664 049/300 Read block 2342 len 20480
18665 050/300 Write block 924 len 24576 data: 3 2 2 3 2 2
18666 051/300 Read block 2804 len 28672
18667 052/300 Write block 2540 len 16384 data: 9 7 6 5
18668 053/300 Write block 4892 len 32768 data: 3 3 3 4 5 4 4 4
18669 054/300 Write block 289 len 20480 data: 6 6 7 6 4
18670 055/300 Read block 1403 len 16384
18671 056/300 Read block 4317 len 20480
18672 057/300 Write block 1533 len 20480 data: 4 5 3 2 2
18673 058/300 Read block 865 len 20480
18674 059/300 Read block 1337 len 24576
18675 060/300 Flush
18676 061/300 Read block 2962 len 24576
18677 062/300 Write block 1980 len 12288 data: 4 5 3
18678 063/300 Read block 1790 len 16384
18679 064/300 Write block 4559 len 16384 data: 4 4 5 6
18680 065/300 Flush
18681 066/300 Write block 4634 len 20480 data: 5 5 4 5 4
18682 067/300 Write block 4830 len 16384 data: 4 5 4 5
18683 068/300 Flush
18684 069/300 Read block 1396 len 32768
18685 070/300 Read block 3664 len 28672
18686 071/300 Write block 820 len 36864 data: 6 7 8 9 6 5 6 5 5
18687 072/300 Read block 4236 len 32768
18688 073/300 Read block 4152 len 20480
18689 074/300 Write block 2167 len 40960 data: 3 3 3 2 2 2 2 4 4 4
18690 075/300 Flush
18691 076/300 Read block 2725 len 12288
18692 077/300 Read block 3479 len 8192
18693 078/300 Read block 1829 len 8192
18694 079/300 Write block 1049 len 40960 data: 7 5 5 4 2 3 3 3 4 4
18695 080/300 Write block 2287 len 8192 data: 4 4
18696 081/300 Write block 4502 len 32768 data: 5 3 4 4 4 4 3 3
18697 082/300 Read block 3758 len 40960
18698 083/300 Flush
18699 084/300 Read block 4074 len 20480
18700 085/300 Read block 563 len 20480
18701 086/300 Read block 179 len 8192
18702 087/300 Read block 2044 len 40960
18703 088/300 Read block 1487 len 4096
18704 089/300 Read block 3076 len 32768
18705 090/300 Write block 786 len 36864 data: 6 6 7 5 4 4 3 3 3
18706 091/300 Write block 3743 len 40960 data: 4 4 5 4 5 5 5 4 4 5
18707 092/300 Read block 4332 len 4096
18708 093/300 Read block 612 len 16384
18709 094/300 Read block 4990 len 28672
18710 095/300 Read block 4387 len 12288
18711 096/300 Write block 1541 len 28672 data: 2 2 3 3 3 3 3
18712 097/300 Flush
18713 098/300 Read block 1708 len 28672
18714 099/300 Write block 4805 len 36864 data: 3 4 5 6 6 6 5 5 5
18715 100/300 Write block 4413 len 24576 data: 3 3 3 4 4 4
18716 101/300 Read block 2861 len 32768
18717 102/300 Write block 2305 len 36864 data: 5 6 6 5 5 5 5 5 7
18718 103/300 Read block 1581 len 24576
18719 104/300 Read block 2161 len 4096
18720 105/300 Write block 1490 len 40960 data: 5 6 5 4 3 3 3 4 3 4
18721 106/300 Write block 4732 len 28672 data: 3 4 4 3 3 3 2
18722 107/300 Flush
18723 108/300 Write block 796 len 12288 data: 3 3 3
18724 109/300 Write block 3640 len 4096 data: 6
18725 110/300 Flush
18726 111/300 Write block 3363 len 12288 data: 7 6 5
18727 112/300 Read block 856 len 28672
18728 113/300 Flush
18729 114/300 Write block 4628 len 40960 data: 6 5 5 5 4 4 6 6 5 6
18730 115/300 Write block 2079 len 40960 data: 7 6 6 5 5 4 4 4 4 4
18731 116/300 Read block 1217 len 36864
18732 117/300 Read block 955 len 4096
18733 118/300 Read block 2443 len 4096
18734 119/300 Read block 1030 len 28672
18735 120/300 Read block 2221 len 20480
18736 121/300 Flush
18737 122/300 Write block 3664 len 8192 data: 3 4
18738 123/300 Read block 1084 len 36864
18739 124/300 Read block 927 len 32768
18740 125/300 Write block 446 len 36864 data: 2 2 2 2 3 3 4 4 5
18741 126/300 Read block 3626 len 4096
18742 127/300 Read block 2341 len 36864
18743 128/300 Flush
18744 129/300 Read block 4934 len 8192
18745 130/300 Read block 4429 len 40960
18746 131/300 Read block 3840 len 36864
18747 132/300 Write block 3163 len 24576 data: 6 5 5 5 6 5
18748 133/300 Read block 4583 len 40960
18749 134/300 Read block 2657 len 28672
18750 135/300 Write block 4164 len 40960 data: 8 6 6 7 7 6 7 8 8 6
18751 136/300 Flush
18752 137/300 Read block 259 len 32768
18753 138/300 Read block 2001 len 12288
18754 139/300 Read block 299 len 24576
18755 140/300 Write block 3806 len 12288 data: 6 5 5
18756 141/300 Write block 815 len 36864 data: 4 4 5 5 5 7 8 9 10
18757 142/300 Read block 1132 len 20480
18758 143/300 Read block 4065 len 4096
18759 144/300 Flush
18760 145/300 Read block 1582 len 36864
18761 146/300 Read block 3663 len 40960
18762 147/300 Read block 3477 len 16384
18763 148/300 Write block 3145 len 28672 data: 3 2 2 2 3 3 2
18764 149/300 Read block 1462 len 16384
18765 150/300 Write block 4852 len 40960 data: 4 4 3 3 3 2 2 3 3 2
18766 151/300 Read block 1605 len 8192
18767 152/300 Flush
18768 153/300 Write block 2254 len 4096 data: 3
18769 154/300 Write block 54 len 4096 data: 2
18770 155/300 Flush
18771 156/300 Read block 2561 len 16384
18772 157/300 Write block 4646 len 28672 data: 6 7 5 4 4 4 3
18773 158/300 Write block 396 len 28672 data: 5 5 5 5 4 4 3
18774 159/300 Read block 2234 len 36864
18775 160/300 Write block 4902 len 20480 data: 3 3 4 4 4
18776 161/300 Flush
18777 162/300 Write block 860 len 24576 data: 2 2 3 4 4 4
18778 163/300 Read block 4504 len 28672
18779 164/300 Flush
18780 165/300 Read block 3578 len 24576
18781 166/300 Read block 1643 len 36864
18782 167/300 Read block 4102 len 40960
18783 168/300 Read block 4662 len 40960
18784 169/300 Write block 4905 len 28672 data: 5 5 3 3 3 3 6
18785 170/300 Read block 2915 len 24576
18786 171/300 Write block 1954 len 24576 data: 4 5 4 4 4 4
18787 172/300 Write block 237 len 24576 data: 3 3 3 3 3 3
18788 173/300 Write block 3221 len 24576 data: 4 4 4 5 5 4
18789 174/300 Write block 3068 len 16384 data: 4 4 4 4
18790 175/300 Write block 3800 len 32768 data: 5 6 6 5 5 4 7 6
18791 176/300 Read block 4689 len 28672
18792 177/300 Write block 4365 len 12288 data: 3 3 3
18793 178/300 Read block 2984 len 28672
18794 179/300 Write block 3272 len 24576 data: 4 4 4 4 4 4
18795 180/300 Read block 1810 len 32768
18796 181/300 Write block 1558 len 12288 data: 5 6 5
18797 182/300 Read block 2400 len 40960
18798 183/300 Read block 1070 len 16384
18799 184/300 Read block 3405 len 40960
18800 185/300 Read block 871 len 32768
18801 186/300 Write block 3362 len 16384 data: 7 8 7 6
18802 187/300 Read block 519 len 40960
18803 188/300 Read block 1377 len 40960
18804 189/300 Write block 3829 len 12288 data: 4 4 4
18805 190/300 Read block 667 len 12288
18806 191/300 Write block 1229 len 28672 data: 3 3 5 5 5 7 6
18807 192/300 Write block 2590 len 8192 data: 5 5
18808 193/300 Read block 1588 len 28672
18809 194/300 Read block 1746 len 16384
18810 195/300 Read block 660 len 12288
18811 196/300 Write block 1023 len 32768 data: 6 5 4 5 5 3 3 5
18812 197/300 Write block 1660 len 36864 data: 5 5 4 5 4 3 4 4 4
18813 198/300 Write block 4188 len 40960 data: 3 3 3 4 3 3 3 4 4 3
18814 199/300 Read block 2489 len 36864
18815 200/300 Read block 2793 len 24576
18816 201/300 Read block 2566 len 40960
18817 202/300 Flush
18818 203/300 Write block 600 len 24576 data: 6 6 9 7 7 6
18819 204/300 Read block 1345 len 36864
18820 205/300 Write block 2606 len 32768 data: 8 7 7 5 5 5 4 5
18821 206/300 Write block 3827 len 24576 data: 4 4 5 5 5 4
18822 207/300 Read block 860 len 28672
18823 208/300 Read block 991 len 16384
18824 209/300 Read block 4246 len 4096
18825 210/300 Write block 2822 len 20480 data: 4 4 4 3 2
18826 211/300 Write block 951 len 32768 data: 2 2 4 4 5 5 6 7
18827 212/300 Read block 1387 len 8192
18828 213/300 Read block 4656 len 4096
18829 214/300 Write block 1858 len 40960 data: 4 5 5 6 7 7 7 6 6 7
18830 215/300 Write block 3021 len 36864 data: 5 4 4 5 5 5 4 4 4
18831 216/300 Write block 4004 len 4096 data: 3
18832 217/300 Write block 624 len 16384 data: 4 4 6 3
18833 218/300 Read block 839 len 32768
18834 219/300 Write block 2233 len 4096 data: 4
18835 220/300 Write block 2448 len 28672 data: 7 7 6 5 5 4 4
18836 221/300 Write block 4805 len 28672 data: 4 5 6 7 7 7 6
18837 222/300 Write block 4025 len 32768 data: 4 3 4 6 7 7 7 5
18838 223/300 Read block 4214 len 32768
18839 224/300 Flush
18840 225/300 Read block 2943 len 8192
18841 226/300 Read block 4265 len 8192
18842 227/300 Read block 4064 len 8192
18843 228/300 Write block 1141 len 28672 data: 6 5 5 5 7 7 6
18844 229/300 Write block 4598 len 36864 data: 5 5 4 4 3 3 3 3 3
18845 230/300 Read block 4769 len 16384
18846 231/300 Read block 4776 len 8192
18847 232/300 Write block 2489 len 20480 data: 4 5 5 5 4
18848 233/300 Write block 266 len 4096 data: 4
18849 234/300 Read block 3511 len 36864
18850 235/300 Read block 3039 len 40960
18851 236/300 Read block 4828 len 20480
18852 237/300 Write block 3570 len 36864 data: 5 4 4 4 5 6 4 5 4
18853 238/300 Read block 1362 len 4096
18854 239/300 Write block 2067 len 24576 data: 3 3 6 6 7 6
18855 240/300 Write block 4988 len 24576 data: 5 3 3 3 3 3
18856 241/300 Read block 3400 len 4096
18857 242/300 Write block 4605 len 20480 data: 4 4 3 3 5
18858 243/300 Read block 3892 len 4096
18859 244/300 Read block 2438 len 12288
18860 245/300 Write block 412 len 28672 data: 4 6 6 8 8 8 8
18861 246/300 Write block 2323 len 20480 data: 5 6 6 5 5
18862 247/300 Read block 933 len 20480
18863 248/300 Flush
18864 249/300 Write block 2185 len 32768 data: 4 3 2 2 2 2 2 2
18865 250/300 Read block 1952 len 12288
18866 251/300 Read block 1664 len 36864
18867 252/300 Read block 379 len 24576
18868 253/300 Write block 3557 len 8192 data: 4 5
18869 254/300 Read block 684 len 16384
18870 255/300 Read block 776 len 32768
18871 256/300 Flush
18872 257/300 Read block 3831 len 36864
18873 258/300 Flush
18874 259/300 Write block 3991 len 20480 data: 3 3 3 4 4
18875 260/300 Write block 1214 len 16384 data: 5 5 5 5
18876 261/300 Read block 4877 len 8192
18877 262/300 Flush
18878 263/300 Read block 2107 len 12288
18879 264/300 Read block 729 len 28672
18880 265/300 Read block 597 len 32768
18881 266/300 Write block 4178 len 40960 data: 5 3 2 2 3 3 3 2 2 3
18882 267/300 Read block 4681 len 24576
18883 268/300 Read block 4838 len 36864
18884 269/300 Write block 2625 len 36864 data: 6 6 4 6 5 4 3 5 5
18885 270/300 Write block 4937 len 8192 data: 4 4
18886 271/300 Flush
18887 272/300 Read block 438 len 24576
18888 273/300 Write block 1186 len 32768 data: 3 3 2 2 2 5 4 5
18889 274/300 Read block 1040 len 36864
18890 275/300 Read block 1564 len 20480
18891 276/300 Read block 1502 len 20480
18892 277/300 Write block 356 len 16384 data: 4 3 3 2
18893 278/300 Write block 4361 len 20480 data: 4 4 4 4 4
18894 279/300 Read block 4116 len 8192
18895 280/300 Write block 3947 len 4096 data: 4
18896 281/300 Write block 3092 len 28672 data: 5 5 5 5 3 3 3
18897 282/300 Write block 3707 len 24576 data: 3 4 4 3 5 6
18898 283/300 Write block 4005 len 24576 data: 3 4 5 5 4 4
18899 284/300 Write block 520 len 40960 data: 4 4 4 3 4 5 7 8 8 7
18900 285/300 Read block 2432 len 36864
18901 286/300 Write block 1465 len 4096 data: 8
18902 287/300 Read block 3331 len 12288
18903 288/300 Flush
18904 289/300 Read block 4270 len 36864
18905 290/300 Read block 4703 len 40960
18906 291/300 Read block 635 len 12288
18907 292/300 Write block 978 len 40960 data: 3 4 4 5 5 5 6 7 7 5
18908 293/300 Read block 1495 len 40960
18909 294/300 Read block 593 len 28672
18910 295/300 Read block 2387 len 8192
18911 296/300 Write block 1495 len 16384 data: 4 4 5 4
18912 297/300 Read block 64 len 24576
18913 298/300 Flush
18914 299/300 Write block 77 len 16384 data: 6 7 6 4
18915 300/300 Write block 3422 len 20480 data: 8 9 8 7 5
18916 301/300 Flush
18917 Main task has work to do, go find it
18918 got dsc Start(2)
18919 start 2
18920 Jan 23 18:51:53.437 INFO request completed, latency_us: 195, response_code: 204, uri: /start/cid/2, method: POST, req_id: a7026b63-1ded-4782-bc89-47ff01170e21, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
18921 Replay: started 2, returned:Ok(())
18922 ----------------------------------------------------------------
18923 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18924 States: Active Active Offline
18925 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18926 6132 Acked Write 8 Done Done Sent false
18927 6133 Acked Write 6 Done Done Sent false
18928 6134 Acked Write 2 Done Done Sent false
18929 6135 Acked Write 1 Done Done Sent false
18930 6136 Acked Write 7 Done Done Sent false
18931 6137 Acked Write 9 Done Done Sent false
18932 6138 Acked Read 7 Done Done Sent false
18933 6139 Acked Write 10 Done Done Sent false
18934 6140 Acked Write 4 Done Done Sent false
18935 6141 Acked Read 9 Done Done Sent false
18936 6142 Acked Read 9 Done Done Sent false
18937 6143 Acked Read 9 Done Done Sent false
18938 6144 Acked Read 9 Done Done Sent false
18939 6145 Acked Read 3 Done Done Sent false
18940 6146 Acked Read 10 Done Done Sent false
18941 6147 Acked Read 2 Done Done Sent false
18942 6148 Acked Read 8 Done Done Sent false
18943 6149 Acked Write 1 Done Done Sent false
18944 6150 Acked Flush 0 Done Done Sent false
18945 6151 Acked Read 1 Done Done Sent false
18946 6152 Acked Write 9 Done Done Sent false
18947 6153 Acked Write 8 Done Done Sent false
18948 6154 Acked Read 5 Done Done Sent false
18949 6155 Acked Write 5 Done Done Sent false
18950 6156 Acked Read 6 Done Done Sent false
18951 6157 Acked Read 1 Done Done Sent false
18952 6158 Acked Read 3 Done Done Sent false
18953 6159 Acked Read 6 Done Done Sent false
18954 6160 Acked Read 8 Done Done Sent false
18955 6161 Acked Read 6 Done Done Sent false
18956 6162 Acked Read 6 Done Done Sent false
18957 6163 Acked Read 10 Done Done Sent false
18958 6164 Acked Read 5 Done Done Sent false
18959 6165 Acked Write 9 Done Done Sent false
18960 6166 Acked Write 1 Done Done Sent false
18961 6167 Acked Read 8 Done Done Sent false
18962 6168 Acked Write 9 Done Done Sent false
18963 6169 Acked Read 4 Done Done Sent false
18964 6170 Acked Write 3 Done Done Sent false
18965 6171 Acked Write 5 Done Done Sent false
18966 6172 Acked Read 6 Done Done Sent false
18967 6173 Acked Write 5 Done Done Sent false
18968 6174 Acked Read 7 Done Done Sent false
18969 6175 Acked Read 8 Done Done Sent false
18970 6176 Acked Write 8 Done Done Sent false
18971 6177 Acked Write 9 Done Done Sent false
18972 6178 Acked Write 8 Done Done Sent false
18973 6179 Acked Write 5 Done Done Sent false
18974 6180 Acked Read 5 Done Done Sent false
18975 6181 Acked Write 6 Done Done Sent false
18976 6182 Acked Read 7 Done Done Sent false
18977 6183 Acked Write 4 Done Done Sent false
18978 6184 Acked Write 8 Done Done Sent false
18979 6185 Acked Write 5 Done Done Sent false
18980 6186 Acked Read 4 Done Done Sent false
18981 6187 Acked Read 5 Done Done Sent false
18982 6188 Acked Write 5 Done Done Sent false
18983 6189 Acked Read 5 Done Done Sent false
18984 6190 Acked Read 6 Done Done Sent false
18985 6191 Acked Flush 0 Done Done Sent false
18986 6192 Acked Read 6 Done Done Sent false
18987 6193 Acked Write 3 Done Done Sent false
18988 6194 Acked Read 4 Done Done Sent false
18989 6195 Acked Write 4 Done Done Sent false
18990 6196 Acked Flush 0 Done Done Sent false
18991 6197 Acked Write 5 Done Done Sent false
18992 6198 Acked Write 4 Done Done Sent false
18993 6199 Acked Flush 0 Done Done Sent false
18994 6200 Acked Read 8 Done Done Sent false
18995 6201 Acked Read 7 Done Done Sent false
18996 6202 Acked Write 9 Done Done Sent false
18997 6203 Acked Read 8 Done Done Sent false
18998 6204 Acked Read 5 Done Done Sent false
18999 6205 Acked Write 10 Done Done Sent false
19000 6206 Acked Flush 0 Done Done Sent false
19001 6207 Acked Read 3 Done Done Sent false
19002 6208 Acked Read 2 Done Done Sent false
19003 6209 Acked Read 2 Done Done Sent false
19004 6210 Acked Write 10 Done Done Sent false
19005 6211 Acked Write 2 Done Done Sent false
19006 6212 Acked Write 8 Done Done Sent false
19007 6213 Acked Read 10 Done Done Sent false
19008 6214 Acked Flush 0 Done Done Sent false
19009 6215 Acked Read 5 Done Done Sent false
19010 6216 Acked Read 5 Done Done Sent false
19011 6217 Acked Read 2 Done Done Sent false
19012 6218 Acked Read 10 Done Done Sent false
19013 6219 Acked Read 1 Done Done Sent false
19014 6220 Acked Read 8 Done Done Sent false
19015 6221 Acked Write 9 Done Done Sent false
19016 6222 Acked Write 10 Done Done Sent false
19017 6223 Acked Read 1 Done Done Sent false
19018 6224 Acked Read 4 Done Done Sent false
19019 6225 Acked Read 7 Done Done Sent false
19020 6226 Acked Read 3 Done Done Sent false
19021 6227 Acked Write 7 Done Done Sent false
19022 6228 Acked Flush 0 Done Done Sent false
19023 6229 Acked Read 7 Done Done Sent false
19024 6230 Acked Write 9 Done Done Sent false
19025 6231 Acked Write 6 Done Done Sent false
19026 6232 Acked Read 8 Done Done Sent false
19027 6233 Acked Write 9 Done Done Sent false
19028 6234 Acked Read 6 Done Done Sent false
19029 6235 Acked Read 1 Done Done Sent false
19030 6236 Acked Write 10 Done Done Sent false
19031 6237 Acked Write 7 Done Done Sent false
19032 6238 Acked Flush 0 Done Done Sent false
19033 6239 Acked Write 3 Done Done Sent false
19034 6240 Acked Write 1 Done Done Sent false
19035 6241 Acked Flush 0 Done Done Sent false
19036 6242 Acked Write 3 Done Done Sent false
19037 6243 Acked Read 7 Done Done Sent false
19038 6244 Acked Flush 0 Done Done Sent false
19039 6245 Acked Write 10 Done Done Sent false
19040 6246 Acked Write 10 Done Done Sent false
19041 6247 Acked Read 9 Done Done Sent false
19042 6248 Acked Read 1 Done Done Sent false
19043 6249 Acked Read 1 Done Done Sent false
19044 6250 Acked Read 7 Done Done Sent false
19045 6251 Acked Read 5 Done Done Sent false
19046 6252 Acked Flush 0 Done Done Sent false
19047 6253 Acked Write 2 Done Done Sent false
19048 6254 Acked Read 9 Done Done Sent false
19049 6255 Acked Read 8 Done Done Sent false
19050 6256 Acked Write 9 Done Done Sent false
19051 6257 Acked Read 1 Done Done Sent false
19052 6258 Acked Read 9 Done Done Sent false
19053 6259 Acked Flush 0 Done Done Sent false
19054 6260 Acked Read 2 Done Done Sent false
19055 6261 Acked Read 10 Done Done Sent false
19056 6262 Acked Read 9 Done Done Sent false
19057 6263 Acked Write 6 Done Done Sent false
19058 6264 Acked Read 10 Done Done Sent false
19059 6265 Acked Read 7 Done Done Sent false
19060 6266 Acked Write 10 Done Done Sent false
19061 6267 Acked Flush 0 Done Done Sent false
19062 6268 Acked Read 8 Done Done Sent false
19063 6269 Acked Read 3 Done Done Sent false
19064 6270 Acked Read 6 Done Done Sent false
19065 6271 Acked Write 3 Done Done Sent false
19066 6272 Acked Write 9 Done Done Sent false
19067 6273 Acked Read 5 Done Done Sent false
19068 6274 Acked Read 1 Done Done Sent false
19069 6275 Acked Flush 0 Done Done Sent false
19070 6276 Acked Read 9 Done Done Sent false
19071 6277 Acked Read 10 Done Done Sent false
19072 6278 Acked Read 4 Done Done Sent false
19073 6279 Acked Write 7 Done Done Sent false
19074 6280 Acked Read 4 Done Done Sent false
19075 6281 Acked Write 10 Done Done Sent false
19076 6282 Acked Read 2 Done Done Sent false
19077 6283 Acked Flush 0 Done Done Sent false
19078 6284 Acked Write 1 Done Done Sent false
19079 6285 Acked Write 1 Done Done Sent false
19080 6286 Acked Flush 0 Done Done Sent false
19081 6287 Acked Read 4 Done Done Sent false
19082 6288 Acked Write 7 Done Done Sent false
19083 6289 Acked Write 7 Done Done Sent false
19084 6290 Acked Read 9 Done Done Sent false
19085 6291 Acked Write 5 Done Done Sent false
19086 6292 Acked Flush 0 Done Done Sent false
19087 6293 Acked Write 6 Done Done Sent false
19088 6294 Acked Read 7 Done Done Sent false
19089 6295 Acked Flush 0 Done Done Sent false
19090 6296 Acked Read 6 Done Done Sent false
19091 6297 Acked Read 9 Done Done Sent false
19092 6298 Acked Read 10 Done Done Sent false
19093 6299 Acked Read 10 Done Done Sent false
19094 6300 Acked Write 7 Done Done Sent false
19095 6301 Acked Read 6 Done Done Sent false
19096 6302 Acked Write 6 Done Done Sent false
19097 6303 Acked Write 6 Done Done Sent false
19098 6304 Acked Write 6 Done Done Sent false
19099 6305 Acked Write 4 Done Done Sent false
19100 6306 Acked Write 8 Done Done Sent false
19101 6307 Acked Read 7 Done Done Sent false
19102 6308 Acked Write 3 Done Done Sent false
19103 6309 Acked Read 7 Done Done Sent false
19104 6310 Acked Write 6 Done Done Sent false
19105 6311 Acked Read 8 Done Done Sent false
19106 6312 Acked Write 3 Done Done Sent false
19107 6313 Acked Read 10 Done Done Sent false
19108 6314 Acked Read 4 Done Done Sent false
19109 6315 Acked Read 10 Done Done Sent false
19110 6316 Acked Read 8 Done Done Sent false
19111 6317 Acked Write 4 Done Done Sent false
19112 6318 Acked Read 10 Done Done Sent false
19113 6319 Acked Read 10 Done Done Sent false
19114 6320 Acked Write 3 Done Done Sent false
19115 6321 Acked Read 3 Done Done Sent false
19116 6322 Acked Write 7 Done Done Sent false
19117 6323 Acked Write 2 Done Done Sent false
19118 6324 Acked Read 7 Done Done Sent false
19119 6325 Acked Read 4 Done Done Sent false
19120 6326 Acked Read 3 Done Done Sent false
19121 6327 Acked Write 8 Done Done Sent false
19122 6328 Acked Write 9 Done Done Sent false
19123 6329 Acked Write 10 Done Done Sent false
19124 6330 Acked Read 9 Done Done Sent false
19125 6331 Acked Read 6 Done Done Sent false
19126 6332 Acked Read 10 Done Done Sent false
19127 6333 Acked Flush 0 Done Done Sent false
19128 6334 Acked Write 6 Done Done Sent false
19129 6335 Acked Read 9 Done Done Sent false
19130 6336 Acked Write 8 Done Done Sent false
19131 6337 Acked Write 6 Done Done Sent false
19132 6338 Acked Read 7 Done Done Sent false
19133 6339 Acked Read 4 Done Done Sent false
19134 6340 Acked Read 1 Done Done Sent false
19135 6341 Acked Write 5 Done Done Sent false
19136 6342 Acked Write 8 Done Done Sent false
19137 6343 Acked Read 2 Done Done Sent false
19138 6344 Acked Read 1 Done Done Sent false
19139 6345 Acked Write 10 Done Done Sent false
19140 6346 Acked Write 9 Done Done Sent false
19141 6347 Acked Write 1 Done Done Sent false
19142 6348 Acked Write 4 Done Done Sent false
19143 6349 Acked Read 8 Done Done Sent false
19144 6350 Acked Write 1 Done Done Sent false
19145 6351 Acked Write 7 Done Done Sent false
19146 6352 Acked Write 7 Done Done Sent false
19147 6353 Acked Write 8 Done Done Sent false
19148 6354 Acked Read 8 Done Done Sent false
19149 6355 Acked Flush 0 Done Done Sent false
19150 6356 Acked Read 2 Done Done Sent false
19151 6357 Acked Read 2 Done Done Sent false
19152 6358 Acked Read 2 Done Done Sent false
19153 6359 Acked Write 7 Done Done Sent false
19154 6360 Acked Write 9 Done Done Sent false
19155 6361 Acked Read 4 Done Done Sent false
19156 6362 Acked Read 2 Done Done Sent false
19157 6363 Acked Write 5 Done Done Sent false
19158 6364 Acked Write 1 Done Done Sent false
19159 6365 Acked Read 9 Done Done Sent false
19160 6366 Acked Read 10 Done Done Sent false
19161 6367 Acked Read 5 Done Done Sent false
19162 6368 Acked Write 9 Done Done Sent false
19163 6369 Acked Read 1 Done Done Sent false
19164 6370 Acked Write 6 Done Done Sent false
19165 6371 Acked Write 6 Done Done Sent false
19166 6372 Acked Read 1 Done Done Sent false
19167 6373 Acked Write 5 Done Done Sent false
19168 6374 Acked Read 1 Done Done Sent false
19169 6375 Acked Read 3 Done Done Sent false
19170 6376 Acked Write 7 Done Done Sent false
19171 6377 Acked Write 5 Done Done Sent false
19172 6378 Acked Read 5 Done Done Sent false
19173 6379 Acked Flush 0 Done Done Sent false
19174 6380 Acked Write 8 Done Done Sent false
19175 6381 Acked Read 3 Done Done Sent false
19176 6382 Acked Read 9 Done Done Sent false
19177 6383 Acked Read 6 Done Done Sent false
19178 6384 Acked Write 2 Done Done Sent false
19179 6385 Acked Read 4 Done Done Sent false
19180 6386 Acked Read 8 Done Done Sent false
19181 6387 Acked Flush 0 Done Done Sent false
19182 6388 Acked Read 9 Done Done Sent false
19183 6389 Acked Flush 0 Done Done Sent false
19184 6390 Acked Write 5 Done Done Sent false
19185 6391 Acked Write 4 Done Done Sent false
19186 6392 Acked Read 2 Done Done Sent false
19187 6393 Acked Flush 0 Done Done Sent false
19188 6394 Acked Read 3 Done Done Sent false
19189 6395 Acked Read 7 Done Done Sent false
19190 6396 Acked Read 8 Done Done Sent false
19191 6397 Acked Write 10 Done Done Sent false
19192 6398 Acked Read 6 Done Done Sent false
19193 6399 Acked Read 9 Done Done Sent false
19194 6400 Acked Write 9 Done Done Sent false
19195 6401 Acked Write 2 Done Done Sent false
19196 6402 Acked Flush 0 Done Done Sent false
19197 6403 Acked Read 6 Done Done Sent false
19198 6404 Acked Write 8 Done Done Sent false
19199 6405 Acked Read 9 Done Done Sent false
19200 6406 Acked Read 5 Done Done Sent false
19201 6407 Acked Read 5 Done Done Sent false
19202 6408 Acked Write 4 Done Done Sent false
19203 6409 Acked Write 5 Done Done Sent false
19204 6410 Acked Read 2 Done Done Sent false
19205 6411 Acked Write 1 Done Done Sent false
19206 6412 Acked Write 7 Done Done Sent false
19207 6413 Acked Write 6 Done Done Sent false
19208 6414 Acked Write 6 Done Done Sent false
19209 6415 Acked Write 10 Done Done Sent false
19210 6416 Acked Read 9 Done Done Sent false
19211 6417 Acked Write 1 Done Done Sent false
19212 6418 Acked Read 3 Done Done Sent false
19213 6419 Acked Flush 0 Done Done Sent false
19214 6420 Acked Read 9 Done Done Sent false
19215 6421 Acked Read 10 Done Done Sent false
19216 6422 Acked Read 3 Done Done Sent false
19217 6423 Acked Write 10 Done Done Sent false
19218 6424 Acked Read 10 Done Done Sent false
19219 6425 Acked Read 7 Done Done Sent false
19220 6426 Acked Read 2 Done Done Sent false
19221 6427 Acked Write 4 Done Done Sent false
19222 6428 Acked Read 6 Done Done Sent false
19223 6429 Acked Flush 0 Done Done Sent false
19224 6430 Acked Write 4 Done Done Sent false
19225 6431 Acked Write 5 Done Done Sent false
19226 6432 Acked Flush 0 Done Done Sent false
19227 STATES DS:0 DS:1 DS:2 TOTAL
19228 Sent 0 0 301 301
19229 Done 301 301 0 602
19230 Skipped 0 0 0 0
19231 Error 0 0 0 0
19232 Last Flush: Some(JobId(6432)) Some(JobId(6432)) Some(JobId(6131))
19233 Downstairs last five completed: 6131 6130 6129 6128 6127
19234 Upstairs last five completed: 6432 6431 6430 6429 6428
19235 CLIENT: Up:0 ds:301 act:2
19236 [8830] got start action
19237 [8830] I am going to restart
19238 Starting downstairs at port 8830
19239 Make output file at "/tmp/dsc/downstairs-8830.txt"
19240 [8830][2] reports Starting
19241 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1218)
19242 [8830][2] reports Running
19243 ----------------------------------------------------------------
19244 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19245 States: Active Active Offline
19246 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19247 6132 Acked Write 8 Done Done Sent false
19248 6133 Acked Write 6 Done Done Sent false
19249 6134 Acked Write 2 Done Done Sent false
19250 6135 Acked Write 1 Done Done Sent false
19251 6136 Acked Write 7 Done Done Sent false
19252 6137 Acked Write 9 Done Done Sent false
19253 6138 Acked Read 7 Done Done Sent false
19254 6139 Acked Write 10 Done Done Sent false
19255 6140 Acked Write 4 Done Done Sent false
19256 6141 Acked Read 9 Done Done Sent false
19257 6142 Acked Read 9 Done Done Sent false
19258 6143 Acked Read 9 Done Done Sent false
19259 6144 Acked Read 9 Done Done Sent false
19260 6145 Acked Read 3 Done Done Sent false
19261 6146 Acked Read 10 Done Done Sent false
19262 6147 Acked Read 2 Done Done Sent false
19263 6148 Acked Read 8 Done Done Sent false
19264 6149 Acked Write 1 Done Done Sent false
19265 6150 Acked Flush 0 Done Done Sent false
19266 6151 Acked Read 1 Done Done Sent false
19267 6152 Acked Write 9 Done Done Sent false
19268 6153 Acked Write 8 Done Done Sent false
19269 6154 Acked Read 5 Done Done Sent false
19270 6155 Acked Write 5 Done Done Sent false
19271 6156 Acked Read 6 Done Done Sent false
19272 6157 Acked Read 1 Done Done Sent false
19273 6158 Acked Read 3 Done Done Sent false
19274 6159 Acked Read 6 Done Done Sent false
19275 6160 Acked Read 8 Done Done Sent false
19276 6161 Acked Read 6 Done Done Sent false
19277 6162 Acked Read 6 Done Done Sent false
19278 6163 Acked Read 10 Done Done Sent false
19279 6164 Acked Read 5 Done Done Sent false
19280 6165 Acked Write 9 Done Done Sent false
19281 6166 Acked Write 1 Done Done Sent false
19282 6167 Acked Read 8 Done Done Sent false
19283 6168 Acked Write 9 Done Done Sent false
19284 6169 Acked Read 4 Done Done Sent false
19285 6170 Acked Write 3 Done Done Sent false
19286 6171 Acked Write 5 Done Done Sent false
19287 6172 Acked Read 6 Done Done Sent false
19288 6173 Acked Write 5 Done Done Sent false
19289 6174 Acked Read 7 Done Done Sent false
19290 6175 Acked Read 8 Done Done Sent false
19291 6176 Acked Write 8 Done Done Sent false
19292 6177 Acked Write 9 Done Done Sent false
19293 6178 Acked Write 8 Done Done Sent false
19294 6179 Acked Write 5 Done Done Sent false
19295 6180 Acked Read 5 Done Done Sent false
19296 6181 Acked Write 6 Done Done Sent false
19297 6182 Acked Read 7 Done Done Sent false
19298 6183 Acked Write 4 Done Done Sent false
19299 6184 Acked Write 8 Done Done Sent false
19300 6185 Acked Write 5 Done Done Sent false
19301 6186 Acked Read 4 Done Done Sent false
19302 6187 Acked Read 5 Done Done Sent false
19303 6188 Acked Write 5 Done Done Sent false
19304 6189 Acked Read 5 Done Done Sent false
19305 6190 Acked Read 6 Done Done Sent false
19306 6191 Acked Flush 0 Done Done Sent false
19307 6192 Acked Read 6 Done Done Sent false
19308 6193 Acked Write 3 Done Done Sent false
19309 6194 Acked Read 4 Done Done Sent false
19310 6195 Acked Write 4 Done Done Sent false
19311 6196 Acked Flush 0 Done Done Sent false
19312 6197 Acked Write 5 Done Done Sent false
19313 6198 Acked Write 4 Done Done Sent false
19314 6199 Acked Flush 0 Done Done Sent false
19315 6200 Acked Read 8 Done Done Sent false
19316 6201 Acked Read 7 Done Done Sent false
19317 6202 Acked Write 9 Done Done Sent false
19318 6203 Acked Read 8 Done Done Sent false
19319 6204 Acked Read 5 Done Done Sent false
19320 6205 Acked Write 10 Done Done Sent false
19321 6206 Acked Flush 0 Done Done Sent false
19322 6207 Acked Read 3 Done Done Sent false
19323 6208 Acked Read 2 Done Done Sent false
19324 6209 Acked Read 2 Done Done Sent false
19325 6210 Acked Write 10 Done Done Sent false
19326 6211 Acked Write 2 Done Done Sent false
19327 6212 Acked Write 8 Done Done Sent false
19328 6213 Acked Read 10 Done Done Sent false
19329 6214 Acked Flush 0 Done Done Sent false
19330 6215 Acked Read 5 Done Done Sent false
19331 6216 Acked Read 5 Done Done Sent false
19332 6217 Acked Read 2 Done Done Sent false
19333 6218 Acked Read 10 Done Done Sent false
19334 6219 Acked Read 1 Done Done Sent false
19335 6220 Acked Read 8 Done Done Sent false
19336 6221 Acked Write 9 Done Done Sent false
19337 6222 Acked Write 10 Done Done Sent false
19338 6223 Acked Read 1 Done Done Sent false
19339 6224 Acked Read 4 Done Done Sent false
19340 6225 Acked Read 7 Done Done Sent false
19341 6226 Acked Read 3 Done Done Sent false
19342 6227 Acked Write 7 Done Done Sent false
19343 6228 Acked Flush 0 Done Done Sent false
19344 6229 Acked Read 7 Done Done Sent false
19345 6230 Acked Write 9 Done Done Sent false
19346 6231 Acked Write 6 Done Done Sent false
19347 6232 Acked Read 8 Done Done Sent false
19348 6233 Acked Write 9 Done Done Sent false
19349 6234 Acked Read 6 Done Done Sent false
19350 6235 Acked Read 1 Done Done Sent false
19351 6236 Acked Write 10 Done Done Sent false
19352 6237 Acked Write 7 Done Done Sent false
19353 6238 Acked Flush 0 Done Done Sent false
19354 6239 Acked Write 3 Done Done Sent false
19355 6240 Acked Write 1 Done Done Sent false
19356 6241 Acked Flush 0 Done Done Sent false
19357 6242 Acked Write 3 Done Done Sent false
19358 6243 Acked Read 7 Done Done Sent false
19359 6244 Acked Flush 0 Done Done Sent false
19360 6245 Acked Write 10 Done Done Sent false
19361 6246 Acked Write 10 Done Done Sent false
19362 6247 Acked Read 9 Done Done Sent false
19363 6248 Acked Read 1 Done Done Sent false
19364 6249 Acked Read 1 Done Done Sent false
19365 6250 Acked Read 7 Done Done Sent false
19366 6251 Acked Read 5 Done Done Sent false
19367 6252 Acked Flush 0 Done Done Sent false
19368 6253 Acked Write 2 Done Done Sent false
19369 6254 Acked Read 9 Done Done Sent false
19370 6255 Acked Read 8 Done Done Sent false
19371 6256 Acked Write 9 Done Done Sent false
19372 6257 Acked Read 1 Done Done Sent false
19373 6258 Acked Read 9 Done Done Sent false
19374 6259 Acked Flush 0 Done Done Sent false
19375 6260 Acked Read 2 Done Done Sent false
19376 6261 Acked Read 10 Done Done Sent false
19377 6262 Acked Read 9 Done Done Sent false
19378 6263 Acked Write 6 Done Done Sent false
19379 6264 Acked Read 10 Done Done Sent false
19380 6265 Acked Read 7 Done Done Sent false
19381 6266 Acked Write 10 Done Done Sent false
19382 6267 Acked Flush 0 Done Done Sent false
19383 6268 Acked Read 8 Done Done Sent false
19384 6269 Acked Read 3 Done Done Sent false
19385 6270 Acked Read 6 Done Done Sent false
19386 6271 Acked Write 3 Done Done Sent false
19387 6272 Acked Write 9 Done Done Sent false
19388 6273 Acked Read 5 Done Done Sent false
19389 6274 Acked Read 1 Done Done Sent false
19390 6275 Acked Flush 0 Done Done Sent false
19391 6276 Acked Read 9 Done Done Sent false
19392 6277 Acked Read 10 Done Done Sent false
19393 6278 Acked Read 4 Done Done Sent false
19394 6279 Acked Write 7 Done Done Sent false
19395 6280 Acked Read 4 Done Done Sent false
19396 6281 Acked Write 10 Done Done Sent false
19397 6282 Acked Read 2 Done Done Sent false
19398 6283 Acked Flush 0 Done Done Sent false
19399 6284 Acked Write 1 Done Done Sent false
19400 6285 Acked Write 1 Done Done Sent false
19401 6286 Acked Flush 0 Done Done Sent false
19402 6287 Acked Read 4 Done Done Sent false
19403 6288 Acked Write 7 Done Done Sent false
19404 6289 Acked Write 7 Done Done Sent false
19405 6290 Acked Read 9 Done Done Sent false
19406 6291 Acked Write 5 Done Done Sent false
19407 6292 Acked Flush 0 Done Done Sent false
19408 6293 Acked Write 6 Done Done Sent false
19409 6294 Acked Read 7 Done Done Sent false
19410 6295 Acked Flush 0 Done Done Sent false
19411 6296 Acked Read 6 Done Done Sent false
19412 6297 Acked Read 9 Done Done Sent false
19413 6298 Acked Read 10 Done Done Sent false
19414 6299 Acked Read 10 Done Done Sent false
19415 6300 Acked Write 7 Done Done Sent false
19416 6301 Acked Read 6 Done Done Sent false
19417 6302 Acked Write 6 Done Done Sent false
19418 6303 Acked Write 6 Done Done Sent false
19419 6304 Acked Write 6 Done Done Sent false
19420 6305 Acked Write 4 Done Done Sent false
19421 6306 Acked Write 8 Done Done Sent false
19422 6307 Acked Read 7 Done Done Sent false
19423 6308 Acked Write 3 Done Done Sent false
19424 6309 Acked Read 7 Done Done Sent false
19425 6310 Acked Write 6 Done Done Sent false
19426 6311 Acked Read 8 Done Done Sent false
19427 6312 Acked Write 3 Done Done Sent false
19428 6313 Acked Read 10 Done Done Sent false
19429 6314 Acked Read 4 Done Done Sent false
19430 6315 Acked Read 10 Done Done Sent false
19431 6316 Acked Read 8 Done Done Sent false
19432 6317 Acked Write 4 Done Done Sent false
19433 6318 Acked Read 10 Done Done Sent false
19434 6319 Acked Read 10 Done Done Sent false
19435 6320 Acked Write 3 Done Done Sent false
19436 6321 Acked Read 3 Done Done Sent false
19437 6322 Acked Write 7 Done Done Sent false
19438 6323 Acked Write 2 Done Done Sent false
19439 6324 Acked Read 7 Done Done Sent false
19440 6325 Acked Read 4 Done Done Sent false
19441 6326 Acked Read 3 Done Done Sent false
19442 6327 Acked Write 8 Done Done Sent false
19443 6328 Acked Write 9 Done Done Sent false
19444 6329 Acked Write 10 Done Done Sent false
19445 6330 Acked Read 9 Done Done Sent false
19446 6331 Acked Read 6 Done Done Sent false
19447 6332 Acked Read 10 Done Done Sent false
19448 6333 Acked Flush 0 Done Done Sent false
19449 6334 Acked Write 6 Done Done Sent false
19450 6335 Acked Read 9 Done Done Sent false
19451 6336 Acked Write 8 Done Done Sent false
19452 6337 Acked Write 6 Done Done Sent false
19453 6338 Acked Read 7 Done Done Sent false
19454 6339 Acked Read 4 Done Done Sent false
19455 6340 Acked Read 1 Done Done Sent false
19456 6341 Acked Write 5 Done Done Sent false
19457 6342 Acked Write 8 Done Done Sent false
19458 6343 Acked Read 2 Done Done Sent false
19459 6344 Acked Read 1 Done Done Sent false
19460 6345 Acked Write 10 Done Done Sent false
19461 6346 Acked Write 9 Done Done Sent false
19462 6347 Acked Write 1 Done Done Sent false
19463 6348 Acked Write 4 Done Done Sent false
19464 6349 Acked Read 8 Done Done Sent false
19465 6350 Acked Write 1 Done Done Sent false
19466 6351 Acked Write 7 Done Done Sent false
19467 6352 Acked Write 7 Done Done Sent false
19468 6353 Acked Write 8 Done Done Sent false
19469 6354 Acked Read 8 Done Done Sent false
19470 6355 Acked Flush 0 Done Done Sent false
19471 6356 Acked Read 2 Done Done Sent false
19472 6357 Acked Read 2 Done Done Sent false
19473 6358 Acked Read 2 Done Done Sent false
19474 6359 Acked Write 7 Done Done Sent false
19475 6360 Acked Write 9 Done Done Sent false
19476 6361 Acked Read 4 Done Done Sent false
19477 6362 Acked Read 2 Done Done Sent false
19478 6363 Acked Write 5 Done Done Sent false
19479 6364 Acked Write 1 Done Done Sent false
19480 6365 Acked Read 9 Done Done Sent false
19481 6366 Acked Read 10 Done Done Sent false
19482 6367 Acked Read 5 Done Done Sent false
19483 6368 Acked Write 9 Done Done Sent false
19484 6369 Acked Read 1 Done Done Sent false
19485 6370 Acked Write 6 Done Done Sent false
19486 6371 Acked Write 6 Done Done Sent false
19487 6372 Acked Read 1 Done Done Sent false
19488 6373 Acked Write 5 Done Done Sent false
19489 6374 Acked Read 1 Done Done Sent false
19490 6375 Acked Read 3 Done Done Sent false
19491 6376 Acked Write 7 Done Done Sent false
19492 6377 Acked Write 5 Done Done Sent false
19493 6378 Acked Read 5 Done Done Sent false
19494 6379 Acked Flush 0 Done Done Sent false
19495 6380 Acked Write 8 Done Done Sent false
19496 6381 Acked Read 3 Done Done Sent false
19497 6382 Acked Read 9 Done Done Sent false
19498 6383 Acked Read 6 Done Done Sent false
19499 6384 Acked Write 2 Done Done Sent false
19500 6385 Acked Read 4 Done Done Sent false
19501 6386 Acked Read 8 Done Done Sent false
19502 6387 Acked Flush 0 Done Done Sent false
19503 6388 Acked Read 9 Done Done Sent false
19504 6389 Acked Flush 0 Done Done Sent false
19505 6390 Acked Write 5 Done Done Sent false
19506 6391 Acked Write 4 Done Done Sent false
19507 6392 Acked Read 2 Done Done Sent false
19508 6393 Acked Flush 0 Done Done Sent false
19509 6394 Acked Read 3 Done Done Sent false
19510 6395 Acked Read 7 Done Done Sent false
19511 6396 Acked Read 8 Done Done Sent false
19512 6397 Acked Write 10 Done Done Sent false
19513 6398 Acked Read 6 Done Done Sent false
19514 6399 Acked Read 9 Done Done Sent false
19515 6400 Acked Write 9 Done Done Sent false
19516 6401 Acked Write 2 Done Done Sent false
19517 6402 Acked Flush 0 Done Done Sent false
19518 6403 Acked Read 6 Done Done Sent false
19519 6404 Acked Write 8 Done Done Sent false
19520 6405 Acked Read 9 Done Done Sent false
19521 6406 Acked Read 5 Done Done Sent false
19522 6407 Acked Read 5 Done Done Sent false
19523 6408 Acked Write 4 Done Done Sent false
19524 6409 Acked Write 5 Done Done Sent false
19525 6410 Acked Read 2 Done Done Sent false
19526 6411 Acked Write 1 Done Done Sent false
19527 6412 Acked Write 7 Done Done Sent false
19528 6413 Acked Write 6 Done Done Sent false
19529 6414 Acked Write 6 Done Done Sent false
19530 6415 Acked Write 10 Done Done Sent false
19531 6416 Acked Read 9 Done Done Sent false
19532 6417 Acked Write 1 Done Done Sent false
19533 6418 Acked Read 3 Done Done Sent false
19534 6419 Acked Flush 0 Done Done Sent false
19535 6420 Acked Read 9 Done Done Sent false
19536 6421 Acked Read 10 Done Done Sent false
19537 6422 Acked Read 3 Done Done Sent false
19538 6423 Acked Write 10 Done Done Sent false
19539 6424 Acked Read 10 Done Done Sent false
19540 6425 Acked Read 7 Done Done Sent false
19541 6426 Acked Read 2 Done Done Sent false
19542 6427 Acked Write 4 Done Done Sent false
19543 6428 Acked Read 6 Done Done Sent false
19544 6429 Acked Flush 0 Done Done Sent false
19545 6430 Acked Write 4 Done Done Sent false
19546 6431 Acked Write 5 Done Done Sent false
19547 6432 Acked Flush 0 Done Done Sent false
19548 STATES DS:0 DS:1 DS:2 TOTAL
19549 Sent 0 0 301 301
19550 Done 301 301 0 602
19551 Skipped 0 0 0 0
19552 Error 0 0 0 0
19553 Last Flush: Some(JobId(6432)) Some(JobId(6432)) Some(JobId(6131))
19554 Downstairs last five completed: 6131 6130 6129 6128 6127
19555 Upstairs last five completed: 6432 6431 6430 6429 6428
19556 CLIENT: Up:0 ds:301 act:2
195572025-01-23T18:51:59.186ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
195582025-01-23T18:51:59.186ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64833) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
195592025-01-23T18:51:59.187ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
195602025-01-23T18:51:59.188ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
195612025-01-23T18:51:59.188ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
195622025-01-23T18:51:59.188ZINFOcrucible: send last flush ID to this DS: Some(JobId(6131)) = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
195632025-01-23T18:51:59.188ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
195642025-01-23T18:51:59.188ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6131)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
195652025-01-23T18:51:59.188ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6131)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
19566 ----------------------------------------------------------------
19567 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19568 Downstairs last five completed: 6432 6431 6430 6429 6428
19569 Upstairs last five completed: 6432 6431 6430 6429 6428
19570 CLIENT: Up:0 ds:0 act:3
19571 Replay: All jobs finished
19572 Main task has work to do, go find it
19573 got dsc Stop(1)
19574 stop 1
19575 Jan 23 18:52:01.441[8820] Got stop action so:false kr:false
19576 INFO request completed, latency_us: 343, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 0d7a1fb7-d761-42da-9f99-64fb5df04372, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
195772025-01-23T18:52:01.442ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
195782025-01-23T18:52:01.442ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
19579 Jan 23 18:52:01.442 INFO request completed, latency_us: 143, response_code: 200{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:52:01.442337088Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
19580 , uri: /state/cid/1, method: GET, req_id: 1cb71c14-af86-46c5-8579-2f871c2234ff, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
195812025-01-23T18:52:01.442ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
19582 [8820] Exited with: ExitStatus(unix_wait_status(9))
19583 [8820][1] reports Exit
19584 Jan 23 18:52:05.445 INFO request completed, latency_us: 223, response_code: 200, uri: /state/cid/1, method: GET, req_id: 15c4ff4d-e02e-4315-9e27-9db6a5937781, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
19585 001/300 Read block 4136 len 4096
19586 002/300 Write block 81 len 8192 data: 3 3
19587 003/300 Flush
19588 004/300 Read block 429 len 12288
19589 005/300 Read block 872 len 8192
19590 006/300 Read block 771 len 28672
19591 007/300 Read block 3567 len 20480
19592 008/300 Write block 3867 len 20480 data: 3 3 3 3 4
19593 009/300 Write block 4638 len 4096 data: 5
19594 010/300 Read block 3698 len 28672
19595 011/300 Read block 2728 len 8192
19596 012/300 Write block 3903 len 16384 data: 5 5 5 4
19597 013/300 Read block 3074 len 28672
19598 014/300 Read block 1416 len 24576
19599 015/300 Read block 2117 len 12288
19600 016/300 Write block 4416 len 36864 data: 5 5 5 4 4 5 5 6 8
19601 017/300 Read block 465 len 24576
19602 018/300 Write block 5 len 8192 data: 2 3
19603 019/300 Read block 1830 len 12288
19604 020/300 Read block 4620 len 24576
19605 021/300 Write block 4708 len 32768 data: 6 5 4 5 5 5 5 4
19606 022/300 Read block 2097 len 40960
19607 023/300 Read block 981 len 4096
19608 024/300 Write block 3859 len 28672 data: 5 6 4 4 4 4 4
19609 025/300 Read block 14 len 16384
19610 026/300 Flush
19611 027/300 Write block 2615 len 16384 data: 6 6 6 7
19612 028/300 Read block 2626 len 24576
19613 029/300 Read block 1780 len 16384
19614 030/300 Write block 134 len 20480 data: 6 6 6 5 7
19615 031/300 Flush
19616 032/300 Write block 2629 len 16384 data: 6 5 4 6
19617 033/300 Read block 2597 len 20480
19618 034/300 Flush
19619 035/300 Read block 1427 len 4096
19620 036/300 Read block 2534 len 24576
19621 037/300 Write block 1828 len 28672 data: 6 5 6 6 5 4 4
19622 038/300 Read block 3642 len 32768
19623 039/300 Read block 954 len 20480
19624 040/300 Read block 4120 len 36864
19625 041/300 Flush
19626 042/300 Write block 4900 len 16384 data: 3 3 4 4
19627 043/300 Write block 1469 len 36864 data: 4 3 2 2 2 2 3 4 4
19628 044/300 Read block 997 len 40960
19629 045/300 Read block 4894 len 24576
19630 046/300 Write block 678 len 20480 data: 4 4 3 2 2
19631 047/300 Write block 4582 len 28672 data: 7 6 5 4 4 4 2
19632 048/300 Write block 1706 len 20480 data: 6 4 4 5 3
19633 049/300 Read block 502 len 8192
19634 050/300 Read block 655 len 4096
19635 051/300 Read block 3005 len 40960
19636 052/300 Read block 3375 len 32768
19637 053/300 Flush
19638 054/300 Read block 4778 len 32768
19639 055/300 Write block 4005 len 4096 data: 4
19640 056/300 Read block 2308 len 8192
19641 057/300 Read block 2472 len 24576
19642 058/300 Write block 1842 len 24576 data: 2 2 2 2 3 4
19643 059/300 Write block 4798 len 36864 data: 2 2 2 2 2 2 2 5 6
19644 060/300 Write block 1129 len 12288 data: 3 3 4
19645 061/300 Flush
19646 062/300 Read block 3582 len 12288
19647 063/300 Read block 4544 len 24576
19648 064/300 Flush
19649 065/300 Read block 1248 len 36864
19650 066/300 Read block 83 len 24576
19651 067/300 Read block 3344 len 16384
19652 068/300 Flush
19653 069/300 Read block 332 len 24576
19654 070/300 Flush
19655 071/300 Write block 4328 len 36864 data: 6 6 6 4 3 2 3 3 5
19656 072/300 Flush
19657 073/300 Read block 3251 len 12288
19658 074/300 Write block 1930 len 20480 data: 5 5 6 6 6
19659 075/300 Write block 2901 len 28672 data: 2 2 2 3 4 4 4
19660 076/300 Read block 654 len 36864
19661 077/300 Write block 2147 len 28672 data: 4 5 5 5 4 3 3
19662 078/300 Read block 3584 len 16384
19663 079/300 Write block 1549 len 32768 data: 5 5 4 4 3 3 3 3
19664 080/300 Write block 820 len 4096 data: 8
19665 081/300 Write block 3669 len 16384 data: 5 5 5 3
19666 082/300 Read block 636 len 24576
19667 083/300 Write block 835 len 36864 data: 5 4 4 4 3 3 3 4 4
19668 084/300 Write block 4344 len 32768 data: 9 7 7 6 7 6 5 4
19669 085/300 Read block 1905 len 4096
19670 086/300 Write block 1595 len 32768 data: 5 7 7 5 3 2 2 3
19671 087/300 Write block 3386 len 36864 data: 8 8 7 6 5 3 3 2 2
19672 088/300 Write block 3298 len 12288 data: 6 5 4
19673 089/300 Write block 1173 len 12288 data: 4 4 5
19674 090/300 Write block 1344 len 16384 data: 4 4 2 2
19675 091/300 Read block 3636 len 16384
19676 092/300 Read block 2289 len 28672
19677 093/300 Read block 4055 len 28672
19678 094/300 Read block 376 len 8192
19679 095/300 Flush
19680 096/300 Read block 4348 len 20480
19681 097/300 Read block 464 len 28672
19682 098/300 Write block 3438 len 24576 data: 4 4 3 4 5 5
19683 099/300 Read block 2325 len 36864
19684 100/300 Write block 4887 len 28672 data: 4 4 3 3 3 4 4
19685 101/300 Read block 4015 len 16384
19686 102/300 Read block 4348 len 24576
19687 103/300 Write block 35 len 20480 data: 5 6 6 6 7
19688 104/300 Read block 237 len 40960
19689 105/300 Write block 2103 len 32768 data: 7 7 7 7 5 6 5 4
19690 106/300 Read block 2033 len 4096
19691 107/300 Write block 4833 len 28672 data: 6 5 5 5 4 3 3
19692 108/300 Read block 4073 len 24576
19693 109/300 Read block 2790 len 24576
19694 110/300 Read block 1290 len 36864
19695 111/300 Read block 1521 len 12288
19696 112/300 Read block 2557 len 24576
19697 113/300 Write block 2848 len 4096 data: 5
19698 114/300 Write block 3955 len 8192 data: 5 5
19699 115/300 Write block 271 len 8192 data: 5 5
19700 116/300 Read block 1747 len 8192
19701 117/300 Read block 255 len 8192
19702 118/300 Write block 1928 len 8192 data: 4 5
19703 119/300 Write block 37 len 12288 data: 7 7 8
19704 120/300 Read block 3381 len 12288
19705 121/300 Write block 895 len 16384 data: 6 8 6 5
19706 122/300 Write block 4706 len 28672 data: 5 5 7 6 5 6 6
19707 123/300 Flush
19708 124/300 Read block 2953 len 24576
19709 125/300 Read block 3430 len 36864
19710 126/300 Write block 3752 len 8192 data: 6 6
19711 127/300 Flush
19712 128/300 Write block 2169 len 28672 data: 4 3 3 3 3 5 5
19713 129/300 Read block 4824 len 28672
19714 130/300 Read block 706 len 24576
19715 131/300 Write block 2574 len 20480 data: 4 5 4 6 3
19716 132/300 Read block 272 len 12288
19717 133/300 Read block 4634 len 28672
19718 134/300 Flush
19719 135/300 Write block 4686 len 12288 data: 3 3 3
19720 136/300 Flush
19721 137/300 Read block 3666 len 20480
19722 138/300 Read block 815 len 8192
19723 139/300 Read block 86 len 16384
19724 140/300 Read block 2731 len 4096
19725 141/300 Flush
19726 142/300 Read block 29 len 20480
19727 143/300 Read block 4128 len 20480
19728 144/300 Write block 4782 len 16384 data: 4 4 4 4
19729 145/300 Write block 3442 len 28672 data: 6 6 6 4 4 4 4
19730 146/300 Flush
19731 147/300 Read block 3627 len 28672
19732 148/300 Write block 3992 len 36864 data: 4 4 5 5 5 6 5 5 5
19733 149/300 Read block 1482 len 28672
19734 150/300 Read block 1302 len 36864
19735 151/300 Flush
19736 152/300 Read block 3715 len 24576
19737 153/300 Write block 3064 len 24576 data: 5 5 5 4 5 5
19738 154/300 Read block 3578 len 36864
19739 155/300 Write block 1343 len 16384 data: 4 5 5 3
19740 156/300 Read block 305 len 4096
19741 157/300 Read block 96 len 28672
19742 158/300 Read block 891 len 28672
19743 159/300 Flush
19744 160/300 Write block 22 len 20480 data: 4 3 3 3 3
19745 161/300 Flush
19746 162/300 Read block 1297 len 8192
19747 163/300 Write block 4682 len 24576 data: 3 4 2 3 4 4
19748 164/300 Write block 236 len 8192 data: 3 4
19749 165/300 Read block 2786 len 40960
19750 166/300 Read block 3550 len 4096
19751 167/300 Flush
19752 168/300 Write block 609 len 12288 data: 5 5 3
19753 169/300 Flush
19754 170/300 Read block 1615 len 20480
19755 171/300 Read block 4316 len 12288
19756 172/300 Write block 1002 len 20480 data: 6 6 6 5 4
19757 173/300 Read block 4146 len 20480
19758 174/300 Read block 2650 len 4096
19759 175/300 Write block 273 len 36864 data: 4 4 5 6 7 5 5 4 3
19760 176/300 Read block 1044 len 12288
19761 177/300 Read block 3463 len 28672
19762 178/300 Read block 2738 len 28672
19763 179/300 Read block 1923 len 16384
19764 180/300 Read block 3594 len 8192
19765 181/300 Read block 201 len 20480
19766 182/300 Read block 4161 len 20480
19767 183/300 Read block 201 len 8192
19768 184/300 Write block 1912 len 28672 data: 6 6 6 6 5 5 5
19769 185/300 Write block 1281 len 32768 data: 4 3 4 5 5 7 6 6
19770 186/300 Write block 1625 len 4096 data: 5
19771 187/300 Read block 3318 len 28672
19772 188/300 Read block 506 len 32768
19773 189/300 Read block 3894 len 12288
19774 190/300 Read block 2684 len 4096
19775 191/300 Write block 984 len 20480 data: 7 8 8 6 4
19776 192/300 Flush
19777 193/300 Read block 2567 len 16384
19778 194/300 Write block 4239 len 32768 data: 3 3 2 3 4 4 4 4
19779 195/300 Read block 762 len 36864
19780 196/300 Flush
19781 197/300 Flush
19782 198/300 Read block 3627 len 20480
19783 199/300 Write block 2692 len 4096 data: 5
19784 200/300 Write block 502 len 4096 data: 3
19785 201/300 Write block 1297 len 16384 data: 3 3 3 5
19786 202/300 Read block 3514 len 40960
19787 203/300 Write block 3672 len 16384 data: 4 4 4 4
19788 204/300 Write block 2155 len 8192 data: 5 7
19789 205/300 Write block 4163 len 40960 data: 7 9 7 7 8 8 7 8 9 9
19790 206/300 Write block 4389 len 8192 data: 3 2
19791 207/300 Read block 1962 len 24576
19792 208/300 Flush
19793 209/300 Read block 3294 len 16384
19794 210/300 Read block 192 len 12288
19795 211/300 Write block 150 len 28672 data: 7 7 6 6 5 5 4
19796 212/300 Read block 722 len 20480
19797 213/300 Write block 716 len 36864 data: 6 5 5 4 3 3 2 2 3
19798 214/300 Flush
19799 215/300 Write block 733 len 16384 data: 5 5 4 5
19800 216/300 Read block 1976 len 20480
19801 217/300 Write block 4167 len 36864 data: 9 9 8 9 10 10 7 6 8
19802 218/300 Write block 765 len 32768 data: 3 3 3 3 3 3 3 3
19803 219/300 Read block 4085 len 12288
19804 220/300 Write block 4016 len 24576 data: 6 3 3 4 4 5
19805 221/300 Read block 4102 len 20480
19806 222/300 Write block 2737 len 24576 data: 5 5 4 3 4 5
19807 223/300 Write block 500 len 36864 data: 3 3 4 2 3 4 4 4 4
19808 224/300 Read block 3156 len 12288
19809 225/300 Write block 3894 len 8192 data: 5 4
19810 226/300 Read block 161 len 28672
19811 227/300 Flush
19812 228/300 Write block 4440 len 24576 data: 2 2 2 2 3 4
19813 229/300 Flush
19814 230/300 Read block 3053 len 40960
19815 231/300 Read block 2422 len 12288
19816 232/300 Write block 2384 len 12288 data: 3 3 2
19817 233/300 Write block 3505 len 36864 data: 5 5 6 5 6 6 6 6 5
19818 234/300 Read block 249 len 4096
19819 235/300 Write block 124 len 20480 data: 5 6 6 4 5
19820 236/300 Read block 707 len 36864
19821 237/300 Write block 2123 len 20480 data: 7 7 6 6 6
19822 238/300 Read block 622 len 24576
19823 239/300 Flush
19824 240/300 Write block 2159 len 28672 data: 6 6 5 5 5 4 4
19825 241/300 Read block 3085 len 24576
19826 242/300 Read block 2236 len 16384
19827 243/300 Write block 4834 len 4096 data: 6
19828 244/300 Read block 4919 len 4096
19829 245/300 Read block 1708 len 40960
19830 246/300 Read block 3175 len 12288
19831 247/300 Write block 898 len 24576 data: 6 6 5 6 5 5
19832 248/300 Read block 4258 len 32768
19833 249/300 Read block 4926 len 40960
19834 250/300 Read block 1643 len 40960
19835 251/300 Write block 3779 len 4096 data: 5
19836 252/300 Flush
19837 253/300 Read block 4357 len 12288
19838 254/300 Write block 1536 len 4096 data: 3
19839 255/300 Write block 2295 len 20480 data: 5 4 4 4 5
19840 256/300 Read block 4338 len 40960
19841 257/300 Write block 3032 len 32768 data: 5 5 4 5 5 5 7 8
19842 258/300 Read block 3247 len 20480
19843 259/300 Write block 4713 len 24576 data: 6 6 5 2 2 2
19844 260/300 Read block 931 len 20480
19845 261/300 Write block 2699 len 12288 data: 5 5 4
19846 262/300 Write block 3506 len 40960 data: 6 7 6 7 7 7 7 6 4 5
19847 263/300 Write block 4447 len 20480 data: 7 8 9 8 8
19848 264/300 Read block 3463 len 20480
19849 265/300 Write block 163 len 12288 data: 3 3 4
19850 266/300 Flush
19851 267/300 Write block 2106 len 36864 data: 8 6 7 6 5 3 3 2 2
19852 268/300 Write block 409 len 8192 data: 6 7
19853 269/300 Flush
19854 270/300 Write block 2942 len 28672 data: 3 3 3 3 3 3 5
19855 271/300 Flush
19856 272/300 Read block 4964 len 36864
19857 273/300 Write block 4508 len 16384 data: 4 4 3 3
19858 274/300 Write block 362 len 28672 data: 2 3 3 5 4 5 4
19859 275/300 Write block 3492 len 36864 data: 3 3 4 5 5 4 5 6 5
19860 276/300 Write block 4180 len 8192 data: 3 3
19861 277/300 Write block 4063 len 36864 data: 3 4 5 6 6 5 4 4 5
19862 278/300 Write block 945 len 8192 data: 7 5
19863 279/300 Write block 3595 len 24576 data: 5 3 4 4 3 3
19864 280/300 Read block 4000 len 40960
19865 281/300 Write block 6 len 20480 data: 4 3 3 3 3
19866 282/300 Read block 1218 len 40960
19867 283/300 Read block 4800 len 40960
19868 284/300 Read block 3727 len 40960
19869 285/300 Write block 428 len 32768 data: 8 9 8 8 5 6 5 3
19870 286/300 Write block 2120 len 36864 data: 5 6 6 8 8 7 7 7 6
19871 287/300 Read block 2727 len 20480
19872 288/300 Write block 963 len 12288 data: 6 6 6
19873 289/300 Read block 3137 len 16384
19874 290/300 Write block 2425 len 16384 data: 3 4 5 4
19875 291/300 Write block 1697 len 32768 data: 6 5 5 4 3 4 6 6
19876 292/300 Write block 4310 len 40960 data: 4 4 5 4 4 4 3 3 3 4
19877 293/300 Read block 442 len 12288
19878 294/300 Read block 948 len 16384
19879 295/300 Write block 1813 len 8192 data: 8 8
19880 296/300 Flush
19881 297/300 Flush
19882 298/300 Read block 299 len 36864
19883 299/300 Read block 4049 len 8192
19884 300/300 Read block 881 len 8192
19885 301/300 Read block 3423 len 24576
19886 Main task has work to do, go find it
19887 got dsc Start(1)
19888 start 1
19889 Jan 23 18:52:05.695 INFO request completed, latency_us: 246, response_code: 204, uri: /start/cid/1, method: POST, req_id: 253a5fab-067d-4691-bc0f-bd89c453cc10, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
19890 Replay: started 1, returned:Ok(())
19891 ----------------------------------------------------------------
19892 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19893 States: Active Offline Active
19894 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19895 6433 Acked Read 1 Done Sent Done false
19896 6434 Acked Write 2 Done Sent Done false
19897 6435 Acked Flush 0 Done Sent Done false
19898 6436 Acked Read 3 Done Sent Done false
19899 6437 Acked Read 2 Done Sent Done false
19900 6438 Acked Read 7 Done Sent Done false
19901 6439 Acked Read 5 Done Sent Done false
19902 6440 Acked Write 5 Done Sent Done false
19903 6441 Acked Write 1 Done Sent Done false
19904 6442 Acked Read 7 Done Sent Done false
19905 6443 Acked Read 2 Done Sent Done false
19906 6444 Acked Write 4 Done Sent Done false
19907 6445 Acked Read 7 Done Sent Done false
19908 6446 Acked Read 6 Done Sent Done false
19909 6447 Acked Read 3 Done Sent Done false
19910 6448 Acked Write 9 Done Sent Done false
19911 6449 Acked Read 6 Done Sent Done false
19912 6450 Acked Write 2 Done Sent Done false
19913 6451 Acked Read 3 Done Sent Done false
19914 6452 Acked Read 6 Done Sent Done false
19915 6453 Acked Write 8 Done Sent Done false
19916 6454 Acked Read 10 Done Sent Done false
19917 6455 Acked Read 1 Done Sent Done false
19918 6456 Acked Write 7 Done Sent Done false
19919 6457 Acked Read 4 Done Sent Done false
19920 6458 Acked Flush 0 Done Sent Done false
19921 6459 Acked Write 4 Done Sent Done false
19922 6460 Acked Read 6 Done Sent Done false
19923 6461 Acked Read 4 Done Sent Done false
19924 6462 Acked Write 5 Done Sent Done false
19925 6463 Acked Flush 0 Done Sent Done false
19926 6464 Acked Write 4 Done Sent Done false
19927 6465 Acked Read 5 Done Sent Done false
19928 6466 Acked Flush 0 Done Sent Done false
19929 6467 Acked Read 1 Done Sent Done false
19930 6468 Acked Read 6 Done Sent Done false
19931 6469 Acked Write 7 Done Sent Done false
19932 6470 Acked Read 8 Done Sent Done false
19933 6471 Acked Read 5 Done Sent Done false
19934 6472 Acked Read 9 Done Sent Done false
19935 6473 Acked Flush 0 Done Sent Done false
19936 6474 Acked Write 4 Done Sent Done false
19937 6475 Acked Write 9 Done Sent Done false
19938 6476 Acked Read 10 Done Sent Done false
19939 6477 Acked Read 6 Done Sent Done false
19940 6478 Acked Write 5 Done Sent Done false
19941 6479 Acked Write 7 Done Sent Done false
19942 6480 Acked Write 5 Done Sent Done false
19943 6481 Acked Read 2 Done Sent Done false
19944 6482 Acked Read 1 Done Sent Done false
19945 6483 Acked Read 10 Done Sent Done false
19946 6484 Acked Read 8 Done Sent Done false
19947 6485 Acked Flush 0 Done Sent Done false
19948 6486 Acked Read 8 Done Sent Done false
19949 6487 Acked Write 1 Done Sent Done false
19950 6488 Acked Read 2 Done Sent Done false
19951 6489 Acked Read 6 Done Sent Done false
19952 6490 Acked Write 6 Done Sent Done false
19953 6491 Acked Write 9 Done Sent Done false
19954 6492 Acked Write 3 Done Sent Done false
19955 6493 Acked Flush 0 Done Sent Done false
19956 6494 Acked Read 3 Done Sent Done false
19957 6495 Acked Read 6 Done Sent Done false
19958 6496 Acked Flush 0 Done Sent Done false
19959 6497 Acked Read 9 Done Sent Done false
19960 6498 Acked Read 6 Done Sent Done false
19961 6499 Acked Read 4 Done Sent Done false
19962 6500 Acked Flush 0 Done Sent Done false
19963 6501 Acked Read 6 Done Sent Done false
19964 6502 Acked Flush 0 Done Sent Done false
19965 6503 Acked Write 9 Done Sent Done false
19966 6504 Acked Flush 0 Done Sent Done false
19967 6505 Acked Read 3 Done Sent Done false
19968 6506 Acked Write 5 Done Sent Done false
19969 6507 Acked Write 7 Done Sent Done false
19970 6508 Acked Read 9 Done Sent Done false
19971 6509 Acked Write 7 Done Sent Done false
19972 6510 Acked Read 4 Done Sent Done false
19973 6511 Acked Write 8 Done Sent Done false
19974 6512 Acked Write 1 Done Sent Done false
19975 6513 Acked Write 4 Done Sent Done false
19976 6514 Acked Read 6 Done Sent Done false
19977 6515 Acked Write 9 Done Sent Done false
19978 6516 Acked Write 8 Done Sent Done false
19979 6517 Acked Read 1 Done Sent Done false
19980 6518 Acked Write 8 Done Sent Done false
19981 6519 Acked Write 9 Done Sent Done false
19982 6520 Acked Write 3 Done Sent Done false
19983 6521 Acked Write 3 Done Sent Done false
19984 6522 Acked Write 4 Done Sent Done false
19985 6523 Acked Read 4 Done Sent Done false
19986 6524 Acked Read 7 Done Sent Done false
19987 6525 Acked Read 7 Done Sent Done false
19988 6526 Acked Read 2 Done Sent Done false
19989 6527 Acked Flush 0 Done Sent Done false
19990 6528 Acked Read 5 Done Sent Done false
19991 6529 Acked Read 7 Done Sent Done false
19992 6530 Acked Write 6 Done Sent Done false
19993 6531 Acked Read 9 Done Sent Done false
19994 6532 Acked Write 7 Done Sent Done false
19995 6533 Acked Read 4 Done Sent Done false
19996 6534 Acked Read 6 Done Sent Done false
19997 6535 Acked Write 5 Done Sent Done false
19998 6536 Acked Read 10 Done Sent Done false
19999 6537 Acked Write 8 Done Sent Done false
20000 6538 Acked Read 1 Done Sent Done false
20001 6539 Acked Write 7 Done Sent Done false
20002 6540 Acked Read 6 Done Sent Done false
20003 6541 Acked Read 6 Done Sent Done false
20004 6542 Acked Read 9 Done Sent Done false
20005 6543 Acked Read 3 Done Sent Done false
20006 6544 Acked Read 6 Done Sent Done false
20007 6545 Acked Write 1 Done Sent Done false
20008 6546 Acked Write 2 Done Sent Done false
20009 6547 Acked Write 2 Done Sent Done false
20010 6548 Acked Read 2 Done Sent Done false
20011 6549 Acked Read 2 Done Sent Done false
20012 6550 Acked Write 2 Done Sent Done false
20013 6551 Acked Write 3 Done Sent Done false
20014 6552 Acked Read 3 Done Sent Done false
20015 6553 Acked Write 4 Done Sent Done false
20016 6554 Acked Write 7 Done Sent Done false
20017 6555 Acked Flush 0 Done Sent Done false
20018 6556 Acked Read 6 Done Sent Done false
20019 6557 Acked Read 9 Done Sent Done false
20020 6558 Acked Write 2 Done Sent Done false
20021 6559 Acked Flush 0 Done Sent Done false
20022 6560 Acked Write 7 Done Sent Done false
20023 6561 Acked Read 7 Done Sent Done false
20024 6562 Acked Read 6 Done Sent Done false
20025 6563 Acked Write 5 Done Sent Done false
20026 6564 Acked Read 3 Done Sent Done false
20027 6565 Acked Read 7 Done Sent Done false
20028 6566 Acked Flush 0 Done Sent Done false
20029 6567 Acked Write 3 Done Sent Done false
20030 6568 Acked Flush 0 Done Sent Done false
20031 6569 Acked Read 5 Done Sent Done false
20032 6570 Acked Read 2 Done Sent Done false
20033 6571 Acked Read 4 Done Sent Done false
20034 6572 Acked Read 1 Done Sent Done false
20035 6573 Acked Flush 0 Done Sent Done false
20036 6574 Acked Read 5 Done Sent Done false
20037 6575 Acked Read 5 Done Sent Done false
20038 6576 Acked Write 4 Done Sent Done false
20039 6577 Acked Write 7 Done Sent Done false
20040 6578 Acked Flush 0 Done Sent Done false
20041 6579 Acked Read 7 Done Sent Done false
20042 6580 Acked Write 9 Done Sent Done false
20043 6581 Acked Read 7 Done Sent Done false
20044 6582 Acked Read 9 Done Sent Done false
20045 6583 Acked Flush 0 Done Sent Done false
20046 6584 Acked Read 6 Done Sent Done false
20047 6585 Acked Write 6 Done Sent Done false
20048 6586 Acked Read 9 Done Sent Done false
20049 6587 Acked Write 4 Done Sent Done false
20050 6588 Acked Read 1 Done Sent Done false
20051 6589 Acked Read 7 Done Sent Done false
20052 6590 Acked Read 7 Done Sent Done false
20053 6591 Acked Flush 0 Done Sent Done false
20054 6592 Acked Write 5 Done Sent Done false
20055 6593 Acked Flush 0 Done Sent Done false
20056 6594 Acked Read 2 Done Sent Done false
20057 6595 Acked Write 6 Done Sent Done false
20058 6596 Acked Write 2 Done Sent Done false
20059 6597 Acked Read 10 Done Sent Done false
20060 6598 Acked Read 1 Done Sent Done false
20061 6599 Acked Flush 0 Done Sent Done false
20062 6600 Acked Write 3 Done Sent Done false
20063 6601 Acked Flush 0 Done Sent Done false
20064 6602 Acked Read 5 Done Sent Done false
20065 6603 Acked Read 3 Done Sent Done false
20066 6604 Acked Write 5 Done Sent Done false
20067 6605 Acked Read 5 Done Sent Done false
20068 6606 Acked Read 1 Done Sent Done false
20069 6607 Acked Write 9 Done Sent Done false
20070 6608 Acked Read 3 Done Sent Done false
20071 6609 Acked Read 7 Done Sent Done false
20072 6610 Acked Read 7 Done Sent Done false
20073 6611 Acked Read 4 Done Sent Done false
20074 6612 Acked Read 2 Done Sent Done false
20075 6613 Acked Read 5 Done Sent Done false
20076 6614 Acked Read 5 Done Sent Done false
20077 6615 Acked Read 2 Done Sent Done false
20078 6616 Acked Write 7 Done Sent Done false
20079 6617 Acked Write 8 Done Sent Done false
20080 6618 Acked Write 1 Done Sent Done false
20081 6619 Acked Read 7 Done Sent Done false
20082 6620 Acked Read 8 Done Sent Done false
20083 6621 Acked Read 3 Done Sent Done false
20084 6622 Acked Read 1 Done Sent Done false
20085 6623 Acked Write 5 Done Sent Done false
20086 6624 Acked Flush 0 Done Sent Done false
20087 6625 Acked Read 4 Done Sent Done false
20088 6626 Acked Write 8 Done Sent Done false
20089 6627 Acked Read 9 Done Sent Done false
20090 6628 Acked Flush 0 Done Sent Done false
20091 6629 Acked Flush 0 Done Sent Done false
20092 6630 Acked Read 5 Done Sent Done false
20093 6631 Acked Write 1 Done Sent Done false
20094 6632 Acked Write 1 Done Sent Done false
20095 6633 Acked Write 4 Done Sent Done false
20096 6634 Acked Read 10 Done Sent Done false
20097 6635 Acked Write 4 Done Sent Done false
20098 6636 Acked Write 2 Done Sent Done false
20099 6637 Acked Write 10 Done Sent Done false
20100 6638 Acked Write 2 Done Sent Done false
20101 6639 Acked Read 6 Done Sent Done false
20102 6640 Acked Flush 0 Done Sent Done false
20103 6641 Acked Read 4 Done Sent Done false
20104 6642 Acked Read 3 Done Sent Done false
20105 6643 Acked Write 7 Done Sent Done false
20106 6644 Acked Read 5 Done Sent Done false
20107 6645 Acked Write 9 Done Sent Done false
20108 6646 Acked Flush 0 Done Sent Done false
20109 6647 Acked Write 4 Done Sent Done false
20110 6648 Acked Read 5 Done Sent Done false
20111 6649 Acked Write 9 Done Sent Done false
20112 6650 Acked Write 8 Done Sent Done false
20113 6651 Acked Read 3 Done Sent Done false
20114 6652 Acked Write 6 Done Sent Done false
20115 6653 Acked Read 5 Done Sent Done false
20116 6654 Acked Write 6 Done Sent Done false
20117 6655 Acked Write 9 Done Sent Done false
20118 6656 Acked Read 3 Done Sent Done false
20119 6657 Acked Write 2 Done Sent Done false
20120 6658 Acked Read 7 Done Sent Done false
20121 6659 Acked Flush 0 Done Sent Done false
20122 6660 Acked Write 6 Done Sent Done false
20123 6661 Acked Flush 0 Done Sent Done false
20124 6662 Acked Read 10 Done Sent Done false
20125 6663 Acked Read 3 Done Sent Done false
20126 6664 Acked Write 3 Done Sent Done false
20127 6665 Acked Write 9 Done Sent Done false
20128 6666 Acked Read 1 Done Sent Done false
20129 6667 Acked Write 5 Done Sent Done false
20130 6668 Acked Read 9 Done Sent Done false
20131 6669 Acked Write 5 Done Sent Done false
20132 6670 Acked Read 6 Done Sent Done false
20133 6671 Acked Flush 0 Done Sent Done false
20134 6672 Acked Write 7 Done Sent Done false
20135 6673 Acked Read 6 Done Sent Done false
20136 6674 Acked Read 4 Done Sent Done false
20137 6675 Acked Write 1 Done Sent Done false
20138 6676 Acked Read 1 Done Sent Done false
20139 6677 Acked Read 10 Done Sent Done false
20140 6678 Acked Read 3 Done Sent Done false
20141 6679 Acked Write 6 Done Sent Done false
20142 6680 Acked Read 8 Done Sent Done false
20143 6681 Acked Read 10 Done Sent Done false
20144 6682 Acked Read 10 Done Sent Done false
20145 6683 Acked Write 1 Done Sent Done false
20146 6684 Acked Flush 0 Done Sent Done false
20147 6685 Acked Read 3 Done Sent Done false
20148 6686 Acked Write 1 Done Sent Done false
20149 6687 Acked Write 5 Done Sent Done false
20150 6688 Acked Read 10 Done Sent Done false
20151 6689 Acked Write 8 Done Sent Done false
20152 6690 Acked Read 5 Done Sent Done false
20153 6691 Acked Write 6 Done Sent Done false
20154 6692 Acked Read 5 Done Sent Done false
20155 6693 Acked Write 3 Done Sent Done false
20156 6694 Acked Write 10 Done Sent Done false
20157 6695 Acked Write 5 Done Sent Done false
20158 6696 Acked Read 5 Done Sent Done false
20159 6697 Acked Write 3 Done Sent Done false
20160 6698 Acked Flush 0 Done Sent Done false
20161 6699 Acked Write 9 Done Sent Done false
20162 6700 Acked Write 2 Done Sent Done false
20163 6701 Acked Flush 0 Done Sent Done false
20164 6702 Acked Write 7 Done Sent Done false
20165 6703 Acked Flush 0 Done Sent Done false
20166 6704 Acked Read 9 Done Sent Done false
20167 6705 Acked Write 4 Done Sent Done false
20168 6706 Acked Write 7 Done Sent Done false
20169 6707 Acked Write 9 Done Sent Done false
20170 6708 Acked Write 2 Done Sent Done false
20171 6709 Acked Write 9 Done Sent Done false
20172 6710 Acked Write 2 Done Sent Done false
20173 6711 Acked Write 6 Done Sent Done false
20174 6712 Acked Read 10 Done Sent Done false
20175 6713 Acked Write 5 Done Sent Done false
20176 6714 Acked Read 10 Done Sent Done false
20177 6715 Acked Read 10 Done Sent Done false
20178 6716 Acked Read 10 Done Sent Done false
20179 6717 Acked Write 8 Done Sent Done false
20180 6718 Acked Write 9 Done Sent Done false
20181 6719 Acked Read 5 Done Sent Done false
20182 6720 Acked Write 3 Done Sent Done false
20183 6721 Acked Read 4 Done Sent Done false
20184 6722 Acked Write 4 Done Sent Done false
20185 6723 Acked Write 8 Done Sent Done false
20186 6724 Acked Write 10 Done Sent Done false
20187 6725 Acked Read 3 Done Sent Done false
20188 6726 Acked Read 4 Done Sent Done false
20189 6727 Acked Write 2 Done Sent Done false
20190 6728 Acked Flush 0 Done Sent Done false
20191 6729 Acked Flush 0 Done Sent Done false
20192 6730 Acked Read 9 Done Sent Done false
20193 6731 Acked Read 2 Done Sent Done false
20194 6732 Acked Read 2 Done Sent Done false
20195 6733 Acked Read 6 Done Sent Done false
20196 STATES DS:0 DS:1 DS:2 TOTAL
20197 Sent 0 301 0 301
20198 Done 301 0 301 602
20199 Skipped 0 0 0 0
20200 Error 0 0 0 0
20201 Last Flush: Some(JobId(6729)) Some(JobId(6432)) Some(JobId(6729))
20202 Downstairs last five completed: 6432 6431 6430 6429 6428
20203 Upstairs last five completed: 6733 6732 6731 6730 6729
20204 CLIENT: Up:0 ds:301 act:2
20205 [8820] got start action
20206 [8820] I am going to restart
20207 Starting downstairs at port 8820
20208 Make output file at "/tmp/dsc/downstairs-8820.txt"
20209 [8820][1] reports Starting
20210 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1219)
20211 [8820][1] reports Running
20212 ----------------------------------------------------------------
20213 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
20214 States: Active Offline Active
20215 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20216 6433 Acked Read 1 Done Sent Done false
20217 6434 Acked Write 2 Done Sent Done false
20218 6435 Acked Flush 0 Done Sent Done false
20219 6436 Acked Read 3 Done Sent Done false
20220 6437 Acked Read 2 Done Sent Done false
20221 6438 Acked Read 7 Done Sent Done false
20222 6439 Acked Read 5 Done Sent Done false
20223 6440 Acked Write 5 Done Sent Done false
20224 6441 Acked Write 1 Done Sent Done false
20225 6442 Acked Read 7 Done Sent Done false
20226 6443 Acked Read 2 Done Sent Done false
20227 6444 Acked Write 4 Done Sent Done false
20228 6445 Acked Read 7 Done Sent Done false
20229 6446 Acked Read 6 Done Sent Done false
20230 6447 Acked Read 3 Done Sent Done false
20231 6448 Acked Write 9 Done Sent Done false
20232 6449 Acked Read 6 Done Sent Done false
20233 6450 Acked Write 2 Done Sent Done false
20234 6451 Acked Read 3 Done Sent Done false
20235 6452 Acked Read 6 Done Sent Done false
20236 6453 Acked Write 8 Done Sent Done false
20237 6454 Acked Read 10 Done Sent Done false
20238 6455 Acked Read 1 Done Sent Done false
20239 6456 Acked Write 7 Done Sent Done false
20240 6457 Acked Read 4 Done Sent Done false
20241 6458 Acked Flush 0 Done Sent Done false
20242 6459 Acked Write 4 Done Sent Done false
20243 6460 Acked Read 6 Done Sent Done false
20244 6461 Acked Read 4 Done Sent Done false
20245 6462 Acked Write 5 Done Sent Done false
20246 6463 Acked Flush 0 Done Sent Done false
20247 6464 Acked Write 4 Done Sent Done false
20248 6465 Acked Read 5 Done Sent Done false
20249 6466 Acked Flush 0 Done Sent Done false
20250 6467 Acked Read 1 Done Sent Done false
20251 6468 Acked Read 6 Done Sent Done false
20252 6469 Acked Write 7 Done Sent Done false
20253 6470 Acked Read 8 Done Sent Done false
20254 6471 Acked Read 5 Done Sent Done false
20255 6472 Acked Read 9 Done Sent Done false
20256 6473 Acked Flush 0 Done Sent Done false
20257 6474 Acked Write 4 Done Sent Done false
20258 6475 Acked Write 9 Done Sent Done false
20259 6476 Acked Read 10 Done Sent Done false
20260 6477 Acked Read 6 Done Sent Done false
20261 6478 Acked Write 5 Done Sent Done false
20262 6479 Acked Write 7 Done Sent Done false
20263 6480 Acked Write 5 Done Sent Done false
20264 6481 Acked Read 2 Done Sent Done false
20265 6482 Acked Read 1 Done Sent Done false
20266 6483 Acked Read 10 Done Sent Done false
20267 6484 Acked Read 8 Done Sent Done false
20268 6485 Acked Flush 0 Done Sent Done false
20269 6486 Acked Read 8 Done Sent Done false
20270 6487 Acked Write 1 Done Sent Done false
20271 6488 Acked Read 2 Done Sent Done false
20272 6489 Acked Read 6 Done Sent Done false
20273 6490 Acked Write 6 Done Sent Done false
20274 6491 Acked Write 9 Done Sent Done false
20275 6492 Acked Write 3 Done Sent Done false
20276 6493 Acked Flush 0 Done Sent Done false
20277 6494 Acked Read 3 Done Sent Done false
20278 6495 Acked Read 6 Done Sent Done false
20279 6496 Acked Flush 0 Done Sent Done false
20280 6497 Acked Read 9 Done Sent Done false
20281 6498 Acked Read 6 Done Sent Done false
20282 6499 Acked Read 4 Done Sent Done false
20283 6500 Acked Flush 0 Done Sent Done false
20284 6501 Acked Read 6 Done Sent Done false
20285 6502 Acked Flush 0 Done Sent Done false
20286 6503 Acked Write 9 Done Sent Done false
20287 6504 Acked Flush 0 Done Sent Done false
20288 6505 Acked Read 3 Done Sent Done false
20289 6506 Acked Write 5 Done Sent Done false
20290 6507 Acked Write 7 Done Sent Done false
20291 6508 Acked Read 9 Done Sent Done false
20292 6509 Acked Write 7 Done Sent Done false
20293 6510 Acked Read 4 Done Sent Done false
20294 6511 Acked Write 8 Done Sent Done false
20295 6512 Acked Write 1 Done Sent Done false
20296 6513 Acked Write 4 Done Sent Done false
20297 6514 Acked Read 6 Done Sent Done false
20298 6515 Acked Write 9 Done Sent Done false
20299 6516 Acked Write 8 Done Sent Done false
20300 6517 Acked Read 1 Done Sent Done false
20301 6518 Acked Write 8 Done Sent Done false
20302 6519 Acked Write 9 Done Sent Done false
20303 6520 Acked Write 3 Done Sent Done false
20304 6521 Acked Write 3 Done Sent Done false
20305 6522 Acked Write 4 Done Sent Done false
20306 6523 Acked Read 4 Done Sent Done false
20307 6524 Acked Read 7 Done Sent Done false
20308 6525 Acked Read 7 Done Sent Done false
20309 6526 Acked Read 2 Done Sent Done false
20310 6527 Acked Flush 0 Done Sent Done false
20311 6528 Acked Read 5 Done Sent Done false
20312 6529 Acked Read 7 Done Sent Done false
20313 6530 Acked Write 6 Done Sent Done false
20314 6531 Acked Read 9 Done Sent Done false
20315 6532 Acked Write 7 Done Sent Done false
20316 6533 Acked Read 4 Done Sent Done false
20317 6534 Acked Read 6 Done Sent Done false
20318 6535 Acked Write 5 Done Sent Done false
20319 6536 Acked Read 10 Done Sent Done false
20320 6537 Acked Write 8 Done Sent Done false
20321 6538 Acked Read 1 Done Sent Done false
20322 6539 Acked Write 7 Done Sent Done false
20323 6540 Acked Read 6 Done Sent Done false
20324 6541 Acked Read 6 Done Sent Done false
20325 6542 Acked Read 9 Done Sent Done false
20326 6543 Acked Read 3 Done Sent Done false
20327 6544 Acked Read 6 Done Sent Done false
20328 6545 Acked Write 1 Done Sent Done false
20329 6546 Acked Write 2 Done Sent Done false
20330 6547 Acked Write 2 Done Sent Done false
20331 6548 Acked Read 2 Done Sent Done false
20332 6549 Acked Read 2 Done Sent Done false
20333 6550 Acked Write 2 Done Sent Done false
20334 6551 Acked Write 3 Done Sent Done false
20335 6552 Acked Read 3 Done Sent Done false
20336 6553 Acked Write 4 Done Sent Done false
20337 6554 Acked Write 7 Done Sent Done false
20338 6555 Acked Flush 0 Done Sent Done false
20339 6556 Acked Read 6 Done Sent Done false
20340 6557 Acked Read 9 Done Sent Done false
20341 6558 Acked Write 2 Done Sent Done false
20342 6559 Acked Flush 0 Done Sent Done false
20343 6560 Acked Write 7 Done Sent Done false
20344 6561 Acked Read 7 Done Sent Done false
20345 6562 Acked Read 6 Done Sent Done false
20346 6563 Acked Write 5 Done Sent Done false
20347 6564 Acked Read 3 Done Sent Done false
20348 6565 Acked Read 7 Done Sent Done false
20349 6566 Acked Flush 0 Done Sent Done false
20350 6567 Acked Write 3 Done Sent Done false
20351 6568 Acked Flush 0 Done Sent Done false
20352 6569 Acked Read 5 Done Sent Done false
20353 6570 Acked Read 2 Done Sent Done false
20354 6571 Acked Read 4 Done Sent Done false
20355 6572 Acked Read 1 Done Sent Done false
20356 6573 Acked Flush 0 Done Sent Done false
20357 6574 Acked Read 5 Done Sent Done false
20358 6575 Acked Read 5 Done Sent Done false
20359 6576 Acked Write 4 Done Sent Done false
20360 6577 Acked Write 7 Done Sent Done false
20361 6578 Acked Flush 0 Done Sent Done false
20362 6579 Acked Read 7 Done Sent Done false
20363 6580 Acked Write 9 Done Sent Done false
20364 6581 Acked Read 7 Done Sent Done false
20365 6582 Acked Read 9 Done Sent Done false
20366 6583 Acked Flush 0 Done Sent Done false
20367 6584 Acked Read 6 Done Sent Done false
20368 6585 Acked Write 6 Done Sent Done false
20369 6586 Acked Read 9 Done Sent Done false
20370 6587 Acked Write 4 Done Sent Done false
20371 6588 Acked Read 1 Done Sent Done false
20372 6589 Acked Read 7 Done Sent Done false
20373 6590 Acked Read 7 Done Sent Done false
20374 6591 Acked Flush 0 Done Sent Done false
20375 6592 Acked Write 5 Done Sent Done false
20376 6593 Acked Flush 0 Done Sent Done false
20377 6594 Acked Read 2 Done Sent Done false
20378 6595 Acked Write 6 Done Sent Done false
20379 6596 Acked Write 2 Done Sent Done false
20380 6597 Acked Read 10 Done Sent Done false
20381 6598 Acked Read 1 Done Sent Done false
20382 6599 Acked Flush 0 Done Sent Done false
20383 6600 Acked Write 3 Done Sent Done false
20384 6601 Acked Flush 0 Done Sent Done false
20385 6602 Acked Read 5 Done Sent Done false
20386 6603 Acked Read 3 Done Sent Done false
20387 6604 Acked Write 5 Done Sent Done false
20388 6605 Acked Read 5 Done Sent Done false
20389 6606 Acked Read 1 Done Sent Done false
20390 6607 Acked Write 9 Done Sent Done false
20391 6608 Acked Read 3 Done Sent Done false
20392 6609 Acked Read 7 Done Sent Done false
20393 6610 Acked Read 7 Done Sent Done false
20394 6611 Acked Read 4 Done Sent Done false
20395 6612 Acked Read 2 Done Sent Done false
20396 6613 Acked Read 5 Done Sent Done false
20397 6614 Acked Read 5 Done Sent Done false
20398 6615 Acked Read 2 Done Sent Done false
20399 6616 Acked Write 7 Done Sent Done false
20400 6617 Acked Write 8 Done Sent Done false
20401 6618 Acked Write 1 Done Sent Done false
20402 6619 Acked Read 7 Done Sent Done false
20403 6620 Acked Read 8 Done Sent Done false
20404 6621 Acked Read 3 Done Sent Done false
20405 6622 Acked Read 1 Done Sent Done false
20406 6623 Acked Write 5 Done Sent Done false
20407 6624 Acked Flush 0 Done Sent Done false
20408 6625 Acked Read 4 Done Sent Done false
20409 6626 Acked Write 8 Done Sent Done false
20410 6627 Acked Read 9 Done Sent Done false
20411 6628 Acked Flush 0 Done Sent Done false
20412 6629 Acked Flush 0 Done Sent Done false
20413 6630 Acked Read 5 Done Sent Done false
20414 6631 Acked Write 1 Done Sent Done false
20415 6632 Acked Write 1 Done Sent Done false
20416 6633 Acked Write 4 Done Sent Done false
20417 6634 Acked Read 10 Done Sent Done false
20418 6635 Acked Write 4 Done Sent Done false
20419 6636 Acked Write 2 Done Sent Done false
20420 6637 Acked Write 10 Done Sent Done false
20421 6638 Acked Write 2 Done Sent Done false
20422 6639 Acked Read 6 Done Sent Done false
20423 6640 Acked Flush 0 Done Sent Done false
20424 6641 Acked Read 4 Done Sent Done false
20425 6642 Acked Read 3 Done Sent Done false
20426 6643 Acked Write 7 Done Sent Done false
20427 6644 Acked Read 5 Done Sent Done false
20428 6645 Acked Write 9 Done Sent Done false
20429 6646 Acked Flush 0 Done Sent Done false
20430 6647 Acked Write 4 Done Sent Done false
20431 6648 Acked Read 5 Done Sent Done false
20432 6649 Acked Write 9 Done Sent Done false
20433 6650 Acked Write 8 Done Sent Done false
20434 6651 Acked Read 3 Done Sent Done false
20435 6652 Acked Write 6 Done Sent Done false
20436 6653 Acked Read 5 Done Sent Done false
20437 6654 Acked Write 6 Done Sent Done false
20438 6655 Acked Write 9 Done Sent Done false
20439 6656 Acked Read 3 Done Sent Done false
20440 6657 Acked Write 2 Done Sent Done false
20441 6658 Acked Read 7 Done Sent Done false
20442 6659 Acked Flush 0 Done Sent Done false
20443 6660 Acked Write 6 Done Sent Done false
20444 6661 Acked Flush 0 Done Sent Done false
20445 6662 Acked Read 10 Done Sent Done false
20446 6663 Acked Read 3 Done Sent Done false
20447 6664 Acked Write 3 Done Sent Done false
20448 6665 Acked Write 9 Done Sent Done false
20449 6666 Acked Read 1 Done Sent Done false
20450 6667 Acked Write 5 Done Sent Done false
20451 6668 Acked Read 9 Done Sent Done false
20452 6669 Acked Write 5 Done Sent Done false
20453 6670 Acked Read 6 Done Sent Done false
20454 6671 Acked Flush 0 Done Sent Done false
20455 6672 Acked Write 7 Done Sent Done false
20456 6673 Acked Read 6 Done Sent Done false
20457 6674 Acked Read 4 Done Sent Done false
20458 6675 Acked Write 1 Done Sent Done false
20459 6676 Acked Read 1 Done Sent Done false
20460 6677 Acked Read 10 Done Sent Done false
20461 6678 Acked Read 3 Done Sent Done false
20462 6679 Acked Write 6 Done Sent Done false
20463 6680 Acked Read 8 Done Sent Done false
20464 6681 Acked Read 10 Done Sent Done false
20465 6682 Acked Read 10 Done Sent Done false
20466 6683 Acked Write 1 Done Sent Done false
20467 6684 Acked Flush 0 Done Sent Done false
20468 6685 Acked Read 3 Done Sent Done false
20469 6686 Acked Write 1 Done Sent Done false
20470 6687 Acked Write 5 Done Sent Done false
20471 6688 Acked Read 10 Done Sent Done false
20472 6689 Acked Write 8 Done Sent Done false
20473 6690 Acked Read 5 Done Sent Done false
20474 6691 Acked Write 6 Done Sent Done false
20475 6692 Acked Read 5 Done Sent Done false
20476 6693 Acked Write 3 Done Sent Done false
20477 6694 Acked Write 10 Done Sent Done false
20478 6695 Acked Write 5 Done Sent Done false
20479 6696 Acked Read 5 Done Sent Done false
20480 6697 Acked Write 3 Done Sent Done false
20481 6698 Acked Flush 0 Done Sent Done false
20482 6699 Acked Write 9 Done Sent Done false
20483 6700 Acked Write 2 Done Sent Done false
20484 6701 Acked Flush 0 Done Sent Done false
20485 6702 Acked Write 7 Done Sent Done false
20486 6703 Acked Flush 0 Done Sent Done false
20487 6704 Acked Read 9 Done Sent Done false
20488 6705 Acked Write 4 Done Sent Done false
20489 6706 Acked Write 7 Done Sent Done false
20490 6707 Acked Write 9 Done Sent Done false
20491 6708 Acked Write 2 Done Sent Done false
20492 6709 Acked Write 9 Done Sent Done false
20493 6710 Acked Write 2 Done Sent Done false
20494 6711 Acked Write 6 Done Sent Done false
20495 6712 Acked Read 10 Done Sent Done false
20496 6713 Acked Write 5 Done Sent Done false
20497 6714 Acked Read 10 Done Sent Done false
20498 6715 Acked Read 10 Done Sent Done false
20499 6716 Acked Read 10 Done Sent Done false
20500 6717 Acked Write 8 Done Sent Done false
20501 6718 Acked Write 9 Done Sent Done false
20502 6719 Acked Read 5 Done Sent Done false
20503 6720 Acked Write 3 Done Sent Done false
20504 6721 Acked Read 4 Done Sent Done false
20505 6722 Acked Write 4 Done Sent Done false
20506 6723 Acked Write 8 Done Sent Done false
20507 6724 Acked Write 10 Done Sent Done false
20508 6725 Acked Read 3 Done Sent Done false
20509 6726 Acked Read 4 Done Sent Done false
20510 6727 Acked Write 2 Done Sent Done false
20511 6728 Acked Flush 0 Done Sent Done false
20512 6729 Acked Flush 0 Done Sent Done false
20513 6730 Acked Read 9 Done Sent Done false
20514 6731 Acked Read 2 Done Sent Done false
20515 6732 Acked Read 2 Done Sent Done false
20516 6733 Acked Read 6 Done Sent Done false
20517 6734 Acked Flush 0 Done Sent Done false
20518 STATES DS:0 DS:1 DS:2 TOTAL
20519 Sent 0 302 0 302
20520 Done 302 0 302 604
20521 Skipped 0 0 0 0
20522 Error 0 0 0 0
20523 Last Flush: Some(JobId(6734)) Some(JobId(6432)) Some(JobId(6734))
20524 Downstairs last five completed: 6432 6431 6430 6429 6428
20525 Upstairs last five completed: 6734 6733 6732 6731 6730
20526 CLIENT: Up:0 ds:302 act:2
205272025-01-23T18:52:11.443ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
205282025-01-23T18:52:11.444ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61380) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
205292025-01-23T18:52:11.445ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
205302025-01-23T18:52:11.445ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
205312025-01-23T18:52:11.445ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
205322025-01-23T18:52:11.445ZINFOcrucible: send last flush ID to this DS: Some(JobId(6432)) = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
205332025-01-23T18:52:11.445ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
205342025-01-23T18:52:11.445ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(6432)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
205352025-01-23T18:52:11.445ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(6432)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
20536 ----------------------------------------------------------------
20537 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20538 Downstairs last five completed: 6734 6733 6732 6731 6730
20539 Upstairs last five completed: 6734 6733 6732 6731 6730
20540 CLIENT: Up:0 ds:0 act:3
20541 Replay: All jobs finished
20542 Main task has work to do, go find it
20543 got dsc Stop(2)
20544 stop 2
20545 Jan 23 18:52:13.700 INFO request completed, latency_us: 342, response_code: 204, uri: /stop/cid/2, method: POST, req_id: a2ddc002-f429-49b8-b9c7-c0488175affb, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
20546 [8830] Got stop action so:false kr:false
20547 Jan 23 18:52:13.701 INFO request completed, latency_us: 165, response_code: 200, uri: /state/cid/2, method: GET, req_id: abdff93e-4ac8-4fbf-90f1-04099aaad9bb, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
205482025-01-23T18:52:13.701ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
205492025-01-23T18:52:13.701ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
205502025-01-23T18:52:13.701ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
205512025-01-23T18:52:13.702ZINFOcrucible: client task is exiting = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
20552 [8830] Exited with: ExitStatus(unix_wait_status(9))
20553 [8830][2] reports Exit
20554 Jan 23 18:52:17.703 INFO request completed, latency_us: 248, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3e382571-3335-4577-bca1-e7f884d7109e, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
20555 001/300 Read block 708 len 4096
20556 002/300 Read block 13 len 12288
20557 003/300 Write block 2496 len 20480 data: 5 8 7 7 6
20558 004/300 Write block 3717 len 4096 data: 6
20559 005/300 Write block 1307 len 36864 data: 5 5 5 5 5 8 8 7 5
20560 006/300 Read block 3498 len 24576
20561 007/300 Read block 3267 len 20480
20562 008/300 Write block 477 len 36864 data: 5 6 5 5 6 4 4 4 5
20563 009/300 Write block 1462 len 4096 data: 5
20564 010/300 Write block 1106 len 20480 data: 5 4 5 4 3
20565 011/300 Flush
20566 012/300 Write block 1404 len 8192 data: 3 3
20567 013/300 Read block 4810 len 24576
20568 014/300 Read block 783 len 24576
20569 015/300 Write block 1961 len 36864 data: 3 3 4 4 4 5 5 6 5
20570 016/300 Read block 1694 len 28672
20571 017/300 Read block 2414 len 40960
20572 018/300 Write block 1950 len 12288 data: 4 4 4
20573 019/300 Write block 1069 len 32768 data: 7 7 5 4 6 5 5 6
20574 020/300 Read block 1096 len 20480
20575 021/300 Write block 3537 len 36864 data: 8 7 7 5 4 4 4 4 5
20576 022/300 Read block 2300 len 16384
20577 023/300 Read block 2773 len 16384
20578 024/300 Read block 1153 len 28672
20579 025/300 Write block 2053 len 4096 data: 6
20580 026/300 Read block 455 len 32768
20581 027/300 Read block 4389 len 4096
20582 028/300 Read block 3753 len 40960
20583 029/300 Read block 2449 len 24576
20584 030/300 Read block 1831 len 8192
20585 031/300 Read block 366 len 32768
20586 032/300 Read block 3837 len 24576
20587 033/300 Read block 4443 len 24576
20588 034/300 Read block 1644 len 12288
20589 035/300 Write block 4752 len 16384 data: 5 6 5 7
20590 036/300 Write block 762 len 40960 data: 8 7 6 4 4 4 4 4 4 4
20591 037/300 Write block 4549 len 40960 data: 3 3 3 2 2 2 3 2 3 4
20592 038/300 Read block 4220 len 4096
20593 039/300 Read block 3621 len 28672
20594 040/300 Flush
20595 041/300 Write block 1131 len 20480 data: 5 6 7 8 8
20596 042/300 Flush
20597 043/300 Read block 3335 len 20480
20598 044/300 Write block 2103 len 28672 data: 8 8 8 9 7 8 7
20599 045/300 Flush
20600 046/300 Flush
20601 047/300 Read block 2134 len 16384
20602 048/300 Read block 4178 len 20480
20603 049/300 Read block 3428 len 32768
20604 050/300 Read block 2991 len 8192
20605 051/300 Flush
20606 052/300 Read block 3894 len 20480
20607 053/300 Write block 4115 len 16384 data: 4 4 2 2
20608 054/300 Write block 4359 len 8192 data: 5 4
20609 055/300 Write block 3877 len 4096 data: 6
20610 056/300 Read block 3839 len 8192
20611 057/300 Read block 509 len 16384
20612 058/300 Write block 1449 len 4096 data: 2
20613 059/300 Write block 875 len 20480 data: 5 5 4 4 4
20614 060/300 Read block 4033 len 20480
20615 061/300 Write block 3042 len 28672 data: 8 9 8 10 10 9 10
20616 062/300 Write block 2030 len 12288 data: 3 3 3
20617 063/300 Write block 4916 len 8192 data: 7 6
20618 064/300 Read block 1448 len 36864
20619 065/300 Write block 385 len 32768 data: 5 3 3 3 3 3 4 4
20620 066/300 Read block 1852 len 20480
20621 067/300 Read block 1097 len 4096
20622 068/300 Flush
20623 069/300 Read block 4817 len 12288
20624 070/300 Write block 859 len 40960 data: 4 3 3 4 5 5 5 4 4 5
20625 071/300 Write block 1401 len 40960 data: 3 3 3 4 4 3 3 3 2 2
20626 072/300 Read block 4525 len 4096
20627 073/300 Read block 4243 len 36864
20628 074/300 Write block 2786 len 20480 data: 3 2 4 4 4
20629 075/300 Read block 1395 len 24576
20630 076/300 Write block 1865 len 40960 data: 7 7 8 6 7 6 5 5 5 3
20631 077/300 Write block 1973 len 36864 data: 4 5 5 4 4 4 4 5 6
20632 078/300 Read block 3428 len 28672
20633 079/300 Read block 4495 len 32768
20634 080/300 Write block 1232 len 16384 data: 6 6 8 7
20635 081/300 Read block 1331 len 36864
20636 082/300 Write block 3276 len 24576 data: 5 5 3 4 6 5
20637 083/300 Write block 671 len 36864 data: 6 5 6 5 6 6 5 5 5
20638 084/300 Write block 3205 len 8192 data: 2 2
20639 085/300 Write block 3022 len 36864 data: 5 5 6 6 6 5 5 5 6
20640 086/300 Read block 1242 len 32768
20641 087/300 Read block 276 len 4096
20642 088/300 Read block 1947 len 4096
20643 089/300 Read block 2858 len 24576
20644 090/300 Write block 1770 len 8192 data: 4 5
20645 091/300 Write block 4939 len 24576 data: 4 5 5 4 4 5
20646 092/300 Write block 3557 len 32768 data: 5 6 5 5 3 3 3 3
20647 093/300 Write block 1835 len 8192 data: 4 4
20648 094/300 Read block 3109 len 20480
20649 095/300 Read block 2890 len 40960
20650 096/300 Read block 1401 len 24576
20651 097/300 Write block 2122 len 12288 data: 7 9 9
20652 098/300 Write block 1069 len 32768 data: 8 8 6 5 7 6 6 7
20653 099/300 Write block 2776 len 4096 data: 7
20654 100/300 Write block 2230 len 40960 data: 3 3 4 5 4 3 3 3 4 4
20655 101/300 Write block 2570 len 12288 data: 3 3 3
20656 102/300 Write block 1356 len 16384 data: 4 2 3 4
20657 103/300 Write block 4885 len 40960 data: 6 5 5 5 4 4 4 5 5 4
20658 104/300 Read block 2178 len 24576
20659 105/300 Flush
20660 106/300 Flush
20661 107/300 Write block 4139 len 16384 data: 5 5 6 7
20662 108/300 Read block 192 len 32768
20663 109/300 Read block 977 len 16384
20664 110/300 Read block 3374 len 4096
20665 111/300 Write block 1771 len 24576 data: 6 6 7 8 7 7
20666 112/300 Read block 4908 len 4096
20667 113/300 Flush
20668 114/300 Read block 4939 len 32768
20669 115/300 Read block 77 len 28672
20670 116/300 Read block 3055 len 36864
20671 117/300 Read block 961 len 20480
20672 118/300 Write block 2194 len 28672 data: 3 3 4 5 5 6 6
20673 119/300 Read block 1312 len 8192
20674 120/300 Write block 796 len 12288 data: 4 4 4
20675 121/300 Read block 3368 len 28672
20676 122/300 Write block 3884 len 24576 data: 6 8 7 6 6 5
20677 123/300 Flush
20678 124/300 Read block 1502 len 40960
20679 125/300 Read block 4521 len 16384
20680 126/300 Read block 4555 len 4096
20681 127/300 Read block 1426 len 36864
20682 128/300 Flush
20683 129/300 Read block 839 len 20480
20684 130/300 Read block 3165 len 36864
20685 131/300 Write block 2193 len 40960 data: 2 4 4 5 6 6 7 7 7 5
20686 132/300 Write block 3477 len 4096 data: 4
20687 133/300 Write block 1750 len 40960 data: 2 2 2 2 3 2 2 2 4 4
20688 134/300 Read block 4688 len 40960
20689 135/300 Write block 1439 len 36864 data: 2 3 4 4 4 4 3 2 2
20690 136/300 Read block 3642 len 20480
20691 137/300 Write block 4899 len 32768 data: 5 4 4 5 5 5 6 6
20692 138/300 Read block 2305 len 16384
20693 139/300 Write block 3196 len 8192 data: 3 4
20694 140/300 Read block 195 len 32768
20695 141/300 Write block 2144 len 4096 data: 4
20696 142/300 Read block 1481 len 28672
20697 143/300 Read block 4501 len 36864
20698 144/300 Write block 3802 len 40960 data: 7 6 6 5 8 7 6 4 3 2
20699 145/300 Read block 2765 len 20480
20700 146/300 Read block 170 len 24576
20701 147/300 Read block 3412 len 16384
20702 148/300 Write block 3416 len 4096 data: 4
20703 149/300 Read block 4623 len 16384
20704 150/300 Write block 3750 len 40960 data: 5 5 7 7 6 6 5 4 4 4
20705 151/300 Write block 1891 len 4096 data: 3
20706 152/300 Write block 3684 len 20480 data: 4 4 4 5 5
20707 153/300 Write block 3693 len 8192 data: 7 5
20708 154/300 Read block 1907 len 36864
20709 155/300 Write block 2365 len 8192 data: 4 3
20710 156/300 Write block 1249 len 4096 data: 5
20711 157/300 Write block 3375 len 12288 data: 4 4 4
20712 158/300 Read block 3526 len 4096
20713 159/300 Read block 2046 len 28672
20714 160/300 Write block 1144 len 36864 data: 6 8 8 7 6 6 6 6 7
20715 161/300 Write block 4282 len 4096 data: 3
20716 162/300 Write block 2803 len 12288 data: 4 4 3
20717 163/300 Read block 1611 len 4096
20718 164/300 Read block 3711 len 16384
20719 165/300 Flush
20720 166/300 Read block 4099 len 12288
20721 167/300 Read block 2311 len 16384
20722 168/300 Read block 1441 len 20480
20723 169/300 Write block 967 len 32768 data: 7 6 4 4 3 3 3 3
20724 170/300 Flush
20725 171/300 Write block 652 len 12288 data: 5 5 6
20726 172/300 Read block 2367 len 12288
20727 173/300 Write block 2624 len 40960 data: 6 7 7 5 7 7 6 5 7 6
20728 174/300 Read block 1421 len 40960
20729 175/300 Read block 3719 len 40960
20730 176/300 Read block 4982 len 36864
20731 177/300 Read block 1538 len 12288
20732 178/300 Write block 3832 len 40960 data: 5 3 3 3 3 3 4 4 4 4
20733 179/300 Write block 832 len 20480 data: 3 4 4 6 5
20734 180/300 Write block 4427 len 32768 data: 5 6 5 5 6 5 5 3
20735 181/300 Read block 2655 len 20480
20736 182/300 Read block 3306 len 40960
20737 183/300 Write block 2327 len 24576 data: 6 4 4 6 7 6
20738 184/300 Write block 932 len 20480 data: 4 4 4 6 7
20739 185/300 Write block 1468 len 36864 data: 5 5 4 3 3 3 3 4 5
20740 186/300 Write block 2181 len 8192 data: 2 2
20741 187/300 Read block 3139 len 36864
20742 188/300 Write block 3614 len 12288 data: 4 4 4
20743 189/300 Flush
20744 190/300 Flush
20745 191/300 Write block 1754 len 8192 data: 4 3
20746 192/300 Read block 582 len 40960
20747 193/300 Read block 3001 len 28672
20748 194/300 Flush
20749 195/300 Write block 1773 len 32768 data: 8 9 8 8 7 5 4 4
20750 196/300 Write block 640 len 8192 data: 3 3
20751 197/300 Write block 3878 len 8192 data: 7 7
20752 198/300 Read block 2522 len 16384
20753 199/300 Write block 618 len 28672 data: 4 4 4 4 4 3 5
20754 200/300 Read block 4409 len 28672
20755 201/300 Write block 3402 len 28672 data: 3 3 3 2 2 2 4
20756 202/300 Write block 1019 len 40960 data: 7 8 9 6 7 6 5 6 6 4
20757 203/300 Write block 3834 len 8192 data: 4 4
20758 204/300 Read block 9 len 12288
20759 205/300 Read block 1889 len 12288
20760 206/300 Write block 4499 len 36864 data: 5 5 5 6 4 5 5 5 5
20761 207/300 Read block 2327 len 12288
20762 208/300 Write block 2217 len 4096 data: 2
20763 209/300 Read block 4396 len 24576
20764 210/300 Flush
20765 211/300 Write block 3020 len 12288 data: 4 6 6
20766 212/300 Write block 4059 len 12288 data: 3 4 4
20767 213/300 Read block 1348 len 4096
20768 214/300 Flush
20769 215/300 Read block 1569 len 4096
20770 216/300 Write block 3326 len 36864 data: 3 4 4 5 5 5 5 4 4
20771 217/300 Write block 2260 len 16384 data: 3 3 3 2
20772 218/300 Read block 3240 len 4096
20773 219/300 Read block 4774 len 12288
20774 220/300 Read block 120 len 20480
20775 221/300 Read block 3169 len 8192
20776 222/300 Flush
20777 223/300 Write block 3834 len 12288 data: 5 5 4
20778 224/300 Write block 147 len 12288 data: 5 6 6
20779 225/300 Read block 433 len 16384
20780 226/300 Read block 2318 len 24576
20781 227/300 Write block 1637 len 40960 data: 5 5 5 4 5 5 4 6 6 6
20782 228/300 Read block 4272 len 24576
20783 229/300 Read block 3816 len 28672
20784 230/300 Read block 4874 len 12288
20785 231/300 Read block 620 len 12288
20786 232/300 Read block 2876 len 36864
20787 233/300 Read block 61 len 12288
20788 234/300 Write block 4267 len 4096 data: 6
20789 235/300 Read block 3079 len 28672
20790 236/300 Flush
20791 237/300 Read block 3102 len 32768
20792 238/300 Write block 3877 len 28672 data: 7 8 8 8 7 6 6
20793 239/300 Write block 3655 len 8192 data: 3 3
20794 240/300 Read block 478 len 24576
20795 241/300 Write block 4743 len 32768 data: 3 4 4 4 5 4 4 5
20796 242/300 Write block 2153 len 36864 data: 4 5 6 8 8 7 7 7 6
20797 243/300 Read block 2259 len 16384
20798 244/300 Flush
20799 245/300 Write block 1027 len 40960 data: 7 5 4 6 5 5 3 3 3 5
20800 246/300 Write block 1130 len 12288 data: 4 6 7
20801 247/300 Read block 2977 len 20480
20802 248/300 Read block 638 len 16384
20803 249/300 Write block 4107 len 24576 data: 2 2 2 3 4 4
20804 250/300 Read block 4733 len 28672
20805 251/300 Read block 3354 len 32768
20806 252/300 Write block 4203 len 36864 data: 4 4 3 2 3 3 3 2 3
20807 253/300 Write block 4265 len 12288 data: 3 3 7
20808 254/300 Write block 3884 len 36864 data: 7 9 8 7 7 6 5 5 5
20809 255/300 Write block 3484 len 16384 data: 3 3 4 3
20810 256/300 Write block 2945 len 12288 data: 4 4 4
20811 257/300 Write block 3897 len 36864 data: 6 6 6 6 6 6 6 6 6
20812 258/300 Read block 3256 len 36864
20813 259/300 Read block 1304 len 36864
20814 260/300 Read block 2542 len 40960
20815 261/300 Read block 3119 len 36864
20816 262/300 Write block 3566 len 12288 data: 4 5 6
20817 263/300 Read block 1197 len 24576
20818 264/300 Read block 31 len 4096
20819 265/300 Read block 1991 len 36864
20820 266/300 Read block 3466 len 24576
20821 267/300 Read block 229 len 28672
20822 268/300 Flush
20823 269/300 Read block 4571 len 20480
20824 270/300 Read block 4803 len 4096
20825 271/300 Write block 2506 len 20480 data: 3 3 3 3 3
20826 272/300 Read block 1043 len 32768
20827 273/300 Read block 4205 len 40960
20828 274/300 Write block 2991 len 20480 data: 4 3 4 5 7
20829 275/300 Write block 3319 len 28672 data: 4 4 3 4 5 4 3
20830 276/300 Write block 3305 len 40960 data: 3 4 4 5 8 9 8 7 7 5
20831 277/300 Write block 4816 len 40960 data: 3 2 3 3 3 3 5 6 6 4
20832 278/300 Read block 1365 len 8192
20833 279/300 Write block 1488 len 36864 data: 5 6 6 7 6 5 4 5 5
20834 280/300 Read block 3046 len 28672
20835 281/300 Read block 1550 len 8192
20836 282/300 Read block 2430 len 36864
20837 283/300 Flush
20838 284/300 Write block 3317 len 12288 data: 6 4 5
20839 285/300 Write block 427 len 16384 data: 5 9 10 9
20840 286/300 Read block 1669 len 24576
20841 287/300 Read block 48 len 32768
20842 288/300 Write block 3005 len 16384 data: 7 6 6 4
20843 289/300 Flush
20844 290/300 Write block 1558 len 36864 data: 6 7 6 5 4 5 5 4 5
20845 291/300 Write block 2304 len 8192 data: 6 6
20846 292/300 Read block 1278 len 4096
20847 293/300 Write block 835 len 12288 data: 7 6 5
20848 294/300 Write block 2210 len 12288 data: 5 5 5
20849 295/300 Read block 220 len 12288
20850 296/300 Write block 4458 len 16384 data: 6 5 6 6
20851 297/300 Write block 4856 len 28672 data: 4 3 3 4 4 3 2
20852 298/300 Write block 4643 len 16384 data: 6 6 6 7
20853 299/300 Write block 1900 len 8192 data: 4 4
20854 300/300 Write block 1487 len 16384 data: 4 6 7 7
20855 301/300 Read block 803 len 20480
20856 Main task has work to do, go find it
20857 got dsc Start(2)
20858 start 2
20859 Jan 23 18:52:17.938 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: 5867cdac-a6f5-43b7-a5fc-f36713c79bf4, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
20860 Replay: started 2, returned:Ok(())
20861 ----------------------------------------------------------------
20862 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20863 States: Active Active Offline
20864 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20865 6735 Acked Read 1 Done Done Sent false
20866 6736 Acked Read 3 Done Done Sent false
20867 6737 Acked Write 5 Done Done Sent false
20868 6738 Acked Write 1 Done Done Sent false
20869 6739 Acked Write 9 Done Done Sent false
20870 6740 Acked Read 6 Done Done Sent false
20871 6741 Acked Read 5 Done Done Sent false
20872 6742 Acked Write 9 Done Done Sent false
20873 6743 Acked Write 1 Done Done Sent false
20874 6744 Acked Write 5 Done Done Sent false
20875 6745 Acked Flush 0 Done Done Sent false
20876 6746 Acked Write 2 Done Done Sent false
20877 6747 Acked Read 6 Done Done Sent false
20878 6748 Acked Read 6 Done Done Sent false
20879 6749 Acked Write 9 Done Done Sent false
20880 6750 Acked Read 7 Done Done Sent false
20881 6751 Acked Read 10 Done Done Sent false
20882 6752 Acked Write 3 Done Done Sent false
20883 6753 Acked Write 8 Done Done Sent false
20884 6754 Acked Read 5 Done Done Sent false
20885 6755 Acked Write 9 Done Done Sent false
20886 6756 Acked Read 4 Done Done Sent false
20887 6757 Acked Read 4 Done Done Sent false
20888 6758 Acked Read 7 Done Done Sent false
20889 6759 Acked Write 1 Done Done Sent false
20890 6760 Acked Read 8 Done Done Sent false
20891 6761 Acked Read 1 Done Done Sent false
20892 6762 Acked Read 10 Done Done Sent false
20893 6763 Acked Read 6 Done Done Sent false
20894 6764 Acked Read 2 Done Done Sent false
20895 6765 Acked Read 8 Done Done Sent false
20896 6766 Acked Read 6 Done Done Sent false
20897 6767 Acked Read 6 Done Done Sent false
20898 6768 Acked Read 3 Done Done Sent false
20899 6769 Acked Write 4 Done Done Sent false
20900 6770 Acked Write 10 Done Done Sent false
20901 6771 Acked Write 10 Done Done Sent false
20902 6772 Acked Read 1 Done Done Sent false
20903 6773 Acked Read 7 Done Done Sent false
20904 6774 Acked Flush 0 Done Done Sent false
20905 6775 Acked Write 5 Done Done Sent false
20906 6776 Acked Flush 0 Done Done Sent false
20907 6777 Acked Read 5 Done Done Sent false
20908 6778 Acked Write 7 Done Done Sent false
20909 6779 Acked Flush 0 Done Done Sent false
20910 6780 Acked Flush 0 Done Done Sent false
20911 6781 Acked Read 4 Done Done Sent false
20912 6782 Acked Read 5 Done Done Sent false
20913 6783 Acked Read 8 Done Done Sent false
20914 6784 Acked Read 2 Done Done Sent false
20915 6785 Acked Flush 0 Done Done Sent false
20916 6786 Acked Read 5 Done Done Sent false
20917 6787 Acked Write 4 Done Done Sent false
20918 6788 Acked Write 2 Done Done Sent false
20919 6789 Acked Write 1 Done Done Sent false
20920 6790 Acked Read 2 Done Done Sent false
20921 6791 Acked Read 4 Done Done Sent false
20922 6792 Acked Write 1 Done Done Sent false
20923 6793 Acked Write 5 Done Done Sent false
20924 6794 Acked Read 5 Done Done Sent false
20925 6795 Acked Write 7 Done Done Sent false
20926 6796 Acked Write 3 Done Done Sent false
20927 6797 Acked Write 2 Done Done Sent false
20928 6798 Acked Read 9 Done Done Sent false
20929 6799 Acked Write 8 Done Done Sent false
20930 6800 Acked Read 5 Done Done Sent false
20931 6801 Acked Read 1 Done Done Sent false
20932 6802 Acked Flush 0 Done Done Sent false
20933 6803 Acked Read 3 Done Done Sent false
20934 6804 Acked Write 10 Done Done Sent false
20935 6805 Acked Write 10 Done Done Sent false
20936 6806 Acked Read 1 Done Done Sent false
20937 6807 Acked Read 9 Done Done Sent false
20938 6808 Acked Write 5 Done Done Sent false
20939 6809 Acked Read 6 Done Done Sent false
20940 6810 Acked Write 10 Done Done Sent false
20941 6811 Acked Write 9 Done Done Sent false
20942 6812 Acked Read 7 Done Done Sent false
20943 6813 Acked Read 8 Done Done Sent false
20944 6814 Acked Write 4 Done Done Sent false
20945 6815 Acked Read 9 Done Done Sent false
20946 6816 Acked Write 6 Done Done Sent false
20947 6817 Acked Write 9 Done Done Sent false
20948 6818 Acked Write 2 Done Done Sent false
20949 6819 Acked Write 9 Done Done Sent false
20950 6820 Acked Read 8 Done Done Sent false
20951 6821 Acked Read 1 Done Done Sent false
20952 6822 Acked Read 1 Done Done Sent false
20953 6823 Acked Read 6 Done Done Sent false
20954 6824 Acked Write 2 Done Done Sent false
20955 6825 Acked Write 6 Done Done Sent false
20956 6826 Acked Write 8 Done Done Sent false
20957 6827 Acked Write 2 Done Done Sent false
20958 6828 Acked Read 5 Done Done Sent false
20959 6829 Acked Read 10 Done Done Sent false
20960 6830 Acked Read 6 Done Done Sent false
20961 6831 Acked Write 3 Done Done Sent false
20962 6832 Acked Write 8 Done Done Sent false
20963 6833 Acked Write 1 Done Done Sent false
20964 6834 Acked Write 10 Done Done Sent false
20965 6835 Acked Write 3 Done Done Sent false
20966 6836 Acked Write 4 Done Done Sent false
20967 6837 Acked Write 10 Done Done Sent false
20968 6838 Acked Read 6 Done Done Sent false
20969 6839 Acked Flush 0 Done Done Sent false
20970 6840 Acked Flush 0 Done Done Sent false
20971 6841 Acked Write 4 Done Done Sent false
20972 6842 Acked Read 8 Done Done Sent false
20973 6843 Acked Read 4 Done Done Sent false
20974 6844 Acked Read 1 Done Done Sent false
20975 6845 Acked Write 6 Done Done Sent false
20976 6846 Acked Read 1 Done Done Sent false
20977 6847 Acked Flush 0 Done Done Sent false
20978 6848 Acked Read 8 Done Done Sent false
20979 6849 Acked Read 7 Done Done Sent false
20980 6850 Acked Read 9 Done Done Sent false
20981 6851 Acked Read 5 Done Done Sent false
20982 6852 Acked Write 7 Done Done Sent false
20983 6853 Acked Read 2 Done Done Sent false
20984 6854 Acked Write 3 Done Done Sent false
20985 6855 Acked Read 7 Done Done Sent false
20986 6856 Acked Write 6 Done Done Sent false
20987 6857 Acked Flush 0 Done Done Sent false
20988 6858 Acked Read 10 Done Done Sent false
20989 6859 Acked Read 4 Done Done Sent false
20990 6860 Acked Read 1 Done Done Sent false
20991 6861 Acked Read 9 Done Done Sent false
20992 6862 Acked Flush 0 Done Done Sent false
20993 6863 Acked Read 5 Done Done Sent false
20994 6864 Acked Read 9 Done Done Sent false
20995 6865 Acked Write 10 Done Done Sent false
20996 6866 Acked Write 1 Done Done Sent false
20997 6867 Acked Write 10 Done Done Sent false
20998 6868 Acked Read 10 Done Done Sent false
20999 6869 Acked Write 9 Done Done Sent false
21000 6870 Acked Read 5 Done Done Sent false
21001 6871 Acked Write 8 Done Done Sent false
21002 6872 Acked Read 4 Done Done Sent false
21003 6873 Acked Write 2 Done Done Sent false
21004 6874 Acked Read 8 Done Done Sent false
21005 6875 Acked Write 1 Done Done Sent false
21006 6876 Acked Read 7 Done Done Sent false
21007 6877 Acked Read 9 Done Done Sent false
21008 6878 Acked Write 10 Done Done Sent false
21009 6879 Acked Read 5 Done Done Sent false
21010 6880 Acked Read 6 Done Done Sent false
21011 6881 Acked Read 4 Done Done Sent false
21012 6882 Acked Write 1 Done Done Sent false
21013 6883 Acked Read 4 Done Done Sent false
21014 6884 Acked Write 10 Done Done Sent false
21015 6885 Acked Write 1 Done Done Sent false
21016 6886 Acked Write 5 Done Done Sent false
21017 6887 Acked Write 2 Done Done Sent false
21018 6888 Acked Read 9 Done Done Sent false
21019 6889 Acked Write 2 Done Done Sent false
21020 6890 Acked Write 1 Done Done Sent false
21021 6891 Acked Write 3 Done Done Sent false
21022 6892 Acked Read 1 Done Done Sent false
21023 6893 Acked Read 7 Done Done Sent false
21024 6894 Acked Write 9 Done Done Sent false
21025 6895 Acked Write 1 Done Done Sent false
21026 6896 Acked Write 3 Done Done Sent false
21027 6897 Acked Read 1 Done Done Sent false
21028 6898 Acked Read 4 Done Done Sent false
21029 6899 Acked Flush 0 Done Done Sent false
21030 6900 Acked Read 3 Done Done Sent false
21031 6901 Acked Read 4 Done Done Sent false
21032 6902 Acked Read 5 Done Done Sent false
21033 6903 Acked Write 8 Done Done Sent false
21034 6904 Acked Flush 0 Done Done Sent false
21035 6905 Acked Write 3 Done Done Sent false
21036 6906 Acked Read 3 Done Done Sent false
21037 6907 Acked Write 10 Done Done Sent false
21038 6908 Acked Read 10 Done Done Sent false
21039 6909 Acked Read 10 Done Done Sent false
21040 6910 Acked Read 9 Done Done Sent false
21041 6911 Acked Read 3 Done Done Sent false
21042 6912 Acked Write 10 Done Done Sent false
21043 6913 Acked Write 5 Done Done Sent false
21044 6914 Acked Write 8 Done Done Sent false
21045 6915 Acked Read 5 Done Done Sent false
21046 6916 Acked Read 10 Done Done Sent false
21047 6917 Acked Write 6 Done Done Sent false
21048 6918 Acked Write 5 Done Done Sent false
21049 6919 Acked Write 9 Done Done Sent false
21050 6920 Acked Write 2 Done Done Sent false
21051 6921 Acked Read 9 Done Done Sent false
21052 6922 Acked Write 3 Done Done Sent false
21053 6923 Acked Flush 0 Done Done Sent false
21054 6924 Acked Flush 0 Done Done Sent false
21055 6925 Acked Write 2 Done Done Sent false
21056 6926 Acked Read 10 Done Done Sent false
21057 6927 Acked Read 7 Done Done Sent false
21058 6928 Acked Flush 0 Done Done Sent false
21059 6929 Acked Write 8 Done Done Sent false
21060 6930 Acked Write 2 Done Done Sent false
21061 6931 Acked Write 2 Done Done Sent false
21062 6932 Acked Read 4 Done Done Sent false
21063 6933 Acked Write 7 Done Done Sent false
21064 6934 Acked Read 7 Done Done Sent false
21065 6935 Acked Write 7 Done Done Sent false
21066 6936 Acked Write 10 Done Done Sent false
21067 6937 Acked Write 2 Done Done Sent false
21068 6938 Acked Read 3 Done Done Sent false
21069 6939 Acked Read 3 Done Done Sent false
21070 6940 Acked Write 9 Done Done Sent false
21071 6941 Acked Read 3 Done Done Sent false
21072 6942 Acked Write 1 Done Done Sent false
21073 6943 Acked Read 6 Done Done Sent false
21074 6944 Acked Flush 0 Done Done Sent false
21075 6945 Acked Write 3 Done Done Sent false
21076 6946 Acked Write 3 Done Done Sent false
21077 6947 Acked Read 1 Done Done Sent false
21078 6948 Acked Flush 0 Done Done Sent false
21079 6949 Acked Read 1 Done Done Sent false
21080 6950 Acked Write 9 Done Done Sent false
21081 6951 Acked Write 4 Done Done Sent false
21082 6952 Acked Read 1 Done Done Sent false
21083 6953 Acked Read 3 Done Done Sent false
21084 6954 Acked Read 5 Done Done Sent false
21085 6955 Acked Read 2 Done Done Sent false
21086 6956 Acked Flush 0 Done Done Sent false
21087 6957 Acked Write 3 Done Done Sent false
21088 6958 Acked Write 3 Done Done Sent false
21089 6959 Acked Read 4 Done Done Sent false
21090 6960 Acked Read 6 Done Done Sent false
21091 6961 Acked Write 10 Done Done Sent false
21092 6962 Acked Read 6 Done Done Sent false
21093 6963 Acked Read 7 Done Done Sent false
21094 6964 Acked Read 3 Done Done Sent false
21095 6965 Acked Read 3 Done Done Sent false
21096 6966 Acked Read 9 Done Done Sent false
21097 6967 Acked Read 3 Done Done Sent false
21098 6968 Acked Write 1 Done Done Sent false
21099 6969 Acked Read 7 Done Done Sent false
21100 6970 Acked Flush 0 Done Done Sent false
21101 6971 Acked Read 8 Done Done Sent false
21102 6972 Acked Write 7 Done Done Sent false
21103 6973 Acked Write 2 Done Done Sent false
21104 6974 Acked Read 6 Done Done Sent false
21105 6975 Acked Write 8 Done Done Sent false
21106 6976 Acked Write 9 Done Done Sent false
21107 6977 Acked Read 4 Done Done Sent false
21108 6978 Acked Flush 0 Done Done Sent false
21109 6979 Acked Write 10 Done Done Sent false
21110 6980 Acked Write 3 Done Done Sent false
21111 6981 Acked Read 5 Done Done Sent false
21112 6982 Acked Read 4 Done Done Sent false
21113 6983 Acked Write 6 Done Done Sent false
21114 6984 Acked Read 7 Done Done Sent false
21115 6985 Acked Read 8 Done Done Sent false
21116 6986 Acked Write 9 Done Done Sent false
21117 6987 Acked Write 3 Done Done Sent false
21118 6988 Acked Write 9 Done Done Sent false
21119 6989 Acked Write 4 Done Done Sent false
21120 6990 Acked Write 3 Done Done Sent false
21121 6991 Acked Write 9 Done Done Sent false
21122 6992 Acked Read 9 Done Done Sent false
21123 6993 Acked Read 9 Done Done Sent false
21124 6994 Acked Read 10 Done Done Sent false
21125 6995 Acked Read 9 Done Done Sent false
21126 6996 Acked Write 3 Done Done Sent false
21127 6997 Acked Read 6 Done Done Sent false
21128 6998 Acked Read 1 Done Done Sent false
21129 6999 Acked Read 9 Done Done Sent false
21130 7000 Acked Read 6 Done Done Sent false
21131 7001 Acked Read 7 Done Done Sent false
21132 7002 Acked Flush 0 Done Done Sent false
21133 7003 Acked Read 5 Done Done Sent false
21134 7004 Acked Read 1 Done Done Sent false
21135 7005 Acked Write 5 Done Done Sent false
21136 7006 Acked Read 8 Done Done Sent false
21137 7007 Acked Read 10 Done Done Sent false
21138 7008 Acked Write 5 Done Done Sent false
21139 7009 Acked Write 7 Done Done Sent false
21140 7010 Acked Write 10 Done Done Sent false
21141 7011 Acked Write 10 Done Done Sent false
21142 7012 Acked Read 2 Done Done Sent false
21143 7013 Acked Write 9 Done Done Sent false
21144 7014 Acked Read 7 Done Done Sent false
21145 7015 Acked Read 2 Done Done Sent false
21146 7016 Acked Read 9 Done Done Sent false
21147 7017 Acked Flush 0 Done Done Sent false
21148 7018 Acked Write 3 Done Done Sent false
21149 7019 Acked Write 4 Done Done Sent false
21150 7020 Acked Read 6 Done Done Sent false
21151 7021 Acked Read 8 Done Done Sent false
21152 7022 Acked Write 4 Done Done Sent false
21153 7023 Acked Flush 0 Done Done Sent false
21154 7024 Acked Write 9 Done Done Sent false
21155 7025 Acked Write 2 Done Done Sent false
21156 7026 Acked Read 1 Done Done Sent false
21157 7027 Acked Write 3 Done Done Sent false
21158 7028 Acked Write 3 Done Done Sent false
21159 7029 Acked Read 3 Done Done Sent false
21160 7030 Acked Write 4 Done Done Sent false
21161 7031 Acked Write 7 Done Done Sent false
21162 7032 Acked Write 4 Done Done Sent false
21163 7033 Acked Write 2 Done Done Sent false
21164 7034 Acked Write 4 Done Done Sent false
21165 7035 Acked Read 5 Done Done Sent false
21166 STATES DS:0 DS:1 DS:2 TOTAL
21167 Sent 0 0 301 301
21168 Done 301 301 0 602
21169 Skipped 0 0 0 0
21170 Error 0 0 0 0
21171 Last Flush: Some(JobId(7023)) Some(JobId(7023)) Some(JobId(6734))
21172 Downstairs last five completed: 6734 6733 6732 6731 6730
21173 Upstairs last five completed: 7035 7034 7033 7032 7031
21174 CLIENT: Up:0 ds:301 act:2
21175 [8830] got start action
21176 [8830] I am going to restart
21177 Starting downstairs at port 8830
21178 Make output file at "/tmp/dsc/downstairs-8830.txt"
21179 [8830][2] reports Starting
21180 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1220)
21181 [8830][2] reports Running
21182 ----------------------------------------------------------------
21183 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
21184 States: Active Active Offline
21185 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21186 6735 Acked Read 1 Done Done Sent false
21187 6736 Acked Read 3 Done Done Sent false
21188 6737 Acked Write 5 Done Done Sent false
21189 6738 Acked Write 1 Done Done Sent false
21190 6739 Acked Write 9 Done Done Sent false
21191 6740 Acked Read 6 Done Done Sent false
21192 6741 Acked Read 5 Done Done Sent false
21193 6742 Acked Write 9 Done Done Sent false
21194 6743 Acked Write 1 Done Done Sent false
21195 6744 Acked Write 5 Done Done Sent false
21196 6745 Acked Flush 0 Done Done Sent false
21197 6746 Acked Write 2 Done Done Sent false
21198 6747 Acked Read 6 Done Done Sent false
21199 6748 Acked Read 6 Done Done Sent false
21200 6749 Acked Write 9 Done Done Sent false
21201 6750 Acked Read 7 Done Done Sent false
21202 6751 Acked Read 10 Done Done Sent false
21203 6752 Acked Write 3 Done Done Sent false
21204 6753 Acked Write 8 Done Done Sent false
21205 6754 Acked Read 5 Done Done Sent false
21206 6755 Acked Write 9 Done Done Sent false
21207 6756 Acked Read 4 Done Done Sent false
21208 6757 Acked Read 4 Done Done Sent false
21209 6758 Acked Read 7 Done Done Sent false
21210 6759 Acked Write 1 Done Done Sent false
21211 6760 Acked Read 8 Done Done Sent false
21212 6761 Acked Read 1 Done Done Sent false
21213 6762 Acked Read 10 Done Done Sent false
21214 6763 Acked Read 6 Done Done Sent false
21215 6764 Acked Read 2 Done Done Sent false
21216 6765 Acked Read 8 Done Done Sent false
21217 6766 Acked Read 6 Done Done Sent false
21218 6767 Acked Read 6 Done Done Sent false
21219 6768 Acked Read 3 Done Done Sent false
21220 6769 Acked Write 4 Done Done Sent false
21221 6770 Acked Write 10 Done Done Sent false
21222 6771 Acked Write 10 Done Done Sent false
21223 6772 Acked Read 1 Done Done Sent false
21224 6773 Acked Read 7 Done Done Sent false
21225 6774 Acked Flush 0 Done Done Sent false
21226 6775 Acked Write 5 Done Done Sent false
21227 6776 Acked Flush 0 Done Done Sent false
21228 6777 Acked Read 5 Done Done Sent false
21229 6778 Acked Write 7 Done Done Sent false
21230 6779 Acked Flush 0 Done Done Sent false
21231 6780 Acked Flush 0 Done Done Sent false
21232 6781 Acked Read 4 Done Done Sent false
21233 6782 Acked Read 5 Done Done Sent false
21234 6783 Acked Read 8 Done Done Sent false
21235 6784 Acked Read 2 Done Done Sent false
21236 6785 Acked Flush 0 Done Done Sent false
21237 6786 Acked Read 5 Done Done Sent false
21238 6787 Acked Write 4 Done Done Sent false
21239 6788 Acked Write 2 Done Done Sent false
21240 6789 Acked Write 1 Done Done Sent false
21241 6790 Acked Read 2 Done Done Sent false
21242 6791 Acked Read 4 Done Done Sent false
21243 6792 Acked Write 1 Done Done Sent false
21244 6793 Acked Write 5 Done Done Sent false
21245 6794 Acked Read 5 Done Done Sent false
21246 6795 Acked Write 7 Done Done Sent false
21247 6796 Acked Write 3 Done Done Sent false
21248 6797 Acked Write 2 Done Done Sent false
21249 6798 Acked Read 9 Done Done Sent false
21250 6799 Acked Write 8 Done Done Sent false
21251 6800 Acked Read 5 Done Done Sent false
21252 6801 Acked Read 1 Done Done Sent false
21253 6802 Acked Flush 0 Done Done Sent false
21254 6803 Acked Read 3 Done Done Sent false
21255 6804 Acked Write 10 Done Done Sent false
21256 6805 Acked Write 10 Done Done Sent false
21257 6806 Acked Read 1 Done Done Sent false
21258 6807 Acked Read 9 Done Done Sent false
21259 6808 Acked Write 5 Done Done Sent false
21260 6809 Acked Read 6 Done Done Sent false
21261 6810 Acked Write 10 Done Done Sent false
21262 6811 Acked Write 9 Done Done Sent false
21263 6812 Acked Read 7 Done Done Sent false
21264 6813 Acked Read 8 Done Done Sent false
21265 6814 Acked Write 4 Done Done Sent false
21266 6815 Acked Read 9 Done Done Sent false
21267 6816 Acked Write 6 Done Done Sent false
21268 6817 Acked Write 9 Done Done Sent false
21269 6818 Acked Write 2 Done Done Sent false
21270 6819 Acked Write 9 Done Done Sent false
21271 6820 Acked Read 8 Done Done Sent false
21272 6821 Acked Read 1 Done Done Sent false
21273 6822 Acked Read 1 Done Done Sent false
21274 6823 Acked Read 6 Done Done Sent false
21275 6824 Acked Write 2 Done Done Sent false
21276 6825 Acked Write 6 Done Done Sent false
21277 6826 Acked Write 8 Done Done Sent false
21278 6827 Acked Write 2 Done Done Sent false
21279 6828 Acked Read 5 Done Done Sent false
21280 6829 Acked Read 10 Done Done Sent false
21281 6830 Acked Read 6 Done Done Sent false
21282 6831 Acked Write 3 Done Done Sent false
21283 6832 Acked Write 8 Done Done Sent false
21284 6833 Acked Write 1 Done Done Sent false
21285 6834 Acked Write 10 Done Done Sent false
21286 6835 Acked Write 3 Done Done Sent false
21287 6836 Acked Write 4 Done Done Sent false
21288 6837 Acked Write 10 Done Done Sent false
21289 6838 Acked Read 6 Done Done Sent false
21290 6839 Acked Flush 0 Done Done Sent false
21291 6840 Acked Flush 0 Done Done Sent false
21292 6841 Acked Write 4 Done Done Sent false
21293 6842 Acked Read 8 Done Done Sent false
21294 6843 Acked Read 4 Done Done Sent false
21295 6844 Acked Read 1 Done Done Sent false
21296 6845 Acked Write 6 Done Done Sent false
21297 6846 Acked Read 1 Done Done Sent false
21298 6847 Acked Flush 0 Done Done Sent false
21299 6848 Acked Read 8 Done Done Sent false
21300 6849 Acked Read 7 Done Done Sent false
21301 6850 Acked Read 9 Done Done Sent false
21302 6851 Acked Read 5 Done Done Sent false
21303 6852 Acked Write 7 Done Done Sent false
21304 6853 Acked Read 2 Done Done Sent false
21305 6854 Acked Write 3 Done Done Sent false
21306 6855 Acked Read 7 Done Done Sent false
21307 6856 Acked Write 6 Done Done Sent false
21308 6857 Acked Flush 0 Done Done Sent false
21309 6858 Acked Read 10 Done Done Sent false
21310 6859 Acked Read 4 Done Done Sent false
21311 6860 Acked Read 1 Done Done Sent false
21312 6861 Acked Read 9 Done Done Sent false
21313 6862 Acked Flush 0 Done Done Sent false
21314 6863 Acked Read 5 Done Done Sent false
21315 6864 Acked Read 9 Done Done Sent false
21316 6865 Acked Write 10 Done Done Sent false
21317 6866 Acked Write 1 Done Done Sent false
21318 6867 Acked Write 10 Done Done Sent false
21319 6868 Acked Read 10 Done Done Sent false
21320 6869 Acked Write 9 Done Done Sent false
21321 6870 Acked Read 5 Done Done Sent false
21322 6871 Acked Write 8 Done Done Sent false
21323 6872 Acked Read 4 Done Done Sent false
21324 6873 Acked Write 2 Done Done Sent false
21325 6874 Acked Read 8 Done Done Sent false
21326 6875 Acked Write 1 Done Done Sent false
21327 6876 Acked Read 7 Done Done Sent false
21328 6877 Acked Read 9 Done Done Sent false
21329 6878 Acked Write 10 Done Done Sent false
21330 6879 Acked Read 5 Done Done Sent false
21331 6880 Acked Read 6 Done Done Sent false
21332 6881 Acked Read 4 Done Done Sent false
21333 6882 Acked Write 1 Done Done Sent false
21334 6883 Acked Read 4 Done Done Sent false
21335 6884 Acked Write 10 Done Done Sent false
21336 6885 Acked Write 1 Done Done Sent false
21337 6886 Acked Write 5 Done Done Sent false
21338 6887 Acked Write 2 Done Done Sent false
21339 6888 Acked Read 9 Done Done Sent false
21340 6889 Acked Write 2 Done Done Sent false
21341 6890 Acked Write 1 Done Done Sent false
21342 6891 Acked Write 3 Done Done Sent false
21343 6892 Acked Read 1 Done Done Sent false
21344 6893 Acked Read 7 Done Done Sent false
21345 6894 Acked Write 9 Done Done Sent false
21346 6895 Acked Write 1 Done Done Sent false
21347 6896 Acked Write 3 Done Done Sent false
21348 6897 Acked Read 1 Done Done Sent false
21349 6898 Acked Read 4 Done Done Sent false
21350 6899 Acked Flush 0 Done Done Sent false
21351 6900 Acked Read 3 Done Done Sent false
21352 6901 Acked Read 4 Done Done Sent false
21353 6902 Acked Read 5 Done Done Sent false
21354 6903 Acked Write 8 Done Done Sent false
21355 6904 Acked Flush 0 Done Done Sent false
21356 6905 Acked Write 3 Done Done Sent false
21357 6906 Acked Read 3 Done Done Sent false
21358 6907 Acked Write 10 Done Done Sent false
21359 6908 Acked Read 10 Done Done Sent false
21360 6909 Acked Read 10 Done Done Sent false
21361 6910 Acked Read 9 Done Done Sent false
21362 6911 Acked Read 3 Done Done Sent false
21363 6912 Acked Write 10 Done Done Sent false
21364 6913 Acked Write 5 Done Done Sent false
21365 6914 Acked Write 8 Done Done Sent false
21366 6915 Acked Read 5 Done Done Sent false
21367 6916 Acked Read 10 Done Done Sent false
21368 6917 Acked Write 6 Done Done Sent false
21369 6918 Acked Write 5 Done Done Sent false
21370 6919 Acked Write 9 Done Done Sent false
21371 6920 Acked Write 2 Done Done Sent false
21372 6921 Acked Read 9 Done Done Sent false
21373 6922 Acked Write 3 Done Done Sent false
21374 6923 Acked Flush 0 Done Done Sent false
21375 6924 Acked Flush 0 Done Done Sent false
21376 6925 Acked Write 2 Done Done Sent false
21377 6926 Acked Read 10 Done Done Sent false
21378 6927 Acked Read 7 Done Done Sent false
21379 6928 Acked Flush 0 Done Done Sent false
21380 6929 Acked Write 8 Done Done Sent false
21381 6930 Acked Write 2 Done Done Sent false
21382 6931 Acked Write 2 Done Done Sent false
21383 6932 Acked Read 4 Done Done Sent false
21384 6933 Acked Write 7 Done Done Sent false
21385 6934 Acked Read 7 Done Done Sent false
21386 6935 Acked Write 7 Done Done Sent false
21387 6936 Acked Write 10 Done Done Sent false
21388 6937 Acked Write 2 Done Done Sent false
21389 6938 Acked Read 3 Done Done Sent false
21390 6939 Acked Read 3 Done Done Sent false
21391 6940 Acked Write 9 Done Done Sent false
21392 6941 Acked Read 3 Done Done Sent false
21393 6942 Acked Write 1 Done Done Sent false
21394 6943 Acked Read 6 Done Done Sent false
21395 6944 Acked Flush 0 Done Done Sent false
21396 6945 Acked Write 3 Done Done Sent false
21397 6946 Acked Write 3 Done Done Sent false
21398 6947 Acked Read 1 Done Done Sent false
21399 6948 Acked Flush 0 Done Done Sent false
21400 6949 Acked Read 1 Done Done Sent false
21401 6950 Acked Write 9 Done Done Sent false
21402 6951 Acked Write 4 Done Done Sent false
21403 6952 Acked Read 1 Done Done Sent false
21404 6953 Acked Read 3 Done Done Sent false
21405 6954 Acked Read 5 Done Done Sent false
21406 6955 Acked Read 2 Done Done Sent false
21407 6956 Acked Flush 0 Done Done Sent false
21408 6957 Acked Write 3 Done Done Sent false
21409 6958 Acked Write 3 Done Done Sent false
21410 6959 Acked Read 4 Done Done Sent false
21411 6960 Acked Read 6 Done Done Sent false
21412 6961 Acked Write 10 Done Done Sent false
21413 6962 Acked Read 6 Done Done Sent false
21414 6963 Acked Read 7 Done Done Sent false
21415 6964 Acked Read 3 Done Done Sent false
21416 6965 Acked Read 3 Done Done Sent false
21417 6966 Acked Read 9 Done Done Sent false
21418 6967 Acked Read 3 Done Done Sent false
21419 6968 Acked Write 1 Done Done Sent false
21420 6969 Acked Read 7 Done Done Sent false
21421 6970 Acked Flush 0 Done Done Sent false
21422 6971 Acked Read 8 Done Done Sent false
21423 6972 Acked Write 7 Done Done Sent false
21424 6973 Acked Write 2 Done Done Sent false
21425 6974 Acked Read 6 Done Done Sent false
21426 6975 Acked Write 8 Done Done Sent false
21427 6976 Acked Write 9 Done Done Sent false
21428 6977 Acked Read 4 Done Done Sent false
21429 6978 Acked Flush 0 Done Done Sent false
21430 6979 Acked Write 10 Done Done Sent false
21431 6980 Acked Write 3 Done Done Sent false
21432 6981 Acked Read 5 Done Done Sent false
21433 6982 Acked Read 4 Done Done Sent false
21434 6983 Acked Write 6 Done Done Sent false
21435 6984 Acked Read 7 Done Done Sent false
21436 6985 Acked Read 8 Done Done Sent false
21437 6986 Acked Write 9 Done Done Sent false
21438 6987 Acked Write 3 Done Done Sent false
21439 6988 Acked Write 9 Done Done Sent false
21440 6989 Acked Write 4 Done Done Sent false
21441 6990 Acked Write 3 Done Done Sent false
21442 6991 Acked Write 9 Done Done Sent false
21443 6992 Acked Read 9 Done Done Sent false
21444 6993 Acked Read 9 Done Done Sent false
21445 6994 Acked Read 10 Done Done Sent false
21446 6995 Acked Read 9 Done Done Sent false
21447 6996 Acked Write 3 Done Done Sent false
21448 6997 Acked Read 6 Done Done Sent false
21449 6998 Acked Read 1 Done Done Sent false
21450 6999 Acked Read 9 Done Done Sent false
21451 7000 Acked Read 6 Done Done Sent false
21452 7001 Acked Read 7 Done Done Sent false
21453 7002 Acked Flush 0 Done Done Sent false
21454 7003 Acked Read 5 Done Done Sent false
21455 7004 Acked Read 1 Done Done Sent false
21456 7005 Acked Write 5 Done Done Sent false
21457 7006 Acked Read 8 Done Done Sent false
21458 7007 Acked Read 10 Done Done Sent false
21459 7008 Acked Write 5 Done Done Sent false
21460 7009 Acked Write 7 Done Done Sent false
21461 7010 Acked Write 10 Done Done Sent false
21462 7011 Acked Write 10 Done Done Sent false
21463 7012 Acked Read 2 Done Done Sent false
21464 7013 Acked Write 9 Done Done Sent false
21465 7014 Acked Read 7 Done Done Sent false
21466 7015 Acked Read 2 Done Done Sent false
21467 7016 Acked Read 9 Done Done Sent false
21468 7017 Acked Flush 0 Done Done Sent false
21469 7018 Acked Write 3 Done Done Sent false
21470 7019 Acked Write 4 Done Done Sent false
21471 7020 Acked Read 6 Done Done Sent false
21472 7021 Acked Read 8 Done Done Sent false
21473 7022 Acked Write 4 Done Done Sent false
21474 7023 Acked Flush 0 Done Done Sent false
21475 7024 Acked Write 9 Done Done Sent false
21476 7025 Acked Write 2 Done Done Sent false
21477 7026 Acked Read 1 Done Done Sent false
21478 7027 Acked Write 3 Done Done Sent false
21479 7028 Acked Write 3 Done Done Sent false
21480 7029 Acked Read 3 Done Done Sent false
21481 7030 Acked Write 4 Done Done Sent false
21482 7031 Acked Write 7 Done Done Sent false
21483 7032 Acked Write 4 Done Done Sent false
21484 7033 Acked Write 2 Done Done Sent false
21485 7034 Acked Write 4 Done Done Sent false
21486 7035 Acked Read 5 Done Done Sent false
21487 7036 Acked Flush 0 Done Done Sent false
21488 STATES DS:0 DS:1 DS:2 TOTAL
21489 Sent 0 0 302 302
21490 Done 302 302 0 604
21491 Skipped 0 0 0 0
21492 Error 0 0 0 0
21493 Last Flush: Some(JobId(7036)) Some(JobId(7036)) Some(JobId(6734))
21494 Downstairs last five completed: 6734 6733 6732 6731 6730
21495 Upstairs last five completed: 7036 7035 7034 7033 7032
21496 CLIENT: Up:0 ds:302 act:2
214972025-01-23T18:52:23.702ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
214982025-01-23T18:52:23.702ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58793) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
214992025-01-23T18:52:23.703ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
215002025-01-23T18:52:23.704ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
215012025-01-23T18:52:23.704ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
215022025-01-23T18:52:23.704ZINFOcrucible: send last flush ID to this DS: Some(JobId(6734)) = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
215032025-01-23T18:52:23.704ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
215042025-01-23T18:52:23.704ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(6734)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
215052025-01-23T18:52:23.704ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(6734)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
21506 ----------------------------------------------------------------
21507 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21508 Downstairs last five completed: 7036 7035 7034 7033 7032
21509 Upstairs last five completed: 7036 7035 7034 7033 7032
21510 CLIENT: Up:0 ds:0 act:3
21511 Replay: All jobs finished
21512 Main task has work to do, go find it
21513 got dsc Stop(1)
21514 stop 1
21515 [8820] Got stop action so:false kr:false
21516 Jan 23 18:52:25.943 INFO request completed, latency_us: 354, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 3a965627-bf02-49de-bc82-31aefaf0daf5, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
21517 Jan 23 18:52:25.944 INFO request completed, latency_us: 119, response_code:{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:52:25.944719206Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
21518 200, uri: /state/cid/1, method: GET, req_id: 13814f6a-318e-41d9-a65f-16cf8420a787, remote_addr{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:52:25.944770629Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
21519 : 127.0.0.1:59295, local_addr{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:52:25.944797256Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
21520 : 127.0.0.1:9998
215212025-01-23T18:52:25.944ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
21522 [8820] Exited with: ExitStatus(unix_wait_status(9))
21523 [8820][1] reports Exit
21524 Jan 23 18:52:29.947 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/1, method: GET, req_id: b871ea04-39c4-4ac8-beb2-02359cf048a4, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
21525 001/300 Write block 2326 len 4096 data: 6
21526 002/300 Read block 3387 len 32768
21527 003/300 Write block 1847 len 12288 data: 5 5 5
21528 004/300 Read block 238 len 32768
21529 005/300 Read block 2654 len 16384
21530 006/300 Write block 2967 len 24576 data: 5 5 4 4 3 3
21531 007/300 Write block 4529 len 20480 data: 2 2 2 2 3
21532 008/300 Write block 341 len 20480 data: 4 3 3 3 3
21533 009/300 Read block 3697 len 20480
21534 010/300 Write block 2579 len 12288 data: 3 4 4
21535 011/300 Write block 4867 len 12288 data: 2 4 5
21536 012/300 Write block 3305 len 4096 data: 4
21537 013/300 Read block 930 len 40960
21538 014/300 Write block 2742 len 24576 data: 6 3 4 3 4 4
21539 015/300 Flush
21540 016/300 Write block 700 len 8192 data: 3 2
21541 017/300 Read block 3045 len 40960
21542 018/300 Write block 1795 len 24576 data: 8 7 7 6 5 3
21543 019/300 Write block 1198 len 4096 data: 3
21544 020/300 Write block 4274 len 8192 data: 4 4
21545 021/300 Flush
21546 022/300 Write block 2987 len 4096 data: 3
21547 023/300 Read block 337 len 4096
21548 024/300 Write block 1506 len 28672 data: 6 5 6 6 6 6 5
21549 025/300 Write block 3606 len 28672 data: 3 3 3 3 4 4 4
21550 026/300 Write block 1673 len 28672 data: 4 4 5 5 5 6 6
21551 027/300 Write block 2578 len 40960 data: 4 4 5 5 4 3 3 5 6 4
21552 028/300 Read block 172 len 20480
21553 029/300 Write block 2149 len 4096 data: 6
21554 030/300 Write block 4283 len 28672 data: 3 3 3 5 5 6 6
21555 031/300 Read block 1267 len 40960
21556 032/300 Write block 180 len 40960 data: 6 5 5 4 4 4 4 5 4 4
21557 033/300 Read block 360 len 8192
21558 034/300 Write block 2071 len 28672 data: 8 7 6 5 5 5 7
21559 035/300 Write block 3087 len 32768 data: 3 4 4 4 5 6 6 6
21560 036/300 Read block 3887 len 20480
21561 037/300 Write block 4783 len 28672 data: 5 5 5 4 4 6 4
21562 038/300 Write block 694 len 16384 data: 5 5 5 4
21563 039/300 Write block 4604 len 32768 data: 4 5 5 4 4 6 3 3
21564 040/300 Write block 2219 len 8192 data: 2 3
21565 041/300 Read block 1078 len 40960
21566 042/300 Read block 2504 len 36864
21567 043/300 Read block 896 len 4096
21568 044/300 Read block 2204 len 36864
21569 045/300 Read block 4684 len 12288
21570 046/300 Write block 3183 len 32768 data: 3 3 4 4 3 3 3 3
21571 047/300 Read block 220 len 4096
21572 048/300 Write block 4117 len 28672 data: 3 3 2 3 3 3 3
21573 049/300 Write block 1988 len 12288 data: 4 3 3
21574 050/300 Write block 895 len 36864 data: 7 9 7 7 7 6 7 6 6
21575 051/300 Write block 4386 len 20480 data: 3 3 3 4 3
21576 052/300 Write block 997 len 4096 data: 3
21577 053/300 Write block 4890 len 8192 data: 5 5
21578 054/300 Flush
21579 055/300 Write block 1388 len 36864 data: 6 5 6 5 3 4 4 6 6
21580 056/300 Read block 834 len 20480
21581 057/300 Read block 293 len 32768
21582 058/300 Write block 323 len 20480 data: 6 5 4 4 3
21583 059/300 Write block 2005 len 8192 data: 3 4
21584 060/300 Write block 3065 len 36864 data: 6 6 5 6 6 5 5 5 5
21585 061/300 Flush
21586 062/300 Read block 84 len 24576
21587 063/300 Flush
21588 064/300 Read block 4401 len 12288
21589 065/300 Write block 3697 len 12288 data: 3 4 3
21590 066/300 Read block 4781 len 24576
21591 067/300 Read block 1862 len 40960
21592 068/300 Write block 1851 len 16384 data: 5 5 4 5
21593 069/300 Write block 1404 len 16384 data: 5 5 4 4
21594 070/300 Write block 2948 len 32768 data: 6 5 5 5 4 4 4 4
21595 071/300 Read block 3988 len 36864
21596 072/300 Flush
21597 073/300 Read block 1095 len 40960
21598 074/300 Flush
21599 075/300 Flush
21600 076/300 Write block 3977 len 36864 data: 5 4 4 4 3 3 3 2 4
21601 077/300 Write block 2952 len 36864 data: 5 5 5 5 5 4 3 3 4
21602 078/300 Read block 2029 len 36864
21603 079/300 Read block 1097 len 12288
21604 080/300 Flush
21605 081/300 Read block 4016 len 32768
21606 082/300 Read block 4676 len 28672
21607 083/300 Read block 406 len 32768
21608 084/300 Read block 2364 len 24576
21609 085/300 Read block 1667 len 4096
21610 086/300 Read block 2027 len 12288
21611 087/300 Read block 2661 len 12288
21612 088/300 Read block 1300 len 12288
21613 089/300 Read block 114 len 28672
21614 090/300 Read block 1241 len 12288
21615 091/300 Read block 2388 len 36864
21616 092/300 Read block 3534 len 28672
21617 093/300 Write block 3691 len 36864 data: 7 6 8 6 5 5 4 5 4
21618 094/300 Flush
21619 095/300 Read block 2490 len 12288
21620 096/300 Write block 2725 len 40960 data: 3 3 3 3 2 2 4 4 4 4
21621 097/300 Read block 3064 len 20480
21622 098/300 Read block 391 len 12288
21623 099/300 Read block 4682 len 32768
21624 100/300 Write block 4378 len 24576 data: 3 3 2 2 2 4
21625 101/300 Read block 4004 len 28672
21626 102/300 Write block 388 len 40960 data: 4 4 4 5 5 4 4 4 6 6
21627 103/300 Write block 3885 len 40960 data: 10 9 8 8 7 6 6 6 5 6
21628 104/300 Read block 1036 len 12288
21629 105/300 Read block 4220 len 32768
21630 106/300 Write block 3462 len 12288 data: 7 7 10
21631 107/300 Flush
21632 108/300 Write block 4134 len 16384 data: 2 3 3 3
21633 109/300 Read block 3052 len 4096
21634 110/300 Read block 4579 len 36864
21635 111/300 Read block 4631 len 8192
21636 112/300 Write block 3641 len 8192 data: 4 3
21637 113/300 Read block 4775 len 24576
21638 114/300 Read block 4552 len 20480
21639 115/300 Write block 2280 len 28672 data: 4 4 4 5 3 3 3
21640 116/300 Read block 4016 len 4096
21641 117/300 Write block 1070 len 16384 data: 9 7 6 8
21642 118/300 Read block 152 len 24576
21643 119/300 Flush
21644 120/300 Read block 805 len 24576
21645 121/300 Read block 3891 len 16384
21646 122/300 Read block 4701 len 16384
21647 123/300 Read block 4039 len 24576
21648 124/300 Write block 2765 len 36864 data: 4 4 4 4 4 3 3 4 4
21649 125/300 Read block 1443 len 8192
21650 126/300 Read block 3833 len 8192
21651 127/300 Write block 1924 len 24576 data: 5 5 4 4 5 6
21652 128/300 Flush
21653 129/300 Flush
21654 130/300 Read block 346 len 24576
21655 131/300 Write block 4488 len 20480 data: 4 5 5 5 5
21656 132/300 Read block 327 len 8192
21657 133/300 Read block 2277 len 20480
21658 134/300 Write block 4724 len 24576 data: 4 3 3 3 3 3
21659 135/300 Read block 8 len 20480
21660 136/300 Write block 822 len 40960 data: 10 11 7 6 7 6 6 5 4 3
21661 137/300 Read block 4724 len 20480
21662 138/300 Read block 2242 len 16384
21663 139/300 Flush
21664 140/300 Read block 1136 len 32768
21665 141/300 Read block 3248 len 20480
21666 142/300 Read block 3398 len 12288
21667 143/300 Write block 3839 len 28672 data: 5 5 5 5 4 4 3
21668 144/300 Read block 4790 len 32768
21669 145/300 Read block 3956 len 4096
21670 146/300 Read block 4546 len 32768
21671 147/300 Write block 3452 len 32768 data: 3 3 4 2 4 7 7 7
21672 148/300 Write block 4503 len 8192 data: 5 6
21673 149/300 Read block 2842 len 12288
21674 150/300 Flush
21675 151/300 Read block 4093 len 32768
21676 152/300 Write block 4191 len 4096 data: 5
21677 153/300 Read block 1160 len 8192
21678 154/300 Read block 1956 len 4096
21679 155/300 Write block 3343 len 24576 data: 5 5 6 6 6 4
21680 156/300 Read block 501 len 40960
21681 157/300 Write block 2502 len 8192 data: 4 3
21682 158/300 Read block 1145 len 12288
21683 159/300 Write block 2461 len 12288 data: 6 5 4
21684 160/300 Flush
21685 161/300 Read block 2604 len 20480
21686 162/300 Read block 4533 len 20480
21687 163/300 Flush
21688 164/300 Write block 773 len 16384 data: 3 3 4 7
21689 165/300 Read block 2505 len 36864
21690 166/300 Read block 264 len 28672
21691 167/300 Read block 3580 len 24576
21692 168/300 Read block 2955 len 4096
21693 169/300 Read block 4621 len 8192
21694 170/300 Read block 432 len 4096
21695 171/300 Read block 2167 len 40960
21696 172/300 Write block 4599 len 16384 data: 6 5 5 4
21697 173/300 Write block 1781 len 4096 data: 4
21698 174/300 Write block 2446 len 32768 data: 5 4 8 8 7 6 6 5
21699 175/300 Read block 4329 len 12288
21700 176/300 Read block 986 len 40960
21701 177/300 Read block 4089 len 12288
21702 178/300 Write block 3955 len 28672 data: 6 6 5 4 3 3 4
21703 179/300 Read block 2429 len 36864
21704 180/300 Flush
21705 181/300 Write block 3248 len 24576 data: 5 5 5 7 7 7
21706 182/300 Write block 2264 len 16384 data: 3 4 5 5
21707 183/300 Read block 2052 len 40960
21708 184/300 Read block 4977 len 40960
21709 185/300 Read block 379 len 24576
21710 186/300 Read block 2313 len 28672
21711 187/300 Read block 3732 len 4096
21712 188/300 Read block 2987 len 16384
21713 189/300 Write block 2450 len 8192 data: 8 7
21714 190/300 Read block 2895 len 32768
21715 191/300 Read block 1318 len 12288
21716 192/300 Read block 526 len 40960
21717 193/300 Read block 4843 len 4096
21718 194/300 Read block 4605 len 16384
21719 195/300 Write block 354 len 36864 data: 3 4 5 4 4 3 2 2 3
21720 196/300 Read block 808 len 28672
21721 197/300 Flush
21722 198/300 Write block 3025 len 40960 data: 7 7 6 6 6 7 6 6 6 5
21723 199/300 Read block 945 len 16384
21724 200/300 Write block 3364 len 8192 data: 8 7
21725 201/300 Flush
21726 202/300 Read block 3120 len 4096
21727 203/300 Read block 4385 len 20480
21728 204/300 Read block 2885 len 4096
21729 205/300 Flush
21730 206/300 Flush
21731 207/300 Read block 2876 len 16384
21732 208/300 Read block 4924 len 4096
21733 209/300 Write block 2235 len 12288 data: 4 4 4
21734 210/300 Read block 4367 len 20480
21735 211/300 Read block 1263 len 36864
21736 212/300 Read block 2188 len 12288
21737 213/300 Read block 3671 len 40960
21738 214/300 Read block 2894 len 24576
21739 215/300 Flush
21740 216/300 Read block 2737 len 40960
21741 217/300 Read block 4359 len 8192
21742 218/300 Write block 3025 len 4096 data: 8
21743 219/300 Read block 2373 len 20480
21744 220/300 Read block 1612 len 40960
21745 221/300 Write block 3099 len 32768 data: 4 5 5 5 6 6 7 7
21746 222/300 Write block 96 len 40960 data: 7 7 4 4 3 3 4 5 4 3
21747 223/300 Read block 2277 len 28672
21748 224/300 Read block 319 len 4096
21749 225/300 Read block 277 len 16384
21750 226/300 Read block 4146 len 20480
21751 227/300 Write block 4879 len 16384 data: 2 2 2 4
21752 228/300 Read block 1691 len 40960
21753 229/300 Write block 3009 len 8192 data: 5 6
21754 230/300 Write block 2108 len 20480 data: 9 8 6 4 4
21755 231/300 Read block 3160 len 36864
21756 232/300 Read block 3310 len 32768
21757 233/300 Write block 1069 len 16384 data: 9 10 8 7
21758 234/300 Read block 652 len 36864
21759 235/300 Read block 3794 len 32768
21760 236/300 Write block 2436 len 40960 data: 5 5 6 5 6 6 5 6 6 5
21761 237/300 Read block 2818 len 8192
21762 238/300 Write block 2016 len 8192 data: 6 6
21763 239/300 Read block 2058 len 12288
21764 240/300 Read block 1353 len 36864
21765 241/300 Read block 4829 len 40960
21766 242/300 Read block 3878 len 32768
21767 243/300 Write block 1357 len 40960 data: 3 4 5 3 4 3 3 4 4 4
21768 244/300 Write block 3594 len 20480 data: 5 6 4 5 5
21769 245/300 Read block 2608 len 40960
21770 246/300 Write block 1433 len 28672 data: 2 2 2 2 2 2 3
21771 247/300 Write block 926 len 4096 data: 3
21772 248/300 Write block 1928 len 20480 data: 6 7 6 6 7
21773 249/300 Read block 3270 len 12288
21774 250/300 Read block 2690 len 28672
21775 251/300 Read block 1885 len 4096
21776 252/300 Read block 2165 len 4096
21777 253/300 Write block 4792 len 16384 data: 3 3 3 3
21778 254/300 Flush
21779 255/300 Write block 2573 len 40960 data: 3 5 6 5 7 5 5 6 6 5
21780 256/300 Write block 1988 len 28672 data: 5 4 4 2 2 3 4
21781 257/300 Write block 626 len 16384 data: 7 4 4 4
21782 258/300 Read block 616 len 8192
21783 259/300 Read block 3203 len 24576
21784 260/300 Write block 857 len 8192 data: 4 4
21785 261/300 Write block 966 len 8192 data: 6 8
21786 262/300 Read block 3929 len 32768
21787 263/300 Read block 2362 len 16384
21788 264/300 Write block 4149 len 36864 data: 10 10 8 6 7 5 3 3 4
21789 265/300 Write block 1393 len 8192 data: 5 5
21790 266/300 Write block 2179 len 20480 data: 3 2 3 3 4
21791 267/300 Read block 3679 len 20480
21792 268/300 Read block 4201 len 40960
21793 269/300 Write block 1114 len 16384 data: 4 4 4 4
21794 270/300 Write block 1289 len 24576 data: 6 6 6 5 5 4
21795 271/300 Read block 826 len 24576
21796 272/300 Read block 2690 len 40960
21797 273/300 Read block 530 len 8192
21798 274/300 Write block 1406 len 28672 data: 5 5 4 3 3 2 3
21799 275/300 Write block 2577 len 24576 data: 8 6 6 7 7 6
21800 276/300 Flush
21801 277/300 Write block 4702 len 12288 data: 4 3 5
21802 278/300 Write block 220 len 4096 data: 5
21803 279/300 Write block 2840 len 28672 data: 5 5 4 4 5 5 7
21804 280/300 Write block 127 len 4096 data: 5
21805 281/300 Flush
21806 282/300 Read block 2845 len 12288
21807 283/300 Read block 675 len 4096
21808 284/300 Read block 4956 len 28672
21809 285/300 Read block 3362 len 20480
21810 286/300 Read block 4924 len 8192
21811 287/300 Flush
21812 288/300 Read block 472 len 28672
21813 289/300 Flush
21814 290/300 Read block 2281 len 36864
21815 291/300 Write block 617 len 40960 data: 3 5 5 5 5 5 4 6 5 8
21816 292/300 Flush
21817 293/300 Read block 1545 len 32768
21818 294/300 Write block 590 len 24576 data: 2 2 4 4 4 4
21819 295/300 Write block 4294 len 8192 data: 4 4
21820 296/300 Read block 2872 len 28672
21821 297/300 Flush
21822 298/300 Flush
21823 299/300 Read block 3371 len 20480
21824 300/300 Read block 4466 len 36864
21825 301/300 Write block 504 len 16384 data: 4 5 5 5
21826 Main task has work to do, go find it
21827 got dsc Start(1)
21828 start 1
21829 Jan 23 18:52:30.189 INFO request completed, latency_us: 179, response_code: 204, uri: /start/cid/1, method: POST, req_id: aaadce48-6a3e-4321-bed3-680ee86d94af, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
21830 Replay: started 1, returned:Ok(())
21831 ----------------------------------------------------------------
21832 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21833 States: Active Offline Active
21834 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21835 7037 Acked Write 1 Done Sent Done false
21836 7038 Acked Read 8 Done Sent Done false
21837 7039 Acked Write 3 Done Sent Done false
21838 7040 Acked Read 8 Done Sent Done false
21839 7041 Acked Read 4 Done Sent Done false
21840 7042 Acked Write 6 Done Sent Done false
21841 7043 Acked Write 5 Done Sent Done false
21842 7044 Acked Write 5 Done Sent Done false
21843 7045 Acked Read 5 Done Sent Done false
21844 7046 Acked Write 3 Done Sent Done false
21845 7047 Acked Write 3 Done Sent Done false
21846 7048 Acked Write 1 Done Sent Done false
21847 7049 Acked Read 10 Done Sent Done false
21848 7050 Acked Write 6 Done Sent Done false
21849 7051 Acked Flush 0 Done Sent Done false
21850 7052 Acked Write 2 Done Sent Done false
21851 7053 Acked Read 10 Done Sent Done false
21852 7054 Acked Write 6 Done Sent Done false
21853 7055 Acked Write 1 Done Sent Done false
21854 7056 Acked Write 2 Done Sent Done false
21855 7057 Acked Flush 0 Done Sent Done false
21856 7058 Acked Write 1 Done Sent Done false
21857 7059 Acked Read 1 Done Sent Done false
21858 7060 Acked Write 7 Done Sent Done false
21859 7061 Acked Write 7 Done Sent Done false
21860 7062 Acked Write 7 Done Sent Done false
21861 7063 Acked Write 10 Done Sent Done false
21862 7064 Acked Read 5 Done Sent Done false
21863 7065 Acked Write 1 Done Sent Done false
21864 7066 Acked Write 7 Done Sent Done false
21865 7067 Acked Read 10 Done Sent Done false
21866 7068 Acked Write 10 Done Sent Done false
21867 7069 Acked Read 2 Done Sent Done false
21868 7070 Acked Write 7 Done Sent Done false
21869 7071 Acked Write 8 Done Sent Done false
21870 7072 Acked Read 5 Done Sent Done false
21871 7073 Acked Write 7 Done Sent Done false
21872 7074 Acked Write 4 Done Sent Done false
21873 7075 Acked Write 8 Done Sent Done false
21874 7076 Acked Write 2 Done Sent Done false
21875 7077 Acked Read 10 Done Sent Done false
21876 7078 Acked Read 9 Done Sent Done false
21877 7079 Acked Read 1 Done Sent Done false
21878 7080 Acked Read 9 Done Sent Done false
21879 7081 Acked Read 3 Done Sent Done false
21880 7082 Acked Write 8 Done Sent Done false
21881 7083 Acked Read 1 Done Sent Done false
21882 7084 Acked Write 7 Done Sent Done false
21883 7085 Acked Write 3 Done Sent Done false
21884 7086 Acked Write 9 Done Sent Done false
21885 7087 Acked Write 5 Done Sent Done false
21886 7088 Acked Write 1 Done Sent Done false
21887 7089 Acked Write 2 Done Sent Done false
21888 7090 Acked Flush 0 Done Sent Done false
21889 7091 Acked Write 9 Done Sent Done false
21890 7092 Acked Read 5 Done Sent Done false
21891 7093 Acked Read 8 Done Sent Done false
21892 7094 Acked Write 5 Done Sent Done false
21893 7095 Acked Write 2 Done Sent Done false
21894 7096 Acked Write 9 Done Sent Done false
21895 7097 Acked Flush 0 Done Sent Done false
21896 7098 Acked Read 6 Done Sent Done false
21897 7099 Acked Flush 0 Done Sent Done false
21898 7100 Acked Read 3 Done Sent Done false
21899 7101 Acked Write 3 Done Sent Done false
21900 7102 Acked Read 6 Done Sent Done false
21901 7103 Acked Read 10 Done Sent Done false
21902 7104 Acked Write 4 Done Sent Done false
21903 7105 Acked Write 4 Done Sent Done false
21904 7106 Acked Write 8 Done Sent Done false
21905 7107 Acked Read 9 Done Sent Done false
21906 7108 Acked Flush 0 Done Sent Done false
21907 7109 Acked Read 10 Done Sent Done false
21908 7110 Acked Flush 0 Done Sent Done false
21909 7111 Acked Flush 0 Done Sent Done false
21910 7112 Acked Write 9 Done Sent Done false
21911 7113 Acked Write 9 Done Sent Done false
21912 7114 Acked Read 9 Done Sent Done false
21913 7115 Acked Read 3 Done Sent Done false
21914 7116 Acked Flush 0 Done Sent Done false
21915 7117 Acked Read 8 Done Sent Done false
21916 7118 Acked Read 7 Done Sent Done false
21917 7119 Acked Read 8 Done Sent Done false
21918 7120 Acked Read 6 Done Sent Done false
21919 7121 Acked Read 1 Done Sent Done false
21920 7122 Acked Read 3 Done Sent Done false
21921 7123 Acked Read 3 Done Sent Done false
21922 7124 Acked Read 3 Done Sent Done false
21923 7125 Acked Read 7 Done Sent Done false
21924 7126 Acked Read 3 Done Sent Done false
21925 7127 Acked Read 9 Done Sent Done false
21926 7128 Acked Read 7 Done Sent Done false
21927 7129 Acked Write 9 Done Sent Done false
21928 7130 Acked Flush 0 Done Sent Done false
21929 7131 Acked Read 3 Done Sent Done false
21930 7132 Acked Write 10 Done Sent Done false
21931 7133 Acked Read 5 Done Sent Done false
21932 7134 Acked Read 3 Done Sent Done false
21933 7135 Acked Read 8 Done Sent Done false
21934 7136 Acked Write 6 Done Sent Done false
21935 7137 Acked Read 7 Done Sent Done false
21936 7138 Acked Write 10 Done Sent Done false
21937 7139 Acked Write 10 Done Sent Done false
21938 7140 Acked Read 3 Done Sent Done false
21939 7141 Acked Read 8 Done Sent Done false
21940 7142 Acked Write 3 Done Sent Done false
21941 7143 Acked Flush 0 Done Sent Done false
21942 7144 Acked Write 4 Done Sent Done false
21943 7145 Acked Read 1 Done Sent Done false
21944 7146 Acked Read 9 Done Sent Done false
21945 7147 Acked Read 2 Done Sent Done false
21946 7148 Acked Write 2 Done Sent Done false
21947 7149 Acked Read 6 Done Sent Done false
21948 7150 Acked Read 5 Done Sent Done false
21949 7151 Acked Write 7 Done Sent Done false
21950 7152 Acked Read 1 Done Sent Done false
21951 7153 Acked Write 4 Done Sent Done false
21952 7154 Acked Read 6 Done Sent Done false
21953 7155 Acked Flush 0 Done Sent Done false
21954 7156 Acked Read 6 Done Sent Done false
21955 7157 Acked Read 4 Done Sent Done false
21956 7158 Acked Read 4 Done Sent Done false
21957 7159 Acked Read 6 Done Sent Done false
21958 7160 Acked Write 9 Done Sent Done false
21959 7161 Acked Read 2 Done Sent Done false
21960 7162 Acked Read 2 Done Sent Done false
21961 7163 Acked Write 6 Done Sent Done false
21962 7164 Acked Flush 0 Done Sent Done false
21963 7165 Acked Flush 0 Done Sent Done false
21964 7166 Acked Read 6 Done Sent Done false
21965 7167 Acked Write 5 Done Sent Done false
21966 7168 Acked Read 2 Done Sent Done false
21967 7169 Acked Read 5 Done Sent Done false
21968 7170 Acked Write 6 Done Sent Done false
21969 7171 Acked Read 5 Done Sent Done false
21970 7172 Acked Write 10 Done Sent Done false
21971 7173 Acked Read 5 Done Sent Done false
21972 7174 Acked Read 4 Done Sent Done false
21973 7175 Acked Flush 0 Done Sent Done false
21974 7176 Acked Read 8 Done Sent Done false
21975 7177 Acked Read 5 Done Sent Done false
21976 7178 Acked Read 3 Done Sent Done false
21977 7179 Acked Write 7 Done Sent Done false
21978 7180 Acked Read 8 Done Sent Done false
21979 7181 Acked Read 1 Done Sent Done false
21980 7182 Acked Read 8 Done Sent Done false
21981 7183 Acked Write 8 Done Sent Done false
21982 7184 Acked Write 2 Done Sent Done false
21983 7185 Acked Read 3 Done Sent Done false
21984 7186 Acked Flush 0 Done Sent Done false
21985 7187 Acked Read 8 Done Sent Done false
21986 7188 Acked Write 1 Done Sent Done false
21987 7189 Acked Read 2 Done Sent Done false
21988 7190 Acked Read 1 Done Sent Done false
21989 7191 Acked Write 6 Done Sent Done false
21990 7192 Acked Read 10 Done Sent Done false
21991 7193 Acked Write 2 Done Sent Done false
21992 7194 Acked Read 3 Done Sent Done false
21993 7195 Acked Write 3 Done Sent Done false
21994 7196 Acked Flush 0 Done Sent Done false
21995 7197 Acked Read 5 Done Sent Done false
21996 7198 Acked Read 5 Done Sent Done false
21997 7199 Acked Flush 0 Done Sent Done false
21998 7200 Acked Write 4 Done Sent Done false
21999 7201 Acked Read 9 Done Sent Done false
22000 7202 Acked Read 7 Done Sent Done false
22001 7203 Acked Read 6 Done Sent Done false
22002 7204 Acked Read 1 Done Sent Done false
22003 7205 Acked Read 2 Done Sent Done false
22004 7206 Acked Read 1 Done Sent Done false
22005 7207 Acked Read 10 Done Sent Done false
22006 7208 Acked Write 4 Done Sent Done false
22007 7209 Acked Write 1 Done Sent Done false
22008 7210 Acked Write 8 Done Sent Done false
22009 7211 Acked Read 3 Done Sent Done false
22010 7212 Acked Read 10 Done Sent Done false
22011 7213 Acked Read 3 Done Sent Done false
22012 7214 Acked Write 7 Done Sent Done false
22013 7215 Acked Read 9 Done Sent Done false
22014 7216 Acked Flush 0 Done Sent Done false
22015 7217 Acked Write 6 Done Sent Done false
22016 7218 Acked Write 4 Done Sent Done false
22017 7219 Acked Read 10 Done Sent Done false
22018 7220 Acked Read 10 Done Sent Done false
22019 7221 Acked Read 6 Done Sent Done false
22020 7222 Acked Read 7 Done Sent Done false
22021 7223 Acked Read 1 Done Sent Done false
22022 7224 Acked Read 4 Done Sent Done false
22023 7225 Acked Write 2 Done Sent Done false
22024 7226 Acked Read 8 Done Sent Done false
22025 7227 Acked Read 3 Done Sent Done false
22026 7228 Acked Read 10 Done Sent Done false
22027 7229 Acked Read 1 Done Sent Done false
22028 7230 Acked Read 4 Done Sent Done false
22029 7231 Acked Write 9 Done Sent Done false
22030 7232 Acked Read 7 Done Sent Done false
22031 7233 Acked Flush 0 Done Sent Done false
22032 7234 Acked Write 10 Done Sent Done false
22033 7235 Acked Read 4 Done Sent Done false
22034 7236 Acked Write 2 Done Sent Done false
22035 7237 Acked Flush 0 Done Sent Done false
22036 7238 Acked Read 1 Done Sent Done false
22037 7239 Acked Read 5 Done Sent Done false
22038 7240 Acked Read 1 Done Sent Done false
22039 7241 Acked Flush 0 Done Sent Done false
22040 7242 Acked Flush 0 Done Sent Done false
22041 7243 Acked Read 4 Done Sent Done false
22042 7244 Acked Read 1 Done Sent Done false
22043 7245 Acked Write 3 Done Sent Done false
22044 7246 Acked Read 5 Done Sent Done false
22045 7247 Acked Read 9 Done Sent Done false
22046 7248 Acked Read 3 Done Sent Done false
22047 7249 Acked Read 10 Done Sent Done false
22048 7250 Acked Read 6 Done Sent Done false
22049 7251 Acked Flush 0 Done Sent Done false
22050 7252 Acked Read 10 Done Sent Done false
22051 7253 Acked Read 2 Done Sent Done false
22052 7254 Acked Write 1 Done Sent Done false
22053 7255 Acked Read 5 Done Sent Done false
22054 7256 Acked Read 10 Done Sent Done false
22055 7257 Acked Write 8 Done Sent Done false
22056 7258 Acked Write 10 Done Sent Done false
22057 7259 Acked Read 7 Done Sent Done false
22058 7260 Acked Read 1 Done Sent Done false
22059 7261 Acked Read 4 Done Sent Done false
22060 7262 Acked Read 5 Done Sent Done false
22061 7263 Acked Write 4 Done Sent Done false
22062 7264 Acked Read 10 Done Sent Done false
22063 7265 Acked Write 2 Done Sent Done false
22064 7266 Acked Write 5 Done Sent Done false
22065 7267 Acked Read 9 Done Sent Done false
22066 7268 Acked Read 8 Done Sent Done false
22067 7269 Acked Write 4 Done Sent Done false
22068 7270 Acked Read 9 Done Sent Done false
22069 7271 Acked Read 8 Done Sent Done false
22070 7272 Acked Write 10 Done Sent Done false
22071 7273 Acked Read 2 Done Sent Done false
22072 7274 Acked Write 2 Done Sent Done false
22073 7275 Acked Read 3 Done Sent Done false
22074 7276 Acked Read 9 Done Sent Done false
22075 7277 Acked Read 10 Done Sent Done false
22076 7278 Acked Read 8 Done Sent Done false
22077 7279 Acked Write 10 Done Sent Done false
22078 7280 Acked Write 5 Done Sent Done false
22079 7281 Acked Read 10 Done Sent Done false
22080 7282 Acked Write 7 Done Sent Done false
22081 7283 Acked Write 1 Done Sent Done false
22082 7284 Acked Write 5 Done Sent Done false
22083 7285 Acked Read 3 Done Sent Done false
22084 7286 Acked Read 7 Done Sent Done false
22085 7287 Acked Read 1 Done Sent Done false
22086 7288 Acked Read 1 Done Sent Done false
22087 7289 Acked Write 4 Done Sent Done false
22088 7290 Acked Flush 0 Done Sent Done false
22089 7291 Acked Write 10 Done Sent Done false
22090 7292 Acked Write 7 Done Sent Done false
22091 7293 Acked Write 4 Done Sent Done false
22092 7294 Acked Read 2 Done Sent Done false
22093 7295 Acked Read 6 Done Sent Done false
22094 7296 Acked Write 2 Done Sent Done false
22095 7297 Acked Write 2 Done Sent Done false
22096 7298 Acked Read 8 Done Sent Done false
22097 7299 Acked Read 4 Done Sent Done false
22098 7300 Acked Write 9 Done Sent Done false
22099 7301 Acked Write 2 Done Sent Done false
22100 7302 Acked Write 5 Done Sent Done false
22101 7303 Acked Read 5 Done Sent Done false
22102 7304 Acked Read 10 Done Sent Done false
22103 7305 Acked Write 4 Done Sent Done false
22104 7306 Acked Write 6 Done Sent Done false
22105 7307 Acked Read 6 Done Sent Done false
22106 7308 Acked Read 10 Done Sent Done false
22107 7309 Acked Read 2 Done Sent Done false
22108 7310 Acked Write 7 Done Sent Done false
22109 7311 Acked Write 6 Done Sent Done false
22110 7312 Acked Flush 0 Done Sent Done false
22111 7313 Acked Write 3 Done Sent Done false
22112 7314 Acked Write 1 Done Sent Done false
22113 7315 Acked Write 7 Done Sent Done false
22114 7316 Acked Write 1 Done Sent Done false
22115 7317 Acked Flush 0 Done Sent Done false
22116 7318 Acked Read 3 Done Sent Done false
22117 7319 Acked Read 1 Done Sent Done false
22118 7320 Acked Read 7 Done Sent Done false
22119 7321 Acked Read 5 Done Sent Done false
22120 7322 Acked Read 2 Done Sent Done false
22121 7323 Acked Flush 0 Done Sent Done false
22122 7324 Acked Read 7 Done Sent Done false
22123 7325 Acked Flush 0 Done Sent Done false
22124 7326 Acked Read 9 Done Sent Done false
22125 7327 Acked Write 10 Done Sent Done false
22126 7328 Acked Flush 0 Done Sent Done false
22127 7329 Acked Read 8 Done Sent Done false
22128 7330 Acked Write 6 Done Sent Done false
22129 7331 Acked Write 2 Done Sent Done false
22130 7332 Acked Read 7 Done Sent Done false
22131 7333 Acked Flush 0 Done Sent Done false
22132 7334 Acked Flush 0 Done Sent Done false
22133 7335 Acked Read 5 Done Sent Done false
22134 7336 Acked Read 9 Done Sent Done false
22135 7337 Acked Write 4 Done Sent Done false
22136 STATES DS:0 DS:1 DS:2 TOTAL
22137 Sent 0 301 0 301
22138 Done 301 0 301 602
22139 Skipped 0 0 0 0
22140 Error 0 0 0 0
22141 Last Flush: Some(JobId(7334)) Some(JobId(7036)) Some(JobId(7334))
22142 Downstairs last five completed: 7036 7035 7034 7033 7032
22143 Upstairs last five completed: 7337 7336 7335 7334 7333
22144 CLIENT: Up:0 ds:301 act:2
22145 [8820] got start action
22146 [8820] I am going to restart
22147 Starting downstairs at port 8820
22148 Make output file at "/tmp/dsc/downstairs-8820.txt"
22149 [8820][1] reports Starting
22150 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1221)
22151 [8820][1] reports Running
22152 ----------------------------------------------------------------
22153 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
22154 States: Active Offline Active
22155 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22156 7037 Acked Write 1 Done Sent Done false
22157 7038 Acked Read 8 Done Sent Done false
22158 7039 Acked Write 3 Done Sent Done false
22159 7040 Acked Read 8 Done Sent Done false
22160 7041 Acked Read 4 Done Sent Done false
22161 7042 Acked Write 6 Done Sent Done false
22162 7043 Acked Write 5 Done Sent Done false
22163 7044 Acked Write 5 Done Sent Done false
22164 7045 Acked Read 5 Done Sent Done false
22165 7046 Acked Write 3 Done Sent Done false
22166 7047 Acked Write 3 Done Sent Done false
22167 7048 Acked Write 1 Done Sent Done false
22168 7049 Acked Read 10 Done Sent Done false
22169 7050 Acked Write 6 Done Sent Done false
22170 7051 Acked Flush 0 Done Sent Done false
22171 7052 Acked Write 2 Done Sent Done false
22172 7053 Acked Read 10 Done Sent Done false
22173 7054 Acked Write 6 Done Sent Done false
22174 7055 Acked Write 1 Done Sent Done false
22175 7056 Acked Write 2 Done Sent Done false
22176 7057 Acked Flush 0 Done Sent Done false
22177 7058 Acked Write 1 Done Sent Done false
22178 7059 Acked Read 1 Done Sent Done false
22179 7060 Acked Write 7 Done Sent Done false
22180 7061 Acked Write 7 Done Sent Done false
22181 7062 Acked Write 7 Done Sent Done false
22182 7063 Acked Write 10 Done Sent Done false
22183 7064 Acked Read 5 Done Sent Done false
22184 7065 Acked Write 1 Done Sent Done false
22185 7066 Acked Write 7 Done Sent Done false
22186 7067 Acked Read 10 Done Sent Done false
22187 7068 Acked Write 10 Done Sent Done false
22188 7069 Acked Read 2 Done Sent Done false
22189 7070 Acked Write 7 Done Sent Done false
22190 7071 Acked Write 8 Done Sent Done false
22191 7072 Acked Read 5 Done Sent Done false
22192 7073 Acked Write 7 Done Sent Done false
22193 7074 Acked Write 4 Done Sent Done false
22194 7075 Acked Write 8 Done Sent Done false
22195 7076 Acked Write 2 Done Sent Done false
22196 7077 Acked Read 10 Done Sent Done false
22197 7078 Acked Read 9 Done Sent Done false
22198 7079 Acked Read 1 Done Sent Done false
22199 7080 Acked Read 9 Done Sent Done false
22200 7081 Acked Read 3 Done Sent Done false
22201 7082 Acked Write 8 Done Sent Done false
22202 7083 Acked Read 1 Done Sent Done false
22203 7084 Acked Write 7 Done Sent Done false
22204 7085 Acked Write 3 Done Sent Done false
22205 7086 Acked Write 9 Done Sent Done false
22206 7087 Acked Write 5 Done Sent Done false
22207 7088 Acked Write 1 Done Sent Done false
22208 7089 Acked Write 2 Done Sent Done false
22209 7090 Acked Flush 0 Done Sent Done false
22210 7091 Acked Write 9 Done Sent Done false
22211 7092 Acked Read 5 Done Sent Done false
22212 7093 Acked Read 8 Done Sent Done false
22213 7094 Acked Write 5 Done Sent Done false
22214 7095 Acked Write 2 Done Sent Done false
22215 7096 Acked Write 9 Done Sent Done false
22216 7097 Acked Flush 0 Done Sent Done false
22217 7098 Acked Read 6 Done Sent Done false
22218 7099 Acked Flush 0 Done Sent Done false
22219 7100 Acked Read 3 Done Sent Done false
22220 7101 Acked Write 3 Done Sent Done false
22221 7102 Acked Read 6 Done Sent Done false
22222 7103 Acked Read 10 Done Sent Done false
22223 7104 Acked Write 4 Done Sent Done false
22224 7105 Acked Write 4 Done Sent Done false
22225 7106 Acked Write 8 Done Sent Done false
22226 7107 Acked Read 9 Done Sent Done false
22227 7108 Acked Flush 0 Done Sent Done false
22228 7109 Acked Read 10 Done Sent Done false
22229 7110 Acked Flush 0 Done Sent Done false
22230 7111 Acked Flush 0 Done Sent Done false
22231 7112 Acked Write 9 Done Sent Done false
22232 7113 Acked Write 9 Done Sent Done false
22233 7114 Acked Read 9 Done Sent Done false
22234 7115 Acked Read 3 Done Sent Done false
22235 7116 Acked Flush 0 Done Sent Done false
22236 7117 Acked Read 8 Done Sent Done false
22237 7118 Acked Read 7 Done Sent Done false
22238 7119 Acked Read 8 Done Sent Done false
22239 7120 Acked Read 6 Done Sent Done false
22240 7121 Acked Read 1 Done Sent Done false
22241 7122 Acked Read 3 Done Sent Done false
22242 7123 Acked Read 3 Done Sent Done false
22243 7124 Acked Read 3 Done Sent Done false
22244 7125 Acked Read 7 Done Sent Done false
22245 7126 Acked Read 3 Done Sent Done false
22246 7127 Acked Read 9 Done Sent Done false
22247 7128 Acked Read 7 Done Sent Done false
22248 7129 Acked Write 9 Done Sent Done false
22249 7130 Acked Flush 0 Done Sent Done false
22250 7131 Acked Read 3 Done Sent Done false
22251 7132 Acked Write 10 Done Sent Done false
22252 7133 Acked Read 5 Done Sent Done false
22253 7134 Acked Read 3 Done Sent Done false
22254 7135 Acked Read 8 Done Sent Done false
22255 7136 Acked Write 6 Done Sent Done false
22256 7137 Acked Read 7 Done Sent Done false
22257 7138 Acked Write 10 Done Sent Done false
22258 7139 Acked Write 10 Done Sent Done false
22259 7140 Acked Read 3 Done Sent Done false
22260 7141 Acked Read 8 Done Sent Done false
22261 7142 Acked Write 3 Done Sent Done false
22262 7143 Acked Flush 0 Done Sent Done false
22263 7144 Acked Write 4 Done Sent Done false
22264 7145 Acked Read 1 Done Sent Done false
22265 7146 Acked Read 9 Done Sent Done false
22266 7147 Acked Read 2 Done Sent Done false
22267 7148 Acked Write 2 Done Sent Done false
22268 7149 Acked Read 6 Done Sent Done false
22269 7150 Acked Read 5 Done Sent Done false
22270 7151 Acked Write 7 Done Sent Done false
22271 7152 Acked Read 1 Done Sent Done false
22272 7153 Acked Write 4 Done Sent Done false
22273 7154 Acked Read 6 Done Sent Done false
22274 7155 Acked Flush 0 Done Sent Done false
22275 7156 Acked Read 6 Done Sent Done false
22276 7157 Acked Read 4 Done Sent Done false
22277 7158 Acked Read 4 Done Sent Done false
22278 7159 Acked Read 6 Done Sent Done false
22279 7160 Acked Write 9 Done Sent Done false
22280 7161 Acked Read 2 Done Sent Done false
22281 7162 Acked Read 2 Done Sent Done false
22282 7163 Acked Write 6 Done Sent Done false
22283 7164 Acked Flush 0 Done Sent Done false
22284 7165 Acked Flush 0 Done Sent Done false
22285 7166 Acked Read 6 Done Sent Done false
22286 7167 Acked Write 5 Done Sent Done false
22287 7168 Acked Read 2 Done Sent Done false
22288 7169 Acked Read 5 Done Sent Done false
22289 7170 Acked Write 6 Done Sent Done false
22290 7171 Acked Read 5 Done Sent Done false
22291 7172 Acked Write 10 Done Sent Done false
22292 7173 Acked Read 5 Done Sent Done false
22293 7174 Acked Read 4 Done Sent Done false
22294 7175 Acked Flush 0 Done Sent Done false
22295 7176 Acked Read 8 Done Sent Done false
22296 7177 Acked Read 5 Done Sent Done false
22297 7178 Acked Read 3 Done Sent Done false
22298 7179 Acked Write 7 Done Sent Done false
22299 7180 Acked Read 8 Done Sent Done false
22300 7181 Acked Read 1 Done Sent Done false
22301 7182 Acked Read 8 Done Sent Done false
22302 7183 Acked Write 8 Done Sent Done false
22303 7184 Acked Write 2 Done Sent Done false
22304 7185 Acked Read 3 Done Sent Done false
22305 7186 Acked Flush 0 Done Sent Done false
22306 7187 Acked Read 8 Done Sent Done false
22307 7188 Acked Write 1 Done Sent Done false
22308 7189 Acked Read 2 Done Sent Done false
22309 7190 Acked Read 1 Done Sent Done false
22310 7191 Acked Write 6 Done Sent Done false
22311 7192 Acked Read 10 Done Sent Done false
22312 7193 Acked Write 2 Done Sent Done false
22313 7194 Acked Read 3 Done Sent Done false
22314 7195 Acked Write 3 Done Sent Done false
22315 7196 Acked Flush 0 Done Sent Done false
22316 7197 Acked Read 5 Done Sent Done false
22317 7198 Acked Read 5 Done Sent Done false
22318 7199 Acked Flush 0 Done Sent Done false
22319 7200 Acked Write 4 Done Sent Done false
22320 7201 Acked Read 9 Done Sent Done false
22321 7202 Acked Read 7 Done Sent Done false
22322 7203 Acked Read 6 Done Sent Done false
22323 7204 Acked Read 1 Done Sent Done false
22324 7205 Acked Read 2 Done Sent Done false
22325 7206 Acked Read 1 Done Sent Done false
22326 7207 Acked Read 10 Done Sent Done false
22327 7208 Acked Write 4 Done Sent Done false
22328 7209 Acked Write 1 Done Sent Done false
22329 7210 Acked Write 8 Done Sent Done false
22330 7211 Acked Read 3 Done Sent Done false
22331 7212 Acked Read 10 Done Sent Done false
22332 7213 Acked Read 3 Done Sent Done false
22333 7214 Acked Write 7 Done Sent Done false
22334 7215 Acked Read 9 Done Sent Done false
22335 7216 Acked Flush 0 Done Sent Done false
22336 7217 Acked Write 6 Done Sent Done false
22337 7218 Acked Write 4 Done Sent Done false
22338 7219 Acked Read 10 Done Sent Done false
22339 7220 Acked Read 10 Done Sent Done false
22340 7221 Acked Read 6 Done Sent Done false
22341 7222 Acked Read 7 Done Sent Done false
22342 7223 Acked Read 1 Done Sent Done false
22343 7224 Acked Read 4 Done Sent Done false
22344 7225 Acked Write 2 Done Sent Done false
22345 7226 Acked Read 8 Done Sent Done false
22346 7227 Acked Read 3 Done Sent Done false
22347 7228 Acked Read 10 Done Sent Done false
22348 7229 Acked Read 1 Done Sent Done false
22349 7230 Acked Read 4 Done Sent Done false
22350 7231 Acked Write 9 Done Sent Done false
22351 7232 Acked Read 7 Done Sent Done false
22352 7233 Acked Flush 0 Done Sent Done false
22353 7234 Acked Write 10 Done Sent Done false
22354 7235 Acked Read 4 Done Sent Done false
22355 7236 Acked Write 2 Done Sent Done false
22356 7237 Acked Flush 0 Done Sent Done false
22357 7238 Acked Read 1 Done Sent Done false
22358 7239 Acked Read 5 Done Sent Done false
22359 7240 Acked Read 1 Done Sent Done false
22360 7241 Acked Flush 0 Done Sent Done false
22361 7242 Acked Flush 0 Done Sent Done false
22362 7243 Acked Read 4 Done Sent Done false
22363 7244 Acked Read 1 Done Sent Done false
22364 7245 Acked Write 3 Done Sent Done false
22365 7246 Acked Read 5 Done Sent Done false
22366 7247 Acked Read 9 Done Sent Done false
22367 7248 Acked Read 3 Done Sent Done false
22368 7249 Acked Read 10 Done Sent Done false
22369 7250 Acked Read 6 Done Sent Done false
22370 7251 Acked Flush 0 Done Sent Done false
22371 7252 Acked Read 10 Done Sent Done false
22372 7253 Acked Read 2 Done Sent Done false
22373 7254 Acked Write 1 Done Sent Done false
22374 7255 Acked Read 5 Done Sent Done false
22375 7256 Acked Read 10 Done Sent Done false
22376 7257 Acked Write 8 Done Sent Done false
22377 7258 Acked Write 10 Done Sent Done false
22378 7259 Acked Read 7 Done Sent Done false
22379 7260 Acked Read 1 Done Sent Done false
22380 7261 Acked Read 4 Done Sent Done false
22381 7262 Acked Read 5 Done Sent Done false
22382 7263 Acked Write 4 Done Sent Done false
22383 7264 Acked Read 10 Done Sent Done false
22384 7265 Acked Write 2 Done Sent Done false
22385 7266 Acked Write 5 Done Sent Done false
22386 7267 Acked Read 9 Done Sent Done false
22387 7268 Acked Read 8 Done Sent Done false
22388 7269 Acked Write 4 Done Sent Done false
22389 7270 Acked Read 9 Done Sent Done false
22390 7271 Acked Read 8 Done Sent Done false
22391 7272 Acked Write 10 Done Sent Done false
22392 7273 Acked Read 2 Done Sent Done false
22393 7274 Acked Write 2 Done Sent Done false
22394 7275 Acked Read 3 Done Sent Done false
22395 7276 Acked Read 9 Done Sent Done false
22396 7277 Acked Read 10 Done Sent Done false
22397 7278 Acked Read 8 Done Sent Done false
22398 7279 Acked Write 10 Done Sent Done false
22399 7280 Acked Write 5 Done Sent Done false
22400 7281 Acked Read 10 Done Sent Done false
22401 7282 Acked Write 7 Done Sent Done false
22402 7283 Acked Write 1 Done Sent Done false
22403 7284 Acked Write 5 Done Sent Done false
22404 7285 Acked Read 3 Done Sent Done false
22405 7286 Acked Read 7 Done Sent Done false
22406 7287 Acked Read 1 Done Sent Done false
22407 7288 Acked Read 1 Done Sent Done false
22408 7289 Acked Write 4 Done Sent Done false
22409 7290 Acked Flush 0 Done Sent Done false
22410 7291 Acked Write 10 Done Sent Done false
22411 7292 Acked Write 7 Done Sent Done false
22412 7293 Acked Write 4 Done Sent Done false
22413 7294 Acked Read 2 Done Sent Done false
22414 7295 Acked Read 6 Done Sent Done false
22415 7296 Acked Write 2 Done Sent Done false
22416 7297 Acked Write 2 Done Sent Done false
22417 7298 Acked Read 8 Done Sent Done false
22418 7299 Acked Read 4 Done Sent Done false
22419 7300 Acked Write 9 Done Sent Done false
22420 7301 Acked Write 2 Done Sent Done false
22421 7302 Acked Write 5 Done Sent Done false
22422 7303 Acked Read 5 Done Sent Done false
22423 7304 Acked Read 10 Done Sent Done false
22424 7305 Acked Write 4 Done Sent Done false
22425 7306 Acked Write 6 Done Sent Done false
22426 7307 Acked Read 6 Done Sent Done false
22427 7308 Acked Read 10 Done Sent Done false
22428 7309 Acked Read 2 Done Sent Done false
22429 7310 Acked Write 7 Done Sent Done false
22430 7311 Acked Write 6 Done Sent Done false
22431 7312 Acked Flush 0 Done Sent Done false
22432 7313 Acked Write 3 Done Sent Done false
22433 7314 Acked Write 1 Done Sent Done false
22434 7315 Acked Write 7 Done Sent Done false
22435 7316 Acked Write 1 Done Sent Done false
22436 7317 Acked Flush 0 Done Sent Done false
22437 7318 Acked Read 3 Done Sent Done false
22438 7319 Acked Read 1 Done Sent Done false
22439 7320 Acked Read 7 Done Sent Done false
22440 7321 Acked Read 5 Done Sent Done false
22441 7322 Acked Read 2 Done Sent Done false
22442 7323 Acked Flush 0 Done Sent Done false
22443 7324 Acked Read 7 Done Sent Done false
22444 7325 Acked Flush 0 Done Sent Done false
22445 7326 Acked Read 9 Done Sent Done false
22446 7327 Acked Write 10 Done Sent Done false
22447 7328 Acked Flush 0 Done Sent Done false
22448 7329 Acked Read 8 Done Sent Done false
22449 7330 Acked Write 6 Done Sent Done false
22450 7331 Acked Write 2 Done Sent Done false
22451 7332 Acked Read 7 Done Sent Done false
22452 7333 Acked Flush 0 Done Sent Done false
22453 7334 Acked Flush 0 Done Sent Done false
22454 7335 Acked Read 5 Done Sent Done false
22455 7336 Acked Read 9 Done Sent Done false
22456 7337 Acked Write 4 Done Sent Done false
22457 7338 Acked Flush 0 Done Sent Done false
22458 STATES DS:0 DS:1 DS:2 TOTAL
22459 Sent 0 302 0 302
22460 Done 302 0 302 604
22461 Skipped 0 0 0 0
22462 Error 0 0 0 0
22463 Last Flush: Some(JobId(7338)) Some(JobId(7036)) Some(JobId(7338))
22464 Downstairs last five completed: 7036 7035 7034 7033 7032
22465 Upstairs last five completed: 7338 7337 7336 7335 7334
22466 CLIENT: Up:0 ds:302 act:2
224672025-01-23T18:52:35.946ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
224682025-01-23T18:52:35.946ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49172) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
224692025-01-23T18:52:35.947ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
224702025-01-23T18:52:35.947ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
224712025-01-23T18:52:35.947ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
224722025-01-23T18:52:35.947ZINFOcrucible: send last flush ID to this DS: Some(JobId(7036)) = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
224732025-01-23T18:52:35.947ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
224742025-01-23T18:52:35.947ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(7036)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
224752025-01-23T18:52:35.948ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(7036)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
22476 ----------------------------------------------------------------
22477 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22478 Downstairs last five completed: 7338 7337 7336 7335 7334
22479 Upstairs last five completed: 7338 7337 7336 7335 7334
22480 CLIENT: Up:0 ds:0 act:3
22481 Replay: All jobs finished
22482 Main task has work to do, go find it
22483 got dsc Stop(1)
22484 stop 1
22485 [8820] Got stop action so:false kr:false
22486 Jan 23 18:52:38.195 INFO request completed, latency_us: 320, response_code: 204, uri: /stop/cid/1, method: POST, req_id: c90ba47d-0471-4896-a422-f15ed04e5bf6, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
224872025-01-23T18:52:38.195ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
224882025-01-23T18:52:38.195ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
224892025-01-23T18:52:38.195ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
22490 Jan 23 18:52:38.195 INFO request completed, latency_us: 121, response_code: 200, uri: /state/cid/1, method: GET, req_id: 661146ac-e56d-406c-836a-7f4d597b43ea, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
224912025-01-23T18:52:38.195ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
22492 [8820] Exited with: ExitStatus(unix_wait_status(9))
22493 [8820][1] reports Exit
22494 Jan 23 18:52:42.198 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/1, method: GET, req_id: 9734ed38-20fe-4474-90f5-acb8abaf3f35, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
22495 001/300 Read block 1896 len 32768
22496 002/300 Read block 4282 len 32768
22497 003/300 Read block 2396 len 8192
22498 004/300 Write block 3075 len 24576 data: 3 3 4 3 4 4
22499 005/300 Flush
22500 006/300 Read block 426 len 24576
22501 007/300 Read block 4822 len 32768
22502 008/300 Read block 1317 len 24576
22503 009/300 Read block 4702 len 20480
22504 010/300 Flush
22505 011/300 Read block 4046 len 4096
22506 012/300 Write block 3387 len 4096 data: 9
22507 013/300 Read block 520 len 16384
22508 014/300 Write block 2735 len 28672 data: 4 5 6 6 5 4 5
22509 015/300 Write block 1593 len 24576 data: 5 6 6 8 8 6
22510 016/300 Read block 884 len 20480
22511 017/300 Read block 2852 len 32768
22512 018/300 Read block 3894 len 24576
22513 019/300 Read block 2277 len 24576
22514 020/300 Write block 3475 len 24576 data: 2 2 5 4 4 5
22515 021/300 Write block 2352 len 20480 data: 3 2 2 2 2
22516 022/300 Read block 2985 len 20480
22517 023/300 Flush
22518 024/300 Read block 3328 len 8192
22519 025/300 Write block 2801 len 16384 data: 4 4 5 5
22520 026/300 Read block 4115 len 24576
22521 027/300 Flush
22522 028/300 Read block 1929 len 12288
22523 029/300 Read block 4257 len 4096
22524 030/300 Write block 855 len 8192 data: 5 5
22525 031/300 Read block 1936 len 20480
22526 032/300 Read block 4525 len 40960
22527 033/300 Write block 1537 len 40960 data: 3 2 2 2 3 3 4 4 4 4
22528 034/300 Write block 428 len 20480 data: 10 11 10 9 6
22529 035/300 Read block 2320 len 24576
22530 036/300 Write block 450 len 28672 data: 4 4 5 5 6 5 6
22531 037/300 Read block 3395 len 16384
22532 038/300 Write block 4834 len 20480 data: 7 6 6 5 4
22533 039/300 Write block 2572 len 40960 data: 4 4 6 7 6 9 7 7 8 8
22534 040/300 Write block 2842 len 24576 data: 5 5 6 6 8 6
22535 041/300 Write block 1284 len 20480 data: 6 6 8 7 7
22536 042/300 Read block 2756 len 32768
22537 043/300 Read block 3001 len 36864
22538 044/300 Write block 2530 len 24576 data: 2 2 3 4 5 6
22539 045/300 Read block 3157 len 40960
22540 046/300 Read block 3094 len 28672
22541 047/300 Write block 3482 len 20480 data: 5 4 4 4 5
22542 048/300 Read block 1298 len 40960
22543 049/300 Write block 1902 len 36864 data: 5 6 6 5 4 3 3 3 3
22544 050/300 Read block 1323 len 16384
22545 051/300 Read block 736 len 16384
22546 052/300 Read block 3979 len 20480
22547 053/300 Write block 3958 len 40960 data: 5 4 4 5 4 3 4 4 3 3
22548 054/300 Write block 63 len 36864 data: 5 5 5 3 3 4 4 5 6
22549 055/300 Read block 158 len 12288
22550 056/300 Read block 4912 len 4096
22551 057/300 Write block 3781 len 12288 data: 4 4 4
22552 058/300 Write block 1631 len 16384 data: 5 5 5 5
22553 059/300 Read block 2611 len 20480
22554 060/300 Flush
22555 061/300 Read block 758 len 28672
22556 062/300 Read block 2690 len 24576
22557 063/300 Write block 481 len 20480 data: 7 5 5 5 6
22558 064/300 Read block 2346 len 4096
22559 065/300 Write block 2503 len 24576 data: 4 4 3 4 4 4
22560 066/300 Write block 262 len 8192 data: 3 3
22561 067/300 Flush
22562 068/300 Write block 2547 len 40960 data: 4 4 3 3 2 2 3 2 3 3
22563 069/300 Read block 3903 len 8192
22564 070/300 Write block 285 len 8192 data: 4 4
22565 071/300 Write block 3545 len 8192 data: 6 4
22566 072/300 Read block 530 len 16384
22567 073/300 Write block 2883 len 20480 data: 2 3 4 5 5
22568 074/300 Write block 752 len 32768 data: 2 2 2 3 3 3 5 4
22569 075/300 Write block 2763 len 36864 data: 5 5 5 5 5 5 5 4 4
22570 076/300 Flush
22571 077/300 Read block 2243 len 32768
22572 078/300 Flush
22573 079/300 Flush
22574 080/300 Write block 874 len 8192 data: 5 6
22575 081/300 Read block 2264 len 40960
22576 082/300 Read block 1266 len 12288
22577 083/300 Flush
22578 084/300 Write block 4579 len 12288 data: 6 6 6
22579 085/300 Read block 3965 len 28672
22580 086/300 Read block 300 len 32768
22581 087/300 Read block 3655 len 20480
22582 088/300 Read block 3930 len 12288
22583 089/300 Read block 2682 len 32768
22584 090/300 Write block 1385 len 12288 data: 5 4 6
22585 091/300 Read block 794 len 20480
22586 092/300 Read block 1341 len 32768
22587 093/300 Flush
22588 094/300 Read block 4321 len 36864
22589 095/300 Write block 3417 len 16384 data: 6 7 6 6
22590 096/300 Read block 2841 len 4096
22591 097/300 Read block 623 len 16384
22592 098/300 Write block 3629 len 8192 data: 4 3
22593 099/300 Write block 776 len 20480 data: 8 8 7 8 9
22594 100/300 Write block 1777 len 4096 data: 8
22595 101/300 Read block 741 len 20480
22596 102/300 Read block 2227 len 24576
22597 103/300 Read block 853 len 16384
22598 104/300 Write block 4503 len 20480 data: 6 7 6 6 6
22599 105/300 Write block 4420 len 8192 data: 5 6
22600 106/300 Read block 4645 len 36864
22601 107/300 Read block 2362 len 8192
22602 108/300 Read block 223 len 40960
22603 109/300 Write block 998 len 28672 data: 4 5 5 5 7 7 7
22604 110/300 Write block 3778 len 8192 data: 6 6
22605 111/300 Flush
22606 112/300 Read block 774 len 32768
22607 113/300 Read block 3483 len 12288
22608 114/300 Read block 3708 len 32768
22609 115/300 Read block 1571 len 32768
22610 116/300 Read block 4959 len 36864
22611 117/300 Write block 4134 len 16384 data: 3 4 4 4
22612 118/300 Read block 1835 len 8192
22613 119/300 Read block 3332 len 20480
22614 120/300 Write block 1378 len 12288 data: 3 3 3
22615 121/300 Read block 2272 len 40960
22616 122/300 Flush
22617 123/300 Read block 173 len 28672
22618 124/300 Read block 4847 len 12288
22619 125/300 Flush
22620 126/300 Read block 1333 len 32768
22621 127/300 Read block 2907 len 28672
22622 128/300 Write block 3750 len 40960 data: 6 6 8 8 7 7 6 5 5 5
22623 129/300 Write block 4811 len 16384 data: 7 6 6 4
22624 130/300 Read block 2238 len 28672
22625 131/300 Write block 2378 len 36864 data: 3 4 3 3 3 3 4 4 3
22626 132/300 Read block 250 len 40960
22627 133/300 Read block 1836 len 36864
22628 134/300 Write block 1790 len 12288 data: 7 6 6
22629 135/300 Read block 463 len 12288
22630 136/300 Read block 2748 len 12288
22631 137/300 Write block 1466 len 36864 data: 8 7 6 6 5 4 4 4 4
22632 138/300 Read block 1227 len 20480
22633 139/300 Read block 1830 len 4096
22634 140/300 Write block 3144 len 16384 data: 3 4 3 3
22635 141/300 Flush
22636 142/300 Write block 3116 len 40960 data: 4 4 3 3 5 6 5 5 4 4
22637 143/300 Write block 3421 len 40960 data: 7 9 10 9 8 6 5 5 4 4
22638 144/300 Write block 3398 len 8192 data: 2 2
22639 145/300 Read block 4643 len 12288
22640 146/300 Flush
22641 147/300 Read block 3601 len 32768
22642 148/300 Flush
22643 149/300 Write block 3556 len 4096 data: 6
22644 150/300 Read block 198 len 12288
22645 151/300 Read block 3342 len 28672
22646 152/300 Read block 1050 len 20480
22647 153/300 Write block 882 len 24576 data: 5 4 4 4 4 5
22648 154/300 Write block 4405 len 4096 data: 3
22649 155/300 Write block 3350 len 40960 data: 5 5 5 5 6 4 4 5 4 6
22650 156/300 Write block 4029 len 28672 data: 8 8 8 6 5 4 3
22651 157/300 Write block 2111 len 4096 data: 5
22652 158/300 Write block 3477 len 36864 data: 6 5 5 6 5 6 5 5 5
22653 159/300 Write block 3648 len 40960 data: 4 3 3 3 5 6 4 4 4 3
22654 160/300 Flush
22655 161/300 Read block 26 len 24576
22656 162/300 Read block 306 len 32768
22657 163/300 Write block 1040 len 20480 data: 3 4 5 5 6
22658 164/300 Write block 3241 len 4096 data: 6
22659 165/300 Flush
22660 166/300 Flush
22661 167/300 Write block 4495 len 20480 data: 7 6 6 5 6
22662 168/300 Write block 883 len 8192 data: 5 5
22663 169/300 Read block 3353 len 4096
22664 170/300 Read block 3047 len 8192
22665 171/300 Write block 4612 len 4096 data: 3
22666 172/300 Write block 556 len 16384 data: 4 5 6 6
22667 173/300 Write block 405 len 8192 data: 8 6
22668 174/300 Read block 2916 len 36864
22669 175/300 Read block 677 len 28672
22670 176/300 Read block 2898 len 36864
22671 177/300 Write block 2583 len 16384 data: 4 4 6 7
22672 178/300 Write block 1854 len 24576 data: 6 5 5 4 5 6
22673 179/300 Read block 3726 len 24576
22674 180/300 Write block 1582 len 36864 data: 4 5 5 3 4 4 3 4 4
22675 181/300 Write block 1896 len 36864 data: 4 4 5 4 5 5 6 7 7
22676 182/300 Read block 1824 len 16384
22677 183/300 Write block 3810 len 4096 data: 4
22678 184/300 Read block 183 len 4096
22679 185/300 Write block 4913 len 32768 data: 5 5 7 8 7 6 3 3
22680 186/300 Read block 2156 len 12288
22681 187/300 Read block 2017 len 32768
22682 188/300 Read block 3991 len 40960
22683 189/300 Read block 1610 len 4096
22684 190/300 Write block 4825 len 20480 data: 5 5 5 5 4
22685 191/300 Write block 2993 len 20480 data: 5 6 8 6 5
22686 192/300 Write block 2240 len 32768 data: 4 5 4 4 4 4 4 3
22687 193/300 Read block 4892 len 16384
22688 194/300 Write block 2194 len 12288 data: 5 5 6
22689 195/300 Write block 4192 len 28672 data: 4 4 4 5 5 4 5
22690 196/300 Read block 3215 len 32768
22691 197/300 Write block 147 len 12288 data: 6 7 7
22692 198/300 Read block 3566 len 12288
22693 199/300 Flush
22694 200/300 Write block 1462 len 36864 data: 6 5 6 9 9 8 7 7 6
22695 201/300 Read block 4927 len 20480
22696 202/300 Flush
22697 203/300 Read block 1095 len 32768
22698 204/300 Read block 4291 len 4096
22699 205/300 Read block 1944 len 16384
22700 206/300 Read block 1563 len 12288
22701 207/300 Flush
22702 208/300 Read block 1650 len 12288
22703 209/300 Write block 4193 len 32768 data: 5 5 6 6 5 6 5 4
22704 210/300 Read block 2893 len 16384
22705 211/300 Read block 3395 len 12288
22706 212/300 Write block 4682 len 12288 data: 4 5 3
22707 213/300 Read block 2922 len 16384
22708 214/300 Write block 2885 len 32768 data: 5 6 6 7 6 6 7 6
22709 215/300 Flush
22710 216/300 Flush
22711 217/300 Read block 3501 len 28672
22712 218/300 Write block 3247 len 16384 data: 6 6 6 6
22713 219/300 Write block 636 len 4096 data: 3
22714 220/300 Read block 2248 len 40960
22715 221/300 Read block 1895 len 40960
22716 222/300 Read block 3232 len 24576
22717 223/300 Read block 16 len 28672
22718 224/300 Write block 3177 len 16384 data: 2 2 2 3
22719 225/300 Write block 3825 len 32768 data: 4 4 5 5 6 6 6 6
22720 226/300 Read block 271 len 32768
22721 227/300 Write block 4961 len 36864 data: 6 7 6 7 6 5 4 4 4
22722 228/300 Write block 2159 len 4096 data: 8
22723 229/300 Read block 3878 len 28672
22724 230/300 Write block 1137 len 12288 data: 5 5 7
22725 231/300 Read block 966 len 16384
22726 232/300 Read block 3637 len 12288
22727 233/300 Write block 183 len 40960 data: 5 5 5 5 6 5 5 4 4 3
22728 234/300 Write block 4959 len 28672 data: 5 6 7 8 7 8 7
22729 235/300 Read block 4906 len 8192
22730 236/300 Read block 2890 len 12288
22731 237/300 Read block 764 len 36864
22732 238/300 Read block 1787 len 24576
22733 239/300 Write block 675 len 12288 data: 7 7 6
22734 240/300 Write block 1563 len 32768 data: 6 6 5 6 6 6 5 6
22735 241/300 Write block 862 len 40960 data: 5 6 6 6 5 5 6 5 4 4
22736 242/300 Write block 2744 len 32768 data: 5 4 5 5 5 5 5 5
22737 243/300 Read block 4126 len 4096
22738 244/300 Flush
22739 245/300 Read block 638 len 24576
22740 246/300 Read block 1445 len 36864
22741 247/300 Read block 3623 len 24576
22742 248/300 Read block 362 len 8192
22743 249/300 Read block 4926 len 36864
22744 250/300 Write block 4739 len 4096 data: 2
22745 251/300 Write block 313 len 28672 data: 7 6 4 3 3 3 3
22746 252/300 Read block 165 len 24576
22747 253/300 Read block 2907 len 36864
22748 254/300 Read block 4771 len 32768
22749 255/300 Write block 2447 len 20480 data: 5 9 9 9 8
22750 256/300 Flush
22751 257/300 Read block 3415 len 4096
22752 258/300 Read block 1547 len 24576
22753 259/300 Read block 767 len 40960
22754 260/300 Read block 4814 len 32768
22755 261/300 Read block 4753 len 12288
22756 262/300 Read block 941 len 8192
22757 263/300 Read block 4963 len 32768
22758 264/300 Write block 1138 len 20480 data: 6 8 7 7 6
22759 265/300 Write block 1184 len 36864 data: 2 2 4 4 3 3 3 6 5
22760 266/300 Read block 3914 len 32768
22761 267/300 Write block 4959 len 12288 data: 6 7 8
22762 268/300 Flush
22763 269/300 Read block 2750 len 32768
22764 270/300 Flush
22765 271/300 Read block 726 len 24576
22766 272/300 Write block 3348 len 12288 data: 5 6 6
22767 273/300 Read block 4649 len 28672
22768 274/300 Read block 3064 len 12288
22769 275/300 Write block 2613 len 40960 data: 6 5 7 7 7 8 7 7 7 6
22770 276/300 Flush
22771 277/300 Read block 1795 len 4096
22772 278/300 Read block 1413 len 12288
22773 279/300 Read block 4027 len 12288
22774 280/300 Read block 1114 len 24576
22775 281/300 Read block 795 len 40960
22776 282/300 Flush
22777 283/300 Write block 4917 len 28672 data: 8 7 4 4 4 3 3
22778 284/300 Read block 3940 len 40960
22779 285/300 Read block 4656 len 24576
22780 286/300 Read block 3958 len 8192
22781 287/300 Read block 4144 len 40960
22782 288/300 Read block 1166 len 12288
22783 289/300 Write block 956 len 20480 data: 6 7 8 7 6
22784 290/300 Write block 537 len 40960 data: 5 4 4 4 5 7 6 5 4 5
22785 291/300 Read block 2770 len 24576
22786 292/300 Read block 3531 len 40960
22787 293/300 Read block 1193 len 8192
22788 294/300 Write block 2186 len 36864 data: 4 3 3 3 3 3 3 3 6
22789 295/300 Write block 629 len 20480 data: 5 4 4 5 5
22790 296/300 Read block 2826 len 20480
22791 297/300 Read block 3046 len 8192
22792 298/300 Read block 1419 len 8192
22793 299/300 Write block 3916 len 28672 data: 4 5 5 6 8 7 7
22794 300/300 Flush
22795 301/300 Write block 47 len 32768 data: 3 4 3 3 2 2 2 3
22796 Main task has work to do, go find it
22797 got dsc Start(1)
22798 start 1
22799 Jan 23 18:52:42.444 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/1, method: POST, req_id: 411630f3-ad60-4f7c-a014-6cd3477887d0, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
22800 Replay: started 1, returned:Ok(())
22801 ----------------------------------------------------------------
22802 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22803 States: Active Offline Active
22804 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22805 7339 Acked Read 8 Done Sent Done false
22806 7340 Acked Read 8 Done Sent Done false
22807 7341 Acked Read 2 Done Sent Done false
22808 7342 Acked Write 6 Done Sent Done false
22809 7343 Acked Flush 0 Done Sent Done false
22810 7344 Acked Read 6 Done Sent Done false
22811 7345 Acked Read 8 Done Sent Done false
22812 7346 Acked Read 6 Done Sent Done false
22813 7347 Acked Read 5 Done Sent Done false
22814 7348 Acked Flush 0 Done Sent Done false
22815 7349 Acked Read 1 Done Sent Done false
22816 7350 Acked Write 1 Done Sent Done false
22817 7351 Acked Read 4 Done Sent Done false
22818 7352 Acked Write 7 Done Sent Done false
22819 7353 Acked Write 6 Done Sent Done false
22820 7354 Acked Read 5 Done Sent Done false
22821 7355 Acked Read 8 Done Sent Done false
22822 7356 Acked Read 6 Done Sent Done false
22823 7357 Acked Read 6 Done Sent Done false
22824 7358 Acked Write 6 Done Sent Done false
22825 7359 Acked Write 5 Done Sent Done false
22826 7360 Acked Read 5 Done Sent Done false
22827 7361 Acked Flush 0 Done Sent Done false
22828 7362 Acked Read 2 Done Sent Done false
22829 7363 Acked Write 4 Done Sent Done false
22830 7364 Acked Read 6 Done Sent Done false
22831 7365 Acked Flush 0 Done Sent Done false
22832 7366 Acked Read 3 Done Sent Done false
22833 7367 Acked Read 1 Done Sent Done false
22834 7368 Acked Write 2 Done Sent Done false
22835 7369 Acked Read 5 Done Sent Done false
22836 7370 Acked Read 10 Done Sent Done false
22837 7371 Acked Write 10 Done Sent Done false
22838 7372 Acked Write 5 Done Sent Done false
22839 7373 Acked Read 6 Done Sent Done false
22840 7374 Acked Write 7 Done Sent Done false
22841 7375 Acked Read 4 Done Sent Done false
22842 7376 Acked Write 5 Done Sent Done false
22843 7377 Acked Write 10 Done Sent Done false
22844 7378 Acked Write 6 Done Sent Done false
22845 7379 Acked Write 5 Done Sent Done false
22846 7380 Acked Read 8 Done Sent Done false
22847 7381 Acked Read 9 Done Sent Done false
22848 7382 Acked Write 6 Done Sent Done false
22849 7383 Acked Read 10 Done Sent Done false
22850 7384 Acked Read 7 Done Sent Done false
22851 7385 Acked Write 5 Done Sent Done false
22852 7386 Acked Read 10 Done Sent Done false
22853 7387 Acked Write 9 Done Sent Done false
22854 7388 Acked Read 4 Done Sent Done false
22855 7389 Acked Read 4 Done Sent Done false
22856 7390 Acked Read 5 Done Sent Done false
22857 7391 Acked Write 10 Done Sent Done false
22858 7392 Acked Write 9 Done Sent Done false
22859 7393 Acked Read 3 Done Sent Done false
22860 7394 Acked Read 1 Done Sent Done false
22861 7395 Acked Write 3 Done Sent Done false
22862 7396 Acked Write 4 Done Sent Done false
22863 7397 Acked Read 5 Done Sent Done false
22864 7398 Acked Flush 0 Done Sent Done false
22865 7399 Acked Read 7 Done Sent Done false
22866 7400 Acked Read 6 Done Sent Done false
22867 7401 Acked Write 5 Done Sent Done false
22868 7402 Acked Read 1 Done Sent Done false
22869 7403 Acked Write 6 Done Sent Done false
22870 7404 Acked Write 2 Done Sent Done false
22871 7405 Acked Flush 0 Done Sent Done false
22872 7406 Acked Write 10 Done Sent Done false
22873 7407 Acked Read 2 Done Sent Done false
22874 7408 Acked Write 2 Done Sent Done false
22875 7409 Acked Write 2 Done Sent Done false
22876 7410 Acked Read 4 Done Sent Done false
22877 7411 Acked Write 5 Done Sent Done false
22878 7412 Acked Write 8 Done Sent Done false
22879 7413 Acked Write 9 Done Sent Done false
22880 7414 Acked Flush 0 Done Sent Done false
22881 7415 Acked Read 8 Done Sent Done false
22882 7416 Acked Flush 0 Done Sent Done false
22883 7417 Acked Flush 0 Done Sent Done false
22884 7418 Acked Write 2 Done Sent Done false
22885 7419 Acked Read 10 Done Sent Done false
22886 7420 Acked Read 3 Done Sent Done false
22887 7421 Acked Flush 0 Done Sent Done false
22888 7422 Acked Write 3 Done Sent Done false
22889 7423 Acked Read 7 Done Sent Done false
22890 7424 Acked Read 8 Done Sent Done false
22891 7425 Acked Read 5 Done Sent Done false
22892 7426 Acked Read 3 Done Sent Done false
22893 7427 Acked Read 8 Done Sent Done false
22894 7428 Acked Write 3 Done Sent Done false
22895 7429 Acked Read 5 Done Sent Done false
22896 7430 Acked Read 8 Done Sent Done false
22897 7431 Acked Flush 0 Done Sent Done false
22898 7432 Acked Read 9 Done Sent Done false
22899 7433 Acked Write 4 Done Sent Done false
22900 7434 Acked Read 1 Done Sent Done false
22901 7435 Acked Read 4 Done Sent Done false
22902 7436 Acked Write 2 Done Sent Done false
22903 7437 Acked Write 5 Done Sent Done false
22904 7438 Acked Write 1 Done Sent Done false
22905 7439 Acked Read 5 Done Sent Done false
22906 7440 Acked Read 6 Done Sent Done false
22907 7441 Acked Read 4 Done Sent Done false
22908 7442 Acked Write 5 Done Sent Done false
22909 7443 Acked Write 2 Done Sent Done false
22910 7444 Acked Read 9 Done Sent Done false
22911 7445 Acked Read 2 Done Sent Done false
22912 7446 Acked Read 10 Done Sent Done false
22913 7447 Acked Write 7 Done Sent Done false
22914 7448 Acked Write 2 Done Sent Done false
22915 7449 Acked Flush 0 Done Sent Done false
22916 7450 Acked Read 8 Done Sent Done false
22917 7451 Acked Read 3 Done Sent Done false
22918 7452 Acked Read 8 Done Sent Done false
22919 7453 Acked Read 8 Done Sent Done false
22920 7454 Acked Read 9 Done Sent Done false
22921 7455 Acked Write 4 Done Sent Done false
22922 7456 Acked Read 2 Done Sent Done false
22923 7457 Acked Read 5 Done Sent Done false
22924 7458 Acked Write 3 Done Sent Done false
22925 7459 Acked Read 10 Done Sent Done false
22926 7460 Acked Flush 0 Done Sent Done false
22927 7461 Acked Read 7 Done Sent Done false
22928 7462 Acked Read 3 Done Sent Done false
22929 7463 Acked Flush 0 Done Sent Done false
22930 7464 Acked Read 8 Done Sent Done false
22931 7465 Acked Read 7 Done Sent Done false
22932 7466 Acked Write 10 Done Sent Done false
22933 7467 Acked Write 4 Done Sent Done false
22934 7468 Acked Read 7 Done Sent Done false
22935 7469 Acked Write 9 Done Sent Done false
22936 7470 Acked Read 10 Done Sent Done false
22937 7471 Acked Read 9 Done Sent Done false
22938 7472 Acked Write 3 Done Sent Done false
22939 7473 Acked Read 3 Done Sent Done false
22940 7474 Acked Read 3 Done Sent Done false
22941 7475 Acked Write 9 Done Sent Done false
22942 7476 Acked Read 5 Done Sent Done false
22943 7477 Acked Read 1 Done Sent Done false
22944 7478 Acked Write 4 Done Sent Done false
22945 7479 Acked Flush 0 Done Sent Done false
22946 7480 Acked Write 10 Done Sent Done false
22947 7481 Acked Write 10 Done Sent Done false
22948 7482 Acked Write 2 Done Sent Done false
22949 7483 Acked Read 3 Done Sent Done false
22950 7484 Acked Flush 0 Done Sent Done false
22951 7485 Acked Read 8 Done Sent Done false
22952 7486 Acked Flush 0 Done Sent Done false
22953 7487 Acked Write 1 Done Sent Done false
22954 7488 Acked Read 3 Done Sent Done false
22955 7489 Acked Read 7 Done Sent Done false
22956 7490 Acked Read 5 Done Sent Done false
22957 7491 Acked Write 6 Done Sent Done false
22958 7492 Acked Write 1 Done Sent Done false
22959 7493 Acked Write 10 Done Sent Done false
22960 7494 Acked Write 7 Done Sent Done false
22961 7495 Acked Write 1 Done Sent Done false
22962 7496 Acked Write 9 Done Sent Done false
22963 7497 Acked Write 10 Done Sent Done false
22964 7498 Acked Flush 0 Done Sent Done false
22965 7499 Acked Read 6 Done Sent Done false
22966 7500 Acked Read 8 Done Sent Done false
22967 7501 Acked Write 5 Done Sent Done false
22968 7502 Acked Write 1 Done Sent Done false
22969 7503 Acked Flush 0 Done Sent Done false
22970 7504 Acked Flush 0 Done Sent Done false
22971 7505 Acked Write 5 Done Sent Done false
22972 7506 Acked Write 2 Done Sent Done false
22973 7507 Acked Read 1 Done Sent Done false
22974 7508 Acked Read 2 Done Sent Done false
22975 7509 Acked Write 1 Done Sent Done false
22976 7510 Acked Write 4 Done Sent Done false
22977 7511 Acked Write 2 Done Sent Done false
22978 7512 Acked Read 9 Done Sent Done false
22979 7513 Acked Read 7 Done Sent Done false
22980 7514 Acked Read 9 Done Sent Done false
22981 7515 Acked Write 4 Done Sent Done false
22982 7516 Acked Write 6 Done Sent Done false
22983 7517 Acked Read 6 Done Sent Done false
22984 7518 Acked Write 9 Done Sent Done false
22985 7519 Acked Write 9 Done Sent Done false
22986 7520 Acked Read 4 Done Sent Done false
22987 7521 Acked Write 1 Done Sent Done false
22988 7522 Acked Read 1 Done Sent Done false
22989 7523 Acked Write 8 Done Sent Done false
22990 7524 Acked Read 3 Done Sent Done false
22991 7525 Acked Read 8 Done Sent Done false
22992 7526 Acked Read 10 Done Sent Done false
22993 7527 Acked Read 1 Done Sent Done false
22994 7528 Acked Write 5 Done Sent Done false
22995 7529 Acked Write 5 Done Sent Done false
22996 7530 Acked Write 8 Done Sent Done false
22997 7531 Acked Read 4 Done Sent Done false
22998 7532 Acked Write 3 Done Sent Done false
22999 7533 Acked Write 7 Done Sent Done false
23000 7534 Acked Read 8 Done Sent Done false
23001 7535 Acked Write 3 Done Sent Done false
23002 7536 Acked Read 3 Done Sent Done false
23003 7537 Acked Flush 0 Done Sent Done false
23004 7538 Acked Write 9 Done Sent Done false
23005 7539 Acked Read 5 Done Sent Done false
23006 7540 Acked Flush 0 Done Sent Done false
23007 7541 Acked Read 8 Done Sent Done false
23008 7542 Acked Read 1 Done Sent Done false
23009 7543 Acked Read 4 Done Sent Done false
23010 7544 Acked Read 3 Done Sent Done false
23011 7545 Acked Flush 0 Done Sent Done false
23012 7546 Acked Read 3 Done Sent Done false
23013 7547 Acked Write 8 Done Sent Done false
23014 7548 Acked Read 4 Done Sent Done false
23015 7549 Acked Read 3 Done Sent Done false
23016 7550 Acked Write 3 Done Sent Done false
23017 7551 Acked Read 4 Done Sent Done false
23018 7552 Acked Write 8 Done Sent Done false
23019 7553 Acked Flush 0 Done Sent Done false
23020 7554 Acked Flush 0 Done Sent Done false
23021 7555 Acked Read 7 Done Sent Done false
23022 7556 Acked Write 4 Done Sent Done false
23023 7557 Acked Write 1 Done Sent Done false
23024 7558 Acked Read 10 Done Sent Done false
23025 7559 Acked Read 10 Done Sent Done false
23026 7560 Acked Read 6 Done Sent Done false
23027 7561 Acked Read 7 Done Sent Done false
23028 7562 Acked Write 4 Done Sent Done false
23029 7563 Acked Write 8 Done Sent Done false
23030 7564 Acked Read 8 Done Sent Done false
23031 7565 Acked Write 9 Done Sent Done false
23032 7566 Acked Write 1 Done Sent Done false
23033 7567 Acked Read 7 Done Sent Done false
23034 7568 Acked Write 3 Done Sent Done false
23035 7569 Acked Read 4 Done Sent Done false
23036 7570 Acked Read 3 Done Sent Done false
23037 7571 Acked Write 10 Done Sent Done false
23038 7572 Acked Write 7 Done Sent Done false
23039 7573 Acked Read 2 Done Sent Done false
23040 7574 Acked Read 3 Done Sent Done false
23041 7575 Acked Read 9 Done Sent Done false
23042 7576 Acked Read 6 Done Sent Done false
23043 7577 Acked Write 3 Done Sent Done false
23044 7578 Acked Write 8 Done Sent Done false
23045 7579 Acked Write 10 Done Sent Done false
23046 7580 Acked Write 8 Done Sent Done false
23047 7581 Acked Read 1 Done Sent Done false
23048 7582 Acked Flush 0 Done Sent Done false
23049 7583 Acked Read 6 Done Sent Done false
23050 7584 Acked Read 9 Done Sent Done false
23051 7585 Acked Read 6 Done Sent Done false
23052 7586 Acked Read 2 Done Sent Done false
23053 7587 Acked Read 9 Done Sent Done false
23054 7588 Acked Write 1 Done Sent Done false
23055 7589 Acked Write 7 Done Sent Done false
23056 7590 Acked Read 6 Done Sent Done false
23057 7591 Acked Read 9 Done Sent Done false
23058 7592 Acked Read 8 Done Sent Done false
23059 7593 Acked Write 5 Done Sent Done false
23060 7594 Acked Flush 0 Done Sent Done false
23061 7595 Acked Read 1 Done Sent Done false
23062 7596 Acked Read 6 Done Sent Done false
23063 7597 Acked Read 10 Done Sent Done false
23064 7598 Acked Read 8 Done Sent Done false
23065 7599 Acked Read 3 Done Sent Done false
23066 7600 Acked Read 2 Done Sent Done false
23067 7601 Acked Read 8 Done Sent Done false
23068 7602 Acked Write 5 Done Sent Done false
23069 7603 Acked Write 9 Done Sent Done false
23070 7604 Acked Read 8 Done Sent Done false
23071 7605 Acked Write 3 Done Sent Done false
23072 7606 Acked Flush 0 Done Sent Done false
23073 7607 Acked Read 8 Done Sent Done false
23074 7608 Acked Flush 0 Done Sent Done false
23075 7609 Acked Read 6 Done Sent Done false
23076 7610 Acked Write 3 Done Sent Done false
23077 7611 Acked Read 7 Done Sent Done false
23078 7612 Acked Read 3 Done Sent Done false
23079 7613 Acked Write 10 Done Sent Done false
23080 7614 Acked Flush 0 Done Sent Done false
23081 7615 Acked Read 1 Done Sent Done false
23082 7616 Acked Read 3 Done Sent Done false
23083 7617 Acked Read 3 Done Sent Done false
23084 7618 Acked Read 6 Done Sent Done false
23085 7619 Acked Read 10 Done Sent Done false
23086 7620 Acked Flush 0 Done Sent Done false
23087 7621 Acked Write 7 Done Sent Done false
23088 7622 Acked Read 10 Done Sent Done false
23089 7623 Acked Read 6 Done Sent Done false
23090 7624 Acked Read 2 Done Sent Done false
23091 7625 Acked Read 10 Done Sent Done false
23092 7626 Acked Read 3 Done Sent Done false
23093 7627 Acked Write 5 Done Sent Done false
23094 7628 Acked Write 10 Done Sent Done false
23095 7629 Acked Read 6 Done Sent Done false
23096 7630 Acked Read 10 Done Sent Done false
23097 7631 Acked Read 2 Done Sent Done false
23098 7632 Acked Write 9 Done Sent Done false
23099 7633 Acked Write 5 Done Sent Done false
23100 7634 Acked Read 5 Done Sent Done false
23101 7635 Acked Read 2 Done Sent Done false
23102 7636 Acked Read 2 Done Sent Done false
23103 7637 Acked Write 7 Done Sent Done false
23104 7638 Acked Flush 0 Done Sent Done false
23105 7639 Acked Write 8 Done Sent Done false
23106 STATES DS:0 DS:1 DS:2 TOTAL
23107 Sent 0 301 0 301
23108 Done 301 0 301 602
23109 Skipped 0 0 0 0
23110 Error 0 0 0 0
23111 Last Flush: Some(JobId(7638)) Some(JobId(7338)) Some(JobId(7638))
23112 Downstairs last five completed: 7338 7337 7336 7335 7334
23113 Upstairs last five completed: 7639 7638 7637 7636 7635
23114 CLIENT: Up:0 ds:301 act:2
23115 [8820] got start action
23116 [8820] I am going to restart
23117 Starting downstairs at port 8820
23118 Make output file at "/tmp/dsc/downstairs-8820.txt"
23119 [8820][1] reports Starting
23120 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1222)
23121 [8820][1] reports Running
23122 ----------------------------------------------------------------
23123 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
23124 States: Active Offline Active
23125 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23126 7339 Acked Read 8 Done Sent Done false
23127 7340 Acked Read 8 Done Sent Done false
23128 7341 Acked Read 2 Done Sent Done false
23129 7342 Acked Write 6 Done Sent Done false
23130 7343 Acked Flush 0 Done Sent Done false
23131 7344 Acked Read 6 Done Sent Done false
23132 7345 Acked Read 8 Done Sent Done false
23133 7346 Acked Read 6 Done Sent Done false
23134 7347 Acked Read 5 Done Sent Done false
23135 7348 Acked Flush 0 Done Sent Done false
23136 7349 Acked Read 1 Done Sent Done false
23137 7350 Acked Write 1 Done Sent Done false
23138 7351 Acked Read 4 Done Sent Done false
23139 7352 Acked Write 7 Done Sent Done false
23140 7353 Acked Write 6 Done Sent Done false
23141 7354 Acked Read 5 Done Sent Done false
23142 7355 Acked Read 8 Done Sent Done false
23143 7356 Acked Read 6 Done Sent Done false
23144 7357 Acked Read 6 Done Sent Done false
23145 7358 Acked Write 6 Done Sent Done false
23146 7359 Acked Write 5 Done Sent Done false
23147 7360 Acked Read 5 Done Sent Done false
23148 7361 Acked Flush 0 Done Sent Done false
23149 7362 Acked Read 2 Done Sent Done false
23150 7363 Acked Write 4 Done Sent Done false
23151 7364 Acked Read 6 Done Sent Done false
23152 7365 Acked Flush 0 Done Sent Done false
23153 7366 Acked Read 3 Done Sent Done false
23154 7367 Acked Read 1 Done Sent Done false
23155 7368 Acked Write 2 Done Sent Done false
23156 7369 Acked Read 5 Done Sent Done false
23157 7370 Acked Read 10 Done Sent Done false
23158 7371 Acked Write 10 Done Sent Done false
23159 7372 Acked Write 5 Done Sent Done false
23160 7373 Acked Read 6 Done Sent Done false
23161 7374 Acked Write 7 Done Sent Done false
23162 7375 Acked Read 4 Done Sent Done false
23163 7376 Acked Write 5 Done Sent Done false
23164 7377 Acked Write 10 Done Sent Done false
23165 7378 Acked Write 6 Done Sent Done false
23166 7379 Acked Write 5 Done Sent Done false
23167 7380 Acked Read 8 Done Sent Done false
23168 7381 Acked Read 9 Done Sent Done false
23169 7382 Acked Write 6 Done Sent Done false
23170 7383 Acked Read 10 Done Sent Done false
23171 7384 Acked Read 7 Done Sent Done false
23172 7385 Acked Write 5 Done Sent Done false
23173 7386 Acked Read 10 Done Sent Done false
23174 7387 Acked Write 9 Done Sent Done false
23175 7388 Acked Read 4 Done Sent Done false
23176 7389 Acked Read 4 Done Sent Done false
23177 7390 Acked Read 5 Done Sent Done false
23178 7391 Acked Write 10 Done Sent Done false
23179 7392 Acked Write 9 Done Sent Done false
23180 7393 Acked Read 3 Done Sent Done false
23181 7394 Acked Read 1 Done Sent Done false
23182 7395 Acked Write 3 Done Sent Done false
23183 7396 Acked Write 4 Done Sent Done false
23184 7397 Acked Read 5 Done Sent Done false
23185 7398 Acked Flush 0 Done Sent Done false
23186 7399 Acked Read 7 Done Sent Done false
23187 7400 Acked Read 6 Done Sent Done false
23188 7401 Acked Write 5 Done Sent Done false
23189 7402 Acked Read 1 Done Sent Done false
23190 7403 Acked Write 6 Done Sent Done false
23191 7404 Acked Write 2 Done Sent Done false
23192 7405 Acked Flush 0 Done Sent Done false
23193 7406 Acked Write 10 Done Sent Done false
23194 7407 Acked Read 2 Done Sent Done false
23195 7408 Acked Write 2 Done Sent Done false
23196 7409 Acked Write 2 Done Sent Done false
23197 7410 Acked Read 4 Done Sent Done false
23198 7411 Acked Write 5 Done Sent Done false
23199 7412 Acked Write 8 Done Sent Done false
23200 7413 Acked Write 9 Done Sent Done false
23201 7414 Acked Flush 0 Done Sent Done false
23202 7415 Acked Read 8 Done Sent Done false
23203 7416 Acked Flush 0 Done Sent Done false
23204 7417 Acked Flush 0 Done Sent Done false
23205 7418 Acked Write 2 Done Sent Done false
23206 7419 Acked Read 10 Done Sent Done false
23207 7420 Acked Read 3 Done Sent Done false
23208 7421 Acked Flush 0 Done Sent Done false
23209 7422 Acked Write 3 Done Sent Done false
23210 7423 Acked Read 7 Done Sent Done false
23211 7424 Acked Read 8 Done Sent Done false
23212 7425 Acked Read 5 Done Sent Done false
23213 7426 Acked Read 3 Done Sent Done false
23214 7427 Acked Read 8 Done Sent Done false
23215 7428 Acked Write 3 Done Sent Done false
23216 7429 Acked Read 5 Done Sent Done false
23217 7430 Acked Read 8 Done Sent Done false
23218 7431 Acked Flush 0 Done Sent Done false
23219 7432 Acked Read 9 Done Sent Done false
23220 7433 Acked Write 4 Done Sent Done false
23221 7434 Acked Read 1 Done Sent Done false
23222 7435 Acked Read 4 Done Sent Done false
23223 7436 Acked Write 2 Done Sent Done false
23224 7437 Acked Write 5 Done Sent Done false
23225 7438 Acked Write 1 Done Sent Done false
23226 7439 Acked Read 5 Done Sent Done false
23227 7440 Acked Read 6 Done Sent Done false
23228 7441 Acked Read 4 Done Sent Done false
23229 7442 Acked Write 5 Done Sent Done false
23230 7443 Acked Write 2 Done Sent Done false
23231 7444 Acked Read 9 Done Sent Done false
23232 7445 Acked Read 2 Done Sent Done false
23233 7446 Acked Read 10 Done Sent Done false
23234 7447 Acked Write 7 Done Sent Done false
23235 7448 Acked Write 2 Done Sent Done false
23236 7449 Acked Flush 0 Done Sent Done false
23237 7450 Acked Read 8 Done Sent Done false
23238 7451 Acked Read 3 Done Sent Done false
23239 7452 Acked Read 8 Done Sent Done false
23240 7453 Acked Read 8 Done Sent Done false
23241 7454 Acked Read 9 Done Sent Done false
23242 7455 Acked Write 4 Done Sent Done false
23243 7456 Acked Read 2 Done Sent Done false
23244 7457 Acked Read 5 Done Sent Done false
23245 7458 Acked Write 3 Done Sent Done false
23246 7459 Acked Read 10 Done Sent Done false
23247 7460 Acked Flush 0 Done Sent Done false
23248 7461 Acked Read 7 Done Sent Done false
23249 7462 Acked Read 3 Done Sent Done false
23250 7463 Acked Flush 0 Done Sent Done false
23251 7464 Acked Read 8 Done Sent Done false
23252 7465 Acked Read 7 Done Sent Done false
23253 7466 Acked Write 10 Done Sent Done false
23254 7467 Acked Write 4 Done Sent Done false
23255 7468 Acked Read 7 Done Sent Done false
23256 7469 Acked Write 9 Done Sent Done false
23257 7470 Acked Read 10 Done Sent Done false
23258 7471 Acked Read 9 Done Sent Done false
23259 7472 Acked Write 3 Done Sent Done false
23260 7473 Acked Read 3 Done Sent Done false
23261 7474 Acked Read 3 Done Sent Done false
23262 7475 Acked Write 9 Done Sent Done false
23263 7476 Acked Read 5 Done Sent Done false
23264 7477 Acked Read 1 Done Sent Done false
23265 7478 Acked Write 4 Done Sent Done false
23266 7479 Acked Flush 0 Done Sent Done false
23267 7480 Acked Write 10 Done Sent Done false
23268 7481 Acked Write 10 Done Sent Done false
23269 7482 Acked Write 2 Done Sent Done false
23270 7483 Acked Read 3 Done Sent Done false
23271 7484 Acked Flush 0 Done Sent Done false
23272 7485 Acked Read 8 Done Sent Done false
23273 7486 Acked Flush 0 Done Sent Done false
23274 7487 Acked Write 1 Done Sent Done false
23275 7488 Acked Read 3 Done Sent Done false
23276 7489 Acked Read 7 Done Sent Done false
23277 7490 Acked Read 5 Done Sent Done false
23278 7491 Acked Write 6 Done Sent Done false
23279 7492 Acked Write 1 Done Sent Done false
23280 7493 Acked Write 10 Done Sent Done false
23281 7494 Acked Write 7 Done Sent Done false
23282 7495 Acked Write 1 Done Sent Done false
23283 7496 Acked Write 9 Done Sent Done false
23284 7497 Acked Write 10 Done Sent Done false
23285 7498 Acked Flush 0 Done Sent Done false
23286 7499 Acked Read 6 Done Sent Done false
23287 7500 Acked Read 8 Done Sent Done false
23288 7501 Acked Write 5 Done Sent Done false
23289 7502 Acked Write 1 Done Sent Done false
23290 7503 Acked Flush 0 Done Sent Done false
23291 7504 Acked Flush 0 Done Sent Done false
23292 7505 Acked Write 5 Done Sent Done false
23293 7506 Acked Write 2 Done Sent Done false
23294 7507 Acked Read 1 Done Sent Done false
23295 7508 Acked Read 2 Done Sent Done false
23296 7509 Acked Write 1 Done Sent Done false
23297 7510 Acked Write 4 Done Sent Done false
23298 7511 Acked Write 2 Done Sent Done false
23299 7512 Acked Read 9 Done Sent Done false
23300 7513 Acked Read 7 Done Sent Done false
23301 7514 Acked Read 9 Done Sent Done false
23302 7515 Acked Write 4 Done Sent Done false
23303 7516 Acked Write 6 Done Sent Done false
23304 7517 Acked Read 6 Done Sent Done false
23305 7518 Acked Write 9 Done Sent Done false
23306 7519 Acked Write 9 Done Sent Done false
23307 7520 Acked Read 4 Done Sent Done false
23308 7521 Acked Write 1 Done Sent Done false
23309 7522 Acked Read 1 Done Sent Done false
23310 7523 Acked Write 8 Done Sent Done false
23311 7524 Acked Read 3 Done Sent Done false
23312 7525 Acked Read 8 Done Sent Done false
23313 7526 Acked Read 10 Done Sent Done false
23314 7527 Acked Read 1 Done Sent Done false
23315 7528 Acked Write 5 Done Sent Done false
23316 7529 Acked Write 5 Done Sent Done false
23317 7530 Acked Write 8 Done Sent Done false
23318 7531 Acked Read 4 Done Sent Done false
23319 7532 Acked Write 3 Done Sent Done false
23320 7533 Acked Write 7 Done Sent Done false
23321 7534 Acked Read 8 Done Sent Done false
23322 7535 Acked Write 3 Done Sent Done false
23323 7536 Acked Read 3 Done Sent Done false
23324 7537 Acked Flush 0 Done Sent Done false
23325 7538 Acked Write 9 Done Sent Done false
23326 7539 Acked Read 5 Done Sent Done false
23327 7540 Acked Flush 0 Done Sent Done false
23328 7541 Acked Read 8 Done Sent Done false
23329 7542 Acked Read 1 Done Sent Done false
23330 7543 Acked Read 4 Done Sent Done false
23331 7544 Acked Read 3 Done Sent Done false
23332 7545 Acked Flush 0 Done Sent Done false
23333 7546 Acked Read 3 Done Sent Done false
23334 7547 Acked Write 8 Done Sent Done false
23335 7548 Acked Read 4 Done Sent Done false
23336 7549 Acked Read 3 Done Sent Done false
23337 7550 Acked Write 3 Done Sent Done false
23338 7551 Acked Read 4 Done Sent Done false
23339 7552 Acked Write 8 Done Sent Done false
23340 7553 Acked Flush 0 Done Sent Done false
23341 7554 Acked Flush 0 Done Sent Done false
23342 7555 Acked Read 7 Done Sent Done false
23343 7556 Acked Write 4 Done Sent Done false
23344 7557 Acked Write 1 Done Sent Done false
23345 7558 Acked Read 10 Done Sent Done false
23346 7559 Acked Read 10 Done Sent Done false
23347 7560 Acked Read 6 Done Sent Done false
23348 7561 Acked Read 7 Done Sent Done false
23349 7562 Acked Write 4 Done Sent Done false
23350 7563 Acked Write 8 Done Sent Done false
23351 7564 Acked Read 8 Done Sent Done false
23352 7565 Acked Write 9 Done Sent Done false
23353 7566 Acked Write 1 Done Sent Done false
23354 7567 Acked Read 7 Done Sent Done false
23355 7568 Acked Write 3 Done Sent Done false
23356 7569 Acked Read 4 Done Sent Done false
23357 7570 Acked Read 3 Done Sent Done false
23358 7571 Acked Write 10 Done Sent Done false
23359 7572 Acked Write 7 Done Sent Done false
23360 7573 Acked Read 2 Done Sent Done false
23361 7574 Acked Read 3 Done Sent Done false
23362 7575 Acked Read 9 Done Sent Done false
23363 7576 Acked Read 6 Done Sent Done false
23364 7577 Acked Write 3 Done Sent Done false
23365 7578 Acked Write 8 Done Sent Done false
23366 7579 Acked Write 10 Done Sent Done false
23367 7580 Acked Write 8 Done Sent Done false
23368 7581 Acked Read 1 Done Sent Done false
23369 7582 Acked Flush 0 Done Sent Done false
23370 7583 Acked Read 6 Done Sent Done false
23371 7584 Acked Read 9 Done Sent Done false
23372 7585 Acked Read 6 Done Sent Done false
23373 7586 Acked Read 2 Done Sent Done false
23374 7587 Acked Read 9 Done Sent Done false
23375 7588 Acked Write 1 Done Sent Done false
23376 7589 Acked Write 7 Done Sent Done false
23377 7590 Acked Read 6 Done Sent Done false
23378 7591 Acked Read 9 Done Sent Done false
23379 7592 Acked Read 8 Done Sent Done false
23380 7593 Acked Write 5 Done Sent Done false
23381 7594 Acked Flush 0 Done Sent Done false
23382 7595 Acked Read 1 Done Sent Done false
23383 7596 Acked Read 6 Done Sent Done false
23384 7597 Acked Read 10 Done Sent Done false
23385 7598 Acked Read 8 Done Sent Done false
23386 7599 Acked Read 3 Done Sent Done false
23387 7600 Acked Read 2 Done Sent Done false
23388 7601 Acked Read 8 Done Sent Done false
23389 7602 Acked Write 5 Done Sent Done false
23390 7603 Acked Write 9 Done Sent Done false
23391 7604 Acked Read 8 Done Sent Done false
23392 7605 Acked Write 3 Done Sent Done false
23393 7606 Acked Flush 0 Done Sent Done false
23394 7607 Acked Read 8 Done Sent Done false
23395 7608 Acked Flush 0 Done Sent Done false
23396 7609 Acked Read 6 Done Sent Done false
23397 7610 Acked Write 3 Done Sent Done false
23398 7611 Acked Read 7 Done Sent Done false
23399 7612 Acked Read 3 Done Sent Done false
23400 7613 Acked Write 10 Done Sent Done false
23401 7614 Acked Flush 0 Done Sent Done false
23402 7615 Acked Read 1 Done Sent Done false
23403 7616 Acked Read 3 Done Sent Done false
23404 7617 Acked Read 3 Done Sent Done false
23405 7618 Acked Read 6 Done Sent Done false
23406 7619 Acked Read 10 Done Sent Done false
23407 7620 Acked Flush 0 Done Sent Done false
23408 7621 Acked Write 7 Done Sent Done false
23409 7622 Acked Read 10 Done Sent Done false
23410 7623 Acked Read 6 Done Sent Done false
23411 7624 Acked Read 2 Done Sent Done false
23412 7625 Acked Read 10 Done Sent Done false
23413 7626 Acked Read 3 Done Sent Done false
23414 7627 Acked Write 5 Done Sent Done false
23415 7628 Acked Write 10 Done Sent Done false
23416 7629 Acked Read 6 Done Sent Done false
23417 7630 Acked Read 10 Done Sent Done false
23418 7631 Acked Read 2 Done Sent Done false
23419 7632 Acked Write 9 Done Sent Done false
23420 7633 Acked Write 5 Done Sent Done false
23421 7634 Acked Read 5 Done Sent Done false
23422 7635 Acked Read 2 Done Sent Done false
23423 7636 Acked Read 2 Done Sent Done false
23424 7637 Acked Write 7 Done Sent Done false
23425 7638 Acked Flush 0 Done Sent Done false
23426 7639 Acked Write 8 Done Sent Done false
23427 7640 Acked Flush 0 Done Sent Done false
23428 STATES DS:0 DS:1 DS:2 TOTAL
23429 Sent 0 302 0 302
23430 Done 302 0 302 604
23431 Skipped 0 0 0 0
23432 Error 0 0 0 0
23433 Last Flush: Some(JobId(7640)) Some(JobId(7338)) Some(JobId(7640))
23434 Downstairs last five completed: 7338 7337 7336 7335 7334
23435 Upstairs last five completed: 7640 7639 7638 7637 7636
23436 CLIENT: Up:0 ds:302 act:2
234372025-01-23T18:52:48.196ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
234382025-01-23T18:52:48.196ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:45600) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
234392025-01-23T18:52:48.197ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
234402025-01-23T18:52:48.198ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
234412025-01-23T18:52:48.198ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
234422025-01-23T18:52:48.198ZINFOcrucible: send last flush ID to this DS: Some(JobId(7338)) = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
234432025-01-23T18:52:48.198ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
234442025-01-23T18:52:48.198ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(7338)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
234452025-01-23T18:52:48.198ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(7338)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
23446 ----------------------------------------------------------------
23447 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23448 Downstairs last five completed: 7640 7639 7638 7637 7636
23449 Upstairs last five completed: 7640 7639 7638 7637 7636
23450 CLIENT: Up:0 ds:0 act:3
23451 Replay: All jobs finished
23452 Main task has work to do, go find it
23453 got dsc Stop(2)
23454 stop 2
23455 [8830] Got stop action so:false kr:false
23456 Jan 23 18:52:50.449 INFO request completed, latency_us: 337, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 59f2c252-69a1-4fbe-9958-14e4d492e9d8, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
234572025-01-23T18:52:50.450ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
234582025-01-23T18:52:50.450ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
23459 Jan 23 18:52:50.450 INFO request completed, {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:52:50.450239782Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
23460 latency_us: 136, response_code: 200, uri: /state/cid/2, method: GET, req_id: 13a673ef-75e7-4eb1-ae91-84a5b8fd0f2d, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
234612025-01-23T18:52:50.450ZINFOcrucible: client task is exiting = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
23462 [8830] Exited with: ExitStatus(unix_wait_status(9))
23463 [8830][2] reports Exit
23464 Jan 23 18:52:54.452 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1b478816-588b-4004-bfc5-eaa9425d5b96, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
23465 001/300 Write block 4060 len 24576 data: 5 5 3 4 5 6
23466 002/300 Write block 2622 len 32768 data: 7 6 7 8 8 6 8 8
23467 003/300 Write block 1236 len 32768 data: 5 5 6 6 7 5 6 6
23468 004/300 Read block 4359 len 24576
23469 005/300 Write block 3098 len 24576 data: 4 5 6 6 6 7
23470 006/300 Write block 1145 len 28672 data: 9 9 8 7 7 7 7
23471 007/300 Read block 2315 len 12288
23472 008/300 Read block 4394 len 32768
23473 009/300 Read block 4675 len 32768
23474 010/300 Read block 4489 len 36864
23475 011/300 Read block 3845 len 4096
23476 012/300 Read block 967 len 36864
23477 013/300 Read block 639 len 16384
23478 014/300 Read block 947 len 24576
23479 015/300 Flush
23480 016/300 Read block 1517 len 12288
23481 017/300 Read block 79 len 8192
23482 018/300 Read block 839 len 24576
23483 019/300 Write block 2384 len 36864 data: 5 5 4 2 3 4 5 6 6
23484 020/300 Write block 3316 len 36864 data: 4 7 5 6 5 4 5 6 5
23485 021/300 Write block 2228 len 20480 data: 3 3 4 4 5
23486 022/300 Read block 2221 len 20480
23487 023/300 Write block 4944 len 16384 data: 6 4 2 3
23488 024/300 Read block 3425 len 8192
23489 025/300 Write block 3501 len 40960 data: 5 3 3 5 6 7 8 7 8 8
23490 026/300 Write block 3245 len 36864 data: 7 6 7 7 7 7 8 8 8
23491 027/300 Read block 4276 len 4096
23492 028/300 Write block 4636 len 32768 data: 6 7 6 5 5 3 4 7
23493 029/300 Write block 848 len 4096 data: 2
23494 030/300 Write block 4800 len 4096 data: 3
23495 031/300 Flush
23496 032/300 Read block 4451 len 8192
23497 033/300 Read block 446 len 16384
23498 034/300 Write block 2227 len 32768 data: 2 4 4 5 5 6 6 5
23499 035/300 Write block 2468 len 8192 data: 4 3
23500 036/300 Flush
23501 037/300 Flush
23502 038/300 Write block 3706 len 24576 data: 3 4 5 5 4 6
23503 039/300 Write block 947 len 28672 data: 4 4 3 3 3 3 5
23504 040/300 Read block 1494 len 32768
23505 041/300 Flush
23506 042/300 Read block 4267 len 32768
23507 043/300 Flush
23508 044/300 Read block 2922 len 8192
23509 045/300 Flush
23510 046/300 Read block 1181 len 12288
23511 047/300 Read block 2042 len 32768
23512 048/300 Read block 1390 len 36864
23513 049/300 Read block 4782 len 8192
23514 050/300 Read block 254 len 40960
23515 051/300 Write block 1259 len 36864 data: 4 5 5 6 6 6 6 4 4
23516 052/300 Write block 1423 len 28672 data: 9 8 7 7 6 4 4
23517 053/300 Write block 87 len 36864 data: 3 4 5 7 7 8 7 8 8
23518 054/300 Read block 2239 len 24576
23519 055/300 Read block 2808 len 32768
23520 056/300 Write block 2970 len 4096 data: 5
23521 057/300 Read block 942 len 40960
23522 058/300 Write block 952 len 4096 data: 4
23523 059/300 Read block 2654 len 36864
23524 060/300 Read block 2442 len 8192
23525 061/300 Read block 4624 len 24576
23526 062/300 Read block 3906 len 28672
23527 063/300 Write block 1743 len 36864 data: 2 2 3 3 2 2 2 3 3
23528 064/300 Write block 4501 len 24576 data: 6 7 7 8 7 7
23529 065/300 Read block 31 len 40960
23530 066/300 Write block 1380 len 4096 data: 4
23531 067/300 Write block 4313 len 4096 data: 5
23532 068/300 Read block 2542 len 4096
23533 069/300 Read block 3420 len 36864
23534 070/300 Read block 4084 len 36864
23535 071/300 Write block 3880 len 24576 data: 9 8 7 7 8 11
23536 072/300 Read block 114 len 8192
23537 073/300 Write block 4946 len 4096 data: 3
23538 074/300 Read block 2499 len 24576
23539 075/300 Write block 2883 len 12288 data: 3 4 6
23540 076/300 Read block 3832 len 16384
23541 077/300 Read block 926 len 32768
23542 078/300 Read block 4083 len 24576
23543 079/300 Read block 1491 len 28672
23544 080/300 Write block 1521 len 16384 data: 4 3 4 4
23545 081/300 Write block 2202 len 28672 data: 6 6 5 5 6 6 6
23546 082/300 Flush
23547 083/300 Write block 242 len 28672 data: 4 3 2 4 4 7 7
23548 084/300 Read block 1516 len 24576
23549 085/300 Write block 1931 len 36864 data: 7 8 7 7 5 6 5 5 6
23550 086/300 Write block 4102 len 24576 data: 3 3 3 3 3 3
23551 087/300 Read block 398 len 4096
23552 088/300 Write block 551 len 4096 data: 6
23553 089/300 Write block 4575 len 12288 data: 5 4 6
23554 090/300 Write block 1379 len 20480 data: 4 5 3 3 4
23555 091/300 Read block 469 len 16384
23556 092/300 Read block 3598 len 20480
23557 093/300 Write block 580 len 8192 data: 4 4
23558 094/300 Read block 1316 len 40960
23559 095/300 Read block 3688 len 16384
23560 096/300 Read block 2339 len 4096
23561 097/300 Read block 3028 len 28672
23562 098/300 Write block 1079 len 24576 data: 5 5 6 5 5 5
23563 099/300 Read block 4211 len 8192
23564 100/300 Write block 1503 len 12288 data: 4 4 6
23565 101/300 Read block 1846 len 32768
23566 102/300 Write block 2286 len 20480 data: 4 5 5 4 5
23567 103/300 Write block 1138 len 28672 data: 7 9 8 8 7 6 7
23568 104/300 Write block 4874 len 28672 data: 7 6 5 3 2 3 3
23569 105/300 Read block 4433 len 4096
23570 106/300 Flush
23571 107/300 Read block 4601 len 4096
23572 108/300 Write block 2908 len 24576 data: 5 6 6 5 6 5
23573 109/300 Write block 4650 len 28672 data: 5 5 4 4 3 3 3
23574 110/300 Flush
23575 111/300 Read block 178 len 8192
23576 112/300 Write block 1287 len 20480 data: 8 8 7 7 7
23577 113/300 Flush
23578 114/300 Read block 2179 len 4096
23579 115/300 Read block 971 len 28672
23580 116/300 Flush
23581 117/300 Flush
23582 118/300 Read block 4144 len 4096
23583 119/300 Read block 856 len 36864
23584 120/300 Flush
23585 121/300 Read block 2690 len 16384
23586 122/300 Write block 1955 len 28672 data: 6 5 5 5 5 3 4
23587 123/300 Write block 1667 len 16384 data: 5 5 5 4
23588 124/300 Read block 1389 len 12288
23589 125/300 Read block 3707 len 12288
23590 126/300 Read block 2511 len 12288
23591 127/300 Read block 2502 len 4096
23592 128/300 Read block 1514 len 8192
23593 129/300 Write block 1201 len 40960 data: 5 5 4 4 3 3 2 2 2 2
23594 130/300 Write block 1093 len 24576 data: 5 5 7 6 5 6
23595 131/300 Read block 3183 len 4096
23596 132/300 Read block 975 len 4096
23597 133/300 Read block 2304 len 4096
23598 134/300 Write block 3779 len 16384 data: 7 5 5 5
23599 135/300 Write block 1635 len 24576 data: 6 5 6 6 6 5
23600 136/300 Write block 1889 len 12288 data: 5 4 4
23601 137/300 Read block 2110 len 24576
23602 138/300 Write block 3226 len 32768 data: 5 4 4 5 6 7 8 8
23603 139/300 Read block 1492 len 40960
23604 140/300 Read block 4005 len 40960
23605 141/300 Write block 998 len 28672 data: 5 6 6 6 8 8 8
23606 142/300 Write block 4204 len 20480 data: 5 4 3 4 4
23607 143/300 Read block 1473 len 8192
23608 144/300 Write block 3441 len 20480 data: 5 7 7 7 5
23609 145/300 Flush
23610 146/300 Read block 3160 len 20480
23611 147/300 Read block 1781 len 40960
23612 148/300 Write block 1111 len 40960 data: 4 4 4 5 5 5 5 3 3 2
23613 149/300 Read block 492 len 24576
23614 150/300 Read block 4653 len 24576
23615 151/300 Write block 4477 len 16384 data: 6 5 6 6
23616 152/300 Read block 4388 len 32768
23617 153/300 Read block 4507 len 24576
23618 154/300 Write block 4782 len 28672 data: 5 6 6 6 5 5 7
23619 155/300 Write block 3586 len 32768 data: 4 4 4 5 5 5 5 5
23620 156/300 Write block 279 len 36864 data: 6 5 4 3 4 3 5 5 3
23621 157/300 Write block 2190 len 8192 data: 4 4
23622 158/300 Write block 2052 len 28672 data: 3 7 4 5 7 5 5
23623 159/300 Flush
23624 160/300 Write block 3185 len 8192 data: 5 5
23625 161/300 Write block 0 len 28672 data: 2 2 2 2 2 3 5
23626 162/300 Read block 4613 len 32768
23627 163/300 Write block 259 len 20480 data: 4 3 3 4 4
23628 164/300 Write block 1950 len 20480 data: 5 5 5 4 5
23629 165/300 Write block 841 len 32768 data: 4 5 5 4 4 4 4 3
23630 166/300 Flush
23631 167/300 Write block 2346 len 36864 data: 3 3 4 4 3 3 4 3 3
23632 168/300 Write block 21 len 4096 data: 4
23633 169/300 Read block 4761 len 12288
23634 170/300 Write block 1984 len 12288 data: 2 4 4
23635 171/300 Flush
23636 172/300 Write block 744 len 40960 data: 6 5 4 4 4 3 3 2 3 3
23637 173/300 Write block 1271 len 40960 data: 2 2 2 2 3 3 3 4 4 3
23638 174/300 Read block 1037 len 4096
23639 175/300 Read block 4118 len 16384
23640 176/300 Read block 3490 len 36864
23641 177/300 Read block 4025 len 12288
23642 178/300 Read block 344 len 16384
23643 179/300 Read block 3067 len 36864
23644 180/300 Write block 1096 len 32768 data: 7 6 7 7 7 6 7 7
23645 181/300 Read block 3334 len 40960
23646 182/300 Read block 3929 len 32768
23647 183/300 Read block 1869 len 32768
23648 184/300 Read block 3380 len 12288
23649 185/300 Write block 4632 len 4096 data: 5
23650 186/300 Read block 3072 len 16384
23651 187/300 Write block 4878 len 28672 data: 3 4 4 3 5 5 6
23652 188/300 Flush
23653 189/300 Write block 4382 len 24576 data: 3 5 3 3 4 4
23654 190/300 Read block 4015 len 36864
23655 191/300 Read block 3861 len 32768
23656 192/300 Read block 4750 len 24576
23657 193/300 Read block 2777 len 28672
23658 194/300 Write block 2833 len 12288 data: 4 4 5
23659 195/300 Read block 1196 len 4096
23660 196/300 Read block 693 len 36864
23661 197/300 Read block 2744 len 8192
23662 198/300 Read block 260 len 8192
23663 199/300 Write block 1098 len 20480 data: 8 8 8 7 8
23664 200/300 Read block 1377 len 28672
23665 201/300 Read block 1066 len 32768
23666 202/300 Read block 4737 len 4096
23667 203/300 Write block 4443 len 40960 data: 3 4 5 6 8 9 10 9 9 8
23668 204/300 Read block 3614 len 4096
23669 205/300 Write block 2313 len 36864 data: 8 8 9 6 6 5 5 6 5
23670 206/300 Write block 1474 len 20480 data: 5 5 6 5 4
23671 207/300 Read block 4063 len 12288
23672 208/300 Read block 548 len 36864
23673 209/300 Read block 3377 len 40960
23674 210/300 Read block 2578 len 32768
23675 211/300 Write block 84 len 24576 data: 3 4 4 4 5 6
23676 212/300 Read block 786 len 12288
23677 213/300 Flush
23678 214/300 Read block 4621 len 12288
23679 215/300 Read block 1633 len 20480
23680 216/300 Read block 245 len 4096
23681 217/300 Write block 4318 len 20480 data: 4 5 5 6 7
23682 218/300 Write block 1282 len 36864 data: 4 5 7 7 9 9 9 8 8
23683 219/300 Write block 1527 len 28672 data: 6 6 5 4 4 4 5
23684 220/300 Write block 4401 len 12288 data: 3 3 4
23685 221/300 Write block 2357 len 16384 data: 2 2 3 4
23686 222/300 Write block 796 len 40960 data: 5 5 5 3 3 3 2 3 3 3
23687 223/300 Read block 1074 len 12288
23688 224/300 Read block 479 len 12288
23689 225/300 Read block 217 len 40960
23690 226/300 Read block 360 len 4096
23691 227/300 Read block 4701 len 24576
23692 228/300 Flush
23693 229/300 Write block 137 len 28672 data: 6 8 6 8 7 5 5
23694 230/300 Read block 2158 len 16384
23695 231/300 Write block 2343 len 16384 data: 5 5 3 4
23696 232/300 Write block 3684 len 4096 data: 5
23697 233/300 Read block 3102 len 20480
23698 234/300 Read block 3422 len 24576
23699 235/300 Read block 1111 len 20480
23700 236/300 Read block 3605 len 16384
23701 237/300 Write block 2205 len 28672 data: 6 7 7 7 5 6 6
23702 238/300 Read block 2828 len 16384
23703 239/300 Read block 4948 len 36864
23704 240/300 Write block 1048 len 32768 data: 8 8 6 6 5 3 4 4
23705 241/300 Write block 2073 len 24576 data: 7 6 6 6 8 6
23706 242/300 Read block 1031 len 20480
23707 243/300 Read block 3536 len 12288
23708 244/300 Read block 1461 len 40960
23709 245/300 Read block 4370 len 32768
23710 246/300 Write block 135 len 40960 data: 7 7 7 9 7 9 8 6 6 5
23711 247/300 Read block 3215 len 16384
23712 248/300 Write block 4342 len 12288 data: 7 6 10
23713 249/300 Flush
23714 250/300 Read block 686 len 12288
23715 251/300 Write block 4330 len 28672 data: 7 5 4 3 4 4 6
23716 252/300 Read block 2688 len 8192
23717 253/300 Flush
23718 254/300 Write block 3908 len 40960 data: 2 3 3 4 4 5 5 5 5 6
23719 255/300 Write block 233 len 24576 data: 2 2 2 4 5 4
23720 256/300 Read block 3267 len 28672
23721 257/300 Read block 4291 len 28672
23722 258/300 Write block 4683 len 20480 data: 6 4 4 5 5
23723 259/300 Read block 2221 len 24576
23724 260/300 Read block 4002 len 20480
23725 261/300 Read block 2990 len 36864
23726 262/300 Read block 1077 len 40960
23727 263/300 Read block 2484 len 4096
23728 264/300 Flush
23729 265/300 Read block 740 len 8192
23730 266/300 Read block 995 len 4096
23731 267/300 Read block 3234 len 4096
23732 268/300 Write block 3486 len 12288 data: 6 4 3
23733 269/300 Write block 2225 len 28672 data: 4 2 3 5 5 6 6
23734 270/300 Read block 2582 len 24576
23735 271/300 Read block 352 len 20480
23736 272/300 Read block 607 len 36864
23737 273/300 Read block 2168 len 24576
23738 274/300 Read block 1892 len 20480
23739 275/300 Flush
23740 276/300 Flush
23741 277/300 Read block 270 len 4096
23742 278/300 Read block 3380 len 8192
23743 279/300 Flush
23744 280/300 Read block 1978 len 8192
23745 281/300 Write block 4614 len 8192 data: 2 2
23746 282/300 Read block 2679 len 8192
23747 283/300 Read block 775 len 8192
23748 284/300 Write block 3171 len 16384 data: 3 4 3 3
23749 285/300 Read block 1580 len 40960
23750 286/300 Flush
23751 287/300 Write block 3499 len 28672 data: 7 6 6 4 4 6 7
23752 288/300 Write block 3559 len 12288 data: 6 6 4
23753 289/300 Write block 4343 len 4096 data: 7
23754 290/300 Read block 1538 len 4096
23755 291/300 Write block 12 len 8192 data: 5 3
23756 292/300 Write block 4907 len 20480 data: 4 4 4 4 7
23757 293/300 Write block 881 len 36864 data: 4 6 6 6 5 5 6 4 3
23758 294/300 Read block 3594 len 16384
23759 295/300 Read block 815 len 36864
23760 296/300 Read block 3072 len 4096
23761 297/300 Read block 3322 len 16384
23762 298/300 Read block 1475 len 20480
23763 299/300 Write block 939 len 36864 data: 8 7 7 7 6 7 8 6 5
23764 300/300 Read block 4887 len 28672
23765 301/300 Write block 1755 len 36864 data: 4 3 3 5 5 4 3 4 4
23766 Main task has work to do, go find it
23767 got dsc Start(2)
23768 start 2
23769 Jan 23 18:52:54.688 INFO request completed, latency_us: 197, response_code: 204, uri: /start/cid/2, method: POST, req_id: b9a58e97-f696-4794-b318-30fba7227b35, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
23770 Replay: started 2, returned:Ok(())
23771 ----------------------------------------------------------------
23772 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23773 States: Active Active Offline
23774 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23775 7641 Acked Write 6 Done Done Sent false
23776 7642 Acked Write 8 Done Done Sent false
23777 7643 Acked Write 8 Done Done Sent false
23778 7644 Acked Read 6 Done Done Sent false
23779 7645 Acked Write 6 Done Done Sent false
23780 7646 Acked Write 7 Done Done Sent false
23781 7647 Acked Read 3 Done Done Sent false
23782 7648 Acked Read 8 Done Done Sent false
23783 7649 Acked Read 8 Done Done Sent false
23784 7650 Acked Read 9 Done Done Sent false
23785 7651 Acked Read 1 Done Done Sent false
23786 7652 Acked Read 9 Done Done Sent false
23787 7653 Acked Read 4 Done Done Sent false
23788 7654 Acked Read 6 Done Done Sent false
23789 7655 Acked Flush 0 Done Done Sent false
23790 7656 Acked Read 3 Done Done Sent false
23791 7657 Acked Read 2 Done Done Sent false
23792 7658 Acked Read 6 Done Done Sent false
23793 7659 Acked Write 9 Done Done Sent false
23794 7660 Acked Write 9 Done Done Sent false
23795 7661 Acked Write 5 Done Done Sent false
23796 7662 Acked Read 5 Done Done Sent false
23797 7663 Acked Write 4 Done Done Sent false
23798 7664 Acked Read 2 Done Done Sent false
23799 7665 Acked Write 10 Done Done Sent false
23800 7666 Acked Write 9 Done Done Sent false
23801 7667 Acked Read 1 Done Done Sent false
23802 7668 Acked Write 8 Done Done Sent false
23803 7669 Acked Write 1 Done Done Sent false
23804 7670 Acked Write 1 Done Done Sent false
23805 7671 Acked Flush 0 Done Done Sent false
23806 7672 Acked Read 2 Done Done Sent false
23807 7673 Acked Read 4 Done Done Sent false
23808 7674 Acked Write 8 Done Done Sent false
23809 7675 Acked Write 2 Done Done Sent false
23810 7676 Acked Flush 0 Done Done Sent false
23811 7677 Acked Flush 0 Done Done Sent false
23812 7678 Acked Write 6 Done Done Sent false
23813 7679 Acked Write 7 Done Done Sent false
23814 7680 Acked Read 8 Done Done Sent false
23815 7681 Acked Flush 0 Done Done Sent false
23816 7682 Acked Read 8 Done Done Sent false
23817 7683 Acked Flush 0 Done Done Sent false
23818 7684 Acked Read 2 Done Done Sent false
23819 7685 Acked Flush 0 Done Done Sent false
23820 7686 Acked Read 3 Done Done Sent false
23821 7687 Acked Read 8 Done Done Sent false
23822 7688 Acked Read 9 Done Done Sent false
23823 7689 Acked Read 2 Done Done Sent false
23824 7690 Acked Read 10 Done Done Sent false
23825 7691 Acked Write 9 Done Done Sent false
23826 7692 Acked Write 7 Done Done Sent false
23827 7693 Acked Write 9 Done Done Sent false
23828 7694 Acked Read 6 Done Done Sent false
23829 7695 Acked Read 8 Done Done Sent false
23830 7696 Acked Write 1 Done Done Sent false
23831 7697 Acked Read 10 Done Done Sent false
23832 7698 Acked Write 1 Done Done Sent false
23833 7699 Acked Read 9 Done Done Sent false
23834 7700 Acked Read 2 Done Done Sent false
23835 7701 Acked Read 6 Done Done Sent false
23836 7702 Acked Read 7 Done Done Sent false
23837 7703 Acked Write 9 Done Done Sent false
23838 7704 Acked Write 6 Done Done Sent false
23839 7705 Acked Read 10 Done Done Sent false
23840 7706 Acked Write 1 Done Done Sent false
23841 7707 Acked Write 1 Done Done Sent false
23842 7708 Acked Read 1 Done Done Sent false
23843 7709 Acked Read 9 Done Done Sent false
23844 7710 Acked Read 9 Done Done Sent false
23845 7711 Acked Write 6 Done Done Sent false
23846 7712 Acked Read 2 Done Done Sent false
23847 7713 Acked Write 1 Done Done Sent false
23848 7714 Acked Read 6 Done Done Sent false
23849 7715 Acked Write 3 Done Done Sent false
23850 7716 Acked Read 4 Done Done Sent false
23851 7717 Acked Read 8 Done Done Sent false
23852 7718 Acked Read 6 Done Done Sent false
23853 7719 Acked Read 7 Done Done Sent false
23854 7720 Acked Write 4 Done Done Sent false
23855 7721 Acked Write 7 Done Done Sent false
23856 7722 Acked Flush 0 Done Done Sent false
23857 7723 Acked Write 7 Done Done Sent false
23858 7724 Acked Read 6 Done Done Sent false
23859 7725 Acked Write 9 Done Done Sent false
23860 7726 Acked Write 6 Done Done Sent false
23861 7727 Acked Read 1 Done Done Sent false
23862 7728 Acked Write 1 Done Done Sent false
23863 7729 Acked Write 3 Done Done Sent false
23864 7730 Acked Write 5 Done Done Sent false
23865 7731 Acked Read 4 Done Done Sent false
23866 7732 Acked Read 5 Done Done Sent false
23867 7733 Acked Write 2 Done Done Sent false
23868 7734 Acked Read 10 Done Done Sent false
23869 7735 Acked Read 4 Done Done Sent false
23870 7736 Acked Read 1 Done Done Sent false
23871 7737 Acked Read 7 Done Done Sent false
23872 7738 Acked Write 6 Done Done Sent false
23873 7739 Acked Read 2 Done Done Sent false
23874 7740 Acked Write 3 Done Done Sent false
23875 7741 Acked Read 8 Done Done Sent false
23876 7742 Acked Write 5 Done Done Sent false
23877 7743 Acked Write 7 Done Done Sent false
23878 7744 Acked Write 7 Done Done Sent false
23879 7745 Acked Read 1 Done Done Sent false
23880 7746 Acked Flush 0 Done Done Sent false
23881 7747 Acked Read 1 Done Done Sent false
23882 7748 Acked Write 6 Done Done Sent false
23883 7749 Acked Write 7 Done Done Sent false
23884 7750 Acked Flush 0 Done Done Sent false
23885 7751 Acked Read 2 Done Done Sent false
23886 7752 Acked Write 5 Done Done Sent false
23887 7753 Acked Flush 0 Done Done Sent false
23888 7754 Acked Read 1 Done Done Sent false
23889 7755 Acked Read 7 Done Done Sent false
23890 7756 Acked Flush 0 Done Done Sent false
23891 7757 Acked Flush 0 Done Done Sent false
23892 7758 Acked Read 1 Done Done Sent false
23893 7759 Acked Read 9 Done Done Sent false
23894 7760 Acked Flush 0 Done Done Sent false
23895 7761 Acked Read 4 Done Done Sent false
23896 7762 Acked Write 7 Done Done Sent false
23897 7763 Acked Write 4 Done Done Sent false
23898 7764 Acked Read 3 Done Done Sent false
23899 7765 Acked Read 3 Done Done Sent false
23900 7766 Acked Read 3 Done Done Sent false
23901 7767 Acked Read 1 Done Done Sent false
23902 7768 Acked Read 2 Done Done Sent false
23903 7769 Acked Write 10 Done Done Sent false
23904 7770 Acked Write 6 Done Done Sent false
23905 7771 Acked Read 1 Done Done Sent false
23906 7772 Acked Read 1 Done Done Sent false
23907 7773 Acked Read 1 Done Done Sent false
23908 7774 Acked Write 4 Done Done Sent false
23909 7775 Acked Write 6 Done Done Sent false
23910 7776 Acked Write 3 Done Done Sent false
23911 7777 Acked Read 6 Done Done Sent false
23912 7778 Acked Write 8 Done Done Sent false
23913 7779 Acked Read 10 Done Done Sent false
23914 7780 Acked Read 10 Done Done Sent false
23915 7781 Acked Write 7 Done Done Sent false
23916 7782 Acked Write 5 Done Done Sent false
23917 7783 Acked Read 2 Done Done Sent false
23918 7784 Acked Write 5 Done Done Sent false
23919 7785 Acked Flush 0 Done Done Sent false
23920 7786 Acked Read 5 Done Done Sent false
23921 7787 Acked Read 10 Done Done Sent false
23922 7788 Acked Write 10 Done Done Sent false
23923 7789 Acked Read 6 Done Done Sent false
23924 7790 Acked Read 6 Done Done Sent false
23925 7791 Acked Write 4 Done Done Sent false
23926 7792 Acked Read 8 Done Done Sent false
23927 7793 Acked Read 6 Done Done Sent false
23928 7794 Acked Write 7 Done Done Sent false
23929 7795 Acked Write 8 Done Done Sent false
23930 7796 Acked Write 9 Done Done Sent false
23931 7797 Acked Write 2 Done Done Sent false
23932 7798 Acked Write 7 Done Done Sent false
23933 7799 Acked Flush 0 Done Done Sent false
23934 7800 Acked Write 2 Done Done Sent false
23935 7801 Acked Write 7 Done Done Sent false
23936 7802 Acked Read 8 Done Done Sent false
23937 7803 Acked Write 5 Done Done Sent false
23938 7804 Acked Write 5 Done Done Sent false
23939 7805 Acked Write 8 Done Done Sent false
23940 7806 Acked Flush 0 Done Done Sent false
23941 7807 Acked Write 9 Done Done Sent false
23942 7808 Acked Write 1 Done Done Sent false
23943 7809 Acked Read 3 Done Done Sent false
23944 7810 Acked Write 3 Done Done Sent false
23945 7811 Acked Flush 0 Done Done Sent false
23946 7812 Acked Write 10 Done Done Sent false
23947 7813 Acked Write 10 Done Done Sent false
23948 7814 Acked Read 1 Done Done Sent false
23949 7815 Acked Read 4 Done Done Sent false
23950 7816 Acked Read 9 Done Done Sent false
23951 7817 Acked Read 3 Done Done Sent false
23952 7818 Acked Read 4 Done Done Sent false
23953 7819 Acked Read 9 Done Done Sent false
23954 7820 Acked Write 8 Done Done Sent false
23955 7821 Acked Read 10 Done Done Sent false
23956 7822 Acked Read 8 Done Done Sent false
23957 7823 Acked Read 8 Done Done Sent false
23958 7824 Acked Read 3 Done Done Sent false
23959 7825 Acked Write 1 Done Done Sent false
23960 7826 Acked Read 4 Done Done Sent false
23961 7827 Acked Write 7 Done Done Sent false
23962 7828 Acked Flush 0 Done Done Sent false
23963 7829 Acked Write 6 Done Done Sent false
23964 7830 Acked Read 9 Done Done Sent false
23965 7831 Acked Read 8 Done Done Sent false
23966 7832 Acked Read 6 Done Done Sent false
23967 7833 Acked Read 7 Done Done Sent false
23968 7834 Acked Write 3 Done Done Sent false
23969 7835 Acked Read 1 Done Done Sent false
23970 7836 Acked Read 9 Done Done Sent false
23971 7837 Acked Read 2 Done Done Sent false
23972 7838 Acked Read 2 Done Done Sent false
23973 7839 Acked Write 5 Done Done Sent false
23974 7840 Acked Read 7 Done Done Sent false
23975 7841 Acked Read 8 Done Done Sent false
23976 7842 Acked Read 1 Done Done Sent false
23977 7843 Acked Write 10 Done Done Sent false
23978 7844 Acked Read 1 Done Done Sent false
23979 7845 Acked Write 9 Done Done Sent false
23980 7846 Acked Write 5 Done Done Sent false
23981 7847 Acked Read 3 Done Done Sent false
23982 7848 Acked Read 9 Done Done Sent false
23983 7849 Acked Read 10 Done Done Sent false
23984 7850 Acked Read 8 Done Done Sent false
23985 7851 Acked Write 6 Done Done Sent false
23986 7852 Acked Read 3 Done Done Sent false
23987 7853 Acked Flush 0 Done Done Sent false
23988 7854 Acked Read 3 Done Done Sent false
23989 7855 Acked Read 5 Done Done Sent false
23990 7856 Acked Read 1 Done Done Sent false
23991 7857 Acked Write 5 Done Done Sent false
23992 7858 Acked Write 9 Done Done Sent false
23993 7859 Acked Write 7 Done Done Sent false
23994 7860 Acked Write 3 Done Done Sent false
23995 7861 Acked Write 4 Done Done Sent false
23996 7862 Acked Write 10 Done Done Sent false
23997 7863 Acked Read 3 Done Done Sent false
23998 7864 Acked Read 3 Done Done Sent false
23999 7865 Acked Read 10 Done Done Sent false
24000 7866 Acked Read 1 Done Done Sent false
24001 7867 Acked Read 6 Done Done Sent false
24002 7868 Acked Flush 0 Done Done Sent false
24003 7869 Acked Write 7 Done Done Sent false
24004 7870 Acked Read 4 Done Done Sent false
24005 7871 Acked Write 4 Done Done Sent false
24006 7872 Acked Write 1 Done Done Sent false
24007 7873 Acked Read 5 Done Done Sent false
24008 7874 Acked Read 6 Done Done Sent false
24009 7875 Acked Read 5 Done Done Sent false
24010 7876 Acked Read 4 Done Done Sent false
24011 7877 Acked Write 7 Done Done Sent false
24012 7878 Acked Read 4 Done Done Sent false
24013 7879 Acked Read 9 Done Done Sent false
24014 7880 Acked Write 8 Done Done Sent false
24015 7881 Acked Write 6 Done Done Sent false
24016 7882 Acked Read 5 Done Done Sent false
24017 7883 Acked Read 3 Done Done Sent false
24018 7884 Acked Read 10 Done Done Sent false
24019 7885 Acked Read 8 Done Done Sent false
24020 7886 Acked Write 10 Done Done Sent false
24021 7887 Acked Read 4 Done Done Sent false
24022 7888 Acked Write 3 Done Done Sent false
24023 7889 Acked Flush 0 Done Done Sent false
24024 7890 Acked Read 3 Done Done Sent false
24025 7891 Acked Write 7 Done Done Sent false
24026 7892 Acked Read 2 Done Done Sent false
24027 7893 Acked Flush 0 Done Done Sent false
24028 7894 Acked Write 10 Done Done Sent false
24029 7895 Acked Write 6 Done Done Sent false
24030 7896 Acked Read 7 Done Done Sent false
24031 7897 Acked Read 7 Done Done Sent false
24032 7898 Acked Write 5 Done Done Sent false
24033 7899 Acked Read 6 Done Done Sent false
24034 7900 Acked Read 5 Done Done Sent false
24035 7901 Acked Read 9 Done Done Sent false
24036 7902 Acked Read 10 Done Done Sent false
24037 7903 Acked Read 1 Done Done Sent false
24038 7904 Acked Flush 0 Done Done Sent false
24039 7905 Acked Read 2 Done Done Sent false
24040 7906 Acked Read 1 Done Done Sent false
24041 7907 Acked Read 1 Done Done Sent false
24042 7908 Acked Write 3 Done Done Sent false
24043 7909 Acked Write 7 Done Done Sent false
24044 7910 Acked Read 6 Done Done Sent false
24045 7911 Acked Read 5 Done Done Sent false
24046 7912 Acked Read 9 Done Done Sent false
24047 7913 Acked Read 6 Done Done Sent false
24048 7914 Acked Read 5 Done Done Sent false
24049 7915 Acked Flush 0 Done Done Sent false
24050 7916 Acked Flush 0 Done Done Sent false
24051 7917 Acked Read 1 Done Done Sent false
24052 7918 Acked Read 2 Done Done Sent false
24053 7919 Acked Flush 0 Done Done Sent false
24054 7920 Acked Read 2 Done Done Sent false
24055 7921 Acked Write 2 Done Done Sent false
24056 7922 Acked Read 2 Done Done Sent false
24057 7923 Acked Read 2 Done Done Sent false
24058 7924 Acked Write 4 Done Done Sent false
24059 7925 Acked Read 10 Done Done Sent false
24060 7926 Acked Flush 0 Done Done Sent false
24061 7927 Acked Write 7 Done Done Sent false
24062 7928 Acked Write 3 Done Done Sent false
24063 7929 Acked Write 1 Done Done Sent false
24064 7930 Acked Read 1 Done Done Sent false
24065 7931 Acked Write 2 Done Done Sent false
24066 7932 Acked Write 5 Done Done Sent false
24067 7933 Acked Write 9 Done Done Sent false
24068 7934 Acked Read 4 Done Done Sent false
24069 7935 Acked Read 9 Done Done Sent false
24070 7936 Acked Read 1 Done Done Sent false
24071 7937 Acked Read 4 Done Done Sent false
24072 7938 Acked Read 5 Done Done Sent false
24073 7939 Acked Write 9 Done Done Sent false
24074 7940 Acked Read 7 Done Done Sent false
24075 7941 Acked Write 9 Done Sent Sent false
24076 STATES DS:0 DS:1 DS:2 TOTAL
24077 Sent 0 1 301 302
24078 Done 301 300 0 601
24079 Skipped 0 0 0 0
24080 Error 0 0 0 0
24081 Last Flush: Some(JobId(7926)) Some(JobId(7926)) Some(JobId(7640))
24082 Downstairs last five completed: 7640 7639 7638 7637 7636
24083 Upstairs last five completed: 7941 7940 7939 7938 7937
24084 CLIENT: Up:0 ds:301 act:2
24085 ----------------------------------------------------------------
24086 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24087 States: Active Active Offline
24088 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24089 7641 Acked Write 6 Done Done Sent false
24090 7642 Acked Write 8 Done Done Sent false
24091 7643 Acked Write 8 Done Done Sent false
24092 7644 Acked Read 6 Done Done Sent false
24093 7645 Acked Write 6 Done Done Sent false
24094 7646 Acked Write 7 Done Done Sent false
24095 7647 Acked Read 3 Done Done Sent false
24096 7648 Acked Read 8 Done Done Sent false
24097 7649 Acked Read 8 Done Done Sent false
24098 7650 Acked Read 9 Done Done Sent false
24099 7651 Acked Read 1 Done Done Sent false
24100 7652 Acked Read 9 Done Done Sent false
24101 7653 Acked Read 4 Done Done Sent false
24102 7654 Acked Read 6 Done Done Sent false
24103 7655 Acked Flush 0 Done Done Sent false
24104 7656 Acked Read 3 Done Done Sent false
24105 7657 Acked Read 2 Done Done Sent false
24106 7658 Acked Read 6 Done Done Sent false
24107 7659 Acked Write 9 Done Done Sent false
24108 7660 Acked Write 9 Done Done Sent false
24109 7661 Acked Write 5 Done Done Sent false
24110 7662 Acked Read 5 Done Done Sent false
24111 7663 Acked Write 4 Done Done Sent false
24112 7664 Acked Read 2 Done Done Sent false
24113 7665 Acked Write 10 Done Done Sent false
24114 7666 Acked Write 9 Done Done Sent false
24115 7667 Acked Read 1 Done Done Sent false
24116 7668 Acked Write 8 Done Done Sent false
24117 7669 Acked Write 1 Done Done Sent false
24118 7670 Acked Write 1 Done Done Sent false
24119 7671 Acked Flush 0 Done Done Sent false
24120 7672 Acked Read 2 Done Done Sent false
24121 7673 Acked Read 4 Done Done Sent false
24122 7674 Acked Write 8 Done Done Sent false
24123 7675 Acked Write 2 Done Done Sent false
24124 7676 Acked Flush 0 Done Done Sent false
24125 7677 Acked Flush 0 Done Done Sent false
24126 7678 Acked Write 6 Done Done Sent false
24127 7679 Acked Write 7 Done Done Sent false
24128 7680 Acked Read 8 Done Done Sent false
24129 7681 Acked Flush 0 Done Done Sent false
24130 7682 Acked Read 8 Done Done Sent false
24131 7683 Acked Flush 0 Done Done Sent false
24132 7684 Acked Read 2 Done Done Sent false
24133 7685 Acked Flush 0 Done Done Sent false
24134 7686 Acked Read 3 Done Done Sent false
24135 7687 Acked Read 8 Done Done Sent false
24136 7688 Acked Read 9 Done Done Sent false
24137 7689 Acked Read 2 Done Done Sent false
24138 7690 Acked Read 10 Done Done Sent false
24139 7691 Acked Write 9 Done Done Sent false
24140 7692 Acked Write 7 Done Done Sent false
24141 7693 Acked Write 9 Done Done Sent false
24142 7694 Acked Read 6 Done Done Sent false
24143 7695 Acked Read 8 Done Done Sent false
24144 7696 Acked Write 1 Done Done Sent false
24145 7697 Acked Read 10 Done Done Sent false
24146 7698 Acked Write 1 Done Done Sent false
24147 7699 Acked Read 9 Done Done Sent false
24148 7700 Acked Read 2 Done Done Sent false
24149 7701 Acked Read 6 Done Done Sent false
24150 7702 Acked Read 7 Done Done Sent false
24151 7703 Acked Write 9 Done Done Sent false
24152 7704 Acked Write 6 Done Done Sent false
24153 7705 Acked Read 10 Done Done Sent false
24154 7706 Acked Write 1 Done Done Sent false
24155 7707 Acked Write 1 Done Done Sent false
24156 7708 Acked Read 1 Done Done Sent false
24157 7709 Acked Read 9 Done Done Sent false
24158 7710 Acked Read 9 Done Done Sent false
24159 7711 Acked Write 6 Done Done Sent false
24160 7712 Acked Read 2 Done Done Sent false
24161 7713 Acked Write 1 Done Done Sent false
24162 7714 Acked Read 6 Done Done Sent false
24163 7715 Acked Write 3 Done Done Sent false
24164 7716 Acked Read 4 Done Done Sent false
24165 7717 Acked Read 8 Done Done Sent false
24166 7718 Acked Read 6 Done Done Sent false
24167 7719 Acked Read 7 Done Done Sent false
24168 7720 Acked Write 4 Done Done Sent false
24169 7721 Acked Write 7 Done Done Sent false
24170 7722 Acked Flush 0 Done Done Sent false
24171 7723 Acked Write 7 Done Done Sent false
24172 7724 Acked Read 6 Done Done Sent false
24173 7725 Acked Write 9 Done Done Sent false
24174 7726 Acked Write 6 Done Done Sent false
24175 7727 Acked Read 1 Done Done Sent false
24176 7728 Acked Write 1 Done Done Sent false
24177 7729 Acked Write 3 Done Done Sent false
24178 7730 Acked Write 5 Done Done Sent false
24179 7731 Acked Read 4 Done Done Sent false
24180 7732 Acked Read 5 Done Done Sent false
24181 7733 Acked Write 2 Done Done Sent false
24182 7734 Acked Read 10 Done Done Sent false
24183 7735 Acked Read 4 Done Done Sent false
24184 7736 Acked Read 1 Done Done Sent false
24185 7737 Acked Read 7 Done Done Sent false
24186 7738 Acked Write 6 Done Done Sent false
24187 7739 Acked Read 2 Done Done Sent false
24188 7740 Acked Write 3 Done Done Sent false
24189 7741 Acked Read 8 Done Done Sent false
24190 7742 Acked Write 5 Done Done Sent false
24191 7743 Acked Write 7 Done Done Sent false
24192 7744 Acked Write 7 Done Done Sent false
24193 7745 Acked Read 1 Done Done Sent false
24194 7746 Acked Flush 0 Done Done Sent false
24195 7747 Acked Read 1 Done Done Sent false
24196 7748 Acked Write 6 Done Done Sent false
24197 7749 Acked Write 7 Done Done Sent false
24198 7750 Acked Flush 0 Done Done Sent false
24199 7751 Acked Read 2 Done Done Sent false
24200 7752 Acked Write 5 Done Done Sent false
24201 7753 Acked Flush 0 Done Done Sent false
24202 7754 Acked Read 1 Done Done Sent false
24203 7755 Acked Read 7 Done Done Sent false
24204 7756 Acked Flush 0 Done Done Sent false
24205 7757 Acked Flush 0 Done Done Sent false
24206 7758 Acked Read 1 Done Done Sent false
24207 7759 Acked Read 9 Done Done Sent false
24208 7760 Acked Flush 0 Done Done Sent false
24209 7761 Acked Read 4 Done Done Sent false
24210 7762 Acked Write 7 Done Done Sent false
24211 7763 Acked Write 4 Done Done Sent false
24212 7764 Acked Read 3 Done Done Sent false
24213 7765 Acked Read 3 Done Done Sent false
24214 7766 Acked Read 3 Done Done Sent false
24215 7767 Acked Read 1 Done Done Sent false
24216 7768 Acked Read 2 Done Done Sent false
24217 7769 Acked Write 10 Done Done Sent false
24218 7770 Acked Write 6 Done Done Sent false
24219 7771 Acked Read 1 Done Done Sent false
24220 7772 Acked Read 1 Done Done Sent false
24221 7773 Acked Read 1 Done Done Sent false
24222 7774 Acked Write 4 Done Done Sent false
24223 7775 Acked Write 6 Done Done Sent false
24224 7776 Acked Write 3 Done Done Sent false
24225 7777 Acked Read 6 Done Done Sent false
24226 7778 Acked Write 8 Done Done Sent false
24227 7779 Acked Read 10 Done Done Sent false
24228 7780 Acked Read 10 Done Done Sent false
24229 7781 Acked Write 7 Done Done Sent false
24230 7782 Acked Write 5 Done Done Sent false
24231 7783 Acked Read 2 Done Done Sent false
24232 7784 Acked Write 5 Done Done Sent false
24233 7785 Acked Flush 0 Done Done Sent false
24234 7786 Acked Read 5 Done Done Sent false
24235 7787 Acked Read 10 Done Done Sent false
24236 7788 Acked Write 10 Done Done Sent false
24237 7789 Acked Read 6 Done Done Sent false
24238 7790 Acked Read 6 Done Done Sent false
24239 7791 Acked Write 4 Done Done Sent false
24240 7792 Acked Read 8 Done Done Sent false
24241 7793 Acked Read 6 Done Done Sent false
24242 7794 Acked Write 7 Done Done Sent false
24243 7795 Acked Write 8 Done Done Sent false
24244 7796 Acked Write 9 Done Done Sent false
24245 7797 Acked Write 2 Done Done Sent false
24246 7798 Acked Write 7 Done Done Sent false
24247 7799 Acked Flush 0 Done Done Sent false
24248 7800 Acked Write 2 Done Done Sent false
24249 7801 Acked Write 7 Done Done Sent false
24250 7802 Acked Read 8 Done Done Sent false
24251 7803 Acked Write 5 Done Done Sent false
24252 7804 Acked Write 5 Done Done Sent false
24253 7805 Acked Write 8 Done Done Sent false
24254 7806 Acked Flush 0 Done Done Sent false
24255 7807 Acked Write 9 Done Done Sent false
24256 7808 Acked Write 1 Done Done Sent false
24257 7809 Acked Read 3 Done Done Sent false
24258 7810 Acked Write 3 Done Done Sent false
24259 7811 Acked Flush 0 Done Done Sent false
24260 7812 Acked Write 10 Done Done Sent false
24261 7813 Acked Write 10 Done Done Sent false
24262 7814 Acked Read 1 Done Done Sent false
24263 7815 Acked Read 4 Done Done Sent false
24264 7816 Acked Read 9 Done Done Sent false
24265 7817 Acked Read 3 Done Done Sent false
24266 7818 Acked Read 4 Done Done Sent false
24267 7819 Acked Read 9 Done Done Sent false
24268 7820 Acked Write 8 Done Done Sent false
24269 7821 Acked Read 10 Done Done Sent false
24270 7822 Acked Read 8 Done Done Sent false
24271 7823 Acked Read 8 Done Done Sent false
24272 7824 Acked Read 3 Done Done Sent false
24273 7825 Acked Write 1 Done Done Sent false
24274 7826 Acked Read 4 Done Done Sent false
24275 7827 Acked Write 7 Done Done Sent false
24276 7828 Acked Flush 0 Done Done Sent false
24277 7829 Acked Write 6 Done Done Sent false
24278 7830 Acked Read 9 Done Done Sent false
24279 7831 Acked Read 8 Done Done Sent false
24280 7832 Acked Read 6 Done Done Sent false
24281 7833 Acked Read 7 Done Done Sent false
24282 7834 Acked Write 3 Done Done Sent false
24283 7835 Acked Read 1 Done Done Sent false
24284 7836 Acked Read 9 Done Done Sent false
24285 7837 Acked Read 2 Done Done Sent false
24286 7838 Acked Read 2 Done Done Sent false
24287 7839 Acked Write 5 Done Done Sent false
24288 7840 Acked Read 7 Done Done Sent false
24289 7841 Acked Read 8 Done Done Sent false
24290 7842 Acked Read 1 Done Done Sent false
24291 7843 Acked Write 10 Done Done Sent false
24292 7844 Acked Read 1 Done Done Sent false
24293 7845 Acked Write 9 Done Done Sent false
24294 7846 Acked Write 5 Done Done Sent false
24295 7847 Acked Read 3 Done Done Sent false
24296 7848 Acked Read 9 Done Done Sent false
24297 7849 Acked Read 10 Done Done Sent false
24298 7850 Acked Read 8 Done Done Sent false
24299 7851 Acked Write 6 Done Done Sent false
24300 7852 Acked Read 3 Done Done Sent false
24301 7853 Acked Flush 0 Done Done Sent false
24302 7854 Acked Read 3 Done Done Sent false
24303 7855 Acked Read 5 Done Done Sent false
24304 7856 Acked Read 1 Done Done Sent false
24305 7857 Acked Write 5 Done Done Sent false
24306 7858 Acked Write 9 Done Done Sent false
24307 7859 Acked Write 7 Done Done Sent false
24308 7860 Acked Write 3 Done Done Sent false
24309 7861 Acked Write 4 Done Done Sent false
24310 7862 Acked Write 10 Done Done Sent false
24311 7863 Acked Read 3 Done Done Sent false
24312 7864 Acked Read 3 Done Done Sent false
24313 7865 Acked Read 10 Done Done Sent false
24314 7866 Acked Read 1 Done Done Sent false
24315 7867 Acked Read 6 Done Done Sent false
24316 7868 Acked Flush 0 Done Done Sent false
24317 7869 Acked Write 7 Done Done Sent false
24318 7870 Acked Read 4 Done Done Sent false
24319 7871 Acked Write 4 Done Done Sent false
24320 7872 Acked Write 1 Done Done Sent false
24321 7873 Acked Read 5 Done Done Sent false
24322 7874 Acked Read 6 Done Done Sent false
24323 7875 Acked Read 5 Done Done Sent false
24324 7876 Acked Read 4 Done Done Sent false
24325 7877 Acked Write 7 Done Done Sent false
24326 7878 Acked Read 4 Done Done Sent false
24327 7879 Acked Read 9 Done Done Sent false
24328 7880 Acked Write 8 Done Done Sent false
24329 7881 Acked Write 6 Done Done Sent false
24330 7882 Acked Read 5 Done Done Sent false
24331 7883 Acked Read 3 Done Done Sent false
24332 7884 Acked Read 10 Done Done Sent false
24333 7885 Acked Read 8 Done Done Sent false
24334 7886 Acked Write 10 Done Done Sent false
24335 7887 Acked Read 4 Done Done Sent false
24336 7888 Acked Write 3 Done Done Sent false
24337 7889 Acked Flush 0 Done Done Sent false
24338 7890 Acked Read 3 Done Done Sent false
24339 7891 Acked Write 7 Done Done Sent false
24340 7892 Acked Read 2 Done Done Sent false
24341 7893 Acked Flush 0 Done Done Sent false
24342 7894 Acked Write 10 Done Done Sent false
24343 7895 Acked Write 6 Done Done Sent false
24344 7896 Acked Read 7 Done Done Sent false
24345 7897 Acked Read 7 Done Done Sent false
24346 7898 Acked Write 5 Done Done Sent false
24347 7899 Acked Read 6 Done Done Sent false
24348 7900 Acked Read 5 Done Done Sent false
24349 7901 Acked Read 9 Done Done Sent false
24350 7902 Acked Read 10 Done Done Sent false
24351 7903 Acked Read 1 Done Done Sent false
24352 7904 Acked Flush 0 Done Done Sent false
24353 7905 Acked Read 2 Done Done Sent false
24354 7906 Acked Read 1 Done Done Sent false
24355 7907 Acked Read 1 Done Done Sent false
24356 7908 Acked Write 3 Done Done Sent false
24357 7909 Acked Write 7 Done Done Sent false
24358 7910 Acked Read 6 Done Done Sent false
24359 7911 Acked Read 5 Done Done Sent false
24360 7912 Acked Read 9 Done Done Sent false
24361 7913 Acked Read 6 Done Done Sent false
24362 7914 Acked Read 5 Done Done Sent false
24363 7915 Acked Flush 0 Done Done Sent false
24364 7916 Acked Flush 0 Done Done Sent false
24365 7917 Acked Read 1 Done Done Sent false
24366 7918 Acked Read 2 Done Done Sent false
24367 7919 Acked Flush 0 Done Done Sent false
24368 7920 Acked Read 2 Done Done Sent false
24369 7921 Acked Write 2 Done Done Sent false
24370 7922 Acked Read 2 Done Done Sent false
24371 7923 Acked Read 2 Done Done Sent false
24372 7924 Acked Write 4 Done Done Sent false
24373 7925 Acked Read 10 Done Done Sent false
24374 7926 Acked Flush 0 Done Done Sent false
24375 7927 Acked Write 7 Done Done Sent false
24376 7928 Acked Write 3 Done Done Sent false
24377 7929 Acked Write 1 Done Done Sent false
24378 7930 Acked Read 1 Done Done Sent false
24379 7931 Acked Write 2 Done Done Sent false
24380 7932 Acked Write 5 Done Done Sent false
24381 7933 Acked Write 9 Done Done Sent false
24382 7934 Acked Read 4 Done Done Sent false
24383 7935 Acked Read 9 Done Done Sent false
24384 7936 Acked Read 1 Done Done Sent false
24385 7937 Acked Read 4 Done Done Sent false
24386 7938 Acked Read 5 Done Done Sent false
24387 7939 Acked Write 9 Done Done Sent false
24388 7940 Acked Read 7 Done Done Sent false
24389 7941 Acked Write 9 Done Done Sent false
24390 7942 Acked Flush 0 Done Done Sent false
24391 STATES DS:0 DS:1 DS:2 TOTAL
24392 Sent 0 0 302 302
24393 Done 302 302 0 604
24394 Skipped 0 0 0 0
24395 Error 0 0 0 0
24396 Last Flush: Some(JobId(7942)) Some(JobId(7942)) Some(JobId(7640))
24397 Downstairs last five completed: 7640 7639 7638 7637 7636
24398 Upstairs last five completed: 7942 7941 7940 7939 7938
24399 CLIENT: Up:0 ds:302 act:2
244002025-01-23T18:53:00.451ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
244012025-01-23T18:53:00.451ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
244022025-01-23T18:53:00.451ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
244032025-01-23T18:53:00.451ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
244042025-01-23T18:53:00.451ZINFOcrucible: client task is exiting = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
24405 [8830] got start action
24406 [8830] I am going to restart
24407 Starting downstairs at port 8830
24408 Make output file at "/tmp/dsc/downstairs-8830.txt"
24409 [8830][2] reports Starting
24410 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1223)
24411 [8830][2] reports Running
24412 ----------------------------------------------------------------
24413 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24414 States: Active Active Offline
24415 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24416 7641 Acked Write 6 Done Done Sent false
24417 7642 Acked Write 8 Done Done Sent false
24418 7643 Acked Write 8 Done Done Sent false
24419 7644 Acked Read 6 Done Done Sent false
24420 7645 Acked Write 6 Done Done Sent false
24421 7646 Acked Write 7 Done Done Sent false
24422 7647 Acked Read 3 Done Done Sent false
24423 7648 Acked Read 8 Done Done Sent false
24424 7649 Acked Read 8 Done Done Sent false
24425 7650 Acked Read 9 Done Done Sent false
24426 7651 Acked Read 1 Done Done Sent false
24427 7652 Acked Read 9 Done Done Sent false
24428 7653 Acked Read 4 Done Done Sent false
24429 7654 Acked Read 6 Done Done Sent false
24430 7655 Acked Flush 0 Done Done Sent false
24431 7656 Acked Read 3 Done Done Sent false
24432 7657 Acked Read 2 Done Done Sent false
24433 7658 Acked Read 6 Done Done Sent false
24434 7659 Acked Write 9 Done Done Sent false
24435 7660 Acked Write 9 Done Done Sent false
24436 7661 Acked Write 5 Done Done Sent false
24437 7662 Acked Read 5 Done Done Sent false
24438 7663 Acked Write 4 Done Done Sent false
24439 7664 Acked Read 2 Done Done Sent false
24440 7665 Acked Write 10 Done Done Sent false
24441 7666 Acked Write 9 Done Done Sent false
24442 7667 Acked Read 1 Done Done Sent false
24443 7668 Acked Write 8 Done Done Sent false
24444 7669 Acked Write 1 Done Done Sent false
24445 7670 Acked Write 1 Done Done Sent false
24446 7671 Acked Flush 0 Done Done Sent false
24447 7672 Acked Read 2 Done Done Sent false
24448 7673 Acked Read 4 Done Done Sent false
24449 7674 Acked Write 8 Done Done Sent false
24450 7675 Acked Write 2 Done Done Sent false
24451 7676 Acked Flush 0 Done Done Sent false
24452 7677 Acked Flush 0 Done Done Sent false
24453 7678 Acked Write 6 Done Done Sent false
24454 7679 Acked Write 7 Done Done Sent false
24455 7680 Acked Read 8 Done Done Sent false
24456 7681 Acked Flush 0 Done Done Sent false
24457 7682 Acked Read 8 Done Done Sent false
24458 7683 Acked Flush 0 Done Done Sent false
24459 7684 Acked Read 2 Done Done Sent false
24460 7685 Acked Flush 0 Done Done Sent false
24461 7686 Acked Read 3 Done Done Sent false
24462 7687 Acked Read 8 Done Done Sent false
24463 7688 Acked Read 9 Done Done Sent false
24464 7689 Acked Read 2 Done Done Sent false
24465 7690 Acked Read 10 Done Done Sent false
24466 7691 Acked Write 9 Done Done Sent false
24467 7692 Acked Write 7 Done Done Sent false
24468 7693 Acked Write 9 Done Done Sent false
24469 7694 Acked Read 6 Done Done Sent false
24470 7695 Acked Read 8 Done Done Sent false
24471 7696 Acked Write 1 Done Done Sent false
24472 7697 Acked Read 10 Done Done Sent false
24473 7698 Acked Write 1 Done Done Sent false
24474 7699 Acked Read 9 Done Done Sent false
24475 7700 Acked Read 2 Done Done Sent false
24476 7701 Acked Read 6 Done Done Sent false
24477 7702 Acked Read 7 Done Done Sent false
24478 7703 Acked Write 9 Done Done Sent false
24479 7704 Acked Write 6 Done Done Sent false
24480 7705 Acked Read 10 Done Done Sent false
24481 7706 Acked Write 1 Done Done Sent false
24482 7707 Acked Write 1 Done Done Sent false
24483 7708 Acked Read 1 Done Done Sent false
24484 7709 Acked Read 9 Done Done Sent false
24485 7710 Acked Read 9 Done Done Sent false
24486 7711 Acked Write 6 Done Done Sent false
24487 7712 Acked Read 2 Done Done Sent false
24488 7713 Acked Write 1 Done Done Sent false
24489 7714 Acked Read 6 Done Done Sent false
24490 7715 Acked Write 3 Done Done Sent false
24491 7716 Acked Read 4 Done Done Sent false
24492 7717 Acked Read 8 Done Done Sent false
24493 7718 Acked Read 6 Done Done Sent false
24494 7719 Acked Read 7 Done Done Sent false
24495 7720 Acked Write 4 Done Done Sent false
24496 7721 Acked Write 7 Done Done Sent false
24497 7722 Acked Flush 0 Done Done Sent false
24498 7723 Acked Write 7 Done Done Sent false
24499 7724 Acked Read 6 Done Done Sent false
24500 7725 Acked Write 9 Done Done Sent false
24501 7726 Acked Write 6 Done Done Sent false
24502 7727 Acked Read 1 Done Done Sent false
24503 7728 Acked Write 1 Done Done Sent false
24504 7729 Acked Write 3 Done Done Sent false
24505 7730 Acked Write 5 Done Done Sent false
24506 7731 Acked Read 4 Done Done Sent false
24507 7732 Acked Read 5 Done Done Sent false
24508 7733 Acked Write 2 Done Done Sent false
24509 7734 Acked Read 10 Done Done Sent false
24510 7735 Acked Read 4 Done Done Sent false
24511 7736 Acked Read 1 Done Done Sent false
24512 7737 Acked Read 7 Done Done Sent false
24513 7738 Acked Write 6 Done Done Sent false
24514 7739 Acked Read 2 Done Done Sent false
24515 7740 Acked Write 3 Done Done Sent false
24516 7741 Acked Read 8 Done Done Sent false
24517 7742 Acked Write 5 Done Done Sent false
24518 7743 Acked Write 7 Done Done Sent false
24519 7744 Acked Write 7 Done Done Sent false
24520 7745 Acked Read 1 Done Done Sent false
24521 7746 Acked Flush 0 Done Done Sent false
24522 7747 Acked Read 1 Done Done Sent false
24523 7748 Acked Write 6 Done Done Sent false
24524 7749 Acked Write 7 Done Done Sent false
24525 7750 Acked Flush 0 Done Done Sent false
24526 7751 Acked Read 2 Done Done Sent false
24527 7752 Acked Write 5 Done Done Sent false
24528 7753 Acked Flush 0 Done Done Sent false
24529 7754 Acked Read 1 Done Done Sent false
24530 7755 Acked Read 7 Done Done Sent false
24531 7756 Acked Flush 0 Done Done Sent false
24532 7757 Acked Flush 0 Done Done Sent false
24533 7758 Acked Read 1 Done Done Sent false
24534 7759 Acked Read 9 Done Done Sent false
24535 7760 Acked Flush 0 Done Done Sent false
24536 7761 Acked Read 4 Done Done Sent false
24537 7762 Acked Write 7 Done Done Sent false
24538 7763 Acked Write 4 Done Done Sent false
24539 7764 Acked Read 3 Done Done Sent false
24540 7765 Acked Read 3 Done Done Sent false
24541 7766 Acked Read 3 Done Done Sent false
24542 7767 Acked Read 1 Done Done Sent false
24543 7768 Acked Read 2 Done Done Sent false
24544 7769 Acked Write 10 Done Done Sent false
24545 7770 Acked Write 6 Done Done Sent false
24546 7771 Acked Read 1 Done Done Sent false
24547 7772 Acked Read 1 Done Done Sent false
24548 7773 Acked Read 1 Done Done Sent false
24549 7774 Acked Write 4 Done Done Sent false
24550 7775 Acked Write 6 Done Done Sent false
24551 7776 Acked Write 3 Done Done Sent false
24552 7777 Acked Read 6 Done Done Sent false
24553 7778 Acked Write 8 Done Done Sent false
24554 7779 Acked Read 10 Done Done Sent false
24555 7780 Acked Read 10 Done Done Sent false
24556 7781 Acked Write 7 Done Done Sent false
24557 7782 Acked Write 5 Done Done Sent false
24558 7783 Acked Read 2 Done Done Sent false
24559 7784 Acked Write 5 Done Done Sent false
24560 7785 Acked Flush 0 Done Done Sent false
24561 7786 Acked Read 5 Done Done Sent false
24562 7787 Acked Read 10 Done Done Sent false
24563 7788 Acked Write 10 Done Done Sent false
24564 7789 Acked Read 6 Done Done Sent false
24565 7790 Acked Read 6 Done Done Sent false
24566 7791 Acked Write 4 Done Done Sent false
24567 7792 Acked Read 8 Done Done Sent false
24568 7793 Acked Read 6 Done Done Sent false
24569 7794 Acked Write 7 Done Done Sent false
24570 7795 Acked Write 8 Done Done Sent false
24571 7796 Acked Write 9 Done Done Sent false
24572 7797 Acked Write 2 Done Done Sent false
24573 7798 Acked Write 7 Done Done Sent false
24574 7799 Acked Flush 0 Done Done Sent false
24575 7800 Acked Write 2 Done Done Sent false
24576 7801 Acked Write 7 Done Done Sent false
24577 7802 Acked Read 8 Done Done Sent false
24578 7803 Acked Write 5 Done Done Sent false
24579 7804 Acked Write 5 Done Done Sent false
24580 7805 Acked Write 8 Done Done Sent false
24581 7806 Acked Flush 0 Done Done Sent false
24582 7807 Acked Write 9 Done Done Sent false
24583 7808 Acked Write 1 Done Done Sent false
24584 7809 Acked Read 3 Done Done Sent false
24585 7810 Acked Write 3 Done Done Sent false
24586 7811 Acked Flush 0 Done Done Sent false
24587 7812 Acked Write 10 Done Done Sent false
24588 7813 Acked Write 10 Done Done Sent false
24589 7814 Acked Read 1 Done Done Sent false
24590 7815 Acked Read 4 Done Done Sent false
24591 7816 Acked Read 9 Done Done Sent false
24592 7817 Acked Read 3 Done Done Sent false
24593 7818 Acked Read 4 Done Done Sent false
24594 7819 Acked Read 9 Done Done Sent false
24595 7820 Acked Write 8 Done Done Sent false
24596 7821 Acked Read 10 Done Done Sent false
24597 7822 Acked Read 8 Done Done Sent false
24598 7823 Acked Read 8 Done Done Sent false
24599 7824 Acked Read 3 Done Done Sent false
24600 7825 Acked Write 1 Done Done Sent false
24601 7826 Acked Read 4 Done Done Sent false
24602 7827 Acked Write 7 Done Done Sent false
24603 7828 Acked Flush 0 Done Done Sent false
24604 7829 Acked Write 6 Done Done Sent false
24605 7830 Acked Read 9 Done Done Sent false
24606 7831 Acked Read 8 Done Done Sent false
24607 7832 Acked Read 6 Done Done Sent false
24608 7833 Acked Read 7 Done Done Sent false
24609 7834 Acked Write 3 Done Done Sent false
24610 7835 Acked Read 1 Done Done Sent false
24611 7836 Acked Read 9 Done Done Sent false
24612 7837 Acked Read 2 Done Done Sent false
24613 7838 Acked Read 2 Done Done Sent false
24614 7839 Acked Write 5 Done Done Sent false
24615 7840 Acked Read 7 Done Done Sent false
24616 7841 Acked Read 8 Done Done Sent false
24617 7842 Acked Read 1 Done Done Sent false
24618 7843 Acked Write 10 Done Done Sent false
24619 7844 Acked Read 1 Done Done Sent false
24620 7845 Acked Write 9 Done Done Sent false
24621 7846 Acked Write 5 Done Done Sent false
24622 7847 Acked Read 3 Done Done Sent false
24623 7848 Acked Read 9 Done Done Sent false
24624 7849 Acked Read 10 Done Done Sent false
24625 7850 Acked Read 8 Done Done Sent false
24626 7851 Acked Write 6 Done Done Sent false
24627 7852 Acked Read 3 Done Done Sent false
24628 7853 Acked Flush 0 Done Done Sent false
24629 7854 Acked Read 3 Done Done Sent false
24630 7855 Acked Read 5 Done Done Sent false
24631 7856 Acked Read 1 Done Done Sent false
24632 7857 Acked Write 5 Done Done Sent false
24633 7858 Acked Write 9 Done Done Sent false
24634 7859 Acked Write 7 Done Done Sent false
24635 7860 Acked Write 3 Done Done Sent false
24636 7861 Acked Write 4 Done Done Sent false
24637 7862 Acked Write 10 Done Done Sent false
24638 7863 Acked Read 3 Done Done Sent false
24639 7864 Acked Read 3 Done Done Sent false
24640 7865 Acked Read 10 Done Done Sent false
24641 7866 Acked Read 1 Done Done Sent false
24642 7867 Acked Read 6 Done Done Sent false
24643 7868 Acked Flush 0 Done Done Sent false
24644 7869 Acked Write 7 Done Done Sent false
24645 7870 Acked Read 4 Done Done Sent false
24646 7871 Acked Write 4 Done Done Sent false
24647 7872 Acked Write 1 Done Done Sent false
24648 7873 Acked Read 5 Done Done Sent false
24649 7874 Acked Read 6 Done Done Sent false
24650 7875 Acked Read 5 Done Done Sent false
24651 7876 Acked Read 4 Done Done Sent false
24652 7877 Acked Write 7 Done Done Sent false
24653 7878 Acked Read 4 Done Done Sent false
24654 7879 Acked Read 9 Done Done Sent false
24655 7880 Acked Write 8 Done Done Sent false
24656 7881 Acked Write 6 Done Done Sent false
24657 7882 Acked Read 5 Done Done Sent false
24658 7883 Acked Read 3 Done Done Sent false
24659 7884 Acked Read 10 Done Done Sent false
24660 7885 Acked Read 8 Done Done Sent false
24661 7886 Acked Write 10 Done Done Sent false
24662 7887 Acked Read 4 Done Done Sent false
24663 7888 Acked Write 3 Done Done Sent false
24664 7889 Acked Flush 0 Done Done Sent false
24665 7890 Acked Read 3 Done Done Sent false
24666 7891 Acked Write 7 Done Done Sent false
24667 7892 Acked Read 2 Done Done Sent false
24668 7893 Acked Flush 0 Done Done Sent false
24669 7894 Acked Write 10 Done Done Sent false
24670 7895 Acked Write 6 Done Done Sent false
24671 7896 Acked Read 7 Done Done Sent false
24672 7897 Acked Read 7 Done Done Sent false
24673 7898 Acked Write 5 Done Done Sent false
24674 7899 Acked Read 6 Done Done Sent false
24675 7900 Acked Read 5 Done Done Sent false
24676 7901 Acked Read 9 Done Done Sent false
24677 7902 Acked Read 10 Done Done Sent false
24678 7903 Acked Read 1 Done Done Sent false
24679 7904 Acked Flush 0 Done Done Sent false
24680 7905 Acked Read 2 Done Done Sent false
24681 7906 Acked Read 1 Done Done Sent false
24682 7907 Acked Read 1 Done Done Sent false
24683 7908 Acked Write 3 Done Done Sent false
24684 7909 Acked Write 7 Done Done Sent false
24685 7910 Acked Read 6 Done Done Sent false
24686 7911 Acked Read 5 Done Done Sent false
24687 7912 Acked Read 9 Done Done Sent false
24688 7913 Acked Read 6 Done Done Sent false
24689 7914 Acked Read 5 Done Done Sent false
24690 7915 Acked Flush 0 Done Done Sent false
24691 7916 Acked Flush 0 Done Done Sent false
24692 7917 Acked Read 1 Done Done Sent false
24693 7918 Acked Read 2 Done Done Sent false
24694 7919 Acked Flush 0 Done Done Sent false
24695 7920 Acked Read 2 Done Done Sent false
24696 7921 Acked Write 2 Done Done Sent false
24697 7922 Acked Read 2 Done Done Sent false
24698 7923 Acked Read 2 Done Done Sent false
24699 7924 Acked Write 4 Done Done Sent false
24700 7925 Acked Read 10 Done Done Sent false
24701 7926 Acked Flush 0 Done Done Sent false
24702 7927 Acked Write 7 Done Done Sent false
24703 7928 Acked Write 3 Done Done Sent false
24704 7929 Acked Write 1 Done Done Sent false
24705 7930 Acked Read 1 Done Done Sent false
24706 7931 Acked Write 2 Done Done Sent false
24707 7932 Acked Write 5 Done Done Sent false
24708 7933 Acked Write 9 Done Done Sent false
24709 7934 Acked Read 4 Done Done Sent false
24710 7935 Acked Read 9 Done Done Sent false
24711 7936 Acked Read 1 Done Done Sent false
24712 7937 Acked Read 4 Done Done Sent false
24713 7938 Acked Read 5 Done Done Sent false
24714 7939 Acked Write 9 Done Done Sent false
24715 7940 Acked Read 7 Done Done Sent false
24716 7941 Acked Write 9 Done Done Sent false
24717 7942 Acked Flush 0 Done Done Sent false
24718 STATES DS:0 DS:1 DS:2 TOTAL
24719 Sent 0 0 302 302
24720 Done 302 302 0 604
24721 Skipped 0 0 0 0
24722 Error 0 0 0 0
24723 Last Flush: Some(JobId(7942)) Some(JobId(7942)) Some(JobId(7640))
24724 Downstairs last five completed: 7640 7639 7638 7637 7636
24725 Upstairs last five completed: 7942 7941 7940 7939 7938
24726 CLIENT: Up:0 ds:302 act:2
24727 ----------------------------------------------------------------
24728 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24729 States: Active Active Offline
24730 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24731 7641 Acked Write 6 Done Done Sent false
24732 7642 Acked Write 8 Done Done Sent false
24733 7643 Acked Write 8 Done Done Sent false
24734 7644 Acked Read 6 Done Done Sent false
24735 7645 Acked Write 6 Done Done Sent false
24736 7646 Acked Write 7 Done Done Sent false
24737 7647 Acked Read 3 Done Done Sent false
24738 7648 Acked Read 8 Done Done Sent false
24739 7649 Acked Read 8 Done Done Sent false
24740 7650 Acked Read 9 Done Done Sent false
24741 7651 Acked Read 1 Done Done Sent false
24742 7652 Acked Read 9 Done Done Sent false
24743 7653 Acked Read 4 Done Done Sent false
24744 7654 Acked Read 6 Done Done Sent false
24745 7655 Acked Flush 0 Done Done Sent false
24746 7656 Acked Read 3 Done Done Sent false
24747 7657 Acked Read 2 Done Done Sent false
24748 7658 Acked Read 6 Done Done Sent false
24749 7659 Acked Write 9 Done Done Sent false
24750 7660 Acked Write 9 Done Done Sent false
24751 7661 Acked Write 5 Done Done Sent false
24752 7662 Acked Read 5 Done Done Sent false
24753 7663 Acked Write 4 Done Done Sent false
24754 7664 Acked Read 2 Done Done Sent false
24755 7665 Acked Write 10 Done Done Sent false
24756 7666 Acked Write 9 Done Done Sent false
24757 7667 Acked Read 1 Done Done Sent false
24758 7668 Acked Write 8 Done Done Sent false
24759 7669 Acked Write 1 Done Done Sent false
24760 7670 Acked Write 1 Done Done Sent false
24761 7671 Acked Flush 0 Done Done Sent false
24762 7672 Acked Read 2 Done Done Sent false
24763 7673 Acked Read 4 Done Done Sent false
24764 7674 Acked Write 8 Done Done Sent false
24765 7675 Acked Write 2 Done Done Sent false
24766 7676 Acked Flush 0 Done Done Sent false
24767 7677 Acked Flush 0 Done Done Sent false
24768 7678 Acked Write 6 Done Done Sent false
24769 7679 Acked Write 7 Done Done Sent false
24770 7680 Acked Read 8 Done Done Sent false
24771 7681 Acked Flush 0 Done Done Sent false
24772 7682 Acked Read 8 Done Done Sent false
24773 7683 Acked Flush 0 Done Done Sent false
24774 7684 Acked Read 2 Done Done Sent false
24775 7685 Acked Flush 0 Done Done Sent false
24776 7686 Acked Read 3 Done Done Sent false
24777 7687 Acked Read 8 Done Done Sent false
24778 7688 Acked Read 9 Done Done Sent false
24779 7689 Acked Read 2 Done Done Sent false
24780 7690 Acked Read 10 Done Done Sent false
24781 7691 Acked Write 9 Done Done Sent false
24782 7692 Acked Write 7 Done Done Sent false
24783 7693 Acked Write 9 Done Done Sent false
24784 7694 Acked Read 6 Done Done Sent false
24785 7695 Acked Read 8 Done Done Sent false
24786 7696 Acked Write 1 Done Done Sent false
24787 7697 Acked Read 10 Done Done Sent false
24788 7698 Acked Write 1 Done Done Sent false
24789 7699 Acked Read 9 Done Done Sent false
24790 7700 Acked Read 2 Done Done Sent false
24791 7701 Acked Read 6 Done Done Sent false
24792 7702 Acked Read 7 Done Done Sent false
24793 7703 Acked Write 9 Done Done Sent false
24794 7704 Acked Write 6 Done Done Sent false
24795 7705 Acked Read 10 Done Done Sent false
24796 7706 Acked Write 1 Done Done Sent false
24797 7707 Acked Write 1 Done Done Sent false
24798 7708 Acked Read 1 Done Done Sent false
24799 7709 Acked Read 9 Done Done Sent false
24800 7710 Acked Read 9 Done Done Sent false
24801 7711 Acked Write 6 Done Done Sent false
24802 7712 Acked Read 2 Done Done Sent false
24803 7713 Acked Write 1 Done Done Sent false
24804 7714 Acked Read 6 Done Done Sent false
24805 7715 Acked Write 3 Done Done Sent false
24806 7716 Acked Read 4 Done Done Sent false
24807 7717 Acked Read 8 Done Done Sent false
24808 7718 Acked Read 6 Done Done Sent false
24809 7719 Acked Read 7 Done Done Sent false
24810 7720 Acked Write 4 Done Done Sent false
24811 7721 Acked Write 7 Done Done Sent false
24812 7722 Acked Flush 0 Done Done Sent false
24813 7723 Acked Write 7 Done Done Sent false
24814 7724 Acked Read 6 Done Done Sent false
24815 7725 Acked Write 9 Done Done Sent false
24816 7726 Acked Write 6 Done Done Sent false
24817 7727 Acked Read 1 Done Done Sent false
24818 7728 Acked Write 1 Done Done Sent false
24819 7729 Acked Write 3 Done Done Sent false
24820 7730 Acked Write 5 Done Done Sent false
24821 7731 Acked Read 4 Done Done Sent false
24822 7732 Acked Read 5 Done Done Sent false
24823 7733 Acked Write 2 Done Done Sent false
24824 7734 Acked Read 10 Done Done Sent false
24825 7735 Acked Read 4 Done Done Sent false
24826 7736 Acked Read 1 Done Done Sent false
24827 7737 Acked Read 7 Done Done Sent false
24828 7738 Acked Write 6 Done Done Sent false
24829 7739 Acked Read 2 Done Done Sent false
24830 7740 Acked Write 3 Done Done Sent false
24831 7741 Acked Read 8 Done Done Sent false
24832 7742 Acked Write 5 Done Done Sent false
24833 7743 Acked Write 7 Done Done Sent false
24834 7744 Acked Write 7 Done Done Sent false
24835 7745 Acked Read 1 Done Done Sent false
24836 7746 Acked Flush 0 Done Done Sent false
24837 7747 Acked Read 1 Done Done Sent false
24838 7748 Acked Write 6 Done Done Sent false
24839 7749 Acked Write 7 Done Done Sent false
24840 7750 Acked Flush 0 Done Done Sent false
24841 7751 Acked Read 2 Done Done Sent false
24842 7752 Acked Write 5 Done Done Sent false
24843 7753 Acked Flush 0 Done Done Sent false
24844 7754 Acked Read 1 Done Done Sent false
24845 7755 Acked Read 7 Done Done Sent false
24846 7756 Acked Flush 0 Done Done Sent false
24847 7757 Acked Flush 0 Done Done Sent false
24848 7758 Acked Read 1 Done Done Sent false
24849 7759 Acked Read 9 Done Done Sent false
24850 7760 Acked Flush 0 Done Done Sent false
24851 7761 Acked Read 4 Done Done Sent false
24852 7762 Acked Write 7 Done Done Sent false
24853 7763 Acked Write 4 Done Done Sent false
24854 7764 Acked Read 3 Done Done Sent false
24855 7765 Acked Read 3 Done Done Sent false
24856 7766 Acked Read 3 Done Done Sent false
24857 7767 Acked Read 1 Done Done Sent false
24858 7768 Acked Read 2 Done Done Sent false
24859 7769 Acked Write 10 Done Done Sent false
24860 7770 Acked Write 6 Done Done Sent false
24861 7771 Acked Read 1 Done Done Sent false
24862 7772 Acked Read 1 Done Done Sent false
24863 7773 Acked Read 1 Done Done Sent false
24864 7774 Acked Write 4 Done Done Sent false
24865 7775 Acked Write 6 Done Done Sent false
24866 7776 Acked Write 3 Done Done Sent false
24867 7777 Acked Read 6 Done Done Sent false
24868 7778 Acked Write 8 Done Done Sent false
24869 7779 Acked Read 10 Done Done Sent false
24870 7780 Acked Read 10 Done Done Sent false
24871 7781 Acked Write 7 Done Done Sent false
24872 7782 Acked Write 5 Done Done Sent false
24873 7783 Acked Read 2 Done Done Sent false
24874 7784 Acked Write 5 Done Done Sent false
24875 7785 Acked Flush 0 Done Done Sent false
24876 7786 Acked Read 5 Done Done Sent false
24877 7787 Acked Read 10 Done Done Sent false
24878 7788 Acked Write 10 Done Done Sent false
24879 7789 Acked Read 6 Done Done Sent false
24880 7790 Acked Read 6 Done Done Sent false
24881 7791 Acked Write 4 Done Done Sent false
24882 7792 Acked Read 8 Done Done Sent false
24883 7793 Acked Read 6 Done Done Sent false
24884 7794 Acked Write 7 Done Done Sent false
24885 7795 Acked Write 8 Done Done Sent false
24886 7796 Acked Write 9 Done Done Sent false
24887 7797 Acked Write 2 Done Done Sent false
24888 7798 Acked Write 7 Done Done Sent false
24889 7799 Acked Flush 0 Done Done Sent false
24890 7800 Acked Write 2 Done Done Sent false
24891 7801 Acked Write 7 Done Done Sent false
24892 7802 Acked Read 8 Done Done Sent false
24893 7803 Acked Write 5 Done Done Sent false
24894 7804 Acked Write 5 Done Done Sent false
24895 7805 Acked Write 8 Done Done Sent false
24896 7806 Acked Flush 0 Done Done Sent false
24897 7807 Acked Write 9 Done Done Sent false
24898 7808 Acked Write 1 Done Done Sent false
24899 7809 Acked Read 3 Done Done Sent false
24900 7810 Acked Write 3 Done Done Sent false
24901 7811 Acked Flush 0 Done Done Sent false
24902 7812 Acked Write 10 Done Done Sent false
24903 7813 Acked Write 10 Done Done Sent false
24904 7814 Acked Read 1 Done Done Sent false
24905 7815 Acked Read 4 Done Done Sent false
24906 7816 Acked Read 9 Done Done Sent false
24907 7817 Acked Read 3 Done Done Sent false
24908 7818 Acked Read 4 Done Done Sent false
24909 7819 Acked Read 9 Done Done Sent false
24910 7820 Acked Write 8 Done Done Sent false
24911 7821 Acked Read 10 Done Done Sent false
24912 7822 Acked Read 8 Done Done Sent false
24913 7823 Acked Read 8 Done Done Sent false
24914 7824 Acked Read 3 Done Done Sent false
24915 7825 Acked Write 1 Done Done Sent false
24916 7826 Acked Read 4 Done Done Sent false
24917 7827 Acked Write 7 Done Done Sent false
24918 7828 Acked Flush 0 Done Done Sent false
24919 7829 Acked Write 6 Done Done Sent false
24920 7830 Acked Read 9 Done Done Sent false
24921 7831 Acked Read 8 Done Done Sent false
24922 7832 Acked Read 6 Done Done Sent false
24923 7833 Acked Read 7 Done Done Sent false
24924 7834 Acked Write 3 Done Done Sent false
24925 7835 Acked Read 1 Done Done Sent false
24926 7836 Acked Read 9 Done Done Sent false
24927 7837 Acked Read 2 Done Done Sent false
24928 7838 Acked Read 2 Done Done Sent false
24929 7839 Acked Write 5 Done Done Sent false
24930 7840 Acked Read 7 Done Done Sent false
24931 7841 Acked Read 8 Done Done Sent false
24932 7842 Acked Read 1 Done Done Sent false
24933 7843 Acked Write 10 Done Done Sent false
24934 7844 Acked Read 1 Done Done Sent false
24935 7845 Acked Write 9 Done Done Sent false
24936 7846 Acked Write 5 Done Done Sent false
24937 7847 Acked Read 3 Done Done Sent false
24938 7848 Acked Read 9 Done Done Sent false
24939 7849 Acked Read 10 Done Done Sent false
24940 7850 Acked Read 8 Done Done Sent false
24941 7851 Acked Write 6 Done Done Sent false
24942 7852 Acked Read 3 Done Done Sent false
24943 7853 Acked Flush 0 Done Done Sent false
24944 7854 Acked Read 3 Done Done Sent false
24945 7855 Acked Read 5 Done Done Sent false
24946 7856 Acked Read 1 Done Done Sent false
24947 7857 Acked Write 5 Done Done Sent false
24948 7858 Acked Write 9 Done Done Sent false
24949 7859 Acked Write 7 Done Done Sent false
24950 7860 Acked Write 3 Done Done Sent false
24951 7861 Acked Write 4 Done Done Sent false
24952 7862 Acked Write 10 Done Done Sent false
24953 7863 Acked Read 3 Done Done Sent false
24954 7864 Acked Read 3 Done Done Sent false
24955 7865 Acked Read 10 Done Done Sent false
24956 7866 Acked Read 1 Done Done Sent false
24957 7867 Acked Read 6 Done Done Sent false
24958 7868 Acked Flush 0 Done Done Sent false
24959 7869 Acked Write 7 Done Done Sent false
24960 7870 Acked Read 4 Done Done Sent false
24961 7871 Acked Write 4 Done Done Sent false
24962 7872 Acked Write 1 Done Done Sent false
24963 7873 Acked Read 5 Done Done Sent false
24964 7874 Acked Read 6 Done Done Sent false
24965 7875 Acked Read 5 Done Done Sent false
24966 7876 Acked Read 4 Done Done Sent false
24967 7877 Acked Write 7 Done Done Sent false
24968 7878 Acked Read 4 Done Done Sent false
24969 7879 Acked Read 9 Done Done Sent false
24970 7880 Acked Write 8 Done Done Sent false
24971 7881 Acked Write 6 Done Done Sent false
24972 7882 Acked Read 5 Done Done Sent false
24973 7883 Acked Read 3 Done Done Sent false
24974 7884 Acked Read 10 Done Done Sent false
24975 7885 Acked Read 8 Done Done Sent false
24976 7886 Acked Write 10 Done Done Sent false
24977 7887 Acked Read 4 Done Done Sent false
24978 7888 Acked Write 3 Done Done Sent false
24979 7889 Acked Flush 0 Done Done Sent false
24980 7890 Acked Read 3 Done Done Sent false
24981 7891 Acked Write 7 Done Done Sent false
24982 7892 Acked Read 2 Done Done Sent false
24983 7893 Acked Flush 0 Done Done Sent false
24984 7894 Acked Write 10 Done Done Sent false
24985 7895 Acked Write 6 Done Done Sent false
24986 7896 Acked Read 7 Done Done Sent false
24987 7897 Acked Read 7 Done Done Sent false
24988 7898 Acked Write 5 Done Done Sent false
24989 7899 Acked Read 6 Done Done Sent false
24990 7900 Acked Read 5 Done Done Sent false
24991 7901 Acked Read 9 Done Done Sent false
24992 7902 Acked Read 10 Done Done Sent false
24993 7903 Acked Read 1 Done Done Sent false
24994 7904 Acked Flush 0 Done Done Sent false
24995 7905 Acked Read 2 Done Done Sent false
24996 7906 Acked Read 1 Done Done Sent false
24997 7907 Acked Read 1 Done Done Sent false
24998 7908 Acked Write 3 Done Done Sent false
24999 7909 Acked Write 7 Done Done Sent false
25000 7910 Acked Read 6 Done Done Sent false
25001 7911 Acked Read 5 Done Done Sent false
25002 7912 Acked Read 9 Done Done Sent false
25003 7913 Acked Read 6 Done Done Sent false
25004 7914 Acked Read 5 Done Done Sent false
25005 7915 Acked Flush 0 Done Done Sent false
25006 7916 Acked Flush 0 Done Done Sent false
25007 7917 Acked Read 1 Done Done Sent false
25008 7918 Acked Read 2 Done Done Sent false
25009 7919 Acked Flush 0 Done Done Sent false
25010 7920 Acked Read 2 Done Done Sent false
25011 7921 Acked Write 2 Done Done Sent false
25012 7922 Acked Read 2 Done Done Sent false
25013 7923 Acked Read 2 Done Done Sent false
25014 7924 Acked Write 4 Done Done Sent false
25015 7925 Acked Read 10 Done Done Sent false
25016 7926 Acked Flush 0 Done Done Sent false
25017 7927 Acked Write 7 Done Done Sent false
25018 7928 Acked Write 3 Done Done Sent false
25019 7929 Acked Write 1 Done Done Sent false
25020 7930 Acked Read 1 Done Done Sent false
25021 7931 Acked Write 2 Done Done Sent false
25022 7932 Acked Write 5 Done Done Sent false
25023 7933 Acked Write 9 Done Done Sent false
25024 7934 Acked Read 4 Done Done Sent false
25025 7935 Acked Read 9 Done Done Sent false
25026 7936 Acked Read 1 Done Done Sent false
25027 7937 Acked Read 4 Done Done Sent false
25028 7938 Acked Read 5 Done Done Sent false
25029 7939 Acked Write 9 Done Done Sent false
25030 7940 Acked Read 7 Done Done Sent false
25031 7941 Acked Write 9 Done Done Sent false
25032 7942 Acked Flush 0 Done Done Sent false
25033 STATES DS:0 DS:1 DS:2 TOTAL
25034 Sent 0 0 302 302
25035 Done 302 302 0 604
25036 Skipped 0 0 0 0
25037 Error 0 0 0 0
25038 Last Flush: Some(JobId(7942)) Some(JobId(7942)) Some(JobId(7640))
25039 Downstairs last five completed: 7640 7639 7638 7637 7636
25040 Upstairs last five completed: 7942 7941 7940 7939 7938
25041 CLIENT: Up:0 ds:302 act:2
250422025-01-23T18:53:10.452ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
250432025-01-23T18:53:10.452ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65387) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
250442025-01-23T18:53:10.453ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
250452025-01-23T18:53:10.454ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
250462025-01-23T18:53:10.454ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
250472025-01-23T18:53:10.454ZINFOcrucible: send last flush ID to this DS: Some(JobId(7640)) = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
250482025-01-23T18:53:10.454ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
250492025-01-23T18:53:10.454ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(7640)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
250502025-01-23T18:53:10.454ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(7640)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
25051 ----------------------------------------------------------------
25052 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25053 Downstairs last five completed: 7942 7941 7940 7939 7938
25054 Upstairs last five completed: 7942 7941 7940 7939 7938
25055 CLIENT: Up:0 ds:0 act:3
25056 Replay: All jobs finished
25057 Main task has work to do, go find it
25058 got dsc Stop(2)
25059 stop 2
25060 [8830] Got stop action so:false kr:false
25061 Jan 23 18:53:10.697 INFO request completed, latency_us: 326, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 682cd191-bae7-416c-863d-650b42882ec8, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
250622025-01-23T18:53:10.698ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
250632025-01-23T18:53:10.698ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
250642025-01-23T18:53:10.698ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
25065 Jan 23 18:53:10.698 INFO request completed, latency_us: 159, response_code: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T18:53:10.698253084Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
25066 200, uri: /state/cid/2, method: GET, req_id: 0e7ffbd0-59ae-47e1-be9e-c89e77794000, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
25067 [8830] Exited with: ExitStatus(unix_wait_status(9))
25068 [8830][2] reports Exit
25069 Jan 23 18:53:14.700 INFO request completed, latency_us: 197, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1827d6d4-917c-4a03-93f0-d9f901e68a02, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
25070 001/300 Read block 920 len 28672
25071 002/300 Write block 958 len 32768 data: 9 8 7 6 7 7 7 7
25072 003/300 Write block 395 len 24576 data: 5 7 7 6 6 5
25073 004/300 Write block 4477 len 12288 data: 7 6 7
25074 005/300 Write block 997 len 20480 data: 4 6 7 7 7
25075 006/300 Read block 4380 len 4096
25076 007/300 Read block 3839 len 8192
25077 008/300 Read block 130 len 36864
25078 009/300 Read block 2656 len 24576
25079 010/300 Write block 4687 len 20480 data: 6 4 4 4 5
25080 011/300 Write block 3537 len 8192 data: 9 8
25081 012/300 Read block 3314 len 36864
25082 013/300 Read block 1436 len 20480
25083 014/300 Flush
25084 015/300 Read block 2756 len 32768
25085 016/300 Read block 4146 len 16384
25086 017/300 Read block 3750 len 24576
25087 018/300 Flush
25088 019/300 Write block 4043 len 12288 data: 4 4 4
25089 020/300 Read block 282 len 32768
25090 021/300 Read block 4346 len 36864
25091 022/300 Write block 2323 len 8192 data: 6 7
25092 023/300 Write block 3773 len 8192 data: 5 5
25093 024/300 Read block 1666 len 12288
25094 025/300 Read block 445 len 12288
25095 026/300 Write block 2679 len 32768 data: 6 6 6 6 6 5 6 5
25096 027/300 Read block 4607 len 24576
25097 028/300 Read block 3075 len 4096
25098 029/300 Read block 3155 len 36864
25099 030/300 Write block 1471 len 12288 data: 5 5 5
25100 031/300 Write block 4146 len 32768 data: 10 11 10 11 11 9 7 8
25101 032/300 Write block 2238 len 20480 data: 5 5 5 6 5
25102 033/300 Write block 2170 len 20480 data: 4 4 4 4 6
25103 034/300 Read block 1476 len 40960
25104 035/300 Flush
25105 036/300 Write block 1254 len 20480 data: 4 3 3 4 4
25106 037/300 Write block 3775 len 24576 data: 5 5 5 7 8 6
25107 038/300 Read block 1334 len 28672
25108 039/300 Flush
25109 040/300 Flush
25110 041/300 Read block 1472 len 20480
25111 042/300 Flush
25112 043/300 Write block 3520 len 8192 data: 3 4
25113 044/300 Write block 4581 len 36864 data: 7 8 7 6 5 5 5 3 2
25114 045/300 Write block 3661 len 32768 data: 4 4 3 4 5 5 5 5
25115 046/300 Flush
25116 047/300 Flush
25117 048/300 Read block 4829 len 32768
25118 049/300 Flush
25119 050/300 Write block 2391 len 36864 data: 7 7 5 7 4 4 4 3 2
25120 051/300 Flush
25121 052/300 Read block 2236 len 4096
25122 053/300 Write block 3860 len 36864 data: 7 5 5 5 5 5 3 4 4
25123 054/300 Read block 1854 len 36864
25124 055/300 Read block 2426 len 28672
25125 056/300 Flush
25126 057/300 Write block 3234 len 32768 data: 8 8 9 9 8 6 6 7
25127 058/300 Read block 3656 len 28672
25128 059/300 Read block 2079 len 24576
25129 060/300 Read block 2896 len 28672
25130 061/300 Read block 1002 len 32768
25131 062/300 Read block 1044 len 32768
25132 063/300 Write block 3004 len 32768 data: 7 8 7 7 5 6 7 5
25133 064/300 Read block 4865 len 32768
25134 065/300 Read block 4455 len 28672
25135 066/300 Read block 2188 len 28672
25136 067/300 Write block 2227 len 32768 data: 4 6 6 7 7 7 7 6
25137 068/300 Read block 3304 len 16384
25138 069/300 Read block 19 len 40960
25139 070/300 Write block 242 len 28672 data: 5 4 3 5 5 8 8
25140 071/300 Write block 2380 len 8192 data: 4 4
25141 072/300 Read block 2659 len 16384
25142 073/300 Flush
25143 074/300 Read block 843 len 40960
25144 075/300 Write block 2826 len 28672 data: 3 2 3 3 4 5 5
25145 076/300 Read block 4816 len 24576
25146 077/300 Read block 239 len 20480
25147 078/300 Write block 3791 len 8192 data: 5 4
25148 079/300 Read block 2903 len 32768
25149 080/300 Read block 3749 len 36864
25150 081/300 Write block 828 len 16384 data: 7 6 5 4
25151 082/300 Write block 3004 len 32768 data: 8 9 8 8 6 7 8 6
25152 083/300 Read block 4410 len 20480
25153 084/300 Flush
25154 085/300 Read block 3969 len 20480
25155 086/300 Write block 3261 len 4096 data: 4
25156 087/300 Write block 1856 len 24576 data: 6 5 6 7 6 7
25157 088/300 Write block 2903 len 32768 data: 3 4 5 5 5 6 7 7
25158 089/300 Write block 4897 len 8192 data: 5 5
25159 090/300 Write block 557 len 24576 data: 6 7 7 5 3 4
25160 091/300 Write block 2630 len 12288 data: 7 6 8
25161 092/300 Read block 2839 len 32768
25162 093/300 Write block 1897 len 40960 data: 5 6 5 6 6 7 8 8 6 5
25163 094/300 Write block 2236 len 20480 data: 5 5 6 6 6
25164 095/300 Write block 4532 len 24576 data: 3 4 5 5 4 5
25165 096/300 Write block 3969 len 36864 data: 4 5 4 4 5 5 5 5 6
25166 097/300 Flush
25167 098/300 Read block 1506 len 40960
25168 099/300 Read block 2239 len 28672
25169 100/300 Read block 1701 len 36864
25170 101/300 Read block 1229 len 36864
25171 102/300 Write block 3411 len 20480 data: 3 3 4 4 4
25172 103/300 Read block 2111 len 4096
25173 104/300 Read block 1852 len 28672
25174 105/300 Read block 2647 len 16384
25175 106/300 Write block 4758 len 32768 data: 6 6 6 6 5 4 4 4
25176 107/300 Write block 2858 len 28672 data: 6 5 5 5 5 4 4
25177 108/300 Read block 4152 len 28672
25178 109/300 Read block 1963 len 28672
25179 110/300 Read block 4884 len 8192
25180 111/300 Write block 297 len 12288 data: 3 3 2
25181 112/300 Read block 4211 len 4096
25182 113/300 Read block 3137 len 12288
25183 114/300 Write block 4741 len 28672 data: 3 3 4 5 5 5 6
25184 115/300 Write block 2630 len 28672 data: 8 7 9 7 4 6 4
25185 116/300 Write block 2407 len 16384 data: 6 7 7 6
25186 117/300 Read block 2946 len 20480
25187 118/300 Read block 4483 len 4096
25188 119/300 Write block 2837 len 32768 data: 5 6 5 6 6 6 6 7
25189 120/300 Flush
25190 121/300 Write block 1197 len 16384 data: 4 4 4 6
25191 122/300 Read block 3448 len 28672
25192 123/300 Read block 752 len 8192
25193 124/300 Write block 319 len 12288 data: 4 3 3
25194 125/300 Read block 4630 len 12288
25195 126/300 Read block 3968 len 12288
25196 127/300 Read block 4301 len 28672
25197 128/300 Flush
25198 129/300 Write block 3124 len 36864 data: 5 5 4 4 3 3 4 3 3
25199 130/300 Flush
25200 131/300 Read block 4638 len 40960
25201 132/300 Write block 472 len 16384 data: 5 6 5 6
25202 133/300 Write block 4474 len 8192 data: 7 7
25203 134/300 Read block 429 len 16384
25204 135/300 Read block 3296 len 40960
25205 136/300 Flush
25206 137/300 Read block 1260 len 40960
25207 138/300 Read block 3747 len 28672
25208 139/300 Flush
25209 140/300 Write block 1290 len 12288 data: 9 8 6
25210 141/300 Write block 2707 len 20480 data: 4 4 4 4 5
25211 142/300 Write block 1785 len 32768 data: 4 5 5 5 6 8 7 7
25212 143/300 Read block 2130 len 36864
25213 144/300 Read block 460 len 4096
25214 145/300 Write block 1345 len 16384 data: 6 4 3 2
25215 146/300 Read block 230 len 12288
25216 147/300 Read block 2479 len 4096
25217 148/300 Write block 4100 len 20480 data: 4 2 4 4 4
25218 149/300 Read block 2992 len 36864
25219 150/300 Write block 4729 len 28672 data: 4 3 3 4 5 5 4
25220 151/300 Write block 292 len 8192 data: 7 5
25221 152/300 Read block 3271 len 20480
25222 153/300 Write block 4773 len 12288 data: 2 2 2
25223 154/300 Read block 3570 len 32768
25224 155/300 Read block 882 len 4096
25225 156/300 Write block 1898 len 40960 data: 7 6 7 7 8 9 9 7 6 4
25226 157/300 Write block 2211 len 8192 data: 7 6
25227 158/300 Flush
25228 159/300 Read block 3159 len 28672
25229 160/300 Write block 2084 len 32768 data: 5 5 5 5 5 2 2 2
25230 161/300 Read block 359 len 8192
25231 162/300 Read block 4855 len 28672
25232 163/300 Read block 2842 len 4096
25233 164/300 Flush
25234 165/300 Write block 1237 len 8192 data: 6 7
25235 166/300 Write block 4642 len 24576 data: 5 8 7 7 8 8
25236 167/300 Flush
25237 168/300 Read block 2615 len 16384
25238 169/300 Write block 1180 len 36864 data: 3 2 2 2 3 3 5 5 4
25239 170/300 Write block 4003 len 12288 data: 5 4 5
25240 171/300 Write block 109 len 24576 data: 7 5 7 7 8 9
25241 172/300 Write block 2535 len 8192 data: 7 6
25242 173/300 Write block 2218 len 8192 data: 2 3
25243 174/300 Flush
25244 175/300 Flush
25245 176/300 Write block 4542 len 16384 data: 5 5 5 4
25246 177/300 Read block 1109 len 16384
25247 178/300 Write block 4390 len 36864 data: 4 2 2 2 3 2 2 4 5
25248 179/300 Read block 472 len 40960
25249 180/300 Read block 118 len 12288
25250 181/300 Read block 263 len 32768
25251 182/300 Write block 4266 len 12288 data: 4 8 6
25252 183/300 Read block 2264 len 28672
25253 184/300 Read block 910 len 4096
25254 185/300 Write block 3694 len 36864 data: 7 6 6 5 6 5 4 3 4
25255 186/300 Write block 971 len 4096 data: 4
25256 187/300 Write block 2397 len 24576 data: 5 4 3 2 3 3
25257 188/300 Read block 1296 len 28672
25258 189/300 Write block 3441 len 4096 data: 6
25259 190/300 Write block 1522 len 12288 data: 4 5 5
25260 191/300 Write block 1073 len 12288 data: 9 7 7
25261 192/300 Write block 1042 len 40960 data: 6 6 7 9 7 8 9 9 7 7
25262 193/300 Write block 4855 len 8192 data: 4 5
25263 194/300 Read block 2407 len 4096
25264 195/300 Flush
25265 196/300 Write block 1257 len 8192 data: 5 5
25266 197/300 Read block 2217 len 8192
25267 198/300 Read block 4657 len 32768
25268 199/300 Read block 3386 len 4096
25269 200/300 Read block 1851 len 36864
25270 201/300 Write block 1759 len 28672 data: 6 5 4 5 5 4 5
25271 202/300 Write block 1818 len 28672 data: 7 6 7 6 4 4 4
25272 203/300 Write block 4206 len 16384 data: 4 5 5 4
25273 204/300 Read block 2407 len 24576
25274 205/300 Read block 1658 len 40960
25275 206/300 Write block 2808 len 20480 data: 2 2 2 3 3
25276 207/300 Write block 2731 len 40960 data: 5 5 5 5 5 6 7 7 6 5
25277 208/300 Flush
25278 209/300 Read block 2505 len 16384
25279 210/300 Read block 4034 len 16384
25280 211/300 Write block 1337 len 8192 data: 3 4
25281 212/300 Read block 2501 len 40960
25282 213/300 Flush
25283 214/300 Read block 1231 len 36864
25284 215/300 Read block 2301 len 4096
25285 216/300 Flush
25286 217/300 Write block 4374 len 8192 data: 6 7
25287 218/300 Read block 3500 len 40960
25288 219/300 Write block 1179 len 16384 data: 3 4 3 3
25289 220/300 Read block 558 len 16384
25290 221/300 Read block 1665 len 8192
25291 222/300 Write block 3016 len 24576 data: 8 6 5 5 5 7
25292 223/300 Write block 1019 len 36864 data: 8 9 10 7 8 7 6 7 8
25293 224/300 Read block 4219 len 40960
25294 225/300 Read block 383 len 8192
25295 226/300 Read block 166 len 40960
25296 227/300 Write block 1784 len 20480 data: 5 5 6 6 6
25297 228/300 Write block 3617 len 32768 data: 5 5 6 6 6 8 7 7
25298 229/300 Read block 1714 len 12288
25299 230/300 Read block 1570 len 40960
25300 231/300 Write block 995 len 24576 data: 3 3 5 7 8 8
25301 232/300 Write block 463 len 12288 data: 7 5 4
25302 233/300 Read block 3112 len 16384
25303 234/300 Write block 1795 len 4096 data: 9
25304 235/300 Write block 2400 len 20480 data: 3 4 4 7 7
25305 236/300 Write block 4600 len 32768 data: 6 6 5 4 5 6 6 5
25306 237/300 Read block 1083 len 20480
25307 238/300 Read block 4689 len 8192
25308 239/300 Read block 3982 len 36864
25309 240/300 Write block 3662 len 28672 data: 5 4 5 6 6 6 6
25310 241/300 Flush
25311 242/300 Read block 901 len 8192
25312 243/300 Read block 689 len 24576
25313 244/300 Read block 4986 len 28672
25314 245/300 Read block 156 len 16384
25315 246/300 Write block 3118 len 32768 data: 4 4 6 7 6 6 6 6
25316 247/300 Read block 3628 len 16384
25317 248/300 Read block 2585 len 4096
25318 249/300 Write block 2854 len 8192 data: 5 6
25319 250/300 Read block 3507 len 36864
25320 251/300 Write block 3056 len 8192 data: 3 5
25321 252/300 Write block 3162 len 4096 data: 5
25322 253/300 Read block 3352 len 8192
25323 254/300 Write block 2550 len 20480 data: 4 3 3 4 3
25324 255/300 Write block 674 len 4096 data: 6
25325 256/300 Read block 2894 len 8192
25326 257/300 Write block 4911 len 24576 data: 8 5 6 6 8 9
25327 258/300 Read block 4594 len 28672
25328 259/300 Read block 3407 len 20480
25329 260/300 Read block 418 len 28672
25330 261/300 Read block 3760 len 28672
25331 262/300 Flush
25332 263/300 Read block 2044 len 12288
25333 264/300 Write block 1761 len 32768 data: 5 6 6 5 6 4 3 3
25334 265/300 Read block 2850 len 8192
25335 266/300 Write block 424 len 32768 data: 4 4 4 6 11 12 11 10
25336 267/300 Read block 4951 len 4096
25337 268/300 Read block 388 len 28672
25338 269/300 Write block 733 len 12288 data: 6 6 5
25339 270/300 Write block 4567 len 40960 data: 5 6 3 3 3 5 4 4 6 5
25340 271/300 Read block 310 len 4096
25341 272/300 Write block 1134 len 24576 data: 9 9 7 6 8 10
25342 273/300 Read block 1798 len 16384
25343 274/300 Read block 3606 len 32768
25344 275/300 Read block 453 len 28672
25345 276/300 Read block 2745 len 8192
25346 277/300 Read block 3129 len 32768
25347 278/300 Write block 1041 len 12288 data: 5 7 7
25348 279/300 Read block 2838 len 28672
25349 280/300 Flush
25350 281/300 Read block 424 len 4096
25351 282/300 Flush
25352 283/300 Write block 2430 len 4096 data: 5
25353 284/300 Write block 2455 len 12288 data: 4 6 5
25354 285/300 Read block 4741 len 8192
25355 286/300 Flush
25356 287/300 Read block 1616 len 8192
25357 288/300 Read block 1503 len 4096
25358 289/300 Write block 1638 len 4096 data: 7
25359 290/300 Read block 744 len 24576
25360 291/300 Flush
25361 292/300 Read block 3103 len 40960
25362 293/300 Read block 186 len 20480
25363 294/300 Write block 632 len 12288 data: 6 6 5
25364 295/300 Read block 3587 len 16384
25365 296/300 Write block 389 len 28672 data: 5 5 6 6 5 5 6
25366 297/300 Read block 53 len 24576
25367 298/300 Read block 4630 len 20480
25368 299/300 Write block 4133 len 40960 data: 3 4 5 5 5 3 6 6 7 8
25369 300/300 Read block 257 len 28672
25370 301/300 Write block 2883 len 32768 data: 4 5 7 7 7 8 7 7
25371 Main task has work to do, go find it
25372 got dsc Start(2)
25373 start 2
25374 Jan 23 18:53:14.939 INFO request completed, latency_us: 316, response_code: 204, uri: /start/cid/2, method: POST, req_id: 01935d45-11b7-46be-92cd-20c869197b22, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
25375 Replay: started 2, returned:Ok(())
25376 ----------------------------------------------------------------
25377 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25378 States: Active Active Offline
25379 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25380 7943 Acked Read 7 Done Done Sent false
25381 7944 Acked Write 8 Done Done Sent false
25382 7945 Acked Write 6 Done Done Sent false
25383 7946 Acked Write 3 Done Done Sent false
25384 7947 Acked Write 5 Done Done Sent false
25385 7948 Acked Read 1 Done Done Sent false
25386 7949 Acked Read 2 Done Done Sent false
25387 7950 Acked Read 9 Done Done Sent false
25388 7951 Acked Read 6 Done Done Sent false
25389 7952 Acked Write 5 Done Done Sent false
25390 7953 Acked Write 2 Done Done Sent false
25391 7954 Acked Read 9 Done Done Sent false
25392 7955 Acked Read 5 Done Done Sent false
25393 7956 Acked Flush 0 Done Done Sent false
25394 7957 Acked Read 8 Done Done Sent false
25395 7958 Acked Read 4 Done Done Sent false
25396 7959 Acked Read 6 Done Done Sent false
25397 7960 Acked Flush 0 Done Done Sent false
25398 7961 Acked Write 3 Done Done Sent false
25399 7962 Acked Read 8 Done Done Sent false
25400 7963 Acked Read 9 Done Done Sent false
25401 7964 Acked Write 2 Done Done Sent false
25402 7965 Acked Write 2 Done Done Sent false
25403 7966 Acked Read 3 Done Done Sent false
25404 7967 Acked Read 3 Done Done Sent false
25405 7968 Acked Write 8 Done Done Sent false
25406 7969 Acked Read 6 Done Done Sent false
25407 7970 Acked Read 1 Done Done Sent false
25408 7971 Acked Read 9 Done Done Sent false
25409 7972 Acked Write 3 Done Done Sent false
25410 7973 Acked Write 8 Done Done Sent false
25411 7974 Acked Write 5 Done Done Sent false
25412 7975 Acked Write 5 Done Done Sent false
25413 7976 Acked Read 10 Done Done Sent false
25414 7977 Acked Flush 0 Done Done Sent false
25415 7978 Acked Write 5 Done Done Sent false
25416 7979 Acked Write 6 Done Done Sent false
25417 7980 Acked Read 7 Done Done Sent false
25418 7981 Acked Flush 0 Done Done Sent false
25419 7982 Acked Flush 0 Done Done Sent false
25420 7983 Acked Read 5 Done Done Sent false
25421 7984 Acked Flush 0 Done Done Sent false
25422 7985 Acked Write 2 Done Done Sent false
25423 7986 Acked Write 9 Done Done Sent false
25424 7987 Acked Write 8 Done Done Sent false
25425 7988 Acked Flush 0 Done Done Sent false
25426 7989 Acked Flush 0 Done Done Sent false
25427 7990 Acked Read 8 Done Done Sent false
25428 7991 Acked Flush 0 Done Done Sent false
25429 7992 Acked Write 9 Done Done Sent false
25430 7993 Acked Flush 0 Done Done Sent false
25431 7994 Acked Read 1 Done Done Sent false
25432 7995 Acked Write 9 Done Done Sent false
25433 7996 Acked Read 9 Done Done Sent false
25434 7997 Acked Read 7 Done Done Sent false
25435 7998 Acked Flush 0 Done Done Sent false
25436 7999 Acked Write 8 Done Done Sent false
25437 8000 Acked Read 7 Done Done Sent false
25438 8001 Acked Read 6 Done Done Sent false
25439 8002 Acked Read 7 Done Done Sent false
25440 8003 Acked Read 8 Done Done Sent false
25441 8004 Acked Read 8 Done Done Sent false
25442 8005 Acked Write 8 Done Done Sent false
25443 8006 Acked Read 8 Done Done Sent false
25444 8007 Acked Read 7 Done Done Sent false
25445 8008 Acked Read 7 Done Done Sent false
25446 8009 Acked Write 8 Done Done Sent false
25447 8010 Acked Read 4 Done Done Sent false
25448 8011 Acked Read 10 Done Done Sent false
25449 8012 Acked Write 7 Done Done Sent false
25450 8013 Acked Write 2 Done Done Sent false
25451 8014 Acked Read 4 Done Done Sent false
25452 8015 Acked Flush 0 Done Done Sent false
25453 8016 Acked Read 10 Done Done Sent false
25454 8017 Acked Write 7 Done Done Sent false
25455 8018 Acked Read 6 Done Done Sent false
25456 8019 Acked Read 5 Done Done Sent false
25457 8020 Acked Write 2 Done Done Sent false
25458 8021 Acked Read 8 Done Done Sent false
25459 8022 Acked Read 9 Done Done Sent false
25460 8023 Acked Write 4 Done Done Sent false
25461 8024 Acked Write 8 Done Done Sent false
25462 8025 Acked Read 5 Done Done Sent false
25463 8026 Acked Flush 0 Done Done Sent false
25464 8027 Acked Read 5 Done Done Sent false
25465 8028 Acked Write 1 Done Done Sent false
25466 8029 Acked Write 6 Done Done Sent false
25467 8030 Acked Write 8 Done Done Sent false
25468 8031 Acked Write 2 Done Done Sent false
25469 8032 Acked Write 6 Done Done Sent false
25470 8033 Acked Write 3 Done Done Sent false
25471 8034 Acked Read 8 Done Done Sent false
25472 8035 Acked Write 10 Done Done Sent false
25473 8036 Acked Write 5 Done Done Sent false
25474 8037 Acked Write 6 Done Done Sent false
25475 8038 Acked Write 9 Done Done Sent false
25476 8039 Acked Flush 0 Done Done Sent false
25477 8040 Acked Read 10 Done Done Sent false
25478 8041 Acked Read 7 Done Done Sent false
25479 8042 Acked Read 9 Done Done Sent false
25480 8043 Acked Read 9 Done Done Sent false
25481 8044 Acked Write 5 Done Done Sent false
25482 8045 Acked Read 1 Done Done Sent false
25483 8046 Acked Read 7 Done Done Sent false
25484 8047 Acked Read 4 Done Done Sent false
25485 8048 Acked Write 8 Done Done Sent false
25486 8049 Acked Write 7 Done Done Sent false
25487 8050 Acked Read 7 Done Done Sent false
25488 8051 Acked Read 7 Done Done Sent false
25489 8052 Acked Read 2 Done Done Sent false
25490 8053 Acked Write 3 Done Done Sent false
25491 8054 Acked Read 1 Done Done Sent false
25492 8055 Acked Read 3 Done Done Sent false
25493 8056 Acked Write 7 Done Done Sent false
25494 8057 Acked Write 7 Done Done Sent false
25495 8058 Acked Write 4 Done Done Sent false
25496 8059 Acked Read 5 Done Done Sent false
25497 8060 Acked Read 1 Done Done Sent false
25498 8061 Acked Write 8 Done Done Sent false
25499 8062 Acked Flush 0 Done Done Sent false
25500 8063 Acked Write 4 Done Done Sent false
25501 8064 Acked Read 7 Done Done Sent false
25502 8065 Acked Read 2 Done Done Sent false
25503 8066 Acked Write 3 Done Done Sent false
25504 8067 Acked Read 3 Done Done Sent false
25505 8068 Acked Read 3 Done Done Sent false
25506 8069 Acked Read 7 Done Done Sent false
25507 8070 Acked Flush 0 Done Done Sent false
25508 8071 Acked Write 9 Done Done Sent false
25509 8072 Acked Flush 0 Done Done Sent false
25510 8073 Acked Read 10 Done Done Sent false
25511 8074 Acked Write 4 Done Done Sent false
25512 8075 Acked Write 2 Done Done Sent false
25513 8076 Acked Read 4 Done Done Sent false
25514 8077 Acked Read 10 Done Done Sent false
25515 8078 Acked Flush 0 Done Done Sent false
25516 8079 Acked Read 10 Done Done Sent false
25517 8080 Acked Read 7 Done Done Sent false
25518 8081 Acked Flush 0 Done Done Sent false
25519 8082 Acked Write 3 Done Done Sent false
25520 8083 Acked Write 5 Done Done Sent false
25521 8084 Acked Write 8 Done Done Sent false
25522 8085 Acked Read 9 Done Done Sent false
25523 8086 Acked Read 1 Done Done Sent false
25524 8087 Acked Write 4 Done Done Sent false
25525 8088 Acked Read 3 Done Done Sent false
25526 8089 Acked Read 1 Done Done Sent false
25527 8090 Acked Write 5 Done Done Sent false
25528 8091 Acked Read 9 Done Done Sent false
25529 8092 Acked Write 7 Done Done Sent false
25530 8093 Acked Write 2 Done Done Sent false
25531 8094 Acked Read 5 Done Done Sent false
25532 8095 Acked Write 3 Done Done Sent false
25533 8096 Acked Read 8 Done Done Sent false
25534 8097 Acked Read 1 Done Done Sent false
25535 8098 Acked Write 10 Done Done Sent false
25536 8099 Acked Write 2 Done Done Sent false
25537 8100 Acked Flush 0 Done Done Sent false
25538 8101 Acked Read 7 Done Done Sent false
25539 8102 Acked Write 8 Done Done Sent false
25540 8103 Acked Read 2 Done Done Sent false
25541 8104 Acked Read 7 Done Done Sent false
25542 8105 Acked Read 1 Done Done Sent false
25543 8106 Acked Flush 0 Done Done Sent false
25544 8107 Acked Write 2 Done Done Sent false
25545 8108 Acked Write 6 Done Done Sent false
25546 8109 Acked Flush 0 Done Done Sent false
25547 8110 Acked Read 4 Done Done Sent false
25548 8111 Acked Write 9 Done Done Sent false
25549 8112 Acked Write 3 Done Done Sent false
25550 8113 Acked Write 6 Done Done Sent false
25551 8114 Acked Write 2 Done Done Sent false
25552 8115 Acked Write 2 Done Done Sent false
25553 8116 Acked Flush 0 Done Done Sent false
25554 8117 Acked Flush 0 Done Done Sent false
25555 8118 Acked Write 4 Done Done Sent false
25556 8119 Acked Read 4 Done Done Sent false
25557 8120 Acked Write 9 Done Done Sent false
25558 8121 Acked Read 10 Done Done Sent false
25559 8122 Acked Read 3 Done Done Sent false
25560 8123 Acked Read 8 Done Done Sent false
25561 8124 Acked Write 3 Done Done Sent false
25562 8125 Acked Read 7 Done Done Sent false
25563 8126 Acked Read 1 Done Done Sent false
25564 8127 Acked Write 9 Done Done Sent false
25565 8128 Acked Write 1 Done Done Sent false
25566 8129 Acked Write 6 Done Done Sent false
25567 8130 Acked Read 7 Done Done Sent false
25568 8131 Acked Write 1 Done Done Sent false
25569 8132 Acked Write 3 Done Done Sent false
25570 8133 Acked Write 3 Done Done Sent false
25571 8134 Acked Write 10 Done Done Sent false
25572 8135 Acked Write 2 Done Done Sent false
25573 8136 Acked Read 1 Done Done Sent false
25574 8137 Acked Flush 0 Done Done Sent false
25575 8138 Acked Write 2 Done Done Sent false
25576 8139 Acked Read 2 Done Done Sent false
25577 8140 Acked Read 8 Done Done Sent false
25578 8141 Acked Read 1 Done Done Sent false
25579 8142 Acked Read 9 Done Done Sent false
25580 8143 Acked Write 7 Done Done Sent false
25581 8144 Acked Write 7 Done Done Sent false
25582 8145 Acked Write 4 Done Done Sent false
25583 8146 Acked Read 6 Done Done Sent false
25584 8147 Acked Read 10 Done Done Sent false
25585 8148 Acked Write 5 Done Done Sent false
25586 8149 Acked Write 10 Done Done Sent false
25587 8150 Acked Flush 0 Done Done Sent false
25588 8151 Acked Read 4 Done Done Sent false
25589 8152 Acked Read 4 Done Done Sent false
25590 8153 Acked Write 2 Done Done Sent false
25591 8154 Acked Read 10 Done Done Sent false
25592 8155 Acked Flush 0 Done Done Sent false
25593 8156 Acked Read 9 Done Done Sent false
25594 8157 Acked Read 1 Done Done Sent false
25595 8158 Acked Flush 0 Done Done Sent false
25596 8159 Acked Write 2 Done Done Sent false
25597 8160 Acked Read 10 Done Done Sent false
25598 8161 Acked Write 4 Done Done Sent false
25599 8162 Acked Read 4 Done Done Sent false
25600 8163 Acked Read 2 Done Done Sent false
25601 8164 Acked Write 6 Done Done Sent false
25602 8165 Acked Write 9 Done Done Sent false
25603 8166 Acked Read 10 Done Done Sent false
25604 8167 Acked Read 2 Done Done Sent false
25605 8168 Acked Read 10 Done Done Sent false
25606 8169 Acked Write 5 Done Done Sent false
25607 8170 Acked Write 8 Done Done Sent false
25608 8171 Acked Read 3 Done Done Sent false
25609 8172 Acked Read 10 Done Done Sent false
25610 8173 Acked Write 6 Done Done Sent false
25611 8174 Acked Write 3 Done Done Sent false
25612 8175 Acked Read 4 Done Done Sent false
25613 8176 Acked Write 1 Done Done Sent false
25614 8177 Acked Write 5 Done Done Sent false
25615 8178 Acked Write 8 Done Done Sent false
25616 8179 Acked Read 5 Done Done Sent false
25617 8180 Acked Read 2 Done Done Sent false
25618 8181 Acked Read 9 Done Done Sent false
25619 8182 Acked Write 7 Done Done Sent false
25620 8183 Acked Flush 0 Done Done Sent false
25621 8184 Acked Read 2 Done Done Sent false
25622 8185 Acked Read 6 Done Done Sent false
25623 8186 Acked Read 7 Done Done Sent false
25624 8187 Acked Read 4 Done Done Sent false
25625 8188 Acked Write 8 Done Done Sent false
25626 8189 Acked Read 4 Done Done Sent false
25627 8190 Acked Read 1 Done Done Sent false
25628 8191 Acked Write 2 Done Done Sent false
25629 8192 Acked Read 9 Done Done Sent false
25630 8193 Acked Write 2 Done Done Sent false
25631 8194 Acked Write 1 Done Done Sent false
25632 8195 Acked Read 2 Done Done Sent false
25633 8196 Acked Write 5 Done Done Sent false
25634 8197 Acked Write 1 Done Done Sent false
25635 8198 Acked Read 2 Done Done Sent false
25636 8199 Acked Write 6 Done Done Sent false
25637 8200 Acked Read 7 Done Done Sent false
25638 8201 Acked Read 5 Done Done Sent false
25639 8202 Acked Read 7 Done Done Sent false
25640 8203 Acked Read 7 Done Done Sent false
25641 8204 Acked Flush 0 Done Done Sent false
25642 8205 Acked Read 3 Done Done Sent false
25643 8206 Acked Write 8 Done Done Sent false
25644 8207 Acked Read 2 Done Done Sent false
25645 8208 Acked Write 8 Done Done Sent false
25646 8209 Acked Read 1 Done Done Sent false
25647 8210 Acked Read 7 Done Done Sent false
25648 8211 Acked Write 3 Done Done Sent false
25649 8212 Acked Write 10 Done Done Sent false
25650 8213 Acked Read 1 Done Done Sent false
25651 8214 Acked Write 6 Done Done Sent false
25652 8215 Acked Read 4 Done Done Sent false
25653 8216 Acked Read 8 Done Done Sent false
25654 8217 Acked Read 7 Done Done Sent false
25655 8218 Acked Read 2 Done Done Sent false
25656 8219 Acked Read 8 Done Done Sent false
25657 8220 Acked Write 3 Done Done Sent false
25658 8221 Acked Read 7 Done Done Sent false
25659 8222 Acked Flush 0 Done Done Sent false
25660 8223 Acked Read 1 Done Done Sent false
25661 8224 Acked Flush 0 Done Done Sent false
25662 8225 Acked Write 1 Done Done Sent false
25663 8226 Acked Write 3 Done Done Sent false
25664 8227 Acked Read 2 Done Done Sent false
25665 8228 Acked Flush 0 Done Done Sent false
25666 8229 Acked Read 2 Done Done Sent false
25667 8230 Acked Read 1 Done Done Sent false
25668 8231 Acked Write 1 Done Done Sent false
25669 8232 Acked Read 6 Done Done Sent false
25670 8233 Acked Flush 0 Done Done Sent false
25671 8234 Acked Read 10 Done Done Sent false
25672 8235 Acked Read 5 Done Done Sent false
25673 8236 Acked Write 3 Done Done Sent false
25674 8237 Acked Read 4 Done Done Sent false
25675 8238 Acked Write 7 Done Done Sent false
25676 8239 Acked Read 6 Done Done Sent false
25677 8240 Acked Read 5 Done Done Sent false
25678 8241 Acked Write 10 Done Done Sent false
25679 8242 Acked Read 7 Done Done Sent false
25680 8243 Acked Write 8 Done Done Sent false
25681 STATES DS:0 DS:1 DS:2 TOTAL
25682 Sent 0 0 301 301
25683 Done 301 301 0 602
25684 Skipped 0 0 0 0
25685 Error 0 0 0 0
25686 Last Flush: Some(JobId(8233)) Some(JobId(8233)) Some(JobId(7942))
25687 Downstairs last five completed: 7942 7941 7940 7939 7938
25688 Upstairs last five completed: 8243 8242 8241 8240 8239
25689 CLIENT: Up:0 ds:301 act:2
25690 [8830] got start action
25691 [8830] I am going to restart
25692 Starting downstairs at port 8830
25693 Make output file at "/tmp/dsc/downstairs-8830.txt"
25694 [8830][2] reports Starting
25695 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1224)
25696 [8830][2] reports Running
25697 ----------------------------------------------------------------
25698 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
25699 States: Active Active Offline
25700 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25701 7943 Acked Read 7 Done Done Sent false
25702 7944 Acked Write 8 Done Done Sent false
25703 7945 Acked Write 6 Done Done Sent false
25704 7946 Acked Write 3 Done Done Sent false
25705 7947 Acked Write 5 Done Done Sent false
25706 7948 Acked Read 1 Done Done Sent false
25707 7949 Acked Read 2 Done Done Sent false
25708 7950 Acked Read 9 Done Done Sent false
25709 7951 Acked Read 6 Done Done Sent false
25710 7952 Acked Write 5 Done Done Sent false
25711 7953 Acked Write 2 Done Done Sent false
25712 7954 Acked Read 9 Done Done Sent false
25713 7955 Acked Read 5 Done Done Sent false
25714 7956 Acked Flush 0 Done Done Sent false
25715 7957 Acked Read 8 Done Done Sent false
25716 7958 Acked Read 4 Done Done Sent false
25717 7959 Acked Read 6 Done Done Sent false
25718 7960 Acked Flush 0 Done Done Sent false
25719 7961 Acked Write 3 Done Done Sent false
25720 7962 Acked Read 8 Done Done Sent false
25721 7963 Acked Read 9 Done Done Sent false
25722 7964 Acked Write 2 Done Done Sent false
25723 7965 Acked Write 2 Done Done Sent false
25724 7966 Acked Read 3 Done Done Sent false
25725 7967 Acked Read 3 Done Done Sent false
25726 7968 Acked Write 8 Done Done Sent false
25727 7969 Acked Read 6 Done Done Sent false
25728 7970 Acked Read 1 Done Done Sent false
25729 7971 Acked Read 9 Done Done Sent false
25730 7972 Acked Write 3 Done Done Sent false
25731 7973 Acked Write 8 Done Done Sent false
25732 7974 Acked Write 5 Done Done Sent false
25733 7975 Acked Write 5 Done Done Sent false
25734 7976 Acked Read 10 Done Done Sent false
25735 7977 Acked Flush 0 Done Done Sent false
25736 7978 Acked Write 5 Done Done Sent false
25737 7979 Acked Write 6 Done Done Sent false
25738 7980 Acked Read 7 Done Done Sent false
25739 7981 Acked Flush 0 Done Done Sent false
25740 7982 Acked Flush 0 Done Done Sent false
25741 7983 Acked Read 5 Done Done Sent false
25742 7984 Acked Flush 0 Done Done Sent false
25743 7985 Acked Write 2 Done Done Sent false
25744 7986 Acked Write 9 Done Done Sent false
25745 7987 Acked Write 8 Done Done Sent false
25746 7988 Acked Flush 0 Done Done Sent false
25747 7989 Acked Flush 0 Done Done Sent false
25748 7990 Acked Read 8 Done Done Sent false
25749 7991 Acked Flush 0 Done Done Sent false
25750 7992 Acked Write 9 Done Done Sent false
25751 7993 Acked Flush 0 Done Done Sent false
25752 7994 Acked Read 1 Done Done Sent false
25753 7995 Acked Write 9 Done Done Sent false
25754 7996 Acked Read 9 Done Done Sent false
25755 7997 Acked Read 7 Done Done Sent false
25756 7998 Acked Flush 0 Done Done Sent false
25757 7999 Acked Write 8 Done Done Sent false
25758 8000 Acked Read 7 Done Done Sent false
25759 8001 Acked Read 6 Done Done Sent false
25760 8002 Acked Read 7 Done Done Sent false
25761 8003 Acked Read 8 Done Done Sent false
25762 8004 Acked Read 8 Done Done Sent false
25763 8005 Acked Write 8 Done Done Sent false
25764 8006 Acked Read 8 Done Done Sent false
25765 8007 Acked Read 7 Done Done Sent false
25766 8008 Acked Read 7 Done Done Sent false
25767 8009 Acked Write 8 Done Done Sent false
25768 8010 Acked Read 4 Done Done Sent false
25769 8011 Acked Read 10 Done Done Sent false
25770 8012 Acked Write 7 Done Done Sent false
25771 8013 Acked Write 2 Done Done Sent false
25772 8014 Acked Read 4 Done Done Sent false
25773 8015 Acked Flush 0 Done Done Sent false
25774 8016 Acked Read 10 Done Done Sent false
25775 8017 Acked Write 7 Done Done Sent false
25776 8018 Acked Read 6 Done Done Sent false
25777 8019 Acked Read 5 Done Done Sent false
25778 8020 Acked Write 2 Done Done Sent false
25779 8021 Acked Read 8 Done Done Sent false
25780 8022 Acked Read 9 Done Done Sent false
25781 8023 Acked Write 4 Done Done Sent false
25782 8024 Acked Write 8 Done Done Sent false
25783 8025 Acked Read 5 Done Done Sent false
25784 8026 Acked Flush 0 Done Done Sent false
25785 8027 Acked Read 5 Done Done Sent false
25786 8028 Acked Write 1 Done Done Sent false
25787 8029 Acked Write 6 Done Done Sent false
25788 8030 Acked Write 8 Done Done Sent false
25789 8031 Acked Write 2 Done Done Sent false
25790 8032 Acked Write 6 Done Done Sent false
25791 8033 Acked Write 3 Done Done Sent false
25792 8034 Acked Read 8 Done Done Sent false
25793 8035 Acked Write 10 Done Done Sent false
25794 8036 Acked Write 5 Done Done Sent false
25795 8037 Acked Write 6 Done Done Sent false
25796 8038 Acked Write 9 Done Done Sent false
25797 8039 Acked Flush 0 Done Done Sent false
25798 8040 Acked Read 10 Done Done Sent false
25799 8041 Acked Read 7 Done Done Sent false
25800 8042 Acked Read 9 Done Done Sent false
25801 8043 Acked Read 9 Done Done Sent false
25802 8044 Acked Write 5 Done Done Sent false
25803 8045 Acked Read 1 Done Done Sent false
25804 8046 Acked Read 7 Done Done Sent false
25805 8047 Acked Read 4 Done Done Sent false
25806 8048 Acked Write 8 Done Done Sent false
25807 8049 Acked Write 7 Done Done Sent false
25808 8050 Acked Read 7 Done Done Sent false
25809 8051 Acked Read 7 Done Done Sent false
25810 8052 Acked Read 2 Done Done Sent false
25811 8053 Acked Write 3 Done Done Sent false
25812 8054 Acked Read 1 Done Done Sent false
25813 8055 Acked Read 3 Done Done Sent false
25814 8056 Acked Write 7 Done Done Sent false
25815 8057 Acked Write 7 Done Done Sent false
25816 8058 Acked Write 4 Done Done Sent false
25817 8059 Acked Read 5 Done Done Sent false
25818 8060 Acked Read 1 Done Done Sent false
25819 8061 Acked Write 8 Done Done Sent false
25820 8062 Acked Flush 0 Done Done Sent false
25821 8063 Acked Write 4 Done Done Sent false
25822 8064 Acked Read 7 Done Done Sent false
25823 8065 Acked Read 2 Done Done Sent false
25824 8066 Acked Write 3 Done Done Sent false
25825 8067 Acked Read 3 Done Done Sent false
25826 8068 Acked Read 3 Done Done Sent false
25827 8069 Acked Read 7 Done Done Sent false
25828 8070 Acked Flush 0 Done Done Sent false
25829 8071 Acked Write 9 Done Done Sent false
25830 8072 Acked Flush 0 Done Done Sent false
25831 8073 Acked Read 10 Done Done Sent false
25832 8074 Acked Write 4 Done Done Sent false
25833 8075 Acked Write 2 Done Done Sent false
25834 8076 Acked Read 4 Done Done Sent false
25835 8077 Acked Read 10 Done Done Sent false
25836 8078 Acked Flush 0 Done Done Sent false
25837 8079 Acked Read 10 Done Done Sent false
25838 8080 Acked Read 7 Done Done Sent false
25839 8081 Acked Flush 0 Done Done Sent false
25840 8082 Acked Write 3 Done Done Sent false
25841 8083 Acked Write 5 Done Done Sent false
25842 8084 Acked Write 8 Done Done Sent false
25843 8085 Acked Read 9 Done Done Sent false
25844 8086 Acked Read 1 Done Done Sent false
25845 8087 Acked Write 4 Done Done Sent false
25846 8088 Acked Read 3 Done Done Sent false
25847 8089 Acked Read 1 Done Done Sent false
25848 8090 Acked Write 5 Done Done Sent false
25849 8091 Acked Read 9 Done Done Sent false
25850 8092 Acked Write 7 Done Done Sent false
25851 8093 Acked Write 2 Done Done Sent false
25852 8094 Acked Read 5 Done Done Sent false
25853 8095 Acked Write 3 Done Done Sent false
25854 8096 Acked Read 8 Done Done Sent false
25855 8097 Acked Read 1 Done Done Sent false
25856 8098 Acked Write 10 Done Done Sent false
25857 8099 Acked Write 2 Done Done Sent false
25858 8100 Acked Flush 0 Done Done Sent false
25859 8101 Acked Read 7 Done Done Sent false
25860 8102 Acked Write 8 Done Done Sent false
25861 8103 Acked Read 2 Done Done Sent false
25862 8104 Acked Read 7 Done Done Sent false
25863 8105 Acked Read 1 Done Done Sent false
25864 8106 Acked Flush 0 Done Done Sent false
25865 8107 Acked Write 2 Done Done Sent false
25866 8108 Acked Write 6 Done Done Sent false
25867 8109 Acked Flush 0 Done Done Sent false
25868 8110 Acked Read 4 Done Done Sent false
25869 8111 Acked Write 9 Done Done Sent false
25870 8112 Acked Write 3 Done Done Sent false
25871 8113 Acked Write 6 Done Done Sent false
25872 8114 Acked Write 2 Done Done Sent false
25873 8115 Acked Write 2 Done Done Sent false
25874 8116 Acked Flush 0 Done Done Sent false
25875 8117 Acked Flush 0 Done Done Sent false
25876 8118 Acked Write 4 Done Done Sent false
25877 8119 Acked Read 4 Done Done Sent false
25878 8120 Acked Write 9 Done Done Sent false
25879 8121 Acked Read 10 Done Done Sent false
25880 8122 Acked Read 3 Done Done Sent false
25881 8123 Acked Read 8 Done Done Sent false
25882 8124 Acked Write 3 Done Done Sent false
25883 8125 Acked Read 7 Done Done Sent false
25884 8126 Acked Read 1 Done Done Sent false
25885 8127 Acked Write 9 Done Done Sent false
25886 8128 Acked Write 1 Done Done Sent false
25887 8129 Acked Write 6 Done Done Sent false
25888 8130 Acked Read 7 Done Done Sent false
25889 8131 Acked Write 1 Done Done Sent false
25890 8132 Acked Write 3 Done Done Sent false
25891 8133 Acked Write 3 Done Done Sent false
25892 8134 Acked Write 10 Done Done Sent false
25893 8135 Acked Write 2 Done Done Sent false
25894 8136 Acked Read 1 Done Done Sent false
25895 8137 Acked Flush 0 Done Done Sent false
25896 8138 Acked Write 2 Done Done Sent false
25897 8139 Acked Read 2 Done Done Sent false
25898 8140 Acked Read 8 Done Done Sent false
25899 8141 Acked Read 1 Done Done Sent false
25900 8142 Acked Read 9 Done Done Sent false
25901 8143 Acked Write 7 Done Done Sent false
25902 8144 Acked Write 7 Done Done Sent false
25903 8145 Acked Write 4 Done Done Sent false
25904 8146 Acked Read 6 Done Done Sent false
25905 8147 Acked Read 10 Done Done Sent false
25906 8148 Acked Write 5 Done Done Sent false
25907 8149 Acked Write 10 Done Done Sent false
25908 8150 Acked Flush 0 Done Done Sent false
25909 8151 Acked Read 4 Done Done Sent false
25910 8152 Acked Read 4 Done Done Sent false
25911 8153 Acked Write 2 Done Done Sent false
25912 8154 Acked Read 10 Done Done Sent false
25913 8155 Acked Flush 0 Done Done Sent false
25914 8156 Acked Read 9 Done Done Sent false
25915 8157 Acked Read 1 Done Done Sent false
25916 8158 Acked Flush 0 Done Done Sent false
25917 8159 Acked Write 2 Done Done Sent false
25918 8160 Acked Read 10 Done Done Sent false
25919 8161 Acked Write 4 Done Done Sent false
25920 8162 Acked Read 4 Done Done Sent false
25921 8163 Acked Read 2 Done Done Sent false
25922 8164 Acked Write 6 Done Done Sent false
25923 8165 Acked Write 9 Done Done Sent false
25924 8166 Acked Read 10 Done Done Sent false
25925 8167 Acked Read 2 Done Done Sent false
25926 8168 Acked Read 10 Done Done Sent false
25927 8169 Acked Write 5 Done Done Sent false
25928 8170 Acked Write 8 Done Done Sent false
25929 8171 Acked Read 3 Done Done Sent false
25930 8172 Acked Read 10 Done Done Sent false
25931 8173 Acked Write 6 Done Done Sent false
25932 8174 Acked Write 3 Done Done Sent false
25933 8175 Acked Read 4 Done Done Sent false
25934 8176 Acked Write 1 Done Done Sent false
25935 8177 Acked Write 5 Done Done Sent false
25936 8178 Acked Write 8 Done Done Sent false
25937 8179 Acked Read 5 Done Done Sent false
25938 8180 Acked Read 2 Done Done Sent false
25939 8181 Acked Read 9 Done Done Sent false
25940 8182 Acked Write 7 Done Done Sent false
25941 8183 Acked Flush 0 Done Done Sent false
25942 8184 Acked Read 2 Done Done Sent false
25943 8185 Acked Read 6 Done Done Sent false
25944 8186 Acked Read 7 Done Done Sent false
25945 8187 Acked Read 4 Done Done Sent false
25946 8188 Acked Write 8 Done Done Sent false
25947 8189 Acked Read 4 Done Done Sent false
25948 8190 Acked Read 1 Done Done Sent false
25949 8191 Acked Write 2 Done Done Sent false
25950 8192 Acked Read 9 Done Done Sent false
25951 8193 Acked Write 2 Done Done Sent false
25952 8194 Acked Write 1 Done Done Sent false
25953 8195 Acked Read 2 Done Done Sent false
25954 8196 Acked Write 5 Done Done Sent false
25955 8197 Acked Write 1 Done Done Sent false
25956 8198 Acked Read 2 Done Done Sent false
25957 8199 Acked Write 6 Done Done Sent false
25958 8200 Acked Read 7 Done Done Sent false
25959 8201 Acked Read 5 Done Done Sent false
25960 8202 Acked Read 7 Done Done Sent false
25961 8203 Acked Read 7 Done Done Sent false
25962 8204 Acked Flush 0 Done Done Sent false
25963 8205 Acked Read 3 Done Done Sent false
25964 8206 Acked Write 8 Done Done Sent false
25965 8207 Acked Read 2 Done Done Sent false
25966 8208 Acked Write 8 Done Done Sent false
25967 8209 Acked Read 1 Done Done Sent false
25968 8210 Acked Read 7 Done Done Sent false
25969 8211 Acked Write 3 Done Done Sent false
25970 8212 Acked Write 10 Done Done Sent false
25971 8213 Acked Read 1 Done Done Sent false
25972 8214 Acked Write 6 Done Done Sent false
25973 8215 Acked Read 4 Done Done Sent false
25974 8216 Acked Read 8 Done Done Sent false
25975 8217 Acked Read 7 Done Done Sent false
25976 8218 Acked Read 2 Done Done Sent false
25977 8219 Acked Read 8 Done Done Sent false
25978 8220 Acked Write 3 Done Done Sent false
25979 8221 Acked Read 7 Done Done Sent false
25980 8222 Acked Flush 0 Done Done Sent false
25981 8223 Acked Read 1 Done Done Sent false
25982 8224 Acked Flush 0 Done Done Sent false
25983 8225 Acked Write 1 Done Done Sent false
25984 8226 Acked Write 3 Done Done Sent false
25985 8227 Acked Read 2 Done Done Sent false
25986 8228 Acked Flush 0 Done Done Sent false
25987 8229 Acked Read 2 Done Done Sent false
25988 8230 Acked Read 1 Done Done Sent false
25989 8231 Acked Write 1 Done Done Sent false
25990 8232 Acked Read 6 Done Done Sent false
25991 8233 Acked Flush 0 Done Done Sent false
25992 8234 Acked Read 10 Done Done Sent false
25993 8235 Acked Read 5 Done Done Sent false
25994 8236 Acked Write 3 Done Done Sent false
25995 8237 Acked Read 4 Done Done Sent false
25996 8238 Acked Write 7 Done Done Sent false
25997 8239 Acked Read 6 Done Done Sent false
25998 8240 Acked Read 5 Done Done Sent false
25999 8241 Acked Write 10 Done Done Sent false
26000 8242 Acked Read 7 Done Done Sent false
26001 8243 Acked Write 8 Done Done Sent false
26002 8244 Acked Flush 0 Done Done Sent false
26003 STATES DS:0 DS:1 DS:2 TOTAL
26004 Sent 0 0 302 302
26005 Done 302 302 0 604
26006 Skipped 0 0 0 0
26007 Error 0 0 0 0
26008 Last Flush: Some(JobId(8244)) Some(JobId(8244)) Some(JobId(7942))
26009 Downstairs last five completed: 7942 7941 7940 7939 7938
26010 Upstairs last five completed: 8244 8243 8242 8241 8240
26011 CLIENT: Up:0 ds:302 act:2
260122025-01-23T18:53:20.699ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
260132025-01-23T18:53:20.699ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53912) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
260142025-01-23T18:53:20.700ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
260152025-01-23T18:53:20.701ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
260162025-01-23T18:53:20.701ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
260172025-01-23T18:53:20.701ZINFOcrucible: send last flush ID to this DS: Some(JobId(7942)) = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
260182025-01-23T18:53:20.701ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
260192025-01-23T18:53:20.701ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(7942)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
260202025-01-23T18:53:20.701ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(7942)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
26021 ----------------------------------------------------------------
26022 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26023 Downstairs last five completed: 8244 8243 8242 8241 8240
26024 Upstairs last five completed: 8244 8243 8242 8241 8240
26025 CLIENT: Up:0 ds:0 act:3
26026 Replay: All jobs finished
26027 Main task has work to do, go find it
26028 got dsc Stop(1)
26029 stop 1
26030 [8820] Got stop action so:false kr:false
26031 Jan 23 18:53:22.945 INFO request completed, latency_us: 324, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 51cd51d9-4db4-482b-93da-e73464003bd8, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
260322025-01-23T18:53:22.945ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
260332025-01-23T18:53:22.945ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
260342025-01-23T18:53:22.945ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
26035 Jan 23 18:53:22.945 INFO request completed, latency_us: 109, response_code: 200, uri: /state/cid/1, method: GET, req_id: 94bbdd5d-9fa9-488a-90bc-5baba8800601, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
260362025-01-23T18:53:22.945ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
26037 [8820] Exited with: ExitStatus(unix_wait_status(9))
26038 [8820][1] reports Exit
26039 Jan 23 18:53:26.948 INFO request completed, latency_us: 236, response_code: 200, uri: /state/cid/1, method: GET, req_id: 656fa37b-508b-4bee-a0ae-1a7c7075a4b0, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
26040 001/300 Write block 3096 len 20480 data: 4 4 5 6 7
26041 002/300 Write block 3342 len 20480 data: 4 6 6 7 7
26042 003/300 Read block 2491 len 12288
26043 004/300 Write block 2397 len 4096 data: 6
26044 005/300 Read block 1910 len 36864
26045 006/300 Write block 3917 len 20480 data: 7 6 7 9 8
26046 007/300 Write block 2818 len 12288 data: 2 2 3
26047 008/300 Read block 3741 len 20480
26048 009/300 Write block 2755 len 24576 data: 5 5 6 5 6 6
26049 010/300 Write block 4090 len 24576 data: 6 5 7 7 6 6
26050 011/300 Read block 3442 len 36864
26051 012/300 Write block 1345 len 24576 data: 7 5 4 3 2 3
26052 013/300 Write block 434 len 4096 data: 6
26053 014/300 Read block 2330 len 32768
26054 015/300 Read block 1851 len 4096
26055 016/300 Flush
26056 017/300 Read block 2523 len 8192
26057 018/300 Read block 4069 len 12288
26058 019/300 Read block 498 len 12288
26059 020/300 Read block 1407 len 4096
26060 021/300 Write block 486 len 20480 data: 5 7 7 6 7
26061 022/300 Write block 2961 len 36864 data: 4 5 5 6 6 5 6 6 5
26062 023/300 Read block 4222 len 12288
26063 024/300 Read block 4687 len 16384
26064 025/300 Read block 4256 len 12288
26065 026/300 Read block 2744 len 8192
26066 027/300 Write block 48 len 20480 data: 5 4 4 3 3
26067 028/300 Read block 2598 len 32768
26068 029/300 Write block 2580 len 32768 data: 9 9 7 5 5 7 8 5
26069 030/300 Read block 3858 len 24576
26070 031/300 Read block 1902 len 24576
26071 032/300 Read block 2288 len 28672
26072 033/300 Write block 4207 len 24576 data: 6 6 5 3 4 3
26073 034/300 Write block 1566 len 24576 data: 7 7 7 6 7 4
26074 035/300 Read block 1201 len 28672
26075 036/300 Write block 728 len 32768 data: 5 7 8 8 7 7 7 6
26076 037/300 Read block 2884 len 24576
26077 038/300 Flush
26078 039/300 Read block 3045 len 12288
26079 040/300 Read block 1051 len 40960
26080 041/300 Read block 2767 len 32768
26081 042/300 Write block 1384 len 28672 data: 6 6 5 7 7 6 7
26082 043/300 Read block 2124 len 40960
26083 044/300 Read block 1435 len 8192
26084 045/300 Read block 3087 len 16384
26085 046/300 Write block 100 len 24576 data: 4 4 5 6 5 4
26086 047/300 Write block 2940 len 32768 data: 4 3 4 4 4 5 5 5
26087 048/300 Flush
26088 049/300 Read block 2041 len 20480
26089 050/300 Read block 977 len 32768
26090 051/300 Write block 748 len 28672 data: 5 4 4 3 4 4 3
26091 052/300 Write block 3945 len 24576 data: 4 4 5 4 4 5
26092 053/300 Read block 1443 len 36864
26093 054/300 Write block 4422 len 40960 data: 6 7 9 6 5 6 7 6 6 7
26094 055/300 Write block 4772 len 40960 data: 2 3 3 3 2 3 6 6 5 4
26095 056/300 Read block 1300 len 12288
26096 057/300 Write block 3405 len 40960 data: 3 3 3 5 4 4 4 4 5 5
26097 058/300 Read block 801 len 8192
26098 059/300 Write block 1423 len 12288 data: 10 9 8
26099 060/300 Write block 539 len 24576 data: 5 5 6 8 7 6
26100 061/300 Read block 756 len 28672
26101 062/300 Write block 879 len 8192 data: 5 4
26102 063/300 Read block 3414 len 20480
26103 064/300 Read block 1248 len 36864
26104 065/300 Write block 3656 len 20480 data: 5 4 3 3 4
26105 066/300 Write block 1680 len 28672 data: 4 4 3 4 4 4 3
26106 067/300 Read block 4084 len 28672
26107 068/300 Read block 3212 len 12288
26108 069/300 Write block 1594 len 28672 data: 7 7 9 9 7 4 3
26109 070/300 Write block 3532 len 16384 data: 4 5 8 8
26110 071/300 Write block 2891 len 24576 data: 8 7 6 6 6 5
26111 072/300 Write block 2870 len 16384 data: 6 7 6 5
26112 073/300 Write block 2936 len 12288 data: 3 4 4
26113 074/300 Write block 3778 len 12288 data: 8 9 7
26114 075/300 Write block 2309 len 12288 data: 6 6 6
26115 076/300 Read block 1951 len 28672
26116 077/300 Read block 391 len 32768
26117 078/300 Write block 4277 len 24576 data: 4 4 6 5 5 4
26118 079/300 Flush
26119 080/300 Flush
26120 081/300 Write block 258 len 16384 data: 3 5 4 4
26121 082/300 Write block 3397 len 36864 data: 2 3 3 2 3 4 4 4 4
26122 083/300 Flush
26123 084/300 Write block 882 len 40960 data: 7 7 7 6 6 7 5 4 4 4
26124 085/300 Write block 77 len 8192 data: 7 8
26125 086/300 Read block 4586 len 8192
26126 087/300 Read block 3554 len 32768
26127 088/300 Flush
26128 089/300 Write block 993 len 12288 data: 4 2 4
26129 090/300 Write block 1090 len 32768 data: 7 6 5 6 6 8 8 7
26130 091/300 Write block 4733 len 24576 data: 6 6 5 4 4 3
26131 092/300 Read block 1420 len 16384
26132 093/300 Write block 839 len 16384 data: 4 4 5 6
26133 094/300 Read block 181 len 8192
26134 095/300 Read block 1997 len 16384
26135 096/300 Read block 801 len 16384
26136 097/300 Read block 1609 len 4096
26137 098/300 Write block 2593 len 32768 data: 4 4 4 4 4 4 4 5
26138 099/300 Write block 3821 len 36864 data: 5 4 5 5 5 5 6 6 7
26139 100/300 Read block 4666 len 28672
26140 101/300 Read block 4733 len 24576
26141 102/300 Write block 871 len 24576 data: 5 4 4 6 7 6
26142 103/300 Flush
26143 104/300 Read block 1823 len 24576
26144 105/300 Write block 3162 len 28672 data: 6 7 6 6 6 7 6
26145 106/300 Write block 1422 len 28672 data: 10 11 10 9 8 7 5
26146 107/300 Read block 194 len 24576
26147 108/300 Read block 3154 len 32768
26148 109/300 Flush
26149 110/300 Read block 1026 len 24576
26150 111/300 Write block 470 len 8192 data: 5 6
26151 112/300 Read block 2541 len 24576
26152 113/300 Read block 2941 len 36864
26153 114/300 Write block 2543 len 36864 data: 6 6 5 5 5 5 4 5 4
26154 115/300 Write block 2718 len 36864 data: 3 3 3 2 2 3 3 4 4
26155 116/300 Read block 4095 len 12288
26156 117/300 Flush
26157 118/300 Read block 1685 len 36864
26158 119/300 Write block 2949 len 28672 data: 6 6 6 6 6 6 6
26159 120/300 Write block 959 len 8192 data: 9 8
26160 121/300 Write block 1880 len 28672 data: 2 2 2 2 2 2 3
26161 122/300 Write block 4611 len 4096 data: 4
26162 123/300 Read block 2559 len 32768
26163 124/300 Flush
26164 125/300 Read block 2289 len 8192
26165 126/300 Read block 4329 len 32768
26166 127/300 Write block 2369 len 16384 data: 3 3 2 2
26167 128/300 Write block 3843 len 28672 data: 5 5 4 3 3 3 3
26168 129/300 Read block 2150 len 16384
26169 130/300 Flush
26170 131/300 Write block 3446 len 36864 data: 5 5 5 4 4 3 4 4 5
26171 132/300 Write block 3196 len 8192 data: 4 5
26172 133/300 Read block 4563 len 28672
26173 134/300 Read block 524 len 12288
26174 135/300 Write block 4766 len 24576 data: 4 3 3 2 2 2
26175 136/300 Write block 3482 len 8192 data: 7 6
26176 137/300 Write block 3871 len 4096 data: 5
26177 138/300 Write block 3606 len 20480 data: 4 4 4 4 5
26178 139/300 Write block 24 len 24576 data: 4 4 4 3 2 2
26179 140/300 Read block 4595 len 12288
26180 141/300 Read block 1319 len 28672
26181 142/300 Flush
26182 143/300 Write block 374 len 20480 data: 5 4 5 5 3
26183 144/300 Read block 4185 len 24576
26184 145/300 Read block 3135 len 12288
26185 146/300 Read block 3508 len 24576
26186 147/300 Read block 3514 len 40960
26187 148/300 Read block 1646 len 12288
26188 149/300 Flush
26189 150/300 Write block 1641 len 28672 data: 6 6 5 7 7 7 5
26190 151/300 Read block 3370 len 28672
26191 152/300 Read block 1384 len 24576
26192 153/300 Read block 1744 len 32768
26193 154/300 Read block 2715 len 20480
26194 155/300 Read block 2413 len 40960
26195 156/300 Read block 3515 len 40960
26196 157/300 Read block 3862 len 16384
26197 158/300 Write block 3228 len 32768 data: 5 6 7 8 9 9 9 9
26198 159/300 Read block 85 len 8192
26199 160/300 Read block 1826 len 32768
26200 161/300 Write block 3676 len 20480 data: 4 4 5 4 5
26201 162/300 Write block 1455 len 20480 data: 3 2 2 2 2
26202 163/300 Read block 1675 len 4096
26203 164/300 Read block 1921 len 40960
26204 165/300 Write block 2512 len 40960 data: 2 2 2 2 3 4 6 6 7 7
26205 166/300 Read block 1995 len 16384
26206 167/300 Read block 4960 len 8192
26207 168/300 Read block 817 len 36864
26208 169/300 Write block 243 len 20480 data: 5 4 6 6 9
26209 170/300 Read block 4068 len 8192
26210 171/300 Write block 1389 len 28672 data: 7 8 6 4 6 6 7
26211 172/300 Write block 4812 len 20480 data: 7 7 5 4 4
26212 173/300 Write block 4261 len 24576 data: 4 4 4 3 4 5
26213 174/300 Write block 4414 len 24576 data: 4 4 6 6 6 5
26214 175/300 Read block 2547 len 16384
26215 176/300 Read block 4597 len 4096
26216 177/300 Write block 3868 len 16384 data: 5 4 4 6
26217 178/300 Write block 4793 len 12288 data: 4 4 4
26218 179/300 Write block 3626 len 28672 data: 7 6 4 5 4 5 5
26219 180/300 Read block 1226 len 20480
26220 181/300 Write block 4053 len 12288 data: 4 4 4
26221 182/300 Write block 4293 len 16384 data: 5 5 5 4
26222 183/300 Flush
26223 184/300 Write block 2786 len 12288 data: 4 3 5
26224 185/300 Read block 761 len 32768
26225 186/300 Write block 3486 len 8192 data: 7 5
26226 187/300 Write block 1073 len 36864 data: 10 8 8 8 5 5 6 6 7
26227 188/300 Write block 557 len 16384 data: 7 8 8 6
26228 189/300 Read block 1589 len 12288
26229 190/300 Write block 1273 len 36864 data: 3 3 4 4 4 5 5 4 5
26230 191/300 Read block 1615 len 8192
26231 192/300 Write block 2732 len 12288 data: 6 6 6
26232 193/300 Read block 3749 len 24576
26233 194/300 Flush
26234 195/300 Read block 1013 len 28672
26235 196/300 Write block 4967 len 40960 data: 5 5 5 4 3 3 2 3 2 3
26236 197/300 Flush
26237 198/300 Read block 2479 len 8192
26238 199/300 Read block 2146 len 36864
26239 200/300 Flush
26240 201/300 Read block 2895 len 32768
26241 202/300 Write block 4617 len 8192 data: 2 2
26242 203/300 Read block 3805 len 28672
26243 204/300 Write block 4570 len 4096 data: 4
26244 205/300 Read block 251 len 40960
26245 206/300 Write block 1064 len 24576 data: 7 7 7 7 6 10
26246 207/300 Write block 3174 len 20480 data: 4 3 3 3 3
26247 208/300 Write block 174 len 24576 data: 6 7 9 7 10 8
26248 209/300 Write block 4888 len 4096 data: 6
26249 210/300 Read block 2549 len 16384
26250 211/300 Read block 1823 len 8192
26251 212/300 Write block 1368 len 20480 data: 2 2 3 3 3
26252 213/300 Write block 4093 len 8192 data: 8 7
26253 214/300 Write block 1021 len 4096 data: 11
26254 215/300 Read block 4753 len 12288
26255 216/300 Read block 507 len 4096
26256 217/300 Read block 154 len 8192
26257 218/300 Write block 4201 len 20480 data: 4 4 5 6 5
26258 219/300 Read block 681 len 20480
26259 220/300 Flush
26260 221/300 Write block 621 len 8192 data: 6 6
26261 222/300 Write block 1582 len 16384 data: 5 6 6 4
26262 223/300 Read block 2332 len 16384
26263 224/300 Write block 2177 len 8192 data: 4 4
26264 225/300 Read block 2591 len 28672
26265 226/300 Write block 2349 len 4096 data: 5
26266 227/300 Read block 487 len 16384
26267 228/300 Write block 556 len 12288 data: 5 8 9
26268 229/300 Write block 2625 len 40960 data: 9 9 7 9 9 9 8 10 8 5
26269 230/300 Read block 1033 len 20480
26270 231/300 Flush
26271 232/300 Read block 1956 len 28672
26272 233/300 Write block 3143 len 32768 data: 3 4 5 4 4 3 4 4
26273 234/300 Write block 3192 len 32768 data: 4 5 5 4 5 6 3 3
26274 235/300 Write block 2427 len 36864 data: 6 5 5 6 4 5 6 5 5
26275 236/300 Flush
26276 237/300 Read block 2163 len 20480
26277 238/300 Read block 2969 len 24576
26278 239/300 Read block 1827 len 4096
26279 240/300 Read block 3099 len 32768
26280 241/300 Flush
26281 242/300 Read block 3177 len 36864
26282 243/300 Write block 3593 len 24576 data: 6 6 7 5 6 6
26283 244/300 Write block 1247 len 28672 data: 7 5 6 6 6 7 7
26284 245/300 Read block 2968 len 40960
26285 246/300 Write block 1745 len 36864 data: 4 4 3 3 3 4 4 3 3
26286 247/300 Read block 2252 len 4096
26287 248/300 Read block 981 len 12288
26288 249/300 Read block 1065 len 8192
26289 250/300 Write block 4593 len 12288 data: 3 4 4
26290 251/300 Read block 4256 len 20480
26291 252/300 Read block 1018 len 36864
26292 253/300 Read block 3963 len 4096
26293 254/300 Write block 3198 len 8192 data: 4 4
26294 255/300 Write block 4384 len 4096 data: 4
26295 256/300 Write block 647 len 12288 data: 4 3 5
26296 257/300 Write block 2554 len 4096 data: 4
26297 258/300 Read block 1864 len 16384
26298 259/300 Write block 4065 len 32768 data: 7 7 7 6 5 5 6 6
26299 260/300 Read block 783 len 16384
26300 261/300 Read block 3224 len 36864
26301 262/300 Read block 1595 len 32768
26302 263/300 Write block 1950 len 40960 data: 6 6 6 5 6 7 6 6 6 6
26303 264/300 Write block 4107 len 28672 data: 4 3 3 4 5 5 4
26304 265/300 Read block 3328 len 12288
26305 266/300 Read block 250 len 20480
26306 267/300 Read block 4602 len 20480
26307 268/300 Write block 3022 len 12288 data: 7 6 7
26308 269/300 Flush
26309 270/300 Read block 1420 len 12288
26310 271/300 Read block 4562 len 36864
26311 272/300 Write block 3212 len 32768 data: 2 2 2 3 3 3 3 4
26312 273/300 Read block 1650 len 40960
26313 274/300 Write block 4403 len 8192 data: 5 3
26314 275/300 Read block 2574 len 28672
26315 276/300 Read block 4651 len 28672
26316 277/300 Read block 2895 len 36864
26317 278/300 Write block 739 len 20480 data: 5 5 6 6 6
26318 279/300 Write block 3342 len 12288 data: 5 7 7
26319 280/300 Write block 2788 len 40960 data: 6 5 5 4 5 5 5 6 5 6
26320 281/300 Read block 1886 len 32768
26321 282/300 Write block 2885 len 4096 data: 8
26322 283/300 Write block 2017 len 8192 data: 7 6
26323 284/300 Read block 3816 len 36864
26324 285/300 Write block 1352 len 32768 data: 4 4 4 4 5 4 5 6
26325 286/300 Flush
26326 287/300 Write block 899 len 20480 data: 8 7 8 7 7
26327 288/300 Read block 1829 len 4096
26328 289/300 Write block 1265 len 24576 data: 7 5 5 3 3 3
26329 290/300 Write block 3373 len 4096 data: 4
26330 291/300 Read block 3126 len 16384
26331 292/300 Write block 1298 len 16384 data: 4 4 6 5
26332 293/300 Flush
26333 294/300 Write block 4964 len 16384 data: 9 8 6 6
26334 295/300 Flush
26335 296/300 Read block 1964 len 32768
26336 297/300 Read block 3673 len 32768
26337 298/300 Write block 3048 len 40960 data: 11 9 9 6 4 4 4 4 4 6
26338 299/300 Read block 4159 len 16384
26339 300/300 Read block 17 len 4096
26340 301/300 Write block 4164 len 40960 data: 10 8 8 10 10 9 10 11 11 8
26341 Main task has work to do, go find it
26342 got dsc Start(1)
26343 start 1
26344 Jan 23 18:53:27.178 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/1, method: POST, req_id: 86e4c57e-2d39-4230-880e-5cbc94a32aaa, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
26345 Replay: started 1, returned:Ok(())
26346 ----------------------------------------------------------------
26347 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26348 States: Active Offline Active
26349 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26350 8245 Acked Write 5 Done Sent Done false
26351 8246 Acked Write 5 Done Sent Done false
26352 8247 Acked Read 3 Done Sent Done false
26353 8248 Acked Write 1 Done Sent Done false
26354 8249 Acked Read 9 Done Sent Done false
26355 8250 Acked Write 5 Done Sent Done false
26356 8251 Acked Write 3 Done Sent Done false
26357 8252 Acked Read 5 Done Sent Done false
26358 8253 Acked Write 6 Done Sent Done false
26359 8254 Acked Write 6 Done Sent Done false
26360 8255 Acked Read 9 Done Sent Done false
26361 8256 Acked Write 6 Done Sent Done false
26362 8257 Acked Write 1 Done Sent Done false
26363 8258 Acked Read 8 Done Sent Done false
26364 8259 Acked Read 1 Done Sent Done false
26365 8260 Acked Flush 0 Done Sent Done false
26366 8261 Acked Read 2 Done Sent Done false
26367 8262 Acked Read 3 Done Sent Done false
26368 8263 Acked Read 3 Done Sent Done false
26369 8264 Acked Read 1 Done Sent Done false
26370 8265 Acked Write 5 Done Sent Done false
26371 8266 Acked Write 9 Done Sent Done false
26372 8267 Acked Read 3 Done Sent Done false
26373 8268 Acked Read 4 Done Sent Done false
26374 8269 Acked Read 3 Done Sent Done false
26375 8270 Acked Read 2 Done Sent Done false
26376 8271 Acked Write 5 Done Sent Done false
26377 8272 Acked Read 8 Done Sent Done false
26378 8273 Acked Write 8 Done Sent Done false
26379 8274 Acked Read 6 Done Sent Done false
26380 8275 Acked Read 6 Done Sent Done false
26381 8276 Acked Read 7 Done Sent Done false
26382 8277 Acked Write 6 Done Sent Done false
26383 8278 Acked Write 6 Done Sent Done false
26384 8279 Acked Read 7 Done Sent Done false
26385 8280 Acked Write 8 Done Sent Done false
26386 8281 Acked Read 6 Done Sent Done false
26387 8282 Acked Flush 0 Done Sent Done false
26388 8283 Acked Read 3 Done Sent Done false
26389 8284 Acked Read 10 Done Sent Done false
26390 8285 Acked Read 8 Done Sent Done false
26391 8286 Acked Write 7 Done Sent Done false
26392 8287 Acked Read 10 Done Sent Done false
26393 8288 Acked Read 2 Done Sent Done false
26394 8289 Acked Read 4 Done Sent Done false
26395 8290 Acked Write 6 Done Sent Done false
26396 8291 Acked Write 8 Done Sent Done false
26397 8292 Acked Flush 0 Done Sent Done false
26398 8293 Acked Read 5 Done Sent Done false
26399 8294 Acked Read 8 Done Sent Done false
26400 8295 Acked Write 7 Done Sent Done false
26401 8296 Acked Write 6 Done Sent Done false
26402 8297 Acked Read 9 Done Sent Done false
26403 8298 Acked Write 10 Done Sent Done false
26404 8299 Acked Write 10 Done Sent Done false
26405 8300 Acked Read 3 Done Sent Done false
26406 8301 Acked Write 10 Done Sent Done false
26407 8302 Acked Read 2 Done Sent Done false
26408 8303 Acked Write 3 Done Sent Done false
26409 8304 Acked Write 6 Done Sent Done false
26410 8305 Acked Read 7 Done Sent Done false
26411 8306 Acked Write 2 Done Sent Done false
26412 8307 Acked Read 5 Done Sent Done false
26413 8308 Acked Read 9 Done Sent Done false
26414 8309 Acked Write 5 Done Sent Done false
26415 8310 Acked Write 7 Done Sent Done false
26416 8311 Acked Read 7 Done Sent Done false
26417 8312 Acked Read 3 Done Sent Done false
26418 8313 Acked Write 7 Done Sent Done false
26419 8314 Acked Write 4 Done Sent Done false
26420 8315 Acked Write 6 Done Sent Done false
26421 8316 Acked Write 4 Done Sent Done false
26422 8317 Acked Write 3 Done Sent Done false
26423 8318 Acked Write 3 Done Sent Done false
26424 8319 Acked Write 3 Done Sent Done false
26425 8320 Acked Read 7 Done Sent Done false
26426 8321 Acked Read 8 Done Sent Done false
26427 8322 Acked Write 6 Done Sent Done false
26428 8323 Acked Flush 0 Done Sent Done false
26429 8324 Acked Flush 0 Done Sent Done false
26430 8325 Acked Write 4 Done Sent Done false
26431 8326 Acked Write 9 Done Sent Done false
26432 8327 Acked Flush 0 Done Sent Done false
26433 8328 Acked Write 10 Done Sent Done false
26434 8329 Acked Write 2 Done Sent Done false
26435 8330 Acked Read 2 Done Sent Done false
26436 8331 Acked Read 8 Done Sent Done false
26437 8332 Acked Flush 0 Done Sent Done false
26438 8333 Acked Write 3 Done Sent Done false
26439 8334 Acked Write 8 Done Sent Done false
26440 8335 Acked Write 6 Done Sent Done false
26441 8336 Acked Read 4 Done Sent Done false
26442 8337 Acked Write 4 Done Sent Done false
26443 8338 Acked Read 2 Done Sent Done false
26444 8339 Acked Read 4 Done Sent Done false
26445 8340 Acked Read 4 Done Sent Done false
26446 8341 Acked Read 1 Done Sent Done false
26447 8342 Acked Write 8 Done Sent Done false
26448 8343 Acked Write 9 Done Sent Done false
26449 8344 Acked Read 7 Done Sent Done false
26450 8345 Acked Read 6 Done Sent Done false
26451 8346 Acked Write 6 Done Sent Done false
26452 8347 Acked Flush 0 Done Sent Done false
26453 8348 Acked Read 6 Done Sent Done false
26454 8349 Acked Write 7 Done Sent Done false
26455 8350 Acked Write 7 Done Sent Done false
26456 8351 Acked Read 6 Done Sent Done false
26457 8352 Acked Read 8 Done Sent Done false
26458 8353 Acked Flush 0 Done Sent Done false
26459 8354 Acked Read 6 Done Sent Done false
26460 8355 Acked Write 2 Done Sent Done false
26461 8356 Acked Read 6 Done Sent Done false
26462 8357 Acked Read 9 Done Sent Done false
26463 8358 Acked Write 9 Done Sent Done false
26464 8359 Acked Write 9 Done Sent Done false
26465 8360 Acked Read 3 Done Sent Done false
26466 8361 Acked Flush 0 Done Sent Done false
26467 8362 Acked Read 9 Done Sent Done false
26468 8363 Acked Write 7 Done Sent Done false
26469 8364 Acked Write 2 Done Sent Done false
26470 8365 Acked Write 7 Done Sent Done false
26471 8366 Acked Write 1 Done Sent Done false
26472 8367 Acked Read 8 Done Sent Done false
26473 8368 Acked Flush 0 Done Sent Done false
26474 8369 Acked Read 2 Done Sent Done false
26475 8370 Acked Read 8 Done Sent Done false
26476 8371 Acked Write 4 Done Sent Done false
26477 8372 Acked Write 7 Done Sent Done false
26478 8373 Acked Read 4 Done Sent Done false
26479 8374 Acked Flush 0 Done Sent Done false
26480 8375 Acked Write 9 Done Sent Done false
26481 8376 Acked Write 2 Done Sent Done false
26482 8377 Acked Read 7 Done Sent Done false
26483 8378 Acked Read 3 Done Sent Done false
26484 8379 Acked Write 6 Done Sent Done false
26485 8380 Acked Write 2 Done Sent Done false
26486 8381 Acked Write 1 Done Sent Done false
26487 8382 Acked Write 5 Done Sent Done false
26488 8383 Acked Write 6 Done Sent Done false
26489 8384 Acked Read 3 Done Sent Done false
26490 8385 Acked Read 7 Done Sent Done false
26491 8386 Acked Flush 0 Done Sent Done false
26492 8387 Acked Write 5 Done Sent Done false
26493 8388 Acked Read 6 Done Sent Done false
26494 8389 Acked Read 3 Done Sent Done false
26495 8390 Acked Read 6 Done Sent Done false
26496 8391 Acked Read 10 Done Sent Done false
26497 8392 Acked Read 3 Done Sent Done false
26498 8393 Acked Flush 0 Done Sent Done false
26499 8394 Acked Write 7 Done Sent Done false
26500 8395 Acked Read 7 Done Sent Done false
26501 8396 Acked Read 6 Done Sent Done false
26502 8397 Acked Read 8 Done Sent Done false
26503 8398 Acked Read 5 Done Sent Done false
26504 8399 Acked Read 10 Done Sent Done false
26505 8400 Acked Read 10 Done Sent Done false
26506 8401 Acked Read 4 Done Sent Done false
26507 8402 Acked Write 8 Done Sent Done false
26508 8403 Acked Read 2 Done Sent Done false
26509 8404 Acked Read 8 Done Sent Done false
26510 8405 Acked Write 5 Done Sent Done false
26511 8406 Acked Write 5 Done Sent Done false
26512 8407 Acked Read 1 Done Sent Done false
26513 8408 Acked Read 10 Done Sent Done false
26514 8409 Acked Write 10 Done Sent Done false
26515 8410 Acked Read 4 Done Sent Done false
26516 8411 Acked Read 2 Done Sent Done false
26517 8412 Acked Read 9 Done Sent Done false
26518 8413 Acked Write 5 Done Sent Done false
26519 8414 Acked Read 2 Done Sent Done false
26520 8415 Acked Write 7 Done Sent Done false
26521 8416 Acked Write 5 Done Sent Done false
26522 8417 Acked Write 6 Done Sent Done false
26523 8418 Acked Write 6 Done Sent Done false
26524 8419 Acked Read 4 Done Sent Done false
26525 8420 Acked Read 1 Done Sent Done false
26526 8421 Acked Write 4 Done Sent Done false
26527 8422 Acked Write 3 Done Sent Done false
26528 8423 Acked Write 7 Done Sent Done false
26529 8424 Acked Read 5 Done Sent Done false
26530 8425 Acked Write 3 Done Sent Done false
26531 8426 Acked Write 4 Done Sent Done false
26532 8427 Acked Flush 0 Done Sent Done false
26533 8428 Acked Write 3 Done Sent Done false
26534 8429 Acked Read 8 Done Sent Done false
26535 8430 Acked Write 2 Done Sent Done false
26536 8431 Acked Write 9 Done Sent Done false
26537 8432 Acked Write 4 Done Sent Done false
26538 8433 Acked Read 3 Done Sent Done false
26539 8434 Acked Write 9 Done Sent Done false
26540 8435 Acked Read 2 Done Sent Done false
26541 8436 Acked Write 3 Done Sent Done false
26542 8437 Acked Read 6 Done Sent Done false
26543 8438 Acked Flush 0 Done Sent Done false
26544 8439 Acked Read 7 Done Sent Done false
26545 8440 Acked Write 10 Done Sent Done false
26546 8441 Acked Flush 0 Done Sent Done false
26547 8442 Acked Read 2 Done Sent Done false
26548 8443 Acked Read 9 Done Sent Done false
26549 8444 Acked Flush 0 Done Sent Done false
26550 8445 Acked Read 8 Done Sent Done false
26551 8446 Acked Write 2 Done Sent Done false
26552 8447 Acked Read 7 Done Sent Done false
26553 8448 Acked Write 1 Done Sent Done false
26554 8449 Acked Read 10 Done Sent Done false
26555 8450 Acked Write 6 Done Sent Done false
26556 8451 Acked Write 5 Done Sent Done false
26557 8452 Acked Write 6 Done Sent Done false
26558 8453 Acked Write 1 Done Sent Done false
26559 8454 Acked Read 4 Done Sent Done false
26560 8455 Acked Read 2 Done Sent Done false
26561 8456 Acked Write 5 Done Sent Done false
26562 8457 Acked Write 2 Done Sent Done false
26563 8458 Acked Write 1 Done Sent Done false
26564 8459 Acked Read 3 Done Sent Done false
26565 8460 Acked Read 1 Done Sent Done false
26566 8461 Acked Read 2 Done Sent Done false
26567 8462 Acked Write 5 Done Sent Done false
26568 8463 Acked Read 5 Done Sent Done false
26569 8464 Acked Flush 0 Done Sent Done false
26570 8465 Acked Write 2 Done Sent Done false
26571 8466 Acked Write 4 Done Sent Done false
26572 8467 Acked Read 4 Done Sent Done false
26573 8468 Acked Write 2 Done Sent Done false
26574 8469 Acked Read 7 Done Sent Done false
26575 8470 Acked Write 1 Done Sent Done false
26576 8471 Acked Read 4 Done Sent Done false
26577 8472 Acked Write 3 Done Sent Done false
26578 8473 Acked Write 10 Done Sent Done false
26579 8474 Acked Read 5 Done Sent Done false
26580 8475 Acked Flush 0 Done Sent Done false
26581 8476 Acked Read 7 Done Sent Done false
26582 8477 Acked Write 8 Done Sent Done false
26583 8478 Acked Write 8 Done Sent Done false
26584 8479 Acked Write 9 Done Sent Done false
26585 8480 Acked Flush 0 Done Sent Done false
26586 8481 Acked Read 5 Done Sent Done false
26587 8482 Acked Read 6 Done Sent Done false
26588 8483 Acked Read 1 Done Sent Done false
26589 8484 Acked Read 8 Done Sent Done false
26590 8485 Acked Flush 0 Done Sent Done false
26591 8486 Acked Read 9 Done Sent Done false
26592 8487 Acked Write 6 Done Sent Done false
26593 8488 Acked Write 7 Done Sent Done false
26594 8489 Acked Read 10 Done Sent Done false
26595 8490 Acked Write 9 Done Sent Done false
26596 8491 Acked Read 1 Done Sent Done false
26597 8492 Acked Read 3 Done Sent Done false
26598 8493 Acked Read 2 Done Sent Done false
26599 8494 Acked Write 3 Done Sent Done false
26600 8495 Acked Read 5 Done Sent Done false
26601 8496 Acked Read 9 Done Sent Done false
26602 8497 Acked Read 1 Done Sent Done false
26603 8498 Acked Write 2 Done Sent Done false
26604 8499 Acked Write 1 Done Sent Done false
26605 8500 Acked Write 3 Done Sent Done false
26606 8501 Acked Write 1 Done Sent Done false
26607 8502 Acked Read 4 Done Sent Done false
26608 8503 Acked Write 8 Done Sent Done false
26609 8504 Acked Read 4 Done Sent Done false
26610 8505 Acked Read 9 Done Sent Done false
26611 8506 Acked Read 8 Done Sent Done false
26612 8507 Acked Write 10 Done Sent Done false
26613 8508 Acked Write 7 Done Sent Done false
26614 8509 Acked Read 3 Done Sent Done false
26615 8510 Acked Read 5 Done Sent Done false
26616 8511 Acked Read 5 Done Sent Done false
26617 8512 Acked Write 3 Done Sent Done false
26618 8513 Acked Flush 0 Done Sent Done false
26619 8514 Acked Read 3 Done Sent Done false
26620 8515 Acked Read 9 Done Sent Done false
26621 8516 Acked Write 8 Done Sent Done false
26622 8517 Acked Read 10 Done Sent Done false
26623 8518 Acked Write 2 Done Sent Done false
26624 8519 Acked Read 7 Done Sent Done false
26625 8520 Acked Read 7 Done Sent Done false
26626 8521 Acked Read 9 Done Sent Done false
26627 8522 Acked Write 5 Done Sent Done false
26628 8523 Acked Write 3 Done Sent Done false
26629 8524 Acked Write 10 Done Sent Done false
26630 8525 Acked Read 8 Done Sent Done false
26631 8526 Acked Write 1 Done Sent Done false
26632 8527 Acked Write 2 Done Sent Done false
26633 8528 Acked Read 9 Done Sent Done false
26634 8529 Acked Write 8 Done Sent Done false
26635 8530 Acked Flush 0 Done Sent Done false
26636 8531 Acked Write 5 Done Sent Done false
26637 8532 Acked Read 1 Done Sent Done false
26638 8533 Acked Write 6 Done Sent Done false
26639 8534 Acked Write 1 Done Sent Done false
26640 8535 Acked Read 4 Done Sent Done false
26641 8536 Acked Write 4 Done Sent Done false
26642 8537 Acked Flush 0 Done Sent Done false
26643 8538 Acked Write 4 Done Sent Done false
26644 8539 Acked Flush 0 Done Sent Done false
26645 8540 Acked Read 8 Done Sent Done false
26646 8541 Acked Read 8 Done Sent Done false
26647 8542 Acked Write 10 Done Sent Done false
26648 8543 Acked Read 4 Done Sent Done false
26649 8544 Acked Read 1 Done Sent Done false
26650 8545 Acked Write 10 Done Sent Sent false
26651 STATES DS:0 DS:1 DS:2 TOTAL
26652 Sent 0 301 1 302
26653 Done 301 0 300 601
26654 Skipped 0 0 0 0
26655 Error 0 0 0 0
26656 Last Flush: Some(JobId(8539)) Some(JobId(8244)) Some(JobId(8539))
26657 Downstairs last five completed: 8244 8243 8242 8241 8240
26658 Upstairs last five completed: 8545 8544 8543 8542 8541
26659 CLIENT: Up:0 ds:301 act:2
26660 ----------------------------------------------------------------
26661 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
26662 States: Active Offline Active
26663 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26664 8245 Acked Write 5 Done Sent Done false
26665 8246 Acked Write 5 Done Sent Done false
26666 8247 Acked Read 3 Done Sent Done false
26667 8248 Acked Write 1 Done Sent Done false
26668 8249 Acked Read 9 Done Sent Done false
26669 8250 Acked Write 5 Done Sent Done false
26670 8251 Acked Write 3 Done Sent Done false
26671 8252 Acked Read 5 Done Sent Done false
26672 8253 Acked Write 6 Done Sent Done false
26673 8254 Acked Write 6 Done Sent Done false
26674 8255 Acked Read 9 Done Sent Done false
26675 8256 Acked Write 6 Done Sent Done false
26676 8257 Acked Write 1 Done Sent Done false
26677 8258 Acked Read 8 Done Sent Done false
26678 8259 Acked Read 1 Done Sent Done false
26679 8260 Acked Flush 0 Done Sent Done false
26680 8261 Acked Read 2 Done Sent Done false
26681 8262 Acked Read 3 Done Sent Done false
26682 8263 Acked Read 3 Done Sent Done false
26683 8264 Acked Read 1 Done Sent Done false
26684 8265 Acked Write 5 Done Sent Done false
26685 8266 Acked Write 9 Done Sent Done false
26686 8267 Acked Read 3 Done Sent Done false
26687 8268 Acked Read 4 Done Sent Done false
26688 8269 Acked Read 3 Done Sent Done false
26689 8270 Acked Read 2 Done Sent Done false
26690 8271 Acked Write 5 Done Sent Done false
26691 8272 Acked Read 8 Done Sent Done false
26692 8273 Acked Write 8 Done Sent Done false
26693 8274 Acked Read 6 Done Sent Done false
26694 8275 Acked Read 6 Done Sent Done false
26695 8276 Acked Read 7 Done Sent Done false
26696 8277 Acked Write 6 Done Sent Done false
26697 8278 Acked Write 6 Done Sent Done false
26698 8279 Acked Read 7 Done Sent Done false
26699 8280 Acked Write 8 Done Sent Done false
26700 8281 Acked Read 6 Done Sent Done false
26701 8282 Acked Flush 0 Done Sent Done false
26702 8283 Acked Read 3 Done Sent Done false
26703 8284 Acked Read 10 Done Sent Done false
26704 8285 Acked Read 8 Done Sent Done false
26705 8286 Acked Write 7 Done Sent Done false
26706 8287 Acked Read 10 Done Sent Done false
26707 8288 Acked Read 2 Done Sent Done false
26708 8289 Acked Read 4 Done Sent Done false
26709 8290 Acked Write 6 Done Sent Done false
26710 8291 Acked Write 8 Done Sent Done false
26711 8292 Acked Flush 0 Done Sent Done false
26712 8293 Acked Read 5 Done Sent Done false
26713 8294 Acked Read 8 Done Sent Done false
26714 8295 Acked Write 7 Done Sent Done false
26715 8296 Acked Write 6 Done Sent Done false
26716 8297 Acked Read 9 Done Sent Done false
26717 8298 Acked Write 10 Done Sent Done false
26718 8299 Acked Write 10 Done Sent Done false
26719 8300 Acked Read 3 Done Sent Done false
26720 8301 Acked Write 10 Done Sent Done false
26721 8302 Acked Read 2 Done Sent Done false
26722 8303 Acked Write 3 Done Sent Done false
26723 8304 Acked Write 6 Done Sent Done false
26724 8305 Acked Read 7 Done Sent Done false
26725 8306 Acked Write 2 Done Sent Done false
26726 8307 Acked Read 5 Done Sent Done false
26727 8308 Acked Read 9 Done Sent Done false
26728 8309 Acked Write 5 Done Sent Done false
26729 8310 Acked Write 7 Done Sent Done false
26730 8311 Acked Read 7 Done Sent Done false
26731 8312 Acked Read 3 Done Sent Done false
26732 8313 Acked Write 7 Done Sent Done false
26733 8314 Acked Write 4 Done Sent Done false
26734 8315 Acked Write 6 Done Sent Done false
26735 8316 Acked Write 4 Done Sent Done false
26736 8317 Acked Write 3 Done Sent Done false
26737 8318 Acked Write 3 Done Sent Done false
26738 8319 Acked Write 3 Done Sent Done false
26739 8320 Acked Read 7 Done Sent Done false
26740 8321 Acked Read 8 Done Sent Done false
26741 8322 Acked Write 6 Done Sent Done false
26742 8323 Acked Flush 0 Done Sent Done false
26743 8324 Acked Flush 0 Done Sent Done false
26744 8325 Acked Write 4 Done Sent Done false
26745 8326 Acked Write 9 Done Sent Done false
26746 8327 Acked Flush 0 Done Sent Done false
26747 8328 Acked Write 10 Done Sent Done false
26748 8329 Acked Write 2 Done Sent Done false
26749 8330 Acked Read 2 Done Sent Done false
26750 8331 Acked Read 8 Done Sent Done false
26751 8332 Acked Flush 0 Done Sent Done false
26752 8333 Acked Write 3 Done Sent Done false
26753 8334 Acked Write 8 Done Sent Done false
26754 8335 Acked Write 6 Done Sent Done false
26755 8336 Acked Read 4 Done Sent Done false
26756 8337 Acked Write 4 Done Sent Done false
26757 8338 Acked Read 2 Done Sent Done false
26758 8339 Acked Read 4 Done Sent Done false
26759 8340 Acked Read 4 Done Sent Done false
26760 8341 Acked Read 1 Done Sent Done false
26761 8342 Acked Write 8 Done Sent Done false
26762 8343 Acked Write 9 Done Sent Done false
26763 8344 Acked Read 7 Done Sent Done false
26764 8345 Acked Read 6 Done Sent Done false
26765 8346 Acked Write 6 Done Sent Done false
26766 8347 Acked Flush 0 Done Sent Done false
26767 8348 Acked Read 6 Done Sent Done false
26768 8349 Acked Write 7 Done Sent Done false
26769 8350 Acked Write 7 Done Sent Done false
26770 8351 Acked Read 6 Done Sent Done false
26771 8352 Acked Read 8 Done Sent Done false
26772 8353 Acked Flush 0 Done Sent Done false
26773 8354 Acked Read 6 Done Sent Done false
26774 8355 Acked Write 2 Done Sent Done false
26775 8356 Acked Read 6 Done Sent Done false
26776 8357 Acked Read 9 Done Sent Done false
26777 8358 Acked Write 9 Done Sent Done false
26778 8359 Acked Write 9 Done Sent Done false
26779 8360 Acked Read 3 Done Sent Done false
26780 8361 Acked Flush 0 Done Sent Done false
26781 8362 Acked Read 9 Done Sent Done false
26782 8363 Acked Write 7 Done Sent Done false
26783 8364 Acked Write 2 Done Sent Done false
26784 8365 Acked Write 7 Done Sent Done false
26785 8366 Acked Write 1 Done Sent Done false
26786 8367 Acked Read 8 Done Sent Done false
26787 8368 Acked Flush 0 Done Sent Done false
26788 8369 Acked Read 2 Done Sent Done false
26789 8370 Acked Read 8 Done Sent Done false
26790 8371 Acked Write 4 Done Sent Done false
26791 8372 Acked Write 7 Done Sent Done false
26792 8373 Acked Read 4 Done Sent Done false
26793 8374 Acked Flush 0 Done Sent Done false
26794 8375 Acked Write 9 Done Sent Done false
26795 8376 Acked Write 2 Done Sent Done false
26796 8377 Acked Read 7 Done Sent Done false
26797 8378 Acked Read 3 Done Sent Done false
26798 8379 Acked Write 6 Done Sent Done false
26799 8380 Acked Write 2 Done Sent Done false
26800 8381 Acked Write 1 Done Sent Done false
26801 8382 Acked Write 5 Done Sent Done false
26802 8383 Acked Write 6 Done Sent Done false
26803 8384 Acked Read 3 Done Sent Done false
26804 8385 Acked Read 7 Done Sent Done false
26805 8386 Acked Flush 0 Done Sent Done false
26806 8387 Acked Write 5 Done Sent Done false
26807 8388 Acked Read 6 Done Sent Done false
26808 8389 Acked Read 3 Done Sent Done false
26809 8390 Acked Read 6 Done Sent Done false
26810 8391 Acked Read 10 Done Sent Done false
26811 8392 Acked Read 3 Done Sent Done false
26812 8393 Acked Flush 0 Done Sent Done false
26813 8394 Acked Write 7 Done Sent Done false
26814 8395 Acked Read 7 Done Sent Done false
26815 8396 Acked Read 6 Done Sent Done false
26816 8397 Acked Read 8 Done Sent Done false
26817 8398 Acked Read 5 Done Sent Done false
26818 8399 Acked Read 10 Done Sent Done false
26819 8400 Acked Read 10 Done Sent Done false
26820 8401 Acked Read 4 Done Sent Done false
26821 8402 Acked Write 8 Done Sent Done false
26822 8403 Acked Read 2 Done Sent Done false
26823 8404 Acked Read 8 Done Sent Done false
26824 8405 Acked Write 5 Done Sent Done false
26825 8406 Acked Write 5 Done Sent Done false
26826 8407 Acked Read 1 Done Sent Done false
26827 8408 Acked Read 10 Done Sent Done false
26828 8409 Acked Write 10 Done Sent Done false
26829 8410 Acked Read 4 Done Sent Done false
26830 8411 Acked Read 2 Done Sent Done false
26831 8412 Acked Read 9 Done Sent Done false
26832 8413 Acked Write 5 Done Sent Done false
26833 8414 Acked Read 2 Done Sent Done false
26834 8415 Acked Write 7 Done Sent Done false
26835 8416 Acked Write 5 Done Sent Done false
26836 8417 Acked Write 6 Done Sent Done false
26837 8418 Acked Write 6 Done Sent Done false
26838 8419 Acked Read 4 Done Sent Done false
26839 8420 Acked Read 1 Done Sent Done false
26840 8421 Acked Write 4 Done Sent Done false
26841 8422 Acked Write 3 Done Sent Done false
26842 8423 Acked Write 7 Done Sent Done false
26843 8424 Acked Read 5 Done Sent Done false
26844 8425 Acked Write 3 Done Sent Done false
26845 8426 Acked Write 4 Done Sent Done false
26846 8427 Acked Flush 0 Done Sent Done false
26847 8428 Acked Write 3 Done Sent Done false
26848 8429 Acked Read 8 Done Sent Done false
26849 8430 Acked Write 2 Done Sent Done false
26850 8431 Acked Write 9 Done Sent Done false
26851 8432 Acked Write 4 Done Sent Done false
26852 8433 Acked Read 3 Done Sent Done false
26853 8434 Acked Write 9 Done Sent Done false
26854 8435 Acked Read 2 Done Sent Done false
26855 8436 Acked Write 3 Done Sent Done false
26856 8437 Acked Read 6 Done Sent Done false
26857 8438 Acked Flush 0 Done Sent Done false
26858 8439 Acked Read 7 Done Sent Done false
26859 8440 Acked Write 10 Done Sent Done false
26860 8441 Acked Flush 0 Done Sent Done false
26861 8442 Acked Read 2 Done Sent Done false
26862 8443 Acked Read 9 Done Sent Done false
26863 8444 Acked Flush 0 Done Sent Done false
26864 8445 Acked Read 8 Done Sent Done false
26865 8446 Acked Write 2 Done Sent Done false
26866 8447 Acked Read 7 Done Sent Done false
26867 8448 Acked Write 1 Done Sent Done false
26868 8449 Acked Read 10 Done Sent Done false
26869 8450 Acked Write 6 Done Sent Done false
26870 8451 Acked Write 5 Done Sent Done false
26871 8452 Acked Write 6 Done Sent Done false
26872 8453 Acked Write 1 Done Sent Done false
26873 8454 Acked Read 4 Done Sent Done false
26874 8455 Acked Read 2 Done Sent Done false
26875 8456 Acked Write 5 Done Sent Done false
26876 8457 Acked Write 2 Done Sent Done false
26877 8458 Acked Write 1 Done Sent Done false
26878 8459 Acked Read 3 Done Sent Done false
26879 8460 Acked Read 1 Done Sent Done false
26880 8461 Acked Read 2 Done Sent Done false
26881 8462 Acked Write 5 Done Sent Done false
26882 8463 Acked Read 5 Done Sent Done false
26883 8464 Acked Flush 0 Done Sent Done false
26884 8465 Acked Write 2 Done Sent Done false
26885 8466 Acked Write 4 Done Sent Done false
26886 8467 Acked Read 4 Done Sent Done false
26887 8468 Acked Write 2 Done Sent Done false
26888 8469 Acked Read 7 Done Sent Done false
26889 8470 Acked Write 1 Done Sent Done false
26890 8471 Acked Read 4 Done Sent Done false
26891 8472 Acked Write 3 Done Sent Done false
26892 8473 Acked Write 10 Done Sent Done false
26893 8474 Acked Read 5 Done Sent Done false
26894 8475 Acked Flush 0 Done Sent Done false
26895 8476 Acked Read 7 Done Sent Done false
26896 8477 Acked Write 8 Done Sent Done false
26897 8478 Acked Write 8 Done Sent Done false
26898 8479 Acked Write 9 Done Sent Done false
26899 8480 Acked Flush 0 Done Sent Done false
26900 8481 Acked Read 5 Done Sent Done false
26901 8482 Acked Read 6 Done Sent Done false
26902 8483 Acked Read 1 Done Sent Done false
26903 8484 Acked Read 8 Done Sent Done false
26904 8485 Acked Flush 0 Done Sent Done false
26905 8486 Acked Read 9 Done Sent Done false
26906 8487 Acked Write 6 Done Sent Done false
26907 8488 Acked Write 7 Done Sent Done false
26908 8489 Acked Read 10 Done Sent Done false
26909 8490 Acked Write 9 Done Sent Done false
26910 8491 Acked Read 1 Done Sent Done false
26911 8492 Acked Read 3 Done Sent Done false
26912 8493 Acked Read 2 Done Sent Done false
26913 8494 Acked Write 3 Done Sent Done false
26914 8495 Acked Read 5 Done Sent Done false
26915 8496 Acked Read 9 Done Sent Done false
26916 8497 Acked Read 1 Done Sent Done false
26917 8498 Acked Write 2 Done Sent Done false
26918 8499 Acked Write 1 Done Sent Done false
26919 8500 Acked Write 3 Done Sent Done false
26920 8501 Acked Write 1 Done Sent Done false
26921 8502 Acked Read 4 Done Sent Done false
26922 8503 Acked Write 8 Done Sent Done false
26923 8504 Acked Read 4 Done Sent Done false
26924 8505 Acked Read 9 Done Sent Done false
26925 8506 Acked Read 8 Done Sent Done false
26926 8507 Acked Write 10 Done Sent Done false
26927 8508 Acked Write 7 Done Sent Done false
26928 8509 Acked Read 3 Done Sent Done false
26929 8510 Acked Read 5 Done Sent Done false
26930 8511 Acked Read 5 Done Sent Done false
26931 8512 Acked Write 3 Done Sent Done false
26932 8513 Acked Flush 0 Done Sent Done false
26933 8514 Acked Read 3 Done Sent Done false
26934 8515 Acked Read 9 Done Sent Done false
26935 8516 Acked Write 8 Done Sent Done false
26936 8517 Acked Read 10 Done Sent Done false
26937 8518 Acked Write 2 Done Sent Done false
26938 8519 Acked Read 7 Done Sent Done false
26939 8520 Acked Read 7 Done Sent Done false
26940 8521 Acked Read 9 Done Sent Done false
26941 8522 Acked Write 5 Done Sent Done false
26942 8523 Acked Write 3 Done Sent Done false
26943 8524 Acked Write 10 Done Sent Done false
26944 8525 Acked Read 8 Done Sent Done false
26945 8526 Acked Write 1 Done Sent Done false
26946 8527 Acked Write 2 Done Sent Done false
26947 8528 Acked Read 9 Done Sent Done false
26948 8529 Acked Write 8 Done Sent Done false
26949 8530 Acked Flush 0 Done Sent Done false
26950 8531 Acked Write 5 Done Sent Done false
26951 8532 Acked Read 1 Done Sent Done false
26952 8533 Acked Write 6 Done Sent Done false
26953 8534 Acked Write 1 Done Sent Done false
26954 8535 Acked Read 4 Done Sent Done false
26955 8536 Acked Write 4 Done Sent Done false
26956 8537 Acked Flush 0 Done Sent Done false
26957 8538 Acked Write 4 Done Sent Done false
26958 8539 Acked Flush 0 Done Sent Done false
26959 8540 Acked Read 8 Done Sent Done false
26960 8541 Acked Read 8 Done Sent Done false
26961 8542 Acked Write 10 Done Sent Done false
26962 8543 Acked Read 4 Done Sent Done false
26963 8544 Acked Read 1 Done Sent Done false
26964 8545 Acked Write 10 Done Sent Done false
26965 8546 Acked Flush 0 Done Sent Done false
26966 STATES DS:0 DS:1 DS:2 TOTAL
26967 Sent 0 302 0 302
26968 Done 302 0 302 604
26969 Skipped 0 0 0 0
26970 Error 0 0 0 0
26971 Last Flush: Some(JobId(8546)) Some(JobId(8244)) Some(JobId(8546))
26972 Downstairs last five completed: 8244 8243 8242 8241 8240
26973 Upstairs last five completed: 8546 8545 8544 8543 8542
26974 CLIENT: Up:0 ds:302 act:2
269752025-01-23T18:53:32.946ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
269762025-01-23T18:53:32.946ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
269772025-01-23T18:53:32.946ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
269782025-01-23T18:53:32.946ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
269792025-01-23T18:53:32.946ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
26980 [8820] got start action
26981 [8820] I am going to restart
26982 Starting downstairs at port 8820
26983 Make output file at "/tmp/dsc/downstairs-8820.txt"
26984 [8820][1] reports Starting
26985 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1225)
26986 [8820][1] reports Running
26987 ----------------------------------------------------------------
26988 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
26989 States: Active Offline Active
26990 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26991 8245 Acked Write 5 Done Sent Done false
26992 8246 Acked Write 5 Done Sent Done false
26993 8247 Acked Read 3 Done Sent Done false
26994 8248 Acked Write 1 Done Sent Done false
26995 8249 Acked Read 9 Done Sent Done false
26996 8250 Acked Write 5 Done Sent Done false
26997 8251 Acked Write 3 Done Sent Done false
26998 8252 Acked Read 5 Done Sent Done false
26999 8253 Acked Write 6 Done Sent Done false
27000 8254 Acked Write 6 Done Sent Done false
27001 8255 Acked Read 9 Done Sent Done false
27002 8256 Acked Write 6 Done Sent Done false
27003 8257 Acked Write 1 Done Sent Done false
27004 8258 Acked Read 8 Done Sent Done false
27005 8259 Acked Read 1 Done Sent Done false
27006 8260 Acked Flush 0 Done Sent Done false
27007 8261 Acked Read 2 Done Sent Done false
27008 8262 Acked Read 3 Done Sent Done false
27009 8263 Acked Read 3 Done Sent Done false
27010 8264 Acked Read 1 Done Sent Done false
27011 8265 Acked Write 5 Done Sent Done false
27012 8266 Acked Write 9 Done Sent Done false
27013 8267 Acked Read 3 Done Sent Done false
27014 8268 Acked Read 4 Done Sent Done false
27015 8269 Acked Read 3 Done Sent Done false
27016 8270 Acked Read 2 Done Sent Done false
27017 8271 Acked Write 5 Done Sent Done false
27018 8272 Acked Read 8 Done Sent Done false
27019 8273 Acked Write 8 Done Sent Done false
27020 8274 Acked Read 6 Done Sent Done false
27021 8275 Acked Read 6 Done Sent Done false
27022 8276 Acked Read 7 Done Sent Done false
27023 8277 Acked Write 6 Done Sent Done false
27024 8278 Acked Write 6 Done Sent Done false
27025 8279 Acked Read 7 Done Sent Done false
27026 8280 Acked Write 8 Done Sent Done false
27027 8281 Acked Read 6 Done Sent Done false
27028 8282 Acked Flush 0 Done Sent Done false
27029 8283 Acked Read 3 Done Sent Done false
27030 8284 Acked Read 10 Done Sent Done false
27031 8285 Acked Read 8 Done Sent Done false
27032 8286 Acked Write 7 Done Sent Done false
27033 8287 Acked Read 10 Done Sent Done false
27034 8288 Acked Read 2 Done Sent Done false
27035 8289 Acked Read 4 Done Sent Done false
27036 8290 Acked Write 6 Done Sent Done false
27037 8291 Acked Write 8 Done Sent Done false
27038 8292 Acked Flush 0 Done Sent Done false
27039 8293 Acked Read 5 Done Sent Done false
27040 8294 Acked Read 8 Done Sent Done false
27041 8295 Acked Write 7 Done Sent Done false
27042 8296 Acked Write 6 Done Sent Done false
27043 8297 Acked Read 9 Done Sent Done false
27044 8298 Acked Write 10 Done Sent Done false
27045 8299 Acked Write 10 Done Sent Done false
27046 8300 Acked Read 3 Done Sent Done false
27047 8301 Acked Write 10 Done Sent Done false
27048 8302 Acked Read 2 Done Sent Done false
27049 8303 Acked Write 3 Done Sent Done false
27050 8304 Acked Write 6 Done Sent Done false
27051 8305 Acked Read 7 Done Sent Done false
27052 8306 Acked Write 2 Done Sent Done false
27053 8307 Acked Read 5 Done Sent Done false
27054 8308 Acked Read 9 Done Sent Done false
27055 8309 Acked Write 5 Done Sent Done false
27056 8310 Acked Write 7 Done Sent Done false
27057 8311 Acked Read 7 Done Sent Done false
27058 8312 Acked Read 3 Done Sent Done false
27059 8313 Acked Write 7 Done Sent Done false
27060 8314 Acked Write 4 Done Sent Done false
27061 8315 Acked Write 6 Done Sent Done false
27062 8316 Acked Write 4 Done Sent Done false
27063 8317 Acked Write 3 Done Sent Done false
27064 8318 Acked Write 3 Done Sent Done false
27065 8319 Acked Write 3 Done Sent Done false
27066 8320 Acked Read 7 Done Sent Done false
27067 8321 Acked Read 8 Done Sent Done false
27068 8322 Acked Write 6 Done Sent Done false
27069 8323 Acked Flush 0 Done Sent Done false
27070 8324 Acked Flush 0 Done Sent Done false
27071 8325 Acked Write 4 Done Sent Done false
27072 8326 Acked Write 9 Done Sent Done false
27073 8327 Acked Flush 0 Done Sent Done false
27074 8328 Acked Write 10 Done Sent Done false
27075 8329 Acked Write 2 Done Sent Done false
27076 8330 Acked Read 2 Done Sent Done false
27077 8331 Acked Read 8 Done Sent Done false
27078 8332 Acked Flush 0 Done Sent Done false
27079 8333 Acked Write 3 Done Sent Done false
27080 8334 Acked Write 8 Done Sent Done false
27081 8335 Acked Write 6 Done Sent Done false
27082 8336 Acked Read 4 Done Sent Done false
27083 8337 Acked Write 4 Done Sent Done false
27084 8338 Acked Read 2 Done Sent Done false
27085 8339 Acked Read 4 Done Sent Done false
27086 8340 Acked Read 4 Done Sent Done false
27087 8341 Acked Read 1 Done Sent Done false
27088 8342 Acked Write 8 Done Sent Done false
27089 8343 Acked Write 9 Done Sent Done false
27090 8344 Acked Read 7 Done Sent Done false
27091 8345 Acked Read 6 Done Sent Done false
27092 8346 Acked Write 6 Done Sent Done false
27093 8347 Acked Flush 0 Done Sent Done false
27094 8348 Acked Read 6 Done Sent Done false
27095 8349 Acked Write 7 Done Sent Done false
27096 8350 Acked Write 7 Done Sent Done false
27097 8351 Acked Read 6 Done Sent Done false
27098 8352 Acked Read 8 Done Sent Done false
27099 8353 Acked Flush 0 Done Sent Done false
27100 8354 Acked Read 6 Done Sent Done false
27101 8355 Acked Write 2 Done Sent Done false
27102 8356 Acked Read 6 Done Sent Done false
27103 8357 Acked Read 9 Done Sent Done false
27104 8358 Acked Write 9 Done Sent Done false
27105 8359 Acked Write 9 Done Sent Done false
27106 8360 Acked Read 3 Done Sent Done false
27107 8361 Acked Flush 0 Done Sent Done false
27108 8362 Acked Read 9 Done Sent Done false
27109 8363 Acked Write 7 Done Sent Done false
27110 8364 Acked Write 2 Done Sent Done false
27111 8365 Acked Write 7 Done Sent Done false
27112 8366 Acked Write 1 Done Sent Done false
27113 8367 Acked Read 8 Done Sent Done false
27114 8368 Acked Flush 0 Done Sent Done false
27115 8369 Acked Read 2 Done Sent Done false
27116 8370 Acked Read 8 Done Sent Done false
27117 8371 Acked Write 4 Done Sent Done false
27118 8372 Acked Write 7 Done Sent Done false
27119 8373 Acked Read 4 Done Sent Done false
27120 8374 Acked Flush 0 Done Sent Done false
27121 8375 Acked Write 9 Done Sent Done false
27122 8376 Acked Write 2 Done Sent Done false
27123 8377 Acked Read 7 Done Sent Done false
27124 8378 Acked Read 3 Done Sent Done false
27125 8379 Acked Write 6 Done Sent Done false
27126 8380 Acked Write 2 Done Sent Done false
27127 8381 Acked Write 1 Done Sent Done false
27128 8382 Acked Write 5 Done Sent Done false
27129 8383 Acked Write 6 Done Sent Done false
27130 8384 Acked Read 3 Done Sent Done false
27131 8385 Acked Read 7 Done Sent Done false
27132 8386 Acked Flush 0 Done Sent Done false
27133 8387 Acked Write 5 Done Sent Done false
27134 8388 Acked Read 6 Done Sent Done false
27135 8389 Acked Read 3 Done Sent Done false
27136 8390 Acked Read 6 Done Sent Done false
27137 8391 Acked Read 10 Done Sent Done false
27138 8392 Acked Read 3 Done Sent Done false
27139 8393 Acked Flush 0 Done Sent Done false
27140 8394 Acked Write 7 Done Sent Done false
27141 8395 Acked Read 7 Done Sent Done false
27142 8396 Acked Read 6 Done Sent Done false
27143 8397 Acked Read 8 Done Sent Done false
27144 8398 Acked Read 5 Done Sent Done false
27145 8399 Acked Read 10 Done Sent Done false
27146 8400 Acked Read 10 Done Sent Done false
27147 8401 Acked Read 4 Done Sent Done false
27148 8402 Acked Write 8 Done Sent Done false
27149 8403 Acked Read 2 Done Sent Done false
27150 8404 Acked Read 8 Done Sent Done false
27151 8405 Acked Write 5 Done Sent Done false
27152 8406 Acked Write 5 Done Sent Done false
27153 8407 Acked Read 1 Done Sent Done false
27154 8408 Acked Read 10 Done Sent Done false
27155 8409 Acked Write 10 Done Sent Done false
27156 8410 Acked Read 4 Done Sent Done false
27157 8411 Acked Read 2 Done Sent Done false
27158 8412 Acked Read 9 Done Sent Done false
27159 8413 Acked Write 5 Done Sent Done false
27160 8414 Acked Read 2 Done Sent Done false
27161 8415 Acked Write 7 Done Sent Done false
27162 8416 Acked Write 5 Done Sent Done false
27163 8417 Acked Write 6 Done Sent Done false
27164 8418 Acked Write 6 Done Sent Done false
27165 8419 Acked Read 4 Done Sent Done false
27166 8420 Acked Read 1 Done Sent Done false
27167 8421 Acked Write 4 Done Sent Done false
27168 8422 Acked Write 3 Done Sent Done false
27169 8423 Acked Write 7 Done Sent Done false
27170 8424 Acked Read 5 Done Sent Done false
27171 8425 Acked Write 3 Done Sent Done false
27172 8426 Acked Write 4 Done Sent Done false
27173 8427 Acked Flush 0 Done Sent Done false
27174 8428 Acked Write 3 Done Sent Done false
27175 8429 Acked Read 8 Done Sent Done false
27176 8430 Acked Write 2 Done Sent Done false
27177 8431 Acked Write 9 Done Sent Done false
27178 8432 Acked Write 4 Done Sent Done false
27179 8433 Acked Read 3 Done Sent Done false
27180 8434 Acked Write 9 Done Sent Done false
27181 8435 Acked Read 2 Done Sent Done false
27182 8436 Acked Write 3 Done Sent Done false
27183 8437 Acked Read 6 Done Sent Done false
27184 8438 Acked Flush 0 Done Sent Done false
27185 8439 Acked Read 7 Done Sent Done false
27186 8440 Acked Write 10 Done Sent Done false
27187 8441 Acked Flush 0 Done Sent Done false
27188 8442 Acked Read 2 Done Sent Done false
27189 8443 Acked Read 9 Done Sent Done false
27190 8444 Acked Flush 0 Done Sent Done false
27191 8445 Acked Read 8 Done Sent Done false
27192 8446 Acked Write 2 Done Sent Done false
27193 8447 Acked Read 7 Done Sent Done false
27194 8448 Acked Write 1 Done Sent Done false
27195 8449 Acked Read 10 Done Sent Done false
27196 8450 Acked Write 6 Done Sent Done false
27197 8451 Acked Write 5 Done Sent Done false
27198 8452 Acked Write 6 Done Sent Done false
27199 8453 Acked Write 1 Done Sent Done false
27200 8454 Acked Read 4 Done Sent Done false
27201 8455 Acked Read 2 Done Sent Done false
27202 8456 Acked Write 5 Done Sent Done false
27203 8457 Acked Write 2 Done Sent Done false
27204 8458 Acked Write 1 Done Sent Done false
27205 8459 Acked Read 3 Done Sent Done false
27206 8460 Acked Read 1 Done Sent Done false
27207 8461 Acked Read 2 Done Sent Done false
27208 8462 Acked Write 5 Done Sent Done false
27209 8463 Acked Read 5 Done Sent Done false
27210 8464 Acked Flush 0 Done Sent Done false
27211 8465 Acked Write 2 Done Sent Done false
27212 8466 Acked Write 4 Done Sent Done false
27213 8467 Acked Read 4 Done Sent Done false
27214 8468 Acked Write 2 Done Sent Done false
27215 8469 Acked Read 7 Done Sent Done false
27216 8470 Acked Write 1 Done Sent Done false
27217 8471 Acked Read 4 Done Sent Done false
27218 8472 Acked Write 3 Done Sent Done false
27219 8473 Acked Write 10 Done Sent Done false
27220 8474 Acked Read 5 Done Sent Done false
27221 8475 Acked Flush 0 Done Sent Done false
27222 8476 Acked Read 7 Done Sent Done false
27223 8477 Acked Write 8 Done Sent Done false
27224 8478 Acked Write 8 Done Sent Done false
27225 8479 Acked Write 9 Done Sent Done false
27226 8480 Acked Flush 0 Done Sent Done false
27227 8481 Acked Read 5 Done Sent Done false
27228 8482 Acked Read 6 Done Sent Done false
27229 8483 Acked Read 1 Done Sent Done false
27230 8484 Acked Read 8 Done Sent Done false
27231 8485 Acked Flush 0 Done Sent Done false
27232 8486 Acked Read 9 Done Sent Done false
27233 8487 Acked Write 6 Done Sent Done false
27234 8488 Acked Write 7 Done Sent Done false
27235 8489 Acked Read 10 Done Sent Done false
27236 8490 Acked Write 9 Done Sent Done false
27237 8491 Acked Read 1 Done Sent Done false
27238 8492 Acked Read 3 Done Sent Done false
27239 8493 Acked Read 2 Done Sent Done false
27240 8494 Acked Write 3 Done Sent Done false
27241 8495 Acked Read 5 Done Sent Done false
27242 8496 Acked Read 9 Done Sent Done false
27243 8497 Acked Read 1 Done Sent Done false
27244 8498 Acked Write 2 Done Sent Done false
27245 8499 Acked Write 1 Done Sent Done false
27246 8500 Acked Write 3 Done Sent Done false
27247 8501 Acked Write 1 Done Sent Done false
27248 8502 Acked Read 4 Done Sent Done false
27249 8503 Acked Write 8 Done Sent Done false
27250 8504 Acked Read 4 Done Sent Done false
27251 8505 Acked Read 9 Done Sent Done false
27252 8506 Acked Read 8 Done Sent Done false
27253 8507 Acked Write 10 Done Sent Done false
27254 8508 Acked Write 7 Done Sent Done false
27255 8509 Acked Read 3 Done Sent Done false
27256 8510 Acked Read 5 Done Sent Done false
27257 8511 Acked Read 5 Done Sent Done false
27258 8512 Acked Write 3 Done Sent Done false
27259 8513 Acked Flush 0 Done Sent Done false
27260 8514 Acked Read 3 Done Sent Done false
27261 8515 Acked Read 9 Done Sent Done false
27262 8516 Acked Write 8 Done Sent Done false
27263 8517 Acked Read 10 Done Sent Done false
27264 8518 Acked Write 2 Done Sent Done false
27265 8519 Acked Read 7 Done Sent Done false
27266 8520 Acked Read 7 Done Sent Done false
27267 8521 Acked Read 9 Done Sent Done false
27268 8522 Acked Write 5 Done Sent Done false
27269 8523 Acked Write 3 Done Sent Done false
27270 8524 Acked Write 10 Done Sent Done false
27271 8525 Acked Read 8 Done Sent Done false
27272 8526 Acked Write 1 Done Sent Done false
27273 8527 Acked Write 2 Done Sent Done false
27274 8528 Acked Read 9 Done Sent Done false
27275 8529 Acked Write 8 Done Sent Done false
27276 8530 Acked Flush 0 Done Sent Done false
27277 8531 Acked Write 5 Done Sent Done false
27278 8532 Acked Read 1 Done Sent Done false
27279 8533 Acked Write 6 Done Sent Done false
27280 8534 Acked Write 1 Done Sent Done false
27281 8535 Acked Read 4 Done Sent Done false
27282 8536 Acked Write 4 Done Sent Done false
27283 8537 Acked Flush 0 Done Sent Done false
27284 8538 Acked Write 4 Done Sent Done false
27285 8539 Acked Flush 0 Done Sent Done false
27286 8540 Acked Read 8 Done Sent Done false
27287 8541 Acked Read 8 Done Sent Done false
27288 8542 Acked Write 10 Done Sent Done false
27289 8543 Acked Read 4 Done Sent Done false
27290 8544 Acked Read 1 Done Sent Done false
27291 8545 Acked Write 10 Done Sent Done false
27292 8546 Acked Flush 0 Done Sent Done false
27293 STATES DS:0 DS:1 DS:2 TOTAL
27294 Sent 0 302 0 302
27295 Done 302 0 302 604
27296 Skipped 0 0 0 0
27297 Error 0 0 0 0
27298 Last Flush: Some(JobId(8546)) Some(JobId(8244)) Some(JobId(8546))
27299 Downstairs last five completed: 8244 8243 8242 8241 8240
27300 Upstairs last five completed: 8546 8545 8544 8543 8542
27301 CLIENT: Up:0 ds:302 act:2
27302 ----------------------------------------------------------------
27303 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
27304 States: Active Offline Active
27305 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27306 8245 Acked Write 5 Done Sent Done false
27307 8246 Acked Write 5 Done Sent Done false
27308 8247 Acked Read 3 Done Sent Done false
27309 8248 Acked Write 1 Done Sent Done false
27310 8249 Acked Read 9 Done Sent Done false
27311 8250 Acked Write 5 Done Sent Done false
27312 8251 Acked Write 3 Done Sent Done false
27313 8252 Acked Read 5 Done Sent Done false
27314 8253 Acked Write 6 Done Sent Done false
27315 8254 Acked Write 6 Done Sent Done false
27316 8255 Acked Read 9 Done Sent Done false
27317 8256 Acked Write 6 Done Sent Done false
27318 8257 Acked Write 1 Done Sent Done false
27319 8258 Acked Read 8 Done Sent Done false
27320 8259 Acked Read 1 Done Sent Done false
27321 8260 Acked Flush 0 Done Sent Done false
27322 8261 Acked Read 2 Done Sent Done false
27323 8262 Acked Read 3 Done Sent Done false
27324 8263 Acked Read 3 Done Sent Done false
27325 8264 Acked Read 1 Done Sent Done false
27326 8265 Acked Write 5 Done Sent Done false
27327 8266 Acked Write 9 Done Sent Done false
27328 8267 Acked Read 3 Done Sent Done false
27329 8268 Acked Read 4 Done Sent Done false
27330 8269 Acked Read 3 Done Sent Done false
27331 8270 Acked Read 2 Done Sent Done false
27332 8271 Acked Write 5 Done Sent Done false
27333 8272 Acked Read 8 Done Sent Done false
27334 8273 Acked Write 8 Done Sent Done false
27335 8274 Acked Read 6 Done Sent Done false
27336 8275 Acked Read 6 Done Sent Done false
27337 8276 Acked Read 7 Done Sent Done false
27338 8277 Acked Write 6 Done Sent Done false
27339 8278 Acked Write 6 Done Sent Done false
27340 8279 Acked Read 7 Done Sent Done false
27341 8280 Acked Write 8 Done Sent Done false
27342 8281 Acked Read 6 Done Sent Done false
27343 8282 Acked Flush 0 Done Sent Done false
27344 8283 Acked Read 3 Done Sent Done false
27345 8284 Acked Read 10 Done Sent Done false
27346 8285 Acked Read 8 Done Sent Done false
27347 8286 Acked Write 7 Done Sent Done false
27348 8287 Acked Read 10 Done Sent Done false
27349 8288 Acked Read 2 Done Sent Done false
27350 8289 Acked Read 4 Done Sent Done false
27351 8290 Acked Write 6 Done Sent Done false
27352 8291 Acked Write 8 Done Sent Done false
27353 8292 Acked Flush 0 Done Sent Done false
27354 8293 Acked Read 5 Done Sent Done false
27355 8294 Acked Read 8 Done Sent Done false
27356 8295 Acked Write 7 Done Sent Done false
27357 8296 Acked Write 6 Done Sent Done false
27358 8297 Acked Read 9 Done Sent Done false
27359 8298 Acked Write 10 Done Sent Done false
27360 8299 Acked Write 10 Done Sent Done false
27361 8300 Acked Read 3 Done Sent Done false
27362 8301 Acked Write 10 Done Sent Done false
27363 8302 Acked Read 2 Done Sent Done false
27364 8303 Acked Write 3 Done Sent Done false
27365 8304 Acked Write 6 Done Sent Done false
27366 8305 Acked Read 7 Done Sent Done false
27367 8306 Acked Write 2 Done Sent Done false
27368 8307 Acked Read 5 Done Sent Done false
27369 8308 Acked Read 9 Done Sent Done false
27370 8309 Acked Write 5 Done Sent Done false
27371 8310 Acked Write 7 Done Sent Done false
27372 8311 Acked Read 7 Done Sent Done false
27373 8312 Acked Read 3 Done Sent Done false
27374 8313 Acked Write 7 Done Sent Done false
27375 8314 Acked Write 4 Done Sent Done false
27376 8315 Acked Write 6 Done Sent Done false
27377 8316 Acked Write 4 Done Sent Done false
27378 8317 Acked Write 3 Done Sent Done false
27379 8318 Acked Write 3 Done Sent Done false
27380 8319 Acked Write 3 Done Sent Done false
27381 8320 Acked Read 7 Done Sent Done false
27382 8321 Acked Read 8 Done Sent Done false
27383 8322 Acked Write 6 Done Sent Done false
27384 8323 Acked Flush 0 Done Sent Done false
27385 8324 Acked Flush 0 Done Sent Done false
27386 8325 Acked Write 4 Done Sent Done false
27387 8326 Acked Write 9 Done Sent Done false
27388 8327 Acked Flush 0 Done Sent Done false
27389 8328 Acked Write 10 Done Sent Done false
27390 8329 Acked Write 2 Done Sent Done false
27391 8330 Acked Read 2 Done Sent Done false
27392 8331 Acked Read 8 Done Sent Done false
27393 8332 Acked Flush 0 Done Sent Done false
27394 8333 Acked Write 3 Done Sent Done false
27395 8334 Acked Write 8 Done Sent Done false
27396 8335 Acked Write 6 Done Sent Done false
27397 8336 Acked Read 4 Done Sent Done false
27398 8337 Acked Write 4 Done Sent Done false
27399 8338 Acked Read 2 Done Sent Done false
27400 8339 Acked Read 4 Done Sent Done false
27401 8340 Acked Read 4 Done Sent Done false
27402 8341 Acked Read 1 Done Sent Done false
27403 8342 Acked Write 8 Done Sent Done false
27404 8343 Acked Write 9 Done Sent Done false
27405 8344 Acked Read 7 Done Sent Done false
27406 8345 Acked Read 6 Done Sent Done false
27407 8346 Acked Write 6 Done Sent Done false
27408 8347 Acked Flush 0 Done Sent Done false
27409 8348 Acked Read 6 Done Sent Done false
27410 8349 Acked Write 7 Done Sent Done false
27411 8350 Acked Write 7 Done Sent Done false
27412 8351 Acked Read 6 Done Sent Done false
27413 8352 Acked Read 8 Done Sent Done false
27414 8353 Acked Flush 0 Done Sent Done false
27415 8354 Acked Read 6 Done Sent Done false
27416 8355 Acked Write 2 Done Sent Done false
27417 8356 Acked Read 6 Done Sent Done false
27418 8357 Acked Read 9 Done Sent Done false
27419 8358 Acked Write 9 Done Sent Done false
27420 8359 Acked Write 9 Done Sent Done false
27421 8360 Acked Read 3 Done Sent Done false
27422 8361 Acked Flush 0 Done Sent Done false
27423 8362 Acked Read 9 Done Sent Done false
27424 8363 Acked Write 7 Done Sent Done false
27425 8364 Acked Write 2 Done Sent Done false
27426 8365 Acked Write 7 Done Sent Done false
27427 8366 Acked Write 1 Done Sent Done false
27428 8367 Acked Read 8 Done Sent Done false
27429 8368 Acked Flush 0 Done Sent Done false
27430 8369 Acked Read 2 Done Sent Done false
27431 8370 Acked Read 8 Done Sent Done false
27432 8371 Acked Write 4 Done Sent Done false
27433 8372 Acked Write 7 Done Sent Done false
27434 8373 Acked Read 4 Done Sent Done false
27435 8374 Acked Flush 0 Done Sent Done false
27436 8375 Acked Write 9 Done Sent Done false
27437 8376 Acked Write 2 Done Sent Done false
27438 8377 Acked Read 7 Done Sent Done false
27439 8378 Acked Read 3 Done Sent Done false
27440 8379 Acked Write 6 Done Sent Done false
27441 8380 Acked Write 2 Done Sent Done false
27442 8381 Acked Write 1 Done Sent Done false
27443 8382 Acked Write 5 Done Sent Done false
27444 8383 Acked Write 6 Done Sent Done false
27445 8384 Acked Read 3 Done Sent Done false
27446 8385 Acked Read 7 Done Sent Done false
27447 8386 Acked Flush 0 Done Sent Done false
27448 8387 Acked Write 5 Done Sent Done false
27449 8388 Acked Read 6 Done Sent Done false
27450 8389 Acked Read 3 Done Sent Done false
27451 8390 Acked Read 6 Done Sent Done false
27452 8391 Acked Read 10 Done Sent Done false
27453 8392 Acked Read 3 Done Sent Done false
27454 8393 Acked Flush 0 Done Sent Done false
27455 8394 Acked Write 7 Done Sent Done false
27456 8395 Acked Read 7 Done Sent Done false
27457 8396 Acked Read 6 Done Sent Done false
27458 8397 Acked Read 8 Done Sent Done false
27459 8398 Acked Read 5 Done Sent Done false
27460 8399 Acked Read 10 Done Sent Done false
27461 8400 Acked Read 10 Done Sent Done false
27462 8401 Acked Read 4 Done Sent Done false
27463 8402 Acked Write 8 Done Sent Done false
27464 8403 Acked Read 2 Done Sent Done false
27465 8404 Acked Read 8 Done Sent Done false
27466 8405 Acked Write 5 Done Sent Done false
27467 8406 Acked Write 5 Done Sent Done false
27468 8407 Acked Read 1 Done Sent Done false
27469 8408 Acked Read 10 Done Sent Done false
27470 8409 Acked Write 10 Done Sent Done false
27471 8410 Acked Read 4 Done Sent Done false
27472 8411 Acked Read 2 Done Sent Done false
27473 8412 Acked Read 9 Done Sent Done false
27474 8413 Acked Write 5 Done Sent Done false
27475 8414 Acked Read 2 Done Sent Done false
27476 8415 Acked Write 7 Done Sent Done false
27477 8416 Acked Write 5 Done Sent Done false
27478 8417 Acked Write 6 Done Sent Done false
27479 8418 Acked Write 6 Done Sent Done false
27480 8419 Acked Read 4 Done Sent Done false
27481 8420 Acked Read 1 Done Sent Done false
27482 8421 Acked Write 4 Done Sent Done false
27483 8422 Acked Write 3 Done Sent Done false
27484 8423 Acked Write 7 Done Sent Done false
27485 8424 Acked Read 5 Done Sent Done false
27486 8425 Acked Write 3 Done Sent Done false
27487 8426 Acked Write 4 Done Sent Done false
27488 8427 Acked Flush 0 Done Sent Done false
27489 8428 Acked Write 3 Done Sent Done false
27490 8429 Acked Read 8 Done Sent Done false
27491 8430 Acked Write 2 Done Sent Done false
27492 8431 Acked Write 9 Done Sent Done false
27493 8432 Acked Write 4 Done Sent Done false
27494 8433 Acked Read 3 Done Sent Done false
27495 8434 Acked Write 9 Done Sent Done false
27496 8435 Acked Read 2 Done Sent Done false
27497 8436 Acked Write 3 Done Sent Done false
27498 8437 Acked Read 6 Done Sent Done false
27499 8438 Acked Flush 0 Done Sent Done false
27500 8439 Acked Read 7 Done Sent Done false
27501 8440 Acked Write 10 Done Sent Done false
27502 8441 Acked Flush 0 Done Sent Done false
27503 8442 Acked Read 2 Done Sent Done false
27504 8443 Acked Read 9 Done Sent Done false
27505 8444 Acked Flush 0 Done Sent Done false
27506 8445 Acked Read 8 Done Sent Done false
27507 8446 Acked Write 2 Done Sent Done false
27508 8447 Acked Read 7 Done Sent Done false
27509 8448 Acked Write 1 Done Sent Done false
27510 8449 Acked Read 10 Done Sent Done false
27511 8450 Acked Write 6 Done Sent Done false
27512 8451 Acked Write 5 Done Sent Done false
27513 8452 Acked Write 6 Done Sent Done false
27514 8453 Acked Write 1 Done Sent Done false
27515 8454 Acked Read 4 Done Sent Done false
27516 8455 Acked Read 2 Done Sent Done false
27517 8456 Acked Write 5 Done Sent Done false
27518 8457 Acked Write 2 Done Sent Done false
27519 8458 Acked Write 1 Done Sent Done false
27520 8459 Acked Read 3 Done Sent Done false
27521 8460 Acked Read 1 Done Sent Done false
27522 8461 Acked Read 2 Done Sent Done false
27523 8462 Acked Write 5 Done Sent Done false
27524 8463 Acked Read 5 Done Sent Done false
27525 8464 Acked Flush 0 Done Sent Done false
27526 8465 Acked Write 2 Done Sent Done false
27527 8466 Acked Write 4 Done Sent Done false
27528 8467 Acked Read 4 Done Sent Done false
27529 8468 Acked Write 2 Done Sent Done false
27530 8469 Acked Read 7 Done Sent Done false
27531 8470 Acked Write 1 Done Sent Done false
27532 8471 Acked Read 4 Done Sent Done false
27533 8472 Acked Write 3 Done Sent Done false
27534 8473 Acked Write 10 Done Sent Done false
27535 8474 Acked Read 5 Done Sent Done false
27536 8475 Acked Flush 0 Done Sent Done false
27537 8476 Acked Read 7 Done Sent Done false
27538 8477 Acked Write 8 Done Sent Done false
27539 8478 Acked Write 8 Done Sent Done false
27540 8479 Acked Write 9 Done Sent Done false
27541 8480 Acked Flush 0 Done Sent Done false
27542 8481 Acked Read 5 Done Sent Done false
27543 8482 Acked Read 6 Done Sent Done false
27544 8483 Acked Read 1 Done Sent Done false
27545 8484 Acked Read 8 Done Sent Done false
27546 8485 Acked Flush 0 Done Sent Done false
27547 8486 Acked Read 9 Done Sent Done false
27548 8487 Acked Write 6 Done Sent Done false
27549 8488 Acked Write 7 Done Sent Done false
27550 8489 Acked Read 10 Done Sent Done false
27551 8490 Acked Write 9 Done Sent Done false
27552 8491 Acked Read 1 Done Sent Done false
27553 8492 Acked Read 3 Done Sent Done false
27554 8493 Acked Read 2 Done Sent Done false
27555 8494 Acked Write 3 Done Sent Done false
27556 8495 Acked Read 5 Done Sent Done false
27557 8496 Acked Read 9 Done Sent Done false
27558 8497 Acked Read 1 Done Sent Done false
27559 8498 Acked Write 2 Done Sent Done false
27560 8499 Acked Write 1 Done Sent Done false
27561 8500 Acked Write 3 Done Sent Done false
27562 8501 Acked Write 1 Done Sent Done false
27563 8502 Acked Read 4 Done Sent Done false
27564 8503 Acked Write 8 Done Sent Done false
27565 8504 Acked Read 4 Done Sent Done false
27566 8505 Acked Read 9 Done Sent Done false
27567 8506 Acked Read 8 Done Sent Done false
27568 8507 Acked Write 10 Done Sent Done false
27569 8508 Acked Write 7 Done Sent Done false
27570 8509 Acked Read 3 Done Sent Done false
27571 8510 Acked Read 5 Done Sent Done false
27572 8511 Acked Read 5 Done Sent Done false
27573 8512 Acked Write 3 Done Sent Done false
27574 8513 Acked Flush 0 Done Sent Done false
27575 8514 Acked Read 3 Done Sent Done false
27576 8515 Acked Read 9 Done Sent Done false
27577 8516 Acked Write 8 Done Sent Done false
27578 8517 Acked Read 10 Done Sent Done false
27579 8518 Acked Write 2 Done Sent Done false
27580 8519 Acked Read 7 Done Sent Done false
27581 8520 Acked Read 7 Done Sent Done false
27582 8521 Acked Read 9 Done Sent Done false
27583 8522 Acked Write 5 Done Sent Done false
27584 8523 Acked Write 3 Done Sent Done false
27585 8524 Acked Write 10 Done Sent Done false
27586 8525 Acked Read 8 Done Sent Done false
27587 8526 Acked Write 1 Done Sent Done false
27588 8527 Acked Write 2 Done Sent Done false
27589 8528 Acked Read 9 Done Sent Done false
27590 8529 Acked Write 8 Done Sent Done false
27591 8530 Acked Flush 0 Done Sent Done false
27592 8531 Acked Write 5 Done Sent Done false
27593 8532 Acked Read 1 Done Sent Done false
27594 8533 Acked Write 6 Done Sent Done false
27595 8534 Acked Write 1 Done Sent Done false
27596 8535 Acked Read 4 Done Sent Done false
27597 8536 Acked Write 4 Done Sent Done false
27598 8537 Acked Flush 0 Done Sent Done false
27599 8538 Acked Write 4 Done Sent Done false
27600 8539 Acked Flush 0 Done Sent Done false
27601 8540 Acked Read 8 Done Sent Done false
27602 8541 Acked Read 8 Done Sent Done false
27603 8542 Acked Write 10 Done Sent Done false
27604 8543 Acked Read 4 Done Sent Done false
27605 8544 Acked Read 1 Done Sent Done false
27606 8545 Acked Write 10 Done Sent Done false
27607 8546 Acked Flush 0 Done Sent Done false
27608 STATES DS:0 DS:1 DS:2 TOTAL
27609 Sent 0 302 0 302
27610 Done 302 0 302 604
27611 Skipped 0 0 0 0
27612 Error 0 0 0 0
27613 Last Flush: Some(JobId(8546)) Some(JobId(8244)) Some(JobId(8546))
27614 Downstairs last five completed: 8244 8243 8242 8241 8240
27615 Upstairs last five completed: 8546 8545 8544 8543 8542
27616 CLIENT: Up:0 ds:302 act:2
276172025-01-23T18:53:42.947ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
276182025-01-23T18:53:42.947ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60123) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
276192025-01-23T18:53:42.948ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
276202025-01-23T18:53:42.949ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
276212025-01-23T18:53:42.949ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
276222025-01-23T18:53:42.949ZINFOcrucible: send last flush ID to this DS: Some(JobId(8244)) = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
276232025-01-23T18:53:42.949ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
276242025-01-23T18:53:42.949ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(8244)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
276252025-01-23T18:53:42.949ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(8244)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
27626 ----------------------------------------------------------------
27627 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
27628 Downstairs last five completed: 8546 8545 8544 8543 8542
27629 Upstairs last five completed: 8546 8545 8544 8543 8542
27630 CLIENT: Up:0 ds:0 act:3
27631 Replay: All jobs finished
27632 Main task has work to do, go find it
27633 got dsc Stop(0)
27634 stop 0
27635 [8810] Got stop action so:false kr:false
27636 Jan 23 18:53:43.187 INFO request completed, latency_us: 305, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 3b384a2e-5c1d-4001-9878-ca12b3e17a0b, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
276372025-01-23T18:53:43.188ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
27638 Jan 23 18:53:43.188 INFO request completed{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:53:43.188214229Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
27639 , latency_us: 126, response_code: 200, uri: /state/cid/0, method: GET, req_id: a9dad340-84e5-493a-931c-28d7be85621a, remote_addr: 127{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:53:43.188248365Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
27640 .0.0.1:59295, local_addr: 127.0.0.1:9998
276412025-01-23T18:53:43.188ZINFOcrucible: client task is exiting = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
27642 [8810] Exited with: ExitStatus(unix_wait_status(9))
27643 [8810][0] reports Exit
27644 Jan 23 18:53:47.190 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/0, method: GET, req_id: a15ad460-e6b8-4790-b037-aaea657f7683, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
27645 001/300 Read block 3275 len 24576
27646 002/300 Read block 812 len 24576
27647 003/300 Write block 4993 len 16384 data: 4 3 2 2
27648 004/300 Read block 428 len 40960
27649 005/300 Read block 913 len 8192
27650 006/300 Flush
27651 007/300 Write block 42 len 8192 data: 3 3
27652 008/300 Read block 2734 len 16384
27653 009/300 Write block 4557 len 16384 data: 4 5 5 5
27654 010/300 Write block 3656 len 40960 data: 6 5 4 4 5 5 6 5 6 7
27655 011/300 Write block 1082 len 24576 data: 6 6 6 6 6 6
27656 012/300 Read block 276 len 20480
27657 013/300 Write block 721 len 16384 data: 4 3 3 4
27658 014/300 Write block 1500 len 12288 data: 4 4 3
27659 015/300 Read block 3916 len 4096
27660 016/300 Write block 3701 len 28672 data: 4 5 4 4 3 4 5
27661 017/300 Flush
27662 018/300 Write block 2686 len 8192 data: 6 6
27663 019/300 Read block 1322 len 24576
27664 020/300 Write block 448 len 28672 data: 3 3 5 5 6 6 7
27665 021/300 Read block 3829 len 12288
27666 022/300 Read block 2434 len 24576
27667 023/300 Write block 3831 len 16384 data: 7 7 4 6
27668 024/300 Read block 1868 len 36864
27669 025/300 Write block 3137 len 28672 data: 5 5 4 4 4 4 4
27670 026/300 Read block 86 len 24576
27671 027/300 Write block 2859 len 32768 data: 6 6 6 6 5 5 4 4
27672 028/300 Read block 2711 len 12288
27673 029/300 Write block 2585 len 16384 data: 8 9 6 5
27674 030/300 Write block 4789 len 40960 data: 5 4 3 4 5 5 5 3 3 3
27675 031/300 Read block 4178 len 4096
27676 032/300 Read block 1178 len 8192
27677 033/300 Write block 3790 len 28672 data: 5 6 5 5 5 6 6
27678 034/300 Flush
27679 035/300 Read block 2144 len 24576
27680 036/300 Flush
27681 037/300 Flush
27682 038/300 Read block 1351 len 36864
27683 039/300 Read block 3390 len 20480
27684 040/300 Flush
27685 041/300 Read block 1884 len 20480
27686 042/300 Read block 1571 len 4096
27687 043/300 Read block 3944 len 28672
27688 044/300 Write block 1698 len 32768 data: 6 6 5 4 5 7 7 6
27689 045/300 Write block 1506 len 32768 data: 7 6 7 7 7 7 6 5
27690 046/300 Write block 3796 len 20480 data: 7 5 5 6 6
27691 047/300 Read block 4227 len 12288
27692 048/300 Write block 1402 len 12288 data: 4 4 6
27693 049/300 Read block 2565 len 16384
27694 050/300 Write block 1007 len 8192 data: 4 4
27695 051/300 Read block 4658 len 16384
27696 052/300 Write block 4448 len 8192 data: 10 11
27697 053/300 Read block 4973 len 16384
27698 054/300 Read block 4494 len 24576
27699 055/300 Write block 1182 len 36864 data: 4 3 4 4 6 6 5 4 4
27700 056/300 Flush
27701 057/300 Read block 2865 len 8192
27702 058/300 Read block 359 len 16384
27703 059/300 Read block 4758 len 20480
27704 060/300 Read block 2294 len 36864
27705 061/300 Read block 2753 len 16384
27706 062/300 Write block 100 len 28672 data: 5 5 6 7 6 5 4
27707 063/300 Read block 1834 len 40960
27708 064/300 Flush
27709 065/300 Write block 1691 len 40960 data: 6 7 8 8 8 8 7 7 7 6
27710 066/300 Read block 618 len 32768
27711 067/300 Flush
27712 068/300 Read block 0 len 12288
27713 069/300 Write block 1640 len 28672 data: 6 7 7 6 8 8 8
27714 070/300 Read block 1117 len 20480
27715 071/300 Read block 657 len 8192
27716 072/300 Read block 1373 len 32768
27717 073/300 Flush
27718 074/300 Write block 2110 len 40960 data: 7 6 5 3 3 2 2 3 3 4
27719 075/300 Write block 4648 len 20480 data: 6 5 6 6 5
27720 076/300 Read block 1682 len 8192
27721 077/300 Read block 618 len 4096
27722 078/300 Write block 3640 len 4096 data: 7
27723 079/300 Read block 4696 len 20480
27724 080/300 Read block 1686 len 36864
27725 081/300 Read block 470 len 16384
27726 082/300 Read block 1546 len 32768
27727 083/300 Read block 1555 len 28672
27728 084/300 Write block 1142 len 12288 data: 8 7 8
27729 085/300 Flush
27730 086/300 Read block 4260 len 16384
27731 087/300 Write block 1303 len 36864 data: 4 6 5 4 6 6 6 6 6
27732 088/300 Read block 2009 len 8192
27733 089/300 Write block 4519 len 20480 data: 3 4 4 4 5
27734 090/300 Read block 2135 len 32768
27735 091/300 Read block 2102 len 40960
27736 092/300 Write block 2308 len 20480 data: 6 7 7 7 6
27737 093/300 Read block 1028 len 36864
27738 094/300 Write block 1990 len 24576 data: 5 3 3 4 5 3
27739 095/300 Read block 2659 len 16384
27740 096/300 Flush
27741 097/300 Read block 1518 len 4096
27742 098/300 Write block 4135 len 20480 data: 6 6 6 4 7
27743 099/300 Read block 107 len 40960
27744 100/300 Write block 1190 len 24576 data: 5 7 6 6 5 5
27745 101/300 Read block 1842 len 16384
27746 102/300 Write block 4561 len 28672 data: 6 7 6 6 5 5 6
27747 103/300 Write block 2665 len 4096 data: 3
27748 104/300 Read block 559 len 20480
27749 105/300 Read block 2906 len 40960
27750 106/300 Read block 2179 len 16384
27751 107/300 Write block 4135 len 12288 data: 7 7 7
27752 108/300 Read block 2388 len 20480
27753 109/300 Read block 3669 len 4096
27754 110/300 Flush
27755 111/300 Read block 4872 len 20480
27756 112/300 Read block 2326 len 40960
27757 113/300 Read block 1136 len 40960
27758 114/300 Flush
27759 115/300 Read block 1961 len 36864
27760 116/300 Read block 1 len 24576
27761 117/300 Read block 2200 len 24576
27762 118/300 Write block 3130 len 32768 data: 5 4 4 3 4 4 4 6
27763 119/300 Flush
27764 120/300 Write block 3035 len 8192 data: 6 6
27765 121/300 Write block 342 len 20480 data: 4 4 4 4 3
27766 122/300 Write block 4482 len 40960 data: 4 4 5 6 5 5 5 6 6 6
27767 123/300 Read block 4019 len 32768
27768 124/300 Read block 4029 len 32768
27769 125/300 Read block 1605 len 8192
27770 126/300 Write block 3736 len 32768 data: 9 8 8 7 7 6 5 5
27771 127/300 Read block 135 len 12288
27772 128/300 Write block 2815 len 28672 data: 3 3 2 3 3 4 4
27773 129/300 Read block 2040 len 4096
27774 130/300 Write block 3063 len 12288 data: 4 6 7
27775 131/300 Read block 2150 len 4096
27776 132/300 Write block 1709 len 24576 data: 6 4 3 4 4 4
27777 133/300 Read block 1107 len 24576
27778 134/300 Write block 4363 len 24576 data: 5 5 5 4 4 5
27779 135/300 Flush
27780 136/300 Read block 1436 len 40960
27781 137/300 Read block 990 len 20480
27782 138/300 Write block 3248 len 16384 data: 8 8 8 9
27783 139/300 Flush
27784 140/300 Write block 2725 len 12288 data: 5 5 4
27785 141/300 Write block 4871 len 4096 data: 6
27786 142/300 Flush
27787 143/300 Read block 1404 len 12288
27788 144/300 Write block 2006 len 4096 data: 5
27789 145/300 Write block 620 len 32768 data: 6 7 7 5 7 6 9 5
27790 146/300 Read block 716 len 28672
27791 147/300 Write block 2669 len 24576 data: 4 7 8 7 8 6
27792 148/300 Read block 208 len 4096
27793 149/300 Write block 1002 len 4096 data: 9
27794 150/300 Read block 498 len 4096
27795 151/300 Write block 1094 len 24576 data: 7 9 9 8 9 9
27796 152/300 Read block 3361 len 24576
27797 153/300 Write block 3225 len 16384 data: 6 6 5 6
27798 154/300 Read block 1499 len 28672
27799 155/300 Write block 4318 len 8192 data: 5 6
27800 156/300 Read block 1883 len 40960
27801 157/300 Read block 3065 len 8192
27802 158/300 Write block 4777 len 36864 data: 4 7 7 6 5 6 7 7 7
27803 159/300 Read block 2858 len 12288
27804 160/300 Write block 2344 len 12288 data: 6 4 5
27805 161/300 Write block 4114 len 4096 data: 4
27806 162/300 Write block 50 len 4096 data: 5
27807 163/300 Read block 3919 len 36864
27808 164/300 Write block 4430 len 16384 data: 7 8 6 6
27809 165/300 Read block 2324 len 16384
27810 166/300 Write block 2566 len 12288 data: 2 2 3
27811 167/300 Write block 4833 len 12288 data: 7 8 7
27812 168/300 Read block 1193 len 16384
27813 169/300 Read block 1842 len 36864
27814 170/300 Write block 2562 len 4096 data: 6
27815 171/300 Write block 187 len 4096 data: 7
27816 172/300 Flush
27817 173/300 Read block 3021 len 36864
27818 174/300 Read block 4957 len 12288
27819 175/300 Read block 3618 len 28672
27820 176/300 Write block 4603 len 28672 data: 5 6 7 7 6 5 7
27821 177/300 Write block 1232 len 4096 data: 7
27822 178/300 Read block 2062 len 24576
27823 179/300 Read block 2993 len 8192
27824 180/300 Write block 458 len 40960 data: 7 7 6 6 8 8 6 5 4 6
27825 181/300 Read block 4815 len 8192
27826 182/300 Read block 2629 len 12288
27827 183/300 Write block 2403 len 36864 data: 8 8 6 6 7 8 8 7 5
27828 184/300 Write block 2959 len 40960 data: 4 5 5 6 6 7 7 6 7 7
27829 185/300 Read block 4914 len 12288
27830 186/300 Read block 2219 len 40960
27831 187/300 Read block 2278 len 36864
27832 188/300 Read block 533 len 36864
27833 189/300 Read block 3435 len 16384
27834 190/300 Write block 2838 len 20480 data: 7 6 7 7 7
27835 191/300 Read block 3632 len 32768
27836 192/300 Read block 3882 len 24576
27837 193/300 Read block 1242 len 40960
27838 194/300 Read block 621 len 28672
27839 195/300 Read block 1741 len 16384
27840 196/300 Write block 1586 len 40960 data: 5 5 4 5 5 4 4 6 8 8
27841 197/300 Write block 4238 len 20480 data: 4 4 4 3 4
27842 198/300 Read block 1395 len 40960
27843 199/300 Read block 424 len 40960
27844 200/300 Flush
27845 201/300 Read block 1914 len 24576
27846 202/300 Read block 4630 len 28672
27847 203/300 Read block 4694 len 32768
27848 204/300 Write block 1947 len 12288 data: 4 4 4
27849 205/300 Flush
27850 206/300 Read block 1648 len 24576
27851 207/300 Write block 2356 len 16384 data: 3 3 3 4
27852 208/300 Write block 3386 len 24576 data: 9 10 8 7 6 4
27853 209/300 Read block 4698 len 32768
27854 210/300 Read block 559 len 16384
27855 211/300 Read block 2049 len 16384
27856 212/300 Read block 207 len 40960
27857 213/300 Write block 3777 len 32768 data: 6 9 10 8 6 6 5 4
27858 214/300 Read block 332 len 16384
27859 215/300 Read block 3647 len 24576
27860 216/300 Write block 321 len 4096 data: 4
27861 217/300 Read block 848 len 4096
27862 218/300 Read block 1123 len 12288
27863 219/300 Read block 345 len 24576
27864 220/300 Read block 3254 len 12288
27865 221/300 Read block 2652 len 28672
27866 222/300 Write block 778 len 12288 data: 8 9 10
27867 223/300 Read block 2386 len 32768
27868 224/300 Read block 4329 len 32768
27869 225/300 Read block 3514 len 8192
27870 226/300 Read block 3972 len 24576
27871 227/300 Write block 1498 len 24576 data: 5 5 5 5 4 5
27872 228/300 Read block 3592 len 4096
27873 229/300 Write block 4411 len 20480 data: 2 2 4 5 5
27874 230/300 Flush
27875 231/300 Read block 3049 len 36864
27876 232/300 Write block 1958 len 20480 data: 7 7 4 5 4
27877 233/300 Read block 631 len 20480
27878 234/300 Read block 4980 len 32768
27879 235/300 Write block 4419 len 32768 data: 6 6 7 7 8 10 7 6
27880 236/300 Read block 4326 len 8192
27881 237/300 Read block 1169 len 40960
27882 238/300 Write block 4090 len 16384 data: 7 6 8 9
27883 239/300 Read block 4669 len 12288
27884 240/300 Write block 4467 len 12288 data: 3 3 3
27885 241/300 Write block 3551 len 32768 data: 3 3 3 4 5 7 6 7
27886 242/300 Flush
27887 243/300 Write block 2726 len 24576 data: 6 5 4 3 3 6
27888 244/300 Flush
27889 245/300 Read block 1531 len 36864
27890 246/300 Read block 4204 len 20480
27891 247/300 Read block 822 len 12288
27892 248/300 Read block 4935 len 4096
27893 249/300 Read block 288 len 8192
27894 250/300 Read block 2918 len 36864
27895 251/300 Read block 1285 len 4096
27896 252/300 Read block 1786 len 12288
27897 253/300 Read block 1573 len 12288
27898 254/300 Write block 4404 len 36864 data: 4 4 2 2 2 2 2 3 3
27899 255/300 Flush
27900 256/300 Write block 4561 len 4096 data: 7
27901 257/300 Read block 1547 len 16384
27902 258/300 Flush
27903 259/300 Read block 4876 len 24576
27904 260/300 Read block 3506 len 36864
27905 261/300 Read block 4556 len 8192
27906 262/300 Read block 2817 len 28672
27907 263/300 Read block 3895 len 40960
27908 264/300 Write block 338 len 40960 data: 5 7 7 5 5 5 5 5 4 5
27909 265/300 Read block 3839 len 36864
27910 266/300 Read block 3718 len 32768
27911 267/300 Write block 678 len 4096 data: 6
27912 268/300 Read block 431 len 20480
27913 269/300 Flush
27914 270/300 Read block 768 len 28672
27915 271/300 Write block 2571 len 24576 data: 4 5 5 7 8 7
27916 272/300 Read block 2062 len 4096
27917 273/300 Write block 1033 len 16384 data: 4 4 4 6
27918 274/300 Read block 2652 len 20480
27919 275/300 Read block 4677 len 12288
27920 276/300 Write block 2807 len 4096 data: 2
27921 277/300 Read block 3373 len 8192
27922 278/300 Read block 36 len 28672
27923 279/300 Read block 3848 len 12288
27924 280/300 Read block 3278 len 20480
27925 281/300 Write block 4386 len 12288 data: 5 5 4
27926 282/300 Write block 1820 len 40960 data: 8 7 5 5 5 4 5 5 7 6
27927 283/300 Read block 1935 len 32768
27928 284/300 Read block 2849 len 4096
27929 285/300 Read block 434 len 16384
27930 286/300 Write block 725 len 8192 data: 4 5
27931 287/300 Read block 1311 len 12288
27932 288/300 Read block 246 len 16384
27933 289/300 Read block 1415 len 16384
27934 290/300 Write block 3000 len 8192 data: 8 10
27935 291/300 Write block 681 len 32768 data: 3 3 2 2 2 2 2 3
27936 292/300 Read block 1928 len 8192
27937 293/300 Read block 947 len 4096
27938 294/300 Write block 576 len 28672 data: 3 5 5 4 5 5 4
27939 295/300 Read block 2740 len 24576
27940 296/300 Read block 32 len 28672
27941 297/300 Write block 2938 len 4096 data: 5
27942 298/300 Write block 1721 len 40960 data: 4 3 3 3 3 4 4 4 4 5
27943 299/300 Read block 4030 len 28672
27944 300/300 Read block 3675 len 28672
27945 301/300 Write block 4835 len 24576 data: 8 7 6 5 4 3
27946 Main task has work to do, go find it
27947 got dsc Start(0)
27948 start 0
27949 Jan 23 18:53:47.442 INFO request completed, latency_us: 259, response_code: 204, uri: /start/cid/0, method: POST, req_id: 15cf9022-9357-465b-917e-bfba0e493b1a, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
27950 Replay: started 0, returned:Ok(())
27951 ----------------------------------------------------------------
27952 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27953 States: Offline Active Active
27954 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27955 8547 Acked Read 6 Sent Done Done false
27956 8548 Acked Read 6 Sent Done Done false
27957 8549 Acked Write 4 Sent Done Done false
27958 8550 Acked Read 10 Sent Done Done false
27959 8551 Acked Read 2 Sent Done Done false
27960 8552 Acked Flush 0 Sent Done Done false
27961 8553 Acked Write 2 Sent Done Done false
27962 8554 Acked Read 4 Sent Done Done false
27963 8555 Acked Write 4 Sent Done Done false
27964 8556 Acked Write 10 Sent Done Done false
27965 8557 Acked Write 6 Sent Done Done false
27966 8558 Acked Read 5 Sent Done Done false
27967 8559 Acked Write 4 Sent Done Done false
27968 8560 Acked Write 3 Sent Done Done false
27969 8561 Acked Read 1 Sent Done Done false
27970 8562 Acked Write 7 Sent Done Done false
27971 8563 Acked Flush 0 Sent Done Done false
27972 8564 Acked Write 2 Sent Done Done false
27973 8565 Acked Read 6 Sent Done Done false
27974 8566 Acked Write 7 Sent Done Done false
27975 8567 Acked Read 3 Sent Done Done false
27976 8568 Acked Read 6 Sent Done Done false
27977 8569 Acked Write 4 Sent Done Done false
27978 8570 Acked Read 9 Sent Done Done false
27979 8571 Acked Write 7 Sent Done Done false
27980 8572 Acked Read 6 Sent Done Done false
27981 8573 Acked Write 8 Sent Done Done false
27982 8574 Acked Read 3 Sent Done Done false
27983 8575 Acked Write 4 Sent Done Done false
27984 8576 Acked Write 10 Sent Done Done false
27985 8577 Acked Read 1 Sent Done Done false
27986 8578 Acked Read 2 Sent Done Done false
27987 8579 Acked Write 7 Sent Done Done false
27988 8580 Acked Flush 0 Sent Done Done false
27989 8581 Acked Read 6 Sent Done Done false
27990 8582 Acked Flush 0 Sent Done Done false
27991 8583 Acked Flush 0 Sent Done Done false
27992 8584 Acked Read 9 Sent Done Done false
27993 8585 Acked Read 5 Sent Done Done false
27994 8586 Acked Flush 0 Sent Done Done false
27995 8587 Acked Read 5 Sent Done Done false
27996 8588 Acked Read 1 Sent Done Done false
27997 8589 Acked Read 7 Sent Done Done false
27998 8590 Acked Write 8 Sent Done Done false
27999 8591 Acked Write 8 Sent Done Done false
28000 8592 Acked Write 5 Sent Done Done false
28001 8593 Acked Read 3 Sent Done Done false
28002 8594 Acked Write 3 Sent Done Done false
28003 8595 Acked Read 4 Sent Done Done false
28004 8596 Acked Write 2 Sent Done Done false
28005 8597 Acked Read 4 Sent Done Done false
28006 8598 Acked Write 2 Sent Done Done false
28007 8599 Acked Read 4 Sent Done Done false
28008 8600 Acked Read 6 Sent Done Done false
28009 8601 Acked Write 9 Sent Done Done false
28010 8602 Acked Flush 0 Sent Done Done false
28011 8603 Acked Read 2 Sent Done Done false
28012 8604 Acked Read 4 Sent Done Done false
28013 8605 Acked Read 5 Sent Done Done false
28014 8606 Acked Read 9 Sent Done Done false
28015 8607 Acked Read 4 Sent Done Done false
28016 8608 Acked Write 7 Sent Done Done false
28017 8609 Acked Read 10 Sent Done Done false
28018 8610 Acked Flush 0 Sent Done Done false
28019 8611 Acked Write 10 Sent Done Done false
28020 8612 Acked Read 8 Sent Done Done false
28021 8613 Acked Flush 0 Sent Done Done false
28022 8614 Acked Read 3 Sent Done Done false
28023 8615 Acked Write 7 Sent Done Done false
28024 8616 Acked Read 5 Sent Done Done false
28025 8617 Acked Read 2 Sent Done Done false
28026 8618 Acked Read 8 Sent Done Done false
28027 8619 Acked Flush 0 Sent Done Done false
28028 8620 Acked Write 10 Sent Done Done false
28029 8621 Acked Write 5 Sent Done Done false
28030 8622 Acked Read 2 Sent Done Done false
28031 8623 Acked Read 1 Sent Done Done false
28032 8624 Acked Write 1 Sent Done Done false
28033 8625 Acked Read 5 Sent Done Done false
28034 8626 Acked Read 9 Sent Done Done false
28035 8627 Acked Read 4 Sent Done Done false
28036 8628 Acked Read 8 Sent Done Done false
28037 8629 Acked Read 7 Sent Done Done false
28038 8630 Acked Write 3 Sent Done Done false
28039 8631 Acked Flush 0 Sent Done Done false
28040 8632 Acked Read 4 Sent Done Done false
28041 8633 Acked Write 9 Sent Done Done false
28042 8634 Acked Read 2 Sent Done Done false
28043 8635 Acked Write 5 Sent Done Done false
28044 8636 Acked Read 8 Sent Done Done false
28045 8637 Acked Read 10 Sent Done Done false
28046 8638 Acked Write 5 Sent Done Done false
28047 8639 Acked Read 9 Sent Done Done false
28048 8640 Acked Write 6 Sent Done Done false
28049 8641 Acked Read 4 Sent Done Done false
28050 8642 Acked Flush 0 Sent Done Done false
28051 8643 Acked Read 1 Sent Done Done false
28052 8644 Acked Write 5 Sent Done Done false
28053 8645 Acked Read 10 Sent Done Done false
28054 8646 Acked Write 6 Sent Done Done false
28055 8647 Acked Read 4 Sent Done Done false
28056 8648 Acked Write 7 Sent Done Done false
28057 8649 Acked Write 1 Sent Done Done false
28058 8650 Acked Read 5 Sent Done Done false
28059 8651 Acked Read 10 Sent Done Done false
28060 8652 Acked Read 4 Sent Done Done false
28061 8653 Acked Write 3 Sent Done Done false
28062 8654 Acked Read 5 Sent Done Done false
28063 8655 Acked Read 1 Sent Done Done false
28064 8656 Acked Flush 0 Sent Done Done false
28065 8657 Acked Read 5 Sent Done Done false
28066 8658 Acked Read 10 Sent Done Done false
28067 8659 Acked Read 10 Sent Done Done false
28068 8660 Acked Flush 0 Sent Done Done false
28069 8661 Acked Read 9 Sent Done Done false
28070 8662 Acked Read 6 Sent Done Done false
28071 8663 Acked Read 6 Sent Done Done false
28072 8664 Acked Write 8 Sent Done Done false
28073 8665 Acked Flush 0 Sent Done Done false
28074 8666 Acked Write 2 Sent Done Done false
28075 8667 Acked Write 5 Sent Done Done false
28076 8668 Acked Write 10 Sent Done Done false
28077 8669 Acked Read 8 Sent Done Done false
28078 8670 Acked Read 8 Sent Done Done false
28079 8671 Acked Read 2 Sent Done Done false
28080 8672 Acked Write 8 Sent Done Done false
28081 8673 Acked Read 3 Sent Done Done false
28082 8674 Acked Write 7 Sent Done Done false
28083 8675 Acked Read 1 Sent Done Done false
28084 8676 Acked Write 3 Sent Done Done false
28085 8677 Acked Read 1 Sent Done Done false
28086 8678 Acked Write 6 Sent Done Done false
28087 8679 Acked Read 6 Sent Done Done false
28088 8680 Acked Write 6 Sent Done Done false
28089 8681 Acked Flush 0 Sent Done Done false
28090 8682 Acked Read 10 Sent Done Done false
28091 8683 Acked Read 5 Sent Done Done false
28092 8684 Acked Write 4 Sent Done Done false
28093 8685 Acked Flush 0 Sent Done Done false
28094 8686 Acked Write 3 Sent Done Done false
28095 8687 Acked Write 1 Sent Done Done false
28096 8688 Acked Flush 0 Sent Done Done false
28097 8689 Acked Read 3 Sent Done Done false
28098 8690 Acked Write 1 Sent Done Done false
28099 8691 Acked Write 8 Sent Done Done false
28100 8692 Acked Read 7 Sent Done Done false
28101 8693 Acked Write 6 Sent Done Done false
28102 8694 Acked Read 1 Sent Done Done false
28103 8695 Acked Write 1 Sent Done Done false
28104 8696 Acked Read 1 Sent Done Done false
28105 8697 Acked Write 6 Sent Done Done false
28106 8698 Acked Read 6 Sent Done Done false
28107 8699 Acked Write 4 Sent Done Done false
28108 8700 Acked Read 7 Sent Done Done false
28109 8701 Acked Write 2 Sent Done Done false
28110 8702 Acked Read 10 Sent Done Done false
28111 8703 Acked Read 2 Sent Done Done false
28112 8704 Acked Write 9 Sent Done Done false
28113 8705 Acked Read 3 Sent Done Done false
28114 8706 Acked Write 3 Sent Done Done false
28115 8707 Acked Write 1 Sent Done Done false
28116 8708 Acked Write 1 Sent Done Done false
28117 8709 Acked Read 9 Sent Done Done false
28118 8710 Acked Write 4 Sent Done Done false
28119 8711 Acked Read 4 Sent Done Done false
28120 8712 Acked Write 3 Sent Done Done false
28121 8713 Acked Write 3 Sent Done Done false
28122 8714 Acked Read 4 Sent Done Done false
28123 8715 Acked Read 9 Sent Done Done false
28124 8716 Acked Write 1 Sent Done Done false
28125 8717 Acked Write 1 Sent Done Done false
28126 8718 Acked Flush 0 Sent Done Done false
28127 8719 Acked Read 9 Sent Done Done false
28128 8720 Acked Read 3 Sent Done Done false
28129 8721 Acked Read 7 Sent Done Done false
28130 8722 Acked Write 7 Sent Done Done false
28131 8723 Acked Write 1 Sent Done Done false
28132 8724 Acked Read 6 Sent Done Done false
28133 8725 Acked Read 2 Sent Done Done false
28134 8726 Acked Write 10 Sent Done Done false
28135 8727 Acked Read 2 Sent Done Done false
28136 8728 Acked Read 3 Sent Done Done false
28137 8729 Acked Write 9 Sent Done Done false
28138 8730 Acked Write 10 Sent Done Done false
28139 8731 Acked Read 3 Sent Done Done false
28140 8732 Acked Read 10 Sent Done Done false
28141 8733 Acked Read 9 Sent Done Done false
28142 8734 Acked Read 9 Sent Done Done false
28143 8735 Acked Read 4 Sent Done Done false
28144 8736 Acked Write 5 Sent Done Done false
28145 8737 Acked Read 8 Sent Done Done false
28146 8738 Acked Read 6 Sent Done Done false
28147 8739 Acked Read 10 Sent Done Done false
28148 8740 Acked Read 7 Sent Done Done false
28149 8741 Acked Read 4 Sent Done Done false
28150 8742 Acked Write 10 Sent Done Done false
28151 8743 Acked Write 5 Sent Done Done false
28152 8744 Acked Read 10 Sent Done Done false
28153 8745 Acked Read 10 Sent Done Done false
28154 8746 Acked Flush 0 Sent Done Done false
28155 8747 Acked Read 6 Sent Done Done false
28156 8748 Acked Read 7 Sent Done Done false
28157 8749 Acked Read 8 Sent Done Done false
28158 8750 Acked Write 3 Sent Done Done false
28159 8751 Acked Flush 0 Sent Done Done false
28160 8752 Acked Read 6 Sent Done Done false
28161 8753 Acked Write 4 Sent Done Done false
28162 8754 Acked Write 6 Sent Done Done false
28163 8755 Acked Read 8 Sent Done Done false
28164 8756 Acked Read 4 Sent Done Done false
28165 8757 Acked Read 4 Sent Done Done false
28166 8758 Acked Read 10 Sent Done Done false
28167 8759 Acked Write 8 Sent Done Done false
28168 8760 Acked Read 4 Sent Done Done false
28169 8761 Acked Read 6 Sent Done Done false
28170 8762 Acked Write 1 Sent Done Done false
28171 8763 Acked Read 1 Sent Done Done false
28172 8764 Acked Read 3 Sent Done Done false
28173 8765 Acked Read 6 Sent Done Done false
28174 8766 Acked Read 3 Sent Done Done false
28175 8767 Acked Read 7 Sent Done Done false
28176 8768 Acked Write 3 Sent Done Done false
28177 8769 Acked Read 8 Sent Done Done false
28178 8770 Acked Read 8 Sent Done Done false
28179 8771 Acked Read 2 Sent Done Done false
28180 8772 Acked Read 6 Sent Done Done false
28181 8773 Acked Write 6 Sent Done Done false
28182 8774 Acked Read 1 Sent Done Done false
28183 8775 Acked Write 5 Sent Done Done false
28184 8776 Acked Flush 0 Sent Done Done false
28185 8777 Acked Read 9 Sent Done Done false
28186 8778 Acked Write 5 Sent Done Done false
28187 8779 Acked Read 5 Sent Done Done false
28188 8780 Acked Read 8 Sent Done Done false
28189 8781 Acked Write 8 Sent Done Done false
28190 8782 Acked Read 2 Sent Done Done false
28191 8783 Acked Read 10 Sent Done Done false
28192 8784 Acked Write 4 Sent Done Done false
28193 8785 Acked Read 3 Sent Done Done false
28194 8786 Acked Write 3 Sent Done Done false
28195 8787 Acked Write 8 Sent Done Done false
28196 8788 Acked Flush 0 Sent Done Done false
28197 8789 Acked Write 6 Sent Done Done false
28198 8790 Acked Flush 0 Sent Done Done false
28199 8791 Acked Read 9 Sent Done Done false
28200 8792 Acked Read 5 Sent Done Done false
28201 8793 Acked Read 3 Sent Done Done false
28202 8794 Acked Read 1 Sent Done Done false
28203 8795 Acked Read 2 Sent Done Done false
28204 8796 Acked Read 9 Sent Done Done false
28205 8797 Acked Read 1 Sent Done Done false
28206 8798 Acked Read 3 Sent Done Done false
28207 8799 Acked Read 3 Sent Done Done false
28208 8800 Acked Write 9 Sent Done Done false
28209 8801 Acked Flush 0 Sent Done Done false
28210 8802 Acked Write 1 Sent Done Done false
28211 8803 Acked Read 4 Sent Done Done false
28212 8804 Acked Flush 0 Sent Done Done false
28213 8805 Acked Read 6 Sent Done Done false
28214 8806 Acked Read 9 Sent Done Done false
28215 8807 Acked Read 2 Sent Done Done false
28216 8808 Acked Read 7 Sent Done Done false
28217 8809 Acked Read 10 Sent Done Done false
28218 8810 Acked Write 10 Sent Done Done false
28219 8811 Acked Read 9 Sent Done Done false
28220 8812 Acked Read 8 Sent Done Done false
28221 8813 Acked Write 1 Sent Done Done false
28222 8814 Acked Read 5 Sent Done Done false
28223 8815 Acked Flush 0 Sent Done Done false
28224 8816 Acked Read 7 Sent Done Done false
28225 8817 Acked Write 6 Sent Done Done false
28226 8818 Acked Read 1 Sent Done Done false
28227 8819 Acked Write 4 Sent Done Done false
28228 8820 Acked Read 5 Sent Done Done false
28229 8821 Acked Read 3 Sent Done Done false
28230 8822 Acked Write 1 Sent Done Done false
28231 8823 Acked Read 2 Sent Done Done false
28232 8824 Acked Read 7 Sent Done Done false
28233 8825 Acked Read 3 Sent Done Done false
28234 8826 Acked Read 5 Sent Done Done false
28235 8827 Acked Write 3 Sent Done Done false
28236 8828 Acked Write 10 Sent Done Done false
28237 8829 Acked Read 8 Sent Done Done false
28238 8830 Acked Read 1 Sent Done Done false
28239 8831 Acked Read 4 Sent Done Done false
28240 8832 Acked Write 2 Sent Done Done false
28241 8833 Acked Read 3 Sent Done Done false
28242 8834 Acked Read 4 Sent Done Done false
28243 8835 Acked Read 4 Sent Done Done false
28244 8836 Acked Write 2 Sent Done Done false
28245 8837 Acked Write 8 Sent Done Done false
28246 8838 Acked Read 2 Sent Done Done false
28247 8839 Acked Read 1 Sent Done Done false
28248 8840 Acked Write 7 Sent Done Done false
28249 8841 Acked Read 6 Sent Done Done false
28250 8842 Acked Read 7 Sent Done Done false
28251 8843 Acked Write 1 Sent Done Done false
28252 8844 Acked Write 10 Sent Done Done false
28253 8845 Acked Read 7 Sent Done Done false
28254 8846 Acked Read 7 Sent Done Done false
28255 8847 Acked Write 6 Sent Done Done false
28256 STATES DS:0 DS:1 DS:2 TOTAL
28257 Sent 301 0 0 301
28258 Done 0 301 301 602
28259 Skipped 0 0 0 0
28260 Error 0 0 0 0
28261 Last Flush: Some(JobId(8546)) Some(JobId(8815)) Some(JobId(8815))
28262 Downstairs last five completed: 8546 8545 8544 8543 8542
28263 Upstairs last five completed: 8847 8846 8845 8844 8843
28264 CLIENT: Up:0 ds:301 act:2
28265 [8810] got start action
28266 [8810] I am going to restart
28267 Starting downstairs at port 8810
28268 Make output file at "/tmp/dsc/downstairs-8810.txt"
28269 [8810][0] reports Starting
28270 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1226)
28271 [8810][0] reports Running
28272 ----------------------------------------------------------------
28273 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
28274 States: Offline Active Active
28275 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28276 8547 Acked Read 6 Sent Done Done false
28277 8548 Acked Read 6 Sent Done Done false
28278 8549 Acked Write 4 Sent Done Done false
28279 8550 Acked Read 10 Sent Done Done false
28280 8551 Acked Read 2 Sent Done Done false
28281 8552 Acked Flush 0 Sent Done Done false
28282 8553 Acked Write 2 Sent Done Done false
28283 8554 Acked Read 4 Sent Done Done false
28284 8555 Acked Write 4 Sent Done Done false
28285 8556 Acked Write 10 Sent Done Done false
28286 8557 Acked Write 6 Sent Done Done false
28287 8558 Acked Read 5 Sent Done Done false
28288 8559 Acked Write 4 Sent Done Done false
28289 8560 Acked Write 3 Sent Done Done false
28290 8561 Acked Read 1 Sent Done Done false
28291 8562 Acked Write 7 Sent Done Done false
28292 8563 Acked Flush 0 Sent Done Done false
28293 8564 Acked Write 2 Sent Done Done false
28294 8565 Acked Read 6 Sent Done Done false
28295 8566 Acked Write 7 Sent Done Done false
28296 8567 Acked Read 3 Sent Done Done false
28297 8568 Acked Read 6 Sent Done Done false
28298 8569 Acked Write 4 Sent Done Done false
28299 8570 Acked Read 9 Sent Done Done false
28300 8571 Acked Write 7 Sent Done Done false
28301 8572 Acked Read 6 Sent Done Done false
28302 8573 Acked Write 8 Sent Done Done false
28303 8574 Acked Read 3 Sent Done Done false
28304 8575 Acked Write 4 Sent Done Done false
28305 8576 Acked Write 10 Sent Done Done false
28306 8577 Acked Read 1 Sent Done Done false
28307 8578 Acked Read 2 Sent Done Done false
28308 8579 Acked Write 7 Sent Done Done false
28309 8580 Acked Flush 0 Sent Done Done false
28310 8581 Acked Read 6 Sent Done Done false
28311 8582 Acked Flush 0 Sent Done Done false
28312 8583 Acked Flush 0 Sent Done Done false
28313 8584 Acked Read 9 Sent Done Done false
28314 8585 Acked Read 5 Sent Done Done false
28315 8586 Acked Flush 0 Sent Done Done false
28316 8587 Acked Read 5 Sent Done Done false
28317 8588 Acked Read 1 Sent Done Done false
28318 8589 Acked Read 7 Sent Done Done false
28319 8590 Acked Write 8 Sent Done Done false
28320 8591 Acked Write 8 Sent Done Done false
28321 8592 Acked Write 5 Sent Done Done false
28322 8593 Acked Read 3 Sent Done Done false
28323 8594 Acked Write 3 Sent Done Done false
28324 8595 Acked Read 4 Sent Done Done false
28325 8596 Acked Write 2 Sent Done Done false
28326 8597 Acked Read 4 Sent Done Done false
28327 8598 Acked Write 2 Sent Done Done false
28328 8599 Acked Read 4 Sent Done Done false
28329 8600 Acked Read 6 Sent Done Done false
28330 8601 Acked Write 9 Sent Done Done false
28331 8602 Acked Flush 0 Sent Done Done false
28332 8603 Acked Read 2 Sent Done Done false
28333 8604 Acked Read 4 Sent Done Done false
28334 8605 Acked Read 5 Sent Done Done false
28335 8606 Acked Read 9 Sent Done Done false
28336 8607 Acked Read 4 Sent Done Done false
28337 8608 Acked Write 7 Sent Done Done false
28338 8609 Acked Read 10 Sent Done Done false
28339 8610 Acked Flush 0 Sent Done Done false
28340 8611 Acked Write 10 Sent Done Done false
28341 8612 Acked Read 8 Sent Done Done false
28342 8613 Acked Flush 0 Sent Done Done false
28343 8614 Acked Read 3 Sent Done Done false
28344 8615 Acked Write 7 Sent Done Done false
28345 8616 Acked Read 5 Sent Done Done false
28346 8617 Acked Read 2 Sent Done Done false
28347 8618 Acked Read 8 Sent Done Done false
28348 8619 Acked Flush 0 Sent Done Done false
28349 8620 Acked Write 10 Sent Done Done false
28350 8621 Acked Write 5 Sent Done Done false
28351 8622 Acked Read 2 Sent Done Done false
28352 8623 Acked Read 1 Sent Done Done false
28353 8624 Acked Write 1 Sent Done Done false
28354 8625 Acked Read 5 Sent Done Done false
28355 8626 Acked Read 9 Sent Done Done false
28356 8627 Acked Read 4 Sent Done Done false
28357 8628 Acked Read 8 Sent Done Done false
28358 8629 Acked Read 7 Sent Done Done false
28359 8630 Acked Write 3 Sent Done Done false
28360 8631 Acked Flush 0 Sent Done Done false
28361 8632 Acked Read 4 Sent Done Done false
28362 8633 Acked Write 9 Sent Done Done false
28363 8634 Acked Read 2 Sent Done Done false
28364 8635 Acked Write 5 Sent Done Done false
28365 8636 Acked Read 8 Sent Done Done false
28366 8637 Acked Read 10 Sent Done Done false
28367 8638 Acked Write 5 Sent Done Done false
28368 8639 Acked Read 9 Sent Done Done false
28369 8640 Acked Write 6 Sent Done Done false
28370 8641 Acked Read 4 Sent Done Done false
28371 8642 Acked Flush 0 Sent Done Done false
28372 8643 Acked Read 1 Sent Done Done false
28373 8644 Acked Write 5 Sent Done Done false
28374 8645 Acked Read 10 Sent Done Done false
28375 8646 Acked Write 6 Sent Done Done false
28376 8647 Acked Read 4 Sent Done Done false
28377 8648 Acked Write 7 Sent Done Done false
28378 8649 Acked Write 1 Sent Done Done false
28379 8650 Acked Read 5 Sent Done Done false
28380 8651 Acked Read 10 Sent Done Done false
28381 8652 Acked Read 4 Sent Done Done false
28382 8653 Acked Write 3 Sent Done Done false
28383 8654 Acked Read 5 Sent Done Done false
28384 8655 Acked Read 1 Sent Done Done false
28385 8656 Acked Flush 0 Sent Done Done false
28386 8657 Acked Read 5 Sent Done Done false
28387 8658 Acked Read 10 Sent Done Done false
28388 8659 Acked Read 10 Sent Done Done false
28389 8660 Acked Flush 0 Sent Done Done false
28390 8661 Acked Read 9 Sent Done Done false
28391 8662 Acked Read 6 Sent Done Done false
28392 8663 Acked Read 6 Sent Done Done false
28393 8664 Acked Write 8 Sent Done Done false
28394 8665 Acked Flush 0 Sent Done Done false
28395 8666 Acked Write 2 Sent Done Done false
28396 8667 Acked Write 5 Sent Done Done false
28397 8668 Acked Write 10 Sent Done Done false
28398 8669 Acked Read 8 Sent Done Done false
28399 8670 Acked Read 8 Sent Done Done false
28400 8671 Acked Read 2 Sent Done Done false
28401 8672 Acked Write 8 Sent Done Done false
28402 8673 Acked Read 3 Sent Done Done false
28403 8674 Acked Write 7 Sent Done Done false
28404 8675 Acked Read 1 Sent Done Done false
28405 8676 Acked Write 3 Sent Done Done false
28406 8677 Acked Read 1 Sent Done Done false
28407 8678 Acked Write 6 Sent Done Done false
28408 8679 Acked Read 6 Sent Done Done false
28409 8680 Acked Write 6 Sent Done Done false
28410 8681 Acked Flush 0 Sent Done Done false
28411 8682 Acked Read 10 Sent Done Done false
28412 8683 Acked Read 5 Sent Done Done false
28413 8684 Acked Write 4 Sent Done Done false
28414 8685 Acked Flush 0 Sent Done Done false
28415 8686 Acked Write 3 Sent Done Done false
28416 8687 Acked Write 1 Sent Done Done false
28417 8688 Acked Flush 0 Sent Done Done false
28418 8689 Acked Read 3 Sent Done Done false
28419 8690 Acked Write 1 Sent Done Done false
28420 8691 Acked Write 8 Sent Done Done false
28421 8692 Acked Read 7 Sent Done Done false
28422 8693 Acked Write 6 Sent Done Done false
28423 8694 Acked Read 1 Sent Done Done false
28424 8695 Acked Write 1 Sent Done Done false
28425 8696 Acked Read 1 Sent Done Done false
28426 8697 Acked Write 6 Sent Done Done false
28427 8698 Acked Read 6 Sent Done Done false
28428 8699 Acked Write 4 Sent Done Done false
28429 8700 Acked Read 7 Sent Done Done false
28430 8701 Acked Write 2 Sent Done Done false
28431 8702 Acked Read 10 Sent Done Done false
28432 8703 Acked Read 2 Sent Done Done false
28433 8704 Acked Write 9 Sent Done Done false
28434 8705 Acked Read 3 Sent Done Done false
28435 8706 Acked Write 3 Sent Done Done false
28436 8707 Acked Write 1 Sent Done Done false
28437 8708 Acked Write 1 Sent Done Done false
28438 8709 Acked Read 9 Sent Done Done false
28439 8710 Acked Write 4 Sent Done Done false
28440 8711 Acked Read 4 Sent Done Done false
28441 8712 Acked Write 3 Sent Done Done false
28442 8713 Acked Write 3 Sent Done Done false
28443 8714 Acked Read 4 Sent Done Done false
28444 8715 Acked Read 9 Sent Done Done false
28445 8716 Acked Write 1 Sent Done Done false
28446 8717 Acked Write 1 Sent Done Done false
28447 8718 Acked Flush 0 Sent Done Done false
28448 8719 Acked Read 9 Sent Done Done false
28449 8720 Acked Read 3 Sent Done Done false
28450 8721 Acked Read 7 Sent Done Done false
28451 8722 Acked Write 7 Sent Done Done false
28452 8723 Acked Write 1 Sent Done Done false
28453 8724 Acked Read 6 Sent Done Done false
28454 8725 Acked Read 2 Sent Done Done false
28455 8726 Acked Write 10 Sent Done Done false
28456 8727 Acked Read 2 Sent Done Done false
28457 8728 Acked Read 3 Sent Done Done false
28458 8729 Acked Write 9 Sent Done Done false
28459 8730 Acked Write 10 Sent Done Done false
28460 8731 Acked Read 3 Sent Done Done false
28461 8732 Acked Read 10 Sent Done Done false
28462 8733 Acked Read 9 Sent Done Done false
28463 8734 Acked Read 9 Sent Done Done false
28464 8735 Acked Read 4 Sent Done Done false
28465 8736 Acked Write 5 Sent Done Done false
28466 8737 Acked Read 8 Sent Done Done false
28467 8738 Acked Read 6 Sent Done Done false
28468 8739 Acked Read 10 Sent Done Done false
28469 8740 Acked Read 7 Sent Done Done false
28470 8741 Acked Read 4 Sent Done Done false
28471 8742 Acked Write 10 Sent Done Done false
28472 8743 Acked Write 5 Sent Done Done false
28473 8744 Acked Read 10 Sent Done Done false
28474 8745 Acked Read 10 Sent Done Done false
28475 8746 Acked Flush 0 Sent Done Done false
28476 8747 Acked Read 6 Sent Done Done false
28477 8748 Acked Read 7 Sent Done Done false
28478 8749 Acked Read 8 Sent Done Done false
28479 8750 Acked Write 3 Sent Done Done false
28480 8751 Acked Flush 0 Sent Done Done false
28481 8752 Acked Read 6 Sent Done Done false
28482 8753 Acked Write 4 Sent Done Done false
28483 8754 Acked Write 6 Sent Done Done false
28484 8755 Acked Read 8 Sent Done Done false
28485 8756 Acked Read 4 Sent Done Done false
28486 8757 Acked Read 4 Sent Done Done false
28487 8758 Acked Read 10 Sent Done Done false
28488 8759 Acked Write 8 Sent Done Done false
28489 8760 Acked Read 4 Sent Done Done false
28490 8761 Acked Read 6 Sent Done Done false
28491 8762 Acked Write 1 Sent Done Done false
28492 8763 Acked Read 1 Sent Done Done false
28493 8764 Acked Read 3 Sent Done Done false
28494 8765 Acked Read 6 Sent Done Done false
28495 8766 Acked Read 3 Sent Done Done false
28496 8767 Acked Read 7 Sent Done Done false
28497 8768 Acked Write 3 Sent Done Done false
28498 8769 Acked Read 8 Sent Done Done false
28499 8770 Acked Read 8 Sent Done Done false
28500 8771 Acked Read 2 Sent Done Done false
28501 8772 Acked Read 6 Sent Done Done false
28502 8773 Acked Write 6 Sent Done Done false
28503 8774 Acked Read 1 Sent Done Done false
28504 8775 Acked Write 5 Sent Done Done false
28505 8776 Acked Flush 0 Sent Done Done false
28506 8777 Acked Read 9 Sent Done Done false
28507 8778 Acked Write 5 Sent Done Done false
28508 8779 Acked Read 5 Sent Done Done false
28509 8780 Acked Read 8 Sent Done Done false
28510 8781 Acked Write 8 Sent Done Done false
28511 8782 Acked Read 2 Sent Done Done false
28512 8783 Acked Read 10 Sent Done Done false
28513 8784 Acked Write 4 Sent Done Done false
28514 8785 Acked Read 3 Sent Done Done false
28515 8786 Acked Write 3 Sent Done Done false
28516 8787 Acked Write 8 Sent Done Done false
28517 8788 Acked Flush 0 Sent Done Done false
28518 8789 Acked Write 6 Sent Done Done false
28519 8790 Acked Flush 0 Sent Done Done false
28520 8791 Acked Read 9 Sent Done Done false
28521 8792 Acked Read 5 Sent Done Done false
28522 8793 Acked Read 3 Sent Done Done false
28523 8794 Acked Read 1 Sent Done Done false
28524 8795 Acked Read 2 Sent Done Done false
28525 8796 Acked Read 9 Sent Done Done false
28526 8797 Acked Read 1 Sent Done Done false
28527 8798 Acked Read 3 Sent Done Done false
28528 8799 Acked Read 3 Sent Done Done false
28529 8800 Acked Write 9 Sent Done Done false
28530 8801 Acked Flush 0 Sent Done Done false
28531 8802 Acked Write 1 Sent Done Done false
28532 8803 Acked Read 4 Sent Done Done false
28533 8804 Acked Flush 0 Sent Done Done false
28534 8805 Acked Read 6 Sent Done Done false
28535 8806 Acked Read 9 Sent Done Done false
28536 8807 Acked Read 2 Sent Done Done false
28537 8808 Acked Read 7 Sent Done Done false
28538 8809 Acked Read 10 Sent Done Done false
28539 8810 Acked Write 10 Sent Done Done false
28540 8811 Acked Read 9 Sent Done Done false
28541 8812 Acked Read 8 Sent Done Done false
28542 8813 Acked Write 1 Sent Done Done false
28543 8814 Acked Read 5 Sent Done Done false
28544 8815 Acked Flush 0 Sent Done Done false
28545 8816 Acked Read 7 Sent Done Done false
28546 8817 Acked Write 6 Sent Done Done false
28547 8818 Acked Read 1 Sent Done Done false
28548 8819 Acked Write 4 Sent Done Done false
28549 8820 Acked Read 5 Sent Done Done false
28550 8821 Acked Read 3 Sent Done Done false
28551 8822 Acked Write 1 Sent Done Done false
28552 8823 Acked Read 2 Sent Done Done false
28553 8824 Acked Read 7 Sent Done Done false
28554 8825 Acked Read 3 Sent Done Done false
28555 8826 Acked Read 5 Sent Done Done false
28556 8827 Acked Write 3 Sent Done Done false
28557 8828 Acked Write 10 Sent Done Done false
28558 8829 Acked Read 8 Sent Done Done false
28559 8830 Acked Read 1 Sent Done Done false
28560 8831 Acked Read 4 Sent Done Done false
28561 8832 Acked Write 2 Sent Done Done false
28562 8833 Acked Read 3 Sent Done Done false
28563 8834 Acked Read 4 Sent Done Done false
28564 8835 Acked Read 4 Sent Done Done false
28565 8836 Acked Write 2 Sent Done Done false
28566 8837 Acked Write 8 Sent Done Done false
28567 8838 Acked Read 2 Sent Done Done false
28568 8839 Acked Read 1 Sent Done Done false
28569 8840 Acked Write 7 Sent Done Done false
28570 8841 Acked Read 6 Sent Done Done false
28571 8842 Acked Read 7 Sent Done Done false
28572 8843 Acked Write 1 Sent Done Done false
28573 8844 Acked Write 10 Sent Done Done false
28574 8845 Acked Read 7 Sent Done Done false
28575 8846 Acked Read 7 Sent Done Done false
28576 8847 Acked Write 6 Sent Done Done false
28577 8848 Acked Flush 0 Sent Done Done false
28578 STATES DS:0 DS:1 DS:2 TOTAL
28579 Sent 302 0 0 302
28580 Done 0 302 302 604
28581 Skipped 0 0 0 0
28582 Error 0 0 0 0
28583 Last Flush: Some(JobId(8546)) Some(JobId(8848)) Some(JobId(8848))
28584 Downstairs last five completed: 8546 8545 8544 8543 8542
28585 Upstairs last five completed: 8848 8847 8846 8845 8844
28586 CLIENT: Up:0 ds:302 act:2
285872025-01-23T18:53:53.190ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
285882025-01-23T18:53:53.190ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50002) = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
285892025-01-23T18:53:53.191ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
285902025-01-23T18:53:53.191ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
285912025-01-23T18:53:53.191ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
285922025-01-23T18:53:53.191ZINFOcrucible: send last flush ID to this DS: Some(JobId(8546)) = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
285932025-01-23T18:53:53.191ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
285942025-01-23T18:53:53.191ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(8546)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
285952025-01-23T18:53:53.191ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(8546)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
28596 ----------------------------------------------------------------
28597 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28598 Downstairs last five completed: 8848 8847 8846 8845 8844
28599 Upstairs last five completed: 8848 8847 8846 8845 8844
28600 CLIENT: Up:0 ds:0 act:3
28601 Replay: All jobs finished
28602 Main task has work to do, go find it
28603 got dsc Stop(0)
28604 stop 0
28605 [8810] Got stop action so:false kr:false
28606 Jan 23 18:53:55.447 INFO request completed, latency_us: 324, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 990fa4a7-06bb-49d4-b39a-8651903bc0a1, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
286072025-01-23T18:53:55.448ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
28608 Jan 23 18:53:55.448 INFO request completed, latency_us: 149, response_code: 200, uri: /state/cid/0, method: GET, req_id: 94aee06b-65b8-401e-886b-2e076f2945d6, remote_addr: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:53:55.448502448Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
28609 127.0.0.1:59295, local_addr: {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:53:55.448526035Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
28610 127.0.0.1:9998
286112025-01-23T18:53:55.448ZINFOcrucible: client task is exiting = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
28612 [8810] Exited with: ExitStatus(unix_wait_status(9))
28613 [8810][0] reports Exit
28614 Jan 23 18:53:59.450 INFO request completed, latency_us: 215, response_code: 200, uri: /state/cid/0, method: GET, req_id: b3f5c10b-bfd7-4824-9fe0-564a8207cbb3, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
28615 001/300 Write block 1252 len 8192 data: 8 8
28616 002/300 Write block 4272 len 24576 data: 5 4 5 5 4 5
28617 003/300 Read block 1333 len 8192
28618 004/300 Read block 1548 len 40960
28619 005/300 Write block 1047 len 24576 data: 9 10 10 8 8 6
28620 006/300 Write block 1136 len 4096 data: 8
28621 007/300 Write block 4979 len 40960 data: 5 5 5 4 4 4 4 4 4 6
28622 008/300 Read block 3881 len 36864
28623 009/300 Write block 3900 len 4096 data: 7
28624 010/300 Read block 2694 len 8192
28625 011/300 Write block 20 len 4096 data: 4
28626 012/300 Read block 3689 len 36864
28627 013/300 Write block 1222 len 40960 data: 6 4 3 3 3 2 2 4 4 6
28628 014/300 Read block 805 len 28672
28629 015/300 Read block 1387 len 36864
28630 016/300 Read block 4072 len 20480
28631 017/300 Flush
28632 018/300 Read block 3422 len 24576
28633 019/300 Read block 3733 len 16384
28634 020/300 Read block 4270 len 24576
28635 021/300 Write block 1909 len 8192 data: 4 4
28636 022/300 Read block 1774 len 36864
28637 023/300 Write block 1599 len 32768 data: 5 4 3 4 4 3 3 3
28638 024/300 Write block 572 len 4096 data: 3
28639 025/300 Read block 2771 len 20480
28640 026/300 Write block 4055 len 40960 data: 5 5 4 3 4 6 6 4 5 6
28641 027/300 Write block 4387 len 20480 data: 6 5 5 5 3
28642 028/300 Read block 920 len 36864
28643 029/300 Read block 4327 len 4096
28644 030/300 Read block 496 len 20480
28645 031/300 Read block 856 len 8192
28646 032/300 Write block 4306 len 8192 data: 5 4
28647 033/300 Write block 3995 len 36864 data: 6 6 7 6 6 6 4 5 6
28648 034/300 Read block 4503 len 20480
28649 035/300 Write block 2643 len 4096 data: 2
28650 036/300 Read block 4983 len 28672
28651 037/300 Read block 2555 len 28672
28652 038/300 Write block 4574 len 20480 data: 5 7 6 7 5
28653 039/300 Write block 1837 len 12288 data: 3 2 3
28654 040/300 Read block 4332 len 28672
28655 041/300 Write block 3103 len 40960 data: 8 7 8 8 7 6 4 4 3 4
28656 042/300 Write block 2454 len 40960 data: 5 5 7 6 4 5 6 7 6 5
28657 043/300 Write block 455 len 32768 data: 6 7 7 8 8 7 7 9
28658 044/300 Write block 4520 len 36864 data: 5 5 5 6 5 4 3 3 2
28659 045/300 Read block 4461 len 4096
28660 046/300 Read block 2916 len 32768
28661 047/300 Write block 4985 len 36864 data: 5 5 5 7 4 4 4 4 5
28662 048/300 Write block 957 len 32768 data: 8 10 10 9 7 8 8 8
28663 049/300 Read block 3258 len 28672
28664 050/300 Write block 421 len 24576 data: 5 5 6 5 5 5
28665 051/300 Read block 2385 len 20480
28666 052/300 Read block 3256 len 28672
28667 053/300 Write block 3203 len 8192 data: 3 4
28668 054/300 Read block 1148 len 8192
28669 055/300 Flush
28670 056/300 Read block 2454 len 24576
28671 057/300 Read block 539 len 4096
28672 058/300 Read block 3378 len 16384
28673 059/300 Read block 1465 len 4096
28674 060/300 Write block 1086 len 24576 data: 7 7 7 8 8 7
28675 061/300 Read block 2142 len 24576
28676 062/300 Write block 4310 len 8192 data: 5 5
28677 063/300 Flush
28678 064/300 Write block 548 len 20480 data: 6 6 6 7 7
28679 065/300 Write block 519 len 16384 data: 4 5 5 5
28680 066/300 Read block 2200 len 28672
28681 067/300 Write block 3676 len 4096 data: 5
28682 068/300 Read block 2578 len 4096
28683 069/300 Write block 32 len 40960 data: 3 5 5 6 7 8 8 9 7 4
28684 070/300 Write block 2059 len 12288 data: 5 4 4
28685 071/300 Write block 4124 len 36864 data: 3 4 3 3 3 3 2 3 3
28686 072/300 Read block 3344 len 40960
28687 073/300 Flush
28688 074/300 Write block 2282 len 28672 data: 5 6 4 4 5 6 6
28689 075/300 Write block 3431 len 20480 data: 5 3 2 2 2
28690 076/300 Read block 3469 len 8192
28691 077/300 Write block 4226 len 28672 data: 3 3 4 5 5 5 4
28692 078/300 Read block 228 len 4096
28693 079/300 Write block 2966 len 16384 data: 7 8 8 6
28694 080/300 Read block 4971 len 8192
28695 081/300 Write block 618 len 16384 data: 6 6 7 8
28696 082/300 Write block 2301 len 20480 data: 5 7 8 7 7
28697 083/300 Read block 2132 len 12288
28698 084/300 Read block 1304 len 36864
28699 085/300 Write block 3768 len 36864 data: 2 3 3 3 4 6 6 6 6
28700 086/300 Read block 4714 len 28672
28701 087/300 Read block 4793 len 8192
28702 088/300 Write block 1297 len 16384 data: 4 5 5 7
28703 089/300 Read block 3429 len 4096
28704 090/300 Read block 42 len 12288
28705 091/300 Write block 4757 len 28672 data: 6 7 7 7 7 6 5
28706 092/300 Write block 3331 len 8192 data: 6 6
28707 093/300 Flush
28708 094/300 Read block 1143 len 40960
28709 095/300 Read block 2203 len 36864
28710 096/300 Read block 517 len 20480
28711 097/300 Read block 1607 len 12288
28712 098/300 Write block 1520 len 16384 data: 3 5 5 6
28713 099/300 Read block 4917 len 28672
28714 100/300 Write block 2302 len 8192 data: 8 9
28715 101/300 Read block 3343 len 12288
28716 102/300 Read block 2302 len 40960
28717 103/300 Read block 4044 len 8192
28718 104/300 Write block 205 len 32768 data: 4 4 4 3 2 2 2 2
28719 105/300 Read block 377 len 4096
28720 106/300 Read block 2829 len 4096
28721 107/300 Write block 1329 len 28672 data: 2 2 2 2 2 2 2
28722 108/300 Write block 1170 len 32768 data: 3 3 4 5 5 6 5 5
28723 109/300 Flush
28724 110/300 Read block 3799 len 40960
28725 111/300 Write block 2867 len 36864 data: 4 5 5 7 8 7 6 4 4
28726 112/300 Read block 2035 len 40960
28727 113/300 Read block 119 len 4096
28728 114/300 Read block 4105 len 28672
28729 115/300 Write block 866 len 16384 data: 6 6 7 6
28730 116/300 Write block 756 len 40960 data: 4 4 6 5 5 6 9 8 7 5
28731 117/300 Read block 3249 len 20480
28732 118/300 Read block 1487 len 8192
28733 119/300 Write block 2671 len 24576 data: 9 8 9 7 7 7
28734 120/300 Flush
28735 121/300 Read block 2584 len 4096
28736 122/300 Read block 2793 len 12288
28737 123/300 Write block 1533 len 40960 data: 6 6 4 4 4 3 3 3 4 4
28738 124/300 Read block 605 len 8192
28739 125/300 Flush
28740 126/300 Read block 3148 len 16384
28741 127/300 Read block 272 len 28672
28742 128/300 Read block 4603 len 24576
28743 129/300 Read block 2276 len 40960
28744 130/300 Write block 160 len 12288 data: 3 2 3
28745 131/300 Read block 2124 len 40960
28746 132/300 Write block 2229 len 36864 data: 7 8 8 8 8 7 5 6 6
28747 133/300 Write block 1115 len 8192 data: 6 6
28748 134/300 Read block 3848 len 32768
28749 135/300 Write block 1761 len 12288 data: 6 7 7
28750 136/300 Flush
28751 137/300 Read block 2744 len 8192
28752 138/300 Write block 622 len 20480 data: 8 6 8 7 10
28753 139/300 Flush
28754 140/300 Read block 2673 len 28672
28755 141/300 Flush
28756 142/300 Write block 4460 len 32768 data: 7 7 6 5 5 4 3 4
28757 143/300 Write block 651 len 20480 data: 5 6 6 7 6
28758 144/300 Write block 784 len 4096 data: 6
28759 145/300 Write block 3963 len 20480 data: 4 5 5 4 4
28760 146/300 Read block 1619 len 40960
28761 147/300 Write block 4971 len 40960 data: 4 4 3 4 3 4 4 5 6 6
28762 148/300 Read block 4728 len 16384
28763 149/300 Write block 1139 len 28672 data: 11 9 9 9 8 9 10
28764 150/300 Read block 151 len 12288
28765 151/300 Read block 3866 len 8192
28766 152/300 Write block 1782 len 4096 data: 6
28767 153/300 Write block 2299 len 20480 data: 6 4 6 9 10
28768 154/300 Read block 4800 len 32768
28769 155/300 Read block 3919 len 12288
28770 156/300 Write block 2696 len 32768 data: 5 4 4 6 6 5 4 5
28771 157/300 Read block 1062 len 32768
28772 158/300 Read block 2084 len 4096
28773 159/300 Read block 733 len 4096
28774 160/300 Read block 4502 len 28672
28775 161/300 Read block 3295 len 24576
28776 162/300 Read block 364 len 28672
28777 163/300 Write block 3614 len 16384 data: 5 5 5 6
28778 164/300 Flush
28779 165/300 Read block 254 len 12288
28780 166/300 Read block 4081 len 20480
28781 167/300 Read block 2699 len 32768
28782 168/300 Read block 4274 len 36864
28783 169/300 Write block 3663 len 8192 data: 6 7
28784 170/300 Write block 285 len 8192 data: 6 6
28785 171/300 Read block 3415 len 24576
28786 172/300 Flush
28787 173/300 Read block 2303 len 12288
28788 174/300 Read block 1673 len 36864
28789 175/300 Flush
28790 176/300 Read block 3635 len 20480
28791 177/300 Write block 1819 len 24576 data: 7 9 8 6 6 6
28792 178/300 Write block 4446 len 40960 data: 7 9 11 12 10 10 9 6 6 6
28793 179/300 Read block 99 len 32768
28794 180/300 Write block 131 len 32768 data: 8 7 6 7 8 8 8 10
28795 181/300 Flush
28796 182/300 Write block 3904 len 8192 data: 7 7
28797 183/300 Read block 1138 len 8192
28798 184/300 Write block 1154 len 20480 data: 6 4 3 3 5
28799 185/300 Write block 3990 len 12288 data: 3 4 5
28800 186/300 Read block 2716 len 32768
28801 187/300 Read block 1068 len 40960
28802 188/300 Read block 3475 len 4096
28803 189/300 Write block 4662 len 24576 data: 6 6 7 6 6 4
28804 190/300 Write block 2473 len 28672 data: 4 4 4 3 3 3 3
28805 191/300 Write block 2593 len 8192 data: 5 5
28806 192/300 Write block 694 len 20480 data: 6 6 6 5 3
28807 193/300 Read block 168 len 32768
28808 194/300 Read block 1532 len 32768
28809 195/300 Write block 2932 len 32768 data: 7 7 7 5 4 5 6 4
28810 196/300 Read block 4222 len 32768
28811 197/300 Write block 331 len 32768 data: 3 4 4 5 5 5 5 6
28812 198/300 Read block 2782 len 16384
28813 199/300 Write block 3750 len 16384 data: 7 7 9 9
28814 200/300 Read block 3065 len 36864
28815 201/300 Read block 831 len 36864
28816 202/300 Write block 2363 len 20480 data: 4 4 5 4 3
28817 203/300 Read block 3391 len 20480
28818 204/300 Flush
28819 205/300 Write block 504 len 20480 data: 5 6 6 6 5
28820 206/300 Read block 2287 len 36864
28821 207/300 Read block 2964 len 4096
28822 208/300 Read block 1876 len 8192
28823 209/300 Read block 2933 len 40960
28824 210/300 Read block 2630 len 32768
28825 211/300 Write block 4215 len 4096 data: 5
28826 212/300 Read block 4451 len 16384
28827 213/300 Read block 4190 len 8192
28828 214/300 Read block 559 len 20480
28829 215/300 Read block 4566 len 28672
28830 216/300 Read block 1190 len 8192
28831 217/300 Read block 4147 len 16384
28832 218/300 Write block 84 len 16384 data: 4 5 5 5
28833 219/300 Write block 831 len 28672 data: 5 4 5 5 8 7 6
28834 220/300 Write block 850 len 36864 data: 3 3 3 3 5 6 6 5 5
28835 221/300 Write block 3022 len 28672 data: 8 7 8 9 8 7 7
28836 222/300 Read block 2015 len 16384
28837 223/300 Read block 3993 len 4096
28838 224/300 Write block 3565 len 20480 data: 3 5 6 7 6
28839 225/300 Read block 1092 len 16384
28840 226/300 Read block 2088 len 32768
28841 227/300 Read block 2776 len 12288
28842 228/300 Read block 4392 len 8192
28843 229/300 Read block 4674 len 24576
28844 230/300 Write block 1688 len 28672 data: 5 6 6 7 8 9 9
28845 231/300 Read block 3708 len 36864
28846 232/300 Write block 187 len 4096 data: 8
28847 233/300 Write block 6 len 36864 data: 6 4 4 4 4 4 6 4 5
28848 234/300 Read block 746 len 12288
28849 235/300 Flush
28850 236/300 Write block 1064 len 24576 data: 8 8 8 8 7 11
28851 237/300 Read block 859 len 24576
28852 238/300 Write block 516 len 12288 data: 4 4 4
28853 239/300 Read block 2138 len 8192
28854 240/300 Read block 1620 len 8192
28855 241/300 Read block 1493 len 12288
28856 242/300 Read block 194 len 36864
28857 243/300 Flush
28858 244/300 Read block 2967 len 16384
28859 245/300 Read block 3944 len 24576
28860 246/300 Write block 2832 len 16384 data: 6 5 5 6
28861 247/300 Flush
28862 248/300 Read block 2748 len 12288
28863 249/300 Read block 4447 len 4096
28864 250/300 Write block 2523 len 36864 data: 9 8 3 3 3 3 3 3 3
28865 251/300 Read block 1778 len 20480
28866 252/300 Write block 2807 len 8192 data: 3 3
28867 253/300 Read block 4090 len 28672
28868 254/300 Write block 675 len 12288 data: 8 8 7
28869 255/300 Read block 4190 len 16384
28870 256/300 Read block 2181 len 32768
28871 257/300 Read block 1642 len 32768
28872 258/300 Read block 2301 len 28672
28873 259/300 Read block 4978 len 20480
28874 260/300 Write block 688 len 8192 data: 4 3
28875 261/300 Read block 2837 len 16384
28876 262/300 Read block 4544 len 28672
28877 263/300 Read block 1617 len 24576
28878 264/300 Read block 614 len 36864
28879 265/300 Read block 3327 len 40960
28880 266/300 Write block 3928 len 12288 data: 6 6 4
28881 267/300 Write block 3831 len 36864 data: 8 8 5 7 6 5 4 5 6
28882 268/300 Write block 4493 len 16384 data: 5 6 8 7
28883 269/300 Read block 4733 len 12288
28884 270/300 Read block 324 len 16384
28885 271/300 Flush
28886 272/300 Write block 2998 len 36864 data: 6 7 9 11 10 8 9 10 9
28887 273/300 Read block 3376 len 20480
28888 274/300 Read block 2249 len 24576
28889 275/300 Write block 1427 len 40960 data: 8 6 5 3 3 2 3 3 3 3
28890 276/300 Flush
28891 277/300 Write block 863 len 12288 data: 7 7 7
28892 278/300 Write block 945 len 24576 data: 9 7 6 5 4 4
28893 279/300 Write block 1601 len 32768 data: 4 5 5 4 4 4 3 5
28894 280/300 Write block 4972 len 4096 data: 5
28895 281/300 Read block 2839 len 8192
28896 282/300 Read block 3456 len 20480
28897 283/300 Write block 3266 len 36864 data: 2 2 4 4 4 4 5 5 5
28898 284/300 Write block 2226 len 16384 data: 3 5 7 8
28899 285/300 Read block 4861 len 12288
28900 286/300 Read block 3854 len 24576
28901 287/300 Flush
28902 288/300 Write block 3905 len 20480 data: 8 5 2 3 4
28903 289/300 Write block 721 len 28672 data: 5 4 4 5 5 6 6
28904 290/300 Read block 1492 len 36864
28905 291/300 Write block 1749 len 16384 data: 4 5 5 4
28906 292/300 Write block 3714 len 16384 data: 6 6 7 7
28907 293/300 Read block 141 len 12288
28908 294/300 Read block 101 len 8192
28909 295/300 Write block 1030 len 36864 data: 7 6 6 5 5 5 7 5 5
28910 296/300 Read block 3538 len 28672
28911 297/300 Write block 2712 len 8192 data: 3 3
28912 298/300 Write block 960 len 4096 data: 10
28913 299/300 Read block 3705 len 20480
28914 300/300 Read block 1692 len 24576
28915 301/300 Read block 4917 len 12288
28916 Main task has work to do, go find it
28917 got dsc Start(0)
28918 start 0
28919 Jan 23 18:53:59.692 INFO request completed, latency_us: 189, response_code: 204, uri: /start/cid/0, method: POST, req_id: a2196781-573a-4a5c-ad37-c82809d2178c, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
28920 Replay: started 0, returned:Ok(())
28921 ----------------------------------------------------------------
28922 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28923 States: Offline Active Active
28924 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28925 8849 Acked Write 2 Sent Done Done false
28926 8850 Acked Write 6 Sent Done Done false
28927 8851 Acked Read 2 Sent Done Done false
28928 8852 Acked Read 10 Sent Done Done false
28929 8853 Acked Write 6 Sent Done Done false
28930 8854 Acked Write 1 Sent Done Done false
28931 8855 Acked Write 10 Sent Done Done false
28932 8856 Acked Read 9 Sent Done Done false
28933 8857 Acked Write 1 Sent Done Done false
28934 8858 Acked Read 2 Sent Done Done false
28935 8859 Acked Write 1 Sent Done Done false
28936 8860 Acked Read 9 Sent Done Done false
28937 8861 Acked Write 10 Sent Done Done false
28938 8862 Acked Read 7 Sent Done Done false
28939 8863 Acked Read 9 Sent Done Done false
28940 8864 Acked Read 5 Sent Done Done false
28941 8865 Acked Flush 0 Sent Done Done false
28942 8866 Acked Read 6 Sent Done Done false
28943 8867 Acked Read 4 Sent Done Done false
28944 8868 Acked Read 6 Sent Done Done false
28945 8869 Acked Write 2 Sent Done Done false
28946 8870 Acked Read 9 Sent Done Done false
28947 8871 Acked Write 8 Sent Done Done false
28948 8872 Acked Write 1 Sent Done Done false
28949 8873 Acked Read 5 Sent Done Done false
28950 8874 Acked Write 10 Sent Done Done false
28951 8875 Acked Write 5 Sent Done Done false
28952 8876 Acked Read 9 Sent Done Done false
28953 8877 Acked Read 1 Sent Done Done false
28954 8878 Acked Read 5 Sent Done Done false
28955 8879 Acked Read 2 Sent Done Done false
28956 8880 Acked Write 2 Sent Done Done false
28957 8881 Acked Write 9 Sent Done Done false
28958 8882 Acked Read 5 Sent Done Done false
28959 8883 Acked Write 1 Sent Done Done false
28960 8884 Acked Read 7 Sent Done Done false
28961 8885 Acked Read 7 Sent Done Done false
28962 8886 Acked Write 5 Sent Done Done false
28963 8887 Acked Write 3 Sent Done Done false
28964 8888 Acked Read 7 Sent Done Done false
28965 8889 Acked Write 10 Sent Done Done false
28966 8890 Acked Write 10 Sent Done Done false
28967 8891 Acked Write 8 Sent Done Done false
28968 8892 Acked Write 9 Sent Done Done false
28969 8893 Acked Read 1 Sent Done Done false
28970 8894 Acked Read 8 Sent Done Done false
28971 8895 Acked Write 9 Sent Done Done false
28972 8896 Acked Write 8 Sent Done Done false
28973 8897 Acked Read 7 Sent Done Done false
28974 8898 Acked Write 6 Sent Done Done false
28975 8899 Acked Read 5 Sent Done Done false
28976 8900 Acked Read 7 Sent Done Done false
28977 8901 Acked Write 2 Sent Done Done false
28978 8902 Acked Read 2 Sent Done Done false
28979 8903 Acked Flush 0 Sent Done Done false
28980 8904 Acked Read 6 Sent Done Done false
28981 8905 Acked Read 1 Sent Done Done false
28982 8906 Acked Read 4 Sent Done Done false
28983 8907 Acked Read 1 Sent Done Done false
28984 8908 Acked Write 6 Sent Done Done false
28985 8909 Acked Read 6 Sent Done Done false
28986 8910 Acked Write 2 Sent Done Done false
28987 8911 Acked Flush 0 Sent Done Done false
28988 8912 Acked Write 5 Sent Done Done false
28989 8913 Acked Write 4 Sent Done Done false
28990 8914 Acked Read 7 Sent Done Done false
28991 8915 Acked Write 1 Sent Done Done false
28992 8916 Acked Read 1 Sent Done Done false
28993 8917 Acked Write 10 Sent Done Done false
28994 8918 Acked Write 3 Sent Done Done false
28995 8919 Acked Write 9 Sent Done Done false
28996 8920 Acked Read 10 Sent Done Done false
28997 8921 Acked Flush 0 Sent Done Done false
28998 8922 Acked Write 7 Sent Done Done false
28999 8923 Acked Write 5 Sent Done Done false
29000 8924 Acked Read 2 Sent Done Done false
29001 8925 Acked Write 7 Sent Done Done false
29002 8926 Acked Read 1 Sent Done Done false
29003 8927 Acked Write 4 Sent Done Done false
29004 8928 Acked Read 2 Sent Done Done false
29005 8929 Acked Write 4 Sent Done Done false
29006 8930 Acked Write 5 Sent Done Done false
29007 8931 Acked Read 3 Sent Done Done false
29008 8932 Acked Read 9 Sent Done Done false
29009 8933 Acked Write 9 Sent Done Done false
29010 8934 Acked Read 7 Sent Done Done false
29011 8935 Acked Read 2 Sent Done Done false
29012 8936 Acked Write 4 Sent Done Done false
29013 8937 Acked Read 1 Sent Done Done false
29014 8938 Acked Read 3 Sent Done Done false
29015 8939 Acked Write 7 Sent Done Done false
29016 8940 Acked Write 2 Sent Done Done false
29017 8941 Acked Flush 0 Sent Done Done false
29018 8942 Acked Read 10 Sent Done Done false
29019 8943 Acked Read 9 Sent Done Done false
29020 8944 Acked Read 5 Sent Done Done false
29021 8945 Acked Read 3 Sent Done Done false
29022 8946 Acked Write 4 Sent Done Done false
29023 8947 Acked Read 7 Sent Done Done false
29024 8948 Acked Write 2 Sent Done Done false
29025 8949 Acked Read 3 Sent Done Done false
29026 8950 Acked Read 10 Sent Done Done false
29027 8951 Acked Read 2 Sent Done Done false
29028 8952 Acked Write 8 Sent Done Done false
29029 8953 Acked Read 1 Sent Done Done false
29030 8954 Acked Read 1 Sent Done Done false
29031 8955 Acked Write 7 Sent Done Done false
29032 8956 Acked Write 8 Sent Done Done false
29033 8957 Acked Flush 0 Sent Done Done false
29034 8958 Acked Read 10 Sent Done Done false
29035 8959 Acked Write 9 Sent Done Done false
29036 8960 Acked Read 10 Sent Done Done false
29037 8961 Acked Read 1 Sent Done Done false
29038 8962 Acked Read 7 Sent Done Done false
29039 8963 Acked Write 4 Sent Done Done false
29040 8964 Acked Write 10 Sent Done Done false
29041 8965 Acked Read 5 Sent Done Done false
29042 8966 Acked Read 2 Sent Done Done false
29043 8967 Acked Write 6 Sent Done Done false
29044 8968 Acked Flush 0 Sent Done Done false
29045 8969 Acked Read 1 Sent Done Done false
29046 8970 Acked Read 3 Sent Done Done false
29047 8971 Acked Write 10 Sent Done Done false
29048 8972 Acked Read 2 Sent Done Done false
29049 8973 Acked Flush 0 Sent Done Done false
29050 8974 Acked Read 4 Sent Done Done false
29051 8975 Acked Read 7 Sent Done Done false
29052 8976 Acked Read 6 Sent Done Done false
29053 8977 Acked Read 10 Sent Done Done false
29054 8978 Acked Write 3 Sent Done Done false
29055 8979 Acked Read 10 Sent Done Done false
29056 8980 Acked Write 9 Sent Done Done false
29057 8981 Acked Write 2 Sent Done Done false
29058 8982 Acked Read 8 Sent Done Done false
29059 8983 Acked Write 3 Sent Done Done false
29060 8984 Acked Flush 0 Sent Done Done false
29061 8985 Acked Read 2 Sent Done Done false
29062 8986 Acked Write 5 Sent Done Done false
29063 8987 Acked Flush 0 Sent Done Done false
29064 8988 Acked Read 7 Sent Done Done false
29065 8989 Acked Flush 0 Sent Done Done false
29066 8990 Acked Write 8 Sent Done Done false
29067 8991 Acked Write 5 Sent Done Done false
29068 8992 Acked Write 1 Sent Done Done false
29069 8993 Acked Write 5 Sent Done Done false
29070 8994 Acked Read 10 Sent Done Done false
29071 8995 Acked Write 10 Sent Done Done false
29072 8996 Acked Read 4 Sent Done Done false
29073 8997 Acked Write 7 Sent Done Done false
29074 8998 Acked Read 3 Sent Done Done false
29075 8999 Acked Read 2 Sent Done Done false
29076 9000 Acked Write 1 Sent Done Done false
29077 9001 Acked Write 5 Sent Done Done false
29078 9002 Acked Read 8 Sent Done Done false
29079 9003 Acked Read 3 Sent Done Done false
29080 9004 Acked Write 8 Sent Done Done false
29081 9005 Acked Read 8 Sent Done Done false
29082 9006 Acked Read 1 Sent Done Done false
29083 9007 Acked Read 1 Sent Done Done false
29084 9008 Acked Read 7 Sent Done Done false
29085 9009 Acked Read 6 Sent Done Done false
29086 9010 Acked Read 7 Sent Done Done false
29087 9011 Acked Write 4 Sent Done Done false
29088 9012 Acked Flush 0 Sent Done Done false
29089 9013 Acked Read 3 Sent Done Done false
29090 9014 Acked Read 5 Sent Done Done false
29091 9015 Acked Read 8 Sent Done Done false
29092 9016 Acked Read 9 Sent Done Done false
29093 9017 Acked Write 2 Sent Done Done false
29094 9018 Acked Write 2 Sent Done Done false
29095 9019 Acked Read 6 Sent Done Done false
29096 9020 Acked Flush 0 Sent Done Done false
29097 9021 Acked Read 3 Sent Done Done false
29098 9022 Acked Read 9 Sent Done Done false
29099 9023 Acked Flush 0 Sent Done Done false
29100 9024 Acked Read 5 Sent Done Done false
29101 9025 Acked Write 6 Sent Done Done false
29102 9026 Acked Write 10 Sent Done Done false
29103 9027 Acked Read 8 Sent Done Done false
29104 9028 Acked Write 8 Sent Done Done false
29105 9029 Acked Flush 0 Sent Done Done false
29106 9030 Acked Write 2 Sent Done Done false
29107 9031 Acked Read 2 Sent Done Done false
29108 9032 Acked Write 5 Sent Done Done false
29109 9033 Acked Write 3 Sent Done Done false
29110 9034 Acked Read 8 Sent Done Done false
29111 9035 Acked Read 10 Sent Done Done false
29112 9036 Acked Read 1 Sent Done Done false
29113 9037 Acked Write 6 Sent Done Done false
29114 9038 Acked Write 7 Sent Done Done false
29115 9039 Acked Write 2 Sent Done Done false
29116 9040 Acked Write 5 Sent Done Done false
29117 9041 Acked Read 8 Sent Done Done false
29118 9042 Acked Read 8 Sent Done Done false
29119 9043 Acked Write 8 Sent Done Done false
29120 9044 Acked Read 8 Sent Done Done false
29121 9045 Acked Write 8 Sent Done Done false
29122 9046 Acked Read 4 Sent Done Done false
29123 9047 Acked Write 4 Sent Done Done false
29124 9048 Acked Read 9 Sent Done Done false
29125 9049 Acked Read 9 Sent Done Done false
29126 9050 Acked Write 5 Sent Done Done false
29127 9051 Acked Read 5 Sent Done Done false
29128 9052 Acked Flush 0 Sent Done Done false
29129 9053 Acked Write 5 Sent Done Done false
29130 9054 Acked Read 9 Sent Done Done false
29131 9055 Acked Read 1 Sent Done Done false
29132 9056 Acked Read 2 Sent Done Done false
29133 9057 Acked Read 10 Sent Done Done false
29134 9058 Acked Read 8 Sent Done Done false
29135 9059 Acked Write 1 Sent Done Done false
29136 9060 Acked Read 4 Sent Done Done false
29137 9061 Acked Read 2 Sent Done Done false
29138 9062 Acked Read 5 Sent Done Done false
29139 9063 Acked Read 7 Sent Done Done false
29140 9064 Acked Read 2 Sent Done Done false
29141 9065 Acked Read 4 Sent Done Done false
29142 9066 Acked Write 4 Sent Done Done false
29143 9067 Acked Write 7 Sent Done Done false
29144 9068 Acked Write 9 Sent Done Done false
29145 9069 Acked Write 7 Sent Done Done false
29146 9070 Acked Read 4 Sent Done Done false
29147 9071 Acked Read 1 Sent Done Done false
29148 9072 Acked Write 5 Sent Done Done false
29149 9073 Acked Read 4 Sent Done Done false
29150 9074 Acked Read 8 Sent Done Done false
29151 9075 Acked Read 3 Sent Done Done false
29152 9076 Acked Read 2 Sent Done Done false
29153 9077 Acked Read 6 Sent Done Done false
29154 9078 Acked Write 7 Sent Done Done false
29155 9079 Acked Read 9 Sent Done Done false
29156 9080 Acked Write 1 Sent Done Done false
29157 9081 Acked Write 9 Sent Done Done false
29158 9082 Acked Read 3 Sent Done Done false
29159 9083 Acked Flush 0 Sent Done Done false
29160 9084 Acked Write 6 Sent Done Done false
29161 9085 Acked Read 6 Sent Done Done false
29162 9086 Acked Write 3 Sent Done Done false
29163 9087 Acked Read 2 Sent Done Done false
29164 9088 Acked Read 2 Sent Done Done false
29165 9089 Acked Read 3 Sent Done Done false
29166 9090 Acked Read 9 Sent Done Done false
29167 9091 Acked Flush 0 Sent Done Done false
29168 9092 Acked Read 4 Sent Done Done false
29169 9093 Acked Read 6 Sent Done Done false
29170 9094 Acked Write 4 Sent Done Done false
29171 9095 Acked Flush 0 Sent Done Done false
29172 9096 Acked Read 3 Sent Done Done false
29173 9097 Acked Read 1 Sent Done Done false
29174 9098 Acked Write 9 Sent Done Done false
29175 9099 Acked Read 5 Sent Done Done false
29176 9100 Acked Write 2 Sent Done Done false
29177 9101 Acked Read 7 Sent Done Done false
29178 9102 Acked Write 3 Sent Done Done false
29179 9103 Acked Read 4 Sent Done Done false
29180 9104 Acked Read 8 Sent Done Done false
29181 9105 Acked Read 8 Sent Done Done false
29182 9106 Acked Read 7 Sent Done Done false
29183 9107 Acked Read 5 Sent Done Done false
29184 9108 Acked Write 2 Sent Done Done false
29185 9109 Acked Read 4 Sent Done Done false
29186 9110 Acked Read 7 Sent Done Done false
29187 9111 Acked Read 6 Sent Done Done false
29188 9112 Acked Read 9 Sent Done Done false
29189 9113 Acked Read 10 Sent Done Done false
29190 9114 Acked Write 3 Sent Done Done false
29191 9115 Acked Write 9 Sent Done Done false
29192 9116 Acked Write 4 Sent Done Done false
29193 9117 Acked Read 3 Sent Done Done false
29194 9118 Acked Read 4 Sent Done Done false
29195 9119 Acked Flush 0 Sent Done Done false
29196 9120 Acked Write 9 Sent Done Done false
29197 9121 Acked Read 5 Sent Done Done false
29198 9122 Acked Read 6 Sent Done Done false
29199 9123 Acked Write 10 Sent Done Done false
29200 9124 Acked Flush 0 Sent Done Done false
29201 9125 Acked Write 3 Sent Done Done false
29202 9126 Acked Write 6 Sent Done Done false
29203 9127 Acked Write 8 Sent Done Done false
29204 9128 Acked Write 1 Sent Done Done false
29205 9129 Acked Read 2 Sent Done Done false
29206 9130 Acked Read 5 Sent Done Done false
29207 9131 Acked Write 9 Sent Done Done false
29208 9132 Acked Write 4 Sent Done Done false
29209 9133 Acked Read 3 Sent Done Done false
29210 9134 Acked Read 6 Sent Done Done false
29211 9135 Acked Flush 0 Sent Done Done false
29212 9136 Acked Write 5 Sent Done Done false
29213 9137 Acked Write 7 Sent Done Done false
29214 9138 Acked Read 9 Sent Done Done false
29215 9139 Acked Write 4 Sent Done Done false
29216 9140 Acked Write 4 Sent Done Done false
29217 9141 Acked Read 3 Sent Done Done false
29218 9142 Acked Read 2 Sent Done Done false
29219 9143 Acked Write 9 Sent Done Done false
29220 9144 Acked Read 7 Sent Done Done false
29221 9145 Acked Write 2 Sent Done Done false
29222 9146 Acked Write 1 Sent Done Done false
29223 9147 Acked Read 5 Sent Done Done false
29224 9148 Acked Read 6 Sent Done Done false
29225 9149 Acked Read 3 Sent Done Done false
29226 STATES DS:0 DS:1 DS:2 TOTAL
29227 Sent 301 0 0 301
29228 Done 0 301 301 602
29229 Skipped 0 0 0 0
29230 Error 0 0 0 0
29231 Last Flush: Some(JobId(8848)) Some(JobId(9135)) Some(JobId(9135))
29232 Downstairs last five completed: 8848 8847 8846 8845 8844
29233 Upstairs last five completed: 9149 9148 9147 9146 9145
29234 CLIENT: Up:0 ds:301 act:2
29235 [8810] got start action
29236 [8810] I am going to restart
29237 Starting downstairs at port 8810
29238 Make output file at "/tmp/dsc/downstairs-8810.txt"
29239 [8810][0] reports Starting
29240 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1227)
29241 [8810][0] reports Running
29242 ----------------------------------------------------------------
29243 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
29244 States: Offline Active Active
29245 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29246 8849 Acked Write 2 Sent Done Done false
29247 8850 Acked Write 6 Sent Done Done false
29248 8851 Acked Read 2 Sent Done Done false
29249 8852 Acked Read 10 Sent Done Done false
29250 8853 Acked Write 6 Sent Done Done false
29251 8854 Acked Write 1 Sent Done Done false
29252 8855 Acked Write 10 Sent Done Done false
29253 8856 Acked Read 9 Sent Done Done false
29254 8857 Acked Write 1 Sent Done Done false
29255 8858 Acked Read 2 Sent Done Done false
29256 8859 Acked Write 1 Sent Done Done false
29257 8860 Acked Read 9 Sent Done Done false
29258 8861 Acked Write 10 Sent Done Done false
29259 8862 Acked Read 7 Sent Done Done false
29260 8863 Acked Read 9 Sent Done Done false
29261 8864 Acked Read 5 Sent Done Done false
29262 8865 Acked Flush 0 Sent Done Done false
29263 8866 Acked Read 6 Sent Done Done false
29264 8867 Acked Read 4 Sent Done Done false
29265 8868 Acked Read 6 Sent Done Done false
29266 8869 Acked Write 2 Sent Done Done false
29267 8870 Acked Read 9 Sent Done Done false
29268 8871 Acked Write 8 Sent Done Done false
29269 8872 Acked Write 1 Sent Done Done false
29270 8873 Acked Read 5 Sent Done Done false
29271 8874 Acked Write 10 Sent Done Done false
29272 8875 Acked Write 5 Sent Done Done false
29273 8876 Acked Read 9 Sent Done Done false
29274 8877 Acked Read 1 Sent Done Done false
29275 8878 Acked Read 5 Sent Done Done false
29276 8879 Acked Read 2 Sent Done Done false
29277 8880 Acked Write 2 Sent Done Done false
29278 8881 Acked Write 9 Sent Done Done false
29279 8882 Acked Read 5 Sent Done Done false
29280 8883 Acked Write 1 Sent Done Done false
29281 8884 Acked Read 7 Sent Done Done false
29282 8885 Acked Read 7 Sent Done Done false
29283 8886 Acked Write 5 Sent Done Done false
29284 8887 Acked Write 3 Sent Done Done false
29285 8888 Acked Read 7 Sent Done Done false
29286 8889 Acked Write 10 Sent Done Done false
29287 8890 Acked Write 10 Sent Done Done false
29288 8891 Acked Write 8 Sent Done Done false
29289 8892 Acked Write 9 Sent Done Done false
29290 8893 Acked Read 1 Sent Done Done false
29291 8894 Acked Read 8 Sent Done Done false
29292 8895 Acked Write 9 Sent Done Done false
29293 8896 Acked Write 8 Sent Done Done false
29294 8897 Acked Read 7 Sent Done Done false
29295 8898 Acked Write 6 Sent Done Done false
29296 8899 Acked Read 5 Sent Done Done false
29297 8900 Acked Read 7 Sent Done Done false
29298 8901 Acked Write 2 Sent Done Done false
29299 8902 Acked Read 2 Sent Done Done false
29300 8903 Acked Flush 0 Sent Done Done false
29301 8904 Acked Read 6 Sent Done Done false
29302 8905 Acked Read 1 Sent Done Done false
29303 8906 Acked Read 4 Sent Done Done false
29304 8907 Acked Read 1 Sent Done Done false
29305 8908 Acked Write 6 Sent Done Done false
29306 8909 Acked Read 6 Sent Done Done false
29307 8910 Acked Write 2 Sent Done Done false
29308 8911 Acked Flush 0 Sent Done Done false
29309 8912 Acked Write 5 Sent Done Done false
29310 8913 Acked Write 4 Sent Done Done false
29311 8914 Acked Read 7 Sent Done Done false
29312 8915 Acked Write 1 Sent Done Done false
29313 8916 Acked Read 1 Sent Done Done false
29314 8917 Acked Write 10 Sent Done Done false
29315 8918 Acked Write 3 Sent Done Done false
29316 8919 Acked Write 9 Sent Done Done false
29317 8920 Acked Read 10 Sent Done Done false
29318 8921 Acked Flush 0 Sent Done Done false
29319 8922 Acked Write 7 Sent Done Done false
29320 8923 Acked Write 5 Sent Done Done false
29321 8924 Acked Read 2 Sent Done Done false
29322 8925 Acked Write 7 Sent Done Done false
29323 8926 Acked Read 1 Sent Done Done false
29324 8927 Acked Write 4 Sent Done Done false
29325 8928 Acked Read 2 Sent Done Done false
29326 8929 Acked Write 4 Sent Done Done false
29327 8930 Acked Write 5 Sent Done Done false
29328 8931 Acked Read 3 Sent Done Done false
29329 8932 Acked Read 9 Sent Done Done false
29330 8933 Acked Write 9 Sent Done Done false
29331 8934 Acked Read 7 Sent Done Done false
29332 8935 Acked Read 2 Sent Done Done false
29333 8936 Acked Write 4 Sent Done Done false
29334 8937 Acked Read 1 Sent Done Done false
29335 8938 Acked Read 3 Sent Done Done false
29336 8939 Acked Write 7 Sent Done Done false
29337 8940 Acked Write 2 Sent Done Done false
29338 8941 Acked Flush 0 Sent Done Done false
29339 8942 Acked Read 10 Sent Done Done false
29340 8943 Acked Read 9 Sent Done Done false
29341 8944 Acked Read 5 Sent Done Done false
29342 8945 Acked Read 3 Sent Done Done false
29343 8946 Acked Write 4 Sent Done Done false
29344 8947 Acked Read 7 Sent Done Done false
29345 8948 Acked Write 2 Sent Done Done false
29346 8949 Acked Read 3 Sent Done Done false
29347 8950 Acked Read 10 Sent Done Done false
29348 8951 Acked Read 2 Sent Done Done false
29349 8952 Acked Write 8 Sent Done Done false
29350 8953 Acked Read 1 Sent Done Done false
29351 8954 Acked Read 1 Sent Done Done false
29352 8955 Acked Write 7 Sent Done Done false
29353 8956 Acked Write 8 Sent Done Done false
29354 8957 Acked Flush 0 Sent Done Done false
29355 8958 Acked Read 10 Sent Done Done false
29356 8959 Acked Write 9 Sent Done Done false
29357 8960 Acked Read 10 Sent Done Done false
29358 8961 Acked Read 1 Sent Done Done false
29359 8962 Acked Read 7 Sent Done Done false
29360 8963 Acked Write 4 Sent Done Done false
29361 8964 Acked Write 10 Sent Done Done false
29362 8965 Acked Read 5 Sent Done Done false
29363 8966 Acked Read 2 Sent Done Done false
29364 8967 Acked Write 6 Sent Done Done false
29365 8968 Acked Flush 0 Sent Done Done false
29366 8969 Acked Read 1 Sent Done Done false
29367 8970 Acked Read 3 Sent Done Done false
29368 8971 Acked Write 10 Sent Done Done false
29369 8972 Acked Read 2 Sent Done Done false
29370 8973 Acked Flush 0 Sent Done Done false
29371 8974 Acked Read 4 Sent Done Done false
29372 8975 Acked Read 7 Sent Done Done false
29373 8976 Acked Read 6 Sent Done Done false
29374 8977 Acked Read 10 Sent Done Done false
29375 8978 Acked Write 3 Sent Done Done false
29376 8979 Acked Read 10 Sent Done Done false
29377 8980 Acked Write 9 Sent Done Done false
29378 8981 Acked Write 2 Sent Done Done false
29379 8982 Acked Read 8 Sent Done Done false
29380 8983 Acked Write 3 Sent Done Done false
29381 8984 Acked Flush 0 Sent Done Done false
29382 8985 Acked Read 2 Sent Done Done false
29383 8986 Acked Write 5 Sent Done Done false
29384 8987 Acked Flush 0 Sent Done Done false
29385 8988 Acked Read 7 Sent Done Done false
29386 8989 Acked Flush 0 Sent Done Done false
29387 8990 Acked Write 8 Sent Done Done false
29388 8991 Acked Write 5 Sent Done Done false
29389 8992 Acked Write 1 Sent Done Done false
29390 8993 Acked Write 5 Sent Done Done false
29391 8994 Acked Read 10 Sent Done Done false
29392 8995 Acked Write 10 Sent Done Done false
29393 8996 Acked Read 4 Sent Done Done false
29394 8997 Acked Write 7 Sent Done Done false
29395 8998 Acked Read 3 Sent Done Done false
29396 8999 Acked Read 2 Sent Done Done false
29397 9000 Acked Write 1 Sent Done Done false
29398 9001 Acked Write 5 Sent Done Done false
29399 9002 Acked Read 8 Sent Done Done false
29400 9003 Acked Read 3 Sent Done Done false
29401 9004 Acked Write 8 Sent Done Done false
29402 9005 Acked Read 8 Sent Done Done false
29403 9006 Acked Read 1 Sent Done Done false
29404 9007 Acked Read 1 Sent Done Done false
29405 9008 Acked Read 7 Sent Done Done false
29406 9009 Acked Read 6 Sent Done Done false
29407 9010 Acked Read 7 Sent Done Done false
29408 9011 Acked Write 4 Sent Done Done false
29409 9012 Acked Flush 0 Sent Done Done false
29410 9013 Acked Read 3 Sent Done Done false
29411 9014 Acked Read 5 Sent Done Done false
29412 9015 Acked Read 8 Sent Done Done false
29413 9016 Acked Read 9 Sent Done Done false
29414 9017 Acked Write 2 Sent Done Done false
29415 9018 Acked Write 2 Sent Done Done false
29416 9019 Acked Read 6 Sent Done Done false
29417 9020 Acked Flush 0 Sent Done Done false
29418 9021 Acked Read 3 Sent Done Done false
29419 9022 Acked Read 9 Sent Done Done false
29420 9023 Acked Flush 0 Sent Done Done false
29421 9024 Acked Read 5 Sent Done Done false
29422 9025 Acked Write 6 Sent Done Done false
29423 9026 Acked Write 10 Sent Done Done false
29424 9027 Acked Read 8 Sent Done Done false
29425 9028 Acked Write 8 Sent Done Done false
29426 9029 Acked Flush 0 Sent Done Done false
29427 9030 Acked Write 2 Sent Done Done false
29428 9031 Acked Read 2 Sent Done Done false
29429 9032 Acked Write 5 Sent Done Done false
29430 9033 Acked Write 3 Sent Done Done false
29431 9034 Acked Read 8 Sent Done Done false
29432 9035 Acked Read 10 Sent Done Done false
29433 9036 Acked Read 1 Sent Done Done false
29434 9037 Acked Write 6 Sent Done Done false
29435 9038 Acked Write 7 Sent Done Done false
29436 9039 Acked Write 2 Sent Done Done false
29437 9040 Acked Write 5 Sent Done Done false
29438 9041 Acked Read 8 Sent Done Done false
29439 9042 Acked Read 8 Sent Done Done false
29440 9043 Acked Write 8 Sent Done Done false
29441 9044 Acked Read 8 Sent Done Done false
29442 9045 Acked Write 8 Sent Done Done false
29443 9046 Acked Read 4 Sent Done Done false
29444 9047 Acked Write 4 Sent Done Done false
29445 9048 Acked Read 9 Sent Done Done false
29446 9049 Acked Read 9 Sent Done Done false
29447 9050 Acked Write 5 Sent Done Done false
29448 9051 Acked Read 5 Sent Done Done false
29449 9052 Acked Flush 0 Sent Done Done false
29450 9053 Acked Write 5 Sent Done Done false
29451 9054 Acked Read 9 Sent Done Done false
29452 9055 Acked Read 1 Sent Done Done false
29453 9056 Acked Read 2 Sent Done Done false
29454 9057 Acked Read 10 Sent Done Done false
29455 9058 Acked Read 8 Sent Done Done false
29456 9059 Acked Write 1 Sent Done Done false
29457 9060 Acked Read 4 Sent Done Done false
29458 9061 Acked Read 2 Sent Done Done false
29459 9062 Acked Read 5 Sent Done Done false
29460 9063 Acked Read 7 Sent Done Done false
29461 9064 Acked Read 2 Sent Done Done false
29462 9065 Acked Read 4 Sent Done Done false
29463 9066 Acked Write 4 Sent Done Done false
29464 9067 Acked Write 7 Sent Done Done false
29465 9068 Acked Write 9 Sent Done Done false
29466 9069 Acked Write 7 Sent Done Done false
29467 9070 Acked Read 4 Sent Done Done false
29468 9071 Acked Read 1 Sent Done Done false
29469 9072 Acked Write 5 Sent Done Done false
29470 9073 Acked Read 4 Sent Done Done false
29471 9074 Acked Read 8 Sent Done Done false
29472 9075 Acked Read 3 Sent Done Done false
29473 9076 Acked Read 2 Sent Done Done false
29474 9077 Acked Read 6 Sent Done Done false
29475 9078 Acked Write 7 Sent Done Done false
29476 9079 Acked Read 9 Sent Done Done false
29477 9080 Acked Write 1 Sent Done Done false
29478 9081 Acked Write 9 Sent Done Done false
29479 9082 Acked Read 3 Sent Done Done false
29480 9083 Acked Flush 0 Sent Done Done false
29481 9084 Acked Write 6 Sent Done Done false
29482 9085 Acked Read 6 Sent Done Done false
29483 9086 Acked Write 3 Sent Done Done false
29484 9087 Acked Read 2 Sent Done Done false
29485 9088 Acked Read 2 Sent Done Done false
29486 9089 Acked Read 3 Sent Done Done false
29487 9090 Acked Read 9 Sent Done Done false
29488 9091 Acked Flush 0 Sent Done Done false
29489 9092 Acked Read 4 Sent Done Done false
29490 9093 Acked Read 6 Sent Done Done false
29491 9094 Acked Write 4 Sent Done Done false
29492 9095 Acked Flush 0 Sent Done Done false
29493 9096 Acked Read 3 Sent Done Done false
29494 9097 Acked Read 1 Sent Done Done false
29495 9098 Acked Write 9 Sent Done Done false
29496 9099 Acked Read 5 Sent Done Done false
29497 9100 Acked Write 2 Sent Done Done false
29498 9101 Acked Read 7 Sent Done Done false
29499 9102 Acked Write 3 Sent Done Done false
29500 9103 Acked Read 4 Sent Done Done false
29501 9104 Acked Read 8 Sent Done Done false
29502 9105 Acked Read 8 Sent Done Done false
29503 9106 Acked Read 7 Sent Done Done false
29504 9107 Acked Read 5 Sent Done Done false
29505 9108 Acked Write 2 Sent Done Done false
29506 9109 Acked Read 4 Sent Done Done false
29507 9110 Acked Read 7 Sent Done Done false
29508 9111 Acked Read 6 Sent Done Done false
29509 9112 Acked Read 9 Sent Done Done false
29510 9113 Acked Read 10 Sent Done Done false
29511 9114 Acked Write 3 Sent Done Done false
29512 9115 Acked Write 9 Sent Done Done false
29513 9116 Acked Write 4 Sent Done Done false
29514 9117 Acked Read 3 Sent Done Done false
29515 9118 Acked Read 4 Sent Done Done false
29516 9119 Acked Flush 0 Sent Done Done false
29517 9120 Acked Write 9 Sent Done Done false
29518 9121 Acked Read 5 Sent Done Done false
29519 9122 Acked Read 6 Sent Done Done false
29520 9123 Acked Write 10 Sent Done Done false
29521 9124 Acked Flush 0 Sent Done Done false
29522 9125 Acked Write 3 Sent Done Done false
29523 9126 Acked Write 6 Sent Done Done false
29524 9127 Acked Write 8 Sent Done Done false
29525 9128 Acked Write 1 Sent Done Done false
29526 9129 Acked Read 2 Sent Done Done false
29527 9130 Acked Read 5 Sent Done Done false
29528 9131 Acked Write 9 Sent Done Done false
29529 9132 Acked Write 4 Sent Done Done false
29530 9133 Acked Read 3 Sent Done Done false
29531 9134 Acked Read 6 Sent Done Done false
29532 9135 Acked Flush 0 Sent Done Done false
29533 9136 Acked Write 5 Sent Done Done false
29534 9137 Acked Write 7 Sent Done Done false
29535 9138 Acked Read 9 Sent Done Done false
29536 9139 Acked Write 4 Sent Done Done false
29537 9140 Acked Write 4 Sent Done Done false
29538 9141 Acked Read 3 Sent Done Done false
29539 9142 Acked Read 2 Sent Done Done false
29540 9143 Acked Write 9 Sent Done Done false
29541 9144 Acked Read 7 Sent Done Done false
29542 9145 Acked Write 2 Sent Done Done false
29543 9146 Acked Write 1 Sent Done Done false
29544 9147 Acked Read 5 Sent Done Done false
29545 9148 Acked Read 6 Sent Done Done false
29546 9149 Acked Read 3 Sent Done Done false
29547 9150 Acked Flush 0 Sent Done Done false
29548 STATES DS:0 DS:1 DS:2 TOTAL
29549 Sent 302 0 0 302
29550 Done 0 302 302 604
29551 Skipped 0 0 0 0
29552 Error 0 0 0 0
29553 Last Flush: Some(JobId(8848)) Some(JobId(9150)) Some(JobId(9150))
29554 Downstairs last five completed: 8848 8847 8846 8845 8844
29555 Upstairs last five completed: 9150 9149 9148 9147 9146
29556 CLIENT: Up:0 ds:302 act:2
295572025-01-23T18:54:05.449ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
295582025-01-23T18:54:05.449ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62322) = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
295592025-01-23T18:54:05.450ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
295602025-01-23T18:54:05.451ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
295612025-01-23T18:54:05.451ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
295622025-01-23T18:54:05.451ZINFOcrucible: send last flush ID to this DS: Some(JobId(8848)) = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
295632025-01-23T18:54:05.451ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
295642025-01-23T18:54:05.451ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(8848)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
295652025-01-23T18:54:05.451ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(8848)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
29566 ----------------------------------------------------------------
29567 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
29568 Downstairs last five completed: 9150 9149 9148 9147 9146
29569 Upstairs last five completed: 9150 9149 9148 9147 9146
29570 CLIENT: Up:0 ds:0 act:3
29571 Replay: All jobs finished
29572 Main task has work to do, go find it
29573 got dsc Stop(2)
29574 stop 2
29575 Jan 23 18:54:07.698[8830] Got stop action so:false kr:false
29576 INFO request completed, latency_us: 334, response_code: 204, uri: /stop/cid/2, method: POST, req_id: d86c8605-a0c8-4bd3-8d2a-d9af250125d4, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
295772025-01-23T18:54:07.698ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
295782025-01-23T18:54:07.698ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
295792025-01-23T18:54:07.698ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
29580 Jan 23 18:54:07.698 INFO request completed, latency_us: 139, response_code: 200, uri: /state/cid/2, method:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T18:54:07.69894448Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"2","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
29581 GET, req_id: f63cfc01-05c0-498e-9fd3-32b25432fc68, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
29582 [8830] Exited with: ExitStatus(unix_wait_status(9))
29583 [8830][2] reports Exit
29584 Jan 23 18:54:11.701 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/2, method: GET, req_id: 07bf4a74-2bda-4ab3-81f2-bc8052b4acf4, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
29585 001/300 Read block 2794 len 32768
29586 002/300 Read block 2675 len 32768
29587 003/300 Write block 4805 len 24576 data: 6 7 7 8 8 8
29588 004/300 Write block 4297 len 12288 data: 3 3 3
29589 005/300 Write block 4294 len 32768 data: 6 6 5 4 4 4 3 2
29590 006/300 Flush
29591 007/300 Read block 2329 len 36864
29592 008/300 Read block 1578 len 36864
29593 009/300 Write block 4791 len 16384 data: 4 5 6 6
29594 010/300 Write block 32 len 16384 data: 4 6 6 7
29595 011/300 Write block 1859 len 12288 data: 8 7 8
29596 012/300 Write block 2592 len 24576 data: 4 6 6 5 5 5
29597 013/300 Write block 783 len 40960 data: 8 7 5 7 7 8 6 5 5 4
29598 014/300 Write block 1174 len 36864 data: 6 7 6 6 4 4 5 4 5
29599 015/300 Write block 1814 len 8192 data: 9 8
29600 016/300 Write block 1901 len 8192 data: 8 9
29601 017/300 Read block 2456 len 32768
29602 018/300 Write block 4440 len 16384 data: 3 3 3 4
29603 019/300 Read block 3305 len 16384
29604 020/300 Write block 3513 len 4096 data: 7
29605 021/300 Read block 543 len 32768
29606 022/300 Write block 2565 len 24576 data: 3 3 3 4 3 4
29607 023/300 Write block 4670 len 4096 data: 5
29608 024/300 Write block 729 len 32768 data: 8 9 9 8 8 8 7 6
29609 025/300 Read block 2034 len 16384
29610 026/300 Read block 842 len 12288
29611 027/300 Flush
29612 028/300 Write block 4186 len 16384 data: 3 4 4 4
29613 029/300 Write block 2071 len 4096 data: 9
29614 030/300 Read block 4769 len 8192
29615 031/300 Read block 2563 len 32768
29616 032/300 Read block 664 len 36864
29617 033/300 Write block 1059 len 20480 data: 3 3 5 7 5
29618 034/300 Read block 3804 len 28672
29619 035/300 Write block 3847 len 24576 data: 4 4 4 4 4 4
29620 036/300 Read block 2661 len 24576
29621 037/300 Flush
29622 038/300 Read block 1471 len 8192
29623 039/300 Read block 714 len 16384
29624 040/300 Write block 1648 len 32768 data: 5 5 5 3 4 3 3 3
29625 041/300 Read block 3164 len 28672
29626 042/300 Read block 3302 len 28672
29627 043/300 Write block 528 len 36864 data: 9 8 6 5 5 5 6 5 6
29628 044/300 Write block 1960 len 40960 data: 5 6 5 5 5 5 6 6 7 6
29629 045/300 Read block 585 len 16384
29630 046/300 Write block 3106 len 8192 data: 9 8
29631 047/300 Read block 4032 len 24576
29632 048/300 Read block 655 len 36864
29633 049/300 Write block 3811 len 16384 data: 3 4 4 4
29634 050/300 Flush
29635 051/300 Write block 1024 len 8192 data: 8 7
29636 052/300 Write block 4197 len 20480 data: 6 7 6 5 5
29637 053/300 Read block 2017 len 4096
29638 054/300 Read block 2959 len 24576
29639 055/300 Read block 3571 len 12288
29640 056/300 Read block 3202 len 28672
29641 057/300 Write block 4551 len 4096 data: 4
29642 058/300 Read block 299 len 32768
29643 059/300 Write block 2398 len 36864 data: 5 4 4 5 5 9 9 7 7
29644 060/300 Read block 2576 len 4096
29645 061/300 Flush
29646 062/300 Write block 2719 len 12288 data: 4 4 3
29647 063/300 Read block 4391 len 16384
29648 064/300 Read block 3451 len 28672
29649 065/300 Flush
29650 066/300 Flush
29651 067/300 Read block 3757 len 32768
29652 068/300 Read block 2546 len 8192
29653 069/300 Read block 2042 len 40960
29654 070/300 Flush
29655 071/300 Write block 2793 len 32768 data: 6 6 7 6 7 7 7 4
29656 072/300 Read block 3974 len 24576
29657 073/300 Write block 3818 len 24576 data: 5 5 5 6 5 6
29658 074/300 Write block 4263 len 4096 data: 5
29659 075/300 Read block 4914 len 8192
29660 076/300 Read block 2943 len 28672
29661 077/300 Read block 711 len 36864
29662 078/300 Read block 1421 len 40960
29663 079/300 Write block 1902 len 20480 data: 10 10 10 8 7
29664 080/300 Read block 2688 len 32768
29665 081/300 Write block 67 len 16384 data: 4 5 5 6
29666 082/300 Read block 4496 len 8192
29667 083/300 Read block 3913 len 12288
29668 084/300 Write block 2316 len 16384 data: 7 7 6 6
29669 085/300 Read block 1163 len 32768
29670 086/300 Read block 2551 len 32768
29671 087/300 Write block 1847 len 32768 data: 6 6 6 5 6 6 5 7
29672 088/300 Write block 3366 len 4096 data: 3
29673 089/300 Write block 2871 len 12288 data: 9 8 7
29674 090/300 Flush
29675 091/300 Read block 1197 len 32768
29676 092/300 Flush
29677 093/300 Read block 2175 len 4096
29678 094/300 Write block 857 len 8192 data: 6 6
29679 095/300 Read block 1721 len 40960
29680 096/300 Write block 1008 len 36864 data: 5 6 6 6 7 6 4 4 6
29681 097/300 Flush
29682 098/300 Read block 1974 len 12288
29683 099/300 Flush
29684 100/300 Read block 3684 len 8192
29685 101/300 Read block 1071 len 32768
29686 102/300 Read block 1840 len 4096
29687 103/300 Write block 2363 len 8192 data: 5 5
29688 104/300 Write block 255 len 20480 data: 4 3 3 4 6
29689 105/300 Read block 1792 len 12288
29690 106/300 Write block 4195 len 40960 data: 7 7 7 8 7 6 6 5 6 7
29691 107/300 Read block 4315 len 32768
29692 108/300 Flush
29693 109/300 Write block 1315 len 4096 data: 6
29694 110/300 Write block 1606 len 28672 data: 5 4 6 5 4 4 4
29695 111/300 Write block 1428 len 24576 data: 7 6 4 4 3 4
29696 112/300 Write block 1897 len 40960 data: 6 8 7 8 9 11 11 11 9 8
29697 113/300 Write block 4503 len 12288 data: 8 9 8
29698 114/300 Write block 2082 len 20480 data: 6 6 6 6 6
29699 115/300 Write block 1330 len 12288 data: 3 3 3
29700 116/300 Write block 3574 len 28672 data: 6 7 5 6 5 4 3
29701 117/300 Read block 1349 len 28672
29702 118/300 Read block 3383 len 36864
29703 119/300 Read block 3576 len 32768
29704 120/300 Read block 3425 len 28672
29705 121/300 Read block 2228 len 24576
29706 122/300 Write block 2589 len 32768 data: 5 6 6 5 7 7 6 6
29707 123/300 Read block 3256 len 36864
29708 124/300 Write block 2889 len 8192 data: 8 8
29709 125/300 Read block 4628 len 32768
29710 126/300 Write block 4782 len 12288 data: 7 8 8
29711 127/300 Flush
29712 128/300 Read block 3173 len 28672
29713 129/300 Read block 105 len 16384
29714 130/300 Flush
29715 131/300 Read block 3139 len 24576
29716 132/300 Write block 974 len 20480 data: 4 4 3 3 4
29717 133/300 Write block 1413 len 4096 data: 3
29718 134/300 Read block 4080 len 8192
29719 135/300 Write block 1285 len 12288 data: 8 10 10
29720 136/300 Write block 777 len 12288 data: 9 9 10
29721 137/300 Read block 1315 len 24576
29722 138/300 Read block 4189 len 16384
29723 139/300 Write block 1513 len 20480 data: 6 6 3 3 3
29724 140/300 Write block 2214 len 24576 data: 2 2 2 3 3 4
29725 141/300 Write block 800 len 12288 data: 4 4 3
29726 142/300 Write block 3525 len 20480 data: 2 2 2 2 2
29727 143/300 Flush
29728 144/300 Read block 2558 len 32768
29729 145/300 Read block 4531 len 40960
29730 146/300 Write block 522 len 4096 data: 6
29731 147/300 Write block 1608 len 8192 data: 7 6
29732 148/300 Read block 2526 len 28672
29733 149/300 Flush
29734 150/300 Write block 2645 len 36864 data: 2 2 2 3 4 3 3 5 5
29735 151/300 Flush
29736 152/300 Read block 4502 len 20480
29737 153/300 Read block 766 len 28672
29738 154/300 Flush
29739 155/300 Read block 4832 len 36864
29740 156/300 Read block 1651 len 24576
29741 157/300 Write block 4369 len 16384 data: 6 6 5 5
29742 158/300 Read block 2653 len 40960
29743 159/300 Read block 2115 len 28672
29744 160/300 Read block 27 len 16384
29745 161/300 Read block 4745 len 36864
29746 162/300 Flush
29747 163/300 Read block 1863 len 32768
29748 164/300 Read block 1684 len 24576
29749 165/300 Write block 2490 len 12288 data: 6 6 6
29750 166/300 Flush
29751 167/300 Read block 4576 len 16384
29752 168/300 Read block 2764 len 24576
29753 169/300 Write block 3958 len 24576 data: 6 5 5 6 5 5
29754 170/300 Flush
29755 171/300 Read block 1412 len 24576
29756 172/300 Write block 359 len 40960 data: 4 3 3 4 4 4 6 5 6 5
29757 173/300 Read block 1442 len 16384
29758 174/300 Write block 3049 len 20480 data: 10 10 7 5 5
29759 175/300 Write block 3301 len 4096 data: 4
29760 176/300 Write block 913 len 16384 data: 4 4 5 4
29761 177/300 Read block 1837 len 12288
29762 178/300 Read block 1409 len 20480
29763 179/300 Read block 2209 len 8192
29764 180/300 Read block 1763 len 32768
29765 181/300 Flush
29766 182/300 Write block 1727 len 36864 data: 5 5 5 6 6 6 6 4 3
29767 183/300 Read block 773 len 12288
29768 184/300 Write block 3116 len 8192 data: 5 5
29769 185/300 Read block 4065 len 4096
29770 186/300 Read block 1596 len 36864
29771 187/300 Write block 2053 len 28672 data: 8 5 6 8 6 6 6
29772 188/300 Read block 538 len 28672
29773 189/300 Write block 3318 len 24576 data: 6 7 6 5 6 7
29774 190/300 Write block 593 len 24576 data: 5 5 5 4 5 6
29775 191/300 Flush
29776 192/300 Read block 2207 len 4096
29777 193/300 Read block 1056 len 28672
29778 194/300 Write block 4994 len 16384 data: 4 3 3 2
29779 195/300 Read block 656 len 20480
29780 196/300 Read block 4955 len 28672
29781 197/300 Read block 3101 len 20480
29782 198/300 Read block 3388 len 40960
29783 199/300 Read block 169 len 12288
29784 200/300 Read block 1652 len 24576
29785 201/300 Write block 3286 len 12288 data: 5 5 5
29786 202/300 Flush
29787 203/300 Write block 2869 len 24576 data: 6 8 10 9 8 5
29788 204/300 Write block 4919 len 4096 data: 5
29789 205/300 Write block 1220 len 40960 data: 6 6 7 5 4 4 4 3 3 5
29790 206/300 Read block 981 len 8192
29791 207/300 Write block 4866 len 12288 data: 2 3 5
29792 208/300 Write block 1485 len 8192 data: 3 3
29793 209/300 Read block 458 len 4096
29794 210/300 Read block 2881 len 24576
29795 211/300 Read block 2993 len 20480
29796 212/300 Flush
29797 213/300 Read block 1480 len 12288
29798 214/300 Read block 2462 len 32768
29799 215/300 Read block 2765 len 36864
29800 216/300 Read block 4459 len 28672
29801 217/300 Write block 136 len 12288 data: 9 9 11
29802 218/300 Flush
29803 219/300 Read block 4108 len 28672
29804 220/300 Read block 4534 len 28672
29805 221/300 Write block 2735 len 24576 data: 6 7 8 8 7 6
29806 222/300 Write block 2495 len 20480 data: 5 6 9 8 8
29807 223/300 Write block 3654 len 8192 data: 5 5
29808 224/300 Write block 1245 len 24576 data: 8 9 8 6 7 7
29809 225/300 Read block 3708 len 20480
29810 226/300 Write block 3648 len 24576 data: 5 4 4 4 6 7
29811 227/300 Write block 4928 len 12288 data: 4 4 5
29812 228/300 Read block 4130 len 28672
29813 229/300 Read block 4945 len 28672
29814 230/300 Write block 4318 len 4096 data: 6
29815 231/300 Write block 586 len 4096 data: 2
29816 232/300 Write block 3304 len 4096 data: 3
29817 233/300 Read block 1802 len 36864
29818 234/300 Write block 2872 len 16384 data: 10 9 6 5
29819 235/300 Read block 2591 len 24576
29820 236/300 Write block 4540 len 28672 data: 4 4 6 6 6 5 4
29821 237/300 Read block 2533 len 20480
29822 238/300 Read block 479 len 24576
29823 239/300 Read block 3610 len 24576
29824 240/300 Read block 359 len 8192
29825 241/300 Write block 130 len 36864 data: 6 9 8 7 8 9 10 10 12
29826 242/300 Write block 3875 len 28672 data: 6 6 8 9 9 10 9
29827 243/300 Read block 1282 len 40960
29828 244/300 Write block 3649 len 36864 data: 5 5 5 7 8 6 6 7 6
29829 245/300 Read block 622 len 4096
29830 246/300 Read block 4218 len 36864
29831 247/300 Write block 1469 len 4096 data: 8
29832 248/300 Write block 248 len 32768 data: 9 6 5 6 7 6 6 5
29833 249/300 Write block 2295 len 40960 data: 6 5 5 5 7 5 7 10 11 8
29834 250/300 Write block 2582 len 4096 data: 8
29835 251/300 Read block 1610 len 32768
29836 252/300 Read block 2250 len 24576
29837 253/300 Read block 3737 len 16384
29838 254/300 Read block 3320 len 40960
29839 255/300 Read block 2492 len 36864
29840 256/300 Write block 3185 len 20480 data: 6 6 4 4 4
29841 257/300 Write block 712 len 20480 data: 5 4 4 5 7
29842 258/300 Read block 1482 len 36864
29843 259/300 Write block 1136 len 8192 data: 9 7
29844 260/300 Read block 4084 len 24576
29845 261/300 Read block 4067 len 24576
29846 262/300 Read block 3974 len 12288
29847 263/300 Flush
29848 264/300 Flush
29849 265/300 Flush
29850 266/300 Write block 350 len 24576 data: 5 5 4 3 4 5
29851 267/300 Read block 595 len 8192
29852 268/300 Write block 1917 len 40960 data: 6 6 4 5 4 5 6 6 6 5
29853 269/300 Read block 1123 len 24576
29854 270/300 Write block 4828 len 8192 data: 6 5
29855 271/300 Read block 1276 len 4096
29856 272/300 Read block 1926 len 4096
29857 273/300 Read block 3602 len 16384
29858 274/300 Write block 4702 len 40960 data: 5 4 6 4 6 6 8 7 6 7
29859 275/300 Write block 3651 len 40960 data: 6 8 9 7 7 8 7 5 5 6
29860 276/300 Write block 578 len 36864 data: 6 5 6 6 5 4 4 2 3
29861 277/300 Write block 4511 len 16384 data: 4 3 3 4
29862 278/300 Read block 3706 len 24576
29863 279/300 Read block 964 len 20480
29864 280/300 Write block 4393 len 8192 data: 3 4
29865 281/300 Read block 2640 len 8192
29866 282/300 Read block 3163 len 36864
29867 283/300 Write block 4626 len 4096 data: 4
29868 284/300 Read block 2712 len 24576
29869 285/300 Read block 634 len 16384
29870 286/300 Read block 4412 len 8192
29871 287/300 Write block 4954 len 20480 data: 5 5 5 5 5
29872 288/300 Write block 1815 len 4096 data: 9
29873 289/300 Read block 1823 len 20480
29874 290/300 Flush
29875 291/300 Read block 4381 len 28672
29876 292/300 Write block 3066 len 16384 data: 7 6 7 7
29877 293/300 Write block 796 len 36864 data: 6 6 6 4 5 5 4 4 4
29878 294/300 Write block 2102 len 28672 data: 7 9 9 9 10 8 10
29879 295/300 Write block 2975 len 24576 data: 3 2 2 2 2 3
29880 296/300 Read block 1512 len 8192
29881 297/300 Read block 1527 len 36864
29882 298/300 Read block 4805 len 8192
29883 299/300 Flush
29884 300/300 Read block 4285 len 32768
29885 301/300 Read block 1156 len 32768
29886 Main task has work to do, go find it
29887 got dsc Start(2)
29888 start 2
29889 Jan 23 18:54:11.940 INFO request completed, latency_us: 202, response_code: 204, uri: /start/cid/2, method: POST, req_id: 855f5849-3314-45eb-8db8-d17d8bbd6b09, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
29890 Replay: started 2, returned:Ok(())
29891 ----------------------------------------------------------------
29892 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29893 States: Active Active Offline
29894 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29895 9151 Acked Read 8 Done Done Sent false
29896 9152 Acked Read 8 Done Done Sent false
29897 9153 Acked Write 6 Done Done Sent false
29898 9154 Acked Write 3 Done Done Sent false
29899 9155 Acked Write 8 Done Done Sent false
29900 9156 Acked Flush 0 Done Done Sent false
29901 9157 Acked Read 9 Done Done Sent false
29902 9158 Acked Read 9 Done Done Sent false
29903 9159 Acked Write 4 Done Done Sent false
29904 9160 Acked Write 4 Done Done Sent false
29905 9161 Acked Write 3 Done Done Sent false
29906 9162 Acked Write 6 Done Done Sent false
29907 9163 Acked Write 10 Done Done Sent false
29908 9164 Acked Write 9 Done Done Sent false
29909 9165 Acked Write 2 Done Done Sent false
29910 9166 Acked Write 2 Done Done Sent false
29911 9167 Acked Read 8 Done Done Sent false
29912 9168 Acked Write 4 Done Done Sent false
29913 9169 Acked Read 4 Done Done Sent false
29914 9170 Acked Write 1 Done Done Sent false
29915 9171 Acked Read 8 Done Done Sent false
29916 9172 Acked Write 6 Done Done Sent false
29917 9173 Acked Write 1 Done Done Sent false
29918 9174 Acked Write 8 Done Done Sent false
29919 9175 Acked Read 4 Done Done Sent false
29920 9176 Acked Read 3 Done Done Sent false
29921 9177 Acked Flush 0 Done Done Sent false
29922 9178 Acked Write 4 Done Done Sent false
29923 9179 Acked Write 1 Done Done Sent false
29924 9180 Acked Read 2 Done Done Sent false
29925 9181 Acked Read 8 Done Done Sent false
29926 9182 Acked Read 9 Done Done Sent false
29927 9183 Acked Write 5 Done Done Sent false
29928 9184 Acked Read 7 Done Done Sent false
29929 9185 Acked Write 6 Done Done Sent false
29930 9186 Acked Read 6 Done Done Sent false
29931 9187 Acked Flush 0 Done Done Sent false
29932 9188 Acked Read 2 Done Done Sent false
29933 9189 Acked Read 4 Done Done Sent false
29934 9190 Acked Write 8 Done Done Sent false
29935 9191 Acked Read 7 Done Done Sent false
29936 9192 Acked Read 7 Done Done Sent false
29937 9193 Acked Write 9 Done Done Sent false
29938 9194 Acked Write 10 Done Done Sent false
29939 9195 Acked Read 4 Done Done Sent false
29940 9196 Acked Write 2 Done Done Sent false
29941 9197 Acked Read 6 Done Done Sent false
29942 9198 Acked Read 9 Done Done Sent false
29943 9199 Acked Write 4 Done Done Sent false
29944 9200 Acked Flush 0 Done Done Sent false
29945 9201 Acked Write 2 Done Done Sent false
29946 9202 Acked Write 5 Done Done Sent false
29947 9203 Acked Read 1 Done Done Sent false
29948 9204 Acked Read 6 Done Done Sent false
29949 9205 Acked Read 3 Done Done Sent false
29950 9206 Acked Read 7 Done Done Sent false
29951 9207 Acked Write 1 Done Done Sent false
29952 9208 Acked Read 8 Done Done Sent false
29953 9209 Acked Write 9 Done Done Sent false
29954 9210 Acked Read 1 Done Done Sent false
29955 9211 Acked Flush 0 Done Done Sent false
29956 9212 Acked Write 3 Done Done Sent false
29957 9213 Acked Read 4 Done Done Sent false
29958 9214 Acked Read 7 Done Done Sent false
29959 9215 Acked Flush 0 Done Done Sent false
29960 9216 Acked Flush 0 Done Done Sent false
29961 9217 Acked Read 8 Done Done Sent false
29962 9218 Acked Read 2 Done Done Sent false
29963 9219 Acked Read 10 Done Done Sent false
29964 9220 Acked Flush 0 Done Done Sent false
29965 9221 Acked Write 8 Done Done Sent false
29966 9222 Acked Read 6 Done Done Sent false
29967 9223 Acked Write 6 Done Done Sent false
29968 9224 Acked Write 1 Done Done Sent false
29969 9225 Acked Read 2 Done Done Sent false
29970 9226 Acked Read 7 Done Done Sent false
29971 9227 Acked Read 9 Done Done Sent false
29972 9228 Acked Read 10 Done Done Sent false
29973 9229 Acked Write 5 Done Done Sent false
29974 9230 Acked Read 8 Done Done Sent false
29975 9231 Acked Write 4 Done Done Sent false
29976 9232 Acked Read 2 Done Done Sent false
29977 9233 Acked Read 3 Done Done Sent false
29978 9234 Acked Write 4 Done Done Sent false
29979 9235 Acked Read 8 Done Done Sent false
29980 9236 Acked Read 8 Done Done Sent false
29981 9237 Acked Write 8 Done Done Sent false
29982 9238 Acked Write 1 Done Done Sent false
29983 9239 Acked Write 3 Done Done Sent false
29984 9240 Acked Flush 0 Done Done Sent false
29985 9241 Acked Read 8 Done Done Sent false
29986 9242 Acked Flush 0 Done Done Sent false
29987 9243 Acked Read 1 Done Done Sent false
29988 9244 Acked Write 2 Done Done Sent false
29989 9245 Acked Read 10 Done Done Sent false
29990 9246 Acked Write 9 Done Done Sent false
29991 9247 Acked Flush 0 Done Done Sent false
29992 9248 Acked Read 3 Done Done Sent false
29993 9249 Acked Flush 0 Done Done Sent false
29994 9250 Acked Read 2 Done Done Sent false
29995 9251 Acked Read 8 Done Done Sent false
29996 9252 Acked Read 1 Done Done Sent false
29997 9253 Acked Write 2 Done Done Sent false
29998 9254 Acked Write 5 Done Done Sent false
29999 9255 Acked Read 3 Done Done Sent false
30000 9256 Acked Write 10 Done Done Sent false
30001 9257 Acked Read 8 Done Done Sent false
30002 9258 Acked Flush 0 Done Done Sent false
30003 9259 Acked Write 1 Done Done Sent false
30004 9260 Acked Write 7 Done Done Sent false
30005 9261 Acked Write 6 Done Done Sent false
30006 9262 Acked Write 10 Done Done Sent false
30007 9263 Acked Write 3 Done Done Sent false
30008 9264 Acked Write 5 Done Done Sent false
30009 9265 Acked Write 3 Done Done Sent false
30010 9266 Acked Write 7 Done Done Sent false
30011 9267 Acked Read 7 Done Done Sent false
30012 9268 Acked Read 9 Done Done Sent false
30013 9269 Acked Read 8 Done Done Sent false
30014 9270 Acked Read 7 Done Done Sent false
30015 9271 Acked Read 6 Done Done Sent false
30016 9272 Acked Write 8 Done Done Sent false
30017 9273 Acked Read 9 Done Done Sent false
30018 9274 Acked Write 2 Done Done Sent false
30019 9275 Acked Read 8 Done Done Sent false
30020 9276 Acked Write 3 Done Done Sent false
30021 9277 Acked Flush 0 Done Done Sent false
30022 9278 Acked Read 7 Done Done Sent false
30023 9279 Acked Read 4 Done Done Sent false
30024 9280 Acked Flush 0 Done Done Sent false
30025 9281 Acked Read 6 Done Done Sent false
30026 9282 Acked Write 5 Done Done Sent false
30027 9283 Acked Write 1 Done Done Sent false
30028 9284 Acked Read 2 Done Done Sent false
30029 9285 Acked Write 3 Done Done Sent false
30030 9286 Acked Write 3 Done Done Sent false
30031 9287 Acked Read 6 Done Done Sent false
30032 9288 Acked Read 4 Done Done Sent false
30033 9289 Acked Write 5 Done Done Sent false
30034 9290 Acked Write 6 Done Done Sent false
30035 9291 Acked Write 3 Done Done Sent false
30036 9292 Acked Write 5 Done Done Sent false
30037 9293 Acked Flush 0 Done Done Sent false
30038 9294 Acked Read 8 Done Done Sent false
30039 9295 Acked Read 10 Done Done Sent false
30040 9296 Acked Write 1 Done Done Sent false
30041 9297 Acked Write 2 Done Done Sent false
30042 9298 Acked Read 7 Done Done Sent false
30043 9299 Acked Flush 0 Done Done Sent false
30044 9300 Acked Write 9 Done Done Sent false
30045 9301 Acked Flush 0 Done Done Sent false
30046 9302 Acked Read 5 Done Done Sent false
30047 9303 Acked Read 7 Done Done Sent false
30048 9304 Acked Flush 0 Done Done Sent false
30049 9305 Acked Read 9 Done Done Sent false
30050 9306 Acked Read 6 Done Done Sent false
30051 9307 Acked Write 4 Done Done Sent false
30052 9308 Acked Read 10 Done Done Sent false
30053 9309 Acked Read 7 Done Done Sent false
30054 9310 Acked Read 4 Done Done Sent false
30055 9311 Acked Read 9 Done Done Sent false
30056 9312 Acked Flush 0 Done Done Sent false
30057 9313 Acked Read 8 Done Done Sent false
30058 9314 Acked Read 6 Done Done Sent false
30059 9315 Acked Write 3 Done Done Sent false
30060 9316 Acked Flush 0 Done Done Sent false
30061 9317 Acked Read 4 Done Done Sent false
30062 9318 Acked Read 6 Done Done Sent false
30063 9319 Acked Write 6 Done Done Sent false
30064 9320 Acked Flush 0 Done Done Sent false
30065 9321 Acked Read 6 Done Done Sent false
30066 9322 Acked Write 10 Done Done Sent false
30067 9323 Acked Read 4 Done Done Sent false
30068 9324 Acked Write 5 Done Done Sent false
30069 9325 Acked Write 1 Done Done Sent false
30070 9326 Acked Write 4 Done Done Sent false
30071 9327 Acked Read 3 Done Done Sent false
30072 9328 Acked Read 5 Done Done Sent false
30073 9329 Acked Read 2 Done Done Sent false
30074 9330 Acked Read 8 Done Done Sent false
30075 9331 Acked Flush 0 Done Done Sent false
30076 9332 Acked Write 9 Done Done Sent false
30077 9333 Acked Read 3 Done Done Sent false
30078 9334 Acked Write 2 Done Done Sent false
30079 9335 Acked Read 1 Done Done Sent false
30080 9336 Acked Read 9 Done Done Sent false
30081 9337 Acked Write 7 Done Done Sent false
30082 9338 Acked Read 7 Done Done Sent false
30083 9339 Acked Write 6 Done Done Sent false
30084 9340 Acked Write 6 Done Done Sent false
30085 9341 Acked Flush 0 Done Done Sent false
30086 9342 Acked Read 1 Done Done Sent false
30087 9343 Acked Read 7 Done Done Sent false
30088 9344 Acked Write 4 Done Done Sent false
30089 9345 Acked Read 5 Done Done Sent false
30090 9346 Acked Read 7 Done Done Sent false
30091 9347 Acked Read 5 Done Done Sent false
30092 9348 Acked Read 10 Done Done Sent false
30093 9349 Acked Read 3 Done Done Sent false
30094 9350 Acked Read 6 Done Done Sent false
30095 9351 Acked Write 3 Done Done Sent false
30096 9352 Acked Flush 0 Done Done Sent false
30097 9353 Acked Write 6 Done Done Sent false
30098 9354 Acked Write 1 Done Done Sent false
30099 9355 Acked Write 10 Done Done Sent false
30100 9356 Acked Read 2 Done Done Sent false
30101 9357 Acked Write 3 Done Done Sent false
30102 9358 Acked Write 2 Done Done Sent false
30103 9359 Acked Read 1 Done Done Sent false
30104 9360 Acked Read 6 Done Done Sent false
30105 9361 Acked Read 5 Done Done Sent false
30106 9362 Acked Flush 0 Done Done Sent false
30107 9363 Acked Read 3 Done Done Sent false
30108 9364 Acked Read 8 Done Done Sent false
30109 9365 Acked Read 9 Done Done Sent false
30110 9366 Acked Read 7 Done Done Sent false
30111 9367 Acked Write 3 Done Done Sent false
30112 9368 Acked Flush 0 Done Done Sent false
30113 9369 Acked Read 7 Done Done Sent false
30114 9370 Acked Read 7 Done Done Sent false
30115 9371 Acked Write 6 Done Done Sent false
30116 9372 Acked Write 5 Done Done Sent false
30117 9373 Acked Write 2 Done Done Sent false
30118 9374 Acked Write 6 Done Done Sent false
30119 9375 Acked Read 5 Done Done Sent false
30120 9376 Acked Write 6 Done Done Sent false
30121 9377 Acked Write 3 Done Done Sent false
30122 9378 Acked Read 7 Done Done Sent false
30123 9379 Acked Read 7 Done Done Sent false
30124 9380 Acked Write 1 Done Done Sent false
30125 9381 Acked Write 1 Done Done Sent false
30126 9382 Acked Write 1 Done Done Sent false
30127 9383 Acked Read 9 Done Done Sent false
30128 9384 Acked Write 4 Done Done Sent false
30129 9385 Acked Read 6 Done Done Sent false
30130 9386 Acked Write 7 Done Done Sent false
30131 9387 Acked Read 5 Done Done Sent false
30132 9388 Acked Read 6 Done Done Sent false
30133 9389 Acked Read 6 Done Done Sent false
30134 9390 Acked Read 2 Done Done Sent false
30135 9391 Acked Write 9 Done Done Sent false
30136 9392 Acked Write 7 Done Done Sent false
30137 9393 Acked Read 10 Done Done Sent false
30138 9394 Acked Write 9 Done Done Sent false
30139 9395 Acked Read 1 Done Done Sent false
30140 9396 Acked Read 9 Done Done Sent false
30141 9397 Acked Write 1 Done Done Sent false
30142 9398 Acked Write 8 Done Done Sent false
30143 9399 Acked Write 10 Done Done Sent false
30144 9400 Acked Write 1 Done Done Sent false
30145 9401 Acked Read 8 Done Done Sent false
30146 9402 Acked Read 6 Done Done Sent false
30147 9403 Acked Read 4 Done Done Sent false
30148 9404 Acked Read 10 Done Done Sent false
30149 9405 Acked Read 9 Done Done Sent false
30150 9406 Acked Write 5 Done Done Sent false
30151 9407 Acked Write 5 Done Done Sent false
30152 9408 Acked Read 9 Done Done Sent false
30153 9409 Acked Write 2 Done Done Sent false
30154 9410 Acked Read 6 Done Done Sent false
30155 9411 Acked Read 6 Done Done Sent false
30156 9412 Acked Read 3 Done Done Sent false
30157 9413 Acked Flush 0 Done Done Sent false
30158 9414 Acked Flush 0 Done Done Sent false
30159 9415 Acked Flush 0 Done Done Sent false
30160 9416 Acked Write 6 Done Done Sent false
30161 9417 Acked Read 2 Done Done Sent false
30162 9418 Acked Write 10 Done Done Sent false
30163 9419 Acked Read 6 Done Done Sent false
30164 9420 Acked Write 2 Done Done Sent false
30165 9421 Acked Read 1 Done Done Sent false
30166 9422 Acked Read 1 Done Done Sent false
30167 9423 Acked Read 4 Done Done Sent false
30168 9424 Acked Write 10 Done Done Sent false
30169 9425 Acked Write 10 Done Done Sent false
30170 9426 Acked Write 9 Done Done Sent false
30171 9427 Acked Write 4 Done Done Sent false
30172 9428 Acked Read 6 Done Done Sent false
30173 9429 Acked Read 5 Done Done Sent false
30174 9430 Acked Write 2 Done Done Sent false
30175 9431 Acked Read 2 Done Done Sent false
30176 9432 Acked Read 9 Done Done Sent false
30177 9433 Acked Write 1 Done Done Sent false
30178 9434 Acked Read 6 Done Done Sent false
30179 9435 Acked Read 4 Done Done Sent false
30180 9436 Acked Read 2 Done Done Sent false
30181 9437 Acked Write 5 Done Done Sent false
30182 9438 Acked Write 1 Done Done Sent false
30183 9439 Acked Read 5 Done Done Sent false
30184 9440 Acked Flush 0 Done Done Sent false
30185 9441 Acked Read 7 Done Done Sent false
30186 9442 Acked Write 4 Done Done Sent false
30187 9443 Acked Write 9 Done Done Sent false
30188 9444 Acked Write 7 Done Done Sent false
30189 9445 Acked Write 6 Done Done Sent false
30190 9446 Acked Read 2 Done Done Sent false
30191 9447 Acked Read 9 Done Done Sent false
30192 9448 Acked Read 2 Done Done Sent false
30193 9449 Acked Flush 0 Done Done Sent false
30194 9450 Acked Read 8 Done Done Sent false
30195 9451 Acked Read 8 Done Done Sent false
30196 STATES DS:0 DS:1 DS:2 TOTAL
30197 Sent 0 0 301 301
30198 Done 301 301 0 602
30199 Skipped 0 0 0 0
30200 Error 0 0 0 0
30201 Last Flush: Some(JobId(9449)) Some(JobId(9449)) Some(JobId(9150))
30202 Downstairs last five completed: 9150 9149 9148 9147 9146
30203 Upstairs last five completed: 9451 9450 9449 9448 9447
30204 CLIENT: Up:0 ds:301 act:2
30205 [8830] got start action
30206 [8830] I am going to restart
30207 Starting downstairs at port 8830
30208 Make output file at "/tmp/dsc/downstairs-8830.txt"
30209 [8830][2] reports Starting
30210 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1228)
30211 [8830][2] reports Running
30212 ----------------------------------------------------------------
30213 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
30214 States: Active Active Offline
30215 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30216 9151 Acked Read 8 Done Done Sent false
30217 9152 Acked Read 8 Done Done Sent false
30218 9153 Acked Write 6 Done Done Sent false
30219 9154 Acked Write 3 Done Done Sent false
30220 9155 Acked Write 8 Done Done Sent false
30221 9156 Acked Flush 0 Done Done Sent false
30222 9157 Acked Read 9 Done Done Sent false
30223 9158 Acked Read 9 Done Done Sent false
30224 9159 Acked Write 4 Done Done Sent false
30225 9160 Acked Write 4 Done Done Sent false
30226 9161 Acked Write 3 Done Done Sent false
30227 9162 Acked Write 6 Done Done Sent false
30228 9163 Acked Write 10 Done Done Sent false
30229 9164 Acked Write 9 Done Done Sent false
30230 9165 Acked Write 2 Done Done Sent false
30231 9166 Acked Write 2 Done Done Sent false
30232 9167 Acked Read 8 Done Done Sent false
30233 9168 Acked Write 4 Done Done Sent false
30234 9169 Acked Read 4 Done Done Sent false
30235 9170 Acked Write 1 Done Done Sent false
30236 9171 Acked Read 8 Done Done Sent false
30237 9172 Acked Write 6 Done Done Sent false
30238 9173 Acked Write 1 Done Done Sent false
30239 9174 Acked Write 8 Done Done Sent false
30240 9175 Acked Read 4 Done Done Sent false
30241 9176 Acked Read 3 Done Done Sent false
30242 9177 Acked Flush 0 Done Done Sent false
30243 9178 Acked Write 4 Done Done Sent false
30244 9179 Acked Write 1 Done Done Sent false
30245 9180 Acked Read 2 Done Done Sent false
30246 9181 Acked Read 8 Done Done Sent false
30247 9182 Acked Read 9 Done Done Sent false
30248 9183 Acked Write 5 Done Done Sent false
30249 9184 Acked Read 7 Done Done Sent false
30250 9185 Acked Write 6 Done Done Sent false
30251 9186 Acked Read 6 Done Done Sent false
30252 9187 Acked Flush 0 Done Done Sent false
30253 9188 Acked Read 2 Done Done Sent false
30254 9189 Acked Read 4 Done Done Sent false
30255 9190 Acked Write 8 Done Done Sent false
30256 9191 Acked Read 7 Done Done Sent false
30257 9192 Acked Read 7 Done Done Sent false
30258 9193 Acked Write 9 Done Done Sent false
30259 9194 Acked Write 10 Done Done Sent false
30260 9195 Acked Read 4 Done Done Sent false
30261 9196 Acked Write 2 Done Done Sent false
30262 9197 Acked Read 6 Done Done Sent false
30263 9198 Acked Read 9 Done Done Sent false
30264 9199 Acked Write 4 Done Done Sent false
30265 9200 Acked Flush 0 Done Done Sent false
30266 9201 Acked Write 2 Done Done Sent false
30267 9202 Acked Write 5 Done Done Sent false
30268 9203 Acked Read 1 Done Done Sent false
30269 9204 Acked Read 6 Done Done Sent false
30270 9205 Acked Read 3 Done Done Sent false
30271 9206 Acked Read 7 Done Done Sent false
30272 9207 Acked Write 1 Done Done Sent false
30273 9208 Acked Read 8 Done Done Sent false
30274 9209 Acked Write 9 Done Done Sent false
30275 9210 Acked Read 1 Done Done Sent false
30276 9211 Acked Flush 0 Done Done Sent false
30277 9212 Acked Write 3 Done Done Sent false
30278 9213 Acked Read 4 Done Done Sent false
30279 9214 Acked Read 7 Done Done Sent false
30280 9215 Acked Flush 0 Done Done Sent false
30281 9216 Acked Flush 0 Done Done Sent false
30282 9217 Acked Read 8 Done Done Sent false
30283 9218 Acked Read 2 Done Done Sent false
30284 9219 Acked Read 10 Done Done Sent false
30285 9220 Acked Flush 0 Done Done Sent false
30286 9221 Acked Write 8 Done Done Sent false
30287 9222 Acked Read 6 Done Done Sent false
30288 9223 Acked Write 6 Done Done Sent false
30289 9224 Acked Write 1 Done Done Sent false
30290 9225 Acked Read 2 Done Done Sent false
30291 9226 Acked Read 7 Done Done Sent false
30292 9227 Acked Read 9 Done Done Sent false
30293 9228 Acked Read 10 Done Done Sent false
30294 9229 Acked Write 5 Done Done Sent false
30295 9230 Acked Read 8 Done Done Sent false
30296 9231 Acked Write 4 Done Done Sent false
30297 9232 Acked Read 2 Done Done Sent false
30298 9233 Acked Read 3 Done Done Sent false
30299 9234 Acked Write 4 Done Done Sent false
30300 9235 Acked Read 8 Done Done Sent false
30301 9236 Acked Read 8 Done Done Sent false
30302 9237 Acked Write 8 Done Done Sent false
30303 9238 Acked Write 1 Done Done Sent false
30304 9239 Acked Write 3 Done Done Sent false
30305 9240 Acked Flush 0 Done Done Sent false
30306 9241 Acked Read 8 Done Done Sent false
30307 9242 Acked Flush 0 Done Done Sent false
30308 9243 Acked Read 1 Done Done Sent false
30309 9244 Acked Write 2 Done Done Sent false
30310 9245 Acked Read 10 Done Done Sent false
30311 9246 Acked Write 9 Done Done Sent false
30312 9247 Acked Flush 0 Done Done Sent false
30313 9248 Acked Read 3 Done Done Sent false
30314 9249 Acked Flush 0 Done Done Sent false
30315 9250 Acked Read 2 Done Done Sent false
30316 9251 Acked Read 8 Done Done Sent false
30317 9252 Acked Read 1 Done Done Sent false
30318 9253 Acked Write 2 Done Done Sent false
30319 9254 Acked Write 5 Done Done Sent false
30320 9255 Acked Read 3 Done Done Sent false
30321 9256 Acked Write 10 Done Done Sent false
30322 9257 Acked Read 8 Done Done Sent false
30323 9258 Acked Flush 0 Done Done Sent false
30324 9259 Acked Write 1 Done Done Sent false
30325 9260 Acked Write 7 Done Done Sent false
30326 9261 Acked Write 6 Done Done Sent false
30327 9262 Acked Write 10 Done Done Sent false
30328 9263 Acked Write 3 Done Done Sent false
30329 9264 Acked Write 5 Done Done Sent false
30330 9265 Acked Write 3 Done Done Sent false
30331 9266 Acked Write 7 Done Done Sent false
30332 9267 Acked Read 7 Done Done Sent false
30333 9268 Acked Read 9 Done Done Sent false
30334 9269 Acked Read 8 Done Done Sent false
30335 9270 Acked Read 7 Done Done Sent false
30336 9271 Acked Read 6 Done Done Sent false
30337 9272 Acked Write 8 Done Done Sent false
30338 9273 Acked Read 9 Done Done Sent false
30339 9274 Acked Write 2 Done Done Sent false
30340 9275 Acked Read 8 Done Done Sent false
30341 9276 Acked Write 3 Done Done Sent false
30342 9277 Acked Flush 0 Done Done Sent false
30343 9278 Acked Read 7 Done Done Sent false
30344 9279 Acked Read 4 Done Done Sent false
30345 9280 Acked Flush 0 Done Done Sent false
30346 9281 Acked Read 6 Done Done Sent false
30347 9282 Acked Write 5 Done Done Sent false
30348 9283 Acked Write 1 Done Done Sent false
30349 9284 Acked Read 2 Done Done Sent false
30350 9285 Acked Write 3 Done Done Sent false
30351 9286 Acked Write 3 Done Done Sent false
30352 9287 Acked Read 6 Done Done Sent false
30353 9288 Acked Read 4 Done Done Sent false
30354 9289 Acked Write 5 Done Done Sent false
30355 9290 Acked Write 6 Done Done Sent false
30356 9291 Acked Write 3 Done Done Sent false
30357 9292 Acked Write 5 Done Done Sent false
30358 9293 Acked Flush 0 Done Done Sent false
30359 9294 Acked Read 8 Done Done Sent false
30360 9295 Acked Read 10 Done Done Sent false
30361 9296 Acked Write 1 Done Done Sent false
30362 9297 Acked Write 2 Done Done Sent false
30363 9298 Acked Read 7 Done Done Sent false
30364 9299 Acked Flush 0 Done Done Sent false
30365 9300 Acked Write 9 Done Done Sent false
30366 9301 Acked Flush 0 Done Done Sent false
30367 9302 Acked Read 5 Done Done Sent false
30368 9303 Acked Read 7 Done Done Sent false
30369 9304 Acked Flush 0 Done Done Sent false
30370 9305 Acked Read 9 Done Done Sent false
30371 9306 Acked Read 6 Done Done Sent false
30372 9307 Acked Write 4 Done Done Sent false
30373 9308 Acked Read 10 Done Done Sent false
30374 9309 Acked Read 7 Done Done Sent false
30375 9310 Acked Read 4 Done Done Sent false
30376 9311 Acked Read 9 Done Done Sent false
30377 9312 Acked Flush 0 Done Done Sent false
30378 9313 Acked Read 8 Done Done Sent false
30379 9314 Acked Read 6 Done Done Sent false
30380 9315 Acked Write 3 Done Done Sent false
30381 9316 Acked Flush 0 Done Done Sent false
30382 9317 Acked Read 4 Done Done Sent false
30383 9318 Acked Read 6 Done Done Sent false
30384 9319 Acked Write 6 Done Done Sent false
30385 9320 Acked Flush 0 Done Done Sent false
30386 9321 Acked Read 6 Done Done Sent false
30387 9322 Acked Write 10 Done Done Sent false
30388 9323 Acked Read 4 Done Done Sent false
30389 9324 Acked Write 5 Done Done Sent false
30390 9325 Acked Write 1 Done Done Sent false
30391 9326 Acked Write 4 Done Done Sent false
30392 9327 Acked Read 3 Done Done Sent false
30393 9328 Acked Read 5 Done Done Sent false
30394 9329 Acked Read 2 Done Done Sent false
30395 9330 Acked Read 8 Done Done Sent false
30396 9331 Acked Flush 0 Done Done Sent false
30397 9332 Acked Write 9 Done Done Sent false
30398 9333 Acked Read 3 Done Done Sent false
30399 9334 Acked Write 2 Done Done Sent false
30400 9335 Acked Read 1 Done Done Sent false
30401 9336 Acked Read 9 Done Done Sent false
30402 9337 Acked Write 7 Done Done Sent false
30403 9338 Acked Read 7 Done Done Sent false
30404 9339 Acked Write 6 Done Done Sent false
30405 9340 Acked Write 6 Done Done Sent false
30406 9341 Acked Flush 0 Done Done Sent false
30407 9342 Acked Read 1 Done Done Sent false
30408 9343 Acked Read 7 Done Done Sent false
30409 9344 Acked Write 4 Done Done Sent false
30410 9345 Acked Read 5 Done Done Sent false
30411 9346 Acked Read 7 Done Done Sent false
30412 9347 Acked Read 5 Done Done Sent false
30413 9348 Acked Read 10 Done Done Sent false
30414 9349 Acked Read 3 Done Done Sent false
30415 9350 Acked Read 6 Done Done Sent false
30416 9351 Acked Write 3 Done Done Sent false
30417 9352 Acked Flush 0 Done Done Sent false
30418 9353 Acked Write 6 Done Done Sent false
30419 9354 Acked Write 1 Done Done Sent false
30420 9355 Acked Write 10 Done Done Sent false
30421 9356 Acked Read 2 Done Done Sent false
30422 9357 Acked Write 3 Done Done Sent false
30423 9358 Acked Write 2 Done Done Sent false
30424 9359 Acked Read 1 Done Done Sent false
30425 9360 Acked Read 6 Done Done Sent false
30426 9361 Acked Read 5 Done Done Sent false
30427 9362 Acked Flush 0 Done Done Sent false
30428 9363 Acked Read 3 Done Done Sent false
30429 9364 Acked Read 8 Done Done Sent false
30430 9365 Acked Read 9 Done Done Sent false
30431 9366 Acked Read 7 Done Done Sent false
30432 9367 Acked Write 3 Done Done Sent false
30433 9368 Acked Flush 0 Done Done Sent false
30434 9369 Acked Read 7 Done Done Sent false
30435 9370 Acked Read 7 Done Done Sent false
30436 9371 Acked Write 6 Done Done Sent false
30437 9372 Acked Write 5 Done Done Sent false
30438 9373 Acked Write 2 Done Done Sent false
30439 9374 Acked Write 6 Done Done Sent false
30440 9375 Acked Read 5 Done Done Sent false
30441 9376 Acked Write 6 Done Done Sent false
30442 9377 Acked Write 3 Done Done Sent false
30443 9378 Acked Read 7 Done Done Sent false
30444 9379 Acked Read 7 Done Done Sent false
30445 9380 Acked Write 1 Done Done Sent false
30446 9381 Acked Write 1 Done Done Sent false
30447 9382 Acked Write 1 Done Done Sent false
30448 9383 Acked Read 9 Done Done Sent false
30449 9384 Acked Write 4 Done Done Sent false
30450 9385 Acked Read 6 Done Done Sent false
30451 9386 Acked Write 7 Done Done Sent false
30452 9387 Acked Read 5 Done Done Sent false
30453 9388 Acked Read 6 Done Done Sent false
30454 9389 Acked Read 6 Done Done Sent false
30455 9390 Acked Read 2 Done Done Sent false
30456 9391 Acked Write 9 Done Done Sent false
30457 9392 Acked Write 7 Done Done Sent false
30458 9393 Acked Read 10 Done Done Sent false
30459 9394 Acked Write 9 Done Done Sent false
30460 9395 Acked Read 1 Done Done Sent false
30461 9396 Acked Read 9 Done Done Sent false
30462 9397 Acked Write 1 Done Done Sent false
30463 9398 Acked Write 8 Done Done Sent false
30464 9399 Acked Write 10 Done Done Sent false
30465 9400 Acked Write 1 Done Done Sent false
30466 9401 Acked Read 8 Done Done Sent false
30467 9402 Acked Read 6 Done Done Sent false
30468 9403 Acked Read 4 Done Done Sent false
30469 9404 Acked Read 10 Done Done Sent false
30470 9405 Acked Read 9 Done Done Sent false
30471 9406 Acked Write 5 Done Done Sent false
30472 9407 Acked Write 5 Done Done Sent false
30473 9408 Acked Read 9 Done Done Sent false
30474 9409 Acked Write 2 Done Done Sent false
30475 9410 Acked Read 6 Done Done Sent false
30476 9411 Acked Read 6 Done Done Sent false
30477 9412 Acked Read 3 Done Done Sent false
30478 9413 Acked Flush 0 Done Done Sent false
30479 9414 Acked Flush 0 Done Done Sent false
30480 9415 Acked Flush 0 Done Done Sent false
30481 9416 Acked Write 6 Done Done Sent false
30482 9417 Acked Read 2 Done Done Sent false
30483 9418 Acked Write 10 Done Done Sent false
30484 9419 Acked Read 6 Done Done Sent false
30485 9420 Acked Write 2 Done Done Sent false
30486 9421 Acked Read 1 Done Done Sent false
30487 9422 Acked Read 1 Done Done Sent false
30488 9423 Acked Read 4 Done Done Sent false
30489 9424 Acked Write 10 Done Done Sent false
30490 9425 Acked Write 10 Done Done Sent false
30491 9426 Acked Write 9 Done Done Sent false
30492 9427 Acked Write 4 Done Done Sent false
30493 9428 Acked Read 6 Done Done Sent false
30494 9429 Acked Read 5 Done Done Sent false
30495 9430 Acked Write 2 Done Done Sent false
30496 9431 Acked Read 2 Done Done Sent false
30497 9432 Acked Read 9 Done Done Sent false
30498 9433 Acked Write 1 Done Done Sent false
30499 9434 Acked Read 6 Done Done Sent false
30500 9435 Acked Read 4 Done Done Sent false
30501 9436 Acked Read 2 Done Done Sent false
30502 9437 Acked Write 5 Done Done Sent false
30503 9438 Acked Write 1 Done Done Sent false
30504 9439 Acked Read 5 Done Done Sent false
30505 9440 Acked Flush 0 Done Done Sent false
30506 9441 Acked Read 7 Done Done Sent false
30507 9442 Acked Write 4 Done Done Sent false
30508 9443 Acked Write 9 Done Done Sent false
30509 9444 Acked Write 7 Done Done Sent false
30510 9445 Acked Write 6 Done Done Sent false
30511 9446 Acked Read 2 Done Done Sent false
30512 9447 Acked Read 9 Done Done Sent false
30513 9448 Acked Read 2 Done Done Sent false
30514 9449 Acked Flush 0 Done Done Sent false
30515 9450 Acked Read 8 Done Done Sent false
30516 9451 Acked Read 8 Done Done Sent false
30517 9452 Acked Flush 0 Done Done Sent false
30518 STATES DS:0 DS:1 DS:2 TOTAL
30519 Sent 0 0 302 302
30520 Done 302 302 0 604
30521 Skipped 0 0 0 0
30522 Error 0 0 0 0
30523 Last Flush: Some(JobId(9452)) Some(JobId(9452)) Some(JobId(9150))
30524 Downstairs last five completed: 9150 9149 9148 9147 9146
30525 Upstairs last five completed: 9452 9451 9450 9449 9448
30526 CLIENT: Up:0 ds:302 act:2
305272025-01-23T18:54:17.699ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
305282025-01-23T18:54:17.699ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44793) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
305292025-01-23T18:54:17.700ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
305302025-01-23T18:54:17.701ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
305312025-01-23T18:54:17.701ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
305322025-01-23T18:54:17.701ZINFOcrucible: send last flush ID to this DS: Some(JobId(9150)) = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
305332025-01-23T18:54:17.701ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
305342025-01-23T18:54:17.701ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(9150)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
305352025-01-23T18:54:17.701ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(9150)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
30536 ----------------------------------------------------------------
30537 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30538 Downstairs last five completed: 9452 9451 9450 9449 9448
30539 Upstairs last five completed: 9452 9451 9450 9449 9448
30540 CLIENT: Up:0 ds:0 act:3
30541 Replay: All jobs finished
30542 Main task has work to do, go find it
30543 got dsc Stop(2)
30544 stop 2
30545 [8830] Got stop action so:false kr:false
30546 Jan 23 18:54:19.945 INFO request completed, latency_us: 322, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 86e3016c-5c93-4bbb-ae2c-30bce97cd5f3, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
305472025-01-23T18:54:19.946ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
305482025-01-23T18:54:19.946ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
305492025-01-23T18:54:19.946ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
305502025-01-23T18:54:19.946ZINFOcrucible: client task is exiting = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
30551 Jan 23 18:54:19.946 INFO request completed, latency_us: 150, response_code: 200, uri: /state/cid/2, method: GET, req_id: fa414931-fc31-469b-a7e4-c8db6ebbfde0, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
30552 [8830] Exited with: ExitStatus(unix_wait_status(9))
30553 [8830][2] reports Exit
30554 Jan 23 18:54:23.948 INFO request completed, latency_us: 191, response_code: 200, uri: /state/cid/2, method: GET, req_id: 437c7fcd-754c-4378-ba99-2edb3c7677c2, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
30555 001/300 Write block 1563 len 12288 data: 7 7 6
30556 002/300 Read block 3286 len 8192
30557 003/300 Read block 527 len 24576
30558 004/300 Write block 1832 len 16384 data: 6 5 5 5
30559 005/300 Write block 3459 len 20480 data: 8 8 8 8 8
30560 006/300 Write block 260 len 32768 data: 5 5 5 5 3 5 5 4
30561 007/300 Write block 1512 len 40960 data: 7 7 7 4 4 4 3 3 4 6
30562 008/300 Read block 2567 len 36864
30563 009/300 Read block 4263 len 28672
30564 010/300 Write block 1249 len 4096 data: 8
30565 011/300 Flush
30566 012/300 Read block 1047 len 20480
30567 013/300 Flush
30568 014/300 Read block 2485 len 24576
30569 015/300 Write block 4228 len 8192 data: 5 6
30570 016/300 Read block 1663 len 32768
30571 017/300 Read block 3269 len 24576
30572 018/300 Write block 2531 len 28672 data: 4 4 5 6 8 7 6
30573 019/300 Write block 916 len 4096 data: 5
30574 020/300 Read block 2802 len 28672
30575 021/300 Write block 3416 len 16384 data: 5 7 8 7
30576 022/300 Read block 954 len 24576
30577 023/300 Read block 4892 len 36864
30578 024/300 Read block 2839 len 8192
30579 025/300 Read block 2304 len 36864
30580 026/300 Write block 950 len 28672 data: 5 4 5 6 5 6 7
30581 027/300 Write block 4824 len 24576 data: 7 6 6 6 7 6
30582 028/300 Flush
30583 029/300 Read block 2039 len 8192
30584 030/300 Read block 3899 len 36864
30585 031/300 Read block 4502 len 4096
30586 032/300 Write block 4214 len 36864 data: 3 6 6 7 6 7 7 7 8
30587 033/300 Flush
30588 034/300 Write block 2810 len 40960 data: 3 4 4 2 2 4 4 3 4 4
30589 035/300 Write block 2573 len 20480 data: 6 8 9 8 10
30590 036/300 Read block 2678 len 12288
30591 037/300 Write block 3894 len 40960 data: 7 5 5 7 7 7 8 7 7 7
30592 038/300 Read block 2303 len 28672
30593 039/300 Write block 4260 len 8192 data: 5 5
30594 040/300 Read block 2680 len 16384
30595 041/300 Write block 3677 len 12288 data: 5 6 5
30596 042/300 Write block 3653 len 40960 data: 10 8 8 9 8 6 6 7 6 7
30597 043/300 Write block 3789 len 8192 data: 5 6
30598 044/300 Write block 766 len 16384 data: 5 5 5 5
30599 045/300 Read block 1186 len 12288
30600 046/300 Read block 937 len 8192
30601 047/300 Read block 2996 len 24576
30602 048/300 Write block 2329 len 4096 data: 5
30603 049/300 Write block 1380 len 4096 data: 6
30604 050/300 Read block 4452 len 36864
30605 051/300 Write block 774 len 24576 data: 4 5 9 10 10 11
30606 052/300 Write block 3591 len 24576 data: 6 6 7 7 8 6
30607 053/300 Write block 2563 len 16384 data: 4 4 4 4
30608 054/300 Read block 1554 len 36864
30609 055/300 Read block 1128 len 32768
30610 056/300 Flush
30611 057/300 Flush
30612 058/300 Write block 4707 len 20480 data: 7 9 8 7 8
30613 059/300 Read block 94 len 16384
30614 060/300 Read block 239 len 28672
30615 061/300 Write block 1864 len 4096 data: 8
30616 062/300 Write block 3577 len 12288 data: 7 6 5
30617 063/300 Write block 4276 len 32768 data: 5 6 5 7 6 6 5 4
30618 064/300 Flush
30619 065/300 Read block 1703 len 20480
30620 066/300 Flush
30621 067/300 Write block 103 len 36864 data: 8 7 6 5 4 5 8 6 8
30622 068/300 Flush
30623 069/300 Read block 1251 len 20480
30624 070/300 Read block 41 len 20480
30625 071/300 Read block 3440 len 36864
30626 072/300 Read block 316 len 28672
30627 073/300 Read block 2667 len 24576
30628 074/300 Write block 4274 len 4096 data: 6
30629 075/300 Write block 4299 len 16384 data: 5 4 3 3
30630 076/300 Read block 3828 len 24576
30631 077/300 Read block 2763 len 36864
30632 078/300 Read block 4272 len 28672
30633 079/300 Read block 2049 len 16384
30634 080/300 Read block 3610 len 36864
30635 081/300 Read block 294 len 24576
30636 082/300 Write block 4042 len 16384 data: 3 5 5 5
30637 083/300 Read block 4609 len 20480
30638 084/300 Flush
30639 085/300 Write block 4518 len 8192 data: 2 4
30640 086/300 Read block 487 len 36864
30641 087/300 Read block 3690 len 28672
30642 088/300 Write block 4 len 36864 data: 3 4 7 5 5 5 5 5 7
30643 089/300 Write block 4119 len 36864 data: 3 4 4 4 4 4 5 4 4
30644 090/300 Write block 507 len 16384 data: 7 6 4 4
30645 091/300 Read block 1628 len 8192
30646 092/300 Flush
30647 093/300 Write block 1588 len 8192 data: 5 6
30648 094/300 Read block 3482 len 36864
30649 095/300 Write block 3808 len 36864 data: 7 5 5 4 5 5 5 4 4
30650 096/300 Flush
30651 097/300 Flush
30652 098/300 Write block 2580 len 36864 data: 10 10 9 6 6 9 10 7 6
30653 099/300 Read block 863 len 20480
30654 100/300 Read block 4121 len 24576
30655 101/300 Write block 3118 len 28672 data: 5 5 7 8 7 7 7
30656 102/300 Write block 2606 len 28672 data: 9 8 8 6 6 6 5
30657 103/300 Write block 2975 len 12288 data: 4 3 3
30658 104/300 Write block 1258 len 20480 data: 6 5 6 6 7
30659 105/300 Write block 3632 len 12288 data: 6 6 6
30660 106/300 Write block 4756 len 16384 data: 7 7 8 8
30661 107/300 Write block 2208 len 16384 data: 8 6 7 8
30662 108/300 Read block 3507 len 4096
30663 109/300 Write block 1235 len 32768 data: 8 6 7 8 7 8 6 7
30664 110/300 Write block 2107 len 28672 data: 9 11 9 8 7 6 4
30665 111/300 Read block 4570 len 20480
30666 112/300 Read block 1741 len 40960
30667 113/300 Write block 3745 len 12288 data: 6 5 6
30668 114/300 Read block 4916 len 36864
30669 115/300 Write block 2523 len 36864 data: 10 9 4 4 4 4 4 4 5
30670 116/300 Write block 2495 len 40960 data: 6 7 10 9 9 7 6 5 5 5
30671 117/300 Write block 930 len 8192 data: 2 4
30672 118/300 Read block 134 len 20480
30673 119/300 Read block 191 len 4096
30674 120/300 Read block 1518 len 12288
30675 121/300 Read block 4825 len 32768
30676 122/300 Read block 187 len 40960
30677 123/300 Read block 2654 len 20480
30678 124/300 Read block 911 len 40960
30679 125/300 Read block 1617 len 40960
30680 126/300 Flush
30681 127/300 Write block 138 len 16384 data: 13 8 10 9
30682 128/300 Write block 4777 len 28672 data: 5 8 8 7 6 8 9
30683 129/300 Read block 3876 len 8192
30684 130/300 Read block 191 len 40960
30685 131/300 Read block 710 len 40960
30686 132/300 Write block 4985 len 28672 data: 6 6 6 8 5 5 5
30687 133/300 Write block 3462 len 40960 data: 9 9 11 8 5 4 4 3 3 3
30688 134/300 Read block 2724 len 8192
30689 135/300 Flush
30690 136/300 Write block 3780 len 32768 data: 9 7 7 6 5 5 5 6
30691 137/300 Read block 3044 len 32768
30692 138/300 Write block 3026 len 8192 data: 9 8
30693 139/300 Read block 2657 len 12288
30694 140/300 Read block 3797 len 36864
30695 141/300 Read block 3125 len 36864
30696 142/300 Write block 1670 len 40960 data: 5 3 5 5 5 6 6 6 7 7
30697 143/300 Read block 3891 len 16384
30698 144/300 Read block 2744 len 40960
30699 145/300 Write block 4806 len 32768 data: 8 8 9 9 9 8 8 8
30700 146/300 Read block 3164 len 36864
30701 147/300 Read block 2636 len 36864
30702 148/300 Read block 734 len 40960
30703 149/300 Flush
30704 150/300 Flush
30705 151/300 Read block 4202 len 20480
30706 152/300 Write block 4187 len 28672 data: 5 5 5 4 6 5 6
30707 153/300 Write block 2352 len 28672 data: 5 4 4 3 4 4 4
30708 154/300 Read block 316 len 20480
30709 155/300 Flush
30710 156/300 Write block 3113 len 20480 data: 4 5 5 6 6
30711 157/300 Read block 1187 len 4096
30712 158/300 Write block 2913 len 40960 data: 6 4 4 4 5 6 6 3 7 6
30713 159/300 Write block 4212 len 4096 data: 4
30714 160/300 Read block 2678 len 4096
30715 161/300 Read block 3081 len 20480
30716 162/300 Write block 3211 len 40960 data: 2 3 3 3 4 4 4 4 5 4
30717 163/300 Read block 1925 len 24576
30718 164/300 Read block 755 len 36864
30719 165/300 Read block 534 len 32768
30720 166/300 Read block 1763 len 28672
30721 167/300 Read block 2065 len 24576
30722 168/300 Write block 2396 len 24576 data: 5 7 6 5 5 6
30723 169/300 Write block 637 len 24576 data: 2 2 3 4 4 4
30724 170/300 Flush
30725 171/300 Write block 988 len 4096 data: 5
30726 172/300 Write block 2558 len 4096 data: 3
30727 173/300 Flush
30728 174/300 Read block 1616 len 24576
30729 175/300 Read block 4290 len 28672
30730 176/300 Write block 1098 len 4096 data: 10
30731 177/300 Write block 4564 len 40960 data: 7 6 6 7 7 4 5 4 6 5
30732 178/300 Flush
30733 179/300 Read block 409 len 4096
30734 180/300 Write block 3012 len 40960 data: 6 8 8 8 9 7 6 6 6 8
30735 181/300 Read block 1717 len 32768
30736 182/300 Read block 336 len 36864
30737 183/300 Read block 1270 len 36864
30738 184/300 Read block 2134 len 12288
30739 185/300 Write block 4394 len 28672 data: 5 3 3 5 6 3 3
30740 186/300 Write block 4426 len 16384 data: 7 7 8 7
30741 187/300 Write block 3071 len 8192 data: 6 6
30742 188/300 Read block 2334 len 16384
30743 189/300 Flush
30744 190/300 Write block 4608 len 36864 data: 6 8 4 5 4 3 3 3 2
30745 191/300 Write block 2354 len 24576 data: 5 4 5 5 5 5
30746 192/300 Write block 2945 len 16384 data: 6 6 6 7
30747 193/300 Write block 2958 len 20480 data: 4 5 6 6 7
30748 194/300 Write block 2666 len 36864 data: 3 3 3 5 8 10 9 10 8
30749 195/300 Flush
30750 196/300 Read block 2110 len 28672
30751 197/300 Read block 1199 len 24576
30752 198/300 Write block 133 len 4096 data: 8
30753 199/300 Read block 1533 len 28672
30754 200/300 Write block 3132 len 12288 data: 5 4 5
30755 201/300 Read block 3681 len 4096
30756 202/300 Write block 2878 len 8192 data: 3 3
30757 203/300 Write block 3874 len 20480 data: 5 7 7 9 10
30758 204/300 Read block 135 len 4096
30759 205/300 Write block 3651 len 20480 data: 7 9 11 9 9
30760 206/300 Read block 1073 len 40960
30761 207/300 Write block 3536 len 32768 data: 8 10 9 8 6 5 5 5
30762 208/300 Write block 4456 len 8192 data: 6 5
30763 209/300 Write block 4099 len 40960 data: 5 5 3 5 5 5 4 4 5 4
30764 210/300 Read block 1719 len 12288
30765 211/300 Read block 3548 len 28672
30766 212/300 Read block 4909 len 40960
30767 213/300 Read block 2410 len 8192
30768 214/300 Read block 4675 len 4096
30769 215/300 Read block 676 len 20480
30770 216/300 Read block 461 len 16384
30771 217/300 Read block 1246 len 8192
30772 218/300 Read block 1934 len 20480
30773 219/300 Write block 1761 len 4096 data: 7
30774 220/300 Write block 3074 len 24576 data: 3 4 4 5 4 5
30775 221/300 Read block 4632 len 12288
30776 222/300 Write block 4188 len 8192 data: 6 6
30777 223/300 Read block 4546 len 4096
30778 224/300 Read block 1736 len 40960
30779 225/300 Read block 4249 len 4096
30780 226/300 Read block 1696 len 8192
30781 227/300 Write block 1046 len 36864 data: 8 10 11 11 9 9 7 4 5
30782 228/300 Write block 4527 len 16384 data: 4 3 3 3
30783 229/300 Flush
30784 230/300 Write block 1969 len 12288 data: 7 6 4
30785 231/300 Read block 3770 len 28672
30786 232/300 Write block 4160 len 28672 data: 7 7 7 8 11 9 9
30787 233/300 Read block 1752 len 8192
30788 234/300 Write block 2605 len 20480 data: 8 10 9 9 7
30789 235/300 Read block 4443 len 32768
30790 236/300 Write block 1069 len 36864 data: 12 11 9 8 11 9 9 9 6
30791 237/300 Read block 143 len 4096
30792 238/300 Write block 3895 len 20480 data: 6 6 8 8 8
30793 239/300 Write block 3081 len 16384 data: 4 4 4 4
30794 240/300 Read block 364 len 28672
30795 241/300 Read block 4051 len 24576
30796 242/300 Write block 3722 len 28672 data: 6 5 4 3 4 4 4
30797 243/300 Write block 4397 len 32768 data: 6 7 4 4 4 4 6 5
30798 244/300 Write block 4450 len 12288 data: 11 11 10
30799 245/300 Read block 3573 len 32768
30800 246/300 Write block 2640 len 20480 data: 4 4 5 3 2
30801 247/300 Read block 3872 len 20480
30802 248/300 Read block 3093 len 28672
30803 249/300 Read block 3169 len 8192
30804 250/300 Write block 4128 len 28672 data: 4 4 3 4 4 4 5
30805 251/300 Read block 2176 len 20480
30806 252/300 Read block 609 len 40960
30807 253/300 Flush
30808 254/300 Write block 1404 len 40960 data: 7 6 6 6 5 4 4 3 4 4
30809 255/300 Read block 2135 len 8192
30810 256/300 Write block 4030 len 12288 data: 9 9 7
30811 257/300 Read block 4457 len 8192
30812 258/300 Write block 4870 len 12288 data: 6 7 7
30813 259/300 Read block 1847 len 28672
30814 260/300 Read block 117 len 16384
30815 261/300 Write block 3292 len 4096 data: 5
30816 262/300 Read block 18 len 4096
30817 263/300 Write block 3464 len 20480 data: 12 9 6 5 5
30818 264/300 Read block 3644 len 12288
30819 265/300 Read block 3360 len 4096
30820 266/300 Write block 3628 len 24576 data: 5 6 5 6 7 7
30821 267/300 Write block 887 len 28672 data: 8 6 5 5 5 4 5
30822 268/300 Read block 4227 len 24576
30823 269/300 Read block 2640 len 28672
30824 270/300 Write block 2461 len 12288 data: 8 7 6
30825 271/300 Write block 2491 len 32768 data: 7 7 5 5 7 8 11 10
30826 272/300 Read block 2025 len 32768
30827 273/300 Read block 2773 len 4096
30828 274/300 Read block 1836 len 20480
30829 275/300 Read block 751 len 40960
30830 276/300 Read block 28 len 32768
30831 277/300 Read block 2211 len 32768
30832 278/300 Write block 603 len 4096 data: 8
30833 279/300 Read block 3201 len 12288
30834 280/300 Write block 3274 len 28672 data: 6 5 6 6 4 5 7
30835 281/300 Write block 4369 len 8192 data: 7 7
30836 282/300 Flush
30837 283/300 Write block 189 len 32768 data: 6 5 5 4 3 3 3 2
30838 284/300 Flush
30839 285/300 Read block 1561 len 16384
30840 286/300 Read block 4373 len 8192
30841 287/300 Write block 2655 len 36864 data: 7 6 5 5 6 5 6 4 4
30842 288/300 Write block 2386 len 28672 data: 5 3 4 5 6 8 8
30843 289/300 Read block 2938 len 20480
30844 290/300 Write block 3662 len 8192 data: 8 7
30845 291/300 Write block 966 len 28672 data: 7 9 7 5 5 5 4
30846 292/300 Read block 3925 len 28672
30847 293/300 Read block 2723 len 20480
30848 294/300 Read block 2215 len 16384
30849 295/300 Write block 2871 len 40960 data: 11 11 10 7 6 3 3 4 4 2
30850 296/300 Write block 3079 len 40960 data: 6 5 5 5 5 5 5 5 4 5
30851 297/300 Read block 3582 len 40960
30852 298/300 Read block 357 len 12288
30853 299/300 Read block 2535 len 40960
30854 300/300 Write block 2075 len 40960 data: 7 7 9 7 8 7 7 7 7 7
30855 301/300 Read block 1804 len 8192
30856 Main task has work to do, go find it
30857 got dsc Start(2)
30858 start 2
30859 Jan 23 18:54:24.188 INFO request completed, latency_us: 199, response_code: 204, uri: /start/cid/2, method: POST, req_id: 772b78e0-7a8c-45f8-9001-b15b81d81437, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
30860 Replay: started 2, returned:Ok(())
30861 ----------------------------------------------------------------
30862 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30863 States: Active Active Offline
30864 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30865 9453 Acked Write 3 Done Done Sent false
30866 9454 Acked Read 2 Done Done Sent false
30867 9455 Acked Read 6 Done Done Sent false
30868 9456 Acked Write 4 Done Done Sent false
30869 9457 Acked Write 5 Done Done Sent false
30870 9458 Acked Write 8 Done Done Sent false
30871 9459 Acked Write 10 Done Done Sent false
30872 9460 Acked Read 9 Done Done Sent false
30873 9461 Acked Read 7 Done Done Sent false
30874 9462 Acked Write 1 Done Done Sent false
30875 9463 Acked Flush 0 Done Done Sent false
30876 9464 Acked Read 5 Done Done Sent false
30877 9465 Acked Flush 0 Done Done Sent false
30878 9466 Acked Read 6 Done Done Sent false
30879 9467 Acked Write 2 Done Done Sent false
30880 9468 Acked Read 8 Done Done Sent false
30881 9469 Acked Read 6 Done Done Sent false
30882 9470 Acked Write 7 Done Done Sent false
30883 9471 Acked Write 1 Done Done Sent false
30884 9472 Acked Read 7 Done Done Sent false
30885 9473 Acked Write 4 Done Done Sent false
30886 9474 Acked Read 6 Done Done Sent false
30887 9475 Acked Read 9 Done Done Sent false
30888 9476 Acked Read 2 Done Done Sent false
30889 9477 Acked Read 9 Done Done Sent false
30890 9478 Acked Write 7 Done Done Sent false
30891 9479 Acked Write 6 Done Done Sent false
30892 9480 Acked Flush 0 Done Done Sent false
30893 9481 Acked Read 2 Done Done Sent false
30894 9482 Acked Read 9 Done Done Sent false
30895 9483 Acked Read 1 Done Done Sent false
30896 9484 Acked Write 9 Done Done Sent false
30897 9485 Acked Flush 0 Done Done Sent false
30898 9486 Acked Write 10 Done Done Sent false
30899 9487 Acked Write 5 Done Done Sent false
30900 9488 Acked Read 3 Done Done Sent false
30901 9489 Acked Write 10 Done Done Sent false
30902 9490 Acked Read 7 Done Done Sent false
30903 9491 Acked Write 2 Done Done Sent false
30904 9492 Acked Read 4 Done Done Sent false
30905 9493 Acked Write 3 Done Done Sent false
30906 9494 Acked Write 10 Done Done Sent false
30907 9495 Acked Write 2 Done Done Sent false
30908 9496 Acked Write 4 Done Done Sent false
30909 9497 Acked Read 3 Done Done Sent false
30910 9498 Acked Read 2 Done Done Sent false
30911 9499 Acked Read 6 Done Done Sent false
30912 9500 Acked Write 1 Done Done Sent false
30913 9501 Acked Write 1 Done Done Sent false
30914 9502 Acked Read 9 Done Done Sent false
30915 9503 Acked Write 6 Done Done Sent false
30916 9504 Acked Write 6 Done Done Sent false
30917 9505 Acked Write 4 Done Done Sent false
30918 9506 Acked Read 9 Done Done Sent false
30919 9507 Acked Read 8 Done Done Sent false
30920 9508 Acked Flush 0 Done Done Sent false
30921 9509 Acked Flush 0 Done Done Sent false
30922 9510 Acked Write 5 Done Done Sent false
30923 9511 Acked Read 4 Done Done Sent false
30924 9512 Acked Read 7 Done Done Sent false
30925 9513 Acked Write 1 Done Done Sent false
30926 9514 Acked Write 3 Done Done Sent false
30927 9515 Acked Write 8 Done Done Sent false
30928 9516 Acked Flush 0 Done Done Sent false
30929 9517 Acked Read 5 Done Done Sent false
30930 9518 Acked Flush 0 Done Done Sent false
30931 9519 Acked Write 9 Done Done Sent false
30932 9520 Acked Flush 0 Done Done Sent false
30933 9521 Acked Read 5 Done Done Sent false
30934 9522 Acked Read 5 Done Done Sent false
30935 9523 Acked Read 9 Done Done Sent false
30936 9524 Acked Read 7 Done Done Sent false
30937 9525 Acked Read 6 Done Done Sent false
30938 9526 Acked Write 1 Done Done Sent false
30939 9527 Acked Write 4 Done Done Sent false
30940 9528 Acked Read 6 Done Done Sent false
30941 9529 Acked Read 9 Done Done Sent false
30942 9530 Acked Read 7 Done Done Sent false
30943 9531 Acked Read 4 Done Done Sent false
30944 9532 Acked Read 9 Done Done Sent false
30945 9533 Acked Read 6 Done Done Sent false
30946 9534 Acked Write 4 Done Done Sent false
30947 9535 Acked Read 5 Done Done Sent false
30948 9536 Acked Flush 0 Done Done Sent false
30949 9537 Acked Write 2 Done Done Sent false
30950 9538 Acked Read 9 Done Done Sent false
30951 9539 Acked Read 7 Done Done Sent false
30952 9540 Acked Write 9 Done Done Sent false
30953 9541 Acked Write 9 Done Done Sent false
30954 9542 Acked Write 4 Done Done Sent false
30955 9543 Acked Read 2 Done Done Sent false
30956 9544 Acked Flush 0 Done Done Sent false
30957 9545 Acked Write 2 Done Done Sent false
30958 9546 Acked Read 9 Done Done Sent false
30959 9547 Acked Write 9 Done Done Sent false
30960 9548 Acked Flush 0 Done Done Sent false
30961 9549 Acked Flush 0 Done Done Sent false
30962 9550 Acked Write 9 Done Done Sent false
30963 9551 Acked Read 5 Done Done Sent false
30964 9552 Acked Read 6 Done Done Sent false
30965 9553 Acked Write 7 Done Done Sent false
30966 9554 Acked Write 7 Done Done Sent false
30967 9555 Acked Write 3 Done Done Sent false
30968 9556 Acked Write 5 Done Done Sent false
30969 9557 Acked Write 3 Done Done Sent false
30970 9558 Acked Write 4 Done Done Sent false
30971 9559 Acked Write 4 Done Done Sent false
30972 9560 Acked Read 1 Done Done Sent false
30973 9561 Acked Write 8 Done Done Sent false
30974 9562 Acked Write 7 Done Done Sent false
30975 9563 Acked Read 5 Done Done Sent false
30976 9564 Acked Read 10 Done Done Sent false
30977 9565 Acked Write 3 Done Done Sent false
30978 9566 Acked Read 9 Done Done Sent false
30979 9567 Acked Write 9 Done Done Sent false
30980 9568 Acked Write 10 Done Done Sent false
30981 9569 Acked Write 2 Done Done Sent false
30982 9570 Acked Read 5 Done Done Sent false
30983 9571 Acked Read 1 Done Done Sent false
30984 9572 Acked Read 3 Done Done Sent false
30985 9573 Acked Read 8 Done Done Sent false
30986 9574 Acked Read 10 Done Done Sent false
30987 9575 Acked Read 5 Done Done Sent false
30988 9576 Acked Read 10 Done Done Sent false
30989 9577 Acked Read 10 Done Done Sent false
30990 9578 Acked Flush 0 Done Done Sent false
30991 9579 Acked Write 4 Done Done Sent false
30992 9580 Acked Write 7 Done Done Sent false
30993 9581 Acked Read 2 Done Done Sent false
30994 9582 Acked Read 10 Done Done Sent false
30995 9583 Acked Read 10 Done Done Sent false
30996 9584 Acked Write 7 Done Done Sent false
30997 9585 Acked Write 10 Done Done Sent false
30998 9586 Acked Read 2 Done Done Sent false
30999 9587 Acked Flush 0 Done Done Sent false
31000 9588 Acked Write 8 Done Done Sent false
31001 9589 Acked Read 8 Done Done Sent false
31002 9590 Acked Write 2 Done Done Sent false
31003 9591 Acked Read 3 Done Done Sent false
31004 9592 Acked Read 9 Done Done Sent false
31005 9593 Acked Read 9 Done Done Sent false
31006 9594 Acked Write 10 Done Done Sent false
31007 9595 Acked Read 4 Done Done Sent false
31008 9596 Acked Read 10 Done Done Sent false
31009 9597 Acked Write 8 Done Done Sent false
31010 9598 Acked Read 9 Done Done Sent false
31011 9599 Acked Read 9 Done Done Sent false
31012 9600 Acked Read 10 Done Done Sent false
31013 9601 Acked Flush 0 Done Done Sent false
31014 9602 Acked Flush 0 Done Done Sent false
31015 9603 Acked Read 5 Done Done Sent false
31016 9604 Acked Write 7 Done Done Sent false
31017 9605 Acked Write 7 Done Done Sent false
31018 9606 Acked Read 5 Done Done Sent false
31019 9607 Acked Flush 0 Done Done Sent false
31020 9608 Acked Write 5 Done Done Sent false
31021 9609 Acked Read 1 Done Done Sent false
31022 9610 Acked Write 10 Done Done Sent false
31023 9611 Acked Write 1 Done Done Sent false
31024 9612 Acked Read 1 Done Done Sent false
31025 9613 Acked Read 5 Done Done Sent false
31026 9614 Acked Write 10 Done Done Sent false
31027 9615 Acked Read 6 Done Done Sent false
31028 9616 Acked Read 9 Done Done Sent false
31029 9617 Acked Read 8 Done Done Sent false
31030 9618 Acked Read 7 Done Done Sent false
31031 9619 Acked Read 6 Done Done Sent false
31032 9620 Acked Write 6 Done Done Sent false
31033 9621 Acked Write 6 Done Done Sent false
31034 9622 Acked Flush 0 Done Done Sent false
31035 9623 Acked Write 1 Done Done Sent false
31036 9624 Acked Write 1 Done Done Sent false
31037 9625 Acked Flush 0 Done Done Sent false
31038 9626 Acked Read 6 Done Done Sent false
31039 9627 Acked Read 7 Done Done Sent false
31040 9628 Acked Write 1 Done Done Sent false
31041 9629 Acked Write 10 Done Done Sent false
31042 9630 Acked Flush 0 Done Done Sent false
31043 9631 Acked Read 1 Done Done Sent false
31044 9632 Acked Write 10 Done Done Sent false
31045 9633 Acked Read 8 Done Done Sent false
31046 9634 Acked Read 9 Done Done Sent false
31047 9635 Acked Read 9 Done Done Sent false
31048 9636 Acked Read 3 Done Done Sent false
31049 9637 Acked Write 7 Done Done Sent false
31050 9638 Acked Write 4 Done Done Sent false
31051 9639 Acked Write 2 Done Done Sent false
31052 9640 Acked Read 4 Done Done Sent false
31053 9641 Acked Flush 0 Done Done Sent false
31054 9642 Acked Write 9 Done Done Sent false
31055 9643 Acked Write 6 Done Done Sent false
31056 9644 Acked Write 4 Done Done Sent false
31057 9645 Acked Write 5 Done Done Sent false
31058 9646 Acked Write 9 Done Done Sent false
31059 9647 Acked Flush 0 Done Done Sent false
31060 9648 Acked Read 7 Done Done Sent false
31061 9649 Acked Read 6 Done Done Sent false
31062 9650 Acked Write 1 Done Done Sent false
31063 9651 Acked Read 7 Done Done Sent false
31064 9652 Acked Write 3 Done Done Sent false
31065 9653 Acked Read 1 Done Done Sent false
31066 9654 Acked Write 2 Done Done Sent false
31067 9655 Acked Write 5 Done Done Sent false
31068 9656 Acked Read 1 Done Done Sent false
31069 9657 Acked Write 5 Done Done Sent false
31070 9658 Acked Read 10 Done Done Sent false
31071 9659 Acked Write 8 Done Done Sent false
31072 9660 Acked Write 2 Done Done Sent false
31073 9661 Acked Write 10 Done Done Sent false
31074 9662 Acked Read 3 Done Done Sent false
31075 9663 Acked Read 7 Done Done Sent false
31076 9664 Acked Read 10 Done Done Sent false
31077 9665 Acked Read 2 Done Done Sent false
31078 9666 Acked Read 1 Done Done Sent false
31079 9667 Acked Read 5 Done Done Sent false
31080 9668 Acked Read 4 Done Done Sent false
31081 9669 Acked Read 2 Done Done Sent false
31082 9670 Acked Read 5 Done Done Sent false
31083 9671 Acked Write 1 Done Done Sent false
31084 9672 Acked Write 6 Done Done Sent false
31085 9673 Acked Read 3 Done Done Sent false
31086 9674 Acked Write 2 Done Done Sent false
31087 9675 Acked Read 1 Done Done Sent false
31088 9676 Acked Read 10 Done Done Sent false
31089 9677 Acked Read 1 Done Done Sent false
31090 9678 Acked Read 2 Done Done Sent false
31091 9679 Acked Write 9 Done Done Sent false
31092 9680 Acked Write 4 Done Done Sent false
31093 9681 Acked Flush 0 Done Done Sent false
31094 9682 Acked Write 3 Done Done Sent false
31095 9683 Acked Read 7 Done Done Sent false
31096 9684 Acked Write 7 Done Done Sent false
31097 9685 Acked Read 2 Done Done Sent false
31098 9686 Acked Write 5 Done Done Sent false
31099 9687 Acked Read 8 Done Done Sent false
31100 9688 Acked Write 9 Done Done Sent false
31101 9689 Acked Read 1 Done Done Sent false
31102 9690 Acked Write 5 Done Done Sent false
31103 9691 Acked Write 4 Done Done Sent false
31104 9692 Acked Read 7 Done Done Sent false
31105 9693 Acked Read 6 Done Done Sent false
31106 9694 Acked Write 7 Done Done Sent false
31107 9695 Acked Write 8 Done Done Sent false
31108 9696 Acked Write 3 Done Done Sent false
31109 9697 Acked Read 8 Done Done Sent false
31110 9698 Acked Write 5 Done Done Sent false
31111 9699 Acked Read 5 Done Done Sent false
31112 9700 Acked Read 7 Done Done Sent false
31113 9701 Acked Read 2 Done Done Sent false
31114 9702 Acked Write 7 Done Done Sent false
31115 9703 Acked Read 5 Done Done Sent false
31116 9704 Acked Read 10 Done Done Sent false
31117 9705 Acked Flush 0 Done Done Sent false
31118 9706 Acked Write 10 Done Done Sent false
31119 9707 Acked Read 2 Done Done Sent false
31120 9708 Acked Write 3 Done Done Sent false
31121 9709 Acked Read 2 Done Done Sent false
31122 9710 Acked Write 3 Done Done Sent false
31123 9711 Acked Read 7 Done Done Sent false
31124 9712 Acked Read 4 Done Done Sent false
31125 9713 Acked Write 1 Done Done Sent false
31126 9714 Acked Read 1 Done Done Sent false
31127 9715 Acked Write 5 Done Done Sent false
31128 9716 Acked Read 3 Done Done Sent false
31129 9717 Acked Read 1 Done Done Sent false
31130 9718 Acked Write 6 Done Done Sent false
31131 9719 Acked Write 7 Done Done Sent false
31132 9720 Acked Read 6 Done Done Sent false
31133 9721 Acked Read 7 Done Done Sent false
31134 9722 Acked Write 3 Done Done Sent false
31135 9723 Acked Write 8 Done Done Sent false
31136 9724 Acked Read 8 Done Done Sent false
31137 9725 Acked Read 1 Done Done Sent false
31138 9726 Acked Read 5 Done Done Sent false
31139 9727 Acked Read 10 Done Done Sent false
31140 9728 Acked Read 8 Done Done Sent false
31141 9729 Acked Read 8 Done Done Sent false
31142 9730 Acked Write 1 Done Done Sent false
31143 9731 Acked Read 3 Done Done Sent false
31144 9732 Acked Write 7 Done Done Sent false
31145 9733 Acked Write 2 Done Done Sent false
31146 9734 Acked Flush 0 Done Done Sent false
31147 9735 Acked Write 8 Done Done Sent false
31148 9736 Acked Flush 0 Done Done Sent false
31149 9737 Acked Read 4 Done Done Sent false
31150 9738 Acked Read 2 Done Done Sent false
31151 9739 Acked Write 9 Done Done Sent false
31152 9740 Acked Write 7 Done Done Sent false
31153 9741 Acked Read 5 Done Done Sent false
31154 9742 Acked Write 2 Done Done Sent false
31155 9743 Acked Write 7 Done Done Sent false
31156 9744 Acked Read 7 Done Done Sent false
31157 9745 Acked Read 5 Done Done Sent false
31158 9746 Acked Read 4 Done Done Sent false
31159 9747 Acked Write 10 Done Done Sent false
31160 9748 Acked Write 10 Done Done Sent false
31161 9749 Acked Read 10 Done Done Sent false
31162 9750 Acked Read 3 Done Done Sent false
31163 9751 Acked Read 10 Done Done Sent false
31164 9752 Acked Write 10 Done Done Sent false
31165 9753 Acked Read 2 Done Done Sent false
31166 STATES DS:0 DS:1 DS:2 TOTAL
31167 Sent 0 0 301 301
31168 Done 301 301 0 602
31169 Skipped 0 0 0 0
31170 Error 0 0 0 0
31171 Last Flush: Some(JobId(9736)) Some(JobId(9736)) Some(JobId(9452))
31172 Downstairs last five completed: 9452 9451 9450 9449 9448
31173 Upstairs last five completed: 9753 9752 9751 9750 9749
31174 CLIENT: Up:0 ds:301 act:2
31175 [8830] got start action
31176 [8830] I am going to restart
31177 Starting downstairs at port 8830
31178 Make output file at "/tmp/dsc/downstairs-8830.txt"
31179 [8830][2] reports Starting
31180 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1229)
31181 [8830][2] reports Running
31182 ----------------------------------------------------------------
31183 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
31184 States: Active Active Offline
31185 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31186 9453 Acked Write 3 Done Done Sent false
31187 9454 Acked Read 2 Done Done Sent false
31188 9455 Acked Read 6 Done Done Sent false
31189 9456 Acked Write 4 Done Done Sent false
31190 9457 Acked Write 5 Done Done Sent false
31191 9458 Acked Write 8 Done Done Sent false
31192 9459 Acked Write 10 Done Done Sent false
31193 9460 Acked Read 9 Done Done Sent false
31194 9461 Acked Read 7 Done Done Sent false
31195 9462 Acked Write 1 Done Done Sent false
31196 9463 Acked Flush 0 Done Done Sent false
31197 9464 Acked Read 5 Done Done Sent false
31198 9465 Acked Flush 0 Done Done Sent false
31199 9466 Acked Read 6 Done Done Sent false
31200 9467 Acked Write 2 Done Done Sent false
31201 9468 Acked Read 8 Done Done Sent false
31202 9469 Acked Read 6 Done Done Sent false
31203 9470 Acked Write 7 Done Done Sent false
31204 9471 Acked Write 1 Done Done Sent false
31205 9472 Acked Read 7 Done Done Sent false
31206 9473 Acked Write 4 Done Done Sent false
31207 9474 Acked Read 6 Done Done Sent false
31208 9475 Acked Read 9 Done Done Sent false
31209 9476 Acked Read 2 Done Done Sent false
31210 9477 Acked Read 9 Done Done Sent false
31211 9478 Acked Write 7 Done Done Sent false
31212 9479 Acked Write 6 Done Done Sent false
31213 9480 Acked Flush 0 Done Done Sent false
31214 9481 Acked Read 2 Done Done Sent false
31215 9482 Acked Read 9 Done Done Sent false
31216 9483 Acked Read 1 Done Done Sent false
31217 9484 Acked Write 9 Done Done Sent false
31218 9485 Acked Flush 0 Done Done Sent false
31219 9486 Acked Write 10 Done Done Sent false
31220 9487 Acked Write 5 Done Done Sent false
31221 9488 Acked Read 3 Done Done Sent false
31222 9489 Acked Write 10 Done Done Sent false
31223 9490 Acked Read 7 Done Done Sent false
31224 9491 Acked Write 2 Done Done Sent false
31225 9492 Acked Read 4 Done Done Sent false
31226 9493 Acked Write 3 Done Done Sent false
31227 9494 Acked Write 10 Done Done Sent false
31228 9495 Acked Write 2 Done Done Sent false
31229 9496 Acked Write 4 Done Done Sent false
31230 9497 Acked Read 3 Done Done Sent false
31231 9498 Acked Read 2 Done Done Sent false
31232 9499 Acked Read 6 Done Done Sent false
31233 9500 Acked Write 1 Done Done Sent false
31234 9501 Acked Write 1 Done Done Sent false
31235 9502 Acked Read 9 Done Done Sent false
31236 9503 Acked Write 6 Done Done Sent false
31237 9504 Acked Write 6 Done Done Sent false
31238 9505 Acked Write 4 Done Done Sent false
31239 9506 Acked Read 9 Done Done Sent false
31240 9507 Acked Read 8 Done Done Sent false
31241 9508 Acked Flush 0 Done Done Sent false
31242 9509 Acked Flush 0 Done Done Sent false
31243 9510 Acked Write 5 Done Done Sent false
31244 9511 Acked Read 4 Done Done Sent false
31245 9512 Acked Read 7 Done Done Sent false
31246 9513 Acked Write 1 Done Done Sent false
31247 9514 Acked Write 3 Done Done Sent false
31248 9515 Acked Write 8 Done Done Sent false
31249 9516 Acked Flush 0 Done Done Sent false
31250 9517 Acked Read 5 Done Done Sent false
31251 9518 Acked Flush 0 Done Done Sent false
31252 9519 Acked Write 9 Done Done Sent false
31253 9520 Acked Flush 0 Done Done Sent false
31254 9521 Acked Read 5 Done Done Sent false
31255 9522 Acked Read 5 Done Done Sent false
31256 9523 Acked Read 9 Done Done Sent false
31257 9524 Acked Read 7 Done Done Sent false
31258 9525 Acked Read 6 Done Done Sent false
31259 9526 Acked Write 1 Done Done Sent false
31260 9527 Acked Write 4 Done Done Sent false
31261 9528 Acked Read 6 Done Done Sent false
31262 9529 Acked Read 9 Done Done Sent false
31263 9530 Acked Read 7 Done Done Sent false
31264 9531 Acked Read 4 Done Done Sent false
31265 9532 Acked Read 9 Done Done Sent false
31266 9533 Acked Read 6 Done Done Sent false
31267 9534 Acked Write 4 Done Done Sent false
31268 9535 Acked Read 5 Done Done Sent false
31269 9536 Acked Flush 0 Done Done Sent false
31270 9537 Acked Write 2 Done Done Sent false
31271 9538 Acked Read 9 Done Done Sent false
31272 9539 Acked Read 7 Done Done Sent false
31273 9540 Acked Write 9 Done Done Sent false
31274 9541 Acked Write 9 Done Done Sent false
31275 9542 Acked Write 4 Done Done Sent false
31276 9543 Acked Read 2 Done Done Sent false
31277 9544 Acked Flush 0 Done Done Sent false
31278 9545 Acked Write 2 Done Done Sent false
31279 9546 Acked Read 9 Done Done Sent false
31280 9547 Acked Write 9 Done Done Sent false
31281 9548 Acked Flush 0 Done Done Sent false
31282 9549 Acked Flush 0 Done Done Sent false
31283 9550 Acked Write 9 Done Done Sent false
31284 9551 Acked Read 5 Done Done Sent false
31285 9552 Acked Read 6 Done Done Sent false
31286 9553 Acked Write 7 Done Done Sent false
31287 9554 Acked Write 7 Done Done Sent false
31288 9555 Acked Write 3 Done Done Sent false
31289 9556 Acked Write 5 Done Done Sent false
31290 9557 Acked Write 3 Done Done Sent false
31291 9558 Acked Write 4 Done Done Sent false
31292 9559 Acked Write 4 Done Done Sent false
31293 9560 Acked Read 1 Done Done Sent false
31294 9561 Acked Write 8 Done Done Sent false
31295 9562 Acked Write 7 Done Done Sent false
31296 9563 Acked Read 5 Done Done Sent false
31297 9564 Acked Read 10 Done Done Sent false
31298 9565 Acked Write 3 Done Done Sent false
31299 9566 Acked Read 9 Done Done Sent false
31300 9567 Acked Write 9 Done Done Sent false
31301 9568 Acked Write 10 Done Done Sent false
31302 9569 Acked Write 2 Done Done Sent false
31303 9570 Acked Read 5 Done Done Sent false
31304 9571 Acked Read 1 Done Done Sent false
31305 9572 Acked Read 3 Done Done Sent false
31306 9573 Acked Read 8 Done Done Sent false
31307 9574 Acked Read 10 Done Done Sent false
31308 9575 Acked Read 5 Done Done Sent false
31309 9576 Acked Read 10 Done Done Sent false
31310 9577 Acked Read 10 Done Done Sent false
31311 9578 Acked Flush 0 Done Done Sent false
31312 9579 Acked Write 4 Done Done Sent false
31313 9580 Acked Write 7 Done Done Sent false
31314 9581 Acked Read 2 Done Done Sent false
31315 9582 Acked Read 10 Done Done Sent false
31316 9583 Acked Read 10 Done Done Sent false
31317 9584 Acked Write 7 Done Done Sent false
31318 9585 Acked Write 10 Done Done Sent false
31319 9586 Acked Read 2 Done Done Sent false
31320 9587 Acked Flush 0 Done Done Sent false
31321 9588 Acked Write 8 Done Done Sent false
31322 9589 Acked Read 8 Done Done Sent false
31323 9590 Acked Write 2 Done Done Sent false
31324 9591 Acked Read 3 Done Done Sent false
31325 9592 Acked Read 9 Done Done Sent false
31326 9593 Acked Read 9 Done Done Sent false
31327 9594 Acked Write 10 Done Done Sent false
31328 9595 Acked Read 4 Done Done Sent false
31329 9596 Acked Read 10 Done Done Sent false
31330 9597 Acked Write 8 Done Done Sent false
31331 9598 Acked Read 9 Done Done Sent false
31332 9599 Acked Read 9 Done Done Sent false
31333 9600 Acked Read 10 Done Done Sent false
31334 9601 Acked Flush 0 Done Done Sent false
31335 9602 Acked Flush 0 Done Done Sent false
31336 9603 Acked Read 5 Done Done Sent false
31337 9604 Acked Write 7 Done Done Sent false
31338 9605 Acked Write 7 Done Done Sent false
31339 9606 Acked Read 5 Done Done Sent false
31340 9607 Acked Flush 0 Done Done Sent false
31341 9608 Acked Write 5 Done Done Sent false
31342 9609 Acked Read 1 Done Done Sent false
31343 9610 Acked Write 10 Done Done Sent false
31344 9611 Acked Write 1 Done Done Sent false
31345 9612 Acked Read 1 Done Done Sent false
31346 9613 Acked Read 5 Done Done Sent false
31347 9614 Acked Write 10 Done Done Sent false
31348 9615 Acked Read 6 Done Done Sent false
31349 9616 Acked Read 9 Done Done Sent false
31350 9617 Acked Read 8 Done Done Sent false
31351 9618 Acked Read 7 Done Done Sent false
31352 9619 Acked Read 6 Done Done Sent false
31353 9620 Acked Write 6 Done Done Sent false
31354 9621 Acked Write 6 Done Done Sent false
31355 9622 Acked Flush 0 Done Done Sent false
31356 9623 Acked Write 1 Done Done Sent false
31357 9624 Acked Write 1 Done Done Sent false
31358 9625 Acked Flush 0 Done Done Sent false
31359 9626 Acked Read 6 Done Done Sent false
31360 9627 Acked Read 7 Done Done Sent false
31361 9628 Acked Write 1 Done Done Sent false
31362 9629 Acked Write 10 Done Done Sent false
31363 9630 Acked Flush 0 Done Done Sent false
31364 9631 Acked Read 1 Done Done Sent false
31365 9632 Acked Write 10 Done Done Sent false
31366 9633 Acked Read 8 Done Done Sent false
31367 9634 Acked Read 9 Done Done Sent false
31368 9635 Acked Read 9 Done Done Sent false
31369 9636 Acked Read 3 Done Done Sent false
31370 9637 Acked Write 7 Done Done Sent false
31371 9638 Acked Write 4 Done Done Sent false
31372 9639 Acked Write 2 Done Done Sent false
31373 9640 Acked Read 4 Done Done Sent false
31374 9641 Acked Flush 0 Done Done Sent false
31375 9642 Acked Write 9 Done Done Sent false
31376 9643 Acked Write 6 Done Done Sent false
31377 9644 Acked Write 4 Done Done Sent false
31378 9645 Acked Write 5 Done Done Sent false
31379 9646 Acked Write 9 Done Done Sent false
31380 9647 Acked Flush 0 Done Done Sent false
31381 9648 Acked Read 7 Done Done Sent false
31382 9649 Acked Read 6 Done Done Sent false
31383 9650 Acked Write 1 Done Done Sent false
31384 9651 Acked Read 7 Done Done Sent false
31385 9652 Acked Write 3 Done Done Sent false
31386 9653 Acked Read 1 Done Done Sent false
31387 9654 Acked Write 2 Done Done Sent false
31388 9655 Acked Write 5 Done Done Sent false
31389 9656 Acked Read 1 Done Done Sent false
31390 9657 Acked Write 5 Done Done Sent false
31391 9658 Acked Read 10 Done Done Sent false
31392 9659 Acked Write 8 Done Done Sent false
31393 9660 Acked Write 2 Done Done Sent false
31394 9661 Acked Write 10 Done Done Sent false
31395 9662 Acked Read 3 Done Done Sent false
31396 9663 Acked Read 7 Done Done Sent false
31397 9664 Acked Read 10 Done Done Sent false
31398 9665 Acked Read 2 Done Done Sent false
31399 9666 Acked Read 1 Done Done Sent false
31400 9667 Acked Read 5 Done Done Sent false
31401 9668 Acked Read 4 Done Done Sent false
31402 9669 Acked Read 2 Done Done Sent false
31403 9670 Acked Read 5 Done Done Sent false
31404 9671 Acked Write 1 Done Done Sent false
31405 9672 Acked Write 6 Done Done Sent false
31406 9673 Acked Read 3 Done Done Sent false
31407 9674 Acked Write 2 Done Done Sent false
31408 9675 Acked Read 1 Done Done Sent false
31409 9676 Acked Read 10 Done Done Sent false
31410 9677 Acked Read 1 Done Done Sent false
31411 9678 Acked Read 2 Done Done Sent false
31412 9679 Acked Write 9 Done Done Sent false
31413 9680 Acked Write 4 Done Done Sent false
31414 9681 Acked Flush 0 Done Done Sent false
31415 9682 Acked Write 3 Done Done Sent false
31416 9683 Acked Read 7 Done Done Sent false
31417 9684 Acked Write 7 Done Done Sent false
31418 9685 Acked Read 2 Done Done Sent false
31419 9686 Acked Write 5 Done Done Sent false
31420 9687 Acked Read 8 Done Done Sent false
31421 9688 Acked Write 9 Done Done Sent false
31422 9689 Acked Read 1 Done Done Sent false
31423 9690 Acked Write 5 Done Done Sent false
31424 9691 Acked Write 4 Done Done Sent false
31425 9692 Acked Read 7 Done Done Sent false
31426 9693 Acked Read 6 Done Done Sent false
31427 9694 Acked Write 7 Done Done Sent false
31428 9695 Acked Write 8 Done Done Sent false
31429 9696 Acked Write 3 Done Done Sent false
31430 9697 Acked Read 8 Done Done Sent false
31431 9698 Acked Write 5 Done Done Sent false
31432 9699 Acked Read 5 Done Done Sent false
31433 9700 Acked Read 7 Done Done Sent false
31434 9701 Acked Read 2 Done Done Sent false
31435 9702 Acked Write 7 Done Done Sent false
31436 9703 Acked Read 5 Done Done Sent false
31437 9704 Acked Read 10 Done Done Sent false
31438 9705 Acked Flush 0 Done Done Sent false
31439 9706 Acked Write 10 Done Done Sent false
31440 9707 Acked Read 2 Done Done Sent false
31441 9708 Acked Write 3 Done Done Sent false
31442 9709 Acked Read 2 Done Done Sent false
31443 9710 Acked Write 3 Done Done Sent false
31444 9711 Acked Read 7 Done Done Sent false
31445 9712 Acked Read 4 Done Done Sent false
31446 9713 Acked Write 1 Done Done Sent false
31447 9714 Acked Read 1 Done Done Sent false
31448 9715 Acked Write 5 Done Done Sent false
31449 9716 Acked Read 3 Done Done Sent false
31450 9717 Acked Read 1 Done Done Sent false
31451 9718 Acked Write 6 Done Done Sent false
31452 9719 Acked Write 7 Done Done Sent false
31453 9720 Acked Read 6 Done Done Sent false
31454 9721 Acked Read 7 Done Done Sent false
31455 9722 Acked Write 3 Done Done Sent false
31456 9723 Acked Write 8 Done Done Sent false
31457 9724 Acked Read 8 Done Done Sent false
31458 9725 Acked Read 1 Done Done Sent false
31459 9726 Acked Read 5 Done Done Sent false
31460 9727 Acked Read 10 Done Done Sent false
31461 9728 Acked Read 8 Done Done Sent false
31462 9729 Acked Read 8 Done Done Sent false
31463 9730 Acked Write 1 Done Done Sent false
31464 9731 Acked Read 3 Done Done Sent false
31465 9732 Acked Write 7 Done Done Sent false
31466 9733 Acked Write 2 Done Done Sent false
31467 9734 Acked Flush 0 Done Done Sent false
31468 9735 Acked Write 8 Done Done Sent false
31469 9736 Acked Flush 0 Done Done Sent false
31470 9737 Acked Read 4 Done Done Sent false
31471 9738 Acked Read 2 Done Done Sent false
31472 9739 Acked Write 9 Done Done Sent false
31473 9740 Acked Write 7 Done Done Sent false
31474 9741 Acked Read 5 Done Done Sent false
31475 9742 Acked Write 2 Done Done Sent false
31476 9743 Acked Write 7 Done Done Sent false
31477 9744 Acked Read 7 Done Done Sent false
31478 9745 Acked Read 5 Done Done Sent false
31479 9746 Acked Read 4 Done Done Sent false
31480 9747 Acked Write 10 Done Done Sent false
31481 9748 Acked Write 10 Done Done Sent false
31482 9749 Acked Read 10 Done Done Sent false
31483 9750 Acked Read 3 Done Done Sent false
31484 9751 Acked Read 10 Done Done Sent false
31485 9752 Acked Write 10 Done Done Sent false
31486 9753 Acked Read 2 Done Done Sent false
31487 9754 Acked Flush 0 Done Done Sent false
31488 STATES DS:0 DS:1 DS:2 TOTAL
31489 Sent 0 0 302 302
31490 Done 302 302 0 604
31491 Skipped 0 0 0 0
31492 Error 0 0 0 0
31493 Last Flush: Some(JobId(9754)) Some(JobId(9754)) Some(JobId(9452))
31494 Downstairs last five completed: 9452 9451 9450 9449 9448
31495 Upstairs last five completed: 9754 9753 9752 9751 9750
31496 CLIENT: Up:0 ds:302 act:2
314972025-01-23T18:54:29.947ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
314982025-01-23T18:54:29.947ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:40867) = io task client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
314992025-01-23T18:54:29.948ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
315002025-01-23T18:54:29.949ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
315012025-01-23T18:54:29.949ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
315022025-01-23T18:54:29.949ZINFOcrucible: send last flush ID to this DS: Some(JobId(9452)) = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
315032025-01-23T18:54:29.949ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
315042025-01-23T18:54:29.949ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(9452)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
315052025-01-23T18:54:29.949ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(9452)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
31506 ----------------------------------------------------------------
31507 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
31508 Downstairs last five completed: 9754 9753 9752 9751 9750
31509 Upstairs last five completed: 9754 9753 9752 9751 9750
31510 CLIENT: Up:0 ds:0 act:3
31511 Replay: All jobs finished
31512 Main task has work to do, go find it
31513 got dsc Stop(0)
31514 stop 0
31515 [8810] Got stop action so:false kr:false
31516 Jan 23 18:54:32.194 INFO request completed, latency_us: 318, response_code: 204, uri: /stop/cid/0, method: POST, req_id: d75238c1-4aaf-4b4b-bccc-1eaf7bbf6497, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
315172025-01-23T18:54:32.194ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
31518 Jan 23 18:54:32.194 INFO request completed, latency_us: 141, response_code: 200{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:54:32.194771091Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"0","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
31519 , uri: /state/cid/0, method: GET, req_id: c8e72012-2db4-4b79-b613-466895bea6a1, remote_addr: 127.{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:54:32.194789348Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
31520 0.0.1:59295, local_addr: 127.0.0.1:9998
315212025-01-23T18:54:32.194ZINFOcrucible: client task is exiting = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
31522 [8810] Exited with: ExitStatus(unix_wait_status(9))
31523 [8810][0] reports Exit
31524 Jan 23 18:54:36.197 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9c2ce8d3-1c93-49f1-b598-4452b9c4d337, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
31525 001/300 Write block 3150 len 40960 data: 5 3 3 4 4 4 3 4 3 3
31526 002/300 Write block 3416 len 12288 data: 6 8 9
31527 003/300 Flush
31528 004/300 Read block 2857 len 4096
31529 005/300 Write block 956 len 16384 data: 8 9 11 11
31530 006/300 Read block 2541 len 24576
31531 007/300 Flush
31532 008/300 Write block 2168 len 24576 data: 4 5 5 5 5 5
31533 009/300 Write block 2721 len 28672 data: 4 3 4 4 6 7 6
31534 010/300 Write block 2535 len 32768 data: 9 8 7 6 7 10 8 7
31535 011/300 Read block 398 len 12288
31536 012/300 Flush
31537 013/300 Read block 1714 len 24576
31538 014/300 Read block 486 len 4096
31539 015/300 Read block 4731 len 8192
31540 016/300 Write block 2861 len 4096 data: 7
31541 017/300 Write block 255 len 36864 data: 6 4 4 5 7 6 6 6 6
31542 018/300 Read block 4901 len 20480
31543 019/300 Read block 1976 len 16384
31544 020/300 Write block 4132 len 4096 data: 5
31545 021/300 Write block 3821 len 20480 data: 7 6 7 6 6
31546 022/300 Read block 514 len 36864
31547 023/300 Write block 2788 len 32768 data: 7 6 6 5 6 7 7 8
31548 024/300 Write block 1065 len 36864 data: 9 9 9 8 13 12 10 9 12
31549 025/300 Write block 1157 len 8192 data: 4 6
31550 026/300 Read block 4262 len 12288
31551 027/300 Read block 4628 len 40960
31552 028/300 Read block 1162 len 4096
31553 029/300 Write block 3541 len 32768 data: 6 6 6 5 7 5 4 4
31554 030/300 Read block 605 len 8192
31555 031/300 Write block 1352 len 8192 data: 5 5
31556 032/300 Read block 3128 len 12288
31557 033/300 Read block 4271 len 12288
31558 034/300 Write block 740 len 8192 data: 6 7
31559 035/300 Write block 2910 len 12288 data: 8 6 7
31560 036/300 Read block 4562 len 32768
31561 037/300 Read block 3902 len 32768
31562 038/300 Read block 2395 len 40960
31563 039/300 Write block 3543 len 8192 data: 7 6
31564 040/300 Read block 3640 len 28672
31565 041/300 Write block 2625 len 8192 data: 10 10
31566 042/300 Write block 4729 len 28672 data: 5 4 4 5 7 7 6
31567 043/300 Read block 1462 len 12288
31568 044/300 Write block 4079 len 16384 data: 6 7 9 9
31569 045/300 Write block 911 len 28672 data: 5 4 5 5 6 6 5
31570 046/300 Read block 4732 len 16384
31571 047/300 Read block 4857 len 12288
31572 048/300 Flush
31573 049/300 Read block 2525 len 16384
31574 050/300 Read block 1153 len 24576
31575 051/300 Write block 1550 len 20480 data: 6 5 5 4 4
31576 052/300 Read block 4946 len 4096
31577 053/300 Write block 1417 len 4096 data: 7
31578 054/300 Read block 1018 len 4096
31579 055/300 Write block 3033 len 36864 data: 7 6 7 7 6 8 9 10 9
31580 056/300 Read block 2839 len 24576
31581 057/300 Read block 2072 len 24576
31582 058/300 Write block 3779 len 36864 data: 11 10 8 8 7 6 6 6 7
31583 059/300 Write block 2673 len 20480 data: 11 9 8 8 7
31584 060/300 Write block 3980 len 24576 data: 5 4 4 4 3 5
31585 061/300 Read block 4902 len 20480
31586 062/300 Read block 4823 len 24576
31587 063/300 Read block 4599 len 28672
31588 064/300 Read block 4058 len 20480
31589 065/300 Read block 171 len 12288
31590 066/300 Read block 2319 len 32768
31591 067/300 Read block 2153 len 4096
31592 068/300 Read block 1196 len 36864
31593 069/300 Write block 837 len 20480 data: 7 5 5 5 6
31594 070/300 Read block 2643 len 12288
31595 071/300 Read block 3208 len 16384
31596 072/300 Write block 1639 len 36864 data: 7 7 8 8 7 9 9 9 6
31597 073/300 Write block 2693 len 4096 data: 4
31598 074/300 Read block 344 len 4096
31599 075/300 Read block 995 len 8192
31600 076/300 Read block 649 len 4096
31601 077/300 Write block 3561 len 36864 data: 5 4 4 4 4 6 7 8 7
31602 078/300 Write block 179 len 16384 data: 9 7 6 6
31603 079/300 Read block 2767 len 36864
31604 080/300 Read block 595 len 32768
31605 081/300 Write block 4025 len 24576 data: 5 4 5 7 9 10
31606 082/300 Write block 3469 len 8192 data: 4 4
31607 083/300 Write block 1814 len 20480 data: 10 10 6 7 8
31608 084/300 Write block 1691 len 20480 data: 8 9 10 10 9
31609 085/300 Write block 1282 len 12288 data: 5 6 8
31610 086/300 Write block 4964 len 16384 data: 10 9 7 7
31611 087/300 Write block 4027 len 40960 data: 6 8 10 11 10 8 6 5 4 3
31612 088/300 Read block 625 len 40960
31613 089/300 Write block 3683 len 4096 data: 3
31614 090/300 Flush
31615 091/300 Flush
31616 092/300 Write block 1799 len 36864 data: 6 4 4 4 4 4 5 3 5
31617 093/300 Flush
31618 094/300 Read block 3459 len 32768
31619 095/300 Write block 4580 len 8192 data: 7 8
31620 096/300 Read block 959 len 32768
31621 097/300 Write block 351 len 16384 data: 6 5 4 5
31622 098/300 Read block 2286 len 20480
31623 099/300 Write block 1778 len 16384 data: 6 5 5 5
31624 100/300 Read block 774 len 32768
31625 101/300 Read block 4479 len 40960
31626 102/300 Write block 2450 len 20480 data: 10 9 7 6 6
31627 103/300 Write block 794 len 40960 data: 4 4 7 7 7 5 6 6 5 5
31628 104/300 Flush
31629 105/300 Read block 385 len 8192
31630 106/300 Read block 3264 len 8192
31631 107/300 Write block 523 len 28672 data: 4 5 6 8 9 10 9
31632 108/300 Read block 1244 len 28672
31633 109/300 Read block 1857 len 28672
31634 110/300 Write block 743 len 20480 data: 7 7 6 5 5
31635 111/300 Read block 3040 len 16384
31636 112/300 Write block 3827 len 12288 data: 7 7 8
31637 113/300 Write block 3893 len 36864 data: 6 8 7 7 9 9 9 9 8
31638 114/300 Write block 2377 len 20480 data: 3 4 5 5 5
31639 115/300 Write block 738 len 24576 data: 6 6 7 8 7 8
31640 116/300 Read block 3814 len 32768
31641 117/300 Write block 4088 len 12288 data: 6 6 8
31642 118/300 Write block 2204 len 12288 data: 6 7 8
31643 119/300 Flush
31644 120/300 Write block 1217 len 32768 data: 6 5 6 7 7 8 6 5
31645 121/300 Read block 2993 len 28672
31646 122/300 Write block 2406 len 4096 data: 8
31647 123/300 Read block 4682 len 36864
31648 124/300 Write block 1660 len 32768 data: 6 6 5 6 5 4 5 6
31649 125/300 Read block 3548 len 36864
31650 126/300 Write block 3667 len 12288 data: 7 7 6
31651 127/300 Write block 3174 len 32768 data: 5 4 4 4 4 3 4 3
31652 128/300 Write block 1627 len 8192 data: 5 5
31653 129/300 Flush
31654 130/300 Flush
31655 131/300 Read block 3012 len 32768
31656 132/300 Flush
31657 133/300 Write block 37 len 36864 data: 9 9 10 8 5 4 4 3 3
31658 134/300 Read block 3112 len 16384
31659 135/300 Write block 2754 len 8192 data: 4 6
31660 136/300 Read block 4451 len 28672
31661 137/300 Read block 2574 len 8192
31662 138/300 Write block 2412 len 36864 data: 3 3 3 5 6 5 6 5 4
31663 139/300 Read block 4490 len 20480
31664 140/300 Read block 2132 len 40960
31665 141/300 Read block 444 len 8192
31666 142/300 Read block 13 len 40960
31667 143/300 Read block 4247 len 20480
31668 144/300 Write block 2247 len 36864 data: 4 2 2 2 2 3 3 4 3
31669 145/300 Read block 3978 len 24576
31670 146/300 Write block 4732 len 36864 data: 6 8 8 7 5 5 4 3 3
31671 147/300 Read block 3706 len 36864
31672 148/300 Write block 4036 len 24576 data: 4 3 2 2 2 2
31673 149/300 Read block 2350 len 4096
31674 150/300 Flush
31675 151/300 Read block 2884 len 12288
31676 152/300 Read block 4978 len 16384
31677 153/300 Write block 3854 len 4096 data: 5
31678 154/300 Write block 2213 len 36864 data: 3 3 3 3 4 4 5 4 3
31679 155/300 Write block 2465 len 20480 data: 3 5 5 5 4
31680 156/300 Write block 257 len 28672 data: 5 6 8 7 7 7 7
31681 157/300 Flush
31682 158/300 Write block 4384 len 16384 data: 5 4 6 7
31683 159/300 Read block 1967 len 8192
31684 160/300 Read block 1407 len 36864
31685 161/300 Write block 4061 len 4096 data: 7
31686 162/300 Write block 2514 len 36864 data: 3 3 4 5 7 7 8 8 7
31687 163/300 Read block 4325 len 40960
31688 164/300 Write block 1622 len 16384 data: 7 7 5 6
31689 165/300 Read block 2987 len 36864
31690 166/300 Flush
31691 167/300 Read block 350 len 36864
31692 168/300 Read block 3456 len 36864
31693 169/300 Read block 3581 len 4096
31694 170/300 Read block 2968 len 32768
31695 171/300 Flush
31696 172/300 Read block 2084 len 32768
31697 173/300 Write block 4693 len 16384 data: 6 6 7 7
31698 174/300 Write block 3822 len 16384 data: 7 8 7 7
31699 175/300 Read block 1698 len 20480
31700 176/300 Read block 1365 len 20480
31701 177/300 Read block 238 len 16384
31702 178/300 Read block 3418 len 8192
31703 179/300 Read block 3602 len 20480
31704 180/300 Write block 571 len 28672 data: 2 4 3 3 5 4 6
31705 181/300 Read block 98 len 12288
31706 182/300 Read block 2986 len 4096
31707 183/300 Write block 4885 len 12288 data: 7 6 6
31708 184/300 Write block 3082 len 16384 data: 6 6 6 6
31709 185/300 Write block 4847 len 20480 data: 6 6 5 3 3
31710 186/300 Read block 4176 len 4096
31711 187/300 Read block 3483 len 12288
31712 188/300 Read block 285 len 40960
31713 189/300 Read block 1456 len 36864
31714 190/300 Write block 3510 len 20480 data: 9 8 8 8 5
31715 191/300 Write block 147 len 36864 data: 7 8 8 8 8 7 7 6 6
31716 192/300 Write block 4641 len 16384 data: 4 6 9 8
31717 193/300 Read block 146 len 24576
31718 194/300 Write block 569 len 12288 data: 5 4 3
31719 195/300 Read block 2277 len 8192
31720 196/300 Read block 3180 len 16384
31721 197/300 Flush
31722 198/300 Write block 959 len 4096 data: 12
31723 199/300 Write block 4645 len 16384 data: 8 9 9 7
31724 200/300 Flush
31725 201/300 Read block 2950 len 4096
31726 202/300 Read block 535 len 36864
31727 203/300 Read block 2531 len 28672
31728 204/300 Flush
31729 205/300 Flush
31730 206/300 Read block 655 len 24576
31731 207/300 Write block 829 len 12288 data: 7 6 6
31732 208/300 Read block 2177 len 28672
31733 209/300 Read block 2460 len 4096
31734 210/300 Write block 3637 len 28672 data: 4 6 7 8 5 4 3
31735 211/300 Read block 4243 len 20480
31736 212/300 Write block 85 len 16384 data: 6 6 6 6
31737 213/300 Write block 2145 len 4096 data: 3
31738 214/300 Read block 2374 len 12288
31739 215/300 Read block 3023 len 28672
31740 216/300 Read block 811 len 16384
31741 217/300 Write block 4264 len 4096 data: 4
31742 218/300 Write block 593 len 32768 data: 6 6 6 5 6 7 7 7
31743 219/300 Read block 3602 len 28672
31744 220/300 Flush
31745 221/300 Read block 3917 len 40960
31746 222/300 Write block 2066 len 28672 data: 3 4 4 7 7 10 8
31747 223/300 Read block 3715 len 28672
31748 224/300 Read block 1091 len 12288
31749 225/300 Write block 3970 len 8192 data: 6 5
31750 226/300 Flush
31751 227/300 Read block 2524 len 32768
31752 228/300 Read block 3422 len 24576
31753 229/300 Read block 30 len 40960
31754 230/300 Read block 1082 len 24576
31755 231/300 Flush
31756 232/300 Write block 3175 len 28672 data: 5 5 5 5 4 5 4
31757 233/300 Write block 2350 len 8192 data: 4 4
31758 234/300 Flush
31759 235/300 Read block 2866 len 40960
31760 236/300 Flush
31761 237/300 Write block 3077 len 36864 data: 6 5 7 6 6 7 7 7 7
31762 238/300 Write block 2512 len 8192 data: 3 3
31763 239/300 Read block 3722 len 40960
31764 240/300 Read block 706 len 32768
31765 241/300 Read block 1419 len 8192
31766 242/300 Read block 1861 len 40960
31767 243/300 Write block 3785 len 4096 data: 7
31768 244/300 Read block 3522 len 36864
31769 245/300 Write block 1530 len 16384 data: 5 5 5 7
31770 246/300 Read block 730 len 12288
31771 247/300 Write block 3099 len 28672 data: 7 8 7 7 9 8 9
31772 248/300 Read block 3833 len 40960
31773 249/300 Flush
31774 250/300 Read block 1838 len 4096
31775 251/300 Write block 1632 len 8192 data: 6 6
31776 252/300 Write block 4571 len 8192 data: 5 7
31777 253/300 Write block 1518 len 36864 data: 4 4 5 7 6 7 6 4 5
31778 254/300 Flush
31779 255/300 Write block 3581 len 20480 data: 2 2 2 4 4
31780 256/300 Read block 435 len 36864
31781 257/300 Flush
31782 258/300 Write block 1735 len 24576 data: 4 4 6 7 6 6
31783 259/300 Write block 1246 len 40960 data: 10 9 7 9 8 7 9 9 5 4
31784 260/300 Read block 1964 len 20480
31785 261/300 Write block 759 len 12288 data: 6 6 7
31786 262/300 Write block 1693 len 36864 data: 11 11 10 9 8 8 8 7 5
31787 263/300 Write block 1024 len 4096 data: 9
31788 264/300 Write block 2046 len 24576 data: 4 3 3 3 4 4
31789 265/300 Flush
31790 266/300 Write block 4208 len 28672 data: 7 6 4 5 5 3 4
31791 267/300 Read block 4842 len 32768
31792 268/300 Write block 3707 len 4096 data: 6
31793 269/300 Read block 4870 len 8192
31794 270/300 Write block 497 len 32768 data: 4 4 3 4 4 5 3 6
31795 271/300 Write block 689 len 16384 data: 4 4 4 5
31796 272/300 Read block 1772 len 8192
31797 273/300 Read block 1697 len 32768
31798 274/300 Read block 3096 len 8192
31799 275/300 Read block 332 len 8192
31800 276/300 Read block 3870 len 32768
31801 277/300 Write block 28 len 8192 data: 3 3
31802 278/300 Read block 491 len 28672
31803 279/300 Read block 1352 len 8192
31804 280/300 Flush
31805 281/300 Write block 4320 len 12288 data: 6 7 8
31806 282/300 Write block 628 len 12288 data: 5 6 5
31807 283/300 Read block 1685 len 12288
31808 284/300 Flush
31809 285/300 Write block 135 len 40960 data: 10 11 11 14 9 11 10 7 7 6
31810 286/300 Read block 2195 len 36864
31811 287/300 Read block 3163 len 40960
31812 288/300 Read block 2904 len 32768
31813 289/300 Read block 4590 len 12288
31814 290/300 Write block 511 len 20480 data: 4 3 3 5 5
31815 291/300 Read block 1759 len 12288
31816 292/300 Write block 885 len 28672 data: 7 7 9 7 6 6 6
31817 293/300 Read block 530 len 8192
31818 294/300 Read block 1351 len 28672
31819 295/300 Read block 879 len 28672
31820 296/300 Flush
31821 297/300 Write block 4890 len 24576 data: 6 6 6 6 5 5
31822 298/300 Write block 125 len 12288 data: 7 7 6
31823 299/300 Write block 978 len 24576 data: 5 5 5 6 6 6
31824 300/300 Flush
31825 301/300 Flush
31826 Main task has work to do, go find it
31827 got dsc Start(0)
31828 start 0
31829 Jan 23 18:54:36.431 INFO request completed, latency_us: 205, response_code: 204, uri: /start/cid/0, method: POST, req_id: 02ff2f7e-a244-46ef-92ec-c5660276cae1, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
31830 Replay: started 0, returned:Ok(())
31831 ----------------------------------------------------------------
31832 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31833 States: Offline Active Active
31834 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31835 9755 Acked Write 10 Sent Done Done false
31836 9756 Acked Write 3 Sent Done Done false
31837 9757 Acked Flush 0 Sent Done Done false
31838 9758 Acked Read 1 Sent Done Done false
31839 9759 Acked Write 4 Sent Done Done false
31840 9760 Acked Read 6 Sent Done Done false
31841 9761 Acked Flush 0 Sent Done Done false
31842 9762 Acked Write 6 Sent Done Done false
31843 9763 Acked Write 7 Sent Done Done false
31844 9764 Acked Write 8 Sent Done Done false
31845 9765 Acked Read 3 Sent Done Done false
31846 9766 Acked Flush 0 Sent Done Done false
31847 9767 Acked Read 6 Sent Done Done false
31848 9768 Acked Read 1 Sent Done Done false
31849 9769 Acked Read 2 Sent Done Done false
31850 9770 Acked Write 1 Sent Done Done false
31851 9771 Acked Write 9 Sent Done Done false
31852 9772 Acked Read 5 Sent Done Done false
31853 9773 Acked Read 4 Sent Done Done false
31854 9774 Acked Write 1 Sent Done Done false
31855 9775 Acked Write 5 Sent Done Done false
31856 9776 Acked Read 9 Sent Done Done false
31857 9777 Acked Write 8 Sent Done Done false
31858 9778 Acked Write 9 Sent Done Done false
31859 9779 Acked Write 2 Sent Done Done false
31860 9780 Acked Read 3 Sent Done Done false
31861 9781 Acked Read 10 Sent Done Done false
31862 9782 Acked Read 1 Sent Done Done false
31863 9783 Acked Write 8 Sent Done Done false
31864 9784 Acked Read 2 Sent Done Done false
31865 9785 Acked Write 2 Sent Done Done false
31866 9786 Acked Read 3 Sent Done Done false
31867 9787 Acked Read 3 Sent Done Done false
31868 9788 Acked Write 2 Sent Done Done false
31869 9789 Acked Write 3 Sent Done Done false
31870 9790 Acked Read 8 Sent Done Done false
31871 9791 Acked Read 8 Sent Done Done false
31872 9792 Acked Read 10 Sent Done Done false
31873 9793 Acked Write 2 Sent Done Done false
31874 9794 Acked Read 7 Sent Done Done false
31875 9795 Acked Write 2 Sent Done Done false
31876 9796 Acked Write 7 Sent Done Done false
31877 9797 Acked Read 3 Sent Done Done false
31878 9798 Acked Write 4 Sent Done Done false
31879 9799 Acked Write 7 Sent Done Done false
31880 9800 Acked Read 4 Sent Done Done false
31881 9801 Acked Read 3 Sent Done Done false
31882 9802 Acked Flush 0 Sent Done Done false
31883 9803 Acked Read 4 Sent Done Done false
31884 9804 Acked Read 6 Sent Done Done false
31885 9805 Acked Write 5 Sent Done Done false
31886 9806 Acked Read 1 Sent Done Done false
31887 9807 Acked Write 1 Sent Done Done false
31888 9808 Acked Read 1 Sent Done Done false
31889 9809 Acked Write 9 Sent Done Done false
31890 9810 Acked Read 6 Sent Done Done false
31891 9811 Acked Read 6 Sent Done Done false
31892 9812 Acked Write 9 Sent Done Done false
31893 9813 Acked Write 5 Sent Done Done false
31894 9814 Acked Write 6 Sent Done Done false
31895 9815 Acked Read 5 Sent Done Done false
31896 9816 Acked Read 6 Sent Done Done false
31897 9817 Acked Read 7 Sent Done Done false
31898 9818 Acked Read 5 Sent Done Done false
31899 9819 Acked Read 3 Sent Done Done false
31900 9820 Acked Read 8 Sent Done Done false
31901 9821 Acked Read 1 Sent Done Done false
31902 9822 Acked Read 9 Sent Done Done false
31903 9823 Acked Write 5 Sent Done Done false
31904 9824 Acked Read 3 Sent Done Done false
31905 9825 Acked Read 4 Sent Done Done false
31906 9826 Acked Write 9 Sent Done Done false
31907 9827 Acked Write 1 Sent Done Done false
31908 9828 Acked Read 1 Sent Done Done false
31909 9829 Acked Read 2 Sent Done Done false
31910 9830 Acked Read 1 Sent Done Done false
31911 9831 Acked Write 9 Sent Done Done false
31912 9832 Acked Write 4 Sent Done Done false
31913 9833 Acked Read 9 Sent Done Done false
31914 9834 Acked Read 8 Sent Done Done false
31915 9835 Acked Write 6 Sent Done Done false
31916 9836 Acked Write 2 Sent Done Done false
31917 9837 Acked Write 5 Sent Done Done false
31918 9838 Acked Write 5 Sent Done Done false
31919 9839 Acked Write 3 Sent Done Done false
31920 9840 Acked Write 4 Sent Done Done false
31921 9841 Acked Write 10 Sent Done Done false
31922 9842 Acked Read 10 Sent Done Done false
31923 9843 Acked Write 1 Sent Done Done false
31924 9844 Acked Flush 0 Sent Done Done false
31925 9845 Acked Flush 0 Sent Done Done false
31926 9846 Acked Write 9 Sent Done Done false
31927 9847 Acked Flush 0 Sent Done Done false
31928 9848 Acked Read 8 Sent Done Done false
31929 9849 Acked Write 2 Sent Done Done false
31930 9850 Acked Read 8 Sent Done Done false
31931 9851 Acked Write 4 Sent Done Done false
31932 9852 Acked Read 5 Sent Done Done false
31933 9853 Acked Write 4 Sent Done Done false
31934 9854 Acked Read 8 Sent Done Done false
31935 9855 Acked Read 10 Sent Done Done false
31936 9856 Acked Write 5 Sent Done Done false
31937 9857 Acked Write 10 Sent Done Done false
31938 9858 Acked Flush 0 Sent Done Done false
31939 9859 Acked Read 2 Sent Done Done false
31940 9860 Acked Read 2 Sent Done Done false
31941 9861 Acked Write 7 Sent Done Done false
31942 9862 Acked Read 7 Sent Done Done false
31943 9863 Acked Read 7 Sent Done Done false
31944 9864 Acked Write 5 Sent Done Done false
31945 9865 Acked Read 4 Sent Done Done false
31946 9866 Acked Write 3 Sent Done Done false
31947 9867 Acked Write 9 Sent Done Done false
31948 9868 Acked Write 5 Sent Done Done false
31949 9869 Acked Write 6 Sent Done Done false
31950 9870 Acked Read 8 Sent Done Done false
31951 9871 Acked Write 3 Sent Done Done false
31952 9872 Acked Write 3 Sent Done Done false
31953 9873 Acked Flush 0 Sent Done Done false
31954 9874 Acked Write 8 Sent Done Done false
31955 9875 Acked Read 7 Sent Done Done false
31956 9876 Acked Write 1 Sent Done Done false
31957 9877 Acked Read 9 Sent Done Done false
31958 9878 Acked Write 8 Sent Done Done false
31959 9879 Acked Read 9 Sent Done Done false
31960 9880 Acked Write 3 Sent Done Done false
31961 9881 Acked Write 8 Sent Done Done false
31962 9882 Acked Write 2 Sent Done Done false
31963 9883 Acked Flush 0 Sent Done Done false
31964 9884 Acked Flush 0 Sent Done Done false
31965 9885 Acked Read 8 Sent Done Done false
31966 9886 Acked Flush 0 Sent Done Done false
31967 9887 Acked Write 9 Sent Done Done false
31968 9888 Acked Read 4 Sent Done Done false
31969 9889 Acked Write 2 Sent Done Done false
31970 9890 Acked Read 7 Sent Done Done false
31971 9891 Acked Read 2 Sent Done Done false
31972 9892 Acked Write 9 Sent Done Done false
31973 9893 Acked Read 5 Sent Done Done false
31974 9894 Acked Read 10 Sent Done Done false
31975 9895 Acked Read 2 Sent Done Done false
31976 9896 Acked Read 10 Sent Done Done false
31977 9897 Acked Read 5 Sent Done Done false
31978 9898 Acked Write 9 Sent Done Done false
31979 9899 Acked Read 6 Sent Done Done false
31980 9900 Acked Write 9 Sent Done Done false
31981 9901 Acked Read 9 Sent Done Done false
31982 9902 Acked Write 6 Sent Done Done false
31983 9903 Acked Read 1 Sent Done Done false
31984 9904 Acked Flush 0 Sent Done Done false
31985 9905 Acked Read 3 Sent Done Done false
31986 9906 Acked Read 4 Sent Done Done false
31987 9907 Acked Write 1 Sent Done Done false
31988 9908 Acked Write 9 Sent Done Done false
31989 9909 Acked Write 5 Sent Done Done false
31990 9910 Acked Write 7 Sent Done Done false
31991 9911 Acked Flush 0 Sent Done Done false
31992 9912 Acked Write 4 Sent Done Done false
31993 9913 Acked Read 2 Sent Done Done false
31994 9914 Acked Read 9 Sent Done Done false
31995 9915 Acked Write 1 Sent Done Done false
31996 9916 Acked Write 9 Sent Done Done false
31997 9917 Acked Read 10 Sent Done Done false
31998 9918 Acked Write 4 Sent Done Done false
31999 9919 Acked Read 9 Sent Done Done false
32000 9920 Acked Flush 0 Sent Done Done false
32001 9921 Acked Read 9 Sent Done Done false
32002 9922 Acked Read 9 Sent Done Done false
32003 9923 Acked Read 1 Sent Done Done false
32004 9924 Acked Read 8 Sent Done Done false
32005 9925 Acked Flush 0 Sent Done Done false
32006 9926 Acked Read 8 Sent Done Done false
32007 9927 Acked Write 4 Sent Done Done false
32008 9928 Acked Write 4 Sent Done Done false
32009 9929 Acked Read 5 Sent Done Done false
32010 9930 Acked Read 5 Sent Done Done false
32011 9931 Acked Read 4 Sent Done Done false
32012 9932 Acked Read 2 Sent Done Done false
32013 9933 Acked Read 5 Sent Done Done false
32014 9934 Acked Write 7 Sent Done Done false
32015 9935 Acked Read 3 Sent Done Done false
32016 9936 Acked Read 1 Sent Done Done false
32017 9937 Acked Write 3 Sent Done Done false
32018 9938 Acked Write 4 Sent Done Done false
32019 9939 Acked Write 5 Sent Done Done false
32020 9940 Acked Read 1 Sent Done Done false
32021 9941 Acked Read 3 Sent Done Done false
32022 9942 Acked Read 10 Sent Done Done false
32023 9943 Acked Read 9 Sent Done Done false
32024 9944 Acked Write 5 Sent Done Done false
32025 9945 Acked Write 9 Sent Done Done false
32026 9946 Acked Write 4 Sent Done Done false
32027 9947 Acked Read 6 Sent Done Done false
32028 9948 Acked Write 3 Sent Done Done false
32029 9949 Acked Read 2 Sent Done Done false
32030 9950 Acked Read 4 Sent Done Done false
32031 9951 Acked Flush 0 Sent Done Done false
32032 9952 Acked Write 1 Sent Done Done false
32033 9953 Acked Write 4 Sent Done Done false
32034 9954 Acked Flush 0 Sent Done Done false
32035 9955 Acked Read 1 Sent Done Done false
32036 9956 Acked Read 9 Sent Done Done false
32037 9957 Acked Read 7 Sent Done Done false
32038 9958 Acked Flush 0 Sent Done Done false
32039 9959 Acked Flush 0 Sent Done Done false
32040 9960 Acked Read 6 Sent Done Done false
32041 9961 Acked Write 3 Sent Done Done false
32042 9962 Acked Read 7 Sent Done Done false
32043 9963 Acked Read 1 Sent Done Done false
32044 9964 Acked Write 7 Sent Done Done false
32045 9965 Acked Read 5 Sent Done Done false
32046 9966 Acked Write 4 Sent Done Done false
32047 9967 Acked Write 1 Sent Done Done false
32048 9968 Acked Read 3 Sent Done Done false
32049 9969 Acked Read 7 Sent Done Done false
32050 9970 Acked Read 4 Sent Done Done false
32051 9971 Acked Write 1 Sent Done Done false
32052 9972 Acked Write 8 Sent Done Done false
32053 9973 Acked Read 7 Sent Done Done false
32054 9974 Acked Flush 0 Sent Done Done false
32055 9975 Acked Read 10 Sent Done Done false
32056 9976 Acked Write 7 Sent Done Done false
32057 9977 Acked Read 7 Sent Done Done false
32058 9978 Acked Read 3 Sent Done Done false
32059 9979 Acked Write 2 Sent Done Done false
32060 9980 Acked Flush 0 Sent Done Done false
32061 9981 Acked Read 8 Sent Done Done false
32062 9982 Acked Read 6 Sent Done Done false
32063 9983 Acked Read 10 Sent Done Done false
32064 9984 Acked Read 6 Sent Done Done false
32065 9985 Acked Flush 0 Sent Done Done false
32066 9986 Acked Write 7 Sent Done Done false
32067 9987 Acked Write 2 Sent Done Done false
32068 9988 Acked Flush 0 Sent Done Done false
32069 9989 Acked Read 10 Sent Done Done false
32070 9990 Acked Flush 0 Sent Done Done false
32071 9991 Acked Write 9 Sent Done Done false
32072 9992 Acked Write 2 Sent Done Done false
32073 9993 Acked Read 10 Sent Done Done false
32074 9994 Acked Read 8 Sent Done Done false
32075 9995 Acked Read 2 Sent Done Done false
32076 9996 Acked Read 10 Sent Done Done false
32077 9997 Acked Write 1 Sent Done Done false
32078 9998 Acked Read 9 Sent Done Done false
32079 9999 Acked Write 4 Sent Done Done false
32080 10000 Acked Read 3 Sent Done Done false
32081 10001 Acked Write 7 Sent Done Done false
32082 10002 Acked Read 10 Sent Done Done false
32083 10003 Acked Flush 0 Sent Done Done false
32084 10004 Acked Read 1 Sent Done Done false
32085 10005 Acked Write 2 Sent Done Done false
32086 10006 Acked Write 2 Sent Done Done false
32087 10007 Acked Write 9 Sent Done Done false
32088 10008 Acked Flush 0 Sent Done Done false
32089 10009 Acked Write 5 Sent Done Done false
32090 10010 Acked Read 9 Sent Done Done false
32091 10011 Acked Flush 0 Sent Done Done false
32092 10012 Acked Write 6 Sent Done Done false
32093 10013 Acked Write 10 Sent Done Done false
32094 10014 Acked Read 5 Sent Done Done false
32095 10015 Acked Write 3 Sent Done Done false
32096 10016 Acked Write 9 Sent Done Done false
32097 10017 Acked Write 1 Sent Done Done false
32098 10018 Acked Write 6 Sent Done Done false
32099 10019 Acked Flush 0 Sent Done Done false
32100 10020 Acked Write 7 Sent Done Done false
32101 10021 Acked Read 8 Sent Done Done false
32102 10022 Acked Write 1 Sent Done Done false
32103 10023 Acked Read 2 Sent Done Done false
32104 10024 Acked Write 8 Sent Done Done false
32105 10025 Acked Write 4 Sent Done Done false
32106 10026 Acked Read 2 Sent Done Done false
32107 10027 Acked Read 8 Sent Done Done false
32108 10028 Acked Read 2 Sent Done Done false
32109 10029 Acked Read 2 Sent Done Done false
32110 10030 Acked Read 8 Sent Done Done false
32111 10031 Acked Write 2 Sent Done Done false
32112 10032 Acked Read 7 Sent Done Done false
32113 10033 Acked Read 2 Sent Done Done false
32114 10034 Acked Flush 0 Sent Done Done false
32115 10035 Acked Write 3 Sent Done Done false
32116 10036 Acked Write 3 Sent Done Done false
32117 10037 Acked Read 3 Sent Done Done false
32118 10038 Acked Flush 0 Sent Done Done false
32119 10039 Acked Write 10 Sent Done Done false
32120 10040 Acked Read 9 Sent Done Done false
32121 10041 Acked Read 10 Sent Done Done false
32122 10042 Acked Read 8 Sent Done Done false
32123 10043 Acked Read 3 Sent Done Done false
32124 10044 Acked Write 5 Sent Done Done false
32125 10045 Acked Read 3 Sent Done Done false
32126 10046 Acked Write 7 Sent Done Done false
32127 10047 Acked Read 2 Sent Done Done false
32128 10048 Acked Read 7 Sent Done Done false
32129 10049 Acked Read 7 Sent Done Done false
32130 10050 Acked Flush 0 Sent Done Done false
32131 10051 Acked Write 6 Sent Done Done false
32132 10052 Acked Write 3 Sent Done Done false
32133 10053 Acked Write 6 Sent Done Done false
32134 10054 Acked Flush 0 Sent Done Done false
32135 10055 Acked Flush 0 Sent Done Done false
32136 STATES DS:0 DS:1 DS:2 TOTAL
32137 Sent 301 0 0 301
32138 Done 0 301 301 602
32139 Skipped 0 0 0 0
32140 Error 0 0 0 0
32141 Last Flush: Some(JobId(9754)) Some(JobId(10055)) Some(JobId(10055))
32142 Downstairs last five completed: 9754 9753 9752 9751 9750
32143 Upstairs last five completed: 10055 10054 10053 10052 10051
32144 CLIENT: Up:0 ds:301 act:2
32145 [8810] got start action
32146 [8810] I am going to restart
32147 Starting downstairs at port 8810
32148 Make output file at "/tmp/dsc/downstairs-8810.txt"
32149 [8810][0] reports Starting
32150 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1230)
32151 [8810][0] reports Running
32152 ----------------------------------------------------------------
32153 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32154 States: Offline Active Active
32155 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32156 9755 Acked Write 10 Sent Done Done false
32157 9756 Acked Write 3 Sent Done Done false
32158 9757 Acked Flush 0 Sent Done Done false
32159 9758 Acked Read 1 Sent Done Done false
32160 9759 Acked Write 4 Sent Done Done false
32161 9760 Acked Read 6 Sent Done Done false
32162 9761 Acked Flush 0 Sent Done Done false
32163 9762 Acked Write 6 Sent Done Done false
32164 9763 Acked Write 7 Sent Done Done false
32165 9764 Acked Write 8 Sent Done Done false
32166 9765 Acked Read 3 Sent Done Done false
32167 9766 Acked Flush 0 Sent Done Done false
32168 9767 Acked Read 6 Sent Done Done false
32169 9768 Acked Read 1 Sent Done Done false
32170 9769 Acked Read 2 Sent Done Done false
32171 9770 Acked Write 1 Sent Done Done false
32172 9771 Acked Write 9 Sent Done Done false
32173 9772 Acked Read 5 Sent Done Done false
32174 9773 Acked Read 4 Sent Done Done false
32175 9774 Acked Write 1 Sent Done Done false
32176 9775 Acked Write 5 Sent Done Done false
32177 9776 Acked Read 9 Sent Done Done false
32178 9777 Acked Write 8 Sent Done Done false
32179 9778 Acked Write 9 Sent Done Done false
32180 9779 Acked Write 2 Sent Done Done false
32181 9780 Acked Read 3 Sent Done Done false
32182 9781 Acked Read 10 Sent Done Done false
32183 9782 Acked Read 1 Sent Done Done false
32184 9783 Acked Write 8 Sent Done Done false
32185 9784 Acked Read 2 Sent Done Done false
32186 9785 Acked Write 2 Sent Done Done false
32187 9786 Acked Read 3 Sent Done Done false
32188 9787 Acked Read 3 Sent Done Done false
32189 9788 Acked Write 2 Sent Done Done false
32190 9789 Acked Write 3 Sent Done Done false
32191 9790 Acked Read 8 Sent Done Done false
32192 9791 Acked Read 8 Sent Done Done false
32193 9792 Acked Read 10 Sent Done Done false
32194 9793 Acked Write 2 Sent Done Done false
32195 9794 Acked Read 7 Sent Done Done false
32196 9795 Acked Write 2 Sent Done Done false
32197 9796 Acked Write 7 Sent Done Done false
32198 9797 Acked Read 3 Sent Done Done false
32199 9798 Acked Write 4 Sent Done Done false
32200 9799 Acked Write 7 Sent Done Done false
32201 9800 Acked Read 4 Sent Done Done false
32202 9801 Acked Read 3 Sent Done Done false
32203 9802 Acked Flush 0 Sent Done Done false
32204 9803 Acked Read 4 Sent Done Done false
32205 9804 Acked Read 6 Sent Done Done false
32206 9805 Acked Write 5 Sent Done Done false
32207 9806 Acked Read 1 Sent Done Done false
32208 9807 Acked Write 1 Sent Done Done false
32209 9808 Acked Read 1 Sent Done Done false
32210 9809 Acked Write 9 Sent Done Done false
32211 9810 Acked Read 6 Sent Done Done false
32212 9811 Acked Read 6 Sent Done Done false
32213 9812 Acked Write 9 Sent Done Done false
32214 9813 Acked Write 5 Sent Done Done false
32215 9814 Acked Write 6 Sent Done Done false
32216 9815 Acked Read 5 Sent Done Done false
32217 9816 Acked Read 6 Sent Done Done false
32218 9817 Acked Read 7 Sent Done Done false
32219 9818 Acked Read 5 Sent Done Done false
32220 9819 Acked Read 3 Sent Done Done false
32221 9820 Acked Read 8 Sent Done Done false
32222 9821 Acked Read 1 Sent Done Done false
32223 9822 Acked Read 9 Sent Done Done false
32224 9823 Acked Write 5 Sent Done Done false
32225 9824 Acked Read 3 Sent Done Done false
32226 9825 Acked Read 4 Sent Done Done false
32227 9826 Acked Write 9 Sent Done Done false
32228 9827 Acked Write 1 Sent Done Done false
32229 9828 Acked Read 1 Sent Done Done false
32230 9829 Acked Read 2 Sent Done Done false
32231 9830 Acked Read 1 Sent Done Done false
32232 9831 Acked Write 9 Sent Done Done false
32233 9832 Acked Write 4 Sent Done Done false
32234 9833 Acked Read 9 Sent Done Done false
32235 9834 Acked Read 8 Sent Done Done false
32236 9835 Acked Write 6 Sent Done Done false
32237 9836 Acked Write 2 Sent Done Done false
32238 9837 Acked Write 5 Sent Done Done false
32239 9838 Acked Write 5 Sent Done Done false
32240 9839 Acked Write 3 Sent Done Done false
32241 9840 Acked Write 4 Sent Done Done false
32242 9841 Acked Write 10 Sent Done Done false
32243 9842 Acked Read 10 Sent Done Done false
32244 9843 Acked Write 1 Sent Done Done false
32245 9844 Acked Flush 0 Sent Done Done false
32246 9845 Acked Flush 0 Sent Done Done false
32247 9846 Acked Write 9 Sent Done Done false
32248 9847 Acked Flush 0 Sent Done Done false
32249 9848 Acked Read 8 Sent Done Done false
32250 9849 Acked Write 2 Sent Done Done false
32251 9850 Acked Read 8 Sent Done Done false
32252 9851 Acked Write 4 Sent Done Done false
32253 9852 Acked Read 5 Sent Done Done false
32254 9853 Acked Write 4 Sent Done Done false
32255 9854 Acked Read 8 Sent Done Done false
32256 9855 Acked Read 10 Sent Done Done false
32257 9856 Acked Write 5 Sent Done Done false
32258 9857 Acked Write 10 Sent Done Done false
32259 9858 Acked Flush 0 Sent Done Done false
32260 9859 Acked Read 2 Sent Done Done false
32261 9860 Acked Read 2 Sent Done Done false
32262 9861 Acked Write 7 Sent Done Done false
32263 9862 Acked Read 7 Sent Done Done false
32264 9863 Acked Read 7 Sent Done Done false
32265 9864 Acked Write 5 Sent Done Done false
32266 9865 Acked Read 4 Sent Done Done false
32267 9866 Acked Write 3 Sent Done Done false
32268 9867 Acked Write 9 Sent Done Done false
32269 9868 Acked Write 5 Sent Done Done false
32270 9869 Acked Write 6 Sent Done Done false
32271 9870 Acked Read 8 Sent Done Done false
32272 9871 Acked Write 3 Sent Done Done false
32273 9872 Acked Write 3 Sent Done Done false
32274 9873 Acked Flush 0 Sent Done Done false
32275 9874 Acked Write 8 Sent Done Done false
32276 9875 Acked Read 7 Sent Done Done false
32277 9876 Acked Write 1 Sent Done Done false
32278 9877 Acked Read 9 Sent Done Done false
32279 9878 Acked Write 8 Sent Done Done false
32280 9879 Acked Read 9 Sent Done Done false
32281 9880 Acked Write 3 Sent Done Done false
32282 9881 Acked Write 8 Sent Done Done false
32283 9882 Acked Write 2 Sent Done Done false
32284 9883 Acked Flush 0 Sent Done Done false
32285 9884 Acked Flush 0 Sent Done Done false
32286 9885 Acked Read 8 Sent Done Done false
32287 9886 Acked Flush 0 Sent Done Done false
32288 9887 Acked Write 9 Sent Done Done false
32289 9888 Acked Read 4 Sent Done Done false
32290 9889 Acked Write 2 Sent Done Done false
32291 9890 Acked Read 7 Sent Done Done false
32292 9891 Acked Read 2 Sent Done Done false
32293 9892 Acked Write 9 Sent Done Done false
32294 9893 Acked Read 5 Sent Done Done false
32295 9894 Acked Read 10 Sent Done Done false
32296 9895 Acked Read 2 Sent Done Done false
32297 9896 Acked Read 10 Sent Done Done false
32298 9897 Acked Read 5 Sent Done Done false
32299 9898 Acked Write 9 Sent Done Done false
32300 9899 Acked Read 6 Sent Done Done false
32301 9900 Acked Write 9 Sent Done Done false
32302 9901 Acked Read 9 Sent Done Done false
32303 9902 Acked Write 6 Sent Done Done false
32304 9903 Acked Read 1 Sent Done Done false
32305 9904 Acked Flush 0 Sent Done Done false
32306 9905 Acked Read 3 Sent Done Done false
32307 9906 Acked Read 4 Sent Done Done false
32308 9907 Acked Write 1 Sent Done Done false
32309 9908 Acked Write 9 Sent Done Done false
32310 9909 Acked Write 5 Sent Done Done false
32311 9910 Acked Write 7 Sent Done Done false
32312 9911 Acked Flush 0 Sent Done Done false
32313 9912 Acked Write 4 Sent Done Done false
32314 9913 Acked Read 2 Sent Done Done false
32315 9914 Acked Read 9 Sent Done Done false
32316 9915 Acked Write 1 Sent Done Done false
32317 9916 Acked Write 9 Sent Done Done false
32318 9917 Acked Read 10 Sent Done Done false
32319 9918 Acked Write 4 Sent Done Done false
32320 9919 Acked Read 9 Sent Done Done false
32321 9920 Acked Flush 0 Sent Done Done false
32322 9921 Acked Read 9 Sent Done Done false
32323 9922 Acked Read 9 Sent Done Done false
32324 9923 Acked Read 1 Sent Done Done false
32325 9924 Acked Read 8 Sent Done Done false
32326 9925 Acked Flush 0 Sent Done Done false
32327 9926 Acked Read 8 Sent Done Done false
32328 9927 Acked Write 4 Sent Done Done false
32329 9928 Acked Write 4 Sent Done Done false
32330 9929 Acked Read 5 Sent Done Done false
32331 9930 Acked Read 5 Sent Done Done false
32332 9931 Acked Read 4 Sent Done Done false
32333 9932 Acked Read 2 Sent Done Done false
32334 9933 Acked Read 5 Sent Done Done false
32335 9934 Acked Write 7 Sent Done Done false
32336 9935 Acked Read 3 Sent Done Done false
32337 9936 Acked Read 1 Sent Done Done false
32338 9937 Acked Write 3 Sent Done Done false
32339 9938 Acked Write 4 Sent Done Done false
32340 9939 Acked Write 5 Sent Done Done false
32341 9940 Acked Read 1 Sent Done Done false
32342 9941 Acked Read 3 Sent Done Done false
32343 9942 Acked Read 10 Sent Done Done false
32344 9943 Acked Read 9 Sent Done Done false
32345 9944 Acked Write 5 Sent Done Done false
32346 9945 Acked Write 9 Sent Done Done false
32347 9946 Acked Write 4 Sent Done Done false
32348 9947 Acked Read 6 Sent Done Done false
32349 9948 Acked Write 3 Sent Done Done false
32350 9949 Acked Read 2 Sent Done Done false
32351 9950 Acked Read 4 Sent Done Done false
32352 9951 Acked Flush 0 Sent Done Done false
32353 9952 Acked Write 1 Sent Done Done false
32354 9953 Acked Write 4 Sent Done Done false
32355 9954 Acked Flush 0 Sent Done Done false
32356 9955 Acked Read 1 Sent Done Done false
32357 9956 Acked Read 9 Sent Done Done false
32358 9957 Acked Read 7 Sent Done Done false
32359 9958 Acked Flush 0 Sent Done Done false
32360 9959 Acked Flush 0 Sent Done Done false
32361 9960 Acked Read 6 Sent Done Done false
32362 9961 Acked Write 3 Sent Done Done false
32363 9962 Acked Read 7 Sent Done Done false
32364 9963 Acked Read 1 Sent Done Done false
32365 9964 Acked Write 7 Sent Done Done false
32366 9965 Acked Read 5 Sent Done Done false
32367 9966 Acked Write 4 Sent Done Done false
32368 9967 Acked Write 1 Sent Done Done false
32369 9968 Acked Read 3 Sent Done Done false
32370 9969 Acked Read 7 Sent Done Done false
32371 9970 Acked Read 4 Sent Done Done false
32372 9971 Acked Write 1 Sent Done Done false
32373 9972 Acked Write 8 Sent Done Done false
32374 9973 Acked Read 7 Sent Done Done false
32375 9974 Acked Flush 0 Sent Done Done false
32376 9975 Acked Read 10 Sent Done Done false
32377 9976 Acked Write 7 Sent Done Done false
32378 9977 Acked Read 7 Sent Done Done false
32379 9978 Acked Read 3 Sent Done Done false
32380 9979 Acked Write 2 Sent Done Done false
32381 9980 Acked Flush 0 Sent Done Done false
32382 9981 Acked Read 8 Sent Done Done false
32383 9982 Acked Read 6 Sent Done Done false
32384 9983 Acked Read 10 Sent Done Done false
32385 9984 Acked Read 6 Sent Done Done false
32386 9985 Acked Flush 0 Sent Done Done false
32387 9986 Acked Write 7 Sent Done Done false
32388 9987 Acked Write 2 Sent Done Done false
32389 9988 Acked Flush 0 Sent Done Done false
32390 9989 Acked Read 10 Sent Done Done false
32391 9990 Acked Flush 0 Sent Done Done false
32392 9991 Acked Write 9 Sent Done Done false
32393 9992 Acked Write 2 Sent Done Done false
32394 9993 Acked Read 10 Sent Done Done false
32395 9994 Acked Read 8 Sent Done Done false
32396 9995 Acked Read 2 Sent Done Done false
32397 9996 Acked Read 10 Sent Done Done false
32398 9997 Acked Write 1 Sent Done Done false
32399 9998 Acked Read 9 Sent Done Done false
32400 9999 Acked Write 4 Sent Done Done false
32401 10000 Acked Read 3 Sent Done Done false
32402 10001 Acked Write 7 Sent Done Done false
32403 10002 Acked Read 10 Sent Done Done false
32404 10003 Acked Flush 0 Sent Done Done false
32405 10004 Acked Read 1 Sent Done Done false
32406 10005 Acked Write 2 Sent Done Done false
32407 10006 Acked Write 2 Sent Done Done false
32408 10007 Acked Write 9 Sent Done Done false
32409 10008 Acked Flush 0 Sent Done Done false
32410 10009 Acked Write 5 Sent Done Done false
32411 10010 Acked Read 9 Sent Done Done false
32412 10011 Acked Flush 0 Sent Done Done false
32413 10012 Acked Write 6 Sent Done Done false
32414 10013 Acked Write 10 Sent Done Done false
32415 10014 Acked Read 5 Sent Done Done false
32416 10015 Acked Write 3 Sent Done Done false
32417 10016 Acked Write 9 Sent Done Done false
32418 10017 Acked Write 1 Sent Done Done false
32419 10018 Acked Write 6 Sent Done Done false
32420 10019 Acked Flush 0 Sent Done Done false
32421 10020 Acked Write 7 Sent Done Done false
32422 10021 Acked Read 8 Sent Done Done false
32423 10022 Acked Write 1 Sent Done Done false
32424 10023 Acked Read 2 Sent Done Done false
32425 10024 Acked Write 8 Sent Done Done false
32426 10025 Acked Write 4 Sent Done Done false
32427 10026 Acked Read 2 Sent Done Done false
32428 10027 Acked Read 8 Sent Done Done false
32429 10028 Acked Read 2 Sent Done Done false
32430 10029 Acked Read 2 Sent Done Done false
32431 10030 Acked Read 8 Sent Done Done false
32432 10031 Acked Write 2 Sent Done Done false
32433 10032 Acked Read 7 Sent Done Done false
32434 10033 Acked Read 2 Sent Done Done false
32435 10034 Acked Flush 0 Sent Done Done false
32436 10035 Acked Write 3 Sent Done Done false
32437 10036 Acked Write 3 Sent Done Done false
32438 10037 Acked Read 3 Sent Done Done false
32439 10038 Acked Flush 0 Sent Done Done false
32440 10039 Acked Write 10 Sent Done Done false
32441 10040 Acked Read 9 Sent Done Done false
32442 10041 Acked Read 10 Sent Done Done false
32443 10042 Acked Read 8 Sent Done Done false
32444 10043 Acked Read 3 Sent Done Done false
32445 10044 Acked Write 5 Sent Done Done false
32446 10045 Acked Read 3 Sent Done Done false
32447 10046 Acked Write 7 Sent Done Done false
32448 10047 Acked Read 2 Sent Done Done false
32449 10048 Acked Read 7 Sent Done Done false
32450 10049 Acked Read 7 Sent Done Done false
32451 10050 Acked Flush 0 Sent Done Done false
32452 10051 Acked Write 6 Sent Done Done false
32453 10052 Acked Write 3 Sent Done Done false
32454 10053 Acked Write 6 Sent Done Done false
32455 10054 Acked Flush 0 Sent Done Done false
32456 10055 Acked Flush 0 Sent Done Done false
32457 STATES DS:0 DS:1 DS:2 TOTAL
32458 Sent 301 0 0 301
32459 Done 0 301 301 602
32460 Skipped 0 0 0 0
32461 Error 0 0 0 0
32462 Last Flush: Some(JobId(9754)) Some(JobId(10055)) Some(JobId(10055))
32463 Downstairs last five completed: 9754 9753 9752 9751 9750
32464 Upstairs last five completed: 10055 10054 10053 10052 10051
32465 CLIENT: Up:0 ds:301 act:2
324662025-01-23T18:54:42.195ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
324672025-01-23T18:54:42.196ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64427) = io task client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
324682025-01-23T18:54:42.197ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
324692025-01-23T18:54:42.197ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
324702025-01-23T18:54:42.197ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
324712025-01-23T18:54:42.197ZINFOcrucible: send last flush ID to this DS: Some(JobId(9754)) = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
324722025-01-23T18:54:42.197ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
324732025-01-23T18:54:42.197ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(9754)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
324742025-01-23T18:54:42.197ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(9754)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
32475 ----------------------------------------------------------------
32476 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
32477 Downstairs last five completed: 10055 10054 10053 10052 10051
32478 Upstairs last five completed: 10055 10054 10053 10052 10051
32479 CLIENT: Up:0 ds:0 act:3
32480 Replay: All jobs finished
32481 Main task has work to do, go find it
32482 got dsc Stop(1)
32483 stop 1
32484 [8820] Got stop action so:false kr:false
32485 Jan 23 18:54:44.437 INFO request completed, latency_us: 348, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 65b52570-2d54-48b2-a776-76c86552d566, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
324862025-01-23T18:54:44.438ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
32487 Jan 23 18:54:44.438 INFO request completed, latency_us: 176, response_code: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:54:44.438192541Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"":"io task","client":"1","":"downstairs","session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
32488 200, uri: /state/cid/1, method{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T18:54:44.438241005Z","hostname":"w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1200,"session_id":"bb6faa00-ad53-4274-84ae-5c5874c7a798"}
32489 : GET, req_id: 90f96ce9-be4d-47a7-a21b-2d3cab5eaa40, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
324902025-01-23T18:54:44.438ZINFOcrucible: client task is exiting = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
32491 [8820] Exited with: ExitStatus(unix_wait_status(9))
32492 [8820][1] reports Exit
32493 Jan 23 18:54:48.440 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/1, method: GET, req_id: bf0b5f40-b9eb-4b64-9545-52266e23cef7, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
32494 001/300 Flush
32495 002/300 Read block 226 len 24576
32496 003/300 Flush
32497 004/300 Read block 4964 len 32768
32498 005/300 Read block 583 len 36864
32499 006/300 Read block 3605 len 28672
32500 007/300 Flush
32501 008/300 Read block 2258 len 36864
32502 009/300 Flush
32503 010/300 Write block 507 len 28672 data: 8 7 5 5 5 4 4
32504 011/300 Read block 3270 len 32768
32505 012/300 Read block 2260 len 40960
32506 013/300 Read block 539 len 36864
32507 014/300 Read block 1502 len 12288
32508 015/300 Flush
32509 016/300 Write block 2276 len 40960 data: 5 6 4 4 5 5 6 7 5 5
32510 017/300 Read block 11 len 12288
32511 018/300 Read block 426 len 40960
32512 019/300 Read block 1475 len 36864
32513 020/300 Read block 3246 len 12288
32514 021/300 Read block 1881 len 16384
32515 022/300 Write block 4334 len 24576 data: 5 5 7 5 5 5
32516 023/300 Write block 1372 len 8192 data: 4 4
32517 024/300 Write block 66 len 8192 data: 4 5
32518 025/300 Read block 1026 len 40960
32519 026/300 Write block 4780 len 40960 data: 8 7 9 10 9 8 6 6 8 6
32520 027/300 Write block 4413 len 8192 data: 5 6
32521 028/300 Read block 3020 len 16384
32522 029/300 Read block 3763 len 24576
32523 030/300 Write block 3033 len 40960 data: 8 7 8 8 7 9 10 11 10 9
32524 031/300 Write block 1419 len 40960 data: 10 10 10 11 12 11 10 9 9 8
32525 032/300 Write block 1364 len 4096 data: 5
32526 033/300 Write block 562 len 12288 data: 5 5 5
32527 034/300 Read block 2042 len 40960
32528 035/300 Read block 2790 len 24576
32529 036/300 Read block 1246 len 4096
32530 037/300 Read block 2800 len 8192
32531 038/300 Write block 3401 len 28672 data: 4 5 5 5 5 4 4
32532 039/300 Write block 4381 len 20480 data: 3 4 6 6 5
32533 040/300 Write block 4238 len 12288 data: 5 5 5
32534 041/300 Write block 2505 len 4096 data: 4
32535 042/300 Write block 200 len 4096 data: 3
32536 043/300 Write block 4889 len 36864 data: 5 7 7 7 7 6 6 6 6
32537 044/300 Write block 2489 len 20480 data: 5 7 8 8 6
32538 045/300 Write block 4527 len 40960 data: 5 4 4 4 3 4 5 6 6 5
32539 046/300 Read block 1386 len 32768
32540 047/300 Write block 4347 len 36864 data: 7 8 7 6 5 5 5 4 6
32541 048/300 Read block 496 len 28672
32542 049/300 Read block 3142 len 24576
32543 050/300 Write block 1819 len 12288 data: 8 10 9
32544 051/300 Read block 4594 len 20480
32545 052/300 Write block 1357 len 4096 data: 5
32546 053/300 Write block 4545 len 28672 data: 6 5 3 3 4 4 5
32547 054/300 Write block 1220 len 16384 data: 8 8 9 7
32548 055/300 Write block 2338 len 8192 data: 5 5
32549 056/300 Flush
32550 057/300 Write block 1048 len 32768 data: 12 12 10 10 8 5 6 5
32551 058/300 Write block 1606 len 16384 data: 6 5 8 7
32552 059/300 Write block 1939 len 24576 data: 7 5 7 7 4 5
32553 060/300 Flush
32554 061/300 Read block 1879 len 16384
32555 062/300 Read block 1866 len 40960
32556 063/300 Write block 3998 len 12288 data: 7 7 7
32557 064/300 Read block 3464 len 12288
32558 065/300 Write block 1508 len 16384 data: 8 8 8 8
32559 066/300 Write block 1354 len 40960 data: 5 5 6 6 6 7 4 5 4 4
32560 067/300 Write block 3954 len 20480 data: 5 7 7 6 7
32561 068/300 Write block 3735 len 36864 data: 6 10 9 9 8 8 7 6 6
32562 069/300 Read block 716 len 8192
32563 070/300 Read block 4988 len 32768
32564 071/300 Read block 996 len 8192
32565 072/300 Read block 464 len 40960
32566 073/300 Write block 240 len 8192 data: 4 4
32567 074/300 Write block 361 len 20480 data: 4 5 5 5 7
32568 075/300 Read block 1725 len 16384
32569 076/300 Flush
32570 077/300 Write block 4160 len 24576 data: 8 8 8 9 12 10
32571 078/300 Read block 1710 len 32768
32572 079/300 Read block 969 len 16384
32573 080/300 Read block 3823 len 20480
32574 081/300 Flush
32575 082/300 Write block 4878 len 28672 data: 4 5 5 4 6 6 7
32576 083/300 Write block 705 len 20480 data: 2 2 3 4 5
32577 084/300 Write block 4591 len 8192 data: 2 3
32578 085/300 Write block 1309 len 12288 data: 7 7 7
32579 086/300 Flush
32580 087/300 Read block 782 len 12288
32581 088/300 Read block 3413 len 32768
32582 089/300 Read block 4282 len 4096
32583 090/300 Write block 3376 len 24576 data: 5 5 4 5 5 4
32584 091/300 Read block 4824 len 16384
32585 092/300 Read block 1285 len 24576
32586 093/300 Read block 129 len 40960
32587 094/300 Write block 4158 len 40960 data: 4 6 9 9 9 10 13 11 10 11
32588 095/300 Write block 316 len 16384 data: 4 4 4 5
32589 096/300 Read block 4071 len 24576
32590 097/300 Read block 2637 len 40960
32591 098/300 Read block 3454 len 12288
32592 099/300 Write block 4866 len 24576 data: 3 4 6 6 7 8
32593 100/300 Read block 2228 len 20480
32594 101/300 Read block 2150 len 36864
32595 102/300 Read block 2465 len 32768
32596 103/300 Read block 3715 len 16384
32597 104/300 Read block 3624 len 20480
32598 105/300 Write block 1533 len 20480 data: 8 7 5 5 5
32599 106/300 Read block 3482 len 32768
32600 107/300 Write block 2299 len 8192 data: 8 6
32601 108/300 Write block 3601 len 4096 data: 3
32602 109/300 Read block 3355 len 12288
32603 110/300 Read block 3272 len 8192
32604 111/300 Read block 1581 len 36864
32605 112/300 Read block 3559 len 16384
32606 113/300 Read block 2575 len 16384
32607 114/300 Write block 2958 len 24576 data: 5 6 7 7 8 7
32608 115/300 Read block 997 len 24576
32609 116/300 Write block 56 len 24576 data: 2 3 3 3 4 5
32610 117/300 Write block 4140 len 36864 data: 7 8 9 9 9 9 11 12 11
32611 118/300 Write block 3321 len 16384 data: 6 7 8 6
32612 119/300 Read block 1392 len 40960
32613 120/300 Read block 4990 len 16384
32614 121/300 Read block 3813 len 8192
32615 122/300 Read block 4322 len 40960
32616 123/300 Write block 3660 len 24576 data: 8 7 9 8 8 8
32617 124/300 Read block 3119 len 28672
32618 125/300 Write block 1655 len 40960 data: 4 3 2 4 4 7 7 6 7 6
32619 126/300 Read block 212 len 36864
32620 127/300 Read block 743 len 20480
32621 128/300 Read block 2212 len 16384
32622 129/300 Read block 2623 len 20480
32623 130/300 Write block 3994 len 40960 data: 6 7 7 8 8 8 8 5 6 7
32624 131/300 Flush
32625 132/300 Read block 3421 len 28672
32626 133/300 Write block 4932 len 28672 data: 6 6 6 6 5 5 5
32627 134/300 Read block 386 len 24576
32628 135/300 Read block 3428 len 32768
32629 136/300 Write block 2336 len 12288 data: 5 5 6
32630 137/300 Read block 386 len 20480
32631 138/300 Read block 1222 len 36864
32632 139/300 Write block 2179 len 28672 data: 4 3 4 4 5 4 5
32633 140/300 Read block 2237 len 4096
32634 141/300 Write block 4376 len 12288 data: 4 3 4
32635 142/300 Read block 2802 len 8192
32636 143/300 Write block 4476 len 36864 data: 6 8 7 8 7 5 5 5 6
32637 144/300 Write block 4355 len 24576 data: 7 6 6 5 6 5
32638 145/300 Write block 1739 len 36864 data: 7 7 3 3 3 3 5 5 4
32639 146/300 Read block 4618 len 24576
32640 147/300 Read block 1569 len 12288
32641 148/300 Write block 2 len 20480 data: 3 3 4 5 8
32642 149/300 Write block 3131 len 4096 data: 5
32643 150/300 Read block 1119 len 16384
32644 151/300 Read block 460 len 32768
32645 152/300 Read block 3330 len 16384
32646 153/300 Write block 2642 len 8192 data: 6 4
32647 154/300 Read block 4215 len 36864
32648 155/300 Write block 3427 len 36864 data: 6 6 5 5 6 4 3 3 3
32649 156/300 Read block 1953 len 28672
32650 157/300 Write block 3643 len 12288 data: 4 6 5
32651 158/300 Read block 2073 len 36864
32652 159/300 Read block 1266 len 24576
32653 160/300 Write block 4658 len 4096 data: 5
32654 161/300 Write block 2662 len 24576 data: 5 5 4 4 4 4
32655 162/300 Write block 3694 len 32768 data: 8 7 7 6 7 6 5 5
32656 163/300 Read block 4155 len 8192
32657 164/300 Write block 3436 len 8192 data: 4 4
32658 165/300 Read block 2310 len 20480
32659 166/300 Write block 4190 len 8192 data: 5 7
32660 167/300 Write block 1910 len 28672 data: 5 5 7 7 7 7 6
32661 168/300 Read block 2869 len 36864
32662 169/300 Read block 4837 len 12288
32663 170/300 Read block 4554 len 20480
32664 171/300 Read block 1709 len 28672
32665 172/300 Read block 4921 len 32768
32666 173/300 Flush
32667 174/300 Write block 3882 len 12288 data: 8 8 9
32668 175/300 Read block 1782 len 40960
32669 176/300 Write block 2534 len 12288 data: 7 10 9
32670 177/300 Write block 149 len 20480 data: 9 9 9 8 8
32671 178/300 Write block 4879 len 32768 data: 6 6 5 7 7 8 8 7
32672 179/300 Read block 942 len 24576
32673 180/300 Read block 2219 len 4096
32674 181/300 Read block 120 len 32768
32675 182/300 Read block 4185 len 28672
32676 183/300 Write block 3621 len 24576 data: 7 9 8 8 8 8
32677 184/300 Write block 1446 len 8192 data: 3 3
32678 185/300 Flush
32679 186/300 Write block 178 len 12288 data: 11 10 8
32680 187/300 Write block 4777 len 16384 data: 6 9 9 9
32681 188/300 Write block 3714 len 36864 data: 7 7 8 8 5 4 5 5 7
32682 189/300 Write block 289 len 12288 data: 7 7 8
32683 190/300 Read block 1905 len 24576
32684 191/300 Write block 2650 len 28672 data: 4 4 6 6 5 8 7
32685 192/300 Read block 777 len 4096
32686 193/300 Read block 3327 len 24576
32687 194/300 Read block 4861 len 28672
32688 195/300 Flush
32689 196/300 Flush
32690 197/300 Read block 2722 len 36864
32691 198/300 Write block 1211 len 40960 data: 3 3 4 6 6 6 7 6 7 9
32692 199/300 Read block 502 len 12288
32693 200/300 Read block 2083 len 36864
32694 201/300 Read block 4756 len 32768
32695 202/300 Flush
32696 203/300 Write block 4312 len 8192 data: 6 6
32697 204/300 Write block 4421 len 36864 data: 8 8 9 11 8 8 8 9 8
32698 205/300 Write block 3871 len 12288 data: 7 4 5
32699 206/300 Write block 4778 len 20480 data: 10 10 10 8 10
32700 207/300 Read block 467 len 36864
32701 208/300 Write block 2478 len 36864 data: 4 4 2 2 3 3 3 3 4
32702 209/300 Read block 3076 len 16384
32703 210/300 Read block 2292 len 28672
32704 211/300 Flush
32705 212/300 Read block 2848 len 40960
32706 213/300 Flush
32707 214/300 Write block 1941 len 4096 data: 8
32708 215/300 Read block 1165 len 32768
32709 216/300 Flush
32710 217/300 Read block 581 len 28672
32711 218/300 Read block 4708 len 12288
32712 219/300 Write block 791 len 24576 data: 6 5 4 5 5 8
32713 220/300 Write block 2982 len 8192 data: 3 3
32714 221/300 Read block 4633 len 28672
32715 222/300 Write block 876 len 16384 data: 7 5 5 6
32716 223/300 Read block 4400 len 36864
32717 224/300 Write block 2417 len 16384 data: 6 7 6 5
32718 225/300 Read block 3407 len 24576
32719 226/300 Read block 4734 len 16384
32720 227/300 Read block 214 len 12288
32721 228/300 Write block 4234 len 16384 data: 4 3 3 4
32722 229/300 Flush
32723 230/300 Read block 2902 len 4096
32724 231/300 Write block 3817 len 16384 data: 5 6 6 6
32725 232/300 Write block 3335 len 8192 data: 4 4
32726 233/300 Write block 920 len 32768 data: 4 3 3 4 4 3 4 4
32727 234/300 Write block 716 len 16384 data: 8 6 6 5
32728 235/300 Read block 1035 len 16384
32729 236/300 Write block 895 len 20480 data: 8 10 8 8 9
32730 237/300 Read block 3730 len 40960
32731 238/300 Flush
32732 239/300 Write block 2689 len 36864 data: 7 7 5 6 5 4 5 6 5
32733 240/300 Flush
32734 241/300 Read block 4246 len 24576
32735 242/300 Write block 2770 len 8192 data: 5 5
32736 243/300 Write block 1282 len 28672 data: 6 7 9 9 11 11 10
32737 244/300 Write block 226 len 8192 data: 3 3
32738 245/300 Write block 458 len 24576 data: 9 9 8 8 10 9
32739 246/300 Read block 2732 len 4096
32740 247/300 Read block 2856 len 4096
32741 248/300 Flush
32742 249/300 Write block 2980 len 28672 data: 4 3 4 4 4 3 3
32743 250/300 Write block 735 len 28672 data: 8 7 6 7 7 8 9
32744 251/300 Write block 3678 len 40960 data: 7 6 6 5 4 4 6 5 5 6
32745 252/300 Read block 4622 len 32768
32746 253/300 Read block 535 len 8192
32747 254/300 Read block 3317 len 28672
32748 255/300 Write block 698 len 28672 data: 4 3 4 3 2 2 2
32749 256/300 Flush
32750 257/300 Write block 4426 len 16384 data: 9 9 10 9
32751 258/300 Read block 1627 len 12288
32752 259/300 Read block 1866 len 32768
32753 260/300 Read block 2620 len 12288
32754 261/300 Write block 283 len 28672 data: 5 4 7 7 4 3 8
32755 262/300 Write block 2617 len 32768 data: 8 9 8 8 8 8 7 8
32756 263/300 Read block 3432 len 12288
32757 264/300 Read block 4615 len 40960
32758 265/300 Read block 1577 len 20480
32759 266/300 Read block 3800 len 36864
32760 267/300 Write block 3352 len 20480 data: 6 6 7 5 5
32761 268/300 Write block 1365 len 4096 data: 5
32762 269/300 Write block 2907 len 24576 data: 6 7 8 9 7 8
32763 270/300 Write block 4829 len 4096 data: 7
32764 271/300 Read block 1917 len 12288
32765 272/300 Write block 4512 len 20480 data: 4 4 5 4 4
32766 273/300 Write block 2958 len 4096 data: 6
32767 274/300 Read block 706 len 24576
32768 275/300 Read block 3376 len 28672
32769 276/300 Read block 120 len 8192
32770 277/300 Read block 2244 len 28672
32771 278/300 Read block 2523 len 16384
32772 279/300 Write block 109 len 24576 data: 9 7 9 8 9 10
32773 280/300 Write block 3928 len 36864 data: 7 7 5 3 2 2 2 2 3
32774 281/300 Read block 4274 len 28672
32775 282/300 Read block 4887 len 4096
32776 283/300 Read block 4351 len 8192
32777 284/300 Write block 2103 len 12288 data: 10 10 10
32778 285/300 Write block 864 len 40960 data: 8 8 7 7 8 7 5 6 5 5
32779 286/300 Write block 1511 len 8192 data: 9 8
32780 287/300 Write block 1249 len 16384 data: 10 9 8 10
32781 288/300 Write block 4770 len 24576 data: 3 3 3 4 4 4
32782 289/300 Read block 1052 len 12288
32783 290/300 Read block 3407 len 40960
32784 291/300 Read block 56 len 32768
32785 292/300 Flush
32786 293/300 Write block 2567 len 16384 data: 4 5 4 5
32787 294/300 Read block 2164 len 24576
32788 295/300 Read block 1454 len 8192
32789 296/300 Read block 2583 len 28672
32790 297/300 Write block 522 len 32768 data: 7 5 6 7 9 10 11 10
32791 298/300 Read block 1294 len 32768
32792 299/300 Read block 3348 len 40960
32793 300/300 Write block 4469 len 8192 data: 4 3
32794 301/300 Read block 2117 len 16384
32795 Main task has work to do, go find it
32796 got dsc Start(1)
32797 start 1
32798 Jan 23 18:54:48.687 INFO request completed, latency_us: 243, response_code: 204, uri: /start/cid/1, method: POST, req_id: 9aa8c676-7727-4ccc-93d7-809183f9b22e, remote_addr: 127.0.0.1:59295, local_addr: 127.0.0.1:9998
32799 Replay: started 1, returned:Ok(())
32800 ----------------------------------------------------------------
32801 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32802 States: Active Offline Active
32803 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32804 10056 Acked Flush 0 Done Sent Done false
32805 10057 Acked Read 6 Done Sent Done false
32806 10058 Acked Flush 0 Done Sent Done false
32807 10059 Acked Read 8 Done Sent Done false
32808 10060 Acked Read 9 Done Sent Done false
32809 10061 Acked Read 7 Done Sent Done false
32810 10062 Acked Flush 0 Done Sent Done false
32811 10063 Acked Read 9 Done Sent Done false
32812 10064 Acked Flush 0 Done Sent Done false
32813 10065 Acked Write 7 Done Sent Done false
32814 10066 Acked Read 8 Done Sent Done false
32815 10067 Acked Read 10 Done Sent Done false
32816 10068 Acked Read 9 Done Sent Done false
32817 10069 Acked Read 3 Done Sent Done false
32818 10070 Acked Flush 0 Done Sent Done false
32819 10071 Acked Write 10 Done Sent Done false
32820 10072 Acked Read 3 Done Sent Done false
32821 10073 Acked Read 10 Done Sent Done false
32822 10074 Acked Read 9 Done Sent Done false
32823 10075 Acked Read 3 Done Sent Done false
32824 10076 Acked Read 4 Done Sent Done false
32825 10077 Acked Write 6 Done Sent Done false
32826 10078 Acked Write 2 Done Sent Done false
32827 10079 Acked Write 2 Done Sent Done false
32828 10080 Acked Read 10 Done Sent Done false
32829 10081 Acked Write 10 Done Sent Done false
32830 10082 Acked Write 2 Done Sent Done false
32831 10083 Acked Read 4 Done Sent Done false
32832 10084 Acked Read 6 Done Sent Done false
32833 10085 Acked Write 10 Done Sent Done false
32834 10086 Acked Write 10 Done Sent Done false
32835 10087 Acked Write 1 Done Sent Done false
32836 10088 Acked Write 3 Done Sent Done false
32837 10089 Acked Read 10 Done Sent Done false
32838 10090 Acked Read 6 Done Sent Done false
32839 10091 Acked Read 1 Done Sent Done false
32840 10092 Acked Read 2 Done Sent Done false
32841 10093 Acked Write 7 Done Sent Done false
32842 10094 Acked Write 5 Done Sent Done false
32843 10095 Acked Write 3 Done Sent Done false
32844 10096 Acked Write 1 Done Sent Done false
32845 10097 Acked Write 1 Done Sent Done false
32846 10098 Acked Write 9 Done Sent Done false
32847 10099 Acked Write 5 Done Sent Done false
32848 10100 Acked Write 10 Done Sent Done false
32849 10101 Acked Read 8 Done Sent Done false
32850 10102 Acked Write 9 Done Sent Done false
32851 10103 Acked Read 7 Done Sent Done false
32852 10104 Acked Read 6 Done Sent Done false
32853 10105 Acked Write 3 Done Sent Done false
32854 10106 Acked Read 5 Done Sent Done false
32855 10107 Acked Write 1 Done Sent Done false
32856 10108 Acked Write 7 Done Sent Done false
32857 10109 Acked Write 4 Done Sent Done false
32858 10110 Acked Write 2 Done Sent Done false
32859 10111 Acked Flush 0 Done Sent Done false
32860 10112 Acked Write 8 Done Sent Done false
32861 10113 Acked Write 4 Done Sent Done false
32862 10114 Acked Write 6 Done Sent Done false
32863 10115 Acked Flush 0 Done Sent Done false
32864 10116 Acked Read 4 Done Sent Done false
32865 10117 Acked Read 10 Done Sent Done false
32866 10118 Acked Write 3 Done Sent Done false
32867 10119 Acked Read 3 Done Sent Done false
32868 10120 Acked Write 4 Done Sent Done false
32869 10121 Acked Write 10 Done Sent Done false
32870 10122 Acked Write 5 Done Sent Done false
32871 10123 Acked Write 9 Done Sent Done false
32872 10124 Acked Read 2 Done Sent Done false
32873 10125 Acked Read 8 Done Sent Done false
32874 10126 Acked Read 2 Done Sent Done false
32875 10127 Acked Read 10 Done Sent Done false
32876 10128 Acked Write 2 Done Sent Done false
32877 10129 Acked Write 5 Done Sent Done false
32878 10130 Acked Read 4 Done Sent Done false
32879 10131 Acked Flush 0 Done Sent Done false
32880 10132 Acked Write 6 Done Sent Done false
32881 10133 Acked Read 8 Done Sent Done false
32882 10134 Acked Read 4 Done Sent Done false
32883 10135 Acked Read 5 Done Sent Done false
32884 10136 Acked Flush 0 Done Sent Done false
32885 10137 Acked Write 7 Done Sent Done false
32886 10138 Acked Write 5 Done Sent Done false
32887 10139 Acked Write 2 Done Sent Done false
32888 10140 Acked Write 3 Done Sent Done false
32889 10141 Acked Flush 0 Done Sent Done false
32890 10142 Acked Read 3 Done Sent Done false
32891 10143 Acked Read 8 Done Sent Done false
32892 10144 Acked Read 1 Done Sent Done false
32893 10145 Acked Write 6 Done Sent Done false
32894 10146 Acked Read 4 Done Sent Done false
32895 10147 Acked Read 6 Done Sent Done false
32896 10148 Acked Read 10 Done Sent Done false
32897 10149 Acked Write 10 Done Sent Done false
32898 10150 Acked Write 4 Done Sent Done false
32899 10151 Acked Read 6 Done Sent Done false
32900 10152 Acked Read 10 Done Sent Done false
32901 10153 Acked Read 3 Done Sent Done false
32902 10154 Acked Write 6 Done Sent Done false
32903 10155 Acked Read 5 Done Sent Done false
32904 10156 Acked Read 9 Done Sent Done false
32905 10157 Acked Read 8 Done Sent Done false
32906 10158 Acked Read 4 Done Sent Done false
32907 10159 Acked Read 5 Done Sent Done false
32908 10160 Acked Write 5 Done Sent Done false
32909 10161 Acked Read 8 Done Sent Done false
32910 10162 Acked Write 2 Done Sent Done false
32911 10163 Acked Write 1 Done Sent Done false
32912 10164 Acked Read 3 Done Sent Done false
32913 10165 Acked Read 2 Done Sent Done false
32914 10166 Acked Read 9 Done Sent Done false
32915 10167 Acked Read 4 Done Sent Done false
32916 10168 Acked Read 4 Done Sent Done false
32917 10169 Acked Write 6 Done Sent Done false
32918 10170 Acked Read 6 Done Sent Done false
32919 10171 Acked Write 6 Done Sent Done false
32920 10172 Acked Write 9 Done Sent Done false
32921 10173 Acked Write 4 Done Sent Done false
32922 10174 Acked Read 10 Done Sent Done false
32923 10175 Acked Read 4 Done Sent Done false
32924 10176 Acked Read 2 Done Sent Done false
32925 10177 Acked Read 10 Done Sent Done false
32926 10178 Acked Write 6 Done Sent Done false
32927 10179 Acked Read 7 Done Sent Done false
32928 10180 Acked Write 10 Done Sent Done false
32929 10181 Acked Read 9 Done Sent Done false
32930 10182 Acked Read 5 Done Sent Done false
32931 10183 Acked Read 4 Done Sent Done false
32932 10184 Acked Read 5 Done Sent Done false
32933 10185 Acked Write 10 Done Sent Done false
32934 10186 Acked Flush 0 Done Sent Done false
32935 10187 Acked Read 7 Done Sent Done false
32936 10188 Acked Write 7 Done Sent Done false
32937 10189 Acked Read 6 Done Sent Done false
32938 10190 Acked Read 8 Done Sent Done false
32939 10191 Acked Write 3 Done Sent Done false
32940 10192 Acked Read 5 Done Sent Done false
32941 10193 Acked Read 9 Done Sent Done false
32942 10194 Acked Write 7 Done Sent Done false
32943 10195 Acked Read 1 Done Sent Done false
32944 10196 Acked Write 3 Done Sent Done false
32945 10197 Acked Read 2 Done Sent Done false
32946 10198 Acked Write 9 Done Sent Done false
32947 10199 Acked Write 6 Done Sent Done false
32948 10200 Acked Write 9 Done Sent Done false
32949 10201 Acked Read 6 Done Sent Done false
32950 10202 Acked Read 3 Done Sent Done false
32951 10203 Acked Write 5 Done Sent Done false
32952 10204 Acked Write 1 Done Sent Done false
32953 10205 Acked Read 4 Done Sent Done false
32954 10206 Acked Read 8 Done Sent Done false
32955 10207 Acked Read 4 Done Sent Done false
32956 10208 Acked Write 2 Done Sent Done false
32957 10209 Acked Read 9 Done Sent Done false
32958 10210 Acked Write 9 Done Sent Done false
32959 10211 Acked Read 7 Done Sent Done false
32960 10212 Acked Write 3 Done Sent Done false
32961 10213 Acked Read 9 Done Sent Done false
32962 10214 Acked Read 6 Done Sent Done false
32963 10215 Acked Write 1 Done Sent Done false
32964 10216 Acked Write 6 Done Sent Done false
32965 10217 Acked Write 8 Done Sent Done false
32966 10218 Acked Read 2 Done Sent Done false
32967 10219 Acked Write 2 Done Sent Done false
32968 10220 Acked Read 5 Done Sent Done false
32969 10221 Acked Write 2 Done Sent Done false
32970 10222 Acked Write 7 Done Sent Done false
32971 10223 Acked Read 9 Done Sent Done false
32972 10224 Acked Read 3 Done Sent Done false
32973 10225 Acked Read 5 Done Sent Done false
32974 10226 Acked Read 7 Done Sent Done false
32975 10227 Acked Read 8 Done Sent Done false
32976 10228 Acked Flush 0 Done Sent Done false
32977 10229 Acked Write 3 Done Sent Done false
32978 10230 Acked Read 10 Done Sent Done false
32979 10231 Acked Write 3 Done Sent Done false
32980 10232 Acked Write 5 Done Sent Done false
32981 10233 Acked Write 8 Done Sent Done false
32982 10234 Acked Read 6 Done Sent Done false
32983 10235 Acked Read 1 Done Sent Done false
32984 10236 Acked Read 8 Done Sent Done false
32985 10237 Acked Read 7 Done Sent Done false
32986 10238 Acked Write 6 Done Sent Done false
32987 10239 Acked Write 2 Done Sent Done false
32988 10240 Acked Flush 0 Done Sent Done false
32989 10241 Acked Write 3 Done Sent Done false
32990 10242 Acked Write 4 Done Sent Done false
32991 10243 Acked Write 9 Done Sent Done false
32992 10244 Acked Write 3 Done Sent Done false
32993 10245 Acked Read 6 Done Sent Done false
32994 10246 Acked Write 7 Done Sent Done false
32995 10247 Acked Read 1 Done Sent Done false
32996 10248 Acked Read 6 Done Sent Done false
32997 10249 Acked Read 7 Done Sent Done false
32998 10250 Acked Flush 0 Done Sent Done false
32999 10251 Acked Flush 0 Done Sent Done false
33000 10252 Acked Read 9 Done Sent Done false
33001 10253 Acked Write 10 Done Sent Done false
33002 10254 Acked Read 3 Done Sent Done false
33003 10255 Acked Read 9 Done Sent Done false
33004 10256 Acked Read 8 Done Sent Done false
33005 10257 Acked Flush 0 Done Sent Done false
33006 10258 Acked Write 2 Done Sent Done false
33007 10259 Acked Write 9 Done Sent Done false
33008 10260 Acked Write 3 Done Sent Done false
33009 10261 Acked Write 5 Done Sent Done false
33010 10262 Acked Read 9 Done Sent Done false
33011 10263 Acked Write 9 Done Sent Done false
33012 10264 Acked Read 4 Done Sent Done false
33013 10265 Acked Read 7 Done Sent Done false
33014 10266 Acked Flush 0 Done Sent Done false
33015 10267 Acked Read 10 Done Sent Done false
33016 10268 Acked Flush 0 Done Sent Done false
33017 10269 Acked Write 1 Done Sent Done false
33018 10270 Acked Read 8 Done Sent Done false
33019 10271 Acked Flush 0 Done Sent Done false
33020 10272 Acked Read 7 Done Sent Done false
33021 10273 Acked Read 3 Done Sent Done false
33022 10274 Acked Write 6 Done Sent Done false
33023 10275 Acked Write 2 Done Sent Done false
33024 10276 Acked Read 7 Done Sent Done false
33025 10277 Acked Write 4 Done Sent Done false
33026 10278 Acked Read 9 Done Sent Done false
33027 10279 Acked Write 4 Done Sent Done false
33028 10280 Acked Read 6 Done Sent Done false
33029 10281 Acked Read 4 Done Sent Done false
33030 10282 Acked Read 3 Done Sent Done false
33031 10283 Acked Write 4 Done Sent Done false
33032 10284 Acked Flush 0 Done Sent Done false
33033 10285 Acked Read 1 Done Sent Done false
33034 10286 Acked Write 4 Done Sent Done false
33035 10287 Acked Write 2 Done Sent Done false
33036 10288 Acked Write 8 Done Sent Done false
33037 10289 Acked Write 4 Done Sent Done false
33038 10290 Acked Read 4 Done Sent Done false
33039 10291 Acked Write 5 Done Sent Done false
33040 10292 Acked Read 10 Done Sent Done false
33041 10293 Acked Flush 0 Done Sent Done false
33042 10294 Acked Write 9 Done Sent Done false
33043 10295 Acked Flush 0 Done Sent Done false
33044 10296 Acked Read 6 Done Sent Done false
33045 10297 Acked Write 2 Done Sent Done false
33046 10298 Acked Write 7 Done Sent Done false
33047 10299 Acked Write 2 Done Sent Done false
33048 10300 Acked Write 6 Done Sent Done false
33049 10301 Acked Read 1 Done Sent Done false
33050 10302 Acked Read 1 Done Sent Done false
33051 10303 Acked Flush 0 Done Sent Done false
33052 10304 Acked Write 7 Done Sent Done false
33053 10305 Acked Write 7 Done Sent Done false
33054 10306 Acked Write 10 Done Sent Done false
33055 10307 Acked Read 8 Done Sent Done false
33056 10308 Acked Read 2 Done Sent Done false
33057 10309 Acked Read 7 Done Sent Done false
33058 10310 Acked Write 7 Done Sent Done false
33059 10311 Acked Flush 0 Done Sent Done false
33060 10312 Acked Write 4 Done Sent Done false
33061 10313 Acked Read 3 Done Sent Done false
33062 10314 Acked Read 8 Done Sent Done false
33063 10315 Acked Read 3 Done Sent Done false
33064 10316 Acked Write 7 Done Sent Done false
33065 10317 Acked Write 8 Done Sent Done false
33066 10318 Acked Read 3 Done Sent Done false
33067 10319 Acked Read 10 Done Sent Done false
33068 10320 Acked Read 5 Done Sent Done false
33069 10321 Acked Read 9 Done Sent Done false
33070 10322 Acked Write 5 Done Sent Done false
33071 10323 Acked Write 1 Done Sent Done false
33072 10324 Acked Write 6 Done Sent Done false
33073 10325 Acked Write 1 Done Sent Done false
33074 10326 Acked Read 3 Done Sent Done false
33075 10327 Acked Write 5 Done Sent Done false
33076 10328 Acked Write 1 Done Sent Done false
33077 10329 Acked Read 6 Done Sent Done false
33078 10330 Acked Read 7 Done Sent Done false
33079 10331 Acked Read 2 Done Sent Done false
33080 10332 Acked Read 7 Done Sent Done false
33081 10333 Acked Read 4 Done Sent Done false
33082 10334 Acked Write 6 Done Sent Done false
33083 10335 Acked Write 9 Done Sent Done false
33084 10336 Acked Read 7 Done Sent Done false
33085 10337 Acked Read 1 Done Sent Done false
33086 10338 Acked Read 2 Done Sent Done false
33087 10339 Acked Write 3 Done Sent Done false
33088 10340 Acked Write 10 Done Sent Done false
33089 10341 Acked Write 2 Done Sent Done false
33090 10342 Acked Write 4 Done Sent Done false
33091 10343 Acked Write 6 Done Sent Done false
33092 10344 Acked Read 3 Done Sent Done false
33093 10345 Acked Read 10 Done Sent Done false
33094 10346 Acked Read 8 Done Sent Done false
33095 10347 Acked Flush 0 Done Sent Done false
33096 10348 Acked Write 4 Done Sent Done false
33097 10349 Acked Read 6 Done Sent Done false
33098 10350 Acked Read 2 Done Sent Done false
33099 10351 Acked Read 7 Done Sent Done false
33100 10352 Acked Write 8 Done Sent Done false
33101 10353 Acked Read 8 Done Sent Done false
33102 10354 Acked Read 10 Done Sent Done false
33103 10355 Acked Write 2 Done Sent Done false
33104 10356 Acked Read 4 Done Sent Done false
33105 STATES DS:0 DS:1 DS:2 TOTAL
33106 Sent 0 301 0 301
33107 Done 301 0 301 602
33108 Skipped 0 0 0 0
33109 Error 0 0 0 0
33110 Last Flush: Some(JobId(10347)) Some(JobId(10055)) Some(JobId(10347))
33111 Downstairs last five completed: 10055 10054 10053 10052 10051
33112 Upstairs last five completed: 10356 10355 10354 10353 10352
33113 CLIENT: Up:0 ds:301 act:2
33114 [8820] got start action
33115 [8820] I am going to restart
33116 Starting downstairs at port 8820
33117 Make output file at "/tmp/dsc/downstairs-8820.txt"
33118 [8820][1] reports Starting
33119 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1231)
33120 [8820][1] reports Running
33121 ----------------------------------------------------------------
33122 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
33123 States: Active Offline Active
33124 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33125 10056 Acked Flush 0 Done Sent Done false
33126 10057 Acked Read 6 Done Sent Done false
33127 10058 Acked Flush 0 Done Sent Done false
33128 10059 Acked Read 8 Done Sent Done false
33129 10060 Acked Read 9 Done Sent Done false
33130 10061 Acked Read 7 Done Sent Done false
33131 10062 Acked Flush 0 Done Sent Done false
33132 10063 Acked Read 9 Done Sent Done false
33133 10064 Acked Flush 0 Done Sent Done false
33134 10065 Acked Write 7 Done Sent Done false
33135 10066 Acked Read 8 Done Sent Done false
33136 10067 Acked Read 10 Done Sent Done false
33137 10068 Acked Read 9 Done Sent Done false
33138 10069 Acked Read 3 Done Sent Done false
33139 10070 Acked Flush 0 Done Sent Done false
33140 10071 Acked Write 10 Done Sent Done false
33141 10072 Acked Read 3 Done Sent Done false
33142 10073 Acked Read 10 Done Sent Done false
33143 10074 Acked Read 9 Done Sent Done false
33144 10075 Acked Read 3 Done Sent Done false
33145 10076 Acked Read 4 Done Sent Done false
33146 10077 Acked Write 6 Done Sent Done false
33147 10078 Acked Write 2 Done Sent Done false
33148 10079 Acked Write 2 Done Sent Done false
33149 10080 Acked Read 10 Done Sent Done false
33150 10081 Acked Write 10 Done Sent Done false
33151 10082 Acked Write 2 Done Sent Done false
33152 10083 Acked Read 4 Done Sent Done false
33153 10084 Acked Read 6 Done Sent Done false
33154 10085 Acked Write 10 Done Sent Done false
33155 10086 Acked Write 10 Done Sent Done false
33156 10087 Acked Write 1 Done Sent Done false
33157 10088 Acked Write 3 Done Sent Done false
33158 10089 Acked Read 10 Done Sent Done false
33159 10090 Acked Read 6 Done Sent Done false
33160 10091 Acked Read 1 Done Sent Done false
33161 10092 Acked Read 2 Done Sent Done false
33162 10093 Acked Write 7 Done Sent Done false
33163 10094 Acked Write 5 Done Sent Done false
33164 10095 Acked Write 3 Done Sent Done false
33165 10096 Acked Write 1 Done Sent Done false
33166 10097 Acked Write 1 Done Sent Done false
33167 10098 Acked Write 9 Done Sent Done false
33168 10099 Acked Write 5 Done Sent Done false
33169 10100 Acked Write 10 Done Sent Done false
33170 10101 Acked Read 8 Done Sent Done false
33171 10102 Acked Write 9 Done Sent Done false
33172 10103 Acked Read 7 Done Sent Done false
33173 10104 Acked Read 6 Done Sent Done false
33174 10105 Acked Write 3 Done Sent Done false
33175 10106 Acked Read 5 Done Sent Done false
33176 10107 Acked Write 1 Done Sent Done false
33177 10108 Acked Write 7 Done Sent Done false
33178 10109 Acked Write 4 Done Sent Done false
33179 10110 Acked Write 2 Done Sent Done false
33180 10111 Acked Flush 0 Done Sent Done false
33181 10112 Acked Write 8 Done Sent Done false
33182 10113 Acked Write 4 Done Sent Done false
33183 10114 Acked Write 6 Done Sent Done false
33184 10115 Acked Flush 0 Done Sent Done false
33185 10116 Acked Read 4 Done Sent Done false
33186 10117 Acked Read 10 Done Sent Done false
33187 10118 Acked Write 3 Done Sent Done false
33188 10119 Acked Read 3 Done Sent Done false
33189 10120 Acked Write 4 Done Sent Done false
33190 10121 Acked Write 10 Done Sent Done false
33191 10122 Acked Write 5 Done Sent Done false
33192 10123 Acked Write 9 Done Sent Done false
33193 10124 Acked Read 2 Done Sent Done false
33194 10125 Acked Read 8 Done Sent Done false
33195 10126 Acked Read 2 Done Sent Done false
33196 10127 Acked Read 10 Done Sent Done false
33197 10128 Acked Write 2 Done Sent Done false
33198 10129 Acked Write 5 Done Sent Done false
33199 10130 Acked Read 4 Done Sent Done false
33200 10131 Acked Flush 0 Done Sent Done false
33201 10132 Acked Write 6 Done Sent Done false
33202 10133 Acked Read 8 Done Sent Done false
33203 10134 Acked Read 4 Done Sent Done false
33204 10135 Acked Read 5 Done Sent Done false
33205 10136 Acked Flush 0 Done Sent Done false
33206 10137 Acked Write 7 Done Sent Done false
33207 10138 Acked Write 5 Done Sent Done false
33208 10139 Acked Write 2 Done Sent Done false
33209 10140 Acked Write 3 Done Sent Done false
33210 10141 Acked Flush 0 Done Sent Done false
33211 10142 Acked Read 3 Done Sent Done false
33212 10143 Acked Read 8 Done Sent Done false
33213 10144 Acked Read 1 Done Sent Done false
33214 10145 Acked Write 6 Done Sent Done false
33215 10146 Acked Read 4 Done Sent Done false
33216 10147 Acked Read 6 Done Sent Done false
33217 10148 Acked Read 10 Done Sent Done false
33218 10149 Acked Write 10 Done Sent Done false
33219 10150 Acked Write 4 Done Sent Done false
33220 10151 Acked Read 6 Done Sent Done false
33221 10152 Acked Read 10 Done Sent Done false
33222 10153 Acked Read 3 Done Sent Done false
33223 10154 Acked Write 6 Done Sent Done false
33224 10155 Acked Read 5 Done Sent Done false
33225 10156 Acked Read 9 Done Sent Done false
33226 10157 Acked Read 8 Done Sent Done false
33227 10158 Acked Read 4 Done Sent Done false
33228 10159 Acked Read 5 Done Sent Done false
33229 10160 Acked Write 5 Done Sent Done false
33230 10161 Acked Read 8 Done Sent Done false
33231 10162 Acked Write 2 Done Sent Done false
33232 10163 Acked Write 1 Done Sent Done false
33233 10164 Acked Read 3 Done Sent Done false
33234 10165 Acked Read 2 Done Sent Done false
33235 10166 Acked Read 9 Done Sent Done false
33236 10167 Acked Read 4 Done Sent Done false
33237 10168 Acked Read 4 Done Sent Done false
33238 10169 Acked Write 6 Done Sent Done false
33239 10170 Acked Read 6 Done Sent Done false
33240 10171 Acked Write 6 Done Sent Done false
33241 10172 Acked Write 9 Done Sent Done false
33242 10173 Acked Write 4 Done Sent Done false
33243 10174 Acked Read 10 Done Sent Done false
33244 10175 Acked Read 4 Done Sent Done false
33245 10176 Acked Read 2 Done Sent Done false
33246 10177 Acked Read 10 Done Sent Done false
33247 10178 Acked Write 6 Done Sent Done false
33248 10179 Acked Read 7 Done Sent Done false
33249 10180 Acked Write 10 Done Sent Done false
33250 10181 Acked Read 9 Done Sent Done false
33251 10182 Acked Read 5 Done Sent Done false
33252 10183 Acked Read 4 Done Sent Done false
33253 10184 Acked Read 5 Done Sent Done false
33254 10185 Acked Write 10 Done Sent Done false
33255 10186 Acked Flush 0 Done Sent Done false
33256 10187 Acked Read 7 Done Sent Done false
33257 10188 Acked Write 7 Done Sent Done false
33258 10189 Acked Read 6 Done Sent Done false
33259 10190 Acked Read 8 Done Sent Done false
33260 10191 Acked Write 3 Done Sent Done false
33261 10192 Acked Read 5 Done Sent Done false
33262 10193 Acked Read 9 Done Sent Done false
33263 10194 Acked Write 7 Done Sent Done false
33264 10195 Acked Read 1 Done Sent Done false
33265 10196 Acked Write 3 Done Sent Done false
33266 10197 Acked Read 2 Done Sent Done false
33267 10198 Acked Write 9 Done Sent Done false
33268 10199 Acked Write 6 Done Sent Done false
33269 10200 Acked Write 9 Done Sent Done false
33270 10201 Acked Read 6 Done Sent Done false
33271 10202 Acked Read 3 Done Sent Done false
33272 10203 Acked Write 5 Done Sent Done false
33273 10204 Acked Write 1 Done Sent Done false
33274 10205 Acked Read 4 Done Sent Done false
33275 10206 Acked Read 8 Done Sent Done false
33276 10207 Acked Read 4 Done Sent Done false
33277 10208 Acked Write 2 Done Sent Done false
33278 10209 Acked Read 9 Done Sent Done false
33279 10210 Acked Write 9 Done Sent Done false
33280 10211 Acked Read 7 Done Sent Done false
33281 10212 Acked Write 3 Done Sent Done false
33282 10213 Acked Read 9 Done Sent Done false
33283 10214 Acked Read 6 Done Sent Done false
33284 10215 Acked Write 1 Done Sent Done false
33285 10216 Acked Write 6 Done Sent Done false
33286 10217 Acked Write 8 Done Sent Done false
33287 10218 Acked Read 2 Done Sent Done false
33288 10219 Acked Write 2 Done Sent Done false
33289 10220 Acked Read 5 Done Sent Done false
33290 10221 Acked Write 2 Done Sent Done false
33291 10222 Acked Write 7 Done Sent Done false
33292 10223 Acked Read 9 Done Sent Done false
33293 10224 Acked Read 3 Done Sent Done false
33294 10225 Acked Read 5 Done Sent Done false
33295 10226 Acked Read 7 Done Sent Done false
33296 10227 Acked Read 8 Done Sent Done false
33297 10228 Acked Flush 0 Done Sent Done false
33298 10229 Acked Write 3 Done Sent Done false
33299 10230 Acked Read 10 Done Sent Done false
33300 10231 Acked Write 3 Done Sent Done false
33301 10232 Acked Write 5 Done Sent Done false
33302 10233 Acked Write 8 Done Sent Done false
33303 10234 Acked Read 6 Done Sent Done false
33304 10235 Acked Read 1 Done Sent Done false
33305 10236 Acked Read 8 Done Sent Done false
33306 10237 Acked Read 7 Done Sent Done false
33307 10238 Acked Write 6 Done Sent Done false
33308 10239 Acked Write 2 Done Sent Done false
33309 10240 Acked Flush 0 Done Sent Done false
33310 10241 Acked Write 3 Done Sent Done false
33311 10242 Acked Write 4 Done Sent Done false
33312 10243 Acked Write 9 Done Sent Done false
33313 10244 Acked Write 3 Done Sent Done false
33314 10245 Acked Read 6 Done Sent Done false
33315 10246 Acked Write 7 Done Sent Done false
33316 10247 Acked Read 1 Done Sent Done false
33317 10248 Acked Read 6 Done Sent Done false
33318 10249 Acked Read 7 Done Sent Done false
33319 10250 Acked Flush 0 Done Sent Done false
33320 10251 Acked Flush 0 Done Sent Done false
33321 10252 Acked Read 9 Done Sent Done false
33322 10253 Acked Write 10 Done Sent Done false
33323 10254 Acked Read 3 Done Sent Done false
33324 10255 Acked Read 9 Done Sent Done false
33325 10256 Acked Read 8 Done Sent Done false
33326 10257 Acked Flush 0 Done Sent Done false
33327 10258 Acked Write 2 Done Sent Done false
33328 10259 Acked Write 9 Done Sent Done false
33329 10260 Acked Write 3 Done Sent Done false
33330 10261 Acked Write 5 Done Sent Done false
33331 10262 Acked Read 9 Done Sent Done false
33332 10263 Acked Write 9 Done Sent Done false
33333 10264 Acked Read 4 Done Sent Done false
33334 10265 Acked Read 7 Done Sent Done false
33335 10266 Acked Flush 0 Done Sent Done false
33336 10267 Acked Read 10 Done Sent Done false
33337 10268 Acked Flush 0 Done Sent Done false
33338 10269 Acked Write 1 Done Sent Done false
33339 10270 Acked Read 8 Done Sent Done false
33340 10271 Acked Flush 0 Done Sent Done false
33341 10272 Acked Read 7 Done Sent Done false
33342 10273 Acked Read 3 Done Sent Done false
33343 10274 Acked Write 6 Done Sent Done false
33344 10275 Acked Write 2 Done Sent Done false
33345 10276 Acked Read 7 Done Sent Done false
33346 10277 Acked Write 4 Done Sent Done false
33347 10278 Acked Read 9 Done Sent Done false
33348 10279 Acked Write 4 Done Sent Done false
33349 10280 Acked Read 6 Done Sent Done false
33350 10281 Acked Read 4 Done Sent Done false
33351 10282 Acked Read 3 Done Sent Done false
33352 10283 Acked Write 4 Done Sent Done false
33353 10284 Acked Flush 0 Done Sent Done false
33354 10285 Acked Read 1 Done Sent Done false
33355 10286 Acked Write 4 Done Sent Done false
33356 10287 Acked Write 2 Done Sent Done false
33357 10288 Acked Write 8 Done Sent Done false
33358 10289 Acked Write 4 Done Sent Done false
33359 10290 Acked Read 4 Done Sent Done false
33360 10291 Acked Write 5 Done Sent Done false
33361 10292 Acked Read 10 Done Sent Done false
33362 10293 Acked Flush 0 Done Sent Done false
33363 10294 Acked Write 9 Done Sent Done false
33364 10295 Acked Flush 0 Done Sent Done false
33365 10296 Acked Read 6 Done Sent Done false
33366 10297 Acked Write 2 Done Sent Done false
33367 10298 Acked Write 7 Done Sent Done false
33368 10299 Acked Write 2 Done Sent Done false
33369 10300 Acked Write 6 Done Sent Done false
33370 10301 Acked Read 1 Done Sent Done false
33371 10302 Acked Read 1 Done Sent Done false
33372 10303 Acked Flush 0 Done Sent Done false
33373 10304 Acked Write 7 Done Sent Done false
33374 10305 Acked Write 7 Done Sent Done false
33375 10306 Acked Write 10 Done Sent Done false
33376 10307 Acked Read 8 Done Sent Done false
33377 10308 Acked Read 2 Done Sent Done false
33378 10309 Acked Read 7 Done Sent Done false
33379 10310 Acked Write 7 Done Sent Done false
33380 10311 Acked Flush 0 Done Sent Done false
33381 10312 Acked Write 4 Done Sent Done false
33382 10313 Acked Read 3 Done Sent Done false
33383 10314 Acked Read 8 Done Sent Done false
33384 10315 Acked Read 3 Done Sent Done false
33385 10316 Acked Write 7 Done Sent Done false
33386 10317 Acked Write 8 Done Sent Done false
33387 10318 Acked Read 3 Done Sent Done false
33388 10319 Acked Read 10 Done Sent Done false
33389 10320 Acked Read 5 Done Sent Done false
33390 10321 Acked Read 9 Done Sent Done false
33391 10322 Acked Write 5 Done Sent Done false
33392 10323 Acked Write 1 Done Sent Done false
33393 10324 Acked Write 6 Done Sent Done false
33394 10325 Acked Write 1 Done Sent Done false
33395 10326 Acked Read 3 Done Sent Done false
33396 10327 Acked Write 5 Done Sent Done false
33397 10328 Acked Write 1 Done Sent Done false
33398 10329 Acked Read 6 Done Sent Done false
33399 10330 Acked Read 7 Done Sent Done false
33400 10331 Acked Read 2 Done Sent Done false
33401 10332 Acked Read 7 Done Sent Done false
33402 10333 Acked Read 4 Done Sent Done false
33403 10334 Acked Write 6 Done Sent Done false
33404 10335 Acked Write 9 Done Sent Done false
33405 10336 Acked Read 7 Done Sent Done false
33406 10337 Acked Read 1 Done Sent Done false
33407 10338 Acked Read 2 Done Sent Done false
33408 10339 Acked Write 3 Done Sent Done false
33409 10340 Acked Write 10 Done Sent Done false
33410 10341 Acked Write 2 Done Sent Done false
33411 10342 Acked Write 4 Done Sent Done false
33412 10343 Acked Write 6 Done Sent Done false
33413 10344 Acked Read 3 Done Sent Done false
33414 10345 Acked Read 10 Done Sent Done false
33415 10346 Acked Read 8 Done Sent Done false
33416 10347 Acked Flush 0 Done Sent Done false
33417 10348 Acked Write 4 Done Sent Done false
33418 10349 Acked Read 6 Done Sent Done false
33419 10350 Acked Read 2 Done Sent Done false
33420 10351 Acked Read 7 Done Sent Done false
33421 10352 Acked Write 8 Done Sent Done false
33422 10353 Acked Read 8 Done Sent Done false
33423 10354 Acked Read 10 Done Sent Done false
33424 10355 Acked Write 2 Done Sent Done false
33425 10356 Acked Read 4 Done Sent Done false
33426 10357 Acked Flush 0 Done Sent Done false
33427 STATES DS:0 DS:1 DS:2 TOTAL
33428 Sent 0 302 0 302
33429 Done 302 0 302 604
33430 Skipped 0 0 0 0
33431 Error 0 0 0 0
33432 Last Flush: Some(JobId(10357)) Some(JobId(10055)) Some(JobId(10357))
33433 Downstairs last five completed: 10055 10054 10053 10052 10051
33434 Upstairs last five completed: 10357 10356 10355 10354 10353
33435 CLIENT: Up:0 ds:302 act:2
334362025-01-23T18:54:54.439ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
334372025-01-23T18:54:54.439ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55418) = io task client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
334382025-01-23T18:54:54.440ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
334392025-01-23T18:54:54.441ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
334402025-01-23T18:54:54.441ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
334412025-01-23T18:54:54.441ZINFOcrucible: send last flush ID to this DS: Some(JobId(10055)) = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
334422025-01-23T18:54:54.441ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
334432025-01-23T18:54:54.441ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(10055)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
334442025-01-23T18:54:54.441ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(10055)) = downstairs session_id = bb6faa00-ad53-4274-84ae-5c5874c7a798
33445 ----------------------------------------------------------------
33446 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33447 Downstairs last five completed: 10357 10356 10355 10354 10353
33448 Upstairs last five completed: 10357 10356 10355 10354 10353
33449 CLIENT: Up:0 ds:0 act:3
33450 Replay: All jobs finished
33451 Test replay has completed
33452 Wrote out file "/tmp/test_replay_verify.log"
33453 CLIENT: Tests done. All submitted work has been ACK'd
33454 ----------------------------------------------------------------
33455 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33456 Downstairs last five completed: 10357 10356 10355 10354 10353
33457 Upstairs last five completed: 10357 10356 10355 10354 10353
33458 CLIENT: Up:0 ds:0 act:3
33459 CLIENT: All jobs finished, all DS active.
334602025-01-23T18:54:56.695ZWARNcrucible: Guest handle has been dropped
33461 Replays:30 time: 7:00
33462 Do final verify
33463 Jan 23 18:54:56.766 INFO accepted connection, remote_addr: 127.0.0.1:64307, local_addr: 127.0.0.1:9998
33464 Jan 23 18:54:56.767 INFO request completed, latency_us: 181, response_code: 200, uri: /regioninfo, method: GET, req_id: dd9bbcee-72f3-4fda-bd13-7dad1d7795cc, remote_addr: 127.0.0.1:64307, local_addr: 127.0.0.1:9998
334652025-01-23T18:54:56.767ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
33466 Jan 23 18:54:56.768 INFO request completed, latency_us: 87, response_code: 200, uri: /regioncount, method: GET, req_id: 5b71ca32-96c1-4918-bef9-8536b8cd6338, remote_addr: 127.0.0.1:64307, local_addr: 127.0.0.1:9998
334672025-01-23T18:54:56.768ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
33468 Jan 23 18:54:56.768 INFO request completed, latency_us: 102, response_code: 200, uri: /port/cid/0, method: GET, req_id: 1ec5baad-8c1b-4faa-a736-b21ff438a963, remote_addr: 127.0.0.1:64307, local_addr: 127.0.0.1:9998
33469 Jan 23 18:54:56.769 INFO request completed, latency_us: 98, response_code: 200, uri: /port/cid/1, method: GET, req_id: c04e370f-73e4-46d2-a840-5aa45fdbac81, remote_addr: 127.0.0.1:64307, local_addr: 127.0.0.1:9998
33470 Jan 23 18:54:56.769 INFO request completed, latency_us: 99, response_code: 200, uri: /port/cid/2, method: GET, req_id: d5e72f8d-0800-401a-a436-dc5553bf6e9d, remote_addr: 127.0.0.1:64307, local_addr: 127.0.0.1:9998
334712025-01-23T18:54:56.770ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
334722025-01-23T18:54:56.770ZINFOcrucible: Upstairs starts
334732025-01-23T18:54:56.770ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "dd1cf1330bf5c5b1551a0ad961a7aa7c6207feae", git_commit_timestamp: "2025-01-23T18:31:41.000000000Z", git_branch: "mkeeter/early-io-rejection", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
334742025-01-23T18:54:56.770ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
334752025-01-23T18:54:56.770ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
334762025-01-23T18:54:56.770ZINFOcrucible: Crucible 3158e9ee-3aaf-40db-bcee-1aa7a6ba4fb5 has session id: cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334772025-01-23T18:54:56.770ZINFOcrucible: Upstairs opts: Upstairs UUID: 3158e9ee-3aaf-40db-bcee-1aa7a6ba4fb5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334782025-01-23T18:54:56.771ZINFOcrucible: Crucible stats registered with UUID: 3158e9ee-3aaf-40db-bcee-1aa7a6ba4fb5 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334792025-01-23T18:54:56.771ZWARNcrucible: could not find Downstairs address for Nexus session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334802025-01-23T18:54:56.771ZINFOcrucible: Downstairs in volume = 3 task = crutest
334812025-01-23T18:54:56.771ZINFOcrucible: The guest has requested activation with gen:3
334822025-01-23T18:54:56.771ZINFOcrucible: 3158e9ee-3aaf-40db-bcee-1aa7a6ba4fb5 active request set session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334832025-01-23T18:54:56.771ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334842025-01-23T18:54:56.772ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334852025-01-23T18:54:56.772ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334862025-01-23T18:54:56.772ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334872025-01-23T18:54:56.772ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334882025-01-23T18:54:56.772ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334892025-01-23T18:54:56.772ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334902025-01-23T18:54:56.772ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334912025-01-23T18:54:56.772ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334922025-01-23T18:54:56.772ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33856) = io task client = 0 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334932025-01-23T18:54:56.772ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59330) = io task client = 1 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334942025-01-23T18:54:56.772ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58875) = io task client = 2 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334952025-01-23T18:54:56.773ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334962025-01-23T18:54:56.773ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334972025-01-23T18:54:56.773ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334982025-01-23T18:54:56.773ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
334992025-01-23T18:54:56.773ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335002025-01-23T18:54:56.773ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335012025-01-23T18:54:56.774ZINFOcrucible: Waiting for more clients to be ready session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335022025-01-23T18:54:56.774ZINFOcrucible: Waiting for more clients to be ready session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335032025-01-23T18:54:56.774ZINFOcrucible: [0]R flush_numbers[0..12]: [900, 913, 913, 899, 911, 914, 912, 912, 913, 909, 895, 858] = downstairs session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335042025-01-23T18:54:56.774ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335052025-01-23T18:54:56.774ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335062025-01-23T18:54:56.774ZINFOcrucible: [1]R flush_numbers[0..12]: [900, 913, 913, 899, 911, 914, 912, 912, 913, 909, 895, 858] = downstairs session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335072025-01-23T18:54:56.774ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335082025-01-23T18:54:56.774ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335092025-01-23T18:54:56.774ZINFOcrucible: [2]R flush_numbers[0..12]: [900, 913, 913, 899, 911, 914, 912, 912, 913, 909, 895, 858] = downstairs session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335102025-01-23T18:54:56.774ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335112025-01-23T18:54:56.774ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
33512 {"msg":"Max found gen is 3","v":0,"name":"crucible","level":30,"time":"2025-01-23T18:54:56.774750811Z","hostname":"Wait for a query_work_queue command to finish before sending IO
33513 w-01JJA7SGAN5RQ5R76VGZEANENG","pid":1236,"":"downstairs","":"downstairs","session_id":"cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77"}
335142025-01-23T18:54:56.774ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335152025-01-23T18:54:56.774ZINFOcrucible: Next flush: 915 = downstairs session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335162025-01-23T18:54:56.774ZINFOcrucible: All extents match = downstairs session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335172025-01-23T18:54:56.774ZINFOcrucible: No downstairs reconciliation required session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335182025-01-23T18:54:56.774ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335192025-01-23T18:54:56.774ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335202025-01-23T18:54:56.774ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335212025-01-23T18:54:56.774ZINFOcrucible: All required reconciliation work is completed session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335222025-01-23T18:54:56.774ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335232025-01-23T18:54:56.774ZINFOcrucible: 3158e9ee-3aaf-40db-bcee-1aa7a6ba4fb5 is now active with session: cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77 session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335242025-01-23T18:54:56.774ZINFOcrucible: Set Active after no reconciliation session_id = cc5d3e85-5d75-4cd3-bc1e-daaf66dc6f77
335252025-01-23T18:54:56.774ZINFOcrucible: The guest has finished waiting for activation with:3
33526 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
33527 Loading write count information from file "/tmp/test_replay_verify.log"
33528 Read and Verify all blocks (0..5000 range:false)
33529 Wrote out file "/tmp/test_replay_verify.log"
33530 Verify test completed
33531 Wrote out file "/tmp/test_replay_verify.log"
33532 CLIENT: Tests done. All submitted work has been ACK'd
33533 ----------------------------------------------------------------
33534 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
33535 States: Active Active Active
33536 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33537 1000 Acked Read 256 Done Done Done false
33538 1001 Acked Read 256 Done Done Done false
33539 1002 Acked Read 256 Done Done Done false
33540 1003 Acked Read 256 Done Done Done false
33541 1004 Acked Read 256 Done Done Done false
33542 1005 Acked Read 256 Done Done Done false
33543 1006 Acked Read 256 Done Done Done false
33544 1007 Acked Read 256 Done Done Done false
33545 1008 Acked Read 256 Done Done Done false
33546 1009 Acked Read 256 Done Done Done false
33547 1010 Acked Read 256 Done Done Done false
33548 1011 Acked Read 256 Done Done Done false
33549 1012 Acked Read 256 Done Done Done false
33550 1013 Acked Read 256 Done Done Done false
33551 1014 Acked Read 256 Done Done Done false
33552 1015 Acked Read 256 Done Done Done false
33553 1016 Acked Read 256 Done Done Done false
33554 1017 Acked Read 136 Done Done Done false
33555 1018 Acked Read 256 Done Done Done false
33556 1019 Acked Read 256 Done Done Done false
33557 STATES DS:0 DS:1 DS:2 TOTAL
33558 Sent 0 0 0 0
33559 Done 20 20 20 60
33560 Skipped 0 0 0 0
33561 Error 0 0 0 0
33562 Last Flush: None None None
33563 Downstairs last five completed:
33564 Upstairs last five completed: 1019 1018 1017 1016 1015
33565 CLIENT: Up:0 ds:20 act:3
33566 ----------------------------------------------------------------
33567 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
33568 Downstairs last five completed: 1020 1019 1018 1017 1016
33569 Upstairs last five completed: 1020 1019 1018 1017 1016
33570 CLIENT: Up:0 ds:0 act:3
33571 CLIENT: All crucible jobs finished, exiting program
335722025-01-23T18:55:01.190ZWARNcrucible: Guest handle has been dropped
33573 Jan 23 18:55:01.254 INFO accepted connection, remote_addr: 127.0.0.1:47692, local_addr: 127.0.0.1:9998
33574 Main task has work to do, go find it
33575 got dsc Shutdown
33576 Shutdown
33577 Jan 23 18:55:01.255Shut it down
33578 INFO request completed, latency_us: 145, response_code: 204, uri: /shutdown, method[8810] Disable keep_running
33579 : POST, req_id:[8810] Got stop action so:false kr:false
33580 c71c8bdf-b1a3-4ccf-8abd-3a3f3541fe30, remote_addr[8820] Disable keep_running
33581 : 127.0.0.1:47692, local_addr: 127.0.0.1:9998
33582 [8820] Got stop action so:false kr:false
33583 [8830] Disable keep_running
33584 [8830] Got stop action so:false kr:false
33585 [8820] Exited with: ExitStatus(unix_wait_status(9))
33586 [8820][1] reports Exit
33587 [8810] Exited with: ExitStatus(unix_wait_status(9))
33588 [8810][0] reports Exit
33589 [8830] Exited with: ExitStatus(unix_wait_status(9))
33590 [8830][2] reports Exit
33591 January 23, 2025 at 06:55:06 PM UTC Test ends with 0