Max level shown:
1 starting November 1, 2024 at 04:50:25 PM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.33889037
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.059087228
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.059410825
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start ds: 8820
14 start ds: 8830
15 start access at:127.0.0.1:9998
16 Starting downstairs at port 8810
17 Starting downstairs at port 8820
18 Make output file at "/tmp/dsc/downstairs-8820.txt"
19 Starting downstairs at port 8830
20 Make output file at "/tmp/dsc/downstairs-8810.txt"
21 Make output file at "/tmp/dsc/downstairs-8830.txt"
22 [8810][0] initial start wait reports Starting
23 [8820][1] initial start wait reports Starting
24 [8830][2] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1188)
26 [8830][2] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1189)
28 [8810][0] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1190)
30 [8820][1] initial start wait reports Running
31 All downstairs are running
32 Nov 01 16:50:26.385 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Nov 01 16:50:31.485 INFO accepted connection, remote_addr: 127.0.0.1:63905, local_addr: 127.0.0.1:9998
36 Nov 01 16:50:31.488 INFO request completed, latency_us: 649, response_code: 200, uri: /regioninfo, method: GET, req_id: aced01be-78b6-46f8-8b63-bea0b797408c, remote_addr: 127.0.0.1:63905, local_addr: 127.0.0.1:9998
37 Nov 01 16:50:31.490 INFO request completed, latency_us: 105, response_code: 200, uri: /regioncount, method: GET, req_id: c004e879-4c50-4deb-83e0-8b1ee9f45f9f, remote_addr: 127.0.0.1:63905, local_addr: 127.0.0.1:9998
382024-11-01T16:50:31.490ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392024-11-01T16:50:31.491ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Nov 01 16:50:31.492 INFO request completed, latency_us: 146, response_code: 200, uri: /port/cid/0, method: GET, req_id: 531ce1ec-a617-4312-9c77-804fee725e46, remote_addr: 127.0.0.1:63905, local_addr: 127.0.0.1:9998
41 Nov 01 16:50:31.492 INFO request completed, latency_us: 92, response_code: 200, uri: /port/cid/1, method: GET, req_id: 8a60b8ec-555e-42ad-bc85-c2023acbe282, remote_addr: 127.0.0.1:63905, local_addr: 127.0.0.1:9998
42 Nov 01 16:50:31.493 INFO request completed, latency_us: 96, response_code: 200, uri: /port/cid/2, method: GET, req_id: daa93d96-3367-4b7a-b173-f65f140d580c, remote_addr: 127.0.0.1:63905, local_addr: 127.0.0.1:9998
432024-11-01T16:50:31.493ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442024-11-01T16:50:31.494ZINFOcrucible: Upstairs starts
452024-11-01T16:50:31.494ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "3818f0fc76fe787c6cc412af30fd59b930203885", git_commit_timestamp: "2024-11-01T16:08:30.000000000Z", git_branch: "renovate/hyper-1.x-lockfile", rustc_semver: "1.82.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "f6e511eec7342f59a25f7c0534f1dbea00d01b14", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462024-11-01T16:50:31.494ZINFOcrucible: Upstairs <-> Downstairs Message Version: 12
472024-11-01T16:50:31.494ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482024-11-01T16:50:31.495ZINFOcrucible: Crucible 3ccccf96-e798-45a6-8faf-b55088fce69a has session id: 74c7df72-98c2-48ff-a60d-99a5ca59575e session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
492024-11-01T16:50:31.495ZINFOcrucible: Upstairs opts: Upstairs UUID: 3ccccf96-e798-45a6-8faf-b55088fce69a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
502024-11-01T16:50:31.495ZINFOcrucible: Crucible stats registered with UUID: 3ccccf96-e798-45a6-8faf-b55088fce69a session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
512024-11-01T16:50:31.530ZINFOcrucible: Downstairs in volume = 3 task = crutest
522024-11-01T16:50:31.530ZINFOcrucible: The guest has requested activation with gen:1
532024-11-01T16:50:31.530ZINFOcrucible: 3ccccf96-e798-45a6-8faf-b55088fce69a active request set session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
542024-11-01T16:50:31.530ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
552024-11-01T16:50:31.530ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
562024-11-01T16:50:31.531ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
572024-11-01T16:50:31.531ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
582024-11-01T16:50:31.531ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
592024-11-01T16:50:31.531ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
602024-11-01T16:50:31.531ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
612024-11-01T16:50:31.531ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
622024-11-01T16:50:31.531ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
632024-11-01T16:50:31.531ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38925) = io task client = 1 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
642024-11-01T16:50:31.531ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:57283) = io task client = 0 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
652024-11-01T16:50:31.531ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52290) = io task client = 2 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
662024-11-01T16:50:31.532ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
672024-11-01T16:50:31.532ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
682024-11-01T16:50:31.532ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
692024-11-01T16:50:31.532ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
702024-11-01T16:50:31.532ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
712024-11-01T16:50:31.532ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
722024-11-01T16:50:31.534ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
732024-11-01T16:50:31.534ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
742024-11-01T16:50:31.534ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
752024-11-01T16:50:31.535ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
762024-11-01T16:50:31.535ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
772024-11-01T16:50:31.535ZINFOcrucible: Waiting for more clients to be ready session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
782024-11-01T16:50:31.535ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
792024-11-01T16:50:31.535ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
802024-11-01T16:50:31.535ZINFOcrucible: Waiting for more clients to be ready session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
812024-11-01T16:50:31.535ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
822024-11-01T16:50:31.535ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
832024-11-01T16:50:31.535ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
842024-11-01T16:50:31.535ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
852024-11-01T16:50:31.535ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
862024-11-01T16:50:31.535ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
872024-11-01T16:50:31.535ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
882024-11-01T16:50:31.535ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
892024-11-01T16:50:31.535ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
902024-11-01T16:50:31.535ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
912024-11-01T16:50:31.535ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
922024-11-01T16:50:31.535ZINFOcrucible: Max found gen is 1 = downstairs session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
932024-11-01T16:50:31.535ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
942024-11-01T16:50:31.535ZINFOcrucible: Next flush: 1 = downstairs session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
952024-11-01T16:50:31.536ZINFOcrucible: All extents match = downstairs session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
962024-11-01T16:50:31.536ZINFOcrucible: No downstairs reconciliation required session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
972024-11-01T16:50:31.536ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
982024-11-01T16:50:31.536ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
992024-11-01T16:50:31.536ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
1002024-11-01T16:50:31.536ZINFOcrucible: All required reconciliation work is completed session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
1012024-11-01T16:50:31.536ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
102 {"msg":"3ccccf96-e798-45a6-8faf-b55088fce69a is now active with session: 74c7df72-98c2-48ff-a60d-99a5ca59575e","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:31.536458386Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1194,"Activate successful
103 session_idWait for a query_work_queue command to finish before sending IO
104 ":"74c7df72-98c2-48ff-a60d-99a5ca59575e"}
1052024-11-01T16:50:31.536ZINFOcrucible: Set Active after no reconciliation session_id = 74c7df72-98c2-48ff-a60d-99a5ca59575e
1062024-11-01T16:50:31.536ZINFOcrucible: The guest has finished waiting for activation with:1
107 Region: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
108 Fill test
109 Read and Verify all blocks (0..5000 range:false)
110 Wrote out file "/tmp/test_replay_verify.log"
111 CLIENT: Tests done. All submitted work has been ACK'd
112 ----------------------------------------------------------------
113 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
114 Downstairs last five completed: 1041 1040 1039 1038 1037
115 Upstairs last five completed: 1041 1040 1039 1038 1037
116 CLIENT: Up:0 ds:0 act:3
117 CLIENT: All crucible jobs finished, exiting program
118 Replay loop starts now November 1, 2024 at 04:50:32 PM UTC
119 Nov 01 16:50:32.179 INFO accepted connection, remote_addr: 127.0.0.1:40299, local_addr: 127.0.0.1:9998
120 Nov 01 16:50:32.180 INFO request completed, latency_us: 175, response_code: 200, uri: /regioninfo, method: GET, req_id: e7fd9c2a-5adf-4c31-ae4d-edb937f221b5, remote_addr: 127.0.0.1:40299, local_addr: 127.0.0.1:9998
1212024-11-01T16:50:32.180ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
122 Nov 01 16:50:32.181 INFO request completed, latency_us: 105, response_code: 200, uri: /regioncount, method: GET, req_id: 9cf2bccc-f399-4139-b648-dea116050356, remote_addr: 127.0.0.1:40299, local_addr: 127.0.0.1:9998
1232024-11-01T16:50:32.181ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
124 Nov 01 16:50:32.181 INFO request completed, latency_us: 121, response_code: 200, uri: /port/cid/0, method: GET, req_id: 2b4df718-6a29-4a7b-89ce-e4151e328c8d, remote_addr: 127.0.0.1:40299, local_addr: 127.0.0.1:9998
125 Nov 01 16:50:32.182 INFO request completed, latency_us: 94, response_code: 200, uri: /port/cid/1, method: GET, req_id: 1aad7185-6c0e-4bc7-94b4-a1b19416d7a8, remote_addr: 127.0.0.1:40299, local_addr: 127.0.0.1:9998
126 Nov 01 16:50:32.182 INFO request completed, latency_us: 92, response_code: 200, uri: /port/cid/2, method: GET, req_id: 7437aa94-ca82-42cc-8cfb-afcb239a5047, remote_addr: 127.0.0.1:40299, local_addr: 127.0.0.1:9998
1272024-11-01T16:50:32.183ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1282024-11-01T16:50:32.183ZINFOcrucible: Upstairs starts
1292024-11-01T16:50:32.183ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "3818f0fc76fe787c6cc412af30fd59b930203885", git_commit_timestamp: "2024-11-01T16:08:30.000000000Z", git_branch: "renovate/hyper-1.x-lockfile", rustc_semver: "1.82.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "f6e511eec7342f59a25f7c0534f1dbea00d01b14", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1302024-11-01T16:50:32.183ZINFOcrucible: Upstairs <-> Downstairs Message Version: 12
1312024-11-01T16:50:32.183ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1322024-11-01T16:50:32.183ZINFOcrucible: Crucible 6ec10072-262a-40ad-a7fe-dd8f2399e051 has session id: 2507b7e9-146d-454e-b579-e099d8c24c78 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1332024-11-01T16:50:32.183ZINFOcrucible: Upstairs opts: Upstairs UUID: 6ec10072-262a-40ad-a7fe-dd8f2399e051, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1342024-11-01T16:50:32.183ZINFOcrucible: Crucible stats registered with UUID: 6ec10072-262a-40ad-a7fe-dd8f2399e051 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1352024-11-01T16:50:32.219ZINFOcrucible: Downstairs in volume = 3 task = crutest
1362024-11-01T16:50:32.219ZINFOcrucible: The guest has requested activation with gen:2
1372024-11-01T16:50:32.219ZINFOcrucible: 6ec10072-262a-40ad-a7fe-dd8f2399e051 active request set session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1382024-11-01T16:50:32.219ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1392024-11-01T16:50:32.219ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1402024-11-01T16:50:32.219ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1412024-11-01T16:50:32.219ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1422024-11-01T16:50:32.219ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1432024-11-01T16:50:32.219ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1442024-11-01T16:50:32.219ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1452024-11-01T16:50:32.219ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1462024-11-01T16:50:32.219ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1472024-11-01T16:50:32.219ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56940) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1482024-11-01T16:50:32.219ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46761) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1492024-11-01T16:50:32.219ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63649) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1502024-11-01T16:50:32.220ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1512024-11-01T16:50:32.220ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1522024-11-01T16:50:32.220ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1532024-11-01T16:50:32.220ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1542024-11-01T16:50:32.220ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1552024-11-01T16:50:32.220ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1562024-11-01T16:50:32.221ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1572024-11-01T16:50:32.221ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1582024-11-01T16:50:32.221ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1592024-11-01T16:50:32.221ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1602024-11-01T16:50:32.221ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1612024-11-01T16:50:32.221ZINFOcrucible: Waiting for more clients to be ready session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1622024-11-01T16:50:32.221ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1632024-11-01T16:50:32.221ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1642024-11-01T16:50:32.221ZINFOcrucible: Waiting for more clients to be ready session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1652024-11-01T16:50:32.221ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1662024-11-01T16:50:32.222ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1672024-11-01T16:50:32.222ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1682024-11-01T16:50:32.222ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1692024-11-01T16:50:32.222ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1702024-11-01T16:50:32.222ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1712024-11-01T16:50:32.222ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1722024-11-01T16:50:32.222ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1732024-11-01T16:50:32.222ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1742024-11-01T16:50:32.222ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
175 {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Activate successful
176 ,Wait for a query_work_queue command to finish before sending IO
177 "time":"2024-11-01T16:50:32.222268815Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"downstairs","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
1782024-11-01T16:50:32.222ZINFOcrucible: Max found gen is 2 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1792024-11-01T16:50:32.222ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1802024-11-01T16:50:32.222ZINFOcrucible: Next flush: 2 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1812024-11-01T16:50:32.222ZINFOcrucible: All extents match = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1822024-11-01T16:50:32.222ZINFOcrucible: No downstairs reconciliation required session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1832024-11-01T16:50:32.222ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
184 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Region: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
185 ,"time":"2024-11-01T16:50:32.222470962Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"client":"1","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
1862024-11-01T16:50:32.222ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1872024-11-01T16:50:32.222ZINFOcrucible: All required reconciliation work is completed session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1882024-11-01T16:50:32.222ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1892024-11-01T16:50:32.222ZINFOcrucible: 6ec10072-262a-40ad-a7fe-dd8f2399e051 is now active with session: 2507b7e9-146d-454e-b579-e099d8c24c78 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1902024-11-01T16:50:32.222ZINFOcrucible: Set Active after no reconciliation session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1912024-11-01T16:50:32.222ZINFOcrucible: The guest has finished waiting for activation with:2
192 Loading write count information from file "/tmp/test_replay_verify.log"
193 Nov 01 16:50:32.257 INFO accepted connection, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
194 Main task has work to do, go find it
195 got dsc Stop(2)
196 stop 2
197 Nov 01 16:50:32.257 INFO request completed, latency_us: 223, response_code: 204, uri: /stop/cid/2[8830] Got stop action so:false kr:false
198 , method: POST, req_id: fa370cd8-bc2f-4f65-93f6-2bc64c2858cd, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
1992024-11-01T16:50:32.258ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
200 Nov 01 16:50:32.258 INFO request completed, latency_us: 114, response_code: 200, uri: /state/cid/2, method: GET, req_id:{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:50:32.25875515Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
201 d631710d-5e0c-4277-b165-bcc72cc58e80, remote_addr: 127.{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:50:32.258784886Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
202 0.0.1:58552, local_addr: 127.0.0.1:9998
2032024-11-01T16:50:32.258ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
2042024-11-01T16:50:32.258ZINFOcrucible: client task is exiting = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
205 [8830] Exited with: ExitStatus(unix_wait_status(9))
206 [8830][2] reports Exit
207 Nov 01 16:50:36.260 INFO request completed, latency_us: 230, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1f8358f3-b053-4948-bab8-cc0fd6dbf59f, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
208 001/300 Read block 2281 len 40960
209 002/300 Flush
210 003/300 Read block 3233 len 28672
211 004/300 Write block 2253 len 20480 data: 2 2 2 2 2
212 005/300 Read block 1727 len 20480
213 006/300 Read block 577 len 4096
214 007/300 Read block 4846 len 24576
215 008/300 Read block 2076 len 24576
216 009/300 Flush
217 010/300 Read block 2414 len 36864
218 011/300 Read block 3860 len 16384
219 012/300 Read block 4899 len 4096
220 013/300 Read block 371 len 12288
221 014/300 Read block 4394 len 36864
222 015/300 Read block 3696 len 40960
223 016/300 Read block 783 len 32768
224 017/300 Read block 3383 len 40960
225 018/300 Flush
226 019/300 Write block 3655 len 24576 data: 2 2 2 2 2 2
227 020/300 Read block 2830 len 4096
228 021/300 Write block 3668 len 28672 data: 2 2 2 2 2 2 2
229 022/300 Read block 4777 len 8192
230 023/300 Read block 3371 len 40960
231 024/300 Read block 4022 len 36864
232 025/300 Write block 1967 len 36864 data: 2 2 2 2 2 2 2 2 2
233 026/300 Read block 2577 len 40960
234 027/300 Read block 4557 len 36864
235 028/300 Read block 3121 len 40960
236 029/300 Read block 4431 len 28672
237 030/300 Read block 669 len 28672
238 031/300 Write block 3866 len 36864 data: 2 2 2 2 2 2 2 2 2
239 032/300 Write block 2491 len 16384 data: 2 2 2 2
240 033/300 Read block 36 len 32768
241 034/300 Read block 1135 len 36864
242 035/300 Read block 3455 len 32768
243 036/300 Write block 687 len 4096 data: 2
244 037/300 Read block 2775 len 40960
245 038/300 Flush
246 039/300 Write block 2054 len 40960 data: 2 2 2 2 2 2 2 2 2 2
247 040/300 Read block 113 len 12288
248 041/300 Write block 2981 len 20480 data: 2 2 2 2 2
249 042/300 Read block 877 len 24576
250 043/300 Write block 2811 len 24576 data: 2 2 2 2 2 2
251 044/300 Flush
252 045/300 Read block 3983 len 20480
253 046/300 Read block 2052 len 8192
254 047/300 Write block 406 len 4096 data: 2
255 048/300 Read block 4101 len 16384
256 049/300 Write block 1293 len 40960 data: 2 2 2 2 2 2 2 2 2 2
257 050/300 Read block 1944 len 12288
258 051/300 Read block 3083 len 40960
259 052/300 Write block 4320 len 16384 data: 2 2 2 2
260 053/300 Write block 844 len 36864 data: 2 2 2 2 2 2 2 2 2
261 054/300 Read block 1714 len 20480
262 055/300 Read block 1211 len 40960
263 056/300 Write block 2304 len 4096 data: 2
264 057/300 Write block 1985 len 20480 data: 2 2 2 2 2
265 058/300 Read block 4430 len 16384
266 059/300 Write block 492 len 40960 data: 2 2 2 2 2 2 2 2 2 2
267 060/300 Write block 3259 len 24576 data: 2 2 2 2 2 2
268 061/300 Write block 3703 len 40960 data: 2 2 2 2 2 2 2 2 2 2
269 062/300 Write block 2496 len 16384 data: 2 2 2 2
270 063/300 Read block 4966 len 8192
271 064/300 Read block 2676 len 32768
272 065/300 Write block 4116 len 20480 data: 2 2 2 2 2
273 066/300 Read block 2634 len 4096
274 067/300 Write block 1431 len 32768 data: 2 2 2 2 2 2 2 2
275 068/300 Write block 4085 len 24576 data: 2 2 2 2 2 2
276 069/300 Read block 717 len 32768
277 070/300 Read block 2917 len 28672
278 071/300 Read block 364 len 32768
279 072/300 Read block 1922 len 36864
280 073/300 Write block 4916 len 12288 data: 2 2 2
281 074/300 Read block 3306 len 4096
282 075/300 Write block 290 len 8192 data: 2 2
283 076/300 Read block 4128 len 32768
284 077/300 Write block 3223 len 28672 data: 2 2 2 2 2 2 2
285 078/300 Write block 3830 len 32768 data: 2 2 2 2 2 2 2 2
286 079/300 Flush
287 080/300 Write block 1790 len 8192 data: 2 2
288 081/300 Write block 2475 len 24576 data: 2 2 2 2 2 2
289 082/300 Read block 99 len 20480
290 083/300 Read block 780 len 8192
291 084/300 Read block 3521 len 24576
292 085/300 Write block 982 len 8192 data: 2 2
293 086/300 Read block 2249 len 36864
294 087/300 Write block 4673 len 20480 data: 2 2 2 2 2
295 088/300 Read block 620 len 28672
296 089/300 Read block 2605 len 36864
297 090/300 Read block 1534 len 20480
298 091/300 Read block 724 len 28672
299 092/300 Write block 1294 len 28672 data: 3 3 3 3 3 3 3
300 093/300 Read block 4808 len 36864
301 094/300 Write block 2924 len 36864 data: 2 2 2 2 2 2 2 2 2
302 095/300 Write block 3204 len 16384 data: 2 2 2 2
303 096/300 Write block 1381 len 12288 data: 2 2 2
304 097/300 Read block 695 len 16384
305 098/300 Read block 3301 len 16384
306 099/300 Write block 3026 len 36864 data: 2 2 2 2 2 2 2 2 2
307 100/300 Write block 4484 len 20480 data: 2 2 2 2 2
308 101/300 Read block 4592 len 16384
309 102/300 Write block 2846 len 16384 data: 2 2 2 2
310 103/300 Flush
311 104/300 Read block 1649 len 36864
312 105/300 Flush
313 106/300 Read block 3129 len 40960
314 107/300 Read block 2845 len 36864
315 108/300 Read block 1917 len 36864
316 109/300 Write block 3734 len 32768 data: 2 2 2 2 2 2 2 2
317 110/300 Read block 3540 len 32768
318 111/300 Write block 1151 len 36864 data: 2 2 2 2 2 2 2 2 2
319 112/300 Write block 3630 len 12288 data: 2 2 2
320 113/300 Write block 727 len 28672 data: 2 2 2 2 2 2 2
321 114/300 Write block 1074 len 36864 data: 2 2 2 2 2 2 2 2 2
322 115/300 Flush
323 116/300 Write block 2798 len 8192 data: 2 2
324 117/300 Write block 739 len 12288 data: 2 2 2
325 118/300 Write block 1018 len 4096 data: 2
326 119/300 Read block 3649 len 20480
327 120/300 Flush
328 121/300 Read block 76 len 4096
329 122/300 Read block 664 len 8192
330 123/300 Flush
331 124/300 Flush
332 125/300 Read block 381 len 28672
333 126/300 Write block 1964 len 40960 data: 2 2 2 3 3 3 3 3 3 3
334 127/300 Write block 607 len 28672 data: 2 2 2 2 2 2 2
335 128/300 Flush
336 129/300 Read block 703 len 20480
337 130/300 Write block 516 len 32768 data: 2 2 2 2 2 2 2 2
338 131/300 Write block 716 len 12288 data: 2 2 2
339 132/300 Write block 2086 len 4096 data: 2
340 133/300 Read block 4997 len 12288
341 134/300 Read block 257 len 20480
342 135/300 Write block 807 len 28672 data: 2 2 2 2 2 2 2
343 136/300 Flush
344 137/300 Write block 3825 len 16384 data: 2 2 2 2
345 138/300 Write block 1879 len 24576 data: 2 2 2 2 2 2
346 139/300 Read block 4945 len 36864
347 140/300 Read block 1750 len 36864
348 141/300 Read block 1115 len 4096
349 142/300 Flush
350 143/300 Read block 3862 len 40960
351 144/300 Read block 2704 len 8192
352 145/300 Read block 4955 len 20480
353 146/300 Read block 4954 len 40960
354 147/300 Write block 1237 len 28672 data: 2 2 2 2 2 2 2
355 148/300 Read block 2693 len 20480
356 149/300 Write block 3421 len 12288 data: 2 2 2
357 150/300 Read block 1959 len 28672
358 151/300 Read block 1576 len 24576
359 152/300 Read block 3679 len 16384
360 153/300 Flush
361 154/300 Write block 3076 len 40960 data: 2 2 2 2 2 2 2 2 2 2
362 155/300 Write block 3946 len 32768 data: 2 2 2 2 2 2 2 2
363 156/300 Flush
364 157/300 Read block 1125 len 8192
365 158/300 Write block 4905 len 8192 data: 2 2
366 159/300 Read block 2227 len 36864
367 160/300 Write block 2641 len 40960 data: 2 2 2 2 2 2 2 2 2 2
368 161/300 Write block 4828 len 24576 data: 2 2 2 2 2 2
369 162/300 Read block 924 len 8192
370 163/300 Read block 4474 len 28672
371 164/300 Read block 1798 len 28672
372 165/300 Read block 1558 len 16384
373 166/300 Read block 1375 len 8192
374 167/300 Flush
375 168/300 Read block 4102 len 4096
376 169/300 Read block 3981 len 12288
377 170/300 Read block 4868 len 28672
378 171/300 Write block 2003 len 24576 data: 2 2 2 2 2 2
379 172/300 Write block 549 len 36864 data: 2 2 2 2 2 2 2 2 2
380 173/300 Read block 2593 len 4096
381 174/300 Flush
382 175/300 Write block 3040 len 16384 data: 2 2 2 2
383 176/300 Write block 831 len 8192 data: 2 2
384 177/300 Read block 3061 len 8192
385 178/300 Write block 4017 len 8192 data: 2 2
386 179/300 Write block 4118 len 4096 data: 3
387 180/300 Read block 1871 len 8192
388 181/300 Read block 3011 len 28672
389 182/300 Write block 4185 len 12288 data: 2 2 2
390 183/300 Read block 4041 len 8192
391 184/300 Read block 3070 len 24576
392 185/300 Write block 2710 len 24576 data: 2 2 2 2 2 2
393 186/300 Read block 1661 len 16384
394 187/300 Flush
395 188/300 Read block 2483 len 16384
396 189/300 Read block 4008 len 28672
397 190/300 Write block 2030 len 24576 data: 2 2 2 2 2 2
398 191/300 Read block 2900 len 12288
399 192/300 Read block 1642 len 24576
400 193/300 Read block 763 len 28672
401 194/300 Read block 3349 len 16384
402 195/300 Write block 1720 len 4096 data: 2
403 196/300 Read block 3347 len 4096
404 197/300 Read block 4425 len 4096
405 198/300 Write block 197 len 12288 data: 2 2 2
406 199/300 Read block 1083 len 28672
407 200/300 Read block 4041 len 8192
408 201/300 Read block 927 len 4096
409 202/300 Write block 120 len 32768 data: 2 2 2 2 2 2 2 2
410 203/300 Read block 3322 len 24576
411 204/300 Flush
412 205/300 Write block 799 len 20480 data: 2 2 2 2 2
413 206/300 Read block 1560 len 20480
414 207/300 Flush
415 208/300 Write block 4130 len 8192 data: 2 2
416 209/300 Read block 1760 len 36864
417 210/300 Read block 1476 len 4096
418 211/300 Write block 1872 len 8192 data: 2 2
419 212/300 Read block 748 len 16384
420 213/300 Read block 3340 len 8192
421 214/300 Write block 2954 len 12288 data: 2 2 2
422 215/300 Read block 647 len 28672
423 216/300 Write block 4035 len 4096 data: 2
424 217/300 Write block 2998 len 16384 data: 2 2 2 2
425 218/300 Read block 2445 len 40960
426 219/300 Write block 3201 len 4096 data: 2
427 220/300 Read block 4505 len 8192
428 221/300 Read block 1551 len 4096
429 222/300 Write block 4051 len 32768 data: 2 2 2 2 2 2 2 2
430 223/300 Write block 970 len 16384 data: 2 2 2 2
431 224/300 Flush
432 225/300 Read block 4086 len 28672
433 226/300 Flush
434 227/300 Write block 1917 len 8192 data: 2 2
435 228/300 Read block 1265 len 32768
436 229/300 Read block 1212 len 36864
437 230/300 Read block 4112 len 4096
438 231/300 Read block 2029 len 28672
439 232/300 Read block 3508 len 12288
440 233/300 Read block 3349 len 24576
441 234/300 Read block 2988 len 28672
442 235/300 Write block 1300 len 16384 data: 4 3 3 2
443 236/300 Write block 4073 len 8192 data: 2 2
444 237/300 Read block 2080 len 20480
445 238/300 Read block 1112 len 12288
446 239/300 Write block 1342 len 20480 data: 2 2 2 2 2
447 240/300 Write block 2485 len 20480 data: 2 2 2 2 2
448 241/300 Read block 2419 len 24576
449 242/300 Write block 4990 len 40960 data: 2 2 2 2 2 2 2 2 2 2
450 243/300 Read block 2290 len 4096
451 244/300 Write block 3737 len 16384 data: 3 3 3 3
452 245/300 Read block 3821 len 4096
453 246/300 Write block 246 len 16384 data: 2 2 2 2
454 247/300 Write block 3245 len 24576 data: 2 2 2 2 2 2
455 248/300 Write block 1689 len 28672 data: 2 2 2 2 2 2 2
456 249/300 Write block 3208 len 4096 data: 2
457 250/300 Read block 3396 len 36864
458 251/300 Flush
459 252/300 Write block 858 len 12288 data: 2 2 2
460 253/300 Write block 2677 len 28672 data: 2 2 2 2 2 2 2
461 254/300 Write block 3833 len 20480 data: 3 3 3 3 3
462 255/300 Read block 1555 len 20480
463 256/300 Read block 1667 len 40960
464 257/300 Read block 1881 len 28672
465 258/300 Read block 4508 len 20480
466 259/300 Write block 1922 len 28672 data: 2 2 2 2 2 2 2
467 260/300 Write block 1476 len 16384 data: 2 2 2 2
468 261/300 Write block 1085 len 24576 data: 2 2 2 2 2 2
469 262/300 Read block 3333 len 36864
470 263/300 Write block 4921 len 28672 data: 2 2 2 2 2 2 2
471 264/300 Read block 3596 len 8192
472 265/300 Write block 4663 len 8192 data: 2 2
473 266/300 Write block 1008 len 16384 data: 2 2 2 2
474 267/300 Write block 3242 len 40960 data: 2 2 2 3 3 3 3 3 3 2
475 268/300 Write block 3134 len 12288 data: 2 2 2
476 269/300 Read block 673 len 20480
477 270/300 Write block 3581 len 32768 data: 2 2 2 2 2 2 2 2
478 271/300 Read block 3996 len 36864
479 272/300 Read block 3293 len 16384
480 273/300 Write block 4220 len 12288 data: 2 2 2
481 274/300 Write block 490 len 20480 data: 2 2 3 3 3
482 275/300 Write block 771 len 32768 data: 2 2 2 2 2 2 2 2
483 276/300 Read block 1359 len 12288
484 277/300 Write block 3184 len 20480 data: 2 2 2 2 2
485 278/300 Read block 3861 len 40960
486 279/300 Read block 2901 len 4096
487 280/300 Flush
488 281/300 Write block 1715 len 36864 data: 2 2 2 2 2 3 2 2 2
489 282/300 Write block 4988 len 12288 data: 2 2 3
490 283/300 Flush
491 284/300 Write block 2244 len 20480 data: 2 2 2 2 2
492 285/300 Read block 239 len 4096
493 286/300 Read block 546 len 36864
494 287/300 Read block 1669 len 28672
495 288/300 Read block 4310 len 4096
496 289/300 Write block 2284 len 4096 data: 2
497 290/300 Read block 4000 len 20480
498 291/300 Read block 2231 len 8192
499 292/300 Read block 4628 len 36864
500 293/300 Read block 1711 len 36864
501 294/300 Read block 4571 len 8192
502 295/300 Write block 3444 len 32768 data: 2 2 2 2 2 2 2 2
503 296/300 Flush
504 297/300 Write block 4940 len 20480 data: 2 2 2 2 2
505 298/300 Write block 510 len 20480 data: 2 2 2 2 2
506 299/300 Write block 750 len 40960 data: 2 2 2 2 2 2 2 2 2 2
507 300/300 Read block 1916 len 20480
508 301/300 Read block 2154 len 4096
509 Main task has work to do, go find it
510 got dsc Start(2)
511 start 2
512 Nov 01 16:50:36.555 INFO request completed, latency_us: 240, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1741e32b-13cf-4c81-afe9-16eb796221d2, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
513 Replay: started 2, returned:Ok(())
514 ----------------------------------------------------------------
515 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
516 States: Active Active Offline
517 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
518 1000 Acked Read 10 Done Done Sent false
519 1001 Acked Flush 0 Done Done Sent false
520 1002 Acked Read 7 Done Done Sent false
521 1003 Acked Write 5 Done Done Sent false
522 1004 Acked Read 5 Done Done Sent false
523 1005 Acked Read 1 Done Done Sent false
524 1006 Acked Read 6 Done Done Sent false
525 1007 Acked Read 6 Done Done Sent false
526 1008 Acked Flush 0 Done Done Sent false
527 1009 Acked Read 9 Done Done Sent false
528 1010 Acked Read 4 Done Done Sent false
529 1011 Acked Read 1 Done Done Sent false
530 1012 Acked Read 3 Done Done Sent false
531 1013 Acked Read 9 Done Done Sent false
532 1014 Acked Read 10 Done Done Sent false
533 1015 Acked Read 8 Done Done Sent false
534 1016 Acked Read 10 Done Done Sent false
535 1017 Acked Flush 0 Done Done Sent false
536 1018 Acked Write 6 Done Done Sent false
537 1019 Acked Read 1 Done Done Sent false
538 1020 Acked Write 7 Done Done Sent false
539 1021 Acked Read 2 Done Done Sent false
540 1022 Acked Read 10 Done Done Sent false
541 1023 Acked Read 9 Done Done Sent false
542 1024 Acked Write 9 Done Done Sent false
543 1025 Acked Read 10 Done Done Sent false
544 1026 Acked Read 9 Done Done Sent false
545 1027 Acked Read 10 Done Done Sent false
546 1028 Acked Read 7 Done Done Sent false
547 1029 Acked Read 7 Done Done Sent false
548 1030 Acked Write 9 Done Done Sent false
549 1031 Acked Write 4 Done Done Sent false
550 1032 Acked Read 8 Done Done Sent false
551 1033 Acked Read 9 Done Done Sent false
552 1034 Acked Read 8 Done Done Sent false
553 1035 Acked Write 1 Done Done Sent false
554 1036 Acked Read 10 Done Done Sent false
555 1037 Acked Flush 0 Done Done Sent false
556 1038 Acked Write 10 Done Done Sent false
557 1039 Acked Read 3 Done Done Sent false
558 1040 Acked Write 5 Done Done Sent false
559 1041 Acked Read 6 Done Done Sent false
560 1042 Acked Write 6 Done Done Sent false
561 1043 Acked Flush 0 Done Done Sent false
562 1044 Acked Read 5 Done Done Sent false
563 1045 Acked Read 2 Done Done Sent false
564 1046 Acked Write 1 Done Done Sent false
565 1047 Acked Read 4 Done Done Sent false
566 1048 Acked Write 10 Done Done Sent false
567 1049 Acked Read 3 Done Done Sent false
568 1050 Acked Read 10 Done Done Sent false
569 1051 Acked Write 4 Done Done Sent false
570 1052 Acked Write 9 Done Done Sent false
571 1053 Acked Read 5 Done Done Sent false
572 1054 Acked Read 10 Done Done Sent false
573 1055 Acked Write 1 Done Done Sent false
574 1056 Acked Write 5 Done Done Sent false
575 1057 Acked Read 4 Done Done Sent false
576 1058 Acked Write 10 Done Done Sent false
577 1059 Acked Write 6 Done Done Sent false
578 1060 Acked Write 10 Done Done Sent false
579 1061 Acked Write 4 Done Done Sent false
580 1062 Acked Read 2 Done Done Sent false
581 1063 Acked Read 8 Done Done Sent false
582 1064 Acked Write 5 Done Done Sent false
583 1065 Acked Read 1 Done Done Sent false
584 1066 Acked Write 8 Done Done Sent false
585 1067 Acked Write 6 Done Done Sent false
586 1068 Acked Read 8 Done Done Sent false
587 1069 Acked Read 7 Done Done Sent false
588 1070 Acked Read 8 Done Done Sent false
589 1071 Acked Read 9 Done Done Sent false
590 1072 Acked Write 3 Done Done Sent false
591 1073 Acked Read 1 Done Done Sent false
592 1074 Acked Write 2 Done Done Sent false
593 1075 Acked Read 8 Done Done Sent false
594 1076 Acked Write 7 Done Done Sent false
595 1077 Acked Write 8 Done Done Sent false
596 1078 Acked Flush 0 Done Done Sent false
597 1079 Acked Write 2 Done Done Sent false
598 1080 Acked Write 6 Done Done Sent false
599 1081 Acked Read 5 Done Done Sent false
600 1082 Acked Read 2 Done Done Sent false
601 1083 Acked Read 6 Done Done Sent false
602 1084 Acked Write 2 Done Done Sent false
603 1085 Acked Read 9 Done Done Sent false
604 1086 Acked Write 5 Done Done Sent false
605 1087 Acked Read 7 Done Done Sent false
606 1088 Acked Read 9 Done Done Sent false
607 1089 Acked Read 5 Done Done Sent false
608 1090 Acked Read 7 Done Done Sent false
609 1091 Acked Write 7 Done Done Sent false
610 1092 Acked Read 9 Done Done Sent false
611 1093 Acked Write 9 Done Done Sent false
612 1094 Acked Write 4 Done Done Sent false
613 1095 Acked Write 3 Done Done Sent false
614 1096 Acked Read 4 Done Done Sent false
615 1097 Acked Read 4 Done Done Sent false
616 1098 Acked Write 9 Done Done Sent false
617 1099 Acked Write 5 Done Done Sent false
618 1100 Acked Read 4 Done Done Sent false
619 1101 Acked Write 4 Done Done Sent false
620 1102 Acked Flush 0 Done Done Sent false
621 1103 Acked Read 9 Done Done Sent false
622 1104 Acked Flush 0 Done Done Sent false
623 1105 Acked Read 10 Done Done Sent false
624 1106 Acked Read 9 Done Done Sent false
625 1107 Acked Read 9 Done Done Sent false
626 1108 Acked Write 8 Done Done Sent false
627 1109 Acked Read 8 Done Done Sent false
628 1110 Acked Write 9 Done Done Sent false
629 1111 Acked Write 3 Done Done Sent false
630 1112 Acked Write 7 Done Done Sent false
631 1113 Acked Write 9 Done Done Sent false
632 1114 Acked Flush 0 Done Done Sent false
633 1115 Acked Write 2 Done Done Sent false
634 1116 Acked Write 3 Done Done Sent false
635 1117 Acked Write 1 Done Done Sent false
636 1118 Acked Read 5 Done Done Sent false
637 1119 Acked Flush 0 Done Done Sent false
638 1120 Acked Read 1 Done Done Sent false
639 1121 Acked Read 2 Done Done Sent false
640 1122 Acked Flush 0 Done Done Sent false
641 1123 Acked Flush 0 Done Done Sent false
642 1124 Acked Read 7 Done Done Sent false
643 1125 Acked Write 10 Done Done Sent false
644 1126 Acked Write 7 Done Done Sent false
645 1127 Acked Flush 0 Done Done Sent false
646 1128 Acked Read 5 Done Done Sent false
647 1129 Acked Write 8 Done Done Sent false
648 1130 Acked Write 3 Done Done Sent false
649 1131 Acked Write 1 Done Done Sent false
650 1132 Acked Read 3 Done Done Sent false
651 1133 Acked Read 5 Done Done Sent false
652 1134 Acked Write 7 Done Done Sent false
653 1135 Acked Flush 0 Done Done Sent false
654 1136 Acked Write 4 Done Done Sent false
655 1137 Acked Write 6 Done Done Sent false
656 1138 Acked Read 9 Done Done Sent false
657 1139 Acked Read 9 Done Done Sent false
658 1140 Acked Read 1 Done Done Sent false
659 1141 Acked Flush 0 Done Done Sent false
660 1142 Acked Read 10 Done Done Sent false
661 1143 Acked Read 2 Done Done Sent false
662 1144 Acked Read 5 Done Done Sent false
663 1145 Acked Read 10 Done Done Sent false
664 1146 Acked Write 7 Done Done Sent false
665 1147 Acked Read 5 Done Done Sent false
666 1148 Acked Write 3 Done Done Sent false
667 1149 Acked Read 7 Done Done Sent false
668 1150 Acked Read 6 Done Done Sent false
669 1151 Acked Read 4 Done Done Sent false
670 1152 Acked Flush 0 Done Done Sent false
671 1153 Acked Write 10 Done Done Sent false
672 1154 Acked Write 8 Done Done Sent false
673 1155 Acked Flush 0 Done Done Sent false
674 1156 Acked Read 2 Done Done Sent false
675 1157 Acked Write 2 Done Done Sent false
676 1158 Acked Read 9 Done Done Sent false
677 1159 Acked Write 10 Done Done Sent false
678 1160 Acked Write 6 Done Done Sent false
679 1161 Acked Read 2 Done Done Sent false
680 1162 Acked Read 7 Done Done Sent false
681 1163 Acked Read 7 Done Done Sent false
682 1164 Acked Read 4 Done Done Sent false
683 1165 Acked Read 2 Done Done Sent false
684 1166 Acked Flush 0 Done Done Sent false
685 1167 Acked Read 1 Done Done Sent false
686 1168 Acked Read 3 Done Done Sent false
687 1169 Acked Read 7 Done Done Sent false
688 1170 Acked Write 6 Done Done Sent false
689 1171 Acked Write 9 Done Done Sent false
690 1172 Acked Read 1 Done Done Sent false
691 1173 Acked Flush 0 Done Done Sent false
692 1174 Acked Write 4 Done Done Sent false
693 1175 Acked Write 2 Done Done Sent false
694 1176 Acked Read 2 Done Done Sent false
695 1177 Acked Write 2 Done Done Sent false
696 1178 Acked Write 1 Done Done Sent false
697 1179 Acked Read 2 Done Done Sent false
698 1180 Acked Read 7 Done Done Sent false
699 1181 Acked Write 3 Done Done Sent false
700 1182 Acked Read 2 Done Done Sent false
701 1183 Acked Read 6 Done Done Sent false
702 1184 Acked Write 6 Done Done Sent false
703 1185 Acked Read 4 Done Done Sent false
704 1186 Acked Flush 0 Done Done Sent false
705 1187 Acked Read 4 Done Done Sent false
706 1188 Acked Read 7 Done Done Sent false
707 1189 Acked Write 6 Done Done Sent false
708 1190 Acked Read 3 Done Done Sent false
709 1191 Acked Read 6 Done Done Sent false
710 1192 Acked Read 7 Done Done Sent false
711 1193 Acked Read 4 Done Done Sent false
712 1194 Acked Write 1 Done Done Sent false
713 1195 Acked Read 1 Done Done Sent false
714 1196 Acked Read 1 Done Done Sent false
715 1197 Acked Write 3 Done Done Sent false
716 1198 Acked Read 7 Done Done Sent false
717 1199 Acked Read 2 Done Done Sent false
718 1200 Acked Read 1 Done Done Sent false
719 1201 Acked Write 8 Done Done Sent false
720 1202 Acked Read 6 Done Done Sent false
721 1203 Acked Flush 0 Done Done Sent false
722 1204 Acked Write 5 Done Done Sent false
723 1205 Acked Read 5 Done Done Sent false
724 1206 Acked Flush 0 Done Done Sent false
725 1207 Acked Write 2 Done Done Sent false
726 1208 Acked Read 9 Done Done Sent false
727 1209 Acked Read 1 Done Done Sent false
728 1210 Acked Write 2 Done Done Sent false
729 1211 Acked Read 4 Done Done Sent false
730 1212 Acked Read 2 Done Done Sent false
731 1213 Acked Write 3 Done Done Sent false
732 1214 Acked Read 7 Done Done Sent false
733 1215 Acked Write 1 Done Done Sent false
734 1216 Acked Write 4 Done Done Sent false
735 1217 Acked Read 10 Done Done Sent false
736 1218 Acked Write 1 Done Done Sent false
737 1219 Acked Read 2 Done Done Sent false
738 1220 Acked Read 1 Done Done Sent false
739 1221 Acked Write 8 Done Done Sent false
740 1222 Acked Write 4 Done Done Sent false
741 1223 Acked Flush 0 Done Done Sent false
742 1224 Acked Read 7 Done Done Sent false
743 1225 Acked Flush 0 Done Done Sent false
744 1226 Acked Write 2 Done Done Sent false
745 1227 Acked Read 8 Done Done Sent false
746 1228 Acked Read 9 Done Done Sent false
747 1229 Acked Read 1 Done Done Sent false
748 1230 Acked Read 7 Done Done Sent false
749 1231 Acked Read 3 Done Done Sent false
750 1232 Acked Read 6 Done Done Sent false
751 1233 Acked Read 7 Done Done Sent false
752 1234 Acked Write 4 Done Done Sent false
753 1235 Acked Write 2 Done Done Sent false
754 1236 Acked Read 5 Done Done Sent false
755 1237 Acked Read 3 Done Done Sent false
756 1238 Acked Write 5 Done Done Sent false
757 1239 Acked Write 5 Done Done Sent false
758 1240 Acked Read 6 Done Done Sent false
759 1241 Acked Write 10 Done Done Sent false
760 1242 Acked Read 1 Done Done Sent false
761 1243 Acked Write 4 Done Done Sent false
762 1244 Acked Read 1 Done Done Sent false
763 1245 Acked Write 4 Done Done Sent false
764 1246 Acked Write 6 Done Done Sent false
765 1247 Acked Write 7 Done Done Sent false
766 1248 Acked Write 1 Done Done Sent false
767 1249 Acked Read 9 Done Done Sent false
768 1250 Acked Flush 0 Done Done Sent false
769 1251 Acked Write 3 Done Done Sent false
770 1252 Acked Write 7 Done Done Sent false
771 1253 Acked Write 5 Done Done Sent false
772 1254 Acked Read 5 Done Done Sent false
773 1255 Acked Read 10 Done Done Sent false
774 1256 Acked Read 7 Done Done Sent false
775 1257 Acked Read 5 Done Done Sent false
776 1258 Acked Write 7 Done Done Sent false
777 1259 Acked Write 4 Done Done Sent false
778 1260 Acked Write 6 Done Done Sent false
779 1261 Acked Read 9 Done Done Sent false
780 1262 Acked Write 7 Done Done Sent false
781 1263 Acked Read 2 Done Done Sent false
782 1264 Acked Write 2 Done Done Sent false
783 1265 Acked Write 4 Done Done Sent false
784 1266 Acked Write 10 Done Done Sent false
785 1267 Acked Write 3 Done Done Sent false
786 1268 Acked Read 5 Done Done Sent false
787 1269 Acked Write 8 Done Done Sent false
788 1270 Acked Read 9 Done Done Sent false
789 1271 Acked Read 4 Done Done Sent false
790 1272 Acked Write 3 Done Done Sent false
791 1273 Acked Write 5 Done Done Sent false
792 1274 Acked Write 8 Done Done Sent false
793 1275 Acked Read 3 Done Done Sent false
794 1276 Acked Write 5 Done Done Sent false
795 1277 Acked Read 10 Done Done Sent false
796 1278 Acked Read 1 Done Done Sent false
797 1279 Acked Flush 0 Done Done Sent false
798 1280 Acked Write 9 Done Done Sent false
799 1281 Acked Write 3 Done Done Sent false
800 1282 Acked Flush 0 Done Done Sent false
801 1283 Acked Write 5 Done Done Sent false
802 1284 Acked Read 1 Done Done Sent false
803 1285 Acked Read 9 Done Done Sent false
804 1286 Acked Read 7 Done Done Sent false
805 1287 Acked Read 1 Done Done Sent false
806 1288 Acked Write 1 Done Done Sent false
807 1289 Acked Read 5 Done Done Sent false
808 1290 Acked Read 2 Done Done Sent false
809 1291 Acked Read 9 Done Done Sent false
810 1292 Acked Read 9 Done Done Sent false
811 1293 Acked Read 2 Done Done Sent false
812 1294 Acked Write 8 Done Done Sent false
813 1295 Acked Flush 0 Done Done Sent false
814 1296 Acked Write 5 Done Done Sent false
815 1297 Acked Write 5 Done Done Sent false
816 1298 Acked Write 10 Done Done Sent false
817 1299 Acked Read 5 Done Done Sent false
818 1300 Acked Read 1 Done Done Sent false
819 STATES DS:0 DS:1 DS:2 TOTAL
820 Sent 0 0 301 301
821 Done 301 301 0 602
822 Skipped 0 0 0 0
823 Error 0 0 0 0
824 Last Flush: 1295 1295 0
825 Downstairs last five completed:
826 Upstairs last five completed: 1300 1299 1298 1297 1296
827 CLIENT: Up:0 ds:301 act:2
828 [8830] got start action
829 [8830] I am going to restart
830 Starting downstairs at port 8830
831 Make output file at "/tmp/dsc/downstairs-8830.txt"
832 [8830][2] reports Starting
833 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1199)
834 [8830][2] reports Running
835 ----------------------------------------------------------------
836 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
837 States: Active Active Offline
838 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
839 1000 Acked Read 10 Done Done Sent false
840 1001 Acked Flush 0 Done Done Sent false
841 1002 Acked Read 7 Done Done Sent false
842 1003 Acked Write 5 Done Done Sent false
843 1004 Acked Read 5 Done Done Sent false
844 1005 Acked Read 1 Done Done Sent false
845 1006 Acked Read 6 Done Done Sent false
846 1007 Acked Read 6 Done Done Sent false
847 1008 Acked Flush 0 Done Done Sent false
848 1009 Acked Read 9 Done Done Sent false
849 1010 Acked Read 4 Done Done Sent false
850 1011 Acked Read 1 Done Done Sent false
851 1012 Acked Read 3 Done Done Sent false
852 1013 Acked Read 9 Done Done Sent false
853 1014 Acked Read 10 Done Done Sent false
854 1015 Acked Read 8 Done Done Sent false
855 1016 Acked Read 10 Done Done Sent false
856 1017 Acked Flush 0 Done Done Sent false
857 1018 Acked Write 6 Done Done Sent false
858 1019 Acked Read 1 Done Done Sent false
859 1020 Acked Write 7 Done Done Sent false
860 1021 Acked Read 2 Done Done Sent false
861 1022 Acked Read 10 Done Done Sent false
862 1023 Acked Read 9 Done Done Sent false
863 1024 Acked Write 9 Done Done Sent false
864 1025 Acked Read 10 Done Done Sent false
865 1026 Acked Read 9 Done Done Sent false
866 1027 Acked Read 10 Done Done Sent false
867 1028 Acked Read 7 Done Done Sent false
868 1029 Acked Read 7 Done Done Sent false
869 1030 Acked Write 9 Done Done Sent false
870 1031 Acked Write 4 Done Done Sent false
871 1032 Acked Read 8 Done Done Sent false
872 1033 Acked Read 9 Done Done Sent false
873 1034 Acked Read 8 Done Done Sent false
874 1035 Acked Write 1 Done Done Sent false
875 1036 Acked Read 10 Done Done Sent false
876 1037 Acked Flush 0 Done Done Sent false
877 1038 Acked Write 10 Done Done Sent false
878 1039 Acked Read 3 Done Done Sent false
879 1040 Acked Write 5 Done Done Sent false
880 1041 Acked Read 6 Done Done Sent false
881 1042 Acked Write 6 Done Done Sent false
882 1043 Acked Flush 0 Done Done Sent false
883 1044 Acked Read 5 Done Done Sent false
884 1045 Acked Read 2 Done Done Sent false
885 1046 Acked Write 1 Done Done Sent false
886 1047 Acked Read 4 Done Done Sent false
887 1048 Acked Write 10 Done Done Sent false
888 1049 Acked Read 3 Done Done Sent false
889 1050 Acked Read 10 Done Done Sent false
890 1051 Acked Write 4 Done Done Sent false
891 1052 Acked Write 9 Done Done Sent false
892 1053 Acked Read 5 Done Done Sent false
893 1054 Acked Read 10 Done Done Sent false
894 1055 Acked Write 1 Done Done Sent false
895 1056 Acked Write 5 Done Done Sent false
896 1057 Acked Read 4 Done Done Sent false
897 1058 Acked Write 10 Done Done Sent false
898 1059 Acked Write 6 Done Done Sent false
899 1060 Acked Write 10 Done Done Sent false
900 1061 Acked Write 4 Done Done Sent false
901 1062 Acked Read 2 Done Done Sent false
902 1063 Acked Read 8 Done Done Sent false
903 1064 Acked Write 5 Done Done Sent false
904 1065 Acked Read 1 Done Done Sent false
905 1066 Acked Write 8 Done Done Sent false
906 1067 Acked Write 6 Done Done Sent false
907 1068 Acked Read 8 Done Done Sent false
908 1069 Acked Read 7 Done Done Sent false
909 1070 Acked Read 8 Done Done Sent false
910 1071 Acked Read 9 Done Done Sent false
911 1072 Acked Write 3 Done Done Sent false
912 1073 Acked Read 1 Done Done Sent false
913 1074 Acked Write 2 Done Done Sent false
914 1075 Acked Read 8 Done Done Sent false
915 1076 Acked Write 7 Done Done Sent false
916 1077 Acked Write 8 Done Done Sent false
917 1078 Acked Flush 0 Done Done Sent false
918 1079 Acked Write 2 Done Done Sent false
919 1080 Acked Write 6 Done Done Sent false
920 1081 Acked Read 5 Done Done Sent false
921 1082 Acked Read 2 Done Done Sent false
922 1083 Acked Read 6 Done Done Sent false
923 1084 Acked Write 2 Done Done Sent false
924 1085 Acked Read 9 Done Done Sent false
925 1086 Acked Write 5 Done Done Sent false
926 1087 Acked Read 7 Done Done Sent false
927 1088 Acked Read 9 Done Done Sent false
928 1089 Acked Read 5 Done Done Sent false
929 1090 Acked Read 7 Done Done Sent false
930 1091 Acked Write 7 Done Done Sent false
931 1092 Acked Read 9 Done Done Sent false
932 1093 Acked Write 9 Done Done Sent false
933 1094 Acked Write 4 Done Done Sent false
934 1095 Acked Write 3 Done Done Sent false
935 1096 Acked Read 4 Done Done Sent false
936 1097 Acked Read 4 Done Done Sent false
937 1098 Acked Write 9 Done Done Sent false
938 1099 Acked Write 5 Done Done Sent false
939 1100 Acked Read 4 Done Done Sent false
940 1101 Acked Write 4 Done Done Sent false
941 1102 Acked Flush 0 Done Done Sent false
942 1103 Acked Read 9 Done Done Sent false
943 1104 Acked Flush 0 Done Done Sent false
944 1105 Acked Read 10 Done Done Sent false
945 1106 Acked Read 9 Done Done Sent false
946 1107 Acked Read 9 Done Done Sent false
947 1108 Acked Write 8 Done Done Sent false
948 1109 Acked Read 8 Done Done Sent false
949 1110 Acked Write 9 Done Done Sent false
950 1111 Acked Write 3 Done Done Sent false
951 1112 Acked Write 7 Done Done Sent false
952 1113 Acked Write 9 Done Done Sent false
953 1114 Acked Flush 0 Done Done Sent false
954 1115 Acked Write 2 Done Done Sent false
955 1116 Acked Write 3 Done Done Sent false
956 1117 Acked Write 1 Done Done Sent false
957 1118 Acked Read 5 Done Done Sent false
958 1119 Acked Flush 0 Done Done Sent false
959 1120 Acked Read 1 Done Done Sent false
960 1121 Acked Read 2 Done Done Sent false
961 1122 Acked Flush 0 Done Done Sent false
962 1123 Acked Flush 0 Done Done Sent false
963 1124 Acked Read 7 Done Done Sent false
964 1125 Acked Write 10 Done Done Sent false
965 1126 Acked Write 7 Done Done Sent false
966 1127 Acked Flush 0 Done Done Sent false
967 1128 Acked Read 5 Done Done Sent false
968 1129 Acked Write 8 Done Done Sent false
969 1130 Acked Write 3 Done Done Sent false
970 1131 Acked Write 1 Done Done Sent false
971 1132 Acked Read 3 Done Done Sent false
972 1133 Acked Read 5 Done Done Sent false
973 1134 Acked Write 7 Done Done Sent false
974 1135 Acked Flush 0 Done Done Sent false
975 1136 Acked Write 4 Done Done Sent false
976 1137 Acked Write 6 Done Done Sent false
977 1138 Acked Read 9 Done Done Sent false
978 1139 Acked Read 9 Done Done Sent false
979 1140 Acked Read 1 Done Done Sent false
980 1141 Acked Flush 0 Done Done Sent false
981 1142 Acked Read 10 Done Done Sent false
982 1143 Acked Read 2 Done Done Sent false
983 1144 Acked Read 5 Done Done Sent false
984 1145 Acked Read 10 Done Done Sent false
985 1146 Acked Write 7 Done Done Sent false
986 1147 Acked Read 5 Done Done Sent false
987 1148 Acked Write 3 Done Done Sent false
988 1149 Acked Read 7 Done Done Sent false
989 1150 Acked Read 6 Done Done Sent false
990 1151 Acked Read 4 Done Done Sent false
991 1152 Acked Flush 0 Done Done Sent false
992 1153 Acked Write 10 Done Done Sent false
993 1154 Acked Write 8 Done Done Sent false
994 1155 Acked Flush 0 Done Done Sent false
995 1156 Acked Read 2 Done Done Sent false
996 1157 Acked Write 2 Done Done Sent false
997 1158 Acked Read 9 Done Done Sent false
998 1159 Acked Write 10 Done Done Sent false
999 1160 Acked Write 6 Done Done Sent false
1000 1161 Acked Read 2 Done Done Sent false
1001 1162 Acked Read 7 Done Done Sent false
1002 1163 Acked Read 7 Done Done Sent false
1003 1164 Acked Read 4 Done Done Sent false
1004 1165 Acked Read 2 Done Done Sent false
1005 1166 Acked Flush 0 Done Done Sent false
1006 1167 Acked Read 1 Done Done Sent false
1007 1168 Acked Read 3 Done Done Sent false
1008 1169 Acked Read 7 Done Done Sent false
1009 1170 Acked Write 6 Done Done Sent false
1010 1171 Acked Write 9 Done Done Sent false
1011 1172 Acked Read 1 Done Done Sent false
1012 1173 Acked Flush 0 Done Done Sent false
1013 1174 Acked Write 4 Done Done Sent false
1014 1175 Acked Write 2 Done Done Sent false
1015 1176 Acked Read 2 Done Done Sent false
1016 1177 Acked Write 2 Done Done Sent false
1017 1178 Acked Write 1 Done Done Sent false
1018 1179 Acked Read 2 Done Done Sent false
1019 1180 Acked Read 7 Done Done Sent false
1020 1181 Acked Write 3 Done Done Sent false
1021 1182 Acked Read 2 Done Done Sent false
1022 1183 Acked Read 6 Done Done Sent false
1023 1184 Acked Write 6 Done Done Sent false
1024 1185 Acked Read 4 Done Done Sent false
1025 1186 Acked Flush 0 Done Done Sent false
1026 1187 Acked Read 4 Done Done Sent false
1027 1188 Acked Read 7 Done Done Sent false
1028 1189 Acked Write 6 Done Done Sent false
1029 1190 Acked Read 3 Done Done Sent false
1030 1191 Acked Read 6 Done Done Sent false
1031 1192 Acked Read 7 Done Done Sent false
1032 1193 Acked Read 4 Done Done Sent false
1033 1194 Acked Write 1 Done Done Sent false
1034 1195 Acked Read 1 Done Done Sent false
1035 1196 Acked Read 1 Done Done Sent false
1036 1197 Acked Write 3 Done Done Sent false
1037 1198 Acked Read 7 Done Done Sent false
1038 1199 Acked Read 2 Done Done Sent false
1039 1200 Acked Read 1 Done Done Sent false
1040 1201 Acked Write 8 Done Done Sent false
1041 1202 Acked Read 6 Done Done Sent false
1042 1203 Acked Flush 0 Done Done Sent false
1043 1204 Acked Write 5 Done Done Sent false
1044 1205 Acked Read 5 Done Done Sent false
1045 1206 Acked Flush 0 Done Done Sent false
1046 1207 Acked Write 2 Done Done Sent false
1047 1208 Acked Read 9 Done Done Sent false
1048 1209 Acked Read 1 Done Done Sent false
1049 1210 Acked Write 2 Done Done Sent false
1050 1211 Acked Read 4 Done Done Sent false
1051 1212 Acked Read 2 Done Done Sent false
1052 1213 Acked Write 3 Done Done Sent false
1053 1214 Acked Read 7 Done Done Sent false
1054 1215 Acked Write 1 Done Done Sent false
1055 1216 Acked Write 4 Done Done Sent false
1056 1217 Acked Read 10 Done Done Sent false
1057 1218 Acked Write 1 Done Done Sent false
1058 1219 Acked Read 2 Done Done Sent false
1059 1220 Acked Read 1 Done Done Sent false
1060 1221 Acked Write 8 Done Done Sent false
1061 1222 Acked Write 4 Done Done Sent false
1062 1223 Acked Flush 0 Done Done Sent false
1063 1224 Acked Read 7 Done Done Sent false
1064 1225 Acked Flush 0 Done Done Sent false
1065 1226 Acked Write 2 Done Done Sent false
1066 1227 Acked Read 8 Done Done Sent false
1067 1228 Acked Read 9 Done Done Sent false
1068 1229 Acked Read 1 Done Done Sent false
1069 1230 Acked Read 7 Done Done Sent false
1070 1231 Acked Read 3 Done Done Sent false
1071 1232 Acked Read 6 Done Done Sent false
1072 1233 Acked Read 7 Done Done Sent false
1073 1234 Acked Write 4 Done Done Sent false
1074 1235 Acked Write 2 Done Done Sent false
1075 1236 Acked Read 5 Done Done Sent false
1076 1237 Acked Read 3 Done Done Sent false
1077 1238 Acked Write 5 Done Done Sent false
1078 1239 Acked Write 5 Done Done Sent false
1079 1240 Acked Read 6 Done Done Sent false
1080 1241 Acked Write 10 Done Done Sent false
1081 1242 Acked Read 1 Done Done Sent false
1082 1243 Acked Write 4 Done Done Sent false
1083 1244 Acked Read 1 Done Done Sent false
1084 1245 Acked Write 4 Done Done Sent false
1085 1246 Acked Write 6 Done Done Sent false
1086 1247 Acked Write 7 Done Done Sent false
1087 1248 Acked Write 1 Done Done Sent false
1088 1249 Acked Read 9 Done Done Sent false
1089 1250 Acked Flush 0 Done Done Sent false
1090 1251 Acked Write 3 Done Done Sent false
1091 1252 Acked Write 7 Done Done Sent false
1092 1253 Acked Write 5 Done Done Sent false
1093 1254 Acked Read 5 Done Done Sent false
1094 1255 Acked Read 10 Done Done Sent false
1095 1256 Acked Read 7 Done Done Sent false
1096 1257 Acked Read 5 Done Done Sent false
1097 1258 Acked Write 7 Done Done Sent false
1098 1259 Acked Write 4 Done Done Sent false
1099 1260 Acked Write 6 Done Done Sent false
1100 1261 Acked Read 9 Done Done Sent false
1101 1262 Acked Write 7 Done Done Sent false
1102 1263 Acked Read 2 Done Done Sent false
1103 1264 Acked Write 2 Done Done Sent false
1104 1265 Acked Write 4 Done Done Sent false
1105 1266 Acked Write 10 Done Done Sent false
1106 1267 Acked Write 3 Done Done Sent false
1107 1268 Acked Read 5 Done Done Sent false
1108 1269 Acked Write 8 Done Done Sent false
1109 1270 Acked Read 9 Done Done Sent false
1110 1271 Acked Read 4 Done Done Sent false
1111 1272 Acked Write 3 Done Done Sent false
1112 1273 Acked Write 5 Done Done Sent false
1113 1274 Acked Write 8 Done Done Sent false
1114 1275 Acked Read 3 Done Done Sent false
1115 1276 Acked Write 5 Done Done Sent false
1116 1277 Acked Read 10 Done Done Sent false
1117 1278 Acked Read 1 Done Done Sent false
1118 1279 Acked Flush 0 Done Done Sent false
1119 1280 Acked Write 9 Done Done Sent false
1120 1281 Acked Write 3 Done Done Sent false
1121 1282 Acked Flush 0 Done Done Sent false
1122 1283 Acked Write 5 Done Done Sent false
1123 1284 Acked Read 1 Done Done Sent false
1124 1285 Acked Read 9 Done Done Sent false
1125 1286 Acked Read 7 Done Done Sent false
1126 1287 Acked Read 1 Done Done Sent false
1127 1288 Acked Write 1 Done Done Sent false
1128 1289 Acked Read 5 Done Done Sent false
1129 1290 Acked Read 2 Done Done Sent false
1130 1291 Acked Read 9 Done Done Sent false
1131 1292 Acked Read 9 Done Done Sent false
1132 1293 Acked Read 2 Done Done Sent false
1133 1294 Acked Write 8 Done Done Sent false
1134 1295 Acked Flush 0 Done Done Sent false
1135 1296 Acked Write 5 Done Done Sent false
1136 1297 Acked Write 5 Done Done Sent false
1137 1298 Acked Write 10 Done Done Sent false
1138 1299 Acked Read 5 Done Done Sent false
1139 1300 Acked Read 1 Done Done Sent false
1140 1301 Acked Flush 0 Done Done Sent false
1141 STATES DS:0 DS:1 DS:2 TOTAL
1142 Sent 0 0 302 302
1143 Done 302 302 0 604
1144 Skipped 0 0 0 0
1145 Error 0 0 0 0
1146 Last Flush: 1301 1301 0
1147 Downstairs last five completed:
1148 Upstairs last five completed: 1301 1300 1299 1298 1297
1149 CLIENT: Up:0 ds:302 act:2
11502024-11-01T16:50:42.259ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11512024-11-01T16:50:42.259ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46817) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11522024-11-01T16:50:42.260ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11532024-11-01T16:50:42.261ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11542024-11-01T16:50:42.261ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11552024-11-01T16:50:42.261ZINFOcrucible: send last flush ID to this DS: 0 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11562024-11-01T16:50:42.261ZINFOcrucible: Replied this last flush ID: 0 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11572024-11-01T16:50:42.261ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11582024-11-01T16:50:42.261ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11592024-11-01T16:50:42.261ZINFOcrucible: [2] client re-new 302 jobs since flush 0 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11602024-11-01T16:50:42.262ZINFOcrucible: [2] Replayed 302 jobs since flush: 0 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1161 ----------------------------------------------------------------
1162 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1163 Downstairs last five completed: 1301 1300 1299 1298 1297
1164 Upstairs last five completed: 1301 1300 1299 1298 1297
1165 CLIENT: Up:0 ds:0 act:3
1166 Replay: All jobs finished
1167 Main task has work to do, go find it
1168 got dsc Stop(1)
1169 stop 1
1170 Nov 01 16:50:44.560 INFO request completed, latency_us: 366[8820] Got stop action so:false kr:false
1171 , response_code: 204, uri: /stop/cid/1, method: POST, req_id: 4d11a978-5559-4ab5-913c-ea7b3cd5ac40, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
11722024-11-01T16:50:44.561ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11732024-11-01T16:50:44.561ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11742024-11-01T16:50:44.561ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11752024-11-01T16:50:44.561ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
1176 Nov 01 16:50:44.561 INFO request completed, latency_us: 255, response_code: 200, uri: /state/cid/1, method: GET, req_id{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:50:44.561392408Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
1177 : 6da64704-99a1-45a0-8343-44b2cdd3824d, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
1178 [8820] Exited with: ExitStatus(unix_wait_status(9))
1179 [8820][1] reports Exit
1180 Nov 01 16:50:48.563 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/1, method: GET, req_id: 12857b07-8493-4b3a-86bf-1c83f0f8fd09, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
1181 001/300 Read block 1031 len 20480
1182 002/300 Read block 3632 len 40960
1183 003/300 Write block 2369 len 20480 data: 2 2 2 2 2
1184 004/300 Write block 2888 len 16384 data: 2 2 2 2
1185 005/300 Flush
1186 006/300 Write block 3511 len 20480 data: 2 2 2 2 2
1187 007/300 Read block 1527 len 36864
1188 008/300 Read block 3081 len 36864
1189 009/300 Write block 4102 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1190 010/300 Write block 4170 len 20480 data: 2 2 2 2 2
1191 011/300 Flush
1192 012/300 Read block 4314 len 4096
1193 013/300 Write block 906 len 8192 data: 2 2
1194 014/300 Write block 246 len 12288 data: 3 3 3
1195 015/300 Write block 1883 len 8192 data: 3 3
1196 016/300 Read block 4382 len 24576
1197 017/300 Read block 352 len 12288
1198 018/300 Write block 2370 len 4096 data: 3
1199 019/300 Write block 1435 len 20480 data: 3 3 3 3 2
1200 020/300 Write block 4942 len 32768 data: 3 3 3 2 2 2 2 2
1201 021/300 Write block 3592 len 36864 data: 2 2 2 2 2 2 2 2 2
1202 022/300 Write block 999 len 28672 data: 2 2 2 2 2 2 2
1203 023/300 Read block 2528 len 16384
1204 024/300 Read block 2580 len 16384
1205 025/300 Write block 2051 len 12288 data: 2 2 2
1206 026/300 Read block 4670 len 36864
1207 027/300 Read block 4607 len 8192
1208 028/300 Write block 3465 len 16384 data: 2 2 2 2
1209 029/300 Read block 4724 len 28672
1210 030/300 Write block 1969 len 12288 data: 4 4 4
1211 031/300 Write block 3778 len 24576 data: 2 2 2 2 2 2
1212 032/300 Write block 4258 len 24576 data: 2 2 2 2 2 2
1213 033/300 Write block 4953 len 24576 data: 2 2 2 2 2 2
1214 034/300 Write block 3966 len 24576 data: 2 2 2 2 2 2
1215 035/300 Write block 4318 len 40960 data: 2 2 3 3 3 3 2 2 2 2
1216 036/300 Flush
1217 037/300 Read block 3264 len 24576
1218 038/300 Read block 4740 len 36864
1219 039/300 Flush
1220 040/300 Flush
1221 041/300 Write block 996 len 24576 data: 2 2 2 3 3 3
1222 042/300 Write block 4670 len 8192 data: 2 2
1223 043/300 Flush
1224 044/300 Read block 2452 len 4096
1225 045/300 Read block 3547 len 8192
1226 046/300 Flush
1227 047/300 Read block 841 len 8192
1228 048/300 Write block 578 len 24576 data: 2 2 2 2 2 2
1229 049/300 Read block 4862 len 40960
1230 050/300 Read block 1299 len 32768
1231 051/300 Flush
1232 052/300 Read block 38 len 8192
1233 053/300 Read block 366 len 16384
1234 054/300 Write block 3043 len 24576 data: 3 2 2 2 2 2
1235 055/300 Read block 4371 len 28672
1236 056/300 Read block 934 len 40960
1237 057/300 Write block 1491 len 8192 data: 2 2
1238 058/300 Write block 286 len 20480 data: 2 2 2 2 3
1239 059/300 Write block 3299 len 24576 data: 2 2 2 2 2 2
1240 060/300 Write block 4205 len 24576 data: 2 2 2 2 2 2
1241 061/300 Write block 4087 len 32768 data: 3 3 3 3 2 2 2 2
1242 062/300 Write block 4532 len 28672 data: 2 2 2 2 2 2 2
1243 063/300 Write block 1739 len 36864 data: 2 2 2 2 2 2 2 2 2
1244 064/300 Read block 1190 len 36864
1245 065/300 Write block 888 len 32768 data: 2 2 2 2 2 2 2 2
1246 066/300 Write block 105 len 8192 data: 2 2
1247 067/300 Read block 1024 len 20480
1248 068/300 Read block 1410 len 4096
1249 069/300 Read block 122 len 16384
1250 070/300 Write block 4460 len 24576 data: 2 2 2 2 2 2
1251 071/300 Write block 1968 len 8192 data: 4 5
1252 072/300 Write block 1282 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1253 073/300 Flush
1254 074/300 Read block 3708 len 20480
1255 075/300 Flush
1256 076/300 Read block 829 len 4096
1257 077/300 Read block 4089 len 40960
1258 078/300 Flush
1259 079/300 Write block 3916 len 24576 data: 2 2 2 2 2 2
1260 080/300 Flush
1261 081/300 Write block 1348 len 28672 data: 2 2 2 2 2 2 2
1262 082/300 Write block 1263 len 16384 data: 2 2 2 2
1263 083/300 Read block 3945 len 20480
1264 084/300 Write block 96 len 32768 data: 2 2 2 2 2 2 2 2
1265 085/300 Write block 519 len 12288 data: 3 3 3
1266 086/300 Write block 651 len 12288 data: 2 2 2
1267 087/300 Read block 3142 len 32768
1268 088/300 Write block 1077 len 16384 data: 3 3 3 3
1269 089/300 Write block 1539 len 20480 data: 2 2 2 2 2
1270 090/300 Read block 3179 len 32768
1271 091/300 Write block 53 len 16384 data: 2 2 2 2
1272 092/300 Read block 1356 len 4096
1273 093/300 Write block 4870 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1274 094/300 Write block 176 len 8192 data: 2 2
1275 095/300 Write block 514 len 12288 data: 3 2 3
1276 096/300 Read block 2569 len 36864
1277 097/300 Read block 1798 len 24576
1278 098/300 Read block 851 len 16384
1279 099/300 Read block 24 len 8192
1280 100/300 Write block 922 len 24576 data: 2 2 2 2 2 2
1281 101/300 Write block 753 len 16384 data: 3 3 3 3
1282 102/300 Write block 1852 len 12288 data: 2 2 2
1283 103/300 Read block 2324 len 36864
1284 104/300 Read block 1418 len 24576
1285 105/300 Read block 896 len 16384
1286 106/300 Read block 3148 len 16384
1287 107/300 Read block 4278 len 32768
1288 108/300 Read block 4250 len 16384
1289 109/300 Read block 1457 len 8192
1290 110/300 Write block 4177 len 28672 data: 2 2 2 2 2 2 2
1291 111/300 Read block 2518 len 16384
1292 112/300 Write block 3558 len 8192 data: 2 2
1293 113/300 Write block 2935 len 4096 data: 2
1294 114/300 Write block 2741 len 28672 data: 2 2 2 2 2 2 2
1295 115/300 Write block 4339 len 32768 data: 2 2 2 2 2 2 2 2
1296 116/300 Flush
1297 117/300 Write block 552 len 20480 data: 3 3 3 3 3
1298 118/300 Read block 3972 len 20480
1299 119/300 Read block 2332 len 24576
1300 120/300 Read block 197 len 40960
1301 121/300 Write block 3092 len 36864 data: 2 2 2 2 2 2 2 2 2
1302 122/300 Read block 4157 len 36864
1303 123/300 Write block 4392 len 8192 data: 2 2
1304 124/300 Read block 4552 len 4096
1305 125/300 Write block 890 len 32768 data: 3 3 3 3 3 3 2 2
1306 126/300 Read block 2758 len 16384
1307 127/300 Read block 2886 len 20480
1308 128/300 Write block 2186 len 4096 data: 2
1309 129/300 Read block 3126 len 8192
1310 130/300 Read block 319 len 32768
1311 131/300 Read block 4449 len 4096
1312 132/300 Read block 204 len 12288
1313 133/300 Read block 91 len 20480
1314 134/300 Read block 4096 len 36864
1315 135/300 Read block 4219 len 20480
1316 136/300 Read block 3409 len 32768
1317 137/300 Read block 2988 len 40960
1318 138/300 Read block 3890 len 4096
1319 139/300 Read block 1679 len 28672
1320 140/300 Read block 4353 len 28672
1321 141/300 Write block 2821 len 8192 data: 2 2
1322 142/300 Write block 800 len 8192 data: 3 3
1323 143/300 Read block 3252 len 12288
1324 144/300 Write block 3424 len 16384 data: 2 2 2 2
1325 145/300 Write block 3699 len 36864 data: 2 2 2 2 3 3 3 3 3
1326 146/300 Write block 97 len 40960 data: 3 3 3 3 3 3 3 2 3 3
1327 147/300 Write block 3858 len 28672 data: 2 2 2 2 2 2 2
1328 148/300 Write block 3 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1329 149/300 Flush
1330 150/300 Read block 3589 len 28672
1331 151/300 Write block 2712 len 20480 data: 3 3 3 3 2
1332 152/300 Read block 2230 len 28672
1333 153/300 Write block 3320 len 20480 data: 2 2 2 2 2
1334 154/300 Read block 3086 len 36864
1335 155/300 Flush
1336 156/300 Write block 3404 len 12288 data: 2 2 2
1337 157/300 Read block 4212 len 4096
1338 158/300 Flush
1339 159/300 Read block 1709 len 12288
1340 160/300 Write block 1429 len 32768 data: 2 2 3 3 3 3 4 4
1341 161/300 Write block 3826 len 32768 data: 3 3 3 2 3 3 3 4
1342 162/300 Read block 1060 len 36864
1343 163/300 Write block 4546 len 20480 data: 2 2 2 2 2
1344 164/300 Write block 2421 len 4096 data: 2
1345 165/300 Read block 3547 len 8192
1346 166/300 Write block 2875 len 24576 data: 2 2 2 2 2 2
1347 167/300 Read block 4538 len 36864
1348 168/300 Flush
1349 169/300 Write block 2815 len 24576 data: 3 3 2 2 2 2
1350 170/300 Read block 3476 len 40960
1351 171/300 Read block 4340 len 24576
1352 172/300 Read block 4550 len 8192
1353 173/300 Read block 3663 len 8192
1354 174/300 Read block 2988 len 24576
1355 175/300 Read block 2879 len 36864
1356 176/300 Write block 3422 len 32768 data: 3 3 3 3 3 3 2 2
1357 177/300 Read block 3728 len 24576
1358 178/300 Read block 4153 len 8192
1359 179/300 Read block 3077 len 24576
1360 180/300 Write block 2208 len 20480 data: 2 2 2 2 2
1361 181/300 Read block 1135 len 16384
1362 182/300 Write block 4023 len 20480 data: 2 2 2 2 2
1363 183/300 Write block 4939 len 4096 data: 2
1364 184/300 Write block 4659 len 36864 data: 2 2 2 2 3 3 2 2 2
1365 185/300 Write block 4321 len 24576 data: 4 4 4 3 3 3
1366 186/300 Write block 239 len 40960 data: 2 2 2 2 2 2 2 4 4 4
1367 187/300 Read block 2108 len 36864
1368 188/300 Read block 1699 len 40960
1369 189/300 Write block 1223 len 12288 data: 2 2 2
1370 190/300 Write block 2732 len 40960 data: 2 2 2 2 2 2 2 2 2 3
1371 191/300 Write block 2207 len 32768 data: 2 3 3 3 3 3 2 2
1372 192/300 Write block 3816 len 16384 data: 2 2 2 2
1373 193/300 Flush
1374 194/300 Read block 3093 len 40960
1375 195/300 Write block 749 len 4096 data: 2
1376 196/300 Write block 3404 len 16384 data: 3 3 3 2
1377 197/300 Read block 2804 len 16384
1378 198/300 Read block 3497 len 16384
1379 199/300 Write block 4629 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1380 200/300 Write block 506 len 12288 data: 2 2 2
1381 201/300 Write block 3817 len 8192 data: 3 3
1382 202/300 Write block 2207 len 8192 data: 3 4
1383 203/300 Write block 1496 len 24576 data: 2 2 2 2 2 2
1384 204/300 Flush
1385 205/300 Read block 427 len 16384
1386 206/300 Read block 2050 len 24576
1387 207/300 Flush
1388 208/300 Read block 3793 len 8192
1389 209/300 Flush
1390 210/300 Read block 3985 len 28672
1391 211/300 Write block 279 len 4096 data: 2
1392 212/300 Write block 1662 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1393 213/300 Read block 957 len 8192
1394 214/300 Write block 2415 len 16384 data: 2 2 2 2
1395 215/300 Flush
1396 216/300 Write block 4389 len 8192 data: 2 2
1397 217/300 Read block 2215 len 20480
1398 218/300 Read block 2661 len 24576
1399 219/300 Read block 713 len 8192
1400 220/300 Read block 829 len 20480
1401 221/300 Read block 1090 len 4096
1402 222/300 Flush
1403 223/300 Read block 291 len 16384
1404 224/300 Write block 529 len 24576 data: 2 2 2 2 2 2
1405 225/300 Read block 295 len 36864
1406 226/300 Write block 2938 len 12288 data: 2 2 2
1407 227/300 Write block 4451 len 20480 data: 2 2 2 2 2
1408 228/300 Read block 4170 len 12288
1409 229/300 Read block 2280 len 12288
1410 230/300 Write block 2235 len 36864 data: 2 2 2 2 2 2 2 2 2
1411 231/300 Read block 946 len 36864
1412 232/300 Read block 417 len 16384
1413 233/300 Write block 3787 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1414 234/300 Write block 2122 len 4096 data: 2
1415 235/300 Read block 2678 len 20480
1416 236/300 Read block 724 len 8192
1417 237/300 Write block 1176 len 24576 data: 2 2 2 2 2 2
1418 238/300 Write block 1602 len 8192 data: 2 2
1419 239/300 Flush
1420 240/300 Write block 3197 len 28672 data: 2 2 2 2 3 2 2
1421 241/300 Write block 4062 len 16384 data: 2 2 2 2
1422 242/300 Read block 2284 len 36864
1423 243/300 Read block 2014 len 24576
1424 244/300 Write block 2716 len 8192 data: 3 2
1425 245/300 Read block 4510 len 4096
1426 246/300 Read block 2448 len 40960
1427 247/300 Write block 50 len 12288 data: 2 2 2
1428 248/300 Read block 3124 len 36864
1429 249/300 Write block 1645 len 12288 data: 2 2 2
1430 250/300 Write block 1815 len 20480 data: 2 2 2 2 2
1431 251/300 Read block 3349 len 8192
1432 252/300 Write block 4134 len 20480 data: 2 2 2 2 2
1433 253/300 Write block 784 len 12288 data: 2 2 2
1434 254/300 Read block 3196 len 40960
1435 255/300 Write block 286 len 8192 data: 3 3
1436 256/300 Read block 2903 len 20480
1437 257/300 Read block 1861 len 36864
1438 258/300 Read block 3576 len 16384
1439 259/300 Read block 782 len 36864
1440 260/300 Read block 240 len 4096
1441 261/300 Write block 413 len 4096 data: 2
1442 262/300 Write block 1078 len 4096 data: 4
1443 263/300 Read block 2158 len 8192
1444 264/300 Read block 2097 len 8192
1445 265/300 Read block 1893 len 12288
1446 266/300 Read block 4606 len 4096
1447 267/300 Read block 4211 len 12288
1448 268/300 Read block 3017 len 28672
1449 269/300 Read block 1059 len 4096
1450 270/300 Read block 1614 len 32768
1451 271/300 Write block 4318 len 20480 data: 3 3 4 5 5
1452 272/300 Read block 3111 len 24576
1453 273/300 Read block 367 len 20480
1454 274/300 Write block 3109 len 4096 data: 2
1455 275/300 Read block 3268 len 28672
1456 276/300 Flush
1457 277/300 Read block 3892 len 4096
1458 278/300 Read block 2026 len 36864
1459 279/300 Write block 1673 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1460 280/300 Write block 216 len 28672 data: 2 2 2 2 2 2 2
1461 281/300 Read block 464 len 32768
1462 282/300 Write block 2293 len 20480 data: 2 2 2 2 2
1463 283/300 Read block 2222 len 40960
1464 284/300 Read block 2071 len 16384
1465 285/300 Write block 1147 len 36864 data: 2 2 2 2 3 3 3 3 3
1466 286/300 Write block 1240 len 20480 data: 3 3 3 3 2
1467 287/300 Read block 708 len 24576
1468 288/300 Read block 4216 len 36864
1469 289/300 Write block 1632 len 36864 data: 2 2 2 2 2 2 2 2 2
1470 290/300 Write block 4459 len 20480 data: 2 3 3 3 3
1471 291/300 Write block 4374 len 32768 data: 2 2 2 2 2 2 2 2
1472 292/300 Write block 3896 len 8192 data: 2 2
1473 293/300 Flush
1474 294/300 Read block 3652 len 4096
1475 295/300 Read block 2078 len 8192
1476 296/300 Write block 592 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1477 297/300 Write block 1961 len 16384 data: 2 2 2 3
1478 298/300 Write block 994 len 28672 data: 2 2 3 3 3 4 4
1479 299/300 Read block 2450 len 20480
1480 300/300 Read block 716 len 24576
1481 301/300 Read block 955 len 16384
1482 Main task has work to do, go find it
1483 got dsc Start(1)
1484 start 1
1485 Nov 01 16:50:48.833 INFO request completed, latency_us: 239, response_code: 204, uri: /start/cid/1, method: POST, req_id: af4f1ec2-2fa1-4724-8db8-544ab85ef11a, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
1486 Replay: started 1, returned:Ok(())
1487 ----------------------------------------------------------------
1488 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
1489 States: Active Offline Active
1490 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1491 1302 Acked Read 5 Done Sent Done false
1492 1303 Acked Read 10 Done Sent Done false
1493 1304 Acked Write 5 Done Sent Done false
1494 1305 Acked Write 4 Done Sent Done false
1495 1306 Acked Flush 0 Done Sent Done false
1496 1307 Acked Write 5 Done Sent Done false
1497 1308 Acked Read 9 Done Sent Done false
1498 1309 Acked Read 9 Done Sent Done false
1499 1310 Acked Write 10 Done Sent Done false
1500 1311 Acked Write 5 Done Sent Done false
1501 1312 Acked Flush 0 Done Sent Done false
1502 1313 Acked Read 1 Done Sent Done false
1503 1314 Acked Write 2 Done Sent Done false
1504 1315 Acked Write 3 Done Sent Done false
1505 1316 Acked Write 2 Done Sent Done false
1506 1317 Acked Read 6 Done Sent Done false
1507 1318 Acked Read 3 Done Sent Done false
1508 1319 Acked Write 1 Done Sent Done false
1509 1320 Acked Write 5 Done Sent Done false
1510 1321 Acked Write 8 Done Sent Done false
1511 1322 Acked Write 9 Done Sent Done false
1512 1323 Acked Write 7 Done Sent Done false
1513 1324 Acked Read 4 Done Sent Done false
1514 1325 Acked Read 4 Done Sent Done false
1515 1326 Acked Write 3 Done Sent Done false
1516 1327 Acked Read 9 Done Sent Done false
1517 1328 Acked Read 2 Done Sent Done false
1518 1329 Acked Write 4 Done Sent Done false
1519 1330 Acked Read 7 Done Sent Done false
1520 1331 Acked Write 3 Done Sent Done false
1521 1332 Acked Write 6 Done Sent Done false
1522 1333 Acked Write 6 Done Sent Done false
1523 1334 Acked Write 6 Done Sent Done false
1524 1335 Acked Write 6 Done Sent Done false
1525 1336 Acked Write 10 Done Sent Done false
1526 1337 Acked Flush 0 Done Sent Done false
1527 1338 Acked Read 6 Done Sent Done false
1528 1339 Acked Read 9 Done Sent Done false
1529 1340 Acked Flush 0 Done Sent Done false
1530 1341 Acked Flush 0 Done Sent Done false
1531 1342 Acked Write 6 Done Sent Done false
1532 1343 Acked Write 2 Done Sent Done false
1533 1344 Acked Flush 0 Done Sent Done false
1534 1345 Acked Read 1 Done Sent Done false
1535 1346 Acked Read 2 Done Sent Done false
1536 1347 Acked Flush 0 Done Sent Done false
1537 1348 Acked Read 2 Done Sent Done false
1538 1349 Acked Write 6 Done Sent Done false
1539 1350 Acked Read 10 Done Sent Done false
1540 1351 Acked Read 8 Done Sent Done false
1541 1352 Acked Flush 0 Done Sent Done false
1542 1353 Acked Read 2 Done Sent Done false
1543 1354 Acked Read 4 Done Sent Done false
1544 1355 Acked Write 6 Done Sent Done false
1545 1356 Acked Read 7 Done Sent Done false
1546 1357 Acked Read 10 Done Sent Done false
1547 1358 Acked Write 2 Done Sent Done false
1548 1359 Acked Write 5 Done Sent Done false
1549 1360 Acked Write 6 Done Sent Done false
1550 1361 Acked Write 6 Done Sent Done false
1551 1362 Acked Write 8 Done Sent Done false
1552 1363 Acked Write 7 Done Sent Done false
1553 1364 Acked Write 9 Done Sent Done false
1554 1365 Acked Read 9 Done Sent Done false
1555 1366 Acked Write 8 Done Sent Done false
1556 1367 Acked Write 2 Done Sent Done false
1557 1368 Acked Read 5 Done Sent Done false
1558 1369 Acked Read 1 Done Sent Done false
1559 1370 Acked Read 4 Done Sent Done false
1560 1371 Acked Write 6 Done Sent Done false
1561 1372 Acked Write 2 Done Sent Done false
1562 1373 Acked Write 10 Done Sent Done false
1563 1374 Acked Flush 0 Done Sent Done false
1564 1375 Acked Read 5 Done Sent Done false
1565 1376 Acked Flush 0 Done Sent Done false
1566 1377 Acked Read 1 Done Sent Done false
1567 1378 Acked Read 10 Done Sent Done false
1568 1379 Acked Flush 0 Done Sent Done false
1569 1380 Acked Write 6 Done Sent Done false
1570 1381 Acked Flush 0 Done Sent Done false
1571 1382 Acked Write 7 Done Sent Done false
1572 1383 Acked Write 4 Done Sent Done false
1573 1384 Acked Read 5 Done Sent Done false
1574 1385 Acked Write 8 Done Sent Done false
1575 1386 Acked Write 3 Done Sent Done false
1576 1387 Acked Write 3 Done Sent Done false
1577 1388 Acked Read 8 Done Sent Done false
1578 1389 Acked Write 4 Done Sent Done false
1579 1390 Acked Write 5 Done Sent Done false
1580 1391 Acked Read 8 Done Sent Done false
1581 1392 Acked Write 4 Done Sent Done false
1582 1393 Acked Read 1 Done Sent Done false
1583 1394 Acked Write 10 Done Sent Done false
1584 1395 Acked Write 2 Done Sent Done false
1585 1396 Acked Write 3 Done Sent Done false
1586 1397 Acked Read 9 Done Sent Done false
1587 1398 Acked Read 6 Done Sent Done false
1588 1399 Acked Read 4 Done Sent Done false
1589 1400 Acked Read 2 Done Sent Done false
1590 1401 Acked Write 6 Done Sent Done false
1591 1402 Acked Write 4 Done Sent Done false
1592 1403 Acked Write 3 Done Sent Done false
1593 1404 Acked Read 9 Done Sent Done false
1594 1405 Acked Read 6 Done Sent Done false
1595 1406 Acked Read 4 Done Sent Done false
1596 1407 Acked Read 4 Done Sent Done false
1597 1408 Acked Read 8 Done Sent Done false
1598 1409 Acked Read 4 Done Sent Done false
1599 1410 Acked Read 2 Done Sent Done false
1600 1411 Acked Write 7 Done Sent Done false
1601 1412 Acked Read 4 Done Sent Done false
1602 1413 Acked Write 2 Done Sent Done false
1603 1414 Acked Write 1 Done Sent Done false
1604 1415 Acked Write 7 Done Sent Done false
1605 1416 Acked Write 8 Done Sent Done false
1606 1417 Acked Flush 0 Done Sent Done false
1607 1418 Acked Write 5 Done Sent Done false
1608 1419 Acked Read 5 Done Sent Done false
1609 1420 Acked Read 6 Done Sent Done false
1610 1421 Acked Read 10 Done Sent Done false
1611 1422 Acked Write 9 Done Sent Done false
1612 1423 Acked Read 9 Done Sent Done false
1613 1424 Acked Write 2 Done Sent Done false
1614 1425 Acked Read 1 Done Sent Done false
1615 1426 Acked Write 8 Done Sent Done false
1616 1427 Acked Read 4 Done Sent Done false
1617 1428 Acked Read 5 Done Sent Done false
1618 1429 Acked Write 1 Done Sent Done false
1619 1430 Acked Read 2 Done Sent Done false
1620 1431 Acked Read 8 Done Sent Done false
1621 1432 Acked Read 1 Done Sent Done false
1622 1433 Acked Read 3 Done Sent Done false
1623 1434 Acked Read 5 Done Sent Done false
1624 1435 Acked Read 9 Done Sent Done false
1625 1436 Acked Read 5 Done Sent Done false
1626 1437 Acked Read 8 Done Sent Done false
1627 1438 Acked Read 10 Done Sent Done false
1628 1439 Acked Read 1 Done Sent Done false
1629 1440 Acked Read 7 Done Sent Done false
1630 1441 Acked Read 7 Done Sent Done false
1631 1442 Acked Write 2 Done Sent Done false
1632 1443 Acked Write 2 Done Sent Done false
1633 1444 Acked Read 3 Done Sent Done false
1634 1445 Acked Write 4 Done Sent Done false
1635 1446 Acked Write 9 Done Sent Done false
1636 1447 Acked Write 10 Done Sent Done false
1637 1448 Acked Write 7 Done Sent Done false
1638 1449 Acked Write 10 Done Sent Done false
1639 1450 Acked Flush 0 Done Sent Done false
1640 1451 Acked Read 7 Done Sent Done false
1641 1452 Acked Write 5 Done Sent Done false
1642 1453 Acked Read 7 Done Sent Done false
1643 1454 Acked Write 5 Done Sent Done false
1644 1455 Acked Read 9 Done Sent Done false
1645 1456 Acked Flush 0 Done Sent Done false
1646 1457 Acked Write 3 Done Sent Done false
1647 1458 Acked Read 1 Done Sent Done false
1648 1459 Acked Flush 0 Done Sent Done false
1649 1460 Acked Read 3 Done Sent Done false
1650 1461 Acked Write 8 Done Sent Done false
1651 1462 Acked Write 8 Done Sent Done false
1652 1463 Acked Read 9 Done Sent Done false
1653 1464 Acked Write 5 Done Sent Done false
1654 1465 Acked Write 1 Done Sent Done false
1655 1466 Acked Read 2 Done Sent Done false
1656 1467 Acked Write 6 Done Sent Done false
1657 1468 Acked Read 9 Done Sent Done false
1658 1469 Acked Flush 0 Done Sent Done false
1659 1470 Acked Write 6 Done Sent Done false
1660 1471 Acked Read 10 Done Sent Done false
1661 1472 Acked Read 6 Done Sent Done false
1662 1473 Acked Read 2 Done Sent Done false
1663 1474 Acked Read 2 Done Sent Done false
1664 1475 Acked Read 6 Done Sent Done false
1665 1476 Acked Read 9 Done Sent Done false
1666 1477 Acked Write 8 Done Sent Done false
1667 1478 Acked Read 6 Done Sent Done false
1668 1479 Acked Read 2 Done Sent Done false
1669 1480 Acked Read 6 Done Sent Done false
1670 1481 Acked Write 5 Done Sent Done false
1671 1482 Acked Read 4 Done Sent Done false
1672 1483 Acked Write 5 Done Sent Done false
1673 1484 Acked Write 1 Done Sent Done false
1674 1485 Acked Write 9 Done Sent Done false
1675 1486 Acked Write 6 Done Sent Done false
1676 1487 Acked Write 10 Done Sent Done false
1677 1488 Acked Read 9 Done Sent Done false
1678 1489 Acked Read 10 Done Sent Done false
1679 1490 Acked Write 3 Done Sent Done false
1680 1491 Acked Write 10 Done Sent Done false
1681 1492 Acked Write 8 Done Sent Done false
1682 1493 Acked Write 4 Done Sent Done false
1683 1494 Acked Flush 0 Done Sent Done false
1684 1495 Acked Read 10 Done Sent Done false
1685 1496 Acked Write 1 Done Sent Done false
1686 1497 Acked Write 4 Done Sent Done false
1687 1498 Acked Read 4 Done Sent Done false
1688 1499 Acked Read 4 Done Sent Done false
1689 1500 Acked Write 10 Done Sent Done false
1690 1501 Acked Write 3 Done Sent Done false
1691 1502 Acked Write 2 Done Sent Done false
1692 1503 Acked Write 2 Done Sent Done false
1693 1504 Acked Write 6 Done Sent Done false
1694 1505 Acked Flush 0 Done Sent Done false
1695 1506 Acked Read 4 Done Sent Done false
1696 1507 Acked Read 6 Done Sent Done false
1697 1508 Acked Flush 0 Done Sent Done false
1698 1509 Acked Read 2 Done Sent Done false
1699 1510 Acked Flush 0 Done Sent Done false
1700 1511 Acked Read 7 Done Sent Done false
1701 1512 Acked Write 1 Done Sent Done false
1702 1513 Acked Write 10 Done Sent Done false
1703 1514 Acked Read 2 Done Sent Done false
1704 1515 Acked Flush 0 Done Sent Done false
1705 1516 Acked Write 4 Done Sent Done false
1706 1517 Acked Flush 0 Done Sent Done false
1707 1518 Acked Write 2 Done Sent Done false
1708 1519 Acked Read 5 Done Sent Done false
1709 1520 Acked Read 6 Done Sent Done false
1710 1521 Acked Read 2 Done Sent Done false
1711 1522 Acked Read 5 Done Sent Done false
1712 1523 Acked Read 1 Done Sent Done false
1713 1524 Acked Flush 0 Done Sent Done false
1714 1525 Acked Read 4 Done Sent Done false
1715 1526 Acked Write 6 Done Sent Done false
1716 1527 Acked Read 9 Done Sent Done false
1717 1528 Acked Write 3 Done Sent Done false
1718 1529 Acked Write 5 Done Sent Done false
1719 1530 Acked Read 3 Done Sent Done false
1720 1531 Acked Read 3 Done Sent Done false
1721 1532 Acked Write 9 Done Sent Done false
1722 1533 Acked Read 9 Done Sent Done false
1723 1534 Acked Read 4 Done Sent Done false
1724 1535 Acked Write 10 Done Sent Done false
1725 1536 Acked Write 1 Done Sent Done false
1726 1537 Acked Read 5 Done Sent Done false
1727 1538 Acked Read 2 Done Sent Done false
1728 1539 Acked Write 6 Done Sent Done false
1729 1540 Acked Write 2 Done Sent Done false
1730 1541 Acked Flush 0 Done Sent Done false
1731 1542 Acked Write 7 Done Sent Done false
1732 1543 Acked Write 4 Done Sent Done false
1733 1544 Acked Read 9 Done Sent Done false
1734 1545 Acked Read 6 Done Sent Done false
1735 1546 Acked Write 2 Done Sent Done false
1736 1547 Acked Read 1 Done Sent Done false
1737 1548 Acked Read 10 Done Sent Done false
1738 1549 Acked Write 3 Done Sent Done false
1739 1550 Acked Read 9 Done Sent Done false
1740 1551 Acked Write 3 Done Sent Done false
1741 1552 Acked Write 5 Done Sent Done false
1742 1553 Acked Read 2 Done Sent Done false
1743 1554 Acked Write 5 Done Sent Done false
1744 1555 Acked Write 3 Done Sent Done false
1745 1556 Acked Read 10 Done Sent Done false
1746 1557 Acked Write 2 Done Sent Done false
1747 1558 Acked Read 5 Done Sent Done false
1748 1559 Acked Read 9 Done Sent Done false
1749 1560 Acked Read 4 Done Sent Done false
1750 1561 Acked Read 9 Done Sent Done false
1751 1562 Acked Read 1 Done Sent Done false
1752 1563 Acked Write 1 Done Sent Done false
1753 1564 Acked Write 1 Done Sent Done false
1754 1565 Acked Read 2 Done Sent Done false
1755 1566 Acked Read 2 Done Sent Done false
1756 1567 Acked Read 3 Done Sent Done false
1757 1568 Acked Read 1 Done Sent Done false
1758 1569 Acked Read 3 Done Sent Done false
1759 1570 Acked Read 7 Done Sent Done false
1760 1571 Acked Read 1 Done Sent Done false
1761 1572 Acked Read 8 Done Sent Done false
1762 1573 Acked Write 5 Done Sent Done false
1763 1574 Acked Read 6 Done Sent Done false
1764 1575 Acked Read 5 Done Sent Done false
1765 1576 Acked Write 1 Done Sent Done false
1766 1577 Acked Read 7 Done Sent Done false
1767 1578 Acked Flush 0 Done Sent Done false
1768 1579 Acked Read 1 Done Sent Done false
1769 1580 Acked Read 9 Done Sent Done false
1770 1581 Acked Write 10 Done Sent Done false
1771 1582 Acked Write 7 Done Sent Done false
1772 1583 Acked Read 8 Done Sent Done false
1773 1584 Acked Write 5 Done Sent Done false
1774 1585 Acked Read 10 Done Sent Done false
1775 1586 Acked Read 4 Done Sent Done false
1776 1587 Acked Write 9 Done Sent Done false
1777 1588 Acked Write 5 Done Sent Done false
1778 1589 Acked Read 6 Done Sent Done false
1779 1590 Acked Read 9 Done Sent Done false
1780 1591 Acked Write 9 Done Sent Done false
1781 1592 Acked Write 5 Done Sent Done false
1782 1593 Acked Write 8 Done Sent Done false
1783 1594 Acked Write 2 Done Sent Done false
1784 1595 Acked Flush 0 Done Sent Done false
1785 1596 Acked Read 1 Done Sent Done false
1786 1597 Acked Read 2 Done Sent Done false
1787 1598 Acked Write 10 Done Sent Done false
1788 1599 Acked Write 4 Done Sent Done false
1789 1600 Acked Write 7 Done Sent Done false
1790 1601 Acked Read 5 Done Sent Done false
1791 1602 Acked Read 6 Done Sent Done false
1792 1603 Acked Read 4 Done Sent Done false
1793 STATES DS:0 DS:1 DS:2 TOTAL
1794 Sent 0 302 0 302
1795 Done 302 0 302 604
1796 Skipped 0 0 0 0
1797 Error 0 0 0 0
1798 Last Flush: 1595 1301 1595
1799 Downstairs last five completed: 1301 1300 1299 1298 1297
1800 Upstairs last five completed: 1603 1602 1601 1600 1599
1801 CLIENT: Up:0 ds:302 act:2
1802 [8820] got start action
1803 [8820] I am going to restart
1804 Starting downstairs at port 8820
1805 Make output file at "/tmp/dsc/downstairs-8820.txt"
1806 [8820][1] reports Starting
1807 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1200)
1808 [8820][1] reports Running
1809 ----------------------------------------------------------------
1810 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
1811 States: Active Offline Active
1812 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1813 1302 Acked Read 5 Done Sent Done false
1814 1303 Acked Read 10 Done Sent Done false
1815 1304 Acked Write 5 Done Sent Done false
1816 1305 Acked Write 4 Done Sent Done false
1817 1306 Acked Flush 0 Done Sent Done false
1818 1307 Acked Write 5 Done Sent Done false
1819 1308 Acked Read 9 Done Sent Done false
1820 1309 Acked Read 9 Done Sent Done false
1821 1310 Acked Write 10 Done Sent Done false
1822 1311 Acked Write 5 Done Sent Done false
1823 1312 Acked Flush 0 Done Sent Done false
1824 1313 Acked Read 1 Done Sent Done false
1825 1314 Acked Write 2 Done Sent Done false
1826 1315 Acked Write 3 Done Sent Done false
1827 1316 Acked Write 2 Done Sent Done false
1828 1317 Acked Read 6 Done Sent Done false
1829 1318 Acked Read 3 Done Sent Done false
1830 1319 Acked Write 1 Done Sent Done false
1831 1320 Acked Write 5 Done Sent Done false
1832 1321 Acked Write 8 Done Sent Done false
1833 1322 Acked Write 9 Done Sent Done false
1834 1323 Acked Write 7 Done Sent Done false
1835 1324 Acked Read 4 Done Sent Done false
1836 1325 Acked Read 4 Done Sent Done false
1837 1326 Acked Write 3 Done Sent Done false
1838 1327 Acked Read 9 Done Sent Done false
1839 1328 Acked Read 2 Done Sent Done false
1840 1329 Acked Write 4 Done Sent Done false
1841 1330 Acked Read 7 Done Sent Done false
1842 1331 Acked Write 3 Done Sent Done false
1843 1332 Acked Write 6 Done Sent Done false
1844 1333 Acked Write 6 Done Sent Done false
1845 1334 Acked Write 6 Done Sent Done false
1846 1335 Acked Write 6 Done Sent Done false
1847 1336 Acked Write 10 Done Sent Done false
1848 1337 Acked Flush 0 Done Sent Done false
1849 1338 Acked Read 6 Done Sent Done false
1850 1339 Acked Read 9 Done Sent Done false
1851 1340 Acked Flush 0 Done Sent Done false
1852 1341 Acked Flush 0 Done Sent Done false
1853 1342 Acked Write 6 Done Sent Done false
1854 1343 Acked Write 2 Done Sent Done false
1855 1344 Acked Flush 0 Done Sent Done false
1856 1345 Acked Read 1 Done Sent Done false
1857 1346 Acked Read 2 Done Sent Done false
1858 1347 Acked Flush 0 Done Sent Done false
1859 1348 Acked Read 2 Done Sent Done false
1860 1349 Acked Write 6 Done Sent Done false
1861 1350 Acked Read 10 Done Sent Done false
1862 1351 Acked Read 8 Done Sent Done false
1863 1352 Acked Flush 0 Done Sent Done false
1864 1353 Acked Read 2 Done Sent Done false
1865 1354 Acked Read 4 Done Sent Done false
1866 1355 Acked Write 6 Done Sent Done false
1867 1356 Acked Read 7 Done Sent Done false
1868 1357 Acked Read 10 Done Sent Done false
1869 1358 Acked Write 2 Done Sent Done false
1870 1359 Acked Write 5 Done Sent Done false
1871 1360 Acked Write 6 Done Sent Done false
1872 1361 Acked Write 6 Done Sent Done false
1873 1362 Acked Write 8 Done Sent Done false
1874 1363 Acked Write 7 Done Sent Done false
1875 1364 Acked Write 9 Done Sent Done false
1876 1365 Acked Read 9 Done Sent Done false
1877 1366 Acked Write 8 Done Sent Done false
1878 1367 Acked Write 2 Done Sent Done false
1879 1368 Acked Read 5 Done Sent Done false
1880 1369 Acked Read 1 Done Sent Done false
1881 1370 Acked Read 4 Done Sent Done false
1882 1371 Acked Write 6 Done Sent Done false
1883 1372 Acked Write 2 Done Sent Done false
1884 1373 Acked Write 10 Done Sent Done false
1885 1374 Acked Flush 0 Done Sent Done false
1886 1375 Acked Read 5 Done Sent Done false
1887 1376 Acked Flush 0 Done Sent Done false
1888 1377 Acked Read 1 Done Sent Done false
1889 1378 Acked Read 10 Done Sent Done false
1890 1379 Acked Flush 0 Done Sent Done false
1891 1380 Acked Write 6 Done Sent Done false
1892 1381 Acked Flush 0 Done Sent Done false
1893 1382 Acked Write 7 Done Sent Done false
1894 1383 Acked Write 4 Done Sent Done false
1895 1384 Acked Read 5 Done Sent Done false
1896 1385 Acked Write 8 Done Sent Done false
1897 1386 Acked Write 3 Done Sent Done false
1898 1387 Acked Write 3 Done Sent Done false
1899 1388 Acked Read 8 Done Sent Done false
1900 1389 Acked Write 4 Done Sent Done false
1901 1390 Acked Write 5 Done Sent Done false
1902 1391 Acked Read 8 Done Sent Done false
1903 1392 Acked Write 4 Done Sent Done false
1904 1393 Acked Read 1 Done Sent Done false
1905 1394 Acked Write 10 Done Sent Done false
1906 1395 Acked Write 2 Done Sent Done false
1907 1396 Acked Write 3 Done Sent Done false
1908 1397 Acked Read 9 Done Sent Done false
1909 1398 Acked Read 6 Done Sent Done false
1910 1399 Acked Read 4 Done Sent Done false
1911 1400 Acked Read 2 Done Sent Done false
1912 1401 Acked Write 6 Done Sent Done false
1913 1402 Acked Write 4 Done Sent Done false
1914 1403 Acked Write 3 Done Sent Done false
1915 1404 Acked Read 9 Done Sent Done false
1916 1405 Acked Read 6 Done Sent Done false
1917 1406 Acked Read 4 Done Sent Done false
1918 1407 Acked Read 4 Done Sent Done false
1919 1408 Acked Read 8 Done Sent Done false
1920 1409 Acked Read 4 Done Sent Done false
1921 1410 Acked Read 2 Done Sent Done false
1922 1411 Acked Write 7 Done Sent Done false
1923 1412 Acked Read 4 Done Sent Done false
1924 1413 Acked Write 2 Done Sent Done false
1925 1414 Acked Write 1 Done Sent Done false
1926 1415 Acked Write 7 Done Sent Done false
1927 1416 Acked Write 8 Done Sent Done false
1928 1417 Acked Flush 0 Done Sent Done false
1929 1418 Acked Write 5 Done Sent Done false
1930 1419 Acked Read 5 Done Sent Done false
1931 1420 Acked Read 6 Done Sent Done false
1932 1421 Acked Read 10 Done Sent Done false
1933 1422 Acked Write 9 Done Sent Done false
1934 1423 Acked Read 9 Done Sent Done false
1935 1424 Acked Write 2 Done Sent Done false
1936 1425 Acked Read 1 Done Sent Done false
1937 1426 Acked Write 8 Done Sent Done false
1938 1427 Acked Read 4 Done Sent Done false
1939 1428 Acked Read 5 Done Sent Done false
1940 1429 Acked Write 1 Done Sent Done false
1941 1430 Acked Read 2 Done Sent Done false
1942 1431 Acked Read 8 Done Sent Done false
1943 1432 Acked Read 1 Done Sent Done false
1944 1433 Acked Read 3 Done Sent Done false
1945 1434 Acked Read 5 Done Sent Done false
1946 1435 Acked Read 9 Done Sent Done false
1947 1436 Acked Read 5 Done Sent Done false
1948 1437 Acked Read 8 Done Sent Done false
1949 1438 Acked Read 10 Done Sent Done false
1950 1439 Acked Read 1 Done Sent Done false
1951 1440 Acked Read 7 Done Sent Done false
1952 1441 Acked Read 7 Done Sent Done false
1953 1442 Acked Write 2 Done Sent Done false
1954 1443 Acked Write 2 Done Sent Done false
1955 1444 Acked Read 3 Done Sent Done false
1956 1445 Acked Write 4 Done Sent Done false
1957 1446 Acked Write 9 Done Sent Done false
1958 1447 Acked Write 10 Done Sent Done false
1959 1448 Acked Write 7 Done Sent Done false
1960 1449 Acked Write 10 Done Sent Done false
1961 1450 Acked Flush 0 Done Sent Done false
1962 1451 Acked Read 7 Done Sent Done false
1963 1452 Acked Write 5 Done Sent Done false
1964 1453 Acked Read 7 Done Sent Done false
1965 1454 Acked Write 5 Done Sent Done false
1966 1455 Acked Read 9 Done Sent Done false
1967 1456 Acked Flush 0 Done Sent Done false
1968 1457 Acked Write 3 Done Sent Done false
1969 1458 Acked Read 1 Done Sent Done false
1970 1459 Acked Flush 0 Done Sent Done false
1971 1460 Acked Read 3 Done Sent Done false
1972 1461 Acked Write 8 Done Sent Done false
1973 1462 Acked Write 8 Done Sent Done false
1974 1463 Acked Read 9 Done Sent Done false
1975 1464 Acked Write 5 Done Sent Done false
1976 1465 Acked Write 1 Done Sent Done false
1977 1466 Acked Read 2 Done Sent Done false
1978 1467 Acked Write 6 Done Sent Done false
1979 1468 Acked Read 9 Done Sent Done false
1980 1469 Acked Flush 0 Done Sent Done false
1981 1470 Acked Write 6 Done Sent Done false
1982 1471 Acked Read 10 Done Sent Done false
1983 1472 Acked Read 6 Done Sent Done false
1984 1473 Acked Read 2 Done Sent Done false
1985 1474 Acked Read 2 Done Sent Done false
1986 1475 Acked Read 6 Done Sent Done false
1987 1476 Acked Read 9 Done Sent Done false
1988 1477 Acked Write 8 Done Sent Done false
1989 1478 Acked Read 6 Done Sent Done false
1990 1479 Acked Read 2 Done Sent Done false
1991 1480 Acked Read 6 Done Sent Done false
1992 1481 Acked Write 5 Done Sent Done false
1993 1482 Acked Read 4 Done Sent Done false
1994 1483 Acked Write 5 Done Sent Done false
1995 1484 Acked Write 1 Done Sent Done false
1996 1485 Acked Write 9 Done Sent Done false
1997 1486 Acked Write 6 Done Sent Done false
1998 1487 Acked Write 10 Done Sent Done false
1999 1488 Acked Read 9 Done Sent Done false
2000 1489 Acked Read 10 Done Sent Done false
2001 1490 Acked Write 3 Done Sent Done false
2002 1491 Acked Write 10 Done Sent Done false
2003 1492 Acked Write 8 Done Sent Done false
2004 1493 Acked Write 4 Done Sent Done false
2005 1494 Acked Flush 0 Done Sent Done false
2006 1495 Acked Read 10 Done Sent Done false
2007 1496 Acked Write 1 Done Sent Done false
2008 1497 Acked Write 4 Done Sent Done false
2009 1498 Acked Read 4 Done Sent Done false
2010 1499 Acked Read 4 Done Sent Done false
2011 1500 Acked Write 10 Done Sent Done false
2012 1501 Acked Write 3 Done Sent Done false
2013 1502 Acked Write 2 Done Sent Done false
2014 1503 Acked Write 2 Done Sent Done false
2015 1504 Acked Write 6 Done Sent Done false
2016 1505 Acked Flush 0 Done Sent Done false
2017 1506 Acked Read 4 Done Sent Done false
2018 1507 Acked Read 6 Done Sent Done false
2019 1508 Acked Flush 0 Done Sent Done false
2020 1509 Acked Read 2 Done Sent Done false
2021 1510 Acked Flush 0 Done Sent Done false
2022 1511 Acked Read 7 Done Sent Done false
2023 1512 Acked Write 1 Done Sent Done false
2024 1513 Acked Write 10 Done Sent Done false
2025 1514 Acked Read 2 Done Sent Done false
2026 1515 Acked Flush 0 Done Sent Done false
2027 1516 Acked Write 4 Done Sent Done false
2028 1517 Acked Flush 0 Done Sent Done false
2029 1518 Acked Write 2 Done Sent Done false
2030 1519 Acked Read 5 Done Sent Done false
2031 1520 Acked Read 6 Done Sent Done false
2032 1521 Acked Read 2 Done Sent Done false
2033 1522 Acked Read 5 Done Sent Done false
2034 1523 Acked Read 1 Done Sent Done false
2035 1524 Acked Flush 0 Done Sent Done false
2036 1525 Acked Read 4 Done Sent Done false
2037 1526 Acked Write 6 Done Sent Done false
2038 1527 Acked Read 9 Done Sent Done false
2039 1528 Acked Write 3 Done Sent Done false
2040 1529 Acked Write 5 Done Sent Done false
2041 1530 Acked Read 3 Done Sent Done false
2042 1531 Acked Read 3 Done Sent Done false
2043 1532 Acked Write 9 Done Sent Done false
2044 1533 Acked Read 9 Done Sent Done false
2045 1534 Acked Read 4 Done Sent Done false
2046 1535 Acked Write 10 Done Sent Done false
2047 1536 Acked Write 1 Done Sent Done false
2048 1537 Acked Read 5 Done Sent Done false
2049 1538 Acked Read 2 Done Sent Done false
2050 1539 Acked Write 6 Done Sent Done false
2051 1540 Acked Write 2 Done Sent Done false
2052 1541 Acked Flush 0 Done Sent Done false
2053 1542 Acked Write 7 Done Sent Done false
2054 1543 Acked Write 4 Done Sent Done false
2055 1544 Acked Read 9 Done Sent Done false
2056 1545 Acked Read 6 Done Sent Done false
2057 1546 Acked Write 2 Done Sent Done false
2058 1547 Acked Read 1 Done Sent Done false
2059 1548 Acked Read 10 Done Sent Done false
2060 1549 Acked Write 3 Done Sent Done false
2061 1550 Acked Read 9 Done Sent Done false
2062 1551 Acked Write 3 Done Sent Done false
2063 1552 Acked Write 5 Done Sent Done false
2064 1553 Acked Read 2 Done Sent Done false
2065 1554 Acked Write 5 Done Sent Done false
2066 1555 Acked Write 3 Done Sent Done false
2067 1556 Acked Read 10 Done Sent Done false
2068 1557 Acked Write 2 Done Sent Done false
2069 1558 Acked Read 5 Done Sent Done false
2070 1559 Acked Read 9 Done Sent Done false
2071 1560 Acked Read 4 Done Sent Done false
2072 1561 Acked Read 9 Done Sent Done false
2073 1562 Acked Read 1 Done Sent Done false
2074 1563 Acked Write 1 Done Sent Done false
2075 1564 Acked Write 1 Done Sent Done false
2076 1565 Acked Read 2 Done Sent Done false
2077 1566 Acked Read 2 Done Sent Done false
2078 1567 Acked Read 3 Done Sent Done false
2079 1568 Acked Read 1 Done Sent Done false
2080 1569 Acked Read 3 Done Sent Done false
2081 1570 Acked Read 7 Done Sent Done false
2082 1571 Acked Read 1 Done Sent Done false
2083 1572 Acked Read 8 Done Sent Done false
2084 1573 Acked Write 5 Done Sent Done false
2085 1574 Acked Read 6 Done Sent Done false
2086 1575 Acked Read 5 Done Sent Done false
2087 1576 Acked Write 1 Done Sent Done false
2088 1577 Acked Read 7 Done Sent Done false
2089 1578 Acked Flush 0 Done Sent Done false
2090 1579 Acked Read 1 Done Sent Done false
2091 1580 Acked Read 9 Done Sent Done false
2092 1581 Acked Write 10 Done Sent Done false
2093 1582 Acked Write 7 Done Sent Done false
2094 1583 Acked Read 8 Done Sent Done false
2095 1584 Acked Write 5 Done Sent Done false
2096 1585 Acked Read 10 Done Sent Done false
2097 1586 Acked Read 4 Done Sent Done false
2098 1587 Acked Write 9 Done Sent Done false
2099 1588 Acked Write 5 Done Sent Done false
2100 1589 Acked Read 6 Done Sent Done false
2101 1590 Acked Read 9 Done Sent Done false
2102 1591 Acked Write 9 Done Sent Done false
2103 1592 Acked Write 5 Done Sent Done false
2104 1593 Acked Write 8 Done Sent Done false
2105 1594 Acked Write 2 Done Sent Done false
2106 1595 Acked Flush 0 Done Sent Done false
2107 1596 Acked Read 1 Done Sent Done false
2108 1597 Acked Read 2 Done Sent Done false
2109 1598 Acked Write 10 Done Sent Done false
2110 1599 Acked Write 4 Done Sent Done false
2111 1600 Acked Write 7 Done Sent Done false
2112 1601 Acked Read 5 Done Sent Done false
2113 1602 Acked Read 6 Done Sent Done false
2114 1603 Acked Read 4 Done Sent Done false
2115 1604 Acked Flush 0 Done Sent Done false
2116 STATES DS:0 DS:1 DS:2 TOTAL
2117 Sent 0 303 0 303
2118 Done 303 0 303 606
2119 Skipped 0 0 0 0
2120 Error 0 0 0 0
2121 Last Flush: 1604 1301 1604
2122 Downstairs last five completed: 1301 1300 1299 1298 1297
2123 Upstairs last five completed: 1604 1603 1602 1601 1600
2124 CLIENT: Up:0 ds:303 act:2
21252024-11-01T16:50:54.561ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21262024-11-01T16:50:54.562ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42721) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21272024-11-01T16:50:54.563ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21282024-11-01T16:50:54.563ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21292024-11-01T16:50:54.563ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21302024-11-01T16:50:54.563ZINFOcrucible: send last flush ID to this DS: 1301 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21312024-11-01T16:50:54.563ZINFOcrucible: Replied this last flush ID: 1301 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21322024-11-01T16:50:54.564ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21332024-11-01T16:50:54.564ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21342024-11-01T16:50:54.564ZINFOcrucible: [1] client re-new 303 jobs since flush 1301 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21352024-11-01T16:50:54.564ZINFOcrucible: [1] Replayed 303 jobs since flush: 1301 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
2136 ----------------------------------------------------------------
2137 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2138 Downstairs last five completed: 1604 1603 1602 1601 1600
2139 Upstairs last five completed: 1604 1603 1602 1601 1600
2140 CLIENT: Up:0 ds:0 act:3
2141 Replay: All jobs finished
2142 Main task has work to do, go find it
2143 got dsc Stop(0)
2144 stop 0
2145 Nov 01 16:50:56.838 INFO request completed, latency_us: 348, response_code: 204, uri: /stop/cid/0, [8810] Got stop action so:false kr:false
2146 method: POST, req_id: 8c9465fa-6652-47eb-ab7c-64e205ab67d2, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
21472024-11-01T16:50:56.839ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21482024-11-01T16:50:56.839ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21492024-11-01T16:50:56.839ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21502024-11-01T16:50:56.839ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
2151 Nov 01 16:50:56.839 INFO request completed, latency_us: 199, response_code: 200, uri: /state/cid/0, method: GET, req_id: 8e9b9bcb-4dc2-4a77-8440-8a3b806b8798, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
21522024-11-01T16:50:56.839ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
2153 [8810] Exited with: ExitStatus(unix_wait_status(9))
2154 [8810][0] reports Exit
2155 Nov 01 16:51:00.841 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/0, method: GET, req_id: 91cbc6ea-0f64-4699-b0b6-d1cdc1f621d1, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
2156 001/300 Write block 2488 len 16384 data: 3 3 2 3
2157 002/300 Write block 3747 len 8192 data: 2 2
2158 003/300 Read block 1850 len 32768
2159 004/300 Write block 2739 len 28672 data: 3 3 4 3 3 3 3
2160 005/300 Write block 3047 len 8192 data: 3 3
2161 006/300 Write block 1502 len 28672 data: 2 2 2 2 2 2 2
2162 007/300 Write block 2807 len 36864 data: 2 2 2 2 3 3 3 3 4
2163 008/300 Read block 2746 len 40960
2164 009/300 Read block 613 len 20480
2165 010/300 Read block 2385 len 32768
2166 011/300 Read block 1500 len 28672
2167 012/300 Read block 2725 len 20480
2168 013/300 Write block 1543 len 36864 data: 3 2 2 2 2 2 2 2 2
2169 014/300 Flush
2170 015/300 Read block 3065 len 16384
2171 016/300 Read block 3527 len 16384
2172 017/300 Write block 4031 len 4096 data: 2
2173 018/300 Read block 3271 len 20480
2174 019/300 Read block 4087 len 28672
2175 020/300 Write block 3870 len 16384 data: 3 3 3 3
2176 021/300 Read block 1998 len 12288
2177 022/300 Write block 528 len 28672 data: 2 3 3 3 3 3 3
2178 023/300 Write block 478 len 16384 data: 2 2 2 2
2179 024/300 Read block 2283 len 12288
2180 025/300 Read block 600 len 24576
2181 026/300 Read block 3770 len 40960
2182 027/300 Read block 4652 len 20480
2183 028/300 Write block 3105 len 36864 data: 2 2 2 2 3 2 2 2 2
2184 029/300 Write block 2281 len 16384 data: 2 2 2 3
2185 030/300 Write block 845 len 40960 data: 3 3 3 3 3 3 3 3 2 2
2186 031/300 Write block 1382 len 12288 data: 3 3 2
2187 032/300 Write block 2545 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2188 033/300 Write block 3859 len 28672 data: 3 3 3 3 3 3 2
2189 034/300 Write block 3879 len 8192 data: 2 2
2190 035/300 Read block 3612 len 20480
2191 036/300 Read block 4756 len 36864
2192 037/300 Read block 2777 len 28672
2193 038/300 Read block 2251 len 24576
2194 039/300 Write block 251 len 32768 data: 2 2 2 2 2 2 2 2
2195 040/300 Read block 311 len 28672
2196 041/300 Flush
2197 042/300 Read block 2269 len 16384
2198 043/300 Read block 1177 len 4096
2199 044/300 Read block 3208 len 24576
2200 045/300 Read block 3214 len 4096
2201 046/300 Read block 858 len 24576
2202 047/300 Read block 2193 len 24576
2203 048/300 Write block 860 len 20480 data: 3 2 2 2 2
2204 049/300 Write block 4813 len 24576 data: 2 2 2 2 2 2
2205 050/300 Write block 2428 len 36864 data: 2 2 2 2 2 2 2 2 2
2206 051/300 Write block 4504 len 32768 data: 2 2 2 2 2 2 2 2
2207 052/300 Write block 1369 len 8192 data: 2 2
2208 053/300 Write block 4951 len 24576 data: 2 2 3 3 3 3
2209 054/300 Read block 257 len 28672
2210 055/300 Write block 2712 len 8192 data: 4 4
2211 056/300 Write block 2270 len 24576 data: 2 2 2 2 2 2
2212 057/300 Read block 4879 len 40960
2213 058/300 Write block 1782 len 20480 data: 2 2 2 2 2
2214 059/300 Write block 2956 len 32768 data: 3 2 2 2 2 2 2 2
2215 060/300 Write block 3793 len 16384 data: 3 3 3 3
2216 061/300 Write block 2873 len 16384 data: 2 2 3 3
2217 062/300 Read block 2927 len 4096
2218 063/300 Flush
2219 064/300 Read block 3779 len 4096
2220 065/300 Write block 3790 len 32768 data: 3 3 3 4 4 4 4 2
2221 066/300 Read block 1894 len 20480
2222 067/300 Write block 4422 len 36864 data: 2 2 2 2 2 2 2 2 2
2223 068/300 Read block 4768 len 28672
2224 069/300 Read block 2629 len 40960
2225 070/300 Read block 4271 len 40960
2226 071/300 Write block 1922 len 12288 data: 3 3 3
2227 072/300 Read block 4342 len 40960
2228 073/300 Read block 4020 len 4096
2229 074/300 Read block 4982 len 12288
2230 075/300 Write block 1735 len 24576 data: 2 2 2 2 3 3
2231 076/300 Flush
2232 077/300 Write block 3338 len 12288 data: 2 2 2
2233 078/300 Write block 4073 len 28672 data: 3 3 2 2 2 2 2
2234 079/300 Read block 651 len 28672
2235 080/300 Write block 793 len 32768 data: 2 2 2 2 2 2 3 4
2236 081/300 Write block 4566 len 4096 data: 2
2237 082/300 Write block 1510 len 16384 data: 2 2 2 2
2238 083/300 Read block 3291 len 8192
2239 084/300 Flush
2240 085/300 Read block 1737 len 40960
2241 086/300 Write block 1403 len 32768 data: 2 2 2 2 2 2 2 2
2242 087/300 Read block 1527 len 4096
2243 088/300 Write block 2288 len 40960 data: 2 2 2 2 2 3 3 3 3 3
2244 089/300 Write block 1936 len 12288 data: 2 2 2
2245 090/300 Write block 1489 len 8192 data: 2 2
2246 091/300 Read block 40 len 32768
2247 092/300 Read block 2473 len 28672
2248 093/300 Read block 1321 len 4096
2249 094/300 Read block 4951 len 24576
2250 095/300 Read block 103 len 40960
2251 096/300 Read block 3721 len 8192
2252 097/300 Read block 4270 len 12288
2253 098/300 Flush
2254 099/300 Read block 3104 len 4096
2255 100/300 Flush
2256 101/300 Read block 4819 len 40960
2257 102/300 Write block 1524 len 32768 data: 2 2 2 2 2 2 2 2
2258 103/300 Flush
2259 104/300 Write block 1307 len 16384 data: 2 2 2 2
2260 105/300 Read block 1965 len 40960
2261 106/300 Write block 4276 len 32768 data: 2 2 2 2 2 2 2 2
2262 107/300 Read block 4967 len 16384
2263 108/300 Read block 3101 len 36864
2264 109/300 Read block 338 len 28672
2265 110/300 Read block 4440 len 32768
2266 111/300 Read block 1856 len 8192
2267 112/300 Read block 2764 len 20480
2268 113/300 Read block 525 len 20480
2269 114/300 Write block 1560 len 16384 data: 2 2 2 2
2270 115/300 Read block 4018 len 32768
2271 116/300 Flush
2272 117/300 Read block 2339 len 16384
2273 118/300 Read block 1343 len 12288
2274 119/300 Read block 1543 len 4096
2275 120/300 Read block 1988 len 28672
2276 121/300 Read block 2678 len 16384
2277 122/300 Write block 4733 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2278 123/300 Read block 1308 len 32768
2279 124/300 Read block 439 len 32768
2280 125/300 Write block 658 len 20480 data: 2 2 2 2 2
2281 126/300 Flush
2282 127/300 Flush
2283 128/300 Write block 1260 len 12288 data: 2 2 2
2284 129/300 Read block 175 len 20480
2285 130/300 Write block 4882 len 32768 data: 2 2 2 2 2 2 2 2
2286 131/300 Write block 3131 len 40960 data: 2 2 2 3 3 3 2 2 2 2
2287 132/300 Write block 2792 len 24576 data: 2 2 2 2 2 2
2288 133/300 Flush
2289 134/300 Write block 915 len 24576 data: 2 2 2 2 2 2
2290 135/300 Write block 1323 len 28672 data: 2 2 2 2 2 2 2
2291 136/300 Write block 168 len 32768 data: 2 2 2 2 2 2 2 2
2292 137/300 Write block 3183 len 8192 data: 2 3
2293 138/300 Write block 4033 len 28672 data: 2 2 3 2 2 2 2
2294 139/300 Write block 2770 len 36864 data: 2 2 2 2 2 2 2 2 2
2295 140/300 Read block 4945 len 24576
2296 141/300 Read block 1521 len 12288
2297 142/300 Read block 3205 len 12288
2298 143/300 Read block 4272 len 8192
2299 144/300 Write block 892 len 24576 data: 4 4 4 4 3 3
2300 145/300 Read block 1754 len 20480
2301 146/300 Read block 806 len 32768
2302 147/300 Write block 3329 len 36864 data: 2 2 2 2 2 2 2 2 2
2303 148/300 Write block 4220 len 36864 data: 3 3 3 2 2 2 2 2 2
2304 149/300 Write block 3367 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2305 150/300 Write block 4443 len 40960 data: 2 2 2 2 2 2 2 2 3 3
2306 151/300 Write block 1454 len 4096 data: 2
2307 152/300 Read block 714 len 40960
2308 153/300 Write block 3941 len 16384 data: 2 2 2 2
2309 154/300 Write block 3912 len 20480 data: 2 2 2 2 3
2310 155/300 Read block 873 len 32768
2311 156/300 Write block 993 len 20480 data: 2 3 3 4 4
2312 157/300 Write block 744 len 12288 data: 2 2 2
2313 158/300 Read block 2553 len 28672
2314 159/300 Flush
2315 160/300 Write block 2329 len 36864 data: 2 2 2 2 2 2 2 2 2
2316 161/300 Flush
2317 162/300 Write block 3099 len 20480 data: 3 3 2 2 2
2318 163/300 Read block 2298 len 28672
2319 164/300 Read block 4831 len 20480
2320 165/300 Write block 948 len 4096 data: 2
2321 166/300 Read block 1178 len 8192
2322 167/300 Write block 2599 len 8192 data: 2 2
2323 168/300 Write block 370 len 8192 data: 2 2
2324 169/300 Read block 1506 len 12288
2325 170/300 Read block 2202 len 40960
2326 171/300 Read block 3483 len 32768
2327 172/300 Write block 775 len 28672 data: 3 3 3 3 2 2 2
2328 173/300 Read block 19 len 20480
2329 174/300 Write block 2092 len 24576 data: 2 2 2 2 2 2
2330 175/300 Write block 3380 len 4096 data: 2
2331 176/300 Read block 3610 len 4096
2332 177/300 Read block 2436 len 32768
2333 178/300 Read block 3777 len 4096
2334 179/300 Write block 1783 len 24576 data: 3 3 3 3 2 2
2335 180/300 Write block 4510 len 24576 data: 3 3 2 2 2 2
2336 181/300 Read block 1079 len 16384
2337 182/300 Read block 3520 len 36864
2338 183/300 Write block 880 len 24576 data: 2 2 2 2 2 2
2339 184/300 Read block 4805 len 8192
2340 185/300 Write block 120 len 32768 data: 3 3 3 3 3 3 3 3
2341 186/300 Flush
2342 187/300 Write block 455 len 36864 data: 2 2 2 2 2 2 2 2 2
2343 188/300 Read block 895 len 16384
2344 189/300 Flush
2345 190/300 Flush
2346 191/300 Flush
2347 192/300 Read block 2405 len 32768
2348 193/300 Write block 4788 len 36864 data: 2 2 2 2 2 2 2 2 2
2349 194/300 Flush
2350 195/300 Write block 3584 len 28672 data: 3 3 3 3 3 2 2
2351 196/300 Write block 3308 len 16384 data: 2 2 2 2
2352 197/300 Write block 206 len 24576 data: 2 2 2 2 2 2
2353 198/300 Read block 3337 len 24576
2354 199/300 Write block 3692 len 40960 data: 2 2 2 2 2 2 2 3 3 3
2355 200/300 Read block 3375 len 8192
2356 201/300 Read block 3604 len 40960
2357 202/300 Flush
2358 203/300 Read block 2385 len 16384
2359 204/300 Flush
2360 205/300 Flush
2361 206/300 Write block 508 len 20480 data: 3 2 3 3 3
2362 207/300 Write block 1080 len 40960 data: 4 3 3 2 2 3 3 3 3 3
2363 208/300 Write block 2717 len 36864 data: 3 2 2 2 2 2 2 2 2
2364 209/300 Read block 850 len 32768
2365 210/300 Write block 1234 len 20480 data: 2 2 2 3 3
2366 211/300 Write block 3401 len 28672 data: 2 2 2 4 4 4 3
2367 212/300 Write block 2092 len 20480 data: 3 3 3 3 3
2368 213/300 Read block 3179 len 28672
2369 214/300 Read block 338 len 32768
2370 215/300 Read block 1205 len 16384
2371 216/300 Write block 4752 len 20480 data: 2 2 2 2 2
2372 217/300 Read block 3935 len 16384
2373 218/300 Read block 2093 len 36864
2374 219/300 Flush
2375 220/300 Write block 4225 len 28672 data: 3 3 3 3 2 2 2
2376 221/300 Read block 2219 len 40960
2377 222/300 Read block 3339 len 16384
2378 223/300 Write block 984 len 4096 data: 2
2379 224/300 Read block 1126 len 4096
2380 225/300 Write block 3798 len 8192 data: 2 2
2381 226/300 Read block 2087 len 24576
2382 227/300 Write block 1691 len 12288 data: 3 3 3
2383 228/300 Read block 2460 len 40960
2384 229/300 Flush
2385 230/300 Write block 2230 len 40960 data: 2 2 2 2 2 3 3 3 3 3
2386 231/300 Write block 4898 len 40960 data: 2 2 2 2 2 2 2 3 3 2
2387 232/300 Write block 716 len 4096 data: 3
2388 233/300 Write block 3441 len 40960 data: 2 2 2 3 3 3 3 3 3 3
2389 234/300 Write block 4194 len 28672 data: 2 2 2 2 2 2 2
2390 235/300 Write block 2810 len 32768 data: 3 4 4 4 4 5 4 3
2391 236/300 Write block 795 len 8192 data: 3 3
2392 237/300 Flush
2393 238/300 Read block 934 len 4096
2394 239/300 Read block 3134 len 20480
2395 240/300 Flush
2396 241/300 Write block 4636 len 4096 data: 3
2397 242/300 Flush
2398 243/300 Flush
2399 244/300 Read block 4069 len 40960
2400 245/300 Read block 4488 len 32768
2401 246/300 Read block 2854 len 32768
2402 247/300 Write block 3241 len 8192 data: 2 3
2403 248/300 Write block 482 len 36864 data: 2 2 2 2 2 2 2 2 3
2404 249/300 Read block 4916 len 28672
2405 250/300 Write block 3919 len 8192 data: 3 3
2406 251/300 Read block 2060 len 40960
2407 252/300 Write block 3146 len 12288 data: 2 2 2
2408 253/300 Flush
2409 254/300 Read block 1996 len 36864
2410 255/300 Write block 2611 len 28672 data: 2 2 2 2 2 2 2
2411 256/300 Flush
2412 257/300 Read block 1320 len 32768
2413 258/300 Write block 3002 len 12288 data: 2 2 2
2414 259/300 Read block 668 len 12288
2415 260/300 Read block 1210 len 4096
2416 261/300 Read block 3531 len 16384
2417 262/300 Write block 2877 len 12288 data: 3 3 3
2418 263/300 Read block 3271 len 4096
2419 264/300 Write block 2973 len 36864 data: 2 2 2 2 2 2 2 2 3
2420 265/300 Write block 1100 len 32768 data: 2 2 2 2 2 2 2 2
2421 266/300 Write block 4684 len 20480 data: 2 2 2 2 2
2422 267/300 Write block 4672 len 40960 data: 2 3 3 3 3 3 2 2 2 2
2423 268/300 Read block 3037 len 28672
2424 269/300 Write block 1088 len 20480 data: 4 4 3 2 2
2425 270/300 Write block 601 len 16384 data: 3 2 2 2
2426 271/300 Read block 422 len 24576
2427 272/300 Read block 292 len 28672
2428 273/300 Write block 4775 len 20480 data: 2 2 2 2 2
2429 274/300 Flush
2430 275/300 Flush
2431 276/300 Write block 3288 len 20480 data: 2 2 2 2 2
2432 277/300 Read block 103 len 24576
2433 278/300 Write block 2562 len 28672 data: 2 2 2 2 2 2 2
2434 279/300 Read block 1721 len 36864
2435 280/300 Read block 3295 len 12288
2436 281/300 Read block 2346 len 12288
2437 282/300 Write block 2181 len 40960 data: 2 2 2 2 2 3 2 2 2 2
2438 283/300 Write block 1698 len 12288 data: 2 2 2
2439 284/300 Write block 775 len 28672 data: 4 4 4 4 3 3 3
2440 285/300 Write block 3486 len 32768 data: 2 2 2 2 2 2 2 2
2441 286/300 Read block 3938 len 12288
2442 287/300 Write block 4163 len 20480 data: 2 2 2 2 2
2443 288/300 Read block 2909 len 36864
2444 289/300 Read block 4884 len 36864
2445 290/300 Flush
2446 291/300 Write block 63 len 12288 data: 2 2 2
2447 292/300 Read block 3236 len 8192
2448 293/300 Read block 2149 len 36864
2449 294/300 Write block 4101 len 16384 data: 2 3 3 3
2450 295/300 Write block 4629 len 16384 data: 3 3 3 3
2451 296/300 Write block 202 len 20480 data: 2 2 2 2 3
2452 297/300 Read block 3723 len 12288
2453 298/300 Flush
2454 299/300 Read block 2262 len 8192
2455 300/300 Read block 1000 len 36864
2456 301/300 Read block 4170 len 8192
2457 Main task has work to do, go find it
2458 got dsc Start(0)
2459 start 0
2460 Nov 01 16:51:01.121 INFO request completed, latency_us: 216, response_code: 204, uri: /start/cid/0, method: POST, req_id: 3bad0dbc-000d-4a2c-b62e-006c3c756561, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
2461 Replay: started 0, returned:Ok(())
2462 ----------------------------------------------------------------
2463 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2464 States: Offline Active Active
2465 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2466 1605 Acked Write 4 Sent Done Done false
2467 1606 Acked Write 2 Sent Done Done false
2468 1607 Acked Read 8 Sent Done Done false
2469 1608 Acked Write 7 Sent Done Done false
2470 1609 Acked Write 2 Sent Done Done false
2471 1610 Acked Write 7 Sent Done Done false
2472 1611 Acked Write 9 Sent Done Done false
2473 1612 Acked Read 10 Sent Done Done false
2474 1613 Acked Read 5 Sent Done Done false
2475 1614 Acked Read 8 Sent Done Done false
2476 1615 Acked Read 7 Sent Done Done false
2477 1616 Acked Read 5 Sent Done Done false
2478 1617 Acked Write 9 Sent Done Done false
2479 1618 Acked Flush 0 Sent Done Done false
2480 1619 Acked Read 4 Sent Done Done false
2481 1620 Acked Read 4 Sent Done Done false
2482 1621 Acked Write 1 Sent Done Done false
2483 1622 Acked Read 5 Sent Done Done false
2484 1623 Acked Read 7 Sent Done Done false
2485 1624 Acked Write 4 Sent Done Done false
2486 1625 Acked Read 3 Sent Done Done false
2487 1626 Acked Write 7 Sent Done Done false
2488 1627 Acked Write 4 Sent Done Done false
2489 1628 Acked Read 3 Sent Done Done false
2490 1629 Acked Read 6 Sent Done Done false
2491 1630 Acked Read 10 Sent Done Done false
2492 1631 Acked Read 5 Sent Done Done false
2493 1632 Acked Write 9 Sent Done Done false
2494 1633 Acked Write 4 Sent Done Done false
2495 1634 Acked Write 10 Sent Done Done false
2496 1635 Acked Write 3 Sent Done Done false
2497 1636 Acked Write 10 Sent Done Done false
2498 1637 Acked Write 7 Sent Done Done false
2499 1638 Acked Write 2 Sent Done Done false
2500 1639 Acked Read 5 Sent Done Done false
2501 1640 Acked Read 9 Sent Done Done false
2502 1641 Acked Read 7 Sent Done Done false
2503 1642 Acked Read 6 Sent Done Done false
2504 1643 Acked Write 8 Sent Done Done false
2505 1644 Acked Read 7 Sent Done Done false
2506 1645 Acked Flush 0 Sent Done Done false
2507 1646 Acked Read 4 Sent Done Done false
2508 1647 Acked Read 1 Sent Done Done false
2509 1648 Acked Read 6 Sent Done Done false
2510 1649 Acked Read 1 Sent Done Done false
2511 1650 Acked Read 6 Sent Done Done false
2512 1651 Acked Read 6 Sent Done Done false
2513 1652 Acked Write 5 Sent Done Done false
2514 1653 Acked Write 6 Sent Done Done false
2515 1654 Acked Write 9 Sent Done Done false
2516 1655 Acked Write 8 Sent Done Done false
2517 1656 Acked Write 2 Sent Done Done false
2518 1657 Acked Write 6 Sent Done Done false
2519 1658 Acked Read 7 Sent Done Done false
2520 1659 Acked Write 2 Sent Done Done false
2521 1660 Acked Write 6 Sent Done Done false
2522 1661 Acked Read 10 Sent Done Done false
2523 1662 Acked Write 5 Sent Done Done false
2524 1663 Acked Write 8 Sent Done Done false
2525 1664 Acked Write 4 Sent Done Done false
2526 1665 Acked Write 4 Sent Done Done false
2527 1666 Acked Read 1 Sent Done Done false
2528 1667 Acked Flush 0 Sent Done Done false
2529 1668 Acked Read 1 Sent Done Done false
2530 1669 Acked Write 8 Sent Done Done false
2531 1670 Acked Read 5 Sent Done Done false
2532 1671 Acked Write 9 Sent Done Done false
2533 1672 Acked Read 7 Sent Done Done false
2534 1673 Acked Read 10 Sent Done Done false
2535 1674 Acked Read 10 Sent Done Done false
2536 1675 Acked Write 3 Sent Done Done false
2537 1676 Acked Read 10 Sent Done Done false
2538 1677 Acked Read 1 Sent Done Done false
2539 1678 Acked Read 3 Sent Done Done false
2540 1679 Acked Write 6 Sent Done Done false
2541 1680 Acked Flush 0 Sent Done Done false
2542 1681 Acked Write 3 Sent Done Done false
2543 1682 Acked Write 7 Sent Done Done false
2544 1683 Acked Read 7 Sent Done Done false
2545 1684 Acked Write 8 Sent Done Done false
2546 1685 Acked Write 1 Sent Done Done false
2547 1686 Acked Write 4 Sent Done Done false
2548 1687 Acked Read 2 Sent Done Done false
2549 1688 Acked Flush 0 Sent Done Done false
2550 1689 Acked Read 10 Sent Done Done false
2551 1690 Acked Write 8 Sent Done Done false
2552 1691 Acked Read 1 Sent Done Done false
2553 1692 Acked Write 10 Sent Done Done false
2554 1693 Acked Write 3 Sent Done Done false
2555 1694 Acked Write 2 Sent Done Done false
2556 1695 Acked Read 8 Sent Done Done false
2557 1696 Acked Read 7 Sent Done Done false
2558 1697 Acked Read 1 Sent Done Done false
2559 1698 Acked Read 6 Sent Done Done false
2560 1699 Acked Read 10 Sent Done Done false
2561 1700 Acked Read 2 Sent Done Done false
2562 1701 Acked Read 3 Sent Done Done false
2563 1702 Acked Flush 0 Sent Done Done false
2564 1703 Acked Read 1 Sent Done Done false
2565 1704 Acked Flush 0 Sent Done Done false
2566 1705 Acked Read 10 Sent Done Done false
2567 1706 Acked Write 8 Sent Done Done false
2568 1707 Acked Flush 0 Sent Done Done false
2569 1708 Acked Write 4 Sent Done Done false
2570 1709 Acked Read 10 Sent Done Done false
2571 1710 Acked Write 8 Sent Done Done false
2572 1711 Acked Read 4 Sent Done Done false
2573 1712 Acked Read 9 Sent Done Done false
2574 1713 Acked Read 7 Sent Done Done false
2575 1714 Acked Read 8 Sent Done Done false
2576 1715 Acked Read 2 Sent Done Done false
2577 1716 Acked Read 5 Sent Done Done false
2578 1717 Acked Read 5 Sent Done Done false
2579 1718 Acked Write 4 Sent Done Done false
2580 1719 Acked Read 8 Sent Done Done false
2581 1720 Acked Flush 0 Sent Done Done false
2582 1721 Acked Read 4 Sent Done Done false
2583 1722 Acked Read 3 Sent Done Done false
2584 1723 Acked Read 1 Sent Done Done false
2585 1724 Acked Read 7 Sent Done Done false
2586 1725 Acked Read 4 Sent Done Done false
2587 1726 Acked Write 10 Sent Done Done false
2588 1727 Acked Read 8 Sent Done Done false
2589 1728 Acked Read 8 Sent Done Done false
2590 1729 Acked Write 5 Sent Done Done false
2591 1730 Acked Flush 0 Sent Done Done false
2592 1731 Acked Flush 0 Sent Done Done false
2593 1732 Acked Write 3 Sent Done Done false
2594 1733 Acked Read 5 Sent Done Done false
2595 1734 Acked Write 8 Sent Done Done false
2596 1735 Acked Write 10 Sent Done Done false
2597 1736 Acked Write 6 Sent Done Done false
2598 1737 Acked Flush 0 Sent Done Done false
2599 1738 Acked Write 6 Sent Done Done false
2600 1739 Acked Write 7 Sent Done Done false
2601 1740 Acked Write 8 Sent Done Done false
2602 1741 Acked Write 2 Sent Done Done false
2603 1742 Acked Write 7 Sent Done Done false
2604 1743 Acked Write 9 Sent Done Done false
2605 1744 Acked Read 6 Sent Done Done false
2606 1745 Acked Read 3 Sent Done Done false
2607 1746 Acked Read 3 Sent Done Done false
2608 1747 Acked Read 2 Sent Done Done false
2609 1748 Acked Write 6 Sent Done Done false
2610 1749 Acked Read 5 Sent Done Done false
2611 1750 Acked Read 8 Sent Done Done false
2612 1751 Acked Write 9 Sent Done Done false
2613 1752 Acked Write 9 Sent Done Done false
2614 1753 Acked Write 10 Sent Done Done false
2615 1754 Acked Write 10 Sent Done Done false
2616 1755 Acked Write 1 Sent Done Done false
2617 1756 Acked Read 10 Sent Done Done false
2618 1757 Acked Write 4 Sent Done Done false
2619 1758 Acked Write 5 Sent Done Done false
2620 1759 Acked Read 8 Sent Done Done false
2621 1760 Acked Write 5 Sent Done Done false
2622 1761 Acked Write 3 Sent Done Done false
2623 1762 Acked Read 7 Sent Done Done false
2624 1763 Acked Flush 0 Sent Done Done false
2625 1764 Acked Write 9 Sent Done Done false
2626 1765 Acked Flush 0 Sent Done Done false
2627 1766 Acked Write 5 Sent Done Done false
2628 1767 Acked Read 7 Sent Done Done false
2629 1768 Acked Read 5 Sent Done Done false
2630 1769 Acked Write 1 Sent Done Done false
2631 1770 Acked Read 2 Sent Done Done false
2632 1771 Acked Write 2 Sent Done Done false
2633 1772 Acked Write 2 Sent Done Done false
2634 1773 Acked Read 3 Sent Done Done false
2635 1774 Acked Read 10 Sent Done Done false
2636 1775 Acked Read 8 Sent Done Done false
2637 1776 Acked Write 7 Sent Done Done false
2638 1777 Acked Read 5 Sent Done Done false
2639 1778 Acked Write 6 Sent Done Done false
2640 1779 Acked Write 1 Sent Done Done false
2641 1780 Acked Read 1 Sent Done Done false
2642 1781 Acked Read 8 Sent Done Done false
2643 1782 Acked Read 1 Sent Done Done false
2644 1783 Acked Write 6 Sent Done Done false
2645 1784 Acked Write 6 Sent Done Done false
2646 1785 Acked Read 4 Sent Done Done false
2647 1786 Acked Read 9 Sent Done Done false
2648 1787 Acked Write 6 Sent Done Done false
2649 1788 Acked Read 2 Sent Done Done false
2650 1789 Acked Write 8 Sent Done Done false
2651 1790 Acked Flush 0 Sent Done Done false
2652 1791 Acked Write 9 Sent Done Done false
2653 1792 Acked Read 4 Sent Done Done false
2654 1793 Acked Flush 0 Sent Done Done false
2655 1794 Acked Flush 0 Sent Done Done false
2656 1795 Acked Flush 0 Sent Done Done false
2657 1796 Acked Read 8 Sent Done Done false
2658 1797 Acked Write 9 Sent Done Done false
2659 1798 Acked Flush 0 Sent Done Done false
2660 1799 Acked Write 7 Sent Done Done false
2661 1800 Acked Write 4 Sent Done Done false
2662 1801 Acked Write 6 Sent Done Done false
2663 1802 Acked Read 6 Sent Done Done false
2664 1803 Acked Write 10 Sent Done Done false
2665 1804 Acked Read 2 Sent Done Done false
2666 1805 Acked Read 10 Sent Done Done false
2667 1806 Acked Flush 0 Sent Done Done false
2668 1807 Acked Read 4 Sent Done Done false
2669 1808 Acked Flush 0 Sent Done Done false
2670 1809 Acked Flush 0 Sent Done Done false
2671 1810 Acked Write 5 Sent Done Done false
2672 1811 Acked Write 10 Sent Done Done false
2673 1812 Acked Write 9 Sent Done Done false
2674 1813 Acked Read 8 Sent Done Done false
2675 1814 Acked Write 5 Sent Done Done false
2676 1815 Acked Write 7 Sent Done Done false
2677 1816 Acked Write 5 Sent Done Done false
2678 1817 Acked Read 7 Sent Done Done false
2679 1818 Acked Read 8 Sent Done Done false
2680 1819 Acked Read 4 Sent Done Done false
2681 1820 Acked Write 5 Sent Done Done false
2682 1821 Acked Read 4 Sent Done Done false
2683 1822 Acked Read 9 Sent Done Done false
2684 1823 Acked Flush 0 Sent Done Done false
2685 1824 Acked Write 7 Sent Done Done false
2686 1825 Acked Read 10 Sent Done Done false
2687 1826 Acked Read 4 Sent Done Done false
2688 1827 Acked Write 1 Sent Done Done false
2689 1828 Acked Read 1 Sent Done Done false
2690 1829 Acked Write 2 Sent Done Done false
2691 1830 Acked Read 6 Sent Done Done false
2692 1831 Acked Write 3 Sent Done Done false
2693 1832 Acked Read 10 Sent Done Done false
2694 1833 Acked Flush 0 Sent Done Done false
2695 1834 Acked Write 10 Sent Done Done false
2696 1835 Acked Write 10 Sent Done Done false
2697 1836 Acked Write 1 Sent Done Done false
2698 1837 Acked Write 10 Sent Done Done false
2699 1838 Acked Write 7 Sent Done Done false
2700 1839 Acked Write 8 Sent Done Done false
2701 1840 Acked Write 2 Sent Done Done false
2702 1841 Acked Flush 0 Sent Done Done false
2703 1842 Acked Read 1 Sent Done Done false
2704 1843 Acked Read 5 Sent Done Done false
2705 1844 Acked Flush 0 Sent Done Done false
2706 1845 Acked Write 1 Sent Done Done false
2707 1846 Acked Flush 0 Sent Done Done false
2708 1847 Acked Flush 0 Sent Done Done false
2709 1848 Acked Read 10 Sent Done Done false
2710 1849 Acked Read 8 Sent Done Done false
2711 1850 Acked Read 8 Sent Done Done false
2712 1851 Acked Write 2 Sent Done Done false
2713 1852 Acked Write 9 Sent Done Done false
2714 1853 Acked Read 7 Sent Done Done false
2715 1854 Acked Write 2 Sent Done Done false
2716 1855 Acked Read 10 Sent Done Done false
2717 1856 Acked Write 3 Sent Done Done false
2718 1857 Acked Flush 0 Sent Done Done false
2719 1858 Acked Read 9 Sent Done Done false
2720 1859 Acked Write 7 Sent Done Done false
2721 1860 Acked Flush 0 Sent Done Done false
2722 1861 Acked Read 8 Sent Done Done false
2723 1862 Acked Write 3 Sent Done Done false
2724 1863 Acked Read 3 Sent Done Done false
2725 1864 Acked Read 1 Sent Done Done false
2726 1865 Acked Read 4 Sent Done Done false
2727 1866 Acked Write 3 Sent Done Done false
2728 1867 Acked Read 1 Sent Done Done false
2729 1868 Acked Write 9 Sent Done Done false
2730 1869 Acked Write 8 Sent Done Done false
2731 1870 Acked Write 5 Sent Done Done false
2732 1871 Acked Write 10 Sent Done Done false
2733 1872 Acked Read 7 Sent Done Done false
2734 1873 Acked Write 5 Sent Done Done false
2735 1874 Acked Write 4 Sent Done Done false
2736 1875 Acked Read 6 Sent Done Done false
2737 1876 Acked Read 7 Sent Done Done false
2738 1877 Acked Write 5 Sent Done Done false
2739 1878 Acked Flush 0 Sent Done Done false
2740 1879 Acked Flush 0 Sent Done Done false
2741 1880 Acked Write 5 Sent Done Done false
2742 1881 Acked Read 6 Sent Done Done false
2743 1882 Acked Write 7 Sent Done Done false
2744 1883 Acked Read 9 Sent Done Done false
2745 1884 Acked Read 3 Sent Done Done false
2746 1885 Acked Read 3 Sent Done Done false
2747 1886 Acked Write 10 Sent Done Done false
2748 1887 Acked Write 3 Sent Done Done false
2749 1888 Acked Write 7 Sent Done Done false
2750 1889 Acked Write 8 Sent Done Done false
2751 1890 Acked Read 3 Sent Done Done false
2752 1891 Acked Write 5 Sent Done Done false
2753 1892 Acked Read 9 Sent Done Done false
2754 1893 Acked Read 9 Sent Done Done false
2755 1894 Acked Flush 0 Sent Done Done false
2756 1895 Acked Write 3 Sent Done Done false
2757 1896 Acked Read 2 Sent Done Done false
2758 1897 Acked Read 9 Sent Done Done false
2759 1898 Acked Write 4 Sent Done Done false
2760 1899 Acked Write 4 Sent Done Done false
2761 1900 Acked Write 5 Sent Done Done false
2762 1901 Acked Read 3 Sent Done Done false
2763 1902 Acked Flush 0 Sent Done Done false
2764 1903 Acked Read 2 Sent Done Done false
2765 1904 Acked Read 9 Sent Done Done false
2766 1905 Acked Read 2 Sent Done Done false
2767 STATES DS:0 DS:1 DS:2 TOTAL
2768 Sent 301 0 0 301
2769 Done 0 301 301 602
2770 Skipped 0 0 0 0
2771 Error 0 0 0 0
2772 Last Flush: 1604 1902 1902
2773 Downstairs last five completed: 1604 1603 1602 1601 1600
2774 Upstairs last five completed: 1905 1904 1903 1902 1901
2775 CLIENT: Up:0 ds:301 act:2
2776 [8810] got start action
2777 [8810] I am going to restart
2778 Starting downstairs at port 8810
2779 Make output file at "/tmp/dsc/downstairs-8810.txt"
2780 [8810][0] reports Starting
2781 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1201)
2782 [8810][0] reports Running
2783 ----------------------------------------------------------------
2784 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
2785 States: Offline Active Active
2786 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2787 1605 Acked Write 4 Sent Done Done false
2788 1606 Acked Write 2 Sent Done Done false
2789 1607 Acked Read 8 Sent Done Done false
2790 1608 Acked Write 7 Sent Done Done false
2791 1609 Acked Write 2 Sent Done Done false
2792 1610 Acked Write 7 Sent Done Done false
2793 1611 Acked Write 9 Sent Done Done false
2794 1612 Acked Read 10 Sent Done Done false
2795 1613 Acked Read 5 Sent Done Done false
2796 1614 Acked Read 8 Sent Done Done false
2797 1615 Acked Read 7 Sent Done Done false
2798 1616 Acked Read 5 Sent Done Done false
2799 1617 Acked Write 9 Sent Done Done false
2800 1618 Acked Flush 0 Sent Done Done false
2801 1619 Acked Read 4 Sent Done Done false
2802 1620 Acked Read 4 Sent Done Done false
2803 1621 Acked Write 1 Sent Done Done false
2804 1622 Acked Read 5 Sent Done Done false
2805 1623 Acked Read 7 Sent Done Done false
2806 1624 Acked Write 4 Sent Done Done false
2807 1625 Acked Read 3 Sent Done Done false
2808 1626 Acked Write 7 Sent Done Done false
2809 1627 Acked Write 4 Sent Done Done false
2810 1628 Acked Read 3 Sent Done Done false
2811 1629 Acked Read 6 Sent Done Done false
2812 1630 Acked Read 10 Sent Done Done false
2813 1631 Acked Read 5 Sent Done Done false
2814 1632 Acked Write 9 Sent Done Done false
2815 1633 Acked Write 4 Sent Done Done false
2816 1634 Acked Write 10 Sent Done Done false
2817 1635 Acked Write 3 Sent Done Done false
2818 1636 Acked Write 10 Sent Done Done false
2819 1637 Acked Write 7 Sent Done Done false
2820 1638 Acked Write 2 Sent Done Done false
2821 1639 Acked Read 5 Sent Done Done false
2822 1640 Acked Read 9 Sent Done Done false
2823 1641 Acked Read 7 Sent Done Done false
2824 1642 Acked Read 6 Sent Done Done false
2825 1643 Acked Write 8 Sent Done Done false
2826 1644 Acked Read 7 Sent Done Done false
2827 1645 Acked Flush 0 Sent Done Done false
2828 1646 Acked Read 4 Sent Done Done false
2829 1647 Acked Read 1 Sent Done Done false
2830 1648 Acked Read 6 Sent Done Done false
2831 1649 Acked Read 1 Sent Done Done false
2832 1650 Acked Read 6 Sent Done Done false
2833 1651 Acked Read 6 Sent Done Done false
2834 1652 Acked Write 5 Sent Done Done false
2835 1653 Acked Write 6 Sent Done Done false
2836 1654 Acked Write 9 Sent Done Done false
2837 1655 Acked Write 8 Sent Done Done false
2838 1656 Acked Write 2 Sent Done Done false
2839 1657 Acked Write 6 Sent Done Done false
2840 1658 Acked Read 7 Sent Done Done false
2841 1659 Acked Write 2 Sent Done Done false
2842 1660 Acked Write 6 Sent Done Done false
2843 1661 Acked Read 10 Sent Done Done false
2844 1662 Acked Write 5 Sent Done Done false
2845 1663 Acked Write 8 Sent Done Done false
2846 1664 Acked Write 4 Sent Done Done false
2847 1665 Acked Write 4 Sent Done Done false
2848 1666 Acked Read 1 Sent Done Done false
2849 1667 Acked Flush 0 Sent Done Done false
2850 1668 Acked Read 1 Sent Done Done false
2851 1669 Acked Write 8 Sent Done Done false
2852 1670 Acked Read 5 Sent Done Done false
2853 1671 Acked Write 9 Sent Done Done false
2854 1672 Acked Read 7 Sent Done Done false
2855 1673 Acked Read 10 Sent Done Done false
2856 1674 Acked Read 10 Sent Done Done false
2857 1675 Acked Write 3 Sent Done Done false
2858 1676 Acked Read 10 Sent Done Done false
2859 1677 Acked Read 1 Sent Done Done false
2860 1678 Acked Read 3 Sent Done Done false
2861 1679 Acked Write 6 Sent Done Done false
2862 1680 Acked Flush 0 Sent Done Done false
2863 1681 Acked Write 3 Sent Done Done false
2864 1682 Acked Write 7 Sent Done Done false
2865 1683 Acked Read 7 Sent Done Done false
2866 1684 Acked Write 8 Sent Done Done false
2867 1685 Acked Write 1 Sent Done Done false
2868 1686 Acked Write 4 Sent Done Done false
2869 1687 Acked Read 2 Sent Done Done false
2870 1688 Acked Flush 0 Sent Done Done false
2871 1689 Acked Read 10 Sent Done Done false
2872 1690 Acked Write 8 Sent Done Done false
2873 1691 Acked Read 1 Sent Done Done false
2874 1692 Acked Write 10 Sent Done Done false
2875 1693 Acked Write 3 Sent Done Done false
2876 1694 Acked Write 2 Sent Done Done false
2877 1695 Acked Read 8 Sent Done Done false
2878 1696 Acked Read 7 Sent Done Done false
2879 1697 Acked Read 1 Sent Done Done false
2880 1698 Acked Read 6 Sent Done Done false
2881 1699 Acked Read 10 Sent Done Done false
2882 1700 Acked Read 2 Sent Done Done false
2883 1701 Acked Read 3 Sent Done Done false
2884 1702 Acked Flush 0 Sent Done Done false
2885 1703 Acked Read 1 Sent Done Done false
2886 1704 Acked Flush 0 Sent Done Done false
2887 1705 Acked Read 10 Sent Done Done false
2888 1706 Acked Write 8 Sent Done Done false
2889 1707 Acked Flush 0 Sent Done Done false
2890 1708 Acked Write 4 Sent Done Done false
2891 1709 Acked Read 10 Sent Done Done false
2892 1710 Acked Write 8 Sent Done Done false
2893 1711 Acked Read 4 Sent Done Done false
2894 1712 Acked Read 9 Sent Done Done false
2895 1713 Acked Read 7 Sent Done Done false
2896 1714 Acked Read 8 Sent Done Done false
2897 1715 Acked Read 2 Sent Done Done false
2898 1716 Acked Read 5 Sent Done Done false
2899 1717 Acked Read 5 Sent Done Done false
2900 1718 Acked Write 4 Sent Done Done false
2901 1719 Acked Read 8 Sent Done Done false
2902 1720 Acked Flush 0 Sent Done Done false
2903 1721 Acked Read 4 Sent Done Done false
2904 1722 Acked Read 3 Sent Done Done false
2905 1723 Acked Read 1 Sent Done Done false
2906 1724 Acked Read 7 Sent Done Done false
2907 1725 Acked Read 4 Sent Done Done false
2908 1726 Acked Write 10 Sent Done Done false
2909 1727 Acked Read 8 Sent Done Done false
2910 1728 Acked Read 8 Sent Done Done false
2911 1729 Acked Write 5 Sent Done Done false
2912 1730 Acked Flush 0 Sent Done Done false
2913 1731 Acked Flush 0 Sent Done Done false
2914 1732 Acked Write 3 Sent Done Done false
2915 1733 Acked Read 5 Sent Done Done false
2916 1734 Acked Write 8 Sent Done Done false
2917 1735 Acked Write 10 Sent Done Done false
2918 1736 Acked Write 6 Sent Done Done false
2919 1737 Acked Flush 0 Sent Done Done false
2920 1738 Acked Write 6 Sent Done Done false
2921 1739 Acked Write 7 Sent Done Done false
2922 1740 Acked Write 8 Sent Done Done false
2923 1741 Acked Write 2 Sent Done Done false
2924 1742 Acked Write 7 Sent Done Done false
2925 1743 Acked Write 9 Sent Done Done false
2926 1744 Acked Read 6 Sent Done Done false
2927 1745 Acked Read 3 Sent Done Done false
2928 1746 Acked Read 3 Sent Done Done false
2929 1747 Acked Read 2 Sent Done Done false
2930 1748 Acked Write 6 Sent Done Done false
2931 1749 Acked Read 5 Sent Done Done false
2932 1750 Acked Read 8 Sent Done Done false
2933 1751 Acked Write 9 Sent Done Done false
2934 1752 Acked Write 9 Sent Done Done false
2935 1753 Acked Write 10 Sent Done Done false
2936 1754 Acked Write 10 Sent Done Done false
2937 1755 Acked Write 1 Sent Done Done false
2938 1756 Acked Read 10 Sent Done Done false
2939 1757 Acked Write 4 Sent Done Done false
2940 1758 Acked Write 5 Sent Done Done false
2941 1759 Acked Read 8 Sent Done Done false
2942 1760 Acked Write 5 Sent Done Done false
2943 1761 Acked Write 3 Sent Done Done false
2944 1762 Acked Read 7 Sent Done Done false
2945 1763 Acked Flush 0 Sent Done Done false
2946 1764 Acked Write 9 Sent Done Done false
2947 1765 Acked Flush 0 Sent Done Done false
2948 1766 Acked Write 5 Sent Done Done false
2949 1767 Acked Read 7 Sent Done Done false
2950 1768 Acked Read 5 Sent Done Done false
2951 1769 Acked Write 1 Sent Done Done false
2952 1770 Acked Read 2 Sent Done Done false
2953 1771 Acked Write 2 Sent Done Done false
2954 1772 Acked Write 2 Sent Done Done false
2955 1773 Acked Read 3 Sent Done Done false
2956 1774 Acked Read 10 Sent Done Done false
2957 1775 Acked Read 8 Sent Done Done false
2958 1776 Acked Write 7 Sent Done Done false
2959 1777 Acked Read 5 Sent Done Done false
2960 1778 Acked Write 6 Sent Done Done false
2961 1779 Acked Write 1 Sent Done Done false
2962 1780 Acked Read 1 Sent Done Done false
2963 1781 Acked Read 8 Sent Done Done false
2964 1782 Acked Read 1 Sent Done Done false
2965 1783 Acked Write 6 Sent Done Done false
2966 1784 Acked Write 6 Sent Done Done false
2967 1785 Acked Read 4 Sent Done Done false
2968 1786 Acked Read 9 Sent Done Done false
2969 1787 Acked Write 6 Sent Done Done false
2970 1788 Acked Read 2 Sent Done Done false
2971 1789 Acked Write 8 Sent Done Done false
2972 1790 Acked Flush 0 Sent Done Done false
2973 1791 Acked Write 9 Sent Done Done false
2974 1792 Acked Read 4 Sent Done Done false
2975 1793 Acked Flush 0 Sent Done Done false
2976 1794 Acked Flush 0 Sent Done Done false
2977 1795 Acked Flush 0 Sent Done Done false
2978 1796 Acked Read 8 Sent Done Done false
2979 1797 Acked Write 9 Sent Done Done false
2980 1798 Acked Flush 0 Sent Done Done false
2981 1799 Acked Write 7 Sent Done Done false
2982 1800 Acked Write 4 Sent Done Done false
2983 1801 Acked Write 6 Sent Done Done false
2984 1802 Acked Read 6 Sent Done Done false
2985 1803 Acked Write 10 Sent Done Done false
2986 1804 Acked Read 2 Sent Done Done false
2987 1805 Acked Read 10 Sent Done Done false
2988 1806 Acked Flush 0 Sent Done Done false
2989 1807 Acked Read 4 Sent Done Done false
2990 1808 Acked Flush 0 Sent Done Done false
2991 1809 Acked Flush 0 Sent Done Done false
2992 1810 Acked Write 5 Sent Done Done false
2993 1811 Acked Write 10 Sent Done Done false
2994 1812 Acked Write 9 Sent Done Done false
2995 1813 Acked Read 8 Sent Done Done false
2996 1814 Acked Write 5 Sent Done Done false
2997 1815 Acked Write 7 Sent Done Done false
2998 1816 Acked Write 5 Sent Done Done false
2999 1817 Acked Read 7 Sent Done Done false
3000 1818 Acked Read 8 Sent Done Done false
3001 1819 Acked Read 4 Sent Done Done false
3002 1820 Acked Write 5 Sent Done Done false
3003 1821 Acked Read 4 Sent Done Done false
3004 1822 Acked Read 9 Sent Done Done false
3005 1823 Acked Flush 0 Sent Done Done false
3006 1824 Acked Write 7 Sent Done Done false
3007 1825 Acked Read 10 Sent Done Done false
3008 1826 Acked Read 4 Sent Done Done false
3009 1827 Acked Write 1 Sent Done Done false
3010 1828 Acked Read 1 Sent Done Done false
3011 1829 Acked Write 2 Sent Done Done false
3012 1830 Acked Read 6 Sent Done Done false
3013 1831 Acked Write 3 Sent Done Done false
3014 1832 Acked Read 10 Sent Done Done false
3015 1833 Acked Flush 0 Sent Done Done false
3016 1834 Acked Write 10 Sent Done Done false
3017 1835 Acked Write 10 Sent Done Done false
3018 1836 Acked Write 1 Sent Done Done false
3019 1837 Acked Write 10 Sent Done Done false
3020 1838 Acked Write 7 Sent Done Done false
3021 1839 Acked Write 8 Sent Done Done false
3022 1840 Acked Write 2 Sent Done Done false
3023 1841 Acked Flush 0 Sent Done Done false
3024 1842 Acked Read 1 Sent Done Done false
3025 1843 Acked Read 5 Sent Done Done false
3026 1844 Acked Flush 0 Sent Done Done false
3027 1845 Acked Write 1 Sent Done Done false
3028 1846 Acked Flush 0 Sent Done Done false
3029 1847 Acked Flush 0 Sent Done Done false
3030 1848 Acked Read 10 Sent Done Done false
3031 1849 Acked Read 8 Sent Done Done false
3032 1850 Acked Read 8 Sent Done Done false
3033 1851 Acked Write 2 Sent Done Done false
3034 1852 Acked Write 9 Sent Done Done false
3035 1853 Acked Read 7 Sent Done Done false
3036 1854 Acked Write 2 Sent Done Done false
3037 1855 Acked Read 10 Sent Done Done false
3038 1856 Acked Write 3 Sent Done Done false
3039 1857 Acked Flush 0 Sent Done Done false
3040 1858 Acked Read 9 Sent Done Done false
3041 1859 Acked Write 7 Sent Done Done false
3042 1860 Acked Flush 0 Sent Done Done false
3043 1861 Acked Read 8 Sent Done Done false
3044 1862 Acked Write 3 Sent Done Done false
3045 1863 Acked Read 3 Sent Done Done false
3046 1864 Acked Read 1 Sent Done Done false
3047 1865 Acked Read 4 Sent Done Done false
3048 1866 Acked Write 3 Sent Done Done false
3049 1867 Acked Read 1 Sent Done Done false
3050 1868 Acked Write 9 Sent Done Done false
3051 1869 Acked Write 8 Sent Done Done false
3052 1870 Acked Write 5 Sent Done Done false
3053 1871 Acked Write 10 Sent Done Done false
3054 1872 Acked Read 7 Sent Done Done false
3055 1873 Acked Write 5 Sent Done Done false
3056 1874 Acked Write 4 Sent Done Done false
3057 1875 Acked Read 6 Sent Done Done false
3058 1876 Acked Read 7 Sent Done Done false
3059 1877 Acked Write 5 Sent Done Done false
3060 1878 Acked Flush 0 Sent Done Done false
3061 1879 Acked Flush 0 Sent Done Done false
3062 1880 Acked Write 5 Sent Done Done false
3063 1881 Acked Read 6 Sent Done Done false
3064 1882 Acked Write 7 Sent Done Done false
3065 1883 Acked Read 9 Sent Done Done false
3066 1884 Acked Read 3 Sent Done Done false
3067 1885 Acked Read 3 Sent Done Done false
3068 1886 Acked Write 10 Sent Done Done false
3069 1887 Acked Write 3 Sent Done Done false
3070 1888 Acked Write 7 Sent Done Done false
3071 1889 Acked Write 8 Sent Done Done false
3072 1890 Acked Read 3 Sent Done Done false
3073 1891 Acked Write 5 Sent Done Done false
3074 1892 Acked Read 9 Sent Done Done false
3075 1893 Acked Read 9 Sent Done Done false
3076 1894 Acked Flush 0 Sent Done Done false
3077 1895 Acked Write 3 Sent Done Done false
3078 1896 Acked Read 2 Sent Done Done false
3079 1897 Acked Read 9 Sent Done Done false
3080 1898 Acked Write 4 Sent Done Done false
3081 1899 Acked Write 4 Sent Done Done false
3082 1900 Acked Write 5 Sent Done Done false
3083 1901 Acked Read 3 Sent Done Done false
3084 1902 Acked Flush 0 Sent Done Done false
3085 1903 Acked Read 2 Sent Done Done false
3086 1904 Acked Read 9 Sent Done Done false
3087 1905 Acked Read 2 Sent Done Done false
3088 1906 Acked Flush 0 Sent Done Done false
3089 STATES DS:0 DS:1 DS:2 TOTAL
3090 Sent 302 0 0 302
3091 Done 0 302 302 604
3092 Skipped 0 0 0 0
3093 Error 0 0 0 0
3094 Last Flush: 1604 1906 1906
3095 Downstairs last five completed: 1604 1603 1602 1601 1600
3096 Upstairs last five completed: 1906 1905 1904 1903 1902
3097 CLIENT: Up:0 ds:302 act:2
30982024-11-01T16:51:06.841ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
30992024-11-01T16:51:06.841ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36008) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31002024-11-01T16:51:06.842ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31012024-11-01T16:51:06.842ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31022024-11-01T16:51:06.842ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31032024-11-01T16:51:06.842ZINFOcrucible: send last flush ID to this DS: 1604 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31042024-11-01T16:51:06.843ZINFOcrucible: Replied this last flush ID: 1604 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31052024-11-01T16:51:06.843ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31062024-11-01T16:51:06.843ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31072024-11-01T16:51:06.843ZINFOcrucible: [0] client re-new 302 jobs since flush 1604 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31082024-11-01T16:51:06.843ZINFOcrucible: [0] Replayed 302 jobs since flush: 1604 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
3109 ----------------------------------------------------------------
3110 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3111 Downstairs last five completed: 1906 1905 1904 1903 1902
3112 Upstairs last five completed: 1906 1905 1904 1903 1902
3113 CLIENT: Up:0 ds:0 act:3
3114 Replay: All jobs finished
3115 Main task has work to do, go find it
3116 got dsc Stop(1)
3117 stop 1
3118 Nov 01 16:51:09.127 INFO request completed, latency_us: 385[8820] Got stop action so:false kr:false
3119 , response_code: 204, uri: /stop/cid/1, method: POST, req_id: 2d385b62-f808-42b6-9bd9-5ab51b056388, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
3120 Nov 01 16:51:09.128 INFO request completed, latency_us: 176, response_code: 200, uri: /state/cid/1, method: GET, req_id: 403301be-0978-4929-9d48-ab611a8ac328, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
31212024-11-01T16:51:09.128ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31222024-11-01T16:51:09.128ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31232024-11-01T16:51:09.128ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31242024-11-01T16:51:09.128ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31252024-11-01T16:51:09.128ZINFOcrucible: client task is exiting = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
3126 [8820] Exited with: ExitStatus(unix_wait_status(9))
3127 [8820][1] reports Exit
3128 Nov 01 16:51:13.131 INFO request completed, latency_us: 231, response_code: 200, uri: /state/cid/1, method: GET, req_id: 37cebda7-9ddd-47d8-a60c-1c9232cf4a26, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
3129 001/300 Read block 4978 len 28672
3130 002/300 Write block 1937 len 20480 data: 3 3 2 2 2
3131 003/300 Write block 2496 len 8192 data: 3 3
3132 004/300 Write block 1076 len 16384 data: 3 4 5 4
3133 005/300 Read block 1963 len 32768
3134 006/300 Write block 834 len 36864 data: 2 2 2 2 2 2 2 2 2
3135 007/300 Read block 274 len 12288
3136 008/300 Read block 847 len 32768
3137 009/300 Read block 4245 len 4096
3138 010/300 Read block 2250 len 24576
3139 011/300 Write block 4913 len 36864 data: 2 2 2 3 3 3 2 2 3
3140 012/300 Read block 4402 len 28672
3141 013/300 Read block 3298 len 40960
3142 014/300 Write block 1153 len 20480 data: 4 4 4 3 3
3143 015/300 Read block 2022 len 32768
3144 016/300 Write block 1041 len 20480 data: 2 2 2 2 2
3145 017/300 Write block 2942 len 4096 data: 2
3146 018/300 Write block 1424 len 4096 data: 2
3147 019/300 Read block 2095 len 24576
3148 020/300 Write block 718 len 28672 data: 3 2 2 2 2 2 2
3149 021/300 Read block 4971 len 12288
3150 022/300 Write block 2292 len 40960 data: 3 4 4 4 4 4 2 2 2 2
3151 023/300 Write block 4688 len 12288 data: 3 2 2
3152 024/300 Read block 1105 len 28672
3153 025/300 Read block 2429 len 24576
3154 026/300 Write block 1500 len 12288 data: 3 3 3
3155 027/300 Read block 659 len 16384
3156 028/300 Write block 3571 len 8192 data: 2 2
3157 029/300 Read block 2587 len 20480
3158 030/300 Write block 999 len 16384 data: 5 5 4 3
3159 031/300 Write block 3079 len 20480 data: 3 3 3 3 3
3160 032/300 Read block 1493 len 4096
3161 033/300 Read block 4321 len 12288
3162 034/300 Write block 1317 len 32768 data: 2 2 2 2 2 2 3 3
3163 035/300 Write block 2674 len 16384 data: 2 2 2 3
3164 036/300 Write block 521 len 20480 data: 4 3 3 2 2
3165 037/300 Read block 4634 len 28672
3166 038/300 Read block 3472 len 24576
3167 039/300 Write block 4525 len 4096 data: 2
3168 040/300 Read block 3432 len 40960
3169 041/300 Write block 4215 len 16384 data: 2 2 2 2
3170 042/300 Read block 1180 len 8192
3171 043/300 Read block 2056 len 16384
3172 044/300 Read block 2096 len 20480
3173 045/300 Read block 4989 len 20480
3174 046/300 Read block 4197 len 36864
3175 047/300 Write block 3946 len 36864 data: 3 3 3 3 3 3 3 3 2
3176 048/300 Read block 3595 len 36864
3177 049/300 Write block 3233 len 4096 data: 2
3178 050/300 Read block 4273 len 12288
3179 051/300 Read block 4199 len 36864
3180 052/300 Read block 3983 len 4096
3181 053/300 Write block 3204 len 16384 data: 3 3 3 3
3182 054/300 Read block 1679 len 20480
3183 055/300 Write block 2640 len 4096 data: 2
3184 056/300 Write block 3761 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3185 057/300 Write block 2158 len 32768 data: 2 2 2 2 2 2 2 2
3186 058/300 Read block 2806 len 32768
3187 059/300 Read block 4748 len 8192
3188 060/300 Write block 1837 len 36864 data: 2 2 2 2 2 2 2 2 2
3189 061/300 Write block 2447 len 28672 data: 2 2 2 2 2 2 2
3190 062/300 Read block 1441 len 16384
3191 063/300 Write block 3437 len 20480 data: 2 2 2 2 3
3192 064/300 Read block 4534 len 8192
3193 065/300 Write block 3813 len 28672 data: 2 2 2 3 4 4 3
3194 066/300 Read block 3859 len 36864
3195 067/300 Read block 4060 len 8192
3196 068/300 Read block 1118 len 4096
3197 069/300 Write block 694 len 32768 data: 2 2 2 2 2 2 2 2
3198 070/300 Read block 622 len 20480
3199 071/300 Read block 2322 len 24576
3200 072/300 Flush
3201 073/300 Read block 1041 len 36864
3202 074/300 Write block 2390 len 28672 data: 2 2 2 2 2 2 2
3203 075/300 Flush
3204 076/300 Write block 1232 len 12288 data: 2 2 3
3205 077/300 Flush
3206 078/300 Read block 3917 len 28672
3207 079/300 Read block 2690 len 16384
3208 080/300 Read block 4364 len 16384
3209 081/300 Read block 1723 len 24576
3210 082/300 Write block 1969 len 36864 data: 6 5 5 4 4 3 3 2 2
3211 083/300 Write block 2966 len 16384 data: 2 2 2 2
3212 084/300 Read block 1383 len 28672
3213 085/300 Write block 198 len 32768 data: 3 3 2 2 3 3 3 3
3214 086/300 Read block 708 len 36864
3215 087/300 Read block 4691 len 28672
3216 088/300 Read block 815 len 4096
3217 089/300 Read block 367 len 20480
3218 090/300 Read block 2740 len 36864
3219 091/300 Read block 411 len 8192
3220 092/300 Read block 3010 len 32768
3221 093/300 Read block 4746 len 4096
3222 094/300 Read block 1813 len 16384
3223 095/300 Write block 4536 len 36864 data: 3 3 3 2 2 2 2 2 2
3224 096/300 Read block 3759 len 28672
3225 097/300 Write block 1583 len 36864 data: 2 2 2 2 2 2 2 2 2
3226 098/300 Read block 3956 len 8192
3227 099/300 Write block 3802 len 32768 data: 2 2 2 2 2 2 2 2
3228 100/300 Read block 1892 len 20480
3229 101/300 Flush
3230 102/300 Read block 890 len 8192
3231 103/300 Read block 1411 len 12288
3232 104/300 Read block 2010 len 12288
3233 105/300 Write block 622 len 32768 data: 2 2 2 2 2 2 2 2
3234 106/300 Write block 3458 len 36864 data: 2 2 2 2 2 2 2 3 3
3235 107/300 Read block 2767 len 16384
3236 108/300 Flush
3237 109/300 Flush
3238 110/300 Read block 2955 len 4096
3239 111/300 Flush
3240 112/300 Write block 2653 len 8192 data: 2 2
3241 113/300 Read block 2137 len 4096
3242 114/300 Read block 284 len 16384
3243 115/300 Read block 4497 len 8192
3244 116/300 Read block 3807 len 8192
3245 117/300 Read block 4203 len 24576
3246 118/300 Read block 1802 len 28672
3247 119/300 Flush
3248 120/300 Read block 4464 len 36864
3249 121/300 Write block 4768 len 8192 data: 2 2
3250 122/300 Write block 4359 len 16384 data: 2 2 2 2
3251 123/300 Read block 517 len 32768
3252 124/300 Read block 119 len 28672
3253 125/300 Read block 2619 len 20480
3254 126/300 Flush
3255 127/300 Write block 3976 len 4096 data: 2
3256 128/300 Read block 1381 len 12288
3257 129/300 Read block 661 len 12288
3258 130/300 Write block 1297 len 40960 data: 4 4 4 5 4 4 3 2 2 2
3259 131/300 Read block 3733 len 36864
3260 132/300 Read block 4143 len 40960
3261 133/300 Write block 523 len 36864 data: 4 3 3 2 2 3 4 4 4
3262 134/300 Flush
3263 135/300 Read block 52 len 4096
3264 136/300 Read block 3770 len 20480
3265 137/300 Read block 3925 len 24576
3266 138/300 Write block 1067 len 4096 data: 2
3267 139/300 Read block 1030 len 8192
3268 140/300 Write block 487 len 4096 data: 3
3269 141/300 Read block 383 len 12288
3270 142/300 Flush
3271 143/300 Write block 568 len 36864 data: 2 2 2 2 2 2 2 2 2
3272 144/300 Read block 881 len 32768
3273 145/300 Flush
3274 146/300 Read block 4103 len 24576
3275 147/300 Read block 2894 len 16384
3276 148/300 Read block 2692 len 36864
3277 149/300 Read block 3607 len 36864
3278 150/300 Read block 2127 len 28672
3279 151/300 Read block 1234 len 40960
3280 152/300 Read block 3520 len 28672
3281 153/300 Read block 1551 len 40960
3282 154/300 Flush
3283 155/300 Read block 4582 len 24576
3284 156/300 Write block 1905 len 24576 data: 2 2 2 2 2 2
3285 157/300 Write block 2121 len 28672 data: 2 3 2 2 2 2 2
3286 158/300 Write block 4057 len 32768 data: 3 3 2 2 2 3 3 3
3287 159/300 Flush
3288 160/300 Write block 3694 len 16384 data: 3 3 3 3
3289 161/300 Flush
3290 162/300 Write block 1781 len 8192 data: 2 3
3291 163/300 Write block 762 len 28672 data: 2 2 2 2 2 2 2
3292 164/300 Read block 4412 len 20480
3293 165/300 Write block 367 len 28672 data: 2 2 2 3 3 2 2
3294 166/300 Read block 507 len 40960
3295 167/300 Read block 2989 len 8192
3296 168/300 Flush
3297 169/300 Write block 4806 len 32768 data: 2 2 2 2 2 2 2 3
3298 170/300 Write block 420 len 20480 data: 2 2 2 2 2
3299 171/300 Read block 3963 len 28672
3300 172/300 Write block 1903 len 32768 data: 2 2 3 3 3 3 3 3
3301 173/300 Read block 1469 len 40960
3302 174/300 Read block 3302 len 24576
3303 175/300 Write block 3813 len 32768 data: 3 3 3 4 5 5 4 2
3304 176/300 Read block 3788 len 36864
3305 177/300 Flush
3306 178/300 Read block 1063 len 40960
3307 179/300 Read block 464 len 36864
3308 180/300 Flush
3309 181/300 Read block 1302 len 20480
3310 182/300 Read block 1657 len 8192
3311 183/300 Write block 409 len 28672 data: 2 2 2 2 3 2 2
3312 184/300 Write block 1694 len 36864 data: 3 3 2 2 3 3 3 2 2
3313 185/300 Read block 422 len 32768
3314 186/300 Flush
3315 187/300 Write block 3109 len 20480 data: 4 3 3 3 3
3316 188/300 Write block 4449 len 28672 data: 3 3 4 4 3 3 3
3317 189/300 Read block 3891 len 16384
3318 190/300 Write block 2199 len 8192 data: 2 2
3319 191/300 Flush
3320 192/300 Flush
3321 193/300 Write block 3122 len 12288 data: 2 2 2
3322 194/300 Read block 4917 len 40960
3323 195/300 Write block 4981 len 24576 data: 2 2 2 2 2 2
3324 196/300 Write block 1772 len 8192 data: 2 2
3325 197/300 Read block 1386 len 12288
3326 198/300 Write block 2071 len 28672 data: 2 2 2 2 2 2 2
3327 199/300 Write block 1432 len 36864 data: 4 4 4 5 5 4 4 3 2
3328 200/300 Read block 1269 len 20480
3329 201/300 Write block 723 len 4096 data: 3
3330 202/300 Read block 4516 len 24576
3331 203/300 Read block 877 len 20480
3332 204/300 Write block 2450 len 36864 data: 3 3 3 3 2 2 2 2 2
3333 205/300 Write block 1796 len 32768 data: 2 2 2 2 2 2 2 2
3334 206/300 Read block 3216 len 40960
3335 207/300 Read block 1740 len 4096
3336 208/300 Read block 3832 len 32768
3337 209/300 Read block 2784 len 32768
3338 210/300 Read block 3974 len 36864
3339 211/300 Write block 2522 len 16384 data: 2 2 2 2
3340 212/300 Read block 434 len 20480
3341 213/300 Write block 28 len 4096 data: 2
3342 214/300 Read block 1339 len 40960
3343 215/300 Read block 2859 len 8192
3344 216/300 Read block 1303 len 24576
3345 217/300 Read block 3960 len 20480
3346 218/300 Write block 919 len 36864 data: 3 3 2 3 3 3 3 3 3
3347 219/300 Write block 3648 len 20480 data: 2 2 2 2 2
3348 220/300 Read block 1895 len 36864
3349 221/300 Read block 2570 len 28672
3350 222/300 Write block 2274 len 40960 data: 3 3 2 2 2 2 2 3 3 3
3351 223/300 Flush
3352 224/300 Write block 943 len 12288 data: 2 2 2
3353 225/300 Write block 2535 len 20480 data: 2 2 2 2 2
3354 226/300 Write block 2561 len 20480 data: 2 3 3 3 3
3355 227/300 Write block 1938 len 36864 data: 4 3 3 3 2 2 2 2 2
3356 228/300 Write block 2424 len 4096 data: 2
3357 229/300 Write block 1264 len 24576 data: 3 3 3 2 2 2
3358 230/300 Write block 1594 len 4096 data: 2
3359 231/300 Read block 1710 len 28672
3360 232/300 Write block 1989 len 16384 data: 3 2 2 2
3361 233/300 Read block 3246 len 20480
3362 234/300 Flush
3363 235/300 Flush
3364 236/300 Read block 4814 len 40960
3365 237/300 Read block 4938 len 32768
3366 238/300 Read block 1803 len 12288
3367 239/300 Read block 211 len 8192
3368 240/300 Read block 1425 len 24576
3369 241/300 Read block 422 len 40960
3370 242/300 Flush
3371 243/300 Read block 1155 len 8192
3372 244/300 Read block 1423 len 20480
3373 245/300 Write block 4103 len 20480 data: 4 4 3 3 3
3374 246/300 Read block 4631 len 12288
3375 247/300 Read block 4227 len 40960
3376 248/300 Read block 785 len 24576
3377 249/300 Read block 868 len 12288
3378 250/300 Read block 4142 len 32768
3379 251/300 Read block 2744 len 12288
3380 252/300 Read block 98 len 36864
3381 253/300 Read block 427 len 32768
3382 254/300 Read block 755 len 4096
3383 255/300 Write block 2119 len 28672 data: 2 2 3 4 3 3 3
3384 256/300 Read block 3919 len 16384
3385 257/300 Write block 3058 len 20480 data: 2 2 2 2 2
3386 258/300 Read block 3436 len 28672
3387 259/300 Write block 248 len 28672 data: 5 3 2 3 3 3 3
3388 260/300 Read block 4184 len 20480
3389 261/300 Write block 4318 len 16384 data: 4 4 5 6
3390 262/300 Read block 1036 len 4096
3391 263/300 Read block 3230 len 8192
3392 264/300 Read block 2272 len 16384
3393 265/300 Write block 2298 len 24576 data: 3 3 3 3 2 2
3394 266/300 Read block 159 len 8192
3395 267/300 Read block 4640 len 8192
3396 268/300 Read block 4372 len 32768
3397 269/300 Write block 2752 len 20480 data: 2 2 2 2 2
3398 270/300 Write block 115 len 16384 data: 2 2 2 2
3399 271/300 Read block 1898 len 40960
3400 272/300 Write block 2722 len 32768 data: 3 3 3 3 2 2 2 2
3401 273/300 Write block 446 len 16384 data: 2 2 2 2
3402 274/300 Read block 2147 len 28672
3403 275/300 Read block 1181 len 4096
3404 276/300 Write block 2607 len 32768 data: 2 2 2 2 3 3 3 3
3405 277/300 Read block 2147 len 4096
3406 278/300 Read block 728 len 40960
3407 279/300 Read block 3268 len 40960
3408 280/300 Read block 3508 len 36864
3409 281/300 Write block 384 len 12288 data: 2 2 2
3410 282/300 Read block 438 len 12288
3411 283/300 Read block 1351 len 24576
3412 284/300 Read block 381 len 36864
3413 285/300 Read block 4305 len 12288
3414 286/300 Flush
3415 287/300 Write block 4943 len 40960 data: 4 4 3 3 3 3 3 2 3 3
3416 288/300 Read block 4248 len 8192
3417 289/300 Write block 1260 len 4096 data: 3
3418 290/300 Write block 3441 len 4096 data: 4
3419 291/300 Write block 594 len 32768 data: 3 3 3 3 3 3 3 4
3420 292/300 Flush
3421 293/300 Read block 4780 len 20480
3422 294/300 Read block 223 len 40960
3423 295/300 Read block 3696 len 36864
3424 296/300 Read block 2125 len 12288
3425 297/300 Read block 802 len 32768
3426 298/300 Write block 4928 len 4096 data: 2
3427 299/300 Write block 1460 len 32768 data: 2 2 2 2 2 2 2 2
3428 300/300 Flush
3429 301/300 Write block 1086 len 40960 data: 4 4 5 5 4 3 3 2 2 2
3430 Main task has work to do, go find it
3431 got dsc Start(1)
3432 start 1
3433 Nov 01 16:51:13.426 INFO request completed, latency_us: 283, response_code: 204, uri: /start/cid/1, method: POST, req_id: 0df1734d-3690-4954-bbf2-a52b071174bb, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
3434 Replay: started 1, returned:Ok(())
3435 ----------------------------------------------------------------
3436 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
3437 States: Active Offline Active
3438 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3439 1907 Acked Read 7 Done Sent Done false
3440 1908 Acked Write 5 Done Sent Done false
3441 1909 Acked Write 2 Done Sent Done false
3442 1910 Acked Write 4 Done Sent Done false
3443 1911 Acked Read 8 Done Sent Done false
3444 1912 Acked Write 9 Done Sent Done false
3445 1913 Acked Read 3 Done Sent Done false
3446 1914 Acked Read 8 Done Sent Done false
3447 1915 Acked Read 1 Done Sent Done false
3448 1916 Acked Read 6 Done Sent Done false
3449 1917 Acked Write 9 Done Sent Done false
3450 1918 Acked Read 7 Done Sent Done false
3451 1919 Acked Read 10 Done Sent Done false
3452 1920 Acked Write 5 Done Sent Done false
3453 1921 Acked Read 8 Done Sent Done false
3454 1922 Acked Write 5 Done Sent Done false
3455 1923 Acked Write 1 Done Sent Done false
3456 1924 Acked Write 1 Done Sent Done false
3457 1925 Acked Read 6 Done Sent Done false
3458 1926 Acked Write 7 Done Sent Done false
3459 1927 Acked Read 3 Done Sent Done false
3460 1928 Acked Write 10 Done Sent Done false
3461 1929 Acked Write 3 Done Sent Done false
3462 1930 Acked Read 7 Done Sent Done false
3463 1931 Acked Read 6 Done Sent Done false
3464 1932 Acked Write 3 Done Sent Done false
3465 1933 Acked Read 4 Done Sent Done false
3466 1934 Acked Write 2 Done Sent Done false
3467 1935 Acked Read 5 Done Sent Done false
3468 1936 Acked Write 4 Done Sent Done false
3469 1937 Acked Write 5 Done Sent Done false
3470 1938 Acked Read 1 Done Sent Done false
3471 1939 Acked Read 3 Done Sent Done false
3472 1940 Acked Write 8 Done Sent Done false
3473 1941 Acked Write 4 Done Sent Done false
3474 1942 Acked Write 5 Done Sent Done false
3475 1943 Acked Read 7 Done Sent Done false
3476 1944 Acked Read 6 Done Sent Done false
3477 1945 Acked Write 1 Done Sent Done false
3478 1946 Acked Read 10 Done Sent Done false
3479 1947 Acked Write 4 Done Sent Done false
3480 1948 Acked Read 2 Done Sent Done false
3481 1949 Acked Read 4 Done Sent Done false
3482 1950 Acked Read 5 Done Sent Done false
3483 1951 Acked Read 5 Done Sent Done false
3484 1952 Acked Read 9 Done Sent Done false
3485 1953 Acked Write 9 Done Sent Done false
3486 1954 Acked Read 9 Done Sent Done false
3487 1955 Acked Write 1 Done Sent Done false
3488 1956 Acked Read 3 Done Sent Done false
3489 1957 Acked Read 9 Done Sent Done false
3490 1958 Acked Read 1 Done Sent Done false
3491 1959 Acked Write 4 Done Sent Done false
3492 1960 Acked Read 5 Done Sent Done false
3493 1961 Acked Write 1 Done Sent Done false
3494 1962 Acked Write 10 Done Sent Done false
3495 1963 Acked Write 8 Done Sent Done false
3496 1964 Acked Read 8 Done Sent Done false
3497 1965 Acked Read 2 Done Sent Done false
3498 1966 Acked Write 9 Done Sent Done false
3499 1967 Acked Write 7 Done Sent Done false
3500 1968 Acked Read 4 Done Sent Done false
3501 1969 Acked Write 5 Done Sent Done false
3502 1970 Acked Read 2 Done Sent Done false
3503 1971 Acked Write 7 Done Sent Done false
3504 1972 Acked Read 9 Done Sent Done false
3505 1973 Acked Read 2 Done Sent Done false
3506 1974 Acked Read 1 Done Sent Done false
3507 1975 Acked Write 8 Done Sent Done false
3508 1976 Acked Read 5 Done Sent Done false
3509 1977 Acked Read 6 Done Sent Done false
3510 1978 Acked Flush 0 Done Sent Done false
3511 1979 Acked Read 9 Done Sent Done false
3512 1980 Acked Write 7 Done Sent Done false
3513 1981 Acked Flush 0 Done Sent Done false
3514 1982 Acked Write 3 Done Sent Done false
3515 1983 Acked Flush 0 Done Sent Done false
3516 1984 Acked Read 7 Done Sent Done false
3517 1985 Acked Read 4 Done Sent Done false
3518 1986 Acked Read 4 Done Sent Done false
3519 1987 Acked Read 6 Done Sent Done false
3520 1988 Acked Write 9 Done Sent Done false
3521 1989 Acked Write 4 Done Sent Done false
3522 1990 Acked Read 7 Done Sent Done false
3523 1991 Acked Write 8 Done Sent Done false
3524 1992 Acked Read 9 Done Sent Done false
3525 1993 Acked Read 7 Done Sent Done false
3526 1994 Acked Read 1 Done Sent Done false
3527 1995 Acked Read 5 Done Sent Done false
3528 1996 Acked Read 9 Done Sent Done false
3529 1997 Acked Read 2 Done Sent Done false
3530 1998 Acked Read 8 Done Sent Done false
3531 1999 Acked Read 1 Done Sent Done false
3532 2000 Acked Read 4 Done Sent Done false
3533 2001 Acked Write 9 Done Sent Done false
3534 2002 Acked Read 7 Done Sent Done false
3535 2003 Acked Write 9 Done Sent Done false
3536 2004 Acked Read 2 Done Sent Done false
3537 2005 Acked Write 8 Done Sent Done false
3538 2006 Acked Read 5 Done Sent Done false
3539 2007 Acked Flush 0 Done Sent Done false
3540 2008 Acked Read 2 Done Sent Done false
3541 2009 Acked Read 3 Done Sent Done false
3542 2010 Acked Read 3 Done Sent Done false
3543 2011 Acked Write 8 Done Sent Done false
3544 2012 Acked Write 9 Done Sent Done false
3545 2013 Acked Read 4 Done Sent Done false
3546 2014 Acked Flush 0 Done Sent Done false
3547 2015 Acked Flush 0 Done Sent Done false
3548 2016 Acked Read 1 Done Sent Done false
3549 2017 Acked Flush 0 Done Sent Done false
3550 2018 Acked Write 2 Done Sent Done false
3551 2019 Acked Read 1 Done Sent Done false
3552 2020 Acked Read 4 Done Sent Done false
3553 2021 Acked Read 2 Done Sent Done false
3554 2022 Acked Read 2 Done Sent Done false
3555 2023 Acked Read 6 Done Sent Done false
3556 2024 Acked Read 7 Done Sent Done false
3557 2025 Acked Flush 0 Done Sent Done false
3558 2026 Acked Read 9 Done Sent Done false
3559 2027 Acked Write 2 Done Sent Done false
3560 2028 Acked Write 4 Done Sent Done false
3561 2029 Acked Read 8 Done Sent Done false
3562 2030 Acked Read 7 Done Sent Done false
3563 2031 Acked Read 5 Done Sent Done false
3564 2032 Acked Flush 0 Done Sent Done false
3565 2033 Acked Write 1 Done Sent Done false
3566 2034 Acked Read 3 Done Sent Done false
3567 2035 Acked Read 3 Done Sent Done false
3568 2036 Acked Write 10 Done Sent Done false
3569 2037 Acked Read 9 Done Sent Done false
3570 2038 Acked Read 10 Done Sent Done false
3571 2039 Acked Write 9 Done Sent Done false
3572 2040 Acked Flush 0 Done Sent Done false
3573 2041 Acked Read 1 Done Sent Done false
3574 2042 Acked Read 5 Done Sent Done false
3575 2043 Acked Read 6 Done Sent Done false
3576 2044 Acked Write 1 Done Sent Done false
3577 2045 Acked Read 2 Done Sent Done false
3578 2046 Acked Write 1 Done Sent Done false
3579 2047 Acked Read 3 Done Sent Done false
3580 2048 Acked Flush 0 Done Sent Done false
3581 2049 Acked Write 9 Done Sent Done false
3582 2050 Acked Read 8 Done Sent Done false
3583 2051 Acked Flush 0 Done Sent Done false
3584 2052 Acked Read 6 Done Sent Done false
3585 2053 Acked Read 4 Done Sent Done false
3586 2054 Acked Read 9 Done Sent Done false
3587 2055 Acked Read 9 Done Sent Done false
3588 2056 Acked Read 7 Done Sent Done false
3589 2057 Acked Read 10 Done Sent Done false
3590 2058 Acked Read 7 Done Sent Done false
3591 2059 Acked Read 10 Done Sent Done false
3592 2060 Acked Flush 0 Done Sent Done false
3593 2061 Acked Read 6 Done Sent Done false
3594 2062 Acked Write 6 Done Sent Done false
3595 2063 Acked Write 7 Done Sent Done false
3596 2064 Acked Write 8 Done Sent Done false
3597 2065 Acked Flush 0 Done Sent Done false
3598 2066 Acked Write 4 Done Sent Done false
3599 2067 Acked Flush 0 Done Sent Done false
3600 2068 Acked Write 2 Done Sent Done false
3601 2069 Acked Write 7 Done Sent Done false
3602 2070 Acked Read 5 Done Sent Done false
3603 2071 Acked Write 7 Done Sent Done false
3604 2072 Acked Read 10 Done Sent Done false
3605 2073 Acked Read 2 Done Sent Done false
3606 2074 Acked Flush 0 Done Sent Done false
3607 2075 Acked Write 8 Done Sent Done false
3608 2076 Acked Write 5 Done Sent Done false
3609 2077 Acked Read 7 Done Sent Done false
3610 2078 Acked Write 8 Done Sent Done false
3611 2079 Acked Read 10 Done Sent Done false
3612 2080 Acked Read 6 Done Sent Done false
3613 2081 Acked Write 8 Done Sent Done false
3614 2082 Acked Read 9 Done Sent Done false
3615 2083 Acked Flush 0 Done Sent Done false
3616 2084 Acked Read 10 Done Sent Done false
3617 2085 Acked Read 9 Done Sent Done false
3618 2086 Acked Flush 0 Done Sent Done false
3619 2087 Acked Read 5 Done Sent Done false
3620 2088 Acked Read 2 Done Sent Done false
3621 2089 Acked Write 7 Done Sent Done false
3622 2090 Acked Flush 0 Done Sent Done false
3623 2091 Acked Write 9 Done Sent Done false
3624 2092 Acked Read 8 Done Sent Done false
3625 2093 Acked Flush 0 Done Sent Done false
3626 2094 Acked Write 5 Done Sent Done false
3627 2095 Acked Write 7 Done Sent Done false
3628 2096 Acked Read 4 Done Sent Done false
3629 2097 Acked Write 2 Done Sent Done false
3630 2098 Acked Flush 0 Done Sent Done false
3631 2099 Acked Flush 0 Done Sent Done false
3632 2100 Acked Write 3 Done Sent Done false
3633 2101 Acked Read 10 Done Sent Done false
3634 2102 Acked Write 6 Done Sent Done false
3635 2103 Acked Write 2 Done Sent Done false
3636 2104 Acked Read 3 Done Sent Done false
3637 2105 Acked Write 7 Done Sent Done false
3638 2106 Acked Write 9 Done Sent Done false
3639 2107 Acked Read 5 Done Sent Done false
3640 2108 Acked Write 1 Done Sent Done false
3641 2109 Acked Read 6 Done Sent Done false
3642 2110 Acked Read 5 Done Sent Done false
3643 2111 Acked Write 9 Done Sent Done false
3644 2112 Acked Write 8 Done Sent Done false
3645 2113 Acked Read 10 Done Sent Done false
3646 2114 Acked Read 1 Done Sent Done false
3647 2115 Acked Read 8 Done Sent Done false
3648 2116 Acked Read 8 Done Sent Done false
3649 2117 Acked Read 9 Done Sent Done false
3650 2118 Acked Write 4 Done Sent Done false
3651 2119 Acked Read 5 Done Sent Done false
3652 2120 Acked Write 1 Done Sent Done false
3653 2121 Acked Read 10 Done Sent Done false
3654 2122 Acked Read 2 Done Sent Done false
3655 2123 Acked Read 6 Done Sent Done false
3656 2124 Acked Read 5 Done Sent Done false
3657 2125 Acked Write 9 Done Sent Done false
3658 2126 Acked Write 5 Done Sent Done false
3659 2127 Acked Read 9 Done Sent Done false
3660 2128 Acked Read 7 Done Sent Done false
3661 2129 Acked Write 10 Done Sent Done false
3662 2130 Acked Flush 0 Done Sent Done false
3663 2131 Acked Write 3 Done Sent Done false
3664 2132 Acked Write 5 Done Sent Done false
3665 2133 Acked Write 5 Done Sent Done false
3666 2134 Acked Write 9 Done Sent Done false
3667 2135 Acked Write 1 Done Sent Done false
3668 2136 Acked Write 6 Done Sent Done false
3669 2137 Acked Write 1 Done Sent Done false
3670 2138 Acked Read 7 Done Sent Done false
3671 2139 Acked Write 4 Done Sent Done false
3672 2140 Acked Read 5 Done Sent Done false
3673 2141 Acked Flush 0 Done Sent Done false
3674 2142 Acked Flush 0 Done Sent Done false
3675 2143 Acked Read 10 Done Sent Done false
3676 2144 Acked Read 8 Done Sent Done false
3677 2145 Acked Read 3 Done Sent Done false
3678 2146 Acked Read 2 Done Sent Done false
3679 2147 Acked Read 6 Done Sent Done false
3680 2148 Acked Read 10 Done Sent Done false
3681 2149 Acked Flush 0 Done Sent Done false
3682 2150 Acked Read 2 Done Sent Done false
3683 2151 Acked Read 5 Done Sent Done false
3684 2152 Acked Write 5 Done Sent Done false
3685 2153 Acked Read 3 Done Sent Done false
3686 2154 Acked Read 10 Done Sent Done false
3687 2155 Acked Read 6 Done Sent Done false
3688 2156 Acked Read 3 Done Sent Done false
3689 2157 Acked Read 8 Done Sent Done false
3690 2158 Acked Read 3 Done Sent Done false
3691 2159 Acked Read 9 Done Sent Done false
3692 2160 Acked Read 8 Done Sent Done false
3693 2161 Acked Read 1 Done Sent Done false
3694 2162 Acked Write 7 Done Sent Done false
3695 2163 Acked Read 4 Done Sent Done false
3696 2164 Acked Write 5 Done Sent Done false
3697 2165 Acked Read 7 Done Sent Done false
3698 2166 Acked Write 7 Done Sent Done false
3699 2167 Acked Read 5 Done Sent Done false
3700 2168 Acked Write 4 Done Sent Done false
3701 2169 Acked Read 1 Done Sent Done false
3702 2170 Acked Read 2 Done Sent Done false
3703 2171 Acked Read 4 Done Sent Done false
3704 2172 Acked Write 6 Done Sent Done false
3705 2173 Acked Read 2 Done Sent Done false
3706 2174 Acked Read 2 Done Sent Done false
3707 2175 Acked Read 8 Done Sent Done false
3708 2176 Acked Write 5 Done Sent Done false
3709 2177 Acked Write 4 Done Sent Done false
3710 2178 Acked Read 10 Done Sent Done false
3711 2179 Acked Write 8 Done Sent Done false
3712 2180 Acked Write 4 Done Sent Done false
3713 2181 Acked Read 7 Done Sent Done false
3714 2182 Acked Read 1 Done Sent Done false
3715 2183 Acked Write 8 Done Sent Done false
3716 2184 Acked Read 1 Done Sent Done false
3717 2185 Acked Read 10 Done Sent Done false
3718 2186 Acked Read 10 Done Sent Done false
3719 2187 Acked Read 9 Done Sent Done false
3720 2188 Acked Write 3 Done Sent Done false
3721 2189 Acked Read 3 Done Sent Done false
3722 2190 Acked Read 6 Done Sent Done false
3723 2191 Acked Read 9 Done Sent Done false
3724 2192 Acked Read 3 Done Sent Done false
3725 2193 Acked Flush 0 Done Sent Done false
3726 2194 Acked Write 10 Done Sent Done false
3727 2195 Acked Read 2 Done Sent Done false
3728 2196 Acked Write 1 Done Sent Done false
3729 2197 Acked Write 1 Done Sent Done false
3730 2198 Acked Write 8 Done Sent Done false
3731 2199 Acked Flush 0 Done Sent Done false
3732 2200 Acked Read 5 Done Sent Done false
3733 2201 Acked Read 10 Done Sent Done false
3734 2202 Acked Read 9 Done Sent Done false
3735 2203 Acked Read 3 Done Sent Done false
3736 2204 Acked Read 8 Done Sent Done false
3737 2205 Acked Write 1 Done Sent Done false
3738 2206 Acked Write 8 Done Sent Done false
3739 2207 Acked Flush 0 Done Sent Done false
3740 2208 Acked Write 10 Sent Sent Sent false
3741 STATES DS:0 DS:1 DS:2 TOTAL
3742 Sent 1 302 1 304
3743 Done 301 0 301 602
3744 Skipped 0 0 0 0
3745 Error 0 0 0 0
3746 Last Flush: 2207 1906 2207
3747 Downstairs last five completed: 1906 1905 1904 1903 1902
3748 Upstairs last five completed: 2208 2207 2206 2205 2204
3749 CLIENT: Up:0 ds:302 act:2
3750 [8820] got start action
3751 [8820] I am going to restart
3752 Starting downstairs at port 8820
3753 Make output file at "/tmp/dsc/downstairs-8820.txt"
3754 [8820][1] reports Starting
3755 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1202)
3756 [8820][1] reports Running
3757 ----------------------------------------------------------------
3758 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
3759 States: Active Offline Active
3760 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3761 1907 Acked Read 7 Done Sent Done false
3762 1908 Acked Write 5 Done Sent Done false
3763 1909 Acked Write 2 Done Sent Done false
3764 1910 Acked Write 4 Done Sent Done false
3765 1911 Acked Read 8 Done Sent Done false
3766 1912 Acked Write 9 Done Sent Done false
3767 1913 Acked Read 3 Done Sent Done false
3768 1914 Acked Read 8 Done Sent Done false
3769 1915 Acked Read 1 Done Sent Done false
3770 1916 Acked Read 6 Done Sent Done false
3771 1917 Acked Write 9 Done Sent Done false
3772 1918 Acked Read 7 Done Sent Done false
3773 1919 Acked Read 10 Done Sent Done false
3774 1920 Acked Write 5 Done Sent Done false
3775 1921 Acked Read 8 Done Sent Done false
3776 1922 Acked Write 5 Done Sent Done false
3777 1923 Acked Write 1 Done Sent Done false
3778 1924 Acked Write 1 Done Sent Done false
3779 1925 Acked Read 6 Done Sent Done false
3780 1926 Acked Write 7 Done Sent Done false
3781 1927 Acked Read 3 Done Sent Done false
3782 1928 Acked Write 10 Done Sent Done false
3783 1929 Acked Write 3 Done Sent Done false
3784 1930 Acked Read 7 Done Sent Done false
3785 1931 Acked Read 6 Done Sent Done false
3786 1932 Acked Write 3 Done Sent Done false
3787 1933 Acked Read 4 Done Sent Done false
3788 1934 Acked Write 2 Done Sent Done false
3789 1935 Acked Read 5 Done Sent Done false
3790 1936 Acked Write 4 Done Sent Done false
3791 1937 Acked Write 5 Done Sent Done false
3792 1938 Acked Read 1 Done Sent Done false
3793 1939 Acked Read 3 Done Sent Done false
3794 1940 Acked Write 8 Done Sent Done false
3795 1941 Acked Write 4 Done Sent Done false
3796 1942 Acked Write 5 Done Sent Done false
3797 1943 Acked Read 7 Done Sent Done false
3798 1944 Acked Read 6 Done Sent Done false
3799 1945 Acked Write 1 Done Sent Done false
3800 1946 Acked Read 10 Done Sent Done false
3801 1947 Acked Write 4 Done Sent Done false
3802 1948 Acked Read 2 Done Sent Done false
3803 1949 Acked Read 4 Done Sent Done false
3804 1950 Acked Read 5 Done Sent Done false
3805 1951 Acked Read 5 Done Sent Done false
3806 1952 Acked Read 9 Done Sent Done false
3807 1953 Acked Write 9 Done Sent Done false
3808 1954 Acked Read 9 Done Sent Done false
3809 1955 Acked Write 1 Done Sent Done false
3810 1956 Acked Read 3 Done Sent Done false
3811 1957 Acked Read 9 Done Sent Done false
3812 1958 Acked Read 1 Done Sent Done false
3813 1959 Acked Write 4 Done Sent Done false
3814 1960 Acked Read 5 Done Sent Done false
3815 1961 Acked Write 1 Done Sent Done false
3816 1962 Acked Write 10 Done Sent Done false
3817 1963 Acked Write 8 Done Sent Done false
3818 1964 Acked Read 8 Done Sent Done false
3819 1965 Acked Read 2 Done Sent Done false
3820 1966 Acked Write 9 Done Sent Done false
3821 1967 Acked Write 7 Done Sent Done false
3822 1968 Acked Read 4 Done Sent Done false
3823 1969 Acked Write 5 Done Sent Done false
3824 1970 Acked Read 2 Done Sent Done false
3825 1971 Acked Write 7 Done Sent Done false
3826 1972 Acked Read 9 Done Sent Done false
3827 1973 Acked Read 2 Done Sent Done false
3828 1974 Acked Read 1 Done Sent Done false
3829 1975 Acked Write 8 Done Sent Done false
3830 1976 Acked Read 5 Done Sent Done false
3831 1977 Acked Read 6 Done Sent Done false
3832 1978 Acked Flush 0 Done Sent Done false
3833 1979 Acked Read 9 Done Sent Done false
3834 1980 Acked Write 7 Done Sent Done false
3835 1981 Acked Flush 0 Done Sent Done false
3836 1982 Acked Write 3 Done Sent Done false
3837 1983 Acked Flush 0 Done Sent Done false
3838 1984 Acked Read 7 Done Sent Done false
3839 1985 Acked Read 4 Done Sent Done false
3840 1986 Acked Read 4 Done Sent Done false
3841 1987 Acked Read 6 Done Sent Done false
3842 1988 Acked Write 9 Done Sent Done false
3843 1989 Acked Write 4 Done Sent Done false
3844 1990 Acked Read 7 Done Sent Done false
3845 1991 Acked Write 8 Done Sent Done false
3846 1992 Acked Read 9 Done Sent Done false
3847 1993 Acked Read 7 Done Sent Done false
3848 1994 Acked Read 1 Done Sent Done false
3849 1995 Acked Read 5 Done Sent Done false
3850 1996 Acked Read 9 Done Sent Done false
3851 1997 Acked Read 2 Done Sent Done false
3852 1998 Acked Read 8 Done Sent Done false
3853 1999 Acked Read 1 Done Sent Done false
3854 2000 Acked Read 4 Done Sent Done false
3855 2001 Acked Write 9 Done Sent Done false
3856 2002 Acked Read 7 Done Sent Done false
3857 2003 Acked Write 9 Done Sent Done false
3858 2004 Acked Read 2 Done Sent Done false
3859 2005 Acked Write 8 Done Sent Done false
3860 2006 Acked Read 5 Done Sent Done false
3861 2007 Acked Flush 0 Done Sent Done false
3862 2008 Acked Read 2 Done Sent Done false
3863 2009 Acked Read 3 Done Sent Done false
3864 2010 Acked Read 3 Done Sent Done false
3865 2011 Acked Write 8 Done Sent Done false
3866 2012 Acked Write 9 Done Sent Done false
3867 2013 Acked Read 4 Done Sent Done false
3868 2014 Acked Flush 0 Done Sent Done false
3869 2015 Acked Flush 0 Done Sent Done false
3870 2016 Acked Read 1 Done Sent Done false
3871 2017 Acked Flush 0 Done Sent Done false
3872 2018 Acked Write 2 Done Sent Done false
3873 2019 Acked Read 1 Done Sent Done false
3874 2020 Acked Read 4 Done Sent Done false
3875 2021 Acked Read 2 Done Sent Done false
3876 2022 Acked Read 2 Done Sent Done false
3877 2023 Acked Read 6 Done Sent Done false
3878 2024 Acked Read 7 Done Sent Done false
3879 2025 Acked Flush 0 Done Sent Done false
3880 2026 Acked Read 9 Done Sent Done false
3881 2027 Acked Write 2 Done Sent Done false
3882 2028 Acked Write 4 Done Sent Done false
3883 2029 Acked Read 8 Done Sent Done false
3884 2030 Acked Read 7 Done Sent Done false
3885 2031 Acked Read 5 Done Sent Done false
3886 2032 Acked Flush 0 Done Sent Done false
3887 2033 Acked Write 1 Done Sent Done false
3888 2034 Acked Read 3 Done Sent Done false
3889 2035 Acked Read 3 Done Sent Done false
3890 2036 Acked Write 10 Done Sent Done false
3891 2037 Acked Read 9 Done Sent Done false
3892 2038 Acked Read 10 Done Sent Done false
3893 2039 Acked Write 9 Done Sent Done false
3894 2040 Acked Flush 0 Done Sent Done false
3895 2041 Acked Read 1 Done Sent Done false
3896 2042 Acked Read 5 Done Sent Done false
3897 2043 Acked Read 6 Done Sent Done false
3898 2044 Acked Write 1 Done Sent Done false
3899 2045 Acked Read 2 Done Sent Done false
3900 2046 Acked Write 1 Done Sent Done false
3901 2047 Acked Read 3 Done Sent Done false
3902 2048 Acked Flush 0 Done Sent Done false
3903 2049 Acked Write 9 Done Sent Done false
3904 2050 Acked Read 8 Done Sent Done false
3905 2051 Acked Flush 0 Done Sent Done false
3906 2052 Acked Read 6 Done Sent Done false
3907 2053 Acked Read 4 Done Sent Done false
3908 2054 Acked Read 9 Done Sent Done false
3909 2055 Acked Read 9 Done Sent Done false
3910 2056 Acked Read 7 Done Sent Done false
3911 2057 Acked Read 10 Done Sent Done false
3912 2058 Acked Read 7 Done Sent Done false
3913 2059 Acked Read 10 Done Sent Done false
3914 2060 Acked Flush 0 Done Sent Done false
3915 2061 Acked Read 6 Done Sent Done false
3916 2062 Acked Write 6 Done Sent Done false
3917 2063 Acked Write 7 Done Sent Done false
3918 2064 Acked Write 8 Done Sent Done false
3919 2065 Acked Flush 0 Done Sent Done false
3920 2066 Acked Write 4 Done Sent Done false
3921 2067 Acked Flush 0 Done Sent Done false
3922 2068 Acked Write 2 Done Sent Done false
3923 2069 Acked Write 7 Done Sent Done false
3924 2070 Acked Read 5 Done Sent Done false
3925 2071 Acked Write 7 Done Sent Done false
3926 2072 Acked Read 10 Done Sent Done false
3927 2073 Acked Read 2 Done Sent Done false
3928 2074 Acked Flush 0 Done Sent Done false
3929 2075 Acked Write 8 Done Sent Done false
3930 2076 Acked Write 5 Done Sent Done false
3931 2077 Acked Read 7 Done Sent Done false
3932 2078 Acked Write 8 Done Sent Done false
3933 2079 Acked Read 10 Done Sent Done false
3934 2080 Acked Read 6 Done Sent Done false
3935 2081 Acked Write 8 Done Sent Done false
3936 2082 Acked Read 9 Done Sent Done false
3937 2083 Acked Flush 0 Done Sent Done false
3938 2084 Acked Read 10 Done Sent Done false
3939 2085 Acked Read 9 Done Sent Done false
3940 2086 Acked Flush 0 Done Sent Done false
3941 2087 Acked Read 5 Done Sent Done false
3942 2088 Acked Read 2 Done Sent Done false
3943 2089 Acked Write 7 Done Sent Done false
3944 2090 Acked Flush 0 Done Sent Done false
3945 2091 Acked Write 9 Done Sent Done false
3946 2092 Acked Read 8 Done Sent Done false
3947 2093 Acked Flush 0 Done Sent Done false
3948 2094 Acked Write 5 Done Sent Done false
3949 2095 Acked Write 7 Done Sent Done false
3950 2096 Acked Read 4 Done Sent Done false
3951 2097 Acked Write 2 Done Sent Done false
3952 2098 Acked Flush 0 Done Sent Done false
3953 2099 Acked Flush 0 Done Sent Done false
3954 2100 Acked Write 3 Done Sent Done false
3955 2101 Acked Read 10 Done Sent Done false
3956 2102 Acked Write 6 Done Sent Done false
3957 2103 Acked Write 2 Done Sent Done false
3958 2104 Acked Read 3 Done Sent Done false
3959 2105 Acked Write 7 Done Sent Done false
3960 2106 Acked Write 9 Done Sent Done false
3961 2107 Acked Read 5 Done Sent Done false
3962 2108 Acked Write 1 Done Sent Done false
3963 2109 Acked Read 6 Done Sent Done false
3964 2110 Acked Read 5 Done Sent Done false
3965 2111 Acked Write 9 Done Sent Done false
3966 2112 Acked Write 8 Done Sent Done false
3967 2113 Acked Read 10 Done Sent Done false
3968 2114 Acked Read 1 Done Sent Done false
3969 2115 Acked Read 8 Done Sent Done false
3970 2116 Acked Read 8 Done Sent Done false
3971 2117 Acked Read 9 Done Sent Done false
3972 2118 Acked Write 4 Done Sent Done false
3973 2119 Acked Read 5 Done Sent Done false
3974 2120 Acked Write 1 Done Sent Done false
3975 2121 Acked Read 10 Done Sent Done false
3976 2122 Acked Read 2 Done Sent Done false
3977 2123 Acked Read 6 Done Sent Done false
3978 2124 Acked Read 5 Done Sent Done false
3979 2125 Acked Write 9 Done Sent Done false
3980 2126 Acked Write 5 Done Sent Done false
3981 2127 Acked Read 9 Done Sent Done false
3982 2128 Acked Read 7 Done Sent Done false
3983 2129 Acked Write 10 Done Sent Done false
3984 2130 Acked Flush 0 Done Sent Done false
3985 2131 Acked Write 3 Done Sent Done false
3986 2132 Acked Write 5 Done Sent Done false
3987 2133 Acked Write 5 Done Sent Done false
3988 2134 Acked Write 9 Done Sent Done false
3989 2135 Acked Write 1 Done Sent Done false
3990 2136 Acked Write 6 Done Sent Done false
3991 2137 Acked Write 1 Done Sent Done false
3992 2138 Acked Read 7 Done Sent Done false
3993 2139 Acked Write 4 Done Sent Done false
3994 2140 Acked Read 5 Done Sent Done false
3995 2141 Acked Flush 0 Done Sent Done false
3996 2142 Acked Flush 0 Done Sent Done false
3997 2143 Acked Read 10 Done Sent Done false
3998 2144 Acked Read 8 Done Sent Done false
3999 2145 Acked Read 3 Done Sent Done false
4000 2146 Acked Read 2 Done Sent Done false
4001 2147 Acked Read 6 Done Sent Done false
4002 2148 Acked Read 10 Done Sent Done false
4003 2149 Acked Flush 0 Done Sent Done false
4004 2150 Acked Read 2 Done Sent Done false
4005 2151 Acked Read 5 Done Sent Done false
4006 2152 Acked Write 5 Done Sent Done false
4007 2153 Acked Read 3 Done Sent Done false
4008 2154 Acked Read 10 Done Sent Done false
4009 2155 Acked Read 6 Done Sent Done false
4010 2156 Acked Read 3 Done Sent Done false
4011 2157 Acked Read 8 Done Sent Done false
4012 2158 Acked Read 3 Done Sent Done false
4013 2159 Acked Read 9 Done Sent Done false
4014 2160 Acked Read 8 Done Sent Done false
4015 2161 Acked Read 1 Done Sent Done false
4016 2162 Acked Write 7 Done Sent Done false
4017 2163 Acked Read 4 Done Sent Done false
4018 2164 Acked Write 5 Done Sent Done false
4019 2165 Acked Read 7 Done Sent Done false
4020 2166 Acked Write 7 Done Sent Done false
4021 2167 Acked Read 5 Done Sent Done false
4022 2168 Acked Write 4 Done Sent Done false
4023 2169 Acked Read 1 Done Sent Done false
4024 2170 Acked Read 2 Done Sent Done false
4025 2171 Acked Read 4 Done Sent Done false
4026 2172 Acked Write 6 Done Sent Done false
4027 2173 Acked Read 2 Done Sent Done false
4028 2174 Acked Read 2 Done Sent Done false
4029 2175 Acked Read 8 Done Sent Done false
4030 2176 Acked Write 5 Done Sent Done false
4031 2177 Acked Write 4 Done Sent Done false
4032 2178 Acked Read 10 Done Sent Done false
4033 2179 Acked Write 8 Done Sent Done false
4034 2180 Acked Write 4 Done Sent Done false
4035 2181 Acked Read 7 Done Sent Done false
4036 2182 Acked Read 1 Done Sent Done false
4037 2183 Acked Write 8 Done Sent Done false
4038 2184 Acked Read 1 Done Sent Done false
4039 2185 Acked Read 10 Done Sent Done false
4040 2186 Acked Read 10 Done Sent Done false
4041 2187 Acked Read 9 Done Sent Done false
4042 2188 Acked Write 3 Done Sent Done false
4043 2189 Acked Read 3 Done Sent Done false
4044 2190 Acked Read 6 Done Sent Done false
4045 2191 Acked Read 9 Done Sent Done false
4046 2192 Acked Read 3 Done Sent Done false
4047 2193 Acked Flush 0 Done Sent Done false
4048 2194 Acked Write 10 Done Sent Done false
4049 2195 Acked Read 2 Done Sent Done false
4050 2196 Acked Write 1 Done Sent Done false
4051 2197 Acked Write 1 Done Sent Done false
4052 2198 Acked Write 8 Done Sent Done false
4053 2199 Acked Flush 0 Done Sent Done false
4054 2200 Acked Read 5 Done Sent Done false
4055 2201 Acked Read 10 Done Sent Done false
4056 2202 Acked Read 9 Done Sent Done false
4057 2203 Acked Read 3 Done Sent Done false
4058 2204 Acked Read 8 Done Sent Done false
4059 2205 Acked Write 1 Done Sent Done false
4060 2206 Acked Write 8 Done Sent Done false
4061 2207 Acked Flush 0 Done Sent Done false
4062 2208 Acked Write 10 Done Sent Done false
4063 2209 Acked Flush 0 Done Sent Done false
4064 STATES DS:0 DS:1 DS:2 TOTAL
4065 Sent 0 303 0 303
4066 Done 303 0 303 606
4067 Skipped 0 0 0 0
4068 Error 0 0 0 0
4069 Last Flush: 2209 1906 2209
4070 Downstairs last five completed: 1906 1905 1904 1903 1902
4071 Upstairs last five completed: 2209 2208 2207 2206 2205
4072 CLIENT: Up:0 ds:303 act:2
40732024-11-01T16:51:19.130ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
40742024-11-01T16:51:19.130ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:32850) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
40752024-11-01T16:51:19.131ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
40762024-11-01T16:51:19.131ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
40772024-11-01T16:51:19.131ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
40782024-11-01T16:51:19.131ZINFOcrucible: send last flush ID to this DS: 1906 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
40792024-11-01T16:51:19.132ZINFOcrucible: Replied this last flush ID: 1906 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
40802024-11-01T16:51:19.132ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
40812024-11-01T16:51:19.132ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
40822024-11-01T16:51:19.132ZINFOcrucible: [1] client re-new 303 jobs since flush 1906 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
40832024-11-01T16:51:19.132ZINFOcrucible: [1] Replayed 303 jobs since flush: 1906 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
4084 ----------------------------------------------------------------
4085 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4086 Downstairs last five completed: 2209 2208 2207 2206 2205
4087 Upstairs last five completed: 2209 2208 2207 2206 2205
4088 CLIENT: Up:0 ds:0 act:3
4089 Replay: All jobs finished
4090 Main task has work to do, go find it
4091 got dsc Stop(0)
4092 stop 0
4093 Nov 01 16:51:21.431 INFO [8810] Got stop action so:false kr:false
4094 request completed, latency_us: 369, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 43d8a0cf-9476-4abb-86c1-dbf7bc0ee43c, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
40952024-11-01T16:51:21.431ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
40962024-11-01T16:51:21.431ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
40972024-11-01T16:51:21.431ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
40982024-11-01T16:51:21.431ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
4099 Nov 01 16:51:21.431 INFO request completed, latency_us: 107, response_code: 200, uri: /state/cid/0, method: GET, req_id: 83623353-99e3-44d3-a4db-d89991d972ed, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
41002024-11-01T16:51:21.431ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
4101 [8810] Exited with: ExitStatus(unix_wait_status(9))
4102 [8810][0] reports Exit
4103 Nov 01 16:51:25.434 INFO request completed, latency_us: 346, response_code: 200, uri: /state/cid/0, method: GET, req_id: 88d46a55-3a7c-40ed-8a5d-191cb41f87c7, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
4104 001/300 Write block 369 len 8192 data: 3 4
4105 002/300 Write block 3500 len 32768 data: 2 2 2 2 2 2 2 2
4106 003/300 Read block 2280 len 8192
4107 004/300 Flush
4108 005/300 Write block 2871 len 4096 data: 2
4109 006/300 Write block 2710 len 12288 data: 3 3 5
4110 007/300 Read block 1729 len 20480
4111 008/300 Write block 4469 len 12288 data: 2 2 2
4112 009/300 Write block 4422 len 24576 data: 3 3 3 3 3 3
4113 010/300 Write block 2969 len 28672 data: 3 2 2 2 3 3 3
4114 011/300 Read block 3843 len 28672
4115 012/300 Read block 1613 len 40960
4116 013/300 Write block 4209 len 28672 data: 3 3 2 2 2 2 3
4117 014/300 Read block 2973 len 28672
4118 015/300 Read block 4107 len 36864
4119 016/300 Read block 3770 len 12288
4120 017/300 Write block 2957 len 24576 data: 3 3 3 3 3 3
4121 018/300 Write block 3990 len 32768 data: 2 2 2 2 2 2 2 2
4122 019/300 Read block 4881 len 12288
4123 020/300 Write block 618 len 32768 data: 2 2 2 2 3 3 3 3
4124 021/300 Read block 2608 len 16384
4125 022/300 Write block 2859 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4126 023/300 Read block 2860 len 20480
4127 024/300 Read block 804 len 32768
4128 025/300 Flush
4129 026/300 Flush
4130 027/300 Write block 2219 len 16384 data: 2 2 2 2
4131 028/300 Read block 2971 len 36864
4132 029/300 Write block 4625 len 40960 data: 2 2 2 2 4 4 4 4 3 3
4133 030/300 Write block 3712 len 32768 data: 3 2 2 2 2 2 2 2
4134 031/300 Read block 754 len 20480
4135 032/300 Write block 1800 len 36864 data: 3 3 3 3 2 2 2 2 2
4136 033/300 Read block 2735 len 28672
4137 034/300 Read block 2195 len 4096
4138 035/300 Flush
4139 036/300 Read block 3811 len 8192
4140 037/300 Read block 613 len 40960
4141 038/300 Write block 3568 len 36864 data: 2 2 2 3 3 2 2 2 2
4142 039/300 Write block 502 len 36864 data: 2 2 2 2 3 3 4 3 4
4143 040/300 Read block 3282 len 16384
4144 041/300 Read block 4016 len 12288
4145 042/300 Write block 4146 len 36864 data: 2 2 2 2 2 2 2 2 2
4146 043/300 Read block 4900 len 20480
4147 044/300 Read block 166 len 40960
4148 045/300 Read block 2273 len 36864
4149 046/300 Read block 885 len 4096
4150 047/300 Read block 2493 len 20480
4151 048/300 Write block 3800 len 4096 data: 2
4152 049/300 Write block 2930 len 32768 data: 3 3 3 2 2 3 2 2
4153 050/300 Flush
4154 051/300 Write block 2170 len 4096 data: 2
4155 052/300 Read block 2079 len 24576
4156 053/300 Write block 168 len 40960 data: 3 3 3 3 3 3 3 3 3 3
4157 054/300 Read block 378 len 32768
4158 055/300 Flush
4159 056/300 Write block 179 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4160 057/300 Write block 3003 len 4096 data: 3
4161 058/300 Read block 3154 len 20480
4162 059/300 Write block 8 len 16384 data: 3 3 3 3
4163 060/300 Flush
4164 061/300 Read block 4323 len 32768
4165 062/300 Read block 3020 len 12288
4166 063/300 Write block 529 len 36864 data: 5 5 5 4 4 4 2 2 2
4167 064/300 Read block 4790 len 8192
4168 065/300 Read block 1812 len 16384
4169 066/300 Read block 991 len 40960
4170 067/300 Read block 4677 len 32768
4171 068/300 Read block 2226 len 8192
4172 069/300 Write block 3251 len 28672 data: 3 2 2 2 2 2 2
4173 070/300 Write block 3795 len 4096 data: 5
4174 071/300 Read block 1479 len 8192
4175 072/300 Read block 1998 len 20480
4176 073/300 Read block 2938 len 16384
4177 074/300 Write block 1942 len 32768 data: 3 3 3 3 3 2 2 2
4178 075/300 Read block 4518 len 24576
4179 076/300 Read block 626 len 20480
4180 077/300 Write block 3287 len 20480 data: 2 3 3 3 3
4181 078/300 Write block 89 len 32768 data: 2 2 2 2 2 2 2 3
4182 079/300 Write block 4651 len 20480 data: 2 2 2 2 2
4183 080/300 Write block 3992 len 4096 data: 3
4184 081/300 Read block 3997 len 32768
4185 082/300 Read block 4875 len 40960
4186 083/300 Write block 4409 len 12288 data: 2 2 2
4187 084/300 Read block 3750 len 12288
4188 085/300 Write block 1065 len 8192 data: 2 2
4189 086/300 Read block 1423 len 4096
4190 087/300 Read block 2723 len 20480
4191 088/300 Flush
4192 089/300 Read block 4463 len 4096
4193 090/300 Read block 3964 len 12288
4194 091/300 Write block 2718 len 12288 data: 3 3 3
4195 092/300 Write block 1265 len 32768 data: 4 4 3 3 3 2 2 2
4196 093/300 Write block 4438 len 28672 data: 2 2 2 2 2 3 3
4197 094/300 Write block 3527 len 12288 data: 2 2 2
4198 095/300 Read block 857 len 28672
4199 096/300 Read block 1498 len 20480
4200 097/300 Read block 4109 len 40960
4201 098/300 Write block 3925 len 4096 data: 2
4202 099/300 Read block 208 len 32768
4203 100/300 Read block 1842 len 16384
4204 101/300 Write block 2739 len 36864 data: 4 4 5 4 4 4 4 3 3
4205 102/300 Write block 4909 len 36864 data: 2 2 2 2 3 3 3 4 4
4206 103/300 Read block 1866 len 36864
4207 104/300 Write block 2291 len 12288 data: 3 4 5
4208 105/300 Write block 2682 len 16384 data: 3 3 2 2
4209 106/300 Read block 893 len 24576
4210 107/300 Write block 1575 len 16384 data: 2 2 2 2
4211 108/300 Read block 3954 len 24576
4212 109/300 Write block 2565 len 40960 data: 4 3 3 3 2 2 2 2 2 2
4213 110/300 Flush
4214 111/300 Read block 339 len 40960
4215 112/300 Read block 4368 len 28672
4216 113/300 Write block 4892 len 20480 data: 2 2 2 2 2
4217 114/300 Read block 4118 len 28672
4218 115/300 Write block 599 len 8192 data: 4 4
4219 116/300 Read block 288 len 32768
4220 117/300 Flush
4221 118/300 Write block 3290 len 36864 data: 4 4 3 2 2 2 2 2 2
4222 119/300 Write block 4626 len 40960 data: 3 3 3 5 5 5 5 4 4 3
4223 120/300 Read block 3138 len 20480
4224 121/300 Write block 2211 len 8192 data: 4 4
4225 122/300 Read block 4906 len 40960
4226 123/300 Read block 2012 len 24576
4227 124/300 Read block 4866 len 32768
4228 125/300 Read block 1944 len 20480
4229 126/300 Read block 673 len 16384
4230 127/300 Read block 1954 len 36864
4231 128/300 Read block 4785 len 28672
4232 129/300 Write block 4310 len 24576 data: 2 2 2 2 2 2
4233 130/300 Read block 3953 len 32768
4234 131/300 Read block 4937 len 24576
4235 132/300 Write block 1809 len 16384 data: 2 2 2 2
4236 133/300 Flush
4237 134/300 Flush
4238 135/300 Write block 2286 len 32768 data: 2 2 3 3 3 4 5 6
4239 136/300 Read block 4106 len 12288
4240 137/300 Write block 845 len 32768 data: 4 4 4 4 4 4 4 4
4241 138/300 Read block 4734 len 4096
4242 139/300 Flush
4243 140/300 Read block 3064 len 4096
4244 141/300 Read block 2879 len 36864
4245 142/300 Read block 3239 len 12288
4246 143/300 Read block 2220 len 20480
4247 144/300 Read block 3053 len 40960
4248 145/300 Flush
4249 146/300 Write block 2747 len 12288 data: 4 2 2
4250 147/300 Read block 4352 len 28672
4251 148/300 Read block 2525 len 16384
4252 149/300 Read block 4509 len 40960
4253 150/300 Write block 654 len 36864 data: 2 2 2 2 3 3 3 3 3
4254 151/300 Read block 254 len 4096
4255 152/300 Read block 4766 len 12288
4256 153/300 Flush
4257 154/300 Write block 86 len 32768 data: 2 2 2 3 3 3 3 3
4258 155/300 Read block 1964 len 36864
4259 156/300 Write block 4024 len 4096 data: 3
4260 157/300 Read block 891 len 16384
4261 158/300 Write block 773 len 28672 data: 3 3 5 5 5 5 4
4262 159/300 Flush
4263 160/300 Write block 3459 len 16384 data: 3 3 3 3
4264 161/300 Read block 4946 len 12288
4265 162/300 Read block 2923 len 4096
4266 163/300 Read block 4277 len 24576
4267 164/300 Read block 1424 len 36864
4268 165/300 Write block 3545 len 4096 data: 2
4269 166/300 Write block 1091 len 4096 data: 4
4270 167/300 Write block 3552 len 36864 data: 2 2 2 2 2 2 3 3 2
4271 168/300 Write block 90 len 40960 data: 4 4 4 4 3 3 4 4 4 4
4272 169/300 Write block 3133 len 36864 data: 3 4 4 4 3 3 3 3 2
4273 170/300 Read block 1374 len 16384
4274 171/300 Flush
4275 172/300 Read block 793 len 16384
4276 173/300 Flush
4277 174/300 Read block 3374 len 16384
4278 175/300 Read block 4950 len 40960
4279 176/300 Read block 4989 len 12288
4280 177/300 Write block 4403 len 20480 data: 2 2 2 2 2
4281 178/300 Read block 2145 len 12288
4282 179/300 Flush
4283 180/300 Read block 403 len 8192
4284 181/300 Write block 3213 len 16384 data: 2 2 2 2
4285 182/300 Read block 1129 len 4096
4286 183/300 Read block 133 len 8192
4287 184/300 Write block 1862 len 36864 data: 2 2 2 2 2 2 2 2 2
4288 185/300 Read block 3540 len 28672
4289 186/300 Write block 2615 len 28672 data: 3 3 3 2 2 2 2
4290 187/300 Read block 4293 len 32768
4291 188/300 Read block 2512 len 8192
4292 189/300 Read block 4365 len 20480
4293 190/300 Write block 2661 len 36864 data: 2 2 2 2 2 2 2 2 2
4294 191/300 Read block 4625 len 20480
4295 192/300 Read block 1615 len 16384
4296 193/300 Read block 1885 len 36864
4297 194/300 Read block 4354 len 32768
4298 195/300 Write block 2396 len 40960 data: 3 2 2 2 2 2 2 2 2 2
4299 196/300 Write block 792 len 20480 data: 2 3 3 4 4
4300 197/300 Read block 3990 len 16384
4301 198/300 Read block 2028 len 36864
4302 199/300 Read block 1275 len 32768
4303 200/300 Write block 3644 len 20480 data: 2 2 2 2 3
4304 201/300 Write block 4694 len 8192 data: 2 2
4305 202/300 Write block 4675 len 16384 data: 4 4 4 3
4306 203/300 Read block 250 len 28672
4307 204/300 Read block 1823 len 16384
4308 205/300 Read block 4324 len 20480
4309 206/300 Read block 3794 len 4096
4310 207/300 Read block 4242 len 12288
4311 208/300 Read block 1314 len 32768
4312 209/300 Write block 3206 len 28672 data: 4 4 3 2 2 2 2
4313 210/300 Read block 4013 len 32768
4314 211/300 Write block 3014 len 28672 data: 2 2 2 2 2 2 2
4315 212/300 Read block 1954 len 32768
4316 213/300 Write block 3465 len 40960 data: 4 4 3 3 2 2 2 2 2 2
4317 214/300 Read block 4783 len 16384
4318 215/300 Read block 2630 len 4096
4319 216/300 Read block 1663 len 12288
4320 217/300 Read block 951 len 12288
4321 218/300 Write block 416 len 12288 data: 2 2 2
4322 219/300 Write block 675 len 16384 data: 2 2 2 2
4323 220/300 Write block 4635 len 28672 data: 4 4 3 3 2 2 2
4324 221/300 Read block 3178 len 32768
4325 222/300 Write block 3187 len 16384 data: 3 3 2 2
4326 223/300 Write block 4110 len 40960 data: 3 3 2 2 2 2 3 3 4 3
4327 224/300 Write block 2657 len 40960 data: 2 2 2 2 3 3 3 3 3 3
4328 225/300 Write block 4578 len 32768 data: 2 2 2 2 2 2 2 2
4329 226/300 Write block 1956 len 32768 data: 2 2 2 2 2 3 3 3
4330 227/300 Write block 4724 len 28672 data: 2 2 2 2 2 2 2
4331 228/300 Read block 4492 len 28672
4332 229/300 Write block 3322 len 24576 data: 3 3 3 2 2 2
4333 230/300 Flush
4334 231/300 Read block 910 len 8192
4335 232/300 Write block 4771 len 24576 data: 2 2 2 2 3 3
4336 233/300 Read block 3437 len 28672
4337 234/300 Flush
4338 235/300 Read block 737 len 16384
4339 236/300 Write block 1760 len 20480 data: 2 2 2 2 2
4340 237/300 Write block 624 len 36864 data: 4 4 3 3 3 3 2 2 2
4341 238/300 Write block 1335 len 12288 data: 2 2 2
4342 239/300 Read block 1664 len 4096
4343 240/300 Read block 4041 len 32768
4344 241/300 Write block 2683 len 8192 data: 4 3
4345 242/300 Write block 1362 len 4096 data: 2
4346 243/300 Write block 3229 len 8192 data: 3 2
4347 244/300 Flush
4348 245/300 Read block 1894 len 16384
4349 246/300 Read block 1673 len 20480
4350 247/300 Read block 574 len 28672
4351 248/300 Read block 4626 len 16384
4352 249/300 Read block 4624 len 4096
4353 250/300 Read block 385 len 24576
4354 251/300 Read block 4249 len 32768
4355 252/300 Flush
4356 253/300 Write block 3747 len 40960 data: 3 3 2 2 2 2 2 2 2 2
4357 254/300 Write block 3460 len 28672 data: 4 4 4 3 3 5 5
4358 255/300 Read block 1212 len 12288
4359 256/300 Read block 3343 len 8192
4360 257/300 Read block 578 len 4096
4361 258/300 Read block 3361 len 40960
4362 259/300 Read block 2955 len 8192
4363 260/300 Read block 3213 len 36864
4364 261/300 Flush
4365 262/300 Write block 2722 len 16384 data: 4 4 4 4
4366 263/300 Read block 2175 len 36864
4367 264/300 Write block 122 len 40960 data: 4 4 4 4 4 4 2 2 2 2
4368 265/300 Read block 3058 len 40960
4369 266/300 Read block 4521 len 4096
4370 267/300 Read block 3863 len 8192
4371 268/300 Flush
4372 269/300 Flush
4373 270/300 Flush
4374 271/300 Read block 295 len 36864
4375 272/300 Read block 755 len 36864
4376 273/300 Write block 1974 len 8192 data: 4 4
4377 274/300 Write block 4947 len 4096 data: 4
4378 275/300 Read block 3839 len 16384
4379 276/300 Write block 3873 len 8192 data: 4 3
4380 277/300 Write block 3696 len 12288 data: 4 4 3
4381 278/300 Flush
4382 279/300 Read block 2415 len 8192
4383 280/300 Read block 731 len 24576
4384 281/300 Read block 2231 len 16384
4385 282/300 Flush
4386 283/300 Read block 3706 len 4096
4387 284/300 Flush
4388 285/300 Write block 3551 len 36864 data: 2 3 3 3 3 3 3 4 4
4389 286/300 Read block 445 len 16384
4390 287/300 Write block 3725 len 20480 data: 2 2 2 2 2
4391 288/300 Read block 1147 len 4096
4392 289/300 Read block 3912 len 40960
4393 290/300 Read block 4402 len 4096
4394 291/300 Read block 748 len 32768
4395 292/300 Read block 3419 len 16384
4396 293/300 Write block 2204 len 12288 data: 2 2 2
4397 294/300 Write block 4368 len 24576 data: 2 2 2 2 2 2
4398 295/300 Read block 2495 len 4096
4399 296/300 Read block 3644 len 40960
4400 297/300 Read block 1377 len 40960
4401 298/300 Flush
4402 299/300 Read block 4595 len 32768
4403 300/300 Flush
4404 301/300 Flush
4405 Main task has work to do, go find it
4406 got dsc Start(0)
4407 start 0
4408 Nov 01 16:51:25.725 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/0, method: POST, req_id: 391b5485-73ff-44f7-a40e-f8ce83591c2c, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
4409 Replay: started 0, returned:Ok(())
4410 ----------------------------------------------------------------
4411 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4412 States: Offline Active Active
4413 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4414 2210 Acked Write 2 Sent Done Done false
4415 2211 Acked Write 8 Sent Done Done false
4416 2212 Acked Read 2 Sent Done Done false
4417 2213 Acked Flush 0 Sent Done Done false
4418 2214 Acked Write 1 Sent Done Done false
4419 2215 Acked Write 3 Sent Done Done false
4420 2216 Acked Read 5 Sent Done Done false
4421 2217 Acked Write 3 Sent Done Done false
4422 2218 Acked Write 6 Sent Done Done false
4423 2219 Acked Write 7 Sent Done Done false
4424 2220 Acked Read 7 Sent Done Done false
4425 2221 Acked Read 10 Sent Done Done false
4426 2222 Acked Write 7 Sent Done Done false
4427 2223 Acked Read 7 Sent Done Done false
4428 2224 Acked Read 9 Sent Done Done false
4429 2225 Acked Read 3 Sent Done Done false
4430 2226 Acked Write 6 Sent Done Done false
4431 2227 Acked Write 8 Sent Done Done false
4432 2228 Acked Read 3 Sent Done Done false
4433 2229 Acked Write 8 Sent Done Done false
4434 2230 Acked Read 4 Sent Done Done false
4435 2231 Acked Write 10 Sent Done Done false
4436 2232 Acked Read 5 Sent Done Done false
4437 2233 Acked Read 8 Sent Done Done false
4438 2234 Acked Flush 0 Sent Done Done false
4439 2235 Acked Flush 0 Sent Done Done false
4440 2236 Acked Write 4 Sent Done Done false
4441 2237 Acked Read 9 Sent Done Done false
4442 2238 Acked Write 10 Sent Done Done false
4443 2239 Acked Write 8 Sent Done Done false
4444 2240 Acked Read 5 Sent Done Done false
4445 2241 Acked Write 9 Sent Done Done false
4446 2242 Acked Read 7 Sent Done Done false
4447 2243 Acked Read 1 Sent Done Done false
4448 2244 Acked Flush 0 Sent Done Done false
4449 2245 Acked Read 2 Sent Done Done false
4450 2246 Acked Read 10 Sent Done Done false
4451 2247 Acked Write 9 Sent Done Done false
4452 2248 Acked Write 9 Sent Done Done false
4453 2249 Acked Read 4 Sent Done Done false
4454 2250 Acked Read 3 Sent Done Done false
4455 2251 Acked Write 9 Sent Done Done false
4456 2252 Acked Read 5 Sent Done Done false
4457 2253 Acked Read 10 Sent Done Done false
4458 2254 Acked Read 9 Sent Done Done false
4459 2255 Acked Read 1 Sent Done Done false
4460 2256 Acked Read 5 Sent Done Done false
4461 2257 Acked Write 1 Sent Done Done false
4462 2258 Acked Write 8 Sent Done Done false
4463 2259 Acked Flush 0 Sent Done Done false
4464 2260 Acked Write 1 Sent Done Done false
4465 2261 Acked Read 6 Sent Done Done false
4466 2262 Acked Write 10 Sent Done Done false
4467 2263 Acked Read 8 Sent Done Done false
4468 2264 Acked Flush 0 Sent Done Done false
4469 2265 Acked Write 10 Sent Done Done false
4470 2266 Acked Write 1 Sent Done Done false
4471 2267 Acked Read 5 Sent Done Done false
4472 2268 Acked Write 4 Sent Done Done false
4473 2269 Acked Flush 0 Sent Done Done false
4474 2270 Acked Read 8 Sent Done Done false
4475 2271 Acked Read 3 Sent Done Done false
4476 2272 Acked Write 9 Sent Done Done false
4477 2273 Acked Read 2 Sent Done Done false
4478 2274 Acked Read 4 Sent Done Done false
4479 2275 Acked Read 10 Sent Done Done false
4480 2276 Acked Read 8 Sent Done Done false
4481 2277 Acked Read 2 Sent Done Done false
4482 2278 Acked Write 7 Sent Done Done false
4483 2279 Acked Write 1 Sent Done Done false
4484 2280 Acked Read 2 Sent Done Done false
4485 2281 Acked Read 5 Sent Done Done false
4486 2282 Acked Read 4 Sent Done Done false
4487 2283 Acked Write 8 Sent Done Done false
4488 2284 Acked Read 6 Sent Done Done false
4489 2285 Acked Read 5 Sent Done Done false
4490 2286 Acked Write 5 Sent Done Done false
4491 2287 Acked Write 8 Sent Done Done false
4492 2288 Acked Write 5 Sent Done Done false
4493 2289 Acked Write 1 Sent Done Done false
4494 2290 Acked Read 8 Sent Done Done false
4495 2291 Acked Read 10 Sent Done Done false
4496 2292 Acked Write 3 Sent Done Done false
4497 2293 Acked Read 3 Sent Done Done false
4498 2294 Acked Write 2 Sent Done Done false
4499 2295 Acked Read 1 Sent Done Done false
4500 2296 Acked Read 5 Sent Done Done false
4501 2297 Acked Flush 0 Sent Done Done false
4502 2298 Acked Read 1 Sent Done Done false
4503 2299 Acked Read 3 Sent Done Done false
4504 2300 Acked Write 3 Sent Done Done false
4505 2301 Acked Write 8 Sent Done Done false
4506 2302 Acked Write 7 Sent Done Done false
4507 2303 Acked Write 3 Sent Done Done false
4508 2304 Acked Read 7 Sent Done Done false
4509 2305 Acked Read 5 Sent Done Done false
4510 2306 Acked Read 10 Sent Done Done false
4511 2307 Acked Write 1 Sent Done Done false
4512 2308 Acked Read 8 Sent Done Done false
4513 2309 Acked Read 4 Sent Done Done false
4514 2310 Acked Write 9 Sent Done Done false
4515 2311 Acked Write 9 Sent Done Done false
4516 2312 Acked Read 9 Sent Done Done false
4517 2313 Acked Write 3 Sent Done Done false
4518 2314 Acked Write 4 Sent Done Done false
4519 2315 Acked Read 6 Sent Done Done false
4520 2316 Acked Write 4 Sent Done Done false
4521 2317 Acked Read 6 Sent Done Done false
4522 2318 Acked Write 10 Sent Done Done false
4523 2319 Acked Flush 0 Sent Done Done false
4524 2320 Acked Read 10 Sent Done Done false
4525 2321 Acked Read 7 Sent Done Done false
4526 2322 Acked Write 5 Sent Done Done false
4527 2323 Acked Read 7 Sent Done Done false
4528 2324 Acked Write 2 Sent Done Done false
4529 2325 Acked Read 8 Sent Done Done false
4530 2326 Acked Flush 0 Sent Done Done false
4531 2327 Acked Write 9 Sent Done Done false
4532 2328 Acked Write 10 Sent Done Done false
4533 2329 Acked Read 5 Sent Done Done false
4534 2330 Acked Write 2 Sent Done Done false
4535 2331 Acked Read 10 Sent Done Done false
4536 2332 Acked Read 6 Sent Done Done false
4537 2333 Acked Read 8 Sent Done Done false
4538 2334 Acked Read 5 Sent Done Done false
4539 2335 Acked Read 4 Sent Done Done false
4540 2336 Acked Read 9 Sent Done Done false
4541 2337 Acked Read 7 Sent Done Done false
4542 2338 Acked Write 6 Sent Done Done false
4543 2339 Acked Read 8 Sent Done Done false
4544 2340 Acked Read 6 Sent Done Done false
4545 2341 Acked Write 4 Sent Done Done false
4546 2342 Acked Flush 0 Sent Done Done false
4547 2343 Acked Flush 0 Sent Done Done false
4548 2344 Acked Write 8 Sent Done Done false
4549 2345 Acked Read 3 Sent Done Done false
4550 2346 Acked Write 8 Sent Done Done false
4551 2347 Acked Read 1 Sent Done Done false
4552 2348 Acked Flush 0 Sent Done Done false
4553 2349 Acked Read 1 Sent Done Done false
4554 2350 Acked Read 9 Sent Done Done false
4555 2351 Acked Read 3 Sent Done Done false
4556 2352 Acked Read 5 Sent Done Done false
4557 2353 Acked Read 10 Sent Done Done false
4558 2354 Acked Flush 0 Sent Done Done false
4559 2355 Acked Write 3 Sent Done Done false
4560 2356 Acked Read 7 Sent Done Done false
4561 2357 Acked Read 4 Sent Done Done false
4562 2358 Acked Read 10 Sent Done Done false
4563 2359 Acked Write 9 Sent Done Done false
4564 2360 Acked Read 1 Sent Done Done false
4565 2361 Acked Read 3 Sent Done Done false
4566 2362 Acked Flush 0 Sent Done Done false
4567 2363 Acked Write 8 Sent Done Done false
4568 2364 Acked Read 9 Sent Done Done false
4569 2365 Acked Write 1 Sent Done Done false
4570 2366 Acked Read 4 Sent Done Done false
4571 2367 Acked Write 7 Sent Done Done false
4572 2368 Acked Flush 0 Sent Done Done false
4573 2369 Acked Write 4 Sent Done Done false
4574 2370 Acked Read 3 Sent Done Done false
4575 2371 Acked Read 1 Sent Done Done false
4576 2372 Acked Read 6 Sent Done Done false
4577 2373 Acked Read 9 Sent Done Done false
4578 2374 Acked Write 1 Sent Done Done false
4579 2375 Acked Write 1 Sent Done Done false
4580 2376 Acked Write 9 Sent Done Done false
4581 2377 Acked Write 10 Sent Done Done false
4582 2378 Acked Write 9 Sent Done Done false
4583 2379 Acked Read 4 Sent Done Done false
4584 2380 Acked Flush 0 Sent Done Done false
4585 2381 Acked Read 4 Sent Done Done false
4586 2382 Acked Flush 0 Sent Done Done false
4587 2383 Acked Read 4 Sent Done Done false
4588 2384 Acked Read 10 Sent Done Done false
4589 2385 Acked Read 3 Sent Done Done false
4590 2386 Acked Write 5 Sent Done Done false
4591 2387 Acked Read 3 Sent Done Done false
4592 2388 Acked Flush 0 Sent Done Done false
4593 2389 Acked Read 2 Sent Done Done false
4594 2390 Acked Write 4 Sent Done Done false
4595 2391 Acked Read 1 Sent Done Done false
4596 2392 Acked Read 2 Sent Done Done false
4597 2393 Acked Write 9 Sent Done Done false
4598 2394 Acked Read 7 Sent Done Done false
4599 2395 Acked Write 7 Sent Done Done false
4600 2396 Acked Read 8 Sent Done Done false
4601 2397 Acked Read 2 Sent Done Done false
4602 2398 Acked Read 5 Sent Done Done false
4603 2399 Acked Write 9 Sent Done Done false
4604 2400 Acked Read 5 Sent Done Done false
4605 2401 Acked Read 4 Sent Done Done false
4606 2402 Acked Read 9 Sent Done Done false
4607 2403 Acked Read 8 Sent Done Done false
4608 2404 Acked Write 10 Sent Done Done false
4609 2405 Acked Write 5 Sent Done Done false
4610 2406 Acked Read 4 Sent Done Done false
4611 2407 Acked Read 9 Sent Done Done false
4612 2408 Acked Read 8 Sent Done Done false
4613 2409 Acked Write 5 Sent Done Done false
4614 2410 Acked Write 2 Sent Done Done false
4615 2411 Acked Write 4 Sent Done Done false
4616 2412 Acked Read 7 Sent Done Done false
4617 2413 Acked Read 4 Sent Done Done false
4618 2414 Acked Read 5 Sent Done Done false
4619 2415 Acked Read 1 Sent Done Done false
4620 2416 Acked Read 3 Sent Done Done false
4621 2417 Acked Read 8 Sent Done Done false
4622 2418 Acked Write 7 Sent Done Done false
4623 2419 Acked Read 8 Sent Done Done false
4624 2420 Acked Write 7 Sent Done Done false
4625 2421 Acked Read 8 Sent Done Done false
4626 2422 Acked Write 10 Sent Done Done false
4627 2423 Acked Read 4 Sent Done Done false
4628 2424 Acked Read 1 Sent Done Done false
4629 2425 Acked Read 3 Sent Done Done false
4630 2426 Acked Read 3 Sent Done Done false
4631 2427 Acked Write 3 Sent Done Done false
4632 2428 Acked Write 4 Sent Done Done false
4633 2429 Acked Write 7 Sent Done Done false
4634 2430 Acked Read 8 Sent Done Done false
4635 2431 Acked Write 4 Sent Done Done false
4636 2432 Acked Write 10 Sent Done Done false
4637 2433 Acked Write 10 Sent Done Done false
4638 2434 Acked Write 8 Sent Done Done false
4639 2435 Acked Write 8 Sent Done Done false
4640 2436 Acked Write 7 Sent Done Done false
4641 2437 Acked Read 7 Sent Done Done false
4642 2438 Acked Write 6 Sent Done Done false
4643 2439 Acked Flush 0 Sent Done Done false
4644 2440 Acked Read 2 Sent Done Done false
4645 2441 Acked Write 6 Sent Done Done false
4646 2442 Acked Read 7 Sent Done Done false
4647 2443 Acked Flush 0 Sent Done Done false
4648 2444 Acked Read 4 Sent Done Done false
4649 2445 Acked Write 5 Sent Done Done false
4650 2446 Acked Write 9 Sent Done Done false
4651 2447 Acked Write 3 Sent Done Done false
4652 2448 Acked Read 1 Sent Done Done false
4653 2449 Acked Read 8 Sent Done Done false
4654 2450 Acked Write 2 Sent Done Done false
4655 2451 Acked Write 1 Sent Done Done false
4656 2452 Acked Write 2 Sent Done Done false
4657 2453 Acked Flush 0 Sent Done Done false
4658 2454 Acked Read 4 Sent Done Done false
4659 2455 Acked Read 5 Sent Done Done false
4660 2456 Acked Read 7 Sent Done Done false
4661 2457 Acked Read 4 Sent Done Done false
4662 2458 Acked Read 1 Sent Done Done false
4663 2459 Acked Read 6 Sent Done Done false
4664 2460 Acked Read 8 Sent Done Done false
4665 2461 Acked Flush 0 Sent Done Done false
4666 2462 Acked Write 10 Sent Done Done false
4667 2463 Acked Write 7 Sent Done Done false
4668 2464 Acked Read 3 Sent Done Done false
4669 2465 Acked Read 2 Sent Done Done false
4670 2466 Acked Read 1 Sent Done Done false
4671 2467 Acked Read 10 Sent Done Done false
4672 2468 Acked Read 2 Sent Done Done false
4673 2469 Acked Read 9 Sent Done Done false
4674 2470 Acked Flush 0 Sent Done Done false
4675 2471 Acked Write 4 Sent Done Done false
4676 2472 Acked Read 9 Sent Done Done false
4677 2473 Acked Write 10 Sent Done Done false
4678 2474 Acked Read 10 Sent Done Done false
4679 2475 Acked Read 1 Sent Done Done false
4680 2476 Acked Read 2 Sent Done Done false
4681 2477 Acked Flush 0 Sent Done Done false
4682 2478 Acked Flush 0 Sent Done Done false
4683 2479 Acked Flush 0 Sent Done Done false
4684 2480 Acked Read 9 Sent Done Done false
4685 2481 Acked Read 9 Sent Done Done false
4686 2482 Acked Write 2 Sent Done Done false
4687 2483 Acked Write 1 Sent Done Done false
4688 2484 Acked Read 4 Sent Done Done false
4689 2485 Acked Write 2 Sent Done Done false
4690 2486 Acked Write 3 Sent Done Done false
4691 2487 Acked Flush 0 Sent Done Done false
4692 2488 Acked Read 2 Sent Done Done false
4693 2489 Acked Read 6 Sent Done Done false
4694 2490 Acked Read 4 Sent Done Done false
4695 2491 Acked Flush 0 Sent Done Done false
4696 2492 Acked Read 1 Sent Done Done false
4697 2493 Acked Flush 0 Sent Done Done false
4698 2494 Acked Write 9 Sent Done Done false
4699 2495 Acked Read 4 Sent Done Done false
4700 2496 Acked Write 5 Sent Done Done false
4701 2497 Acked Read 1 Sent Done Done false
4702 2498 Acked Read 10 Sent Done Done false
4703 2499 Acked Read 1 Sent Done Done false
4704 2500 Acked Read 8 Sent Done Done false
4705 2501 Acked Read 4 Sent Done Done false
4706 2502 Acked Write 3 Sent Done Done false
4707 2503 Acked Write 6 Sent Done Done false
4708 2504 Acked Read 1 Sent Done Done false
4709 2505 Acked Read 10 Sent Done Done false
4710 2506 Acked Read 10 Sent Done Done false
4711 2507 Acked Flush 0 Sent Done Done false
4712 2508 Acked Read 8 Sent Done Done false
4713 2509 Acked Flush 0 Sent Done Done false
4714 2510 Acked Flush 0 Sent Done Done false
4715 STATES DS:0 DS:1 DS:2 TOTAL
4716 Sent 301 0 0 301
4717 Done 0 301 301 602
4718 Skipped 0 0 0 0
4719 Error 0 0 0 0
4720 Last Flush: 2209 2510 2510
4721 Downstairs last five completed: 2209 2208 2207 2206 2205
4722 Upstairs last five completed: 2510 2509 2508 2507 2506
4723 CLIENT: Up:0 ds:301 act:2
4724 [8810] got start action
4725 [8810] I am going to restart
4726 Starting downstairs at port 8810
4727 Make output file at "/tmp/dsc/downstairs-8810.txt"
4728 [8810][0] reports Starting
4729 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1203)
4730 [8810][0] reports Running
4731 ----------------------------------------------------------------
4732 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4733 States: Offline Active Active
4734 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4735 2210 Acked Write 2 Sent Done Done false
4736 2211 Acked Write 8 Sent Done Done false
4737 2212 Acked Read 2 Sent Done Done false
4738 2213 Acked Flush 0 Sent Done Done false
4739 2214 Acked Write 1 Sent Done Done false
4740 2215 Acked Write 3 Sent Done Done false
4741 2216 Acked Read 5 Sent Done Done false
4742 2217 Acked Write 3 Sent Done Done false
4743 2218 Acked Write 6 Sent Done Done false
4744 2219 Acked Write 7 Sent Done Done false
4745 2220 Acked Read 7 Sent Done Done false
4746 2221 Acked Read 10 Sent Done Done false
4747 2222 Acked Write 7 Sent Done Done false
4748 2223 Acked Read 7 Sent Done Done false
4749 2224 Acked Read 9 Sent Done Done false
4750 2225 Acked Read 3 Sent Done Done false
4751 2226 Acked Write 6 Sent Done Done false
4752 2227 Acked Write 8 Sent Done Done false
4753 2228 Acked Read 3 Sent Done Done false
4754 2229 Acked Write 8 Sent Done Done false
4755 2230 Acked Read 4 Sent Done Done false
4756 2231 Acked Write 10 Sent Done Done false
4757 2232 Acked Read 5 Sent Done Done false
4758 2233 Acked Read 8 Sent Done Done false
4759 2234 Acked Flush 0 Sent Done Done false
4760 2235 Acked Flush 0 Sent Done Done false
4761 2236 Acked Write 4 Sent Done Done false
4762 2237 Acked Read 9 Sent Done Done false
4763 2238 Acked Write 10 Sent Done Done false
4764 2239 Acked Write 8 Sent Done Done false
4765 2240 Acked Read 5 Sent Done Done false
4766 2241 Acked Write 9 Sent Done Done false
4767 2242 Acked Read 7 Sent Done Done false
4768 2243 Acked Read 1 Sent Done Done false
4769 2244 Acked Flush 0 Sent Done Done false
4770 2245 Acked Read 2 Sent Done Done false
4771 2246 Acked Read 10 Sent Done Done false
4772 2247 Acked Write 9 Sent Done Done false
4773 2248 Acked Write 9 Sent Done Done false
4774 2249 Acked Read 4 Sent Done Done false
4775 2250 Acked Read 3 Sent Done Done false
4776 2251 Acked Write 9 Sent Done Done false
4777 2252 Acked Read 5 Sent Done Done false
4778 2253 Acked Read 10 Sent Done Done false
4779 2254 Acked Read 9 Sent Done Done false
4780 2255 Acked Read 1 Sent Done Done false
4781 2256 Acked Read 5 Sent Done Done false
4782 2257 Acked Write 1 Sent Done Done false
4783 2258 Acked Write 8 Sent Done Done false
4784 2259 Acked Flush 0 Sent Done Done false
4785 2260 Acked Write 1 Sent Done Done false
4786 2261 Acked Read 6 Sent Done Done false
4787 2262 Acked Write 10 Sent Done Done false
4788 2263 Acked Read 8 Sent Done Done false
4789 2264 Acked Flush 0 Sent Done Done false
4790 2265 Acked Write 10 Sent Done Done false
4791 2266 Acked Write 1 Sent Done Done false
4792 2267 Acked Read 5 Sent Done Done false
4793 2268 Acked Write 4 Sent Done Done false
4794 2269 Acked Flush 0 Sent Done Done false
4795 2270 Acked Read 8 Sent Done Done false
4796 2271 Acked Read 3 Sent Done Done false
4797 2272 Acked Write 9 Sent Done Done false
4798 2273 Acked Read 2 Sent Done Done false
4799 2274 Acked Read 4 Sent Done Done false
4800 2275 Acked Read 10 Sent Done Done false
4801 2276 Acked Read 8 Sent Done Done false
4802 2277 Acked Read 2 Sent Done Done false
4803 2278 Acked Write 7 Sent Done Done false
4804 2279 Acked Write 1 Sent Done Done false
4805 2280 Acked Read 2 Sent Done Done false
4806 2281 Acked Read 5 Sent Done Done false
4807 2282 Acked Read 4 Sent Done Done false
4808 2283 Acked Write 8 Sent Done Done false
4809 2284 Acked Read 6 Sent Done Done false
4810 2285 Acked Read 5 Sent Done Done false
4811 2286 Acked Write 5 Sent Done Done false
4812 2287 Acked Write 8 Sent Done Done false
4813 2288 Acked Write 5 Sent Done Done false
4814 2289 Acked Write 1 Sent Done Done false
4815 2290 Acked Read 8 Sent Done Done false
4816 2291 Acked Read 10 Sent Done Done false
4817 2292 Acked Write 3 Sent Done Done false
4818 2293 Acked Read 3 Sent Done Done false
4819 2294 Acked Write 2 Sent Done Done false
4820 2295 Acked Read 1 Sent Done Done false
4821 2296 Acked Read 5 Sent Done Done false
4822 2297 Acked Flush 0 Sent Done Done false
4823 2298 Acked Read 1 Sent Done Done false
4824 2299 Acked Read 3 Sent Done Done false
4825 2300 Acked Write 3 Sent Done Done false
4826 2301 Acked Write 8 Sent Done Done false
4827 2302 Acked Write 7 Sent Done Done false
4828 2303 Acked Write 3 Sent Done Done false
4829 2304 Acked Read 7 Sent Done Done false
4830 2305 Acked Read 5 Sent Done Done false
4831 2306 Acked Read 10 Sent Done Done false
4832 2307 Acked Write 1 Sent Done Done false
4833 2308 Acked Read 8 Sent Done Done false
4834 2309 Acked Read 4 Sent Done Done false
4835 2310 Acked Write 9 Sent Done Done false
4836 2311 Acked Write 9 Sent Done Done false
4837 2312 Acked Read 9 Sent Done Done false
4838 2313 Acked Write 3 Sent Done Done false
4839 2314 Acked Write 4 Sent Done Done false
4840 2315 Acked Read 6 Sent Done Done false
4841 2316 Acked Write 4 Sent Done Done false
4842 2317 Acked Read 6 Sent Done Done false
4843 2318 Acked Write 10 Sent Done Done false
4844 2319 Acked Flush 0 Sent Done Done false
4845 2320 Acked Read 10 Sent Done Done false
4846 2321 Acked Read 7 Sent Done Done false
4847 2322 Acked Write 5 Sent Done Done false
4848 2323 Acked Read 7 Sent Done Done false
4849 2324 Acked Write 2 Sent Done Done false
4850 2325 Acked Read 8 Sent Done Done false
4851 2326 Acked Flush 0 Sent Done Done false
4852 2327 Acked Write 9 Sent Done Done false
4853 2328 Acked Write 10 Sent Done Done false
4854 2329 Acked Read 5 Sent Done Done false
4855 2330 Acked Write 2 Sent Done Done false
4856 2331 Acked Read 10 Sent Done Done false
4857 2332 Acked Read 6 Sent Done Done false
4858 2333 Acked Read 8 Sent Done Done false
4859 2334 Acked Read 5 Sent Done Done false
4860 2335 Acked Read 4 Sent Done Done false
4861 2336 Acked Read 9 Sent Done Done false
4862 2337 Acked Read 7 Sent Done Done false
4863 2338 Acked Write 6 Sent Done Done false
4864 2339 Acked Read 8 Sent Done Done false
4865 2340 Acked Read 6 Sent Done Done false
4866 2341 Acked Write 4 Sent Done Done false
4867 2342 Acked Flush 0 Sent Done Done false
4868 2343 Acked Flush 0 Sent Done Done false
4869 2344 Acked Write 8 Sent Done Done false
4870 2345 Acked Read 3 Sent Done Done false
4871 2346 Acked Write 8 Sent Done Done false
4872 2347 Acked Read 1 Sent Done Done false
4873 2348 Acked Flush 0 Sent Done Done false
4874 2349 Acked Read 1 Sent Done Done false
4875 2350 Acked Read 9 Sent Done Done false
4876 2351 Acked Read 3 Sent Done Done false
4877 2352 Acked Read 5 Sent Done Done false
4878 2353 Acked Read 10 Sent Done Done false
4879 2354 Acked Flush 0 Sent Done Done false
4880 2355 Acked Write 3 Sent Done Done false
4881 2356 Acked Read 7 Sent Done Done false
4882 2357 Acked Read 4 Sent Done Done false
4883 2358 Acked Read 10 Sent Done Done false
4884 2359 Acked Write 9 Sent Done Done false
4885 2360 Acked Read 1 Sent Done Done false
4886 2361 Acked Read 3 Sent Done Done false
4887 2362 Acked Flush 0 Sent Done Done false
4888 2363 Acked Write 8 Sent Done Done false
4889 2364 Acked Read 9 Sent Done Done false
4890 2365 Acked Write 1 Sent Done Done false
4891 2366 Acked Read 4 Sent Done Done false
4892 2367 Acked Write 7 Sent Done Done false
4893 2368 Acked Flush 0 Sent Done Done false
4894 2369 Acked Write 4 Sent Done Done false
4895 2370 Acked Read 3 Sent Done Done false
4896 2371 Acked Read 1 Sent Done Done false
4897 2372 Acked Read 6 Sent Done Done false
4898 2373 Acked Read 9 Sent Done Done false
4899 2374 Acked Write 1 Sent Done Done false
4900 2375 Acked Write 1 Sent Done Done false
4901 2376 Acked Write 9 Sent Done Done false
4902 2377 Acked Write 10 Sent Done Done false
4903 2378 Acked Write 9 Sent Done Done false
4904 2379 Acked Read 4 Sent Done Done false
4905 2380 Acked Flush 0 Sent Done Done false
4906 2381 Acked Read 4 Sent Done Done false
4907 2382 Acked Flush 0 Sent Done Done false
4908 2383 Acked Read 4 Sent Done Done false
4909 2384 Acked Read 10 Sent Done Done false
4910 2385 Acked Read 3 Sent Done Done false
4911 2386 Acked Write 5 Sent Done Done false
4912 2387 Acked Read 3 Sent Done Done false
4913 2388 Acked Flush 0 Sent Done Done false
4914 2389 Acked Read 2 Sent Done Done false
4915 2390 Acked Write 4 Sent Done Done false
4916 2391 Acked Read 1 Sent Done Done false
4917 2392 Acked Read 2 Sent Done Done false
4918 2393 Acked Write 9 Sent Done Done false
4919 2394 Acked Read 7 Sent Done Done false
4920 2395 Acked Write 7 Sent Done Done false
4921 2396 Acked Read 8 Sent Done Done false
4922 2397 Acked Read 2 Sent Done Done false
4923 2398 Acked Read 5 Sent Done Done false
4924 2399 Acked Write 9 Sent Done Done false
4925 2400 Acked Read 5 Sent Done Done false
4926 2401 Acked Read 4 Sent Done Done false
4927 2402 Acked Read 9 Sent Done Done false
4928 2403 Acked Read 8 Sent Done Done false
4929 2404 Acked Write 10 Sent Done Done false
4930 2405 Acked Write 5 Sent Done Done false
4931 2406 Acked Read 4 Sent Done Done false
4932 2407 Acked Read 9 Sent Done Done false
4933 2408 Acked Read 8 Sent Done Done false
4934 2409 Acked Write 5 Sent Done Done false
4935 2410 Acked Write 2 Sent Done Done false
4936 2411 Acked Write 4 Sent Done Done false
4937 2412 Acked Read 7 Sent Done Done false
4938 2413 Acked Read 4 Sent Done Done false
4939 2414 Acked Read 5 Sent Done Done false
4940 2415 Acked Read 1 Sent Done Done false
4941 2416 Acked Read 3 Sent Done Done false
4942 2417 Acked Read 8 Sent Done Done false
4943 2418 Acked Write 7 Sent Done Done false
4944 2419 Acked Read 8 Sent Done Done false
4945 2420 Acked Write 7 Sent Done Done false
4946 2421 Acked Read 8 Sent Done Done false
4947 2422 Acked Write 10 Sent Done Done false
4948 2423 Acked Read 4 Sent Done Done false
4949 2424 Acked Read 1 Sent Done Done false
4950 2425 Acked Read 3 Sent Done Done false
4951 2426 Acked Read 3 Sent Done Done false
4952 2427 Acked Write 3 Sent Done Done false
4953 2428 Acked Write 4 Sent Done Done false
4954 2429 Acked Write 7 Sent Done Done false
4955 2430 Acked Read 8 Sent Done Done false
4956 2431 Acked Write 4 Sent Done Done false
4957 2432 Acked Write 10 Sent Done Done false
4958 2433 Acked Write 10 Sent Done Done false
4959 2434 Acked Write 8 Sent Done Done false
4960 2435 Acked Write 8 Sent Done Done false
4961 2436 Acked Write 7 Sent Done Done false
4962 2437 Acked Read 7 Sent Done Done false
4963 2438 Acked Write 6 Sent Done Done false
4964 2439 Acked Flush 0 Sent Done Done false
4965 2440 Acked Read 2 Sent Done Done false
4966 2441 Acked Write 6 Sent Done Done false
4967 2442 Acked Read 7 Sent Done Done false
4968 2443 Acked Flush 0 Sent Done Done false
4969 2444 Acked Read 4 Sent Done Done false
4970 2445 Acked Write 5 Sent Done Done false
4971 2446 Acked Write 9 Sent Done Done false
4972 2447 Acked Write 3 Sent Done Done false
4973 2448 Acked Read 1 Sent Done Done false
4974 2449 Acked Read 8 Sent Done Done false
4975 2450 Acked Write 2 Sent Done Done false
4976 2451 Acked Write 1 Sent Done Done false
4977 2452 Acked Write 2 Sent Done Done false
4978 2453 Acked Flush 0 Sent Done Done false
4979 2454 Acked Read 4 Sent Done Done false
4980 2455 Acked Read 5 Sent Done Done false
4981 2456 Acked Read 7 Sent Done Done false
4982 2457 Acked Read 4 Sent Done Done false
4983 2458 Acked Read 1 Sent Done Done false
4984 2459 Acked Read 6 Sent Done Done false
4985 2460 Acked Read 8 Sent Done Done false
4986 2461 Acked Flush 0 Sent Done Done false
4987 2462 Acked Write 10 Sent Done Done false
4988 2463 Acked Write 7 Sent Done Done false
4989 2464 Acked Read 3 Sent Done Done false
4990 2465 Acked Read 2 Sent Done Done false
4991 2466 Acked Read 1 Sent Done Done false
4992 2467 Acked Read 10 Sent Done Done false
4993 2468 Acked Read 2 Sent Done Done false
4994 2469 Acked Read 9 Sent Done Done false
4995 2470 Acked Flush 0 Sent Done Done false
4996 2471 Acked Write 4 Sent Done Done false
4997 2472 Acked Read 9 Sent Done Done false
4998 2473 Acked Write 10 Sent Done Done false
4999 2474 Acked Read 10 Sent Done Done false
5000 2475 Acked Read 1 Sent Done Done false
5001 2476 Acked Read 2 Sent Done Done false
5002 2477 Acked Flush 0 Sent Done Done false
5003 2478 Acked Flush 0 Sent Done Done false
5004 2479 Acked Flush 0 Sent Done Done false
5005 2480 Acked Read 9 Sent Done Done false
5006 2481 Acked Read 9 Sent Done Done false
5007 2482 Acked Write 2 Sent Done Done false
5008 2483 Acked Write 1 Sent Done Done false
5009 2484 Acked Read 4 Sent Done Done false
5010 2485 Acked Write 2 Sent Done Done false
5011 2486 Acked Write 3 Sent Done Done false
5012 2487 Acked Flush 0 Sent Done Done false
5013 2488 Acked Read 2 Sent Done Done false
5014 2489 Acked Read 6 Sent Done Done false
5015 2490 Acked Read 4 Sent Done Done false
5016 2491 Acked Flush 0 Sent Done Done false
5017 2492 Acked Read 1 Sent Done Done false
5018 2493 Acked Flush 0 Sent Done Done false
5019 2494 Acked Write 9 Sent Done Done false
5020 2495 Acked Read 4 Sent Done Done false
5021 2496 Acked Write 5 Sent Done Done false
5022 2497 Acked Read 1 Sent Done Done false
5023 2498 Acked Read 10 Sent Done Done false
5024 2499 Acked Read 1 Sent Done Done false
5025 2500 Acked Read 8 Sent Done Done false
5026 2501 Acked Read 4 Sent Done Done false
5027 2502 Acked Write 3 Sent Done Done false
5028 2503 Acked Write 6 Sent Done Done false
5029 2504 Acked Read 1 Sent Done Done false
5030 2505 Acked Read 10 Sent Done Done false
5031 2506 Acked Read 10 Sent Done Done false
5032 2507 Acked Flush 0 Sent Done Done false
5033 2508 Acked Read 8 Sent Done Done false
5034 2509 Acked Flush 0 Sent Done Done false
5035 2510 Acked Flush 0 Sent Done Done false
5036 STATES DS:0 DS:1 DS:2 TOTAL
5037 Sent 301 0 0 301
5038 Done 0 301 301 602
5039 Skipped 0 0 0 0
5040 Error 0 0 0 0
5041 Last Flush: 2209 2510 2510
5042 Downstairs last five completed: 2209 2208 2207 2206 2205
5043 Upstairs last five completed: 2510 2509 2508 2507 2506
5044 CLIENT: Up:0 ds:301 act:2
50452024-11-01T16:51:31.432ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
50462024-11-01T16:51:31.432ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58142) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
50472024-11-01T16:51:31.433ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
50482024-11-01T16:51:31.434ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
50492024-11-01T16:51:31.434ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
50502024-11-01T16:51:31.434ZINFOcrucible: send last flush ID to this DS: 2209 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
50512024-11-01T16:51:31.434ZINFOcrucible: Replied this last flush ID: 2209 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
50522024-11-01T16:51:31.434ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
50532024-11-01T16:51:31.434ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
50542024-11-01T16:51:31.434ZINFOcrucible: [0] client re-new 301 jobs since flush 2209 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
50552024-11-01T16:51:31.434ZINFOcrucible: [0] Replayed 301 jobs since flush: 2209 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
5056 ----------------------------------------------------------------
5057 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5058 Downstairs last five completed: 2510 2509 2508 2507 2506
5059 Upstairs last five completed: 2510 2509 2508 2507 2506
5060 CLIENT: Up:0 ds:0 act:3
5061 Replay: All jobs finished
5062 Main task has work to do, go find it
5063 got dsc Stop(1)
5064 stop 1
5065 Nov 01 16:51:33.730 INFO request completed, latency_us: 357, [8820] Got stop action so:false kr:false
5066 response_code: 204, uri: /stop/cid/1, method: POST, req_id: c6812d70-ab1a-4e1a-9a06-a278477c9383, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
5067 Nov 01 16:51:33.731 INFO request completed, latency_us: 146, response_code: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:51:33.731599759Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
5068 200, uri: /state/cid/1, method: GET, req_id: 3d7c1395-286e-4228-90b7-cfbf2e478cec, remote_addr: 127.0.{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:51:33.731681586Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
5069 0{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:51:33.731710241Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
5070 .1:58552, local_addr: 127.0.0.{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:51:33.731737137Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"client":"1","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
5071 1:9998
50722024-11-01T16:51:33.731ZINFOcrucible: client task is exiting = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
5073 [8820] Exited with: ExitStatus(unix_wait_status(9))
5074 [8820][1] reports Exit
5075 Nov 01 16:51:37.733 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/1, method: GET, req_id: 09fe51e5-c1df-4a33-8c4f-5daadc2012ca, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
5076 001/300 Read block 2674 len 24576
5077 002/300 Read block 4568 len 16384
5078 003/300 Read block 363 len 16384
5079 004/300 Read block 2178 len 20480
5080 005/300 Flush
5081 006/300 Write block 3528 len 16384 data: 3 3 2 2
5082 007/300 Write block 3370 len 16384 data: 3 3 3 3
5083 008/300 Read block 113 len 16384
5084 009/300 Write block 2842 len 8192 data: 2 2
5085 010/300 Write block 4822 len 24576 data: 2 2 2 2 2 2
5086 011/300 Flush
5087 012/300 Write block 2021 len 8192 data: 2 2
5088 013/300 Flush
5089 014/300 Write block 1080 len 16384 data: 5 4 4 3
5090 015/300 Flush
5091 016/300 Read block 582 len 12288
5092 017/300 Write block 4506 len 8192 data: 3 3
5093 018/300 Write block 889 len 24576 data: 3 4 4 5 5 5
5094 019/300 Read block 25 len 20480
5095 020/300 Read block 3735 len 16384
5096 021/300 Read block 4144 len 4096
5097 022/300 Read block 3069 len 20480
5098 023/300 Read block 2715 len 24576
5099 024/300 Write block 3802 len 12288 data: 3 3 3
5100 025/300 Read block 2775 len 12288
5101 026/300 Read block 2741 len 28672
5102 027/300 Write block 929 len 20480 data: 2 2 2 2 2
5103 028/300 Read block 4668 len 36864
5104 029/300 Read block 216 len 32768
5105 030/300 Write block 3171 len 12288 data: 2 2 2
5106 031/300 Read block 1329 len 16384
5107 032/300 Flush
5108 033/300 Write block 670 len 28672 data: 2 2 2 2 2 3 3
5109 034/300 Flush
5110 035/300 Read block 417 len 8192
5111 036/300 Write block 2867 len 20480 data: 3 3 2 2 3
5112 037/300 Read block 771 len 28672
5113 038/300 Read block 16 len 16384
5114 039/300 Write block 2124 len 4096 data: 4
5115 040/300 Write block 1474 len 4096 data: 2
5116 041/300 Write block 3714 len 40960 data: 3 3 3 3 3 3 2 2 2 2
5117 042/300 Read block 4590 len 20480
5118 043/300 Write block 3424 len 8192 data: 4 4
5119 044/300 Read block 4306 len 40960
5120 045/300 Write block 3253 len 32768 data: 3 3 3 3 3 2 3 3
5121 046/300 Write block 3543 len 32768 data: 2 2 3 2 2 2 2 2
5122 047/300 Flush
5123 048/300 Write block 886 len 4096 data: 2
5124 049/300 Read block 3121 len 16384
5125 050/300 Read block 554 len 16384
5126 051/300 Flush
5127 052/300 Read block 957 len 40960
5128 053/300 Read block 2966 len 4096
5129 054/300 Write block 503 len 40960 data: 3 3 3 4 4 5 4 5 4 4
5130 055/300 Write block 372 len 4096 data: 3
5131 056/300 Flush
5132 057/300 Write block 4869 len 40960 data: 2 3 3 3 3 3 3 3 3 3
5133 058/300 Read block 1955 len 8192
5134 059/300 Read block 2183 len 32768
5135 060/300 Read block 452 len 20480
5136 061/300 Write block 4275 len 16384 data: 2 3 3 3
5137 062/300 Flush
5138 063/300 Read block 249 len 32768
5139 064/300 Read block 4036 len 28672
5140 065/300 Read block 1542 len 20480
5141 066/300 Write block 3471 len 32768 data: 3 3 3 3 2 2 2 2
5142 067/300 Read block 2151 len 20480
5143 068/300 Read block 90 len 24576
5144 069/300 Read block 1582 len 40960
5145 070/300 Read block 2660 len 4096
5146 071/300 Write block 641 len 28672 data: 2 2 2 2 2 2 2
5147 072/300 Write block 1441 len 28672 data: 2 2 2 2 2 2 2
5148 073/300 Write block 825 len 28672 data: 2 2 2 2 2 2 3
5149 074/300 Write block 144 len 12288 data: 2 2 2
5150 075/300 Read block 3537 len 4096
5151 076/300 Write block 2210 len 28672 data: 4 5 5 3 3 2 2
5152 077/300 Read block 2900 len 36864
5153 078/300 Read block 3763 len 12288
5154 079/300 Read block 440 len 8192
5155 080/300 Read block 1510 len 12288
5156 081/300 Read block 4529 len 16384
5157 082/300 Write block 4509 len 4096 data: 3
5158 083/300 Write block 4814 len 12288 data: 3 3 3
5159 084/300 Read block 248 len 16384
5160 085/300 Read block 2118 len 8192
5161 086/300 Read block 3848 len 8192
5162 087/300 Flush
5163 088/300 Write block 3473 len 36864 data: 4 4 3 3 3 3 2 2 2
5164 089/300 Read block 546 len 4096
5165 090/300 Read block 4963 len 20480
5166 091/300 Read block 1231 len 36864
5167 092/300 Write block 3739 len 20480 data: 4 4 3 2 2
5168 093/300 Write block 2476 len 8192 data: 3 3
5169 094/300 Read block 3899 len 40960
5170 095/300 Write block 655 len 28672 data: 3 3 3 4 4 4 4
5171 096/300 Read block 2665 len 32768
5172 097/300 Flush
5173 098/300 Read block 1668 len 8192
5174 099/300 Read block 4682 len 4096
5175 100/300 Read block 70 len 32768
5176 101/300 Read block 3366 len 16384
5177 102/300 Flush
5178 103/300 Read block 2764 len 16384
5179 104/300 Read block 2998 len 32768
5180 105/300 Write block 2541 len 24576 data: 2 2 2 2 3 3
5181 106/300 Read block 570 len 12288
5182 107/300 Read block 2309 len 4096
5183 108/300 Flush
5184 109/300 Flush
5185 110/300 Read block 1191 len 20480
5186 111/300 Read block 3078 len 12288
5187 112/300 Write block 1783 len 24576 data: 4 4 4 4 3 3
5188 113/300 Write block 1999 len 40960 data: 2 2 2 2 3 3 3 3 3 3
5189 114/300 Flush
5190 115/300 Read block 2858 len 32768
5191 116/300 Read block 4526 len 28672
5192 117/300 Write block 538 len 24576 data: 2 2 2 2 2 2
5193 118/300 Read block 4088 len 8192
5194 119/300 Read block 3358 len 40960
5195 120/300 Write block 3748 len 36864 data: 4 3 3 3 3 3 3 3 3
5196 121/300 Read block 2275 len 16384
5197 122/300 Write block 3373 len 24576 data: 4 3 3 3 2 2
5198 123/300 Write block 1794 len 8192 data: 2 2
5199 124/300 Write block 1970 len 24576 data: 6 6 5 5 5 5
5200 125/300 Write block 209 len 4096 data: 3
5201 126/300 Read block 3364 len 8192
5202 127/300 Read block 617 len 36864
5203 128/300 Write block 4597 len 16384 data: 2 2 2 2
5204 129/300 Read block 558 len 24576
5205 130/300 Write block 3588 len 40960 data: 4 3 3 2 3 3 3 3 3 3
5206 131/300 Flush
5207 132/300 Write block 3394 len 36864 data: 2 2 2 2 2 2 2 3 3
5208 133/300 Flush
5209 134/300 Read block 1485 len 20480
5210 135/300 Read block 2854 len 16384
5211 136/300 Read block 396 len 12288
5212 137/300 Write block 2936 len 20480 data: 3 3 3 3 3
5213 138/300 Read block 3665 len 12288
5214 139/300 Read block 2780 len 4096
5215 140/300 Write block 195 len 32768 data: 2 2 3 4 4 3 3 4
5216 141/300 Write block 1807 len 24576 data: 3 3 3 3 3 3
5217 142/300 Write block 745 len 4096 data: 3
5218 143/300 Write block 1781 len 12288 data: 3 4 5
5219 144/300 Read block 768 len 4096
5220 145/300 Flush
5221 146/300 Write block 974 len 16384 data: 2 2 2 2
5222 147/300 Read block 886 len 24576
5223 148/300 Write block 3794 len 28672 data: 5 6 5 3 3 3 3
5224 149/300 Read block 2513 len 36864
5225 150/300 Read block 4899 len 36864
5226 151/300 Read block 3628 len 16384
5227 152/300 Write block 3532 len 28672 data: 2 2 2 2 2 2 2
5228 153/300 Read block 2268 len 12288
5229 154/300 Write block 4997 len 4096 data: 3
5230 155/300 Write block 2424 len 8192 data: 3 2
5231 156/300 Write block 4917 len 40960 data: 5 4 3 3 4 3 3 3 3 3
5232 157/300 Read block 1782 len 36864
5233 158/300 Write block 749 len 4096 data: 3
5234 159/300 Write block 1976 len 36864 data: 3 3 2 2 2 2 2 2 2
5235 160/300 Read block 920 len 36864
5236 161/300 Read block 1058 len 24576
5237 162/300 Read block 42 len 8192
5238 163/300 Flush
5239 164/300 Write block 4505 len 36864 data: 3 4 4 3 4 4 4 3 3
5240 165/300 Write block 4689 len 24576 data: 3 3 2 2 2 3
5241 166/300 Read block 1544 len 24576
5242 167/300 Read block 204 len 4096
5243 168/300 Read block 4782 len 20480
5244 169/300 Read block 4388 len 32768
5245 170/300 Read block 4813 len 28672
5246 171/300 Read block 2324 len 32768
5247 172/300 Flush
5248 173/300 Write block 182 len 8192 data: 3 3
5249 174/300 Read block 3873 len 28672
5250 175/300 Write block 2952 len 36864 data: 2 2 3 3 4 4 4 4 4
5251 176/300 Read block 3770 len 28672
5252 177/300 Write block 1765 len 32768 data: 2 2 2 2 2 2 2 3
5253 178/300 Read block 1055 len 16384
5254 179/300 Read block 3591 len 12288
5255 180/300 Write block 4925 len 8192 data: 4 4
5256 181/300 Read block 1249 len 24576
5257 182/300 Write block 4965 len 28672 data: 2 2 2 2 2 2 2
5258 183/300 Read block 2447 len 36864
5259 184/300 Read block 2412 len 16384
5260 185/300 Write block 2238 len 36864 data: 4 4 3 3 3 3 3 3 3
5261 186/300 Read block 2103 len 8192
5262 187/300 Flush
5263 188/300 Read block 2201 len 40960
5264 189/300 Read block 4037 len 24576
5265 190/300 Read block 1641 len 20480
5266 191/300 Write block 3793 len 24576 data: 5 6 7 6 4 4
5267 192/300 Flush
5268 193/300 Read block 3686 len 28672
5269 194/300 Read block 391 len 12288
5270 195/300 Write block 3094 len 20480 data: 3 3 3 3 3
5271 196/300 Read block 3851 len 40960
5272 197/300 Write block 3962 len 32768 data: 2 2 2 2 3 3 3 3
5273 198/300 Write block 4135 len 36864 data: 3 3 3 3 2 2 2 2 2
5274 199/300 Read block 1887 len 12288
5275 200/300 Write block 762 len 16384 data: 3 3 3 3
5276 201/300 Write block 4440 len 4096 data: 3
5277 202/300 Read block 3130 len 32768
5278 203/300 Read block 2192 len 4096
5279 204/300 Flush
5280 205/300 Read block 1477 len 28672
5281 206/300 Read block 1917 len 32768
5282 207/300 Write block 1725 len 12288 data: 2 2 2
5283 208/300 Read block 3754 len 40960
5284 209/300 Write block 3717 len 8192 data: 4 4
5285 210/300 Read block 1489 len 12288
5286 211/300 Read block 1452 len 4096
5287 212/300 Read block 1792 len 36864
5288 213/300 Read block 195 len 16384
5289 214/300 Read block 793 len 36864
5290 215/300 Write block 3916 len 20480 data: 4 3 3 4 4
5291 216/300 Write block 1218 len 16384 data: 2 2 2 2
5292 217/300 Read block 1125 len 28672
5293 218/300 Write block 1660 len 24576 data: 2 2 3 3 3 3
5294 219/300 Write block 3866 len 24576 data: 3 3 3 3 4 4
5295 220/300 Write block 3685 len 24576 data: 2 2 2 2 2 2
5296 221/300 Flush
5297 222/300 Write block 1748 len 12288 data: 2 2 2
5298 223/300 Write block 2766 len 24576 data: 2 2 2 2 3 3
5299 224/300 Write block 3746 len 40960 data: 2 4 5 4 4 4 4 4 4 4
5300 225/300 Read block 3153 len 36864
5301 226/300 Read block 3283 len 4096
5302 227/300 Write block 1431 len 28672 data: 4 5 5 5 6 6 5
5303 228/300 Write block 1019 len 8192 data: 2 2
5304 229/300 Write block 203 len 24576 data: 4 4 4 4 3 3
5305 230/300 Write block 752 len 24576 data: 3 4 4 4 4 3
5306 231/300 Read block 201 len 36864
5307 232/300 Write block 1709 len 8192 data: 2 2
5308 233/300 Read block 4550 len 8192
5309 234/300 Read block 4473 len 20480
5310 235/300 Write block 2679 len 4096 data: 3
5311 236/300 Read block 3808 len 24576
5312 237/300 Read block 1234 len 24576
5313 238/300 Read block 3705 len 40960
5314 239/300 Read block 1574 len 28672
5315 240/300 Write block 1275 len 40960 data: 2 2 2 2 2 2 2 3 3 3
5316 241/300 Read block 2696 len 12288
5317 242/300 Write block 192 len 32768 data: 2 2 2 3 3 4 5 5
5318 243/300 Read block 4640 len 20480
5319 244/300 Read block 1345 len 12288
5320 245/300 Read block 1895 len 40960
5321 246/300 Write block 1938 len 8192 data: 5 4
5322 247/300 Read block 81 len 8192
5323 248/300 Write block 1478 len 4096 data: 3
5324 249/300 Read block 4352 len 16384
5325 250/300 Read block 3989 len 12288
5326 251/300 Write block 4263 len 32768 data: 3 2 2 2 2 2 2 2
5327 252/300 Write block 2837 len 36864 data: 2 2 2 2 2 3 3 2 2
5328 253/300 Read block 3251 len 32768
5329 254/300 Read block 3486 len 28672
5330 255/300 Read block 1789 len 8192
5331 256/300 Read block 4236 len 24576
5332 257/300 Flush
5333 258/300 Write block 2107 len 32768 data: 2 2 2 2 2 2 2 2
5334 259/300 Read block 3498 len 40960
5335 260/300 Read block 1331 len 8192
5336 261/300 Read block 1218 len 32768
5337 262/300 Read block 1986 len 16384
5338 263/300 Write block 3862 len 20480 data: 4 4 4 3 4
5339 264/300 Write block 3949 len 12288 data: 4 4 4
5340 265/300 Write block 3504 len 36864 data: 3 3 3 3 2 2 2 3 3
5341 266/300 Write block 4278 len 4096 data: 4
5342 267/300 Read block 1357 len 36864
5343 268/300 Write block 3383 len 16384 data: 2 2 2 2
5344 269/300 Write block 1740 len 4096 data: 4
5345 270/300 Write block 2187 len 16384 data: 3 3 3 3
5346 271/300 Read block 233 len 40960
5347 272/300 Write block 2399 len 24576 data: 3 3 3 3 3 3
5348 273/300 Write block 4760 len 16384 data: 2 2 2 2
5349 274/300 Read block 686 len 24576
5350 275/300 Read block 45 len 8192
5351 276/300 Write block 975 len 4096 data: 3
5352 277/300 Read block 251 len 8192
5353 278/300 Read block 4790 len 32768
5354 279/300 Write block 3499 len 28672 data: 2 3 3 3 3 4 4
5355 280/300 Write block 3298 len 20480 data: 3 3 3 3 3
5356 281/300 Write block 1975 len 8192 data: 6 4
5357 282/300 Write block 4784 len 40960 data: 2 2 2 2 3 3 3 3 3 3
5358 283/300 Write block 1401 len 16384 data: 2 2 3 3
5359 284/300 Read block 3023 len 8192
5360 285/300 Write block 1361 len 28672 data: 2 3 2 2 2 2 2
5361 286/300 Read block 3620 len 4096
5362 287/300 Write block 85 len 8192 data: 2 3
5363 288/300 Flush
5364 289/300 Write block 3731 len 12288 data: 2 2 2
5365 290/300 Read block 508 len 36864
5366 291/300 Read block 1251 len 36864
5367 292/300 Read block 4817 len 12288
5368 293/300 Write block 3084 len 36864 data: 3 3 2 2 2 2 2 2 3
5369 294/300 Read block 1909 len 16384
5370 295/300 Write block 508 len 40960 data: 6 5 6 5 5 3 4 3 4 3
5371 296/300 Read block 2313 len 16384
5372 297/300 Read block 4709 len 8192
5373 298/300 Write block 4234 len 8192 data: 2 2
5374 299/300 Write block 639 len 36864 data: 2 2 3 3 3 3 3 3 3
5375 300/300 Write block 4956 len 40960 data: 4 3 3 2 2 2 2 2 2 3
5376 301/300 Read block 643 len 16384
5377 Main task has work to do, go find it
5378 got dsc Start(1)
5379 start 1
5380 Nov 01 16:51:38.019 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/1, method: POST, req_id: 263934ca-aacb-4058-8a30-8a9650953c5a, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
5381 Replay: started 1, returned:Ok(())
5382 ----------------------------------------------------------------
5383 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5384 States: Active Offline Active
5385 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5386 2511 Acked Read 6 Done Sent Done false
5387 2512 Acked Read 4 Done Sent Done false
5388 2513 Acked Read 4 Done Sent Done false
5389 2514 Acked Read 5 Done Sent Done false
5390 2515 Acked Flush 0 Done Sent Done false
5391 2516 Acked Write 4 Done Sent Done false
5392 2517 Acked Write 4 Done Sent Done false
5393 2518 Acked Read 4 Done Sent Done false
5394 2519 Acked Write 2 Done Sent Done false
5395 2520 Acked Write 6 Done Sent Done false
5396 2521 Acked Flush 0 Done Sent Done false
5397 2522 Acked Write 2 Done Sent Done false
5398 2523 Acked Flush 0 Done Sent Done false
5399 2524 Acked Write 4 Done Sent Done false
5400 2525 Acked Flush 0 Done Sent Done false
5401 2526 Acked Read 3 Done Sent Done false
5402 2527 Acked Write 2 Done Sent Done false
5403 2528 Acked Write 6 Done Sent Done false
5404 2529 Acked Read 5 Done Sent Done false
5405 2530 Acked Read 4 Done Sent Done false
5406 2531 Acked Read 1 Done Sent Done false
5407 2532 Acked Read 5 Done Sent Done false
5408 2533 Acked Read 6 Done Sent Done false
5409 2534 Acked Write 3 Done Sent Done false
5410 2535 Acked Read 3 Done Sent Done false
5411 2536 Acked Read 7 Done Sent Done false
5412 2537 Acked Write 5 Done Sent Done false
5413 2538 Acked Read 9 Done Sent Done false
5414 2539 Acked Read 8 Done Sent Done false
5415 2540 Acked Write 3 Done Sent Done false
5416 2541 Acked Read 4 Done Sent Done false
5417 2542 Acked Flush 0 Done Sent Done false
5418 2543 Acked Write 7 Done Sent Done false
5419 2544 Acked Flush 0 Done Sent Done false
5420 2545 Acked Read 2 Done Sent Done false
5421 2546 Acked Write 5 Done Sent Done false
5422 2547 Acked Read 7 Done Sent Done false
5423 2548 Acked Read 4 Done Sent Done false
5424 2549 Acked Write 1 Done Sent Done false
5425 2550 Acked Write 1 Done Sent Done false
5426 2551 Acked Write 10 Done Sent Done false
5427 2552 Acked Read 5 Done Sent Done false
5428 2553 Acked Write 2 Done Sent Done false
5429 2554 Acked Read 10 Done Sent Done false
5430 2555 Acked Write 8 Done Sent Done false
5431 2556 Acked Write 8 Done Sent Done false
5432 2557 Acked Flush 0 Done Sent Done false
5433 2558 Acked Write 1 Done Sent Done false
5434 2559 Acked Read 4 Done Sent Done false
5435 2560 Acked Read 4 Done Sent Done false
5436 2561 Acked Flush 0 Done Sent Done false
5437 2562 Acked Read 10 Done Sent Done false
5438 2563 Acked Read 1 Done Sent Done false
5439 2564 Acked Write 10 Done Sent Done false
5440 2565 Acked Write 1 Done Sent Done false
5441 2566 Acked Flush 0 Done Sent Done false
5442 2567 Acked Write 10 Done Sent Done false
5443 2568 Acked Read 2 Done Sent Done false
5444 2569 Acked Read 8 Done Sent Done false
5445 2570 Acked Read 5 Done Sent Done false
5446 2571 Acked Write 4 Done Sent Done false
5447 2572 Acked Flush 0 Done Sent Done false
5448 2573 Acked Read 8 Done Sent Done false
5449 2574 Acked Read 7 Done Sent Done false
5450 2575 Acked Read 5 Done Sent Done false
5451 2576 Acked Write 8 Done Sent Done false
5452 2577 Acked Read 5 Done Sent Done false
5453 2578 Acked Read 6 Done Sent Done false
5454 2579 Acked Read 10 Done Sent Done false
5455 2580 Acked Read 1 Done Sent Done false
5456 2581 Acked Write 7 Done Sent Done false
5457 2582 Acked Write 7 Done Sent Done false
5458 2583 Acked Write 7 Done Sent Done false
5459 2584 Acked Write 3 Done Sent Done false
5460 2585 Acked Read 1 Done Sent Done false
5461 2586 Acked Write 7 Done Sent Done false
5462 2587 Acked Read 9 Done Sent Done false
5463 2588 Acked Read 3 Done Sent Done false
5464 2589 Acked Read 2 Done Sent Done false
5465 2590 Acked Read 3 Done Sent Done false
5466 2591 Acked Read 4 Done Sent Done false
5467 2592 Acked Write 1 Done Sent Done false
5468 2593 Acked Write 3 Done Sent Done false
5469 2594 Acked Read 4 Done Sent Done false
5470 2595 Acked Read 2 Done Sent Done false
5471 2596 Acked Read 2 Done Sent Done false
5472 2597 Acked Flush 0 Done Sent Done false
5473 2598 Acked Write 9 Done Sent Done false
5474 2599 Acked Read 1 Done Sent Done false
5475 2600 Acked Read 5 Done Sent Done false
5476 2601 Acked Read 9 Done Sent Done false
5477 2602 Acked Write 5 Done Sent Done false
5478 2603 Acked Write 2 Done Sent Done false
5479 2604 Acked Read 10 Done Sent Done false
5480 2605 Acked Write 7 Done Sent Done false
5481 2606 Acked Read 8 Done Sent Done false
5482 2607 Acked Flush 0 Done Sent Done false
5483 2608 Acked Read 2 Done Sent Done false
5484 2609 Acked Read 1 Done Sent Done false
5485 2610 Acked Read 8 Done Sent Done false
5486 2611 Acked Read 4 Done Sent Done false
5487 2612 Acked Flush 0 Done Sent Done false
5488 2613 Acked Read 4 Done Sent Done false
5489 2614 Acked Read 8 Done Sent Done false
5490 2615 Acked Write 6 Done Sent Done false
5491 2616 Acked Read 3 Done Sent Done false
5492 2617 Acked Read 1 Done Sent Done false
5493 2618 Acked Flush 0 Done Sent Done false
5494 2619 Acked Flush 0 Done Sent Done false
5495 2620 Acked Read 5 Done Sent Done false
5496 2621 Acked Read 3 Done Sent Done false
5497 2622 Acked Write 6 Done Sent Done false
5498 2623 Acked Write 10 Done Sent Done false
5499 2624 Acked Flush 0 Done Sent Done false
5500 2625 Acked Read 8 Done Sent Done false
5501 2626 Acked Read 7 Done Sent Done false
5502 2627 Acked Write 6 Done Sent Done false
5503 2628 Acked Read 2 Done Sent Done false
5504 2629 Acked Read 10 Done Sent Done false
5505 2630 Acked Write 9 Done Sent Done false
5506 2631 Acked Read 4 Done Sent Done false
5507 2632 Acked Write 6 Done Sent Done false
5508 2633 Acked Write 2 Done Sent Done false
5509 2634 Acked Write 6 Done Sent Done false
5510 2635 Acked Write 1 Done Sent Done false
5511 2636 Acked Read 2 Done Sent Done false
5512 2637 Acked Read 9 Done Sent Done false
5513 2638 Acked Write 4 Done Sent Done false
5514 2639 Acked Read 6 Done Sent Done false
5515 2640 Acked Write 10 Done Sent Done false
5516 2641 Acked Flush 0 Done Sent Done false
5517 2642 Acked Write 9 Done Sent Done false
5518 2643 Acked Flush 0 Done Sent Done false
5519 2644 Acked Read 5 Done Sent Done false
5520 2645 Acked Read 4 Done Sent Done false
5521 2646 Acked Read 3 Done Sent Done false
5522 2647 Acked Write 5 Done Sent Done false
5523 2648 Acked Read 3 Done Sent Done false
5524 2649 Acked Read 1 Done Sent Done false
5525 2650 Acked Write 8 Done Sent Done false
5526 2651 Acked Write 6 Done Sent Done false
5527 2652 Acked Write 1 Done Sent Done false
5528 2653 Acked Write 3 Done Sent Done false
5529 2654 Acked Read 1 Done Sent Done false
5530 2655 Acked Flush 0 Done Sent Done false
5531 2656 Acked Write 4 Done Sent Done false
5532 2657 Acked Read 6 Done Sent Done false
5533 2658 Acked Write 7 Done Sent Done false
5534 2659 Acked Read 9 Done Sent Done false
5535 2660 Acked Read 9 Done Sent Done false
5536 2661 Acked Read 4 Done Sent Done false
5537 2662 Acked Write 7 Done Sent Done false
5538 2663 Acked Read 3 Done Sent Done false
5539 2664 Acked Write 1 Done Sent Done false
5540 2665 Acked Write 2 Done Sent Done false
5541 2666 Acked Write 10 Done Sent Done false
5542 2667 Acked Read 9 Done Sent Done false
5543 2668 Acked Write 1 Done Sent Done false
5544 2669 Acked Write 9 Done Sent Done false
5545 2670 Acked Read 9 Done Sent Done false
5546 2671 Acked Read 6 Done Sent Done false
5547 2672 Acked Read 2 Done Sent Done false
5548 2673 Acked Flush 0 Done Sent Done false
5549 2674 Acked Write 9 Done Sent Done false
5550 2675 Acked Write 6 Done Sent Done false
5551 2676 Acked Read 6 Done Sent Done false
5552 2677 Acked Read 1 Done Sent Done false
5553 2678 Acked Read 5 Done Sent Done false
5554 2679 Acked Read 8 Done Sent Done false
5555 2680 Acked Read 7 Done Sent Done false
5556 2681 Acked Read 8 Done Sent Done false
5557 2682 Acked Flush 0 Done Sent Done false
5558 2683 Acked Write 2 Done Sent Done false
5559 2684 Acked Read 7 Done Sent Done false
5560 2685 Acked Write 9 Done Sent Done false
5561 2686 Acked Read 7 Done Sent Done false
5562 2687 Acked Write 8 Done Sent Done false
5563 2688 Acked Read 4 Done Sent Done false
5564 2689 Acked Read 3 Done Sent Done false
5565 2690 Acked Write 2 Done Sent Done false
5566 2691 Acked Read 6 Done Sent Done false
5567 2692 Acked Write 7 Done Sent Done false
5568 2693 Acked Read 9 Done Sent Done false
5569 2694 Acked Read 4 Done Sent Done false
5570 2695 Acked Write 9 Done Sent Done false
5571 2696 Acked Read 2 Done Sent Done false
5572 2697 Acked Flush 0 Done Sent Done false
5573 2698 Acked Read 10 Done Sent Done false
5574 2699 Acked Read 6 Done Sent Done false
5575 2700 Acked Read 5 Done Sent Done false
5576 2701 Acked Write 6 Done Sent Done false
5577 2702 Acked Flush 0 Done Sent Done false
5578 2703 Acked Read 7 Done Sent Done false
5579 2704 Acked Read 3 Done Sent Done false
5580 2705 Acked Write 5 Done Sent Done false
5581 2706 Acked Read 10 Done Sent Done false
5582 2707 Acked Write 8 Done Sent Done false
5583 2708 Acked Write 9 Done Sent Done false
5584 2709 Acked Read 3 Done Sent Done false
5585 2710 Acked Write 4 Done Sent Done false
5586 2711 Acked Write 1 Done Sent Done false
5587 2712 Acked Read 8 Done Sent Done false
5588 2713 Acked Read 1 Done Sent Done false
5589 2714 Acked Flush 0 Done Sent Done false
5590 2715 Acked Read 7 Done Sent Done false
5591 2716 Acked Read 8 Done Sent Done false
5592 2717 Acked Write 3 Done Sent Done false
5593 2718 Acked Read 10 Done Sent Done false
5594 2719 Acked Write 2 Done Sent Done false
5595 2720 Acked Read 3 Done Sent Done false
5596 2721 Acked Read 1 Done Sent Done false
5597 2722 Acked Read 9 Done Sent Done false
5598 2723 Acked Read 4 Done Sent Done false
5599 2724 Acked Read 9 Done Sent Done false
5600 2725 Acked Write 5 Done Sent Done false
5601 2726 Acked Write 4 Done Sent Done false
5602 2727 Acked Read 7 Done Sent Done false
5603 2728 Acked Write 6 Done Sent Done false
5604 2729 Acked Write 6 Done Sent Done false
5605 2730 Acked Write 6 Done Sent Done false
5606 2731 Acked Flush 0 Done Sent Done false
5607 2732 Acked Write 3 Done Sent Done false
5608 2733 Acked Write 6 Done Sent Done false
5609 2734 Acked Write 10 Done Sent Done false
5610 2735 Acked Read 9 Done Sent Done false
5611 2736 Acked Read 1 Done Sent Done false
5612 2737 Acked Write 7 Done Sent Done false
5613 2738 Acked Write 2 Done Sent Done false
5614 2739 Acked Write 6 Done Sent Done false
5615 2740 Acked Write 6 Done Sent Done false
5616 2741 Acked Read 9 Done Sent Done false
5617 2742 Acked Write 2 Done Sent Done false
5618 2743 Acked Read 2 Done Sent Done false
5619 2744 Acked Read 5 Done Sent Done false
5620 2745 Acked Write 1 Done Sent Done false
5621 2746 Acked Read 6 Done Sent Done false
5622 2747 Acked Read 6 Done Sent Done false
5623 2748 Acked Read 10 Done Sent Done false
5624 2749 Acked Read 7 Done Sent Done false
5625 2750 Acked Write 10 Done Sent Done false
5626 2751 Acked Read 3 Done Sent Done false
5627 2752 Acked Write 8 Done Sent Done false
5628 2753 Acked Read 5 Done Sent Done false
5629 2754 Acked Read 3 Done Sent Done false
5630 2755 Acked Read 10 Done Sent Done false
5631 2756 Acked Write 2 Done Sent Done false
5632 2757 Acked Read 2 Done Sent Done false
5633 2758 Acked Write 1 Done Sent Done false
5634 2759 Acked Read 4 Done Sent Done false
5635 2760 Acked Read 3 Done Sent Done false
5636 2761 Acked Write 8 Done Sent Done false
5637 2762 Acked Write 9 Done Sent Done false
5638 2763 Acked Read 8 Done Sent Done false
5639 2764 Acked Read 7 Done Sent Done false
5640 2765 Acked Read 2 Done Sent Done false
5641 2766 Acked Read 6 Done Sent Done false
5642 2767 Acked Flush 0 Done Sent Done false
5643 2768 Acked Write 8 Done Sent Done false
5644 2769 Acked Read 10 Done Sent Done false
5645 2770 Acked Read 2 Done Sent Done false
5646 2771 Acked Read 8 Done Sent Done false
5647 2772 Acked Read 4 Done Sent Done false
5648 2773 Acked Write 5 Done Sent Done false
5649 2774 Acked Write 3 Done Sent Done false
5650 2775 Acked Write 9 Done Sent Done false
5651 2776 Acked Write 1 Done Sent Done false
5652 2777 Acked Read 9 Done Sent Done false
5653 2778 Acked Write 4 Done Sent Done false
5654 2779 Acked Write 1 Done Sent Done false
5655 2780 Acked Write 4 Done Sent Done false
5656 2781 Acked Read 10 Done Sent Done false
5657 2782 Acked Write 6 Done Sent Done false
5658 2783 Acked Write 4 Done Sent Done false
5659 2784 Acked Read 6 Done Sent Done false
5660 2785 Acked Read 2 Done Sent Done false
5661 2786 Acked Write 1 Done Sent Done false
5662 2787 Acked Read 2 Done Sent Done false
5663 2788 Acked Read 8 Done Sent Done false
5664 2789 Acked Write 7 Done Sent Done false
5665 2790 Acked Write 5 Done Sent Done false
5666 2791 Acked Write 2 Done Sent Done false
5667 2792 Acked Write 10 Done Sent Done false
5668 2793 Acked Write 4 Done Sent Done false
5669 2794 Acked Read 2 Done Sent Done false
5670 2795 Acked Write 7 Done Sent Done false
5671 2796 Acked Read 1 Done Sent Done false
5672 2797 Acked Write 2 Done Sent Done false
5673 2798 Acked Flush 0 Done Sent Done false
5674 2799 Acked Write 3 Done Sent Done false
5675 2800 Acked Read 9 Done Sent Done false
5676 2801 Acked Read 9 Done Sent Done false
5677 2802 Acked Read 3 Done Sent Done false
5678 2803 Acked Write 9 Done Sent Done false
5679 2804 Acked Read 4 Done Sent Done false
5680 2805 Acked Write 10 Done Sent Done false
5681 2806 Acked Read 4 Done Sent Done false
5682 2807 Acked Read 2 Done Sent Done false
5683 2808 Acked Write 2 Done Sent Done false
5684 2809 Acked Write 9 Done Sent Done false
5685 2810 Acked Write 10 Done Sent Done false
5686 2811 Acked Read 4 Done Sent Done false
5687 STATES DS:0 DS:1 DS:2 TOTAL
5688 Sent 0 301 0 301
5689 Done 301 0 301 602
5690 Skipped 0 0 0 0
5691 Error 0 0 0 0
5692 Last Flush: 2798 2510 2798
5693 Downstairs last five completed: 2510 2509 2508 2507 2506
5694 Upstairs last five completed: 2811 2810 2809 2808 2807
5695 CLIENT: Up:0 ds:301 act:2
5696 ----------------------------------------------------------------
5697 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
5698 States: Active Offline Active
5699 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5700 2511 Acked Read 6 Done Sent Done false
5701 2512 Acked Read 4 Done Sent Done false
5702 2513 Acked Read 4 Done Sent Done false
5703 2514 Acked Read 5 Done Sent Done false
5704 2515 Acked Flush 0 Done Sent Done false
5705 2516 Acked Write 4 Done Sent Done false
5706 2517 Acked Write 4 Done Sent Done false
5707 2518 Acked Read 4 Done Sent Done false
5708 2519 Acked Write 2 Done Sent Done false
5709 2520 Acked Write 6 Done Sent Done false
5710 2521 Acked Flush 0 Done Sent Done false
5711 2522 Acked Write 2 Done Sent Done false
5712 2523 Acked Flush 0 Done Sent Done false
5713 2524 Acked Write 4 Done Sent Done false
5714 2525 Acked Flush 0 Done Sent Done false
5715 2526 Acked Read 3 Done Sent Done false
5716 2527 Acked Write 2 Done Sent Done false
5717 2528 Acked Write 6 Done Sent Done false
5718 2529 Acked Read 5 Done Sent Done false
5719 2530 Acked Read 4 Done Sent Done false
5720 2531 Acked Read 1 Done Sent Done false
5721 2532 Acked Read 5 Done Sent Done false
5722 2533 Acked Read 6 Done Sent Done false
5723 2534 Acked Write 3 Done Sent Done false
5724 2535 Acked Read 3 Done Sent Done false
5725 2536 Acked Read 7 Done Sent Done false
5726 2537 Acked Write 5 Done Sent Done false
5727 2538 Acked Read 9 Done Sent Done false
5728 2539 Acked Read 8 Done Sent Done false
5729 2540 Acked Write 3 Done Sent Done false
5730 2541 Acked Read 4 Done Sent Done false
5731 2542 Acked Flush 0 Done Sent Done false
5732 2543 Acked Write 7 Done Sent Done false
5733 2544 Acked Flush 0 Done Sent Done false
5734 2545 Acked Read 2 Done Sent Done false
5735 2546 Acked Write 5 Done Sent Done false
5736 2547 Acked Read 7 Done Sent Done false
5737 2548 Acked Read 4 Done Sent Done false
5738 2549 Acked Write 1 Done Sent Done false
5739 2550 Acked Write 1 Done Sent Done false
5740 2551 Acked Write 10 Done Sent Done false
5741 2552 Acked Read 5 Done Sent Done false
5742 2553 Acked Write 2 Done Sent Done false
5743 2554 Acked Read 10 Done Sent Done false
5744 2555 Acked Write 8 Done Sent Done false
5745 2556 Acked Write 8 Done Sent Done false
5746 2557 Acked Flush 0 Done Sent Done false
5747 2558 Acked Write 1 Done Sent Done false
5748 2559 Acked Read 4 Done Sent Done false
5749 2560 Acked Read 4 Done Sent Done false
5750 2561 Acked Flush 0 Done Sent Done false
5751 2562 Acked Read 10 Done Sent Done false
5752 2563 Acked Read 1 Done Sent Done false
5753 2564 Acked Write 10 Done Sent Done false
5754 2565 Acked Write 1 Done Sent Done false
5755 2566 Acked Flush 0 Done Sent Done false
5756 2567 Acked Write 10 Done Sent Done false
5757 2568 Acked Read 2 Done Sent Done false
5758 2569 Acked Read 8 Done Sent Done false
5759 2570 Acked Read 5 Done Sent Done false
5760 2571 Acked Write 4 Done Sent Done false
5761 2572 Acked Flush 0 Done Sent Done false
5762 2573 Acked Read 8 Done Sent Done false
5763 2574 Acked Read 7 Done Sent Done false
5764 2575 Acked Read 5 Done Sent Done false
5765 2576 Acked Write 8 Done Sent Done false
5766 2577 Acked Read 5 Done Sent Done false
5767 2578 Acked Read 6 Done Sent Done false
5768 2579 Acked Read 10 Done Sent Done false
5769 2580 Acked Read 1 Done Sent Done false
5770 2581 Acked Write 7 Done Sent Done false
5771 2582 Acked Write 7 Done Sent Done false
5772 2583 Acked Write 7 Done Sent Done false
5773 2584 Acked Write 3 Done Sent Done false
5774 2585 Acked Read 1 Done Sent Done false
5775 2586 Acked Write 7 Done Sent Done false
5776 2587 Acked Read 9 Done Sent Done false
5777 2588 Acked Read 3 Done Sent Done false
5778 2589 Acked Read 2 Done Sent Done false
5779 2590 Acked Read 3 Done Sent Done false
5780 2591 Acked Read 4 Done Sent Done false
5781 2592 Acked Write 1 Done Sent Done false
5782 2593 Acked Write 3 Done Sent Done false
5783 2594 Acked Read 4 Done Sent Done false
5784 2595 Acked Read 2 Done Sent Done false
5785 2596 Acked Read 2 Done Sent Done false
5786 2597 Acked Flush 0 Done Sent Done false
5787 2598 Acked Write 9 Done Sent Done false
5788 2599 Acked Read 1 Done Sent Done false
5789 2600 Acked Read 5 Done Sent Done false
5790 2601 Acked Read 9 Done Sent Done false
5791 2602 Acked Write 5 Done Sent Done false
5792 2603 Acked Write 2 Done Sent Done false
5793 2604 Acked Read 10 Done Sent Done false
5794 2605 Acked Write 7 Done Sent Done false
5795 2606 Acked Read 8 Done Sent Done false
5796 2607 Acked Flush 0 Done Sent Done false
5797 2608 Acked Read 2 Done Sent Done false
5798 2609 Acked Read 1 Done Sent Done false
5799 2610 Acked Read 8 Done Sent Done false
5800 2611 Acked Read 4 Done Sent Done false
5801 2612 Acked Flush 0 Done Sent Done false
5802 2613 Acked Read 4 Done Sent Done false
5803 2614 Acked Read 8 Done Sent Done false
5804 2615 Acked Write 6 Done Sent Done false
5805 2616 Acked Read 3 Done Sent Done false
5806 2617 Acked Read 1 Done Sent Done false
5807 2618 Acked Flush 0 Done Sent Done false
5808 2619 Acked Flush 0 Done Sent Done false
5809 2620 Acked Read 5 Done Sent Done false
5810 2621 Acked Read 3 Done Sent Done false
5811 2622 Acked Write 6 Done Sent Done false
5812 2623 Acked Write 10 Done Sent Done false
5813 2624 Acked Flush 0 Done Sent Done false
5814 2625 Acked Read 8 Done Sent Done false
5815 2626 Acked Read 7 Done Sent Done false
5816 2627 Acked Write 6 Done Sent Done false
5817 2628 Acked Read 2 Done Sent Done false
5818 2629 Acked Read 10 Done Sent Done false
5819 2630 Acked Write 9 Done Sent Done false
5820 2631 Acked Read 4 Done Sent Done false
5821 2632 Acked Write 6 Done Sent Done false
5822 2633 Acked Write 2 Done Sent Done false
5823 2634 Acked Write 6 Done Sent Done false
5824 2635 Acked Write 1 Done Sent Done false
5825 2636 Acked Read 2 Done Sent Done false
5826 2637 Acked Read 9 Done Sent Done false
5827 2638 Acked Write 4 Done Sent Done false
5828 2639 Acked Read 6 Done Sent Done false
5829 2640 Acked Write 10 Done Sent Done false
5830 2641 Acked Flush 0 Done Sent Done false
5831 2642 Acked Write 9 Done Sent Done false
5832 2643 Acked Flush 0 Done Sent Done false
5833 2644 Acked Read 5 Done Sent Done false
5834 2645 Acked Read 4 Done Sent Done false
5835 2646 Acked Read 3 Done Sent Done false
5836 2647 Acked Write 5 Done Sent Done false
5837 2648 Acked Read 3 Done Sent Done false
5838 2649 Acked Read 1 Done Sent Done false
5839 2650 Acked Write 8 Done Sent Done false
5840 2651 Acked Write 6 Done Sent Done false
5841 2652 Acked Write 1 Done Sent Done false
5842 2653 Acked Write 3 Done Sent Done false
5843 2654 Acked Read 1 Done Sent Done false
5844 2655 Acked Flush 0 Done Sent Done false
5845 2656 Acked Write 4 Done Sent Done false
5846 2657 Acked Read 6 Done Sent Done false
5847 2658 Acked Write 7 Done Sent Done false
5848 2659 Acked Read 9 Done Sent Done false
5849 2660 Acked Read 9 Done Sent Done false
5850 2661 Acked Read 4 Done Sent Done false
5851 2662 Acked Write 7 Done Sent Done false
5852 2663 Acked Read 3 Done Sent Done false
5853 2664 Acked Write 1 Done Sent Done false
5854 2665 Acked Write 2 Done Sent Done false
5855 2666 Acked Write 10 Done Sent Done false
5856 2667 Acked Read 9 Done Sent Done false
5857 2668 Acked Write 1 Done Sent Done false
5858 2669 Acked Write 9 Done Sent Done false
5859 2670 Acked Read 9 Done Sent Done false
5860 2671 Acked Read 6 Done Sent Done false
5861 2672 Acked Read 2 Done Sent Done false
5862 2673 Acked Flush 0 Done Sent Done false
5863 2674 Acked Write 9 Done Sent Done false
5864 2675 Acked Write 6 Done Sent Done false
5865 2676 Acked Read 6 Done Sent Done false
5866 2677 Acked Read 1 Done Sent Done false
5867 2678 Acked Read 5 Done Sent Done false
5868 2679 Acked Read 8 Done Sent Done false
5869 2680 Acked Read 7 Done Sent Done false
5870 2681 Acked Read 8 Done Sent Done false
5871 2682 Acked Flush 0 Done Sent Done false
5872 2683 Acked Write 2 Done Sent Done false
5873 2684 Acked Read 7 Done Sent Done false
5874 2685 Acked Write 9 Done Sent Done false
5875 2686 Acked Read 7 Done Sent Done false
5876 2687 Acked Write 8 Done Sent Done false
5877 2688 Acked Read 4 Done Sent Done false
5878 2689 Acked Read 3 Done Sent Done false
5879 2690 Acked Write 2 Done Sent Done false
5880 2691 Acked Read 6 Done Sent Done false
5881 2692 Acked Write 7 Done Sent Done false
5882 2693 Acked Read 9 Done Sent Done false
5883 2694 Acked Read 4 Done Sent Done false
5884 2695 Acked Write 9 Done Sent Done false
5885 2696 Acked Read 2 Done Sent Done false
5886 2697 Acked Flush 0 Done Sent Done false
5887 2698 Acked Read 10 Done Sent Done false
5888 2699 Acked Read 6 Done Sent Done false
5889 2700 Acked Read 5 Done Sent Done false
5890 2701 Acked Write 6 Done Sent Done false
5891 2702 Acked Flush 0 Done Sent Done false
5892 2703 Acked Read 7 Done Sent Done false
5893 2704 Acked Read 3 Done Sent Done false
5894 2705 Acked Write 5 Done Sent Done false
5895 2706 Acked Read 10 Done Sent Done false
5896 2707 Acked Write 8 Done Sent Done false
5897 2708 Acked Write 9 Done Sent Done false
5898 2709 Acked Read 3 Done Sent Done false
5899 2710 Acked Write 4 Done Sent Done false
5900 2711 Acked Write 1 Done Sent Done false
5901 2712 Acked Read 8 Done Sent Done false
5902 2713 Acked Read 1 Done Sent Done false
5903 2714 Acked Flush 0 Done Sent Done false
5904 2715 Acked Read 7 Done Sent Done false
5905 2716 Acked Read 8 Done Sent Done false
5906 2717 Acked Write 3 Done Sent Done false
5907 2718 Acked Read 10 Done Sent Done false
5908 2719 Acked Write 2 Done Sent Done false
5909 2720 Acked Read 3 Done Sent Done false
5910 2721 Acked Read 1 Done Sent Done false
5911 2722 Acked Read 9 Done Sent Done false
5912 2723 Acked Read 4 Done Sent Done false
5913 2724 Acked Read 9 Done Sent Done false
5914 2725 Acked Write 5 Done Sent Done false
5915 2726 Acked Write 4 Done Sent Done false
5916 2727 Acked Read 7 Done Sent Done false
5917 2728 Acked Write 6 Done Sent Done false
5918 2729 Acked Write 6 Done Sent Done false
5919 2730 Acked Write 6 Done Sent Done false
5920 2731 Acked Flush 0 Done Sent Done false
5921 2732 Acked Write 3 Done Sent Done false
5922 2733 Acked Write 6 Done Sent Done false
5923 2734 Acked Write 10 Done Sent Done false
5924 2735 Acked Read 9 Done Sent Done false
5925 2736 Acked Read 1 Done Sent Done false
5926 2737 Acked Write 7 Done Sent Done false
5927 2738 Acked Write 2 Done Sent Done false
5928 2739 Acked Write 6 Done Sent Done false
5929 2740 Acked Write 6 Done Sent Done false
5930 2741 Acked Read 9 Done Sent Done false
5931 2742 Acked Write 2 Done Sent Done false
5932 2743 Acked Read 2 Done Sent Done false
5933 2744 Acked Read 5 Done Sent Done false
5934 2745 Acked Write 1 Done Sent Done false
5935 2746 Acked Read 6 Done Sent Done false
5936 2747 Acked Read 6 Done Sent Done false
5937 2748 Acked Read 10 Done Sent Done false
5938 2749 Acked Read 7 Done Sent Done false
5939 2750 Acked Write 10 Done Sent Done false
5940 2751 Acked Read 3 Done Sent Done false
5941 2752 Acked Write 8 Done Sent Done false
5942 2753 Acked Read 5 Done Sent Done false
5943 2754 Acked Read 3 Done Sent Done false
5944 2755 Acked Read 10 Done Sent Done false
5945 2756 Acked Write 2 Done Sent Done false
5946 2757 Acked Read 2 Done Sent Done false
5947 2758 Acked Write 1 Done Sent Done false
5948 2759 Acked Read 4 Done Sent Done false
5949 2760 Acked Read 3 Done Sent Done false
5950 2761 Acked Write 8 Done Sent Done false
5951 2762 Acked Write 9 Done Sent Done false
5952 2763 Acked Read 8 Done Sent Done false
5953 2764 Acked Read 7 Done Sent Done false
5954 2765 Acked Read 2 Done Sent Done false
5955 2766 Acked Read 6 Done Sent Done false
5956 2767 Acked Flush 0 Done Sent Done false
5957 2768 Acked Write 8 Done Sent Done false
5958 2769 Acked Read 10 Done Sent Done false
5959 2770 Acked Read 2 Done Sent Done false
5960 2771 Acked Read 8 Done Sent Done false
5961 2772 Acked Read 4 Done Sent Done false
5962 2773 Acked Write 5 Done Sent Done false
5963 2774 Acked Write 3 Done Sent Done false
5964 2775 Acked Write 9 Done Sent Done false
5965 2776 Acked Write 1 Done Sent Done false
5966 2777 Acked Read 9 Done Sent Done false
5967 2778 Acked Write 4 Done Sent Done false
5968 2779 Acked Write 1 Done Sent Done false
5969 2780 Acked Write 4 Done Sent Done false
5970 2781 Acked Read 10 Done Sent Done false
5971 2782 Acked Write 6 Done Sent Done false
5972 2783 Acked Write 4 Done Sent Done false
5973 2784 Acked Read 6 Done Sent Done false
5974 2785 Acked Read 2 Done Sent Done false
5975 2786 Acked Write 1 Done Sent Done false
5976 2787 Acked Read 2 Done Sent Done false
5977 2788 Acked Read 8 Done Sent Done false
5978 2789 Acked Write 7 Done Sent Done false
5979 2790 Acked Write 5 Done Sent Done false
5980 2791 Acked Write 2 Done Sent Done false
5981 2792 Acked Write 10 Done Sent Done false
5982 2793 Acked Write 4 Done Sent Done false
5983 2794 Acked Read 2 Done Sent Done false
5984 2795 Acked Write 7 Done Sent Done false
5985 2796 Acked Read 1 Done Sent Done false
5986 2797 Acked Write 2 Done Sent Done false
5987 2798 Acked Flush 0 Done Sent Done false
5988 2799 Acked Write 3 Done Sent Done false
5989 2800 Acked Read 9 Done Sent Done false
5990 2801 Acked Read 9 Done Sent Done false
5991 2802 Acked Read 3 Done Sent Done false
5992 2803 Acked Write 9 Done Sent Done false
5993 2804 Acked Read 4 Done Sent Done false
5994 2805 Acked Write 10 Done Sent Done false
5995 2806 Acked Read 4 Done Sent Done false
5996 2807 Acked Read 2 Done Sent Done false
5997 2808 Acked Write 2 Done Sent Done false
5998 2809 Acked Write 9 Done Sent Done false
5999 2810 Acked Write 10 Done Sent Done false
6000 2811 Acked Read 4 Done Sent Done false
6001 2812 Acked Flush 0 Done Sent Done false
6002 STATES DS:0 DS:1 DS:2 TOTAL
6003 Sent 0 302 0 302
6004 Done 302 0 302 604
6005 Skipped 0 0 0 0
6006 Error 0 0 0 0
6007 Last Flush: 2812 2510 2812
6008 Downstairs last five completed: 2510 2509 2508 2507 2506
6009 Upstairs last five completed: 2812 2811 2810 2809 2808
6010 CLIENT: Up:0 ds:302 act:2
60112024-11-01T16:51:43.733ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
60122024-11-01T16:51:43.733ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
60132024-11-01T16:51:43.733ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
60142024-11-01T16:51:43.733ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
60152024-11-01T16:51:43.733ZINFOcrucible: client task is exiting = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
6016 [8820] got start action
6017 [8820] I am going to restart
6018 Starting downstairs at port 8820
6019 Make output file at "/tmp/dsc/downstairs-8820.txt"
6020 [8820][1] reports Starting
6021 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1204)
6022 [8820][1] reports Running
6023 ----------------------------------------------------------------
6024 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
6025 States: Active Offline Active
6026 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6027 2511 Acked Read 6 Done Sent Done false
6028 2512 Acked Read 4 Done Sent Done false
6029 2513 Acked Read 4 Done Sent Done false
6030 2514 Acked Read 5 Done Sent Done false
6031 2515 Acked Flush 0 Done Sent Done false
6032 2516 Acked Write 4 Done Sent Done false
6033 2517 Acked Write 4 Done Sent Done false
6034 2518 Acked Read 4 Done Sent Done false
6035 2519 Acked Write 2 Done Sent Done false
6036 2520 Acked Write 6 Done Sent Done false
6037 2521 Acked Flush 0 Done Sent Done false
6038 2522 Acked Write 2 Done Sent Done false
6039 2523 Acked Flush 0 Done Sent Done false
6040 2524 Acked Write 4 Done Sent Done false
6041 2525 Acked Flush 0 Done Sent Done false
6042 2526 Acked Read 3 Done Sent Done false
6043 2527 Acked Write 2 Done Sent Done false
6044 2528 Acked Write 6 Done Sent Done false
6045 2529 Acked Read 5 Done Sent Done false
6046 2530 Acked Read 4 Done Sent Done false
6047 2531 Acked Read 1 Done Sent Done false
6048 2532 Acked Read 5 Done Sent Done false
6049 2533 Acked Read 6 Done Sent Done false
6050 2534 Acked Write 3 Done Sent Done false
6051 2535 Acked Read 3 Done Sent Done false
6052 2536 Acked Read 7 Done Sent Done false
6053 2537 Acked Write 5 Done Sent Done false
6054 2538 Acked Read 9 Done Sent Done false
6055 2539 Acked Read 8 Done Sent Done false
6056 2540 Acked Write 3 Done Sent Done false
6057 2541 Acked Read 4 Done Sent Done false
6058 2542 Acked Flush 0 Done Sent Done false
6059 2543 Acked Write 7 Done Sent Done false
6060 2544 Acked Flush 0 Done Sent Done false
6061 2545 Acked Read 2 Done Sent Done false
6062 2546 Acked Write 5 Done Sent Done false
6063 2547 Acked Read 7 Done Sent Done false
6064 2548 Acked Read 4 Done Sent Done false
6065 2549 Acked Write 1 Done Sent Done false
6066 2550 Acked Write 1 Done Sent Done false
6067 2551 Acked Write 10 Done Sent Done false
6068 2552 Acked Read 5 Done Sent Done false
6069 2553 Acked Write 2 Done Sent Done false
6070 2554 Acked Read 10 Done Sent Done false
6071 2555 Acked Write 8 Done Sent Done false
6072 2556 Acked Write 8 Done Sent Done false
6073 2557 Acked Flush 0 Done Sent Done false
6074 2558 Acked Write 1 Done Sent Done false
6075 2559 Acked Read 4 Done Sent Done false
6076 2560 Acked Read 4 Done Sent Done false
6077 2561 Acked Flush 0 Done Sent Done false
6078 2562 Acked Read 10 Done Sent Done false
6079 2563 Acked Read 1 Done Sent Done false
6080 2564 Acked Write 10 Done Sent Done false
6081 2565 Acked Write 1 Done Sent Done false
6082 2566 Acked Flush 0 Done Sent Done false
6083 2567 Acked Write 10 Done Sent Done false
6084 2568 Acked Read 2 Done Sent Done false
6085 2569 Acked Read 8 Done Sent Done false
6086 2570 Acked Read 5 Done Sent Done false
6087 2571 Acked Write 4 Done Sent Done false
6088 2572 Acked Flush 0 Done Sent Done false
6089 2573 Acked Read 8 Done Sent Done false
6090 2574 Acked Read 7 Done Sent Done false
6091 2575 Acked Read 5 Done Sent Done false
6092 2576 Acked Write 8 Done Sent Done false
6093 2577 Acked Read 5 Done Sent Done false
6094 2578 Acked Read 6 Done Sent Done false
6095 2579 Acked Read 10 Done Sent Done false
6096 2580 Acked Read 1 Done Sent Done false
6097 2581 Acked Write 7 Done Sent Done false
6098 2582 Acked Write 7 Done Sent Done false
6099 2583 Acked Write 7 Done Sent Done false
6100 2584 Acked Write 3 Done Sent Done false
6101 2585 Acked Read 1 Done Sent Done false
6102 2586 Acked Write 7 Done Sent Done false
6103 2587 Acked Read 9 Done Sent Done false
6104 2588 Acked Read 3 Done Sent Done false
6105 2589 Acked Read 2 Done Sent Done false
6106 2590 Acked Read 3 Done Sent Done false
6107 2591 Acked Read 4 Done Sent Done false
6108 2592 Acked Write 1 Done Sent Done false
6109 2593 Acked Write 3 Done Sent Done false
6110 2594 Acked Read 4 Done Sent Done false
6111 2595 Acked Read 2 Done Sent Done false
6112 2596 Acked Read 2 Done Sent Done false
6113 2597 Acked Flush 0 Done Sent Done false
6114 2598 Acked Write 9 Done Sent Done false
6115 2599 Acked Read 1 Done Sent Done false
6116 2600 Acked Read 5 Done Sent Done false
6117 2601 Acked Read 9 Done Sent Done false
6118 2602 Acked Write 5 Done Sent Done false
6119 2603 Acked Write 2 Done Sent Done false
6120 2604 Acked Read 10 Done Sent Done false
6121 2605 Acked Write 7 Done Sent Done false
6122 2606 Acked Read 8 Done Sent Done false
6123 2607 Acked Flush 0 Done Sent Done false
6124 2608 Acked Read 2 Done Sent Done false
6125 2609 Acked Read 1 Done Sent Done false
6126 2610 Acked Read 8 Done Sent Done false
6127 2611 Acked Read 4 Done Sent Done false
6128 2612 Acked Flush 0 Done Sent Done false
6129 2613 Acked Read 4 Done Sent Done false
6130 2614 Acked Read 8 Done Sent Done false
6131 2615 Acked Write 6 Done Sent Done false
6132 2616 Acked Read 3 Done Sent Done false
6133 2617 Acked Read 1 Done Sent Done false
6134 2618 Acked Flush 0 Done Sent Done false
6135 2619 Acked Flush 0 Done Sent Done false
6136 2620 Acked Read 5 Done Sent Done false
6137 2621 Acked Read 3 Done Sent Done false
6138 2622 Acked Write 6 Done Sent Done false
6139 2623 Acked Write 10 Done Sent Done false
6140 2624 Acked Flush 0 Done Sent Done false
6141 2625 Acked Read 8 Done Sent Done false
6142 2626 Acked Read 7 Done Sent Done false
6143 2627 Acked Write 6 Done Sent Done false
6144 2628 Acked Read 2 Done Sent Done false
6145 2629 Acked Read 10 Done Sent Done false
6146 2630 Acked Write 9 Done Sent Done false
6147 2631 Acked Read 4 Done Sent Done false
6148 2632 Acked Write 6 Done Sent Done false
6149 2633 Acked Write 2 Done Sent Done false
6150 2634 Acked Write 6 Done Sent Done false
6151 2635 Acked Write 1 Done Sent Done false
6152 2636 Acked Read 2 Done Sent Done false
6153 2637 Acked Read 9 Done Sent Done false
6154 2638 Acked Write 4 Done Sent Done false
6155 2639 Acked Read 6 Done Sent Done false
6156 2640 Acked Write 10 Done Sent Done false
6157 2641 Acked Flush 0 Done Sent Done false
6158 2642 Acked Write 9 Done Sent Done false
6159 2643 Acked Flush 0 Done Sent Done false
6160 2644 Acked Read 5 Done Sent Done false
6161 2645 Acked Read 4 Done Sent Done false
6162 2646 Acked Read 3 Done Sent Done false
6163 2647 Acked Write 5 Done Sent Done false
6164 2648 Acked Read 3 Done Sent Done false
6165 2649 Acked Read 1 Done Sent Done false
6166 2650 Acked Write 8 Done Sent Done false
6167 2651 Acked Write 6 Done Sent Done false
6168 2652 Acked Write 1 Done Sent Done false
6169 2653 Acked Write 3 Done Sent Done false
6170 2654 Acked Read 1 Done Sent Done false
6171 2655 Acked Flush 0 Done Sent Done false
6172 2656 Acked Write 4 Done Sent Done false
6173 2657 Acked Read 6 Done Sent Done false
6174 2658 Acked Write 7 Done Sent Done false
6175 2659 Acked Read 9 Done Sent Done false
6176 2660 Acked Read 9 Done Sent Done false
6177 2661 Acked Read 4 Done Sent Done false
6178 2662 Acked Write 7 Done Sent Done false
6179 2663 Acked Read 3 Done Sent Done false
6180 2664 Acked Write 1 Done Sent Done false
6181 2665 Acked Write 2 Done Sent Done false
6182 2666 Acked Write 10 Done Sent Done false
6183 2667 Acked Read 9 Done Sent Done false
6184 2668 Acked Write 1 Done Sent Done false
6185 2669 Acked Write 9 Done Sent Done false
6186 2670 Acked Read 9 Done Sent Done false
6187 2671 Acked Read 6 Done Sent Done false
6188 2672 Acked Read 2 Done Sent Done false
6189 2673 Acked Flush 0 Done Sent Done false
6190 2674 Acked Write 9 Done Sent Done false
6191 2675 Acked Write 6 Done Sent Done false
6192 2676 Acked Read 6 Done Sent Done false
6193 2677 Acked Read 1 Done Sent Done false
6194 2678 Acked Read 5 Done Sent Done false
6195 2679 Acked Read 8 Done Sent Done false
6196 2680 Acked Read 7 Done Sent Done false
6197 2681 Acked Read 8 Done Sent Done false
6198 2682 Acked Flush 0 Done Sent Done false
6199 2683 Acked Write 2 Done Sent Done false
6200 2684 Acked Read 7 Done Sent Done false
6201 2685 Acked Write 9 Done Sent Done false
6202 2686 Acked Read 7 Done Sent Done false
6203 2687 Acked Write 8 Done Sent Done false
6204 2688 Acked Read 4 Done Sent Done false
6205 2689 Acked Read 3 Done Sent Done false
6206 2690 Acked Write 2 Done Sent Done false
6207 2691 Acked Read 6 Done Sent Done false
6208 2692 Acked Write 7 Done Sent Done false
6209 2693 Acked Read 9 Done Sent Done false
6210 2694 Acked Read 4 Done Sent Done false
6211 2695 Acked Write 9 Done Sent Done false
6212 2696 Acked Read 2 Done Sent Done false
6213 2697 Acked Flush 0 Done Sent Done false
6214 2698 Acked Read 10 Done Sent Done false
6215 2699 Acked Read 6 Done Sent Done false
6216 2700 Acked Read 5 Done Sent Done false
6217 2701 Acked Write 6 Done Sent Done false
6218 2702 Acked Flush 0 Done Sent Done false
6219 2703 Acked Read 7 Done Sent Done false
6220 2704 Acked Read 3 Done Sent Done false
6221 2705 Acked Write 5 Done Sent Done false
6222 2706 Acked Read 10 Done Sent Done false
6223 2707 Acked Write 8 Done Sent Done false
6224 2708 Acked Write 9 Done Sent Done false
6225 2709 Acked Read 3 Done Sent Done false
6226 2710 Acked Write 4 Done Sent Done false
6227 2711 Acked Write 1 Done Sent Done false
6228 2712 Acked Read 8 Done Sent Done false
6229 2713 Acked Read 1 Done Sent Done false
6230 2714 Acked Flush 0 Done Sent Done false
6231 2715 Acked Read 7 Done Sent Done false
6232 2716 Acked Read 8 Done Sent Done false
6233 2717 Acked Write 3 Done Sent Done false
6234 2718 Acked Read 10 Done Sent Done false
6235 2719 Acked Write 2 Done Sent Done false
6236 2720 Acked Read 3 Done Sent Done false
6237 2721 Acked Read 1 Done Sent Done false
6238 2722 Acked Read 9 Done Sent Done false
6239 2723 Acked Read 4 Done Sent Done false
6240 2724 Acked Read 9 Done Sent Done false
6241 2725 Acked Write 5 Done Sent Done false
6242 2726 Acked Write 4 Done Sent Done false
6243 2727 Acked Read 7 Done Sent Done false
6244 2728 Acked Write 6 Done Sent Done false
6245 2729 Acked Write 6 Done Sent Done false
6246 2730 Acked Write 6 Done Sent Done false
6247 2731 Acked Flush 0 Done Sent Done false
6248 2732 Acked Write 3 Done Sent Done false
6249 2733 Acked Write 6 Done Sent Done false
6250 2734 Acked Write 10 Done Sent Done false
6251 2735 Acked Read 9 Done Sent Done false
6252 2736 Acked Read 1 Done Sent Done false
6253 2737 Acked Write 7 Done Sent Done false
6254 2738 Acked Write 2 Done Sent Done false
6255 2739 Acked Write 6 Done Sent Done false
6256 2740 Acked Write 6 Done Sent Done false
6257 2741 Acked Read 9 Done Sent Done false
6258 2742 Acked Write 2 Done Sent Done false
6259 2743 Acked Read 2 Done Sent Done false
6260 2744 Acked Read 5 Done Sent Done false
6261 2745 Acked Write 1 Done Sent Done false
6262 2746 Acked Read 6 Done Sent Done false
6263 2747 Acked Read 6 Done Sent Done false
6264 2748 Acked Read 10 Done Sent Done false
6265 2749 Acked Read 7 Done Sent Done false
6266 2750 Acked Write 10 Done Sent Done false
6267 2751 Acked Read 3 Done Sent Done false
6268 2752 Acked Write 8 Done Sent Done false
6269 2753 Acked Read 5 Done Sent Done false
6270 2754 Acked Read 3 Done Sent Done false
6271 2755 Acked Read 10 Done Sent Done false
6272 2756 Acked Write 2 Done Sent Done false
6273 2757 Acked Read 2 Done Sent Done false
6274 2758 Acked Write 1 Done Sent Done false
6275 2759 Acked Read 4 Done Sent Done false
6276 2760 Acked Read 3 Done Sent Done false
6277 2761 Acked Write 8 Done Sent Done false
6278 2762 Acked Write 9 Done Sent Done false
6279 2763 Acked Read 8 Done Sent Done false
6280 2764 Acked Read 7 Done Sent Done false
6281 2765 Acked Read 2 Done Sent Done false
6282 2766 Acked Read 6 Done Sent Done false
6283 2767 Acked Flush 0 Done Sent Done false
6284 2768 Acked Write 8 Done Sent Done false
6285 2769 Acked Read 10 Done Sent Done false
6286 2770 Acked Read 2 Done Sent Done false
6287 2771 Acked Read 8 Done Sent Done false
6288 2772 Acked Read 4 Done Sent Done false
6289 2773 Acked Write 5 Done Sent Done false
6290 2774 Acked Write 3 Done Sent Done false
6291 2775 Acked Write 9 Done Sent Done false
6292 2776 Acked Write 1 Done Sent Done false
6293 2777 Acked Read 9 Done Sent Done false
6294 2778 Acked Write 4 Done Sent Done false
6295 2779 Acked Write 1 Done Sent Done false
6296 2780 Acked Write 4 Done Sent Done false
6297 2781 Acked Read 10 Done Sent Done false
6298 2782 Acked Write 6 Done Sent Done false
6299 2783 Acked Write 4 Done Sent Done false
6300 2784 Acked Read 6 Done Sent Done false
6301 2785 Acked Read 2 Done Sent Done false
6302 2786 Acked Write 1 Done Sent Done false
6303 2787 Acked Read 2 Done Sent Done false
6304 2788 Acked Read 8 Done Sent Done false
6305 2789 Acked Write 7 Done Sent Done false
6306 2790 Acked Write 5 Done Sent Done false
6307 2791 Acked Write 2 Done Sent Done false
6308 2792 Acked Write 10 Done Sent Done false
6309 2793 Acked Write 4 Done Sent Done false
6310 2794 Acked Read 2 Done Sent Done false
6311 2795 Acked Write 7 Done Sent Done false
6312 2796 Acked Read 1 Done Sent Done false
6313 2797 Acked Write 2 Done Sent Done false
6314 2798 Acked Flush 0 Done Sent Done false
6315 2799 Acked Write 3 Done Sent Done false
6316 2800 Acked Read 9 Done Sent Done false
6317 2801 Acked Read 9 Done Sent Done false
6318 2802 Acked Read 3 Done Sent Done false
6319 2803 Acked Write 9 Done Sent Done false
6320 2804 Acked Read 4 Done Sent Done false
6321 2805 Acked Write 10 Done Sent Done false
6322 2806 Acked Read 4 Done Sent Done false
6323 2807 Acked Read 2 Done Sent Done false
6324 2808 Acked Write 2 Done Sent Done false
6325 2809 Acked Write 9 Done Sent Done false
6326 2810 Acked Write 10 Done Sent Done false
6327 2811 Acked Read 4 Done Sent Done false
6328 2812 Acked Flush 0 Done Sent Done false
6329 STATES DS:0 DS:1 DS:2 TOTAL
6330 Sent 0 302 0 302
6331 Done 302 0 302 604
6332 Skipped 0 0 0 0
6333 Error 0 0 0 0
6334 Last Flush: 2812 2510 2812
6335 Downstairs last five completed: 2510 2509 2508 2507 2506
6336 Upstairs last five completed: 2812 2811 2810 2809 2808
6337 CLIENT: Up:0 ds:302 act:2
6338 ----------------------------------------------------------------
6339 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
6340 States: Active Offline Active
6341 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6342 2511 Acked Read 6 Done Sent Done false
6343 2512 Acked Read 4 Done Sent Done false
6344 2513 Acked Read 4 Done Sent Done false
6345 2514 Acked Read 5 Done Sent Done false
6346 2515 Acked Flush 0 Done Sent Done false
6347 2516 Acked Write 4 Done Sent Done false
6348 2517 Acked Write 4 Done Sent Done false
6349 2518 Acked Read 4 Done Sent Done false
6350 2519 Acked Write 2 Done Sent Done false
6351 2520 Acked Write 6 Done Sent Done false
6352 2521 Acked Flush 0 Done Sent Done false
6353 2522 Acked Write 2 Done Sent Done false
6354 2523 Acked Flush 0 Done Sent Done false
6355 2524 Acked Write 4 Done Sent Done false
6356 2525 Acked Flush 0 Done Sent Done false
6357 2526 Acked Read 3 Done Sent Done false
6358 2527 Acked Write 2 Done Sent Done false
6359 2528 Acked Write 6 Done Sent Done false
6360 2529 Acked Read 5 Done Sent Done false
6361 2530 Acked Read 4 Done Sent Done false
6362 2531 Acked Read 1 Done Sent Done false
6363 2532 Acked Read 5 Done Sent Done false
6364 2533 Acked Read 6 Done Sent Done false
6365 2534 Acked Write 3 Done Sent Done false
6366 2535 Acked Read 3 Done Sent Done false
6367 2536 Acked Read 7 Done Sent Done false
6368 2537 Acked Write 5 Done Sent Done false
6369 2538 Acked Read 9 Done Sent Done false
6370 2539 Acked Read 8 Done Sent Done false
6371 2540 Acked Write 3 Done Sent Done false
6372 2541 Acked Read 4 Done Sent Done false
6373 2542 Acked Flush 0 Done Sent Done false
6374 2543 Acked Write 7 Done Sent Done false
6375 2544 Acked Flush 0 Done Sent Done false
6376 2545 Acked Read 2 Done Sent Done false
6377 2546 Acked Write 5 Done Sent Done false
6378 2547 Acked Read 7 Done Sent Done false
6379 2548 Acked Read 4 Done Sent Done false
6380 2549 Acked Write 1 Done Sent Done false
6381 2550 Acked Write 1 Done Sent Done false
6382 2551 Acked Write 10 Done Sent Done false
6383 2552 Acked Read 5 Done Sent Done false
6384 2553 Acked Write 2 Done Sent Done false
6385 2554 Acked Read 10 Done Sent Done false
6386 2555 Acked Write 8 Done Sent Done false
6387 2556 Acked Write 8 Done Sent Done false
6388 2557 Acked Flush 0 Done Sent Done false
6389 2558 Acked Write 1 Done Sent Done false
6390 2559 Acked Read 4 Done Sent Done false
6391 2560 Acked Read 4 Done Sent Done false
6392 2561 Acked Flush 0 Done Sent Done false
6393 2562 Acked Read 10 Done Sent Done false
6394 2563 Acked Read 1 Done Sent Done false
6395 2564 Acked Write 10 Done Sent Done false
6396 2565 Acked Write 1 Done Sent Done false
6397 2566 Acked Flush 0 Done Sent Done false
6398 2567 Acked Write 10 Done Sent Done false
6399 2568 Acked Read 2 Done Sent Done false
6400 2569 Acked Read 8 Done Sent Done false
6401 2570 Acked Read 5 Done Sent Done false
6402 2571 Acked Write 4 Done Sent Done false
6403 2572 Acked Flush 0 Done Sent Done false
6404 2573 Acked Read 8 Done Sent Done false
6405 2574 Acked Read 7 Done Sent Done false
6406 2575 Acked Read 5 Done Sent Done false
6407 2576 Acked Write 8 Done Sent Done false
6408 2577 Acked Read 5 Done Sent Done false
6409 2578 Acked Read 6 Done Sent Done false
6410 2579 Acked Read 10 Done Sent Done false
6411 2580 Acked Read 1 Done Sent Done false
6412 2581 Acked Write 7 Done Sent Done false
6413 2582 Acked Write 7 Done Sent Done false
6414 2583 Acked Write 7 Done Sent Done false
6415 2584 Acked Write 3 Done Sent Done false
6416 2585 Acked Read 1 Done Sent Done false
6417 2586 Acked Write 7 Done Sent Done false
6418 2587 Acked Read 9 Done Sent Done false
6419 2588 Acked Read 3 Done Sent Done false
6420 2589 Acked Read 2 Done Sent Done false
6421 2590 Acked Read 3 Done Sent Done false
6422 2591 Acked Read 4 Done Sent Done false
6423 2592 Acked Write 1 Done Sent Done false
6424 2593 Acked Write 3 Done Sent Done false
6425 2594 Acked Read 4 Done Sent Done false
6426 2595 Acked Read 2 Done Sent Done false
6427 2596 Acked Read 2 Done Sent Done false
6428 2597 Acked Flush 0 Done Sent Done false
6429 2598 Acked Write 9 Done Sent Done false
6430 2599 Acked Read 1 Done Sent Done false
6431 2600 Acked Read 5 Done Sent Done false
6432 2601 Acked Read 9 Done Sent Done false
6433 2602 Acked Write 5 Done Sent Done false
6434 2603 Acked Write 2 Done Sent Done false
6435 2604 Acked Read 10 Done Sent Done false
6436 2605 Acked Write 7 Done Sent Done false
6437 2606 Acked Read 8 Done Sent Done false
6438 2607 Acked Flush 0 Done Sent Done false
6439 2608 Acked Read 2 Done Sent Done false
6440 2609 Acked Read 1 Done Sent Done false
6441 2610 Acked Read 8 Done Sent Done false
6442 2611 Acked Read 4 Done Sent Done false
6443 2612 Acked Flush 0 Done Sent Done false
6444 2613 Acked Read 4 Done Sent Done false
6445 2614 Acked Read 8 Done Sent Done false
6446 2615 Acked Write 6 Done Sent Done false
6447 2616 Acked Read 3 Done Sent Done false
6448 2617 Acked Read 1 Done Sent Done false
6449 2618 Acked Flush 0 Done Sent Done false
6450 2619 Acked Flush 0 Done Sent Done false
6451 2620 Acked Read 5 Done Sent Done false
6452 2621 Acked Read 3 Done Sent Done false
6453 2622 Acked Write 6 Done Sent Done false
6454 2623 Acked Write 10 Done Sent Done false
6455 2624 Acked Flush 0 Done Sent Done false
6456 2625 Acked Read 8 Done Sent Done false
6457 2626 Acked Read 7 Done Sent Done false
6458 2627 Acked Write 6 Done Sent Done false
6459 2628 Acked Read 2 Done Sent Done false
6460 2629 Acked Read 10 Done Sent Done false
6461 2630 Acked Write 9 Done Sent Done false
6462 2631 Acked Read 4 Done Sent Done false
6463 2632 Acked Write 6 Done Sent Done false
6464 2633 Acked Write 2 Done Sent Done false
6465 2634 Acked Write 6 Done Sent Done false
6466 2635 Acked Write 1 Done Sent Done false
6467 2636 Acked Read 2 Done Sent Done false
6468 2637 Acked Read 9 Done Sent Done false
6469 2638 Acked Write 4 Done Sent Done false
6470 2639 Acked Read 6 Done Sent Done false
6471 2640 Acked Write 10 Done Sent Done false
6472 2641 Acked Flush 0 Done Sent Done false
6473 2642 Acked Write 9 Done Sent Done false
6474 2643 Acked Flush 0 Done Sent Done false
6475 2644 Acked Read 5 Done Sent Done false
6476 2645 Acked Read 4 Done Sent Done false
6477 2646 Acked Read 3 Done Sent Done false
6478 2647 Acked Write 5 Done Sent Done false
6479 2648 Acked Read 3 Done Sent Done false
6480 2649 Acked Read 1 Done Sent Done false
6481 2650 Acked Write 8 Done Sent Done false
6482 2651 Acked Write 6 Done Sent Done false
6483 2652 Acked Write 1 Done Sent Done false
6484 2653 Acked Write 3 Done Sent Done false
6485 2654 Acked Read 1 Done Sent Done false
6486 2655 Acked Flush 0 Done Sent Done false
6487 2656 Acked Write 4 Done Sent Done false
6488 2657 Acked Read 6 Done Sent Done false
6489 2658 Acked Write 7 Done Sent Done false
6490 2659 Acked Read 9 Done Sent Done false
6491 2660 Acked Read 9 Done Sent Done false
6492 2661 Acked Read 4 Done Sent Done false
6493 2662 Acked Write 7 Done Sent Done false
6494 2663 Acked Read 3 Done Sent Done false
6495 2664 Acked Write 1 Done Sent Done false
6496 2665 Acked Write 2 Done Sent Done false
6497 2666 Acked Write 10 Done Sent Done false
6498 2667 Acked Read 9 Done Sent Done false
6499 2668 Acked Write 1 Done Sent Done false
6500 2669 Acked Write 9 Done Sent Done false
6501 2670 Acked Read 9 Done Sent Done false
6502 2671 Acked Read 6 Done Sent Done false
6503 2672 Acked Read 2 Done Sent Done false
6504 2673 Acked Flush 0 Done Sent Done false
6505 2674 Acked Write 9 Done Sent Done false
6506 2675 Acked Write 6 Done Sent Done false
6507 2676 Acked Read 6 Done Sent Done false
6508 2677 Acked Read 1 Done Sent Done false
6509 2678 Acked Read 5 Done Sent Done false
6510 2679 Acked Read 8 Done Sent Done false
6511 2680 Acked Read 7 Done Sent Done false
6512 2681 Acked Read 8 Done Sent Done false
6513 2682 Acked Flush 0 Done Sent Done false
6514 2683 Acked Write 2 Done Sent Done false
6515 2684 Acked Read 7 Done Sent Done false
6516 2685 Acked Write 9 Done Sent Done false
6517 2686 Acked Read 7 Done Sent Done false
6518 2687 Acked Write 8 Done Sent Done false
6519 2688 Acked Read 4 Done Sent Done false
6520 2689 Acked Read 3 Done Sent Done false
6521 2690 Acked Write 2 Done Sent Done false
6522 2691 Acked Read 6 Done Sent Done false
6523 2692 Acked Write 7 Done Sent Done false
6524 2693 Acked Read 9 Done Sent Done false
6525 2694 Acked Read 4 Done Sent Done false
6526 2695 Acked Write 9 Done Sent Done false
6527 2696 Acked Read 2 Done Sent Done false
6528 2697 Acked Flush 0 Done Sent Done false
6529 2698 Acked Read 10 Done Sent Done false
6530 2699 Acked Read 6 Done Sent Done false
6531 2700 Acked Read 5 Done Sent Done false
6532 2701 Acked Write 6 Done Sent Done false
6533 2702 Acked Flush 0 Done Sent Done false
6534 2703 Acked Read 7 Done Sent Done false
6535 2704 Acked Read 3 Done Sent Done false
6536 2705 Acked Write 5 Done Sent Done false
6537 2706 Acked Read 10 Done Sent Done false
6538 2707 Acked Write 8 Done Sent Done false
6539 2708 Acked Write 9 Done Sent Done false
6540 2709 Acked Read 3 Done Sent Done false
6541 2710 Acked Write 4 Done Sent Done false
6542 2711 Acked Write 1 Done Sent Done false
6543 2712 Acked Read 8 Done Sent Done false
6544 2713 Acked Read 1 Done Sent Done false
6545 2714 Acked Flush 0 Done Sent Done false
6546 2715 Acked Read 7 Done Sent Done false
6547 2716 Acked Read 8 Done Sent Done false
6548 2717 Acked Write 3 Done Sent Done false
6549 2718 Acked Read 10 Done Sent Done false
6550 2719 Acked Write 2 Done Sent Done false
6551 2720 Acked Read 3 Done Sent Done false
6552 2721 Acked Read 1 Done Sent Done false
6553 2722 Acked Read 9 Done Sent Done false
6554 2723 Acked Read 4 Done Sent Done false
6555 2724 Acked Read 9 Done Sent Done false
6556 2725 Acked Write 5 Done Sent Done false
6557 2726 Acked Write 4 Done Sent Done false
6558 2727 Acked Read 7 Done Sent Done false
6559 2728 Acked Write 6 Done Sent Done false
6560 2729 Acked Write 6 Done Sent Done false
6561 2730 Acked Write 6 Done Sent Done false
6562 2731 Acked Flush 0 Done Sent Done false
6563 2732 Acked Write 3 Done Sent Done false
6564 2733 Acked Write 6 Done Sent Done false
6565 2734 Acked Write 10 Done Sent Done false
6566 2735 Acked Read 9 Done Sent Done false
6567 2736 Acked Read 1 Done Sent Done false
6568 2737 Acked Write 7 Done Sent Done false
6569 2738 Acked Write 2 Done Sent Done false
6570 2739 Acked Write 6 Done Sent Done false
6571 2740 Acked Write 6 Done Sent Done false
6572 2741 Acked Read 9 Done Sent Done false
6573 2742 Acked Write 2 Done Sent Done false
6574 2743 Acked Read 2 Done Sent Done false
6575 2744 Acked Read 5 Done Sent Done false
6576 2745 Acked Write 1 Done Sent Done false
6577 2746 Acked Read 6 Done Sent Done false
6578 2747 Acked Read 6 Done Sent Done false
6579 2748 Acked Read 10 Done Sent Done false
6580 2749 Acked Read 7 Done Sent Done false
6581 2750 Acked Write 10 Done Sent Done false
6582 2751 Acked Read 3 Done Sent Done false
6583 2752 Acked Write 8 Done Sent Done false
6584 2753 Acked Read 5 Done Sent Done false
6585 2754 Acked Read 3 Done Sent Done false
6586 2755 Acked Read 10 Done Sent Done false
6587 2756 Acked Write 2 Done Sent Done false
6588 2757 Acked Read 2 Done Sent Done false
6589 2758 Acked Write 1 Done Sent Done false
6590 2759 Acked Read 4 Done Sent Done false
6591 2760 Acked Read 3 Done Sent Done false
6592 2761 Acked Write 8 Done Sent Done false
6593 2762 Acked Write 9 Done Sent Done false
6594 2763 Acked Read 8 Done Sent Done false
6595 2764 Acked Read 7 Done Sent Done false
6596 2765 Acked Read 2 Done Sent Done false
6597 2766 Acked Read 6 Done Sent Done false
6598 2767 Acked Flush 0 Done Sent Done false
6599 2768 Acked Write 8 Done Sent Done false
6600 2769 Acked Read 10 Done Sent Done false
6601 2770 Acked Read 2 Done Sent Done false
6602 2771 Acked Read 8 Done Sent Done false
6603 2772 Acked Read 4 Done Sent Done false
6604 2773 Acked Write 5 Done Sent Done false
6605 2774 Acked Write 3 Done Sent Done false
6606 2775 Acked Write 9 Done Sent Done false
6607 2776 Acked Write 1 Done Sent Done false
6608 2777 Acked Read 9 Done Sent Done false
6609 2778 Acked Write 4 Done Sent Done false
6610 2779 Acked Write 1 Done Sent Done false
6611 2780 Acked Write 4 Done Sent Done false
6612 2781 Acked Read 10 Done Sent Done false
6613 2782 Acked Write 6 Done Sent Done false
6614 2783 Acked Write 4 Done Sent Done false
6615 2784 Acked Read 6 Done Sent Done false
6616 2785 Acked Read 2 Done Sent Done false
6617 2786 Acked Write 1 Done Sent Done false
6618 2787 Acked Read 2 Done Sent Done false
6619 2788 Acked Read 8 Done Sent Done false
6620 2789 Acked Write 7 Done Sent Done false
6621 2790 Acked Write 5 Done Sent Done false
6622 2791 Acked Write 2 Done Sent Done false
6623 2792 Acked Write 10 Done Sent Done false
6624 2793 Acked Write 4 Done Sent Done false
6625 2794 Acked Read 2 Done Sent Done false
6626 2795 Acked Write 7 Done Sent Done false
6627 2796 Acked Read 1 Done Sent Done false
6628 2797 Acked Write 2 Done Sent Done false
6629 2798 Acked Flush 0 Done Sent Done false
6630 2799 Acked Write 3 Done Sent Done false
6631 2800 Acked Read 9 Done Sent Done false
6632 2801 Acked Read 9 Done Sent Done false
6633 2802 Acked Read 3 Done Sent Done false
6634 2803 Acked Write 9 Done Sent Done false
6635 2804 Acked Read 4 Done Sent Done false
6636 2805 Acked Write 10 Done Sent Done false
6637 2806 Acked Read 4 Done Sent Done false
6638 2807 Acked Read 2 Done Sent Done false
6639 2808 Acked Write 2 Done Sent Done false
6640 2809 Acked Write 9 Done Sent Done false
6641 2810 Acked Write 10 Done Sent Done false
6642 2811 Acked Read 4 Done Sent Done false
6643 2812 Acked Flush 0 Done Sent Done false
6644 STATES DS:0 DS:1 DS:2 TOTAL
6645 Sent 0 302 0 302
6646 Done 302 0 302 604
6647 Skipped 0 0 0 0
6648 Error 0 0 0 0
6649 Last Flush: 2812 2510 2812
6650 Downstairs last five completed: 2510 2509 2508 2507 2506
6651 Upstairs last five completed: 2812 2811 2810 2809 2808
6652 CLIENT: Up:0 ds:302 act:2
66532024-11-01T16:51:53.734ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66542024-11-01T16:51:53.735ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47226) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66552024-11-01T16:51:53.736ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66562024-11-01T16:51:53.736ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66572024-11-01T16:51:53.736ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66582024-11-01T16:51:53.736ZINFOcrucible: send last flush ID to this DS: 2510 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66592024-11-01T16:51:53.737ZINFOcrucible: Replied this last flush ID: 2510 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66602024-11-01T16:51:53.737ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66612024-11-01T16:51:53.737ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66622024-11-01T16:51:53.737ZINFOcrucible: [1] client re-new 302 jobs since flush 2510 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66632024-11-01T16:51:53.737ZINFOcrucible: [1] Replayed 302 jobs since flush: 2510 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
6664 ----------------------------------------------------------------
6665 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6666 Downstairs last five completed: 2812 2811 2810 2809 2808
6667 Upstairs last five completed: 2812 2811 2810 2809 2808
6668 CLIENT: Up:0 ds:0 act:3
6669 Replay: All jobs finished
6670 Main task has work to do, go find it
6671 got dsc Stop(1)
6672 stop 1
6673 Nov 01 16:51:54.029 INFO request completed[8820] Got stop action so:false kr:false
6674 , latency_us: 330, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 4e5323d3-9964-4ce0-8c57-165852b87dd0, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
6675 Nov 01 16:51:54.029 INFO accepted connection, remote_addr: 127.0.0.1:55183, local_addr: 127.0.0.1:9998
66762024-11-01T16:51:54.030ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66772024-11-01T16:51:54.030ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66782024-11-01T16:51:54.030ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66792024-11-01T16:51:54.030ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
66802024-11-01T16:51:54.030ZINFOcrucible: client task is exiting = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
6681 Nov 01 16:51:54.030 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/1, method: GET, req_id: 53013bb7-d118-4094-beb1-66acc8869f97, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
6682 [8820] Exited with: ExitStatus(unix_wait_status(9))
6683 [8820][1] reports Exit
6684 Nov 01 16:51:58.032 INFO request completed, latency_us: 276, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1e1b4081-25eb-41d7-806f-21cb397e644e, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
6685 001/300 Write block 1836 len 16384 data: 2 3 3 3
6686 002/300 Read block 3935 len 4096
6687 003/300 Read block 1561 len 40960
6688 004/300 Write block 3838 len 32768 data: 2 2 2 2 2 2 2 2
6689 005/300 Write block 2651 len 40960 data: 2 2 3 3 2 2 3 3 3 3
6690 006/300 Read block 3753 len 32768
6691 007/300 Flush
6692 008/300 Write block 4970 len 4096 data: 3
6693 009/300 Flush
6694 010/300 Read block 2391 len 28672
6695 011/300 Flush
6696 012/300 Read block 1855 len 8192
6697 013/300 Read block 2363 len 36864
6698 014/300 Write block 4737 len 20480 data: 3 3 3 3 3
6699 015/300 Read block 3335 len 12288
6700 016/300 Read block 4604 len 32768
6701 017/300 Write block 1410 len 32768 data: 3 2 2 2 2 2 2 2
6702 018/300 Read block 2218 len 8192
6703 019/300 Write block 4903 len 32768 data: 3 3 4 4 3 2 3 3
6704 020/300 Write block 3281 len 24576 data: 2 2 2 2 2 2
6705 021/300 Read block 3455 len 20480
6706 022/300 Read block 2351 len 28672
6707 023/300 Write block 3774 len 12288 data: 2 2 2
6708 024/300 Write block 3510 len 24576 data: 3 4 4 3 3 3
6709 025/300 Read block 1049 len 4096
6710 026/300 Read block 1606 len 16384
6711 027/300 Read block 4063 len 36864
6712 028/300 Read block 2513 len 16384
6713 029/300 Read block 611 len 40960
6714 030/300 Read block 3409 len 36864
6715 031/300 Read block 1388 len 4096
6716 032/300 Read block 3938 len 32768
6717 033/300 Read block 2896 len 32768
6718 034/300 Read block 1626 len 40960
6719 035/300 Read block 1483 len 36864
6720 036/300 Read block 689 len 32768
6721 037/300 Write block 2461 len 36864 data: 2 2 2 2 2 2 2 2 2
6722 038/300 Read block 249 len 40960
6723 039/300 Write block 4182 len 8192 data: 3 3
6724 040/300 Read block 1695 len 28672
6725 041/300 Read block 2097 len 8192
6726 042/300 Read block 489 len 4096
6727 043/300 Write block 1234 len 28672 data: 4 3 3 4 4 3 4
6728 044/300 Write block 2686 len 4096 data: 2
6729 045/300 Write block 1866 len 4096 data: 3
6730 046/300 Write block 190 len 40960 data: 2 2 3 3 3 4 4 5 6 6
6731 047/300 Read block 2401 len 24576
6732 048/300 Flush
6733 049/300 Write block 255 len 28672 data: 3 3 3 3 2 2 2
6734 050/300 Read block 984 len 28672
6735 051/300 Write block 4315 len 24576 data: 3 2 2 5 5 6
6736 052/300 Write block 332 len 32768 data: 2 2 2 2 2 2 2 2
6737 053/300 Write block 2033 len 36864 data: 3 3 3 2 2 2 2 2 2
6738 054/300 Flush
6739 055/300 Write block 4370 len 8192 data: 3 3
6740 056/300 Read block 3947 len 12288
6741 057/300 Write block 4381 len 40960 data: 3 2 2 2 2 2 2 2 3 3
6742 058/300 Flush
6743 059/300 Read block 2565 len 40960
6744 060/300 Read block 4523 len 20480
6745 061/300 Flush
6746 062/300 Read block 80 len 24576
6747 063/300 Read block 3252 len 32768
6748 064/300 Read block 4033 len 12288
6749 065/300 Read block 2998 len 16384
6750 066/300 Write block 4521 len 32768 data: 2 2 2 2 3 2 2 2
6751 067/300 Read block 4705 len 24576
6752 068/300 Write block 344 len 12288 data: 2 2 2
6753 069/300 Write block 3001 len 4096 data: 3
6754 070/300 Write block 2085 len 12288 data: 2 3 2
6755 071/300 Read block 1734 len 4096
6756 072/300 Write block 2956 len 20480 data: 5 5 5 5 5
6757 073/300 Write block 1876 len 16384 data: 2 2 2 3
6758 074/300 Read block 3253 len 12288
6759 075/300 Write block 1116 len 20480 data: 2 2 2 2 2
6760 076/300 Read block 2871 len 16384
6761 077/300 Write block 2623 len 8192 data: 2 2
6762 078/300 Write block 687 len 24576 data: 3 2 2 2 2 2
6763 079/300 Read block 976 len 12288
6764 080/300 Write block 3622 len 24576 data: 2 2 2 2 2 2
6765 081/300 Write block 3831 len 24576 data: 4 4 5 4 4 4
6766 082/300 Read block 1650 len 28672
6767 083/300 Write block 4710 len 12288 data: 2 2 2
6768 084/300 Write block 3611 len 20480 data: 2 2 2 2 2
6769 085/300 Read block 1913 len 4096
6770 086/300 Read block 1257 len 4096
6771 087/300 Write block 4459 len 20480 data: 3 4 4 4 4
6772 088/300 Write block 3988 len 32768 data: 2 2 3 3 4 3 3 3
6773 089/300 Write block 4427 len 32768 data: 4 3 3 3 2 2 2 2
6774 090/300 Read block 3185 len 4096
6775 091/300 Flush
6776 092/300 Read block 4999 len 4096
6777 093/300 Read block 1190 len 8192
6778 094/300 Read block 3512 len 8192
6779 095/300 Read block 4537 len 24576
6780 096/300 Write block 1063 len 20480 data: 2 2 3 3 3
6781 097/300 Read block 4642 len 28672
6782 098/300 Read block 4309 len 36864
6783 099/300 Read block 4308 len 40960
6784 100/300 Write block 1132 len 8192 data: 2 2
6785 101/300 Read block 4825 len 20480
6786 102/300 Write block 40 len 20480 data: 2 2 2 2 2
6787 103/300 Read block 1037 len 8192
6788 104/300 Read block 1484 len 8192
6789 105/300 Read block 3300 len 4096
6790 106/300 Write block 2356 len 32768 data: 2 2 2 2 2 2 2 2
6791 107/300 Flush
6792 108/300 Write block 3082 len 40960 data: 4 4 4 4 3 3 3 3 3 3
6793 109/300 Read block 3612 len 28672
6794 110/300 Write block 2553 len 16384 data: 3 3 2 2
6795 111/300 Flush
6796 112/300 Write block 4940 len 12288 data: 3 3 4
6797 113/300 Write block 2318 len 28672 data: 2 2 2 2 2 2 2
6798 114/300 Read block 4146 len 16384
6799 115/300 Flush
6800 116/300 Read block 4825 len 16384
6801 117/300 Read block 3761 len 12288
6802 118/300 Read block 3552 len 20480
6803 119/300 Read block 4484 len 4096
6804 120/300 Read block 156 len 4096
6805 121/300 Write block 3518 len 16384 data: 2 2 2 2
6806 122/300 Read block 1022 len 12288
6807 123/300 Read block 1741 len 36864
6808 124/300 Read block 2422 len 40960
6809 125/300 Flush
6810 126/300 Read block 2255 len 8192
6811 127/300 Read block 488 len 24576
6812 128/300 Read block 1802 len 32768
6813 129/300 Write block 2316 len 12288 data: 2 2 3
6814 130/300 Read block 2204 len 16384
6815 131/300 Write block 470 len 36864 data: 2 2 2 2 2 2 2 2 3
6816 132/300 Read block 383 len 24576
6817 133/300 Write block 3560 len 40960 data: 3 2 2 2 2 2 2 2 3 3
6818 134/300 Flush
6819 135/300 Write block 524 len 28672 data: 4 4 3 3 4 6 6
6820 136/300 Read block 3151 len 16384
6821 137/300 Write block 4542 len 12288 data: 3 3 3
6822 138/300 Read block 1398 len 4096
6823 139/300 Read block 1166 len 32768
6824 140/300 Read block 3206 len 16384
6825 141/300 Write block 1188 len 32768 data: 2 2 2 2 2 2 2 2
6826 142/300 Write block 2297 len 28672 data: 5 4 4 4 4 3 3
6827 143/300 Read block 3780 len 16384
6828 144/300 Write block 3035 len 12288 data: 2 2 2
6829 145/300 Read block 1088 len 8192
6830 146/300 Read block 4861 len 40960
6831 147/300 Read block 2757 len 40960
6832 148/300 Flush
6833 149/300 Read block 102 len 32768
6834 150/300 Write block 241 len 24576 data: 3 3 3 3 3 5
6835 151/300 Write block 2791 len 8192 data: 2 3
6836 152/300 Read block 1366 len 8192
6837 153/300 Read block 4466 len 16384
6838 154/300 Read block 2698 len 12288
6839 155/300 Read block 3462 len 32768
6840 156/300 Read block 1911 len 40960
6841 157/300 Write block 622 len 4096 data: 4
6842 158/300 Flush
6843 159/300 Flush
6844 160/300 Write block 4279 len 12288 data: 3 3 3
6845 161/300 Write block 730 len 28672 data: 3 3 3 3 2 2 2
6846 162/300 Flush
6847 163/300 Write block 4372 len 8192 data: 3 3
6848 164/300 Write block 2857 len 40960 data: 2 2 3 3 3 3 3 3 3 3
6849 165/300 Read block 1681 len 40960
6850 166/300 Read block 1559 len 8192
6851 167/300 Flush
6852 168/300 Write block 1298 len 12288 data: 5 5 6
6853 169/300 Flush
6854 170/300 Write block 1213 len 24576 data: 2 2 2 2 2 3
6855 171/300 Write block 3089 len 28672 data: 4 4 4 4 3 4 4
6856 172/300 Write block 1660 len 28672 data: 3 3 4 4 4 4 3
6857 173/300 Read block 3381 len 28672
6858 174/300 Read block 730 len 36864
6859 175/300 Read block 808 len 32768
6860 176/300 Read block 131 len 40960
6861 177/300 Read block 286 len 12288
6862 178/300 Write block 4022 len 36864 data: 2 3 4 3 3 3 2 2 2
6863 179/300 Read block 4109 len 40960
6864 180/300 Read block 481 len 20480
6865 181/300 Write block 526 len 40960 data: 4 4 5 7 7 6 5 5 5 3
6866 182/300 Write block 528 len 40960 data: 6 8 8 7 6 6 6 4 3 3
6867 183/300 Read block 345 len 24576
6868 184/300 Read block 895 len 24576
6869 185/300 Write block 4090 len 24576 data: 4 3 3 3 3 2
6870 186/300 Write block 4031 len 24576 data: 3 2 3 3 4 3
6871 187/300 Read block 2314 len 40960
6872 188/300 Read block 3318 len 20480
6873 189/300 Read block 3134 len 24576
6874 190/300 Read block 3742 len 24576
6875 191/300 Write block 2720 len 24576 data: 4 3 5 5 5 5
6876 192/300 Flush
6877 193/300 Read block 866 len 32768
6878 194/300 Write block 4595 len 8192 data: 2 2
6879 195/300 Read block 3742 len 16384
6880 196/300 Read block 4062 len 32768
6881 197/300 Read block 670 len 12288
6882 198/300 Write block 4808 len 28672 data: 3 3 3 3 3 4 4
6883 199/300 Read block 1219 len 8192
6884 200/300 Write block 3197 len 24576 data: 3 3 3 3 4 3
6885 201/300 Read block 2572 len 28672
6886 202/300 Read block 2326 len 40960
6887 203/300 Read block 4808 len 32768
6888 204/300 Write block 2131 len 28672 data: 2 2 2 2 2 2 2
6889 205/300 Write block 3733 len 20480 data: 3 3 3 3 4
6890 206/300 Flush
6891 207/300 Read block 227 len 4096
6892 208/300 Write block 3226 len 40960 data: 3 3 3 4 3 2 2 3 2 2
6893 209/300 Write block 4588 len 36864 data: 2 2 2 2 2 2 2 3 3
6894 210/300 Write block 3074 len 16384 data: 2 2 3 3
6895 211/300 Write block 859 len 36864 data: 3 4 3 3 3 3 2 2 2
6896 212/300 Read block 2950 len 8192
6897 213/300 Read block 1042 len 28672
6898 214/300 Write block 862 len 8192 data: 4 4
6899 215/300 Write block 4549 len 20480 data: 3 3 2 2 2
6900 216/300 Read block 1458 len 24576
6901 217/300 Flush
6902 218/300 Read block 3412 len 24576
6903 219/300 Read block 4392 len 36864
6904 220/300 Read block 4528 len 28672
6905 221/300 Write block 267 len 20480 data: 2 2 2 2 2
6906 222/300 Read block 3557 len 8192
6907 223/300 Write block 2091 len 32768 data: 2 4 4 4 4 4 3 2
6908 224/300 Write block 824 len 4096 data: 2
6909 225/300 Read block 1605 len 28672
6910 226/300 Read block 2137 len 4096
6911 227/300 Write block 1560 len 12288 data: 3 3 3
6912 228/300 Read block 2191 len 40960
6913 229/300 Read block 1850 len 8192
6914 230/300 Flush
6915 231/300 Read block 3334 len 16384
6916 232/300 Read block 584 len 24576
6917 233/300 Write block 4578 len 40960 data: 3 3 3 3 3 3 3 3 2 2
6918 234/300 Flush
6919 235/300 Read block 2456 len 8192
6920 236/300 Write block 182 len 12288 data: 4 4 3
6921 237/300 Read block 1139 len 16384
6922 238/300 Write block 1436 len 12288 data: 7 6 5
6923 239/300 Read block 1085 len 16384
6924 240/300 Write block 4608 len 4096 data: 2
6925 241/300 Read block 1889 len 40960
6926 242/300 Read block 3879 len 36864
6927 243/300 Read block 4612 len 4096
6928 244/300 Read block 3942 len 24576
6929 245/300 Write block 1650 len 36864 data: 2 2 2 2 2 2 2 2 2
6930 246/300 Read block 4492 len 20480
6931 247/300 Read block 4952 len 24576
6932 248/300 Write block 974 len 36864 data: 3 4 3 3 2 2 2 2 3
6933 249/300 Read block 1574 len 28672
6934 250/300 Read block 477 len 12288
6935 251/300 Write block 2987 len 24576 data: 2 2 2 2 2 2
6936 252/300 Flush
6937 253/300 Write block 1942 len 36864 data: 4 4 4 4 4 3 3 3 2
6938 254/300 Read block 4241 len 36864
6939 255/300 Write block 3956 len 12288 data: 2 2 2
6940 256/300 Read block 4286 len 36864
6941 257/300 Write block 927 len 4096 data: 4
6942 258/300 Read block 581 len 36864
6943 259/300 Read block 4571 len 8192
6944 260/300 Write block 2866 len 32768 data: 4 4 4 3 3 4 2 3
6945 261/300 Read block 2576 len 20480
6946 262/300 Write block 3979 len 4096 data: 2
6947 263/300 Write block 4107 len 40960 data: 4 3 3 4 4 3 3 3 3 4
6948 264/300 Write block 3437 len 24576 data: 3 3 3 3 5 3
6949 265/300 Read block 3764 len 36864
6950 266/300 Read block 663 len 40960
6951 267/300 Write block 1608 len 20480 data: 2 2 2 2 2
6952 268/300 Write block 1605 len 24576 data: 2 2 2 3 3 3
6953 269/300 Read block 3728 len 8192
6954 270/300 Read block 588 len 8192
6955 271/300 Read block 2633 len 28672
6956 272/300 Read block 3405 len 20480
6957 273/300 Write block 946 len 8192 data: 2 2
6958 274/300 Write block 3381 len 28672 data: 2 2 3 3 3 3 2
6959 275/300 Write block 3274 len 16384 data: 2 2 2 2
6960 276/300 Write block 115 len 36864 data: 3 3 3 3 2 4 4 5 5
6961 277/300 Read block 2595 len 40960
6962 278/300 Read block 2754 len 16384
6963 279/300 Read block 1864 len 4096
6964 280/300 Write block 4033 len 32768 data: 4 4 5 4 3 3 3 2
6965 281/300 Write block 3305 len 24576 data: 2 2 2 3 3 3
6966 282/300 Write block 4165 len 12288 data: 3 3 3
6967 283/300 Read block 3145 len 40960
6968 284/300 Read block 4500 len 28672
6969 285/300 Read block 29 len 16384
6970 286/300 Read block 1721 len 36864
6971 287/300 Read block 3809 len 32768
6972 288/300 Read block 1714 len 16384
6973 289/300 Read block 3220 len 32768
6974 290/300 Write block 3181 len 36864 data: 2 2 3 4 3 3 4 4 3
6975 291/300 Read block 526 len 4096
6976 292/300 Read block 4084 len 36864
6977 293/300 Write block 1348 len 32768 data: 3 3 3 3 3 3 3 2
6978 294/300 Write block 4848 len 32768 data: 2 2 2 2 2 2 2 2
6979 295/300 Read block 3080 len 24576
6980 296/300 Flush
6981 297/300 Read block 1435 len 16384
6982 298/300 Read block 806 len 40960
6983 299/300 Read block 3219 len 16384
6984 300/300 Flush
6985 301/300 Flush
6986 Main task has work to do, go find it
6987 got dsc Start(1)
6988 start 1
6989 Nov 01 16:51:58.325 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/1, method: POST, req_id: a7543f19-3d23-47b5-bd21-31dc117ad9d1, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
6990 Replay: started 1, returned:Ok(())
6991 ----------------------------------------------------------------
6992 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6993 States: Active Offline Active
6994 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6995 2813 Acked Write 4 Done Sent Done false
6996 2814 Acked Read 1 Done Sent Done false
6997 2815 Acked Read 10 Done Sent Done false
6998 2816 Acked Write 8 Done Sent Done false
6999 2817 Acked Write 10 Done Sent Done false
7000 2818 Acked Read 8 Done Sent Done false
7001 2819 Acked Flush 0 Done Sent Done false
7002 2820 Acked Write 1 Done Sent Done false
7003 2821 Acked Flush 0 Done Sent Done false
7004 2822 Acked Read 7 Done Sent Done false
7005 2823 Acked Flush 0 Done Sent Done false
7006 2824 Acked Read 2 Done Sent Done false
7007 2825 Acked Read 9 Done Sent Done false
7008 2826 Acked Write 5 Done Sent Done false
7009 2827 Acked Read 3 Done Sent Done false
7010 2828 Acked Read 8 Done Sent Done false
7011 2829 Acked Write 8 Done Sent Done false
7012 2830 Acked Read 2 Done Sent Done false
7013 2831 Acked Write 8 Done Sent Done false
7014 2832 Acked Write 6 Done Sent Done false
7015 2833 Acked Read 5 Done Sent Done false
7016 2834 Acked Read 7 Done Sent Done false
7017 2835 Acked Write 3 Done Sent Done false
7018 2836 Acked Write 6 Done Sent Done false
7019 2837 Acked Read 1 Done Sent Done false
7020 2838 Acked Read 4 Done Sent Done false
7021 2839 Acked Read 9 Done Sent Done false
7022 2840 Acked Read 4 Done Sent Done false
7023 2841 Acked Read 10 Done Sent Done false
7024 2842 Acked Read 9 Done Sent Done false
7025 2843 Acked Read 1 Done Sent Done false
7026 2844 Acked Read 8 Done Sent Done false
7027 2845 Acked Read 8 Done Sent Done false
7028 2846 Acked Read 10 Done Sent Done false
7029 2847 Acked Read 9 Done Sent Done false
7030 2848 Acked Read 8 Done Sent Done false
7031 2849 Acked Write 9 Done Sent Done false
7032 2850 Acked Read 10 Done Sent Done false
7033 2851 Acked Write 2 Done Sent Done false
7034 2852 Acked Read 7 Done Sent Done false
7035 2853 Acked Read 2 Done Sent Done false
7036 2854 Acked Read 1 Done Sent Done false
7037 2855 Acked Write 7 Done Sent Done false
7038 2856 Acked Write 1 Done Sent Done false
7039 2857 Acked Write 1 Done Sent Done false
7040 2858 Acked Write 10 Done Sent Done false
7041 2859 Acked Read 6 Done Sent Done false
7042 2860 Acked Flush 0 Done Sent Done false
7043 2861 Acked Write 7 Done Sent Done false
7044 2862 Acked Read 7 Done Sent Done false
7045 2863 Acked Write 6 Done Sent Done false
7046 2864 Acked Write 8 Done Sent Done false
7047 2865 Acked Write 9 Done Sent Done false
7048 2866 Acked Flush 0 Done Sent Done false
7049 2867 Acked Write 2 Done Sent Done false
7050 2868 Acked Read 3 Done Sent Done false
7051 2869 Acked Write 10 Done Sent Done false
7052 2870 Acked Flush 0 Done Sent Done false
7053 2871 Acked Read 10 Done Sent Done false
7054 2872 Acked Read 5 Done Sent Done false
7055 2873 Acked Flush 0 Done Sent Done false
7056 2874 Acked Read 6 Done Sent Done false
7057 2875 Acked Read 8 Done Sent Done false
7058 2876 Acked Read 3 Done Sent Done false
7059 2877 Acked Read 4 Done Sent Done false
7060 2878 Acked Write 8 Done Sent Done false
7061 2879 Acked Read 6 Done Sent Done false
7062 2880 Acked Write 3 Done Sent Done false
7063 2881 Acked Write 1 Done Sent Done false
7064 2882 Acked Write 3 Done Sent Done false
7065 2883 Acked Read 1 Done Sent Done false
7066 2884 Acked Write 5 Done Sent Done false
7067 2885 Acked Write 4 Done Sent Done false
7068 2886 Acked Read 3 Done Sent Done false
7069 2887 Acked Write 5 Done Sent Done false
7070 2888 Acked Read 4 Done Sent Done false
7071 2889 Acked Write 2 Done Sent Done false
7072 2890 Acked Write 6 Done Sent Done false
7073 2891 Acked Read 3 Done Sent Done false
7074 2892 Acked Write 6 Done Sent Done false
7075 2893 Acked Write 6 Done Sent Done false
7076 2894 Acked Read 7 Done Sent Done false
7077 2895 Acked Write 3 Done Sent Done false
7078 2896 Acked Write 5 Done Sent Done false
7079 2897 Acked Read 1 Done Sent Done false
7080 2898 Acked Read 1 Done Sent Done false
7081 2899 Acked Write 5 Done Sent Done false
7082 2900 Acked Write 8 Done Sent Done false
7083 2901 Acked Write 8 Done Sent Done false
7084 2902 Acked Read 1 Done Sent Done false
7085 2903 Acked Flush 0 Done Sent Done false
7086 2904 Acked Read 1 Done Sent Done false
7087 2905 Acked Read 2 Done Sent Done false
7088 2906 Acked Read 2 Done Sent Done false
7089 2907 Acked Read 6 Done Sent Done false
7090 2908 Acked Write 5 Done Sent Done false
7091 2909 Acked Read 7 Done Sent Done false
7092 2910 Acked Read 9 Done Sent Done false
7093 2911 Acked Read 10 Done Sent Done false
7094 2912 Acked Write 2 Done Sent Done false
7095 2913 Acked Read 5 Done Sent Done false
7096 2914 Acked Write 5 Done Sent Done false
7097 2915 Acked Read 2 Done Sent Done false
7098 2916 Acked Read 2 Done Sent Done false
7099 2917 Acked Read 1 Done Sent Done false
7100 2918 Acked Write 8 Done Sent Done false
7101 2919 Acked Flush 0 Done Sent Done false
7102 2920 Acked Write 10 Done Sent Done false
7103 2921 Acked Read 7 Done Sent Done false
7104 2922 Acked Write 4 Done Sent Done false
7105 2923 Acked Flush 0 Done Sent Done false
7106 2924 Acked Write 3 Done Sent Done false
7107 2925 Acked Write 7 Done Sent Done false
7108 2926 Acked Read 4 Done Sent Done false
7109 2927 Acked Flush 0 Done Sent Done false
7110 2928 Acked Read 4 Done Sent Done false
7111 2929 Acked Read 3 Done Sent Done false
7112 2930 Acked Read 5 Done Sent Done false
7113 2931 Acked Read 1 Done Sent Done false
7114 2932 Acked Read 1 Done Sent Done false
7115 2933 Acked Write 4 Done Sent Done false
7116 2934 Acked Read 3 Done Sent Done false
7117 2935 Acked Read 9 Done Sent Done false
7118 2936 Acked Read 10 Done Sent Done false
7119 2937 Acked Flush 0 Done Sent Done false
7120 2938 Acked Read 2 Done Sent Done false
7121 2939 Acked Read 6 Done Sent Done false
7122 2940 Acked Read 8 Done Sent Done false
7123 2941 Acked Write 3 Done Sent Done false
7124 2942 Acked Read 4 Done Sent Done false
7125 2943 Acked Write 9 Done Sent Done false
7126 2944 Acked Read 6 Done Sent Done false
7127 2945 Acked Write 10 Done Sent Done false
7128 2946 Acked Flush 0 Done Sent Done false
7129 2947 Acked Write 7 Done Sent Done false
7130 2948 Acked Read 4 Done Sent Done false
7131 2949 Acked Write 3 Done Sent Done false
7132 2950 Acked Read 1 Done Sent Done false
7133 2951 Acked Read 8 Done Sent Done false
7134 2952 Acked Read 4 Done Sent Done false
7135 2953 Acked Write 8 Done Sent Done false
7136 2954 Acked Write 7 Done Sent Done false
7137 2955 Acked Read 4 Done Sent Done false
7138 2956 Acked Write 3 Done Sent Done false
7139 2957 Acked Read 2 Done Sent Done false
7140 2958 Acked Read 10 Done Sent Done false
7141 2959 Acked Read 10 Done Sent Done false
7142 2960 Acked Flush 0 Done Sent Done false
7143 2961 Acked Read 8 Done Sent Done false
7144 2962 Acked Write 6 Done Sent Done false
7145 2963 Acked Write 2 Done Sent Done false
7146 2964 Acked Read 2 Done Sent Done false
7147 2965 Acked Read 4 Done Sent Done false
7148 2966 Acked Read 3 Done Sent Done false
7149 2967 Acked Read 8 Done Sent Done false
7150 2968 Acked Read 10 Done Sent Done false
7151 2969 Acked Write 1 Done Sent Done false
7152 2970 Acked Flush 0 Done Sent Done false
7153 2971 Acked Flush 0 Done Sent Done false
7154 2972 Acked Write 3 Done Sent Done false
7155 2973 Acked Write 7 Done Sent Done false
7156 2974 Acked Flush 0 Done Sent Done false
7157 2975 Acked Write 2 Done Sent Done false
7158 2976 Acked Write 10 Done Sent Done false
7159 2977 Acked Read 10 Done Sent Done false
7160 2978 Acked Read 2 Done Sent Done false
7161 2979 Acked Flush 0 Done Sent Done false
7162 2980 Acked Write 3 Done Sent Done false
7163 2981 Acked Flush 0 Done Sent Done false
7164 2982 Acked Write 6 Done Sent Done false
7165 2983 Acked Write 7 Done Sent Done false
7166 2984 Acked Write 7 Done Sent Done false
7167 2985 Acked Read 7 Done Sent Done false
7168 2986 Acked Read 9 Done Sent Done false
7169 2987 Acked Read 8 Done Sent Done false
7170 2988 Acked Read 10 Done Sent Done false
7171 2989 Acked Read 3 Done Sent Done false
7172 2990 Acked Write 9 Done Sent Done false
7173 2991 Acked Read 10 Done Sent Done false
7174 2992 Acked Read 5 Done Sent Done false
7175 2993 Acked Write 10 Done Sent Done false
7176 2994 Acked Write 10 Done Sent Done false
7177 2995 Acked Read 6 Done Sent Done false
7178 2996 Acked Read 6 Done Sent Done false
7179 2997 Acked Write 6 Done Sent Done false
7180 2998 Acked Write 6 Done Sent Done false
7181 2999 Acked Read 10 Done Sent Done false
7182 3000 Acked Read 5 Done Sent Done false
7183 3001 Acked Read 6 Done Sent Done false
7184 3002 Acked Read 6 Done Sent Done false
7185 3003 Acked Write 6 Done Sent Done false
7186 3004 Acked Flush 0 Done Sent Done false
7187 3005 Acked Read 8 Done Sent Done false
7188 3006 Acked Write 2 Done Sent Done false
7189 3007 Acked Read 4 Done Sent Done false
7190 3008 Acked Read 8 Done Sent Done false
7191 3009 Acked Read 3 Done Sent Done false
7192 3010 Acked Write 7 Done Sent Done false
7193 3011 Acked Read 2 Done Sent Done false
7194 3012 Acked Write 6 Done Sent Done false
7195 3013 Acked Read 7 Done Sent Done false
7196 3014 Acked Read 10 Done Sent Done false
7197 3015 Acked Read 8 Done Sent Done false
7198 3016 Acked Write 7 Done Sent Done false
7199 3017 Acked Write 5 Done Sent Done false
7200 3018 Acked Flush 0 Done Sent Done false
7201 3019 Acked Read 1 Done Sent Done false
7202 3020 Acked Write 10 Done Sent Done false
7203 3021 Acked Write 9 Done Sent Done false
7204 3022 Acked Write 4 Done Sent Done false
7205 3023 Acked Write 9 Done Sent Done false
7206 3024 Acked Read 2 Done Sent Done false
7207 3025 Acked Read 7 Done Sent Done false
7208 3026 Acked Write 2 Done Sent Done false
7209 3027 Acked Write 5 Done Sent Done false
7210 3028 Acked Read 6 Done Sent Done false
7211 3029 Acked Flush 0 Done Sent Done false
7212 3030 Acked Read 6 Done Sent Done false
7213 3031 Acked Read 9 Done Sent Done false
7214 3032 Acked Read 7 Done Sent Done false
7215 3033 Acked Write 5 Done Sent Done false
7216 3034 Acked Read 2 Done Sent Done false
7217 3035 Acked Write 8 Done Sent Done false
7218 3036 Acked Write 1 Done Sent Done false
7219 3037 Acked Read 7 Done Sent Done false
7220 3038 Acked Read 1 Done Sent Done false
7221 3039 Acked Write 3 Done Sent Done false
7222 3040 Acked Read 10 Done Sent Done false
7223 3041 Acked Read 2 Done Sent Done false
7224 3042 Acked Flush 0 Done Sent Done false
7225 3043 Acked Read 4 Done Sent Done false
7226 3044 Acked Read 6 Done Sent Done false
7227 3045 Acked Write 10 Done Sent Done false
7228 3046 Acked Flush 0 Done Sent Done false
7229 3047 Acked Read 2 Done Sent Done false
7230 3048 Acked Write 3 Done Sent Done false
7231 3049 Acked Read 4 Done Sent Done false
7232 3050 Acked Write 3 Done Sent Done false
7233 3051 Acked Read 4 Done Sent Done false
7234 3052 Acked Write 1 Done Sent Done false
7235 3053 Acked Read 10 Done Sent Done false
7236 3054 Acked Read 9 Done Sent Done false
7237 3055 Acked Read 1 Done Sent Done false
7238 3056 Acked Read 6 Done Sent Done false
7239 3057 Acked Write 9 Done Sent Done false
7240 3058 Acked Read 5 Done Sent Done false
7241 3059 Acked Read 6 Done Sent Done false
7242 3060 Acked Write 9 Done Sent Done false
7243 3061 Acked Read 7 Done Sent Done false
7244 3062 Acked Read 3 Done Sent Done false
7245 3063 Acked Write 6 Done Sent Done false
7246 3064 Acked Flush 0 Done Sent Done false
7247 3065 Acked Write 9 Done Sent Done false
7248 3066 Acked Read 9 Done Sent Done false
7249 3067 Acked Write 3 Done Sent Done false
7250 3068 Acked Read 9 Done Sent Done false
7251 3069 Acked Write 1 Done Sent Done false
7252 3070 Acked Read 9 Done Sent Done false
7253 3071 Acked Read 2 Done Sent Done false
7254 3072 Acked Write 8 Done Sent Done false
7255 3073 Acked Read 5 Done Sent Done false
7256 3074 Acked Write 1 Done Sent Done false
7257 3075 Acked Write 10 Done Sent Done false
7258 3076 Acked Write 6 Done Sent Done false
7259 3077 Acked Read 9 Done Sent Done false
7260 3078 Acked Read 10 Done Sent Done false
7261 3079 Acked Write 5 Done Sent Done false
7262 3080 Acked Write 6 Done Sent Done false
7263 3081 Acked Read 2 Done Sent Done false
7264 3082 Acked Read 2 Done Sent Done false
7265 3083 Acked Read 7 Done Sent Done false
7266 3084 Acked Read 5 Done Sent Done false
7267 3085 Acked Write 2 Done Sent Done false
7268 3086 Acked Write 7 Done Sent Done false
7269 3087 Acked Write 4 Done Sent Done false
7270 3088 Acked Write 9 Done Sent Done false
7271 3089 Acked Read 10 Done Sent Done false
7272 3090 Acked Read 4 Done Sent Done false
7273 3091 Acked Read 1 Done Sent Done false
7274 3092 Acked Write 8 Done Sent Done false
7275 3093 Acked Write 6 Done Sent Done false
7276 3094 Acked Write 3 Done Sent Done false
7277 3095 Acked Read 10 Done Sent Done false
7278 3096 Acked Read 7 Done Sent Done false
7279 3097 Acked Read 4 Done Sent Done false
7280 3098 Acked Read 9 Done Sent Done false
7281 3099 Acked Read 8 Done Sent Done false
7282 3100 Acked Read 4 Done Sent Done false
7283 3101 Acked Read 8 Done Sent Done false
7284 3102 Acked Write 9 Done Sent Done false
7285 3103 Acked Read 1 Done Sent Done false
7286 3104 Acked Read 9 Done Sent Done false
7287 3105 Acked Write 8 Done Sent Done false
7288 3106 Acked Write 8 Done Sent Done false
7289 3107 Acked Read 6 Done Sent Done false
7290 3108 Acked Flush 0 Done Sent Done false
7291 3109 Acked Read 4 Done Sent Done false
7292 3110 Acked Read 10 Done Sent Done false
7293 3111 Acked Read 4 Done Sent Done false
7294 3112 Acked Flush 0 Done Sent Done false
7295 3113 Acked Flush 0 Done Sent Done false
7296 STATES DS:0 DS:1 DS:2 TOTAL
7297 Sent 0 301 0 301
7298 Done 301 0 301 602
7299 Skipped 0 0 0 0
7300 Error 0 0 0 0
7301 Last Flush: 3113 2812 3113
7302 Downstairs last five completed: 2812 2811 2810 2809 2808
7303 Upstairs last five completed: 3113 3112 3111 3110 3109
7304 CLIENT: Up:0 ds:301 act:2
7305 [8820] got start action
7306 [8820] I am going to restart
7307 Starting downstairs at port 8820
7308 Make output file at "/tmp/dsc/downstairs-8820.txt"
7309 [8820][1] reports Starting
7310 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1205)
7311 [8820][1] reports Running
7312 ----------------------------------------------------------------
7313 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7314 States: Active Offline Active
7315 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7316 2813 Acked Write 4 Done Sent Done false
7317 2814 Acked Read 1 Done Sent Done false
7318 2815 Acked Read 10 Done Sent Done false
7319 2816 Acked Write 8 Done Sent Done false
7320 2817 Acked Write 10 Done Sent Done false
7321 2818 Acked Read 8 Done Sent Done false
7322 2819 Acked Flush 0 Done Sent Done false
7323 2820 Acked Write 1 Done Sent Done false
7324 2821 Acked Flush 0 Done Sent Done false
7325 2822 Acked Read 7 Done Sent Done false
7326 2823 Acked Flush 0 Done Sent Done false
7327 2824 Acked Read 2 Done Sent Done false
7328 2825 Acked Read 9 Done Sent Done false
7329 2826 Acked Write 5 Done Sent Done false
7330 2827 Acked Read 3 Done Sent Done false
7331 2828 Acked Read 8 Done Sent Done false
7332 2829 Acked Write 8 Done Sent Done false
7333 2830 Acked Read 2 Done Sent Done false
7334 2831 Acked Write 8 Done Sent Done false
7335 2832 Acked Write 6 Done Sent Done false
7336 2833 Acked Read 5 Done Sent Done false
7337 2834 Acked Read 7 Done Sent Done false
7338 2835 Acked Write 3 Done Sent Done false
7339 2836 Acked Write 6 Done Sent Done false
7340 2837 Acked Read 1 Done Sent Done false
7341 2838 Acked Read 4 Done Sent Done false
7342 2839 Acked Read 9 Done Sent Done false
7343 2840 Acked Read 4 Done Sent Done false
7344 2841 Acked Read 10 Done Sent Done false
7345 2842 Acked Read 9 Done Sent Done false
7346 2843 Acked Read 1 Done Sent Done false
7347 2844 Acked Read 8 Done Sent Done false
7348 2845 Acked Read 8 Done Sent Done false
7349 2846 Acked Read 10 Done Sent Done false
7350 2847 Acked Read 9 Done Sent Done false
7351 2848 Acked Read 8 Done Sent Done false
7352 2849 Acked Write 9 Done Sent Done false
7353 2850 Acked Read 10 Done Sent Done false
7354 2851 Acked Write 2 Done Sent Done false
7355 2852 Acked Read 7 Done Sent Done false
7356 2853 Acked Read 2 Done Sent Done false
7357 2854 Acked Read 1 Done Sent Done false
7358 2855 Acked Write 7 Done Sent Done false
7359 2856 Acked Write 1 Done Sent Done false
7360 2857 Acked Write 1 Done Sent Done false
7361 2858 Acked Write 10 Done Sent Done false
7362 2859 Acked Read 6 Done Sent Done false
7363 2860 Acked Flush 0 Done Sent Done false
7364 2861 Acked Write 7 Done Sent Done false
7365 2862 Acked Read 7 Done Sent Done false
7366 2863 Acked Write 6 Done Sent Done false
7367 2864 Acked Write 8 Done Sent Done false
7368 2865 Acked Write 9 Done Sent Done false
7369 2866 Acked Flush 0 Done Sent Done false
7370 2867 Acked Write 2 Done Sent Done false
7371 2868 Acked Read 3 Done Sent Done false
7372 2869 Acked Write 10 Done Sent Done false
7373 2870 Acked Flush 0 Done Sent Done false
7374 2871 Acked Read 10 Done Sent Done false
7375 2872 Acked Read 5 Done Sent Done false
7376 2873 Acked Flush 0 Done Sent Done false
7377 2874 Acked Read 6 Done Sent Done false
7378 2875 Acked Read 8 Done Sent Done false
7379 2876 Acked Read 3 Done Sent Done false
7380 2877 Acked Read 4 Done Sent Done false
7381 2878 Acked Write 8 Done Sent Done false
7382 2879 Acked Read 6 Done Sent Done false
7383 2880 Acked Write 3 Done Sent Done false
7384 2881 Acked Write 1 Done Sent Done false
7385 2882 Acked Write 3 Done Sent Done false
7386 2883 Acked Read 1 Done Sent Done false
7387 2884 Acked Write 5 Done Sent Done false
7388 2885 Acked Write 4 Done Sent Done false
7389 2886 Acked Read 3 Done Sent Done false
7390 2887 Acked Write 5 Done Sent Done false
7391 2888 Acked Read 4 Done Sent Done false
7392 2889 Acked Write 2 Done Sent Done false
7393 2890 Acked Write 6 Done Sent Done false
7394 2891 Acked Read 3 Done Sent Done false
7395 2892 Acked Write 6 Done Sent Done false
7396 2893 Acked Write 6 Done Sent Done false
7397 2894 Acked Read 7 Done Sent Done false
7398 2895 Acked Write 3 Done Sent Done false
7399 2896 Acked Write 5 Done Sent Done false
7400 2897 Acked Read 1 Done Sent Done false
7401 2898 Acked Read 1 Done Sent Done false
7402 2899 Acked Write 5 Done Sent Done false
7403 2900 Acked Write 8 Done Sent Done false
7404 2901 Acked Write 8 Done Sent Done false
7405 2902 Acked Read 1 Done Sent Done false
7406 2903 Acked Flush 0 Done Sent Done false
7407 2904 Acked Read 1 Done Sent Done false
7408 2905 Acked Read 2 Done Sent Done false
7409 2906 Acked Read 2 Done Sent Done false
7410 2907 Acked Read 6 Done Sent Done false
7411 2908 Acked Write 5 Done Sent Done false
7412 2909 Acked Read 7 Done Sent Done false
7413 2910 Acked Read 9 Done Sent Done false
7414 2911 Acked Read 10 Done Sent Done false
7415 2912 Acked Write 2 Done Sent Done false
7416 2913 Acked Read 5 Done Sent Done false
7417 2914 Acked Write 5 Done Sent Done false
7418 2915 Acked Read 2 Done Sent Done false
7419 2916 Acked Read 2 Done Sent Done false
7420 2917 Acked Read 1 Done Sent Done false
7421 2918 Acked Write 8 Done Sent Done false
7422 2919 Acked Flush 0 Done Sent Done false
7423 2920 Acked Write 10 Done Sent Done false
7424 2921 Acked Read 7 Done Sent Done false
7425 2922 Acked Write 4 Done Sent Done false
7426 2923 Acked Flush 0 Done Sent Done false
7427 2924 Acked Write 3 Done Sent Done false
7428 2925 Acked Write 7 Done Sent Done false
7429 2926 Acked Read 4 Done Sent Done false
7430 2927 Acked Flush 0 Done Sent Done false
7431 2928 Acked Read 4 Done Sent Done false
7432 2929 Acked Read 3 Done Sent Done false
7433 2930 Acked Read 5 Done Sent Done false
7434 2931 Acked Read 1 Done Sent Done false
7435 2932 Acked Read 1 Done Sent Done false
7436 2933 Acked Write 4 Done Sent Done false
7437 2934 Acked Read 3 Done Sent Done false
7438 2935 Acked Read 9 Done Sent Done false
7439 2936 Acked Read 10 Done Sent Done false
7440 2937 Acked Flush 0 Done Sent Done false
7441 2938 Acked Read 2 Done Sent Done false
7442 2939 Acked Read 6 Done Sent Done false
7443 2940 Acked Read 8 Done Sent Done false
7444 2941 Acked Write 3 Done Sent Done false
7445 2942 Acked Read 4 Done Sent Done false
7446 2943 Acked Write 9 Done Sent Done false
7447 2944 Acked Read 6 Done Sent Done false
7448 2945 Acked Write 10 Done Sent Done false
7449 2946 Acked Flush 0 Done Sent Done false
7450 2947 Acked Write 7 Done Sent Done false
7451 2948 Acked Read 4 Done Sent Done false
7452 2949 Acked Write 3 Done Sent Done false
7453 2950 Acked Read 1 Done Sent Done false
7454 2951 Acked Read 8 Done Sent Done false
7455 2952 Acked Read 4 Done Sent Done false
7456 2953 Acked Write 8 Done Sent Done false
7457 2954 Acked Write 7 Done Sent Done false
7458 2955 Acked Read 4 Done Sent Done false
7459 2956 Acked Write 3 Done Sent Done false
7460 2957 Acked Read 2 Done Sent Done false
7461 2958 Acked Read 10 Done Sent Done false
7462 2959 Acked Read 10 Done Sent Done false
7463 2960 Acked Flush 0 Done Sent Done false
7464 2961 Acked Read 8 Done Sent Done false
7465 2962 Acked Write 6 Done Sent Done false
7466 2963 Acked Write 2 Done Sent Done false
7467 2964 Acked Read 2 Done Sent Done false
7468 2965 Acked Read 4 Done Sent Done false
7469 2966 Acked Read 3 Done Sent Done false
7470 2967 Acked Read 8 Done Sent Done false
7471 2968 Acked Read 10 Done Sent Done false
7472 2969 Acked Write 1 Done Sent Done false
7473 2970 Acked Flush 0 Done Sent Done false
7474 2971 Acked Flush 0 Done Sent Done false
7475 2972 Acked Write 3 Done Sent Done false
7476 2973 Acked Write 7 Done Sent Done false
7477 2974 Acked Flush 0 Done Sent Done false
7478 2975 Acked Write 2 Done Sent Done false
7479 2976 Acked Write 10 Done Sent Done false
7480 2977 Acked Read 10 Done Sent Done false
7481 2978 Acked Read 2 Done Sent Done false
7482 2979 Acked Flush 0 Done Sent Done false
7483 2980 Acked Write 3 Done Sent Done false
7484 2981 Acked Flush 0 Done Sent Done false
7485 2982 Acked Write 6 Done Sent Done false
7486 2983 Acked Write 7 Done Sent Done false
7487 2984 Acked Write 7 Done Sent Done false
7488 2985 Acked Read 7 Done Sent Done false
7489 2986 Acked Read 9 Done Sent Done false
7490 2987 Acked Read 8 Done Sent Done false
7491 2988 Acked Read 10 Done Sent Done false
7492 2989 Acked Read 3 Done Sent Done false
7493 2990 Acked Write 9 Done Sent Done false
7494 2991 Acked Read 10 Done Sent Done false
7495 2992 Acked Read 5 Done Sent Done false
7496 2993 Acked Write 10 Done Sent Done false
7497 2994 Acked Write 10 Done Sent Done false
7498 2995 Acked Read 6 Done Sent Done false
7499 2996 Acked Read 6 Done Sent Done false
7500 2997 Acked Write 6 Done Sent Done false
7501 2998 Acked Write 6 Done Sent Done false
7502 2999 Acked Read 10 Done Sent Done false
7503 3000 Acked Read 5 Done Sent Done false
7504 3001 Acked Read 6 Done Sent Done false
7505 3002 Acked Read 6 Done Sent Done false
7506 3003 Acked Write 6 Done Sent Done false
7507 3004 Acked Flush 0 Done Sent Done false
7508 3005 Acked Read 8 Done Sent Done false
7509 3006 Acked Write 2 Done Sent Done false
7510 3007 Acked Read 4 Done Sent Done false
7511 3008 Acked Read 8 Done Sent Done false
7512 3009 Acked Read 3 Done Sent Done false
7513 3010 Acked Write 7 Done Sent Done false
7514 3011 Acked Read 2 Done Sent Done false
7515 3012 Acked Write 6 Done Sent Done false
7516 3013 Acked Read 7 Done Sent Done false
7517 3014 Acked Read 10 Done Sent Done false
7518 3015 Acked Read 8 Done Sent Done false
7519 3016 Acked Write 7 Done Sent Done false
7520 3017 Acked Write 5 Done Sent Done false
7521 3018 Acked Flush 0 Done Sent Done false
7522 3019 Acked Read 1 Done Sent Done false
7523 3020 Acked Write 10 Done Sent Done false
7524 3021 Acked Write 9 Done Sent Done false
7525 3022 Acked Write 4 Done Sent Done false
7526 3023 Acked Write 9 Done Sent Done false
7527 3024 Acked Read 2 Done Sent Done false
7528 3025 Acked Read 7 Done Sent Done false
7529 3026 Acked Write 2 Done Sent Done false
7530 3027 Acked Write 5 Done Sent Done false
7531 3028 Acked Read 6 Done Sent Done false
7532 3029 Acked Flush 0 Done Sent Done false
7533 3030 Acked Read 6 Done Sent Done false
7534 3031 Acked Read 9 Done Sent Done false
7535 3032 Acked Read 7 Done Sent Done false
7536 3033 Acked Write 5 Done Sent Done false
7537 3034 Acked Read 2 Done Sent Done false
7538 3035 Acked Write 8 Done Sent Done false
7539 3036 Acked Write 1 Done Sent Done false
7540 3037 Acked Read 7 Done Sent Done false
7541 3038 Acked Read 1 Done Sent Done false
7542 3039 Acked Write 3 Done Sent Done false
7543 3040 Acked Read 10 Done Sent Done false
7544 3041 Acked Read 2 Done Sent Done false
7545 3042 Acked Flush 0 Done Sent Done false
7546 3043 Acked Read 4 Done Sent Done false
7547 3044 Acked Read 6 Done Sent Done false
7548 3045 Acked Write 10 Done Sent Done false
7549 3046 Acked Flush 0 Done Sent Done false
7550 3047 Acked Read 2 Done Sent Done false
7551 3048 Acked Write 3 Done Sent Done false
7552 3049 Acked Read 4 Done Sent Done false
7553 3050 Acked Write 3 Done Sent Done false
7554 3051 Acked Read 4 Done Sent Done false
7555 3052 Acked Write 1 Done Sent Done false
7556 3053 Acked Read 10 Done Sent Done false
7557 3054 Acked Read 9 Done Sent Done false
7558 3055 Acked Read 1 Done Sent Done false
7559 3056 Acked Read 6 Done Sent Done false
7560 3057 Acked Write 9 Done Sent Done false
7561 3058 Acked Read 5 Done Sent Done false
7562 3059 Acked Read 6 Done Sent Done false
7563 3060 Acked Write 9 Done Sent Done false
7564 3061 Acked Read 7 Done Sent Done false
7565 3062 Acked Read 3 Done Sent Done false
7566 3063 Acked Write 6 Done Sent Done false
7567 3064 Acked Flush 0 Done Sent Done false
7568 3065 Acked Write 9 Done Sent Done false
7569 3066 Acked Read 9 Done Sent Done false
7570 3067 Acked Write 3 Done Sent Done false
7571 3068 Acked Read 9 Done Sent Done false
7572 3069 Acked Write 1 Done Sent Done false
7573 3070 Acked Read 9 Done Sent Done false
7574 3071 Acked Read 2 Done Sent Done false
7575 3072 Acked Write 8 Done Sent Done false
7576 3073 Acked Read 5 Done Sent Done false
7577 3074 Acked Write 1 Done Sent Done false
7578 3075 Acked Write 10 Done Sent Done false
7579 3076 Acked Write 6 Done Sent Done false
7580 3077 Acked Read 9 Done Sent Done false
7581 3078 Acked Read 10 Done Sent Done false
7582 3079 Acked Write 5 Done Sent Done false
7583 3080 Acked Write 6 Done Sent Done false
7584 3081 Acked Read 2 Done Sent Done false
7585 3082 Acked Read 2 Done Sent Done false
7586 3083 Acked Read 7 Done Sent Done false
7587 3084 Acked Read 5 Done Sent Done false
7588 3085 Acked Write 2 Done Sent Done false
7589 3086 Acked Write 7 Done Sent Done false
7590 3087 Acked Write 4 Done Sent Done false
7591 3088 Acked Write 9 Done Sent Done false
7592 3089 Acked Read 10 Done Sent Done false
7593 3090 Acked Read 4 Done Sent Done false
7594 3091 Acked Read 1 Done Sent Done false
7595 3092 Acked Write 8 Done Sent Done false
7596 3093 Acked Write 6 Done Sent Done false
7597 3094 Acked Write 3 Done Sent Done false
7598 3095 Acked Read 10 Done Sent Done false
7599 3096 Acked Read 7 Done Sent Done false
7600 3097 Acked Read 4 Done Sent Done false
7601 3098 Acked Read 9 Done Sent Done false
7602 3099 Acked Read 8 Done Sent Done false
7603 3100 Acked Read 4 Done Sent Done false
7604 3101 Acked Read 8 Done Sent Done false
7605 3102 Acked Write 9 Done Sent Done false
7606 3103 Acked Read 1 Done Sent Done false
7607 3104 Acked Read 9 Done Sent Done false
7608 3105 Acked Write 8 Done Sent Done false
7609 3106 Acked Write 8 Done Sent Done false
7610 3107 Acked Read 6 Done Sent Done false
7611 3108 Acked Flush 0 Done Sent Done false
7612 3109 Acked Read 4 Done Sent Done false
7613 3110 Acked Read 10 Done Sent Done false
7614 3111 Acked Read 4 Done Sent Done false
7615 3112 Acked Flush 0 Done Sent Done false
7616 3113 Acked Flush 0 Done Sent Done false
7617 STATES DS:0 DS:1 DS:2 TOTAL
7618 Sent 0 301 0 301
7619 Done 301 0 301 602
7620 Skipped 0 0 0 0
7621 Error 0 0 0 0
7622 Last Flush: 3113 2812 3113
7623 Downstairs last five completed: 2812 2811 2810 2809 2808
7624 Upstairs last five completed: 3113 3112 3111 3110 3109
7625 CLIENT: Up:0 ds:301 act:2
76262024-11-01T16:52:04.030ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
76272024-11-01T16:52:04.030ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38093) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
76282024-11-01T16:52:04.031ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
76292024-11-01T16:52:04.032ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
76302024-11-01T16:52:04.032ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
76312024-11-01T16:52:04.032ZINFOcrucible: send last flush ID to this DS: 2812 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
76322024-11-01T16:52:04.032ZINFOcrucible: Replied this last flush ID: 2812 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
76332024-11-01T16:52:04.032ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
76342024-11-01T16:52:04.032ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
76352024-11-01T16:52:04.032ZINFOcrucible: [1] client re-new 301 jobs since flush 2812 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
76362024-11-01T16:52:04.032ZINFOcrucible: [1] Replayed 301 jobs since flush: 2812 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
7637 ----------------------------------------------------------------
7638 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7639 Downstairs last five completed: 3113 3112 3111 3110 3109
7640 Upstairs last five completed: 3113 3112 3111 3110 3109
7641 CLIENT: Up:0 ds:0 act:3
7642 Replay: All jobs finished
7643 Main task has work to do, go find it
7644 got dsc Stop(1)
7645 stop 1
7646 Nov 01 16:52:06.331 INFO request completed, latency_us: 353[8820] Got stop action so:false kr:false
7647 , response_code: 204, uri: /stop/cid/1, method: POST, req_id: 19e32e7a-191b-4c34-a784-1cae1786f7d4, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
76482024-11-01T16:52:06.332ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
76492024-11-01T16:52:06.332ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
7650 Nov 01 16:52:06.332{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:52:06.332433062Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
7651 INFO request completed, latency_us: 176, response_code:{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:52:06.332472866Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"client":"1","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
7652 200, uri: /state/cid/1, method: GET, req_id: 3bc31cf5-d0da-440e-83b7-9f4ac485d911, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
76532024-11-01T16:52:06.332ZINFOcrucible: client task is exiting = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
7654 [8820] Exited with: ExitStatus(unix_wait_status(9))
7655 [8820][1] reports Exit
7656 Nov 01 16:52:10.335 INFO request completed, latency_us: 308, response_code: 200, uri: /state/cid/1, method: GET, req_id: 419707fc-cd23-4317-9727-36b5d22eef83, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
7657 001/300 Read block 4665 len 12288
7658 002/300 Read block 3252 len 36864
7659 003/300 Write block 2594 len 20480 data: 2 2 2 2 2
7660 004/300 Read block 3503 len 28672
7661 005/300 Write block 3199 len 40960 data: 4 4 5 4 3 4 4 5 5 4
7662 006/300 Read block 3485 len 28672
7663 007/300 Read block 4428 len 28672
7664 008/300 Read block 1541 len 40960
7665 009/300 Read block 4750 len 20480
7666 010/300 Write block 1362 len 28672 data: 4 3 3 3 3 3 2
7667 011/300 Read block 3419 len 8192
7668 012/300 Read block 2384 len 36864
7669 013/300 Flush
7670 014/300 Write block 4941 len 40960 data: 4 5 5 5 4 4 5 4 4 3
7671 015/300 Write block 4527 len 36864 data: 3 3 2 2 2 3 3 3 3
7672 016/300 Read block 526 len 28672
7673 017/300 Write block 2062 len 20480 data: 3 3 2 2 2
7674 018/300 Flush
7675 019/300 Write block 840 len 32768 data: 3 3 3 2 3 5 5 5
7676 020/300 Write block 2786 len 40960 data: 2 2 2 2 2 3 4 3 3 3
7677 021/300 Write block 1773 len 36864 data: 3 2 2 2 2 2 2 2 4
7678 022/300 Read block 4623 len 12288
7679 023/300 Flush
7680 024/300 Read block 3967 len 36864
7681 025/300 Write block 455 len 32768 data: 3 3 3 3 3 3 3 3
7682 026/300 Read block 3987 len 16384
7683 027/300 Read block 3596 len 8192
7684 028/300 Flush
7685 029/300 Read block 2342 len 4096
7686 030/300 Read block 693 len 32768
7687 031/300 Write block 584 len 12288 data: 2 2 2
7688 032/300 Flush
7689 033/300 Read block 2875 len 40960
7690 034/300 Write block 3582 len 16384 data: 3 3 4 4
7691 035/300 Read block 4548 len 36864
7692 036/300 Write block 1333 len 32768 data: 2 2 3 3 3 2 2 2
7693 037/300 Write block 735 len 20480 data: 3 3 2 2 3
7694 038/300 Read block 4946 len 16384
7695 039/300 Write block 3171 len 20480 data: 3 3 3 2 2
7696 040/300 Read block 4307 len 4096
7697 041/300 Write block 4366 len 28672 data: 2 2 3 3 4 4 4
7698 042/300 Read block 4730 len 36864
7699 043/300 Read block 3564 len 8192
7700 044/300 Flush
7701 045/300 Read block 2889 len 16384
7702 046/300 Write block 548 len 28672 data: 2 3 3 3 4 4 4
7703 047/300 Read block 2742 len 40960
7704 048/300 Write block 1613 len 4096 data: 2
7705 049/300 Write block 3633 len 8192 data: 2 2
7706 050/300 Read block 3675 len 40960
7707 051/300 Write block 3452 len 20480 data: 2 2 2 2 2
7708 052/300 Read block 1139 len 36864
7709 053/300 Read block 1482 len 16384
7710 054/300 Read block 217 len 36864
7711 055/300 Read block 2152 len 8192
7712 056/300 Write block 944 len 28672 data: 3 3 3 3 3 2 2
7713 057/300 Flush
7714 058/300 Read block 2632 len 20480
7715 059/300 Write block 846 len 36864 data: 6 6 5 5 5 5 5 3 3
7716 060/300 Write block 3373 len 12288 data: 5 4 4
7717 061/300 Read block 3539 len 32768
7718 062/300 Write block 1136 len 20480 data: 2 2 2 2 2
7719 063/300 Read block 1147 len 12288
7720 064/300 Write block 1385 len 16384 data: 2 2 2 2
7721 065/300 Read block 4599 len 28672
7722 066/300 Write block 69 len 16384 data: 2 2 2 2
7723 067/300 Read block 4629 len 40960
7724 068/300 Write block 2228 len 32768 data: 2 2 3 3 3 3 3 4
7725 069/300 Write block 436 len 4096 data: 2
7726 070/300 Read block 3404 len 40960
7727 071/300 Read block 1599 len 12288
7728 072/300 Write block 4673 len 8192 data: 4 4
7729 073/300 Read block 1707 len 20480
7730 074/300 Read block 3782 len 12288
7731 075/300 Write block 3207 len 8192 data: 6 5
7732 076/300 Write block 4107 len 8192 data: 5 4
7733 077/300 Read block 168 len 32768
7734 078/300 Read block 1544 len 8192
7735 079/300 Read block 3485 len 28672
7736 080/300 Read block 2970 len 12288
7737 081/300 Write block 3130 len 36864 data: 2 3 3 4 5 5 5 4 4
7738 082/300 Write block 3544 len 36864 data: 3 4 3 3 3 3 3 3 4
7739 083/300 Read block 912 len 32768
7740 084/300 Read block 1631 len 36864
7741 085/300 Write block 3621 len 16384 data: 2 3 3 3
7742 086/300 Read block 2030 len 12288
7743 087/300 Read block 3745 len 28672
7744 088/300 Write block 676 len 36864 data: 4 3 3 2 2 2 2 2 2
7745 089/300 Write block 66 len 12288 data: 2 2 2
7746 090/300 Read block 4223 len 24576
7747 091/300 Read block 4934 len 12288
7748 092/300 Read block 4902 len 4096
7749 093/300 Write block 1195 len 40960 data: 3 2 2 2 2 2 2 2 2 2
7750 094/300 Write block 1119 len 12288 data: 3 3 2
7751 095/300 Read block 4782 len 32768
7752 096/300 Write block 593 len 32768 data: 3 4 4 4 4 4 5 5
7753 097/300 Read block 4798 len 40960
7754 098/300 Read block 4087 len 4096
7755 099/300 Read block 1422 len 40960
7756 100/300 Read block 2115 len 20480
7757 101/300 Read block 4944 len 40960
7758 102/300 Write block 1789 len 4096 data: 2
7759 103/300 Read block 3323 len 20480
7760 104/300 Read block 2089 len 24576
7761 105/300 Write block 3194 len 4096 data: 2
7762 106/300 Read block 3976 len 8192
7763 107/300 Write block 2580 len 28672 data: 2 2 2 2 2 2 2
7764 108/300 Read block 349 len 32768
7765 109/300 Write block 3732 len 20480 data: 3 4 4 4 4
7766 110/300 Flush
7767 111/300 Read block 2306 len 20480
7768 112/300 Write block 2172 len 12288 data: 2 2 2
7769 113/300 Write block 1597 len 4096 data: 2
7770 114/300 Read block 4965 len 28672
7771 115/300 Read block 1454 len 12288
7772 116/300 Read block 2592 len 40960
7773 117/300 Read block 4322 len 8192
7774 118/300 Flush
7775 119/300 Write block 1188 len 36864 data: 3 3 3 3 3 3 3 4 3
7776 120/300 Write block 108 len 4096 data: 2
7777 121/300 Write block 587 len 28672 data: 2 2 2 2 2 3 4
7778 122/300 Flush
7779 123/300 Read block 1360 len 36864
7780 124/300 Read block 4300 len 20480
7781 125/300 Read block 3237 len 24576
7782 126/300 Read block 4375 len 20480
7783 127/300 Write block 4949 len 24576 data: 5 4 4 4 4 4
7784 128/300 Write block 3915 len 16384 data: 3 5 4 4
7785 129/300 Read block 489 len 16384
7786 130/300 Read block 3170 len 36864
7787 131/300 Write block 2312 len 16384 data: 2 2 2 2
7788 132/300 Read block 1899 len 24576
7789 133/300 Flush
7790 134/300 Flush
7791 135/300 Flush
7792 136/300 Read block 2706 len 4096
7793 137/300 Read block 4717 len 16384
7794 138/300 Read block 4928 len 12288
7795 139/300 Write block 2330 len 16384 data: 3 3 3 3
7796 140/300 Read block 1657 len 24576
7797 141/300 Read block 3051 len 36864
7798 142/300 Write block 483 len 36864 data: 3 3 3 3 4 3 3 4 3
7799 143/300 Read block 1458 len 8192
7800 144/300 Read block 672 len 36864
7801 145/300 Write block 3804 len 4096 data: 4
7802 146/300 Read block 3937 len 12288
7803 147/300 Read block 4474 len 8192
7804 148/300 Read block 3097 len 40960
7805 149/300 Write block 1929 len 28672 data: 2 2 2 2 2 2 2
7806 150/300 Read block 2049 len 36864
7807 151/300 Read block 4423 len 20480
7808 152/300 Read block 4289 len 12288
7809 153/300 Read block 4329 len 32768
7810 154/300 Read block 870 len 16384
7811 155/300 Read block 4950 len 24576
7812 156/300 Write block 1491 len 28672 data: 3 3 2 2 2 3 3
7813 157/300 Read block 3046 len 8192
7814 158/300 Read block 2346 len 36864
7815 159/300 Read block 938 len 36864
7816 160/300 Read block 627 len 4096
7817 161/300 Read block 3945 len 16384
7818 162/300 Read block 3430 len 40960
7819 163/300 Write block 2255 len 36864 data: 3 3 3 2 2 2 2 2 2
7820 164/300 Write block 87 len 40960 data: 3 3 4 5 5 5 5 4 4 5
7821 165/300 Write block 2886 len 20480 data: 2 2 3 3 3
7822 166/300 Flush
7823 167/300 Read block 1096 len 8192
7824 168/300 Read block 3262 len 8192
7825 169/300 Read block 4675 len 36864
7826 170/300 Read block 170 len 20480
7827 171/300 Write block 4050 len 24576 data: 2 3 3 3 3 3
7828 172/300 Write block 4376 len 40960 data: 3 3 3 3 3 4 3 3 3 3
7829 173/300 Flush
7830 174/300 Write block 3382 len 40960 data: 3 4 4 4 4 3 2 2 2 2
7831 175/300 Read block 4495 len 32768
7832 176/300 Write block 2729 len 24576 data: 3 2 2 3 3 3
7833 177/300 Read block 2042 len 12288
7834 178/300 Read block 2049 len 40960
7835 179/300 Flush
7836 180/300 Flush
7837 181/300 Flush
7838 182/300 Write block 1171 len 40960 data: 2 2 2 2 2 3 3 3 3 3
7839 183/300 Write block 1403 len 24576 data: 4 4 3 3 3 3
7840 184/300 Read block 1625 len 36864
7841 185/300 Read block 2922 len 40960
7842 186/300 Read block 4175 len 32768
7843 187/300 Write block 2265 len 20480 data: 2 2 2 2 2
7844 188/300 Read block 342 len 8192
7845 189/300 Write block 261 len 36864 data: 3 2 2 2 2 2 3 3 3
7846 190/300 Read block 2648 len 32768
7847 191/300 Write block 1842 len 36864 data: 3 3 3 3 2 2 2 2 2
7848 192/300 Read block 2728 len 24576
7849 193/300 Read block 2843 len 12288
7850 194/300 Read block 308 len 24576
7851 195/300 Write block 4556 len 20480 data: 2 2 2 2 2
7852 196/300 Flush
7853 197/300 Write block 1962 len 28672 data: 4 4 4 3 3 4 5
7854 198/300 Read block 1255 len 8192
7855 199/300 Write block 3957 len 24576 data: 3 3 2 2 2 3
7856 200/300 Read block 4766 len 36864
7857 201/300 Read block 3132 len 32768
7858 202/300 Read block 1205 len 24576
7859 203/300 Flush
7860 204/300 Read block 2754 len 32768
7861 205/300 Write block 302 len 24576 data: 2 2 2 2 2 2
7862 206/300 Read block 2369 len 24576
7863 207/300 Write block 4672 len 24576 data: 3 5 5 5 5 5
7864 208/300 Read block 2457 len 36864
7865 209/300 Write block 2258 len 8192 data: 3 3
7866 210/300 Read block 77 len 28672
7867 211/300 Write block 3891 len 20480 data: 2 2 2 2 2
7868 212/300 Write block 4284 len 32768 data: 2 2 2 2 2 2 2 2
7869 213/300 Read block 910 len 24576
7870 214/300 Read block 3469 len 16384
7871 215/300 Write block 3876 len 4096 data: 2
7872 216/300 Read block 4427 len 28672
7873 217/300 Write block 1911 len 4096 data: 2
7874 218/300 Write block 3583 len 32768 data: 4 5 5 4 4 5 4 4
7875 219/300 Write block 3178 len 40960 data: 2 2 2 3 3 4 5 4 4 5
7876 220/300 Write block 3272 len 40960 data: 2 2 3 3 3 3 2 2 2 3
7877 221/300 Write block 1773 len 4096 data: 4
7878 222/300 Write block 1512 len 8192 data: 3 3
7879 223/300 Write block 447 len 24576 data: 3 3 3 2 2 2
7880 224/300 Read block 2631 len 20480
7881 225/300 Read block 1084 len 36864
7882 226/300 Read block 3182 len 8192
7883 227/300 Read block 1287 len 4096
7884 228/300 Write block 4140 len 12288 data: 3 3 3
7885 229/300 Read block 389 len 28672
7886 230/300 Write block 279 len 32768 data: 3 2 2 2 2 2 2 4
7887 231/300 Write block 2040 len 4096 data: 3
7888 232/300 Write block 246 len 32768 data: 6 5 6 4 3 4 4 4
7889 233/300 Write block 3961 len 20480 data: 3 4 3 3 3
7890 234/300 Write block 898 len 24576 data: 2 2 2 2 2 2
7891 235/300 Write block 4893 len 16384 data: 3 3 3 3
7892 236/300 Read block 4663 len 16384
7893 237/300 Write block 1175 len 28672 data: 3 4 4 4 4 4 3
7894 238/300 Flush
7895 239/300 Read block 471 len 32768
7896 240/300 Read block 4828 len 16384
7897 241/300 Read block 883 len 4096
7898 242/300 Write block 749 len 4096 data: 4
7899 243/300 Read block 4226 len 12288
7900 244/300 Write block 3523 len 4096 data: 2
7901 245/300 Flush
7902 246/300 Flush
7903 247/300 Read block 3897 len 4096
7904 248/300 Read block 1620 len 8192
7905 249/300 Read block 247 len 28672
7906 250/300 Write block 2183 len 28672 data: 3 3 3 4 4 4 4
7907 251/300 Read block 646 len 12288
7908 252/300 Write block 4036 len 12288 data: 5 4 4
7909 253/300 Read block 3309 len 16384
7910 254/300 Read block 4052 len 20480
7911 255/300 Read block 4654 len 4096
7912 256/300 Write block 3455 len 36864 data: 3 3 2 3 4 5 5 5 4
7913 257/300 Write block 639 len 36864 data: 3 3 4 4 4 4 4 4 4
7914 258/300 Write block 4820 len 20480 data: 2 2 3 3 3
7915 259/300 Write block 4548 len 20480 data: 3 4 4 3 3
7916 260/300 Read block 1355 len 8192
7917 261/300 Write block 632 len 16384 data: 3 2 2 2
7918 262/300 Read block 829 len 40960
7919 263/300 Write block 2636 len 32768 data: 2 2 2 2 3 3 3 3
7920 264/300 Write block 4768 len 16384 data: 3 3 2 3
7921 265/300 Read block 2781 len 8192
7922 266/300 Read block 3454 len 36864
7923 267/300 Read block 2102 len 24576
7924 268/300 Write block 4009 len 20480 data: 2 2 2 2 2
7925 269/300 Write block 2450 len 8192 data: 4 4
7926 270/300 Write block 550 len 40960 data: 4 4 5 5 5 4 4 3 2 2
7927 271/300 Flush
7928 272/300 Read block 3257 len 32768
7929 273/300 Read block 4282 len 4096
7930 274/300 Write block 584 len 36864 data: 3 3 3 3 3 3 3 3 4
7931 275/300 Read block 3483 len 16384
7932 276/300 Write block 36 len 24576 data: 2 2 2 2 3 3
7933 277/300 Read block 788 len 20480
7934 278/300 Read block 3730 len 28672
7935 279/300 Read block 4499 len 4096
7936 280/300 Write block 4671 len 24576 data: 3 4 6 6 6 6
7937 281/300 Read block 1790 len 4096
7938 282/300 Write block 2222 len 24576 data: 3 2 2 2 2 2
7939 283/300 Read block 3625 len 4096
7940 284/300 Flush
7941 285/300 Read block 4061 len 8192
7942 286/300 Flush
7943 287/300 Write block 158 len 36864 data: 2 2 2 2 2 2 2 2 2
7944 288/300 Write block 4512 len 24576 data: 4 4 3 3 2 2
7945 289/300 Write block 1795 len 16384 data: 3 3 3 3
7946 290/300 Read block 1809 len 4096
7947 291/300 Read block 1443 len 40960
7948 292/300 Write block 4050 len 16384 data: 3 4 4 4
7949 293/300 Write block 1474 len 40960 data: 3 2 3 3 4 3 2 2 2 2
7950 294/300 Read block 3500 len 28672
7951 295/300 Write block 158 len 8192 data: 3 3
7952 296/300 Write block 3964 len 8192 data: 4 4
7953 297/300 Write block 197 len 32768 data: 6 7 7 4 4 5 5 5
7954 298/300 Write block 342 len 20480 data: 2 2 3 3 3
7955 299/300 Write block 3231 len 16384 data: 3 3 4 3
7956 300/300 Write block 2741 len 24576 data: 6 5 5 5 5 4
7957 301/300 Read block 4396 len 20480
7958 Main task has work to do, go find it
7959 got dsc Start(1)
7960 start 1
7961 Nov 01 16:52:10.619 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/1, method: POST, req_id: 8ed7f6f4-a499-4a2e-9b8f-b940b7342706, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
7962 Replay: started 1, returned:Ok(())
7963 ----------------------------------------------------------------
7964 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7965 States: Active Offline Active
7966 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7967 3114 Acked Read 3 Done Sent Done false
7968 3115 Acked Read 9 Done Sent Done false
7969 3116 Acked Write 5 Done Sent Done false
7970 3117 Acked Read 7 Done Sent Done false
7971 3118 Acked Write 10 Done Sent Done false
7972 3119 Acked Read 7 Done Sent Done false
7973 3120 Acked Read 7 Done Sent Done false
7974 3121 Acked Read 10 Done Sent Done false
7975 3122 Acked Read 5 Done Sent Done false
7976 3123 Acked Write 7 Done Sent Done false
7977 3124 Acked Read 2 Done Sent Done false
7978 3125 Acked Read 9 Done Sent Done false
7979 3126 Acked Flush 0 Done Sent Done false
7980 3127 Acked Write 10 Done Sent Done false
7981 3128 Acked Write 9 Done Sent Done false
7982 3129 Acked Read 7 Done Sent Done false
7983 3130 Acked Write 5 Done Sent Done false
7984 3131 Acked Flush 0 Done Sent Done false
7985 3132 Acked Write 8 Done Sent Done false
7986 3133 Acked Write 10 Done Sent Done false
7987 3134 Acked Write 9 Done Sent Done false
7988 3135 Acked Read 3 Done Sent Done false
7989 3136 Acked Flush 0 Done Sent Done false
7990 3137 Acked Read 9 Done Sent Done false
7991 3138 Acked Write 8 Done Sent Done false
7992 3139 Acked Read 4 Done Sent Done false
7993 3140 Acked Read 2 Done Sent Done false
7994 3141 Acked Flush 0 Done Sent Done false
7995 3142 Acked Read 1 Done Sent Done false
7996 3143 Acked Read 8 Done Sent Done false
7997 3144 Acked Write 3 Done Sent Done false
7998 3145 Acked Flush 0 Done Sent Done false
7999 3146 Acked Read 10 Done Sent Done false
8000 3147 Acked Write 4 Done Sent Done false
8001 3148 Acked Read 9 Done Sent Done false
8002 3149 Acked Write 8 Done Sent Done false
8003 3150 Acked Write 5 Done Sent Done false
8004 3151 Acked Read 4 Done Sent Done false
8005 3152 Acked Write 5 Done Sent Done false
8006 3153 Acked Read 1 Done Sent Done false
8007 3154 Acked Write 7 Done Sent Done false
8008 3155 Acked Read 9 Done Sent Done false
8009 3156 Acked Read 2 Done Sent Done false
8010 3157 Acked Flush 0 Done Sent Done false
8011 3158 Acked Read 4 Done Sent Done false
8012 3159 Acked Write 7 Done Sent Done false
8013 3160 Acked Read 10 Done Sent Done false
8014 3161 Acked Write 1 Done Sent Done false
8015 3162 Acked Write 2 Done Sent Done false
8016 3163 Acked Read 10 Done Sent Done false
8017 3164 Acked Write 5 Done Sent Done false
8018 3165 Acked Read 9 Done Sent Done false
8019 3166 Acked Read 4 Done Sent Done false
8020 3167 Acked Read 9 Done Sent Done false
8021 3168 Acked Read 2 Done Sent Done false
8022 3169 Acked Write 7 Done Sent Done false
8023 3170 Acked Flush 0 Done Sent Done false
8024 3171 Acked Read 5 Done Sent Done false
8025 3172 Acked Write 9 Done Sent Done false
8026 3173 Acked Write 3 Done Sent Done false
8027 3174 Acked Read 8 Done Sent Done false
8028 3175 Acked Write 5 Done Sent Done false
8029 3176 Acked Read 3 Done Sent Done false
8030 3177 Acked Write 4 Done Sent Done false
8031 3178 Acked Read 7 Done Sent Done false
8032 3179 Acked Write 4 Done Sent Done false
8033 3180 Acked Read 10 Done Sent Done false
8034 3181 Acked Write 8 Done Sent Done false
8035 3182 Acked Write 1 Done Sent Done false
8036 3183 Acked Read 10 Done Sent Done false
8037 3184 Acked Read 3 Done Sent Done false
8038 3185 Acked Write 2 Done Sent Done false
8039 3186 Acked Read 5 Done Sent Done false
8040 3187 Acked Read 3 Done Sent Done false
8041 3188 Acked Write 2 Done Sent Done false
8042 3189 Acked Write 2 Done Sent Done false
8043 3190 Acked Read 8 Done Sent Done false
8044 3191 Acked Read 2 Done Sent Done false
8045 3192 Acked Read 7 Done Sent Done false
8046 3193 Acked Read 3 Done Sent Done false
8047 3194 Acked Write 9 Done Sent Done false
8048 3195 Acked Write 9 Done Sent Done false
8049 3196 Acked Read 8 Done Sent Done false
8050 3197 Acked Read 9 Done Sent Done false
8051 3198 Acked Write 4 Done Sent Done false
8052 3199 Acked Read 3 Done Sent Done false
8053 3200 Acked Read 7 Done Sent Done false
8054 3201 Acked Write 9 Done Sent Done false
8055 3202 Acked Write 3 Done Sent Done false
8056 3203 Acked Read 6 Done Sent Done false
8057 3204 Acked Read 3 Done Sent Done false
8058 3205 Acked Read 1 Done Sent Done false
8059 3206 Acked Write 10 Done Sent Done false
8060 3207 Acked Write 3 Done Sent Done false
8061 3208 Acked Read 8 Done Sent Done false
8062 3209 Acked Write 8 Done Sent Done false
8063 3210 Acked Read 10 Done Sent Done false
8064 3211 Acked Read 1 Done Sent Done false
8065 3212 Acked Read 10 Done Sent Done false
8066 3213 Acked Read 5 Done Sent Done false
8067 3214 Acked Read 10 Done Sent Done false
8068 3215 Acked Write 1 Done Sent Done false
8069 3216 Acked Read 5 Done Sent Done false
8070 3217 Acked Read 6 Done Sent Done false
8071 3218 Acked Write 1 Done Sent Done false
8072 3219 Acked Read 2 Done Sent Done false
8073 3220 Acked Write 7 Done Sent Done false
8074 3221 Acked Read 8 Done Sent Done false
8075 3222 Acked Write 5 Done Sent Done false
8076 3223 Acked Flush 0 Done Sent Done false
8077 3224 Acked Read 5 Done Sent Done false
8078 3225 Acked Write 3 Done Sent Done false
8079 3226 Acked Write 1 Done Sent Done false
8080 3227 Acked Read 7 Done Sent Done false
8081 3228 Acked Read 3 Done Sent Done false
8082 3229 Acked Read 10 Done Sent Done false
8083 3230 Acked Read 2 Done Sent Done false
8084 3231 Acked Flush 0 Done Sent Done false
8085 3232 Acked Write 9 Done Sent Done false
8086 3233 Acked Write 1 Done Sent Done false
8087 3234 Acked Write 7 Done Sent Done false
8088 3235 Acked Flush 0 Done Sent Done false
8089 3236 Acked Read 9 Done Sent Done false
8090 3237 Acked Read 5 Done Sent Done false
8091 3238 Acked Read 6 Done Sent Done false
8092 3239 Acked Read 5 Done Sent Done false
8093 3240 Acked Write 6 Done Sent Done false
8094 3241 Acked Write 4 Done Sent Done false
8095 3242 Acked Read 4 Done Sent Done false
8096 3243 Acked Read 9 Done Sent Done false
8097 3244 Acked Write 4 Done Sent Done false
8098 3245 Acked Read 6 Done Sent Done false
8099 3246 Acked Flush 0 Done Sent Done false
8100 3247 Acked Flush 0 Done Sent Done false
8101 3248 Acked Flush 0 Done Sent Done false
8102 3249 Acked Read 1 Done Sent Done false
8103 3250 Acked Read 4 Done Sent Done false
8104 3251 Acked Read 3 Done Sent Done false
8105 3252 Acked Write 4 Done Sent Done false
8106 3253 Acked Read 6 Done Sent Done false
8107 3254 Acked Read 9 Done Sent Done false
8108 3255 Acked Write 9 Done Sent Done false
8109 3256 Acked Read 2 Done Sent Done false
8110 3257 Acked Read 9 Done Sent Done false
8111 3258 Acked Write 1 Done Sent Done false
8112 3259 Acked Read 3 Done Sent Done false
8113 3260 Acked Read 2 Done Sent Done false
8114 3261 Acked Read 10 Done Sent Done false
8115 3262 Acked Write 7 Done Sent Done false
8116 3263 Acked Read 9 Done Sent Done false
8117 3264 Acked Read 5 Done Sent Done false
8118 3265 Acked Read 3 Done Sent Done false
8119 3266 Acked Read 8 Done Sent Done false
8120 3267 Acked Read 4 Done Sent Done false
8121 3268 Acked Read 6 Done Sent Done false
8122 3269 Acked Write 7 Done Sent Done false
8123 3270 Acked Read 2 Done Sent Done false
8124 3271 Acked Read 9 Done Sent Done false
8125 3272 Acked Read 9 Done Sent Done false
8126 3273 Acked Read 1 Done Sent Done false
8127 3274 Acked Read 4 Done Sent Done false
8128 3275 Acked Read 10 Done Sent Done false
8129 3276 Acked Write 9 Done Sent Done false
8130 3277 Acked Write 10 Done Sent Done false
8131 3278 Acked Write 5 Done Sent Done false
8132 3279 Acked Flush 0 Done Sent Done false
8133 3280 Acked Read 2 Done Sent Done false
8134 3281 Acked Read 2 Done Sent Done false
8135 3282 Acked Read 9 Done Sent Done false
8136 3283 Acked Read 5 Done Sent Done false
8137 3284 Acked Write 6 Done Sent Done false
8138 3285 Acked Write 10 Done Sent Done false
8139 3286 Acked Flush 0 Done Sent Done false
8140 3287 Acked Write 10 Done Sent Done false
8141 3288 Acked Read 8 Done Sent Done false
8142 3289 Acked Write 6 Done Sent Done false
8143 3290 Acked Read 3 Done Sent Done false
8144 3291 Acked Read 10 Done Sent Done false
8145 3292 Acked Flush 0 Done Sent Done false
8146 3293 Acked Flush 0 Done Sent Done false
8147 3294 Acked Flush 0 Done Sent Done false
8148 3295 Acked Write 10 Done Sent Done false
8149 3296 Acked Write 6 Done Sent Done false
8150 3297 Acked Read 9 Done Sent Done false
8151 3298 Acked Read 10 Done Sent Done false
8152 3299 Acked Read 8 Done Sent Done false
8153 3300 Acked Write 5 Done Sent Done false
8154 3301 Acked Read 2 Done Sent Done false
8155 3302 Acked Write 9 Done Sent Done false
8156 3303 Acked Read 8 Done Sent Done false
8157 3304 Acked Write 9 Done Sent Done false
8158 3305 Acked Read 6 Done Sent Done false
8159 3306 Acked Read 3 Done Sent Done false
8160 3307 Acked Read 6 Done Sent Done false
8161 3308 Acked Write 5 Done Sent Done false
8162 3309 Acked Flush 0 Done Sent Done false
8163 3310 Acked Write 7 Done Sent Done false
8164 3311 Acked Read 2 Done Sent Done false
8165 3312 Acked Write 6 Done Sent Done false
8166 3313 Acked Read 9 Done Sent Done false
8167 3314 Acked Read 8 Done Sent Done false
8168 3315 Acked Read 6 Done Sent Done false
8169 3316 Acked Flush 0 Done Sent Done false
8170 3317 Acked Read 8 Done Sent Done false
8171 3318 Acked Write 6 Done Sent Done false
8172 3319 Acked Read 6 Done Sent Done false
8173 3320 Acked Write 6 Done Sent Done false
8174 3321 Acked Read 9 Done Sent Done false
8175 3322 Acked Write 2 Done Sent Done false
8176 3323 Acked Read 7 Done Sent Done false
8177 3324 Acked Write 5 Done Sent Done false
8178 3325 Acked Write 8 Done Sent Done false
8179 3326 Acked Read 6 Done Sent Done false
8180 3327 Acked Read 4 Done Sent Done false
8181 3328 Acked Write 1 Done Sent Done false
8182 3329 Acked Read 7 Done Sent Done false
8183 3330 Acked Write 1 Done Sent Done false
8184 3331 Acked Write 8 Done Sent Done false
8185 3332 Acked Write 10 Done Sent Done false
8186 3333 Acked Write 10 Done Sent Done false
8187 3334 Acked Write 1 Done Sent Done false
8188 3335 Acked Write 2 Done Sent Done false
8189 3336 Acked Write 6 Done Sent Done false
8190 3337 Acked Read 5 Done Sent Done false
8191 3338 Acked Read 9 Done Sent Done false
8192 3339 Acked Read 2 Done Sent Done false
8193 3340 Acked Read 1 Done Sent Done false
8194 3341 Acked Write 3 Done Sent Done false
8195 3342 Acked Read 7 Done Sent Done false
8196 3343 Acked Write 8 Done Sent Done false
8197 3344 Acked Write 1 Done Sent Done false
8198 3345 Acked Write 8 Done Sent Done false
8199 3346 Acked Write 5 Done Sent Done false
8200 3347 Acked Write 6 Done Sent Done false
8201 3348 Acked Write 4 Done Sent Done false
8202 3349 Acked Read 4 Done Sent Done false
8203 3350 Acked Write 7 Done Sent Done false
8204 3351 Acked Flush 0 Done Sent Done false
8205 3352 Acked Read 8 Done Sent Done false
8206 3353 Acked Read 4 Done Sent Done false
8207 3354 Acked Read 1 Done Sent Done false
8208 3355 Acked Write 1 Done Sent Done false
8209 3356 Acked Read 3 Done Sent Done false
8210 3357 Acked Write 1 Done Sent Done false
8211 3358 Acked Flush 0 Done Sent Done false
8212 3359 Acked Flush 0 Done Sent Done false
8213 3360 Acked Read 1 Done Sent Done false
8214 3361 Acked Read 2 Done Sent Done false
8215 3362 Acked Read 7 Done Sent Done false
8216 3363 Acked Write 7 Done Sent Done false
8217 3364 Acked Read 3 Done Sent Done false
8218 3365 Acked Write 3 Done Sent Done false
8219 3366 Acked Read 4 Done Sent Done false
8220 3367 Acked Read 5 Done Sent Done false
8221 3368 Acked Read 1 Done Sent Done false
8222 3369 Acked Write 9 Done Sent Done false
8223 3370 Acked Write 9 Done Sent Done false
8224 3371 Acked Write 5 Done Sent Done false
8225 3372 Acked Write 5 Done Sent Done false
8226 3373 Acked Read 2 Done Sent Done false
8227 3374 Acked Write 4 Done Sent Done false
8228 3375 Acked Read 10 Done Sent Done false
8229 3376 Acked Write 8 Done Sent Done false
8230 3377 Acked Write 4 Done Sent Done false
8231 3378 Acked Read 2 Done Sent Done false
8232 3379 Acked Read 9 Done Sent Done false
8233 3380 Acked Read 6 Done Sent Done false
8234 3381 Acked Write 5 Done Sent Done false
8235 3382 Acked Write 2 Done Sent Done false
8236 3383 Acked Write 10 Done Sent Done false
8237 3384 Acked Flush 0 Done Sent Done false
8238 3385 Acked Read 8 Done Sent Done false
8239 3386 Acked Read 1 Done Sent Done false
8240 3387 Acked Write 9 Done Sent Done false
8241 3388 Acked Read 4 Done Sent Done false
8242 3389 Acked Write 6 Done Sent Done false
8243 3390 Acked Read 5 Done Sent Done false
8244 3391 Acked Read 7 Done Sent Done false
8245 3392 Acked Read 1 Done Sent Done false
8246 3393 Acked Write 6 Done Sent Done false
8247 3394 Acked Read 1 Done Sent Done false
8248 3395 Acked Write 6 Done Sent Done false
8249 3396 Acked Read 1 Done Sent Done false
8250 3397 Acked Flush 0 Done Sent Done false
8251 3398 Acked Read 2 Done Sent Done false
8252 3399 Acked Flush 0 Done Sent Done false
8253 3400 Acked Write 9 Done Sent Done false
8254 3401 Acked Write 6 Done Sent Done false
8255 3402 Acked Write 4 Done Sent Done false
8256 3403 Acked Read 1 Done Sent Done false
8257 3404 Acked Read 10 Done Sent Done false
8258 3405 Acked Write 4 Done Sent Done false
8259 3406 Acked Write 10 Done Sent Done false
8260 3407 Acked Read 7 Done Sent Done false
8261 3408 Acked Write 2 Done Sent Done false
8262 3409 Acked Write 2 Done Sent Done false
8263 3410 Acked Write 8 Done Sent Done false
8264 3411 Acked Write 5 Done Sent Done false
8265 3412 Acked Write 4 Done Sent Done false
8266 3413 Acked Write 6 Done Sent Done false
8267 3414 Acked Read 5 Done Sent Done false
8268 STATES DS:0 DS:1 DS:2 TOTAL
8269 Sent 0 301 0 301
8270 Done 301 0 301 602
8271 Skipped 0 0 0 0
8272 Error 0 0 0 0
8273 Last Flush: 3399 3113 3399
8274 Downstairs last five completed: 3113 3112 3111 3110 3109
8275 Upstairs last five completed: 3414 3413 3412 3411 3410
8276 CLIENT: Up:0 ds:301 act:2
8277 [8820] got start action
8278 [8820] I am going to restart
8279 Starting downstairs at port 8820
8280 Make output file at "/tmp/dsc/downstairs-8820.txt"
8281 [8820][1] reports Starting
8282 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1206)
8283 [8820][1] reports Running
8284 ----------------------------------------------------------------
8285 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
8286 States: Active Offline Active
8287 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8288 3114 Acked Read 3 Done Sent Done false
8289 3115 Acked Read 9 Done Sent Done false
8290 3116 Acked Write 5 Done Sent Done false
8291 3117 Acked Read 7 Done Sent Done false
8292 3118 Acked Write 10 Done Sent Done false
8293 3119 Acked Read 7 Done Sent Done false
8294 3120 Acked Read 7 Done Sent Done false
8295 3121 Acked Read 10 Done Sent Done false
8296 3122 Acked Read 5 Done Sent Done false
8297 3123 Acked Write 7 Done Sent Done false
8298 3124 Acked Read 2 Done Sent Done false
8299 3125 Acked Read 9 Done Sent Done false
8300 3126 Acked Flush 0 Done Sent Done false
8301 3127 Acked Write 10 Done Sent Done false
8302 3128 Acked Write 9 Done Sent Done false
8303 3129 Acked Read 7 Done Sent Done false
8304 3130 Acked Write 5 Done Sent Done false
8305 3131 Acked Flush 0 Done Sent Done false
8306 3132 Acked Write 8 Done Sent Done false
8307 3133 Acked Write 10 Done Sent Done false
8308 3134 Acked Write 9 Done Sent Done false
8309 3135 Acked Read 3 Done Sent Done false
8310 3136 Acked Flush 0 Done Sent Done false
8311 3137 Acked Read 9 Done Sent Done false
8312 3138 Acked Write 8 Done Sent Done false
8313 3139 Acked Read 4 Done Sent Done false
8314 3140 Acked Read 2 Done Sent Done false
8315 3141 Acked Flush 0 Done Sent Done false
8316 3142 Acked Read 1 Done Sent Done false
8317 3143 Acked Read 8 Done Sent Done false
8318 3144 Acked Write 3 Done Sent Done false
8319 3145 Acked Flush 0 Done Sent Done false
8320 3146 Acked Read 10 Done Sent Done false
8321 3147 Acked Write 4 Done Sent Done false
8322 3148 Acked Read 9 Done Sent Done false
8323 3149 Acked Write 8 Done Sent Done false
8324 3150 Acked Write 5 Done Sent Done false
8325 3151 Acked Read 4 Done Sent Done false
8326 3152 Acked Write 5 Done Sent Done false
8327 3153 Acked Read 1 Done Sent Done false
8328 3154 Acked Write 7 Done Sent Done false
8329 3155 Acked Read 9 Done Sent Done false
8330 3156 Acked Read 2 Done Sent Done false
8331 3157 Acked Flush 0 Done Sent Done false
8332 3158 Acked Read 4 Done Sent Done false
8333 3159 Acked Write 7 Done Sent Done false
8334 3160 Acked Read 10 Done Sent Done false
8335 3161 Acked Write 1 Done Sent Done false
8336 3162 Acked Write 2 Done Sent Done false
8337 3163 Acked Read 10 Done Sent Done false
8338 3164 Acked Write 5 Done Sent Done false
8339 3165 Acked Read 9 Done Sent Done false
8340 3166 Acked Read 4 Done Sent Done false
8341 3167 Acked Read 9 Done Sent Done false
8342 3168 Acked Read 2 Done Sent Done false
8343 3169 Acked Write 7 Done Sent Done false
8344 3170 Acked Flush 0 Done Sent Done false
8345 3171 Acked Read 5 Done Sent Done false
8346 3172 Acked Write 9 Done Sent Done false
8347 3173 Acked Write 3 Done Sent Done false
8348 3174 Acked Read 8 Done Sent Done false
8349 3175 Acked Write 5 Done Sent Done false
8350 3176 Acked Read 3 Done Sent Done false
8351 3177 Acked Write 4 Done Sent Done false
8352 3178 Acked Read 7 Done Sent Done false
8353 3179 Acked Write 4 Done Sent Done false
8354 3180 Acked Read 10 Done Sent Done false
8355 3181 Acked Write 8 Done Sent Done false
8356 3182 Acked Write 1 Done Sent Done false
8357 3183 Acked Read 10 Done Sent Done false
8358 3184 Acked Read 3 Done Sent Done false
8359 3185 Acked Write 2 Done Sent Done false
8360 3186 Acked Read 5 Done Sent Done false
8361 3187 Acked Read 3 Done Sent Done false
8362 3188 Acked Write 2 Done Sent Done false
8363 3189 Acked Write 2 Done Sent Done false
8364 3190 Acked Read 8 Done Sent Done false
8365 3191 Acked Read 2 Done Sent Done false
8366 3192 Acked Read 7 Done Sent Done false
8367 3193 Acked Read 3 Done Sent Done false
8368 3194 Acked Write 9 Done Sent Done false
8369 3195 Acked Write 9 Done Sent Done false
8370 3196 Acked Read 8 Done Sent Done false
8371 3197 Acked Read 9 Done Sent Done false
8372 3198 Acked Write 4 Done Sent Done false
8373 3199 Acked Read 3 Done Sent Done false
8374 3200 Acked Read 7 Done Sent Done false
8375 3201 Acked Write 9 Done Sent Done false
8376 3202 Acked Write 3 Done Sent Done false
8377 3203 Acked Read 6 Done Sent Done false
8378 3204 Acked Read 3 Done Sent Done false
8379 3205 Acked Read 1 Done Sent Done false
8380 3206 Acked Write 10 Done Sent Done false
8381 3207 Acked Write 3 Done Sent Done false
8382 3208 Acked Read 8 Done Sent Done false
8383 3209 Acked Write 8 Done Sent Done false
8384 3210 Acked Read 10 Done Sent Done false
8385 3211 Acked Read 1 Done Sent Done false
8386 3212 Acked Read 10 Done Sent Done false
8387 3213 Acked Read 5 Done Sent Done false
8388 3214 Acked Read 10 Done Sent Done false
8389 3215 Acked Write 1 Done Sent Done false
8390 3216 Acked Read 5 Done Sent Done false
8391 3217 Acked Read 6 Done Sent Done false
8392 3218 Acked Write 1 Done Sent Done false
8393 3219 Acked Read 2 Done Sent Done false
8394 3220 Acked Write 7 Done Sent Done false
8395 3221 Acked Read 8 Done Sent Done false
8396 3222 Acked Write 5 Done Sent Done false
8397 3223 Acked Flush 0 Done Sent Done false
8398 3224 Acked Read 5 Done Sent Done false
8399 3225 Acked Write 3 Done Sent Done false
8400 3226 Acked Write 1 Done Sent Done false
8401 3227 Acked Read 7 Done Sent Done false
8402 3228 Acked Read 3 Done Sent Done false
8403 3229 Acked Read 10 Done Sent Done false
8404 3230 Acked Read 2 Done Sent Done false
8405 3231 Acked Flush 0 Done Sent Done false
8406 3232 Acked Write 9 Done Sent Done false
8407 3233 Acked Write 1 Done Sent Done false
8408 3234 Acked Write 7 Done Sent Done false
8409 3235 Acked Flush 0 Done Sent Done false
8410 3236 Acked Read 9 Done Sent Done false
8411 3237 Acked Read 5 Done Sent Done false
8412 3238 Acked Read 6 Done Sent Done false
8413 3239 Acked Read 5 Done Sent Done false
8414 3240 Acked Write 6 Done Sent Done false
8415 3241 Acked Write 4 Done Sent Done false
8416 3242 Acked Read 4 Done Sent Done false
8417 3243 Acked Read 9 Done Sent Done false
8418 3244 Acked Write 4 Done Sent Done false
8419 3245 Acked Read 6 Done Sent Done false
8420 3246 Acked Flush 0 Done Sent Done false
8421 3247 Acked Flush 0 Done Sent Done false
8422 3248 Acked Flush 0 Done Sent Done false
8423 3249 Acked Read 1 Done Sent Done false
8424 3250 Acked Read 4 Done Sent Done false
8425 3251 Acked Read 3 Done Sent Done false
8426 3252 Acked Write 4 Done Sent Done false
8427 3253 Acked Read 6 Done Sent Done false
8428 3254 Acked Read 9 Done Sent Done false
8429 3255 Acked Write 9 Done Sent Done false
8430 3256 Acked Read 2 Done Sent Done false
8431 3257 Acked Read 9 Done Sent Done false
8432 3258 Acked Write 1 Done Sent Done false
8433 3259 Acked Read 3 Done Sent Done false
8434 3260 Acked Read 2 Done Sent Done false
8435 3261 Acked Read 10 Done Sent Done false
8436 3262 Acked Write 7 Done Sent Done false
8437 3263 Acked Read 9 Done Sent Done false
8438 3264 Acked Read 5 Done Sent Done false
8439 3265 Acked Read 3 Done Sent Done false
8440 3266 Acked Read 8 Done Sent Done false
8441 3267 Acked Read 4 Done Sent Done false
8442 3268 Acked Read 6 Done Sent Done false
8443 3269 Acked Write 7 Done Sent Done false
8444 3270 Acked Read 2 Done Sent Done false
8445 3271 Acked Read 9 Done Sent Done false
8446 3272 Acked Read 9 Done Sent Done false
8447 3273 Acked Read 1 Done Sent Done false
8448 3274 Acked Read 4 Done Sent Done false
8449 3275 Acked Read 10 Done Sent Done false
8450 3276 Acked Write 9 Done Sent Done false
8451 3277 Acked Write 10 Done Sent Done false
8452 3278 Acked Write 5 Done Sent Done false
8453 3279 Acked Flush 0 Done Sent Done false
8454 3280 Acked Read 2 Done Sent Done false
8455 3281 Acked Read 2 Done Sent Done false
8456 3282 Acked Read 9 Done Sent Done false
8457 3283 Acked Read 5 Done Sent Done false
8458 3284 Acked Write 6 Done Sent Done false
8459 3285 Acked Write 10 Done Sent Done false
8460 3286 Acked Flush 0 Done Sent Done false
8461 3287 Acked Write 10 Done Sent Done false
8462 3288 Acked Read 8 Done Sent Done false
8463 3289 Acked Write 6 Done Sent Done false
8464 3290 Acked Read 3 Done Sent Done false
8465 3291 Acked Read 10 Done Sent Done false
8466 3292 Acked Flush 0 Done Sent Done false
8467 3293 Acked Flush 0 Done Sent Done false
8468 3294 Acked Flush 0 Done Sent Done false
8469 3295 Acked Write 10 Done Sent Done false
8470 3296 Acked Write 6 Done Sent Done false
8471 3297 Acked Read 9 Done Sent Done false
8472 3298 Acked Read 10 Done Sent Done false
8473 3299 Acked Read 8 Done Sent Done false
8474 3300 Acked Write 5 Done Sent Done false
8475 3301 Acked Read 2 Done Sent Done false
8476 3302 Acked Write 9 Done Sent Done false
8477 3303 Acked Read 8 Done Sent Done false
8478 3304 Acked Write 9 Done Sent Done false
8479 3305 Acked Read 6 Done Sent Done false
8480 3306 Acked Read 3 Done Sent Done false
8481 3307 Acked Read 6 Done Sent Done false
8482 3308 Acked Write 5 Done Sent Done false
8483 3309 Acked Flush 0 Done Sent Done false
8484 3310 Acked Write 7 Done Sent Done false
8485 3311 Acked Read 2 Done Sent Done false
8486 3312 Acked Write 6 Done Sent Done false
8487 3313 Acked Read 9 Done Sent Done false
8488 3314 Acked Read 8 Done Sent Done false
8489 3315 Acked Read 6 Done Sent Done false
8490 3316 Acked Flush 0 Done Sent Done false
8491 3317 Acked Read 8 Done Sent Done false
8492 3318 Acked Write 6 Done Sent Done false
8493 3319 Acked Read 6 Done Sent Done false
8494 3320 Acked Write 6 Done Sent Done false
8495 3321 Acked Read 9 Done Sent Done false
8496 3322 Acked Write 2 Done Sent Done false
8497 3323 Acked Read 7 Done Sent Done false
8498 3324 Acked Write 5 Done Sent Done false
8499 3325 Acked Write 8 Done Sent Done false
8500 3326 Acked Read 6 Done Sent Done false
8501 3327 Acked Read 4 Done Sent Done false
8502 3328 Acked Write 1 Done Sent Done false
8503 3329 Acked Read 7 Done Sent Done false
8504 3330 Acked Write 1 Done Sent Done false
8505 3331 Acked Write 8 Done Sent Done false
8506 3332 Acked Write 10 Done Sent Done false
8507 3333 Acked Write 10 Done Sent Done false
8508 3334 Acked Write 1 Done Sent Done false
8509 3335 Acked Write 2 Done Sent Done false
8510 3336 Acked Write 6 Done Sent Done false
8511 3337 Acked Read 5 Done Sent Done false
8512 3338 Acked Read 9 Done Sent Done false
8513 3339 Acked Read 2 Done Sent Done false
8514 3340 Acked Read 1 Done Sent Done false
8515 3341 Acked Write 3 Done Sent Done false
8516 3342 Acked Read 7 Done Sent Done false
8517 3343 Acked Write 8 Done Sent Done false
8518 3344 Acked Write 1 Done Sent Done false
8519 3345 Acked Write 8 Done Sent Done false
8520 3346 Acked Write 5 Done Sent Done false
8521 3347 Acked Write 6 Done Sent Done false
8522 3348 Acked Write 4 Done Sent Done false
8523 3349 Acked Read 4 Done Sent Done false
8524 3350 Acked Write 7 Done Sent Done false
8525 3351 Acked Flush 0 Done Sent Done false
8526 3352 Acked Read 8 Done Sent Done false
8527 3353 Acked Read 4 Done Sent Done false
8528 3354 Acked Read 1 Done Sent Done false
8529 3355 Acked Write 1 Done Sent Done false
8530 3356 Acked Read 3 Done Sent Done false
8531 3357 Acked Write 1 Done Sent Done false
8532 3358 Acked Flush 0 Done Sent Done false
8533 3359 Acked Flush 0 Done Sent Done false
8534 3360 Acked Read 1 Done Sent Done false
8535 3361 Acked Read 2 Done Sent Done false
8536 3362 Acked Read 7 Done Sent Done false
8537 3363 Acked Write 7 Done Sent Done false
8538 3364 Acked Read 3 Done Sent Done false
8539 3365 Acked Write 3 Done Sent Done false
8540 3366 Acked Read 4 Done Sent Done false
8541 3367 Acked Read 5 Done Sent Done false
8542 3368 Acked Read 1 Done Sent Done false
8543 3369 Acked Write 9 Done Sent Done false
8544 3370 Acked Write 9 Done Sent Done false
8545 3371 Acked Write 5 Done Sent Done false
8546 3372 Acked Write 5 Done Sent Done false
8547 3373 Acked Read 2 Done Sent Done false
8548 3374 Acked Write 4 Done Sent Done false
8549 3375 Acked Read 10 Done Sent Done false
8550 3376 Acked Write 8 Done Sent Done false
8551 3377 Acked Write 4 Done Sent Done false
8552 3378 Acked Read 2 Done Sent Done false
8553 3379 Acked Read 9 Done Sent Done false
8554 3380 Acked Read 6 Done Sent Done false
8555 3381 Acked Write 5 Done Sent Done false
8556 3382 Acked Write 2 Done Sent Done false
8557 3383 Acked Write 10 Done Sent Done false
8558 3384 Acked Flush 0 Done Sent Done false
8559 3385 Acked Read 8 Done Sent Done false
8560 3386 Acked Read 1 Done Sent Done false
8561 3387 Acked Write 9 Done Sent Done false
8562 3388 Acked Read 4 Done Sent Done false
8563 3389 Acked Write 6 Done Sent Done false
8564 3390 Acked Read 5 Done Sent Done false
8565 3391 Acked Read 7 Done Sent Done false
8566 3392 Acked Read 1 Done Sent Done false
8567 3393 Acked Write 6 Done Sent Done false
8568 3394 Acked Read 1 Done Sent Done false
8569 3395 Acked Write 6 Done Sent Done false
8570 3396 Acked Read 1 Done Sent Done false
8571 3397 Acked Flush 0 Done Sent Done false
8572 3398 Acked Read 2 Done Sent Done false
8573 3399 Acked Flush 0 Done Sent Done false
8574 3400 Acked Write 9 Done Sent Done false
8575 3401 Acked Write 6 Done Sent Done false
8576 3402 Acked Write 4 Done Sent Done false
8577 3403 Acked Read 1 Done Sent Done false
8578 3404 Acked Read 10 Done Sent Done false
8579 3405 Acked Write 4 Done Sent Done false
8580 3406 Acked Write 10 Done Sent Done false
8581 3407 Acked Read 7 Done Sent Done false
8582 3408 Acked Write 2 Done Sent Done false
8583 3409 Acked Write 2 Done Sent Done false
8584 3410 Acked Write 8 Done Sent Done false
8585 3411 Acked Write 5 Done Sent Done false
8586 3412 Acked Write 4 Done Sent Done false
8587 3413 Acked Write 6 Done Sent Done false
8588 3414 Acked Read 5 Done Sent Done false
8589 3415 Acked Flush 0 Done Sent Done false
8590 STATES DS:0 DS:1 DS:2 TOTAL
8591 Sent 0 302 0 302
8592 Done 302 0 302 604
8593 Skipped 0 0 0 0
8594 Error 0 0 0 0
8595 Last Flush: 3415 3113 3415
8596 Downstairs last five completed: 3113 3112 3111 3110 3109
8597 Upstairs last five completed: 3415 3414 3413 3412 3411
8598 CLIENT: Up:0 ds:302 act:2
85992024-11-01T16:52:16.334ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
86002024-11-01T16:52:16.334ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44020) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
86012024-11-01T16:52:16.335ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
86022024-11-01T16:52:16.336ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
86032024-11-01T16:52:16.336ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
86042024-11-01T16:52:16.336ZINFOcrucible: send last flush ID to this DS: 3113 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
86052024-11-01T16:52:16.336ZINFOcrucible: Replied this last flush ID: 3113 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
86062024-11-01T16:52:16.336ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
86072024-11-01T16:52:16.336ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
86082024-11-01T16:52:16.336ZINFOcrucible: [1] client re-new 302 jobs since flush 3113 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
86092024-11-01T16:52:16.336ZINFOcrucible: [1] Replayed 302 jobs since flush: 3113 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
8610 ----------------------------------------------------------------
8611 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8612 Downstairs last five completed: 3415 3414 3413 3412 3411
8613 Upstairs last five completed: 3415 3414 3413 3412 3411
8614 CLIENT: Up:0 ds:0 act:3
8615 Replay: All jobs finished
8616 Main task has work to do, go find it
8617 got dsc Stop(1)
8618 stop 1
8619 Nov 01 16:52:18.624 INFO request completed, latency_us[8820] Got stop action so:false kr:false
8620 : 348, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 52381c95-c23d-4671-8738-c34662a5b51f, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
86212024-11-01T16:52:18.625ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
86222024-11-01T16:52:18.625ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
86232024-11-01T16:52:18.625ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
86242024-11-01T16:52:18.625ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
8625 Nov 01 16:52:18.625 INFO request completed, latency_us: 164, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6d6c8cbc-31d9-4b3c-a6f4-9c3f0c887471, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
86262024-11-01T16:52:18.625ZINFOcrucible: client task is exiting = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
8627 [8820] Exited with: ExitStatus(unix_wait_status(9))
8628 [8820][1] reports Exit
8629 Nov 01 16:52:22.627 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6e57d8b3-84fa-4383-b03b-9400b63b6d47, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
8630 001/300 Write block 1715 len 36864 data: 3 3 3 3 3 4 3 3 3
8631 002/300 Write block 2073 len 8192 data: 3 3
8632 003/300 Read block 1086 len 8192
8633 004/300 Flush
8634 005/300 Write block 1087 len 32768 data: 5 6 6 5 5 4 3 3
8635 006/300 Read block 3465 len 12288
8636 007/300 Read block 3256 len 4096
8637 008/300 Read block 2836 len 16384
8638 009/300 Flush
8639 010/300 Read block 4924 len 4096
8640 011/300 Write block 4553 len 16384 data: 3 2 2 3
8641 012/300 Read block 78 len 8192
8642 013/300 Write block 4942 len 16384 data: 6 6 6 5
8643 014/300 Write block 1528 len 28672 data: 3 3 3 3 2 2 2
8644 015/300 Read block 870 len 24576
8645 016/300 Read block 4230 len 40960
8646 017/300 Read block 2208 len 20480
8647 018/300 Read block 1801 len 28672
8648 019/300 Write block 2471 len 36864 data: 2 2 2 2 3 4 4 3 3
8649 020/300 Read block 2371 len 4096
8650 021/300 Write block 3823 len 28672 data: 2 2 3 4 4 4 3
8651 022/300 Read block 4178 len 32768
8652 023/300 Write block 1104 len 20480 data: 3 3 3 3 2
8653 024/300 Read block 1444 len 12288
8654 025/300 Write block 4076 len 20480 data: 3 3 3 3 2
8655 026/300 Flush
8656 027/300 Write block 1711 len 32768 data: 2 2 2 2 4 4 4 4
8657 028/300 Write block 2514 len 12288 data: 2 2 2
8658 029/300 Write block 2424 len 8192 data: 4 3
8659 030/300 Read block 3565 len 40960
8660 031/300 Write block 3539 len 20480 data: 2 2 2 2 3
8661 032/300 Read block 2289 len 8192
8662 033/300 Read block 2098 len 40960
8663 034/300 Write block 1173 len 28672 data: 3 3 4 5 5 5 5
8664 035/300 Read block 4422 len 8192
8665 036/300 Write block 4710 len 12288 data: 3 3 3
8666 037/300 Write block 142 len 20480 data: 2 2 3 3 3
8667 038/300 Write block 411 len 4096 data: 3
8668 039/300 Read block 4394 len 40960
8669 040/300 Write block 2361 len 36864 data: 3 3 3 2 2 2 2 2 3
8670 041/300 Write block 2174 len 36864 data: 3 2 2 2 2 2 2 3 3
8671 042/300 Flush
8672 043/300 Read block 1623 len 4096
8673 044/300 Write block 4685 len 8192 data: 3 3
8674 045/300 Read block 4346 len 8192
8675 046/300 Read block 4847 len 12288
8676 047/300 Read block 2991 len 16384
8677 048/300 Write block 754 len 40960 data: 5 5 5 4 3 3 2 2 4 4
8678 049/300 Read block 220 len 40960
8679 050/300 Flush
8680 051/300 Read block 2530 len 24576
8681 052/300 Read block 3664 len 4096
8682 053/300 Write block 478 len 24576 data: 4 3 3 3 3 4
8683 054/300 Flush
8684 055/300 Write block 4733 len 20480 data: 3 3 3 3 4
8685 056/300 Write block 2844 len 4096 data: 3
8686 057/300 Write block 2031 len 8192 data: 3 3
8687 058/300 Write block 3300 len 12288 data: 4 4 4
8688 059/300 Read block 4258 len 12288
8689 060/300 Flush
8690 061/300 Write block 4002 len 32768 data: 2 2 2 2 2 2 2 3
8691 062/300 Read block 4351 len 4096
8692 063/300 Read block 4048 len 32768
8693 064/300 Read block 2834 len 40960
8694 065/300 Write block 2652 len 4096 data: 3
8695 066/300 Read block 2061 len 4096
8696 067/300 Read block 3530 len 28672
8697 068/300 Write block 2368 len 40960 data: 3 4 4 3 3 3 2 2 2 2
8698 069/300 Flush
8699 070/300 Write block 4331 len 16384 data: 2 2 2 2
8700 071/300 Read block 4225 len 24576
8701 072/300 Read block 249 len 36864
8702 073/300 Flush
8703 074/300 Write block 4498 len 28672 data: 2 2 2 2 2 2 3
8704 075/300 Write block 2427 len 8192 data: 2 3
8705 076/300 Write block 925 len 12288 data: 4 4 5
8706 077/300 Read block 2526 len 40960
8707 078/300 Read block 492 len 12288
8708 079/300 Write block 843 len 32768 data: 3 4 6 7 7 6 6 6
8709 080/300 Flush
8710 081/300 Write block 4461 len 36864 data: 5 5 5 3 3 2 2 2 3
8711 082/300 Read block 2907 len 32768
8712 083/300 Write block 3687 len 24576 data: 3 3 3 3 2 3
8713 084/300 Read block 1930 len 28672
8714 085/300 Write block 4874 len 4096 data: 4
8715 086/300 Flush
8716 087/300 Read block 1573 len 4096
8717 088/300 Write block 3882 len 8192 data: 2 2
8718 089/300 Write block 58 len 28672 data: 2 2 2 2 2 3 3
8719 090/300 Write block 1537 len 20480 data: 2 2 3 3 3
8720 091/300 Write block 2652 len 8192 data: 4 4
8721 092/300 Read block 1152 len 12288
8722 093/300 Flush
8723 094/300 Read block 2978 len 40960
8724 095/300 Write block 3201 len 16384 data: 6 5 4 5
8725 096/300 Write block 4377 len 8192 data: 4 4
8726 097/300 Write block 1721 len 20480 data: 4 4 4 2 3
8727 098/300 Write block 843 len 16384 data: 4 5 7 8
8728 099/300 Read block 3350 len 40960
8729 100/300 Write block 4104 len 36864 data: 5 4 4 6 5 4 5 5 4
8730 101/300 Write block 4270 len 32768 data: 3 2 2 2 2 3 4 4
8731 102/300 Read block 956 len 36864
8732 103/300 Read block 3894 len 8192
8733 104/300 Read block 570 len 8192
8734 105/300 Read block 4899 len 20480
8735 106/300 Read block 208 len 16384
8736 107/300 Write block 2453 len 28672 data: 4 3 3 3 3 3 2
8737 108/300 Write block 2413 len 24576 data: 2 2 3 3 3 3
8738 109/300 Flush
8739 110/300 Flush
8740 111/300 Read block 1451 len 36864
8741 112/300 Flush
8742 113/300 Read block 3566 len 28672
8743 114/300 Read block 845 len 16384
8744 115/300 Read block 1864 len 24576
8745 116/300 Read block 4858 len 24576
8746 117/300 Read block 2507 len 36864
8747 118/300 Read block 4572 len 8192
8748 119/300 Read block 4727 len 12288
8749 120/300 Read block 788 len 32768
8750 121/300 Write block 1915 len 4096 data: 2
8751 122/300 Read block 3522 len 40960
8752 123/300 Read block 109 len 40960
8753 124/300 Flush
8754 125/300 Read block 2260 len 32768
8755 126/300 Write block 4056 len 4096 data: 3
8756 127/300 Write block 4876 len 4096 data: 4
8757 128/300 Write block 2475 len 40960 data: 4 5 5 4 4 3 2 2 2 2
8758 129/300 Read block 163 len 28672
8759 130/300 Read block 187 len 32768
8760 131/300 Read block 4226 len 16384
8761 132/300 Write block 2510 len 32768 data: 2 2 2 2 3 3 3 2
8762 133/300 Read block 2598 len 40960
8763 134/300 Read block 4074 len 20480
8764 135/300 Read block 450 len 40960
8765 136/300 Read block 4283 len 36864
8766 137/300 Read block 850 len 40960
8767 138/300 Read block 3341 len 36864
8768 139/300 Write block 1144 len 8192 data: 2 2
8769 140/300 Read block 4182 len 20480
8770 141/300 Write block 1760 len 40960 data: 3 3 3 3 3 3 3 3 3 3
8771 142/300 Write block 2751 len 40960 data: 2 3 3 3 3 3 2 2 2 2
8772 143/300 Read block 2060 len 20480
8773 144/300 Read block 2826 len 20480
8774 145/300 Read block 777 len 8192
8775 146/300 Read block 3567 len 4096
8776 147/300 Write block 3820 len 8192 data: 3 2
8777 148/300 Read block 1261 len 28672
8778 149/300 Read block 141 len 24576
8779 150/300 Read block 1492 len 12288
8780 151/300 Read block 3289 len 20480
8781 152/300 Write block 1925 len 12288 data: 3 3 3
8782 153/300 Write block 4280 len 16384 data: 4 4 3 3
8783 154/300 Write block 3267 len 4096 data: 2
8784 155/300 Write block 4179 len 28672 data: 3 3 3 4 4 2 3
8785 156/300 Write block 3073 len 16384 data: 2 3 3 4
8786 157/300 Read block 4681 len 32768
8787 158/300 Write block 162 len 24576 data: 3 3 3 3 3 2
8788 159/300 Write block 3173 len 8192 data: 4 3
8789 160/300 Read block 2557 len 28672
8790 161/300 Read block 3733 len 16384
8791 162/300 Write block 2643 len 28672 data: 4 3 3 3 3 3 3
8792 163/300 Read block 3769 len 28672
8793 164/300 Write block 4349 len 4096 data: 2
8794 165/300 Flush
8795 166/300 Write block 2200 len 16384 data: 3 2 2 2
8796 167/300 Write block 444 len 4096 data: 2
8797 168/300 Write block 1798 len 8192 data: 4 3
8798 169/300 Read block 1689 len 12288
8799 170/300 Write block 1747 len 36864 data: 3 3 3 3 2 2 2 2 2
8800 171/300 Read block 383 len 28672
8801 172/300 Read block 3599 len 12288
8802 173/300 Write block 1803 len 24576 data: 4 3 3 3 4 4
8803 174/300 Read block 277 len 16384
8804 175/300 Write block 157 len 20480 data: 2 4 4 3 3
8805 176/300 Read block 2117 len 24576
8806 177/300 Flush
8807 178/300 Read block 1391 len 12288
8808 179/300 Read block 3221 len 16384
8809 180/300 Write block 4923 len 32768 data: 4 4 5 5 3 3 2 2
8810 181/300 Read block 4227 len 36864
8811 182/300 Read block 871 len 8192
8812 183/300 Read block 1404 len 16384
8813 184/300 Read block 3064 len 24576
8814 185/300 Write block 3906 len 16384 data: 2 2 2 2
8815 186/300 Read block 4883 len 32768
8816 187/300 Read block 1732 len 32768
8817 188/300 Write block 75 len 28672 data: 2 2 2 2 2 2 2
8818 189/300 Read block 3155 len 8192
8819 190/300 Read block 4793 len 4096
8820 191/300 Read block 3337 len 20480
8821 192/300 Read block 1643 len 16384
8822 193/300 Flush
8823 194/300 Write block 4358 len 8192 data: 2 3
8824 195/300 Write block 3728 len 16384 data: 3 3 2 3
8825 196/300 Write block 1372 len 16384 data: 2 2 2 2
8826 197/300 Read block 3750 len 36864
8827 198/300 Read block 2225 len 16384
8828 199/300 Write block 3407 len 36864 data: 4 2 2 2 2 2 2 2 2
8829 200/300 Read block 4421 len 32768
8830 201/300 Read block 3003 len 36864
8831 202/300 Flush
8832 203/300 Write block 147 len 8192 data: 2 2
8833 204/300 Flush
8834 205/300 Read block 1819 len 36864
8835 206/300 Read block 4614 len 4096
8836 207/300 Read block 3702 len 4096
8837 208/300 Write block 1189 len 32768 data: 4 4 4 4 4 4 5 4
8838 209/300 Read block 1785 len 24576
8839 210/300 Write block 2236 len 40960 data: 4 4 5 5 4 4 4 4 4 4
8840 211/300 Write block 2620 len 24576 data: 3 3 2 3 3 2
8841 212/300 Flush
8842 213/300 Read block 4210 len 32768
8843 214/300 Write block 4664 len 4096 data: 4
8844 215/300 Read block 462 len 40960
8845 216/300 Write block 4883 len 24576 data: 3 3 3 3 3 3
8846 217/300 Write block 1523 len 20480 data: 2 3 3 3 3
8847 218/300 Flush
8848 219/300 Write block 451 len 32768 data: 3 3 2 2 4 4 4 4
8849 220/300 Write block 270 len 40960 data: 3 3 2 2 2 2 2 2 2 4
8850 221/300 Write block 3932 len 16384 data: 2 2 2 2
8851 222/300 Read block 388 len 32768
8852 223/300 Flush
8853 224/300 Read block 822 len 16384
8854 225/300 Write block 933 len 16384 data: 3 2 2 2
8855 226/300 Read block 2427 len 24576
8856 227/300 Flush
8857 228/300 Read block 4742 len 16384
8858 229/300 Write block 2531 len 8192 data: 2 2
8859 230/300 Write block 3360 len 36864 data: 2 2 2 2 2 2 2 3 3
8860 231/300 Write block 2339 len 28672 data: 2 2 2 2 2 2 2
8861 232/300 Read block 3901 len 12288
8862 233/300 Flush
8863 234/300 Write block 2775 len 24576 data: 3 3 3 3 2 2
8864 235/300 Read block 3071 len 4096
8865 236/300 Write block 2365 len 32768 data: 3 3 3 4 5 5 4 4
8866 237/300 Read block 4109 len 16384
8867 238/300 Write block 2181 len 28672 data: 4 4 4 4 4 5 5
8868 239/300 Write block 3912 len 24576 data: 3 3 3 4 6 5
8869 240/300 Write block 2289 len 24576 data: 4 4 5 6 7 5
8870 241/300 Read block 1310 len 8192
8871 242/300 Write block 916 len 28672 data: 3 3 3 4 4 3 4
8872 243/300 Write block 2786 len 12288 data: 3 3 3
8873 244/300 Write block 1616 len 20480 data: 2 2 2 2 2
8874 245/300 Read block 3634 len 12288
8875 246/300 Read block 2212 len 28672
8876 247/300 Read block 369 len 28672
8877 248/300 Write block 4969 len 12288 data: 3 4 3
8878 249/300 Read block 4842 len 8192
8879 250/300 Read block 2048 len 28672
8880 251/300 Write block 3039 len 4096 data: 2
8881 252/300 Read block 3499 len 16384
8882 253/300 Flush
8883 254/300 Write block 558 len 32768 data: 3 3 2 2 2 2 2 2
8884 255/300 Flush
8885 256/300 Write block 4138 len 32768 data: 4 3 4 4 4 3 2 2
8886 257/300 Flush
8887 258/300 Flush
8888 259/300 Read block 3718 len 28672
8889 260/300 Write block 391 len 36864 data: 2 2 2 2 2 2 2 2 2
8890 261/300 Read block 4387 len 36864
8891 262/300 Read block 2090 len 40960
8892 263/300 Flush
8893 264/300 Read block 3219 len 32768
8894 265/300 Flush
8895 266/300 Read block 110 len 20480
8896 267/300 Read block 908 len 28672
8897 268/300 Read block 1211 len 36864
8898 269/300 Read block 4607 len 8192
8899 270/300 Write block 3325 len 12288 data: 3 3 3
8900 271/300 Read block 2785 len 24576
8901 272/300 Read block 3977 len 32768
8902 273/300 Read block 4492 len 32768
8903 274/300 Flush
8904 275/300 Write block 2218 len 4096 data: 2
8905 276/300 Write block 4772 len 24576 data: 3 3 3 4 4 3
8906 277/300 Write block 968 len 12288 data: 2 2 3
8907 278/300 Read block 787 len 40960
8908 279/300 Flush
8909 280/300 Read block 1436 len 24576
8910 281/300 Write block 838 len 24576 data: 3 3 4 4 4 5
8911 282/300 Read block 2242 len 20480
8912 283/300 Flush
8913 284/300 Read block 1794 len 36864
8914 285/300 Read block 4418 len 16384
8915 286/300 Flush
8916 287/300 Read block 521 len 24576
8917 288/300 Write block 766 len 16384 data: 3 3 3 2
8918 289/300 Read block 3598 len 4096
8919 290/300 Flush
8920 291/300 Read block 4166 len 40960
8921 292/300 Read block 917 len 16384
8922 293/300 Read block 4759 len 4096
8923 294/300 Write block 2094 len 8192 data: 5 5
8924 295/300 Read block 2610 len 4096
8925 296/300 Read block 3586 len 4096
8926 297/300 Write block 4081 len 8192 data: 2 2
8927 298/300 Read block 1155 len 16384
8928 299/300 Write block 1116 len 12288 data: 3 3 3
8929 300/300 Read block 4127 len 20480
8930 301/300 Read block 3752 len 32768
8931 Main task has work to do, go find it
8932 got dsc Start(1)
8933 start 1
8934 Nov 01 16:52:22.915 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/1, method: POST, req_id: 8501ac2c-2ba2-41df-93dc-d3f94456d1d0, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
8935 Replay: started 1, returned:Ok(())
8936 ----------------------------------------------------------------
8937 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8938 States: Active Offline Active
8939 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8940 3416 Acked Write 9 Done Sent Done false
8941 3417 Acked Write 2 Done Sent Done false
8942 3418 Acked Read 2 Done Sent Done false
8943 3419 Acked Flush 0 Done Sent Done false
8944 3420 Acked Write 8 Done Sent Done false
8945 3421 Acked Read 3 Done Sent Done false
8946 3422 Acked Read 1 Done Sent Done false
8947 3423 Acked Read 4 Done Sent Done false
8948 3424 Acked Flush 0 Done Sent Done false
8949 3425 Acked Read 1 Done Sent Done false
8950 3426 Acked Write 4 Done Sent Done false
8951 3427 Acked Read 2 Done Sent Done false
8952 3428 Acked Write 4 Done Sent Done false
8953 3429 Acked Write 7 Done Sent Done false
8954 3430 Acked Read 6 Done Sent Done false
8955 3431 Acked Read 10 Done Sent Done false
8956 3432 Acked Read 5 Done Sent Done false
8957 3433 Acked Read 7 Done Sent Done false
8958 3434 Acked Write 9 Done Sent Done false
8959 3435 Acked Read 1 Done Sent Done false
8960 3436 Acked Write 7 Done Sent Done false
8961 3437 Acked Read 8 Done Sent Done false
8962 3438 Acked Write 5 Done Sent Done false
8963 3439 Acked Read 3 Done Sent Done false
8964 3440 Acked Write 5 Done Sent Done false
8965 3441 Acked Flush 0 Done Sent Done false
8966 3442 Acked Write 8 Done Sent Done false
8967 3443 Acked Write 3 Done Sent Done false
8968 3444 Acked Write 2 Done Sent Done false
8969 3445 Acked Read 10 Done Sent Done false
8970 3446 Acked Write 5 Done Sent Done false
8971 3447 Acked Read 2 Done Sent Done false
8972 3448 Acked Read 10 Done Sent Done false
8973 3449 Acked Write 7 Done Sent Done false
8974 3450 Acked Read 2 Done Sent Done false
8975 3451 Acked Write 3 Done Sent Done false
8976 3452 Acked Write 5 Done Sent Done false
8977 3453 Acked Write 1 Done Sent Done false
8978 3454 Acked Read 10 Done Sent Done false
8979 3455 Acked Write 9 Done Sent Done false
8980 3456 Acked Write 9 Done Sent Done false
8981 3457 Acked Flush 0 Done Sent Done false
8982 3458 Acked Read 1 Done Sent Done false
8983 3459 Acked Write 2 Done Sent Done false
8984 3460 Acked Read 2 Done Sent Done false
8985 3461 Acked Read 3 Done Sent Done false
8986 3462 Acked Read 4 Done Sent Done false
8987 3463 Acked Write 10 Done Sent Done false
8988 3464 Acked Read 10 Done Sent Done false
8989 3465 Acked Flush 0 Done Sent Done false
8990 3466 Acked Read 6 Done Sent Done false
8991 3467 Acked Read 1 Done Sent Done false
8992 3468 Acked Write 6 Done Sent Done false
8993 3469 Acked Flush 0 Done Sent Done false
8994 3470 Acked Write 5 Done Sent Done false
8995 3471 Acked Write 1 Done Sent Done false
8996 3472 Acked Write 2 Done Sent Done false
8997 3473 Acked Write 3 Done Sent Done false
8998 3474 Acked Read 3 Done Sent Done false
8999 3475 Acked Flush 0 Done Sent Done false
9000 3476 Acked Write 8 Done Sent Done false
9001 3477 Acked Read 1 Done Sent Done false
9002 3478 Acked Read 8 Done Sent Done false
9003 3479 Acked Read 10 Done Sent Done false
9004 3480 Acked Write 1 Done Sent Done false
9005 3481 Acked Read 1 Done Sent Done false
9006 3482 Acked Read 7 Done Sent Done false
9007 3483 Acked Write 10 Done Sent Done false
9008 3484 Acked Flush 0 Done Sent Done false
9009 3485 Acked Write 4 Done Sent Done false
9010 3486 Acked Read 6 Done Sent Done false
9011 3487 Acked Read 9 Done Sent Done false
9012 3488 Acked Flush 0 Done Sent Done false
9013 3489 Acked Write 7 Done Sent Done false
9014 3490 Acked Write 2 Done Sent Done false
9015 3491 Acked Write 3 Done Sent Done false
9016 3492 Acked Read 10 Done Sent Done false
9017 3493 Acked Read 3 Done Sent Done false
9018 3494 Acked Write 8 Done Sent Done false
9019 3495 Acked Flush 0 Done Sent Done false
9020 3496 Acked Write 9 Done Sent Done false
9021 3497 Acked Read 8 Done Sent Done false
9022 3498 Acked Write 6 Done Sent Done false
9023 3499 Acked Read 7 Done Sent Done false
9024 3500 Acked Write 1 Done Sent Done false
9025 3501 Acked Flush 0 Done Sent Done false
9026 3502 Acked Read 1 Done Sent Done false
9027 3503 Acked Write 2 Done Sent Done false
9028 3504 Acked Write 7 Done Sent Done false
9029 3505 Acked Write 5 Done Sent Done false
9030 3506 Acked Write 2 Done Sent Done false
9031 3507 Acked Read 3 Done Sent Done false
9032 3508 Acked Flush 0 Done Sent Done false
9033 3509 Acked Read 10 Done Sent Done false
9034 3510 Acked Write 4 Done Sent Done false
9035 3511 Acked Write 2 Done Sent Done false
9036 3512 Acked Write 5 Done Sent Done false
9037 3513 Acked Write 4 Done Sent Done false
9038 3514 Acked Read 10 Done Sent Done false
9039 3515 Acked Write 9 Done Sent Done false
9040 3516 Acked Write 8 Done Sent Done false
9041 3517 Acked Read 9 Done Sent Done false
9042 3518 Acked Read 2 Done Sent Done false
9043 3519 Acked Read 2 Done Sent Done false
9044 3520 Acked Read 5 Done Sent Done false
9045 3521 Acked Read 4 Done Sent Done false
9046 3522 Acked Write 7 Done Sent Done false
9047 3523 Acked Write 6 Done Sent Done false
9048 3524 Acked Flush 0 Done Sent Done false
9049 3525 Acked Flush 0 Done Sent Done false
9050 3526 Acked Read 9 Done Sent Done false
9051 3527 Acked Flush 0 Done Sent Done false
9052 3528 Acked Read 7 Done Sent Done false
9053 3529 Acked Read 4 Done Sent Done false
9054 3530 Acked Read 6 Done Sent Done false
9055 3531 Acked Read 6 Done Sent Done false
9056 3532 Acked Read 9 Done Sent Done false
9057 3533 Acked Read 2 Done Sent Done false
9058 3534 Acked Read 3 Done Sent Done false
9059 3535 Acked Read 8 Done Sent Done false
9060 3536 Acked Write 1 Done Sent Done false
9061 3537 Acked Read 10 Done Sent Done false
9062 3538 Acked Read 10 Done Sent Done false
9063 3539 Acked Flush 0 Done Sent Done false
9064 3540 Acked Read 8 Done Sent Done false
9065 3541 Acked Write 1 Done Sent Done false
9066 3542 Acked Write 1 Done Sent Done false
9067 3543 Acked Write 10 Done Sent Done false
9068 3544 Acked Read 7 Done Sent Done false
9069 3545 Acked Read 8 Done Sent Done false
9070 3546 Acked Read 4 Done Sent Done false
9071 3547 Acked Write 8 Done Sent Done false
9072 3548 Acked Read 10 Done Sent Done false
9073 3549 Acked Read 5 Done Sent Done false
9074 3550 Acked Read 10 Done Sent Done false
9075 3551 Acked Read 9 Done Sent Done false
9076 3552 Acked Read 10 Done Sent Done false
9077 3553 Acked Read 9 Done Sent Done false
9078 3554 Acked Write 2 Done Sent Done false
9079 3555 Acked Read 5 Done Sent Done false
9080 3556 Acked Write 10 Done Sent Done false
9081 3557 Acked Write 10 Done Sent Done false
9082 3558 Acked Read 5 Done Sent Done false
9083 3559 Acked Read 5 Done Sent Done false
9084 3560 Acked Read 2 Done Sent Done false
9085 3561 Acked Read 1 Done Sent Done false
9086 3562 Acked Write 2 Done Sent Done false
9087 3563 Acked Read 7 Done Sent Done false
9088 3564 Acked Read 6 Done Sent Done false
9089 3565 Acked Read 3 Done Sent Done false
9090 3566 Acked Read 5 Done Sent Done false
9091 3567 Acked Write 3 Done Sent Done false
9092 3568 Acked Write 4 Done Sent Done false
9093 3569 Acked Write 1 Done Sent Done false
9094 3570 Acked Write 7 Done Sent Done false
9095 3571 Acked Write 4 Done Sent Done false
9096 3572 Acked Read 8 Done Sent Done false
9097 3573 Acked Write 6 Done Sent Done false
9098 3574 Acked Write 2 Done Sent Done false
9099 3575 Acked Read 7 Done Sent Done false
9100 3576 Acked Read 4 Done Sent Done false
9101 3577 Acked Write 7 Done Sent Done false
9102 3578 Acked Read 7 Done Sent Done false
9103 3579 Acked Write 1 Done Sent Done false
9104 3580 Acked Flush 0 Done Sent Done false
9105 3581 Acked Write 4 Done Sent Done false
9106 3582 Acked Write 1 Done Sent Done false
9107 3583 Acked Write 2 Done Sent Done false
9108 3584 Acked Read 3 Done Sent Done false
9109 3585 Acked Write 9 Done Sent Done false
9110 3586 Acked Read 7 Done Sent Done false
9111 3587 Acked Read 3 Done Sent Done false
9112 3588 Acked Write 6 Done Sent Done false
9113 3589 Acked Read 4 Done Sent Done false
9114 3590 Acked Write 5 Done Sent Done false
9115 3591 Acked Read 6 Done Sent Done false
9116 3592 Acked Flush 0 Done Sent Done false
9117 3593 Acked Read 3 Done Sent Done false
9118 3594 Acked Read 4 Done Sent Done false
9119 3595 Acked Write 8 Done Sent Done false
9120 3596 Acked Read 9 Done Sent Done false
9121 3597 Acked Read 2 Done Sent Done false
9122 3598 Acked Read 4 Done Sent Done false
9123 3599 Acked Read 6 Done Sent Done false
9124 3600 Acked Write 4 Done Sent Done false
9125 3601 Acked Read 8 Done Sent Done false
9126 3602 Acked Read 8 Done Sent Done false
9127 3603 Acked Write 7 Done Sent Done false
9128 3604 Acked Read 2 Done Sent Done false
9129 3605 Acked Read 1 Done Sent Done false
9130 3606 Acked Read 5 Done Sent Done false
9131 3607 Acked Read 4 Done Sent Done false
9132 3608 Acked Flush 0 Done Sent Done false
9133 3609 Acked Write 2 Done Sent Done false
9134 3610 Acked Write 4 Done Sent Done false
9135 3611 Acked Write 4 Done Sent Done false
9136 3612 Acked Read 9 Done Sent Done false
9137 3613 Acked Read 4 Done Sent Done false
9138 3614 Acked Write 9 Done Sent Done false
9139 3615 Acked Read 8 Done Sent Done false
9140 3616 Acked Read 9 Done Sent Done false
9141 3617 Acked Flush 0 Done Sent Done false
9142 3618 Acked Write 2 Done Sent Done false
9143 3619 Acked Flush 0 Done Sent Done false
9144 3620 Acked Read 9 Done Sent Done false
9145 3621 Acked Read 1 Done Sent Done false
9146 3622 Acked Read 1 Done Sent Done false
9147 3623 Acked Write 8 Done Sent Done false
9148 3624 Acked Read 6 Done Sent Done false
9149 3625 Acked Write 10 Done Sent Done false
9150 3626 Acked Write 6 Done Sent Done false
9151 3627 Acked Flush 0 Done Sent Done false
9152 3628 Acked Read 8 Done Sent Done false
9153 3629 Acked Write 1 Done Sent Done false
9154 3630 Acked Read 10 Done Sent Done false
9155 3631 Acked Write 6 Done Sent Done false
9156 3632 Acked Write 5 Done Sent Done false
9157 3633 Acked Flush 0 Done Sent Done false
9158 3634 Acked Write 8 Done Sent Done false
9159 3635 Acked Write 10 Done Sent Done false
9160 3636 Acked Write 4 Done Sent Done false
9161 3637 Acked Read 8 Done Sent Done false
9162 3638 Acked Flush 0 Done Sent Done false
9163 3639 Acked Read 4 Done Sent Done false
9164 3640 Acked Write 4 Done Sent Done false
9165 3641 Acked Read 6 Done Sent Done false
9166 3642 Acked Flush 0 Done Sent Done false
9167 3643 Acked Read 4 Done Sent Done false
9168 3644 Acked Write 2 Done Sent Done false
9169 3645 Acked Write 9 Done Sent Done false
9170 3646 Acked Write 7 Done Sent Done false
9171 3647 Acked Read 3 Done Sent Done false
9172 3648 Acked Flush 0 Done Sent Done false
9173 3649 Acked Write 6 Done Sent Done false
9174 3650 Acked Read 1 Done Sent Done false
9175 3651 Acked Write 8 Done Sent Done false
9176 3652 Acked Read 4 Done Sent Done false
9177 3653 Acked Write 7 Done Sent Done false
9178 3654 Acked Write 6 Done Sent Done false
9179 3655 Acked Write 6 Done Sent Done false
9180 3656 Acked Read 2 Done Sent Done false
9181 3657 Acked Write 7 Done Sent Done false
9182 3658 Acked Write 3 Done Sent Done false
9183 3659 Acked Write 5 Done Sent Done false
9184 3660 Acked Read 3 Done Sent Done false
9185 3661 Acked Read 7 Done Sent Done false
9186 3662 Acked Read 7 Done Sent Done false
9187 3663 Acked Write 3 Done Sent Done false
9188 3664 Acked Read 2 Done Sent Done false
9189 3665 Acked Read 7 Done Sent Done false
9190 3666 Acked Write 1 Done Sent Done false
9191 3667 Acked Read 4 Done Sent Done false
9192 3668 Acked Flush 0 Done Sent Done false
9193 3669 Acked Write 8 Done Sent Done false
9194 3670 Acked Flush 0 Done Sent Done false
9195 3671 Acked Write 8 Done Sent Done false
9196 3672 Acked Flush 0 Done Sent Done false
9197 3673 Acked Flush 0 Done Sent Done false
9198 3674 Acked Read 7 Done Sent Done false
9199 3675 Acked Write 9 Done Sent Done false
9200 3676 Acked Read 9 Done Sent Done false
9201 3677 Acked Read 10 Done Sent Done false
9202 3678 Acked Flush 0 Done Sent Done false
9203 3679 Acked Read 8 Done Sent Done false
9204 3680 Acked Flush 0 Done Sent Done false
9205 3681 Acked Read 5 Done Sent Done false
9206 3682 Acked Read 7 Done Sent Done false
9207 3683 Acked Read 9 Done Sent Done false
9208 3684 Acked Read 2 Done Sent Done false
9209 3685 Acked Write 3 Done Sent Done false
9210 3686 Acked Read 6 Done Sent Done false
9211 3687 Acked Read 8 Done Sent Done false
9212 3688 Acked Read 8 Done Sent Done false
9213 3689 Acked Flush 0 Done Sent Done false
9214 3690 Acked Write 1 Done Sent Done false
9215 3691 Acked Write 6 Done Sent Done false
9216 3692 Acked Write 3 Done Sent Done false
9217 3693 Acked Read 10 Done Sent Done false
9218 3694 Acked Flush 0 Done Sent Done false
9219 3695 Acked Read 6 Done Sent Done false
9220 3696 Acked Write 6 Done Sent Done false
9221 3697 Acked Read 5 Done Sent Done false
9222 3698 Acked Flush 0 Done Sent Done false
9223 3699 Acked Read 9 Done Sent Done false
9224 3700 Acked Read 4 Done Sent Done false
9225 3701 Acked Flush 0 Done Sent Done false
9226 3702 Acked Read 6 Done Sent Done false
9227 3703 Acked Write 4 Done Sent Done false
9228 3704 Acked Read 1 Done Sent Done false
9229 3705 Acked Flush 0 Done Sent Done false
9230 3706 Acked Read 10 Done Sent Done false
9231 3707 Acked Read 4 Done Sent Done false
9232 3708 Acked Read 1 Done Sent Done false
9233 3709 Acked Write 2 Done Sent Done false
9234 3710 Acked Read 1 Done Sent Done false
9235 3711 Acked Read 1 Done Sent Done false
9236 3712 Acked Write 2 Done Sent Done false
9237 3713 Acked Read 4 Done Sent Done false
9238 3714 Acked Write 3 Done Sent Done false
9239 3715 Acked Read 5 Done Sent Done false
9240 3716 Acked Read 8 Done Sent Done false
9241 STATES DS:0 DS:1 DS:2 TOTAL
9242 Sent 0 301 0 301
9243 Done 301 0 301 602
9244 Skipped 0 0 0 0
9245 Error 0 0 0 0
9246 Last Flush: 3705 3415 3705
9247 Downstairs last five completed: 3415 3414 3413 3412 3411
9248 Upstairs last five completed: 3716 3715 3714 3713 3712
9249 CLIENT: Up:0 ds:301 act:2
9250 ----------------------------------------------------------------
9251 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
9252 States: Active Offline Active
9253 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9254 3416 Acked Write 9 Done Sent Done false
9255 3417 Acked Write 2 Done Sent Done false
9256 3418 Acked Read 2 Done Sent Done false
9257 3419 Acked Flush 0 Done Sent Done false
9258 3420 Acked Write 8 Done Sent Done false
9259 3421 Acked Read 3 Done Sent Done false
9260 3422 Acked Read 1 Done Sent Done false
9261 3423 Acked Read 4 Done Sent Done false
9262 3424 Acked Flush 0 Done Sent Done false
9263 3425 Acked Read 1 Done Sent Done false
9264 3426 Acked Write 4 Done Sent Done false
9265 3427 Acked Read 2 Done Sent Done false
9266 3428 Acked Write 4 Done Sent Done false
9267 3429 Acked Write 7 Done Sent Done false
9268 3430 Acked Read 6 Done Sent Done false
9269 3431 Acked Read 10 Done Sent Done false
9270 3432 Acked Read 5 Done Sent Done false
9271 3433 Acked Read 7 Done Sent Done false
9272 3434 Acked Write 9 Done Sent Done false
9273 3435 Acked Read 1 Done Sent Done false
9274 3436 Acked Write 7 Done Sent Done false
9275 3437 Acked Read 8 Done Sent Done false
9276 3438 Acked Write 5 Done Sent Done false
9277 3439 Acked Read 3 Done Sent Done false
9278 3440 Acked Write 5 Done Sent Done false
9279 3441 Acked Flush 0 Done Sent Done false
9280 3442 Acked Write 8 Done Sent Done false
9281 3443 Acked Write 3 Done Sent Done false
9282 3444 Acked Write 2 Done Sent Done false
9283 3445 Acked Read 10 Done Sent Done false
9284 3446 Acked Write 5 Done Sent Done false
9285 3447 Acked Read 2 Done Sent Done false
9286 3448 Acked Read 10 Done Sent Done false
9287 3449 Acked Write 7 Done Sent Done false
9288 3450 Acked Read 2 Done Sent Done false
9289 3451 Acked Write 3 Done Sent Done false
9290 3452 Acked Write 5 Done Sent Done false
9291 3453 Acked Write 1 Done Sent Done false
9292 3454 Acked Read 10 Done Sent Done false
9293 3455 Acked Write 9 Done Sent Done false
9294 3456 Acked Write 9 Done Sent Done false
9295 3457 Acked Flush 0 Done Sent Done false
9296 3458 Acked Read 1 Done Sent Done false
9297 3459 Acked Write 2 Done Sent Done false
9298 3460 Acked Read 2 Done Sent Done false
9299 3461 Acked Read 3 Done Sent Done false
9300 3462 Acked Read 4 Done Sent Done false
9301 3463 Acked Write 10 Done Sent Done false
9302 3464 Acked Read 10 Done Sent Done false
9303 3465 Acked Flush 0 Done Sent Done false
9304 3466 Acked Read 6 Done Sent Done false
9305 3467 Acked Read 1 Done Sent Done false
9306 3468 Acked Write 6 Done Sent Done false
9307 3469 Acked Flush 0 Done Sent Done false
9308 3470 Acked Write 5 Done Sent Done false
9309 3471 Acked Write 1 Done Sent Done false
9310 3472 Acked Write 2 Done Sent Done false
9311 3473 Acked Write 3 Done Sent Done false
9312 3474 Acked Read 3 Done Sent Done false
9313 3475 Acked Flush 0 Done Sent Done false
9314 3476 Acked Write 8 Done Sent Done false
9315 3477 Acked Read 1 Done Sent Done false
9316 3478 Acked Read 8 Done Sent Done false
9317 3479 Acked Read 10 Done Sent Done false
9318 3480 Acked Write 1 Done Sent Done false
9319 3481 Acked Read 1 Done Sent Done false
9320 3482 Acked Read 7 Done Sent Done false
9321 3483 Acked Write 10 Done Sent Done false
9322 3484 Acked Flush 0 Done Sent Done false
9323 3485 Acked Write 4 Done Sent Done false
9324 3486 Acked Read 6 Done Sent Done false
9325 3487 Acked Read 9 Done Sent Done false
9326 3488 Acked Flush 0 Done Sent Done false
9327 3489 Acked Write 7 Done Sent Done false
9328 3490 Acked Write 2 Done Sent Done false
9329 3491 Acked Write 3 Done Sent Done false
9330 3492 Acked Read 10 Done Sent Done false
9331 3493 Acked Read 3 Done Sent Done false
9332 3494 Acked Write 8 Done Sent Done false
9333 3495 Acked Flush 0 Done Sent Done false
9334 3496 Acked Write 9 Done Sent Done false
9335 3497 Acked Read 8 Done Sent Done false
9336 3498 Acked Write 6 Done Sent Done false
9337 3499 Acked Read 7 Done Sent Done false
9338 3500 Acked Write 1 Done Sent Done false
9339 3501 Acked Flush 0 Done Sent Done false
9340 3502 Acked Read 1 Done Sent Done false
9341 3503 Acked Write 2 Done Sent Done false
9342 3504 Acked Write 7 Done Sent Done false
9343 3505 Acked Write 5 Done Sent Done false
9344 3506 Acked Write 2 Done Sent Done false
9345 3507 Acked Read 3 Done Sent Done false
9346 3508 Acked Flush 0 Done Sent Done false
9347 3509 Acked Read 10 Done Sent Done false
9348 3510 Acked Write 4 Done Sent Done false
9349 3511 Acked Write 2 Done Sent Done false
9350 3512 Acked Write 5 Done Sent Done false
9351 3513 Acked Write 4 Done Sent Done false
9352 3514 Acked Read 10 Done Sent Done false
9353 3515 Acked Write 9 Done Sent Done false
9354 3516 Acked Write 8 Done Sent Done false
9355 3517 Acked Read 9 Done Sent Done false
9356 3518 Acked Read 2 Done Sent Done false
9357 3519 Acked Read 2 Done Sent Done false
9358 3520 Acked Read 5 Done Sent Done false
9359 3521 Acked Read 4 Done Sent Done false
9360 3522 Acked Write 7 Done Sent Done false
9361 3523 Acked Write 6 Done Sent Done false
9362 3524 Acked Flush 0 Done Sent Done false
9363 3525 Acked Flush 0 Done Sent Done false
9364 3526 Acked Read 9 Done Sent Done false
9365 3527 Acked Flush 0 Done Sent Done false
9366 3528 Acked Read 7 Done Sent Done false
9367 3529 Acked Read 4 Done Sent Done false
9368 3530 Acked Read 6 Done Sent Done false
9369 3531 Acked Read 6 Done Sent Done false
9370 3532 Acked Read 9 Done Sent Done false
9371 3533 Acked Read 2 Done Sent Done false
9372 3534 Acked Read 3 Done Sent Done false
9373 3535 Acked Read 8 Done Sent Done false
9374 3536 Acked Write 1 Done Sent Done false
9375 3537 Acked Read 10 Done Sent Done false
9376 3538 Acked Read 10 Done Sent Done false
9377 3539 Acked Flush 0 Done Sent Done false
9378 3540 Acked Read 8 Done Sent Done false
9379 3541 Acked Write 1 Done Sent Done false
9380 3542 Acked Write 1 Done Sent Done false
9381 3543 Acked Write 10 Done Sent Done false
9382 3544 Acked Read 7 Done Sent Done false
9383 3545 Acked Read 8 Done Sent Done false
9384 3546 Acked Read 4 Done Sent Done false
9385 3547 Acked Write 8 Done Sent Done false
9386 3548 Acked Read 10 Done Sent Done false
9387 3549 Acked Read 5 Done Sent Done false
9388 3550 Acked Read 10 Done Sent Done false
9389 3551 Acked Read 9 Done Sent Done false
9390 3552 Acked Read 10 Done Sent Done false
9391 3553 Acked Read 9 Done Sent Done false
9392 3554 Acked Write 2 Done Sent Done false
9393 3555 Acked Read 5 Done Sent Done false
9394 3556 Acked Write 10 Done Sent Done false
9395 3557 Acked Write 10 Done Sent Done false
9396 3558 Acked Read 5 Done Sent Done false
9397 3559 Acked Read 5 Done Sent Done false
9398 3560 Acked Read 2 Done Sent Done false
9399 3561 Acked Read 1 Done Sent Done false
9400 3562 Acked Write 2 Done Sent Done false
9401 3563 Acked Read 7 Done Sent Done false
9402 3564 Acked Read 6 Done Sent Done false
9403 3565 Acked Read 3 Done Sent Done false
9404 3566 Acked Read 5 Done Sent Done false
9405 3567 Acked Write 3 Done Sent Done false
9406 3568 Acked Write 4 Done Sent Done false
9407 3569 Acked Write 1 Done Sent Done false
9408 3570 Acked Write 7 Done Sent Done false
9409 3571 Acked Write 4 Done Sent Done false
9410 3572 Acked Read 8 Done Sent Done false
9411 3573 Acked Write 6 Done Sent Done false
9412 3574 Acked Write 2 Done Sent Done false
9413 3575 Acked Read 7 Done Sent Done false
9414 3576 Acked Read 4 Done Sent Done false
9415 3577 Acked Write 7 Done Sent Done false
9416 3578 Acked Read 7 Done Sent Done false
9417 3579 Acked Write 1 Done Sent Done false
9418 3580 Acked Flush 0 Done Sent Done false
9419 3581 Acked Write 4 Done Sent Done false
9420 3582 Acked Write 1 Done Sent Done false
9421 3583 Acked Write 2 Done Sent Done false
9422 3584 Acked Read 3 Done Sent Done false
9423 3585 Acked Write 9 Done Sent Done false
9424 3586 Acked Read 7 Done Sent Done false
9425 3587 Acked Read 3 Done Sent Done false
9426 3588 Acked Write 6 Done Sent Done false
9427 3589 Acked Read 4 Done Sent Done false
9428 3590 Acked Write 5 Done Sent Done false
9429 3591 Acked Read 6 Done Sent Done false
9430 3592 Acked Flush 0 Done Sent Done false
9431 3593 Acked Read 3 Done Sent Done false
9432 3594 Acked Read 4 Done Sent Done false
9433 3595 Acked Write 8 Done Sent Done false
9434 3596 Acked Read 9 Done Sent Done false
9435 3597 Acked Read 2 Done Sent Done false
9436 3598 Acked Read 4 Done Sent Done false
9437 3599 Acked Read 6 Done Sent Done false
9438 3600 Acked Write 4 Done Sent Done false
9439 3601 Acked Read 8 Done Sent Done false
9440 3602 Acked Read 8 Done Sent Done false
9441 3603 Acked Write 7 Done Sent Done false
9442 3604 Acked Read 2 Done Sent Done false
9443 3605 Acked Read 1 Done Sent Done false
9444 3606 Acked Read 5 Done Sent Done false
9445 3607 Acked Read 4 Done Sent Done false
9446 3608 Acked Flush 0 Done Sent Done false
9447 3609 Acked Write 2 Done Sent Done false
9448 3610 Acked Write 4 Done Sent Done false
9449 3611 Acked Write 4 Done Sent Done false
9450 3612 Acked Read 9 Done Sent Done false
9451 3613 Acked Read 4 Done Sent Done false
9452 3614 Acked Write 9 Done Sent Done false
9453 3615 Acked Read 8 Done Sent Done false
9454 3616 Acked Read 9 Done Sent Done false
9455 3617 Acked Flush 0 Done Sent Done false
9456 3618 Acked Write 2 Done Sent Done false
9457 3619 Acked Flush 0 Done Sent Done false
9458 3620 Acked Read 9 Done Sent Done false
9459 3621 Acked Read 1 Done Sent Done false
9460 3622 Acked Read 1 Done Sent Done false
9461 3623 Acked Write 8 Done Sent Done false
9462 3624 Acked Read 6 Done Sent Done false
9463 3625 Acked Write 10 Done Sent Done false
9464 3626 Acked Write 6 Done Sent Done false
9465 3627 Acked Flush 0 Done Sent Done false
9466 3628 Acked Read 8 Done Sent Done false
9467 3629 Acked Write 1 Done Sent Done false
9468 3630 Acked Read 10 Done Sent Done false
9469 3631 Acked Write 6 Done Sent Done false
9470 3632 Acked Write 5 Done Sent Done false
9471 3633 Acked Flush 0 Done Sent Done false
9472 3634 Acked Write 8 Done Sent Done false
9473 3635 Acked Write 10 Done Sent Done false
9474 3636 Acked Write 4 Done Sent Done false
9475 3637 Acked Read 8 Done Sent Done false
9476 3638 Acked Flush 0 Done Sent Done false
9477 3639 Acked Read 4 Done Sent Done false
9478 3640 Acked Write 4 Done Sent Done false
9479 3641 Acked Read 6 Done Sent Done false
9480 3642 Acked Flush 0 Done Sent Done false
9481 3643 Acked Read 4 Done Sent Done false
9482 3644 Acked Write 2 Done Sent Done false
9483 3645 Acked Write 9 Done Sent Done false
9484 3646 Acked Write 7 Done Sent Done false
9485 3647 Acked Read 3 Done Sent Done false
9486 3648 Acked Flush 0 Done Sent Done false
9487 3649 Acked Write 6 Done Sent Done false
9488 3650 Acked Read 1 Done Sent Done false
9489 3651 Acked Write 8 Done Sent Done false
9490 3652 Acked Read 4 Done Sent Done false
9491 3653 Acked Write 7 Done Sent Done false
9492 3654 Acked Write 6 Done Sent Done false
9493 3655 Acked Write 6 Done Sent Done false
9494 3656 Acked Read 2 Done Sent Done false
9495 3657 Acked Write 7 Done Sent Done false
9496 3658 Acked Write 3 Done Sent Done false
9497 3659 Acked Write 5 Done Sent Done false
9498 3660 Acked Read 3 Done Sent Done false
9499 3661 Acked Read 7 Done Sent Done false
9500 3662 Acked Read 7 Done Sent Done false
9501 3663 Acked Write 3 Done Sent Done false
9502 3664 Acked Read 2 Done Sent Done false
9503 3665 Acked Read 7 Done Sent Done false
9504 3666 Acked Write 1 Done Sent Done false
9505 3667 Acked Read 4 Done Sent Done false
9506 3668 Acked Flush 0 Done Sent Done false
9507 3669 Acked Write 8 Done Sent Done false
9508 3670 Acked Flush 0 Done Sent Done false
9509 3671 Acked Write 8 Done Sent Done false
9510 3672 Acked Flush 0 Done Sent Done false
9511 3673 Acked Flush 0 Done Sent Done false
9512 3674 Acked Read 7 Done Sent Done false
9513 3675 Acked Write 9 Done Sent Done false
9514 3676 Acked Read 9 Done Sent Done false
9515 3677 Acked Read 10 Done Sent Done false
9516 3678 Acked Flush 0 Done Sent Done false
9517 3679 Acked Read 8 Done Sent Done false
9518 3680 Acked Flush 0 Done Sent Done false
9519 3681 Acked Read 5 Done Sent Done false
9520 3682 Acked Read 7 Done Sent Done false
9521 3683 Acked Read 9 Done Sent Done false
9522 3684 Acked Read 2 Done Sent Done false
9523 3685 Acked Write 3 Done Sent Done false
9524 3686 Acked Read 6 Done Sent Done false
9525 3687 Acked Read 8 Done Sent Done false
9526 3688 Acked Read 8 Done Sent Done false
9527 3689 Acked Flush 0 Done Sent Done false
9528 3690 Acked Write 1 Done Sent Done false
9529 3691 Acked Write 6 Done Sent Done false
9530 3692 Acked Write 3 Done Sent Done false
9531 3693 Acked Read 10 Done Sent Done false
9532 3694 Acked Flush 0 Done Sent Done false
9533 3695 Acked Read 6 Done Sent Done false
9534 3696 Acked Write 6 Done Sent Done false
9535 3697 Acked Read 5 Done Sent Done false
9536 3698 Acked Flush 0 Done Sent Done false
9537 3699 Acked Read 9 Done Sent Done false
9538 3700 Acked Read 4 Done Sent Done false
9539 3701 Acked Flush 0 Done Sent Done false
9540 3702 Acked Read 6 Done Sent Done false
9541 3703 Acked Write 4 Done Sent Done false
9542 3704 Acked Read 1 Done Sent Done false
9543 3705 Acked Flush 0 Done Sent Done false
9544 3706 Acked Read 10 Done Sent Done false
9545 3707 Acked Read 4 Done Sent Done false
9546 3708 Acked Read 1 Done Sent Done false
9547 3709 Acked Write 2 Done Sent Done false
9548 3710 Acked Read 1 Done Sent Done false
9549 3711 Acked Read 1 Done Sent Done false
9550 3712 Acked Write 2 Done Sent Done false
9551 3713 Acked Read 4 Done Sent Done false
9552 3714 Acked Write 3 Done Sent Done false
9553 3715 Acked Read 5 Done Sent Done false
9554 3716 Acked Read 8 Done Sent Done false
9555 3717 Acked Flush 0 Done Sent Done false
9556 STATES DS:0 DS:1 DS:2 TOTAL
9557 Sent 0 302 0 302
9558 Done 302 0 302 604
9559 Skipped 0 0 0 0
9560 Error 0 0 0 0
9561 Last Flush: 3717 3415 3717
9562 Downstairs last five completed: 3415 3414 3413 3412 3411
9563 Upstairs last five completed: 3717 3716 3715 3714 3713
9564 CLIENT: Up:0 ds:302 act:2
95652024-11-01T16:52:28.626ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
95662024-11-01T16:52:28.626ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
95672024-11-01T16:52:28.626ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
95682024-11-01T16:52:28.626ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
95692024-11-01T16:52:28.626ZINFOcrucible: client task is exiting = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
9570 [8820] got start action
9571 [8820] I am going to restart
9572 Starting downstairs at port 8820
9573 Make output file at "/tmp/dsc/downstairs-8820.txt"
9574 [8820][1] reports Starting
9575 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1207)
9576 [8820][1] reports Running
9577 ----------------------------------------------------------------
9578 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
9579 States: Active Offline Active
9580 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9581 3416 Acked Write 9 Done Sent Done false
9582 3417 Acked Write 2 Done Sent Done false
9583 3418 Acked Read 2 Done Sent Done false
9584 3419 Acked Flush 0 Done Sent Done false
9585 3420 Acked Write 8 Done Sent Done false
9586 3421 Acked Read 3 Done Sent Done false
9587 3422 Acked Read 1 Done Sent Done false
9588 3423 Acked Read 4 Done Sent Done false
9589 3424 Acked Flush 0 Done Sent Done false
9590 3425 Acked Read 1 Done Sent Done false
9591 3426 Acked Write 4 Done Sent Done false
9592 3427 Acked Read 2 Done Sent Done false
9593 3428 Acked Write 4 Done Sent Done false
9594 3429 Acked Write 7 Done Sent Done false
9595 3430 Acked Read 6 Done Sent Done false
9596 3431 Acked Read 10 Done Sent Done false
9597 3432 Acked Read 5 Done Sent Done false
9598 3433 Acked Read 7 Done Sent Done false
9599 3434 Acked Write 9 Done Sent Done false
9600 3435 Acked Read 1 Done Sent Done false
9601 3436 Acked Write 7 Done Sent Done false
9602 3437 Acked Read 8 Done Sent Done false
9603 3438 Acked Write 5 Done Sent Done false
9604 3439 Acked Read 3 Done Sent Done false
9605 3440 Acked Write 5 Done Sent Done false
9606 3441 Acked Flush 0 Done Sent Done false
9607 3442 Acked Write 8 Done Sent Done false
9608 3443 Acked Write 3 Done Sent Done false
9609 3444 Acked Write 2 Done Sent Done false
9610 3445 Acked Read 10 Done Sent Done false
9611 3446 Acked Write 5 Done Sent Done false
9612 3447 Acked Read 2 Done Sent Done false
9613 3448 Acked Read 10 Done Sent Done false
9614 3449 Acked Write 7 Done Sent Done false
9615 3450 Acked Read 2 Done Sent Done false
9616 3451 Acked Write 3 Done Sent Done false
9617 3452 Acked Write 5 Done Sent Done false
9618 3453 Acked Write 1 Done Sent Done false
9619 3454 Acked Read 10 Done Sent Done false
9620 3455 Acked Write 9 Done Sent Done false
9621 3456 Acked Write 9 Done Sent Done false
9622 3457 Acked Flush 0 Done Sent Done false
9623 3458 Acked Read 1 Done Sent Done false
9624 3459 Acked Write 2 Done Sent Done false
9625 3460 Acked Read 2 Done Sent Done false
9626 3461 Acked Read 3 Done Sent Done false
9627 3462 Acked Read 4 Done Sent Done false
9628 3463 Acked Write 10 Done Sent Done false
9629 3464 Acked Read 10 Done Sent Done false
9630 3465 Acked Flush 0 Done Sent Done false
9631 3466 Acked Read 6 Done Sent Done false
9632 3467 Acked Read 1 Done Sent Done false
9633 3468 Acked Write 6 Done Sent Done false
9634 3469 Acked Flush 0 Done Sent Done false
9635 3470 Acked Write 5 Done Sent Done false
9636 3471 Acked Write 1 Done Sent Done false
9637 3472 Acked Write 2 Done Sent Done false
9638 3473 Acked Write 3 Done Sent Done false
9639 3474 Acked Read 3 Done Sent Done false
9640 3475 Acked Flush 0 Done Sent Done false
9641 3476 Acked Write 8 Done Sent Done false
9642 3477 Acked Read 1 Done Sent Done false
9643 3478 Acked Read 8 Done Sent Done false
9644 3479 Acked Read 10 Done Sent Done false
9645 3480 Acked Write 1 Done Sent Done false
9646 3481 Acked Read 1 Done Sent Done false
9647 3482 Acked Read 7 Done Sent Done false
9648 3483 Acked Write 10 Done Sent Done false
9649 3484 Acked Flush 0 Done Sent Done false
9650 3485 Acked Write 4 Done Sent Done false
9651 3486 Acked Read 6 Done Sent Done false
9652 3487 Acked Read 9 Done Sent Done false
9653 3488 Acked Flush 0 Done Sent Done false
9654 3489 Acked Write 7 Done Sent Done false
9655 3490 Acked Write 2 Done Sent Done false
9656 3491 Acked Write 3 Done Sent Done false
9657 3492 Acked Read 10 Done Sent Done false
9658 3493 Acked Read 3 Done Sent Done false
9659 3494 Acked Write 8 Done Sent Done false
9660 3495 Acked Flush 0 Done Sent Done false
9661 3496 Acked Write 9 Done Sent Done false
9662 3497 Acked Read 8 Done Sent Done false
9663 3498 Acked Write 6 Done Sent Done false
9664 3499 Acked Read 7 Done Sent Done false
9665 3500 Acked Write 1 Done Sent Done false
9666 3501 Acked Flush 0 Done Sent Done false
9667 3502 Acked Read 1 Done Sent Done false
9668 3503 Acked Write 2 Done Sent Done false
9669 3504 Acked Write 7 Done Sent Done false
9670 3505 Acked Write 5 Done Sent Done false
9671 3506 Acked Write 2 Done Sent Done false
9672 3507 Acked Read 3 Done Sent Done false
9673 3508 Acked Flush 0 Done Sent Done false
9674 3509 Acked Read 10 Done Sent Done false
9675 3510 Acked Write 4 Done Sent Done false
9676 3511 Acked Write 2 Done Sent Done false
9677 3512 Acked Write 5 Done Sent Done false
9678 3513 Acked Write 4 Done Sent Done false
9679 3514 Acked Read 10 Done Sent Done false
9680 3515 Acked Write 9 Done Sent Done false
9681 3516 Acked Write 8 Done Sent Done false
9682 3517 Acked Read 9 Done Sent Done false
9683 3518 Acked Read 2 Done Sent Done false
9684 3519 Acked Read 2 Done Sent Done false
9685 3520 Acked Read 5 Done Sent Done false
9686 3521 Acked Read 4 Done Sent Done false
9687 3522 Acked Write 7 Done Sent Done false
9688 3523 Acked Write 6 Done Sent Done false
9689 3524 Acked Flush 0 Done Sent Done false
9690 3525 Acked Flush 0 Done Sent Done false
9691 3526 Acked Read 9 Done Sent Done false
9692 3527 Acked Flush 0 Done Sent Done false
9693 3528 Acked Read 7 Done Sent Done false
9694 3529 Acked Read 4 Done Sent Done false
9695 3530 Acked Read 6 Done Sent Done false
9696 3531 Acked Read 6 Done Sent Done false
9697 3532 Acked Read 9 Done Sent Done false
9698 3533 Acked Read 2 Done Sent Done false
9699 3534 Acked Read 3 Done Sent Done false
9700 3535 Acked Read 8 Done Sent Done false
9701 3536 Acked Write 1 Done Sent Done false
9702 3537 Acked Read 10 Done Sent Done false
9703 3538 Acked Read 10 Done Sent Done false
9704 3539 Acked Flush 0 Done Sent Done false
9705 3540 Acked Read 8 Done Sent Done false
9706 3541 Acked Write 1 Done Sent Done false
9707 3542 Acked Write 1 Done Sent Done false
9708 3543 Acked Write 10 Done Sent Done false
9709 3544 Acked Read 7 Done Sent Done false
9710 3545 Acked Read 8 Done Sent Done false
9711 3546 Acked Read 4 Done Sent Done false
9712 3547 Acked Write 8 Done Sent Done false
9713 3548 Acked Read 10 Done Sent Done false
9714 3549 Acked Read 5 Done Sent Done false
9715 3550 Acked Read 10 Done Sent Done false
9716 3551 Acked Read 9 Done Sent Done false
9717 3552 Acked Read 10 Done Sent Done false
9718 3553 Acked Read 9 Done Sent Done false
9719 3554 Acked Write 2 Done Sent Done false
9720 3555 Acked Read 5 Done Sent Done false
9721 3556 Acked Write 10 Done Sent Done false
9722 3557 Acked Write 10 Done Sent Done false
9723 3558 Acked Read 5 Done Sent Done false
9724 3559 Acked Read 5 Done Sent Done false
9725 3560 Acked Read 2 Done Sent Done false
9726 3561 Acked Read 1 Done Sent Done false
9727 3562 Acked Write 2 Done Sent Done false
9728 3563 Acked Read 7 Done Sent Done false
9729 3564 Acked Read 6 Done Sent Done false
9730 3565 Acked Read 3 Done Sent Done false
9731 3566 Acked Read 5 Done Sent Done false
9732 3567 Acked Write 3 Done Sent Done false
9733 3568 Acked Write 4 Done Sent Done false
9734 3569 Acked Write 1 Done Sent Done false
9735 3570 Acked Write 7 Done Sent Done false
9736 3571 Acked Write 4 Done Sent Done false
9737 3572 Acked Read 8 Done Sent Done false
9738 3573 Acked Write 6 Done Sent Done false
9739 3574 Acked Write 2 Done Sent Done false
9740 3575 Acked Read 7 Done Sent Done false
9741 3576 Acked Read 4 Done Sent Done false
9742 3577 Acked Write 7 Done Sent Done false
9743 3578 Acked Read 7 Done Sent Done false
9744 3579 Acked Write 1 Done Sent Done false
9745 3580 Acked Flush 0 Done Sent Done false
9746 3581 Acked Write 4 Done Sent Done false
9747 3582 Acked Write 1 Done Sent Done false
9748 3583 Acked Write 2 Done Sent Done false
9749 3584 Acked Read 3 Done Sent Done false
9750 3585 Acked Write 9 Done Sent Done false
9751 3586 Acked Read 7 Done Sent Done false
9752 3587 Acked Read 3 Done Sent Done false
9753 3588 Acked Write 6 Done Sent Done false
9754 3589 Acked Read 4 Done Sent Done false
9755 3590 Acked Write 5 Done Sent Done false
9756 3591 Acked Read 6 Done Sent Done false
9757 3592 Acked Flush 0 Done Sent Done false
9758 3593 Acked Read 3 Done Sent Done false
9759 3594 Acked Read 4 Done Sent Done false
9760 3595 Acked Write 8 Done Sent Done false
9761 3596 Acked Read 9 Done Sent Done false
9762 3597 Acked Read 2 Done Sent Done false
9763 3598 Acked Read 4 Done Sent Done false
9764 3599 Acked Read 6 Done Sent Done false
9765 3600 Acked Write 4 Done Sent Done false
9766 3601 Acked Read 8 Done Sent Done false
9767 3602 Acked Read 8 Done Sent Done false
9768 3603 Acked Write 7 Done Sent Done false
9769 3604 Acked Read 2 Done Sent Done false
9770 3605 Acked Read 1 Done Sent Done false
9771 3606 Acked Read 5 Done Sent Done false
9772 3607 Acked Read 4 Done Sent Done false
9773 3608 Acked Flush 0 Done Sent Done false
9774 3609 Acked Write 2 Done Sent Done false
9775 3610 Acked Write 4 Done Sent Done false
9776 3611 Acked Write 4 Done Sent Done false
9777 3612 Acked Read 9 Done Sent Done false
9778 3613 Acked Read 4 Done Sent Done false
9779 3614 Acked Write 9 Done Sent Done false
9780 3615 Acked Read 8 Done Sent Done false
9781 3616 Acked Read 9 Done Sent Done false
9782 3617 Acked Flush 0 Done Sent Done false
9783 3618 Acked Write 2 Done Sent Done false
9784 3619 Acked Flush 0 Done Sent Done false
9785 3620 Acked Read 9 Done Sent Done false
9786 3621 Acked Read 1 Done Sent Done false
9787 3622 Acked Read 1 Done Sent Done false
9788 3623 Acked Write 8 Done Sent Done false
9789 3624 Acked Read 6 Done Sent Done false
9790 3625 Acked Write 10 Done Sent Done false
9791 3626 Acked Write 6 Done Sent Done false
9792 3627 Acked Flush 0 Done Sent Done false
9793 3628 Acked Read 8 Done Sent Done false
9794 3629 Acked Write 1 Done Sent Done false
9795 3630 Acked Read 10 Done Sent Done false
9796 3631 Acked Write 6 Done Sent Done false
9797 3632 Acked Write 5 Done Sent Done false
9798 3633 Acked Flush 0 Done Sent Done false
9799 3634 Acked Write 8 Done Sent Done false
9800 3635 Acked Write 10 Done Sent Done false
9801 3636 Acked Write 4 Done Sent Done false
9802 3637 Acked Read 8 Done Sent Done false
9803 3638 Acked Flush 0 Done Sent Done false
9804 3639 Acked Read 4 Done Sent Done false
9805 3640 Acked Write 4 Done Sent Done false
9806 3641 Acked Read 6 Done Sent Done false
9807 3642 Acked Flush 0 Done Sent Done false
9808 3643 Acked Read 4 Done Sent Done false
9809 3644 Acked Write 2 Done Sent Done false
9810 3645 Acked Write 9 Done Sent Done false
9811 3646 Acked Write 7 Done Sent Done false
9812 3647 Acked Read 3 Done Sent Done false
9813 3648 Acked Flush 0 Done Sent Done false
9814 3649 Acked Write 6 Done Sent Done false
9815 3650 Acked Read 1 Done Sent Done false
9816 3651 Acked Write 8 Done Sent Done false
9817 3652 Acked Read 4 Done Sent Done false
9818 3653 Acked Write 7 Done Sent Done false
9819 3654 Acked Write 6 Done Sent Done false
9820 3655 Acked Write 6 Done Sent Done false
9821 3656 Acked Read 2 Done Sent Done false
9822 3657 Acked Write 7 Done Sent Done false
9823 3658 Acked Write 3 Done Sent Done false
9824 3659 Acked Write 5 Done Sent Done false
9825 3660 Acked Read 3 Done Sent Done false
9826 3661 Acked Read 7 Done Sent Done false
9827 3662 Acked Read 7 Done Sent Done false
9828 3663 Acked Write 3 Done Sent Done false
9829 3664 Acked Read 2 Done Sent Done false
9830 3665 Acked Read 7 Done Sent Done false
9831 3666 Acked Write 1 Done Sent Done false
9832 3667 Acked Read 4 Done Sent Done false
9833 3668 Acked Flush 0 Done Sent Done false
9834 3669 Acked Write 8 Done Sent Done false
9835 3670 Acked Flush 0 Done Sent Done false
9836 3671 Acked Write 8 Done Sent Done false
9837 3672 Acked Flush 0 Done Sent Done false
9838 3673 Acked Flush 0 Done Sent Done false
9839 3674 Acked Read 7 Done Sent Done false
9840 3675 Acked Write 9 Done Sent Done false
9841 3676 Acked Read 9 Done Sent Done false
9842 3677 Acked Read 10 Done Sent Done false
9843 3678 Acked Flush 0 Done Sent Done false
9844 3679 Acked Read 8 Done Sent Done false
9845 3680 Acked Flush 0 Done Sent Done false
9846 3681 Acked Read 5 Done Sent Done false
9847 3682 Acked Read 7 Done Sent Done false
9848 3683 Acked Read 9 Done Sent Done false
9849 3684 Acked Read 2 Done Sent Done false
9850 3685 Acked Write 3 Done Sent Done false
9851 3686 Acked Read 6 Done Sent Done false
9852 3687 Acked Read 8 Done Sent Done false
9853 3688 Acked Read 8 Done Sent Done false
9854 3689 Acked Flush 0 Done Sent Done false
9855 3690 Acked Write 1 Done Sent Done false
9856 3691 Acked Write 6 Done Sent Done false
9857 3692 Acked Write 3 Done Sent Done false
9858 3693 Acked Read 10 Done Sent Done false
9859 3694 Acked Flush 0 Done Sent Done false
9860 3695 Acked Read 6 Done Sent Done false
9861 3696 Acked Write 6 Done Sent Done false
9862 3697 Acked Read 5 Done Sent Done false
9863 3698 Acked Flush 0 Done Sent Done false
9864 3699 Acked Read 9 Done Sent Done false
9865 3700 Acked Read 4 Done Sent Done false
9866 3701 Acked Flush 0 Done Sent Done false
9867 3702 Acked Read 6 Done Sent Done false
9868 3703 Acked Write 4 Done Sent Done false
9869 3704 Acked Read 1 Done Sent Done false
9870 3705 Acked Flush 0 Done Sent Done false
9871 3706 Acked Read 10 Done Sent Done false
9872 3707 Acked Read 4 Done Sent Done false
9873 3708 Acked Read 1 Done Sent Done false
9874 3709 Acked Write 2 Done Sent Done false
9875 3710 Acked Read 1 Done Sent Done false
9876 3711 Acked Read 1 Done Sent Done false
9877 3712 Acked Write 2 Done Sent Done false
9878 3713 Acked Read 4 Done Sent Done false
9879 3714 Acked Write 3 Done Sent Done false
9880 3715 Acked Read 5 Done Sent Done false
9881 3716 Acked Read 8 Done Sent Done false
9882 3717 Acked Flush 0 Done Sent Done false
9883 STATES DS:0 DS:1 DS:2 TOTAL
9884 Sent 0 302 0 302
9885 Done 302 0 302 604
9886 Skipped 0 0 0 0
9887 Error 0 0 0 0
9888 Last Flush: 3717 3415 3717
9889 Downstairs last five completed: 3415 3414 3413 3412 3411
9890 Upstairs last five completed: 3717 3716 3715 3714 3713
9891 CLIENT: Up:0 ds:302 act:2
9892 ----------------------------------------------------------------
9893 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
9894 States: Active Offline Active
9895 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9896 3416 Acked Write 9 Done Sent Done false
9897 3417 Acked Write 2 Done Sent Done false
9898 3418 Acked Read 2 Done Sent Done false
9899 3419 Acked Flush 0 Done Sent Done false
9900 3420 Acked Write 8 Done Sent Done false
9901 3421 Acked Read 3 Done Sent Done false
9902 3422 Acked Read 1 Done Sent Done false
9903 3423 Acked Read 4 Done Sent Done false
9904 3424 Acked Flush 0 Done Sent Done false
9905 3425 Acked Read 1 Done Sent Done false
9906 3426 Acked Write 4 Done Sent Done false
9907 3427 Acked Read 2 Done Sent Done false
9908 3428 Acked Write 4 Done Sent Done false
9909 3429 Acked Write 7 Done Sent Done false
9910 3430 Acked Read 6 Done Sent Done false
9911 3431 Acked Read 10 Done Sent Done false
9912 3432 Acked Read 5 Done Sent Done false
9913 3433 Acked Read 7 Done Sent Done false
9914 3434 Acked Write 9 Done Sent Done false
9915 3435 Acked Read 1 Done Sent Done false
9916 3436 Acked Write 7 Done Sent Done false
9917 3437 Acked Read 8 Done Sent Done false
9918 3438 Acked Write 5 Done Sent Done false
9919 3439 Acked Read 3 Done Sent Done false
9920 3440 Acked Write 5 Done Sent Done false
9921 3441 Acked Flush 0 Done Sent Done false
9922 3442 Acked Write 8 Done Sent Done false
9923 3443 Acked Write 3 Done Sent Done false
9924 3444 Acked Write 2 Done Sent Done false
9925 3445 Acked Read 10 Done Sent Done false
9926 3446 Acked Write 5 Done Sent Done false
9927 3447 Acked Read 2 Done Sent Done false
9928 3448 Acked Read 10 Done Sent Done false
9929 3449 Acked Write 7 Done Sent Done false
9930 3450 Acked Read 2 Done Sent Done false
9931 3451 Acked Write 3 Done Sent Done false
9932 3452 Acked Write 5 Done Sent Done false
9933 3453 Acked Write 1 Done Sent Done false
9934 3454 Acked Read 10 Done Sent Done false
9935 3455 Acked Write 9 Done Sent Done false
9936 3456 Acked Write 9 Done Sent Done false
9937 3457 Acked Flush 0 Done Sent Done false
9938 3458 Acked Read 1 Done Sent Done false
9939 3459 Acked Write 2 Done Sent Done false
9940 3460 Acked Read 2 Done Sent Done false
9941 3461 Acked Read 3 Done Sent Done false
9942 3462 Acked Read 4 Done Sent Done false
9943 3463 Acked Write 10 Done Sent Done false
9944 3464 Acked Read 10 Done Sent Done false
9945 3465 Acked Flush 0 Done Sent Done false
9946 3466 Acked Read 6 Done Sent Done false
9947 3467 Acked Read 1 Done Sent Done false
9948 3468 Acked Write 6 Done Sent Done false
9949 3469 Acked Flush 0 Done Sent Done false
9950 3470 Acked Write 5 Done Sent Done false
9951 3471 Acked Write 1 Done Sent Done false
9952 3472 Acked Write 2 Done Sent Done false
9953 3473 Acked Write 3 Done Sent Done false
9954 3474 Acked Read 3 Done Sent Done false
9955 3475 Acked Flush 0 Done Sent Done false
9956 3476 Acked Write 8 Done Sent Done false
9957 3477 Acked Read 1 Done Sent Done false
9958 3478 Acked Read 8 Done Sent Done false
9959 3479 Acked Read 10 Done Sent Done false
9960 3480 Acked Write 1 Done Sent Done false
9961 3481 Acked Read 1 Done Sent Done false
9962 3482 Acked Read 7 Done Sent Done false
9963 3483 Acked Write 10 Done Sent Done false
9964 3484 Acked Flush 0 Done Sent Done false
9965 3485 Acked Write 4 Done Sent Done false
9966 3486 Acked Read 6 Done Sent Done false
9967 3487 Acked Read 9 Done Sent Done false
9968 3488 Acked Flush 0 Done Sent Done false
9969 3489 Acked Write 7 Done Sent Done false
9970 3490 Acked Write 2 Done Sent Done false
9971 3491 Acked Write 3 Done Sent Done false
9972 3492 Acked Read 10 Done Sent Done false
9973 3493 Acked Read 3 Done Sent Done false
9974 3494 Acked Write 8 Done Sent Done false
9975 3495 Acked Flush 0 Done Sent Done false
9976 3496 Acked Write 9 Done Sent Done false
9977 3497 Acked Read 8 Done Sent Done false
9978 3498 Acked Write 6 Done Sent Done false
9979 3499 Acked Read 7 Done Sent Done false
9980 3500 Acked Write 1 Done Sent Done false
9981 3501 Acked Flush 0 Done Sent Done false
9982 3502 Acked Read 1 Done Sent Done false
9983 3503 Acked Write 2 Done Sent Done false
9984 3504 Acked Write 7 Done Sent Done false
9985 3505 Acked Write 5 Done Sent Done false
9986 3506 Acked Write 2 Done Sent Done false
9987 3507 Acked Read 3 Done Sent Done false
9988 3508 Acked Flush 0 Done Sent Done false
9989 3509 Acked Read 10 Done Sent Done false
9990 3510 Acked Write 4 Done Sent Done false
9991 3511 Acked Write 2 Done Sent Done false
9992 3512 Acked Write 5 Done Sent Done false
9993 3513 Acked Write 4 Done Sent Done false
9994 3514 Acked Read 10 Done Sent Done false
9995 3515 Acked Write 9 Done Sent Done false
9996 3516 Acked Write 8 Done Sent Done false
9997 3517 Acked Read 9 Done Sent Done false
9998 3518 Acked Read 2 Done Sent Done false
9999 3519 Acked Read 2 Done Sent Done false
10000 3520 Acked Read 5 Done Sent Done false
10001 3521 Acked Read 4 Done Sent Done false
10002 3522 Acked Write 7 Done Sent Done false
10003 3523 Acked Write 6 Done Sent Done false
10004 3524 Acked Flush 0 Done Sent Done false
10005 3525 Acked Flush 0 Done Sent Done false
10006 3526 Acked Read 9 Done Sent Done false
10007 3527 Acked Flush 0 Done Sent Done false
10008 3528 Acked Read 7 Done Sent Done false
10009 3529 Acked Read 4 Done Sent Done false
10010 3530 Acked Read 6 Done Sent Done false
10011 3531 Acked Read 6 Done Sent Done false
10012 3532 Acked Read 9 Done Sent Done false
10013 3533 Acked Read 2 Done Sent Done false
10014 3534 Acked Read 3 Done Sent Done false
10015 3535 Acked Read 8 Done Sent Done false
10016 3536 Acked Write 1 Done Sent Done false
10017 3537 Acked Read 10 Done Sent Done false
10018 3538 Acked Read 10 Done Sent Done false
10019 3539 Acked Flush 0 Done Sent Done false
10020 3540 Acked Read 8 Done Sent Done false
10021 3541 Acked Write 1 Done Sent Done false
10022 3542 Acked Write 1 Done Sent Done false
10023 3543 Acked Write 10 Done Sent Done false
10024 3544 Acked Read 7 Done Sent Done false
10025 3545 Acked Read 8 Done Sent Done false
10026 3546 Acked Read 4 Done Sent Done false
10027 3547 Acked Write 8 Done Sent Done false
10028 3548 Acked Read 10 Done Sent Done false
10029 3549 Acked Read 5 Done Sent Done false
10030 3550 Acked Read 10 Done Sent Done false
10031 3551 Acked Read 9 Done Sent Done false
10032 3552 Acked Read 10 Done Sent Done false
10033 3553 Acked Read 9 Done Sent Done false
10034 3554 Acked Write 2 Done Sent Done false
10035 3555 Acked Read 5 Done Sent Done false
10036 3556 Acked Write 10 Done Sent Done false
10037 3557 Acked Write 10 Done Sent Done false
10038 3558 Acked Read 5 Done Sent Done false
10039 3559 Acked Read 5 Done Sent Done false
10040 3560 Acked Read 2 Done Sent Done false
10041 3561 Acked Read 1 Done Sent Done false
10042 3562 Acked Write 2 Done Sent Done false
10043 3563 Acked Read 7 Done Sent Done false
10044 3564 Acked Read 6 Done Sent Done false
10045 3565 Acked Read 3 Done Sent Done false
10046 3566 Acked Read 5 Done Sent Done false
10047 3567 Acked Write 3 Done Sent Done false
10048 3568 Acked Write 4 Done Sent Done false
10049 3569 Acked Write 1 Done Sent Done false
10050 3570 Acked Write 7 Done Sent Done false
10051 3571 Acked Write 4 Done Sent Done false
10052 3572 Acked Read 8 Done Sent Done false
10053 3573 Acked Write 6 Done Sent Done false
10054 3574 Acked Write 2 Done Sent Done false
10055 3575 Acked Read 7 Done Sent Done false
10056 3576 Acked Read 4 Done Sent Done false
10057 3577 Acked Write 7 Done Sent Done false
10058 3578 Acked Read 7 Done Sent Done false
10059 3579 Acked Write 1 Done Sent Done false
10060 3580 Acked Flush 0 Done Sent Done false
10061 3581 Acked Write 4 Done Sent Done false
10062 3582 Acked Write 1 Done Sent Done false
10063 3583 Acked Write 2 Done Sent Done false
10064 3584 Acked Read 3 Done Sent Done false
10065 3585 Acked Write 9 Done Sent Done false
10066 3586 Acked Read 7 Done Sent Done false
10067 3587 Acked Read 3 Done Sent Done false
10068 3588 Acked Write 6 Done Sent Done false
10069 3589 Acked Read 4 Done Sent Done false
10070 3590 Acked Write 5 Done Sent Done false
10071 3591 Acked Read 6 Done Sent Done false
10072 3592 Acked Flush 0 Done Sent Done false
10073 3593 Acked Read 3 Done Sent Done false
10074 3594 Acked Read 4 Done Sent Done false
10075 3595 Acked Write 8 Done Sent Done false
10076 3596 Acked Read 9 Done Sent Done false
10077 3597 Acked Read 2 Done Sent Done false
10078 3598 Acked Read 4 Done Sent Done false
10079 3599 Acked Read 6 Done Sent Done false
10080 3600 Acked Write 4 Done Sent Done false
10081 3601 Acked Read 8 Done Sent Done false
10082 3602 Acked Read 8 Done Sent Done false
10083 3603 Acked Write 7 Done Sent Done false
10084 3604 Acked Read 2 Done Sent Done false
10085 3605 Acked Read 1 Done Sent Done false
10086 3606 Acked Read 5 Done Sent Done false
10087 3607 Acked Read 4 Done Sent Done false
10088 3608 Acked Flush 0 Done Sent Done false
10089 3609 Acked Write 2 Done Sent Done false
10090 3610 Acked Write 4 Done Sent Done false
10091 3611 Acked Write 4 Done Sent Done false
10092 3612 Acked Read 9 Done Sent Done false
10093 3613 Acked Read 4 Done Sent Done false
10094 3614 Acked Write 9 Done Sent Done false
10095 3615 Acked Read 8 Done Sent Done false
10096 3616 Acked Read 9 Done Sent Done false
10097 3617 Acked Flush 0 Done Sent Done false
10098 3618 Acked Write 2 Done Sent Done false
10099 3619 Acked Flush 0 Done Sent Done false
10100 3620 Acked Read 9 Done Sent Done false
10101 3621 Acked Read 1 Done Sent Done false
10102 3622 Acked Read 1 Done Sent Done false
10103 3623 Acked Write 8 Done Sent Done false
10104 3624 Acked Read 6 Done Sent Done false
10105 3625 Acked Write 10 Done Sent Done false
10106 3626 Acked Write 6 Done Sent Done false
10107 3627 Acked Flush 0 Done Sent Done false
10108 3628 Acked Read 8 Done Sent Done false
10109 3629 Acked Write 1 Done Sent Done false
10110 3630 Acked Read 10 Done Sent Done false
10111 3631 Acked Write 6 Done Sent Done false
10112 3632 Acked Write 5 Done Sent Done false
10113 3633 Acked Flush 0 Done Sent Done false
10114 3634 Acked Write 8 Done Sent Done false
10115 3635 Acked Write 10 Done Sent Done false
10116 3636 Acked Write 4 Done Sent Done false
10117 3637 Acked Read 8 Done Sent Done false
10118 3638 Acked Flush 0 Done Sent Done false
10119 3639 Acked Read 4 Done Sent Done false
10120 3640 Acked Write 4 Done Sent Done false
10121 3641 Acked Read 6 Done Sent Done false
10122 3642 Acked Flush 0 Done Sent Done false
10123 3643 Acked Read 4 Done Sent Done false
10124 3644 Acked Write 2 Done Sent Done false
10125 3645 Acked Write 9 Done Sent Done false
10126 3646 Acked Write 7 Done Sent Done false
10127 3647 Acked Read 3 Done Sent Done false
10128 3648 Acked Flush 0 Done Sent Done false
10129 3649 Acked Write 6 Done Sent Done false
10130 3650 Acked Read 1 Done Sent Done false
10131 3651 Acked Write 8 Done Sent Done false
10132 3652 Acked Read 4 Done Sent Done false
10133 3653 Acked Write 7 Done Sent Done false
10134 3654 Acked Write 6 Done Sent Done false
10135 3655 Acked Write 6 Done Sent Done false
10136 3656 Acked Read 2 Done Sent Done false
10137 3657 Acked Write 7 Done Sent Done false
10138 3658 Acked Write 3 Done Sent Done false
10139 3659 Acked Write 5 Done Sent Done false
10140 3660 Acked Read 3 Done Sent Done false
10141 3661 Acked Read 7 Done Sent Done false
10142 3662 Acked Read 7 Done Sent Done false
10143 3663 Acked Write 3 Done Sent Done false
10144 3664 Acked Read 2 Done Sent Done false
10145 3665 Acked Read 7 Done Sent Done false
10146 3666 Acked Write 1 Done Sent Done false
10147 3667 Acked Read 4 Done Sent Done false
10148 3668 Acked Flush 0 Done Sent Done false
10149 3669 Acked Write 8 Done Sent Done false
10150 3670 Acked Flush 0 Done Sent Done false
10151 3671 Acked Write 8 Done Sent Done false
10152 3672 Acked Flush 0 Done Sent Done false
10153 3673 Acked Flush 0 Done Sent Done false
10154 3674 Acked Read 7 Done Sent Done false
10155 3675 Acked Write 9 Done Sent Done false
10156 3676 Acked Read 9 Done Sent Done false
10157 3677 Acked Read 10 Done Sent Done false
10158 3678 Acked Flush 0 Done Sent Done false
10159 3679 Acked Read 8 Done Sent Done false
10160 3680 Acked Flush 0 Done Sent Done false
10161 3681 Acked Read 5 Done Sent Done false
10162 3682 Acked Read 7 Done Sent Done false
10163 3683 Acked Read 9 Done Sent Done false
10164 3684 Acked Read 2 Done Sent Done false
10165 3685 Acked Write 3 Done Sent Done false
10166 3686 Acked Read 6 Done Sent Done false
10167 3687 Acked Read 8 Done Sent Done false
10168 3688 Acked Read 8 Done Sent Done false
10169 3689 Acked Flush 0 Done Sent Done false
10170 3690 Acked Write 1 Done Sent Done false
10171 3691 Acked Write 6 Done Sent Done false
10172 3692 Acked Write 3 Done Sent Done false
10173 3693 Acked Read 10 Done Sent Done false
10174 3694 Acked Flush 0 Done Sent Done false
10175 3695 Acked Read 6 Done Sent Done false
10176 3696 Acked Write 6 Done Sent Done false
10177 3697 Acked Read 5 Done Sent Done false
10178 3698 Acked Flush 0 Done Sent Done false
10179 3699 Acked Read 9 Done Sent Done false
10180 3700 Acked Read 4 Done Sent Done false
10181 3701 Acked Flush 0 Done Sent Done false
10182 3702 Acked Read 6 Done Sent Done false
10183 3703 Acked Write 4 Done Sent Done false
10184 3704 Acked Read 1 Done Sent Done false
10185 3705 Acked Flush 0 Done Sent Done false
10186 3706 Acked Read 10 Done Sent Done false
10187 3707 Acked Read 4 Done Sent Done false
10188 3708 Acked Read 1 Done Sent Done false
10189 3709 Acked Write 2 Done Sent Done false
10190 3710 Acked Read 1 Done Sent Done false
10191 3711 Acked Read 1 Done Sent Done false
10192 3712 Acked Write 2 Done Sent Done false
10193 3713 Acked Read 4 Done Sent Done false
10194 3714 Acked Write 3 Done Sent Done false
10195 3715 Acked Read 5 Done Sent Done false
10196 3716 Acked Read 8 Done Sent Done false
10197 3717 Acked Flush 0 Done Sent Done false
10198 STATES DS:0 DS:1 DS:2 TOTAL
10199 Sent 0 302 0 302
10200 Done 302 0 302 604
10201 Skipped 0 0 0 0
10202 Error 0 0 0 0
10203 Last Flush: 3717 3415 3717
10204 Downstairs last five completed: 3415 3414 3413 3412 3411
10205 Upstairs last five completed: 3717 3716 3715 3714 3713
10206 CLIENT: Up:0 ds:302 act:2
102072024-11-01T16:52:38.626ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102082024-11-01T16:52:38.627ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:45206) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102092024-11-01T16:52:38.627ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102102024-11-01T16:52:38.628ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102112024-11-01T16:52:38.628ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102122024-11-01T16:52:38.628ZINFOcrucible: send last flush ID to this DS: 3415 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102132024-11-01T16:52:38.628ZINFOcrucible: Replied this last flush ID: 3415 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102142024-11-01T16:52:38.628ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102152024-11-01T16:52:38.628ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102162024-11-01T16:52:38.628ZINFOcrucible: [1] client re-new 302 jobs since flush 3415 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102172024-11-01T16:52:38.629ZINFOcrucible: [1] Replayed 302 jobs since flush: 3415 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
10218 ----------------------------------------------------------------
10219 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10220 Downstairs last five completed: 3717 3716 3715 3714 3713
10221 Upstairs last five completed: 3717 3716 3715 3714 3713
10222 CLIENT: Up:0 ds:0 act:3
10223 Replay: All jobs finished
10224 Main task has work to do, go find it
10225 got dsc Stop(2)
10226 stop 2
10227 Nov 01 16:52:38.926 INFO request completed, latency_us: 303, response_code: 204, uri: /stop/cid/2, method: POST, [8830] Got stop action so:false kr:false
10228 req_id: fc0125b7-6fe7-4260-8732-0e633eb892ba, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
102292024-11-01T16:52:38.926ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102302024-11-01T16:52:38.926ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102312024-11-01T16:52:38.926ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102322024-11-01T16:52:38.926ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
102332024-11-01T16:52:38.927ZINFOcrucible: client task is exiting = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
10234 Nov 01 16:52:38.927 INFO request completed, latency_us: 170, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5aba9636-e80e-4a9a-8f06-d5b543bbcd07, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
10235 [8830] Exited with: ExitStatus(unix_wait_status(9))
10236 [8830][2] reports Exit
10237 Nov 01 16:52:42.928 INFO request completed, latency_us: 221, response_code: 200, uri: /state/cid/2, method: GET, req_id: c318b837-3081-4656-a45f-10e8ccd7aa41, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
10238 001/300 Read block 1923 len 4096
10239 002/300 Flush
10240 003/300 Flush
10241 004/300 Read block 2633 len 24576
10242 005/300 Read block 793 len 8192
10243 006/300 Read block 281 len 32768
10244 007/300 Read block 4731 len 28672
10245 008/300 Read block 3153 len 40960
10246 009/300 Write block 2058 len 20480 data: 3 3 3 3 4
10247 010/300 Read block 3511 len 36864
10248 011/300 Write block 3860 len 4096 data: 4
10249 012/300 Read block 1480 len 28672
10250 013/300 Flush
10251 014/300 Read block 1322 len 36864
10252 015/300 Flush
10253 016/300 Read block 1841 len 4096
10254 017/300 Write block 4361 len 20480 data: 3 3 2 2 2
10255 018/300 Write block 4506 len 4096 data: 5
10256 019/300 Write block 4051 len 36864 data: 5 5 5 4 4 4 4 4 3
10257 020/300 Flush
10258 021/300 Read block 3967 len 24576
10259 022/300 Read block 527 len 20480
10260 023/300 Write block 1207 len 36864 data: 2 2 2 2 2 2 3 3 3
10261 024/300 Read block 3787 len 20480
10262 025/300 Read block 1907 len 16384
10263 026/300 Write block 1267 len 12288 data: 4 4 4
10264 027/300 Write block 527 len 12288 data: 5 7 9
10265 028/300 Write block 1490 len 40960 data: 3 4 4 3 3 3 4 4 3 3
10266 029/300 Read block 3406 len 8192
10267 030/300 Read block 2153 len 12288
10268 031/300 Write block 3813 len 4096 data: 4
10269 032/300 Flush
10270 033/300 Flush
10271 034/300 Read block 1253 len 32768
10272 035/300 Read block 661 len 12288
10273 036/300 Write block 3857 len 8192 data: 2 3
10274 037/300 Read block 1571 len 40960
10275 038/300 Flush
10276 039/300 Read block 1584 len 24576
10277 040/300 Write block 2542 len 20480 data: 3 3 3 4 4
10278 041/300 Flush
10279 042/300 Write block 3782 len 4096 data: 3
10280 043/300 Flush
10281 044/300 Read block 1400 len 24576
10282 045/300 Read block 2917 len 40960
10283 046/300 Write block 4200 len 32768 data: 3 2 2 2 2 3 3 3
10284 047/300 Read block 1596 len 8192
10285 048/300 Read block 4388 len 4096
10286 049/300 Read block 2533 len 20480
10287 050/300 Flush
10288 051/300 Write block 2293 len 20480 data: 8 6 5 5 6
10289 052/300 Read block 490 len 28672
10290 053/300 Write block 776 len 24576 data: 6 6 6 5 4 4
10291 054/300 Read block 4108 len 16384
10292 055/300 Read block 2635 len 32768
10293 056/300 Read block 4035 len 28672
10294 057/300 Write block 1211 len 12288 data: 3 3 4
10295 058/300 Flush
10296 059/300 Flush
10297 060/300 Write block 4317 len 32768 data: 3 6 6 7 7 6 5 4
10298 061/300 Read block 296 len 28672
10299 062/300 Read block 4014 len 12288
10300 063/300 Read block 2196 len 28672
10301 064/300 Read block 108 len 20480
10302 065/300 Read block 1149 len 40960
10303 066/300 Read block 3087 len 4096
10304 067/300 Flush
10305 068/300 Write block 1104 len 20480 data: 4 4 4 4 3
10306 069/300 Read block 973 len 24576
10307 070/300 Write block 2773 len 12288 data: 3 3 4
10308 071/300 Read block 3811 len 16384
10309 072/300 Write block 4431 len 40960 data: 3 3 3 3 2 2 2 3 3 4
10310 073/300 Read block 2762 len 16384
10311 074/300 Read block 3292 len 16384
10312 075/300 Read block 1553 len 8192
10313 076/300 Read block 1333 len 28672
10314 077/300 Read block 1768 len 12288
10315 078/300 Write block 2531 len 28672 data: 3 3 2 2 3 3 3
10316 079/300 Write block 3567 len 8192 data: 3 4
10317 080/300 Write block 2033 len 16384 data: 4 4 4 3
10318 081/300 Read block 111 len 8192
10319 082/300 Flush
10320 083/300 Write block 3442 len 40960 data: 4 3 4 4 4 4 4 4 4 3
10321 084/300 Write block 4483 len 28672 data: 2 3 3 3 3 3 2
10322 085/300 Read block 4627 len 20480
10323 086/300 Flush
10324 087/300 Read block 3216 len 32768
10325 088/300 Write block 2444 len 12288 data: 2 2 2
10326 089/300 Write block 2106 len 36864 data: 2 3 3 3 3 3 3 3 3
10327 090/300 Read block 1305 len 40960
10328 091/300 Write block 1807 len 8192 data: 5 5
10329 092/300 Write block 853 len 24576 data: 4 4 2 2 2 3
10330 093/300 Flush
10331 094/300 Write block 1331 len 32768 data: 2 2 3 3 4 4 4 3
10332 095/300 Read block 2819 len 12288
10333 096/300 Write block 4754 len 36864 data: 3 3 3 2 2 2 3 3 3
10334 097/300 Write block 4682 len 24576 data: 2 2 3 4 4 3
10335 098/300 Read block 2170 len 36864
10336 099/300 Write block 295 len 4096 data: 2
10337 100/300 Write block 1614 len 28672 data: 2 2 3 3 3 3 3
10338 101/300 Write block 4760 len 40960 data: 4 4 4 3 2 2 2 2 4 4
10339 102/300 Read block 2690 len 28672
10340 103/300 Read block 484 len 28672
10341 104/300 Flush
10342 105/300 Read block 1297 len 8192
10343 106/300 Write block 1314 len 40960 data: 2 2 2 3 3 3 3 3 3 4
10344 107/300 Read block 4712 len 28672
10345 108/300 Write block 2190 len 32768 data: 4 2 2 2 2 2 2 2
10346 109/300 Read block 1816 len 8192
10347 110/300 Write block 3150 len 36864 data: 2 2 2 2 2 2 2 2 2
10348 111/300 Read block 668 len 12288
10349 112/300 Flush
10350 113/300 Flush
10351 114/300 Flush
10352 115/300 Write block 107 len 16384 data: 2 3 2 2
10353 116/300 Write block 4428 len 20480 data: 4 4 4 4 4
10354 117/300 Write block 2371 len 16384 data: 5 5 4 3
10355 118/300 Read block 2701 len 4096
10356 119/300 Read block 2950 len 40960
10357 120/300 Write block 2585 len 28672 data: 3 3 2 2 2 2 2
10358 121/300 Flush
10359 122/300 Read block 4551 len 12288
10360 123/300 Write block 3929 len 16384 data: 2 2 2 3
10361 124/300 Write block 1870 len 32768 data: 3 2 3 3 2 2 3 3
10362 125/300 Write block 4542 len 12288 data: 4 4 4
10363 126/300 Read block 664 len 12288
10364 127/300 Read block 1008 len 32768
10365 128/300 Read block 4746 len 28672
10366 129/300 Write block 1509 len 28672 data: 2 3 3 4 4 2 2
10367 130/300 Read block 2604 len 8192
10368 131/300 Read block 915 len 12288
10369 132/300 Read block 3559 len 16384
10370 133/300 Read block 4029 len 12288
10371 134/300 Flush
10372 135/300 Read block 1122 len 4096
10373 136/300 Read block 3450 len 4096
10374 137/300 Flush
10375 138/300 Flush
10376 139/300 Write block 1269 len 40960 data: 5 3 3 3 2 2 3 3 3 3
10377 140/300 Read block 2034 len 28672
10378 141/300 Read block 1702 len 24576
10379 142/300 Read block 2288 len 20480
10380 143/300 Read block 4966 len 36864
10381 144/300 Write block 4516 len 32768 data: 3 3 2 2 2 3 3 3
10382 145/300 Write block 478 len 36864 data: 5 4 4 4 4 5 4 4 4
10383 146/300 Read block 363 len 8192
10384 147/300 Write block 1804 len 4096 data: 4
10385 148/300 Read block 747 len 36864
10386 149/300 Write block 535 len 12288 data: 5 4 4
10387 150/300 Read block 4353 len 4096
10388 151/300 Write block 3972 len 8192 data: 2 2
10389 152/300 Write block 2929 len 16384 data: 3 4 4 4
10390 153/300 Write block 2584 len 16384 data: 3 4 4 3
10391 154/300 Read block 2172 len 40960
10392 155/300 Read block 368 len 4096
10393 156/300 Write block 1251 len 4096 data: 2
10394 157/300 Read block 3246 len 20480
10395 158/300 Write block 2757 len 40960 data: 3 3 3 3 2 2 2 2 2 3
10396 159/300 Read block 2704 len 36864
10397 160/300 Write block 1162 len 20480 data: 2 2 2 2 2
10398 161/300 Write block 3831 len 16384 data: 5 5 6 5
10399 162/300 Read block 3650 len 4096
10400 163/300 Write block 1583 len 40960 data: 3 3 3 3 3 3 3 3 3 2
10401 164/300 Read block 713 len 28672
10402 165/300 Read block 627 len 8192
10403 166/300 Write block 1254 len 36864 data: 2 2 2 2 2 2 4 3 3
10404 167/300 Read block 4774 len 24576
10405 168/300 Write block 1353 len 32768 data: 4 4 3 2 2 2 2 2
10406 169/300 Read block 3350 len 36864
10407 170/300 Read block 1486 len 20480
10408 171/300 Write block 2210 len 32768 data: 5 6 6 4 4 3 3 2
10409 172/300 Write block 2315 len 12288 data: 3 3 3
10410 173/300 Write block 1056 len 20480 data: 2 2 2 2 2
10411 174/300 Read block 3381 len 20480
10412 175/300 Read block 3984 len 28672
10413 176/300 Read block 2792 len 36864
10414 177/300 Read block 1558 len 16384
10415 178/300 Read block 1627 len 40960
10416 179/300 Read block 1574 len 32768
10417 180/300 Write block 3372 len 4096 data: 4
10418 181/300 Read block 937 len 40960
10419 182/300 Write block 4136 len 8192 data: 4 4
10420 183/300 Write block 4745 len 36864 data: 2 2 2 2 2 2 2 3 3
10421 184/300 Read block 3346 len 24576
10422 185/300 Read block 2751 len 32768
10423 186/300 Write block 218 len 36864 data: 3 3 3 3 3 2 2 2 2
10424 187/300 Read block 3548 len 12288
10425 188/300 Write block 4314 len 36864 data: 3 4 3 4 7 7 8 8 7
10426 189/300 Read block 3645 len 24576
10427 190/300 Read block 4754 len 20480
10428 191/300 Read block 4641 len 36864
10429 192/300 Read block 1334 len 24576
10430 193/300 Flush
10431 194/300 Read block 4886 len 16384
10432 195/300 Read block 501 len 40960
10433 196/300 Flush
10434 197/300 Read block 2821 len 28672
10435 198/300 Flush
10436 199/300 Write block 3006 len 8192 data: 2 2
10437 200/300 Read block 1733 len 20480
10438 201/300 Read block 3618 len 8192
10439 202/300 Read block 2182 len 20480
10440 203/300 Write block 365 len 32768 data: 2 2 3 3 4 5 4 4
10441 204/300 Write block 1872 len 40960 data: 4 4 3 3 4 4 3 4 3 3
10442 205/300 Flush
10443 206/300 Flush
10444 207/300 Read block 1938 len 40960
10445 208/300 Read block 3832 len 8192
10446 209/300 Write block 4417 len 16384 data: 2 2 2 2
10447 210/300 Write block 1172 len 32768 data: 3 4 4 5 6 6 6 6
10448 211/300 Read block 2131 len 36864
10449 212/300 Write block 2535 len 40960 data: 4 4 4 3 3 2 3 4 4 4
10450 213/300 Read block 428 len 36864
10451 214/300 Read block 4036 len 20480
10452 215/300 Flush
10453 216/300 Read block 2986 len 4096
10454 217/300 Read block 1263 len 4096
10455 218/300 Read block 3063 len 20480
10456 219/300 Read block 755 len 40960
10457 220/300 Write block 4066 len 8192 data: 2 2
10458 221/300 Write block 983 len 32768 data: 3 3 2 2 2 2 2 2
10459 222/300 Read block 3157 len 32768
10460 223/300 Read block 176 len 32768
10461 224/300 Read block 4267 len 8192
10462 225/300 Read block 2181 len 12288
10463 226/300 Write block 4604 len 20480 data: 2 2 2 2 3
10464 227/300 Read block 611 len 16384
10465 228/300 Read block 1306 len 32768
10466 229/300 Read block 4190 len 20480
10467 230/300 Read block 3400 len 32768
10468 231/300 Read block 1142 len 28672
10469 232/300 Read block 2426 len 36864
10470 233/300 Read block 3290 len 24576
10471 234/300 Read block 1553 len 36864
10472 235/300 Read block 3056 len 32768
10473 236/300 Write block 4512 len 32768 data: 5 5 4 4 4 4 3 3
10474 237/300 Read block 912 len 28672
10475 238/300 Read block 2373 len 28672
10476 239/300 Write block 2914 len 40960 data: 2 2 2 2 2 2 2 2 2 2
10477 240/300 Write block 4657 len 24576 data: 2 2 3 3 3 3
10478 241/300 Write block 920 len 36864 data: 5 4 5 4 4 5 5 6 2
10479 242/300 Read block 331 len 12288
10480 243/300 Read block 169 len 12288
10481 244/300 Flush
10482 245/300 Read block 1849 len 8192
10483 246/300 Write block 2487 len 16384 data: 3 4 4 3
10484 247/300 Write block 4080 len 16384 data: 3 3 3 2
10485 248/300 Flush
10486 249/300 Read block 406 len 28672
10487 250/300 Read block 3571 len 16384
10488 251/300 Read block 2877 len 8192
10489 252/300 Read block 4507 len 32768
10490 253/300 Read block 4330 len 12288
10491 254/300 Write block 4539 len 36864 data: 3 3 3 5 5 5 2 3 3
10492 255/300 Flush
10493 256/300 Write block 2644 len 36864 data: 4 4 4 4 4 4 3 3 5
10494 257/300 Write block 659 len 28672 data: 5 5 5 4 2 2 2
10495 258/300 Write block 2674 len 24576 data: 3 3 3 4 3 4
10496 259/300 Write block 240 len 20480 data: 3 4 4 4 4
10497 260/300 Write block 1904 len 16384 data: 3 4 4 4
10498 261/300 Read block 692 len 32768
10499 262/300 Read block 1467 len 16384
10500 263/300 Write block 2617 len 16384 data: 4 3 3 4
10501 264/300 Flush
10502 265/300 Flush
10503 266/300 Read block 3423 len 16384
10504 267/300 Write block 2436 len 12288 data: 3 2 2
10505 268/300 Read block 1870 len 4096
10506 269/300 Write block 2354 len 36864 data: 2 2 3 3 3 3 3 4 4
10507 270/300 Write block 896 len 4096 data: 4
10508 271/300 Read block 1237 len 32768
10509 272/300 Read block 394 len 24576
10510 273/300 Write block 732 len 4096 data: 4
10511 274/300 Write block 808 len 32768 data: 3 3 3 3 3 3 2 2
10512 275/300 Read block 4132 len 36864
10513 276/300 Read block 2476 len 12288
10514 277/300 Read block 4572 len 4096
10515 278/300 Write block 3824 len 32768 data: 3 4 5 5 5 4 4 6
10516 279/300 Write block 3808 len 24576 data: 3 3 2 2 2 5
10517 280/300 Read block 4589 len 4096
10518 281/300 Write block 2973 len 28672 data: 4 4 4 3 3 3 3
10519 282/300 Read block 300 len 4096
10520 283/300 Write block 113 len 16384 data: 2 2 4 4
10521 284/300 Write block 1865 len 40960 data: 3 4 3 3 3 4 3 5 5 4
10522 285/300 Read block 4531 len 16384
10523 286/300 Read block 3470 len 24576
10524 287/300 Write block 2754 len 36864 data: 4 4 4 4 4 4 4 3 3
10525 288/300 Write block 707 len 8192 data: 2 2
10526 289/300 Read block 4184 len 4096
10527 290/300 Flush
10528 291/300 Flush
10529 292/300 Write block 618 len 32768 data: 3 3 3 3 5 4 5 5
10530 293/300 Write block 1514 len 24576 data: 3 3 2 2 2 2
10531 294/300 Write block 4821 len 4096 data: 3
10532 295/300 Write block 1155 len 4096 data: 5
10533 296/300 Read block 3902 len 4096
10534 297/300 Read block 1145 len 16384
10535 298/300 Write block 1144 len 32768 data: 3 3 2 3 3 3 3 4
10536 299/300 Read block 1183 len 8192
10537 300/300 Read block 4758 len 32768
10538 301/300 Read block 289 len 8192
10539 Main task has work to do, go find it
10540 got dsc Start(2)
10541 start 2
10542 Nov 01 16:52:43.216 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/2, method: POST, req_id: 39080440-8616-4256-aa63-0e1c3284e072, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
10543 Replay: started 2, returned:Ok(())
10544 ----------------------------------------------------------------
10545 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
10546 States: Active Active Offline
10547 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10548 3718 Acked Read 1 Done Done Sent false
10549 3719 Acked Flush 0 Done Done Sent false
10550 3720 Acked Flush 0 Done Done Sent false
10551 3721 Acked Read 6 Done Done Sent false
10552 3722 Acked Read 2 Done Done Sent false
10553 3723 Acked Read 8 Done Done Sent false
10554 3724 Acked Read 7 Done Done Sent false
10555 3725 Acked Read 10 Done Done Sent false
10556 3726 Acked Write 5 Done Done Sent false
10557 3727 Acked Read 9 Done Done Sent false
10558 3728 Acked Write 1 Done Done Sent false
10559 3729 Acked Read 7 Done Done Sent false
10560 3730 Acked Flush 0 Done Done Sent false
10561 3731 Acked Read 9 Done Done Sent false
10562 3732 Acked Flush 0 Done Done Sent false
10563 3733 Acked Read 1 Done Done Sent false
10564 3734 Acked Write 5 Done Done Sent false
10565 3735 Acked Write 1 Done Done Sent false
10566 3736 Acked Write 9 Done Done Sent false
10567 3737 Acked Flush 0 Done Done Sent false
10568 3738 Acked Read 6 Done Done Sent false
10569 3739 Acked Read 5 Done Done Sent false
10570 3740 Acked Write 9 Done Done Sent false
10571 3741 Acked Read 5 Done Done Sent false
10572 3742 Acked Read 4 Done Done Sent false
10573 3743 Acked Write 3 Done Done Sent false
10574 3744 Acked Write 3 Done Done Sent false
10575 3745 Acked Write 10 Done Done Sent false
10576 3746 Acked Read 2 Done Done Sent false
10577 3747 Acked Read 3 Done Done Sent false
10578 3748 Acked Write 1 Done Done Sent false
10579 3749 Acked Flush 0 Done Done Sent false
10580 3750 Acked Flush 0 Done Done Sent false
10581 3751 Acked Read 8 Done Done Sent false
10582 3752 Acked Read 3 Done Done Sent false
10583 3753 Acked Write 2 Done Done Sent false
10584 3754 Acked Read 10 Done Done Sent false
10585 3755 Acked Flush 0 Done Done Sent false
10586 3756 Acked Read 6 Done Done Sent false
10587 3757 Acked Write 5 Done Done Sent false
10588 3758 Acked Flush 0 Done Done Sent false
10589 3759 Acked Write 1 Done Done Sent false
10590 3760 Acked Flush 0 Done Done Sent false
10591 3761 Acked Read 6 Done Done Sent false
10592 3762 Acked Read 10 Done Done Sent false
10593 3763 Acked Write 8 Done Done Sent false
10594 3764 Acked Read 2 Done Done Sent false
10595 3765 Acked Read 1 Done Done Sent false
10596 3766 Acked Read 5 Done Done Sent false
10597 3767 Acked Flush 0 Done Done Sent false
10598 3768 Acked Write 5 Done Done Sent false
10599 3769 Acked Read 7 Done Done Sent false
10600 3770 Acked Write 6 Done Done Sent false
10601 3771 Acked Read 4 Done Done Sent false
10602 3772 Acked Read 8 Done Done Sent false
10603 3773 Acked Read 7 Done Done Sent false
10604 3774 Acked Write 3 Done Done Sent false
10605 3775 Acked Flush 0 Done Done Sent false
10606 3776 Acked Flush 0 Done Done Sent false
10607 3777 Acked Write 8 Done Done Sent false
10608 3778 Acked Read 7 Done Done Sent false
10609 3779 Acked Read 3 Done Done Sent false
10610 3780 Acked Read 7 Done Done Sent false
10611 3781 Acked Read 5 Done Done Sent false
10612 3782 Acked Read 10 Done Done Sent false
10613 3783 Acked Read 1 Done Done Sent false
10614 3784 Acked Flush 0 Done Done Sent false
10615 3785 Acked Write 5 Done Done Sent false
10616 3786 Acked Read 6 Done Done Sent false
10617 3787 Acked Write 3 Done Done Sent false
10618 3788 Acked Read 4 Done Done Sent false
10619 3789 Acked Write 10 Done Done Sent false
10620 3790 Acked Read 4 Done Done Sent false
10621 3791 Acked Read 4 Done Done Sent false
10622 3792 Acked Read 2 Done Done Sent false
10623 3793 Acked Read 7 Done Done Sent false
10624 3794 Acked Read 3 Done Done Sent false
10625 3795 Acked Write 7 Done Done Sent false
10626 3796 Acked Write 2 Done Done Sent false
10627 3797 Acked Write 4 Done Done Sent false
10628 3798 Acked Read 2 Done Done Sent false
10629 3799 Acked Flush 0 Done Done Sent false
10630 3800 Acked Write 10 Done Done Sent false
10631 3801 Acked Flush 0 Done Done Sent false
10632 3802 Acked Write 7 Done Done Sent false
10633 3803 Acked Read 5 Done Done Sent false
10634 3804 Acked Flush 0 Done Done Sent false
10635 3805 Acked Read 8 Done Done Sent false
10636 3806 Acked Write 3 Done Done Sent false
10637 3807 Acked Write 9 Done Done Sent false
10638 3808 Acked Read 10 Done Done Sent false
10639 3809 Acked Write 2 Done Done Sent false
10640 3810 Acked Write 6 Done Done Sent false
10641 3811 Acked Flush 0 Done Done Sent false
10642 3812 Acked Write 8 Done Done Sent false
10643 3813 Acked Read 3 Done Done Sent false
10644 3814 Acked Write 9 Done Done Sent false
10645 3815 Acked Write 6 Done Done Sent false
10646 3816 Acked Read 9 Done Done Sent false
10647 3817 Acked Write 1 Done Done Sent false
10648 3818 Acked Write 7 Done Done Sent false
10649 3819 Acked Write 10 Done Done Sent false
10650 3820 Acked Read 7 Done Done Sent false
10651 3821 Acked Read 7 Done Done Sent false
10652 3822 Acked Flush 0 Done Done Sent false
10653 3823 Acked Read 2 Done Done Sent false
10654 3824 Acked Write 10 Done Done Sent false
10655 3825 Acked Read 7 Done Done Sent false
10656 3826 Acked Write 8 Done Done Sent false
10657 3827 Acked Read 2 Done Done Sent false
10658 3828 Acked Write 9 Done Done Sent false
10659 3829 Acked Read 3 Done Done Sent false
10660 3830 Acked Flush 0 Done Done Sent false
10661 3831 Acked Flush 0 Done Done Sent false
10662 3832 Acked Flush 0 Done Done Sent false
10663 3833 Acked Write 4 Done Done Sent false
10664 3834 Acked Write 5 Done Done Sent false
10665 3835 Acked Write 4 Done Done Sent false
10666 3836 Acked Read 1 Done Done Sent false
10667 3837 Acked Read 10 Done Done Sent false
10668 3838 Acked Write 7 Done Done Sent false
10669 3839 Acked Flush 0 Done Done Sent false
10670 3840 Acked Read 3 Done Done Sent false
10671 3841 Acked Write 4 Done Done Sent false
10672 3842 Acked Write 8 Done Done Sent false
10673 3843 Acked Write 3 Done Done Sent false
10674 3844 Acked Read 3 Done Done Sent false
10675 3845 Acked Read 8 Done Done Sent false
10676 3846 Acked Read 7 Done Done Sent false
10677 3847 Acked Write 7 Done Done Sent false
10678 3848 Acked Read 2 Done Done Sent false
10679 3849 Acked Read 3 Done Done Sent false
10680 3850 Acked Read 4 Done Done Sent false
10681 3851 Acked Read 3 Done Done Sent false
10682 3852 Acked Flush 0 Done Done Sent false
10683 3853 Acked Read 1 Done Done Sent false
10684 3854 Acked Read 1 Done Done Sent false
10685 3855 Acked Flush 0 Done Done Sent false
10686 3856 Acked Flush 0 Done Done Sent false
10687 3857 Acked Write 10 Done Done Sent false
10688 3858 Acked Read 7 Done Done Sent false
10689 3859 Acked Read 6 Done Done Sent false
10690 3860 Acked Read 5 Done Done Sent false
10691 3861 Acked Read 9 Done Done Sent false
10692 3862 Acked Write 8 Done Done Sent false
10693 3863 Acked Write 9 Done Done Sent false
10694 3864 Acked Read 2 Done Done Sent false
10695 3865 Acked Write 1 Done Done Sent false
10696 3866 Acked Read 9 Done Done Sent false
10697 3867 Acked Write 3 Done Done Sent false
10698 3868 Acked Read 1 Done Done Sent false
10699 3869 Acked Write 2 Done Done Sent false
10700 3870 Acked Write 4 Done Done Sent false
10701 3871 Acked Write 4 Done Done Sent false
10702 3872 Acked Read 10 Done Done Sent false
10703 3873 Acked Read 1 Done Done Sent false
10704 3874 Acked Write 1 Done Done Sent false
10705 3875 Acked Read 5 Done Done Sent false
10706 3876 Acked Write 10 Done Done Sent false
10707 3877 Acked Read 9 Done Done Sent false
10708 3878 Acked Write 5 Done Done Sent false
10709 3879 Acked Write 4 Done Done Sent false
10710 3880 Acked Read 1 Done Done Sent false
10711 3881 Acked Write 10 Done Done Sent false
10712 3882 Acked Read 7 Done Done Sent false
10713 3883 Acked Read 2 Done Done Sent false
10714 3884 Acked Write 9 Done Done Sent false
10715 3885 Acked Read 6 Done Done Sent false
10716 3886 Acked Write 8 Done Done Sent false
10717 3887 Acked Read 9 Done Done Sent false
10718 3888 Acked Read 5 Done Done Sent false
10719 3889 Acked Write 8 Done Done Sent false
10720 3890 Acked Write 3 Done Done Sent false
10721 3891 Acked Write 5 Done Done Sent false
10722 3892 Acked Read 5 Done Done Sent false
10723 3893 Acked Read 7 Done Done Sent false
10724 3894 Acked Read 9 Done Done Sent false
10725 3895 Acked Read 4 Done Done Sent false
10726 3896 Acked Read 10 Done Done Sent false
10727 3897 Acked Read 8 Done Done Sent false
10728 3898 Acked Write 1 Done Done Sent false
10729 3899 Acked Read 10 Done Done Sent false
10730 3900 Acked Write 2 Done Done Sent false
10731 3901 Acked Write 9 Done Done Sent false
10732 3902 Acked Read 6 Done Done Sent false
10733 3903 Acked Read 8 Done Done Sent false
10734 3904 Acked Write 9 Done Done Sent false
10735 3905 Acked Read 3 Done Done Sent false
10736 3906 Acked Write 9 Done Done Sent false
10737 3907 Acked Read 6 Done Done Sent false
10738 3908 Acked Read 5 Done Done Sent false
10739 3909 Acked Read 9 Done Done Sent false
10740 3910 Acked Read 6 Done Done Sent false
10741 3911 Acked Flush 0 Done Done Sent false
10742 3912 Acked Read 4 Done Done Sent false
10743 3913 Acked Read 10 Done Done Sent false
10744 3914 Acked Flush 0 Done Done Sent false
10745 3915 Acked Read 7 Done Done Sent false
10746 3916 Acked Flush 0 Done Done Sent false
10747 3917 Acked Write 2 Done Done Sent false
10748 3918 Acked Read 5 Done Done Sent false
10749 3919 Acked Read 2 Done Done Sent false
10750 3920 Acked Read 5 Done Done Sent false
10751 3921 Acked Write 8 Done Done Sent false
10752 3922 Acked Write 10 Done Done Sent false
10753 3923 Acked Flush 0 Done Done Sent false
10754 3924 Acked Flush 0 Done Done Sent false
10755 3925 Acked Read 10 Done Done Sent false
10756 3926 Acked Read 2 Done Done Sent false
10757 3927 Acked Write 4 Done Done Sent false
10758 3928 Acked Write 8 Done Done Sent false
10759 3929 Acked Read 9 Done Done Sent false
10760 3930 Acked Write 10 Done Done Sent false
10761 3931 Acked Read 9 Done Done Sent false
10762 3932 Acked Read 5 Done Done Sent false
10763 3933 Acked Flush 0 Done Done Sent false
10764 3934 Acked Read 1 Done Done Sent false
10765 3935 Acked Read 1 Done Done Sent false
10766 3936 Acked Read 5 Done Done Sent false
10767 3937 Acked Read 10 Done Done Sent false
10768 3938 Acked Write 2 Done Done Sent false
10769 3939 Acked Write 8 Done Done Sent false
10770 3940 Acked Read 8 Done Done Sent false
10771 3941 Acked Read 8 Done Done Sent false
10772 3942 Acked Read 2 Done Done Sent false
10773 3943 Acked Read 3 Done Done Sent false
10774 3944 Acked Write 5 Done Done Sent false
10775 3945 Acked Read 4 Done Done Sent false
10776 3946 Acked Read 8 Done Done Sent false
10777 3947 Acked Read 5 Done Done Sent false
10778 3948 Acked Read 8 Done Done Sent false
10779 3949 Acked Read 7 Done Done Sent false
10780 3950 Acked Read 9 Done Done Sent false
10781 3951 Acked Read 6 Done Done Sent false
10782 3952 Acked Read 9 Done Done Sent false
10783 3953 Acked Read 8 Done Done Sent false
10784 3954 Acked Write 8 Done Done Sent false
10785 3955 Acked Read 7 Done Done Sent false
10786 3956 Acked Read 7 Done Done Sent false
10787 3957 Acked Write 10 Done Done Sent false
10788 3958 Acked Write 6 Done Done Sent false
10789 3959 Acked Write 9 Done Done Sent false
10790 3960 Acked Read 3 Done Done Sent false
10791 3961 Acked Read 3 Done Done Sent false
10792 3962 Acked Flush 0 Done Done Sent false
10793 3963 Acked Read 2 Done Done Sent false
10794 3964 Acked Write 4 Done Done Sent false
10795 3965 Acked Write 4 Done Done Sent false
10796 3966 Acked Flush 0 Done Done Sent false
10797 3967 Acked Read 7 Done Done Sent false
10798 3968 Acked Read 4 Done Done Sent false
10799 3969 Acked Read 2 Done Done Sent false
10800 3970 Acked Read 8 Done Done Sent false
10801 3971 Acked Read 3 Done Done Sent false
10802 3972 Acked Write 9 Done Done Sent false
10803 3973 Acked Flush 0 Done Done Sent false
10804 3974 Acked Write 9 Done Done Sent false
10805 3975 Acked Write 7 Done Done Sent false
10806 3976 Acked Write 6 Done Done Sent false
10807 3977 Acked Write 5 Done Done Sent false
10808 3978 Acked Write 4 Done Done Sent false
10809 3979 Acked Read 8 Done Done Sent false
10810 3980 Acked Read 4 Done Done Sent false
10811 3981 Acked Write 4 Done Done Sent false
10812 3982 Acked Flush 0 Done Done Sent false
10813 3983 Acked Flush 0 Done Done Sent false
10814 3984 Acked Read 4 Done Done Sent false
10815 3985 Acked Write 3 Done Done Sent false
10816 3986 Acked Read 1 Done Done Sent false
10817 3987 Acked Write 9 Done Done Sent false
10818 3988 Acked Write 1 Done Done Sent false
10819 3989 Acked Read 8 Done Done Sent false
10820 3990 Acked Read 6 Done Done Sent false
10821 3991 Acked Write 1 Done Done Sent false
10822 3992 Acked Write 8 Done Done Sent false
10823 3993 Acked Read 9 Done Done Sent false
10824 3994 Acked Read 3 Done Done Sent false
10825 3995 Acked Read 1 Done Done Sent false
10826 3996 Acked Write 8 Done Done Sent false
10827 3997 Acked Write 6 Done Done Sent false
10828 3998 Acked Read 1 Done Done Sent false
10829 3999 Acked Write 7 Done Done Sent false
10830 4000 Acked Read 1 Done Done Sent false
10831 4001 Acked Write 4 Done Done Sent false
10832 4002 Acked Write 10 Done Done Sent false
10833 4003 Acked Read 4 Done Done Sent false
10834 4004 Acked Read 6 Done Done Sent false
10835 4005 Acked Write 9 Done Done Sent false
10836 4006 Acked Write 2 Done Done Sent false
10837 4007 Acked Read 1 Done Done Sent false
10838 4008 Acked Flush 0 Done Done Sent false
10839 4009 Acked Flush 0 Done Done Sent false
10840 4010 Acked Write 8 Done Done Sent false
10841 4011 Acked Write 6 Done Done Sent false
10842 4012 Acked Write 1 Done Done Sent false
10843 4013 Acked Write 1 Done Done Sent false
10844 4014 Acked Read 1 Done Done Sent false
10845 4015 Acked Read 4 Done Done Sent false
10846 4016 Acked Write 8 Done Done Sent false
10847 4017 Acked Read 2 Done Done Sent false
10848 4018 Acked Read 8 Done Done Sent false
10849 4019 Acked Read 2 Done Done Sent false
10850 STATES DS:0 DS:1 DS:2 TOTAL
10851 Sent 0 0 302 302
10852 Done 302 302 0 604
10853 Skipped 0 0 0 0
10854 Error 0 0 0 0
10855 Last Flush: 4009 4009 3717
10856 Downstairs last five completed: 3717 3716 3715 3714 3713
10857 Upstairs last five completed: 4019 4018 4017 4016 4015
10858 CLIENT: Up:0 ds:302 act:2
10859 [8830] got start action
10860 [8830] I am going to restart
10861 Starting downstairs at port 8830
10862 Make output file at "/tmp/dsc/downstairs-8830.txt"
10863 [8830][2] reports Starting
10864 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1208)
10865 [8830][2] reports Running
10866 ----------------------------------------------------------------
10867 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
10868 States: Active Active Offline
10869 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10870 3718 Acked Read 1 Done Done Sent false
10871 3719 Acked Flush 0 Done Done Sent false
10872 3720 Acked Flush 0 Done Done Sent false
10873 3721 Acked Read 6 Done Done Sent false
10874 3722 Acked Read 2 Done Done Sent false
10875 3723 Acked Read 8 Done Done Sent false
10876 3724 Acked Read 7 Done Done Sent false
10877 3725 Acked Read 10 Done Done Sent false
10878 3726 Acked Write 5 Done Done Sent false
10879 3727 Acked Read 9 Done Done Sent false
10880 3728 Acked Write 1 Done Done Sent false
10881 3729 Acked Read 7 Done Done Sent false
10882 3730 Acked Flush 0 Done Done Sent false
10883 3731 Acked Read 9 Done Done Sent false
10884 3732 Acked Flush 0 Done Done Sent false
10885 3733 Acked Read 1 Done Done Sent false
10886 3734 Acked Write 5 Done Done Sent false
10887 3735 Acked Write 1 Done Done Sent false
10888 3736 Acked Write 9 Done Done Sent false
10889 3737 Acked Flush 0 Done Done Sent false
10890 3738 Acked Read 6 Done Done Sent false
10891 3739 Acked Read 5 Done Done Sent false
10892 3740 Acked Write 9 Done Done Sent false
10893 3741 Acked Read 5 Done Done Sent false
10894 3742 Acked Read 4 Done Done Sent false
10895 3743 Acked Write 3 Done Done Sent false
10896 3744 Acked Write 3 Done Done Sent false
10897 3745 Acked Write 10 Done Done Sent false
10898 3746 Acked Read 2 Done Done Sent false
10899 3747 Acked Read 3 Done Done Sent false
10900 3748 Acked Write 1 Done Done Sent false
10901 3749 Acked Flush 0 Done Done Sent false
10902 3750 Acked Flush 0 Done Done Sent false
10903 3751 Acked Read 8 Done Done Sent false
10904 3752 Acked Read 3 Done Done Sent false
10905 3753 Acked Write 2 Done Done Sent false
10906 3754 Acked Read 10 Done Done Sent false
10907 3755 Acked Flush 0 Done Done Sent false
10908 3756 Acked Read 6 Done Done Sent false
10909 3757 Acked Write 5 Done Done Sent false
10910 3758 Acked Flush 0 Done Done Sent false
10911 3759 Acked Write 1 Done Done Sent false
10912 3760 Acked Flush 0 Done Done Sent false
10913 3761 Acked Read 6 Done Done Sent false
10914 3762 Acked Read 10 Done Done Sent false
10915 3763 Acked Write 8 Done Done Sent false
10916 3764 Acked Read 2 Done Done Sent false
10917 3765 Acked Read 1 Done Done Sent false
10918 3766 Acked Read 5 Done Done Sent false
10919 3767 Acked Flush 0 Done Done Sent false
10920 3768 Acked Write 5 Done Done Sent false
10921 3769 Acked Read 7 Done Done Sent false
10922 3770 Acked Write 6 Done Done Sent false
10923 3771 Acked Read 4 Done Done Sent false
10924 3772 Acked Read 8 Done Done Sent false
10925 3773 Acked Read 7 Done Done Sent false
10926 3774 Acked Write 3 Done Done Sent false
10927 3775 Acked Flush 0 Done Done Sent false
10928 3776 Acked Flush 0 Done Done Sent false
10929 3777 Acked Write 8 Done Done Sent false
10930 3778 Acked Read 7 Done Done Sent false
10931 3779 Acked Read 3 Done Done Sent false
10932 3780 Acked Read 7 Done Done Sent false
10933 3781 Acked Read 5 Done Done Sent false
10934 3782 Acked Read 10 Done Done Sent false
10935 3783 Acked Read 1 Done Done Sent false
10936 3784 Acked Flush 0 Done Done Sent false
10937 3785 Acked Write 5 Done Done Sent false
10938 3786 Acked Read 6 Done Done Sent false
10939 3787 Acked Write 3 Done Done Sent false
10940 3788 Acked Read 4 Done Done Sent false
10941 3789 Acked Write 10 Done Done Sent false
10942 3790 Acked Read 4 Done Done Sent false
10943 3791 Acked Read 4 Done Done Sent false
10944 3792 Acked Read 2 Done Done Sent false
10945 3793 Acked Read 7 Done Done Sent false
10946 3794 Acked Read 3 Done Done Sent false
10947 3795 Acked Write 7 Done Done Sent false
10948 3796 Acked Write 2 Done Done Sent false
10949 3797 Acked Write 4 Done Done Sent false
10950 3798 Acked Read 2 Done Done Sent false
10951 3799 Acked Flush 0 Done Done Sent false
10952 3800 Acked Write 10 Done Done Sent false
10953 3801 Acked Flush 0 Done Done Sent false
10954 3802 Acked Write 7 Done Done Sent false
10955 3803 Acked Read 5 Done Done Sent false
10956 3804 Acked Flush 0 Done Done Sent false
10957 3805 Acked Read 8 Done Done Sent false
10958 3806 Acked Write 3 Done Done Sent false
10959 3807 Acked Write 9 Done Done Sent false
10960 3808 Acked Read 10 Done Done Sent false
10961 3809 Acked Write 2 Done Done Sent false
10962 3810 Acked Write 6 Done Done Sent false
10963 3811 Acked Flush 0 Done Done Sent false
10964 3812 Acked Write 8 Done Done Sent false
10965 3813 Acked Read 3 Done Done Sent false
10966 3814 Acked Write 9 Done Done Sent false
10967 3815 Acked Write 6 Done Done Sent false
10968 3816 Acked Read 9 Done Done Sent false
10969 3817 Acked Write 1 Done Done Sent false
10970 3818 Acked Write 7 Done Done Sent false
10971 3819 Acked Write 10 Done Done Sent false
10972 3820 Acked Read 7 Done Done Sent false
10973 3821 Acked Read 7 Done Done Sent false
10974 3822 Acked Flush 0 Done Done Sent false
10975 3823 Acked Read 2 Done Done Sent false
10976 3824 Acked Write 10 Done Done Sent false
10977 3825 Acked Read 7 Done Done Sent false
10978 3826 Acked Write 8 Done Done Sent false
10979 3827 Acked Read 2 Done Done Sent false
10980 3828 Acked Write 9 Done Done Sent false
10981 3829 Acked Read 3 Done Done Sent false
10982 3830 Acked Flush 0 Done Done Sent false
10983 3831 Acked Flush 0 Done Done Sent false
10984 3832 Acked Flush 0 Done Done Sent false
10985 3833 Acked Write 4 Done Done Sent false
10986 3834 Acked Write 5 Done Done Sent false
10987 3835 Acked Write 4 Done Done Sent false
10988 3836 Acked Read 1 Done Done Sent false
10989 3837 Acked Read 10 Done Done Sent false
10990 3838 Acked Write 7 Done Done Sent false
10991 3839 Acked Flush 0 Done Done Sent false
10992 3840 Acked Read 3 Done Done Sent false
10993 3841 Acked Write 4 Done Done Sent false
10994 3842 Acked Write 8 Done Done Sent false
10995 3843 Acked Write 3 Done Done Sent false
10996 3844 Acked Read 3 Done Done Sent false
10997 3845 Acked Read 8 Done Done Sent false
10998 3846 Acked Read 7 Done Done Sent false
10999 3847 Acked Write 7 Done Done Sent false
11000 3848 Acked Read 2 Done Done Sent false
11001 3849 Acked Read 3 Done Done Sent false
11002 3850 Acked Read 4 Done Done Sent false
11003 3851 Acked Read 3 Done Done Sent false
11004 3852 Acked Flush 0 Done Done Sent false
11005 3853 Acked Read 1 Done Done Sent false
11006 3854 Acked Read 1 Done Done Sent false
11007 3855 Acked Flush 0 Done Done Sent false
11008 3856 Acked Flush 0 Done Done Sent false
11009 3857 Acked Write 10 Done Done Sent false
11010 3858 Acked Read 7 Done Done Sent false
11011 3859 Acked Read 6 Done Done Sent false
11012 3860 Acked Read 5 Done Done Sent false
11013 3861 Acked Read 9 Done Done Sent false
11014 3862 Acked Write 8 Done Done Sent false
11015 3863 Acked Write 9 Done Done Sent false
11016 3864 Acked Read 2 Done Done Sent false
11017 3865 Acked Write 1 Done Done Sent false
11018 3866 Acked Read 9 Done Done Sent false
11019 3867 Acked Write 3 Done Done Sent false
11020 3868 Acked Read 1 Done Done Sent false
11021 3869 Acked Write 2 Done Done Sent false
11022 3870 Acked Write 4 Done Done Sent false
11023 3871 Acked Write 4 Done Done Sent false
11024 3872 Acked Read 10 Done Done Sent false
11025 3873 Acked Read 1 Done Done Sent false
11026 3874 Acked Write 1 Done Done Sent false
11027 3875 Acked Read 5 Done Done Sent false
11028 3876 Acked Write 10 Done Done Sent false
11029 3877 Acked Read 9 Done Done Sent false
11030 3878 Acked Write 5 Done Done Sent false
11031 3879 Acked Write 4 Done Done Sent false
11032 3880 Acked Read 1 Done Done Sent false
11033 3881 Acked Write 10 Done Done Sent false
11034 3882 Acked Read 7 Done Done Sent false
11035 3883 Acked Read 2 Done Done Sent false
11036 3884 Acked Write 9 Done Done Sent false
11037 3885 Acked Read 6 Done Done Sent false
11038 3886 Acked Write 8 Done Done Sent false
11039 3887 Acked Read 9 Done Done Sent false
11040 3888 Acked Read 5 Done Done Sent false
11041 3889 Acked Write 8 Done Done Sent false
11042 3890 Acked Write 3 Done Done Sent false
11043 3891 Acked Write 5 Done Done Sent false
11044 3892 Acked Read 5 Done Done Sent false
11045 3893 Acked Read 7 Done Done Sent false
11046 3894 Acked Read 9 Done Done Sent false
11047 3895 Acked Read 4 Done Done Sent false
11048 3896 Acked Read 10 Done Done Sent false
11049 3897 Acked Read 8 Done Done Sent false
11050 3898 Acked Write 1 Done Done Sent false
11051 3899 Acked Read 10 Done Done Sent false
11052 3900 Acked Write 2 Done Done Sent false
11053 3901 Acked Write 9 Done Done Sent false
11054 3902 Acked Read 6 Done Done Sent false
11055 3903 Acked Read 8 Done Done Sent false
11056 3904 Acked Write 9 Done Done Sent false
11057 3905 Acked Read 3 Done Done Sent false
11058 3906 Acked Write 9 Done Done Sent false
11059 3907 Acked Read 6 Done Done Sent false
11060 3908 Acked Read 5 Done Done Sent false
11061 3909 Acked Read 9 Done Done Sent false
11062 3910 Acked Read 6 Done Done Sent false
11063 3911 Acked Flush 0 Done Done Sent false
11064 3912 Acked Read 4 Done Done Sent false
11065 3913 Acked Read 10 Done Done Sent false
11066 3914 Acked Flush 0 Done Done Sent false
11067 3915 Acked Read 7 Done Done Sent false
11068 3916 Acked Flush 0 Done Done Sent false
11069 3917 Acked Write 2 Done Done Sent false
11070 3918 Acked Read 5 Done Done Sent false
11071 3919 Acked Read 2 Done Done Sent false
11072 3920 Acked Read 5 Done Done Sent false
11073 3921 Acked Write 8 Done Done Sent false
11074 3922 Acked Write 10 Done Done Sent false
11075 3923 Acked Flush 0 Done Done Sent false
11076 3924 Acked Flush 0 Done Done Sent false
11077 3925 Acked Read 10 Done Done Sent false
11078 3926 Acked Read 2 Done Done Sent false
11079 3927 Acked Write 4 Done Done Sent false
11080 3928 Acked Write 8 Done Done Sent false
11081 3929 Acked Read 9 Done Done Sent false
11082 3930 Acked Write 10 Done Done Sent false
11083 3931 Acked Read 9 Done Done Sent false
11084 3932 Acked Read 5 Done Done Sent false
11085 3933 Acked Flush 0 Done Done Sent false
11086 3934 Acked Read 1 Done Done Sent false
11087 3935 Acked Read 1 Done Done Sent false
11088 3936 Acked Read 5 Done Done Sent false
11089 3937 Acked Read 10 Done Done Sent false
11090 3938 Acked Write 2 Done Done Sent false
11091 3939 Acked Write 8 Done Done Sent false
11092 3940 Acked Read 8 Done Done Sent false
11093 3941 Acked Read 8 Done Done Sent false
11094 3942 Acked Read 2 Done Done Sent false
11095 3943 Acked Read 3 Done Done Sent false
11096 3944 Acked Write 5 Done Done Sent false
11097 3945 Acked Read 4 Done Done Sent false
11098 3946 Acked Read 8 Done Done Sent false
11099 3947 Acked Read 5 Done Done Sent false
11100 3948 Acked Read 8 Done Done Sent false
11101 3949 Acked Read 7 Done Done Sent false
11102 3950 Acked Read 9 Done Done Sent false
11103 3951 Acked Read 6 Done Done Sent false
11104 3952 Acked Read 9 Done Done Sent false
11105 3953 Acked Read 8 Done Done Sent false
11106 3954 Acked Write 8 Done Done Sent false
11107 3955 Acked Read 7 Done Done Sent false
11108 3956 Acked Read 7 Done Done Sent false
11109 3957 Acked Write 10 Done Done Sent false
11110 3958 Acked Write 6 Done Done Sent false
11111 3959 Acked Write 9 Done Done Sent false
11112 3960 Acked Read 3 Done Done Sent false
11113 3961 Acked Read 3 Done Done Sent false
11114 3962 Acked Flush 0 Done Done Sent false
11115 3963 Acked Read 2 Done Done Sent false
11116 3964 Acked Write 4 Done Done Sent false
11117 3965 Acked Write 4 Done Done Sent false
11118 3966 Acked Flush 0 Done Done Sent false
11119 3967 Acked Read 7 Done Done Sent false
11120 3968 Acked Read 4 Done Done Sent false
11121 3969 Acked Read 2 Done Done Sent false
11122 3970 Acked Read 8 Done Done Sent false
11123 3971 Acked Read 3 Done Done Sent false
11124 3972 Acked Write 9 Done Done Sent false
11125 3973 Acked Flush 0 Done Done Sent false
11126 3974 Acked Write 9 Done Done Sent false
11127 3975 Acked Write 7 Done Done Sent false
11128 3976 Acked Write 6 Done Done Sent false
11129 3977 Acked Write 5 Done Done Sent false
11130 3978 Acked Write 4 Done Done Sent false
11131 3979 Acked Read 8 Done Done Sent false
11132 3980 Acked Read 4 Done Done Sent false
11133 3981 Acked Write 4 Done Done Sent false
11134 3982 Acked Flush 0 Done Done Sent false
11135 3983 Acked Flush 0 Done Done Sent false
11136 3984 Acked Read 4 Done Done Sent false
11137 3985 Acked Write 3 Done Done Sent false
11138 3986 Acked Read 1 Done Done Sent false
11139 3987 Acked Write 9 Done Done Sent false
11140 3988 Acked Write 1 Done Done Sent false
11141 3989 Acked Read 8 Done Done Sent false
11142 3990 Acked Read 6 Done Done Sent false
11143 3991 Acked Write 1 Done Done Sent false
11144 3992 Acked Write 8 Done Done Sent false
11145 3993 Acked Read 9 Done Done Sent false
11146 3994 Acked Read 3 Done Done Sent false
11147 3995 Acked Read 1 Done Done Sent false
11148 3996 Acked Write 8 Done Done Sent false
11149 3997 Acked Write 6 Done Done Sent false
11150 3998 Acked Read 1 Done Done Sent false
11151 3999 Acked Write 7 Done Done Sent false
11152 4000 Acked Read 1 Done Done Sent false
11153 4001 Acked Write 4 Done Done Sent false
11154 4002 Acked Write 10 Done Done Sent false
11155 4003 Acked Read 4 Done Done Sent false
11156 4004 Acked Read 6 Done Done Sent false
11157 4005 Acked Write 9 Done Done Sent false
11158 4006 Acked Write 2 Done Done Sent false
11159 4007 Acked Read 1 Done Done Sent false
11160 4008 Acked Flush 0 Done Done Sent false
11161 4009 Acked Flush 0 Done Done Sent false
11162 4010 Acked Write 8 Done Done Sent false
11163 4011 Acked Write 6 Done Done Sent false
11164 4012 Acked Write 1 Done Done Sent false
11165 4013 Acked Write 1 Done Done Sent false
11166 4014 Acked Read 1 Done Done Sent false
11167 4015 Acked Read 4 Done Done Sent false
11168 4016 Acked Write 8 Done Done Sent false
11169 4017 Acked Read 2 Done Done Sent false
11170 4018 Acked Read 8 Done Done Sent false
11171 4019 Acked Read 2 Done Done Sent false
11172 4020 Acked Flush 0 Done Done Sent false
11173 STATES DS:0 DS:1 DS:2 TOTAL
11174 Sent 0 0 303 303
11175 Done 303 303 0 606
11176 Skipped 0 0 0 0
11177 Error 0 0 0 0
11178 Last Flush: 4020 4020 3717
11179 Downstairs last five completed: 3717 3716 3715 3714 3713
11180 Upstairs last five completed: 4020 4019 4018 4017 4016
11181 CLIENT: Up:0 ds:303 act:2
111822024-11-01T16:52:48.927ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
111832024-11-01T16:52:48.928ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:32999) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
111842024-11-01T16:52:48.929ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
111852024-11-01T16:52:48.930ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
111862024-11-01T16:52:48.930ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
111872024-11-01T16:52:48.930ZINFOcrucible: send last flush ID to this DS: 3717 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
111882024-11-01T16:52:48.930ZINFOcrucible: Replied this last flush ID: 3717 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
111892024-11-01T16:52:48.930ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
111902024-11-01T16:52:48.930ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
111912024-11-01T16:52:48.930ZINFOcrucible: [2] client re-new 303 jobs since flush 3717 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
111922024-11-01T16:52:48.930ZINFOcrucible: [2] Replayed 303 jobs since flush: 3717 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11193 ----------------------------------------------------------------
11194 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11195 Downstairs last five completed: 4020 4019 4018 4017 4016
11196 Upstairs last five completed: 4020 4019 4018 4017 4016
11197 CLIENT: Up:0 ds:0 act:3
11198 Replay: All jobs finished
11199 Main task has work to do, go find it
11200 got dsc Stop(2)
11201 stop 2
11202 Nov 01 16:52:51.221[8830] Got stop action so:false kr:false
11203 INFO request completed, latency_us: 355, response_code: 204, uri: /stop/cid/2, method: POST, req_id: a10857b6-66d7-4958-bbf3-69943fd9ccb2, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
112042024-11-01T16:52:51.222ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11205 Nov 01 16:52:51.222 INFO request completed, latency_us: 150, response_code: 200, uri: /state/cid/2, method: GET, req_id: 924adb12-7fbd-42dc-a7a0-bf5a34574fc2, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
112062024-11-01T16:52:51.222ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
112072024-11-01T16:52:51.222ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
112082024-11-01T16:52:51.222ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
112092024-11-01T16:52:51.222ZINFOcrucible: client task is exiting = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
11210 [8830] Exited with: ExitStatus(unix_wait_status(9))
11211 [8830][2] reports Exit
11212 Nov 01 16:52:55.225 INFO request completed, latency_us: 264, response_code: 200, uri: /state/cid/2, method: GET, req_id: f233da20-13bf-41d6-bdfa-6ce1636f0003, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
11213 001/300 Read block 514 len 36864
11214 002/300 Write block 63 len 40960 data: 4 4 3 3 3 3 3 3 3 3
11215 003/300 Write block 3846 len 24576 data: 2 2 2 2 2 2
11216 004/300 Write block 2822 len 24576 data: 3 2 2 2 2 2
11217 005/300 Read block 4289 len 24576
11218 006/300 Write block 1809 len 24576 data: 4 4 4 4 2 2
11219 007/300 Write block 3974 len 20480 data: 2 2 3 2 2
11220 008/300 Flush
11221 009/300 Write block 1068 len 24576 data: 2 2 2 2 2 2
11222 010/300 Write block 4301 len 12288 data: 2 2 2
11223 011/300 Read block 1606 len 16384
11224 012/300 Write block 3417 len 28672 data: 2 2 2 2 3 4 4
11225 013/300 Read block 659 len 16384
11226 014/300 Read block 114 len 20480
11227 015/300 Write block 4266 len 32768 data: 3 3 3 3 4 3 3 3
11228 016/300 Read block 2470 len 8192
11229 017/300 Read block 80 len 16384
11230 018/300 Flush
11231 019/300 Read block 2418 len 8192
11232 020/300 Read block 1724 len 12288
11233 021/300 Read block 4125 len 12288
11234 022/300 Write block 2979 len 16384 data: 4 3 4 3
11235 023/300 Write block 50 len 4096 data: 3
11236 024/300 Write block 4223 len 24576 data: 3 3 4 4 4 4
11237 025/300 Read block 212 len 36864
11238 026/300 Read block 1747 len 24576
11239 027/300 Read block 4449 len 8192
11240 028/300 Read block 2613 len 40960
11241 029/300 Write block 2613 len 40960 data: 4 4 4 4 5 4 4 5 4 3
11242 030/300 Write block 3665 len 32768 data: 2 2 2 3 3 3 3 3
11243 031/300 Write block 199 len 8192 data: 8 5
11244 032/300 Write block 2434 len 20480 data: 3 3 4 3 3
11245 033/300 Read block 550 len 24576
11246 034/300 Read block 428 len 4096
11247 035/300 Read block 1560 len 28672
11248 036/300 Write block 254 len 32768 data: 4 4 4 4 4 3 3 4
11249 037/300 Read block 830 len 12288
11250 038/300 Write block 929 len 20480 data: 3 3 3 3 4
11251 039/300 Read block 298 len 12288
11252 040/300 Write block 3483 len 4096 data: 2
11253 041/300 Read block 2312 len 32768
11254 042/300 Write block 3003 len 28672 data: 4 3 2 3 3 2 2
11255 043/300 Read block 4310 len 40960
11256 044/300 Read block 4838 len 28672
11257 045/300 Read block 907 len 32768
11258 046/300 Write block 2384 len 32768 data: 2 2 2 2 2 2 3 3
11259 047/300 Read block 4160 len 36864
11260 048/300 Write block 2979 len 8192 data: 5 4
11261 049/300 Read block 2057 len 8192
11262 050/300 Read block 2201 len 8192
11263 051/300 Read block 1534 len 20480
11264 052/300 Read block 4928 len 12288
11265 053/300 Read block 613 len 8192
11266 054/300 Read block 4270 len 4096
11267 055/300 Read block 3800 len 28672
11268 056/300 Read block 800 len 12288
11269 057/300 Read block 1780 len 24576
11270 058/300 Read block 2617 len 40960
11271 059/300 Write block 3848 len 36864 data: 3 3 3 3 2 2 2 2 2
11272 060/300 Read block 332 len 36864
11273 061/300 Flush
11274 062/300 Write block 2090 len 8192 data: 2 3
11275 063/300 Write block 2284 len 32768 data: 4 2 3 3 4 5 5 6
11276 064/300 Read block 3259 len 12288
11277 065/300 Write block 4546 len 8192 data: 4 4
11278 066/300 Flush
11279 067/300 Read block 683 len 32768
11280 068/300 Read block 1819 len 24576
11281 069/300 Flush
11282 070/300 Write block 2953 len 20480 data: 3 4 4 6 6
11283 071/300 Read block 307 len 8192
11284 072/300 Flush
11285 073/300 Write block 113 len 40960 data: 3 3 5 5 4 4 3 5 5 6
11286 074/300 Flush
11287 075/300 Write block 3416 len 12288 data: 2 3 3
11288 076/300 Write block 1359 len 4096 data: 3
11289 077/300 Flush
11290 078/300 Read block 3704 len 24576
11291 079/300 Read block 3229 len 36864
11292 080/300 Write block 3619 len 12288 data: 2 2 3
11293 081/300 Read block 1160 len 20480
11294 082/300 Read block 1565 len 40960
11295 083/300 Read block 2607 len 40960
11296 084/300 Read block 929 len 4096
11297 085/300 Read block 2360 len 32768
11298 086/300 Write block 3594 len 8192 data: 4 4
11299 087/300 Read block 3613 len 12288
11300 088/300 Read block 2304 len 20480
11301 089/300 Read block 3090 len 4096
11302 090/300 Read block 2754 len 16384
11303 091/300 Read block 2449 len 20480
11304 092/300 Write block 1673 len 32768 data: 3 3 3 3 3 3 3 3
11305 093/300 Write block 2792 len 32768 data: 5 4 4 4 3 3 3 3
11306 094/300 Write block 4695 len 12288 data: 3 2 2
11307 095/300 Read block 990 len 20480
11308 096/300 Read block 2700 len 8192
11309 097/300 Write block 243 len 36864 data: 5 5 4 7 6 7 5 4 5
11310 098/300 Read block 1322 len 8192
11311 099/300 Write block 4312 len 32768 data: 3 3 4 5 4 5 8 8
11312 100/300 Read block 1790 len 40960
11313 101/300 Read block 2428 len 24576
11314 102/300 Read block 3178 len 16384
11315 103/300 Flush
11316 104/300 Read block 221 len 36864
11317 105/300 Read block 3937 len 36864
11318 106/300 Flush
11319 107/300 Read block 4134 len 40960
11320 108/300 Write block 2832 len 24576 data: 2 2 2 2 2 3
11321 109/300 Flush
11322 110/300 Read block 859 len 36864
11323 111/300 Read block 4306 len 32768
11324 112/300 Flush
11325 113/300 Read block 2205 len 16384
11326 114/300 Read block 2557 len 40960
11327 115/300 Read block 2762 len 28672
11328 116/300 Read block 2172 len 4096
11329 117/300 Write block 2091 len 12288 data: 4 5 5
11330 118/300 Read block 3588 len 24576
11331 119/300 Flush
11332 120/300 Read block 2519 len 8192
11333 121/300 Write block 3722 len 16384 data: 3 3 2 3
11334 122/300 Write block 3670 len 36864 data: 4 4 4 3 3 2 2 2 2
11335 123/300 Write block 2546 len 36864 data: 5 3 3 3 3 3 3 4 4
11336 124/300 Write block 4767 len 20480 data: 3 5 5 3 4
11337 125/300 Flush
11338 126/300 Write block 2777 len 28672 data: 4 4 3 3 2 2 2
11339 127/300 Write block 3903 len 24576 data: 2 2 2 3 3 3
11340 128/300 Read block 3938 len 4096
11341 129/300 Read block 3122 len 36864
11342 130/300 Write block 2987 len 32768 data: 3 3 3 3 3 3 2 2
11343 131/300 Read block 3629 len 24576
11344 132/300 Write block 1993 len 4096 data: 2
11345 133/300 Write block 1662 len 36864 data: 5 5 5 5 4 3 3 3 3
11346 134/300 Read block 2603 len 24576
11347 135/300 Read block 2050 len 28672
11348 136/300 Write block 3479 len 40960 data: 3 3 3 2 3 2 2 3 3 3
11349 137/300 Read block 3356 len 20480
11350 138/300 Read block 95 len 16384
11351 139/300 Read block 3097 len 8192
11352 140/300 Write block 117 len 8192 data: 5 5
11353 141/300 Read block 4486 len 12288
11354 142/300 Read block 4680 len 28672
11355 143/300 Write block 2451 len 28672 data: 5 4 5 4 4 4 4
11356 144/300 Read block 39 len 24576
11357 145/300 Write block 2633 len 28672 data: 2 2 2 3 3 3 3
11358 146/300 Write block 1323 len 4096 data: 5
11359 147/300 Read block 3792 len 8192
11360 148/300 Read block 1849 len 28672
11361 149/300 Read block 4331 len 24576
11362 150/300 Write block 3880 len 32768 data: 3 2 3 3 2 2 2 2
11363 151/300 Read block 3373 len 36864
11364 152/300 Write block 4639 len 12288 data: 3 3 3
11365 153/300 Write block 4517 len 24576 data: 5 4 4 3 4 4
11366 154/300 Write block 2283 len 24576 data: 4 5 3 4 4 5
11367 155/300 Write block 3463 len 40960 data: 5 4 6 6 4 4 3 3 4 4
11368 156/300 Read block 2376 len 8192
11369 157/300 Read block 2078 len 24576
11370 158/300 Write block 2539 len 20480 data: 4 3 4 5 5
11371 159/300 Write block 216 len 8192 data: 3 3
11372 160/300 Flush
11373 161/300 Read block 4735 len 12288
11374 162/300 Read block 4492 len 36864
11375 163/300 Read block 4181 len 24576
11376 164/300 Flush
11377 165/300 Write block 663 len 12288 data: 3 3 3
11378 166/300 Read block 998 len 40960
11379 167/300 Write block 4554 len 32768 data: 3 3 4 3 3 3 3 2
11380 168/300 Flush
11381 169/300 Write block 823 len 32768 data: 2 3 3 3 3 3 3 3
11382 170/300 Flush
11383 171/300 Flush
11384 172/300 Read block 4503 len 4096
11385 173/300 Read block 1417 len 36864
11386 174/300 Write block 3434 len 16384 data: 2 2 2 4
11387 175/300 Flush
11388 176/300 Read block 870 len 16384
11389 177/300 Write block 2770 len 16384 data: 4 4 3 4
11390 178/300 Read block 2314 len 28672
11391 179/300 Read block 1407 len 32768
11392 180/300 Read block 280 len 24576
11393 181/300 Write block 2597 len 40960 data: 3 3 3 3 2 2 2 2 2 2
11394 182/300 Read block 883 len 16384
11395 183/300 Read block 1421 len 36864
11396 184/300 Write block 4030 len 16384 data: 3 4 3 5
11397 185/300 Write block 2472 len 16384 data: 3 3 3 5
11398 186/300 Write block 3236 len 28672 data: 2 2 2 2 2 3 4
11399 187/300 Write block 2084 len 36864 data: 2 3 4 3 2 2 3 5 6
11400 188/300 Write block 2962 len 28672 data: 4 3 2 2 3 3 3
11401 189/300 Write block 3911 len 28672 data: 2 4 4 4 5 7 6
11402 190/300 Read block 171 len 28672
11403 191/300 Write block 4490 len 24576 data: 2 2 2 2 2 2
11404 192/300 Read block 208 len 36864
11405 193/300 Read block 4904 len 40960
11406 194/300 Write block 2698 len 36864 data: 2 2 2 2 2 2 2 2 2
11407 195/300 Read block 1746 len 28672
11408 196/300 Read block 4093 len 8192
11409 197/300 Flush
11410 198/300 Read block 4675 len 8192
11411 199/300 Read block 2988 len 16384
11412 200/300 Read block 573 len 40960
11413 201/300 Flush
11414 202/300 Read block 3065 len 32768
11415 203/300 Write block 1505 len 20480 data: 3 3 3 3 3
11416 204/300 Read block 79 len 40960
11417 205/300 Read block 42 len 20480
11418 206/300 Read block 1652 len 32768
11419 207/300 Write block 1039 len 40960 data: 2 2 3 3 3 3 3 2 2 2
11420 208/300 Write block 656 len 20480 data: 4 4 5 6 6
11421 209/300 Read block 2042 len 16384
11422 210/300 Read block 3957 len 28672
11423 211/300 Read block 653 len 36864
11424 212/300 Write block 1563 len 28672 data: 3 2 2 2 2 2 2
11425 213/300 Read block 1208 len 20480
11426 214/300 Read block 2416 len 32768
11427 215/300 Write block 2995 len 40960 data: 2 2 2 3 3 3 4 3 5 4
11428 216/300 Write block 1563 len 32768 data: 4 3 3 3 3 3 3 2
11429 217/300 Read block 1329 len 32768
11430 218/300 Read block 3000 len 12288
11431 219/300 Read block 3148 len 8192
11432 220/300 Write block 2804 len 4096 data: 2
11433 221/300 Read block 3049 len 8192
11434 222/300 Flush
11435 223/300 Read block 958 len 24576
11436 224/300 Write block 3274 len 32768 data: 4 4 4 4 3 3 3 4
11437 225/300 Write block 2205 len 8192 data: 3 3
11438 226/300 Read block 3294 len 16384
11439 227/300 Write block 1832 len 12288 data: 2 2 2
11440 228/300 Flush
11441 229/300 Write block 4143 len 20480 data: 4 3 3 3 3
11442 230/300 Write block 4293 len 8192 data: 2 2
11443 231/300 Write block 4632 len 28672 data: 6 5 5 5 5 4 4
11444 232/300 Write block 3143 len 36864 data: 2 2 2 3 3 3 2 3 3
11445 233/300 Write block 4496 len 40960 data: 2 2 3 3 3 3 3 3 4 4
11446 234/300 Read block 3788 len 4096
11447 235/300 Flush
11448 236/300 Read block 201 len 20480
11449 237/300 Write block 4311 len 12288 data: 3 4 4
11450 238/300 Read block 3583 len 4096
11451 239/300 Write block 897 len 8192 data: 4 3
11452 240/300 Write block 3042 len 28672 data: 3 4 3 3 3 4 4
11453 241/300 Write block 4749 len 12288 data: 3 3 3
11454 242/300 Read block 1452 len 28672
11455 243/300 Read block 1348 len 16384
11456 244/300 Read block 3758 len 28672
11457 245/300 Write block 4985 len 20480 data: 3 3 2 3 3
11458 246/300 Read block 4249 len 24576
11459 247/300 Read block 1032 len 16384
11460 248/300 Write block 3155 len 4096 data: 3
11461 249/300 Read block 4550 len 16384
11462 250/300 Read block 794 len 36864
11463 251/300 Read block 3387 len 20480
11464 252/300 Write block 3122 len 4096 data: 3
11465 253/300 Read block 1520 len 8192
11466 254/300 Write block 3298 len 16384 data: 4 4 5 5
11467 255/300 Write block 2292 len 40960 data: 7 9 7 6 6 7 5 5 5 5
11468 256/300 Read block 550 len 24576
11469 257/300 Write block 765 len 8192 data: 4 4
11470 258/300 Read block 2689 len 20480
11471 259/300 Read block 1929 len 36864
11472 260/300 Read block 1348 len 16384
11473 261/300 Read block 3715 len 8192
11474 262/300 Read block 4800 len 12288
11475 263/300 Read block 2541 len 40960
11476 264/300 Write block 502 len 32768 data: 3 4 4 4 5 5 7 6
11477 265/300 Write block 3068 len 32768 data: 2 2 2 2 2 3 4 4
11478 266/300 Read block 2968 len 8192
11479 267/300 Read block 397 len 4096
11480 268/300 Flush
11481 269/300 Write block 1613 len 8192 data: 3 3
11482 270/300 Write block 43 len 40960 data: 3 3 2 2 2 2 2 4 3 3
11483 271/300 Write block 4314 len 28672 data: 5 6 5 6 9 9 9
11484 272/300 Write block 778 len 20480 data: 7 6 5 5 2
11485 273/300 Read block 4227 len 8192
11486 274/300 Read block 4985 len 40960
11487 275/300 Write block 3295 len 32768 data: 3 3 3 5 5 6 6 5
11488 276/300 Read block 1664 len 16384
11489 277/300 Write block 3436 len 4096 data: 3
11490 278/300 Read block 3066 len 36864
11491 279/300 Read block 3821 len 20480
11492 280/300 Read block 3609 len 8192
11493 281/300 Flush
11494 282/300 Flush
11495 283/300 Write block 695 len 32768 data: 3 3 3 3 3 3 3 2
11496 284/300 Write block 694 len 24576 data: 3 4 4 4 4 4
11497 285/300 Read block 598 len 32768
11498 286/300 Read block 24 len 20480
11499 287/300 Read block 1521 len 4096
11500 288/300 Read block 1015 len 40960
11501 289/300 Read block 4405 len 36864
11502 290/300 Write block 4409 len 16384 data: 3 3 3 2
11503 291/300 Write block 3188 len 16384 data: 5 4 3 2
11504 292/300 Flush
11505 293/300 Write block 1345 len 4096 data: 3
11506 294/300 Write block 1023 len 20480 data: 2 2 2 2 2
11507 295/300 Read block 1146 len 12288
11508 296/300 Read block 813 len 24576
11509 297/300 Read block 4123 len 8192
11510 298/300 Write block 4098 len 8192 data: 2 2
11511 299/300 Write block 1391 len 28672 data: 2 2 2 2 2 2 2
11512 300/300 Write block 1206 len 32768 data: 2 3 3 3 3 4 4 5
11513 301/300 Read block 2473 len 16384
11514 Main task has work to do, go find it
11515 got dsc Start(2)
11516 start 2
11517 Nov 01 16:52:55.514 INFO request completed, latency_us: 206, response_code: 204, uri: /start/cid/2, method: POST, req_id: 8d941f60-854a-4c1c-8dfb-0ef7a45ca679, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
11518 Replay: started 2, returned:Ok(())
11519 ----------------------------------------------------------------
11520 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11521 States: Active Active Offline
11522 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11523 4021 Acked Read 9 Done Done Sent false
11524 4022 Acked Write 10 Done Done Sent false
11525 4023 Acked Write 6 Done Done Sent false
11526 4024 Acked Write 6 Done Done Sent false
11527 4025 Acked Read 6 Done Done Sent false
11528 4026 Acked Write 6 Done Done Sent false
11529 4027 Acked Write 5 Done Done Sent false
11530 4028 Acked Flush 0 Done Done Sent false
11531 4029 Acked Write 6 Done Done Sent false
11532 4030 Acked Write 3 Done Done Sent false
11533 4031 Acked Read 4 Done Done Sent false
11534 4032 Acked Write 7 Done Done Sent false
11535 4033 Acked Read 4 Done Done Sent false
11536 4034 Acked Read 5 Done Done Sent false
11537 4035 Acked Write 8 Done Done Sent false
11538 4036 Acked Read 2 Done Done Sent false
11539 4037 Acked Read 4 Done Done Sent false
11540 4038 Acked Flush 0 Done Done Sent false
11541 4039 Acked Read 2 Done Done Sent false
11542 4040 Acked Read 3 Done Done Sent false
11543 4041 Acked Read 3 Done Done Sent false
11544 4042 Acked Write 4 Done Done Sent false
11545 4043 Acked Write 1 Done Done Sent false
11546 4044 Acked Write 6 Done Done Sent false
11547 4045 Acked Read 9 Done Done Sent false
11548 4046 Acked Read 6 Done Done Sent false
11549 4047 Acked Read 2 Done Done Sent false
11550 4048 Acked Read 10 Done Done Sent false
11551 4049 Acked Write 10 Done Done Sent false
11552 4050 Acked Write 8 Done Done Sent false
11553 4051 Acked Write 2 Done Done Sent false
11554 4052 Acked Write 5 Done Done Sent false
11555 4053 Acked Read 6 Done Done Sent false
11556 4054 Acked Read 1 Done Done Sent false
11557 4055 Acked Read 7 Done Done Sent false
11558 4056 Acked Write 8 Done Done Sent false
11559 4057 Acked Read 3 Done Done Sent false
11560 4058 Acked Write 5 Done Done Sent false
11561 4059 Acked Read 3 Done Done Sent false
11562 4060 Acked Write 1 Done Done Sent false
11563 4061 Acked Read 8 Done Done Sent false
11564 4062 Acked Write 7 Done Done Sent false
11565 4063 Acked Read 10 Done Done Sent false
11566 4064 Acked Read 7 Done Done Sent false
11567 4065 Acked Read 8 Done Done Sent false
11568 4066 Acked Write 8 Done Done Sent false
11569 4067 Acked Read 9 Done Done Sent false
11570 4068 Acked Write 2 Done Done Sent false
11571 4069 Acked Read 2 Done Done Sent false
11572 4070 Acked Read 2 Done Done Sent false
11573 4071 Acked Read 5 Done Done Sent false
11574 4072 Acked Read 3 Done Done Sent false
11575 4073 Acked Read 2 Done Done Sent false
11576 4074 Acked Read 1 Done Done Sent false
11577 4075 Acked Read 7 Done Done Sent false
11578 4076 Acked Read 3 Done Done Sent false
11579 4077 Acked Read 6 Done Done Sent false
11580 4078 Acked Read 10 Done Done Sent false
11581 4079 Acked Write 9 Done Done Sent false
11582 4080 Acked Read 9 Done Done Sent false
11583 4081 Acked Flush 0 Done Done Sent false
11584 4082 Acked Write 2 Done Done Sent false
11585 4083 Acked Write 8 Done Done Sent false
11586 4084 Acked Read 3 Done Done Sent false
11587 4085 Acked Write 2 Done Done Sent false
11588 4086 Acked Flush 0 Done Done Sent false
11589 4087 Acked Read 8 Done Done Sent false
11590 4088 Acked Read 6 Done Done Sent false
11591 4089 Acked Flush 0 Done Done Sent false
11592 4090 Acked Write 5 Done Done Sent false
11593 4091 Acked Read 2 Done Done Sent false
11594 4092 Acked Flush 0 Done Done Sent false
11595 4093 Acked Write 10 Done Done Sent false
11596 4094 Acked Flush 0 Done Done Sent false
11597 4095 Acked Write 3 Done Done Sent false
11598 4096 Acked Write 1 Done Done Sent false
11599 4097 Acked Flush 0 Done Done Sent false
11600 4098 Acked Read 6 Done Done Sent false
11601 4099 Acked Read 9 Done Done Sent false
11602 4100 Acked Write 3 Done Done Sent false
11603 4101 Acked Read 5 Done Done Sent false
11604 4102 Acked Read 10 Done Done Sent false
11605 4103 Acked Read 10 Done Done Sent false
11606 4104 Acked Read 1 Done Done Sent false
11607 4105 Acked Read 8 Done Done Sent false
11608 4106 Acked Write 2 Done Done Sent false
11609 4107 Acked Read 3 Done Done Sent false
11610 4108 Acked Read 5 Done Done Sent false
11611 4109 Acked Read 1 Done Done Sent false
11612 4110 Acked Read 4 Done Done Sent false
11613 4111 Acked Read 5 Done Done Sent false
11614 4112 Acked Write 8 Done Done Sent false
11615 4113 Acked Write 8 Done Done Sent false
11616 4114 Acked Write 3 Done Done Sent false
11617 4115 Acked Read 5 Done Done Sent false
11618 4116 Acked Read 2 Done Done Sent false
11619 4117 Acked Write 9 Done Done Sent false
11620 4118 Acked Read 2 Done Done Sent false
11621 4119 Acked Write 8 Done Done Sent false
11622 4120 Acked Read 10 Done Done Sent false
11623 4121 Acked Read 6 Done Done Sent false
11624 4122 Acked Read 4 Done Done Sent false
11625 4123 Acked Flush 0 Done Done Sent false
11626 4124 Acked Read 9 Done Done Sent false
11627 4125 Acked Read 9 Done Done Sent false
11628 4126 Acked Flush 0 Done Done Sent false
11629 4127 Acked Read 10 Done Done Sent false
11630 4128 Acked Write 6 Done Done Sent false
11631 4129 Acked Flush 0 Done Done Sent false
11632 4130 Acked Read 9 Done Done Sent false
11633 4131 Acked Read 8 Done Done Sent false
11634 4132 Acked Flush 0 Done Done Sent false
11635 4133 Acked Read 4 Done Done Sent false
11636 4134 Acked Read 10 Done Done Sent false
11637 4135 Acked Read 7 Done Done Sent false
11638 4136 Acked Read 1 Done Done Sent false
11639 4137 Acked Write 3 Done Done Sent false
11640 4138 Acked Read 6 Done Done Sent false
11641 4139 Acked Flush 0 Done Done Sent false
11642 4140 Acked Read 2 Done Done Sent false
11643 4141 Acked Write 4 Done Done Sent false
11644 4142 Acked Write 9 Done Done Sent false
11645 4143 Acked Write 9 Done Done Sent false
11646 4144 Acked Write 5 Done Done Sent false
11647 4145 Acked Flush 0 Done Done Sent false
11648 4146 Acked Write 7 Done Done Sent false
11649 4147 Acked Write 6 Done Done Sent false
11650 4148 Acked Read 1 Done Done Sent false
11651 4149 Acked Read 9 Done Done Sent false
11652 4150 Acked Write 8 Done Done Sent false
11653 4151 Acked Read 6 Done Done Sent false
11654 4152 Acked Write 1 Done Done Sent false
11655 4153 Acked Write 9 Done Done Sent false
11656 4154 Acked Read 6 Done Done Sent false
11657 4155 Acked Read 7 Done Done Sent false
11658 4156 Acked Write 10 Done Done Sent false
11659 4157 Acked Read 5 Done Done Sent false
11660 4158 Acked Read 4 Done Done Sent false
11661 4159 Acked Read 2 Done Done Sent false
11662 4160 Acked Write 2 Done Done Sent false
11663 4161 Acked Read 3 Done Done Sent false
11664 4162 Acked Read 7 Done Done Sent false
11665 4163 Acked Write 7 Done Done Sent false
11666 4164 Acked Read 6 Done Done Sent false
11667 4165 Acked Write 7 Done Done Sent false
11668 4166 Acked Write 1 Done Done Sent false
11669 4167 Acked Read 2 Done Done Sent false
11670 4168 Acked Read 7 Done Done Sent false
11671 4169 Acked Read 6 Done Done Sent false
11672 4170 Acked Write 8 Done Done Sent false
11673 4171 Acked Read 9 Done Done Sent false
11674 4172 Acked Write 3 Done Done Sent false
11675 4173 Acked Write 6 Done Done Sent false
11676 4174 Acked Write 6 Done Done Sent false
11677 4175 Acked Write 10 Done Done Sent false
11678 4176 Acked Read 2 Done Done Sent false
11679 4177 Acked Read 6 Done Done Sent false
11680 4178 Acked Write 5 Done Done Sent false
11681 4179 Acked Write 2 Done Done Sent false
11682 4180 Acked Flush 0 Done Done Sent false
11683 4181 Acked Read 3 Done Done Sent false
11684 4182 Acked Read 9 Done Done Sent false
11685 4183 Acked Read 6 Done Done Sent false
11686 4184 Acked Flush 0 Done Done Sent false
11687 4185 Acked Write 3 Done Done Sent false
11688 4186 Acked Read 10 Done Done Sent false
11689 4187 Acked Write 8 Done Done Sent false
11690 4188 Acked Flush 0 Done Done Sent false
11691 4189 Acked Write 8 Done Done Sent false
11692 4190 Acked Flush 0 Done Done Sent false
11693 4191 Acked Flush 0 Done Done Sent false
11694 4192 Acked Read 1 Done Done Sent false
11695 4193 Acked Read 9 Done Done Sent false
11696 4194 Acked Write 4 Done Done Sent false
11697 4195 Acked Flush 0 Done Done Sent false
11698 4196 Acked Read 4 Done Done Sent false
11699 4197 Acked Write 4 Done Done Sent false
11700 4198 Acked Read 7 Done Done Sent false
11701 4199 Acked Read 8 Done Done Sent false
11702 4200 Acked Read 6 Done Done Sent false
11703 4201 Acked Write 10 Done Done Sent false
11704 4202 Acked Read 4 Done Done Sent false
11705 4203 Acked Read 9 Done Done Sent false
11706 4204 Acked Write 4 Done Done Sent false
11707 4205 Acked Write 4 Done Done Sent false
11708 4206 Acked Write 7 Done Done Sent false
11709 4207 Acked Write 9 Done Done Sent false
11710 4208 Acked Write 7 Done Done Sent false
11711 4209 Acked Write 7 Done Done Sent false
11712 4210 Acked Read 7 Done Done Sent false
11713 4211 Acked Write 6 Done Done Sent false
11714 4212 Acked Read 9 Done Done Sent false
11715 4213 Acked Read 10 Done Done Sent false
11716 4214 Acked Write 9 Done Done Sent false
11717 4215 Acked Read 7 Done Done Sent false
11718 4216 Acked Read 2 Done Done Sent false
11719 4217 Acked Flush 0 Done Done Sent false
11720 4218 Acked Read 2 Done Done Sent false
11721 4219 Acked Read 4 Done Done Sent false
11722 4220 Acked Read 10 Done Done Sent false
11723 4221 Acked Flush 0 Done Done Sent false
11724 4222 Acked Read 8 Done Done Sent false
11725 4223 Acked Write 5 Done Done Sent false
11726 4224 Acked Read 10 Done Done Sent false
11727 4225 Acked Read 5 Done Done Sent false
11728 4226 Acked Read 8 Done Done Sent false
11729 4227 Acked Write 10 Done Done Sent false
11730 4228 Acked Write 5 Done Done Sent false
11731 4229 Acked Read 4 Done Done Sent false
11732 4230 Acked Read 7 Done Done Sent false
11733 4231 Acked Read 9 Done Done Sent false
11734 4232 Acked Write 7 Done Done Sent false
11735 4233 Acked Read 5 Done Done Sent false
11736 4234 Acked Read 8 Done Done Sent false
11737 4235 Acked Write 10 Done Done Sent false
11738 4236 Acked Write 8 Done Done Sent false
11739 4237 Acked Read 8 Done Done Sent false
11740 4238 Acked Read 3 Done Done Sent false
11741 4239 Acked Read 2 Done Done Sent false
11742 4240 Acked Write 1 Done Done Sent false
11743 4241 Acked Read 2 Done Done Sent false
11744 4242 Acked Flush 0 Done Done Sent false
11745 4243 Acked Read 6 Done Done Sent false
11746 4244 Acked Write 8 Done Done Sent false
11747 4245 Acked Write 2 Done Done Sent false
11748 4246 Acked Read 4 Done Done Sent false
11749 4247 Acked Write 3 Done Done Sent false
11750 4248 Acked Flush 0 Done Done Sent false
11751 4249 Acked Write 5 Done Done Sent false
11752 4250 Acked Write 2 Done Done Sent false
11753 4251 Acked Write 7 Done Done Sent false
11754 4252 Acked Write 9 Done Done Sent false
11755 4253 Acked Write 10 Done Done Sent false
11756 4254 Acked Read 1 Done Done Sent false
11757 4255 Acked Flush 0 Done Done Sent false
11758 4256 Acked Read 5 Done Done Sent false
11759 4257 Acked Write 3 Done Done Sent false
11760 4258 Acked Read 1 Done Done Sent false
11761 4259 Acked Write 2 Done Done Sent false
11762 4260 Acked Write 7 Done Done Sent false
11763 4261 Acked Write 3 Done Done Sent false
11764 4262 Acked Read 7 Done Done Sent false
11765 4263 Acked Read 4 Done Done Sent false
11766 4264 Acked Read 7 Done Done Sent false
11767 4265 Acked Write 5 Done Done Sent false
11768 4266 Acked Read 6 Done Done Sent false
11769 4267 Acked Read 4 Done Done Sent false
11770 4268 Acked Write 1 Done Done Sent false
11771 4269 Acked Read 4 Done Done Sent false
11772 4270 Acked Read 9 Done Done Sent false
11773 4271 Acked Read 5 Done Done Sent false
11774 4272 Acked Write 1 Done Done Sent false
11775 4273 Acked Read 2 Done Done Sent false
11776 4274 Acked Write 4 Done Done Sent false
11777 4275 Acked Write 10 Done Done Sent false
11778 4276 Acked Read 6 Done Done Sent false
11779 4277 Acked Write 2 Done Done Sent false
11780 4278 Acked Read 5 Done Done Sent false
11781 4279 Acked Read 9 Done Done Sent false
11782 4280 Acked Read 4 Done Done Sent false
11783 4281 Acked Read 2 Done Done Sent false
11784 4282 Acked Read 3 Done Done Sent false
11785 4283 Acked Read 10 Done Done Sent false
11786 4284 Acked Write 8 Done Done Sent false
11787 4285 Acked Write 8 Done Done Sent false
11788 4286 Acked Read 2 Done Done Sent false
11789 4287 Acked Read 1 Done Done Sent false
11790 4288 Acked Flush 0 Done Done Sent false
11791 4289 Acked Write 2 Done Done Sent false
11792 4290 Acked Write 10 Done Done Sent false
11793 4291 Acked Write 7 Done Done Sent false
11794 4292 Acked Write 5 Done Done Sent false
11795 4293 Acked Read 2 Done Done Sent false
11796 4294 Acked Read 10 Done Done Sent false
11797 4295 Acked Write 8 Done Done Sent false
11798 4296 Acked Read 4 Done Done Sent false
11799 4297 Acked Write 1 Done Done Sent false
11800 4298 Acked Read 9 Done Done Sent false
11801 4299 Acked Read 5 Done Done Sent false
11802 4300 Acked Read 2 Done Done Sent false
11803 4301 Acked Flush 0 Done Done Sent false
11804 4302 Acked Flush 0 Done Done Sent false
11805 4303 Acked Write 8 Done Done Sent false
11806 4304 Acked Write 6 Done Done Sent false
11807 4305 Acked Read 8 Done Done Sent false
11808 4306 Acked Read 5 Done Done Sent false
11809 4307 Acked Read 1 Done Done Sent false
11810 4308 Acked Read 10 Done Done Sent false
11811 4309 Acked Read 9 Done Done Sent false
11812 4310 Acked Write 4 Done Done Sent false
11813 4311 Acked Write 4 Done Done Sent false
11814 4312 Acked Flush 0 Done Done Sent false
11815 4313 Acked Write 1 Done Done Sent false
11816 4314 Acked Write 5 Done Done Sent false
11817 4315 Acked Read 3 Done Done Sent false
11818 4316 Acked Read 6 Done Done Sent false
11819 4317 Acked Read 2 Done Done Sent false
11820 4318 Acked Write 2 Done Done Sent false
11821 4319 Acked Write 7 Done Done Sent false
11822 4320 Acked Write 8 Done Done Sent false
11823 4321 Acked Read 4 Done Done Sent false
11824 STATES DS:0 DS:1 DS:2 TOTAL
11825 Sent 0 0 301 301
11826 Done 301 301 0 602
11827 Skipped 0 0 0 0
11828 Error 0 0 0 0
11829 Last Flush: 4312 4312 4020
11830 Downstairs last five completed: 4020 4019 4018 4017 4016
11831 Upstairs last five completed: 4321 4320 4319 4318 4317
11832 CLIENT: Up:0 ds:301 act:2
11833 [8830] got start action
11834 [8830] I am going to restart
11835 Starting downstairs at port 8830
11836 Make output file at "/tmp/dsc/downstairs-8830.txt"
11837 [8830][2] reports Starting
11838 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1209)
11839 [8830][2] reports Running
11840 ----------------------------------------------------------------
11841 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
11842 States: Active Active Offline
11843 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11844 4021 Acked Read 9 Done Done Sent false
11845 4022 Acked Write 10 Done Done Sent false
11846 4023 Acked Write 6 Done Done Sent false
11847 4024 Acked Write 6 Done Done Sent false
11848 4025 Acked Read 6 Done Done Sent false
11849 4026 Acked Write 6 Done Done Sent false
11850 4027 Acked Write 5 Done Done Sent false
11851 4028 Acked Flush 0 Done Done Sent false
11852 4029 Acked Write 6 Done Done Sent false
11853 4030 Acked Write 3 Done Done Sent false
11854 4031 Acked Read 4 Done Done Sent false
11855 4032 Acked Write 7 Done Done Sent false
11856 4033 Acked Read 4 Done Done Sent false
11857 4034 Acked Read 5 Done Done Sent false
11858 4035 Acked Write 8 Done Done Sent false
11859 4036 Acked Read 2 Done Done Sent false
11860 4037 Acked Read 4 Done Done Sent false
11861 4038 Acked Flush 0 Done Done Sent false
11862 4039 Acked Read 2 Done Done Sent false
11863 4040 Acked Read 3 Done Done Sent false
11864 4041 Acked Read 3 Done Done Sent false
11865 4042 Acked Write 4 Done Done Sent false
11866 4043 Acked Write 1 Done Done Sent false
11867 4044 Acked Write 6 Done Done Sent false
11868 4045 Acked Read 9 Done Done Sent false
11869 4046 Acked Read 6 Done Done Sent false
11870 4047 Acked Read 2 Done Done Sent false
11871 4048 Acked Read 10 Done Done Sent false
11872 4049 Acked Write 10 Done Done Sent false
11873 4050 Acked Write 8 Done Done Sent false
11874 4051 Acked Write 2 Done Done Sent false
11875 4052 Acked Write 5 Done Done Sent false
11876 4053 Acked Read 6 Done Done Sent false
11877 4054 Acked Read 1 Done Done Sent false
11878 4055 Acked Read 7 Done Done Sent false
11879 4056 Acked Write 8 Done Done Sent false
11880 4057 Acked Read 3 Done Done Sent false
11881 4058 Acked Write 5 Done Done Sent false
11882 4059 Acked Read 3 Done Done Sent false
11883 4060 Acked Write 1 Done Done Sent false
11884 4061 Acked Read 8 Done Done Sent false
11885 4062 Acked Write 7 Done Done Sent false
11886 4063 Acked Read 10 Done Done Sent false
11887 4064 Acked Read 7 Done Done Sent false
11888 4065 Acked Read 8 Done Done Sent false
11889 4066 Acked Write 8 Done Done Sent false
11890 4067 Acked Read 9 Done Done Sent false
11891 4068 Acked Write 2 Done Done Sent false
11892 4069 Acked Read 2 Done Done Sent false
11893 4070 Acked Read 2 Done Done Sent false
11894 4071 Acked Read 5 Done Done Sent false
11895 4072 Acked Read 3 Done Done Sent false
11896 4073 Acked Read 2 Done Done Sent false
11897 4074 Acked Read 1 Done Done Sent false
11898 4075 Acked Read 7 Done Done Sent false
11899 4076 Acked Read 3 Done Done Sent false
11900 4077 Acked Read 6 Done Done Sent false
11901 4078 Acked Read 10 Done Done Sent false
11902 4079 Acked Write 9 Done Done Sent false
11903 4080 Acked Read 9 Done Done Sent false
11904 4081 Acked Flush 0 Done Done Sent false
11905 4082 Acked Write 2 Done Done Sent false
11906 4083 Acked Write 8 Done Done Sent false
11907 4084 Acked Read 3 Done Done Sent false
11908 4085 Acked Write 2 Done Done Sent false
11909 4086 Acked Flush 0 Done Done Sent false
11910 4087 Acked Read 8 Done Done Sent false
11911 4088 Acked Read 6 Done Done Sent false
11912 4089 Acked Flush 0 Done Done Sent false
11913 4090 Acked Write 5 Done Done Sent false
11914 4091 Acked Read 2 Done Done Sent false
11915 4092 Acked Flush 0 Done Done Sent false
11916 4093 Acked Write 10 Done Done Sent false
11917 4094 Acked Flush 0 Done Done Sent false
11918 4095 Acked Write 3 Done Done Sent false
11919 4096 Acked Write 1 Done Done Sent false
11920 4097 Acked Flush 0 Done Done Sent false
11921 4098 Acked Read 6 Done Done Sent false
11922 4099 Acked Read 9 Done Done Sent false
11923 4100 Acked Write 3 Done Done Sent false
11924 4101 Acked Read 5 Done Done Sent false
11925 4102 Acked Read 10 Done Done Sent false
11926 4103 Acked Read 10 Done Done Sent false
11927 4104 Acked Read 1 Done Done Sent false
11928 4105 Acked Read 8 Done Done Sent false
11929 4106 Acked Write 2 Done Done Sent false
11930 4107 Acked Read 3 Done Done Sent false
11931 4108 Acked Read 5 Done Done Sent false
11932 4109 Acked Read 1 Done Done Sent false
11933 4110 Acked Read 4 Done Done Sent false
11934 4111 Acked Read 5 Done Done Sent false
11935 4112 Acked Write 8 Done Done Sent false
11936 4113 Acked Write 8 Done Done Sent false
11937 4114 Acked Write 3 Done Done Sent false
11938 4115 Acked Read 5 Done Done Sent false
11939 4116 Acked Read 2 Done Done Sent false
11940 4117 Acked Write 9 Done Done Sent false
11941 4118 Acked Read 2 Done Done Sent false
11942 4119 Acked Write 8 Done Done Sent false
11943 4120 Acked Read 10 Done Done Sent false
11944 4121 Acked Read 6 Done Done Sent false
11945 4122 Acked Read 4 Done Done Sent false
11946 4123 Acked Flush 0 Done Done Sent false
11947 4124 Acked Read 9 Done Done Sent false
11948 4125 Acked Read 9 Done Done Sent false
11949 4126 Acked Flush 0 Done Done Sent false
11950 4127 Acked Read 10 Done Done Sent false
11951 4128 Acked Write 6 Done Done Sent false
11952 4129 Acked Flush 0 Done Done Sent false
11953 4130 Acked Read 9 Done Done Sent false
11954 4131 Acked Read 8 Done Done Sent false
11955 4132 Acked Flush 0 Done Done Sent false
11956 4133 Acked Read 4 Done Done Sent false
11957 4134 Acked Read 10 Done Done Sent false
11958 4135 Acked Read 7 Done Done Sent false
11959 4136 Acked Read 1 Done Done Sent false
11960 4137 Acked Write 3 Done Done Sent false
11961 4138 Acked Read 6 Done Done Sent false
11962 4139 Acked Flush 0 Done Done Sent false
11963 4140 Acked Read 2 Done Done Sent false
11964 4141 Acked Write 4 Done Done Sent false
11965 4142 Acked Write 9 Done Done Sent false
11966 4143 Acked Write 9 Done Done Sent false
11967 4144 Acked Write 5 Done Done Sent false
11968 4145 Acked Flush 0 Done Done Sent false
11969 4146 Acked Write 7 Done Done Sent false
11970 4147 Acked Write 6 Done Done Sent false
11971 4148 Acked Read 1 Done Done Sent false
11972 4149 Acked Read 9 Done Done Sent false
11973 4150 Acked Write 8 Done Done Sent false
11974 4151 Acked Read 6 Done Done Sent false
11975 4152 Acked Write 1 Done Done Sent false
11976 4153 Acked Write 9 Done Done Sent false
11977 4154 Acked Read 6 Done Done Sent false
11978 4155 Acked Read 7 Done Done Sent false
11979 4156 Acked Write 10 Done Done Sent false
11980 4157 Acked Read 5 Done Done Sent false
11981 4158 Acked Read 4 Done Done Sent false
11982 4159 Acked Read 2 Done Done Sent false
11983 4160 Acked Write 2 Done Done Sent false
11984 4161 Acked Read 3 Done Done Sent false
11985 4162 Acked Read 7 Done Done Sent false
11986 4163 Acked Write 7 Done Done Sent false
11987 4164 Acked Read 6 Done Done Sent false
11988 4165 Acked Write 7 Done Done Sent false
11989 4166 Acked Write 1 Done Done Sent false
11990 4167 Acked Read 2 Done Done Sent false
11991 4168 Acked Read 7 Done Done Sent false
11992 4169 Acked Read 6 Done Done Sent false
11993 4170 Acked Write 8 Done Done Sent false
11994 4171 Acked Read 9 Done Done Sent false
11995 4172 Acked Write 3 Done Done Sent false
11996 4173 Acked Write 6 Done Done Sent false
11997 4174 Acked Write 6 Done Done Sent false
11998 4175 Acked Write 10 Done Done Sent false
11999 4176 Acked Read 2 Done Done Sent false
12000 4177 Acked Read 6 Done Done Sent false
12001 4178 Acked Write 5 Done Done Sent false
12002 4179 Acked Write 2 Done Done Sent false
12003 4180 Acked Flush 0 Done Done Sent false
12004 4181 Acked Read 3 Done Done Sent false
12005 4182 Acked Read 9 Done Done Sent false
12006 4183 Acked Read 6 Done Done Sent false
12007 4184 Acked Flush 0 Done Done Sent false
12008 4185 Acked Write 3 Done Done Sent false
12009 4186 Acked Read 10 Done Done Sent false
12010 4187 Acked Write 8 Done Done Sent false
12011 4188 Acked Flush 0 Done Done Sent false
12012 4189 Acked Write 8 Done Done Sent false
12013 4190 Acked Flush 0 Done Done Sent false
12014 4191 Acked Flush 0 Done Done Sent false
12015 4192 Acked Read 1 Done Done Sent false
12016 4193 Acked Read 9 Done Done Sent false
12017 4194 Acked Write 4 Done Done Sent false
12018 4195 Acked Flush 0 Done Done Sent false
12019 4196 Acked Read 4 Done Done Sent false
12020 4197 Acked Write 4 Done Done Sent false
12021 4198 Acked Read 7 Done Done Sent false
12022 4199 Acked Read 8 Done Done Sent false
12023 4200 Acked Read 6 Done Done Sent false
12024 4201 Acked Write 10 Done Done Sent false
12025 4202 Acked Read 4 Done Done Sent false
12026 4203 Acked Read 9 Done Done Sent false
12027 4204 Acked Write 4 Done Done Sent false
12028 4205 Acked Write 4 Done Done Sent false
12029 4206 Acked Write 7 Done Done Sent false
12030 4207 Acked Write 9 Done Done Sent false
12031 4208 Acked Write 7 Done Done Sent false
12032 4209 Acked Write 7 Done Done Sent false
12033 4210 Acked Read 7 Done Done Sent false
12034 4211 Acked Write 6 Done Done Sent false
12035 4212 Acked Read 9 Done Done Sent false
12036 4213 Acked Read 10 Done Done Sent false
12037 4214 Acked Write 9 Done Done Sent false
12038 4215 Acked Read 7 Done Done Sent false
12039 4216 Acked Read 2 Done Done Sent false
12040 4217 Acked Flush 0 Done Done Sent false
12041 4218 Acked Read 2 Done Done Sent false
12042 4219 Acked Read 4 Done Done Sent false
12043 4220 Acked Read 10 Done Done Sent false
12044 4221 Acked Flush 0 Done Done Sent false
12045 4222 Acked Read 8 Done Done Sent false
12046 4223 Acked Write 5 Done Done Sent false
12047 4224 Acked Read 10 Done Done Sent false
12048 4225 Acked Read 5 Done Done Sent false
12049 4226 Acked Read 8 Done Done Sent false
12050 4227 Acked Write 10 Done Done Sent false
12051 4228 Acked Write 5 Done Done Sent false
12052 4229 Acked Read 4 Done Done Sent false
12053 4230 Acked Read 7 Done Done Sent false
12054 4231 Acked Read 9 Done Done Sent false
12055 4232 Acked Write 7 Done Done Sent false
12056 4233 Acked Read 5 Done Done Sent false
12057 4234 Acked Read 8 Done Done Sent false
12058 4235 Acked Write 10 Done Done Sent false
12059 4236 Acked Write 8 Done Done Sent false
12060 4237 Acked Read 8 Done Done Sent false
12061 4238 Acked Read 3 Done Done Sent false
12062 4239 Acked Read 2 Done Done Sent false
12063 4240 Acked Write 1 Done Done Sent false
12064 4241 Acked Read 2 Done Done Sent false
12065 4242 Acked Flush 0 Done Done Sent false
12066 4243 Acked Read 6 Done Done Sent false
12067 4244 Acked Write 8 Done Done Sent false
12068 4245 Acked Write 2 Done Done Sent false
12069 4246 Acked Read 4 Done Done Sent false
12070 4247 Acked Write 3 Done Done Sent false
12071 4248 Acked Flush 0 Done Done Sent false
12072 4249 Acked Write 5 Done Done Sent false
12073 4250 Acked Write 2 Done Done Sent false
12074 4251 Acked Write 7 Done Done Sent false
12075 4252 Acked Write 9 Done Done Sent false
12076 4253 Acked Write 10 Done Done Sent false
12077 4254 Acked Read 1 Done Done Sent false
12078 4255 Acked Flush 0 Done Done Sent false
12079 4256 Acked Read 5 Done Done Sent false
12080 4257 Acked Write 3 Done Done Sent false
12081 4258 Acked Read 1 Done Done Sent false
12082 4259 Acked Write 2 Done Done Sent false
12083 4260 Acked Write 7 Done Done Sent false
12084 4261 Acked Write 3 Done Done Sent false
12085 4262 Acked Read 7 Done Done Sent false
12086 4263 Acked Read 4 Done Done Sent false
12087 4264 Acked Read 7 Done Done Sent false
12088 4265 Acked Write 5 Done Done Sent false
12089 4266 Acked Read 6 Done Done Sent false
12090 4267 Acked Read 4 Done Done Sent false
12091 4268 Acked Write 1 Done Done Sent false
12092 4269 Acked Read 4 Done Done Sent false
12093 4270 Acked Read 9 Done Done Sent false
12094 4271 Acked Read 5 Done Done Sent false
12095 4272 Acked Write 1 Done Done Sent false
12096 4273 Acked Read 2 Done Done Sent false
12097 4274 Acked Write 4 Done Done Sent false
12098 4275 Acked Write 10 Done Done Sent false
12099 4276 Acked Read 6 Done Done Sent false
12100 4277 Acked Write 2 Done Done Sent false
12101 4278 Acked Read 5 Done Done Sent false
12102 4279 Acked Read 9 Done Done Sent false
12103 4280 Acked Read 4 Done Done Sent false
12104 4281 Acked Read 2 Done Done Sent false
12105 4282 Acked Read 3 Done Done Sent false
12106 4283 Acked Read 10 Done Done Sent false
12107 4284 Acked Write 8 Done Done Sent false
12108 4285 Acked Write 8 Done Done Sent false
12109 4286 Acked Read 2 Done Done Sent false
12110 4287 Acked Read 1 Done Done Sent false
12111 4288 Acked Flush 0 Done Done Sent false
12112 4289 Acked Write 2 Done Done Sent false
12113 4290 Acked Write 10 Done Done Sent false
12114 4291 Acked Write 7 Done Done Sent false
12115 4292 Acked Write 5 Done Done Sent false
12116 4293 Acked Read 2 Done Done Sent false
12117 4294 Acked Read 10 Done Done Sent false
12118 4295 Acked Write 8 Done Done Sent false
12119 4296 Acked Read 4 Done Done Sent false
12120 4297 Acked Write 1 Done Done Sent false
12121 4298 Acked Read 9 Done Done Sent false
12122 4299 Acked Read 5 Done Done Sent false
12123 4300 Acked Read 2 Done Done Sent false
12124 4301 Acked Flush 0 Done Done Sent false
12125 4302 Acked Flush 0 Done Done Sent false
12126 4303 Acked Write 8 Done Done Sent false
12127 4304 Acked Write 6 Done Done Sent false
12128 4305 Acked Read 8 Done Done Sent false
12129 4306 Acked Read 5 Done Done Sent false
12130 4307 Acked Read 1 Done Done Sent false
12131 4308 Acked Read 10 Done Done Sent false
12132 4309 Acked Read 9 Done Done Sent false
12133 4310 Acked Write 4 Done Done Sent false
12134 4311 Acked Write 4 Done Done Sent false
12135 4312 Acked Flush 0 Done Done Sent false
12136 4313 Acked Write 1 Done Done Sent false
12137 4314 Acked Write 5 Done Done Sent false
12138 4315 Acked Read 3 Done Done Sent false
12139 4316 Acked Read 6 Done Done Sent false
12140 4317 Acked Read 2 Done Done Sent false
12141 4318 Acked Write 2 Done Done Sent false
12142 4319 Acked Write 7 Done Done Sent false
12143 4320 Acked Write 8 Done Done Sent false
12144 4321 Acked Read 4 Done Done Sent false
12145 4322 Acked Flush 0 Done Done Sent false
12146 STATES DS:0 DS:1 DS:2 TOTAL
12147 Sent 0 0 302 302
12148 Done 302 302 0 604
12149 Skipped 0 0 0 0
12150 Error 0 0 0 0
12151 Last Flush: 4322 4322 4020
12152 Downstairs last five completed: 4020 4019 4018 4017 4016
12153 Upstairs last five completed: 4322 4321 4320 4319 4318
12154 CLIENT: Up:0 ds:302 act:2
121552024-11-01T16:53:01.223ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
121562024-11-01T16:53:01.224ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48614) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
121572024-11-01T16:53:01.225ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
121582024-11-01T16:53:01.225ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
121592024-11-01T16:53:01.225ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
121602024-11-01T16:53:01.225ZINFOcrucible: send last flush ID to this DS: 4020 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
121612024-11-01T16:53:01.225ZINFOcrucible: Replied this last flush ID: 4020 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
121622024-11-01T16:53:01.225ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
121632024-11-01T16:53:01.226ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
121642024-11-01T16:53:01.226ZINFOcrucible: [2] client re-new 302 jobs since flush 4020 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
121652024-11-01T16:53:01.226ZINFOcrucible: [2] Replayed 302 jobs since flush: 4020 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
12166 ----------------------------------------------------------------
12167 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12168 Downstairs last five completed: 4322 4321 4320 4319 4318
12169 Upstairs last five completed: 4322 4321 4320 4319 4318
12170 CLIENT: Up:0 ds:0 act:3
12171 Replay: All jobs finished
12172 Main task has work to do, go find it
12173 got dsc Stop(0)
12174 stop 0
12175 Nov 01 16:53:03.520 INFO request completed, latency_us: 370, response_code: 204, uri:[8810] Got stop action so:false kr:false
12176 /stop/cid/0, method: POST, req_id: b9bdc4ab-721d-4dcc-a20d-a5b87c0f7c93, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
121772024-11-01T16:53:03.521ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
121782024-11-01T16:53:03.521ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
121792024-11-01T16:53:03.521ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
121802024-11-01T16:53:03.521ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
12181 Nov 01 16:53:03.521 INFO request completed, latency_us: 178, response_code: 200, uri: /state/cid/0, method: GET, req_id: 47f9fe50-e8f8-447b-a7a0-82c5012e4137, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
121822024-11-01T16:53:03.521ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
12183 [8810] Exited with: ExitStatus(unix_wait_status(9))
12184 [8810][0] reports Exit
12185 Nov 01 16:53:07.524 INFO request completed, latency_us: 238, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4dab21d4-e13b-43a0-94a9-785550c0af77, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
12186 001/300 Flush
12187 002/300 Read block 1949 len 40960
12188 003/300 Write block 2813 len 40960 data: 5 5 6 5 4 3 3 3 3 4
12189 004/300 Read block 2834 len 40960
12190 005/300 Write block 3173 len 4096 data: 5
12191 006/300 Read block 868 len 36864
12192 007/300 Read block 4019 len 40960
12193 008/300 Flush
12194 009/300 Read block 3369 len 4096
12195 010/300 Read block 4165 len 12288
12196 011/300 Read block 1004 len 40960
12197 012/300 Write block 344 len 20480 data: 4 4 4 2 2
12198 013/300 Flush
12199 014/300 Write block 1022 len 36864 data: 2 3 3 3 3 3 2 2 2
12200 015/300 Write block 71 len 28672 data: 4 4 2 2 3 3 3
12201 016/300 Read block 570 len 36864
12202 017/300 Write block 1314 len 12288 data: 3 3 3
12203 018/300 Read block 465 len 16384
12204 019/300 Read block 865 len 24576
12205 020/300 Write block 541 len 8192 data: 3 3
12206 021/300 Write block 4449 len 40960 data: 4 4 5 5 4 4 4 2 2 2
12207 022/300 Write block 2082 len 12288 data: 2 2 3
12208 023/300 Write block 2232 len 16384 data: 4 4 4 5
12209 024/300 Read block 2268 len 4096
12210 025/300 Write block 3364 len 40960 data: 3 3 3 4 4 3 4 4 5 6
12211 026/300 Read block 4436 len 4096
12212 027/300 Read block 1823 len 40960
12213 028/300 Read block 3979 len 40960
12214 029/300 Write block 3479 len 16384 data: 4 4 4 3
12215 030/300 Write block 132 len 4096 data: 2
12216 031/300 Write block 2235 len 12288 data: 6 5 5
12217 032/300 Write block 380 len 4096 data: 2
12218 033/300 Write block 1093 len 4096 data: 4
12219 034/300 Write block 4481 len 8192 data: 2 2
12220 035/300 Read block 3023 len 36864
12221 036/300 Read block 3349 len 36864
12222 037/300 Write block 1868 len 32768 data: 4 4 5 4 6 6 5 4
12223 038/300 Write block 2550 len 28672 data: 4 4 4 5 5 3 3
12224 039/300 Read block 4347 len 12288
12225 040/300 Write block 2180 len 12288 data: 3 5 5
12226 041/300 Write block 3819 len 24576 data: 5 4 3 2 3 4
12227 042/300 Read block 4294 len 32768
12228 043/300 Read block 4186 len 16384
12229 044/300 Read block 3101 len 24576
12230 045/300 Write block 4800 len 12288 data: 2 2 2
12231 046/300 Write block 1913 len 12288 data: 2 2 3
12232 047/300 Write block 3300 len 4096 data: 7
12233 048/300 Read block 641 len 40960
12234 049/300 Write block 3713 len 28672 data: 3 4 4 4 5 5 4
12235 050/300 Write block 2116 len 40960 data: 2 2 2 3 3 4 5 4 5 4
12236 051/300 Read block 712 len 4096
12237 052/300 Read block 415 len 24576
12238 053/300 Read block 402 len 16384
12239 054/300 Write block 4349 len 32768 data: 3 2 2 2 2 2 2 2
12240 055/300 Write block 3981 len 20480 data: 2 2 2 2 2
12241 056/300 Write block 364 len 8192 data: 2 3
12242 057/300 Read block 4349 len 32768
12243 058/300 Read block 3310 len 4096
12244 059/300 Flush
12245 060/300 Read block 3102 len 24576
12246 061/300 Read block 1187 len 32768
12247 062/300 Read block 2688 len 20480
12248 063/300 Read block 3779 len 40960
12249 064/300 Write block 1948 len 24576 data: 4 4 3 2 2 2
12250 065/300 Write block 2829 len 28672 data: 2 2 2 3 3 3 3
12251 066/300 Write block 4067 len 40960 data: 3 2 2 2 2 2 4 4 3 4
12252 067/300 Read block 4452 len 32768
12253 068/300 Read block 1949 len 16384
12254 069/300 Write block 1997 len 32768 data: 2 2 3 3 3 3 4 4
12255 070/300 Flush
12256 071/300 Read block 78 len 28672
12257 072/300 Read block 1225 len 4096
12258 073/300 Read block 3845 len 40960
12259 074/300 Read block 1186 len 20480
12260 075/300 Read block 3595 len 32768
12261 076/300 Write block 3830 len 4096 data: 5
12262 077/300 Write block 2085 len 24576 data: 4 5 4 3 3 4
12263 078/300 Write block 4426 len 36864 data: 4 5 5 5 5 5 5 4 4
12264 079/300 Read block 3448 len 36864
12265 080/300 Flush
12266 081/300 Write block 1858 len 32768 data: 2 2 2 2 3 3 3 4
12267 082/300 Read block 3803 len 40960
12268 083/300 Read block 4545 len 12288
12269 084/300 Flush
12270 085/300 Write block 2223 len 32768 data: 3 3 3 3 3 3 3 4
12271 086/300 Write block 1418 len 36864 data: 2 2 2 2 2 2 3 2 2
12272 087/300 Read block 4256 len 4096
12273 088/300 Flush
12274 089/300 Read block 1042 len 28672
12275 090/300 Read block 2111 len 40960
12276 091/300 Read block 4909 len 16384
12277 092/300 Read block 517 len 12288
12278 093/300 Write block 3475 len 28672 data: 4 4 4 4 5 5 5
12279 094/300 Write block 277 len 8192 data: 3 3
12280 095/300 Write block 1900 len 40960 data: 2 2 2 3 4 5 5 5 4 4
12281 096/300 Write block 3286 len 32768 data: 3 3 4 4 5 5 4 3
12282 097/300 Read block 3441 len 28672
12283 098/300 Read block 3563 len 20480
12284 099/300 Read block 3944 len 8192
12285 100/300 Read block 1353 len 4096
12286 101/300 Write block 3752 len 36864 data: 5 5 5 5 4 2 2 2 2
12287 102/300 Write block 417 len 28672 data: 3 3 2 3 3 3 3
12288 103/300 Read block 1333 len 20480
12289 104/300 Write block 2287 len 32768 data: 5 6 6 6 7 8 10 8
12290 105/300 Write block 4815 len 36864 data: 4 4 3 3 2 3 4 4 4
12291 106/300 Read block 53 len 20480
12292 107/300 Flush
12293 108/300 Write block 3973 len 40960 data: 3 3 3 4 3 3 3 2 3 3
12294 109/300 Write block 3583 len 40960 data: 5 6 6 5 5 6 5 5 3 4
12295 110/300 Read block 27 len 4096
12296 111/300 Read block 2794 len 4096
12297 112/300 Read block 3998 len 28672
12298 113/300 Write block 4914 len 20480 data: 4 4 5 6 5
12299 114/300 Write block 3448 len 32768 data: 5 5 5 4 3 3 3 4
12300 115/300 Read block 1792 len 40960
12301 116/300 Read block 1543 len 32768
12302 117/300 Write block 732 len 40960 data: 5 4 3 4 4 3 3 4 3 3
12303 118/300 Read block 1818 len 20480
12304 119/300 Read block 4084 len 24576
12305 120/300 Write block 2609 len 16384 data: 3 3 4 4
12306 121/300 Write block 1873 len 36864 data: 7 6 5 5 5 4 5 4 4
12307 122/300 Read block 3623 len 20480
12308 123/300 Read block 1186 len 24576
12309 124/300 Read block 1341 len 4096
12310 125/300 Write block 4872 len 24576 data: 4 4 5 4 5 4
12311 126/300 Flush
12312 127/300 Read block 831 len 4096
12313 128/300 Write block 2736 len 32768 data: 3 3 3 5 5 7 6 6
12314 129/300 Write block 1651 len 20480 data: 3 3 3 3 3
12315 130/300 Write block 1470 len 8192 data: 2 2
12316 131/300 Write block 4301 len 16384 data: 3 3 3 2
12317 132/300 Read block 919 len 36864
12318 133/300 Write block 2986 len 20480 data: 2 4 4 4 4
12319 134/300 Flush
12320 135/300 Read block 3445 len 12288
12321 136/300 Write block 4524 len 32768 data: 3 4 3 4 4 3 3 3
12322 137/300 Read block 4736 len 32768
12323 138/300 Write block 2111 len 4096 data: 4
12324 139/300 Write block 2714 len 24576 data: 4 4 4 4 4 4
12325 140/300 Read block 1515 len 24576
12326 141/300 Read block 4211 len 20480
12327 142/300 Read block 3291 len 40960
12328 143/300 Read block 703 len 40960
12329 144/300 Write block 4950 len 20480 data: 5 5 5 5 5
12330 145/300 Read block 4473 len 12288
12331 146/300 Flush
12332 147/300 Read block 2474 len 4096
12333 148/300 Read block 3114 len 36864
12334 149/300 Read block 3584 len 4096
12335 150/300 Read block 4812 len 12288
12336 151/300 Write block 1982 len 8192 data: 3 3
12337 152/300 Read block 471 len 36864
12338 153/300 Flush
12339 154/300 Write block 2330 len 8192 data: 4 4
12340 155/300 Read block 1661 len 16384
12341 156/300 Write block 4006 len 24576 data: 3 3 3 4 3 3
12342 157/300 Read block 592 len 40960
12343 158/300 Flush
12344 159/300 Write block 2286 len 4096 data: 5
12345 160/300 Read block 4595 len 8192
12346 161/300 Write block 2546 len 12288 data: 6 4 4
12347 162/300 Write block 1784 len 36864 data: 5 5 5 4 4 3 3 3 2
12348 163/300 Write block 457 len 8192 data: 5 5
12349 164/300 Write block 519 len 28672 data: 4 4 5 4 5 5 5
12350 165/300 Write block 1022 len 36864 data: 3 4 4 4 4 4 3 3 3
12351 166/300 Read block 2228 len 32768
12352 167/300 Write block 1587 len 32768 data: 4 4 4 4 4 3 2 3
12353 168/300 Read block 1576 len 8192
12354 169/300 Read block 2569 len 12288
12355 170/300 Write block 3702 len 4096 data: 3
12356 171/300 Read block 1669 len 36864
12357 172/300 Write block 3160 len 24576 data: 2 2 2 2 2 2
12358 173/300 Read block 1507 len 40960
12359 174/300 Flush
12360 175/300 Flush
12361 176/300 Read block 4708 len 32768
12362 177/300 Read block 310 len 36864
12363 178/300 Write block 475 len 32768 data: 3 3 3 6 5 5 5 5
12364 179/300 Read block 2461 len 28672
12365 180/300 Write block 776 len 40960 data: 7 7 8 7 6 6 3 2 3 3
12366 181/300 Write block 4900 len 20480 data: 3 3 3 4 4
12367 182/300 Read block 3352 len 36864
12368 183/300 Read block 3121 len 36864
12369 184/300 Read block 4698 len 40960
12370 185/300 Read block 4247 len 16384
12371 186/300 Read block 3731 len 36864
12372 187/300 Flush
12373 188/300 Write block 1790 len 16384 data: 4 4 3 2
12374 189/300 Read block 2191 len 24576
12375 190/300 Read block 2446 len 32768
12376 191/300 Read block 1808 len 4096
12377 192/300 Flush
12378 193/300 Read block 3000 len 32768
12379 194/300 Read block 1802 len 12288
12380 195/300 Read block 206 len 20480
12381 196/300 Read block 919 len 4096
12382 197/300 Write block 3542 len 4096 data: 3
12383 198/300 Write block 1770 len 36864 data: 3 3 4 5 3 3 3 3 3
12384 199/300 Write block 1902 len 36864 data: 3 4 5 6 6 6 5 5 4
12385 200/300 Write block 4915 len 40960 data: 5 6 7 6 4 4 5 4 5 5
12386 201/300 Read block 1539 len 40960
12387 202/300 Read block 1035 len 24576
12388 203/300 Read block 2587 len 36864
12389 204/300 Read block 659 len 24576
12390 205/300 Write block 1869 len 4096 data: 5
12391 206/300 Read block 3013 len 36864
12392 207/300 Flush
12393 208/300 Write block 4008 len 16384 data: 4 5 4 4
12394 209/300 Read block 1347 len 36864
12395 210/300 Flush
12396 211/300 Read block 4994 len 24576
12397 212/300 Read block 457 len 8192
12398 213/300 Write block 1053 len 24576 data: 2 2 2 3 3 3
12399 214/300 Read block 3209 len 32768
12400 215/300 Write block 2729 len 8192 data: 4 3
12401 216/300 Write block 691 len 16384 data: 3 3 2 4
12402 217/300 Read block 4528 len 32768
12403 218/300 Flush
12404 219/300 Write block 4995 len 12288 data: 3 3 4
12405 220/300 Read block 1745 len 40960
12406 221/300 Read block 1212 len 36864
12407 222/300 Write block 2550 len 4096 data: 5
12408 223/300 Read block 3338 len 40960
12409 224/300 Write block 436 len 20480 data: 3 2 2 2 2
12410 225/300 Write block 2303 len 32768 data: 4 3 2 2 2 2 2 2
12411 226/300 Write block 1372 len 8192 data: 3 3
12412 227/300 Read block 3736 len 24576
12413 228/300 Read block 2295 len 8192
12414 229/300 Read block 4721 len 20480
12415 230/300 Write block 4072 len 24576 data: 3 5 5 4 5 4
12416 231/300 Write block 2854 len 28672 data: 2 2 2 3 3 4 4
12417 232/300 Read block 2573 len 36864
12418 233/300 Read block 2249 len 16384
12419 234/300 Read block 923 len 36864
12420 235/300 Read block 3814 len 8192
12421 236/300 Flush
12422 237/300 Read block 4286 len 16384
12423 238/300 Write block 4280 len 24576 data: 5 5 4 4 3 3
12424 239/300 Write block 4001 len 32768 data: 2 3 3 3 3 4 4 5
12425 240/300 Flush
12426 241/300 Write block 2765 len 4096 data: 3
12427 242/300 Read block 1 len 28672
12428 243/300 Read block 4800 len 12288
12429 244/300 Read block 1187 len 40960
12430 245/300 Read block 4407 len 36864
12431 246/300 Flush
12432 247/300 Read block 4390 len 36864
12433 248/300 Write block 2949 len 4096 data: 2
12434 249/300 Read block 2350 len 4096
12435 250/300 Write block 373 len 32768 data: 3 2 2 2 2 2 2 3
12436 251/300 Write block 1503 len 40960 data: 3 3 4 4 4 4 4 4 4 5
12437 252/300 Read block 1900 len 12288
12438 253/300 Flush
12439 254/300 Read block 930 len 40960
12440 255/300 Write block 3745 len 40960 data: 2 3 5 6 5 5 5 6 6 6
12441 256/300 Read block 2368 len 4096
12442 257/300 Read block 3959 len 4096
12443 258/300 Write block 1207 len 8192 data: 4 4
12444 259/300 Write block 1171 len 40960 data: 3 4 5 5 6 7 7 7 7 5
12445 260/300 Read block 204 len 28672
12446 261/300 Write block 1044 len 40960 data: 4 4 3 3 3 2 2 2 2 3
12447 262/300 Read block 3144 len 4096
12448 263/300 Write block 2641 len 24576 data: 4 4 5 5 5 5
12449 264/300 Read block 4122 len 28672
12450 265/300 Read block 2775 len 20480
12451 266/300 Read block 3514 len 24576
12452 267/300 Write block 1454 len 24576 data: 3 2 2 2 2 2
12453 268/300 Flush
12454 269/300 Read block 3928 len 28672
12455 270/300 Write block 2369 len 12288 data: 6 6 6
12456 271/300 Read block 2123 len 32768
12457 272/300 Write block 4019 len 36864 data: 2 2 2 3 4 5 4 4 4
12458 273/300 Read block 3535 len 12288
12459 274/300 Write block 2241 len 40960 data: 5 5 5 5 5 4 3 3 2 2
12460 275/300 Write block 2554 len 32768 data: 6 4 4 2 2 2 2 3
12461 276/300 Read block 4264 len 32768
12462 277/300 Flush
12463 278/300 Write block 1820 len 36864 data: 2 2 2 2 2 2 2 2 2
12464 279/300 Read block 2095 len 36864
12465 280/300 Write block 4503 len 24576 data: 4 5 5 6 5 4
12466 281/300 Write block 4376 len 40960 data: 4 5 5 4 4 5 4 4 4 4
12467 282/300 Read block 2483 len 28672
12468 283/300 Read block 4359 len 16384
12469 284/300 Read block 2277 len 8192
12470 285/300 Read block 1058 len 8192
12471 286/300 Write block 2216 len 16384 data: 4 3 3 3
12472 287/300 Read block 433 len 32768
12473 288/300 Read block 1294 len 20480
12474 289/300 Read block 3636 len 4096
12475 290/300 Read block 1139 len 12288
12476 291/300 Read block 3150 len 16384
12477 292/300 Write block 2449 len 4096 data: 3
12478 293/300 Read block 3441 len 8192
12479 294/300 Write block 1593 len 40960 data: 3 4 2 2 3 2 2 2 2 3
12480 295/300 Read block 4124 len 24576
12481 296/300 Read block 2110 len 20480
12482 297/300 Read block 4602 len 40960
12483 298/300 Write block 181 len 8192 data: 3 5
12484 299/300 Write block 1693 len 4096 data: 4
12485 300/300 Read block 4959 len 20480
12486 301/300 Read block 3480 len 4096
12487 Main task has work to do, go find it
12488 got dsc Start(0)
12489 start 0
12490 Nov 01 16:53:07.818 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/0, method: POST, req_id: 7f40213d-4298-4c7b-b300-95844d21ec3b, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
12491 Replay: started 0, returned:Ok(())
12492 ----------------------------------------------------------------
12493 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
12494 States: Offline Active Active
12495 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12496 4323 Acked Flush 0 Sent Done Done false
12497 4324 Acked Read 10 Sent Done Done false
12498 4325 Acked Write 10 Sent Done Done false
12499 4326 Acked Read 10 Sent Done Done false
12500 4327 Acked Write 1 Sent Done Done false
12501 4328 Acked Read 9 Sent Done Done false
12502 4329 Acked Read 10 Sent Done Done false
12503 4330 Acked Flush 0 Sent Done Done false
12504 4331 Acked Read 1 Sent Done Done false
12505 4332 Acked Read 3 Sent Done Done false
12506 4333 Acked Read 10 Sent Done Done false
12507 4334 Acked Write 5 Sent Done Done false
12508 4335 Acked Flush 0 Sent Done Done false
12509 4336 Acked Write 9 Sent Done Done false
12510 4337 Acked Write 7 Sent Done Done false
12511 4338 Acked Read 9 Sent Done Done false
12512 4339 Acked Write 3 Sent Done Done false
12513 4340 Acked Read 4 Sent Done Done false
12514 4341 Acked Read 6 Sent Done Done false
12515 4342 Acked Write 2 Sent Done Done false
12516 4343 Acked Write 10 Sent Done Done false
12517 4344 Acked Write 3 Sent Done Done false
12518 4345 Acked Write 4 Sent Done Done false
12519 4346 Acked Read 1 Sent Done Done false
12520 4347 Acked Write 10 Sent Done Done false
12521 4348 Acked Read 1 Sent Done Done false
12522 4349 Acked Read 10 Sent Done Done false
12523 4350 Acked Read 10 Sent Done Done false
12524 4351 Acked Write 4 Sent Done Done false
12525 4352 Acked Write 1 Sent Done Done false
12526 4353 Acked Write 3 Sent Done Done false
12527 4354 Acked Write 1 Sent Done Done false
12528 4355 Acked Write 1 Sent Done Done false
12529 4356 Acked Write 2 Sent Done Done false
12530 4357 Acked Read 9 Sent Done Done false
12531 4358 Acked Read 9 Sent Done Done false
12532 4359 Acked Write 8 Sent Done Done false
12533 4360 Acked Write 7 Sent Done Done false
12534 4361 Acked Read 3 Sent Done Done false
12535 4362 Acked Write 3 Sent Done Done false
12536 4363 Acked Write 6 Sent Done Done false
12537 4364 Acked Read 8 Sent Done Done false
12538 4365 Acked Read 4 Sent Done Done false
12539 4366 Acked Read 6 Sent Done Done false
12540 4367 Acked Write 3 Sent Done Done false
12541 4368 Acked Write 3 Sent Done Done false
12542 4369 Acked Write 1 Sent Done Done false
12543 4370 Acked Read 10 Sent Done Done false
12544 4371 Acked Write 7 Sent Done Done false
12545 4372 Acked Write 10 Sent Done Done false
12546 4373 Acked Read 1 Sent Done Done false
12547 4374 Acked Read 6 Sent Done Done false
12548 4375 Acked Flush 0 Sent Done Done false
12549 4376 Acked Read 4 Sent Done Done false
12550 4377 Acked Write 8 Sent Done Done false
12551 4378 Acked Write 5 Sent Done Done false
12552 4379 Acked Write 2 Sent Done Done false
12553 4380 Acked Read 8 Sent Done Done false
12554 4381 Acked Read 1 Sent Done Done false
12555 4382 Acked Flush 0 Sent Done Done false
12556 4383 Acked Read 6 Sent Done Done false
12557 4384 Acked Read 8 Sent Done Done false
12558 4385 Acked Read 5 Sent Done Done false
12559 4386 Acked Read 10 Sent Done Done false
12560 4387 Acked Write 6 Sent Done Done false
12561 4388 Acked Write 7 Sent Done Done false
12562 4389 Acked Write 10 Sent Done Done false
12563 4390 Acked Read 8 Sent Done Done false
12564 4391 Acked Read 4 Sent Done Done false
12565 4392 Acked Write 8 Sent Done Done false
12566 4393 Acked Flush 0 Sent Done Done false
12567 4394 Acked Read 7 Sent Done Done false
12568 4395 Acked Read 1 Sent Done Done false
12569 4396 Acked Read 10 Sent Done Done false
12570 4397 Acked Read 5 Sent Done Done false
12571 4398 Acked Read 8 Sent Done Done false
12572 4399 Acked Write 1 Sent Done Done false
12573 4400 Acked Write 6 Sent Done Done false
12574 4401 Acked Write 9 Sent Done Done false
12575 4402 Acked Read 9 Sent Done Done false
12576 4403 Acked Flush 0 Sent Done Done false
12577 4404 Acked Write 8 Sent Done Done false
12578 4405 Acked Read 10 Sent Done Done false
12579 4406 Acked Read 3 Sent Done Done false
12580 4407 Acked Flush 0 Sent Done Done false
12581 4408 Acked Write 8 Sent Done Done false
12582 4409 Acked Write 9 Sent Done Done false
12583 4410 Acked Read 1 Sent Done Done false
12584 4411 Acked Flush 0 Sent Done Done false
12585 4412 Acked Read 7 Sent Done Done false
12586 4413 Acked Read 10 Sent Done Done false
12587 4414 Acked Read 4 Sent Done Done false
12588 4415 Acked Read 3 Sent Done Done false
12589 4416 Acked Write 7 Sent Done Done false
12590 4417 Acked Write 2 Sent Done Done false
12591 4418 Acked Write 10 Sent Done Done false
12592 4419 Acked Write 8 Sent Done Done false
12593 4420 Acked Read 7 Sent Done Done false
12594 4421 Acked Read 5 Sent Done Done false
12595 4422 Acked Read 2 Sent Done Done false
12596 4423 Acked Read 1 Sent Done Done false
12597 4424 Acked Write 9 Sent Done Done false
12598 4425 Acked Write 7 Sent Done Done false
12599 4426 Acked Read 5 Sent Done Done false
12600 4427 Acked Write 8 Sent Done Done false
12601 4428 Acked Write 9 Sent Done Done false
12602 4429 Acked Read 5 Sent Done Done false
12603 4430 Acked Flush 0 Sent Done Done false
12604 4431 Acked Write 10 Sent Done Done false
12605 4432 Acked Write 10 Sent Done Done false
12606 4433 Acked Read 1 Sent Done Done false
12607 4434 Acked Read 1 Sent Done Done false
12608 4435 Acked Read 7 Sent Done Done false
12609 4436 Acked Write 5 Sent Done Done false
12610 4437 Acked Write 8 Sent Done Done false
12611 4438 Acked Read 10 Sent Done Done false
12612 4439 Acked Read 8 Sent Done Done false
12613 4440 Acked Write 10 Sent Done Done false
12614 4441 Acked Read 5 Sent Done Done false
12615 4442 Acked Read 6 Sent Done Done false
12616 4443 Acked Write 4 Sent Done Done false
12617 4444 Acked Write 9 Sent Done Done false
12618 4445 Acked Read 5 Sent Done Done false
12619 4446 Acked Read 6 Sent Done Done false
12620 4447 Acked Read 1 Sent Done Done false
12621 4448 Acked Write 6 Sent Done Done false
12622 4449 Acked Flush 0 Sent Done Done false
12623 4450 Acked Read 1 Sent Done Done false
12624 4451 Acked Write 8 Sent Done Done false
12625 4452 Acked Write 5 Sent Done Done false
12626 4453 Acked Write 2 Sent Done Done false
12627 4454 Acked Write 4 Sent Done Done false
12628 4455 Acked Read 9 Sent Done Done false
12629 4456 Acked Write 5 Sent Done Done false
12630 4457 Acked Flush 0 Sent Done Done false
12631 4458 Acked Read 3 Sent Done Done false
12632 4459 Acked Write 8 Sent Done Done false
12633 4460 Acked Read 8 Sent Done Done false
12634 4461 Acked Write 1 Sent Done Done false
12635 4462 Acked Write 6 Sent Done Done false
12636 4463 Acked Read 6 Sent Done Done false
12637 4464 Acked Read 5 Sent Done Done false
12638 4465 Acked Read 10 Sent Done Done false
12639 4466 Acked Read 10 Sent Done Done false
12640 4467 Acked Write 5 Sent Done Done false
12641 4468 Acked Read 3 Sent Done Done false
12642 4469 Acked Flush 0 Sent Done Done false
12643 4470 Acked Read 1 Sent Done Done false
12644 4471 Acked Read 9 Sent Done Done false
12645 4472 Acked Read 1 Sent Done Done false
12646 4473 Acked Read 3 Sent Done Done false
12647 4474 Acked Write 2 Sent Done Done false
12648 4475 Acked Read 9 Sent Done Done false
12649 4476 Acked Flush 0 Sent Done Done false
12650 4477 Acked Write 2 Sent Done Done false
12651 4478 Acked Read 4 Sent Done Done false
12652 4479 Acked Write 6 Sent Done Done false
12653 4480 Acked Read 10 Sent Done Done false
12654 4481 Acked Flush 0 Sent Done Done false
12655 4482 Acked Write 1 Sent Done Done false
12656 4483 Acked Read 2 Sent Done Done false
12657 4484 Acked Write 3 Sent Done Done false
12658 4485 Acked Write 9 Sent Done Done false
12659 4486 Acked Write 2 Sent Done Done false
12660 4487 Acked Write 7 Sent Done Done false
12661 4488 Acked Write 9 Sent Done Done false
12662 4489 Acked Read 8 Sent Done Done false
12663 4490 Acked Write 8 Sent Done Done false
12664 4491 Acked Read 2 Sent Done Done false
12665 4492 Acked Read 3 Sent Done Done false
12666 4493 Acked Write 1 Sent Done Done false
12667 4494 Acked Read 9 Sent Done Done false
12668 4495 Acked Write 6 Sent Done Done false
12669 4496 Acked Read 10 Sent Done Done false
12670 4497 Acked Flush 0 Sent Done Done false
12671 4498 Acked Flush 0 Sent Done Done false
12672 4499 Acked Read 8 Sent Done Done false
12673 4500 Acked Read 9 Sent Done Done false
12674 4501 Acked Write 8 Sent Done Done false
12675 4502 Acked Read 7 Sent Done Done false
12676 4503 Acked Write 10 Sent Done Done false
12677 4504 Acked Write 5 Sent Done Done false
12678 4505 Acked Read 9 Sent Done Done false
12679 4506 Acked Read 9 Sent Done Done false
12680 4507 Acked Read 10 Sent Done Done false
12681 4508 Acked Read 4 Sent Done Done false
12682 4509 Acked Read 9 Sent Done Done false
12683 4510 Acked Flush 0 Sent Done Done false
12684 4511 Acked Write 4 Sent Done Done false
12685 4512 Acked Read 6 Sent Done Done false
12686 4513 Acked Read 8 Sent Done Done false
12687 4514 Acked Read 1 Sent Done Done false
12688 4515 Acked Flush 0 Sent Done Done false
12689 4516 Acked Read 8 Sent Done Done false
12690 4517 Acked Read 3 Sent Done Done false
12691 4518 Acked Read 5 Sent Done Done false
12692 4519 Acked Read 1 Sent Done Done false
12693 4520 Acked Write 1 Sent Done Done false
12694 4521 Acked Write 9 Sent Done Done false
12695 4522 Acked Write 9 Sent Done Done false
12696 4523 Acked Write 10 Sent Done Done false
12697 4524 Acked Read 10 Sent Done Done false
12698 4525 Acked Read 6 Sent Done Done false
12699 4526 Acked Read 9 Sent Done Done false
12700 4527 Acked Read 6 Sent Done Done false
12701 4528 Acked Write 1 Sent Done Done false
12702 4529 Acked Read 9 Sent Done Done false
12703 4530 Acked Flush 0 Sent Done Done false
12704 4531 Acked Write 4 Sent Done Done false
12705 4532 Acked Read 9 Sent Done Done false
12706 4533 Acked Flush 0 Sent Done Done false
12707 4534 Acked Read 6 Sent Done Done false
12708 4535 Acked Read 2 Sent Done Done false
12709 4536 Acked Write 6 Sent Done Done false
12710 4537 Acked Read 8 Sent Done Done false
12711 4538 Acked Write 2 Sent Done Done false
12712 4539 Acked Write 4 Sent Done Done false
12713 4540 Acked Read 8 Sent Done Done false
12714 4541 Acked Flush 0 Sent Done Done false
12715 4542 Acked Write 3 Sent Done Done false
12716 4543 Acked Read 10 Sent Done Done false
12717 4544 Acked Read 9 Sent Done Done false
12718 4545 Acked Write 1 Sent Done Done false
12719 4546 Acked Read 10 Sent Done Done false
12720 4547 Acked Write 5 Sent Done Done false
12721 4548 Acked Write 8 Sent Done Done false
12722 4549 Acked Write 2 Sent Done Done false
12723 4550 Acked Read 6 Sent Done Done false
12724 4551 Acked Read 2 Sent Done Done false
12725 4552 Acked Read 5 Sent Done Done false
12726 4553 Acked Write 6 Sent Done Done false
12727 4554 Acked Write 7 Sent Done Done false
12728 4555 Acked Read 9 Sent Done Done false
12729 4556 Acked Read 4 Sent Done Done false
12730 4557 Acked Read 9 Sent Done Done false
12731 4558 Acked Read 2 Sent Done Done false
12732 4559 Acked Flush 0 Sent Done Done false
12733 4560 Acked Read 4 Sent Done Done false
12734 4561 Acked Write 6 Sent Done Done false
12735 4562 Acked Write 8 Sent Done Done false
12736 4563 Acked Flush 0 Sent Done Done false
12737 4564 Acked Write 1 Sent Done Done false
12738 4565 Acked Read 7 Sent Done Done false
12739 4566 Acked Read 3 Sent Done Done false
12740 4567 Acked Read 10 Sent Done Done false
12741 4568 Acked Read 9 Sent Done Done false
12742 4569 Acked Flush 0 Sent Done Done false
12743 4570 Acked Read 9 Sent Done Done false
12744 4571 Acked Write 1 Sent Done Done false
12745 4572 Acked Read 1 Sent Done Done false
12746 4573 Acked Write 8 Sent Done Done false
12747 4574 Acked Write 10 Sent Done Done false
12748 4575 Acked Read 3 Sent Done Done false
12749 4576 Acked Flush 0 Sent Done Done false
12750 4577 Acked Read 10 Sent Done Done false
12751 4578 Acked Write 10 Sent Done Done false
12752 4579 Acked Read 1 Sent Done Done false
12753 4580 Acked Read 1 Sent Done Done false
12754 4581 Acked Write 2 Sent Done Done false
12755 4582 Acked Write 10 Sent Done Done false
12756 4583 Acked Read 7 Sent Done Done false
12757 4584 Acked Write 10 Sent Done Done false
12758 4585 Acked Read 1 Sent Done Done false
12759 4586 Acked Write 6 Sent Done Done false
12760 4587 Acked Read 7 Sent Done Done false
12761 4588 Acked Read 5 Sent Done Done false
12762 4589 Acked Read 6 Sent Done Done false
12763 4590 Acked Write 6 Sent Done Done false
12764 4591 Acked Flush 0 Sent Done Done false
12765 4592 Acked Read 7 Sent Done Done false
12766 4593 Acked Write 3 Sent Done Done false
12767 4594 Acked Read 8 Sent Done Done false
12768 4595 Acked Write 9 Sent Done Done false
12769 4596 Acked Read 3 Sent Done Done false
12770 4597 Acked Write 10 Sent Done Done false
12771 4598 Acked Write 8 Sent Done Done false
12772 4599 Acked Read 8 Sent Done Done false
12773 4600 Acked Flush 0 Sent Done Done false
12774 4601 Acked Write 9 Sent Done Done false
12775 4602 Acked Read 9 Sent Done Done false
12776 4603 Acked Write 6 Sent Done Done false
12777 4604 Acked Write 10 Sent Done Done false
12778 4605 Acked Read 7 Sent Done Done false
12779 4606 Acked Read 4 Sent Done Done false
12780 4607 Acked Read 2 Sent Done Done false
12781 4608 Acked Read 2 Sent Done Done false
12782 4609 Acked Write 4 Sent Done Done false
12783 4610 Acked Read 8 Sent Done Done false
12784 4611 Acked Read 5 Sent Done Done false
12785 4612 Acked Read 1 Sent Done Done false
12786 4613 Acked Read 3 Sent Done Done false
12787 4614 Acked Read 4 Sent Done Done false
12788 4615 Acked Write 1 Sent Done Done false
12789 4616 Acked Read 2 Sent Done Done false
12790 4617 Acked Write 10 Sent Done Done false
12791 4618 Acked Read 6 Sent Done Done false
12792 4619 Acked Read 5 Sent Done Done false
12793 4620 Acked Read 10 Sent Done Done false
12794 4621 Acked Write 2 Sent Done Done false
12795 4622 Acked Write 1 Sent Done Done false
12796 4623 Acked Read 5 Sent Done Done false
12797 4624 Acked Read 1 Sent Done Done false
12798 STATES DS:0 DS:1 DS:2 TOTAL
12799 Sent 302 0 0 302
12800 Done 0 302 302 604
12801 Skipped 0 0 0 0
12802 Error 0 0 0 0
12803 Last Flush: 4322 4600 4600
12804 Downstairs last five completed: 4322 4321 4320 4319 4318
12805 Upstairs last five completed: 4624 4623 4622 4621 4620
12806 CLIENT: Up:0 ds:302 act:2
12807 [8810] got start action
12808 [8810] I am going to restart
12809 Starting downstairs at port 8810
12810 Make output file at "/tmp/dsc/downstairs-8810.txt"
12811 [8810][0] reports Starting
12812 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1210)
12813 [8810][0] reports Running
12814 ----------------------------------------------------------------
12815 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
12816 States: Offline Active Active
12817 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12818 4323 Acked Flush 0 Sent Done Done false
12819 4324 Acked Read 10 Sent Done Done false
12820 4325 Acked Write 10 Sent Done Done false
12821 4326 Acked Read 10 Sent Done Done false
12822 4327 Acked Write 1 Sent Done Done false
12823 4328 Acked Read 9 Sent Done Done false
12824 4329 Acked Read 10 Sent Done Done false
12825 4330 Acked Flush 0 Sent Done Done false
12826 4331 Acked Read 1 Sent Done Done false
12827 4332 Acked Read 3 Sent Done Done false
12828 4333 Acked Read 10 Sent Done Done false
12829 4334 Acked Write 5 Sent Done Done false
12830 4335 Acked Flush 0 Sent Done Done false
12831 4336 Acked Write 9 Sent Done Done false
12832 4337 Acked Write 7 Sent Done Done false
12833 4338 Acked Read 9 Sent Done Done false
12834 4339 Acked Write 3 Sent Done Done false
12835 4340 Acked Read 4 Sent Done Done false
12836 4341 Acked Read 6 Sent Done Done false
12837 4342 Acked Write 2 Sent Done Done false
12838 4343 Acked Write 10 Sent Done Done false
12839 4344 Acked Write 3 Sent Done Done false
12840 4345 Acked Write 4 Sent Done Done false
12841 4346 Acked Read 1 Sent Done Done false
12842 4347 Acked Write 10 Sent Done Done false
12843 4348 Acked Read 1 Sent Done Done false
12844 4349 Acked Read 10 Sent Done Done false
12845 4350 Acked Read 10 Sent Done Done false
12846 4351 Acked Write 4 Sent Done Done false
12847 4352 Acked Write 1 Sent Done Done false
12848 4353 Acked Write 3 Sent Done Done false
12849 4354 Acked Write 1 Sent Done Done false
12850 4355 Acked Write 1 Sent Done Done false
12851 4356 Acked Write 2 Sent Done Done false
12852 4357 Acked Read 9 Sent Done Done false
12853 4358 Acked Read 9 Sent Done Done false
12854 4359 Acked Write 8 Sent Done Done false
12855 4360 Acked Write 7 Sent Done Done false
12856 4361 Acked Read 3 Sent Done Done false
12857 4362 Acked Write 3 Sent Done Done false
12858 4363 Acked Write 6 Sent Done Done false
12859 4364 Acked Read 8 Sent Done Done false
12860 4365 Acked Read 4 Sent Done Done false
12861 4366 Acked Read 6 Sent Done Done false
12862 4367 Acked Write 3 Sent Done Done false
12863 4368 Acked Write 3 Sent Done Done false
12864 4369 Acked Write 1 Sent Done Done false
12865 4370 Acked Read 10 Sent Done Done false
12866 4371 Acked Write 7 Sent Done Done false
12867 4372 Acked Write 10 Sent Done Done false
12868 4373 Acked Read 1 Sent Done Done false
12869 4374 Acked Read 6 Sent Done Done false
12870 4375 Acked Flush 0 Sent Done Done false
12871 4376 Acked Read 4 Sent Done Done false
12872 4377 Acked Write 8 Sent Done Done false
12873 4378 Acked Write 5 Sent Done Done false
12874 4379 Acked Write 2 Sent Done Done false
12875 4380 Acked Read 8 Sent Done Done false
12876 4381 Acked Read 1 Sent Done Done false
12877 4382 Acked Flush 0 Sent Done Done false
12878 4383 Acked Read 6 Sent Done Done false
12879 4384 Acked Read 8 Sent Done Done false
12880 4385 Acked Read 5 Sent Done Done false
12881 4386 Acked Read 10 Sent Done Done false
12882 4387 Acked Write 6 Sent Done Done false
12883 4388 Acked Write 7 Sent Done Done false
12884 4389 Acked Write 10 Sent Done Done false
12885 4390 Acked Read 8 Sent Done Done false
12886 4391 Acked Read 4 Sent Done Done false
12887 4392 Acked Write 8 Sent Done Done false
12888 4393 Acked Flush 0 Sent Done Done false
12889 4394 Acked Read 7 Sent Done Done false
12890 4395 Acked Read 1 Sent Done Done false
12891 4396 Acked Read 10 Sent Done Done false
12892 4397 Acked Read 5 Sent Done Done false
12893 4398 Acked Read 8 Sent Done Done false
12894 4399 Acked Write 1 Sent Done Done false
12895 4400 Acked Write 6 Sent Done Done false
12896 4401 Acked Write 9 Sent Done Done false
12897 4402 Acked Read 9 Sent Done Done false
12898 4403 Acked Flush 0 Sent Done Done false
12899 4404 Acked Write 8 Sent Done Done false
12900 4405 Acked Read 10 Sent Done Done false
12901 4406 Acked Read 3 Sent Done Done false
12902 4407 Acked Flush 0 Sent Done Done false
12903 4408 Acked Write 8 Sent Done Done false
12904 4409 Acked Write 9 Sent Done Done false
12905 4410 Acked Read 1 Sent Done Done false
12906 4411 Acked Flush 0 Sent Done Done false
12907 4412 Acked Read 7 Sent Done Done false
12908 4413 Acked Read 10 Sent Done Done false
12909 4414 Acked Read 4 Sent Done Done false
12910 4415 Acked Read 3 Sent Done Done false
12911 4416 Acked Write 7 Sent Done Done false
12912 4417 Acked Write 2 Sent Done Done false
12913 4418 Acked Write 10 Sent Done Done false
12914 4419 Acked Write 8 Sent Done Done false
12915 4420 Acked Read 7 Sent Done Done false
12916 4421 Acked Read 5 Sent Done Done false
12917 4422 Acked Read 2 Sent Done Done false
12918 4423 Acked Read 1 Sent Done Done false
12919 4424 Acked Write 9 Sent Done Done false
12920 4425 Acked Write 7 Sent Done Done false
12921 4426 Acked Read 5 Sent Done Done false
12922 4427 Acked Write 8 Sent Done Done false
12923 4428 Acked Write 9 Sent Done Done false
12924 4429 Acked Read 5 Sent Done Done false
12925 4430 Acked Flush 0 Sent Done Done false
12926 4431 Acked Write 10 Sent Done Done false
12927 4432 Acked Write 10 Sent Done Done false
12928 4433 Acked Read 1 Sent Done Done false
12929 4434 Acked Read 1 Sent Done Done false
12930 4435 Acked Read 7 Sent Done Done false
12931 4436 Acked Write 5 Sent Done Done false
12932 4437 Acked Write 8 Sent Done Done false
12933 4438 Acked Read 10 Sent Done Done false
12934 4439 Acked Read 8 Sent Done Done false
12935 4440 Acked Write 10 Sent Done Done false
12936 4441 Acked Read 5 Sent Done Done false
12937 4442 Acked Read 6 Sent Done Done false
12938 4443 Acked Write 4 Sent Done Done false
12939 4444 Acked Write 9 Sent Done Done false
12940 4445 Acked Read 5 Sent Done Done false
12941 4446 Acked Read 6 Sent Done Done false
12942 4447 Acked Read 1 Sent Done Done false
12943 4448 Acked Write 6 Sent Done Done false
12944 4449 Acked Flush 0 Sent Done Done false
12945 4450 Acked Read 1 Sent Done Done false
12946 4451 Acked Write 8 Sent Done Done false
12947 4452 Acked Write 5 Sent Done Done false
12948 4453 Acked Write 2 Sent Done Done false
12949 4454 Acked Write 4 Sent Done Done false
12950 4455 Acked Read 9 Sent Done Done false
12951 4456 Acked Write 5 Sent Done Done false
12952 4457 Acked Flush 0 Sent Done Done false
12953 4458 Acked Read 3 Sent Done Done false
12954 4459 Acked Write 8 Sent Done Done false
12955 4460 Acked Read 8 Sent Done Done false
12956 4461 Acked Write 1 Sent Done Done false
12957 4462 Acked Write 6 Sent Done Done false
12958 4463 Acked Read 6 Sent Done Done false
12959 4464 Acked Read 5 Sent Done Done false
12960 4465 Acked Read 10 Sent Done Done false
12961 4466 Acked Read 10 Sent Done Done false
12962 4467 Acked Write 5 Sent Done Done false
12963 4468 Acked Read 3 Sent Done Done false
12964 4469 Acked Flush 0 Sent Done Done false
12965 4470 Acked Read 1 Sent Done Done false
12966 4471 Acked Read 9 Sent Done Done false
12967 4472 Acked Read 1 Sent Done Done false
12968 4473 Acked Read 3 Sent Done Done false
12969 4474 Acked Write 2 Sent Done Done false
12970 4475 Acked Read 9 Sent Done Done false
12971 4476 Acked Flush 0 Sent Done Done false
12972 4477 Acked Write 2 Sent Done Done false
12973 4478 Acked Read 4 Sent Done Done false
12974 4479 Acked Write 6 Sent Done Done false
12975 4480 Acked Read 10 Sent Done Done false
12976 4481 Acked Flush 0 Sent Done Done false
12977 4482 Acked Write 1 Sent Done Done false
12978 4483 Acked Read 2 Sent Done Done false
12979 4484 Acked Write 3 Sent Done Done false
12980 4485 Acked Write 9 Sent Done Done false
12981 4486 Acked Write 2 Sent Done Done false
12982 4487 Acked Write 7 Sent Done Done false
12983 4488 Acked Write 9 Sent Done Done false
12984 4489 Acked Read 8 Sent Done Done false
12985 4490 Acked Write 8 Sent Done Done false
12986 4491 Acked Read 2 Sent Done Done false
12987 4492 Acked Read 3 Sent Done Done false
12988 4493 Acked Write 1 Sent Done Done false
12989 4494 Acked Read 9 Sent Done Done false
12990 4495 Acked Write 6 Sent Done Done false
12991 4496 Acked Read 10 Sent Done Done false
12992 4497 Acked Flush 0 Sent Done Done false
12993 4498 Acked Flush 0 Sent Done Done false
12994 4499 Acked Read 8 Sent Done Done false
12995 4500 Acked Read 9 Sent Done Done false
12996 4501 Acked Write 8 Sent Done Done false
12997 4502 Acked Read 7 Sent Done Done false
12998 4503 Acked Write 10 Sent Done Done false
12999 4504 Acked Write 5 Sent Done Done false
13000 4505 Acked Read 9 Sent Done Done false
13001 4506 Acked Read 9 Sent Done Done false
13002 4507 Acked Read 10 Sent Done Done false
13003 4508 Acked Read 4 Sent Done Done false
13004 4509 Acked Read 9 Sent Done Done false
13005 4510 Acked Flush 0 Sent Done Done false
13006 4511 Acked Write 4 Sent Done Done false
13007 4512 Acked Read 6 Sent Done Done false
13008 4513 Acked Read 8 Sent Done Done false
13009 4514 Acked Read 1 Sent Done Done false
13010 4515 Acked Flush 0 Sent Done Done false
13011 4516 Acked Read 8 Sent Done Done false
13012 4517 Acked Read 3 Sent Done Done false
13013 4518 Acked Read 5 Sent Done Done false
13014 4519 Acked Read 1 Sent Done Done false
13015 4520 Acked Write 1 Sent Done Done false
13016 4521 Acked Write 9 Sent Done Done false
13017 4522 Acked Write 9 Sent Done Done false
13018 4523 Acked Write 10 Sent Done Done false
13019 4524 Acked Read 10 Sent Done Done false
13020 4525 Acked Read 6 Sent Done Done false
13021 4526 Acked Read 9 Sent Done Done false
13022 4527 Acked Read 6 Sent Done Done false
13023 4528 Acked Write 1 Sent Done Done false
13024 4529 Acked Read 9 Sent Done Done false
13025 4530 Acked Flush 0 Sent Done Done false
13026 4531 Acked Write 4 Sent Done Done false
13027 4532 Acked Read 9 Sent Done Done false
13028 4533 Acked Flush 0 Sent Done Done false
13029 4534 Acked Read 6 Sent Done Done false
13030 4535 Acked Read 2 Sent Done Done false
13031 4536 Acked Write 6 Sent Done Done false
13032 4537 Acked Read 8 Sent Done Done false
13033 4538 Acked Write 2 Sent Done Done false
13034 4539 Acked Write 4 Sent Done Done false
13035 4540 Acked Read 8 Sent Done Done false
13036 4541 Acked Flush 0 Sent Done Done false
13037 4542 Acked Write 3 Sent Done Done false
13038 4543 Acked Read 10 Sent Done Done false
13039 4544 Acked Read 9 Sent Done Done false
13040 4545 Acked Write 1 Sent Done Done false
13041 4546 Acked Read 10 Sent Done Done false
13042 4547 Acked Write 5 Sent Done Done false
13043 4548 Acked Write 8 Sent Done Done false
13044 4549 Acked Write 2 Sent Done Done false
13045 4550 Acked Read 6 Sent Done Done false
13046 4551 Acked Read 2 Sent Done Done false
13047 4552 Acked Read 5 Sent Done Done false
13048 4553 Acked Write 6 Sent Done Done false
13049 4554 Acked Write 7 Sent Done Done false
13050 4555 Acked Read 9 Sent Done Done false
13051 4556 Acked Read 4 Sent Done Done false
13052 4557 Acked Read 9 Sent Done Done false
13053 4558 Acked Read 2 Sent Done Done false
13054 4559 Acked Flush 0 Sent Done Done false
13055 4560 Acked Read 4 Sent Done Done false
13056 4561 Acked Write 6 Sent Done Done false
13057 4562 Acked Write 8 Sent Done Done false
13058 4563 Acked Flush 0 Sent Done Done false
13059 4564 Acked Write 1 Sent Done Done false
13060 4565 Acked Read 7 Sent Done Done false
13061 4566 Acked Read 3 Sent Done Done false
13062 4567 Acked Read 10 Sent Done Done false
13063 4568 Acked Read 9 Sent Done Done false
13064 4569 Acked Flush 0 Sent Done Done false
13065 4570 Acked Read 9 Sent Done Done false
13066 4571 Acked Write 1 Sent Done Done false
13067 4572 Acked Read 1 Sent Done Done false
13068 4573 Acked Write 8 Sent Done Done false
13069 4574 Acked Write 10 Sent Done Done false
13070 4575 Acked Read 3 Sent Done Done false
13071 4576 Acked Flush 0 Sent Done Done false
13072 4577 Acked Read 10 Sent Done Done false
13073 4578 Acked Write 10 Sent Done Done false
13074 4579 Acked Read 1 Sent Done Done false
13075 4580 Acked Read 1 Sent Done Done false
13076 4581 Acked Write 2 Sent Done Done false
13077 4582 Acked Write 10 Sent Done Done false
13078 4583 Acked Read 7 Sent Done Done false
13079 4584 Acked Write 10 Sent Done Done false
13080 4585 Acked Read 1 Sent Done Done false
13081 4586 Acked Write 6 Sent Done Done false
13082 4587 Acked Read 7 Sent Done Done false
13083 4588 Acked Read 5 Sent Done Done false
13084 4589 Acked Read 6 Sent Done Done false
13085 4590 Acked Write 6 Sent Done Done false
13086 4591 Acked Flush 0 Sent Done Done false
13087 4592 Acked Read 7 Sent Done Done false
13088 4593 Acked Write 3 Sent Done Done false
13089 4594 Acked Read 8 Sent Done Done false
13090 4595 Acked Write 9 Sent Done Done false
13091 4596 Acked Read 3 Sent Done Done false
13092 4597 Acked Write 10 Sent Done Done false
13093 4598 Acked Write 8 Sent Done Done false
13094 4599 Acked Read 8 Sent Done Done false
13095 4600 Acked Flush 0 Sent Done Done false
13096 4601 Acked Write 9 Sent Done Done false
13097 4602 Acked Read 9 Sent Done Done false
13098 4603 Acked Write 6 Sent Done Done false
13099 4604 Acked Write 10 Sent Done Done false
13100 4605 Acked Read 7 Sent Done Done false
13101 4606 Acked Read 4 Sent Done Done false
13102 4607 Acked Read 2 Sent Done Done false
13103 4608 Acked Read 2 Sent Done Done false
13104 4609 Acked Write 4 Sent Done Done false
13105 4610 Acked Read 8 Sent Done Done false
13106 4611 Acked Read 5 Sent Done Done false
13107 4612 Acked Read 1 Sent Done Done false
13108 4613 Acked Read 3 Sent Done Done false
13109 4614 Acked Read 4 Sent Done Done false
13110 4615 Acked Write 1 Sent Done Done false
13111 4616 Acked Read 2 Sent Done Done false
13112 4617 Acked Write 10 Sent Done Done false
13113 4618 Acked Read 6 Sent Done Done false
13114 4619 Acked Read 5 Sent Done Done false
13115 4620 Acked Read 10 Sent Done Done false
13116 4621 Acked Write 2 Sent Done Done false
13117 4622 Acked Write 1 Sent Done Done false
13118 4623 Acked Read 5 Sent Done Done false
13119 4624 Acked Read 1 Sent Done Done false
13120 4625 Acked Flush 0 Sent Done Done false
13121 STATES DS:0 DS:1 DS:2 TOTAL
13122 Sent 303 0 0 303
13123 Done 0 303 303 606
13124 Skipped 0 0 0 0
13125 Error 0 0 0 0
13126 Last Flush: 4322 4625 4625
13127 Downstairs last five completed: 4322 4321 4320 4319 4318
13128 Upstairs last five completed: 4625 4624 4623 4622 4621
13129 CLIENT: Up:0 ds:303 act:2
131302024-11-01T16:53:13.522ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131312024-11-01T16:53:13.523ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46162) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131322024-11-01T16:53:13.524ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131332024-11-01T16:53:13.524ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131342024-11-01T16:53:13.524ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131352024-11-01T16:53:13.524ZINFOcrucible: send last flush ID to this DS: 4322 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131362024-11-01T16:53:13.524ZINFOcrucible: Replied this last flush ID: 4322 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131372024-11-01T16:53:13.524ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131382024-11-01T16:53:13.524ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131392024-11-01T16:53:13.524ZINFOcrucible: [0] client re-new 303 jobs since flush 4322 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131402024-11-01T16:53:13.525ZINFOcrucible: [0] Replayed 303 jobs since flush: 4322 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
13141 ----------------------------------------------------------------
13142 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13143 Downstairs last five completed: 4625 4624 4623 4622 4621
13144 Upstairs last five completed: 4625 4624 4623 4622 4621
13145 CLIENT: Up:0 ds:0 act:3
13146 Replay: All jobs finished
13147 Main task has work to do, go find it
13148 got dsc Stop(0)
13149 stop 0
13150 [8810] Got stop action so:false kr:false
13151 Nov 01 16:53:15.825 INFO request completed, latency_us: 331, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 9b732c3e-307d-4b04-a90b-366b9628018a, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
131522024-11-01T16:53:15.825ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131532024-11-01T16:53:15.825ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131542024-11-01T16:53:15.825ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131552024-11-01T16:53:15.825ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
131562024-11-01T16:53:15.825ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
13157 Nov 01 16:53:15.825 INFO request completed, latency_us: 143, response_code: 200, uri: /state/cid/0, method: GET, req_id: 60dd322f-0ea5-40be-9543-77aabccc9d45, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
13158 [8810] Exited with: ExitStatus(unix_wait_status(9))
13159 [8810][0] reports Exit
13160 Nov 01 16:53:19.828 INFO request completed, latency_us: 294, response_code: 200, uri: /state/cid/0, method: GET, req_id: 412f61f2-11dc-49d1-a8b1-115fe1f58e63, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
13161 001/300 Read block 627 len 24576
13162 002/300 Read block 757 len 32768
13163 003/300 Read block 2012 len 36864
13164 004/300 Write block 1052 len 4096 data: 3
13165 005/300 Read block 3126 len 20480
13166 006/300 Read block 4891 len 40960
13167 007/300 Read block 3119 len 4096
13168 008/300 Read block 902 len 32768
13169 009/300 Read block 4406 len 32768
13170 010/300 Write block 4738 len 20480 data: 4 4 4 4 3
13171 011/300 Flush
13172 012/300 Write block 3816 len 32768 data: 5 6 6 6 5 4 3 4
13173 013/300 Read block 1317 len 24576
13174 014/300 Read block 3436 len 4096
13175 015/300 Read block 4961 len 40960
13176 016/300 Write block 608 len 28672 data: 3 3 3 3 3 3 2
13177 017/300 Read block 1545 len 36864
13178 018/300 Read block 4103 len 4096
13179 019/300 Read block 1818 len 16384
13180 020/300 Read block 2781 len 12288
13181 021/300 Write block 2898 len 12288 data: 2 2 2
13182 022/300 Read block 4825 len 12288
13183 023/300 Read block 691 len 12288
13184 024/300 Read block 998 len 16384
13185 025/300 Read block 1136 len 20480
13186 026/300 Read block 4570 len 12288
13187 027/300 Read block 256 len 28672
13188 028/300 Read block 2574 len 28672
13189 029/300 Read block 3342 len 12288
13190 030/300 Write block 2490 len 32768 data: 4 4 3 3 3 2 4 4
13191 031/300 Write block 3336 len 40960 data: 3 3 3 3 3 2 2 2 2 2
13192 032/300 Read block 4948 len 24576
13193 033/300 Read block 4318 len 12288
13194 034/300 Flush
13195 035/300 Write block 1720 len 40960 data: 5 5 5 5 3 4 3 3 2 2
13196 036/300 Write block 3978 len 40960 data: 4 4 3 4 4 3 3 3 2 2
13197 037/300 Read block 2616 len 24576
13198 038/300 Write block 3460 len 20480 data: 6 6 6 6 5
13199 039/300 Read block 2007 len 32768
13200 040/300 Write block 3360 len 28672 data: 3 3 3 3 4 4 4
13201 041/300 Write block 2373 len 24576 data: 5 4 3 3 3 2
13202 042/300 Read block 4054 len 28672
13203 043/300 Write block 3035 len 4096 data: 3
13204 044/300 Write block 209 len 8192 data: 4 3
13205 045/300 Write block 2268 len 28672 data: 3 3 3 3 3 3 4
13206 046/300 Write block 2870 len 36864 data: 4 5 3 4 3 4 4 4 4
13207 047/300 Read block 441 len 20480
13208 048/300 Write block 2889 len 32768 data: 4 4 3 2 2 2 2 2
13209 049/300 Read block 1647 len 12288
13210 050/300 Read block 3040 len 32768
13211 051/300 Read block 2909 len 4096
13212 052/300 Read block 2919 len 28672
13213 053/300 Read block 77 len 36864
13214 054/300 Read block 2910 len 8192
13215 055/300 Write block 380 len 40960 data: 4 2 2 2 3 3 3 2 2 2
13216 056/300 Read block 2991 len 8192
13217 057/300 Write block 3199 len 24576 data: 5 5 7 6 5 6
13218 058/300 Read block 4935 len 36864
13219 059/300 Read block 1181 len 8192
13220 060/300 Read block 149 len 4096
13221 061/300 Read block 808 len 40960
13222 062/300 Write block 2853 len 20480 data: 2 3 3 3 4
13223 063/300 Flush
13224 064/300 Read block 3727 len 12288
13225 065/300 Read block 1856 len 20480
13226 066/300 Read block 3278 len 20480
13227 067/300 Read block 2180 len 4096
13228 068/300 Write block 2394 len 20480 data: 3 3 4 3 3
13229 069/300 Write block 2564 len 20480 data: 4 5 4 4 4
13230 070/300 Write block 1032 len 4096 data: 2
13231 071/300 Write block 886 len 36864 data: 3 2 3 4 5 5 6 6 6
13232 072/300 Read block 2078 len 28672
13233 073/300 Write block 3510 len 24576 data: 4 5 5 4 4 4
13234 074/300 Read block 4988 len 8192
13235 075/300 Read block 2838 len 20480
13236 076/300 Write block 1068 len 32768 data: 3 3 3 3 3 3 3 3
13237 077/300 Write block 849 len 20480 data: 7 7 6 6 5
13238 078/300 Write block 127 len 4096 data: 5
13239 079/300 Write block 2481 len 28672 data: 3 3 3 3 3 3 4
13240 080/300 Write block 3541 len 24576 data: 3 4 4 4 5 4
13241 081/300 Read block 3179 len 4096
13242 082/300 Write block 4533 len 24576 data: 4 4 4 4 4 4
13243 083/300 Read block 4260 len 36864
13244 084/300 Read block 2 len 8192
13245 085/300 Read block 4203 len 20480
13246 086/300 Flush
13247 087/300 Read block 2836 len 16384
13248 088/300 Read block 1195 len 20480
13249 089/300 Read block 1676 len 12288
13250 090/300 Write block 3382 len 24576 data: 4 5 5 5 5 4
13251 091/300 Read block 1685 len 40960
13252 092/300 Read block 3411 len 40960
13253 093/300 Read block 4529 len 20480
13254 094/300 Read block 3198 len 8192
13255 095/300 Read block 3738 len 20480
13256 096/300 Read block 4512 len 4096
13257 097/300 Write block 4554 len 20480 data: 4 4 5 4 4
13258 098/300 Read block 2855 len 8192
13259 099/300 Write block 1001 len 24576 data: 5 4 3 3 3 2
13260 100/300 Write block 3485 len 16384 data: 3 4 4 4
13261 101/300 Write block 3294 len 28672 data: 3 4 4 4 6 6 8
13262 102/300 Write block 902 len 36864 data: 3 3 2 2 3 3 2 2 2
13263 103/300 Write block 4246 len 24576 data: 2 2 2 2 2 2
13264 104/300 Read block 1949 len 20480
13265 105/300 Read block 554 len 16384
13266 106/300 Read block 3061 len 16384
13267 107/300 Flush
13268 108/300 Write block 1507 len 40960 data: 5 5 5 5 5 6 5 4 4 3
13269 109/300 Read block 660 len 12288
13270 110/300 Read block 2080 len 40960
13271 111/300 Write block 4891 len 12288 data: 2 3 4
13272 112/300 Read block 3541 len 40960
13273 113/300 Write block 3457 len 32768 data: 3 4 5 7 7 7 7 6
13274 114/300 Flush
13275 115/300 Read block 3665 len 40960
13276 116/300 Read block 4736 len 20480
13277 117/300 Flush
13278 118/300 Write block 3031 len 16384 data: 3 3 3 3
13279 119/300 Read block 1512 len 8192
13280 120/300 Read block 946 len 12288
13281 121/300 Write block 3227 len 28672 data: 4 4 5 4 4 4 5
13282 122/300 Write block 3930 len 8192 data: 3 3
13283 123/300 Read block 3406 len 24576
13284 124/300 Read block 1598 len 24576
13285 125/300 Write block 4307 len 20480 data: 2 2 2 3 4
13286 126/300 Read block 130 len 8192
13287 127/300 Read block 2344 len 28672
13288 128/300 Write block 1603 len 16384 data: 3 2 3 3
13289 129/300 Read block 2463 len 8192
13290 130/300 Read block 1582 len 4096
13291 131/300 Write block 1761 len 12288 data: 4 4 4
13292 132/300 Read block 2235 len 20480
13293 133/300 Read block 3502 len 16384
13294 134/300 Read block 48 len 24576
13295 135/300 Read block 2007 len 4096
13296 136/300 Write block 1317 len 4096 data: 4
13297 137/300 Read block 1634 len 8192
13298 138/300 Write block 1393 len 20480 data: 3 3 3 3 3
13299 139/300 Write block 2050 len 32768 data: 2 3 3 3 3 3 3 3
13300 140/300 Write block 403 len 32768 data: 2 2 2 3 2 2 3 3
13301 141/300 Write block 967 len 28672 data: 2 3 3 4 3 3 3
13302 142/300 Read block 1542 len 32768
13303 143/300 Write block 2498 len 32768 data: 3 3 2 2 2 2 2 2
13304 144/300 Write block 3250 len 12288 data: 4 4 3
13305 145/300 Write block 4890 len 16384 data: 2 3 4 5
13306 146/300 Read block 3834 len 32768
13307 147/300 Flush
13308 148/300 Read block 2680 len 40960
13309 149/300 Write block 2177 len 4096 data: 3
13310 150/300 Write block 15 len 20480 data: 2 2 2 2 2
13311 151/300 Read block 4891 len 32768
13312 152/300 Read block 4172 len 24576
13313 153/300 Write block 3257 len 36864 data: 4 3 4 4 3 3 3 3 2
13314 154/300 Flush
13315 155/300 Write block 2944 len 20480 data: 2 2 2 2 2
13316 156/300 Write block 3538 len 4096 data: 3
13317 157/300 Write block 4992 len 24576 data: 3 3 3 4 4 5
13318 158/300 Write block 4247 len 12288 data: 3 3 3
13319 159/300 Read block 2163 len 24576
13320 160/300 Write block 2741 len 20480 data: 8 7 7 6 6
13321 161/300 Read block 1411 len 4096
13322 162/300 Write block 2556 len 28672 data: 5 3 3 3 3 4 4
13323 163/300 Read block 4919 len 12288
13324 164/300 Write block 2950 len 28672 data: 2 2 3 4 5 5 7
13325 165/300 Read block 3435 len 36864
13326 166/300 Write block 1669 len 12288 data: 4 4 3
13327 167/300 Write block 4009 len 20480 data: 6 5 5 3 3
13328 168/300 Write block 767 len 16384 data: 4 4 3 2
13329 169/300 Write block 1945 len 36864 data: 5 5 4 5 5 4 3 3 3
13330 170/300 Read block 1892 len 36864
13331 171/300 Read block 1996 len 24576
13332 172/300 Read block 2008 len 4096
13333 173/300 Write block 2832 len 20480 data: 4 4 4 4 3
13334 174/300 Read block 2997 len 16384
13335 175/300 Write block 4335 len 16384 data: 2 2 2 2
13336 176/300 Write block 128 len 12288 data: 3 3 3
13337 177/300 Read block 645 len 24576
13338 178/300 Write block 2144 len 40960 data: 2 2 2 2 2 2 2 2 2 2
13339 179/300 Read block 2416 len 24576
13340 180/300 Read block 1864 len 20480
13341 181/300 Write block 4185 len 32768 data: 4 3 3 2 2 2 2 2
13342 182/300 Write block 4942 len 4096 data: 7
13343 183/300 Flush
13344 184/300 Flush
13345 185/300 Read block 4267 len 12288
13346 186/300 Read block 1072 len 16384
13347 187/300 Write block 2522 len 28672 data: 3 3 3 3 2 2 2
13348 188/300 Flush
13349 189/300 Read block 4075 len 36864
13350 190/300 Read block 2611 len 8192
13351 191/300 Read block 1266 len 12288
13352 192/300 Read block 3554 len 24576
13353 193/300 Write block 1595 len 28672 data: 3 3 4 3 3 3 3
13354 194/300 Write block 2995 len 4096 data: 3
13355 195/300 Read block 4694 len 20480
13356 196/300 Read block 213 len 36864
13357 197/300 Write block 2167 len 28672 data: 2 2 2 3 2 3 3
13358 198/300 Read block 4575 len 16384
13359 199/300 Read block 693 len 8192
13360 200/300 Read block 620 len 8192
13361 201/300 Read block 3352 len 32768
13362 202/300 Flush
13363 203/300 Write block 621 len 28672 data: 4 6 5 6 6 4 4
13364 204/300 Write block 2222 len 28672 data: 4 4 4 4 4 4 4
13365 205/300 Write block 3443 len 40960 data: 4 5 5 5 5 6 6 6 5 4
13366 206/300 Write block 503 len 8192 data: 5 5
13367 207/300 Read block 2089 len 28672
13368 208/300 Write block 2557 len 32768 data: 4 4 4 4 5 5 4 5
13369 209/300 Read block 63 len 20480
13370 210/300 Write block 999 len 32768 data: 6 6 6 5 4 4 4 3
13371 211/300 Read block 202 len 20480
13372 212/300 Flush
13373 213/300 Read block 2056 len 36864
13374 214/300 Write block 4301 len 16384 data: 4 4 4 3
13375 215/300 Read block 451 len 20480
13376 216/300 Read block 3143 len 12288
13377 217/300 Read block 3751 len 12288
13378 218/300 Flush
13379 219/300 Write block 4899 len 32768 data: 3 4 4 4 5 5 5 5
13380 220/300 Write block 1251 len 20480 data: 3 2 2 3 3
13381 221/300 Write block 2564 len 32768 data: 6 6 5 5 5 3 3 3
13382 222/300 Read block 2035 len 8192
13383 223/300 Write block 3105 len 32768 data: 3 3 3 3 5 4 4 4
13384 224/300 Write block 2233 len 4096 data: 5
13385 225/300 Read block 2906 len 24576
13386 226/300 Read block 1664 len 20480
13387 227/300 Read block 4574 len 36864
13388 228/300 Read block 4724 len 36864
13389 229/300 Read block 1861 len 24576
13390 230/300 Write block 4962 len 24576 data: 3 3 3 4 3 3
13391 231/300 Write block 2922 len 8192 data: 3 3
13392 232/300 Read block 1245 len 36864
13393 233/300 Read block 3789 len 24576
13394 234/300 Write block 1819 len 16384 data: 3 3 3 3
13395 235/300 Read block 947 len 36864
13396 236/300 Read block 1700 len 12288
13397 237/300 Read block 3945 len 32768
13398 238/300 Read block 1165 len 32768
13399 239/300 Write block 1413 len 20480 data: 3 3 3 3 3
13400 240/300 Read block 370 len 24576
13401 241/300 Read block 1614 len 12288
13402 242/300 Read block 4434 len 32768
13403 243/300 Write block 4108 len 32768 data: 6 5 6 6 5 4 4 4
13404 244/300 Write block 2563 len 36864 data: 5 7 7 6 6 6 4 4 4
13405 245/300 Read block 1949 len 36864
13406 246/300 Read block 3094 len 32768
13407 247/300 Write block 4530 len 40960 data: 4 4 4 5 5 5 5 5 5 4
13408 248/300 Write block 4429 len 4096 data: 6
13409 249/300 Read block 1871 len 32768
13410 250/300 Write block 138 len 8192 data: 2 2
13411 251/300 Read block 1183 len 36864
13412 252/300 Read block 3577 len 28672
13413 253/300 Read block 4924 len 40960
13414 254/300 Read block 4222 len 16384
13415 255/300 Write block 2807 len 28672 data: 3 3 3 4 5 5 6
13416 256/300 Read block 1438 len 4096
13417 257/300 Flush
13418 258/300 Flush
13419 259/300 Read block 4887 len 24576
13420 260/300 Write block 2388 len 24576 data: 3 3 4 4 3 3
13421 261/300 Read block 1197 len 8192
13422 262/300 Read block 2962 len 24576
13423 263/300 Read block 4803 len 12288
13424 264/300 Write block 2764 len 12288 data: 3 4 4
13425 265/300 Read block 1572 len 8192
13426 266/300 Read block 2995 len 4096
13427 267/300 Read block 205 len 28672
13428 268/300 Write block 1890 len 4096 data: 2
13429 269/300 Read block 2659 len 36864
13430 270/300 Read block 1318 len 16384
13431 271/300 Read block 3748 len 28672
13432 272/300 Write block 2683 len 16384 data: 5 4 3 3
13433 273/300 Write block 3093 len 8192 data: 4 5
13434 274/300 Write block 2922 len 12288 data: 4 4 3
13435 275/300 Read block 1705 len 20480
13436 276/300 Flush
13437 277/300 Read block 3442 len 4096
13438 278/300 Read block 4212 len 36864
13439 279/300 Read block 3410 len 4096
13440 280/300 Read block 333 len 24576
13441 281/300 Write block 3907 len 16384 data: 4 4 3 2
13442 282/300 Read block 4161 len 4096
13443 283/300 Flush
13444 284/300 Write block 4305 len 40960 data: 2 2 3 3 3 4 5 5 5 6
13445 285/300 Read block 2351 len 20480
13446 286/300 Read block 3604 len 24576
13447 287/300 Read block 1337 len 20480
13448 288/300 Read block 2744 len 32768
13449 289/300 Read block 3508 len 28672
13450 290/300 Read block 4493 len 8192
13451 291/300 Write block 4661 len 28672 data: 4 4 4 5 3 3 3
13452 292/300 Write block 1348 len 8192 data: 4 4
13453 293/300 Read block 3404 len 12288
13454 294/300 Flush
13455 295/300 Flush
13456 296/300 Write block 2356 len 20480 data: 4 4 4 4 4
13457 297/300 Read block 619 len 8192
13458 298/300 Write block 2674 len 24576 data: 4 4 4 5 4 5
13459 299/300 Write block 3522 len 36864 data: 2 3 2 2 2 3 4 4 3
13460 300/300 Write block 433 len 20480 data: 2 2 2 4 3
13461 301/300 Read block 1843 len 24576
13462 Main task has work to do, go find it
13463 got dsc Start(0)
13464 start 0
13465 Nov 01 16:53:20.110 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/0, method: POST, req_id: 686835bb-9d72-4511-85a8-7c3831dedb79, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
13466 Replay: started 0, returned:Ok(())
13467 ----------------------------------------------------------------
13468 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13469 States: Offline Active Active
13470 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13471 4626 Acked Read 6 Sent Done Done false
13472 4627 Acked Read 8 Sent Done Done false
13473 4628 Acked Read 9 Sent Done Done false
13474 4629 Acked Write 1 Sent Done Done false
13475 4630 Acked Read 5 Sent Done Done false
13476 4631 Acked Read 10 Sent Done Done false
13477 4632 Acked Read 1 Sent Done Done false
13478 4633 Acked Read 8 Sent Done Done false
13479 4634 Acked Read 8 Sent Done Done false
13480 4635 Acked Write 5 Sent Done Done false
13481 4636 Acked Flush 0 Sent Done Done false
13482 4637 Acked Write 8 Sent Done Done false
13483 4638 Acked Read 6 Sent Done Done false
13484 4639 Acked Read 1 Sent Done Done false
13485 4640 Acked Read 10 Sent Done Done false
13486 4641 Acked Write 7 Sent Done Done false
13487 4642 Acked Read 9 Sent Done Done false
13488 4643 Acked Read 1 Sent Done Done false
13489 4644 Acked Read 4 Sent Done Done false
13490 4645 Acked Read 3 Sent Done Done false
13491 4646 Acked Write 3 Sent Done Done false
13492 4647 Acked Read 3 Sent Done Done false
13493 4648 Acked Read 3 Sent Done Done false
13494 4649 Acked Read 4 Sent Done Done false
13495 4650 Acked Read 5 Sent Done Done false
13496 4651 Acked Read 3 Sent Done Done false
13497 4652 Acked Read 7 Sent Done Done false
13498 4653 Acked Read 7 Sent Done Done false
13499 4654 Acked Read 3 Sent Done Done false
13500 4655 Acked Write 8 Sent Done Done false
13501 4656 Acked Write 10 Sent Done Done false
13502 4657 Acked Read 6 Sent Done Done false
13503 4658 Acked Read 3 Sent Done Done false
13504 4659 Acked Flush 0 Sent Done Done false
13505 4660 Acked Write 10 Sent Done Done false
13506 4661 Acked Write 10 Sent Done Done false
13507 4662 Acked Read 6 Sent Done Done false
13508 4663 Acked Write 5 Sent Done Done false
13509 4664 Acked Read 8 Sent Done Done false
13510 4665 Acked Write 7 Sent Done Done false
13511 4666 Acked Write 6 Sent Done Done false
13512 4667 Acked Read 7 Sent Done Done false
13513 4668 Acked Write 1 Sent Done Done false
13514 4669 Acked Write 2 Sent Done Done false
13515 4670 Acked Write 7 Sent Done Done false
13516 4671 Acked Write 9 Sent Done Done false
13517 4672 Acked Read 5 Sent Done Done false
13518 4673 Acked Write 8 Sent Done Done false
13519 4674 Acked Read 3 Sent Done Done false
13520 4675 Acked Read 8 Sent Done Done false
13521 4676 Acked Read 1 Sent Done Done false
13522 4677 Acked Read 7 Sent Done Done false
13523 4678 Acked Read 9 Sent Done Done false
13524 4679 Acked Read 2 Sent Done Done false
13525 4680 Acked Write 10 Sent Done Done false
13526 4681 Acked Read 2 Sent Done Done false
13527 4682 Acked Write 6 Sent Done Done false
13528 4683 Acked Read 9 Sent Done Done false
13529 4684 Acked Read 2 Sent Done Done false
13530 4685 Acked Read 1 Sent Done Done false
13531 4686 Acked Read 10 Sent Done Done false
13532 4687 Acked Write 5 Sent Done Done false
13533 4688 Acked Flush 0 Sent Done Done false
13534 4689 Acked Read 3 Sent Done Done false
13535 4690 Acked Read 5 Sent Done Done false
13536 4691 Acked Read 5 Sent Done Done false
13537 4692 Acked Read 1 Sent Done Done false
13538 4693 Acked Write 5 Sent Done Done false
13539 4694 Acked Write 5 Sent Done Done false
13540 4695 Acked Write 1 Sent Done Done false
13541 4696 Acked Write 9 Sent Done Done false
13542 4697 Acked Read 7 Sent Done Done false
13543 4698 Acked Write 6 Sent Done Done false
13544 4699 Acked Read 2 Sent Done Done false
13545 4700 Acked Read 5 Sent Done Done false
13546 4701 Acked Write 8 Sent Done Done false
13547 4702 Acked Write 5 Sent Done Done false
13548 4703 Acked Write 1 Sent Done Done false
13549 4704 Acked Write 7 Sent Done Done false
13550 4705 Acked Write 6 Sent Done Done false
13551 4706 Acked Read 1 Sent Done Done false
13552 4707 Acked Write 6 Sent Done Done false
13553 4708 Acked Read 9 Sent Done Done false
13554 4709 Acked Read 2 Sent Done Done false
13555 4710 Acked Read 5 Sent Done Done false
13556 4711 Acked Flush 0 Sent Done Done false
13557 4712 Acked Read 4 Sent Done Done false
13558 4713 Acked Read 5 Sent Done Done false
13559 4714 Acked Read 3 Sent Done Done false
13560 4715 Acked Write 6 Sent Done Done false
13561 4716 Acked Read 10 Sent Done Done false
13562 4717 Acked Read 10 Sent Done Done false
13563 4718 Acked Read 5 Sent Done Done false
13564 4719 Acked Read 2 Sent Done Done false
13565 4720 Acked Read 5 Sent Done Done false
13566 4721 Acked Read 1 Sent Done Done false
13567 4722 Acked Write 5 Sent Done Done false
13568 4723 Acked Read 2 Sent Done Done false
13569 4724 Acked Write 6 Sent Done Done false
13570 4725 Acked Write 4 Sent Done Done false
13571 4726 Acked Write 7 Sent Done Done false
13572 4727 Acked Write 9 Sent Done Done false
13573 4728 Acked Write 6 Sent Done Done false
13574 4729 Acked Read 5 Sent Done Done false
13575 4730 Acked Read 4 Sent Done Done false
13576 4731 Acked Read 4 Sent Done Done false
13577 4732 Acked Flush 0 Sent Done Done false
13578 4733 Acked Write 10 Sent Done Done false
13579 4734 Acked Read 3 Sent Done Done false
13580 4735 Acked Read 10 Sent Done Done false
13581 4736 Acked Write 3 Sent Done Done false
13582 4737 Acked Read 10 Sent Done Done false
13583 4738 Acked Write 8 Sent Done Done false
13584 4739 Acked Flush 0 Sent Done Done false
13585 4740 Acked Read 10 Sent Done Done false
13586 4741 Acked Read 5 Sent Done Done false
13587 4742 Acked Flush 0 Sent Done Done false
13588 4743 Acked Write 4 Sent Done Done false
13589 4744 Acked Read 2 Sent Done Done false
13590 4745 Acked Read 3 Sent Done Done false
13591 4746 Acked Write 7 Sent Done Done false
13592 4747 Acked Write 2 Sent Done Done false
13593 4748 Acked Read 6 Sent Done Done false
13594 4749 Acked Read 6 Sent Done Done false
13595 4750 Acked Write 5 Sent Done Done false
13596 4751 Acked Read 2 Sent Done Done false
13597 4752 Acked Read 7 Sent Done Done false
13598 4753 Acked Write 4 Sent Done Done false
13599 4754 Acked Read 2 Sent Done Done false
13600 4755 Acked Read 1 Sent Done Done false
13601 4756 Acked Write 3 Sent Done Done false
13602 4757 Acked Read 5 Sent Done Done false
13603 4758 Acked Read 4 Sent Done Done false
13604 4759 Acked Read 6 Sent Done Done false
13605 4760 Acked Read 1 Sent Done Done false
13606 4761 Acked Write 1 Sent Done Done false
13607 4762 Acked Read 2 Sent Done Done false
13608 4763 Acked Write 5 Sent Done Done false
13609 4764 Acked Write 8 Sent Done Done false
13610 4765 Acked Write 8 Sent Done Done false
13611 4766 Acked Write 7 Sent Done Done false
13612 4767 Acked Read 8 Sent Done Done false
13613 4768 Acked Write 8 Sent Done Done false
13614 4769 Acked Write 3 Sent Done Done false
13615 4770 Acked Write 4 Sent Done Done false
13616 4771 Acked Read 8 Sent Done Done false
13617 4772 Acked Flush 0 Sent Done Done false
13618 4773 Acked Read 10 Sent Done Done false
13619 4774 Acked Write 1 Sent Done Done false
13620 4775 Acked Write 5 Sent Done Done false
13621 4776 Acked Read 8 Sent Done Done false
13622 4777 Acked Read 6 Sent Done Done false
13623 4778 Acked Write 9 Sent Done Done false
13624 4779 Acked Flush 0 Sent Done Done false
13625 4780 Acked Write 5 Sent Done Done false
13626 4781 Acked Write 1 Sent Done Done false
13627 4782 Acked Write 6 Sent Done Done false
13628 4783 Acked Write 3 Sent Done Done false
13629 4784 Acked Read 6 Sent Done Done false
13630 4785 Acked Write 5 Sent Done Done false
13631 4786 Acked Read 1 Sent Done Done false
13632 4787 Acked Write 7 Sent Done Done false
13633 4788 Acked Read 3 Sent Done Done false
13634 4789 Acked Write 7 Sent Done Done false
13635 4790 Acked Read 9 Sent Done Done false
13636 4791 Acked Write 3 Sent Done Done false
13637 4792 Acked Write 5 Sent Done Done false
13638 4793 Acked Write 4 Sent Done Done false
13639 4794 Acked Write 9 Sent Done Done false
13640 4795 Acked Read 9 Sent Done Done false
13641 4796 Acked Read 6 Sent Done Done false
13642 4797 Acked Read 1 Sent Done Done false
13643 4798 Acked Write 5 Sent Done Done false
13644 4799 Acked Read 4 Sent Done Done false
13645 4800 Acked Write 4 Sent Done Done false
13646 4801 Acked Write 3 Sent Done Done false
13647 4802 Acked Read 6 Sent Done Done false
13648 4803 Acked Write 10 Sent Done Done false
13649 4804 Acked Read 6 Sent Done Done false
13650 4805 Acked Read 5 Sent Done Done false
13651 4806 Acked Write 8 Sent Done Done false
13652 4807 Acked Write 1 Sent Done Done false
13653 4808 Acked Flush 0 Sent Done Done false
13654 4809 Acked Flush 0 Sent Done Done false
13655 4810 Acked Read 3 Sent Done Done false
13656 4811 Acked Read 4 Sent Done Done false
13657 4812 Acked Write 7 Sent Done Done false
13658 4813 Acked Flush 0 Sent Done Done false
13659 4814 Acked Read 9 Sent Done Done false
13660 4815 Acked Read 2 Sent Done Done false
13661 4816 Acked Read 3 Sent Done Done false
13662 4817 Acked Read 6 Sent Done Done false
13663 4818 Acked Write 7 Sent Done Done false
13664 4819 Acked Write 1 Sent Done Done false
13665 4820 Acked Read 5 Sent Done Done false
13666 4821 Acked Read 9 Sent Done Done false
13667 4822 Acked Write 7 Sent Done Done false
13668 4823 Acked Read 4 Sent Done Done false
13669 4824 Acked Read 2 Sent Done Done false
13670 4825 Acked Read 2 Sent Done Done false
13671 4826 Acked Read 8 Sent Done Done false
13672 4827 Acked Flush 0 Sent Done Done false
13673 4828 Acked Write 7 Sent Done Done false
13674 4829 Acked Write 7 Sent Done Done false
13675 4830 Acked Write 10 Sent Done Done false
13676 4831 Acked Write 2 Sent Done Done false
13677 4832 Acked Read 7 Sent Done Done false
13678 4833 Acked Write 8 Sent Done Done false
13679 4834 Acked Read 5 Sent Done Done false
13680 4835 Acked Write 8 Sent Done Done false
13681 4836 Acked Read 5 Sent Done Done false
13682 4837 Acked Flush 0 Sent Done Done false
13683 4838 Acked Read 9 Sent Done Done false
13684 4839 Acked Write 4 Sent Done Done false
13685 4840 Acked Read 5 Sent Done Done false
13686 4841 Acked Read 3 Sent Done Done false
13687 4842 Acked Read 3 Sent Done Done false
13688 4843 Acked Flush 0 Sent Done Done false
13689 4844 Acked Write 8 Sent Done Done false
13690 4845 Acked Write 5 Sent Done Done false
13691 4846 Acked Write 8 Sent Done Done false
13692 4847 Acked Read 2 Sent Done Done false
13693 4848 Acked Write 8 Sent Done Done false
13694 4849 Acked Write 1 Sent Done Done false
13695 4850 Acked Read 6 Sent Done Done false
13696 4851 Acked Read 5 Sent Done Done false
13697 4852 Acked Read 9 Sent Done Done false
13698 4853 Acked Read 9 Sent Done Done false
13699 4854 Acked Read 6 Sent Done Done false
13700 4855 Acked Write 6 Sent Done Done false
13701 4856 Acked Write 2 Sent Done Done false
13702 4857 Acked Read 9 Sent Done Done false
13703 4858 Acked Read 6 Sent Done Done false
13704 4859 Acked Write 4 Sent Done Done false
13705 4860 Acked Read 9 Sent Done Done false
13706 4861 Acked Read 3 Sent Done Done false
13707 4862 Acked Read 8 Sent Done Done false
13708 4863 Acked Read 8 Sent Done Done false
13709 4864 Acked Write 5 Sent Done Done false
13710 4865 Acked Read 6 Sent Done Done false
13711 4866 Acked Read 3 Sent Done Done false
13712 4867 Acked Read 8 Sent Done Done false
13713 4868 Acked Write 8 Sent Done Done false
13714 4869 Acked Write 9 Sent Done Done false
13715 4870 Acked Read 9 Sent Done Done false
13716 4871 Acked Read 8 Sent Done Done false
13717 4872 Acked Write 10 Sent Done Done false
13718 4873 Acked Write 1 Sent Done Done false
13719 4874 Acked Read 8 Sent Done Done false
13720 4875 Acked Write 2 Sent Done Done false
13721 4876 Acked Read 9 Sent Done Done false
13722 4877 Acked Read 7 Sent Done Done false
13723 4878 Acked Read 10 Sent Done Done false
13724 4879 Acked Read 4 Sent Done Done false
13725 4880 Acked Write 7 Sent Done Done false
13726 4881 Acked Read 1 Sent Done Done false
13727 4882 Acked Flush 0 Sent Done Done false
13728 4883 Acked Flush 0 Sent Done Done false
13729 4884 Acked Read 6 Sent Done Done false
13730 4885 Acked Write 6 Sent Done Done false
13731 4886 Acked Read 2 Sent Done Done false
13732 4887 Acked Read 6 Sent Done Done false
13733 4888 Acked Read 3 Sent Done Done false
13734 4889 Acked Write 3 Sent Done Done false
13735 4890 Acked Read 2 Sent Done Done false
13736 4891 Acked Read 1 Sent Done Done false
13737 4892 Acked Read 7 Sent Done Done false
13738 4893 Acked Write 1 Sent Done Done false
13739 4894 Acked Read 9 Sent Done Done false
13740 4895 Acked Read 4 Sent Done Done false
13741 4896 Acked Read 7 Sent Done Done false
13742 4897 Acked Write 4 Sent Done Done false
13743 4898 Acked Write 2 Sent Done Done false
13744 4899 Acked Write 3 Sent Done Done false
13745 4900 Acked Read 5 Sent Done Done false
13746 4901 Acked Flush 0 Sent Done Done false
13747 4902 Acked Read 1 Sent Done Done false
13748 4903 Acked Read 9 Sent Done Done false
13749 4904 Acked Read 1 Sent Done Done false
13750 4905 Acked Read 6 Sent Done Done false
13751 4906 Acked Write 4 Sent Done Done false
13752 4907 Acked Read 1 Sent Done Done false
13753 4908 Acked Flush 0 Sent Done Done false
13754 4909 Acked Write 10 Sent Done Done false
13755 4910 Acked Read 5 Sent Done Done false
13756 4911 Acked Read 6 Sent Done Done false
13757 4912 Acked Read 5 Sent Done Done false
13758 4913 Acked Read 8 Sent Done Done false
13759 4914 Acked Read 7 Sent Done Done false
13760 4915 Acked Read 2 Sent Done Done false
13761 4916 Acked Write 7 Sent Done Done false
13762 4917 Acked Write 2 Sent Done Done false
13763 4918 Acked Read 3 Sent Done Done false
13764 4919 Acked Flush 0 Sent Done Done false
13765 4920 Acked Flush 0 Sent Done Done false
13766 4921 Acked Write 5 Sent Done Done false
13767 4922 Acked Read 2 Sent Done Done false
13768 4923 Acked Write 6 Sent Done Done false
13769 4924 Acked Write 9 Sent Done Done false
13770 4925 Acked Write 5 Sent Done Done false
13771 4926 Acked Read 6 Sent Done Done false
13772 STATES DS:0 DS:1 DS:2 TOTAL
13773 Sent 301 0 0 301
13774 Done 0 301 301 602
13775 Skipped 0 0 0 0
13776 Error 0 0 0 0
13777 Last Flush: 4625 4920 4920
13778 Downstairs last five completed: 4625 4624 4623 4622 4621
13779 Upstairs last five completed: 4926 4925 4924 4923 4922
13780 CLIENT: Up:0 ds:301 act:2
13781 ----------------------------------------------------------------
13782 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
13783 States: Offline Active Active
13784 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13785 4626 Acked Read 6 Sent Done Done false
13786 4627 Acked Read 8 Sent Done Done false
13787 4628 Acked Read 9 Sent Done Done false
13788 4629 Acked Write 1 Sent Done Done false
13789 4630 Acked Read 5 Sent Done Done false
13790 4631 Acked Read 10 Sent Done Done false
13791 4632 Acked Read 1 Sent Done Done false
13792 4633 Acked Read 8 Sent Done Done false
13793 4634 Acked Read 8 Sent Done Done false
13794 4635 Acked Write 5 Sent Done Done false
13795 4636 Acked Flush 0 Sent Done Done false
13796 4637 Acked Write 8 Sent Done Done false
13797 4638 Acked Read 6 Sent Done Done false
13798 4639 Acked Read 1 Sent Done Done false
13799 4640 Acked Read 10 Sent Done Done false
13800 4641 Acked Write 7 Sent Done Done false
13801 4642 Acked Read 9 Sent Done Done false
13802 4643 Acked Read 1 Sent Done Done false
13803 4644 Acked Read 4 Sent Done Done false
13804 4645 Acked Read 3 Sent Done Done false
13805 4646 Acked Write 3 Sent Done Done false
13806 4647 Acked Read 3 Sent Done Done false
13807 4648 Acked Read 3 Sent Done Done false
13808 4649 Acked Read 4 Sent Done Done false
13809 4650 Acked Read 5 Sent Done Done false
13810 4651 Acked Read 3 Sent Done Done false
13811 4652 Acked Read 7 Sent Done Done false
13812 4653 Acked Read 7 Sent Done Done false
13813 4654 Acked Read 3 Sent Done Done false
13814 4655 Acked Write 8 Sent Done Done false
13815 4656 Acked Write 10 Sent Done Done false
13816 4657 Acked Read 6 Sent Done Done false
13817 4658 Acked Read 3 Sent Done Done false
13818 4659 Acked Flush 0 Sent Done Done false
13819 4660 Acked Write 10 Sent Done Done false
13820 4661 Acked Write 10 Sent Done Done false
13821 4662 Acked Read 6 Sent Done Done false
13822 4663 Acked Write 5 Sent Done Done false
13823 4664 Acked Read 8 Sent Done Done false
13824 4665 Acked Write 7 Sent Done Done false
13825 4666 Acked Write 6 Sent Done Done false
13826 4667 Acked Read 7 Sent Done Done false
13827 4668 Acked Write 1 Sent Done Done false
13828 4669 Acked Write 2 Sent Done Done false
13829 4670 Acked Write 7 Sent Done Done false
13830 4671 Acked Write 9 Sent Done Done false
13831 4672 Acked Read 5 Sent Done Done false
13832 4673 Acked Write 8 Sent Done Done false
13833 4674 Acked Read 3 Sent Done Done false
13834 4675 Acked Read 8 Sent Done Done false
13835 4676 Acked Read 1 Sent Done Done false
13836 4677 Acked Read 7 Sent Done Done false
13837 4678 Acked Read 9 Sent Done Done false
13838 4679 Acked Read 2 Sent Done Done false
13839 4680 Acked Write 10 Sent Done Done false
13840 4681 Acked Read 2 Sent Done Done false
13841 4682 Acked Write 6 Sent Done Done false
13842 4683 Acked Read 9 Sent Done Done false
13843 4684 Acked Read 2 Sent Done Done false
13844 4685 Acked Read 1 Sent Done Done false
13845 4686 Acked Read 10 Sent Done Done false
13846 4687 Acked Write 5 Sent Done Done false
13847 4688 Acked Flush 0 Sent Done Done false
13848 4689 Acked Read 3 Sent Done Done false
13849 4690 Acked Read 5 Sent Done Done false
13850 4691 Acked Read 5 Sent Done Done false
13851 4692 Acked Read 1 Sent Done Done false
13852 4693 Acked Write 5 Sent Done Done false
13853 4694 Acked Write 5 Sent Done Done false
13854 4695 Acked Write 1 Sent Done Done false
13855 4696 Acked Write 9 Sent Done Done false
13856 4697 Acked Read 7 Sent Done Done false
13857 4698 Acked Write 6 Sent Done Done false
13858 4699 Acked Read 2 Sent Done Done false
13859 4700 Acked Read 5 Sent Done Done false
13860 4701 Acked Write 8 Sent Done Done false
13861 4702 Acked Write 5 Sent Done Done false
13862 4703 Acked Write 1 Sent Done Done false
13863 4704 Acked Write 7 Sent Done Done false
13864 4705 Acked Write 6 Sent Done Done false
13865 4706 Acked Read 1 Sent Done Done false
13866 4707 Acked Write 6 Sent Done Done false
13867 4708 Acked Read 9 Sent Done Done false
13868 4709 Acked Read 2 Sent Done Done false
13869 4710 Acked Read 5 Sent Done Done false
13870 4711 Acked Flush 0 Sent Done Done false
13871 4712 Acked Read 4 Sent Done Done false
13872 4713 Acked Read 5 Sent Done Done false
13873 4714 Acked Read 3 Sent Done Done false
13874 4715 Acked Write 6 Sent Done Done false
13875 4716 Acked Read 10 Sent Done Done false
13876 4717 Acked Read 10 Sent Done Done false
13877 4718 Acked Read 5 Sent Done Done false
13878 4719 Acked Read 2 Sent Done Done false
13879 4720 Acked Read 5 Sent Done Done false
13880 4721 Acked Read 1 Sent Done Done false
13881 4722 Acked Write 5 Sent Done Done false
13882 4723 Acked Read 2 Sent Done Done false
13883 4724 Acked Write 6 Sent Done Done false
13884 4725 Acked Write 4 Sent Done Done false
13885 4726 Acked Write 7 Sent Done Done false
13886 4727 Acked Write 9 Sent Done Done false
13887 4728 Acked Write 6 Sent Done Done false
13888 4729 Acked Read 5 Sent Done Done false
13889 4730 Acked Read 4 Sent Done Done false
13890 4731 Acked Read 4 Sent Done Done false
13891 4732 Acked Flush 0 Sent Done Done false
13892 4733 Acked Write 10 Sent Done Done false
13893 4734 Acked Read 3 Sent Done Done false
13894 4735 Acked Read 10 Sent Done Done false
13895 4736 Acked Write 3 Sent Done Done false
13896 4737 Acked Read 10 Sent Done Done false
13897 4738 Acked Write 8 Sent Done Done false
13898 4739 Acked Flush 0 Sent Done Done false
13899 4740 Acked Read 10 Sent Done Done false
13900 4741 Acked Read 5 Sent Done Done false
13901 4742 Acked Flush 0 Sent Done Done false
13902 4743 Acked Write 4 Sent Done Done false
13903 4744 Acked Read 2 Sent Done Done false
13904 4745 Acked Read 3 Sent Done Done false
13905 4746 Acked Write 7 Sent Done Done false
13906 4747 Acked Write 2 Sent Done Done false
13907 4748 Acked Read 6 Sent Done Done false
13908 4749 Acked Read 6 Sent Done Done false
13909 4750 Acked Write 5 Sent Done Done false
13910 4751 Acked Read 2 Sent Done Done false
13911 4752 Acked Read 7 Sent Done Done false
13912 4753 Acked Write 4 Sent Done Done false
13913 4754 Acked Read 2 Sent Done Done false
13914 4755 Acked Read 1 Sent Done Done false
13915 4756 Acked Write 3 Sent Done Done false
13916 4757 Acked Read 5 Sent Done Done false
13917 4758 Acked Read 4 Sent Done Done false
13918 4759 Acked Read 6 Sent Done Done false
13919 4760 Acked Read 1 Sent Done Done false
13920 4761 Acked Write 1 Sent Done Done false
13921 4762 Acked Read 2 Sent Done Done false
13922 4763 Acked Write 5 Sent Done Done false
13923 4764 Acked Write 8 Sent Done Done false
13924 4765 Acked Write 8 Sent Done Done false
13925 4766 Acked Write 7 Sent Done Done false
13926 4767 Acked Read 8 Sent Done Done false
13927 4768 Acked Write 8 Sent Done Done false
13928 4769 Acked Write 3 Sent Done Done false
13929 4770 Acked Write 4 Sent Done Done false
13930 4771 Acked Read 8 Sent Done Done false
13931 4772 Acked Flush 0 Sent Done Done false
13932 4773 Acked Read 10 Sent Done Done false
13933 4774 Acked Write 1 Sent Done Done false
13934 4775 Acked Write 5 Sent Done Done false
13935 4776 Acked Read 8 Sent Done Done false
13936 4777 Acked Read 6 Sent Done Done false
13937 4778 Acked Write 9 Sent Done Done false
13938 4779 Acked Flush 0 Sent Done Done false
13939 4780 Acked Write 5 Sent Done Done false
13940 4781 Acked Write 1 Sent Done Done false
13941 4782 Acked Write 6 Sent Done Done false
13942 4783 Acked Write 3 Sent Done Done false
13943 4784 Acked Read 6 Sent Done Done false
13944 4785 Acked Write 5 Sent Done Done false
13945 4786 Acked Read 1 Sent Done Done false
13946 4787 Acked Write 7 Sent Done Done false
13947 4788 Acked Read 3 Sent Done Done false
13948 4789 Acked Write 7 Sent Done Done false
13949 4790 Acked Read 9 Sent Done Done false
13950 4791 Acked Write 3 Sent Done Done false
13951 4792 Acked Write 5 Sent Done Done false
13952 4793 Acked Write 4 Sent Done Done false
13953 4794 Acked Write 9 Sent Done Done false
13954 4795 Acked Read 9 Sent Done Done false
13955 4796 Acked Read 6 Sent Done Done false
13956 4797 Acked Read 1 Sent Done Done false
13957 4798 Acked Write 5 Sent Done Done false
13958 4799 Acked Read 4 Sent Done Done false
13959 4800 Acked Write 4 Sent Done Done false
13960 4801 Acked Write 3 Sent Done Done false
13961 4802 Acked Read 6 Sent Done Done false
13962 4803 Acked Write 10 Sent Done Done false
13963 4804 Acked Read 6 Sent Done Done false
13964 4805 Acked Read 5 Sent Done Done false
13965 4806 Acked Write 8 Sent Done Done false
13966 4807 Acked Write 1 Sent Done Done false
13967 4808 Acked Flush 0 Sent Done Done false
13968 4809 Acked Flush 0 Sent Done Done false
13969 4810 Acked Read 3 Sent Done Done false
13970 4811 Acked Read 4 Sent Done Done false
13971 4812 Acked Write 7 Sent Done Done false
13972 4813 Acked Flush 0 Sent Done Done false
13973 4814 Acked Read 9 Sent Done Done false
13974 4815 Acked Read 2 Sent Done Done false
13975 4816 Acked Read 3 Sent Done Done false
13976 4817 Acked Read 6 Sent Done Done false
13977 4818 Acked Write 7 Sent Done Done false
13978 4819 Acked Write 1 Sent Done Done false
13979 4820 Acked Read 5 Sent Done Done false
13980 4821 Acked Read 9 Sent Done Done false
13981 4822 Acked Write 7 Sent Done Done false
13982 4823 Acked Read 4 Sent Done Done false
13983 4824 Acked Read 2 Sent Done Done false
13984 4825 Acked Read 2 Sent Done Done false
13985 4826 Acked Read 8 Sent Done Done false
13986 4827 Acked Flush 0 Sent Done Done false
13987 4828 Acked Write 7 Sent Done Done false
13988 4829 Acked Write 7 Sent Done Done false
13989 4830 Acked Write 10 Sent Done Done false
13990 4831 Acked Write 2 Sent Done Done false
13991 4832 Acked Read 7 Sent Done Done false
13992 4833 Acked Write 8 Sent Done Done false
13993 4834 Acked Read 5 Sent Done Done false
13994 4835 Acked Write 8 Sent Done Done false
13995 4836 Acked Read 5 Sent Done Done false
13996 4837 Acked Flush 0 Sent Done Done false
13997 4838 Acked Read 9 Sent Done Done false
13998 4839 Acked Write 4 Sent Done Done false
13999 4840 Acked Read 5 Sent Done Done false
14000 4841 Acked Read 3 Sent Done Done false
14001 4842 Acked Read 3 Sent Done Done false
14002 4843 Acked Flush 0 Sent Done Done false
14003 4844 Acked Write 8 Sent Done Done false
14004 4845 Acked Write 5 Sent Done Done false
14005 4846 Acked Write 8 Sent Done Done false
14006 4847 Acked Read 2 Sent Done Done false
14007 4848 Acked Write 8 Sent Done Done false
14008 4849 Acked Write 1 Sent Done Done false
14009 4850 Acked Read 6 Sent Done Done false
14010 4851 Acked Read 5 Sent Done Done false
14011 4852 Acked Read 9 Sent Done Done false
14012 4853 Acked Read 9 Sent Done Done false
14013 4854 Acked Read 6 Sent Done Done false
14014 4855 Acked Write 6 Sent Done Done false
14015 4856 Acked Write 2 Sent Done Done false
14016 4857 Acked Read 9 Sent Done Done false
14017 4858 Acked Read 6 Sent Done Done false
14018 4859 Acked Write 4 Sent Done Done false
14019 4860 Acked Read 9 Sent Done Done false
14020 4861 Acked Read 3 Sent Done Done false
14021 4862 Acked Read 8 Sent Done Done false
14022 4863 Acked Read 8 Sent Done Done false
14023 4864 Acked Write 5 Sent Done Done false
14024 4865 Acked Read 6 Sent Done Done false
14025 4866 Acked Read 3 Sent Done Done false
14026 4867 Acked Read 8 Sent Done Done false
14027 4868 Acked Write 8 Sent Done Done false
14028 4869 Acked Write 9 Sent Done Done false
14029 4870 Acked Read 9 Sent Done Done false
14030 4871 Acked Read 8 Sent Done Done false
14031 4872 Acked Write 10 Sent Done Done false
14032 4873 Acked Write 1 Sent Done Done false
14033 4874 Acked Read 8 Sent Done Done false
14034 4875 Acked Write 2 Sent Done Done false
14035 4876 Acked Read 9 Sent Done Done false
14036 4877 Acked Read 7 Sent Done Done false
14037 4878 Acked Read 10 Sent Done Done false
14038 4879 Acked Read 4 Sent Done Done false
14039 4880 Acked Write 7 Sent Done Done false
14040 4881 Acked Read 1 Sent Done Done false
14041 4882 Acked Flush 0 Sent Done Done false
14042 4883 Acked Flush 0 Sent Done Done false
14043 4884 Acked Read 6 Sent Done Done false
14044 4885 Acked Write 6 Sent Done Done false
14045 4886 Acked Read 2 Sent Done Done false
14046 4887 Acked Read 6 Sent Done Done false
14047 4888 Acked Read 3 Sent Done Done false
14048 4889 Acked Write 3 Sent Done Done false
14049 4890 Acked Read 2 Sent Done Done false
14050 4891 Acked Read 1 Sent Done Done false
14051 4892 Acked Read 7 Sent Done Done false
14052 4893 Acked Write 1 Sent Done Done false
14053 4894 Acked Read 9 Sent Done Done false
14054 4895 Acked Read 4 Sent Done Done false
14055 4896 Acked Read 7 Sent Done Done false
14056 4897 Acked Write 4 Sent Done Done false
14057 4898 Acked Write 2 Sent Done Done false
14058 4899 Acked Write 3 Sent Done Done false
14059 4900 Acked Read 5 Sent Done Done false
14060 4901 Acked Flush 0 Sent Done Done false
14061 4902 Acked Read 1 Sent Done Done false
14062 4903 Acked Read 9 Sent Done Done false
14063 4904 Acked Read 1 Sent Done Done false
14064 4905 Acked Read 6 Sent Done Done false
14065 4906 Acked Write 4 Sent Done Done false
14066 4907 Acked Read 1 Sent Done Done false
14067 4908 Acked Flush 0 Sent Done Done false
14068 4909 Acked Write 10 Sent Done Done false
14069 4910 Acked Read 5 Sent Done Done false
14070 4911 Acked Read 6 Sent Done Done false
14071 4912 Acked Read 5 Sent Done Done false
14072 4913 Acked Read 8 Sent Done Done false
14073 4914 Acked Read 7 Sent Done Done false
14074 4915 Acked Read 2 Sent Done Done false
14075 4916 Acked Write 7 Sent Done Done false
14076 4917 Acked Write 2 Sent Done Done false
14077 4918 Acked Read 3 Sent Done Done false
14078 4919 Acked Flush 0 Sent Done Done false
14079 4920 Acked Flush 0 Sent Done Done false
14080 4921 Acked Write 5 Sent Done Done false
14081 4922 Acked Read 2 Sent Done Done false
14082 4923 Acked Write 6 Sent Done Done false
14083 4924 Acked Write 9 Sent Done Done false
14084 4925 Acked Write 5 Sent Done Done false
14085 4926 Acked Read 6 Sent Done Done false
14086 4927 Acked Flush 0 Sent Done Done false
14087 STATES DS:0 DS:1 DS:2 TOTAL
14088 Sent 302 0 0 302
14089 Done 0 302 302 604
14090 Skipped 0 0 0 0
14091 Error 0 0 0 0
14092 Last Flush: 4625 4927 4927
14093 Downstairs last five completed: 4625 4624 4623 4622 4621
14094 Upstairs last five completed: 4927 4926 4925 4924 4923
14095 CLIENT: Up:0 ds:302 act:2
140962024-11-01T16:53:25.826ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
140972024-11-01T16:53:25.826ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
140982024-11-01T16:53:25.826ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
140992024-11-01T16:53:25.826ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
141002024-11-01T16:53:25.827ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
14101 [8810] got start action
14102 [8810] I am going to restart
14103 Starting downstairs at port 8810
14104 Make output file at "/tmp/dsc/downstairs-8810.txt"
14105 [8810][0] reports Starting
14106 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1211)
14107 [8810][0] reports Running
14108 ----------------------------------------------------------------
14109 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
14110 States: Offline Active Active
14111 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14112 4626 Acked Read 6 Sent Done Done false
14113 4627 Acked Read 8 Sent Done Done false
14114 4628 Acked Read 9 Sent Done Done false
14115 4629 Acked Write 1 Sent Done Done false
14116 4630 Acked Read 5 Sent Done Done false
14117 4631 Acked Read 10 Sent Done Done false
14118 4632 Acked Read 1 Sent Done Done false
14119 4633 Acked Read 8 Sent Done Done false
14120 4634 Acked Read 8 Sent Done Done false
14121 4635 Acked Write 5 Sent Done Done false
14122 4636 Acked Flush 0 Sent Done Done false
14123 4637 Acked Write 8 Sent Done Done false
14124 4638 Acked Read 6 Sent Done Done false
14125 4639 Acked Read 1 Sent Done Done false
14126 4640 Acked Read 10 Sent Done Done false
14127 4641 Acked Write 7 Sent Done Done false
14128 4642 Acked Read 9 Sent Done Done false
14129 4643 Acked Read 1 Sent Done Done false
14130 4644 Acked Read 4 Sent Done Done false
14131 4645 Acked Read 3 Sent Done Done false
14132 4646 Acked Write 3 Sent Done Done false
14133 4647 Acked Read 3 Sent Done Done false
14134 4648 Acked Read 3 Sent Done Done false
14135 4649 Acked Read 4 Sent Done Done false
14136 4650 Acked Read 5 Sent Done Done false
14137 4651 Acked Read 3 Sent Done Done false
14138 4652 Acked Read 7 Sent Done Done false
14139 4653 Acked Read 7 Sent Done Done false
14140 4654 Acked Read 3 Sent Done Done false
14141 4655 Acked Write 8 Sent Done Done false
14142 4656 Acked Write 10 Sent Done Done false
14143 4657 Acked Read 6 Sent Done Done false
14144 4658 Acked Read 3 Sent Done Done false
14145 4659 Acked Flush 0 Sent Done Done false
14146 4660 Acked Write 10 Sent Done Done false
14147 4661 Acked Write 10 Sent Done Done false
14148 4662 Acked Read 6 Sent Done Done false
14149 4663 Acked Write 5 Sent Done Done false
14150 4664 Acked Read 8 Sent Done Done false
14151 4665 Acked Write 7 Sent Done Done false
14152 4666 Acked Write 6 Sent Done Done false
14153 4667 Acked Read 7 Sent Done Done false
14154 4668 Acked Write 1 Sent Done Done false
14155 4669 Acked Write 2 Sent Done Done false
14156 4670 Acked Write 7 Sent Done Done false
14157 4671 Acked Write 9 Sent Done Done false
14158 4672 Acked Read 5 Sent Done Done false
14159 4673 Acked Write 8 Sent Done Done false
14160 4674 Acked Read 3 Sent Done Done false
14161 4675 Acked Read 8 Sent Done Done false
14162 4676 Acked Read 1 Sent Done Done false
14163 4677 Acked Read 7 Sent Done Done false
14164 4678 Acked Read 9 Sent Done Done false
14165 4679 Acked Read 2 Sent Done Done false
14166 4680 Acked Write 10 Sent Done Done false
14167 4681 Acked Read 2 Sent Done Done false
14168 4682 Acked Write 6 Sent Done Done false
14169 4683 Acked Read 9 Sent Done Done false
14170 4684 Acked Read 2 Sent Done Done false
14171 4685 Acked Read 1 Sent Done Done false
14172 4686 Acked Read 10 Sent Done Done false
14173 4687 Acked Write 5 Sent Done Done false
14174 4688 Acked Flush 0 Sent Done Done false
14175 4689 Acked Read 3 Sent Done Done false
14176 4690 Acked Read 5 Sent Done Done false
14177 4691 Acked Read 5 Sent Done Done false
14178 4692 Acked Read 1 Sent Done Done false
14179 4693 Acked Write 5 Sent Done Done false
14180 4694 Acked Write 5 Sent Done Done false
14181 4695 Acked Write 1 Sent Done Done false
14182 4696 Acked Write 9 Sent Done Done false
14183 4697 Acked Read 7 Sent Done Done false
14184 4698 Acked Write 6 Sent Done Done false
14185 4699 Acked Read 2 Sent Done Done false
14186 4700 Acked Read 5 Sent Done Done false
14187 4701 Acked Write 8 Sent Done Done false
14188 4702 Acked Write 5 Sent Done Done false
14189 4703 Acked Write 1 Sent Done Done false
14190 4704 Acked Write 7 Sent Done Done false
14191 4705 Acked Write 6 Sent Done Done false
14192 4706 Acked Read 1 Sent Done Done false
14193 4707 Acked Write 6 Sent Done Done false
14194 4708 Acked Read 9 Sent Done Done false
14195 4709 Acked Read 2 Sent Done Done false
14196 4710 Acked Read 5 Sent Done Done false
14197 4711 Acked Flush 0 Sent Done Done false
14198 4712 Acked Read 4 Sent Done Done false
14199 4713 Acked Read 5 Sent Done Done false
14200 4714 Acked Read 3 Sent Done Done false
14201 4715 Acked Write 6 Sent Done Done false
14202 4716 Acked Read 10 Sent Done Done false
14203 4717 Acked Read 10 Sent Done Done false
14204 4718 Acked Read 5 Sent Done Done false
14205 4719 Acked Read 2 Sent Done Done false
14206 4720 Acked Read 5 Sent Done Done false
14207 4721 Acked Read 1 Sent Done Done false
14208 4722 Acked Write 5 Sent Done Done false
14209 4723 Acked Read 2 Sent Done Done false
14210 4724 Acked Write 6 Sent Done Done false
14211 4725 Acked Write 4 Sent Done Done false
14212 4726 Acked Write 7 Sent Done Done false
14213 4727 Acked Write 9 Sent Done Done false
14214 4728 Acked Write 6 Sent Done Done false
14215 4729 Acked Read 5 Sent Done Done false
14216 4730 Acked Read 4 Sent Done Done false
14217 4731 Acked Read 4 Sent Done Done false
14218 4732 Acked Flush 0 Sent Done Done false
14219 4733 Acked Write 10 Sent Done Done false
14220 4734 Acked Read 3 Sent Done Done false
14221 4735 Acked Read 10 Sent Done Done false
14222 4736 Acked Write 3 Sent Done Done false
14223 4737 Acked Read 10 Sent Done Done false
14224 4738 Acked Write 8 Sent Done Done false
14225 4739 Acked Flush 0 Sent Done Done false
14226 4740 Acked Read 10 Sent Done Done false
14227 4741 Acked Read 5 Sent Done Done false
14228 4742 Acked Flush 0 Sent Done Done false
14229 4743 Acked Write 4 Sent Done Done false
14230 4744 Acked Read 2 Sent Done Done false
14231 4745 Acked Read 3 Sent Done Done false
14232 4746 Acked Write 7 Sent Done Done false
14233 4747 Acked Write 2 Sent Done Done false
14234 4748 Acked Read 6 Sent Done Done false
14235 4749 Acked Read 6 Sent Done Done false
14236 4750 Acked Write 5 Sent Done Done false
14237 4751 Acked Read 2 Sent Done Done false
14238 4752 Acked Read 7 Sent Done Done false
14239 4753 Acked Write 4 Sent Done Done false
14240 4754 Acked Read 2 Sent Done Done false
14241 4755 Acked Read 1 Sent Done Done false
14242 4756 Acked Write 3 Sent Done Done false
14243 4757 Acked Read 5 Sent Done Done false
14244 4758 Acked Read 4 Sent Done Done false
14245 4759 Acked Read 6 Sent Done Done false
14246 4760 Acked Read 1 Sent Done Done false
14247 4761 Acked Write 1 Sent Done Done false
14248 4762 Acked Read 2 Sent Done Done false
14249 4763 Acked Write 5 Sent Done Done false
14250 4764 Acked Write 8 Sent Done Done false
14251 4765 Acked Write 8 Sent Done Done false
14252 4766 Acked Write 7 Sent Done Done false
14253 4767 Acked Read 8 Sent Done Done false
14254 4768 Acked Write 8 Sent Done Done false
14255 4769 Acked Write 3 Sent Done Done false
14256 4770 Acked Write 4 Sent Done Done false
14257 4771 Acked Read 8 Sent Done Done false
14258 4772 Acked Flush 0 Sent Done Done false
14259 4773 Acked Read 10 Sent Done Done false
14260 4774 Acked Write 1 Sent Done Done false
14261 4775 Acked Write 5 Sent Done Done false
14262 4776 Acked Read 8 Sent Done Done false
14263 4777 Acked Read 6 Sent Done Done false
14264 4778 Acked Write 9 Sent Done Done false
14265 4779 Acked Flush 0 Sent Done Done false
14266 4780 Acked Write 5 Sent Done Done false
14267 4781 Acked Write 1 Sent Done Done false
14268 4782 Acked Write 6 Sent Done Done false
14269 4783 Acked Write 3 Sent Done Done false
14270 4784 Acked Read 6 Sent Done Done false
14271 4785 Acked Write 5 Sent Done Done false
14272 4786 Acked Read 1 Sent Done Done false
14273 4787 Acked Write 7 Sent Done Done false
14274 4788 Acked Read 3 Sent Done Done false
14275 4789 Acked Write 7 Sent Done Done false
14276 4790 Acked Read 9 Sent Done Done false
14277 4791 Acked Write 3 Sent Done Done false
14278 4792 Acked Write 5 Sent Done Done false
14279 4793 Acked Write 4 Sent Done Done false
14280 4794 Acked Write 9 Sent Done Done false
14281 4795 Acked Read 9 Sent Done Done false
14282 4796 Acked Read 6 Sent Done Done false
14283 4797 Acked Read 1 Sent Done Done false
14284 4798 Acked Write 5 Sent Done Done false
14285 4799 Acked Read 4 Sent Done Done false
14286 4800 Acked Write 4 Sent Done Done false
14287 4801 Acked Write 3 Sent Done Done false
14288 4802 Acked Read 6 Sent Done Done false
14289 4803 Acked Write 10 Sent Done Done false
14290 4804 Acked Read 6 Sent Done Done false
14291 4805 Acked Read 5 Sent Done Done false
14292 4806 Acked Write 8 Sent Done Done false
14293 4807 Acked Write 1 Sent Done Done false
14294 4808 Acked Flush 0 Sent Done Done false
14295 4809 Acked Flush 0 Sent Done Done false
14296 4810 Acked Read 3 Sent Done Done false
14297 4811 Acked Read 4 Sent Done Done false
14298 4812 Acked Write 7 Sent Done Done false
14299 4813 Acked Flush 0 Sent Done Done false
14300 4814 Acked Read 9 Sent Done Done false
14301 4815 Acked Read 2 Sent Done Done false
14302 4816 Acked Read 3 Sent Done Done false
14303 4817 Acked Read 6 Sent Done Done false
14304 4818 Acked Write 7 Sent Done Done false
14305 4819 Acked Write 1 Sent Done Done false
14306 4820 Acked Read 5 Sent Done Done false
14307 4821 Acked Read 9 Sent Done Done false
14308 4822 Acked Write 7 Sent Done Done false
14309 4823 Acked Read 4 Sent Done Done false
14310 4824 Acked Read 2 Sent Done Done false
14311 4825 Acked Read 2 Sent Done Done false
14312 4826 Acked Read 8 Sent Done Done false
14313 4827 Acked Flush 0 Sent Done Done false
14314 4828 Acked Write 7 Sent Done Done false
14315 4829 Acked Write 7 Sent Done Done false
14316 4830 Acked Write 10 Sent Done Done false
14317 4831 Acked Write 2 Sent Done Done false
14318 4832 Acked Read 7 Sent Done Done false
14319 4833 Acked Write 8 Sent Done Done false
14320 4834 Acked Read 5 Sent Done Done false
14321 4835 Acked Write 8 Sent Done Done false
14322 4836 Acked Read 5 Sent Done Done false
14323 4837 Acked Flush 0 Sent Done Done false
14324 4838 Acked Read 9 Sent Done Done false
14325 4839 Acked Write 4 Sent Done Done false
14326 4840 Acked Read 5 Sent Done Done false
14327 4841 Acked Read 3 Sent Done Done false
14328 4842 Acked Read 3 Sent Done Done false
14329 4843 Acked Flush 0 Sent Done Done false
14330 4844 Acked Write 8 Sent Done Done false
14331 4845 Acked Write 5 Sent Done Done false
14332 4846 Acked Write 8 Sent Done Done false
14333 4847 Acked Read 2 Sent Done Done false
14334 4848 Acked Write 8 Sent Done Done false
14335 4849 Acked Write 1 Sent Done Done false
14336 4850 Acked Read 6 Sent Done Done false
14337 4851 Acked Read 5 Sent Done Done false
14338 4852 Acked Read 9 Sent Done Done false
14339 4853 Acked Read 9 Sent Done Done false
14340 4854 Acked Read 6 Sent Done Done false
14341 4855 Acked Write 6 Sent Done Done false
14342 4856 Acked Write 2 Sent Done Done false
14343 4857 Acked Read 9 Sent Done Done false
14344 4858 Acked Read 6 Sent Done Done false
14345 4859 Acked Write 4 Sent Done Done false
14346 4860 Acked Read 9 Sent Done Done false
14347 4861 Acked Read 3 Sent Done Done false
14348 4862 Acked Read 8 Sent Done Done false
14349 4863 Acked Read 8 Sent Done Done false
14350 4864 Acked Write 5 Sent Done Done false
14351 4865 Acked Read 6 Sent Done Done false
14352 4866 Acked Read 3 Sent Done Done false
14353 4867 Acked Read 8 Sent Done Done false
14354 4868 Acked Write 8 Sent Done Done false
14355 4869 Acked Write 9 Sent Done Done false
14356 4870 Acked Read 9 Sent Done Done false
14357 4871 Acked Read 8 Sent Done Done false
14358 4872 Acked Write 10 Sent Done Done false
14359 4873 Acked Write 1 Sent Done Done false
14360 4874 Acked Read 8 Sent Done Done false
14361 4875 Acked Write 2 Sent Done Done false
14362 4876 Acked Read 9 Sent Done Done false
14363 4877 Acked Read 7 Sent Done Done false
14364 4878 Acked Read 10 Sent Done Done false
14365 4879 Acked Read 4 Sent Done Done false
14366 4880 Acked Write 7 Sent Done Done false
14367 4881 Acked Read 1 Sent Done Done false
14368 4882 Acked Flush 0 Sent Done Done false
14369 4883 Acked Flush 0 Sent Done Done false
14370 4884 Acked Read 6 Sent Done Done false
14371 4885 Acked Write 6 Sent Done Done false
14372 4886 Acked Read 2 Sent Done Done false
14373 4887 Acked Read 6 Sent Done Done false
14374 4888 Acked Read 3 Sent Done Done false
14375 4889 Acked Write 3 Sent Done Done false
14376 4890 Acked Read 2 Sent Done Done false
14377 4891 Acked Read 1 Sent Done Done false
14378 4892 Acked Read 7 Sent Done Done false
14379 4893 Acked Write 1 Sent Done Done false
14380 4894 Acked Read 9 Sent Done Done false
14381 4895 Acked Read 4 Sent Done Done false
14382 4896 Acked Read 7 Sent Done Done false
14383 4897 Acked Write 4 Sent Done Done false
14384 4898 Acked Write 2 Sent Done Done false
14385 4899 Acked Write 3 Sent Done Done false
14386 4900 Acked Read 5 Sent Done Done false
14387 4901 Acked Flush 0 Sent Done Done false
14388 4902 Acked Read 1 Sent Done Done false
14389 4903 Acked Read 9 Sent Done Done false
14390 4904 Acked Read 1 Sent Done Done false
14391 4905 Acked Read 6 Sent Done Done false
14392 4906 Acked Write 4 Sent Done Done false
14393 4907 Acked Read 1 Sent Done Done false
14394 4908 Acked Flush 0 Sent Done Done false
14395 4909 Acked Write 10 Sent Done Done false
14396 4910 Acked Read 5 Sent Done Done false
14397 4911 Acked Read 6 Sent Done Done false
14398 4912 Acked Read 5 Sent Done Done false
14399 4913 Acked Read 8 Sent Done Done false
14400 4914 Acked Read 7 Sent Done Done false
14401 4915 Acked Read 2 Sent Done Done false
14402 4916 Acked Write 7 Sent Done Done false
14403 4917 Acked Write 2 Sent Done Done false
14404 4918 Acked Read 3 Sent Done Done false
14405 4919 Acked Flush 0 Sent Done Done false
14406 4920 Acked Flush 0 Sent Done Done false
14407 4921 Acked Write 5 Sent Done Done false
14408 4922 Acked Read 2 Sent Done Done false
14409 4923 Acked Write 6 Sent Done Done false
14410 4924 Acked Write 9 Sent Done Done false
14411 4925 Acked Write 5 Sent Done Done false
14412 4926 Acked Read 6 Sent Done Done false
14413 4927 Acked Flush 0 Sent Done Done false
14414 STATES DS:0 DS:1 DS:2 TOTAL
14415 Sent 302 0 0 302
14416 Done 0 302 302 604
14417 Skipped 0 0 0 0
14418 Error 0 0 0 0
14419 Last Flush: 4625 4927 4927
14420 Downstairs last five completed: 4625 4624 4623 4622 4621
14421 Upstairs last five completed: 4927 4926 4925 4924 4923
14422 CLIENT: Up:0 ds:302 act:2
14423 ----------------------------------------------------------------
14424 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
14425 States: Offline Active Active
14426 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14427 4626 Acked Read 6 Sent Done Done false
14428 4627 Acked Read 8 Sent Done Done false
14429 4628 Acked Read 9 Sent Done Done false
14430 4629 Acked Write 1 Sent Done Done false
14431 4630 Acked Read 5 Sent Done Done false
14432 4631 Acked Read 10 Sent Done Done false
14433 4632 Acked Read 1 Sent Done Done false
14434 4633 Acked Read 8 Sent Done Done false
14435 4634 Acked Read 8 Sent Done Done false
14436 4635 Acked Write 5 Sent Done Done false
14437 4636 Acked Flush 0 Sent Done Done false
14438 4637 Acked Write 8 Sent Done Done false
14439 4638 Acked Read 6 Sent Done Done false
14440 4639 Acked Read 1 Sent Done Done false
14441 4640 Acked Read 10 Sent Done Done false
14442 4641 Acked Write 7 Sent Done Done false
14443 4642 Acked Read 9 Sent Done Done false
14444 4643 Acked Read 1 Sent Done Done false
14445 4644 Acked Read 4 Sent Done Done false
14446 4645 Acked Read 3 Sent Done Done false
14447 4646 Acked Write 3 Sent Done Done false
14448 4647 Acked Read 3 Sent Done Done false
14449 4648 Acked Read 3 Sent Done Done false
14450 4649 Acked Read 4 Sent Done Done false
14451 4650 Acked Read 5 Sent Done Done false
14452 4651 Acked Read 3 Sent Done Done false
14453 4652 Acked Read 7 Sent Done Done false
14454 4653 Acked Read 7 Sent Done Done false
14455 4654 Acked Read 3 Sent Done Done false
14456 4655 Acked Write 8 Sent Done Done false
14457 4656 Acked Write 10 Sent Done Done false
14458 4657 Acked Read 6 Sent Done Done false
14459 4658 Acked Read 3 Sent Done Done false
14460 4659 Acked Flush 0 Sent Done Done false
14461 4660 Acked Write 10 Sent Done Done false
14462 4661 Acked Write 10 Sent Done Done false
14463 4662 Acked Read 6 Sent Done Done false
14464 4663 Acked Write 5 Sent Done Done false
14465 4664 Acked Read 8 Sent Done Done false
14466 4665 Acked Write 7 Sent Done Done false
14467 4666 Acked Write 6 Sent Done Done false
14468 4667 Acked Read 7 Sent Done Done false
14469 4668 Acked Write 1 Sent Done Done false
14470 4669 Acked Write 2 Sent Done Done false
14471 4670 Acked Write 7 Sent Done Done false
14472 4671 Acked Write 9 Sent Done Done false
14473 4672 Acked Read 5 Sent Done Done false
14474 4673 Acked Write 8 Sent Done Done false
14475 4674 Acked Read 3 Sent Done Done false
14476 4675 Acked Read 8 Sent Done Done false
14477 4676 Acked Read 1 Sent Done Done false
14478 4677 Acked Read 7 Sent Done Done false
14479 4678 Acked Read 9 Sent Done Done false
14480 4679 Acked Read 2 Sent Done Done false
14481 4680 Acked Write 10 Sent Done Done false
14482 4681 Acked Read 2 Sent Done Done false
14483 4682 Acked Write 6 Sent Done Done false
14484 4683 Acked Read 9 Sent Done Done false
14485 4684 Acked Read 2 Sent Done Done false
14486 4685 Acked Read 1 Sent Done Done false
14487 4686 Acked Read 10 Sent Done Done false
14488 4687 Acked Write 5 Sent Done Done false
14489 4688 Acked Flush 0 Sent Done Done false
14490 4689 Acked Read 3 Sent Done Done false
14491 4690 Acked Read 5 Sent Done Done false
14492 4691 Acked Read 5 Sent Done Done false
14493 4692 Acked Read 1 Sent Done Done false
14494 4693 Acked Write 5 Sent Done Done false
14495 4694 Acked Write 5 Sent Done Done false
14496 4695 Acked Write 1 Sent Done Done false
14497 4696 Acked Write 9 Sent Done Done false
14498 4697 Acked Read 7 Sent Done Done false
14499 4698 Acked Write 6 Sent Done Done false
14500 4699 Acked Read 2 Sent Done Done false
14501 4700 Acked Read 5 Sent Done Done false
14502 4701 Acked Write 8 Sent Done Done false
14503 4702 Acked Write 5 Sent Done Done false
14504 4703 Acked Write 1 Sent Done Done false
14505 4704 Acked Write 7 Sent Done Done false
14506 4705 Acked Write 6 Sent Done Done false
14507 4706 Acked Read 1 Sent Done Done false
14508 4707 Acked Write 6 Sent Done Done false
14509 4708 Acked Read 9 Sent Done Done false
14510 4709 Acked Read 2 Sent Done Done false
14511 4710 Acked Read 5 Sent Done Done false
14512 4711 Acked Flush 0 Sent Done Done false
14513 4712 Acked Read 4 Sent Done Done false
14514 4713 Acked Read 5 Sent Done Done false
14515 4714 Acked Read 3 Sent Done Done false
14516 4715 Acked Write 6 Sent Done Done false
14517 4716 Acked Read 10 Sent Done Done false
14518 4717 Acked Read 10 Sent Done Done false
14519 4718 Acked Read 5 Sent Done Done false
14520 4719 Acked Read 2 Sent Done Done false
14521 4720 Acked Read 5 Sent Done Done false
14522 4721 Acked Read 1 Sent Done Done false
14523 4722 Acked Write 5 Sent Done Done false
14524 4723 Acked Read 2 Sent Done Done false
14525 4724 Acked Write 6 Sent Done Done false
14526 4725 Acked Write 4 Sent Done Done false
14527 4726 Acked Write 7 Sent Done Done false
14528 4727 Acked Write 9 Sent Done Done false
14529 4728 Acked Write 6 Sent Done Done false
14530 4729 Acked Read 5 Sent Done Done false
14531 4730 Acked Read 4 Sent Done Done false
14532 4731 Acked Read 4 Sent Done Done false
14533 4732 Acked Flush 0 Sent Done Done false
14534 4733 Acked Write 10 Sent Done Done false
14535 4734 Acked Read 3 Sent Done Done false
14536 4735 Acked Read 10 Sent Done Done false
14537 4736 Acked Write 3 Sent Done Done false
14538 4737 Acked Read 10 Sent Done Done false
14539 4738 Acked Write 8 Sent Done Done false
14540 4739 Acked Flush 0 Sent Done Done false
14541 4740 Acked Read 10 Sent Done Done false
14542 4741 Acked Read 5 Sent Done Done false
14543 4742 Acked Flush 0 Sent Done Done false
14544 4743 Acked Write 4 Sent Done Done false
14545 4744 Acked Read 2 Sent Done Done false
14546 4745 Acked Read 3 Sent Done Done false
14547 4746 Acked Write 7 Sent Done Done false
14548 4747 Acked Write 2 Sent Done Done false
14549 4748 Acked Read 6 Sent Done Done false
14550 4749 Acked Read 6 Sent Done Done false
14551 4750 Acked Write 5 Sent Done Done false
14552 4751 Acked Read 2 Sent Done Done false
14553 4752 Acked Read 7 Sent Done Done false
14554 4753 Acked Write 4 Sent Done Done false
14555 4754 Acked Read 2 Sent Done Done false
14556 4755 Acked Read 1 Sent Done Done false
14557 4756 Acked Write 3 Sent Done Done false
14558 4757 Acked Read 5 Sent Done Done false
14559 4758 Acked Read 4 Sent Done Done false
14560 4759 Acked Read 6 Sent Done Done false
14561 4760 Acked Read 1 Sent Done Done false
14562 4761 Acked Write 1 Sent Done Done false
14563 4762 Acked Read 2 Sent Done Done false
14564 4763 Acked Write 5 Sent Done Done false
14565 4764 Acked Write 8 Sent Done Done false
14566 4765 Acked Write 8 Sent Done Done false
14567 4766 Acked Write 7 Sent Done Done false
14568 4767 Acked Read 8 Sent Done Done false
14569 4768 Acked Write 8 Sent Done Done false
14570 4769 Acked Write 3 Sent Done Done false
14571 4770 Acked Write 4 Sent Done Done false
14572 4771 Acked Read 8 Sent Done Done false
14573 4772 Acked Flush 0 Sent Done Done false
14574 4773 Acked Read 10 Sent Done Done false
14575 4774 Acked Write 1 Sent Done Done false
14576 4775 Acked Write 5 Sent Done Done false
14577 4776 Acked Read 8 Sent Done Done false
14578 4777 Acked Read 6 Sent Done Done false
14579 4778 Acked Write 9 Sent Done Done false
14580 4779 Acked Flush 0 Sent Done Done false
14581 4780 Acked Write 5 Sent Done Done false
14582 4781 Acked Write 1 Sent Done Done false
14583 4782 Acked Write 6 Sent Done Done false
14584 4783 Acked Write 3 Sent Done Done false
14585 4784 Acked Read 6 Sent Done Done false
14586 4785 Acked Write 5 Sent Done Done false
14587 4786 Acked Read 1 Sent Done Done false
14588 4787 Acked Write 7 Sent Done Done false
14589 4788 Acked Read 3 Sent Done Done false
14590 4789 Acked Write 7 Sent Done Done false
14591 4790 Acked Read 9 Sent Done Done false
14592 4791 Acked Write 3 Sent Done Done false
14593 4792 Acked Write 5 Sent Done Done false
14594 4793 Acked Write 4 Sent Done Done false
14595 4794 Acked Write 9 Sent Done Done false
14596 4795 Acked Read 9 Sent Done Done false
14597 4796 Acked Read 6 Sent Done Done false
14598 4797 Acked Read 1 Sent Done Done false
14599 4798 Acked Write 5 Sent Done Done false
14600 4799 Acked Read 4 Sent Done Done false
14601 4800 Acked Write 4 Sent Done Done false
14602 4801 Acked Write 3 Sent Done Done false
14603 4802 Acked Read 6 Sent Done Done false
14604 4803 Acked Write 10 Sent Done Done false
14605 4804 Acked Read 6 Sent Done Done false
14606 4805 Acked Read 5 Sent Done Done false
14607 4806 Acked Write 8 Sent Done Done false
14608 4807 Acked Write 1 Sent Done Done false
14609 4808 Acked Flush 0 Sent Done Done false
14610 4809 Acked Flush 0 Sent Done Done false
14611 4810 Acked Read 3 Sent Done Done false
14612 4811 Acked Read 4 Sent Done Done false
14613 4812 Acked Write 7 Sent Done Done false
14614 4813 Acked Flush 0 Sent Done Done false
14615 4814 Acked Read 9 Sent Done Done false
14616 4815 Acked Read 2 Sent Done Done false
14617 4816 Acked Read 3 Sent Done Done false
14618 4817 Acked Read 6 Sent Done Done false
14619 4818 Acked Write 7 Sent Done Done false
14620 4819 Acked Write 1 Sent Done Done false
14621 4820 Acked Read 5 Sent Done Done false
14622 4821 Acked Read 9 Sent Done Done false
14623 4822 Acked Write 7 Sent Done Done false
14624 4823 Acked Read 4 Sent Done Done false
14625 4824 Acked Read 2 Sent Done Done false
14626 4825 Acked Read 2 Sent Done Done false
14627 4826 Acked Read 8 Sent Done Done false
14628 4827 Acked Flush 0 Sent Done Done false
14629 4828 Acked Write 7 Sent Done Done false
14630 4829 Acked Write 7 Sent Done Done false
14631 4830 Acked Write 10 Sent Done Done false
14632 4831 Acked Write 2 Sent Done Done false
14633 4832 Acked Read 7 Sent Done Done false
14634 4833 Acked Write 8 Sent Done Done false
14635 4834 Acked Read 5 Sent Done Done false
14636 4835 Acked Write 8 Sent Done Done false
14637 4836 Acked Read 5 Sent Done Done false
14638 4837 Acked Flush 0 Sent Done Done false
14639 4838 Acked Read 9 Sent Done Done false
14640 4839 Acked Write 4 Sent Done Done false
14641 4840 Acked Read 5 Sent Done Done false
14642 4841 Acked Read 3 Sent Done Done false
14643 4842 Acked Read 3 Sent Done Done false
14644 4843 Acked Flush 0 Sent Done Done false
14645 4844 Acked Write 8 Sent Done Done false
14646 4845 Acked Write 5 Sent Done Done false
14647 4846 Acked Write 8 Sent Done Done false
14648 4847 Acked Read 2 Sent Done Done false
14649 4848 Acked Write 8 Sent Done Done false
14650 4849 Acked Write 1 Sent Done Done false
14651 4850 Acked Read 6 Sent Done Done false
14652 4851 Acked Read 5 Sent Done Done false
14653 4852 Acked Read 9 Sent Done Done false
14654 4853 Acked Read 9 Sent Done Done false
14655 4854 Acked Read 6 Sent Done Done false
14656 4855 Acked Write 6 Sent Done Done false
14657 4856 Acked Write 2 Sent Done Done false
14658 4857 Acked Read 9 Sent Done Done false
14659 4858 Acked Read 6 Sent Done Done false
14660 4859 Acked Write 4 Sent Done Done false
14661 4860 Acked Read 9 Sent Done Done false
14662 4861 Acked Read 3 Sent Done Done false
14663 4862 Acked Read 8 Sent Done Done false
14664 4863 Acked Read 8 Sent Done Done false
14665 4864 Acked Write 5 Sent Done Done false
14666 4865 Acked Read 6 Sent Done Done false
14667 4866 Acked Read 3 Sent Done Done false
14668 4867 Acked Read 8 Sent Done Done false
14669 4868 Acked Write 8 Sent Done Done false
14670 4869 Acked Write 9 Sent Done Done false
14671 4870 Acked Read 9 Sent Done Done false
14672 4871 Acked Read 8 Sent Done Done false
14673 4872 Acked Write 10 Sent Done Done false
14674 4873 Acked Write 1 Sent Done Done false
14675 4874 Acked Read 8 Sent Done Done false
14676 4875 Acked Write 2 Sent Done Done false
14677 4876 Acked Read 9 Sent Done Done false
14678 4877 Acked Read 7 Sent Done Done false
14679 4878 Acked Read 10 Sent Done Done false
14680 4879 Acked Read 4 Sent Done Done false
14681 4880 Acked Write 7 Sent Done Done false
14682 4881 Acked Read 1 Sent Done Done false
14683 4882 Acked Flush 0 Sent Done Done false
14684 4883 Acked Flush 0 Sent Done Done false
14685 4884 Acked Read 6 Sent Done Done false
14686 4885 Acked Write 6 Sent Done Done false
14687 4886 Acked Read 2 Sent Done Done false
14688 4887 Acked Read 6 Sent Done Done false
14689 4888 Acked Read 3 Sent Done Done false
14690 4889 Acked Write 3 Sent Done Done false
14691 4890 Acked Read 2 Sent Done Done false
14692 4891 Acked Read 1 Sent Done Done false
14693 4892 Acked Read 7 Sent Done Done false
14694 4893 Acked Write 1 Sent Done Done false
14695 4894 Acked Read 9 Sent Done Done false
14696 4895 Acked Read 4 Sent Done Done false
14697 4896 Acked Read 7 Sent Done Done false
14698 4897 Acked Write 4 Sent Done Done false
14699 4898 Acked Write 2 Sent Done Done false
14700 4899 Acked Write 3 Sent Done Done false
14701 4900 Acked Read 5 Sent Done Done false
14702 4901 Acked Flush 0 Sent Done Done false
14703 4902 Acked Read 1 Sent Done Done false
14704 4903 Acked Read 9 Sent Done Done false
14705 4904 Acked Read 1 Sent Done Done false
14706 4905 Acked Read 6 Sent Done Done false
14707 4906 Acked Write 4 Sent Done Done false
14708 4907 Acked Read 1 Sent Done Done false
14709 4908 Acked Flush 0 Sent Done Done false
14710 4909 Acked Write 10 Sent Done Done false
14711 4910 Acked Read 5 Sent Done Done false
14712 4911 Acked Read 6 Sent Done Done false
14713 4912 Acked Read 5 Sent Done Done false
14714 4913 Acked Read 8 Sent Done Done false
14715 4914 Acked Read 7 Sent Done Done false
14716 4915 Acked Read 2 Sent Done Done false
14717 4916 Acked Write 7 Sent Done Done false
14718 4917 Acked Write 2 Sent Done Done false
14719 4918 Acked Read 3 Sent Done Done false
14720 4919 Acked Flush 0 Sent Done Done false
14721 4920 Acked Flush 0 Sent Done Done false
14722 4921 Acked Write 5 Sent Done Done false
14723 4922 Acked Read 2 Sent Done Done false
14724 4923 Acked Write 6 Sent Done Done false
14725 4924 Acked Write 9 Sent Done Done false
14726 4925 Acked Write 5 Sent Done Done false
14727 4926 Acked Read 6 Sent Done Done false
14728 4927 Acked Flush 0 Sent Done Done false
14729 STATES DS:0 DS:1 DS:2 TOTAL
14730 Sent 302 0 0 302
14731 Done 0 302 302 604
14732 Skipped 0 0 0 0
14733 Error 0 0 0 0
14734 Last Flush: 4625 4927 4927
14735 Downstairs last five completed: 4625 4624 4623 4622 4621
14736 Upstairs last five completed: 4927 4926 4925 4924 4923
14737 CLIENT: Up:0 ds:302 act:2
147382024-11-01T16:53:35.827ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147392024-11-01T16:53:35.828ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36018) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147402024-11-01T16:53:35.829ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147412024-11-01T16:53:35.829ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147422024-11-01T16:53:35.829ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147432024-11-01T16:53:35.829ZINFOcrucible: send last flush ID to this DS: 4625 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147442024-11-01T16:53:35.830ZINFOcrucible: Replied this last flush ID: 4625 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147452024-11-01T16:53:35.830ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147462024-11-01T16:53:35.830ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147472024-11-01T16:53:35.830ZINFOcrucible: [0] client re-new 302 jobs since flush 4625 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147482024-11-01T16:53:35.830ZINFOcrucible: [0] Replayed 302 jobs since flush: 4625 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
14749 ----------------------------------------------------------------
14750 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14751 Downstairs last five completed: 4927 4926 4925 4924 4923
14752 Upstairs last five completed: 4927 4926 4925 4924 4923
14753 CLIENT: Up:0 ds:0 act:3
14754 Replay: All jobs finished
14755 Main task has work to do, go find it
14756 got dsc Stop(0)
14757 stop 0
14758 [8810] Got stop action so:false kr:false
14759 Nov 01 16:53:36.120 INFO request completed, latency_us: 323, response_code: 204, uri: /stop/cid/0, method: POST, req_id: a94096db-b30f-4034-a451-eb398d4c739e, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
147602024-11-01T16:53:36.121ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147612024-11-01T16:53:36.121ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147622024-11-01T16:53:36.121ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147632024-11-01T16:53:36.121ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
147642024-11-01T16:53:36.121ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
14765 Nov 01 16:53:36.121 INFO request completed, latency_us: 140, response_code: 200, uri: /state/cid/0, method: GET, req_id: b946e556-7544-4a11-849e-d036e223f44e, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
14766 [8810] Exited with: ExitStatus(unix_wait_status(9))
14767 [8810][0] reports Exit
14768 Nov 01 16:53:40.124 INFO request completed, latency_us: 334, response_code: 200, uri: /state/cid/0, method: GET, req_id: e970d02e-723d-4425-ab97-eb0020247389, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
14769 001/300 Write block 1138 len 4096 data: 3
14770 002/300 Read block 3513 len 20480
14771 003/300 Read block 4478 len 28672
14772 004/300 Flush
14773 005/300 Flush
14774 006/300 Write block 413 len 12288 data: 4 3 3
14775 007/300 Read block 2927 len 24576
14776 008/300 Read block 2806 len 40960
14777 009/300 Read block 2776 len 16384
14778 010/300 Read block 723 len 20480
14779 011/300 Read block 2442 len 24576
14780 012/300 Read block 2392 len 16384
14781 013/300 Read block 1335 len 36864
14782 014/300 Read block 2391 len 4096
14783 015/300 Read block 4264 len 28672
14784 016/300 Read block 2117 len 32768
14785 017/300 Write block 2885 len 36864 data: 2 3 3 4 5 5 4 3 3
14786 018/300 Write block 4245 len 16384 data: 2 3 4 4
14787 019/300 Write block 1327 len 12288 data: 3 3 3
14788 020/300 Write block 4258 len 36864 data: 3 3 3 3 3 4 3 3 4
14789 021/300 Write block 3057 len 4096 data: 2
14790 022/300 Read block 3310 len 28672
14791 023/300 Read block 1147 len 8192
14792 024/300 Read block 3859 len 16384
14793 025/300 Read block 1042 len 32768
14794 026/300 Write block 3109 len 24576 data: 6 5 5 5 4 2
14795 027/300 Flush
14796 028/300 Read block 1886 len 36864
14797 029/300 Read block 1054 len 12288
14798 030/300 Write block 57 len 4096 data: 2
14799 031/300 Write block 2474 len 4096 data: 4
14800 032/300 Read block 2099 len 4096
14801 033/300 Read block 129 len 24576
14802 034/300 Write block 548 len 24576 data: 3 4 5 5 6 6
14803 035/300 Read block 916 len 16384
14804 036/300 Write block 1737 len 40960 data: 3 3 4 5 3 3 3 3 3 3
14805 037/300 Flush
14806 038/300 Write block 4829 len 24576 data: 3 3 3 3 3 2
14807 039/300 Read block 4984 len 4096
14808 040/300 Read block 2107 len 24576
14809 041/300 Write block 1583 len 8192 data: 4 4
14810 042/300 Write block 2232 len 4096 data: 5
14811 043/300 Read block 352 len 8192
14812 044/300 Write block 178 len 20480 data: 2 3 3 4 6
14813 045/300 Read block 4781 len 16384
14814 046/300 Write block 3471 len 12288 data: 5 5 5
14815 047/300 Read block 417 len 20480
14816 048/300 Read block 1119 len 36864
14817 049/300 Read block 3773 len 28672
14818 050/300 Read block 2535 len 36864
14819 051/300 Write block 549 len 16384 data: 5 6 6 7
14820 052/300 Write block 4668 len 20480 data: 2 2 3 4 5
14821 053/300 Write block 3264 len 16384 data: 4 3 2 3
14822 054/300 Write block 1034 len 36864 data: 2 2 2 2 2 3 3 4 4
14823 055/300 Write block 2880 len 28672 data: 3 2 2 2 2 3 4
14824 056/300 Read block 4105 len 16384
14825 057/300 Read block 4104 len 36864
14826 058/300 Write block 4919 len 4096 data: 5
14827 059/300 Read block 2640 len 4096
14828 060/300 Write block 1168 len 20480 data: 2 2 2 4 5
14829 061/300 Read block 1474 len 40960
14830 062/300 Read block 2121 len 20480
14831 063/300 Write block 1572 len 32768 data: 2 2 2 3 3 3 3 2
14832 064/300 Read block 2431 len 8192
14833 065/300 Read block 3002 len 4096
14834 066/300 Read block 2230 len 28672
14835 067/300 Write block 3147 len 36864 data: 4 4 3 4 4 3 3 3 4
14836 068/300 Read block 3345 len 40960
14837 069/300 Write block 2983 len 24576 data: 3 3 3 3 5 5
14838 070/300 Write block 296 len 20480 data: 2 2 2 2 2
14839 071/300 Flush
14840 072/300 Write block 1320 len 4096 data: 4
14841 073/300 Write block 2849 len 20480 data: 3 2 2 2 3
14842 074/300 Read block 3150 len 20480
14843 075/300 Write block 3222 len 8192 data: 2 3
14844 076/300 Read block 832 len 4096
14845 077/300 Write block 4404 len 40960 data: 3 3 3 3 2 4 4 4 3 2
14846 078/300 Write block 761 len 16384 data: 3 5 5 4
14847 079/300 Read block 4943 len 40960
14848 080/300 Read block 1919 len 16384
14849 081/300 Read block 1239 len 36864
14850 082/300 Read block 4367 len 4096
14851 083/300 Write block 4317 len 20480 data: 7 10 10 10 9
14852 084/300 Read block 2317 len 20480
14853 085/300 Write block 850 len 16384 data: 8 7 7 6
14854 086/300 Write block 4473 len 36864 data: 2 2 2 2 2 2 2 2 3
14855 087/300 Write block 917 len 4096 data: 4
14856 088/300 Flush
14857 089/300 Write block 1425 len 28672 data: 3 3 2 2 3 3 5
14858 090/300 Write block 4051 len 16384 data: 6 6 6 5
14859 091/300 Read block 3909 len 12288
14860 092/300 Read block 98 len 8192
14861 093/300 Read block 1531 len 12288
14862 094/300 Read block 2109 len 32768
14863 095/300 Read block 519 len 24576
14864 096/300 Read block 3315 len 28672
14865 097/300 Read block 4242 len 28672
14866 098/300 Write block 2437 len 20480 data: 4 4 2 2 2
14867 099/300 Write block 1331 len 32768 data: 3 3 4 4 5 5 5 4
14868 100/300 Write block 2948 len 40960 data: 3 3 3 3 4 5 6 6 8 7
14869 101/300 Flush
14870 102/300 Write block 1034 len 24576 data: 3 3 3 3 3 4
14871 103/300 Read block 3004 len 12288
14872 104/300 Write block 3009 len 16384 data: 3 2 2 2
14873 105/300 Read block 910 len 20480
14874 106/300 Write block 3860 len 36864 data: 5 4 5 5 5 4 5 4 4
14875 107/300 Flush
14876 108/300 Write block 1717 len 8192 data: 5 5
14877 109/300 Write block 4676 len 32768 data: 7 6 4 3 3 3 3 3
14878 110/300 Read block 3846 len 4096
14879 111/300 Read block 3694 len 28672
14880 112/300 Write block 4852 len 4096 data: 3
14881 113/300 Read block 4031 len 16384
14882 114/300 Flush
14883 115/300 Write block 3085 len 12288 data: 5 4 4
14884 116/300 Flush
14885 117/300 Read block 4490 len 16384
14886 118/300 Flush
14887 119/300 Read block 509 len 12288
14888 120/300 Write block 4371 len 8192 data: 5 5
14889 121/300 Read block 2743 len 4096
14890 122/300 Read block 107 len 20480
14891 123/300 Write block 3092 len 40960 data: 5 5 6 5 4 4 4 4 4 3
14892 124/300 Read block 437 len 12288
14893 125/300 Read block 3556 len 16384
14894 126/300 Read block 174 len 32768
14895 127/300 Write block 2805 len 24576 data: 2 2 4 4 4 5
14896 128/300 Write block 1914 len 24576 data: 3 4 2 3 3 2
14897 129/300 Write block 4830 len 24576 data: 4 4 4 4 3 2
14898 130/300 Read block 996 len 16384
14899 131/300 Flush
14900 132/300 Write block 107 len 40960 data: 3 4 3 3 2 2 4 4 6 6
14901 133/300 Read block 3318 len 36864
14902 134/300 Read block 1989 len 4096
14903 135/300 Read block 1674 len 16384
14904 136/300 Read block 3526 len 32768
14905 137/300 Read block 1357 len 32768
14906 138/300 Write block 73 len 40960 data: 3 3 4 4 4 3 3 3 3 2
14907 139/300 Read block 759 len 8192
14908 140/300 Read block 4912 len 20480
14909 141/300 Write block 3554 len 28672 data: 4 4 4 4 5 5 4
14910 142/300 Flush
14911 143/300 Write block 3265 len 12288 data: 4 3 4
14912 144/300 Write block 1410 len 16384 data: 4 3 3 4
14913 145/300 Flush
14914 146/300 Read block 4944 len 32768
14915 147/300 Write block 3787 len 16384 data: 3 3 3 4
14916 148/300 Read block 1406 len 16384
14917 149/300 Write block 2949 len 4096 data: 4
14918 150/300 Read block 4215 len 4096
14919 151/300 Read block 3297 len 32768
14920 152/300 Read block 4527 len 28672
14921 153/300 Write block 4676 len 24576 data: 8 7 5 4 4 4
14922 154/300 Read block 3318 len 20480
14923 155/300 Read block 1286 len 4096
14924 156/300 Read block 4129 len 8192
14925 157/300 Read block 3964 len 16384
14926 158/300 Read block 4093 len 40960
14927 159/300 Write block 4682 len 28672 data: 4 4 4 5 5 4 4
14928 160/300 Write block 3579 len 36864 data: 2 2 3 4 6 7 7 6 6
14929 161/300 Read block 3093 len 24576
14930 162/300 Flush
14931 163/300 Read block 4148 len 4096
14932 164/300 Write block 2196 len 36864 data: 3 3 2 3 4 3 3 3 3
14933 165/300 Write block 4249 len 36864 data: 4 3 3 2 2 2 2 2 2
14934 166/300 Write block 607 len 16384 data: 3 4 4 4
14935 167/300 Write block 3722 len 40960 data: 4 4 3 4 3 3 4 4 3 4
14936 168/300 Write block 4553 len 16384 data: 4 5 5 6
14937 169/300 Read block 2162 len 36864
14938 170/300 Flush
14939 171/300 Read block 625 len 32768
14940 172/300 Flush
14941 173/300 Write block 1280 len 8192 data: 3 3
14942 174/300 Write block 4139 len 8192 data: 4 5
14943 175/300 Read block 2368 len 40960
14944 176/300 Read block 2655 len 36864
14945 177/300 Read block 2635 len 28672
14946 178/300 Read block 3800 len 36864
14947 179/300 Write block 1193 len 36864 data: 5 5 6 5 3 3 3 3 3
14948 180/300 Write block 4609 len 36864 data: 2 2 2 2 2 2 2 2 2
14949 181/300 Write block 2723 len 8192 data: 6 6
14950 182/300 Read block 273 len 32768
14951 183/300 Read block 3108 len 16384
14952 184/300 Read block 140 len 36864
14953 185/300 Read block 4340 len 20480
14954 186/300 Write block 4762 len 8192 data: 5 4
14955 187/300 Read block 1349 len 12288
14956 188/300 Write block 3996 len 8192 data: 3 3
14957 189/300 Read block 3812 len 24576
14958 190/300 Read block 2516 len 12288
14959 191/300 Write block 1761 len 24576 data: 5 5 5 4 4 4
14960 192/300 Read block 4673 len 28672
14961 193/300 Read block 427 len 12288
14962 194/300 Write block 3928 len 28672 data: 2 3 4 4 4 3 3
14963 195/300 Flush
14964 196/300 Read block 2662 len 4096
14965 197/300 Read block 4127 len 36864
14966 198/300 Write block 4582 len 8192 data: 4 4
14967 199/300 Flush
14968 200/300 Read block 2720 len 28672
14969 201/300 Write block 1821 len 20480 data: 4 4 3 3 3
14970 202/300 Flush
14971 203/300 Write block 3195 len 12288 data: 2 2 4
14972 204/300 Write block 934 len 24576 data: 3 3 3 2 2 2
14973 205/300 Read block 3315 len 32768
14974 206/300 Write block 1958 len 24576 data: 3 3 3 4 5 5
14975 207/300 Write block 2683 len 16384 data: 6 5 4 4
14976 208/300 Write block 4508 len 40960 data: 5 5 5 5 6 6 5 5 5 6
14977 209/300 Flush
14978 210/300 Write block 2365 len 28672 data: 4 4 4 5 7 7 7
14979 211/300 Write block 680 len 32768 data: 3 3 3 3 3 2 2 4
14980 212/300 Flush
14981 213/300 Flush
14982 214/300 Write block 2992 len 8192 data: 4 3
14983 215/300 Flush
14984 216/300 Write block 4438 len 8192 data: 4 4
14985 217/300 Read block 4940 len 28672
14986 218/300 Read block 2194 len 32768
14987 219/300 Read block 2344 len 28672
14988 220/300 Write block 2278 len 40960 data: 3 3 3 4 4 5 6 4 6 6
14989 221/300 Flush
14990 222/300 Read block 2173 len 16384
14991 223/300 Read block 2592 len 20480
14992 224/300 Read block 2617 len 4096
14993 225/300 Write block 2576 len 32768 data: 2 2 2 2 3 3 3 3
14994 226/300 Read block 637 len 40960
14995 227/300 Read block 3915 len 28672
14996 228/300 Write block 1120 len 16384 data: 4 3 2 2
14997 229/300 Read block 1696 len 16384
14998 230/300 Flush
14999 231/300 Write block 3316 len 4096 data: 2
15000 232/300 Flush
15001 233/300 Read block 1221 len 28672
15002 234/300 Read block 134 len 24576
15003 235/300 Flush
15004 236/300 Read block 1320 len 4096
15005 237/300 Read block 2776 len 36864
15006 238/300 Write block 2060 len 20480 data: 4 4 5 4 3
15007 239/300 Write block 3228 len 16384 data: 5 6 5 5
15008 240/300 Write block 4576 len 12288 data: 2 2 4
15009 241/300 Write block 2859 len 24576 data: 5 5 4 4 4 4
15010 242/300 Read block 881 len 24576
15011 243/300 Read block 3397 len 4096
15012 244/300 Read block 2897 len 28672
15013 245/300 Read block 3105 len 32768
15014 246/300 Read block 564 len 12288
15015 247/300 Flush
15016 248/300 Read block 927 len 32768
15017 249/300 Read block 2309 len 16384
15018 250/300 Write block 2705 len 24576 data: 3 3 2 2 2 4
15019 251/300 Flush
15020 252/300 Read block 466 len 4096
15021 253/300 Flush
15022 254/300 Read block 4627 len 8192
15023 255/300 Read block 2456 len 12288
15024 256/300 Flush
15025 257/300 Read block 2434 len 28672
15026 258/300 Write block 4202 len 20480 data: 3 3 3 4 4
15027 259/300 Flush
15028 260/300 Read block 3324 len 32768
15029 261/300 Write block 4392 len 36864 data: 3 3 2 2 2 2 2 2 2
15030 262/300 Write block 634 len 28672 data: 3 3 2 2 2 4 4
15031 263/300 Flush
15032 264/300 Read block 4215 len 12288
15033 265/300 Read block 102 len 4096
15034 266/300 Write block 3381 len 40960 data: 3 5 6 6 6 6 5 3 3 3
15035 267/300 Read block 4205 len 28672
15036 268/300 Read block 3217 len 8192
15037 269/300 Write block 2631 len 36864 data: 2 2 3 3 3 4 4 4 4
15038 270/300 Write block 1203 len 36864 data: 3 3 2 3 5 5 4 4 5
15039 271/300 Read block 4730 len 28672
15040 272/300 Read block 1194 len 40960
15041 273/300 Read block 3123 len 20480
15042 274/300 Read block 3278 len 12288
15043 275/300 Write block 991 len 8192 data: 2 2
15044 276/300 Write block 4809 len 28672 data: 4 4 4 4 5 5 5
15045 277/300 Write block 3018 len 24576 data: 3 3 3 2 2 2
15046 278/300 Write block 1595 len 12288 data: 4 4 5
15047 279/300 Read block 3072 len 36864
15048 280/300 Write block 2048 len 8192 data: 2 2
15049 281/300 Write block 2080 len 8192 data: 2 2
15050 282/300 Read block 2548 len 40960
15051 283/300 Write block 1971 len 16384 data: 7 6 6 6
15052 284/300 Read block 912 len 24576
15053 285/300 Read block 2031 len 8192
15054 286/300 Write block 3978 len 4096 data: 5
15055 287/300 Read block 2596 len 4096
15056 288/300 Read block 4422 len 40960
15057 289/300 Read block 1403 len 24576
15058 290/300 Write block 3483 len 12288 data: 4 3 4
15059 291/300 Read block 2461 len 36864
15060 292/300 Read block 4878 len 28672
15061 293/300 Write block 4030 len 20480 data: 4 5 4 6 5
15062 294/300 Read block 1110 len 40960
15063 295/300 Read block 4924 len 28672
15064 296/300 Write block 2786 len 28672 data: 4 4 4 3 3 4 6
15065 297/300 Read block 3521 len 4096
15066 298/300 Read block 1930 len 36864
15067 299/300 Read block 2659 len 8192
15068 300/300 Read block 2960 len 20480
15069 301/300 Read block 188 len 8192
15070 Main task has work to do, go find it
15071 got dsc Start(0)
15072 start 0
15073 Nov 01 16:53:40.409 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/0, method: POST, req_id: cc3abac1-8fc1-40e9-ab83-688a813928d3, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
15074 Replay: started 0, returned:Ok(())
15075 ----------------------------------------------------------------
15076 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
15077 States: Offline Active Active
15078 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15079 4928 Acked Write 1 Sent Done Done false
15080 4929 Acked Read 5 Sent Done Done false
15081 4930 Acked Read 7 Sent Done Done false
15082 4931 Acked Flush 0 Sent Done Done false
15083 4932 Acked Flush 0 Sent Done Done false
15084 4933 Acked Write 3 Sent Done Done false
15085 4934 Acked Read 6 Sent Done Done false
15086 4935 Acked Read 10 Sent Done Done false
15087 4936 Acked Read 4 Sent Done Done false
15088 4937 Acked Read 5 Sent Done Done false
15089 4938 Acked Read 6 Sent Done Done false
15090 4939 Acked Read 4 Sent Done Done false
15091 4940 Acked Read 9 Sent Done Done false
15092 4941 Acked Read 1 Sent Done Done false
15093 4942 Acked Read 7 Sent Done Done false
15094 4943 Acked Flush 0 Sent Done Done false
15095 4944 Acked Read 8 Sent Done Done false
15096 4945 Acked Write 9 Sent Done Done false
15097 4946 Acked Write 4 Sent Done Done false
15098 4947 Acked Write 3 Sent Done Done false
15099 4948 Acked Write 9 Sent Done Done false
15100 4949 Acked Write 1 Sent Done Done false
15101 4950 Acked Read 7 Sent Done Done false
15102 4951 Acked Read 2 Sent Done Done false
15103 4952 Acked Read 4 Sent Done Done false
15104 4953 Acked Read 8 Sent Done Done false
15105 4954 Acked Write 6 Sent Done Done false
15106 4955 Acked Flush 0 Sent Done Done false
15107 4956 Acked Read 9 Sent Done Done false
15108 4957 Acked Read 3 Sent Done Done false
15109 4958 Acked Write 1 Sent Done Done false
15110 4959 Acked Write 1 Sent Done Done false
15111 4960 Acked Read 1 Sent Done Done false
15112 4961 Acked Read 6 Sent Done Done false
15113 4962 Acked Write 6 Sent Done Done false
15114 4963 Acked Read 4 Sent Done Done false
15115 4964 Acked Write 10 Sent Done Done false
15116 4965 Acked Flush 0 Sent Done Done false
15117 4966 Acked Write 6 Sent Done Done false
15118 4967 Acked Read 1 Sent Done Done false
15119 4968 Acked Read 6 Sent Done Done false
15120 4969 Acked Write 2 Sent Done Done false
15121 4970 Acked Write 1 Sent Done Done false
15122 4971 Acked Read 2 Sent Done Done false
15123 4972 Acked Write 5 Sent Done Done false
15124 4973 Acked Read 4 Sent Done Done false
15125 4974 Acked Write 3 Sent Done Done false
15126 4975 Acked Read 5 Sent Done Done false
15127 4976 Acked Read 9 Sent Done Done false
15128 4977 Acked Read 7 Sent Done Done false
15129 4978 Acked Read 9 Sent Done Done false
15130 4979 Acked Write 4 Sent Done Done false
15131 4980 Acked Write 5 Sent Done Done false
15132 4981 Acked Write 4 Sent Done Done false
15133 4982 Acked Write 9 Sent Done Done false
15134 4983 Acked Write 7 Sent Done Done false
15135 4984 Acked Read 4 Sent Done Done false
15136 4985 Acked Read 9 Sent Done Done false
15137 4986 Acked Write 1 Sent Done Done false
15138 4987 Acked Read 1 Sent Done Done false
15139 4988 Acked Write 5 Sent Done Done false
15140 4989 Acked Read 10 Sent Done Done false
15141 4990 Acked Read 5 Sent Done Done false
15142 4991 Acked Write 8 Sent Done Done false
15143 4992 Acked Read 2 Sent Done Done false
15144 4993 Acked Read 1 Sent Done Done false
15145 4994 Acked Read 7 Sent Done Done false
15146 4995 Acked Write 9 Sent Done Done false
15147 4996 Acked Read 10 Sent Done Done false
15148 4997 Acked Write 6 Sent Done Done false
15149 4998 Acked Write 5 Sent Done Done false
15150 4999 Acked Flush 0 Sent Done Done false
15151 5000 Acked Write 1 Sent Done Done false
15152 5001 Acked Write 5 Sent Done Done false
15153 5002 Acked Read 5 Sent Done Done false
15154 5003 Acked Write 2 Sent Done Done false
15155 5004 Acked Read 1 Sent Done Done false
15156 5005 Acked Write 10 Sent Done Done false
15157 5006 Acked Write 4 Sent Done Done false
15158 5007 Acked Read 10 Sent Done Done false
15159 5008 Acked Read 4 Sent Done Done false
15160 5009 Acked Read 9 Sent Done Done false
15161 5010 Acked Read 1 Sent Done Done false
15162 5011 Acked Write 5 Sent Done Done false
15163 5012 Acked Read 5 Sent Done Done false
15164 5013 Acked Write 4 Sent Done Done false
15165 5014 Acked Write 9 Sent Done Done false
15166 5015 Acked Write 1 Sent Done Done false
15167 5016 Acked Flush 0 Sent Done Done false
15168 5017 Acked Write 7 Sent Done Done false
15169 5018 Acked Write 4 Sent Done Done false
15170 5019 Acked Read 3 Sent Done Done false
15171 5020 Acked Read 2 Sent Done Done false
15172 5021 Acked Read 3 Sent Done Done false
15173 5022 Acked Read 8 Sent Done Done false
15174 5023 Acked Read 6 Sent Done Done false
15175 5024 Acked Read 7 Sent Done Done false
15176 5025 Acked Read 7 Sent Done Done false
15177 5026 Acked Write 5 Sent Done Done false
15178 5027 Acked Write 8 Sent Done Done false
15179 5028 Acked Write 10 Sent Done Done false
15180 5029 Acked Flush 0 Sent Done Done false
15181 5030 Acked Write 6 Sent Done Done false
15182 5031 Acked Read 3 Sent Done Done false
15183 5032 Acked Write 4 Sent Done Done false
15184 5033 Acked Read 5 Sent Done Done false
15185 5034 Acked Write 9 Sent Done Done false
15186 5035 Acked Flush 0 Sent Done Done false
15187 5036 Acked Write 2 Sent Done Done false
15188 5037 Acked Write 8 Sent Done Done false
15189 5038 Acked Read 1 Sent Done Done false
15190 5039 Acked Read 7 Sent Done Done false
15191 5040 Acked Write 1 Sent Done Done false
15192 5041 Acked Read 4 Sent Done Done false
15193 5042 Acked Flush 0 Sent Done Done false
15194 5043 Acked Write 3 Sent Done Done false
15195 5044 Acked Flush 0 Sent Done Done false
15196 5045 Acked Read 4 Sent Done Done false
15197 5046 Acked Flush 0 Sent Done Done false
15198 5047 Acked Read 3 Sent Done Done false
15199 5048 Acked Write 2 Sent Done Done false
15200 5049 Acked Read 1 Sent Done Done false
15201 5050 Acked Read 5 Sent Done Done false
15202 5051 Acked Write 10 Sent Done Done false
15203 5052 Acked Read 3 Sent Done Done false
15204 5053 Acked Read 4 Sent Done Done false
15205 5054 Acked Read 8 Sent Done Done false
15206 5055 Acked Write 6 Sent Done Done false
15207 5056 Acked Write 6 Sent Done Done false
15208 5057 Acked Write 6 Sent Done Done false
15209 5058 Acked Read 4 Sent Done Done false
15210 5059 Acked Flush 0 Sent Done Done false
15211 5060 Acked Write 10 Sent Done Done false
15212 5061 Acked Read 9 Sent Done Done false
15213 5062 Acked Read 1 Sent Done Done false
15214 5063 Acked Read 4 Sent Done Done false
15215 5064 Acked Read 8 Sent Done Done false
15216 5065 Acked Read 8 Sent Done Done false
15217 5066 Acked Write 10 Sent Done Done false
15218 5067 Acked Read 2 Sent Done Done false
15219 5068 Acked Read 5 Sent Done Done false
15220 5069 Acked Write 7 Sent Done Done false
15221 5070 Acked Flush 0 Sent Done Done false
15222 5071 Acked Write 3 Sent Done Done false
15223 5072 Acked Write 4 Sent Done Done false
15224 5073 Acked Flush 0 Sent Done Done false
15225 5074 Acked Read 8 Sent Done Done false
15226 5075 Acked Write 4 Sent Done Done false
15227 5076 Acked Read 4 Sent Done Done false
15228 5077 Acked Write 1 Sent Done Done false
15229 5078 Acked Read 1 Sent Done Done false
15230 5079 Acked Read 8 Sent Done Done false
15231 5080 Acked Read 7 Sent Done Done false
15232 5081 Acked Write 6 Sent Done Done false
15233 5082 Acked Read 5 Sent Done Done false
15234 5083 Acked Read 1 Sent Done Done false
15235 5084 Acked Read 2 Sent Done Done false
15236 5085 Acked Read 4 Sent Done Done false
15237 5086 Acked Read 10 Sent Done Done false
15238 5087 Acked Write 7 Sent Done Done false
15239 5088 Acked Write 9 Sent Done Done false
15240 5089 Acked Read 6 Sent Done Done false
15241 5090 Acked Flush 0 Sent Done Done false
15242 5091 Acked Read 1 Sent Done Done false
15243 5092 Acked Write 9 Sent Done Done false
15244 5093 Acked Write 9 Sent Done Done false
15245 5094 Acked Write 4 Sent Done Done false
15246 5095 Acked Write 10 Sent Done Done false
15247 5096 Acked Write 4 Sent Done Done false
15248 5097 Acked Read 9 Sent Done Done false
15249 5098 Acked Flush 0 Sent Done Done false
15250 5099 Acked Read 8 Sent Done Done false
15251 5100 Acked Flush 0 Sent Done Done false
15252 5101 Acked Write 2 Sent Done Done false
15253 5102 Acked Write 2 Sent Done Done false
15254 5103 Acked Read 10 Sent Done Done false
15255 5104 Acked Read 9 Sent Done Done false
15256 5105 Acked Read 7 Sent Done Done false
15257 5106 Acked Read 9 Sent Done Done false
15258 5107 Acked Write 9 Sent Done Done false
15259 5108 Acked Write 9 Sent Done Done false
15260 5109 Acked Write 2 Sent Done Done false
15261 5110 Acked Read 8 Sent Done Done false
15262 5111 Acked Read 4 Sent Done Done false
15263 5112 Acked Read 9 Sent Done Done false
15264 5113 Acked Read 5 Sent Done Done false
15265 5114 Acked Write 2 Sent Done Done false
15266 5115 Acked Read 3 Sent Done Done false
15267 5116 Acked Write 2 Sent Done Done false
15268 5117 Acked Read 6 Sent Done Done false
15269 5118 Acked Read 3 Sent Done Done false
15270 5119 Acked Write 6 Sent Done Done false
15271 5120 Acked Read 7 Sent Done Done false
15272 5121 Acked Read 3 Sent Done Done false
15273 5122 Acked Write 7 Sent Done Done false
15274 5123 Acked Flush 0 Sent Done Done false
15275 5124 Acked Read 1 Sent Done Done false
15276 5125 Acked Read 9 Sent Done Done false
15277 5126 Acked Write 2 Sent Done Done false
15278 5127 Acked Flush 0 Sent Done Done false
15279 5128 Acked Read 7 Sent Done Done false
15280 5129 Acked Write 5 Sent Done Done false
15281 5130 Acked Flush 0 Sent Done Done false
15282 5131 Acked Write 3 Sent Done Done false
15283 5132 Acked Write 6 Sent Done Done false
15284 5133 Acked Read 8 Sent Done Done false
15285 5134 Acked Write 6 Sent Done Done false
15286 5135 Acked Write 4 Sent Done Done false
15287 5136 Acked Write 10 Sent Done Done false
15288 5137 Acked Flush 0 Sent Done Done false
15289 5138 Acked Write 7 Sent Done Done false
15290 5139 Acked Write 8 Sent Done Done false
15291 5140 Acked Flush 0 Sent Done Done false
15292 5141 Acked Flush 0 Sent Done Done false
15293 5142 Acked Write 2 Sent Done Done false
15294 5143 Acked Flush 0 Sent Done Done false
15295 5144 Acked Write 2 Sent Done Done false
15296 5145 Acked Read 7 Sent Done Done false
15297 5146 Acked Read 8 Sent Done Done false
15298 5147 Acked Read 7 Sent Done Done false
15299 5148 Acked Write 10 Sent Done Done false
15300 5149 Acked Flush 0 Sent Done Done false
15301 5150 Acked Read 4 Sent Done Done false
15302 5151 Acked Read 5 Sent Done Done false
15303 5152 Acked Read 1 Sent Done Done false
15304 5153 Acked Write 8 Sent Done Done false
15305 5154 Acked Read 10 Sent Done Done false
15306 5155 Acked Read 7 Sent Done Done false
15307 5156 Acked Write 4 Sent Done Done false
15308 5157 Acked Read 4 Sent Done Done false
15309 5158 Acked Flush 0 Sent Done Done false
15310 5159 Acked Write 1 Sent Done Done false
15311 5160 Acked Flush 0 Sent Done Done false
15312 5161 Acked Read 7 Sent Done Done false
15313 5162 Acked Read 6 Sent Done Done false
15314 5163 Acked Flush 0 Sent Done Done false
15315 5164 Acked Read 1 Sent Done Done false
15316 5165 Acked Read 9 Sent Done Done false
15317 5166 Acked Write 5 Sent Done Done false
15318 5167 Acked Write 4 Sent Done Done false
15319 5168 Acked Write 3 Sent Done Done false
15320 5169 Acked Write 6 Sent Done Done false
15321 5170 Acked Read 6 Sent Done Done false
15322 5171 Acked Read 1 Sent Done Done false
15323 5172 Acked Read 7 Sent Done Done false
15324 5173 Acked Read 8 Sent Done Done false
15325 5174 Acked Read 3 Sent Done Done false
15326 5175 Acked Flush 0 Sent Done Done false
15327 5176 Acked Read 8 Sent Done Done false
15328 5177 Acked Read 4 Sent Done Done false
15329 5178 Acked Write 6 Sent Done Done false
15330 5179 Acked Flush 0 Sent Done Done false
15331 5180 Acked Read 1 Sent Done Done false
15332 5181 Acked Flush 0 Sent Done Done false
15333 5182 Acked Read 2 Sent Done Done false
15334 5183 Acked Read 3 Sent Done Done false
15335 5184 Acked Flush 0 Sent Done Done false
15336 5185 Acked Read 7 Sent Done Done false
15337 5186 Acked Write 5 Sent Done Done false
15338 5187 Acked Flush 0 Sent Done Done false
15339 5188 Acked Read 8 Sent Done Done false
15340 5189 Acked Write 9 Sent Done Done false
15341 5190 Acked Write 7 Sent Done Done false
15342 5191 Acked Flush 0 Sent Done Done false
15343 5192 Acked Read 3 Sent Done Done false
15344 5193 Acked Read 1 Sent Done Done false
15345 5194 Acked Write 10 Sent Done Done false
15346 5195 Acked Read 7 Sent Done Done false
15347 5196 Acked Read 2 Sent Done Done false
15348 5197 Acked Write 9 Sent Done Done false
15349 5198 Acked Write 9 Sent Done Done false
15350 5199 Acked Read 7 Sent Done Done false
15351 5200 Acked Read 10 Sent Done Done false
15352 5201 Acked Read 5 Sent Done Done false
15353 5202 Acked Read 3 Sent Done Done false
15354 5203 Acked Write 2 Sent Done Done false
15355 5204 Acked Write 7 Sent Done Done false
15356 5205 Acked Write 6 Sent Done Done false
15357 5206 Acked Write 3 Sent Done Done false
15358 5207 Acked Read 9 Sent Done Done false
15359 5208 Acked Write 2 Sent Done Done false
15360 5209 Acked Write 2 Sent Done Done false
15361 5210 Acked Read 10 Sent Done Done false
15362 5211 Acked Write 4 Sent Done Done false
15363 5212 Acked Read 6 Sent Done Done false
15364 5213 Acked Read 2 Sent Done Done false
15365 5214 Acked Write 1 Sent Done Done false
15366 5215 Acked Read 1 Sent Done Done false
15367 5216 Acked Read 10 Sent Done Done false
15368 5217 Acked Read 6 Sent Done Done false
15369 5218 Acked Write 3 Sent Done Done false
15370 5219 Acked Read 9 Sent Done Done false
15371 5220 Acked Read 7 Sent Done Done false
15372 5221 Acked Write 5 Sent Done Done false
15373 5222 Acked Read 10 Sent Done Done false
15374 5223 Acked Read 7 Sent Done Done false
15375 5224 Acked Write 7 Sent Done Done false
15376 5225 Acked Read 1 Sent Done Done false
15377 5226 Acked Read 9 Sent Done Done false
15378 5227 Acked Read 2 Sent Done Done false
15379 5228 Acked Read 5 Sent Done Done false
15380 5229 Acked Read 2 Sent Done Done false
15381 STATES DS:0 DS:1 DS:2 TOTAL
15382 Sent 302 0 0 302
15383 Done 0 302 302 604
15384 Skipped 0 0 0 0
15385 Error 0 0 0 0
15386 Last Flush: 4927 5191 5191
15387 Downstairs last five completed: 4927 4926 4925 4924 4923
15388 Upstairs last five completed: 5229 5228 5227 5226 5225
15389 CLIENT: Up:0 ds:302 act:2
15390 [8810] got start action
15391 [8810] I am going to restart
15392 Starting downstairs at port 8810
15393 Make output file at "/tmp/dsc/downstairs-8810.txt"
15394 [8810][0] reports Starting
15395 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1212)
15396 [8810][0] reports Running
15397 ----------------------------------------------------------------
15398 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
15399 States: Offline Active Active
15400 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15401 4928 Acked Write 1 Sent Done Done false
15402 4929 Acked Read 5 Sent Done Done false
15403 4930 Acked Read 7 Sent Done Done false
15404 4931 Acked Flush 0 Sent Done Done false
15405 4932 Acked Flush 0 Sent Done Done false
15406 4933 Acked Write 3 Sent Done Done false
15407 4934 Acked Read 6 Sent Done Done false
15408 4935 Acked Read 10 Sent Done Done false
15409 4936 Acked Read 4 Sent Done Done false
15410 4937 Acked Read 5 Sent Done Done false
15411 4938 Acked Read 6 Sent Done Done false
15412 4939 Acked Read 4 Sent Done Done false
15413 4940 Acked Read 9 Sent Done Done false
15414 4941 Acked Read 1 Sent Done Done false
15415 4942 Acked Read 7 Sent Done Done false
15416 4943 Acked Flush 0 Sent Done Done false
15417 4944 Acked Read 8 Sent Done Done false
15418 4945 Acked Write 9 Sent Done Done false
15419 4946 Acked Write 4 Sent Done Done false
15420 4947 Acked Write 3 Sent Done Done false
15421 4948 Acked Write 9 Sent Done Done false
15422 4949 Acked Write 1 Sent Done Done false
15423 4950 Acked Read 7 Sent Done Done false
15424 4951 Acked Read 2 Sent Done Done false
15425 4952 Acked Read 4 Sent Done Done false
15426 4953 Acked Read 8 Sent Done Done false
15427 4954 Acked Write 6 Sent Done Done false
15428 4955 Acked Flush 0 Sent Done Done false
15429 4956 Acked Read 9 Sent Done Done false
15430 4957 Acked Read 3 Sent Done Done false
15431 4958 Acked Write 1 Sent Done Done false
15432 4959 Acked Write 1 Sent Done Done false
15433 4960 Acked Read 1 Sent Done Done false
15434 4961 Acked Read 6 Sent Done Done false
15435 4962 Acked Write 6 Sent Done Done false
15436 4963 Acked Read 4 Sent Done Done false
15437 4964 Acked Write 10 Sent Done Done false
15438 4965 Acked Flush 0 Sent Done Done false
15439 4966 Acked Write 6 Sent Done Done false
15440 4967 Acked Read 1 Sent Done Done false
15441 4968 Acked Read 6 Sent Done Done false
15442 4969 Acked Write 2 Sent Done Done false
15443 4970 Acked Write 1 Sent Done Done false
15444 4971 Acked Read 2 Sent Done Done false
15445 4972 Acked Write 5 Sent Done Done false
15446 4973 Acked Read 4 Sent Done Done false
15447 4974 Acked Write 3 Sent Done Done false
15448 4975 Acked Read 5 Sent Done Done false
15449 4976 Acked Read 9 Sent Done Done false
15450 4977 Acked Read 7 Sent Done Done false
15451 4978 Acked Read 9 Sent Done Done false
15452 4979 Acked Write 4 Sent Done Done false
15453 4980 Acked Write 5 Sent Done Done false
15454 4981 Acked Write 4 Sent Done Done false
15455 4982 Acked Write 9 Sent Done Done false
15456 4983 Acked Write 7 Sent Done Done false
15457 4984 Acked Read 4 Sent Done Done false
15458 4985 Acked Read 9 Sent Done Done false
15459 4986 Acked Write 1 Sent Done Done false
15460 4987 Acked Read 1 Sent Done Done false
15461 4988 Acked Write 5 Sent Done Done false
15462 4989 Acked Read 10 Sent Done Done false
15463 4990 Acked Read 5 Sent Done Done false
15464 4991 Acked Write 8 Sent Done Done false
15465 4992 Acked Read 2 Sent Done Done false
15466 4993 Acked Read 1 Sent Done Done false
15467 4994 Acked Read 7 Sent Done Done false
15468 4995 Acked Write 9 Sent Done Done false
15469 4996 Acked Read 10 Sent Done Done false
15470 4997 Acked Write 6 Sent Done Done false
15471 4998 Acked Write 5 Sent Done Done false
15472 4999 Acked Flush 0 Sent Done Done false
15473 5000 Acked Write 1 Sent Done Done false
15474 5001 Acked Write 5 Sent Done Done false
15475 5002 Acked Read 5 Sent Done Done false
15476 5003 Acked Write 2 Sent Done Done false
15477 5004 Acked Read 1 Sent Done Done false
15478 5005 Acked Write 10 Sent Done Done false
15479 5006 Acked Write 4 Sent Done Done false
15480 5007 Acked Read 10 Sent Done Done false
15481 5008 Acked Read 4 Sent Done Done false
15482 5009 Acked Read 9 Sent Done Done false
15483 5010 Acked Read 1 Sent Done Done false
15484 5011 Acked Write 5 Sent Done Done false
15485 5012 Acked Read 5 Sent Done Done false
15486 5013 Acked Write 4 Sent Done Done false
15487 5014 Acked Write 9 Sent Done Done false
15488 5015 Acked Write 1 Sent Done Done false
15489 5016 Acked Flush 0 Sent Done Done false
15490 5017 Acked Write 7 Sent Done Done false
15491 5018 Acked Write 4 Sent Done Done false
15492 5019 Acked Read 3 Sent Done Done false
15493 5020 Acked Read 2 Sent Done Done false
15494 5021 Acked Read 3 Sent Done Done false
15495 5022 Acked Read 8 Sent Done Done false
15496 5023 Acked Read 6 Sent Done Done false
15497 5024 Acked Read 7 Sent Done Done false
15498 5025 Acked Read 7 Sent Done Done false
15499 5026 Acked Write 5 Sent Done Done false
15500 5027 Acked Write 8 Sent Done Done false
15501 5028 Acked Write 10 Sent Done Done false
15502 5029 Acked Flush 0 Sent Done Done false
15503 5030 Acked Write 6 Sent Done Done false
15504 5031 Acked Read 3 Sent Done Done false
15505 5032 Acked Write 4 Sent Done Done false
15506 5033 Acked Read 5 Sent Done Done false
15507 5034 Acked Write 9 Sent Done Done false
15508 5035 Acked Flush 0 Sent Done Done false
15509 5036 Acked Write 2 Sent Done Done false
15510 5037 Acked Write 8 Sent Done Done false
15511 5038 Acked Read 1 Sent Done Done false
15512 5039 Acked Read 7 Sent Done Done false
15513 5040 Acked Write 1 Sent Done Done false
15514 5041 Acked Read 4 Sent Done Done false
15515 5042 Acked Flush 0 Sent Done Done false
15516 5043 Acked Write 3 Sent Done Done false
15517 5044 Acked Flush 0 Sent Done Done false
15518 5045 Acked Read 4 Sent Done Done false
15519 5046 Acked Flush 0 Sent Done Done false
15520 5047 Acked Read 3 Sent Done Done false
15521 5048 Acked Write 2 Sent Done Done false
15522 5049 Acked Read 1 Sent Done Done false
15523 5050 Acked Read 5 Sent Done Done false
15524 5051 Acked Write 10 Sent Done Done false
15525 5052 Acked Read 3 Sent Done Done false
15526 5053 Acked Read 4 Sent Done Done false
15527 5054 Acked Read 8 Sent Done Done false
15528 5055 Acked Write 6 Sent Done Done false
15529 5056 Acked Write 6 Sent Done Done false
15530 5057 Acked Write 6 Sent Done Done false
15531 5058 Acked Read 4 Sent Done Done false
15532 5059 Acked Flush 0 Sent Done Done false
15533 5060 Acked Write 10 Sent Done Done false
15534 5061 Acked Read 9 Sent Done Done false
15535 5062 Acked Read 1 Sent Done Done false
15536 5063 Acked Read 4 Sent Done Done false
15537 5064 Acked Read 8 Sent Done Done false
15538 5065 Acked Read 8 Sent Done Done false
15539 5066 Acked Write 10 Sent Done Done false
15540 5067 Acked Read 2 Sent Done Done false
15541 5068 Acked Read 5 Sent Done Done false
15542 5069 Acked Write 7 Sent Done Done false
15543 5070 Acked Flush 0 Sent Done Done false
15544 5071 Acked Write 3 Sent Done Done false
15545 5072 Acked Write 4 Sent Done Done false
15546 5073 Acked Flush 0 Sent Done Done false
15547 5074 Acked Read 8 Sent Done Done false
15548 5075 Acked Write 4 Sent Done Done false
15549 5076 Acked Read 4 Sent Done Done false
15550 5077 Acked Write 1 Sent Done Done false
15551 5078 Acked Read 1 Sent Done Done false
15552 5079 Acked Read 8 Sent Done Done false
15553 5080 Acked Read 7 Sent Done Done false
15554 5081 Acked Write 6 Sent Done Done false
15555 5082 Acked Read 5 Sent Done Done false
15556 5083 Acked Read 1 Sent Done Done false
15557 5084 Acked Read 2 Sent Done Done false
15558 5085 Acked Read 4 Sent Done Done false
15559 5086 Acked Read 10 Sent Done Done false
15560 5087 Acked Write 7 Sent Done Done false
15561 5088 Acked Write 9 Sent Done Done false
15562 5089 Acked Read 6 Sent Done Done false
15563 5090 Acked Flush 0 Sent Done Done false
15564 5091 Acked Read 1 Sent Done Done false
15565 5092 Acked Write 9 Sent Done Done false
15566 5093 Acked Write 9 Sent Done Done false
15567 5094 Acked Write 4 Sent Done Done false
15568 5095 Acked Write 10 Sent Done Done false
15569 5096 Acked Write 4 Sent Done Done false
15570 5097 Acked Read 9 Sent Done Done false
15571 5098 Acked Flush 0 Sent Done Done false
15572 5099 Acked Read 8 Sent Done Done false
15573 5100 Acked Flush 0 Sent Done Done false
15574 5101 Acked Write 2 Sent Done Done false
15575 5102 Acked Write 2 Sent Done Done false
15576 5103 Acked Read 10 Sent Done Done false
15577 5104 Acked Read 9 Sent Done Done false
15578 5105 Acked Read 7 Sent Done Done false
15579 5106 Acked Read 9 Sent Done Done false
15580 5107 Acked Write 9 Sent Done Done false
15581 5108 Acked Write 9 Sent Done Done false
15582 5109 Acked Write 2 Sent Done Done false
15583 5110 Acked Read 8 Sent Done Done false
15584 5111 Acked Read 4 Sent Done Done false
15585 5112 Acked Read 9 Sent Done Done false
15586 5113 Acked Read 5 Sent Done Done false
15587 5114 Acked Write 2 Sent Done Done false
15588 5115 Acked Read 3 Sent Done Done false
15589 5116 Acked Write 2 Sent Done Done false
15590 5117 Acked Read 6 Sent Done Done false
15591 5118 Acked Read 3 Sent Done Done false
15592 5119 Acked Write 6 Sent Done Done false
15593 5120 Acked Read 7 Sent Done Done false
15594 5121 Acked Read 3 Sent Done Done false
15595 5122 Acked Write 7 Sent Done Done false
15596 5123 Acked Flush 0 Sent Done Done false
15597 5124 Acked Read 1 Sent Done Done false
15598 5125 Acked Read 9 Sent Done Done false
15599 5126 Acked Write 2 Sent Done Done false
15600 5127 Acked Flush 0 Sent Done Done false
15601 5128 Acked Read 7 Sent Done Done false
15602 5129 Acked Write 5 Sent Done Done false
15603 5130 Acked Flush 0 Sent Done Done false
15604 5131 Acked Write 3 Sent Done Done false
15605 5132 Acked Write 6 Sent Done Done false
15606 5133 Acked Read 8 Sent Done Done false
15607 5134 Acked Write 6 Sent Done Done false
15608 5135 Acked Write 4 Sent Done Done false
15609 5136 Acked Write 10 Sent Done Done false
15610 5137 Acked Flush 0 Sent Done Done false
15611 5138 Acked Write 7 Sent Done Done false
15612 5139 Acked Write 8 Sent Done Done false
15613 5140 Acked Flush 0 Sent Done Done false
15614 5141 Acked Flush 0 Sent Done Done false
15615 5142 Acked Write 2 Sent Done Done false
15616 5143 Acked Flush 0 Sent Done Done false
15617 5144 Acked Write 2 Sent Done Done false
15618 5145 Acked Read 7 Sent Done Done false
15619 5146 Acked Read 8 Sent Done Done false
15620 5147 Acked Read 7 Sent Done Done false
15621 5148 Acked Write 10 Sent Done Done false
15622 5149 Acked Flush 0 Sent Done Done false
15623 5150 Acked Read 4 Sent Done Done false
15624 5151 Acked Read 5 Sent Done Done false
15625 5152 Acked Read 1 Sent Done Done false
15626 5153 Acked Write 8 Sent Done Done false
15627 5154 Acked Read 10 Sent Done Done false
15628 5155 Acked Read 7 Sent Done Done false
15629 5156 Acked Write 4 Sent Done Done false
15630 5157 Acked Read 4 Sent Done Done false
15631 5158 Acked Flush 0 Sent Done Done false
15632 5159 Acked Write 1 Sent Done Done false
15633 5160 Acked Flush 0 Sent Done Done false
15634 5161 Acked Read 7 Sent Done Done false
15635 5162 Acked Read 6 Sent Done Done false
15636 5163 Acked Flush 0 Sent Done Done false
15637 5164 Acked Read 1 Sent Done Done false
15638 5165 Acked Read 9 Sent Done Done false
15639 5166 Acked Write 5 Sent Done Done false
15640 5167 Acked Write 4 Sent Done Done false
15641 5168 Acked Write 3 Sent Done Done false
15642 5169 Acked Write 6 Sent Done Done false
15643 5170 Acked Read 6 Sent Done Done false
15644 5171 Acked Read 1 Sent Done Done false
15645 5172 Acked Read 7 Sent Done Done false
15646 5173 Acked Read 8 Sent Done Done false
15647 5174 Acked Read 3 Sent Done Done false
15648 5175 Acked Flush 0 Sent Done Done false
15649 5176 Acked Read 8 Sent Done Done false
15650 5177 Acked Read 4 Sent Done Done false
15651 5178 Acked Write 6 Sent Done Done false
15652 5179 Acked Flush 0 Sent Done Done false
15653 5180 Acked Read 1 Sent Done Done false
15654 5181 Acked Flush 0 Sent Done Done false
15655 5182 Acked Read 2 Sent Done Done false
15656 5183 Acked Read 3 Sent Done Done false
15657 5184 Acked Flush 0 Sent Done Done false
15658 5185 Acked Read 7 Sent Done Done false
15659 5186 Acked Write 5 Sent Done Done false
15660 5187 Acked Flush 0 Sent Done Done false
15661 5188 Acked Read 8 Sent Done Done false
15662 5189 Acked Write 9 Sent Done Done false
15663 5190 Acked Write 7 Sent Done Done false
15664 5191 Acked Flush 0 Sent Done Done false
15665 5192 Acked Read 3 Sent Done Done false
15666 5193 Acked Read 1 Sent Done Done false
15667 5194 Acked Write 10 Sent Done Done false
15668 5195 Acked Read 7 Sent Done Done false
15669 5196 Acked Read 2 Sent Done Done false
15670 5197 Acked Write 9 Sent Done Done false
15671 5198 Acked Write 9 Sent Done Done false
15672 5199 Acked Read 7 Sent Done Done false
15673 5200 Acked Read 10 Sent Done Done false
15674 5201 Acked Read 5 Sent Done Done false
15675 5202 Acked Read 3 Sent Done Done false
15676 5203 Acked Write 2 Sent Done Done false
15677 5204 Acked Write 7 Sent Done Done false
15678 5205 Acked Write 6 Sent Done Done false
15679 5206 Acked Write 3 Sent Done Done false
15680 5207 Acked Read 9 Sent Done Done false
15681 5208 Acked Write 2 Sent Done Done false
15682 5209 Acked Write 2 Sent Done Done false
15683 5210 Acked Read 10 Sent Done Done false
15684 5211 Acked Write 4 Sent Done Done false
15685 5212 Acked Read 6 Sent Done Done false
15686 5213 Acked Read 2 Sent Done Done false
15687 5214 Acked Write 1 Sent Done Done false
15688 5215 Acked Read 1 Sent Done Done false
15689 5216 Acked Read 10 Sent Done Done false
15690 5217 Acked Read 6 Sent Done Done false
15691 5218 Acked Write 3 Sent Done Done false
15692 5219 Acked Read 9 Sent Done Done false
15693 5220 Acked Read 7 Sent Done Done false
15694 5221 Acked Write 5 Sent Done Done false
15695 5222 Acked Read 10 Sent Done Done false
15696 5223 Acked Read 7 Sent Done Done false
15697 5224 Acked Write 7 Sent Done Done false
15698 5225 Acked Read 1 Sent Done Done false
15699 5226 Acked Read 9 Sent Done Done false
15700 5227 Acked Read 2 Sent Done Done false
15701 5228 Acked Read 5 Sent Done Done false
15702 5229 Acked Read 2 Sent Done Done false
15703 5230 Acked Flush 0 Sent Done Done false
15704 STATES DS:0 DS:1 DS:2 TOTAL
15705 Sent 303 0 0 303
15706 Done 0 303 303 606
15707 Skipped 0 0 0 0
15708 Error 0 0 0 0
15709 Last Flush: 4927 5230 5230
15710 Downstairs last five completed: 4927 4926 4925 4924 4923
15711 Upstairs last five completed: 5230 5229 5228 5227 5226
15712 CLIENT: Up:0 ds:303 act:2
157132024-11-01T16:53:46.122ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
157142024-11-01T16:53:46.122ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54521) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
157152024-11-01T16:53:46.123ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
157162024-11-01T16:53:46.124ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
157172024-11-01T16:53:46.124ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
157182024-11-01T16:53:46.124ZINFOcrucible: send last flush ID to this DS: 4927 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
157192024-11-01T16:53:46.124ZINFOcrucible: Replied this last flush ID: 4927 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
157202024-11-01T16:53:46.124ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
157212024-11-01T16:53:46.124ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
157222024-11-01T16:53:46.124ZINFOcrucible: [0] client re-new 303 jobs since flush 4927 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
157232024-11-01T16:53:46.124ZINFOcrucible: [0] Replayed 303 jobs since flush: 4927 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
15724 ----------------------------------------------------------------
15725 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15726 Downstairs last five completed: 5230 5229 5228 5227 5226
15727 Upstairs last five completed: 5230 5229 5228 5227 5226
15728 CLIENT: Up:0 ds:0 act:3
15729 Replay: All jobs finished
15730 Main task has work to do, go find it
15731 got dsc Stop(0)
15732 stop 0
15733 [8810] Got stop action so:false kr:false
15734 Nov 01 16:53:48.416 INFO request completed, latency_us: 336, response_code: 204, uri: /stop/cid/0, method: POST, req_id: f2d7ae67-f5c0-4db6-a889-9dbf1038a90f, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
157352024-11-01T16:53:48.417ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
157362024-11-01T16:53:48.417ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
157372024-11-01T16:53:48.417ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
157382024-11-01T16:53:48.417ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
15739 Nov 01 16:53:48.417 INFO request completed, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:53:48.41721691Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
15740 latency_us: 162, response_code: 200, uri: /state/cid/0, method: GET, req_id: f6672fb6-a8c0-480e-8583-b4f8f19c43a7, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
15741 [8810] Exited with: ExitStatus(unix_wait_status(9))
15742 [8810][0] reports Exit
15743 Nov 01 16:53:52.420 INFO request completed, latency_us: 296, response_code: 200, uri: /state/cid/0, method: GET, req_id: 737c6db0-9585-4427-8ff2-5bc2dc194dd0, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
15744 001/300 Flush
15745 002/300 Flush
15746 003/300 Read block 1811 len 28672
15747 004/300 Write block 2940 len 24576 data: 4 2 3 2 3 3
15748 005/300 Read block 142 len 8192
15749 006/300 Write block 3255 len 36864 data: 4 4 5 4 5 5 4 4 4
15750 007/300 Flush
15751 008/300 Read block 1868 len 16384
15752 009/300 Read block 4970 len 24576
15753 010/300 Read block 24 len 28672
15754 011/300 Read block 3954 len 20480
15755 012/300 Write block 1662 len 8192 data: 6 6
15756 013/300 Read block 1913 len 40960
15757 014/300 Write block 147 len 4096 data: 3
15758 015/300 Write block 3425 len 28672 data: 5 4 4 3 3 2 2
15759 016/300 Read block 3126 len 32768
15760 017/300 Read block 1665 len 24576
15761 018/300 Write block 4714 len 8192 data: 2 2
15762 019/300 Flush
15763 020/300 Read block 296 len 40960
15764 021/300 Write block 1520 len 28672 data: 2 2 2 3 4 4 4
15765 022/300 Read block 4060 len 12288
15766 023/300 Read block 1922 len 12288
15767 024/300 Write block 4828 len 4096 data: 3
15768 025/300 Read block 2782 len 8192
15769 026/300 Write block 1445 len 36864 data: 3 3 3 2 2 2 2 2 2
15770 027/300 Read block 788 len 28672
15771 028/300 Write block 4933 len 12288 data: 2 2 2
15772 029/300 Read block 4564 len 16384
15773 030/300 Read block 907 len 4096
15774 031/300 Write block 3314 len 40960 data: 2 2 3 2 2 2 3 3 4 4
15775 032/300 Read block 3343 len 36864
15776 033/300 Write block 1450 len 4096 data: 3
15777 034/300 Read block 4543 len 36864
15778 035/300 Read block 3534 len 40960
15779 036/300 Read block 110 len 20480
15780 037/300 Read block 3497 len 4096
15781 038/300 Write block 1594 len 36864 data: 5 5 5 6 4 4 4 4 4
15782 039/300 Write block 502 len 16384 data: 4 6 6 5
15783 040/300 Write block 38 len 24576 data: 3 3 4 4 3 4
15784 041/300 Read block 1669 len 16384
15785 042/300 Read block 1027 len 32768
15786 043/300 Read block 3842 len 20480
15787 044/300 Read block 502 len 40960
15788 045/300 Read block 3046 len 16384
15789 046/300 Write block 1379 len 8192 data: 2 2
15790 047/300 Flush
15791 048/300 Write block 832 len 4096 data: 3
15792 049/300 Write block 1510 len 4096 data: 6
15793 050/300 Read block 2617 len 20480
15794 051/300 Write block 1508 len 36864 data: 6 6 7 6 7 6 5 5 4
15795 052/300 Write block 2091 len 8192 data: 6 7
15796 053/300 Read block 4510 len 20480
15797 054/300 Read block 2715 len 12288
15798 055/300 Read block 2526 len 40960
15799 056/300 Flush
15800 057/300 Write block 2024 len 28672 data: 2 2 2 2 2 2 3
15801 058/300 Flush
15802 059/300 Flush
15803 060/300 Read block 1638 len 24576
15804 061/300 Read block 1344 len 16384
15805 062/300 Read block 4734 len 28672
15806 063/300 Read block 4944 len 4096
15807 064/300 Write block 1303 len 16384 data: 4 3 3 3
15808 065/300 Read block 3568 len 28672
15809 066/300 Write block 271 len 8192 data: 4 3
15810 067/300 Read block 1727 len 16384
15811 068/300 Flush
15812 069/300 Read block 3768 len 24576
15813 070/300 Read block 285 len 28672
15814 071/300 Read block 1731 len 28672
15815 072/300 Flush
15816 073/300 Write block 480 len 28672 data: 6 6 6 6 5 5 5
15817 074/300 Read block 3058 len 32768
15818 075/300 Read block 229 len 24576
15819 076/300 Write block 2383 len 4096 data: 2
15820 077/300 Write block 47 len 32768 data: 3 3 3 5 4 4 3 3
15821 078/300 Read block 4603 len 4096
15822 079/300 Read block 3923 len 4096
15823 080/300 Read block 4508 len 4096
15824 081/300 Read block 2513 len 16384
15825 082/300 Write block 3237 len 8192 data: 3 3
15826 083/300 Write block 4202 len 12288 data: 4 4 4
15827 084/300 Read block 1642 len 24576
15828 085/300 Read block 4883 len 16384
15829 086/300 Read block 3406 len 16384
15830 087/300 Write block 775 len 20480 data: 6 8 8 9 8
15831 088/300 Read block 4803 len 32768
15832 089/300 Read block 2889 len 16384
15833 090/300 Write block 90 len 8192 data: 6 6
15834 091/300 Write block 220 len 40960 data: 4 4 4 3 3 3 3 2 2 2
15835 092/300 Read block 577 len 32768
15836 093/300 Read block 4822 len 24576
15837 094/300 Write block 1551 len 16384 data: 3 2 2 2
15838 095/300 Read block 181 len 24576
15839 096/300 Write block 4858 len 8192 data: 2 2
15840 097/300 Read block 1884 len 40960
15841 098/300 Write block 3798 len 28672 data: 5 4 4 2 4 4 5
15842 099/300 Flush
15843 100/300 Read block 3284 len 20480
15844 101/300 Read block 4836 len 40960
15845 102/300 Write block 1943 len 40960 data: 5 5 6 6 5 6 6 5 4 4
15846 103/300 Read block 946 len 28672
15847 104/300 Read block 4664 len 12288
15848 105/300 Flush
15849 106/300 Read block 464 len 24576
15850 107/300 Read block 3049 len 12288
15851 108/300 Read block 1280 len 28672
15852 109/300 Read block 1194 len 36864
15853 110/300 Write block 1265 len 8192 data: 5 5
15854 111/300 Write block 2473 len 32768 data: 4 5 6 6 6 5 5 4
15855 112/300 Read block 3713 len 28672
15856 113/300 Write block 4373 len 20480 data: 4 3 3 5 6
15857 114/300 Read block 990 len 24576
15858 115/300 Write block 4420 len 8192 data: 3 2
15859 116/300 Write block 1125 len 24576 data: 2 2 2 2 2 2
15860 117/300 Read block 1105 len 32768
15861 118/300 Write block 3565 len 20480 data: 3 3 4 5 4
15862 119/300 Read block 259 len 16384
15863 120/300 Write block 1800 len 16384 data: 4 4 4 5
15864 121/300 Read block 4072 len 40960
15865 122/300 Write block 3190 len 16384 data: 4 3 2 2
15866 123/300 Read block 3492 len 20480
15867 124/300 Read block 3335 len 24576
15868 125/300 Write block 1843 len 36864 data: 4 4 4 3 3 3 3 3 2
15869 126/300 Read block 527 len 36864
15870 127/300 Write block 2487 len 32768 data: 5 5 5 5 5 4 4 4
15871 128/300 Write block 3081 len 4096 data: 4
15872 129/300 Flush
15873 130/300 Read block 1143 len 4096
15874 131/300 Flush
15875 132/300 Flush
15876 133/300 Write block 2071 len 28672 data: 3 3 4 4 3 3 3
15877 134/300 Read block 1174 len 32768
15878 135/300 Read block 2638 len 8192
15879 136/300 Read block 4908 len 40960
15880 137/300 Write block 1046 len 8192 data: 4 4
15881 138/300 Write block 425 len 8192 data: 2 2
15882 139/300 Read block 1118 len 28672
15883 140/300 Read block 2750 len 36864
15884 141/300 Write block 1053 len 8192 data: 4 3
15885 142/300 Read block 4114 len 32768
15886 143/300 Write block 4188 len 40960 data: 3 3 3 3 3 2 3 3 3 3
15887 144/300 Read block 3326 len 40960
15888 145/300 Write block 1131 len 32768 data: 2 3 3 2 2 3 3 4
15889 146/300 Read block 1463 len 16384
15890 147/300 Write block 940 len 32768 data: 2 2 2 3 4 4 4 4
15891 148/300 Read block 68 len 4096
15892 149/300 Read block 4986 len 12288
15893 150/300 Read block 207 len 28672
15894 151/300 Write block 3524 len 28672 data: 3 3 3 4 5 5 4
15895 152/300 Read block 4242 len 12288
15896 153/300 Write block 3163 len 12288 data: 3 3 3
15897 154/300 Read block 4358 len 20480
15898 155/300 Write block 2294 len 16384 data: 9 7 7 8
15899 156/300 Read block 1856 len 32768
15900 157/300 Read block 1802 len 16384
15901 158/300 Read block 1393 len 24576
15902 159/300 Read block 3682 len 24576
15903 160/300 Read block 19 len 20480
15904 161/300 Write block 2766 len 8192 data: 5 3
15905 162/300 Write block 3859 len 24576 data: 4 6 5 6 6 6
15906 163/300 Read block 3155 len 12288
15907 164/300 Read block 2013 len 32768
15908 165/300 Read block 450 len 40960
15909 166/300 Read block 3161 len 20480
15910 167/300 Read block 4551 len 8192
15911 168/300 Read block 113 len 4096
15912 169/300 Write block 456 len 20480 data: 5 6 6 4 4
15913 170/300 Read block 3125 len 24576
15914 171/300 Read block 3926 len 20480
15915 172/300 Read block 1395 len 28672
15916 173/300 Flush
15917 174/300 Read block 4480 len 8192
15918 175/300 Read block 1247 len 12288
15919 176/300 Read block 272 len 32768
15920 177/300 Read block 2451 len 20480
15921 178/300 Read block 2433 len 12288
15922 179/300 Write block 498 len 8192 data: 3 3
15923 180/300 Write block 4974 len 40960 data: 2 2 2 2 2 2 2 3 3 3
15924 181/300 Read block 2094 len 8192
15925 182/300 Read block 2763 len 24576
15926 183/300 Write block 1855 len 12288 data: 2 2 2
15927 184/300 Flush
15928 185/300 Write block 1057 len 40960 data: 4 4 3 3 2 2 3 3 4 4
15929 186/300 Read block 4690 len 20480
15930 187/300 Write block 3376 len 20480 data: 4 3 3 2 3
15931 188/300 Read block 3291 len 12288
15932 189/300 Flush
15933 190/300 Read block 708 len 12288
15934 191/300 Write block 1332 len 8192 data: 4 5
15935 192/300 Write block 4677 len 8192 data: 8 6
15936 193/300 Write block 989 len 32768 data: 3 3 3 3 3 4 4 5
15937 194/300 Write block 1591 len 4096 data: 5
15938 195/300 Read block 1359 len 32768
15939 196/300 Read block 3336 len 8192
15940 197/300 Read block 4132 len 12288
15941 198/300 Read block 3094 len 16384
15942 199/300 Flush
15943 200/300 Read block 4923 len 36864
15944 201/300 Write block 298 len 8192 data: 3 3
15945 202/300 Write block 317 len 8192 data: 2 2
15946 203/300 Write block 3707 len 16384 data: 4 3 3 3
15947 204/300 Read block 1489 len 12288
15948 205/300 Read block 719 len 40960
15949 206/300 Read block 1336 len 16384
15950 207/300 Write block 1848 len 28672 data: 4 4 4 3 3 3 3
15951 208/300 Read block 980 len 40960
15952 209/300 Flush
15953 210/300 Read block 4024 len 32768
15954 211/300 Read block 1068 len 24576
15955 212/300 Write block 1740 len 24576 data: 6 4 4 4 4 4
15956 213/300 Read block 4511 len 4096
15957 214/300 Read block 673 len 4096
15958 215/300 Read block 2512 len 32768
15959 216/300 Write block 70 len 28672 data: 4 5 5 4 4 5 5
15960 217/300 Flush
15961 218/300 Read block 1726 len 8192
15962 219/300 Write block 234 len 28672 data: 2 2 2 2 2 3 4
15963 220/300 Read block 13 len 20480
15964 221/300 Read block 2333 len 32768
15965 222/300 Read block 2462 len 32768
15966 223/300 Read block 4782 len 4096
15967 224/300 Write block 2091 len 20480 data: 7 8 6 6 6
15968 225/300 Write block 643 len 12288 data: 5 5 5
15969 226/300 Write block 2266 len 40960 data: 3 3 4 4 4 4 4 4 5 4
15970 227/300 Read block 4203 len 24576
15971 228/300 Read block 1446 len 40960
15972 229/300 Write block 1469 len 24576 data: 2 3 3 2 2 4
15973 230/300 Write block 3793 len 36864 data: 6 7 8 7 5 6 5 5 3
15974 231/300 Read block 4212 len 12288
15975 232/300 Read block 3976 len 36864
15976 233/300 Read block 2349 len 24576
15977 234/300 Read block 3472 len 12288
15978 235/300 Write block 430 len 32768 data: 2 2 2 3 3 3 5 4
15979 236/300 Read block 3032 len 20480
15980 237/300 Read block 360 len 4096
15981 238/300 Write block 414 len 8192 data: 4 4
15982 239/300 Write block 4146 len 24576 data: 4 4 3 3 3 3
15983 240/300 Write block 2500 len 32768 data: 3 3 3 3 3 3 2 2
15984 241/300 Read block 3071 len 28672
15985 242/300 Read block 2646 len 12288
15986 243/300 Write block 970 len 24576 data: 5 4 4 4 4 5
15987 244/300 Write block 1520 len 12288 data: 3 3 3
15988 245/300 Read block 3411 len 12288
15989 246/300 Flush
15990 247/300 Read block 1466 len 16384
15991 248/300 Flush
15992 249/300 Write block 4075 len 20480 data: 5 6 5 4 4
15993 250/300 Write block 1427 len 4096 data: 3
15994 251/300 Write block 1333 len 36864 data: 6 5 6 6 6 5 3 3 2
15995 252/300 Read block 396 len 36864
15996 253/300 Write block 2289 len 28672 data: 7 7 8 9 11 10 8
15997 254/300 Write block 2218 len 28672 data: 4 4 3 3 5 5 5
15998 255/300 Write block 4310 len 36864 data: 5 6 6 6 7 7 6 8 11
15999 256/300 Read block 4461 len 32768
16000 257/300 Read block 2980 len 8192
16001 258/300 Flush
16002 259/300 Read block 914 len 8192
16003 260/300 Flush
16004 261/300 Read block 3566 len 28672
16005 262/300 Read block 368 len 16384
16006 263/300 Write block 4196 len 36864 data: 4 4 3 3 4 3 5 5 5
16007 264/300 Read block 3682 len 36864
16008 265/300 Read block 1399 len 40960
16009 266/300 Write block 4921 len 36864 data: 6 5 6 6 6 6 4 4 3
16010 267/300 Write block 3527 len 28672 data: 5 6 6 5 3 3 3
16011 268/300 Read block 3097 len 16384
16012 269/300 Read block 291 len 40960
16013 270/300 Write block 4849 len 8192 data: 3 3
16014 271/300 Write block 2057 len 4096 data: 4
16015 272/300 Read block 1467 len 28672
16016 273/300 Read block 1379 len 20480
16017 274/300 Read block 4681 len 20480
16018 275/300 Write block 2982 len 16384 data: 4 4 4 4
16019 276/300 Write block 4142 len 20480 data: 5 5 4 4 5
16020 277/300 Read block 1623 len 20480
16021 278/300 Write block 1521 len 40960 data: 4 4 4 5 5 5 4 4 4 4
16022 279/300 Read block 3593 len 8192
16023 280/300 Write block 1221 len 28672 data: 3 2 3 3 3 2 2
16024 281/300 Write block 4772 len 16384 data: 4 4 4 5
16025 282/300 Read block 1287 len 36864
16026 283/300 Write block 2403 len 24576 data: 4 4 3 2 2 2
16027 284/300 Write block 4162 len 16384 data: 2 3 3 4
16028 285/300 Write block 3349 len 28672 data: 2 2 2 2 2 2 2
16029 286/300 Read block 2608 len 40960
16030 287/300 Read block 1823 len 16384
16031 288/300 Write block 3865 len 12288 data: 5 6 5
16032 289/300 Flush
16033 290/300 Read block 3420 len 40960
16034 291/300 Read block 4632 len 28672
16035 292/300 Read block 4354 len 16384
16036 293/300 Read block 2137 len 4096
16037 294/300 Read block 3589 len 40960
16038 295/300 Read block 4067 len 4096
16039 296/300 Read block 4312 len 12288
16040 297/300 Read block 1420 len 32768
16041 298/300 Write block 4046 len 20480 data: 2 2 2 2 4
16042 299/300 Flush
16043 300/300 Read block 1602 len 8192
16044 301/300 Read block 3032 len 8192
16045 Main task has work to do, go find it
16046 got dsc Start(0)
16047 start 0
16048 Nov 01 16:53:52.712 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/0, method: POST, req_id: 4b20967d-bb49-42fe-95f6-9a1f5eec687e, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
16049 Replay: started 0, returned:Ok(())
16050 ----------------------------------------------------------------
16051 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
16052 States: Offline Active Active
16053 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16054 5231 Acked Flush 0 Sent Done Done false
16055 5232 Acked Flush 0 Sent Done Done false
16056 5233 Acked Read 7 Sent Done Done false
16057 5234 Acked Write 6 Sent Done Done false
16058 5235 Acked Read 2 Sent Done Done false
16059 5236 Acked Write 9 Sent Done Done false
16060 5237 Acked Flush 0 Sent Done Done false
16061 5238 Acked Read 4 Sent Done Done false
16062 5239 Acked Read 6 Sent Done Done false
16063 5240 Acked Read 7 Sent Done Done false
16064 5241 Acked Read 5 Sent Done Done false
16065 5242 Acked Write 2 Sent Done Done false
16066 5243 Acked Read 10 Sent Done Done false
16067 5244 Acked Write 1 Sent Done Done false
16068 5245 Acked Write 7 Sent Done Done false
16069 5246 Acked Read 8 Sent Done Done false
16070 5247 Acked Read 6 Sent Done Done false
16071 5248 Acked Write 2 Sent Done Done false
16072 5249 Acked Flush 0 Sent Done Done false
16073 5250 Acked Read 10 Sent Done Done false
16074 5251 Acked Write 7 Sent Done Done false
16075 5252 Acked Read 3 Sent Done Done false
16076 5253 Acked Read 3 Sent Done Done false
16077 5254 Acked Write 1 Sent Done Done false
16078 5255 Acked Read 2 Sent Done Done false
16079 5256 Acked Write 9 Sent Done Done false
16080 5257 Acked Read 7 Sent Done Done false
16081 5258 Acked Write 3 Sent Done Done false
16082 5259 Acked Read 4 Sent Done Done false
16083 5260 Acked Read 1 Sent Done Done false
16084 5261 Acked Write 10 Sent Done Done false
16085 5262 Acked Read 9 Sent Done Done false
16086 5263 Acked Write 1 Sent Done Done false
16087 5264 Acked Read 9 Sent Done Done false
16088 5265 Acked Read 10 Sent Done Done false
16089 5266 Acked Read 5 Sent Done Done false
16090 5267 Acked Read 1 Sent Done Done false
16091 5268 Acked Write 9 Sent Done Done false
16092 5269 Acked Write 4 Sent Done Done false
16093 5270 Acked Write 6 Sent Done Done false
16094 5271 Acked Read 4 Sent Done Done false
16095 5272 Acked Read 8 Sent Done Done false
16096 5273 Acked Read 5 Sent Done Done false
16097 5274 Acked Read 10 Sent Done Done false
16098 5275 Acked Read 4 Sent Done Done false
16099 5276 Acked Write 2 Sent Done Done false
16100 5277 Acked Flush 0 Sent Done Done false
16101 5278 Acked Write 1 Sent Done Done false
16102 5279 Acked Write 1 Sent Done Done false
16103 5280 Acked Read 5 Sent Done Done false
16104 5281 Acked Write 9 Sent Done Done false
16105 5282 Acked Write 2 Sent Done Done false
16106 5283 Acked Read 5 Sent Done Done false
16107 5284 Acked Read 3 Sent Done Done false
16108 5285 Acked Read 10 Sent Done Done false
16109 5286 Acked Flush 0 Sent Done Done false
16110 5287 Acked Write 7 Sent Done Done false
16111 5288 Acked Flush 0 Sent Done Done false
16112 5289 Acked Flush 0 Sent Done Done false
16113 5290 Acked Read 6 Sent Done Done false
16114 5291 Acked Read 4 Sent Done Done false
16115 5292 Acked Read 7 Sent Done Done false
16116 5293 Acked Read 1 Sent Done Done false
16117 5294 Acked Write 4 Sent Done Done false
16118 5295 Acked Read 7 Sent Done Done false
16119 5296 Acked Write 2 Sent Done Done false
16120 5297 Acked Read 4 Sent Done Done false
16121 5298 Acked Flush 0 Sent Done Done false
16122 5299 Acked Read 6 Sent Done Done false
16123 5300 Acked Read 7 Sent Done Done false
16124 5301 Acked Read 7 Sent Done Done false
16125 5302 Acked Flush 0 Sent Done Done false
16126 5303 Acked Write 7 Sent Done Done false
16127 5304 Acked Read 8 Sent Done Done false
16128 5305 Acked Read 6 Sent Done Done false
16129 5306 Acked Write 1 Sent Done Done false
16130 5307 Acked Write 8 Sent Done Done false
16131 5308 Acked Read 1 Sent Done Done false
16132 5309 Acked Read 1 Sent Done Done false
16133 5310 Acked Read 1 Sent Done Done false
16134 5311 Acked Read 4 Sent Done Done false
16135 5312 Acked Write 2 Sent Done Done false
16136 5313 Acked Write 3 Sent Done Done false
16137 5314 Acked Read 6 Sent Done Done false
16138 5315 Acked Read 4 Sent Done Done false
16139 5316 Acked Read 4 Sent Done Done false
16140 5317 Acked Write 5 Sent Done Done false
16141 5318 Acked Read 8 Sent Done Done false
16142 5319 Acked Read 4 Sent Done Done false
16143 5320 Acked Write 2 Sent Done Done false
16144 5321 Acked Write 10 Sent Done Done false
16145 5322 Acked Read 8 Sent Done Done false
16146 5323 Acked Read 6 Sent Done Done false
16147 5324 Acked Write 4 Sent Done Done false
16148 5325 Acked Read 6 Sent Done Done false
16149 5326 Acked Write 2 Sent Done Done false
16150 5327 Acked Read 10 Sent Done Done false
16151 5328 Acked Write 7 Sent Done Done false
16152 5329 Acked Flush 0 Sent Done Done false
16153 5330 Acked Read 5 Sent Done Done false
16154 5331 Acked Read 10 Sent Done Done false
16155 5332 Acked Write 10 Sent Done Done false
16156 5333 Acked Read 7 Sent Done Done false
16157 5334 Acked Read 3 Sent Done Done false
16158 5335 Acked Flush 0 Sent Done Done false
16159 5336 Acked Read 6 Sent Done Done false
16160 5337 Acked Read 3 Sent Done Done false
16161 5338 Acked Read 7 Sent Done Done false
16162 5339 Acked Read 9 Sent Done Done false
16163 5340 Acked Write 2 Sent Done Done false
16164 5341 Acked Write 8 Sent Done Done false
16165 5342 Acked Read 7 Sent Done Done false
16166 5343 Acked Write 5 Sent Done Done false
16167 5344 Acked Read 6 Sent Done Done false
16168 5345 Acked Write 2 Sent Done Done false
16169 5346 Acked Write 6 Sent Done Done false
16170 5347 Acked Read 8 Sent Done Done false
16171 5348 Acked Write 5 Sent Done Done false
16172 5349 Acked Read 4 Sent Done Done false
16173 5350 Acked Write 4 Sent Done Done false
16174 5351 Acked Read 10 Sent Done Done false
16175 5352 Acked Write 4 Sent Done Done false
16176 5353 Acked Read 5 Sent Done Done false
16177 5354 Acked Read 6 Sent Done Done false
16178 5355 Acked Write 9 Sent Done Done false
16179 5356 Acked Read 9 Sent Done Done false
16180 5357 Acked Write 8 Sent Done Done false
16181 5358 Acked Write 1 Sent Done Done false
16182 5359 Acked Flush 0 Sent Done Done false
16183 5360 Acked Read 1 Sent Done Done false
16184 5361 Acked Flush 0 Sent Done Done false
16185 5362 Acked Flush 0 Sent Done Done false
16186 5363 Acked Write 7 Sent Done Done false
16187 5364 Acked Read 8 Sent Done Done false
16188 5365 Acked Read 2 Sent Done Done false
16189 5366 Acked Read 10 Sent Done Done false
16190 5367 Acked Write 2 Sent Done Done false
16191 5368 Acked Write 2 Sent Done Done false
16192 5369 Acked Read 7 Sent Done Done false
16193 5370 Acked Read 9 Sent Done Done false
16194 5371 Acked Write 2 Sent Done Done false
16195 5372 Acked Read 8 Sent Done Done false
16196 5373 Acked Write 10 Sent Done Done false
16197 5374 Acked Read 10 Sent Done Done false
16198 5375 Acked Write 8 Sent Done Done false
16199 5376 Acked Read 4 Sent Done Done false
16200 5377 Acked Write 8 Sent Done Done false
16201 5378 Acked Read 1 Sent Done Done false
16202 5379 Acked Read 3 Sent Done Done false
16203 5380 Acked Read 7 Sent Done Done false
16204 5381 Acked Write 7 Sent Done Done false
16205 5382 Acked Read 3 Sent Done Done false
16206 5383 Acked Write 3 Sent Done Done false
16207 5384 Acked Read 5 Sent Done Done false
16208 5385 Acked Write 4 Sent Done Done false
16209 5386 Acked Read 8 Sent Done Done false
16210 5387 Acked Read 4 Sent Done Done false
16211 5388 Acked Read 6 Sent Done Done false
16212 5389 Acked Read 6 Sent Done Done false
16213 5390 Acked Read 5 Sent Done Done false
16214 5391 Acked Write 2 Sent Done Done false
16215 5392 Acked Write 6 Sent Done Done false
16216 5393 Acked Read 3 Sent Done Done false
16217 5394 Acked Read 8 Sent Done Done false
16218 5395 Acked Read 10 Sent Done Done false
16219 5396 Acked Read 5 Sent Done Done false
16220 5397 Acked Read 2 Sent Done Done false
16221 5398 Acked Read 1 Sent Done Done false
16222 5399 Acked Write 5 Sent Done Done false
16223 5400 Acked Read 6 Sent Done Done false
16224 5401 Acked Read 5 Sent Done Done false
16225 5402 Acked Read 7 Sent Done Done false
16226 5403 Acked Flush 0 Sent Done Done false
16227 5404 Acked Read 2 Sent Done Done false
16228 5405 Acked Read 3 Sent Done Done false
16229 5406 Acked Read 8 Sent Done Done false
16230 5407 Acked Read 5 Sent Done Done false
16231 5408 Acked Read 3 Sent Done Done false
16232 5409 Acked Write 2 Sent Done Done false
16233 5410 Acked Write 10 Sent Done Done false
16234 5411 Acked Read 2 Sent Done Done false
16235 5412 Acked Read 6 Sent Done Done false
16236 5413 Acked Write 3 Sent Done Done false
16237 5414 Acked Flush 0 Sent Done Done false
16238 5415 Acked Write 10 Sent Done Done false
16239 5416 Acked Read 5 Sent Done Done false
16240 5417 Acked Write 5 Sent Done Done false
16241 5418 Acked Read 3 Sent Done Done false
16242 5419 Acked Flush 0 Sent Done Done false
16243 5420 Acked Read 3 Sent Done Done false
16244 5421 Acked Write 2 Sent Done Done false
16245 5422 Acked Write 2 Sent Done Done false
16246 5423 Acked Write 8 Sent Done Done false
16247 5424 Acked Write 1 Sent Done Done false
16248 5425 Acked Read 8 Sent Done Done false
16249 5426 Acked Read 2 Sent Done Done false
16250 5427 Acked Read 3 Sent Done Done false
16251 5428 Acked Read 4 Sent Done Done false
16252 5429 Acked Flush 0 Sent Done Done false
16253 5430 Acked Read 9 Sent Done Done false
16254 5431 Acked Write 2 Sent Done Done false
16255 5432 Acked Write 2 Sent Done Done false
16256 5433 Acked Write 4 Sent Done Done false
16257 5434 Acked Read 3 Sent Done Done false
16258 5435 Acked Read 10 Sent Done Done false
16259 5436 Acked Read 4 Sent Done Done false
16260 5437 Acked Write 7 Sent Done Done false
16261 5438 Acked Read 10 Sent Done Done false
16262 5439 Acked Flush 0 Sent Done Done false
16263 5440 Acked Read 8 Sent Done Done false
16264 5441 Acked Read 6 Sent Done Done false
16265 5442 Acked Write 6 Sent Done Done false
16266 5443 Acked Read 1 Sent Done Done false
16267 5444 Acked Read 1 Sent Done Done false
16268 5445 Acked Read 8 Sent Done Done false
16269 5446 Acked Write 7 Sent Done Done false
16270 5447 Acked Flush 0 Sent Done Done false
16271 5448 Acked Read 2 Sent Done Done false
16272 5449 Acked Write 7 Sent Done Done false
16273 5450 Acked Read 5 Sent Done Done false
16274 5451 Acked Read 8 Sent Done Done false
16275 5452 Acked Read 8 Sent Done Done false
16276 5453 Acked Read 1 Sent Done Done false
16277 5454 Acked Write 5 Sent Done Done false
16278 5455 Acked Write 3 Sent Done Done false
16279 5456 Acked Write 10 Sent Done Done false
16280 5457 Acked Read 6 Sent Done Done false
16281 5458 Acked Read 10 Sent Done Done false
16282 5459 Acked Write 6 Sent Done Done false
16283 5460 Acked Write 9 Sent Done Done false
16284 5461 Acked Read 3 Sent Done Done false
16285 5462 Acked Read 9 Sent Done Done false
16286 5463 Acked Read 6 Sent Done Done false
16287 5464 Acked Read 3 Sent Done Done false
16288 5465 Acked Write 8 Sent Done Done false
16289 5466 Acked Read 5 Sent Done Done false
16290 5467 Acked Read 1 Sent Done Done false
16291 5468 Acked Write 2 Sent Done Done false
16292 5469 Acked Write 6 Sent Done Done false
16293 5470 Acked Write 8 Sent Done Done false
16294 5471 Acked Read 7 Sent Done Done false
16295 5472 Acked Read 3 Sent Done Done false
16296 5473 Acked Write 6 Sent Done Done false
16297 5474 Acked Write 3 Sent Done Done false
16298 5475 Acked Read 3 Sent Done Done false
16299 5476 Acked Flush 0 Sent Done Done false
16300 5477 Acked Read 4 Sent Done Done false
16301 5478 Acked Flush 0 Sent Done Done false
16302 5479 Acked Write 5 Sent Done Done false
16303 5480 Acked Write 1 Sent Done Done false
16304 5481 Acked Write 9 Sent Done Done false
16305 5482 Acked Read 9 Sent Done Done false
16306 5483 Acked Write 7 Sent Done Done false
16307 5484 Acked Write 7 Sent Done Done false
16308 5485 Acked Write 9 Sent Done Done false
16309 5486 Acked Read 8 Sent Done Done false
16310 5487 Acked Flush 0 Sent Done Done false
16311 5488 Acked Read 2 Sent Done Done false
16312 5489 Acked Flush 0 Sent Done Done false
16313 5490 Acked Read 2 Sent Done Done false
16314 5491 Acked Flush 0 Sent Done Done false
16315 5492 Acked Read 7 Sent Done Done false
16316 5493 Acked Read 4 Sent Done Done false
16317 5494 Acked Write 9 Sent Done Done false
16318 5495 Acked Read 9 Sent Done Done false
16319 5496 Acked Read 10 Sent Done Done false
16320 5497 Acked Write 9 Sent Done Done false
16321 5498 Acked Write 7 Sent Done Done false
16322 5499 Acked Read 4 Sent Done Done false
16323 5500 Acked Read 10 Sent Done Done false
16324 5501 Acked Write 2 Sent Done Done false
16325 5502 Acked Write 1 Sent Done Done false
16326 5503 Acked Read 7 Sent Done Done false
16327 5504 Acked Read 5 Sent Done Done false
16328 5505 Acked Read 5 Sent Done Done false
16329 5506 Acked Write 4 Sent Done Done false
16330 5507 Acked Write 5 Sent Done Done false
16331 5508 Acked Read 5 Sent Done Done false
16332 5509 Acked Write 10 Sent Done Done false
16333 5510 Acked Read 2 Sent Done Done false
16334 5511 Acked Write 7 Sent Done Done false
16335 5512 Acked Write 4 Sent Done Done false
16336 5513 Acked Read 9 Sent Done Done false
16337 5514 Acked Write 6 Sent Done Done false
16338 5515 Acked Write 4 Sent Done Done false
16339 5516 Acked Write 7 Sent Done Done false
16340 5517 Acked Read 10 Sent Done Done false
16341 5518 Acked Read 4 Sent Done Done false
16342 5519 Acked Write 3 Sent Done Done false
16343 5520 Acked Flush 0 Sent Done Done false
16344 5521 Acked Read 10 Sent Done Done false
16345 5522 Acked Read 7 Sent Done Done false
16346 5523 Acked Read 4 Sent Done Done false
16347 5524 Acked Read 1 Sent Done Done false
16348 5525 Acked Read 10 Sent Done Done false
16349 5526 Acked Read 1 Sent Done Done false
16350 5527 Acked Read 3 Sent Done Done false
16351 5528 Acked Read 8 Sent Done Done false
16352 5529 Acked Write 5 Sent Done Done false
16353 5530 Acked Flush 0 Sent Done Done false
16354 5531 Acked Read 2 Sent Done Done false
16355 5532 Acked Read 2 Sent Done Done false
16356 STATES DS:0 DS:1 DS:2 TOTAL
16357 Sent 302 0 0 302
16358 Done 0 302 302 604
16359 Skipped 0 0 0 0
16360 Error 0 0 0 0
16361 Last Flush: 5230 5530 5530
16362 Downstairs last five completed: 5230 5229 5228 5227 5226
16363 Upstairs last five completed: 5532 5531 5530 5529 5528
16364 CLIENT: Up:0 ds:302 act:2
16365 [8810] got start action
16366 [8810] I am going to restart
16367 Starting downstairs at port 8810
16368 Make output file at "/tmp/dsc/downstairs-8810.txt"
16369 [8810][0] reports Starting
16370 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1213)
16371 [8810][0] reports Running
16372 ----------------------------------------------------------------
16373 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
16374 States: Offline Active Active
16375 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16376 5231 Acked Flush 0 Sent Done Done false
16377 5232 Acked Flush 0 Sent Done Done false
16378 5233 Acked Read 7 Sent Done Done false
16379 5234 Acked Write 6 Sent Done Done false
16380 5235 Acked Read 2 Sent Done Done false
16381 5236 Acked Write 9 Sent Done Done false
16382 5237 Acked Flush 0 Sent Done Done false
16383 5238 Acked Read 4 Sent Done Done false
16384 5239 Acked Read 6 Sent Done Done false
16385 5240 Acked Read 7 Sent Done Done false
16386 5241 Acked Read 5 Sent Done Done false
16387 5242 Acked Write 2 Sent Done Done false
16388 5243 Acked Read 10 Sent Done Done false
16389 5244 Acked Write 1 Sent Done Done false
16390 5245 Acked Write 7 Sent Done Done false
16391 5246 Acked Read 8 Sent Done Done false
16392 5247 Acked Read 6 Sent Done Done false
16393 5248 Acked Write 2 Sent Done Done false
16394 5249 Acked Flush 0 Sent Done Done false
16395 5250 Acked Read 10 Sent Done Done false
16396 5251 Acked Write 7 Sent Done Done false
16397 5252 Acked Read 3 Sent Done Done false
16398 5253 Acked Read 3 Sent Done Done false
16399 5254 Acked Write 1 Sent Done Done false
16400 5255 Acked Read 2 Sent Done Done false
16401 5256 Acked Write 9 Sent Done Done false
16402 5257 Acked Read 7 Sent Done Done false
16403 5258 Acked Write 3 Sent Done Done false
16404 5259 Acked Read 4 Sent Done Done false
16405 5260 Acked Read 1 Sent Done Done false
16406 5261 Acked Write 10 Sent Done Done false
16407 5262 Acked Read 9 Sent Done Done false
16408 5263 Acked Write 1 Sent Done Done false
16409 5264 Acked Read 9 Sent Done Done false
16410 5265 Acked Read 10 Sent Done Done false
16411 5266 Acked Read 5 Sent Done Done false
16412 5267 Acked Read 1 Sent Done Done false
16413 5268 Acked Write 9 Sent Done Done false
16414 5269 Acked Write 4 Sent Done Done false
16415 5270 Acked Write 6 Sent Done Done false
16416 5271 Acked Read 4 Sent Done Done false
16417 5272 Acked Read 8 Sent Done Done false
16418 5273 Acked Read 5 Sent Done Done false
16419 5274 Acked Read 10 Sent Done Done false
16420 5275 Acked Read 4 Sent Done Done false
16421 5276 Acked Write 2 Sent Done Done false
16422 5277 Acked Flush 0 Sent Done Done false
16423 5278 Acked Write 1 Sent Done Done false
16424 5279 Acked Write 1 Sent Done Done false
16425 5280 Acked Read 5 Sent Done Done false
16426 5281 Acked Write 9 Sent Done Done false
16427 5282 Acked Write 2 Sent Done Done false
16428 5283 Acked Read 5 Sent Done Done false
16429 5284 Acked Read 3 Sent Done Done false
16430 5285 Acked Read 10 Sent Done Done false
16431 5286 Acked Flush 0 Sent Done Done false
16432 5287 Acked Write 7 Sent Done Done false
16433 5288 Acked Flush 0 Sent Done Done false
16434 5289 Acked Flush 0 Sent Done Done false
16435 5290 Acked Read 6 Sent Done Done false
16436 5291 Acked Read 4 Sent Done Done false
16437 5292 Acked Read 7 Sent Done Done false
16438 5293 Acked Read 1 Sent Done Done false
16439 5294 Acked Write 4 Sent Done Done false
16440 5295 Acked Read 7 Sent Done Done false
16441 5296 Acked Write 2 Sent Done Done false
16442 5297 Acked Read 4 Sent Done Done false
16443 5298 Acked Flush 0 Sent Done Done false
16444 5299 Acked Read 6 Sent Done Done false
16445 5300 Acked Read 7 Sent Done Done false
16446 5301 Acked Read 7 Sent Done Done false
16447 5302 Acked Flush 0 Sent Done Done false
16448 5303 Acked Write 7 Sent Done Done false
16449 5304 Acked Read 8 Sent Done Done false
16450 5305 Acked Read 6 Sent Done Done false
16451 5306 Acked Write 1 Sent Done Done false
16452 5307 Acked Write 8 Sent Done Done false
16453 5308 Acked Read 1 Sent Done Done false
16454 5309 Acked Read 1 Sent Done Done false
16455 5310 Acked Read 1 Sent Done Done false
16456 5311 Acked Read 4 Sent Done Done false
16457 5312 Acked Write 2 Sent Done Done false
16458 5313 Acked Write 3 Sent Done Done false
16459 5314 Acked Read 6 Sent Done Done false
16460 5315 Acked Read 4 Sent Done Done false
16461 5316 Acked Read 4 Sent Done Done false
16462 5317 Acked Write 5 Sent Done Done false
16463 5318 Acked Read 8 Sent Done Done false
16464 5319 Acked Read 4 Sent Done Done false
16465 5320 Acked Write 2 Sent Done Done false
16466 5321 Acked Write 10 Sent Done Done false
16467 5322 Acked Read 8 Sent Done Done false
16468 5323 Acked Read 6 Sent Done Done false
16469 5324 Acked Write 4 Sent Done Done false
16470 5325 Acked Read 6 Sent Done Done false
16471 5326 Acked Write 2 Sent Done Done false
16472 5327 Acked Read 10 Sent Done Done false
16473 5328 Acked Write 7 Sent Done Done false
16474 5329 Acked Flush 0 Sent Done Done false
16475 5330 Acked Read 5 Sent Done Done false
16476 5331 Acked Read 10 Sent Done Done false
16477 5332 Acked Write 10 Sent Done Done false
16478 5333 Acked Read 7 Sent Done Done false
16479 5334 Acked Read 3 Sent Done Done false
16480 5335 Acked Flush 0 Sent Done Done false
16481 5336 Acked Read 6 Sent Done Done false
16482 5337 Acked Read 3 Sent Done Done false
16483 5338 Acked Read 7 Sent Done Done false
16484 5339 Acked Read 9 Sent Done Done false
16485 5340 Acked Write 2 Sent Done Done false
16486 5341 Acked Write 8 Sent Done Done false
16487 5342 Acked Read 7 Sent Done Done false
16488 5343 Acked Write 5 Sent Done Done false
16489 5344 Acked Read 6 Sent Done Done false
16490 5345 Acked Write 2 Sent Done Done false
16491 5346 Acked Write 6 Sent Done Done false
16492 5347 Acked Read 8 Sent Done Done false
16493 5348 Acked Write 5 Sent Done Done false
16494 5349 Acked Read 4 Sent Done Done false
16495 5350 Acked Write 4 Sent Done Done false
16496 5351 Acked Read 10 Sent Done Done false
16497 5352 Acked Write 4 Sent Done Done false
16498 5353 Acked Read 5 Sent Done Done false
16499 5354 Acked Read 6 Sent Done Done false
16500 5355 Acked Write 9 Sent Done Done false
16501 5356 Acked Read 9 Sent Done Done false
16502 5357 Acked Write 8 Sent Done Done false
16503 5358 Acked Write 1 Sent Done Done false
16504 5359 Acked Flush 0 Sent Done Done false
16505 5360 Acked Read 1 Sent Done Done false
16506 5361 Acked Flush 0 Sent Done Done false
16507 5362 Acked Flush 0 Sent Done Done false
16508 5363 Acked Write 7 Sent Done Done false
16509 5364 Acked Read 8 Sent Done Done false
16510 5365 Acked Read 2 Sent Done Done false
16511 5366 Acked Read 10 Sent Done Done false
16512 5367 Acked Write 2 Sent Done Done false
16513 5368 Acked Write 2 Sent Done Done false
16514 5369 Acked Read 7 Sent Done Done false
16515 5370 Acked Read 9 Sent Done Done false
16516 5371 Acked Write 2 Sent Done Done false
16517 5372 Acked Read 8 Sent Done Done false
16518 5373 Acked Write 10 Sent Done Done false
16519 5374 Acked Read 10 Sent Done Done false
16520 5375 Acked Write 8 Sent Done Done false
16521 5376 Acked Read 4 Sent Done Done false
16522 5377 Acked Write 8 Sent Done Done false
16523 5378 Acked Read 1 Sent Done Done false
16524 5379 Acked Read 3 Sent Done Done false
16525 5380 Acked Read 7 Sent Done Done false
16526 5381 Acked Write 7 Sent Done Done false
16527 5382 Acked Read 3 Sent Done Done false
16528 5383 Acked Write 3 Sent Done Done false
16529 5384 Acked Read 5 Sent Done Done false
16530 5385 Acked Write 4 Sent Done Done false
16531 5386 Acked Read 8 Sent Done Done false
16532 5387 Acked Read 4 Sent Done Done false
16533 5388 Acked Read 6 Sent Done Done false
16534 5389 Acked Read 6 Sent Done Done false
16535 5390 Acked Read 5 Sent Done Done false
16536 5391 Acked Write 2 Sent Done Done false
16537 5392 Acked Write 6 Sent Done Done false
16538 5393 Acked Read 3 Sent Done Done false
16539 5394 Acked Read 8 Sent Done Done false
16540 5395 Acked Read 10 Sent Done Done false
16541 5396 Acked Read 5 Sent Done Done false
16542 5397 Acked Read 2 Sent Done Done false
16543 5398 Acked Read 1 Sent Done Done false
16544 5399 Acked Write 5 Sent Done Done false
16545 5400 Acked Read 6 Sent Done Done false
16546 5401 Acked Read 5 Sent Done Done false
16547 5402 Acked Read 7 Sent Done Done false
16548 5403 Acked Flush 0 Sent Done Done false
16549 5404 Acked Read 2 Sent Done Done false
16550 5405 Acked Read 3 Sent Done Done false
16551 5406 Acked Read 8 Sent Done Done false
16552 5407 Acked Read 5 Sent Done Done false
16553 5408 Acked Read 3 Sent Done Done false
16554 5409 Acked Write 2 Sent Done Done false
16555 5410 Acked Write 10 Sent Done Done false
16556 5411 Acked Read 2 Sent Done Done false
16557 5412 Acked Read 6 Sent Done Done false
16558 5413 Acked Write 3 Sent Done Done false
16559 5414 Acked Flush 0 Sent Done Done false
16560 5415 Acked Write 10 Sent Done Done false
16561 5416 Acked Read 5 Sent Done Done false
16562 5417 Acked Write 5 Sent Done Done false
16563 5418 Acked Read 3 Sent Done Done false
16564 5419 Acked Flush 0 Sent Done Done false
16565 5420 Acked Read 3 Sent Done Done false
16566 5421 Acked Write 2 Sent Done Done false
16567 5422 Acked Write 2 Sent Done Done false
16568 5423 Acked Write 8 Sent Done Done false
16569 5424 Acked Write 1 Sent Done Done false
16570 5425 Acked Read 8 Sent Done Done false
16571 5426 Acked Read 2 Sent Done Done false
16572 5427 Acked Read 3 Sent Done Done false
16573 5428 Acked Read 4 Sent Done Done false
16574 5429 Acked Flush 0 Sent Done Done false
16575 5430 Acked Read 9 Sent Done Done false
16576 5431 Acked Write 2 Sent Done Done false
16577 5432 Acked Write 2 Sent Done Done false
16578 5433 Acked Write 4 Sent Done Done false
16579 5434 Acked Read 3 Sent Done Done false
16580 5435 Acked Read 10 Sent Done Done false
16581 5436 Acked Read 4 Sent Done Done false
16582 5437 Acked Write 7 Sent Done Done false
16583 5438 Acked Read 10 Sent Done Done false
16584 5439 Acked Flush 0 Sent Done Done false
16585 5440 Acked Read 8 Sent Done Done false
16586 5441 Acked Read 6 Sent Done Done false
16587 5442 Acked Write 6 Sent Done Done false
16588 5443 Acked Read 1 Sent Done Done false
16589 5444 Acked Read 1 Sent Done Done false
16590 5445 Acked Read 8 Sent Done Done false
16591 5446 Acked Write 7 Sent Done Done false
16592 5447 Acked Flush 0 Sent Done Done false
16593 5448 Acked Read 2 Sent Done Done false
16594 5449 Acked Write 7 Sent Done Done false
16595 5450 Acked Read 5 Sent Done Done false
16596 5451 Acked Read 8 Sent Done Done false
16597 5452 Acked Read 8 Sent Done Done false
16598 5453 Acked Read 1 Sent Done Done false
16599 5454 Acked Write 5 Sent Done Done false
16600 5455 Acked Write 3 Sent Done Done false
16601 5456 Acked Write 10 Sent Done Done false
16602 5457 Acked Read 6 Sent Done Done false
16603 5458 Acked Read 10 Sent Done Done false
16604 5459 Acked Write 6 Sent Done Done false
16605 5460 Acked Write 9 Sent Done Done false
16606 5461 Acked Read 3 Sent Done Done false
16607 5462 Acked Read 9 Sent Done Done false
16608 5463 Acked Read 6 Sent Done Done false
16609 5464 Acked Read 3 Sent Done Done false
16610 5465 Acked Write 8 Sent Done Done false
16611 5466 Acked Read 5 Sent Done Done false
16612 5467 Acked Read 1 Sent Done Done false
16613 5468 Acked Write 2 Sent Done Done false
16614 5469 Acked Write 6 Sent Done Done false
16615 5470 Acked Write 8 Sent Done Done false
16616 5471 Acked Read 7 Sent Done Done false
16617 5472 Acked Read 3 Sent Done Done false
16618 5473 Acked Write 6 Sent Done Done false
16619 5474 Acked Write 3 Sent Done Done false
16620 5475 Acked Read 3 Sent Done Done false
16621 5476 Acked Flush 0 Sent Done Done false
16622 5477 Acked Read 4 Sent Done Done false
16623 5478 Acked Flush 0 Sent Done Done false
16624 5479 Acked Write 5 Sent Done Done false
16625 5480 Acked Write 1 Sent Done Done false
16626 5481 Acked Write 9 Sent Done Done false
16627 5482 Acked Read 9 Sent Done Done false
16628 5483 Acked Write 7 Sent Done Done false
16629 5484 Acked Write 7 Sent Done Done false
16630 5485 Acked Write 9 Sent Done Done false
16631 5486 Acked Read 8 Sent Done Done false
16632 5487 Acked Flush 0 Sent Done Done false
16633 5488 Acked Read 2 Sent Done Done false
16634 5489 Acked Flush 0 Sent Done Done false
16635 5490 Acked Read 2 Sent Done Done false
16636 5491 Acked Flush 0 Sent Done Done false
16637 5492 Acked Read 7 Sent Done Done false
16638 5493 Acked Read 4 Sent Done Done false
16639 5494 Acked Write 9 Sent Done Done false
16640 5495 Acked Read 9 Sent Done Done false
16641 5496 Acked Read 10 Sent Done Done false
16642 5497 Acked Write 9 Sent Done Done false
16643 5498 Acked Write 7 Sent Done Done false
16644 5499 Acked Read 4 Sent Done Done false
16645 5500 Acked Read 10 Sent Done Done false
16646 5501 Acked Write 2 Sent Done Done false
16647 5502 Acked Write 1 Sent Done Done false
16648 5503 Acked Read 7 Sent Done Done false
16649 5504 Acked Read 5 Sent Done Done false
16650 5505 Acked Read 5 Sent Done Done false
16651 5506 Acked Write 4 Sent Done Done false
16652 5507 Acked Write 5 Sent Done Done false
16653 5508 Acked Read 5 Sent Done Done false
16654 5509 Acked Write 10 Sent Done Done false
16655 5510 Acked Read 2 Sent Done Done false
16656 5511 Acked Write 7 Sent Done Done false
16657 5512 Acked Write 4 Sent Done Done false
16658 5513 Acked Read 9 Sent Done Done false
16659 5514 Acked Write 6 Sent Done Done false
16660 5515 Acked Write 4 Sent Done Done false
16661 5516 Acked Write 7 Sent Done Done false
16662 5517 Acked Read 10 Sent Done Done false
16663 5518 Acked Read 4 Sent Done Done false
16664 5519 Acked Write 3 Sent Done Done false
16665 5520 Acked Flush 0 Sent Done Done false
16666 5521 Acked Read 10 Sent Done Done false
16667 5522 Acked Read 7 Sent Done Done false
16668 5523 Acked Read 4 Sent Done Done false
16669 5524 Acked Read 1 Sent Done Done false
16670 5525 Acked Read 10 Sent Done Done false
16671 5526 Acked Read 1 Sent Done Done false
16672 5527 Acked Read 3 Sent Done Done false
16673 5528 Acked Read 8 Sent Done Done false
16674 5529 Acked Write 5 Sent Done Done false
16675 5530 Acked Flush 0 Sent Done Done false
16676 5531 Acked Read 2 Sent Done Done false
16677 5532 Acked Read 2 Sent Done Done false
16678 5533 Acked Flush 0 Sent Done Done false
16679 STATES DS:0 DS:1 DS:2 TOTAL
16680 Sent 303 0 0 303
16681 Done 0 303 303 606
16682 Skipped 0 0 0 0
16683 Error 0 0 0 0
16684 Last Flush: 5230 5533 5533
16685 Downstairs last five completed: 5230 5229 5228 5227 5226
16686 Upstairs last five completed: 5533 5532 5531 5530 5529
16687 CLIENT: Up:0 ds:303 act:2
166882024-11-01T16:53:58.417ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
166892024-11-01T16:53:58.417ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33378) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
166902024-11-01T16:53:58.419ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
166912024-11-01T16:53:58.419ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
166922024-11-01T16:53:58.419ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
166932024-11-01T16:53:58.419ZINFOcrucible: send last flush ID to this DS: 5230 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
166942024-11-01T16:53:58.419ZINFOcrucible: Replied this last flush ID: 5230 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
166952024-11-01T16:53:58.419ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
166962024-11-01T16:53:58.419ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
166972024-11-01T16:53:58.420ZINFOcrucible: [0] client re-new 303 jobs since flush 5230 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
166982024-11-01T16:53:58.420ZINFOcrucible: [0] Replayed 303 jobs since flush: 5230 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
16699 ----------------------------------------------------------------
16700 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16701 Downstairs last five completed: 5533 5532 5531 5530 5529
16702 Upstairs last five completed: 5533 5532 5531 5530 5529
16703 CLIENT: Up:0 ds:0 act:3
16704 Replay: All jobs finished
16705 Main task has work to do, go find it
16706 got dsc Stop(2)
16707 stop 2
16708 Nov 01 16:54:00.718 INFO [8830] Got stop action so:false kr:false
16709 request completed, latency_us: 344, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 90ccc69c-bc1b-4e2a-80db-6af186c33099, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
16710 Nov 01 16:54:00.719 INFO request completed, latency_us: 106, response_code: 200, uri: /state/cid/2, method: GET, req_id: a7aa270c-dd67-4b48-bd3c-469a93c82f69, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
167112024-11-01T16:54:00.719ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
167122024-11-01T16:54:00.719ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
167132024-11-01T16:54:00.719ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
167142024-11-01T16:54:00.719ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
167152024-11-01T16:54:00.719ZINFOcrucible: client task is exiting = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
16716 [8830] Exited with: ExitStatus(unix_wait_status(9))
16717 [8830][2] reports Exit
16718 Nov 01 16:54:04.721 INFO request completed, latency_us: 193, response_code: 200, uri: /state/cid/2, method: GET, req_id: 05c4be4c-93fb-4bb5-b33c-6ba678128818, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
16719 001/300 Read block 728 len 32768
16720 002/300 Flush
16721 003/300 Write block 4729 len 12288 data: 3 3 2
16722 004/300 Read block 3681 len 4096
16723 005/300 Read block 3249 len 32768
16724 006/300 Read block 178 len 4096
16725 007/300 Write block 1012 len 8192 data: 2 2
16726 008/300 Write block 758 len 8192 data: 4 4
16727 009/300 Flush
16728 010/300 Write block 882 len 32768 data: 3 3 3 3 4 3 4 5
16729 011/300 Read block 121 len 28672
16730 012/300 Write block 3688 len 40960 data: 4 4 4 3 4 3 4 4 5 5
16731 013/300 Write block 956 len 8192 data: 2 2
16732 014/300 Flush
16733 015/300 Write block 3315 len 20480 data: 3 4 3 3 3
16734 016/300 Read block 3200 len 24576
16735 017/300 Write block 1196 len 8192 data: 6 4
16736 018/300 Write block 3224 len 12288 data: 3 3 4
16737 019/300 Write block 1667 len 8192 data: 4 4
16738 020/300 Read block 3651 len 12288
16739 021/300 Write block 4998 len 8192 data: 3 3
16740 022/300 Read block 2972 len 36864
16741 023/300 Read block 3576 len 32768
16742 024/300 Write block 167 len 40960 data: 3 4 4 4 4 4 4 4 4 4
16743 025/300 Read block 2410 len 12288
16744 026/300 Read block 468 len 36864
16745 027/300 Flush
16746 028/300 Read block 4416 len 24576
16747 029/300 Write block 36 len 32768 data: 3 3 4 4 5 5 4 5
16748 030/300 Read block 3418 len 4096
16749 031/300 Write block 4488 len 32768 data: 4 3 3 3 3 3 3 3
16750 032/300 Read block 4308 len 16384
16751 033/300 Read block 4642 len 36864
16752 034/300 Read block 4916 len 8192
16753 035/300 Read block 4170 len 20480
16754 036/300 Write block 462 len 4096 data: 4
16755 037/300 Write block 4328 len 24576 data: 2 2 2 3 3 3
16756 038/300 Write block 4650 len 28672 data: 2 3 3 3 3 3 2
16757 039/300 Flush
16758 040/300 Read block 2458 len 24576
16759 041/300 Read block 2307 len 4096
16760 042/300 Read block 4614 len 32768
16761 043/300 Write block 3825 len 36864 data: 5 6 6 6 5 6 7 6 7
16762 044/300 Write block 637 len 4096 data: 3
16763 045/300 Write block 1823 len 32768 data: 4 4 4 3 3 3 2 2
16764 046/300 Write block 3498 len 40960 data: 2 3 4 4 4 4 5 5 4 4
16765 047/300 Read block 936 len 16384
16766 048/300 Read block 4247 len 20480
16767 049/300 Read block 1036 len 16384
16768 050/300 Write block 1228 len 16384 data: 2 2 2 2
16769 051/300 Flush
16770 052/300 Flush
16771 053/300 Read block 4134 len 4096
16772 054/300 Flush
16773 055/300 Read block 1280 len 8192
16774 056/300 Read block 4972 len 32768
16775 057/300 Flush
16776 058/300 Write block 2983 len 36864 data: 5 5 5 4 6 6 5 5 4
16777 059/300 Flush
16778 060/300 Write block 3350 len 32768 data: 3 3 3 3 3 3 2 2
16779 061/300 Write block 3660 len 36864 data: 3 2 2 2 2 3 3 3 4
16780 062/300 Flush
16781 063/300 Read block 4143 len 28672
16782 064/300 Write block 3695 len 16384 data: 5 6 6 4
16783 065/300 Write block 4759 len 36864 data: 3 5 5 6 5 3 3 3 4
16784 066/300 Read block 509 len 36864
16785 067/300 Read block 264 len 40960
16786 068/300 Flush
16787 069/300 Read block 2687 len 16384
16788 070/300 Write block 4074 len 8192 data: 6 6
16789 071/300 Read block 1583 len 36864
16790 072/300 Write block 2476 len 16384 data: 7 7 6 6
16791 073/300 Write block 3094 len 28672 data: 7 6 5 5 5 5 5
16792 074/300 Write block 1097 len 8192 data: 2 2
16793 075/300 Write block 4309 len 24576 data: 4 6 7 7 7 8
16794 076/300 Write block 3202 len 32768 data: 7 6 7 5 6 7 6 3
16795 077/300 Read block 1449 len 36864
16796 078/300 Write block 1684 len 12288 data: 2 2 2
16797 079/300 Write block 501 len 24576 data: 3 5 7 7 6 6
16798 080/300 Read block 4670 len 36864
16799 081/300 Write block 18 len 16384 data: 3 3 2 2
16800 082/300 Flush
16801 083/300 Write block 3612 len 24576 data: 3 3 3 3 2 2
16802 084/300 Read block 3710 len 4096
16803 085/300 Write block 909 len 28672 data: 3 3 2 2 2 2 3
16804 086/300 Read block 1105 len 32768
16805 087/300 Write block 2362 len 24576 data: 5 4 3 5 5 5
16806 088/300 Read block 3983 len 20480
16807 089/300 Read block 3595 len 20480
16808 090/300 Read block 1431 len 16384
16809 091/300 Write block 2013 len 8192 data: 2 2
16810 092/300 Write block 2213 len 36864 data: 5 5 4 5 4 5 5 4 4
16811 093/300 Flush
16812 094/300 Write block 3333 len 32768 data: 3 3 3 4 4 4 4 4
16813 095/300 Read block 701 len 24576
16814 096/300 Write block 220 len 24576 data: 5 5 5 4 4 4
16815 097/300 Write block 670 len 12288 data: 3 3 3
16816 098/300 Write block 3262 len 20480 data: 5 5 5 5 4
16817 099/300 Flush
16818 100/300 Write block 4653 len 8192 data: 4 4
16819 101/300 Read block 4818 len 28672
16820 102/300 Write block 1214 len 4096 data: 4
16821 103/300 Write block 4041 len 16384 data: 2 2 2 2
16822 104/300 Write block 4972 len 40960 data: 2 2 3 3 3 3 3 3 3 4
16823 105/300 Read block 1885 len 4096
16824 106/300 Write block 4083 len 28672 data: 3 2 3 3 4 4 4
16825 107/300 Read block 2149 len 32768
16826 108/300 Read block 3588 len 20480
16827 109/300 Read block 2695 len 40960
16828 110/300 Read block 2753 len 28672
16829 111/300 Read block 24 len 32768
16830 112/300 Write block 4485 len 20480 data: 4 4 4 5 4
16831 113/300 Write block 160 len 4096 data: 4
16832 114/300 Flush
16833 115/300 Write block 3864 len 32768 data: 7 6 7 6 5 4 5 5
16834 116/300 Write block 3854 len 32768 data: 3 3 3 3 4 5 7 6
16835 117/300 Read block 1188 len 36864
16836 118/300 Read block 4301 len 24576
16837 119/300 Write block 127 len 8192 data: 6 4
16838 120/300 Read block 948 len 32768
16839 121/300 Read block 4746 len 28672
16840 122/300 Flush
16841 123/300 Read block 122 len 8192
16842 124/300 Read block 4228 len 4096
16843 125/300 Write block 2325 len 16384 data: 2 2 2 2
16844 126/300 Read block 4366 len 16384
16845 127/300 Read block 2789 len 32768
16846 128/300 Write block 4717 len 24576 data: 2 2 2 2 2 2
16847 129/300 Read block 674 len 16384
16848 130/300 Read block 1265 len 4096
16849 131/300 Read block 611 len 28672
16850 132/300 Read block 1038 len 16384
16851 133/300 Flush
16852 134/300 Read block 219 len 8192
16853 135/300 Read block 2080 len 32768
16854 136/300 Read block 2450 len 32768
16855 137/300 Write block 1896 len 4096 data: 2
16856 138/300 Write block 909 len 36864 data: 4 4 3 3 3 3 4 4 5
16857 139/300 Write block 2273 len 12288 data: 5 6 5
16858 140/300 Write block 4355 len 32768 data: 3 3 2 3 4 3 4 4
16859 141/300 Write block 4854 len 36864 data: 3 3 2 2 3 3 2 2 2
16860 142/300 Read block 3840 len 12288
16861 143/300 Read block 154 len 4096
16862 144/300 Read block 1533 len 24576
16863 145/300 Write block 4538 len 36864 data: 6 5 4 4 6 6 6 3 5
16864 146/300 Read block 3615 len 36864
16865 147/300 Read block 1659 len 24576
16866 148/300 Write block 4216 len 40960 data: 3 3 3 2 4 4 4 4 4 5
16867 149/300 Flush
16868 150/300 Flush
16869 151/300 Read block 4105 len 28672
16870 152/300 Read block 4012 len 16384
16871 153/300 Write block 3941 len 28672 data: 3 3 3 3 2 4 4
16872 154/300 Write block 1076 len 4096 data: 4
16873 155/300 Flush
16874 156/300 Read block 3036 len 16384
16875 157/300 Read block 2340 len 28672
16876 158/300 Write block 4888 len 40960 data: 4 3 3 4 5 6 4 4 4 2
16877 159/300 Read block 3289 len 12288
16878 160/300 Read block 3747 len 20480
16879 161/300 Write block 4716 len 16384 data: 2 3 3 3
16880 162/300 Read block 4207 len 36864
16881 163/300 Read block 4210 len 36864
16882 164/300 Read block 3264 len 24576
16883 165/300 Read block 886 len 20480
16884 166/300 Read block 1009 len 24576
16885 167/300 Read block 1782 len 28672
16886 168/300 Read block 868 len 28672
16887 169/300 Write block 655 len 28672 data: 4 5 5 6 7 7 6
16888 170/300 Write block 3092 len 16384 data: 6 6 8 7
16889 171/300 Flush
16890 172/300 Write block 726 len 36864 data: 2 3 3 3 4 4 6 5 4
16891 173/300 Read block 532 len 28672
16892 174/300 Flush
16893 175/300 Read block 63 len 20480
16894 176/300 Read block 710 len 12288
16895 177/300 Read block 1863 len 28672
16896 178/300 Read block 4280 len 24576
16897 179/300 Read block 2091 len 16384
16898 180/300 Write block 2440 len 20480 data: 3 3 2 2 3
16899 181/300 Flush
16900 182/300 Read block 1106 len 40960
16901 183/300 Write block 1570 len 16384 data: 3 2 3 3
16902 184/300 Read block 156 len 36864
16903 185/300 Read block 1292 len 36864
16904 186/300 Read block 3947 len 4096
16905 187/300 Write block 4537 len 24576 data: 6 7 6 5 5 7
16906 188/300 Read block 4818 len 12288
16907 189/300 Flush
16908 190/300 Read block 2070 len 24576
16909 191/300 Write block 2736 len 36864 data: 4 4 4 6 6 9 8 8 7
16910 192/300 Write block 2348 len 24576 data: 2 2 2 2 2 2
16911 193/300 Write block 2076 len 8192 data: 4 4
16912 194/300 Read block 2881 len 40960
16913 195/300 Write block 4209 len 20480 data: 4 4 3 3 3
16914 196/300 Read block 2356 len 16384
16915 197/300 Read block 4063 len 8192
16916 198/300 Read block 1604 len 28672
16917 199/300 Read block 1955 len 40960
16918 200/300 Read block 3755 len 40960
16919 201/300 Write block 4734 len 40960 data: 4 4 4 5 5 5 5 5 4 2
16920 202/300 Read block 584 len 20480
16921 203/300 Read block 1582 len 8192
16922 204/300 Write block 226 len 4096 data: 4
16923 205/300 Write block 1346 len 28672 data: 3 2 5 5 4 4 4
16924 206/300 Read block 2184 len 36864
16925 207/300 Read block 117 len 32768
16926 208/300 Flush
16927 209/300 Write block 734 len 36864 data: 5 5 5 4 4 5 4 4 2
16928 210/300 Flush
16929 211/300 Write block 3936 len 12288 data: 2 2 2
16930 212/300 Write block 1840 len 16384 data: 3 3 4 5
16931 213/300 Write block 517 len 8192 data: 4 3
16932 214/300 Read block 3184 len 40960
16933 215/300 Write block 4336 len 28672 data: 3 3 3 3 3 3 3
16934 216/300 Flush
16935 217/300 Write block 2850 len 20480 data: 3 3 3 4 4
16936 218/300 Write block 340 len 4096 data: 2
16937 219/300 Flush
16938 220/300 Read block 3030 len 36864
16939 221/300 Flush
16940 222/300 Read block 4635 len 12288
16941 223/300 Read block 4335 len 20480
16942 224/300 Write block 4518 len 16384 data: 5 5 4 5
16943 225/300 Write block 3491 len 40960 data: 3 3 3 2 2 2 2 3 4 5
16944 226/300 Write block 1214 len 32768 data: 5 4 3 3 4 3 3 4
16945 227/300 Read block 2690 len 4096
16946 228/300 Read block 1180 len 28672
16947 229/300 Write block 3278 len 8192 data: 4 4
16948 230/300 Read block 215 len 20480
16949 231/300 Read block 2005 len 40960
16950 232/300 Write block 925 len 32768 data: 6 6 7 3 4 4 4 4
16951 233/300 Read block 3336 len 32768
16952 234/300 Read block 1327 len 12288
16953 235/300 Write block 516 len 28672 data: 5 5 4 5 5 6 5
16954 236/300 Read block 1214 len 4096
16955 237/300 Flush
16956 238/300 Flush
16957 239/300 Flush
16958 240/300 Write block 4962 len 40960 data: 4 4 4 5 4 4 3 4 5 4
16959 241/300 Read block 1473 len 12288
16960 242/300 Write block 81 len 16384 data: 4 3 2 2
16961 243/300 Write block 3562 len 12288 data: 3 3 3
16962 244/300 Read block 4159 len 28672
16963 245/300 Read block 3046 len 36864
16964 246/300 Write block 3106 len 28672 data: 4 4 4 7 6 6 6
16965 247/300 Flush
16966 248/300 Write block 1646 len 8192 data: 3 3
16967 249/300 Write block 2765 len 4096 data: 5
16968 250/300 Read block 796 len 4096
16969 251/300 Write block 494 len 8192 data: 4 3
16970 252/300 Write block 3017 len 28672 data: 3 4 4 4 3 3 3
16971 253/300 Write block 2974 len 40960 data: 5 5 4 4 4 6 5 5 5 6
16972 254/300 Read block 1195 len 40960
16973 255/300 Read block 4182 len 20480
16974 256/300 Write block 2089 len 32768 data: 4 5 8 9 7 7 7 5
16975 257/300 Read block 3957 len 12288
16976 258/300 Read block 2398 len 4096
16977 259/300 Write block 462 len 28672 data: 5 3 2 2 2 2 2
16978 260/300 Write block 3625 len 40960 data: 3 3 3 2 2 3 3 3 3 3
16979 261/300 Read block 3877 len 12288
16980 262/300 Read block 2417 len 20480
16981 263/300 Write block 3918 len 40960 data: 5 5 5 3 2 2 2 3 2 2
16982 264/300 Read block 2869 len 4096
16983 265/300 Read block 1013 len 32768
16984 266/300 Read block 4170 len 24576
16985 267/300 Write block 3624 len 24576 data: 4 4 4 4 3 3
16986 268/300 Write block 4995 len 8192 data: 5 5
16987 269/300 Write block 2136 len 40960 data: 3 3 2 2 2 2 2 2 3 3
16988 270/300 Write block 2143 len 28672 data: 3 4 4 3 3 3 3
16989 271/300 Read block 2365 len 8192
16990 272/300 Write block 2727 len 4096 data: 3
16991 273/300 Write block 1479 len 16384 data: 4 3 3 3
16992 274/300 Read block 2726 len 24576
16993 275/300 Read block 604 len 40960
16994 276/300 Write block 4777 len 36864 data: 4 3 3 2 2 2 2 3 3
16995 277/300 Write block 4356 len 32768 data: 4 3 4 5 4 5 5 3
16996 278/300 Read block 3885 len 32768
16997 279/300 Write block 1497 len 8192 data: 5 4
16998 280/300 Write block 3479 len 36864 data: 6 6 6 4 5 4 5 5 5
16999 281/300 Write block 1646 len 32768 data: 4 4 2 2 3 4 4 4
17000 282/300 Read block 168 len 8192
17001 283/300 Write block 2187 len 20480 data: 6 5 5 5 3
17002 284/300 Write block 3076 len 12288 data: 5 4 3
17003 285/300 Read block 4928 len 4096
17004 286/300 Read block 4496 len 4096
17005 287/300 Read block 2548 len 8192
17006 288/300 Write block 1476 len 24576 data: 4 4 5 5 4 4
17007 289/300 Write block 691 len 28672 data: 4 4 3 5 5 5 5
17008 290/300 Write block 409 len 20480 data: 4 4 4 3 5
17009 291/300 Read block 770 len 8192
17010 292/300 Flush
17011 293/300 Read block 2278 len 40960
17012 294/300 Read block 3326 len 40960
17013 295/300 Write block 1499 len 40960 data: 4 4 4 4 4 4 5 5 6 7
17014 296/300 Read block 4111 len 16384
17015 297/300 Flush
17016 298/300 Write block 4232 len 4096 data: 2
17017 299/300 Write block 1658 len 24576 data: 3 2 4 4 7 7
17018 300/300 Read block 2841 len 40960
17019 301/300 Write block 4336 len 8192 data: 4 4
17020 Main task has work to do, go find it
17021 got dsc Start(2)
17022 start 2
17023 Nov 01 16:54:05.006 INFO request completed, latency_us: 241, response_code: 204, uri: /start/cid/2, method: POST, req_id: d9377b48-d516-47d4-8a13-dc8d275bc647, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
17024 Replay: started 2, returned:Ok(())
17025 ----------------------------------------------------------------
17026 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17027 States: Active Active Offline
17028 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17029 5534 Acked Read 8 Done Done Sent false
17030 5535 Acked Flush 0 Done Done Sent false
17031 5536 Acked Write 3 Done Done Sent false
17032 5537 Acked Read 1 Done Done Sent false
17033 5538 Acked Read 8 Done Done Sent false
17034 5539 Acked Read 1 Done Done Sent false
17035 5540 Acked Write 2 Done Done Sent false
17036 5541 Acked Write 2 Done Done Sent false
17037 5542 Acked Flush 0 Done Done Sent false
17038 5543 Acked Write 8 Done Done Sent false
17039 5544 Acked Read 7 Done Done Sent false
17040 5545 Acked Write 10 Done Done Sent false
17041 5546 Acked Write 2 Done Done Sent false
17042 5547 Acked Flush 0 Done Done Sent false
17043 5548 Acked Write 5 Done Done Sent false
17044 5549 Acked Read 6 Done Done Sent false
17045 5550 Acked Write 2 Done Done Sent false
17046 5551 Acked Write 3 Done Done Sent false
17047 5552 Acked Write 2 Done Done Sent false
17048 5553 Acked Read 3 Done Done Sent false
17049 5554 Acked Write 2 Done Done Sent false
17050 5555 Acked Read 9 Done Done Sent false
17051 5556 Acked Read 8 Done Done Sent false
17052 5557 Acked Write 10 Done Done Sent false
17053 5558 Acked Read 3 Done Done Sent false
17054 5559 Acked Read 9 Done Done Sent false
17055 5560 Acked Flush 0 Done Done Sent false
17056 5561 Acked Read 6 Done Done Sent false
17057 5562 Acked Write 8 Done Done Sent false
17058 5563 Acked Read 1 Done Done Sent false
17059 5564 Acked Write 8 Done Done Sent false
17060 5565 Acked Read 4 Done Done Sent false
17061 5566 Acked Read 9 Done Done Sent false
17062 5567 Acked Read 2 Done Done Sent false
17063 5568 Acked Read 5 Done Done Sent false
17064 5569 Acked Write 1 Done Done Sent false
17065 5570 Acked Write 6 Done Done Sent false
17066 5571 Acked Write 7 Done Done Sent false
17067 5572 Acked Flush 0 Done Done Sent false
17068 5573 Acked Read 6 Done Done Sent false
17069 5574 Acked Read 1 Done Done Sent false
17070 5575 Acked Read 8 Done Done Sent false
17071 5576 Acked Write 9 Done Done Sent false
17072 5577 Acked Write 1 Done Done Sent false
17073 5578 Acked Write 8 Done Done Sent false
17074 5579 Acked Write 10 Done Done Sent false
17075 5580 Acked Read 4 Done Done Sent false
17076 5581 Acked Read 5 Done Done Sent false
17077 5582 Acked Read 4 Done Done Sent false
17078 5583 Acked Write 4 Done Done Sent false
17079 5584 Acked Flush 0 Done Done Sent false
17080 5585 Acked Flush 0 Done Done Sent false
17081 5586 Acked Read 1 Done Done Sent false
17082 5587 Acked Flush 0 Done Done Sent false
17083 5588 Acked Read 2 Done Done Sent false
17084 5589 Acked Read 8 Done Done Sent false
17085 5590 Acked Flush 0 Done Done Sent false
17086 5591 Acked Write 9 Done Done Sent false
17087 5592 Acked Flush 0 Done Done Sent false
17088 5593 Acked Write 8 Done Done Sent false
17089 5594 Acked Write 9 Done Done Sent false
17090 5595 Acked Flush 0 Done Done Sent false
17091 5596 Acked Read 7 Done Done Sent false
17092 5597 Acked Write 4 Done Done Sent false
17093 5598 Acked Write 9 Done Done Sent false
17094 5599 Acked Read 9 Done Done Sent false
17095 5600 Acked Read 10 Done Done Sent false
17096 5601 Acked Flush 0 Done Done Sent false
17097 5602 Acked Read 4 Done Done Sent false
17098 5603 Acked Write 2 Done Done Sent false
17099 5604 Acked Read 9 Done Done Sent false
17100 5605 Acked Write 4 Done Done Sent false
17101 5606 Acked Write 7 Done Done Sent false
17102 5607 Acked Write 2 Done Done Sent false
17103 5608 Acked Write 6 Done Done Sent false
17104 5609 Acked Write 8 Done Done Sent false
17105 5610 Acked Read 9 Done Done Sent false
17106 5611 Acked Write 3 Done Done Sent false
17107 5612 Acked Write 6 Done Done Sent false
17108 5613 Acked Read 9 Done Done Sent false
17109 5614 Acked Write 4 Done Done Sent false
17110 5615 Acked Flush 0 Done Done Sent false
17111 5616 Acked Write 6 Done Done Sent false
17112 5617 Acked Read 1 Done Done Sent false
17113 5618 Acked Write 7 Done Done Sent false
17114 5619 Acked Read 8 Done Done Sent false
17115 5620 Acked Write 6 Done Done Sent false
17116 5621 Acked Read 5 Done Done Sent false
17117 5622 Acked Read 5 Done Done Sent false
17118 5623 Acked Read 4 Done Done Sent false
17119 5624 Acked Write 2 Done Done Sent false
17120 5625 Acked Write 9 Done Done Sent false
17121 5626 Acked Flush 0 Done Done Sent false
17122 5627 Acked Write 8 Done Done Sent false
17123 5628 Acked Read 6 Done Done Sent false
17124 5629 Acked Write 6 Done Done Sent false
17125 5630 Acked Write 3 Done Done Sent false
17126 5631 Acked Write 5 Done Done Sent false
17127 5632 Acked Flush 0 Done Done Sent false
17128 5633 Acked Write 2 Done Done Sent false
17129 5634 Acked Read 7 Done Done Sent false
17130 5635 Acked Write 1 Done Done Sent false
17131 5636 Acked Write 4 Done Done Sent false
17132 5637 Acked Write 10 Done Done Sent false
17133 5638 Acked Read 1 Done Done Sent false
17134 5639 Acked Write 7 Done Done Sent false
17135 5640 Acked Read 8 Done Done Sent false
17136 5641 Acked Read 5 Done Done Sent false
17137 5642 Acked Read 10 Done Done Sent false
17138 5643 Acked Read 7 Done Done Sent false
17139 5644 Acked Read 8 Done Done Sent false
17140 5645 Acked Write 5 Done Done Sent false
17141 5646 Acked Write 1 Done Done Sent false
17142 5647 Acked Flush 0 Done Done Sent false
17143 5648 Acked Write 8 Done Done Sent false
17144 5649 Acked Write 8 Done Done Sent false
17145 5650 Acked Read 9 Done Done Sent false
17146 5651 Acked Read 6 Done Done Sent false
17147 5652 Acked Write 2 Done Done Sent false
17148 5653 Acked Read 8 Done Done Sent false
17149 5654 Acked Read 7 Done Done Sent false
17150 5655 Acked Flush 0 Done Done Sent false
17151 5656 Acked Read 2 Done Done Sent false
17152 5657 Acked Read 1 Done Done Sent false
17153 5658 Acked Write 4 Done Done Sent false
17154 5659 Acked Read 4 Done Done Sent false
17155 5660 Acked Read 8 Done Done Sent false
17156 5661 Acked Write 6 Done Done Sent false
17157 5662 Acked Read 4 Done Done Sent false
17158 5663 Acked Read 1 Done Done Sent false
17159 5664 Acked Read 7 Done Done Sent false
17160 5665 Acked Read 4 Done Done Sent false
17161 5666 Acked Flush 0 Done Done Sent false
17162 5667 Acked Read 2 Done Done Sent false
17163 5668 Acked Read 8 Done Done Sent false
17164 5669 Acked Read 8 Done Done Sent false
17165 5670 Acked Write 1 Done Done Sent false
17166 5671 Acked Write 9 Done Done Sent false
17167 5672 Acked Write 3 Done Done Sent false
17168 5673 Acked Write 8 Done Done Sent false
17169 5674 Acked Write 9 Done Done Sent false
17170 5675 Acked Read 3 Done Done Sent false
17171 5676 Acked Read 1 Done Done Sent false
17172 5677 Acked Read 6 Done Done Sent false
17173 5678 Acked Write 9 Done Done Sent false
17174 5679 Acked Read 9 Done Done Sent false
17175 5680 Acked Read 6 Done Done Sent false
17176 5681 Acked Write 10 Done Done Sent false
17177 5682 Acked Flush 0 Done Done Sent false
17178 5683 Acked Flush 0 Done Done Sent false
17179 5684 Acked Read 7 Done Done Sent false
17180 5685 Acked Read 4 Done Done Sent false
17181 5686 Acked Write 7 Done Done Sent false
17182 5687 Acked Write 1 Done Done Sent false
17183 5688 Acked Flush 0 Done Done Sent false
17184 5689 Acked Read 4 Done Done Sent false
17185 5690 Acked Read 7 Done Done Sent false
17186 5691 Acked Write 10 Done Done Sent false
17187 5692 Acked Read 3 Done Done Sent false
17188 5693 Acked Read 5 Done Done Sent false
17189 5694 Acked Write 4 Done Done Sent false
17190 5695 Acked Read 9 Done Done Sent false
17191 5696 Acked Read 9 Done Done Sent false
17192 5697 Acked Read 6 Done Done Sent false
17193 5698 Acked Read 5 Done Done Sent false
17194 5699 Acked Read 6 Done Done Sent false
17195 5700 Acked Read 7 Done Done Sent false
17196 5701 Acked Read 7 Done Done Sent false
17197 5702 Acked Write 7 Done Done Sent false
17198 5703 Acked Write 4 Done Done Sent false
17199 5704 Acked Flush 0 Done Done Sent false
17200 5705 Acked Write 9 Done Done Sent false
17201 5706 Acked Read 7 Done Done Sent false
17202 5707 Acked Flush 0 Done Done Sent false
17203 5708 Acked Read 5 Done Done Sent false
17204 5709 Acked Read 3 Done Done Sent false
17205 5710 Acked Read 7 Done Done Sent false
17206 5711 Acked Read 6 Done Done Sent false
17207 5712 Acked Read 4 Done Done Sent false
17208 5713 Acked Write 5 Done Done Sent false
17209 5714 Acked Flush 0 Done Done Sent false
17210 5715 Acked Read 10 Done Done Sent false
17211 5716 Acked Write 4 Done Done Sent false
17212 5717 Acked Read 9 Done Done Sent false
17213 5718 Acked Read 9 Done Done Sent false
17214 5719 Acked Read 1 Done Done Sent false
17215 5720 Acked Write 6 Done Done Sent false
17216 5721 Acked Read 3 Done Done Sent false
17217 5722 Acked Flush 0 Done Done Sent false
17218 5723 Acked Read 6 Done Done Sent false
17219 5724 Acked Write 9 Done Done Sent false
17220 5725 Acked Write 6 Done Done Sent false
17221 5726 Acked Write 2 Done Done Sent false
17222 5727 Acked Read 10 Done Done Sent false
17223 5728 Acked Write 5 Done Done Sent false
17224 5729 Acked Read 4 Done Done Sent false
17225 5730 Acked Read 2 Done Done Sent false
17226 5731 Acked Read 7 Done Done Sent false
17227 5732 Acked Read 10 Done Done Sent false
17228 5733 Acked Read 10 Done Done Sent false
17229 5734 Acked Write 10 Done Done Sent false
17230 5735 Acked Read 5 Done Done Sent false
17231 5736 Acked Read 2 Done Done Sent false
17232 5737 Acked Write 1 Done Done Sent false
17233 5738 Acked Write 7 Done Done Sent false
17234 5739 Acked Read 9 Done Done Sent false
17235 5740 Acked Read 8 Done Done Sent false
17236 5741 Acked Flush 0 Done Done Sent false
17237 5742 Acked Write 9 Done Done Sent false
17238 5743 Acked Flush 0 Done Done Sent false
17239 5744 Acked Write 3 Done Done Sent false
17240 5745 Acked Write 4 Done Done Sent false
17241 5746 Acked Write 2 Done Done Sent false
17242 5747 Acked Read 10 Done Done Sent false
17243 5748 Acked Write 7 Done Done Sent false
17244 5749 Acked Flush 0 Done Done Sent false
17245 5750 Acked Write 5 Done Done Sent false
17246 5751 Acked Write 1 Done Done Sent false
17247 5752 Acked Flush 0 Done Done Sent false
17248 5753 Acked Read 9 Done Done Sent false
17249 5754 Acked Flush 0 Done Done Sent false
17250 5755 Acked Read 3 Done Done Sent false
17251 5756 Acked Read 5 Done Done Sent false
17252 5757 Acked Write 4 Done Done Sent false
17253 5758 Acked Write 10 Done Done Sent false
17254 5759 Acked Write 8 Done Done Sent false
17255 5760 Acked Read 1 Done Done Sent false
17256 5761 Acked Read 7 Done Done Sent false
17257 5762 Acked Write 2 Done Done Sent false
17258 5763 Acked Read 5 Done Done Sent false
17259 5764 Acked Read 10 Done Done Sent false
17260 5765 Acked Write 8 Done Done Sent false
17261 5766 Acked Read 8 Done Done Sent false
17262 5767 Acked Read 3 Done Done Sent false
17263 5768 Acked Write 7 Done Done Sent false
17264 5769 Acked Read 1 Done Done Sent false
17265 5770 Acked Flush 0 Done Done Sent false
17266 5771 Acked Flush 0 Done Done Sent false
17267 5772 Acked Flush 0 Done Done Sent false
17268 5773 Acked Write 10 Done Done Sent false
17269 5774 Acked Read 3 Done Done Sent false
17270 5775 Acked Write 4 Done Done Sent false
17271 5776 Acked Write 3 Done Done Sent false
17272 5777 Acked Read 7 Done Done Sent false
17273 5778 Acked Read 9 Done Done Sent false
17274 5779 Acked Write 7 Done Done Sent false
17275 5780 Acked Flush 0 Done Done Sent false
17276 5781 Acked Write 2 Done Done Sent false
17277 5782 Acked Write 1 Done Done Sent false
17278 5783 Acked Read 1 Done Done Sent false
17279 5784 Acked Write 2 Done Done Sent false
17280 5785 Acked Write 7 Done Done Sent false
17281 5786 Acked Write 10 Done Done Sent false
17282 5787 Acked Read 10 Done Done Sent false
17283 5788 Acked Read 5 Done Done Sent false
17284 5789 Acked Write 8 Done Done Sent false
17285 5790 Acked Read 3 Done Done Sent false
17286 5791 Acked Read 1 Done Done Sent false
17287 5792 Acked Write 7 Done Done Sent false
17288 5793 Acked Write 10 Done Done Sent false
17289 5794 Acked Read 3 Done Done Sent false
17290 5795 Acked Read 5 Done Done Sent false
17291 5796 Acked Write 10 Done Done Sent false
17292 5797 Acked Read 1 Done Done Sent false
17293 5798 Acked Read 8 Done Done Sent false
17294 5799 Acked Read 6 Done Done Sent false
17295 5800 Acked Write 6 Done Done Sent false
17296 5801 Acked Write 2 Done Done Sent false
17297 5802 Acked Write 10 Done Done Sent false
17298 5803 Acked Write 7 Done Done Sent false
17299 5804 Acked Read 2 Done Done Sent false
17300 5805 Acked Write 1 Done Done Sent false
17301 5806 Acked Write 4 Done Done Sent false
17302 5807 Acked Read 6 Done Done Sent false
17303 5808 Acked Read 10 Done Done Sent false
17304 5809 Acked Write 9 Done Done Sent false
17305 5810 Acked Write 8 Done Done Sent false
17306 5811 Acked Read 8 Done Done Sent false
17307 5812 Acked Write 2 Done Done Sent false
17308 5813 Acked Write 9 Done Done Sent false
17309 5814 Acked Write 8 Done Done Sent false
17310 5815 Acked Read 2 Done Done Sent false
17311 5816 Acked Write 5 Done Done Sent false
17312 5817 Acked Write 3 Done Done Sent false
17313 5818 Acked Read 1 Done Done Sent false
17314 5819 Acked Read 1 Done Done Sent false
17315 5820 Acked Read 2 Done Done Sent false
17316 5821 Acked Write 6 Done Done Sent false
17317 5822 Acked Write 7 Done Done Sent false
17318 5823 Acked Write 5 Done Done Sent false
17319 5824 Acked Read 2 Done Done Sent false
17320 5825 Acked Flush 0 Done Done Sent false
17321 5826 Acked Read 10 Done Done Sent false
17322 5827 Acked Read 10 Done Done Sent false
17323 5828 Acked Write 10 Done Done Sent false
17324 5829 Acked Read 4 Done Done Sent false
17325 5830 Acked Flush 0 Done Done Sent false
17326 5831 Acked Write 1 Done Done Sent false
17327 5832 Acked Write 6 Done Done Sent false
17328 5833 Acked Read 10 Done Done Sent false
17329 5834 Acked Write 2 Done Done Sent false
17330 STATES DS:0 DS:1 DS:2 TOTAL
17331 Sent 0 0 301 301
17332 Done 301 301 0 602
17333 Skipped 0 0 0 0
17334 Error 0 0 0 0
17335 Last Flush: 5830 5830 5533
17336 Downstairs last five completed: 5533 5532 5531 5530 5529
17337 Upstairs last five completed: 5834 5833 5832 5831 5830
17338 CLIENT: Up:0 ds:301 act:2
17339 [8830] got start action
17340 [8830] I am going to restart
17341 Starting downstairs at port 8830
17342 Make output file at "/tmp/dsc/downstairs-8830.txt"
17343 [8830][2] reports Starting
17344 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1214)
17345 [8830][2] reports Running
17346 ----------------------------------------------------------------
17347 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
17348 States: Active Active Offline
17349 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17350 5534 Acked Read 8 Done Done Sent false
17351 5535 Acked Flush 0 Done Done Sent false
17352 5536 Acked Write 3 Done Done Sent false
17353 5537 Acked Read 1 Done Done Sent false
17354 5538 Acked Read 8 Done Done Sent false
17355 5539 Acked Read 1 Done Done Sent false
17356 5540 Acked Write 2 Done Done Sent false
17357 5541 Acked Write 2 Done Done Sent false
17358 5542 Acked Flush 0 Done Done Sent false
17359 5543 Acked Write 8 Done Done Sent false
17360 5544 Acked Read 7 Done Done Sent false
17361 5545 Acked Write 10 Done Done Sent false
17362 5546 Acked Write 2 Done Done Sent false
17363 5547 Acked Flush 0 Done Done Sent false
17364 5548 Acked Write 5 Done Done Sent false
17365 5549 Acked Read 6 Done Done Sent false
17366 5550 Acked Write 2 Done Done Sent false
17367 5551 Acked Write 3 Done Done Sent false
17368 5552 Acked Write 2 Done Done Sent false
17369 5553 Acked Read 3 Done Done Sent false
17370 5554 Acked Write 2 Done Done Sent false
17371 5555 Acked Read 9 Done Done Sent false
17372 5556 Acked Read 8 Done Done Sent false
17373 5557 Acked Write 10 Done Done Sent false
17374 5558 Acked Read 3 Done Done Sent false
17375 5559 Acked Read 9 Done Done Sent false
17376 5560 Acked Flush 0 Done Done Sent false
17377 5561 Acked Read 6 Done Done Sent false
17378 5562 Acked Write 8 Done Done Sent false
17379 5563 Acked Read 1 Done Done Sent false
17380 5564 Acked Write 8 Done Done Sent false
17381 5565 Acked Read 4 Done Done Sent false
17382 5566 Acked Read 9 Done Done Sent false
17383 5567 Acked Read 2 Done Done Sent false
17384 5568 Acked Read 5 Done Done Sent false
17385 5569 Acked Write 1 Done Done Sent false
17386 5570 Acked Write 6 Done Done Sent false
17387 5571 Acked Write 7 Done Done Sent false
17388 5572 Acked Flush 0 Done Done Sent false
17389 5573 Acked Read 6 Done Done Sent false
17390 5574 Acked Read 1 Done Done Sent false
17391 5575 Acked Read 8 Done Done Sent false
17392 5576 Acked Write 9 Done Done Sent false
17393 5577 Acked Write 1 Done Done Sent false
17394 5578 Acked Write 8 Done Done Sent false
17395 5579 Acked Write 10 Done Done Sent false
17396 5580 Acked Read 4 Done Done Sent false
17397 5581 Acked Read 5 Done Done Sent false
17398 5582 Acked Read 4 Done Done Sent false
17399 5583 Acked Write 4 Done Done Sent false
17400 5584 Acked Flush 0 Done Done Sent false
17401 5585 Acked Flush 0 Done Done Sent false
17402 5586 Acked Read 1 Done Done Sent false
17403 5587 Acked Flush 0 Done Done Sent false
17404 5588 Acked Read 2 Done Done Sent false
17405 5589 Acked Read 8 Done Done Sent false
17406 5590 Acked Flush 0 Done Done Sent false
17407 5591 Acked Write 9 Done Done Sent false
17408 5592 Acked Flush 0 Done Done Sent false
17409 5593 Acked Write 8 Done Done Sent false
17410 5594 Acked Write 9 Done Done Sent false
17411 5595 Acked Flush 0 Done Done Sent false
17412 5596 Acked Read 7 Done Done Sent false
17413 5597 Acked Write 4 Done Done Sent false
17414 5598 Acked Write 9 Done Done Sent false
17415 5599 Acked Read 9 Done Done Sent false
17416 5600 Acked Read 10 Done Done Sent false
17417 5601 Acked Flush 0 Done Done Sent false
17418 5602 Acked Read 4 Done Done Sent false
17419 5603 Acked Write 2 Done Done Sent false
17420 5604 Acked Read 9 Done Done Sent false
17421 5605 Acked Write 4 Done Done Sent false
17422 5606 Acked Write 7 Done Done Sent false
17423 5607 Acked Write 2 Done Done Sent false
17424 5608 Acked Write 6 Done Done Sent false
17425 5609 Acked Write 8 Done Done Sent false
17426 5610 Acked Read 9 Done Done Sent false
17427 5611 Acked Write 3 Done Done Sent false
17428 5612 Acked Write 6 Done Done Sent false
17429 5613 Acked Read 9 Done Done Sent false
17430 5614 Acked Write 4 Done Done Sent false
17431 5615 Acked Flush 0 Done Done Sent false
17432 5616 Acked Write 6 Done Done Sent false
17433 5617 Acked Read 1 Done Done Sent false
17434 5618 Acked Write 7 Done Done Sent false
17435 5619 Acked Read 8 Done Done Sent false
17436 5620 Acked Write 6 Done Done Sent false
17437 5621 Acked Read 5 Done Done Sent false
17438 5622 Acked Read 5 Done Done Sent false
17439 5623 Acked Read 4 Done Done Sent false
17440 5624 Acked Write 2 Done Done Sent false
17441 5625 Acked Write 9 Done Done Sent false
17442 5626 Acked Flush 0 Done Done Sent false
17443 5627 Acked Write 8 Done Done Sent false
17444 5628 Acked Read 6 Done Done Sent false
17445 5629 Acked Write 6 Done Done Sent false
17446 5630 Acked Write 3 Done Done Sent false
17447 5631 Acked Write 5 Done Done Sent false
17448 5632 Acked Flush 0 Done Done Sent false
17449 5633 Acked Write 2 Done Done Sent false
17450 5634 Acked Read 7 Done Done Sent false
17451 5635 Acked Write 1 Done Done Sent false
17452 5636 Acked Write 4 Done Done Sent false
17453 5637 Acked Write 10 Done Done Sent false
17454 5638 Acked Read 1 Done Done Sent false
17455 5639 Acked Write 7 Done Done Sent false
17456 5640 Acked Read 8 Done Done Sent false
17457 5641 Acked Read 5 Done Done Sent false
17458 5642 Acked Read 10 Done Done Sent false
17459 5643 Acked Read 7 Done Done Sent false
17460 5644 Acked Read 8 Done Done Sent false
17461 5645 Acked Write 5 Done Done Sent false
17462 5646 Acked Write 1 Done Done Sent false
17463 5647 Acked Flush 0 Done Done Sent false
17464 5648 Acked Write 8 Done Done Sent false
17465 5649 Acked Write 8 Done Done Sent false
17466 5650 Acked Read 9 Done Done Sent false
17467 5651 Acked Read 6 Done Done Sent false
17468 5652 Acked Write 2 Done Done Sent false
17469 5653 Acked Read 8 Done Done Sent false
17470 5654 Acked Read 7 Done Done Sent false
17471 5655 Acked Flush 0 Done Done Sent false
17472 5656 Acked Read 2 Done Done Sent false
17473 5657 Acked Read 1 Done Done Sent false
17474 5658 Acked Write 4 Done Done Sent false
17475 5659 Acked Read 4 Done Done Sent false
17476 5660 Acked Read 8 Done Done Sent false
17477 5661 Acked Write 6 Done Done Sent false
17478 5662 Acked Read 4 Done Done Sent false
17479 5663 Acked Read 1 Done Done Sent false
17480 5664 Acked Read 7 Done Done Sent false
17481 5665 Acked Read 4 Done Done Sent false
17482 5666 Acked Flush 0 Done Done Sent false
17483 5667 Acked Read 2 Done Done Sent false
17484 5668 Acked Read 8 Done Done Sent false
17485 5669 Acked Read 8 Done Done Sent false
17486 5670 Acked Write 1 Done Done Sent false
17487 5671 Acked Write 9 Done Done Sent false
17488 5672 Acked Write 3 Done Done Sent false
17489 5673 Acked Write 8 Done Done Sent false
17490 5674 Acked Write 9 Done Done Sent false
17491 5675 Acked Read 3 Done Done Sent false
17492 5676 Acked Read 1 Done Done Sent false
17493 5677 Acked Read 6 Done Done Sent false
17494 5678 Acked Write 9 Done Done Sent false
17495 5679 Acked Read 9 Done Done Sent false
17496 5680 Acked Read 6 Done Done Sent false
17497 5681 Acked Write 10 Done Done Sent false
17498 5682 Acked Flush 0 Done Done Sent false
17499 5683 Acked Flush 0 Done Done Sent false
17500 5684 Acked Read 7 Done Done Sent false
17501 5685 Acked Read 4 Done Done Sent false
17502 5686 Acked Write 7 Done Done Sent false
17503 5687 Acked Write 1 Done Done Sent false
17504 5688 Acked Flush 0 Done Done Sent false
17505 5689 Acked Read 4 Done Done Sent false
17506 5690 Acked Read 7 Done Done Sent false
17507 5691 Acked Write 10 Done Done Sent false
17508 5692 Acked Read 3 Done Done Sent false
17509 5693 Acked Read 5 Done Done Sent false
17510 5694 Acked Write 4 Done Done Sent false
17511 5695 Acked Read 9 Done Done Sent false
17512 5696 Acked Read 9 Done Done Sent false
17513 5697 Acked Read 6 Done Done Sent false
17514 5698 Acked Read 5 Done Done Sent false
17515 5699 Acked Read 6 Done Done Sent false
17516 5700 Acked Read 7 Done Done Sent false
17517 5701 Acked Read 7 Done Done Sent false
17518 5702 Acked Write 7 Done Done Sent false
17519 5703 Acked Write 4 Done Done Sent false
17520 5704 Acked Flush 0 Done Done Sent false
17521 5705 Acked Write 9 Done Done Sent false
17522 5706 Acked Read 7 Done Done Sent false
17523 5707 Acked Flush 0 Done Done Sent false
17524 5708 Acked Read 5 Done Done Sent false
17525 5709 Acked Read 3 Done Done Sent false
17526 5710 Acked Read 7 Done Done Sent false
17527 5711 Acked Read 6 Done Done Sent false
17528 5712 Acked Read 4 Done Done Sent false
17529 5713 Acked Write 5 Done Done Sent false
17530 5714 Acked Flush 0 Done Done Sent false
17531 5715 Acked Read 10 Done Done Sent false
17532 5716 Acked Write 4 Done Done Sent false
17533 5717 Acked Read 9 Done Done Sent false
17534 5718 Acked Read 9 Done Done Sent false
17535 5719 Acked Read 1 Done Done Sent false
17536 5720 Acked Write 6 Done Done Sent false
17537 5721 Acked Read 3 Done Done Sent false
17538 5722 Acked Flush 0 Done Done Sent false
17539 5723 Acked Read 6 Done Done Sent false
17540 5724 Acked Write 9 Done Done Sent false
17541 5725 Acked Write 6 Done Done Sent false
17542 5726 Acked Write 2 Done Done Sent false
17543 5727 Acked Read 10 Done Done Sent false
17544 5728 Acked Write 5 Done Done Sent false
17545 5729 Acked Read 4 Done Done Sent false
17546 5730 Acked Read 2 Done Done Sent false
17547 5731 Acked Read 7 Done Done Sent false
17548 5732 Acked Read 10 Done Done Sent false
17549 5733 Acked Read 10 Done Done Sent false
17550 5734 Acked Write 10 Done Done Sent false
17551 5735 Acked Read 5 Done Done Sent false
17552 5736 Acked Read 2 Done Done Sent false
17553 5737 Acked Write 1 Done Done Sent false
17554 5738 Acked Write 7 Done Done Sent false
17555 5739 Acked Read 9 Done Done Sent false
17556 5740 Acked Read 8 Done Done Sent false
17557 5741 Acked Flush 0 Done Done Sent false
17558 5742 Acked Write 9 Done Done Sent false
17559 5743 Acked Flush 0 Done Done Sent false
17560 5744 Acked Write 3 Done Done Sent false
17561 5745 Acked Write 4 Done Done Sent false
17562 5746 Acked Write 2 Done Done Sent false
17563 5747 Acked Read 10 Done Done Sent false
17564 5748 Acked Write 7 Done Done Sent false
17565 5749 Acked Flush 0 Done Done Sent false
17566 5750 Acked Write 5 Done Done Sent false
17567 5751 Acked Write 1 Done Done Sent false
17568 5752 Acked Flush 0 Done Done Sent false
17569 5753 Acked Read 9 Done Done Sent false
17570 5754 Acked Flush 0 Done Done Sent false
17571 5755 Acked Read 3 Done Done Sent false
17572 5756 Acked Read 5 Done Done Sent false
17573 5757 Acked Write 4 Done Done Sent false
17574 5758 Acked Write 10 Done Done Sent false
17575 5759 Acked Write 8 Done Done Sent false
17576 5760 Acked Read 1 Done Done Sent false
17577 5761 Acked Read 7 Done Done Sent false
17578 5762 Acked Write 2 Done Done Sent false
17579 5763 Acked Read 5 Done Done Sent false
17580 5764 Acked Read 10 Done Done Sent false
17581 5765 Acked Write 8 Done Done Sent false
17582 5766 Acked Read 8 Done Done Sent false
17583 5767 Acked Read 3 Done Done Sent false
17584 5768 Acked Write 7 Done Done Sent false
17585 5769 Acked Read 1 Done Done Sent false
17586 5770 Acked Flush 0 Done Done Sent false
17587 5771 Acked Flush 0 Done Done Sent false
17588 5772 Acked Flush 0 Done Done Sent false
17589 5773 Acked Write 10 Done Done Sent false
17590 5774 Acked Read 3 Done Done Sent false
17591 5775 Acked Write 4 Done Done Sent false
17592 5776 Acked Write 3 Done Done Sent false
17593 5777 Acked Read 7 Done Done Sent false
17594 5778 Acked Read 9 Done Done Sent false
17595 5779 Acked Write 7 Done Done Sent false
17596 5780 Acked Flush 0 Done Done Sent false
17597 5781 Acked Write 2 Done Done Sent false
17598 5782 Acked Write 1 Done Done Sent false
17599 5783 Acked Read 1 Done Done Sent false
17600 5784 Acked Write 2 Done Done Sent false
17601 5785 Acked Write 7 Done Done Sent false
17602 5786 Acked Write 10 Done Done Sent false
17603 5787 Acked Read 10 Done Done Sent false
17604 5788 Acked Read 5 Done Done Sent false
17605 5789 Acked Write 8 Done Done Sent false
17606 5790 Acked Read 3 Done Done Sent false
17607 5791 Acked Read 1 Done Done Sent false
17608 5792 Acked Write 7 Done Done Sent false
17609 5793 Acked Write 10 Done Done Sent false
17610 5794 Acked Read 3 Done Done Sent false
17611 5795 Acked Read 5 Done Done Sent false
17612 5796 Acked Write 10 Done Done Sent false
17613 5797 Acked Read 1 Done Done Sent false
17614 5798 Acked Read 8 Done Done Sent false
17615 5799 Acked Read 6 Done Done Sent false
17616 5800 Acked Write 6 Done Done Sent false
17617 5801 Acked Write 2 Done Done Sent false
17618 5802 Acked Write 10 Done Done Sent false
17619 5803 Acked Write 7 Done Done Sent false
17620 5804 Acked Read 2 Done Done Sent false
17621 5805 Acked Write 1 Done Done Sent false
17622 5806 Acked Write 4 Done Done Sent false
17623 5807 Acked Read 6 Done Done Sent false
17624 5808 Acked Read 10 Done Done Sent false
17625 5809 Acked Write 9 Done Done Sent false
17626 5810 Acked Write 8 Done Done Sent false
17627 5811 Acked Read 8 Done Done Sent false
17628 5812 Acked Write 2 Done Done Sent false
17629 5813 Acked Write 9 Done Done Sent false
17630 5814 Acked Write 8 Done Done Sent false
17631 5815 Acked Read 2 Done Done Sent false
17632 5816 Acked Write 5 Done Done Sent false
17633 5817 Acked Write 3 Done Done Sent false
17634 5818 Acked Read 1 Done Done Sent false
17635 5819 Acked Read 1 Done Done Sent false
17636 5820 Acked Read 2 Done Done Sent false
17637 5821 Acked Write 6 Done Done Sent false
17638 5822 Acked Write 7 Done Done Sent false
17639 5823 Acked Write 5 Done Done Sent false
17640 5824 Acked Read 2 Done Done Sent false
17641 5825 Acked Flush 0 Done Done Sent false
17642 5826 Acked Read 10 Done Done Sent false
17643 5827 Acked Read 10 Done Done Sent false
17644 5828 Acked Write 10 Done Done Sent false
17645 5829 Acked Read 4 Done Done Sent false
17646 5830 Acked Flush 0 Done Done Sent false
17647 5831 Acked Write 1 Done Done Sent false
17648 5832 Acked Write 6 Done Done Sent false
17649 5833 Acked Read 10 Done Done Sent false
17650 5834 Acked Write 2 Done Done Sent false
17651 5835 Acked Flush 0 Done Done Sent false
17652 STATES DS:0 DS:1 DS:2 TOTAL
17653 Sent 0 0 302 302
17654 Done 302 302 0 604
17655 Skipped 0 0 0 0
17656 Error 0 0 0 0
17657 Last Flush: 5835 5835 5533
17658 Downstairs last five completed: 5533 5532 5531 5530 5529
17659 Upstairs last five completed: 5835 5834 5833 5832 5831
17660 CLIENT: Up:0 ds:302 act:2
176612024-11-01T16:54:10.720ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
176622024-11-01T16:54:10.721ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58940) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
176632024-11-01T16:54:10.722ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
176642024-11-01T16:54:10.722ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
176652024-11-01T16:54:10.722ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
176662024-11-01T16:54:10.722ZINFOcrucible: send last flush ID to this DS: 5533 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
176672024-11-01T16:54:10.722ZINFOcrucible: Replied this last flush ID: 5533 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
176682024-11-01T16:54:10.723ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
176692024-11-01T16:54:10.723ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
176702024-11-01T16:54:10.723ZINFOcrucible: [2] client re-new 302 jobs since flush 5533 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
176712024-11-01T16:54:10.723ZINFOcrucible: [2] Replayed 302 jobs since flush: 5533 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
17672 ----------------------------------------------------------------
17673 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17674 Downstairs last five completed: 5835 5834 5833 5832 5831
17675 Upstairs last five completed: 5835 5834 5833 5832 5831
17676 CLIENT: Up:0 ds:0 act:3
17677 Replay: All jobs finished
17678 Main task has work to do, go find it
17679 got dsc Stop(2)
17680 stop 2
17681 Nov 01 16:54:13.011 INFO request completed, latency_us: 341, response_code: 204, uri: /stop/cid/2, method[8830] Got stop action so:false kr:false
17682 : POST, req_id: 4fc3b084-e824-4ad6-b259-cbbd02f5365d, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
176832024-11-01T16:54:13.012ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
176842024-11-01T16:54:13.012ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
176852024-11-01T16:54:13.012ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
176862024-11-01T16:54:13.012ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
17687 Nov 01 16:54:13.012 INFO request completed, latency_us: 230, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9f1b23cc-4ec1-4227-9b3d-5c5a0c2e13d7, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:54:13.012769181Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
17688 
17689 [8830] Exited with: ExitStatus(unix_wait_status(9))
17690 [8830][2] reports Exit
17691 Nov 01 16:54:17.014 INFO request completed, latency_us: 195, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9a3b1578-9081-4510-91e2-7723af9d088c, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
17692 001/300 Write block 1315 len 36864 data: 4 4 5 4 4 5 4 4 6
17693 002/300 Read block 1585 len 32768
17694 003/300 Write block 3287 len 20480 data: 4 5 5 6 6
17695 004/300 Read block 2451 len 20480
17696 005/300 Read block 2422 len 16384
17697 006/300 Read block 4667 len 28672
17698 007/300 Flush
17699 008/300 Write block 4226 len 36864 data: 5 5 5 3 3 3 3 2 3
17700 009/300 Read block 4826 len 32768
17701 010/300 Read block 4353 len 32768
17702 011/300 Write block 1891 len 16384 data: 2 2 2 2
17703 012/300 Write block 731 len 24576 data: 5 7 6 6 6 6
17704 013/300 Read block 307 len 24576
17705 014/300 Read block 4315 len 20480
17706 015/300 Write block 1594 len 8192 data: 6 6
17707 016/300 Read block 1117 len 20480
17708 017/300 Flush
17709 018/300 Write block 4344 len 24576 data: 3 3 3 2 2 4
17710 019/300 Write block 4209 len 16384 data: 5 5 4 4
17711 020/300 Write block 2471 len 36864 data: 3 4 5 6 7 8 8 7 7
17712 021/300 Read block 3392 len 4096
17713 022/300 Read block 2897 len 36864
17714 023/300 Read block 1715 len 12288
17715 024/300 Write block 1499 len 4096 data: 5
17716 025/300 Read block 3059 len 40960
17717 026/300 Read block 2696 len 16384
17718 027/300 Read block 3979 len 24576
17719 028/300 Read block 2566 len 28672
17720 029/300 Write block 4495 len 16384 data: 4 3 3 4
17721 030/300 Read block 2146 len 12288
17722 031/300 Read block 3447 len 36864
17723 032/300 Write block 661 len 28672 data: 7 5 4 4 4 2 2
17724 033/300 Write block 3361 len 40960 data: 4 4 4 5 5 5 5 5 4 5
17725 034/300 Read block 1838 len 4096
17726 035/300 Write block 3116 len 32768 data: 2 2 2 2 2 2 4 3
17727 036/300 Read block 3627 len 16384
17728 037/300 Read block 289 len 28672
17729 038/300 Write block 3153 len 4096 data: 4
17730 039/300 Read block 4310 len 12288
17731 040/300 Read block 2994 len 8192
17732 041/300 Read block 1992 len 4096
17733 042/300 Write block 2096 len 20480 data: 6 4 3 2 2
17734 043/300 Read block 853 len 24576
17735 044/300 Read block 3022 len 8192
17736 045/300 Read block 1240 len 8192
17737 046/300 Read block 1937 len 28672
17738 047/300 Read block 428 len 32768
17739 048/300 Flush
17740 049/300 Write block 299 len 4096 data: 4
17741 050/300 Read block 110 len 16384
17742 051/300 Write block 455 len 4096 data: 5
17743 052/300 Read block 641 len 16384
17744 053/300 Write block 4993 len 20480 data: 4 4 6 6 6
17745 054/300 Read block 1822 len 16384
17746 055/300 Write block 91 len 40960 data: 7 6 6 5 5 6 5 5 5 4
17747 056/300 Read block 4665 len 4096
17748 057/300 Write block 3272 len 12288 data: 3 3 5
17749 058/300 Read block 1827 len 16384
17750 059/300 Read block 3605 len 40960
17751 060/300 Write block 1213 len 40960 data: 6 6 5 4 4 5 4 4 5 3
17752 061/300 Write block 3863 len 32768 data: 7 8 7 8 7 6 5 6
17753 062/300 Read block 2012 len 32768
17754 063/300 Write block 3059 len 20480 data: 3 3 3 3 2
17755 064/300 Flush
17756 065/300 Read block 4761 len 12288
17757 066/300 Read block 3447 len 16384
17758 067/300 Write block 2343 len 24576 data: 3 3 3 2 2 3
17759 068/300 Read block 2629 len 8192
17760 069/300 Write block 4715 len 40960 data: 3 3 4 4 4 3 3 3 2 3
17761 070/300 Read block 18 len 36864
17762 071/300 Flush
17763 072/300 Read block 1300 len 16384
17764 073/300 Read block 4663 len 28672
17765 074/300 Write block 849 len 8192 data: 8 9
17766 075/300 Write block 4432 len 12288 data: 6 5 5
17767 076/300 Read block 4326 len 24576
17768 077/300 Read block 884 len 4096
17769 078/300 Flush
17770 079/300 Read block 1810 len 24576
17771 080/300 Read block 3950 len 4096
17772 081/300 Write block 657 len 8192 data: 6 7
17773 082/300 Write block 985 len 24576 data: 3 3 3 3 4 4
17774 083/300 Write block 4762 len 8192 data: 7 6
17775 084/300 Read block 1351 len 12288
17776 085/300 Read block 4828 len 20480
17777 086/300 Write block 360 len 28672 data: 2 2 2 2 3 4 3
17778 087/300 Write block 149 len 16384 data: 2 2 2 2
17779 088/300 Write block 4658 len 40960 data: 3 4 4 5 5 5 6 4 4 4
17780 089/300 Write block 3882 len 36864 data: 4 4 3 3 3 3 2 2 2
17781 090/300 Write block 2076 len 32768 data: 5 5 2 2 3 3 3 3
17782 091/300 Read block 548 len 32768
17783 092/300 Flush
17784 093/300 Write block 3017 len 40960 data: 4 5 5 5 4 4 4 2 2 3
17785 094/300 Write block 730 len 24576 data: 5 6 8 7 7 7
17786 095/300 Write block 399 len 16384 data: 3 2 2 2
17787 096/300 Write block 3687 len 40960 data: 4 5 5 5 4 5 4 5 6 7
17788 097/300 Write block 1091 len 28672 data: 6 5 5 4 3 2 3
17789 098/300 Read block 420 len 24576
17790 099/300 Read block 760 len 28672
17791 100/300 Write block 1280 len 16384 data: 4 4 4 4
17792 101/300 Flush
17793 102/300 Read block 892 len 16384
17794 103/300 Write block 4739 len 16384 data: 6 6 6 5
17795 104/300 Read block 2043 len 36864
17796 105/300 Write block 4304 len 40960 data: 4 3 3 4 4 5 7 8 8 8
17797 106/300 Write block 2887 len 24576 data: 4 5 6 6 5 4
17798 107/300 Write block 4330 len 8192 data: 3 4
17799 108/300 Read block 2855 len 24576
17800 109/300 Write block 3756 len 16384 data: 5 3 3 3
17801 110/300 Read block 4987 len 8192
17802 111/300 Read block 2415 len 4096
17803 112/300 Read block 2273 len 24576
17804 113/300 Read block 4750 len 12288
17805 114/300 Write block 4 len 20480 data: 3 3 3 3 4
17806 115/300 Write block 4729 len 32768 data: 4 4 3 2 4 5 5 5
17807 116/300 Read block 3745 len 12288
17808 117/300 Write block 1831 len 20480 data: 2 3 3 3 2
17809 118/300 Read block 3438 len 8192
17810 119/300 Write block 4203 len 32768 data: 6 6 5 5 4 3 6 6
17811 120/300 Read block 104 len 4096
17812 121/300 Read block 4672 len 24576
17813 122/300 Write block 1952 len 16384 data: 5 4 2 2
17814 123/300 Read block 3249 len 20480
17815 124/300 Read block 4065 len 4096
17816 125/300 Read block 55 len 40960
17817 126/300 Write block 1992 len 28672 data: 3 3 2 2 2 3 3
17818 127/300 Flush
17819 128/300 Read block 144 len 12288
17820 129/300 Read block 267 len 36864
17821 130/300 Read block 3587 len 20480
17822 131/300 Read block 220 len 12288
17823 132/300 Read block 4494 len 28672
17824 133/300 Read block 2552 len 24576
17825 134/300 Read block 4027 len 28672
17826 135/300 Read block 411 len 20480
17827 136/300 Flush
17828 137/300 Read block 2624 len 4096
17829 138/300 Read block 4491 len 4096
17830 139/300 Read block 2960 len 40960
17831 140/300 Write block 1632 len 28672 data: 3 3 3 3 3 3 3
17832 141/300 Read block 899 len 40960
17833 142/300 Read block 3823 len 32768
17834 143/300 Write block 3184 len 8192 data: 6 5
17835 144/300 Write block 1533 len 40960 data: 3 3 2 2 3 3 4 4 4 3
17836 145/300 Read block 56 len 32768
17837 146/300 Read block 4030 len 36864
17838 147/300 Read block 4632 len 40960
17839 148/300 Read block 4987 len 32768
17840 149/300 Read block 2755 len 28672
17841 150/300 Write block 1051 len 28672 data: 3 4 5 4 3 4 5
17842 151/300 Write block 4362 len 16384 data: 6 4 3 3
17843 152/300 Read block 4883 len 36864
17844 153/300 Write block 1069 len 32768 data: 4 4 4 4 4 4 4 5
17845 154/300 Read block 2625 len 24576
17846 155/300 Write block 4866 len 12288 data: 2 2 2
17847 156/300 Flush
17848 157/300 Write block 2965 len 20480 data: 3 4 4 4 4
17849 158/300 Read block 3433 len 24576
17850 159/300 Read block 3563 len 24576
17851 160/300 Read block 676 len 8192
17852 161/300 Read block 3866 len 16384
17853 162/300 Read block 3830 len 32768
17854 163/300 Read block 2741 len 32768
17855 164/300 Read block 1036 len 32768
17856 165/300 Read block 1857 len 12288
17857 166/300 Write block 4882 len 12288 data: 3 4 4
17858 167/300 Write block 1729 len 36864 data: 3 2 2 2 2 2 3 3 4
17859 168/300 Read block 4645 len 20480
17860 169/300 Write block 2161 len 24576 data: 3 3 3 3 3 2
17861 170/300 Write block 3766 len 20480 data: 3 3 3 3 3
17862 171/300 Read block 3183 len 4096
17863 172/300 Read block 2700 len 20480
17864 173/300 Read block 1327 len 28672
17865 174/300 Flush
17866 175/300 Read block 284 len 16384
17867 176/300 Write block 3307 len 16384 data: 3 4 4 4
17868 177/300 Read block 23 len 12288
17869 178/300 Read block 1489 len 16384
17870 179/300 Write block 1839 len 8192 data: 4 4
17871 180/300 Write block 3959 len 40960 data: 3 3 4 5 4 5 5 4 4 4
17872 181/300 Write block 3838 len 12288 data: 3 3 3
17873 182/300 Read block 4448 len 16384
17874 183/300 Write block 4597 len 20480 data: 3 3 3 3 2
17875 184/300 Flush
17876 185/300 Flush
17877 186/300 Read block 3080 len 4096
17878 187/300 Write block 3011 len 4096 data: 3
17879 188/300 Read block 3723 len 16384
17880 189/300 Write block 3132 len 20480 data: 4 5 6 6 6
17881 190/300 Write block 306 len 32768 data: 3 3 2 2 2 2 2 2
17882 191/300 Read block 2056 len 8192
17883 192/300 Write block 3341 len 8192 data: 3 3
17884 193/300 Write block 1149 len 20480 data: 4 4 5 4 5
17885 194/300 Flush
17886 195/300 Flush
17887 196/300 Flush
17888 197/300 Read block 232 len 4096
17889 198/300 Flush
17890 199/300 Write block 4886 len 20480 data: 4 4 5 4 4
17891 200/300 Flush
17892 201/300 Read block 2073 len 40960
17893 202/300 Read block 3288 len 12288
17894 203/300 Write block 3946 len 32768 data: 5 5 4 5 5 5 4 4
17895 204/300 Write block 3163 len 28672 data: 4 4 4 2 2 2 2
17896 205/300 Write block 28 len 24576 data: 3 2 2 2 2 2
17897 206/300 Write block 4449 len 32768 data: 5 5 6 6 5 5 5 3
17898 207/300 Read block 550 len 16384
17899 208/300 Write block 3930 len 24576 data: 5 5 5 4 4 3
17900 209/300 Read block 3832 len 16384
17901 210/300 Write block 3692 len 16384 data: 6 5 6 7
17902 211/300 Read block 3705 len 4096
17903 212/300 Flush
17904 213/300 Write block 3853 len 36864 data: 3 4 4 4 4 5 6 8 7
17905 214/300 Write block 3779 len 16384 data: 3 3 3 4
17906 215/300 Flush
17907 216/300 Read block 2236 len 8192
17908 217/300 Read block 1326 len 4096
17909 218/300 Write block 3763 len 36864 data: 3 3 3 4 4 4 4 4 2
17910 219/300 Read block 2273 len 16384
17911 220/300 Read block 590 len 36864
17912 221/300 Write block 4866 len 28672 data: 3 3 3 3 4 4 5
17913 222/300 Read block 684 len 20480
17914 223/300 Write block 2911 len 28672 data: 2 2 2 3 3 3 3
17915 224/300 Write block 89 len 8192 data: 5 7
17916 225/300 Read block 4640 len 4096
17917 226/300 Write block 2379 len 24576 data: 2 2 2 2 3 3
17918 227/300 Write block 269 len 40960 data: 4 4 5 4 3 3 3 3 4 4
17919 228/300 Read block 1490 len 28672
17920 229/300 Read block 2839 len 28672
17921 230/300 Write block 4015 len 8192 data: 2 2
17922 231/300 Read block 677 len 24576
17923 232/300 Read block 2879 len 24576
17924 233/300 Write block 3369 len 24576 data: 5 6 5 6 7 5
17925 234/300 Read block 3352 len 8192
17926 235/300 Read block 1200 len 28672
17927 236/300 Write block 1317 len 8192 data: 6 5
17928 237/300 Read block 3423 len 36864
17929 238/300 Write block 276 len 28672 data: 4 5 5 5 3 3 3
17930 239/300 Write block 1586 len 24576 data: 4 5 5 5 5 6
17931 240/300 Flush
17932 241/300 Write block 3530 len 36864 data: 6 4 4 4 3 3 3 3 4
17933 242/300 Flush
17934 243/300 Write block 2578 len 24576 data: 3 3 4 4 4 4
17935 244/300 Write block 768 len 24576 data: 5 4 3 3 3 4
17936 245/300 Flush
17937 246/300 Write block 3474 len 12288 data: 5 5 5
17938 247/300 Write block 1165 len 4096 data: 3
17939 248/300 Read block 453 len 20480
17940 249/300 Write block 4700 len 32768 data: 2 2 2 2 2 2 2 2
17941 250/300 Write block 2567 len 4096 data: 7
17942 251/300 Write block 3672 len 28672 data: 5 4 4 3 3 3 3
17943 252/300 Write block 830 len 28672 data: 4 4 4 2 3 3 3
17944 253/300 Write block 1119 len 12288 data: 4 5 4
17945 254/300 Flush
17946 255/300 Read block 2896 len 36864
17947 256/300 Write block 4193 len 36864 data: 3 4 4 5 5 4 4 5 4
17948 257/300 Read block 126 len 16384
17949 258/300 Write block 1764 len 4096 data: 5
17950 259/300 Write block 1688 len 4096 data: 2
17951 260/300 Read block 748 len 12288
17952 261/300 Read block 4355 len 8192
17953 262/300 Read block 1136 len 32768
17954 263/300 Write block 3509 len 12288 data: 3 5 6
17955 264/300 Read block 552 len 12288
17956 265/300 Read block 3004 len 36864
17957 266/300 Write block 4684 len 36864 data: 5 6 6 5 5 4 4 3 3
17958 267/300 Read block 740 len 4096
17959 268/300 Write block 3550 len 24576 data: 4 4 5 4 5 5
17960 269/300 Read block 4876 len 4096
17961 270/300 Read block 4849 len 20480
17962 271/300 Read block 1340 len 36864
17963 272/300 Flush
17964 273/300 Write block 320 len 4096 data: 2
17965 274/300 Write block 2787 len 24576 data: 5 5 4 4 5 7
17966 275/300 Read block 3010 len 36864
17967 276/300 Read block 3632 len 20480
17968 277/300 Read block 553 len 28672
17969 278/300 Read block 1820 len 12288
17970 279/300 Write block 3024 len 20480 data: 3 3 4 3 3
17971 280/300 Read block 2984 len 4096
17972 281/300 Read block 2872 len 4096
17973 282/300 Write block 4523 len 32768 data: 4 4 5 4 5 5 4 5
17974 283/300 Read block 1099 len 4096
17975 284/300 Write block 3653 len 36864 data: 2 2 3 3 3 3 3 4 3
17976 285/300 Read block 1867 len 28672
17977 286/300 Read block 376 len 36864
17978 287/300 Flush
17979 288/300 Write block 676 len 16384 data: 5 4 4 3
17980 289/300 Write block 2101 len 36864 data: 2 2 2 2 2 3 4 4 4
17981 290/300 Write block 3964 len 20480 data: 6 6 5 5 5
17982 291/300 Read block 323 len 32768
17983 292/300 Write block 1030 len 4096 data: 4
17984 293/300 Write block 3132 len 40960 data: 5 6 7 7 7 5 5 4 4 3
17985 294/300 Write block 1519 len 40960 data: 3 4 5 5 5 6 6 6 5 5
17986 295/300 Write block 3893 len 16384 data: 3 3 3 3
17987 296/300 Flush
17988 297/300 Write block 969 len 36864 data: 4 6 5 5 5 5 6 4 4
17989 298/300 Read block 3832 len 24576
17990 299/300 Read block 872 len 24576
17991 300/300 Read block 4913 len 12288
17992 301/300 Read block 4545 len 24576
17993 Main task has work to do, go find it
17994 got dsc Start(2)
17995 start 2
17996 Nov 01 16:54:17.306 INFO request completed, latency_us: 224, response_code: 204, uri: /start/cid/2, method: POST, req_id: 3b0e01ad-c44f-4044-93c0-e9c7734bb735, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
17997 Replay: started 2, returned:Ok(())
17998 ----------------------------------------------------------------
17999 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18000 States: Active Active Offline
18001 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18002 5836 Acked Write 9 Done Done Sent false
18003 5837 Acked Read 8 Done Done Sent false
18004 5838 Acked Write 5 Done Done Sent false
18005 5839 Acked Read 5 Done Done Sent false
18006 5840 Acked Read 4 Done Done Sent false
18007 5841 Acked Read 7 Done Done Sent false
18008 5842 Acked Flush 0 Done Done Sent false
18009 5843 Acked Write 9 Done Done Sent false
18010 5844 Acked Read 8 Done Done Sent false
18011 5845 Acked Read 8 Done Done Sent false
18012 5846 Acked Write 4 Done Done Sent false
18013 5847 Acked Write 6 Done Done Sent false
18014 5848 Acked Read 6 Done Done Sent false
18015 5849 Acked Read 5 Done Done Sent false
18016 5850 Acked Write 2 Done Done Sent false
18017 5851 Acked Read 5 Done Done Sent false
18018 5852 Acked Flush 0 Done Done Sent false
18019 5853 Acked Write 6 Done Done Sent false
18020 5854 Acked Write 4 Done Done Sent false
18021 5855 Acked Write 9 Done Done Sent false
18022 5856 Acked Read 1 Done Done Sent false
18023 5857 Acked Read 9 Done Done Sent false
18024 5858 Acked Read 3 Done Done Sent false
18025 5859 Acked Write 1 Done Done Sent false
18026 5860 Acked Read 10 Done Done Sent false
18027 5861 Acked Read 4 Done Done Sent false
18028 5862 Acked Read 6 Done Done Sent false
18029 5863 Acked Read 7 Done Done Sent false
18030 5864 Acked Write 4 Done Done Sent false
18031 5865 Acked Read 3 Done Done Sent false
18032 5866 Acked Read 9 Done Done Sent false
18033 5867 Acked Write 7 Done Done Sent false
18034 5868 Acked Write 10 Done Done Sent false
18035 5869 Acked Read 1 Done Done Sent false
18036 5870 Acked Write 8 Done Done Sent false
18037 5871 Acked Read 4 Done Done Sent false
18038 5872 Acked Read 7 Done Done Sent false
18039 5873 Acked Write 1 Done Done Sent false
18040 5874 Acked Read 3 Done Done Sent false
18041 5875 Acked Read 2 Done Done Sent false
18042 5876 Acked Read 1 Done Done Sent false
18043 5877 Acked Write 5 Done Done Sent false
18044 5878 Acked Read 6 Done Done Sent false
18045 5879 Acked Read 2 Done Done Sent false
18046 5880 Acked Read 2 Done Done Sent false
18047 5881 Acked Read 7 Done Done Sent false
18048 5882 Acked Read 8 Done Done Sent false
18049 5883 Acked Flush 0 Done Done Sent false
18050 5884 Acked Write 1 Done Done Sent false
18051 5885 Acked Read 4 Done Done Sent false
18052 5886 Acked Write 1 Done Done Sent false
18053 5887 Acked Read 4 Done Done Sent false
18054 5888 Acked Write 5 Done Done Sent false
18055 5889 Acked Read 4 Done Done Sent false
18056 5890 Acked Write 10 Done Done Sent false
18057 5891 Acked Read 1 Done Done Sent false
18058 5892 Acked Write 3 Done Done Sent false
18059 5893 Acked Read 4 Done Done Sent false
18060 5894 Acked Read 10 Done Done Sent false
18061 5895 Acked Write 10 Done Done Sent false
18062 5896 Acked Write 8 Done Done Sent false
18063 5897 Acked Read 8 Done Done Sent false
18064 5898 Acked Write 5 Done Done Sent false
18065 5899 Acked Flush 0 Done Done Sent false
18066 5900 Acked Read 3 Done Done Sent false
18067 5901 Acked Read 4 Done Done Sent false
18068 5902 Acked Write 6 Done Done Sent false
18069 5903 Acked Read 2 Done Done Sent false
18070 5904 Acked Write 10 Done Done Sent false
18071 5905 Acked Read 9 Done Done Sent false
18072 5906 Acked Flush 0 Done Done Sent false
18073 5907 Acked Read 4 Done Done Sent false
18074 5908 Acked Read 7 Done Done Sent false
18075 5909 Acked Write 2 Done Done Sent false
18076 5910 Acked Write 3 Done Done Sent false
18077 5911 Acked Read 6 Done Done Sent false
18078 5912 Acked Read 1 Done Done Sent false
18079 5913 Acked Flush 0 Done Done Sent false
18080 5914 Acked Read 6 Done Done Sent false
18081 5915 Acked Read 1 Done Done Sent false
18082 5916 Acked Write 2 Done Done Sent false
18083 5917 Acked Write 6 Done Done Sent false
18084 5918 Acked Write 2 Done Done Sent false
18085 5919 Acked Read 3 Done Done Sent false
18086 5920 Acked Read 5 Done Done Sent false
18087 5921 Acked Write 7 Done Done Sent false
18088 5922 Acked Write 4 Done Done Sent false
18089 5923 Acked Write 10 Done Done Sent false
18090 5924 Acked Write 9 Done Done Sent false
18091 5925 Acked Write 8 Done Done Sent false
18092 5926 Acked Read 8 Done Done Sent false
18093 5927 Acked Flush 0 Done Done Sent false
18094 5928 Acked Write 10 Done Done Sent false
18095 5929 Acked Write 6 Done Done Sent false
18096 5930 Acked Write 4 Done Done Sent false
18097 5931 Acked Write 10 Done Done Sent false
18098 5932 Acked Write 7 Done Done Sent false
18099 5933 Acked Read 6 Done Done Sent false
18100 5934 Acked Read 7 Done Done Sent false
18101 5935 Acked Write 4 Done Done Sent false
18102 5936 Acked Flush 0 Done Done Sent false
18103 5937 Acked Read 4 Done Done Sent false
18104 5938 Acked Write 4 Done Done Sent false
18105 5939 Acked Read 9 Done Done Sent false
18106 5940 Acked Write 10 Done Done Sent false
18107 5941 Acked Write 6 Done Done Sent false
18108 5942 Acked Write 2 Done Done Sent false
18109 5943 Acked Read 6 Done Done Sent false
18110 5944 Acked Write 4 Done Done Sent false
18111 5945 Acked Read 2 Done Done Sent false
18112 5946 Acked Read 1 Done Done Sent false
18113 5947 Acked Read 6 Done Done Sent false
18114 5948 Acked Read 3 Done Done Sent false
18115 5949 Acked Write 5 Done Done Sent false
18116 5950 Acked Write 8 Done Done Sent false
18117 5951 Acked Read 3 Done Done Sent false
18118 5952 Acked Write 5 Done Done Sent false
18119 5953 Acked Read 2 Done Done Sent false
18120 5954 Acked Write 8 Done Done Sent false
18121 5955 Acked Read 1 Done Done Sent false
18122 5956 Acked Read 6 Done Done Sent false
18123 5957 Acked Write 4 Done Done Sent false
18124 5958 Acked Read 5 Done Done Sent false
18125 5959 Acked Read 1 Done Done Sent false
18126 5960 Acked Read 10 Done Done Sent false
18127 5961 Acked Write 7 Done Done Sent false
18128 5962 Acked Flush 0 Done Done Sent false
18129 5963 Acked Read 3 Done Done Sent false
18130 5964 Acked Read 9 Done Done Sent false
18131 5965 Acked Read 5 Done Done Sent false
18132 5966 Acked Read 3 Done Done Sent false
18133 5967 Acked Read 7 Done Done Sent false
18134 5968 Acked Read 6 Done Done Sent false
18135 5969 Acked Read 7 Done Done Sent false
18136 5970 Acked Read 5 Done Done Sent false
18137 5971 Acked Flush 0 Done Done Sent false
18138 5972 Acked Read 1 Done Done Sent false
18139 5973 Acked Read 1 Done Done Sent false
18140 5974 Acked Read 10 Done Done Sent false
18141 5975 Acked Write 7 Done Done Sent false
18142 5976 Acked Read 10 Done Done Sent false
18143 5977 Acked Read 8 Done Done Sent false
18144 5978 Acked Write 2 Done Done Sent false
18145 5979 Acked Write 10 Done Done Sent false
18146 5980 Acked Read 8 Done Done Sent false
18147 5981 Acked Read 9 Done Done Sent false
18148 5982 Acked Read 10 Done Done Sent false
18149 5983 Acked Read 8 Done Done Sent false
18150 5984 Acked Read 7 Done Done Sent false
18151 5985 Acked Write 7 Done Done Sent false
18152 5986 Acked Write 4 Done Done Sent false
18153 5987 Acked Read 9 Done Done Sent false
18154 5988 Acked Write 8 Done Done Sent false
18155 5989 Acked Read 6 Done Done Sent false
18156 5990 Acked Write 3 Done Done Sent false
18157 5991 Acked Flush 0 Done Done Sent false
18158 5992 Acked Write 5 Done Done Sent false
18159 5993 Acked Read 6 Done Done Sent false
18160 5994 Acked Read 6 Done Done Sent false
18161 5995 Acked Read 2 Done Done Sent false
18162 5996 Acked Read 4 Done Done Sent false
18163 5997 Acked Read 8 Done Done Sent false
18164 5998 Acked Read 8 Done Done Sent false
18165 5999 Acked Read 8 Done Done Sent false
18166 6000 Acked Read 3 Done Done Sent false
18167 6001 Acked Write 3 Done Done Sent false
18168 6002 Acked Write 9 Done Done Sent false
18169 6003 Acked Read 5 Done Done Sent false
18170 6004 Acked Write 6 Done Done Sent false
18171 6005 Acked Write 5 Done Done Sent false
18172 6006 Acked Read 1 Done Done Sent false
18173 6007 Acked Read 5 Done Done Sent false
18174 6008 Acked Read 7 Done Done Sent false
18175 6009 Acked Flush 0 Done Done Sent false
18176 6010 Acked Read 4 Done Done Sent false
18177 6011 Acked Write 4 Done Done Sent false
18178 6012 Acked Read 3 Done Done Sent false
18179 6013 Acked Read 4 Done Done Sent false
18180 6014 Acked Write 2 Done Done Sent false
18181 6015 Acked Write 10 Done Done Sent false
18182 6016 Acked Write 3 Done Done Sent false
18183 6017 Acked Read 4 Done Done Sent false
18184 6018 Acked Write 5 Done Done Sent false
18185 6019 Acked Flush 0 Done Done Sent false
18186 6020 Acked Flush 0 Done Done Sent false
18187 6021 Acked Read 1 Done Done Sent false
18188 6022 Acked Write 1 Done Done Sent false
18189 6023 Acked Read 4 Done Done Sent false
18190 6024 Acked Write 5 Done Done Sent false
18191 6025 Acked Write 8 Done Done Sent false
18192 6026 Acked Read 2 Done Done Sent false
18193 6027 Acked Write 2 Done Done Sent false
18194 6028 Acked Write 5 Done Done Sent false
18195 6029 Acked Flush 0 Done Done Sent false
18196 6030 Acked Flush 0 Done Done Sent false
18197 6031 Acked Flush 0 Done Done Sent false
18198 6032 Acked Read 1 Done Done Sent false
18199 6033 Acked Flush 0 Done Done Sent false
18200 6034 Acked Write 5 Done Done Sent false
18201 6035 Acked Flush 0 Done Done Sent false
18202 6036 Acked Read 10 Done Done Sent false
18203 6037 Acked Read 3 Done Done Sent false
18204 6038 Acked Write 8 Done Done Sent false
18205 6039 Acked Write 7 Done Done Sent false
18206 6040 Acked Write 6 Done Done Sent false
18207 6041 Acked Write 8 Done Done Sent false
18208 6042 Acked Read 4 Done Done Sent false
18209 6043 Acked Write 6 Done Done Sent false
18210 6044 Acked Read 4 Done Done Sent false
18211 6045 Acked Write 4 Done Done Sent false
18212 6046 Acked Read 1 Done Done Sent false
18213 6047 Acked Flush 0 Done Done Sent false
18214 6048 Acked Write 9 Done Done Sent false
18215 6049 Acked Write 4 Done Done Sent false
18216 6050 Acked Flush 0 Done Done Sent false
18217 6051 Acked Read 2 Done Done Sent false
18218 6052 Acked Read 1 Done Done Sent false
18219 6053 Acked Write 9 Done Done Sent false
18220 6054 Acked Read 4 Done Done Sent false
18221 6055 Acked Read 9 Done Done Sent false
18222 6056 Acked Write 7 Done Done Sent false
18223 6057 Acked Read 5 Done Done Sent false
18224 6058 Acked Write 7 Done Done Sent false
18225 6059 Acked Write 2 Done Done Sent false
18226 6060 Acked Read 1 Done Done Sent false
18227 6061 Acked Write 6 Done Done Sent false
18228 6062 Acked Write 10 Done Done Sent false
18229 6063 Acked Read 7 Done Done Sent false
18230 6064 Acked Read 7 Done Done Sent false
18231 6065 Acked Write 2 Done Done Sent false
18232 6066 Acked Read 6 Done Done Sent false
18233 6067 Acked Read 6 Done Done Sent false
18234 6068 Acked Write 6 Done Done Sent false
18235 6069 Acked Read 2 Done Done Sent false
18236 6070 Acked Read 7 Done Done Sent false
18237 6071 Acked Write 2 Done Done Sent false
18238 6072 Acked Read 9 Done Done Sent false
18239 6073 Acked Write 7 Done Done Sent false
18240 6074 Acked Write 6 Done Done Sent false
18241 6075 Acked Flush 0 Done Done Sent false
18242 6076 Acked Write 9 Done Done Sent false
18243 6077 Acked Flush 0 Done Done Sent false
18244 6078 Acked Write 6 Done Done Sent false
18245 6079 Acked Write 6 Done Done Sent false
18246 6080 Acked Flush 0 Done Done Sent false
18247 6081 Acked Write 3 Done Done Sent false
18248 6082 Acked Write 1 Done Done Sent false
18249 6083 Acked Read 5 Done Done Sent false
18250 6084 Acked Write 8 Done Done Sent false
18251 6085 Acked Write 1 Done Done Sent false
18252 6086 Acked Write 7 Done Done Sent false
18253 6087 Acked Write 7 Done Done Sent false
18254 6088 Acked Write 3 Done Done Sent false
18255 6089 Acked Flush 0 Done Done Sent false
18256 6090 Acked Read 9 Done Done Sent false
18257 6091 Acked Write 9 Done Done Sent false
18258 6092 Acked Read 4 Done Done Sent false
18259 6093 Acked Write 1 Done Done Sent false
18260 6094 Acked Write 1 Done Done Sent false
18261 6095 Acked Read 3 Done Done Sent false
18262 6096 Acked Read 2 Done Done Sent false
18263 6097 Acked Read 8 Done Done Sent false
18264 6098 Acked Write 3 Done Done Sent false
18265 6099 Acked Read 3 Done Done Sent false
18266 6100 Acked Read 9 Done Done Sent false
18267 6101 Acked Write 9 Done Done Sent false
18268 6102 Acked Read 1 Done Done Sent false
18269 6103 Acked Write 6 Done Done Sent false
18270 6104 Acked Read 1 Done Done Sent false
18271 6105 Acked Read 5 Done Done Sent false
18272 6106 Acked Read 9 Done Done Sent false
18273 6107 Acked Flush 0 Done Done Sent false
18274 6108 Acked Write 1 Done Done Sent false
18275 6109 Acked Write 6 Done Done Sent false
18276 6110 Acked Read 9 Done Done Sent false
18277 6111 Acked Read 5 Done Done Sent false
18278 6112 Acked Read 7 Done Done Sent false
18279 6113 Acked Read 3 Done Done Sent false
18280 6114 Acked Write 5 Done Done Sent false
18281 6115 Acked Read 1 Done Done Sent false
18282 6116 Acked Read 1 Done Done Sent false
18283 6117 Acked Write 8 Done Done Sent false
18284 6118 Acked Read 1 Done Done Sent false
18285 6119 Acked Write 9 Done Done Sent false
18286 6120 Acked Read 7 Done Done Sent false
18287 6121 Acked Read 9 Done Done Sent false
18288 6122 Acked Flush 0 Done Done Sent false
18289 6123 Acked Write 4 Done Done Sent false
18290 6124 Acked Write 9 Done Done Sent false
18291 6125 Acked Write 5 Done Done Sent false
18292 6126 Acked Read 8 Done Done Sent false
18293 6127 Acked Write 1 Done Done Sent false
18294 6128 Acked Write 10 Done Done Sent false
18295 6129 Acked Write 10 Done Done Sent false
18296 6130 Acked Write 4 Done Done Sent false
18297 6131 Acked Flush 0 Done Done Sent false
18298 6132 Acked Write 9 Done Done Sent false
18299 6133 Acked Read 6 Done Done Sent false
18300 6134 Acked Read 6 Done Done Sent false
18301 6135 Acked Read 3 Done Done Sent false
18302 6136 Acked Read 6 Done Done Sent false
18303 STATES DS:0 DS:1 DS:2 TOTAL
18304 Sent 0 0 301 301
18305 Done 301 301 0 602
18306 Skipped 0 0 0 0
18307 Error 0 0 0 0
18308 Last Flush: 6131 6131 5835
18309 Downstairs last five completed: 5835 5834 5833 5832 5831
18310 Upstairs last five completed: 6136 6135 6134 6133 6132
18311 CLIENT: Up:0 ds:301 act:2
18312 [8830] got start action
18313 [8830] I am going to restart
18314 Starting downstairs at port 8830
18315 Make output file at "/tmp/dsc/downstairs-8830.txt"
18316 [8830][2] reports Starting
18317 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1215)
18318 [8830][2] reports Running
18319 ----------------------------------------------------------------
18320 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
18321 States: Active Active Offline
18322 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18323 5836 Acked Write 9 Done Done Sent false
18324 5837 Acked Read 8 Done Done Sent false
18325 5838 Acked Write 5 Done Done Sent false
18326 5839 Acked Read 5 Done Done Sent false
18327 5840 Acked Read 4 Done Done Sent false
18328 5841 Acked Read 7 Done Done Sent false
18329 5842 Acked Flush 0 Done Done Sent false
18330 5843 Acked Write 9 Done Done Sent false
18331 5844 Acked Read 8 Done Done Sent false
18332 5845 Acked Read 8 Done Done Sent false
18333 5846 Acked Write 4 Done Done Sent false
18334 5847 Acked Write 6 Done Done Sent false
18335 5848 Acked Read 6 Done Done Sent false
18336 5849 Acked Read 5 Done Done Sent false
18337 5850 Acked Write 2 Done Done Sent false
18338 5851 Acked Read 5 Done Done Sent false
18339 5852 Acked Flush 0 Done Done Sent false
18340 5853 Acked Write 6 Done Done Sent false
18341 5854 Acked Write 4 Done Done Sent false
18342 5855 Acked Write 9 Done Done Sent false
18343 5856 Acked Read 1 Done Done Sent false
18344 5857 Acked Read 9 Done Done Sent false
18345 5858 Acked Read 3 Done Done Sent false
18346 5859 Acked Write 1 Done Done Sent false
18347 5860 Acked Read 10 Done Done Sent false
18348 5861 Acked Read 4 Done Done Sent false
18349 5862 Acked Read 6 Done Done Sent false
18350 5863 Acked Read 7 Done Done Sent false
18351 5864 Acked Write 4 Done Done Sent false
18352 5865 Acked Read 3 Done Done Sent false
18353 5866 Acked Read 9 Done Done Sent false
18354 5867 Acked Write 7 Done Done Sent false
18355 5868 Acked Write 10 Done Done Sent false
18356 5869 Acked Read 1 Done Done Sent false
18357 5870 Acked Write 8 Done Done Sent false
18358 5871 Acked Read 4 Done Done Sent false
18359 5872 Acked Read 7 Done Done Sent false
18360 5873 Acked Write 1 Done Done Sent false
18361 5874 Acked Read 3 Done Done Sent false
18362 5875 Acked Read 2 Done Done Sent false
18363 5876 Acked Read 1 Done Done Sent false
18364 5877 Acked Write 5 Done Done Sent false
18365 5878 Acked Read 6 Done Done Sent false
18366 5879 Acked Read 2 Done Done Sent false
18367 5880 Acked Read 2 Done Done Sent false
18368 5881 Acked Read 7 Done Done Sent false
18369 5882 Acked Read 8 Done Done Sent false
18370 5883 Acked Flush 0 Done Done Sent false
18371 5884 Acked Write 1 Done Done Sent false
18372 5885 Acked Read 4 Done Done Sent false
18373 5886 Acked Write 1 Done Done Sent false
18374 5887 Acked Read 4 Done Done Sent false
18375 5888 Acked Write 5 Done Done Sent false
18376 5889 Acked Read 4 Done Done Sent false
18377 5890 Acked Write 10 Done Done Sent false
18378 5891 Acked Read 1 Done Done Sent false
18379 5892 Acked Write 3 Done Done Sent false
18380 5893 Acked Read 4 Done Done Sent false
18381 5894 Acked Read 10 Done Done Sent false
18382 5895 Acked Write 10 Done Done Sent false
18383 5896 Acked Write 8 Done Done Sent false
18384 5897 Acked Read 8 Done Done Sent false
18385 5898 Acked Write 5 Done Done Sent false
18386 5899 Acked Flush 0 Done Done Sent false
18387 5900 Acked Read 3 Done Done Sent false
18388 5901 Acked Read 4 Done Done Sent false
18389 5902 Acked Write 6 Done Done Sent false
18390 5903 Acked Read 2 Done Done Sent false
18391 5904 Acked Write 10 Done Done Sent false
18392 5905 Acked Read 9 Done Done Sent false
18393 5906 Acked Flush 0 Done Done Sent false
18394 5907 Acked Read 4 Done Done Sent false
18395 5908 Acked Read 7 Done Done Sent false
18396 5909 Acked Write 2 Done Done Sent false
18397 5910 Acked Write 3 Done Done Sent false
18398 5911 Acked Read 6 Done Done Sent false
18399 5912 Acked Read 1 Done Done Sent false
18400 5913 Acked Flush 0 Done Done Sent false
18401 5914 Acked Read 6 Done Done Sent false
18402 5915 Acked Read 1 Done Done Sent false
18403 5916 Acked Write 2 Done Done Sent false
18404 5917 Acked Write 6 Done Done Sent false
18405 5918 Acked Write 2 Done Done Sent false
18406 5919 Acked Read 3 Done Done Sent false
18407 5920 Acked Read 5 Done Done Sent false
18408 5921 Acked Write 7 Done Done Sent false
18409 5922 Acked Write 4 Done Done Sent false
18410 5923 Acked Write 10 Done Done Sent false
18411 5924 Acked Write 9 Done Done Sent false
18412 5925 Acked Write 8 Done Done Sent false
18413 5926 Acked Read 8 Done Done Sent false
18414 5927 Acked Flush 0 Done Done Sent false
18415 5928 Acked Write 10 Done Done Sent false
18416 5929 Acked Write 6 Done Done Sent false
18417 5930 Acked Write 4 Done Done Sent false
18418 5931 Acked Write 10 Done Done Sent false
18419 5932 Acked Write 7 Done Done Sent false
18420 5933 Acked Read 6 Done Done Sent false
18421 5934 Acked Read 7 Done Done Sent false
18422 5935 Acked Write 4 Done Done Sent false
18423 5936 Acked Flush 0 Done Done Sent false
18424 5937 Acked Read 4 Done Done Sent false
18425 5938 Acked Write 4 Done Done Sent false
18426 5939 Acked Read 9 Done Done Sent false
18427 5940 Acked Write 10 Done Done Sent false
18428 5941 Acked Write 6 Done Done Sent false
18429 5942 Acked Write 2 Done Done Sent false
18430 5943 Acked Read 6 Done Done Sent false
18431 5944 Acked Write 4 Done Done Sent false
18432 5945 Acked Read 2 Done Done Sent false
18433 5946 Acked Read 1 Done Done Sent false
18434 5947 Acked Read 6 Done Done Sent false
18435 5948 Acked Read 3 Done Done Sent false
18436 5949 Acked Write 5 Done Done Sent false
18437 5950 Acked Write 8 Done Done Sent false
18438 5951 Acked Read 3 Done Done Sent false
18439 5952 Acked Write 5 Done Done Sent false
18440 5953 Acked Read 2 Done Done Sent false
18441 5954 Acked Write 8 Done Done Sent false
18442 5955 Acked Read 1 Done Done Sent false
18443 5956 Acked Read 6 Done Done Sent false
18444 5957 Acked Write 4 Done Done Sent false
18445 5958 Acked Read 5 Done Done Sent false
18446 5959 Acked Read 1 Done Done Sent false
18447 5960 Acked Read 10 Done Done Sent false
18448 5961 Acked Write 7 Done Done Sent false
18449 5962 Acked Flush 0 Done Done Sent false
18450 5963 Acked Read 3 Done Done Sent false
18451 5964 Acked Read 9 Done Done Sent false
18452 5965 Acked Read 5 Done Done Sent false
18453 5966 Acked Read 3 Done Done Sent false
18454 5967 Acked Read 7 Done Done Sent false
18455 5968 Acked Read 6 Done Done Sent false
18456 5969 Acked Read 7 Done Done Sent false
18457 5970 Acked Read 5 Done Done Sent false
18458 5971 Acked Flush 0 Done Done Sent false
18459 5972 Acked Read 1 Done Done Sent false
18460 5973 Acked Read 1 Done Done Sent false
18461 5974 Acked Read 10 Done Done Sent false
18462 5975 Acked Write 7 Done Done Sent false
18463 5976 Acked Read 10 Done Done Sent false
18464 5977 Acked Read 8 Done Done Sent false
18465 5978 Acked Write 2 Done Done Sent false
18466 5979 Acked Write 10 Done Done Sent false
18467 5980 Acked Read 8 Done Done Sent false
18468 5981 Acked Read 9 Done Done Sent false
18469 5982 Acked Read 10 Done Done Sent false
18470 5983 Acked Read 8 Done Done Sent false
18471 5984 Acked Read 7 Done Done Sent false
18472 5985 Acked Write 7 Done Done Sent false
18473 5986 Acked Write 4 Done Done Sent false
18474 5987 Acked Read 9 Done Done Sent false
18475 5988 Acked Write 8 Done Done Sent false
18476 5989 Acked Read 6 Done Done Sent false
18477 5990 Acked Write 3 Done Done Sent false
18478 5991 Acked Flush 0 Done Done Sent false
18479 5992 Acked Write 5 Done Done Sent false
18480 5993 Acked Read 6 Done Done Sent false
18481 5994 Acked Read 6 Done Done Sent false
18482 5995 Acked Read 2 Done Done Sent false
18483 5996 Acked Read 4 Done Done Sent false
18484 5997 Acked Read 8 Done Done Sent false
18485 5998 Acked Read 8 Done Done Sent false
18486 5999 Acked Read 8 Done Done Sent false
18487 6000 Acked Read 3 Done Done Sent false
18488 6001 Acked Write 3 Done Done Sent false
18489 6002 Acked Write 9 Done Done Sent false
18490 6003 Acked Read 5 Done Done Sent false
18491 6004 Acked Write 6 Done Done Sent false
18492 6005 Acked Write 5 Done Done Sent false
18493 6006 Acked Read 1 Done Done Sent false
18494 6007 Acked Read 5 Done Done Sent false
18495 6008 Acked Read 7 Done Done Sent false
18496 6009 Acked Flush 0 Done Done Sent false
18497 6010 Acked Read 4 Done Done Sent false
18498 6011 Acked Write 4 Done Done Sent false
18499 6012 Acked Read 3 Done Done Sent false
18500 6013 Acked Read 4 Done Done Sent false
18501 6014 Acked Write 2 Done Done Sent false
18502 6015 Acked Write 10 Done Done Sent false
18503 6016 Acked Write 3 Done Done Sent false
18504 6017 Acked Read 4 Done Done Sent false
18505 6018 Acked Write 5 Done Done Sent false
18506 6019 Acked Flush 0 Done Done Sent false
18507 6020 Acked Flush 0 Done Done Sent false
18508 6021 Acked Read 1 Done Done Sent false
18509 6022 Acked Write 1 Done Done Sent false
18510 6023 Acked Read 4 Done Done Sent false
18511 6024 Acked Write 5 Done Done Sent false
18512 6025 Acked Write 8 Done Done Sent false
18513 6026 Acked Read 2 Done Done Sent false
18514 6027 Acked Write 2 Done Done Sent false
18515 6028 Acked Write 5 Done Done Sent false
18516 6029 Acked Flush 0 Done Done Sent false
18517 6030 Acked Flush 0 Done Done Sent false
18518 6031 Acked Flush 0 Done Done Sent false
18519 6032 Acked Read 1 Done Done Sent false
18520 6033 Acked Flush 0 Done Done Sent false
18521 6034 Acked Write 5 Done Done Sent false
18522 6035 Acked Flush 0 Done Done Sent false
18523 6036 Acked Read 10 Done Done Sent false
18524 6037 Acked Read 3 Done Done Sent false
18525 6038 Acked Write 8 Done Done Sent false
18526 6039 Acked Write 7 Done Done Sent false
18527 6040 Acked Write 6 Done Done Sent false
18528 6041 Acked Write 8 Done Done Sent false
18529 6042 Acked Read 4 Done Done Sent false
18530 6043 Acked Write 6 Done Done Sent false
18531 6044 Acked Read 4 Done Done Sent false
18532 6045 Acked Write 4 Done Done Sent false
18533 6046 Acked Read 1 Done Done Sent false
18534 6047 Acked Flush 0 Done Done Sent false
18535 6048 Acked Write 9 Done Done Sent false
18536 6049 Acked Write 4 Done Done Sent false
18537 6050 Acked Flush 0 Done Done Sent false
18538 6051 Acked Read 2 Done Done Sent false
18539 6052 Acked Read 1 Done Done Sent false
18540 6053 Acked Write 9 Done Done Sent false
18541 6054 Acked Read 4 Done Done Sent false
18542 6055 Acked Read 9 Done Done Sent false
18543 6056 Acked Write 7 Done Done Sent false
18544 6057 Acked Read 5 Done Done Sent false
18545 6058 Acked Write 7 Done Done Sent false
18546 6059 Acked Write 2 Done Done Sent false
18547 6060 Acked Read 1 Done Done Sent false
18548 6061 Acked Write 6 Done Done Sent false
18549 6062 Acked Write 10 Done Done Sent false
18550 6063 Acked Read 7 Done Done Sent false
18551 6064 Acked Read 7 Done Done Sent false
18552 6065 Acked Write 2 Done Done Sent false
18553 6066 Acked Read 6 Done Done Sent false
18554 6067 Acked Read 6 Done Done Sent false
18555 6068 Acked Write 6 Done Done Sent false
18556 6069 Acked Read 2 Done Done Sent false
18557 6070 Acked Read 7 Done Done Sent false
18558 6071 Acked Write 2 Done Done Sent false
18559 6072 Acked Read 9 Done Done Sent false
18560 6073 Acked Write 7 Done Done Sent false
18561 6074 Acked Write 6 Done Done Sent false
18562 6075 Acked Flush 0 Done Done Sent false
18563 6076 Acked Write 9 Done Done Sent false
18564 6077 Acked Flush 0 Done Done Sent false
18565 6078 Acked Write 6 Done Done Sent false
18566 6079 Acked Write 6 Done Done Sent false
18567 6080 Acked Flush 0 Done Done Sent false
18568 6081 Acked Write 3 Done Done Sent false
18569 6082 Acked Write 1 Done Done Sent false
18570 6083 Acked Read 5 Done Done Sent false
18571 6084 Acked Write 8 Done Done Sent false
18572 6085 Acked Write 1 Done Done Sent false
18573 6086 Acked Write 7 Done Done Sent false
18574 6087 Acked Write 7 Done Done Sent false
18575 6088 Acked Write 3 Done Done Sent false
18576 6089 Acked Flush 0 Done Done Sent false
18577 6090 Acked Read 9 Done Done Sent false
18578 6091 Acked Write 9 Done Done Sent false
18579 6092 Acked Read 4 Done Done Sent false
18580 6093 Acked Write 1 Done Done Sent false
18581 6094 Acked Write 1 Done Done Sent false
18582 6095 Acked Read 3 Done Done Sent false
18583 6096 Acked Read 2 Done Done Sent false
18584 6097 Acked Read 8 Done Done Sent false
18585 6098 Acked Write 3 Done Done Sent false
18586 6099 Acked Read 3 Done Done Sent false
18587 6100 Acked Read 9 Done Done Sent false
18588 6101 Acked Write 9 Done Done Sent false
18589 6102 Acked Read 1 Done Done Sent false
18590 6103 Acked Write 6 Done Done Sent false
18591 6104 Acked Read 1 Done Done Sent false
18592 6105 Acked Read 5 Done Done Sent false
18593 6106 Acked Read 9 Done Done Sent false
18594 6107 Acked Flush 0 Done Done Sent false
18595 6108 Acked Write 1 Done Done Sent false
18596 6109 Acked Write 6 Done Done Sent false
18597 6110 Acked Read 9 Done Done Sent false
18598 6111 Acked Read 5 Done Done Sent false
18599 6112 Acked Read 7 Done Done Sent false
18600 6113 Acked Read 3 Done Done Sent false
18601 6114 Acked Write 5 Done Done Sent false
18602 6115 Acked Read 1 Done Done Sent false
18603 6116 Acked Read 1 Done Done Sent false
18604 6117 Acked Write 8 Done Done Sent false
18605 6118 Acked Read 1 Done Done Sent false
18606 6119 Acked Write 9 Done Done Sent false
18607 6120 Acked Read 7 Done Done Sent false
18608 6121 Acked Read 9 Done Done Sent false
18609 6122 Acked Flush 0 Done Done Sent false
18610 6123 Acked Write 4 Done Done Sent false
18611 6124 Acked Write 9 Done Done Sent false
18612 6125 Acked Write 5 Done Done Sent false
18613 6126 Acked Read 8 Done Done Sent false
18614 6127 Acked Write 1 Done Done Sent false
18615 6128 Acked Write 10 Done Done Sent false
18616 6129 Acked Write 10 Done Done Sent false
18617 6130 Acked Write 4 Done Done Sent false
18618 6131 Acked Flush 0 Done Done Sent false
18619 6132 Acked Write 9 Done Done Sent false
18620 6133 Acked Read 6 Done Done Sent false
18621 6134 Acked Read 6 Done Done Sent false
18622 6135 Acked Read 3 Done Done Sent false
18623 6136 Acked Read 6 Done Done Sent false
18624 6137 Acked Flush 0 Done Done Sent false
18625 STATES DS:0 DS:1 DS:2 TOTAL
18626 Sent 0 0 302 302
18627 Done 302 302 0 604
18628 Skipped 0 0 0 0
18629 Error 0 0 0 0
18630 Last Flush: 6137 6137 5835
18631 Downstairs last five completed: 5835 5834 5833 5832 5831
18632 Upstairs last five completed: 6137 6136 6135 6134 6133
18633 CLIENT: Up:0 ds:302 act:2
186342024-11-01T16:54:23.014ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
186352024-11-01T16:54:23.014ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37592) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
186362024-11-01T16:54:23.016ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
186372024-11-01T16:54:23.016ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
186382024-11-01T16:54:23.016ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
186392024-11-01T16:54:23.016ZINFOcrucible: send last flush ID to this DS: 5835 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
186402024-11-01T16:54:23.016ZINFOcrucible: Replied this last flush ID: 5835 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
186412024-11-01T16:54:23.016ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
186422024-11-01T16:54:23.016ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
186432024-11-01T16:54:23.017ZINFOcrucible: [2] client re-new 302 jobs since flush 5835 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
186442024-11-01T16:54:23.017ZINFOcrucible: [2] Replayed 302 jobs since flush: 5835 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
18645 ----------------------------------------------------------------
18646 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18647 Downstairs last five completed: 6137 6136 6135 6134 6133
18648 Upstairs last five completed: 6137 6136 6135 6134 6133
18649 CLIENT: Up:0 ds:0 act:3
18650 Replay: All jobs finished
18651 Main task has work to do, go find it
18652 got dsc Stop(2)
18653 stop 2
18654 [8830] Got stop action so:false kr:false
18655 Nov 01 16:54:25.311 INFO request completed, latency_us: 350, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 6e3f8d75-cafc-4258-90ff-436e1e5e1841, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
186562024-11-01T16:54:25.311ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
186572024-11-01T16:54:25.311ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
186582024-11-01T16:54:25.311ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
18659 Nov 01 16:54:25.312 INFO request completed, latency_us: 128, response_code: 200, uri: /state/cid/2{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:54:25.312046174Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"client":"2","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
18660 , method: GET, req_id: 2c4b7ea2-4403-4b77-830b-59c33d2a4da5, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
186612024-11-01T16:54:25.312ZINFOcrucible: client task is exiting = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
18662 [8830] Exited with: ExitStatus(unix_wait_status(9))
18663 [8830][2] reports Exit
18664 Nov 01 16:54:29.313 INFO request completed, latency_us: 252, response_code: 200, uri: /state/cid/2, method: GET, req_id: e8243d2f-98e9-447d-8a65-93684ee2a8b1, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
18665 001/300 Write block 976 len 4096 data: 5
18666 002/300 Read block 3054 len 16384
18667 003/300 Write block 3949 len 28672 data: 6 6 6 5 5 3 2
18668 004/300 Read block 724 len 8192
18669 005/300 Write block 3119 len 16384 data: 3 3 3 5
18670 006/300 Write block 3843 len 16384 data: 3 3 3 3
18671 007/300 Write block 190 len 8192 data: 3 3
18672 008/300 Write block 4286 len 12288 data: 3 3 3
18673 009/300 Write block 3076 len 12288 data: 6 5 4
18674 010/300 Read block 2359 len 12288
18675 011/300 Read block 3403 len 20480
18676 012/300 Write block 2087 len 12288 data: 5 4 5
18677 013/300 Flush
18678 014/300 Read block 2494 len 32768
18679 015/300 Write block 2994 len 28672 data: 3 4 3 3 4 4 4
18680 016/300 Read block 54 len 8192
18681 017/300 Write block 3936 len 4096 data: 3
18682 018/300 Read block 2799 len 16384
18683 019/300 Write block 3996 len 40960 data: 4 4 2 2 2 3 4 4 4 4
18684 020/300 Write block 2472 len 32768 data: 5 6 7 8 9 9 8 8
18685 021/300 Read block 3656 len 4096
18686 022/300 Write block 2658 len 4096 data: 4
18687 023/300 Read block 4831 len 4096
18688 024/300 Read block 2699 len 32768
18689 025/300 Read block 666 len 20480
18690 026/300 Read block 4521 len 32768
18691 027/300 Write block 1572 len 28672 data: 4 4 3 4 4 4 4
18692 028/300 Read block 429 len 20480
18693 029/300 Write block 1689 len 32768 data: 3 3 4 4 5 4 4 3
18694 030/300 Write block 4179 len 24576 data: 4 4 4 5 5 3
18695 031/300 Write block 3118 len 28672 data: 3 4 4 4 6 4 3
18696 032/300 Write block 4500 len 24576 data: 4 4 4 5 6 6
18697 033/300 Read block 3688 len 36864
18698 034/300 Write block 4527 len 36864 data: 6 6 5 6 5 5 6 6 6
18699 035/300 Write block 2267 len 20480 data: 4 5 5 5 5
18700 036/300 Read block 1659 len 36864
18701 037/300 Read block 2709 len 28672
18702 038/300 Read block 3797 len 20480
18703 039/300 Write block 3118 len 4096 data: 4
18704 040/300 Read block 3758 len 16384
18705 041/300 Read block 2515 len 24576
18706 042/300 Read block 394 len 8192
18707 043/300 Read block 1 len 40960
18708 044/300 Write block 4194 len 24576 data: 5 5 6 6 5 5
18709 045/300 Read block 2429 len 40960
18710 046/300 Read block 3991 len 32768
18711 047/300 Write block 1933 len 24576 data: 3 3 3 3 4 6
18712 048/300 Read block 3143 len 24576
18713 049/300 Read block 2166 len 20480
18714 050/300 Read block 417 len 28672
18715 051/300 Read block 2422 len 36864
18716 052/300 Read block 4280 len 16384
18717 053/300 Write block 1917 len 32768 data: 4 4 3 2 2 4 4 4
18718 054/300 Read block 386 len 24576
18719 055/300 Write block 919 len 4096 data: 5
18720 056/300 Read block 1651 len 32768
18721 057/300 Write block 2466 len 40960 data: 3 3 3 3 2 4 6 7 8 9
18722 058/300 Write block 2604 len 40960 data: 3 3 3 3 3 4 4 5 5 5
18723 059/300 Read block 219 len 12288
18724 060/300 Write block 4878 len 32768 data: 4 3 2 2 4 5 5 4
18725 061/300 Write block 162 len 4096 data: 4
18726 062/300 Read block 4171 len 32768
18727 063/300 Flush
18728 064/300 Read block 1168 len 28672
18729 065/300 Write block 2584 len 28672 data: 4 5 5 4 3 3 3
18730 066/300 Read block 578 len 40960
18731 067/300 Read block 3582 len 8192
18732 068/300 Read block 4369 len 8192
18733 069/300 Read block 4443 len 24576
18734 070/300 Write block 1639 len 40960 data: 3 3 2 2 2 2 3 5 5 3
18735 071/300 Write block 2264 len 20480 data: 2 3 4 5 6
18736 072/300 Write block 773 len 36864 data: 5 4 7 9 9 10 9 7 7
18737 073/300 Read block 4950 len 20480
18738 074/300 Write block 3819 len 12288 data: 7 6 5
18739 075/300 Write block 101 len 12288 data: 4 4 4
18740 076/300 Write block 3413 len 24576 data: 3 3 3 3 4 4
18741 077/300 Read block 3744 len 4096
18742 078/300 Write block 4157 len 36864 data: 2 2 2 2 2 3 4 4 5
18743 079/300 Write block 1419 len 40960 data: 3 3 3 3 3 4 4 4 4 3
18744 080/300 Read block 3040 len 8192
18745 081/300 Read block 2478 len 20480
18746 082/300 Read block 3164 len 24576
18747 083/300 Write block 1202 len 32768 data: 3 4 4 3 4 6 6 5
18748 084/300 Read block 2214 len 12288
18749 085/300 Read block 1150 len 16384
18750 086/300 Write block 2462 len 28672 data: 3 3 3 3 4 4 4
18751 087/300 Read block 4470 len 40960
18752 088/300 Read block 865 len 20480
18753 089/300 Read block 2681 len 20480
18754 090/300 Flush
18755 091/300 Write block 3470 len 8192 data: 4 6
18756 092/300 Write block 2305 len 4096 data: 3
18757 093/300 Read block 3926 len 32768
18758 094/300 Read block 3675 len 24576
18759 095/300 Flush
18760 096/300 Read block 3279 len 24576
18761 097/300 Read block 317 len 16384
18762 098/300 Read block 2612 len 40960
18763 099/300 Write block 4428 len 40960 data: 6 7 6 6 7 6 6 3 3 3
18764 100/300 Write block 1730 len 24576 data: 3 3 3 3 3 4
18765 101/300 Read block 1863 len 12288
18766 102/300 Read block 397 len 20480
18767 103/300 Read block 919 len 4096
18768 104/300 Read block 3835 len 24576
18769 105/300 Write block 1805 len 12288 data: 4 4 6
18770 106/300 Read block 4127 len 16384
18771 107/300 Write block 4851 len 36864 data: 3 4 3 4 4 3 3 4 4
18772 108/300 Write block 264 len 12288 data: 3 3 3
18773 109/300 Read block 2735 len 28672
18774 110/300 Read block 797 len 24576
18775 111/300 Write block 4807 len 12288 data: 3 4 5
18776 112/300 Read block 4054 len 32768
18777 113/300 Write block 2309 len 24576 data: 3 3 2 3 3 3
18778 114/300 Read block 3667 len 20480
18779 115/300 Write block 2619 len 28672 data: 5 6 5 4 4 4 3
18780 116/300 Flush
18781 117/300 Write block 3441 len 20480 data: 6 5 5 6 6
18782 118/300 Read block 3989 len 20480
18783 119/300 Write block 1469 len 36864 data: 3 4 4 3 3 5 3 5 5
18784 120/300 Write block 2745 len 40960 data: 7 5 5 3 3 2 3 4 4 5
18785 121/300 Read block 3910 len 24576
18786 122/300 Read block 3047 len 20480
18787 123/300 Read block 3706 len 16384
18788 124/300 Read block 2227 len 24576
18789 125/300 Read block 2304 len 4096
18790 126/300 Read block 2478 len 16384
18791 127/300 Read block 1305 len 32768
18792 128/300 Write block 4762 len 20480 data: 8 7 4 4 4
18793 129/300 Write block 1509 len 4096 data: 7
18794 130/300 Read block 402 len 32768
18795 131/300 Read block 1534 len 32768
18796 132/300 Write block 2205 len 36864 data: 4 4 4 5 4 6 7 7 6
18797 133/300 Write block 2929 len 28672 data: 4 5 5 5 3 3 4
18798 134/300 Write block 3170 len 20480 data: 2 4 4 6 4
18799 135/300 Write block 2342 len 32768 data: 3 4 4 4 3 3 4 3
18800 136/300 Read block 3753 len 36864
18801 137/300 Flush
18802 138/300 Write block 3866 len 28672 data: 9 8 7 6 7 6 4
18803 139/300 Write block 1876 len 20480 data: 6 6 5 6 5
18804 140/300 Read block 3306 len 32768
18805 141/300 Read block 1771 len 32768
18806 142/300 Read block 3313 len 20480
18807 143/300 Write block 2302 len 12288 data: 4 5 4
18808 144/300 Write block 1640 len 12288 data: 4 3 3
18809 145/300 Read block 432 len 24576
18810 146/300 Read block 4279 len 28672
18811 147/300 Read block 1526 len 12288
18812 148/300 Flush
18813 149/300 Write block 1638 len 32768 data: 4 4 5 4 4 3 3 4
18814 150/300 Write block 3466 len 40960 data: 7 5 5 4 5 7 6 6 6 6
18815 151/300 Read block 3028 len 28672
18816 152/300 Read block 1981 len 32768
18817 153/300 Write block 3539 len 4096 data: 3
18818 154/300 Read block 194 len 24576
18819 155/300 Write block 3239 len 8192 data: 3 3
18820 156/300 Read block 4258 len 40960
18821 157/300 Read block 2836 len 4096
18822 158/300 Read block 663 len 32768
18823 159/300 Read block 1781 len 32768
18824 160/300 Write block 4846 len 24576 data: 2 2 3 4 4 4
18825 161/300 Write block 3108 len 4096 data: 5
18826 162/300 Read block 444 len 24576
18827 163/300 Write block 4523 len 8192 data: 5 5
18828 164/300 Read block 4806 len 32768
18829 165/300 Read block 2687 len 32768
18830 166/300 Write block 2503 len 20480 data: 4 4 4 3 3
18831 167/300 Read block 4839 len 12288
18832 168/300 Read block 1406 len 28672
18833 169/300 Flush
18834 170/300 Read block 4107 len 40960
18835 171/300 Flush
18836 172/300 Flush
18837 173/300 Read block 4126 len 20480
18838 174/300 Read block 3041 len 4096
18839 175/300 Write block 475 len 16384 data: 4 4 4 7
18840 176/300 Write block 4190 len 20480 data: 4 4 4 4 6
18841 177/300 Write block 3857 len 24576 data: 5 6 7 9 8 7
18842 178/300 Read block 884 len 24576
18843 179/300 Write block 3029 len 20480 data: 3 3 4 4 4
18844 180/300 Read block 3865 len 16384
18845 181/300 Write block 1225 len 12288 data: 4 3 3
18846 182/300 Write block 3174 len 24576 data: 5 3 2 2 3 3
18847 183/300 Read block 697 len 32768
18848 184/300 Read block 745 len 24576
18849 185/300 Read block 4498 len 36864
18850 186/300 Write block 3250 len 8192 data: 5 5
18851 187/300 Read block 3344 len 32768
18852 188/300 Read block 360 len 16384
18853 189/300 Write block 3905 len 40960 data: 3 4 5 5 4 3 3 5 5 5
18854 190/300 Flush
18855 191/300 Read block 653 len 16384
18856 192/300 Read block 2261 len 24576
18857 193/300 Write block 4745 len 24576 data: 3 3 3 3 4 4
18858 194/300 Flush
18859 195/300 Write block 509 len 8192 data: 7 7
18860 196/300 Read block 2628 len 32768
18861 197/300 Read block 2871 len 36864
18862 198/300 Write block 4 len 4096 data: 4
18863 199/300 Read block 4894 len 8192
18864 200/300 Flush
18865 201/300 Read block 762 len 40960
18866 202/300 Write block 2728 len 20480 data: 3 5 4 3 4
18867 203/300 Read block 4281 len 16384
18868 204/300 Read block 520 len 32768
18869 205/300 Flush
18870 206/300 Read block 4321 len 40960
18871 207/300 Write block 2358 len 36864 data: 5 5 5 5 6 5 4 6 6
18872 208/300 Read block 3312 len 40960
18873 209/300 Read block 1208 len 20480
18874 210/300 Read block 1189 len 16384
18875 211/300 Write block 1927 len 12288 data: 4 3 3
18876 212/300 Write block 1765 len 24576 data: 5 5 4 4 4 4
18877 213/300 Write block 1294 len 20480 data: 4 4 4 5 6
18878 214/300 Read block 928 len 36864
18879 215/300 Read block 972 len 40960
18880 216/300 Read block 756 len 28672
18881 217/300 Read block 873 len 8192
18882 218/300 Write block 2404 len 12288 data: 5 4 3
18883 219/300 Read block 943 len 16384
18884 220/300 Read block 3939 len 28672
18885 221/300 Read block 2231 len 20480
18886 222/300 Write block 1213 len 24576 data: 7 7 6 5 5 6
18887 223/300 Write block 4419 len 8192 data: 3 4
18888 224/300 Read block 2644 len 36864
18889 225/300 Read block 3176 len 12288
18890 226/300 Read block 4840 len 36864
18891 227/300 Write block 2774 len 20480 data: 4 5 4 5 5
18892 228/300 Read block 3857 len 32768
18893 229/300 Write block 4911 len 24576 data: 3 3 4 5 6 7
18894 230/300 Read block 3777 len 12288
18895 231/300 Write block 4373 len 20480 data: 5 4 4 6 7
18896 232/300 Write block 1419 len 8192 data: 4 4
18897 233/300 Read block 412 len 24576
18898 234/300 Flush
18899 235/300 Write block 2360 len 28672 data: 6 6 7 6 5 7 7
18900 236/300 Write block 4964 len 28672 data: 5 6 5 5 4 5 6
18901 237/300 Write block 3933 len 24576 data: 5 5 4 4 3 3
18902 238/300 Write block 3459 len 12288 data: 6 8 8
18903 239/300 Read block 3853 len 28672
18904 240/300 Read block 3398 len 40960
18905 241/300 Write block 4141 len 12288 data: 5 6 6
18906 242/300 Read block 4742 len 4096
18907 243/300 Write block 894 len 16384 data: 7 5 5 5
18908 244/300 Read block 220 len 28672
18909 245/300 Flush
18910 246/300 Write block 966 len 24576 data: 2 3 4 5 7 6
18911 247/300 Read block 672 len 12288
18912 248/300 Read block 2560 len 8192
18913 249/300 Read block 2776 len 12288
18914 250/300 Write block 3878 len 12288 data: 2 3 4
18915 251/300 Read block 161 len 28672
18916 252/300 Write block 3875 len 4096 data: 2
18917 253/300 Write block 3254 len 8192 data: 4 5
18918 254/300 Write block 2903 len 24576 data: 2 2 2 2 2 2
18919 255/300 Read block 4398 len 36864
18920 256/300 Write block 1137 len 20480 data: 4 5 3 3 2
18921 257/300 Write block 4875 len 8192 data: 5 6
18922 258/300 Write block 1017 len 20480 data: 2 3 3 3 2
18923 259/300 Write block 1475 len 16384 data: 4 6 6 6
18924 260/300 Read block 3046 len 16384
18925 261/300 Write block 4344 len 4096 data: 4
18926 262/300 Write block 825 len 28672 data: 4 4 4 4 4 5 5
18927 263/300 Write block 1038 len 24576 data: 4 5 4 5 5 4
18928 264/300 Read block 2545 len 32768
18929 265/300 Write block 4866 len 12288 data: 4 4 4
18930 266/300 Read block 2572 len 20480
18931 267/300 Write block 2283 len 8192 data: 6 7
18932 268/300 Write block 4447 len 28672 data: 3 3 6 6 7 7 6
18933 269/300 Write block 2280 len 28672 data: 4 5 5 7 8 5 7
18934 270/300 Write block 2636 len 28672 data: 5 5 5 5 4 5 5
18935 271/300 Write block 363 len 16384 data: 3 4 5 4
18936 272/300 Write block 4787 len 12288 data: 3 4 4
18937 273/300 Write block 1534 len 28672 data: 4 3 3 4 4 5 5
18938 274/300 Write block 695 len 40960 data: 6 6 6 5 5 4 4 3 2 2
18939 275/300 Write block 3345 len 24576 data: 3 2 2 2 3 4
18940 276/300 Write block 277 len 40960 data: 6 6 6 4 4 4 3 3 3 5
18941 277/300 Read block 1677 len 20480
18942 278/300 Write block 1077 len 20480 data: 5 6 5 6 5
18943 279/300 Read block 3588 len 4096
18944 280/300 Read block 1056 len 24576
18945 281/300 Flush
18946 282/300 Write block 1877 len 32768 data: 7 6 7 6 5 3 4 4
18947 283/300 Read block 777 len 36864
18948 284/300 Read block 2966 len 40960
18949 285/300 Read block 840 len 20480
18950 286/300 Write block 4251 len 28672 data: 4 3 3 3 3 3 3
18951 287/300 Write block 2409 len 4096 data: 2
18952 288/300 Read block 1094 len 24576
18953 289/300 Write block 2526 len 20480 data: 3 3 3 2 2
18954 290/300 Write block 4523 len 20480 data: 6 6 6 5 7
18955 291/300 Write block 3797 len 40960 data: 6 7 6 6 4 5 5 6 3 3
18956 292/300 Read block 2423 len 36864
18957 293/300 Write block 2960 len 8192 data: 6 4
18958 294/300 Read block 364 len 24576
18959 295/300 Write block 3988 len 20480 data: 3 3 4 4 5
18960 296/300 Read block 4402 len 24576
18961 297/300 Read block 2264 len 28672
18962 298/300 Read block 909 len 8192
18963 299/300 Write block 926 len 32768 data: 7 8 4 5 5 5 5 5
18964 300/300 Write block 1722 len 16384 data: 6 6 4 5
18965 301/300 Read block 742 len 16384
18966 Main task has work to do, go find it
18967 got dsc Start(2)
18968 start 2
18969 Nov 01 16:54:29.621 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/2, method: POST, req_id: 3dbcc2f2-384e-466b-b142-bb5054514f68, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
18970 Replay: started 2, returned:Ok(())
18971 ----------------------------------------------------------------
18972 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18973 States: Active Active Offline
18974 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18975 6138 Acked Write 1 Done Done Sent false
18976 6139 Acked Read 4 Done Done Sent false
18977 6140 Acked Write 7 Done Done Sent false
18978 6141 Acked Read 2 Done Done Sent false
18979 6142 Acked Write 4 Done Done Sent false
18980 6143 Acked Write 4 Done Done Sent false
18981 6144 Acked Write 2 Done Done Sent false
18982 6145 Acked Write 3 Done Done Sent false
18983 6146 Acked Write 3 Done Done Sent false
18984 6147 Acked Read 3 Done Done Sent false
18985 6148 Acked Read 5 Done Done Sent false
18986 6149 Acked Write 3 Done Done Sent false
18987 6150 Acked Flush 0 Done Done Sent false
18988 6151 Acked Read 8 Done Done Sent false
18989 6152 Acked Write 7 Done Done Sent false
18990 6153 Acked Read 2 Done Done Sent false
18991 6154 Acked Write 1 Done Done Sent false
18992 6155 Acked Read 4 Done Done Sent false
18993 6156 Acked Write 10 Done Done Sent false
18994 6157 Acked Write 8 Done Done Sent false
18995 6158 Acked Read 1 Done Done Sent false
18996 6159 Acked Write 1 Done Done Sent false
18997 6160 Acked Read 1 Done Done Sent false
18998 6161 Acked Read 8 Done Done Sent false
18999 6162 Acked Read 5 Done Done Sent false
19000 6163 Acked Read 8 Done Done Sent false
19001 6164 Acked Write 7 Done Done Sent false
19002 6165 Acked Read 5 Done Done Sent false
19003 6166 Acked Write 8 Done Done Sent false
19004 6167 Acked Write 6 Done Done Sent false
19005 6168 Acked Write 7 Done Done Sent false
19006 6169 Acked Write 6 Done Done Sent false
19007 6170 Acked Read 9 Done Done Sent false
19008 6171 Acked Write 9 Done Done Sent false
19009 6172 Acked Write 5 Done Done Sent false
19010 6173 Acked Read 9 Done Done Sent false
19011 6174 Acked Read 7 Done Done Sent false
19012 6175 Acked Read 5 Done Done Sent false
19013 6176 Acked Write 1 Done Done Sent false
19014 6177 Acked Read 4 Done Done Sent false
19015 6178 Acked Read 6 Done Done Sent false
19016 6179 Acked Read 2 Done Done Sent false
19017 6180 Acked Read 10 Done Done Sent false
19018 6181 Acked Write 6 Done Done Sent false
19019 6182 Acked Read 10 Done Done Sent false
19020 6183 Acked Read 8 Done Done Sent false
19021 6184 Acked Write 6 Done Done Sent false
19022 6185 Acked Read 6 Done Done Sent false
19023 6186 Acked Read 5 Done Done Sent false
19024 6187 Acked Read 7 Done Done Sent false
19025 6188 Acked Read 9 Done Done Sent false
19026 6189 Acked Read 4 Done Done Sent false
19027 6190 Acked Write 8 Done Done Sent false
19028 6191 Acked Read 6 Done Done Sent false
19029 6192 Acked Write 1 Done Done Sent false
19030 6193 Acked Read 8 Done Done Sent false
19031 6194 Acked Write 10 Done Done Sent false
19032 6195 Acked Write 10 Done Done Sent false
19033 6196 Acked Read 3 Done Done Sent false
19034 6197 Acked Write 8 Done Done Sent false
19035 6198 Acked Write 1 Done Done Sent false
19036 6199 Acked Read 8 Done Done Sent false
19037 6200 Acked Flush 0 Done Done Sent false
19038 6201 Acked Read 7 Done Done Sent false
19039 6202 Acked Write 7 Done Done Sent false
19040 6203 Acked Read 10 Done Done Sent false
19041 6204 Acked Read 2 Done Done Sent false
19042 6205 Acked Read 2 Done Done Sent false
19043 6206 Acked Read 6 Done Done Sent false
19044 6207 Acked Write 10 Done Done Sent false
19045 6208 Acked Write 5 Done Done Sent false
19046 6209 Acked Write 9 Done Done Sent false
19047 6210 Acked Read 5 Done Done Sent false
19048 6211 Acked Write 3 Done Done Sent false
19049 6212 Acked Write 3 Done Done Sent false
19050 6213 Acked Write 6 Done Done Sent false
19051 6214 Acked Read 1 Done Done Sent false
19052 6215 Acked Write 9 Done Done Sent false
19053 6216 Acked Write 10 Done Done Sent false
19054 6217 Acked Read 2 Done Done Sent false
19055 6218 Acked Read 5 Done Done Sent false
19056 6219 Acked Read 6 Done Done Sent false
19057 6220 Acked Write 8 Done Done Sent false
19058 6221 Acked Read 3 Done Done Sent false
19059 6222 Acked Read 4 Done Done Sent false
19060 6223 Acked Write 7 Done Done Sent false
19061 6224 Acked Read 10 Done Done Sent false
19062 6225 Acked Read 5 Done Done Sent false
19063 6226 Acked Read 5 Done Done Sent false
19064 6227 Acked Flush 0 Done Done Sent false
19065 6228 Acked Write 2 Done Done Sent false
19066 6229 Acked Write 1 Done Done Sent false
19067 6230 Acked Read 8 Done Done Sent false
19068 6231 Acked Read 6 Done Done Sent false
19069 6232 Acked Flush 0 Done Done Sent false
19070 6233 Acked Read 6 Done Done Sent false
19071 6234 Acked Read 4 Done Done Sent false
19072 6235 Acked Read 10 Done Done Sent false
19073 6236 Acked Write 10 Done Done Sent false
19074 6237 Acked Write 6 Done Done Sent false
19075 6238 Acked Read 3 Done Done Sent false
19076 6239 Acked Read 5 Done Done Sent false
19077 6240 Acked Read 1 Done Done Sent false
19078 6241 Acked Read 6 Done Done Sent false
19079 6242 Acked Write 3 Done Done Sent false
19080 6243 Acked Read 4 Done Done Sent false
19081 6244 Acked Write 9 Done Done Sent false
19082 6245 Acked Write 3 Done Done Sent false
19083 6246 Acked Read 7 Done Done Sent false
19084 6247 Acked Read 6 Done Done Sent false
19085 6248 Acked Write 3 Done Done Sent false
19086 6249 Acked Read 8 Done Done Sent false
19087 6250 Acked Write 6 Done Done Sent false
19088 6251 Acked Read 5 Done Done Sent false
19089 6252 Acked Write 7 Done Done Sent false
19090 6253 Acked Flush 0 Done Done Sent false
19091 6254 Acked Write 5 Done Done Sent false
19092 6255 Acked Read 5 Done Done Sent false
19093 6256 Acked Write 9 Done Done Sent false
19094 6257 Acked Write 10 Done Done Sent false
19095 6258 Acked Read 6 Done Done Sent false
19096 6259 Acked Read 5 Done Done Sent false
19097 6260 Acked Read 4 Done Done Sent false
19098 6261 Acked Read 6 Done Done Sent false
19099 6262 Acked Read 1 Done Done Sent false
19100 6263 Acked Read 4 Done Done Sent false
19101 6264 Acked Read 8 Done Done Sent false
19102 6265 Acked Write 5 Done Done Sent false
19103 6266 Acked Write 1 Done Done Sent false
19104 6267 Acked Read 8 Done Done Sent false
19105 6268 Acked Read 8 Done Done Sent false
19106 6269 Acked Write 9 Done Done Sent false
19107 6270 Acked Write 7 Done Done Sent false
19108 6271 Acked Write 5 Done Done Sent false
19109 6272 Acked Write 8 Done Done Sent false
19110 6273 Acked Read 9 Done Done Sent false
19111 6274 Acked Flush 0 Done Done Sent false
19112 6275 Acked Write 7 Done Done Sent false
19113 6276 Acked Write 5 Done Done Sent false
19114 6277 Acked Read 8 Done Done Sent false
19115 6278 Acked Read 8 Done Done Sent false
19116 6279 Acked Read 5 Done Done Sent false
19117 6280 Acked Write 3 Done Done Sent false
19118 6281 Acked Write 3 Done Done Sent false
19119 6282 Acked Read 6 Done Done Sent false
19120 6283 Acked Read 7 Done Done Sent false
19121 6284 Acked Read 3 Done Done Sent false
19122 6285 Acked Flush 0 Done Done Sent false
19123 6286 Acked Write 8 Done Done Sent false
19124 6287 Acked Write 10 Done Done Sent false
19125 6288 Acked Read 7 Done Done Sent false
19126 6289 Acked Read 8 Done Done Sent false
19127 6290 Acked Write 1 Done Done Sent false
19128 6291 Acked Read 6 Done Done Sent false
19129 6292 Acked Write 2 Done Done Sent false
19130 6293 Acked Read 10 Done Done Sent false
19131 6294 Acked Read 1 Done Done Sent false
19132 6295 Acked Read 8 Done Done Sent false
19133 6296 Acked Read 8 Done Done Sent false
19134 6297 Acked Write 6 Done Done Sent false
19135 6298 Acked Write 1 Done Done Sent false
19136 6299 Acked Read 6 Done Done Sent false
19137 6300 Acked Write 2 Done Done Sent false
19138 6301 Acked Read 8 Done Done Sent false
19139 6302 Acked Read 8 Done Done Sent false
19140 6303 Acked Write 5 Done Done Sent false
19141 6304 Acked Read 3 Done Done Sent false
19142 6305 Acked Read 7 Done Done Sent false
19143 6306 Acked Flush 0 Done Done Sent false
19144 6307 Acked Read 10 Done Done Sent false
19145 6308 Acked Flush 0 Done Done Sent false
19146 6309 Acked Flush 0 Done Done Sent false
19147 6310 Acked Read 5 Done Done Sent false
19148 6311 Acked Read 1 Done Done Sent false
19149 6312 Acked Write 4 Done Done Sent false
19150 6313 Acked Write 5 Done Done Sent false
19151 6314 Acked Write 6 Done Done Sent false
19152 6315 Acked Read 6 Done Done Sent false
19153 6316 Acked Write 5 Done Done Sent false
19154 6317 Acked Read 4 Done Done Sent false
19155 6318 Acked Write 3 Done Done Sent false
19156 6319 Acked Write 6 Done Done Sent false
19157 6320 Acked Read 8 Done Done Sent false
19158 6321 Acked Read 6 Done Done Sent false
19159 6322 Acked Read 9 Done Done Sent false
19160 6323 Acked Write 2 Done Done Sent false
19161 6324 Acked Read 8 Done Done Sent false
19162 6325 Acked Read 4 Done Done Sent false
19163 6326 Acked Write 10 Done Done Sent false
19164 6327 Acked Flush 0 Done Done Sent false
19165 6328 Acked Read 4 Done Done Sent false
19166 6329 Acked Read 6 Done Done Sent false
19167 6330 Acked Write 6 Done Done Sent false
19168 6331 Acked Flush 0 Done Done Sent false
19169 6332 Acked Write 2 Done Done Sent false
19170 6333 Acked Read 8 Done Done Sent false
19171 6334 Acked Read 9 Done Done Sent false
19172 6335 Acked Write 1 Done Done Sent false
19173 6336 Acked Read 2 Done Done Sent false
19174 6337 Acked Flush 0 Done Done Sent false
19175 6338 Acked Read 10 Done Done Sent false
19176 6339 Acked Write 5 Done Done Sent false
19177 6340 Acked Read 4 Done Done Sent false
19178 6341 Acked Read 8 Done Done Sent false
19179 6342 Acked Flush 0 Done Done Sent false
19180 6343 Acked Read 10 Done Done Sent false
19181 6344 Acked Write 9 Done Done Sent false
19182 6345 Acked Read 10 Done Done Sent false
19183 6346 Acked Read 5 Done Done Sent false
19184 6347 Acked Read 4 Done Done Sent false
19185 6348 Acked Write 3 Done Done Sent false
19186 6349 Acked Write 6 Done Done Sent false
19187 6350 Acked Write 5 Done Done Sent false
19188 6351 Acked Read 9 Done Done Sent false
19189 6352 Acked Read 10 Done Done Sent false
19190 6353 Acked Read 7 Done Done Sent false
19191 6354 Acked Read 2 Done Done Sent false
19192 6355 Acked Write 3 Done Done Sent false
19193 6356 Acked Read 4 Done Done Sent false
19194 6357 Acked Read 7 Done Done Sent false
19195 6358 Acked Read 5 Done Done Sent false
19196 6359 Acked Write 6 Done Done Sent false
19197 6360 Acked Write 2 Done Done Sent false
19198 6361 Acked Read 9 Done Done Sent false
19199 6362 Acked Read 3 Done Done Sent false
19200 6363 Acked Read 9 Done Done Sent false
19201 6364 Acked Write 5 Done Done Sent false
19202 6365 Acked Read 8 Done Done Sent false
19203 6366 Acked Write 6 Done Done Sent false
19204 6367 Acked Read 3 Done Done Sent false
19205 6368 Acked Write 5 Done Done Sent false
19206 6369 Acked Write 2 Done Done Sent false
19207 6370 Acked Read 6 Done Done Sent false
19208 6371 Acked Flush 0 Done Done Sent false
19209 6372 Acked Write 7 Done Done Sent false
19210 6373 Acked Write 7 Done Done Sent false
19211 6374 Acked Write 6 Done Done Sent false
19212 6375 Acked Write 3 Done Done Sent false
19213 6376 Acked Read 7 Done Done Sent false
19214 6377 Acked Read 10 Done Done Sent false
19215 6378 Acked Write 3 Done Done Sent false
19216 6379 Acked Read 1 Done Done Sent false
19217 6380 Acked Write 4 Done Done Sent false
19218 6381 Acked Read 7 Done Done Sent false
19219 6382 Acked Flush 0 Done Done Sent false
19220 6383 Acked Write 6 Done Done Sent false
19221 6384 Acked Read 3 Done Done Sent false
19222 6385 Acked Read 2 Done Done Sent false
19223 6386 Acked Read 3 Done Done Sent false
19224 6387 Acked Write 3 Done Done Sent false
19225 6388 Acked Read 7 Done Done Sent false
19226 6389 Acked Write 1 Done Done Sent false
19227 6390 Acked Write 2 Done Done Sent false
19228 6391 Acked Write 6 Done Done Sent false
19229 6392 Acked Read 9 Done Done Sent false
19230 6393 Acked Write 5 Done Done Sent false
19231 6394 Acked Write 2 Done Done Sent false
19232 6395 Acked Write 5 Done Done Sent false
19233 6396 Acked Write 4 Done Done Sent false
19234 6397 Acked Read 4 Done Done Sent false
19235 6398 Acked Write 1 Done Done Sent false
19236 6399 Acked Write 7 Done Done Sent false
19237 6400 Acked Write 6 Done Done Sent false
19238 6401 Acked Read 8 Done Done Sent false
19239 6402 Acked Write 3 Done Done Sent false
19240 6403 Acked Read 5 Done Done Sent false
19241 6404 Acked Write 2 Done Done Sent false
19242 6405 Acked Write 7 Done Done Sent false
19243 6406 Acked Write 7 Done Done Sent false
19244 6407 Acked Write 7 Done Done Sent false
19245 6408 Acked Write 4 Done Done Sent false
19246 6409 Acked Write 3 Done Done Sent false
19247 6410 Acked Write 7 Done Done Sent false
19248 6411 Acked Write 10 Done Done Sent false
19249 6412 Acked Write 6 Done Done Sent false
19250 6413 Acked Write 10 Done Done Sent false
19251 6414 Acked Read 5 Done Done Sent false
19252 6415 Acked Write 5 Done Done Sent false
19253 6416 Acked Read 1 Done Done Sent false
19254 6417 Acked Read 6 Done Done Sent false
19255 6418 Acked Flush 0 Done Done Sent false
19256 6419 Acked Write 8 Done Done Sent false
19257 6420 Acked Read 9 Done Done Sent false
19258 6421 Acked Read 10 Done Done Sent false
19259 6422 Acked Read 5 Done Done Sent false
19260 6423 Acked Write 7 Done Done Sent false
19261 6424 Acked Write 1 Done Done Sent false
19262 6425 Acked Read 6 Done Done Sent false
19263 6426 Acked Write 5 Done Done Sent false
19264 6427 Acked Write 5 Done Done Sent false
19265 6428 Acked Write 10 Done Done Sent false
19266 6429 Acked Read 9 Done Done Sent false
19267 6430 Acked Write 2 Done Done Sent false
19268 6431 Acked Read 6 Done Done Sent false
19269 6432 Acked Write 5 Done Done Sent false
19270 6433 Acked Read 6 Done Done Sent false
19271 6434 Acked Read 7 Done Done Sent false
19272 6435 Acked Read 2 Done Done Sent false
19273 6436 Acked Write 8 Done Done Sent false
19274 6437 Acked Write 4 Done Done Sent false
19275 6438 Acked Read 4 Done Done Sent false
19276 STATES DS:0 DS:1 DS:2 TOTAL
19277 Sent 0 0 301 301
19278 Done 301 301 0 602
19279 Skipped 0 0 0 0
19280 Error 0 0 0 0
19281 Last Flush: 6418 6418 6137
19282 Downstairs last five completed: 6137 6136 6135 6134 6133
19283 Upstairs last five completed: 6438 6437 6436 6435 6434
19284 CLIENT: Up:0 ds:301 act:2
19285 [8830] got start action
19286 [8830] I am going to restart
19287 Starting downstairs at port 8830
19288 Make output file at "/tmp/dsc/downstairs-8830.txt"
19289 [8830][2] reports Starting
19290 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1216)
19291 [8830][2] reports Running
19292 ----------------------------------------------------------------
19293 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
19294 States: Active Active Offline
19295 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19296 6138 Acked Write 1 Done Done Sent false
19297 6139 Acked Read 4 Done Done Sent false
19298 6140 Acked Write 7 Done Done Sent false
19299 6141 Acked Read 2 Done Done Sent false
19300 6142 Acked Write 4 Done Done Sent false
19301 6143 Acked Write 4 Done Done Sent false
19302 6144 Acked Write 2 Done Done Sent false
19303 6145 Acked Write 3 Done Done Sent false
19304 6146 Acked Write 3 Done Done Sent false
19305 6147 Acked Read 3 Done Done Sent false
19306 6148 Acked Read 5 Done Done Sent false
19307 6149 Acked Write 3 Done Done Sent false
19308 6150 Acked Flush 0 Done Done Sent false
19309 6151 Acked Read 8 Done Done Sent false
19310 6152 Acked Write 7 Done Done Sent false
19311 6153 Acked Read 2 Done Done Sent false
19312 6154 Acked Write 1 Done Done Sent false
19313 6155 Acked Read 4 Done Done Sent false
19314 6156 Acked Write 10 Done Done Sent false
19315 6157 Acked Write 8 Done Done Sent false
19316 6158 Acked Read 1 Done Done Sent false
19317 6159 Acked Write 1 Done Done Sent false
19318 6160 Acked Read 1 Done Done Sent false
19319 6161 Acked Read 8 Done Done Sent false
19320 6162 Acked Read 5 Done Done Sent false
19321 6163 Acked Read 8 Done Done Sent false
19322 6164 Acked Write 7 Done Done Sent false
19323 6165 Acked Read 5 Done Done Sent false
19324 6166 Acked Write 8 Done Done Sent false
19325 6167 Acked Write 6 Done Done Sent false
19326 6168 Acked Write 7 Done Done Sent false
19327 6169 Acked Write 6 Done Done Sent false
19328 6170 Acked Read 9 Done Done Sent false
19329 6171 Acked Write 9 Done Done Sent false
19330 6172 Acked Write 5 Done Done Sent false
19331 6173 Acked Read 9 Done Done Sent false
19332 6174 Acked Read 7 Done Done Sent false
19333 6175 Acked Read 5 Done Done Sent false
19334 6176 Acked Write 1 Done Done Sent false
19335 6177 Acked Read 4 Done Done Sent false
19336 6178 Acked Read 6 Done Done Sent false
19337 6179 Acked Read 2 Done Done Sent false
19338 6180 Acked Read 10 Done Done Sent false
19339 6181 Acked Write 6 Done Done Sent false
19340 6182 Acked Read 10 Done Done Sent false
19341 6183 Acked Read 8 Done Done Sent false
19342 6184 Acked Write 6 Done Done Sent false
19343 6185 Acked Read 6 Done Done Sent false
19344 6186 Acked Read 5 Done Done Sent false
19345 6187 Acked Read 7 Done Done Sent false
19346 6188 Acked Read 9 Done Done Sent false
19347 6189 Acked Read 4 Done Done Sent false
19348 6190 Acked Write 8 Done Done Sent false
19349 6191 Acked Read 6 Done Done Sent false
19350 6192 Acked Write 1 Done Done Sent false
19351 6193 Acked Read 8 Done Done Sent false
19352 6194 Acked Write 10 Done Done Sent false
19353 6195 Acked Write 10 Done Done Sent false
19354 6196 Acked Read 3 Done Done Sent false
19355 6197 Acked Write 8 Done Done Sent false
19356 6198 Acked Write 1 Done Done Sent false
19357 6199 Acked Read 8 Done Done Sent false
19358 6200 Acked Flush 0 Done Done Sent false
19359 6201 Acked Read 7 Done Done Sent false
19360 6202 Acked Write 7 Done Done Sent false
19361 6203 Acked Read 10 Done Done Sent false
19362 6204 Acked Read 2 Done Done Sent false
19363 6205 Acked Read 2 Done Done Sent false
19364 6206 Acked Read 6 Done Done Sent false
19365 6207 Acked Write 10 Done Done Sent false
19366 6208 Acked Write 5 Done Done Sent false
19367 6209 Acked Write 9 Done Done Sent false
19368 6210 Acked Read 5 Done Done Sent false
19369 6211 Acked Write 3 Done Done Sent false
19370 6212 Acked Write 3 Done Done Sent false
19371 6213 Acked Write 6 Done Done Sent false
19372 6214 Acked Read 1 Done Done Sent false
19373 6215 Acked Write 9 Done Done Sent false
19374 6216 Acked Write 10 Done Done Sent false
19375 6217 Acked Read 2 Done Done Sent false
19376 6218 Acked Read 5 Done Done Sent false
19377 6219 Acked Read 6 Done Done Sent false
19378 6220 Acked Write 8 Done Done Sent false
19379 6221 Acked Read 3 Done Done Sent false
19380 6222 Acked Read 4 Done Done Sent false
19381 6223 Acked Write 7 Done Done Sent false
19382 6224 Acked Read 10 Done Done Sent false
19383 6225 Acked Read 5 Done Done Sent false
19384 6226 Acked Read 5 Done Done Sent false
19385 6227 Acked Flush 0 Done Done Sent false
19386 6228 Acked Write 2 Done Done Sent false
19387 6229 Acked Write 1 Done Done Sent false
19388 6230 Acked Read 8 Done Done Sent false
19389 6231 Acked Read 6 Done Done Sent false
19390 6232 Acked Flush 0 Done Done Sent false
19391 6233 Acked Read 6 Done Done Sent false
19392 6234 Acked Read 4 Done Done Sent false
19393 6235 Acked Read 10 Done Done Sent false
19394 6236 Acked Write 10 Done Done Sent false
19395 6237 Acked Write 6 Done Done Sent false
19396 6238 Acked Read 3 Done Done Sent false
19397 6239 Acked Read 5 Done Done Sent false
19398 6240 Acked Read 1 Done Done Sent false
19399 6241 Acked Read 6 Done Done Sent false
19400 6242 Acked Write 3 Done Done Sent false
19401 6243 Acked Read 4 Done Done Sent false
19402 6244 Acked Write 9 Done Done Sent false
19403 6245 Acked Write 3 Done Done Sent false
19404 6246 Acked Read 7 Done Done Sent false
19405 6247 Acked Read 6 Done Done Sent false
19406 6248 Acked Write 3 Done Done Sent false
19407 6249 Acked Read 8 Done Done Sent false
19408 6250 Acked Write 6 Done Done Sent false
19409 6251 Acked Read 5 Done Done Sent false
19410 6252 Acked Write 7 Done Done Sent false
19411 6253 Acked Flush 0 Done Done Sent false
19412 6254 Acked Write 5 Done Done Sent false
19413 6255 Acked Read 5 Done Done Sent false
19414 6256 Acked Write 9 Done Done Sent false
19415 6257 Acked Write 10 Done Done Sent false
19416 6258 Acked Read 6 Done Done Sent false
19417 6259 Acked Read 5 Done Done Sent false
19418 6260 Acked Read 4 Done Done Sent false
19419 6261 Acked Read 6 Done Done Sent false
19420 6262 Acked Read 1 Done Done Sent false
19421 6263 Acked Read 4 Done Done Sent false
19422 6264 Acked Read 8 Done Done Sent false
19423 6265 Acked Write 5 Done Done Sent false
19424 6266 Acked Write 1 Done Done Sent false
19425 6267 Acked Read 8 Done Done Sent false
19426 6268 Acked Read 8 Done Done Sent false
19427 6269 Acked Write 9 Done Done Sent false
19428 6270 Acked Write 7 Done Done Sent false
19429 6271 Acked Write 5 Done Done Sent false
19430 6272 Acked Write 8 Done Done Sent false
19431 6273 Acked Read 9 Done Done Sent false
19432 6274 Acked Flush 0 Done Done Sent false
19433 6275 Acked Write 7 Done Done Sent false
19434 6276 Acked Write 5 Done Done Sent false
19435 6277 Acked Read 8 Done Done Sent false
19436 6278 Acked Read 8 Done Done Sent false
19437 6279 Acked Read 5 Done Done Sent false
19438 6280 Acked Write 3 Done Done Sent false
19439 6281 Acked Write 3 Done Done Sent false
19440 6282 Acked Read 6 Done Done Sent false
19441 6283 Acked Read 7 Done Done Sent false
19442 6284 Acked Read 3 Done Done Sent false
19443 6285 Acked Flush 0 Done Done Sent false
19444 6286 Acked Write 8 Done Done Sent false
19445 6287 Acked Write 10 Done Done Sent false
19446 6288 Acked Read 7 Done Done Sent false
19447 6289 Acked Read 8 Done Done Sent false
19448 6290 Acked Write 1 Done Done Sent false
19449 6291 Acked Read 6 Done Done Sent false
19450 6292 Acked Write 2 Done Done Sent false
19451 6293 Acked Read 10 Done Done Sent false
19452 6294 Acked Read 1 Done Done Sent false
19453 6295 Acked Read 8 Done Done Sent false
19454 6296 Acked Read 8 Done Done Sent false
19455 6297 Acked Write 6 Done Done Sent false
19456 6298 Acked Write 1 Done Done Sent false
19457 6299 Acked Read 6 Done Done Sent false
19458 6300 Acked Write 2 Done Done Sent false
19459 6301 Acked Read 8 Done Done Sent false
19460 6302 Acked Read 8 Done Done Sent false
19461 6303 Acked Write 5 Done Done Sent false
19462 6304 Acked Read 3 Done Done Sent false
19463 6305 Acked Read 7 Done Done Sent false
19464 6306 Acked Flush 0 Done Done Sent false
19465 6307 Acked Read 10 Done Done Sent false
19466 6308 Acked Flush 0 Done Done Sent false
19467 6309 Acked Flush 0 Done Done Sent false
19468 6310 Acked Read 5 Done Done Sent false
19469 6311 Acked Read 1 Done Done Sent false
19470 6312 Acked Write 4 Done Done Sent false
19471 6313 Acked Write 5 Done Done Sent false
19472 6314 Acked Write 6 Done Done Sent false
19473 6315 Acked Read 6 Done Done Sent false
19474 6316 Acked Write 5 Done Done Sent false
19475 6317 Acked Read 4 Done Done Sent false
19476 6318 Acked Write 3 Done Done Sent false
19477 6319 Acked Write 6 Done Done Sent false
19478 6320 Acked Read 8 Done Done Sent false
19479 6321 Acked Read 6 Done Done Sent false
19480 6322 Acked Read 9 Done Done Sent false
19481 6323 Acked Write 2 Done Done Sent false
19482 6324 Acked Read 8 Done Done Sent false
19483 6325 Acked Read 4 Done Done Sent false
19484 6326 Acked Write 10 Done Done Sent false
19485 6327 Acked Flush 0 Done Done Sent false
19486 6328 Acked Read 4 Done Done Sent false
19487 6329 Acked Read 6 Done Done Sent false
19488 6330 Acked Write 6 Done Done Sent false
19489 6331 Acked Flush 0 Done Done Sent false
19490 6332 Acked Write 2 Done Done Sent false
19491 6333 Acked Read 8 Done Done Sent false
19492 6334 Acked Read 9 Done Done Sent false
19493 6335 Acked Write 1 Done Done Sent false
19494 6336 Acked Read 2 Done Done Sent false
19495 6337 Acked Flush 0 Done Done Sent false
19496 6338 Acked Read 10 Done Done Sent false
19497 6339 Acked Write 5 Done Done Sent false
19498 6340 Acked Read 4 Done Done Sent false
19499 6341 Acked Read 8 Done Done Sent false
19500 6342 Acked Flush 0 Done Done Sent false
19501 6343 Acked Read 10 Done Done Sent false
19502 6344 Acked Write 9 Done Done Sent false
19503 6345 Acked Read 10 Done Done Sent false
19504 6346 Acked Read 5 Done Done Sent false
19505 6347 Acked Read 4 Done Done Sent false
19506 6348 Acked Write 3 Done Done Sent false
19507 6349 Acked Write 6 Done Done Sent false
19508 6350 Acked Write 5 Done Done Sent false
19509 6351 Acked Read 9 Done Done Sent false
19510 6352 Acked Read 10 Done Done Sent false
19511 6353 Acked Read 7 Done Done Sent false
19512 6354 Acked Read 2 Done Done Sent false
19513 6355 Acked Write 3 Done Done Sent false
19514 6356 Acked Read 4 Done Done Sent false
19515 6357 Acked Read 7 Done Done Sent false
19516 6358 Acked Read 5 Done Done Sent false
19517 6359 Acked Write 6 Done Done Sent false
19518 6360 Acked Write 2 Done Done Sent false
19519 6361 Acked Read 9 Done Done Sent false
19520 6362 Acked Read 3 Done Done Sent false
19521 6363 Acked Read 9 Done Done Sent false
19522 6364 Acked Write 5 Done Done Sent false
19523 6365 Acked Read 8 Done Done Sent false
19524 6366 Acked Write 6 Done Done Sent false
19525 6367 Acked Read 3 Done Done Sent false
19526 6368 Acked Write 5 Done Done Sent false
19527 6369 Acked Write 2 Done Done Sent false
19528 6370 Acked Read 6 Done Done Sent false
19529 6371 Acked Flush 0 Done Done Sent false
19530 6372 Acked Write 7 Done Done Sent false
19531 6373 Acked Write 7 Done Done Sent false
19532 6374 Acked Write 6 Done Done Sent false
19533 6375 Acked Write 3 Done Done Sent false
19534 6376 Acked Read 7 Done Done Sent false
19535 6377 Acked Read 10 Done Done Sent false
19536 6378 Acked Write 3 Done Done Sent false
19537 6379 Acked Read 1 Done Done Sent false
19538 6380 Acked Write 4 Done Done Sent false
19539 6381 Acked Read 7 Done Done Sent false
19540 6382 Acked Flush 0 Done Done Sent false
19541 6383 Acked Write 6 Done Done Sent false
19542 6384 Acked Read 3 Done Done Sent false
19543 6385 Acked Read 2 Done Done Sent false
19544 6386 Acked Read 3 Done Done Sent false
19545 6387 Acked Write 3 Done Done Sent false
19546 6388 Acked Read 7 Done Done Sent false
19547 6389 Acked Write 1 Done Done Sent false
19548 6390 Acked Write 2 Done Done Sent false
19549 6391 Acked Write 6 Done Done Sent false
19550 6392 Acked Read 9 Done Done Sent false
19551 6393 Acked Write 5 Done Done Sent false
19552 6394 Acked Write 2 Done Done Sent false
19553 6395 Acked Write 5 Done Done Sent false
19554 6396 Acked Write 4 Done Done Sent false
19555 6397 Acked Read 4 Done Done Sent false
19556 6398 Acked Write 1 Done Done Sent false
19557 6399 Acked Write 7 Done Done Sent false
19558 6400 Acked Write 6 Done Done Sent false
19559 6401 Acked Read 8 Done Done Sent false
19560 6402 Acked Write 3 Done Done Sent false
19561 6403 Acked Read 5 Done Done Sent false
19562 6404 Acked Write 2 Done Done Sent false
19563 6405 Acked Write 7 Done Done Sent false
19564 6406 Acked Write 7 Done Done Sent false
19565 6407 Acked Write 7 Done Done Sent false
19566 6408 Acked Write 4 Done Done Sent false
19567 6409 Acked Write 3 Done Done Sent false
19568 6410 Acked Write 7 Done Done Sent false
19569 6411 Acked Write 10 Done Done Sent false
19570 6412 Acked Write 6 Done Done Sent false
19571 6413 Acked Write 10 Done Done Sent false
19572 6414 Acked Read 5 Done Done Sent false
19573 6415 Acked Write 5 Done Done Sent false
19574 6416 Acked Read 1 Done Done Sent false
19575 6417 Acked Read 6 Done Done Sent false
19576 6418 Acked Flush 0 Done Done Sent false
19577 6419 Acked Write 8 Done Done Sent false
19578 6420 Acked Read 9 Done Done Sent false
19579 6421 Acked Read 10 Done Done Sent false
19580 6422 Acked Read 5 Done Done Sent false
19581 6423 Acked Write 7 Done Done Sent false
19582 6424 Acked Write 1 Done Done Sent false
19583 6425 Acked Read 6 Done Done Sent false
19584 6426 Acked Write 5 Done Done Sent false
19585 6427 Acked Write 5 Done Done Sent false
19586 6428 Acked Write 10 Done Done Sent false
19587 6429 Acked Read 9 Done Done Sent false
19588 6430 Acked Write 2 Done Done Sent false
19589 6431 Acked Read 6 Done Done Sent false
19590 6432 Acked Write 5 Done Done Sent false
19591 6433 Acked Read 6 Done Done Sent false
19592 6434 Acked Read 7 Done Done Sent false
19593 6435 Acked Read 2 Done Done Sent false
19594 6436 Acked Write 8 Done Done Sent false
19595 6437 Acked Write 4 Done Done Sent false
19596 6438 Acked Read 4 Done Done Sent false
19597 6439 Acked Flush 0 Done Done Sent false
19598 STATES DS:0 DS:1 DS:2 TOTAL
19599 Sent 0 0 302 302
19600 Done 302 302 0 604
19601 Skipped 0 0 0 0
19602 Error 0 0 0 0
19603 Last Flush: 6439 6439 6137
19604 Downstairs last five completed: 6137 6136 6135 6134 6133
19605 Upstairs last five completed: 6439 6438 6437 6436 6435
19606 CLIENT: Up:0 ds:302 act:2
196072024-11-01T16:54:35.313ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
196082024-11-01T16:54:35.313ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46797) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
196092024-11-01T16:54:35.314ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
196102024-11-01T16:54:35.315ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
196112024-11-01T16:54:35.315ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
196122024-11-01T16:54:35.315ZINFOcrucible: send last flush ID to this DS: 6137 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
196132024-11-01T16:54:35.315ZINFOcrucible: Replied this last flush ID: 6137 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
196142024-11-01T16:54:35.315ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
196152024-11-01T16:54:35.315ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
196162024-11-01T16:54:35.315ZINFOcrucible: [2] client re-new 302 jobs since flush 6137 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
196172024-11-01T16:54:35.315ZINFOcrucible: [2] Replayed 302 jobs since flush: 6137 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
19618 ----------------------------------------------------------------
19619 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19620 Downstairs last five completed: 6439 6438 6437 6436 6435
19621 Upstairs last five completed: 6439 6438 6437 6436 6435
19622 CLIENT: Up:0 ds:0 act:3
19623 Replay: All jobs finished
19624 Main task has work to do, go find it
19625 got dsc Stop(1)
19626 stop 1
19627 [8820] Got stop action so:false kr:false
19628 Nov 01 16:54:37.627 INFO request completed, latency_us: 422, response_code: 204, uri: /stop/cid/1, method: POST, req_id: ceac98d7-640f-443d-813b-d1959abde8ce, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
196292024-11-01T16:54:37.628ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
196302024-11-01T16:54:37.628ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
19631 Nov 01 16:54:37.628 INFO request completed, latency_us: 115, response_code: 200, {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:54:37.628334731Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
19632 uri: /state/cid/1, method: GET, req_id: 371fea0e-b759-444b-84f7-085e02b1c464{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:54:37.628361247Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"client":"1","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
19633 , remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
196342024-11-01T16:54:37.628ZINFOcrucible: client task is exiting = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
19635 [8820] Exited with: ExitStatus(unix_wait_status(9))
19636 [8820][1] reports Exit
19637 Nov 01 16:54:41.630 INFO request completed, latency_us: 222, response_code: 200, uri: /state/cid/1, method: GET, req_id: d6c55c74-b2a7-4a51-b591-323890eaea6f, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
19638 001/300 Write block 113 len 36864 data: 5 5 7 7 6 6 4 6 6
19639 002/300 Read block 2406 len 16384
19640 003/300 Write block 4124 len 40960 data: 2 2 2 2 2 2 3 3 2 2
19641 004/300 Read block 4238 len 40960
19642 005/300 Read block 1795 len 28672
19643 006/300 Read block 4268 len 28672
19644 007/300 Write block 2831 len 4096 data: 3
19645 008/300 Write block 1095 len 8192 data: 4 3
19646 009/300 Write block 207 len 20480 data: 4 4 5 4 3
19647 010/300 Flush
19648 011/300 Flush
19649 012/300 Write block 2745 len 24576 data: 8 6 6 4 4 3
19650 013/300 Read block 3787 len 20480
19651 014/300 Flush
19652 015/300 Read block 4323 len 32768
19653 016/300 Write block 1088 len 20480 data: 7 7 6 7 6
19654 017/300 Read block 3959 len 20480
19655 018/300 Read block 2455 len 4096
19656 019/300 Read block 790 len 8192
19657 020/300 Write block 3428 len 32768 data: 4 4 3 3 2 2 3 3
19658 021/300 Write block 3214 len 12288 data: 3 3 3
19659 022/300 Write block 185 len 36864 data: 3 3 3 3 2 4 4 4 4
19660 023/300 Read block 4817 len 32768
19661 024/300 Write block 2600 len 16384 data: 4 3 3 3
19662 025/300 Write block 1443 len 32768 data: 3 3 4 4 4 3 3 4
19663 026/300 Read block 1887 len 24576
19664 027/300 Read block 1019 len 32768
19665 028/300 Read block 3022 len 36864
19666 029/300 Read block 3459 len 4096
19667 030/300 Flush
19668 031/300 Read block 1527 len 24576
19669 032/300 Write block 3779 len 12288 data: 4 4 4
19670 033/300 Read block 2519 len 4096
19671 034/300 Write block 3063 len 20480 data: 3 2 2 2 2
19672 035/300 Write block 2104 len 8192 data: 3 3
19673 036/300 Write block 4309 len 16384 data: 6 8 9 9
19674 037/300 Write block 1846 len 24576 data: 4 4 5 5 5 4
19675 038/300 Write block 3168 len 40960 data: 3 3 3 5 5 7 6 4 3 3
19676 039/300 Write block 1201 len 20480 data: 4 4 5 5 4
19677 040/300 Read block 3659 len 4096
19678 041/300 Write block 1033 len 20480 data: 2 4 4 4 4
19679 042/300 Read block 2472 len 32768
19680 043/300 Write block 1123 len 20480 data: 3 2 3 3 3
19681 044/300 Read block 202 len 40960
19682 045/300 Write block 899 len 40960 data: 3 3 3 4 4 3 3 4 4 3
19683 046/300 Read block 3396 len 32768
19684 047/300 Read block 1440 len 8192
19685 048/300 Read block 3299 len 28672
19686 049/300 Flush
19687 050/300 Read block 2215 len 16384
19688 051/300 Write block 3071 len 32768 data: 3 3 4 5 5 7 6 5
19689 052/300 Read block 4229 len 20480
19690 053/300 Read block 915 len 12288
19691 054/300 Read block 3154 len 4096
19692 055/300 Read block 1675 len 16384
19693 056/300 Read block 4369 len 4096
19694 057/300 Read block 1990 len 32768
19695 058/300 Write block 3656 len 4096 data: 4
19696 059/300 Flush
19697 060/300 Write block 233 len 20480 data: 2 3 3 3 3
19698 061/300 Read block 58 len 12288
19699 062/300 Read block 1697 len 20480
19700 063/300 Write block 4569 len 20480 data: 2 2 2 2 2
19701 064/300 Read block 2700 len 24576
19702 065/300 Write block 2655 len 40960 data: 3 3 4 5 4 4 4 4 4 4
19703 066/300 Read block 1447 len 36864
19704 067/300 Read block 643 len 4096
19705 068/300 Write block 1810 len 24576 data: 5 5 5 3 3 3
19706 069/300 Flush
19707 070/300 Write block 1000 len 28672 data: 7 7 6 5 5 5 4
19708 071/300 Read block 4562 len 28672
19709 072/300 Read block 3310 len 40960
19710 073/300 Flush
19711 074/300 Read block 3458 len 32768
19712 075/300 Read block 2127 len 24576
19713 076/300 Read block 76 len 8192
19714 077/300 Flush
19715 078/300 Read block 2726 len 20480
19716 079/300 Read block 1025 len 20480
19717 080/300 Write block 1317 len 32768 data: 7 6 5 6 5 5 7 4
19718 081/300 Read block 695 len 32768
19719 082/300 Read block 2991 len 24576
19720 083/300 Read block 3699 len 16384
19721 084/300 Read block 3891 len 12288
19722 085/300 Read block 3773 len 36864
19723 086/300 Write block 1561 len 4096 data: 4
19724 087/300 Write block 3195 len 8192 data: 3 3
19725 088/300 Read block 4327 len 20480
19726 089/300 Read block 4935 len 36864
19727 090/300 Read block 1312 len 32768
19728 091/300 Read block 3580 len 8192
19729 092/300 Write block 3980 len 20480 data: 4 5 5 4 4
19730 093/300 Read block 708 len 24576
19731 094/300 Read block 272 len 28672
19732 095/300 Read block 4641 len 20480
19733 096/300 Flush
19734 097/300 Write block 3076 len 20480 data: 8 7 6 4 4
19735 098/300 Read block 2192 len 4096
19736 099/300 Read block 3765 len 40960
19737 100/300 Write block 1700 len 4096 data: 4
19738 101/300 Read block 3245 len 32768
19739 102/300 Write block 1113 len 28672 data: 2 2 2 4 4 4 5
19740 103/300 Read block 2898 len 28672
19741 104/300 Write block 2048 len 32768 data: 3 3 3 4 4 4 4 4
19742 105/300 Flush
19743 106/300 Read block 1063 len 20480
19744 107/300 Flush
19745 108/300 Read block 2371 len 28672
19746 109/300 Write block 4491 len 4096 data: 4
19747 110/300 Write block 93 len 8192 data: 7 6
19748 111/300 Read block 3896 len 32768
19749 112/300 Write block 1435 len 24576 data: 7 8 7 6 4 3
19750 113/300 Write block 895 len 4096 data: 6
19751 114/300 Read block 213 len 4096
19752 115/300 Write block 4553 len 8192 data: 5 6
19753 116/300 Write block 1859 len 8192 data: 3 3
19754 117/300 Read block 1161 len 32768
19755 118/300 Read block 1253 len 8192
19756 119/300 Write block 4933 len 36864 data: 3 3 3 2 2 2 3 4 5
19757 120/300 Write block 4503 len 16384 data: 6 7 7 7
19758 121/300 Read block 2221 len 20480
19759 122/300 Flush
19760 123/300 Write block 1544 len 24576 data: 3 3 3 3 3 3
19761 124/300 Write block 657 len 4096 data: 7
19762 125/300 Read block 2128 len 32768
19763 126/300 Write block 257 len 40960 data: 5 5 4 4 5 3 3 4 4 4
19764 127/300 Write block 1139 len 12288 data: 4 4 3
19765 128/300 Read block 837 len 36864
19766 129/300 Read block 2189 len 12288
19767 130/300 Read block 1411 len 8192
19768 131/300 Flush
19769 132/300 Write block 242 len 28672 data: 5 6 6 5 8 7 8
19770 133/300 Read block 2852 len 8192
19771 134/300 Write block 4315 len 24576 data: 8 7 9 12 11 11
19772 135/300 Read block 2801 len 28672
19773 136/300 Read block 40 len 4096
19774 137/300 Read block 2003 len 20480
19775 138/300 Write block 1786 len 24576 data: 6 5 5 4 5 5
19776 139/300 Read block 787 len 20480
19777 140/300 Read block 1808 len 32768
19778 141/300 Read block 1165 len 28672
19779 142/300 Read block 3297 len 4096
19780 143/300 Read block 3148 len 16384
19781 144/300 Read block 2428 len 16384
19782 145/300 Flush
19783 146/300 Write block 2133 len 24576 data: 3 3 3 4 4 3
19784 147/300 Write block 3872 len 20480 data: 5 5 4 3 3
19785 148/300 Read block 1993 len 20480
19786 149/300 Read block 3898 len 20480
19787 150/300 Write block 4572 len 16384 data: 3 3 2 2
19788 151/300 Read block 948 len 36864
19789 152/300 Write block 4024 len 4096 data: 6
19790 153/300 Read block 2504 len 8192
19791 154/300 Write block 1364 len 4096 data: 4
19792 155/300 Read block 571 len 12288
19793 156/300 Read block 4518 len 20480
19794 157/300 Read block 4623 len 8192
19795 158/300 Flush
19796 159/300 Read block 567 len 4096
19797 160/300 Write block 3239 len 4096 data: 4
19798 161/300 Flush
19799 162/300 Write block 1426 len 28672 data: 5 5 4 4 4 6 6
19800 163/300 Read block 2251 len 4096
19801 164/300 Read block 988 len 28672
19802 165/300 Read block 2487 len 24576
19803 166/300 Write block 1870 len 12288 data: 6 5 7
19804 167/300 Read block 4605 len 32768
19805 168/300 Read block 4971 len 16384
19806 169/300 Read block 1701 len 36864
19807 170/300 Read block 1856 len 8192
19808 171/300 Write block 2914 len 24576 data: 4 4 4 4 3 3
19809 172/300 Read block 38 len 28672
19810 173/300 Write block 1510 len 28672 data: 8 7 8 7 6 6 5
19811 174/300 Write block 1020 len 20480 data: 4 3 4 5 5
19812 175/300 Flush
19813 176/300 Read block 4825 len 24576
19814 177/300 Read block 735 len 36864
19815 178/300 Write block 4077 len 36864 data: 6 5 5 4 4 4 4 3 4
19816 179/300 Read block 3248 len 16384
19817 180/300 Read block 1347 len 32768
19818 181/300 Read block 2894 len 12288
19819 182/300 Flush
19820 183/300 Flush
19821 184/300 Write block 4516 len 24576 data: 6 7 6 6 5 6
19822 185/300 Flush
19823 186/300 Read block 2098 len 16384
19824 187/300 Read block 4978 len 4096
19825 188/300 Read block 1452 len 28672
19826 189/300 Write block 3414 len 36864 data: 4 4 4 5 5 3 3 4 5
19827 190/300 Read block 1221 len 4096
19828 191/300 Read block 3750 len 32768
19829 192/300 Read block 4412 len 16384
19830 193/300 Write block 2000 len 24576 data: 4 4 4 5 5 4
19831 194/300 Write block 563 len 12288 data: 3 3 3
19832 195/300 Read block 4610 len 12288
19833 196/300 Read block 983 len 8192
19834 197/300 Read block 503 len 4096
19835 198/300 Read block 674 len 28672
19836 199/300 Read block 2488 len 28672
19837 200/300 Write block 3092 len 20480 data: 7 7 9 8 6
19838 201/300 Read block 4627 len 36864
19839 202/300 Write block 1384 len 28672 data: 3 3 3 3 3 2 2
19840 203/300 Write block 690 len 20480 data: 3 5 5 4 6
19841 204/300 Read block 3803 len 40960
19842 205/300 Read block 3228 len 12288
19843 206/300 Write block 2055 len 8192 data: 5 4
19844 207/300 Read block 292 len 16384
19845 208/300 Write block 2812 len 40960 data: 6 7 6 7 6 5 4 4 4 4
19846 209/300 Write block 1703 len 4096 data: 2
19847 210/300 Read block 2807 len 40960
19848 211/300 Read block 1849 len 40960
19849 212/300 Read block 673 len 40960
19850 213/300 Write block 1362 len 36864 data: 5 4 5 4 4 4 3 3 3
19851 214/300 Write block 2407 len 24576 data: 3 3 3 2 2 2
19852 215/300 Read block 3839 len 16384
19853 216/300 Write block 2162 len 24576 data: 4 4 4 4 3 3
19854 217/300 Write block 4052 len 24576 data: 7 7 6 5 5 5
19855 218/300 Read block 2638 len 28672
19856 219/300 Read block 1747 len 20480
19857 220/300 Write block 1594 len 4096 data: 7
19858 221/300 Read block 4993 len 16384
19859 222/300 Read block 4655 len 4096
19860 223/300 Write block 2868 len 20480 data: 5 4 5 6 4
19861 224/300 Read block 3159 len 32768
19862 225/300 Read block 1256 len 32768
19863 226/300 Read block 1231 len 20480
19864 227/300 Read block 1122 len 36864
19865 228/300 Flush
19866 229/300 Write block 127 len 16384 data: 7 5 4 4
19867 230/300 Write block 1188 len 16384 data: 4 5 5 5
19868 231/300 Read block 1464 len 24576
19869 232/300 Read block 4171 len 36864
19870 233/300 Write block 2685 len 12288 data: 5 5 2
19871 234/300 Write block 3070 len 20480 data: 3 4 4 5 6
19872 235/300 Read block 4518 len 12288
19873 236/300 Read block 3019 len 24576
19874 237/300 Read block 3059 len 12288
19875 238/300 Read block 416 len 28672
19876 239/300 Read block 2384 len 4096
19877 240/300 Read block 2083 len 40960
19878 241/300 Read block 2770 len 32768
19879 242/300 Read block 4526 len 36864
19880 243/300 Flush
19881 244/300 Read block 2051 len 8192
19882 245/300 Write block 1167 len 4096 data: 2
19883 246/300 Read block 3782 len 28672
19884 247/300 Write block 348 len 24576 data: 3 2 2 2 2 2
19885 248/300 Write block 2322 len 12288 data: 3 3 3
19886 249/300 Read block 2440 len 28672
19887 250/300 Read block 3376 len 16384
19888 251/300 Read block 1079 len 32768
19889 252/300 Read block 2813 len 8192
19890 253/300 Read block 3941 len 32768
19891 254/300 Read block 1979 len 36864
19892 255/300 Read block 1230 len 8192
19893 256/300 Read block 3356 len 24576
19894 257/300 Read block 3797 len 16384
19895 258/300 Read block 2683 len 16384
19896 259/300 Read block 2526 len 4096
19897 260/300 Write block 2551 len 32768 data: 5 5 6 7 5 6 5 5
19898 261/300 Flush
19899 262/300 Read block 1218 len 16384
19900 263/300 Write block 4272 len 4096 data: 4
19901 264/300 Write block 104 len 32768 data: 3 4 4 4 5 4 4 3
19902 265/300 Read block 923 len 32768
19903 266/300 Read block 4255 len 4096
19904 267/300 Read block 4505 len 24576
19905 268/300 Write block 4789 len 8192 data: 5 4
19906 269/300 Read block 2738 len 20480
19907 270/300 Read block 3388 len 36864
19908 271/300 Read block 1203 len 36864
19909 272/300 Flush
19910 273/300 Read block 2870 len 36864
19911 274/300 Read block 3199 len 16384
19912 275/300 Read block 1644 len 32768
19913 276/300 Write block 1940 len 28672 data: 4 4 5 6 6 7 7
19914 277/300 Read block 2784 len 8192
19915 278/300 Write block 3836 len 12288 data: 5 4 4
19916 279/300 Read block 4928 len 36864
19917 280/300 Write block 2926 len 32768 data: 3 3 3 5 6 6 6 4
19918 281/300 Write block 4742 len 24576 data: 6 3 2 4 4 4
19919 282/300 Read block 2403 len 32768
19920 283/300 Read block 4467 len 40960
19921 284/300 Read block 3050 len 24576
19922 285/300 Read block 60 len 40960
19923 286/300 Write block 299 len 40960 data: 5 3 2 3 3 3 3 4 4 3
19924 287/300 Write block 1884 len 20480 data: 5 2 2 2 2
19925 288/300 Read block 3017 len 4096
19926 289/300 Write block 511 len 32768 data: 6 6 4 5 4 6 6 5
19927 290/300 Read block 1509 len 32768
19928 291/300 Read block 2912 len 32768
19929 292/300 Read block 800 len 24576
19930 293/300 Write block 3173 len 16384 data: 8 7 5 4
19931 294/300 Read block 3418 len 4096
19932 295/300 Read block 2586 len 16384
19933 296/300 Write block 2484 len 12288 data: 4 4 4
19934 297/300 Read block 3084 len 12288
19935 298/300 Write block 379 len 40960 data: 3 5 3 3 3 4 4 4 3 3
19936 299/300 Write block 4573 len 32768 data: 4 3 3 3 3 5 4 4
19937 300/300 Read block 826 len 20480
19938 301/300 Read block 1367 len 12288
19939 Main task has work to do, go find it
19940 got dsc Start(1)
19941 start 1
19942 Nov 01 16:54:41.916 INFO request completed, latency_us: 327, response_code: 204, uri: /start/cid/1, method: POST, req_id: a6328c5c-8c3f-4a42-a625-8e5d673d745a, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
19943 Replay: started 1, returned:Ok(())
19944 ----------------------------------------------------------------
19945 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
19946 States: Active Offline Active
19947 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19948 6440 Acked Write 9 Done Sent Done false
19949 6441 Acked Read 4 Done Sent Done false
19950 6442 Acked Write 10 Done Sent Done false
19951 6443 Acked Read 10 Done Sent Done false
19952 6444 Acked Read 7 Done Sent Done false
19953 6445 Acked Read 7 Done Sent Done false
19954 6446 Acked Write 1 Done Sent Done false
19955 6447 Acked Write 2 Done Sent Done false
19956 6448 Acked Write 5 Done Sent Done false
19957 6449 Acked Flush 0 Done Sent Done false
19958 6450 Acked Flush 0 Done Sent Done false
19959 6451 Acked Write 6 Done Sent Done false
19960 6452 Acked Read 5 Done Sent Done false
19961 6453 Acked Flush 0 Done Sent Done false
19962 6454 Acked Read 8 Done Sent Done false
19963 6455 Acked Write 5 Done Sent Done false
19964 6456 Acked Read 5 Done Sent Done false
19965 6457 Acked Read 1 Done Sent Done false
19966 6458 Acked Read 2 Done Sent Done false
19967 6459 Acked Write 8 Done Sent Done false
19968 6460 Acked Write 3 Done Sent Done false
19969 6461 Acked Write 9 Done Sent Done false
19970 6462 Acked Read 8 Done Sent Done false
19971 6463 Acked Write 4 Done Sent Done false
19972 6464 Acked Write 8 Done Sent Done false
19973 6465 Acked Read 6 Done Sent Done false
19974 6466 Acked Read 8 Done Sent Done false
19975 6467 Acked Read 9 Done Sent Done false
19976 6468 Acked Read 1 Done Sent Done false
19977 6469 Acked Flush 0 Done Sent Done false
19978 6470 Acked Read 6 Done Sent Done false
19979 6471 Acked Write 3 Done Sent Done false
19980 6472 Acked Read 1 Done Sent Done false
19981 6473 Acked Write 5 Done Sent Done false
19982 6474 Acked Write 2 Done Sent Done false
19983 6475 Acked Write 4 Done Sent Done false
19984 6476 Acked Write 6 Done Sent Done false
19985 6477 Acked Write 10 Done Sent Done false
19986 6478 Acked Write 5 Done Sent Done false
19987 6479 Acked Read 1 Done Sent Done false
19988 6480 Acked Write 5 Done Sent Done false
19989 6481 Acked Read 8 Done Sent Done false
19990 6482 Acked Write 5 Done Sent Done false
19991 6483 Acked Read 10 Done Sent Done false
19992 6484 Acked Write 10 Done Sent Done false
19993 6485 Acked Read 8 Done Sent Done false
19994 6486 Acked Read 2 Done Sent Done false
19995 6487 Acked Read 7 Done Sent Done false
19996 6488 Acked Flush 0 Done Sent Done false
19997 6489 Acked Read 4 Done Sent Done false
19998 6490 Acked Write 8 Done Sent Done false
19999 6491 Acked Read 5 Done Sent Done false
20000 6492 Acked Read 3 Done Sent Done false
20001 6493 Acked Read 1 Done Sent Done false
20002 6494 Acked Read 4 Done Sent Done false
20003 6495 Acked Read 1 Done Sent Done false
20004 6496 Acked Read 8 Done Sent Done false
20005 6497 Acked Write 1 Done Sent Done false
20006 6498 Acked Flush 0 Done Sent Done false
20007 6499 Acked Write 5 Done Sent Done false
20008 6500 Acked Read 3 Done Sent Done false
20009 6501 Acked Read 5 Done Sent Done false
20010 6502 Acked Write 5 Done Sent Done false
20011 6503 Acked Read 6 Done Sent Done false
20012 6504 Acked Write 10 Done Sent Done false
20013 6505 Acked Read 9 Done Sent Done false
20014 6506 Acked Read 1 Done Sent Done false
20015 6507 Acked Write 6 Done Sent Done false
20016 6508 Acked Flush 0 Done Sent Done false
20017 6509 Acked Write 7 Done Sent Done false
20018 6510 Acked Read 7 Done Sent Done false
20019 6511 Acked Read 10 Done Sent Done false
20020 6512 Acked Flush 0 Done Sent Done false
20021 6513 Acked Read 8 Done Sent Done false
20022 6514 Acked Read 6 Done Sent Done false
20023 6515 Acked Read 2 Done Sent Done false
20024 6516 Acked Flush 0 Done Sent Done false
20025 6517 Acked Read 5 Done Sent Done false
20026 6518 Acked Read 5 Done Sent Done false
20027 6519 Acked Write 8 Done Sent Done false
20028 6520 Acked Read 8 Done Sent Done false
20029 6521 Acked Read 6 Done Sent Done false
20030 6522 Acked Read 4 Done Sent Done false
20031 6523 Acked Read 3 Done Sent Done false
20032 6524 Acked Read 9 Done Sent Done false
20033 6525 Acked Write 1 Done Sent Done false
20034 6526 Acked Write 2 Done Sent Done false
20035 6527 Acked Read 5 Done Sent Done false
20036 6528 Acked Read 9 Done Sent Done false
20037 6529 Acked Read 8 Done Sent Done false
20038 6530 Acked Read 2 Done Sent Done false
20039 6531 Acked Write 5 Done Sent Done false
20040 6532 Acked Read 6 Done Sent Done false
20041 6533 Acked Read 7 Done Sent Done false
20042 6534 Acked Read 5 Done Sent Done false
20043 6535 Acked Flush 0 Done Sent Done false
20044 6536 Acked Write 5 Done Sent Done false
20045 6537 Acked Read 1 Done Sent Done false
20046 6538 Acked Read 10 Done Sent Done false
20047 6539 Acked Write 1 Done Sent Done false
20048 6540 Acked Read 8 Done Sent Done false
20049 6541 Acked Write 7 Done Sent Done false
20050 6542 Acked Read 7 Done Sent Done false
20051 6543 Acked Write 8 Done Sent Done false
20052 6544 Acked Flush 0 Done Sent Done false
20053 6545 Acked Read 5 Done Sent Done false
20054 6546 Acked Flush 0 Done Sent Done false
20055 6547 Acked Read 7 Done Sent Done false
20056 6548 Acked Write 1 Done Sent Done false
20057 6549 Acked Write 2 Done Sent Done false
20058 6550 Acked Read 8 Done Sent Done false
20059 6551 Acked Write 6 Done Sent Done false
20060 6552 Acked Write 1 Done Sent Done false
20061 6553 Acked Read 1 Done Sent Done false
20062 6554 Acked Write 2 Done Sent Done false
20063 6555 Acked Write 2 Done Sent Done false
20064 6556 Acked Read 8 Done Sent Done false
20065 6557 Acked Read 2 Done Sent Done false
20066 6558 Acked Write 9 Done Sent Done false
20067 6559 Acked Write 4 Done Sent Done false
20068 6560 Acked Read 5 Done Sent Done false
20069 6561 Acked Flush 0 Done Sent Done false
20070 6562 Acked Write 6 Done Sent Done false
20071 6563 Acked Write 1 Done Sent Done false
20072 6564 Acked Read 8 Done Sent Done false
20073 6565 Acked Write 10 Done Sent Done false
20074 6566 Acked Write 3 Done Sent Done false
20075 6567 Acked Read 9 Done Sent Done false
20076 6568 Acked Read 3 Done Sent Done false
20077 6569 Acked Read 2 Done Sent Done false
20078 6570 Acked Flush 0 Done Sent Done false
20079 6571 Acked Write 7 Done Sent Done false
20080 6572 Acked Read 2 Done Sent Done false
20081 6573 Acked Write 6 Done Sent Done false
20082 6574 Acked Read 7 Done Sent Done false
20083 6575 Acked Read 1 Done Sent Done false
20084 6576 Acked Read 5 Done Sent Done false
20085 6577 Acked Write 6 Done Sent Done false
20086 6578 Acked Read 5 Done Sent Done false
20087 6579 Acked Read 8 Done Sent Done false
20088 6580 Acked Read 7 Done Sent Done false
20089 6581 Acked Read 1 Done Sent Done false
20090 6582 Acked Read 4 Done Sent Done false
20091 6583 Acked Read 4 Done Sent Done false
20092 6584 Acked Flush 0 Done Sent Done false
20093 6585 Acked Write 6 Done Sent Done false
20094 6586 Acked Write 5 Done Sent Done false
20095 6587 Acked Read 5 Done Sent Done false
20096 6588 Acked Read 5 Done Sent Done false
20097 6589 Acked Write 4 Done Sent Done false
20098 6590 Acked Read 9 Done Sent Done false
20099 6591 Acked Write 1 Done Sent Done false
20100 6592 Acked Read 2 Done Sent Done false
20101 6593 Acked Write 1 Done Sent Done false
20102 6594 Acked Read 3 Done Sent Done false
20103 6595 Acked Flush 0 Done Sent Done false
20104 6596 Acked Read 5 Done Sent Done false
20105 6597 Acked Read 2 Done Sent Done false
20106 6598 Acked Flush 0 Done Sent Done false
20107 6599 Acked Read 1 Done Sent Done false
20108 6600 Acked Write 1 Done Sent Done false
20109 6601 Acked Flush 0 Done Sent Done false
20110 6602 Acked Write 7 Done Sent Done false
20111 6603 Acked Read 1 Done Sent Done false
20112 6604 Acked Read 7 Done Sent Done false
20113 6605 Acked Read 6 Done Sent Done false
20114 6606 Acked Write 3 Done Sent Done false
20115 6607 Acked Read 8 Done Sent Done false
20116 6608 Acked Read 4 Done Sent Done false
20117 6609 Acked Read 9 Done Sent Done false
20118 6610 Acked Read 2 Done Sent Done false
20119 6611 Acked Write 6 Done Sent Done false
20120 6612 Acked Read 7 Done Sent Done false
20121 6613 Acked Write 7 Done Sent Done false
20122 6614 Acked Write 5 Done Sent Done false
20123 6615 Acked Flush 0 Done Sent Done false
20124 6616 Acked Read 6 Done Sent Done false
20125 6617 Acked Read 9 Done Sent Done false
20126 6618 Acked Write 9 Done Sent Done false
20127 6619 Acked Read 4 Done Sent Done false
20128 6620 Acked Read 8 Done Sent Done false
20129 6621 Acked Read 3 Done Sent Done false
20130 6622 Acked Flush 0 Done Sent Done false
20131 6623 Acked Flush 0 Done Sent Done false
20132 6624 Acked Write 6 Done Sent Done false
20133 6625 Acked Flush 0 Done Sent Done false
20134 6626 Acked Read 4 Done Sent Done false
20135 6627 Acked Read 1 Done Sent Done false
20136 6628 Acked Read 7 Done Sent Done false
20137 6629 Acked Write 9 Done Sent Done false
20138 6630 Acked Read 1 Done Sent Done false
20139 6631 Acked Read 8 Done Sent Done false
20140 6632 Acked Read 4 Done Sent Done false
20141 6633 Acked Write 6 Done Sent Done false
20142 6634 Acked Write 3 Done Sent Done false
20143 6635 Acked Read 3 Done Sent Done false
20144 6636 Acked Read 2 Done Sent Done false
20145 6637 Acked Read 1 Done Sent Done false
20146 6638 Acked Read 7 Done Sent Done false
20147 6639 Acked Read 7 Done Sent Done false
20148 6640 Acked Write 5 Done Sent Done false
20149 6641 Acked Read 9 Done Sent Done false
20150 6642 Acked Write 7 Done Sent Done false
20151 6643 Acked Write 5 Done Sent Done false
20152 6644 Acked Read 10 Done Sent Done false
20153 6645 Acked Read 3 Done Sent Done false
20154 6646 Acked Write 2 Done Sent Done false
20155 6647 Acked Read 4 Done Sent Done false
20156 6648 Acked Write 10 Done Sent Done false
20157 6649 Acked Write 1 Done Sent Done false
20158 6650 Acked Read 10 Done Sent Done false
20159 6651 Acked Read 10 Done Sent Done false
20160 6652 Acked Read 10 Done Sent Done false
20161 6653 Acked Write 9 Done Sent Done false
20162 6654 Acked Write 6 Done Sent Done false
20163 6655 Acked Read 4 Done Sent Done false
20164 6656 Acked Write 6 Done Sent Done false
20165 6657 Acked Write 6 Done Sent Done false
20166 6658 Acked Read 7 Done Sent Done false
20167 6659 Acked Read 5 Done Sent Done false
20168 6660 Acked Write 1 Done Sent Done false
20169 6661 Acked Read 4 Done Sent Done false
20170 6662 Acked Read 1 Done Sent Done false
20171 6663 Acked Write 5 Done Sent Done false
20172 6664 Acked Read 8 Done Sent Done false
20173 6665 Acked Read 8 Done Sent Done false
20174 6666 Acked Read 5 Done Sent Done false
20175 6667 Acked Read 9 Done Sent Done false
20176 6668 Acked Flush 0 Done Sent Done false
20177 6669 Acked Write 4 Done Sent Done false
20178 6670 Acked Write 4 Done Sent Done false
20179 6671 Acked Read 6 Done Sent Done false
20180 6672 Acked Read 9 Done Sent Done false
20181 6673 Acked Write 3 Done Sent Done false
20182 6674 Acked Write 5 Done Sent Done false
20183 6675 Acked Read 3 Done Sent Done false
20184 6676 Acked Read 6 Done Sent Done false
20185 6677 Acked Read 3 Done Sent Done false
20186 6678 Acked Read 7 Done Sent Done false
20187 6679 Acked Read 1 Done Sent Done false
20188 6680 Acked Read 10 Done Sent Done false
20189 6681 Acked Read 8 Done Sent Done false
20190 6682 Acked Read 9 Done Sent Done false
20191 6683 Acked Flush 0 Done Sent Done false
20192 6684 Acked Read 2 Done Sent Done false
20193 6685 Acked Write 1 Done Sent Done false
20194 6686 Acked Read 7 Done Sent Done false
20195 6687 Acked Write 6 Done Sent Done false
20196 6688 Acked Write 3 Done Sent Done false
20197 6689 Acked Read 7 Done Sent Done false
20198 6690 Acked Read 4 Done Sent Done false
20199 6691 Acked Read 8 Done Sent Done false
20200 6692 Acked Read 2 Done Sent Done false
20201 6693 Acked Read 8 Done Sent Done false
20202 6694 Acked Read 9 Done Sent Done false
20203 6695 Acked Read 2 Done Sent Done false
20204 6696 Acked Read 6 Done Sent Done false
20205 6697 Acked Read 4 Done Sent Done false
20206 6698 Acked Read 4 Done Sent Done false
20207 6699 Acked Read 1 Done Sent Done false
20208 6700 Acked Write 8 Done Sent Done false
20209 6701 Acked Flush 0 Done Sent Done false
20210 6702 Acked Read 4 Done Sent Done false
20211 6703 Acked Write 1 Done Sent Done false
20212 6704 Acked Write 8 Done Sent Done false
20213 6705 Acked Read 8 Done Sent Done false
20214 6706 Acked Read 1 Done Sent Done false
20215 6707 Acked Read 6 Done Sent Done false
20216 6708 Acked Write 2 Done Sent Done false
20217 6709 Acked Read 5 Done Sent Done false
20218 6710 Acked Read 9 Done Sent Done false
20219 6711 Acked Read 9 Done Sent Done false
20220 6712 Acked Flush 0 Done Sent Done false
20221 6713 Acked Read 9 Done Sent Done false
20222 6714 Acked Read 4 Done Sent Done false
20223 6715 Acked Read 8 Done Sent Done false
20224 6716 Acked Write 7 Done Sent Done false
20225 6717 Acked Read 2 Done Sent Done false
20226 6718 Acked Write 3 Done Sent Done false
20227 6719 Acked Read 9 Done Sent Done false
20228 6720 Acked Write 8 Done Sent Done false
20229 6721 Acked Write 6 Done Sent Done false
20230 6722 Acked Read 8 Done Sent Done false
20231 6723 Acked Read 10 Done Sent Done false
20232 6724 Acked Read 6 Done Sent Done false
20233 6725 Acked Read 10 Done Sent Done false
20234 6726 Acked Write 10 Done Sent Done false
20235 6727 Acked Write 5 Done Sent Done false
20236 6728 Acked Read 1 Done Sent Done false
20237 6729 Acked Write 8 Done Sent Done false
20238 6730 Acked Read 8 Done Sent Done false
20239 6731 Acked Read 8 Done Sent Done false
20240 6732 Acked Read 6 Done Sent Done false
20241 6733 Acked Write 4 Done Sent Done false
20242 6734 Acked Read 1 Done Sent Done false
20243 6735 Acked Read 4 Done Sent Done false
20244 6736 Acked Write 3 Done Sent Done false
20245 6737 Acked Read 3 Done Sent Done false
20246 6738 Acked Write 10 Done Sent Done false
20247 6739 Acked Write 8 Done Sent Done false
20248 6740 Acked Read 5 Done Sent Done false
20249 6741 Acked Read 3 Done Sent Done false
20250 STATES DS:0 DS:1 DS:2 TOTAL
20251 Sent 0 302 0 302
20252 Done 302 0 302 604
20253 Skipped 0 0 0 0
20254 Error 0 0 0 0
20255 Last Flush: 6712 6439 6712
20256 Downstairs last five completed: 6439 6438 6437 6436 6435
20257 Upstairs last five completed: 6741 6740 6739 6738 6737
20258 CLIENT: Up:0 ds:302 act:2
20259 ----------------------------------------------------------------
20260 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
20261 States: Active Offline Active
20262 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20263 6440 Acked Write 9 Done Sent Done false
20264 6441 Acked Read 4 Done Sent Done false
20265 6442 Acked Write 10 Done Sent Done false
20266 6443 Acked Read 10 Done Sent Done false
20267 6444 Acked Read 7 Done Sent Done false
20268 6445 Acked Read 7 Done Sent Done false
20269 6446 Acked Write 1 Done Sent Done false
20270 6447 Acked Write 2 Done Sent Done false
20271 6448 Acked Write 5 Done Sent Done false
20272 6449 Acked Flush 0 Done Sent Done false
20273 6450 Acked Flush 0 Done Sent Done false
20274 6451 Acked Write 6 Done Sent Done false
20275 6452 Acked Read 5 Done Sent Done false
20276 6453 Acked Flush 0 Done Sent Done false
20277 6454 Acked Read 8 Done Sent Done false
20278 6455 Acked Write 5 Done Sent Done false
20279 6456 Acked Read 5 Done Sent Done false
20280 6457 Acked Read 1 Done Sent Done false
20281 6458 Acked Read 2 Done Sent Done false
20282 6459 Acked Write 8 Done Sent Done false
20283 6460 Acked Write 3 Done Sent Done false
20284 6461 Acked Write 9 Done Sent Done false
20285 6462 Acked Read 8 Done Sent Done false
20286 6463 Acked Write 4 Done Sent Done false
20287 6464 Acked Write 8 Done Sent Done false
20288 6465 Acked Read 6 Done Sent Done false
20289 6466 Acked Read 8 Done Sent Done false
20290 6467 Acked Read 9 Done Sent Done false
20291 6468 Acked Read 1 Done Sent Done false
20292 6469 Acked Flush 0 Done Sent Done false
20293 6470 Acked Read 6 Done Sent Done false
20294 6471 Acked Write 3 Done Sent Done false
20295 6472 Acked Read 1 Done Sent Done false
20296 6473 Acked Write 5 Done Sent Done false
20297 6474 Acked Write 2 Done Sent Done false
20298 6475 Acked Write 4 Done Sent Done false
20299 6476 Acked Write 6 Done Sent Done false
20300 6477 Acked Write 10 Done Sent Done false
20301 6478 Acked Write 5 Done Sent Done false
20302 6479 Acked Read 1 Done Sent Done false
20303 6480 Acked Write 5 Done Sent Done false
20304 6481 Acked Read 8 Done Sent Done false
20305 6482 Acked Write 5 Done Sent Done false
20306 6483 Acked Read 10 Done Sent Done false
20307 6484 Acked Write 10 Done Sent Done false
20308 6485 Acked Read 8 Done Sent Done false
20309 6486 Acked Read 2 Done Sent Done false
20310 6487 Acked Read 7 Done Sent Done false
20311 6488 Acked Flush 0 Done Sent Done false
20312 6489 Acked Read 4 Done Sent Done false
20313 6490 Acked Write 8 Done Sent Done false
20314 6491 Acked Read 5 Done Sent Done false
20315 6492 Acked Read 3 Done Sent Done false
20316 6493 Acked Read 1 Done Sent Done false
20317 6494 Acked Read 4 Done Sent Done false
20318 6495 Acked Read 1 Done Sent Done false
20319 6496 Acked Read 8 Done Sent Done false
20320 6497 Acked Write 1 Done Sent Done false
20321 6498 Acked Flush 0 Done Sent Done false
20322 6499 Acked Write 5 Done Sent Done false
20323 6500 Acked Read 3 Done Sent Done false
20324 6501 Acked Read 5 Done Sent Done false
20325 6502 Acked Write 5 Done Sent Done false
20326 6503 Acked Read 6 Done Sent Done false
20327 6504 Acked Write 10 Done Sent Done false
20328 6505 Acked Read 9 Done Sent Done false
20329 6506 Acked Read 1 Done Sent Done false
20330 6507 Acked Write 6 Done Sent Done false
20331 6508 Acked Flush 0 Done Sent Done false
20332 6509 Acked Write 7 Done Sent Done false
20333 6510 Acked Read 7 Done Sent Done false
20334 6511 Acked Read 10 Done Sent Done false
20335 6512 Acked Flush 0 Done Sent Done false
20336 6513 Acked Read 8 Done Sent Done false
20337 6514 Acked Read 6 Done Sent Done false
20338 6515 Acked Read 2 Done Sent Done false
20339 6516 Acked Flush 0 Done Sent Done false
20340 6517 Acked Read 5 Done Sent Done false
20341 6518 Acked Read 5 Done Sent Done false
20342 6519 Acked Write 8 Done Sent Done false
20343 6520 Acked Read 8 Done Sent Done false
20344 6521 Acked Read 6 Done Sent Done false
20345 6522 Acked Read 4 Done Sent Done false
20346 6523 Acked Read 3 Done Sent Done false
20347 6524 Acked Read 9 Done Sent Done false
20348 6525 Acked Write 1 Done Sent Done false
20349 6526 Acked Write 2 Done Sent Done false
20350 6527 Acked Read 5 Done Sent Done false
20351 6528 Acked Read 9 Done Sent Done false
20352 6529 Acked Read 8 Done Sent Done false
20353 6530 Acked Read 2 Done Sent Done false
20354 6531 Acked Write 5 Done Sent Done false
20355 6532 Acked Read 6 Done Sent Done false
20356 6533 Acked Read 7 Done Sent Done false
20357 6534 Acked Read 5 Done Sent Done false
20358 6535 Acked Flush 0 Done Sent Done false
20359 6536 Acked Write 5 Done Sent Done false
20360 6537 Acked Read 1 Done Sent Done false
20361 6538 Acked Read 10 Done Sent Done false
20362 6539 Acked Write 1 Done Sent Done false
20363 6540 Acked Read 8 Done Sent Done false
20364 6541 Acked Write 7 Done Sent Done false
20365 6542 Acked Read 7 Done Sent Done false
20366 6543 Acked Write 8 Done Sent Done false
20367 6544 Acked Flush 0 Done Sent Done false
20368 6545 Acked Read 5 Done Sent Done false
20369 6546 Acked Flush 0 Done Sent Done false
20370 6547 Acked Read 7 Done Sent Done false
20371 6548 Acked Write 1 Done Sent Done false
20372 6549 Acked Write 2 Done Sent Done false
20373 6550 Acked Read 8 Done Sent Done false
20374 6551 Acked Write 6 Done Sent Done false
20375 6552 Acked Write 1 Done Sent Done false
20376 6553 Acked Read 1 Done Sent Done false
20377 6554 Acked Write 2 Done Sent Done false
20378 6555 Acked Write 2 Done Sent Done false
20379 6556 Acked Read 8 Done Sent Done false
20380 6557 Acked Read 2 Done Sent Done false
20381 6558 Acked Write 9 Done Sent Done false
20382 6559 Acked Write 4 Done Sent Done false
20383 6560 Acked Read 5 Done Sent Done false
20384 6561 Acked Flush 0 Done Sent Done false
20385 6562 Acked Write 6 Done Sent Done false
20386 6563 Acked Write 1 Done Sent Done false
20387 6564 Acked Read 8 Done Sent Done false
20388 6565 Acked Write 10 Done Sent Done false
20389 6566 Acked Write 3 Done Sent Done false
20390 6567 Acked Read 9 Done Sent Done false
20391 6568 Acked Read 3 Done Sent Done false
20392 6569 Acked Read 2 Done Sent Done false
20393 6570 Acked Flush 0 Done Sent Done false
20394 6571 Acked Write 7 Done Sent Done false
20395 6572 Acked Read 2 Done Sent Done false
20396 6573 Acked Write 6 Done Sent Done false
20397 6574 Acked Read 7 Done Sent Done false
20398 6575 Acked Read 1 Done Sent Done false
20399 6576 Acked Read 5 Done Sent Done false
20400 6577 Acked Write 6 Done Sent Done false
20401 6578 Acked Read 5 Done Sent Done false
20402 6579 Acked Read 8 Done Sent Done false
20403 6580 Acked Read 7 Done Sent Done false
20404 6581 Acked Read 1 Done Sent Done false
20405 6582 Acked Read 4 Done Sent Done false
20406 6583 Acked Read 4 Done Sent Done false
20407 6584 Acked Flush 0 Done Sent Done false
20408 6585 Acked Write 6 Done Sent Done false
20409 6586 Acked Write 5 Done Sent Done false
20410 6587 Acked Read 5 Done Sent Done false
20411 6588 Acked Read 5 Done Sent Done false
20412 6589 Acked Write 4 Done Sent Done false
20413 6590 Acked Read 9 Done Sent Done false
20414 6591 Acked Write 1 Done Sent Done false
20415 6592 Acked Read 2 Done Sent Done false
20416 6593 Acked Write 1 Done Sent Done false
20417 6594 Acked Read 3 Done Sent Done false
20418 6595 Acked Flush 0 Done Sent Done false
20419 6596 Acked Read 5 Done Sent Done false
20420 6597 Acked Read 2 Done Sent Done false
20421 6598 Acked Flush 0 Done Sent Done false
20422 6599 Acked Read 1 Done Sent Done false
20423 6600 Acked Write 1 Done Sent Done false
20424 6601 Acked Flush 0 Done Sent Done false
20425 6602 Acked Write 7 Done Sent Done false
20426 6603 Acked Read 1 Done Sent Done false
20427 6604 Acked Read 7 Done Sent Done false
20428 6605 Acked Read 6 Done Sent Done false
20429 6606 Acked Write 3 Done Sent Done false
20430 6607 Acked Read 8 Done Sent Done false
20431 6608 Acked Read 4 Done Sent Done false
20432 6609 Acked Read 9 Done Sent Done false
20433 6610 Acked Read 2 Done Sent Done false
20434 6611 Acked Write 6 Done Sent Done false
20435 6612 Acked Read 7 Done Sent Done false
20436 6613 Acked Write 7 Done Sent Done false
20437 6614 Acked Write 5 Done Sent Done false
20438 6615 Acked Flush 0 Done Sent Done false
20439 6616 Acked Read 6 Done Sent Done false
20440 6617 Acked Read 9 Done Sent Done false
20441 6618 Acked Write 9 Done Sent Done false
20442 6619 Acked Read 4 Done Sent Done false
20443 6620 Acked Read 8 Done Sent Done false
20444 6621 Acked Read 3 Done Sent Done false
20445 6622 Acked Flush 0 Done Sent Done false
20446 6623 Acked Flush 0 Done Sent Done false
20447 6624 Acked Write 6 Done Sent Done false
20448 6625 Acked Flush 0 Done Sent Done false
20449 6626 Acked Read 4 Done Sent Done false
20450 6627 Acked Read 1 Done Sent Done false
20451 6628 Acked Read 7 Done Sent Done false
20452 6629 Acked Write 9 Done Sent Done false
20453 6630 Acked Read 1 Done Sent Done false
20454 6631 Acked Read 8 Done Sent Done false
20455 6632 Acked Read 4 Done Sent Done false
20456 6633 Acked Write 6 Done Sent Done false
20457 6634 Acked Write 3 Done Sent Done false
20458 6635 Acked Read 3 Done Sent Done false
20459 6636 Acked Read 2 Done Sent Done false
20460 6637 Acked Read 1 Done Sent Done false
20461 6638 Acked Read 7 Done Sent Done false
20462 6639 Acked Read 7 Done Sent Done false
20463 6640 Acked Write 5 Done Sent Done false
20464 6641 Acked Read 9 Done Sent Done false
20465 6642 Acked Write 7 Done Sent Done false
20466 6643 Acked Write 5 Done Sent Done false
20467 6644 Acked Read 10 Done Sent Done false
20468 6645 Acked Read 3 Done Sent Done false
20469 6646 Acked Write 2 Done Sent Done false
20470 6647 Acked Read 4 Done Sent Done false
20471 6648 Acked Write 10 Done Sent Done false
20472 6649 Acked Write 1 Done Sent Done false
20473 6650 Acked Read 10 Done Sent Done false
20474 6651 Acked Read 10 Done Sent Done false
20475 6652 Acked Read 10 Done Sent Done false
20476 6653 Acked Write 9 Done Sent Done false
20477 6654 Acked Write 6 Done Sent Done false
20478 6655 Acked Read 4 Done Sent Done false
20479 6656 Acked Write 6 Done Sent Done false
20480 6657 Acked Write 6 Done Sent Done false
20481 6658 Acked Read 7 Done Sent Done false
20482 6659 Acked Read 5 Done Sent Done false
20483 6660 Acked Write 1 Done Sent Done false
20484 6661 Acked Read 4 Done Sent Done false
20485 6662 Acked Read 1 Done Sent Done false
20486 6663 Acked Write 5 Done Sent Done false
20487 6664 Acked Read 8 Done Sent Done false
20488 6665 Acked Read 8 Done Sent Done false
20489 6666 Acked Read 5 Done Sent Done false
20490 6667 Acked Read 9 Done Sent Done false
20491 6668 Acked Flush 0 Done Sent Done false
20492 6669 Acked Write 4 Done Sent Done false
20493 6670 Acked Write 4 Done Sent Done false
20494 6671 Acked Read 6 Done Sent Done false
20495 6672 Acked Read 9 Done Sent Done false
20496 6673 Acked Write 3 Done Sent Done false
20497 6674 Acked Write 5 Done Sent Done false
20498 6675 Acked Read 3 Done Sent Done false
20499 6676 Acked Read 6 Done Sent Done false
20500 6677 Acked Read 3 Done Sent Done false
20501 6678 Acked Read 7 Done Sent Done false
20502 6679 Acked Read 1 Done Sent Done false
20503 6680 Acked Read 10 Done Sent Done false
20504 6681 Acked Read 8 Done Sent Done false
20505 6682 Acked Read 9 Done Sent Done false
20506 6683 Acked Flush 0 Done Sent Done false
20507 6684 Acked Read 2 Done Sent Done false
20508 6685 Acked Write 1 Done Sent Done false
20509 6686 Acked Read 7 Done Sent Done false
20510 6687 Acked Write 6 Done Sent Done false
20511 6688 Acked Write 3 Done Sent Done false
20512 6689 Acked Read 7 Done Sent Done false
20513 6690 Acked Read 4 Done Sent Done false
20514 6691 Acked Read 8 Done Sent Done false
20515 6692 Acked Read 2 Done Sent Done false
20516 6693 Acked Read 8 Done Sent Done false
20517 6694 Acked Read 9 Done Sent Done false
20518 6695 Acked Read 2 Done Sent Done false
20519 6696 Acked Read 6 Done Sent Done false
20520 6697 Acked Read 4 Done Sent Done false
20521 6698 Acked Read 4 Done Sent Done false
20522 6699 Acked Read 1 Done Sent Done false
20523 6700 Acked Write 8 Done Sent Done false
20524 6701 Acked Flush 0 Done Sent Done false
20525 6702 Acked Read 4 Done Sent Done false
20526 6703 Acked Write 1 Done Sent Done false
20527 6704 Acked Write 8 Done Sent Done false
20528 6705 Acked Read 8 Done Sent Done false
20529 6706 Acked Read 1 Done Sent Done false
20530 6707 Acked Read 6 Done Sent Done false
20531 6708 Acked Write 2 Done Sent Done false
20532 6709 Acked Read 5 Done Sent Done false
20533 6710 Acked Read 9 Done Sent Done false
20534 6711 Acked Read 9 Done Sent Done false
20535 6712 Acked Flush 0 Done Sent Done false
20536 6713 Acked Read 9 Done Sent Done false
20537 6714 Acked Read 4 Done Sent Done false
20538 6715 Acked Read 8 Done Sent Done false
20539 6716 Acked Write 7 Done Sent Done false
20540 6717 Acked Read 2 Done Sent Done false
20541 6718 Acked Write 3 Done Sent Done false
20542 6719 Acked Read 9 Done Sent Done false
20543 6720 Acked Write 8 Done Sent Done false
20544 6721 Acked Write 6 Done Sent Done false
20545 6722 Acked Read 8 Done Sent Done false
20546 6723 Acked Read 10 Done Sent Done false
20547 6724 Acked Read 6 Done Sent Done false
20548 6725 Acked Read 10 Done Sent Done false
20549 6726 Acked Write 10 Done Sent Done false
20550 6727 Acked Write 5 Done Sent Done false
20551 6728 Acked Read 1 Done Sent Done false
20552 6729 Acked Write 8 Done Sent Done false
20553 6730 Acked Read 8 Done Sent Done false
20554 6731 Acked Read 8 Done Sent Done false
20555 6732 Acked Read 6 Done Sent Done false
20556 6733 Acked Write 4 Done Sent Done false
20557 6734 Acked Read 1 Done Sent Done false
20558 6735 Acked Read 4 Done Sent Done false
20559 6736 Acked Write 3 Done Sent Done false
20560 6737 Acked Read 3 Done Sent Done false
20561 6738 Acked Write 10 Done Sent Done false
20562 6739 Acked Write 8 Done Sent Done false
20563 6740 Acked Read 5 Done Sent Done false
20564 6741 Acked Read 3 Done Sent Done false
20565 6742 Acked Flush 0 Done Sent Done false
20566 STATES DS:0 DS:1 DS:2 TOTAL
20567 Sent 0 303 0 303
20568 Done 303 0 303 606
20569 Skipped 0 0 0 0
20570 Error 0 0 0 0
20571 Last Flush: 6742 6439 6742
20572 Downstairs last five completed: 6439 6438 6437 6436 6435
20573 Upstairs last five completed: 6742 6741 6740 6739 6738
20574 CLIENT: Up:0 ds:303 act:2
205752024-11-01T16:54:47.629ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
205762024-11-01T16:54:47.629ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
205772024-11-01T16:54:47.629ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
205782024-11-01T16:54:47.629ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
205792024-11-01T16:54:47.629ZINFOcrucible: client task is exiting = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
20580 [8820] got start action
20581 [8820] I am going to restart
20582 Starting downstairs at port 8820
20583 Make output file at "/tmp/dsc/downstairs-8820.txt"
20584 [8820][1] reports Starting
20585 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1217)
20586 [8820][1] reports Running
20587 ----------------------------------------------------------------
20588 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
20589 States: Active Offline Active
20590 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20591 6440 Acked Write 9 Done Sent Done false
20592 6441 Acked Read 4 Done Sent Done false
20593 6442 Acked Write 10 Done Sent Done false
20594 6443 Acked Read 10 Done Sent Done false
20595 6444 Acked Read 7 Done Sent Done false
20596 6445 Acked Read 7 Done Sent Done false
20597 6446 Acked Write 1 Done Sent Done false
20598 6447 Acked Write 2 Done Sent Done false
20599 6448 Acked Write 5 Done Sent Done false
20600 6449 Acked Flush 0 Done Sent Done false
20601 6450 Acked Flush 0 Done Sent Done false
20602 6451 Acked Write 6 Done Sent Done false
20603 6452 Acked Read 5 Done Sent Done false
20604 6453 Acked Flush 0 Done Sent Done false
20605 6454 Acked Read 8 Done Sent Done false
20606 6455 Acked Write 5 Done Sent Done false
20607 6456 Acked Read 5 Done Sent Done false
20608 6457 Acked Read 1 Done Sent Done false
20609 6458 Acked Read 2 Done Sent Done false
20610 6459 Acked Write 8 Done Sent Done false
20611 6460 Acked Write 3 Done Sent Done false
20612 6461 Acked Write 9 Done Sent Done false
20613 6462 Acked Read 8 Done Sent Done false
20614 6463 Acked Write 4 Done Sent Done false
20615 6464 Acked Write 8 Done Sent Done false
20616 6465 Acked Read 6 Done Sent Done false
20617 6466 Acked Read 8 Done Sent Done false
20618 6467 Acked Read 9 Done Sent Done false
20619 6468 Acked Read 1 Done Sent Done false
20620 6469 Acked Flush 0 Done Sent Done false
20621 6470 Acked Read 6 Done Sent Done false
20622 6471 Acked Write 3 Done Sent Done false
20623 6472 Acked Read 1 Done Sent Done false
20624 6473 Acked Write 5 Done Sent Done false
20625 6474 Acked Write 2 Done Sent Done false
20626 6475 Acked Write 4 Done Sent Done false
20627 6476 Acked Write 6 Done Sent Done false
20628 6477 Acked Write 10 Done Sent Done false
20629 6478 Acked Write 5 Done Sent Done false
20630 6479 Acked Read 1 Done Sent Done false
20631 6480 Acked Write 5 Done Sent Done false
20632 6481 Acked Read 8 Done Sent Done false
20633 6482 Acked Write 5 Done Sent Done false
20634 6483 Acked Read 10 Done Sent Done false
20635 6484 Acked Write 10 Done Sent Done false
20636 6485 Acked Read 8 Done Sent Done false
20637 6486 Acked Read 2 Done Sent Done false
20638 6487 Acked Read 7 Done Sent Done false
20639 6488 Acked Flush 0 Done Sent Done false
20640 6489 Acked Read 4 Done Sent Done false
20641 6490 Acked Write 8 Done Sent Done false
20642 6491 Acked Read 5 Done Sent Done false
20643 6492 Acked Read 3 Done Sent Done false
20644 6493 Acked Read 1 Done Sent Done false
20645 6494 Acked Read 4 Done Sent Done false
20646 6495 Acked Read 1 Done Sent Done false
20647 6496 Acked Read 8 Done Sent Done false
20648 6497 Acked Write 1 Done Sent Done false
20649 6498 Acked Flush 0 Done Sent Done false
20650 6499 Acked Write 5 Done Sent Done false
20651 6500 Acked Read 3 Done Sent Done false
20652 6501 Acked Read 5 Done Sent Done false
20653 6502 Acked Write 5 Done Sent Done false
20654 6503 Acked Read 6 Done Sent Done false
20655 6504 Acked Write 10 Done Sent Done false
20656 6505 Acked Read 9 Done Sent Done false
20657 6506 Acked Read 1 Done Sent Done false
20658 6507 Acked Write 6 Done Sent Done false
20659 6508 Acked Flush 0 Done Sent Done false
20660 6509 Acked Write 7 Done Sent Done false
20661 6510 Acked Read 7 Done Sent Done false
20662 6511 Acked Read 10 Done Sent Done false
20663 6512 Acked Flush 0 Done Sent Done false
20664 6513 Acked Read 8 Done Sent Done false
20665 6514 Acked Read 6 Done Sent Done false
20666 6515 Acked Read 2 Done Sent Done false
20667 6516 Acked Flush 0 Done Sent Done false
20668 6517 Acked Read 5 Done Sent Done false
20669 6518 Acked Read 5 Done Sent Done false
20670 6519 Acked Write 8 Done Sent Done false
20671 6520 Acked Read 8 Done Sent Done false
20672 6521 Acked Read 6 Done Sent Done false
20673 6522 Acked Read 4 Done Sent Done false
20674 6523 Acked Read 3 Done Sent Done false
20675 6524 Acked Read 9 Done Sent Done false
20676 6525 Acked Write 1 Done Sent Done false
20677 6526 Acked Write 2 Done Sent Done false
20678 6527 Acked Read 5 Done Sent Done false
20679 6528 Acked Read 9 Done Sent Done false
20680 6529 Acked Read 8 Done Sent Done false
20681 6530 Acked Read 2 Done Sent Done false
20682 6531 Acked Write 5 Done Sent Done false
20683 6532 Acked Read 6 Done Sent Done false
20684 6533 Acked Read 7 Done Sent Done false
20685 6534 Acked Read 5 Done Sent Done false
20686 6535 Acked Flush 0 Done Sent Done false
20687 6536 Acked Write 5 Done Sent Done false
20688 6537 Acked Read 1 Done Sent Done false
20689 6538 Acked Read 10 Done Sent Done false
20690 6539 Acked Write 1 Done Sent Done false
20691 6540 Acked Read 8 Done Sent Done false
20692 6541 Acked Write 7 Done Sent Done false
20693 6542 Acked Read 7 Done Sent Done false
20694 6543 Acked Write 8 Done Sent Done false
20695 6544 Acked Flush 0 Done Sent Done false
20696 6545 Acked Read 5 Done Sent Done false
20697 6546 Acked Flush 0 Done Sent Done false
20698 6547 Acked Read 7 Done Sent Done false
20699 6548 Acked Write 1 Done Sent Done false
20700 6549 Acked Write 2 Done Sent Done false
20701 6550 Acked Read 8 Done Sent Done false
20702 6551 Acked Write 6 Done Sent Done false
20703 6552 Acked Write 1 Done Sent Done false
20704 6553 Acked Read 1 Done Sent Done false
20705 6554 Acked Write 2 Done Sent Done false
20706 6555 Acked Write 2 Done Sent Done false
20707 6556 Acked Read 8 Done Sent Done false
20708 6557 Acked Read 2 Done Sent Done false
20709 6558 Acked Write 9 Done Sent Done false
20710 6559 Acked Write 4 Done Sent Done false
20711 6560 Acked Read 5 Done Sent Done false
20712 6561 Acked Flush 0 Done Sent Done false
20713 6562 Acked Write 6 Done Sent Done false
20714 6563 Acked Write 1 Done Sent Done false
20715 6564 Acked Read 8 Done Sent Done false
20716 6565 Acked Write 10 Done Sent Done false
20717 6566 Acked Write 3 Done Sent Done false
20718 6567 Acked Read 9 Done Sent Done false
20719 6568 Acked Read 3 Done Sent Done false
20720 6569 Acked Read 2 Done Sent Done false
20721 6570 Acked Flush 0 Done Sent Done false
20722 6571 Acked Write 7 Done Sent Done false
20723 6572 Acked Read 2 Done Sent Done false
20724 6573 Acked Write 6 Done Sent Done false
20725 6574 Acked Read 7 Done Sent Done false
20726 6575 Acked Read 1 Done Sent Done false
20727 6576 Acked Read 5 Done Sent Done false
20728 6577 Acked Write 6 Done Sent Done false
20729 6578 Acked Read 5 Done Sent Done false
20730 6579 Acked Read 8 Done Sent Done false
20731 6580 Acked Read 7 Done Sent Done false
20732 6581 Acked Read 1 Done Sent Done false
20733 6582 Acked Read 4 Done Sent Done false
20734 6583 Acked Read 4 Done Sent Done false
20735 6584 Acked Flush 0 Done Sent Done false
20736 6585 Acked Write 6 Done Sent Done false
20737 6586 Acked Write 5 Done Sent Done false
20738 6587 Acked Read 5 Done Sent Done false
20739 6588 Acked Read 5 Done Sent Done false
20740 6589 Acked Write 4 Done Sent Done false
20741 6590 Acked Read 9 Done Sent Done false
20742 6591 Acked Write 1 Done Sent Done false
20743 6592 Acked Read 2 Done Sent Done false
20744 6593 Acked Write 1 Done Sent Done false
20745 6594 Acked Read 3 Done Sent Done false
20746 6595 Acked Flush 0 Done Sent Done false
20747 6596 Acked Read 5 Done Sent Done false
20748 6597 Acked Read 2 Done Sent Done false
20749 6598 Acked Flush 0 Done Sent Done false
20750 6599 Acked Read 1 Done Sent Done false
20751 6600 Acked Write 1 Done Sent Done false
20752 6601 Acked Flush 0 Done Sent Done false
20753 6602 Acked Write 7 Done Sent Done false
20754 6603 Acked Read 1 Done Sent Done false
20755 6604 Acked Read 7 Done Sent Done false
20756 6605 Acked Read 6 Done Sent Done false
20757 6606 Acked Write 3 Done Sent Done false
20758 6607 Acked Read 8 Done Sent Done false
20759 6608 Acked Read 4 Done Sent Done false
20760 6609 Acked Read 9 Done Sent Done false
20761 6610 Acked Read 2 Done Sent Done false
20762 6611 Acked Write 6 Done Sent Done false
20763 6612 Acked Read 7 Done Sent Done false
20764 6613 Acked Write 7 Done Sent Done false
20765 6614 Acked Write 5 Done Sent Done false
20766 6615 Acked Flush 0 Done Sent Done false
20767 6616 Acked Read 6 Done Sent Done false
20768 6617 Acked Read 9 Done Sent Done false
20769 6618 Acked Write 9 Done Sent Done false
20770 6619 Acked Read 4 Done Sent Done false
20771 6620 Acked Read 8 Done Sent Done false
20772 6621 Acked Read 3 Done Sent Done false
20773 6622 Acked Flush 0 Done Sent Done false
20774 6623 Acked Flush 0 Done Sent Done false
20775 6624 Acked Write 6 Done Sent Done false
20776 6625 Acked Flush 0 Done Sent Done false
20777 6626 Acked Read 4 Done Sent Done false
20778 6627 Acked Read 1 Done Sent Done false
20779 6628 Acked Read 7 Done Sent Done false
20780 6629 Acked Write 9 Done Sent Done false
20781 6630 Acked Read 1 Done Sent Done false
20782 6631 Acked Read 8 Done Sent Done false
20783 6632 Acked Read 4 Done Sent Done false
20784 6633 Acked Write 6 Done Sent Done false
20785 6634 Acked Write 3 Done Sent Done false
20786 6635 Acked Read 3 Done Sent Done false
20787 6636 Acked Read 2 Done Sent Done false
20788 6637 Acked Read 1 Done Sent Done false
20789 6638 Acked Read 7 Done Sent Done false
20790 6639 Acked Read 7 Done Sent Done false
20791 6640 Acked Write 5 Done Sent Done false
20792 6641 Acked Read 9 Done Sent Done false
20793 6642 Acked Write 7 Done Sent Done false
20794 6643 Acked Write 5 Done Sent Done false
20795 6644 Acked Read 10 Done Sent Done false
20796 6645 Acked Read 3 Done Sent Done false
20797 6646 Acked Write 2 Done Sent Done false
20798 6647 Acked Read 4 Done Sent Done false
20799 6648 Acked Write 10 Done Sent Done false
20800 6649 Acked Write 1 Done Sent Done false
20801 6650 Acked Read 10 Done Sent Done false
20802 6651 Acked Read 10 Done Sent Done false
20803 6652 Acked Read 10 Done Sent Done false
20804 6653 Acked Write 9 Done Sent Done false
20805 6654 Acked Write 6 Done Sent Done false
20806 6655 Acked Read 4 Done Sent Done false
20807 6656 Acked Write 6 Done Sent Done false
20808 6657 Acked Write 6 Done Sent Done false
20809 6658 Acked Read 7 Done Sent Done false
20810 6659 Acked Read 5 Done Sent Done false
20811 6660 Acked Write 1 Done Sent Done false
20812 6661 Acked Read 4 Done Sent Done false
20813 6662 Acked Read 1 Done Sent Done false
20814 6663 Acked Write 5 Done Sent Done false
20815 6664 Acked Read 8 Done Sent Done false
20816 6665 Acked Read 8 Done Sent Done false
20817 6666 Acked Read 5 Done Sent Done false
20818 6667 Acked Read 9 Done Sent Done false
20819 6668 Acked Flush 0 Done Sent Done false
20820 6669 Acked Write 4 Done Sent Done false
20821 6670 Acked Write 4 Done Sent Done false
20822 6671 Acked Read 6 Done Sent Done false
20823 6672 Acked Read 9 Done Sent Done false
20824 6673 Acked Write 3 Done Sent Done false
20825 6674 Acked Write 5 Done Sent Done false
20826 6675 Acked Read 3 Done Sent Done false
20827 6676 Acked Read 6 Done Sent Done false
20828 6677 Acked Read 3 Done Sent Done false
20829 6678 Acked Read 7 Done Sent Done false
20830 6679 Acked Read 1 Done Sent Done false
20831 6680 Acked Read 10 Done Sent Done false
20832 6681 Acked Read 8 Done Sent Done false
20833 6682 Acked Read 9 Done Sent Done false
20834 6683 Acked Flush 0 Done Sent Done false
20835 6684 Acked Read 2 Done Sent Done false
20836 6685 Acked Write 1 Done Sent Done false
20837 6686 Acked Read 7 Done Sent Done false
20838 6687 Acked Write 6 Done Sent Done false
20839 6688 Acked Write 3 Done Sent Done false
20840 6689 Acked Read 7 Done Sent Done false
20841 6690 Acked Read 4 Done Sent Done false
20842 6691 Acked Read 8 Done Sent Done false
20843 6692 Acked Read 2 Done Sent Done false
20844 6693 Acked Read 8 Done Sent Done false
20845 6694 Acked Read 9 Done Sent Done false
20846 6695 Acked Read 2 Done Sent Done false
20847 6696 Acked Read 6 Done Sent Done false
20848 6697 Acked Read 4 Done Sent Done false
20849 6698 Acked Read 4 Done Sent Done false
20850 6699 Acked Read 1 Done Sent Done false
20851 6700 Acked Write 8 Done Sent Done false
20852 6701 Acked Flush 0 Done Sent Done false
20853 6702 Acked Read 4 Done Sent Done false
20854 6703 Acked Write 1 Done Sent Done false
20855 6704 Acked Write 8 Done Sent Done false
20856 6705 Acked Read 8 Done Sent Done false
20857 6706 Acked Read 1 Done Sent Done false
20858 6707 Acked Read 6 Done Sent Done false
20859 6708 Acked Write 2 Done Sent Done false
20860 6709 Acked Read 5 Done Sent Done false
20861 6710 Acked Read 9 Done Sent Done false
20862 6711 Acked Read 9 Done Sent Done false
20863 6712 Acked Flush 0 Done Sent Done false
20864 6713 Acked Read 9 Done Sent Done false
20865 6714 Acked Read 4 Done Sent Done false
20866 6715 Acked Read 8 Done Sent Done false
20867 6716 Acked Write 7 Done Sent Done false
20868 6717 Acked Read 2 Done Sent Done false
20869 6718 Acked Write 3 Done Sent Done false
20870 6719 Acked Read 9 Done Sent Done false
20871 6720 Acked Write 8 Done Sent Done false
20872 6721 Acked Write 6 Done Sent Done false
20873 6722 Acked Read 8 Done Sent Done false
20874 6723 Acked Read 10 Done Sent Done false
20875 6724 Acked Read 6 Done Sent Done false
20876 6725 Acked Read 10 Done Sent Done false
20877 6726 Acked Write 10 Done Sent Done false
20878 6727 Acked Write 5 Done Sent Done false
20879 6728 Acked Read 1 Done Sent Done false
20880 6729 Acked Write 8 Done Sent Done false
20881 6730 Acked Read 8 Done Sent Done false
20882 6731 Acked Read 8 Done Sent Done false
20883 6732 Acked Read 6 Done Sent Done false
20884 6733 Acked Write 4 Done Sent Done false
20885 6734 Acked Read 1 Done Sent Done false
20886 6735 Acked Read 4 Done Sent Done false
20887 6736 Acked Write 3 Done Sent Done false
20888 6737 Acked Read 3 Done Sent Done false
20889 6738 Acked Write 10 Done Sent Done false
20890 6739 Acked Write 8 Done Sent Done false
20891 6740 Acked Read 5 Done Sent Done false
20892 6741 Acked Read 3 Done Sent Done false
20893 6742 Acked Flush 0 Done Sent Done false
20894 STATES DS:0 DS:1 DS:2 TOTAL
20895 Sent 0 303 0 303
20896 Done 303 0 303 606
20897 Skipped 0 0 0 0
20898 Error 0 0 0 0
20899 Last Flush: 6742 6439 6742
20900 Downstairs last five completed: 6439 6438 6437 6436 6435
20901 Upstairs last five completed: 6742 6741 6740 6739 6738
20902 CLIENT: Up:0 ds:303 act:2
20903 ----------------------------------------------------------------
20904 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
20905 States: Active Offline Active
20906 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20907 6440 Acked Write 9 Done Sent Done false
20908 6441 Acked Read 4 Done Sent Done false
20909 6442 Acked Write 10 Done Sent Done false
20910 6443 Acked Read 10 Done Sent Done false
20911 6444 Acked Read 7 Done Sent Done false
20912 6445 Acked Read 7 Done Sent Done false
20913 6446 Acked Write 1 Done Sent Done false
20914 6447 Acked Write 2 Done Sent Done false
20915 6448 Acked Write 5 Done Sent Done false
20916 6449 Acked Flush 0 Done Sent Done false
20917 6450 Acked Flush 0 Done Sent Done false
20918 6451 Acked Write 6 Done Sent Done false
20919 6452 Acked Read 5 Done Sent Done false
20920 6453 Acked Flush 0 Done Sent Done false
20921 6454 Acked Read 8 Done Sent Done false
20922 6455 Acked Write 5 Done Sent Done false
20923 6456 Acked Read 5 Done Sent Done false
20924 6457 Acked Read 1 Done Sent Done false
20925 6458 Acked Read 2 Done Sent Done false
20926 6459 Acked Write 8 Done Sent Done false
20927 6460 Acked Write 3 Done Sent Done false
20928 6461 Acked Write 9 Done Sent Done false
20929 6462 Acked Read 8 Done Sent Done false
20930 6463 Acked Write 4 Done Sent Done false
20931 6464 Acked Write 8 Done Sent Done false
20932 6465 Acked Read 6 Done Sent Done false
20933 6466 Acked Read 8 Done Sent Done false
20934 6467 Acked Read 9 Done Sent Done false
20935 6468 Acked Read 1 Done Sent Done false
20936 6469 Acked Flush 0 Done Sent Done false
20937 6470 Acked Read 6 Done Sent Done false
20938 6471 Acked Write 3 Done Sent Done false
20939 6472 Acked Read 1 Done Sent Done false
20940 6473 Acked Write 5 Done Sent Done false
20941 6474 Acked Write 2 Done Sent Done false
20942 6475 Acked Write 4 Done Sent Done false
20943 6476 Acked Write 6 Done Sent Done false
20944 6477 Acked Write 10 Done Sent Done false
20945 6478 Acked Write 5 Done Sent Done false
20946 6479 Acked Read 1 Done Sent Done false
20947 6480 Acked Write 5 Done Sent Done false
20948 6481 Acked Read 8 Done Sent Done false
20949 6482 Acked Write 5 Done Sent Done false
20950 6483 Acked Read 10 Done Sent Done false
20951 6484 Acked Write 10 Done Sent Done false
20952 6485 Acked Read 8 Done Sent Done false
20953 6486 Acked Read 2 Done Sent Done false
20954 6487 Acked Read 7 Done Sent Done false
20955 6488 Acked Flush 0 Done Sent Done false
20956 6489 Acked Read 4 Done Sent Done false
20957 6490 Acked Write 8 Done Sent Done false
20958 6491 Acked Read 5 Done Sent Done false
20959 6492 Acked Read 3 Done Sent Done false
20960 6493 Acked Read 1 Done Sent Done false
20961 6494 Acked Read 4 Done Sent Done false
20962 6495 Acked Read 1 Done Sent Done false
20963 6496 Acked Read 8 Done Sent Done false
20964 6497 Acked Write 1 Done Sent Done false
20965 6498 Acked Flush 0 Done Sent Done false
20966 6499 Acked Write 5 Done Sent Done false
20967 6500 Acked Read 3 Done Sent Done false
20968 6501 Acked Read 5 Done Sent Done false
20969 6502 Acked Write 5 Done Sent Done false
20970 6503 Acked Read 6 Done Sent Done false
20971 6504 Acked Write 10 Done Sent Done false
20972 6505 Acked Read 9 Done Sent Done false
20973 6506 Acked Read 1 Done Sent Done false
20974 6507 Acked Write 6 Done Sent Done false
20975 6508 Acked Flush 0 Done Sent Done false
20976 6509 Acked Write 7 Done Sent Done false
20977 6510 Acked Read 7 Done Sent Done false
20978 6511 Acked Read 10 Done Sent Done false
20979 6512 Acked Flush 0 Done Sent Done false
20980 6513 Acked Read 8 Done Sent Done false
20981 6514 Acked Read 6 Done Sent Done false
20982 6515 Acked Read 2 Done Sent Done false
20983 6516 Acked Flush 0 Done Sent Done false
20984 6517 Acked Read 5 Done Sent Done false
20985 6518 Acked Read 5 Done Sent Done false
20986 6519 Acked Write 8 Done Sent Done false
20987 6520 Acked Read 8 Done Sent Done false
20988 6521 Acked Read 6 Done Sent Done false
20989 6522 Acked Read 4 Done Sent Done false
20990 6523 Acked Read 3 Done Sent Done false
20991 6524 Acked Read 9 Done Sent Done false
20992 6525 Acked Write 1 Done Sent Done false
20993 6526 Acked Write 2 Done Sent Done false
20994 6527 Acked Read 5 Done Sent Done false
20995 6528 Acked Read 9 Done Sent Done false
20996 6529 Acked Read 8 Done Sent Done false
20997 6530 Acked Read 2 Done Sent Done false
20998 6531 Acked Write 5 Done Sent Done false
20999 6532 Acked Read 6 Done Sent Done false
21000 6533 Acked Read 7 Done Sent Done false
21001 6534 Acked Read 5 Done Sent Done false
21002 6535 Acked Flush 0 Done Sent Done false
21003 6536 Acked Write 5 Done Sent Done false
21004 6537 Acked Read 1 Done Sent Done false
21005 6538 Acked Read 10 Done Sent Done false
21006 6539 Acked Write 1 Done Sent Done false
21007 6540 Acked Read 8 Done Sent Done false
21008 6541 Acked Write 7 Done Sent Done false
21009 6542 Acked Read 7 Done Sent Done false
21010 6543 Acked Write 8 Done Sent Done false
21011 6544 Acked Flush 0 Done Sent Done false
21012 6545 Acked Read 5 Done Sent Done false
21013 6546 Acked Flush 0 Done Sent Done false
21014 6547 Acked Read 7 Done Sent Done false
21015 6548 Acked Write 1 Done Sent Done false
21016 6549 Acked Write 2 Done Sent Done false
21017 6550 Acked Read 8 Done Sent Done false
21018 6551 Acked Write 6 Done Sent Done false
21019 6552 Acked Write 1 Done Sent Done false
21020 6553 Acked Read 1 Done Sent Done false
21021 6554 Acked Write 2 Done Sent Done false
21022 6555 Acked Write 2 Done Sent Done false
21023 6556 Acked Read 8 Done Sent Done false
21024 6557 Acked Read 2 Done Sent Done false
21025 6558 Acked Write 9 Done Sent Done false
21026 6559 Acked Write 4 Done Sent Done false
21027 6560 Acked Read 5 Done Sent Done false
21028 6561 Acked Flush 0 Done Sent Done false
21029 6562 Acked Write 6 Done Sent Done false
21030 6563 Acked Write 1 Done Sent Done false
21031 6564 Acked Read 8 Done Sent Done false
21032 6565 Acked Write 10 Done Sent Done false
21033 6566 Acked Write 3 Done Sent Done false
21034 6567 Acked Read 9 Done Sent Done false
21035 6568 Acked Read 3 Done Sent Done false
21036 6569 Acked Read 2 Done Sent Done false
21037 6570 Acked Flush 0 Done Sent Done false
21038 6571 Acked Write 7 Done Sent Done false
21039 6572 Acked Read 2 Done Sent Done false
21040 6573 Acked Write 6 Done Sent Done false
21041 6574 Acked Read 7 Done Sent Done false
21042 6575 Acked Read 1 Done Sent Done false
21043 6576 Acked Read 5 Done Sent Done false
21044 6577 Acked Write 6 Done Sent Done false
21045 6578 Acked Read 5 Done Sent Done false
21046 6579 Acked Read 8 Done Sent Done false
21047 6580 Acked Read 7 Done Sent Done false
21048 6581 Acked Read 1 Done Sent Done false
21049 6582 Acked Read 4 Done Sent Done false
21050 6583 Acked Read 4 Done Sent Done false
21051 6584 Acked Flush 0 Done Sent Done false
21052 6585 Acked Write 6 Done Sent Done false
21053 6586 Acked Write 5 Done Sent Done false
21054 6587 Acked Read 5 Done Sent Done false
21055 6588 Acked Read 5 Done Sent Done false
21056 6589 Acked Write 4 Done Sent Done false
21057 6590 Acked Read 9 Done Sent Done false
21058 6591 Acked Write 1 Done Sent Done false
21059 6592 Acked Read 2 Done Sent Done false
21060 6593 Acked Write 1 Done Sent Done false
21061 6594 Acked Read 3 Done Sent Done false
21062 6595 Acked Flush 0 Done Sent Done false
21063 6596 Acked Read 5 Done Sent Done false
21064 6597 Acked Read 2 Done Sent Done false
21065 6598 Acked Flush 0 Done Sent Done false
21066 6599 Acked Read 1 Done Sent Done false
21067 6600 Acked Write 1 Done Sent Done false
21068 6601 Acked Flush 0 Done Sent Done false
21069 6602 Acked Write 7 Done Sent Done false
21070 6603 Acked Read 1 Done Sent Done false
21071 6604 Acked Read 7 Done Sent Done false
21072 6605 Acked Read 6 Done Sent Done false
21073 6606 Acked Write 3 Done Sent Done false
21074 6607 Acked Read 8 Done Sent Done false
21075 6608 Acked Read 4 Done Sent Done false
21076 6609 Acked Read 9 Done Sent Done false
21077 6610 Acked Read 2 Done Sent Done false
21078 6611 Acked Write 6 Done Sent Done false
21079 6612 Acked Read 7 Done Sent Done false
21080 6613 Acked Write 7 Done Sent Done false
21081 6614 Acked Write 5 Done Sent Done false
21082 6615 Acked Flush 0 Done Sent Done false
21083 6616 Acked Read 6 Done Sent Done false
21084 6617 Acked Read 9 Done Sent Done false
21085 6618 Acked Write 9 Done Sent Done false
21086 6619 Acked Read 4 Done Sent Done false
21087 6620 Acked Read 8 Done Sent Done false
21088 6621 Acked Read 3 Done Sent Done false
21089 6622 Acked Flush 0 Done Sent Done false
21090 6623 Acked Flush 0 Done Sent Done false
21091 6624 Acked Write 6 Done Sent Done false
21092 6625 Acked Flush 0 Done Sent Done false
21093 6626 Acked Read 4 Done Sent Done false
21094 6627 Acked Read 1 Done Sent Done false
21095 6628 Acked Read 7 Done Sent Done false
21096 6629 Acked Write 9 Done Sent Done false
21097 6630 Acked Read 1 Done Sent Done false
21098 6631 Acked Read 8 Done Sent Done false
21099 6632 Acked Read 4 Done Sent Done false
21100 6633 Acked Write 6 Done Sent Done false
21101 6634 Acked Write 3 Done Sent Done false
21102 6635 Acked Read 3 Done Sent Done false
21103 6636 Acked Read 2 Done Sent Done false
21104 6637 Acked Read 1 Done Sent Done false
21105 6638 Acked Read 7 Done Sent Done false
21106 6639 Acked Read 7 Done Sent Done false
21107 6640 Acked Write 5 Done Sent Done false
21108 6641 Acked Read 9 Done Sent Done false
21109 6642 Acked Write 7 Done Sent Done false
21110 6643 Acked Write 5 Done Sent Done false
21111 6644 Acked Read 10 Done Sent Done false
21112 6645 Acked Read 3 Done Sent Done false
21113 6646 Acked Write 2 Done Sent Done false
21114 6647 Acked Read 4 Done Sent Done false
21115 6648 Acked Write 10 Done Sent Done false
21116 6649 Acked Write 1 Done Sent Done false
21117 6650 Acked Read 10 Done Sent Done false
21118 6651 Acked Read 10 Done Sent Done false
21119 6652 Acked Read 10 Done Sent Done false
21120 6653 Acked Write 9 Done Sent Done false
21121 6654 Acked Write 6 Done Sent Done false
21122 6655 Acked Read 4 Done Sent Done false
21123 6656 Acked Write 6 Done Sent Done false
21124 6657 Acked Write 6 Done Sent Done false
21125 6658 Acked Read 7 Done Sent Done false
21126 6659 Acked Read 5 Done Sent Done false
21127 6660 Acked Write 1 Done Sent Done false
21128 6661 Acked Read 4 Done Sent Done false
21129 6662 Acked Read 1 Done Sent Done false
21130 6663 Acked Write 5 Done Sent Done false
21131 6664 Acked Read 8 Done Sent Done false
21132 6665 Acked Read 8 Done Sent Done false
21133 6666 Acked Read 5 Done Sent Done false
21134 6667 Acked Read 9 Done Sent Done false
21135 6668 Acked Flush 0 Done Sent Done false
21136 6669 Acked Write 4 Done Sent Done false
21137 6670 Acked Write 4 Done Sent Done false
21138 6671 Acked Read 6 Done Sent Done false
21139 6672 Acked Read 9 Done Sent Done false
21140 6673 Acked Write 3 Done Sent Done false
21141 6674 Acked Write 5 Done Sent Done false
21142 6675 Acked Read 3 Done Sent Done false
21143 6676 Acked Read 6 Done Sent Done false
21144 6677 Acked Read 3 Done Sent Done false
21145 6678 Acked Read 7 Done Sent Done false
21146 6679 Acked Read 1 Done Sent Done false
21147 6680 Acked Read 10 Done Sent Done false
21148 6681 Acked Read 8 Done Sent Done false
21149 6682 Acked Read 9 Done Sent Done false
21150 6683 Acked Flush 0 Done Sent Done false
21151 6684 Acked Read 2 Done Sent Done false
21152 6685 Acked Write 1 Done Sent Done false
21153 6686 Acked Read 7 Done Sent Done false
21154 6687 Acked Write 6 Done Sent Done false
21155 6688 Acked Write 3 Done Sent Done false
21156 6689 Acked Read 7 Done Sent Done false
21157 6690 Acked Read 4 Done Sent Done false
21158 6691 Acked Read 8 Done Sent Done false
21159 6692 Acked Read 2 Done Sent Done false
21160 6693 Acked Read 8 Done Sent Done false
21161 6694 Acked Read 9 Done Sent Done false
21162 6695 Acked Read 2 Done Sent Done false
21163 6696 Acked Read 6 Done Sent Done false
21164 6697 Acked Read 4 Done Sent Done false
21165 6698 Acked Read 4 Done Sent Done false
21166 6699 Acked Read 1 Done Sent Done false
21167 6700 Acked Write 8 Done Sent Done false
21168 6701 Acked Flush 0 Done Sent Done false
21169 6702 Acked Read 4 Done Sent Done false
21170 6703 Acked Write 1 Done Sent Done false
21171 6704 Acked Write 8 Done Sent Done false
21172 6705 Acked Read 8 Done Sent Done false
21173 6706 Acked Read 1 Done Sent Done false
21174 6707 Acked Read 6 Done Sent Done false
21175 6708 Acked Write 2 Done Sent Done false
21176 6709 Acked Read 5 Done Sent Done false
21177 6710 Acked Read 9 Done Sent Done false
21178 6711 Acked Read 9 Done Sent Done false
21179 6712 Acked Flush 0 Done Sent Done false
21180 6713 Acked Read 9 Done Sent Done false
21181 6714 Acked Read 4 Done Sent Done false
21182 6715 Acked Read 8 Done Sent Done false
21183 6716 Acked Write 7 Done Sent Done false
21184 6717 Acked Read 2 Done Sent Done false
21185 6718 Acked Write 3 Done Sent Done false
21186 6719 Acked Read 9 Done Sent Done false
21187 6720 Acked Write 8 Done Sent Done false
21188 6721 Acked Write 6 Done Sent Done false
21189 6722 Acked Read 8 Done Sent Done false
21190 6723 Acked Read 10 Done Sent Done false
21191 6724 Acked Read 6 Done Sent Done false
21192 6725 Acked Read 10 Done Sent Done false
21193 6726 Acked Write 10 Done Sent Done false
21194 6727 Acked Write 5 Done Sent Done false
21195 6728 Acked Read 1 Done Sent Done false
21196 6729 Acked Write 8 Done Sent Done false
21197 6730 Acked Read 8 Done Sent Done false
21198 6731 Acked Read 8 Done Sent Done false
21199 6732 Acked Read 6 Done Sent Done false
21200 6733 Acked Write 4 Done Sent Done false
21201 6734 Acked Read 1 Done Sent Done false
21202 6735 Acked Read 4 Done Sent Done false
21203 6736 Acked Write 3 Done Sent Done false
21204 6737 Acked Read 3 Done Sent Done false
21205 6738 Acked Write 10 Done Sent Done false
21206 6739 Acked Write 8 Done Sent Done false
21207 6740 Acked Read 5 Done Sent Done false
21208 6741 Acked Read 3 Done Sent Done false
21209 6742 Acked Flush 0 Done Sent Done false
21210 STATES DS:0 DS:1 DS:2 TOTAL
21211 Sent 0 303 0 303
21212 Done 303 0 303 606
21213 Skipped 0 0 0 0
21214 Error 0 0 0 0
21215 Last Flush: 6742 6439 6742
21216 Downstairs last five completed: 6439 6438 6437 6436 6435
21217 Upstairs last five completed: 6742 6741 6740 6739 6738
21218 CLIENT: Up:0 ds:303 act:2
212192024-11-01T16:54:57.631ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
212202024-11-01T16:54:57.631ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51086) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
212212024-11-01T16:54:57.632ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
212222024-11-01T16:54:57.633ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
212232024-11-01T16:54:57.633ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
212242024-11-01T16:54:57.633ZINFOcrucible: send last flush ID to this DS: 6439 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
212252024-11-01T16:54:57.633ZINFOcrucible: Replied this last flush ID: 6439 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
212262024-11-01T16:54:57.633ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
212272024-11-01T16:54:57.633ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
212282024-11-01T16:54:57.633ZINFOcrucible: [1] client re-new 303 jobs since flush 6439 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
212292024-11-01T16:54:57.633ZINFOcrucible: [1] Replayed 303 jobs since flush: 6439 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21230 ----------------------------------------------------------------
21231 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21232 Downstairs last five completed: 6742 6741 6740 6739 6738
21233 Upstairs last five completed: 6742 6741 6740 6739 6738
21234 CLIENT: Up:0 ds:0 act:3
21235 Replay: All jobs finished
21236 Main task has work to do, go find it
21237 got dsc Stop(2)
21238 stop 2
21239 [8830] Got stop action so:false kr:false
21240 Nov 01 16:54:57.926 INFO request completed, latency_us: 358, response_code: 204, uri: /stop/cid/2, method: POST, req_id: d7cad579-c45a-41ef-be7c-ff7631399cb0, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
21241 Nov 01 16:54:57.927 INFO request completed, latency_us: 131, response_code: 200, uri: /state/cid/2, {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:54:57.927266765Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
21242 method: GET, req_id: 65ea81cb-b45e-4a84-bea6-c7a5ab7ce394, remote_addr: 127.0.0.1:58552{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:54:57.927343442Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
21243 , {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:54:57.927379707Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
21244 local_addr: 127.0.0.1:9998
212452024-11-01T16:54:57.927ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
212462024-11-01T16:54:57.927ZINFOcrucible: client task is exiting = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
21247 [8830] Exited with: ExitStatus(unix_wait_status(9))
21248 [8830][2] reports Exit
21249 Nov 01 16:55:01.929 INFO request completed, latency_us: 246, response_code: 200, uri: /state/cid/2, method: GET, req_id: dc3a9114-ccad-4bca-ae58-079e90cecbe3, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
21250 001/300 Write block 454 len 12288 data: 3 6 6
21251 002/300 Read block 3895 len 36864
21252 003/300 Read block 4891 len 28672
21253 004/300 Read block 4567 len 12288
21254 005/300 Read block 4483 len 36864
21255 006/300 Write block 2340 len 24576 data: 3 3 4 5 5 5
21256 007/300 Write block 2223 len 24576 data: 6 6 5 5 5 5
21257 008/300 Read block 4574 len 32768
21258 009/300 Write block 3597 len 28672 data: 4 3 3 3 2 2 2
21259 010/300 Write block 64 len 28672 data: 5 4 4 4 4 4 5
21260 011/300 Read block 313 len 20480
21261 012/300 Read block 1832 len 28672
21262 013/300 Read block 1449 len 28672
21263 014/300 Write block 2835 len 32768 data: 5 4 4 3 3 3 3 4
21264 015/300 Read block 3501 len 36864
21265 016/300 Write block 602 len 24576 data: 3 3 3 2 2 4
21266 017/300 Read block 95 len 4096
21267 018/300 Write block 1560 len 40960 data: 4 5 4 5 4 4 4 4 4 4
21268 019/300 Flush
21269 020/300 Read block 3946 len 32768
21270 021/300 Write block 1193 len 40960 data: 6 6 7 7 5 4 4 4 5 5
21271 022/300 Write block 2138 len 36864 data: 4 3 3 3 3 4 5 5 4
21272 023/300 Write block 2584 len 12288 data: 5 6 6
21273 024/300 Write block 4712 len 40960 data: 4 2 3 4 4 5 5 5 4 4
21274 025/300 Write block 4123 len 12288 data: 2 3 3
21275 026/300 Write block 2572 len 32768 data: 3 3 3 2 3 3 4 4
21276 027/300 Read block 4238 len 20480
21277 028/300 Read block 626 len 4096
21278 029/300 Read block 1048 len 24576
21279 030/300 Read block 1932 len 28672
21280 031/300 Read block 844 len 4096
21281 032/300 Read block 4849 len 32768
21282 033/300 Read block 2054 len 20480
21283 034/300 Read block 621 len 28672
21284 035/300 Read block 2172 len 40960
21285 036/300 Read block 4755 len 4096
21286 037/300 Write block 252 len 20480 data: 5 5 5 5 5
21287 038/300 Read block 651 len 8192
21288 039/300 Write block 3968 len 20480 data: 6 4 3 3 3
21289 040/300 Read block 4224 len 12288
21290 041/300 Read block 805 len 8192
21291 042/300 Read block 3256 len 16384
21292 043/300 Flush
21293 044/300 Write block 1743 len 40960 data: 5 5 5 4 4 4 4 4 3 3
21294 045/300 Flush
21295 046/300 Write block 1549 len 20480 data: 4 3 4 3 3
21296 047/300 Write block 3037 len 36864 data: 3 2 3 3 3 4 5 4 4
21297 048/300 Write block 3671 len 40960 data: 5 6 5 5 4 4 4 4 2 2
21298 049/300 Write block 4703 len 4096 data: 3
21299 050/300 Write block 547 len 8192 data: 2 4
21300 051/300 Write block 321 len 4096 data: 2
21301 052/300 Write block 4636 len 28672 data: 6 5 5 4 4 4 2
21302 053/300 Write block 3601 len 16384 data: 3 3 3 2
21303 054/300 Read block 2778 len 12288
21304 055/300 Read block 3230 len 20480
21305 056/300 Write block 2106 len 8192 data: 4 5
21306 057/300 Read block 911 len 28672
21307 058/300 Flush
21308 059/300 Read block 4166 len 8192
21309 060/300 Flush
21310 061/300 Read block 496 len 40960
21311 062/300 Write block 2829 len 36864 data: 3 3 4 5 5 5 6 5 5
21312 063/300 Write block 2293 len 28672 data: 12 11 9 8 9 6 6
21313 064/300 Read block 2930 len 20480
21314 065/300 Write block 519 len 8192 data: 6 6
21315 066/300 Write block 4169 len 8192 data: 2 3
21316 067/300 Write block 2402 len 8192 data: 4 5
21317 068/300 Flush
21318 069/300 Read block 2213 len 8192
21319 070/300 Read block 392 len 4096
21320 071/300 Flush
21321 072/300 Read block 641 len 16384
21322 073/300 Write block 4156 len 24576 data: 2 3 3 3 3 3
21323 074/300 Read block 3880 len 20480
21324 075/300 Read block 3774 len 8192
21325 076/300 Write block 4192 len 28672 data: 5 5 7 6 7 7 6
21326 077/300 Write block 4783 len 20480 data: 3 4 4 3 4
21327 078/300 Read block 1488 len 12288
21328 079/300 Read block 2266 len 16384
21329 080/300 Write block 3955 len 20480 data: 3 3 4 4 4
21330 081/300 Read block 4394 len 12288
21331 082/300 Read block 2222 len 36864
21332 083/300 Flush
21333 084/300 Read block 4294 len 24576
21334 085/300 Read block 3005 len 36864
21335 086/300 Write block 4982 len 16384 data: 4 4 3 4
21336 087/300 Read block 3335 len 24576
21337 088/300 Read block 1318 len 12288
21338 089/300 Read block 556 len 16384
21339 090/300 Write block 4254 len 16384 data: 4 4 4 4
21340 091/300 Flush
21341 092/300 Write block 374 len 24576 data: 3 3 3 3 3 4
21342 093/300 Write block 1726 len 8192 data: 4 4
21343 094/300 Read block 3617 len 24576
21344 095/300 Read block 4678 len 4096
21345 096/300 Read block 1930 len 16384
21346 097/300 Read block 2627 len 24576
21347 098/300 Read block 1892 len 24576
21348 099/300 Write block 3750 len 36864 data: 6 6 7 7 7 6 6 4 4
21349 100/300 Write block 1551 len 24576 data: 5 4 4 3 2 2
21350 101/300 Read block 4956 len 8192
21351 102/300 Read block 4809 len 20480
21352 103/300 Read block 717 len 40960
21353 104/300 Read block 3544 len 20480
21354 105/300 Write block 3458 len 8192 data: 5 7
21355 106/300 Write block 752 len 36864 data: 4 5 6 6 6 5 5 5 3
21356 107/300 Read block 3317 len 32768
21357 108/300 Flush
21358 109/300 Read block 162 len 4096
21359 110/300 Flush
21360 111/300 Read block 2519 len 20480
21361 112/300 Read block 346 len 24576
21362 113/300 Write block 2874 len 16384 data: 4 5 5 5
21363 114/300 Read block 232 len 12288
21364 115/300 Read block 1265 len 40960
21365 116/300 Flush
21366 117/300 Read block 1209 len 4096
21367 118/300 Read block 26 len 16384
21368 119/300 Read block 2004 len 32768
21369 120/300 Write block 4721 len 12288 data: 5 4 3
21370 121/300 Write block 3758 len 12288 data: 5 4 3
21371 122/300 Read block 2203 len 36864
21372 123/300 Flush
21373 124/300 Write block 3659 len 24576 data: 4 5 4 3 3 3
21374 125/300 Write block 1154 len 24576 data: 5 6 4 4 3 3
21375 126/300 Write block 4073 len 8192 data: 6 7
21376 127/300 Write block 3653 len 4096 data: 3
21377 128/300 Read block 991 len 8192
21378 129/300 Write block 4575 len 8192 data: 4 4
21379 130/300 Write block 292 len 40960 data: 2 2 2 3 3 3 4 6 4 3
21380 131/300 Write block 3214 len 8192 data: 4 4
21381 132/300 Flush
21382 133/300 Read block 1269 len 12288
21383 134/300 Read block 2482 len 28672
21384 135/300 Write block 776 len 40960 data: 10 10 11 10 8 8 4 3 4 4
21385 136/300 Read block 3597 len 16384
21386 137/300 Write block 875 len 40960 data: 2 2 2 2 2 3 3 4 4 4
21387 138/300 Write block 2894 len 16384 data: 3 3 3 2
21388 139/300 Write block 2211 len 12288 data: 8 8 7
21389 140/300 Read block 2066 len 28672
21390 141/300 Read block 1635 len 16384
21391 142/300 Flush
21392 143/300 Read block 4115 len 36864
21393 144/300 Read block 3168 len 16384
21394 145/300 Read block 1797 len 36864
21395 146/300 Read block 4020 len 16384
21396 147/300 Flush
21397 148/300 Read block 1422 len 8192
21398 149/300 Write block 1832 len 32768 data: 4 4 4 3 3 4 4 5
21399 150/300 Write block 737 len 16384 data: 5 5 6 5
21400 151/300 Read block 1790 len 28672
21401 152/300 Read block 2803 len 24576
21402 153/300 Write block 2034 len 4096 data: 5
21403 154/300 Write block 2150 len 8192 data: 3 3
21404 155/300 Flush
21405 156/300 Write block 4697 len 12288 data: 3 2 2
21406 157/300 Write block 1644 len 16384 data: 4 5 6 6
21407 158/300 Flush
21408 159/300 Flush
21409 160/300 Write block 3272 len 36864 data: 4 4 6 5 5 5 5 5 4
21410 161/300 Read block 3765 len 8192
21411 162/300 Write block 248 len 36864 data: 9 6 5 6 6 6 6 6 6
21412 163/300 Write block 751 len 12288 data: 3 5 6
21413 164/300 Read block 3893 len 36864
21414 165/300 Write block 2401 len 32768 data: 4 5 6 6 5 4 4 4
21415 166/300 Read block 4138 len 36864
21416 167/300 Write block 2972 len 24576 data: 3 5 6 6 5 5
21417 168/300 Read block 106 len 20480
21418 169/300 Read block 2942 len 24576
21419 170/300 Read block 2071 len 16384
21420 171/300 Write block 4498 len 12288 data: 5 4 5
21421 172/300 Write block 299 len 32768 data: 7 5 4 4 4 4 4 5
21422 173/300 Write block 881 len 20480 data: 4 5 5 5 4
21423 174/300 Read block 475 len 4096
21424 175/300 Read block 1607 len 12288
21425 176/300 Read block 2910 len 24576
21426 177/300 Read block 2413 len 12288
21427 178/300 Read block 3206 len 32768
21428 179/300 Write block 3940 len 32768 data: 2 4 4 4 4 3 6 6
21429 180/300 Read block 4674 len 8192
21430 181/300 Write block 2810 len 4096 data: 6
21431 182/300 Read block 2440 len 24576
21432 183/300 Write block 2787 len 28672 data: 6 6 5 5 6 8 5
21433 184/300 Read block 1374 len 24576
21434 185/300 Write block 1872 len 28672 data: 8 8 7 6 7 8 7
21435 186/300 Flush
21436 187/300 Write block 2218 len 24576 data: 6 6 5 5 6 7
21437 188/300 Flush
21438 189/300 Read block 4247 len 28672
21439 190/300 Read block 112 len 20480
21440 191/300 Read block 428 len 36864
21441 192/300 Read block 3304 len 40960
21442 193/300 Read block 3610 len 20480
21443 194/300 Write block 3942 len 8192 data: 5 5
21444 195/300 Read block 4307 len 8192
21445 196/300 Write block 649 len 40960 data: 2 2 3 3 3 3 5 6 8 8
21446 197/300 Write block 3806 len 20480 data: 4 3 4 4 3
21447 198/300 Write block 1071 len 28672 data: 5 5 5 5 5 6 6
21448 199/300 Read block 307 len 40960
21449 200/300 Write block 1691 len 28672 data: 5 5 6 5 5 4 3
21450 201/300 Write block 3709 len 4096 data: 4
21451 202/300 Flush
21452 203/300 Flush
21453 204/300 Read block 2600 len 8192
21454 205/300 Write block 3420 len 32768 data: 4 5 6 5 5 6 5 5
21455 206/300 Read block 4585 len 36864
21456 207/300 Read block 4988 len 4096
21457 208/300 Write block 645 len 12288 data: 6 5 5
21458 209/300 Read block 4549 len 20480
21459 210/300 Flush
21460 211/300 Write block 4093 len 4096 data: 4
21461 212/300 Write block 4309 len 12288 data: 7 9 10
21462 213/300 Write block 3616 len 4096 data: 3
21463 214/300 Read block 4562 len 40960
21464 215/300 Read block 4122 len 28672
21465 216/300 Flush
21466 217/300 Flush
21467 218/300 Write block 4255 len 32768 data: 5 5 5 4 4 4 4 4
21468 219/300 Write block 291 len 28672 data: 3 3 3 3 4 4 4
21469 220/300 Flush
21470 221/300 Read block 849 len 40960
21471 222/300 Write block 211 len 32768 data: 4 2 2 2 2 4 4 4
21472 223/300 Read block 3508 len 16384
21473 224/300 Write block 1251 len 20480 data: 4 3 3 4 4
21474 225/300 Read block 643 len 12288
21475 226/300 Flush
21476 227/300 Read block 1018 len 24576
21477 228/300 Read block 2587 len 36864
21478 229/300 Write block 1909 len 24576 data: 6 5 3 2 3 4
21479 230/300 Write block 1367 len 32768 data: 5 4 4 4 2 4 4 3
21480 231/300 Write block 1347 len 4096 data: 3
21481 232/300 Write block 4798 len 20480 data: 2 2 3 3 3
21482 233/300 Read block 2452 len 32768
21483 234/300 Read block 4798 len 20480
21484 235/300 Read block 653 len 4096
21485 236/300 Flush
21486 237/300 Read block 4825 len 28672
21487 238/300 Write block 4579 len 28672 data: 5 5 4 5 5 4 4
21488 239/300 Write block 1024 len 20480 data: 6 5 5 5 4
21489 240/300 Write block 3124 len 12288 data: 4 2 2
21490 241/300 Write block 81 len 24576 data: 5 4 3 3 3 4
21491 242/300 Read block 4165 len 40960
21492 243/300 Read block 592 len 8192
21493 244/300 Write block 292 len 16384 data: 4 4 4 5
21494 245/300 Write block 4045 len 36864 data: 2 3 3 3 3 5 7 8 8
21495 246/300 Write block 1152 len 36864 data: 5 6 6 7 5 5 4 4 2
21496 247/300 Read block 169 len 24576
21497 248/300 Read block 1947 len 8192
21498 249/300 Write block 1771 len 20480 data: 4 5 6 4 4
21499 250/300 Flush
21500 251/300 Write block 2656 len 12288 data: 4 5 6
21501 252/300 Write block 2820 len 32768 data: 5 5 5 3 3 3 3 3
21502 253/300 Read block 84 len 28672
21503 254/300 Write block 4610 len 24576 data: 3 3 3 3 3 3
21504 255/300 Write block 930 len 28672 data: 6 6 6 6 4 4 4
21505 256/300 Write block 1084 len 8192 data: 3 4
21506 257/300 Write block 3327 len 24576 data: 4 2 3 3 3 3
21507 258/300 Write block 446 len 40960 data: 3 4 4 4 3 4 4 3 4 7
21508 259/300 Write block 4075 len 16384 data: 7 7 7 6
21509 260/300 Flush
21510 261/300 Read block 4765 len 12288
21511 262/300 Write block 3313 len 20480 data: 2 3 4 5 4
21512 263/300 Flush
21513 264/300 Read block 506 len 24576
21514 265/300 Write block 2816 len 36864 data: 7 6 5 5 6 6 6 4 4
21515 266/300 Read block 2206 len 8192
21516 267/300 Write block 2230 len 32768 data: 5 4 6 6 5 7 6 6
21517 268/300 Read block 457 len 24576
21518 269/300 Write block 3816 len 28672 data: 6 7 7 8 7 6 4
21519 270/300 Read block 3084 len 28672
21520 271/300 Read block 2817 len 24576
21521 272/300 Read block 1408 len 12288
21522 273/300 Write block 4109 len 28672 data: 6 7 7 6 5 5 5
21523 274/300 Write block 3909 len 24576 data: 5 4 4 6 6 6
21524 275/300 Read block 3150 len 40960
21525 276/300 Read block 3595 len 4096
21526 277/300 Read block 1765 len 20480
21527 278/300 Read block 1954 len 24576
21528 279/300 Read block 1980 len 32768
21529 280/300 Write block 1475 len 32768 data: 5 7 7 7 6 5 5 4
21530 281/300 Write block 3590 len 16384 data: 6 4 5 4
21531 282/300 Read block 2662 len 28672
21532 283/300 Read block 2023 len 16384
21533 284/300 Read block 1694 len 8192
21534 285/300 Read block 3188 len 28672
21535 286/300 Read block 3898 len 40960
21536 287/300 Read block 3279 len 20480
21537 288/300 Read block 4128 len 36864
21538 289/300 Read block 1123 len 16384
21539 290/300 Read block 1952 len 16384
21540 291/300 Read block 4155 len 32768
21541 292/300 Read block 2920 len 12288
21542 293/300 Read block 4211 len 20480
21543 294/300 Write block 2878 len 40960 data: 5 4 4 3 3 3 3 4 5 5
21544 295/300 Write block 0 len 36864 data: 2 2 2 3 5 4 4 4 5
21545 296/300 Write block 2195 len 24576 data: 3 4 4 3 4 5
21546 297/300 Read block 3128 len 20480
21547 298/300 Flush
21548 299/300 Write block 1891 len 12288 data: 3 3 3
21549 300/300 Flush
21550 301/300 Write block 3092 len 16384 data: 8 8 10 9
21551 Main task has work to do, go find it
21552 got dsc Start(2)
21553 start 2
21554 Nov 01 16:55:02.224 INFO request completed, latency_us: 249, response_code: 204, uri: /start/cid/2, method: POST, req_id: 6109e35c-1eb1-4649-8839-14260492730e, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
21555 Replay: started 2, returned:Ok(())
21556 ----------------------------------------------------------------
21557 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21558 States: Active Active Offline
21559 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21560 6743 Acked Write 3 Done Done Sent false
21561 6744 Acked Read 9 Done Done Sent false
21562 6745 Acked Read 7 Done Done Sent false
21563 6746 Acked Read 3 Done Done Sent false
21564 6747 Acked Read 9 Done Done Sent false
21565 6748 Acked Write 6 Done Done Sent false
21566 6749 Acked Write 6 Done Done Sent false
21567 6750 Acked Read 8 Done Done Sent false
21568 6751 Acked Write 7 Done Done Sent false
21569 6752 Acked Write 7 Done Done Sent false
21570 6753 Acked Read 5 Done Done Sent false
21571 6754 Acked Read 7 Done Done Sent false
21572 6755 Acked Read 7 Done Done Sent false
21573 6756 Acked Write 8 Done Done Sent false
21574 6757 Acked Read 9 Done Done Sent false
21575 6758 Acked Write 6 Done Done Sent false
21576 6759 Acked Read 1 Done Done Sent false
21577 6760 Acked Write 10 Done Done Sent false
21578 6761 Acked Flush 0 Done Done Sent false
21579 6762 Acked Read 8 Done Done Sent false
21580 6763 Acked Write 10 Done Done Sent false
21581 6764 Acked Write 9 Done Done Sent false
21582 6765 Acked Write 3 Done Done Sent false
21583 6766 Acked Write 10 Done Done Sent false
21584 6767 Acked Write 3 Done Done Sent false
21585 6768 Acked Write 8 Done Done Sent false
21586 6769 Acked Read 5 Done Done Sent false
21587 6770 Acked Read 1 Done Done Sent false
21588 6771 Acked Read 6 Done Done Sent false
21589 6772 Acked Read 7 Done Done Sent false
21590 6773 Acked Read 1 Done Done Sent false
21591 6774 Acked Read 8 Done Done Sent false
21592 6775 Acked Read 5 Done Done Sent false
21593 6776 Acked Read 7 Done Done Sent false
21594 6777 Acked Read 10 Done Done Sent false
21595 6778 Acked Read 1 Done Done Sent false
21596 6779 Acked Write 5 Done Done Sent false
21597 6780 Acked Read 2 Done Done Sent false
21598 6781 Acked Write 5 Done Done Sent false
21599 6782 Acked Read 3 Done Done Sent false
21600 6783 Acked Read 2 Done Done Sent false
21601 6784 Acked Read 4 Done Done Sent false
21602 6785 Acked Flush 0 Done Done Sent false
21603 6786 Acked Write 10 Done Done Sent false
21604 6787 Acked Flush 0 Done Done Sent false
21605 6788 Acked Write 5 Done Done Sent false
21606 6789 Acked Write 9 Done Done Sent false
21607 6790 Acked Write 10 Done Done Sent false
21608 6791 Acked Write 1 Done Done Sent false
21609 6792 Acked Write 2 Done Done Sent false
21610 6793 Acked Write 1 Done Done Sent false
21611 6794 Acked Write 7 Done Done Sent false
21612 6795 Acked Write 4 Done Done Sent false
21613 6796 Acked Read 3 Done Done Sent false
21614 6797 Acked Read 5 Done Done Sent false
21615 6798 Acked Write 2 Done Done Sent false
21616 6799 Acked Read 7 Done Done Sent false
21617 6800 Acked Flush 0 Done Done Sent false
21618 6801 Acked Read 2 Done Done Sent false
21619 6802 Acked Flush 0 Done Done Sent false
21620 6803 Acked Read 10 Done Done Sent false
21621 6804 Acked Write 9 Done Done Sent false
21622 6805 Acked Write 7 Done Done Sent false
21623 6806 Acked Read 5 Done Done Sent false
21624 6807 Acked Write 2 Done Done Sent false
21625 6808 Acked Write 2 Done Done Sent false
21626 6809 Acked Write 2 Done Done Sent false
21627 6810 Acked Flush 0 Done Done Sent false
21628 6811 Acked Read 2 Done Done Sent false
21629 6812 Acked Read 1 Done Done Sent false
21630 6813 Acked Flush 0 Done Done Sent false
21631 6814 Acked Read 4 Done Done Sent false
21632 6815 Acked Write 6 Done Done Sent false
21633 6816 Acked Read 5 Done Done Sent false
21634 6817 Acked Read 2 Done Done Sent false
21635 6818 Acked Write 7 Done Done Sent false
21636 6819 Acked Write 5 Done Done Sent false
21637 6820 Acked Read 3 Done Done Sent false
21638 6821 Acked Read 4 Done Done Sent false
21639 6822 Acked Write 5 Done Done Sent false
21640 6823 Acked Read 3 Done Done Sent false
21641 6824 Acked Read 9 Done Done Sent false
21642 6825 Acked Flush 0 Done Done Sent false
21643 6826 Acked Read 6 Done Done Sent false
21644 6827 Acked Read 9 Done Done Sent false
21645 6828 Acked Write 4 Done Done Sent false
21646 6829 Acked Read 6 Done Done Sent false
21647 6830 Acked Read 3 Done Done Sent false
21648 6831 Acked Read 4 Done Done Sent false
21649 6832 Acked Write 4 Done Done Sent false
21650 6833 Acked Flush 0 Done Done Sent false
21651 6834 Acked Write 6 Done Done Sent false
21652 6835 Acked Write 2 Done Done Sent false
21653 6836 Acked Read 6 Done Done Sent false
21654 6837 Acked Read 1 Done Done Sent false
21655 6838 Acked Read 4 Done Done Sent false
21656 6839 Acked Read 6 Done Done Sent false
21657 6840 Acked Read 6 Done Done Sent false
21658 6841 Acked Write 9 Done Done Sent false
21659 6842 Acked Write 6 Done Done Sent false
21660 6843 Acked Read 2 Done Done Sent false
21661 6844 Acked Read 5 Done Done Sent false
21662 6845 Acked Read 10 Done Done Sent false
21663 6846 Acked Read 5 Done Done Sent false
21664 6847 Acked Write 2 Done Done Sent false
21665 6848 Acked Write 9 Done Done Sent false
21666 6849 Acked Read 8 Done Done Sent false
21667 6850 Acked Flush 0 Done Done Sent false
21668 6851 Acked Read 1 Done Done Sent false
21669 6852 Acked Flush 0 Done Done Sent false
21670 6853 Acked Read 5 Done Done Sent false
21671 6854 Acked Read 6 Done Done Sent false
21672 6855 Acked Write 4 Done Done Sent false
21673 6856 Acked Read 3 Done Done Sent false
21674 6857 Acked Read 10 Done Done Sent false
21675 6858 Acked Flush 0 Done Done Sent false
21676 6859 Acked Read 1 Done Done Sent false
21677 6860 Acked Read 4 Done Done Sent false
21678 6861 Acked Read 8 Done Done Sent false
21679 6862 Acked Write 3 Done Done Sent false
21680 6863 Acked Write 3 Done Done Sent false
21681 6864 Acked Read 9 Done Done Sent false
21682 6865 Acked Flush 0 Done Done Sent false
21683 6866 Acked Write 6 Done Done Sent false
21684 6867 Acked Write 6 Done Done Sent false
21685 6868 Acked Write 2 Done Done Sent false
21686 6869 Acked Write 1 Done Done Sent false
21687 6870 Acked Read 2 Done Done Sent false
21688 6871 Acked Write 2 Done Done Sent false
21689 6872 Acked Write 10 Done Done Sent false
21690 6873 Acked Write 2 Done Done Sent false
21691 6874 Acked Flush 0 Done Done Sent false
21692 6875 Acked Read 3 Done Done Sent false
21693 6876 Acked Read 7 Done Done Sent false
21694 6877 Acked Write 10 Done Done Sent false
21695 6878 Acked Read 4 Done Done Sent false
21696 6879 Acked Write 10 Done Done Sent false
21697 6880 Acked Write 4 Done Done Sent false
21698 6881 Acked Write 3 Done Done Sent false
21699 6882 Acked Read 7 Done Done Sent false
21700 6883 Acked Read 4 Done Done Sent false
21701 6884 Acked Flush 0 Done Done Sent false
21702 6885 Acked Read 9 Done Done Sent false
21703 6886 Acked Read 4 Done Done Sent false
21704 6887 Acked Read 9 Done Done Sent false
21705 6888 Acked Read 4 Done Done Sent false
21706 6889 Acked Flush 0 Done Done Sent false
21707 6890 Acked Read 2 Done Done Sent false
21708 6891 Acked Write 8 Done Done Sent false
21709 6892 Acked Write 4 Done Done Sent false
21710 6893 Acked Read 7 Done Done Sent false
21711 6894 Acked Read 6 Done Done Sent false
21712 6895 Acked Write 1 Done Done Sent false
21713 6896 Acked Write 2 Done Done Sent false
21714 6897 Acked Flush 0 Done Done Sent false
21715 6898 Acked Write 3 Done Done Sent false
21716 6899 Acked Write 4 Done Done Sent false
21717 6900 Acked Flush 0 Done Done Sent false
21718 6901 Acked Flush 0 Done Done Sent false
21719 6902 Acked Write 9 Done Done Sent false
21720 6903 Acked Read 2 Done Done Sent false
21721 6904 Acked Write 9 Done Done Sent false
21722 6905 Acked Write 3 Done Done Sent false
21723 6906 Acked Read 9 Done Done Sent false
21724 6907 Acked Write 8 Done Done Sent false
21725 6908 Acked Read 9 Done Done Sent false
21726 6909 Acked Write 6 Done Done Sent false
21727 6910 Acked Read 5 Done Done Sent false
21728 6911 Acked Read 6 Done Done Sent false
21729 6912 Acked Read 4 Done Done Sent false
21730 6913 Acked Write 3 Done Done Sent false
21731 6914 Acked Write 8 Done Done Sent false
21732 6915 Acked Write 5 Done Done Sent false
21733 6916 Acked Read 1 Done Done Sent false
21734 6917 Acked Read 3 Done Done Sent false
21735 6918 Acked Read 6 Done Done Sent false
21736 6919 Acked Read 3 Done Done Sent false
21737 6920 Acked Read 8 Done Done Sent false
21738 6921 Acked Write 8 Done Done Sent false
21739 6922 Acked Read 2 Done Done Sent false
21740 6923 Acked Write 1 Done Done Sent false
21741 6924 Acked Read 6 Done Done Sent false
21742 6925 Acked Write 7 Done Done Sent false
21743 6926 Acked Read 6 Done Done Sent false
21744 6927 Acked Write 7 Done Done Sent false
21745 6928 Acked Flush 0 Done Done Sent false
21746 6929 Acked Write 6 Done Done Sent false
21747 6930 Acked Flush 0 Done Done Sent false
21748 6931 Acked Read 7 Done Done Sent false
21749 6932 Acked Read 5 Done Done Sent false
21750 6933 Acked Read 9 Done Done Sent false
21751 6934 Acked Read 10 Done Done Sent false
21752 6935 Acked Read 5 Done Done Sent false
21753 6936 Acked Write 2 Done Done Sent false
21754 6937 Acked Read 2 Done Done Sent false
21755 6938 Acked Write 10 Done Done Sent false
21756 6939 Acked Write 5 Done Done Sent false
21757 6940 Acked Write 7 Done Done Sent false
21758 6941 Acked Read 10 Done Done Sent false
21759 6942 Acked Write 7 Done Done Sent false
21760 6943 Acked Write 1 Done Done Sent false
21761 6944 Acked Flush 0 Done Done Sent false
21762 6945 Acked Flush 0 Done Done Sent false
21763 6946 Acked Read 2 Done Done Sent false
21764 6947 Acked Write 8 Done Done Sent false
21765 6948 Acked Read 9 Done Done Sent false
21766 6949 Acked Read 1 Done Done Sent false
21767 6950 Acked Write 3 Done Done Sent false
21768 6951 Acked Read 5 Done Done Sent false
21769 6952 Acked Flush 0 Done Done Sent false
21770 6953 Acked Write 1 Done Done Sent false
21771 6954 Acked Write 3 Done Done Sent false
21772 6955 Acked Write 1 Done Done Sent false
21773 6956 Acked Read 10 Done Done Sent false
21774 6957 Acked Read 7 Done Done Sent false
21775 6958 Acked Flush 0 Done Done Sent false
21776 6959 Acked Flush 0 Done Done Sent false
21777 6960 Acked Write 8 Done Done Sent false
21778 6961 Acked Write 7 Done Done Sent false
21779 6962 Acked Flush 0 Done Done Sent false
21780 6963 Acked Read 10 Done Done Sent false
21781 6964 Acked Write 8 Done Done Sent false
21782 6965 Acked Read 4 Done Done Sent false
21783 6966 Acked Write 5 Done Done Sent false
21784 6967 Acked Read 3 Done Done Sent false
21785 6968 Acked Flush 0 Done Done Sent false
21786 6969 Acked Read 6 Done Done Sent false
21787 6970 Acked Read 9 Done Done Sent false
21788 6971 Acked Write 6 Done Done Sent false
21789 6972 Acked Write 8 Done Done Sent false
21790 6973 Acked Write 1 Done Done Sent false
21791 6974 Acked Write 5 Done Done Sent false
21792 6975 Acked Read 8 Done Done Sent false
21793 6976 Acked Read 5 Done Done Sent false
21794 6977 Acked Read 1 Done Done Sent false
21795 6978 Acked Flush 0 Done Done Sent false
21796 6979 Acked Read 7 Done Done Sent false
21797 6980 Acked Write 7 Done Done Sent false
21798 6981 Acked Write 5 Done Done Sent false
21799 6982 Acked Write 3 Done Done Sent false
21800 6983 Acked Write 6 Done Done Sent false
21801 6984 Acked Read 10 Done Done Sent false
21802 6985 Acked Read 2 Done Done Sent false
21803 6986 Acked Write 4 Done Done Sent false
21804 6987 Acked Write 9 Done Done Sent false
21805 6988 Acked Write 9 Done Done Sent false
21806 6989 Acked Read 6 Done Done Sent false
21807 6990 Acked Read 2 Done Done Sent false
21808 6991 Acked Write 5 Done Done Sent false
21809 6992 Acked Flush 0 Done Done Sent false
21810 6993 Acked Write 3 Done Done Sent false
21811 6994 Acked Write 8 Done Done Sent false
21812 6995 Acked Read 7 Done Done Sent false
21813 6996 Acked Write 6 Done Done Sent false
21814 6997 Acked Write 7 Done Done Sent false
21815 6998 Acked Write 2 Done Done Sent false
21816 6999 Acked Write 6 Done Done Sent false
21817 7000 Acked Write 10 Done Done Sent false
21818 7001 Acked Write 4 Done Done Sent false
21819 7002 Acked Flush 0 Done Done Sent false
21820 7003 Acked Read 3 Done Done Sent false
21821 7004 Acked Write 5 Done Done Sent false
21822 7005 Acked Flush 0 Done Done Sent false
21823 7006 Acked Read 6 Done Done Sent false
21824 7007 Acked Write 9 Done Done Sent false
21825 7008 Acked Read 2 Done Done Sent false
21826 7009 Acked Write 8 Done Done Sent false
21827 7010 Acked Read 6 Done Done Sent false
21828 7011 Acked Write 7 Done Done Sent false
21829 7012 Acked Read 7 Done Done Sent false
21830 7013 Acked Read 6 Done Done Sent false
21831 7014 Acked Read 3 Done Done Sent false
21832 7015 Acked Write 7 Done Done Sent false
21833 7016 Acked Write 6 Done Done Sent false
21834 7017 Acked Read 10 Done Done Sent false
21835 7018 Acked Read 1 Done Done Sent false
21836 7019 Acked Read 5 Done Done Sent false
21837 7020 Acked Read 6 Done Done Sent false
21838 7021 Acked Read 8 Done Done Sent false
21839 7022 Acked Write 8 Done Done Sent false
21840 7023 Acked Write 4 Done Done Sent false
21841 7024 Acked Read 7 Done Done Sent false
21842 7025 Acked Read 4 Done Done Sent false
21843 7026 Acked Read 2 Done Done Sent false
21844 7027 Acked Read 7 Done Done Sent false
21845 7028 Acked Read 10 Done Done Sent false
21846 7029 Acked Read 5 Done Done Sent false
21847 7030 Acked Read 9 Done Done Sent false
21848 7031 Acked Read 4 Done Done Sent false
21849 7032 Acked Read 4 Done Done Sent false
21850 7033 Acked Read 8 Done Done Sent false
21851 7034 Acked Read 3 Done Done Sent false
21852 7035 Acked Read 5 Done Done Sent false
21853 7036 Acked Write 10 Done Done Sent false
21854 7037 Acked Write 9 Done Done Sent false
21855 7038 Acked Write 6 Done Done Sent false
21856 7039 Acked Read 5 Done Done Sent false
21857 7040 Acked Flush 0 Done Done Sent false
21858 7041 Acked Write 3 Done Done Sent false
21859 7042 Acked Flush 0 Done Done Sent false
21860 7043 Acked Write 4 Done Done Sent false
21861 STATES DS:0 DS:1 DS:2 TOTAL
21862 Sent 0 0 301 301
21863 Done 301 301 0 602
21864 Skipped 0 0 0 0
21865 Error 0 0 0 0
21866 Last Flush: 7042 7042 6742
21867 Downstairs last five completed: 6742 6741 6740 6739 6738
21868 Upstairs last five completed: 7043 7042 7041 7040 7039
21869 CLIENT: Up:0 ds:301 act:2
21870 [8830] got start action
21871 [8830] I am going to restart
21872 Starting downstairs at port 8830
21873 Make output file at "/tmp/dsc/downstairs-8830.txt"
21874 [8830][2] reports Starting
21875 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1218)
21876 [8830][2] reports Running
21877 ----------------------------------------------------------------
21878 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
21879 States: Active Active Offline
21880 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21881 6743 Acked Write 3 Done Done Sent false
21882 6744 Acked Read 9 Done Done Sent false
21883 6745 Acked Read 7 Done Done Sent false
21884 6746 Acked Read 3 Done Done Sent false
21885 6747 Acked Read 9 Done Done Sent false
21886 6748 Acked Write 6 Done Done Sent false
21887 6749 Acked Write 6 Done Done Sent false
21888 6750 Acked Read 8 Done Done Sent false
21889 6751 Acked Write 7 Done Done Sent false
21890 6752 Acked Write 7 Done Done Sent false
21891 6753 Acked Read 5 Done Done Sent false
21892 6754 Acked Read 7 Done Done Sent false
21893 6755 Acked Read 7 Done Done Sent false
21894 6756 Acked Write 8 Done Done Sent false
21895 6757 Acked Read 9 Done Done Sent false
21896 6758 Acked Write 6 Done Done Sent false
21897 6759 Acked Read 1 Done Done Sent false
21898 6760 Acked Write 10 Done Done Sent false
21899 6761 Acked Flush 0 Done Done Sent false
21900 6762 Acked Read 8 Done Done Sent false
21901 6763 Acked Write 10 Done Done Sent false
21902 6764 Acked Write 9 Done Done Sent false
21903 6765 Acked Write 3 Done Done Sent false
21904 6766 Acked Write 10 Done Done Sent false
21905 6767 Acked Write 3 Done Done Sent false
21906 6768 Acked Write 8 Done Done Sent false
21907 6769 Acked Read 5 Done Done Sent false
21908 6770 Acked Read 1 Done Done Sent false
21909 6771 Acked Read 6 Done Done Sent false
21910 6772 Acked Read 7 Done Done Sent false
21911 6773 Acked Read 1 Done Done Sent false
21912 6774 Acked Read 8 Done Done Sent false
21913 6775 Acked Read 5 Done Done Sent false
21914 6776 Acked Read 7 Done Done Sent false
21915 6777 Acked Read 10 Done Done Sent false
21916 6778 Acked Read 1 Done Done Sent false
21917 6779 Acked Write 5 Done Done Sent false
21918 6780 Acked Read 2 Done Done Sent false
21919 6781 Acked Write 5 Done Done Sent false
21920 6782 Acked Read 3 Done Done Sent false
21921 6783 Acked Read 2 Done Done Sent false
21922 6784 Acked Read 4 Done Done Sent false
21923 6785 Acked Flush 0 Done Done Sent false
21924 6786 Acked Write 10 Done Done Sent false
21925 6787 Acked Flush 0 Done Done Sent false
21926 6788 Acked Write 5 Done Done Sent false
21927 6789 Acked Write 9 Done Done Sent false
21928 6790 Acked Write 10 Done Done Sent false
21929 6791 Acked Write 1 Done Done Sent false
21930 6792 Acked Write 2 Done Done Sent false
21931 6793 Acked Write 1 Done Done Sent false
21932 6794 Acked Write 7 Done Done Sent false
21933 6795 Acked Write 4 Done Done Sent false
21934 6796 Acked Read 3 Done Done Sent false
21935 6797 Acked Read 5 Done Done Sent false
21936 6798 Acked Write 2 Done Done Sent false
21937 6799 Acked Read 7 Done Done Sent false
21938 6800 Acked Flush 0 Done Done Sent false
21939 6801 Acked Read 2 Done Done Sent false
21940 6802 Acked Flush 0 Done Done Sent false
21941 6803 Acked Read 10 Done Done Sent false
21942 6804 Acked Write 9 Done Done Sent false
21943 6805 Acked Write 7 Done Done Sent false
21944 6806 Acked Read 5 Done Done Sent false
21945 6807 Acked Write 2 Done Done Sent false
21946 6808 Acked Write 2 Done Done Sent false
21947 6809 Acked Write 2 Done Done Sent false
21948 6810 Acked Flush 0 Done Done Sent false
21949 6811 Acked Read 2 Done Done Sent false
21950 6812 Acked Read 1 Done Done Sent false
21951 6813 Acked Flush 0 Done Done Sent false
21952 6814 Acked Read 4 Done Done Sent false
21953 6815 Acked Write 6 Done Done Sent false
21954 6816 Acked Read 5 Done Done Sent false
21955 6817 Acked Read 2 Done Done Sent false
21956 6818 Acked Write 7 Done Done Sent false
21957 6819 Acked Write 5 Done Done Sent false
21958 6820 Acked Read 3 Done Done Sent false
21959 6821 Acked Read 4 Done Done Sent false
21960 6822 Acked Write 5 Done Done Sent false
21961 6823 Acked Read 3 Done Done Sent false
21962 6824 Acked Read 9 Done Done Sent false
21963 6825 Acked Flush 0 Done Done Sent false
21964 6826 Acked Read 6 Done Done Sent false
21965 6827 Acked Read 9 Done Done Sent false
21966 6828 Acked Write 4 Done Done Sent false
21967 6829 Acked Read 6 Done Done Sent false
21968 6830 Acked Read 3 Done Done Sent false
21969 6831 Acked Read 4 Done Done Sent false
21970 6832 Acked Write 4 Done Done Sent false
21971 6833 Acked Flush 0 Done Done Sent false
21972 6834 Acked Write 6 Done Done Sent false
21973 6835 Acked Write 2 Done Done Sent false
21974 6836 Acked Read 6 Done Done Sent false
21975 6837 Acked Read 1 Done Done Sent false
21976 6838 Acked Read 4 Done Done Sent false
21977 6839 Acked Read 6 Done Done Sent false
21978 6840 Acked Read 6 Done Done Sent false
21979 6841 Acked Write 9 Done Done Sent false
21980 6842 Acked Write 6 Done Done Sent false
21981 6843 Acked Read 2 Done Done Sent false
21982 6844 Acked Read 5 Done Done Sent false
21983 6845 Acked Read 10 Done Done Sent false
21984 6846 Acked Read 5 Done Done Sent false
21985 6847 Acked Write 2 Done Done Sent false
21986 6848 Acked Write 9 Done Done Sent false
21987 6849 Acked Read 8 Done Done Sent false
21988 6850 Acked Flush 0 Done Done Sent false
21989 6851 Acked Read 1 Done Done Sent false
21990 6852 Acked Flush 0 Done Done Sent false
21991 6853 Acked Read 5 Done Done Sent false
21992 6854 Acked Read 6 Done Done Sent false
21993 6855 Acked Write 4 Done Done Sent false
21994 6856 Acked Read 3 Done Done Sent false
21995 6857 Acked Read 10 Done Done Sent false
21996 6858 Acked Flush 0 Done Done Sent false
21997 6859 Acked Read 1 Done Done Sent false
21998 6860 Acked Read 4 Done Done Sent false
21999 6861 Acked Read 8 Done Done Sent false
22000 6862 Acked Write 3 Done Done Sent false
22001 6863 Acked Write 3 Done Done Sent false
22002 6864 Acked Read 9 Done Done Sent false
22003 6865 Acked Flush 0 Done Done Sent false
22004 6866 Acked Write 6 Done Done Sent false
22005 6867 Acked Write 6 Done Done Sent false
22006 6868 Acked Write 2 Done Done Sent false
22007 6869 Acked Write 1 Done Done Sent false
22008 6870 Acked Read 2 Done Done Sent false
22009 6871 Acked Write 2 Done Done Sent false
22010 6872 Acked Write 10 Done Done Sent false
22011 6873 Acked Write 2 Done Done Sent false
22012 6874 Acked Flush 0 Done Done Sent false
22013 6875 Acked Read 3 Done Done Sent false
22014 6876 Acked Read 7 Done Done Sent false
22015 6877 Acked Write 10 Done Done Sent false
22016 6878 Acked Read 4 Done Done Sent false
22017 6879 Acked Write 10 Done Done Sent false
22018 6880 Acked Write 4 Done Done Sent false
22019 6881 Acked Write 3 Done Done Sent false
22020 6882 Acked Read 7 Done Done Sent false
22021 6883 Acked Read 4 Done Done Sent false
22022 6884 Acked Flush 0 Done Done Sent false
22023 6885 Acked Read 9 Done Done Sent false
22024 6886 Acked Read 4 Done Done Sent false
22025 6887 Acked Read 9 Done Done Sent false
22026 6888 Acked Read 4 Done Done Sent false
22027 6889 Acked Flush 0 Done Done Sent false
22028 6890 Acked Read 2 Done Done Sent false
22029 6891 Acked Write 8 Done Done Sent false
22030 6892 Acked Write 4 Done Done Sent false
22031 6893 Acked Read 7 Done Done Sent false
22032 6894 Acked Read 6 Done Done Sent false
22033 6895 Acked Write 1 Done Done Sent false
22034 6896 Acked Write 2 Done Done Sent false
22035 6897 Acked Flush 0 Done Done Sent false
22036 6898 Acked Write 3 Done Done Sent false
22037 6899 Acked Write 4 Done Done Sent false
22038 6900 Acked Flush 0 Done Done Sent false
22039 6901 Acked Flush 0 Done Done Sent false
22040 6902 Acked Write 9 Done Done Sent false
22041 6903 Acked Read 2 Done Done Sent false
22042 6904 Acked Write 9 Done Done Sent false
22043 6905 Acked Write 3 Done Done Sent false
22044 6906 Acked Read 9 Done Done Sent false
22045 6907 Acked Write 8 Done Done Sent false
22046 6908 Acked Read 9 Done Done Sent false
22047 6909 Acked Write 6 Done Done Sent false
22048 6910 Acked Read 5 Done Done Sent false
22049 6911 Acked Read 6 Done Done Sent false
22050 6912 Acked Read 4 Done Done Sent false
22051 6913 Acked Write 3 Done Done Sent false
22052 6914 Acked Write 8 Done Done Sent false
22053 6915 Acked Write 5 Done Done Sent false
22054 6916 Acked Read 1 Done Done Sent false
22055 6917 Acked Read 3 Done Done Sent false
22056 6918 Acked Read 6 Done Done Sent false
22057 6919 Acked Read 3 Done Done Sent false
22058 6920 Acked Read 8 Done Done Sent false
22059 6921 Acked Write 8 Done Done Sent false
22060 6922 Acked Read 2 Done Done Sent false
22061 6923 Acked Write 1 Done Done Sent false
22062 6924 Acked Read 6 Done Done Sent false
22063 6925 Acked Write 7 Done Done Sent false
22064 6926 Acked Read 6 Done Done Sent false
22065 6927 Acked Write 7 Done Done Sent false
22066 6928 Acked Flush 0 Done Done Sent false
22067 6929 Acked Write 6 Done Done Sent false
22068 6930 Acked Flush 0 Done Done Sent false
22069 6931 Acked Read 7 Done Done Sent false
22070 6932 Acked Read 5 Done Done Sent false
22071 6933 Acked Read 9 Done Done Sent false
22072 6934 Acked Read 10 Done Done Sent false
22073 6935 Acked Read 5 Done Done Sent false
22074 6936 Acked Write 2 Done Done Sent false
22075 6937 Acked Read 2 Done Done Sent false
22076 6938 Acked Write 10 Done Done Sent false
22077 6939 Acked Write 5 Done Done Sent false
22078 6940 Acked Write 7 Done Done Sent false
22079 6941 Acked Read 10 Done Done Sent false
22080 6942 Acked Write 7 Done Done Sent false
22081 6943 Acked Write 1 Done Done Sent false
22082 6944 Acked Flush 0 Done Done Sent false
22083 6945 Acked Flush 0 Done Done Sent false
22084 6946 Acked Read 2 Done Done Sent false
22085 6947 Acked Write 8 Done Done Sent false
22086 6948 Acked Read 9 Done Done Sent false
22087 6949 Acked Read 1 Done Done Sent false
22088 6950 Acked Write 3 Done Done Sent false
22089 6951 Acked Read 5 Done Done Sent false
22090 6952 Acked Flush 0 Done Done Sent false
22091 6953 Acked Write 1 Done Done Sent false
22092 6954 Acked Write 3 Done Done Sent false
22093 6955 Acked Write 1 Done Done Sent false
22094 6956 Acked Read 10 Done Done Sent false
22095 6957 Acked Read 7 Done Done Sent false
22096 6958 Acked Flush 0 Done Done Sent false
22097 6959 Acked Flush 0 Done Done Sent false
22098 6960 Acked Write 8 Done Done Sent false
22099 6961 Acked Write 7 Done Done Sent false
22100 6962 Acked Flush 0 Done Done Sent false
22101 6963 Acked Read 10 Done Done Sent false
22102 6964 Acked Write 8 Done Done Sent false
22103 6965 Acked Read 4 Done Done Sent false
22104 6966 Acked Write 5 Done Done Sent false
22105 6967 Acked Read 3 Done Done Sent false
22106 6968 Acked Flush 0 Done Done Sent false
22107 6969 Acked Read 6 Done Done Sent false
22108 6970 Acked Read 9 Done Done Sent false
22109 6971 Acked Write 6 Done Done Sent false
22110 6972 Acked Write 8 Done Done Sent false
22111 6973 Acked Write 1 Done Done Sent false
22112 6974 Acked Write 5 Done Done Sent false
22113 6975 Acked Read 8 Done Done Sent false
22114 6976 Acked Read 5 Done Done Sent false
22115 6977 Acked Read 1 Done Done Sent false
22116 6978 Acked Flush 0 Done Done Sent false
22117 6979 Acked Read 7 Done Done Sent false
22118 6980 Acked Write 7 Done Done Sent false
22119 6981 Acked Write 5 Done Done Sent false
22120 6982 Acked Write 3 Done Done Sent false
22121 6983 Acked Write 6 Done Done Sent false
22122 6984 Acked Read 10 Done Done Sent false
22123 6985 Acked Read 2 Done Done Sent false
22124 6986 Acked Write 4 Done Done Sent false
22125 6987 Acked Write 9 Done Done Sent false
22126 6988 Acked Write 9 Done Done Sent false
22127 6989 Acked Read 6 Done Done Sent false
22128 6990 Acked Read 2 Done Done Sent false
22129 6991 Acked Write 5 Done Done Sent false
22130 6992 Acked Flush 0 Done Done Sent false
22131 6993 Acked Write 3 Done Done Sent false
22132 6994 Acked Write 8 Done Done Sent false
22133 6995 Acked Read 7 Done Done Sent false
22134 6996 Acked Write 6 Done Done Sent false
22135 6997 Acked Write 7 Done Done Sent false
22136 6998 Acked Write 2 Done Done Sent false
22137 6999 Acked Write 6 Done Done Sent false
22138 7000 Acked Write 10 Done Done Sent false
22139 7001 Acked Write 4 Done Done Sent false
22140 7002 Acked Flush 0 Done Done Sent false
22141 7003 Acked Read 3 Done Done Sent false
22142 7004 Acked Write 5 Done Done Sent false
22143 7005 Acked Flush 0 Done Done Sent false
22144 7006 Acked Read 6 Done Done Sent false
22145 7007 Acked Write 9 Done Done Sent false
22146 7008 Acked Read 2 Done Done Sent false
22147 7009 Acked Write 8 Done Done Sent false
22148 7010 Acked Read 6 Done Done Sent false
22149 7011 Acked Write 7 Done Done Sent false
22150 7012 Acked Read 7 Done Done Sent false
22151 7013 Acked Read 6 Done Done Sent false
22152 7014 Acked Read 3 Done Done Sent false
22153 7015 Acked Write 7 Done Done Sent false
22154 7016 Acked Write 6 Done Done Sent false
22155 7017 Acked Read 10 Done Done Sent false
22156 7018 Acked Read 1 Done Done Sent false
22157 7019 Acked Read 5 Done Done Sent false
22158 7020 Acked Read 6 Done Done Sent false
22159 7021 Acked Read 8 Done Done Sent false
22160 7022 Acked Write 8 Done Done Sent false
22161 7023 Acked Write 4 Done Done Sent false
22162 7024 Acked Read 7 Done Done Sent false
22163 7025 Acked Read 4 Done Done Sent false
22164 7026 Acked Read 2 Done Done Sent false
22165 7027 Acked Read 7 Done Done Sent false
22166 7028 Acked Read 10 Done Done Sent false
22167 7029 Acked Read 5 Done Done Sent false
22168 7030 Acked Read 9 Done Done Sent false
22169 7031 Acked Read 4 Done Done Sent false
22170 7032 Acked Read 4 Done Done Sent false
22171 7033 Acked Read 8 Done Done Sent false
22172 7034 Acked Read 3 Done Done Sent false
22173 7035 Acked Read 5 Done Done Sent false
22174 7036 Acked Write 10 Done Done Sent false
22175 7037 Acked Write 9 Done Done Sent false
22176 7038 Acked Write 6 Done Done Sent false
22177 7039 Acked Read 5 Done Done Sent false
22178 7040 Acked Flush 0 Done Done Sent false
22179 7041 Acked Write 3 Done Done Sent false
22180 7042 Acked Flush 0 Done Done Sent false
22181 7043 Acked Write 4 Done Done Sent false
22182 7044 Acked Flush 0 Done Done Sent false
22183 STATES DS:0 DS:1 DS:2 TOTAL
22184 Sent 0 0 302 302
22185 Done 302 302 0 604
22186 Skipped 0 0 0 0
22187 Error 0 0 0 0
22188 Last Flush: 7044 7044 6742
22189 Downstairs last five completed: 6742 6741 6740 6739 6738
22190 Upstairs last five completed: 7044 7043 7042 7041 7040
22191 CLIENT: Up:0 ds:302 act:2
221922024-11-01T16:55:07.929ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
221932024-11-01T16:55:07.929ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52694) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
221942024-11-01T16:55:07.930ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
221952024-11-01T16:55:07.930ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
221962024-11-01T16:55:07.930ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
221972024-11-01T16:55:07.930ZINFOcrucible: send last flush ID to this DS: 6742 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
221982024-11-01T16:55:07.931ZINFOcrucible: Replied this last flush ID: 6742 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
221992024-11-01T16:55:07.931ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
222002024-11-01T16:55:07.931ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
222012024-11-01T16:55:07.931ZINFOcrucible: [2] client re-new 302 jobs since flush 6742 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
222022024-11-01T16:55:07.931ZINFOcrucible: [2] Replayed 302 jobs since flush: 6742 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
22203 ----------------------------------------------------------------
22204 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22205 Downstairs last five completed: 7044 7043 7042 7041 7040
22206 Upstairs last five completed: 7044 7043 7042 7041 7040
22207 CLIENT: Up:0 ds:0 act:3
22208 Replay: All jobs finished
22209 Main task has work to do, go find it
22210 got dsc Stop(0)
22211 stop 0
22212 [8810] Got stop action so:false kr:false
22213 Nov 01 16:55:10.229 INFO request completed, latency_us: 366, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 7452679a-a6f6-44f2-a3ff-917c5ae68fee, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
222142024-11-01T16:55:10.229ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
22215 Nov 01 16:55:10.229 INFO request completed{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:55:10.22992449Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
22216 , latency_us: 122, response_code: 200, uri: /state/cid/0, method:{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:55:10.229945957Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
22217 GET, req_id: 0c8e7ec5-c784-4b61-bad8-3a7b08269e74, remote_addr: 127{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:55:10.229965603Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"client":"0","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
22218 .0.0.1:58552, local_addr: 127.0.0.1:9998
222192024-11-01T16:55:10.230ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
22220 [8810] Exited with: ExitStatus(unix_wait_status(9))
22221 [8810][0] reports Exit
22222 Nov 01 16:55:14.232 INFO request completed, latency_us: 232, response_code: 200, uri: /state/cid/0, method: GET, req_id: 296b5c8a-c12f-417c-a4b4-1adf58f721b5, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
22223 001/300 Read block 2693 len 36864
22224 002/300 Flush
22225 003/300 Write block 382 len 20480 data: 4 4 5 5 5
22226 004/300 Write block 1870 len 24576 data: 7 6 9 9 8 7
22227 005/300 Write block 810 len 4096 data: 4
22228 006/300 Flush
22229 007/300 Write block 2757 len 40960 data: 5 5 5 5 4 4 3 4 6 6
22230 008/300 Read block 1403 len 24576
22231 009/300 Read block 2926 len 8192
22232 010/300 Read block 2864 len 40960
22233 011/300 Flush
22234 012/300 Read block 2366 len 28672
22235 013/300 Read block 2165 len 16384
22236 014/300 Write block 660 len 12288 data: 8 8 6
22237 015/300 Write block 4129 len 4096 data: 3
22238 016/300 Write block 2690 len 20480 data: 2 2 2 2 2
22239 017/300 Read block 3325 len 8192
22240 018/300 Write block 3135 len 24576 data: 8 8 6 6 5 5
22241 019/300 Flush
22242 020/300 Read block 1894 len 28672
22243 021/300 Write block 2970 len 4096 data: 3
22244 022/300 Read block 3348 len 24576
22245 023/300 Flush
22246 024/300 Write block 4510 len 36864 data: 6 6 7 7 6 6 7 8 7
22247 025/300 Read block 3210 len 12288
22248 026/300 Read block 3348 len 36864
22249 027/300 Flush
22250 028/300 Read block 776 len 8192
22251 029/300 Read block 3639 len 28672
22252 030/300 Read block 2652 len 4096
22253 031/300 Read block 3867 len 40960
22254 032/300 Read block 1120 len 12288
22255 033/300 Write block 683 len 32768 data: 4 4 3 3 5 3 3 4
22256 034/300 Flush
22257 035/300 Read block 812 len 20480
22258 036/300 Write block 2311 len 36864 data: 3 4 4 4 4 4 4 4 3
22259 037/300 Flush
22260 038/300 Read block 4518 len 16384
22261 039/300 Read block 640 len 40960
22262 040/300 Read block 1242 len 40960
22263 041/300 Write block 407 len 24576 data: 3 3 5 5 5 4
22264 042/300 Flush
22265 043/300 Write block 3054 len 12288 data: 2 2 2
22266 044/300 Flush
22267 045/300 Write block 1089 len 32768 data: 8 7 8 7 6 5 5 4
22268 046/300 Read block 3644 len 20480
22269 047/300 Write block 2560 len 24576 data: 5 6 6 6 8 8
22270 048/300 Write block 1705 len 32768 data: 2 2 2 2 3 3 3 3
22271 049/300 Read block 784 len 12288
22272 050/300 Read block 1930 len 40960
22273 051/300 Write block 3588 len 20480 data: 7 6 7 5 6
22274 052/300 Write block 4600 len 32768 data: 4 3 2 2 3 3 3 3
22275 053/300 Read block 2093 len 28672
22276 054/300 Read block 4328 len 36864
22277 055/300 Write block 1818 len 36864 data: 3 4 4 5 5 5 5 5 4
22278 056/300 Read block 200 len 32768
22279 057/300 Write block 4780 len 20480 data: 3 3 3 4 5
22280 058/300 Read block 1856 len 24576
22281 059/300 Read block 903 len 4096
22282 060/300 Read block 4640 len 28672
22283 061/300 Read block 538 len 24576
22284 062/300 Read block 1943 len 28672
22285 063/300 Write block 2086 len 16384 data: 6 6 5 6
22286 064/300 Write block 2456 len 28672 data: 5 5 4 3 2 3 4
22287 065/300 Read block 4046 len 32768
22288 066/300 Read block 2167 len 36864
22289 067/300 Flush
22290 068/300 Flush
22291 069/300 Write block 1902 len 40960 data: 4 5 6 7 7 7 6 7 6 4
22292 070/300 Read block 1346 len 20480
22293 071/300 Write block 4852 len 36864 data: 5 4 5 5 4 4 5 5 3
22294 072/300 Write block 1218 len 16384 data: 7 5 5 6
22295 073/300 Read block 1651 len 20480
22296 074/300 Read block 1549 len 28672
22297 075/300 Read block 2208 len 4096
22298 076/300 Write block 1442 len 20480 data: 3 4 4 5 5
22299 077/300 Read block 1078 len 8192
22300 078/300 Read block 4931 len 16384
22301 079/300 Write block 2506 len 40960 data: 4 4 2 2 3 3 3 3 4 4
22302 080/300 Write block 4683 len 12288 data: 5 6 7
22303 081/300 Write block 3195 len 28672 data: 4 4 5 4 6 6 8
22304 082/300 Read block 4512 len 28672
22305 083/300 Write block 4052 len 20480 data: 9 9 7 6 6
22306 084/300 Write block 3935 len 4096 data: 5
22307 085/300 Write block 2377 len 28672 data: 4 3 3 3 3 3 4
22308 086/300 Read block 966 len 8192
22309 087/300 Read block 3850 len 8192
22310 088/300 Write block 2774 len 36864 data: 5 6 5 6 6 4 4 3 3
22311 089/300 Read block 771 len 12288
22312 090/300 Flush
22313 091/300 Flush
22314 092/300 Read block 4262 len 8192
22315 093/300 Read block 3018 len 36864
22316 094/300 Write block 4486 len 28672 data: 5 5 6 5 4 5 4
22317 095/300 Write block 526 len 8192 data: 5 6
22318 096/300 Write block 3290 len 4096 data: 7
22319 097/300 Write block 2028 len 32768 data: 3 3 4 4 4 5 6 5
22320 098/300 Write block 3100 len 4096 data: 6
22321 099/300 Read block 3298 len 28672
22322 100/300 Flush
22323 101/300 Read block 4569 len 16384
22324 102/300 Write block 3191 len 12288 data: 4 3 3
22325 103/300 Read block 1109 len 16384
22326 104/300 Write block 531 len 12288 data: 8 7 7
22327 105/300 Read block 2118 len 4096
22328 106/300 Write block 1664 len 4096 data: 6
22329 107/300 Read block 1893 len 28672
22330 108/300 Read block 758 len 20480
22331 109/300 Write block 2601 len 12288 data: 4 4 4
22332 110/300 Read block 1810 len 8192
22333 111/300 Read block 165 len 20480
22334 112/300 Write block 4096 len 40960 data: 2 2 3 3 2 3 4 5 6 5
22335 113/300 Write block 4041 len 40960 data: 3 3 3 3 3 4 4 4 4 6
22336 114/300 Read block 3898 len 16384
22337 115/300 Read block 2273 len 20480
22338 116/300 Flush
22339 117/300 Read block 4532 len 24576
22340 118/300 Read block 3694 len 40960
22341 119/300 Read block 692 len 24576
22342 120/300 Read block 281 len 20480
22343 121/300 Write block 3348 len 40960 data: 3 4 5 4 4 4 4 4 3 3
22344 122/300 Write block 2639 len 32768 data: 6 5 6 6 6 6 6 6
22345 123/300 Read block 405 len 24576
22346 124/300 Write block 581 len 32768 data: 3 3 3 4 4 4 4 4
22347 125/300 Read block 1215 len 4096
22348 126/300 Read block 2667 len 12288
22349 127/300 Read block 3056 len 8192
22350 128/300 Write block 4525 len 40960 data: 7 6 8 7 6 7 6 6 7 7
22351 129/300 Write block 1135 len 20480 data: 3 4 5 6 5
22352 130/300 Write block 2703 len 28672 data: 3 3 4 4 3 3 3
22353 131/300 Write block 2432 len 16384 data: 3 3 4 4
22354 132/300 Read block 3126 len 40960
22355 133/300 Read block 4144 len 28672
22356 134/300 Write block 4596 len 32768 data: 4 4 4 4 5 4 3 3
22357 135/300 Read block 4976 len 36864
22358 136/300 Read block 3701 len 8192
22359 137/300 Write block 1956 len 32768 data: 3 3 4 4 4 5 6 6
22360 138/300 Write block 4484 len 40960 data: 4 5 6 6 7 6 5 6 5 4
22361 139/300 Write block 1856 len 16384 data: 3 3 3 4
22362 140/300 Flush
22363 141/300 Write block 3599 len 36864 data: 4 4 4 4 4 3 2 2 2
22364 142/300 Write block 3082 len 20480 data: 5 5 5 6 5
22365 143/300 Read block 3248 len 4096
22366 144/300 Read block 4605 len 40960
22367 145/300 Read block 1824 len 32768
22368 146/300 Read block 981 len 24576
22369 147/300 Write block 3967 len 12288 data: 6 7 5
22370 148/300 Write block 542 len 8192 data: 4 3
22371 149/300 Write block 2177 len 36864 data: 4 3 3 4 6 6 5 5 5
22372 150/300 Read block 217 len 12288
22373 151/300 Read block 3230 len 32768
22374 152/300 Write block 4945 len 32768 data: 6 5 6 5 6 6 6 6
22375 153/300 Flush
22376 154/300 Write block 2901 len 16384 data: 2 2 3 3
22377 155/300 Read block 4292 len 36864
22378 156/300 Read block 1493 len 4096
22379 157/300 Read block 664 len 8192
22380 158/300 Write block 3969 len 32768 data: 6 4 4 4 4 4 4 5
22381 159/300 Read block 340 len 40960
22382 160/300 Write block 4401 len 16384 data: 2 2 3 4
22383 161/300 Read block 4834 len 12288
22384 162/300 Write block 3339 len 12288 data: 5 5 4
22385 163/300 Read block 1184 len 36864
22386 164/300 Read block 2128 len 16384
22387 165/300 Write block 858 len 4096 data: 4
22388 166/300 Write block 1186 len 12288 data: 2 2 5
22389 167/300 Read block 2392 len 36864
22390 168/300 Read block 2967 len 8192
22391 169/300 Write block 4969 len 4096 data: 6
22392 170/300 Read block 1860 len 20480
22393 171/300 Read block 4164 len 28672
22394 172/300 Flush
22395 173/300 Write block 1573 len 40960 data: 5 4 5 5 5 5 3 2 2 2
22396 174/300 Read block 730 len 36864
22397 175/300 Flush
22398 176/300 Write block 1118 len 40960 data: 5 6 6 5 3 4 3 4 4 4
22399 177/300 Read block 301 len 32768
22400 178/300 Write block 3440 len 4096 data: 4
22401 179/300 Write block 4242 len 4096 data: 2
22402 180/300 Read block 2334 len 24576
22403 181/300 Flush
22404 182/300 Read block 1893 len 16384
22405 183/300 Flush
22406 184/300 Read block 3036 len 16384
22407 185/300 Flush
22408 186/300 Read block 3494 len 4096
22409 187/300 Read block 2535 len 32768
22410 188/300 Read block 34 len 32768
22411 189/300 Read block 3285 len 28672
22412 190/300 Read block 1906 len 12288
22413 191/300 Write block 4571 len 20480 data: 3 4 5 4 5
22414 192/300 Flush
22415 193/300 Write block 4018 len 16384 data: 3 3 3 3
22416 194/300 Write block 2788 len 28672 data: 7 6 6 7 9 6 5
22417 195/300 Write block 1494 len 36864 data: 4 4 5 6 5 6 5 5 5
22418 196/300 Write block 167 len 32768 data: 4 5 5 5 5 5 5 5
22419 197/300 Read block 3877 len 8192
22420 198/300 Flush
22421 199/300 Read block 3479 len 24576
22422 200/300 Read block 4040 len 40960
22423 201/300 Write block 822 len 20480 data: 2 3 4 5 5
22424 202/300 Write block 2112 len 12288 data: 4 4 4
22425 203/300 Write block 1276 len 4096 data: 4
22426 204/300 Read block 4483 len 36864
22427 205/300 Read block 4884 len 32768
22428 206/300 Read block 3315 len 24576
22429 207/300 Read block 578 len 40960
22430 208/300 Read block 1916 len 8192
22431 209/300 Read block 2251 len 12288
22432 210/300 Write block 2862 len 4096 data: 5
22433 211/300 Read block 875 len 32768
22434 212/300 Write block 2339 len 36864 data: 3 4 4 5 6 6 6 4 4
22435 213/300 Flush
22436 214/300 Write block 2591 len 36864 data: 3 2 2 3 3 3 4 4 4
22437 215/300 Write block 3120 len 32768 data: 5 5 7 5 5 3 3 2
22438 216/300 Read block 40 len 24576
22439 217/300 Flush
22440 218/300 Write block 1235 len 8192 data: 4 4
22441 219/300 Read block 3175 len 32768
22442 220/300 Read block 4771 len 24576
22443 221/300 Write block 3697 len 8192 data: 7 5
22444 222/300 Read block 218 len 36864
22445 223/300 Write block 3742 len 28672 data: 3 3 2 3 4 6 7
22446 224/300 Write block 3321 len 8192 data: 4 5
22447 225/300 Read block 3154 len 36864
22448 226/300 Write block 269 len 28672 data: 5 5 6 5 4 4 4
22449 227/300 Flush
22450 228/300 Read block 3635 len 24576
22451 229/300 Flush
22452 230/300 Write block 3439 len 24576 data: 4 5 7 6 6 7
22453 231/300 Read block 4795 len 4096
22454 232/300 Read block 4554 len 40960
22455 233/300 Write block 180 len 20480 data: 4 5 7 5 4
22456 234/300 Write block 2084 len 24576 data: 4 5 7 7 6 7
22457 235/300 Read block 1334 len 32768
22458 236/300 Flush
22459 237/300 Write block 2931 len 32768 data: 7 7 5 4 5 4 4 4
22460 238/300 Read block 1637 len 4096
22461 239/300 Flush
22462 240/300 Read block 2083 len 36864
22463 241/300 Write block 3081 len 8192 data: 5 6
22464 242/300 Write block 1436 len 12288 data: 9 8 7
22465 243/300 Read block 3312 len 16384
22466 244/300 Write block 962 len 36864 data: 2 2 2 2 3 4 5 6 8
22467 245/300 Write block 4819 len 36864 data: 3 4 5 5 5 4 3 3 3
22468 246/300 Flush
22469 247/300 Write block 910 len 32768 data: 5 4 4 4 4 5 5 6
22470 248/300 Read block 1228 len 12288
22471 249/300 Write block 4216 len 32768 data: 4 4 4 3 5 5 5 5
22472 250/300 Write block 1940 len 24576 data: 5 5 6 7 7 8
22473 251/300 Read block 1145 len 20480
22474 252/300 Read block 3207 len 28672
22475 253/300 Read block 542 len 36864
22476 254/300 Read block 3149 len 4096
22477 255/300 Write block 4843 len 16384 data: 2 2 2 3
22478 256/300 Read block 4343 len 28672
22479 257/300 Read block 2163 len 4096
22480 258/300 Read block 4794 len 4096
22481 259/300 Write block 2062 len 40960 data: 6 5 4 3 3 2 2 2 2 4
22482 260/300 Read block 1260 len 24576
22483 261/300 Read block 3960 len 36864
22484 262/300 Read block 4692 len 24576
22485 263/300 Read block 1097 len 4096
22486 264/300 Write block 3972 len 20480 data: 5 5 5 5 6
22487 265/300 Flush
22488 266/300 Read block 901 len 20480
22489 267/300 Read block 3606 len 4096
22490 268/300 Write block 2893 len 12288 data: 4 4 4
22491 269/300 Write block 2595 len 4096 data: 4
22492 270/300 Read block 4942 len 16384
22493 271/300 Read block 4287 len 40960
22494 272/300 Read block 2316 len 24576
22495 273/300 Read block 2223 len 28672
22496 274/300 Read block 1730 len 4096
22497 275/300 Write block 4348 len 24576 data: 3 5 3 3 3 3
22498 276/300 Write block 2617 len 40960 data: 6 5 6 7 6 5 5 5 4 2
22499 277/300 Read block 2979 len 36864
22500 278/300 Write block 1772 len 28672 data: 6 7 5 5 4 4 4
22501 279/300 Read block 3040 len 20480
22502 280/300 Read block 3613 len 32768
22503 281/300 Write block 1771 len 8192 data: 5 7
22504 282/300 Write block 1225 len 4096 data: 5
22505 283/300 Read block 423 len 4096
22506 284/300 Flush
22507 285/300 Write block 1551 len 4096 data: 6
22508 286/300 Write block 1053 len 16384 data: 6 5 4 5
22509 287/300 Write block 4992 len 4096 data: 4
22510 288/300 Read block 488 len 28672
22511 289/300 Write block 2845 len 32768 data: 3 3 3 3 4 4 4 4
22512 290/300 Read block 3228 len 12288
22513 291/300 Read block 3163 len 32768
22514 292/300 Read block 3328 len 20480
22515 293/300 Write block 2259 len 16384 data: 4 3 3 3
22516 294/300 Read block 2887 len 32768
22517 295/300 Read block 582 len 20480
22518 296/300 Write block 3689 len 28672 data: 6 6 5 7 6 7 8
22519 297/300 Read block 2613 len 36864
22520 298/300 Write block 1132 len 12288 data: 4 4 3
22521 299/300 Write block 831 len 32768 data: 6 5 3 4 4 4 3 4
22522 300/300 Flush
22523 301/300 Flush
22524 Main task has work to do, go find it
22525 got dsc Start(0)
22526 start 0
22527 Nov 01 16:55:14.536 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/0, method: POST, req_id: 7551301a-51a5-478e-83b7-08be907e3937, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
22528 Replay: started 0, returned:Ok(())
22529 ----------------------------------------------------------------
22530 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22531 States: Offline Active Active
22532 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22533 7045 Acked Read 9 Sent Done Done false
22534 7046 Acked Flush 0 Sent Done Done false
22535 7047 Acked Write 5 Sent Done Done false
22536 7048 Acked Write 6 Sent Done Done false
22537 7049 Acked Write 1 Sent Done Done false
22538 7050 Acked Flush 0 Sent Done Done false
22539 7051 Acked Write 10 Sent Done Done false
22540 7052 Acked Read 6 Sent Done Done false
22541 7053 Acked Read 2 Sent Done Done false
22542 7054 Acked Read 10 Sent Done Done false
22543 7055 Acked Flush 0 Sent Done Done false
22544 7056 Acked Read 7 Sent Done Done false
22545 7057 Acked Read 4 Sent Done Done false
22546 7058 Acked Write 3 Sent Done Done false
22547 7059 Acked Write 1 Sent Done Done false
22548 7060 Acked Write 5 Sent Done Done false
22549 7061 Acked Read 2 Sent Done Done false
22550 7062 Acked Write 6 Sent Done Done false
22551 7063 Acked Flush 0 Sent Done Done false
22552 7064 Acked Read 7 Sent Done Done false
22553 7065 Acked Write 1 Sent Done Done false
22554 7066 Acked Read 6 Sent Done Done false
22555 7067 Acked Flush 0 Sent Done Done false
22556 7068 Acked Write 9 Sent Done Done false
22557 7069 Acked Read 3 Sent Done Done false
22558 7070 Acked Read 9 Sent Done Done false
22559 7071 Acked Flush 0 Sent Done Done false
22560 7072 Acked Read 2 Sent Done Done false
22561 7073 Acked Read 7 Sent Done Done false
22562 7074 Acked Read 1 Sent Done Done false
22563 7075 Acked Read 10 Sent Done Done false
22564 7076 Acked Read 3 Sent Done Done false
22565 7077 Acked Write 8 Sent Done Done false
22566 7078 Acked Flush 0 Sent Done Done false
22567 7079 Acked Read 5 Sent Done Done false
22568 7080 Acked Write 9 Sent Done Done false
22569 7081 Acked Flush 0 Sent Done Done false
22570 7082 Acked Read 4 Sent Done Done false
22571 7083 Acked Read 10 Sent Done Done false
22572 7084 Acked Read 10 Sent Done Done false
22573 7085 Acked Write 6 Sent Done Done false
22574 7086 Acked Flush 0 Sent Done Done false
22575 7087 Acked Write 3 Sent Done Done false
22576 7088 Acked Flush 0 Sent Done Done false
22577 7089 Acked Write 8 Sent Done Done false
22578 7090 Acked Read 5 Sent Done Done false
22579 7091 Acked Write 6 Sent Done Done false
22580 7092 Acked Write 8 Sent Done Done false
22581 7093 Acked Read 3 Sent Done Done false
22582 7094 Acked Read 10 Sent Done Done false
22583 7095 Acked Write 5 Sent Done Done false
22584 7096 Acked Write 8 Sent Done Done false
22585 7097 Acked Read 7 Sent Done Done false
22586 7098 Acked Read 9 Sent Done Done false
22587 7099 Acked Write 9 Sent Done Done false
22588 7100 Acked Read 8 Sent Done Done false
22589 7101 Acked Write 5 Sent Done Done false
22590 7102 Acked Read 6 Sent Done Done false
22591 7103 Acked Read 1 Sent Done Done false
22592 7104 Acked Read 7 Sent Done Done false
22593 7105 Acked Read 6 Sent Done Done false
22594 7106 Acked Read 7 Sent Done Done false
22595 7107 Acked Write 4 Sent Done Done false
22596 7108 Acked Write 7 Sent Done Done false
22597 7109 Acked Read 8 Sent Done Done false
22598 7110 Acked Read 9 Sent Done Done false
22599 7111 Acked Flush 0 Sent Done Done false
22600 7112 Acked Flush 0 Sent Done Done false
22601 7113 Acked Write 10 Sent Done Done false
22602 7114 Acked Read 5 Sent Done Done false
22603 7115 Acked Write 9 Sent Done Done false
22604 7116 Acked Write 4 Sent Done Done false
22605 7117 Acked Read 5 Sent Done Done false
22606 7118 Acked Read 7 Sent Done Done false
22607 7119 Acked Read 1 Sent Done Done false
22608 7120 Acked Write 5 Sent Done Done false
22609 7121 Acked Read 2 Sent Done Done false
22610 7122 Acked Read 4 Sent Done Done false
22611 7123 Acked Write 10 Sent Done Done false
22612 7124 Acked Write 3 Sent Done Done false
22613 7125 Acked Write 7 Sent Done Done false
22614 7126 Acked Read 7 Sent Done Done false
22615 7127 Acked Write 5 Sent Done Done false
22616 7128 Acked Write 1 Sent Done Done false
22617 7129 Acked Write 7 Sent Done Done false
22618 7130 Acked Read 2 Sent Done Done false
22619 7131 Acked Read 2 Sent Done Done false
22620 7132 Acked Write 9 Sent Done Done false
22621 7133 Acked Read 3 Sent Done Done false
22622 7134 Acked Flush 0 Sent Done Done false
22623 7135 Acked Flush 0 Sent Done Done false
22624 7136 Acked Read 2 Sent Done Done false
22625 7137 Acked Read 9 Sent Done Done false
22626 7138 Acked Write 7 Sent Done Done false
22627 7139 Acked Write 2 Sent Done Done false
22628 7140 Acked Write 1 Sent Done Done false
22629 7141 Acked Write 8 Sent Done Done false
22630 7142 Acked Write 1 Sent Done Done false
22631 7143 Acked Read 7 Sent Done Done false
22632 7144 Acked Flush 0 Sent Done Done false
22633 7145 Acked Read 4 Sent Done Done false
22634 7146 Acked Write 3 Sent Done Done false
22635 7147 Acked Read 4 Sent Done Done false
22636 7148 Acked Write 3 Sent Done Done false
22637 7149 Acked Read 1 Sent Done Done false
22638 7150 Acked Write 1 Sent Done Done false
22639 7151 Acked Read 7 Sent Done Done false
22640 7152 Acked Read 5 Sent Done Done false
22641 7153 Acked Write 3 Sent Done Done false
22642 7154 Acked Read 2 Sent Done Done false
22643 7155 Acked Read 5 Sent Done Done false
22644 7156 Acked Write 10 Sent Done Done false
22645 7157 Acked Write 10 Sent Done Done false
22646 7158 Acked Read 4 Sent Done Done false
22647 7159 Acked Read 5 Sent Done Done false
22648 7160 Acked Flush 0 Sent Done Done false
22649 7161 Acked Read 6 Sent Done Done false
22650 7162 Acked Read 10 Sent Done Done false
22651 7163 Acked Read 6 Sent Done Done false
22652 7164 Acked Read 5 Sent Done Done false
22653 7165 Acked Write 10 Sent Done Done false
22654 7166 Acked Write 8 Sent Done Done false
22655 7167 Acked Read 6 Sent Done Done false
22656 7168 Acked Write 8 Sent Done Done false
22657 7169 Acked Read 1 Sent Done Done false
22658 7170 Acked Read 3 Sent Done Done false
22659 7171 Acked Read 2 Sent Done Done false
22660 7172 Acked Write 10 Sent Done Done false
22661 7173 Acked Write 5 Sent Done Done false
22662 7174 Acked Write 7 Sent Done Done false
22663 7175 Acked Write 4 Sent Done Done false
22664 7176 Acked Read 10 Sent Done Done false
22665 7177 Acked Read 7 Sent Done Done false
22666 7178 Acked Write 8 Sent Done Done false
22667 7179 Acked Read 9 Sent Done Done false
22668 7180 Acked Read 2 Sent Done Done false
22669 7181 Acked Write 8 Sent Done Done false
22670 7182 Acked Write 10 Sent Done Done false
22671 7183 Acked Write 4 Sent Done Done false
22672 7184 Acked Flush 0 Sent Done Done false
22673 7185 Acked Write 9 Sent Done Done false
22674 7186 Acked Write 5 Sent Done Done false
22675 7187 Acked Read 1 Sent Done Done false
22676 7188 Acked Read 10 Sent Done Done false
22677 7189 Acked Read 8 Sent Done Done false
22678 7190 Acked Read 6 Sent Done Done false
22679 7191 Acked Write 3 Sent Done Done false
22680 7192 Acked Write 2 Sent Done Done false
22681 7193 Acked Write 9 Sent Done Done false
22682 7194 Acked Read 3 Sent Done Done false
22683 7195 Acked Read 8 Sent Done Done false
22684 7196 Acked Write 8 Sent Done Done false
22685 7197 Acked Flush 0 Sent Done Done false
22686 7198 Acked Write 4 Sent Done Done false
22687 7199 Acked Read 9 Sent Done Done false
22688 7200 Acked Read 1 Sent Done Done false
22689 7201 Acked Read 2 Sent Done Done false
22690 7202 Acked Write 8 Sent Done Done false
22691 7203 Acked Read 10 Sent Done Done false
22692 7204 Acked Write 4 Sent Done Done false
22693 7205 Acked Read 3 Sent Done Done false
22694 7206 Acked Write 3 Sent Done Done false
22695 7207 Acked Read 9 Sent Done Done false
22696 7208 Acked Read 4 Sent Done Done false
22697 7209 Acked Write 1 Sent Done Done false
22698 7210 Acked Write 3 Sent Done Done false
22699 7211 Acked Read 9 Sent Done Done false
22700 7212 Acked Read 2 Sent Done Done false
22701 7213 Acked Write 1 Sent Done Done false
22702 7214 Acked Read 5 Sent Done Done false
22703 7215 Acked Read 7 Sent Done Done false
22704 7216 Acked Flush 0 Sent Done Done false
22705 7217 Acked Write 10 Sent Done Done false
22706 7218 Acked Read 9 Sent Done Done false
22707 7219 Acked Flush 0 Sent Done Done false
22708 7220 Acked Write 10 Sent Done Done false
22709 7221 Acked Read 8 Sent Done Done false
22710 7222 Acked Write 1 Sent Done Done false
22711 7223 Acked Write 1 Sent Done Done false
22712 7224 Acked Read 6 Sent Done Done false
22713 7225 Acked Flush 0 Sent Done Done false
22714 7226 Acked Read 4 Sent Done Done false
22715 7227 Acked Flush 0 Sent Done Done false
22716 7228 Acked Read 4 Sent Done Done false
22717 7229 Acked Flush 0 Sent Done Done false
22718 7230 Acked Read 1 Sent Done Done false
22719 7231 Acked Read 8 Sent Done Done false
22720 7232 Acked Read 8 Sent Done Done false
22721 7233 Acked Read 7 Sent Done Done false
22722 7234 Acked Read 3 Sent Done Done false
22723 7235 Acked Write 5 Sent Done Done false
22724 7236 Acked Flush 0 Sent Done Done false
22725 7237 Acked Write 4 Sent Done Done false
22726 7238 Acked Write 7 Sent Done Done false
22727 7239 Acked Write 9 Sent Done Done false
22728 7240 Acked Write 8 Sent Done Done false
22729 7241 Acked Read 2 Sent Done Done false
22730 7242 Acked Flush 0 Sent Done Done false
22731 7243 Acked Read 6 Sent Done Done false
22732 7244 Acked Read 10 Sent Done Done false
22733 7245 Acked Write 5 Sent Done Done false
22734 7246 Acked Write 3 Sent Done Done false
22735 7247 Acked Write 1 Sent Done Done false
22736 7248 Acked Read 9 Sent Done Done false
22737 7249 Acked Read 8 Sent Done Done false
22738 7250 Acked Read 6 Sent Done Done false
22739 7251 Acked Read 10 Sent Done Done false
22740 7252 Acked Read 2 Sent Done Done false
22741 7253 Acked Read 3 Sent Done Done false
22742 7254 Acked Write 1 Sent Done Done false
22743 7255 Acked Read 8 Sent Done Done false
22744 7256 Acked Write 9 Sent Done Done false
22745 7257 Acked Flush 0 Sent Done Done false
22746 7258 Acked Write 9 Sent Done Done false
22747 7259 Acked Write 8 Sent Done Done false
22748 7260 Acked Read 6 Sent Done Done false
22749 7261 Acked Flush 0 Sent Done Done false
22750 7262 Acked Write 2 Sent Done Done false
22751 7263 Acked Read 8 Sent Done Done false
22752 7264 Acked Read 6 Sent Done Done false
22753 7265 Acked Write 2 Sent Done Done false
22754 7266 Acked Read 9 Sent Done Done false
22755 7267 Acked Write 7 Sent Done Done false
22756 7268 Acked Write 2 Sent Done Done false
22757 7269 Acked Read 9 Sent Done Done false
22758 7270 Acked Write 7 Sent Done Done false
22759 7271 Acked Flush 0 Sent Done Done false
22760 7272 Acked Read 6 Sent Done Done false
22761 7273 Acked Flush 0 Sent Done Done false
22762 7274 Acked Write 6 Sent Done Done false
22763 7275 Acked Read 1 Sent Done Done false
22764 7276 Acked Read 10 Sent Done Done false
22765 7277 Acked Write 5 Sent Done Done false
22766 7278 Acked Write 6 Sent Done Done false
22767 7279 Acked Read 8 Sent Done Done false
22768 7280 Acked Flush 0 Sent Done Done false
22769 7281 Acked Write 8 Sent Done Done false
22770 7282 Acked Read 1 Sent Done Done false
22771 7283 Acked Flush 0 Sent Done Done false
22772 7284 Acked Read 9 Sent Done Done false
22773 7285 Acked Write 2 Sent Done Done false
22774 7286 Acked Write 3 Sent Done Done false
22775 7287 Acked Read 4 Sent Done Done false
22776 7288 Acked Write 9 Sent Done Done false
22777 7289 Acked Write 9 Sent Done Done false
22778 7290 Acked Flush 0 Sent Done Done false
22779 7291 Acked Write 8 Sent Done Done false
22780 7292 Acked Read 3 Sent Done Done false
22781 7293 Acked Write 8 Sent Done Done false
22782 7294 Acked Write 6 Sent Done Done false
22783 7295 Acked Read 5 Sent Done Done false
22784 7296 Acked Read 7 Sent Done Done false
22785 7297 Acked Read 9 Sent Done Done false
22786 7298 Acked Read 1 Sent Done Done false
22787 7299 Acked Write 4 Sent Done Done false
22788 7300 Acked Read 7 Sent Done Done false
22789 7301 Acked Read 1 Sent Done Done false
22790 7302 Acked Read 1 Sent Done Done false
22791 7303 Acked Write 10 Sent Done Done false
22792 7304 Acked Read 6 Sent Done Done false
22793 7305 Acked Read 9 Sent Done Done false
22794 7306 Acked Read 6 Sent Done Done false
22795 7307 Acked Read 1 Sent Done Done false
22796 7308 Acked Write 5 Sent Done Done false
22797 7309 Acked Flush 0 Sent Done Done false
22798 7310 Acked Read 5 Sent Done Done false
22799 7311 Acked Read 1 Sent Done Done false
22800 7312 Acked Write 3 Sent Done Done false
22801 7313 Acked Write 1 Sent Done Done false
22802 7314 Acked Read 4 Sent Done Done false
22803 7315 Acked Read 10 Sent Done Done false
22804 7316 Acked Read 6 Sent Done Done false
22805 7317 Acked Read 7 Sent Done Done false
22806 7318 Acked Read 1 Sent Done Done false
22807 7319 Acked Write 6 Sent Done Done false
22808 7320 Acked Write 10 Sent Done Done false
22809 7321 Acked Read 9 Sent Done Done false
22810 7322 Acked Write 7 Sent Done Done false
22811 7323 Acked Read 5 Sent Done Done false
22812 7324 Acked Read 8 Sent Done Done false
22813 7325 Acked Write 2 Sent Done Done false
22814 7326 Acked Write 1 Sent Done Done false
22815 7327 Acked Read 1 Sent Done Done false
22816 7328 Acked Flush 0 Sent Done Done false
22817 7329 Acked Write 1 Sent Done Done false
22818 7330 Acked Write 4 Sent Done Done false
22819 7331 Acked Write 1 Sent Done Done false
22820 7332 Acked Read 7 Sent Done Done false
22821 7333 Acked Write 8 Sent Done Done false
22822 7334 Acked Read 3 Sent Done Done false
22823 7335 Acked Read 8 Sent Done Done false
22824 7336 Acked Read 5 Sent Done Done false
22825 7337 Acked Write 4 Sent Done Done false
22826 7338 Acked Read 8 Sent Done Done false
22827 7339 Acked Read 5 Sent Done Done false
22828 7340 Acked Write 7 Sent Done Done false
22829 7341 Acked Read 9 Sent Done Done false
22830 7342 Acked Write 3 Sent Done Done false
22831 7343 Acked Write 8 Sent Done Done false
22832 7344 Acked Flush 0 Sent Done Done false
22833 7345 Acked Flush 0 Sent Done Done false
22834 STATES DS:0 DS:1 DS:2 TOTAL
22835 Sent 301 0 0 301
22836 Done 0 301 301 602
22837 Skipped 0 0 0 0
22838 Error 0 0 0 0
22839 Last Flush: 7044 7345 7345
22840 Downstairs last five completed: 7044 7043 7042 7041 7040
22841 Upstairs last five completed: 7345 7344 7343 7342 7341
22842 CLIENT: Up:0 ds:301 act:2
22843 [8810] got start action
22844 [8810] I am going to restart
22845 Starting downstairs at port 8810
22846 Make output file at "/tmp/dsc/downstairs-8810.txt"
22847 [8810][0] reports Starting
22848 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1219)
22849 [8810][0] reports Running
22850 ----------------------------------------------------------------
22851 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22852 States: Offline Active Active
22853 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22854 7045 Acked Read 9 Sent Done Done false
22855 7046 Acked Flush 0 Sent Done Done false
22856 7047 Acked Write 5 Sent Done Done false
22857 7048 Acked Write 6 Sent Done Done false
22858 7049 Acked Write 1 Sent Done Done false
22859 7050 Acked Flush 0 Sent Done Done false
22860 7051 Acked Write 10 Sent Done Done false
22861 7052 Acked Read 6 Sent Done Done false
22862 7053 Acked Read 2 Sent Done Done false
22863 7054 Acked Read 10 Sent Done Done false
22864 7055 Acked Flush 0 Sent Done Done false
22865 7056 Acked Read 7 Sent Done Done false
22866 7057 Acked Read 4 Sent Done Done false
22867 7058 Acked Write 3 Sent Done Done false
22868 7059 Acked Write 1 Sent Done Done false
22869 7060 Acked Write 5 Sent Done Done false
22870 7061 Acked Read 2 Sent Done Done false
22871 7062 Acked Write 6 Sent Done Done false
22872 7063 Acked Flush 0 Sent Done Done false
22873 7064 Acked Read 7 Sent Done Done false
22874 7065 Acked Write 1 Sent Done Done false
22875 7066 Acked Read 6 Sent Done Done false
22876 7067 Acked Flush 0 Sent Done Done false
22877 7068 Acked Write 9 Sent Done Done false
22878 7069 Acked Read 3 Sent Done Done false
22879 7070 Acked Read 9 Sent Done Done false
22880 7071 Acked Flush 0 Sent Done Done false
22881 7072 Acked Read 2 Sent Done Done false
22882 7073 Acked Read 7 Sent Done Done false
22883 7074 Acked Read 1 Sent Done Done false
22884 7075 Acked Read 10 Sent Done Done false
22885 7076 Acked Read 3 Sent Done Done false
22886 7077 Acked Write 8 Sent Done Done false
22887 7078 Acked Flush 0 Sent Done Done false
22888 7079 Acked Read 5 Sent Done Done false
22889 7080 Acked Write 9 Sent Done Done false
22890 7081 Acked Flush 0 Sent Done Done false
22891 7082 Acked Read 4 Sent Done Done false
22892 7083 Acked Read 10 Sent Done Done false
22893 7084 Acked Read 10 Sent Done Done false
22894 7085 Acked Write 6 Sent Done Done false
22895 7086 Acked Flush 0 Sent Done Done false
22896 7087 Acked Write 3 Sent Done Done false
22897 7088 Acked Flush 0 Sent Done Done false
22898 7089 Acked Write 8 Sent Done Done false
22899 7090 Acked Read 5 Sent Done Done false
22900 7091 Acked Write 6 Sent Done Done false
22901 7092 Acked Write 8 Sent Done Done false
22902 7093 Acked Read 3 Sent Done Done false
22903 7094 Acked Read 10 Sent Done Done false
22904 7095 Acked Write 5 Sent Done Done false
22905 7096 Acked Write 8 Sent Done Done false
22906 7097 Acked Read 7 Sent Done Done false
22907 7098 Acked Read 9 Sent Done Done false
22908 7099 Acked Write 9 Sent Done Done false
22909 7100 Acked Read 8 Sent Done Done false
22910 7101 Acked Write 5 Sent Done Done false
22911 7102 Acked Read 6 Sent Done Done false
22912 7103 Acked Read 1 Sent Done Done false
22913 7104 Acked Read 7 Sent Done Done false
22914 7105 Acked Read 6 Sent Done Done false
22915 7106 Acked Read 7 Sent Done Done false
22916 7107 Acked Write 4 Sent Done Done false
22917 7108 Acked Write 7 Sent Done Done false
22918 7109 Acked Read 8 Sent Done Done false
22919 7110 Acked Read 9 Sent Done Done false
22920 7111 Acked Flush 0 Sent Done Done false
22921 7112 Acked Flush 0 Sent Done Done false
22922 7113 Acked Write 10 Sent Done Done false
22923 7114 Acked Read 5 Sent Done Done false
22924 7115 Acked Write 9 Sent Done Done false
22925 7116 Acked Write 4 Sent Done Done false
22926 7117 Acked Read 5 Sent Done Done false
22927 7118 Acked Read 7 Sent Done Done false
22928 7119 Acked Read 1 Sent Done Done false
22929 7120 Acked Write 5 Sent Done Done false
22930 7121 Acked Read 2 Sent Done Done false
22931 7122 Acked Read 4 Sent Done Done false
22932 7123 Acked Write 10 Sent Done Done false
22933 7124 Acked Write 3 Sent Done Done false
22934 7125 Acked Write 7 Sent Done Done false
22935 7126 Acked Read 7 Sent Done Done false
22936 7127 Acked Write 5 Sent Done Done false
22937 7128 Acked Write 1 Sent Done Done false
22938 7129 Acked Write 7 Sent Done Done false
22939 7130 Acked Read 2 Sent Done Done false
22940 7131 Acked Read 2 Sent Done Done false
22941 7132 Acked Write 9 Sent Done Done false
22942 7133 Acked Read 3 Sent Done Done false
22943 7134 Acked Flush 0 Sent Done Done false
22944 7135 Acked Flush 0 Sent Done Done false
22945 7136 Acked Read 2 Sent Done Done false
22946 7137 Acked Read 9 Sent Done Done false
22947 7138 Acked Write 7 Sent Done Done false
22948 7139 Acked Write 2 Sent Done Done false
22949 7140 Acked Write 1 Sent Done Done false
22950 7141 Acked Write 8 Sent Done Done false
22951 7142 Acked Write 1 Sent Done Done false
22952 7143 Acked Read 7 Sent Done Done false
22953 7144 Acked Flush 0 Sent Done Done false
22954 7145 Acked Read 4 Sent Done Done false
22955 7146 Acked Write 3 Sent Done Done false
22956 7147 Acked Read 4 Sent Done Done false
22957 7148 Acked Write 3 Sent Done Done false
22958 7149 Acked Read 1 Sent Done Done false
22959 7150 Acked Write 1 Sent Done Done false
22960 7151 Acked Read 7 Sent Done Done false
22961 7152 Acked Read 5 Sent Done Done false
22962 7153 Acked Write 3 Sent Done Done false
22963 7154 Acked Read 2 Sent Done Done false
22964 7155 Acked Read 5 Sent Done Done false
22965 7156 Acked Write 10 Sent Done Done false
22966 7157 Acked Write 10 Sent Done Done false
22967 7158 Acked Read 4 Sent Done Done false
22968 7159 Acked Read 5 Sent Done Done false
22969 7160 Acked Flush 0 Sent Done Done false
22970 7161 Acked Read 6 Sent Done Done false
22971 7162 Acked Read 10 Sent Done Done false
22972 7163 Acked Read 6 Sent Done Done false
22973 7164 Acked Read 5 Sent Done Done false
22974 7165 Acked Write 10 Sent Done Done false
22975 7166 Acked Write 8 Sent Done Done false
22976 7167 Acked Read 6 Sent Done Done false
22977 7168 Acked Write 8 Sent Done Done false
22978 7169 Acked Read 1 Sent Done Done false
22979 7170 Acked Read 3 Sent Done Done false
22980 7171 Acked Read 2 Sent Done Done false
22981 7172 Acked Write 10 Sent Done Done false
22982 7173 Acked Write 5 Sent Done Done false
22983 7174 Acked Write 7 Sent Done Done false
22984 7175 Acked Write 4 Sent Done Done false
22985 7176 Acked Read 10 Sent Done Done false
22986 7177 Acked Read 7 Sent Done Done false
22987 7178 Acked Write 8 Sent Done Done false
22988 7179 Acked Read 9 Sent Done Done false
22989 7180 Acked Read 2 Sent Done Done false
22990 7181 Acked Write 8 Sent Done Done false
22991 7182 Acked Write 10 Sent Done Done false
22992 7183 Acked Write 4 Sent Done Done false
22993 7184 Acked Flush 0 Sent Done Done false
22994 7185 Acked Write 9 Sent Done Done false
22995 7186 Acked Write 5 Sent Done Done false
22996 7187 Acked Read 1 Sent Done Done false
22997 7188 Acked Read 10 Sent Done Done false
22998 7189 Acked Read 8 Sent Done Done false
22999 7190 Acked Read 6 Sent Done Done false
23000 7191 Acked Write 3 Sent Done Done false
23001 7192 Acked Write 2 Sent Done Done false
23002 7193 Acked Write 9 Sent Done Done false
23003 7194 Acked Read 3 Sent Done Done false
23004 7195 Acked Read 8 Sent Done Done false
23005 7196 Acked Write 8 Sent Done Done false
23006 7197 Acked Flush 0 Sent Done Done false
23007 7198 Acked Write 4 Sent Done Done false
23008 7199 Acked Read 9 Sent Done Done false
23009 7200 Acked Read 1 Sent Done Done false
23010 7201 Acked Read 2 Sent Done Done false
23011 7202 Acked Write 8 Sent Done Done false
23012 7203 Acked Read 10 Sent Done Done false
23013 7204 Acked Write 4 Sent Done Done false
23014 7205 Acked Read 3 Sent Done Done false
23015 7206 Acked Write 3 Sent Done Done false
23016 7207 Acked Read 9 Sent Done Done false
23017 7208 Acked Read 4 Sent Done Done false
23018 7209 Acked Write 1 Sent Done Done false
23019 7210 Acked Write 3 Sent Done Done false
23020 7211 Acked Read 9 Sent Done Done false
23021 7212 Acked Read 2 Sent Done Done false
23022 7213 Acked Write 1 Sent Done Done false
23023 7214 Acked Read 5 Sent Done Done false
23024 7215 Acked Read 7 Sent Done Done false
23025 7216 Acked Flush 0 Sent Done Done false
23026 7217 Acked Write 10 Sent Done Done false
23027 7218 Acked Read 9 Sent Done Done false
23028 7219 Acked Flush 0 Sent Done Done false
23029 7220 Acked Write 10 Sent Done Done false
23030 7221 Acked Read 8 Sent Done Done false
23031 7222 Acked Write 1 Sent Done Done false
23032 7223 Acked Write 1 Sent Done Done false
23033 7224 Acked Read 6 Sent Done Done false
23034 7225 Acked Flush 0 Sent Done Done false
23035 7226 Acked Read 4 Sent Done Done false
23036 7227 Acked Flush 0 Sent Done Done false
23037 7228 Acked Read 4 Sent Done Done false
23038 7229 Acked Flush 0 Sent Done Done false
23039 7230 Acked Read 1 Sent Done Done false
23040 7231 Acked Read 8 Sent Done Done false
23041 7232 Acked Read 8 Sent Done Done false
23042 7233 Acked Read 7 Sent Done Done false
23043 7234 Acked Read 3 Sent Done Done false
23044 7235 Acked Write 5 Sent Done Done false
23045 7236 Acked Flush 0 Sent Done Done false
23046 7237 Acked Write 4 Sent Done Done false
23047 7238 Acked Write 7 Sent Done Done false
23048 7239 Acked Write 9 Sent Done Done false
23049 7240 Acked Write 8 Sent Done Done false
23050 7241 Acked Read 2 Sent Done Done false
23051 7242 Acked Flush 0 Sent Done Done false
23052 7243 Acked Read 6 Sent Done Done false
23053 7244 Acked Read 10 Sent Done Done false
23054 7245 Acked Write 5 Sent Done Done false
23055 7246 Acked Write 3 Sent Done Done false
23056 7247 Acked Write 1 Sent Done Done false
23057 7248 Acked Read 9 Sent Done Done false
23058 7249 Acked Read 8 Sent Done Done false
23059 7250 Acked Read 6 Sent Done Done false
23060 7251 Acked Read 10 Sent Done Done false
23061 7252 Acked Read 2 Sent Done Done false
23062 7253 Acked Read 3 Sent Done Done false
23063 7254 Acked Write 1 Sent Done Done false
23064 7255 Acked Read 8 Sent Done Done false
23065 7256 Acked Write 9 Sent Done Done false
23066 7257 Acked Flush 0 Sent Done Done false
23067 7258 Acked Write 9 Sent Done Done false
23068 7259 Acked Write 8 Sent Done Done false
23069 7260 Acked Read 6 Sent Done Done false
23070 7261 Acked Flush 0 Sent Done Done false
23071 7262 Acked Write 2 Sent Done Done false
23072 7263 Acked Read 8 Sent Done Done false
23073 7264 Acked Read 6 Sent Done Done false
23074 7265 Acked Write 2 Sent Done Done false
23075 7266 Acked Read 9 Sent Done Done false
23076 7267 Acked Write 7 Sent Done Done false
23077 7268 Acked Write 2 Sent Done Done false
23078 7269 Acked Read 9 Sent Done Done false
23079 7270 Acked Write 7 Sent Done Done false
23080 7271 Acked Flush 0 Sent Done Done false
23081 7272 Acked Read 6 Sent Done Done false
23082 7273 Acked Flush 0 Sent Done Done false
23083 7274 Acked Write 6 Sent Done Done false
23084 7275 Acked Read 1 Sent Done Done false
23085 7276 Acked Read 10 Sent Done Done false
23086 7277 Acked Write 5 Sent Done Done false
23087 7278 Acked Write 6 Sent Done Done false
23088 7279 Acked Read 8 Sent Done Done false
23089 7280 Acked Flush 0 Sent Done Done false
23090 7281 Acked Write 8 Sent Done Done false
23091 7282 Acked Read 1 Sent Done Done false
23092 7283 Acked Flush 0 Sent Done Done false
23093 7284 Acked Read 9 Sent Done Done false
23094 7285 Acked Write 2 Sent Done Done false
23095 7286 Acked Write 3 Sent Done Done false
23096 7287 Acked Read 4 Sent Done Done false
23097 7288 Acked Write 9 Sent Done Done false
23098 7289 Acked Write 9 Sent Done Done false
23099 7290 Acked Flush 0 Sent Done Done false
23100 7291 Acked Write 8 Sent Done Done false
23101 7292 Acked Read 3 Sent Done Done false
23102 7293 Acked Write 8 Sent Done Done false
23103 7294 Acked Write 6 Sent Done Done false
23104 7295 Acked Read 5 Sent Done Done false
23105 7296 Acked Read 7 Sent Done Done false
23106 7297 Acked Read 9 Sent Done Done false
23107 7298 Acked Read 1 Sent Done Done false
23108 7299 Acked Write 4 Sent Done Done false
23109 7300 Acked Read 7 Sent Done Done false
23110 7301 Acked Read 1 Sent Done Done false
23111 7302 Acked Read 1 Sent Done Done false
23112 7303 Acked Write 10 Sent Done Done false
23113 7304 Acked Read 6 Sent Done Done false
23114 7305 Acked Read 9 Sent Done Done false
23115 7306 Acked Read 6 Sent Done Done false
23116 7307 Acked Read 1 Sent Done Done false
23117 7308 Acked Write 5 Sent Done Done false
23118 7309 Acked Flush 0 Sent Done Done false
23119 7310 Acked Read 5 Sent Done Done false
23120 7311 Acked Read 1 Sent Done Done false
23121 7312 Acked Write 3 Sent Done Done false
23122 7313 Acked Write 1 Sent Done Done false
23123 7314 Acked Read 4 Sent Done Done false
23124 7315 Acked Read 10 Sent Done Done false
23125 7316 Acked Read 6 Sent Done Done false
23126 7317 Acked Read 7 Sent Done Done false
23127 7318 Acked Read 1 Sent Done Done false
23128 7319 Acked Write 6 Sent Done Done false
23129 7320 Acked Write 10 Sent Done Done false
23130 7321 Acked Read 9 Sent Done Done false
23131 7322 Acked Write 7 Sent Done Done false
23132 7323 Acked Read 5 Sent Done Done false
23133 7324 Acked Read 8 Sent Done Done false
23134 7325 Acked Write 2 Sent Done Done false
23135 7326 Acked Write 1 Sent Done Done false
23136 7327 Acked Read 1 Sent Done Done false
23137 7328 Acked Flush 0 Sent Done Done false
23138 7329 Acked Write 1 Sent Done Done false
23139 7330 Acked Write 4 Sent Done Done false
23140 7331 Acked Write 1 Sent Done Done false
23141 7332 Acked Read 7 Sent Done Done false
23142 7333 Acked Write 8 Sent Done Done false
23143 7334 Acked Read 3 Sent Done Done false
23144 7335 Acked Read 8 Sent Done Done false
23145 7336 Acked Read 5 Sent Done Done false
23146 7337 Acked Write 4 Sent Done Done false
23147 7338 Acked Read 8 Sent Done Done false
23148 7339 Acked Read 5 Sent Done Done false
23149 7340 Acked Write 7 Sent Done Done false
23150 7341 Acked Read 9 Sent Done Done false
23151 7342 Acked Write 3 Sent Done Done false
23152 7343 Acked Write 8 Sent Done Done false
23153 7344 Acked Flush 0 Sent Done Done false
23154 7345 Acked Flush 0 Sent Done Done false
23155 STATES DS:0 DS:1 DS:2 TOTAL
23156 Sent 301 0 0 301
23157 Done 0 301 301 602
23158 Skipped 0 0 0 0
23159 Error 0 0 0 0
23160 Last Flush: 7044 7345 7345
23161 Downstairs last five completed: 7044 7043 7042 7041 7040
23162 Upstairs last five completed: 7345 7344 7343 7342 7341
23163 CLIENT: Up:0 ds:301 act:2
231642024-11-01T16:55:20.231ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
231652024-11-01T16:55:20.231ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50470) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
231662024-11-01T16:55:20.233ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
231672024-11-01T16:55:20.233ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
231682024-11-01T16:55:20.233ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
231692024-11-01T16:55:20.233ZINFOcrucible: send last flush ID to this DS: 7044 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
231702024-11-01T16:55:20.233ZINFOcrucible: Replied this last flush ID: 7044 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
231712024-11-01T16:55:20.234ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
231722024-11-01T16:55:20.234ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
231732024-11-01T16:55:20.234ZINFOcrucible: [0] client re-new 301 jobs since flush 7044 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
231742024-11-01T16:55:20.234ZINFOcrucible: [0] Replayed 301 jobs since flush: 7044 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
23175 ----------------------------------------------------------------
23176 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23177 Downstairs last five completed: 7345 7344 7343 7342 7341
23178 Upstairs last five completed: 7345 7344 7343 7342 7341
23179 CLIENT: Up:0 ds:0 act:3
23180 Replay: All jobs finished
23181 Main task has work to do, go find it
23182 got dsc Stop(0)
23183 stop 0
23184 Nov 01 16:55:22.541 INFO request completed, latency_us: 366[8810] Got stop action so:false kr:false
23185 , response_code: 204, uri: /stop/cid/0, method: POST, req_id: 7ad82d2f-e26f-4430-8107-52eb2a70c613, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
23186 Nov 01 16:55:22.541{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:55:22.541879382Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
23187 INFO request completed, latency_us: 132, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:55:22.54195262Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
23188 response_code: 200, uri: /state/cid/0, method: GET, req_id: {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:55:22.541980626Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
23189 27a5f6f2-87e7-4bd6-a92d-4788fb48defd, remote_addr: 127.0.{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:55:22.542025318Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"client":"0","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
23190 0.1:58552, local_addr: 127.0.0.1:9998
231912024-11-01T16:55:22.542ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
23192 [8810] Exited with: ExitStatus(unix_wait_status(9))
23193 [8810][0] reports Exit
23194 Nov 01 16:55:26.543 INFO request completed, latency_us: 220, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9cfa1361-fd00-4418-9f15-823310792acc, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
23195 001/300 Read block 400 len 16384
23196 002/300 Read block 2275 len 32768
23197 003/300 Write block 3643 len 36864 data: 2 3 3 3 3 4 3 3 3
23198 004/300 Read block 3470 len 4096
23199 005/300 Read block 78 len 20480
23200 006/300 Write block 4955 len 40960 data: 4 5 4 4 3 3 3 5 5 6
23201 007/300 Flush
23202 008/300 Read block 3465 len 16384
23203 009/300 Write block 2313 len 8192 data: 5 5
23204 010/300 Read block 4282 len 40960
23205 011/300 Read block 3819 len 36864
23206 012/300 Flush
23207 013/300 Read block 50 len 20480
23208 014/300 Write block 1065 len 28672 data: 5 5 4 4 5 5 6
23209 015/300 Write block 4436 len 24576 data: 4 4 5 5 5 3
23210 016/300 Write block 903 len 16384 data: 5 4 4 5
23211 017/300 Read block 3963 len 8192
23212 018/300 Write block 434 len 16384 data: 4 4 6 5
23213 019/300 Write block 2518 len 12288 data: 2 2 2
23214 020/300 Read block 299 len 40960
23215 021/300 Write block 3486 len 12288 data: 6 6 5
23216 022/300 Read block 2453 len 16384
23217 023/300 Write block 94 len 32768 data: 7 6 7 6 6 6 5 5
23218 024/300 Read block 4406 len 16384
23219 025/300 Flush
23220 026/300 Read block 1088 len 28672
23221 027/300 Write block 859 len 32768 data: 4 5 4 5 5 4 3 3
23222 028/300 Read block 2061 len 8192
23223 029/300 Read block 3312 len 4096
23224 030/300 Read block 4790 len 40960
23225 031/300 Read block 3512 len 8192
23226 032/300 Write block 277 len 40960 data: 7 7 7 5 5 5 4 4 4 6
23227 033/300 Read block 3390 len 24576
23228 034/300 Write block 3171 len 40960 data: 6 6 9 8 6 5 4 4 4 3
23229 035/300 Read block 165 len 32768
23230 036/300 Write block 2777 len 16384 data: 7 7 5 5
23231 037/300 Read block 361 len 20480
23232 038/300 Read block 674 len 8192
23233 039/300 Write block 42 len 8192 data: 5 6
23234 040/300 Flush
23235 041/300 Write block 4012 len 28672 data: 4 4 2 3 3 3 4
23236 042/300 Write block 1964 len 16384 data: 5 4 4 5
23237 043/300 Read block 3927 len 16384
23238 044/300 Flush
23239 045/300 Write block 3990 len 20480 data: 5 5 6 4 4
23240 046/300 Read block 3905 len 20480
23241 047/300 Read block 3202 len 36864
23242 048/300 Write block 4107 len 12288 data: 7 7 7
23243 049/300 Read block 3263 len 28672
23244 050/300 Write block 600 len 8192 data: 6 5
23245 051/300 Read block 3498 len 32768
23246 052/300 Read block 851 len 16384
23247 053/300 Write block 2256 len 20480 data: 4 4 4 5 4
23248 054/300 Read block 846 len 40960
23249 055/300 Read block 1812 len 28672
23250 056/300 Read block 2962 len 40960
23251 057/300 Write block 1808 len 32768 data: 6 5 6 6 6 4 4 4
23252 058/300 Flush
23253 059/300 Write block 2552 len 4096 data: 6
23254 060/300 Write block 2425 len 4096 data: 4
23255 061/300 Read block 1444 len 36864
23256 062/300 Read block 4853 len 12288
23257 063/300 Read block 2369 len 36864
23258 064/300 Read block 3085 len 20480
23259 065/300 Write block 3674 len 40960 data: 6 5 5 5 5 3 3 2 2 2
23260 066/300 Read block 149 len 8192
23261 067/300 Write block 2409 len 28672 data: 4 3 3 3 3 3 4
23262 068/300 Read block 1803 len 8192
23263 069/300 Read block 572 len 36864
23264 070/300 Write block 2913 len 20480 data: 3 5 5 5 5
23265 071/300 Flush
23266 072/300 Write block 896 len 8192 data: 6 6
23267 073/300 Write block 2517 len 8192 data: 3 3
23268 074/300 Read block 3461 len 32768
23269 075/300 Flush
23270 076/300 Read block 3277 len 32768
23271 077/300 Write block 1885 len 20480 data: 3 3 3 3 2
23272 078/300 Read block 1584 len 16384
23273 079/300 Read block 194 len 28672
23274 080/300 Read block 3495 len 8192
23275 081/300 Read block 1211 len 36864
23276 082/300 Read block 2252 len 12288
23277 083/300 Read block 1350 len 12288
23278 084/300 Read block 1467 len 16384
23279 085/300 Write block 4994 len 8192 data: 5 7
23280 086/300 Read block 2458 len 8192
23281 087/300 Write block 4107 len 36864 data: 8 8 8 8 8 7 6 6 6
23282 088/300 Flush
23283 089/300 Flush
23284 090/300 Write block 359 len 8192 data: 2 3
23285 091/300 Read block 3247 len 36864
23286 092/300 Read block 3056 len 8192
23287 093/300 Write block 425 len 32768 data: 3 3 2 2 2 3 3 3
23288 094/300 Write block 49 len 40960 data: 4 6 5 5 4 4 3 3 3 3
23289 095/300 Read block 3068 len 32768
23290 096/300 Read block 3742 len 28672
23291 097/300 Write block 946 len 24576 data: 5 5 4 3 3 2
23292 098/300 Read block 3779 len 36864
23293 099/300 Flush
23294 100/300 Write block 2221 len 40960 data: 6 7 8 7 6 6 6 6 4 6
23295 101/300 Write block 1051 len 16384 data: 4 5 7 6
23296 102/300 Read block 2118 len 24576
23297 103/300 Write block 379 len 32768 data: 5 6 4 5 5 6 6 6
23298 104/300 Write block 1201 len 28672 data: 6 6 6 6 5 5 7
23299 105/300 Write block 1058 len 4096 data: 5
23300 106/300 Write block 861 len 20480 data: 5 6 6 5 4
23301 107/300 Read block 4269 len 4096
23302 108/300 Write block 1535 len 36864 data: 4 4 5 5 6 6 5 4 4
23303 109/300 Write block 2083 len 24576 data: 4 5 6 8 8 7
23304 110/300 Read block 4713 len 16384
23305 111/300 Write block 1197 len 40960 data: 6 5 5 5 7 7 7 7 6 6
23306 112/300 Write block 4885 len 24576 data: 5 5 5 6 5 5
23307 113/300 Flush
23308 114/300 Write block 627 len 24576 data: 5 4 4 3 3 4
23309 115/300 Write block 1765 len 12288 data: 6 6 5
23310 116/300 Write block 4123 len 24576 data: 3 4 4 3 3 3
23311 117/300 Read block 1257 len 4096
23312 118/300 Read block 2936 len 32768
23313 119/300 Read block 4155 len 36864
23314 120/300 Read block 73 len 32768
23315 121/300 Read block 2120 len 20480
23316 122/300 Write block 4651 len 16384 data: 4 4 5 5
23317 123/300 Write block 2936 len 8192 data: 5 5
23318 124/300 Write block 1208 len 28672 data: 7 6 5 6 5 8 8
23319 125/300 Read block 204 len 8192
23320 126/300 Write block 3356 len 24576 data: 4 4 2 2 4 5
23321 127/300 Read block 3908 len 4096
23322 128/300 Read block 4373 len 16384
23323 129/300 Read block 1546 len 40960
23324 130/300 Read block 3725 len 36864
23325 131/300 Read block 3399 len 20480
23326 132/300 Write block 782 len 12288 data: 5 4 5
23327 133/300 Write block 4417 len 16384 data: 3 3 4 5
23328 134/300 Read block 3928 len 16384
23329 135/300 Write block 529 len 32768 data: 10 9 9 8 8 7 6 5
23330 136/300 Write block 4204 len 20480 data: 7 6 6 5 4
23331 137/300 Read block 1605 len 24576
23332 138/300 Read block 839 len 4096
23333 139/300 Write block 3716 len 20480 data: 5 6 6 5 3
23334 140/300 Write block 1165 len 32768 data: 4 3 3 3 3 3 5 6
23335 141/300 Read block 3575 len 32768
23336 142/300 Flush
23337 143/300 Write block 3799 len 40960 data: 7 7 5 6 6 7 4 5 4 5
23338 144/300 Read block 2345 len 32768
23339 145/300 Write block 1138 len 12288 data: 7 6 5
23340 146/300 Flush
23341 147/300 Write block 3162 len 8192 data: 3 5
23342 148/300 Read block 4196 len 32768
23343 149/300 Write block 829 len 28672 data: 5 6 7 6 4 5 5
23344 150/300 Write block 3866 len 20480 data: 10 9 8 7 8
23345 151/300 Read block 803 len 4096
23346 152/300 Flush
23347 153/300 Read block 1489 len 40960
23348 154/300 Read block 2930 len 8192
23349 155/300 Write block 1694 len 32768 data: 6 6 5 4 4 4 5 3
23350 156/300 Read block 3449 len 40960
23351 157/300 Flush
23352 158/300 Write block 3067 len 12288 data: 3 3 3
23353 159/300 Write block 2581 len 4096 data: 5
23354 160/300 Read block 2763 len 16384
23355 161/300 Read block 61 len 24576
23356 162/300 Write block 94 len 24576 data: 8 7 8 7 7 7
23357 163/300 Write block 204 len 32768 data: 6 5 5 5 5 6 5 5
23358 164/300 Write block 868 len 4096 data: 2
23359 165/300 Write block 4925 len 40960 data: 7 7 5 5 4 3 2 2 4 4
23360 166/300 Read block 611 len 12288
23361 167/300 Read block 2932 len 12288
23362 168/300 Read block 2470 len 16384
23363 169/300 Read block 825 len 12288
23364 170/300 Read block 3787 len 40960
23365 171/300 Flush
23366 172/300 Read block 3728 len 28672
23367 173/300 Write block 2885 len 28672 data: 5 6 6 6 7 7 6
23368 174/300 Read block 1995 len 36864
23369 175/300 Read block 665 len 24576
23370 176/300 Write block 11 len 8192 data: 4 3
23371 177/300 Write block 129 len 20480 data: 5 5 3 3 2
23372 178/300 Read block 3897 len 12288
23373 179/300 Write block 2202 len 36864 data: 4 4 4 5 5 5 6 5 7
23374 180/300 Read block 305 len 16384
23375 181/300 Read block 4247 len 8192
23376 182/300 Write block 1524 len 40960 data: 7 7 7 6 6 5 5 4 3 4
23377 183/300 Read block 1628 len 28672
23378 184/300 Read block 1013 len 4096
23379 185/300 Write block 4186 len 36864 data: 4 4 4 4 5 5 6 6 8
23380 186/300 Write block 3100 len 36864 data: 7 4 3 3 2 4 5 5 6
23381 187/300 Write block 406 len 4096 data: 4
23382 188/300 Read block 253 len 4096
23383 189/300 Read block 613 len 8192
23384 190/300 Write block 4568 len 32768 data: 2 3 3 4 5 6 5 6
23385 191/300 Read block 2806 len 24576
23386 192/300 Flush
23387 193/300 Read block 3159 len 40960
23388 194/300 Read block 3916 len 12288
23389 195/300 Write block 3873 len 36864 data: 6 5 4 4 2 3 4 5 3
23390 196/300 Flush
23391 197/300 Write block 361 len 28672 data: 3 3 4 5 6 5 4
23392 198/300 Read block 3639 len 20480
23393 199/300 Write block 4883 len 12288 data: 6 6 6
23394 200/300 Read block 3469 len 8192
23395 201/300 Read block 3449 len 28672
23396 202/300 Write block 2663 len 4096 data: 5
23397 203/300 Read block 4498 len 24576
23398 204/300 Read block 85 len 24576
23399 205/300 Write block 3302 len 32768 data: 6 3 3 3 3 4 5 5
23400 206/300 Flush
23401 207/300 Read block 2470 len 24576
23402 208/300 Read block 4310 len 8192
23403 209/300 Read block 3727 len 28672
23404 210/300 Write block 1459 len 24576 data: 3 3 3 3 3 3
23405 211/300 Write block 198 len 12288 data: 8 9 6
23406 212/300 Write block 4563 len 28672 data: 2 2 2 3 2 3 4
23407 213/300 Flush
23408 214/300 Write block 3345 len 20480 data: 4 3 3 4 5
23409 215/300 Flush
23410 216/300 Read block 4103 len 32768
23411 217/300 Read block 3006 len 8192
23412 218/300 Read block 2684 len 20480
23413 219/300 Write block 4494 len 12288 data: 4 5 4
23414 220/300 Read block 2855 len 4096
23415 221/300 Write block 4243 len 40960 data: 2 2 3 4 5 5 5 4 5 4
23416 222/300 Read block 152 len 8192
23417 223/300 Write block 4146 len 32768 data: 6 5 4 4 4 4 3 3
23418 224/300 Read block 3093 len 36864
23419 225/300 Read block 1060 len 36864
23420 226/300 Read block 4874 len 20480
23421 227/300 Read block 2132 len 36864
23422 228/300 Read block 3683 len 16384
23423 229/300 Read block 1648 len 8192
23424 230/300 Write block 4626 len 32768 data: 4 4 4 6 6 6 7 6
23425 231/300 Flush
23426 232/300 Flush
23427 233/300 Read block 1407 len 24576
23428 234/300 Read block 1363 len 32768
23429 235/300 Write block 3849 len 8192 data: 4 4
23430 236/300 Read block 1144 len 28672
23431 237/300 Read block 2915 len 8192
23432 238/300 Read block 4011 len 24576
23433 239/300 Read block 1239 len 20480
23434 240/300 Write block 2496 len 20480 data: 5 5 4 4 4
23435 241/300 Read block 4837 len 24576
23436 242/300 Write block 4219 len 4096 data: 4
23437 243/300 Write block 890 len 24576 data: 6 6 7 7 8 7
23438 244/300 Write block 2316 len 16384 data: 5 5 5 4
23439 245/300 Read block 4579 len 36864
23440 246/300 Read block 3180 len 40960
23441 247/300 Read block 690 len 16384
23442 248/300 Flush
23443 249/300 Read block 1078 len 20480
23444 250/300 Read block 374 len 24576
23445 251/300 Read block 3214 len 28672
23446 252/300 Read block 486 len 8192
23447 253/300 Write block 407 len 40960 data: 4 4 6 6 6 5 6 5 5 3
23448 254/300 Read block 2756 len 20480
23449 255/300 Read block 1003 len 16384
23450 256/300 Read block 1290 len 36864
23451 257/300 Read block 909 len 28672
23452 258/300 Read block 3041 len 12288
23453 259/300 Read block 897 len 28672
23454 260/300 Flush
23455 261/300 Write block 3763 len 28672 data: 4 4 4 5 5 5 5
23456 262/300 Write block 899 len 24576 data: 4 4 4 5 6 5
23457 263/300 Read block 4846 len 12288
23458 264/300 Flush
23459 265/300 Read block 1405 len 28672
23460 266/300 Write block 2046 len 28672 data: 2 2 4 4 4 5 5
23461 267/300 Read block 1765 len 40960
23462 268/300 Read block 4917 len 8192
23463 269/300 Read block 4779 len 20480
23464 270/300 Read block 1716 len 40960
23465 271/300 Flush
23466 272/300 Read block 2583 len 32768
23467 273/300 Write block 2252 len 4096 data: 2
23468 274/300 Read block 2633 len 32768
23469 275/300 Flush
23470 276/300 Write block 4979 len 40960 data: 4 4 5 5 5 4 5 4 3 4
23471 277/300 Write block 4591 len 24576 data: 3 3 3 3 4 5
23472 278/300 Write block 2032 len 40960 data: 5 6 7 6 4 3 3 3 4 3
23473 279/300 Read block 725 len 4096
23474 280/300 Read block 4452 len 32768
23475 281/300 Write block 3670 len 24576 data: 5 6 7 6 7 6
23476 282/300 Read block 2776 len 40960
23477 283/300 Read block 4633 len 32768
23478 284/300 Write block 1631 len 36864 data: 2 4 4 4 4 4 4 5 5
23479 285/300 Read block 2521 len 40960
23480 286/300 Read block 803 len 4096
23481 287/300 Write block 4415 len 24576 data: 2 2 4 4 5 6
23482 288/300 Read block 4008 len 36864
23483 289/300 Write block 1682 len 4096 data: 3
23484 290/300 Read block 4624 len 24576
23485 291/300 Read block 4811 len 28672
23486 292/300 Read block 3598 len 24576
23487 293/300 Write block 1446 len 32768 data: 6 5 4 4 5 3 3 3
23488 294/300 Flush
23489 295/300 Read block 1392 len 28672
23490 296/300 Read block 4690 len 40960
23491 297/300 Read block 810 len 32768
23492 298/300 Flush
23493 299/300 Read block 3600 len 32768
23494 300/300 Read block 1162 len 8192
23495 301/300 Read block 1558 len 4096
23496 Main task has work to do, go find it
23497 got dsc Start(0)
23498 start 0
23499 Nov 01 16:55:26.841 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/0, method: POST, req_id: 544227ed-93bd-4f2b-ba87-954b208587be, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
23500 Replay: started 0, returned:Ok(())
23501 ----------------------------------------------------------------
23502 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23503 States: Offline Active Active
23504 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23505 7346 Acked Read 4 Sent Done Done false
23506 7347 Acked Read 8 Sent Done Done false
23507 7348 Acked Write 9 Sent Done Done false
23508 7349 Acked Read 1 Sent Done Done false
23509 7350 Acked Read 5 Sent Done Done false
23510 7351 Acked Write 10 Sent Done Done false
23511 7352 Acked Flush 0 Sent Done Done false
23512 7353 Acked Read 4 Sent Done Done false
23513 7354 Acked Write 2 Sent Done Done false
23514 7355 Acked Read 10 Sent Done Done false
23515 7356 Acked Read 9 Sent Done Done false
23516 7357 Acked Flush 0 Sent Done Done false
23517 7358 Acked Read 5 Sent Done Done false
23518 7359 Acked Write 7 Sent Done Done false
23519 7360 Acked Write 6 Sent Done Done false
23520 7361 Acked Write 4 Sent Done Done false
23521 7362 Acked Read 2 Sent Done Done false
23522 7363 Acked Write 4 Sent Done Done false
23523 7364 Acked Write 3 Sent Done Done false
23524 7365 Acked Read 10 Sent Done Done false
23525 7366 Acked Write 3 Sent Done Done false
23526 7367 Acked Read 4 Sent Done Done false
23527 7368 Acked Write 8 Sent Done Done false
23528 7369 Acked Read 4 Sent Done Done false
23529 7370 Acked Flush 0 Sent Done Done false
23530 7371 Acked Read 7 Sent Done Done false
23531 7372 Acked Write 8 Sent Done Done false
23532 7373 Acked Read 2 Sent Done Done false
23533 7374 Acked Read 1 Sent Done Done false
23534 7375 Acked Read 10 Sent Done Done false
23535 7376 Acked Read 2 Sent Done Done false
23536 7377 Acked Write 10 Sent Done Done false
23537 7378 Acked Read 6 Sent Done Done false
23538 7379 Acked Write 10 Sent Done Done false
23539 7380 Acked Read 8 Sent Done Done false
23540 7381 Acked Write 4 Sent Done Done false
23541 7382 Acked Read 5 Sent Done Done false
23542 7383 Acked Read 2 Sent Done Done false
23543 7384 Acked Write 2 Sent Done Done false
23544 7385 Acked Flush 0 Sent Done Done false
23545 7386 Acked Write 7 Sent Done Done false
23546 7387 Acked Write 4 Sent Done Done false
23547 7388 Acked Read 4 Sent Done Done false
23548 7389 Acked Flush 0 Sent Done Done false
23549 7390 Acked Write 5 Sent Done Done false
23550 7391 Acked Read 5 Sent Done Done false
23551 7392 Acked Read 9 Sent Done Done false
23552 7393 Acked Write 3 Sent Done Done false
23553 7394 Acked Read 7 Sent Done Done false
23554 7395 Acked Write 2 Sent Done Done false
23555 7396 Acked Read 8 Sent Done Done false
23556 7397 Acked Read 4 Sent Done Done false
23557 7398 Acked Write 5 Sent Done Done false
23558 7399 Acked Read 10 Sent Done Done false
23559 7400 Acked Read 7 Sent Done Done false
23560 7401 Acked Read 10 Sent Done Done false
23561 7402 Acked Write 8 Sent Done Done false
23562 7403 Acked Flush 0 Sent Done Done false
23563 7404 Acked Write 1 Sent Done Done false
23564 7405 Acked Write 1 Sent Done Done false
23565 7406 Acked Read 9 Sent Done Done false
23566 7407 Acked Read 3 Sent Done Done false
23567 7408 Acked Read 9 Sent Done Done false
23568 7409 Acked Read 5 Sent Done Done false
23569 7410 Acked Write 10 Sent Done Done false
23570 7411 Acked Read 2 Sent Done Done false
23571 7412 Acked Write 7 Sent Done Done false
23572 7413 Acked Read 2 Sent Done Done false
23573 7414 Acked Read 9 Sent Done Done false
23574 7415 Acked Write 5 Sent Done Done false
23575 7416 Acked Flush 0 Sent Done Done false
23576 7417 Acked Write 2 Sent Done Done false
23577 7418 Acked Write 2 Sent Done Done false
23578 7419 Acked Read 8 Sent Done Done false
23579 7420 Acked Flush 0 Sent Done Done false
23580 7421 Acked Read 8 Sent Done Done false
23581 7422 Acked Write 5 Sent Done Done false
23582 7423 Acked Read 4 Sent Done Done false
23583 7424 Acked Read 7 Sent Done Done false
23584 7425 Acked Read 2 Sent Done Done false
23585 7426 Acked Read 9 Sent Done Done false
23586 7427 Acked Read 3 Sent Done Done false
23587 7428 Acked Read 3 Sent Done Done false
23588 7429 Acked Read 4 Sent Done Done false
23589 7430 Acked Write 2 Sent Done Done false
23590 7431 Acked Read 2 Sent Done Done false
23591 7432 Acked Write 9 Sent Done Done false
23592 7433 Acked Flush 0 Sent Done Done false
23593 7434 Acked Flush 0 Sent Done Done false
23594 7435 Acked Write 2 Sent Done Done false
23595 7436 Acked Read 9 Sent Done Done false
23596 7437 Acked Read 2 Sent Done Done false
23597 7438 Acked Write 8 Sent Done Done false
23598 7439 Acked Write 10 Sent Done Done false
23599 7440 Acked Read 8 Sent Done Done false
23600 7441 Acked Read 7 Sent Done Done false
23601 7442 Acked Write 6 Sent Done Done false
23602 7443 Acked Read 9 Sent Done Done false
23603 7444 Acked Flush 0 Sent Done Done false
23604 7445 Acked Write 10 Sent Done Done false
23605 7446 Acked Write 4 Sent Done Done false
23606 7447 Acked Read 6 Sent Done Done false
23607 7448 Acked Write 8 Sent Done Done false
23608 7449 Acked Write 7 Sent Done Done false
23609 7450 Acked Write 1 Sent Done Done false
23610 7451 Acked Write 5 Sent Done Done false
23611 7452 Acked Read 1 Sent Done Done false
23612 7453 Acked Write 9 Sent Done Done false
23613 7454 Acked Write 6 Sent Done Done false
23614 7455 Acked Read 4 Sent Done Done false
23615 7456 Acked Write 10 Sent Done Done false
23616 7457 Acked Write 6 Sent Done Done false
23617 7458 Acked Flush 0 Sent Done Done false
23618 7459 Acked Write 6 Sent Done Done false
23619 7460 Acked Write 3 Sent Done Done false
23620 7461 Acked Write 6 Sent Done Done false
23621 7462 Acked Read 1 Sent Done Done false
23622 7463 Acked Read 8 Sent Done Done false
23623 7464 Acked Read 9 Sent Done Done false
23624 7465 Acked Read 8 Sent Done Done false
23625 7466 Acked Read 5 Sent Done Done false
23626 7467 Acked Write 4 Sent Done Done false
23627 7468 Acked Write 2 Sent Done Done false
23628 7469 Acked Write 7 Sent Done Done false
23629 7470 Acked Read 2 Sent Done Done false
23630 7471 Acked Write 6 Sent Done Done false
23631 7472 Acked Read 1 Sent Done Done false
23632 7473 Acked Read 4 Sent Done Done false
23633 7474 Acked Read 10 Sent Done Done false
23634 7475 Acked Read 9 Sent Done Done false
23635 7476 Acked Read 5 Sent Done Done false
23636 7477 Acked Write 3 Sent Done Done false
23637 7478 Acked Write 4 Sent Done Done false
23638 7479 Acked Read 4 Sent Done Done false
23639 7480 Acked Write 8 Sent Done Done false
23640 7481 Acked Write 5 Sent Done Done false
23641 7482 Acked Read 6 Sent Done Done false
23642 7483 Acked Read 1 Sent Done Done false
23643 7484 Acked Write 5 Sent Done Done false
23644 7485 Acked Write 8 Sent Done Done false
23645 7486 Acked Read 8 Sent Done Done false
23646 7487 Acked Flush 0 Sent Done Done false
23647 7488 Acked Write 10 Sent Done Done false
23648 7489 Acked Read 8 Sent Done Done false
23649 7490 Acked Write 3 Sent Done Done false
23650 7491 Acked Flush 0 Sent Done Done false
23651 7492 Acked Write 2 Sent Done Done false
23652 7493 Acked Read 8 Sent Done Done false
23653 7494 Acked Write 7 Sent Done Done false
23654 7495 Acked Write 5 Sent Done Done false
23655 7496 Acked Read 1 Sent Done Done false
23656 7497 Acked Flush 0 Sent Done Done false
23657 7498 Acked Read 10 Sent Done Done false
23658 7499 Acked Read 2 Sent Done Done false
23659 7500 Acked Write 8 Sent Done Done false
23660 7501 Acked Read 10 Sent Done Done false
23661 7502 Acked Flush 0 Sent Done Done false
23662 7503 Acked Write 3 Sent Done Done false
23663 7504 Acked Write 1 Sent Done Done false
23664 7505 Acked Read 4 Sent Done Done false
23665 7506 Acked Read 6 Sent Done Done false
23666 7507 Acked Write 6 Sent Done Done false
23667 7508 Acked Write 8 Sent Done Done false
23668 7509 Acked Write 1 Sent Done Done false
23669 7510 Acked Write 10 Sent Done Done false
23670 7511 Acked Read 3 Sent Done Done false
23671 7512 Acked Read 3 Sent Done Done false
23672 7513 Acked Read 4 Sent Done Done false
23673 7514 Acked Read 3 Sent Done Done false
23674 7515 Acked Read 10 Sent Done Done false
23675 7516 Acked Flush 0 Sent Done Done false
23676 7517 Acked Read 7 Sent Done Done false
23677 7518 Acked Write 7 Sent Done Done false
23678 7519 Acked Read 9 Sent Done Done false
23679 7520 Acked Read 6 Sent Done Done false
23680 7521 Acked Write 2 Sent Done Done false
23681 7522 Acked Write 5 Sent Done Done false
23682 7523 Acked Read 3 Sent Done Done false
23683 7524 Acked Write 9 Sent Done Done false
23684 7525 Acked Read 4 Sent Done Done false
23685 7526 Acked Read 2 Sent Done Done false
23686 7527 Acked Write 10 Sent Done Done false
23687 7528 Acked Read 7 Sent Done Done false
23688 7529 Acked Read 1 Sent Done Done false
23689 7530 Acked Write 9 Sent Done Done false
23690 7531 Acked Write 9 Sent Done Done false
23691 7532 Acked Write 1 Sent Done Done false
23692 7533 Acked Read 1 Sent Done Done false
23693 7534 Acked Read 2 Sent Done Done false
23694 7535 Acked Write 8 Sent Done Done false
23695 7536 Acked Read 6 Sent Done Done false
23696 7537 Acked Flush 0 Sent Done Done false
23697 7538 Acked Read 10 Sent Done Done false
23698 7539 Acked Read 3 Sent Done Done false
23699 7540 Acked Write 9 Sent Done Done false
23700 7541 Acked Flush 0 Sent Done Done false
23701 7542 Acked Write 7 Sent Done Done false
23702 7543 Acked Read 5 Sent Done Done false
23703 7544 Acked Write 3 Sent Done Done false
23704 7545 Acked Read 2 Sent Done Done false
23705 7546 Acked Read 7 Sent Done Done false
23706 7547 Acked Write 1 Sent Done Done false
23707 7548 Acked Read 6 Sent Done Done false
23708 7549 Acked Read 6 Sent Done Done false
23709 7550 Acked Write 8 Sent Done Done false
23710 7551 Acked Flush 0 Sent Done Done false
23711 7552 Acked Read 6 Sent Done Done false
23712 7553 Acked Read 2 Sent Done Done false
23713 7554 Acked Read 7 Sent Done Done false
23714 7555 Acked Write 6 Sent Done Done false
23715 7556 Acked Write 3 Sent Done Done false
23716 7557 Acked Write 7 Sent Done Done false
23717 7558 Acked Flush 0 Sent Done Done false
23718 7559 Acked Write 5 Sent Done Done false
23719 7560 Acked Flush 0 Sent Done Done false
23720 7561 Acked Read 8 Sent Done Done false
23721 7562 Acked Read 2 Sent Done Done false
23722 7563 Acked Read 5 Sent Done Done false
23723 7564 Acked Write 3 Sent Done Done false
23724 7565 Acked Read 1 Sent Done Done false
23725 7566 Acked Write 10 Sent Done Done false
23726 7567 Acked Read 2 Sent Done Done false
23727 7568 Acked Write 8 Sent Done Done false
23728 7569 Acked Read 9 Sent Done Done false
23729 7570 Acked Read 9 Sent Done Done false
23730 7571 Acked Read 5 Sent Done Done false
23731 7572 Acked Read 9 Sent Done Done false
23732 7573 Acked Read 4 Sent Done Done false
23733 7574 Acked Read 2 Sent Done Done false
23734 7575 Acked Write 8 Sent Done Done false
23735 7576 Acked Flush 0 Sent Done Done false
23736 7577 Acked Flush 0 Sent Done Done false
23737 7578 Acked Read 6 Sent Done Done false
23738 7579 Acked Read 8 Sent Done Done false
23739 7580 Acked Write 2 Sent Done Done false
23740 7581 Acked Read 7 Sent Done Done false
23741 7582 Acked Read 2 Sent Done Done false
23742 7583 Acked Read 6 Sent Done Done false
23743 7584 Acked Read 5 Sent Done Done false
23744 7585 Acked Write 5 Sent Done Done false
23745 7586 Acked Read 6 Sent Done Done false
23746 7587 Acked Write 1 Sent Done Done false
23747 7588 Acked Write 6 Sent Done Done false
23748 7589 Acked Write 4 Sent Done Done false
23749 7590 Acked Read 9 Sent Done Done false
23750 7591 Acked Read 10 Sent Done Done false
23751 7592 Acked Read 4 Sent Done Done false
23752 7593 Acked Flush 0 Sent Done Done false
23753 7594 Acked Read 5 Sent Done Done false
23754 7595 Acked Read 6 Sent Done Done false
23755 7596 Acked Read 7 Sent Done Done false
23756 7597 Acked Read 2 Sent Done Done false
23757 7598 Acked Write 10 Sent Done Done false
23758 7599 Acked Read 5 Sent Done Done false
23759 7600 Acked Read 4 Sent Done Done false
23760 7601 Acked Read 9 Sent Done Done false
23761 7602 Acked Read 7 Sent Done Done false
23762 7603 Acked Read 3 Sent Done Done false
23763 7604 Acked Read 7 Sent Done Done false
23764 7605 Acked Flush 0 Sent Done Done false
23765 7606 Acked Write 7 Sent Done Done false
23766 7607 Acked Write 6 Sent Done Done false
23767 7608 Acked Read 3 Sent Done Done false
23768 7609 Acked Flush 0 Sent Done Done false
23769 7610 Acked Read 7 Sent Done Done false
23770 7611 Acked Write 7 Sent Done Done false
23771 7612 Acked Read 10 Sent Done Done false
23772 7613 Acked Read 2 Sent Done Done false
23773 7614 Acked Read 5 Sent Done Done false
23774 7615 Acked Read 10 Sent Done Done false
23775 7616 Acked Flush 0 Sent Done Done false
23776 7617 Acked Read 8 Sent Done Done false
23777 7618 Acked Write 1 Sent Done Done false
23778 7619 Acked Read 8 Sent Done Done false
23779 7620 Acked Flush 0 Sent Done Done false
23780 7621 Acked Write 10 Sent Done Done false
23781 7622 Acked Write 6 Sent Done Done false
23782 7623 Acked Write 10 Sent Done Done false
23783 7624 Acked Read 1 Sent Done Done false
23784 7625 Acked Read 8 Sent Done Done false
23785 7626 Acked Write 6 Sent Done Done false
23786 7627 Acked Read 10 Sent Done Done false
23787 7628 Acked Read 8 Sent Done Done false
23788 7629 Acked Write 9 Sent Done Done false
23789 7630 Acked Read 10 Sent Done Done false
23790 7631 Acked Read 1 Sent Done Done false
23791 7632 Acked Write 6 Sent Done Done false
23792 7633 Acked Read 9 Sent Done Done false
23793 7634 Acked Write 1 Sent Done Done false
23794 7635 Acked Read 6 Sent Done Done false
23795 7636 Acked Read 7 Sent Done Done false
23796 7637 Acked Read 6 Sent Done Done false
23797 7638 Acked Write 8 Sent Done Done false
23798 7639 Acked Flush 0 Sent Done Done false
23799 7640 Acked Read 7 Sent Done Done false
23800 7641 Acked Read 10 Sent Done Done false
23801 7642 Acked Read 8 Sent Done Done false
23802 7643 Acked Flush 0 Sent Done Done false
23803 7644 Acked Read 8 Sent Done Done false
23804 7645 Acked Read 2 Sent Done Done false
23805 7646 Acked Read 1 Sent Done Done false
23806 STATES DS:0 DS:1 DS:2 TOTAL
23807 Sent 301 0 0 301
23808 Done 0 301 301 602
23809 Skipped 0 0 0 0
23810 Error 0 0 0 0
23811 Last Flush: 7345 7643 7643
23812 Downstairs last five completed: 7345 7344 7343 7342 7341
23813 Upstairs last five completed: 7646 7645 7644 7643 7642
23814 CLIENT: Up:0 ds:301 act:2
23815 ----------------------------------------------------------------
23816 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
23817 States: Offline Active Active
23818 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23819 7346 Acked Read 4 Sent Done Done false
23820 7347 Acked Read 8 Sent Done Done false
23821 7348 Acked Write 9 Sent Done Done false
23822 7349 Acked Read 1 Sent Done Done false
23823 7350 Acked Read 5 Sent Done Done false
23824 7351 Acked Write 10 Sent Done Done false
23825 7352 Acked Flush 0 Sent Done Done false
23826 7353 Acked Read 4 Sent Done Done false
23827 7354 Acked Write 2 Sent Done Done false
23828 7355 Acked Read 10 Sent Done Done false
23829 7356 Acked Read 9 Sent Done Done false
23830 7357 Acked Flush 0 Sent Done Done false
23831 7358 Acked Read 5 Sent Done Done false
23832 7359 Acked Write 7 Sent Done Done false
23833 7360 Acked Write 6 Sent Done Done false
23834 7361 Acked Write 4 Sent Done Done false
23835 7362 Acked Read 2 Sent Done Done false
23836 7363 Acked Write 4 Sent Done Done false
23837 7364 Acked Write 3 Sent Done Done false
23838 7365 Acked Read 10 Sent Done Done false
23839 7366 Acked Write 3 Sent Done Done false
23840 7367 Acked Read 4 Sent Done Done false
23841 7368 Acked Write 8 Sent Done Done false
23842 7369 Acked Read 4 Sent Done Done false
23843 7370 Acked Flush 0 Sent Done Done false
23844 7371 Acked Read 7 Sent Done Done false
23845 7372 Acked Write 8 Sent Done Done false
23846 7373 Acked Read 2 Sent Done Done false
23847 7374 Acked Read 1 Sent Done Done false
23848 7375 Acked Read 10 Sent Done Done false
23849 7376 Acked Read 2 Sent Done Done false
23850 7377 Acked Write 10 Sent Done Done false
23851 7378 Acked Read 6 Sent Done Done false
23852 7379 Acked Write 10 Sent Done Done false
23853 7380 Acked Read 8 Sent Done Done false
23854 7381 Acked Write 4 Sent Done Done false
23855 7382 Acked Read 5 Sent Done Done false
23856 7383 Acked Read 2 Sent Done Done false
23857 7384 Acked Write 2 Sent Done Done false
23858 7385 Acked Flush 0 Sent Done Done false
23859 7386 Acked Write 7 Sent Done Done false
23860 7387 Acked Write 4 Sent Done Done false
23861 7388 Acked Read 4 Sent Done Done false
23862 7389 Acked Flush 0 Sent Done Done false
23863 7390 Acked Write 5 Sent Done Done false
23864 7391 Acked Read 5 Sent Done Done false
23865 7392 Acked Read 9 Sent Done Done false
23866 7393 Acked Write 3 Sent Done Done false
23867 7394 Acked Read 7 Sent Done Done false
23868 7395 Acked Write 2 Sent Done Done false
23869 7396 Acked Read 8 Sent Done Done false
23870 7397 Acked Read 4 Sent Done Done false
23871 7398 Acked Write 5 Sent Done Done false
23872 7399 Acked Read 10 Sent Done Done false
23873 7400 Acked Read 7 Sent Done Done false
23874 7401 Acked Read 10 Sent Done Done false
23875 7402 Acked Write 8 Sent Done Done false
23876 7403 Acked Flush 0 Sent Done Done false
23877 7404 Acked Write 1 Sent Done Done false
23878 7405 Acked Write 1 Sent Done Done false
23879 7406 Acked Read 9 Sent Done Done false
23880 7407 Acked Read 3 Sent Done Done false
23881 7408 Acked Read 9 Sent Done Done false
23882 7409 Acked Read 5 Sent Done Done false
23883 7410 Acked Write 10 Sent Done Done false
23884 7411 Acked Read 2 Sent Done Done false
23885 7412 Acked Write 7 Sent Done Done false
23886 7413 Acked Read 2 Sent Done Done false
23887 7414 Acked Read 9 Sent Done Done false
23888 7415 Acked Write 5 Sent Done Done false
23889 7416 Acked Flush 0 Sent Done Done false
23890 7417 Acked Write 2 Sent Done Done false
23891 7418 Acked Write 2 Sent Done Done false
23892 7419 Acked Read 8 Sent Done Done false
23893 7420 Acked Flush 0 Sent Done Done false
23894 7421 Acked Read 8 Sent Done Done false
23895 7422 Acked Write 5 Sent Done Done false
23896 7423 Acked Read 4 Sent Done Done false
23897 7424 Acked Read 7 Sent Done Done false
23898 7425 Acked Read 2 Sent Done Done false
23899 7426 Acked Read 9 Sent Done Done false
23900 7427 Acked Read 3 Sent Done Done false
23901 7428 Acked Read 3 Sent Done Done false
23902 7429 Acked Read 4 Sent Done Done false
23903 7430 Acked Write 2 Sent Done Done false
23904 7431 Acked Read 2 Sent Done Done false
23905 7432 Acked Write 9 Sent Done Done false
23906 7433 Acked Flush 0 Sent Done Done false
23907 7434 Acked Flush 0 Sent Done Done false
23908 7435 Acked Write 2 Sent Done Done false
23909 7436 Acked Read 9 Sent Done Done false
23910 7437 Acked Read 2 Sent Done Done false
23911 7438 Acked Write 8 Sent Done Done false
23912 7439 Acked Write 10 Sent Done Done false
23913 7440 Acked Read 8 Sent Done Done false
23914 7441 Acked Read 7 Sent Done Done false
23915 7442 Acked Write 6 Sent Done Done false
23916 7443 Acked Read 9 Sent Done Done false
23917 7444 Acked Flush 0 Sent Done Done false
23918 7445 Acked Write 10 Sent Done Done false
23919 7446 Acked Write 4 Sent Done Done false
23920 7447 Acked Read 6 Sent Done Done false
23921 7448 Acked Write 8 Sent Done Done false
23922 7449 Acked Write 7 Sent Done Done false
23923 7450 Acked Write 1 Sent Done Done false
23924 7451 Acked Write 5 Sent Done Done false
23925 7452 Acked Read 1 Sent Done Done false
23926 7453 Acked Write 9 Sent Done Done false
23927 7454 Acked Write 6 Sent Done Done false
23928 7455 Acked Read 4 Sent Done Done false
23929 7456 Acked Write 10 Sent Done Done false
23930 7457 Acked Write 6 Sent Done Done false
23931 7458 Acked Flush 0 Sent Done Done false
23932 7459 Acked Write 6 Sent Done Done false
23933 7460 Acked Write 3 Sent Done Done false
23934 7461 Acked Write 6 Sent Done Done false
23935 7462 Acked Read 1 Sent Done Done false
23936 7463 Acked Read 8 Sent Done Done false
23937 7464 Acked Read 9 Sent Done Done false
23938 7465 Acked Read 8 Sent Done Done false
23939 7466 Acked Read 5 Sent Done Done false
23940 7467 Acked Write 4 Sent Done Done false
23941 7468 Acked Write 2 Sent Done Done false
23942 7469 Acked Write 7 Sent Done Done false
23943 7470 Acked Read 2 Sent Done Done false
23944 7471 Acked Write 6 Sent Done Done false
23945 7472 Acked Read 1 Sent Done Done false
23946 7473 Acked Read 4 Sent Done Done false
23947 7474 Acked Read 10 Sent Done Done false
23948 7475 Acked Read 9 Sent Done Done false
23949 7476 Acked Read 5 Sent Done Done false
23950 7477 Acked Write 3 Sent Done Done false
23951 7478 Acked Write 4 Sent Done Done false
23952 7479 Acked Read 4 Sent Done Done false
23953 7480 Acked Write 8 Sent Done Done false
23954 7481 Acked Write 5 Sent Done Done false
23955 7482 Acked Read 6 Sent Done Done false
23956 7483 Acked Read 1 Sent Done Done false
23957 7484 Acked Write 5 Sent Done Done false
23958 7485 Acked Write 8 Sent Done Done false
23959 7486 Acked Read 8 Sent Done Done false
23960 7487 Acked Flush 0 Sent Done Done false
23961 7488 Acked Write 10 Sent Done Done false
23962 7489 Acked Read 8 Sent Done Done false
23963 7490 Acked Write 3 Sent Done Done false
23964 7491 Acked Flush 0 Sent Done Done false
23965 7492 Acked Write 2 Sent Done Done false
23966 7493 Acked Read 8 Sent Done Done false
23967 7494 Acked Write 7 Sent Done Done false
23968 7495 Acked Write 5 Sent Done Done false
23969 7496 Acked Read 1 Sent Done Done false
23970 7497 Acked Flush 0 Sent Done Done false
23971 7498 Acked Read 10 Sent Done Done false
23972 7499 Acked Read 2 Sent Done Done false
23973 7500 Acked Write 8 Sent Done Done false
23974 7501 Acked Read 10 Sent Done Done false
23975 7502 Acked Flush 0 Sent Done Done false
23976 7503 Acked Write 3 Sent Done Done false
23977 7504 Acked Write 1 Sent Done Done false
23978 7505 Acked Read 4 Sent Done Done false
23979 7506 Acked Read 6 Sent Done Done false
23980 7507 Acked Write 6 Sent Done Done false
23981 7508 Acked Write 8 Sent Done Done false
23982 7509 Acked Write 1 Sent Done Done false
23983 7510 Acked Write 10 Sent Done Done false
23984 7511 Acked Read 3 Sent Done Done false
23985 7512 Acked Read 3 Sent Done Done false
23986 7513 Acked Read 4 Sent Done Done false
23987 7514 Acked Read 3 Sent Done Done false
23988 7515 Acked Read 10 Sent Done Done false
23989 7516 Acked Flush 0 Sent Done Done false
23990 7517 Acked Read 7 Sent Done Done false
23991 7518 Acked Write 7 Sent Done Done false
23992 7519 Acked Read 9 Sent Done Done false
23993 7520 Acked Read 6 Sent Done Done false
23994 7521 Acked Write 2 Sent Done Done false
23995 7522 Acked Write 5 Sent Done Done false
23996 7523 Acked Read 3 Sent Done Done false
23997 7524 Acked Write 9 Sent Done Done false
23998 7525 Acked Read 4 Sent Done Done false
23999 7526 Acked Read 2 Sent Done Done false
24000 7527 Acked Write 10 Sent Done Done false
24001 7528 Acked Read 7 Sent Done Done false
24002 7529 Acked Read 1 Sent Done Done false
24003 7530 Acked Write 9 Sent Done Done false
24004 7531 Acked Write 9 Sent Done Done false
24005 7532 Acked Write 1 Sent Done Done false
24006 7533 Acked Read 1 Sent Done Done false
24007 7534 Acked Read 2 Sent Done Done false
24008 7535 Acked Write 8 Sent Done Done false
24009 7536 Acked Read 6 Sent Done Done false
24010 7537 Acked Flush 0 Sent Done Done false
24011 7538 Acked Read 10 Sent Done Done false
24012 7539 Acked Read 3 Sent Done Done false
24013 7540 Acked Write 9 Sent Done Done false
24014 7541 Acked Flush 0 Sent Done Done false
24015 7542 Acked Write 7 Sent Done Done false
24016 7543 Acked Read 5 Sent Done Done false
24017 7544 Acked Write 3 Sent Done Done false
24018 7545 Acked Read 2 Sent Done Done false
24019 7546 Acked Read 7 Sent Done Done false
24020 7547 Acked Write 1 Sent Done Done false
24021 7548 Acked Read 6 Sent Done Done false
24022 7549 Acked Read 6 Sent Done Done false
24023 7550 Acked Write 8 Sent Done Done false
24024 7551 Acked Flush 0 Sent Done Done false
24025 7552 Acked Read 6 Sent Done Done false
24026 7553 Acked Read 2 Sent Done Done false
24027 7554 Acked Read 7 Sent Done Done false
24028 7555 Acked Write 6 Sent Done Done false
24029 7556 Acked Write 3 Sent Done Done false
24030 7557 Acked Write 7 Sent Done Done false
24031 7558 Acked Flush 0 Sent Done Done false
24032 7559 Acked Write 5 Sent Done Done false
24033 7560 Acked Flush 0 Sent Done Done false
24034 7561 Acked Read 8 Sent Done Done false
24035 7562 Acked Read 2 Sent Done Done false
24036 7563 Acked Read 5 Sent Done Done false
24037 7564 Acked Write 3 Sent Done Done false
24038 7565 Acked Read 1 Sent Done Done false
24039 7566 Acked Write 10 Sent Done Done false
24040 7567 Acked Read 2 Sent Done Done false
24041 7568 Acked Write 8 Sent Done Done false
24042 7569 Acked Read 9 Sent Done Done false
24043 7570 Acked Read 9 Sent Done Done false
24044 7571 Acked Read 5 Sent Done Done false
24045 7572 Acked Read 9 Sent Done Done false
24046 7573 Acked Read 4 Sent Done Done false
24047 7574 Acked Read 2 Sent Done Done false
24048 7575 Acked Write 8 Sent Done Done false
24049 7576 Acked Flush 0 Sent Done Done false
24050 7577 Acked Flush 0 Sent Done Done false
24051 7578 Acked Read 6 Sent Done Done false
24052 7579 Acked Read 8 Sent Done Done false
24053 7580 Acked Write 2 Sent Done Done false
24054 7581 Acked Read 7 Sent Done Done false
24055 7582 Acked Read 2 Sent Done Done false
24056 7583 Acked Read 6 Sent Done Done false
24057 7584 Acked Read 5 Sent Done Done false
24058 7585 Acked Write 5 Sent Done Done false
24059 7586 Acked Read 6 Sent Done Done false
24060 7587 Acked Write 1 Sent Done Done false
24061 7588 Acked Write 6 Sent Done Done false
24062 7589 Acked Write 4 Sent Done Done false
24063 7590 Acked Read 9 Sent Done Done false
24064 7591 Acked Read 10 Sent Done Done false
24065 7592 Acked Read 4 Sent Done Done false
24066 7593 Acked Flush 0 Sent Done Done false
24067 7594 Acked Read 5 Sent Done Done false
24068 7595 Acked Read 6 Sent Done Done false
24069 7596 Acked Read 7 Sent Done Done false
24070 7597 Acked Read 2 Sent Done Done false
24071 7598 Acked Write 10 Sent Done Done false
24072 7599 Acked Read 5 Sent Done Done false
24073 7600 Acked Read 4 Sent Done Done false
24074 7601 Acked Read 9 Sent Done Done false
24075 7602 Acked Read 7 Sent Done Done false
24076 7603 Acked Read 3 Sent Done Done false
24077 7604 Acked Read 7 Sent Done Done false
24078 7605 Acked Flush 0 Sent Done Done false
24079 7606 Acked Write 7 Sent Done Done false
24080 7607 Acked Write 6 Sent Done Done false
24081 7608 Acked Read 3 Sent Done Done false
24082 7609 Acked Flush 0 Sent Done Done false
24083 7610 Acked Read 7 Sent Done Done false
24084 7611 Acked Write 7 Sent Done Done false
24085 7612 Acked Read 10 Sent Done Done false
24086 7613 Acked Read 2 Sent Done Done false
24087 7614 Acked Read 5 Sent Done Done false
24088 7615 Acked Read 10 Sent Done Done false
24089 7616 Acked Flush 0 Sent Done Done false
24090 7617 Acked Read 8 Sent Done Done false
24091 7618 Acked Write 1 Sent Done Done false
24092 7619 Acked Read 8 Sent Done Done false
24093 7620 Acked Flush 0 Sent Done Done false
24094 7621 Acked Write 10 Sent Done Done false
24095 7622 Acked Write 6 Sent Done Done false
24096 7623 Acked Write 10 Sent Done Done false
24097 7624 Acked Read 1 Sent Done Done false
24098 7625 Acked Read 8 Sent Done Done false
24099 7626 Acked Write 6 Sent Done Done false
24100 7627 Acked Read 10 Sent Done Done false
24101 7628 Acked Read 8 Sent Done Done false
24102 7629 Acked Write 9 Sent Done Done false
24103 7630 Acked Read 10 Sent Done Done false
24104 7631 Acked Read 1 Sent Done Done false
24105 7632 Acked Write 6 Sent Done Done false
24106 7633 Acked Read 9 Sent Done Done false
24107 7634 Acked Write 1 Sent Done Done false
24108 7635 Acked Read 6 Sent Done Done false
24109 7636 Acked Read 7 Sent Done Done false
24110 7637 Acked Read 6 Sent Done Done false
24111 7638 Acked Write 8 Sent Done Done false
24112 7639 Acked Flush 0 Sent Done Done false
24113 7640 Acked Read 7 Sent Done Done false
24114 7641 Acked Read 10 Sent Done Done false
24115 7642 Acked Read 8 Sent Done Done false
24116 7643 Acked Flush 0 Sent Done Done false
24117 7644 Acked Read 8 Sent Done Done false
24118 7645 Acked Read 2 Sent Done Done false
24119 7646 Acked Read 1 Sent Done Done false
24120 7647 Acked Flush 0 Sent Done Done false
24121 STATES DS:0 DS:1 DS:2 TOTAL
24122 Sent 302 0 0 302
24123 Done 0 302 302 604
24124 Skipped 0 0 0 0
24125 Error 0 0 0 0
24126 Last Flush: 7345 7647 7647
24127 Downstairs last five completed: 7345 7344 7343 7342 7341
24128 Upstairs last five completed: 7647 7646 7645 7644 7643
24129 CLIENT: Up:0 ds:302 act:2
241302024-11-01T16:55:32.542ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
241312024-11-01T16:55:32.542ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
241322024-11-01T16:55:32.542ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
241332024-11-01T16:55:32.542ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
241342024-11-01T16:55:32.543ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
24135 [8810] got start action
24136 [8810] I am going to restart
24137 Starting downstairs at port 8810
24138 Make output file at "/tmp/dsc/downstairs-8810.txt"
24139 [8810][0] reports Starting
24140 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1220)
24141 [8810][0] reports Running
24142 ----------------------------------------------------------------
24143 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24144 States: Offline Active Active
24145 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24146 7346 Acked Read 4 Sent Done Done false
24147 7347 Acked Read 8 Sent Done Done false
24148 7348 Acked Write 9 Sent Done Done false
24149 7349 Acked Read 1 Sent Done Done false
24150 7350 Acked Read 5 Sent Done Done false
24151 7351 Acked Write 10 Sent Done Done false
24152 7352 Acked Flush 0 Sent Done Done false
24153 7353 Acked Read 4 Sent Done Done false
24154 7354 Acked Write 2 Sent Done Done false
24155 7355 Acked Read 10 Sent Done Done false
24156 7356 Acked Read 9 Sent Done Done false
24157 7357 Acked Flush 0 Sent Done Done false
24158 7358 Acked Read 5 Sent Done Done false
24159 7359 Acked Write 7 Sent Done Done false
24160 7360 Acked Write 6 Sent Done Done false
24161 7361 Acked Write 4 Sent Done Done false
24162 7362 Acked Read 2 Sent Done Done false
24163 7363 Acked Write 4 Sent Done Done false
24164 7364 Acked Write 3 Sent Done Done false
24165 7365 Acked Read 10 Sent Done Done false
24166 7366 Acked Write 3 Sent Done Done false
24167 7367 Acked Read 4 Sent Done Done false
24168 7368 Acked Write 8 Sent Done Done false
24169 7369 Acked Read 4 Sent Done Done false
24170 7370 Acked Flush 0 Sent Done Done false
24171 7371 Acked Read 7 Sent Done Done false
24172 7372 Acked Write 8 Sent Done Done false
24173 7373 Acked Read 2 Sent Done Done false
24174 7374 Acked Read 1 Sent Done Done false
24175 7375 Acked Read 10 Sent Done Done false
24176 7376 Acked Read 2 Sent Done Done false
24177 7377 Acked Write 10 Sent Done Done false
24178 7378 Acked Read 6 Sent Done Done false
24179 7379 Acked Write 10 Sent Done Done false
24180 7380 Acked Read 8 Sent Done Done false
24181 7381 Acked Write 4 Sent Done Done false
24182 7382 Acked Read 5 Sent Done Done false
24183 7383 Acked Read 2 Sent Done Done false
24184 7384 Acked Write 2 Sent Done Done false
24185 7385 Acked Flush 0 Sent Done Done false
24186 7386 Acked Write 7 Sent Done Done false
24187 7387 Acked Write 4 Sent Done Done false
24188 7388 Acked Read 4 Sent Done Done false
24189 7389 Acked Flush 0 Sent Done Done false
24190 7390 Acked Write 5 Sent Done Done false
24191 7391 Acked Read 5 Sent Done Done false
24192 7392 Acked Read 9 Sent Done Done false
24193 7393 Acked Write 3 Sent Done Done false
24194 7394 Acked Read 7 Sent Done Done false
24195 7395 Acked Write 2 Sent Done Done false
24196 7396 Acked Read 8 Sent Done Done false
24197 7397 Acked Read 4 Sent Done Done false
24198 7398 Acked Write 5 Sent Done Done false
24199 7399 Acked Read 10 Sent Done Done false
24200 7400 Acked Read 7 Sent Done Done false
24201 7401 Acked Read 10 Sent Done Done false
24202 7402 Acked Write 8 Sent Done Done false
24203 7403 Acked Flush 0 Sent Done Done false
24204 7404 Acked Write 1 Sent Done Done false
24205 7405 Acked Write 1 Sent Done Done false
24206 7406 Acked Read 9 Sent Done Done false
24207 7407 Acked Read 3 Sent Done Done false
24208 7408 Acked Read 9 Sent Done Done false
24209 7409 Acked Read 5 Sent Done Done false
24210 7410 Acked Write 10 Sent Done Done false
24211 7411 Acked Read 2 Sent Done Done false
24212 7412 Acked Write 7 Sent Done Done false
24213 7413 Acked Read 2 Sent Done Done false
24214 7414 Acked Read 9 Sent Done Done false
24215 7415 Acked Write 5 Sent Done Done false
24216 7416 Acked Flush 0 Sent Done Done false
24217 7417 Acked Write 2 Sent Done Done false
24218 7418 Acked Write 2 Sent Done Done false
24219 7419 Acked Read 8 Sent Done Done false
24220 7420 Acked Flush 0 Sent Done Done false
24221 7421 Acked Read 8 Sent Done Done false
24222 7422 Acked Write 5 Sent Done Done false
24223 7423 Acked Read 4 Sent Done Done false
24224 7424 Acked Read 7 Sent Done Done false
24225 7425 Acked Read 2 Sent Done Done false
24226 7426 Acked Read 9 Sent Done Done false
24227 7427 Acked Read 3 Sent Done Done false
24228 7428 Acked Read 3 Sent Done Done false
24229 7429 Acked Read 4 Sent Done Done false
24230 7430 Acked Write 2 Sent Done Done false
24231 7431 Acked Read 2 Sent Done Done false
24232 7432 Acked Write 9 Sent Done Done false
24233 7433 Acked Flush 0 Sent Done Done false
24234 7434 Acked Flush 0 Sent Done Done false
24235 7435 Acked Write 2 Sent Done Done false
24236 7436 Acked Read 9 Sent Done Done false
24237 7437 Acked Read 2 Sent Done Done false
24238 7438 Acked Write 8 Sent Done Done false
24239 7439 Acked Write 10 Sent Done Done false
24240 7440 Acked Read 8 Sent Done Done false
24241 7441 Acked Read 7 Sent Done Done false
24242 7442 Acked Write 6 Sent Done Done false
24243 7443 Acked Read 9 Sent Done Done false
24244 7444 Acked Flush 0 Sent Done Done false
24245 7445 Acked Write 10 Sent Done Done false
24246 7446 Acked Write 4 Sent Done Done false
24247 7447 Acked Read 6 Sent Done Done false
24248 7448 Acked Write 8 Sent Done Done false
24249 7449 Acked Write 7 Sent Done Done false
24250 7450 Acked Write 1 Sent Done Done false
24251 7451 Acked Write 5 Sent Done Done false
24252 7452 Acked Read 1 Sent Done Done false
24253 7453 Acked Write 9 Sent Done Done false
24254 7454 Acked Write 6 Sent Done Done false
24255 7455 Acked Read 4 Sent Done Done false
24256 7456 Acked Write 10 Sent Done Done false
24257 7457 Acked Write 6 Sent Done Done false
24258 7458 Acked Flush 0 Sent Done Done false
24259 7459 Acked Write 6 Sent Done Done false
24260 7460 Acked Write 3 Sent Done Done false
24261 7461 Acked Write 6 Sent Done Done false
24262 7462 Acked Read 1 Sent Done Done false
24263 7463 Acked Read 8 Sent Done Done false
24264 7464 Acked Read 9 Sent Done Done false
24265 7465 Acked Read 8 Sent Done Done false
24266 7466 Acked Read 5 Sent Done Done false
24267 7467 Acked Write 4 Sent Done Done false
24268 7468 Acked Write 2 Sent Done Done false
24269 7469 Acked Write 7 Sent Done Done false
24270 7470 Acked Read 2 Sent Done Done false
24271 7471 Acked Write 6 Sent Done Done false
24272 7472 Acked Read 1 Sent Done Done false
24273 7473 Acked Read 4 Sent Done Done false
24274 7474 Acked Read 10 Sent Done Done false
24275 7475 Acked Read 9 Sent Done Done false
24276 7476 Acked Read 5 Sent Done Done false
24277 7477 Acked Write 3 Sent Done Done false
24278 7478 Acked Write 4 Sent Done Done false
24279 7479 Acked Read 4 Sent Done Done false
24280 7480 Acked Write 8 Sent Done Done false
24281 7481 Acked Write 5 Sent Done Done false
24282 7482 Acked Read 6 Sent Done Done false
24283 7483 Acked Read 1 Sent Done Done false
24284 7484 Acked Write 5 Sent Done Done false
24285 7485 Acked Write 8 Sent Done Done false
24286 7486 Acked Read 8 Sent Done Done false
24287 7487 Acked Flush 0 Sent Done Done false
24288 7488 Acked Write 10 Sent Done Done false
24289 7489 Acked Read 8 Sent Done Done false
24290 7490 Acked Write 3 Sent Done Done false
24291 7491 Acked Flush 0 Sent Done Done false
24292 7492 Acked Write 2 Sent Done Done false
24293 7493 Acked Read 8 Sent Done Done false
24294 7494 Acked Write 7 Sent Done Done false
24295 7495 Acked Write 5 Sent Done Done false
24296 7496 Acked Read 1 Sent Done Done false
24297 7497 Acked Flush 0 Sent Done Done false
24298 7498 Acked Read 10 Sent Done Done false
24299 7499 Acked Read 2 Sent Done Done false
24300 7500 Acked Write 8 Sent Done Done false
24301 7501 Acked Read 10 Sent Done Done false
24302 7502 Acked Flush 0 Sent Done Done false
24303 7503 Acked Write 3 Sent Done Done false
24304 7504 Acked Write 1 Sent Done Done false
24305 7505 Acked Read 4 Sent Done Done false
24306 7506 Acked Read 6 Sent Done Done false
24307 7507 Acked Write 6 Sent Done Done false
24308 7508 Acked Write 8 Sent Done Done false
24309 7509 Acked Write 1 Sent Done Done false
24310 7510 Acked Write 10 Sent Done Done false
24311 7511 Acked Read 3 Sent Done Done false
24312 7512 Acked Read 3 Sent Done Done false
24313 7513 Acked Read 4 Sent Done Done false
24314 7514 Acked Read 3 Sent Done Done false
24315 7515 Acked Read 10 Sent Done Done false
24316 7516 Acked Flush 0 Sent Done Done false
24317 7517 Acked Read 7 Sent Done Done false
24318 7518 Acked Write 7 Sent Done Done false
24319 7519 Acked Read 9 Sent Done Done false
24320 7520 Acked Read 6 Sent Done Done false
24321 7521 Acked Write 2 Sent Done Done false
24322 7522 Acked Write 5 Sent Done Done false
24323 7523 Acked Read 3 Sent Done Done false
24324 7524 Acked Write 9 Sent Done Done false
24325 7525 Acked Read 4 Sent Done Done false
24326 7526 Acked Read 2 Sent Done Done false
24327 7527 Acked Write 10 Sent Done Done false
24328 7528 Acked Read 7 Sent Done Done false
24329 7529 Acked Read 1 Sent Done Done false
24330 7530 Acked Write 9 Sent Done Done false
24331 7531 Acked Write 9 Sent Done Done false
24332 7532 Acked Write 1 Sent Done Done false
24333 7533 Acked Read 1 Sent Done Done false
24334 7534 Acked Read 2 Sent Done Done false
24335 7535 Acked Write 8 Sent Done Done false
24336 7536 Acked Read 6 Sent Done Done false
24337 7537 Acked Flush 0 Sent Done Done false
24338 7538 Acked Read 10 Sent Done Done false
24339 7539 Acked Read 3 Sent Done Done false
24340 7540 Acked Write 9 Sent Done Done false
24341 7541 Acked Flush 0 Sent Done Done false
24342 7542 Acked Write 7 Sent Done Done false
24343 7543 Acked Read 5 Sent Done Done false
24344 7544 Acked Write 3 Sent Done Done false
24345 7545 Acked Read 2 Sent Done Done false
24346 7546 Acked Read 7 Sent Done Done false
24347 7547 Acked Write 1 Sent Done Done false
24348 7548 Acked Read 6 Sent Done Done false
24349 7549 Acked Read 6 Sent Done Done false
24350 7550 Acked Write 8 Sent Done Done false
24351 7551 Acked Flush 0 Sent Done Done false
24352 7552 Acked Read 6 Sent Done Done false
24353 7553 Acked Read 2 Sent Done Done false
24354 7554 Acked Read 7 Sent Done Done false
24355 7555 Acked Write 6 Sent Done Done false
24356 7556 Acked Write 3 Sent Done Done false
24357 7557 Acked Write 7 Sent Done Done false
24358 7558 Acked Flush 0 Sent Done Done false
24359 7559 Acked Write 5 Sent Done Done false
24360 7560 Acked Flush 0 Sent Done Done false
24361 7561 Acked Read 8 Sent Done Done false
24362 7562 Acked Read 2 Sent Done Done false
24363 7563 Acked Read 5 Sent Done Done false
24364 7564 Acked Write 3 Sent Done Done false
24365 7565 Acked Read 1 Sent Done Done false
24366 7566 Acked Write 10 Sent Done Done false
24367 7567 Acked Read 2 Sent Done Done false
24368 7568 Acked Write 8 Sent Done Done false
24369 7569 Acked Read 9 Sent Done Done false
24370 7570 Acked Read 9 Sent Done Done false
24371 7571 Acked Read 5 Sent Done Done false
24372 7572 Acked Read 9 Sent Done Done false
24373 7573 Acked Read 4 Sent Done Done false
24374 7574 Acked Read 2 Sent Done Done false
24375 7575 Acked Write 8 Sent Done Done false
24376 7576 Acked Flush 0 Sent Done Done false
24377 7577 Acked Flush 0 Sent Done Done false
24378 7578 Acked Read 6 Sent Done Done false
24379 7579 Acked Read 8 Sent Done Done false
24380 7580 Acked Write 2 Sent Done Done false
24381 7581 Acked Read 7 Sent Done Done false
24382 7582 Acked Read 2 Sent Done Done false
24383 7583 Acked Read 6 Sent Done Done false
24384 7584 Acked Read 5 Sent Done Done false
24385 7585 Acked Write 5 Sent Done Done false
24386 7586 Acked Read 6 Sent Done Done false
24387 7587 Acked Write 1 Sent Done Done false
24388 7588 Acked Write 6 Sent Done Done false
24389 7589 Acked Write 4 Sent Done Done false
24390 7590 Acked Read 9 Sent Done Done false
24391 7591 Acked Read 10 Sent Done Done false
24392 7592 Acked Read 4 Sent Done Done false
24393 7593 Acked Flush 0 Sent Done Done false
24394 7594 Acked Read 5 Sent Done Done false
24395 7595 Acked Read 6 Sent Done Done false
24396 7596 Acked Read 7 Sent Done Done false
24397 7597 Acked Read 2 Sent Done Done false
24398 7598 Acked Write 10 Sent Done Done false
24399 7599 Acked Read 5 Sent Done Done false
24400 7600 Acked Read 4 Sent Done Done false
24401 7601 Acked Read 9 Sent Done Done false
24402 7602 Acked Read 7 Sent Done Done false
24403 7603 Acked Read 3 Sent Done Done false
24404 7604 Acked Read 7 Sent Done Done false
24405 7605 Acked Flush 0 Sent Done Done false
24406 7606 Acked Write 7 Sent Done Done false
24407 7607 Acked Write 6 Sent Done Done false
24408 7608 Acked Read 3 Sent Done Done false
24409 7609 Acked Flush 0 Sent Done Done false
24410 7610 Acked Read 7 Sent Done Done false
24411 7611 Acked Write 7 Sent Done Done false
24412 7612 Acked Read 10 Sent Done Done false
24413 7613 Acked Read 2 Sent Done Done false
24414 7614 Acked Read 5 Sent Done Done false
24415 7615 Acked Read 10 Sent Done Done false
24416 7616 Acked Flush 0 Sent Done Done false
24417 7617 Acked Read 8 Sent Done Done false
24418 7618 Acked Write 1 Sent Done Done false
24419 7619 Acked Read 8 Sent Done Done false
24420 7620 Acked Flush 0 Sent Done Done false
24421 7621 Acked Write 10 Sent Done Done false
24422 7622 Acked Write 6 Sent Done Done false
24423 7623 Acked Write 10 Sent Done Done false
24424 7624 Acked Read 1 Sent Done Done false
24425 7625 Acked Read 8 Sent Done Done false
24426 7626 Acked Write 6 Sent Done Done false
24427 7627 Acked Read 10 Sent Done Done false
24428 7628 Acked Read 8 Sent Done Done false
24429 7629 Acked Write 9 Sent Done Done false
24430 7630 Acked Read 10 Sent Done Done false
24431 7631 Acked Read 1 Sent Done Done false
24432 7632 Acked Write 6 Sent Done Done false
24433 7633 Acked Read 9 Sent Done Done false
24434 7634 Acked Write 1 Sent Done Done false
24435 7635 Acked Read 6 Sent Done Done false
24436 7636 Acked Read 7 Sent Done Done false
24437 7637 Acked Read 6 Sent Done Done false
24438 7638 Acked Write 8 Sent Done Done false
24439 7639 Acked Flush 0 Sent Done Done false
24440 7640 Acked Read 7 Sent Done Done false
24441 7641 Acked Read 10 Sent Done Done false
24442 7642 Acked Read 8 Sent Done Done false
24443 7643 Acked Flush 0 Sent Done Done false
24444 7644 Acked Read 8 Sent Done Done false
24445 7645 Acked Read 2 Sent Done Done false
24446 7646 Acked Read 1 Sent Done Done false
24447 7647 Acked Flush 0 Sent Done Done false
24448 STATES DS:0 DS:1 DS:2 TOTAL
24449 Sent 302 0 0 302
24450 Done 0 302 302 604
24451 Skipped 0 0 0 0
24452 Error 0 0 0 0
24453 Last Flush: 7345 7647 7647
24454 Downstairs last five completed: 7345 7344 7343 7342 7341
24455 Upstairs last five completed: 7647 7646 7645 7644 7643
24456 CLIENT: Up:0 ds:302 act:2
24457 ----------------------------------------------------------------
24458 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24459 States: Offline Active Active
24460 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24461 7346 Acked Read 4 Sent Done Done false
24462 7347 Acked Read 8 Sent Done Done false
24463 7348 Acked Write 9 Sent Done Done false
24464 7349 Acked Read 1 Sent Done Done false
24465 7350 Acked Read 5 Sent Done Done false
24466 7351 Acked Write 10 Sent Done Done false
24467 7352 Acked Flush 0 Sent Done Done false
24468 7353 Acked Read 4 Sent Done Done false
24469 7354 Acked Write 2 Sent Done Done false
24470 7355 Acked Read 10 Sent Done Done false
24471 7356 Acked Read 9 Sent Done Done false
24472 7357 Acked Flush 0 Sent Done Done false
24473 7358 Acked Read 5 Sent Done Done false
24474 7359 Acked Write 7 Sent Done Done false
24475 7360 Acked Write 6 Sent Done Done false
24476 7361 Acked Write 4 Sent Done Done false
24477 7362 Acked Read 2 Sent Done Done false
24478 7363 Acked Write 4 Sent Done Done false
24479 7364 Acked Write 3 Sent Done Done false
24480 7365 Acked Read 10 Sent Done Done false
24481 7366 Acked Write 3 Sent Done Done false
24482 7367 Acked Read 4 Sent Done Done false
24483 7368 Acked Write 8 Sent Done Done false
24484 7369 Acked Read 4 Sent Done Done false
24485 7370 Acked Flush 0 Sent Done Done false
24486 7371 Acked Read 7 Sent Done Done false
24487 7372 Acked Write 8 Sent Done Done false
24488 7373 Acked Read 2 Sent Done Done false
24489 7374 Acked Read 1 Sent Done Done false
24490 7375 Acked Read 10 Sent Done Done false
24491 7376 Acked Read 2 Sent Done Done false
24492 7377 Acked Write 10 Sent Done Done false
24493 7378 Acked Read 6 Sent Done Done false
24494 7379 Acked Write 10 Sent Done Done false
24495 7380 Acked Read 8 Sent Done Done false
24496 7381 Acked Write 4 Sent Done Done false
24497 7382 Acked Read 5 Sent Done Done false
24498 7383 Acked Read 2 Sent Done Done false
24499 7384 Acked Write 2 Sent Done Done false
24500 7385 Acked Flush 0 Sent Done Done false
24501 7386 Acked Write 7 Sent Done Done false
24502 7387 Acked Write 4 Sent Done Done false
24503 7388 Acked Read 4 Sent Done Done false
24504 7389 Acked Flush 0 Sent Done Done false
24505 7390 Acked Write 5 Sent Done Done false
24506 7391 Acked Read 5 Sent Done Done false
24507 7392 Acked Read 9 Sent Done Done false
24508 7393 Acked Write 3 Sent Done Done false
24509 7394 Acked Read 7 Sent Done Done false
24510 7395 Acked Write 2 Sent Done Done false
24511 7396 Acked Read 8 Sent Done Done false
24512 7397 Acked Read 4 Sent Done Done false
24513 7398 Acked Write 5 Sent Done Done false
24514 7399 Acked Read 10 Sent Done Done false
24515 7400 Acked Read 7 Sent Done Done false
24516 7401 Acked Read 10 Sent Done Done false
24517 7402 Acked Write 8 Sent Done Done false
24518 7403 Acked Flush 0 Sent Done Done false
24519 7404 Acked Write 1 Sent Done Done false
24520 7405 Acked Write 1 Sent Done Done false
24521 7406 Acked Read 9 Sent Done Done false
24522 7407 Acked Read 3 Sent Done Done false
24523 7408 Acked Read 9 Sent Done Done false
24524 7409 Acked Read 5 Sent Done Done false
24525 7410 Acked Write 10 Sent Done Done false
24526 7411 Acked Read 2 Sent Done Done false
24527 7412 Acked Write 7 Sent Done Done false
24528 7413 Acked Read 2 Sent Done Done false
24529 7414 Acked Read 9 Sent Done Done false
24530 7415 Acked Write 5 Sent Done Done false
24531 7416 Acked Flush 0 Sent Done Done false
24532 7417 Acked Write 2 Sent Done Done false
24533 7418 Acked Write 2 Sent Done Done false
24534 7419 Acked Read 8 Sent Done Done false
24535 7420 Acked Flush 0 Sent Done Done false
24536 7421 Acked Read 8 Sent Done Done false
24537 7422 Acked Write 5 Sent Done Done false
24538 7423 Acked Read 4 Sent Done Done false
24539 7424 Acked Read 7 Sent Done Done false
24540 7425 Acked Read 2 Sent Done Done false
24541 7426 Acked Read 9 Sent Done Done false
24542 7427 Acked Read 3 Sent Done Done false
24543 7428 Acked Read 3 Sent Done Done false
24544 7429 Acked Read 4 Sent Done Done false
24545 7430 Acked Write 2 Sent Done Done false
24546 7431 Acked Read 2 Sent Done Done false
24547 7432 Acked Write 9 Sent Done Done false
24548 7433 Acked Flush 0 Sent Done Done false
24549 7434 Acked Flush 0 Sent Done Done false
24550 7435 Acked Write 2 Sent Done Done false
24551 7436 Acked Read 9 Sent Done Done false
24552 7437 Acked Read 2 Sent Done Done false
24553 7438 Acked Write 8 Sent Done Done false
24554 7439 Acked Write 10 Sent Done Done false
24555 7440 Acked Read 8 Sent Done Done false
24556 7441 Acked Read 7 Sent Done Done false
24557 7442 Acked Write 6 Sent Done Done false
24558 7443 Acked Read 9 Sent Done Done false
24559 7444 Acked Flush 0 Sent Done Done false
24560 7445 Acked Write 10 Sent Done Done false
24561 7446 Acked Write 4 Sent Done Done false
24562 7447 Acked Read 6 Sent Done Done false
24563 7448 Acked Write 8 Sent Done Done false
24564 7449 Acked Write 7 Sent Done Done false
24565 7450 Acked Write 1 Sent Done Done false
24566 7451 Acked Write 5 Sent Done Done false
24567 7452 Acked Read 1 Sent Done Done false
24568 7453 Acked Write 9 Sent Done Done false
24569 7454 Acked Write 6 Sent Done Done false
24570 7455 Acked Read 4 Sent Done Done false
24571 7456 Acked Write 10 Sent Done Done false
24572 7457 Acked Write 6 Sent Done Done false
24573 7458 Acked Flush 0 Sent Done Done false
24574 7459 Acked Write 6 Sent Done Done false
24575 7460 Acked Write 3 Sent Done Done false
24576 7461 Acked Write 6 Sent Done Done false
24577 7462 Acked Read 1 Sent Done Done false
24578 7463 Acked Read 8 Sent Done Done false
24579 7464 Acked Read 9 Sent Done Done false
24580 7465 Acked Read 8 Sent Done Done false
24581 7466 Acked Read 5 Sent Done Done false
24582 7467 Acked Write 4 Sent Done Done false
24583 7468 Acked Write 2 Sent Done Done false
24584 7469 Acked Write 7 Sent Done Done false
24585 7470 Acked Read 2 Sent Done Done false
24586 7471 Acked Write 6 Sent Done Done false
24587 7472 Acked Read 1 Sent Done Done false
24588 7473 Acked Read 4 Sent Done Done false
24589 7474 Acked Read 10 Sent Done Done false
24590 7475 Acked Read 9 Sent Done Done false
24591 7476 Acked Read 5 Sent Done Done false
24592 7477 Acked Write 3 Sent Done Done false
24593 7478 Acked Write 4 Sent Done Done false
24594 7479 Acked Read 4 Sent Done Done false
24595 7480 Acked Write 8 Sent Done Done false
24596 7481 Acked Write 5 Sent Done Done false
24597 7482 Acked Read 6 Sent Done Done false
24598 7483 Acked Read 1 Sent Done Done false
24599 7484 Acked Write 5 Sent Done Done false
24600 7485 Acked Write 8 Sent Done Done false
24601 7486 Acked Read 8 Sent Done Done false
24602 7487 Acked Flush 0 Sent Done Done false
24603 7488 Acked Write 10 Sent Done Done false
24604 7489 Acked Read 8 Sent Done Done false
24605 7490 Acked Write 3 Sent Done Done false
24606 7491 Acked Flush 0 Sent Done Done false
24607 7492 Acked Write 2 Sent Done Done false
24608 7493 Acked Read 8 Sent Done Done false
24609 7494 Acked Write 7 Sent Done Done false
24610 7495 Acked Write 5 Sent Done Done false
24611 7496 Acked Read 1 Sent Done Done false
24612 7497 Acked Flush 0 Sent Done Done false
24613 7498 Acked Read 10 Sent Done Done false
24614 7499 Acked Read 2 Sent Done Done false
24615 7500 Acked Write 8 Sent Done Done false
24616 7501 Acked Read 10 Sent Done Done false
24617 7502 Acked Flush 0 Sent Done Done false
24618 7503 Acked Write 3 Sent Done Done false
24619 7504 Acked Write 1 Sent Done Done false
24620 7505 Acked Read 4 Sent Done Done false
24621 7506 Acked Read 6 Sent Done Done false
24622 7507 Acked Write 6 Sent Done Done false
24623 7508 Acked Write 8 Sent Done Done false
24624 7509 Acked Write 1 Sent Done Done false
24625 7510 Acked Write 10 Sent Done Done false
24626 7511 Acked Read 3 Sent Done Done false
24627 7512 Acked Read 3 Sent Done Done false
24628 7513 Acked Read 4 Sent Done Done false
24629 7514 Acked Read 3 Sent Done Done false
24630 7515 Acked Read 10 Sent Done Done false
24631 7516 Acked Flush 0 Sent Done Done false
24632 7517 Acked Read 7 Sent Done Done false
24633 7518 Acked Write 7 Sent Done Done false
24634 7519 Acked Read 9 Sent Done Done false
24635 7520 Acked Read 6 Sent Done Done false
24636 7521 Acked Write 2 Sent Done Done false
24637 7522 Acked Write 5 Sent Done Done false
24638 7523 Acked Read 3 Sent Done Done false
24639 7524 Acked Write 9 Sent Done Done false
24640 7525 Acked Read 4 Sent Done Done false
24641 7526 Acked Read 2 Sent Done Done false
24642 7527 Acked Write 10 Sent Done Done false
24643 7528 Acked Read 7 Sent Done Done false
24644 7529 Acked Read 1 Sent Done Done false
24645 7530 Acked Write 9 Sent Done Done false
24646 7531 Acked Write 9 Sent Done Done false
24647 7532 Acked Write 1 Sent Done Done false
24648 7533 Acked Read 1 Sent Done Done false
24649 7534 Acked Read 2 Sent Done Done false
24650 7535 Acked Write 8 Sent Done Done false
24651 7536 Acked Read 6 Sent Done Done false
24652 7537 Acked Flush 0 Sent Done Done false
24653 7538 Acked Read 10 Sent Done Done false
24654 7539 Acked Read 3 Sent Done Done false
24655 7540 Acked Write 9 Sent Done Done false
24656 7541 Acked Flush 0 Sent Done Done false
24657 7542 Acked Write 7 Sent Done Done false
24658 7543 Acked Read 5 Sent Done Done false
24659 7544 Acked Write 3 Sent Done Done false
24660 7545 Acked Read 2 Sent Done Done false
24661 7546 Acked Read 7 Sent Done Done false
24662 7547 Acked Write 1 Sent Done Done false
24663 7548 Acked Read 6 Sent Done Done false
24664 7549 Acked Read 6 Sent Done Done false
24665 7550 Acked Write 8 Sent Done Done false
24666 7551 Acked Flush 0 Sent Done Done false
24667 7552 Acked Read 6 Sent Done Done false
24668 7553 Acked Read 2 Sent Done Done false
24669 7554 Acked Read 7 Sent Done Done false
24670 7555 Acked Write 6 Sent Done Done false
24671 7556 Acked Write 3 Sent Done Done false
24672 7557 Acked Write 7 Sent Done Done false
24673 7558 Acked Flush 0 Sent Done Done false
24674 7559 Acked Write 5 Sent Done Done false
24675 7560 Acked Flush 0 Sent Done Done false
24676 7561 Acked Read 8 Sent Done Done false
24677 7562 Acked Read 2 Sent Done Done false
24678 7563 Acked Read 5 Sent Done Done false
24679 7564 Acked Write 3 Sent Done Done false
24680 7565 Acked Read 1 Sent Done Done false
24681 7566 Acked Write 10 Sent Done Done false
24682 7567 Acked Read 2 Sent Done Done false
24683 7568 Acked Write 8 Sent Done Done false
24684 7569 Acked Read 9 Sent Done Done false
24685 7570 Acked Read 9 Sent Done Done false
24686 7571 Acked Read 5 Sent Done Done false
24687 7572 Acked Read 9 Sent Done Done false
24688 7573 Acked Read 4 Sent Done Done false
24689 7574 Acked Read 2 Sent Done Done false
24690 7575 Acked Write 8 Sent Done Done false
24691 7576 Acked Flush 0 Sent Done Done false
24692 7577 Acked Flush 0 Sent Done Done false
24693 7578 Acked Read 6 Sent Done Done false
24694 7579 Acked Read 8 Sent Done Done false
24695 7580 Acked Write 2 Sent Done Done false
24696 7581 Acked Read 7 Sent Done Done false
24697 7582 Acked Read 2 Sent Done Done false
24698 7583 Acked Read 6 Sent Done Done false
24699 7584 Acked Read 5 Sent Done Done false
24700 7585 Acked Write 5 Sent Done Done false
24701 7586 Acked Read 6 Sent Done Done false
24702 7587 Acked Write 1 Sent Done Done false
24703 7588 Acked Write 6 Sent Done Done false
24704 7589 Acked Write 4 Sent Done Done false
24705 7590 Acked Read 9 Sent Done Done false
24706 7591 Acked Read 10 Sent Done Done false
24707 7592 Acked Read 4 Sent Done Done false
24708 7593 Acked Flush 0 Sent Done Done false
24709 7594 Acked Read 5 Sent Done Done false
24710 7595 Acked Read 6 Sent Done Done false
24711 7596 Acked Read 7 Sent Done Done false
24712 7597 Acked Read 2 Sent Done Done false
24713 7598 Acked Write 10 Sent Done Done false
24714 7599 Acked Read 5 Sent Done Done false
24715 7600 Acked Read 4 Sent Done Done false
24716 7601 Acked Read 9 Sent Done Done false
24717 7602 Acked Read 7 Sent Done Done false
24718 7603 Acked Read 3 Sent Done Done false
24719 7604 Acked Read 7 Sent Done Done false
24720 7605 Acked Flush 0 Sent Done Done false
24721 7606 Acked Write 7 Sent Done Done false
24722 7607 Acked Write 6 Sent Done Done false
24723 7608 Acked Read 3 Sent Done Done false
24724 7609 Acked Flush 0 Sent Done Done false
24725 7610 Acked Read 7 Sent Done Done false
24726 7611 Acked Write 7 Sent Done Done false
24727 7612 Acked Read 10 Sent Done Done false
24728 7613 Acked Read 2 Sent Done Done false
24729 7614 Acked Read 5 Sent Done Done false
24730 7615 Acked Read 10 Sent Done Done false
24731 7616 Acked Flush 0 Sent Done Done false
24732 7617 Acked Read 8 Sent Done Done false
24733 7618 Acked Write 1 Sent Done Done false
24734 7619 Acked Read 8 Sent Done Done false
24735 7620 Acked Flush 0 Sent Done Done false
24736 7621 Acked Write 10 Sent Done Done false
24737 7622 Acked Write 6 Sent Done Done false
24738 7623 Acked Write 10 Sent Done Done false
24739 7624 Acked Read 1 Sent Done Done false
24740 7625 Acked Read 8 Sent Done Done false
24741 7626 Acked Write 6 Sent Done Done false
24742 7627 Acked Read 10 Sent Done Done false
24743 7628 Acked Read 8 Sent Done Done false
24744 7629 Acked Write 9 Sent Done Done false
24745 7630 Acked Read 10 Sent Done Done false
24746 7631 Acked Read 1 Sent Done Done false
24747 7632 Acked Write 6 Sent Done Done false
24748 7633 Acked Read 9 Sent Done Done false
24749 7634 Acked Write 1 Sent Done Done false
24750 7635 Acked Read 6 Sent Done Done false
24751 7636 Acked Read 7 Sent Done Done false
24752 7637 Acked Read 6 Sent Done Done false
24753 7638 Acked Write 8 Sent Done Done false
24754 7639 Acked Flush 0 Sent Done Done false
24755 7640 Acked Read 7 Sent Done Done false
24756 7641 Acked Read 10 Sent Done Done false
24757 7642 Acked Read 8 Sent Done Done false
24758 7643 Acked Flush 0 Sent Done Done false
24759 7644 Acked Read 8 Sent Done Done false
24760 7645 Acked Read 2 Sent Done Done false
24761 7646 Acked Read 1 Sent Done Done false
24762 7647 Acked Flush 0 Sent Done Done false
24763 STATES DS:0 DS:1 DS:2 TOTAL
24764 Sent 302 0 0 302
24765 Done 0 302 302 604
24766 Skipped 0 0 0 0
24767 Error 0 0 0 0
24768 Last Flush: 7345 7647 7647
24769 Downstairs last five completed: 7345 7344 7343 7342 7341
24770 Upstairs last five completed: 7647 7646 7645 7644 7643
24771 CLIENT: Up:0 ds:302 act:2
247722024-11-01T16:55:42.544ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
247732024-11-01T16:55:42.544ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:35532) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
247742024-11-01T16:55:42.545ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
247752024-11-01T16:55:42.545ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
247762024-11-01T16:55:42.545ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
247772024-11-01T16:55:42.545ZINFOcrucible: send last flush ID to this DS: 7345 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
247782024-11-01T16:55:42.546ZINFOcrucible: Replied this last flush ID: 7345 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
247792024-11-01T16:55:42.546ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
247802024-11-01T16:55:42.546ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
247812024-11-01T16:55:42.546ZINFOcrucible: [0] client re-new 302 jobs since flush 7345 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
247822024-11-01T16:55:42.546ZINFOcrucible: [0] Replayed 302 jobs since flush: 7345 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
24783 ----------------------------------------------------------------
24784 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24785 Downstairs last five completed: 7647 7646 7645 7644 7643
24786 Upstairs last five completed: 7647 7646 7645 7644 7643
24787 CLIENT: Up:0 ds:0 act:3
24788 Replay: All jobs finished
24789 Main task has work to do, go find it
24790 got dsc Stop(0)
24791 stop 0
24792 [8810] Got stop action so:false kr:false
24793 Nov 01 16:55:42.851 INFO request completed, latency_us: 352, response_code: 204, uri: /stop/cid/0, method: POST, req_id: b25739d5-009b-4410-9966-42886bdf9243, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
247942024-11-01T16:55:42.851ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
247952024-11-01T16:55:42.851ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
247962024-11-01T16:55:42.851ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
247972024-11-01T16:55:42.851ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
24798 Nov 01 16:55:42.851 INFO request completed, latency_us: 136, response_code: 200, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:55:42.851983154Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
24799 uri: /state/cid/0, method: GET, req_id: ca5eea0b-a28c-4140-a5e5-8a41097b4199, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
24800 [8810] Exited with: ExitStatus(unix_wait_status(9))
24801 [8810][0] reports Exit
24802 Nov 01 16:55:46.853 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/0, method: GET, req_id: ce364f6b-cec1-4c2d-aa7a-e3f741d8f20e, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
24803 001/300 Write block 2481 len 40960 data: 4 4 4 5 5 5 6 6 6 6
24804 002/300 Write block 880 len 24576 data: 4 5 6 6 6 5
24805 003/300 Read block 1648 len 28672
24806 004/300 Read block 2667 len 4096
24807 005/300 Write block 1618 len 20480 data: 4 4 4 2 2
24808 006/300 Write block 4441 len 36864 data: 4 3 4 4 3 3 4 4 7
24809 007/300 Write block 330 len 12288 data: 2 2 3
24810 008/300 Read block 2905 len 36864
24811 009/300 Write block 2116 len 20480 data: 3 3 3 4 4
24812 010/300 Read block 1099 len 20480
24813 011/300 Write block 512 len 36864 data: 7 5 6 5 7 7 6 7 7
24814 012/300 Flush
24815 013/300 Read block 2182 len 28672
24816 014/300 Write block 2350 len 8192 data: 3 3
24817 015/300 Read block 2728 len 32768
24818 016/300 Read block 3207 len 36864
24819 017/300 Write block 3734 len 8192 data: 5 5
24820 018/300 Read block 4230 len 40960
24821 019/300 Read block 19 len 28672
24822 020/300 Read block 3318 len 8192
24823 021/300 Write block 1264 len 20480 data: 4 6 6 5 5
24824 022/300 Flush
24825 023/300 Read block 3555 len 28672
24826 024/300 Write block 2872 len 40960 data: 5 5 5 6 6 6 6 5 5 4
24827 025/300 Write block 4045 len 12288 data: 4 5 5
24828 026/300 Read block 1510 len 8192
24829 027/300 Write block 3469 len 12288 data: 5 6 8
24830 028/300 Read block 4496 len 20480
24831 029/300 Read block 538 len 36864
24832 030/300 Read block 113 len 36864
24833 031/300 Write block 2723 len 24576 data: 7 7 6 3 4 4
24834 032/300 Read block 1570 len 12288
24835 033/300 Write block 3060 len 40960 data: 4 4 4 4 3 3 3 4 4 4
24836 034/300 Read block 1807 len 16384
24837 035/300 Read block 854 len 8192
24838 036/300 Write block 4889 len 4096 data: 6
24839 037/300 Read block 2865 len 12288
24840 038/300 Write block 3325 len 4096 data: 4
24841 039/300 Write block 4472 len 16384 data: 2 3 3 3
24842 040/300 Read block 4028 len 12288
24843 041/300 Write block 4728 len 28672 data: 3 5 5 4 3 5 6
24844 042/300 Write block 2164 len 16384 data: 5 5 4 4
24845 043/300 Read block 248 len 16384
24846 044/300 Write block 1825 len 28672 data: 6 5 4 4 3 3 3
24847 045/300 Read block 4519 len 28672
24848 046/300 Read block 4802 len 16384
24849 047/300 Write block 944 len 16384 data: 5 5 6 6
24850 048/300 Read block 4156 len 36864
24851 049/300 Read block 569 len 28672
24852 050/300 Write block 2665 len 16384 data: 4 4 3 3
24853 051/300 Read block 4474 len 8192
24854 052/300 Write block 4617 len 24576 data: 3 2 2 2 2 2
24855 053/300 Read block 749 len 8192
24856 054/300 Read block 1620 len 20480
24857 055/300 Write block 4468 len 28672 data: 3 4 3 3 3 4 4
24858 056/300 Read block 1745 len 32768
24859 057/300 Write block 3800 len 24576 data: 8 6 7 7 8 5
24860 058/300 Read block 2682 len 20480
24861 059/300 Read block 1750 len 32768
24862 060/300 Read block 3206 len 20480
24863 061/300 Read block 4526 len 32768
24864 062/300 Write block 2640 len 20480 data: 6 7 7 7 7
24865 063/300 Read block 788 len 12288
24866 064/300 Read block 1157 len 24576
24867 065/300 Write block 2488 len 8192 data: 7 7
24868 066/300 Read block 1511 len 4096
24869 067/300 Flush
24870 068/300 Read block 125 len 32768
24871 069/300 Flush
24872 070/300 Read block 2707 len 16384
24873 071/300 Read block 1421 len 36864
24874 072/300 Write block 310 len 20480 data: 3 3 3 3 2
24875 073/300 Read block 855 len 24576
24876 074/300 Write block 1666 len 40960 data: 5 5 5 5 5 4 2 4 4 4
24877 075/300 Write block 4390 len 20480 data: 4 2 4 4 3
24878 076/300 Write block 2428 len 12288 data: 4 3 3
24879 077/300 Write block 3739 len 20480 data: 5 5 4 4 4
24880 078/300 Write block 784 len 36864 data: 6 5 3 2 2 2 2 2 3
24881 079/300 Read block 1484 len 24576
24882 080/300 Read block 4269 len 20480
24883 081/300 Write block 2437 len 8192 data: 5 5
24884 082/300 Write block 2030 len 12288 data: 5 5 6
24885 083/300 Flush
24886 084/300 Read block 2465 len 20480
24887 085/300 Read block 2385 len 40960
24888 086/300 Read block 189 len 40960
24889 087/300 Read block 1962 len 20480
24890 088/300 Read block 711 len 32768
24891 089/300 Read block 3658 len 28672
24892 090/300 Read block 4265 len 12288
24893 091/300 Flush
24894 092/300 Write block 425 len 20480 data: 4 4 3 3 3
24895 093/300 Write block 746 len 12288 data: 3 2 2
24896 094/300 Write block 872 len 36864 data: 2 2 2 3 3 3 3 3 5
24897 095/300 Write block 243 len 40960 data: 7 7 6 9 8 10 7 6 7 7
24898 096/300 Read block 3159 len 20480
24899 097/300 Write block 524 len 12288 data: 6 6 6
24900 098/300 Write block 2966 len 36864 data: 5 5 5 5 4 3 4 6 7
24901 099/300 Write block 3009 len 4096 data: 4
24902 100/300 Write block 3434 len 8192 data: 4 4
24903 101/300 Read block 3291 len 32768
24904 102/300 Write block 3375 len 8192 data: 5 5
24905 103/300 Read block 4520 len 12288
24906 104/300 Read block 2962 len 20480
24907 105/300 Write block 2499 len 8192 data: 5 5
24908 106/300 Write block 2413 len 8192 data: 4 4
24909 107/300 Read block 504 len 28672
24910 108/300 Read block 2700 len 20480
24911 109/300 Write block 3669 len 24576 data: 4 6 7 8 7 8
24912 110/300 Write block 4427 len 24576 data: 6 7 8 7 7 8
24913 111/300 Write block 1527 len 12288 data: 7 7 6
24914 112/300 Write block 1799 len 20480 data: 4 5 5 5 6
24915 113/300 Read block 4114 len 4096
24916 114/300 Write block 1367 len 20480 data: 6 5 5 5 3
24917 115/300 Read block 1583 len 16384
24918 116/300 Read block 214 len 12288
24919 117/300 Write block 4799 len 20480 data: 3 4 4 4 2
24920 118/300 Write block 3426 len 28672 data: 6 6 5 5 4 4 3
24921 119/300 Read block 3191 len 16384
24922 120/300 Read block 63 len 32768
24923 121/300 Write block 3001 len 20480 data: 5 4 6 5 3
24924 122/300 Write block 4042 len 40960 data: 4 4 4 5 6 6 5 5 7 8
24925 123/300 Read block 4207 len 16384
24926 124/300 Read block 3177 len 8192
24927 125/300 Write block 1727 len 24576 data: 5 3 4 4 4 4
24928 126/300 Flush
24929 127/300 Read block 3122 len 40960
24930 128/300 Write block 240 len 8192 data: 5 5
24931 129/300 Read block 2392 len 12288
24932 130/300 Flush
24933 131/300 Read block 4463 len 24576
24934 132/300 Write block 1782 len 40960 data: 5 6 6 6 7 6 6 5 6 6
24935 133/300 Read block 1667 len 36864
24936 134/300 Read block 2351 len 28672
24937 135/300 Read block 2072 len 20480
24938 136/300 Write block 2373 len 24576 data: 6 5 4 4 5 4
24939 137/300 Write block 3122 len 28672 data: 8 6 6 4 4 3 2
24940 138/300 Read block 2275 len 32768
24941 139/300 Write block 2478 len 20480 data: 9 9 5 5 5
24942 140/300 Read block 1917 len 32768
24943 141/300 Read block 2446 len 24576
24944 142/300 Write block 977 len 28672 data: 5 3 3 3 3 4 4
24945 143/300 Write block 3898 len 24576 data: 2 2 2 2 2 3
24946 144/300 Read block 95 len 12288
24947 145/300 Write block 3687 len 12288 data: 5 6 7
24948 146/300 Write block 2133 len 16384 data: 4 4 4 5
24949 147/300 Write block 3929 len 8192 data: 4 6
24950 148/300 Read block 2105 len 4096
24951 149/300 Write block 1988 len 24576 data: 3 4 3 3 4 4
24952 150/300 Write block 2959 len 28672 data: 6 7 5 5 4 3 4
24953 151/300 Read block 1258 len 16384
24954 152/300 Read block 2248 len 28672
24955 153/300 Write block 1073 len 8192 data: 6 6
24956 154/300 Read block 4269 len 16384
24957 155/300 Write block 3410 len 36864 data: 3 3 3 4 5 5 5 6 6
24958 156/300 Write block 369 len 28672 data: 5 6 5 5 4 4 4
24959 157/300 Flush
24960 158/300 Write block 3933 len 20480 data: 6 6 6 5 4
24961 159/300 Write block 4865 len 36864 data: 2 5 5 5 4 5 5 6 5
24962 160/300 Read block 398 len 28672
24963 161/300 Write block 1054 len 24576 data: 7 5 6 6 6 4
24964 162/300 Write block 2063 len 28672 data: 6 5 4 4 3 3 3
24965 163/300 Read block 2614 len 24576
24966 164/300 Flush
24967 165/300 Read block 3494 len 24576
24968 166/300 Flush
24969 167/300 Read block 3768 len 4096
24970 168/300 Read block 3153 len 40960
24971 169/300 Write block 1678 len 4096 data: 4
24972 170/300 Flush
24973 171/300 Write block 1181 len 28672 data: 4 2 2 2 2 3 3
24974 172/300 Read block 980 len 12288
24975 173/300 Read block 3351 len 20480
24976 174/300 Read block 1281 len 40960
24977 175/300 Read block 3621 len 4096
24978 176/300 Read block 1910 len 40960
24979 177/300 Read block 4783 len 28672
24980 178/300 Write block 983 len 24576 data: 5 4 4 4 4 4
24981 179/300 Write block 1304 len 8192 data: 4 4
24982 180/300 Read block 1348 len 24576
24983 181/300 Read block 3996 len 12288
24984 182/300 Write block 4010 len 24576 data: 6 6 5 5 3 4
24985 183/300 Read block 2656 len 36864
24986 184/300 Read block 1970 len 36864
24987 185/300 Read block 1704 len 20480
24988 186/300 Read block 4518 len 28672
24989 187/300 Read block 2375 len 28672
24990 188/300 Write block 4203 len 12288 data: 7 8 7
24991 189/300 Write block 54 len 8192 data: 5 4
24992 190/300 Write block 1345 len 16384 data: 4 4 4 6
24993 191/300 Read block 618 len 40960
24994 192/300 Read block 1099 len 28672
24995 193/300 Flush
24996 194/300 Read block 1778 len 28672
24997 195/300 Write block 2423 len 12288 data: 2 5 5
24998 196/300 Read block 397 len 36864
24999 197/300 Read block 2683 len 4096
25000 198/300 Flush
25001 199/300 Read block 4806 len 40960
25002 200/300 Flush
25003 201/300 Write block 1205 len 36864 data: 7 7 8 8 7 6 7 6 9
25004 202/300 Read block 394 len 20480
25005 203/300 Read block 212 len 40960
25006 204/300 Flush
25007 205/300 Flush
25008 206/300 Flush
25009 207/300 Read block 4175 len 28672
25010 208/300 Flush
25011 209/300 Read block 1884 len 40960
25012 210/300 Flush
25013 211/300 Write block 22 len 36864 data: 2 2 2 2 2 2 4 3 3
25014 212/300 Read block 1930 len 40960
25015 213/300 Flush
25016 214/300 Flush
25017 215/300 Write block 4664 len 28672 data: 7 5 5 5 3 3 4
25018 216/300 Write block 4169 len 8192 data: 3 4
25019 217/300 Write block 0 len 24576 data: 3 3 3 4 6 5
25020 218/300 Read block 452 len 28672
25021 219/300 Read block 3174 len 40960
25022 220/300 Write block 4474 len 8192 data: 5 4
25023 221/300 Read block 243 len 12288
25024 222/300 Read block 3882 len 24576
25025 223/300 Flush
25026 224/300 Flush
25027 225/300 Read block 813 len 12288
25028 226/300 Read block 2744 len 40960
25029 227/300 Read block 3261 len 24576
25030 228/300 Read block 3846 len 28672
25031 229/300 Read block 251 len 4096
25032 230/300 Write block 3948 len 8192 data: 5 7
25033 231/300 Write block 3993 len 16384 data: 5 5 4 5
25034 232/300 Read block 2908 len 24576
25035 233/300 Write block 4941 len 12288 data: 6 8 7
25036 234/300 Read block 2925 len 36864
25037 235/300 Write block 2677 len 20480 data: 6 5 6 3 3
25038 236/300 Write block 4498 len 28672 data: 6 5 6 5 5 7 8
25039 237/300 Write block 1087 len 40960 data: 6 8 9 8 9 8 7 6 6 5
25040 238/300 Read block 2948 len 16384
25041 239/300 Read block 193 len 8192
25042 240/300 Read block 4669 len 16384
25043 241/300 Read block 1669 len 16384
25044 242/300 Write block 3148 len 20480 data: 5 4 5 5 4
25045 243/300 Read block 4630 len 28672
25046 244/300 Flush
25047 245/300 Read block 305 len 40960
25048 246/300 Write block 1283 len 12288 data: 5 4 3
25049 247/300 Flush
25050 248/300 Read block 1941 len 16384
25051 249/300 Read block 2012 len 4096
25052 250/300 Write block 2054 len 40960 data: 5 6 5 5 4 4 5 5 7 7
25053 251/300 Write block 1199 len 32768 data: 6 6 8 8 8 8 8 8
25054 252/300 Write block 4558 len 28672 data: 5 4 4 3 2 3 3
25055 253/300 Read block 1388 len 36864
25056 254/300 Write block 3534 len 16384 data: 4 4 4 4
25057 255/300 Read block 4628 len 12288
25058 256/300 Read block 497 len 20480
25059 257/300 Write block 1968 len 24576 data: 6 7 7 8 7 7
25060 258/300 Read block 635 len 20480
25061 259/300 Read block 1356 len 32768
25062 260/300 Write block 2402 len 4096 data: 6
25063 261/300 Read block 4258 len 24576
25064 262/300 Write block 793 len 32768 data: 4 4 5 5 3 3 4 5
25065 263/300 Read block 2508 len 28672
25066 264/300 Write block 428 len 20480 data: 4 4 4 4 4
25067 265/300 Write block 261 len 24576 data: 6 4 4 5 5 5
25068 266/300 Write block 1237 len 16384 data: 5 5 4 5
25069 267/300 Read block 1853 len 12288
25070 268/300 Read block 591 len 32768
25071 269/300 Read block 2809 len 20480
25072 270/300 Read block 113 len 24576
25073 271/300 Read block 2984 len 40960
25074 272/300 Read block 418 len 4096
25075 273/300 Read block 3534 len 40960
25076 274/300 Read block 970 len 36864
25077 275/300 Read block 4696 len 20480
25078 276/300 Read block 1226 len 8192
25079 277/300 Write block 588 len 16384 data: 5 4 4 4
25080 278/300 Read block 916 len 16384
25081 279/300 Flush
25082 280/300 Read block 2703 len 12288
25083 281/300 Flush
25084 282/300 Read block 3990 len 4096
25085 283/300 Read block 1039 len 16384
25086 284/300 Read block 33 len 12288
25087 285/300 Read block 4892 len 20480
25088 286/300 Read block 4930 len 24576
25089 287/300 Read block 656 len 16384
25090 288/300 Write block 2507 len 36864 data: 5 3 3 4 4 4 4 5 5
25091 289/300 Read block 3964 len 12288
25092 290/300 Read block 2643 len 12288
25093 291/300 Write block 3053 len 28672 data: 2 3 3 3 3 3 4
25094 292/300 Write block 4726 len 8192 data: 3 3
25095 293/300 Flush
25096 294/300 Write block 680 len 20480 data: 4 4 4 5 5
25097 295/300 Write block 2539 len 16384 data: 5 4 5 6
25098 296/300 Write block 3529 len 24576 data: 7 7 5 5 5 5
25099 297/300 Write block 3255 len 24576 data: 6 5 6 5 6 6
25100 298/300 Write block 4613 len 8192 data: 4 4
25101 299/300 Read block 2801 len 12288
25102 300/300 Read block 4113 len 12288
25103 301/300 Read block 1266 len 36864
25104 Main task has work to do, go find it
25105 got dsc Start(0)
25106 start 0
25107 Nov 01 16:55:47.138 INFO request completed, latency_us: 237, response_code: 204, uri: /start/cid/0, method: POST, req_id: d6757dea-15d9-43ed-9016-f079a19b508f, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
25108 Replay: started 0, returned:Ok(())
25109 ----------------------------------------------------------------
25110 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
25111 States: Offline Active Active
25112 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25113 7648 Acked Write 10 Sent Done Done false
25114 7649 Acked Write 6 Sent Done Done false
25115 7650 Acked Read 7 Sent Done Done false
25116 7651 Acked Read 1 Sent Done Done false
25117 7652 Acked Write 5 Sent Done Done false
25118 7653 Acked Write 9 Sent Done Done false
25119 7654 Acked Write 3 Sent Done Done false
25120 7655 Acked Read 9 Sent Done Done false
25121 7656 Acked Write 5 Sent Done Done false
25122 7657 Acked Read 5 Sent Done Done false
25123 7658 Acked Write 9 Sent Done Done false
25124 7659 Acked Flush 0 Sent Done Done false
25125 7660 Acked Read 7 Sent Done Done false
25126 7661 Acked Write 2 Sent Done Done false
25127 7662 Acked Read 8 Sent Done Done false
25128 7663 Acked Read 9 Sent Done Done false
25129 7664 Acked Write 2 Sent Done Done false
25130 7665 Acked Read 10 Sent Done Done false
25131 7666 Acked Read 7 Sent Done Done false
25132 7667 Acked Read 2 Sent Done Done false
25133 7668 Acked Write 5 Sent Done Done false
25134 7669 Acked Flush 0 Sent Done Done false
25135 7670 Acked Read 7 Sent Done Done false
25136 7671 Acked Write 10 Sent Done Done false
25137 7672 Acked Write 3 Sent Done Done false
25138 7673 Acked Read 2 Sent Done Done false
25139 7674 Acked Write 3 Sent Done Done false
25140 7675 Acked Read 5 Sent Done Done false
25141 7676 Acked Read 9 Sent Done Done false
25142 7677 Acked Read 9 Sent Done Done false
25143 7678 Acked Write 6 Sent Done Done false
25144 7679 Acked Read 3 Sent Done Done false
25145 7680 Acked Write 10 Sent Done Done false
25146 7681 Acked Read 4 Sent Done Done false
25147 7682 Acked Read 2 Sent Done Done false
25148 7683 Acked Write 1 Sent Done Done false
25149 7684 Acked Read 3 Sent Done Done false
25150 7685 Acked Write 1 Sent Done Done false
25151 7686 Acked Write 4 Sent Done Done false
25152 7687 Acked Read 3 Sent Done Done false
25153 7688 Acked Write 7 Sent Done Done false
25154 7689 Acked Write 4 Sent Done Done false
25155 7690 Acked Read 4 Sent Done Done false
25156 7691 Acked Write 7 Sent Done Done false
25157 7692 Acked Read 7 Sent Done Done false
25158 7693 Acked Read 4 Sent Done Done false
25159 7694 Acked Write 4 Sent Done Done false
25160 7695 Acked Read 9 Sent Done Done false
25161 7696 Acked Read 7 Sent Done Done false
25162 7697 Acked Write 4 Sent Done Done false
25163 7698 Acked Read 2 Sent Done Done false
25164 7699 Acked Write 6 Sent Done Done false
25165 7700 Acked Read 2 Sent Done Done false
25166 7701 Acked Read 5 Sent Done Done false
25167 7702 Acked Write 7 Sent Done Done false
25168 7703 Acked Read 8 Sent Done Done false
25169 7704 Acked Write 6 Sent Done Done false
25170 7705 Acked Read 5 Sent Done Done false
25171 7706 Acked Read 8 Sent Done Done false
25172 7707 Acked Read 5 Sent Done Done false
25173 7708 Acked Read 8 Sent Done Done false
25174 7709 Acked Write 5 Sent Done Done false
25175 7710 Acked Read 3 Sent Done Done false
25176 7711 Acked Read 6 Sent Done Done false
25177 7712 Acked Write 2 Sent Done Done false
25178 7713 Acked Read 1 Sent Done Done false
25179 7714 Acked Flush 0 Sent Done Done false
25180 7715 Acked Read 8 Sent Done Done false
25181 7716 Acked Flush 0 Sent Done Done false
25182 7717 Acked Read 4 Sent Done Done false
25183 7718 Acked Read 9 Sent Done Done false
25184 7719 Acked Write 5 Sent Done Done false
25185 7720 Acked Read 6 Sent Done Done false
25186 7721 Acked Flush 0 Sent Done Done false
25187 7722 Acked Write 10 Sent Done Done false
25188 7723 Acked Write 5 Sent Done Done false
25189 7724 Acked Write 3 Sent Done Done false
25190 7725 Acked Write 5 Sent Done Done false
25191 7726 Acked Write 9 Sent Done Done false
25192 7727 Acked Read 6 Sent Done Done false
25193 7728 Acked Read 5 Sent Done Done false
25194 7729 Acked Write 2 Sent Done Done false
25195 7730 Acked Write 3 Sent Done Done false
25196 7731 Acked Flush 0 Sent Done Done false
25197 7732 Acked Read 5 Sent Done Done false
25198 7733 Acked Read 10 Sent Done Done false
25199 7734 Acked Read 10 Sent Done Done false
25200 7735 Acked Read 5 Sent Done Done false
25201 7736 Acked Read 8 Sent Done Done false
25202 7737 Acked Read 7 Sent Done Done false
25203 7738 Acked Read 3 Sent Done Done false
25204 7739 Acked Flush 0 Sent Done Done false
25205 7740 Acked Write 5 Sent Done Done false
25206 7741 Acked Write 3 Sent Done Done false
25207 7742 Acked Write 9 Sent Done Done false
25208 7743 Acked Write 10 Sent Done Done false
25209 7744 Acked Read 5 Sent Done Done false
25210 7745 Acked Write 3 Sent Done Done false
25211 7746 Acked Write 9 Sent Done Done false
25212 7747 Acked Write 1 Sent Done Done false
25213 7748 Acked Write 2 Sent Done Done false
25214 7749 Acked Read 8 Sent Done Done false
25215 7750 Acked Write 2 Sent Done Done false
25216 7751 Acked Read 3 Sent Done Done false
25217 7752 Acked Read 5 Sent Done Done false
25218 7753 Acked Write 2 Sent Done Done false
25219 7754 Acked Write 2 Sent Done Done false
25220 7755 Acked Read 7 Sent Done Done false
25221 7756 Acked Read 5 Sent Done Done false
25222 7757 Acked Write 6 Sent Done Done false
25223 7758 Acked Write 6 Sent Done Done false
25224 7759 Acked Write 3 Sent Done Done false
25225 7760 Acked Write 5 Sent Done Done false
25226 7761 Acked Read 1 Sent Done Done false
25227 7762 Acked Write 5 Sent Done Done false
25228 7763 Acked Read 4 Sent Done Done false
25229 7764 Acked Read 3 Sent Done Done false
25230 7765 Acked Write 5 Sent Done Done false
25231 7766 Acked Write 7 Sent Done Done false
25232 7767 Acked Read 4 Sent Done Done false
25233 7768 Acked Read 8 Sent Done Done false
25234 7769 Acked Write 5 Sent Done Done false
25235 7770 Acked Write 10 Sent Done Done false
25236 7771 Acked Read 4 Sent Done Done false
25237 7772 Acked Read 2 Sent Done Done false
25238 7773 Acked Write 6 Sent Done Done false
25239 7774 Acked Flush 0 Sent Done Done false
25240 7775 Acked Read 10 Sent Done Done false
25241 7776 Acked Write 2 Sent Done Done false
25242 7777 Acked Read 3 Sent Done Done false
25243 7778 Acked Flush 0 Sent Done Done false
25244 7779 Acked Read 6 Sent Done Done false
25245 7780 Acked Write 10 Sent Done Done false
25246 7781 Acked Read 9 Sent Done Done false
25247 7782 Acked Read 7 Sent Done Done false
25248 7783 Acked Read 5 Sent Done Done false
25249 7784 Acked Write 6 Sent Done Done false
25250 7785 Acked Write 7 Sent Done Done false
25251 7786 Acked Read 8 Sent Done Done false
25252 7787 Acked Write 5 Sent Done Done false
25253 7788 Acked Read 8 Sent Done Done false
25254 7789 Acked Read 6 Sent Done Done false
25255 7790 Acked Write 7 Sent Done Done false
25256 7791 Acked Write 6 Sent Done Done false
25257 7792 Acked Read 3 Sent Done Done false
25258 7793 Acked Write 3 Sent Done Done false
25259 7794 Acked Write 4 Sent Done Done false
25260 7795 Acked Write 2 Sent Done Done false
25261 7796 Acked Read 1 Sent Done Done false
25262 7797 Acked Write 6 Sent Done Done false
25263 7798 Acked Write 7 Sent Done Done false
25264 7799 Acked Read 4 Sent Done Done false
25265 7800 Acked Read 7 Sent Done Done false
25266 7801 Acked Write 2 Sent Done Done false
25267 7802 Acked Read 4 Sent Done Done false
25268 7803 Acked Write 9 Sent Done Done false
25269 7804 Acked Write 7 Sent Done Done false
25270 7805 Acked Flush 0 Sent Done Done false
25271 7806 Acked Write 5 Sent Done Done false
25272 7807 Acked Write 9 Sent Done Done false
25273 7808 Acked Read 7 Sent Done Done false
25274 7809 Acked Write 6 Sent Done Done false
25275 7810 Acked Write 7 Sent Done Done false
25276 7811 Acked Read 6 Sent Done Done false
25277 7812 Acked Flush 0 Sent Done Done false
25278 7813 Acked Read 6 Sent Done Done false
25279 7814 Acked Flush 0 Sent Done Done false
25280 7815 Acked Read 1 Sent Done Done false
25281 7816 Acked Read 10 Sent Done Done false
25282 7817 Acked Write 1 Sent Done Done false
25283 7818 Acked Flush 0 Sent Done Done false
25284 7819 Acked Write 7 Sent Done Done false
25285 7820 Acked Read 3 Sent Done Done false
25286 7821 Acked Read 5 Sent Done Done false
25287 7822 Acked Read 10 Sent Done Done false
25288 7823 Acked Read 1 Sent Done Done false
25289 7824 Acked Read 10 Sent Done Done false
25290 7825 Acked Read 7 Sent Done Done false
25291 7826 Acked Write 6 Sent Done Done false
25292 7827 Acked Write 2 Sent Done Done false
25293 7828 Acked Read 6 Sent Done Done false
25294 7829 Acked Read 3 Sent Done Done false
25295 7830 Acked Write 6 Sent Done Done false
25296 7831 Acked Read 9 Sent Done Done false
25297 7832 Acked Read 9 Sent Done Done false
25298 7833 Acked Read 5 Sent Done Done false
25299 7834 Acked Read 7 Sent Done Done false
25300 7835 Acked Read 7 Sent Done Done false
25301 7836 Acked Write 3 Sent Done Done false
25302 7837 Acked Write 2 Sent Done Done false
25303 7838 Acked Write 4 Sent Done Done false
25304 7839 Acked Read 10 Sent Done Done false
25305 7840 Acked Read 7 Sent Done Done false
25306 7841 Acked Flush 0 Sent Done Done false
25307 7842 Acked Read 7 Sent Done Done false
25308 7843 Acked Write 3 Sent Done Done false
25309 7844 Acked Read 9 Sent Done Done false
25310 7845 Acked Read 1 Sent Done Done false
25311 7846 Acked Flush 0 Sent Done Done false
25312 7847 Acked Read 10 Sent Done Done false
25313 7848 Acked Flush 0 Sent Done Done false
25314 7849 Acked Write 9 Sent Done Done false
25315 7850 Acked Read 5 Sent Done Done false
25316 7851 Acked Read 10 Sent Done Done false
25317 7852 Acked Flush 0 Sent Done Done false
25318 7853 Acked Flush 0 Sent Done Done false
25319 7854 Acked Flush 0 Sent Done Done false
25320 7855 Acked Read 7 Sent Done Done false
25321 7856 Acked Flush 0 Sent Done Done false
25322 7857 Acked Read 10 Sent Done Done false
25323 7858 Acked Flush 0 Sent Done Done false
25324 7859 Acked Write 9 Sent Done Done false
25325 7860 Acked Read 10 Sent Done Done false
25326 7861 Acked Flush 0 Sent Done Done false
25327 7862 Acked Flush 0 Sent Done Done false
25328 7863 Acked Write 7 Sent Done Done false
25329 7864 Acked Write 2 Sent Done Done false
25330 7865 Acked Write 6 Sent Done Done false
25331 7866 Acked Read 7 Sent Done Done false
25332 7867 Acked Read 10 Sent Done Done false
25333 7868 Acked Write 2 Sent Done Done false
25334 7869 Acked Read 3 Sent Done Done false
25335 7870 Acked Read 6 Sent Done Done false
25336 7871 Acked Flush 0 Sent Done Done false
25337 7872 Acked Flush 0 Sent Done Done false
25338 7873 Acked Read 3 Sent Done Done false
25339 7874 Acked Read 10 Sent Done Done false
25340 7875 Acked Read 6 Sent Done Done false
25341 7876 Acked Read 7 Sent Done Done false
25342 7877 Acked Read 1 Sent Done Done false
25343 7878 Acked Write 2 Sent Done Done false
25344 7879 Acked Write 4 Sent Done Done false
25345 7880 Acked Read 6 Sent Done Done false
25346 7881 Acked Write 3 Sent Done Done false
25347 7882 Acked Read 9 Sent Done Done false
25348 7883 Acked Write 5 Sent Done Done false
25349 7884 Acked Write 7 Sent Done Done false
25350 7885 Acked Write 10 Sent Done Done false
25351 7886 Acked Read 4 Sent Done Done false
25352 7887 Acked Read 2 Sent Done Done false
25353 7888 Acked Read 4 Sent Done Done false
25354 7889 Acked Read 4 Sent Done Done false
25355 7890 Acked Write 5 Sent Done Done false
25356 7891 Acked Read 7 Sent Done Done false
25357 7892 Acked Flush 0 Sent Done Done false
25358 7893 Acked Read 10 Sent Done Done false
25359 7894 Acked Write 3 Sent Done Done false
25360 7895 Acked Flush 0 Sent Done Done false
25361 7896 Acked Read 4 Sent Done Done false
25362 7897 Acked Read 1 Sent Done Done false
25363 7898 Acked Write 10 Sent Done Done false
25364 7899 Acked Write 8 Sent Done Done false
25365 7900 Acked Write 7 Sent Done Done false
25366 7901 Acked Read 9 Sent Done Done false
25367 7902 Acked Write 4 Sent Done Done false
25368 7903 Acked Read 3 Sent Done Done false
25369 7904 Acked Read 5 Sent Done Done false
25370 7905 Acked Write 6 Sent Done Done false
25371 7906 Acked Read 5 Sent Done Done false
25372 7907 Acked Read 8 Sent Done Done false
25373 7908 Acked Write 1 Sent Done Done false
25374 7909 Acked Read 6 Sent Done Done false
25375 7910 Acked Write 8 Sent Done Done false
25376 7911 Acked Read 7 Sent Done Done false
25377 7912 Acked Write 5 Sent Done Done false
25378 7913 Acked Write 6 Sent Done Done false
25379 7914 Acked Write 4 Sent Done Done false
25380 7915 Acked Read 3 Sent Done Done false
25381 7916 Acked Read 8 Sent Done Done false
25382 7917 Acked Read 5 Sent Done Done false
25383 7918 Acked Read 6 Sent Done Done false
25384 7919 Acked Read 10 Sent Done Done false
25385 7920 Acked Read 1 Sent Done Done false
25386 7921 Acked Read 10 Sent Done Done false
25387 7922 Acked Read 9 Sent Done Done false
25388 7923 Acked Read 5 Sent Done Done false
25389 7924 Acked Read 2 Sent Done Done false
25390 7925 Acked Write 4 Sent Done Done false
25391 7926 Acked Read 4 Sent Done Done false
25392 7927 Acked Flush 0 Sent Done Done false
25393 7928 Acked Read 3 Sent Done Done false
25394 7929 Acked Flush 0 Sent Done Done false
25395 7930 Acked Read 1 Sent Done Done false
25396 7931 Acked Read 4 Sent Done Done false
25397 7932 Acked Read 3 Sent Done Done false
25398 7933 Acked Read 5 Sent Done Done false
25399 7934 Acked Read 6 Sent Done Done false
25400 7935 Acked Read 4 Sent Done Done false
25401 7936 Acked Write 9 Sent Done Done false
25402 7937 Acked Read 3 Sent Done Done false
25403 7938 Acked Read 3 Sent Done Done false
25404 7939 Acked Write 7 Sent Done Done false
25405 7940 Acked Write 2 Sent Done Done false
25406 7941 Acked Flush 0 Sent Done Done false
25407 7942 Acked Write 5 Sent Done Done false
25408 7943 Acked Write 4 Sent Done Done false
25409 7944 Acked Write 6 Sent Done Done false
25410 7945 Acked Write 6 Sent Done Done false
25411 7946 Acked Write 2 Sent Done Done false
25412 7947 Acked Read 3 Sent Done Done false
25413 7948 Acked Read 3 Sent Done Done false
25414 7949 Acked Read 9 Sent Done Done false
25415 STATES DS:0 DS:1 DS:2 TOTAL
25416 Sent 302 0 0 302
25417 Done 0 302 302 604
25418 Skipped 0 0 0 0
25419 Error 0 0 0 0
25420 Last Flush: 7647 7941 7941
25421 Downstairs last five completed: 7647 7646 7645 7644 7643
25422 Upstairs last five completed: 7949 7948 7947 7946 7945
25423 CLIENT: Up:0 ds:302 act:2
25424 ----------------------------------------------------------------
25425 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
25426 States: Offline Active Active
25427 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25428 7648 Acked Write 10 Sent Done Done false
25429 7649 Acked Write 6 Sent Done Done false
25430 7650 Acked Read 7 Sent Done Done false
25431 7651 Acked Read 1 Sent Done Done false
25432 7652 Acked Write 5 Sent Done Done false
25433 7653 Acked Write 9 Sent Done Done false
25434 7654 Acked Write 3 Sent Done Done false
25435 7655 Acked Read 9 Sent Done Done false
25436 7656 Acked Write 5 Sent Done Done false
25437 7657 Acked Read 5 Sent Done Done false
25438 7658 Acked Write 9 Sent Done Done false
25439 7659 Acked Flush 0 Sent Done Done false
25440 7660 Acked Read 7 Sent Done Done false
25441 7661 Acked Write 2 Sent Done Done false
25442 7662 Acked Read 8 Sent Done Done false
25443 7663 Acked Read 9 Sent Done Done false
25444 7664 Acked Write 2 Sent Done Done false
25445 7665 Acked Read 10 Sent Done Done false
25446 7666 Acked Read 7 Sent Done Done false
25447 7667 Acked Read 2 Sent Done Done false
25448 7668 Acked Write 5 Sent Done Done false
25449 7669 Acked Flush 0 Sent Done Done false
25450 7670 Acked Read 7 Sent Done Done false
25451 7671 Acked Write 10 Sent Done Done false
25452 7672 Acked Write 3 Sent Done Done false
25453 7673 Acked Read 2 Sent Done Done false
25454 7674 Acked Write 3 Sent Done Done false
25455 7675 Acked Read 5 Sent Done Done false
25456 7676 Acked Read 9 Sent Done Done false
25457 7677 Acked Read 9 Sent Done Done false
25458 7678 Acked Write 6 Sent Done Done false
25459 7679 Acked Read 3 Sent Done Done false
25460 7680 Acked Write 10 Sent Done Done false
25461 7681 Acked Read 4 Sent Done Done false
25462 7682 Acked Read 2 Sent Done Done false
25463 7683 Acked Write 1 Sent Done Done false
25464 7684 Acked Read 3 Sent Done Done false
25465 7685 Acked Write 1 Sent Done Done false
25466 7686 Acked Write 4 Sent Done Done false
25467 7687 Acked Read 3 Sent Done Done false
25468 7688 Acked Write 7 Sent Done Done false
25469 7689 Acked Write 4 Sent Done Done false
25470 7690 Acked Read 4 Sent Done Done false
25471 7691 Acked Write 7 Sent Done Done false
25472 7692 Acked Read 7 Sent Done Done false
25473 7693 Acked Read 4 Sent Done Done false
25474 7694 Acked Write 4 Sent Done Done false
25475 7695 Acked Read 9 Sent Done Done false
25476 7696 Acked Read 7 Sent Done Done false
25477 7697 Acked Write 4 Sent Done Done false
25478 7698 Acked Read 2 Sent Done Done false
25479 7699 Acked Write 6 Sent Done Done false
25480 7700 Acked Read 2 Sent Done Done false
25481 7701 Acked Read 5 Sent Done Done false
25482 7702 Acked Write 7 Sent Done Done false
25483 7703 Acked Read 8 Sent Done Done false
25484 7704 Acked Write 6 Sent Done Done false
25485 7705 Acked Read 5 Sent Done Done false
25486 7706 Acked Read 8 Sent Done Done false
25487 7707 Acked Read 5 Sent Done Done false
25488 7708 Acked Read 8 Sent Done Done false
25489 7709 Acked Write 5 Sent Done Done false
25490 7710 Acked Read 3 Sent Done Done false
25491 7711 Acked Read 6 Sent Done Done false
25492 7712 Acked Write 2 Sent Done Done false
25493 7713 Acked Read 1 Sent Done Done false
25494 7714 Acked Flush 0 Sent Done Done false
25495 7715 Acked Read 8 Sent Done Done false
25496 7716 Acked Flush 0 Sent Done Done false
25497 7717 Acked Read 4 Sent Done Done false
25498 7718 Acked Read 9 Sent Done Done false
25499 7719 Acked Write 5 Sent Done Done false
25500 7720 Acked Read 6 Sent Done Done false
25501 7721 Acked Flush 0 Sent Done Done false
25502 7722 Acked Write 10 Sent Done Done false
25503 7723 Acked Write 5 Sent Done Done false
25504 7724 Acked Write 3 Sent Done Done false
25505 7725 Acked Write 5 Sent Done Done false
25506 7726 Acked Write 9 Sent Done Done false
25507 7727 Acked Read 6 Sent Done Done false
25508 7728 Acked Read 5 Sent Done Done false
25509 7729 Acked Write 2 Sent Done Done false
25510 7730 Acked Write 3 Sent Done Done false
25511 7731 Acked Flush 0 Sent Done Done false
25512 7732 Acked Read 5 Sent Done Done false
25513 7733 Acked Read 10 Sent Done Done false
25514 7734 Acked Read 10 Sent Done Done false
25515 7735 Acked Read 5 Sent Done Done false
25516 7736 Acked Read 8 Sent Done Done false
25517 7737 Acked Read 7 Sent Done Done false
25518 7738 Acked Read 3 Sent Done Done false
25519 7739 Acked Flush 0 Sent Done Done false
25520 7740 Acked Write 5 Sent Done Done false
25521 7741 Acked Write 3 Sent Done Done false
25522 7742 Acked Write 9 Sent Done Done false
25523 7743 Acked Write 10 Sent Done Done false
25524 7744 Acked Read 5 Sent Done Done false
25525 7745 Acked Write 3 Sent Done Done false
25526 7746 Acked Write 9 Sent Done Done false
25527 7747 Acked Write 1 Sent Done Done false
25528 7748 Acked Write 2 Sent Done Done false
25529 7749 Acked Read 8 Sent Done Done false
25530 7750 Acked Write 2 Sent Done Done false
25531 7751 Acked Read 3 Sent Done Done false
25532 7752 Acked Read 5 Sent Done Done false
25533 7753 Acked Write 2 Sent Done Done false
25534 7754 Acked Write 2 Sent Done Done false
25535 7755 Acked Read 7 Sent Done Done false
25536 7756 Acked Read 5 Sent Done Done false
25537 7757 Acked Write 6 Sent Done Done false
25538 7758 Acked Write 6 Sent Done Done false
25539 7759 Acked Write 3 Sent Done Done false
25540 7760 Acked Write 5 Sent Done Done false
25541 7761 Acked Read 1 Sent Done Done false
25542 7762 Acked Write 5 Sent Done Done false
25543 7763 Acked Read 4 Sent Done Done false
25544 7764 Acked Read 3 Sent Done Done false
25545 7765 Acked Write 5 Sent Done Done false
25546 7766 Acked Write 7 Sent Done Done false
25547 7767 Acked Read 4 Sent Done Done false
25548 7768 Acked Read 8 Sent Done Done false
25549 7769 Acked Write 5 Sent Done Done false
25550 7770 Acked Write 10 Sent Done Done false
25551 7771 Acked Read 4 Sent Done Done false
25552 7772 Acked Read 2 Sent Done Done false
25553 7773 Acked Write 6 Sent Done Done false
25554 7774 Acked Flush 0 Sent Done Done false
25555 7775 Acked Read 10 Sent Done Done false
25556 7776 Acked Write 2 Sent Done Done false
25557 7777 Acked Read 3 Sent Done Done false
25558 7778 Acked Flush 0 Sent Done Done false
25559 7779 Acked Read 6 Sent Done Done false
25560 7780 Acked Write 10 Sent Done Done false
25561 7781 Acked Read 9 Sent Done Done false
25562 7782 Acked Read 7 Sent Done Done false
25563 7783 Acked Read 5 Sent Done Done false
25564 7784 Acked Write 6 Sent Done Done false
25565 7785 Acked Write 7 Sent Done Done false
25566 7786 Acked Read 8 Sent Done Done false
25567 7787 Acked Write 5 Sent Done Done false
25568 7788 Acked Read 8 Sent Done Done false
25569 7789 Acked Read 6 Sent Done Done false
25570 7790 Acked Write 7 Sent Done Done false
25571 7791 Acked Write 6 Sent Done Done false
25572 7792 Acked Read 3 Sent Done Done false
25573 7793 Acked Write 3 Sent Done Done false
25574 7794 Acked Write 4 Sent Done Done false
25575 7795 Acked Write 2 Sent Done Done false
25576 7796 Acked Read 1 Sent Done Done false
25577 7797 Acked Write 6 Sent Done Done false
25578 7798 Acked Write 7 Sent Done Done false
25579 7799 Acked Read 4 Sent Done Done false
25580 7800 Acked Read 7 Sent Done Done false
25581 7801 Acked Write 2 Sent Done Done false
25582 7802 Acked Read 4 Sent Done Done false
25583 7803 Acked Write 9 Sent Done Done false
25584 7804 Acked Write 7 Sent Done Done false
25585 7805 Acked Flush 0 Sent Done Done false
25586 7806 Acked Write 5 Sent Done Done false
25587 7807 Acked Write 9 Sent Done Done false
25588 7808 Acked Read 7 Sent Done Done false
25589 7809 Acked Write 6 Sent Done Done false
25590 7810 Acked Write 7 Sent Done Done false
25591 7811 Acked Read 6 Sent Done Done false
25592 7812 Acked Flush 0 Sent Done Done false
25593 7813 Acked Read 6 Sent Done Done false
25594 7814 Acked Flush 0 Sent Done Done false
25595 7815 Acked Read 1 Sent Done Done false
25596 7816 Acked Read 10 Sent Done Done false
25597 7817 Acked Write 1 Sent Done Done false
25598 7818 Acked Flush 0 Sent Done Done false
25599 7819 Acked Write 7 Sent Done Done false
25600 7820 Acked Read 3 Sent Done Done false
25601 7821 Acked Read 5 Sent Done Done false
25602 7822 Acked Read 10 Sent Done Done false
25603 7823 Acked Read 1 Sent Done Done false
25604 7824 Acked Read 10 Sent Done Done false
25605 7825 Acked Read 7 Sent Done Done false
25606 7826 Acked Write 6 Sent Done Done false
25607 7827 Acked Write 2 Sent Done Done false
25608 7828 Acked Read 6 Sent Done Done false
25609 7829 Acked Read 3 Sent Done Done false
25610 7830 Acked Write 6 Sent Done Done false
25611 7831 Acked Read 9 Sent Done Done false
25612 7832 Acked Read 9 Sent Done Done false
25613 7833 Acked Read 5 Sent Done Done false
25614 7834 Acked Read 7 Sent Done Done false
25615 7835 Acked Read 7 Sent Done Done false
25616 7836 Acked Write 3 Sent Done Done false
25617 7837 Acked Write 2 Sent Done Done false
25618 7838 Acked Write 4 Sent Done Done false
25619 7839 Acked Read 10 Sent Done Done false
25620 7840 Acked Read 7 Sent Done Done false
25621 7841 Acked Flush 0 Sent Done Done false
25622 7842 Acked Read 7 Sent Done Done false
25623 7843 Acked Write 3 Sent Done Done false
25624 7844 Acked Read 9 Sent Done Done false
25625 7845 Acked Read 1 Sent Done Done false
25626 7846 Acked Flush 0 Sent Done Done false
25627 7847 Acked Read 10 Sent Done Done false
25628 7848 Acked Flush 0 Sent Done Done false
25629 7849 Acked Write 9 Sent Done Done false
25630 7850 Acked Read 5 Sent Done Done false
25631 7851 Acked Read 10 Sent Done Done false
25632 7852 Acked Flush 0 Sent Done Done false
25633 7853 Acked Flush 0 Sent Done Done false
25634 7854 Acked Flush 0 Sent Done Done false
25635 7855 Acked Read 7 Sent Done Done false
25636 7856 Acked Flush 0 Sent Done Done false
25637 7857 Acked Read 10 Sent Done Done false
25638 7858 Acked Flush 0 Sent Done Done false
25639 7859 Acked Write 9 Sent Done Done false
25640 7860 Acked Read 10 Sent Done Done false
25641 7861 Acked Flush 0 Sent Done Done false
25642 7862 Acked Flush 0 Sent Done Done false
25643 7863 Acked Write 7 Sent Done Done false
25644 7864 Acked Write 2 Sent Done Done false
25645 7865 Acked Write 6 Sent Done Done false
25646 7866 Acked Read 7 Sent Done Done false
25647 7867 Acked Read 10 Sent Done Done false
25648 7868 Acked Write 2 Sent Done Done false
25649 7869 Acked Read 3 Sent Done Done false
25650 7870 Acked Read 6 Sent Done Done false
25651 7871 Acked Flush 0 Sent Done Done false
25652 7872 Acked Flush 0 Sent Done Done false
25653 7873 Acked Read 3 Sent Done Done false
25654 7874 Acked Read 10 Sent Done Done false
25655 7875 Acked Read 6 Sent Done Done false
25656 7876 Acked Read 7 Sent Done Done false
25657 7877 Acked Read 1 Sent Done Done false
25658 7878 Acked Write 2 Sent Done Done false
25659 7879 Acked Write 4 Sent Done Done false
25660 7880 Acked Read 6 Sent Done Done false
25661 7881 Acked Write 3 Sent Done Done false
25662 7882 Acked Read 9 Sent Done Done false
25663 7883 Acked Write 5 Sent Done Done false
25664 7884 Acked Write 7 Sent Done Done false
25665 7885 Acked Write 10 Sent Done Done false
25666 7886 Acked Read 4 Sent Done Done false
25667 7887 Acked Read 2 Sent Done Done false
25668 7888 Acked Read 4 Sent Done Done false
25669 7889 Acked Read 4 Sent Done Done false
25670 7890 Acked Write 5 Sent Done Done false
25671 7891 Acked Read 7 Sent Done Done false
25672 7892 Acked Flush 0 Sent Done Done false
25673 7893 Acked Read 10 Sent Done Done false
25674 7894 Acked Write 3 Sent Done Done false
25675 7895 Acked Flush 0 Sent Done Done false
25676 7896 Acked Read 4 Sent Done Done false
25677 7897 Acked Read 1 Sent Done Done false
25678 7898 Acked Write 10 Sent Done Done false
25679 7899 Acked Write 8 Sent Done Done false
25680 7900 Acked Write 7 Sent Done Done false
25681 7901 Acked Read 9 Sent Done Done false
25682 7902 Acked Write 4 Sent Done Done false
25683 7903 Acked Read 3 Sent Done Done false
25684 7904 Acked Read 5 Sent Done Done false
25685 7905 Acked Write 6 Sent Done Done false
25686 7906 Acked Read 5 Sent Done Done false
25687 7907 Acked Read 8 Sent Done Done false
25688 7908 Acked Write 1 Sent Done Done false
25689 7909 Acked Read 6 Sent Done Done false
25690 7910 Acked Write 8 Sent Done Done false
25691 7911 Acked Read 7 Sent Done Done false
25692 7912 Acked Write 5 Sent Done Done false
25693 7913 Acked Write 6 Sent Done Done false
25694 7914 Acked Write 4 Sent Done Done false
25695 7915 Acked Read 3 Sent Done Done false
25696 7916 Acked Read 8 Sent Done Done false
25697 7917 Acked Read 5 Sent Done Done false
25698 7918 Acked Read 6 Sent Done Done false
25699 7919 Acked Read 10 Sent Done Done false
25700 7920 Acked Read 1 Sent Done Done false
25701 7921 Acked Read 10 Sent Done Done false
25702 7922 Acked Read 9 Sent Done Done false
25703 7923 Acked Read 5 Sent Done Done false
25704 7924 Acked Read 2 Sent Done Done false
25705 7925 Acked Write 4 Sent Done Done false
25706 7926 Acked Read 4 Sent Done Done false
25707 7927 Acked Flush 0 Sent Done Done false
25708 7928 Acked Read 3 Sent Done Done false
25709 7929 Acked Flush 0 Sent Done Done false
25710 7930 Acked Read 1 Sent Done Done false
25711 7931 Acked Read 4 Sent Done Done false
25712 7932 Acked Read 3 Sent Done Done false
25713 7933 Acked Read 5 Sent Done Done false
25714 7934 Acked Read 6 Sent Done Done false
25715 7935 Acked Read 4 Sent Done Done false
25716 7936 Acked Write 9 Sent Done Done false
25717 7937 Acked Read 3 Sent Done Done false
25718 7938 Acked Read 3 Sent Done Done false
25719 7939 Acked Write 7 Sent Done Done false
25720 7940 Acked Write 2 Sent Done Done false
25721 7941 Acked Flush 0 Sent Done Done false
25722 7942 Acked Write 5 Sent Done Done false
25723 7943 Acked Write 4 Sent Done Done false
25724 7944 Acked Write 6 Sent Done Done false
25725 7945 Acked Write 6 Sent Done Done false
25726 7946 Acked Write 2 Sent Done Done false
25727 7947 Acked Read 3 Sent Done Done false
25728 7948 Acked Read 3 Sent Done Done false
25729 7949 Acked Read 9 Sent Done Done false
25730 7950 Acked Flush 0 Sent Done Done false
25731 STATES DS:0 DS:1 DS:2 TOTAL
25732 Sent 303 0 0 303
25733 Done 0 303 303 606
25734 Skipped 0 0 0 0
25735 Error 0 0 0 0
25736 Last Flush: 7647 7950 7950
25737 Downstairs last five completed: 7647 7646 7645 7644 7643
25738 Upstairs last five completed: 7950 7949 7948 7947 7946
25739 CLIENT: Up:0 ds:303 act:2
257402024-11-01T16:55:52.852ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
257412024-11-01T16:55:52.853ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
257422024-11-01T16:55:52.853ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
257432024-11-01T16:55:52.853ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
257442024-11-01T16:55:52.853ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
25745 [8810] got start action
25746 [8810] I am going to restart
25747 Starting downstairs at port 8810
25748 Make output file at "/tmp/dsc/downstairs-8810.txt"
25749 [8810][0] reports Starting
25750 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1221)
25751 [8810][0] reports Running
25752 ----------------------------------------------------------------
25753 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
25754 States: Offline Active Active
25755 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25756 7648 Acked Write 10 Sent Done Done false
25757 7649 Acked Write 6 Sent Done Done false
25758 7650 Acked Read 7 Sent Done Done false
25759 7651 Acked Read 1 Sent Done Done false
25760 7652 Acked Write 5 Sent Done Done false
25761 7653 Acked Write 9 Sent Done Done false
25762 7654 Acked Write 3 Sent Done Done false
25763 7655 Acked Read 9 Sent Done Done false
25764 7656 Acked Write 5 Sent Done Done false
25765 7657 Acked Read 5 Sent Done Done false
25766 7658 Acked Write 9 Sent Done Done false
25767 7659 Acked Flush 0 Sent Done Done false
25768 7660 Acked Read 7 Sent Done Done false
25769 7661 Acked Write 2 Sent Done Done false
25770 7662 Acked Read 8 Sent Done Done false
25771 7663 Acked Read 9 Sent Done Done false
25772 7664 Acked Write 2 Sent Done Done false
25773 7665 Acked Read 10 Sent Done Done false
25774 7666 Acked Read 7 Sent Done Done false
25775 7667 Acked Read 2 Sent Done Done false
25776 7668 Acked Write 5 Sent Done Done false
25777 7669 Acked Flush 0 Sent Done Done false
25778 7670 Acked Read 7 Sent Done Done false
25779 7671 Acked Write 10 Sent Done Done false
25780 7672 Acked Write 3 Sent Done Done false
25781 7673 Acked Read 2 Sent Done Done false
25782 7674 Acked Write 3 Sent Done Done false
25783 7675 Acked Read 5 Sent Done Done false
25784 7676 Acked Read 9 Sent Done Done false
25785 7677 Acked Read 9 Sent Done Done false
25786 7678 Acked Write 6 Sent Done Done false
25787 7679 Acked Read 3 Sent Done Done false
25788 7680 Acked Write 10 Sent Done Done false
25789 7681 Acked Read 4 Sent Done Done false
25790 7682 Acked Read 2 Sent Done Done false
25791 7683 Acked Write 1 Sent Done Done false
25792 7684 Acked Read 3 Sent Done Done false
25793 7685 Acked Write 1 Sent Done Done false
25794 7686 Acked Write 4 Sent Done Done false
25795 7687 Acked Read 3 Sent Done Done false
25796 7688 Acked Write 7 Sent Done Done false
25797 7689 Acked Write 4 Sent Done Done false
25798 7690 Acked Read 4 Sent Done Done false
25799 7691 Acked Write 7 Sent Done Done false
25800 7692 Acked Read 7 Sent Done Done false
25801 7693 Acked Read 4 Sent Done Done false
25802 7694 Acked Write 4 Sent Done Done false
25803 7695 Acked Read 9 Sent Done Done false
25804 7696 Acked Read 7 Sent Done Done false
25805 7697 Acked Write 4 Sent Done Done false
25806 7698 Acked Read 2 Sent Done Done false
25807 7699 Acked Write 6 Sent Done Done false
25808 7700 Acked Read 2 Sent Done Done false
25809 7701 Acked Read 5 Sent Done Done false
25810 7702 Acked Write 7 Sent Done Done false
25811 7703 Acked Read 8 Sent Done Done false
25812 7704 Acked Write 6 Sent Done Done false
25813 7705 Acked Read 5 Sent Done Done false
25814 7706 Acked Read 8 Sent Done Done false
25815 7707 Acked Read 5 Sent Done Done false
25816 7708 Acked Read 8 Sent Done Done false
25817 7709 Acked Write 5 Sent Done Done false
25818 7710 Acked Read 3 Sent Done Done false
25819 7711 Acked Read 6 Sent Done Done false
25820 7712 Acked Write 2 Sent Done Done false
25821 7713 Acked Read 1 Sent Done Done false
25822 7714 Acked Flush 0 Sent Done Done false
25823 7715 Acked Read 8 Sent Done Done false
25824 7716 Acked Flush 0 Sent Done Done false
25825 7717 Acked Read 4 Sent Done Done false
25826 7718 Acked Read 9 Sent Done Done false
25827 7719 Acked Write 5 Sent Done Done false
25828 7720 Acked Read 6 Sent Done Done false
25829 7721 Acked Flush 0 Sent Done Done false
25830 7722 Acked Write 10 Sent Done Done false
25831 7723 Acked Write 5 Sent Done Done false
25832 7724 Acked Write 3 Sent Done Done false
25833 7725 Acked Write 5 Sent Done Done false
25834 7726 Acked Write 9 Sent Done Done false
25835 7727 Acked Read 6 Sent Done Done false
25836 7728 Acked Read 5 Sent Done Done false
25837 7729 Acked Write 2 Sent Done Done false
25838 7730 Acked Write 3 Sent Done Done false
25839 7731 Acked Flush 0 Sent Done Done false
25840 7732 Acked Read 5 Sent Done Done false
25841 7733 Acked Read 10 Sent Done Done false
25842 7734 Acked Read 10 Sent Done Done false
25843 7735 Acked Read 5 Sent Done Done false
25844 7736 Acked Read 8 Sent Done Done false
25845 7737 Acked Read 7 Sent Done Done false
25846 7738 Acked Read 3 Sent Done Done false
25847 7739 Acked Flush 0 Sent Done Done false
25848 7740 Acked Write 5 Sent Done Done false
25849 7741 Acked Write 3 Sent Done Done false
25850 7742 Acked Write 9 Sent Done Done false
25851 7743 Acked Write 10 Sent Done Done false
25852 7744 Acked Read 5 Sent Done Done false
25853 7745 Acked Write 3 Sent Done Done false
25854 7746 Acked Write 9 Sent Done Done false
25855 7747 Acked Write 1 Sent Done Done false
25856 7748 Acked Write 2 Sent Done Done false
25857 7749 Acked Read 8 Sent Done Done false
25858 7750 Acked Write 2 Sent Done Done false
25859 7751 Acked Read 3 Sent Done Done false
25860 7752 Acked Read 5 Sent Done Done false
25861 7753 Acked Write 2 Sent Done Done false
25862 7754 Acked Write 2 Sent Done Done false
25863 7755 Acked Read 7 Sent Done Done false
25864 7756 Acked Read 5 Sent Done Done false
25865 7757 Acked Write 6 Sent Done Done false
25866 7758 Acked Write 6 Sent Done Done false
25867 7759 Acked Write 3 Sent Done Done false
25868 7760 Acked Write 5 Sent Done Done false
25869 7761 Acked Read 1 Sent Done Done false
25870 7762 Acked Write 5 Sent Done Done false
25871 7763 Acked Read 4 Sent Done Done false
25872 7764 Acked Read 3 Sent Done Done false
25873 7765 Acked Write 5 Sent Done Done false
25874 7766 Acked Write 7 Sent Done Done false
25875 7767 Acked Read 4 Sent Done Done false
25876 7768 Acked Read 8 Sent Done Done false
25877 7769 Acked Write 5 Sent Done Done false
25878 7770 Acked Write 10 Sent Done Done false
25879 7771 Acked Read 4 Sent Done Done false
25880 7772 Acked Read 2 Sent Done Done false
25881 7773 Acked Write 6 Sent Done Done false
25882 7774 Acked Flush 0 Sent Done Done false
25883 7775 Acked Read 10 Sent Done Done false
25884 7776 Acked Write 2 Sent Done Done false
25885 7777 Acked Read 3 Sent Done Done false
25886 7778 Acked Flush 0 Sent Done Done false
25887 7779 Acked Read 6 Sent Done Done false
25888 7780 Acked Write 10 Sent Done Done false
25889 7781 Acked Read 9 Sent Done Done false
25890 7782 Acked Read 7 Sent Done Done false
25891 7783 Acked Read 5 Sent Done Done false
25892 7784 Acked Write 6 Sent Done Done false
25893 7785 Acked Write 7 Sent Done Done false
25894 7786 Acked Read 8 Sent Done Done false
25895 7787 Acked Write 5 Sent Done Done false
25896 7788 Acked Read 8 Sent Done Done false
25897 7789 Acked Read 6 Sent Done Done false
25898 7790 Acked Write 7 Sent Done Done false
25899 7791 Acked Write 6 Sent Done Done false
25900 7792 Acked Read 3 Sent Done Done false
25901 7793 Acked Write 3 Sent Done Done false
25902 7794 Acked Write 4 Sent Done Done false
25903 7795 Acked Write 2 Sent Done Done false
25904 7796 Acked Read 1 Sent Done Done false
25905 7797 Acked Write 6 Sent Done Done false
25906 7798 Acked Write 7 Sent Done Done false
25907 7799 Acked Read 4 Sent Done Done false
25908 7800 Acked Read 7 Sent Done Done false
25909 7801 Acked Write 2 Sent Done Done false
25910 7802 Acked Read 4 Sent Done Done false
25911 7803 Acked Write 9 Sent Done Done false
25912 7804 Acked Write 7 Sent Done Done false
25913 7805 Acked Flush 0 Sent Done Done false
25914 7806 Acked Write 5 Sent Done Done false
25915 7807 Acked Write 9 Sent Done Done false
25916 7808 Acked Read 7 Sent Done Done false
25917 7809 Acked Write 6 Sent Done Done false
25918 7810 Acked Write 7 Sent Done Done false
25919 7811 Acked Read 6 Sent Done Done false
25920 7812 Acked Flush 0 Sent Done Done false
25921 7813 Acked Read 6 Sent Done Done false
25922 7814 Acked Flush 0 Sent Done Done false
25923 7815 Acked Read 1 Sent Done Done false
25924 7816 Acked Read 10 Sent Done Done false
25925 7817 Acked Write 1 Sent Done Done false
25926 7818 Acked Flush 0 Sent Done Done false
25927 7819 Acked Write 7 Sent Done Done false
25928 7820 Acked Read 3 Sent Done Done false
25929 7821 Acked Read 5 Sent Done Done false
25930 7822 Acked Read 10 Sent Done Done false
25931 7823 Acked Read 1 Sent Done Done false
25932 7824 Acked Read 10 Sent Done Done false
25933 7825 Acked Read 7 Sent Done Done false
25934 7826 Acked Write 6 Sent Done Done false
25935 7827 Acked Write 2 Sent Done Done false
25936 7828 Acked Read 6 Sent Done Done false
25937 7829 Acked Read 3 Sent Done Done false
25938 7830 Acked Write 6 Sent Done Done false
25939 7831 Acked Read 9 Sent Done Done false
25940 7832 Acked Read 9 Sent Done Done false
25941 7833 Acked Read 5 Sent Done Done false
25942 7834 Acked Read 7 Sent Done Done false
25943 7835 Acked Read 7 Sent Done Done false
25944 7836 Acked Write 3 Sent Done Done false
25945 7837 Acked Write 2 Sent Done Done false
25946 7838 Acked Write 4 Sent Done Done false
25947 7839 Acked Read 10 Sent Done Done false
25948 7840 Acked Read 7 Sent Done Done false
25949 7841 Acked Flush 0 Sent Done Done false
25950 7842 Acked Read 7 Sent Done Done false
25951 7843 Acked Write 3 Sent Done Done false
25952 7844 Acked Read 9 Sent Done Done false
25953 7845 Acked Read 1 Sent Done Done false
25954 7846 Acked Flush 0 Sent Done Done false
25955 7847 Acked Read 10 Sent Done Done false
25956 7848 Acked Flush 0 Sent Done Done false
25957 7849 Acked Write 9 Sent Done Done false
25958 7850 Acked Read 5 Sent Done Done false
25959 7851 Acked Read 10 Sent Done Done false
25960 7852 Acked Flush 0 Sent Done Done false
25961 7853 Acked Flush 0 Sent Done Done false
25962 7854 Acked Flush 0 Sent Done Done false
25963 7855 Acked Read 7 Sent Done Done false
25964 7856 Acked Flush 0 Sent Done Done false
25965 7857 Acked Read 10 Sent Done Done false
25966 7858 Acked Flush 0 Sent Done Done false
25967 7859 Acked Write 9 Sent Done Done false
25968 7860 Acked Read 10 Sent Done Done false
25969 7861 Acked Flush 0 Sent Done Done false
25970 7862 Acked Flush 0 Sent Done Done false
25971 7863 Acked Write 7 Sent Done Done false
25972 7864 Acked Write 2 Sent Done Done false
25973 7865 Acked Write 6 Sent Done Done false
25974 7866 Acked Read 7 Sent Done Done false
25975 7867 Acked Read 10 Sent Done Done false
25976 7868 Acked Write 2 Sent Done Done false
25977 7869 Acked Read 3 Sent Done Done false
25978 7870 Acked Read 6 Sent Done Done false
25979 7871 Acked Flush 0 Sent Done Done false
25980 7872 Acked Flush 0 Sent Done Done false
25981 7873 Acked Read 3 Sent Done Done false
25982 7874 Acked Read 10 Sent Done Done false
25983 7875 Acked Read 6 Sent Done Done false
25984 7876 Acked Read 7 Sent Done Done false
25985 7877 Acked Read 1 Sent Done Done false
25986 7878 Acked Write 2 Sent Done Done false
25987 7879 Acked Write 4 Sent Done Done false
25988 7880 Acked Read 6 Sent Done Done false
25989 7881 Acked Write 3 Sent Done Done false
25990 7882 Acked Read 9 Sent Done Done false
25991 7883 Acked Write 5 Sent Done Done false
25992 7884 Acked Write 7 Sent Done Done false
25993 7885 Acked Write 10 Sent Done Done false
25994 7886 Acked Read 4 Sent Done Done false
25995 7887 Acked Read 2 Sent Done Done false
25996 7888 Acked Read 4 Sent Done Done false
25997 7889 Acked Read 4 Sent Done Done false
25998 7890 Acked Write 5 Sent Done Done false
25999 7891 Acked Read 7 Sent Done Done false
26000 7892 Acked Flush 0 Sent Done Done false
26001 7893 Acked Read 10 Sent Done Done false
26002 7894 Acked Write 3 Sent Done Done false
26003 7895 Acked Flush 0 Sent Done Done false
26004 7896 Acked Read 4 Sent Done Done false
26005 7897 Acked Read 1 Sent Done Done false
26006 7898 Acked Write 10 Sent Done Done false
26007 7899 Acked Write 8 Sent Done Done false
26008 7900 Acked Write 7 Sent Done Done false
26009 7901 Acked Read 9 Sent Done Done false
26010 7902 Acked Write 4 Sent Done Done false
26011 7903 Acked Read 3 Sent Done Done false
26012 7904 Acked Read 5 Sent Done Done false
26013 7905 Acked Write 6 Sent Done Done false
26014 7906 Acked Read 5 Sent Done Done false
26015 7907 Acked Read 8 Sent Done Done false
26016 7908 Acked Write 1 Sent Done Done false
26017 7909 Acked Read 6 Sent Done Done false
26018 7910 Acked Write 8 Sent Done Done false
26019 7911 Acked Read 7 Sent Done Done false
26020 7912 Acked Write 5 Sent Done Done false
26021 7913 Acked Write 6 Sent Done Done false
26022 7914 Acked Write 4 Sent Done Done false
26023 7915 Acked Read 3 Sent Done Done false
26024 7916 Acked Read 8 Sent Done Done false
26025 7917 Acked Read 5 Sent Done Done false
26026 7918 Acked Read 6 Sent Done Done false
26027 7919 Acked Read 10 Sent Done Done false
26028 7920 Acked Read 1 Sent Done Done false
26029 7921 Acked Read 10 Sent Done Done false
26030 7922 Acked Read 9 Sent Done Done false
26031 7923 Acked Read 5 Sent Done Done false
26032 7924 Acked Read 2 Sent Done Done false
26033 7925 Acked Write 4 Sent Done Done false
26034 7926 Acked Read 4 Sent Done Done false
26035 7927 Acked Flush 0 Sent Done Done false
26036 7928 Acked Read 3 Sent Done Done false
26037 7929 Acked Flush 0 Sent Done Done false
26038 7930 Acked Read 1 Sent Done Done false
26039 7931 Acked Read 4 Sent Done Done false
26040 7932 Acked Read 3 Sent Done Done false
26041 7933 Acked Read 5 Sent Done Done false
26042 7934 Acked Read 6 Sent Done Done false
26043 7935 Acked Read 4 Sent Done Done false
26044 7936 Acked Write 9 Sent Done Done false
26045 7937 Acked Read 3 Sent Done Done false
26046 7938 Acked Read 3 Sent Done Done false
26047 7939 Acked Write 7 Sent Done Done false
26048 7940 Acked Write 2 Sent Done Done false
26049 7941 Acked Flush 0 Sent Done Done false
26050 7942 Acked Write 5 Sent Done Done false
26051 7943 Acked Write 4 Sent Done Done false
26052 7944 Acked Write 6 Sent Done Done false
26053 7945 Acked Write 6 Sent Done Done false
26054 7946 Acked Write 2 Sent Done Done false
26055 7947 Acked Read 3 Sent Done Done false
26056 7948 Acked Read 3 Sent Done Done false
26057 7949 Acked Read 9 Sent Done Done false
26058 7950 Acked Flush 0 Sent Done Done false
26059 STATES DS:0 DS:1 DS:2 TOTAL
26060 Sent 303 0 0 303
26061 Done 0 303 303 606
26062 Skipped 0 0 0 0
26063 Error 0 0 0 0
26064 Last Flush: 7647 7950 7950
26065 Downstairs last five completed: 7647 7646 7645 7644 7643
26066 Upstairs last five completed: 7950 7949 7948 7947 7946
26067 CLIENT: Up:0 ds:303 act:2
26068 ----------------------------------------------------------------
26069 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
26070 States: Offline Active Active
26071 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26072 7648 Acked Write 10 Sent Done Done false
26073 7649 Acked Write 6 Sent Done Done false
26074 7650 Acked Read 7 Sent Done Done false
26075 7651 Acked Read 1 Sent Done Done false
26076 7652 Acked Write 5 Sent Done Done false
26077 7653 Acked Write 9 Sent Done Done false
26078 7654 Acked Write 3 Sent Done Done false
26079 7655 Acked Read 9 Sent Done Done false
26080 7656 Acked Write 5 Sent Done Done false
26081 7657 Acked Read 5 Sent Done Done false
26082 7658 Acked Write 9 Sent Done Done false
26083 7659 Acked Flush 0 Sent Done Done false
26084 7660 Acked Read 7 Sent Done Done false
26085 7661 Acked Write 2 Sent Done Done false
26086 7662 Acked Read 8 Sent Done Done false
26087 7663 Acked Read 9 Sent Done Done false
26088 7664 Acked Write 2 Sent Done Done false
26089 7665 Acked Read 10 Sent Done Done false
26090 7666 Acked Read 7 Sent Done Done false
26091 7667 Acked Read 2 Sent Done Done false
26092 7668 Acked Write 5 Sent Done Done false
26093 7669 Acked Flush 0 Sent Done Done false
26094 7670 Acked Read 7 Sent Done Done false
26095 7671 Acked Write 10 Sent Done Done false
26096 7672 Acked Write 3 Sent Done Done false
26097 7673 Acked Read 2 Sent Done Done false
26098 7674 Acked Write 3 Sent Done Done false
26099 7675 Acked Read 5 Sent Done Done false
26100 7676 Acked Read 9 Sent Done Done false
26101 7677 Acked Read 9 Sent Done Done false
26102 7678 Acked Write 6 Sent Done Done false
26103 7679 Acked Read 3 Sent Done Done false
26104 7680 Acked Write 10 Sent Done Done false
26105 7681 Acked Read 4 Sent Done Done false
26106 7682 Acked Read 2 Sent Done Done false
26107 7683 Acked Write 1 Sent Done Done false
26108 7684 Acked Read 3 Sent Done Done false
26109 7685 Acked Write 1 Sent Done Done false
26110 7686 Acked Write 4 Sent Done Done false
26111 7687 Acked Read 3 Sent Done Done false
26112 7688 Acked Write 7 Sent Done Done false
26113 7689 Acked Write 4 Sent Done Done false
26114 7690 Acked Read 4 Sent Done Done false
26115 7691 Acked Write 7 Sent Done Done false
26116 7692 Acked Read 7 Sent Done Done false
26117 7693 Acked Read 4 Sent Done Done false
26118 7694 Acked Write 4 Sent Done Done false
26119 7695 Acked Read 9 Sent Done Done false
26120 7696 Acked Read 7 Sent Done Done false
26121 7697 Acked Write 4 Sent Done Done false
26122 7698 Acked Read 2 Sent Done Done false
26123 7699 Acked Write 6 Sent Done Done false
26124 7700 Acked Read 2 Sent Done Done false
26125 7701 Acked Read 5 Sent Done Done false
26126 7702 Acked Write 7 Sent Done Done false
26127 7703 Acked Read 8 Sent Done Done false
26128 7704 Acked Write 6 Sent Done Done false
26129 7705 Acked Read 5 Sent Done Done false
26130 7706 Acked Read 8 Sent Done Done false
26131 7707 Acked Read 5 Sent Done Done false
26132 7708 Acked Read 8 Sent Done Done false
26133 7709 Acked Write 5 Sent Done Done false
26134 7710 Acked Read 3 Sent Done Done false
26135 7711 Acked Read 6 Sent Done Done false
26136 7712 Acked Write 2 Sent Done Done false
26137 7713 Acked Read 1 Sent Done Done false
26138 7714 Acked Flush 0 Sent Done Done false
26139 7715 Acked Read 8 Sent Done Done false
26140 7716 Acked Flush 0 Sent Done Done false
26141 7717 Acked Read 4 Sent Done Done false
26142 7718 Acked Read 9 Sent Done Done false
26143 7719 Acked Write 5 Sent Done Done false
26144 7720 Acked Read 6 Sent Done Done false
26145 7721 Acked Flush 0 Sent Done Done false
26146 7722 Acked Write 10 Sent Done Done false
26147 7723 Acked Write 5 Sent Done Done false
26148 7724 Acked Write 3 Sent Done Done false
26149 7725 Acked Write 5 Sent Done Done false
26150 7726 Acked Write 9 Sent Done Done false
26151 7727 Acked Read 6 Sent Done Done false
26152 7728 Acked Read 5 Sent Done Done false
26153 7729 Acked Write 2 Sent Done Done false
26154 7730 Acked Write 3 Sent Done Done false
26155 7731 Acked Flush 0 Sent Done Done false
26156 7732 Acked Read 5 Sent Done Done false
26157 7733 Acked Read 10 Sent Done Done false
26158 7734 Acked Read 10 Sent Done Done false
26159 7735 Acked Read 5 Sent Done Done false
26160 7736 Acked Read 8 Sent Done Done false
26161 7737 Acked Read 7 Sent Done Done false
26162 7738 Acked Read 3 Sent Done Done false
26163 7739 Acked Flush 0 Sent Done Done false
26164 7740 Acked Write 5 Sent Done Done false
26165 7741 Acked Write 3 Sent Done Done false
26166 7742 Acked Write 9 Sent Done Done false
26167 7743 Acked Write 10 Sent Done Done false
26168 7744 Acked Read 5 Sent Done Done false
26169 7745 Acked Write 3 Sent Done Done false
26170 7746 Acked Write 9 Sent Done Done false
26171 7747 Acked Write 1 Sent Done Done false
26172 7748 Acked Write 2 Sent Done Done false
26173 7749 Acked Read 8 Sent Done Done false
26174 7750 Acked Write 2 Sent Done Done false
26175 7751 Acked Read 3 Sent Done Done false
26176 7752 Acked Read 5 Sent Done Done false
26177 7753 Acked Write 2 Sent Done Done false
26178 7754 Acked Write 2 Sent Done Done false
26179 7755 Acked Read 7 Sent Done Done false
26180 7756 Acked Read 5 Sent Done Done false
26181 7757 Acked Write 6 Sent Done Done false
26182 7758 Acked Write 6 Sent Done Done false
26183 7759 Acked Write 3 Sent Done Done false
26184 7760 Acked Write 5 Sent Done Done false
26185 7761 Acked Read 1 Sent Done Done false
26186 7762 Acked Write 5 Sent Done Done false
26187 7763 Acked Read 4 Sent Done Done false
26188 7764 Acked Read 3 Sent Done Done false
26189 7765 Acked Write 5 Sent Done Done false
26190 7766 Acked Write 7 Sent Done Done false
26191 7767 Acked Read 4 Sent Done Done false
26192 7768 Acked Read 8 Sent Done Done false
26193 7769 Acked Write 5 Sent Done Done false
26194 7770 Acked Write 10 Sent Done Done false
26195 7771 Acked Read 4 Sent Done Done false
26196 7772 Acked Read 2 Sent Done Done false
26197 7773 Acked Write 6 Sent Done Done false
26198 7774 Acked Flush 0 Sent Done Done false
26199 7775 Acked Read 10 Sent Done Done false
26200 7776 Acked Write 2 Sent Done Done false
26201 7777 Acked Read 3 Sent Done Done false
26202 7778 Acked Flush 0 Sent Done Done false
26203 7779 Acked Read 6 Sent Done Done false
26204 7780 Acked Write 10 Sent Done Done false
26205 7781 Acked Read 9 Sent Done Done false
26206 7782 Acked Read 7 Sent Done Done false
26207 7783 Acked Read 5 Sent Done Done false
26208 7784 Acked Write 6 Sent Done Done false
26209 7785 Acked Write 7 Sent Done Done false
26210 7786 Acked Read 8 Sent Done Done false
26211 7787 Acked Write 5 Sent Done Done false
26212 7788 Acked Read 8 Sent Done Done false
26213 7789 Acked Read 6 Sent Done Done false
26214 7790 Acked Write 7 Sent Done Done false
26215 7791 Acked Write 6 Sent Done Done false
26216 7792 Acked Read 3 Sent Done Done false
26217 7793 Acked Write 3 Sent Done Done false
26218 7794 Acked Write 4 Sent Done Done false
26219 7795 Acked Write 2 Sent Done Done false
26220 7796 Acked Read 1 Sent Done Done false
26221 7797 Acked Write 6 Sent Done Done false
26222 7798 Acked Write 7 Sent Done Done false
26223 7799 Acked Read 4 Sent Done Done false
26224 7800 Acked Read 7 Sent Done Done false
26225 7801 Acked Write 2 Sent Done Done false
26226 7802 Acked Read 4 Sent Done Done false
26227 7803 Acked Write 9 Sent Done Done false
26228 7804 Acked Write 7 Sent Done Done false
26229 7805 Acked Flush 0 Sent Done Done false
26230 7806 Acked Write 5 Sent Done Done false
26231 7807 Acked Write 9 Sent Done Done false
26232 7808 Acked Read 7 Sent Done Done false
26233 7809 Acked Write 6 Sent Done Done false
26234 7810 Acked Write 7 Sent Done Done false
26235 7811 Acked Read 6 Sent Done Done false
26236 7812 Acked Flush 0 Sent Done Done false
26237 7813 Acked Read 6 Sent Done Done false
26238 7814 Acked Flush 0 Sent Done Done false
26239 7815 Acked Read 1 Sent Done Done false
26240 7816 Acked Read 10 Sent Done Done false
26241 7817 Acked Write 1 Sent Done Done false
26242 7818 Acked Flush 0 Sent Done Done false
26243 7819 Acked Write 7 Sent Done Done false
26244 7820 Acked Read 3 Sent Done Done false
26245 7821 Acked Read 5 Sent Done Done false
26246 7822 Acked Read 10 Sent Done Done false
26247 7823 Acked Read 1 Sent Done Done false
26248 7824 Acked Read 10 Sent Done Done false
26249 7825 Acked Read 7 Sent Done Done false
26250 7826 Acked Write 6 Sent Done Done false
26251 7827 Acked Write 2 Sent Done Done false
26252 7828 Acked Read 6 Sent Done Done false
26253 7829 Acked Read 3 Sent Done Done false
26254 7830 Acked Write 6 Sent Done Done false
26255 7831 Acked Read 9 Sent Done Done false
26256 7832 Acked Read 9 Sent Done Done false
26257 7833 Acked Read 5 Sent Done Done false
26258 7834 Acked Read 7 Sent Done Done false
26259 7835 Acked Read 7 Sent Done Done false
26260 7836 Acked Write 3 Sent Done Done false
26261 7837 Acked Write 2 Sent Done Done false
26262 7838 Acked Write 4 Sent Done Done false
26263 7839 Acked Read 10 Sent Done Done false
26264 7840 Acked Read 7 Sent Done Done false
26265 7841 Acked Flush 0 Sent Done Done false
26266 7842 Acked Read 7 Sent Done Done false
26267 7843 Acked Write 3 Sent Done Done false
26268 7844 Acked Read 9 Sent Done Done false
26269 7845 Acked Read 1 Sent Done Done false
26270 7846 Acked Flush 0 Sent Done Done false
26271 7847 Acked Read 10 Sent Done Done false
26272 7848 Acked Flush 0 Sent Done Done false
26273 7849 Acked Write 9 Sent Done Done false
26274 7850 Acked Read 5 Sent Done Done false
26275 7851 Acked Read 10 Sent Done Done false
26276 7852 Acked Flush 0 Sent Done Done false
26277 7853 Acked Flush 0 Sent Done Done false
26278 7854 Acked Flush 0 Sent Done Done false
26279 7855 Acked Read 7 Sent Done Done false
26280 7856 Acked Flush 0 Sent Done Done false
26281 7857 Acked Read 10 Sent Done Done false
26282 7858 Acked Flush 0 Sent Done Done false
26283 7859 Acked Write 9 Sent Done Done false
26284 7860 Acked Read 10 Sent Done Done false
26285 7861 Acked Flush 0 Sent Done Done false
26286 7862 Acked Flush 0 Sent Done Done false
26287 7863 Acked Write 7 Sent Done Done false
26288 7864 Acked Write 2 Sent Done Done false
26289 7865 Acked Write 6 Sent Done Done false
26290 7866 Acked Read 7 Sent Done Done false
26291 7867 Acked Read 10 Sent Done Done false
26292 7868 Acked Write 2 Sent Done Done false
26293 7869 Acked Read 3 Sent Done Done false
26294 7870 Acked Read 6 Sent Done Done false
26295 7871 Acked Flush 0 Sent Done Done false
26296 7872 Acked Flush 0 Sent Done Done false
26297 7873 Acked Read 3 Sent Done Done false
26298 7874 Acked Read 10 Sent Done Done false
26299 7875 Acked Read 6 Sent Done Done false
26300 7876 Acked Read 7 Sent Done Done false
26301 7877 Acked Read 1 Sent Done Done false
26302 7878 Acked Write 2 Sent Done Done false
26303 7879 Acked Write 4 Sent Done Done false
26304 7880 Acked Read 6 Sent Done Done false
26305 7881 Acked Write 3 Sent Done Done false
26306 7882 Acked Read 9 Sent Done Done false
26307 7883 Acked Write 5 Sent Done Done false
26308 7884 Acked Write 7 Sent Done Done false
26309 7885 Acked Write 10 Sent Done Done false
26310 7886 Acked Read 4 Sent Done Done false
26311 7887 Acked Read 2 Sent Done Done false
26312 7888 Acked Read 4 Sent Done Done false
26313 7889 Acked Read 4 Sent Done Done false
26314 7890 Acked Write 5 Sent Done Done false
26315 7891 Acked Read 7 Sent Done Done false
26316 7892 Acked Flush 0 Sent Done Done false
26317 7893 Acked Read 10 Sent Done Done false
26318 7894 Acked Write 3 Sent Done Done false
26319 7895 Acked Flush 0 Sent Done Done false
26320 7896 Acked Read 4 Sent Done Done false
26321 7897 Acked Read 1 Sent Done Done false
26322 7898 Acked Write 10 Sent Done Done false
26323 7899 Acked Write 8 Sent Done Done false
26324 7900 Acked Write 7 Sent Done Done false
26325 7901 Acked Read 9 Sent Done Done false
26326 7902 Acked Write 4 Sent Done Done false
26327 7903 Acked Read 3 Sent Done Done false
26328 7904 Acked Read 5 Sent Done Done false
26329 7905 Acked Write 6 Sent Done Done false
26330 7906 Acked Read 5 Sent Done Done false
26331 7907 Acked Read 8 Sent Done Done false
26332 7908 Acked Write 1 Sent Done Done false
26333 7909 Acked Read 6 Sent Done Done false
26334 7910 Acked Write 8 Sent Done Done false
26335 7911 Acked Read 7 Sent Done Done false
26336 7912 Acked Write 5 Sent Done Done false
26337 7913 Acked Write 6 Sent Done Done false
26338 7914 Acked Write 4 Sent Done Done false
26339 7915 Acked Read 3 Sent Done Done false
26340 7916 Acked Read 8 Sent Done Done false
26341 7917 Acked Read 5 Sent Done Done false
26342 7918 Acked Read 6 Sent Done Done false
26343 7919 Acked Read 10 Sent Done Done false
26344 7920 Acked Read 1 Sent Done Done false
26345 7921 Acked Read 10 Sent Done Done false
26346 7922 Acked Read 9 Sent Done Done false
26347 7923 Acked Read 5 Sent Done Done false
26348 7924 Acked Read 2 Sent Done Done false
26349 7925 Acked Write 4 Sent Done Done false
26350 7926 Acked Read 4 Sent Done Done false
26351 7927 Acked Flush 0 Sent Done Done false
26352 7928 Acked Read 3 Sent Done Done false
26353 7929 Acked Flush 0 Sent Done Done false
26354 7930 Acked Read 1 Sent Done Done false
26355 7931 Acked Read 4 Sent Done Done false
26356 7932 Acked Read 3 Sent Done Done false
26357 7933 Acked Read 5 Sent Done Done false
26358 7934 Acked Read 6 Sent Done Done false
26359 7935 Acked Read 4 Sent Done Done false
26360 7936 Acked Write 9 Sent Done Done false
26361 7937 Acked Read 3 Sent Done Done false
26362 7938 Acked Read 3 Sent Done Done false
26363 7939 Acked Write 7 Sent Done Done false
26364 7940 Acked Write 2 Sent Done Done false
26365 7941 Acked Flush 0 Sent Done Done false
26366 7942 Acked Write 5 Sent Done Done false
26367 7943 Acked Write 4 Sent Done Done false
26368 7944 Acked Write 6 Sent Done Done false
26369 7945 Acked Write 6 Sent Done Done false
26370 7946 Acked Write 2 Sent Done Done false
26371 7947 Acked Read 3 Sent Done Done false
26372 7948 Acked Read 3 Sent Done Done false
26373 7949 Acked Read 9 Sent Done Done false
26374 7950 Acked Flush 0 Sent Done Done false
26375 STATES DS:0 DS:1 DS:2 TOTAL
26376 Sent 303 0 0 303
26377 Done 0 303 303 606
26378 Skipped 0 0 0 0
26379 Error 0 0 0 0
26380 Last Flush: 7647 7950 7950
26381 Downstairs last five completed: 7647 7646 7645 7644 7643
26382 Upstairs last five completed: 7950 7949 7948 7947 7946
26383 CLIENT: Up:0 ds:303 act:2
263842024-11-01T16:56:02.853ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
263852024-11-01T16:56:02.854ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65378) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
263862024-11-01T16:56:02.855ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
263872024-11-01T16:56:02.855ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
263882024-11-01T16:56:02.855ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
263892024-11-01T16:56:02.855ZINFOcrucible: send last flush ID to this DS: 7647 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
263902024-11-01T16:56:02.856ZINFOcrucible: Replied this last flush ID: 7647 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
263912024-11-01T16:56:02.856ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
263922024-11-01T16:56:02.856ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
263932024-11-01T16:56:02.856ZINFOcrucible: [0] client re-new 303 jobs since flush 7647 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
263942024-11-01T16:56:02.856ZINFOcrucible: [0] Replayed 303 jobs since flush: 7647 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
26395 ----------------------------------------------------------------
26396 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26397 Downstairs last five completed: 7950 7949 7948 7947 7946
26398 Upstairs last five completed: 7950 7949 7948 7947 7946
26399 CLIENT: Up:0 ds:0 act:3
26400 Replay: All jobs finished
26401 Main task has work to do, go find it
26402 got dsc Stop(0)
26403 stop 0
26404 Nov 01 16:56:03.147 INFO request completed[8810] Got stop action so:false kr:false
26405 , latency_us: 353, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 07103a57-b9f4-4949-a44f-4f17d141a4b4, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
264062024-11-01T16:56:03.148ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
264072024-11-01T16:56:03.148ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
264082024-11-01T16:56:03.148ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
264092024-11-01T16:56:03.148ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
264102024-11-01T16:56:03.148ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
26411 Nov 01 16:56:03.148 INFO request completed, latency_us: 174, response_code: 200, uri: /state/cid/0, method: GET, req_id: d97dea56-bd4f-401c-bc73-40073c485d9b, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
26412 [8810] Exited with: ExitStatus(unix_wait_status(9))
26413 [8810][0] reports Exit
26414 Nov 01 16:56:07.151 INFO request completed, latency_us: 250, response_code: 200, uri: /state/cid/0, method: GET, req_id: 3fcc11f3-5376-4247-99d5-6d77f54f93ba, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
26415 001/300 Read block 716 len 16384
26416 002/300 Write block 4954 len 20480 data: 6 5 6 5 5
26417 003/300 Read block 4594 len 8192
26418 004/300 Write block 4677 len 8192 data: 9 7
26419 005/300 Read block 2387 len 4096
26420 006/300 Read block 2194 len 32768
26421 007/300 Read block 4393 len 36864
26422 008/300 Read block 2478 len 32768
26423 009/300 Read block 3394 len 8192
26424 010/300 Flush
26425 011/300 Read block 1114 len 20480
26426 012/300 Write block 3251 len 32768 data: 6 4 4 5 7 6 7 6
26427 013/300 Flush
26428 014/300 Read block 3425 len 4096
26429 015/300 Write block 3781 len 12288 data: 5 5 3
26430 016/300 Read block 589 len 28672
26431 017/300 Read block 81 len 8192
26432 018/300 Read block 2278 len 16384
26433 019/300 Write block 4535 len 24576 data: 7 6 7 8 7 6
26434 020/300 Write block 447 len 32768 data: 5 5 5 4 5 5 4 5
26435 021/300 Write block 2135 len 24576 data: 5 6 5 5 4 4
26436 022/300 Write block 3253 len 16384 data: 5 6 8 7
26437 023/300 Read block 4938 len 24576
26438 024/300 Read block 3805 len 4096
26439 025/300 Read block 1783 len 32768
26440 026/300 Read block 1704 len 8192
26441 027/300 Read block 4669 len 20480
26442 028/300 Read block 2043 len 40960
26443 029/300 Read block 306 len 40960
26444 030/300 Read block 4607 len 40960
26445 031/300 Write block 1547 len 20480 data: 4 4 5 4 7
26446 032/300 Read block 256 len 24576
26447 033/300 Write block 119 len 8192 data: 5 7
26448 034/300 Write block 789 len 24576 data: 3 3 3 4 5 5
26449 035/300 Write block 3285 len 28672 data: 3 4 5 6 6 8 7
26450 036/300 Write block 3482 len 8192 data: 5 6
26451 037/300 Read block 73 len 36864
26452 038/300 Read block 2667 len 4096
26453 039/300 Read block 602 len 8192
26454 040/300 Read block 2628 len 20480
26455 041/300 Read block 1240 len 28672
26456 042/300 Read block 918 len 28672
26457 043/300 Read block 2074 len 28672
26458 044/300 Flush
26459 045/300 Read block 3144 len 12288
26460 046/300 Read block 1068 len 24576
26461 047/300 Read block 1978 len 24576
26462 048/300 Write block 1757 len 32768 data: 2 2 2 4 6 6 6 6
26463 049/300 Read block 545 len 16384
26464 050/300 Write block 341 len 24576 data: 2 3 3 5 5 5
26465 051/300 Write block 3377 len 40960 data: 4 4 3 4 4 6 7 7 7 7
26466 052/300 Read block 1213 len 12288
26467 053/300 Write block 4981 len 8192 data: 6 6
26468 054/300 Read block 2049 len 24576
26469 055/300 Read block 1792 len 36864
26470 056/300 Write block 2531 len 12288 data: 4 4 3
26471 057/300 Read block 1631 len 12288
26472 058/300 Write block 3933 len 36864 data: 7 7 7 6 5 4 2 3 5
26473 059/300 Write block 1237 len 8192 data: 6 6
26474 060/300 Read block 3487 len 32768
26475 061/300 Flush
26476 062/300 Write block 718 len 40960 data: 4 3 3 3 3 4 3 2 3 4
26477 063/300 Read block 2878 len 28672
26478 064/300 Read block 3128 len 40960
26479 065/300 Read block 3456 len 40960
26480 066/300 Write block 2486 len 4096 data: 6
26481 067/300 Read block 3768 len 12288
26482 068/300 Read block 3942 len 8192
26483 069/300 Read block 3165 len 36864
26484 070/300 Write block 3750 len 28672 data: 7 7 8 8 8 7 7
26485 071/300 Read block 1661 len 20480
26486 072/300 Write block 1225 len 16384 data: 6 4 4 3
26487 073/300 Read block 4237 len 16384
26488 074/300 Read block 4520 len 40960
26489 075/300 Write block 147 len 16384 data: 4 3 3 3
26490 076/300 Read block 2872 len 28672
26491 077/300 Read block 4227 len 20480
26492 078/300 Write block 1013 len 36864 data: 3 2 2 2 3 4 4 5 4
26493 079/300 Write block 3172 len 28672 data: 7 10 9 7 6 5 5
26494 080/300 Read block 4583 len 12288
26495 081/300 Read block 3922 len 12288
26496 082/300 Read block 490 len 24576
26497 083/300 Write block 302 len 36864 data: 5 5 5 5 6 5 4 3 4
26498 084/300 Read block 3774 len 4096
26499 085/300 Read block 3022 len 32768
26500 086/300 Flush
26501 087/300 Write block 2596 len 32768 data: 4 5 5 5 5 5 5 5
26502 088/300 Read block 4638 len 4096
26503 089/300 Write block 3111 len 36864 data: 7 7 5 3 2 3 3 5 5
26504 090/300 Read block 463 len 4096
26505 091/300 Read block 4106 len 20480
26506 092/300 Read block 4631 len 32768
26507 093/300 Write block 4686 len 4096 data: 7
26508 094/300 Read block 2134 len 4096
26509 095/300 Read block 3872 len 8192
26510 096/300 Write block 4610 len 8192 data: 4 4
26511 097/300 Read block 4921 len 40960
26512 098/300 Read block 4130 len 36864
26513 099/300 Read block 2419 len 24576
26514 100/300 Read block 2867 len 16384
26515 101/300 Read block 84 len 24576
26516 102/300 Write block 2652 len 20480 data: 6 5 4 4 5
26517 103/300 Read block 410 len 4096
26518 104/300 Read block 1527 len 36864
26519 105/300 Read block 4243 len 32768
26520 106/300 Read block 857 len 16384
26521 107/300 Write block 430 len 12288 data: 5 5 5
26522 108/300 Read block 2341 len 24576
26523 109/300 Flush
26524 110/300 Write block 4230 len 24576 data: 4 4 4 3 4 3
26525 111/300 Read block 4768 len 28672
26526 112/300 Read block 3160 len 28672
26527 113/300 Write block 1783 len 32768 data: 7 7 7 8 7 7 6 7
26528 114/300 Read block 4769 len 4096
26529 115/300 Read block 4378 len 36864
26530 116/300 Read block 3767 len 4096
26531 117/300 Read block 2852 len 4096
26532 118/300 Write block 3721 len 40960 data: 3 5 5 4 5 4 4 5 5 4
26533 119/300 Read block 1465 len 24576
26534 120/300 Write block 169 len 40960 data: 6 6 6 6 6 6 5 5 4 3
26535 121/300 Write block 1657 len 16384 data: 3 4 3 5
26536 122/300 Read block 2077 len 12288
26537 123/300 Write block 4974 len 16384 data: 4 4 4 4
26538 124/300 Read block 41 len 36864
26539 125/300 Read block 2188 len 24576
26540 126/300 Read block 1342 len 32768
26541 127/300 Write block 3145 len 8192 data: 3 4
26542 128/300 Read block 4110 len 16384
26543 129/300 Read block 3027 len 40960
26544 130/300 Read block 910 len 36864
26545 131/300 Read block 4519 len 16384
26546 132/300 Write block 2022 len 20480 data: 3 2 3 3 3
26547 133/300 Write block 4159 len 40960 data: 4 4 4 4 5 5 6 4 4 2
26548 134/300 Write block 924 len 12288 data: 5 7 8
26549 135/300 Read block 2084 len 36864
26550 136/300 Read block 297 len 24576
26551 137/300 Flush
26552 138/300 Write block 3204 len 4096 data: 8
26553 139/300 Read block 2074 len 8192
26554 140/300 Write block 3825 len 8192 data: 6 7
26555 141/300 Write block 3155 len 32768 data: 5 3 3 3 2 3 3 4
26556 142/300 Read block 708 len 28672
26557 143/300 Read block 3008 len 8192
26558 144/300 Write block 4649 len 36864 data: 2 3 5 5 6 6 4 3 3
26559 145/300 Read block 161 len 40960
26560 146/300 Read block 62 len 8192
26561 147/300 Flush
26562 148/300 Flush
26563 149/300 Flush
26564 150/300 Read block 859 len 4096
26565 151/300 Write block 4224 len 28672 data: 5 6 6 6 6 4 5
26566 152/300 Read block 3264 len 40960
26567 153/300 Read block 2594 len 8192
26568 154/300 Write block 2425 len 20480 data: 6 2 3 5 4
26569 155/300 Read block 4072 len 24576
26570 156/300 Write block 1116 len 8192 data: 5 5
26571 157/300 Read block 3490 len 24576
26572 158/300 Read block 1753 len 24576
26573 159/300 Read block 1368 len 12288
26574 160/300 Write block 3858 len 28672 data: 7 8 10 9 8 8 9
26575 161/300 Read block 2666 len 32768
26576 162/300 Read block 3261 len 28672
26577 163/300 Flush
26578 164/300 Read block 1523 len 12288
26579 165/300 Read block 3734 len 40960
26580 166/300 Read block 15 len 8192
26581 167/300 Read block 1032 len 12288
26582 168/300 Write block 1877 len 28672 data: 9 8 8 7 6 4 5
26583 169/300 Read block 1671 len 24576
26584 170/300 Write block 1215 len 24576 data: 7 6 6 8 6 6
26585 171/300 Write block 322 len 28672 data: 2 2 2 2 2 2 2
26586 172/300 Read block 1280 len 8192
26587 173/300 Write block 77 len 24576 data: 5 4 4 4 6 5
26588 174/300 Write block 56 len 20480 data: 4 4 4 3 3
26589 175/300 Write block 2222 len 20480 data: 8 9 8 7 7
26590 176/300 Write block 833 len 24576 data: 5 6 6 5 4 5
26591 177/300 Write block 457 len 40960 data: 7 7 5 5 4 6 4 3 3 3
26592 178/300 Write block 3800 len 12288 data: 9 7 8
26593 179/300 Write block 3467 len 32768 data: 6 6 6 7 9 7 7 7
26594 180/300 Write block 1136 len 20480 data: 5 6 8 7 6
26595 181/300 Flush
26596 182/300 Read block 3132 len 16384
26597 183/300 Write block 3711 len 12288 data: 3 4 4
26598 184/300 Flush
26599 185/300 Write block 4806 len 28672 data: 3 4 5 6 5 5 5
26600 186/300 Read block 2887 len 24576
26601 187/300 Write block 1199 len 20480 data: 7 7 9 9 9
26602 188/300 Flush
26603 189/300 Read block 4589 len 24576
26604 190/300 Write block 602 len 32768 data: 4 4 4 3 3 5 5 5
26605 191/300 Read block 4986 len 28672
26606 192/300 Write block 421 len 32768 data: 4 4 4 3 5 5 4 5
26607 193/300 Read block 3330 len 40960
26608 194/300 Read block 1847 len 12288
26609 195/300 Read block 1350 len 24576
26610 196/300 Write block 3124 len 12288 data: 7 5 5
26611 197/300 Read block 4426 len 28672
26612 198/300 Read block 1306 len 8192
26613 199/300 Read block 4126 len 40960
26614 200/300 Read block 401 len 20480
26615 201/300 Read block 3495 len 24576
26616 202/300 Flush
26617 203/300 Flush
26618 204/300 Read block 3188 len 24576
26619 205/300 Read block 1748 len 28672
26620 206/300 Read block 1660 len 8192
26621 207/300 Read block 1358 len 16384
26622 208/300 Read block 320 len 40960
26623 209/300 Read block 2541 len 4096
26624 210/300 Read block 204 len 24576
26625 211/300 Write block 3304 len 36864 data: 4 4 4 5 6 6 5 3 2
26626 212/300 Read block 213 len 28672
26627 213/300 Read block 3733 len 40960
26628 214/300 Read block 2929 len 32768
26629 215/300 Write block 1480 len 4096 data: 6
26630 216/300 Write block 215 len 16384 data: 3 5 5 5
26631 217/300 Write block 910 len 32768 data: 6 5 5 5 5 6 6 7
26632 218/300 Write block 3029 len 24576 data: 4 4 5 5 5 4
26633 219/300 Read block 4675 len 12288
26634 220/300 Write block 365 len 24576 data: 7 6 5 4 6 7
26635 221/300 Write block 4233 len 8192 data: 4 5
26636 222/300 Read block 2592 len 32768
26637 223/300 Read block 1206 len 28672
26638 224/300 Read block 2490 len 24576
26639 225/300 Read block 6 len 28672
26640 226/300 Write block 285 len 32768 data: 5 7 4 3 3 4 4 5
26641 227/300 Read block 1616 len 8192
26642 228/300 Read block 4188 len 8192
26643 229/300 Write block 3165 len 24576 data: 5 3 3 4 4 4
26644 230/300 Write block 2085 len 32768 data: 7 9 9 8 8 6 9 10
26645 231/300 Write block 2696 len 32768 data: 2 2 3 3 3 3 3 4
26646 232/300 Read block 4528 len 12288
26647 233/300 Flush
26648 234/300 Write block 3454 len 28672 data: 4 5 4 4 6 8 9
26649 235/300 Read block 1273 len 4096
26650 236/300 Write block 1105 len 36864 data: 5 5 5 4 2 2 2 2 3
26651 237/300 Read block 4881 len 40960
26652 238/300 Write block 2161 len 36864 data: 4 5 5 6 6 5 5 3 3
26653 239/300 Read block 2839 len 28672
26654 240/300 Write block 4209 len 32768 data: 7 7 5 5 4 3 4 5
26655 241/300 Write block 1780 len 8192 data: 3 5
26656 242/300 Write block 663 len 40960 data: 5 5 5 3 3 2 2 4 4 4
26657 243/300 Read block 1645 len 28672
26658 244/300 Read block 4092 len 12288
26659 245/300 Flush
26660 246/300 Write block 749 len 28672 data: 5 3 4 6 7 7 7
26661 247/300 Write block 677 len 12288 data: 5 5 4
26662 248/300 Write block 4454 len 4096 data: 6
26663 249/300 Read block 2598 len 40960
26664 250/300 Write block 4269 len 40960 data: 4 5 4 5 4 3 4 5 5 5
26665 251/300 Write block 3321 len 4096 data: 5
26666 252/300 Read block 468 len 12288
26667 253/300 Read block 3471 len 32768
26668 254/300 Write block 3584 len 40960 data: 8 8 7 7 8 7 8 6 7 5
26669 255/300 Read block 4500 len 20480
26670 256/300 Read block 3044 len 28672
26671 257/300 Read block 2337 len 36864
26672 258/300 Read block 798 len 4096
26673 259/300 Read block 1236 len 24576
26674 260/300 Read block 4377 len 24576
26675 261/300 Read block 2430 len 24576
26676 262/300 Flush
26677 263/300 Read block 330 len 4096
26678 264/300 Write block 2731 len 28672 data: 4 5 4 4 3 5 5
26679 265/300 Write block 1297 len 24576 data: 6 7 6 7 5 5
26680 266/300 Read block 1239 len 24576
26681 267/300 Flush
26682 268/300 Flush
26683 269/300 Write block 1202 len 36864 data: 10 10 9 9 9 9 9 8 7
26684 270/300 Read block 1543 len 40960
26685 271/300 Read block 3794 len 24576
26686 272/300 Write block 734 len 8192 data: 8 8
26687 273/300 Write block 3959 len 8192 data: 5 4
26688 274/300 Read block 4586 len 16384
26689 275/300 Write block 98 len 24576 data: 8 8 6 6 5 5
26690 276/300 Flush
26691 277/300 Write block 2732 len 16384 data: 6 5 5 4
26692 278/300 Flush
26693 279/300 Write block 2997 len 8192 data: 4 5
26694 280/300 Write block 928 len 8192 data: 5 6
26695 281/300 Read block 2801 len 8192
26696 282/300 Read block 2681 len 4096
26697 283/300 Flush
26698 284/300 Flush
26699 285/300 Read block 3954 len 36864
26700 286/300 Write block 3077 len 20480 data: 8 7 5 5 6
26701 287/300 Flush
26702 288/300 Read block 846 len 32768
26703 289/300 Read block 1589 len 40960
26704 290/300 Write block 1194 len 24576 data: 7 8 8 7 6 8
26705 291/300 Write block 4749 len 12288 data: 5 5 4
26706 292/300 Read block 776 len 24576
26707 293/300 Read block 74 len 12288
26708 294/300 Read block 1882 len 24576
26709 295/300 Flush
26710 296/300 Read block 3036 len 8192
26711 297/300 Read block 1318 len 12288
26712 298/300 Read block 2903 len 40960
26713 299/300 Read block 4812 len 4096
26714 300/300 Write block 3625 len 32768 data: 5 5 5 4 4 4 4 4
26715 301/300 Read block 4226 len 36864
26716 Main task has work to do, go find it
26717 got dsc Start(0)
26718 start 0
26719 Nov 01 16:56:07.446 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/0, method: POST, req_id: becc3e58-7cf0-4236-8439-f66ea9961d85, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
26720 Replay: started 0, returned:Ok(())
26721 ----------------------------------------------------------------
26722 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26723 States: Offline Active Active
26724 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26725 7951 Acked Read 4 Sent Done Done false
26726 7952 Acked Write 5 Sent Done Done false
26727 7953 Acked Read 2 Sent Done Done false
26728 7954 Acked Write 2 Sent Done Done false
26729 7955 Acked Read 1 Sent Done Done false
26730 7956 Acked Read 8 Sent Done Done false
26731 7957 Acked Read 9 Sent Done Done false
26732 7958 Acked Read 8 Sent Done Done false
26733 7959 Acked Read 2 Sent Done Done false
26734 7960 Acked Flush 0 Sent Done Done false
26735 7961 Acked Read 5 Sent Done Done false
26736 7962 Acked Write 8 Sent Done Done false
26737 7963 Acked Flush 0 Sent Done Done false
26738 7964 Acked Read 1 Sent Done Done false
26739 7965 Acked Write 3 Sent Done Done false
26740 7966 Acked Read 7 Sent Done Done false
26741 7967 Acked Read 2 Sent Done Done false
26742 7968 Acked Read 4 Sent Done Done false
26743 7969 Acked Write 6 Sent Done Done false
26744 7970 Acked Write 8 Sent Done Done false
26745 7971 Acked Write 6 Sent Done Done false
26746 7972 Acked Write 4 Sent Done Done false
26747 7973 Acked Read 6 Sent Done Done false
26748 7974 Acked Read 1 Sent Done Done false
26749 7975 Acked Read 8 Sent Done Done false
26750 7976 Acked Read 2 Sent Done Done false
26751 7977 Acked Read 5 Sent Done Done false
26752 7978 Acked Read 10 Sent Done Done false
26753 7979 Acked Read 10 Sent Done Done false
26754 7980 Acked Read 10 Sent Done Done false
26755 7981 Acked Write 5 Sent Done Done false
26756 7982 Acked Read 6 Sent Done Done false
26757 7983 Acked Write 2 Sent Done Done false
26758 7984 Acked Write 6 Sent Done Done false
26759 7985 Acked Write 7 Sent Done Done false
26760 7986 Acked Write 2 Sent Done Done false
26761 7987 Acked Read 9 Sent Done Done false
26762 7988 Acked Read 1 Sent Done Done false
26763 7989 Acked Read 2 Sent Done Done false
26764 7990 Acked Read 5 Sent Done Done false
26765 7991 Acked Read 7 Sent Done Done false
26766 7992 Acked Read 7 Sent Done Done false
26767 7993 Acked Read 7 Sent Done Done false
26768 7994 Acked Flush 0 Sent Done Done false
26769 7995 Acked Read 3 Sent Done Done false
26770 7996 Acked Read 6 Sent Done Done false
26771 7997 Acked Read 6 Sent Done Done false
26772 7998 Acked Write 8 Sent Done Done false
26773 7999 Acked Read 4 Sent Done Done false
26774 8000 Acked Write 6 Sent Done Done false
26775 8001 Acked Write 10 Sent Done Done false
26776 8002 Acked Read 3 Sent Done Done false
26777 8003 Acked Write 2 Sent Done Done false
26778 8004 Acked Read 6 Sent Done Done false
26779 8005 Acked Read 9 Sent Done Done false
26780 8006 Acked Write 3 Sent Done Done false
26781 8007 Acked Read 3 Sent Done Done false
26782 8008 Acked Write 9 Sent Done Done false
26783 8009 Acked Write 2 Sent Done Done false
26784 8010 Acked Read 8 Sent Done Done false
26785 8011 Acked Flush 0 Sent Done Done false
26786 8012 Acked Write 10 Sent Done Done false
26787 8013 Acked Read 7 Sent Done Done false
26788 8014 Acked Read 10 Sent Done Done false
26789 8015 Acked Read 10 Sent Done Done false
26790 8016 Acked Write 1 Sent Done Done false
26791 8017 Acked Read 3 Sent Done Done false
26792 8018 Acked Read 2 Sent Done Done false
26793 8019 Acked Read 9 Sent Done Done false
26794 8020 Acked Write 7 Sent Done Done false
26795 8021 Acked Read 5 Sent Done Done false
26796 8022 Acked Write 4 Sent Done Done false
26797 8023 Acked Read 4 Sent Done Done false
26798 8024 Acked Read 10 Sent Done Done false
26799 8025 Acked Write 4 Sent Done Done false
26800 8026 Acked Read 7 Sent Done Done false
26801 8027 Acked Read 5 Sent Done Done false
26802 8028 Acked Write 9 Sent Done Done false
26803 8029 Acked Write 7 Sent Done Done false
26804 8030 Acked Read 3 Sent Done Done false
26805 8031 Acked Read 3 Sent Done Done false
26806 8032 Acked Read 6 Sent Done Done false
26807 8033 Acked Write 9 Sent Done Done false
26808 8034 Acked Read 1 Sent Done Done false
26809 8035 Acked Read 8 Sent Done Done false
26810 8036 Acked Flush 0 Sent Done Done false
26811 8037 Acked Write 8 Sent Done Done false
26812 8038 Acked Read 1 Sent Done Done false
26813 8039 Acked Write 9 Sent Done Done false
26814 8040 Acked Read 1 Sent Done Done false
26815 8041 Acked Read 5 Sent Done Done false
26816 8042 Acked Read 8 Sent Done Done false
26817 8043 Acked Write 1 Sent Done Done false
26818 8044 Acked Read 1 Sent Done Done false
26819 8045 Acked Read 2 Sent Done Done false
26820 8046 Acked Write 2 Sent Done Done false
26821 8047 Acked Read 10 Sent Done Done false
26822 8048 Acked Read 9 Sent Done Done false
26823 8049 Acked Read 6 Sent Done Done false
26824 8050 Acked Read 4 Sent Done Done false
26825 8051 Acked Read 6 Sent Done Done false
26826 8052 Acked Write 5 Sent Done Done false
26827 8053 Acked Read 1 Sent Done Done false
26828 8054 Acked Read 9 Sent Done Done false
26829 8055 Acked Read 8 Sent Done Done false
26830 8056 Acked Read 4 Sent Done Done false
26831 8057 Acked Write 3 Sent Done Done false
26832 8058 Acked Read 6 Sent Done Done false
26833 8059 Acked Flush 0 Sent Done Done false
26834 8060 Acked Write 6 Sent Done Done false
26835 8061 Acked Read 7 Sent Done Done false
26836 8062 Acked Read 7 Sent Done Done false
26837 8063 Acked Write 8 Sent Done Done false
26838 8064 Acked Read 1 Sent Done Done false
26839 8065 Acked Read 9 Sent Done Done false
26840 8066 Acked Read 1 Sent Done Done false
26841 8067 Acked Read 1 Sent Done Done false
26842 8068 Acked Write 10 Sent Done Done false
26843 8069 Acked Read 6 Sent Done Done false
26844 8070 Acked Write 10 Sent Done Done false
26845 8071 Acked Write 4 Sent Done Done false
26846 8072 Acked Read 3 Sent Done Done false
26847 8073 Acked Write 4 Sent Done Done false
26848 8074 Acked Read 9 Sent Done Done false
26849 8075 Acked Read 6 Sent Done Done false
26850 8076 Acked Read 8 Sent Done Done false
26851 8077 Acked Write 2 Sent Done Done false
26852 8078 Acked Read 4 Sent Done Done false
26853 8079 Acked Read 10 Sent Done Done false
26854 8080 Acked Read 9 Sent Done Done false
26855 8081 Acked Read 4 Sent Done Done false
26856 8082 Acked Write 5 Sent Done Done false
26857 8083 Acked Write 10 Sent Done Done false
26858 8084 Acked Write 3 Sent Done Done false
26859 8085 Acked Read 9 Sent Done Done false
26860 8086 Acked Read 6 Sent Done Done false
26861 8087 Acked Flush 0 Sent Done Done false
26862 8088 Acked Write 1 Sent Done Done false
26863 8089 Acked Read 2 Sent Done Done false
26864 8090 Acked Write 2 Sent Done Done false
26865 8091 Acked Write 8 Sent Done Done false
26866 8092 Acked Read 7 Sent Done Done false
26867 8093 Acked Read 2 Sent Done Done false
26868 8094 Acked Write 9 Sent Done Done false
26869 8095 Acked Read 10 Sent Done Done false
26870 8096 Acked Read 2 Sent Done Done false
26871 8097 Acked Flush 0 Sent Done Done false
26872 8098 Acked Flush 0 Sent Done Done false
26873 8099 Acked Flush 0 Sent Done Done false
26874 8100 Acked Read 1 Sent Done Done false
26875 8101 Acked Write 7 Sent Done Done false
26876 8102 Acked Read 10 Sent Done Done false
26877 8103 Acked Read 2 Sent Done Done false
26878 8104 Acked Write 5 Sent Done Done false
26879 8105 Acked Read 6 Sent Done Done false
26880 8106 Acked Write 2 Sent Done Done false
26881 8107 Acked Read 6 Sent Done Done false
26882 8108 Acked Read 6 Sent Done Done false
26883 8109 Acked Read 3 Sent Done Done false
26884 8110 Acked Write 7 Sent Done Done false
26885 8111 Acked Read 8 Sent Done Done false
26886 8112 Acked Read 7 Sent Done Done false
26887 8113 Acked Flush 0 Sent Done Done false
26888 8114 Acked Read 3 Sent Done Done false
26889 8115 Acked Read 10 Sent Done Done false
26890 8116 Acked Read 2 Sent Done Done false
26891 8117 Acked Read 3 Sent Done Done false
26892 8118 Acked Write 7 Sent Done Done false
26893 8119 Acked Read 6 Sent Done Done false
26894 8120 Acked Write 6 Sent Done Done false
26895 8121 Acked Write 7 Sent Done Done false
26896 8122 Acked Read 2 Sent Done Done false
26897 8123 Acked Write 6 Sent Done Done false
26898 8124 Acked Write 5 Sent Done Done false
26899 8125 Acked Write 5 Sent Done Done false
26900 8126 Acked Write 6 Sent Done Done false
26901 8127 Acked Write 10 Sent Done Done false
26902 8128 Acked Write 3 Sent Done Done false
26903 8129 Acked Write 8 Sent Done Done false
26904 8130 Acked Write 5 Sent Done Done false
26905 8131 Acked Flush 0 Sent Done Done false
26906 8132 Acked Read 4 Sent Done Done false
26907 8133 Acked Write 3 Sent Done Done false
26908 8134 Acked Flush 0 Sent Done Done false
26909 8135 Acked Write 7 Sent Done Done false
26910 8136 Acked Read 6 Sent Done Done false
26911 8137 Acked Write 5 Sent Done Done false
26912 8138 Acked Flush 0 Sent Done Done false
26913 8139 Acked Read 6 Sent Done Done false
26914 8140 Acked Write 8 Sent Done Done false
26915 8141 Acked Read 7 Sent Done Done false
26916 8142 Acked Write 8 Sent Done Done false
26917 8143 Acked Read 10 Sent Done Done false
26918 8144 Acked Read 3 Sent Done Done false
26919 8145 Acked Read 6 Sent Done Done false
26920 8146 Acked Write 3 Sent Done Done false
26921 8147 Acked Read 7 Sent Done Done false
26922 8148 Acked Read 2 Sent Done Done false
26923 8149 Acked Read 10 Sent Done Done false
26924 8150 Acked Read 5 Sent Done Done false
26925 8151 Acked Read 6 Sent Done Done false
26926 8152 Acked Flush 0 Sent Done Done false
26927 8153 Acked Flush 0 Sent Done Done false
26928 8154 Acked Read 6 Sent Done Done false
26929 8155 Acked Read 7 Sent Done Done false
26930 8156 Acked Read 2 Sent Done Done false
26931 8157 Acked Read 4 Sent Done Done false
26932 8158 Acked Read 10 Sent Done Done false
26933 8159 Acked Read 1 Sent Done Done false
26934 8160 Acked Read 6 Sent Done Done false
26935 8161 Acked Write 9 Sent Done Done false
26936 8162 Acked Read 7 Sent Done Done false
26937 8163 Acked Read 10 Sent Done Done false
26938 8164 Acked Read 8 Sent Done Done false
26939 8165 Acked Write 1 Sent Done Done false
26940 8166 Acked Write 4 Sent Done Done false
26941 8167 Acked Write 8 Sent Done Done false
26942 8168 Acked Write 6 Sent Done Done false
26943 8169 Acked Read 3 Sent Done Done false
26944 8170 Acked Write 6 Sent Done Done false
26945 8171 Acked Write 2 Sent Done Done false
26946 8172 Acked Read 8 Sent Done Done false
26947 8173 Acked Read 7 Sent Done Done false
26948 8174 Acked Read 6 Sent Done Done false
26949 8175 Acked Read 7 Sent Done Done false
26950 8176 Acked Write 8 Sent Done Done false
26951 8177 Acked Read 2 Sent Done Done false
26952 8178 Acked Read 2 Sent Done Done false
26953 8179 Acked Write 6 Sent Done Done false
26954 8180 Acked Write 8 Sent Done Done false
26955 8181 Acked Write 8 Sent Done Done false
26956 8182 Acked Read 3 Sent Done Done false
26957 8183 Acked Flush 0 Sent Done Done false
26958 8184 Acked Write 7 Sent Done Done false
26959 8185 Acked Read 1 Sent Done Done false
26960 8186 Acked Write 9 Sent Done Done false
26961 8187 Acked Read 10 Sent Done Done false
26962 8188 Acked Write 9 Sent Done Done false
26963 8189 Acked Read 7 Sent Done Done false
26964 8190 Acked Write 8 Sent Done Done false
26965 8191 Acked Write 2 Sent Done Done false
26966 8192 Acked Write 10 Sent Done Done false
26967 8193 Acked Read 7 Sent Done Done false
26968 8194 Acked Read 3 Sent Done Done false
26969 8195 Acked Flush 0 Sent Done Done false
26970 8196 Acked Write 7 Sent Done Done false
26971 8197 Acked Write 3 Sent Done Done false
26972 8198 Acked Write 1 Sent Done Done false
26973 8199 Acked Read 10 Sent Done Done false
26974 8200 Acked Write 10 Sent Done Done false
26975 8201 Acked Write 1 Sent Done Done false
26976 8202 Acked Read 3 Sent Done Done false
26977 8203 Acked Read 8 Sent Done Done false
26978 8204 Acked Write 10 Sent Done Done false
26979 8205 Acked Read 5 Sent Done Done false
26980 8206 Acked Read 7 Sent Done Done false
26981 8207 Acked Read 9 Sent Done Done false
26982 8208 Acked Read 1 Sent Done Done false
26983 8209 Acked Read 6 Sent Done Done false
26984 8210 Acked Read 6 Sent Done Done false
26985 8211 Acked Read 6 Sent Done Done false
26986 8212 Acked Flush 0 Sent Done Done false
26987 8213 Acked Read 1 Sent Done Done false
26988 8214 Acked Write 7 Sent Done Done false
26989 8215 Acked Write 6 Sent Done Done false
26990 8216 Acked Read 6 Sent Done Done false
26991 8217 Acked Flush 0 Sent Done Done false
26992 8218 Acked Flush 0 Sent Done Done false
26993 8219 Acked Write 9 Sent Done Done false
26994 8220 Acked Read 10 Sent Done Done false
26995 8221 Acked Read 6 Sent Done Done false
26996 8222 Acked Write 2 Sent Done Done false
26997 8223 Acked Write 2 Sent Done Done false
26998 8224 Acked Read 4 Sent Done Done false
26999 8225 Acked Write 6 Sent Done Done false
27000 8226 Acked Flush 0 Sent Done Done false
27001 8227 Acked Write 4 Sent Done Done false
27002 8228 Acked Flush 0 Sent Done Done false
27003 8229 Acked Write 2 Sent Done Done false
27004 8230 Acked Write 2 Sent Done Done false
27005 8231 Acked Read 2 Sent Done Done false
27006 8232 Acked Read 1 Sent Done Done false
27007 8233 Acked Flush 0 Sent Done Done false
27008 8234 Acked Flush 0 Sent Done Done false
27009 8235 Acked Read 9 Sent Done Done false
27010 8236 Acked Write 5 Sent Done Done false
27011 8237 Acked Flush 0 Sent Done Done false
27012 8238 Acked Read 8 Sent Done Done false
27013 8239 Acked Read 10 Sent Done Done false
27014 8240 Acked Write 6 Sent Done Done false
27015 8241 Acked Write 3 Sent Done Done false
27016 8242 Acked Read 6 Sent Done Done false
27017 8243 Acked Read 3 Sent Done Done false
27018 8244 Acked Read 6 Sent Done Done false
27019 8245 Acked Flush 0 Sent Done Done false
27020 8246 Acked Read 2 Sent Done Done false
27021 8247 Acked Read 3 Sent Done Done false
27022 8248 Acked Read 10 Sent Done Done false
27023 8249 Acked Read 1 Sent Done Done false
27024 8250 Acked Write 8 Sent Done Done false
27025 8251 Acked Read 9 Sent Done Done false
27026 STATES DS:0 DS:1 DS:2 TOTAL
27027 Sent 301 0 0 301
27028 Done 0 301 301 602
27029 Skipped 0 0 0 0
27030 Error 0 0 0 0
27031 Last Flush: 7950 8245 8245
27032 Downstairs last five completed: 7950 7949 7948 7947 7946
27033 Upstairs last five completed: 8251 8250 8249 8248 8247
27034 CLIENT: Up:0 ds:301 act:2
27035 ----------------------------------------------------------------
27036 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
27037 States: Offline Active Active
27038 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27039 7951 Acked Read 4 Sent Done Done false
27040 7952 Acked Write 5 Sent Done Done false
27041 7953 Acked Read 2 Sent Done Done false
27042 7954 Acked Write 2 Sent Done Done false
27043 7955 Acked Read 1 Sent Done Done false
27044 7956 Acked Read 8 Sent Done Done false
27045 7957 Acked Read 9 Sent Done Done false
27046 7958 Acked Read 8 Sent Done Done false
27047 7959 Acked Read 2 Sent Done Done false
27048 7960 Acked Flush 0 Sent Done Done false
27049 7961 Acked Read 5 Sent Done Done false
27050 7962 Acked Write 8 Sent Done Done false
27051 7963 Acked Flush 0 Sent Done Done false
27052 7964 Acked Read 1 Sent Done Done false
27053 7965 Acked Write 3 Sent Done Done false
27054 7966 Acked Read 7 Sent Done Done false
27055 7967 Acked Read 2 Sent Done Done false
27056 7968 Acked Read 4 Sent Done Done false
27057 7969 Acked Write 6 Sent Done Done false
27058 7970 Acked Write 8 Sent Done Done false
27059 7971 Acked Write 6 Sent Done Done false
27060 7972 Acked Write 4 Sent Done Done false
27061 7973 Acked Read 6 Sent Done Done false
27062 7974 Acked Read 1 Sent Done Done false
27063 7975 Acked Read 8 Sent Done Done false
27064 7976 Acked Read 2 Sent Done Done false
27065 7977 Acked Read 5 Sent Done Done false
27066 7978 Acked Read 10 Sent Done Done false
27067 7979 Acked Read 10 Sent Done Done false
27068 7980 Acked Read 10 Sent Done Done false
27069 7981 Acked Write 5 Sent Done Done false
27070 7982 Acked Read 6 Sent Done Done false
27071 7983 Acked Write 2 Sent Done Done false
27072 7984 Acked Write 6 Sent Done Done false
27073 7985 Acked Write 7 Sent Done Done false
27074 7986 Acked Write 2 Sent Done Done false
27075 7987 Acked Read 9 Sent Done Done false
27076 7988 Acked Read 1 Sent Done Done false
27077 7989 Acked Read 2 Sent Done Done false
27078 7990 Acked Read 5 Sent Done Done false
27079 7991 Acked Read 7 Sent Done Done false
27080 7992 Acked Read 7 Sent Done Done false
27081 7993 Acked Read 7 Sent Done Done false
27082 7994 Acked Flush 0 Sent Done Done false
27083 7995 Acked Read 3 Sent Done Done false
27084 7996 Acked Read 6 Sent Done Done false
27085 7997 Acked Read 6 Sent Done Done false
27086 7998 Acked Write 8 Sent Done Done false
27087 7999 Acked Read 4 Sent Done Done false
27088 8000 Acked Write 6 Sent Done Done false
27089 8001 Acked Write 10 Sent Done Done false
27090 8002 Acked Read 3 Sent Done Done false
27091 8003 Acked Write 2 Sent Done Done false
27092 8004 Acked Read 6 Sent Done Done false
27093 8005 Acked Read 9 Sent Done Done false
27094 8006 Acked Write 3 Sent Done Done false
27095 8007 Acked Read 3 Sent Done Done false
27096 8008 Acked Write 9 Sent Done Done false
27097 8009 Acked Write 2 Sent Done Done false
27098 8010 Acked Read 8 Sent Done Done false
27099 8011 Acked Flush 0 Sent Done Done false
27100 8012 Acked Write 10 Sent Done Done false
27101 8013 Acked Read 7 Sent Done Done false
27102 8014 Acked Read 10 Sent Done Done false
27103 8015 Acked Read 10 Sent Done Done false
27104 8016 Acked Write 1 Sent Done Done false
27105 8017 Acked Read 3 Sent Done Done false
27106 8018 Acked Read 2 Sent Done Done false
27107 8019 Acked Read 9 Sent Done Done false
27108 8020 Acked Write 7 Sent Done Done false
27109 8021 Acked Read 5 Sent Done Done false
27110 8022 Acked Write 4 Sent Done Done false
27111 8023 Acked Read 4 Sent Done Done false
27112 8024 Acked Read 10 Sent Done Done false
27113 8025 Acked Write 4 Sent Done Done false
27114 8026 Acked Read 7 Sent Done Done false
27115 8027 Acked Read 5 Sent Done Done false
27116 8028 Acked Write 9 Sent Done Done false
27117 8029 Acked Write 7 Sent Done Done false
27118 8030 Acked Read 3 Sent Done Done false
27119 8031 Acked Read 3 Sent Done Done false
27120 8032 Acked Read 6 Sent Done Done false
27121 8033 Acked Write 9 Sent Done Done false
27122 8034 Acked Read 1 Sent Done Done false
27123 8035 Acked Read 8 Sent Done Done false
27124 8036 Acked Flush 0 Sent Done Done false
27125 8037 Acked Write 8 Sent Done Done false
27126 8038 Acked Read 1 Sent Done Done false
27127 8039 Acked Write 9 Sent Done Done false
27128 8040 Acked Read 1 Sent Done Done false
27129 8041 Acked Read 5 Sent Done Done false
27130 8042 Acked Read 8 Sent Done Done false
27131 8043 Acked Write 1 Sent Done Done false
27132 8044 Acked Read 1 Sent Done Done false
27133 8045 Acked Read 2 Sent Done Done false
27134 8046 Acked Write 2 Sent Done Done false
27135 8047 Acked Read 10 Sent Done Done false
27136 8048 Acked Read 9 Sent Done Done false
27137 8049 Acked Read 6 Sent Done Done false
27138 8050 Acked Read 4 Sent Done Done false
27139 8051 Acked Read 6 Sent Done Done false
27140 8052 Acked Write 5 Sent Done Done false
27141 8053 Acked Read 1 Sent Done Done false
27142 8054 Acked Read 9 Sent Done Done false
27143 8055 Acked Read 8 Sent Done Done false
27144 8056 Acked Read 4 Sent Done Done false
27145 8057 Acked Write 3 Sent Done Done false
27146 8058 Acked Read 6 Sent Done Done false
27147 8059 Acked Flush 0 Sent Done Done false
27148 8060 Acked Write 6 Sent Done Done false
27149 8061 Acked Read 7 Sent Done Done false
27150 8062 Acked Read 7 Sent Done Done false
27151 8063 Acked Write 8 Sent Done Done false
27152 8064 Acked Read 1 Sent Done Done false
27153 8065 Acked Read 9 Sent Done Done false
27154 8066 Acked Read 1 Sent Done Done false
27155 8067 Acked Read 1 Sent Done Done false
27156 8068 Acked Write 10 Sent Done Done false
27157 8069 Acked Read 6 Sent Done Done false
27158 8070 Acked Write 10 Sent Done Done false
27159 8071 Acked Write 4 Sent Done Done false
27160 8072 Acked Read 3 Sent Done Done false
27161 8073 Acked Write 4 Sent Done Done false
27162 8074 Acked Read 9 Sent Done Done false
27163 8075 Acked Read 6 Sent Done Done false
27164 8076 Acked Read 8 Sent Done Done false
27165 8077 Acked Write 2 Sent Done Done false
27166 8078 Acked Read 4 Sent Done Done false
27167 8079 Acked Read 10 Sent Done Done false
27168 8080 Acked Read 9 Sent Done Done false
27169 8081 Acked Read 4 Sent Done Done false
27170 8082 Acked Write 5 Sent Done Done false
27171 8083 Acked Write 10 Sent Done Done false
27172 8084 Acked Write 3 Sent Done Done false
27173 8085 Acked Read 9 Sent Done Done false
27174 8086 Acked Read 6 Sent Done Done false
27175 8087 Acked Flush 0 Sent Done Done false
27176 8088 Acked Write 1 Sent Done Done false
27177 8089 Acked Read 2 Sent Done Done false
27178 8090 Acked Write 2 Sent Done Done false
27179 8091 Acked Write 8 Sent Done Done false
27180 8092 Acked Read 7 Sent Done Done false
27181 8093 Acked Read 2 Sent Done Done false
27182 8094 Acked Write 9 Sent Done Done false
27183 8095 Acked Read 10 Sent Done Done false
27184 8096 Acked Read 2 Sent Done Done false
27185 8097 Acked Flush 0 Sent Done Done false
27186 8098 Acked Flush 0 Sent Done Done false
27187 8099 Acked Flush 0 Sent Done Done false
27188 8100 Acked Read 1 Sent Done Done false
27189 8101 Acked Write 7 Sent Done Done false
27190 8102 Acked Read 10 Sent Done Done false
27191 8103 Acked Read 2 Sent Done Done false
27192 8104 Acked Write 5 Sent Done Done false
27193 8105 Acked Read 6 Sent Done Done false
27194 8106 Acked Write 2 Sent Done Done false
27195 8107 Acked Read 6 Sent Done Done false
27196 8108 Acked Read 6 Sent Done Done false
27197 8109 Acked Read 3 Sent Done Done false
27198 8110 Acked Write 7 Sent Done Done false
27199 8111 Acked Read 8 Sent Done Done false
27200 8112 Acked Read 7 Sent Done Done false
27201 8113 Acked Flush 0 Sent Done Done false
27202 8114 Acked Read 3 Sent Done Done false
27203 8115 Acked Read 10 Sent Done Done false
27204 8116 Acked Read 2 Sent Done Done false
27205 8117 Acked Read 3 Sent Done Done false
27206 8118 Acked Write 7 Sent Done Done false
27207 8119 Acked Read 6 Sent Done Done false
27208 8120 Acked Write 6 Sent Done Done false
27209 8121 Acked Write 7 Sent Done Done false
27210 8122 Acked Read 2 Sent Done Done false
27211 8123 Acked Write 6 Sent Done Done false
27212 8124 Acked Write 5 Sent Done Done false
27213 8125 Acked Write 5 Sent Done Done false
27214 8126 Acked Write 6 Sent Done Done false
27215 8127 Acked Write 10 Sent Done Done false
27216 8128 Acked Write 3 Sent Done Done false
27217 8129 Acked Write 8 Sent Done Done false
27218 8130 Acked Write 5 Sent Done Done false
27219 8131 Acked Flush 0 Sent Done Done false
27220 8132 Acked Read 4 Sent Done Done false
27221 8133 Acked Write 3 Sent Done Done false
27222 8134 Acked Flush 0 Sent Done Done false
27223 8135 Acked Write 7 Sent Done Done false
27224 8136 Acked Read 6 Sent Done Done false
27225 8137 Acked Write 5 Sent Done Done false
27226 8138 Acked Flush 0 Sent Done Done false
27227 8139 Acked Read 6 Sent Done Done false
27228 8140 Acked Write 8 Sent Done Done false
27229 8141 Acked Read 7 Sent Done Done false
27230 8142 Acked Write 8 Sent Done Done false
27231 8143 Acked Read 10 Sent Done Done false
27232 8144 Acked Read 3 Sent Done Done false
27233 8145 Acked Read 6 Sent Done Done false
27234 8146 Acked Write 3 Sent Done Done false
27235 8147 Acked Read 7 Sent Done Done false
27236 8148 Acked Read 2 Sent Done Done false
27237 8149 Acked Read 10 Sent Done Done false
27238 8150 Acked Read 5 Sent Done Done false
27239 8151 Acked Read 6 Sent Done Done false
27240 8152 Acked Flush 0 Sent Done Done false
27241 8153 Acked Flush 0 Sent Done Done false
27242 8154 Acked Read 6 Sent Done Done false
27243 8155 Acked Read 7 Sent Done Done false
27244 8156 Acked Read 2 Sent Done Done false
27245 8157 Acked Read 4 Sent Done Done false
27246 8158 Acked Read 10 Sent Done Done false
27247 8159 Acked Read 1 Sent Done Done false
27248 8160 Acked Read 6 Sent Done Done false
27249 8161 Acked Write 9 Sent Done Done false
27250 8162 Acked Read 7 Sent Done Done false
27251 8163 Acked Read 10 Sent Done Done false
27252 8164 Acked Read 8 Sent Done Done false
27253 8165 Acked Write 1 Sent Done Done false
27254 8166 Acked Write 4 Sent Done Done false
27255 8167 Acked Write 8 Sent Done Done false
27256 8168 Acked Write 6 Sent Done Done false
27257 8169 Acked Read 3 Sent Done Done false
27258 8170 Acked Write 6 Sent Done Done false
27259 8171 Acked Write 2 Sent Done Done false
27260 8172 Acked Read 8 Sent Done Done false
27261 8173 Acked Read 7 Sent Done Done false
27262 8174 Acked Read 6 Sent Done Done false
27263 8175 Acked Read 7 Sent Done Done false
27264 8176 Acked Write 8 Sent Done Done false
27265 8177 Acked Read 2 Sent Done Done false
27266 8178 Acked Read 2 Sent Done Done false
27267 8179 Acked Write 6 Sent Done Done false
27268 8180 Acked Write 8 Sent Done Done false
27269 8181 Acked Write 8 Sent Done Done false
27270 8182 Acked Read 3 Sent Done Done false
27271 8183 Acked Flush 0 Sent Done Done false
27272 8184 Acked Write 7 Sent Done Done false
27273 8185 Acked Read 1 Sent Done Done false
27274 8186 Acked Write 9 Sent Done Done false
27275 8187 Acked Read 10 Sent Done Done false
27276 8188 Acked Write 9 Sent Done Done false
27277 8189 Acked Read 7 Sent Done Done false
27278 8190 Acked Write 8 Sent Done Done false
27279 8191 Acked Write 2 Sent Done Done false
27280 8192 Acked Write 10 Sent Done Done false
27281 8193 Acked Read 7 Sent Done Done false
27282 8194 Acked Read 3 Sent Done Done false
27283 8195 Acked Flush 0 Sent Done Done false
27284 8196 Acked Write 7 Sent Done Done false
27285 8197 Acked Write 3 Sent Done Done false
27286 8198 Acked Write 1 Sent Done Done false
27287 8199 Acked Read 10 Sent Done Done false
27288 8200 Acked Write 10 Sent Done Done false
27289 8201 Acked Write 1 Sent Done Done false
27290 8202 Acked Read 3 Sent Done Done false
27291 8203 Acked Read 8 Sent Done Done false
27292 8204 Acked Write 10 Sent Done Done false
27293 8205 Acked Read 5 Sent Done Done false
27294 8206 Acked Read 7 Sent Done Done false
27295 8207 Acked Read 9 Sent Done Done false
27296 8208 Acked Read 1 Sent Done Done false
27297 8209 Acked Read 6 Sent Done Done false
27298 8210 Acked Read 6 Sent Done Done false
27299 8211 Acked Read 6 Sent Done Done false
27300 8212 Acked Flush 0 Sent Done Done false
27301 8213 Acked Read 1 Sent Done Done false
27302 8214 Acked Write 7 Sent Done Done false
27303 8215 Acked Write 6 Sent Done Done false
27304 8216 Acked Read 6 Sent Done Done false
27305 8217 Acked Flush 0 Sent Done Done false
27306 8218 Acked Flush 0 Sent Done Done false
27307 8219 Acked Write 9 Sent Done Done false
27308 8220 Acked Read 10 Sent Done Done false
27309 8221 Acked Read 6 Sent Done Done false
27310 8222 Acked Write 2 Sent Done Done false
27311 8223 Acked Write 2 Sent Done Done false
27312 8224 Acked Read 4 Sent Done Done false
27313 8225 Acked Write 6 Sent Done Done false
27314 8226 Acked Flush 0 Sent Done Done false
27315 8227 Acked Write 4 Sent Done Done false
27316 8228 Acked Flush 0 Sent Done Done false
27317 8229 Acked Write 2 Sent Done Done false
27318 8230 Acked Write 2 Sent Done Done false
27319 8231 Acked Read 2 Sent Done Done false
27320 8232 Acked Read 1 Sent Done Done false
27321 8233 Acked Flush 0 Sent Done Done false
27322 8234 Acked Flush 0 Sent Done Done false
27323 8235 Acked Read 9 Sent Done Done false
27324 8236 Acked Write 5 Sent Done Done false
27325 8237 Acked Flush 0 Sent Done Done false
27326 8238 Acked Read 8 Sent Done Done false
27327 8239 Acked Read 10 Sent Done Done false
27328 8240 Acked Write 6 Sent Done Done false
27329 8241 Acked Write 3 Sent Done Done false
27330 8242 Acked Read 6 Sent Done Done false
27331 8243 Acked Read 3 Sent Done Done false
27332 8244 Acked Read 6 Sent Done Done false
27333 8245 Acked Flush 0 Sent Done Done false
27334 8246 Acked Read 2 Sent Done Done false
27335 8247 Acked Read 3 Sent Done Done false
27336 8248 Acked Read 10 Sent Done Done false
27337 8249 Acked Read 1 Sent Done Done false
27338 8250 Acked Write 8 Sent Done Done false
27339 8251 Acked Read 9 Sent Done Done false
27340 8252 Acked Flush 0 Sent Done Done false
27341 STATES DS:0 DS:1 DS:2 TOTAL
27342 Sent 302 0 0 302
27343 Done 0 302 302 604
27344 Skipped 0 0 0 0
27345 Error 0 0 0 0
27346 Last Flush: 7950 8252 8252
27347 Downstairs last five completed: 7950 7949 7948 7947 7946
27348 Upstairs last five completed: 8252 8251 8250 8249 8248
27349 CLIENT: Up:0 ds:302 act:2
273502024-11-01T16:56:13.149ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
273512024-11-01T16:56:13.149ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
273522024-11-01T16:56:13.149ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
273532024-11-01T16:56:13.149ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
273542024-11-01T16:56:13.149ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
27355 [8810] got start action
27356 [8810] I am going to restart
27357 Starting downstairs at port 8810
27358 Make output file at "/tmp/dsc/downstairs-8810.txt"
27359 [8810][0] reports Starting
27360 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1222)
27361 [8810][0] reports Running
27362 ----------------------------------------------------------------
27363 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
27364 States: Offline Active Active
27365 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27366 7951 Acked Read 4 Sent Done Done false
27367 7952 Acked Write 5 Sent Done Done false
27368 7953 Acked Read 2 Sent Done Done false
27369 7954 Acked Write 2 Sent Done Done false
27370 7955 Acked Read 1 Sent Done Done false
27371 7956 Acked Read 8 Sent Done Done false
27372 7957 Acked Read 9 Sent Done Done false
27373 7958 Acked Read 8 Sent Done Done false
27374 7959 Acked Read 2 Sent Done Done false
27375 7960 Acked Flush 0 Sent Done Done false
27376 7961 Acked Read 5 Sent Done Done false
27377 7962 Acked Write 8 Sent Done Done false
27378 7963 Acked Flush 0 Sent Done Done false
27379 7964 Acked Read 1 Sent Done Done false
27380 7965 Acked Write 3 Sent Done Done false
27381 7966 Acked Read 7 Sent Done Done false
27382 7967 Acked Read 2 Sent Done Done false
27383 7968 Acked Read 4 Sent Done Done false
27384 7969 Acked Write 6 Sent Done Done false
27385 7970 Acked Write 8 Sent Done Done false
27386 7971 Acked Write 6 Sent Done Done false
27387 7972 Acked Write 4 Sent Done Done false
27388 7973 Acked Read 6 Sent Done Done false
27389 7974 Acked Read 1 Sent Done Done false
27390 7975 Acked Read 8 Sent Done Done false
27391 7976 Acked Read 2 Sent Done Done false
27392 7977 Acked Read 5 Sent Done Done false
27393 7978 Acked Read 10 Sent Done Done false
27394 7979 Acked Read 10 Sent Done Done false
27395 7980 Acked Read 10 Sent Done Done false
27396 7981 Acked Write 5 Sent Done Done false
27397 7982 Acked Read 6 Sent Done Done false
27398 7983 Acked Write 2 Sent Done Done false
27399 7984 Acked Write 6 Sent Done Done false
27400 7985 Acked Write 7 Sent Done Done false
27401 7986 Acked Write 2 Sent Done Done false
27402 7987 Acked Read 9 Sent Done Done false
27403 7988 Acked Read 1 Sent Done Done false
27404 7989 Acked Read 2 Sent Done Done false
27405 7990 Acked Read 5 Sent Done Done false
27406 7991 Acked Read 7 Sent Done Done false
27407 7992 Acked Read 7 Sent Done Done false
27408 7993 Acked Read 7 Sent Done Done false
27409 7994 Acked Flush 0 Sent Done Done false
27410 7995 Acked Read 3 Sent Done Done false
27411 7996 Acked Read 6 Sent Done Done false
27412 7997 Acked Read 6 Sent Done Done false
27413 7998 Acked Write 8 Sent Done Done false
27414 7999 Acked Read 4 Sent Done Done false
27415 8000 Acked Write 6 Sent Done Done false
27416 8001 Acked Write 10 Sent Done Done false
27417 8002 Acked Read 3 Sent Done Done false
27418 8003 Acked Write 2 Sent Done Done false
27419 8004 Acked Read 6 Sent Done Done false
27420 8005 Acked Read 9 Sent Done Done false
27421 8006 Acked Write 3 Sent Done Done false
27422 8007 Acked Read 3 Sent Done Done false
27423 8008 Acked Write 9 Sent Done Done false
27424 8009 Acked Write 2 Sent Done Done false
27425 8010 Acked Read 8 Sent Done Done false
27426 8011 Acked Flush 0 Sent Done Done false
27427 8012 Acked Write 10 Sent Done Done false
27428 8013 Acked Read 7 Sent Done Done false
27429 8014 Acked Read 10 Sent Done Done false
27430 8015 Acked Read 10 Sent Done Done false
27431 8016 Acked Write 1 Sent Done Done false
27432 8017 Acked Read 3 Sent Done Done false
27433 8018 Acked Read 2 Sent Done Done false
27434 8019 Acked Read 9 Sent Done Done false
27435 8020 Acked Write 7 Sent Done Done false
27436 8021 Acked Read 5 Sent Done Done false
27437 8022 Acked Write 4 Sent Done Done false
27438 8023 Acked Read 4 Sent Done Done false
27439 8024 Acked Read 10 Sent Done Done false
27440 8025 Acked Write 4 Sent Done Done false
27441 8026 Acked Read 7 Sent Done Done false
27442 8027 Acked Read 5 Sent Done Done false
27443 8028 Acked Write 9 Sent Done Done false
27444 8029 Acked Write 7 Sent Done Done false
27445 8030 Acked Read 3 Sent Done Done false
27446 8031 Acked Read 3 Sent Done Done false
27447 8032 Acked Read 6 Sent Done Done false
27448 8033 Acked Write 9 Sent Done Done false
27449 8034 Acked Read 1 Sent Done Done false
27450 8035 Acked Read 8 Sent Done Done false
27451 8036 Acked Flush 0 Sent Done Done false
27452 8037 Acked Write 8 Sent Done Done false
27453 8038 Acked Read 1 Sent Done Done false
27454 8039 Acked Write 9 Sent Done Done false
27455 8040 Acked Read 1 Sent Done Done false
27456 8041 Acked Read 5 Sent Done Done false
27457 8042 Acked Read 8 Sent Done Done false
27458 8043 Acked Write 1 Sent Done Done false
27459 8044 Acked Read 1 Sent Done Done false
27460 8045 Acked Read 2 Sent Done Done false
27461 8046 Acked Write 2 Sent Done Done false
27462 8047 Acked Read 10 Sent Done Done false
27463 8048 Acked Read 9 Sent Done Done false
27464 8049 Acked Read 6 Sent Done Done false
27465 8050 Acked Read 4 Sent Done Done false
27466 8051 Acked Read 6 Sent Done Done false
27467 8052 Acked Write 5 Sent Done Done false
27468 8053 Acked Read 1 Sent Done Done false
27469 8054 Acked Read 9 Sent Done Done false
27470 8055 Acked Read 8 Sent Done Done false
27471 8056 Acked Read 4 Sent Done Done false
27472 8057 Acked Write 3 Sent Done Done false
27473 8058 Acked Read 6 Sent Done Done false
27474 8059 Acked Flush 0 Sent Done Done false
27475 8060 Acked Write 6 Sent Done Done false
27476 8061 Acked Read 7 Sent Done Done false
27477 8062 Acked Read 7 Sent Done Done false
27478 8063 Acked Write 8 Sent Done Done false
27479 8064 Acked Read 1 Sent Done Done false
27480 8065 Acked Read 9 Sent Done Done false
27481 8066 Acked Read 1 Sent Done Done false
27482 8067 Acked Read 1 Sent Done Done false
27483 8068 Acked Write 10 Sent Done Done false
27484 8069 Acked Read 6 Sent Done Done false
27485 8070 Acked Write 10 Sent Done Done false
27486 8071 Acked Write 4 Sent Done Done false
27487 8072 Acked Read 3 Sent Done Done false
27488 8073 Acked Write 4 Sent Done Done false
27489 8074 Acked Read 9 Sent Done Done false
27490 8075 Acked Read 6 Sent Done Done false
27491 8076 Acked Read 8 Sent Done Done false
27492 8077 Acked Write 2 Sent Done Done false
27493 8078 Acked Read 4 Sent Done Done false
27494 8079 Acked Read 10 Sent Done Done false
27495 8080 Acked Read 9 Sent Done Done false
27496 8081 Acked Read 4 Sent Done Done false
27497 8082 Acked Write 5 Sent Done Done false
27498 8083 Acked Write 10 Sent Done Done false
27499 8084 Acked Write 3 Sent Done Done false
27500 8085 Acked Read 9 Sent Done Done false
27501 8086 Acked Read 6 Sent Done Done false
27502 8087 Acked Flush 0 Sent Done Done false
27503 8088 Acked Write 1 Sent Done Done false
27504 8089 Acked Read 2 Sent Done Done false
27505 8090 Acked Write 2 Sent Done Done false
27506 8091 Acked Write 8 Sent Done Done false
27507 8092 Acked Read 7 Sent Done Done false
27508 8093 Acked Read 2 Sent Done Done false
27509 8094 Acked Write 9 Sent Done Done false
27510 8095 Acked Read 10 Sent Done Done false
27511 8096 Acked Read 2 Sent Done Done false
27512 8097 Acked Flush 0 Sent Done Done false
27513 8098 Acked Flush 0 Sent Done Done false
27514 8099 Acked Flush 0 Sent Done Done false
27515 8100 Acked Read 1 Sent Done Done false
27516 8101 Acked Write 7 Sent Done Done false
27517 8102 Acked Read 10 Sent Done Done false
27518 8103 Acked Read 2 Sent Done Done false
27519 8104 Acked Write 5 Sent Done Done false
27520 8105 Acked Read 6 Sent Done Done false
27521 8106 Acked Write 2 Sent Done Done false
27522 8107 Acked Read 6 Sent Done Done false
27523 8108 Acked Read 6 Sent Done Done false
27524 8109 Acked Read 3 Sent Done Done false
27525 8110 Acked Write 7 Sent Done Done false
27526 8111 Acked Read 8 Sent Done Done false
27527 8112 Acked Read 7 Sent Done Done false
27528 8113 Acked Flush 0 Sent Done Done false
27529 8114 Acked Read 3 Sent Done Done false
27530 8115 Acked Read 10 Sent Done Done false
27531 8116 Acked Read 2 Sent Done Done false
27532 8117 Acked Read 3 Sent Done Done false
27533 8118 Acked Write 7 Sent Done Done false
27534 8119 Acked Read 6 Sent Done Done false
27535 8120 Acked Write 6 Sent Done Done false
27536 8121 Acked Write 7 Sent Done Done false
27537 8122 Acked Read 2 Sent Done Done false
27538 8123 Acked Write 6 Sent Done Done false
27539 8124 Acked Write 5 Sent Done Done false
27540 8125 Acked Write 5 Sent Done Done false
27541 8126 Acked Write 6 Sent Done Done false
27542 8127 Acked Write 10 Sent Done Done false
27543 8128 Acked Write 3 Sent Done Done false
27544 8129 Acked Write 8 Sent Done Done false
27545 8130 Acked Write 5 Sent Done Done false
27546 8131 Acked Flush 0 Sent Done Done false
27547 8132 Acked Read 4 Sent Done Done false
27548 8133 Acked Write 3 Sent Done Done false
27549 8134 Acked Flush 0 Sent Done Done false
27550 8135 Acked Write 7 Sent Done Done false
27551 8136 Acked Read 6 Sent Done Done false
27552 8137 Acked Write 5 Sent Done Done false
27553 8138 Acked Flush 0 Sent Done Done false
27554 8139 Acked Read 6 Sent Done Done false
27555 8140 Acked Write 8 Sent Done Done false
27556 8141 Acked Read 7 Sent Done Done false
27557 8142 Acked Write 8 Sent Done Done false
27558 8143 Acked Read 10 Sent Done Done false
27559 8144 Acked Read 3 Sent Done Done false
27560 8145 Acked Read 6 Sent Done Done false
27561 8146 Acked Write 3 Sent Done Done false
27562 8147 Acked Read 7 Sent Done Done false
27563 8148 Acked Read 2 Sent Done Done false
27564 8149 Acked Read 10 Sent Done Done false
27565 8150 Acked Read 5 Sent Done Done false
27566 8151 Acked Read 6 Sent Done Done false
27567 8152 Acked Flush 0 Sent Done Done false
27568 8153 Acked Flush 0 Sent Done Done false
27569 8154 Acked Read 6 Sent Done Done false
27570 8155 Acked Read 7 Sent Done Done false
27571 8156 Acked Read 2 Sent Done Done false
27572 8157 Acked Read 4 Sent Done Done false
27573 8158 Acked Read 10 Sent Done Done false
27574 8159 Acked Read 1 Sent Done Done false
27575 8160 Acked Read 6 Sent Done Done false
27576 8161 Acked Write 9 Sent Done Done false
27577 8162 Acked Read 7 Sent Done Done false
27578 8163 Acked Read 10 Sent Done Done false
27579 8164 Acked Read 8 Sent Done Done false
27580 8165 Acked Write 1 Sent Done Done false
27581 8166 Acked Write 4 Sent Done Done false
27582 8167 Acked Write 8 Sent Done Done false
27583 8168 Acked Write 6 Sent Done Done false
27584 8169 Acked Read 3 Sent Done Done false
27585 8170 Acked Write 6 Sent Done Done false
27586 8171 Acked Write 2 Sent Done Done false
27587 8172 Acked Read 8 Sent Done Done false
27588 8173 Acked Read 7 Sent Done Done false
27589 8174 Acked Read 6 Sent Done Done false
27590 8175 Acked Read 7 Sent Done Done false
27591 8176 Acked Write 8 Sent Done Done false
27592 8177 Acked Read 2 Sent Done Done false
27593 8178 Acked Read 2 Sent Done Done false
27594 8179 Acked Write 6 Sent Done Done false
27595 8180 Acked Write 8 Sent Done Done false
27596 8181 Acked Write 8 Sent Done Done false
27597 8182 Acked Read 3 Sent Done Done false
27598 8183 Acked Flush 0 Sent Done Done false
27599 8184 Acked Write 7 Sent Done Done false
27600 8185 Acked Read 1 Sent Done Done false
27601 8186 Acked Write 9 Sent Done Done false
27602 8187 Acked Read 10 Sent Done Done false
27603 8188 Acked Write 9 Sent Done Done false
27604 8189 Acked Read 7 Sent Done Done false
27605 8190 Acked Write 8 Sent Done Done false
27606 8191 Acked Write 2 Sent Done Done false
27607 8192 Acked Write 10 Sent Done Done false
27608 8193 Acked Read 7 Sent Done Done false
27609 8194 Acked Read 3 Sent Done Done false
27610 8195 Acked Flush 0 Sent Done Done false
27611 8196 Acked Write 7 Sent Done Done false
27612 8197 Acked Write 3 Sent Done Done false
27613 8198 Acked Write 1 Sent Done Done false
27614 8199 Acked Read 10 Sent Done Done false
27615 8200 Acked Write 10 Sent Done Done false
27616 8201 Acked Write 1 Sent Done Done false
27617 8202 Acked Read 3 Sent Done Done false
27618 8203 Acked Read 8 Sent Done Done false
27619 8204 Acked Write 10 Sent Done Done false
27620 8205 Acked Read 5 Sent Done Done false
27621 8206 Acked Read 7 Sent Done Done false
27622 8207 Acked Read 9 Sent Done Done false
27623 8208 Acked Read 1 Sent Done Done false
27624 8209 Acked Read 6 Sent Done Done false
27625 8210 Acked Read 6 Sent Done Done false
27626 8211 Acked Read 6 Sent Done Done false
27627 8212 Acked Flush 0 Sent Done Done false
27628 8213 Acked Read 1 Sent Done Done false
27629 8214 Acked Write 7 Sent Done Done false
27630 8215 Acked Write 6 Sent Done Done false
27631 8216 Acked Read 6 Sent Done Done false
27632 8217 Acked Flush 0 Sent Done Done false
27633 8218 Acked Flush 0 Sent Done Done false
27634 8219 Acked Write 9 Sent Done Done false
27635 8220 Acked Read 10 Sent Done Done false
27636 8221 Acked Read 6 Sent Done Done false
27637 8222 Acked Write 2 Sent Done Done false
27638 8223 Acked Write 2 Sent Done Done false
27639 8224 Acked Read 4 Sent Done Done false
27640 8225 Acked Write 6 Sent Done Done false
27641 8226 Acked Flush 0 Sent Done Done false
27642 8227 Acked Write 4 Sent Done Done false
27643 8228 Acked Flush 0 Sent Done Done false
27644 8229 Acked Write 2 Sent Done Done false
27645 8230 Acked Write 2 Sent Done Done false
27646 8231 Acked Read 2 Sent Done Done false
27647 8232 Acked Read 1 Sent Done Done false
27648 8233 Acked Flush 0 Sent Done Done false
27649 8234 Acked Flush 0 Sent Done Done false
27650 8235 Acked Read 9 Sent Done Done false
27651 8236 Acked Write 5 Sent Done Done false
27652 8237 Acked Flush 0 Sent Done Done false
27653 8238 Acked Read 8 Sent Done Done false
27654 8239 Acked Read 10 Sent Done Done false
27655 8240 Acked Write 6 Sent Done Done false
27656 8241 Acked Write 3 Sent Done Done false
27657 8242 Acked Read 6 Sent Done Done false
27658 8243 Acked Read 3 Sent Done Done false
27659 8244 Acked Read 6 Sent Done Done false
27660 8245 Acked Flush 0 Sent Done Done false
27661 8246 Acked Read 2 Sent Done Done false
27662 8247 Acked Read 3 Sent Done Done false
27663 8248 Acked Read 10 Sent Done Done false
27664 8249 Acked Read 1 Sent Done Done false
27665 8250 Acked Write 8 Sent Done Done false
27666 8251 Acked Read 9 Sent Done Done false
27667 8252 Acked Flush 0 Sent Done Done false
27668 STATES DS:0 DS:1 DS:2 TOTAL
27669 Sent 302 0 0 302
27670 Done 0 302 302 604
27671 Skipped 0 0 0 0
27672 Error 0 0 0 0
27673 Last Flush: 7950 8252 8252
27674 Downstairs last five completed: 7950 7949 7948 7947 7946
27675 Upstairs last five completed: 8252 8251 8250 8249 8248
27676 CLIENT: Up:0 ds:302 act:2
27677 ----------------------------------------------------------------
27678 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
27679 States: Offline Active Active
27680 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27681 7951 Acked Read 4 Sent Done Done false
27682 7952 Acked Write 5 Sent Done Done false
27683 7953 Acked Read 2 Sent Done Done false
27684 7954 Acked Write 2 Sent Done Done false
27685 7955 Acked Read 1 Sent Done Done false
27686 7956 Acked Read 8 Sent Done Done false
27687 7957 Acked Read 9 Sent Done Done false
27688 7958 Acked Read 8 Sent Done Done false
27689 7959 Acked Read 2 Sent Done Done false
27690 7960 Acked Flush 0 Sent Done Done false
27691 7961 Acked Read 5 Sent Done Done false
27692 7962 Acked Write 8 Sent Done Done false
27693 7963 Acked Flush 0 Sent Done Done false
27694 7964 Acked Read 1 Sent Done Done false
27695 7965 Acked Write 3 Sent Done Done false
27696 7966 Acked Read 7 Sent Done Done false
27697 7967 Acked Read 2 Sent Done Done false
27698 7968 Acked Read 4 Sent Done Done false
27699 7969 Acked Write 6 Sent Done Done false
27700 7970 Acked Write 8 Sent Done Done false
27701 7971 Acked Write 6 Sent Done Done false
27702 7972 Acked Write 4 Sent Done Done false
27703 7973 Acked Read 6 Sent Done Done false
27704 7974 Acked Read 1 Sent Done Done false
27705 7975 Acked Read 8 Sent Done Done false
27706 7976 Acked Read 2 Sent Done Done false
27707 7977 Acked Read 5 Sent Done Done false
27708 7978 Acked Read 10 Sent Done Done false
27709 7979 Acked Read 10 Sent Done Done false
27710 7980 Acked Read 10 Sent Done Done false
27711 7981 Acked Write 5 Sent Done Done false
27712 7982 Acked Read 6 Sent Done Done false
27713 7983 Acked Write 2 Sent Done Done false
27714 7984 Acked Write 6 Sent Done Done false
27715 7985 Acked Write 7 Sent Done Done false
27716 7986 Acked Write 2 Sent Done Done false
27717 7987 Acked Read 9 Sent Done Done false
27718 7988 Acked Read 1 Sent Done Done false
27719 7989 Acked Read 2 Sent Done Done false
27720 7990 Acked Read 5 Sent Done Done false
27721 7991 Acked Read 7 Sent Done Done false
27722 7992 Acked Read 7 Sent Done Done false
27723 7993 Acked Read 7 Sent Done Done false
27724 7994 Acked Flush 0 Sent Done Done false
27725 7995 Acked Read 3 Sent Done Done false
27726 7996 Acked Read 6 Sent Done Done false
27727 7997 Acked Read 6 Sent Done Done false
27728 7998 Acked Write 8 Sent Done Done false
27729 7999 Acked Read 4 Sent Done Done false
27730 8000 Acked Write 6 Sent Done Done false
27731 8001 Acked Write 10 Sent Done Done false
27732 8002 Acked Read 3 Sent Done Done false
27733 8003 Acked Write 2 Sent Done Done false
27734 8004 Acked Read 6 Sent Done Done false
27735 8005 Acked Read 9 Sent Done Done false
27736 8006 Acked Write 3 Sent Done Done false
27737 8007 Acked Read 3 Sent Done Done false
27738 8008 Acked Write 9 Sent Done Done false
27739 8009 Acked Write 2 Sent Done Done false
27740 8010 Acked Read 8 Sent Done Done false
27741 8011 Acked Flush 0 Sent Done Done false
27742 8012 Acked Write 10 Sent Done Done false
27743 8013 Acked Read 7 Sent Done Done false
27744 8014 Acked Read 10 Sent Done Done false
27745 8015 Acked Read 10 Sent Done Done false
27746 8016 Acked Write 1 Sent Done Done false
27747 8017 Acked Read 3 Sent Done Done false
27748 8018 Acked Read 2 Sent Done Done false
27749 8019 Acked Read 9 Sent Done Done false
27750 8020 Acked Write 7 Sent Done Done false
27751 8021 Acked Read 5 Sent Done Done false
27752 8022 Acked Write 4 Sent Done Done false
27753 8023 Acked Read 4 Sent Done Done false
27754 8024 Acked Read 10 Sent Done Done false
27755 8025 Acked Write 4 Sent Done Done false
27756 8026 Acked Read 7 Sent Done Done false
27757 8027 Acked Read 5 Sent Done Done false
27758 8028 Acked Write 9 Sent Done Done false
27759 8029 Acked Write 7 Sent Done Done false
27760 8030 Acked Read 3 Sent Done Done false
27761 8031 Acked Read 3 Sent Done Done false
27762 8032 Acked Read 6 Sent Done Done false
27763 8033 Acked Write 9 Sent Done Done false
27764 8034 Acked Read 1 Sent Done Done false
27765 8035 Acked Read 8 Sent Done Done false
27766 8036 Acked Flush 0 Sent Done Done false
27767 8037 Acked Write 8 Sent Done Done false
27768 8038 Acked Read 1 Sent Done Done false
27769 8039 Acked Write 9 Sent Done Done false
27770 8040 Acked Read 1 Sent Done Done false
27771 8041 Acked Read 5 Sent Done Done false
27772 8042 Acked Read 8 Sent Done Done false
27773 8043 Acked Write 1 Sent Done Done false
27774 8044 Acked Read 1 Sent Done Done false
27775 8045 Acked Read 2 Sent Done Done false
27776 8046 Acked Write 2 Sent Done Done false
27777 8047 Acked Read 10 Sent Done Done false
27778 8048 Acked Read 9 Sent Done Done false
27779 8049 Acked Read 6 Sent Done Done false
27780 8050 Acked Read 4 Sent Done Done false
27781 8051 Acked Read 6 Sent Done Done false
27782 8052 Acked Write 5 Sent Done Done false
27783 8053 Acked Read 1 Sent Done Done false
27784 8054 Acked Read 9 Sent Done Done false
27785 8055 Acked Read 8 Sent Done Done false
27786 8056 Acked Read 4 Sent Done Done false
27787 8057 Acked Write 3 Sent Done Done false
27788 8058 Acked Read 6 Sent Done Done false
27789 8059 Acked Flush 0 Sent Done Done false
27790 8060 Acked Write 6 Sent Done Done false
27791 8061 Acked Read 7 Sent Done Done false
27792 8062 Acked Read 7 Sent Done Done false
27793 8063 Acked Write 8 Sent Done Done false
27794 8064 Acked Read 1 Sent Done Done false
27795 8065 Acked Read 9 Sent Done Done false
27796 8066 Acked Read 1 Sent Done Done false
27797 8067 Acked Read 1 Sent Done Done false
27798 8068 Acked Write 10 Sent Done Done false
27799 8069 Acked Read 6 Sent Done Done false
27800 8070 Acked Write 10 Sent Done Done false
27801 8071 Acked Write 4 Sent Done Done false
27802 8072 Acked Read 3 Sent Done Done false
27803 8073 Acked Write 4 Sent Done Done false
27804 8074 Acked Read 9 Sent Done Done false
27805 8075 Acked Read 6 Sent Done Done false
27806 8076 Acked Read 8 Sent Done Done false
27807 8077 Acked Write 2 Sent Done Done false
27808 8078 Acked Read 4 Sent Done Done false
27809 8079 Acked Read 10 Sent Done Done false
27810 8080 Acked Read 9 Sent Done Done false
27811 8081 Acked Read 4 Sent Done Done false
27812 8082 Acked Write 5 Sent Done Done false
27813 8083 Acked Write 10 Sent Done Done false
27814 8084 Acked Write 3 Sent Done Done false
27815 8085 Acked Read 9 Sent Done Done false
27816 8086 Acked Read 6 Sent Done Done false
27817 8087 Acked Flush 0 Sent Done Done false
27818 8088 Acked Write 1 Sent Done Done false
27819 8089 Acked Read 2 Sent Done Done false
27820 8090 Acked Write 2 Sent Done Done false
27821 8091 Acked Write 8 Sent Done Done false
27822 8092 Acked Read 7 Sent Done Done false
27823 8093 Acked Read 2 Sent Done Done false
27824 8094 Acked Write 9 Sent Done Done false
27825 8095 Acked Read 10 Sent Done Done false
27826 8096 Acked Read 2 Sent Done Done false
27827 8097 Acked Flush 0 Sent Done Done false
27828 8098 Acked Flush 0 Sent Done Done false
27829 8099 Acked Flush 0 Sent Done Done false
27830 8100 Acked Read 1 Sent Done Done false
27831 8101 Acked Write 7 Sent Done Done false
27832 8102 Acked Read 10 Sent Done Done false
27833 8103 Acked Read 2 Sent Done Done false
27834 8104 Acked Write 5 Sent Done Done false
27835 8105 Acked Read 6 Sent Done Done false
27836 8106 Acked Write 2 Sent Done Done false
27837 8107 Acked Read 6 Sent Done Done false
27838 8108 Acked Read 6 Sent Done Done false
27839 8109 Acked Read 3 Sent Done Done false
27840 8110 Acked Write 7 Sent Done Done false
27841 8111 Acked Read 8 Sent Done Done false
27842 8112 Acked Read 7 Sent Done Done false
27843 8113 Acked Flush 0 Sent Done Done false
27844 8114 Acked Read 3 Sent Done Done false
27845 8115 Acked Read 10 Sent Done Done false
27846 8116 Acked Read 2 Sent Done Done false
27847 8117 Acked Read 3 Sent Done Done false
27848 8118 Acked Write 7 Sent Done Done false
27849 8119 Acked Read 6 Sent Done Done false
27850 8120 Acked Write 6 Sent Done Done false
27851 8121 Acked Write 7 Sent Done Done false
27852 8122 Acked Read 2 Sent Done Done false
27853 8123 Acked Write 6 Sent Done Done false
27854 8124 Acked Write 5 Sent Done Done false
27855 8125 Acked Write 5 Sent Done Done false
27856 8126 Acked Write 6 Sent Done Done false
27857 8127 Acked Write 10 Sent Done Done false
27858 8128 Acked Write 3 Sent Done Done false
27859 8129 Acked Write 8 Sent Done Done false
27860 8130 Acked Write 5 Sent Done Done false
27861 8131 Acked Flush 0 Sent Done Done false
27862 8132 Acked Read 4 Sent Done Done false
27863 8133 Acked Write 3 Sent Done Done false
27864 8134 Acked Flush 0 Sent Done Done false
27865 8135 Acked Write 7 Sent Done Done false
27866 8136 Acked Read 6 Sent Done Done false
27867 8137 Acked Write 5 Sent Done Done false
27868 8138 Acked Flush 0 Sent Done Done false
27869 8139 Acked Read 6 Sent Done Done false
27870 8140 Acked Write 8 Sent Done Done false
27871 8141 Acked Read 7 Sent Done Done false
27872 8142 Acked Write 8 Sent Done Done false
27873 8143 Acked Read 10 Sent Done Done false
27874 8144 Acked Read 3 Sent Done Done false
27875 8145 Acked Read 6 Sent Done Done false
27876 8146 Acked Write 3 Sent Done Done false
27877 8147 Acked Read 7 Sent Done Done false
27878 8148 Acked Read 2 Sent Done Done false
27879 8149 Acked Read 10 Sent Done Done false
27880 8150 Acked Read 5 Sent Done Done false
27881 8151 Acked Read 6 Sent Done Done false
27882 8152 Acked Flush 0 Sent Done Done false
27883 8153 Acked Flush 0 Sent Done Done false
27884 8154 Acked Read 6 Sent Done Done false
27885 8155 Acked Read 7 Sent Done Done false
27886 8156 Acked Read 2 Sent Done Done false
27887 8157 Acked Read 4 Sent Done Done false
27888 8158 Acked Read 10 Sent Done Done false
27889 8159 Acked Read 1 Sent Done Done false
27890 8160 Acked Read 6 Sent Done Done false
27891 8161 Acked Write 9 Sent Done Done false
27892 8162 Acked Read 7 Sent Done Done false
27893 8163 Acked Read 10 Sent Done Done false
27894 8164 Acked Read 8 Sent Done Done false
27895 8165 Acked Write 1 Sent Done Done false
27896 8166 Acked Write 4 Sent Done Done false
27897 8167 Acked Write 8 Sent Done Done false
27898 8168 Acked Write 6 Sent Done Done false
27899 8169 Acked Read 3 Sent Done Done false
27900 8170 Acked Write 6 Sent Done Done false
27901 8171 Acked Write 2 Sent Done Done false
27902 8172 Acked Read 8 Sent Done Done false
27903 8173 Acked Read 7 Sent Done Done false
27904 8174 Acked Read 6 Sent Done Done false
27905 8175 Acked Read 7 Sent Done Done false
27906 8176 Acked Write 8 Sent Done Done false
27907 8177 Acked Read 2 Sent Done Done false
27908 8178 Acked Read 2 Sent Done Done false
27909 8179 Acked Write 6 Sent Done Done false
27910 8180 Acked Write 8 Sent Done Done false
27911 8181 Acked Write 8 Sent Done Done false
27912 8182 Acked Read 3 Sent Done Done false
27913 8183 Acked Flush 0 Sent Done Done false
27914 8184 Acked Write 7 Sent Done Done false
27915 8185 Acked Read 1 Sent Done Done false
27916 8186 Acked Write 9 Sent Done Done false
27917 8187 Acked Read 10 Sent Done Done false
27918 8188 Acked Write 9 Sent Done Done false
27919 8189 Acked Read 7 Sent Done Done false
27920 8190 Acked Write 8 Sent Done Done false
27921 8191 Acked Write 2 Sent Done Done false
27922 8192 Acked Write 10 Sent Done Done false
27923 8193 Acked Read 7 Sent Done Done false
27924 8194 Acked Read 3 Sent Done Done false
27925 8195 Acked Flush 0 Sent Done Done false
27926 8196 Acked Write 7 Sent Done Done false
27927 8197 Acked Write 3 Sent Done Done false
27928 8198 Acked Write 1 Sent Done Done false
27929 8199 Acked Read 10 Sent Done Done false
27930 8200 Acked Write 10 Sent Done Done false
27931 8201 Acked Write 1 Sent Done Done false
27932 8202 Acked Read 3 Sent Done Done false
27933 8203 Acked Read 8 Sent Done Done false
27934 8204 Acked Write 10 Sent Done Done false
27935 8205 Acked Read 5 Sent Done Done false
27936 8206 Acked Read 7 Sent Done Done false
27937 8207 Acked Read 9 Sent Done Done false
27938 8208 Acked Read 1 Sent Done Done false
27939 8209 Acked Read 6 Sent Done Done false
27940 8210 Acked Read 6 Sent Done Done false
27941 8211 Acked Read 6 Sent Done Done false
27942 8212 Acked Flush 0 Sent Done Done false
27943 8213 Acked Read 1 Sent Done Done false
27944 8214 Acked Write 7 Sent Done Done false
27945 8215 Acked Write 6 Sent Done Done false
27946 8216 Acked Read 6 Sent Done Done false
27947 8217 Acked Flush 0 Sent Done Done false
27948 8218 Acked Flush 0 Sent Done Done false
27949 8219 Acked Write 9 Sent Done Done false
27950 8220 Acked Read 10 Sent Done Done false
27951 8221 Acked Read 6 Sent Done Done false
27952 8222 Acked Write 2 Sent Done Done false
27953 8223 Acked Write 2 Sent Done Done false
27954 8224 Acked Read 4 Sent Done Done false
27955 8225 Acked Write 6 Sent Done Done false
27956 8226 Acked Flush 0 Sent Done Done false
27957 8227 Acked Write 4 Sent Done Done false
27958 8228 Acked Flush 0 Sent Done Done false
27959 8229 Acked Write 2 Sent Done Done false
27960 8230 Acked Write 2 Sent Done Done false
27961 8231 Acked Read 2 Sent Done Done false
27962 8232 Acked Read 1 Sent Done Done false
27963 8233 Acked Flush 0 Sent Done Done false
27964 8234 Acked Flush 0 Sent Done Done false
27965 8235 Acked Read 9 Sent Done Done false
27966 8236 Acked Write 5 Sent Done Done false
27967 8237 Acked Flush 0 Sent Done Done false
27968 8238 Acked Read 8 Sent Done Done false
27969 8239 Acked Read 10 Sent Done Done false
27970 8240 Acked Write 6 Sent Done Done false
27971 8241 Acked Write 3 Sent Done Done false
27972 8242 Acked Read 6 Sent Done Done false
27973 8243 Acked Read 3 Sent Done Done false
27974 8244 Acked Read 6 Sent Done Done false
27975 8245 Acked Flush 0 Sent Done Done false
27976 8246 Acked Read 2 Sent Done Done false
27977 8247 Acked Read 3 Sent Done Done false
27978 8248 Acked Read 10 Sent Done Done false
27979 8249 Acked Read 1 Sent Done Done false
27980 8250 Acked Write 8 Sent Done Done false
27981 8251 Acked Read 9 Sent Done Done false
27982 8252 Acked Flush 0 Sent Done Done false
27983 STATES DS:0 DS:1 DS:2 TOTAL
27984 Sent 302 0 0 302
27985 Done 0 302 302 604
27986 Skipped 0 0 0 0
27987 Error 0 0 0 0
27988 Last Flush: 7950 8252 8252
27989 Downstairs last five completed: 7950 7949 7948 7947 7946
27990 Upstairs last five completed: 8252 8251 8250 8249 8248
27991 CLIENT: Up:0 ds:302 act:2
279922024-11-01T16:56:23.151ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
279932024-11-01T16:56:23.151ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39512) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
279942024-11-01T16:56:23.152ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
279952024-11-01T16:56:23.153ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
279962024-11-01T16:56:23.153ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
279972024-11-01T16:56:23.153ZINFOcrucible: send last flush ID to this DS: 7950 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
279982024-11-01T16:56:23.153ZINFOcrucible: Replied this last flush ID: 7950 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
279992024-11-01T16:56:23.153ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
280002024-11-01T16:56:23.153ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
280012024-11-01T16:56:23.153ZINFOcrucible: [0] client re-new 302 jobs since flush 7950 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
280022024-11-01T16:56:23.153ZINFOcrucible: [0] Replayed 302 jobs since flush: 7950 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
28003 ----------------------------------------------------------------
28004 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28005 Downstairs last five completed: 8252 8251 8250 8249 8248
28006 Upstairs last five completed: 8252 8251 8250 8249 8248
28007 CLIENT: Up:0 ds:0 act:3
28008 Replay: All jobs finished
28009 Main task has work to do, go find it
28010 got dsc Stop(0)
28011 stop 0
28012 Nov 01 16:56:23.456 INFO request completed, latency_us: 364, response_code: 204, uri: /stop/cid/0, method: POST, [8810] Got stop action so:false kr:false
28013 req_id: a03b8852-be06-424d-8a77-80d9d0564c7c, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
28014 Nov 01 16:56:23.456 INFO request completed, latency_us: 128, response_code: 200, uri: /state/cid/0, method: GET, req_id: 3d13c514-cc96-4e26-b751-c426a0b25602, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
280152024-11-01T16:56:23.456ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
280162024-11-01T16:56:23.457ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
280172024-11-01T16:56:23.457ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
280182024-11-01T16:56:23.457ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
280192024-11-01T16:56:23.457ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
28020 [8810] Exited with: ExitStatus(unix_wait_status(9))
28021 [8810][0] reports Exit
28022 Nov 01 16:56:27.459 INFO request completed, latency_us: 235, response_code: 200, uri: /state/cid/0, method: GET, req_id: d6f7ad1c-5665-45d5-9839-5e7a8fd1e3b9, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
28023 001/300 Read block 356 len 16384
28024 002/300 Write block 1071 len 16384 data: 7 6 7 7
28025 003/300 Write block 2478 len 16384 data: 10 10 6 6
28026 004/300 Write block 448 len 24576 data: 6 6 5 6 6 5
28027 005/300 Write block 826 len 4096 data: 6
28028 006/300 Read block 2827 len 36864
28029 007/300 Read block 248 len 32768
28030 008/300 Read block 3470 len 28672
28031 009/300 Read block 2367 len 12288
28032 010/300 Read block 4173 len 12288
28033 011/300 Read block 2386 len 40960
28034 012/300 Write block 1297 len 4096 data: 7
28035 013/300 Write block 3211 len 24576 data: 3 3 3 5 5 4
28036 014/300 Read block 4442 len 36864
28037 015/300 Write block 3465 len 32768 data: 7 8 7 7 7 8 10 8
28038 016/300 Flush
28039 017/300 Flush
28040 018/300 Read block 3301 len 16384
28041 019/300 Read block 1375 len 36864
28042 020/300 Write block 2060 len 12288 data: 6 6 8
28043 021/300 Write block 2337 len 40960 data: 3 2 4 5 5 6 7 7 7 5
28044 022/300 Flush
28045 023/300 Read block 2408 len 32768
28046 024/300 Write block 4599 len 36864 data: 5 6 5 4 4 4 4 4 4
28047 025/300 Read block 2489 len 12288
28048 026/300 Flush
28049 027/300 Read block 3498 len 20480
28050 028/300 Write block 3784 len 28672 data: 2 2 2 4 4 4 5
28051 029/300 Write block 4916 len 20480 data: 8 8 7 6 5
28052 030/300 Read block 448 len 4096
28053 031/300 Write block 826 len 40960 data: 7 5 5 6 7 8 7 6 7 7
28054 032/300 Read block 4684 len 36864
28055 033/300 Read block 2173 len 4096
28056 034/300 Read block 2895 len 24576
28057 035/300 Read block 388 len 16384
28058 036/300 Write block 1870 len 16384 data: 8 7 10 10
28059 037/300 Read block 1577 len 36864
28060 038/300 Read block 1544 len 12288
28061 039/300 Write block 1510 len 28672 data: 9 8 9 8 7 7 6
28062 040/300 Read block 807 len 16384
28063 041/300 Write block 592 len 8192 data: 5 5
28064 042/300 Read block 1746 len 32768
28065 043/300 Flush
28066 044/300 Write block 1467 len 40960 data: 3 2 4 5 5 4 4 6 6 8
28067 045/300 Write block 273 len 24576 data: 5 5 5 5 8 8
28068 046/300 Flush
28069 047/300 Read block 4205 len 24576
28070 048/300 Read block 2615 len 32768
28071 049/300 Write block 3535 len 36864 data: 5 5 5 5 4 3 4 5 5
28072 050/300 Read block 798 len 16384
28073 051/300 Flush
28074 052/300 Flush
28075 053/300 Write block 2057 len 4096 data: 6
28076 054/300 Read block 2765 len 16384
28077 055/300 Read block 145 len 8192
28078 056/300 Read block 4926 len 16384
28079 057/300 Read block 3369 len 36864
28080 058/300 Read block 4638 len 16384
28081 059/300 Read block 4432 len 20480
28082 060/300 Read block 729 len 12288
28083 061/300 Write block 4790 len 28672 data: 5 4 4 4 3 3 3
28084 062/300 Read block 1332 len 36864
28085 063/300 Read block 32 len 40960
28086 064/300 Read block 1423 len 4096
28087 065/300 Read block 2931 len 16384
28088 066/300 Read block 4879 len 24576
28089 067/300 Read block 3873 len 24576
28090 068/300 Read block 2598 len 4096
28091 069/300 Write block 908 len 36864 data: 4 5 7 6 6 6 6 7 7
28092 070/300 Write block 149 len 24576 data: 4 4 3 3 2 2
28093 071/300 Write block 2045 len 36864 data: 2 3 3 5 5 5 6 6 5
28094 072/300 Read block 4624 len 36864
28095 073/300 Read block 2612 len 36864
28096 074/300 Read block 1658 len 8192
28097 075/300 Read block 2518 len 24576
28098 076/300 Read block 3393 len 20480
28099 077/300 Read block 1513 len 40960
28100 078/300 Read block 1223 len 24576
28101 079/300 Write block 3563 len 40960 data: 4 4 4 4 5 6 5 3 4 4
28102 080/300 Read block 3334 len 16384
28103 081/300 Flush
28104 082/300 Read block 3993 len 36864
28105 083/300 Read block 1875 len 28672
28106 084/300 Read block 706 len 36864
28107 085/300 Write block 529 len 16384 data: 11 10 10 9
28108 086/300 Flush
28109 087/300 Write block 2330 len 40960 data: 5 5 4 4 3 3 3 4 3 5
28110 088/300 Read block 3053 len 8192
28111 089/300 Read block 893 len 4096
28112 090/300 Read block 3859 len 32768
28113 091/300 Write block 397 len 40960 data: 3 3 4 3 3 3 3 3 3 5
28114 092/300 Read block 4595 len 12288
28115 093/300 Read block 4711 len 12288
28116 094/300 Write block 4199 len 8192 data: 6 6
28117 095/300 Write block 4218 len 4096 data: 5
28118 096/300 Flush
28119 097/300 Write block 464 len 28672 data: 4 4 4 3 3 2 3
28120 098/300 Read block 1095 len 20480
28121 099/300 Write block 2245 len 40960 data: 6 5 4 4 3 3 2 3 3 3
28122 100/300 Flush
28123 101/300 Flush
28124 102/300 Read block 3587 len 20480
28125 103/300 Read block 1241 len 40960
28126 104/300 Read block 4778 len 4096
28127 105/300 Read block 2046 len 32768
28128 106/300 Read block 3935 len 20480
28129 107/300 Read block 799 len 36864
28130 108/300 Read block 4013 len 36864
28131 109/300 Read block 3618 len 28672
28132 110/300 Read block 3075 len 16384
28133 111/300 Write block 1260 len 40960 data: 5 4 4 3 5 7 7 6 6 6
28134 112/300 Flush
28135 113/300 Write block 1041 len 12288 data: 6 6 5
28136 114/300 Read block 3024 len 20480
28137 115/300 Write block 199 len 32768 data: 10 7 5 6 6 7 6 6
28138 116/300 Read block 4110 len 32768
28139 117/300 Write block 2592 len 4096 data: 3
28140 118/300 Write block 1712 len 4096 data: 4
28141 119/300 Read block 1691 len 36864
28142 120/300 Write block 4322 len 24576 data: 8 6 5 4 4 3
28143 121/300 Write block 1736 len 40960 data: 4 5 4 5 7 5 5 6 6 6
28144 122/300 Write block 3919 len 36864 data: 6 6 4 3 3 3 4 3 3
28145 123/300 Flush
28146 124/300 Write block 437 len 16384 data: 6 3 3 3
28147 125/300 Write block 577 len 32768 data: 2 3 3 3 4 4 4 5
28148 126/300 Write block 4550 len 40960 data: 5 4 4 6 7 6 7 5 6 5
28149 127/300 Write block 3828 len 40960 data: 7 6 7 8 7 8 6 5 6 5
28150 128/300 Read block 1050 len 40960
28151 129/300 Flush
28152 130/300 Flush
28153 131/300 Read block 2670 len 24576
28154 132/300 Read block 3626 len 12288
28155 133/300 Write block 950 len 12288 data: 4 3 2
28156 134/300 Read block 2385 len 40960
28157 135/300 Read block 2978 len 16384
28158 136/300 Write block 1123 len 32768 data: 5 4 5 5 5 3 3 3
28159 137/300 Write block 3967 len 36864 data: 7 8 7 5 5 6 6 6 6
28160 138/300 Read block 224 len 32768
28161 139/300 Read block 3653 len 4096
28162 140/300 Flush
28163 141/300 Read block 1830 len 12288
28164 142/300 Write block 2433 len 40960 data: 4 5 5 5 6 6 3 4 4 3
28165 143/300 Write block 1119 len 32768 data: 7 7 6 4 6 5 6 6
28166 144/300 Read block 507 len 12288
28167 145/300 Flush
28168 146/300 Write block 1078 len 28672 data: 7 6 7 6 5 4 4
28169 147/300 Write block 2584 len 8192 data: 6 7
28170 148/300 Read block 1115 len 8192
28171 149/300 Read block 4334 len 16384
28172 150/300 Read block 775 len 32768
28173 151/300 Write block 738 len 20480 data: 6 7 6 5 3
28174 152/300 Read block 1110 len 12288
28175 153/300 Write block 4815 len 20480 data: 6 5 4 4 4
28176 154/300 Write block 2034 len 40960 data: 8 7 5 4 4 4 5 4 2 2
28177 155/300 Write block 401 len 12288 data: 4 4 4
28178 156/300 Read block 1246 len 28672
28179 157/300 Read block 3697 len 16384
28180 158/300 Write block 1648 len 16384 data: 4 3 4 5
28181 159/300 Read block 3595 len 40960
28182 160/300 Read block 3884 len 4096
28183 161/300 Write block 1091 len 28672 data: 10 9 8 7 7 6 4
28184 162/300 Write block 732 len 20480 data: 9 8 9 9 7
28185 163/300 Read block 4099 len 24576
28186 164/300 Write block 4366 len 8192 data: 3 3
28187 165/300 Read block 2277 len 8192
28188 166/300 Read block 1402 len 4096
28189 167/300 Read block 3897 len 32768
28190 168/300 Write block 4183 len 4096 data: 6
28191 169/300 Read block 3724 len 4096
28192 170/300 Read block 991 len 16384
28193 171/300 Read block 420 len 32768
28194 172/300 Write block 4752 len 24576 data: 4 4 4 4 4 3
28195 173/300 Read block 1545 len 16384
28196 174/300 Flush
28197 175/300 Write block 4660 len 16384 data: 5 6 6 6
28198 176/300 Read block 2573 len 8192
28199 177/300 Write block 3848 len 32768 data: 4 5 5 4 3 4 5 5
28200 178/300 Write block 3827 len 16384 data: 7 8 7 8
28201 179/300 Write block 3347 len 36864 data: 4 5 6 6 5 5 5 5 5
28202 180/300 Read block 793 len 32768
28203 181/300 Write block 1613 len 4096 data: 4
28204 182/300 Flush
28205 183/300 Read block 3100 len 40960
28206 184/300 Read block 3162 len 32768
28207 185/300 Flush
28208 186/300 Write block 105 len 12288 data: 5 5 5
28209 187/300 Read block 1549 len 16384
28210 188/300 Write block 1226 len 20480 data: 5 5 4 3 3
28211 189/300 Write block 1122 len 36864 data: 5 7 6 7 7 6 4 4 4
28212 190/300 Read block 3471 len 12288
28213 191/300 Write block 513 len 8192 data: 6 7
28214 192/300 Read block 1949 len 40960
28215 193/300 Read block 2488 len 20480
28216 194/300 Write block 895 len 16384 data: 8 7 7 4
28217 195/300 Read block 563 len 24576
28218 196/300 Read block 3020 len 40960
28219 197/300 Read block 2245 len 36864
28220 198/300 Write block 1802 len 40960 data: 6 7 5 5 5 7 7 6 7 7
28221 199/300 Read block 961 len 16384
28222 200/300 Read block 2485 len 28672
28223 201/300 Read block 383 len 16384
28224 202/300 Write block 439 len 16384 data: 4 4 2 2
28225 203/300 Flush
28226 204/300 Read block 3774 len 20480
28227 205/300 Write block 1994 len 12288 data: 3 3 3
28228 206/300 Read block 3546 len 12288
28229 207/300 Write block 707 len 36864 data: 3 3 2 2 2 2 2 2 2
28230 208/300 Read block 3649 len 20480
28231 209/300 Write block 3038 len 40960 data: 3 4 4 4 5 6 5 5 4 5
28232 210/300 Read block 3990 len 20480
28233 211/300 Read block 4465 len 4096
28234 212/300 Write block 4340 len 16384 data: 4 4 4 3
28235 213/300 Write block 1820 len 24576 data: 5 6 6 6 6 7
28236 214/300 Write block 4841 len 4096 data: 2
28237 215/300 Write block 3884 len 24576 data: 4 4 4 4 3 3
28238 216/300 Write block 2509 len 32768 data: 4 5 5 5 5 6 6 4
28239 217/300 Read block 856 len 32768
28240 218/300 Read block 4955 len 4096
28241 219/300 Write block 923 len 4096 data: 5
28242 220/300 Write block 1690 len 4096 data: 4
28243 221/300 Flush
28244 222/300 Read block 3328 len 8192
28245 223/300 Write block 1239 len 28672 data: 5 6 4 4 4 3 2
28246 224/300 Write block 1135 len 24576 data: 4 6 7 9 8 7
28247 225/300 Read block 2625 len 32768
28248 226/300 Write block 4979 len 8192 data: 5 5
28249 227/300 Write block 4934 len 4096 data: 5
28250 228/300 Write block 3330 len 20480 data: 4 4 4 4 4
28251 229/300 Read block 3299 len 12288
28252 230/300 Read block 1890 len 36864
28253 231/300 Write block 3121 len 24576 data: 6 9 7 8 6 6
28254 232/300 Read block 4865 len 28672
28255 233/300 Write block 461 len 16384 data: 5 7 5 5
28256 234/300 Read block 2337 len 16384
28257 235/300 Flush
28258 236/300 Read block 285 len 28672
28259 237/300 Read block 3148 len 28672
28260 238/300 Write block 1792 len 24576 data: 4 3 3 4 4 4
28261 239/300 Write block 2124 len 32768 data: 6 5 3 3 2 2 2 3
28262 240/300 Write block 4389 len 40960 data: 4 5 3 5 5 4 3 3 3 3
28263 241/300 Write block 1465 len 40960 data: 3 3 4 3 5 6 6 5 5 7
28264 242/300 Flush
28265 243/300 Read block 2722 len 40960
28266 244/300 Read block 4493 len 20480
28267 245/300 Read block 4967 len 24576
28268 246/300 Read block 3974 len 28672
28269 247/300 Read block 4144 len 36864
28270 248/300 Read block 383 len 16384
28271 249/300 Write block 2416 len 28672 data: 4 4 4 2 2 3 2
28272 250/300 Write block 1484 len 24576 data: 2 2 2 2 2 3
28273 251/300 Write block 2757 len 36864 data: 6 6 6 6 5 5 4 5 7
28274 252/300 Read block 4346 len 20480
28275 253/300 Write block 4617 len 28672 data: 4 3 3 3 3 3 2
28276 254/300 Read block 4713 len 40960
28277 255/300 Write block 4458 len 32768 data: 3 4 5 6 6 6 4 4
28278 256/300 Write block 2364 len 16384 data: 6 8 8 6
28279 257/300 Write block 1429 len 24576 data: 5 5 7 7 6 6
28280 258/300 Write block 2011 len 4096 data: 2
28281 259/300 Read block 371 len 36864
28282 260/300 Read block 4176 len 36864
28283 261/300 Write block 2723 len 32768 data: 8 8 7 4 5 5 6 5
28284 262/300 Read block 4163 len 36864
28285 263/300 Read block 3505 len 28672
28286 264/300 Write block 1759 len 12288 data: 3 5 7
28287 265/300 Write block 4860 len 28672 data: 4 3 3 2 2 3 6
28288 266/300 Read block 1041 len 28672
28289 267/300 Write block 1218 len 36864 data: 9 7 7 7 4 4 4 7 6
28290 268/300 Read block 3429 len 28672
28291 269/300 Read block 4790 len 16384
28292 270/300 Read block 2415 len 16384
28293 271/300 Flush
28294 272/300 Write block 2512 len 40960 data: 6 6 7 7 5 4 4 3 3 2
28295 273/300 Write block 507 len 32768 data: 6 8 8 8 7 8 7 8
28296 274/300 Write block 2639 len 4096 data: 7
28297 275/300 Read block 873 len 36864
28298 276/300 Write block 2433 len 24576 data: 5 6 6 6 7 7
28299 277/300 Write block 1178 len 40960 data: 8 8 6 5 3 3 3 3 4 4
28300 278/300 Read block 2670 len 28672
28301 279/300 Read block 2954 len 4096
28302 280/300 Read block 1566 len 16384
28303 281/300 Write block 848 len 20480 data: 7 9 10 8 8
28304 282/300 Flush
28305 283/300 Flush
28306 284/300 Flush
28307 285/300 Read block 839 len 16384
28308 286/300 Read block 4723 len 28672
28309 287/300 Write block 4661 len 32768 data: 7 7 7 8 6 6 6 4
28310 288/300 Write block 4204 len 28672 data: 9 8 7 6 5 8 8
28311 289/300 Read block 2938 len 32768
28312 290/300 Write block 403 len 20480 data: 5 4 4 6 5
28313 291/300 Read block 1548 len 12288
28314 292/300 Write block 364 len 16384 data: 6 8 7 6
28315 293/300 Write block 1777 len 4096 data: 5
28316 294/300 Read block 1493 len 20480
28317 295/300 Read block 1728 len 40960
28318 296/300 Read block 2886 len 28672
28319 297/300 Write block 3151 len 4096 data: 6
28320 298/300 Flush
28321 299/300 Write block 1864 len 8192 data: 4 5
28322 300/300 Read block 3816 len 32768
28323 301/300 Read block 389 len 8192
28324 Main task has work to do, go find it
28325 got dsc Start(0)
28326 start 0
28327 Nov 01 16:56:27.760 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/0, method: POST, req_id: 3a33a977-ef29-4ede-b50c-ffa870662a5f, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
28328 Replay: started 0, returned:Ok(())
28329 ----------------------------------------------------------------
28330 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28331 States: Offline Active Active
28332 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28333 8253 Acked Read 4 Sent Done Done false
28334 8254 Acked Write 4 Sent Done Done false
28335 8255 Acked Write 4 Sent Done Done false
28336 8256 Acked Write 6 Sent Done Done false
28337 8257 Acked Write 1 Sent Done Done false
28338 8258 Acked Read 9 Sent Done Done false
28339 8259 Acked Read 8 Sent Done Done false
28340 8260 Acked Read 7 Sent Done Done false
28341 8261 Acked Read 3 Sent Done Done false
28342 8262 Acked Read 3 Sent Done Done false
28343 8263 Acked Read 10 Sent Done Done false
28344 8264 Acked Write 1 Sent Done Done false
28345 8265 Acked Write 6 Sent Done Done false
28346 8266 Acked Read 9 Sent Done Done false
28347 8267 Acked Write 8 Sent Done Done false
28348 8268 Acked Flush 0 Sent Done Done false
28349 8269 Acked Flush 0 Sent Done Done false
28350 8270 Acked Read 4 Sent Done Done false
28351 8271 Acked Read 9 Sent Done Done false
28352 8272 Acked Write 3 Sent Done Done false
28353 8273 Acked Write 10 Sent Done Done false
28354 8274 Acked Flush 0 Sent Done Done false
28355 8275 Acked Read 8 Sent Done Done false
28356 8276 Acked Write 9 Sent Done Done false
28357 8277 Acked Read 3 Sent Done Done false
28358 8278 Acked Flush 0 Sent Done Done false
28359 8279 Acked Read 5 Sent Done Done false
28360 8280 Acked Write 7 Sent Done Done false
28361 8281 Acked Write 5 Sent Done Done false
28362 8282 Acked Read 1 Sent Done Done false
28363 8283 Acked Write 10 Sent Done Done false
28364 8284 Acked Read 9 Sent Done Done false
28365 8285 Acked Read 1 Sent Done Done false
28366 8286 Acked Read 6 Sent Done Done false
28367 8287 Acked Read 4 Sent Done Done false
28368 8288 Acked Write 4 Sent Done Done false
28369 8289 Acked Read 9 Sent Done Done false
28370 8290 Acked Read 3 Sent Done Done false
28371 8291 Acked Write 7 Sent Done Done false
28372 8292 Acked Read 4 Sent Done Done false
28373 8293 Acked Write 2 Sent Done Done false
28374 8294 Acked Read 8 Sent Done Done false
28375 8295 Acked Flush 0 Sent Done Done false
28376 8296 Acked Write 10 Sent Done Done false
28377 8297 Acked Write 6 Sent Done Done false
28378 8298 Acked Flush 0 Sent Done Done false
28379 8299 Acked Read 6 Sent Done Done false
28380 8300 Acked Read 8 Sent Done Done false
28381 8301 Acked Write 9 Sent Done Done false
28382 8302 Acked Read 4 Sent Done Done false
28383 8303 Acked Flush 0 Sent Done Done false
28384 8304 Acked Flush 0 Sent Done Done false
28385 8305 Acked Write 1 Sent Done Done false
28386 8306 Acked Read 4 Sent Done Done false
28387 8307 Acked Read 2 Sent Done Done false
28388 8308 Acked Read 4 Sent Done Done false
28389 8309 Acked Read 9 Sent Done Done false
28390 8310 Acked Read 4 Sent Done Done false
28391 8311 Acked Read 5 Sent Done Done false
28392 8312 Acked Read 3 Sent Done Done false
28393 8313 Acked Write 7 Sent Done Done false
28394 8314 Acked Read 9 Sent Done Done false
28395 8315 Acked Read 10 Sent Done Done false
28396 8316 Acked Read 1 Sent Done Done false
28397 8317 Acked Read 4 Sent Done Done false
28398 8318 Acked Read 6 Sent Done Done false
28399 8319 Acked Read 6 Sent Done Done false
28400 8320 Acked Read 1 Sent Done Done false
28401 8321 Acked Write 9 Sent Done Done false
28402 8322 Acked Write 6 Sent Done Done false
28403 8323 Acked Write 9 Sent Done Done false
28404 8324 Acked Read 9 Sent Done Done false
28405 8325 Acked Read 9 Sent Done Done false
28406 8326 Acked Read 2 Sent Done Done false
28407 8327 Acked Read 6 Sent Done Done false
28408 8328 Acked Read 5 Sent Done Done false
28409 8329 Acked Read 10 Sent Done Done false
28410 8330 Acked Read 6 Sent Done Done false
28411 8331 Acked Write 10 Sent Done Done false
28412 8332 Acked Read 4 Sent Done Done false
28413 8333 Acked Flush 0 Sent Done Done false
28414 8334 Acked Read 9 Sent Done Done false
28415 8335 Acked Read 7 Sent Done Done false
28416 8336 Acked Read 9 Sent Done Done false
28417 8337 Acked Write 4 Sent Done Done false
28418 8338 Acked Flush 0 Sent Done Done false
28419 8339 Acked Write 10 Sent Done Done false
28420 8340 Acked Read 2 Sent Done Done false
28421 8341 Acked Read 1 Sent Done Done false
28422 8342 Acked Read 8 Sent Done Done false
28423 8343 Acked Write 10 Sent Done Done false
28424 8344 Acked Read 3 Sent Done Done false
28425 8345 Acked Read 3 Sent Done Done false
28426 8346 Acked Write 2 Sent Done Done false
28427 8347 Acked Write 1 Sent Done Done false
28428 8348 Acked Flush 0 Sent Done Done false
28429 8349 Acked Write 7 Sent Done Done false
28430 8350 Acked Read 5 Sent Done Done false
28431 8351 Acked Write 10 Sent Done Done false
28432 8352 Acked Flush 0 Sent Done Done false
28433 8353 Acked Flush 0 Sent Done Done false
28434 8354 Acked Read 5 Sent Done Done false
28435 8355 Acked Read 10 Sent Done Done false
28436 8356 Acked Read 1 Sent Done Done false
28437 8357 Acked Read 8 Sent Done Done false
28438 8358 Acked Read 5 Sent Done Done false
28439 8359 Acked Read 9 Sent Done Done false
28440 8360 Acked Read 9 Sent Done Done false
28441 8361 Acked Read 7 Sent Done Done false
28442 8362 Acked Read 4 Sent Done Done false
28443 8363 Acked Write 10 Sent Done Done false
28444 8364 Acked Flush 0 Sent Done Done false
28445 8365 Acked Write 3 Sent Done Done false
28446 8366 Acked Read 5 Sent Done Done false
28447 8367 Acked Write 8 Sent Done Done false
28448 8368 Acked Read 8 Sent Done Done false
28449 8369 Acked Write 1 Sent Done Done false
28450 8370 Acked Write 1 Sent Done Done false
28451 8371 Acked Read 9 Sent Done Done false
28452 8372 Acked Write 6 Sent Done Done false
28453 8373 Acked Write 10 Sent Done Done false
28454 8374 Acked Write 9 Sent Done Done false
28455 8375 Acked Flush 0 Sent Done Done false
28456 8376 Acked Write 4 Sent Done Done false
28457 8377 Acked Write 8 Sent Done Done false
28458 8378 Acked Write 10 Sent Done Done false
28459 8379 Acked Write 10 Sent Done Done false
28460 8380 Acked Read 10 Sent Done Done false
28461 8381 Acked Flush 0 Sent Done Done false
28462 8382 Acked Flush 0 Sent Done Done false
28463 8383 Acked Read 6 Sent Done Done false
28464 8384 Acked Read 3 Sent Done Done false
28465 8385 Acked Write 3 Sent Done Done false
28466 8386 Acked Read 10 Sent Done Done false
28467 8387 Acked Read 4 Sent Done Done false
28468 8388 Acked Write 8 Sent Done Done false
28469 8389 Acked Write 9 Sent Done Done false
28470 8390 Acked Read 8 Sent Done Done false
28471 8391 Acked Read 1 Sent Done Done false
28472 8392 Acked Flush 0 Sent Done Done false
28473 8393 Acked Read 3 Sent Done Done false
28474 8394 Acked Write 10 Sent Done Done false
28475 8395 Acked Write 8 Sent Done Done false
28476 8396 Acked Read 3 Sent Done Done false
28477 8397 Acked Flush 0 Sent Done Done false
28478 8398 Acked Write 7 Sent Done Done false
28479 8399 Acked Write 2 Sent Done Done false
28480 8400 Acked Read 2 Sent Done Done false
28481 8401 Acked Read 4 Sent Done Done false
28482 8402 Acked Read 8 Sent Done Done false
28483 8403 Acked Write 5 Sent Done Done false
28484 8404 Acked Read 3 Sent Done Done false
28485 8405 Acked Write 5 Sent Done Done false
28486 8406 Acked Write 10 Sent Done Done false
28487 8407 Acked Write 3 Sent Done Done false
28488 8408 Acked Read 7 Sent Done Done false
28489 8409 Acked Read 4 Sent Done Done false
28490 8410 Acked Write 4 Sent Done Done false
28491 8411 Acked Read 10 Sent Done Done false
28492 8412 Acked Read 1 Sent Done Done false
28493 8413 Acked Write 7 Sent Done Done false
28494 8414 Acked Write 5 Sent Done Done false
28495 8415 Acked Read 6 Sent Done Done false
28496 8416 Acked Write 2 Sent Done Done false
28497 8417 Acked Read 2 Sent Done Done false
28498 8418 Acked Read 1 Sent Done Done false
28499 8419 Acked Read 8 Sent Done Done false
28500 8420 Acked Write 1 Sent Done Done false
28501 8421 Acked Read 1 Sent Done Done false
28502 8422 Acked Read 4 Sent Done Done false
28503 8423 Acked Read 8 Sent Done Done false
28504 8424 Acked Write 6 Sent Done Done false
28505 8425 Acked Read 4 Sent Done Done false
28506 8426 Acked Flush 0 Sent Done Done false
28507 8427 Acked Write 4 Sent Done Done false
28508 8428 Acked Read 2 Sent Done Done false
28509 8429 Acked Write 8 Sent Done Done false
28510 8430 Acked Write 4 Sent Done Done false
28511 8431 Acked Write 9 Sent Done Done false
28512 8432 Acked Read 8 Sent Done Done false
28513 8433 Acked Write 1 Sent Done Done false
28514 8434 Acked Flush 0 Sent Done Done false
28515 8435 Acked Read 10 Sent Done Done false
28516 8436 Acked Read 8 Sent Done Done false
28517 8437 Acked Flush 0 Sent Done Done false
28518 8438 Acked Write 3 Sent Done Done false
28519 8439 Acked Read 4 Sent Done Done false
28520 8440 Acked Write 5 Sent Done Done false
28521 8441 Acked Write 9 Sent Done Done false
28522 8442 Acked Read 3 Sent Done Done false
28523 8443 Acked Write 2 Sent Done Done false
28524 8444 Acked Read 10 Sent Done Done false
28525 8445 Acked Read 5 Sent Done Done false
28526 8446 Acked Write 4 Sent Done Done false
28527 8447 Acked Read 6 Sent Done Done false
28528 8448 Acked Read 10 Sent Done Done false
28529 8449 Acked Read 9 Sent Done Done false
28530 8450 Acked Write 10 Sent Done Done false
28531 8451 Acked Read 4 Sent Done Done false
28532 8452 Acked Read 7 Sent Done Done false
28533 8453 Acked Read 4 Sent Done Done false
28534 8454 Acked Write 4 Sent Done Done false
28535 8455 Acked Flush 0 Sent Done Done false
28536 8456 Acked Read 5 Sent Done Done false
28537 8457 Acked Write 3 Sent Done Done false
28538 8458 Acked Read 3 Sent Done Done false
28539 8459 Acked Write 9 Sent Done Done false
28540 8460 Acked Read 5 Sent Done Done false
28541 8461 Acked Write 10 Sent Done Done false
28542 8462 Acked Read 5 Sent Done Done false
28543 8463 Acked Read 1 Sent Done Done false
28544 8464 Acked Write 4 Sent Done Done false
28545 8465 Acked Write 6 Sent Done Done false
28546 8466 Acked Write 1 Sent Done Done false
28547 8467 Acked Write 6 Sent Done Done false
28548 8468 Acked Write 8 Sent Done Done false
28549 8469 Acked Read 8 Sent Done Done false
28550 8470 Acked Read 1 Sent Done Done false
28551 8471 Acked Write 1 Sent Done Done false
28552 8472 Acked Write 1 Sent Done Done false
28553 8473 Acked Flush 0 Sent Done Done false
28554 8474 Acked Read 2 Sent Done Done false
28555 8475 Acked Write 7 Sent Done Done false
28556 8476 Acked Write 6 Sent Done Done false
28557 8477 Acked Read 8 Sent Done Done false
28558 8478 Acked Write 2 Sent Done Done false
28559 8479 Acked Write 1 Sent Done Done false
28560 8480 Acked Write 5 Sent Done Done false
28561 8481 Acked Read 3 Sent Done Done false
28562 8482 Acked Read 9 Sent Done Done false
28563 8483 Acked Write 6 Sent Done Done false
28564 8484 Acked Read 7 Sent Done Done false
28565 8485 Acked Write 4 Sent Done Done false
28566 8486 Acked Read 4 Sent Done Done false
28567 8487 Acked Flush 0 Sent Done Done false
28568 8488 Acked Read 7 Sent Done Done false
28569 8489 Acked Read 7 Sent Done Done false
28570 8490 Acked Write 6 Sent Done Done false
28571 8491 Acked Write 8 Sent Done Done false
28572 8492 Acked Write 10 Sent Done Done false
28573 8493 Acked Write 10 Sent Done Done false
28574 8494 Acked Flush 0 Sent Done Done false
28575 8495 Acked Read 10 Sent Done Done false
28576 8496 Acked Read 5 Sent Done Done false
28577 8497 Acked Read 6 Sent Done Done false
28578 8498 Acked Read 7 Sent Done Done false
28579 8499 Acked Read 9 Sent Done Done false
28580 8500 Acked Read 4 Sent Done Done false
28581 8501 Acked Write 7 Sent Done Done false
28582 8502 Acked Write 6 Sent Done Done false
28583 8503 Acked Write 9 Sent Done Done false
28584 8504 Acked Read 5 Sent Done Done false
28585 8505 Acked Write 7 Sent Done Done false
28586 8506 Acked Read 10 Sent Done Done false
28587 8507 Acked Write 8 Sent Done Done false
28588 8508 Acked Write 4 Sent Done Done false
28589 8509 Acked Write 6 Sent Done Done false
28590 8510 Acked Write 1 Sent Done Done false
28591 8511 Acked Read 9 Sent Done Done false
28592 8512 Acked Read 9 Sent Done Done false
28593 8513 Acked Write 8 Sent Done Done false
28594 8514 Acked Read 9 Sent Done Done false
28595 8515 Acked Read 7 Sent Done Done false
28596 8516 Acked Write 3 Sent Done Done false
28597 8517 Acked Write 7 Sent Done Done false
28598 8518 Acked Read 7 Sent Done Done false
28599 8519 Acked Write 9 Sent Done Done false
28600 8520 Acked Read 7 Sent Done Done false
28601 8521 Acked Read 4 Sent Done Done false
28602 8522 Acked Read 4 Sent Done Done false
28603 8523 Acked Flush 0 Sent Done Done false
28604 8524 Acked Write 10 Sent Done Done false
28605 8525 Acked Write 8 Sent Done Done false
28606 8526 Acked Write 1 Sent Done Done false
28607 8527 Acked Read 9 Sent Done Done false
28608 8528 Acked Write 6 Sent Done Done false
28609 8529 Acked Write 10 Sent Done Done false
28610 8530 Acked Read 7 Sent Done Done false
28611 8531 Acked Read 1 Sent Done Done false
28612 8532 Acked Read 4 Sent Done Done false
28613 8533 Acked Write 5 Sent Done Done false
28614 8534 Acked Flush 0 Sent Done Done false
28615 8535 Acked Flush 0 Sent Done Done false
28616 8536 Acked Flush 0 Sent Done Done false
28617 8537 Acked Read 4 Sent Done Done false
28618 8538 Acked Read 7 Sent Done Done false
28619 8539 Acked Write 8 Sent Done Done false
28620 8540 Acked Write 7 Sent Done Done false
28621 8541 Acked Read 8 Sent Done Done false
28622 8542 Acked Write 5 Sent Done Done false
28623 8543 Acked Read 3 Sent Done Done false
28624 8544 Acked Write 4 Sent Done Done false
28625 8545 Acked Write 1 Sent Done Done false
28626 8546 Acked Read 5 Sent Done Done false
28627 8547 Acked Read 10 Sent Done Done false
28628 8548 Acked Read 7 Sent Done Done false
28629 8549 Acked Write 1 Sent Done Done false
28630 8550 Acked Flush 0 Sent Done Done false
28631 8551 Acked Write 2 Sent Done Done false
28632 8552 Acked Read 8 Sent Done Done false
28633 8553 Acked Read 2 Sent Done Done false
28634 STATES DS:0 DS:1 DS:2 TOTAL
28635 Sent 301 0 0 301
28636 Done 0 301 301 602
28637 Skipped 0 0 0 0
28638 Error 0 0 0 0
28639 Last Flush: 8252 8550 8550
28640 Downstairs last five completed: 8252 8251 8250 8249 8248
28641 Upstairs last five completed: 8553 8552 8551 8550 8549
28642 CLIENT: Up:0 ds:301 act:2
28643 [8810] got start action
28644 [8810] I am going to restart
28645 Starting downstairs at port 8810
28646 Make output file at "/tmp/dsc/downstairs-8810.txt"
28647 [8810][0] reports Starting
28648 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1223)
28649 [8810][0] reports Running
28650 ----------------------------------------------------------------
28651 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
28652 States: Offline Active Active
28653 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28654 8253 Acked Read 4 Sent Done Done false
28655 8254 Acked Write 4 Sent Done Done false
28656 8255 Acked Write 4 Sent Done Done false
28657 8256 Acked Write 6 Sent Done Done false
28658 8257 Acked Write 1 Sent Done Done false
28659 8258 Acked Read 9 Sent Done Done false
28660 8259 Acked Read 8 Sent Done Done false
28661 8260 Acked Read 7 Sent Done Done false
28662 8261 Acked Read 3 Sent Done Done false
28663 8262 Acked Read 3 Sent Done Done false
28664 8263 Acked Read 10 Sent Done Done false
28665 8264 Acked Write 1 Sent Done Done false
28666 8265 Acked Write 6 Sent Done Done false
28667 8266 Acked Read 9 Sent Done Done false
28668 8267 Acked Write 8 Sent Done Done false
28669 8268 Acked Flush 0 Sent Done Done false
28670 8269 Acked Flush 0 Sent Done Done false
28671 8270 Acked Read 4 Sent Done Done false
28672 8271 Acked Read 9 Sent Done Done false
28673 8272 Acked Write 3 Sent Done Done false
28674 8273 Acked Write 10 Sent Done Done false
28675 8274 Acked Flush 0 Sent Done Done false
28676 8275 Acked Read 8 Sent Done Done false
28677 8276 Acked Write 9 Sent Done Done false
28678 8277 Acked Read 3 Sent Done Done false
28679 8278 Acked Flush 0 Sent Done Done false
28680 8279 Acked Read 5 Sent Done Done false
28681 8280 Acked Write 7 Sent Done Done false
28682 8281 Acked Write 5 Sent Done Done false
28683 8282 Acked Read 1 Sent Done Done false
28684 8283 Acked Write 10 Sent Done Done false
28685 8284 Acked Read 9 Sent Done Done false
28686 8285 Acked Read 1 Sent Done Done false
28687 8286 Acked Read 6 Sent Done Done false
28688 8287 Acked Read 4 Sent Done Done false
28689 8288 Acked Write 4 Sent Done Done false
28690 8289 Acked Read 9 Sent Done Done false
28691 8290 Acked Read 3 Sent Done Done false
28692 8291 Acked Write 7 Sent Done Done false
28693 8292 Acked Read 4 Sent Done Done false
28694 8293 Acked Write 2 Sent Done Done false
28695 8294 Acked Read 8 Sent Done Done false
28696 8295 Acked Flush 0 Sent Done Done false
28697 8296 Acked Write 10 Sent Done Done false
28698 8297 Acked Write 6 Sent Done Done false
28699 8298 Acked Flush 0 Sent Done Done false
28700 8299 Acked Read 6 Sent Done Done false
28701 8300 Acked Read 8 Sent Done Done false
28702 8301 Acked Write 9 Sent Done Done false
28703 8302 Acked Read 4 Sent Done Done false
28704 8303 Acked Flush 0 Sent Done Done false
28705 8304 Acked Flush 0 Sent Done Done false
28706 8305 Acked Write 1 Sent Done Done false
28707 8306 Acked Read 4 Sent Done Done false
28708 8307 Acked Read 2 Sent Done Done false
28709 8308 Acked Read 4 Sent Done Done false
28710 8309 Acked Read 9 Sent Done Done false
28711 8310 Acked Read 4 Sent Done Done false
28712 8311 Acked Read 5 Sent Done Done false
28713 8312 Acked Read 3 Sent Done Done false
28714 8313 Acked Write 7 Sent Done Done false
28715 8314 Acked Read 9 Sent Done Done false
28716 8315 Acked Read 10 Sent Done Done false
28717 8316 Acked Read 1 Sent Done Done false
28718 8317 Acked Read 4 Sent Done Done false
28719 8318 Acked Read 6 Sent Done Done false
28720 8319 Acked Read 6 Sent Done Done false
28721 8320 Acked Read 1 Sent Done Done false
28722 8321 Acked Write 9 Sent Done Done false
28723 8322 Acked Write 6 Sent Done Done false
28724 8323 Acked Write 9 Sent Done Done false
28725 8324 Acked Read 9 Sent Done Done false
28726 8325 Acked Read 9 Sent Done Done false
28727 8326 Acked Read 2 Sent Done Done false
28728 8327 Acked Read 6 Sent Done Done false
28729 8328 Acked Read 5 Sent Done Done false
28730 8329 Acked Read 10 Sent Done Done false
28731 8330 Acked Read 6 Sent Done Done false
28732 8331 Acked Write 10 Sent Done Done false
28733 8332 Acked Read 4 Sent Done Done false
28734 8333 Acked Flush 0 Sent Done Done false
28735 8334 Acked Read 9 Sent Done Done false
28736 8335 Acked Read 7 Sent Done Done false
28737 8336 Acked Read 9 Sent Done Done false
28738 8337 Acked Write 4 Sent Done Done false
28739 8338 Acked Flush 0 Sent Done Done false
28740 8339 Acked Write 10 Sent Done Done false
28741 8340 Acked Read 2 Sent Done Done false
28742 8341 Acked Read 1 Sent Done Done false
28743 8342 Acked Read 8 Sent Done Done false
28744 8343 Acked Write 10 Sent Done Done false
28745 8344 Acked Read 3 Sent Done Done false
28746 8345 Acked Read 3 Sent Done Done false
28747 8346 Acked Write 2 Sent Done Done false
28748 8347 Acked Write 1 Sent Done Done false
28749 8348 Acked Flush 0 Sent Done Done false
28750 8349 Acked Write 7 Sent Done Done false
28751 8350 Acked Read 5 Sent Done Done false
28752 8351 Acked Write 10 Sent Done Done false
28753 8352 Acked Flush 0 Sent Done Done false
28754 8353 Acked Flush 0 Sent Done Done false
28755 8354 Acked Read 5 Sent Done Done false
28756 8355 Acked Read 10 Sent Done Done false
28757 8356 Acked Read 1 Sent Done Done false
28758 8357 Acked Read 8 Sent Done Done false
28759 8358 Acked Read 5 Sent Done Done false
28760 8359 Acked Read 9 Sent Done Done false
28761 8360 Acked Read 9 Sent Done Done false
28762 8361 Acked Read 7 Sent Done Done false
28763 8362 Acked Read 4 Sent Done Done false
28764 8363 Acked Write 10 Sent Done Done false
28765 8364 Acked Flush 0 Sent Done Done false
28766 8365 Acked Write 3 Sent Done Done false
28767 8366 Acked Read 5 Sent Done Done false
28768 8367 Acked Write 8 Sent Done Done false
28769 8368 Acked Read 8 Sent Done Done false
28770 8369 Acked Write 1 Sent Done Done false
28771 8370 Acked Write 1 Sent Done Done false
28772 8371 Acked Read 9 Sent Done Done false
28773 8372 Acked Write 6 Sent Done Done false
28774 8373 Acked Write 10 Sent Done Done false
28775 8374 Acked Write 9 Sent Done Done false
28776 8375 Acked Flush 0 Sent Done Done false
28777 8376 Acked Write 4 Sent Done Done false
28778 8377 Acked Write 8 Sent Done Done false
28779 8378 Acked Write 10 Sent Done Done false
28780 8379 Acked Write 10 Sent Done Done false
28781 8380 Acked Read 10 Sent Done Done false
28782 8381 Acked Flush 0 Sent Done Done false
28783 8382 Acked Flush 0 Sent Done Done false
28784 8383 Acked Read 6 Sent Done Done false
28785 8384 Acked Read 3 Sent Done Done false
28786 8385 Acked Write 3 Sent Done Done false
28787 8386 Acked Read 10 Sent Done Done false
28788 8387 Acked Read 4 Sent Done Done false
28789 8388 Acked Write 8 Sent Done Done false
28790 8389 Acked Write 9 Sent Done Done false
28791 8390 Acked Read 8 Sent Done Done false
28792 8391 Acked Read 1 Sent Done Done false
28793 8392 Acked Flush 0 Sent Done Done false
28794 8393 Acked Read 3 Sent Done Done false
28795 8394 Acked Write 10 Sent Done Done false
28796 8395 Acked Write 8 Sent Done Done false
28797 8396 Acked Read 3 Sent Done Done false
28798 8397 Acked Flush 0 Sent Done Done false
28799 8398 Acked Write 7 Sent Done Done false
28800 8399 Acked Write 2 Sent Done Done false
28801 8400 Acked Read 2 Sent Done Done false
28802 8401 Acked Read 4 Sent Done Done false
28803 8402 Acked Read 8 Sent Done Done false
28804 8403 Acked Write 5 Sent Done Done false
28805 8404 Acked Read 3 Sent Done Done false
28806 8405 Acked Write 5 Sent Done Done false
28807 8406 Acked Write 10 Sent Done Done false
28808 8407 Acked Write 3 Sent Done Done false
28809 8408 Acked Read 7 Sent Done Done false
28810 8409 Acked Read 4 Sent Done Done false
28811 8410 Acked Write 4 Sent Done Done false
28812 8411 Acked Read 10 Sent Done Done false
28813 8412 Acked Read 1 Sent Done Done false
28814 8413 Acked Write 7 Sent Done Done false
28815 8414 Acked Write 5 Sent Done Done false
28816 8415 Acked Read 6 Sent Done Done false
28817 8416 Acked Write 2 Sent Done Done false
28818 8417 Acked Read 2 Sent Done Done false
28819 8418 Acked Read 1 Sent Done Done false
28820 8419 Acked Read 8 Sent Done Done false
28821 8420 Acked Write 1 Sent Done Done false
28822 8421 Acked Read 1 Sent Done Done false
28823 8422 Acked Read 4 Sent Done Done false
28824 8423 Acked Read 8 Sent Done Done false
28825 8424 Acked Write 6 Sent Done Done false
28826 8425 Acked Read 4 Sent Done Done false
28827 8426 Acked Flush 0 Sent Done Done false
28828 8427 Acked Write 4 Sent Done Done false
28829 8428 Acked Read 2 Sent Done Done false
28830 8429 Acked Write 8 Sent Done Done false
28831 8430 Acked Write 4 Sent Done Done false
28832 8431 Acked Write 9 Sent Done Done false
28833 8432 Acked Read 8 Sent Done Done false
28834 8433 Acked Write 1 Sent Done Done false
28835 8434 Acked Flush 0 Sent Done Done false
28836 8435 Acked Read 10 Sent Done Done false
28837 8436 Acked Read 8 Sent Done Done false
28838 8437 Acked Flush 0 Sent Done Done false
28839 8438 Acked Write 3 Sent Done Done false
28840 8439 Acked Read 4 Sent Done Done false
28841 8440 Acked Write 5 Sent Done Done false
28842 8441 Acked Write 9 Sent Done Done false
28843 8442 Acked Read 3 Sent Done Done false
28844 8443 Acked Write 2 Sent Done Done false
28845 8444 Acked Read 10 Sent Done Done false
28846 8445 Acked Read 5 Sent Done Done false
28847 8446 Acked Write 4 Sent Done Done false
28848 8447 Acked Read 6 Sent Done Done false
28849 8448 Acked Read 10 Sent Done Done false
28850 8449 Acked Read 9 Sent Done Done false
28851 8450 Acked Write 10 Sent Done Done false
28852 8451 Acked Read 4 Sent Done Done false
28853 8452 Acked Read 7 Sent Done Done false
28854 8453 Acked Read 4 Sent Done Done false
28855 8454 Acked Write 4 Sent Done Done false
28856 8455 Acked Flush 0 Sent Done Done false
28857 8456 Acked Read 5 Sent Done Done false
28858 8457 Acked Write 3 Sent Done Done false
28859 8458 Acked Read 3 Sent Done Done false
28860 8459 Acked Write 9 Sent Done Done false
28861 8460 Acked Read 5 Sent Done Done false
28862 8461 Acked Write 10 Sent Done Done false
28863 8462 Acked Read 5 Sent Done Done false
28864 8463 Acked Read 1 Sent Done Done false
28865 8464 Acked Write 4 Sent Done Done false
28866 8465 Acked Write 6 Sent Done Done false
28867 8466 Acked Write 1 Sent Done Done false
28868 8467 Acked Write 6 Sent Done Done false
28869 8468 Acked Write 8 Sent Done Done false
28870 8469 Acked Read 8 Sent Done Done false
28871 8470 Acked Read 1 Sent Done Done false
28872 8471 Acked Write 1 Sent Done Done false
28873 8472 Acked Write 1 Sent Done Done false
28874 8473 Acked Flush 0 Sent Done Done false
28875 8474 Acked Read 2 Sent Done Done false
28876 8475 Acked Write 7 Sent Done Done false
28877 8476 Acked Write 6 Sent Done Done false
28878 8477 Acked Read 8 Sent Done Done false
28879 8478 Acked Write 2 Sent Done Done false
28880 8479 Acked Write 1 Sent Done Done false
28881 8480 Acked Write 5 Sent Done Done false
28882 8481 Acked Read 3 Sent Done Done false
28883 8482 Acked Read 9 Sent Done Done false
28884 8483 Acked Write 6 Sent Done Done false
28885 8484 Acked Read 7 Sent Done Done false
28886 8485 Acked Write 4 Sent Done Done false
28887 8486 Acked Read 4 Sent Done Done false
28888 8487 Acked Flush 0 Sent Done Done false
28889 8488 Acked Read 7 Sent Done Done false
28890 8489 Acked Read 7 Sent Done Done false
28891 8490 Acked Write 6 Sent Done Done false
28892 8491 Acked Write 8 Sent Done Done false
28893 8492 Acked Write 10 Sent Done Done false
28894 8493 Acked Write 10 Sent Done Done false
28895 8494 Acked Flush 0 Sent Done Done false
28896 8495 Acked Read 10 Sent Done Done false
28897 8496 Acked Read 5 Sent Done Done false
28898 8497 Acked Read 6 Sent Done Done false
28899 8498 Acked Read 7 Sent Done Done false
28900 8499 Acked Read 9 Sent Done Done false
28901 8500 Acked Read 4 Sent Done Done false
28902 8501 Acked Write 7 Sent Done Done false
28903 8502 Acked Write 6 Sent Done Done false
28904 8503 Acked Write 9 Sent Done Done false
28905 8504 Acked Read 5 Sent Done Done false
28906 8505 Acked Write 7 Sent Done Done false
28907 8506 Acked Read 10 Sent Done Done false
28908 8507 Acked Write 8 Sent Done Done false
28909 8508 Acked Write 4 Sent Done Done false
28910 8509 Acked Write 6 Sent Done Done false
28911 8510 Acked Write 1 Sent Done Done false
28912 8511 Acked Read 9 Sent Done Done false
28913 8512 Acked Read 9 Sent Done Done false
28914 8513 Acked Write 8 Sent Done Done false
28915 8514 Acked Read 9 Sent Done Done false
28916 8515 Acked Read 7 Sent Done Done false
28917 8516 Acked Write 3 Sent Done Done false
28918 8517 Acked Write 7 Sent Done Done false
28919 8518 Acked Read 7 Sent Done Done false
28920 8519 Acked Write 9 Sent Done Done false
28921 8520 Acked Read 7 Sent Done Done false
28922 8521 Acked Read 4 Sent Done Done false
28923 8522 Acked Read 4 Sent Done Done false
28924 8523 Acked Flush 0 Sent Done Done false
28925 8524 Acked Write 10 Sent Done Done false
28926 8525 Acked Write 8 Sent Done Done false
28927 8526 Acked Write 1 Sent Done Done false
28928 8527 Acked Read 9 Sent Done Done false
28929 8528 Acked Write 6 Sent Done Done false
28930 8529 Acked Write 10 Sent Done Done false
28931 8530 Acked Read 7 Sent Done Done false
28932 8531 Acked Read 1 Sent Done Done false
28933 8532 Acked Read 4 Sent Done Done false
28934 8533 Acked Write 5 Sent Done Done false
28935 8534 Acked Flush 0 Sent Done Done false
28936 8535 Acked Flush 0 Sent Done Done false
28937 8536 Acked Flush 0 Sent Done Done false
28938 8537 Acked Read 4 Sent Done Done false
28939 8538 Acked Read 7 Sent Done Done false
28940 8539 Acked Write 8 Sent Done Done false
28941 8540 Acked Write 7 Sent Done Done false
28942 8541 Acked Read 8 Sent Done Done false
28943 8542 Acked Write 5 Sent Done Done false
28944 8543 Acked Read 3 Sent Done Done false
28945 8544 Acked Write 4 Sent Done Done false
28946 8545 Acked Write 1 Sent Done Done false
28947 8546 Acked Read 5 Sent Done Done false
28948 8547 Acked Read 10 Sent Done Done false
28949 8548 Acked Read 7 Sent Done Done false
28950 8549 Acked Write 1 Sent Done Done false
28951 8550 Acked Flush 0 Sent Done Done false
28952 8551 Acked Write 2 Sent Done Done false
28953 8552 Acked Read 8 Sent Done Done false
28954 8553 Acked Read 2 Sent Done Done false
28955 8554 Acked Flush 0 Sent Done Done false
28956 STATES DS:0 DS:1 DS:2 TOTAL
28957 Sent 302 0 0 302
28958 Done 0 302 302 604
28959 Skipped 0 0 0 0
28960 Error 0 0 0 0
28961 Last Flush: 8252 8554 8554
28962 Downstairs last five completed: 8252 8251 8250 8249 8248
28963 Upstairs last five completed: 8554 8553 8552 8551 8550
28964 CLIENT: Up:0 ds:302 act:2
289652024-11-01T16:56:33.458ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
289662024-11-01T16:56:33.458ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36806) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
289672024-11-01T16:56:33.459ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
289682024-11-01T16:56:33.459ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
289692024-11-01T16:56:33.459ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
289702024-11-01T16:56:33.459ZINFOcrucible: send last flush ID to this DS: 8252 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
289712024-11-01T16:56:33.460ZINFOcrucible: Replied this last flush ID: 8252 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
289722024-11-01T16:56:33.460ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
289732024-11-01T16:56:33.460ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
289742024-11-01T16:56:33.460ZINFOcrucible: [0] client re-new 302 jobs since flush 8252 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
289752024-11-01T16:56:33.460ZINFOcrucible: [0] Replayed 302 jobs since flush: 8252 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
28976 ----------------------------------------------------------------
28977 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28978 Downstairs last five completed: 8554 8553 8552 8551 8550
28979 Upstairs last five completed: 8554 8553 8552 8551 8550
28980 CLIENT: Up:0 ds:0 act:3
28981 Replay: All jobs finished
28982 Main task has work to do, go find it
28983 got dsc Stop(0)
28984 stop 0
28985 [8810] Got stop action so:false kr:false
28986 Nov 01 16:56:35.766 INFO request completed, latency_us: 329, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 97479cb9-32e6-4aa0-a27b-395d831062e0, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
289872024-11-01T16:56:35.766ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
289882024-11-01T16:56:35.767ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
289892024-11-01T16:56:35.767ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
289902024-11-01T16:56:35.767ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
28991 Nov 01 16:56:35.767 INFO request completed, latency_us: 128, response_code: 200, uri: /state/cid/0, method: GET, req_id: d4823bcf-2d3f-4ced-9f6d-2b5bdd776b87, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
289922024-11-01T16:56:35.767ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
28993 [8810] Exited with: ExitStatus(unix_wait_status(9))
28994 [8810][0] reports Exit
28995 Nov 01 16:56:39.768 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/0, method: GET, req_id: d6a4abae-43b9-4573-8b3e-caef8ee2ef46, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
28996 001/300 Write block 2290 len 8192 data: 8 9
28997 002/300 Read block 2165 len 4096
28998 003/300 Read block 3702 len 16384
28999 004/300 Write block 4522 len 20480 data: 5 7 7 8 7
29000 005/300 Read block 2392 len 12288
29001 006/300 Write block 3590 len 16384 data: 9 7 8 6
29002 007/300 Read block 1350 len 28672
29003 008/300 Read block 2373 len 36864
29004 009/300 Write block 3860 len 40960 data: 11 10 9 9 10 8 11 10 9 8
29005 010/300 Write block 3354 len 16384 data: 6 6 5 5
29006 011/300 Read block 4765 len 20480
29007 012/300 Read block 1228 len 20480
29008 013/300 Flush
29009 014/300 Write block 4245 len 40960 data: 4 5 6 6 6 5 6 5 4 5
29010 015/300 Read block 3280 len 24576
29011 016/300 Write block 2976 len 20480 data: 6 6 5 7 6
29012 017/300 Read block 831 len 28672
29013 018/300 Read block 3977 len 36864
29014 019/300 Read block 1925 len 24576
29015 020/300 Read block 161 len 8192
29016 021/300 Flush
29017 022/300 Read block 1034 len 4096
29018 023/300 Write block 249 len 20480 data: 8 7 8 8 7
29019 024/300 Read block 23 len 8192
29020 025/300 Read block 2040 len 8192
29021 026/300 Write block 1988 len 12288 data: 4 5 4
29022 027/300 Write block 1619 len 20480 data: 5 5 3 3 2
29023 028/300 Read block 3353 len 20480
29024 029/300 Write block 2110 len 16384 data: 4 5 5 5
29025 030/300 Flush
29026 031/300 Write block 3591 len 36864 data: 8 9 7 5 5 4 5 4 5
29027 032/300 Write block 384 len 4096 data: 7
29028 033/300 Write block 4534 len 4096 data: 8
29029 034/300 Write block 3292 len 8192 data: 5 4
29030 035/300 Read block 2631 len 32768
29031 036/300 Write block 4623 len 4096 data: 3
29032 037/300 Write block 4257 len 40960 data: 6 5 5 5 5 5 5 4 4 5
29033 038/300 Flush
29034 039/300 Flush
29035 040/300 Write block 4688 len 32768 data: 6 5 5 4 4 3 4 4
29036 041/300 Write block 3629 len 8192 data: 5 5
29037 042/300 Write block 3495 len 20480 data: 3 3 3 4 5
29038 043/300 Write block 4906 len 36864 data: 6 4 3 4 4 4 4 5 6
29039 044/300 Write block 490 len 28672 data: 5 4 4 4 5 4 3
29040 045/300 Read block 913 len 40960
29041 046/300 Read block 3385 len 8192
29042 047/300 Write block 3867 len 16384 data: 11 10 9 9
29043 048/300 Read block 2949 len 8192
29044 049/300 Read block 276 len 4096
29045 050/300 Write block 4328 len 4096 data: 3
29046 051/300 Write block 914 len 8192 data: 7 8
29047 052/300 Flush
29048 053/300 Flush
29049 054/300 Read block 1907 len 32768
29050 055/300 Read block 432 len 28672
29051 056/300 Read block 468 len 12288
29052 057/300 Write block 2541 len 20480 data: 6 7 6 5 5
29053 058/300 Write block 3875 len 12288 data: 5 5 3
29054 059/300 Write block 2222 len 4096 data: 9
29055 060/300 Flush
29056 061/300 Read block 743 len 40960
29057 062/300 Read block 3489 len 40960
29058 063/300 Write block 535 len 36864 data: 7 6 5 3 3 3 4 5 4
29059 064/300 Write block 2274 len 32768 data: 7 6 3 3 4 4 5 6
29060 065/300 Read block 1157 len 20480
29061 066/300 Read block 663 len 4096
29062 067/300 Write block 1139 len 20480 data: 9 8 4 2 2
29063 068/300 Read block 2710 len 4096
29064 069/300 Flush
29065 070/300 Read block 3071 len 36864
29066 071/300 Read block 4380 len 40960
29067 072/300 Write block 2850 len 8192 data: 5 5
29068 073/300 Write block 1585 len 40960 data: 4 5 6 6 6 6 7 4 4 8
29069 074/300 Write block 1841 len 36864 data: 4 5 6 5 5 5 5 6 6
29070 075/300 Read block 129 len 12288
29071 076/300 Read block 3803 len 20480
29072 077/300 Read block 3363 len 32768
29073 078/300 Read block 2411 len 32768
29074 079/300 Flush
29075 080/300 Read block 1868 len 24576
29076 081/300 Write block 3943 len 36864 data: 6 5 4 7 7 6 8 7 7
29077 082/300 Read block 2128 len 8192
29078 083/300 Flush
29079 084/300 Read block 3026 len 24576
29080 085/300 Read block 3464 len 8192
29081 086/300 Write block 2724 len 36864 data: 9 8 5 6 6 7 6 5 7
29082 087/300 Write block 4079 len 20480 data: 6 5 5 5 5
29083 088/300 Flush
29084 089/300 Read block 651 len 28672
29085 090/300 Read block 2320 len 40960
29086 091/300 Write block 4339 len 12288 data: 4 5 5
29087 092/300 Read block 411 len 40960
29088 093/300 Write block 4454 len 40960 data: 7 6 4 3 4 5 6 7 7 7
29089 094/300 Write block 2758 len 36864 data: 7 7 7 6 6 5 6 8 7
29090 095/300 Write block 4392 len 32768 data: 6 6 5 4 4 4 4 3
29091 096/300 Flush
29092 097/300 Write block 2325 len 28672 data: 3 3 3 3 3 6 6
29093 098/300 Read block 4463 len 12288
29094 099/300 Read block 1981 len 32768
29095 100/300 Write block 4404 len 4096 data: 5
29096 101/300 Read block 885 len 32768
29097 102/300 Write block 1601 len 28672 data: 5 5 4 3 4 4 3
29098 103/300 Read block 250 len 24576
29099 104/300 Write block 3692 len 4096 data: 8
29100 105/300 Read block 3656 len 32768
29101 106/300 Write block 69 len 8192 data: 5 6
29102 107/300 Read block 3722 len 36864
29103 108/300 Write block 357 len 36864 data: 2 2 3 4 4 4 5 7 9
29104 109/300 Read block 4557 len 20480
29105 110/300 Read block 4075 len 36864
29106 111/300 Read block 3360 len 24576
29107 112/300 Write block 597 len 36864 data: 5 5 6 7 6 5 5 5 4
29108 113/300 Read block 2395 len 20480
29109 114/300 Flush
29110 115/300 Write block 4345 len 40960 data: 4 4 3 4 6 4 4 4 4 3
29111 116/300 Read block 3899 len 16384
29112 117/300 Read block 537 len 28672
29113 118/300 Read block 1689 len 12288
29114 119/300 Write block 2972 len 12288 data: 5 7 8
29115 120/300 Read block 3950 len 12288
29116 121/300 Read block 951 len 32768
29117 122/300 Write block 3003 len 32768 data: 7 6 4 4 4 3 5 3
29118 123/300 Read block 214 len 4096
29119 124/300 Read block 105 len 28672
29120 125/300 Read block 1523 len 20480
29121 126/300 Read block 3611 len 40960
29122 127/300 Read block 4789 len 36864
29123 128/300 Read block 999 len 40960
29124 129/300 Flush
29125 130/300 Write block 981 len 4096 data: 4
29126 131/300 Read block 2361 len 4096
29127 132/300 Write block 1526 len 24576 data: 8 8 8 7 6 5
29128 133/300 Read block 2595 len 8192
29129 134/300 Read block 4702 len 12288
29130 135/300 Read block 393 len 24576
29131 136/300 Write block 1212 len 28672 data: 7 10 9 8 7 7 10
29132 137/300 Read block 4240 len 24576
29133 138/300 Write block 736 len 4096 data: 8
29134 139/300 Write block 81 len 28672 data: 7 6 4 4 4 5 4
29135 140/300 Write block 3872 len 12288 data: 6 7 6
29136 141/300 Read block 2021 len 4096
29137 142/300 Write block 522 len 24576 data: 6 6 7 7 7 7
29138 143/300 Read block 4702 len 20480
29139 144/300 Read block 178 len 4096
29140 145/300 Write block 291 len 4096 data: 5
29141 146/300 Read block 2436 len 16384
29142 147/300 Flush
29143 148/300 Read block 1682 len 20480
29144 149/300 Write block 670 len 12288 data: 5 5 5
29145 150/300 Read block 345 len 20480
29146 151/300 Flush
29147 152/300 Read block 3772 len 12288
29148 153/300 Read block 2540 len 8192
29149 154/300 Read block 4387 len 40960
29150 155/300 Write block 198 len 16384 data: 9 11 8 6
29151 156/300 Flush
29152 157/300 Write block 2395 len 24576 data: 4 5 4 4 4 4
29153 158/300 Read block 3829 len 12288
29154 159/300 Write block 468 len 36864 data: 4 3 4 3 3 3 3 5 5
29155 160/300 Read block 4681 len 8192
29156 161/300 Read block 2698 len 16384
29157 162/300 Read block 2999 len 20480
29158 163/300 Write block 327 len 16384 data: 3 3 2 3
29159 164/300 Read block 4784 len 32768
29160 165/300 Write block 4980 len 12288 data: 6 7 7
29161 166/300 Read block 3244 len 16384
29162 167/300 Read block 4530 len 36864
29163 168/300 Read block 4734 len 20480
29164 169/300 Read block 4848 len 4096
29165 170/300 Read block 1122 len 40960
29166 171/300 Write block 4180 len 8192 data: 5 5
29167 172/300 Read block 4958 len 36864
29168 173/300 Write block 810 len 40960 data: 5 4 4 4 3 3 2 2 2 2
29169 174/300 Read block 3880 len 4096
29170 175/300 Write block 1527 len 12288 data: 9 9 8
29171 176/300 Read block 4814 len 12288
29172 177/300 Write block 3556 len 24576 data: 5 5 6 6 5 3
29173 178/300 Read block 764 len 8192
29174 179/300 Write block 3314 len 4096 data: 4
29175 180/300 Write block 4361 len 8192 data: 6 7
29176 181/300 Read block 3508 len 32768
29177 182/300 Read block 3675 len 28672
29178 183/300 Read block 574 len 24576
29179 184/300 Write block 2893 len 16384 data: 5 5 5 4
29180 185/300 Read block 999 len 4096
29181 186/300 Read block 3373 len 32768
29182 187/300 Read block 4766 len 24576
29183 188/300 Write block 376 len 36864 data: 4 4 4 6 7 5 6 6 8
29184 189/300 Read block 1756 len 12288
29185 190/300 Write block 4323 len 4096 data: 7
29186 191/300 Write block 1003 len 24576 data: 6 6 6 5 2 3
29187 192/300 Write block 4608 len 12288 data: 4 3 5
29188 193/300 Write block 4259 len 8192 data: 6 6
29189 194/300 Write block 2473 len 40960 data: 8 9 10 10 10 11 11 7 7 6
29190 195/300 Write block 439 len 32768 data: 5 5 3 3 2 3 2 4
29191 196/300 Read block 478 len 12288
29192 197/300 Write block 4967 len 24576 data: 6 5 7 7 5 3
29193 198/300 Read block 2400 len 24576
29194 199/300 Read block 3712 len 16384
29195 200/300 Read block 4766 len 28672
29196 201/300 Read block 4310 len 20480
29197 202/300 Flush
29198 203/300 Write block 2327 len 28672 data: 4 4 4 7 7 5 5
29199 204/300 Flush
29200 205/300 Read block 2144 len 28672
29201 206/300 Write block 3346 len 24576 data: 4 5 6 7 7 6
29202 207/300 Write block 2415 len 8192 data: 5 5
29203 208/300 Flush
29204 209/300 Read block 3617 len 32768
29205 210/300 Read block 2405 len 32768
29206 211/300 Read block 4757 len 32768
29207 212/300 Write block 961 len 4096 data: 2
29208 213/300 Read block 3652 len 40960
29209 214/300 Read block 4678 len 20480
29210 215/300 Read block 684 len 8192
29211 216/300 Write block 4915 len 36864 data: 7 9 9 8 7 6 7 6 7
29212 217/300 Write block 2046 len 8192 data: 4 4
29213 218/300 Read block 4281 len 12288
29214 219/300 Read block 4138 len 24576
29215 220/300 Read block 2696 len 24576
29216 221/300 Flush
29217 222/300 Write block 3318 len 36864 data: 4 4 4 6 6 5 4 5 4
29218 223/300 Read block 1227 len 12288
29219 224/300 Read block 102 len 20480
29220 225/300 Write block 1193 len 32768 data: 7 8 9 9 8 7 9 8
29221 226/300 Read block 1554 len 40960
29222 227/300 Flush
29223 228/300 Write block 2198 len 16384 data: 4 5 6 4
29224 229/300 Read block 1134 len 36864
29225 230/300 Write block 1688 len 20480 data: 3 4 5 6 6
29226 231/300 Write block 1950 len 8192 data: 6 5
29227 232/300 Write block 4557 len 40960 data: 6 7 6 5 4 3 4 4 3 4
29228 233/300 Write block 1408 len 24576 data: 4 3 5 4 4 5
29229 234/300 Read block 1524 len 32768
29230 235/300 Flush
29231 236/300 Write block 4204 len 20480 data: 10 9 8 7 6
29232 237/300 Write block 2886 len 36864 data: 7 7 7 8 8 7 5 6 6
29233 238/300 Write block 2976 len 4096 data: 7
29234 239/300 Read block 1702 len 24576
29235 240/300 Write block 597 len 36864 data: 6 6 7 8 7 6 6 6 5
29236 241/300 Write block 1495 len 20480 data: 5 6 7 6 7
29237 242/300 Read block 2075 len 28672
29238 243/300 Read block 13 len 12288
29239 244/300 Flush
29240 245/300 Write block 3941 len 40960 data: 6 6 7 6 5 8 8 7 9 8
29241 246/300 Flush
29242 247/300 Read block 450 len 8192
29243 248/300 Write block 387 len 40960 data: 4 4 3 2 3 3 3 3 3 3
29244 249/300 Read block 1943 len 8192
29245 250/300 Read block 3161 len 28672
29246 251/300 Read block 2624 len 8192
29247 252/300 Read block 835 len 4096
29248 253/300 Read block 2247 len 28672
29249 254/300 Read block 3539 len 32768
29250 255/300 Write block 3479 len 28672 data: 7 7 7 6 7 5 6
29251 256/300 Read block 3646 len 20480
29252 257/300 Read block 165 len 12288
29253 258/300 Read block 424 len 36864
29254 259/300 Write block 622 len 36864 data: 7 6 7 7 5 6 5 5 4
29255 260/300 Read block 3973 len 36864
29256 261/300 Read block 605 len 4096
29257 262/300 Write block 1887 len 40960 data: 4 4 3 3 4 4 4 3 2 3
29258 263/300 Write block 3577 len 16384 data: 2 2 3 3
29259 264/300 Read block 4711 len 16384
29260 265/300 Read block 1300 len 12288
29261 266/300 Write block 4905 len 20480 data: 6 7 5 4 5
29262 267/300 Flush
29263 268/300 Read block 1021 len 16384
29264 269/300 Write block 4932 len 36864 data: 3 5 6 4 3 3 3 4 5
29265 270/300 Read block 843 len 24576
29266 271/300 Read block 4085 len 12288
29267 272/300 Read block 1128 len 20480
29268 273/300 Read block 3319 len 36864
29269 274/300 Read block 1091 len 8192
29270 275/300 Write block 1827 len 12288 data: 5 5 4
29271 276/300 Read block 3223 len 12288
29272 277/300 Read block 1086 len 20480
29273 278/300 Write block 4940 len 8192 data: 6 7
29274 279/300 Read block 1287 len 28672
29275 280/300 Read block 3662 len 32768
29276 281/300 Read block 2889 len 28672
29277 282/300 Write block 4629 len 16384 data: 7 7 7 8
29278 283/300 Write block 3282 len 24576 data: 3 3 3 4 5 6
29279 284/300 Read block 1935 len 12288
29280 285/300 Read block 4976 len 4096
29281 286/300 Write block 4965 len 4096 data: 7
29282 287/300 Flush
29283 288/300 Read block 2785 len 36864
29284 289/300 Write block 4724 len 28672 data: 4 3 4 4 4 6 6
29285 290/300 Read block 2717 len 8192
29286 291/300 Write block 1432 len 20480 data: 8 7 7 8 10
29287 292/300 Read block 1986 len 20480
29288 293/300 Write block 3178 len 32768 data: 6 5 4 4 4 5 7 6
29289 294/300 Read block 2188 len 24576
29290 295/300 Write block 2224 len 32768 data: 9 8 8 7 7 5 7 5
29291 296/300 Flush
29292 297/300 Write block 1192 len 16384 data: 5 8 9 10
29293 298/300 Read block 4782 len 40960
29294 299/300 Read block 2445 len 16384
29295 300/300 Flush
29296 301/300 Write block 1477 len 16384 data: 8 8 7 7
29297 Main task has work to do, go find it
29298 got dsc Start(0)
29299 start 0
29300 Nov 01 16:56:40.078 INFO request completed, latency_us: 257, response_code: 204, uri: /start/cid/0, method: POST, req_id: 06bb9773-6a68-4a14-a195-8c6ea2478e20, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
29301 Replay: started 0, returned:Ok(())
29302 ----------------------------------------------------------------
29303 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29304 States: Offline Active Active
29305 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29306 8555 Acked Write 2 Sent Done Done false
29307 8556 Acked Read 1 Sent Done Done false
29308 8557 Acked Read 4 Sent Done Done false
29309 8558 Acked Write 5 Sent Done Done false
29310 8559 Acked Read 3 Sent Done Done false
29311 8560 Acked Write 4 Sent Done Done false
29312 8561 Acked Read 7 Sent Done Done false
29313 8562 Acked Read 9 Sent Done Done false
29314 8563 Acked Write 10 Sent Done Done false
29315 8564 Acked Write 4 Sent Done Done false
29316 8565 Acked Read 5 Sent Done Done false
29317 8566 Acked Read 5 Sent Done Done false
29318 8567 Acked Flush 0 Sent Done Done false
29319 8568 Acked Write 10 Sent Done Done false
29320 8569 Acked Read 6 Sent Done Done false
29321 8570 Acked Write 5 Sent Done Done false
29322 8571 Acked Read 7 Sent Done Done false
29323 8572 Acked Read 9 Sent Done Done false
29324 8573 Acked Read 6 Sent Done Done false
29325 8574 Acked Read 2 Sent Done Done false
29326 8575 Acked Flush 0 Sent Done Done false
29327 8576 Acked Read 1 Sent Done Done false
29328 8577 Acked Write 5 Sent Done Done false
29329 8578 Acked Read 2 Sent Done Done false
29330 8579 Acked Read 2 Sent Done Done false
29331 8580 Acked Write 3 Sent Done Done false
29332 8581 Acked Write 5 Sent Done Done false
29333 8582 Acked Read 5 Sent Done Done false
29334 8583 Acked Write 4 Sent Done Done false
29335 8584 Acked Flush 0 Sent Done Done false
29336 8585 Acked Write 9 Sent Done Done false
29337 8586 Acked Write 1 Sent Done Done false
29338 8587 Acked Write 1 Sent Done Done false
29339 8588 Acked Write 2 Sent Done Done false
29340 8589 Acked Read 8 Sent Done Done false
29341 8590 Acked Write 1 Sent Done Done false
29342 8591 Acked Write 10 Sent Done Done false
29343 8592 Acked Flush 0 Sent Done Done false
29344 8593 Acked Flush 0 Sent Done Done false
29345 8594 Acked Write 8 Sent Done Done false
29346 8595 Acked Write 2 Sent Done Done false
29347 8596 Acked Write 5 Sent Done Done false
29348 8597 Acked Write 9 Sent Done Done false
29349 8598 Acked Write 7 Sent Done Done false
29350 8599 Acked Read 10 Sent Done Done false
29351 8600 Acked Read 2 Sent Done Done false
29352 8601 Acked Write 4 Sent Done Done false
29353 8602 Acked Read 2 Sent Done Done false
29354 8603 Acked Read 1 Sent Done Done false
29355 8604 Acked Write 1 Sent Done Done false
29356 8605 Acked Write 2 Sent Done Done false
29357 8606 Acked Flush 0 Sent Done Done false
29358 8607 Acked Flush 0 Sent Done Done false
29359 8608 Acked Read 8 Sent Done Done false
29360 8609 Acked Read 7 Sent Done Done false
29361 8610 Acked Read 3 Sent Done Done false
29362 8611 Acked Write 5 Sent Done Done false
29363 8612 Acked Write 3 Sent Done Done false
29364 8613 Acked Write 1 Sent Done Done false
29365 8614 Acked Flush 0 Sent Done Done false
29366 8615 Acked Read 10 Sent Done Done false
29367 8616 Acked Read 10 Sent Done Done false
29368 8617 Acked Write 9 Sent Done Done false
29369 8618 Acked Write 8 Sent Done Done false
29370 8619 Acked Read 5 Sent Done Done false
29371 8620 Acked Read 1 Sent Done Done false
29372 8621 Acked Write 5 Sent Done Done false
29373 8622 Acked Read 1 Sent Done Done false
29374 8623 Acked Flush 0 Sent Done Done false
29375 8624 Acked Read 9 Sent Done Done false
29376 8625 Acked Read 10 Sent Done Done false
29377 8626 Acked Write 2 Sent Done Done false
29378 8627 Acked Write 10 Sent Done Done false
29379 8628 Acked Write 9 Sent Done Done false
29380 8629 Acked Read 3 Sent Done Done false
29381 8630 Acked Read 5 Sent Done Done false
29382 8631 Acked Read 8 Sent Done Done false
29383 8632 Acked Read 8 Sent Done Done false
29384 8633 Acked Flush 0 Sent Done Done false
29385 8634 Acked Read 6 Sent Done Done false
29386 8635 Acked Write 9 Sent Done Done false
29387 8636 Acked Read 2 Sent Done Done false
29388 8637 Acked Flush 0 Sent Done Done false
29389 8638 Acked Read 6 Sent Done Done false
29390 8639 Acked Read 2 Sent Done Done false
29391 8640 Acked Write 9 Sent Done Done false
29392 8641 Acked Write 5 Sent Done Done false
29393 8642 Acked Flush 0 Sent Done Done false
29394 8643 Acked Read 7 Sent Done Done false
29395 8644 Acked Read 10 Sent Done Done false
29396 8645 Acked Write 3 Sent Done Done false
29397 8646 Acked Read 10 Sent Done Done false
29398 8647 Acked Write 10 Sent Done Done false
29399 8648 Acked Write 9 Sent Done Done false
29400 8649 Acked Write 8 Sent Done Done false
29401 8650 Acked Flush 0 Sent Done Done false
29402 8651 Acked Write 7 Sent Done Done false
29403 8652 Acked Read 3 Sent Done Done false
29404 8653 Acked Read 8 Sent Done Done false
29405 8654 Acked Write 1 Sent Done Done false
29406 8655 Acked Read 8 Sent Done Done false
29407 8656 Acked Write 7 Sent Done Done false
29408 8657 Acked Read 6 Sent Done Done false
29409 8658 Acked Write 1 Sent Done Done false
29410 8659 Acked Read 8 Sent Done Done false
29411 8660 Acked Write 2 Sent Done Done false
29412 8661 Acked Read 9 Sent Done Done false
29413 8662 Acked Write 9 Sent Done Done false
29414 8663 Acked Read 5 Sent Done Done false
29415 8664 Acked Read 9 Sent Done Done false
29416 8665 Acked Read 6 Sent Done Done false
29417 8666 Acked Write 9 Sent Done Done false
29418 8667 Acked Read 5 Sent Done Done false
29419 8668 Acked Flush 0 Sent Done Done false
29420 8669 Acked Write 10 Sent Done Done false
29421 8670 Acked Read 4 Sent Done Done false
29422 8671 Acked Read 7 Sent Done Done false
29423 8672 Acked Read 3 Sent Done Done false
29424 8673 Acked Write 3 Sent Done Done false
29425 8674 Acked Read 3 Sent Done Done false
29426 8675 Acked Read 8 Sent Done Done false
29427 8676 Acked Write 8 Sent Done Done false
29428 8677 Acked Read 1 Sent Done Done false
29429 8678 Acked Read 7 Sent Done Done false
29430 8679 Acked Read 5 Sent Done Done false
29431 8680 Acked Read 10 Sent Done Done false
29432 8681 Acked Read 9 Sent Done Done false
29433 8682 Acked Read 10 Sent Done Done false
29434 8683 Acked Flush 0 Sent Done Done false
29435 8684 Acked Write 1 Sent Done Done false
29436 8685 Acked Read 1 Sent Done Done false
29437 8686 Acked Write 6 Sent Done Done false
29438 8687 Acked Read 2 Sent Done Done false
29439 8688 Acked Read 3 Sent Done Done false
29440 8689 Acked Read 6 Sent Done Done false
29441 8690 Acked Write 7 Sent Done Done false
29442 8691 Acked Read 6 Sent Done Done false
29443 8692 Acked Write 1 Sent Done Done false
29444 8693 Acked Write 7 Sent Done Done false
29445 8694 Acked Write 3 Sent Done Done false
29446 8695 Acked Read 1 Sent Done Done false
29447 8696 Acked Write 6 Sent Done Done false
29448 8697 Acked Read 5 Sent Done Done false
29449 8698 Acked Read 1 Sent Done Done false
29450 8699 Acked Write 1 Sent Done Done false
29451 8700 Acked Read 4 Sent Done Done false
29452 8701 Acked Flush 0 Sent Done Done false
29453 8702 Acked Read 5 Sent Done Done false
29454 8703 Acked Write 3 Sent Done Done false
29455 8704 Acked Read 5 Sent Done Done false
29456 8705 Acked Flush 0 Sent Done Done false
29457 8706 Acked Read 3 Sent Done Done false
29458 8707 Acked Read 2 Sent Done Done false
29459 8708 Acked Read 10 Sent Done Done false
29460 8709 Acked Write 4 Sent Done Done false
29461 8710 Acked Flush 0 Sent Done Done false
29462 8711 Acked Write 6 Sent Done Done false
29463 8712 Acked Read 3 Sent Done Done false
29464 8713 Acked Write 9 Sent Done Done false
29465 8714 Acked Read 2 Sent Done Done false
29466 8715 Acked Read 4 Sent Done Done false
29467 8716 Acked Read 5 Sent Done Done false
29468 8717 Acked Write 4 Sent Done Done false
29469 8718 Acked Read 8 Sent Done Done false
29470 8719 Acked Write 3 Sent Done Done false
29471 8720 Acked Read 4 Sent Done Done false
29472 8721 Acked Read 9 Sent Done Done false
29473 8722 Acked Read 5 Sent Done Done false
29474 8723 Acked Read 1 Sent Done Done false
29475 8724 Acked Read 10 Sent Done Done false
29476 8725 Acked Write 2 Sent Done Done false
29477 8726 Acked Read 9 Sent Done Done false
29478 8727 Acked Write 10 Sent Done Done false
29479 8728 Acked Read 1 Sent Done Done false
29480 8729 Acked Write 3 Sent Done Done false
29481 8730 Acked Read 3 Sent Done Done false
29482 8731 Acked Write 6 Sent Done Done false
29483 8732 Acked Read 2 Sent Done Done false
29484 8733 Acked Write 1 Sent Done Done false
29485 8734 Acked Write 2 Sent Done Done false
29486 8735 Acked Read 8 Sent Done Done false
29487 8736 Acked Read 7 Sent Done Done false
29488 8737 Acked Read 6 Sent Done Done false
29489 8738 Acked Write 4 Sent Done Done false
29490 8739 Acked Read 1 Sent Done Done false
29491 8740 Acked Read 8 Sent Done Done false
29492 8741 Acked Read 6 Sent Done Done false
29493 8742 Acked Write 9 Sent Done Done false
29494 8743 Acked Read 3 Sent Done Done false
29495 8744 Acked Write 1 Sent Done Done false
29496 8745 Acked Write 6 Sent Done Done false
29497 8746 Acked Write 3 Sent Done Done false
29498 8747 Acked Write 2 Sent Done Done false
29499 8748 Acked Write 10 Sent Done Done false
29500 8749 Acked Write 8 Sent Done Done false
29501 8750 Acked Read 3 Sent Done Done false
29502 8751 Acked Write 6 Sent Done Done false
29503 8752 Acked Read 6 Sent Done Done false
29504 8753 Acked Read 4 Sent Done Done false
29505 8754 Acked Read 7 Sent Done Done false
29506 8755 Acked Read 5 Sent Done Done false
29507 8756 Acked Flush 0 Sent Done Done false
29508 8757 Acked Write 7 Sent Done Done false
29509 8758 Acked Flush 0 Sent Done Done false
29510 8759 Acked Read 7 Sent Done Done false
29511 8760 Acked Write 6 Sent Done Done false
29512 8761 Acked Write 2 Sent Done Done false
29513 8762 Acked Flush 0 Sent Done Done false
29514 8763 Acked Read 8 Sent Done Done false
29515 8764 Acked Read 8 Sent Done Done false
29516 8765 Acked Read 8 Sent Done Done false
29517 8766 Acked Write 1 Sent Done Done false
29518 8767 Acked Read 10 Sent Done Done false
29519 8768 Acked Read 5 Sent Done Done false
29520 8769 Acked Read 2 Sent Done Done false
29521 8770 Acked Write 9 Sent Done Done false
29522 8771 Acked Write 2 Sent Done Done false
29523 8772 Acked Read 3 Sent Done Done false
29524 8773 Acked Read 6 Sent Done Done false
29525 8774 Acked Read 6 Sent Done Done false
29526 8775 Acked Flush 0 Sent Done Done false
29527 8776 Acked Write 9 Sent Done Done false
29528 8777 Acked Read 3 Sent Done Done false
29529 8778 Acked Read 5 Sent Done Done false
29530 8779 Acked Write 8 Sent Done Done false
29531 8780 Acked Read 10 Sent Done Done false
29532 8781 Acked Flush 0 Sent Done Done false
29533 8782 Acked Write 4 Sent Done Done false
29534 8783 Acked Read 9 Sent Done Done false
29535 8784 Acked Write 5 Sent Done Done false
29536 8785 Acked Write 2 Sent Done Done false
29537 8786 Acked Write 10 Sent Done Done false
29538 8787 Acked Write 6 Sent Done Done false
29539 8788 Acked Read 8 Sent Done Done false
29540 8789 Acked Flush 0 Sent Done Done false
29541 8790 Acked Write 5 Sent Done Done false
29542 8791 Acked Write 9 Sent Done Done false
29543 8792 Acked Write 1 Sent Done Done false
29544 8793 Acked Read 6 Sent Done Done false
29545 8794 Acked Write 9 Sent Done Done false
29546 8795 Acked Write 5 Sent Done Done false
29547 8796 Acked Read 7 Sent Done Done false
29548 8797 Acked Read 3 Sent Done Done false
29549 8798 Acked Flush 0 Sent Done Done false
29550 8799 Acked Write 10 Sent Done Done false
29551 8800 Acked Flush 0 Sent Done Done false
29552 8801 Acked Read 2 Sent Done Done false
29553 8802 Acked Write 10 Sent Done Done false
29554 8803 Acked Read 2 Sent Done Done false
29555 8804 Acked Read 7 Sent Done Done false
29556 8805 Acked Read 2 Sent Done Done false
29557 8806 Acked Read 1 Sent Done Done false
29558 8807 Acked Read 7 Sent Done Done false
29559 8808 Acked Read 8 Sent Done Done false
29560 8809 Acked Write 7 Sent Done Done false
29561 8810 Acked Read 5 Sent Done Done false
29562 8811 Acked Read 3 Sent Done Done false
29563 8812 Acked Read 9 Sent Done Done false
29564 8813 Acked Write 9 Sent Done Done false
29565 8814 Acked Read 9 Sent Done Done false
29566 8815 Acked Read 1 Sent Done Done false
29567 8816 Acked Write 10 Sent Done Done false
29568 8817 Acked Write 4 Sent Done Done false
29569 8818 Acked Read 4 Sent Done Done false
29570 8819 Acked Read 3 Sent Done Done false
29571 8820 Acked Write 5 Sent Done Done false
29572 8821 Acked Flush 0 Sent Done Done false
29573 8822 Acked Read 4 Sent Done Done false
29574 8823 Acked Write 9 Sent Done Done false
29575 8824 Acked Read 6 Sent Done Done false
29576 8825 Acked Read 3 Sent Done Done false
29577 8826 Acked Read 5 Sent Done Done false
29578 8827 Acked Read 9 Sent Done Done false
29579 8828 Acked Read 2 Sent Done Done false
29580 8829 Acked Write 3 Sent Done Done false
29581 8830 Acked Read 3 Sent Done Done false
29582 8831 Acked Read 5 Sent Done Done false
29583 8832 Acked Write 2 Sent Done Done false
29584 8833 Acked Read 7 Sent Done Done false
29585 8834 Acked Read 8 Sent Done Done false
29586 8835 Acked Read 7 Sent Done Done false
29587 8836 Acked Write 4 Sent Done Done false
29588 8837 Acked Write 6 Sent Done Done false
29589 8838 Acked Read 3 Sent Done Done false
29590 8839 Acked Read 1 Sent Done Done false
29591 8840 Acked Write 1 Sent Done Done false
29592 8841 Acked Flush 0 Sent Done Done false
29593 8842 Acked Read 9 Sent Done Done false
29594 8843 Acked Write 7 Sent Done Done false
29595 8844 Acked Read 2 Sent Done Done false
29596 8845 Acked Write 5 Sent Done Done false
29597 8846 Acked Read 5 Sent Done Done false
29598 8847 Acked Write 8 Sent Done Done false
29599 8848 Acked Read 6 Sent Done Done false
29600 8849 Acked Write 8 Sent Done Done false
29601 8850 Acked Flush 0 Sent Done Done false
29602 8851 Acked Write 4 Sent Done Done false
29603 8852 Acked Read 10 Sent Done Done false
29604 8853 Acked Read 4 Sent Done Done false
29605 8854 Acked Flush 0 Sent Done Done false
29606 8855 Acked Write 4 Sent Done Done false
29607 STATES DS:0 DS:1 DS:2 TOTAL
29608 Sent 301 0 0 301
29609 Done 0 301 301 602
29610 Skipped 0 0 0 0
29611 Error 0 0 0 0
29612 Last Flush: 8554 8854 8854
29613 Downstairs last five completed: 8554 8553 8552 8551 8550
29614 Upstairs last five completed: 8855 8854 8853 8852 8851
29615 CLIENT: Up:0 ds:301 act:2
29616 [8810] got start action
29617 [8810] I am going to restart
29618 Starting downstairs at port 8810
29619 Make output file at "/tmp/dsc/downstairs-8810.txt"
29620 [8810][0] reports Starting
29621 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1224)
29622 [8810][0] reports Running
29623 ----------------------------------------------------------------
29624 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
29625 States: Offline Active Active
29626 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29627 8555 Acked Write 2 Sent Done Done false
29628 8556 Acked Read 1 Sent Done Done false
29629 8557 Acked Read 4 Sent Done Done false
29630 8558 Acked Write 5 Sent Done Done false
29631 8559 Acked Read 3 Sent Done Done false
29632 8560 Acked Write 4 Sent Done Done false
29633 8561 Acked Read 7 Sent Done Done false
29634 8562 Acked Read 9 Sent Done Done false
29635 8563 Acked Write 10 Sent Done Done false
29636 8564 Acked Write 4 Sent Done Done false
29637 8565 Acked Read 5 Sent Done Done false
29638 8566 Acked Read 5 Sent Done Done false
29639 8567 Acked Flush 0 Sent Done Done false
29640 8568 Acked Write 10 Sent Done Done false
29641 8569 Acked Read 6 Sent Done Done false
29642 8570 Acked Write 5 Sent Done Done false
29643 8571 Acked Read 7 Sent Done Done false
29644 8572 Acked Read 9 Sent Done Done false
29645 8573 Acked Read 6 Sent Done Done false
29646 8574 Acked Read 2 Sent Done Done false
29647 8575 Acked Flush 0 Sent Done Done false
29648 8576 Acked Read 1 Sent Done Done false
29649 8577 Acked Write 5 Sent Done Done false
29650 8578 Acked Read 2 Sent Done Done false
29651 8579 Acked Read 2 Sent Done Done false
29652 8580 Acked Write 3 Sent Done Done false
29653 8581 Acked Write 5 Sent Done Done false
29654 8582 Acked Read 5 Sent Done Done false
29655 8583 Acked Write 4 Sent Done Done false
29656 8584 Acked Flush 0 Sent Done Done false
29657 8585 Acked Write 9 Sent Done Done false
29658 8586 Acked Write 1 Sent Done Done false
29659 8587 Acked Write 1 Sent Done Done false
29660 8588 Acked Write 2 Sent Done Done false
29661 8589 Acked Read 8 Sent Done Done false
29662 8590 Acked Write 1 Sent Done Done false
29663 8591 Acked Write 10 Sent Done Done false
29664 8592 Acked Flush 0 Sent Done Done false
29665 8593 Acked Flush 0 Sent Done Done false
29666 8594 Acked Write 8 Sent Done Done false
29667 8595 Acked Write 2 Sent Done Done false
29668 8596 Acked Write 5 Sent Done Done false
29669 8597 Acked Write 9 Sent Done Done false
29670 8598 Acked Write 7 Sent Done Done false
29671 8599 Acked Read 10 Sent Done Done false
29672 8600 Acked Read 2 Sent Done Done false
29673 8601 Acked Write 4 Sent Done Done false
29674 8602 Acked Read 2 Sent Done Done false
29675 8603 Acked Read 1 Sent Done Done false
29676 8604 Acked Write 1 Sent Done Done false
29677 8605 Acked Write 2 Sent Done Done false
29678 8606 Acked Flush 0 Sent Done Done false
29679 8607 Acked Flush 0 Sent Done Done false
29680 8608 Acked Read 8 Sent Done Done false
29681 8609 Acked Read 7 Sent Done Done false
29682 8610 Acked Read 3 Sent Done Done false
29683 8611 Acked Write 5 Sent Done Done false
29684 8612 Acked Write 3 Sent Done Done false
29685 8613 Acked Write 1 Sent Done Done false
29686 8614 Acked Flush 0 Sent Done Done false
29687 8615 Acked Read 10 Sent Done Done false
29688 8616 Acked Read 10 Sent Done Done false
29689 8617 Acked Write 9 Sent Done Done false
29690 8618 Acked Write 8 Sent Done Done false
29691 8619 Acked Read 5 Sent Done Done false
29692 8620 Acked Read 1 Sent Done Done false
29693 8621 Acked Write 5 Sent Done Done false
29694 8622 Acked Read 1 Sent Done Done false
29695 8623 Acked Flush 0 Sent Done Done false
29696 8624 Acked Read 9 Sent Done Done false
29697 8625 Acked Read 10 Sent Done Done false
29698 8626 Acked Write 2 Sent Done Done false
29699 8627 Acked Write 10 Sent Done Done false
29700 8628 Acked Write 9 Sent Done Done false
29701 8629 Acked Read 3 Sent Done Done false
29702 8630 Acked Read 5 Sent Done Done false
29703 8631 Acked Read 8 Sent Done Done false
29704 8632 Acked Read 8 Sent Done Done false
29705 8633 Acked Flush 0 Sent Done Done false
29706 8634 Acked Read 6 Sent Done Done false
29707 8635 Acked Write 9 Sent Done Done false
29708 8636 Acked Read 2 Sent Done Done false
29709 8637 Acked Flush 0 Sent Done Done false
29710 8638 Acked Read 6 Sent Done Done false
29711 8639 Acked Read 2 Sent Done Done false
29712 8640 Acked Write 9 Sent Done Done false
29713 8641 Acked Write 5 Sent Done Done false
29714 8642 Acked Flush 0 Sent Done Done false
29715 8643 Acked Read 7 Sent Done Done false
29716 8644 Acked Read 10 Sent Done Done false
29717 8645 Acked Write 3 Sent Done Done false
29718 8646 Acked Read 10 Sent Done Done false
29719 8647 Acked Write 10 Sent Done Done false
29720 8648 Acked Write 9 Sent Done Done false
29721 8649 Acked Write 8 Sent Done Done false
29722 8650 Acked Flush 0 Sent Done Done false
29723 8651 Acked Write 7 Sent Done Done false
29724 8652 Acked Read 3 Sent Done Done false
29725 8653 Acked Read 8 Sent Done Done false
29726 8654 Acked Write 1 Sent Done Done false
29727 8655 Acked Read 8 Sent Done Done false
29728 8656 Acked Write 7 Sent Done Done false
29729 8657 Acked Read 6 Sent Done Done false
29730 8658 Acked Write 1 Sent Done Done false
29731 8659 Acked Read 8 Sent Done Done false
29732 8660 Acked Write 2 Sent Done Done false
29733 8661 Acked Read 9 Sent Done Done false
29734 8662 Acked Write 9 Sent Done Done false
29735 8663 Acked Read 5 Sent Done Done false
29736 8664 Acked Read 9 Sent Done Done false
29737 8665 Acked Read 6 Sent Done Done false
29738 8666 Acked Write 9 Sent Done Done false
29739 8667 Acked Read 5 Sent Done Done false
29740 8668 Acked Flush 0 Sent Done Done false
29741 8669 Acked Write 10 Sent Done Done false
29742 8670 Acked Read 4 Sent Done Done false
29743 8671 Acked Read 7 Sent Done Done false
29744 8672 Acked Read 3 Sent Done Done false
29745 8673 Acked Write 3 Sent Done Done false
29746 8674 Acked Read 3 Sent Done Done false
29747 8675 Acked Read 8 Sent Done Done false
29748 8676 Acked Write 8 Sent Done Done false
29749 8677 Acked Read 1 Sent Done Done false
29750 8678 Acked Read 7 Sent Done Done false
29751 8679 Acked Read 5 Sent Done Done false
29752 8680 Acked Read 10 Sent Done Done false
29753 8681 Acked Read 9 Sent Done Done false
29754 8682 Acked Read 10 Sent Done Done false
29755 8683 Acked Flush 0 Sent Done Done false
29756 8684 Acked Write 1 Sent Done Done false
29757 8685 Acked Read 1 Sent Done Done false
29758 8686 Acked Write 6 Sent Done Done false
29759 8687 Acked Read 2 Sent Done Done false
29760 8688 Acked Read 3 Sent Done Done false
29761 8689 Acked Read 6 Sent Done Done false
29762 8690 Acked Write 7 Sent Done Done false
29763 8691 Acked Read 6 Sent Done Done false
29764 8692 Acked Write 1 Sent Done Done false
29765 8693 Acked Write 7 Sent Done Done false
29766 8694 Acked Write 3 Sent Done Done false
29767 8695 Acked Read 1 Sent Done Done false
29768 8696 Acked Write 6 Sent Done Done false
29769 8697 Acked Read 5 Sent Done Done false
29770 8698 Acked Read 1 Sent Done Done false
29771 8699 Acked Write 1 Sent Done Done false
29772 8700 Acked Read 4 Sent Done Done false
29773 8701 Acked Flush 0 Sent Done Done false
29774 8702 Acked Read 5 Sent Done Done false
29775 8703 Acked Write 3 Sent Done Done false
29776 8704 Acked Read 5 Sent Done Done false
29777 8705 Acked Flush 0 Sent Done Done false
29778 8706 Acked Read 3 Sent Done Done false
29779 8707 Acked Read 2 Sent Done Done false
29780 8708 Acked Read 10 Sent Done Done false
29781 8709 Acked Write 4 Sent Done Done false
29782 8710 Acked Flush 0 Sent Done Done false
29783 8711 Acked Write 6 Sent Done Done false
29784 8712 Acked Read 3 Sent Done Done false
29785 8713 Acked Write 9 Sent Done Done false
29786 8714 Acked Read 2 Sent Done Done false
29787 8715 Acked Read 4 Sent Done Done false
29788 8716 Acked Read 5 Sent Done Done false
29789 8717 Acked Write 4 Sent Done Done false
29790 8718 Acked Read 8 Sent Done Done false
29791 8719 Acked Write 3 Sent Done Done false
29792 8720 Acked Read 4 Sent Done Done false
29793 8721 Acked Read 9 Sent Done Done false
29794 8722 Acked Read 5 Sent Done Done false
29795 8723 Acked Read 1 Sent Done Done false
29796 8724 Acked Read 10 Sent Done Done false
29797 8725 Acked Write 2 Sent Done Done false
29798 8726 Acked Read 9 Sent Done Done false
29799 8727 Acked Write 10 Sent Done Done false
29800 8728 Acked Read 1 Sent Done Done false
29801 8729 Acked Write 3 Sent Done Done false
29802 8730 Acked Read 3 Sent Done Done false
29803 8731 Acked Write 6 Sent Done Done false
29804 8732 Acked Read 2 Sent Done Done false
29805 8733 Acked Write 1 Sent Done Done false
29806 8734 Acked Write 2 Sent Done Done false
29807 8735 Acked Read 8 Sent Done Done false
29808 8736 Acked Read 7 Sent Done Done false
29809 8737 Acked Read 6 Sent Done Done false
29810 8738 Acked Write 4 Sent Done Done false
29811 8739 Acked Read 1 Sent Done Done false
29812 8740 Acked Read 8 Sent Done Done false
29813 8741 Acked Read 6 Sent Done Done false
29814 8742 Acked Write 9 Sent Done Done false
29815 8743 Acked Read 3 Sent Done Done false
29816 8744 Acked Write 1 Sent Done Done false
29817 8745 Acked Write 6 Sent Done Done false
29818 8746 Acked Write 3 Sent Done Done false
29819 8747 Acked Write 2 Sent Done Done false
29820 8748 Acked Write 10 Sent Done Done false
29821 8749 Acked Write 8 Sent Done Done false
29822 8750 Acked Read 3 Sent Done Done false
29823 8751 Acked Write 6 Sent Done Done false
29824 8752 Acked Read 6 Sent Done Done false
29825 8753 Acked Read 4 Sent Done Done false
29826 8754 Acked Read 7 Sent Done Done false
29827 8755 Acked Read 5 Sent Done Done false
29828 8756 Acked Flush 0 Sent Done Done false
29829 8757 Acked Write 7 Sent Done Done false
29830 8758 Acked Flush 0 Sent Done Done false
29831 8759 Acked Read 7 Sent Done Done false
29832 8760 Acked Write 6 Sent Done Done false
29833 8761 Acked Write 2 Sent Done Done false
29834 8762 Acked Flush 0 Sent Done Done false
29835 8763 Acked Read 8 Sent Done Done false
29836 8764 Acked Read 8 Sent Done Done false
29837 8765 Acked Read 8 Sent Done Done false
29838 8766 Acked Write 1 Sent Done Done false
29839 8767 Acked Read 10 Sent Done Done false
29840 8768 Acked Read 5 Sent Done Done false
29841 8769 Acked Read 2 Sent Done Done false
29842 8770 Acked Write 9 Sent Done Done false
29843 8771 Acked Write 2 Sent Done Done false
29844 8772 Acked Read 3 Sent Done Done false
29845 8773 Acked Read 6 Sent Done Done false
29846 8774 Acked Read 6 Sent Done Done false
29847 8775 Acked Flush 0 Sent Done Done false
29848 8776 Acked Write 9 Sent Done Done false
29849 8777 Acked Read 3 Sent Done Done false
29850 8778 Acked Read 5 Sent Done Done false
29851 8779 Acked Write 8 Sent Done Done false
29852 8780 Acked Read 10 Sent Done Done false
29853 8781 Acked Flush 0 Sent Done Done false
29854 8782 Acked Write 4 Sent Done Done false
29855 8783 Acked Read 9 Sent Done Done false
29856 8784 Acked Write 5 Sent Done Done false
29857 8785 Acked Write 2 Sent Done Done false
29858 8786 Acked Write 10 Sent Done Done false
29859 8787 Acked Write 6 Sent Done Done false
29860 8788 Acked Read 8 Sent Done Done false
29861 8789 Acked Flush 0 Sent Done Done false
29862 8790 Acked Write 5 Sent Done Done false
29863 8791 Acked Write 9 Sent Done Done false
29864 8792 Acked Write 1 Sent Done Done false
29865 8793 Acked Read 6 Sent Done Done false
29866 8794 Acked Write 9 Sent Done Done false
29867 8795 Acked Write 5 Sent Done Done false
29868 8796 Acked Read 7 Sent Done Done false
29869 8797 Acked Read 3 Sent Done Done false
29870 8798 Acked Flush 0 Sent Done Done false
29871 8799 Acked Write 10 Sent Done Done false
29872 8800 Acked Flush 0 Sent Done Done false
29873 8801 Acked Read 2 Sent Done Done false
29874 8802 Acked Write 10 Sent Done Done false
29875 8803 Acked Read 2 Sent Done Done false
29876 8804 Acked Read 7 Sent Done Done false
29877 8805 Acked Read 2 Sent Done Done false
29878 8806 Acked Read 1 Sent Done Done false
29879 8807 Acked Read 7 Sent Done Done false
29880 8808 Acked Read 8 Sent Done Done false
29881 8809 Acked Write 7 Sent Done Done false
29882 8810 Acked Read 5 Sent Done Done false
29883 8811 Acked Read 3 Sent Done Done false
29884 8812 Acked Read 9 Sent Done Done false
29885 8813 Acked Write 9 Sent Done Done false
29886 8814 Acked Read 9 Sent Done Done false
29887 8815 Acked Read 1 Sent Done Done false
29888 8816 Acked Write 10 Sent Done Done false
29889 8817 Acked Write 4 Sent Done Done false
29890 8818 Acked Read 4 Sent Done Done false
29891 8819 Acked Read 3 Sent Done Done false
29892 8820 Acked Write 5 Sent Done Done false
29893 8821 Acked Flush 0 Sent Done Done false
29894 8822 Acked Read 4 Sent Done Done false
29895 8823 Acked Write 9 Sent Done Done false
29896 8824 Acked Read 6 Sent Done Done false
29897 8825 Acked Read 3 Sent Done Done false
29898 8826 Acked Read 5 Sent Done Done false
29899 8827 Acked Read 9 Sent Done Done false
29900 8828 Acked Read 2 Sent Done Done false
29901 8829 Acked Write 3 Sent Done Done false
29902 8830 Acked Read 3 Sent Done Done false
29903 8831 Acked Read 5 Sent Done Done false
29904 8832 Acked Write 2 Sent Done Done false
29905 8833 Acked Read 7 Sent Done Done false
29906 8834 Acked Read 8 Sent Done Done false
29907 8835 Acked Read 7 Sent Done Done false
29908 8836 Acked Write 4 Sent Done Done false
29909 8837 Acked Write 6 Sent Done Done false
29910 8838 Acked Read 3 Sent Done Done false
29911 8839 Acked Read 1 Sent Done Done false
29912 8840 Acked Write 1 Sent Done Done false
29913 8841 Acked Flush 0 Sent Done Done false
29914 8842 Acked Read 9 Sent Done Done false
29915 8843 Acked Write 7 Sent Done Done false
29916 8844 Acked Read 2 Sent Done Done false
29917 8845 Acked Write 5 Sent Done Done false
29918 8846 Acked Read 5 Sent Done Done false
29919 8847 Acked Write 8 Sent Done Done false
29920 8848 Acked Read 6 Sent Done Done false
29921 8849 Acked Write 8 Sent Done Done false
29922 8850 Acked Flush 0 Sent Done Done false
29923 8851 Acked Write 4 Sent Done Done false
29924 8852 Acked Read 10 Sent Done Done false
29925 8853 Acked Read 4 Sent Done Done false
29926 8854 Acked Flush 0 Sent Done Done false
29927 8855 Acked Write 4 Sent Done Done false
29928 8856 Acked Flush 0 Sent Done Done false
29929 STATES DS:0 DS:1 DS:2 TOTAL
29930 Sent 302 0 0 302
29931 Done 0 302 302 604
29932 Skipped 0 0 0 0
29933 Error 0 0 0 0
29934 Last Flush: 8554 8856 8856
29935 Downstairs last five completed: 8554 8553 8552 8551 8550
29936 Upstairs last five completed: 8856 8855 8854 8853 8852
29937 CLIENT: Up:0 ds:302 act:2
299382024-11-01T16:56:45.768ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
299392024-11-01T16:56:45.768ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42716) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
299402024-11-01T16:56:45.769ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
299412024-11-01T16:56:45.770ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
299422024-11-01T16:56:45.770ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
299432024-11-01T16:56:45.770ZINFOcrucible: send last flush ID to this DS: 8554 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
299442024-11-01T16:56:45.770ZINFOcrucible: Replied this last flush ID: 8554 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
299452024-11-01T16:56:45.770ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
299462024-11-01T16:56:45.770ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
299472024-11-01T16:56:45.770ZINFOcrucible: [0] client re-new 302 jobs since flush 8554 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
299482024-11-01T16:56:45.770ZINFOcrucible: [0] Replayed 302 jobs since flush: 8554 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
29949 ----------------------------------------------------------------
29950 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
29951 Downstairs last five completed: 8856 8855 8854 8853 8852
29952 Upstairs last five completed: 8856 8855 8854 8853 8852
29953 CLIENT: Up:0 ds:0 act:3
29954 Replay: All jobs finished
29955 Main task has work to do, go find it
29956 got dsc Stop(0)
29957 stop 0
29958 [8810] Got stop action so:false kr:false
29959 Nov 01 16:56:48.084 INFO request completed, latency_us: 342, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 1a9e6658-1dee-4317-a631-f61bab345da1, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
299602024-11-01T16:56:48.085ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
299612024-11-01T16:56:48.085ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
29962 Nov 01 16:56:48.085 INFO request completed{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:56:48.085137971Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
29963 , latency_us: 146, response_code: 200, uri: /state/cid/0, method: GET, req_id: {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:56:48.085164537Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"client":"0","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
29964 7393b6d1-160d-4c7f-ba87-b12e1cf2db55, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:56:48.085188413Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
29965 1:9998
29966 [8810] Exited with: ExitStatus(unix_wait_status(9))
29967 [8810][0] reports Exit
29968 Nov 01 16:56:52.087 INFO request completed, latency_us: 394, response_code: 200, uri: /state/cid/0, method: GET, req_id: 58d39418-89b0-4206-836d-3efbe283c0af, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
29969 001/300 Read block 661 len 36864
29970 002/300 Read block 4774 len 32768
29971 003/300 Write block 1667 len 28672 data: 6 6 6 6 5 3 5
29972 004/300 Write block 3659 len 20480 data: 5 6 5 4 4
29973 005/300 Read block 1737 len 24576
29974 006/300 Read block 1906 len 36864
29975 007/300 Flush
29976 008/300 Read block 351 len 8192
29977 009/300 Read block 3100 len 4096
29978 010/300 Write block 2185 len 8192 data: 6 6
29979 011/300 Write block 878 len 8192 data: 4 4
29980 012/300 Read block 2703 len 40960
29981 013/300 Write block 4833 len 4096 data: 5
29982 014/300 Write block 3277 len 4096 data: 6
29983 015/300 Write block 2882 len 36864 data: 4 4 4 6 8 8 8 9 9
29984 016/300 Write block 1581 len 16384 data: 3 3 5 5
29985 017/300 Read block 831 len 24576
29986 018/300 Read block 4596 len 16384
29987 019/300 Write block 1248 len 12288 data: 2 2 2
29988 020/300 Read block 1744 len 36864
29989 021/300 Read block 207 len 16384
29990 022/300 Read block 3133 len 12288
29991 023/300 Flush
29992 024/300 Write block 2514 len 40960 data: 8 8 6 5 5 4 4 3 4 4
29993 025/300 Read block 4173 len 36864
29994 026/300 Flush
29995 027/300 Write block 2273 len 36864 data: 6 8 7 4 4 5 5 6 7
29996 028/300 Read block 4312 len 4096
29997 029/300 Read block 2008 len 32768
29998 030/300 Write block 3833 len 36864 data: 9 7 6 7 6 5 4 4 3
29999 031/300 Flush
30000 032/300 Read block 1481 len 8192
30001 033/300 Read block 1280 len 28672
30002 034/300 Read block 2435 len 36864
30003 035/300 Write block 1795 len 32768 data: 5 5 5 5 5 6 6 7
30004 036/300 Write block 3787 len 8192 data: 5 5
30005 037/300 Read block 4534 len 12288
30006 038/300 Write block 2083 len 36864 data: 5 6 8 10 10 9 9 7 10
30007 039/300 Write block 1273 len 32768 data: 3 3 4 5 4 4 3 5
30008 040/300 Read block 3036 len 4096
30009 041/300 Flush
30010 042/300 Read block 136 len 16384
30011 043/300 Flush
30012 044/300 Write block 3404 len 28672 data: 5 5 5 5 3 3 4
30013 045/300 Read block 2290 len 28672
30014 046/300 Read block 4938 len 8192
30015 047/300 Write block 4243 len 8192 data: 3 3
30016 048/300 Write block 4422 len 28672 data: 4 4 4 4 5 7 8
30017 049/300 Read block 1111 len 40960
30018 050/300 Flush
30019 051/300 Read block 1118 len 12288
30020 052/300 Write block 304 len 12288 data: 6 6 7
30021 053/300 Read block 3649 len 8192
30022 054/300 Write block 3341 len 40960 data: 5 4 3 3 5 5 6 7 8 8
30023 055/300 Read block 2177 len 20480
30024 056/300 Read block 111 len 4096
30025 057/300 Read block 3493 len 20480
30026 058/300 Read block 2880 len 16384
30027 059/300 Read block 1156 len 8192
30028 060/300 Write block 3745 len 40960 data: 4 5 7 8 6 8 8 9 9 9
30029 061/300 Flush
30030 062/300 Write block 2876 len 12288 data: 7 7 7
30031 063/300 Write block 815 len 8192 data: 4 3
30032 064/300 Read block 3770 len 32768
30033 065/300 Read block 1124 len 12288
30034 066/300 Write block 1773 len 28672 data: 8 6 6 5 6 5 3
30035 067/300 Read block 2972 len 12288
30036 068/300 Write block 2042 len 8192 data: 3 3
30037 069/300 Read block 3425 len 16384
30038 070/300 Flush
30039 071/300 Flush
30040 072/300 Read block 3743 len 24576
30041 073/300 Write block 670 len 12288 data: 6 6 6
30042 074/300 Flush
30043 075/300 Read block 2306 len 28672
30044 076/300 Write block 2105 len 4096 data: 4
30045 077/300 Write block 3560 len 12288 data: 6 4 4
30046 078/300 Read block 3470 len 28672
30047 079/300 Read block 4975 len 32768
30048 080/300 Read block 936 len 28672
30049 081/300 Read block 1676 len 20480
30050 082/300 Read block 864 len 8192
30051 083/300 Write block 3157 len 12288 data: 4 4 3
30052 084/300 Read block 1366 len 4096
30053 085/300 Write block 438 len 40960 data: 4 6 6 4 4 3 4 3 5 6
30054 086/300 Write block 4323 len 20480 data: 8 6 5 5 4
30055 087/300 Read block 4293 len 8192
30056 088/300 Write block 4363 len 24576 data: 5 4 4 4 4 4
30057 089/300 Read block 3533 len 20480
30058 090/300 Write block 4632 len 40960 data: 9 7 6 6 7 6 6 5 5 5
30059 091/300 Write block 1864 len 40960 data: 5 6 5 4 5 6 9 8 11 11
30060 092/300 Read block 1604 len 36864
30061 093/300 Write block 2655 len 12288 data: 5 6 6
30062 094/300 Read block 3601 len 24576
30063 095/300 Write block 3849 len 20480 data: 6 6 5 4 5
30064 096/300 Write block 1673 len 40960 data: 6 5 5 4 4 5 4 4 3 4
30065 097/300 Read block 613 len 8192
30066 098/300 Read block 2992 len 12288
30067 099/300 Flush
30068 100/300 Write block 1112 len 12288 data: 3 4 3
30069 101/300 Read block 4804 len 24576
30070 102/300 Write block 2562 len 24576 data: 7 7 9 9 7 8
30071 103/300 Write block 926 len 8192 data: 9 9
30072 104/300 Write block 2981 len 40960 data: 6 6 7 6 6 5 7 7 6 6
30073 105/300 Write block 4136 len 4096 data: 5
30074 106/300 Read block 633 len 4096
30075 107/300 Flush
30076 108/300 Flush
30077 109/300 Write block 2791 len 20480 data: 8 10 7 6 5
30078 110/300 Flush
30079 111/300 Read block 1203 len 4096
30080 112/300 Read block 1891 len 8192
30081 113/300 Read block 84 len 24576
30082 114/300 Read block 2594 len 28672
30083 115/300 Read block 2760 len 24576
30084 116/300 Write block 1262 len 16384 data: 5 4 6 8
30085 117/300 Read block 2003 len 32768
30086 118/300 Read block 3665 len 20480
30087 119/300 Flush
30088 120/300 Read block 889 len 8192
30089 121/300 Write block 376 len 4096 data: 5
30090 122/300 Read block 3540 len 40960
30091 123/300 Read block 167 len 4096
30092 124/300 Write block 708 len 28672 data: 4 3 3 3 3 3 3
30093 125/300 Read block 2740 len 32768
30094 126/300 Write block 1417 len 8192 data: 4 3
30095 127/300 Write block 1284 len 8192 data: 5 4
30096 128/300 Read block 302 len 32768
30097 129/300 Read block 734 len 32768
30098 130/300 Read block 1508 len 8192
30099 131/300 Read block 1079 len 16384
30100 132/300 Read block 2485 len 8192
30101 133/300 Read block 2121 len 28672
30102 134/300 Read block 4442 len 28672
30103 135/300 Write block 4738 len 20480 data: 6 7 7 7 7
30104 136/300 Read block 2912 len 24576
30105 137/300 Read block 2898 len 12288
30106 138/300 Read block 453 len 24576
30107 139/300 Flush
30108 140/300 Read block 4530 len 28672
30109 141/300 Read block 2995 len 8192
30110 142/300 Write block 2764 len 20480 data: 7 9 8 4 3
30111 143/300 Write block 3025 len 28672 data: 4 5 4 4 5 5 6
30112 144/300 Write block 3751 len 8192 data: 9 10
30113 145/300 Flush
30114 146/300 Read block 4370 len 4096
30115 147/300 Flush
30116 148/300 Write block 366 len 40960 data: 8 7 5 7 8 6 6 5 5 5
30117 149/300 Read block 3929 len 40960
30118 150/300 Write block 1802 len 36864 data: 8 8 6 6 6 8 8 7 8
30119 151/300 Read block 2815 len 20480
30120 152/300 Write block 2027 len 16384 data: 3 4 4 6
30121 153/300 Read block 1167 len 20480
30122 154/300 Write block 1840 len 24576 data: 5 5 6 7 6 6
30123 155/300 Write block 1485 len 32768 data: 3 3 3 3 4 4 5 5
30124 156/300 Read block 1091 len 32768
30125 157/300 Write block 1340 len 28672 data: 4 3 3 3 3 5 5
30126 158/300 Write block 303 len 32768 data: 6 7 7 8 6 5 4 5
30127 159/300 Write block 4016 len 16384 data: 4 4 5 4
30128 160/300 Write block 4860 len 36864 data: 5 4 4 3 3 4 7 6 6
30129 161/300 Read block 4837 len 16384
30130 162/300 Read block 997 len 20480
30131 163/300 Write block 3824 len 36864 data: 5 7 8 8 9 8 9 9 8
30132 164/300 Write block 3944 len 36864 data: 7 6 9 9 8 10 9 8 6
30133 165/300 Write block 2406 len 20480 data: 5 5 5 5 4
30134 166/300 Read block 1869 len 32768
30135 167/300 Read block 2923 len 28672
30136 168/300 Flush
30137 169/300 Read block 4594 len 36864
30138 170/300 Read block 1500 len 12288
30139 171/300 Write block 3918 len 32768 data: 6 7 7 5 4 4 4 5
30140 172/300 Flush
30141 173/300 Read block 1980 len 24576
30142 174/300 Read block 3255 len 24576
30143 175/300 Read block 2921 len 24576
30144 176/300 Read block 3480 len 24576
30145 177/300 Write block 3914 len 40960 data: 7 6 8 7 7 8 8 6 5 5
30146 178/300 Read block 1485 len 32768
30147 179/300 Read block 4042 len 20480
30148 180/300 Write block 436 len 8192 data: 7 7
30149 181/300 Read block 4124 len 40960
30150 182/300 Read block 4323 len 28672
30151 183/300 Flush
30152 184/300 Write block 3590 len 8192 data: 10 9
30153 185/300 Read block 1600 len 36864
30154 186/300 Write block 132 len 12288 data: 4 3 2
30155 187/300 Flush
30156 188/300 Read block 3211 len 28672
30157 189/300 Write block 4667 len 36864 data: 7 5 4 5 5 6 7 7 7
30158 190/300 Write block 2642 len 16384 data: 8 8 8 7
30159 191/300 Write block 1887 len 4096 data: 5
30160 192/300 Write block 1469 len 24576 data: 6 7 7 6 6 8
30161 193/300 Write block 3888 len 12288 data: 4 4 3
30162 194/300 Write block 645 len 20480 data: 7 6 6 2 3
30163 195/300 Flush
30164 196/300 Read block 3625 len 16384
30165 197/300 Read block 118 len 36864
30166 198/300 Write block 824 len 8192 data: 5 6
30167 199/300 Read block 3394 len 36864
30168 200/300 Read block 1835 len 8192
30169 201/300 Write block 3163 len 20480 data: 6 5 6 4 4
30170 202/300 Read block 2225 len 20480
30171 203/300 Read block 2638 len 36864
30172 204/300 Flush
30173 205/300 Flush
30174 206/300 Write block 2045 len 20480 data: 3 5 5 6 6
30175 207/300 Write block 1084 len 40960 data: 5 5 5 7 9 10 9 11 10 9
30176 208/300 Write block 3849 len 28672 data: 7 7 6 5 6 6 6
30177 209/300 Read block 1502 len 8192
30178 210/300 Write block 371 len 12288 data: 7 7 6
30179 211/300 Read block 389 len 40960
30180 212/300 Write block 1712 len 40960 data: 5 3 3 5 5 6 6 4 6 6
30181 213/300 Write block 1661 len 16384 data: 5 8 8 7
30182 214/300 Read block 3866 len 40960
30183 215/300 Read block 2317 len 20480
30184 216/300 Read block 3367 len 16384
30185 217/300 Read block 2704 len 24576
30186 218/300 Read block 2931 len 36864
30187 219/300 Write block 1484 len 40960 data: 3 4 4 4 4 5 5 6 6 4
30188 220/300 Flush
30189 221/300 Read block 578 len 40960
30190 222/300 Read block 1165 len 12288
30191 223/300 Read block 3488 len 16384
30192 224/300 Read block 1900 len 4096
30193 225/300 Read block 3537 len 32768
30194 226/300 Write block 3063 len 16384 data: 5 4 4 4
30195 227/300 Flush
30196 228/300 Write block 1506 len 8192 data: 6 7
30197 229/300 Write block 3679 len 20480 data: 4 4 3 3 3
30198 230/300 Read block 4862 len 24576
30199 231/300 Flush
30200 232/300 Read block 2294 len 16384
30201 233/300 Read block 2154 len 4096
30202 234/300 Read block 1322 len 16384
30203 235/300 Read block 2636 len 40960
30204 236/300 Write block 498 len 12288 data: 4 4 3
30205 237/300 Read block 1862 len 8192
30206 238/300 Write block 2557 len 16384 data: 6 6 5 6
30207 239/300 Flush
30208 240/300 Write block 4110 len 32768 data: 9 9 8 7 7 7 5 4
30209 241/300 Read block 4269 len 20480
30210 242/300 Read block 2332 len 16384
30211 243/300 Read block 3333 len 20480
30212 244/300 Read block 4785 len 20480
30213 245/300 Read block 2200 len 24576
30214 246/300 Write block 1140 len 8192 data: 9 5
30215 247/300 Flush
30216 248/300 Read block 4659 len 8192
30217 249/300 Read block 1392 len 4096
30218 250/300 Write block 2157 len 12288 data: 2 3 3
30219 251/300 Write block 662 len 20480 data: 7 6 6 6 4
30220 252/300 Read block 1186 len 32768
30221 253/300 Read block 1839 len 16384
30222 254/300 Write block 2830 len 32768 data: 4 5 6 6 6 7 6 6
30223 255/300 Read block 1907 len 32768
30224 256/300 Write block 3771 len 4096 data: 3
30225 257/300 Write block 1257 len 24576 data: 3 3 3 6 5 6
30226 258/300 Flush
30227 259/300 Read block 3293 len 28672
30228 260/300 Read block 3346 len 16384
30229 261/300 Read block 4509 len 36864
30230 262/300 Read block 3094 len 20480
30231 263/300 Flush
30232 264/300 Write block 3996 len 12288 data: 6 5 3
30233 265/300 Write block 3869 len 12288 data: 10 10 7
30234 266/300 Read block 423 len 36864
30235 267/300 Read block 4595 len 40960
30236 268/300 Flush
30237 269/300 Read block 4865 len 32768
30238 270/300 Read block 4143 len 36864
30239 271/300 Read block 2211 len 8192
30240 272/300 Read block 4142 len 32768
30241 273/300 Read block 4096 len 20480
30242 274/300 Write block 144 len 28672 data: 4 4 4 5 4 5 5
30243 275/300 Write block 1465 len 20480 data: 4 4 5 4 7
30244 276/300 Read block 4902 len 40960
30245 277/300 Write block 40 len 16384 data: 6 6 6 7
30246 278/300 Read block 2875 len 40960
30247 279/300 Read block 4941 len 4096
30248 280/300 Write block 1498 len 20480 data: 7 8 6 6 6
30249 281/300 Read block 3521 len 40960
30250 282/300 Read block 4751 len 40960
30251 283/300 Flush
30252 284/300 Write block 835 len 24576 data: 8 6 5 6 4 5
30253 285/300 Flush
30254 286/300 Flush
30255 287/300 Read block 2961 len 40960
30256 288/300 Read block 1612 len 28672
30257 289/300 Flush
30258 290/300 Read block 81 len 40960
30259 291/300 Read block 3592 len 12288
30260 292/300 Read block 4912 len 12288
30261 293/300 Read block 4790 len 28672
30262 294/300 Write block 4919 len 24576 data: 8 7 8 7 8 7
30263 295/300 Flush
30264 296/300 Read block 41 len 36864
30265 297/300 Write block 2025 len 20480 data: 4 4 4 5 5
30266 298/300 Write block 1859 len 4096 data: 5
30267 299/300 Read block 2806 len 24576
30268 300/300 Write block 4278 len 12288 data: 6 4 6
30269 301/300 Read block 418 len 40960
30270 Main task has work to do, go find it
30271 got dsc Start(0)
30272 start 0
30273 Nov 01 16:56:52.390 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/0, method: POST, req_id: e69701d1-91c4-4cf5-aa41-0483183f07b9, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
30274 Replay: started 0, returned:Ok(())
30275 ----------------------------------------------------------------
30276 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30277 States: Offline Active Active
30278 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30279 8857 Acked Read 9 Sent Done Done false
30280 8858 Acked Read 8 Sent Done Done false
30281 8859 Acked Write 7 Sent Done Done false
30282 8860 Acked Write 5 Sent Done Done false
30283 8861 Acked Read 6 Sent Done Done false
30284 8862 Acked Read 9 Sent Done Done false
30285 8863 Acked Flush 0 Sent Done Done false
30286 8864 Acked Read 2 Sent Done Done false
30287 8865 Acked Read 1 Sent Done Done false
30288 8866 Acked Write 2 Sent Done Done false
30289 8867 Acked Write 2 Sent Done Done false
30290 8868 Acked Read 10 Sent Done Done false
30291 8869 Acked Write 1 Sent Done Done false
30292 8870 Acked Write 1 Sent Done Done false
30293 8871 Acked Write 9 Sent Done Done false
30294 8872 Acked Write 4 Sent Done Done false
30295 8873 Acked Read 6 Sent Done Done false
30296 8874 Acked Read 4 Sent Done Done false
30297 8875 Acked Write 3 Sent Done Done false
30298 8876 Acked Read 9 Sent Done Done false
30299 8877 Acked Read 4 Sent Done Done false
30300 8878 Acked Read 3 Sent Done Done false
30301 8879 Acked Flush 0 Sent Done Done false
30302 8880 Acked Write 10 Sent Done Done false
30303 8881 Acked Read 9 Sent Done Done false
30304 8882 Acked Flush 0 Sent Done Done false
30305 8883 Acked Write 9 Sent Done Done false
30306 8884 Acked Read 1 Sent Done Done false
30307 8885 Acked Read 8 Sent Done Done false
30308 8886 Acked Write 9 Sent Done Done false
30309 8887 Acked Flush 0 Sent Done Done false
30310 8888 Acked Read 2 Sent Done Done false
30311 8889 Acked Read 7 Sent Done Done false
30312 8890 Acked Read 9 Sent Done Done false
30313 8891 Acked Write 8 Sent Done Done false
30314 8892 Acked Write 2 Sent Done Done false
30315 8893 Acked Read 3 Sent Done Done false
30316 8894 Acked Write 9 Sent Done Done false
30317 8895 Acked Write 8 Sent Done Done false
30318 8896 Acked Read 1 Sent Done Done false
30319 8897 Acked Flush 0 Sent Done Done false
30320 8898 Acked Read 4 Sent Done Done false
30321 8899 Acked Flush 0 Sent Done Done false
30322 8900 Acked Write 7 Sent Done Done false
30323 8901 Acked Read 7 Sent Done Done false
30324 8902 Acked Read 2 Sent Done Done false
30325 8903 Acked Write 2 Sent Done Done false
30326 8904 Acked Write 7 Sent Done Done false
30327 8905 Acked Read 10 Sent Done Done false
30328 8906 Acked Flush 0 Sent Done Done false
30329 8907 Acked Read 3 Sent Done Done false
30330 8908 Acked Write 3 Sent Done Done false
30331 8909 Acked Read 2 Sent Done Done false
30332 8910 Acked Write 10 Sent Done Done false
30333 8911 Acked Read 5 Sent Done Done false
30334 8912 Acked Read 1 Sent Done Done false
30335 8913 Acked Read 5 Sent Done Done false
30336 8914 Acked Read 4 Sent Done Done false
30337 8915 Acked Read 2 Sent Done Done false
30338 8916 Acked Write 10 Sent Done Done false
30339 8917 Acked Flush 0 Sent Done Done false
30340 8918 Acked Write 3 Sent Done Done false
30341 8919 Acked Write 2 Sent Done Done false
30342 8920 Acked Read 8 Sent Done Done false
30343 8921 Acked Read 3 Sent Done Done false
30344 8922 Acked Write 7 Sent Done Done false
30345 8923 Acked Read 3 Sent Done Done false
30346 8924 Acked Write 2 Sent Done Done false
30347 8925 Acked Read 4 Sent Done Done false
30348 8926 Acked Flush 0 Sent Done Done false
30349 8927 Acked Flush 0 Sent Done Done false
30350 8928 Acked Read 6 Sent Done Done false
30351 8929 Acked Write 3 Sent Done Done false
30352 8930 Acked Flush 0 Sent Done Done false
30353 8931 Acked Read 7 Sent Done Done false
30354 8932 Acked Write 1 Sent Done Done false
30355 8933 Acked Write 3 Sent Done Done false
30356 8934 Acked Read 7 Sent Done Done false
30357 8935 Acked Read 8 Sent Done Done false
30358 8936 Acked Read 7 Sent Done Done false
30359 8937 Acked Read 5 Sent Done Done false
30360 8938 Acked Read 2 Sent Done Done false
30361 8939 Acked Write 3 Sent Done Done false
30362 8940 Acked Read 1 Sent Done Done false
30363 8941 Acked Write 10 Sent Done Done false
30364 8942 Acked Write 5 Sent Done Done false
30365 8943 Acked Read 2 Sent Done Done false
30366 8944 Acked Write 6 Sent Done Done false
30367 8945 Acked Read 5 Sent Done Done false
30368 8946 Acked Write 10 Sent Done Done false
30369 8947 Acked Write 10 Sent Done Done false
30370 8948 Acked Read 9 Sent Done Done false
30371 8949 Acked Write 3 Sent Done Done false
30372 8950 Acked Read 6 Sent Done Done false
30373 8951 Acked Write 5 Sent Done Done false
30374 8952 Acked Write 10 Sent Done Done false
30375 8953 Acked Read 2 Sent Done Done false
30376 8954 Acked Read 3 Sent Done Done false
30377 8955 Acked Flush 0 Sent Done Done false
30378 8956 Acked Write 3 Sent Done Done false
30379 8957 Acked Read 6 Sent Done Done false
30380 8958 Acked Write 6 Sent Done Done false
30381 8959 Acked Write 2 Sent Done Done false
30382 8960 Acked Write 10 Sent Done Done false
30383 8961 Acked Write 1 Sent Done Done false
30384 8962 Acked Read 1 Sent Done Done false
30385 8963 Acked Flush 0 Sent Done Done false
30386 8964 Acked Flush 0 Sent Done Done false
30387 8965 Acked Write 5 Sent Done Done false
30388 8966 Acked Flush 0 Sent Done Done false
30389 8967 Acked Read 1 Sent Done Done false
30390 8968 Acked Read 2 Sent Done Done false
30391 8969 Acked Read 6 Sent Done Done false
30392 8970 Acked Read 7 Sent Done Done false
30393 8971 Acked Read 6 Sent Done Done false
30394 8972 Acked Write 4 Sent Done Done false
30395 8973 Acked Read 8 Sent Done Done false
30396 8974 Acked Read 5 Sent Done Done false
30397 8975 Acked Flush 0 Sent Done Done false
30398 8976 Acked Read 2 Sent Done Done false
30399 8977 Acked Write 1 Sent Done Done false
30400 8978 Acked Read 10 Sent Done Done false
30401 8979 Acked Read 1 Sent Done Done false
30402 8980 Acked Write 7 Sent Done Done false
30403 8981 Acked Read 8 Sent Done Done false
30404 8982 Acked Write 2 Sent Done Done false
30405 8983 Acked Write 2 Sent Done Done false
30406 8984 Acked Read 8 Sent Done Done false
30407 8985 Acked Read 8 Sent Done Done false
30408 8986 Acked Read 2 Sent Done Done false
30409 8987 Acked Read 4 Sent Done Done false
30410 8988 Acked Read 2 Sent Done Done false
30411 8989 Acked Read 7 Sent Done Done false
30412 8990 Acked Read 7 Sent Done Done false
30413 8991 Acked Write 5 Sent Done Done false
30414 8992 Acked Read 6 Sent Done Done false
30415 8993 Acked Read 3 Sent Done Done false
30416 8994 Acked Read 6 Sent Done Done false
30417 8995 Acked Flush 0 Sent Done Done false
30418 8996 Acked Read 7 Sent Done Done false
30419 8997 Acked Read 2 Sent Done Done false
30420 8998 Acked Write 5 Sent Done Done false
30421 8999 Acked Write 7 Sent Done Done false
30422 9000 Acked Write 2 Sent Done Done false
30423 9001 Acked Flush 0 Sent Done Done false
30424 9002 Acked Read 1 Sent Done Done false
30425 9003 Acked Flush 0 Sent Done Done false
30426 9004 Acked Write 10 Sent Done Done false
30427 9005 Acked Read 10 Sent Done Done false
30428 9006 Acked Write 9 Sent Done Done false
30429 9007 Acked Read 5 Sent Done Done false
30430 9008 Acked Write 4 Sent Done Done false
30431 9009 Acked Read 5 Sent Done Done false
30432 9010 Acked Write 6 Sent Done Done false
30433 9011 Acked Write 8 Sent Done Done false
30434 9012 Acked Read 8 Sent Done Done false
30435 9013 Acked Write 7 Sent Done Done false
30436 9014 Acked Write 8 Sent Done Done false
30437 9015 Acked Write 4 Sent Done Done false
30438 9016 Acked Write 9 Sent Done Done false
30439 9017 Acked Read 4 Sent Done Done false
30440 9018 Acked Read 5 Sent Done Done false
30441 9019 Acked Write 9 Sent Done Done false
30442 9020 Acked Write 9 Sent Done Done false
30443 9021 Acked Write 5 Sent Done Done false
30444 9022 Acked Read 8 Sent Done Done false
30445 9023 Acked Read 7 Sent Done Done false
30446 9024 Acked Flush 0 Sent Done Done false
30447 9025 Acked Read 9 Sent Done Done false
30448 9026 Acked Read 3 Sent Done Done false
30449 9027 Acked Write 8 Sent Done Done false
30450 9028 Acked Flush 0 Sent Done Done false
30451 9029 Acked Read 6 Sent Done Done false
30452 9030 Acked Read 6 Sent Done Done false
30453 9031 Acked Read 6 Sent Done Done false
30454 9032 Acked Read 6 Sent Done Done false
30455 9033 Acked Write 10 Sent Done Done false
30456 9034 Acked Read 8 Sent Done Done false
30457 9035 Acked Read 5 Sent Done Done false
30458 9036 Acked Write 2 Sent Done Done false
30459 9037 Acked Read 10 Sent Done Done false
30460 9038 Acked Read 7 Sent Done Done false
30461 9039 Acked Flush 0 Sent Done Done false
30462 9040 Acked Write 2 Sent Done Done false
30463 9041 Acked Read 9 Sent Done Done false
30464 9042 Acked Write 3 Sent Done Done false
30465 9043 Acked Flush 0 Sent Done Done false
30466 9044 Acked Read 7 Sent Done Done false
30467 9045 Acked Write 9 Sent Done Done false
30468 9046 Acked Write 4 Sent Done Done false
30469 9047 Acked Write 1 Sent Done Done false
30470 9048 Acked Write 6 Sent Done Done false
30471 9049 Acked Write 3 Sent Done Done false
30472 9050 Acked Write 5 Sent Done Done false
30473 9051 Acked Flush 0 Sent Done Done false
30474 9052 Acked Read 4 Sent Done Done false
30475 9053 Acked Read 9 Sent Done Done false
30476 9054 Acked Write 2 Sent Done Done false
30477 9055 Acked Read 9 Sent Done Done false
30478 9056 Acked Read 2 Sent Done Done false
30479 9057 Acked Write 5 Sent Done Done false
30480 9058 Acked Read 5 Sent Done Done false
30481 9059 Acked Read 9 Sent Done Done false
30482 9060 Acked Flush 0 Sent Done Done false
30483 9061 Acked Flush 0 Sent Done Done false
30484 9062 Acked Write 5 Sent Done Done false
30485 9063 Acked Write 10 Sent Done Done false
30486 9064 Acked Write 7 Sent Done Done false
30487 9065 Acked Read 2 Sent Done Done false
30488 9066 Acked Write 3 Sent Done Done false
30489 9067 Acked Read 10 Sent Done Done false
30490 9068 Acked Write 10 Sent Done Done false
30491 9069 Acked Write 4 Sent Done Done false
30492 9070 Acked Read 10 Sent Done Done false
30493 9071 Acked Read 5 Sent Done Done false
30494 9072 Acked Read 4 Sent Done Done false
30495 9073 Acked Read 6 Sent Done Done false
30496 9074 Acked Read 9 Sent Done Done false
30497 9075 Acked Write 10 Sent Done Done false
30498 9076 Acked Flush 0 Sent Done Done false
30499 9077 Acked Read 10 Sent Done Done false
30500 9078 Acked Read 3 Sent Done Done false
30501 9079 Acked Read 4 Sent Done Done false
30502 9080 Acked Read 1 Sent Done Done false
30503 9081 Acked Read 8 Sent Done Done false
30504 9082 Acked Write 4 Sent Done Done false
30505 9083 Acked Flush 0 Sent Done Done false
30506 9084 Acked Write 2 Sent Done Done false
30507 9085 Acked Write 5 Sent Done Done false
30508 9086 Acked Read 6 Sent Done Done false
30509 9087 Acked Flush 0 Sent Done Done false
30510 9088 Acked Read 4 Sent Done Done false
30511 9089 Acked Read 1 Sent Done Done false
30512 9090 Acked Read 4 Sent Done Done false
30513 9091 Acked Read 10 Sent Done Done false
30514 9092 Acked Write 3 Sent Done Done false
30515 9093 Acked Read 2 Sent Done Done false
30516 9094 Acked Write 4 Sent Done Done false
30517 9095 Acked Flush 0 Sent Done Done false
30518 9096 Acked Write 8 Sent Done Done false
30519 9097 Acked Read 5 Sent Done Done false
30520 9098 Acked Read 4 Sent Done Done false
30521 9099 Acked Read 5 Sent Done Done false
30522 9100 Acked Read 5 Sent Done Done false
30523 9101 Acked Read 6 Sent Done Done false
30524 9102 Acked Write 2 Sent Done Done false
30525 9103 Acked Flush 0 Sent Done Done false
30526 9104 Acked Read 2 Sent Done Done false
30527 9105 Acked Read 1 Sent Done Done false
30528 9106 Acked Write 3 Sent Done Done false
30529 9107 Acked Write 5 Sent Done Done false
30530 9108 Acked Read 8 Sent Done Done false
30531 9109 Acked Read 4 Sent Done Done false
30532 9110 Acked Write 8 Sent Done Done false
30533 9111 Acked Read 8 Sent Done Done false
30534 9112 Acked Write 1 Sent Done Done false
30535 9113 Acked Write 6 Sent Done Done false
30536 9114 Acked Flush 0 Sent Done Done false
30537 9115 Acked Read 7 Sent Done Done false
30538 9116 Acked Read 4 Sent Done Done false
30539 9117 Acked Read 9 Sent Done Done false
30540 9118 Acked Read 5 Sent Done Done false
30541 9119 Acked Flush 0 Sent Done Done false
30542 9120 Acked Write 3 Sent Done Done false
30543 9121 Acked Write 3 Sent Done Done false
30544 9122 Acked Read 9 Sent Done Done false
30545 9123 Acked Read 10 Sent Done Done false
30546 9124 Acked Flush 0 Sent Done Done false
30547 9125 Acked Read 8 Sent Done Done false
30548 9126 Acked Read 9 Sent Done Done false
30549 9127 Acked Read 2 Sent Done Done false
30550 9128 Acked Read 8 Sent Done Done false
30551 9129 Acked Read 5 Sent Done Done false
30552 9130 Acked Write 7 Sent Done Done false
30553 9131 Acked Write 5 Sent Done Done false
30554 9132 Acked Read 10 Sent Done Done false
30555 9133 Acked Write 4 Sent Done Done false
30556 9134 Acked Read 10 Sent Done Done false
30557 9135 Acked Read 1 Sent Done Done false
30558 9136 Acked Write 5 Sent Done Done false
30559 9137 Acked Read 10 Sent Done Done false
30560 9138 Acked Read 10 Sent Done Done false
30561 9139 Acked Flush 0 Sent Done Done false
30562 9140 Acked Write 6 Sent Done Done false
30563 9141 Acked Flush 0 Sent Done Done false
30564 9142 Acked Flush 0 Sent Done Done false
30565 9143 Acked Read 10 Sent Done Done false
30566 9144 Acked Read 7 Sent Done Done false
30567 9145 Acked Flush 0 Sent Done Done false
30568 9146 Acked Read 10 Sent Done Done false
30569 9147 Acked Read 3 Sent Done Done false
30570 9148 Acked Read 3 Sent Done Done false
30571 9149 Acked Read 7 Sent Done Done false
30572 9150 Acked Write 6 Sent Done Done false
30573 9151 Acked Flush 0 Sent Done Done false
30574 9152 Acked Read 9 Sent Done Done false
30575 9153 Acked Write 5 Sent Done Done false
30576 9154 Acked Write 1 Sent Done Done false
30577 9155 Acked Read 6 Sent Done Done false
30578 9156 Acked Write 3 Sent Done Done false
30579 9157 Acked Read 10 Sent Done Done false
30580 STATES DS:0 DS:1 DS:2 TOTAL
30581 Sent 301 0 0 301
30582 Done 0 301 301 602
30583 Skipped 0 0 0 0
30584 Error 0 0 0 0
30585 Last Flush: 8856 9151 9151
30586 Downstairs last five completed: 8856 8855 8854 8853 8852
30587 Upstairs last five completed: 9157 9156 9155 9154 9153
30588 CLIENT: Up:0 ds:301 act:2
30589 [8810] got start action
30590 [8810] I am going to restart
30591 Starting downstairs at port 8810
30592 Make output file at "/tmp/dsc/downstairs-8810.txt"
30593 [8810][0] reports Starting
30594 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1225)
30595 [8810][0] reports Running
30596 ----------------------------------------------------------------
30597 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
30598 States: Offline Active Active
30599 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30600 8857 Acked Read 9 Sent Done Done false
30601 8858 Acked Read 8 Sent Done Done false
30602 8859 Acked Write 7 Sent Done Done false
30603 8860 Acked Write 5 Sent Done Done false
30604 8861 Acked Read 6 Sent Done Done false
30605 8862 Acked Read 9 Sent Done Done false
30606 8863 Acked Flush 0 Sent Done Done false
30607 8864 Acked Read 2 Sent Done Done false
30608 8865 Acked Read 1 Sent Done Done false
30609 8866 Acked Write 2 Sent Done Done false
30610 8867 Acked Write 2 Sent Done Done false
30611 8868 Acked Read 10 Sent Done Done false
30612 8869 Acked Write 1 Sent Done Done false
30613 8870 Acked Write 1 Sent Done Done false
30614 8871 Acked Write 9 Sent Done Done false
30615 8872 Acked Write 4 Sent Done Done false
30616 8873 Acked Read 6 Sent Done Done false
30617 8874 Acked Read 4 Sent Done Done false
30618 8875 Acked Write 3 Sent Done Done false
30619 8876 Acked Read 9 Sent Done Done false
30620 8877 Acked Read 4 Sent Done Done false
30621 8878 Acked Read 3 Sent Done Done false
30622 8879 Acked Flush 0 Sent Done Done false
30623 8880 Acked Write 10 Sent Done Done false
30624 8881 Acked Read 9 Sent Done Done false
30625 8882 Acked Flush 0 Sent Done Done false
30626 8883 Acked Write 9 Sent Done Done false
30627 8884 Acked Read 1 Sent Done Done false
30628 8885 Acked Read 8 Sent Done Done false
30629 8886 Acked Write 9 Sent Done Done false
30630 8887 Acked Flush 0 Sent Done Done false
30631 8888 Acked Read 2 Sent Done Done false
30632 8889 Acked Read 7 Sent Done Done false
30633 8890 Acked Read 9 Sent Done Done false
30634 8891 Acked Write 8 Sent Done Done false
30635 8892 Acked Write 2 Sent Done Done false
30636 8893 Acked Read 3 Sent Done Done false
30637 8894 Acked Write 9 Sent Done Done false
30638 8895 Acked Write 8 Sent Done Done false
30639 8896 Acked Read 1 Sent Done Done false
30640 8897 Acked Flush 0 Sent Done Done false
30641 8898 Acked Read 4 Sent Done Done false
30642 8899 Acked Flush 0 Sent Done Done false
30643 8900 Acked Write 7 Sent Done Done false
30644 8901 Acked Read 7 Sent Done Done false
30645 8902 Acked Read 2 Sent Done Done false
30646 8903 Acked Write 2 Sent Done Done false
30647 8904 Acked Write 7 Sent Done Done false
30648 8905 Acked Read 10 Sent Done Done false
30649 8906 Acked Flush 0 Sent Done Done false
30650 8907 Acked Read 3 Sent Done Done false
30651 8908 Acked Write 3 Sent Done Done false
30652 8909 Acked Read 2 Sent Done Done false
30653 8910 Acked Write 10 Sent Done Done false
30654 8911 Acked Read 5 Sent Done Done false
30655 8912 Acked Read 1 Sent Done Done false
30656 8913 Acked Read 5 Sent Done Done false
30657 8914 Acked Read 4 Sent Done Done false
30658 8915 Acked Read 2 Sent Done Done false
30659 8916 Acked Write 10 Sent Done Done false
30660 8917 Acked Flush 0 Sent Done Done false
30661 8918 Acked Write 3 Sent Done Done false
30662 8919 Acked Write 2 Sent Done Done false
30663 8920 Acked Read 8 Sent Done Done false
30664 8921 Acked Read 3 Sent Done Done false
30665 8922 Acked Write 7 Sent Done Done false
30666 8923 Acked Read 3 Sent Done Done false
30667 8924 Acked Write 2 Sent Done Done false
30668 8925 Acked Read 4 Sent Done Done false
30669 8926 Acked Flush 0 Sent Done Done false
30670 8927 Acked Flush 0 Sent Done Done false
30671 8928 Acked Read 6 Sent Done Done false
30672 8929 Acked Write 3 Sent Done Done false
30673 8930 Acked Flush 0 Sent Done Done false
30674 8931 Acked Read 7 Sent Done Done false
30675 8932 Acked Write 1 Sent Done Done false
30676 8933 Acked Write 3 Sent Done Done false
30677 8934 Acked Read 7 Sent Done Done false
30678 8935 Acked Read 8 Sent Done Done false
30679 8936 Acked Read 7 Sent Done Done false
30680 8937 Acked Read 5 Sent Done Done false
30681 8938 Acked Read 2 Sent Done Done false
30682 8939 Acked Write 3 Sent Done Done false
30683 8940 Acked Read 1 Sent Done Done false
30684 8941 Acked Write 10 Sent Done Done false
30685 8942 Acked Write 5 Sent Done Done false
30686 8943 Acked Read 2 Sent Done Done false
30687 8944 Acked Write 6 Sent Done Done false
30688 8945 Acked Read 5 Sent Done Done false
30689 8946 Acked Write 10 Sent Done Done false
30690 8947 Acked Write 10 Sent Done Done false
30691 8948 Acked Read 9 Sent Done Done false
30692 8949 Acked Write 3 Sent Done Done false
30693 8950 Acked Read 6 Sent Done Done false
30694 8951 Acked Write 5 Sent Done Done false
30695 8952 Acked Write 10 Sent Done Done false
30696 8953 Acked Read 2 Sent Done Done false
30697 8954 Acked Read 3 Sent Done Done false
30698 8955 Acked Flush 0 Sent Done Done false
30699 8956 Acked Write 3 Sent Done Done false
30700 8957 Acked Read 6 Sent Done Done false
30701 8958 Acked Write 6 Sent Done Done false
30702 8959 Acked Write 2 Sent Done Done false
30703 8960 Acked Write 10 Sent Done Done false
30704 8961 Acked Write 1 Sent Done Done false
30705 8962 Acked Read 1 Sent Done Done false
30706 8963 Acked Flush 0 Sent Done Done false
30707 8964 Acked Flush 0 Sent Done Done false
30708 8965 Acked Write 5 Sent Done Done false
30709 8966 Acked Flush 0 Sent Done Done false
30710 8967 Acked Read 1 Sent Done Done false
30711 8968 Acked Read 2 Sent Done Done false
30712 8969 Acked Read 6 Sent Done Done false
30713 8970 Acked Read 7 Sent Done Done false
30714 8971 Acked Read 6 Sent Done Done false
30715 8972 Acked Write 4 Sent Done Done false
30716 8973 Acked Read 8 Sent Done Done false
30717 8974 Acked Read 5 Sent Done Done false
30718 8975 Acked Flush 0 Sent Done Done false
30719 8976 Acked Read 2 Sent Done Done false
30720 8977 Acked Write 1 Sent Done Done false
30721 8978 Acked Read 10 Sent Done Done false
30722 8979 Acked Read 1 Sent Done Done false
30723 8980 Acked Write 7 Sent Done Done false
30724 8981 Acked Read 8 Sent Done Done false
30725 8982 Acked Write 2 Sent Done Done false
30726 8983 Acked Write 2 Sent Done Done false
30727 8984 Acked Read 8 Sent Done Done false
30728 8985 Acked Read 8 Sent Done Done false
30729 8986 Acked Read 2 Sent Done Done false
30730 8987 Acked Read 4 Sent Done Done false
30731 8988 Acked Read 2 Sent Done Done false
30732 8989 Acked Read 7 Sent Done Done false
30733 8990 Acked Read 7 Sent Done Done false
30734 8991 Acked Write 5 Sent Done Done false
30735 8992 Acked Read 6 Sent Done Done false
30736 8993 Acked Read 3 Sent Done Done false
30737 8994 Acked Read 6 Sent Done Done false
30738 8995 Acked Flush 0 Sent Done Done false
30739 8996 Acked Read 7 Sent Done Done false
30740 8997 Acked Read 2 Sent Done Done false
30741 8998 Acked Write 5 Sent Done Done false
30742 8999 Acked Write 7 Sent Done Done false
30743 9000 Acked Write 2 Sent Done Done false
30744 9001 Acked Flush 0 Sent Done Done false
30745 9002 Acked Read 1 Sent Done Done false
30746 9003 Acked Flush 0 Sent Done Done false
30747 9004 Acked Write 10 Sent Done Done false
30748 9005 Acked Read 10 Sent Done Done false
30749 9006 Acked Write 9 Sent Done Done false
30750 9007 Acked Read 5 Sent Done Done false
30751 9008 Acked Write 4 Sent Done Done false
30752 9009 Acked Read 5 Sent Done Done false
30753 9010 Acked Write 6 Sent Done Done false
30754 9011 Acked Write 8 Sent Done Done false
30755 9012 Acked Read 8 Sent Done Done false
30756 9013 Acked Write 7 Sent Done Done false
30757 9014 Acked Write 8 Sent Done Done false
30758 9015 Acked Write 4 Sent Done Done false
30759 9016 Acked Write 9 Sent Done Done false
30760 9017 Acked Read 4 Sent Done Done false
30761 9018 Acked Read 5 Sent Done Done false
30762 9019 Acked Write 9 Sent Done Done false
30763 9020 Acked Write 9 Sent Done Done false
30764 9021 Acked Write 5 Sent Done Done false
30765 9022 Acked Read 8 Sent Done Done false
30766 9023 Acked Read 7 Sent Done Done false
30767 9024 Acked Flush 0 Sent Done Done false
30768 9025 Acked Read 9 Sent Done Done false
30769 9026 Acked Read 3 Sent Done Done false
30770 9027 Acked Write 8 Sent Done Done false
30771 9028 Acked Flush 0 Sent Done Done false
30772 9029 Acked Read 6 Sent Done Done false
30773 9030 Acked Read 6 Sent Done Done false
30774 9031 Acked Read 6 Sent Done Done false
30775 9032 Acked Read 6 Sent Done Done false
30776 9033 Acked Write 10 Sent Done Done false
30777 9034 Acked Read 8 Sent Done Done false
30778 9035 Acked Read 5 Sent Done Done false
30779 9036 Acked Write 2 Sent Done Done false
30780 9037 Acked Read 10 Sent Done Done false
30781 9038 Acked Read 7 Sent Done Done false
30782 9039 Acked Flush 0 Sent Done Done false
30783 9040 Acked Write 2 Sent Done Done false
30784 9041 Acked Read 9 Sent Done Done false
30785 9042 Acked Write 3 Sent Done Done false
30786 9043 Acked Flush 0 Sent Done Done false
30787 9044 Acked Read 7 Sent Done Done false
30788 9045 Acked Write 9 Sent Done Done false
30789 9046 Acked Write 4 Sent Done Done false
30790 9047 Acked Write 1 Sent Done Done false
30791 9048 Acked Write 6 Sent Done Done false
30792 9049 Acked Write 3 Sent Done Done false
30793 9050 Acked Write 5 Sent Done Done false
30794 9051 Acked Flush 0 Sent Done Done false
30795 9052 Acked Read 4 Sent Done Done false
30796 9053 Acked Read 9 Sent Done Done false
30797 9054 Acked Write 2 Sent Done Done false
30798 9055 Acked Read 9 Sent Done Done false
30799 9056 Acked Read 2 Sent Done Done false
30800 9057 Acked Write 5 Sent Done Done false
30801 9058 Acked Read 5 Sent Done Done false
30802 9059 Acked Read 9 Sent Done Done false
30803 9060 Acked Flush 0 Sent Done Done false
30804 9061 Acked Flush 0 Sent Done Done false
30805 9062 Acked Write 5 Sent Done Done false
30806 9063 Acked Write 10 Sent Done Done false
30807 9064 Acked Write 7 Sent Done Done false
30808 9065 Acked Read 2 Sent Done Done false
30809 9066 Acked Write 3 Sent Done Done false
30810 9067 Acked Read 10 Sent Done Done false
30811 9068 Acked Write 10 Sent Done Done false
30812 9069 Acked Write 4 Sent Done Done false
30813 9070 Acked Read 10 Sent Done Done false
30814 9071 Acked Read 5 Sent Done Done false
30815 9072 Acked Read 4 Sent Done Done false
30816 9073 Acked Read 6 Sent Done Done false
30817 9074 Acked Read 9 Sent Done Done false
30818 9075 Acked Write 10 Sent Done Done false
30819 9076 Acked Flush 0 Sent Done Done false
30820 9077 Acked Read 10 Sent Done Done false
30821 9078 Acked Read 3 Sent Done Done false
30822 9079 Acked Read 4 Sent Done Done false
30823 9080 Acked Read 1 Sent Done Done false
30824 9081 Acked Read 8 Sent Done Done false
30825 9082 Acked Write 4 Sent Done Done false
30826 9083 Acked Flush 0 Sent Done Done false
30827 9084 Acked Write 2 Sent Done Done false
30828 9085 Acked Write 5 Sent Done Done false
30829 9086 Acked Read 6 Sent Done Done false
30830 9087 Acked Flush 0 Sent Done Done false
30831 9088 Acked Read 4 Sent Done Done false
30832 9089 Acked Read 1 Sent Done Done false
30833 9090 Acked Read 4 Sent Done Done false
30834 9091 Acked Read 10 Sent Done Done false
30835 9092 Acked Write 3 Sent Done Done false
30836 9093 Acked Read 2 Sent Done Done false
30837 9094 Acked Write 4 Sent Done Done false
30838 9095 Acked Flush 0 Sent Done Done false
30839 9096 Acked Write 8 Sent Done Done false
30840 9097 Acked Read 5 Sent Done Done false
30841 9098 Acked Read 4 Sent Done Done false
30842 9099 Acked Read 5 Sent Done Done false
30843 9100 Acked Read 5 Sent Done Done false
30844 9101 Acked Read 6 Sent Done Done false
30845 9102 Acked Write 2 Sent Done Done false
30846 9103 Acked Flush 0 Sent Done Done false
30847 9104 Acked Read 2 Sent Done Done false
30848 9105 Acked Read 1 Sent Done Done false
30849 9106 Acked Write 3 Sent Done Done false
30850 9107 Acked Write 5 Sent Done Done false
30851 9108 Acked Read 8 Sent Done Done false
30852 9109 Acked Read 4 Sent Done Done false
30853 9110 Acked Write 8 Sent Done Done false
30854 9111 Acked Read 8 Sent Done Done false
30855 9112 Acked Write 1 Sent Done Done false
30856 9113 Acked Write 6 Sent Done Done false
30857 9114 Acked Flush 0 Sent Done Done false
30858 9115 Acked Read 7 Sent Done Done false
30859 9116 Acked Read 4 Sent Done Done false
30860 9117 Acked Read 9 Sent Done Done false
30861 9118 Acked Read 5 Sent Done Done false
30862 9119 Acked Flush 0 Sent Done Done false
30863 9120 Acked Write 3 Sent Done Done false
30864 9121 Acked Write 3 Sent Done Done false
30865 9122 Acked Read 9 Sent Done Done false
30866 9123 Acked Read 10 Sent Done Done false
30867 9124 Acked Flush 0 Sent Done Done false
30868 9125 Acked Read 8 Sent Done Done false
30869 9126 Acked Read 9 Sent Done Done false
30870 9127 Acked Read 2 Sent Done Done false
30871 9128 Acked Read 8 Sent Done Done false
30872 9129 Acked Read 5 Sent Done Done false
30873 9130 Acked Write 7 Sent Done Done false
30874 9131 Acked Write 5 Sent Done Done false
30875 9132 Acked Read 10 Sent Done Done false
30876 9133 Acked Write 4 Sent Done Done false
30877 9134 Acked Read 10 Sent Done Done false
30878 9135 Acked Read 1 Sent Done Done false
30879 9136 Acked Write 5 Sent Done Done false
30880 9137 Acked Read 10 Sent Done Done false
30881 9138 Acked Read 10 Sent Done Done false
30882 9139 Acked Flush 0 Sent Done Done false
30883 9140 Acked Write 6 Sent Done Done false
30884 9141 Acked Flush 0 Sent Done Done false
30885 9142 Acked Flush 0 Sent Done Done false
30886 9143 Acked Read 10 Sent Done Done false
30887 9144 Acked Read 7 Sent Done Done false
30888 9145 Acked Flush 0 Sent Done Done false
30889 9146 Acked Read 10 Sent Done Done false
30890 9147 Acked Read 3 Sent Done Done false
30891 9148 Acked Read 3 Sent Done Done false
30892 9149 Acked Read 7 Sent Done Done false
30893 9150 Acked Write 6 Sent Done Done false
30894 9151 Acked Flush 0 Sent Done Done false
30895 9152 Acked Read 9 Sent Done Done false
30896 9153 Acked Write 5 Sent Done Done false
30897 9154 Acked Write 1 Sent Done Done false
30898 9155 Acked Read 6 Sent Done Done false
30899 9156 Acked Write 3 Sent Done Done false
30900 9157 Acked Read 10 Sent Done Done false
30901 9158 Acked Flush 0 Sent Done Done false
30902 STATES DS:0 DS:1 DS:2 TOTAL
30903 Sent 302 0 0 302
30904 Done 0 302 302 604
30905 Skipped 0 0 0 0
30906 Error 0 0 0 0
30907 Last Flush: 8856 9158 9158
30908 Downstairs last five completed: 8856 8855 8854 8853 8852
30909 Upstairs last five completed: 9158 9157 9156 9155 9154
30910 CLIENT: Up:0 ds:302 act:2
309112024-11-01T16:56:58.085ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
309122024-11-01T16:56:58.085ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47041) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
309132024-11-01T16:56:58.086ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
309142024-11-01T16:56:58.087ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
309152024-11-01T16:56:58.087ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
309162024-11-01T16:56:58.087ZINFOcrucible: send last flush ID to this DS: 8856 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
309172024-11-01T16:56:58.087ZINFOcrucible: Replied this last flush ID: 8856 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
309182024-11-01T16:56:58.087ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
309192024-11-01T16:56:58.087ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
309202024-11-01T16:56:58.087ZINFOcrucible: [0] client re-new 302 jobs since flush 8856 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
309212024-11-01T16:56:58.087ZINFOcrucible: [0] Replayed 302 jobs since flush: 8856 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
30922 ----------------------------------------------------------------
30923 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30924 Downstairs last five completed: 9158 9157 9156 9155 9154
30925 Upstairs last five completed: 9158 9157 9156 9155 9154
30926 CLIENT: Up:0 ds:0 act:3
30927 Replay: All jobs finished
30928 Main task has work to do, go find it
30929 got dsc Stop(0)
30930 stop 0
30931 [8810] Got stop action so:false kr:false
30932 Nov 01 16:57:00.395 INFO request completed, latency_us: 456, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 5c8c50c6-3c65-4508-b6ef-2d8efa68946c, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
309332024-11-01T16:57:00.396ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
309342024-11-01T16:57:00.396ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
309352024-11-01T16:57:00.396ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
30936 Nov 01 16:57:00.396 INFO{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:57:00.396114866Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"client":"0","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
30937 request completed, latency_us: 144, response_code: 200, uri: /state/cid/0, method: GET, req_id: d9dbac99-16eb-4e9b-b023-9ca4cf15b3d5, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
309382024-11-01T16:57:00.396ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
30939 [8810] Exited with: ExitStatus(unix_wait_status(9))
30940 [8810][0] reports Exit
30941 Nov 01 16:57:04.397 INFO request completed, latency_us: 243, response_code: 200, uri: /state/cid/0, method: GET, req_id: e0868923-69fd-44b6-b12c-662ebec31c01, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
30942 001/300 Read block 2658 len 36864
30943 002/300 Write block 4847 len 28672 data: 3 4 5 5 5 6 5
30944 003/300 Flush
30945 004/300 Read block 2508 len 32768
30946 005/300 Write block 1341 len 20480 data: 4 4 4 4 6
30947 006/300 Read block 4361 len 8192
30948 007/300 Flush
30949 008/300 Write block 930 len 16384 data: 7 7 7 7
30950 009/300 Write block 4963 len 12288 data: 6 7 8
30951 010/300 Read block 3072 len 20480
30952 011/300 Read block 3122 len 4096
30953 012/300 Flush
30954 013/300 Read block 2155 len 12288
30955 014/300 Write block 4847 len 24576 data: 4 5 6 6 6 7
30956 015/300 Read block 1368 len 12288
30957 016/300 Write block 4348 len 40960 data: 5 7 5 5 5 5 4 4 5 4
30958 017/300 Read block 3564 len 28672
30959 018/300 Write block 1258 len 8192 data: 4 4
30960 019/300 Write block 2229 len 28672 data: 6 8 6 7 7 6 8
30961 020/300 Write block 4572 len 12288 data: 6 7 6
30962 021/300 Write block 2278 len 12288 data: 6 6 7
30963 022/300 Read block 2193 len 40960
30964 023/300 Write block 949 len 32768 data: 4 5 4 3 2 2 2 3
30965 024/300 Flush
30966 025/300 Write block 953 len 4096 data: 3
30967 026/300 Read block 1471 len 32768
30968 027/300 Read block 991 len 12288
30969 028/300 Read block 787 len 12288
30970 029/300 Read block 1538 len 16384
30971 030/300 Read block 1051 len 36864
30972 031/300 Read block 2719 len 8192
30973 032/300 Read block 1950 len 4096
30974 033/300 Write block 1008 len 24576 data: 4 3 3 3 3 4
30975 034/300 Flush
30976 035/300 Read block 1155 len 32768
30977 036/300 Read block 1120 len 8192
30978 037/300 Read block 4602 len 20480
30979 038/300 Read block 4404 len 36864
30980 039/300 Read block 1713 len 20480
30981 040/300 Read block 866 len 12288
30982 041/300 Write block 1689 len 12288 data: 5 6 7
30983 042/300 Read block 3848 len 20480
30984 043/300 Flush
30985 044/300 Write block 1079 len 40960 data: 7 8 7 6 5 6 6 6 8 10
30986 045/300 Read block 2491 len 32768
30987 046/300 Read block 326 len 28672
30988 047/300 Write block 4948 len 24576 data: 6 7 7 7 7 6
30989 048/300 Write block 4107 len 36864 data: 9 9 9 10 10 9 8 8 8
30990 049/300 Read block 2812 len 32768
30991 050/300 Read block 2519 len 4096
30992 051/300 Flush
30993 052/300 Flush
30994 053/300 Read block 2345 len 20480
30995 054/300 Write block 930 len 24576 data: 8 8 8 8 5 5
30996 055/300 Write block 386 len 32768 data: 7 5 5 4 3 4 4 4
30997 056/300 Read block 241 len 32768
30998 057/300 Read block 843 len 32768
30999 058/300 Read block 2390 len 16384
31000 059/300 Write block 3826 len 28672 data: 9 9 10 9 10 10 9
31001 060/300 Write block 4218 len 28672 data: 6 5 6 6 6 6 6
31002 061/300 Read block 3195 len 40960
31003 062/300 Read block 3411 len 28672
31004 063/300 Read block 2072 len 8192
31005 064/300 Write block 4684 len 20480 data: 7 8 8 6 7
31006 065/300 Read block 2368 len 24576
31007 066/300 Write block 1218 len 24576 data: 11 8 8 8 5 5
31008 067/300 Read block 1354 len 12288
31009 068/300 Flush
31010 069/300 Write block 4268 len 16384 data: 4 5 6 5
31011 070/300 Write block 508 len 24576 data: 9 9 9 8 9 8
31012 071/300 Read block 1420 len 12288
31013 072/300 Write block 2868 len 36864 data: 6 5 6 7 6 6 6 7 8
31014 073/300 Write block 94 len 28672 data: 9 8 9 8 9 9 7
31015 074/300 Read block 1027 len 32768
31016 075/300 Read block 2513 len 8192
31017 076/300 Write block 4009 len 36864 data: 7 7 7 6 6 4 5 5 5
31018 077/300 Read block 62 len 12288
31019 078/300 Write block 83 len 8192 data: 5 5
31020 079/300 Read block 4338 len 36864
31021 080/300 Flush
31022 081/300 Flush
31023 082/300 Write block 34 len 16384 data: 2 2 4 4
31024 083/300 Read block 4560 len 24576
31025 084/300 Read block 521 len 16384
31026 085/300 Read block 4233 len 4096
31027 086/300 Read block 4229 len 36864
31028 087/300 Read block 1742 len 40960
31029 088/300 Write block 2173 len 16384 data: 4 4 3 3
31030 089/300 Read block 2380 len 16384
31031 090/300 Read block 4261 len 32768
31032 091/300 Write block 539 len 28672 data: 4 4 5 6 5 2 2
31033 092/300 Read block 3168 len 32768
31034 093/300 Write block 1895 len 20480 data: 3 4 2 2 2
31035 094/300 Write block 453 len 16384 data: 6 6 8 7
31036 095/300 Write block 1058 len 16384 data: 7 5 4 3
31037 096/300 Read block 2678 len 24576
31038 097/300 Read block 3347 len 20480
31039 098/300 Read block 3146 len 16384
31040 099/300 Write block 767 len 24576 data: 5 6 5 4 4 4
31041 100/300 Write block 1205 len 8192 data: 10 10
31042 101/300 Read block 266 len 12288
31043 102/300 Write block 1366 len 20480 data: 5 7 6 6 6
31044 103/300 Flush
31045 104/300 Read block 3893 len 12288
31046 105/300 Write block 3087 len 12288 data: 5 4 5
31047 106/300 Read block 1600 len 36864
31048 107/300 Read block 3412 len 16384
31049 108/300 Read block 672 len 12288
31050 109/300 Read block 2999 len 4096
31051 110/300 Write block 776 len 20480 data: 11 11 12 11 9
31052 111/300 Read block 2221 len 20480
31053 112/300 Read block 4023 len 4096
31054 113/300 Read block 1306 len 32768
31055 114/300 Read block 1678 len 36864
31056 115/300 Read block 476 len 4096
31057 116/300 Read block 3021 len 20480
31058 117/300 Read block 1589 len 32768
31059 118/300 Flush
31060 119/300 Write block 558 len 16384 data: 4 4 3 3
31061 120/300 Read block 4662 len 24576
31062 121/300 Write block 3378 len 24576 data: 5 4 5 5 7 8
31063 122/300 Write block 1231 len 12288 data: 3 3 3
31064 123/300 Read block 3594 len 24576
31065 124/300 Read block 12 len 16384
31066 125/300 Read block 4590 len 40960
31067 126/300 Read block 2785 len 8192
31068 127/300 Write block 732 len 32768 data: 10 9 10 10 9 6 7 8
31069 128/300 Write block 381 len 20480 data: 6 7 7 9 7
31070 129/300 Write block 383 len 32768 data: 8 10 8 8 6 6 5 4
31071 130/300 Write block 2196 len 24576 data: 5 5 5 6 7 5
31072 131/300 Read block 2812 len 36864
31073 132/300 Flush
31074 133/300 Write block 906 len 24576 data: 6 5 5 6 8 7
31075 134/300 Read block 1157 len 32768
31076 135/300 Read block 2774 len 16384
31077 136/300 Read block 4472 len 36864
31078 137/300 Read block 988 len 12288
31079 138/300 Write block 1173 len 12288 data: 6 6 7
31080 139/300 Read block 4315 len 28672
31081 140/300 Flush
31082 141/300 Write block 232 len 40960 data: 2 3 4 4 4 4 3 4 6 6
31083 142/300 Read block 1971 len 32768
31084 143/300 Read block 780 len 36864
31085 144/300 Flush
31086 145/300 Write block 4332 len 16384 data: 4 4 3 3
31087 146/300 Read block 1615 len 28672
31088 147/300 Read block 3834 len 40960
31089 148/300 Read block 899 len 16384
31090 149/300 Read block 1437 len 24576
31091 150/300 Write block 227 len 4096 data: 3
31092 151/300 Write block 4210 len 20480 data: 9 6 6 5 4
31093 152/300 Write block 1296 len 36864 data: 5 8 8 7 8 6 6 5 5
31094 153/300 Write block 4850 len 4096 data: 7
31095 154/300 Write block 3684 len 20480 data: 2 3 3 6 7
31096 155/300 Write block 1979 len 20480 data: 3 3 3 4 4
31097 156/300 Write block 2331 len 40960 data: 8 6 6 4 4 4 5 4 6 6
31098 157/300 Read block 2216 len 40960
31099 158/300 Read block 4101 len 8192
31100 159/300 Flush
31101 160/300 Flush
31102 161/300 Read block 306 len 24576
31103 162/300 Write block 1086 len 36864 data: 7 9 11 11 10 12 11 10 8
31104 163/300 Read block 4744 len 36864
31105 164/300 Read block 135 len 4096
31106 165/300 Write block 1951 len 4096 data: 6
31107 166/300 Write block 2245 len 20480 data: 7 6 5 5 4
31108 167/300 Read block 1026 len 16384
31109 168/300 Read block 2204 len 32768
31110 169/300 Read block 3597 len 28672
31111 170/300 Read block 1268 len 32768
31112 171/300 Write block 2369 len 28672 data: 8 8 8 6 7 6 5
31113 172/300 Flush
31114 173/300 Read block 3806 len 24576
31115 174/300 Read block 3989 len 32768
31116 175/300 Read block 940 len 28672
31117 176/300 Write block 4437 len 8192 data: 5 6
31118 177/300 Write block 4279 len 4096 data: 5
31119 178/300 Read block 1096 len 28672
31120 179/300 Flush
31121 180/300 Write block 1265 len 16384 data: 9 8 7 7
31122 181/300 Read block 2769 len 36864
31123 182/300 Flush
31124 183/300 Write block 778 len 8192 data: 13 12
31125 184/300 Read block 2370 len 8192
31126 185/300 Write block 2732 len 20480 data: 8 6 6 5 6
31127 186/300 Write block 1052 len 20480 data: 6 8 8 6 7
31128 187/300 Read block 1125 len 8192
31129 188/300 Read block 75 len 24576
31130 189/300 Read block 3852 len 32768
31131 190/300 Read block 2319 len 24576
31132 191/300 Read block 4009 len 20480
31133 192/300 Read block 1744 len 12288
31134 193/300 Read block 3819 len 28672
31135 194/300 Read block 2397 len 4096
31136 195/300 Write block 3422 len 20480 data: 7 6 6 7 7
31137 196/300 Write block 4306 len 36864 data: 4 5 5 8 10 11 10 9 9
31138 197/300 Read block 1770 len 4096
31139 198/300 Write block 2201 len 4096 data: 6
31140 199/300 Write block 2822 len 40960 data: 7 5 5 4 4 4 2 4 5 6
31141 200/300 Write block 4780 len 12288 data: 4 4 4
31142 201/300 Read block 3229 len 24576
31143 202/300 Write block 2111 len 32768 data: 6 6 6 5 2 4 4 4
31144 203/300 Flush
31145 204/300 Read block 2629 len 16384
31146 205/300 Read block 901 len 16384
31147 206/300 Write block 1516 len 36864 data: 7 3 3 4 5 6 6 6 8
31148 207/300 Write block 3024 len 12288 data: 4 5 6
31149 208/300 Read block 3691 len 28672
31150 209/300 Write block 4745 len 32768 data: 5 5 5 4 6 6 5 5
31151 210/300 Read block 4919 len 12288
31152 211/300 Write block 2378 len 36864 data: 5 4 4 4 4 5 4 3 3
31153 212/300 Write block 3367 len 4096 data: 6
31154 213/300 Read block 2823 len 40960
31155 214/300 Read block 1077 len 28672
31156 215/300 Flush
31157 216/300 Read block 3575 len 20480
31158 217/300 Write block 4242 len 40960 data: 3 4 4 5 6 7 7 7 6 7
31159 218/300 Flush
31160 219/300 Write block 2468 len 28672 data: 5 4 3 5 7 9 10
31161 220/300 Read block 364 len 16384
31162 221/300 Write block 4236 len 40960 data: 2 2 2 2 2 2 4 5 5 6
31163 222/300 Write block 723 len 16384 data: 5 4 3 4
31164 223/300 Read block 913 len 4096
31165 224/300 Write block 642 len 20480 data: 5 6 6 8 7
31166 225/300 Write block 883 len 4096 data: 7
31167 226/300 Read block 675 len 24576
31168 227/300 Read block 4651 len 8192
31169 228/300 Write block 3135 len 24576 data: 9 9 7 7 6 6
31170 229/300 Read block 4201 len 8192
31171 230/300 Write block 1114 len 12288 data: 4 3 6
31172 231/300 Write block 4981 len 32768 data: 8 8 6 5 6 5 4 5
31173 232/300 Write block 2121 len 40960 data: 5 6 5 7 6 4 4 3 3 3
31174 233/300 Read block 4624 len 4096
31175 234/300 Read block 4351 len 24576
31176 235/300 Write block 3957 len 8192 data: 5 5
31177 236/300 Flush
31178 237/300 Read block 1815 len 8192
31179 238/300 Write block 28 len 4096 data: 5
31180 239/300 Write block 4693 len 8192 data: 4 5
31181 240/300 Write block 2184 len 36864 data: 6 7 7 7 6 6 6 4 3
31182 241/300 Write block 4093 len 12288 data: 5 4 3
31183 242/300 Read block 2478 len 16384
31184 243/300 Write block 1337 len 16384 data: 7 6 4 5
31185 244/300 Write block 4012 len 16384 data: 7 7 5 6
31186 245/300 Read block 3027 len 32768
31187 246/300 Flush
31188 247/300 Write block 1474 len 36864 data: 9 7 9 9 9 8 8 6 5
31189 248/300 Read block 2146 len 32768
31190 249/300 Write block 4010 len 24576 data: 8 8 8 8 6 7
31191 250/300 Write block 3982 len 16384 data: 6 5 5 4
31192 251/300 Read block 4045 len 40960
31193 252/300 Read block 4329 len 12288
31194 253/300 Write block 2233 len 8192 data: 8 7
31195 254/300 Read block 2 len 12288
31196 255/300 Write block 2510 len 40960 data: 6 6 7 7 9 9 7 6 6 5
31197 256/300 Write block 192 len 20480 data: 5 5 4 5 5
31198 257/300 Read block 3191 len 28672
31199 258/300 Flush
31200 259/300 Write block 2343 len 36864 data: 8 8 8 6 5 5 4 4 4
31201 260/300 Read block 2986 len 16384
31202 261/300 Read block 3280 len 8192
31203 262/300 Read block 830 len 28672
31204 263/300 Read block 527 len 4096
31205 264/300 Read block 3003 len 40960
31206 265/300 Write block 761 len 20480 data: 4 6 6 5 5
31207 266/300 Read block 4280 len 40960
31208 267/300 Read block 1027 len 36864
31209 268/300 Write block 1853 len 36864 data: 4 4 3 4 4 4 6 4 3
31210 269/300 Write block 1870 len 40960 data: 10 9 12 12 9 8 8 10 9 9
31211 270/300 Write block 143 len 4096 data: 3
31212 271/300 Write block 1337 len 8192 data: 8 7
31213 272/300 Read block 491 len 12288
31214 273/300 Read block 4796 len 4096
31215 274/300 Read block 3111 len 4096
31216 275/300 Write block 2499 len 12288 data: 6 6 4
31217 276/300 Write block 402 len 4096 data: 5
31218 277/300 Read block 3228 len 8192
31219 278/300 Write block 912 len 20480 data: 7 7 8 9 8
31220 279/300 Write block 1820 len 16384 data: 6 7 7 7
31221 280/300 Write block 2359 len 28672 data: 6 7 7 8 7 7 9
31222 281/300 Write block 4579 len 40960 data: 6 6 5 6 6 5 5 3 3 3
31223 282/300 Write block 2192 len 20480 data: 4 3 3 4 6
31224 283/300 Write block 4525 len 16384 data: 9 8 9 8
31225 284/300 Read block 4632 len 32768
31226 285/300 Flush
31227 286/300 Write block 3191 len 8192 data: 5 4
31228 287/300 Read block 2401 len 4096
31229 288/300 Write block 616 len 20480 data: 2 2 4 4 4
31230 289/300 Read block 4016 len 4096
31231 290/300 Write block 3592 len 12288 data: 10 8 6
31232 291/300 Read block 1323 len 8192
31233 292/300 Write block 3537 len 40960 data: 6 6 5 4 5 6 6 5 6 5
31234 293/300 Write block 4466 len 40960 data: 3 3 4 5 4 4 4 5 6 5
31235 294/300 Write block 1325 len 40960 data: 3 3 4 4 4 2 4 5 7 6
31236 295/300 Write block 2620 len 20480 data: 8 7 6 6 6
31237 296/300 Flush
31238 297/300 Write block 3299 len 20480 data: 7 9 7 7 4
31239 298/300 Read block 3698 len 4096
31240 299/300 Read block 3074 len 4096
31241 300/300 Write block 4475 len 24576 data: 6 3 3 3 3 3
31242 301/300 Read block 3208 len 8192
31243 Main task has work to do, go find it
31244 got dsc Start(0)
31245 start 0
31246 Nov 01 16:57:04.697 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/0, method: POST, req_id: 06a9e366-50f7-417c-8a3e-0703f4ea62f5, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
31247 Replay: started 0, returned:Ok(())
31248 ----------------------------------------------------------------
31249 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
31250 States: Offline Active Active
31251 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31252 9159 Acked Read 9 Sent Done Done false
31253 9160 Acked Write 7 Sent Done Done false
31254 9161 Acked Flush 0 Sent Done Done false
31255 9162 Acked Read 8 Sent Done Done false
31256 9163 Acked Write 5 Sent Done Done false
31257 9164 Acked Read 2 Sent Done Done false
31258 9165 Acked Flush 0 Sent Done Done false
31259 9166 Acked Write 4 Sent Done Done false
31260 9167 Acked Write 3 Sent Done Done false
31261 9168 Acked Read 5 Sent Done Done false
31262 9169 Acked Read 1 Sent Done Done false
31263 9170 Acked Flush 0 Sent Done Done false
31264 9171 Acked Read 3 Sent Done Done false
31265 9172 Acked Write 6 Sent Done Done false
31266 9173 Acked Read 3 Sent Done Done false
31267 9174 Acked Write 10 Sent Done Done false
31268 9175 Acked Read 7 Sent Done Done false
31269 9176 Acked Write 2 Sent Done Done false
31270 9177 Acked Write 7 Sent Done Done false
31271 9178 Acked Write 3 Sent Done Done false
31272 9179 Acked Write 3 Sent Done Done false
31273 9180 Acked Read 10 Sent Done Done false
31274 9181 Acked Write 8 Sent Done Done false
31275 9182 Acked Flush 0 Sent Done Done false
31276 9183 Acked Write 1 Sent Done Done false
31277 9184 Acked Read 8 Sent Done Done false
31278 9185 Acked Read 3 Sent Done Done false
31279 9186 Acked Read 3 Sent Done Done false
31280 9187 Acked Read 4 Sent Done Done false
31281 9188 Acked Read 9 Sent Done Done false
31282 9189 Acked Read 2 Sent Done Done false
31283 9190 Acked Read 1 Sent Done Done false
31284 9191 Acked Write 6 Sent Done Done false
31285 9192 Acked Flush 0 Sent Done Done false
31286 9193 Acked Read 8 Sent Done Done false
31287 9194 Acked Read 2 Sent Done Done false
31288 9195 Acked Read 5 Sent Done Done false
31289 9196 Acked Read 9 Sent Done Done false
31290 9197 Acked Read 5 Sent Done Done false
31291 9198 Acked Read 3 Sent Done Done false
31292 9199 Acked Write 3 Sent Done Done false
31293 9200 Acked Read 5 Sent Done Done false
31294 9201 Acked Flush 0 Sent Done Done false
31295 9202 Acked Write 10 Sent Done Done false
31296 9203 Acked Read 8 Sent Done Done false
31297 9204 Acked Read 7 Sent Done Done false
31298 9205 Acked Write 6 Sent Done Done false
31299 9206 Acked Write 9 Sent Done Done false
31300 9207 Acked Read 8 Sent Done Done false
31301 9208 Acked Read 1 Sent Done Done false
31302 9209 Acked Flush 0 Sent Done Done false
31303 9210 Acked Flush 0 Sent Done Done false
31304 9211 Acked Read 5 Sent Done Done false
31305 9212 Acked Write 6 Sent Done Done false
31306 9213 Acked Write 8 Sent Done Done false
31307 9214 Acked Read 8 Sent Done Done false
31308 9215 Acked Read 8 Sent Done Done false
31309 9216 Acked Read 4 Sent Done Done false
31310 9217 Acked Write 7 Sent Done Done false
31311 9218 Acked Write 7 Sent Done Done false
31312 9219 Acked Read 10 Sent Done Done false
31313 9220 Acked Read 7 Sent Done Done false
31314 9221 Acked Read 2 Sent Done Done false
31315 9222 Acked Write 5 Sent Done Done false
31316 9223 Acked Read 6 Sent Done Done false
31317 9224 Acked Write 6 Sent Done Done false
31318 9225 Acked Read 3 Sent Done Done false
31319 9226 Acked Flush 0 Sent Done Done false
31320 9227 Acked Write 4 Sent Done Done false
31321 9228 Acked Write 6 Sent Done Done false
31322 9229 Acked Read 3 Sent Done Done false
31323 9230 Acked Write 9 Sent Done Done false
31324 9231 Acked Write 7 Sent Done Done false
31325 9232 Acked Read 8 Sent Done Done false
31326 9233 Acked Read 2 Sent Done Done false
31327 9234 Acked Write 9 Sent Done Done false
31328 9235 Acked Read 3 Sent Done Done false
31329 9236 Acked Write 2 Sent Done Done false
31330 9237 Acked Read 9 Sent Done Done false
31331 9238 Acked Flush 0 Sent Done Done false
31332 9239 Acked Flush 0 Sent Done Done false
31333 9240 Acked Write 4 Sent Done Done false
31334 9241 Acked Read 6 Sent Done Done false
31335 9242 Acked Read 4 Sent Done Done false
31336 9243 Acked Read 1 Sent Done Done false
31337 9244 Acked Read 9 Sent Done Done false
31338 9245 Acked Read 10 Sent Done Done false
31339 9246 Acked Write 4 Sent Done Done false
31340 9247 Acked Read 4 Sent Done Done false
31341 9248 Acked Read 8 Sent Done Done false
31342 9249 Acked Write 7 Sent Done Done false
31343 9250 Acked Read 8 Sent Done Done false
31344 9251 Acked Write 5 Sent Done Done false
31345 9252 Acked Write 4 Sent Done Done false
31346 9253 Acked Write 4 Sent Done Done false
31347 9254 Acked Read 6 Sent Done Done false
31348 9255 Acked Read 5 Sent Done Done false
31349 9256 Acked Read 4 Sent Done Done false
31350 9257 Acked Write 6 Sent Done Done false
31351 9258 Acked Write 2 Sent Done Done false
31352 9259 Acked Read 3 Sent Done Done false
31353 9260 Acked Write 5 Sent Done Done false
31354 9261 Acked Flush 0 Sent Done Done false
31355 9262 Acked Read 3 Sent Done Done false
31356 9263 Acked Write 3 Sent Done Done false
31357 9264 Acked Read 9 Sent Done Done false
31358 9265 Acked Read 4 Sent Done Done false
31359 9266 Acked Read 3 Sent Done Done false
31360 9267 Acked Read 1 Sent Done Done false
31361 9268 Acked Write 5 Sent Done Done false
31362 9269 Acked Read 5 Sent Done Done false
31363 9270 Acked Read 1 Sent Done Done false
31364 9271 Acked Read 8 Sent Done Done false
31365 9272 Acked Read 9 Sent Done Done false
31366 9273 Acked Read 1 Sent Done Done false
31367 9274 Acked Read 5 Sent Done Done false
31368 9275 Acked Read 8 Sent Done Done false
31369 9276 Acked Flush 0 Sent Done Done false
31370 9277 Acked Write 4 Sent Done Done false
31371 9278 Acked Read 6 Sent Done Done false
31372 9279 Acked Write 6 Sent Done Done false
31373 9280 Acked Write 3 Sent Done Done false
31374 9281 Acked Read 6 Sent Done Done false
31375 9282 Acked Read 4 Sent Done Done false
31376 9283 Acked Read 10 Sent Done Done false
31377 9284 Acked Read 2 Sent Done Done false
31378 9285 Acked Write 8 Sent Done Done false
31379 9286 Acked Write 5 Sent Done Done false
31380 9287 Acked Write 8 Sent Done Done false
31381 9288 Acked Write 6 Sent Done Done false
31382 9289 Acked Read 9 Sent Done Done false
31383 9290 Acked Flush 0 Sent Done Done false
31384 9291 Acked Write 6 Sent Done Done false
31385 9292 Acked Read 8 Sent Done Done false
31386 9293 Acked Read 4 Sent Done Done false
31387 9294 Acked Read 9 Sent Done Done false
31388 9295 Acked Read 3 Sent Done Done false
31389 9296 Acked Write 3 Sent Done Done false
31390 9297 Acked Read 7 Sent Done Done false
31391 9298 Acked Flush 0 Sent Done Done false
31392 9299 Acked Write 10 Sent Done Done false
31393 9300 Acked Read 8 Sent Done Done false
31394 9301 Acked Read 9 Sent Done Done false
31395 9302 Acked Flush 0 Sent Done Done false
31396 9303 Acked Write 4 Sent Done Done false
31397 9304 Acked Read 7 Sent Done Done false
31398 9305 Acked Read 10 Sent Done Done false
31399 9306 Acked Read 4 Sent Done Done false
31400 9307 Acked Read 6 Sent Done Done false
31401 9308 Acked Write 1 Sent Done Done false
31402 9309 Acked Write 5 Sent Done Done false
31403 9310 Acked Write 9 Sent Done Done false
31404 9311 Acked Write 1 Sent Done Done false
31405 9312 Acked Write 5 Sent Done Done false
31406 9313 Acked Write 5 Sent Done Done false
31407 9314 Acked Write 10 Sent Done Done false
31408 9315 Acked Read 10 Sent Done Done false
31409 9316 Acked Read 2 Sent Done Done false
31410 9317 Acked Flush 0 Sent Done Done false
31411 9318 Acked Flush 0 Sent Done Done false
31412 9319 Acked Read 6 Sent Done Done false
31413 9320 Acked Write 9 Sent Done Done false
31414 9321 Acked Read 9 Sent Done Done false
31415 9322 Acked Read 1 Sent Done Done false
31416 9323 Acked Write 1 Sent Done Done false
31417 9324 Acked Write 5 Sent Done Done false
31418 9325 Acked Read 4 Sent Done Done false
31419 9326 Acked Read 8 Sent Done Done false
31420 9327 Acked Read 7 Sent Done Done false
31421 9328 Acked Read 8 Sent Done Done false
31422 9329 Acked Write 7 Sent Done Done false
31423 9330 Acked Flush 0 Sent Done Done false
31424 9331 Acked Read 6 Sent Done Done false
31425 9332 Acked Read 8 Sent Done Done false
31426 9333 Acked Read 7 Sent Done Done false
31427 9334 Acked Write 2 Sent Done Done false
31428 9335 Acked Write 1 Sent Done Done false
31429 9336 Acked Read 7 Sent Done Done false
31430 9337 Acked Flush 0 Sent Done Done false
31431 9338 Acked Write 4 Sent Done Done false
31432 9339 Acked Read 9 Sent Done Done false
31433 9340 Acked Flush 0 Sent Done Done false
31434 9341 Acked Write 2 Sent Done Done false
31435 9342 Acked Read 2 Sent Done Done false
31436 9343 Acked Write 5 Sent Done Done false
31437 9344 Acked Write 5 Sent Done Done false
31438 9345 Acked Read 2 Sent Done Done false
31439 9346 Acked Read 6 Sent Done Done false
31440 9347 Acked Read 8 Sent Done Done false
31441 9348 Acked Read 6 Sent Done Done false
31442 9349 Acked Read 5 Sent Done Done false
31443 9350 Acked Read 3 Sent Done Done false
31444 9351 Acked Flush 0 Sent Done Done false
31445 9352 Acked Read 7 Sent Done Done false
31446 9353 Acked Read 1 Sent Done Done false
31447 9354 Acked Write 5 Sent Done Done false
31448 9355 Acked Write 9 Sent Done Done false
31449 9356 Acked Read 1 Sent Done Done false
31450 9357 Acked Write 1 Sent Done Done false
31451 9358 Acked Write 10 Sent Done Done false
31452 9359 Acked Write 3 Sent Done Done false
31453 9360 Acked Read 6 Sent Done Done false
31454 9361 Acked Write 8 Sent Done Done false
31455 9362 Acked Flush 0 Sent Done Done false
31456 9363 Acked Read 4 Sent Done Done false
31457 9364 Acked Read 4 Sent Done Done false
31458 9365 Acked Write 9 Sent Done Done false
31459 9366 Acked Write 3 Sent Done Done false
31460 9367 Acked Read 7 Sent Done Done false
31461 9368 Acked Write 8 Sent Done Done false
31462 9369 Acked Read 3 Sent Done Done false
31463 9370 Acked Write 9 Sent Done Done false
31464 9371 Acked Write 1 Sent Done Done false
31465 9372 Acked Read 10 Sent Done Done false
31466 9373 Acked Read 7 Sent Done Done false
31467 9374 Acked Flush 0 Sent Done Done false
31468 9375 Acked Read 5 Sent Done Done false
31469 9376 Acked Write 10 Sent Done Done false
31470 9377 Acked Flush 0 Sent Done Done false
31471 9378 Acked Write 7 Sent Done Done false
31472 9379 Acked Read 4 Sent Done Done false
31473 9380 Acked Write 10 Sent Done Done false
31474 9381 Acked Write 4 Sent Done Done false
31475 9382 Acked Read 1 Sent Done Done false
31476 9383 Acked Write 5 Sent Done Done false
31477 9384 Acked Write 1 Sent Done Done false
31478 9385 Acked Read 6 Sent Done Done false
31479 9386 Acked Read 2 Sent Done Done false
31480 9387 Acked Write 6 Sent Done Done false
31481 9388 Acked Read 2 Sent Done Done false
31482 9389 Acked Write 3 Sent Done Done false
31483 9390 Acked Write 8 Sent Done Done false
31484 9391 Acked Write 10 Sent Done Done false
31485 9392 Acked Read 1 Sent Done Done false
31486 9393 Acked Read 6 Sent Done Done false
31487 9394 Acked Write 2 Sent Done Done false
31488 9395 Acked Flush 0 Sent Done Done false
31489 9396 Acked Read 2 Sent Done Done false
31490 9397 Acked Write 1 Sent Done Done false
31491 9398 Acked Write 2 Sent Done Done false
31492 9399 Acked Write 9 Sent Done Done false
31493 9400 Acked Write 3 Sent Done Done false
31494 9401 Acked Read 4 Sent Done Done false
31495 9402 Acked Write 4 Sent Done Done false
31496 9403 Acked Write 4 Sent Done Done false
31497 9404 Acked Read 8 Sent Done Done false
31498 9405 Acked Flush 0 Sent Done Done false
31499 9406 Acked Write 9 Sent Done Done false
31500 9407 Acked Read 8 Sent Done Done false
31501 9408 Acked Write 6 Sent Done Done false
31502 9409 Acked Write 4 Sent Done Done false
31503 9410 Acked Read 10 Sent Done Done false
31504 9411 Acked Read 3 Sent Done Done false
31505 9412 Acked Write 2 Sent Done Done false
31506 9413 Acked Read 3 Sent Done Done false
31507 9414 Acked Write 10 Sent Done Done false
31508 9415 Acked Write 5 Sent Done Done false
31509 9416 Acked Read 7 Sent Done Done false
31510 9417 Acked Flush 0 Sent Done Done false
31511 9418 Acked Write 9 Sent Done Done false
31512 9419 Acked Read 4 Sent Done Done false
31513 9420 Acked Read 2 Sent Done Done false
31514 9421 Acked Read 7 Sent Done Done false
31515 9422 Acked Read 1 Sent Done Done false
31516 9423 Acked Read 10 Sent Done Done false
31517 9424 Acked Write 5 Sent Done Done false
31518 9425 Acked Read 10 Sent Done Done false
31519 9426 Acked Read 9 Sent Done Done false
31520 9427 Acked Write 9 Sent Done Done false
31521 9428 Acked Write 10 Sent Done Done false
31522 9429 Acked Write 1 Sent Done Done false
31523 9430 Acked Write 2 Sent Done Done false
31524 9431 Acked Read 3 Sent Done Done false
31525 9432 Acked Read 1 Sent Done Done false
31526 9433 Acked Read 1 Sent Done Done false
31527 9434 Acked Write 3 Sent Done Done false
31528 9435 Acked Write 1 Sent Done Done false
31529 9436 Acked Read 2 Sent Done Done false
31530 9437 Acked Write 5 Sent Done Done false
31531 9438 Acked Write 4 Sent Done Done false
31532 9439 Acked Write 7 Sent Done Done false
31533 9440 Acked Write 10 Sent Done Done false
31534 9441 Acked Write 5 Sent Done Done false
31535 9442 Acked Write 4 Sent Done Done false
31536 9443 Acked Read 8 Sent Done Done false
31537 9444 Acked Flush 0 Sent Done Done false
31538 9445 Acked Write 2 Sent Done Done false
31539 9446 Acked Read 1 Sent Done Done false
31540 9447 Acked Write 5 Sent Done Done false
31541 9448 Acked Read 1 Sent Done Done false
31542 9449 Acked Write 3 Sent Done Done false
31543 9450 Acked Read 2 Sent Done Done false
31544 9451 Acked Write 10 Sent Done Done false
31545 9452 Acked Write 10 Sent Done Done false
31546 9453 Acked Write 10 Sent Done Done false
31547 9454 Acked Write 5 Sent Done Done false
31548 9455 Acked Flush 0 Sent Done Done false
31549 9456 Acked Write 5 Sent Done Done false
31550 9457 Acked Read 1 Sent Done Done false
31551 9458 Acked Read 1 Sent Done Done false
31552 9459 Acked Write 6 Sent Done Done false
31553 9460 Acked Read 2 Sent Done Done false
31554 STATES DS:0 DS:1 DS:2 TOTAL
31555 Sent 302 0 0 302
31556 Done 0 302 302 604
31557 Skipped 0 0 0 0
31558 Error 0 0 0 0
31559 Last Flush: 9158 9455 9455
31560 Downstairs last five completed: 9158 9157 9156 9155 9154
31561 Upstairs last five completed: 9460 9459 9458 9457 9456
31562 CLIENT: Up:0 ds:302 act:2
31563 ----------------------------------------------------------------
31564 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
31565 States: Offline Active Active
31566 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31567 9159 Acked Read 9 Sent Done Done false
31568 9160 Acked Write 7 Sent Done Done false
31569 9161 Acked Flush 0 Sent Done Done false
31570 9162 Acked Read 8 Sent Done Done false
31571 9163 Acked Write 5 Sent Done Done false
31572 9164 Acked Read 2 Sent Done Done false
31573 9165 Acked Flush 0 Sent Done Done false
31574 9166 Acked Write 4 Sent Done Done false
31575 9167 Acked Write 3 Sent Done Done false
31576 9168 Acked Read 5 Sent Done Done false
31577 9169 Acked Read 1 Sent Done Done false
31578 9170 Acked Flush 0 Sent Done Done false
31579 9171 Acked Read 3 Sent Done Done false
31580 9172 Acked Write 6 Sent Done Done false
31581 9173 Acked Read 3 Sent Done Done false
31582 9174 Acked Write 10 Sent Done Done false
31583 9175 Acked Read 7 Sent Done Done false
31584 9176 Acked Write 2 Sent Done Done false
31585 9177 Acked Write 7 Sent Done Done false
31586 9178 Acked Write 3 Sent Done Done false
31587 9179 Acked Write 3 Sent Done Done false
31588 9180 Acked Read 10 Sent Done Done false
31589 9181 Acked Write 8 Sent Done Done false
31590 9182 Acked Flush 0 Sent Done Done false
31591 9183 Acked Write 1 Sent Done Done false
31592 9184 Acked Read 8 Sent Done Done false
31593 9185 Acked Read 3 Sent Done Done false
31594 9186 Acked Read 3 Sent Done Done false
31595 9187 Acked Read 4 Sent Done Done false
31596 9188 Acked Read 9 Sent Done Done false
31597 9189 Acked Read 2 Sent Done Done false
31598 9190 Acked Read 1 Sent Done Done false
31599 9191 Acked Write 6 Sent Done Done false
31600 9192 Acked Flush 0 Sent Done Done false
31601 9193 Acked Read 8 Sent Done Done false
31602 9194 Acked Read 2 Sent Done Done false
31603 9195 Acked Read 5 Sent Done Done false
31604 9196 Acked Read 9 Sent Done Done false
31605 9197 Acked Read 5 Sent Done Done false
31606 9198 Acked Read 3 Sent Done Done false
31607 9199 Acked Write 3 Sent Done Done false
31608 9200 Acked Read 5 Sent Done Done false
31609 9201 Acked Flush 0 Sent Done Done false
31610 9202 Acked Write 10 Sent Done Done false
31611 9203 Acked Read 8 Sent Done Done false
31612 9204 Acked Read 7 Sent Done Done false
31613 9205 Acked Write 6 Sent Done Done false
31614 9206 Acked Write 9 Sent Done Done false
31615 9207 Acked Read 8 Sent Done Done false
31616 9208 Acked Read 1 Sent Done Done false
31617 9209 Acked Flush 0 Sent Done Done false
31618 9210 Acked Flush 0 Sent Done Done false
31619 9211 Acked Read 5 Sent Done Done false
31620 9212 Acked Write 6 Sent Done Done false
31621 9213 Acked Write 8 Sent Done Done false
31622 9214 Acked Read 8 Sent Done Done false
31623 9215 Acked Read 8 Sent Done Done false
31624 9216 Acked Read 4 Sent Done Done false
31625 9217 Acked Write 7 Sent Done Done false
31626 9218 Acked Write 7 Sent Done Done false
31627 9219 Acked Read 10 Sent Done Done false
31628 9220 Acked Read 7 Sent Done Done false
31629 9221 Acked Read 2 Sent Done Done false
31630 9222 Acked Write 5 Sent Done Done false
31631 9223 Acked Read 6 Sent Done Done false
31632 9224 Acked Write 6 Sent Done Done false
31633 9225 Acked Read 3 Sent Done Done false
31634 9226 Acked Flush 0 Sent Done Done false
31635 9227 Acked Write 4 Sent Done Done false
31636 9228 Acked Write 6 Sent Done Done false
31637 9229 Acked Read 3 Sent Done Done false
31638 9230 Acked Write 9 Sent Done Done false
31639 9231 Acked Write 7 Sent Done Done false
31640 9232 Acked Read 8 Sent Done Done false
31641 9233 Acked Read 2 Sent Done Done false
31642 9234 Acked Write 9 Sent Done Done false
31643 9235 Acked Read 3 Sent Done Done false
31644 9236 Acked Write 2 Sent Done Done false
31645 9237 Acked Read 9 Sent Done Done false
31646 9238 Acked Flush 0 Sent Done Done false
31647 9239 Acked Flush 0 Sent Done Done false
31648 9240 Acked Write 4 Sent Done Done false
31649 9241 Acked Read 6 Sent Done Done false
31650 9242 Acked Read 4 Sent Done Done false
31651 9243 Acked Read 1 Sent Done Done false
31652 9244 Acked Read 9 Sent Done Done false
31653 9245 Acked Read 10 Sent Done Done false
31654 9246 Acked Write 4 Sent Done Done false
31655 9247 Acked Read 4 Sent Done Done false
31656 9248 Acked Read 8 Sent Done Done false
31657 9249 Acked Write 7 Sent Done Done false
31658 9250 Acked Read 8 Sent Done Done false
31659 9251 Acked Write 5 Sent Done Done false
31660 9252 Acked Write 4 Sent Done Done false
31661 9253 Acked Write 4 Sent Done Done false
31662 9254 Acked Read 6 Sent Done Done false
31663 9255 Acked Read 5 Sent Done Done false
31664 9256 Acked Read 4 Sent Done Done false
31665 9257 Acked Write 6 Sent Done Done false
31666 9258 Acked Write 2 Sent Done Done false
31667 9259 Acked Read 3 Sent Done Done false
31668 9260 Acked Write 5 Sent Done Done false
31669 9261 Acked Flush 0 Sent Done Done false
31670 9262 Acked Read 3 Sent Done Done false
31671 9263 Acked Write 3 Sent Done Done false
31672 9264 Acked Read 9 Sent Done Done false
31673 9265 Acked Read 4 Sent Done Done false
31674 9266 Acked Read 3 Sent Done Done false
31675 9267 Acked Read 1 Sent Done Done false
31676 9268 Acked Write 5 Sent Done Done false
31677 9269 Acked Read 5 Sent Done Done false
31678 9270 Acked Read 1 Sent Done Done false
31679 9271 Acked Read 8 Sent Done Done false
31680 9272 Acked Read 9 Sent Done Done false
31681 9273 Acked Read 1 Sent Done Done false
31682 9274 Acked Read 5 Sent Done Done false
31683 9275 Acked Read 8 Sent Done Done false
31684 9276 Acked Flush 0 Sent Done Done false
31685 9277 Acked Write 4 Sent Done Done false
31686 9278 Acked Read 6 Sent Done Done false
31687 9279 Acked Write 6 Sent Done Done false
31688 9280 Acked Write 3 Sent Done Done false
31689 9281 Acked Read 6 Sent Done Done false
31690 9282 Acked Read 4 Sent Done Done false
31691 9283 Acked Read 10 Sent Done Done false
31692 9284 Acked Read 2 Sent Done Done false
31693 9285 Acked Write 8 Sent Done Done false
31694 9286 Acked Write 5 Sent Done Done false
31695 9287 Acked Write 8 Sent Done Done false
31696 9288 Acked Write 6 Sent Done Done false
31697 9289 Acked Read 9 Sent Done Done false
31698 9290 Acked Flush 0 Sent Done Done false
31699 9291 Acked Write 6 Sent Done Done false
31700 9292 Acked Read 8 Sent Done Done false
31701 9293 Acked Read 4 Sent Done Done false
31702 9294 Acked Read 9 Sent Done Done false
31703 9295 Acked Read 3 Sent Done Done false
31704 9296 Acked Write 3 Sent Done Done false
31705 9297 Acked Read 7 Sent Done Done false
31706 9298 Acked Flush 0 Sent Done Done false
31707 9299 Acked Write 10 Sent Done Done false
31708 9300 Acked Read 8 Sent Done Done false
31709 9301 Acked Read 9 Sent Done Done false
31710 9302 Acked Flush 0 Sent Done Done false
31711 9303 Acked Write 4 Sent Done Done false
31712 9304 Acked Read 7 Sent Done Done false
31713 9305 Acked Read 10 Sent Done Done false
31714 9306 Acked Read 4 Sent Done Done false
31715 9307 Acked Read 6 Sent Done Done false
31716 9308 Acked Write 1 Sent Done Done false
31717 9309 Acked Write 5 Sent Done Done false
31718 9310 Acked Write 9 Sent Done Done false
31719 9311 Acked Write 1 Sent Done Done false
31720 9312 Acked Write 5 Sent Done Done false
31721 9313 Acked Write 5 Sent Done Done false
31722 9314 Acked Write 10 Sent Done Done false
31723 9315 Acked Read 10 Sent Done Done false
31724 9316 Acked Read 2 Sent Done Done false
31725 9317 Acked Flush 0 Sent Done Done false
31726 9318 Acked Flush 0 Sent Done Done false
31727 9319 Acked Read 6 Sent Done Done false
31728 9320 Acked Write 9 Sent Done Done false
31729 9321 Acked Read 9 Sent Done Done false
31730 9322 Acked Read 1 Sent Done Done false
31731 9323 Acked Write 1 Sent Done Done false
31732 9324 Acked Write 5 Sent Done Done false
31733 9325 Acked Read 4 Sent Done Done false
31734 9326 Acked Read 8 Sent Done Done false
31735 9327 Acked Read 7 Sent Done Done false
31736 9328 Acked Read 8 Sent Done Done false
31737 9329 Acked Write 7 Sent Done Done false
31738 9330 Acked Flush 0 Sent Done Done false
31739 9331 Acked Read 6 Sent Done Done false
31740 9332 Acked Read 8 Sent Done Done false
31741 9333 Acked Read 7 Sent Done Done false
31742 9334 Acked Write 2 Sent Done Done false
31743 9335 Acked Write 1 Sent Done Done false
31744 9336 Acked Read 7 Sent Done Done false
31745 9337 Acked Flush 0 Sent Done Done false
31746 9338 Acked Write 4 Sent Done Done false
31747 9339 Acked Read 9 Sent Done Done false
31748 9340 Acked Flush 0 Sent Done Done false
31749 9341 Acked Write 2 Sent Done Done false
31750 9342 Acked Read 2 Sent Done Done false
31751 9343 Acked Write 5 Sent Done Done false
31752 9344 Acked Write 5 Sent Done Done false
31753 9345 Acked Read 2 Sent Done Done false
31754 9346 Acked Read 6 Sent Done Done false
31755 9347 Acked Read 8 Sent Done Done false
31756 9348 Acked Read 6 Sent Done Done false
31757 9349 Acked Read 5 Sent Done Done false
31758 9350 Acked Read 3 Sent Done Done false
31759 9351 Acked Flush 0 Sent Done Done false
31760 9352 Acked Read 7 Sent Done Done false
31761 9353 Acked Read 1 Sent Done Done false
31762 9354 Acked Write 5 Sent Done Done false
31763 9355 Acked Write 9 Sent Done Done false
31764 9356 Acked Read 1 Sent Done Done false
31765 9357 Acked Write 1 Sent Done Done false
31766 9358 Acked Write 10 Sent Done Done false
31767 9359 Acked Write 3 Sent Done Done false
31768 9360 Acked Read 6 Sent Done Done false
31769 9361 Acked Write 8 Sent Done Done false
31770 9362 Acked Flush 0 Sent Done Done false
31771 9363 Acked Read 4 Sent Done Done false
31772 9364 Acked Read 4 Sent Done Done false
31773 9365 Acked Write 9 Sent Done Done false
31774 9366 Acked Write 3 Sent Done Done false
31775 9367 Acked Read 7 Sent Done Done false
31776 9368 Acked Write 8 Sent Done Done false
31777 9369 Acked Read 3 Sent Done Done false
31778 9370 Acked Write 9 Sent Done Done false
31779 9371 Acked Write 1 Sent Done Done false
31780 9372 Acked Read 10 Sent Done Done false
31781 9373 Acked Read 7 Sent Done Done false
31782 9374 Acked Flush 0 Sent Done Done false
31783 9375 Acked Read 5 Sent Done Done false
31784 9376 Acked Write 10 Sent Done Done false
31785 9377 Acked Flush 0 Sent Done Done false
31786 9378 Acked Write 7 Sent Done Done false
31787 9379 Acked Read 4 Sent Done Done false
31788 9380 Acked Write 10 Sent Done Done false
31789 9381 Acked Write 4 Sent Done Done false
31790 9382 Acked Read 1 Sent Done Done false
31791 9383 Acked Write 5 Sent Done Done false
31792 9384 Acked Write 1 Sent Done Done false
31793 9385 Acked Read 6 Sent Done Done false
31794 9386 Acked Read 2 Sent Done Done false
31795 9387 Acked Write 6 Sent Done Done false
31796 9388 Acked Read 2 Sent Done Done false
31797 9389 Acked Write 3 Sent Done Done false
31798 9390 Acked Write 8 Sent Done Done false
31799 9391 Acked Write 10 Sent Done Done false
31800 9392 Acked Read 1 Sent Done Done false
31801 9393 Acked Read 6 Sent Done Done false
31802 9394 Acked Write 2 Sent Done Done false
31803 9395 Acked Flush 0 Sent Done Done false
31804 9396 Acked Read 2 Sent Done Done false
31805 9397 Acked Write 1 Sent Done Done false
31806 9398 Acked Write 2 Sent Done Done false
31807 9399 Acked Write 9 Sent Done Done false
31808 9400 Acked Write 3 Sent Done Done false
31809 9401 Acked Read 4 Sent Done Done false
31810 9402 Acked Write 4 Sent Done Done false
31811 9403 Acked Write 4 Sent Done Done false
31812 9404 Acked Read 8 Sent Done Done false
31813 9405 Acked Flush 0 Sent Done Done false
31814 9406 Acked Write 9 Sent Done Done false
31815 9407 Acked Read 8 Sent Done Done false
31816 9408 Acked Write 6 Sent Done Done false
31817 9409 Acked Write 4 Sent Done Done false
31818 9410 Acked Read 10 Sent Done Done false
31819 9411 Acked Read 3 Sent Done Done false
31820 9412 Acked Write 2 Sent Done Done false
31821 9413 Acked Read 3 Sent Done Done false
31822 9414 Acked Write 10 Sent Done Done false
31823 9415 Acked Write 5 Sent Done Done false
31824 9416 Acked Read 7 Sent Done Done false
31825 9417 Acked Flush 0 Sent Done Done false
31826 9418 Acked Write 9 Sent Done Done false
31827 9419 Acked Read 4 Sent Done Done false
31828 9420 Acked Read 2 Sent Done Done false
31829 9421 Acked Read 7 Sent Done Done false
31830 9422 Acked Read 1 Sent Done Done false
31831 9423 Acked Read 10 Sent Done Done false
31832 9424 Acked Write 5 Sent Done Done false
31833 9425 Acked Read 10 Sent Done Done false
31834 9426 Acked Read 9 Sent Done Done false
31835 9427 Acked Write 9 Sent Done Done false
31836 9428 Acked Write 10 Sent Done Done false
31837 9429 Acked Write 1 Sent Done Done false
31838 9430 Acked Write 2 Sent Done Done false
31839 9431 Acked Read 3 Sent Done Done false
31840 9432 Acked Read 1 Sent Done Done false
31841 9433 Acked Read 1 Sent Done Done false
31842 9434 Acked Write 3 Sent Done Done false
31843 9435 Acked Write 1 Sent Done Done false
31844 9436 Acked Read 2 Sent Done Done false
31845 9437 Acked Write 5 Sent Done Done false
31846 9438 Acked Write 4 Sent Done Done false
31847 9439 Acked Write 7 Sent Done Done false
31848 9440 Acked Write 10 Sent Done Done false
31849 9441 Acked Write 5 Sent Done Done false
31850 9442 Acked Write 4 Sent Done Done false
31851 9443 Acked Read 8 Sent Done Done false
31852 9444 Acked Flush 0 Sent Done Done false
31853 9445 Acked Write 2 Sent Done Done false
31854 9446 Acked Read 1 Sent Done Done false
31855 9447 Acked Write 5 Sent Done Done false
31856 9448 Acked Read 1 Sent Done Done false
31857 9449 Acked Write 3 Sent Done Done false
31858 9450 Acked Read 2 Sent Done Done false
31859 9451 Acked Write 10 Sent Done Done false
31860 9452 Acked Write 10 Sent Done Done false
31861 9453 Acked Write 10 Sent Done Done false
31862 9454 Acked Write 5 Sent Done Done false
31863 9455 Acked Flush 0 Sent Done Done false
31864 9456 Acked Write 5 Sent Done Done false
31865 9457 Acked Read 1 Sent Done Done false
31866 9458 Acked Read 1 Sent Done Done false
31867 9459 Acked Write 6 Sent Done Done false
31868 9460 Acked Read 2 Sent Done Done false
31869 9461 Acked Flush 0 Sent Done Done false
31870 STATES DS:0 DS:1 DS:2 TOTAL
31871 Sent 303 0 0 303
31872 Done 0 303 303 606
31873 Skipped 0 0 0 0
31874 Error 0 0 0 0
31875 Last Flush: 9158 9461 9461
31876 Downstairs last five completed: 9158 9157 9156 9155 9154
31877 Upstairs last five completed: 9461 9460 9459 9458 9457
31878 CLIENT: Up:0 ds:303 act:2
318792024-11-01T16:57:10.397ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
318802024-11-01T16:57:10.397ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
318812024-11-01T16:57:10.397ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
318822024-11-01T16:57:10.397ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
318832024-11-01T16:57:10.397ZINFOcrucible: client task is exiting = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
31884 [8810] got start action
31885 [8810] I am going to restart
31886 Starting downstairs at port 8810
31887 Make output file at "/tmp/dsc/downstairs-8810.txt"
31888 [8810][0] reports Starting
31889 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1226)
31890 [8810][0] reports Running
31891 ----------------------------------------------------------------
31892 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
31893 States: Offline Active Active
31894 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31895 9159 Acked Read 9 Sent Done Done false
31896 9160 Acked Write 7 Sent Done Done false
31897 9161 Acked Flush 0 Sent Done Done false
31898 9162 Acked Read 8 Sent Done Done false
31899 9163 Acked Write 5 Sent Done Done false
31900 9164 Acked Read 2 Sent Done Done false
31901 9165 Acked Flush 0 Sent Done Done false
31902 9166 Acked Write 4 Sent Done Done false
31903 9167 Acked Write 3 Sent Done Done false
31904 9168 Acked Read 5 Sent Done Done false
31905 9169 Acked Read 1 Sent Done Done false
31906 9170 Acked Flush 0 Sent Done Done false
31907 9171 Acked Read 3 Sent Done Done false
31908 9172 Acked Write 6 Sent Done Done false
31909 9173 Acked Read 3 Sent Done Done false
31910 9174 Acked Write 10 Sent Done Done false
31911 9175 Acked Read 7 Sent Done Done false
31912 9176 Acked Write 2 Sent Done Done false
31913 9177 Acked Write 7 Sent Done Done false
31914 9178 Acked Write 3 Sent Done Done false
31915 9179 Acked Write 3 Sent Done Done false
31916 9180 Acked Read 10 Sent Done Done false
31917 9181 Acked Write 8 Sent Done Done false
31918 9182 Acked Flush 0 Sent Done Done false
31919 9183 Acked Write 1 Sent Done Done false
31920 9184 Acked Read 8 Sent Done Done false
31921 9185 Acked Read 3 Sent Done Done false
31922 9186 Acked Read 3 Sent Done Done false
31923 9187 Acked Read 4 Sent Done Done false
31924 9188 Acked Read 9 Sent Done Done false
31925 9189 Acked Read 2 Sent Done Done false
31926 9190 Acked Read 1 Sent Done Done false
31927 9191 Acked Write 6 Sent Done Done false
31928 9192 Acked Flush 0 Sent Done Done false
31929 9193 Acked Read 8 Sent Done Done false
31930 9194 Acked Read 2 Sent Done Done false
31931 9195 Acked Read 5 Sent Done Done false
31932 9196 Acked Read 9 Sent Done Done false
31933 9197 Acked Read 5 Sent Done Done false
31934 9198 Acked Read 3 Sent Done Done false
31935 9199 Acked Write 3 Sent Done Done false
31936 9200 Acked Read 5 Sent Done Done false
31937 9201 Acked Flush 0 Sent Done Done false
31938 9202 Acked Write 10 Sent Done Done false
31939 9203 Acked Read 8 Sent Done Done false
31940 9204 Acked Read 7 Sent Done Done false
31941 9205 Acked Write 6 Sent Done Done false
31942 9206 Acked Write 9 Sent Done Done false
31943 9207 Acked Read 8 Sent Done Done false
31944 9208 Acked Read 1 Sent Done Done false
31945 9209 Acked Flush 0 Sent Done Done false
31946 9210 Acked Flush 0 Sent Done Done false
31947 9211 Acked Read 5 Sent Done Done false
31948 9212 Acked Write 6 Sent Done Done false
31949 9213 Acked Write 8 Sent Done Done false
31950 9214 Acked Read 8 Sent Done Done false
31951 9215 Acked Read 8 Sent Done Done false
31952 9216 Acked Read 4 Sent Done Done false
31953 9217 Acked Write 7 Sent Done Done false
31954 9218 Acked Write 7 Sent Done Done false
31955 9219 Acked Read 10 Sent Done Done false
31956 9220 Acked Read 7 Sent Done Done false
31957 9221 Acked Read 2 Sent Done Done false
31958 9222 Acked Write 5 Sent Done Done false
31959 9223 Acked Read 6 Sent Done Done false
31960 9224 Acked Write 6 Sent Done Done false
31961 9225 Acked Read 3 Sent Done Done false
31962 9226 Acked Flush 0 Sent Done Done false
31963 9227 Acked Write 4 Sent Done Done false
31964 9228 Acked Write 6 Sent Done Done false
31965 9229 Acked Read 3 Sent Done Done false
31966 9230 Acked Write 9 Sent Done Done false
31967 9231 Acked Write 7 Sent Done Done false
31968 9232 Acked Read 8 Sent Done Done false
31969 9233 Acked Read 2 Sent Done Done false
31970 9234 Acked Write 9 Sent Done Done false
31971 9235 Acked Read 3 Sent Done Done false
31972 9236 Acked Write 2 Sent Done Done false
31973 9237 Acked Read 9 Sent Done Done false
31974 9238 Acked Flush 0 Sent Done Done false
31975 9239 Acked Flush 0 Sent Done Done false
31976 9240 Acked Write 4 Sent Done Done false
31977 9241 Acked Read 6 Sent Done Done false
31978 9242 Acked Read 4 Sent Done Done false
31979 9243 Acked Read 1 Sent Done Done false
31980 9244 Acked Read 9 Sent Done Done false
31981 9245 Acked Read 10 Sent Done Done false
31982 9246 Acked Write 4 Sent Done Done false
31983 9247 Acked Read 4 Sent Done Done false
31984 9248 Acked Read 8 Sent Done Done false
31985 9249 Acked Write 7 Sent Done Done false
31986 9250 Acked Read 8 Sent Done Done false
31987 9251 Acked Write 5 Sent Done Done false
31988 9252 Acked Write 4 Sent Done Done false
31989 9253 Acked Write 4 Sent Done Done false
31990 9254 Acked Read 6 Sent Done Done false
31991 9255 Acked Read 5 Sent Done Done false
31992 9256 Acked Read 4 Sent Done Done false
31993 9257 Acked Write 6 Sent Done Done false
31994 9258 Acked Write 2 Sent Done Done false
31995 9259 Acked Read 3 Sent Done Done false
31996 9260 Acked Write 5 Sent Done Done false
31997 9261 Acked Flush 0 Sent Done Done false
31998 9262 Acked Read 3 Sent Done Done false
31999 9263 Acked Write 3 Sent Done Done false
32000 9264 Acked Read 9 Sent Done Done false
32001 9265 Acked Read 4 Sent Done Done false
32002 9266 Acked Read 3 Sent Done Done false
32003 9267 Acked Read 1 Sent Done Done false
32004 9268 Acked Write 5 Sent Done Done false
32005 9269 Acked Read 5 Sent Done Done false
32006 9270 Acked Read 1 Sent Done Done false
32007 9271 Acked Read 8 Sent Done Done false
32008 9272 Acked Read 9 Sent Done Done false
32009 9273 Acked Read 1 Sent Done Done false
32010 9274 Acked Read 5 Sent Done Done false
32011 9275 Acked Read 8 Sent Done Done false
32012 9276 Acked Flush 0 Sent Done Done false
32013 9277 Acked Write 4 Sent Done Done false
32014 9278 Acked Read 6 Sent Done Done false
32015 9279 Acked Write 6 Sent Done Done false
32016 9280 Acked Write 3 Sent Done Done false
32017 9281 Acked Read 6 Sent Done Done false
32018 9282 Acked Read 4 Sent Done Done false
32019 9283 Acked Read 10 Sent Done Done false
32020 9284 Acked Read 2 Sent Done Done false
32021 9285 Acked Write 8 Sent Done Done false
32022 9286 Acked Write 5 Sent Done Done false
32023 9287 Acked Write 8 Sent Done Done false
32024 9288 Acked Write 6 Sent Done Done false
32025 9289 Acked Read 9 Sent Done Done false
32026 9290 Acked Flush 0 Sent Done Done false
32027 9291 Acked Write 6 Sent Done Done false
32028 9292 Acked Read 8 Sent Done Done false
32029 9293 Acked Read 4 Sent Done Done false
32030 9294 Acked Read 9 Sent Done Done false
32031 9295 Acked Read 3 Sent Done Done false
32032 9296 Acked Write 3 Sent Done Done false
32033 9297 Acked Read 7 Sent Done Done false
32034 9298 Acked Flush 0 Sent Done Done false
32035 9299 Acked Write 10 Sent Done Done false
32036 9300 Acked Read 8 Sent Done Done false
32037 9301 Acked Read 9 Sent Done Done false
32038 9302 Acked Flush 0 Sent Done Done false
32039 9303 Acked Write 4 Sent Done Done false
32040 9304 Acked Read 7 Sent Done Done false
32041 9305 Acked Read 10 Sent Done Done false
32042 9306 Acked Read 4 Sent Done Done false
32043 9307 Acked Read 6 Sent Done Done false
32044 9308 Acked Write 1 Sent Done Done false
32045 9309 Acked Write 5 Sent Done Done false
32046 9310 Acked Write 9 Sent Done Done false
32047 9311 Acked Write 1 Sent Done Done false
32048 9312 Acked Write 5 Sent Done Done false
32049 9313 Acked Write 5 Sent Done Done false
32050 9314 Acked Write 10 Sent Done Done false
32051 9315 Acked Read 10 Sent Done Done false
32052 9316 Acked Read 2 Sent Done Done false
32053 9317 Acked Flush 0 Sent Done Done false
32054 9318 Acked Flush 0 Sent Done Done false
32055 9319 Acked Read 6 Sent Done Done false
32056 9320 Acked Write 9 Sent Done Done false
32057 9321 Acked Read 9 Sent Done Done false
32058 9322 Acked Read 1 Sent Done Done false
32059 9323 Acked Write 1 Sent Done Done false
32060 9324 Acked Write 5 Sent Done Done false
32061 9325 Acked Read 4 Sent Done Done false
32062 9326 Acked Read 8 Sent Done Done false
32063 9327 Acked Read 7 Sent Done Done false
32064 9328 Acked Read 8 Sent Done Done false
32065 9329 Acked Write 7 Sent Done Done false
32066 9330 Acked Flush 0 Sent Done Done false
32067 9331 Acked Read 6 Sent Done Done false
32068 9332 Acked Read 8 Sent Done Done false
32069 9333 Acked Read 7 Sent Done Done false
32070 9334 Acked Write 2 Sent Done Done false
32071 9335 Acked Write 1 Sent Done Done false
32072 9336 Acked Read 7 Sent Done Done false
32073 9337 Acked Flush 0 Sent Done Done false
32074 9338 Acked Write 4 Sent Done Done false
32075 9339 Acked Read 9 Sent Done Done false
32076 9340 Acked Flush 0 Sent Done Done false
32077 9341 Acked Write 2 Sent Done Done false
32078 9342 Acked Read 2 Sent Done Done false
32079 9343 Acked Write 5 Sent Done Done false
32080 9344 Acked Write 5 Sent Done Done false
32081 9345 Acked Read 2 Sent Done Done false
32082 9346 Acked Read 6 Sent Done Done false
32083 9347 Acked Read 8 Sent Done Done false
32084 9348 Acked Read 6 Sent Done Done false
32085 9349 Acked Read 5 Sent Done Done false
32086 9350 Acked Read 3 Sent Done Done false
32087 9351 Acked Flush 0 Sent Done Done false
32088 9352 Acked Read 7 Sent Done Done false
32089 9353 Acked Read 1 Sent Done Done false
32090 9354 Acked Write 5 Sent Done Done false
32091 9355 Acked Write 9 Sent Done Done false
32092 9356 Acked Read 1 Sent Done Done false
32093 9357 Acked Write 1 Sent Done Done false
32094 9358 Acked Write 10 Sent Done Done false
32095 9359 Acked Write 3 Sent Done Done false
32096 9360 Acked Read 6 Sent Done Done false
32097 9361 Acked Write 8 Sent Done Done false
32098 9362 Acked Flush 0 Sent Done Done false
32099 9363 Acked Read 4 Sent Done Done false
32100 9364 Acked Read 4 Sent Done Done false
32101 9365 Acked Write 9 Sent Done Done false
32102 9366 Acked Write 3 Sent Done Done false
32103 9367 Acked Read 7 Sent Done Done false
32104 9368 Acked Write 8 Sent Done Done false
32105 9369 Acked Read 3 Sent Done Done false
32106 9370 Acked Write 9 Sent Done Done false
32107 9371 Acked Write 1 Sent Done Done false
32108 9372 Acked Read 10 Sent Done Done false
32109 9373 Acked Read 7 Sent Done Done false
32110 9374 Acked Flush 0 Sent Done Done false
32111 9375 Acked Read 5 Sent Done Done false
32112 9376 Acked Write 10 Sent Done Done false
32113 9377 Acked Flush 0 Sent Done Done false
32114 9378 Acked Write 7 Sent Done Done false
32115 9379 Acked Read 4 Sent Done Done false
32116 9380 Acked Write 10 Sent Done Done false
32117 9381 Acked Write 4 Sent Done Done false
32118 9382 Acked Read 1 Sent Done Done false
32119 9383 Acked Write 5 Sent Done Done false
32120 9384 Acked Write 1 Sent Done Done false
32121 9385 Acked Read 6 Sent Done Done false
32122 9386 Acked Read 2 Sent Done Done false
32123 9387 Acked Write 6 Sent Done Done false
32124 9388 Acked Read 2 Sent Done Done false
32125 9389 Acked Write 3 Sent Done Done false
32126 9390 Acked Write 8 Sent Done Done false
32127 9391 Acked Write 10 Sent Done Done false
32128 9392 Acked Read 1 Sent Done Done false
32129 9393 Acked Read 6 Sent Done Done false
32130 9394 Acked Write 2 Sent Done Done false
32131 9395 Acked Flush 0 Sent Done Done false
32132 9396 Acked Read 2 Sent Done Done false
32133 9397 Acked Write 1 Sent Done Done false
32134 9398 Acked Write 2 Sent Done Done false
32135 9399 Acked Write 9 Sent Done Done false
32136 9400 Acked Write 3 Sent Done Done false
32137 9401 Acked Read 4 Sent Done Done false
32138 9402 Acked Write 4 Sent Done Done false
32139 9403 Acked Write 4 Sent Done Done false
32140 9404 Acked Read 8 Sent Done Done false
32141 9405 Acked Flush 0 Sent Done Done false
32142 9406 Acked Write 9 Sent Done Done false
32143 9407 Acked Read 8 Sent Done Done false
32144 9408 Acked Write 6 Sent Done Done false
32145 9409 Acked Write 4 Sent Done Done false
32146 9410 Acked Read 10 Sent Done Done false
32147 9411 Acked Read 3 Sent Done Done false
32148 9412 Acked Write 2 Sent Done Done false
32149 9413 Acked Read 3 Sent Done Done false
32150 9414 Acked Write 10 Sent Done Done false
32151 9415 Acked Write 5 Sent Done Done false
32152 9416 Acked Read 7 Sent Done Done false
32153 9417 Acked Flush 0 Sent Done Done false
32154 9418 Acked Write 9 Sent Done Done false
32155 9419 Acked Read 4 Sent Done Done false
32156 9420 Acked Read 2 Sent Done Done false
32157 9421 Acked Read 7 Sent Done Done false
32158 9422 Acked Read 1 Sent Done Done false
32159 9423 Acked Read 10 Sent Done Done false
32160 9424 Acked Write 5 Sent Done Done false
32161 9425 Acked Read 10 Sent Done Done false
32162 9426 Acked Read 9 Sent Done Done false
32163 9427 Acked Write 9 Sent Done Done false
32164 9428 Acked Write 10 Sent Done Done false
32165 9429 Acked Write 1 Sent Done Done false
32166 9430 Acked Write 2 Sent Done Done false
32167 9431 Acked Read 3 Sent Done Done false
32168 9432 Acked Read 1 Sent Done Done false
32169 9433 Acked Read 1 Sent Done Done false
32170 9434 Acked Write 3 Sent Done Done false
32171 9435 Acked Write 1 Sent Done Done false
32172 9436 Acked Read 2 Sent Done Done false
32173 9437 Acked Write 5 Sent Done Done false
32174 9438 Acked Write 4 Sent Done Done false
32175 9439 Acked Write 7 Sent Done Done false
32176 9440 Acked Write 10 Sent Done Done false
32177 9441 Acked Write 5 Sent Done Done false
32178 9442 Acked Write 4 Sent Done Done false
32179 9443 Acked Read 8 Sent Done Done false
32180 9444 Acked Flush 0 Sent Done Done false
32181 9445 Acked Write 2 Sent Done Done false
32182 9446 Acked Read 1 Sent Done Done false
32183 9447 Acked Write 5 Sent Done Done false
32184 9448 Acked Read 1 Sent Done Done false
32185 9449 Acked Write 3 Sent Done Done false
32186 9450 Acked Read 2 Sent Done Done false
32187 9451 Acked Write 10 Sent Done Done false
32188 9452 Acked Write 10 Sent Done Done false
32189 9453 Acked Write 10 Sent Done Done false
32190 9454 Acked Write 5 Sent Done Done false
32191 9455 Acked Flush 0 Sent Done Done false
32192 9456 Acked Write 5 Sent Done Done false
32193 9457 Acked Read 1 Sent Done Done false
32194 9458 Acked Read 1 Sent Done Done false
32195 9459 Acked Write 6 Sent Done Done false
32196 9460 Acked Read 2 Sent Done Done false
32197 9461 Acked Flush 0 Sent Done Done false
32198 STATES DS:0 DS:1 DS:2 TOTAL
32199 Sent 303 0 0 303
32200 Done 0 303 303 606
32201 Skipped 0 0 0 0
32202 Error 0 0 0 0
32203 Last Flush: 9158 9461 9461
32204 Downstairs last five completed: 9158 9157 9156 9155 9154
32205 Upstairs last five completed: 9461 9460 9459 9458 9457
32206 CLIENT: Up:0 ds:303 act:2
32207 ----------------------------------------------------------------
32208 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
32209 States: Offline Active Active
32210 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32211 9159 Acked Read 9 Sent Done Done false
32212 9160 Acked Write 7 Sent Done Done false
32213 9161 Acked Flush 0 Sent Done Done false
32214 9162 Acked Read 8 Sent Done Done false
32215 9163 Acked Write 5 Sent Done Done false
32216 9164 Acked Read 2 Sent Done Done false
32217 9165 Acked Flush 0 Sent Done Done false
32218 9166 Acked Write 4 Sent Done Done false
32219 9167 Acked Write 3 Sent Done Done false
32220 9168 Acked Read 5 Sent Done Done false
32221 9169 Acked Read 1 Sent Done Done false
32222 9170 Acked Flush 0 Sent Done Done false
32223 9171 Acked Read 3 Sent Done Done false
32224 9172 Acked Write 6 Sent Done Done false
32225 9173 Acked Read 3 Sent Done Done false
32226 9174 Acked Write 10 Sent Done Done false
32227 9175 Acked Read 7 Sent Done Done false
32228 9176 Acked Write 2 Sent Done Done false
32229 9177 Acked Write 7 Sent Done Done false
32230 9178 Acked Write 3 Sent Done Done false
32231 9179 Acked Write 3 Sent Done Done false
32232 9180 Acked Read 10 Sent Done Done false
32233 9181 Acked Write 8 Sent Done Done false
32234 9182 Acked Flush 0 Sent Done Done false
32235 9183 Acked Write 1 Sent Done Done false
32236 9184 Acked Read 8 Sent Done Done false
32237 9185 Acked Read 3 Sent Done Done false
32238 9186 Acked Read 3 Sent Done Done false
32239 9187 Acked Read 4 Sent Done Done false
32240 9188 Acked Read 9 Sent Done Done false
32241 9189 Acked Read 2 Sent Done Done false
32242 9190 Acked Read 1 Sent Done Done false
32243 9191 Acked Write 6 Sent Done Done false
32244 9192 Acked Flush 0 Sent Done Done false
32245 9193 Acked Read 8 Sent Done Done false
32246 9194 Acked Read 2 Sent Done Done false
32247 9195 Acked Read 5 Sent Done Done false
32248 9196 Acked Read 9 Sent Done Done false
32249 9197 Acked Read 5 Sent Done Done false
32250 9198 Acked Read 3 Sent Done Done false
32251 9199 Acked Write 3 Sent Done Done false
32252 9200 Acked Read 5 Sent Done Done false
32253 9201 Acked Flush 0 Sent Done Done false
32254 9202 Acked Write 10 Sent Done Done false
32255 9203 Acked Read 8 Sent Done Done false
32256 9204 Acked Read 7 Sent Done Done false
32257 9205 Acked Write 6 Sent Done Done false
32258 9206 Acked Write 9 Sent Done Done false
32259 9207 Acked Read 8 Sent Done Done false
32260 9208 Acked Read 1 Sent Done Done false
32261 9209 Acked Flush 0 Sent Done Done false
32262 9210 Acked Flush 0 Sent Done Done false
32263 9211 Acked Read 5 Sent Done Done false
32264 9212 Acked Write 6 Sent Done Done false
32265 9213 Acked Write 8 Sent Done Done false
32266 9214 Acked Read 8 Sent Done Done false
32267 9215 Acked Read 8 Sent Done Done false
32268 9216 Acked Read 4 Sent Done Done false
32269 9217 Acked Write 7 Sent Done Done false
32270 9218 Acked Write 7 Sent Done Done false
32271 9219 Acked Read 10 Sent Done Done false
32272 9220 Acked Read 7 Sent Done Done false
32273 9221 Acked Read 2 Sent Done Done false
32274 9222 Acked Write 5 Sent Done Done false
32275 9223 Acked Read 6 Sent Done Done false
32276 9224 Acked Write 6 Sent Done Done false
32277 9225 Acked Read 3 Sent Done Done false
32278 9226 Acked Flush 0 Sent Done Done false
32279 9227 Acked Write 4 Sent Done Done false
32280 9228 Acked Write 6 Sent Done Done false
32281 9229 Acked Read 3 Sent Done Done false
32282 9230 Acked Write 9 Sent Done Done false
32283 9231 Acked Write 7 Sent Done Done false
32284 9232 Acked Read 8 Sent Done Done false
32285 9233 Acked Read 2 Sent Done Done false
32286 9234 Acked Write 9 Sent Done Done false
32287 9235 Acked Read 3 Sent Done Done false
32288 9236 Acked Write 2 Sent Done Done false
32289 9237 Acked Read 9 Sent Done Done false
32290 9238 Acked Flush 0 Sent Done Done false
32291 9239 Acked Flush 0 Sent Done Done false
32292 9240 Acked Write 4 Sent Done Done false
32293 9241 Acked Read 6 Sent Done Done false
32294 9242 Acked Read 4 Sent Done Done false
32295 9243 Acked Read 1 Sent Done Done false
32296 9244 Acked Read 9 Sent Done Done false
32297 9245 Acked Read 10 Sent Done Done false
32298 9246 Acked Write 4 Sent Done Done false
32299 9247 Acked Read 4 Sent Done Done false
32300 9248 Acked Read 8 Sent Done Done false
32301 9249 Acked Write 7 Sent Done Done false
32302 9250 Acked Read 8 Sent Done Done false
32303 9251 Acked Write 5 Sent Done Done false
32304 9252 Acked Write 4 Sent Done Done false
32305 9253 Acked Write 4 Sent Done Done false
32306 9254 Acked Read 6 Sent Done Done false
32307 9255 Acked Read 5 Sent Done Done false
32308 9256 Acked Read 4 Sent Done Done false
32309 9257 Acked Write 6 Sent Done Done false
32310 9258 Acked Write 2 Sent Done Done false
32311 9259 Acked Read 3 Sent Done Done false
32312 9260 Acked Write 5 Sent Done Done false
32313 9261 Acked Flush 0 Sent Done Done false
32314 9262 Acked Read 3 Sent Done Done false
32315 9263 Acked Write 3 Sent Done Done false
32316 9264 Acked Read 9 Sent Done Done false
32317 9265 Acked Read 4 Sent Done Done false
32318 9266 Acked Read 3 Sent Done Done false
32319 9267 Acked Read 1 Sent Done Done false
32320 9268 Acked Write 5 Sent Done Done false
32321 9269 Acked Read 5 Sent Done Done false
32322 9270 Acked Read 1 Sent Done Done false
32323 9271 Acked Read 8 Sent Done Done false
32324 9272 Acked Read 9 Sent Done Done false
32325 9273 Acked Read 1 Sent Done Done false
32326 9274 Acked Read 5 Sent Done Done false
32327 9275 Acked Read 8 Sent Done Done false
32328 9276 Acked Flush 0 Sent Done Done false
32329 9277 Acked Write 4 Sent Done Done false
32330 9278 Acked Read 6 Sent Done Done false
32331 9279 Acked Write 6 Sent Done Done false
32332 9280 Acked Write 3 Sent Done Done false
32333 9281 Acked Read 6 Sent Done Done false
32334 9282 Acked Read 4 Sent Done Done false
32335 9283 Acked Read 10 Sent Done Done false
32336 9284 Acked Read 2 Sent Done Done false
32337 9285 Acked Write 8 Sent Done Done false
32338 9286 Acked Write 5 Sent Done Done false
32339 9287 Acked Write 8 Sent Done Done false
32340 9288 Acked Write 6 Sent Done Done false
32341 9289 Acked Read 9 Sent Done Done false
32342 9290 Acked Flush 0 Sent Done Done false
32343 9291 Acked Write 6 Sent Done Done false
32344 9292 Acked Read 8 Sent Done Done false
32345 9293 Acked Read 4 Sent Done Done false
32346 9294 Acked Read 9 Sent Done Done false
32347 9295 Acked Read 3 Sent Done Done false
32348 9296 Acked Write 3 Sent Done Done false
32349 9297 Acked Read 7 Sent Done Done false
32350 9298 Acked Flush 0 Sent Done Done false
32351 9299 Acked Write 10 Sent Done Done false
32352 9300 Acked Read 8 Sent Done Done false
32353 9301 Acked Read 9 Sent Done Done false
32354 9302 Acked Flush 0 Sent Done Done false
32355 9303 Acked Write 4 Sent Done Done false
32356 9304 Acked Read 7 Sent Done Done false
32357 9305 Acked Read 10 Sent Done Done false
32358 9306 Acked Read 4 Sent Done Done false
32359 9307 Acked Read 6 Sent Done Done false
32360 9308 Acked Write 1 Sent Done Done false
32361 9309 Acked Write 5 Sent Done Done false
32362 9310 Acked Write 9 Sent Done Done false
32363 9311 Acked Write 1 Sent Done Done false
32364 9312 Acked Write 5 Sent Done Done false
32365 9313 Acked Write 5 Sent Done Done false
32366 9314 Acked Write 10 Sent Done Done false
32367 9315 Acked Read 10 Sent Done Done false
32368 9316 Acked Read 2 Sent Done Done false
32369 9317 Acked Flush 0 Sent Done Done false
32370 9318 Acked Flush 0 Sent Done Done false
32371 9319 Acked Read 6 Sent Done Done false
32372 9320 Acked Write 9 Sent Done Done false
32373 9321 Acked Read 9 Sent Done Done false
32374 9322 Acked Read 1 Sent Done Done false
32375 9323 Acked Write 1 Sent Done Done false
32376 9324 Acked Write 5 Sent Done Done false
32377 9325 Acked Read 4 Sent Done Done false
32378 9326 Acked Read 8 Sent Done Done false
32379 9327 Acked Read 7 Sent Done Done false
32380 9328 Acked Read 8 Sent Done Done false
32381 9329 Acked Write 7 Sent Done Done false
32382 9330 Acked Flush 0 Sent Done Done false
32383 9331 Acked Read 6 Sent Done Done false
32384 9332 Acked Read 8 Sent Done Done false
32385 9333 Acked Read 7 Sent Done Done false
32386 9334 Acked Write 2 Sent Done Done false
32387 9335 Acked Write 1 Sent Done Done false
32388 9336 Acked Read 7 Sent Done Done false
32389 9337 Acked Flush 0 Sent Done Done false
32390 9338 Acked Write 4 Sent Done Done false
32391 9339 Acked Read 9 Sent Done Done false
32392 9340 Acked Flush 0 Sent Done Done false
32393 9341 Acked Write 2 Sent Done Done false
32394 9342 Acked Read 2 Sent Done Done false
32395 9343 Acked Write 5 Sent Done Done false
32396 9344 Acked Write 5 Sent Done Done false
32397 9345 Acked Read 2 Sent Done Done false
32398 9346 Acked Read 6 Sent Done Done false
32399 9347 Acked Read 8 Sent Done Done false
32400 9348 Acked Read 6 Sent Done Done false
32401 9349 Acked Read 5 Sent Done Done false
32402 9350 Acked Read 3 Sent Done Done false
32403 9351 Acked Flush 0 Sent Done Done false
32404 9352 Acked Read 7 Sent Done Done false
32405 9353 Acked Read 1 Sent Done Done false
32406 9354 Acked Write 5 Sent Done Done false
32407 9355 Acked Write 9 Sent Done Done false
32408 9356 Acked Read 1 Sent Done Done false
32409 9357 Acked Write 1 Sent Done Done false
32410 9358 Acked Write 10 Sent Done Done false
32411 9359 Acked Write 3 Sent Done Done false
32412 9360 Acked Read 6 Sent Done Done false
32413 9361 Acked Write 8 Sent Done Done false
32414 9362 Acked Flush 0 Sent Done Done false
32415 9363 Acked Read 4 Sent Done Done false
32416 9364 Acked Read 4 Sent Done Done false
32417 9365 Acked Write 9 Sent Done Done false
32418 9366 Acked Write 3 Sent Done Done false
32419 9367 Acked Read 7 Sent Done Done false
32420 9368 Acked Write 8 Sent Done Done false
32421 9369 Acked Read 3 Sent Done Done false
32422 9370 Acked Write 9 Sent Done Done false
32423 9371 Acked Write 1 Sent Done Done false
32424 9372 Acked Read 10 Sent Done Done false
32425 9373 Acked Read 7 Sent Done Done false
32426 9374 Acked Flush 0 Sent Done Done false
32427 9375 Acked Read 5 Sent Done Done false
32428 9376 Acked Write 10 Sent Done Done false
32429 9377 Acked Flush 0 Sent Done Done false
32430 9378 Acked Write 7 Sent Done Done false
32431 9379 Acked Read 4 Sent Done Done false
32432 9380 Acked Write 10 Sent Done Done false
32433 9381 Acked Write 4 Sent Done Done false
32434 9382 Acked Read 1 Sent Done Done false
32435 9383 Acked Write 5 Sent Done Done false
32436 9384 Acked Write 1 Sent Done Done false
32437 9385 Acked Read 6 Sent Done Done false
32438 9386 Acked Read 2 Sent Done Done false
32439 9387 Acked Write 6 Sent Done Done false
32440 9388 Acked Read 2 Sent Done Done false
32441 9389 Acked Write 3 Sent Done Done false
32442 9390 Acked Write 8 Sent Done Done false
32443 9391 Acked Write 10 Sent Done Done false
32444 9392 Acked Read 1 Sent Done Done false
32445 9393 Acked Read 6 Sent Done Done false
32446 9394 Acked Write 2 Sent Done Done false
32447 9395 Acked Flush 0 Sent Done Done false
32448 9396 Acked Read 2 Sent Done Done false
32449 9397 Acked Write 1 Sent Done Done false
32450 9398 Acked Write 2 Sent Done Done false
32451 9399 Acked Write 9 Sent Done Done false
32452 9400 Acked Write 3 Sent Done Done false
32453 9401 Acked Read 4 Sent Done Done false
32454 9402 Acked Write 4 Sent Done Done false
32455 9403 Acked Write 4 Sent Done Done false
32456 9404 Acked Read 8 Sent Done Done false
32457 9405 Acked Flush 0 Sent Done Done false
32458 9406 Acked Write 9 Sent Done Done false
32459 9407 Acked Read 8 Sent Done Done false
32460 9408 Acked Write 6 Sent Done Done false
32461 9409 Acked Write 4 Sent Done Done false
32462 9410 Acked Read 10 Sent Done Done false
32463 9411 Acked Read 3 Sent Done Done false
32464 9412 Acked Write 2 Sent Done Done false
32465 9413 Acked Read 3 Sent Done Done false
32466 9414 Acked Write 10 Sent Done Done false
32467 9415 Acked Write 5 Sent Done Done false
32468 9416 Acked Read 7 Sent Done Done false
32469 9417 Acked Flush 0 Sent Done Done false
32470 9418 Acked Write 9 Sent Done Done false
32471 9419 Acked Read 4 Sent Done Done false
32472 9420 Acked Read 2 Sent Done Done false
32473 9421 Acked Read 7 Sent Done Done false
32474 9422 Acked Read 1 Sent Done Done false
32475 9423 Acked Read 10 Sent Done Done false
32476 9424 Acked Write 5 Sent Done Done false
32477 9425 Acked Read 10 Sent Done Done false
32478 9426 Acked Read 9 Sent Done Done false
32479 9427 Acked Write 9 Sent Done Done false
32480 9428 Acked Write 10 Sent Done Done false
32481 9429 Acked Write 1 Sent Done Done false
32482 9430 Acked Write 2 Sent Done Done false
32483 9431 Acked Read 3 Sent Done Done false
32484 9432 Acked Read 1 Sent Done Done false
32485 9433 Acked Read 1 Sent Done Done false
32486 9434 Acked Write 3 Sent Done Done false
32487 9435 Acked Write 1 Sent Done Done false
32488 9436 Acked Read 2 Sent Done Done false
32489 9437 Acked Write 5 Sent Done Done false
32490 9438 Acked Write 4 Sent Done Done false
32491 9439 Acked Write 7 Sent Done Done false
32492 9440 Acked Write 10 Sent Done Done false
32493 9441 Acked Write 5 Sent Done Done false
32494 9442 Acked Write 4 Sent Done Done false
32495 9443 Acked Read 8 Sent Done Done false
32496 9444 Acked Flush 0 Sent Done Done false
32497 9445 Acked Write 2 Sent Done Done false
32498 9446 Acked Read 1 Sent Done Done false
32499 9447 Acked Write 5 Sent Done Done false
32500 9448 Acked Read 1 Sent Done Done false
32501 9449 Acked Write 3 Sent Done Done false
32502 9450 Acked Read 2 Sent Done Done false
32503 9451 Acked Write 10 Sent Done Done false
32504 9452 Acked Write 10 Sent Done Done false
32505 9453 Acked Write 10 Sent Done Done false
32506 9454 Acked Write 5 Sent Done Done false
32507 9455 Acked Flush 0 Sent Done Done false
32508 9456 Acked Write 5 Sent Done Done false
32509 9457 Acked Read 1 Sent Done Done false
32510 9458 Acked Read 1 Sent Done Done false
32511 9459 Acked Write 6 Sent Done Done false
32512 9460 Acked Read 2 Sent Done Done false
32513 9461 Acked Flush 0 Sent Done Done false
32514 STATES DS:0 DS:1 DS:2 TOTAL
32515 Sent 303 0 0 303
32516 Done 0 303 303 606
32517 Skipped 0 0 0 0
32518 Error 0 0 0 0
32519 Last Flush: 9158 9461 9461
32520 Downstairs last five completed: 9158 9157 9156 9155 9154
32521 Upstairs last five completed: 9461 9460 9459 9458 9457
32522 CLIENT: Up:0 ds:303 act:2
325232024-11-01T16:57:20.399ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325242024-11-01T16:57:20.399ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46160) = io task client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325252024-11-01T16:57:20.400ZWARNcrucible: version negotiation from state Offline = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325262024-11-01T16:57:20.401ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325272024-11-01T16:57:20.401ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325282024-11-01T16:57:20.401ZINFOcrucible: send last flush ID to this DS: 9158 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325292024-11-01T16:57:20.401ZINFOcrucible: Replied this last flush ID: 9158 = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325302024-11-01T16:57:20.401ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325312024-11-01T16:57:20.401ZINFOcrucible: [0] Transition from Offline to Active = downstairs client = 0 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325322024-11-01T16:57:20.401ZINFOcrucible: [0] client re-new 303 jobs since flush 9158 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325332024-11-01T16:57:20.401ZINFOcrucible: [0] Replayed 303 jobs since flush: 9158 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
32534 ----------------------------------------------------------------
32535 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
32536 Downstairs last five completed: 9461 9460 9459 9458 9457
32537 Upstairs last five completed: 9461 9460 9459 9458 9457
32538 CLIENT: Up:0 ds:0 act:3
32539 Replay: All jobs finished
32540 Main task has work to do, go find it
32541 got dsc Stop(2)
32542 stop 2
32543 [8830] Got stop action so:false kr:false
32544 Nov 01 16:57:20.707 INFO request completed, latency_us: 371, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 0b6898a3-d4e6-49ed-b975-3c60e6f1badc, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
32545 Nov 01 16:57:20.708 INFO request completed, latency_us: 119, response_code: 200, uri: /state/cid/2, method: GET, req_id: 80862304-fb8b-43cf-97d0-ae5acbcbd672, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
325462024-11-01T16:57:20.708ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325472024-11-01T16:57:20.708ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325482024-11-01T16:57:20.708ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325492024-11-01T16:57:20.708ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
325502024-11-01T16:57:20.708ZINFOcrucible: client task is exiting = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
32551 [8830] Exited with: ExitStatus(unix_wait_status(9))
32552 [8830][2] reports Exit
32553 Nov 01 16:57:24.711 INFO request completed, latency_us: 247, response_code: 200, uri: /state/cid/2, method: GET, req_id: e2a8d4d1-49c1-4318-a5d4-4adfa2144cff, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
32554 001/300 Write block 4549 len 28672 data: 5 6 5 5 7 8 7
32555 002/300 Write block 870 len 8192 data: 2 2
32556 003/300 Write block 1205 len 36864 data: 11 11 10 10 9 8 8 8 11
32557 004/300 Flush
32558 005/300 Read block 42 len 12288
32559 006/300 Read block 1990 len 16384
32560 007/300 Write block 515 len 16384 data: 6 8 8 7
32561 008/300 Read block 2721 len 28672
32562 009/300 Read block 4618 len 4096
32563 010/300 Read block 2048 len 28672
32564 011/300 Write block 4516 len 32768 data: 8 9 8 7 6 7 6 8
32565 012/300 Read block 4784 len 4096
32566 013/300 Read block 3693 len 4096
32567 014/300 Write block 3138 len 32768 data: 8 7 7 4 2 3 3 4
32568 015/300 Write block 289 len 32768 data: 4 5 6 6 5 5 6 5
32569 016/300 Read block 3252 len 40960
32570 017/300 Write block 4146 len 40960 data: 7 6 5 5 5 5 4 4 3 2
32571 018/300 Read block 3668 len 28672
32572 019/300 Write block 960 len 8192 data: 2 3
32573 020/300 Write block 1385 len 8192 data: 4 4
32574 021/300 Read block 4074 len 8192
32575 022/300 Read block 2783 len 16384
32576 023/300 Write block 3050 len 36864 data: 2 2 2 3 4 4 4 4 4
32577 024/300 Write block 1173 len 24576 data: 7 7 8 8 8 9
32578 025/300 Write block 2925 len 24576 data: 3 4 4 4 6 7
32579 026/300 Write block 2865 len 28672 data: 4 5 5 7 6 7 8
32580 027/300 Write block 3556 len 24576 data: 6 6 7 7 7 5
32581 028/300 Flush
32582 029/300 Read block 2796 len 16384
32583 030/300 Read block 3725 len 32768
32584 031/300 Read block 131 len 16384
32585 032/300 Read block 1851 len 12288
32586 033/300 Flush
32587 034/300 Read block 3436 len 8192
32588 035/300 Flush
32589 036/300 Write block 3872 len 36864 data: 7 8 7 6 6 4 4 5 6
32590 037/300 Read block 2636 len 24576
32591 038/300 Read block 4833 len 40960
32592 039/300 Read block 27 len 8192
32593 040/300 Read block 2722 len 32768
32594 041/300 Read block 4267 len 20480
32595 042/300 Write block 3288 len 24576 data: 7 7 9 8 6 5
32596 043/300 Write block 619 len 8192 data: 5 5
32597 044/300 Read block 4765 len 24576
32598 045/300 Flush
32599 046/300 Read block 3827 len 36864
32600 047/300 Read block 3966 len 4096
32601 048/300 Read block 3256 len 36864
32602 049/300 Read block 445 len 40960
32603 050/300 Write block 71 len 20480 data: 6 6 5 5 6
32604 051/300 Read block 4479 len 20480
32605 052/300 Flush
32606 053/300 Write block 4792 len 20480 data: 5 5 4 4 4
32607 054/300 Read block 532 len 24576
32608 055/300 Read block 493 len 32768
32609 056/300 Read block 1836 len 4096
32610 057/300 Flush
32611 058/300 Flush
32612 059/300 Write block 2929 len 4096 data: 7
32613 060/300 Read block 3752 len 20480
32614 061/300 Flush
32615 062/300 Write block 1574 len 36864 data: 5 6 6 6 6 4 3 4 4
32616 063/300 Write block 2890 len 4096 data: 10
32617 064/300 Read block 2678 len 12288
32618 065/300 Write block 4973 len 12288 data: 3 5 5
32619 066/300 Write block 4225 len 12288 data: 7 7 7
32620 067/300 Read block 4700 len 16384
32621 068/300 Write block 4923 len 28672 data: 9 8 8 8 6 6 5
32622 069/300 Read block 2199 len 8192
32623 070/300 Read block 4383 len 32768
32624 071/300 Read block 2449 len 28672
32625 072/300 Write block 4914 len 8192 data: 7 8
32626 073/300 Flush
32627 074/300 Flush
32628 075/300 Read block 2396 len 16384
32629 076/300 Read block 3596 len 32768
32630 077/300 Read block 1122 len 12288
32631 078/300 Read block 2473 len 4096
32632 079/300 Read block 55 len 4096
32633 080/300 Write block 279 len 36864 data: 8 6 6 6 5 5 6 8 5
32634 081/300 Write block 2374 len 16384 data: 7 6 5 6
32635 082/300 Write block 3434 len 24576 data: 5 5 4 5 4 5
32636 083/300 Read block 1082 len 16384
32637 084/300 Write block 2627 len 36864 data: 2 2 2 2 3 3 4 4 4
32638 085/300 Write block 4241 len 4096 data: 3
32639 086/300 Read block 4283 len 24576
32640 087/300 Flush
32641 088/300 Write block 516 len 32768 data: 9 9 8 8 8 7 7 7
32642 089/300 Read block 4651 len 24576
32643 090/300 Read block 3774 len 8192
32644 091/300 Write block 3106 len 4096 data: 6
32645 092/300 Read block 323 len 20480
32646 093/300 Write block 2349 len 4096 data: 5
32647 094/300 Flush
32648 095/300 Write block 663 len 16384 data: 7 7 7 5
32649 096/300 Read block 1481 len 4096
32650 097/300 Read block 4978 len 32768
32651 098/300 Read block 1000 len 20480
32652 099/300 Read block 2000 len 40960
32653 100/300 Write block 4521 len 28672 data: 8 7 9 8 10 9 10
32654 101/300 Read block 1991 len 40960
32655 102/300 Write block 89 len 32768 data: 6 8 8 7 8 10 9 10
32656 103/300 Read block 2011 len 12288
32657 104/300 Write block 4270 len 28672 data: 7 6 6 5 4 5 6
32658 105/300 Read block 4555 len 4096
32659 106/300 Read block 2077 len 16384
32660 107/300 Read block 4562 len 36864
32661 108/300 Read block 2134 len 28672
32662 109/300 Write block 2612 len 4096 data: 6
32663 110/300 Write block 2202 len 32768 data: 5 5 5 6 6 6 7 6
32664 111/300 Write block 2124 len 28672 data: 8 7 5 5 4 4 4
32665 112/300 Write block 1959 len 32768 data: 5 5 6 7 7 6 5 5
32666 113/300 Read block 2074 len 32768
32667 114/300 Flush
32668 115/300 Write block 2080 len 28672 data: 4 4 4 6 7 9 11
32669 116/300 Flush
32670 117/300 Read block 4738 len 20480
32671 118/300 Read block 3955 len 28672
32672 119/300 Read block 4808 len 36864
32673 120/300 Read block 399 len 32768
32674 121/300 Write block 4676 len 12288 data: 9 10 8
32675 122/300 Write block 1924 len 32768 data: 5 4 4 5 4 4 3 3
32676 123/300 Read block 3051 len 28672
32677 124/300 Read block 1423 len 36864
32678 125/300 Read block 2418 len 32768
32679 126/300 Write block 153 len 4096 data: 3
32680 127/300 Write block 4073 len 40960 data: 7 8 8 8 8 7 7 6 6 6
32681 128/300 Flush
32682 129/300 Read block 2109 len 20480
32683 130/300 Read block 1306 len 8192
32684 131/300 Read block 3234 len 36864
32685 132/300 Flush
32686 133/300 Read block 283 len 28672
32687 134/300 Read block 398 len 20480
32688 135/300 Read block 2301 len 28672
32689 136/300 Write block 1659 len 32768 data: 4 6 6 9 9 8 6 6
32690 137/300 Write block 415 len 40960 data: 6 4 4 4 3 4 5 5 5 4
32691 138/300 Read block 3508 len 24576
32692 139/300 Flush
32693 140/300 Write block 1694 len 28672 data: 7 7 6 5 5 5 6
32694 141/300 Read block 232 len 32768
32695 142/300 Write block 317 len 8192 data: 3 3
32696 143/300 Read block 463 len 40960
32697 144/300 Read block 1127 len 36864
32698 145/300 Read block 1045 len 36864
32699 146/300 Read block 3840 len 4096
32700 147/300 Write block 2382 len 28672 data: 5 6 5 4 4 3 4
32701 148/300 Write block 3351 len 28672 data: 7 6 6 7 7 6 6
32702 149/300 Write block 716 len 24576 data: 4 3 5 4 4 4
32703 150/300 Read block 3521 len 16384
32704 151/300 Write block 1657 len 24576 data: 4 5 5 7 7 10
32705 152/300 Flush
32706 153/300 Read block 1861 len 12288
32707 154/300 Read block 1078 len 24576
32708 155/300 Read block 4484 len 12288
32709 156/300 Write block 126 len 28672 data: 5 8 6 6 6 4 5
32710 157/300 Write block 4899 len 12288 data: 4 5 5
32711 158/300 Write block 2470 len 36864 data: 4 6 8 10 11 11 11 11 12
32712 159/300 Read block 240 len 40960
32713 160/300 Write block 3299 len 24576 data: 8 10 8 8 5 5
32714 161/300 Flush
32715 162/300 Read block 374 len 24576
32716 163/300 Write block 3341 len 36864 data: 6 5 4 4 6 6 7 8 9
32717 164/300 Read block 4230 len 36864
32718 165/300 Read block 3259 len 32768
32719 166/300 Flush
32720 167/300 Write block 2190 len 20480 data: 7 5 5 4 4
32721 168/300 Flush
32722 169/300 Read block 3406 len 28672
32723 170/300 Write block 2 len 36864 data: 4 5 7 6 5 5 6 4 4
32724 171/300 Write block 4250 len 12288 data: 7 8 6
32725 172/300 Write block 3245 len 32768 data: 4 4 4 4 4 6 7 5
32726 173/300 Write block 1367 len 12288 data: 8 7 7
32727 174/300 Write block 4424 len 4096 data: 5
32728 175/300 Read block 4034 len 12288
32729 176/300 Write block 304 len 4096 data: 8
32730 177/300 Read block 2917 len 36864
32731 178/300 Read block 3813 len 8192
32732 179/300 Read block 1126 len 4096
32733 180/300 Read block 412 len 8192
32734 181/300 Read block 4414 len 36864
32735 182/300 Read block 2165 len 4096
32736 183/300 Write block 4380 len 8192 data: 5 6
32737 184/300 Write block 3362 len 20480 data: 5 5 6 6 6
32738 185/300 Read block 4275 len 32768
32739 186/300 Read block 3396 len 20480
32740 187/300 Read block 2829 len 40960
32741 188/300 Flush
32742 189/300 Read block 367 len 8192
32743 190/300 Read block 3756 len 32768
32744 191/300 Read block 2322 len 28672
32745 192/300 Read block 4288 len 12288
32746 193/300 Read block 1873 len 24576
32747 194/300 Read block 4760 len 16384
32748 195/300 Read block 2458 len 24576
32749 196/300 Read block 3053 len 36864
32750 197/300 Read block 3569 len 24576
32751 198/300 Read block 2155 len 20480
32752 199/300 Flush
32753 200/300 Read block 198 len 16384
32754 201/300 Write block 4448 len 36864 data: 5 8 7 8 8 7 8 7 5
32755 202/300 Flush
32756 203/300 Read block 3856 len 24576
32757 204/300 Read block 3564 len 12288
32758 205/300 Read block 1702 len 24576
32759 206/300 Read block 234 len 4096
32760 207/300 Read block 2040 len 16384
32761 208/300 Read block 1732 len 28672
32762 209/300 Read block 4806 len 4096
32763 210/300 Read block 1467 len 16384
32764 211/300 Write block 3731 len 32768 data: 5 4 5 6 6 5 5 4
32765 212/300 Read block 4049 len 20480
32766 213/300 Read block 4901 len 4096
32767 214/300 Write block 290 len 8192 data: 6 7
32768 215/300 Read block 2090 len 28672
32769 216/300 Read block 570 len 8192
32770 217/300 Write block 2683 len 28672 data: 7 6 6 6 3 2 2
32771 218/300 Write block 992 len 28672 data: 4 4 5 5 6 5 4
32772 219/300 Read block 512 len 20480
32773 220/300 Write block 1301 len 32768 data: 7 7 6 6 5 4 3 3
32774 221/300 Read block 4011 len 32768
32775 222/300 Read block 954 len 24576
32776 223/300 Flush
32777 224/300 Write block 3570 len 16384 data: 4 5 5 3
32778 225/300 Read block 114 len 32768
32779 226/300 Flush
32780 227/300 Flush
32781 228/300 Write block 2561 len 16384 data: 7 8 8 10
32782 229/300 Write block 521 len 20480 data: 8 8 8 8 8
32783 230/300 Flush
32784 231/300 Write block 3966 len 8192 data: 6 8
32785 232/300 Read block 4456 len 8192
32786 233/300 Write block 1492 len 20480 data: 7 5 5 6 7
32787 234/300 Read block 3638 len 40960
32788 235/300 Write block 4487 len 32768 data: 7 8 7 6 7 6 5 5
32789 236/300 Write block 4964 len 20480 data: 8 9 6 7 6
32790 237/300 Write block 287 len 36864 data: 6 4 5 7 8 7 6 6 7
32791 238/300 Flush
32792 239/300 Read block 2871 len 8192
32793 240/300 Read block 1126 len 8192
32794 241/300 Flush
32795 242/300 Write block 852 len 4096 data: 9
32796 243/300 Read block 602 len 36864
32797 244/300 Write block 1617 len 20480 data: 4 5 6 6 4
32798 245/300 Read block 3718 len 28672
32799 246/300 Read block 735 len 12288
32800 247/300 Read block 1344 len 12288
32801 248/300 Read block 2266 len 24576
32802 249/300 Read block 2814 len 8192
32803 250/300 Write block 1507 len 36864 data: 8 8 8 10 9 10 9 8 8
32804 251/300 Read block 3293 len 16384
32805 252/300 Write block 3625 len 8192 data: 6 6
32806 253/300 Read block 1383 len 20480
32807 254/300 Read block 653 len 36864
32808 255/300 Read block 673 len 24576
32809 256/300 Write block 1506 len 32768 data: 7 9 9 9 11 10 11 10
32810 257/300 Read block 717 len 8192
32811 258/300 Read block 372 len 8192
32812 259/300 Write block 3282 len 16384 data: 4 4 4 5
32813 260/300 Flush
32814 261/300 Write block 1609 len 16384 data: 4 4 3 3
32815 262/300 Read block 1256 len 36864
32816 263/300 Write block 4373 len 28672 data: 6 5 5 7 8 6 5
32817 264/300 Write block 1916 len 36864 data: 3 5 5 4 3 3 5 5 6
32818 265/300 Write block 4275 len 4096 data: 6
32819 266/300 Read block 4079 len 32768
32820 267/300 Read block 1414 len 40960
32821 268/300 Flush
32822 269/300 Write block 1630 len 12288 data: 2 3 5
32823 270/300 Write block 1687 len 28672 data: 2 4 6 7 8 7 7
32824 271/300 Flush
32825 272/300 Write block 162 len 20480 data: 5 4 4 4 4
32826 273/300 Write block 4535 len 36864 data: 8 7 8 9 8 7 6 8 7
32827 274/300 Write block 2173 len 28672 data: 5 5 4 4 5 4 4
32828 275/300 Write block 2477 len 24576 data: 12 13 12 8 8 7
32829 276/300 Read block 2761 len 16384
32830 277/300 Read block 1058 len 20480
32831 278/300 Flush
32832 279/300 Write block 3788 len 24576 data: 6 5 6 4 4 7
32833 280/300 Read block 2377 len 28672
32834 281/300 Write block 2290 len 40960 data: 9 10 10 13 12 10 9 10 7 7
32835 282/300 Read block 2217 len 36864
32836 283/300 Write block 2054 len 24576 data: 6 7 6 7 5 5
32837 284/300 Write block 3357 len 24576 data: 7 3 3 5 6 6
32838 285/300 Read block 2562 len 24576
32839 286/300 Write block 1858 len 16384 data: 5 7 5 4
32840 287/300 Write block 326 len 12288 data: 3 4 4
32841 288/300 Write block 3652 len 8192 data: 3 4
32842 289/300 Write block 2764 len 32768 data: 8 10 9 5 4 3 5 5
32843 290/300 Write block 327 len 40960 data: 5 5 3 4 3 4 3 3 3 3
32844 291/300 Read block 1648 len 16384
32845 292/300 Write block 2559 len 28672 data: 6 7 8 9 9 11 10
32846 293/300 Read block 1864 len 12288
32847 294/300 Write block 4759 len 12288 data: 4 6 6
32848 295/300 Write block 4313 len 12288 data: 10 10 9
32849 296/300 Write block 2879 len 4096 data: 6
32850 297/300 Write block 4205 len 8192 data: 10 9
32851 298/300 Write block 1315 len 8192 data: 5 5
32852 299/300 Read block 4628 len 40960
32853 300/300 Flush
32854 301/300 Write block 2113 len 16384 data: 7 6 3 5
32855 Main task has work to do, go find it
32856 got dsc Start(2)
32857 start 2
32858 Nov 01 16:57:25.004 INFO request completed, latency_us: 261, response_code: 204, uri: /start/cid/2, method: POST, req_id: 0483be9f-4565-4b2f-a7c4-dc310f399b31, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
32859 Replay: started 2, returned:Ok(())
32860 ----------------------------------------------------------------
32861 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32862 States: Active Active Offline
32863 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32864 9462 Acked Write 7 Done Done Sent false
32865 9463 Acked Write 2 Done Done Sent false
32866 9464 Acked Write 9 Done Done Sent false
32867 9465 Acked Flush 0 Done Done Sent false
32868 9466 Acked Read 3 Done Done Sent false
32869 9467 Acked Read 4 Done Done Sent false
32870 9468 Acked Write 4 Done Done Sent false
32871 9469 Acked Read 7 Done Done Sent false
32872 9470 Acked Read 1 Done Done Sent false
32873 9471 Acked Read 7 Done Done Sent false
32874 9472 Acked Write 8 Done Done Sent false
32875 9473 Acked Read 1 Done Done Sent false
32876 9474 Acked Read 1 Done Done Sent false
32877 9475 Acked Write 8 Done Done Sent false
32878 9476 Acked Write 8 Done Done Sent false
32879 9477 Acked Read 10 Done Done Sent false
32880 9478 Acked Write 10 Done Done Sent false
32881 9479 Acked Read 7 Done Done Sent false
32882 9480 Acked Write 2 Done Done Sent false
32883 9481 Acked Write 2 Done Done Sent false
32884 9482 Acked Read 2 Done Done Sent false
32885 9483 Acked Read 4 Done Done Sent false
32886 9484 Acked Write 9 Done Done Sent false
32887 9485 Acked Write 6 Done Done Sent false
32888 9486 Acked Write 6 Done Done Sent false
32889 9487 Acked Write 7 Done Done Sent false
32890 9488 Acked Write 6 Done Done Sent false
32891 9489 Acked Flush 0 Done Done Sent false
32892 9490 Acked Read 4 Done Done Sent false
32893 9491 Acked Read 8 Done Done Sent false
32894 9492 Acked Read 4 Done Done Sent false
32895 9493 Acked Read 3 Done Done Sent false
32896 9494 Acked Flush 0 Done Done Sent false
32897 9495 Acked Read 2 Done Done Sent false
32898 9496 Acked Flush 0 Done Done Sent false
32899 9497 Acked Write 9 Done Done Sent false
32900 9498 Acked Read 6 Done Done Sent false
32901 9499 Acked Read 10 Done Done Sent false
32902 9500 Acked Read 2 Done Done Sent false
32903 9501 Acked Read 8 Done Done Sent false
32904 9502 Acked Read 5 Done Done Sent false
32905 9503 Acked Write 6 Done Done Sent false
32906 9504 Acked Write 2 Done Done Sent false
32907 9505 Acked Read 6 Done Done Sent false
32908 9506 Acked Flush 0 Done Done Sent false
32909 9507 Acked Read 9 Done Done Sent false
32910 9508 Acked Read 1 Done Done Sent false
32911 9509 Acked Read 9 Done Done Sent false
32912 9510 Acked Read 10 Done Done Sent false
32913 9511 Acked Write 5 Done Done Sent false
32914 9512 Acked Read 5 Done Done Sent false
32915 9513 Acked Flush 0 Done Done Sent false
32916 9514 Acked Write 5 Done Done Sent false
32917 9515 Acked Read 6 Done Done Sent false
32918 9516 Acked Read 8 Done Done Sent false
32919 9517 Acked Read 1 Done Done Sent false
32920 9518 Acked Flush 0 Done Done Sent false
32921 9519 Acked Flush 0 Done Done Sent false
32922 9520 Acked Write 1 Done Done Sent false
32923 9521 Acked Read 5 Done Done Sent false
32924 9522 Acked Flush 0 Done Done Sent false
32925 9523 Acked Write 9 Done Done Sent false
32926 9524 Acked Write 1 Done Done Sent false
32927 9525 Acked Read 3 Done Done Sent false
32928 9526 Acked Write 3 Done Done Sent false
32929 9527 Acked Write 3 Done Done Sent false
32930 9528 Acked Read 4 Done Done Sent false
32931 9529 Acked Write 7 Done Done Sent false
32932 9530 Acked Read 2 Done Done Sent false
32933 9531 Acked Read 8 Done Done Sent false
32934 9532 Acked Read 7 Done Done Sent false
32935 9533 Acked Write 2 Done Done Sent false
32936 9534 Acked Flush 0 Done Done Sent false
32937 9535 Acked Flush 0 Done Done Sent false
32938 9536 Acked Read 4 Done Done Sent false
32939 9537 Acked Read 8 Done Done Sent false
32940 9538 Acked Read 3 Done Done Sent false
32941 9539 Acked Read 1 Done Done Sent false
32942 9540 Acked Read 1 Done Done Sent false
32943 9541 Acked Write 9 Done Done Sent false
32944 9542 Acked Write 4 Done Done Sent false
32945 9543 Acked Write 6 Done Done Sent false
32946 9544 Acked Read 4 Done Done Sent false
32947 9545 Acked Write 9 Done Done Sent false
32948 9546 Acked Write 1 Done Done Sent false
32949 9547 Acked Read 6 Done Done Sent false
32950 9548 Acked Flush 0 Done Done Sent false
32951 9549 Acked Write 8 Done Done Sent false
32952 9550 Acked Read 6 Done Done Sent false
32953 9551 Acked Read 2 Done Done Sent false
32954 9552 Acked Write 1 Done Done Sent false
32955 9553 Acked Read 5 Done Done Sent false
32956 9554 Acked Write 1 Done Done Sent false
32957 9555 Acked Flush 0 Done Done Sent false
32958 9556 Acked Write 4 Done Done Sent false
32959 9557 Acked Read 1 Done Done Sent false
32960 9558 Acked Read 8 Done Done Sent false
32961 9559 Acked Read 5 Done Done Sent false
32962 9560 Acked Read 10 Done Done Sent false
32963 9561 Acked Write 7 Done Done Sent false
32964 9562 Acked Read 10 Done Done Sent false
32965 9563 Acked Write 8 Done Done Sent false
32966 9564 Acked Read 3 Done Done Sent false
32967 9565 Acked Write 7 Done Done Sent false
32968 9566 Acked Read 1 Done Done Sent false
32969 9567 Acked Read 4 Done Done Sent false
32970 9568 Acked Read 9 Done Done Sent false
32971 9569 Acked Read 7 Done Done Sent false
32972 9570 Acked Write 1 Done Done Sent false
32973 9571 Acked Write 8 Done Done Sent false
32974 9572 Acked Write 7 Done Done Sent false
32975 9573 Acked Write 8 Done Done Sent false
32976 9574 Acked Read 8 Done Done Sent false
32977 9575 Acked Flush 0 Done Done Sent false
32978 9576 Acked Write 7 Done Done Sent false
32979 9577 Acked Flush 0 Done Done Sent false
32980 9578 Acked Read 5 Done Done Sent false
32981 9579 Acked Read 7 Done Done Sent false
32982 9580 Acked Read 9 Done Done Sent false
32983 9581 Acked Read 8 Done Done Sent false
32984 9582 Acked Write 3 Done Done Sent false
32985 9583 Acked Write 8 Done Done Sent false
32986 9584 Acked Read 7 Done Done Sent false
32987 9585 Acked Read 9 Done Done Sent false
32988 9586 Acked Read 8 Done Done Sent false
32989 9587 Acked Write 1 Done Done Sent false
32990 9588 Acked Write 10 Done Done Sent false
32991 9589 Acked Flush 0 Done Done Sent false
32992 9590 Acked Read 5 Done Done Sent false
32993 9591 Acked Read 2 Done Done Sent false
32994 9592 Acked Read 9 Done Done Sent false
32995 9593 Acked Flush 0 Done Done Sent false
32996 9594 Acked Read 7 Done Done Sent false
32997 9595 Acked Read 5 Done Done Sent false
32998 9596 Acked Read 7 Done Done Sent false
32999 9597 Acked Write 8 Done Done Sent false
33000 9598 Acked Write 10 Done Done Sent false
33001 9599 Acked Read 6 Done Done Sent false
33002 9600 Acked Flush 0 Done Done Sent false
33003 9601 Acked Write 7 Done Done Sent false
33004 9602 Acked Read 8 Done Done Sent false
33005 9603 Acked Write 2 Done Done Sent false
33006 9604 Acked Read 10 Done Done Sent false
33007 9605 Acked Read 9 Done Done Sent false
33008 9606 Acked Read 9 Done Done Sent false
33009 9607 Acked Read 1 Done Done Sent false
33010 9608 Acked Write 7 Done Done Sent false
33011 9609 Acked Write 7 Done Done Sent false
33012 9610 Acked Write 6 Done Done Sent false
33013 9611 Acked Read 4 Done Done Sent false
33014 9612 Acked Write 6 Done Done Sent false
33015 9613 Acked Flush 0 Done Done Sent false
33016 9614 Acked Read 3 Done Done Sent false
33017 9615 Acked Read 6 Done Done Sent false
33018 9616 Acked Read 3 Done Done Sent false
33019 9617 Acked Write 7 Done Done Sent false
33020 9618 Acked Write 3 Done Done Sent false
33021 9619 Acked Write 9 Done Done Sent false
33022 9620 Acked Read 10 Done Done Sent false
33023 9621 Acked Write 6 Done Done Sent false
33024 9622 Acked Flush 0 Done Done Sent false
33025 9623 Acked Read 6 Done Done Sent false
33026 9624 Acked Write 9 Done Done Sent false
33027 9625 Acked Read 9 Done Done Sent false
33028 9626 Acked Read 8 Done Done Sent false
33029 9627 Acked Flush 0 Done Done Sent false
33030 9628 Acked Write 5 Done Done Sent false
33031 9629 Acked Flush 0 Done Done Sent false
33032 9630 Acked Read 7 Done Done Sent false
33033 9631 Acked Write 9 Done Done Sent false
33034 9632 Acked Write 3 Done Done Sent false
33035 9633 Acked Write 8 Done Done Sent false
33036 9634 Acked Write 3 Done Done Sent false
33037 9635 Acked Write 1 Done Done Sent false
33038 9636 Acked Read 3 Done Done Sent false
33039 9637 Acked Write 1 Done Done Sent false
33040 9638 Acked Read 9 Done Done Sent false
33041 9639 Acked Read 2 Done Done Sent false
33042 9640 Acked Read 1 Done Done Sent false
33043 9641 Acked Read 2 Done Done Sent false
33044 9642 Acked Read 9 Done Done Sent false
33045 9643 Acked Read 1 Done Done Sent false
33046 9644 Acked Write 2 Done Done Sent false
33047 9645 Acked Write 5 Done Done Sent false
33048 9646 Acked Read 8 Done Done Sent false
33049 9647 Acked Read 5 Done Done Sent false
33050 9648 Acked Read 10 Done Done Sent false
33051 9649 Acked Flush 0 Done Done Sent false
33052 9650 Acked Read 2 Done Done Sent false
33053 9651 Acked Read 8 Done Done Sent false
33054 9652 Acked Read 7 Done Done Sent false
33055 9653 Acked Read 3 Done Done Sent false
33056 9654 Acked Read 6 Done Done Sent false
33057 9655 Acked Read 4 Done Done Sent false
33058 9656 Acked Read 6 Done Done Sent false
33059 9657 Acked Read 9 Done Done Sent false
33060 9658 Acked Read 6 Done Done Sent false
33061 9659 Acked Read 5 Done Done Sent false
33062 9660 Acked Flush 0 Done Done Sent false
33063 9661 Acked Read 4 Done Done Sent false
33064 9662 Acked Write 9 Done Done Sent false
33065 9663 Acked Flush 0 Done Done Sent false
33066 9664 Acked Read 6 Done Done Sent false
33067 9665 Acked Read 3 Done Done Sent false
33068 9666 Acked Read 6 Done Done Sent false
33069 9667 Acked Read 1 Done Done Sent false
33070 9668 Acked Read 4 Done Done Sent false
33071 9669 Acked Read 7 Done Done Sent false
33072 9670 Acked Read 1 Done Done Sent false
33073 9671 Acked Read 4 Done Done Sent false
33074 9672 Acked Write 8 Done Done Sent false
33075 9673 Acked Read 5 Done Done Sent false
33076 9674 Acked Read 1 Done Done Sent false
33077 9675 Acked Write 2 Done Done Sent false
33078 9676 Acked Read 7 Done Done Sent false
33079 9677 Acked Read 2 Done Done Sent false
33080 9678 Acked Write 7 Done Done Sent false
33081 9679 Acked Write 7 Done Done Sent false
33082 9680 Acked Read 5 Done Done Sent false
33083 9681 Acked Write 8 Done Done Sent false
33084 9682 Acked Read 8 Done Done Sent false
33085 9683 Acked Read 6 Done Done Sent false
33086 9684 Acked Flush 0 Done Done Sent false
33087 9685 Acked Write 4 Done Done Sent false
33088 9686 Acked Read 8 Done Done Sent false
33089 9687 Acked Flush 0 Done Done Sent false
33090 9688 Acked Flush 0 Done Done Sent false
33091 9689 Acked Write 4 Done Done Sent false
33092 9690 Acked Write 5 Done Done Sent false
33093 9691 Acked Flush 0 Done Done Sent false
33094 9692 Acked Write 2 Done Done Sent false
33095 9693 Acked Read 2 Done Done Sent false
33096 9694 Acked Write 5 Done Done Sent false
33097 9695 Acked Read 10 Done Done Sent false
33098 9696 Acked Write 8 Done Done Sent false
33099 9697 Acked Write 5 Done Done Sent false
33100 9698 Acked Write 9 Done Done Sent false
33101 9699 Acked Flush 0 Done Done Sent false
33102 9700 Acked Read 2 Done Done Sent false
33103 9701 Acked Read 2 Done Done Sent false
33104 9702 Acked Flush 0 Done Done Sent false
33105 9703 Acked Write 1 Done Done Sent false
33106 9704 Acked Read 9 Done Done Sent false
33107 9705 Acked Write 5 Done Done Sent false
33108 9706 Acked Read 7 Done Done Sent false
33109 9707 Acked Read 3 Done Done Sent false
33110 9708 Acked Read 3 Done Done Sent false
33111 9709 Acked Read 6 Done Done Sent false
33112 9710 Acked Read 2 Done Done Sent false
33113 9711 Acked Write 9 Done Done Sent false
33114 9712 Acked Read 4 Done Done Sent false
33115 9713 Acked Write 2 Done Done Sent false
33116 9714 Acked Read 5 Done Done Sent false
33117 9715 Acked Read 9 Done Done Sent false
33118 9716 Acked Read 6 Done Done Sent false
33119 9717 Acked Write 8 Done Done Sent false
33120 9718 Acked Read 2 Done Done Sent false
33121 9719 Acked Read 2 Done Done Sent false
33122 9720 Acked Write 4 Done Done Sent false
33123 9721 Acked Flush 0 Done Done Sent false
33124 9722 Acked Write 4 Done Done Sent false
33125 9723 Acked Read 9 Done Done Sent false
33126 9724 Acked Write 7 Done Done Sent false
33127 9725 Acked Write 9 Done Done Sent false
33128 9726 Acked Write 1 Done Done Sent false
33129 9727 Acked Read 8 Done Done Sent false
33130 9728 Acked Read 10 Done Done Sent false
33131 9729 Acked Flush 0 Done Done Sent false
33132 9730 Acked Write 3 Done Done Sent false
33133 9731 Acked Write 7 Done Done Sent false
33134 9732 Acked Flush 0 Done Done Sent false
33135 9733 Acked Write 5 Done Done Sent false
33136 9734 Acked Write 9 Done Done Sent false
33137 9735 Acked Write 7 Done Done Sent false
33138 9736 Acked Write 6 Done Done Sent false
33139 9737 Acked Read 4 Done Done Sent false
33140 9738 Acked Read 5 Done Done Sent false
33141 9739 Acked Flush 0 Done Done Sent false
33142 9740 Acked Write 6 Done Done Sent false
33143 9741 Acked Read 7 Done Done Sent false
33144 9742 Acked Write 10 Done Done Sent false
33145 9743 Acked Read 9 Done Done Sent false
33146 9744 Acked Write 6 Done Done Sent false
33147 9745 Acked Write 6 Done Done Sent false
33148 9746 Acked Read 6 Done Done Sent false
33149 9747 Acked Write 4 Done Done Sent false
33150 9748 Acked Write 3 Done Done Sent false
33151 9749 Acked Write 2 Done Done Sent false
33152 9750 Acked Write 8 Done Done Sent false
33153 9751 Acked Write 10 Done Done Sent false
33154 9752 Acked Read 4 Done Done Sent false
33155 9753 Acked Write 7 Done Done Sent false
33156 9754 Acked Read 3 Done Done Sent false
33157 9755 Acked Write 3 Done Done Sent false
33158 9756 Acked Write 3 Done Done Sent false
33159 9757 Acked Write 1 Done Done Sent false
33160 9758 Acked Write 2 Done Done Sent false
33161 9759 Acked Write 2 Done Done Sent false
33162 9760 Acked Read 10 Done Done Sent false
33163 9761 Acked Flush 0 Done Done Sent false
33164 9762 Acked Write 4 Done Done Sent false
33165 STATES DS:0 DS:1 DS:2 TOTAL
33166 Sent 0 0 301 301
33167 Done 301 301 0 602
33168 Skipped 0 0 0 0
33169 Error 0 0 0 0
33170 Last Flush: 9761 9761 9461
33171 Downstairs last five completed: 9461 9460 9459 9458 9457
33172 Upstairs last five completed: 9762 9761 9760 9759 9758
33173 CLIENT: Up:0 ds:301 act:2
33174 [8830] got start action
33175 [8830] I am going to restart
33176 Starting downstairs at port 8830
33177 Make output file at "/tmp/dsc/downstairs-8830.txt"
33178 [8830][2] reports Starting
33179 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1227)
33180 [8830][2] reports Running
33181 ----------------------------------------------------------------
33182 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
33183 States: Active Active Offline
33184 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33185 9462 Acked Write 7 Done Done Sent false
33186 9463 Acked Write 2 Done Done Sent false
33187 9464 Acked Write 9 Done Done Sent false
33188 9465 Acked Flush 0 Done Done Sent false
33189 9466 Acked Read 3 Done Done Sent false
33190 9467 Acked Read 4 Done Done Sent false
33191 9468 Acked Write 4 Done Done Sent false
33192 9469 Acked Read 7 Done Done Sent false
33193 9470 Acked Read 1 Done Done Sent false
33194 9471 Acked Read 7 Done Done Sent false
33195 9472 Acked Write 8 Done Done Sent false
33196 9473 Acked Read 1 Done Done Sent false
33197 9474 Acked Read 1 Done Done Sent false
33198 9475 Acked Write 8 Done Done Sent false
33199 9476 Acked Write 8 Done Done Sent false
33200 9477 Acked Read 10 Done Done Sent false
33201 9478 Acked Write 10 Done Done Sent false
33202 9479 Acked Read 7 Done Done Sent false
33203 9480 Acked Write 2 Done Done Sent false
33204 9481 Acked Write 2 Done Done Sent false
33205 9482 Acked Read 2 Done Done Sent false
33206 9483 Acked Read 4 Done Done Sent false
33207 9484 Acked Write 9 Done Done Sent false
33208 9485 Acked Write 6 Done Done Sent false
33209 9486 Acked Write 6 Done Done Sent false
33210 9487 Acked Write 7 Done Done Sent false
33211 9488 Acked Write 6 Done Done Sent false
33212 9489 Acked Flush 0 Done Done Sent false
33213 9490 Acked Read 4 Done Done Sent false
33214 9491 Acked Read 8 Done Done Sent false
33215 9492 Acked Read 4 Done Done Sent false
33216 9493 Acked Read 3 Done Done Sent false
33217 9494 Acked Flush 0 Done Done Sent false
33218 9495 Acked Read 2 Done Done Sent false
33219 9496 Acked Flush 0 Done Done Sent false
33220 9497 Acked Write 9 Done Done Sent false
33221 9498 Acked Read 6 Done Done Sent false
33222 9499 Acked Read 10 Done Done Sent false
33223 9500 Acked Read 2 Done Done Sent false
33224 9501 Acked Read 8 Done Done Sent false
33225 9502 Acked Read 5 Done Done Sent false
33226 9503 Acked Write 6 Done Done Sent false
33227 9504 Acked Write 2 Done Done Sent false
33228 9505 Acked Read 6 Done Done Sent false
33229 9506 Acked Flush 0 Done Done Sent false
33230 9507 Acked Read 9 Done Done Sent false
33231 9508 Acked Read 1 Done Done Sent false
33232 9509 Acked Read 9 Done Done Sent false
33233 9510 Acked Read 10 Done Done Sent false
33234 9511 Acked Write 5 Done Done Sent false
33235 9512 Acked Read 5 Done Done Sent false
33236 9513 Acked Flush 0 Done Done Sent false
33237 9514 Acked Write 5 Done Done Sent false
33238 9515 Acked Read 6 Done Done Sent false
33239 9516 Acked Read 8 Done Done Sent false
33240 9517 Acked Read 1 Done Done Sent false
33241 9518 Acked Flush 0 Done Done Sent false
33242 9519 Acked Flush 0 Done Done Sent false
33243 9520 Acked Write 1 Done Done Sent false
33244 9521 Acked Read 5 Done Done Sent false
33245 9522 Acked Flush 0 Done Done Sent false
33246 9523 Acked Write 9 Done Done Sent false
33247 9524 Acked Write 1 Done Done Sent false
33248 9525 Acked Read 3 Done Done Sent false
33249 9526 Acked Write 3 Done Done Sent false
33250 9527 Acked Write 3 Done Done Sent false
33251 9528 Acked Read 4 Done Done Sent false
33252 9529 Acked Write 7 Done Done Sent false
33253 9530 Acked Read 2 Done Done Sent false
33254 9531 Acked Read 8 Done Done Sent false
33255 9532 Acked Read 7 Done Done Sent false
33256 9533 Acked Write 2 Done Done Sent false
33257 9534 Acked Flush 0 Done Done Sent false
33258 9535 Acked Flush 0 Done Done Sent false
33259 9536 Acked Read 4 Done Done Sent false
33260 9537 Acked Read 8 Done Done Sent false
33261 9538 Acked Read 3 Done Done Sent false
33262 9539 Acked Read 1 Done Done Sent false
33263 9540 Acked Read 1 Done Done Sent false
33264 9541 Acked Write 9 Done Done Sent false
33265 9542 Acked Write 4 Done Done Sent false
33266 9543 Acked Write 6 Done Done Sent false
33267 9544 Acked Read 4 Done Done Sent false
33268 9545 Acked Write 9 Done Done Sent false
33269 9546 Acked Write 1 Done Done Sent false
33270 9547 Acked Read 6 Done Done Sent false
33271 9548 Acked Flush 0 Done Done Sent false
33272 9549 Acked Write 8 Done Done Sent false
33273 9550 Acked Read 6 Done Done Sent false
33274 9551 Acked Read 2 Done Done Sent false
33275 9552 Acked Write 1 Done Done Sent false
33276 9553 Acked Read 5 Done Done Sent false
33277 9554 Acked Write 1 Done Done Sent false
33278 9555 Acked Flush 0 Done Done Sent false
33279 9556 Acked Write 4 Done Done Sent false
33280 9557 Acked Read 1 Done Done Sent false
33281 9558 Acked Read 8 Done Done Sent false
33282 9559 Acked Read 5 Done Done Sent false
33283 9560 Acked Read 10 Done Done Sent false
33284 9561 Acked Write 7 Done Done Sent false
33285 9562 Acked Read 10 Done Done Sent false
33286 9563 Acked Write 8 Done Done Sent false
33287 9564 Acked Read 3 Done Done Sent false
33288 9565 Acked Write 7 Done Done Sent false
33289 9566 Acked Read 1 Done Done Sent false
33290 9567 Acked Read 4 Done Done Sent false
33291 9568 Acked Read 9 Done Done Sent false
33292 9569 Acked Read 7 Done Done Sent false
33293 9570 Acked Write 1 Done Done Sent false
33294 9571 Acked Write 8 Done Done Sent false
33295 9572 Acked Write 7 Done Done Sent false
33296 9573 Acked Write 8 Done Done Sent false
33297 9574 Acked Read 8 Done Done Sent false
33298 9575 Acked Flush 0 Done Done Sent false
33299 9576 Acked Write 7 Done Done Sent false
33300 9577 Acked Flush 0 Done Done Sent false
33301 9578 Acked Read 5 Done Done Sent false
33302 9579 Acked Read 7 Done Done Sent false
33303 9580 Acked Read 9 Done Done Sent false
33304 9581 Acked Read 8 Done Done Sent false
33305 9582 Acked Write 3 Done Done Sent false
33306 9583 Acked Write 8 Done Done Sent false
33307 9584 Acked Read 7 Done Done Sent false
33308 9585 Acked Read 9 Done Done Sent false
33309 9586 Acked Read 8 Done Done Sent false
33310 9587 Acked Write 1 Done Done Sent false
33311 9588 Acked Write 10 Done Done Sent false
33312 9589 Acked Flush 0 Done Done Sent false
33313 9590 Acked Read 5 Done Done Sent false
33314 9591 Acked Read 2 Done Done Sent false
33315 9592 Acked Read 9 Done Done Sent false
33316 9593 Acked Flush 0 Done Done Sent false
33317 9594 Acked Read 7 Done Done Sent false
33318 9595 Acked Read 5 Done Done Sent false
33319 9596 Acked Read 7 Done Done Sent false
33320 9597 Acked Write 8 Done Done Sent false
33321 9598 Acked Write 10 Done Done Sent false
33322 9599 Acked Read 6 Done Done Sent false
33323 9600 Acked Flush 0 Done Done Sent false
33324 9601 Acked Write 7 Done Done Sent false
33325 9602 Acked Read 8 Done Done Sent false
33326 9603 Acked Write 2 Done Done Sent false
33327 9604 Acked Read 10 Done Done Sent false
33328 9605 Acked Read 9 Done Done Sent false
33329 9606 Acked Read 9 Done Done Sent false
33330 9607 Acked Read 1 Done Done Sent false
33331 9608 Acked Write 7 Done Done Sent false
33332 9609 Acked Write 7 Done Done Sent false
33333 9610 Acked Write 6 Done Done Sent false
33334 9611 Acked Read 4 Done Done Sent false
33335 9612 Acked Write 6 Done Done Sent false
33336 9613 Acked Flush 0 Done Done Sent false
33337 9614 Acked Read 3 Done Done Sent false
33338 9615 Acked Read 6 Done Done Sent false
33339 9616 Acked Read 3 Done Done Sent false
33340 9617 Acked Write 7 Done Done Sent false
33341 9618 Acked Write 3 Done Done Sent false
33342 9619 Acked Write 9 Done Done Sent false
33343 9620 Acked Read 10 Done Done Sent false
33344 9621 Acked Write 6 Done Done Sent false
33345 9622 Acked Flush 0 Done Done Sent false
33346 9623 Acked Read 6 Done Done Sent false
33347 9624 Acked Write 9 Done Done Sent false
33348 9625 Acked Read 9 Done Done Sent false
33349 9626 Acked Read 8 Done Done Sent false
33350 9627 Acked Flush 0 Done Done Sent false
33351 9628 Acked Write 5 Done Done Sent false
33352 9629 Acked Flush 0 Done Done Sent false
33353 9630 Acked Read 7 Done Done Sent false
33354 9631 Acked Write 9 Done Done Sent false
33355 9632 Acked Write 3 Done Done Sent false
33356 9633 Acked Write 8 Done Done Sent false
33357 9634 Acked Write 3 Done Done Sent false
33358 9635 Acked Write 1 Done Done Sent false
33359 9636 Acked Read 3 Done Done Sent false
33360 9637 Acked Write 1 Done Done Sent false
33361 9638 Acked Read 9 Done Done Sent false
33362 9639 Acked Read 2 Done Done Sent false
33363 9640 Acked Read 1 Done Done Sent false
33364 9641 Acked Read 2 Done Done Sent false
33365 9642 Acked Read 9 Done Done Sent false
33366 9643 Acked Read 1 Done Done Sent false
33367 9644 Acked Write 2 Done Done Sent false
33368 9645 Acked Write 5 Done Done Sent false
33369 9646 Acked Read 8 Done Done Sent false
33370 9647 Acked Read 5 Done Done Sent false
33371 9648 Acked Read 10 Done Done Sent false
33372 9649 Acked Flush 0 Done Done Sent false
33373 9650 Acked Read 2 Done Done Sent false
33374 9651 Acked Read 8 Done Done Sent false
33375 9652 Acked Read 7 Done Done Sent false
33376 9653 Acked Read 3 Done Done Sent false
33377 9654 Acked Read 6 Done Done Sent false
33378 9655 Acked Read 4 Done Done Sent false
33379 9656 Acked Read 6 Done Done Sent false
33380 9657 Acked Read 9 Done Done Sent false
33381 9658 Acked Read 6 Done Done Sent false
33382 9659 Acked Read 5 Done Done Sent false
33383 9660 Acked Flush 0 Done Done Sent false
33384 9661 Acked Read 4 Done Done Sent false
33385 9662 Acked Write 9 Done Done Sent false
33386 9663 Acked Flush 0 Done Done Sent false
33387 9664 Acked Read 6 Done Done Sent false
33388 9665 Acked Read 3 Done Done Sent false
33389 9666 Acked Read 6 Done Done Sent false
33390 9667 Acked Read 1 Done Done Sent false
33391 9668 Acked Read 4 Done Done Sent false
33392 9669 Acked Read 7 Done Done Sent false
33393 9670 Acked Read 1 Done Done Sent false
33394 9671 Acked Read 4 Done Done Sent false
33395 9672 Acked Write 8 Done Done Sent false
33396 9673 Acked Read 5 Done Done Sent false
33397 9674 Acked Read 1 Done Done Sent false
33398 9675 Acked Write 2 Done Done Sent false
33399 9676 Acked Read 7 Done Done Sent false
33400 9677 Acked Read 2 Done Done Sent false
33401 9678 Acked Write 7 Done Done Sent false
33402 9679 Acked Write 7 Done Done Sent false
33403 9680 Acked Read 5 Done Done Sent false
33404 9681 Acked Write 8 Done Done Sent false
33405 9682 Acked Read 8 Done Done Sent false
33406 9683 Acked Read 6 Done Done Sent false
33407 9684 Acked Flush 0 Done Done Sent false
33408 9685 Acked Write 4 Done Done Sent false
33409 9686 Acked Read 8 Done Done Sent false
33410 9687 Acked Flush 0 Done Done Sent false
33411 9688 Acked Flush 0 Done Done Sent false
33412 9689 Acked Write 4 Done Done Sent false
33413 9690 Acked Write 5 Done Done Sent false
33414 9691 Acked Flush 0 Done Done Sent false
33415 9692 Acked Write 2 Done Done Sent false
33416 9693 Acked Read 2 Done Done Sent false
33417 9694 Acked Write 5 Done Done Sent false
33418 9695 Acked Read 10 Done Done Sent false
33419 9696 Acked Write 8 Done Done Sent false
33420 9697 Acked Write 5 Done Done Sent false
33421 9698 Acked Write 9 Done Done Sent false
33422 9699 Acked Flush 0 Done Done Sent false
33423 9700 Acked Read 2 Done Done Sent false
33424 9701 Acked Read 2 Done Done Sent false
33425 9702 Acked Flush 0 Done Done Sent false
33426 9703 Acked Write 1 Done Done Sent false
33427 9704 Acked Read 9 Done Done Sent false
33428 9705 Acked Write 5 Done Done Sent false
33429 9706 Acked Read 7 Done Done Sent false
33430 9707 Acked Read 3 Done Done Sent false
33431 9708 Acked Read 3 Done Done Sent false
33432 9709 Acked Read 6 Done Done Sent false
33433 9710 Acked Read 2 Done Done Sent false
33434 9711 Acked Write 9 Done Done Sent false
33435 9712 Acked Read 4 Done Done Sent false
33436 9713 Acked Write 2 Done Done Sent false
33437 9714 Acked Read 5 Done Done Sent false
33438 9715 Acked Read 9 Done Done Sent false
33439 9716 Acked Read 6 Done Done Sent false
33440 9717 Acked Write 8 Done Done Sent false
33441 9718 Acked Read 2 Done Done Sent false
33442 9719 Acked Read 2 Done Done Sent false
33443 9720 Acked Write 4 Done Done Sent false
33444 9721 Acked Flush 0 Done Done Sent false
33445 9722 Acked Write 4 Done Done Sent false
33446 9723 Acked Read 9 Done Done Sent false
33447 9724 Acked Write 7 Done Done Sent false
33448 9725 Acked Write 9 Done Done Sent false
33449 9726 Acked Write 1 Done Done Sent false
33450 9727 Acked Read 8 Done Done Sent false
33451 9728 Acked Read 10 Done Done Sent false
33452 9729 Acked Flush 0 Done Done Sent false
33453 9730 Acked Write 3 Done Done Sent false
33454 9731 Acked Write 7 Done Done Sent false
33455 9732 Acked Flush 0 Done Done Sent false
33456 9733 Acked Write 5 Done Done Sent false
33457 9734 Acked Write 9 Done Done Sent false
33458 9735 Acked Write 7 Done Done Sent false
33459 9736 Acked Write 6 Done Done Sent false
33460 9737 Acked Read 4 Done Done Sent false
33461 9738 Acked Read 5 Done Done Sent false
33462 9739 Acked Flush 0 Done Done Sent false
33463 9740 Acked Write 6 Done Done Sent false
33464 9741 Acked Read 7 Done Done Sent false
33465 9742 Acked Write 10 Done Done Sent false
33466 9743 Acked Read 9 Done Done Sent false
33467 9744 Acked Write 6 Done Done Sent false
33468 9745 Acked Write 6 Done Done Sent false
33469 9746 Acked Read 6 Done Done Sent false
33470 9747 Acked Write 4 Done Done Sent false
33471 9748 Acked Write 3 Done Done Sent false
33472 9749 Acked Write 2 Done Done Sent false
33473 9750 Acked Write 8 Done Done Sent false
33474 9751 Acked Write 10 Done Done Sent false
33475 9752 Acked Read 4 Done Done Sent false
33476 9753 Acked Write 7 Done Done Sent false
33477 9754 Acked Read 3 Done Done Sent false
33478 9755 Acked Write 3 Done Done Sent false
33479 9756 Acked Write 3 Done Done Sent false
33480 9757 Acked Write 1 Done Done Sent false
33481 9758 Acked Write 2 Done Done Sent false
33482 9759 Acked Write 2 Done Done Sent false
33483 9760 Acked Read 10 Done Done Sent false
33484 9761 Acked Flush 0 Done Done Sent false
33485 9762 Acked Write 4 Done Done Sent false
33486 9763 Acked Flush 0 Done Done Sent false
33487 STATES DS:0 DS:1 DS:2 TOTAL
33488 Sent 0 0 302 302
33489 Done 302 302 0 604
33490 Skipped 0 0 0 0
33491 Error 0 0 0 0
33492 Last Flush: 9763 9763 9461
33493 Downstairs last five completed: 9461 9460 9459 9458 9457
33494 Upstairs last five completed: 9763 9762 9761 9760 9759
33495 CLIENT: Up:0 ds:302 act:2
334962024-11-01T16:57:30.708ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
334972024-11-01T16:57:30.708ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:45981) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
334982024-11-01T16:57:30.710ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
334992024-11-01T16:57:30.710ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
335002024-11-01T16:57:30.710ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
335012024-11-01T16:57:30.710ZINFOcrucible: send last flush ID to this DS: 9461 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
335022024-11-01T16:57:30.710ZINFOcrucible: Replied this last flush ID: 9461 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
335032024-11-01T16:57:30.710ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
335042024-11-01T16:57:30.710ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
335052024-11-01T16:57:30.710ZINFOcrucible: [2] client re-new 302 jobs since flush 9461 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
335062024-11-01T16:57:30.711ZINFOcrucible: [2] Replayed 302 jobs since flush: 9461 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
33507 ----------------------------------------------------------------
33508 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33509 Downstairs last five completed: 9763 9762 9761 9760 9759
33510 Upstairs last five completed: 9763 9762 9761 9760 9759
33511 CLIENT: Up:0 ds:0 act:3
33512 Replay: All jobs finished
33513 Main task has work to do, go find it
33514 got dsc Stop(2)
33515 stop 2
33516 [8830] Got stop action so:false kr:false
33517 Nov 01 16:57:33.010 INFO request completed, latency_us: 364, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 8eef8582-5912-4d60-aeed-80074327af76, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
33518 Nov 01 16:57:33.011 INFO request completed{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:57:33.011267119Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
33519 , latency_us: 118, response_code: 200, uri: /state/cid/2, method: GET, req_id: 561f858f-d02e-434e-a937-3a0ff9dc0503, remote_addr: 127.0.0.1:58552{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-11-01T16:57:33.011341997Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
33520 , local_addr: 127.0.0.1:9998
335212024-11-01T16:57:33.011ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
335222024-11-01T16:57:33.011ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
335232024-11-01T16:57:33.011ZINFOcrucible: client task is exiting = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
33524 [8830] Exited with: ExitStatus(unix_wait_status(9))
33525 [8830][2] reports Exit
33526 Nov 01 16:57:37.014 INFO request completed, latency_us: 269, response_code: 200, uri: /state/cid/2, method: GET, req_id: fdc90d6b-8e15-4e19-91e7-4c638b0d79e7, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
33527 001/300 Flush
33528 002/300 Write block 653 len 32768 data: 4 4 6 7 9 9 8 9
33529 003/300 Write block 355 len 12288 data: 2 2 3
33530 004/300 Flush
33531 005/300 Write block 2594 len 24576 data: 4 5 5 6 6 6
33532 006/300 Read block 4846 len 4096
33533 007/300 Read block 2800 len 20480
33534 008/300 Write block 738 len 8192 data: 8 9
33535 009/300 Write block 4291 len 8192 data: 3 2
33536 010/300 Write block 3045 len 36864 data: 6 5 6 5 2 3 3 3 4
33537 011/300 Read block 2325 len 28672
33538 012/300 Write block 171 len 20480 data: 7 7 7 7 6
33539 013/300 Write block 1176 len 12288 data: 9 9 10
33540 014/300 Write block 2935 len 24576 data: 6 6 6 5 4 5
33541 015/300 Read block 3251 len 36864
33542 016/300 Write block 2464 len 20480 data: 4 4 5 5 6
33543 017/300 Flush
33544 018/300 Read block 1784 len 40960
33545 019/300 Read block 3965 len 12288
33546 020/300 Read block 4390 len 20480
33547 021/300 Write block 4948 len 8192 data: 7 8
33548 022/300 Write block 637 len 8192 data: 4 3
33549 023/300 Write block 2733 len 4096 data: 7
33550 024/300 Read block 3015 len 24576
33551 025/300 Flush
33552 026/300 Write block 485 len 36864 data: 6 6 5 4 4 6 5 5 5
33553 027/300 Write block 4307 len 40960 data: 6 6 9 11 12 11 11 11 10 8
33554 028/300 Write block 492 len 28672 data: 6 6 6 5 4 3 5
33555 029/300 Read block 2620 len 12288
33556 030/300 Flush
33557 031/300 Read block 4232 len 36864
33558 032/300 Write block 561 len 4096 data: 4
33559 033/300 Read block 3982 len 36864
33560 034/300 Read block 2596 len 28672
33561 035/300 Write block 150 len 4096 data: 6
33562 036/300 Read block 1452 len 28672
33563 037/300 Write block 373 len 28672 data: 7 6 6 6 5 5 7
33564 038/300 Write block 2185 len 40960 data: 8 8 8 7 7 8 6 6 5 5
33565 039/300 Write block 2039 len 12288 data: 5 6 5
33566 040/300 Read block 953 len 28672
33567 041/300 Write block 1792 len 36864 data: 5 4 4 6 6 6 6 6 7
33568 042/300 Read block 1901 len 40960
33569 043/300 Read block 4059 len 24576
33570 044/300 Read block 3841 len 36864
33571 045/300 Flush
33572 046/300 Read block 433 len 32768
33573 047/300 Write block 4465 len 20480 data: 5 4 4 5 6
33574 048/300 Read block 3121 len 4096
33575 049/300 Write block 1962 len 40960 data: 8 8 7 6 6 6 7 8 8 9
33576 050/300 Write block 4480 len 20480 data: 4 4 3 3 5
33577 051/300 Flush
33578 052/300 Write block 44 len 12288 data: 4 3 3
33579 053/300 Write block 3260 len 36864 data: 7 5 6 6 6 6 5 5 2
33580 054/300 Flush
33581 055/300 Read block 826 len 36864
33582 056/300 Read block 2840 len 12288
33583 057/300 Read block 3195 len 24576
33584 058/300 Write block 278 len 12288 data: 9 9 7
33585 059/300 Read block 1207 len 12288
33586 060/300 Read block 2532 len 12288
33587 061/300 Write block 3707 len 28672 data: 5 4 5 4 4 5 5
33588 062/300 Write block 1410 len 4096 data: 6
33589 063/300 Read block 1988 len 28672
33590 064/300 Read block 2852 len 40960
33591 065/300 Write block 4721 len 24576 data: 6 5 4 5 4 5
33592 066/300 Read block 288 len 36864
33593 067/300 Write block 1304 len 20480 data: 7 6 5 4 4
33594 068/300 Read block 2652 len 36864
33595 069/300 Write block 715 len 24576 data: 3 5 4 6 5 5
33596 070/300 Write block 4 len 8192 data: 8 7
33597 071/300 Read block 3331 len 12288
33598 072/300 Read block 93 len 12288
33599 073/300 Read block 671 len 16384
33600 074/300 Read block 4140 len 40960
33601 075/300 Read block 3395 len 40960
33602 076/300 Read block 596 len 8192
33603 077/300 Write block 3155 len 8192 data: 6 4
33604 078/300 Read block 1093 len 12288
33605 079/300 Read block 4954 len 4096
33606 080/300 Write block 77 len 28672 data: 6 5 5 5 8 7 6
33607 081/300 Read block 48 len 40960
33608 082/300 Write block 4471 len 4096 data: 5
33609 083/300 Read block 521 len 40960
33610 084/300 Read block 1731 len 32768
33611 085/300 Write block 2263 len 40960 data: 3 3 4 5 6 7 6 6 6 5
33612 086/300 Read block 2973 len 28672
33613 087/300 Write block 3583 len 24576 data: 7 9 9 8 8 9
33614 088/300 Write block 2272 len 40960 data: 6 7 9 8 5 5 7 7 8 8
33615 089/300 Read block 3805 len 32768
33616 090/300 Write block 3328 len 12288 data: 3 4 5
33617 091/300 Write block 4729 len 12288 data: 7 7 5
33618 092/300 Write block 1691 len 24576 data: 9 8 8 8 8 7
33619 093/300 Read block 1885 len 8192
33620 094/300 Write block 3633 len 36864 data: 4 4 2 2 2 2 2 2 2
33621 095/300 Flush
33622 096/300 Read block 2742 len 16384
33623 097/300 Write block 2946 len 24576 data: 3 3 4 5 4 4
33624 098/300 Write block 3591 len 32768 data: 10 11 9 7 6 5 6 5
33625 099/300 Read block 1944 len 12288
33626 100/300 Write block 3034 len 12288 data: 5 4 3
33627 101/300 Write block 3922 len 28672 data: 6 6 5 6 4 4 3
33628 102/300 Read block 1441 len 36864
33629 103/300 Write block 4664 len 8192 data: 9 7
33630 104/300 Write block 3992 len 36864 data: 7 6 6 5 7 6 4 3 3
33631 105/300 Write block 547 len 40960 data: 3 5 6 7 7 8 7 6 5 5
33632 106/300 Read block 4056 len 28672
33633 107/300 Read block 4979 len 16384
33634 108/300 Write block 3676 len 40960 data: 6 6 6 5 5 4 4 4 3 4
33635 109/300 Read block 902 len 36864
33636 110/300 Read block 2172 len 24576
33637 111/300 Write block 3697 len 8192 data: 8 6
33638 112/300 Read block 1706 len 24576
33639 113/300 Write block 926 len 40960 data: 10 10 6 7 9 9 9 9 6 6
33640 114/300 Read block 2854 len 16384
33641 115/300 Read block 4983 len 36864
33642 116/300 Write block 923 len 20480 data: 6 6 8 11 11
33643 117/300 Read block 666 len 36864
33644 118/300 Write block 2257 len 12288 data: 5 5 6
33645 119/300 Write block 4005 len 24576 data: 5 5 5 6 8 9
33646 120/300 Flush
33647 121/300 Write block 1211 len 24576 data: 9 9 12 10 9 8
33648 122/300 Write block 83 len 4096 data: 7
33649 123/300 Read block 4774 len 20480
33650 124/300 Write block 4318 len 24576 data: 13 12 12 10 9 9
33651 125/300 Read block 2230 len 40960
33652 126/300 Read block 4734 len 36864
33653 127/300 Read block 828 len 32768
33654 128/300 Write block 2571 len 20480 data: 5 4 4 4 3
33655 129/300 Read block 1942 len 8192
33656 130/300 Read block 1983 len 40960
33657 131/300 Flush
33658 132/300 Read block 2022 len 12288
33659 133/300 Write block 3413 len 40960 data: 5 6 6 6 7 7 4 5 6 8
33660 134/300 Read block 3217 len 28672
33661 135/300 Write block 1255 len 28672 data: 5 3 4 5 5 7 6
33662 136/300 Write block 4953 len 8192 data: 7 7
33663 137/300 Flush
33664 138/300 Write block 3403 len 28672 data: 3 6 6 6 6 4 4
33665 139/300 Read block 1734 len 28672
33666 140/300 Read block 2522 len 24576
33667 141/300 Flush
33668 142/300 Write block 974 len 12288 data: 6 7 6
33669 143/300 Write block 899 len 32768 data: 5 5 5 6 7 6 5 7
33670 144/300 Write block 4108 len 32768 data: 10 10 11 11 10 9 9 9
33671 145/300 Write block 4614 len 12288 data: 5 4 3
33672 146/300 Read block 3677 len 16384
33673 147/300 Read block 1282 len 20480
33674 148/300 Write block 2073 len 24576 data: 5 5 4 6 6 3
33675 149/300 Write block 3383 len 40960 data: 9 8 8 8 6 4 4 4 3 2
33676 150/300 Read block 2278 len 20480
33677 151/300 Write block 3146 len 4096 data: 5
33678 152/300 Write block 3883 len 40960 data: 5 5 5 5 5 5 5 4 3 3
33679 153/300 Read block 2261 len 4096
33680 154/300 Write block 2011 len 28672 data: 3 2 3 3 2 2 2
33681 155/300 Write block 2078 len 32768 data: 4 3 5 5 5 7 8 10
33682 156/300 Write block 4594 len 40960 data: 4 5 6 5 5 6 7 6 5 5
33683 157/300 Write block 3173 len 32768 data: 11 10 8 7 6 7 6 5
33684 158/300 Write block 2321 len 16384 data: 3 4 4 4
33685 159/300 Flush
33686 160/300 Read block 4214 len 4096
33687 161/300 Write block 4194 len 24576 data: 9 7 8 8 7 7
33688 162/300 Flush
33689 163/300 Read block 4485 len 12288
33690 164/300 Write block 3130 len 8192 data: 3 4
33691 165/300 Read block 4059 len 24576
33692 166/300 Write block 3731 len 40960 data: 6 5 6 7 7 6 6 5 6 6
33693 167/300 Write block 118 len 32768 data: 7 6 8 7 7 6 5 5
33694 168/300 Read block 4540 len 32768
33695 169/300 Write block 909 len 8192 data: 7 9
33696 170/300 Read block 1734 len 20480
33697 171/300 Write block 4752 len 40960 data: 6 5 5 5 5 4 3 5 7 7
33698 172/300 Read block 3639 len 4096
33699 173/300 Write block 3421 len 12288 data: 7 9 7
33700 174/300 Read block 3856 len 24576
33701 175/300 Read block 3392 len 40960
33702 176/300 Write block 739 len 24576 data: 10 7 6 4 2 3
33703 177/300 Read block 2522 len 4096
33704 178/300 Write block 2888 len 32768 data: 9 10 11 8 6 7 7 6
33705 179/300 Flush
33706 180/300 Write block 4334 len 32768 data: 4 4 5 5 4 5 6 6
33707 181/300 Read block 1563 len 40960
33708 182/300 Write block 289 len 20480 data: 6 8 9 8 7
33709 183/300 Write block 4936 len 16384 data: 4 4 4 5
33710 184/300 Write block 1214 len 32768 data: 11 10 9 8 12 9 9 9
33711 185/300 Read block 58 len 12288
33712 186/300 Read block 1214 len 32768
33713 187/300 Read block 4508 len 32768
33714 188/300 Write block 3923 len 24576 data: 7 6 7 5 5 4
33715 189/300 Read block 2642 len 40960
33716 190/300 Read block 4226 len 28672
33717 191/300 Flush
33718 192/300 Read block 771 len 32768
33719 193/300 Read block 3725 len 16384
33720 194/300 Read block 1487 len 32768
33721 195/300 Write block 3398 len 36864 data: 3 3 3 4 4 4 7 7 7
33722 196/300 Read block 2464 len 40960
33723 197/300 Write block 1437 len 8192 data: 9 8
33724 198/300 Read block 2732 len 12288
33725 199/300 Read block 4551 len 28672
33726 200/300 Write block 561 len 20480 data: 5 3 4 4 4
33727 201/300 Write block 3677 len 36864 data: 7 7 6 6 5 5 5 4 5
33728 202/300 Write block 1285 len 28672 data: 5 3 3 3 3 3 3
33729 203/300 Read block 103 len 36864
33730 204/300 Flush
33731 205/300 Read block 846 len 4096
33732 206/300 Read block 4324 len 12288
33733 207/300 Write block 3934 len 28672 data: 8 8 7 6 5 3 4
33734 208/300 Read block 1379 len 16384
33735 209/300 Read block 3833 len 4096
33736 210/300 Write block 1340 len 36864 data: 6 5 5 5 5 7 6 5 7
33737 211/300 Flush
33738 212/300 Read block 4877 len 28672
33739 213/300 Write block 391 len 4096 data: 5
33740 214/300 Flush
33741 215/300 Write block 2197 len 24576 data: 6 6 7 8 7 6
33742 216/300 Write block 1099 len 16384 data: 2 3 3 3
33743 217/300 Read block 323 len 8192
33744 218/300 Write block 2262 len 40960 data: 4 4 4 5 6 7 8 7 7 7
33745 219/300 Read block 2014 len 4096
33746 220/300 Read block 1913 len 32768
33747 221/300 Write block 3177 len 12288 data: 7 8 7
33748 222/300 Flush
33749 223/300 Flush
33750 224/300 Read block 606 len 16384
33751 225/300 Write block 3675 len 36864 data: 7 7 8 8 7 7 6 6 6
33752 226/300 Write block 3345 len 8192 data: 7 7
33753 227/300 Write block 4771 len 36864 data: 5 5 5 5 6 5 5 4 4
33754 228/300 Read block 1781 len 36864
33755 229/300 Write block 4973 len 32768 data: 4 6 6 5 5 4 6 7
33756 230/300 Read block 2831 len 40960
33757 231/300 Write block 671 len 32768 data: 7 7 3 3 4 6 6 6
33758 232/300 Write block 1656 len 24576 data: 3 5 6 6 8 8
33759 233/300 Read block 4044 len 20480
33760 234/300 Write block 4498 len 24576 data: 7 6 7 6 6 8
33761 235/300 Read block 1268 len 28672
33762 236/300 Read block 1756 len 36864
33763 237/300 Write block 4834 len 12288 data: 4 3 2
33764 238/300 Write block 2929 len 8192 data: 8 8
33765 239/300 Read block 2100 len 24576
33766 240/300 Write block 2686 len 4096 data: 7
33767 241/300 Flush
33768 242/300 Read block 710 len 16384
33769 243/300 Write block 35 len 40960 data: 3 5 5 5 5 7 7 7 8 5
33770 244/300 Write block 2481 len 28672 data: 9 8 5 6 6 7 7
33771 245/300 Read block 1879 len 16384
33772 246/300 Flush
33773 247/300 Write block 4024 len 28672 data: 7 5 5 5 3 3 5
33774 248/300 Read block 1354 len 24576
33775 249/300 Write block 4298 len 40960 data: 2 2 2 5 5 5 5 4 5 7
33776 250/300 Write block 493 len 8192 data: 7 7
33777 251/300 Read block 4320 len 12288
33778 252/300 Read block 4583 len 32768
33779 253/300 Flush
33780 254/300 Read block 2330 len 36864
33781 255/300 Write block 422 len 4096 data: 6
33782 256/300 Read block 3425 len 16384
33783 257/300 Read block 4045 len 28672
33784 258/300 Write block 3506 len 28672 data: 5 5 3 4 6 7 6
33785 259/300 Write block 4744 len 16384 data: 3 6 6 6
33786 260/300 Read block 1964 len 8192
33787 261/300 Read block 3237 len 4096
33788 262/300 Read block 1309 len 28672
33789 263/300 Read block 2231 len 36864
33790 264/300 Read block 3152 len 40960
33791 265/300 Read block 3544 len 20480
33792 266/300 Read block 4969 len 24576
33793 267/300 Read block 2623 len 20480
33794 268/300 Write block 3104 len 8192 data: 3 5
33795 269/300 Read block 1149 len 28672
33796 270/300 Read block 132 len 16384
33797 271/300 Read block 2154 len 4096
33798 272/300 Read block 1502 len 40960
33799 273/300 Write block 1968 len 16384 data: 8 9 9 10
33800 274/300 Read block 2335 len 16384
33801 275/300 Read block 3461 len 32768
33802 276/300 Write block 3084 len 28672 data: 6 7 6 6 5 6 5
33803 277/300 Flush
33804 278/300 Read block 811 len 28672
33805 279/300 Read block 820 len 40960
33806 280/300 Read block 3829 len 12288
33807 281/300 Write block 2255 len 40960 data: 4 5 6 6 7 5 4 5 5 5
33808 282/300 Flush
33809 283/300 Read block 542 len 12288
33810 284/300 Read block 197 len 24576
33811 285/300 Write block 1474 len 8192 data: 10 8
33812 286/300 Read block 1078 len 40960
33813 287/300 Write block 4631 len 36864 data: 8 10 8 7 7 8 7 7 6
33814 288/300 Write block 3483 len 36864 data: 8 6 7 7 7 6 3 3 4
33815 289/300 Read block 3945 len 36864
33816 290/300 Write block 3059 len 36864 data: 5 5 5 5 6 5 5 5 5
33817 291/300 Read block 4159 len 32768
33818 292/300 Flush
33819 293/300 Read block 39 len 20480
33820 294/300 Read block 2828 len 20480
33821 295/300 Read block 3333 len 16384
33822 296/300 Write block 2581 len 40960 data: 6 5 5 7 8 7 5 4 4 4
33823 297/300 Write block 4113 len 8192 data: 10 10
33824 298/300 Read block 4071 len 16384
33825 299/300 Read block 1311 len 4096
33826 300/300 Flush
33827 301/300 Read block 958 len 4096
33828 Main task has work to do, go find it
33829 got dsc Start(2)
33830 start 2
33831 Nov 01 16:57:37.314 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: aece7143-1f91-44c3-82c0-e1f6c19ceae5, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
33832 Replay: started 2, returned:Ok(())
33833 ----------------------------------------------------------------
33834 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33835 States: Active Active Offline
33836 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33837 9764 Acked Flush 0 Done Done Sent false
33838 9765 Acked Write 8 Done Done Sent false
33839 9766 Acked Write 3 Done Done Sent false
33840 9767 Acked Flush 0 Done Done Sent false
33841 9768 Acked Write 6 Done Done Sent false
33842 9769 Acked Read 1 Done Done Sent false
33843 9770 Acked Read 5 Done Done Sent false
33844 9771 Acked Write 2 Done Done Sent false
33845 9772 Acked Write 2 Done Done Sent false
33846 9773 Acked Write 9 Done Done Sent false
33847 9774 Acked Read 7 Done Done Sent false
33848 9775 Acked Write 5 Done Done Sent false
33849 9776 Acked Write 3 Done Done Sent false
33850 9777 Acked Write 6 Done Done Sent false
33851 9778 Acked Read 9 Done Done Sent false
33852 9779 Acked Write 5 Done Done Sent false
33853 9780 Acked Flush 0 Done Done Sent false
33854 9781 Acked Read 10 Done Done Sent false
33855 9782 Acked Read 3 Done Done Sent false
33856 9783 Acked Read 5 Done Done Sent false
33857 9784 Acked Write 2 Done Done Sent false
33858 9785 Acked Write 2 Done Done Sent false
33859 9786 Acked Write 1 Done Done Sent false
33860 9787 Acked Read 6 Done Done Sent false
33861 9788 Acked Flush 0 Done Done Sent false
33862 9789 Acked Write 9 Done Done Sent false
33863 9790 Acked Write 10 Done Done Sent false
33864 9791 Acked Write 7 Done Done Sent false
33865 9792 Acked Read 3 Done Done Sent false
33866 9793 Acked Flush 0 Done Done Sent false
33867 9794 Acked Read 9 Done Done Sent false
33868 9795 Acked Write 1 Done Done Sent false
33869 9796 Acked Read 9 Done Done Sent false
33870 9797 Acked Read 7 Done Done Sent false
33871 9798 Acked Write 1 Done Done Sent false
33872 9799 Acked Read 7 Done Done Sent false
33873 9800 Acked Write 7 Done Done Sent false
33874 9801 Acked Write 10 Done Done Sent false
33875 9802 Acked Write 3 Done Done Sent false
33876 9803 Acked Read 7 Done Done Sent false
33877 9804 Acked Write 9 Done Done Sent false
33878 9805 Acked Read 10 Done Done Sent false
33879 9806 Acked Read 6 Done Done Sent false
33880 9807 Acked Read 9 Done Done Sent false
33881 9808 Acked Flush 0 Done Done Sent false
33882 9809 Acked Read 8 Done Done Sent false
33883 9810 Acked Write 5 Done Done Sent false
33884 9811 Acked Read 1 Done Done Sent false
33885 9812 Acked Write 10 Done Done Sent false
33886 9813 Acked Write 5 Done Done Sent false
33887 9814 Acked Flush 0 Done Done Sent false
33888 9815 Acked Write 3 Done Done Sent false
33889 9816 Acked Write 9 Done Done Sent false
33890 9817 Acked Flush 0 Done Done Sent false
33891 9818 Acked Read 9 Done Done Sent false
33892 9819 Acked Read 3 Done Done Sent false
33893 9820 Acked Read 6 Done Done Sent false
33894 9821 Acked Write 3 Done Done Sent false
33895 9822 Acked Read 3 Done Done Sent false
33896 9823 Acked Read 3 Done Done Sent false
33897 9824 Acked Write 7 Done Done Sent false
33898 9825 Acked Write 1 Done Done Sent false
33899 9826 Acked Read 7 Done Done Sent false
33900 9827 Acked Read 10 Done Done Sent false
33901 9828 Acked Write 6 Done Done Sent false
33902 9829 Acked Read 9 Done Done Sent false
33903 9830 Acked Write 5 Done Done Sent false
33904 9831 Acked Read 9 Done Done Sent false
33905 9832 Acked Write 6 Done Done Sent false
33906 9833 Acked Write 2 Done Done Sent false
33907 9834 Acked Read 3 Done Done Sent false
33908 9835 Acked Read 3 Done Done Sent false
33909 9836 Acked Read 4 Done Done Sent false
33910 9837 Acked Read 10 Done Done Sent false
33911 9838 Acked Read 10 Done Done Sent false
33912 9839 Acked Read 2 Done Done Sent false
33913 9840 Acked Write 2 Done Done Sent false
33914 9841 Acked Read 3 Done Done Sent false
33915 9842 Acked Read 1 Done Done Sent false
33916 9843 Acked Write 7 Done Done Sent false
33917 9844 Acked Read 10 Done Done Sent false
33918 9845 Acked Write 1 Done Done Sent false
33919 9846 Acked Read 10 Done Done Sent false
33920 9847 Acked Read 8 Done Done Sent false
33921 9848 Acked Write 10 Done Done Sent false
33922 9849 Acked Read 7 Done Done Sent false
33923 9850 Acked Write 6 Done Done Sent false
33924 9851 Acked Write 10 Done Done Sent false
33925 9852 Acked Read 8 Done Done Sent false
33926 9853 Acked Write 3 Done Done Sent false
33927 9854 Acked Write 3 Done Done Sent false
33928 9855 Acked Write 6 Done Done Sent false
33929 9856 Acked Read 2 Done Done Sent false
33930 9857 Acked Write 9 Done Done Sent false
33931 9858 Acked Flush 0 Done Done Sent false
33932 9859 Acked Read 4 Done Done Sent false
33933 9860 Acked Write 6 Done Done Sent false
33934 9861 Acked Write 8 Done Done Sent false
33935 9862 Acked Read 3 Done Done Sent false
33936 9863 Acked Write 3 Done Done Sent false
33937 9864 Acked Write 7 Done Done Sent false
33938 9865 Acked Read 9 Done Done Sent false
33939 9866 Acked Write 2 Done Done Sent false
33940 9867 Acked Write 9 Done Done Sent false
33941 9868 Acked Write 10 Done Done Sent false
33942 9869 Acked Read 7 Done Done Sent false
33943 9870 Acked Read 4 Done Done Sent false
33944 9871 Acked Write 10 Done Done Sent false
33945 9872 Acked Read 9 Done Done Sent false
33946 9873 Acked Read 6 Done Done Sent false
33947 9874 Acked Write 2 Done Done Sent false
33948 9875 Acked Read 6 Done Done Sent false
33949 9876 Acked Write 10 Done Done Sent false
33950 9877 Acked Read 4 Done Done Sent false
33951 9878 Acked Read 9 Done Done Sent false
33952 9879 Acked Write 5 Done Done Sent false
33953 9880 Acked Read 9 Done Done Sent false
33954 9881 Acked Write 3 Done Done Sent false
33955 9882 Acked Write 6 Done Done Sent false
33956 9883 Acked Flush 0 Done Done Sent false
33957 9884 Acked Write 6 Done Done Sent false
33958 9885 Acked Write 1 Done Done Sent false
33959 9886 Acked Read 5 Done Done Sent false
33960 9887 Acked Write 6 Done Done Sent false
33961 9888 Acked Read 10 Done Done Sent false
33962 9889 Acked Read 9 Done Done Sent false
33963 9890 Acked Read 8 Done Done Sent false
33964 9891 Acked Write 5 Done Done Sent false
33965 9892 Acked Read 2 Done Done Sent false
33966 9893 Acked Read 10 Done Done Sent false
33967 9894 Acked Flush 0 Done Done Sent false
33968 9895 Acked Read 3 Done Done Sent false
33969 9896 Acked Write 10 Done Done Sent false
33970 9897 Acked Read 7 Done Done Sent false
33971 9898 Acked Write 7 Done Done Sent false
33972 9899 Acked Write 2 Done Done Sent false
33973 9900 Acked Flush 0 Done Done Sent false
33974 9901 Acked Write 7 Done Done Sent false
33975 9902 Acked Read 7 Done Done Sent false
33976 9903 Acked Read 6 Done Done Sent false
33977 9904 Acked Flush 0 Done Done Sent false
33978 9905 Acked Write 3 Done Done Sent false
33979 9906 Acked Write 8 Done Done Sent false
33980 9907 Acked Write 8 Done Done Sent false
33981 9908 Acked Write 3 Done Done Sent false
33982 9909 Acked Read 4 Done Done Sent false
33983 9910 Acked Read 5 Done Done Sent false
33984 9911 Acked Write 6 Done Done Sent false
33985 9912 Acked Write 10 Done Done Sent false
33986 9913 Acked Read 5 Done Done Sent false
33987 9914 Acked Write 1 Done Done Sent false
33988 9915 Acked Write 10 Done Done Sent false
33989 9916 Acked Read 1 Done Done Sent false
33990 9917 Acked Write 7 Done Done Sent false
33991 9918 Acked Write 8 Done Done Sent false
33992 9919 Acked Write 10 Done Done Sent false
33993 9920 Acked Write 8 Done Done Sent false
33994 9921 Acked Write 4 Done Done Sent false
33995 9922 Acked Flush 0 Done Done Sent false
33996 9923 Acked Read 1 Done Done Sent false
33997 9924 Acked Write 6 Done Done Sent false
33998 9925 Acked Flush 0 Done Done Sent false
33999 9926 Acked Read 3 Done Done Sent false
34000 9927 Acked Write 2 Done Done Sent false
34001 9928 Acked Read 6 Done Done Sent false
34002 9929 Acked Write 10 Done Done Sent false
34003 9930 Acked Write 8 Done Done Sent false
34004 9931 Acked Read 8 Done Done Sent false
34005 9932 Acked Write 2 Done Done Sent false
34006 9933 Acked Read 5 Done Done Sent false
34007 9934 Acked Write 10 Done Done Sent false
34008 9935 Acked Read 1 Done Done Sent false
34009 9936 Acked Write 3 Done Done Sent false
34010 9937 Acked Read 6 Done Done Sent false
34011 9938 Acked Read 10 Done Done Sent false
34012 9939 Acked Write 6 Done Done Sent false
34013 9940 Acked Read 1 Done Done Sent false
34014 9941 Acked Write 8 Done Done Sent false
34015 9942 Acked Flush 0 Done Done Sent false
34016 9943 Acked Write 8 Done Done Sent false
34017 9944 Acked Read 10 Done Done Sent false
34018 9945 Acked Write 5 Done Done Sent false
34019 9946 Acked Write 4 Done Done Sent false
34020 9947 Acked Write 8 Done Done Sent false
34021 9948 Acked Read 3 Done Done Sent false
34022 9949 Acked Read 8 Done Done Sent false
34023 9950 Acked Read 8 Done Done Sent false
34024 9951 Acked Write 6 Done Done Sent false
34025 9952 Acked Read 10 Done Done Sent false
34026 9953 Acked Read 7 Done Done Sent false
34027 9954 Acked Flush 0 Done Done Sent false
34028 9955 Acked Read 8 Done Done Sent false
34029 9956 Acked Read 4 Done Done Sent false
34030 9957 Acked Read 8 Done Done Sent false
34031 9958 Acked Write 9 Done Done Sent false
34032 9959 Acked Read 10 Done Done Sent false
34033 9960 Acked Write 2 Done Done Sent false
34034 9961 Acked Read 3 Done Done Sent false
34035 9962 Acked Read 7 Done Done Sent false
34036 9963 Acked Write 5 Done Done Sent false
34037 9964 Acked Write 9 Done Done Sent false
34038 9965 Acked Write 7 Done Done Sent false
34039 9966 Acked Read 9 Done Done Sent false
34040 9967 Acked Flush 0 Done Done Sent false
34041 9968 Acked Read 1 Done Done Sent false
34042 9969 Acked Read 3 Done Done Sent false
34043 9970 Acked Write 7 Done Done Sent false
34044 9971 Acked Read 4 Done Done Sent false
34045 9972 Acked Read 1 Done Done Sent false
34046 9973 Acked Write 9 Done Done Sent false
34047 9974 Acked Flush 0 Done Done Sent false
34048 9975 Acked Read 7 Done Done Sent false
34049 9976 Acked Write 1 Done Done Sent false
34050 9977 Acked Flush 0 Done Done Sent false
34051 9978 Acked Write 6 Done Done Sent false
34052 9979 Acked Write 4 Done Done Sent false
34053 9980 Acked Read 2 Done Done Sent false
34054 9981 Acked Write 10 Done Done Sent false
34055 9982 Acked Read 1 Done Done Sent false
34056 9983 Acked Read 8 Done Done Sent false
34057 9984 Acked Write 3 Done Done Sent false
34058 9985 Acked Flush 0 Done Done Sent false
34059 9986 Acked Flush 0 Done Done Sent false
34060 9987 Acked Read 4 Done Done Sent false
34061 9988 Acked Write 9 Done Done Sent false
34062 9989 Acked Write 2 Done Done Sent false
34063 9990 Acked Write 9 Done Done Sent false
34064 9991 Acked Read 9 Done Done Sent false
34065 9992 Acked Write 8 Done Done Sent false
34066 9993 Acked Read 10 Done Done Sent false
34067 9994 Acked Write 8 Done Done Sent false
34068 9995 Acked Write 6 Done Done Sent false
34069 9996 Acked Read 5 Done Done Sent false
34070 9997 Acked Write 6 Done Done Sent false
34071 9998 Acked Read 7 Done Done Sent false
34072 9999 Acked Read 9 Done Done Sent false
34073 10000 Acked Write 3 Done Done Sent false
34074 10001 Acked Write 2 Done Done Sent false
34075 10002 Acked Read 6 Done Done Sent false
34076 10003 Acked Write 1 Done Done Sent false
34077 10004 Acked Flush 0 Done Done Sent false
34078 10005 Acked Read 4 Done Done Sent false
34079 10006 Acked Write 10 Done Done Sent false
34080 10007 Acked Write 7 Done Done Sent false
34081 10008 Acked Read 4 Done Done Sent false
34082 10009 Acked Flush 0 Done Done Sent false
34083 10010 Acked Write 7 Done Done Sent false
34084 10011 Acked Read 6 Done Done Sent false
34085 10012 Acked Write 10 Done Done Sent false
34086 10013 Acked Write 2 Done Done Sent false
34087 10014 Acked Read 3 Done Done Sent false
34088 10015 Acked Read 8 Done Done Sent false
34089 10016 Acked Flush 0 Done Done Sent false
34090 10017 Acked Read 9 Done Done Sent false
34091 10018 Acked Write 1 Done Done Sent false
34092 10019 Acked Read 4 Done Done Sent false
34093 10020 Acked Read 7 Done Done Sent false
34094 10021 Acked Write 7 Done Done Sent false
34095 10022 Acked Write 4 Done Done Sent false
34096 10023 Acked Read 2 Done Done Sent false
34097 10024 Acked Read 1 Done Done Sent false
34098 10025 Acked Read 7 Done Done Sent false
34099 10026 Acked Read 9 Done Done Sent false
34100 10027 Acked Read 10 Done Done Sent false
34101 10028 Acked Read 5 Done Done Sent false
34102 10029 Acked Read 6 Done Done Sent false
34103 10030 Acked Read 5 Done Done Sent false
34104 10031 Acked Write 2 Done Done Sent false
34105 10032 Acked Read 7 Done Done Sent false
34106 10033 Acked Read 4 Done Done Sent false
34107 10034 Acked Read 1 Done Done Sent false
34108 10035 Acked Read 10 Done Done Sent false
34109 10036 Acked Write 4 Done Done Sent false
34110 10037 Acked Read 4 Done Done Sent false
34111 10038 Acked Read 8 Done Done Sent false
34112 10039 Acked Write 7 Done Done Sent false
34113 10040 Acked Flush 0 Done Done Sent false
34114 10041 Acked Read 7 Done Done Sent false
34115 10042 Acked Read 10 Done Done Sent false
34116 10043 Acked Read 3 Done Done Sent false
34117 10044 Acked Write 10 Done Done Sent false
34118 10045 Acked Flush 0 Done Done Sent false
34119 10046 Acked Read 3 Done Done Sent false
34120 10047 Acked Read 6 Done Done Sent false
34121 10048 Acked Write 2 Done Done Sent false
34122 10049 Acked Read 10 Done Done Sent false
34123 10050 Acked Write 9 Done Done Sent false
34124 10051 Acked Write 9 Done Done Sent false
34125 10052 Acked Read 9 Done Done Sent false
34126 10053 Acked Write 9 Done Done Sent false
34127 10054 Acked Read 8 Done Done Sent false
34128 10055 Acked Flush 0 Done Done Sent false
34129 10056 Acked Read 5 Done Done Sent false
34130 10057 Acked Read 5 Done Done Sent false
34131 10058 Acked Read 4 Done Done Sent false
34132 10059 Acked Write 10 Done Done Sent false
34133 10060 Acked Write 2 Done Done Sent false
34134 10061 Acked Read 4 Done Done Sent false
34135 10062 Acked Read 1 Done Done Sent false
34136 10063 Acked Flush 0 Done Done Sent false
34137 10064 Acked Read 1 Done Done Sent false
34138 STATES DS:0 DS:1 DS:2 TOTAL
34139 Sent 0 0 301 301
34140 Done 301 301 0 602
34141 Skipped 0 0 0 0
34142 Error 0 0 0 0
34143 Last Flush: 10063 10063 9763
34144 Downstairs last five completed: 9763 9762 9761 9760 9759
34145 Upstairs last five completed: 10064 10063 10062 10061 10060
34146 CLIENT: Up:0 ds:301 act:2
34147 ----------------------------------------------------------------
34148 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
34149 States: Active Active Offline
34150 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34151 9764 Acked Flush 0 Done Done Sent false
34152 9765 Acked Write 8 Done Done Sent false
34153 9766 Acked Write 3 Done Done Sent false
34154 9767 Acked Flush 0 Done Done Sent false
34155 9768 Acked Write 6 Done Done Sent false
34156 9769 Acked Read 1 Done Done Sent false
34157 9770 Acked Read 5 Done Done Sent false
34158 9771 Acked Write 2 Done Done Sent false
34159 9772 Acked Write 2 Done Done Sent false
34160 9773 Acked Write 9 Done Done Sent false
34161 9774 Acked Read 7 Done Done Sent false
34162 9775 Acked Write 5 Done Done Sent false
34163 9776 Acked Write 3 Done Done Sent false
34164 9777 Acked Write 6 Done Done Sent false
34165 9778 Acked Read 9 Done Done Sent false
34166 9779 Acked Write 5 Done Done Sent false
34167 9780 Acked Flush 0 Done Done Sent false
34168 9781 Acked Read 10 Done Done Sent false
34169 9782 Acked Read 3 Done Done Sent false
34170 9783 Acked Read 5 Done Done Sent false
34171 9784 Acked Write 2 Done Done Sent false
34172 9785 Acked Write 2 Done Done Sent false
34173 9786 Acked Write 1 Done Done Sent false
34174 9787 Acked Read 6 Done Done Sent false
34175 9788 Acked Flush 0 Done Done Sent false
34176 9789 Acked Write 9 Done Done Sent false
34177 9790 Acked Write 10 Done Done Sent false
34178 9791 Acked Write 7 Done Done Sent false
34179 9792 Acked Read 3 Done Done Sent false
34180 9793 Acked Flush 0 Done Done Sent false
34181 9794 Acked Read 9 Done Done Sent false
34182 9795 Acked Write 1 Done Done Sent false
34183 9796 Acked Read 9 Done Done Sent false
34184 9797 Acked Read 7 Done Done Sent false
34185 9798 Acked Write 1 Done Done Sent false
34186 9799 Acked Read 7 Done Done Sent false
34187 9800 Acked Write 7 Done Done Sent false
34188 9801 Acked Write 10 Done Done Sent false
34189 9802 Acked Write 3 Done Done Sent false
34190 9803 Acked Read 7 Done Done Sent false
34191 9804 Acked Write 9 Done Done Sent false
34192 9805 Acked Read 10 Done Done Sent false
34193 9806 Acked Read 6 Done Done Sent false
34194 9807 Acked Read 9 Done Done Sent false
34195 9808 Acked Flush 0 Done Done Sent false
34196 9809 Acked Read 8 Done Done Sent false
34197 9810 Acked Write 5 Done Done Sent false
34198 9811 Acked Read 1 Done Done Sent false
34199 9812 Acked Write 10 Done Done Sent false
34200 9813 Acked Write 5 Done Done Sent false
34201 9814 Acked Flush 0 Done Done Sent false
34202 9815 Acked Write 3 Done Done Sent false
34203 9816 Acked Write 9 Done Done Sent false
34204 9817 Acked Flush 0 Done Done Sent false
34205 9818 Acked Read 9 Done Done Sent false
34206 9819 Acked Read 3 Done Done Sent false
34207 9820 Acked Read 6 Done Done Sent false
34208 9821 Acked Write 3 Done Done Sent false
34209 9822 Acked Read 3 Done Done Sent false
34210 9823 Acked Read 3 Done Done Sent false
34211 9824 Acked Write 7 Done Done Sent false
34212 9825 Acked Write 1 Done Done Sent false
34213 9826 Acked Read 7 Done Done Sent false
34214 9827 Acked Read 10 Done Done Sent false
34215 9828 Acked Write 6 Done Done Sent false
34216 9829 Acked Read 9 Done Done Sent false
34217 9830 Acked Write 5 Done Done Sent false
34218 9831 Acked Read 9 Done Done Sent false
34219 9832 Acked Write 6 Done Done Sent false
34220 9833 Acked Write 2 Done Done Sent false
34221 9834 Acked Read 3 Done Done Sent false
34222 9835 Acked Read 3 Done Done Sent false
34223 9836 Acked Read 4 Done Done Sent false
34224 9837 Acked Read 10 Done Done Sent false
34225 9838 Acked Read 10 Done Done Sent false
34226 9839 Acked Read 2 Done Done Sent false
34227 9840 Acked Write 2 Done Done Sent false
34228 9841 Acked Read 3 Done Done Sent false
34229 9842 Acked Read 1 Done Done Sent false
34230 9843 Acked Write 7 Done Done Sent false
34231 9844 Acked Read 10 Done Done Sent false
34232 9845 Acked Write 1 Done Done Sent false
34233 9846 Acked Read 10 Done Done Sent false
34234 9847 Acked Read 8 Done Done Sent false
34235 9848 Acked Write 10 Done Done Sent false
34236 9849 Acked Read 7 Done Done Sent false
34237 9850 Acked Write 6 Done Done Sent false
34238 9851 Acked Write 10 Done Done Sent false
34239 9852 Acked Read 8 Done Done Sent false
34240 9853 Acked Write 3 Done Done Sent false
34241 9854 Acked Write 3 Done Done Sent false
34242 9855 Acked Write 6 Done Done Sent false
34243 9856 Acked Read 2 Done Done Sent false
34244 9857 Acked Write 9 Done Done Sent false
34245 9858 Acked Flush 0 Done Done Sent false
34246 9859 Acked Read 4 Done Done Sent false
34247 9860 Acked Write 6 Done Done Sent false
34248 9861 Acked Write 8 Done Done Sent false
34249 9862 Acked Read 3 Done Done Sent false
34250 9863 Acked Write 3 Done Done Sent false
34251 9864 Acked Write 7 Done Done Sent false
34252 9865 Acked Read 9 Done Done Sent false
34253 9866 Acked Write 2 Done Done Sent false
34254 9867 Acked Write 9 Done Done Sent false
34255 9868 Acked Write 10 Done Done Sent false
34256 9869 Acked Read 7 Done Done Sent false
34257 9870 Acked Read 4 Done Done Sent false
34258 9871 Acked Write 10 Done Done Sent false
34259 9872 Acked Read 9 Done Done Sent false
34260 9873 Acked Read 6 Done Done Sent false
34261 9874 Acked Write 2 Done Done Sent false
34262 9875 Acked Read 6 Done Done Sent false
34263 9876 Acked Write 10 Done Done Sent false
34264 9877 Acked Read 4 Done Done Sent false
34265 9878 Acked Read 9 Done Done Sent false
34266 9879 Acked Write 5 Done Done Sent false
34267 9880 Acked Read 9 Done Done Sent false
34268 9881 Acked Write 3 Done Done Sent false
34269 9882 Acked Write 6 Done Done Sent false
34270 9883 Acked Flush 0 Done Done Sent false
34271 9884 Acked Write 6 Done Done Sent false
34272 9885 Acked Write 1 Done Done Sent false
34273 9886 Acked Read 5 Done Done Sent false
34274 9887 Acked Write 6 Done Done Sent false
34275 9888 Acked Read 10 Done Done Sent false
34276 9889 Acked Read 9 Done Done Sent false
34277 9890 Acked Read 8 Done Done Sent false
34278 9891 Acked Write 5 Done Done Sent false
34279 9892 Acked Read 2 Done Done Sent false
34280 9893 Acked Read 10 Done Done Sent false
34281 9894 Acked Flush 0 Done Done Sent false
34282 9895 Acked Read 3 Done Done Sent false
34283 9896 Acked Write 10 Done Done Sent false
34284 9897 Acked Read 7 Done Done Sent false
34285 9898 Acked Write 7 Done Done Sent false
34286 9899 Acked Write 2 Done Done Sent false
34287 9900 Acked Flush 0 Done Done Sent false
34288 9901 Acked Write 7 Done Done Sent false
34289 9902 Acked Read 7 Done Done Sent false
34290 9903 Acked Read 6 Done Done Sent false
34291 9904 Acked Flush 0 Done Done Sent false
34292 9905 Acked Write 3 Done Done Sent false
34293 9906 Acked Write 8 Done Done Sent false
34294 9907 Acked Write 8 Done Done Sent false
34295 9908 Acked Write 3 Done Done Sent false
34296 9909 Acked Read 4 Done Done Sent false
34297 9910 Acked Read 5 Done Done Sent false
34298 9911 Acked Write 6 Done Done Sent false
34299 9912 Acked Write 10 Done Done Sent false
34300 9913 Acked Read 5 Done Done Sent false
34301 9914 Acked Write 1 Done Done Sent false
34302 9915 Acked Write 10 Done Done Sent false
34303 9916 Acked Read 1 Done Done Sent false
34304 9917 Acked Write 7 Done Done Sent false
34305 9918 Acked Write 8 Done Done Sent false
34306 9919 Acked Write 10 Done Done Sent false
34307 9920 Acked Write 8 Done Done Sent false
34308 9921 Acked Write 4 Done Done Sent false
34309 9922 Acked Flush 0 Done Done Sent false
34310 9923 Acked Read 1 Done Done Sent false
34311 9924 Acked Write 6 Done Done Sent false
34312 9925 Acked Flush 0 Done Done Sent false
34313 9926 Acked Read 3 Done Done Sent false
34314 9927 Acked Write 2 Done Done Sent false
34315 9928 Acked Read 6 Done Done Sent false
34316 9929 Acked Write 10 Done Done Sent false
34317 9930 Acked Write 8 Done Done Sent false
34318 9931 Acked Read 8 Done Done Sent false
34319 9932 Acked Write 2 Done Done Sent false
34320 9933 Acked Read 5 Done Done Sent false
34321 9934 Acked Write 10 Done Done Sent false
34322 9935 Acked Read 1 Done Done Sent false
34323 9936 Acked Write 3 Done Done Sent false
34324 9937 Acked Read 6 Done Done Sent false
34325 9938 Acked Read 10 Done Done Sent false
34326 9939 Acked Write 6 Done Done Sent false
34327 9940 Acked Read 1 Done Done Sent false
34328 9941 Acked Write 8 Done Done Sent false
34329 9942 Acked Flush 0 Done Done Sent false
34330 9943 Acked Write 8 Done Done Sent false
34331 9944 Acked Read 10 Done Done Sent false
34332 9945 Acked Write 5 Done Done Sent false
34333 9946 Acked Write 4 Done Done Sent false
34334 9947 Acked Write 8 Done Done Sent false
34335 9948 Acked Read 3 Done Done Sent false
34336 9949 Acked Read 8 Done Done Sent false
34337 9950 Acked Read 8 Done Done Sent false
34338 9951 Acked Write 6 Done Done Sent false
34339 9952 Acked Read 10 Done Done Sent false
34340 9953 Acked Read 7 Done Done Sent false
34341 9954 Acked Flush 0 Done Done Sent false
34342 9955 Acked Read 8 Done Done Sent false
34343 9956 Acked Read 4 Done Done Sent false
34344 9957 Acked Read 8 Done Done Sent false
34345 9958 Acked Write 9 Done Done Sent false
34346 9959 Acked Read 10 Done Done Sent false
34347 9960 Acked Write 2 Done Done Sent false
34348 9961 Acked Read 3 Done Done Sent false
34349 9962 Acked Read 7 Done Done Sent false
34350 9963 Acked Write 5 Done Done Sent false
34351 9964 Acked Write 9 Done Done Sent false
34352 9965 Acked Write 7 Done Done Sent false
34353 9966 Acked Read 9 Done Done Sent false
34354 9967 Acked Flush 0 Done Done Sent false
34355 9968 Acked Read 1 Done Done Sent false
34356 9969 Acked Read 3 Done Done Sent false
34357 9970 Acked Write 7 Done Done Sent false
34358 9971 Acked Read 4 Done Done Sent false
34359 9972 Acked Read 1 Done Done Sent false
34360 9973 Acked Write 9 Done Done Sent false
34361 9974 Acked Flush 0 Done Done Sent false
34362 9975 Acked Read 7 Done Done Sent false
34363 9976 Acked Write 1 Done Done Sent false
34364 9977 Acked Flush 0 Done Done Sent false
34365 9978 Acked Write 6 Done Done Sent false
34366 9979 Acked Write 4 Done Done Sent false
34367 9980 Acked Read 2 Done Done Sent false
34368 9981 Acked Write 10 Done Done Sent false
34369 9982 Acked Read 1 Done Done Sent false
34370 9983 Acked Read 8 Done Done Sent false
34371 9984 Acked Write 3 Done Done Sent false
34372 9985 Acked Flush 0 Done Done Sent false
34373 9986 Acked Flush 0 Done Done Sent false
34374 9987 Acked Read 4 Done Done Sent false
34375 9988 Acked Write 9 Done Done Sent false
34376 9989 Acked Write 2 Done Done Sent false
34377 9990 Acked Write 9 Done Done Sent false
34378 9991 Acked Read 9 Done Done Sent false
34379 9992 Acked Write 8 Done Done Sent false
34380 9993 Acked Read 10 Done Done Sent false
34381 9994 Acked Write 8 Done Done Sent false
34382 9995 Acked Write 6 Done Done Sent false
34383 9996 Acked Read 5 Done Done Sent false
34384 9997 Acked Write 6 Done Done Sent false
34385 9998 Acked Read 7 Done Done Sent false
34386 9999 Acked Read 9 Done Done Sent false
34387 10000 Acked Write 3 Done Done Sent false
34388 10001 Acked Write 2 Done Done Sent false
34389 10002 Acked Read 6 Done Done Sent false
34390 10003 Acked Write 1 Done Done Sent false
34391 10004 Acked Flush 0 Done Done Sent false
34392 10005 Acked Read 4 Done Done Sent false
34393 10006 Acked Write 10 Done Done Sent false
34394 10007 Acked Write 7 Done Done Sent false
34395 10008 Acked Read 4 Done Done Sent false
34396 10009 Acked Flush 0 Done Done Sent false
34397 10010 Acked Write 7 Done Done Sent false
34398 10011 Acked Read 6 Done Done Sent false
34399 10012 Acked Write 10 Done Done Sent false
34400 10013 Acked Write 2 Done Done Sent false
34401 10014 Acked Read 3 Done Done Sent false
34402 10015 Acked Read 8 Done Done Sent false
34403 10016 Acked Flush 0 Done Done Sent false
34404 10017 Acked Read 9 Done Done Sent false
34405 10018 Acked Write 1 Done Done Sent false
34406 10019 Acked Read 4 Done Done Sent false
34407 10020 Acked Read 7 Done Done Sent false
34408 10021 Acked Write 7 Done Done Sent false
34409 10022 Acked Write 4 Done Done Sent false
34410 10023 Acked Read 2 Done Done Sent false
34411 10024 Acked Read 1 Done Done Sent false
34412 10025 Acked Read 7 Done Done Sent false
34413 10026 Acked Read 9 Done Done Sent false
34414 10027 Acked Read 10 Done Done Sent false
34415 10028 Acked Read 5 Done Done Sent false
34416 10029 Acked Read 6 Done Done Sent false
34417 10030 Acked Read 5 Done Done Sent false
34418 10031 Acked Write 2 Done Done Sent false
34419 10032 Acked Read 7 Done Done Sent false
34420 10033 Acked Read 4 Done Done Sent false
34421 10034 Acked Read 1 Done Done Sent false
34422 10035 Acked Read 10 Done Done Sent false
34423 10036 Acked Write 4 Done Done Sent false
34424 10037 Acked Read 4 Done Done Sent false
34425 10038 Acked Read 8 Done Done Sent false
34426 10039 Acked Write 7 Done Done Sent false
34427 10040 Acked Flush 0 Done Done Sent false
34428 10041 Acked Read 7 Done Done Sent false
34429 10042 Acked Read 10 Done Done Sent false
34430 10043 Acked Read 3 Done Done Sent false
34431 10044 Acked Write 10 Done Done Sent false
34432 10045 Acked Flush 0 Done Done Sent false
34433 10046 Acked Read 3 Done Done Sent false
34434 10047 Acked Read 6 Done Done Sent false
34435 10048 Acked Write 2 Done Done Sent false
34436 10049 Acked Read 10 Done Done Sent false
34437 10050 Acked Write 9 Done Done Sent false
34438 10051 Acked Write 9 Done Done Sent false
34439 10052 Acked Read 9 Done Done Sent false
34440 10053 Acked Write 9 Done Done Sent false
34441 10054 Acked Read 8 Done Done Sent false
34442 10055 Acked Flush 0 Done Done Sent false
34443 10056 Acked Read 5 Done Done Sent false
34444 10057 Acked Read 5 Done Done Sent false
34445 10058 Acked Read 4 Done Done Sent false
34446 10059 Acked Write 10 Done Done Sent false
34447 10060 Acked Write 2 Done Done Sent false
34448 10061 Acked Read 4 Done Done Sent false
34449 10062 Acked Read 1 Done Done Sent false
34450 10063 Acked Flush 0 Done Done Sent false
34451 10064 Acked Read 1 Done Done Sent false
34452 10065 Acked Flush 0 Done Done Sent false
34453 STATES DS:0 DS:1 DS:2 TOTAL
34454 Sent 0 0 302 302
34455 Done 302 302 0 604
34456 Skipped 0 0 0 0
34457 Error 0 0 0 0
34458 Last Flush: 10065 10065 9763
34459 Downstairs last five completed: 9763 9762 9761 9760 9759
34460 Upstairs last five completed: 10065 10064 10063 10062 10061
34461 CLIENT: Up:0 ds:302 act:2
344622024-11-01T16:57:43.013ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
344632024-11-01T16:57:43.013ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
344642024-11-01T16:57:43.013ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
344652024-11-01T16:57:43.013ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
344662024-11-01T16:57:43.013ZINFOcrucible: client task is exiting = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
34467 [8830] got start action
34468 [8830] I am going to restart
34469 Starting downstairs at port 8830
34470 Make output file at "/tmp/dsc/downstairs-8830.txt"
34471 [8830][2] reports Starting
34472 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1228)
34473 [8830][2] reports Running
34474 ----------------------------------------------------------------
34475 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
34476 States: Active Active Offline
34477 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34478 9764 Acked Flush 0 Done Done Sent false
34479 9765 Acked Write 8 Done Done Sent false
34480 9766 Acked Write 3 Done Done Sent false
34481 9767 Acked Flush 0 Done Done Sent false
34482 9768 Acked Write 6 Done Done Sent false
34483 9769 Acked Read 1 Done Done Sent false
34484 9770 Acked Read 5 Done Done Sent false
34485 9771 Acked Write 2 Done Done Sent false
34486 9772 Acked Write 2 Done Done Sent false
34487 9773 Acked Write 9 Done Done Sent false
34488 9774 Acked Read 7 Done Done Sent false
34489 9775 Acked Write 5 Done Done Sent false
34490 9776 Acked Write 3 Done Done Sent false
34491 9777 Acked Write 6 Done Done Sent false
34492 9778 Acked Read 9 Done Done Sent false
34493 9779 Acked Write 5 Done Done Sent false
34494 9780 Acked Flush 0 Done Done Sent false
34495 9781 Acked Read 10 Done Done Sent false
34496 9782 Acked Read 3 Done Done Sent false
34497 9783 Acked Read 5 Done Done Sent false
34498 9784 Acked Write 2 Done Done Sent false
34499 9785 Acked Write 2 Done Done Sent false
34500 9786 Acked Write 1 Done Done Sent false
34501 9787 Acked Read 6 Done Done Sent false
34502 9788 Acked Flush 0 Done Done Sent false
34503 9789 Acked Write 9 Done Done Sent false
34504 9790 Acked Write 10 Done Done Sent false
34505 9791 Acked Write 7 Done Done Sent false
34506 9792 Acked Read 3 Done Done Sent false
34507 9793 Acked Flush 0 Done Done Sent false
34508 9794 Acked Read 9 Done Done Sent false
34509 9795 Acked Write 1 Done Done Sent false
34510 9796 Acked Read 9 Done Done Sent false
34511 9797 Acked Read 7 Done Done Sent false
34512 9798 Acked Write 1 Done Done Sent false
34513 9799 Acked Read 7 Done Done Sent false
34514 9800 Acked Write 7 Done Done Sent false
34515 9801 Acked Write 10 Done Done Sent false
34516 9802 Acked Write 3 Done Done Sent false
34517 9803 Acked Read 7 Done Done Sent false
34518 9804 Acked Write 9 Done Done Sent false
34519 9805 Acked Read 10 Done Done Sent false
34520 9806 Acked Read 6 Done Done Sent false
34521 9807 Acked Read 9 Done Done Sent false
34522 9808 Acked Flush 0 Done Done Sent false
34523 9809 Acked Read 8 Done Done Sent false
34524 9810 Acked Write 5 Done Done Sent false
34525 9811 Acked Read 1 Done Done Sent false
34526 9812 Acked Write 10 Done Done Sent false
34527 9813 Acked Write 5 Done Done Sent false
34528 9814 Acked Flush 0 Done Done Sent false
34529 9815 Acked Write 3 Done Done Sent false
34530 9816 Acked Write 9 Done Done Sent false
34531 9817 Acked Flush 0 Done Done Sent false
34532 9818 Acked Read 9 Done Done Sent false
34533 9819 Acked Read 3 Done Done Sent false
34534 9820 Acked Read 6 Done Done Sent false
34535 9821 Acked Write 3 Done Done Sent false
34536 9822 Acked Read 3 Done Done Sent false
34537 9823 Acked Read 3 Done Done Sent false
34538 9824 Acked Write 7 Done Done Sent false
34539 9825 Acked Write 1 Done Done Sent false
34540 9826 Acked Read 7 Done Done Sent false
34541 9827 Acked Read 10 Done Done Sent false
34542 9828 Acked Write 6 Done Done Sent false
34543 9829 Acked Read 9 Done Done Sent false
34544 9830 Acked Write 5 Done Done Sent false
34545 9831 Acked Read 9 Done Done Sent false
34546 9832 Acked Write 6 Done Done Sent false
34547 9833 Acked Write 2 Done Done Sent false
34548 9834 Acked Read 3 Done Done Sent false
34549 9835 Acked Read 3 Done Done Sent false
34550 9836 Acked Read 4 Done Done Sent false
34551 9837 Acked Read 10 Done Done Sent false
34552 9838 Acked Read 10 Done Done Sent false
34553 9839 Acked Read 2 Done Done Sent false
34554 9840 Acked Write 2 Done Done Sent false
34555 9841 Acked Read 3 Done Done Sent false
34556 9842 Acked Read 1 Done Done Sent false
34557 9843 Acked Write 7 Done Done Sent false
34558 9844 Acked Read 10 Done Done Sent false
34559 9845 Acked Write 1 Done Done Sent false
34560 9846 Acked Read 10 Done Done Sent false
34561 9847 Acked Read 8 Done Done Sent false
34562 9848 Acked Write 10 Done Done Sent false
34563 9849 Acked Read 7 Done Done Sent false
34564 9850 Acked Write 6 Done Done Sent false
34565 9851 Acked Write 10 Done Done Sent false
34566 9852 Acked Read 8 Done Done Sent false
34567 9853 Acked Write 3 Done Done Sent false
34568 9854 Acked Write 3 Done Done Sent false
34569 9855 Acked Write 6 Done Done Sent false
34570 9856 Acked Read 2 Done Done Sent false
34571 9857 Acked Write 9 Done Done Sent false
34572 9858 Acked Flush 0 Done Done Sent false
34573 9859 Acked Read 4 Done Done Sent false
34574 9860 Acked Write 6 Done Done Sent false
34575 9861 Acked Write 8 Done Done Sent false
34576 9862 Acked Read 3 Done Done Sent false
34577 9863 Acked Write 3 Done Done Sent false
34578 9864 Acked Write 7 Done Done Sent false
34579 9865 Acked Read 9 Done Done Sent false
34580 9866 Acked Write 2 Done Done Sent false
34581 9867 Acked Write 9 Done Done Sent false
34582 9868 Acked Write 10 Done Done Sent false
34583 9869 Acked Read 7 Done Done Sent false
34584 9870 Acked Read 4 Done Done Sent false
34585 9871 Acked Write 10 Done Done Sent false
34586 9872 Acked Read 9 Done Done Sent false
34587 9873 Acked Read 6 Done Done Sent false
34588 9874 Acked Write 2 Done Done Sent false
34589 9875 Acked Read 6 Done Done Sent false
34590 9876 Acked Write 10 Done Done Sent false
34591 9877 Acked Read 4 Done Done Sent false
34592 9878 Acked Read 9 Done Done Sent false
34593 9879 Acked Write 5 Done Done Sent false
34594 9880 Acked Read 9 Done Done Sent false
34595 9881 Acked Write 3 Done Done Sent false
34596 9882 Acked Write 6 Done Done Sent false
34597 9883 Acked Flush 0 Done Done Sent false
34598 9884 Acked Write 6 Done Done Sent false
34599 9885 Acked Write 1 Done Done Sent false
34600 9886 Acked Read 5 Done Done Sent false
34601 9887 Acked Write 6 Done Done Sent false
34602 9888 Acked Read 10 Done Done Sent false
34603 9889 Acked Read 9 Done Done Sent false
34604 9890 Acked Read 8 Done Done Sent false
34605 9891 Acked Write 5 Done Done Sent false
34606 9892 Acked Read 2 Done Done Sent false
34607 9893 Acked Read 10 Done Done Sent false
34608 9894 Acked Flush 0 Done Done Sent false
34609 9895 Acked Read 3 Done Done Sent false
34610 9896 Acked Write 10 Done Done Sent false
34611 9897 Acked Read 7 Done Done Sent false
34612 9898 Acked Write 7 Done Done Sent false
34613 9899 Acked Write 2 Done Done Sent false
34614 9900 Acked Flush 0 Done Done Sent false
34615 9901 Acked Write 7 Done Done Sent false
34616 9902 Acked Read 7 Done Done Sent false
34617 9903 Acked Read 6 Done Done Sent false
34618 9904 Acked Flush 0 Done Done Sent false
34619 9905 Acked Write 3 Done Done Sent false
34620 9906 Acked Write 8 Done Done Sent false
34621 9907 Acked Write 8 Done Done Sent false
34622 9908 Acked Write 3 Done Done Sent false
34623 9909 Acked Read 4 Done Done Sent false
34624 9910 Acked Read 5 Done Done Sent false
34625 9911 Acked Write 6 Done Done Sent false
34626 9912 Acked Write 10 Done Done Sent false
34627 9913 Acked Read 5 Done Done Sent false
34628 9914 Acked Write 1 Done Done Sent false
34629 9915 Acked Write 10 Done Done Sent false
34630 9916 Acked Read 1 Done Done Sent false
34631 9917 Acked Write 7 Done Done Sent false
34632 9918 Acked Write 8 Done Done Sent false
34633 9919 Acked Write 10 Done Done Sent false
34634 9920 Acked Write 8 Done Done Sent false
34635 9921 Acked Write 4 Done Done Sent false
34636 9922 Acked Flush 0 Done Done Sent false
34637 9923 Acked Read 1 Done Done Sent false
34638 9924 Acked Write 6 Done Done Sent false
34639 9925 Acked Flush 0 Done Done Sent false
34640 9926 Acked Read 3 Done Done Sent false
34641 9927 Acked Write 2 Done Done Sent false
34642 9928 Acked Read 6 Done Done Sent false
34643 9929 Acked Write 10 Done Done Sent false
34644 9930 Acked Write 8 Done Done Sent false
34645 9931 Acked Read 8 Done Done Sent false
34646 9932 Acked Write 2 Done Done Sent false
34647 9933 Acked Read 5 Done Done Sent false
34648 9934 Acked Write 10 Done Done Sent false
34649 9935 Acked Read 1 Done Done Sent false
34650 9936 Acked Write 3 Done Done Sent false
34651 9937 Acked Read 6 Done Done Sent false
34652 9938 Acked Read 10 Done Done Sent false
34653 9939 Acked Write 6 Done Done Sent false
34654 9940 Acked Read 1 Done Done Sent false
34655 9941 Acked Write 8 Done Done Sent false
34656 9942 Acked Flush 0 Done Done Sent false
34657 9943 Acked Write 8 Done Done Sent false
34658 9944 Acked Read 10 Done Done Sent false
34659 9945 Acked Write 5 Done Done Sent false
34660 9946 Acked Write 4 Done Done Sent false
34661 9947 Acked Write 8 Done Done Sent false
34662 9948 Acked Read 3 Done Done Sent false
34663 9949 Acked Read 8 Done Done Sent false
34664 9950 Acked Read 8 Done Done Sent false
34665 9951 Acked Write 6 Done Done Sent false
34666 9952 Acked Read 10 Done Done Sent false
34667 9953 Acked Read 7 Done Done Sent false
34668 9954 Acked Flush 0 Done Done Sent false
34669 9955 Acked Read 8 Done Done Sent false
34670 9956 Acked Read 4 Done Done Sent false
34671 9957 Acked Read 8 Done Done Sent false
34672 9958 Acked Write 9 Done Done Sent false
34673 9959 Acked Read 10 Done Done Sent false
34674 9960 Acked Write 2 Done Done Sent false
34675 9961 Acked Read 3 Done Done Sent false
34676 9962 Acked Read 7 Done Done Sent false
34677 9963 Acked Write 5 Done Done Sent false
34678 9964 Acked Write 9 Done Done Sent false
34679 9965 Acked Write 7 Done Done Sent false
34680 9966 Acked Read 9 Done Done Sent false
34681 9967 Acked Flush 0 Done Done Sent false
34682 9968 Acked Read 1 Done Done Sent false
34683 9969 Acked Read 3 Done Done Sent false
34684 9970 Acked Write 7 Done Done Sent false
34685 9971 Acked Read 4 Done Done Sent false
34686 9972 Acked Read 1 Done Done Sent false
34687 9973 Acked Write 9 Done Done Sent false
34688 9974 Acked Flush 0 Done Done Sent false
34689 9975 Acked Read 7 Done Done Sent false
34690 9976 Acked Write 1 Done Done Sent false
34691 9977 Acked Flush 0 Done Done Sent false
34692 9978 Acked Write 6 Done Done Sent false
34693 9979 Acked Write 4 Done Done Sent false
34694 9980 Acked Read 2 Done Done Sent false
34695 9981 Acked Write 10 Done Done Sent false
34696 9982 Acked Read 1 Done Done Sent false
34697 9983 Acked Read 8 Done Done Sent false
34698 9984 Acked Write 3 Done Done Sent false
34699 9985 Acked Flush 0 Done Done Sent false
34700 9986 Acked Flush 0 Done Done Sent false
34701 9987 Acked Read 4 Done Done Sent false
34702 9988 Acked Write 9 Done Done Sent false
34703 9989 Acked Write 2 Done Done Sent false
34704 9990 Acked Write 9 Done Done Sent false
34705 9991 Acked Read 9 Done Done Sent false
34706 9992 Acked Write 8 Done Done Sent false
34707 9993 Acked Read 10 Done Done Sent false
34708 9994 Acked Write 8 Done Done Sent false
34709 9995 Acked Write 6 Done Done Sent false
34710 9996 Acked Read 5 Done Done Sent false
34711 9997 Acked Write 6 Done Done Sent false
34712 9998 Acked Read 7 Done Done Sent false
34713 9999 Acked Read 9 Done Done Sent false
34714 10000 Acked Write 3 Done Done Sent false
34715 10001 Acked Write 2 Done Done Sent false
34716 10002 Acked Read 6 Done Done Sent false
34717 10003 Acked Write 1 Done Done Sent false
34718 10004 Acked Flush 0 Done Done Sent false
34719 10005 Acked Read 4 Done Done Sent false
34720 10006 Acked Write 10 Done Done Sent false
34721 10007 Acked Write 7 Done Done Sent false
34722 10008 Acked Read 4 Done Done Sent false
34723 10009 Acked Flush 0 Done Done Sent false
34724 10010 Acked Write 7 Done Done Sent false
34725 10011 Acked Read 6 Done Done Sent false
34726 10012 Acked Write 10 Done Done Sent false
34727 10013 Acked Write 2 Done Done Sent false
34728 10014 Acked Read 3 Done Done Sent false
34729 10015 Acked Read 8 Done Done Sent false
34730 10016 Acked Flush 0 Done Done Sent false
34731 10017 Acked Read 9 Done Done Sent false
34732 10018 Acked Write 1 Done Done Sent false
34733 10019 Acked Read 4 Done Done Sent false
34734 10020 Acked Read 7 Done Done Sent false
34735 10021 Acked Write 7 Done Done Sent false
34736 10022 Acked Write 4 Done Done Sent false
34737 10023 Acked Read 2 Done Done Sent false
34738 10024 Acked Read 1 Done Done Sent false
34739 10025 Acked Read 7 Done Done Sent false
34740 10026 Acked Read 9 Done Done Sent false
34741 10027 Acked Read 10 Done Done Sent false
34742 10028 Acked Read 5 Done Done Sent false
34743 10029 Acked Read 6 Done Done Sent false
34744 10030 Acked Read 5 Done Done Sent false
34745 10031 Acked Write 2 Done Done Sent false
34746 10032 Acked Read 7 Done Done Sent false
34747 10033 Acked Read 4 Done Done Sent false
34748 10034 Acked Read 1 Done Done Sent false
34749 10035 Acked Read 10 Done Done Sent false
34750 10036 Acked Write 4 Done Done Sent false
34751 10037 Acked Read 4 Done Done Sent false
34752 10038 Acked Read 8 Done Done Sent false
34753 10039 Acked Write 7 Done Done Sent false
34754 10040 Acked Flush 0 Done Done Sent false
34755 10041 Acked Read 7 Done Done Sent false
34756 10042 Acked Read 10 Done Done Sent false
34757 10043 Acked Read 3 Done Done Sent false
34758 10044 Acked Write 10 Done Done Sent false
34759 10045 Acked Flush 0 Done Done Sent false
34760 10046 Acked Read 3 Done Done Sent false
34761 10047 Acked Read 6 Done Done Sent false
34762 10048 Acked Write 2 Done Done Sent false
34763 10049 Acked Read 10 Done Done Sent false
34764 10050 Acked Write 9 Done Done Sent false
34765 10051 Acked Write 9 Done Done Sent false
34766 10052 Acked Read 9 Done Done Sent false
34767 10053 Acked Write 9 Done Done Sent false
34768 10054 Acked Read 8 Done Done Sent false
34769 10055 Acked Flush 0 Done Done Sent false
34770 10056 Acked Read 5 Done Done Sent false
34771 10057 Acked Read 5 Done Done Sent false
34772 10058 Acked Read 4 Done Done Sent false
34773 10059 Acked Write 10 Done Done Sent false
34774 10060 Acked Write 2 Done Done Sent false
34775 10061 Acked Read 4 Done Done Sent false
34776 10062 Acked Read 1 Done Done Sent false
34777 10063 Acked Flush 0 Done Done Sent false
34778 10064 Acked Read 1 Done Done Sent false
34779 10065 Acked Flush 0 Done Done Sent false
34780 STATES DS:0 DS:1 DS:2 TOTAL
34781 Sent 0 0 302 302
34782 Done 302 302 0 604
34783 Skipped 0 0 0 0
34784 Error 0 0 0 0
34785 Last Flush: 10065 10065 9763
34786 Downstairs last five completed: 9763 9762 9761 9760 9759
34787 Upstairs last five completed: 10065 10064 10063 10062 10061
34788 CLIENT: Up:0 ds:302 act:2
34789 ----------------------------------------------------------------
34790 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
34791 States: Active Active Offline
34792 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34793 9764 Acked Flush 0 Done Done Sent false
34794 9765 Acked Write 8 Done Done Sent false
34795 9766 Acked Write 3 Done Done Sent false
34796 9767 Acked Flush 0 Done Done Sent false
34797 9768 Acked Write 6 Done Done Sent false
34798 9769 Acked Read 1 Done Done Sent false
34799 9770 Acked Read 5 Done Done Sent false
34800 9771 Acked Write 2 Done Done Sent false
34801 9772 Acked Write 2 Done Done Sent false
34802 9773 Acked Write 9 Done Done Sent false
34803 9774 Acked Read 7 Done Done Sent false
34804 9775 Acked Write 5 Done Done Sent false
34805 9776 Acked Write 3 Done Done Sent false
34806 9777 Acked Write 6 Done Done Sent false
34807 9778 Acked Read 9 Done Done Sent false
34808 9779 Acked Write 5 Done Done Sent false
34809 9780 Acked Flush 0 Done Done Sent false
34810 9781 Acked Read 10 Done Done Sent false
34811 9782 Acked Read 3 Done Done Sent false
34812 9783 Acked Read 5 Done Done Sent false
34813 9784 Acked Write 2 Done Done Sent false
34814 9785 Acked Write 2 Done Done Sent false
34815 9786 Acked Write 1 Done Done Sent false
34816 9787 Acked Read 6 Done Done Sent false
34817 9788 Acked Flush 0 Done Done Sent false
34818 9789 Acked Write 9 Done Done Sent false
34819 9790 Acked Write 10 Done Done Sent false
34820 9791 Acked Write 7 Done Done Sent false
34821 9792 Acked Read 3 Done Done Sent false
34822 9793 Acked Flush 0 Done Done Sent false
34823 9794 Acked Read 9 Done Done Sent false
34824 9795 Acked Write 1 Done Done Sent false
34825 9796 Acked Read 9 Done Done Sent false
34826 9797 Acked Read 7 Done Done Sent false
34827 9798 Acked Write 1 Done Done Sent false
34828 9799 Acked Read 7 Done Done Sent false
34829 9800 Acked Write 7 Done Done Sent false
34830 9801 Acked Write 10 Done Done Sent false
34831 9802 Acked Write 3 Done Done Sent false
34832 9803 Acked Read 7 Done Done Sent false
34833 9804 Acked Write 9 Done Done Sent false
34834 9805 Acked Read 10 Done Done Sent false
34835 9806 Acked Read 6 Done Done Sent false
34836 9807 Acked Read 9 Done Done Sent false
34837 9808 Acked Flush 0 Done Done Sent false
34838 9809 Acked Read 8 Done Done Sent false
34839 9810 Acked Write 5 Done Done Sent false
34840 9811 Acked Read 1 Done Done Sent false
34841 9812 Acked Write 10 Done Done Sent false
34842 9813 Acked Write 5 Done Done Sent false
34843 9814 Acked Flush 0 Done Done Sent false
34844 9815 Acked Write 3 Done Done Sent false
34845 9816 Acked Write 9 Done Done Sent false
34846 9817 Acked Flush 0 Done Done Sent false
34847 9818 Acked Read 9 Done Done Sent false
34848 9819 Acked Read 3 Done Done Sent false
34849 9820 Acked Read 6 Done Done Sent false
34850 9821 Acked Write 3 Done Done Sent false
34851 9822 Acked Read 3 Done Done Sent false
34852 9823 Acked Read 3 Done Done Sent false
34853 9824 Acked Write 7 Done Done Sent false
34854 9825 Acked Write 1 Done Done Sent false
34855 9826 Acked Read 7 Done Done Sent false
34856 9827 Acked Read 10 Done Done Sent false
34857 9828 Acked Write 6 Done Done Sent false
34858 9829 Acked Read 9 Done Done Sent false
34859 9830 Acked Write 5 Done Done Sent false
34860 9831 Acked Read 9 Done Done Sent false
34861 9832 Acked Write 6 Done Done Sent false
34862 9833 Acked Write 2 Done Done Sent false
34863 9834 Acked Read 3 Done Done Sent false
34864 9835 Acked Read 3 Done Done Sent false
34865 9836 Acked Read 4 Done Done Sent false
34866 9837 Acked Read 10 Done Done Sent false
34867 9838 Acked Read 10 Done Done Sent false
34868 9839 Acked Read 2 Done Done Sent false
34869 9840 Acked Write 2 Done Done Sent false
34870 9841 Acked Read 3 Done Done Sent false
34871 9842 Acked Read 1 Done Done Sent false
34872 9843 Acked Write 7 Done Done Sent false
34873 9844 Acked Read 10 Done Done Sent false
34874 9845 Acked Write 1 Done Done Sent false
34875 9846 Acked Read 10 Done Done Sent false
34876 9847 Acked Read 8 Done Done Sent false
34877 9848 Acked Write 10 Done Done Sent false
34878 9849 Acked Read 7 Done Done Sent false
34879 9850 Acked Write 6 Done Done Sent false
34880 9851 Acked Write 10 Done Done Sent false
34881 9852 Acked Read 8 Done Done Sent false
34882 9853 Acked Write 3 Done Done Sent false
34883 9854 Acked Write 3 Done Done Sent false
34884 9855 Acked Write 6 Done Done Sent false
34885 9856 Acked Read 2 Done Done Sent false
34886 9857 Acked Write 9 Done Done Sent false
34887 9858 Acked Flush 0 Done Done Sent false
34888 9859 Acked Read 4 Done Done Sent false
34889 9860 Acked Write 6 Done Done Sent false
34890 9861 Acked Write 8 Done Done Sent false
34891 9862 Acked Read 3 Done Done Sent false
34892 9863 Acked Write 3 Done Done Sent false
34893 9864 Acked Write 7 Done Done Sent false
34894 9865 Acked Read 9 Done Done Sent false
34895 9866 Acked Write 2 Done Done Sent false
34896 9867 Acked Write 9 Done Done Sent false
34897 9868 Acked Write 10 Done Done Sent false
34898 9869 Acked Read 7 Done Done Sent false
34899 9870 Acked Read 4 Done Done Sent false
34900 9871 Acked Write 10 Done Done Sent false
34901 9872 Acked Read 9 Done Done Sent false
34902 9873 Acked Read 6 Done Done Sent false
34903 9874 Acked Write 2 Done Done Sent false
34904 9875 Acked Read 6 Done Done Sent false
34905 9876 Acked Write 10 Done Done Sent false
34906 9877 Acked Read 4 Done Done Sent false
34907 9878 Acked Read 9 Done Done Sent false
34908 9879 Acked Write 5 Done Done Sent false
34909 9880 Acked Read 9 Done Done Sent false
34910 9881 Acked Write 3 Done Done Sent false
34911 9882 Acked Write 6 Done Done Sent false
34912 9883 Acked Flush 0 Done Done Sent false
34913 9884 Acked Write 6 Done Done Sent false
34914 9885 Acked Write 1 Done Done Sent false
34915 9886 Acked Read 5 Done Done Sent false
34916 9887 Acked Write 6 Done Done Sent false
34917 9888 Acked Read 10 Done Done Sent false
34918 9889 Acked Read 9 Done Done Sent false
34919 9890 Acked Read 8 Done Done Sent false
34920 9891 Acked Write 5 Done Done Sent false
34921 9892 Acked Read 2 Done Done Sent false
34922 9893 Acked Read 10 Done Done Sent false
34923 9894 Acked Flush 0 Done Done Sent false
34924 9895 Acked Read 3 Done Done Sent false
34925 9896 Acked Write 10 Done Done Sent false
34926 9897 Acked Read 7 Done Done Sent false
34927 9898 Acked Write 7 Done Done Sent false
34928 9899 Acked Write 2 Done Done Sent false
34929 9900 Acked Flush 0 Done Done Sent false
34930 9901 Acked Write 7 Done Done Sent false
34931 9902 Acked Read 7 Done Done Sent false
34932 9903 Acked Read 6 Done Done Sent false
34933 9904 Acked Flush 0 Done Done Sent false
34934 9905 Acked Write 3 Done Done Sent false
34935 9906 Acked Write 8 Done Done Sent false
34936 9907 Acked Write 8 Done Done Sent false
34937 9908 Acked Write 3 Done Done Sent false
34938 9909 Acked Read 4 Done Done Sent false
34939 9910 Acked Read 5 Done Done Sent false
34940 9911 Acked Write 6 Done Done Sent false
34941 9912 Acked Write 10 Done Done Sent false
34942 9913 Acked Read 5 Done Done Sent false
34943 9914 Acked Write 1 Done Done Sent false
34944 9915 Acked Write 10 Done Done Sent false
34945 9916 Acked Read 1 Done Done Sent false
34946 9917 Acked Write 7 Done Done Sent false
34947 9918 Acked Write 8 Done Done Sent false
34948 9919 Acked Write 10 Done Done Sent false
34949 9920 Acked Write 8 Done Done Sent false
34950 9921 Acked Write 4 Done Done Sent false
34951 9922 Acked Flush 0 Done Done Sent false
34952 9923 Acked Read 1 Done Done Sent false
34953 9924 Acked Write 6 Done Done Sent false
34954 9925 Acked Flush 0 Done Done Sent false
34955 9926 Acked Read 3 Done Done Sent false
34956 9927 Acked Write 2 Done Done Sent false
34957 9928 Acked Read 6 Done Done Sent false
34958 9929 Acked Write 10 Done Done Sent false
34959 9930 Acked Write 8 Done Done Sent false
34960 9931 Acked Read 8 Done Done Sent false
34961 9932 Acked Write 2 Done Done Sent false
34962 9933 Acked Read 5 Done Done Sent false
34963 9934 Acked Write 10 Done Done Sent false
34964 9935 Acked Read 1 Done Done Sent false
34965 9936 Acked Write 3 Done Done Sent false
34966 9937 Acked Read 6 Done Done Sent false
34967 9938 Acked Read 10 Done Done Sent false
34968 9939 Acked Write 6 Done Done Sent false
34969 9940 Acked Read 1 Done Done Sent false
34970 9941 Acked Write 8 Done Done Sent false
34971 9942 Acked Flush 0 Done Done Sent false
34972 9943 Acked Write 8 Done Done Sent false
34973 9944 Acked Read 10 Done Done Sent false
34974 9945 Acked Write 5 Done Done Sent false
34975 9946 Acked Write 4 Done Done Sent false
34976 9947 Acked Write 8 Done Done Sent false
34977 9948 Acked Read 3 Done Done Sent false
34978 9949 Acked Read 8 Done Done Sent false
34979 9950 Acked Read 8 Done Done Sent false
34980 9951 Acked Write 6 Done Done Sent false
34981 9952 Acked Read 10 Done Done Sent false
34982 9953 Acked Read 7 Done Done Sent false
34983 9954 Acked Flush 0 Done Done Sent false
34984 9955 Acked Read 8 Done Done Sent false
34985 9956 Acked Read 4 Done Done Sent false
34986 9957 Acked Read 8 Done Done Sent false
34987 9958 Acked Write 9 Done Done Sent false
34988 9959 Acked Read 10 Done Done Sent false
34989 9960 Acked Write 2 Done Done Sent false
34990 9961 Acked Read 3 Done Done Sent false
34991 9962 Acked Read 7 Done Done Sent false
34992 9963 Acked Write 5 Done Done Sent false
34993 9964 Acked Write 9 Done Done Sent false
34994 9965 Acked Write 7 Done Done Sent false
34995 9966 Acked Read 9 Done Done Sent false
34996 9967 Acked Flush 0 Done Done Sent false
34997 9968 Acked Read 1 Done Done Sent false
34998 9969 Acked Read 3 Done Done Sent false
34999 9970 Acked Write 7 Done Done Sent false
35000 9971 Acked Read 4 Done Done Sent false
35001 9972 Acked Read 1 Done Done Sent false
35002 9973 Acked Write 9 Done Done Sent false
35003 9974 Acked Flush 0 Done Done Sent false
35004 9975 Acked Read 7 Done Done Sent false
35005 9976 Acked Write 1 Done Done Sent false
35006 9977 Acked Flush 0 Done Done Sent false
35007 9978 Acked Write 6 Done Done Sent false
35008 9979 Acked Write 4 Done Done Sent false
35009 9980 Acked Read 2 Done Done Sent false
35010 9981 Acked Write 10 Done Done Sent false
35011 9982 Acked Read 1 Done Done Sent false
35012 9983 Acked Read 8 Done Done Sent false
35013 9984 Acked Write 3 Done Done Sent false
35014 9985 Acked Flush 0 Done Done Sent false
35015 9986 Acked Flush 0 Done Done Sent false
35016 9987 Acked Read 4 Done Done Sent false
35017 9988 Acked Write 9 Done Done Sent false
35018 9989 Acked Write 2 Done Done Sent false
35019 9990 Acked Write 9 Done Done Sent false
35020 9991 Acked Read 9 Done Done Sent false
35021 9992 Acked Write 8 Done Done Sent false
35022 9993 Acked Read 10 Done Done Sent false
35023 9994 Acked Write 8 Done Done Sent false
35024 9995 Acked Write 6 Done Done Sent false
35025 9996 Acked Read 5 Done Done Sent false
35026 9997 Acked Write 6 Done Done Sent false
35027 9998 Acked Read 7 Done Done Sent false
35028 9999 Acked Read 9 Done Done Sent false
35029 10000 Acked Write 3 Done Done Sent false
35030 10001 Acked Write 2 Done Done Sent false
35031 10002 Acked Read 6 Done Done Sent false
35032 10003 Acked Write 1 Done Done Sent false
35033 10004 Acked Flush 0 Done Done Sent false
35034 10005 Acked Read 4 Done Done Sent false
35035 10006 Acked Write 10 Done Done Sent false
35036 10007 Acked Write 7 Done Done Sent false
35037 10008 Acked Read 4 Done Done Sent false
35038 10009 Acked Flush 0 Done Done Sent false
35039 10010 Acked Write 7 Done Done Sent false
35040 10011 Acked Read 6 Done Done Sent false
35041 10012 Acked Write 10 Done Done Sent false
35042 10013 Acked Write 2 Done Done Sent false
35043 10014 Acked Read 3 Done Done Sent false
35044 10015 Acked Read 8 Done Done Sent false
35045 10016 Acked Flush 0 Done Done Sent false
35046 10017 Acked Read 9 Done Done Sent false
35047 10018 Acked Write 1 Done Done Sent false
35048 10019 Acked Read 4 Done Done Sent false
35049 10020 Acked Read 7 Done Done Sent false
35050 10021 Acked Write 7 Done Done Sent false
35051 10022 Acked Write 4 Done Done Sent false
35052 10023 Acked Read 2 Done Done Sent false
35053 10024 Acked Read 1 Done Done Sent false
35054 10025 Acked Read 7 Done Done Sent false
35055 10026 Acked Read 9 Done Done Sent false
35056 10027 Acked Read 10 Done Done Sent false
35057 10028 Acked Read 5 Done Done Sent false
35058 10029 Acked Read 6 Done Done Sent false
35059 10030 Acked Read 5 Done Done Sent false
35060 10031 Acked Write 2 Done Done Sent false
35061 10032 Acked Read 7 Done Done Sent false
35062 10033 Acked Read 4 Done Done Sent false
35063 10034 Acked Read 1 Done Done Sent false
35064 10035 Acked Read 10 Done Done Sent false
35065 10036 Acked Write 4 Done Done Sent false
35066 10037 Acked Read 4 Done Done Sent false
35067 10038 Acked Read 8 Done Done Sent false
35068 10039 Acked Write 7 Done Done Sent false
35069 10040 Acked Flush 0 Done Done Sent false
35070 10041 Acked Read 7 Done Done Sent false
35071 10042 Acked Read 10 Done Done Sent false
35072 10043 Acked Read 3 Done Done Sent false
35073 10044 Acked Write 10 Done Done Sent false
35074 10045 Acked Flush 0 Done Done Sent false
35075 10046 Acked Read 3 Done Done Sent false
35076 10047 Acked Read 6 Done Done Sent false
35077 10048 Acked Write 2 Done Done Sent false
35078 10049 Acked Read 10 Done Done Sent false
35079 10050 Acked Write 9 Done Done Sent false
35080 10051 Acked Write 9 Done Done Sent false
35081 10052 Acked Read 9 Done Done Sent false
35082 10053 Acked Write 9 Done Done Sent false
35083 10054 Acked Read 8 Done Done Sent false
35084 10055 Acked Flush 0 Done Done Sent false
35085 10056 Acked Read 5 Done Done Sent false
35086 10057 Acked Read 5 Done Done Sent false
35087 10058 Acked Read 4 Done Done Sent false
35088 10059 Acked Write 10 Done Done Sent false
35089 10060 Acked Write 2 Done Done Sent false
35090 10061 Acked Read 4 Done Done Sent false
35091 10062 Acked Read 1 Done Done Sent false
35092 10063 Acked Flush 0 Done Done Sent false
35093 10064 Acked Read 1 Done Done Sent false
35094 10065 Acked Flush 0 Done Done Sent false
35095 STATES DS:0 DS:1 DS:2 TOTAL
35096 Sent 0 0 302 302
35097 Done 302 302 0 604
35098 Skipped 0 0 0 0
35099 Error 0 0 0 0
35100 Last Flush: 10065 10065 9763
35101 Downstairs last five completed: 9763 9762 9761 9760 9759
35102 Upstairs last five completed: 10065 10064 10063 10062 10061
35103 CLIENT: Up:0 ds:302 act:2
351042024-11-01T16:57:53.015ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
351052024-11-01T16:57:53.015ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61781) = io task client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
351062024-11-01T16:57:53.016ZWARNcrucible: version negotiation from state Offline = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
351072024-11-01T16:57:53.017ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
351082024-11-01T16:57:53.017ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
351092024-11-01T16:57:53.017ZINFOcrucible: send last flush ID to this DS: 9763 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
351102024-11-01T16:57:53.017ZINFOcrucible: Replied this last flush ID: 9763 = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
351112024-11-01T16:57:53.017ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
351122024-11-01T16:57:53.017ZINFOcrucible: [2] Transition from Offline to Active = downstairs client = 2 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
351132024-11-01T16:57:53.017ZINFOcrucible: [2] client re-new 302 jobs since flush 9763 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
351142024-11-01T16:57:53.017ZINFOcrucible: [2] Replayed 302 jobs since flush: 9763 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
35115 ----------------------------------------------------------------
35116 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
35117 Downstairs last five completed: 10065 10064 10063 10062 10061
35118 Upstairs last five completed: 10065 10064 10063 10062 10061
35119 CLIENT: Up:0 ds:0 act:3
35120 Replay: All jobs finished
35121 Main task has work to do, go find it
35122 got dsc Stop(1)
35123 stop 1
35124 Nov 01 16:57:53.324[8820] Got stop action so:false kr:false
35125 INFO request completed, latency_us: 336, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 14bfd6f7-c01c-4e38-adfc-b43fb7c5ef9e, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
351262024-11-01T16:57:53.325ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
351272024-11-01T16:57:53.325ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
351282024-11-01T16:57:53.325ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
351292024-11-01T16:57:53.325ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
35130 Nov 01 16:57:53.325 INFO request completed, latency_us: 158, response_code: 200, uri: /state/cid/1, method: GET, req_id: d44ee6a9-ca63-4a7a-9db0-35338cdf7e4a, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:57:53.325753977Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"2507b7e9-146d-454e-b579-e099d8c24c78"}
35131 .1:9998
35132 [8820] Exited with: ExitStatus(unix_wait_status(9))
35133 [8820][1] reports Exit
35134 Nov 01 16:57:57.328 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/1, method: GET, req_id: d0877ef6-f86a-497c-9503-2ca3a1139332, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
35135 001/300 Flush
35136 002/300 Write block 739 len 8192 data: 11 8
35137 003/300 Read block 3065 len 20480
35138 004/300 Write block 189 len 40960 data: 3 5 5 6 6 5 6 6 7 10
35139 005/300 Write block 3146 len 36864 data: 6 5 6 5 6 7 5 5 4
35140 006/300 Write block 1490 len 12288 data: 6 7 8
35141 007/300 Read block 3622 len 28672
35142 008/300 Read block 1205 len 32768
35143 009/300 Write block 4654 len 40960 data: 7 5 4 4 4 5 6 8 8 8
35144 010/300 Read block 2341 len 4096
35145 011/300 Read block 4782 len 36864
35146 012/300 Write block 4418 len 20480 data: 5 6 7 3 5
35147 013/300 Write block 2864 len 40960 data: 5 5 6 6 8 7 8 9 7 7
35148 014/300 Write block 2510 len 36864 data: 7 7 8 8 10 10 8 7 7
35149 015/300 Write block 618 len 24576 data: 5 6 6 5 8 7
35150 016/300 Write block 111 len 24576 data: 4 3 6 6 8 8
35151 017/300 Write block 2697 len 40960 data: 3 4 4 4 4 4 5 4 5 5
35152 018/300 Write block 1941 len 12288 data: 6 7 8
35153 019/300 Write block 2871 len 36864 data: 10 8 8 7 8 9 8 8 7
35154 020/300 Write block 1759 len 36864 data: 4 6 8 7 7 7 7 7 6
35155 021/300 Read block 4963 len 24576
35156 022/300 Read block 1367 len 20480
35157 023/300 Write block 401 len 16384 data: 5 6 6 5
35158 024/300 Flush
35159 025/300 Write block 818 len 20480 data: 3 3 2 2 3
35160 026/300 Write block 2518 len 40960 data: 8 6 5 4 5 5 4 4 4 4
35161 027/300 Read block 3755 len 16384
35162 028/300 Read block 2082 len 12288
35163 029/300 Write block 2467 len 12288 data: 6 7 5
35164 030/300 Write block 4786 len 32768 data: 4 5 5 6 6 5 6 6
35165 031/300 Flush
35166 032/300 Flush
35167 033/300 Write block 1997 len 12288 data: 4 4 4
35168 034/300 Write block 1612 len 20480 data: 4 5 4 3 4
35169 035/300 Write block 3952 len 12288 data: 7 6 4
35170 036/300 Write block 1829 len 40960 data: 5 4 4 5 5 5 4 4 5 5
35171 037/300 Read block 1401 len 40960
35172 038/300 Read block 3755 len 28672
35173 039/300 Read block 3506 len 28672
35174 040/300 Write block 2829 len 24576 data: 5 6 7 7 7 7
35175 041/300 Read block 3430 len 4096
35176 042/300 Read block 3643 len 4096
35177 043/300 Write block 4812 len 8192 data: 6 6
35178 044/300 Read block 2331 len 28672
35179 045/300 Write block 333 len 36864 data: 4 4 4 4 3 3 3 3 3
35180 046/300 Read block 259 len 8192
35181 047/300 Write block 4575 len 40960 data: 7 5 4 6 7 7 6 7 7 6
35182 048/300 Write block 3081 len 36864 data: 7 7 6 7 8 7 7 6 7
35183 049/300 Read block 2818 len 4096
35184 050/300 Flush
35185 051/300 Write block 361 len 36864 data: 5 5 6 8 10 9 8 6 8
35186 052/300 Write block 1362 len 32768 data: 6 5 6 5 6 9 8 8
35187 053/300 Flush
35188 054/300 Flush
35189 055/300 Read block 2126 len 32768
35190 056/300 Write block 3634 len 24576 data: 5 3 3 3 3 3
35191 057/300 Write block 4469 len 16384 data: 7 5 6 5
35192 058/300 Read block 2051 len 28672
35193 059/300 Write block 3333 len 12288 data: 5 5 4
35194 060/300 Write block 4711 len 8192 data: 4 5
35195 061/300 Write block 2885 len 24576 data: 7 9 9 10 11 12
35196 062/300 Write block 3132 len 4096 data: 6
35197 063/300 Write block 4110 len 24576 data: 12 12 11 11 11 10
35198 064/300 Read block 3542 len 32768
35199 065/300 Read block 756 len 24576
35200 066/300 Write block 2878 len 24576 data: 9 8 6 5 5 5
35201 067/300 Write block 2699 len 36864 data: 5 5 5 5 6 5 6 6 4
35202 068/300 Read block 1831 len 12288
35203 069/300 Read block 988 len 8192
35204 070/300 Read block 1461 len 4096
35205 071/300 Write block 3063 len 20480 data: 7 6 6 6 6
35206 072/300 Write block 4688 len 40960 data: 8 6 6 5 5 5 6 5 3 4
35207 073/300 Read block 202 len 32768
35208 074/300 Write block 1533 len 32768 data: 5 5 5 5 6 6 7 7
35209 075/300 Write block 2797 len 4096 data: 4
35210 076/300 Read block 2817 len 16384
35211 077/300 Write block 547 len 8192 data: 4 6
35212 078/300 Read block 2233 len 20480
35213 079/300 Read block 4945 len 36864
35214 080/300 Read block 1347 len 4096
35215 081/300 Read block 892 len 20480
35216 082/300 Flush
35217 083/300 Read block 4402 len 40960
35218 084/300 Write block 189 len 20480 data: 4 6 6 7 7
35219 085/300 Write block 1459 len 16384 data: 4 4 4 4
35220 086/300 Read block 1895 len 16384
35221 087/300 Read block 1040 len 28672
35222 088/300 Read block 4908 len 12288
35223 089/300 Read block 1943 len 4096
35224 090/300 Write block 1723 len 32768 data: 7 5 6 5 6 4 5 5
35225 091/300 Write block 1752 len 12288 data: 4 3 3
35226 092/300 Read block 2735 len 16384
35227 093/300 Write block 2900 len 40960 data: 3 3 3 4 4 3 3 3 3 2
35228 094/300 Write block 2061 len 16384 data: 7 9 8 6
35229 095/300 Read block 2463 len 4096
35230 096/300 Write block 428 len 4096 data: 6
35231 097/300 Write block 3267 len 4096 data: 6
35232 098/300 Write block 530 len 16384 data: 11 11 10 9
35233 099/300 Write block 727 len 4096 data: 5
35234 100/300 Read block 3231 len 4096
35235 101/300 Read block 1673 len 24576
35236 102/300 Read block 3378 len 16384
35237 103/300 Read block 886 len 32768
35238 104/300 Read block 4170 len 12288
35239 105/300 Read block 4423 len 4096
35240 106/300 Write block 1526 len 4096 data: 9
35241 107/300 Write block 4526 len 20480 data: 10 11 9 7 8
35242 108/300 Read block 4874 len 20480
35243 109/300 Write block 364 len 36864 data: 9 11 10 9 7 9 9 8 8
35244 110/300 Read block 2662 len 28672
35245 111/300 Read block 614 len 20480
35246 112/300 Write block 631 len 32768 data: 4 5 3 4 4 3 5 4
35247 113/300 Flush
35248 114/300 Read block 4750 len 8192
35249 115/300 Flush
35250 116/300 Read block 910 len 12288
35251 117/300 Write block 1169 len 16384 data: 4 4 6 7
35252 118/300 Write block 2039 len 12288 data: 6 7 6
35253 119/300 Write block 235 len 4096 data: 5
35254 120/300 Write block 3606 len 8192 data: 3 3
35255 121/300 Read block 4570 len 8192
35256 122/300 Read block 641 len 4096
35257 123/300 Write block 4582 len 40960 data: 8 8 7 6 4 4 4 3 3 4
35258 124/300 Write block 647 len 32768 data: 7 3 4 3 4 4 5 5
35259 125/300 Read block 1705 len 32768
35260 126/300 Read block 1607 len 24576
35261 127/300 Write block 3221 len 40960 data: 2 3 4 4 4 5 5 6 7 6
35262 128/300 Read block 2221 len 24576
35263 129/300 Read block 1005 len 28672
35264 130/300 Read block 409 len 20480
35265 131/300 Read block 4362 len 28672
35266 132/300 Read block 3096 len 36864
35267 133/300 Read block 1537 len 8192
35268 134/300 Read block 1814 len 8192
35269 135/300 Write block 2457 len 12288 data: 6 5 4
35270 136/300 Read block 1677 len 24576
35271 137/300 Read block 3534 len 4096
35272 138/300 Write block 1835 len 24576 data: 5 5 6 6 6 6
35273 139/300 Write block 4443 len 8192 data: 5 5
35274 140/300 Read block 1161 len 12288
35275 141/300 Read block 1088 len 28672
35276 142/300 Write block 3483 len 16384 data: 9 7 8 8
35277 143/300 Write block 4061 len 32768 data: 3 4 4 4 3 3 4 3
35278 144/300 Read block 4276 len 36864
35279 145/300 Read block 1406 len 32768
35280 146/300 Write block 3414 len 4096 data: 7
35281 147/300 Read block 4669 len 36864
35282 148/300 Flush
35283 149/300 Flush
35284 150/300 Read block 4918 len 12288
35285 151/300 Read block 4721 len 20480
35286 152/300 Read block 4974 len 32768
35287 153/300 Write block 707 len 32768 data: 4 5 4 4 4 4 4 4
35288 154/300 Read block 338 len 32768
35289 155/300 Write block 3192 len 28672 data: 5 4 3 5 5 6 5
35290 156/300 Read block 4928 len 28672
35291 157/300 Write block 2224 len 8192 data: 10 9
35292 158/300 Write block 2427 len 28672 data: 4 6 5 4 3 4 6
35293 159/300 Read block 1527 len 28672
35294 160/300 Write block 818 len 40960 data: 4 4 3 3 4 4 6 7 8 6
35295 161/300 Read block 1005 len 4096
35296 162/300 Write block 3408 len 16384 data: 5 5 5 4
35297 163/300 Read block 1195 len 40960
35298 164/300 Read block 1808 len 40960
35299 165/300 Write block 1825 len 4096 data: 8
35300 166/300 Read block 3722 len 8192
35301 167/300 Read block 2657 len 20480
35302 168/300 Read block 4220 len 8192
35303 169/300 Read block 113 len 12288
35304 170/300 Write block 47 len 24576 data: 4 4 5 7 6 6
35305 171/300 Read block 3277 len 16384
35306 172/300 Write block 2259 len 40960 data: 8 6 5 6 6 6 6 7 8 9
35307 173/300 Flush
35308 174/300 Write block 2835 len 32768 data: 8 7 7 4 4 4 4 5
35309 175/300 Write block 1986 len 28672 data: 3 3 5 6 5 4 5
35310 176/300 Read block 2163 len 24576
35311 177/300 Read block 3105 len 36864
35312 178/300 Write block 2837 len 8192 data: 8 5
35313 179/300 Flush
35314 180/300 Read block 2423 len 8192
35315 181/300 Write block 3186 len 32768 data: 5 6 6 5 5 6 6 5
35316 182/300 Read block 3127 len 40960
35317 183/300 Write block 3520 len 16384 data: 3 3 3 4
35318 184/300 Write block 2740 len 20480 data: 7 10 9 9 8
35319 185/300 Read block 1165 len 32768
35320 186/300 Write block 1801 len 20480 data: 7 9 9 7 7
35321 187/300 Read block 2864 len 28672
35322 188/300 Read block 1262 len 36864
35323 189/300 Write block 2433 len 36864 data: 7 7 7 7 8 8 4 5 5
35324 190/300 Write block 3365 len 16384 data: 7 7 7 6
35325 191/300 Write block 4424 len 28672 data: 6 5 6 8 9 9 8
35326 192/300 Read block 1998 len 16384
35327 193/300 Write block 4557 len 36864 data: 7 8 7 6 5 4 5 5 4
35328 194/300 Read block 612 len 28672
35329 195/300 Write block 2771 len 32768 data: 6 4 5 6 7 6 8 8
35330 196/300 Write block 3659 len 40960 data: 6 7 6 5 5 4 4 4 4 5
35331 197/300 Write block 1168 len 28672 data: 4 5 5 7 8 8 8
35332 198/300 Read block 1247 len 40960
35333 199/300 Read block 586 len 8192
35334 200/300 Write block 153 len 32768 data: 4 3 2 2 3 5 5 5
35335 201/300 Read block 3066 len 20480
35336 202/300 Read block 2184 len 28672
35337 203/300 Write block 1568 len 4096 data: 5
35338 204/300 Read block 4718 len 20480
35339 205/300 Write block 3180 len 24576 data: 6 5 5 6 8 7
35340 206/300 Read block 2735 len 36864
35341 207/300 Read block 1241 len 16384
35342 208/300 Write block 107 len 20480 data: 6 6 5 5 5
35343 209/300 Write block 1926 len 36864 data: 5 6 5 5 4 4 3 4 4
35344 210/300 Read block 4283 len 16384
35345 211/300 Write block 86 len 24576 data: 6 5 4 7 9 9
35346 212/300 Write block 14 len 4096 data: 2
35347 213/300 Read block 4113 len 40960
35348 214/300 Read block 4499 len 12288
35349 215/300 Write block 4679 len 32768 data: 5 5 5 5 6 8 9 9
35350 216/300 Read block 3755 len 24576
35351 217/300 Write block 1148 len 8192 data: 4 5
35352 218/300 Write block 1129 len 20480 data: 5 5 3 5 5
35353 219/300 Flush
35354 220/300 Write block 957 len 8192 data: 3 2
35355 221/300 Write block 4763 len 20480 data: 8 5 5 5 5
35356 222/300 Write block 1590 len 16384 data: 7 8 5 5
35357 223/300 Read block 364 len 12288
35358 224/300 Read block 3887 len 32768
35359 225/300 Read block 76 len 36864
35360 226/300 Write block 3826 len 20480 data: 10 10 11 10 11
35361 227/300 Write block 221 len 32768 data: 6 6 5 5 5 5 4 3
35362 228/300 Write block 4682 len 32768 data: 6 7 9 10 10 7 9 7
35363 229/300 Write block 4256 len 28672 data: 6 7 6 7 7 6 6
35364 230/300 Read block 561 len 32768
35365 231/300 Write block 1976 len 16384 data: 5 4 3 4
35366 232/300 Read block 3901 len 28672
35367 233/300 Read block 4244 len 24576
35368 234/300 Flush
35369 235/300 Read block 3190 len 8192
35370 236/300 Write block 1900 len 36864 data: 3 3 5 6 7 8 8 8 7
35371 237/300 Write block 3518 len 28672 data: 3 3 4 4 4 5 4
35372 238/300 Write block 4244 len 32768 data: 6 7 7 8 8 8 8 9
35373 239/300 Read block 1557 len 16384
35374 240/300 Write block 965 len 4096 data: 3
35375 241/300 Read block 766 len 28672
35376 242/300 Read block 292 len 28672
35377 243/300 Write block 2562 len 40960 data: 10 10 12 11 8 9 7 5 5 6
35378 244/300 Write block 4774 len 16384 data: 6 7 6 6
35379 245/300 Write block 2059 len 24576 data: 6 7 8 10 9 7
35380 246/300 Write block 3586 len 8192 data: 9 9
35381 247/300 Write block 203 len 36864 data: 7 8 7 7 6 6 7 6 6
35382 248/300 Read block 1268 len 16384
35383 249/300 Write block 2795 len 4096 data: 6
35384 250/300 Write block 1347 len 24576 data: 6 8 6 5 5 5
35385 251/300 Read block 93 len 28672
35386 252/300 Write block 769 len 36864 data: 6 5 5 5 6 5 8 12 12
35387 253/300 Read block 1629 len 28672
35388 254/300 Write block 2928 len 4096 data: 5
35389 255/300 Read block 1293 len 40960
35390 256/300 Write block 3364 len 36864 data: 7 8 8 8 7 6 7 6 7
35391 257/300 Read block 4391 len 24576
35392 258/300 Write block 4260 len 8192 data: 8 7
35393 259/300 Write block 1077 len 12288 data: 7 8 8
35394 260/300 Read block 2474 len 4096
35395 261/300 Read block 1511 len 20480
35396 262/300 Read block 2063 len 36864
35397 263/300 Write block 4607 len 40960 data: 5 5 4 6 5 4 5 6 5 4
35398 264/300 Read block 2147 len 24576
35399 265/300 Read block 3816 len 20480
35400 266/300 Read block 1967 len 40960
35401 267/300 Read block 2519 len 40960
35402 268/300 Write block 42 len 4096 data: 8
35403 269/300 Write block 271 len 4096 data: 7
35404 270/300 Read block 2131 len 24576
35405 271/300 Write block 911 len 8192 data: 8 8
35406 272/300 Read block 1162 len 8192
35407 273/300 Write block 1467 len 28672 data: 6 5 8 8 8 7 7
35408 274/300 Read block 3636 len 16384
35409 275/300 Read block 303 len 12288
35410 276/300 Write block 783 len 36864 data: 5 7 6 4 3 3 4 4 4
35411 277/300 Write block 845 len 40960 data: 8 9 8 8 10 11 9 10 7 5
35412 278/300 Read block 1624 len 32768
35413 279/300 Read block 2570 len 20480
35414 280/300 Read block 1765 len 32768
35415 281/300 Read block 4792 len 4096
35416 282/300 Flush
35417 283/300 Read block 1634 len 4096
35418 284/300 Write block 1730 len 36864 data: 6 5 5 4 4 5 5 6 5
35419 285/300 Read block 1463 len 36864
35420 286/300 Read block 4904 len 36864
35421 287/300 Write block 4218 len 24576 data: 7 6 7 7 7 7
35422 288/300 Read block 2453 len 4096
35423 289/300 Read block 4497 len 12288
35424 290/300 Read block 630 len 36864
35425 291/300 Write block 2870 len 16384 data: 9 11 9 9
35426 292/300 Flush
35427 293/300 Read block 2863 len 24576
35428 294/300 Read block 1121 len 8192
35429 295/300 Read block 3027 len 20480
35430 296/300 Read block 4726 len 36864
35431 297/300 Read block 1252 len 4096
35432 298/300 Read block 3651 len 32768
35433 299/300 Write block 555 len 36864 data: 6 6 4 5 5 4 6 4 5
35434 300/300 Write block 3573 len 20480 data: 4 3 3 3 3
35435 301/300 Read block 33 len 36864
35436 Main task has work to do, go find it
35437 got dsc Start(1)
35438 start 1
35439 Nov 01 16:57:57.616 INFO request completed, latency_us: 232, response_code: 204, uri: /start/cid/1, method: POST, req_id: 2eb95358-b387-45aa-8c36-bbd2d03c92d0, remote_addr: 127.0.0.1:58552, local_addr: 127.0.0.1:9998
35440 Replay: started 1, returned:Ok(())
35441 ----------------------------------------------------------------
35442 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35443 States: Active Offline Active
35444 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35445 10066 Acked Flush 0 Done Sent Done false
35446 10067 Acked Write 2 Done Sent Done false
35447 10068 Acked Read 5 Done Sent Done false
35448 10069 Acked Write 10 Done Sent Done false
35449 10070 Acked Write 9 Done Sent Done false
35450 10071 Acked Write 3 Done Sent Done false
35451 10072 Acked Read 7 Done Sent Done false
35452 10073 Acked Read 8 Done Sent Done false
35453 10074 Acked Write 10 Done Sent Done false
35454 10075 Acked Read 1 Done Sent Done false
35455 10076 Acked Read 9 Done Sent Done false
35456 10077 Acked Write 5 Done Sent Done false
35457 10078 Acked Write 10 Done Sent Done false
35458 10079 Acked Write 9 Done Sent Done false
35459 10080 Acked Write 6 Done Sent Done false
35460 10081 Acked Write 6 Done Sent Done false
35461 10082 Acked Write 10 Done Sent Done false
35462 10083 Acked Write 3 Done Sent Done false
35463 10084 Acked Write 9 Done Sent Done false
35464 10085 Acked Write 9 Done Sent Done false
35465 10086 Acked Read 6 Done Sent Done false
35466 10087 Acked Read 5 Done Sent Done false
35467 10088 Acked Write 4 Done Sent Done false
35468 10089 Acked Flush 0 Done Sent Done false
35469 10090 Acked Write 5 Done Sent Done false
35470 10091 Acked Write 10 Done Sent Done false
35471 10092 Acked Read 4 Done Sent Done false
35472 10093 Acked Read 3 Done Sent Done false
35473 10094 Acked Write 3 Done Sent Done false
35474 10095 Acked Write 8 Done Sent Done false
35475 10096 Acked Flush 0 Done Sent Done false
35476 10097 Acked Flush 0 Done Sent Done false
35477 10098 Acked Write 3 Done Sent Done false
35478 10099 Acked Write 5 Done Sent Done false
35479 10100 Acked Write 3 Done Sent Done false
35480 10101 Acked Write 10 Done Sent Done false
35481 10102 Acked Read 10 Done Sent Done false
35482 10103 Acked Read 7 Done Sent Done false
35483 10104 Acked Read 7 Done Sent Done false
35484 10105 Acked Write 6 Done Sent Done false
35485 10106 Acked Read 1 Done Sent Done false
35486 10107 Acked Read 1 Done Sent Done false
35487 10108 Acked Write 2 Done Sent Done false
35488 10109 Acked Read 7 Done Sent Done false
35489 10110 Acked Write 9 Done Sent Done false
35490 10111 Acked Read 2 Done Sent Done false
35491 10112 Acked Write 10 Done Sent Done false
35492 10113 Acked Write 9 Done Sent Done false
35493 10114 Acked Read 1 Done Sent Done false
35494 10115 Acked Flush 0 Done Sent Done false
35495 10116 Acked Write 9 Done Sent Done false
35496 10117 Acked Write 8 Done Sent Done false
35497 10118 Acked Flush 0 Done Sent Done false
35498 10119 Acked Flush 0 Done Sent Done false
35499 10120 Acked Read 8 Done Sent Done false
35500 10121 Acked Write 6 Done Sent Done false
35501 10122 Acked Write 4 Done Sent Done false
35502 10123 Acked Read 7 Done Sent Done false
35503 10124 Acked Write 3 Done Sent Done false
35504 10125 Acked Write 2 Done Sent Done false
35505 10126 Acked Write 6 Done Sent Done false
35506 10127 Acked Write 1 Done Sent Done false
35507 10128 Acked Write 6 Done Sent Done false
35508 10129 Acked Read 8 Done Sent Done false
35509 10130 Acked Read 6 Done Sent Done false
35510 10131 Acked Write 6 Done Sent Done false
35511 10132 Acked Write 9 Done Sent Done false
35512 10133 Acked Read 3 Done Sent Done false
35513 10134 Acked Read 2 Done Sent Done false
35514 10135 Acked Read 1 Done Sent Done false
35515 10136 Acked Write 5 Done Sent Done false
35516 10137 Acked Write 10 Done Sent Done false
35517 10138 Acked Read 8 Done Sent Done false
35518 10139 Acked Write 8 Done Sent Done false
35519 10140 Acked Write 1 Done Sent Done false
35520 10141 Acked Read 4 Done Sent Done false
35521 10142 Acked Write 2 Done Sent Done false
35522 10143 Acked Read 5 Done Sent Done false
35523 10144 Acked Read 9 Done Sent Done false
35524 10145 Acked Read 1 Done Sent Done false
35525 10146 Acked Read 5 Done Sent Done false
35526 10147 Acked Flush 0 Done Sent Done false
35527 10148 Acked Read 10 Done Sent Done false
35528 10149 Acked Write 5 Done Sent Done false
35529 10150 Acked Write 4 Done Sent Done false
35530 10151 Acked Read 4 Done Sent Done false
35531 10152 Acked Read 7 Done Sent Done false
35532 10153 Acked Read 3 Done Sent Done false
35533 10154 Acked Read 1 Done Sent Done false
35534 10155 Acked Write 8 Done Sent Done false
35535 10156 Acked Write 3 Done Sent Done false
35536 10157 Acked Read 4 Done Sent Done false
35537 10158 Acked Write 10 Done Sent Done false
35538 10159 Acked Write 4 Done Sent Done false
35539 10160 Acked Read 1 Done Sent Done false
35540 10161 Acked Write 1 Done Sent Done false
35541 10162 Acked Write 1 Done Sent Done false
35542 10163 Acked Write 4 Done Sent Done false
35543 10164 Acked Write 1 Done Sent Done false
35544 10165 Acked Read 1 Done Sent Done false
35545 10166 Acked Read 6 Done Sent Done false
35546 10167 Acked Read 4 Done Sent Done false
35547 10168 Acked Read 8 Done Sent Done false
35548 10169 Acked Read 3 Done Sent Done false
35549 10170 Acked Read 1 Done Sent Done false
35550 10171 Acked Write 1 Done Sent Done false
35551 10172 Acked Write 5 Done Sent Done false
35552 10173 Acked Read 5 Done Sent Done false
35553 10174 Acked Write 9 Done Sent Done false
35554 10175 Acked Read 7 Done Sent Done false
35555 10176 Acked Read 5 Done Sent Done false
35556 10177 Acked Write 8 Done Sent Done false
35557 10178 Acked Flush 0 Done Sent Done false
35558 10179 Acked Read 2 Done Sent Done false
35559 10180 Acked Flush 0 Done Sent Done false
35560 10181 Acked Read 3 Done Sent Done false
35561 10182 Acked Write 4 Done Sent Done false
35562 10183 Acked Write 3 Done Sent Done false
35563 10184 Acked Write 1 Done Sent Done false
35564 10185 Acked Write 2 Done Sent Done false
35565 10186 Acked Read 2 Done Sent Done false
35566 10187 Acked Read 1 Done Sent Done false
35567 10188 Acked Write 10 Done Sent Done false
35568 10189 Acked Write 8 Done Sent Done false
35569 10190 Acked Read 8 Done Sent Done false
35570 10191 Acked Read 6 Done Sent Done false
35571 10192 Acked Write 10 Done Sent Done false
35572 10193 Acked Read 6 Done Sent Done false
35573 10194 Acked Read 7 Done Sent Done false
35574 10195 Acked Read 5 Done Sent Done false
35575 10196 Acked Read 7 Done Sent Done false
35576 10197 Acked Read 9 Done Sent Done false
35577 10198 Acked Read 2 Done Sent Done false
35578 10199 Acked Read 2 Done Sent Done false
35579 10200 Acked Write 3 Done Sent Done false
35580 10201 Acked Read 6 Done Sent Done false
35581 10202 Acked Read 1 Done Sent Done false
35582 10203 Acked Write 6 Done Sent Done false
35583 10204 Acked Write 2 Done Sent Done false
35584 10205 Acked Read 3 Done Sent Done false
35585 10206 Acked Read 7 Done Sent Done false
35586 10207 Acked Write 4 Done Sent Done false
35587 10208 Acked Write 8 Done Sent Done false
35588 10209 Acked Read 9 Done Sent Done false
35589 10210 Acked Read 8 Done Sent Done false
35590 10211 Acked Write 1 Done Sent Done false
35591 10212 Acked Read 9 Done Sent Done false
35592 10213 Acked Flush 0 Done Sent Done false
35593 10214 Acked Flush 0 Done Sent Done false
35594 10215 Acked Read 3 Done Sent Done false
35595 10216 Acked Read 5 Done Sent Done false
35596 10217 Acked Read 8 Done Sent Done false
35597 10218 Acked Write 8 Done Sent Done false
35598 10219 Acked Read 8 Done Sent Done false
35599 10220 Acked Write 7 Done Sent Done false
35600 10221 Acked Read 7 Done Sent Done false
35601 10222 Acked Write 2 Done Sent Done false
35602 10223 Acked Write 7 Done Sent Done false
35603 10224 Acked Read 7 Done Sent Done false
35604 10225 Acked Write 10 Done Sent Done false
35605 10226 Acked Read 1 Done Sent Done false
35606 10227 Acked Write 4 Done Sent Done false
35607 10228 Acked Read 10 Done Sent Done false
35608 10229 Acked Read 10 Done Sent Done false
35609 10230 Acked Write 1 Done Sent Done false
35610 10231 Acked Read 2 Done Sent Done false
35611 10232 Acked Read 5 Done Sent Done false
35612 10233 Acked Read 2 Done Sent Done false
35613 10234 Acked Read 3 Done Sent Done false
35614 10235 Acked Write 6 Done Sent Done false
35615 10236 Acked Read 4 Done Sent Done false
35616 10237 Acked Write 10 Done Sent Done false
35617 10238 Acked Flush 0 Done Sent Done false
35618 10239 Acked Write 8 Done Sent Done false
35619 10240 Acked Write 7 Done Sent Done false
35620 10241 Acked Read 6 Done Sent Done false
35621 10242 Acked Read 9 Done Sent Done false
35622 10243 Acked Write 2 Done Sent Done false
35623 10244 Acked Flush 0 Done Sent Done false
35624 10245 Acked Read 2 Done Sent Done false
35625 10246 Acked Write 8 Done Sent Done false
35626 10247 Acked Read 10 Done Sent Done false
35627 10248 Acked Write 4 Done Sent Done false
35628 10249 Acked Write 5 Done Sent Done false
35629 10250 Acked Read 8 Done Sent Done false
35630 10251 Acked Write 5 Done Sent Done false
35631 10252 Acked Read 7 Done Sent Done false
35632 10253 Acked Read 9 Done Sent Done false
35633 10254 Acked Write 9 Done Sent Done false
35634 10255 Acked Write 4 Done Sent Done false
35635 10256 Acked Write 7 Done Sent Done false
35636 10257 Acked Read 4 Done Sent Done false
35637 10258 Acked Write 9 Done Sent Done false
35638 10259 Acked Read 7 Done Sent Done false
35639 10260 Acked Write 8 Done Sent Done false
35640 10261 Acked Write 10 Done Sent Done false
35641 10262 Acked Write 7 Done Sent Done false
35642 10263 Acked Read 10 Done Sent Done false
35643 10264 Acked Read 2 Done Sent Done false
35644 10265 Acked Write 8 Done Sent Done false
35645 10266 Acked Read 5 Done Sent Done false
35646 10267 Acked Read 7 Done Sent Done false
35647 10268 Acked Write 1 Done Sent Done false
35648 10269 Acked Read 5 Done Sent Done false
35649 10270 Acked Write 6 Done Sent Done false
35650 10271 Acked Read 9 Done Sent Done false
35651 10272 Acked Read 4 Done Sent Done false
35652 10273 Acked Write 5 Done Sent Done false
35653 10274 Acked Write 9 Done Sent Done false
35654 10275 Acked Read 4 Done Sent Done false
35655 10276 Acked Write 6 Done Sent Done false
35656 10277 Acked Write 1 Done Sent Done false
35657 10278 Acked Read 10 Done Sent Done false
35658 10279 Acked Read 3 Done Sent Done false
35659 10280 Acked Write 8 Done Sent Done false
35660 10281 Acked Read 6 Done Sent Done false
35661 10282 Acked Write 2 Done Sent Done false
35662 10283 Acked Write 5 Done Sent Done false
35663 10284 Acked Flush 0 Done Sent Done false
35664 10285 Acked Write 2 Done Sent Done false
35665 10286 Acked Write 5 Done Sent Done false
35666 10287 Acked Write 4 Done Sent Done false
35667 10288 Acked Read 3 Done Sent Done false
35668 10289 Acked Read 8 Done Sent Done false
35669 10290 Acked Read 9 Done Sent Done false
35670 10291 Acked Write 5 Done Sent Done false
35671 10292 Acked Write 8 Done Sent Done false
35672 10293 Acked Write 8 Done Sent Done false
35673 10294 Acked Write 7 Done Sent Done false
35674 10295 Acked Read 8 Done Sent Done false
35675 10296 Acked Write 4 Done Sent Done false
35676 10297 Acked Read 7 Done Sent Done false
35677 10298 Acked Read 6 Done Sent Done false
35678 10299 Acked Flush 0 Done Sent Done false
35679 10300 Acked Read 2 Done Sent Done false
35680 10301 Acked Write 9 Done Sent Done false
35681 10302 Acked Write 7 Done Sent Done false
35682 10303 Acked Write 8 Done Sent Done false
35683 10304 Acked Read 4 Done Sent Done false
35684 10305 Acked Write 1 Done Sent Done false
35685 10306 Acked Read 7 Done Sent Done false
35686 10307 Acked Read 7 Done Sent Done false
35687 10308 Acked Write 10 Done Sent Done false
35688 10309 Acked Write 4 Done Sent Done false
35689 10310 Acked Write 6 Done Sent Done false
35690 10311 Acked Write 2 Done Sent Done false
35691 10312 Acked Write 9 Done Sent Done false
35692 10313 Acked Read 4 Done Sent Done false
35693 10314 Acked Write 1 Done Sent Done false
35694 10315 Acked Write 6 Done Sent Done false
35695 10316 Acked Read 7 Done Sent Done false
35696 10317 Acked Write 9 Done Sent Done false
35697 10318 Acked Read 7 Done Sent Done false
35698 10319 Acked Write 1 Done Sent Done false
35699 10320 Acked Read 10 Done Sent Done false
35700 10321 Acked Write 9 Done Sent Done false
35701 10322 Acked Read 6 Done Sent Done false
35702 10323 Acked Write 2 Done Sent Done false
35703 10324 Acked Write 3 Done Sent Done false
35704 10325 Acked Read 1 Done Sent Done false
35705 10326 Acked Read 5 Done Sent Done false
35706 10327 Acked Read 9 Done Sent Done false
35707 10328 Acked Write 10 Done Sent Done false
35708 10329 Acked Read 6 Done Sent Done false
35709 10330 Acked Read 5 Done Sent Done false
35710 10331 Acked Read 10 Done Sent Done false
35711 10332 Acked Read 10 Done Sent Done false
35712 10333 Acked Write 1 Done Sent Done false
35713 10334 Acked Write 1 Done Sent Done false
35714 10335 Acked Read 6 Done Sent Done false
35715 10336 Acked Write 2 Done Sent Done false
35716 10337 Acked Read 2 Done Sent Done false
35717 10338 Acked Write 7 Done Sent Done false
35718 10339 Acked Read 4 Done Sent Done false
35719 10340 Acked Read 3 Done Sent Done false
35720 10341 Acked Write 9 Done Sent Done false
35721 10342 Acked Write 10 Done Sent Done false
35722 10343 Acked Read 8 Done Sent Done false
35723 10344 Acked Read 5 Done Sent Done false
35724 10345 Acked Read 8 Done Sent Done false
35725 10346 Acked Read 1 Done Sent Done false
35726 10347 Acked Flush 0 Done Sent Done false
35727 10348 Acked Read 1 Done Sent Done false
35728 10349 Acked Write 9 Done Sent Done false
35729 10350 Acked Read 9 Done Sent Done false
35730 10351 Acked Read 9 Done Sent Done false
35731 10352 Acked Write 6 Done Sent Done false
35732 10353 Acked Read 1 Done Sent Done false
35733 10354 Acked Read 3 Done Sent Done false
35734 10355 Acked Read 9 Done Sent Done false
35735 10356 Acked Write 4 Done Sent Done false
35736 10357 Acked Flush 0 Done Sent Done false
35737 10358 Acked Read 6 Done Sent Done false
35738 10359 Acked Read 2 Done Sent Done false
35739 10360 Acked Read 5 Done Sent Done false
35740 10361 Acked Read 9 Done Sent Done false
35741 10362 Acked Read 1 Done Sent Done false
35742 10363 Acked Read 8 Done Sent Done false
35743 10364 Acked Write 9 Done Sent Done false
35744 10365 Acked Write 5 Done Sent Done false
35745 10366 Acked Read 9 Done Sent Done false
35746 STATES DS:0 DS:1 DS:2 TOTAL
35747 Sent 0 301 0 301
35748 Done 301 0 301 602
35749 Skipped 0 0 0 0
35750 Error 0 0 0 0
35751 Last Flush: 10357 10065 10357
35752 Downstairs last five completed: 10065 10064 10063 10062 10061
35753 Upstairs last five completed: 10366 10365 10364 10363 10362
35754 CLIENT: Up:0 ds:301 act:2
35755 [8820] got start action
35756 [8820] I am going to restart
35757 Starting downstairs at port 8820
35758 Make output file at "/tmp/dsc/downstairs-8820.txt"
35759 [8820][1] reports Starting
35760 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1229)
35761 [8820][1] reports Running
35762 ----------------------------------------------------------------
35763 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
35764 States: Active Offline Active
35765 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35766 10066 Acked Flush 0 Done Sent Done false
35767 10067 Acked Write 2 Done Sent Done false
35768 10068 Acked Read 5 Done Sent Done false
35769 10069 Acked Write 10 Done Sent Done false
35770 10070 Acked Write 9 Done Sent Done false
35771 10071 Acked Write 3 Done Sent Done false
35772 10072 Acked Read 7 Done Sent Done false
35773 10073 Acked Read 8 Done Sent Done false
35774 10074 Acked Write 10 Done Sent Done false
35775 10075 Acked Read 1 Done Sent Done false
35776 10076 Acked Read 9 Done Sent Done false
35777 10077 Acked Write 5 Done Sent Done false
35778 10078 Acked Write 10 Done Sent Done false
35779 10079 Acked Write 9 Done Sent Done false
35780 10080 Acked Write 6 Done Sent Done false
35781 10081 Acked Write 6 Done Sent Done false
35782 10082 Acked Write 10 Done Sent Done false
35783 10083 Acked Write 3 Done Sent Done false
35784 10084 Acked Write 9 Done Sent Done false
35785 10085 Acked Write 9 Done Sent Done false
35786 10086 Acked Read 6 Done Sent Done false
35787 10087 Acked Read 5 Done Sent Done false
35788 10088 Acked Write 4 Done Sent Done false
35789 10089 Acked Flush 0 Done Sent Done false
35790 10090 Acked Write 5 Done Sent Done false
35791 10091 Acked Write 10 Done Sent Done false
35792 10092 Acked Read 4 Done Sent Done false
35793 10093 Acked Read 3 Done Sent Done false
35794 10094 Acked Write 3 Done Sent Done false
35795 10095 Acked Write 8 Done Sent Done false
35796 10096 Acked Flush 0 Done Sent Done false
35797 10097 Acked Flush 0 Done Sent Done false
35798 10098 Acked Write 3 Done Sent Done false
35799 10099 Acked Write 5 Done Sent Done false
35800 10100 Acked Write 3 Done Sent Done false
35801 10101 Acked Write 10 Done Sent Done false
35802 10102 Acked Read 10 Done Sent Done false
35803 10103 Acked Read 7 Done Sent Done false
35804 10104 Acked Read 7 Done Sent Done false
35805 10105 Acked Write 6 Done Sent Done false
35806 10106 Acked Read 1 Done Sent Done false
35807 10107 Acked Read 1 Done Sent Done false
35808 10108 Acked Write 2 Done Sent Done false
35809 10109 Acked Read 7 Done Sent Done false
35810 10110 Acked Write 9 Done Sent Done false
35811 10111 Acked Read 2 Done Sent Done false
35812 10112 Acked Write 10 Done Sent Done false
35813 10113 Acked Write 9 Done Sent Done false
35814 10114 Acked Read 1 Done Sent Done false
35815 10115 Acked Flush 0 Done Sent Done false
35816 10116 Acked Write 9 Done Sent Done false
35817 10117 Acked Write 8 Done Sent Done false
35818 10118 Acked Flush 0 Done Sent Done false
35819 10119 Acked Flush 0 Done Sent Done false
35820 10120 Acked Read 8 Done Sent Done false
35821 10121 Acked Write 6 Done Sent Done false
35822 10122 Acked Write 4 Done Sent Done false
35823 10123 Acked Read 7 Done Sent Done false
35824 10124 Acked Write 3 Done Sent Done false
35825 10125 Acked Write 2 Done Sent Done false
35826 10126 Acked Write 6 Done Sent Done false
35827 10127 Acked Write 1 Done Sent Done false
35828 10128 Acked Write 6 Done Sent Done false
35829 10129 Acked Read 8 Done Sent Done false
35830 10130 Acked Read 6 Done Sent Done false
35831 10131 Acked Write 6 Done Sent Done false
35832 10132 Acked Write 9 Done Sent Done false
35833 10133 Acked Read 3 Done Sent Done false
35834 10134 Acked Read 2 Done Sent Done false
35835 10135 Acked Read 1 Done Sent Done false
35836 10136 Acked Write 5 Done Sent Done false
35837 10137 Acked Write 10 Done Sent Done false
35838 10138 Acked Read 8 Done Sent Done false
35839 10139 Acked Write 8 Done Sent Done false
35840 10140 Acked Write 1 Done Sent Done false
35841 10141 Acked Read 4 Done Sent Done false
35842 10142 Acked Write 2 Done Sent Done false
35843 10143 Acked Read 5 Done Sent Done false
35844 10144 Acked Read 9 Done Sent Done false
35845 10145 Acked Read 1 Done Sent Done false
35846 10146 Acked Read 5 Done Sent Done false
35847 10147 Acked Flush 0 Done Sent Done false
35848 10148 Acked Read 10 Done Sent Done false
35849 10149 Acked Write 5 Done Sent Done false
35850 10150 Acked Write 4 Done Sent Done false
35851 10151 Acked Read 4 Done Sent Done false
35852 10152 Acked Read 7 Done Sent Done false
35853 10153 Acked Read 3 Done Sent Done false
35854 10154 Acked Read 1 Done Sent Done false
35855 10155 Acked Write 8 Done Sent Done false
35856 10156 Acked Write 3 Done Sent Done false
35857 10157 Acked Read 4 Done Sent Done false
35858 10158 Acked Write 10 Done Sent Done false
35859 10159 Acked Write 4 Done Sent Done false
35860 10160 Acked Read 1 Done Sent Done false
35861 10161 Acked Write 1 Done Sent Done false
35862 10162 Acked Write 1 Done Sent Done false
35863 10163 Acked Write 4 Done Sent Done false
35864 10164 Acked Write 1 Done Sent Done false
35865 10165 Acked Read 1 Done Sent Done false
35866 10166 Acked Read 6 Done Sent Done false
35867 10167 Acked Read 4 Done Sent Done false
35868 10168 Acked Read 8 Done Sent Done false
35869 10169 Acked Read 3 Done Sent Done false
35870 10170 Acked Read 1 Done Sent Done false
35871 10171 Acked Write 1 Done Sent Done false
35872 10172 Acked Write 5 Done Sent Done false
35873 10173 Acked Read 5 Done Sent Done false
35874 10174 Acked Write 9 Done Sent Done false
35875 10175 Acked Read 7 Done Sent Done false
35876 10176 Acked Read 5 Done Sent Done false
35877 10177 Acked Write 8 Done Sent Done false
35878 10178 Acked Flush 0 Done Sent Done false
35879 10179 Acked Read 2 Done Sent Done false
35880 10180 Acked Flush 0 Done Sent Done false
35881 10181 Acked Read 3 Done Sent Done false
35882 10182 Acked Write 4 Done Sent Done false
35883 10183 Acked Write 3 Done Sent Done false
35884 10184 Acked Write 1 Done Sent Done false
35885 10185 Acked Write 2 Done Sent Done false
35886 10186 Acked Read 2 Done Sent Done false
35887 10187 Acked Read 1 Done Sent Done false
35888 10188 Acked Write 10 Done Sent Done false
35889 10189 Acked Write 8 Done Sent Done false
35890 10190 Acked Read 8 Done Sent Done false
35891 10191 Acked Read 6 Done Sent Done false
35892 10192 Acked Write 10 Done Sent Done false
35893 10193 Acked Read 6 Done Sent Done false
35894 10194 Acked Read 7 Done Sent Done false
35895 10195 Acked Read 5 Done Sent Done false
35896 10196 Acked Read 7 Done Sent Done false
35897 10197 Acked Read 9 Done Sent Done false
35898 10198 Acked Read 2 Done Sent Done false
35899 10199 Acked Read 2 Done Sent Done false
35900 10200 Acked Write 3 Done Sent Done false
35901 10201 Acked Read 6 Done Sent Done false
35902 10202 Acked Read 1 Done Sent Done false
35903 10203 Acked Write 6 Done Sent Done false
35904 10204 Acked Write 2 Done Sent Done false
35905 10205 Acked Read 3 Done Sent Done false
35906 10206 Acked Read 7 Done Sent Done false
35907 10207 Acked Write 4 Done Sent Done false
35908 10208 Acked Write 8 Done Sent Done false
35909 10209 Acked Read 9 Done Sent Done false
35910 10210 Acked Read 8 Done Sent Done false
35911 10211 Acked Write 1 Done Sent Done false
35912 10212 Acked Read 9 Done Sent Done false
35913 10213 Acked Flush 0 Done Sent Done false
35914 10214 Acked Flush 0 Done Sent Done false
35915 10215 Acked Read 3 Done Sent Done false
35916 10216 Acked Read 5 Done Sent Done false
35917 10217 Acked Read 8 Done Sent Done false
35918 10218 Acked Write 8 Done Sent Done false
35919 10219 Acked Read 8 Done Sent Done false
35920 10220 Acked Write 7 Done Sent Done false
35921 10221 Acked Read 7 Done Sent Done false
35922 10222 Acked Write 2 Done Sent Done false
35923 10223 Acked Write 7 Done Sent Done false
35924 10224 Acked Read 7 Done Sent Done false
35925 10225 Acked Write 10 Done Sent Done false
35926 10226 Acked Read 1 Done Sent Done false
35927 10227 Acked Write 4 Done Sent Done false
35928 10228 Acked Read 10 Done Sent Done false
35929 10229 Acked Read 10 Done Sent Done false
35930 10230 Acked Write 1 Done Sent Done false
35931 10231 Acked Read 2 Done Sent Done false
35932 10232 Acked Read 5 Done Sent Done false
35933 10233 Acked Read 2 Done Sent Done false
35934 10234 Acked Read 3 Done Sent Done false
35935 10235 Acked Write 6 Done Sent Done false
35936 10236 Acked Read 4 Done Sent Done false
35937 10237 Acked Write 10 Done Sent Done false
35938 10238 Acked Flush 0 Done Sent Done false
35939 10239 Acked Write 8 Done Sent Done false
35940 10240 Acked Write 7 Done Sent Done false
35941 10241 Acked Read 6 Done Sent Done false
35942 10242 Acked Read 9 Done Sent Done false
35943 10243 Acked Write 2 Done Sent Done false
35944 10244 Acked Flush 0 Done Sent Done false
35945 10245 Acked Read 2 Done Sent Done false
35946 10246 Acked Write 8 Done Sent Done false
35947 10247 Acked Read 10 Done Sent Done false
35948 10248 Acked Write 4 Done Sent Done false
35949 10249 Acked Write 5 Done Sent Done false
35950 10250 Acked Read 8 Done Sent Done false
35951 10251 Acked Write 5 Done Sent Done false
35952 10252 Acked Read 7 Done Sent Done false
35953 10253 Acked Read 9 Done Sent Done false
35954 10254 Acked Write 9 Done Sent Done false
35955 10255 Acked Write 4 Done Sent Done false
35956 10256 Acked Write 7 Done Sent Done false
35957 10257 Acked Read 4 Done Sent Done false
35958 10258 Acked Write 9 Done Sent Done false
35959 10259 Acked Read 7 Done Sent Done false
35960 10260 Acked Write 8 Done Sent Done false
35961 10261 Acked Write 10 Done Sent Done false
35962 10262 Acked Write 7 Done Sent Done false
35963 10263 Acked Read 10 Done Sent Done false
35964 10264 Acked Read 2 Done Sent Done false
35965 10265 Acked Write 8 Done Sent Done false
35966 10266 Acked Read 5 Done Sent Done false
35967 10267 Acked Read 7 Done Sent Done false
35968 10268 Acked Write 1 Done Sent Done false
35969 10269 Acked Read 5 Done Sent Done false
35970 10270 Acked Write 6 Done Sent Done false
35971 10271 Acked Read 9 Done Sent Done false
35972 10272 Acked Read 4 Done Sent Done false
35973 10273 Acked Write 5 Done Sent Done false
35974 10274 Acked Write 9 Done Sent Done false
35975 10275 Acked Read 4 Done Sent Done false
35976 10276 Acked Write 6 Done Sent Done false
35977 10277 Acked Write 1 Done Sent Done false
35978 10278 Acked Read 10 Done Sent Done false
35979 10279 Acked Read 3 Done Sent Done false
35980 10280 Acked Write 8 Done Sent Done false
35981 10281 Acked Read 6 Done Sent Done false
35982 10282 Acked Write 2 Done Sent Done false
35983 10283 Acked Write 5 Done Sent Done false
35984 10284 Acked Flush 0 Done Sent Done false
35985 10285 Acked Write 2 Done Sent Done false
35986 10286 Acked Write 5 Done Sent Done false
35987 10287 Acked Write 4 Done Sent Done false
35988 10288 Acked Read 3 Done Sent Done false
35989 10289 Acked Read 8 Done Sent Done false
35990 10290 Acked Read 9 Done Sent Done false
35991 10291 Acked Write 5 Done Sent Done false
35992 10292 Acked Write 8 Done Sent Done false
35993 10293 Acked Write 8 Done Sent Done false
35994 10294 Acked Write 7 Done Sent Done false
35995 10295 Acked Read 8 Done Sent Done false
35996 10296 Acked Write 4 Done Sent Done false
35997 10297 Acked Read 7 Done Sent Done false
35998 10298 Acked Read 6 Done Sent Done false
35999 10299 Acked Flush 0 Done Sent Done false
36000 10300 Acked Read 2 Done Sent Done false
36001 10301 Acked Write 9 Done Sent Done false
36002 10302 Acked Write 7 Done Sent Done false
36003 10303 Acked Write 8 Done Sent Done false
36004 10304 Acked Read 4 Done Sent Done false
36005 10305 Acked Write 1 Done Sent Done false
36006 10306 Acked Read 7 Done Sent Done false
36007 10307 Acked Read 7 Done Sent Done false
36008 10308 Acked Write 10 Done Sent Done false
36009 10309 Acked Write 4 Done Sent Done false
36010 10310 Acked Write 6 Done Sent Done false
36011 10311 Acked Write 2 Done Sent Done false
36012 10312 Acked Write 9 Done Sent Done false
36013 10313 Acked Read 4 Done Sent Done false
36014 10314 Acked Write 1 Done Sent Done false
36015 10315 Acked Write 6 Done Sent Done false
36016 10316 Acked Read 7 Done Sent Done false
36017 10317 Acked Write 9 Done Sent Done false
36018 10318 Acked Read 7 Done Sent Done false
36019 10319 Acked Write 1 Done Sent Done false
36020 10320 Acked Read 10 Done Sent Done false
36021 10321 Acked Write 9 Done Sent Done false
36022 10322 Acked Read 6 Done Sent Done false
36023 10323 Acked Write 2 Done Sent Done false
36024 10324 Acked Write 3 Done Sent Done false
36025 10325 Acked Read 1 Done Sent Done false
36026 10326 Acked Read 5 Done Sent Done false
36027 10327 Acked Read 9 Done Sent Done false
36028 10328 Acked Write 10 Done Sent Done false
36029 10329 Acked Read 6 Done Sent Done false
36030 10330 Acked Read 5 Done Sent Done false
36031 10331 Acked Read 10 Done Sent Done false
36032 10332 Acked Read 10 Done Sent Done false
36033 10333 Acked Write 1 Done Sent Done false
36034 10334 Acked Write 1 Done Sent Done false
36035 10335 Acked Read 6 Done Sent Done false
36036 10336 Acked Write 2 Done Sent Done false
36037 10337 Acked Read 2 Done Sent Done false
36038 10338 Acked Write 7 Done Sent Done false
36039 10339 Acked Read 4 Done Sent Done false
36040 10340 Acked Read 3 Done Sent Done false
36041 10341 Acked Write 9 Done Sent Done false
36042 10342 Acked Write 10 Done Sent Done false
36043 10343 Acked Read 8 Done Sent Done false
36044 10344 Acked Read 5 Done Sent Done false
36045 10345 Acked Read 8 Done Sent Done false
36046 10346 Acked Read 1 Done Sent Done false
36047 10347 Acked Flush 0 Done Sent Done false
36048 10348 Acked Read 1 Done Sent Done false
36049 10349 Acked Write 9 Done Sent Done false
36050 10350 Acked Read 9 Done Sent Done false
36051 10351 Acked Read 9 Done Sent Done false
36052 10352 Acked Write 6 Done Sent Done false
36053 10353 Acked Read 1 Done Sent Done false
36054 10354 Acked Read 3 Done Sent Done false
36055 10355 Acked Read 9 Done Sent Done false
36056 10356 Acked Write 4 Done Sent Done false
36057 10357 Acked Flush 0 Done Sent Done false
36058 10358 Acked Read 6 Done Sent Done false
36059 10359 Acked Read 2 Done Sent Done false
36060 10360 Acked Read 5 Done Sent Done false
36061 10361 Acked Read 9 Done Sent Done false
36062 10362 Acked Read 1 Done Sent Done false
36063 10363 Acked Read 8 Done Sent Done false
36064 10364 Acked Write 9 Done Sent Done false
36065 10365 Acked Write 5 Done Sent Done false
36066 10366 Acked Read 9 Done Sent Done false
36067 10367 Acked Flush 0 Done Sent Done false
36068 STATES DS:0 DS:1 DS:2 TOTAL
36069 Sent 0 302 0 302
36070 Done 302 0 302 604
36071 Skipped 0 0 0 0
36072 Error 0 0 0 0
36073 Last Flush: 10367 10065 10367
36074 Downstairs last five completed: 10065 10064 10063 10062 10061
36075 Upstairs last five completed: 10367 10366 10365 10364 10363
36076 CLIENT: Up:0 ds:302 act:2
360772024-11-01T16:58:03.326ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
360782024-11-01T16:58:03.326ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53392) = io task client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
360792024-11-01T16:58:03.328ZWARNcrucible: version negotiation from state Offline = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
360802024-11-01T16:58:03.328ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
360812024-11-01T16:58:03.328ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
360822024-11-01T16:58:03.328ZINFOcrucible: send last flush ID to this DS: 10065 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
360832024-11-01T16:58:03.328ZINFOcrucible: Replied this last flush ID: 10065 = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
360842024-11-01T16:58:03.328ZINFOcrucible: ds_transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
360852024-11-01T16:58:03.328ZINFOcrucible: [1] Transition from Offline to Active = downstairs client = 1 session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
360862024-11-01T16:58:03.328ZINFOcrucible: [1] client re-new 302 jobs since flush 10065 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
360872024-11-01T16:58:03.329ZINFOcrucible: [1] Replayed 302 jobs since flush: 10065 = downstairs session_id = 2507b7e9-146d-454e-b579-e099d8c24c78
36088 ----------------------------------------------------------------
36089 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36090 Downstairs last five completed: 10367 10366 10365 10364 10363
36091 Upstairs last five completed: 10367 10366 10365 10364 10363
36092 CLIENT: Up:0 ds:0 act:3
36093 Replay: All jobs finished
36094 Test replay has completed
36095 Wrote out file "/tmp/test_replay_verify.log"
36096 CLIENT: Tests done. All submitted work has been ACK'd
36097 ----------------------------------------------------------------
36098 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36099 Downstairs last five completed: 10367 10366 10365 10364 10363
36100 Upstairs last five completed: 10367 10366 10365 10364 10363
36101 CLIENT: Up:0 ds:0 act:3
36102 CLIENT: All jobs finished, all DS active.
361032024-11-01T16:58:05.624ZWARNcrucible: Guest handle has been dropped
36104 Replays:30 time: 7:33
36105 Do final verify
36106 Nov 01 16:58:05.700 INFO accepted connection, remote_addr: 127.0.0.1:46467, local_addr: 127.0.0.1:9998
36107 Nov 01 16:58:05.701 INFO request completed, latency_us: 205, response_code: 200, uri: /regioninfo, method: GET, req_id: 779e4e89-c42b-4489-ad48-9667bd332bdd, remote_addr: 127.0.0.1:46467, local_addr: 127.0.0.1:9998
361082024-11-01T16:58:05.702ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
36109 Nov 01 16:58:05.702 INFO request completed, latency_us: 113, response_code: 200, uri: /regioncount, method: GET, req_id: e9f1979e-40f4-4b62-a5c4-e4d910ce39aa, remote_addr: 127.0.0.1:46467, local_addr: 127.0.0.1:9998
361102024-11-01T16:58:05.702ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
36111 Nov 01 16:58:05.703 INFO request completed, latency_us: 125, response_code: 200, uri: /port/cid/0, method: GET, req_id: a5816748-5093-4258-be36-6cf8d4ba3820, remote_addr: 127.0.0.1:46467, local_addr: 127.0.0.1:9998
36112 Nov 01 16:58:05.703 INFO request completed, latency_us: 91, response_code: 200, uri: /port/cid/1, method: GET, req_id: e4644d26-ba55-4138-9bf4-1c1f712d88a9, remote_addr: 127.0.0.1:46467, local_addr: 127.0.0.1:9998
36113 Nov 01 16:58:05.704 INFO request completed, latency_us: 94, response_code: 200, uri: /port/cid/2, method: GET, req_id: 57d97cf6-df5c-48d9-8f2d-15a27d9b2712, remote_addr: 127.0.0.1:46467, local_addr: 127.0.0.1:9998
361142024-11-01T16:58:05.704ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
361152024-11-01T16:58:05.704ZINFOcrucible: Upstairs starts
361162024-11-01T16:58:05.704ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "3818f0fc76fe787c6cc412af30fd59b930203885", git_commit_timestamp: "2024-11-01T16:08:30.000000000Z", git_branch: "renovate/hyper-1.x-lockfile", rustc_semver: "1.82.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "f6e511eec7342f59a25f7c0534f1dbea00d01b14", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
361172024-11-01T16:58:05.704ZINFOcrucible: Upstairs <-> Downstairs Message Version: 12
361182024-11-01T16:58:05.704ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
361192024-11-01T16:58:05.705ZINFOcrucible: Crucible d161f363-7a3c-4eeb-8d5a-c3ca27a48572 has session id: 1a5e5554-6944-4845-a51f-66afdef3d982 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361202024-11-01T16:58:05.705ZINFOcrucible: Upstairs opts: Upstairs UUID: d161f363-7a3c-4eeb-8d5a-c3ca27a48572, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361212024-11-01T16:58:05.705ZINFOcrucible: Crucible stats registered with UUID: d161f363-7a3c-4eeb-8d5a-c3ca27a48572 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361222024-11-01T16:58:05.738ZINFOcrucible: Downstairs in volume = 3 task = crutest
361232024-11-01T16:58:05.738ZINFOcrucible: The guest has requested activation with gen:3
361242024-11-01T16:58:05.738ZINFOcrucible: d161f363-7a3c-4eeb-8d5a-c3ca27a48572 active request set session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361252024-11-01T16:58:05.738ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361262024-11-01T16:58:05.738ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361272024-11-01T16:58:05.739ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361282024-11-01T16:58:05.739ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361292024-11-01T16:58:05.739ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361302024-11-01T16:58:05.739ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361312024-11-01T16:58:05.739ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361322024-11-01T16:58:05.739ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361332024-11-01T16:58:05.739ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361342024-11-01T16:58:05.739ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:35450) = io task client = 2 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361352024-11-01T16:58:05.739ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39074) = io task client = 0 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361362024-11-01T16:58:05.739ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39380) = io task client = 1 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361372024-11-01T16:58:05.740ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361382024-11-01T16:58:05.740ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361392024-11-01T16:58:05.740ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361402024-11-01T16:58:05.740ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361412024-11-01T16:58:05.740ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361422024-11-01T16:58:05.740ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361432024-11-01T16:58:05.740ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361442024-11-01T16:58:05.740ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361452024-11-01T16:58:05.740ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361462024-11-01T16:58:05.741ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361472024-11-01T16:58:05.741ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361482024-11-01T16:58:05.741ZINFOcrucible: Waiting for more clients to be ready session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361492024-11-01T16:58:05.741ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361502024-11-01T16:58:05.741ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361512024-11-01T16:58:05.741ZINFOcrucible: Waiting for more clients to be ready session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361522024-11-01T16:58:05.741ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361532024-11-01T16:58:05.741ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361542024-11-01T16:58:05.741ZINFOcrucible: [0]R flush_numbers[0..12]: [954, 952, 954, 946, 946, 956, 948, 954, 954, 954, 954, 952] = downstairs session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361552024-11-01T16:58:05.741ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361562024-11-01T16:58:05.741ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361572024-11-01T16:58:05.741ZINFOcrucible: [1]R flush_numbers[0..12]: [954, 952, 954, 946, 946, 956, 948, 954, 954, 954, 954, 952] = downstairs session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361582024-11-01T16:58:05.741ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361592024-11-01T16:58:05.741ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361602024-11-01T16:58:05.741ZINFOcrucible: [2]R flush_numbers[0..12]: [954, 952, 954, 946, 946, 956, 948, 954, 954, 954, 954, 952] = downstairs session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361612024-11-01T16:58:05.741ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361622024-11-01T16:58:05.741ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361632024-11-01T16:58:05.741ZINFOcrucible: Max found gen is 3 = downstairs session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361642024-11-01T16:58:05.741ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
36165 {"msg":"Next flush: 957","v":0,"name":"crucible","level":30,"time":"2024-11-01T16:58:05.741858342Z","hostname":"w-01JBMA2V1ED74CMYQ1QEAX6KWF","pid":1234,"Wait for a query_work_queue command to finish before sending IO
36166 ":"downstairs","":"downstairs","session_id":"1a5e5554-6944-4845-a51f-66afdef3d982"}
361672024-11-01T16:58:05.741ZINFOcrucible: All extents match = downstairs session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361682024-11-01T16:58:05.741ZINFOcrucible: No downstairs reconciliation required session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361692024-11-01T16:58:05.741ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361702024-11-01T16:58:05.741ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361712024-11-01T16:58:05.741ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361722024-11-01T16:58:05.741ZINFOcrucible: All required reconciliation work is completed session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361732024-11-01T16:58:05.741ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361742024-11-01T16:58:05.741ZINFOcrucible: d161f363-7a3c-4eeb-8d5a-c3ca27a48572 is now active with session: 1a5e5554-6944-4845-a51f-66afdef3d982 session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361752024-11-01T16:58:05.741ZINFOcrucible: Set Active after no reconciliation session_id = 1a5e5554-6944-4845-a51f-66afdef3d982
361762024-11-01T16:58:05.742ZINFOcrucible: The guest has finished waiting for activation with:3
36177 Region: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
36178 Loading write count information from file "/tmp/test_replay_verify.log"
36179 Read and Verify all blocks (0..5000 range:false)
36180 Wrote out file "/tmp/test_replay_verify.log"
36181 Verify test completed
36182 Wrote out file "/tmp/test_replay_verify.log"
36183 CLIENT: Tests done. All submitted work has been ACK'd
36184 ----------------------------------------------------------------
36185 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
36186 States: Active Active Active
36187 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36188 1000 Acked Read 256 Done Done Done false
36189 1001 Acked Read 256 Done Done Done false
36190 1002 Acked Read 256 Done Done Done false
36191 1003 Acked Read 256 Done Done Done false
36192 1004 Acked Read 256 Done Done Done false
36193 1005 Acked Read 256 Done Done Done false
36194 1006 Acked Read 256 Done Done Done false
36195 1007 Acked Read 256 Done Done Done false
36196 1008 Acked Read 256 Done Done Done false
36197 1009 Acked Read 256 Done Done Done false
36198 1010 Acked Read 256 Done Done Done false
36199 1011 Acked Read 256 Done Done Done false
36200 1012 Acked Read 256 Done Done Done false
36201 1013 Acked Read 256 Done Done Done false
36202 1014 Acked Read 256 Done Done Done false
36203 1015 Acked Read 256 Done Done Done false
36204 1016 Acked Read 256 Done Done Done false
36205 1017 Acked Read 256 Done Done Done false
36206 1018 Acked Read 136 Done Done Done false
36207 1019 Acked Read 256 Done Done Done false
36208 STATES DS:0 DS:1 DS:2 TOTAL
36209 Sent 0 0 0 0
36210 Done 20 20 20 60
36211 Skipped 0 0 0 0
36212 Error 0 0 0 0
36213 Last Flush: 0 0 0
36214 Downstairs last five completed:
36215 Upstairs last five completed: 1019 1018 1017 1016 1015
36216 CLIENT: Up:0 ds:20 act:3
36217 ----------------------------------------------------------------
36218 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
36219 Downstairs last five completed: 1020 1019 1018 1017 1016
36220 Upstairs last five completed: 1020 1019 1018 1017 1016
36221 CLIENT: Up:0 ds:0 act:3
36222 CLIENT: All crucible jobs finished, exiting program
362232024-11-01T16:58:10.150ZWARNcrucible: Guest handle has been dropped
36224 Nov 01 16:58:10.218 INFO accepted connection, remote_addr: 127.0.0.1:42922, local_addr: 127.0.0.1:9998
36225 Main task has work to do, go find it
36226 got dsc Shutdown
36227 Shutdown
36228 Shut it down
36229 Nov 01 16:58:10.219 INFO request completed, latency_us[8810] Disable keep_running
36230 : [8820] Disable keep_running
36231 159[8810] Got stop action so:false kr:false
36232 , [8830] Disable keep_running
36233 response_code: 204, uri: [8830] Got stop action so:false kr:false
36234 /shutdown, method: POST, req_id: e48eeee9-1443-443d-b0ee-f05f23e15a31, remote_addr: 127.0.0.1:42922, local_addr: 127.0.0.1:9998
36235 [8820] Got stop action so:false kr:false
36236 [8810] Exited with: ExitStatus(unix_wait_status(9))
36237 [8810][0] reports Exit
36238 [8820] Exited with: ExitStatus(unix_wait_status(9))
36239 [8820][1] reports Exit
36240 [8830] Exited with: ExitStatus(unix_wait_status(9))
36241 [8830][2] reports Exit
36242 November 1, 2024 at 04:58:15 PM UTC Test ends with 0