Max level shown:
1 starting January 23, 2025 at 05:37:37 PM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.37333947
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.02160676
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.02039724
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start access at:127.0.0.1:9998
14 start ds: 8820
15 start ds: 8830
16 Starting downstairs at port 8810
17 Starting downstairs at port 8830
18 Make output file at "/tmp/dsc/downstairs-8830.txt"
19 Starting downstairs at port 8820
20 Make output file at "/tmp/dsc/downstairs-8820.txt"
21 Make output file at "/tmp/dsc/downstairs-8810.txt"
22 [8830][2] initial start wait reports Starting
23 [8810][0] initial start wait reports Starting
24 [8820][1] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1195)
26 [8830][2] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1196)
28 [8810][0] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1197)
30 [8820][1] initial start wait reports Running
31 All downstairs are running
32 Jan 23 17:37:37.823 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Jan 23 17:37:42.890 INFO accepted connection, remote_addr: 127.0.0.1:33157, local_addr: 127.0.0.1:9998
36 Jan 23 17:37:42.892 INFO request completed, latency_us: 605, response_code: 200, uri: /regioninfo, method: GET, req_id: ad36156f-c0dc-4b0f-a7b5-482efb05caa5, remote_addr: 127.0.0.1:33157, local_addr: 127.0.0.1:9998
37 Jan 23 17:37:42.894 INFO request completed, latency_us: 113, response_code: 200, uri: /regioncount, method: GET, req_id: b1122adb-bdeb-435a-a0c2-763d110532ef, remote_addr: 127.0.0.1:33157, local_addr: 127.0.0.1:9998
382025-01-23T17:37:42.894ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392025-01-23T17:37:42.894ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Jan 23 17:37:42.895 INFO request completed, latency_us: 157, response_code: 200, uri: /port/cid/0, method: GET, req_id: eeb9e6c1-b316-4ae9-9038-950c43a73fb8, remote_addr: 127.0.0.1:33157, local_addr: 127.0.0.1:9998
41 Jan 23 17:37:42.895 INFO request completed, latency_us: 88, response_code: 200, uri: /port/cid/1, method: GET, req_id: 6098ec8b-e1f9-4296-ad41-cfdcf244d401, remote_addr: 127.0.0.1:33157, local_addr: 127.0.0.1:9998
42 Jan 23 17:37:42.896 INFO request completed, latency_us: 87, response_code: 200, uri: /port/cid/2, method: GET, req_id: 70595bf5-4aeb-4890-886c-a18a19212379, remote_addr: 127.0.0.1:33157, local_addr: 127.0.0.1:9998
432025-01-23T17:37:42.896ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442025-01-23T17:37:42.897ZINFOcrucible: Upstairs starts
452025-01-23T17:37:42.897ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "2d334f767f4e70887c1c804fc50babae5e22bede", git_commit_timestamp: "2025-01-23T17:21:44.000000000Z", git_branch: "renovate/tokio-util-0.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462025-01-23T17:37:42.897ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
472025-01-23T17:37:42.897ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482025-01-23T17:37:42.898ZINFOcrucible: Crucible 06a4496b-5007-4ce8-a59b-bd6d0747c44b has session id: eb8b1230-3ad2-4a50-a76c-75e729bb792d session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
492025-01-23T17:37:42.898ZINFOcrucible: Upstairs opts: Upstairs UUID: 06a4496b-5007-4ce8-a59b-bd6d0747c44b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
502025-01-23T17:37:42.898ZINFOcrucible: Crucible stats registered with UUID: 06a4496b-5007-4ce8-a59b-bd6d0747c44b session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
512025-01-23T17:37:42.898ZWARNcrucible: could not find Downstairs address for Nexus session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
522025-01-23T17:37:42.900ZINFOcrucible: Downstairs in volume = 3 task = crutest
532025-01-23T17:37:42.900ZINFOcrucible: The guest has requested activation with gen:1
542025-01-23T17:37:42.900ZINFOcrucible: 06a4496b-5007-4ce8-a59b-bd6d0747c44b active request set session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
552025-01-23T17:37:42.900ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
562025-01-23T17:37:42.900ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
572025-01-23T17:37:42.900ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
582025-01-23T17:37:42.900ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
592025-01-23T17:37:42.900ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
602025-01-23T17:37:42.900ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
612025-01-23T17:37:42.900ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
622025-01-23T17:37:42.900ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
632025-01-23T17:37:42.900ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
642025-01-23T17:37:42.900ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52802) = io task client = 0 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
652025-01-23T17:37:42.900ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49764) = io task client = 1 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
662025-01-23T17:37:42.900ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33446) = io task client = 2 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
672025-01-23T17:37:42.902ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
682025-01-23T17:37:42.902ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
692025-01-23T17:37:42.902ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
702025-01-23T17:37:42.903ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
712025-01-23T17:37:42.903ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
722025-01-23T17:37:42.903ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
732025-01-23T17:37:42.904ZINFOcrucible: Waiting for more clients to be ready session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
742025-01-23T17:37:42.904ZINFOcrucible: Waiting for more clients to be ready session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
752025-01-23T17:37:42.904ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
762025-01-23T17:37:42.904ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
772025-01-23T17:37:42.904ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
782025-01-23T17:37:42.904ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
792025-01-23T17:37:42.904ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
802025-01-23T17:37:42.904ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
812025-01-23T17:37:42.904ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
822025-01-23T17:37:42.904ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
832025-01-23T17:37:42.904ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
842025-01-23T17:37:42.904ZINFOcrucible: Max found gen is 1 = downstairs session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
852025-01-23T17:37:42.904ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
862025-01-23T17:37:42.904ZINFOcrucible: Next flush: 1 = downstairs session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
872025-01-23T17:37:42.904ZINFOcrucible: All extents match = downstairs session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
882025-01-23T17:37:42.904ZINFOcrucible: No downstairs reconciliation required session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
892025-01-23T17:37:42.904ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
902025-01-23T17:37:42.904ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
912025-01-23T17:37:42.904ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
92 {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"Activate successful
93 2025-01-23T17:37:42.904970608ZWait for a query_work_queue command to finish before sending IO
94 ","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1201,"session_id":"eb8b1230-3ad2-4a50-a76c-75e729bb792d"}
952025-01-23T17:37:42.904ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
962025-01-23T17:37:42.905ZINFOcrucible: 06a4496b-5007-4ce8-a59b-bd6d0747c44b is now active with session: eb8b1230-3ad2-4a50-a76c-75e729bb792d session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
972025-01-23T17:37:42.905ZINFOcrucible: Set Active after no reconciliation session_id = eb8b1230-3ad2-4a50-a76c-75e729bb792d
982025-01-23T17:37:42.905ZINFOcrucible: The guest has finished waiting for activation with:1
99 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
100 Fill test
101 Read and Verify all blocks (0..5000 range:false)
102 Wrote out file "/tmp/test_replay_verify.log"
103 CLIENT: Tests done. All submitted work has been ACK'd
104 ----------------------------------------------------------------
105 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
106 States: Active Active Active
107 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
108 1021 Acked Read 256 Done Done Done false
109 1022 Acked Read 256 Done Done Done false
110 1023 Acked Read 256 Done Done Done false
111 1024 Acked Read 256 Done Done Done false
112 1025 Acked Read 256 Done Done Done false
113 1026 Acked Read 256 Done Done Done false
114 1027 Acked Read 256 Done Done Done false
115 1028 Acked Read 256 Done Done Done false
116 1029 Acked Read 256 Done Done Done false
117 1030 Acked Read 256 Done Done Done false
118 1031 Acked Read 256 Done Done Done false
119 1032 Acked Read 256 Done Done Done false
120 1033 Acked Read 256 Done Done Done false
121 1034 Acked Read 256 Done Done Done false
122 1035 Acked Read 256 Done Done Done false
123 1036 Acked Read 256 Done Done Done false
124 1037 Acked Read 256 Done Done Done false
125 1038 Acked Read 256 Done Done Done false
126 1039 Acked Read 256 Done Done Done false
127 1040 Acked Read 136 Done Done Done false
128 STATES DS:0 DS:1 DS:2 TOTAL
129 Sent 0 0 0 0
130 Done 20 20 20 60
131 Skipped 0 0 0 0
132 Error 0 0 0 0
133 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
134 Downstairs last five completed: 1020 1019 1018 1017 1016
135 Upstairs last five completed: 1040 1039 1038 1037 1036
136 CLIENT: Up:0 ds:20 act:3
137 ----------------------------------------------------------------
138 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
139 Downstairs last five completed: 1041 1040 1039 1038 1037
140 Upstairs last five completed: 1041 1040 1039 1038 1037
141 CLIENT: Up:0 ds:0 act:3
142 CLIENT: All crucible jobs finished, exiting program
1432025-01-23T17:37:47.483ZWARNcrucible: Guest handle has been dropped
144 Replay loop starts now January 23, 2025 at 05:37:47 PM UTC
145 Jan 23 17:37:47.558 INFO accepted connection, remote_addr: 127.0.0.1:38360, local_addr: 127.0.0.1:9998
146 Jan 23 17:37:47.558 INFO request completed, latency_us: 169, response_code: 200, uri: /regioninfo, method: GET, req_id: bcfac01b-d7a4-422a-aca1-d131be20d146, remote_addr: 127.0.0.1:38360, local_addr: 127.0.0.1:9998
1472025-01-23T17:37:47.559ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
148 Jan 23 17:37:47.559 INFO request completed, latency_us: 92, response_code: 200, uri: /regioncount, method: GET, req_id: 485d1bf1-8e08-495d-977f-7128f2a0f58f, remote_addr: 127.0.0.1:38360, local_addr: 127.0.0.1:9998
1492025-01-23T17:37:47.559ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
150 Jan 23 17:37:47.560 INFO request completed, latency_us: 134, response_code: 200, uri: /port/cid/0, method: GET, req_id: 836fb73b-3aec-4e0a-a17e-6c9051599cc7, remote_addr: 127.0.0.1:38360, local_addr: 127.0.0.1:9998
151 Jan 23 17:37:47.561 INFO request completed, latency_us: 107, response_code: 200, uri: /port/cid/1, method: GET, req_id: 829cf222-74a2-49d3-8d68-027265c8c378, remote_addr: 127.0.0.1:38360, local_addr: 127.0.0.1:9998
152 Jan 23 17:37:47.561 INFO request completed, latency_us: 87, response_code: 200, uri: /port/cid/2, method: GET, req_id: c0c24d2a-0c1c-4222-b404-bab2c1c90b5f, remote_addr: 127.0.0.1:38360, local_addr: 127.0.0.1:9998
1532025-01-23T17:37:47.561ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1542025-01-23T17:37:47.561ZINFOcrucible: Upstairs starts
1552025-01-23T17:37:47.561ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "2d334f767f4e70887c1c804fc50babae5e22bede", git_commit_timestamp: "2025-01-23T17:21:44.000000000Z", git_branch: "renovate/tokio-util-0.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1562025-01-23T17:37:47.562ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
1572025-01-23T17:37:47.562ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1582025-01-23T17:37:47.562ZINFOcrucible: Crucible a7202a6c-0ab1-4a5b-bd91-7f562e3b27f3 has session id: 14e5020b-a7a5-48d4-836f-67e5604bd35c session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1592025-01-23T17:37:47.562ZINFOcrucible: Upstairs opts: Upstairs UUID: a7202a6c-0ab1-4a5b-bd91-7f562e3b27f3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1602025-01-23T17:37:47.562ZINFOcrucible: Crucible stats registered with UUID: a7202a6c-0ab1-4a5b-bd91-7f562e3b27f3 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1612025-01-23T17:37:47.562ZWARNcrucible: could not find Downstairs address for Nexus session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1622025-01-23T17:37:47.563ZINFOcrucible: Downstairs in volume = 3 task = crutest
1632025-01-23T17:37:47.563ZINFOcrucible: The guest has requested activation with gen:2
1642025-01-23T17:37:47.563ZINFOcrucible: a7202a6c-0ab1-4a5b-bd91-7f562e3b27f3 active request set session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1652025-01-23T17:37:47.563ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1662025-01-23T17:37:47.563ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1672025-01-23T17:37:47.563ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1682025-01-23T17:37:47.563ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1692025-01-23T17:37:47.563ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1702025-01-23T17:37:47.563ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1712025-01-23T17:37:47.563ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1722025-01-23T17:37:47.563ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1732025-01-23T17:37:47.563ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1742025-01-23T17:37:47.563ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:45039) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1752025-01-23T17:37:47.564ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42448) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1762025-01-23T17:37:47.564ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49253) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1772025-01-23T17:37:47.564ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1782025-01-23T17:37:47.564ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1792025-01-23T17:37:47.564ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1802025-01-23T17:37:47.565ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1812025-01-23T17:37:47.565ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1822025-01-23T17:37:47.565ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1832025-01-23T17:37:47.566ZINFOcrucible: Waiting for more clients to be ready session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1842025-01-23T17:37:47.566ZINFOcrucible: Waiting for more clients to be ready session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1852025-01-23T17:37:47.566ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1862025-01-23T17:37:47.566ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1872025-01-23T17:37:47.566ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1882025-01-23T17:37:47.566ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1892025-01-23T17:37:47.566ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1902025-01-23T17:37:47.566ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1912025-01-23T17:37:47.566ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1922025-01-23T17:37:47.566ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1932025-01-23T17:37:47.566ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
194 {"msg":"Max found gen is 2","v":0,"name":Activate successful
195 Wait for a query_work_queue command to finish before sending IO
196 "crucible","level":30,"time":"2025-01-23T17:37:47.566393551Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"downstairs","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
1972025-01-23T17:37:47.566ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1982025-01-23T17:37:47.566ZINFOcrucible: Next flush: 2 = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1992025-01-23T17:37:47.566ZINFOcrucible: All extents match = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2002025-01-23T17:37:47.566ZINFOcrucible: No downstairs reconciliation required session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2012025-01-23T17:37:47.566ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2022025-01-23T17:37:47.566ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2032025-01-23T17:37:47.566ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2042025-01-23T17:37:47.566ZINFOcrucible: All required reconciliation work is completed session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2052025-01-23T17:37:47.566ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2062025-01-23T17:37:47.566ZINFOcrucible: a7202a6c-0ab1-4a5b-bd91-7f562e3b27f3 is now active with session: 14e5020b-a7a5-48d4-836f-67e5604bd35c session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2072025-01-23T17:37:47.566ZINFOcrucible: Set Active after no reconciliation session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2082025-01-23T17:37:47.566ZINFOcrucible: The guest has finished waiting for activation with:2
209 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
210 Loading write count information from file "/tmp/test_replay_verify.log"
211 Jan 23 17:37:47.599 INFO accepted connection, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
212 Main task has work to do, go find it
213 got dsc Stop(0)
214 stop 0
215 Jan 23 17:37:47.599 INFO request completed, latency_us: 180, response_code: 204, uri: /stop/cid/0, method: POST, req_id: bc104f08-6d51-4e3c-af11-211c171b8859, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0[8810] Got stop action so:false kr:false
216 .1:9998
2172025-01-23T17:37:47.600ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2182025-01-23T17:37:47.600ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2192025-01-23T17:37:47.600ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2202025-01-23T17:37:47.600ZINFOcrucible: client task is exiting = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
221 Jan 23 17:37:47.600 INFO request completed, latency_us: 251, response_code: 200, uri: /state/cid/0, method: GET, req_id: 517c9c6d-27dd-4e08-a214-b1b7d13ca23d, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
222 [8810] Exited with: ExitStatus(unix_wait_status(9))
223 [8810][0] reports Exit
224 Jan 23 17:37:51.603 INFO request completed, latency_us: 240, response_code: 200, uri: /state/cid/0, method: GET, req_id: 8d40470f-7ff7-4c24-bbe9-36f3aa1c54c1, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
225 001/300 Flush
226 002/300 Write block 125 len 16384 data: 2 2 2 2
227 003/300 Write block 1364 len 32768 data: 2 2 2 2 2 2 2 2
228 004/300 Read block 800 len 8192
229 005/300 Read block 16 len 32768
230 006/300 Read block 1420 len 40960
231 007/300 Read block 4555 len 20480
232 008/300 Write block 496 len 28672 data: 2 2 2 2 2 2 2
233 009/300 Read block 3239 len 20480
234 010/300 Write block 2262 len 24576 data: 2 2 2 2 2 2
235 011/300 Write block 2587 len 32768 data: 2 2 2 2 2 2 2 2
236 012/300 Write block 157 len 24576 data: 2 2 2 2 2 2
237 013/300 Write block 3234 len 36864 data: 2 2 2 2 2 2 2 2 2
238 014/300 Read block 3511 len 16384
239 015/300 Read block 2320 len 20480
240 016/300 Write block 4034 len 24576 data: 2 2 2 2 2 2
241 017/300 Read block 90 len 28672
242 018/300 Write block 1419 len 8192 data: 2 2
243 019/300 Read block 4427 len 40960
244 020/300 Write block 773 len 8192 data: 2 2
245 021/300 Write block 2707 len 32768 data: 2 2 2 2 2 2 2 2
246 022/300 Write block 1719 len 8192 data: 2 2
247 023/300 Write block 4079 len 28672 data: 2 2 2 2 2 2 2
248 024/300 Write block 2528 len 36864 data: 2 2 2 2 2 2 2 2 2
249 025/300 Write block 2051 len 36864 data: 2 2 2 2 2 2 2 2 2
250 026/300 Read block 3698 len 16384
251 027/300 Write block 1506 len 20480 data: 2 2 2 2 2
252 028/300 Read block 408 len 32768
253 029/300 Read block 319 len 28672
254 030/300 Flush
255 031/300 Write block 912 len 8192 data: 2 2
256 032/300 Read block 490 len 40960
257 033/300 Read block 307 len 16384
258 034/300 Read block 817 len 8192
259 035/300 Read block 1499 len 36864
260 036/300 Write block 1907 len 40960 data: 2 2 2 2 2 2 2 2 2 2
261 037/300 Write block 298 len 32768 data: 2 2 2 2 2 2 2 2
262 038/300 Read block 51 len 24576
263 039/300 Read block 3722 len 8192
264 040/300 Read block 2298 len 16384
265 041/300 Read block 2625 len 4096
266 042/300 Write block 1275 len 32768 data: 2 2 2 2 2 2 2 2
267 043/300 Flush
268 044/300 Read block 4487 len 36864
269 045/300 Flush
270 046/300 Write block 1390 len 20480 data: 2 2 2 2 2
271 047/300 Read block 495 len 12288
272 048/300 Read block 1855 len 40960
273 049/300 Read block 92 len 4096
274 050/300 Write block 3346 len 36864 data: 2 2 2 2 2 2 2 2 2
275 051/300 Read block 201 len 12288
276 052/300 Write block 2887 len 36864 data: 2 2 2 2 2 2 2 2 2
277 053/300 Read block 354 len 20480
278 054/300 Write block 595 len 8192 data: 2 2
279 055/300 Read block 693 len 12288
280 056/300 Read block 1902 len 4096
281 057/300 Write block 3430 len 24576 data: 2 2 2 2 2 2
282 058/300 Read block 3982 len 36864
283 059/300 Read block 4702 len 12288
284 060/300 Write block 102 len 12288 data: 2 2 2
285 061/300 Read block 4545 len 28672
286 062/300 Read block 1246 len 20480
287 063/300 Read block 4626 len 32768
288 064/300 Write block 630 len 4096 data: 2
289 065/300 Read block 3494 len 4096
290 066/300 Write block 2960 len 32768 data: 2 2 2 2 2 2 2 2
291 067/300 Read block 3064 len 36864
292 068/300 Flush
293 069/300 Write block 822 len 32768 data: 2 2 2 2 2 2 2 2
294 070/300 Read block 4233 len 12288
295 071/300 Read block 510 len 36864
296 072/300 Write block 406 len 12288 data: 2 2 2
297 073/300 Read block 1533 len 4096
298 074/300 Write block 1665 len 4096 data: 2
299 075/300 Read block 4072 len 28672
300 076/300 Write block 125 len 16384 data: 3 3 3 3
301 077/300 Read block 1820 len 16384
302 078/300 Read block 754 len 40960
303 079/300 Read block 2322 len 8192
304 080/300 Read block 207 len 36864
305 081/300 Write block 3183 len 20480 data: 2 2 2 2 2
306 082/300 Read block 751 len 36864
307 083/300 Read block 1392 len 16384
308 084/300 Read block 2053 len 8192
309 085/300 Flush
310 086/300 Read block 2805 len 40960
311 087/300 Write block 3835 len 28672 data: 2 2 2 2 2 2 2
312 088/300 Read block 657 len 4096
313 089/300 Write block 627 len 24576 data: 2 2 2 3 2 2
314 090/300 Flush
315 091/300 Write block 2203 len 24576 data: 2 2 2 2 2 2
316 092/300 Read block 2873 len 20480
317 093/300 Read block 4839 len 32768
318 094/300 Write block 1477 len 20480 data: 2 2 2 2 2
319 095/300 Read block 2450 len 12288
320 096/300 Flush
321 097/300 Write block 282 len 36864 data: 2 2 2 2 2 2 2 2 2
322 098/300 Write block 4200 len 12288 data: 2 2 2
323 099/300 Write block 367 len 8192 data: 2 2
324 100/300 Read block 475 len 16384
325 101/300 Write block 4239 len 40960 data: 2 2 2 2 2 2 2 2 2 2
326 102/300 Write block 2105 len 24576 data: 2 2 2 2 2 2
327 103/300 Read block 2006 len 40960
328 104/300 Read block 3250 len 36864
329 105/300 Write block 4677 len 28672 data: 2 2 2 2 2 2 2
330 106/300 Write block 1510 len 36864 data: 3 2 2 2 2 2 2 2 2
331 107/300 Read block 3973 len 32768
332 108/300 Read block 2604 len 20480
333 109/300 Read block 2017 len 36864
334 110/300 Write block 2971 len 28672 data: 2 2 2 2 2 2 2
335 111/300 Flush
336 112/300 Write block 171 len 32768 data: 2 2 2 2 2 2 2 2
337 113/300 Read block 2 len 32768
338 114/300 Write block 3806 len 28672 data: 2 2 2 2 2 2 2
339 115/300 Read block 1917 len 28672
340 116/300 Read block 3189 len 16384
341 117/300 Read block 575 len 12288
342 118/300 Write block 2554 len 20480 data: 2 2 2 2 2
343 119/300 Flush
344 120/300 Read block 4328 len 4096
345 121/300 Write block 2824 len 4096 data: 2
346 122/300 Write block 705 len 8192 data: 2 2
347 123/300 Read block 415 len 40960
348 124/300 Read block 4852 len 36864
349 125/300 Write block 1577 len 4096 data: 2
350 126/300 Read block 3423 len 12288
351 127/300 Write block 1869 len 4096 data: 2
352 128/300 Read block 4228 len 32768
353 129/300 Read block 2144 len 12288
354 130/300 Read block 189 len 20480
355 131/300 Write block 2327 len 32768 data: 2 2 2 2 2 2 2 2
356 132/300 Read block 1965 len 16384
357 133/300 Write block 2624 len 4096 data: 2
358 134/300 Read block 2712 len 28672
359 135/300 Write block 4721 len 32768 data: 2 2 2 2 2 2 2 2
360 136/300 Write block 619 len 20480 data: 2 2 2 2 2
361 137/300 Read block 1828 len 12288
362 138/300 Read block 3800 len 20480
363 139/300 Read block 4552 len 16384
364 140/300 Write block 237 len 4096 data: 2
365 141/300 Read block 1325 len 36864
366 142/300 Read block 3465 len 8192
367 143/300 Read block 468 len 40960
368 144/300 Flush
369 145/300 Write block 4773 len 32768 data: 2 2 2 2 2 2 2 2
370 146/300 Write block 367 len 36864 data: 3 3 2 2 2 2 2 2 2
371 147/300 Read block 446 len 4096
372 148/300 Flush
373 149/300 Read block 99 len 40960
374 150/300 Read block 4669 len 16384
375 151/300 Write block 335 len 4096 data: 2
376 152/300 Flush
377 153/300 Read block 2199 len 12288
378 154/300 Write block 3857 len 12288 data: 2 2 2
379 155/300 Read block 4757 len 24576
380 156/300 Read block 4571 len 16384
381 157/300 Read block 2122 len 4096
382 158/300 Write block 2802 len 20480 data: 2 2 2 2 2
383 159/300 Read block 2037 len 32768
384 160/300 Flush
385 161/300 Read block 722 len 40960
386 162/300 Read block 4558 len 36864
387 163/300 Read block 4013 len 4096
388 164/300 Flush
389 165/300 Write block 2866 len 16384 data: 2 2 2 2
390 166/300 Read block 1883 len 28672
391 167/300 Read block 4435 len 12288
392 168/300 Read block 2425 len 20480
393 169/300 Read block 4112 len 32768
394 170/300 Write block 771 len 40960 data: 2 2 3 3 2 2 2 2 2 2
395 171/300 Read block 1827 len 36864
396 172/300 Write block 1755 len 40960 data: 2 2 2 2 2 2 2 2 2 2
397 173/300 Read block 1370 len 4096
398 174/300 Read block 1248 len 16384
399 175/300 Read block 3454 len 36864
400 176/300 Read block 2761 len 40960
401 177/300 Read block 4518 len 24576
402 178/300 Write block 3433 len 20480 data: 3 3 3 2 2
403 179/300 Write block 1957 len 40960 data: 2 2 2 2 2 2 2 2 2 2
404 180/300 Read block 1989 len 28672
405 181/300 Write block 3522 len 20480 data: 2 2 2 2 2
406 182/300 Read block 1688 len 16384
407 183/300 Read block 2310 len 40960
408 184/300 Read block 2699 len 36864
409 185/300 Read block 1689 len 28672
410 186/300 Read block 98 len 20480
411 187/300 Write block 1097 len 28672 data: 2 2 2 2 2 2 2
412 188/300 Read block 151 len 40960
413 189/300 Read block 1896 len 12288
414 190/300 Write block 3732 len 4096 data: 2
415 191/300 Read block 3354 len 16384
416 192/300 Read block 4363 len 20480
417 193/300 Read block 1978 len 8192
418 194/300 Write block 3033 len 12288 data: 2 2 2
419 195/300 Read block 4973 len 20480
420 196/300 Read block 3098 len 16384
421 197/300 Read block 211 len 32768
422 198/300 Write block 2639 len 40960 data: 2 2 2 2 2 2 2 2 2 2
423 199/300 Write block 571 len 16384 data: 2 2 2 2
424 200/300 Write block 369 len 40960 data: 3 3 3 3 3 3 3 2 2 2
425 201/300 Read block 901 len 32768
426 202/300 Write block 4757 len 32768 data: 2 2 2 2 2 2 2 2
427 203/300 Read block 1846 len 28672
428 204/300 Read block 3322 len 28672
429 205/300 Read block 4343 len 36864
430 206/300 Flush
431 207/300 Write block 3532 len 12288 data: 2 2 2
432 208/300 Write block 3893 len 20480 data: 2 2 2 2 2
433 209/300 Read block 3060 len 8192
434 210/300 Read block 318 len 36864
435 211/300 Read block 1763 len 28672
436 212/300 Read block 648 len 36864
437 213/300 Read block 3655 len 32768
438 214/300 Write block 3958 len 32768 data: 2 2 2 2 2 2 2 2
439 215/300 Read block 3588 len 4096
440 216/300 Read block 2828 len 8192
441 217/300 Read block 2912 len 8192
442 218/300 Read block 757 len 24576
443 219/300 Write block 1139 len 28672 data: 2 2 2 2 2 2 2
444 220/300 Read block 2096 len 4096
445 221/300 Write block 192 len 40960 data: 2 2 2 2 2 2 2 2 2 2
446 222/300 Read block 1587 len 24576
447 223/300 Read block 4510 len 20480
448 224/300 Read block 2523 len 36864
449 225/300 Write block 4431 len 24576 data: 2 2 2 2 2 2
450 226/300 Write block 4971 len 8192 data: 2 2
451 227/300 Read block 4381 len 32768
452 228/300 Write block 77 len 16384 data: 2 2 2 2
453 229/300 Read block 3767 len 20480
454 230/300 Write block 2067 len 12288 data: 2 2 2
455 231/300 Read block 2446 len 40960
456 232/300 Read block 848 len 8192
457 233/300 Write block 2234 len 4096 data: 2
458 234/300 Flush
459 235/300 Read block 2666 len 28672
460 236/300 Read block 82 len 28672
461 237/300 Read block 3357 len 8192
462 238/300 Read block 1531 len 24576
463 239/300 Write block 3509 len 40960 data: 2 2 2 2 2 2 2 2 2 2
464 240/300 Flush
465 241/300 Flush
466 242/300 Read block 1771 len 20480
467 243/300 Read block 3686 len 24576
468 244/300 Write block 17 len 32768 data: 2 2 2 2 2 2 2 2
469 245/300 Flush
470 246/300 Write block 2837 len 4096 data: 2
471 247/300 Flush
472 248/300 Write block 4709 len 8192 data: 2 2
473 249/300 Read block 272 len 20480
474 250/300 Read block 1508 len 32768
475 251/300 Write block 1616 len 8192 data: 2 2
476 252/300 Write block 1401 len 16384 data: 2 2 2 2
477 253/300 Read block 4771 len 28672
478 254/300 Read block 1221 len 8192
479 255/300 Write block 3920 len 20480 data: 2 2 2 2 2
480 256/300 Read block 949 len 20480
481 257/300 Write block 1324 len 20480 data: 2 2 2 2 2
482 258/300 Write block 1838 len 12288 data: 2 2 2
483 259/300 Read block 821 len 16384
484 260/300 Write block 441 len 32768 data: 2 2 2 2 2 2 2 2
485 261/300 Write block 1724 len 32768 data: 2 2 2 2 2 2 2 2
486 262/300 Flush
487 263/300 Read block 2335 len 20480
488 264/300 Flush
489 265/300 Write block 1284 len 40960 data: 2 2 2 2 2 2 2 2 2 2
490 266/300 Flush
491 267/300 Read block 1998 len 16384
492 268/300 Write block 1307 len 20480 data: 2 2 2 2 2
493 269/300 Write block 3391 len 36864 data: 2 2 2 2 2 2 2 2 2
494 270/300 Read block 3911 len 24576
495 271/300 Read block 2856 len 28672
496 272/300 Write block 1421 len 32768 data: 2 2 2 2 2 2 2 2
497 273/300 Read block 3656 len 40960
498 274/300 Write block 3940 len 32768 data: 2 2 2 2 2 2 2 2
499 275/300 Read block 2859 len 8192
500 276/300 Flush
501 277/300 Write block 4140 len 40960 data: 2 2 2 2 2 2 2 2 2 2
502 278/300 Write block 123 len 36864 data: 2 2 4 4 4 4 2 2 2
503 279/300 Read block 2258 len 4096
504 280/300 Flush
505 281/300 Write block 3611 len 40960 data: 2 2 2 2 2 2 2 2 2 2
506 282/300 Read block 2386 len 32768
507 283/300 Write block 2301 len 36864 data: 2 2 2 2 2 2 2 2 2
508 284/300 Read block 2170 len 8192
509 285/300 Write block 4032 len 32768 data: 2 2 3 3 3 3 3 3
510 286/300 Read block 3443 len 40960
511 287/300 Read block 3610 len 8192
512 288/300 Read block 3809 len 16384
513 289/300 Write block 61 len 40960 data: 2 2 2 2 2 2 2 2 2 2
514 290/300 Write block 4193 len 40960 data: 2 2 2 2 2 2 2 3 3 3
515 291/300 Read block 2314 len 4096
516 292/300 Read block 3163 len 20480
517 293/300 Read block 211 len 40960
518 294/300 Read block 117 len 8192
519 295/300 Write block 1470 len 20480 data: 2 2 2 2 2
520 296/300 Read block 722 len 36864
521 297/300 Write block 1642 len 8192 data: 2 2
522 298/300 Flush
523 299/300 Write block 533 len 36864 data: 2 2 2 2 2 2 2 2 2
524 300/300 Read block 165 len 32768
525 301/300 Read block 1679 len 20480
526 Main task has work to do, go find it
527 got dsc Start(0)
528 start 0
529 Jan 23 17:37:51.860 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/0, method: POST, req_id: 6958aee9-7f38-44d4-9b3c-5be6aa4c78b3, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
530 Replay: started 0, returned:Ok(())
531 ----------------------------------------------------------------
532 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
533 States: Offline Active Active
534 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
535 1000 Acked Flush 0 Sent Done Done false
536 1001 Acked Write 4 Sent Done Done false
537 1002 Acked Write 8 Sent Done Done false
538 1003 Acked Read 2 Sent Done Done false
539 1004 Acked Read 8 Sent Done Done false
540 1005 Acked Read 10 Sent Done Done false
541 1006 Acked Read 5 Sent Done Done false
542 1007 Acked Write 7 Sent Done Done false
543 1008 Acked Read 5 Sent Done Done false
544 1009 Acked Write 6 Sent Done Done false
545 1010 Acked Write 8 Sent Done Done false
546 1011 Acked Write 6 Sent Done Done false
547 1012 Acked Write 9 Sent Done Done false
548 1013 Acked Read 4 Sent Done Done false
549 1014 Acked Read 5 Sent Done Done false
550 1015 Acked Write 6 Sent Done Done false
551 1016 Acked Read 7 Sent Done Done false
552 1017 Acked Write 2 Sent Done Done false
553 1018 Acked Read 10 Sent Done Done false
554 1019 Acked Write 2 Sent Done Done false
555 1020 Acked Write 8 Sent Done Done false
556 1021 Acked Write 2 Sent Done Done false
557 1022 Acked Write 7 Sent Done Done false
558 1023 Acked Write 9 Sent Done Done false
559 1024 Acked Write 9 Sent Done Done false
560 1025 Acked Read 4 Sent Done Done false
561 1026 Acked Write 5 Sent Done Done false
562 1027 Acked Read 8 Sent Done Done false
563 1028 Acked Read 7 Sent Done Done false
564 1029 Acked Flush 0 Sent Done Done false
565 1030 Acked Write 2 Sent Done Done false
566 1031 Acked Read 10 Sent Done Done false
567 1032 Acked Read 4 Sent Done Done false
568 1033 Acked Read 2 Sent Done Done false
569 1034 Acked Read 9 Sent Done Done false
570 1035 Acked Write 10 Sent Done Done false
571 1036 Acked Write 8 Sent Done Done false
572 1037 Acked Read 6 Sent Done Done false
573 1038 Acked Read 2 Sent Done Done false
574 1039 Acked Read 4 Sent Done Done false
575 1040 Acked Read 1 Sent Done Done false
576 1041 Acked Write 8 Sent Done Done false
577 1042 Acked Flush 0 Sent Done Done false
578 1043 Acked Read 9 Sent Done Done false
579 1044 Acked Flush 0 Sent Done Done false
580 1045 Acked Write 5 Sent Done Done false
581 1046 Acked Read 3 Sent Done Done false
582 1047 Acked Read 10 Sent Done Done false
583 1048 Acked Read 1 Sent Done Done false
584 1049 Acked Write 9 Sent Done Done false
585 1050 Acked Read 3 Sent Done Done false
586 1051 Acked Write 9 Sent Done Done false
587 1052 Acked Read 5 Sent Done Done false
588 1053 Acked Write 2 Sent Done Done false
589 1054 Acked Read 3 Sent Done Done false
590 1055 Acked Read 1 Sent Done Done false
591 1056 Acked Write 6 Sent Done Done false
592 1057 Acked Read 9 Sent Done Done false
593 1058 Acked Read 3 Sent Done Done false
594 1059 Acked Write 3 Sent Done Done false
595 1060 Acked Read 7 Sent Done Done false
596 1061 Acked Read 5 Sent Done Done false
597 1062 Acked Read 8 Sent Done Done false
598 1063 Acked Write 1 Sent Done Done false
599 1064 Acked Read 1 Sent Done Done false
600 1065 Acked Write 8 Sent Done Done false
601 1066 Acked Read 9 Sent Done Done false
602 1067 Acked Flush 0 Sent Done Done false
603 1068 Acked Write 8 Sent Done Done false
604 1069 Acked Read 3 Sent Done Done false
605 1070 Acked Read 9 Sent Done Done false
606 1071 Acked Write 3 Sent Done Done false
607 1072 Acked Read 1 Sent Done Done false
608 1073 Acked Write 1 Sent Done Done false
609 1074 Acked Read 7 Sent Done Done false
610 1075 Acked Write 4 Sent Done Done false
611 1076 Acked Read 4 Sent Done Done false
612 1077 Acked Read 10 Sent Done Done false
613 1078 Acked Read 2 Sent Done Done false
614 1079 Acked Read 9 Sent Done Done false
615 1080 Acked Write 5 Sent Done Done false
616 1081 Acked Read 9 Sent Done Done false
617 1082 Acked Read 4 Sent Done Done false
618 1083 Acked Read 2 Sent Done Done false
619 1084 Acked Flush 0 Sent Done Done false
620 1085 Acked Read 10 Sent Done Done false
621 1086 Acked Write 7 Sent Done Done false
622 1087 Acked Read 1 Sent Done Done false
623 1088 Acked Write 6 Sent Done Done false
624 1089 Acked Flush 0 Sent Done Done false
625 1090 Acked Write 6 Sent Done Done false
626 1091 Acked Read 5 Sent Done Done false
627 1092 Acked Read 8 Sent Done Done false
628 1093 Acked Write 5 Sent Done Done false
629 1094 Acked Read 3 Sent Done Done false
630 1095 Acked Flush 0 Sent Done Done false
631 1096 Acked Write 9 Sent Done Done false
632 1097 Acked Write 3 Sent Done Done false
633 1098 Acked Write 2 Sent Done Done false
634 1099 Acked Read 4 Sent Done Done false
635 1100 Acked Write 10 Sent Done Done false
636 1101 Acked Write 6 Sent Done Done false
637 1102 Acked Read 10 Sent Done Done false
638 1103 Acked Read 9 Sent Done Done false
639 1104 Acked Write 7 Sent Done Done false
640 1105 Acked Write 9 Sent Done Done false
641 1106 Acked Read 8 Sent Done Done false
642 1107 Acked Read 5 Sent Done Done false
643 1108 Acked Read 9 Sent Done Done false
644 1109 Acked Write 7 Sent Done Done false
645 1110 Acked Flush 0 Sent Done Done false
646 1111 Acked Write 8 Sent Done Done false
647 1112 Acked Read 8 Sent Done Done false
648 1113 Acked Write 7 Sent Done Done false
649 1114 Acked Read 7 Sent Done Done false
650 1115 Acked Read 4 Sent Done Done false
651 1116 Acked Read 3 Sent Done Done false
652 1117 Acked Write 5 Sent Done Done false
653 1118 Acked Flush 0 Sent Done Done false
654 1119 Acked Read 1 Sent Done Done false
655 1120 Acked Write 1 Sent Done Done false
656 1121 Acked Write 2 Sent Done Done false
657 1122 Acked Read 10 Sent Done Done false
658 1123 Acked Read 9 Sent Done Done false
659 1124 Acked Write 1 Sent Done Done false
660 1125 Acked Read 3 Sent Done Done false
661 1126 Acked Write 1 Sent Done Done false
662 1127 Acked Read 8 Sent Done Done false
663 1128 Acked Read 3 Sent Done Done false
664 1129 Acked Read 5 Sent Done Done false
665 1130 Acked Write 8 Sent Done Done false
666 1131 Acked Read 4 Sent Done Done false
667 1132 Acked Write 1 Sent Done Done false
668 1133 Acked Read 7 Sent Done Done false
669 1134 Acked Write 8 Sent Done Done false
670 1135 Acked Write 5 Sent Done Done false
671 1136 Acked Read 3 Sent Done Done false
672 1137 Acked Read 5 Sent Done Done false
673 1138 Acked Read 4 Sent Done Done false
674 1139 Acked Write 1 Sent Done Done false
675 1140 Acked Read 9 Sent Done Done false
676 1141 Acked Read 2 Sent Done Done false
677 1142 Acked Read 10 Sent Done Done false
678 1143 Acked Flush 0 Sent Done Done false
679 1144 Acked Write 8 Sent Done Done false
680 1145 Acked Write 9 Sent Done Done false
681 1146 Acked Read 1 Sent Done Done false
682 1147 Acked Flush 0 Sent Done Done false
683 1148 Acked Read 10 Sent Done Done false
684 1149 Acked Read 4 Sent Done Done false
685 1150 Acked Write 1 Sent Done Done false
686 1151 Acked Flush 0 Sent Done Done false
687 1152 Acked Read 3 Sent Done Done false
688 1153 Acked Write 3 Sent Done Done false
689 1154 Acked Read 6 Sent Done Done false
690 1155 Acked Read 4 Sent Done Done false
691 1156 Acked Read 1 Sent Done Done false
692 1157 Acked Write 5 Sent Done Done false
693 1158 Acked Read 8 Sent Done Done false
694 1159 Acked Flush 0 Sent Done Done false
695 1160 Acked Read 10 Sent Done Done false
696 1161 Acked Read 9 Sent Done Done false
697 1162 Acked Read 1 Sent Done Done false
698 1163 Acked Flush 0 Sent Done Done false
699 1164 Acked Write 4 Sent Done Done false
700 1165 Acked Read 7 Sent Done Done false
701 1166 Acked Read 3 Sent Done Done false
702 1167 Acked Read 5 Sent Done Done false
703 1168 Acked Read 8 Sent Done Done false
704 1169 Acked Write 10 Sent Done Done false
705 1170 Acked Read 9 Sent Done Done false
706 1171 Acked Write 10 Sent Done Done false
707 1172 Acked Read 1 Sent Done Done false
708 1173 Acked Read 4 Sent Done Done false
709 1174 Acked Read 9 Sent Done Done false
710 1175 Acked Read 10 Sent Done Done false
711 1176 Acked Read 6 Sent Done Done false
712 1177 Acked Write 5 Sent Done Done false
713 1178 Acked Write 10 Sent Done Done false
714 1179 Acked Read 7 Sent Done Done false
715 1180 Acked Write 5 Sent Done Done false
716 1181 Acked Read 4 Sent Done Done false
717 1182 Acked Read 10 Sent Done Done false
718 1183 Acked Read 9 Sent Done Done false
719 1184 Acked Read 7 Sent Done Done false
720 1185 Acked Read 5 Sent Done Done false
721 1186 Acked Write 7 Sent Done Done false
722 1187 Acked Read 10 Sent Done Done false
723 1188 Acked Read 3 Sent Done Done false
724 1189 Acked Write 1 Sent Done Done false
725 1190 Acked Read 4 Sent Done Done false
726 1191 Acked Read 5 Sent Done Done false
727 1192 Acked Read 2 Sent Done Done false
728 1193 Acked Write 3 Sent Done Done false
729 1194 Acked Read 5 Sent Done Done false
730 1195 Acked Read 4 Sent Done Done false
731 1196 Acked Read 8 Sent Done Done false
732 1197 Acked Write 10 Sent Done Done false
733 1198 Acked Write 4 Sent Done Done false
734 1199 Acked Write 10 Sent Done Done false
735 1200 Acked Read 8 Sent Done Done false
736 1201 Acked Write 8 Sent Done Done false
737 1202 Acked Read 7 Sent Done Done false
738 1203 Acked Read 7 Sent Done Done false
739 1204 Acked Read 9 Sent Done Done false
740 1205 Acked Flush 0 Sent Done Done false
741 1206 Acked Write 3 Sent Done Done false
742 1207 Acked Write 5 Sent Done Done false
743 1208 Acked Read 2 Sent Done Done false
744 1209 Acked Read 9 Sent Done Done false
745 1210 Acked Read 7 Sent Done Done false
746 1211 Acked Read 9 Sent Done Done false
747 1212 Acked Read 8 Sent Done Done false
748 1213 Acked Write 8 Sent Done Done false
749 1214 Acked Read 1 Sent Done Done false
750 1215 Acked Read 2 Sent Done Done false
751 1216 Acked Read 2 Sent Done Done false
752 1217 Acked Read 6 Sent Done Done false
753 1218 Acked Write 7 Sent Done Done false
754 1219 Acked Read 1 Sent Done Done false
755 1220 Acked Write 10 Sent Done Done false
756 1221 Acked Read 6 Sent Done Done false
757 1222 Acked Read 5 Sent Done Done false
758 1223 Acked Read 9 Sent Done Done false
759 1224 Acked Write 6 Sent Done Done false
760 1225 Acked Write 2 Sent Done Done false
761 1226 Acked Read 8 Sent Done Done false
762 1227 Acked Write 4 Sent Done Done false
763 1228 Acked Read 5 Sent Done Done false
764 1229 Acked Write 3 Sent Done Done false
765 1230 Acked Read 10 Sent Done Done false
766 1231 Acked Read 2 Sent Done Done false
767 1232 Acked Write 1 Sent Done Done false
768 1233 Acked Flush 0 Sent Done Done false
769 1234 Acked Read 7 Sent Done Done false
770 1235 Acked Read 7 Sent Done Done false
771 1236 Acked Read 2 Sent Done Done false
772 1237 Acked Read 6 Sent Done Done false
773 1238 Acked Write 10 Sent Done Done false
774 1239 Acked Flush 0 Sent Done Done false
775 1240 Acked Flush 0 Sent Done Done false
776 1241 Acked Read 5 Sent Done Done false
777 1242 Acked Read 6 Sent Done Done false
778 1243 Acked Write 8 Sent Done Done false
779 1244 Acked Flush 0 Sent Done Done false
780 1245 Acked Write 1 Sent Done Done false
781 1246 Acked Flush 0 Sent Done Done false
782 1247 Acked Write 2 Sent Done Done false
783 1248 Acked Read 5 Sent Done Done false
784 1249 Acked Read 8 Sent Done Done false
785 1250 Acked Write 2 Sent Done Done false
786 1251 Acked Write 4 Sent Done Done false
787 1252 Acked Read 7 Sent Done Done false
788 1253 Acked Read 2 Sent Done Done false
789 1254 Acked Write 5 Sent Done Done false
790 1255 Acked Read 5 Sent Done Done false
791 1256 Acked Write 5 Sent Done Done false
792 1257 Acked Write 3 Sent Done Done false
793 1258 Acked Read 4 Sent Done Done false
794 1259 Acked Write 8 Sent Done Done false
795 1260 Acked Write 8 Sent Done Done false
796 1261 Acked Flush 0 Sent Done Done false
797 1262 Acked Read 5 Sent Done Done false
798 1263 Acked Flush 0 Sent Done Done false
799 1264 Acked Write 10 Sent Done Done false
800 1265 Acked Flush 0 Sent Done Done false
801 1266 Acked Read 4 Sent Done Done false
802 1267 Acked Write 5 Sent Done Done false
803 1268 Acked Write 9 Sent Done Done false
804 1269 Acked Read 6 Sent Done Done false
805 1270 Acked Read 7 Sent Done Done false
806 1271 Acked Write 8 Sent Done Done false
807 1272 Acked Read 10 Sent Done Done false
808 1273 Acked Write 8 Sent Done Done false
809 1274 Acked Read 2 Sent Done Done false
810 1275 Acked Flush 0 Sent Done Done false
811 1276 Acked Write 10 Sent Done Done false
812 1277 Acked Write 9 Sent Done Done false
813 1278 Acked Read 1 Sent Done Done false
814 1279 Acked Flush 0 Sent Done Done false
815 1280 Acked Write 10 Sent Done Done false
816 1281 Acked Read 8 Sent Done Done false
817 1282 Acked Write 9 Sent Done Done false
818 1283 Acked Read 2 Sent Done Done false
819 1284 Acked Write 8 Sent Done Done false
820 1285 Acked Read 10 Sent Done Done false
821 1286 Acked Read 2 Sent Done Done false
822 1287 Acked Read 4 Sent Done Done false
823 1288 Acked Write 10 Sent Done Done false
824 1289 Acked Write 10 Sent Done Done false
825 1290 Acked Read 1 Sent Done Done false
826 1291 Acked Read 5 Sent Done Done false
827 1292 Acked Read 10 Sent Done Done false
828 1293 Acked Read 2 Sent Done Done false
829 1294 Acked Write 5 Sent Done Done false
830 1295 Acked Read 9 Sent Done Done false
831 1296 Acked Write 2 Sent Done Done false
832 1297 Acked Flush 0 Sent Done Done false
833 1298 Acked Write 9 Sent Done Done false
834 1299 Acked Read 8 Sent Done Done false
835 1300 Acked Read 5 Sent Done Done false
836 STATES DS:0 DS:1 DS:2 TOTAL
837 Sent 301 0 0 301
838 Done 0 301 301 602
839 Skipped 0 0 0 0
840 Error 0 0 0 0
841 Last Flush: None Some(JobId(1297)) Some(JobId(1297))
842 Downstairs last five completed:
843 Upstairs last five completed: 1300 1299 1298 1297 1296
844 CLIENT: Up:0 ds:301 act:2
845 [8810] got start action
846 [8810] I am going to restart
847 Starting downstairs at port 8810
848 Make output file at "/tmp/dsc/downstairs-8810.txt"
849 [8810][0] reports Starting
850 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1206)
851 [8810][0] reports Running
852 ----------------------------------------------------------------
853 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
854 States: Offline Active Active
855 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
856 1000 Acked Flush 0 Sent Done Done false
857 1001 Acked Write 4 Sent Done Done false
858 1002 Acked Write 8 Sent Done Done false
859 1003 Acked Read 2 Sent Done Done false
860 1004 Acked Read 8 Sent Done Done false
861 1005 Acked Read 10 Sent Done Done false
862 1006 Acked Read 5 Sent Done Done false
863 1007 Acked Write 7 Sent Done Done false
864 1008 Acked Read 5 Sent Done Done false
865 1009 Acked Write 6 Sent Done Done false
866 1010 Acked Write 8 Sent Done Done false
867 1011 Acked Write 6 Sent Done Done false
868 1012 Acked Write 9 Sent Done Done false
869 1013 Acked Read 4 Sent Done Done false
870 1014 Acked Read 5 Sent Done Done false
871 1015 Acked Write 6 Sent Done Done false
872 1016 Acked Read 7 Sent Done Done false
873 1017 Acked Write 2 Sent Done Done false
874 1018 Acked Read 10 Sent Done Done false
875 1019 Acked Write 2 Sent Done Done false
876 1020 Acked Write 8 Sent Done Done false
877 1021 Acked Write 2 Sent Done Done false
878 1022 Acked Write 7 Sent Done Done false
879 1023 Acked Write 9 Sent Done Done false
880 1024 Acked Write 9 Sent Done Done false
881 1025 Acked Read 4 Sent Done Done false
882 1026 Acked Write 5 Sent Done Done false
883 1027 Acked Read 8 Sent Done Done false
884 1028 Acked Read 7 Sent Done Done false
885 1029 Acked Flush 0 Sent Done Done false
886 1030 Acked Write 2 Sent Done Done false
887 1031 Acked Read 10 Sent Done Done false
888 1032 Acked Read 4 Sent Done Done false
889 1033 Acked Read 2 Sent Done Done false
890 1034 Acked Read 9 Sent Done Done false
891 1035 Acked Write 10 Sent Done Done false
892 1036 Acked Write 8 Sent Done Done false
893 1037 Acked Read 6 Sent Done Done false
894 1038 Acked Read 2 Sent Done Done false
895 1039 Acked Read 4 Sent Done Done false
896 1040 Acked Read 1 Sent Done Done false
897 1041 Acked Write 8 Sent Done Done false
898 1042 Acked Flush 0 Sent Done Done false
899 1043 Acked Read 9 Sent Done Done false
900 1044 Acked Flush 0 Sent Done Done false
901 1045 Acked Write 5 Sent Done Done false
902 1046 Acked Read 3 Sent Done Done false
903 1047 Acked Read 10 Sent Done Done false
904 1048 Acked Read 1 Sent Done Done false
905 1049 Acked Write 9 Sent Done Done false
906 1050 Acked Read 3 Sent Done Done false
907 1051 Acked Write 9 Sent Done Done false
908 1052 Acked Read 5 Sent Done Done false
909 1053 Acked Write 2 Sent Done Done false
910 1054 Acked Read 3 Sent Done Done false
911 1055 Acked Read 1 Sent Done Done false
912 1056 Acked Write 6 Sent Done Done false
913 1057 Acked Read 9 Sent Done Done false
914 1058 Acked Read 3 Sent Done Done false
915 1059 Acked Write 3 Sent Done Done false
916 1060 Acked Read 7 Sent Done Done false
917 1061 Acked Read 5 Sent Done Done false
918 1062 Acked Read 8 Sent Done Done false
919 1063 Acked Write 1 Sent Done Done false
920 1064 Acked Read 1 Sent Done Done false
921 1065 Acked Write 8 Sent Done Done false
922 1066 Acked Read 9 Sent Done Done false
923 1067 Acked Flush 0 Sent Done Done false
924 1068 Acked Write 8 Sent Done Done false
925 1069 Acked Read 3 Sent Done Done false
926 1070 Acked Read 9 Sent Done Done false
927 1071 Acked Write 3 Sent Done Done false
928 1072 Acked Read 1 Sent Done Done false
929 1073 Acked Write 1 Sent Done Done false
930 1074 Acked Read 7 Sent Done Done false
931 1075 Acked Write 4 Sent Done Done false
932 1076 Acked Read 4 Sent Done Done false
933 1077 Acked Read 10 Sent Done Done false
934 1078 Acked Read 2 Sent Done Done false
935 1079 Acked Read 9 Sent Done Done false
936 1080 Acked Write 5 Sent Done Done false
937 1081 Acked Read 9 Sent Done Done false
938 1082 Acked Read 4 Sent Done Done false
939 1083 Acked Read 2 Sent Done Done false
940 1084 Acked Flush 0 Sent Done Done false
941 1085 Acked Read 10 Sent Done Done false
942 1086 Acked Write 7 Sent Done Done false
943 1087 Acked Read 1 Sent Done Done false
944 1088 Acked Write 6 Sent Done Done false
945 1089 Acked Flush 0 Sent Done Done false
946 1090 Acked Write 6 Sent Done Done false
947 1091 Acked Read 5 Sent Done Done false
948 1092 Acked Read 8 Sent Done Done false
949 1093 Acked Write 5 Sent Done Done false
950 1094 Acked Read 3 Sent Done Done false
951 1095 Acked Flush 0 Sent Done Done false
952 1096 Acked Write 9 Sent Done Done false
953 1097 Acked Write 3 Sent Done Done false
954 1098 Acked Write 2 Sent Done Done false
955 1099 Acked Read 4 Sent Done Done false
956 1100 Acked Write 10 Sent Done Done false
957 1101 Acked Write 6 Sent Done Done false
958 1102 Acked Read 10 Sent Done Done false
959 1103 Acked Read 9 Sent Done Done false
960 1104 Acked Write 7 Sent Done Done false
961 1105 Acked Write 9 Sent Done Done false
962 1106 Acked Read 8 Sent Done Done false
963 1107 Acked Read 5 Sent Done Done false
964 1108 Acked Read 9 Sent Done Done false
965 1109 Acked Write 7 Sent Done Done false
966 1110 Acked Flush 0 Sent Done Done false
967 1111 Acked Write 8 Sent Done Done false
968 1112 Acked Read 8 Sent Done Done false
969 1113 Acked Write 7 Sent Done Done false
970 1114 Acked Read 7 Sent Done Done false
971 1115 Acked Read 4 Sent Done Done false
972 1116 Acked Read 3 Sent Done Done false
973 1117 Acked Write 5 Sent Done Done false
974 1118 Acked Flush 0 Sent Done Done false
975 1119 Acked Read 1 Sent Done Done false
976 1120 Acked Write 1 Sent Done Done false
977 1121 Acked Write 2 Sent Done Done false
978 1122 Acked Read 10 Sent Done Done false
979 1123 Acked Read 9 Sent Done Done false
980 1124 Acked Write 1 Sent Done Done false
981 1125 Acked Read 3 Sent Done Done false
982 1126 Acked Write 1 Sent Done Done false
983 1127 Acked Read 8 Sent Done Done false
984 1128 Acked Read 3 Sent Done Done false
985 1129 Acked Read 5 Sent Done Done false
986 1130 Acked Write 8 Sent Done Done false
987 1131 Acked Read 4 Sent Done Done false
988 1132 Acked Write 1 Sent Done Done false
989 1133 Acked Read 7 Sent Done Done false
990 1134 Acked Write 8 Sent Done Done false
991 1135 Acked Write 5 Sent Done Done false
992 1136 Acked Read 3 Sent Done Done false
993 1137 Acked Read 5 Sent Done Done false
994 1138 Acked Read 4 Sent Done Done false
995 1139 Acked Write 1 Sent Done Done false
996 1140 Acked Read 9 Sent Done Done false
997 1141 Acked Read 2 Sent Done Done false
998 1142 Acked Read 10 Sent Done Done false
999 1143 Acked Flush 0 Sent Done Done false
1000 1144 Acked Write 8 Sent Done Done false
1001 1145 Acked Write 9 Sent Done Done false
1002 1146 Acked Read 1 Sent Done Done false
1003 1147 Acked Flush 0 Sent Done Done false
1004 1148 Acked Read 10 Sent Done Done false
1005 1149 Acked Read 4 Sent Done Done false
1006 1150 Acked Write 1 Sent Done Done false
1007 1151 Acked Flush 0 Sent Done Done false
1008 1152 Acked Read 3 Sent Done Done false
1009 1153 Acked Write 3 Sent Done Done false
1010 1154 Acked Read 6 Sent Done Done false
1011 1155 Acked Read 4 Sent Done Done false
1012 1156 Acked Read 1 Sent Done Done false
1013 1157 Acked Write 5 Sent Done Done false
1014 1158 Acked Read 8 Sent Done Done false
1015 1159 Acked Flush 0 Sent Done Done false
1016 1160 Acked Read 10 Sent Done Done false
1017 1161 Acked Read 9 Sent Done Done false
1018 1162 Acked Read 1 Sent Done Done false
1019 1163 Acked Flush 0 Sent Done Done false
1020 1164 Acked Write 4 Sent Done Done false
1021 1165 Acked Read 7 Sent Done Done false
1022 1166 Acked Read 3 Sent Done Done false
1023 1167 Acked Read 5 Sent Done Done false
1024 1168 Acked Read 8 Sent Done Done false
1025 1169 Acked Write 10 Sent Done Done false
1026 1170 Acked Read 9 Sent Done Done false
1027 1171 Acked Write 10 Sent Done Done false
1028 1172 Acked Read 1 Sent Done Done false
1029 1173 Acked Read 4 Sent Done Done false
1030 1174 Acked Read 9 Sent Done Done false
1031 1175 Acked Read 10 Sent Done Done false
1032 1176 Acked Read 6 Sent Done Done false
1033 1177 Acked Write 5 Sent Done Done false
1034 1178 Acked Write 10 Sent Done Done false
1035 1179 Acked Read 7 Sent Done Done false
1036 1180 Acked Write 5 Sent Done Done false
1037 1181 Acked Read 4 Sent Done Done false
1038 1182 Acked Read 10 Sent Done Done false
1039 1183 Acked Read 9 Sent Done Done false
1040 1184 Acked Read 7 Sent Done Done false
1041 1185 Acked Read 5 Sent Done Done false
1042 1186 Acked Write 7 Sent Done Done false
1043 1187 Acked Read 10 Sent Done Done false
1044 1188 Acked Read 3 Sent Done Done false
1045 1189 Acked Write 1 Sent Done Done false
1046 1190 Acked Read 4 Sent Done Done false
1047 1191 Acked Read 5 Sent Done Done false
1048 1192 Acked Read 2 Sent Done Done false
1049 1193 Acked Write 3 Sent Done Done false
1050 1194 Acked Read 5 Sent Done Done false
1051 1195 Acked Read 4 Sent Done Done false
1052 1196 Acked Read 8 Sent Done Done false
1053 1197 Acked Write 10 Sent Done Done false
1054 1198 Acked Write 4 Sent Done Done false
1055 1199 Acked Write 10 Sent Done Done false
1056 1200 Acked Read 8 Sent Done Done false
1057 1201 Acked Write 8 Sent Done Done false
1058 1202 Acked Read 7 Sent Done Done false
1059 1203 Acked Read 7 Sent Done Done false
1060 1204 Acked Read 9 Sent Done Done false
1061 1205 Acked Flush 0 Sent Done Done false
1062 1206 Acked Write 3 Sent Done Done false
1063 1207 Acked Write 5 Sent Done Done false
1064 1208 Acked Read 2 Sent Done Done false
1065 1209 Acked Read 9 Sent Done Done false
1066 1210 Acked Read 7 Sent Done Done false
1067 1211 Acked Read 9 Sent Done Done false
1068 1212 Acked Read 8 Sent Done Done false
1069 1213 Acked Write 8 Sent Done Done false
1070 1214 Acked Read 1 Sent Done Done false
1071 1215 Acked Read 2 Sent Done Done false
1072 1216 Acked Read 2 Sent Done Done false
1073 1217 Acked Read 6 Sent Done Done false
1074 1218 Acked Write 7 Sent Done Done false
1075 1219 Acked Read 1 Sent Done Done false
1076 1220 Acked Write 10 Sent Done Done false
1077 1221 Acked Read 6 Sent Done Done false
1078 1222 Acked Read 5 Sent Done Done false
1079 1223 Acked Read 9 Sent Done Done false
1080 1224 Acked Write 6 Sent Done Done false
1081 1225 Acked Write 2 Sent Done Done false
1082 1226 Acked Read 8 Sent Done Done false
1083 1227 Acked Write 4 Sent Done Done false
1084 1228 Acked Read 5 Sent Done Done false
1085 1229 Acked Write 3 Sent Done Done false
1086 1230 Acked Read 10 Sent Done Done false
1087 1231 Acked Read 2 Sent Done Done false
1088 1232 Acked Write 1 Sent Done Done false
1089 1233 Acked Flush 0 Sent Done Done false
1090 1234 Acked Read 7 Sent Done Done false
1091 1235 Acked Read 7 Sent Done Done false
1092 1236 Acked Read 2 Sent Done Done false
1093 1237 Acked Read 6 Sent Done Done false
1094 1238 Acked Write 10 Sent Done Done false
1095 1239 Acked Flush 0 Sent Done Done false
1096 1240 Acked Flush 0 Sent Done Done false
1097 1241 Acked Read 5 Sent Done Done false
1098 1242 Acked Read 6 Sent Done Done false
1099 1243 Acked Write 8 Sent Done Done false
1100 1244 Acked Flush 0 Sent Done Done false
1101 1245 Acked Write 1 Sent Done Done false
1102 1246 Acked Flush 0 Sent Done Done false
1103 1247 Acked Write 2 Sent Done Done false
1104 1248 Acked Read 5 Sent Done Done false
1105 1249 Acked Read 8 Sent Done Done false
1106 1250 Acked Write 2 Sent Done Done false
1107 1251 Acked Write 4 Sent Done Done false
1108 1252 Acked Read 7 Sent Done Done false
1109 1253 Acked Read 2 Sent Done Done false
1110 1254 Acked Write 5 Sent Done Done false
1111 1255 Acked Read 5 Sent Done Done false
1112 1256 Acked Write 5 Sent Done Done false
1113 1257 Acked Write 3 Sent Done Done false
1114 1258 Acked Read 4 Sent Done Done false
1115 1259 Acked Write 8 Sent Done Done false
1116 1260 Acked Write 8 Sent Done Done false
1117 1261 Acked Flush 0 Sent Done Done false
1118 1262 Acked Read 5 Sent Done Done false
1119 1263 Acked Flush 0 Sent Done Done false
1120 1264 Acked Write 10 Sent Done Done false
1121 1265 Acked Flush 0 Sent Done Done false
1122 1266 Acked Read 4 Sent Done Done false
1123 1267 Acked Write 5 Sent Done Done false
1124 1268 Acked Write 9 Sent Done Done false
1125 1269 Acked Read 6 Sent Done Done false
1126 1270 Acked Read 7 Sent Done Done false
1127 1271 Acked Write 8 Sent Done Done false
1128 1272 Acked Read 10 Sent Done Done false
1129 1273 Acked Write 8 Sent Done Done false
1130 1274 Acked Read 2 Sent Done Done false
1131 1275 Acked Flush 0 Sent Done Done false
1132 1276 Acked Write 10 Sent Done Done false
1133 1277 Acked Write 9 Sent Done Done false
1134 1278 Acked Read 1 Sent Done Done false
1135 1279 Acked Flush 0 Sent Done Done false
1136 1280 Acked Write 10 Sent Done Done false
1137 1281 Acked Read 8 Sent Done Done false
1138 1282 Acked Write 9 Sent Done Done false
1139 1283 Acked Read 2 Sent Done Done false
1140 1284 Acked Write 8 Sent Done Done false
1141 1285 Acked Read 10 Sent Done Done false
1142 1286 Acked Read 2 Sent Done Done false
1143 1287 Acked Read 4 Sent Done Done false
1144 1288 Acked Write 10 Sent Done Done false
1145 1289 Acked Write 10 Sent Done Done false
1146 1290 Acked Read 1 Sent Done Done false
1147 1291 Acked Read 5 Sent Done Done false
1148 1292 Acked Read 10 Sent Done Done false
1149 1293 Acked Read 2 Sent Done Done false
1150 1294 Acked Write 5 Sent Done Done false
1151 1295 Acked Read 9 Sent Done Done false
1152 1296 Acked Write 2 Sent Done Done false
1153 1297 Acked Flush 0 Sent Done Done false
1154 1298 Acked Write 9 Sent Done Done false
1155 1299 Acked Read 8 Sent Done Done false
1156 1300 Acked Read 5 Sent Done Done false
1157 1301 Acked Flush 0 Sent Done Done false
1158 STATES DS:0 DS:1 DS:2 TOTAL
1159 Sent 302 0 0 302
1160 Done 0 302 302 604
1161 Skipped 0 0 0 0
1162 Error 0 0 0 0
1163 Last Flush: None Some(JobId(1301)) Some(JobId(1301))
1164 Downstairs last five completed:
1165 Upstairs last five completed: 1301 1300 1299 1298 1297
1166 CLIENT: Up:0 ds:302 act:2
11672025-01-23T17:37:57.601ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
11682025-01-23T17:37:57.601ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56079) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
11692025-01-23T17:37:57.602ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
11702025-01-23T17:37:57.603ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
11712025-01-23T17:37:57.603ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
11722025-01-23T17:37:57.603ZINFOcrucible: send last flush ID to this DS: None = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
11732025-01-23T17:37:57.603ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
11742025-01-23T17:37:57.603ZINFOcrucible: [0] client re-new 302 jobs since flush None = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
11752025-01-23T17:37:57.603ZINFOcrucible: [0] Replayed 302 jobs since flush: None = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1176 ----------------------------------------------------------------
1177 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1178 Downstairs last five completed: 1301 1300 1299 1298 1297
1179 Upstairs last five completed: 1301 1300 1299 1298 1297
1180 CLIENT: Up:0 ds:0 act:3
1181 Replay: All jobs finished
1182 Main task has work to do, go find it
1183 got dsc Stop(2)
1184 stop 2
1185 Jan 23 17:37:59.865 INFO request completed, latency_us: 337, response_code[8830] Got stop action so:false kr:false
1186 : 204, uri: /stop/cid/2, method: POST, req_id: f066859c-43ad-4a4c-ac00-19e79c4afb13, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
11872025-01-23T17:37:59.866ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
11882025-01-23T17:37:59.866ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
11892025-01-23T17:37:59.866ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1190 Jan 23 17:37:59.866 INFO request completed, latency_us: 173, response_code: 200, uri: /state/cid/2, method: GET, req_id: 35e54c75-f997-4316-b66c-d8caa7857485, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
11912025-01-23T17:37:59.866ZINFOcrucible: client task is exiting = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
1192 [8830] Exited with: ExitStatus(unix_wait_status(9))
1193 [8830][2] reports Exit
1194 Jan 23 17:38:03.868 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/2, method: GET, req_id: abeb31b5-0913-4cf3-8f5c-f5135d39d5e0, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
1195 001/300 Read block 576 len 20480
1196 002/300 Read block 1562 len 12288
1197 003/300 Read block 2186 len 28672
1198 004/300 Read block 1902 len 8192
1199 005/300 Read block 948 len 4096
1200 006/300 Write block 4 len 8192 data: 2 2
1201 007/300 Read block 4125 len 32768
1202 008/300 Read block 3419 len 16384
1203 009/300 Write block 2957 len 24576 data: 2 2 2 3 3 3
1204 010/300 Write block 279 len 4096 data: 2
1205 011/300 Read block 1757 len 40960
1206 012/300 Read block 8 len 16384
1207 013/300 Write block 2080 len 12288 data: 2 2 2
1208 014/300 Write block 4933 len 24576 data: 2 2 2 2 2 2
1209 015/300 Read block 452 len 8192
1210 016/300 Read block 2875 len 16384
1211 017/300 Read block 1626 len 28672
1212 018/300 Write block 3853 len 32768 data: 2 2 2 2 3 3 3 2
1213 019/300 Read block 2550 len 28672
1214 020/300 Read block 1010 len 8192
1215 021/300 Write block 1874 len 32768 data: 2 2 2 2 2 2 2 2
1216 022/300 Read block 4875 len 28672
1217 023/300 Read block 3878 len 12288
1218 024/300 Write block 3597 len 32768 data: 2 2 2 2 2 2 2 2
1219 025/300 Write block 3225 len 8192 data: 2 2
1220 026/300 Write block 4267 len 24576 data: 2 2 2 2 2 2
1221 027/300 Read block 3398 len 32768
1222 028/300 Write block 3667 len 12288 data: 2 2 2
1223 029/300 Read block 2780 len 36864
1224 030/300 Read block 1558 len 8192
1225 031/300 Read block 3663 len 4096
1226 032/300 Write block 618 len 12288 data: 2 3 3
1227 033/300 Read block 1186 len 36864
1228 034/300 Read block 1589 len 32768
1229 035/300 Write block 4267 len 36864 data: 3 3 3 3 3 3 2 2 2
1230 036/300 Read block 3529 len 12288
1231 037/300 Write block 1680 len 36864 data: 2 2 2 2 2 2 2 2 2
1232 038/300 Flush
1233 039/300 Read block 2143 len 8192
1234 040/300 Write block 3857 len 28672 data: 4 4 4 3 2 2 2
1235 041/300 Write block 3874 len 8192 data: 2 2
1236 042/300 Write block 3310 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1237 043/300 Write block 2280 len 16384 data: 2 2 2 2
1238 044/300 Flush
1239 045/300 Read block 2520 len 24576
1240 046/300 Read block 2782 len 4096
1241 047/300 Write block 2150 len 24576 data: 2 2 2 2 2 2
1242 048/300 Read block 1910 len 36864
1243 049/300 Read block 3403 len 4096
1244 050/300 Read block 1416 len 40960
1245 051/300 Read block 563 len 20480
1246 052/300 Write block 2628 len 24576 data: 2 2 2 2 2 2
1247 053/300 Read block 82 len 8192
1248 054/300 Read block 3903 len 8192
1249 055/300 Read block 1479 len 4096
1250 056/300 Write block 2082 len 12288 data: 3 2 2
1251 057/300 Read block 4462 len 20480
1252 058/300 Write block 592 len 16384 data: 2 2 2 3
1253 059/300 Read block 824 len 8192
1254 060/300 Read block 4584 len 24576
1255 061/300 Write block 3613 len 4096 data: 3
1256 062/300 Read block 1304 len 32768
1257 063/300 Write block 1653 len 12288 data: 2 2 2
1258 064/300 Write block 3337 len 16384 data: 2 2 2 2
1259 065/300 Write block 1416 len 20480 data: 2 2 2 3 3
1260 066/300 Read block 3716 len 36864
1261 067/300 Write block 1854 len 36864 data: 2 2 2 2 2 2 2 2 2
1262 068/300 Read block 605 len 24576
1263 069/300 Write block 1969 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1264 070/300 Read block 2864 len 16384
1265 071/300 Write block 2372 len 24576 data: 2 2 2 2 2 2
1266 072/300 Write block 1618 len 12288 data: 2 2 2
1267 073/300 Read block 3904 len 16384
1268 074/300 Read block 203 len 12288
1269 075/300 Read block 132 len 16384
1270 076/300 Write block 3992 len 36864 data: 2 2 2 2 2 2 2 2 2
1271 077/300 Read block 376 len 12288
1272 078/300 Write block 2894 len 8192 data: 3 3
1273 079/300 Read block 960 len 12288
1274 080/300 Read block 2995 len 36864
1275 081/300 Read block 2420 len 32768
1276 082/300 Read block 3773 len 24576
1277 083/300 Read block 1895 len 32768
1278 084/300 Flush
1279 085/300 Read block 4711 len 8192
1280 086/300 Read block 3598 len 4096
1281 087/300 Write block 2707 len 40960 data: 3 3 3 3 3 3 3 3 2 2
1282 088/300 Write block 2604 len 20480 data: 2 2 2 2 2
1283 089/300 Read block 3425 len 24576
1284 090/300 Read block 3719 len 20480
1285 091/300 Read block 947 len 16384
1286 092/300 Read block 2214 len 4096
1287 093/300 Read block 1246 len 4096
1288 094/300 Read block 2205 len 36864
1289 095/300 Read block 4744 len 16384
1290 096/300 Flush
1291 097/300 Write block 1156 len 36864 data: 2 2 2 2 2 2 2 2 2
1292 098/300 Write block 1161 len 28672 data: 3 3 3 3 2 2 2
1293 099/300 Write block 4812 len 28672 data: 2 2 2 2 2 2 2
1294 100/300 Read block 894 len 8192
1295 101/300 Write block 113 len 36864 data: 2 2 2 2 2 2 2 2 2
1296 102/300 Read block 3053 len 40960
1297 103/300 Read block 1129 len 36864
1298 104/300 Write block 2112 len 36864 data: 2 2 2 2 2 2 2 2 2
1299 105/300 Read block 4413 len 8192
1300 106/300 Write block 1882 len 32768 data: 2 2 2 2 2 2 2 2
1301 107/300 Write block 3931 len 32768 data: 2 2 2 2 2 2 2 2
1302 108/300 Read block 4772 len 4096
1303 109/300 Read block 2726 len 8192
1304 110/300 Flush
1305 111/300 Flush
1306 112/300 Write block 1110 len 24576 data: 2 2 2 2 2 2
1307 113/300 Write block 984 len 24576 data: 2 2 2 2 2 2
1308 114/300 Read block 2634 len 24576
1309 115/300 Write block 3204 len 24576 data: 2 2 2 2 2 2
1310 116/300 Write block 3609 len 8192 data: 2 2
1311 117/300 Write block 4708 len 4096 data: 2
1312 118/300 Write block 110 len 40960 data: 2 2 2 3 3 3 3 3 3 3
1313 119/300 Write block 3307 len 36864 data: 2 2 2 3 3 3 3 3 3
1314 120/300 Read block 1090 len 40960
1315 121/300 Read block 1951 len 8192
1316 122/300 Write block 3655 len 12288 data: 2 2 2
1317 123/300 Write block 1407 len 20480 data: 2 2 2 2 2
1318 124/300 Write block 2534 len 32768 data: 3 3 3 2 2 2 2 2
1319 125/300 Write block 1368 len 16384 data: 3 3 3 3
1320 126/300 Read block 3126 len 16384
1321 127/300 Read block 4302 len 4096
1322 128/300 Read block 1827 len 12288
1323 129/300 Read block 4377 len 4096
1324 130/300 Read block 2904 len 4096
1325 131/300 Read block 1441 len 24576
1326 132/300 Write block 4986 len 12288 data: 2 2 2
1327 133/300 Read block 509 len 20480
1328 134/300 Write block 2554 len 20480 data: 3 3 3 3 3
1329 135/300 Write block 812 len 20480 data: 2 2 2 2 2
1330 136/300 Write block 1330 len 12288 data: 2 2 2
1331 137/300 Write block 3540 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1332 138/300 Read block 4772 len 16384
1333 139/300 Read block 4339 len 4096
1334 140/300 Read block 1742 len 20480
1335 141/300 Read block 4990 len 8192
1336 142/300 Read block 4804 len 24576
1337 143/300 Write block 3549 len 24576 data: 3 2 2 2 2 2
1338 144/300 Read block 1609 len 20480
1339 145/300 Read block 1102 len 40960
1340 146/300 Read block 1060 len 28672
1341 147/300 Read block 2780 len 32768
1342 148/300 Write block 2240 len 24576 data: 2 2 2 2 2 2
1343 149/300 Read block 3722 len 8192
1344 150/300 Write block 13 len 16384 data: 2 2 2 2
1345 151/300 Write block 2393 len 32768 data: 2 2 2 2 2 2 2 2
1346 152/300 Flush
1347 153/300 Read block 88 len 24576
1348 154/300 Read block 1023 len 20480
1349 155/300 Read block 1580 len 20480
1350 156/300 Read block 998 len 28672
1351 157/300 Read block 1809 len 16384
1352 158/300 Read block 2931 len 12288
1353 159/300 Read block 2931 len 36864
1354 160/300 Write block 2614 len 16384 data: 2 2 2 2
1355 161/300 Write block 4046 len 16384 data: 2 2 2 2
1356 162/300 Read block 3935 len 32768
1357 163/300 Write block 3961 len 24576 data: 3 3 3 3 3 2
1358 164/300 Read block 1012 len 12288
1359 165/300 Flush
1360 166/300 Read block 2367 len 20480
1361 167/300 Read block 2491 len 24576
1362 168/300 Read block 839 len 32768
1363 169/300 Read block 4249 len 40960
1364 170/300 Write block 3869 len 24576 data: 2 2 2 2 2 3
1365 171/300 Flush
1366 172/300 Flush
1367 173/300 Read block 4499 len 4096
1368 174/300 Flush
1369 175/300 Flush
1370 176/300 Flush
1371 177/300 Read block 4091 len 8192
1372 178/300 Read block 3266 len 4096
1373 179/300 Flush
1374 180/300 Write block 3873 len 28672 data: 3 4 3 2 2 2 2
1375 181/300 Read block 713 len 20480
1376 182/300 Flush
1377 183/300 Read block 16 len 8192
1378 184/300 Read block 3299 len 32768
1379 185/300 Write block 126 len 28672 data: 5 5 5 3 3 3 2
1380 186/300 Write block 2911 len 20480 data: 2 2 2 2 2
1381 187/300 Read block 4963 len 12288
1382 188/300 Write block 3069 len 4096 data: 2
1383 189/300 Write block 210 len 16384 data: 2 2 2 2
1384 190/300 Write block 3414 len 28672 data: 2 2 2 2 2 2 2
1385 191/300 Read block 367 len 40960
1386 192/300 Read block 3113 len 40960
1387 193/300 Write block 53 len 28672 data: 2 2 2 2 2 2 2
1388 194/300 Write block 1200 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1389 195/300 Write block 82 len 28672 data: 2 2 2 2 2 2 2
1390 196/300 Read block 271 len 36864
1391 197/300 Write block 2857 len 32768 data: 2 2 2 2 2 2 2 2
1392 198/300 Read block 233 len 28672
1393 199/300 Read block 1971 len 12288
1394 200/300 Read block 1528 len 24576
1395 201/300 Write block 653 len 24576 data: 2 2 2 2 2 2
1396 202/300 Write block 4534 len 16384 data: 2 2 2 2
1397 203/300 Read block 3016 len 40960
1398 204/300 Write block 2603 len 8192 data: 2 3
1399 205/300 Read block 2083 len 36864
1400 206/300 Read block 1993 len 8192
1401 207/300 Flush
1402 208/300 Read block 2433 len 16384
1403 209/300 Read block 3608 len 40960
1404 210/300 Write block 4455 len 8192 data: 2 2
1405 211/300 Write block 2228 len 36864 data: 2 2 2 2 2 2 3 2 2
1406 212/300 Read block 2329 len 28672
1407 213/300 Write block 541 len 12288 data: 3 2 2
1408 214/300 Write block 218 len 4096 data: 2
1409 215/300 Read block 1315 len 40960
1410 216/300 Flush
1411 217/300 Read block 1455 len 24576
1412 218/300 Read block 812 len 40960
1413 219/300 Write block 657 len 16384 data: 3 3 2 2
1414 220/300 Write block 2525 len 32768 data: 2 2 2 3 3 3 3 3
1415 221/300 Write block 3264 len 4096 data: 2
1416 222/300 Write block 1825 len 24576 data: 2 2 2 2 2 2
1417 223/300 Read block 4899 len 36864
1418 224/300 Write block 3590 len 28672 data: 2 2 2 2 2 2 2
1419 225/300 Read block 256 len 28672
1420 226/300 Read block 371 len 16384
1421 227/300 Read block 1824 len 12288
1422 228/300 Read block 779 len 8192
1423 229/300 Read block 444 len 12288
1424 230/300 Read block 24 len 12288
1425 231/300 Write block 570 len 12288 data: 2 3 3
1426 232/300 Flush
1427 233/300 Read block 1401 len 8192
1428 234/300 Write block 1389 len 8192 data: 2 3
1429 235/300 Read block 4886 len 20480
1430 236/300 Read block 2339 len 28672
1431 237/300 Read block 4304 len 24576
1432 238/300 Flush
1433 239/300 Write block 1796 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1434 240/300 Read block 4485 len 20480
1435 241/300 Read block 3432 len 24576
1436 242/300 Write block 1971 len 8192 data: 3 3
1437 243/300 Write block 4233 len 8192 data: 2 2
1438 244/300 Read block 4606 len 40960
1439 245/300 Read block 2025 len 16384
1440 246/300 Write block 4816 len 24576 data: 3 3 3 2 2 2
1441 247/300 Write block 4931 len 24576 data: 2 2 3 3 3 3
1442 248/300 Flush
1443 249/300 Flush
1444 250/300 Write block 2684 len 20480 data: 2 2 2 2 2
1445 251/300 Write block 1473 len 24576 data: 3 3 2 2 3 3
1446 252/300 Write block 2576 len 4096 data: 2
1447 253/300 Read block 786 len 28672
1448 254/300 Write block 2661 len 8192 data: 2 2
1449 255/300 Write block 2060 len 20480 data: 2 2 2 2 2
1450 256/300 Write block 2491 len 32768 data: 2 2 2 2 2 2 2 2
1451 257/300 Read block 1379 len 40960
1452 258/300 Flush
1453 259/300 Write block 4564 len 28672 data: 2 2 2 2 2 2 2
1454 260/300 Write block 3604 len 36864 data: 3 2 2 2 2 3 3 3 3
1455 261/300 Flush
1456 262/300 Read block 4077 len 32768
1457 263/300 Read block 4304 len 20480
1458 264/300 Write block 2126 len 28672 data: 2 2 2 2 2 2 2
1459 265/300 Flush
1460 266/300 Read block 3314 len 4096
1461 267/300 Flush
1462 268/300 Read block 936 len 32768
1463 269/300 Flush
1464 270/300 Write block 2068 len 24576 data: 3 3 2 2 2 2
1465 271/300 Read block 4342 len 12288
1466 272/300 Read block 1257 len 40960
1467 273/300 Write block 1533 len 20480 data: 2 2 2 2 2
1468 274/300 Write block 4533 len 12288 data: 2 3 3
1469 275/300 Read block 3712 len 12288
1470 276/300 Write block 203 len 16384 data: 2 2 2 2
1471 277/300 Write block 4990 len 36864 data: 2 2 2 2 2 2 2 2 2
1472 278/300 Read block 10 len 24576
1473 279/300 Write block 3324 len 12288 data: 2 2 2
1474 280/300 Read block 1456 len 32768
1475 281/300 Read block 1979 len 4096
1476 282/300 Write block 1043 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1477 283/300 Write block 1292 len 8192 data: 3 3
1478 284/300 Write block 4057 len 32768 data: 2 2 2 2 2 2 2 2
1479 285/300 Write block 4949 len 20480 data: 2 2 2 2 2
1480 286/300 Read block 3085 len 8192
1481 287/300 Read block 308 len 4096
1482 288/300 Flush
1483 289/300 Read block 4677 len 24576
1484 290/300 Write block 4334 len 16384 data: 2 2 2 2
1485 291/300 Write block 3198 len 12288 data: 2 2 2
1486 292/300 Read block 4252 len 16384
1487 293/300 Write block 1257 len 28672 data: 2 2 2 2 2 2 2
1488 294/300 Read block 2083 len 40960
1489 295/300 Write block 2220 len 20480 data: 2 2 2 2 2
1490 296/300 Read block 4192 len 4096
1491 297/300 Read block 3391 len 40960
1492 298/300 Write block 2474 len 24576 data: 2 2 2 2 2 2
1493 299/300 Flush
1494 300/300 Read block 553 len 20480
1495 301/300 Write block 4496 len 20480 data: 2 2 2 2 2
1496 Main task has work to do, go find it
1497 got dsc Start(2)
1498 start 2
1499 Jan 23 17:38:04.107 INFO request completed, latency_us: 225, response_code: 204, uri: /start/cid/2, method: POST, req_id: eec8eefd-81a3-4fa6-96d8-ede087a21ca4, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
1500 Replay: started 2, returned:Ok(())
1501 ----------------------------------------------------------------
1502 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1503 States: Active Active Offline
1504 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1505 1302 Acked Read 5 Done Done Sent false
1506 1303 Acked Read 3 Done Done Sent false
1507 1304 Acked Read 7 Done Done Sent false
1508 1305 Acked Read 2 Done Done Sent false
1509 1306 Acked Read 1 Done Done Sent false
1510 1307 Acked Write 2 Done Done Sent false
1511 1308 Acked Read 8 Done Done Sent false
1512 1309 Acked Read 4 Done Done Sent false
1513 1310 Acked Write 6 Done Done Sent false
1514 1311 Acked Write 1 Done Done Sent false
1515 1312 Acked Read 10 Done Done Sent false
1516 1313 Acked Read 4 Done Done Sent false
1517 1314 Acked Write 3 Done Done Sent false
1518 1315 Acked Write 6 Done Done Sent false
1519 1316 Acked Read 2 Done Done Sent false
1520 1317 Acked Read 4 Done Done Sent false
1521 1318 Acked Read 7 Done Done Sent false
1522 1319 Acked Write 8 Done Done Sent false
1523 1320 Acked Read 7 Done Done Sent false
1524 1321 Acked Read 2 Done Done Sent false
1525 1322 Acked Write 8 Done Done Sent false
1526 1323 Acked Read 7 Done Done Sent false
1527 1324 Acked Read 3 Done Done Sent false
1528 1325 Acked Write 8 Done Done Sent false
1529 1326 Acked Write 2 Done Done Sent false
1530 1327 Acked Write 6 Done Done Sent false
1531 1328 Acked Read 8 Done Done Sent false
1532 1329 Acked Write 3 Done Done Sent false
1533 1330 Acked Read 9 Done Done Sent false
1534 1331 Acked Read 2 Done Done Sent false
1535 1332 Acked Read 1 Done Done Sent false
1536 1333 Acked Write 3 Done Done Sent false
1537 1334 Acked Read 9 Done Done Sent false
1538 1335 Acked Read 8 Done Done Sent false
1539 1336 Acked Write 9 Done Done Sent false
1540 1337 Acked Read 3 Done Done Sent false
1541 1338 Acked Write 9 Done Done Sent false
1542 1339 Acked Flush 0 Done Done Sent false
1543 1340 Acked Read 2 Done Done Sent false
1544 1341 Acked Write 7 Done Done Sent false
1545 1342 Acked Write 2 Done Done Sent false
1546 1343 Acked Write 10 Done Done Sent false
1547 1344 Acked Write 4 Done Done Sent false
1548 1345 Acked Flush 0 Done Done Sent false
1549 1346 Acked Read 6 Done Done Sent false
1550 1347 Acked Read 1 Done Done Sent false
1551 1348 Acked Write 6 Done Done Sent false
1552 1349 Acked Read 9 Done Done Sent false
1553 1350 Acked Read 1 Done Done Sent false
1554 1351 Acked Read 10 Done Done Sent false
1555 1352 Acked Read 5 Done Done Sent false
1556 1353 Acked Write 6 Done Done Sent false
1557 1354 Acked Read 2 Done Done Sent false
1558 1355 Acked Read 2 Done Done Sent false
1559 1356 Acked Read 1 Done Done Sent false
1560 1357 Acked Write 3 Done Done Sent false
1561 1358 Acked Read 5 Done Done Sent false
1562 1359 Acked Write 4 Done Done Sent false
1563 1360 Acked Read 2 Done Done Sent false
1564 1361 Acked Read 6 Done Done Sent false
1565 1362 Acked Write 1 Done Done Sent false
1566 1363 Acked Read 8 Done Done Sent false
1567 1364 Acked Write 3 Done Done Sent false
1568 1365 Acked Write 4 Done Done Sent false
1569 1366 Acked Write 5 Done Done Sent false
1570 1367 Acked Read 9 Done Done Sent false
1571 1368 Acked Write 9 Done Done Sent false
1572 1369 Acked Read 6 Done Done Sent false
1573 1370 Acked Write 10 Done Done Sent false
1574 1371 Acked Read 4 Done Done Sent false
1575 1372 Acked Write 6 Done Done Sent false
1576 1373 Acked Write 3 Done Done Sent false
1577 1374 Acked Read 4 Done Done Sent false
1578 1375 Acked Read 3 Done Done Sent false
1579 1376 Acked Read 4 Done Done Sent false
1580 1377 Acked Write 9 Done Done Sent false
1581 1378 Acked Read 3 Done Done Sent false
1582 1379 Acked Write 2 Done Done Sent false
1583 1380 Acked Read 3 Done Done Sent false
1584 1381 Acked Read 9 Done Done Sent false
1585 1382 Acked Read 8 Done Done Sent false
1586 1383 Acked Read 6 Done Done Sent false
1587 1384 Acked Read 8 Done Done Sent false
1588 1385 Acked Flush 0 Done Done Sent false
1589 1386 Acked Read 2 Done Done Sent false
1590 1387 Acked Read 1 Done Done Sent false
1591 1388 Acked Write 10 Done Done Sent false
1592 1389 Acked Write 5 Done Done Sent false
1593 1390 Acked Read 6 Done Done Sent false
1594 1391 Acked Read 5 Done Done Sent false
1595 1392 Acked Read 4 Done Done Sent false
1596 1393 Acked Read 1 Done Done Sent false
1597 1394 Acked Read 1 Done Done Sent false
1598 1395 Acked Read 9 Done Done Sent false
1599 1396 Acked Read 4 Done Done Sent false
1600 1397 Acked Flush 0 Done Done Sent false
1601 1398 Acked Write 9 Done Done Sent false
1602 1399 Acked Write 7 Done Done Sent false
1603 1400 Acked Write 7 Done Done Sent false
1604 1401 Acked Read 2 Done Done Sent false
1605 1402 Acked Write 9 Done Done Sent false
1606 1403 Acked Read 10 Done Done Sent false
1607 1404 Acked Read 9 Done Done Sent false
1608 1405 Acked Write 9 Done Done Sent false
1609 1406 Acked Read 2 Done Done Sent false
1610 1407 Acked Write 8 Done Done Sent false
1611 1408 Acked Write 8 Done Done Sent false
1612 1409 Acked Read 1 Done Done Sent false
1613 1410 Acked Read 2 Done Done Sent false
1614 1411 Acked Flush 0 Done Done Sent false
1615 1412 Acked Flush 0 Done Done Sent false
1616 1413 Acked Write 6 Done Done Sent false
1617 1414 Acked Write 6 Done Done Sent false
1618 1415 Acked Read 6 Done Done Sent false
1619 1416 Acked Write 6 Done Done Sent false
1620 1417 Acked Write 2 Done Done Sent false
1621 1418 Acked Write 1 Done Done Sent false
1622 1419 Acked Write 10 Done Done Sent false
1623 1420 Acked Write 9 Done Done Sent false
1624 1421 Acked Read 10 Done Done Sent false
1625 1422 Acked Read 2 Done Done Sent false
1626 1423 Acked Write 3 Done Done Sent false
1627 1424 Acked Write 5 Done Done Sent false
1628 1425 Acked Write 8 Done Done Sent false
1629 1426 Acked Write 4 Done Done Sent false
1630 1427 Acked Read 4 Done Done Sent false
1631 1428 Acked Read 1 Done Done Sent false
1632 1429 Acked Read 3 Done Done Sent false
1633 1430 Acked Read 1 Done Done Sent false
1634 1431 Acked Read 1 Done Done Sent false
1635 1432 Acked Read 6 Done Done Sent false
1636 1433 Acked Write 3 Done Done Sent false
1637 1434 Acked Read 5 Done Done Sent false
1638 1435 Acked Write 5 Done Done Sent false
1639 1436 Acked Write 5 Done Done Sent false
1640 1437 Acked Write 3 Done Done Sent false
1641 1438 Acked Write 10 Done Done Sent false
1642 1439 Acked Read 4 Done Done Sent false
1643 1440 Acked Read 1 Done Done Sent false
1644 1441 Acked Read 5 Done Done Sent false
1645 1442 Acked Read 2 Done Done Sent false
1646 1443 Acked Read 6 Done Done Sent false
1647 1444 Acked Write 6 Done Done Sent false
1648 1445 Acked Read 5 Done Done Sent false
1649 1446 Acked Read 10 Done Done Sent false
1650 1447 Acked Read 7 Done Done Sent false
1651 1448 Acked Read 8 Done Done Sent false
1652 1449 Acked Write 6 Done Done Sent false
1653 1450 Acked Read 2 Done Done Sent false
1654 1451 Acked Write 4 Done Done Sent false
1655 1452 Acked Write 8 Done Done Sent false
1656 1453 Acked Flush 0 Done Done Sent false
1657 1454 Acked Read 6 Done Done Sent false
1658 1455 Acked Read 5 Done Done Sent false
1659 1456 Acked Read 5 Done Done Sent false
1660 1457 Acked Read 7 Done Done Sent false
1661 1458 Acked Read 4 Done Done Sent false
1662 1459 Acked Read 3 Done Done Sent false
1663 1460 Acked Read 9 Done Done Sent false
1664 1461 Acked Write 4 Done Done Sent false
1665 1462 Acked Write 4 Done Done Sent false
1666 1463 Acked Read 8 Done Done Sent false
1667 1464 Acked Write 6 Done Done Sent false
1668 1465 Acked Read 3 Done Done Sent false
1669 1466 Acked Flush 0 Done Done Sent false
1670 1467 Acked Read 5 Done Done Sent false
1671 1468 Acked Read 6 Done Done Sent false
1672 1469 Acked Read 8 Done Done Sent false
1673 1470 Acked Read 10 Done Done Sent false
1674 1471 Acked Write 6 Done Done Sent false
1675 1472 Acked Flush 0 Done Done Sent false
1676 1473 Acked Flush 0 Done Done Sent false
1677 1474 Acked Read 1 Done Done Sent false
1678 1475 Acked Flush 0 Done Done Sent false
1679 1476 Acked Flush 0 Done Done Sent false
1680 1477 Acked Flush 0 Done Done Sent false
1681 1478 Acked Read 2 Done Done Sent false
1682 1479 Acked Read 1 Done Done Sent false
1683 1480 Acked Flush 0 Done Done Sent false
1684 1481 Acked Write 7 Done Done Sent false
1685 1482 Acked Read 5 Done Done Sent false
1686 1483 Acked Flush 0 Done Done Sent false
1687 1484 Acked Read 2 Done Done Sent false
1688 1485 Acked Read 8 Done Done Sent false
1689 1486 Acked Write 7 Done Done Sent false
1690 1487 Acked Write 5 Done Done Sent false
1691 1488 Acked Read 3 Done Done Sent false
1692 1489 Acked Write 1 Done Done Sent false
1693 1490 Acked Write 4 Done Done Sent false
1694 1491 Acked Write 7 Done Done Sent false
1695 1492 Acked Read 10 Done Done Sent false
1696 1493 Acked Read 10 Done Done Sent false
1697 1494 Acked Write 7 Done Done Sent false
1698 1495 Acked Write 10 Done Done Sent false
1699 1496 Acked Write 7 Done Done Sent false
1700 1497 Acked Read 9 Done Done Sent false
1701 1498 Acked Write 8 Done Done Sent false
1702 1499 Acked Read 7 Done Done Sent false
1703 1500 Acked Read 3 Done Done Sent false
1704 1501 Acked Read 6 Done Done Sent false
1705 1502 Acked Write 6 Done Done Sent false
1706 1503 Acked Write 4 Done Done Sent false
1707 1504 Acked Read 10 Done Done Sent false
1708 1505 Acked Write 2 Done Done Sent false
1709 1506 Acked Read 9 Done Done Sent false
1710 1507 Acked Read 2 Done Done Sent false
1711 1508 Acked Flush 0 Done Done Sent false
1712 1509 Acked Read 4 Done Done Sent false
1713 1510 Acked Read 10 Done Done Sent false
1714 1511 Acked Write 2 Done Done Sent false
1715 1512 Acked Write 9 Done Done Sent false
1716 1513 Acked Read 7 Done Done Sent false
1717 1514 Acked Write 3 Done Done Sent false
1718 1515 Acked Write 1 Done Done Sent false
1719 1516 Acked Read 10 Done Done Sent false
1720 1517 Acked Flush 0 Done Done Sent false
1721 1518 Acked Read 6 Done Done Sent false
1722 1519 Acked Read 10 Done Done Sent false
1723 1520 Acked Write 4 Done Done Sent false
1724 1521 Acked Write 8 Done Done Sent false
1725 1522 Acked Write 1 Done Done Sent false
1726 1523 Acked Write 6 Done Done Sent false
1727 1524 Acked Read 9 Done Done Sent false
1728 1525 Acked Write 7 Done Done Sent false
1729 1526 Acked Read 7 Done Done Sent false
1730 1527 Acked Read 4 Done Done Sent false
1731 1528 Acked Read 3 Done Done Sent false
1732 1529 Acked Read 2 Done Done Sent false
1733 1530 Acked Read 3 Done Done Sent false
1734 1531 Acked Read 3 Done Done Sent false
1735 1532 Acked Write 3 Done Done Sent false
1736 1533 Acked Flush 0 Done Done Sent false
1737 1534 Acked Read 2 Done Done Sent false
1738 1535 Acked Write 2 Done Done Sent false
1739 1536 Acked Read 5 Done Done Sent false
1740 1537 Acked Read 7 Done Done Sent false
1741 1538 Acked Read 6 Done Done Sent false
1742 1539 Acked Flush 0 Done Done Sent false
1743 1540 Acked Write 10 Done Done Sent false
1744 1541 Acked Read 5 Done Done Sent false
1745 1542 Acked Read 6 Done Done Sent false
1746 1543 Acked Write 2 Done Done Sent false
1747 1544 Acked Write 2 Done Done Sent false
1748 1545 Acked Read 10 Done Done Sent false
1749 1546 Acked Read 4 Done Done Sent false
1750 1547 Acked Write 6 Done Done Sent false
1751 1548 Acked Write 6 Done Done Sent false
1752 1549 Acked Flush 0 Done Done Sent false
1753 1550 Acked Flush 0 Done Done Sent false
1754 1551 Acked Write 5 Done Done Sent false
1755 1552 Acked Write 6 Done Done Sent false
1756 1553 Acked Write 1 Done Done Sent false
1757 1554 Acked Read 7 Done Done Sent false
1758 1555 Acked Write 2 Done Done Sent false
1759 1556 Acked Write 5 Done Done Sent false
1760 1557 Acked Write 8 Done Done Sent false
1761 1558 Acked Read 10 Done Done Sent false
1762 1559 Acked Flush 0 Done Done Sent false
1763 1560 Acked Write 7 Done Done Sent false
1764 1561 Acked Write 9 Done Done Sent false
1765 1562 Acked Flush 0 Done Done Sent false
1766 1563 Acked Read 8 Done Done Sent false
1767 1564 Acked Read 5 Done Done Sent false
1768 1565 Acked Write 7 Done Done Sent false
1769 1566 Acked Flush 0 Done Done Sent false
1770 1567 Acked Read 1 Done Done Sent false
1771 1568 Acked Flush 0 Done Done Sent false
1772 1569 Acked Read 8 Done Done Sent false
1773 1570 Acked Flush 0 Done Done Sent false
1774 1571 Acked Write 6 Done Done Sent false
1775 1572 Acked Read 3 Done Done Sent false
1776 1573 Acked Read 10 Done Done Sent false
1777 1574 Acked Write 5 Done Done Sent false
1778 1575 Acked Write 3 Done Done Sent false
1779 1576 Acked Read 3 Done Done Sent false
1780 1577 Acked Write 4 Done Done Sent false
1781 1578 Acked Write 9 Done Done Sent false
1782 1579 Acked Read 6 Done Done Sent false
1783 1580 Acked Write 3 Done Done Sent false
1784 1581 Acked Read 8 Done Done Sent false
1785 1582 Acked Read 1 Done Done Sent false
1786 1583 Acked Write 10 Done Done Sent false
1787 1584 Acked Write 2 Done Done Sent false
1788 1585 Acked Write 8 Done Done Sent false
1789 1586 Acked Write 5 Done Done Sent false
1790 1587 Acked Read 2 Done Done Sent false
1791 1588 Acked Read 1 Done Done Sent false
1792 1589 Acked Flush 0 Done Done Sent false
1793 1590 Acked Read 6 Done Done Sent false
1794 1591 Acked Write 4 Done Done Sent false
1795 1592 Acked Write 3 Done Done Sent false
1796 1593 Acked Read 4 Done Done Sent false
1797 1594 Acked Write 7 Done Done Sent false
1798 1595 Acked Read 10 Done Done Sent false
1799 1596 Acked Write 5 Done Done Sent false
1800 1597 Acked Read 1 Done Done Sent false
1801 1598 Acked Read 10 Done Done Sent false
1802 1599 Acked Write 6 Done Done Sent false
1803 1600 Acked Flush 0 Done Done Sent false
1804 1601 Acked Read 5 Done Done Sent false
1805 1602 Acked Write 5 Done Done Sent false
1806 STATES DS:0 DS:1 DS:2 TOTAL
1807 Sent 0 0 301 301
1808 Done 301 301 0 602
1809 Skipped 0 0 0 0
1810 Error 0 0 0 0
1811 Last Flush: Some(JobId(1600)) Some(JobId(1600)) Some(JobId(1301))
1812 Downstairs last five completed: 1301 1300 1299 1298 1297
1813 Upstairs last five completed: 1602 1601 1600 1599 1598
1814 CLIENT: Up:0 ds:301 act:2
1815 [8830] got start action
1816 [8830] I am going to restart
1817 Starting downstairs at port 8830
1818 Make output file at "/tmp/dsc/downstairs-8830.txt"
1819 [8830][2] reports Starting
1820 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1207)
1821 [8830][2] reports Running
1822 ----------------------------------------------------------------
1823 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
1824 States: Active Active Offline
1825 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1826 1302 Acked Read 5 Done Done Sent false
1827 1303 Acked Read 3 Done Done Sent false
1828 1304 Acked Read 7 Done Done Sent false
1829 1305 Acked Read 2 Done Done Sent false
1830 1306 Acked Read 1 Done Done Sent false
1831 1307 Acked Write 2 Done Done Sent false
1832 1308 Acked Read 8 Done Done Sent false
1833 1309 Acked Read 4 Done Done Sent false
1834 1310 Acked Write 6 Done Done Sent false
1835 1311 Acked Write 1 Done Done Sent false
1836 1312 Acked Read 10 Done Done Sent false
1837 1313 Acked Read 4 Done Done Sent false
1838 1314 Acked Write 3 Done Done Sent false
1839 1315 Acked Write 6 Done Done Sent false
1840 1316 Acked Read 2 Done Done Sent false
1841 1317 Acked Read 4 Done Done Sent false
1842 1318 Acked Read 7 Done Done Sent false
1843 1319 Acked Write 8 Done Done Sent false
1844 1320 Acked Read 7 Done Done Sent false
1845 1321 Acked Read 2 Done Done Sent false
1846 1322 Acked Write 8 Done Done Sent false
1847 1323 Acked Read 7 Done Done Sent false
1848 1324 Acked Read 3 Done Done Sent false
1849 1325 Acked Write 8 Done Done Sent false
1850 1326 Acked Write 2 Done Done Sent false
1851 1327 Acked Write 6 Done Done Sent false
1852 1328 Acked Read 8 Done Done Sent false
1853 1329 Acked Write 3 Done Done Sent false
1854 1330 Acked Read 9 Done Done Sent false
1855 1331 Acked Read 2 Done Done Sent false
1856 1332 Acked Read 1 Done Done Sent false
1857 1333 Acked Write 3 Done Done Sent false
1858 1334 Acked Read 9 Done Done Sent false
1859 1335 Acked Read 8 Done Done Sent false
1860 1336 Acked Write 9 Done Done Sent false
1861 1337 Acked Read 3 Done Done Sent false
1862 1338 Acked Write 9 Done Done Sent false
1863 1339 Acked Flush 0 Done Done Sent false
1864 1340 Acked Read 2 Done Done Sent false
1865 1341 Acked Write 7 Done Done Sent false
1866 1342 Acked Write 2 Done Done Sent false
1867 1343 Acked Write 10 Done Done Sent false
1868 1344 Acked Write 4 Done Done Sent false
1869 1345 Acked Flush 0 Done Done Sent false
1870 1346 Acked Read 6 Done Done Sent false
1871 1347 Acked Read 1 Done Done Sent false
1872 1348 Acked Write 6 Done Done Sent false
1873 1349 Acked Read 9 Done Done Sent false
1874 1350 Acked Read 1 Done Done Sent false
1875 1351 Acked Read 10 Done Done Sent false
1876 1352 Acked Read 5 Done Done Sent false
1877 1353 Acked Write 6 Done Done Sent false
1878 1354 Acked Read 2 Done Done Sent false
1879 1355 Acked Read 2 Done Done Sent false
1880 1356 Acked Read 1 Done Done Sent false
1881 1357 Acked Write 3 Done Done Sent false
1882 1358 Acked Read 5 Done Done Sent false
1883 1359 Acked Write 4 Done Done Sent false
1884 1360 Acked Read 2 Done Done Sent false
1885 1361 Acked Read 6 Done Done Sent false
1886 1362 Acked Write 1 Done Done Sent false
1887 1363 Acked Read 8 Done Done Sent false
1888 1364 Acked Write 3 Done Done Sent false
1889 1365 Acked Write 4 Done Done Sent false
1890 1366 Acked Write 5 Done Done Sent false
1891 1367 Acked Read 9 Done Done Sent false
1892 1368 Acked Write 9 Done Done Sent false
1893 1369 Acked Read 6 Done Done Sent false
1894 1370 Acked Write 10 Done Done Sent false
1895 1371 Acked Read 4 Done Done Sent false
1896 1372 Acked Write 6 Done Done Sent false
1897 1373 Acked Write 3 Done Done Sent false
1898 1374 Acked Read 4 Done Done Sent false
1899 1375 Acked Read 3 Done Done Sent false
1900 1376 Acked Read 4 Done Done Sent false
1901 1377 Acked Write 9 Done Done Sent false
1902 1378 Acked Read 3 Done Done Sent false
1903 1379 Acked Write 2 Done Done Sent false
1904 1380 Acked Read 3 Done Done Sent false
1905 1381 Acked Read 9 Done Done Sent false
1906 1382 Acked Read 8 Done Done Sent false
1907 1383 Acked Read 6 Done Done Sent false
1908 1384 Acked Read 8 Done Done Sent false
1909 1385 Acked Flush 0 Done Done Sent false
1910 1386 Acked Read 2 Done Done Sent false
1911 1387 Acked Read 1 Done Done Sent false
1912 1388 Acked Write 10 Done Done Sent false
1913 1389 Acked Write 5 Done Done Sent false
1914 1390 Acked Read 6 Done Done Sent false
1915 1391 Acked Read 5 Done Done Sent false
1916 1392 Acked Read 4 Done Done Sent false
1917 1393 Acked Read 1 Done Done Sent false
1918 1394 Acked Read 1 Done Done Sent false
1919 1395 Acked Read 9 Done Done Sent false
1920 1396 Acked Read 4 Done Done Sent false
1921 1397 Acked Flush 0 Done Done Sent false
1922 1398 Acked Write 9 Done Done Sent false
1923 1399 Acked Write 7 Done Done Sent false
1924 1400 Acked Write 7 Done Done Sent false
1925 1401 Acked Read 2 Done Done Sent false
1926 1402 Acked Write 9 Done Done Sent false
1927 1403 Acked Read 10 Done Done Sent false
1928 1404 Acked Read 9 Done Done Sent false
1929 1405 Acked Write 9 Done Done Sent false
1930 1406 Acked Read 2 Done Done Sent false
1931 1407 Acked Write 8 Done Done Sent false
1932 1408 Acked Write 8 Done Done Sent false
1933 1409 Acked Read 1 Done Done Sent false
1934 1410 Acked Read 2 Done Done Sent false
1935 1411 Acked Flush 0 Done Done Sent false
1936 1412 Acked Flush 0 Done Done Sent false
1937 1413 Acked Write 6 Done Done Sent false
1938 1414 Acked Write 6 Done Done Sent false
1939 1415 Acked Read 6 Done Done Sent false
1940 1416 Acked Write 6 Done Done Sent false
1941 1417 Acked Write 2 Done Done Sent false
1942 1418 Acked Write 1 Done Done Sent false
1943 1419 Acked Write 10 Done Done Sent false
1944 1420 Acked Write 9 Done Done Sent false
1945 1421 Acked Read 10 Done Done Sent false
1946 1422 Acked Read 2 Done Done Sent false
1947 1423 Acked Write 3 Done Done Sent false
1948 1424 Acked Write 5 Done Done Sent false
1949 1425 Acked Write 8 Done Done Sent false
1950 1426 Acked Write 4 Done Done Sent false
1951 1427 Acked Read 4 Done Done Sent false
1952 1428 Acked Read 1 Done Done Sent false
1953 1429 Acked Read 3 Done Done Sent false
1954 1430 Acked Read 1 Done Done Sent false
1955 1431 Acked Read 1 Done Done Sent false
1956 1432 Acked Read 6 Done Done Sent false
1957 1433 Acked Write 3 Done Done Sent false
1958 1434 Acked Read 5 Done Done Sent false
1959 1435 Acked Write 5 Done Done Sent false
1960 1436 Acked Write 5 Done Done Sent false
1961 1437 Acked Write 3 Done Done Sent false
1962 1438 Acked Write 10 Done Done Sent false
1963 1439 Acked Read 4 Done Done Sent false
1964 1440 Acked Read 1 Done Done Sent false
1965 1441 Acked Read 5 Done Done Sent false
1966 1442 Acked Read 2 Done Done Sent false
1967 1443 Acked Read 6 Done Done Sent false
1968 1444 Acked Write 6 Done Done Sent false
1969 1445 Acked Read 5 Done Done Sent false
1970 1446 Acked Read 10 Done Done Sent false
1971 1447 Acked Read 7 Done Done Sent false
1972 1448 Acked Read 8 Done Done Sent false
1973 1449 Acked Write 6 Done Done Sent false
1974 1450 Acked Read 2 Done Done Sent false
1975 1451 Acked Write 4 Done Done Sent false
1976 1452 Acked Write 8 Done Done Sent false
1977 1453 Acked Flush 0 Done Done Sent false
1978 1454 Acked Read 6 Done Done Sent false
1979 1455 Acked Read 5 Done Done Sent false
1980 1456 Acked Read 5 Done Done Sent false
1981 1457 Acked Read 7 Done Done Sent false
1982 1458 Acked Read 4 Done Done Sent false
1983 1459 Acked Read 3 Done Done Sent false
1984 1460 Acked Read 9 Done Done Sent false
1985 1461 Acked Write 4 Done Done Sent false
1986 1462 Acked Write 4 Done Done Sent false
1987 1463 Acked Read 8 Done Done Sent false
1988 1464 Acked Write 6 Done Done Sent false
1989 1465 Acked Read 3 Done Done Sent false
1990 1466 Acked Flush 0 Done Done Sent false
1991 1467 Acked Read 5 Done Done Sent false
1992 1468 Acked Read 6 Done Done Sent false
1993 1469 Acked Read 8 Done Done Sent false
1994 1470 Acked Read 10 Done Done Sent false
1995 1471 Acked Write 6 Done Done Sent false
1996 1472 Acked Flush 0 Done Done Sent false
1997 1473 Acked Flush 0 Done Done Sent false
1998 1474 Acked Read 1 Done Done Sent false
1999 1475 Acked Flush 0 Done Done Sent false
2000 1476 Acked Flush 0 Done Done Sent false
2001 1477 Acked Flush 0 Done Done Sent false
2002 1478 Acked Read 2 Done Done Sent false
2003 1479 Acked Read 1 Done Done Sent false
2004 1480 Acked Flush 0 Done Done Sent false
2005 1481 Acked Write 7 Done Done Sent false
2006 1482 Acked Read 5 Done Done Sent false
2007 1483 Acked Flush 0 Done Done Sent false
2008 1484 Acked Read 2 Done Done Sent false
2009 1485 Acked Read 8 Done Done Sent false
2010 1486 Acked Write 7 Done Done Sent false
2011 1487 Acked Write 5 Done Done Sent false
2012 1488 Acked Read 3 Done Done Sent false
2013 1489 Acked Write 1 Done Done Sent false
2014 1490 Acked Write 4 Done Done Sent false
2015 1491 Acked Write 7 Done Done Sent false
2016 1492 Acked Read 10 Done Done Sent false
2017 1493 Acked Read 10 Done Done Sent false
2018 1494 Acked Write 7 Done Done Sent false
2019 1495 Acked Write 10 Done Done Sent false
2020 1496 Acked Write 7 Done Done Sent false
2021 1497 Acked Read 9 Done Done Sent false
2022 1498 Acked Write 8 Done Done Sent false
2023 1499 Acked Read 7 Done Done Sent false
2024 1500 Acked Read 3 Done Done Sent false
2025 1501 Acked Read 6 Done Done Sent false
2026 1502 Acked Write 6 Done Done Sent false
2027 1503 Acked Write 4 Done Done Sent false
2028 1504 Acked Read 10 Done Done Sent false
2029 1505 Acked Write 2 Done Done Sent false
2030 1506 Acked Read 9 Done Done Sent false
2031 1507 Acked Read 2 Done Done Sent false
2032 1508 Acked Flush 0 Done Done Sent false
2033 1509 Acked Read 4 Done Done Sent false
2034 1510 Acked Read 10 Done Done Sent false
2035 1511 Acked Write 2 Done Done Sent false
2036 1512 Acked Write 9 Done Done Sent false
2037 1513 Acked Read 7 Done Done Sent false
2038 1514 Acked Write 3 Done Done Sent false
2039 1515 Acked Write 1 Done Done Sent false
2040 1516 Acked Read 10 Done Done Sent false
2041 1517 Acked Flush 0 Done Done Sent false
2042 1518 Acked Read 6 Done Done Sent false
2043 1519 Acked Read 10 Done Done Sent false
2044 1520 Acked Write 4 Done Done Sent false
2045 1521 Acked Write 8 Done Done Sent false
2046 1522 Acked Write 1 Done Done Sent false
2047 1523 Acked Write 6 Done Done Sent false
2048 1524 Acked Read 9 Done Done Sent false
2049 1525 Acked Write 7 Done Done Sent false
2050 1526 Acked Read 7 Done Done Sent false
2051 1527 Acked Read 4 Done Done Sent false
2052 1528 Acked Read 3 Done Done Sent false
2053 1529 Acked Read 2 Done Done Sent false
2054 1530 Acked Read 3 Done Done Sent false
2055 1531 Acked Read 3 Done Done Sent false
2056 1532 Acked Write 3 Done Done Sent false
2057 1533 Acked Flush 0 Done Done Sent false
2058 1534 Acked Read 2 Done Done Sent false
2059 1535 Acked Write 2 Done Done Sent false
2060 1536 Acked Read 5 Done Done Sent false
2061 1537 Acked Read 7 Done Done Sent false
2062 1538 Acked Read 6 Done Done Sent false
2063 1539 Acked Flush 0 Done Done Sent false
2064 1540 Acked Write 10 Done Done Sent false
2065 1541 Acked Read 5 Done Done Sent false
2066 1542 Acked Read 6 Done Done Sent false
2067 1543 Acked Write 2 Done Done Sent false
2068 1544 Acked Write 2 Done Done Sent false
2069 1545 Acked Read 10 Done Done Sent false
2070 1546 Acked Read 4 Done Done Sent false
2071 1547 Acked Write 6 Done Done Sent false
2072 1548 Acked Write 6 Done Done Sent false
2073 1549 Acked Flush 0 Done Done Sent false
2074 1550 Acked Flush 0 Done Done Sent false
2075 1551 Acked Write 5 Done Done Sent false
2076 1552 Acked Write 6 Done Done Sent false
2077 1553 Acked Write 1 Done Done Sent false
2078 1554 Acked Read 7 Done Done Sent false
2079 1555 Acked Write 2 Done Done Sent false
2080 1556 Acked Write 5 Done Done Sent false
2081 1557 Acked Write 8 Done Done Sent false
2082 1558 Acked Read 10 Done Done Sent false
2083 1559 Acked Flush 0 Done Done Sent false
2084 1560 Acked Write 7 Done Done Sent false
2085 1561 Acked Write 9 Done Done Sent false
2086 1562 Acked Flush 0 Done Done Sent false
2087 1563 Acked Read 8 Done Done Sent false
2088 1564 Acked Read 5 Done Done Sent false
2089 1565 Acked Write 7 Done Done Sent false
2090 1566 Acked Flush 0 Done Done Sent false
2091 1567 Acked Read 1 Done Done Sent false
2092 1568 Acked Flush 0 Done Done Sent false
2093 1569 Acked Read 8 Done Done Sent false
2094 1570 Acked Flush 0 Done Done Sent false
2095 1571 Acked Write 6 Done Done Sent false
2096 1572 Acked Read 3 Done Done Sent false
2097 1573 Acked Read 10 Done Done Sent false
2098 1574 Acked Write 5 Done Done Sent false
2099 1575 Acked Write 3 Done Done Sent false
2100 1576 Acked Read 3 Done Done Sent false
2101 1577 Acked Write 4 Done Done Sent false
2102 1578 Acked Write 9 Done Done Sent false
2103 1579 Acked Read 6 Done Done Sent false
2104 1580 Acked Write 3 Done Done Sent false
2105 1581 Acked Read 8 Done Done Sent false
2106 1582 Acked Read 1 Done Done Sent false
2107 1583 Acked Write 10 Done Done Sent false
2108 1584 Acked Write 2 Done Done Sent false
2109 1585 Acked Write 8 Done Done Sent false
2110 1586 Acked Write 5 Done Done Sent false
2111 1587 Acked Read 2 Done Done Sent false
2112 1588 Acked Read 1 Done Done Sent false
2113 1589 Acked Flush 0 Done Done Sent false
2114 1590 Acked Read 6 Done Done Sent false
2115 1591 Acked Write 4 Done Done Sent false
2116 1592 Acked Write 3 Done Done Sent false
2117 1593 Acked Read 4 Done Done Sent false
2118 1594 Acked Write 7 Done Done Sent false
2119 1595 Acked Read 10 Done Done Sent false
2120 1596 Acked Write 5 Done Done Sent false
2121 1597 Acked Read 1 Done Done Sent false
2122 1598 Acked Read 10 Done Done Sent false
2123 1599 Acked Write 6 Done Done Sent false
2124 1600 Acked Flush 0 Done Done Sent false
2125 1601 Acked Read 5 Done Done Sent false
2126 1602 Acked Write 5 Done Done Sent false
2127 1603 Acked Flush 0 Done Done Sent false
2128 STATES DS:0 DS:1 DS:2 TOTAL
2129 Sent 0 0 302 302
2130 Done 302 302 0 604
2131 Skipped 0 0 0 0
2132 Error 0 0 0 0
2133 Last Flush: Some(JobId(1603)) Some(JobId(1603)) Some(JobId(1301))
2134 Downstairs last five completed: 1301 1300 1299 1298 1297
2135 Upstairs last five completed: 1603 1602 1601 1600 1599
2136 CLIENT: Up:0 ds:302 act:2
21372025-01-23T17:38:09.867ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
21382025-01-23T17:38:09.867ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60788) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
21392025-01-23T17:38:09.868ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
21402025-01-23T17:38:09.869ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
21412025-01-23T17:38:09.869ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
21422025-01-23T17:38:09.869ZINFOcrucible: send last flush ID to this DS: Some(JobId(1301)) = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
21432025-01-23T17:38:09.869ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
21442025-01-23T17:38:09.869ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(1301)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
21452025-01-23T17:38:09.869ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(1301)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2146 ----------------------------------------------------------------
2147 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2148 Downstairs last five completed: 1603 1602 1601 1600 1599
2149 Upstairs last five completed: 1603 1602 1601 1600 1599
2150 CLIENT: Up:0 ds:0 act:3
2151 Replay: All jobs finished
2152 Main task has work to do, go find it
2153 got dsc Stop(0)
2154 stop 0
2155 Jan 23 17:38:12.113 INFO request completed, latency_us: 335, [8810] Got stop action so:false kr:false
2156 response_code: 204, uri: /stop/cid/0, method: POST, req_id: 76d032a7-011a-4903-9b9c-1d173c7ab944, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
21572025-01-23T17:38:12.114ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
21582025-01-23T17:38:12.114ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
21592025-01-23T17:38:12.114ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2160 Jan 23 17:38:12.114 INFO request completed, latency_us: 139, response_code: 200, uri: /state/cid/0, method: GET, req_id: 7500275d-7f06-4402-9c41-7f5ce67a61d8, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
21612025-01-23T17:38:12.114ZINFOcrucible: client task is exiting = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
2162 [8810] Exited with: ExitStatus(unix_wait_status(9))
2163 [8810][0] reports Exit
2164 Jan 23 17:38:16.116 INFO request completed, latency_us: 249, response_code: 200, uri: /state/cid/0, method: GET, req_id: daa642e5-fbb7-445a-bf21-52ebf2b825f6, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
2165 001/300 Write block 984 len 36864 data: 3 3 3 3 3 3 2 2 2
2166 002/300 Read block 2702 len 20480
2167 003/300 Write block 2464 len 16384 data: 2 2 2 2
2168 004/300 Write block 4232 len 24576 data: 2 3 3 2 2 2
2169 005/300 Write block 832 len 32768 data: 2 2 2 2 2 2 2 2
2170 006/300 Write block 3092 len 8192 data: 2 2
2171 007/300 Write block 521 len 24576 data: 2 2 2 2 2 2
2172 008/300 Read block 1556 len 36864
2173 009/300 Read block 946 len 40960
2174 010/300 Write block 3183 len 20480 data: 3 3 3 3 3
2175 011/300 Read block 1499 len 20480
2176 012/300 Read block 2928 len 4096
2177 013/300 Read block 3430 len 20480
2178 014/300 Read block 2924 len 40960
2179 015/300 Flush
2180 016/300 Write block 2610 len 28672 data: 2 2 2 2 3 3 3
2181 017/300 Read block 2458 len 12288
2182 018/300 Read block 951 len 32768
2183 019/300 Read block 2268 len 20480
2184 020/300 Read block 255 len 40960
2185 021/300 Write block 2256 len 24576 data: 2 2 2 2 2 2
2186 022/300 Write block 483 len 32768 data: 2 2 2 2 2 2 2 2
2187 023/300 Write block 3720 len 20480 data: 2 2 2 2 2
2188 024/300 Read block 2943 len 24576
2189 025/300 Write block 2222 len 4096 data: 3
2190 026/300 Read block 3241 len 32768
2191 027/300 Flush
2192 028/300 Read block 3166 len 40960
2193 029/300 Write block 2941 len 36864 data: 2 2 2 2 2 2 2 2 2
2194 030/300 Write block 2767 len 16384 data: 2 2 2 2
2195 031/300 Write block 69 len 12288 data: 3 3 2
2196 032/300 Read block 2242 len 16384
2197 033/300 Flush
2198 034/300 Read block 4066 len 16384
2199 035/300 Write block 2479 len 28672 data: 3 2 2 2 2 2 2
2200 036/300 Write block 3968 len 16384 data: 2 2 2 2
2201 037/300 Flush
2202 038/300 Read block 3122 len 40960
2203 039/300 Read block 3890 len 28672
2204 040/300 Read block 1885 len 20480
2205 041/300 Read block 909 len 40960
2206 042/300 Read block 4582 len 20480
2207 043/300 Read block 1850 len 16384
2208 044/300 Write block 4869 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2209 045/300 Flush
2210 046/300 Read block 1934 len 40960
2211 047/300 Read block 4294 len 16384
2212 048/300 Write block 4952 len 16384 data: 3 3 2 2
2213 049/300 Flush
2214 050/300 Read block 841 len 16384
2215 051/300 Flush
2216 052/300 Write block 820 len 32768 data: 2 2 3 3 3 3 3 3
2217 053/300 Read block 3179 len 12288
2218 054/300 Read block 2689 len 4096
2219 055/300 Flush
2220 056/300 Write block 2351 len 4096 data: 2
2221 057/300 Read block 1170 len 4096
2222 058/300 Write block 288 len 12288 data: 3 3 3
2223 059/300 Read block 4485 len 12288
2224 060/300 Write block 1219 len 28672 data: 2 2 2 2 2 2 2
2225 061/300 Read block 333 len 8192
2226 062/300 Read block 2119 len 12288
2227 063/300 Read block 4107 len 16384
2228 064/300 Read block 3924 len 32768
2229 065/300 Read block 424 len 40960
2230 066/300 Flush
2231 067/300 Read block 3801 len 20480
2232 068/300 Read block 1080 len 12288
2233 069/300 Read block 2426 len 40960
2234 070/300 Read block 1955 len 4096
2235 071/300 Read block 3257 len 24576
2236 072/300 Read block 669 len 32768
2237 073/300 Read block 2317 len 16384
2238 074/300 Flush
2239 075/300 Flush
2240 076/300 Read block 4589 len 4096
2241 077/300 Read block 1675 len 24576
2242 078/300 Read block 4377 len 4096
2243 079/300 Read block 2481 len 36864
2244 080/300 Write block 718 len 36864 data: 2 2 2 2 2 2 2 2 2
2245 081/300 Read block 1275 len 8192
2246 082/300 Read block 3887 len 28672
2247 083/300 Read block 4664 len 4096
2248 084/300 Write block 1894 len 8192 data: 2 2
2249 085/300 Read block 114 len 8192
2250 086/300 Read block 1580 len 32768
2251 087/300 Write block 2348 len 16384 data: 2 2 2 3
2252 088/300 Write block 3529 len 20480 data: 2 2 2 3 3
2253 089/300 Flush
2254 090/300 Read block 1781 len 4096
2255 091/300 Read block 448 len 24576
2256 092/300 Read block 1004 len 20480
2257 093/300 Write block 4398 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2258 094/300 Flush
2259 095/300 Write block 1904 len 40960 data: 2 2 2 3 3 3 3 3 3 3
2260 096/300 Flush
2261 097/300 Read block 1684 len 8192
2262 098/300 Read block 4700 len 28672
2263 099/300 Read block 713 len 16384
2264 100/300 Write block 1257 len 40960 data: 3 3 3 3 3 3 3 2 2 2
2265 101/300 Write block 160 len 32768 data: 3 3 3 2 2 2 2 2
2266 102/300 Read block 911 len 28672
2267 103/300 Read block 3103 len 40960
2268 104/300 Read block 3050 len 12288
2269 105/300 Read block 3342 len 12288
2270 106/300 Read block 374 len 36864
2271 107/300 Write block 764 len 36864 data: 2 2 2 2 2 2 2 3 3
2272 108/300 Read block 3768 len 24576
2273 109/300 Read block 981 len 16384
2274 110/300 Read block 3781 len 12288
2275 111/300 Write block 1071 len 24576 data: 2 2 2 2 2 2
2276 112/300 Write block 1993 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2277 113/300 Read block 3239 len 8192
2278 114/300 Read block 4024 len 8192
2279 115/300 Flush
2280 116/300 Read block 4199 len 12288
2281 117/300 Write block 1061 len 20480 data: 2 2 2 2 2
2282 118/300 Read block 4926 len 12288
2283 119/300 Read block 3820 len 36864
2284 120/300 Write block 2193 len 32768 data: 2 2 2 2 2 2 2 2
2285 121/300 Write block 3523 len 4096 data: 3
2286 122/300 Read block 4613 len 24576
2287 123/300 Read block 4658 len 28672
2288 124/300 Read block 292 len 8192
2289 125/300 Write block 1734 len 4096 data: 2
2290 126/300 Write block 1623 len 24576 data: 2 2 2 2 2 2
2291 127/300 Flush
2292 128/300 Write block 2239 len 4096 data: 2
2293 129/300 Read block 4375 len 8192
2294 130/300 Write block 4205 len 16384 data: 2 2 2 2
2295 131/300 Flush
2296 132/300 Read block 4563 len 8192
2297 133/300 Read block 3963 len 20480
2298 134/300 Read block 4947 len 40960
2299 135/300 Read block 3295 len 20480
2300 136/300 Write block 4206 len 28672 data: 3 3 3 2 2 2 2
2301 137/300 Write block 832 len 12288 data: 3 3 3
2302 138/300 Read block 727 len 36864
2303 139/300 Read block 2369 len 12288
2304 140/300 Read block 4253 len 40960
2305 141/300 Write block 4179 len 12288 data: 2 2 2
2306 142/300 Read block 3321 len 8192
2307 143/300 Flush
2308 144/300 Read block 3974 len 28672
2309 145/300 Write block 2345 len 36864 data: 2 2 2 3 3 3 4 2 2
2310 146/300 Write block 3907 len 36864 data: 2 2 2 2 2 2 2 2 2
2311 147/300 Read block 2952 len 16384
2312 148/300 Read block 4928 len 20480
2313 149/300 Read block 1649 len 36864
2314 150/300 Write block 1907 len 8192 data: 4 4
2315 151/300 Read block 16 len 32768
2316 152/300 Read block 2703 len 24576
2317 153/300 Write block 2898 len 24576 data: 2 2 2 2 2 2
2318 154/300 Read block 512 len 20480
2319 155/300 Read block 1798 len 36864
2320 156/300 Read block 2392 len 32768
2321 157/300 Read block 4264 len 36864
2322 158/300 Write block 722 len 28672 data: 3 3 3 3 3 2 2
2323 159/300 Read block 2700 len 16384
2324 160/300 Write block 936 len 12288 data: 2 2 2
2325 161/300 Read block 4735 len 32768
2326 162/300 Read block 8 len 32768
2327 163/300 Read block 1128 len 24576
2328 164/300 Flush
2329 165/300 Read block 2894 len 4096
2330 166/300 Write block 459 len 32768 data: 2 2 2 2 2 2 2 2
2331 167/300 Write block 3550 len 40960 data: 3 3 3 3 3 2 2 2 2 2
2332 168/300 Write block 1477 len 16384 data: 4 4 3 3
2333 169/300 Write block 461 len 28672 data: 3 3 3 3 3 3 2
2334 170/300 Write block 110 len 40960 data: 3 3 3 4 4 4 4 4 4 4
2335 171/300 Write block 4435 len 20480 data: 3 3 2 2 2
2336 172/300 Write block 1004 len 16384 data: 2 2 2 2
2337 173/300 Write block 4578 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2338 174/300 Read block 1620 len 16384
2339 175/300 Write block 1427 len 4096 data: 3
2340 176/300 Read block 886 len 20480
2341 177/300 Write block 2797 len 20480 data: 2 2 2 2 2
2342 178/300 Write block 3578 len 36864 data: 2 2 2 2 2 2 2 2 2
2343 179/300 Read block 3577 len 40960
2344 180/300 Write block 1904 len 24576 data: 3 3 3 5 5 4
2345 181/300 Write block 4943 len 20480 data: 2 2 2 2 2
2346 182/300 Flush
2347 183/300 Flush
2348 184/300 Flush
2349 185/300 Flush
2350 186/300 Write block 3439 len 24576 data: 2 2 2 2 2 2
2351 187/300 Read block 664 len 32768
2352 188/300 Write block 1866 len 8192 data: 2 2
2353 189/300 Read block 4800 len 4096
2354 190/300 Write block 4767 len 20480 data: 2 2 2 2 2
2355 191/300 Write block 616 len 16384 data: 2 2 3 4
2356 192/300 Read block 1317 len 20480
2357 193/300 Read block 939 len 36864
2358 194/300 Read block 1210 len 36864
2359 195/300 Flush
2360 196/300 Read block 525 len 20480
2361 197/300 Read block 4503 len 40960
2362 198/300 Read block 4908 len 36864
2363 199/300 Read block 579 len 12288
2364 200/300 Read block 4670 len 36864
2365 201/300 Read block 4753 len 16384
2366 202/300 Read block 1728 len 40960
2367 203/300 Write block 217 len 12288 data: 2 3 2
2368 204/300 Read block 4286 len 24576
2369 205/300 Read block 2782 len 40960
2370 206/300 Write block 982 len 4096 data: 2
2371 207/300 Write block 4399 len 32768 data: 3 3 3 3 3 3 3 3
2372 208/300 Read block 2984 len 12288
2373 209/300 Read block 3506 len 12288
2374 210/300 Write block 651 len 8192 data: 2 2
2375 211/300 Read block 2344 len 20480
2376 212/300 Read block 4594 len 28672
2377 213/300 Write block 901 len 28672 data: 2 2 2 2 2 2 2
2378 214/300 Read block 3260 len 40960
2379 215/300 Write block 2971 len 20480 data: 3 3 3 3 3
2380 216/300 Write block 2737 len 20480 data: 2 2 2 2 2
2381 217/300 Write block 328 len 24576 data: 2 2 2 2 2 2
2382 218/300 Read block 3849 len 20480
2383 219/300 Read block 3375 len 12288
2384 220/300 Write block 1955 len 4096 data: 2
2385 221/300 Flush
2386 222/300 Write block 4283 len 8192 data: 2 2
2387 223/300 Read block 1615 len 24576
2388 224/300 Read block 2392 len 32768
2389 225/300 Read block 1124 len 40960
2390 226/300 Write block 842 len 16384 data: 2 2 2 2
2391 227/300 Write block 171 len 40960 data: 3 3 3 3 3 3 3 3 2 2
2392 228/300 Write block 2104 len 4096 data: 2
2393 229/300 Read block 1994 len 32768
2394 230/300 Read block 686 len 4096
2395 231/300 Read block 1466 len 32768
2396 232/300 Write block 4034 len 20480 data: 4 4 4 4 4
2397 233/300 Read block 2906 len 32768
2398 234/300 Write block 3292 len 4096 data: 2
2399 235/300 Read block 1546 len 8192
2400 236/300 Read block 916 len 20480
2401 237/300 Read block 2048 len 32768
2402 238/300 Write block 2851 len 36864 data: 2 2 2 2 2 2 3 3 3
2403 239/300 Read block 2222 len 8192
2404 240/300 Read block 1454 len 8192
2405 241/300 Flush
2406 242/300 Read block 4404 len 20480
2407 243/300 Write block 106 len 4096 data: 2
2408 244/300 Write block 1569 len 24576 data: 2 2 2 2 2 2
2409 245/300 Read block 3268 len 4096
2410 246/300 Write block 3374 len 8192 data: 2 2
2411 247/300 Read block 4205 len 16384
2412 248/300 Read block 3493 len 12288
2413 249/300 Write block 962 len 4096 data: 2
2414 250/300 Read block 2548 len 32768
2415 251/300 Flush
2416 252/300 Read block 4812 len 12288
2417 253/300 Read block 4056 len 40960
2418 254/300 Read block 2255 len 24576
2419 255/300 Write block 4609 len 24576 data: 2 2 2 2 2 2
2420 256/300 Write block 2189 len 40960 data: 2 2 2 2 3 3 3 3 3 3
2421 257/300 Write block 295 len 20480 data: 2 2 2 3 3
2422 258/300 Write block 2378 len 32768 data: 2 2 2 2 2 2 2 2
2423 259/300 Read block 4315 len 16384
2424 260/300 Read block 2421 len 36864
2425 261/300 Read block 2109 len 32768
2426 262/300 Read block 316 len 12288
2427 263/300 Write block 2830 len 24576 data: 2 2 2 2 2 2
2428 264/300 Write block 3032 len 40960 data: 2 3 3 3 2 2 2 2 2 2
2429 265/300 Read block 1641 len 16384
2430 266/300 Write block 2398 len 20480 data: 3 3 3 2 2
2431 267/300 Read block 2998 len 36864
2432 268/300 Write block 203 len 16384 data: 3 3 3 3
2433 269/300 Write block 995 len 24576 data: 2 2 2 2 2 2
2434 270/300 Write block 3197 len 32768 data: 2 3 3 3 2 2 2 3
2435 271/300 Read block 2096 len 20480
2436 272/300 Read block 4569 len 16384
2437 273/300 Write block 728 len 36864 data: 3 2 2 2 2 2 2 2 2
2438 274/300 Write block 4426 len 8192 data: 2 2
2439 275/300 Write block 1292 len 40960 data: 4 4 2 2 2 2 2 2 2 2
2440 276/300 Read block 2985 len 16384
2441 277/300 Read block 4253 len 16384
2442 278/300 Write block 2189 len 8192 data: 3 3
2443 279/300 Read block 4787 len 28672
2444 280/300 Read block 2280 len 28672
2445 281/300 Write block 876 len 24576 data: 2 2 2 2 2 2
2446 282/300 Write block 3814 len 4096 data: 2
2447 283/300 Write block 4870 len 28672 data: 3 3 3 3 3 3 3
2448 284/300 Flush
2449 285/300 Read block 913 len 4096
2450 286/300 Write block 4923 len 24576 data: 2 2 2 2 2 2
2451 287/300 Read block 3185 len 12288
2452 288/300 Write block 1763 len 8192 data: 3 3
2453 289/300 Write block 981 len 28672 data: 2 3 2 4 4 4 4
2454 290/300 Write block 2009 len 16384 data: 2 2 2 2
2455 291/300 Read block 2549 len 16384
2456 292/300 Read block 814 len 16384
2457 293/300 Read block 1475 len 36864
2458 294/300 Read block 87 len 16384
2459 295/300 Read block 4273 len 24576
2460 296/300 Read block 346 len 20480
2461 297/300 Read block 2362 len 4096
2462 298/300 Read block 711 len 40960
2463 299/300 Read block 4431 len 20480
2464 300/300 Read block 2767 len 24576
2465 301/300 Write block 2996 len 8192 data: 2 2
2466 Main task has work to do, go find it
2467 got dsc Start(0)
2468 start 0
2469 Jan 23 17:38:16.363 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/0, method: POST, req_id: 900271bc-4fc4-4a56-89f8-49f6ae48dffe, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
2470 Replay: started 0, returned:Ok(())
2471 ----------------------------------------------------------------
2472 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2473 States: Offline Active Active
2474 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2475 1604 Acked Write 9 Sent Done Done false
2476 1605 Acked Read 5 Sent Done Done false
2477 1606 Acked Write 4 Sent Done Done false
2478 1607 Acked Write 6 Sent Done Done false
2479 1608 Acked Write 8 Sent Done Done false
2480 1609 Acked Write 2 Sent Done Done false
2481 1610 Acked Write 6 Sent Done Done false
2482 1611 Acked Read 9 Sent Done Done false
2483 1612 Acked Read 10 Sent Done Done false
2484 1613 Acked Write 5 Sent Done Done false
2485 1614 Acked Read 5 Sent Done Done false
2486 1615 Acked Read 1 Sent Done Done false
2487 1616 Acked Read 5 Sent Done Done false
2488 1617 Acked Read 10 Sent Done Done false
2489 1618 Acked Flush 0 Sent Done Done false
2490 1619 Acked Write 7 Sent Done Done false
2491 1620 Acked Read 3 Sent Done Done false
2492 1621 Acked Read 8 Sent Done Done false
2493 1622 Acked Read 5 Sent Done Done false
2494 1623 Acked Read 10 Sent Done Done false
2495 1624 Acked Write 6 Sent Done Done false
2496 1625 Acked Write 8 Sent Done Done false
2497 1626 Acked Write 5 Sent Done Done false
2498 1627 Acked Read 6 Sent Done Done false
2499 1628 Acked Write 1 Sent Done Done false
2500 1629 Acked Read 8 Sent Done Done false
2501 1630 Acked Flush 0 Sent Done Done false
2502 1631 Acked Read 10 Sent Done Done false
2503 1632 Acked Write 9 Sent Done Done false
2504 1633 Acked Write 4 Sent Done Done false
2505 1634 Acked Write 3 Sent Done Done false
2506 1635 Acked Read 4 Sent Done Done false
2507 1636 Acked Flush 0 Sent Done Done false
2508 1637 Acked Read 4 Sent Done Done false
2509 1638 Acked Write 7 Sent Done Done false
2510 1639 Acked Write 4 Sent Done Done false
2511 1640 Acked Flush 0 Sent Done Done false
2512 1641 Acked Read 10 Sent Done Done false
2513 1642 Acked Read 7 Sent Done Done false
2514 1643 Acked Read 5 Sent Done Done false
2515 1644 Acked Read 10 Sent Done Done false
2516 1645 Acked Read 5 Sent Done Done false
2517 1646 Acked Read 4 Sent Done Done false
2518 1647 Acked Write 10 Sent Done Done false
2519 1648 Acked Flush 0 Sent Done Done false
2520 1649 Acked Read 10 Sent Done Done false
2521 1650 Acked Read 4 Sent Done Done false
2522 1651 Acked Write 4 Sent Done Done false
2523 1652 Acked Flush 0 Sent Done Done false
2524 1653 Acked Read 4 Sent Done Done false
2525 1654 Acked Flush 0 Sent Done Done false
2526 1655 Acked Write 8 Sent Done Done false
2527 1656 Acked Read 3 Sent Done Done false
2528 1657 Acked Read 1 Sent Done Done false
2529 1658 Acked Flush 0 Sent Done Done false
2530 1659 Acked Write 1 Sent Done Done false
2531 1660 Acked Read 1 Sent Done Done false
2532 1661 Acked Write 3 Sent Done Done false
2533 1662 Acked Read 3 Sent Done Done false
2534 1663 Acked Write 7 Sent Done Done false
2535 1664 Acked Read 2 Sent Done Done false
2536 1665 Acked Read 3 Sent Done Done false
2537 1666 Acked Read 4 Sent Done Done false
2538 1667 Acked Read 8 Sent Done Done false
2539 1668 Acked Read 10 Sent Done Done false
2540 1669 Acked Flush 0 Sent Done Done false
2541 1670 Acked Read 5 Sent Done Done false
2542 1671 Acked Read 3 Sent Done Done false
2543 1672 Acked Read 10 Sent Done Done false
2544 1673 Acked Read 1 Sent Done Done false
2545 1674 Acked Read 6 Sent Done Done false
2546 1675 Acked Read 8 Sent Done Done false
2547 1676 Acked Read 4 Sent Done Done false
2548 1677 Acked Flush 0 Sent Done Done false
2549 1678 Acked Flush 0 Sent Done Done false
2550 1679 Acked Read 1 Sent Done Done false
2551 1680 Acked Read 6 Sent Done Done false
2552 1681 Acked Read 1 Sent Done Done false
2553 1682 Acked Read 9 Sent Done Done false
2554 1683 Acked Write 9 Sent Done Done false
2555 1684 Acked Read 2 Sent Done Done false
2556 1685 Acked Read 7 Sent Done Done false
2557 1686 Acked Read 1 Sent Done Done false
2558 1687 Acked Write 2 Sent Done Done false
2559 1688 Acked Read 2 Sent Done Done false
2560 1689 Acked Read 8 Sent Done Done false
2561 1690 Acked Write 4 Sent Done Done false
2562 1691 Acked Write 5 Sent Done Done false
2563 1692 Acked Flush 0 Sent Done Done false
2564 1693 Acked Read 1 Sent Done Done false
2565 1694 Acked Read 6 Sent Done Done false
2566 1695 Acked Read 5 Sent Done Done false
2567 1696 Acked Write 10 Sent Done Done false
2568 1697 Acked Flush 0 Sent Done Done false
2569 1698 Acked Write 10 Sent Done Done false
2570 1699 Acked Flush 0 Sent Done Done false
2571 1700 Acked Read 2 Sent Done Done false
2572 1701 Acked Read 7 Sent Done Done false
2573 1702 Acked Read 4 Sent Done Done false
2574 1703 Acked Write 10 Sent Done Done false
2575 1704 Acked Write 8 Sent Done Done false
2576 1705 Acked Read 7 Sent Done Done false
2577 1706 Acked Read 10 Sent Done Done false
2578 1707 Acked Read 3 Sent Done Done false
2579 1708 Acked Read 3 Sent Done Done false
2580 1709 Acked Read 9 Sent Done Done false
2581 1710 Acked Write 9 Sent Done Done false
2582 1711 Acked Read 6 Sent Done Done false
2583 1712 Acked Read 4 Sent Done Done false
2584 1713 Acked Read 3 Sent Done Done false
2585 1714 Acked Write 6 Sent Done Done false
2586 1715 Acked Write 10 Sent Done Done false
2587 1716 Acked Read 2 Sent Done Done false
2588 1717 Acked Read 2 Sent Done Done false
2589 1718 Acked Flush 0 Sent Done Done false
2590 1719 Acked Read 3 Sent Done Done false
2591 1720 Acked Write 5 Sent Done Done false
2592 1721 Acked Read 3 Sent Done Done false
2593 1722 Acked Read 9 Sent Done Done false
2594 1723 Acked Write 8 Sent Done Done false
2595 1724 Acked Write 1 Sent Done Done false
2596 1725 Acked Read 6 Sent Done Done false
2597 1726 Acked Read 7 Sent Done Done false
2598 1727 Acked Read 2 Sent Done Done false
2599 1728 Acked Write 1 Sent Done Done false
2600 1729 Acked Write 6 Sent Done Done false
2601 1730 Acked Flush 0 Sent Done Done false
2602 1731 Acked Write 1 Sent Done Done false
2603 1732 Acked Read 2 Sent Done Done false
2604 1733 Acked Write 4 Sent Done Done false
2605 1734 Acked Flush 0 Sent Done Done false
2606 1735 Acked Read 2 Sent Done Done false
2607 1736 Acked Read 5 Sent Done Done false
2608 1737 Acked Read 10 Sent Done Done false
2609 1738 Acked Read 5 Sent Done Done false
2610 1739 Acked Write 7 Sent Done Done false
2611 1740 Acked Write 3 Sent Done Done false
2612 1741 Acked Read 9 Sent Done Done false
2613 1742 Acked Read 3 Sent Done Done false
2614 1743 Acked Read 10 Sent Done Done false
2615 1744 Acked Write 3 Sent Done Done false
2616 1745 Acked Read 2 Sent Done Done false
2617 1746 Acked Flush 0 Sent Done Done false
2618 1747 Acked Read 7 Sent Done Done false
2619 1748 Acked Write 9 Sent Done Done false
2620 1749 Acked Write 9 Sent Done Done false
2621 1750 Acked Read 4 Sent Done Done false
2622 1751 Acked Read 5 Sent Done Done false
2623 1752 Acked Read 9 Sent Done Done false
2624 1753 Acked Write 2 Sent Done Done false
2625 1754 Acked Read 8 Sent Done Done false
2626 1755 Acked Read 6 Sent Done Done false
2627 1756 Acked Write 6 Sent Done Done false
2628 1757 Acked Read 5 Sent Done Done false
2629 1758 Acked Read 9 Sent Done Done false
2630 1759 Acked Read 8 Sent Done Done false
2631 1760 Acked Read 9 Sent Done Done false
2632 1761 Acked Write 7 Sent Done Done false
2633 1762 Acked Read 4 Sent Done Done false
2634 1763 Acked Write 3 Sent Done Done false
2635 1764 Acked Read 8 Sent Done Done false
2636 1765 Acked Read 8 Sent Done Done false
2637 1766 Acked Read 6 Sent Done Done false
2638 1767 Acked Flush 0 Sent Done Done false
2639 1768 Acked Read 1 Sent Done Done false
2640 1769 Acked Write 8 Sent Done Done false
2641 1770 Acked Write 10 Sent Done Done false
2642 1771 Acked Write 4 Sent Done Done false
2643 1772 Acked Write 7 Sent Done Done false
2644 1773 Acked Write 10 Sent Done Done false
2645 1774 Acked Write 5 Sent Done Done false
2646 1775 Acked Write 4 Sent Done Done false
2647 1776 Acked Write 10 Sent Done Done false
2648 1777 Acked Read 4 Sent Done Done false
2649 1778 Acked Write 1 Sent Done Done false
2650 1779 Acked Read 5 Sent Done Done false
2651 1780 Acked Write 5 Sent Done Done false
2652 1781 Acked Write 9 Sent Done Done false
2653 1782 Acked Read 10 Sent Done Done false
2654 1783 Acked Write 6 Sent Done Done false
2655 1784 Acked Write 5 Sent Done Done false
2656 1785 Acked Flush 0 Sent Done Done false
2657 1786 Acked Flush 0 Sent Done Done false
2658 1787 Acked Flush 0 Sent Done Done false
2659 1788 Acked Flush 0 Sent Done Done false
2660 1789 Acked Write 6 Sent Done Done false
2661 1790 Acked Read 8 Sent Done Done false
2662 1791 Acked Write 2 Sent Done Done false
2663 1792 Acked Read 1 Sent Done Done false
2664 1793 Acked Write 5 Sent Done Done false
2665 1794 Acked Write 4 Sent Done Done false
2666 1795 Acked Read 5 Sent Done Done false
2667 1796 Acked Read 9 Sent Done Done false
2668 1797 Acked Read 9 Sent Done Done false
2669 1798 Acked Flush 0 Sent Done Done false
2670 1799 Acked Read 5 Sent Done Done false
2671 1800 Acked Read 10 Sent Done Done false
2672 1801 Acked Read 9 Sent Done Done false
2673 1802 Acked Read 3 Sent Done Done false
2674 1803 Acked Read 9 Sent Done Done false
2675 1804 Acked Read 4 Sent Done Done false
2676 1805 Acked Read 10 Sent Done Done false
2677 1806 Acked Write 3 Sent Done Done false
2678 1807 Acked Read 6 Sent Done Done false
2679 1808 Acked Read 10 Sent Done Done false
2680 1809 Acked Write 1 Sent Done Done false
2681 1810 Acked Write 8 Sent Done Done false
2682 1811 Acked Read 3 Sent Done Done false
2683 1812 Acked Read 3 Sent Done Done false
2684 1813 Acked Write 2 Sent Done Done false
2685 1814 Acked Read 5 Sent Done Done false
2686 1815 Acked Read 7 Sent Done Done false
2687 1816 Acked Write 7 Sent Done Done false
2688 1817 Acked Read 10 Sent Done Done false
2689 1818 Acked Write 5 Sent Done Done false
2690 1819 Acked Write 5 Sent Done Done false
2691 1820 Acked Write 6 Sent Done Done false
2692 1821 Acked Read 5 Sent Done Done false
2693 1822 Acked Read 3 Sent Done Done false
2694 1823 Acked Write 1 Sent Done Done false
2695 1824 Acked Flush 0 Sent Done Done false
2696 1825 Acked Write 2 Sent Done Done false
2697 1826 Acked Read 6 Sent Done Done false
2698 1827 Acked Read 8 Sent Done Done false
2699 1828 Acked Read 10 Sent Done Done false
2700 1829 Acked Write 4 Sent Done Done false
2701 1830 Acked Write 10 Sent Done Done false
2702 1831 Acked Write 1 Sent Done Done false
2703 1832 Acked Read 8 Sent Done Done false
2704 1833 Acked Read 1 Sent Done Done false
2705 1834 Acked Read 8 Sent Done Done false
2706 1835 Acked Write 5 Sent Done Done false
2707 1836 Acked Read 8 Sent Done Done false
2708 1837 Acked Write 1 Sent Done Done false
2709 1838 Acked Read 2 Sent Done Done false
2710 1839 Acked Read 5 Sent Done Done false
2711 1840 Acked Read 8 Sent Done Done false
2712 1841 Acked Write 9 Sent Done Done false
2713 1842 Acked Read 2 Sent Done Done false
2714 1843 Acked Read 2 Sent Done Done false
2715 1844 Acked Flush 0 Sent Done Done false
2716 1845 Acked Read 5 Sent Done Done false
2717 1846 Acked Write 1 Sent Done Done false
2718 1847 Acked Write 6 Sent Done Done false
2719 1848 Acked Read 1 Sent Done Done false
2720 1849 Acked Write 2 Sent Done Done false
2721 1850 Acked Read 4 Sent Done Done false
2722 1851 Acked Read 3 Sent Done Done false
2723 1852 Acked Write 1 Sent Done Done false
2724 1853 Acked Read 8 Sent Done Done false
2725 1854 Acked Flush 0 Sent Done Done false
2726 1855 Acked Read 3 Sent Done Done false
2727 1856 Acked Read 10 Sent Done Done false
2728 1857 Acked Read 6 Sent Done Done false
2729 1858 Acked Write 6 Sent Done Done false
2730 1859 Acked Write 10 Sent Done Done false
2731 1860 Acked Write 5 Sent Done Done false
2732 1861 Acked Write 8 Sent Done Done false
2733 1862 Acked Read 4 Sent Done Done false
2734 1863 Acked Read 9 Sent Done Done false
2735 1864 Acked Read 8 Sent Done Done false
2736 1865 Acked Read 3 Sent Done Done false
2737 1866 Acked Write 6 Sent Done Done false
2738 1867 Acked Write 10 Sent Done Done false
2739 1868 Acked Read 4 Sent Done Done false
2740 1869 Acked Write 5 Sent Done Done false
2741 1870 Acked Read 9 Sent Done Done false
2742 1871 Acked Write 4 Sent Done Done false
2743 1872 Acked Write 6 Sent Done Done false
2744 1873 Acked Write 8 Sent Done Done false
2745 1874 Acked Read 5 Sent Done Done false
2746 1875 Acked Read 4 Sent Done Done false
2747 1876 Acked Write 9 Sent Done Done false
2748 1877 Acked Write 2 Sent Done Done false
2749 1878 Acked Write 10 Sent Done Done false
2750 1879 Acked Read 4 Sent Done Done false
2751 1880 Acked Read 4 Sent Done Done false
2752 1881 Acked Write 2 Sent Done Done false
2753 1882 Acked Read 7 Sent Done Done false
2754 1883 Acked Read 7 Sent Done Done false
2755 1884 Acked Write 6 Sent Done Done false
2756 1885 Acked Write 1 Sent Done Done false
2757 1886 Acked Write 7 Sent Done Done false
2758 1887 Acked Flush 0 Sent Done Done false
2759 1888 Acked Read 1 Sent Done Done false
2760 1889 Acked Write 6 Sent Done Done false
2761 1890 Acked Read 3 Sent Done Done false
2762 1891 Acked Write 2 Sent Done Done false
2763 1892 Acked Write 7 Sent Done Done false
2764 1893 Acked Write 4 Sent Done Done false
2765 1894 Acked Read 4 Sent Done Done false
2766 1895 Acked Read 4 Sent Done Done false
2767 1896 Acked Read 9 Sent Done Done false
2768 1897 Acked Read 4 Sent Done Done false
2769 1898 Acked Read 6 Sent Done Done false
2770 1899 Acked Read 5 Sent Done Done false
2771 1900 Acked Read 1 Sent Done Done false
2772 1901 Acked Read 10 Sent Done Done false
2773 1902 Acked Read 5 Sent Done Done false
2774 1903 Acked Read 6 Sent Done Done false
2775 1904 Acked Write 2 Sent Done Done false
2776 STATES DS:0 DS:1 DS:2 TOTAL
2777 Sent 301 0 0 301
2778 Done 0 301 301 602
2779 Skipped 0 0 0 0
2780 Error 0 0 0 0
2781 Last Flush: Some(JobId(1603)) Some(JobId(1887)) Some(JobId(1887))
2782 Downstairs last five completed: 1603 1602 1601 1600 1599
2783 Upstairs last five completed: 1904 1903 1902 1901 1900
2784 CLIENT: Up:0 ds:301 act:2
2785 ----------------------------------------------------------------
2786 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
2787 States: Offline Active Active
2788 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2789 1604 Acked Write 9 Sent Done Done false
2790 1605 Acked Read 5 Sent Done Done false
2791 1606 Acked Write 4 Sent Done Done false
2792 1607 Acked Write 6 Sent Done Done false
2793 1608 Acked Write 8 Sent Done Done false
2794 1609 Acked Write 2 Sent Done Done false
2795 1610 Acked Write 6 Sent Done Done false
2796 1611 Acked Read 9 Sent Done Done false
2797 1612 Acked Read 10 Sent Done Done false
2798 1613 Acked Write 5 Sent Done Done false
2799 1614 Acked Read 5 Sent Done Done false
2800 1615 Acked Read 1 Sent Done Done false
2801 1616 Acked Read 5 Sent Done Done false
2802 1617 Acked Read 10 Sent Done Done false
2803 1618 Acked Flush 0 Sent Done Done false
2804 1619 Acked Write 7 Sent Done Done false
2805 1620 Acked Read 3 Sent Done Done false
2806 1621 Acked Read 8 Sent Done Done false
2807 1622 Acked Read 5 Sent Done Done false
2808 1623 Acked Read 10 Sent Done Done false
2809 1624 Acked Write 6 Sent Done Done false
2810 1625 Acked Write 8 Sent Done Done false
2811 1626 Acked Write 5 Sent Done Done false
2812 1627 Acked Read 6 Sent Done Done false
2813 1628 Acked Write 1 Sent Done Done false
2814 1629 Acked Read 8 Sent Done Done false
2815 1630 Acked Flush 0 Sent Done Done false
2816 1631 Acked Read 10 Sent Done Done false
2817 1632 Acked Write 9 Sent Done Done false
2818 1633 Acked Write 4 Sent Done Done false
2819 1634 Acked Write 3 Sent Done Done false
2820 1635 Acked Read 4 Sent Done Done false
2821 1636 Acked Flush 0 Sent Done Done false
2822 1637 Acked Read 4 Sent Done Done false
2823 1638 Acked Write 7 Sent Done Done false
2824 1639 Acked Write 4 Sent Done Done false
2825 1640 Acked Flush 0 Sent Done Done false
2826 1641 Acked Read 10 Sent Done Done false
2827 1642 Acked Read 7 Sent Done Done false
2828 1643 Acked Read 5 Sent Done Done false
2829 1644 Acked Read 10 Sent Done Done false
2830 1645 Acked Read 5 Sent Done Done false
2831 1646 Acked Read 4 Sent Done Done false
2832 1647 Acked Write 10 Sent Done Done false
2833 1648 Acked Flush 0 Sent Done Done false
2834 1649 Acked Read 10 Sent Done Done false
2835 1650 Acked Read 4 Sent Done Done false
2836 1651 Acked Write 4 Sent Done Done false
2837 1652 Acked Flush 0 Sent Done Done false
2838 1653 Acked Read 4 Sent Done Done false
2839 1654 Acked Flush 0 Sent Done Done false
2840 1655 Acked Write 8 Sent Done Done false
2841 1656 Acked Read 3 Sent Done Done false
2842 1657 Acked Read 1 Sent Done Done false
2843 1658 Acked Flush 0 Sent Done Done false
2844 1659 Acked Write 1 Sent Done Done false
2845 1660 Acked Read 1 Sent Done Done false
2846 1661 Acked Write 3 Sent Done Done false
2847 1662 Acked Read 3 Sent Done Done false
2848 1663 Acked Write 7 Sent Done Done false
2849 1664 Acked Read 2 Sent Done Done false
2850 1665 Acked Read 3 Sent Done Done false
2851 1666 Acked Read 4 Sent Done Done false
2852 1667 Acked Read 8 Sent Done Done false
2853 1668 Acked Read 10 Sent Done Done false
2854 1669 Acked Flush 0 Sent Done Done false
2855 1670 Acked Read 5 Sent Done Done false
2856 1671 Acked Read 3 Sent Done Done false
2857 1672 Acked Read 10 Sent Done Done false
2858 1673 Acked Read 1 Sent Done Done false
2859 1674 Acked Read 6 Sent Done Done false
2860 1675 Acked Read 8 Sent Done Done false
2861 1676 Acked Read 4 Sent Done Done false
2862 1677 Acked Flush 0 Sent Done Done false
2863 1678 Acked Flush 0 Sent Done Done false
2864 1679 Acked Read 1 Sent Done Done false
2865 1680 Acked Read 6 Sent Done Done false
2866 1681 Acked Read 1 Sent Done Done false
2867 1682 Acked Read 9 Sent Done Done false
2868 1683 Acked Write 9 Sent Done Done false
2869 1684 Acked Read 2 Sent Done Done false
2870 1685 Acked Read 7 Sent Done Done false
2871 1686 Acked Read 1 Sent Done Done false
2872 1687 Acked Write 2 Sent Done Done false
2873 1688 Acked Read 2 Sent Done Done false
2874 1689 Acked Read 8 Sent Done Done false
2875 1690 Acked Write 4 Sent Done Done false
2876 1691 Acked Write 5 Sent Done Done false
2877 1692 Acked Flush 0 Sent Done Done false
2878 1693 Acked Read 1 Sent Done Done false
2879 1694 Acked Read 6 Sent Done Done false
2880 1695 Acked Read 5 Sent Done Done false
2881 1696 Acked Write 10 Sent Done Done false
2882 1697 Acked Flush 0 Sent Done Done false
2883 1698 Acked Write 10 Sent Done Done false
2884 1699 Acked Flush 0 Sent Done Done false
2885 1700 Acked Read 2 Sent Done Done false
2886 1701 Acked Read 7 Sent Done Done false
2887 1702 Acked Read 4 Sent Done Done false
2888 1703 Acked Write 10 Sent Done Done false
2889 1704 Acked Write 8 Sent Done Done false
2890 1705 Acked Read 7 Sent Done Done false
2891 1706 Acked Read 10 Sent Done Done false
2892 1707 Acked Read 3 Sent Done Done false
2893 1708 Acked Read 3 Sent Done Done false
2894 1709 Acked Read 9 Sent Done Done false
2895 1710 Acked Write 9 Sent Done Done false
2896 1711 Acked Read 6 Sent Done Done false
2897 1712 Acked Read 4 Sent Done Done false
2898 1713 Acked Read 3 Sent Done Done false
2899 1714 Acked Write 6 Sent Done Done false
2900 1715 Acked Write 10 Sent Done Done false
2901 1716 Acked Read 2 Sent Done Done false
2902 1717 Acked Read 2 Sent Done Done false
2903 1718 Acked Flush 0 Sent Done Done false
2904 1719 Acked Read 3 Sent Done Done false
2905 1720 Acked Write 5 Sent Done Done false
2906 1721 Acked Read 3 Sent Done Done false
2907 1722 Acked Read 9 Sent Done Done false
2908 1723 Acked Write 8 Sent Done Done false
2909 1724 Acked Write 1 Sent Done Done false
2910 1725 Acked Read 6 Sent Done Done false
2911 1726 Acked Read 7 Sent Done Done false
2912 1727 Acked Read 2 Sent Done Done false
2913 1728 Acked Write 1 Sent Done Done false
2914 1729 Acked Write 6 Sent Done Done false
2915 1730 Acked Flush 0 Sent Done Done false
2916 1731 Acked Write 1 Sent Done Done false
2917 1732 Acked Read 2 Sent Done Done false
2918 1733 Acked Write 4 Sent Done Done false
2919 1734 Acked Flush 0 Sent Done Done false
2920 1735 Acked Read 2 Sent Done Done false
2921 1736 Acked Read 5 Sent Done Done false
2922 1737 Acked Read 10 Sent Done Done false
2923 1738 Acked Read 5 Sent Done Done false
2924 1739 Acked Write 7 Sent Done Done false
2925 1740 Acked Write 3 Sent Done Done false
2926 1741 Acked Read 9 Sent Done Done false
2927 1742 Acked Read 3 Sent Done Done false
2928 1743 Acked Read 10 Sent Done Done false
2929 1744 Acked Write 3 Sent Done Done false
2930 1745 Acked Read 2 Sent Done Done false
2931 1746 Acked Flush 0 Sent Done Done false
2932 1747 Acked Read 7 Sent Done Done false
2933 1748 Acked Write 9 Sent Done Done false
2934 1749 Acked Write 9 Sent Done Done false
2935 1750 Acked Read 4 Sent Done Done false
2936 1751 Acked Read 5 Sent Done Done false
2937 1752 Acked Read 9 Sent Done Done false
2938 1753 Acked Write 2 Sent Done Done false
2939 1754 Acked Read 8 Sent Done Done false
2940 1755 Acked Read 6 Sent Done Done false
2941 1756 Acked Write 6 Sent Done Done false
2942 1757 Acked Read 5 Sent Done Done false
2943 1758 Acked Read 9 Sent Done Done false
2944 1759 Acked Read 8 Sent Done Done false
2945 1760 Acked Read 9 Sent Done Done false
2946 1761 Acked Write 7 Sent Done Done false
2947 1762 Acked Read 4 Sent Done Done false
2948 1763 Acked Write 3 Sent Done Done false
2949 1764 Acked Read 8 Sent Done Done false
2950 1765 Acked Read 8 Sent Done Done false
2951 1766 Acked Read 6 Sent Done Done false
2952 1767 Acked Flush 0 Sent Done Done false
2953 1768 Acked Read 1 Sent Done Done false
2954 1769 Acked Write 8 Sent Done Done false
2955 1770 Acked Write 10 Sent Done Done false
2956 1771 Acked Write 4 Sent Done Done false
2957 1772 Acked Write 7 Sent Done Done false
2958 1773 Acked Write 10 Sent Done Done false
2959 1774 Acked Write 5 Sent Done Done false
2960 1775 Acked Write 4 Sent Done Done false
2961 1776 Acked Write 10 Sent Done Done false
2962 1777 Acked Read 4 Sent Done Done false
2963 1778 Acked Write 1 Sent Done Done false
2964 1779 Acked Read 5 Sent Done Done false
2965 1780 Acked Write 5 Sent Done Done false
2966 1781 Acked Write 9 Sent Done Done false
2967 1782 Acked Read 10 Sent Done Done false
2968 1783 Acked Write 6 Sent Done Done false
2969 1784 Acked Write 5 Sent Done Done false
2970 1785 Acked Flush 0 Sent Done Done false
2971 1786 Acked Flush 0 Sent Done Done false
2972 1787 Acked Flush 0 Sent Done Done false
2973 1788 Acked Flush 0 Sent Done Done false
2974 1789 Acked Write 6 Sent Done Done false
2975 1790 Acked Read 8 Sent Done Done false
2976 1791 Acked Write 2 Sent Done Done false
2977 1792 Acked Read 1 Sent Done Done false
2978 1793 Acked Write 5 Sent Done Done false
2979 1794 Acked Write 4 Sent Done Done false
2980 1795 Acked Read 5 Sent Done Done false
2981 1796 Acked Read 9 Sent Done Done false
2982 1797 Acked Read 9 Sent Done Done false
2983 1798 Acked Flush 0 Sent Done Done false
2984 1799 Acked Read 5 Sent Done Done false
2985 1800 Acked Read 10 Sent Done Done false
2986 1801 Acked Read 9 Sent Done Done false
2987 1802 Acked Read 3 Sent Done Done false
2988 1803 Acked Read 9 Sent Done Done false
2989 1804 Acked Read 4 Sent Done Done false
2990 1805 Acked Read 10 Sent Done Done false
2991 1806 Acked Write 3 Sent Done Done false
2992 1807 Acked Read 6 Sent Done Done false
2993 1808 Acked Read 10 Sent Done Done false
2994 1809 Acked Write 1 Sent Done Done false
2995 1810 Acked Write 8 Sent Done Done false
2996 1811 Acked Read 3 Sent Done Done false
2997 1812 Acked Read 3 Sent Done Done false
2998 1813 Acked Write 2 Sent Done Done false
2999 1814 Acked Read 5 Sent Done Done false
3000 1815 Acked Read 7 Sent Done Done false
3001 1816 Acked Write 7 Sent Done Done false
3002 1817 Acked Read 10 Sent Done Done false
3003 1818 Acked Write 5 Sent Done Done false
3004 1819 Acked Write 5 Sent Done Done false
3005 1820 Acked Write 6 Sent Done Done false
3006 1821 Acked Read 5 Sent Done Done false
3007 1822 Acked Read 3 Sent Done Done false
3008 1823 Acked Write 1 Sent Done Done false
3009 1824 Acked Flush 0 Sent Done Done false
3010 1825 Acked Write 2 Sent Done Done false
3011 1826 Acked Read 6 Sent Done Done false
3012 1827 Acked Read 8 Sent Done Done false
3013 1828 Acked Read 10 Sent Done Done false
3014 1829 Acked Write 4 Sent Done Done false
3015 1830 Acked Write 10 Sent Done Done false
3016 1831 Acked Write 1 Sent Done Done false
3017 1832 Acked Read 8 Sent Done Done false
3018 1833 Acked Read 1 Sent Done Done false
3019 1834 Acked Read 8 Sent Done Done false
3020 1835 Acked Write 5 Sent Done Done false
3021 1836 Acked Read 8 Sent Done Done false
3022 1837 Acked Write 1 Sent Done Done false
3023 1838 Acked Read 2 Sent Done Done false
3024 1839 Acked Read 5 Sent Done Done false
3025 1840 Acked Read 8 Sent Done Done false
3026 1841 Acked Write 9 Sent Done Done false
3027 1842 Acked Read 2 Sent Done Done false
3028 1843 Acked Read 2 Sent Done Done false
3029 1844 Acked Flush 0 Sent Done Done false
3030 1845 Acked Read 5 Sent Done Done false
3031 1846 Acked Write 1 Sent Done Done false
3032 1847 Acked Write 6 Sent Done Done false
3033 1848 Acked Read 1 Sent Done Done false
3034 1849 Acked Write 2 Sent Done Done false
3035 1850 Acked Read 4 Sent Done Done false
3036 1851 Acked Read 3 Sent Done Done false
3037 1852 Acked Write 1 Sent Done Done false
3038 1853 Acked Read 8 Sent Done Done false
3039 1854 Acked Flush 0 Sent Done Done false
3040 1855 Acked Read 3 Sent Done Done false
3041 1856 Acked Read 10 Sent Done Done false
3042 1857 Acked Read 6 Sent Done Done false
3043 1858 Acked Write 6 Sent Done Done false
3044 1859 Acked Write 10 Sent Done Done false
3045 1860 Acked Write 5 Sent Done Done false
3046 1861 Acked Write 8 Sent Done Done false
3047 1862 Acked Read 4 Sent Done Done false
3048 1863 Acked Read 9 Sent Done Done false
3049 1864 Acked Read 8 Sent Done Done false
3050 1865 Acked Read 3 Sent Done Done false
3051 1866 Acked Write 6 Sent Done Done false
3052 1867 Acked Write 10 Sent Done Done false
3053 1868 Acked Read 4 Sent Done Done false
3054 1869 Acked Write 5 Sent Done Done false
3055 1870 Acked Read 9 Sent Done Done false
3056 1871 Acked Write 4 Sent Done Done false
3057 1872 Acked Write 6 Sent Done Done false
3058 1873 Acked Write 8 Sent Done Done false
3059 1874 Acked Read 5 Sent Done Done false
3060 1875 Acked Read 4 Sent Done Done false
3061 1876 Acked Write 9 Sent Done Done false
3062 1877 Acked Write 2 Sent Done Done false
3063 1878 Acked Write 10 Sent Done Done false
3064 1879 Acked Read 4 Sent Done Done false
3065 1880 Acked Read 4 Sent Done Done false
3066 1881 Acked Write 2 Sent Done Done false
3067 1882 Acked Read 7 Sent Done Done false
3068 1883 Acked Read 7 Sent Done Done false
3069 1884 Acked Write 6 Sent Done Done false
3070 1885 Acked Write 1 Sent Done Done false
3071 1886 Acked Write 7 Sent Done Done false
3072 1887 Acked Flush 0 Sent Done Done false
3073 1888 Acked Read 1 Sent Done Done false
3074 1889 Acked Write 6 Sent Done Done false
3075 1890 Acked Read 3 Sent Done Done false
3076 1891 Acked Write 2 Sent Done Done false
3077 1892 Acked Write 7 Sent Done Done false
3078 1893 Acked Write 4 Sent Done Done false
3079 1894 Acked Read 4 Sent Done Done false
3080 1895 Acked Read 4 Sent Done Done false
3081 1896 Acked Read 9 Sent Done Done false
3082 1897 Acked Read 4 Sent Done Done false
3083 1898 Acked Read 6 Sent Done Done false
3084 1899 Acked Read 5 Sent Done Done false
3085 1900 Acked Read 1 Sent Done Done false
3086 1901 Acked Read 10 Sent Done Done false
3087 1902 Acked Read 5 Sent Done Done false
3088 1903 Acked Read 6 Sent Done Done false
3089 1904 Acked Write 2 Sent Done Done false
3090 1905 Acked Flush 0 Sent Done Done false
3091 STATES DS:0 DS:1 DS:2 TOTAL
3092 Sent 302 0 0 302
3093 Done 0 302 302 604
3094 Skipped 0 0 0 0
3095 Error 0 0 0 0
3096 Last Flush: Some(JobId(1603)) Some(JobId(1905)) Some(JobId(1905))
3097 Downstairs last five completed: 1603 1602 1601 1600 1599
3098 Upstairs last five completed: 1905 1904 1903 1902 1901
3099 CLIENT: Up:0 ds:302 act:2
31002025-01-23T17:38:22.115ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
31012025-01-23T17:38:22.115ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
31022025-01-23T17:38:22.115ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
31032025-01-23T17:38:22.115ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
31042025-01-23T17:38:22.115ZINFOcrucible: client task is exiting = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
3105 [8810] got start action
3106 [8810] I am going to restart
3107 Starting downstairs at port 8810
3108 Make output file at "/tmp/dsc/downstairs-8810.txt"
3109 [8810][0] reports Starting
3110 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1208)
3111 [8810][0] reports Running
3112 ----------------------------------------------------------------
3113 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
3114 States: Offline Active Active
3115 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3116 1604 Acked Write 9 Sent Done Done false
3117 1605 Acked Read 5 Sent Done Done false
3118 1606 Acked Write 4 Sent Done Done false
3119 1607 Acked Write 6 Sent Done Done false
3120 1608 Acked Write 8 Sent Done Done false
3121 1609 Acked Write 2 Sent Done Done false
3122 1610 Acked Write 6 Sent Done Done false
3123 1611 Acked Read 9 Sent Done Done false
3124 1612 Acked Read 10 Sent Done Done false
3125 1613 Acked Write 5 Sent Done Done false
3126 1614 Acked Read 5 Sent Done Done false
3127 1615 Acked Read 1 Sent Done Done false
3128 1616 Acked Read 5 Sent Done Done false
3129 1617 Acked Read 10 Sent Done Done false
3130 1618 Acked Flush 0 Sent Done Done false
3131 1619 Acked Write 7 Sent Done Done false
3132 1620 Acked Read 3 Sent Done Done false
3133 1621 Acked Read 8 Sent Done Done false
3134 1622 Acked Read 5 Sent Done Done false
3135 1623 Acked Read 10 Sent Done Done false
3136 1624 Acked Write 6 Sent Done Done false
3137 1625 Acked Write 8 Sent Done Done false
3138 1626 Acked Write 5 Sent Done Done false
3139 1627 Acked Read 6 Sent Done Done false
3140 1628 Acked Write 1 Sent Done Done false
3141 1629 Acked Read 8 Sent Done Done false
3142 1630 Acked Flush 0 Sent Done Done false
3143 1631 Acked Read 10 Sent Done Done false
3144 1632 Acked Write 9 Sent Done Done false
3145 1633 Acked Write 4 Sent Done Done false
3146 1634 Acked Write 3 Sent Done Done false
3147 1635 Acked Read 4 Sent Done Done false
3148 1636 Acked Flush 0 Sent Done Done false
3149 1637 Acked Read 4 Sent Done Done false
3150 1638 Acked Write 7 Sent Done Done false
3151 1639 Acked Write 4 Sent Done Done false
3152 1640 Acked Flush 0 Sent Done Done false
3153 1641 Acked Read 10 Sent Done Done false
3154 1642 Acked Read 7 Sent Done Done false
3155 1643 Acked Read 5 Sent Done Done false
3156 1644 Acked Read 10 Sent Done Done false
3157 1645 Acked Read 5 Sent Done Done false
3158 1646 Acked Read 4 Sent Done Done false
3159 1647 Acked Write 10 Sent Done Done false
3160 1648 Acked Flush 0 Sent Done Done false
3161 1649 Acked Read 10 Sent Done Done false
3162 1650 Acked Read 4 Sent Done Done false
3163 1651 Acked Write 4 Sent Done Done false
3164 1652 Acked Flush 0 Sent Done Done false
3165 1653 Acked Read 4 Sent Done Done false
3166 1654 Acked Flush 0 Sent Done Done false
3167 1655 Acked Write 8 Sent Done Done false
3168 1656 Acked Read 3 Sent Done Done false
3169 1657 Acked Read 1 Sent Done Done false
3170 1658 Acked Flush 0 Sent Done Done false
3171 1659 Acked Write 1 Sent Done Done false
3172 1660 Acked Read 1 Sent Done Done false
3173 1661 Acked Write 3 Sent Done Done false
3174 1662 Acked Read 3 Sent Done Done false
3175 1663 Acked Write 7 Sent Done Done false
3176 1664 Acked Read 2 Sent Done Done false
3177 1665 Acked Read 3 Sent Done Done false
3178 1666 Acked Read 4 Sent Done Done false
3179 1667 Acked Read 8 Sent Done Done false
3180 1668 Acked Read 10 Sent Done Done false
3181 1669 Acked Flush 0 Sent Done Done false
3182 1670 Acked Read 5 Sent Done Done false
3183 1671 Acked Read 3 Sent Done Done false
3184 1672 Acked Read 10 Sent Done Done false
3185 1673 Acked Read 1 Sent Done Done false
3186 1674 Acked Read 6 Sent Done Done false
3187 1675 Acked Read 8 Sent Done Done false
3188 1676 Acked Read 4 Sent Done Done false
3189 1677 Acked Flush 0 Sent Done Done false
3190 1678 Acked Flush 0 Sent Done Done false
3191 1679 Acked Read 1 Sent Done Done false
3192 1680 Acked Read 6 Sent Done Done false
3193 1681 Acked Read 1 Sent Done Done false
3194 1682 Acked Read 9 Sent Done Done false
3195 1683 Acked Write 9 Sent Done Done false
3196 1684 Acked Read 2 Sent Done Done false
3197 1685 Acked Read 7 Sent Done Done false
3198 1686 Acked Read 1 Sent Done Done false
3199 1687 Acked Write 2 Sent Done Done false
3200 1688 Acked Read 2 Sent Done Done false
3201 1689 Acked Read 8 Sent Done Done false
3202 1690 Acked Write 4 Sent Done Done false
3203 1691 Acked Write 5 Sent Done Done false
3204 1692 Acked Flush 0 Sent Done Done false
3205 1693 Acked Read 1 Sent Done Done false
3206 1694 Acked Read 6 Sent Done Done false
3207 1695 Acked Read 5 Sent Done Done false
3208 1696 Acked Write 10 Sent Done Done false
3209 1697 Acked Flush 0 Sent Done Done false
3210 1698 Acked Write 10 Sent Done Done false
3211 1699 Acked Flush 0 Sent Done Done false
3212 1700 Acked Read 2 Sent Done Done false
3213 1701 Acked Read 7 Sent Done Done false
3214 1702 Acked Read 4 Sent Done Done false
3215 1703 Acked Write 10 Sent Done Done false
3216 1704 Acked Write 8 Sent Done Done false
3217 1705 Acked Read 7 Sent Done Done false
3218 1706 Acked Read 10 Sent Done Done false
3219 1707 Acked Read 3 Sent Done Done false
3220 1708 Acked Read 3 Sent Done Done false
3221 1709 Acked Read 9 Sent Done Done false
3222 1710 Acked Write 9 Sent Done Done false
3223 1711 Acked Read 6 Sent Done Done false
3224 1712 Acked Read 4 Sent Done Done false
3225 1713 Acked Read 3 Sent Done Done false
3226 1714 Acked Write 6 Sent Done Done false
3227 1715 Acked Write 10 Sent Done Done false
3228 1716 Acked Read 2 Sent Done Done false
3229 1717 Acked Read 2 Sent Done Done false
3230 1718 Acked Flush 0 Sent Done Done false
3231 1719 Acked Read 3 Sent Done Done false
3232 1720 Acked Write 5 Sent Done Done false
3233 1721 Acked Read 3 Sent Done Done false
3234 1722 Acked Read 9 Sent Done Done false
3235 1723 Acked Write 8 Sent Done Done false
3236 1724 Acked Write 1 Sent Done Done false
3237 1725 Acked Read 6 Sent Done Done false
3238 1726 Acked Read 7 Sent Done Done false
3239 1727 Acked Read 2 Sent Done Done false
3240 1728 Acked Write 1 Sent Done Done false
3241 1729 Acked Write 6 Sent Done Done false
3242 1730 Acked Flush 0 Sent Done Done false
3243 1731 Acked Write 1 Sent Done Done false
3244 1732 Acked Read 2 Sent Done Done false
3245 1733 Acked Write 4 Sent Done Done false
3246 1734 Acked Flush 0 Sent Done Done false
3247 1735 Acked Read 2 Sent Done Done false
3248 1736 Acked Read 5 Sent Done Done false
3249 1737 Acked Read 10 Sent Done Done false
3250 1738 Acked Read 5 Sent Done Done false
3251 1739 Acked Write 7 Sent Done Done false
3252 1740 Acked Write 3 Sent Done Done false
3253 1741 Acked Read 9 Sent Done Done false
3254 1742 Acked Read 3 Sent Done Done false
3255 1743 Acked Read 10 Sent Done Done false
3256 1744 Acked Write 3 Sent Done Done false
3257 1745 Acked Read 2 Sent Done Done false
3258 1746 Acked Flush 0 Sent Done Done false
3259 1747 Acked Read 7 Sent Done Done false
3260 1748 Acked Write 9 Sent Done Done false
3261 1749 Acked Write 9 Sent Done Done false
3262 1750 Acked Read 4 Sent Done Done false
3263 1751 Acked Read 5 Sent Done Done false
3264 1752 Acked Read 9 Sent Done Done false
3265 1753 Acked Write 2 Sent Done Done false
3266 1754 Acked Read 8 Sent Done Done false
3267 1755 Acked Read 6 Sent Done Done false
3268 1756 Acked Write 6 Sent Done Done false
3269 1757 Acked Read 5 Sent Done Done false
3270 1758 Acked Read 9 Sent Done Done false
3271 1759 Acked Read 8 Sent Done Done false
3272 1760 Acked Read 9 Sent Done Done false
3273 1761 Acked Write 7 Sent Done Done false
3274 1762 Acked Read 4 Sent Done Done false
3275 1763 Acked Write 3 Sent Done Done false
3276 1764 Acked Read 8 Sent Done Done false
3277 1765 Acked Read 8 Sent Done Done false
3278 1766 Acked Read 6 Sent Done Done false
3279 1767 Acked Flush 0 Sent Done Done false
3280 1768 Acked Read 1 Sent Done Done false
3281 1769 Acked Write 8 Sent Done Done false
3282 1770 Acked Write 10 Sent Done Done false
3283 1771 Acked Write 4 Sent Done Done false
3284 1772 Acked Write 7 Sent Done Done false
3285 1773 Acked Write 10 Sent Done Done false
3286 1774 Acked Write 5 Sent Done Done false
3287 1775 Acked Write 4 Sent Done Done false
3288 1776 Acked Write 10 Sent Done Done false
3289 1777 Acked Read 4 Sent Done Done false
3290 1778 Acked Write 1 Sent Done Done false
3291 1779 Acked Read 5 Sent Done Done false
3292 1780 Acked Write 5 Sent Done Done false
3293 1781 Acked Write 9 Sent Done Done false
3294 1782 Acked Read 10 Sent Done Done false
3295 1783 Acked Write 6 Sent Done Done false
3296 1784 Acked Write 5 Sent Done Done false
3297 1785 Acked Flush 0 Sent Done Done false
3298 1786 Acked Flush 0 Sent Done Done false
3299 1787 Acked Flush 0 Sent Done Done false
3300 1788 Acked Flush 0 Sent Done Done false
3301 1789 Acked Write 6 Sent Done Done false
3302 1790 Acked Read 8 Sent Done Done false
3303 1791 Acked Write 2 Sent Done Done false
3304 1792 Acked Read 1 Sent Done Done false
3305 1793 Acked Write 5 Sent Done Done false
3306 1794 Acked Write 4 Sent Done Done false
3307 1795 Acked Read 5 Sent Done Done false
3308 1796 Acked Read 9 Sent Done Done false
3309 1797 Acked Read 9 Sent Done Done false
3310 1798 Acked Flush 0 Sent Done Done false
3311 1799 Acked Read 5 Sent Done Done false
3312 1800 Acked Read 10 Sent Done Done false
3313 1801 Acked Read 9 Sent Done Done false
3314 1802 Acked Read 3 Sent Done Done false
3315 1803 Acked Read 9 Sent Done Done false
3316 1804 Acked Read 4 Sent Done Done false
3317 1805 Acked Read 10 Sent Done Done false
3318 1806 Acked Write 3 Sent Done Done false
3319 1807 Acked Read 6 Sent Done Done false
3320 1808 Acked Read 10 Sent Done Done false
3321 1809 Acked Write 1 Sent Done Done false
3322 1810 Acked Write 8 Sent Done Done false
3323 1811 Acked Read 3 Sent Done Done false
3324 1812 Acked Read 3 Sent Done Done false
3325 1813 Acked Write 2 Sent Done Done false
3326 1814 Acked Read 5 Sent Done Done false
3327 1815 Acked Read 7 Sent Done Done false
3328 1816 Acked Write 7 Sent Done Done false
3329 1817 Acked Read 10 Sent Done Done false
3330 1818 Acked Write 5 Sent Done Done false
3331 1819 Acked Write 5 Sent Done Done false
3332 1820 Acked Write 6 Sent Done Done false
3333 1821 Acked Read 5 Sent Done Done false
3334 1822 Acked Read 3 Sent Done Done false
3335 1823 Acked Write 1 Sent Done Done false
3336 1824 Acked Flush 0 Sent Done Done false
3337 1825 Acked Write 2 Sent Done Done false
3338 1826 Acked Read 6 Sent Done Done false
3339 1827 Acked Read 8 Sent Done Done false
3340 1828 Acked Read 10 Sent Done Done false
3341 1829 Acked Write 4 Sent Done Done false
3342 1830 Acked Write 10 Sent Done Done false
3343 1831 Acked Write 1 Sent Done Done false
3344 1832 Acked Read 8 Sent Done Done false
3345 1833 Acked Read 1 Sent Done Done false
3346 1834 Acked Read 8 Sent Done Done false
3347 1835 Acked Write 5 Sent Done Done false
3348 1836 Acked Read 8 Sent Done Done false
3349 1837 Acked Write 1 Sent Done Done false
3350 1838 Acked Read 2 Sent Done Done false
3351 1839 Acked Read 5 Sent Done Done false
3352 1840 Acked Read 8 Sent Done Done false
3353 1841 Acked Write 9 Sent Done Done false
3354 1842 Acked Read 2 Sent Done Done false
3355 1843 Acked Read 2 Sent Done Done false
3356 1844 Acked Flush 0 Sent Done Done false
3357 1845 Acked Read 5 Sent Done Done false
3358 1846 Acked Write 1 Sent Done Done false
3359 1847 Acked Write 6 Sent Done Done false
3360 1848 Acked Read 1 Sent Done Done false
3361 1849 Acked Write 2 Sent Done Done false
3362 1850 Acked Read 4 Sent Done Done false
3363 1851 Acked Read 3 Sent Done Done false
3364 1852 Acked Write 1 Sent Done Done false
3365 1853 Acked Read 8 Sent Done Done false
3366 1854 Acked Flush 0 Sent Done Done false
3367 1855 Acked Read 3 Sent Done Done false
3368 1856 Acked Read 10 Sent Done Done false
3369 1857 Acked Read 6 Sent Done Done false
3370 1858 Acked Write 6 Sent Done Done false
3371 1859 Acked Write 10 Sent Done Done false
3372 1860 Acked Write 5 Sent Done Done false
3373 1861 Acked Write 8 Sent Done Done false
3374 1862 Acked Read 4 Sent Done Done false
3375 1863 Acked Read 9 Sent Done Done false
3376 1864 Acked Read 8 Sent Done Done false
3377 1865 Acked Read 3 Sent Done Done false
3378 1866 Acked Write 6 Sent Done Done false
3379 1867 Acked Write 10 Sent Done Done false
3380 1868 Acked Read 4 Sent Done Done false
3381 1869 Acked Write 5 Sent Done Done false
3382 1870 Acked Read 9 Sent Done Done false
3383 1871 Acked Write 4 Sent Done Done false
3384 1872 Acked Write 6 Sent Done Done false
3385 1873 Acked Write 8 Sent Done Done false
3386 1874 Acked Read 5 Sent Done Done false
3387 1875 Acked Read 4 Sent Done Done false
3388 1876 Acked Write 9 Sent Done Done false
3389 1877 Acked Write 2 Sent Done Done false
3390 1878 Acked Write 10 Sent Done Done false
3391 1879 Acked Read 4 Sent Done Done false
3392 1880 Acked Read 4 Sent Done Done false
3393 1881 Acked Write 2 Sent Done Done false
3394 1882 Acked Read 7 Sent Done Done false
3395 1883 Acked Read 7 Sent Done Done false
3396 1884 Acked Write 6 Sent Done Done false
3397 1885 Acked Write 1 Sent Done Done false
3398 1886 Acked Write 7 Sent Done Done false
3399 1887 Acked Flush 0 Sent Done Done false
3400 1888 Acked Read 1 Sent Done Done false
3401 1889 Acked Write 6 Sent Done Done false
3402 1890 Acked Read 3 Sent Done Done false
3403 1891 Acked Write 2 Sent Done Done false
3404 1892 Acked Write 7 Sent Done Done false
3405 1893 Acked Write 4 Sent Done Done false
3406 1894 Acked Read 4 Sent Done Done false
3407 1895 Acked Read 4 Sent Done Done false
3408 1896 Acked Read 9 Sent Done Done false
3409 1897 Acked Read 4 Sent Done Done false
3410 1898 Acked Read 6 Sent Done Done false
3411 1899 Acked Read 5 Sent Done Done false
3412 1900 Acked Read 1 Sent Done Done false
3413 1901 Acked Read 10 Sent Done Done false
3414 1902 Acked Read 5 Sent Done Done false
3415 1903 Acked Read 6 Sent Done Done false
3416 1904 Acked Write 2 Sent Done Done false
3417 1905 Acked Flush 0 Sent Done Done false
3418 STATES DS:0 DS:1 DS:2 TOTAL
3419 Sent 302 0 0 302
3420 Done 0 302 302 604
3421 Skipped 0 0 0 0
3422 Error 0 0 0 0
3423 Last Flush: Some(JobId(1603)) Some(JobId(1905)) Some(JobId(1905))
3424 Downstairs last five completed: 1603 1602 1601 1600 1599
3425 Upstairs last five completed: 1905 1904 1903 1902 1901
3426 CLIENT: Up:0 ds:302 act:2
3427 ----------------------------------------------------------------
3428 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
3429 States: Offline Active Active
3430 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3431 1604 Acked Write 9 Sent Done Done false
3432 1605 Acked Read 5 Sent Done Done false
3433 1606 Acked Write 4 Sent Done Done false
3434 1607 Acked Write 6 Sent Done Done false
3435 1608 Acked Write 8 Sent Done Done false
3436 1609 Acked Write 2 Sent Done Done false
3437 1610 Acked Write 6 Sent Done Done false
3438 1611 Acked Read 9 Sent Done Done false
3439 1612 Acked Read 10 Sent Done Done false
3440 1613 Acked Write 5 Sent Done Done false
3441 1614 Acked Read 5 Sent Done Done false
3442 1615 Acked Read 1 Sent Done Done false
3443 1616 Acked Read 5 Sent Done Done false
3444 1617 Acked Read 10 Sent Done Done false
3445 1618 Acked Flush 0 Sent Done Done false
3446 1619 Acked Write 7 Sent Done Done false
3447 1620 Acked Read 3 Sent Done Done false
3448 1621 Acked Read 8 Sent Done Done false
3449 1622 Acked Read 5 Sent Done Done false
3450 1623 Acked Read 10 Sent Done Done false
3451 1624 Acked Write 6 Sent Done Done false
3452 1625 Acked Write 8 Sent Done Done false
3453 1626 Acked Write 5 Sent Done Done false
3454 1627 Acked Read 6 Sent Done Done false
3455 1628 Acked Write 1 Sent Done Done false
3456 1629 Acked Read 8 Sent Done Done false
3457 1630 Acked Flush 0 Sent Done Done false
3458 1631 Acked Read 10 Sent Done Done false
3459 1632 Acked Write 9 Sent Done Done false
3460 1633 Acked Write 4 Sent Done Done false
3461 1634 Acked Write 3 Sent Done Done false
3462 1635 Acked Read 4 Sent Done Done false
3463 1636 Acked Flush 0 Sent Done Done false
3464 1637 Acked Read 4 Sent Done Done false
3465 1638 Acked Write 7 Sent Done Done false
3466 1639 Acked Write 4 Sent Done Done false
3467 1640 Acked Flush 0 Sent Done Done false
3468 1641 Acked Read 10 Sent Done Done false
3469 1642 Acked Read 7 Sent Done Done false
3470 1643 Acked Read 5 Sent Done Done false
3471 1644 Acked Read 10 Sent Done Done false
3472 1645 Acked Read 5 Sent Done Done false
3473 1646 Acked Read 4 Sent Done Done false
3474 1647 Acked Write 10 Sent Done Done false
3475 1648 Acked Flush 0 Sent Done Done false
3476 1649 Acked Read 10 Sent Done Done false
3477 1650 Acked Read 4 Sent Done Done false
3478 1651 Acked Write 4 Sent Done Done false
3479 1652 Acked Flush 0 Sent Done Done false
3480 1653 Acked Read 4 Sent Done Done false
3481 1654 Acked Flush 0 Sent Done Done false
3482 1655 Acked Write 8 Sent Done Done false
3483 1656 Acked Read 3 Sent Done Done false
3484 1657 Acked Read 1 Sent Done Done false
3485 1658 Acked Flush 0 Sent Done Done false
3486 1659 Acked Write 1 Sent Done Done false
3487 1660 Acked Read 1 Sent Done Done false
3488 1661 Acked Write 3 Sent Done Done false
3489 1662 Acked Read 3 Sent Done Done false
3490 1663 Acked Write 7 Sent Done Done false
3491 1664 Acked Read 2 Sent Done Done false
3492 1665 Acked Read 3 Sent Done Done false
3493 1666 Acked Read 4 Sent Done Done false
3494 1667 Acked Read 8 Sent Done Done false
3495 1668 Acked Read 10 Sent Done Done false
3496 1669 Acked Flush 0 Sent Done Done false
3497 1670 Acked Read 5 Sent Done Done false
3498 1671 Acked Read 3 Sent Done Done false
3499 1672 Acked Read 10 Sent Done Done false
3500 1673 Acked Read 1 Sent Done Done false
3501 1674 Acked Read 6 Sent Done Done false
3502 1675 Acked Read 8 Sent Done Done false
3503 1676 Acked Read 4 Sent Done Done false
3504 1677 Acked Flush 0 Sent Done Done false
3505 1678 Acked Flush 0 Sent Done Done false
3506 1679 Acked Read 1 Sent Done Done false
3507 1680 Acked Read 6 Sent Done Done false
3508 1681 Acked Read 1 Sent Done Done false
3509 1682 Acked Read 9 Sent Done Done false
3510 1683 Acked Write 9 Sent Done Done false
3511 1684 Acked Read 2 Sent Done Done false
3512 1685 Acked Read 7 Sent Done Done false
3513 1686 Acked Read 1 Sent Done Done false
3514 1687 Acked Write 2 Sent Done Done false
3515 1688 Acked Read 2 Sent Done Done false
3516 1689 Acked Read 8 Sent Done Done false
3517 1690 Acked Write 4 Sent Done Done false
3518 1691 Acked Write 5 Sent Done Done false
3519 1692 Acked Flush 0 Sent Done Done false
3520 1693 Acked Read 1 Sent Done Done false
3521 1694 Acked Read 6 Sent Done Done false
3522 1695 Acked Read 5 Sent Done Done false
3523 1696 Acked Write 10 Sent Done Done false
3524 1697 Acked Flush 0 Sent Done Done false
3525 1698 Acked Write 10 Sent Done Done false
3526 1699 Acked Flush 0 Sent Done Done false
3527 1700 Acked Read 2 Sent Done Done false
3528 1701 Acked Read 7 Sent Done Done false
3529 1702 Acked Read 4 Sent Done Done false
3530 1703 Acked Write 10 Sent Done Done false
3531 1704 Acked Write 8 Sent Done Done false
3532 1705 Acked Read 7 Sent Done Done false
3533 1706 Acked Read 10 Sent Done Done false
3534 1707 Acked Read 3 Sent Done Done false
3535 1708 Acked Read 3 Sent Done Done false
3536 1709 Acked Read 9 Sent Done Done false
3537 1710 Acked Write 9 Sent Done Done false
3538 1711 Acked Read 6 Sent Done Done false
3539 1712 Acked Read 4 Sent Done Done false
3540 1713 Acked Read 3 Sent Done Done false
3541 1714 Acked Write 6 Sent Done Done false
3542 1715 Acked Write 10 Sent Done Done false
3543 1716 Acked Read 2 Sent Done Done false
3544 1717 Acked Read 2 Sent Done Done false
3545 1718 Acked Flush 0 Sent Done Done false
3546 1719 Acked Read 3 Sent Done Done false
3547 1720 Acked Write 5 Sent Done Done false
3548 1721 Acked Read 3 Sent Done Done false
3549 1722 Acked Read 9 Sent Done Done false
3550 1723 Acked Write 8 Sent Done Done false
3551 1724 Acked Write 1 Sent Done Done false
3552 1725 Acked Read 6 Sent Done Done false
3553 1726 Acked Read 7 Sent Done Done false
3554 1727 Acked Read 2 Sent Done Done false
3555 1728 Acked Write 1 Sent Done Done false
3556 1729 Acked Write 6 Sent Done Done false
3557 1730 Acked Flush 0 Sent Done Done false
3558 1731 Acked Write 1 Sent Done Done false
3559 1732 Acked Read 2 Sent Done Done false
3560 1733 Acked Write 4 Sent Done Done false
3561 1734 Acked Flush 0 Sent Done Done false
3562 1735 Acked Read 2 Sent Done Done false
3563 1736 Acked Read 5 Sent Done Done false
3564 1737 Acked Read 10 Sent Done Done false
3565 1738 Acked Read 5 Sent Done Done false
3566 1739 Acked Write 7 Sent Done Done false
3567 1740 Acked Write 3 Sent Done Done false
3568 1741 Acked Read 9 Sent Done Done false
3569 1742 Acked Read 3 Sent Done Done false
3570 1743 Acked Read 10 Sent Done Done false
3571 1744 Acked Write 3 Sent Done Done false
3572 1745 Acked Read 2 Sent Done Done false
3573 1746 Acked Flush 0 Sent Done Done false
3574 1747 Acked Read 7 Sent Done Done false
3575 1748 Acked Write 9 Sent Done Done false
3576 1749 Acked Write 9 Sent Done Done false
3577 1750 Acked Read 4 Sent Done Done false
3578 1751 Acked Read 5 Sent Done Done false
3579 1752 Acked Read 9 Sent Done Done false
3580 1753 Acked Write 2 Sent Done Done false
3581 1754 Acked Read 8 Sent Done Done false
3582 1755 Acked Read 6 Sent Done Done false
3583 1756 Acked Write 6 Sent Done Done false
3584 1757 Acked Read 5 Sent Done Done false
3585 1758 Acked Read 9 Sent Done Done false
3586 1759 Acked Read 8 Sent Done Done false
3587 1760 Acked Read 9 Sent Done Done false
3588 1761 Acked Write 7 Sent Done Done false
3589 1762 Acked Read 4 Sent Done Done false
3590 1763 Acked Write 3 Sent Done Done false
3591 1764 Acked Read 8 Sent Done Done false
3592 1765 Acked Read 8 Sent Done Done false
3593 1766 Acked Read 6 Sent Done Done false
3594 1767 Acked Flush 0 Sent Done Done false
3595 1768 Acked Read 1 Sent Done Done false
3596 1769 Acked Write 8 Sent Done Done false
3597 1770 Acked Write 10 Sent Done Done false
3598 1771 Acked Write 4 Sent Done Done false
3599 1772 Acked Write 7 Sent Done Done false
3600 1773 Acked Write 10 Sent Done Done false
3601 1774 Acked Write 5 Sent Done Done false
3602 1775 Acked Write 4 Sent Done Done false
3603 1776 Acked Write 10 Sent Done Done false
3604 1777 Acked Read 4 Sent Done Done false
3605 1778 Acked Write 1 Sent Done Done false
3606 1779 Acked Read 5 Sent Done Done false
3607 1780 Acked Write 5 Sent Done Done false
3608 1781 Acked Write 9 Sent Done Done false
3609 1782 Acked Read 10 Sent Done Done false
3610 1783 Acked Write 6 Sent Done Done false
3611 1784 Acked Write 5 Sent Done Done false
3612 1785 Acked Flush 0 Sent Done Done false
3613 1786 Acked Flush 0 Sent Done Done false
3614 1787 Acked Flush 0 Sent Done Done false
3615 1788 Acked Flush 0 Sent Done Done false
3616 1789 Acked Write 6 Sent Done Done false
3617 1790 Acked Read 8 Sent Done Done false
3618 1791 Acked Write 2 Sent Done Done false
3619 1792 Acked Read 1 Sent Done Done false
3620 1793 Acked Write 5 Sent Done Done false
3621 1794 Acked Write 4 Sent Done Done false
3622 1795 Acked Read 5 Sent Done Done false
3623 1796 Acked Read 9 Sent Done Done false
3624 1797 Acked Read 9 Sent Done Done false
3625 1798 Acked Flush 0 Sent Done Done false
3626 1799 Acked Read 5 Sent Done Done false
3627 1800 Acked Read 10 Sent Done Done false
3628 1801 Acked Read 9 Sent Done Done false
3629 1802 Acked Read 3 Sent Done Done false
3630 1803 Acked Read 9 Sent Done Done false
3631 1804 Acked Read 4 Sent Done Done false
3632 1805 Acked Read 10 Sent Done Done false
3633 1806 Acked Write 3 Sent Done Done false
3634 1807 Acked Read 6 Sent Done Done false
3635 1808 Acked Read 10 Sent Done Done false
3636 1809 Acked Write 1 Sent Done Done false
3637 1810 Acked Write 8 Sent Done Done false
3638 1811 Acked Read 3 Sent Done Done false
3639 1812 Acked Read 3 Sent Done Done false
3640 1813 Acked Write 2 Sent Done Done false
3641 1814 Acked Read 5 Sent Done Done false
3642 1815 Acked Read 7 Sent Done Done false
3643 1816 Acked Write 7 Sent Done Done false
3644 1817 Acked Read 10 Sent Done Done false
3645 1818 Acked Write 5 Sent Done Done false
3646 1819 Acked Write 5 Sent Done Done false
3647 1820 Acked Write 6 Sent Done Done false
3648 1821 Acked Read 5 Sent Done Done false
3649 1822 Acked Read 3 Sent Done Done false
3650 1823 Acked Write 1 Sent Done Done false
3651 1824 Acked Flush 0 Sent Done Done false
3652 1825 Acked Write 2 Sent Done Done false
3653 1826 Acked Read 6 Sent Done Done false
3654 1827 Acked Read 8 Sent Done Done false
3655 1828 Acked Read 10 Sent Done Done false
3656 1829 Acked Write 4 Sent Done Done false
3657 1830 Acked Write 10 Sent Done Done false
3658 1831 Acked Write 1 Sent Done Done false
3659 1832 Acked Read 8 Sent Done Done false
3660 1833 Acked Read 1 Sent Done Done false
3661 1834 Acked Read 8 Sent Done Done false
3662 1835 Acked Write 5 Sent Done Done false
3663 1836 Acked Read 8 Sent Done Done false
3664 1837 Acked Write 1 Sent Done Done false
3665 1838 Acked Read 2 Sent Done Done false
3666 1839 Acked Read 5 Sent Done Done false
3667 1840 Acked Read 8 Sent Done Done false
3668 1841 Acked Write 9 Sent Done Done false
3669 1842 Acked Read 2 Sent Done Done false
3670 1843 Acked Read 2 Sent Done Done false
3671 1844 Acked Flush 0 Sent Done Done false
3672 1845 Acked Read 5 Sent Done Done false
3673 1846 Acked Write 1 Sent Done Done false
3674 1847 Acked Write 6 Sent Done Done false
3675 1848 Acked Read 1 Sent Done Done false
3676 1849 Acked Write 2 Sent Done Done false
3677 1850 Acked Read 4 Sent Done Done false
3678 1851 Acked Read 3 Sent Done Done false
3679 1852 Acked Write 1 Sent Done Done false
3680 1853 Acked Read 8 Sent Done Done false
3681 1854 Acked Flush 0 Sent Done Done false
3682 1855 Acked Read 3 Sent Done Done false
3683 1856 Acked Read 10 Sent Done Done false
3684 1857 Acked Read 6 Sent Done Done false
3685 1858 Acked Write 6 Sent Done Done false
3686 1859 Acked Write 10 Sent Done Done false
3687 1860 Acked Write 5 Sent Done Done false
3688 1861 Acked Write 8 Sent Done Done false
3689 1862 Acked Read 4 Sent Done Done false
3690 1863 Acked Read 9 Sent Done Done false
3691 1864 Acked Read 8 Sent Done Done false
3692 1865 Acked Read 3 Sent Done Done false
3693 1866 Acked Write 6 Sent Done Done false
3694 1867 Acked Write 10 Sent Done Done false
3695 1868 Acked Read 4 Sent Done Done false
3696 1869 Acked Write 5 Sent Done Done false
3697 1870 Acked Read 9 Sent Done Done false
3698 1871 Acked Write 4 Sent Done Done false
3699 1872 Acked Write 6 Sent Done Done false
3700 1873 Acked Write 8 Sent Done Done false
3701 1874 Acked Read 5 Sent Done Done false
3702 1875 Acked Read 4 Sent Done Done false
3703 1876 Acked Write 9 Sent Done Done false
3704 1877 Acked Write 2 Sent Done Done false
3705 1878 Acked Write 10 Sent Done Done false
3706 1879 Acked Read 4 Sent Done Done false
3707 1880 Acked Read 4 Sent Done Done false
3708 1881 Acked Write 2 Sent Done Done false
3709 1882 Acked Read 7 Sent Done Done false
3710 1883 Acked Read 7 Sent Done Done false
3711 1884 Acked Write 6 Sent Done Done false
3712 1885 Acked Write 1 Sent Done Done false
3713 1886 Acked Write 7 Sent Done Done false
3714 1887 Acked Flush 0 Sent Done Done false
3715 1888 Acked Read 1 Sent Done Done false
3716 1889 Acked Write 6 Sent Done Done false
3717 1890 Acked Read 3 Sent Done Done false
3718 1891 Acked Write 2 Sent Done Done false
3719 1892 Acked Write 7 Sent Done Done false
3720 1893 Acked Write 4 Sent Done Done false
3721 1894 Acked Read 4 Sent Done Done false
3722 1895 Acked Read 4 Sent Done Done false
3723 1896 Acked Read 9 Sent Done Done false
3724 1897 Acked Read 4 Sent Done Done false
3725 1898 Acked Read 6 Sent Done Done false
3726 1899 Acked Read 5 Sent Done Done false
3727 1900 Acked Read 1 Sent Done Done false
3728 1901 Acked Read 10 Sent Done Done false
3729 1902 Acked Read 5 Sent Done Done false
3730 1903 Acked Read 6 Sent Done Done false
3731 1904 Acked Write 2 Sent Done Done false
3732 1905 Acked Flush 0 Sent Done Done false
3733 STATES DS:0 DS:1 DS:2 TOTAL
3734 Sent 302 0 0 302
3735 Done 0 302 302 604
3736 Skipped 0 0 0 0
3737 Error 0 0 0 0
3738 Last Flush: Some(JobId(1603)) Some(JobId(1905)) Some(JobId(1905))
3739 Downstairs last five completed: 1603 1602 1601 1600 1599
3740 Upstairs last five completed: 1905 1904 1903 1902 1901
3741 CLIENT: Up:0 ds:302 act:2
37422025-01-23T17:38:32.116ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
37432025-01-23T17:38:32.117ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34684) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
37442025-01-23T17:38:32.118ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
37452025-01-23T17:38:32.118ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
37462025-01-23T17:38:32.118ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
37472025-01-23T17:38:32.118ZINFOcrucible: send last flush ID to this DS: Some(JobId(1603)) = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
37482025-01-23T17:38:32.118ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
37492025-01-23T17:38:32.118ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(1603)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
37502025-01-23T17:38:32.118ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(1603)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
3751 ----------------------------------------------------------------
3752 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3753 Downstairs last five completed: 1905 1904 1903 1902 1901
3754 Upstairs last five completed: 1905 1904 1903 1902 1901
3755 CLIENT: Up:0 ds:0 act:3
3756 Replay: All jobs finished
3757 Main task has work to do, go find it
3758 got dsc Stop(0)
3759 stop 0
3760 Jan 23 17:38:32.373 INFO request completed, latency_us: 359, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 11b619f8-3f61-49e8-9cb2-18b8bd3583d9[8810] Got stop action so:false kr:false
3761 , remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
37622025-01-23T17:38:32.373ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
37632025-01-23T17:38:32.373ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
37642025-01-23T17:38:32.373ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
3765 Jan 23 17:38:32.373 INFO{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T17:38:32.373908741Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
3766 request completed, latency_us: 135, response_code: 200, uri: /state/cid/0, method: GET, req_id: 96e15a51-393f-4228-a595-c55b86397a1a, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
3767 [8810] Exited with: ExitStatus(unix_wait_status(9))
3768 [8810][0] reports Exit
3769 Jan 23 17:38:36.376 INFO request completed, latency_us: 243, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6b0ab9fb-a272-4dca-852a-3add1807ddc8, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
3770 001/300 Write block 1027 len 16384 data: 2 2 2 2
3771 002/300 Read block 4213 len 32768
3772 003/300 Read block 226 len 12288
3773 004/300 Write block 4971 len 12288 data: 3 3 2
3774 005/300 Read block 2417 len 4096
3775 006/300 Read block 2834 len 12288
3776 007/300 Read block 2430 len 4096
3777 008/300 Write block 2954 len 8192 data: 2 2
3778 009/300 Read block 1640 len 32768
3779 010/300 Write block 4879 len 36864 data: 2 2 2 2 2 2 2 2 2
3780 011/300 Read block 2653 len 24576
3781 012/300 Read block 3470 len 32768
3782 013/300 Write block 538 len 36864 data: 3 3 3 4 3 3 2 2 2
3783 014/300 Write block 2571 len 4096 data: 2
3784 015/300 Write block 3210 len 8192 data: 2 2
3785 016/300 Write block 529 len 24576 data: 2 2 2 2 3 3
3786 017/300 Write block 2486 len 32768 data: 2 2 2 2 2 3 3 3
3787 018/300 Read block 4557 len 8192
3788 019/300 Write block 1251 len 4096 data: 2
3789 020/300 Write block 1815 len 12288 data: 2 2 2
3790 021/300 Read block 4369 len 36864
3791 022/300 Flush
3792 023/300 Write block 1937 len 32768 data: 2 2 2 2 2 2 2 2
3793 024/300 Write block 3183 len 8192 data: 4 4
3794 025/300 Flush
3795 026/300 Read block 1343 len 32768
3796 027/300 Flush
3797 028/300 Flush
3798 029/300 Read block 4682 len 16384
3799 030/300 Read block 680 len 16384
3800 031/300 Read block 1228 len 16384
3801 032/300 Write block 2629 len 12288 data: 3 3 3
3802 033/300 Read block 2486 len 32768
3803 034/300 Write block 2355 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3804 035/300 Read block 2104 len 4096
3805 036/300 Flush
3806 037/300 Read block 2916 len 24576
3807 038/300 Read block 1553 len 40960
3808 039/300 Read block 2527 len 24576
3809 040/300 Write block 2445 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3810 041/300 Read block 1835 len 4096
3811 042/300 Read block 2189 len 20480
3812 043/300 Read block 4168 len 16384
3813 044/300 Read block 3244 len 36864
3814 045/300 Read block 3690 len 12288
3815 046/300 Write block 3593 len 16384 data: 3 3 3 3
3816 047/300 Write block 2889 len 4096 data: 3
3817 048/300 Read block 4853 len 36864
3818 049/300 Flush
3819 050/300 Write block 2333 len 24576 data: 3 3 2 2 2 2
3820 051/300 Write block 2574 len 8192 data: 2 2
3821 052/300 Write block 2425 len 8192 data: 2 2
3822 053/300 Write block 461 len 40960 data: 4 4 4 4 4 4 3 2 2 2
3823 054/300 Write block 4894 len 8192 data: 2 2
3824 055/300 Write block 1787 len 28672 data: 2 2 2 2 2 2 2
3825 056/300 Read block 4956 len 36864
3826 057/300 Write block 759 len 12288 data: 2 2 2
3827 058/300 Write block 3298 len 4096 data: 2
3828 059/300 Write block 1227 len 4096 data: 2
3829 060/300 Read block 3169 len 32768
3830 061/300 Flush
3831 062/300 Write block 3848 len 32768 data: 2 2 2 2 2 3 3 3
3832 063/300 Read block 2310 len 28672
3833 064/300 Write block 3024 len 16384 data: 2 2 2 2
3834 065/300 Write block 1941 len 24576 data: 3 3 3 3 2 2
3835 066/300 Write block 52 len 36864 data: 2 3 3 3 3 3 3 3 2
3836 067/300 Read block 2320 len 32768
3837 068/300 Write block 2057 len 32768 data: 3 3 3 3 3 3 3 3
3838 069/300 Read block 1392 len 8192
3839 070/300 Write block 4726 len 8192 data: 3 3
3840 071/300 Read block 1025 len 28672
3841 072/300 Write block 340 len 8192 data: 2 2
3842 073/300 Write block 299 len 12288 data: 4 3 3
3843 074/300 Write block 2378 len 36864 data: 3 3 3 3 3 3 3 3 2
3844 075/300 Write block 731 len 28672 data: 3 3 3 3 3 3 2
3845 076/300 Flush
3846 077/300 Read block 3326 len 4096
3847 078/300 Read block 3579 len 20480
3848 079/300 Read block 2682 len 24576
3849 080/300 Write block 703 len 4096 data: 2
3850 081/300 Write block 1749 len 20480 data: 2 2 2 2 2
3851 082/300 Write block 980 len 4096 data: 2
3852 083/300 Write block 839 len 8192 data: 3 2
3853 084/300 Read block 3314 len 4096
3854 085/300 Flush
3855 086/300 Write block 2000 len 28672 data: 3 3 3 2 2 2 2
3856 087/300 Write block 3660 len 4096 data: 2
3857 088/300 Read block 1784 len 28672
3858 089/300 Read block 1825 len 36864
3859 090/300 Write block 1493 len 4096 data: 2
3860 091/300 Flush
3861 092/300 Flush
3862 093/300 Write block 4278 len 32768 data: 2 2 2 2 2 3 3 2
3863 094/300 Read block 3729 len 8192
3864 095/300 Write block 3825 len 8192 data: 2 2
3865 096/300 Read block 4583 len 40960
3866 097/300 Write block 260 len 12288 data: 2 2 2
3867 098/300 Write block 4682 len 32768 data: 3 3 2 2 2 2 2 2
3868 099/300 Read block 2782 len 8192
3869 100/300 Read block 2225 len 24576
3870 101/300 Read block 3909 len 28672
3871 102/300 Write block 36 len 36864 data: 2 2 2 2 2 2 2 2 2
3872 103/300 Read block 1190 len 36864
3873 104/300 Write block 3851 len 24576 data: 3 3 4 4 4 3
3874 105/300 Read block 3305 len 28672
3875 106/300 Read block 3471 len 40960
3876 107/300 Read block 1604 len 28672
3877 108/300 Write block 3299 len 12288 data: 2 2 2
3878 109/300 Write block 3233 len 32768 data: 2 3 3 3 3 3 3 3
3879 110/300 Read block 2825 len 40960
3880 111/300 Flush
3881 112/300 Write block 3625 len 32768 data: 2 2 2 2 2 2 2 2
3882 113/300 Write block 2373 len 20480 data: 3 3 3 3 3
3883 114/300 Write block 2342 len 32768 data: 2 2 2 3 3 3 4 4
3884 115/300 Write block 2925 len 32768 data: 2 2 2 2 2 2 2 2
3885 116/300 Read block 2596 len 40960
3886 117/300 Write block 3669 len 40960 data: 3 2 2 2 2 2 2 2 2 2
3887 118/300 Flush
3888 119/300 Write block 4318 len 28672 data: 2 2 2 2 2 2 2
3889 120/300 Write block 3110 len 8192 data: 2 2
3890 121/300 Read block 1859 len 36864
3891 122/300 Read block 3408 len 32768
3892 123/300 Read block 2698 len 16384
3893 124/300 Read block 4686 len 8192
3894 125/300 Write block 256 len 8192 data: 2 2
3895 126/300 Write block 4203 len 24576 data: 2 2 3 4 4 4
3896 127/300 Write block 1821 len 28672 data: 2 2 2 2 3 3 3
3897 128/300 Read block 2330 len 36864
3898 129/300 Read block 1669 len 36864
3899 130/300 Read block 4892 len 28672
3900 131/300 Read block 3527 len 4096
3901 132/300 Read block 3129 len 40960
3902 133/300 Write block 4600 len 40960 data: 2 2 2 2 2 2 2 2 2 3
3903 134/300 Write block 367 len 40960 data: 4 4 4 4 4 4 4 4 4 3
3904 135/300 Read block 2038 len 24576
3905 136/300 Read block 1286 len 28672
3906 137/300 Flush
3907 138/300 Write block 2171 len 20480 data: 2 2 2 2 2
3908 139/300 Write block 3098 len 20480 data: 2 2 2 2 2
3909 140/300 Write block 2371 len 36864 data: 2 3 4 4 4 4 4 4 4
3910 141/300 Read block 1162 len 40960
3911 142/300 Read block 3555 len 36864
3912 143/300 Read block 2733 len 8192
3913 144/300 Write block 1450 len 4096 data: 2
3914 145/300 Write block 4685 len 36864 data: 3 3 3 3 3 2 2 2 2
3915 146/300 Write block 843 len 12288 data: 3 3 3
3916 147/300 Write block 2457 len 36864 data: 2 2 2 2 2 2 2 3 3
3917 148/300 Read block 272 len 20480
3918 149/300 Write block 2230 len 24576 data: 3 3 3 3 4 3
3919 150/300 Read block 3019 len 4096
3920 151/300 Read block 2679 len 32768
3921 152/300 Write block 388 len 16384 data: 2 2 2 2
3922 153/300 Read block 4802 len 36864
3923 154/300 Write block 3368 len 20480 data: 2 2 2 2 2
3924 155/300 Read block 3108 len 40960
3925 156/300 Read block 2558 len 20480
3926 157/300 Read block 253 len 20480
3927 158/300 Write block 151 len 40960 data: 2 2 2 2 2 2 3 3 3 4
3928 159/300 Read block 2845 len 36864
3929 160/300 Read block 1885 len 20480
3930 161/300 Write block 1462 len 24576 data: 2 2 2 2 2 2
3931 162/300 Read block 4492 len 8192
3932 163/300 Write block 4693 len 36864 data: 3 2 2 2 2 2 2 2 2
3933 164/300 Write block 1964 len 8192 data: 3 3
3934 165/300 Write block 3783 len 28672 data: 2 2 2 2 2 2 2
3935 166/300 Flush
3936 167/300 Write block 522 len 8192 data: 3 3
3937 168/300 Read block 99 len 32768
3938 169/300 Write block 3692 len 20480 data: 2 2 2 2 2
3939 170/300 Write block 4910 len 24576 data: 2 2 2 2 2 2
3940 171/300 Read block 1433 len 16384
3941 172/300 Write block 3660 len 16384 data: 3 2 2 2
3942 173/300 Read block 1073 len 40960
3943 174/300 Read block 3613 len 36864
3944 175/300 Read block 2307 len 32768
3945 176/300 Write block 1918 len 28672 data: 2 2 2 2 2 2 2
3946 177/300 Read block 650 len 40960
3947 178/300 Read block 2848 len 8192
3948 179/300 Read block 4348 len 32768
3949 180/300 Write block 2986 len 28672 data: 2 2 2 2 2 2 2
3950 181/300 Write block 4628 len 20480 data: 2 2 2 2 2
3951 182/300 Write block 2357 len 4096 data: 3
3952 183/300 Read block 4875 len 4096
3953 184/300 Read block 1453 len 20480
3954 185/300 Write block 3985 len 16384 data: 2 2 2 2
3955 186/300 Flush
3956 187/300 Write block 4368 len 32768 data: 2 2 2 2 2 2 2 2
3957 188/300 Flush
3958 189/300 Write block 2340 len 4096 data: 2
3959 190/300 Write block 3835 len 20480 data: 3 3 3 3 3
3960 191/300 Write block 1303 len 36864 data: 2 2 2 2 3 3 3 3 3
3961 192/300 Read block 674 len 16384
3962 193/300 Flush
3963 194/300 Flush
3964 195/300 Read block 2945 len 36864
3965 196/300 Write block 367 len 40960 data: 5 5 5 5 5 5 5 5 5 4
3966 197/300 Write block 3140 len 28672 data: 2 2 2 2 2 2 2
3967 198/300 Flush
3968 199/300 Write block 4610 len 40960 data: 3 3 3 3 3 2 2 2 2 2
3969 200/300 Write block 4564 len 32768 data: 3 3 3 3 3 3 3 2
3970 201/300 Flush
3971 202/300 Flush
3972 203/300 Read block 4884 len 32768
3973 204/300 Write block 1782 len 12288 data: 2 2 2
3974 205/300 Read block 3655 len 40960
3975 206/300 Write block 2441 len 16384 data: 2 2 2 2
3976 207/300 Write block 2745 len 32768 data: 2 2 2 2 2 2 2 2
3977 208/300 Read block 1296 len 20480
3978 209/300 Write block 2011 len 12288 data: 3 3 2
3979 210/300 Read block 4371 len 8192
3980 211/300 Write block 2742 len 40960 data: 2 2 2 3 3 3 3 3 3 3
3981 212/300 Flush
3982 213/300 Read block 4245 len 28672
3983 214/300 Read block 4410 len 32768
3984 215/300 Write block 1862 len 36864 data: 3 2 2 2 3 3 2 3 2
3985 216/300 Read block 2299 len 28672
3986 217/300 Write block 116 len 4096 data: 5
3987 218/300 Write block 4440 len 24576 data: 2 2 2 2 2 2
3988 219/300 Read block 3394 len 8192
3989 220/300 Read block 2606 len 40960
3990 221/300 Read block 478 len 32768
3991 222/300 Write block 413 len 4096 data: 2
3992 223/300 Write block 4370 len 20480 data: 3 3 3 3 3
3993 224/300 Read block 2008 len 12288
3994 225/300 Write block 1487 len 16384 data: 2 2 2 2
3995 226/300 Read block 3134 len 16384
3996 227/300 Read block 26 len 24576
3997 228/300 Read block 4207 len 4096
3998 229/300 Read block 3752 len 8192
3999 230/300 Read block 2549 len 24576
4000 231/300 Write block 3008 len 28672 data: 2 2 2 2 2 2 2
4001 232/300 Read block 3477 len 12288
4002 233/300 Read block 1892 len 20480
4003 234/300 Write block 1329 len 28672 data: 2 3 3 3 2 2 2
4004 235/300 Write block 4702 len 36864 data: 2 2 2 2 2 2 3 3 3
4005 236/300 Read block 4854 len 16384
4006 237/300 Read block 823 len 32768
4007 238/300 Write block 3455 len 36864 data: 2 2 2 2 2 2 2 2 2
4008 239/300 Write block 1233 len 20480 data: 2 2 2 2 2
4009 240/300 Read block 541 len 24576
4010 241/300 Write block 3031 len 28672 data: 2 3 4 4 4 3 3
4011 242/300 Read block 1318 len 12288
4012 243/300 Write block 2768 len 4096 data: 3
4013 244/300 Write block 1932 len 20480 data: 2 2 2 2 2
4014 245/300 Read block 3665 len 8192
4015 246/300 Read block 4946 len 4096
4016 247/300 Read block 2829 len 24576
4017 248/300 Read block 2947 len 8192
4018 249/300 Write block 3623 len 4096 data: 2
4019 250/300 Write block 2918 len 28672 data: 2 2 2 2 2 2 2
4020 251/300 Write block 1021 len 32768 data: 2 2 2 2 2 2 3 3
4021 252/300 Flush
4022 253/300 Write block 3425 len 36864 data: 2 2 2 2 2 3 3 3 4
4023 254/300 Read block 989 len 20480
4024 255/300 Read block 1972 len 16384
4025 256/300 Write block 4216 len 8192 data: 2 2
4026 257/300 Read block 3016 len 8192
4027 258/300 Write block 2181 len 16384 data: 2 2 2 2
4028 259/300 Read block 4205 len 40960
4029 260/300 Read block 677 len 36864
4030 261/300 Write block 3111 len 28672 data: 3 2 2 2 2 2 2
4031 262/300 Read block 560 len 20480
4032 263/300 Write block 3441 len 12288 data: 3 3 3
4033 264/300 Write block 742 len 36864 data: 2 2 2 2 2 2 2 2 2
4034 265/300 Read block 3362 len 4096
4035 266/300 Write block 713 len 24576 data: 2 2 2 2 2 3
4036 267/300 Write block 17 len 16384 data: 3 3 3 3
4037 268/300 Read block 4824 len 28672
4038 269/300 Read block 3029 len 32768
4039 270/300 Read block 1291 len 28672
4040 271/300 Read block 512 len 40960
4041 272/300 Write block 1825 len 4096 data: 4
4042 273/300 Write block 4361 len 28672 data: 2 2 2 2 2 2 2
4043 274/300 Write block 1681 len 16384 data: 3 3 3 3
4044 275/300 Read block 4254 len 4096
4045 276/300 Write block 4958 len 8192 data: 2 2
4046 277/300 Read block 4140 len 24576
4047 278/300 Flush
4048 279/300 Read block 3552 len 12288
4049 280/300 Write block 3911 len 28672 data: 3 3 3 3 3 2 2
4050 281/300 Write block 1339 len 4096 data: 2
4051 282/300 Flush
4052 283/300 Write block 1345 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4053 284/300 Write block 4618 len 20480 data: 3 3 2 2 2
4054 285/300 Write block 4371 len 12288 data: 4 4 4
4055 286/300 Write block 1974 len 28672 data: 3 3 3 3 3 2 2
4056 287/300 Write block 1882 len 24576 data: 3 3 3 3 3 3
4057 288/300 Read block 3607 len 32768
4058 289/300 Read block 4047 len 4096
4059 290/300 Write block 4556 len 12288 data: 2 2 2
4060 291/300 Write block 455 len 32768 data: 2 2 2 2 3 3 5 5
4061 292/300 Flush
4062 293/300 Read block 360 len 12288
4063 294/300 Read block 3683 len 40960
4064 295/300 Read block 3375 len 40960
4065 296/300 Read block 1251 len 8192
4066 297/300 Write block 285 len 28672 data: 3 3 3 4 4 4 2
4067 298/300 Read block 3946 len 16384
4068 299/300 Write block 3250 len 4096 data: 2
4069 300/300 Write block 1136 len 28672 data: 2 2 2 3 3 3 3
4070 301/300 Read block 1783 len 32768
4071 Main task has work to do, go find it
4072 got dsc Start(0)
4073 start 0
4074 Jan 23 17:38:36.611 INFO request completed, latency_us: 224, response_code: 204, uri: /start/cid/0, method: POST, req_id: f32c8bec-7561-4f42-aaea-afb452d8cb02, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
4075 Replay: started 0, returned:Ok(())
4076 ----------------------------------------------------------------
4077 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4078 States: Offline Active Active
4079 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4080 1906 Acked Write 4 Sent Done Done false
4081 1907 Acked Read 8 Sent Done Done false
4082 1908 Acked Read 3 Sent Done Done false
4083 1909 Acked Write 3 Sent Done Done false
4084 1910 Acked Read 1 Sent Done Done false
4085 1911 Acked Read 3 Sent Done Done false
4086 1912 Acked Read 1 Sent Done Done false
4087 1913 Acked Write 2 Sent Done Done false
4088 1914 Acked Read 8 Sent Done Done false
4089 1915 Acked Write 9 Sent Done Done false
4090 1916 Acked Read 6 Sent Done Done false
4091 1917 Acked Read 8 Sent Done Done false
4092 1918 Acked Write 9 Sent Done Done false
4093 1919 Acked Write 1 Sent Done Done false
4094 1920 Acked Write 2 Sent Done Done false
4095 1921 Acked Write 6 Sent Done Done false
4096 1922 Acked Write 8 Sent Done Done false
4097 1923 Acked Read 2 Sent Done Done false
4098 1924 Acked Write 1 Sent Done Done false
4099 1925 Acked Write 3 Sent Done Done false
4100 1926 Acked Read 9 Sent Done Done false
4101 1927 Acked Flush 0 Sent Done Done false
4102 1928 Acked Write 8 Sent Done Done false
4103 1929 Acked Write 2 Sent Done Done false
4104 1930 Acked Flush 0 Sent Done Done false
4105 1931 Acked Read 8 Sent Done Done false
4106 1932 Acked Flush 0 Sent Done Done false
4107 1933 Acked Flush 0 Sent Done Done false
4108 1934 Acked Read 4 Sent Done Done false
4109 1935 Acked Read 4 Sent Done Done false
4110 1936 Acked Read 4 Sent Done Done false
4111 1937 Acked Write 3 Sent Done Done false
4112 1938 Acked Read 8 Sent Done Done false
4113 1939 Acked Write 10 Sent Done Done false
4114 1940 Acked Read 1 Sent Done Done false
4115 1941 Acked Flush 0 Sent Done Done false
4116 1942 Acked Read 6 Sent Done Done false
4117 1943 Acked Read 10 Sent Done Done false
4118 1944 Acked Read 6 Sent Done Done false
4119 1945 Acked Write 10 Sent Done Done false
4120 1946 Acked Read 1 Sent Done Done false
4121 1947 Acked Read 5 Sent Done Done false
4122 1948 Acked Read 4 Sent Done Done false
4123 1949 Acked Read 9 Sent Done Done false
4124 1950 Acked Read 3 Sent Done Done false
4125 1951 Acked Write 4 Sent Done Done false
4126 1952 Acked Write 1 Sent Done Done false
4127 1953 Acked Read 9 Sent Done Done false
4128 1954 Acked Flush 0 Sent Done Done false
4129 1955 Acked Write 6 Sent Done Done false
4130 1956 Acked Write 2 Sent Done Done false
4131 1957 Acked Write 2 Sent Done Done false
4132 1958 Acked Write 10 Sent Done Done false
4133 1959 Acked Write 2 Sent Done Done false
4134 1960 Acked Write 7 Sent Done Done false
4135 1961 Acked Read 9 Sent Done Done false
4136 1962 Acked Write 3 Sent Done Done false
4137 1963 Acked Write 1 Sent Done Done false
4138 1964 Acked Write 1 Sent Done Done false
4139 1965 Acked Read 8 Sent Done Done false
4140 1966 Acked Flush 0 Sent Done Done false
4141 1967 Acked Write 8 Sent Done Done false
4142 1968 Acked Read 7 Sent Done Done false
4143 1969 Acked Write 4 Sent Done Done false
4144 1970 Acked Write 6 Sent Done Done false
4145 1971 Acked Write 9 Sent Done Done false
4146 1972 Acked Read 8 Sent Done Done false
4147 1973 Acked Write 8 Sent Done Done false
4148 1974 Acked Read 2 Sent Done Done false
4149 1975 Acked Write 2 Sent Done Done false
4150 1976 Acked Read 7 Sent Done Done false
4151 1977 Acked Write 2 Sent Done Done false
4152 1978 Acked Write 3 Sent Done Done false
4153 1979 Acked Write 9 Sent Done Done false
4154 1980 Acked Write 7 Sent Done Done false
4155 1981 Acked Flush 0 Sent Done Done false
4156 1982 Acked Read 1 Sent Done Done false
4157 1983 Acked Read 5 Sent Done Done false
4158 1984 Acked Read 6 Sent Done Done false
4159 1985 Acked Write 1 Sent Done Done false
4160 1986 Acked Write 5 Sent Done Done false
4161 1987 Acked Write 1 Sent Done Done false
4162 1988 Acked Write 2 Sent Done Done false
4163 1989 Acked Read 1 Sent Done Done false
4164 1990 Acked Flush 0 Sent Done Done false
4165 1991 Acked Write 7 Sent Done Done false
4166 1992 Acked Write 1 Sent Done Done false
4167 1993 Acked Read 7 Sent Done Done false
4168 1994 Acked Read 9 Sent Done Done false
4169 1995 Acked Write 1 Sent Done Done false
4170 1996 Acked Flush 0 Sent Done Done false
4171 1997 Acked Flush 0 Sent Done Done false
4172 1998 Acked Write 8 Sent Done Done false
4173 1999 Acked Read 2 Sent Done Done false
4174 2000 Acked Write 2 Sent Done Done false
4175 2001 Acked Read 10 Sent Done Done false
4176 2002 Acked Write 3 Sent Done Done false
4177 2003 Acked Write 8 Sent Done Done false
4178 2004 Acked Read 2 Sent Done Done false
4179 2005 Acked Read 6 Sent Done Done false
4180 2006 Acked Read 7 Sent Done Done false
4181 2007 Acked Write 9 Sent Done Done false
4182 2008 Acked Read 9 Sent Done Done false
4183 2009 Acked Write 6 Sent Done Done false
4184 2010 Acked Read 7 Sent Done Done false
4185 2011 Acked Read 10 Sent Done Done false
4186 2012 Acked Read 7 Sent Done Done false
4187 2013 Acked Write 3 Sent Done Done false
4188 2014 Acked Write 8 Sent Done Done false
4189 2015 Acked Read 10 Sent Done Done false
4190 2016 Acked Flush 0 Sent Done Done false
4191 2017 Acked Write 8 Sent Done Done false
4192 2018 Acked Write 5 Sent Done Done false
4193 2019 Acked Write 8 Sent Done Done false
4194 2020 Acked Write 8 Sent Done Done false
4195 2021 Acked Read 10 Sent Done Done false
4196 2022 Acked Write 10 Sent Done Done false
4197 2023 Acked Flush 0 Sent Done Done false
4198 2024 Acked Write 7 Sent Done Done false
4199 2025 Acked Write 2 Sent Done Done false
4200 2026 Acked Read 9 Sent Done Done false
4201 2027 Acked Read 8 Sent Done Done false
4202 2028 Acked Read 4 Sent Done Done false
4203 2029 Acked Read 2 Sent Done Done false
4204 2030 Acked Write 2 Sent Done Done false
4205 2031 Acked Write 6 Sent Done Done false
4206 2032 Acked Write 7 Sent Done Done false
4207 2033 Acked Read 9 Sent Done Done false
4208 2034 Acked Read 9 Sent Done Done false
4209 2035 Acked Read 7 Sent Done Done false
4210 2036 Acked Read 1 Sent Done Done false
4211 2037 Acked Read 10 Sent Done Done false
4212 2038 Acked Write 10 Sent Done Done false
4213 2039 Acked Write 10 Sent Done Done false
4214 2040 Acked Read 6 Sent Done Done false
4215 2041 Acked Read 7 Sent Done Done false
4216 2042 Acked Flush 0 Sent Done Done false
4217 2043 Acked Write 5 Sent Done Done false
4218 2044 Acked Write 5 Sent Done Done false
4219 2045 Acked Write 9 Sent Done Done false
4220 2046 Acked Read 10 Sent Done Done false
4221 2047 Acked Read 9 Sent Done Done false
4222 2048 Acked Read 2 Sent Done Done false
4223 2049 Acked Write 1 Sent Done Done false
4224 2050 Acked Write 9 Sent Done Done false
4225 2051 Acked Write 3 Sent Done Done false
4226 2052 Acked Write 9 Sent Done Done false
4227 2053 Acked Read 5 Sent Done Done false
4228 2054 Acked Write 6 Sent Done Done false
4229 2055 Acked Read 1 Sent Done Done false
4230 2056 Acked Read 8 Sent Done Done false
4231 2057 Acked Write 4 Sent Done Done false
4232 2058 Acked Read 9 Sent Done Done false
4233 2059 Acked Write 5 Sent Done Done false
4234 2060 Acked Read 10 Sent Done Done false
4235 2061 Acked Read 5 Sent Done Done false
4236 2062 Acked Read 5 Sent Done Done false
4237 2063 Acked Write 10 Sent Done Done false
4238 2064 Acked Read 9 Sent Done Done false
4239 2065 Acked Read 5 Sent Done Done false
4240 2066 Acked Write 6 Sent Done Done false
4241 2067 Acked Read 2 Sent Done Done false
4242 2068 Acked Write 9 Sent Done Done false
4243 2069 Acked Write 2 Sent Done Done false
4244 2070 Acked Write 7 Sent Done Done false
4245 2071 Acked Flush 0 Sent Done Done false
4246 2072 Acked Write 2 Sent Done Done false
4247 2073 Acked Read 8 Sent Done Done false
4248 2074 Acked Write 5 Sent Done Done false
4249 2075 Acked Write 6 Sent Done Done false
4250 2076 Acked Read 4 Sent Done Done false
4251 2077 Acked Write 4 Sent Done Done false
4252 2078 Acked Read 10 Sent Done Done false
4253 2079 Acked Read 9 Sent Done Done false
4254 2080 Acked Read 8 Sent Done Done false
4255 2081 Acked Write 7 Sent Done Done false
4256 2082 Acked Read 10 Sent Done Done false
4257 2083 Acked Read 2 Sent Done Done false
4258 2084 Acked Read 8 Sent Done Done false
4259 2085 Acked Write 7 Sent Done Done false
4260 2086 Acked Write 5 Sent Done Done false
4261 2087 Acked Write 1 Sent Done Done false
4262 2088 Acked Read 1 Sent Done Done false
4263 2089 Acked Read 5 Sent Done Done false
4264 2090 Acked Write 4 Sent Done Done false
4265 2091 Acked Flush 0 Sent Done Done false
4266 2092 Acked Write 8 Sent Done Done false
4267 2093 Acked Flush 0 Sent Done Done false
4268 2094 Acked Write 1 Sent Done Done false
4269 2095 Acked Write 5 Sent Done Done false
4270 2096 Acked Write 9 Sent Done Done false
4271 2097 Acked Read 4 Sent Done Done false
4272 2098 Acked Flush 0 Sent Done Done false
4273 2099 Acked Flush 0 Sent Done Done false
4274 2100 Acked Read 9 Sent Done Done false
4275 2101 Acked Write 10 Sent Done Done false
4276 2102 Acked Write 7 Sent Done Done false
4277 2103 Acked Flush 0 Sent Done Done false
4278 2104 Acked Write 10 Sent Done Done false
4279 2105 Acked Write 8 Sent Done Done false
4280 2106 Acked Flush 0 Sent Done Done false
4281 2107 Acked Flush 0 Sent Done Done false
4282 2108 Acked Read 8 Sent Done Done false
4283 2109 Acked Write 3 Sent Done Done false
4284 2110 Acked Read 10 Sent Done Done false
4285 2111 Acked Write 4 Sent Done Done false
4286 2112 Acked Write 8 Sent Done Done false
4287 2113 Acked Read 5 Sent Done Done false
4288 2114 Acked Write 3 Sent Done Done false
4289 2115 Acked Read 2 Sent Done Done false
4290 2116 Acked Write 10 Sent Done Done false
4291 2117 Acked Flush 0 Sent Done Done false
4292 2118 Acked Read 7 Sent Done Done false
4293 2119 Acked Read 8 Sent Done Done false
4294 2120 Acked Write 9 Sent Done Done false
4295 2121 Acked Read 7 Sent Done Done false
4296 2122 Acked Write 1 Sent Done Done false
4297 2123 Acked Write 6 Sent Done Done false
4298 2124 Acked Read 2 Sent Done Done false
4299 2125 Acked Read 10 Sent Done Done false
4300 2126 Acked Read 8 Sent Done Done false
4301 2127 Acked Write 1 Sent Done Done false
4302 2128 Acked Write 5 Sent Done Done false
4303 2129 Acked Read 3 Sent Done Done false
4304 2130 Acked Write 4 Sent Done Done false
4305 2131 Acked Read 4 Sent Done Done false
4306 2132 Acked Read 6 Sent Done Done false
4307 2133 Acked Read 1 Sent Done Done false
4308 2134 Acked Read 2 Sent Done Done false
4309 2135 Acked Read 6 Sent Done Done false
4310 2136 Acked Write 7 Sent Done Done false
4311 2137 Acked Read 3 Sent Done Done false
4312 2138 Acked Read 5 Sent Done Done false
4313 2139 Acked Write 7 Sent Done Done false
4314 2140 Acked Write 9 Sent Done Done false
4315 2141 Acked Read 4 Sent Done Done false
4316 2142 Acked Read 8 Sent Done Done false
4317 2143 Acked Write 9 Sent Done Done false
4318 2144 Acked Write 5 Sent Done Done false
4319 2145 Acked Read 6 Sent Done Done false
4320 2146 Acked Write 7 Sent Done Done false
4321 2147 Acked Read 3 Sent Done Done false
4322 2148 Acked Write 1 Sent Done Done false
4323 2149 Acked Write 5 Sent Done Done false
4324 2150 Acked Read 2 Sent Done Done false
4325 2151 Acked Read 1 Sent Done Done false
4326 2152 Acked Read 6 Sent Done Done false
4327 2153 Acked Read 2 Sent Done Done false
4328 2154 Acked Write 1 Sent Done Done false
4329 2155 Acked Write 7 Sent Done Done false
4330 2156 Acked Write 8 Sent Done Done false
4331 2157 Acked Flush 0 Sent Done Done false
4332 2158 Acked Write 9 Sent Done Done false
4333 2159 Acked Read 5 Sent Done Done false
4334 2160 Acked Read 4 Sent Done Done false
4335 2161 Acked Write 2 Sent Done Done false
4336 2162 Acked Read 2 Sent Done Done false
4337 2163 Acked Write 4 Sent Done Done false
4338 2164 Acked Read 10 Sent Done Done false
4339 2165 Acked Read 9 Sent Done Done false
4340 2166 Acked Write 7 Sent Done Done false
4341 2167 Acked Read 5 Sent Done Done false
4342 2168 Acked Write 3 Sent Done Done false
4343 2169 Acked Write 9 Sent Done Done false
4344 2170 Acked Read 1 Sent Done Done false
4345 2171 Acked Write 6 Sent Done Done false
4346 2172 Acked Write 4 Sent Done Done false
4347 2173 Acked Read 7 Sent Done Done false
4348 2174 Acked Read 8 Sent Done Done false
4349 2175 Acked Read 7 Sent Done Done false
4350 2176 Acked Read 10 Sent Done Done false
4351 2177 Acked Write 1 Sent Done Done false
4352 2178 Acked Write 7 Sent Done Done false
4353 2179 Acked Write 4 Sent Done Done false
4354 2180 Acked Read 1 Sent Done Done false
4355 2181 Acked Write 2 Sent Done Done false
4356 2182 Acked Read 6 Sent Done Done false
4357 2183 Acked Flush 0 Sent Done Done false
4358 2184 Acked Read 3 Sent Done Done false
4359 2185 Acked Write 7 Sent Done Done false
4360 2186 Acked Write 1 Sent Done Done false
4361 2187 Acked Flush 0 Sent Done Done false
4362 2188 Acked Write 10 Sent Done Done false
4363 2189 Acked Write 5 Sent Done Done false
4364 2190 Acked Write 3 Sent Done Done false
4365 2191 Acked Write 7 Sent Done Done false
4366 2192 Acked Write 6 Sent Done Done false
4367 2193 Acked Read 8 Sent Done Done false
4368 2194 Acked Read 1 Sent Done Done false
4369 2195 Acked Write 3 Sent Done Done false
4370 2196 Acked Write 8 Sent Done Done false
4371 2197 Acked Flush 0 Sent Done Done false
4372 2198 Acked Read 3 Sent Done Done false
4373 2199 Acked Read 10 Sent Done Done false
4374 2200 Acked Read 10 Sent Done Done false
4375 2201 Acked Read 2 Sent Done Done false
4376 2202 Acked Write 7 Sent Done Done false
4377 2203 Acked Read 4 Sent Done Done false
4378 2204 Acked Write 1 Sent Done Done false
4379 2205 Acked Write 7 Sent Done Done false
4380 2206 Acked Read 8 Sent Done Done false
4381 STATES DS:0 DS:1 DS:2 TOTAL
4382 Sent 301 0 0 301
4383 Done 0 301 301 602
4384 Skipped 0 0 0 0
4385 Error 0 0 0 0
4386 Last Flush: Some(JobId(1905)) Some(JobId(2197)) Some(JobId(2197))
4387 Downstairs last five completed: 1905 1904 1903 1902 1901
4388 Upstairs last five completed: 2206 2205 2204 2203 2202
4389 CLIENT: Up:0 ds:301 act:2
4390 [8810] got start action
4391 [8810] I am going to restart
4392 Starting downstairs at port 8810
4393 Make output file at "/tmp/dsc/downstairs-8810.txt"
4394 [8810][0] reports Starting
4395 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1209)
4396 [8810][0] reports Running
4397 ----------------------------------------------------------------
4398 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
4399 States: Offline Active Active
4400 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4401 1906 Acked Write 4 Sent Done Done false
4402 1907 Acked Read 8 Sent Done Done false
4403 1908 Acked Read 3 Sent Done Done false
4404 1909 Acked Write 3 Sent Done Done false
4405 1910 Acked Read 1 Sent Done Done false
4406 1911 Acked Read 3 Sent Done Done false
4407 1912 Acked Read 1 Sent Done Done false
4408 1913 Acked Write 2 Sent Done Done false
4409 1914 Acked Read 8 Sent Done Done false
4410 1915 Acked Write 9 Sent Done Done false
4411 1916 Acked Read 6 Sent Done Done false
4412 1917 Acked Read 8 Sent Done Done false
4413 1918 Acked Write 9 Sent Done Done false
4414 1919 Acked Write 1 Sent Done Done false
4415 1920 Acked Write 2 Sent Done Done false
4416 1921 Acked Write 6 Sent Done Done false
4417 1922 Acked Write 8 Sent Done Done false
4418 1923 Acked Read 2 Sent Done Done false
4419 1924 Acked Write 1 Sent Done Done false
4420 1925 Acked Write 3 Sent Done Done false
4421 1926 Acked Read 9 Sent Done Done false
4422 1927 Acked Flush 0 Sent Done Done false
4423 1928 Acked Write 8 Sent Done Done false
4424 1929 Acked Write 2 Sent Done Done false
4425 1930 Acked Flush 0 Sent Done Done false
4426 1931 Acked Read 8 Sent Done Done false
4427 1932 Acked Flush 0 Sent Done Done false
4428 1933 Acked Flush 0 Sent Done Done false
4429 1934 Acked Read 4 Sent Done Done false
4430 1935 Acked Read 4 Sent Done Done false
4431 1936 Acked Read 4 Sent Done Done false
4432 1937 Acked Write 3 Sent Done Done false
4433 1938 Acked Read 8 Sent Done Done false
4434 1939 Acked Write 10 Sent Done Done false
4435 1940 Acked Read 1 Sent Done Done false
4436 1941 Acked Flush 0 Sent Done Done false
4437 1942 Acked Read 6 Sent Done Done false
4438 1943 Acked Read 10 Sent Done Done false
4439 1944 Acked Read 6 Sent Done Done false
4440 1945 Acked Write 10 Sent Done Done false
4441 1946 Acked Read 1 Sent Done Done false
4442 1947 Acked Read 5 Sent Done Done false
4443 1948 Acked Read 4 Sent Done Done false
4444 1949 Acked Read 9 Sent Done Done false
4445 1950 Acked Read 3 Sent Done Done false
4446 1951 Acked Write 4 Sent Done Done false
4447 1952 Acked Write 1 Sent Done Done false
4448 1953 Acked Read 9 Sent Done Done false
4449 1954 Acked Flush 0 Sent Done Done false
4450 1955 Acked Write 6 Sent Done Done false
4451 1956 Acked Write 2 Sent Done Done false
4452 1957 Acked Write 2 Sent Done Done false
4453 1958 Acked Write 10 Sent Done Done false
4454 1959 Acked Write 2 Sent Done Done false
4455 1960 Acked Write 7 Sent Done Done false
4456 1961 Acked Read 9 Sent Done Done false
4457 1962 Acked Write 3 Sent Done Done false
4458 1963 Acked Write 1 Sent Done Done false
4459 1964 Acked Write 1 Sent Done Done false
4460 1965 Acked Read 8 Sent Done Done false
4461 1966 Acked Flush 0 Sent Done Done false
4462 1967 Acked Write 8 Sent Done Done false
4463 1968 Acked Read 7 Sent Done Done false
4464 1969 Acked Write 4 Sent Done Done false
4465 1970 Acked Write 6 Sent Done Done false
4466 1971 Acked Write 9 Sent Done Done false
4467 1972 Acked Read 8 Sent Done Done false
4468 1973 Acked Write 8 Sent Done Done false
4469 1974 Acked Read 2 Sent Done Done false
4470 1975 Acked Write 2 Sent Done Done false
4471 1976 Acked Read 7 Sent Done Done false
4472 1977 Acked Write 2 Sent Done Done false
4473 1978 Acked Write 3 Sent Done Done false
4474 1979 Acked Write 9 Sent Done Done false
4475 1980 Acked Write 7 Sent Done Done false
4476 1981 Acked Flush 0 Sent Done Done false
4477 1982 Acked Read 1 Sent Done Done false
4478 1983 Acked Read 5 Sent Done Done false
4479 1984 Acked Read 6 Sent Done Done false
4480 1985 Acked Write 1 Sent Done Done false
4481 1986 Acked Write 5 Sent Done Done false
4482 1987 Acked Write 1 Sent Done Done false
4483 1988 Acked Write 2 Sent Done Done false
4484 1989 Acked Read 1 Sent Done Done false
4485 1990 Acked Flush 0 Sent Done Done false
4486 1991 Acked Write 7 Sent Done Done false
4487 1992 Acked Write 1 Sent Done Done false
4488 1993 Acked Read 7 Sent Done Done false
4489 1994 Acked Read 9 Sent Done Done false
4490 1995 Acked Write 1 Sent Done Done false
4491 1996 Acked Flush 0 Sent Done Done false
4492 1997 Acked Flush 0 Sent Done Done false
4493 1998 Acked Write 8 Sent Done Done false
4494 1999 Acked Read 2 Sent Done Done false
4495 2000 Acked Write 2 Sent Done Done false
4496 2001 Acked Read 10 Sent Done Done false
4497 2002 Acked Write 3 Sent Done Done false
4498 2003 Acked Write 8 Sent Done Done false
4499 2004 Acked Read 2 Sent Done Done false
4500 2005 Acked Read 6 Sent Done Done false
4501 2006 Acked Read 7 Sent Done Done false
4502 2007 Acked Write 9 Sent Done Done false
4503 2008 Acked Read 9 Sent Done Done false
4504 2009 Acked Write 6 Sent Done Done false
4505 2010 Acked Read 7 Sent Done Done false
4506 2011 Acked Read 10 Sent Done Done false
4507 2012 Acked Read 7 Sent Done Done false
4508 2013 Acked Write 3 Sent Done Done false
4509 2014 Acked Write 8 Sent Done Done false
4510 2015 Acked Read 10 Sent Done Done false
4511 2016 Acked Flush 0 Sent Done Done false
4512 2017 Acked Write 8 Sent Done Done false
4513 2018 Acked Write 5 Sent Done Done false
4514 2019 Acked Write 8 Sent Done Done false
4515 2020 Acked Write 8 Sent Done Done false
4516 2021 Acked Read 10 Sent Done Done false
4517 2022 Acked Write 10 Sent Done Done false
4518 2023 Acked Flush 0 Sent Done Done false
4519 2024 Acked Write 7 Sent Done Done false
4520 2025 Acked Write 2 Sent Done Done false
4521 2026 Acked Read 9 Sent Done Done false
4522 2027 Acked Read 8 Sent Done Done false
4523 2028 Acked Read 4 Sent Done Done false
4524 2029 Acked Read 2 Sent Done Done false
4525 2030 Acked Write 2 Sent Done Done false
4526 2031 Acked Write 6 Sent Done Done false
4527 2032 Acked Write 7 Sent Done Done false
4528 2033 Acked Read 9 Sent Done Done false
4529 2034 Acked Read 9 Sent Done Done false
4530 2035 Acked Read 7 Sent Done Done false
4531 2036 Acked Read 1 Sent Done Done false
4532 2037 Acked Read 10 Sent Done Done false
4533 2038 Acked Write 10 Sent Done Done false
4534 2039 Acked Write 10 Sent Done Done false
4535 2040 Acked Read 6 Sent Done Done false
4536 2041 Acked Read 7 Sent Done Done false
4537 2042 Acked Flush 0 Sent Done Done false
4538 2043 Acked Write 5 Sent Done Done false
4539 2044 Acked Write 5 Sent Done Done false
4540 2045 Acked Write 9 Sent Done Done false
4541 2046 Acked Read 10 Sent Done Done false
4542 2047 Acked Read 9 Sent Done Done false
4543 2048 Acked Read 2 Sent Done Done false
4544 2049 Acked Write 1 Sent Done Done false
4545 2050 Acked Write 9 Sent Done Done false
4546 2051 Acked Write 3 Sent Done Done false
4547 2052 Acked Write 9 Sent Done Done false
4548 2053 Acked Read 5 Sent Done Done false
4549 2054 Acked Write 6 Sent Done Done false
4550 2055 Acked Read 1 Sent Done Done false
4551 2056 Acked Read 8 Sent Done Done false
4552 2057 Acked Write 4 Sent Done Done false
4553 2058 Acked Read 9 Sent Done Done false
4554 2059 Acked Write 5 Sent Done Done false
4555 2060 Acked Read 10 Sent Done Done false
4556 2061 Acked Read 5 Sent Done Done false
4557 2062 Acked Read 5 Sent Done Done false
4558 2063 Acked Write 10 Sent Done Done false
4559 2064 Acked Read 9 Sent Done Done false
4560 2065 Acked Read 5 Sent Done Done false
4561 2066 Acked Write 6 Sent Done Done false
4562 2067 Acked Read 2 Sent Done Done false
4563 2068 Acked Write 9 Sent Done Done false
4564 2069 Acked Write 2 Sent Done Done false
4565 2070 Acked Write 7 Sent Done Done false
4566 2071 Acked Flush 0 Sent Done Done false
4567 2072 Acked Write 2 Sent Done Done false
4568 2073 Acked Read 8 Sent Done Done false
4569 2074 Acked Write 5 Sent Done Done false
4570 2075 Acked Write 6 Sent Done Done false
4571 2076 Acked Read 4 Sent Done Done false
4572 2077 Acked Write 4 Sent Done Done false
4573 2078 Acked Read 10 Sent Done Done false
4574 2079 Acked Read 9 Sent Done Done false
4575 2080 Acked Read 8 Sent Done Done false
4576 2081 Acked Write 7 Sent Done Done false
4577 2082 Acked Read 10 Sent Done Done false
4578 2083 Acked Read 2 Sent Done Done false
4579 2084 Acked Read 8 Sent Done Done false
4580 2085 Acked Write 7 Sent Done Done false
4581 2086 Acked Write 5 Sent Done Done false
4582 2087 Acked Write 1 Sent Done Done false
4583 2088 Acked Read 1 Sent Done Done false
4584 2089 Acked Read 5 Sent Done Done false
4585 2090 Acked Write 4 Sent Done Done false
4586 2091 Acked Flush 0 Sent Done Done false
4587 2092 Acked Write 8 Sent Done Done false
4588 2093 Acked Flush 0 Sent Done Done false
4589 2094 Acked Write 1 Sent Done Done false
4590 2095 Acked Write 5 Sent Done Done false
4591 2096 Acked Write 9 Sent Done Done false
4592 2097 Acked Read 4 Sent Done Done false
4593 2098 Acked Flush 0 Sent Done Done false
4594 2099 Acked Flush 0 Sent Done Done false
4595 2100 Acked Read 9 Sent Done Done false
4596 2101 Acked Write 10 Sent Done Done false
4597 2102 Acked Write 7 Sent Done Done false
4598 2103 Acked Flush 0 Sent Done Done false
4599 2104 Acked Write 10 Sent Done Done false
4600 2105 Acked Write 8 Sent Done Done false
4601 2106 Acked Flush 0 Sent Done Done false
4602 2107 Acked Flush 0 Sent Done Done false
4603 2108 Acked Read 8 Sent Done Done false
4604 2109 Acked Write 3 Sent Done Done false
4605 2110 Acked Read 10 Sent Done Done false
4606 2111 Acked Write 4 Sent Done Done false
4607 2112 Acked Write 8 Sent Done Done false
4608 2113 Acked Read 5 Sent Done Done false
4609 2114 Acked Write 3 Sent Done Done false
4610 2115 Acked Read 2 Sent Done Done false
4611 2116 Acked Write 10 Sent Done Done false
4612 2117 Acked Flush 0 Sent Done Done false
4613 2118 Acked Read 7 Sent Done Done false
4614 2119 Acked Read 8 Sent Done Done false
4615 2120 Acked Write 9 Sent Done Done false
4616 2121 Acked Read 7 Sent Done Done false
4617 2122 Acked Write 1 Sent Done Done false
4618 2123 Acked Write 6 Sent Done Done false
4619 2124 Acked Read 2 Sent Done Done false
4620 2125 Acked Read 10 Sent Done Done false
4621 2126 Acked Read 8 Sent Done Done false
4622 2127 Acked Write 1 Sent Done Done false
4623 2128 Acked Write 5 Sent Done Done false
4624 2129 Acked Read 3 Sent Done Done false
4625 2130 Acked Write 4 Sent Done Done false
4626 2131 Acked Read 4 Sent Done Done false
4627 2132 Acked Read 6 Sent Done Done false
4628 2133 Acked Read 1 Sent Done Done false
4629 2134 Acked Read 2 Sent Done Done false
4630 2135 Acked Read 6 Sent Done Done false
4631 2136 Acked Write 7 Sent Done Done false
4632 2137 Acked Read 3 Sent Done Done false
4633 2138 Acked Read 5 Sent Done Done false
4634 2139 Acked Write 7 Sent Done Done false
4635 2140 Acked Write 9 Sent Done Done false
4636 2141 Acked Read 4 Sent Done Done false
4637 2142 Acked Read 8 Sent Done Done false
4638 2143 Acked Write 9 Sent Done Done false
4639 2144 Acked Write 5 Sent Done Done false
4640 2145 Acked Read 6 Sent Done Done false
4641 2146 Acked Write 7 Sent Done Done false
4642 2147 Acked Read 3 Sent Done Done false
4643 2148 Acked Write 1 Sent Done Done false
4644 2149 Acked Write 5 Sent Done Done false
4645 2150 Acked Read 2 Sent Done Done false
4646 2151 Acked Read 1 Sent Done Done false
4647 2152 Acked Read 6 Sent Done Done false
4648 2153 Acked Read 2 Sent Done Done false
4649 2154 Acked Write 1 Sent Done Done false
4650 2155 Acked Write 7 Sent Done Done false
4651 2156 Acked Write 8 Sent Done Done false
4652 2157 Acked Flush 0 Sent Done Done false
4653 2158 Acked Write 9 Sent Done Done false
4654 2159 Acked Read 5 Sent Done Done false
4655 2160 Acked Read 4 Sent Done Done false
4656 2161 Acked Write 2 Sent Done Done false
4657 2162 Acked Read 2 Sent Done Done false
4658 2163 Acked Write 4 Sent Done Done false
4659 2164 Acked Read 10 Sent Done Done false
4660 2165 Acked Read 9 Sent Done Done false
4661 2166 Acked Write 7 Sent Done Done false
4662 2167 Acked Read 5 Sent Done Done false
4663 2168 Acked Write 3 Sent Done Done false
4664 2169 Acked Write 9 Sent Done Done false
4665 2170 Acked Read 1 Sent Done Done false
4666 2171 Acked Write 6 Sent Done Done false
4667 2172 Acked Write 4 Sent Done Done false
4668 2173 Acked Read 7 Sent Done Done false
4669 2174 Acked Read 8 Sent Done Done false
4670 2175 Acked Read 7 Sent Done Done false
4671 2176 Acked Read 10 Sent Done Done false
4672 2177 Acked Write 1 Sent Done Done false
4673 2178 Acked Write 7 Sent Done Done false
4674 2179 Acked Write 4 Sent Done Done false
4675 2180 Acked Read 1 Sent Done Done false
4676 2181 Acked Write 2 Sent Done Done false
4677 2182 Acked Read 6 Sent Done Done false
4678 2183 Acked Flush 0 Sent Done Done false
4679 2184 Acked Read 3 Sent Done Done false
4680 2185 Acked Write 7 Sent Done Done false
4681 2186 Acked Write 1 Sent Done Done false
4682 2187 Acked Flush 0 Sent Done Done false
4683 2188 Acked Write 10 Sent Done Done false
4684 2189 Acked Write 5 Sent Done Done false
4685 2190 Acked Write 3 Sent Done Done false
4686 2191 Acked Write 7 Sent Done Done false
4687 2192 Acked Write 6 Sent Done Done false
4688 2193 Acked Read 8 Sent Done Done false
4689 2194 Acked Read 1 Sent Done Done false
4690 2195 Acked Write 3 Sent Done Done false
4691 2196 Acked Write 8 Sent Done Done false
4692 2197 Acked Flush 0 Sent Done Done false
4693 2198 Acked Read 3 Sent Done Done false
4694 2199 Acked Read 10 Sent Done Done false
4695 2200 Acked Read 10 Sent Done Done false
4696 2201 Acked Read 2 Sent Done Done false
4697 2202 Acked Write 7 Sent Done Done false
4698 2203 Acked Read 4 Sent Done Done false
4699 2204 Acked Write 1 Sent Done Done false
4700 2205 Acked Write 7 Sent Done Done false
4701 2206 Acked Read 8 Sent Done Done false
4702 2207 Acked Flush 0 Sent Done Done false
4703 STATES DS:0 DS:1 DS:2 TOTAL
4704 Sent 302 0 0 302
4705 Done 0 302 302 604
4706 Skipped 0 0 0 0
4707 Error 0 0 0 0
4708 Last Flush: Some(JobId(1905)) Some(JobId(2207)) Some(JobId(2207))
4709 Downstairs last five completed: 1905 1904 1903 1902 1901
4710 Upstairs last five completed: 2207 2206 2205 2204 2203
4711 CLIENT: Up:0 ds:302 act:2
47122025-01-23T17:38:42.374ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
47132025-01-23T17:38:42.374ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41736) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
47142025-01-23T17:38:42.376ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
47152025-01-23T17:38:42.376ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
47162025-01-23T17:38:42.376ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
47172025-01-23T17:38:42.376ZINFOcrucible: send last flush ID to this DS: Some(JobId(1905)) = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
47182025-01-23T17:38:42.376ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
47192025-01-23T17:38:42.376ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(1905)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
47202025-01-23T17:38:42.376ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(1905)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
4721 ----------------------------------------------------------------
4722 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4723 Downstairs last five completed: 2207 2206 2205 2204 2203
4724 Upstairs last five completed: 2207 2206 2205 2204 2203
4725 CLIENT: Up:0 ds:0 act:3
4726 Replay: All jobs finished
4727 Main task has work to do, go find it
4728 got dsc Stop(0)
4729 stop 0
4730 Jan 23 17:38:44.616 INFO request completed, latency_us:[8810] Got stop action so:false kr:false
4731 376, response_code: 204, uri: /stop/cid/0, method: POST, req_id: bd408ef3-e5e6-4f48-b85a-00a07ff24751, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
4732 Jan 23 17:38:44.617 INFO request completed{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:38:44.617341128Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
4733 , latency_us: 117, response_code: 200, uri: /state/cid/0, method{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:38:44.61742469Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
4734 : GET, req_id: 2ce3559a-777d-4ee0-ba18-144d24e93ce4, remote_addr{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:38:44.617477765Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
4735 : 127.0.0.1:42015, local_addr: 127.0.0.1:9998
47362025-01-23T17:38:44.617ZINFOcrucible: client task is exiting = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
4737 [8810] Exited with: ExitStatus(unix_wait_status(9))
4738 [8810][0] reports Exit
4739 Jan 23 17:38:48.620 INFO request completed, latency_us: 372, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1cc8427a-cbc2-4f74-80f9-86a3792cbf27, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
4740 001/300 Read block 1437 len 36864
4741 002/300 Write block 760 len 12288 data: 3 3 2
4742 003/300 Read block 4381 len 36864
4743 004/300 Read block 1415 len 32768
4744 005/300 Flush
4745 006/300 Read block 2425 len 16384
4746 007/300 Write block 889 len 4096 data: 2
4747 008/300 Write block 2112 len 4096 data: 3
4748 009/300 Write block 1514 len 8192 data: 3 3
4749 010/300 Write block 4616 len 32768 data: 3 3 4 4 3 3 3 2
4750 011/300 Read block 2224 len 4096
4751 012/300 Read block 721 len 12288
4752 013/300 Write block 1257 len 32768 data: 4 4 4 4 4 4 4 3
4753 014/300 Read block 1380 len 8192
4754 015/300 Write block 2145 len 24576 data: 2 2 2 2 2 3
4755 016/300 Read block 1672 len 28672
4756 017/300 Write block 1264 len 8192 data: 4 3
4757 018/300 Read block 2781 len 8192
4758 019/300 Read block 1678 len 40960
4759 020/300 Read block 1460 len 4096
4760 021/300 Write block 2996 len 24576 data: 3 3 2 2 2 2
4761 022/300 Read block 2010 len 40960
4762 023/300 Read block 1141 len 20480
4763 024/300 Read block 1388 len 12288
4764 025/300 Read block 1962 len 4096
4765 026/300 Read block 3824 len 32768
4766 027/300 Write block 80 len 20480 data: 3 2 3 3 3
4767 028/300 Flush
4768 029/300 Flush
4769 030/300 Read block 1131 len 36864
4770 031/300 Write block 1644 len 40960 data: 2 2 2 2 2 2 2 2 2 3
4771 032/300 Write block 2417 len 24576 data: 2 2 2 2 2 2
4772 033/300 Read block 3363 len 28672
4773 034/300 Read block 4819 len 20480
4774 035/300 Read block 2733 len 24576
4775 036/300 Read block 1983 len 4096
4776 037/300 Write block 4203 len 12288 data: 3 3 4
4777 038/300 Read block 2086 len 8192
4778 039/300 Write block 1861 len 32768 data: 3 4 3 3 3 4 4 3
4779 040/300 Read block 2328 len 28672
4780 041/300 Read block 4970 len 12288
4781 042/300 Write block 252 len 12288 data: 2 2 2
4782 043/300 Write block 825 len 32768 data: 4 4 4 3 3 2 2 4
4783 044/300 Read block 2740 len 16384
4784 045/300 Flush
4785 046/300 Write block 1179 len 20480 data: 2 2 2 2 2
4786 047/300 Read block 4389 len 32768
4787 048/300 Write block 3960 len 16384 data: 3 4 4 4
4788 049/300 Write block 2758 len 36864 data: 2 2 2 2 2 2 2 2 2
4789 050/300 Read block 3627 len 12288
4790 051/300 Write block 4827 len 12288 data: 2 2 2
4791 052/300 Flush
4792 053/300 Write block 785 len 4096 data: 2
4793 054/300 Write block 833 len 28672 data: 4 4 3 3 3 3 4
4794 055/300 Write block 3611 len 24576 data: 4 4 4 3 3 3
4795 056/300 Read block 4700 len 20480
4796 057/300 Write block 3792 len 36864 data: 2 2 2 2 2 2 2 2 2
4797 058/300 Read block 4417 len 28672
4798 059/300 Flush
4799 060/300 Read block 4434 len 4096
4800 061/300 Read block 3177 len 24576
4801 062/300 Flush
4802 063/300 Flush
4803 064/300 Read block 1863 len 4096
4804 065/300 Read block 2596 len 4096
4805 066/300 Read block 2400 len 28672
4806 067/300 Read block 2166 len 28672
4807 068/300 Write block 3706 len 20480 data: 2 2 2 2 2
4808 069/300 Read block 170 len 8192
4809 070/300 Read block 1724 len 36864
4810 071/300 Write block 543 len 32768 data: 4 3 3 3 2 2 2 2
4811 072/300 Read block 2033 len 24576
4812 073/300 Read block 2034 len 24576
4813 074/300 Write block 165 len 24576 data: 3 3 3 2 2 2
4814 075/300 Write block 4176 len 32768 data: 2 2 2 3 3 3 2 2
4815 076/300 Read block 159 len 40960
4816 077/300 Flush
4817 078/300 Write block 2413 len 28672 data: 2 2 2 2 3 3 3
4818 079/300 Write block 3662 len 32768 data: 3 3 2 2 2 3 3 4
4819 080/300 Read block 963 len 16384
4820 081/300 Write block 1142 len 32768 data: 4 3 3 3 2 2 2 2
4821 082/300 Flush
4822 083/300 Write block 1094 len 40960 data: 2 2 2 3 3 3 3 3 3 3
4823 084/300 Write block 3473 len 32768 data: 2 2 2 2 2 2 2 2
4824 085/300 Write block 784 len 8192 data: 2 3
4825 086/300 Read block 4903 len 36864
4826 087/300 Write block 2603 len 40960 data: 3 4 3 3 3 3 2 3 3 3
4827 088/300 Read block 4496 len 16384
4828 089/300 Write block 3564 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4829 090/300 Read block 2869 len 20480
4830 091/300 Read block 2577 len 4096
4831 092/300 Read block 3778 len 36864
4832 093/300 Write block 4550 len 28672 data: 2 2 2 2 2 2 3
4833 094/300 Read block 2958 len 40960
4834 095/300 Write block 1254 len 28672 data: 2 2 2 5 5 5 5
4835 096/300 Read block 2670 len 24576
4836 097/300 Read block 2983 len 16384
4837 098/300 Read block 2478 len 20480
4838 099/300 Write block 1577 len 4096 data: 3
4839 100/300 Read block 2932 len 12288
4840 101/300 Write block 274 len 8192 data: 2 2
4841 102/300 Write block 4723 len 4096 data: 3
4842 103/300 Read block 1889 len 8192
4843 104/300 Read block 538 len 8192
4844 105/300 Write block 2950 len 24576 data: 2 2 2 2 3 3
4845 106/300 Read block 3651 len 32768
4846 107/300 Read block 138 len 8192
4847 108/300 Flush
4848 109/300 Read block 2125 len 4096
4849 110/300 Flush
4850 111/300 Read block 4085 len 20480
4851 112/300 Flush
4852 113/300 Read block 776 len 36864
4853 114/300 Read block 3053 len 24576
4854 115/300 Read block 2174 len 16384
4855 116/300 Write block 1732 len 32768 data: 2 2 3 2 2 2 2 2
4856 117/300 Read block 1987 len 16384
4857 118/300 Write block 3503 len 36864 data: 2 2 2 2 2 2 3 3 3
4858 119/300 Write block 1900 len 28672 data: 2 2 2 2 4 4 4
4859 120/300 Read block 3708 len 28672
4860 121/300 Write block 3701 len 16384 data: 2 2 2 2
4861 122/300 Write block 600 len 36864 data: 2 2 2 2 2 2 2 2 2
4862 123/300 Read block 3183 len 8192
4863 124/300 Write block 738 len 8192 data: 2 2
4864 125/300 Write block 27 len 40960 data: 2 2 2 2 2 2 2 2 2 3
4865 126/300 Flush
4866 127/300 Flush
4867 128/300 Flush
4868 129/300 Read block 4955 len 20480
4869 130/300 Write block 3429 len 32768 data: 3 4 4 4 5 4 4 3
4870 131/300 Write block 3286 len 12288 data: 2 2 2
4871 132/300 Write block 3385 len 4096 data: 2
4872 133/300 Write block 1550 len 20480 data: 2 2 2 2 2
4873 134/300 Read block 2974 len 28672
4874 135/300 Read block 1348 len 4096
4875 136/300 Read block 1819 len 36864
4876 137/300 Read block 130 len 32768
4877 138/300 Write block 3584 len 28672 data: 3 3 3 2 2 2 3
4878 139/300 Write block 1555 len 4096 data: 2
4879 140/300 Write block 2857 len 4096 data: 4
4880 141/300 Write block 4986 len 4096 data: 3
4881 142/300 Read block 3181 len 36864
4882 143/300 Read block 2451 len 24576
4883 144/300 Write block 1893 len 40960 data: 2 3 3 2 2 2 2 3 3 3
4884 145/300 Read block 3714 len 32768
4885 146/300 Read block 758 len 20480
4886 147/300 Read block 1659 len 40960
4887 148/300 Write block 3803 len 8192 data: 2 2
4888 149/300 Read block 4589 len 28672
4889 150/300 Read block 4070 len 36864
4890 151/300 Read block 972 len 32768
4891 152/300 Write block 4032 len 8192 data: 3 3
4892 153/300 Read block 788 len 16384
4893 154/300 Read block 4850 len 24576
4894 155/300 Flush
4895 156/300 Write block 2014 len 4096 data: 2
4896 157/300 Write block 348 len 16384 data: 2 2 2 2
4897 158/300 Read block 1297 len 4096
4898 159/300 Flush
4899 160/300 Write block 1964 len 28672 data: 4 4 3 2 2 3 3
4900 161/300 Write block 1750 len 28672 data: 3 3 3 3 2 3 3
4901 162/300 Read block 2671 len 28672
4902 163/300 Write block 1235 len 12288 data: 3 3 3
4903 164/300 Write block 3429 len 28672 data: 4 5 5 5 6 5 5
4904 165/300 Read block 2014 len 32768
4905 166/300 Flush
4906 167/300 Write block 3346 len 16384 data: 3 3 3 3
4907 168/300 Read block 2163 len 12288
4908 169/300 Write block 1504 len 36864 data: 2 2 3 3 3 3 4 3 3
4909 170/300 Write block 4435 len 36864 data: 4 4 3 3 3 3 3 3 3
4910 171/300 Write block 3343 len 24576 data: 2 2 2 4 4 4
4911 172/300 Flush
4912 173/300 Write block 140 len 16384 data: 2 2 2 2
4913 174/300 Read block 1856 len 24576
4914 175/300 Flush
4915 176/300 Write block 2443 len 36864 data: 3 3 3 3 3 3 3 3 3
4916 177/300 Read block 3720 len 20480
4917 178/300 Write block 1165 len 4096 data: 3
4918 179/300 Read block 1752 len 36864
4919 180/300 Write block 3510 len 36864 data: 4 4 3 3 3 3 3 3 3
4920 181/300 Read block 84 len 32768
4921 182/300 Write block 3838 len 16384 data: 4 4 3 3
4922 183/300 Write block 3920 len 24576 data: 3 3 3 3 3 2
4923 184/300 Write block 4248 len 16384 data: 3 2 2 2
4924 185/300 Write block 4341 len 36864 data: 2 2 2 2 2 2 2 2 2
4925 186/300 Write block 4845 len 20480 data: 2 2 2 2 2
4926 187/300 Write block 970 len 24576 data: 2 2 2 2 2 2
4927 188/300 Flush
4928 189/300 Flush
4929 190/300 Read block 2697 len 12288
4930 191/300 Read block 3889 len 28672
4931 192/300 Read block 309 len 28672
4932 193/300 Read block 3111 len 28672
4933 194/300 Read block 2877 len 28672
4934 195/300 Read block 1036 len 4096
4935 196/300 Write block 2069 len 32768 data: 4 3 3 3 3 2 2 2
4936 197/300 Write block 810 len 12288 data: 2 2 3
4937 198/300 Read block 1121 len 36864
4938 199/300 Read block 1890 len 20480
4939 200/300 Write block 2636 len 20480 data: 2 2 2 3 3
4940 201/300 Write block 1615 len 20480 data: 2 3 3 3 3
4941 202/300 Write block 4049 len 24576 data: 3 2 2 2 2 2
4942 203/300 Read block 2358 len 12288
4943 204/300 Write block 4280 len 36864 data: 3 3 3 4 4 3 2 2 2
4944 205/300 Read block 1911 len 12288
4945 206/300 Read block 3144 len 28672
4946 207/300 Read block 4530 len 16384
4947 208/300 Write block 4014 len 20480 data: 2 2 2 2 2
4948 209/300 Flush
4949 210/300 Read block 1597 len 28672
4950 211/300 Write block 3110 len 40960 data: 3 4 3 3 3 3 3 3 2 2
4951 212/300 Read block 496 len 40960
4952 213/300 Write block 3836 len 36864 data: 4 4 5 5 4 4 2 2 2
4953 214/300 Read block 178 len 40960
4954 215/300 Write block 390 len 20480 data: 3 3 2 2 2
4955 216/300 Read block 4898 len 24576
4956 217/300 Read block 1354 len 12288
4957 218/300 Write block 2007 len 40960 data: 2 2 3 3 4 4 3 3 2 2
4958 219/300 Write block 2999 len 20480 data: 3 3 3 2 2
4959 220/300 Write block 4271 len 24576 data: 4 4 3 3 3 2
4960 221/300 Read block 3987 len 36864
4961 222/300 Write block 2590 len 24576 data: 3 3 3 3 3 2
4962 223/300 Read block 180 len 36864
4963 224/300 Read block 186 len 4096
4964 225/300 Read block 4376 len 32768
4965 226/300 Read block 2700 len 4096
4966 227/300 Read block 4998 len 8192
4967 228/300 Write block 3700 len 36864 data: 2 3 3 3 3 2 3 3 3
4968 229/300 Write block 3982 len 12288 data: 2 2 2
4969 230/300 Read block 3903 len 20480
4970 231/300 Read block 1235 len 20480
4971 232/300 Write block 2859 len 36864 data: 4 3 3 3 3 3 2 3 3
4972 233/300 Read block 3860 len 12288
4973 234/300 Write block 4726 len 28672 data: 4 4 3 2 2 2 2
4974 235/300 Write block 4667 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4975 236/300 Write block 3840 len 20480 data: 5 5 3 3 3
4976 237/300 Read block 183 len 8192
4977 238/300 Write block 640 len 4096 data: 2
4978 239/300 Read block 2066 len 28672
4979 240/300 Flush
4980 241/300 Read block 3327 len 8192
4981 242/300 Read block 2443 len 4096
4982 243/300 Read block 2007 len 24576
4983 244/300 Write block 107 len 40960 data: 2 2 2 4 4 4 5 5 5 6
4984 245/300 Read block 4259 len 20480
4985 246/300 Write block 2867 len 12288 data: 4 3 3
4986 247/300 Write block 1647 len 24576 data: 3 3 3 3 3 3
4987 248/300 Write block 3589 len 8192 data: 3 4
4988 249/300 Write block 1189 len 20480 data: 2 2 2 2 2
4989 250/300 Read block 3002 len 12288
4990 251/300 Write block 3012 len 32768 data: 3 3 3 2 2 2 2 2
4991 252/300 Write block 765 len 8192 data: 3 3
4992 253/300 Read block 1131 len 40960
4993 254/300 Read block 3841 len 28672
4994 255/300 Write block 4268 len 32768 data: 4 4 4 5 5 4 4 4
4995 256/300 Read block 684 len 4096
4996 257/300 Read block 2886 len 12288
4997 258/300 Read block 4256 len 28672
4998 259/300 Read block 3732 len 16384
4999 260/300 Write block 1162 len 12288 data: 4 4 4
5000 261/300 Write block 3817 len 28672 data: 2 2 2 2 2 2 2
5001 262/300 Write block 3762 len 20480 data: 2 2 2 2 2
5002 263/300 Flush
5003 264/300 Read block 2979 len 24576
5004 265/300 Flush
5005 266/300 Read block 1151 len 32768
5006 267/300 Flush
5007 268/300 Write block 4670 len 36864 data: 3 3 3 3 3 3 3 3 3
5008 269/300 Write block 3938 len 24576 data: 3 2 3 3 3 3
5009 270/300 Read block 2782 len 20480
5010 271/300 Read block 3083 len 36864
5011 272/300 Read block 2032 len 28672
5012 273/300 Flush
5013 274/300 Write block 3542 len 12288 data: 3 3 3
5014 275/300 Read block 2458 len 32768
5015 276/300 Read block 3814 len 24576
5016 277/300 Read block 4588 len 12288
5017 278/300 Read block 1150 len 16384
5018 279/300 Write block 4618 len 24576 data: 5 5 4 4 4 3
5019 280/300 Write block 3615 len 36864 data: 4 4 3 3 3 3 2 2 3
5020 281/300 Read block 1547 len 36864
5021 282/300 Write block 4177 len 28672 data: 3 3 4 4 4 3 3
5022 283/300 Write block 4914 len 12288 data: 3 3 2
5023 284/300 Read block 2026 len 24576
5024 285/300 Read block 16 len 40960
5025 286/300 Read block 4348 len 16384
5026 287/300 Read block 462 len 20480
5027 288/300 Write block 3468 len 4096 data: 2
5028 289/300 Flush
5029 290/300 Read block 1490 len 32768
5030 291/300 Write block 4288 len 16384 data: 3 2 2 2
5031 292/300 Read block 4492 len 4096
5032 293/300 Read block 3606 len 36864
5033 294/300 Read block 2976 len 36864
5034 295/300 Write block 3953 len 20480 data: 2 2 2 2 2
5035 296/300 Read block 2971 len 32768
5036 297/300 Write block 3523 len 20480 data: 4 3 3 3 2
5037 298/300 Write block 4798 len 8192 data: 2 2
5038 299/300 Read block 2813 len 32768
5039 300/300 Read block 414 len 8192
5040 301/300 Read block 4118 len 24576
5041 Main task has work to do, go find it
5042 got dsc Start(0)
5043 start 0
5044 Jan 23 17:38:48.861 INFO request completed, latency_us: 192, response_code: 204, uri: /start/cid/0, method: POST, req_id: 9f791401-3a99-4106-b3c8-50b51b9270be, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
5045 Replay: started 0, returned:Ok(())
5046 ----------------------------------------------------------------
5047 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5048 States: Offline Active Active
5049 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5050 2208 Acked Read 9 Sent Done Done false
5051 2209 Acked Write 3 Sent Done Done false
5052 2210 Acked Read 9 Sent Done Done false
5053 2211 Acked Read 8 Sent Done Done false
5054 2212 Acked Flush 0 Sent Done Done false
5055 2213 Acked Read 4 Sent Done Done false
5056 2214 Acked Write 1 Sent Done Done false
5057 2215 Acked Write 1 Sent Done Done false
5058 2216 Acked Write 2 Sent Done Done false
5059 2217 Acked Write 8 Sent Done Done false
5060 2218 Acked Read 1 Sent Done Done false
5061 2219 Acked Read 3 Sent Done Done false
5062 2220 Acked Write 8 Sent Done Done false
5063 2221 Acked Read 2 Sent Done Done false
5064 2222 Acked Write 6 Sent Done Done false
5065 2223 Acked Read 7 Sent Done Done false
5066 2224 Acked Write 2 Sent Done Done false
5067 2225 Acked Read 2 Sent Done Done false
5068 2226 Acked Read 10 Sent Done Done false
5069 2227 Acked Read 1 Sent Done Done false
5070 2228 Acked Write 6 Sent Done Done false
5071 2229 Acked Read 10 Sent Done Done false
5072 2230 Acked Read 5 Sent Done Done false
5073 2231 Acked Read 3 Sent Done Done false
5074 2232 Acked Read 1 Sent Done Done false
5075 2233 Acked Read 8 Sent Done Done false
5076 2234 Acked Write 5 Sent Done Done false
5077 2235 Acked Flush 0 Sent Done Done false
5078 2236 Acked Flush 0 Sent Done Done false
5079 2237 Acked Read 9 Sent Done Done false
5080 2238 Acked Write 10 Sent Done Done false
5081 2239 Acked Write 6 Sent Done Done false
5082 2240 Acked Read 7 Sent Done Done false
5083 2241 Acked Read 5 Sent Done Done false
5084 2242 Acked Read 6 Sent Done Done false
5085 2243 Acked Read 1 Sent Done Done false
5086 2244 Acked Write 3 Sent Done Done false
5087 2245 Acked Read 2 Sent Done Done false
5088 2246 Acked Write 8 Sent Done Done false
5089 2247 Acked Read 7 Sent Done Done false
5090 2248 Acked Read 3 Sent Done Done false
5091 2249 Acked Write 3 Sent Done Done false
5092 2250 Acked Write 8 Sent Done Done false
5093 2251 Acked Read 4 Sent Done Done false
5094 2252 Acked Flush 0 Sent Done Done false
5095 2253 Acked Write 5 Sent Done Done false
5096 2254 Acked Read 8 Sent Done Done false
5097 2255 Acked Write 4 Sent Done Done false
5098 2256 Acked Write 9 Sent Done Done false
5099 2257 Acked Read 3 Sent Done Done false
5100 2258 Acked Write 3 Sent Done Done false
5101 2259 Acked Flush 0 Sent Done Done false
5102 2260 Acked Write 1 Sent Done Done false
5103 2261 Acked Write 7 Sent Done Done false
5104 2262 Acked Write 6 Sent Done Done false
5105 2263 Acked Read 5 Sent Done Done false
5106 2264 Acked Write 9 Sent Done Done false
5107 2265 Acked Read 7 Sent Done Done false
5108 2266 Acked Flush 0 Sent Done Done false
5109 2267 Acked Read 1 Sent Done Done false
5110 2268 Acked Read 6 Sent Done Done false
5111 2269 Acked Flush 0 Sent Done Done false
5112 2270 Acked Flush 0 Sent Done Done false
5113 2271 Acked Read 1 Sent Done Done false
5114 2272 Acked Read 1 Sent Done Done false
5115 2273 Acked Read 7 Sent Done Done false
5116 2274 Acked Read 7 Sent Done Done false
5117 2275 Acked Write 5 Sent Done Done false
5118 2276 Acked Read 2 Sent Done Done false
5119 2277 Acked Read 9 Sent Done Done false
5120 2278 Acked Write 8 Sent Done Done false
5121 2279 Acked Read 6 Sent Done Done false
5122 2280 Acked Read 6 Sent Done Done false
5123 2281 Acked Write 6 Sent Done Done false
5124 2282 Acked Write 8 Sent Done Done false
5125 2283 Acked Read 10 Sent Done Done false
5126 2284 Acked Flush 0 Sent Done Done false
5127 2285 Acked Write 7 Sent Done Done false
5128 2286 Acked Write 8 Sent Done Done false
5129 2287 Acked Read 4 Sent Done Done false
5130 2288 Acked Write 8 Sent Done Done false
5131 2289 Acked Flush 0 Sent Done Done false
5132 2290 Acked Write 10 Sent Done Done false
5133 2291 Acked Write 8 Sent Done Done false
5134 2292 Acked Write 2 Sent Done Done false
5135 2293 Acked Read 9 Sent Done Done false
5136 2294 Acked Write 10 Sent Done Done false
5137 2295 Acked Read 4 Sent Done Done false
5138 2296 Acked Write 10 Sent Done Done false
5139 2297 Acked Read 5 Sent Done Done false
5140 2298 Acked Read 1 Sent Done Done false
5141 2299 Acked Read 9 Sent Done Done false
5142 2300 Acked Write 7 Sent Done Done false
5143 2301 Acked Read 10 Sent Done Done false
5144 2302 Acked Write 7 Sent Done Done false
5145 2303 Acked Read 6 Sent Done Done false
5146 2304 Acked Read 4 Sent Done Done false
5147 2305 Acked Read 5 Sent Done Done false
5148 2306 Acked Write 1 Sent Done Done false
5149 2307 Acked Read 3 Sent Done Done false
5150 2308 Acked Write 2 Sent Done Done false
5151 2309 Acked Write 1 Sent Done Done false
5152 2310 Acked Read 2 Sent Done Done false
5153 2311 Acked Read 2 Sent Done Done false
5154 2312 Acked Write 6 Sent Done Done false
5155 2313 Acked Read 8 Sent Done Done false
5156 2314 Acked Read 2 Sent Done Done false
5157 2315 Acked Flush 0 Sent Done Done false
5158 2316 Acked Read 1 Sent Done Done false
5159 2317 Acked Flush 0 Sent Done Done false
5160 2318 Acked Read 5 Sent Done Done false
5161 2319 Acked Flush 0 Sent Done Done false
5162 2320 Acked Read 9 Sent Done Done false
5163 2321 Acked Read 6 Sent Done Done false
5164 2322 Acked Read 4 Sent Done Done false
5165 2323 Acked Write 8 Sent Done Done false
5166 2324 Acked Read 4 Sent Done Done false
5167 2325 Acked Write 9 Sent Done Done false
5168 2326 Acked Write 7 Sent Done Done false
5169 2327 Acked Read 7 Sent Done Done false
5170 2328 Acked Write 4 Sent Done Done false
5171 2329 Acked Write 9 Sent Done Done false
5172 2330 Acked Read 2 Sent Done Done false
5173 2331 Acked Write 2 Sent Done Done false
5174 2332 Acked Write 10 Sent Done Done false
5175 2333 Acked Flush 0 Sent Done Done false
5176 2334 Acked Flush 0 Sent Done Done false
5177 2335 Acked Flush 0 Sent Done Done false
5178 2336 Acked Read 5 Sent Done Done false
5179 2337 Acked Write 8 Sent Done Done false
5180 2338 Acked Write 3 Sent Done Done false
5181 2339 Acked Write 1 Sent Done Done false
5182 2340 Acked Write 5 Sent Done Done false
5183 2341 Acked Read 7 Sent Done Done false
5184 2342 Acked Read 1 Sent Done Done false
5185 2343 Acked Read 9 Sent Done Done false
5186 2344 Acked Read 8 Sent Done Done false
5187 2345 Acked Write 7 Sent Done Done false
5188 2346 Acked Write 1 Sent Done Done false
5189 2347 Acked Write 1 Sent Done Done false
5190 2348 Acked Write 1 Sent Done Done false
5191 2349 Acked Read 9 Sent Done Done false
5192 2350 Acked Read 6 Sent Done Done false
5193 2351 Acked Write 10 Sent Done Done false
5194 2352 Acked Read 8 Sent Done Done false
5195 2353 Acked Read 5 Sent Done Done false
5196 2354 Acked Read 10 Sent Done Done false
5197 2355 Acked Write 2 Sent Done Done false
5198 2356 Acked Read 7 Sent Done Done false
5199 2357 Acked Read 9 Sent Done Done false
5200 2358 Acked Read 8 Sent Done Done false
5201 2359 Acked Write 2 Sent Done Done false
5202 2360 Acked Read 4 Sent Done Done false
5203 2361 Acked Read 6 Sent Done Done false
5204 2362 Acked Flush 0 Sent Done Done false
5205 2363 Acked Write 1 Sent Done Done false
5206 2364 Acked Write 4 Sent Done Done false
5207 2365 Acked Read 1 Sent Done Done false
5208 2366 Acked Flush 0 Sent Done Done false
5209 2367 Acked Write 7 Sent Done Done false
5210 2368 Acked Write 7 Sent Done Done false
5211 2369 Acked Read 7 Sent Done Done false
5212 2370 Acked Write 3 Sent Done Done false
5213 2371 Acked Write 7 Sent Done Done false
5214 2372 Acked Read 8 Sent Done Done false
5215 2373 Acked Flush 0 Sent Done Done false
5216 2374 Acked Write 4 Sent Done Done false
5217 2375 Acked Read 3 Sent Done Done false
5218 2376 Acked Write 9 Sent Done Done false
5219 2377 Acked Write 9 Sent Done Done false
5220 2378 Acked Write 6 Sent Done Done false
5221 2379 Acked Flush 0 Sent Done Done false
5222 2380 Acked Write 4 Sent Done Done false
5223 2381 Acked Read 6 Sent Done Done false
5224 2382 Acked Flush 0 Sent Done Done false
5225 2383 Acked Write 9 Sent Done Done false
5226 2384 Acked Read 5 Sent Done Done false
5227 2385 Acked Write 1 Sent Done Done false
5228 2386 Acked Read 9 Sent Done Done false
5229 2387 Acked Write 9 Sent Done Done false
5230 2388 Acked Read 8 Sent Done Done false
5231 2389 Acked Write 4 Sent Done Done false
5232 2390 Acked Write 6 Sent Done Done false
5233 2391 Acked Write 4 Sent Done Done false
5234 2392 Acked Write 9 Sent Done Done false
5235 2393 Acked Write 5 Sent Done Done false
5236 2394 Acked Write 6 Sent Done Done false
5237 2395 Acked Flush 0 Sent Done Done false
5238 2396 Acked Flush 0 Sent Done Done false
5239 2397 Acked Read 3 Sent Done Done false
5240 2398 Acked Read 7 Sent Done Done false
5241 2399 Acked Read 7 Sent Done Done false
5242 2400 Acked Read 7 Sent Done Done false
5243 2401 Acked Read 7 Sent Done Done false
5244 2402 Acked Read 1 Sent Done Done false
5245 2403 Acked Write 8 Sent Done Done false
5246 2404 Acked Write 3 Sent Done Done false
5247 2405 Acked Read 9 Sent Done Done false
5248 2406 Acked Read 5 Sent Done Done false
5249 2407 Acked Write 5 Sent Done Done false
5250 2408 Acked Write 5 Sent Done Done false
5251 2409 Acked Write 6 Sent Done Done false
5252 2410 Acked Read 3 Sent Done Done false
5253 2411 Acked Write 9 Sent Done Done false
5254 2412 Acked Read 3 Sent Done Done false
5255 2413 Acked Read 7 Sent Done Done false
5256 2414 Acked Read 4 Sent Done Done false
5257 2415 Acked Write 5 Sent Done Done false
5258 2416 Acked Flush 0 Sent Done Done false
5259 2417 Acked Read 7 Sent Done Done false
5260 2418 Acked Write 10 Sent Done Done false
5261 2419 Acked Read 10 Sent Done Done false
5262 2420 Acked Write 9 Sent Done Done false
5263 2421 Acked Read 10 Sent Done Done false
5264 2422 Acked Write 5 Sent Done Done false
5265 2423 Acked Read 6 Sent Done Done false
5266 2424 Acked Read 3 Sent Done Done false
5267 2425 Acked Write 10 Sent Done Done false
5268 2426 Acked Write 5 Sent Done Done false
5269 2427 Acked Write 6 Sent Done Done false
5270 2428 Acked Read 9 Sent Done Done false
5271 2429 Acked Write 6 Sent Done Done false
5272 2430 Acked Read 9 Sent Done Done false
5273 2431 Acked Read 1 Sent Done Done false
5274 2432 Acked Read 8 Sent Done Done false
5275 2433 Acked Read 1 Sent Done Done false
5276 2434 Acked Read 2 Sent Done Done false
5277 2435 Acked Write 9 Sent Done Done false
5278 2436 Acked Write 3 Sent Done Done false
5279 2437 Acked Read 5 Sent Done Done false
5280 2438 Acked Read 5 Sent Done Done false
5281 2439 Acked Write 9 Sent Done Done false
5282 2440 Acked Read 3 Sent Done Done false
5283 2441 Acked Write 7 Sent Done Done false
5284 2442 Acked Write 10 Sent Done Done false
5285 2443 Acked Write 5 Sent Done Done false
5286 2444 Acked Read 2 Sent Done Done false
5287 2445 Acked Write 1 Sent Done Done false
5288 2446 Acked Read 7 Sent Done Done false
5289 2447 Acked Flush 0 Sent Done Done false
5290 2448 Acked Read 2 Sent Done Done false
5291 2449 Acked Read 1 Sent Done Done false
5292 2450 Acked Read 6 Sent Done Done false
5293 2451 Acked Write 10 Sent Done Done false
5294 2452 Acked Read 5 Sent Done Done false
5295 2453 Acked Write 3 Sent Done Done false
5296 2454 Acked Write 6 Sent Done Done false
5297 2455 Acked Write 2 Sent Done Done false
5298 2456 Acked Write 5 Sent Done Done false
5299 2457 Acked Read 3 Sent Done Done false
5300 2458 Acked Write 8 Sent Done Done false
5301 2459 Acked Write 2 Sent Done Done false
5302 2460 Acked Read 10 Sent Done Done false
5303 2461 Acked Read 7 Sent Done Done false
5304 2462 Acked Write 8 Sent Done Done false
5305 2463 Acked Read 1 Sent Done Done false
5306 2464 Acked Read 3 Sent Done Done false
5307 2465 Acked Read 7 Sent Done Done false
5308 2466 Acked Read 4 Sent Done Done false
5309 2467 Acked Write 3 Sent Done Done false
5310 2468 Acked Write 7 Sent Done Done false
5311 2469 Acked Write 5 Sent Done Done false
5312 2470 Acked Flush 0 Sent Done Done false
5313 2471 Acked Read 6 Sent Done Done false
5314 2472 Acked Flush 0 Sent Done Done false
5315 2473 Acked Read 8 Sent Done Done false
5316 2474 Acked Flush 0 Sent Done Done false
5317 2475 Acked Write 9 Sent Done Done false
5318 2476 Acked Write 6 Sent Done Done false
5319 2477 Acked Read 5 Sent Done Done false
5320 2478 Acked Read 9 Sent Done Done false
5321 2479 Acked Read 7 Sent Done Done false
5322 2480 Acked Flush 0 Sent Done Done false
5323 2481 Acked Write 3 Sent Done Done false
5324 2482 Acked Read 8 Sent Done Done false
5325 2483 Acked Read 6 Sent Done Done false
5326 2484 Acked Read 3 Sent Done Done false
5327 2485 Acked Read 4 Sent Done Done false
5328 2486 Acked Write 6 Sent Done Done false
5329 2487 Acked Write 9 Sent Done Done false
5330 2488 Acked Read 9 Sent Done Done false
5331 2489 Acked Write 7 Sent Done Done false
5332 2490 Acked Write 3 Sent Done Done false
5333 2491 Acked Read 6 Sent Done Done false
5334 2492 Acked Read 10 Sent Done Done false
5335 2493 Acked Read 4 Sent Done Done false
5336 2494 Acked Read 5 Sent Done Done false
5337 2495 Acked Write 1 Sent Done Done false
5338 2496 Acked Flush 0 Sent Done Done false
5339 2497 Acked Read 8 Sent Done Done false
5340 2498 Acked Write 4 Sent Done Done false
5341 2499 Acked Read 1 Sent Done Done false
5342 2500 Acked Read 9 Sent Done Done false
5343 2501 Acked Read 9 Sent Done Done false
5344 2502 Acked Write 5 Sent Done Done false
5345 2503 Acked Read 8 Sent Done Done false
5346 2504 Acked Write 5 Sent Done Done false
5347 2505 Acked Write 2 Sent Done Done false
5348 2506 Acked Read 8 Sent Done Done false
5349 2507 Acked Read 2 Sent Done Done false
5350 2508 Acked Read 6 Sent Done Done false
5351 STATES DS:0 DS:1 DS:2 TOTAL
5352 Sent 301 0 0 301
5353 Done 0 301 301 602
5354 Skipped 0 0 0 0
5355 Error 0 0 0 0
5356 Last Flush: Some(JobId(2207)) Some(JobId(2496)) Some(JobId(2496))
5357 Downstairs last five completed: 2207 2206 2205 2204 2203
5358 Upstairs last five completed: 2508 2507 2506 2505 2504
5359 CLIENT: Up:0 ds:301 act:2
5360 [8810] got start action
5361 [8810] I am going to restart
5362 Starting downstairs at port 8810
5363 Make output file at "/tmp/dsc/downstairs-8810.txt"
5364 [8810][0] reports Starting
5365 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1210)
5366 [8810][0] reports Running
5367 ----------------------------------------------------------------
5368 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
5369 States: Offline Active Active
5370 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5371 2208 Acked Read 9 Sent Done Done false
5372 2209 Acked Write 3 Sent Done Done false
5373 2210 Acked Read 9 Sent Done Done false
5374 2211 Acked Read 8 Sent Done Done false
5375 2212 Acked Flush 0 Sent Done Done false
5376 2213 Acked Read 4 Sent Done Done false
5377 2214 Acked Write 1 Sent Done Done false
5378 2215 Acked Write 1 Sent Done Done false
5379 2216 Acked Write 2 Sent Done Done false
5380 2217 Acked Write 8 Sent Done Done false
5381 2218 Acked Read 1 Sent Done Done false
5382 2219 Acked Read 3 Sent Done Done false
5383 2220 Acked Write 8 Sent Done Done false
5384 2221 Acked Read 2 Sent Done Done false
5385 2222 Acked Write 6 Sent Done Done false
5386 2223 Acked Read 7 Sent Done Done false
5387 2224 Acked Write 2 Sent Done Done false
5388 2225 Acked Read 2 Sent Done Done false
5389 2226 Acked Read 10 Sent Done Done false
5390 2227 Acked Read 1 Sent Done Done false
5391 2228 Acked Write 6 Sent Done Done false
5392 2229 Acked Read 10 Sent Done Done false
5393 2230 Acked Read 5 Sent Done Done false
5394 2231 Acked Read 3 Sent Done Done false
5395 2232 Acked Read 1 Sent Done Done false
5396 2233 Acked Read 8 Sent Done Done false
5397 2234 Acked Write 5 Sent Done Done false
5398 2235 Acked Flush 0 Sent Done Done false
5399 2236 Acked Flush 0 Sent Done Done false
5400 2237 Acked Read 9 Sent Done Done false
5401 2238 Acked Write 10 Sent Done Done false
5402 2239 Acked Write 6 Sent Done Done false
5403 2240 Acked Read 7 Sent Done Done false
5404 2241 Acked Read 5 Sent Done Done false
5405 2242 Acked Read 6 Sent Done Done false
5406 2243 Acked Read 1 Sent Done Done false
5407 2244 Acked Write 3 Sent Done Done false
5408 2245 Acked Read 2 Sent Done Done false
5409 2246 Acked Write 8 Sent Done Done false
5410 2247 Acked Read 7 Sent Done Done false
5411 2248 Acked Read 3 Sent Done Done false
5412 2249 Acked Write 3 Sent Done Done false
5413 2250 Acked Write 8 Sent Done Done false
5414 2251 Acked Read 4 Sent Done Done false
5415 2252 Acked Flush 0 Sent Done Done false
5416 2253 Acked Write 5 Sent Done Done false
5417 2254 Acked Read 8 Sent Done Done false
5418 2255 Acked Write 4 Sent Done Done false
5419 2256 Acked Write 9 Sent Done Done false
5420 2257 Acked Read 3 Sent Done Done false
5421 2258 Acked Write 3 Sent Done Done false
5422 2259 Acked Flush 0 Sent Done Done false
5423 2260 Acked Write 1 Sent Done Done false
5424 2261 Acked Write 7 Sent Done Done false
5425 2262 Acked Write 6 Sent Done Done false
5426 2263 Acked Read 5 Sent Done Done false
5427 2264 Acked Write 9 Sent Done Done false
5428 2265 Acked Read 7 Sent Done Done false
5429 2266 Acked Flush 0 Sent Done Done false
5430 2267 Acked Read 1 Sent Done Done false
5431 2268 Acked Read 6 Sent Done Done false
5432 2269 Acked Flush 0 Sent Done Done false
5433 2270 Acked Flush 0 Sent Done Done false
5434 2271 Acked Read 1 Sent Done Done false
5435 2272 Acked Read 1 Sent Done Done false
5436 2273 Acked Read 7 Sent Done Done false
5437 2274 Acked Read 7 Sent Done Done false
5438 2275 Acked Write 5 Sent Done Done false
5439 2276 Acked Read 2 Sent Done Done false
5440 2277 Acked Read 9 Sent Done Done false
5441 2278 Acked Write 8 Sent Done Done false
5442 2279 Acked Read 6 Sent Done Done false
5443 2280 Acked Read 6 Sent Done Done false
5444 2281 Acked Write 6 Sent Done Done false
5445 2282 Acked Write 8 Sent Done Done false
5446 2283 Acked Read 10 Sent Done Done false
5447 2284 Acked Flush 0 Sent Done Done false
5448 2285 Acked Write 7 Sent Done Done false
5449 2286 Acked Write 8 Sent Done Done false
5450 2287 Acked Read 4 Sent Done Done false
5451 2288 Acked Write 8 Sent Done Done false
5452 2289 Acked Flush 0 Sent Done Done false
5453 2290 Acked Write 10 Sent Done Done false
5454 2291 Acked Write 8 Sent Done Done false
5455 2292 Acked Write 2 Sent Done Done false
5456 2293 Acked Read 9 Sent Done Done false
5457 2294 Acked Write 10 Sent Done Done false
5458 2295 Acked Read 4 Sent Done Done false
5459 2296 Acked Write 10 Sent Done Done false
5460 2297 Acked Read 5 Sent Done Done false
5461 2298 Acked Read 1 Sent Done Done false
5462 2299 Acked Read 9 Sent Done Done false
5463 2300 Acked Write 7 Sent Done Done false
5464 2301 Acked Read 10 Sent Done Done false
5465 2302 Acked Write 7 Sent Done Done false
5466 2303 Acked Read 6 Sent Done Done false
5467 2304 Acked Read 4 Sent Done Done false
5468 2305 Acked Read 5 Sent Done Done false
5469 2306 Acked Write 1 Sent Done Done false
5470 2307 Acked Read 3 Sent Done Done false
5471 2308 Acked Write 2 Sent Done Done false
5472 2309 Acked Write 1 Sent Done Done false
5473 2310 Acked Read 2 Sent Done Done false
5474 2311 Acked Read 2 Sent Done Done false
5475 2312 Acked Write 6 Sent Done Done false
5476 2313 Acked Read 8 Sent Done Done false
5477 2314 Acked Read 2 Sent Done Done false
5478 2315 Acked Flush 0 Sent Done Done false
5479 2316 Acked Read 1 Sent Done Done false
5480 2317 Acked Flush 0 Sent Done Done false
5481 2318 Acked Read 5 Sent Done Done false
5482 2319 Acked Flush 0 Sent Done Done false
5483 2320 Acked Read 9 Sent Done Done false
5484 2321 Acked Read 6 Sent Done Done false
5485 2322 Acked Read 4 Sent Done Done false
5486 2323 Acked Write 8 Sent Done Done false
5487 2324 Acked Read 4 Sent Done Done false
5488 2325 Acked Write 9 Sent Done Done false
5489 2326 Acked Write 7 Sent Done Done false
5490 2327 Acked Read 7 Sent Done Done false
5491 2328 Acked Write 4 Sent Done Done false
5492 2329 Acked Write 9 Sent Done Done false
5493 2330 Acked Read 2 Sent Done Done false
5494 2331 Acked Write 2 Sent Done Done false
5495 2332 Acked Write 10 Sent Done Done false
5496 2333 Acked Flush 0 Sent Done Done false
5497 2334 Acked Flush 0 Sent Done Done false
5498 2335 Acked Flush 0 Sent Done Done false
5499 2336 Acked Read 5 Sent Done Done false
5500 2337 Acked Write 8 Sent Done Done false
5501 2338 Acked Write 3 Sent Done Done false
5502 2339 Acked Write 1 Sent Done Done false
5503 2340 Acked Write 5 Sent Done Done false
5504 2341 Acked Read 7 Sent Done Done false
5505 2342 Acked Read 1 Sent Done Done false
5506 2343 Acked Read 9 Sent Done Done false
5507 2344 Acked Read 8 Sent Done Done false
5508 2345 Acked Write 7 Sent Done Done false
5509 2346 Acked Write 1 Sent Done Done false
5510 2347 Acked Write 1 Sent Done Done false
5511 2348 Acked Write 1 Sent Done Done false
5512 2349 Acked Read 9 Sent Done Done false
5513 2350 Acked Read 6 Sent Done Done false
5514 2351 Acked Write 10 Sent Done Done false
5515 2352 Acked Read 8 Sent Done Done false
5516 2353 Acked Read 5 Sent Done Done false
5517 2354 Acked Read 10 Sent Done Done false
5518 2355 Acked Write 2 Sent Done Done false
5519 2356 Acked Read 7 Sent Done Done false
5520 2357 Acked Read 9 Sent Done Done false
5521 2358 Acked Read 8 Sent Done Done false
5522 2359 Acked Write 2 Sent Done Done false
5523 2360 Acked Read 4 Sent Done Done false
5524 2361 Acked Read 6 Sent Done Done false
5525 2362 Acked Flush 0 Sent Done Done false
5526 2363 Acked Write 1 Sent Done Done false
5527 2364 Acked Write 4 Sent Done Done false
5528 2365 Acked Read 1 Sent Done Done false
5529 2366 Acked Flush 0 Sent Done Done false
5530 2367 Acked Write 7 Sent Done Done false
5531 2368 Acked Write 7 Sent Done Done false
5532 2369 Acked Read 7 Sent Done Done false
5533 2370 Acked Write 3 Sent Done Done false
5534 2371 Acked Write 7 Sent Done Done false
5535 2372 Acked Read 8 Sent Done Done false
5536 2373 Acked Flush 0 Sent Done Done false
5537 2374 Acked Write 4 Sent Done Done false
5538 2375 Acked Read 3 Sent Done Done false
5539 2376 Acked Write 9 Sent Done Done false
5540 2377 Acked Write 9 Sent Done Done false
5541 2378 Acked Write 6 Sent Done Done false
5542 2379 Acked Flush 0 Sent Done Done false
5543 2380 Acked Write 4 Sent Done Done false
5544 2381 Acked Read 6 Sent Done Done false
5545 2382 Acked Flush 0 Sent Done Done false
5546 2383 Acked Write 9 Sent Done Done false
5547 2384 Acked Read 5 Sent Done Done false
5548 2385 Acked Write 1 Sent Done Done false
5549 2386 Acked Read 9 Sent Done Done false
5550 2387 Acked Write 9 Sent Done Done false
5551 2388 Acked Read 8 Sent Done Done false
5552 2389 Acked Write 4 Sent Done Done false
5553 2390 Acked Write 6 Sent Done Done false
5554 2391 Acked Write 4 Sent Done Done false
5555 2392 Acked Write 9 Sent Done Done false
5556 2393 Acked Write 5 Sent Done Done false
5557 2394 Acked Write 6 Sent Done Done false
5558 2395 Acked Flush 0 Sent Done Done false
5559 2396 Acked Flush 0 Sent Done Done false
5560 2397 Acked Read 3 Sent Done Done false
5561 2398 Acked Read 7 Sent Done Done false
5562 2399 Acked Read 7 Sent Done Done false
5563 2400 Acked Read 7 Sent Done Done false
5564 2401 Acked Read 7 Sent Done Done false
5565 2402 Acked Read 1 Sent Done Done false
5566 2403 Acked Write 8 Sent Done Done false
5567 2404 Acked Write 3 Sent Done Done false
5568 2405 Acked Read 9 Sent Done Done false
5569 2406 Acked Read 5 Sent Done Done false
5570 2407 Acked Write 5 Sent Done Done false
5571 2408 Acked Write 5 Sent Done Done false
5572 2409 Acked Write 6 Sent Done Done false
5573 2410 Acked Read 3 Sent Done Done false
5574 2411 Acked Write 9 Sent Done Done false
5575 2412 Acked Read 3 Sent Done Done false
5576 2413 Acked Read 7 Sent Done Done false
5577 2414 Acked Read 4 Sent Done Done false
5578 2415 Acked Write 5 Sent Done Done false
5579 2416 Acked Flush 0 Sent Done Done false
5580 2417 Acked Read 7 Sent Done Done false
5581 2418 Acked Write 10 Sent Done Done false
5582 2419 Acked Read 10 Sent Done Done false
5583 2420 Acked Write 9 Sent Done Done false
5584 2421 Acked Read 10 Sent Done Done false
5585 2422 Acked Write 5 Sent Done Done false
5586 2423 Acked Read 6 Sent Done Done false
5587 2424 Acked Read 3 Sent Done Done false
5588 2425 Acked Write 10 Sent Done Done false
5589 2426 Acked Write 5 Sent Done Done false
5590 2427 Acked Write 6 Sent Done Done false
5591 2428 Acked Read 9 Sent Done Done false
5592 2429 Acked Write 6 Sent Done Done false
5593 2430 Acked Read 9 Sent Done Done false
5594 2431 Acked Read 1 Sent Done Done false
5595 2432 Acked Read 8 Sent Done Done false
5596 2433 Acked Read 1 Sent Done Done false
5597 2434 Acked Read 2 Sent Done Done false
5598 2435 Acked Write 9 Sent Done Done false
5599 2436 Acked Write 3 Sent Done Done false
5600 2437 Acked Read 5 Sent Done Done false
5601 2438 Acked Read 5 Sent Done Done false
5602 2439 Acked Write 9 Sent Done Done false
5603 2440 Acked Read 3 Sent Done Done false
5604 2441 Acked Write 7 Sent Done Done false
5605 2442 Acked Write 10 Sent Done Done false
5606 2443 Acked Write 5 Sent Done Done false
5607 2444 Acked Read 2 Sent Done Done false
5608 2445 Acked Write 1 Sent Done Done false
5609 2446 Acked Read 7 Sent Done Done false
5610 2447 Acked Flush 0 Sent Done Done false
5611 2448 Acked Read 2 Sent Done Done false
5612 2449 Acked Read 1 Sent Done Done false
5613 2450 Acked Read 6 Sent Done Done false
5614 2451 Acked Write 10 Sent Done Done false
5615 2452 Acked Read 5 Sent Done Done false
5616 2453 Acked Write 3 Sent Done Done false
5617 2454 Acked Write 6 Sent Done Done false
5618 2455 Acked Write 2 Sent Done Done false
5619 2456 Acked Write 5 Sent Done Done false
5620 2457 Acked Read 3 Sent Done Done false
5621 2458 Acked Write 8 Sent Done Done false
5622 2459 Acked Write 2 Sent Done Done false
5623 2460 Acked Read 10 Sent Done Done false
5624 2461 Acked Read 7 Sent Done Done false
5625 2462 Acked Write 8 Sent Done Done false
5626 2463 Acked Read 1 Sent Done Done false
5627 2464 Acked Read 3 Sent Done Done false
5628 2465 Acked Read 7 Sent Done Done false
5629 2466 Acked Read 4 Sent Done Done false
5630 2467 Acked Write 3 Sent Done Done false
5631 2468 Acked Write 7 Sent Done Done false
5632 2469 Acked Write 5 Sent Done Done false
5633 2470 Acked Flush 0 Sent Done Done false
5634 2471 Acked Read 6 Sent Done Done false
5635 2472 Acked Flush 0 Sent Done Done false
5636 2473 Acked Read 8 Sent Done Done false
5637 2474 Acked Flush 0 Sent Done Done false
5638 2475 Acked Write 9 Sent Done Done false
5639 2476 Acked Write 6 Sent Done Done false
5640 2477 Acked Read 5 Sent Done Done false
5641 2478 Acked Read 9 Sent Done Done false
5642 2479 Acked Read 7 Sent Done Done false
5643 2480 Acked Flush 0 Sent Done Done false
5644 2481 Acked Write 3 Sent Done Done false
5645 2482 Acked Read 8 Sent Done Done false
5646 2483 Acked Read 6 Sent Done Done false
5647 2484 Acked Read 3 Sent Done Done false
5648 2485 Acked Read 4 Sent Done Done false
5649 2486 Acked Write 6 Sent Done Done false
5650 2487 Acked Write 9 Sent Done Done false
5651 2488 Acked Read 9 Sent Done Done false
5652 2489 Acked Write 7 Sent Done Done false
5653 2490 Acked Write 3 Sent Done Done false
5654 2491 Acked Read 6 Sent Done Done false
5655 2492 Acked Read 10 Sent Done Done false
5656 2493 Acked Read 4 Sent Done Done false
5657 2494 Acked Read 5 Sent Done Done false
5658 2495 Acked Write 1 Sent Done Done false
5659 2496 Acked Flush 0 Sent Done Done false
5660 2497 Acked Read 8 Sent Done Done false
5661 2498 Acked Write 4 Sent Done Done false
5662 2499 Acked Read 1 Sent Done Done false
5663 2500 Acked Read 9 Sent Done Done false
5664 2501 Acked Read 9 Sent Done Done false
5665 2502 Acked Write 5 Sent Done Done false
5666 2503 Acked Read 8 Sent Done Done false
5667 2504 Acked Write 5 Sent Done Done false
5668 2505 Acked Write 2 Sent Done Done false
5669 2506 Acked Read 8 Sent Done Done false
5670 2507 Acked Read 2 Sent Done Done false
5671 2508 Acked Read 6 Sent Done Done false
5672 2509 Acked Flush 0 Sent Done Done false
5673 STATES DS:0 DS:1 DS:2 TOTAL
5674 Sent 302 0 0 302
5675 Done 0 302 302 604
5676 Skipped 0 0 0 0
5677 Error 0 0 0 0
5678 Last Flush: Some(JobId(2207)) Some(JobId(2509)) Some(JobId(2509))
5679 Downstairs last five completed: 2207 2206 2205 2204 2203
5680 Upstairs last five completed: 2509 2508 2507 2506 2505
5681 CLIENT: Up:0 ds:302 act:2
56822025-01-23T17:38:54.618ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
56832025-01-23T17:38:54.618ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55172) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
56842025-01-23T17:38:54.619ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
56852025-01-23T17:38:54.620ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
56862025-01-23T17:38:54.620ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
56872025-01-23T17:38:54.620ZINFOcrucible: send last flush ID to this DS: Some(JobId(2207)) = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
56882025-01-23T17:38:54.620ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
56892025-01-23T17:38:54.620ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(2207)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
56902025-01-23T17:38:54.620ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(2207)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
5691 ----------------------------------------------------------------
5692 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5693 Downstairs last five completed: 2509 2508 2507 2506 2505
5694 Upstairs last five completed: 2509 2508 2507 2506 2505
5695 CLIENT: Up:0 ds:0 act:3
5696 Replay: All jobs finished
5697 Main task has work to do, go find it
5698 got dsc Stop(0)
5699 stop 0
5700 Jan 23 17:38:56.867 INFO request completed, latency_us: 394, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 86c4129d-bd31-4c63-9772-6c75bf54ea9b, remote_addr: 127[8810] Got stop action so:false kr:false
5701 .0.0.1:42015, local_addr: 127.0.0.1:9998
57022025-01-23T17:38:56.868ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
57032025-01-23T17:38:56.868ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
57042025-01-23T17:38:56.868ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
5705 Jan 23 17:38:56.868 INFO request completed, latency_us: 148, response_code: 200, uri: /state/cid/0, method: GET, req_id: aaf20be3-7900-4c66-a244-4f46d0317dbb, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
57062025-01-23T17:38:56.868ZINFOcrucible: client task is exiting = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
5707 [8810] Exited with: ExitStatus(unix_wait_status(9))
5708 [8810][0] reports Exit
5709 Jan 23 17:39:00.870 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/0, method: GET, req_id: e133d943-dc55-487b-8b67-f9ad3a46ef7f, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
5710 001/300 Write block 1247 len 16384 data: 2 2 2 2
5711 002/300 Write block 2252 len 24576 data: 2 2 2 2 3 3
5712 003/300 Write block 2335 len 40960 data: 3 3 3 3 2 3 2 3 3 3
5713 004/300 Read block 573 len 36864
5714 005/300 Read block 2931 len 8192
5715 006/300 Read block 4886 len 4096
5716 007/300 Write block 3405 len 12288 data: 2 2 2
5717 008/300 Read block 52 len 20480
5718 009/300 Read block 579 len 4096
5719 010/300 Write block 668 len 20480 data: 2 2 2 2 2
5720 011/300 Read block 2043 len 20480
5721 012/300 Read block 3907 len 4096
5722 013/300 Flush
5723 014/300 Read block 2112 len 24576
5724 015/300 Write block 1618 len 16384 data: 4 4 3 2
5725 016/300 Read block 2048 len 28672
5726 017/300 Write block 1853 len 20480 data: 2 3 3 3 3
5727 018/300 Write block 1611 len 12288 data: 2 2 2
5728 019/300 Flush
5729 020/300 Flush
5730 021/300 Read block 2594 len 12288
5731 022/300 Write block 3106 len 28672 data: 2 2 2 2 4 5 4
5732 023/300 Read block 1020 len 28672
5733 024/300 Read block 2685 len 40960
5734 025/300 Write block 1558 len 12288 data: 2 2 2
5735 026/300 Write block 3805 len 16384 data: 2 3 3 3
5736 027/300 Read block 4474 len 8192
5737 028/300 Write block 2062 len 24576 data: 4 4 4 2 2 3
5738 029/300 Read block 3703 len 16384
5739 030/300 Write block 2191 len 40960 data: 3 3 4 4 4 4 4 4 3 3
5740 031/300 Write block 2670 len 4096 data: 2
5741 032/300 Flush
5742 033/300 Read block 4864 len 36864
5743 034/300 Read block 4544 len 36864
5744 035/300 Write block 3420 len 40960 data: 3 2 2 2 2 3 3 3 3 5
5745 036/300 Write block 3157 len 24576 data: 2 2 2 2 2 2
5746 037/300 Read block 1673 len 16384
5747 038/300 Read block 4496 len 24576
5748 039/300 Write block 181 len 32768 data: 2 2 2 2 2 2 2 2
5749 040/300 Flush
5750 041/300 Read block 4863 len 28672
5751 042/300 Write block 2159 len 20480 data: 2 2 2 2 2
5752 043/300 Read block 2529 len 8192
5753 044/300 Read block 607 len 16384
5754 045/300 Write block 4326 len 28672 data: 2 2 2 2 2 2 2
5755 046/300 Write block 3848 len 4096 data: 3
5756 047/300 Write block 988 len 40960 data: 4 4 3 3 3 2 2 3 3 3
5757 048/300 Write block 3544 len 36864 data: 4 3 3 3 3 4 4 4 4
5758 049/300 Read block 841 len 4096
5759 050/300 Read block 263 len 36864
5760 051/300 Read block 4147 len 12288
5761 052/300 Read block 3062 len 16384
5762 053/300 Read block 4757 len 8192
5763 054/300 Write block 4066 len 4096 data: 2
5764 055/300 Read block 2168 len 24576
5765 056/300 Flush
5766 057/300 Read block 2866 len 40960
5767 058/300 Write block 447 len 8192 data: 3 3
5768 059/300 Read block 112 len 40960
5769 060/300 Flush
5770 061/300 Read block 2978 len 4096
5771 062/300 Read block 4818 len 24576
5772 063/300 Read block 2721 len 16384
5773 064/300 Read block 1818 len 8192
5774 065/300 Read block 1 len 28672
5775 066/300 Write block 4760 len 36864 data: 3 3 3 3 3 2 2 3 3
5776 067/300 Write block 2990 len 4096 data: 3
5777 068/300 Read block 3030 len 16384
5778 069/300 Write block 4256 len 24576 data: 2 2 2 2 2 2
5779 070/300 Write block 2354 len 20480 data: 2 3 3 4 3
5780 071/300 Read block 4954 len 16384
5781 072/300 Read block 1094 len 12288
5782 073/300 Read block 4750 len 24576
5783 074/300 Read block 697 len 4096
5784 075/300 Write block 3489 len 8192 data: 2 2
5785 076/300 Write block 2620 len 28672 data: 2 2 2 2 3 2 2
5786 077/300 Read block 3212 len 16384
5787 078/300 Read block 329 len 32768
5788 079/300 Write block 1270 len 8192 data: 2 2
5789 080/300 Write block 1043 len 12288 data: 3 3 3
5790 081/300 Read block 976 len 28672
5791 082/300 Write block 716 len 4096 data: 3
5792 083/300 Write block 3576 len 40960 data: 2 2 3 3 3 3 3 3 4 4
5793 084/300 Write block 461 len 32768 data: 6 6 5 5 5 5 4 3
5794 085/300 Write block 3113 len 32768 data: 4 4 4 4 4 3 3 2
5795 086/300 Write block 804 len 20480 data: 2 2 2 2 2
5796 087/300 Read block 3248 len 24576
5797 088/300 Read block 3033 len 12288
5798 089/300 Flush
5799 090/300 Read block 2964 len 4096
5800 091/300 Write block 535 len 16384 data: 3 3 3 4
5801 092/300 Flush
5802 093/300 Read block 3363 len 8192
5803 094/300 Write block 549 len 40960 data: 3 3 2 2 2 2 2 2 2 2
5804 095/300 Write block 1711 len 8192 data: 2 2
5805 096/300 Write block 2837 len 36864 data: 3 2 2 2 2 2 2 2 2
5806 097/300 Write block 2845 len 12288 data: 3 2 2
5807 098/300 Write block 4423 len 16384 data: 2 2 2 3
5808 099/300 Write block 1776 len 16384 data: 2 2 2 2
5809 100/300 Write block 4112 len 28672 data: 2 2 2 2 2 2 2
5810 101/300 Write block 1142 len 24576 data: 5 4 4 4 3 3
5811 102/300 Flush
5812 103/300 Write block 34 len 4096 data: 3
5813 104/300 Flush
5814 105/300 Read block 4645 len 36864
5815 106/300 Write block 2982 len 28672 data: 2 2 2 2 3 3 3
5816 107/300 Read block 3014 len 12288
5817 108/300 Write block 313 len 12288 data: 2 2 2
5818 109/300 Read block 1239 len 4096
5819 110/300 Write block 2938 len 28672 data: 2 2 2 3 3 3 3
5820 111/300 Write block 4361 len 40960 data: 3 3 3 3 3 3 3 3 3 4
5821 112/300 Read block 115 len 40960
5822 113/300 Write block 3253 len 12288 data: 2 2 2
5823 114/300 Write block 3491 len 28672 data: 2 2 2 2 2 2 2
5824 115/300 Flush
5825 116/300 Flush
5826 117/300 Read block 4516 len 24576
5827 118/300 Read block 3915 len 40960
5828 119/300 Read block 4337 len 40960
5829 120/300 Read block 4591 len 16384
5830 121/300 Read block 4743 len 36864
5831 122/300 Read block 919 len 32768
5832 123/300 Flush
5833 124/300 Write block 1722 len 4096 data: 2
5834 125/300 Write block 331 len 16384 data: 3 3 3 2
5835 126/300 Write block 4899 len 28672 data: 2 2 2 2 2 2 2
5836 127/300 Write block 2641 len 24576 data: 3 3 3 3 3 3
5837 128/300 Read block 3039 len 28672
5838 129/300 Write block 599 len 40960 data: 2 3 3 3 3 3 3 3 3 3
5839 130/300 Read block 4239 len 8192
5840 131/300 Read block 1279 len 32768
5841 132/300 Write block 2403 len 16384 data: 2 2 2 2
5842 133/300 Read block 2275 len 40960
5843 134/300 Write block 695 len 32768 data: 2 2 2 2 2 2 2 2
5844 135/300 Read block 1571 len 20480
5845 136/300 Write block 4010 len 12288 data: 2 2 2
5846 137/300 Write block 4394 len 8192 data: 2 2
5847 138/300 Read block 3506 len 40960
5848 139/300 Write block 1176 len 36864 data: 2 2 2 3 3 3 3 3 2
5849 140/300 Write block 2032 len 8192 data: 2 2
5850 141/300 Write block 1310 len 28672 data: 4 4 2 2 2 2 2
5851 142/300 Write block 3685 len 20480 data: 2 2 2 2 2
5852 143/300 Read block 900 len 28672
5853 144/300 Read block 3924 len 12288
5854 145/300 Read block 680 len 20480
5855 146/300 Read block 2228 len 40960
5856 147/300 Write block 2376 len 40960 data: 5 5 5 5 4 4 4 4 4 4
5857 148/300 Write block 1619 len 20480 data: 5 4 3 2 3
5858 149/300 Read block 3222 len 16384
5859 150/300 Write block 2453 len 20480 data: 3 3 2 2 3
5860 151/300 Read block 4290 len 24576
5861 152/300 Write block 4515 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5862 153/300 Write block 3636 len 8192 data: 2 2
5863 154/300 Write block 1554 len 4096 data: 3
5864 155/300 Read block 625 len 28672
5865 156/300 Write block 1960 len 32768 data: 3 3 3 3 5 5 4 3
5866 157/300 Read block 3945 len 40960
5867 158/300 Read block 149 len 40960
5868 159/300 Write block 3330 len 40960 data: 2 2 2 2 2 2 2 3 3 3
5869 160/300 Write block 599 len 36864 data: 3 4 4 4 4 4 4 4 4
5870 161/300 Write block 1236 len 20480 data: 4 4 2 2 2
5871 162/300 Read block 4469 len 12288
5872 163/300 Write block 2992 len 8192 data: 3 2
5873 164/300 Read block 212 len 28672
5874 165/300 Write block 3361 len 24576 data: 2 2 2 2 2 2
5875 166/300 Write block 4458 len 28672 data: 2 2 2 2 2 2 2
5876 167/300 Read block 4279 len 20480
5877 168/300 Write block 4936 len 20480 data: 4 3 3 2 2
5878 169/300 Write block 338 len 8192 data: 2 2
5879 170/300 Write block 431 len 4096 data: 2
5880 171/300 Write block 1080 len 24576 data: 2 2 2 2 2 2
5881 172/300 Write block 153 len 20480 data: 3 3 3 3 4
5882 173/300 Read block 1791 len 16384
5883 174/300 Read block 3309 len 12288
5884 175/300 Read block 1572 len 16384
5885 176/300 Write block 615 len 12288 data: 2 3 3
5886 177/300 Flush
5887 178/300 Read block 4809 len 20480
5888 179/300 Read block 4887 len 32768
5889 180/300 Read block 238 len 12288
5890 181/300 Read block 3410 len 32768
5891 182/300 Read block 261 len 16384
5892 183/300 Read block 4950 len 4096
5893 184/300 Write block 2547 len 28672 data: 2 2 2 2 2 2 2
5894 185/300 Write block 524 len 12288 data: 3 3 3
5895 186/300 Write block 824 len 4096 data: 4
5896 187/300 Write block 584 len 40960 data: 2 2 2 2 2 2 2 2 3 3
5897 188/300 Read block 3324 len 28672
5898 189/300 Read block 3363 len 16384
5899 190/300 Write block 3647 len 36864 data: 2 2 2 2 2 2 2 2 3
5900 191/300 Read block 1739 len 20480
5901 192/300 Read block 539 len 20480
5902 193/300 Write block 4316 len 4096 data: 2
5903 194/300 Read block 1988 len 28672
5904 195/300 Flush
5905 196/300 Write block 3849 len 28672 data: 3 3 4 4 5 5 5
5906 197/300 Flush
5907 198/300 Write block 1922 len 24576 data: 3 3 3 2 2 2
5908 199/300 Read block 4944 len 28672
5909 200/300 Read block 1022 len 40960
5910 201/300 Write block 1828 len 40960 data: 3 3 3 2 2 2 2 2 2 2
5911 202/300 Read block 4005 len 36864
5912 203/300 Read block 395 len 4096
5913 204/300 Write block 1905 len 28672 data: 5 5 6 6 5 4 4
5914 205/300 Read block 1545 len 28672
5915 206/300 Read block 3527 len 20480
5916 207/300 Read block 4486 len 36864
5917 208/300 Read block 1339 len 36864
5918 209/300 Read block 3536 len 32768
5919 210/300 Write block 3142 len 40960 data: 3 3 3 3 3 2 2 2 2 2
5920 211/300 Write block 2025 len 4096 data: 2
5921 212/300 Write block 1005 len 36864 data: 3 3 3 2 2 2 2 2 2
5922 213/300 Read block 3559 len 16384
5923 214/300 Write block 1866 len 24576 data: 5 5 4 4 3 2
5924 215/300 Read block 3398 len 16384
5925 216/300 Read block 4410 len 20480
5926 217/300 Write block 2626 len 40960 data: 3 2 3 4 4 4 3 3 2 2
5927 218/300 Write block 2254 len 16384 data: 3 3 4 4
5928 219/300 Write block 1698 len 36864 data: 2 2 2 2 2 2 2 2 2
5929 220/300 Write block 2151 len 8192 data: 3 3
5930 221/300 Read block 1122 len 32768
5931 222/300 Flush
5932 223/300 Write block 3739 len 12288 data: 2 2 2
5933 224/300 Read block 1010 len 28672
5934 225/300 Read block 4862 len 40960
5935 226/300 Read block 2395 len 28672
5936 227/300 Write block 2067 len 32768 data: 4 4 5 4 4 4 4 3
5937 228/300 Read block 3668 len 32768
5938 229/300 Flush
5939 230/300 Write block 3098 len 16384 data: 3 3 3 3
5940 231/300 Read block 4467 len 24576
5941 232/300 Read block 2517 len 36864
5942 233/300 Read block 4730 len 20480
5943 234/300 Read block 4657 len 24576
5944 235/300 Read block 4546 len 4096
5945 236/300 Write block 2046 len 12288 data: 2 2 2
5946 237/300 Write block 1155 len 8192 data: 2 3
5947 238/300 Read block 4831 len 16384
5948 239/300 Read block 1141 len 36864
5949 240/300 Write block 1297 len 24576 data: 3 3 3 3 3 2
5950 241/300 Read block 3691 len 32768
5951 242/300 Read block 818 len 36864
5952 243/300 Read block 4701 len 20480
5953 244/300 Read block 2998 len 8192
5954 245/300 Write block 1304 len 16384 data: 3 3 3 4
5955 246/300 Flush
5956 247/300 Flush
5957 248/300 Write block 201 len 24576 data: 3 2 4 4 4 4
5958 249/300 Read block 2026 len 8192
5959 250/300 Write block 556 len 8192 data: 3 3
5960 251/300 Read block 1446 len 36864
5961 252/300 Read block 473 len 20480
5962 253/300 Write block 1800 len 24576 data: 3 3 3 3 3 3
5963 254/300 Write block 1862 len 28672 data: 5 4 4 4 6 6 5
5964 255/300 Flush
5965 256/300 Read block 3764 len 36864
5966 257/300 Read block 2460 len 28672
5967 258/300 Read block 1275 len 24576
5968 259/300 Write block 4334 len 12288 data: 3 3 3
5969 260/300 Read block 1937 len 36864
5970 261/300 Read block 2971 len 28672
5971 262/300 Read block 3590 len 12288
5972 263/300 Read block 675 len 40960
5973 264/300 Read block 2709 len 16384
5974 265/300 Write block 2322 len 28672 data: 2 2 2 2 2 3 3
5975 266/300 Read block 2323 len 24576
5976 267/300 Write block 103 len 40960 data: 3 3 2 3 3 3 3 5 5 5
5977 268/300 Read block 104 len 36864
5978 269/300 Read block 4638 len 32768
5979 270/300 Read block 3135 len 12288
5980 271/300 Write block 3081 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5981 272/300 Read block 3169 len 36864
5982 273/300 Write block 1360 len 24576 data: 2 2 2 2 3 3
5983 274/300 Write block 356 len 24576 data: 2 2 2 2 2 2
5984 275/300 Flush
5985 276/300 Read block 1448 len 4096
5986 277/300 Read block 2967 len 8192
5987 278/300 Read block 2731 len 12288
5988 279/300 Read block 2070 len 8192
5989 280/300 Write block 1924 len 4096 data: 4
5990 281/300 Write block 1924 len 28672 data: 5 3 3 3 2 2 2
5991 282/300 Read block 3413 len 40960
5992 283/300 Write block 773 len 32768 data: 4 4 3 3 3 3 3 3
5993 284/300 Write block 2286 len 36864 data: 2 2 2 2 2 2 2 2 2
5994 285/300 Write block 1370 len 32768 data: 4 4 2 2 2 2 2 2
5995 286/300 Read block 3247 len 16384
5996 287/300 Read block 890 len 8192
5997 288/300 Read block 1660 len 4096
5998 289/300 Read block 3581 len 24576
5999 290/300 Write block 3568 len 8192 data: 3 3
6000 291/300 Write block 3383 len 32768 data: 2 2 3 2 2 2 2 2
6001 292/300 Read block 49 len 36864
6002 293/300 Read block 1728 len 16384
6003 294/300 Read block 911 len 16384
6004 295/300 Write block 3048 len 36864 data: 2 2 2 2 2 2 2 2 2
6005 296/300 Write block 2827 len 28672 data: 2 2 2 3 3 3 3
6006 297/300 Write block 3726 len 4096 data: 2
6007 298/300 Read block 2617 len 40960
6008 299/300 Read block 1889 len 20480
6009 300/300 Write block 1880 len 24576 data: 3 3 4 4 4 4
6010 301/300 Write block 1565 len 24576 data: 2 2 2 2 3 3
6011 Main task has work to do, go find it
6012 got dsc Start(0)
6013 start 0
6014 Jan 23 17:39:01.114 INFO request completed, latency_us: 264, response_code: 204, uri: /start/cid/0, method: POST, req_id: 285f03fb-dea3-4039-8a32-13145e9d87fc, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
6015 Replay: started 0, returned:Ok(())
6016 ----------------------------------------------------------------
6017 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6018 States: Offline Active Active
6019 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6020 2510 Acked Write 4 Sent Done Done false
6021 2511 Acked Write 6 Sent Done Done false
6022 2512 Acked Write 10 Sent Done Done false
6023 2513 Acked Read 9 Sent Done Done false
6024 2514 Acked Read 2 Sent Done Done false
6025 2515 Acked Read 1 Sent Done Done false
6026 2516 Acked Write 3 Sent Done Done false
6027 2517 Acked Read 5 Sent Done Done false
6028 2518 Acked Read 1 Sent Done Done false
6029 2519 Acked Write 5 Sent Done Done false
6030 2520 Acked Read 5 Sent Done Done false
6031 2521 Acked Read 1 Sent Done Done false
6032 2522 Acked Flush 0 Sent Done Done false
6033 2523 Acked Read 6 Sent Done Done false
6034 2524 Acked Write 4 Sent Done Done false
6035 2525 Acked Read 7 Sent Done Done false
6036 2526 Acked Write 5 Sent Done Done false
6037 2527 Acked Write 3 Sent Done Done false
6038 2528 Acked Flush 0 Sent Done Done false
6039 2529 Acked Flush 0 Sent Done Done false
6040 2530 Acked Read 3 Sent Done Done false
6041 2531 Acked Write 7 Sent Done Done false
6042 2532 Acked Read 7 Sent Done Done false
6043 2533 Acked Read 10 Sent Done Done false
6044 2534 Acked Write 3 Sent Done Done false
6045 2535 Acked Write 4 Sent Done Done false
6046 2536 Acked Read 2 Sent Done Done false
6047 2537 Acked Write 6 Sent Done Done false
6048 2538 Acked Read 4 Sent Done Done false
6049 2539 Acked Write 10 Sent Done Done false
6050 2540 Acked Write 1 Sent Done Done false
6051 2541 Acked Flush 0 Sent Done Done false
6052 2542 Acked Read 9 Sent Done Done false
6053 2543 Acked Read 9 Sent Done Done false
6054 2544 Acked Write 10 Sent Done Done false
6055 2545 Acked Write 6 Sent Done Done false
6056 2546 Acked Read 4 Sent Done Done false
6057 2547 Acked Read 6 Sent Done Done false
6058 2548 Acked Write 8 Sent Done Done false
6059 2549 Acked Flush 0 Sent Done Done false
6060 2550 Acked Read 7 Sent Done Done false
6061 2551 Acked Write 5 Sent Done Done false
6062 2552 Acked Read 2 Sent Done Done false
6063 2553 Acked Read 4 Sent Done Done false
6064 2554 Acked Write 7 Sent Done Done false
6065 2555 Acked Write 1 Sent Done Done false
6066 2556 Acked Write 10 Sent Done Done false
6067 2557 Acked Write 9 Sent Done Done false
6068 2558 Acked Read 1 Sent Done Done false
6069 2559 Acked Read 9 Sent Done Done false
6070 2560 Acked Read 3 Sent Done Done false
6071 2561 Acked Read 4 Sent Done Done false
6072 2562 Acked Read 2 Sent Done Done false
6073 2563 Acked Write 1 Sent Done Done false
6074 2564 Acked Read 6 Sent Done Done false
6075 2565 Acked Flush 0 Sent Done Done false
6076 2566 Acked Read 10 Sent Done Done false
6077 2567 Acked Write 2 Sent Done Done false
6078 2568 Acked Read 10 Sent Done Done false
6079 2569 Acked Flush 0 Sent Done Done false
6080 2570 Acked Read 1 Sent Done Done false
6081 2571 Acked Read 6 Sent Done Done false
6082 2572 Acked Read 4 Sent Done Done false
6083 2573 Acked Read 2 Sent Done Done false
6084 2574 Acked Read 7 Sent Done Done false
6085 2575 Acked Write 9 Sent Done Done false
6086 2576 Acked Write 1 Sent Done Done false
6087 2577 Acked Read 4 Sent Done Done false
6088 2578 Acked Write 6 Sent Done Done false
6089 2579 Acked Write 5 Sent Done Done false
6090 2580 Acked Read 4 Sent Done Done false
6091 2581 Acked Read 3 Sent Done Done false
6092 2582 Acked Read 6 Sent Done Done false
6093 2583 Acked Read 1 Sent Done Done false
6094 2584 Acked Write 2 Sent Done Done false
6095 2585 Acked Write 7 Sent Done Done false
6096 2586 Acked Read 4 Sent Done Done false
6097 2587 Acked Read 8 Sent Done Done false
6098 2588 Acked Write 2 Sent Done Done false
6099 2589 Acked Write 3 Sent Done Done false
6100 2590 Acked Read 7 Sent Done Done false
6101 2591 Acked Write 1 Sent Done Done false
6102 2592 Acked Write 10 Sent Done Done false
6103 2593 Acked Write 8 Sent Done Done false
6104 2594 Acked Write 8 Sent Done Done false
6105 2595 Acked Write 5 Sent Done Done false
6106 2596 Acked Read 6 Sent Done Done false
6107 2597 Acked Read 3 Sent Done Done false
6108 2598 Acked Flush 0 Sent Done Done false
6109 2599 Acked Read 1 Sent Done Done false
6110 2600 Acked Write 4 Sent Done Done false
6111 2601 Acked Flush 0 Sent Done Done false
6112 2602 Acked Read 2 Sent Done Done false
6113 2603 Acked Write 10 Sent Done Done false
6114 2604 Acked Write 2 Sent Done Done false
6115 2605 Acked Write 9 Sent Done Done false
6116 2606 Acked Write 3 Sent Done Done false
6117 2607 Acked Write 4 Sent Done Done false
6118 2608 Acked Write 4 Sent Done Done false
6119 2609 Acked Write 7 Sent Done Done false
6120 2610 Acked Write 6 Sent Done Done false
6121 2611 Acked Flush 0 Sent Done Done false
6122 2612 Acked Write 1 Sent Done Done false
6123 2613 Acked Flush 0 Sent Done Done false
6124 2614 Acked Read 9 Sent Done Done false
6125 2615 Acked Write 7 Sent Done Done false
6126 2616 Acked Read 3 Sent Done Done false
6127 2617 Acked Write 3 Sent Done Done false
6128 2618 Acked Read 1 Sent Done Done false
6129 2619 Acked Write 7 Sent Done Done false
6130 2620 Acked Write 10 Sent Done Done false
6131 2621 Acked Read 10 Sent Done Done false
6132 2622 Acked Write 3 Sent Done Done false
6133 2623 Acked Write 7 Sent Done Done false
6134 2624 Acked Flush 0 Sent Done Done false
6135 2625 Acked Flush 0 Sent Done Done false
6136 2626 Acked Read 6 Sent Done Done false
6137 2627 Acked Read 10 Sent Done Done false
6138 2628 Acked Read 10 Sent Done Done false
6139 2629 Acked Read 4 Sent Done Done false
6140 2630 Acked Read 9 Sent Done Done false
6141 2631 Acked Read 8 Sent Done Done false
6142 2632 Acked Flush 0 Sent Done Done false
6143 2633 Acked Write 1 Sent Done Done false
6144 2634 Acked Write 4 Sent Done Done false
6145 2635 Acked Write 7 Sent Done Done false
6146 2636 Acked Write 6 Sent Done Done false
6147 2637 Acked Read 7 Sent Done Done false
6148 2638 Acked Write 10 Sent Done Done false
6149 2639 Acked Read 2 Sent Done Done false
6150 2640 Acked Read 8 Sent Done Done false
6151 2641 Acked Write 4 Sent Done Done false
6152 2642 Acked Read 10 Sent Done Done false
6153 2643 Acked Write 8 Sent Done Done false
6154 2644 Acked Read 5 Sent Done Done false
6155 2645 Acked Write 3 Sent Done Done false
6156 2646 Acked Write 2 Sent Done Done false
6157 2647 Acked Read 10 Sent Done Done false
6158 2648 Acked Write 9 Sent Done Done false
6159 2649 Acked Write 2 Sent Done Done false
6160 2650 Acked Write 7 Sent Done Done false
6161 2651 Acked Write 5 Sent Done Done false
6162 2652 Acked Read 7 Sent Done Done false
6163 2653 Acked Read 3 Sent Done Done false
6164 2654 Acked Read 5 Sent Done Done false
6165 2655 Acked Read 10 Sent Done Done false
6166 2656 Acked Write 10 Sent Done Done false
6167 2657 Acked Write 5 Sent Done Done false
6168 2658 Acked Read 4 Sent Done Done false
6169 2659 Acked Write 5 Sent Done Done false
6170 2660 Acked Read 6 Sent Done Done false
6171 2661 Acked Write 10 Sent Done Done false
6172 2662 Acked Write 2 Sent Done Done false
6173 2663 Acked Write 1 Sent Done Done false
6174 2664 Acked Read 7 Sent Done Done false
6175 2665 Acked Write 8 Sent Done Done false
6176 2666 Acked Read 10 Sent Done Done false
6177 2667 Acked Read 10 Sent Done Done false
6178 2668 Acked Write 10 Sent Done Done false
6179 2669 Acked Write 9 Sent Done Done false
6180 2670 Acked Write 5 Sent Done Done false
6181 2671 Acked Read 3 Sent Done Done false
6182 2672 Acked Write 2 Sent Done Done false
6183 2673 Acked Read 7 Sent Done Done false
6184 2674 Acked Write 6 Sent Done Done false
6185 2675 Acked Write 7 Sent Done Done false
6186 2676 Acked Read 5 Sent Done Done false
6187 2677 Acked Write 5 Sent Done Done false
6188 2678 Acked Write 2 Sent Done Done false
6189 2679 Acked Write 1 Sent Done Done false
6190 2680 Acked Write 6 Sent Done Done false
6191 2681 Acked Write 5 Sent Done Done false
6192 2682 Acked Read 4 Sent Done Done false
6193 2683 Acked Read 3 Sent Done Done false
6194 2684 Acked Read 4 Sent Done Done false
6195 2685 Acked Write 3 Sent Done Done false
6196 2686 Acked Flush 0 Sent Done Done false
6197 2687 Acked Read 5 Sent Done Done false
6198 2688 Acked Read 8 Sent Done Done false
6199 2689 Acked Read 3 Sent Done Done false
6200 2690 Acked Read 8 Sent Done Done false
6201 2691 Acked Read 4 Sent Done Done false
6202 2692 Acked Read 1 Sent Done Done false
6203 2693 Acked Write 7 Sent Done Done false
6204 2694 Acked Write 3 Sent Done Done false
6205 2695 Acked Write 1 Sent Done Done false
6206 2696 Acked Write 10 Sent Done Done false
6207 2697 Acked Read 7 Sent Done Done false
6208 2698 Acked Read 4 Sent Done Done false
6209 2699 Acked Write 9 Sent Done Done false
6210 2700 Acked Read 5 Sent Done Done false
6211 2701 Acked Read 5 Sent Done Done false
6212 2702 Acked Write 1 Sent Done Done false
6213 2703 Acked Read 7 Sent Done Done false
6214 2704 Acked Flush 0 Sent Done Done false
6215 2705 Acked Write 7 Sent Done Done false
6216 2706 Acked Flush 0 Sent Done Done false
6217 2707 Acked Write 6 Sent Done Done false
6218 2708 Acked Read 7 Sent Done Done false
6219 2709 Acked Read 10 Sent Done Done false
6220 2710 Acked Write 10 Sent Done Done false
6221 2711 Acked Read 9 Sent Done Done false
6222 2712 Acked Read 1 Sent Done Done false
6223 2713 Acked Write 7 Sent Done Done false
6224 2714 Acked Read 7 Sent Done Done false
6225 2715 Acked Read 5 Sent Done Done false
6226 2716 Acked Read 9 Sent Done Done false
6227 2717 Acked Read 9 Sent Done Done false
6228 2718 Acked Read 8 Sent Done Done false
6229 2719 Acked Write 10 Sent Done Done false
6230 2720 Acked Write 1 Sent Done Done false
6231 2721 Acked Write 9 Sent Done Done false
6232 2722 Acked Read 4 Sent Done Done false
6233 2723 Acked Write 6 Sent Done Done false
6234 2724 Acked Read 4 Sent Done Done false
6235 2725 Acked Read 5 Sent Done Done false
6236 2726 Acked Write 10 Sent Done Done false
6237 2727 Acked Write 4 Sent Done Done false
6238 2728 Acked Write 9 Sent Done Done false
6239 2729 Acked Write 2 Sent Done Done false
6240 2730 Acked Read 8 Sent Done Done false
6241 2731 Acked Flush 0 Sent Done Done false
6242 2732 Acked Write 3 Sent Done Done false
6243 2733 Acked Read 7 Sent Done Done false
6244 2734 Acked Read 10 Sent Done Done false
6245 2735 Acked Read 7 Sent Done Done false
6246 2736 Acked Write 8 Sent Done Done false
6247 2737 Acked Read 8 Sent Done Done false
6248 2738 Acked Flush 0 Sent Done Done false
6249 2739 Acked Write 4 Sent Done Done false
6250 2740 Acked Read 6 Sent Done Done false
6251 2741 Acked Read 9 Sent Done Done false
6252 2742 Acked Read 5 Sent Done Done false
6253 2743 Acked Read 6 Sent Done Done false
6254 2744 Acked Read 1 Sent Done Done false
6255 2745 Acked Write 3 Sent Done Done false
6256 2746 Acked Write 2 Sent Done Done false
6257 2747 Acked Read 4 Sent Done Done false
6258 2748 Acked Read 9 Sent Done Done false
6259 2749 Acked Write 6 Sent Done Done false
6260 2750 Acked Read 8 Sent Done Done false
6261 2751 Acked Read 9 Sent Done Done false
6262 2752 Acked Read 5 Sent Done Done false
6263 2753 Acked Read 2 Sent Done Done false
6264 2754 Acked Write 4 Sent Done Done false
6265 2755 Acked Flush 0 Sent Done Done false
6266 2756 Acked Flush 0 Sent Done Done false
6267 2757 Acked Write 6 Sent Done Done false
6268 2758 Acked Read 2 Sent Done Done false
6269 2759 Acked Write 2 Sent Done Done false
6270 2760 Acked Read 9 Sent Done Done false
6271 2761 Acked Read 5 Sent Done Done false
6272 2762 Acked Write 6 Sent Done Done false
6273 2763 Acked Write 7 Sent Done Done false
6274 2764 Acked Flush 0 Sent Done Done false
6275 2765 Acked Read 9 Sent Done Done false
6276 2766 Acked Read 7 Sent Done Done false
6277 2767 Acked Read 6 Sent Done Done false
6278 2768 Acked Write 3 Sent Done Done false
6279 2769 Acked Read 9 Sent Done Done false
6280 2770 Acked Read 7 Sent Done Done false
6281 2771 Acked Read 3 Sent Done Done false
6282 2772 Acked Read 10 Sent Done Done false
6283 2773 Acked Read 4 Sent Done Done false
6284 2774 Acked Write 7 Sent Done Done false
6285 2775 Acked Read 6 Sent Done Done false
6286 2776 Acked Write 10 Sent Done Done false
6287 2777 Acked Read 9 Sent Done Done false
6288 2778 Acked Read 8 Sent Done Done false
6289 2779 Acked Read 3 Sent Done Done false
6290 2780 Acked Write 10 Sent Done Done false
6291 2781 Acked Read 9 Sent Done Done false
6292 2782 Acked Write 6 Sent Done Done false
6293 2783 Acked Write 6 Sent Done Done false
6294 2784 Acked Flush 0 Sent Done Done false
6295 2785 Acked Read 1 Sent Done Done false
6296 2786 Acked Read 2 Sent Done Done false
6297 2787 Acked Read 3 Sent Done Done false
6298 2788 Acked Read 2 Sent Done Done false
6299 2789 Acked Write 1 Sent Done Done false
6300 2790 Acked Write 7 Sent Done Done false
6301 2791 Acked Read 10 Sent Done Done false
6302 2792 Acked Write 8 Sent Done Done false
6303 2793 Acked Write 9 Sent Done Done false
6304 2794 Acked Write 8 Sent Done Done false
6305 2795 Acked Read 4 Sent Done Done false
6306 2796 Acked Read 2 Sent Done Done false
6307 2797 Acked Read 1 Sent Done Done false
6308 2798 Acked Read 6 Sent Done Done false
6309 2799 Acked Write 2 Sent Done Done false
6310 2800 Acked Write 8 Sent Done Done false
6311 2801 Acked Read 9 Sent Done Done false
6312 2802 Acked Read 4 Sent Done Done false
6313 2803 Acked Read 4 Sent Done Done false
6314 2804 Acked Write 9 Sent Done Done false
6315 2805 Acked Write 7 Sent Done Done false
6316 2806 Acked Write 1 Sent Done Done false
6317 2807 Acked Read 10 Sent Done Done false
6318 2808 Acked Read 5 Sent Done Done false
6319 2809 Acked Write 6 Sent Done Done false
6320 2810 Acked Write 6 Sent Done Done false
6321 STATES DS:0 DS:1 DS:2 TOTAL
6322 Sent 301 0 0 301
6323 Done 0 301 301 602
6324 Skipped 0 0 0 0
6325 Error 0 0 0 0
6326 Last Flush: Some(JobId(2509)) Some(JobId(2784)) Some(JobId(2784))
6327 Downstairs last five completed: 2509 2508 2507 2506 2505
6328 Upstairs last five completed: 2810 2809 2808 2807 2806
6329 CLIENT: Up:0 ds:301 act:2
6330 [8810] got start action
6331 [8810] I am going to restart
6332 Starting downstairs at port 8810
6333 Make output file at "/tmp/dsc/downstairs-8810.txt"
6334 [8810][0] reports Starting
6335 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1211)
6336 [8810][0] reports Running
6337 ----------------------------------------------------------------
6338 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
6339 States: Offline Active Active
6340 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6341 2510 Acked Write 4 Sent Done Done false
6342 2511 Acked Write 6 Sent Done Done false
6343 2512 Acked Write 10 Sent Done Done false
6344 2513 Acked Read 9 Sent Done Done false
6345 2514 Acked Read 2 Sent Done Done false
6346 2515 Acked Read 1 Sent Done Done false
6347 2516 Acked Write 3 Sent Done Done false
6348 2517 Acked Read 5 Sent Done Done false
6349 2518 Acked Read 1 Sent Done Done false
6350 2519 Acked Write 5 Sent Done Done false
6351 2520 Acked Read 5 Sent Done Done false
6352 2521 Acked Read 1 Sent Done Done false
6353 2522 Acked Flush 0 Sent Done Done false
6354 2523 Acked Read 6 Sent Done Done false
6355 2524 Acked Write 4 Sent Done Done false
6356 2525 Acked Read 7 Sent Done Done false
6357 2526 Acked Write 5 Sent Done Done false
6358 2527 Acked Write 3 Sent Done Done false
6359 2528 Acked Flush 0 Sent Done Done false
6360 2529 Acked Flush 0 Sent Done Done false
6361 2530 Acked Read 3 Sent Done Done false
6362 2531 Acked Write 7 Sent Done Done false
6363 2532 Acked Read 7 Sent Done Done false
6364 2533 Acked Read 10 Sent Done Done false
6365 2534 Acked Write 3 Sent Done Done false
6366 2535 Acked Write 4 Sent Done Done false
6367 2536 Acked Read 2 Sent Done Done false
6368 2537 Acked Write 6 Sent Done Done false
6369 2538 Acked Read 4 Sent Done Done false
6370 2539 Acked Write 10 Sent Done Done false
6371 2540 Acked Write 1 Sent Done Done false
6372 2541 Acked Flush 0 Sent Done Done false
6373 2542 Acked Read 9 Sent Done Done false
6374 2543 Acked Read 9 Sent Done Done false
6375 2544 Acked Write 10 Sent Done Done false
6376 2545 Acked Write 6 Sent Done Done false
6377 2546 Acked Read 4 Sent Done Done false
6378 2547 Acked Read 6 Sent Done Done false
6379 2548 Acked Write 8 Sent Done Done false
6380 2549 Acked Flush 0 Sent Done Done false
6381 2550 Acked Read 7 Sent Done Done false
6382 2551 Acked Write 5 Sent Done Done false
6383 2552 Acked Read 2 Sent Done Done false
6384 2553 Acked Read 4 Sent Done Done false
6385 2554 Acked Write 7 Sent Done Done false
6386 2555 Acked Write 1 Sent Done Done false
6387 2556 Acked Write 10 Sent Done Done false
6388 2557 Acked Write 9 Sent Done Done false
6389 2558 Acked Read 1 Sent Done Done false
6390 2559 Acked Read 9 Sent Done Done false
6391 2560 Acked Read 3 Sent Done Done false
6392 2561 Acked Read 4 Sent Done Done false
6393 2562 Acked Read 2 Sent Done Done false
6394 2563 Acked Write 1 Sent Done Done false
6395 2564 Acked Read 6 Sent Done Done false
6396 2565 Acked Flush 0 Sent Done Done false
6397 2566 Acked Read 10 Sent Done Done false
6398 2567 Acked Write 2 Sent Done Done false
6399 2568 Acked Read 10 Sent Done Done false
6400 2569 Acked Flush 0 Sent Done Done false
6401 2570 Acked Read 1 Sent Done Done false
6402 2571 Acked Read 6 Sent Done Done false
6403 2572 Acked Read 4 Sent Done Done false
6404 2573 Acked Read 2 Sent Done Done false
6405 2574 Acked Read 7 Sent Done Done false
6406 2575 Acked Write 9 Sent Done Done false
6407 2576 Acked Write 1 Sent Done Done false
6408 2577 Acked Read 4 Sent Done Done false
6409 2578 Acked Write 6 Sent Done Done false
6410 2579 Acked Write 5 Sent Done Done false
6411 2580 Acked Read 4 Sent Done Done false
6412 2581 Acked Read 3 Sent Done Done false
6413 2582 Acked Read 6 Sent Done Done false
6414 2583 Acked Read 1 Sent Done Done false
6415 2584 Acked Write 2 Sent Done Done false
6416 2585 Acked Write 7 Sent Done Done false
6417 2586 Acked Read 4 Sent Done Done false
6418 2587 Acked Read 8 Sent Done Done false
6419 2588 Acked Write 2 Sent Done Done false
6420 2589 Acked Write 3 Sent Done Done false
6421 2590 Acked Read 7 Sent Done Done false
6422 2591 Acked Write 1 Sent Done Done false
6423 2592 Acked Write 10 Sent Done Done false
6424 2593 Acked Write 8 Sent Done Done false
6425 2594 Acked Write 8 Sent Done Done false
6426 2595 Acked Write 5 Sent Done Done false
6427 2596 Acked Read 6 Sent Done Done false
6428 2597 Acked Read 3 Sent Done Done false
6429 2598 Acked Flush 0 Sent Done Done false
6430 2599 Acked Read 1 Sent Done Done false
6431 2600 Acked Write 4 Sent Done Done false
6432 2601 Acked Flush 0 Sent Done Done false
6433 2602 Acked Read 2 Sent Done Done false
6434 2603 Acked Write 10 Sent Done Done false
6435 2604 Acked Write 2 Sent Done Done false
6436 2605 Acked Write 9 Sent Done Done false
6437 2606 Acked Write 3 Sent Done Done false
6438 2607 Acked Write 4 Sent Done Done false
6439 2608 Acked Write 4 Sent Done Done false
6440 2609 Acked Write 7 Sent Done Done false
6441 2610 Acked Write 6 Sent Done Done false
6442 2611 Acked Flush 0 Sent Done Done false
6443 2612 Acked Write 1 Sent Done Done false
6444 2613 Acked Flush 0 Sent Done Done false
6445 2614 Acked Read 9 Sent Done Done false
6446 2615 Acked Write 7 Sent Done Done false
6447 2616 Acked Read 3 Sent Done Done false
6448 2617 Acked Write 3 Sent Done Done false
6449 2618 Acked Read 1 Sent Done Done false
6450 2619 Acked Write 7 Sent Done Done false
6451 2620 Acked Write 10 Sent Done Done false
6452 2621 Acked Read 10 Sent Done Done false
6453 2622 Acked Write 3 Sent Done Done false
6454 2623 Acked Write 7 Sent Done Done false
6455 2624 Acked Flush 0 Sent Done Done false
6456 2625 Acked Flush 0 Sent Done Done false
6457 2626 Acked Read 6 Sent Done Done false
6458 2627 Acked Read 10 Sent Done Done false
6459 2628 Acked Read 10 Sent Done Done false
6460 2629 Acked Read 4 Sent Done Done false
6461 2630 Acked Read 9 Sent Done Done false
6462 2631 Acked Read 8 Sent Done Done false
6463 2632 Acked Flush 0 Sent Done Done false
6464 2633 Acked Write 1 Sent Done Done false
6465 2634 Acked Write 4 Sent Done Done false
6466 2635 Acked Write 7 Sent Done Done false
6467 2636 Acked Write 6 Sent Done Done false
6468 2637 Acked Read 7 Sent Done Done false
6469 2638 Acked Write 10 Sent Done Done false
6470 2639 Acked Read 2 Sent Done Done false
6471 2640 Acked Read 8 Sent Done Done false
6472 2641 Acked Write 4 Sent Done Done false
6473 2642 Acked Read 10 Sent Done Done false
6474 2643 Acked Write 8 Sent Done Done false
6475 2644 Acked Read 5 Sent Done Done false
6476 2645 Acked Write 3 Sent Done Done false
6477 2646 Acked Write 2 Sent Done Done false
6478 2647 Acked Read 10 Sent Done Done false
6479 2648 Acked Write 9 Sent Done Done false
6480 2649 Acked Write 2 Sent Done Done false
6481 2650 Acked Write 7 Sent Done Done false
6482 2651 Acked Write 5 Sent Done Done false
6483 2652 Acked Read 7 Sent Done Done false
6484 2653 Acked Read 3 Sent Done Done false
6485 2654 Acked Read 5 Sent Done Done false
6486 2655 Acked Read 10 Sent Done Done false
6487 2656 Acked Write 10 Sent Done Done false
6488 2657 Acked Write 5 Sent Done Done false
6489 2658 Acked Read 4 Sent Done Done false
6490 2659 Acked Write 5 Sent Done Done false
6491 2660 Acked Read 6 Sent Done Done false
6492 2661 Acked Write 10 Sent Done Done false
6493 2662 Acked Write 2 Sent Done Done false
6494 2663 Acked Write 1 Sent Done Done false
6495 2664 Acked Read 7 Sent Done Done false
6496 2665 Acked Write 8 Sent Done Done false
6497 2666 Acked Read 10 Sent Done Done false
6498 2667 Acked Read 10 Sent Done Done false
6499 2668 Acked Write 10 Sent Done Done false
6500 2669 Acked Write 9 Sent Done Done false
6501 2670 Acked Write 5 Sent Done Done false
6502 2671 Acked Read 3 Sent Done Done false
6503 2672 Acked Write 2 Sent Done Done false
6504 2673 Acked Read 7 Sent Done Done false
6505 2674 Acked Write 6 Sent Done Done false
6506 2675 Acked Write 7 Sent Done Done false
6507 2676 Acked Read 5 Sent Done Done false
6508 2677 Acked Write 5 Sent Done Done false
6509 2678 Acked Write 2 Sent Done Done false
6510 2679 Acked Write 1 Sent Done Done false
6511 2680 Acked Write 6 Sent Done Done false
6512 2681 Acked Write 5 Sent Done Done false
6513 2682 Acked Read 4 Sent Done Done false
6514 2683 Acked Read 3 Sent Done Done false
6515 2684 Acked Read 4 Sent Done Done false
6516 2685 Acked Write 3 Sent Done Done false
6517 2686 Acked Flush 0 Sent Done Done false
6518 2687 Acked Read 5 Sent Done Done false
6519 2688 Acked Read 8 Sent Done Done false
6520 2689 Acked Read 3 Sent Done Done false
6521 2690 Acked Read 8 Sent Done Done false
6522 2691 Acked Read 4 Sent Done Done false
6523 2692 Acked Read 1 Sent Done Done false
6524 2693 Acked Write 7 Sent Done Done false
6525 2694 Acked Write 3 Sent Done Done false
6526 2695 Acked Write 1 Sent Done Done false
6527 2696 Acked Write 10 Sent Done Done false
6528 2697 Acked Read 7 Sent Done Done false
6529 2698 Acked Read 4 Sent Done Done false
6530 2699 Acked Write 9 Sent Done Done false
6531 2700 Acked Read 5 Sent Done Done false
6532 2701 Acked Read 5 Sent Done Done false
6533 2702 Acked Write 1 Sent Done Done false
6534 2703 Acked Read 7 Sent Done Done false
6535 2704 Acked Flush 0 Sent Done Done false
6536 2705 Acked Write 7 Sent Done Done false
6537 2706 Acked Flush 0 Sent Done Done false
6538 2707 Acked Write 6 Sent Done Done false
6539 2708 Acked Read 7 Sent Done Done false
6540 2709 Acked Read 10 Sent Done Done false
6541 2710 Acked Write 10 Sent Done Done false
6542 2711 Acked Read 9 Sent Done Done false
6543 2712 Acked Read 1 Sent Done Done false
6544 2713 Acked Write 7 Sent Done Done false
6545 2714 Acked Read 7 Sent Done Done false
6546 2715 Acked Read 5 Sent Done Done false
6547 2716 Acked Read 9 Sent Done Done false
6548 2717 Acked Read 9 Sent Done Done false
6549 2718 Acked Read 8 Sent Done Done false
6550 2719 Acked Write 10 Sent Done Done false
6551 2720 Acked Write 1 Sent Done Done false
6552 2721 Acked Write 9 Sent Done Done false
6553 2722 Acked Read 4 Sent Done Done false
6554 2723 Acked Write 6 Sent Done Done false
6555 2724 Acked Read 4 Sent Done Done false
6556 2725 Acked Read 5 Sent Done Done false
6557 2726 Acked Write 10 Sent Done Done false
6558 2727 Acked Write 4 Sent Done Done false
6559 2728 Acked Write 9 Sent Done Done false
6560 2729 Acked Write 2 Sent Done Done false
6561 2730 Acked Read 8 Sent Done Done false
6562 2731 Acked Flush 0 Sent Done Done false
6563 2732 Acked Write 3 Sent Done Done false
6564 2733 Acked Read 7 Sent Done Done false
6565 2734 Acked Read 10 Sent Done Done false
6566 2735 Acked Read 7 Sent Done Done false
6567 2736 Acked Write 8 Sent Done Done false
6568 2737 Acked Read 8 Sent Done Done false
6569 2738 Acked Flush 0 Sent Done Done false
6570 2739 Acked Write 4 Sent Done Done false
6571 2740 Acked Read 6 Sent Done Done false
6572 2741 Acked Read 9 Sent Done Done false
6573 2742 Acked Read 5 Sent Done Done false
6574 2743 Acked Read 6 Sent Done Done false
6575 2744 Acked Read 1 Sent Done Done false
6576 2745 Acked Write 3 Sent Done Done false
6577 2746 Acked Write 2 Sent Done Done false
6578 2747 Acked Read 4 Sent Done Done false
6579 2748 Acked Read 9 Sent Done Done false
6580 2749 Acked Write 6 Sent Done Done false
6581 2750 Acked Read 8 Sent Done Done false
6582 2751 Acked Read 9 Sent Done Done false
6583 2752 Acked Read 5 Sent Done Done false
6584 2753 Acked Read 2 Sent Done Done false
6585 2754 Acked Write 4 Sent Done Done false
6586 2755 Acked Flush 0 Sent Done Done false
6587 2756 Acked Flush 0 Sent Done Done false
6588 2757 Acked Write 6 Sent Done Done false
6589 2758 Acked Read 2 Sent Done Done false
6590 2759 Acked Write 2 Sent Done Done false
6591 2760 Acked Read 9 Sent Done Done false
6592 2761 Acked Read 5 Sent Done Done false
6593 2762 Acked Write 6 Sent Done Done false
6594 2763 Acked Write 7 Sent Done Done false
6595 2764 Acked Flush 0 Sent Done Done false
6596 2765 Acked Read 9 Sent Done Done false
6597 2766 Acked Read 7 Sent Done Done false
6598 2767 Acked Read 6 Sent Done Done false
6599 2768 Acked Write 3 Sent Done Done false
6600 2769 Acked Read 9 Sent Done Done false
6601 2770 Acked Read 7 Sent Done Done false
6602 2771 Acked Read 3 Sent Done Done false
6603 2772 Acked Read 10 Sent Done Done false
6604 2773 Acked Read 4 Sent Done Done false
6605 2774 Acked Write 7 Sent Done Done false
6606 2775 Acked Read 6 Sent Done Done false
6607 2776 Acked Write 10 Sent Done Done false
6608 2777 Acked Read 9 Sent Done Done false
6609 2778 Acked Read 8 Sent Done Done false
6610 2779 Acked Read 3 Sent Done Done false
6611 2780 Acked Write 10 Sent Done Done false
6612 2781 Acked Read 9 Sent Done Done false
6613 2782 Acked Write 6 Sent Done Done false
6614 2783 Acked Write 6 Sent Done Done false
6615 2784 Acked Flush 0 Sent Done Done false
6616 2785 Acked Read 1 Sent Done Done false
6617 2786 Acked Read 2 Sent Done Done false
6618 2787 Acked Read 3 Sent Done Done false
6619 2788 Acked Read 2 Sent Done Done false
6620 2789 Acked Write 1 Sent Done Done false
6621 2790 Acked Write 7 Sent Done Done false
6622 2791 Acked Read 10 Sent Done Done false
6623 2792 Acked Write 8 Sent Done Done false
6624 2793 Acked Write 9 Sent Done Done false
6625 2794 Acked Write 8 Sent Done Done false
6626 2795 Acked Read 4 Sent Done Done false
6627 2796 Acked Read 2 Sent Done Done false
6628 2797 Acked Read 1 Sent Done Done false
6629 2798 Acked Read 6 Sent Done Done false
6630 2799 Acked Write 2 Sent Done Done false
6631 2800 Acked Write 8 Sent Done Done false
6632 2801 Acked Read 9 Sent Done Done false
6633 2802 Acked Read 4 Sent Done Done false
6634 2803 Acked Read 4 Sent Done Done false
6635 2804 Acked Write 9 Sent Done Done false
6636 2805 Acked Write 7 Sent Done Done false
6637 2806 Acked Write 1 Sent Done Done false
6638 2807 Acked Read 10 Sent Done Done false
6639 2808 Acked Read 5 Sent Done Done false
6640 2809 Acked Write 6 Sent Done Done false
6641 2810 Acked Write 6 Sent Done Done false
6642 2811 Acked Flush 0 Sent Done Done false
6643 STATES DS:0 DS:1 DS:2 TOTAL
6644 Sent 302 0 0 302
6645 Done 0 302 302 604
6646 Skipped 0 0 0 0
6647 Error 0 0 0 0
6648 Last Flush: Some(JobId(2509)) Some(JobId(2811)) Some(JobId(2811))
6649 Downstairs last five completed: 2509 2508 2507 2506 2505
6650 Upstairs last five completed: 2811 2810 2809 2808 2807
6651 CLIENT: Up:0 ds:302 act:2
66522025-01-23T17:39:06.869ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
66532025-01-23T17:39:06.870ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65466) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
66542025-01-23T17:39:06.871ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
66552025-01-23T17:39:06.871ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
66562025-01-23T17:39:06.871ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
66572025-01-23T17:39:06.871ZINFOcrucible: send last flush ID to this DS: Some(JobId(2509)) = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
66582025-01-23T17:39:06.871ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
66592025-01-23T17:39:06.871ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(2509)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
66602025-01-23T17:39:06.871ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(2509)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
6661 ----------------------------------------------------------------
6662 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6663 Downstairs last five completed: 2811 2810 2809 2808 2807
6664 Upstairs last five completed: 2811 2810 2809 2808 2807
6665 CLIENT: Up:0 ds:0 act:3
6666 Replay: All jobs finished
6667 Main task has work to do, go find it
6668 got dsc Stop(2)
6669 stop 2
6670 Jan 23 17:39:09.118 INFO[8830] Got stop action so:false kr:false
6671 request completed, latency_us: 373, response_code: 204, uri: /stop/cid/2, method: POST, req_id: f1cef934-a00a-43a7-bb26-a7cabbc971f6, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
66722025-01-23T17:39:09.119ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
66732025-01-23T17:39:09.119ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
66742025-01-23T17:39:09.119ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
6675 Jan 23 17:39:09.119 INFO request completed, latency_us: 146, response_code{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T17:39:09.119549904Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
6676 : 200, uri: /state/cid/2, method: GET, req_id: ce089213-db27-4075-9db1-06f9b37c1114, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
6677 [8830] Exited with: ExitStatus(unix_wait_status(9))
6678 [8830][2] reports Exit
6679 Jan 23 17:39:13.121 INFO request completed, latency_us: 193, response_code: 200, uri: /state/cid/2, method: GET, req_id: a0ff4f3c-5cd9-4233-b056-bcff9543209d, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
6680 001/300 Write block 287 len 16384 data: 4 5 5 5
6681 002/300 Read block 2091 len 4096
6682 003/300 Read block 2151 len 40960
6683 004/300 Read block 1385 len 40960
6684 005/300 Read block 1315 len 16384
6685 006/300 Read block 1850 len 20480
6686 007/300 Write block 3103 len 28672 data: 2 2 2 3 3 3 3
6687 008/300 Read block 3492 len 40960
6688 009/300 Read block 328 len 36864
6689 010/300 Read block 87 len 28672
6690 011/300 Read block 4415 len 40960
6691 012/300 Write block 2367 len 20480 data: 2 2 2 2 3
6692 013/300 Read block 4541 len 20480
6693 014/300 Read block 2772 len 32768
6694 015/300 Write block 4134 len 32768 data: 2 2 2 2 2 2 3 3
6695 016/300 Write block 783 len 16384 data: 2 3 4 2
6696 017/300 Read block 3007 len 12288
6697 018/300 Write block 3475 len 8192 data: 3 3
6698 019/300 Read block 4873 len 12288
6699 020/300 Read block 1158 len 8192
6700 021/300 Flush
6701 022/300 Write block 640 len 28672 data: 3 2 2 2 2 2 2
6702 023/300 Read block 4983 len 20480
6703 024/300 Write block 3884 len 8192 data: 2 2
6704 025/300 Read block 514 len 20480
6705 026/300 Write block 2726 len 32768 data: 2 2 2 2 2 2 2 2
6706 027/300 Read block 466 len 24576
6707 028/300 Read block 27 len 12288
6708 029/300 Flush
6709 030/300 Read block 4964 len 16384
6710 031/300 Read block 423 len 16384
6711 032/300 Read block 2854 len 32768
6712 033/300 Write block 3642 len 24576 data: 2 2 2 2 2 3
6713 034/300 Write block 108 len 20480 data: 4 4 6 6 6
6714 035/300 Read block 2739 len 4096
6715 036/300 Read block 3044 len 36864
6716 037/300 Read block 42 len 12288
6717 038/300 Write block 4799 len 24576 data: 3 2 2 2 2 2
6718 039/300 Read block 335 len 12288
6719 040/300 Read block 3509 len 8192
6720 041/300 Read block 1199 len 28672
6721 042/300 Read block 4863 len 8192
6722 043/300 Read block 4468 len 20480
6723 044/300 Read block 1850 len 24576
6724 045/300 Read block 386 len 8192
6725 046/300 Flush
6726 047/300 Read block 723 len 16384
6727 048/300 Flush
6728 049/300 Read block 2441 len 12288
6729 050/300 Flush
6730 051/300 Read block 4759 len 4096
6731 052/300 Read block 1405 len 16384
6732 053/300 Read block 4892 len 8192
6733 054/300 Read block 2786 len 12288
6734 055/300 Read block 1843 len 28672
6735 056/300 Read block 4939 len 8192
6736 057/300 Read block 1702 len 20480
6737 058/300 Read block 1272 len 28672
6738 059/300 Write block 1691 len 12288 data: 2 2 2
6739 060/300 Read block 1808 len 8192
6740 061/300 Read block 4369 len 32768
6741 062/300 Read block 4683 len 36864
6742 063/300 Write block 556 len 4096 data: 4
6743 064/300 Read block 4157 len 24576
6744 065/300 Read block 1806 len 24576
6745 066/300 Flush
6746 067/300 Read block 4492 len 12288
6747 068/300 Write block 2216 len 16384 data: 2 2 2 2
6748 069/300 Write block 2836 len 40960 data: 2 4 3 3 3 3 3 3 3 4
6749 070/300 Read block 1529 len 4096
6750 071/300 Read block 4692 len 24576
6751 072/300 Read block 3074 len 32768
6752 073/300 Write block 1652 len 8192 data: 4 4
6753 074/300 Flush
6754 075/300 Flush
6755 076/300 Write block 3744 len 12288 data: 2 2 2
6756 077/300 Write block 3732 len 4096 data: 3
6757 078/300 Write block 1728 len 12288 data: 3 3 3
6758 079/300 Read block 612 len 24576
6759 080/300 Read block 674 len 40960
6760 081/300 Read block 2155 len 28672
6761 082/300 Write block 886 len 32768 data: 2 2 2 3 2 2 2 2
6762 083/300 Read block 2042 len 16384
6763 084/300 Write block 1035 len 8192 data: 2 2
6764 085/300 Read block 690 len 20480
6765 086/300 Write block 25 len 28672 data: 2 2 3 3 3 3 3
6766 087/300 Write block 337 len 32768 data: 2 3 3 3 3 2 2 2
6767 088/300 Read block 2291 len 16384
6768 089/300 Read block 597 len 40960
6769 090/300 Read block 2478 len 36864
6770 091/300 Read block 618 len 8192
6771 092/300 Read block 4370 len 16384
6772 093/300 Write block 2251 len 20480 data: 2 3 3 4 4
6773 094/300 Flush
6774 095/300 Read block 4769 len 20480
6775 096/300 Flush
6776 097/300 Write block 208 len 40960 data: 2 2 3 3 3 3 2 2 2 3
6777 098/300 Flush
6778 099/300 Read block 2675 len 28672
6779 100/300 Write block 4993 len 28672 data: 3 3 3 3 3 3 2
6780 101/300 Write block 525 len 8192 data: 4 4
6781 102/300 Write block 4563 len 36864 data: 2 4 4 4 4 4 4 4 3
6782 103/300 Read block 3038 len 12288
6783 104/300 Write block 3986 len 12288 data: 3 3 3
6784 105/300 Read block 528 len 32768
6785 106/300 Read block 3235 len 8192
6786 107/300 Flush
6787 108/300 Write block 1723 len 4096 data: 2
6788 109/300 Flush
6789 110/300 Read block 1767 len 16384
6790 111/300 Read block 327 len 32768
6791 112/300 Read block 4314 len 32768
6792 113/300 Read block 1320 len 28672
6793 114/300 Read block 339 len 8192
6794 115/300 Read block 3110 len 12288
6795 116/300 Flush
6796 117/300 Write block 1815 len 32768 data: 3 3 3 2 2 2 3 3
6797 118/300 Write block 697 len 20480 data: 3 3 3 3 3
6798 119/300 Read block 4218 len 16384
6799 120/300 Read block 2977 len 8192
6800 121/300 Read block 1324 len 28672
6801 122/300 Write block 2748 len 32768 data: 4 4 4 4 3 2 2 2
6802 123/300 Read block 3396 len 40960
6803 124/300 Read block 2706 len 28672
6804 125/300 Write block 2571 len 24576 data: 3 2 2 3 3 3
6805 126/300 Write block 99 len 16384 data: 2 2 2 3
6806 127/300 Read block 4554 len 40960
6807 128/300 Read block 888 len 36864
6808 129/300 Read block 2979 len 28672
6809 130/300 Read block 4748 len 36864
6810 131/300 Read block 960 len 16384
6811 132/300 Write block 1852 len 4096 data: 2
6812 133/300 Flush
6813 134/300 Read block 349 len 36864
6814 135/300 Read block 3497 len 16384
6815 136/300 Write block 165 len 36864 data: 4 4 4 3 3 3 4 4 4
6816 137/300 Write block 4576 len 28672 data: 2 2 3 3 3 3 3
6817 138/300 Read block 134 len 36864
6818 139/300 Read block 2324 len 12288
6819 140/300 Read block 4937 len 12288
6820 141/300 Write block 4277 len 24576 data: 2 3 3 4 4 4
6821 142/300 Read block 2307 len 4096
6822 143/300 Read block 2331 len 12288
6823 144/300 Write block 3457 len 24576 data: 3 3 3 3 3 3
6824 145/300 Write block 3304 len 28672 data: 2 2 2 3 3 3 4
6825 146/300 Read block 1298 len 20480
6826 147/300 Write block 4537 len 12288 data: 3 2 2
6827 148/300 Write block 4026 len 32768 data: 2 2 2 2 2 2 4 4
6828 149/300 Read block 787 len 4096
6829 150/300 Write block 3363 len 12288 data: 3 3 3
6830 151/300 Write block 292 len 16384 data: 2 2 2 3
6831 152/300 Read block 2875 len 8192
6832 153/300 Read block 4896 len 12288
6833 154/300 Read block 1466 len 40960
6834 155/300 Write block 3574 len 12288 data: 2 2 3
6835 156/300 Write block 2865 len 24576 data: 3 4 5 4 4 2
6836 157/300 Flush
6837 158/300 Write block 2441 len 40960 data: 3 3 4 4 4 4 4 4 4 4
6838 159/300 Write block 3949 len 36864 data: 2 2 2 2 3 3 3 3 3
6839 160/300 Write block 1737 len 12288 data: 3 3 3
6840 161/300 Read block 2779 len 36864
6841 162/300 Read block 110 len 20480
6842 163/300 Read block 3659 len 24576
6843 164/300 Read block 706 len 36864
6844 165/300 Write block 1416 len 4096 data: 3
6845 166/300 Read block 1228 len 32768
6846 167/300 Write block 252 len 12288 data: 3 3 3
6847 168/300 Read block 1509 len 12288
6848 169/300 Write block 1825 len 28672 data: 5 4 4 4 4 4 3
6849 170/300 Read block 2600 len 36864
6850 171/300 Read block 1319 len 28672
6851 172/300 Read block 1742 len 36864
6852 173/300 Write block 2651 len 8192 data: 2 2
6853 174/300 Read block 268 len 40960
6854 175/300 Write block 2207 len 28672 data: 3 3 2 2 2 2 2
6855 176/300 Read block 2213 len 28672
6856 177/300 Write block 3493 len 24576 data: 3 3 3 3 3 2
6857 178/300 Write block 2479 len 4096 data: 4
6858 179/300 Read block 3084 len 40960
6859 180/300 Write block 2738 len 24576 data: 3 3 3 3 3 3
6860 181/300 Read block 717 len 12288
6861 182/300 Write block 2195 len 4096 data: 5
6862 183/300 Write block 170 len 24576 data: 4 5 5 5 4 4
6863 184/300 Read block 2519 len 24576
6864 185/300 Write block 3497 len 40960 data: 4 3 2 2 2 2 3 3 3 3
6865 186/300 Write block 413 len 12288 data: 3 2 2
6866 187/300 Read block 4327 len 36864
6867 188/300 Write block 2782 len 16384 data: 2 2 2 2
6868 189/300 Flush
6869 190/300 Write block 2216 len 16384 data: 3 3 3 3
6870 191/300 Read block 3348 len 40960
6871 192/300 Write block 3946 len 16384 data: 3 3 2 3
6872 193/300 Read block 4687 len 40960
6873 194/300 Read block 1522 len 20480
6874 195/300 Write block 2676 len 4096 data: 2
6875 196/300 Write block 4570 len 40960 data: 5 4 2 2 2 2 3 3 4 4
6876 197/300 Read block 403 len 28672
6877 198/300 Write block 1640 len 20480 data: 2 2 3 3 3
6878 199/300 Write block 2841 len 28672 data: 4 4 4 4 5 3 3
6879 200/300 Write block 2123 len 20480 data: 2 2 2 3 3
6880 201/300 Read block 1589 len 28672
6881 202/300 Read block 861 len 16384
6882 203/300 Flush
6883 204/300 Write block 1156 len 36864 data: 4 3 3 3 3 4 5 5 5
6884 205/300 Flush
6885 206/300 Write block 4349 len 16384 data: 3 2 2 2
6886 207/300 Write block 3933 len 8192 data: 3 3
6887 208/300 Read block 199 len 8192
6888 209/300 Write block 4020 len 8192 data: 2 2
6889 210/300 Read block 842 len 24576
6890 211/300 Read block 610 len 8192
6891 212/300 Read block 3583 len 4096
6892 213/300 Write block 3374 len 4096 data: 3
6893 214/300 Write block 480 len 32768 data: 2 2 2 3 3 3 3 3
6894 215/300 Write block 3750 len 32768 data: 2 2 2 2 2 2 2 2
6895 216/300 Read block 3723 len 4096
6896 217/300 Write block 4089 len 28672 data: 2 2 2 2 2 2 2
6897 218/300 Read block 2470 len 16384
6898 219/300 Flush
6899 220/300 Read block 3062 len 16384
6900 221/300 Write block 796 len 36864 data: 2 2 2 2 2 2 2 2 3
6901 222/300 Flush
6902 223/300 Read block 3404 len 12288
6903 224/300 Write block 3579 len 40960 data: 4 4 4 4 4 5 5 4 3 3
6904 225/300 Read block 3926 len 32768
6905 226/300 Write block 4600 len 32768 data: 3 3 3 3 3 3 3 3
6906 227/300 Read block 1303 len 16384
6907 228/300 Read block 4665 len 16384
6908 229/300 Read block 3735 len 36864
6909 230/300 Flush
6910 231/300 Flush
6911 232/300 Read block 1160 len 16384
6912 233/300 Read block 62 len 20480
6913 234/300 Flush
6914 235/300 Read block 4350 len 40960
6915 236/300 Write block 4391 len 32768 data: 2 2 2 3 3 2 2 3
6916 237/300 Read block 4375 len 32768
6917 238/300 Write block 328 len 24576 data: 3 3 3 4 4 4
6918 239/300 Write block 4335 len 40960 data: 4 4 3 2 2 2 3 3 3 3
6919 240/300 Write block 1258 len 12288 data: 6 6 6
6920 241/300 Write block 3967 len 32768 data: 2 3 3 3 3 2 2 2
6921 242/300 Write block 3654 len 24576 data: 3 4 3 3 2 2
6922 243/300 Read block 4026 len 32768
6923 244/300 Read block 3261 len 20480
6924 245/300 Write block 1756 len 20480 data: 4 3 3 3 3
6925 246/300 Read block 345 len 16384
6926 247/300 Write block 2904 len 16384 data: 2 2 2 2
6927 248/300 Write block 2105 len 32768 data: 3 3 3 3 3 3 2 4
6928 249/300 Write block 2964 len 8192 data: 3 3
6929 250/300 Write block 1083 len 40960 data: 3 3 3 2 2 2 2 2 2 2
6930 251/300 Write block 1331 len 12288 data: 4 4 3
6931 252/300 Read block 685 len 16384
6932 253/300 Read block 1788 len 12288
6933 254/300 Write block 403 len 20480 data: 2 2 2 3 3
6934 255/300 Write block 4637 len 24576 data: 2 2 2 2 2 2
6935 256/300 Write block 3298 len 28672 data: 3 3 3 3 2 2 3
6936 257/300 Write block 392 len 36864 data: 3 3 3 2 2 2 2 2 2
6937 258/300 Write block 3416 len 8192 data: 3 3
6938 259/300 Write block 1548 len 12288 data: 2 2 3
6939 260/300 Write block 4713 len 8192 data: 2 2
6940 261/300 Read block 4722 len 4096
6941 262/300 Write block 4140 len 24576 data: 4 4 3 3 3 3
6942 263/300 Flush
6943 264/300 Read block 4055 len 20480
6944 265/300 Read block 1508 len 32768
6945 266/300 Read block 3866 len 40960
6946 267/300 Write block 3607 len 32768 data: 3 3 4 4 5 5 5 4
6947 268/300 Read block 260 len 28672
6948 269/300 Read block 3345 len 32768
6949 270/300 Write block 2239 len 20480 data: 3 3 3 3 3
6950 271/300 Write block 569 len 12288 data: 2 3 4
6951 272/300 Write block 1049 len 20480 data: 3 3 3 3 2
6952 273/300 Read block 4188 len 20480
6953 274/300 Read block 2343 len 36864
6954 275/300 Write block 4284 len 12288 data: 5 4 3
6955 276/300 Read block 3657 len 32768
6956 277/300 Read block 1272 len 36864
6957 278/300 Write block 300 len 4096 data: 4
6958 279/300 Flush
6959 280/300 Read block 4790 len 36864
6960 281/300 Write block 820 len 28672 data: 3 3 4 4 5 5 5
6961 282/300 Flush
6962 283/300 Read block 120 len 12288
6963 284/300 Read block 2657 len 36864
6964 285/300 Write block 989 len 24576 data: 5 4 4 4 3 3
6965 286/300 Read block 1184 len 24576
6966 287/300 Write block 972 len 16384 data: 3 3 3 3
6967 288/300 Write block 1177 len 12288 data: 3 3 4
6968 289/300 Read block 4253 len 28672
6969 290/300 Flush
6970 291/300 Flush
6971 292/300 Flush
6972 293/300 Read block 627 len 8192
6973 294/300 Write block 3189 len 4096 data: 2
6974 295/300 Write block 4894 len 16384 data: 3 3 2 2
6975 296/300 Read block 4466 len 28672
6976 297/300 Write block 1849 len 12288 data: 2 2 2
6977 298/300 Read block 4232 len 32768
6978 299/300 Read block 4344 len 12288
6979 300/300 Read block 337 len 36864
6980 301/300 Read block 368 len 20480
6981 Main task has work to do, go find it
6982 got dsc Start(2)
6983 start 2
6984 Jan 23 17:39:13.365 INFO request completed, latency_us: 196, response_code: 204, uri: /start/cid/2, method: POST, req_id: 95e4fb86-de8e-40fc-84fe-bc0851350c39, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
6985 Replay: started 2, returned:Ok(())
6986 ----------------------------------------------------------------
6987 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6988 States: Active Active Offline
6989 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6990 2812 Acked Write 4 Done Done Sent false
6991 2813 Acked Read 1 Done Done Sent false
6992 2814 Acked Read 10 Done Done Sent false
6993 2815 Acked Read 10 Done Done Sent false
6994 2816 Acked Read 4 Done Done Sent false
6995 2817 Acked Read 5 Done Done Sent false
6996 2818 Acked Write 7 Done Done Sent false
6997 2819 Acked Read 10 Done Done Sent false
6998 2820 Acked Read 9 Done Done Sent false
6999 2821 Acked Read 7 Done Done Sent false
7000 2822 Acked Read 10 Done Done Sent false
7001 2823 Acked Write 5 Done Done Sent false
7002 2824 Acked Read 5 Done Done Sent false
7003 2825 Acked Read 8 Done Done Sent false
7004 2826 Acked Write 8 Done Done Sent false
7005 2827 Acked Write 4 Done Done Sent false
7006 2828 Acked Read 3 Done Done Sent false
7007 2829 Acked Write 2 Done Done Sent false
7008 2830 Acked Read 3 Done Done Sent false
7009 2831 Acked Read 2 Done Done Sent false
7010 2832 Acked Flush 0 Done Done Sent false
7011 2833 Acked Write 7 Done Done Sent false
7012 2834 Acked Read 5 Done Done Sent false
7013 2835 Acked Write 2 Done Done Sent false
7014 2836 Acked Read 5 Done Done Sent false
7015 2837 Acked Write 8 Done Done Sent false
7016 2838 Acked Read 6 Done Done Sent false
7017 2839 Acked Read 3 Done Done Sent false
7018 2840 Acked Flush 0 Done Done Sent false
7019 2841 Acked Read 4 Done Done Sent false
7020 2842 Acked Read 4 Done Done Sent false
7021 2843 Acked Read 8 Done Done Sent false
7022 2844 Acked Write 6 Done Done Sent false
7023 2845 Acked Write 5 Done Done Sent false
7024 2846 Acked Read 1 Done Done Sent false
7025 2847 Acked Read 9 Done Done Sent false
7026 2848 Acked Read 3 Done Done Sent false
7027 2849 Acked Write 6 Done Done Sent false
7028 2850 Acked Read 3 Done Done Sent false
7029 2851 Acked Read 2 Done Done Sent false
7030 2852 Acked Read 7 Done Done Sent false
7031 2853 Acked Read 2 Done Done Sent false
7032 2854 Acked Read 5 Done Done Sent false
7033 2855 Acked Read 6 Done Done Sent false
7034 2856 Acked Read 2 Done Done Sent false
7035 2857 Acked Flush 0 Done Done Sent false
7036 2858 Acked Read 4 Done Done Sent false
7037 2859 Acked Flush 0 Done Done Sent false
7038 2860 Acked Read 3 Done Done Sent false
7039 2861 Acked Flush 0 Done Done Sent false
7040 2862 Acked Read 1 Done Done Sent false
7041 2863 Acked Read 4 Done Done Sent false
7042 2864 Acked Read 2 Done Done Sent false
7043 2865 Acked Read 3 Done Done Sent false
7044 2866 Acked Read 7 Done Done Sent false
7045 2867 Acked Read 2 Done Done Sent false
7046 2868 Acked Read 5 Done Done Sent false
7047 2869 Acked Read 7 Done Done Sent false
7048 2870 Acked Write 3 Done Done Sent false
7049 2871 Acked Read 2 Done Done Sent false
7050 2872 Acked Read 8 Done Done Sent false
7051 2873 Acked Read 9 Done Done Sent false
7052 2874 Acked Write 1 Done Done Sent false
7053 2875 Acked Read 6 Done Done Sent false
7054 2876 Acked Read 6 Done Done Sent false
7055 2877 Acked Flush 0 Done Done Sent false
7056 2878 Acked Read 3 Done Done Sent false
7057 2879 Acked Write 4 Done Done Sent false
7058 2880 Acked Write 10 Done Done Sent false
7059 2881 Acked Read 1 Done Done Sent false
7060 2882 Acked Read 6 Done Done Sent false
7061 2883 Acked Read 8 Done Done Sent false
7062 2884 Acked Write 2 Done Done Sent false
7063 2885 Acked Flush 0 Done Done Sent false
7064 2886 Acked Flush 0 Done Done Sent false
7065 2887 Acked Write 3 Done Done Sent false
7066 2888 Acked Write 1 Done Done Sent false
7067 2889 Acked Write 3 Done Done Sent false
7068 2890 Acked Read 6 Done Done Sent false
7069 2891 Acked Read 10 Done Done Sent false
7070 2892 Acked Read 7 Done Done Sent false
7071 2893 Acked Write 8 Done Done Sent false
7072 2894 Acked Read 4 Done Done Sent false
7073 2895 Acked Write 2 Done Done Sent false
7074 2896 Acked Read 5 Done Done Sent false
7075 2897 Acked Write 7 Done Done Sent false
7076 2898 Acked Write 8 Done Done Sent false
7077 2899 Acked Read 4 Done Done Sent false
7078 2900 Acked Read 10 Done Done Sent false
7079 2901 Acked Read 9 Done Done Sent false
7080 2902 Acked Read 2 Done Done Sent false
7081 2903 Acked Read 4 Done Done Sent false
7082 2904 Acked Write 5 Done Done Sent false
7083 2905 Acked Flush 0 Done Done Sent false
7084 2906 Acked Read 5 Done Done Sent false
7085 2907 Acked Flush 0 Done Done Sent false
7086 2908 Acked Write 10 Done Done Sent false
7087 2909 Acked Flush 0 Done Done Sent false
7088 2910 Acked Read 7 Done Done Sent false
7089 2911 Acked Write 7 Done Done Sent false
7090 2912 Acked Write 2 Done Done Sent false
7091 2913 Acked Write 9 Done Done Sent false
7092 2914 Acked Read 3 Done Done Sent false
7093 2915 Acked Write 3 Done Done Sent false
7094 2916 Acked Read 8 Done Done Sent false
7095 2917 Acked Read 2 Done Done Sent false
7096 2918 Acked Flush 0 Done Done Sent false
7097 2919 Acked Write 1 Done Done Sent false
7098 2920 Acked Flush 0 Done Done Sent false
7099 2921 Acked Read 4 Done Done Sent false
7100 2922 Acked Read 8 Done Done Sent false
7101 2923 Acked Read 8 Done Done Sent false
7102 2924 Acked Read 7 Done Done Sent false
7103 2925 Acked Read 2 Done Done Sent false
7104 2926 Acked Read 3 Done Done Sent false
7105 2927 Acked Flush 0 Done Done Sent false
7106 2928 Acked Write 8 Done Done Sent false
7107 2929 Acked Write 5 Done Done Sent false
7108 2930 Acked Read 4 Done Done Sent false
7109 2931 Acked Read 2 Done Done Sent false
7110 2932 Acked Read 7 Done Done Sent false
7111 2933 Acked Write 8 Done Done Sent false
7112 2934 Acked Read 10 Done Done Sent false
7113 2935 Acked Read 7 Done Done Sent false
7114 2936 Acked Write 6 Done Done Sent false
7115 2937 Acked Write 4 Done Done Sent false
7116 2938 Acked Read 10 Done Done Sent false
7117 2939 Acked Read 9 Done Done Sent false
7118 2940 Acked Read 7 Done Done Sent false
7119 2941 Acked Read 9 Done Done Sent false
7120 2942 Acked Read 4 Done Done Sent false
7121 2943 Acked Write 1 Done Done Sent false
7122 2944 Acked Flush 0 Done Done Sent false
7123 2945 Acked Read 9 Done Done Sent false
7124 2946 Acked Read 4 Done Done Sent false
7125 2947 Acked Write 9 Done Done Sent false
7126 2948 Acked Write 7 Done Done Sent false
7127 2949 Acked Read 9 Done Done Sent false
7128 2950 Acked Read 3 Done Done Sent false
7129 2951 Acked Read 3 Done Done Sent false
7130 2952 Acked Write 6 Done Done Sent false
7131 2953 Acked Read 1 Done Done Sent false
7132 2954 Acked Read 3 Done Done Sent false
7133 2955 Acked Write 6 Done Done Sent false
7134 2956 Acked Write 7 Done Done Sent false
7135 2957 Acked Read 5 Done Done Sent false
7136 2958 Acked Write 3 Done Done Sent false
7137 2959 Acked Write 8 Done Done Sent false
7138 2960 Acked Read 1 Done Done Sent false
7139 2961 Acked Write 3 Done Done Sent false
7140 2962 Acked Write 4 Done Done Sent false
7141 2963 Acked Read 2 Done Done Sent false
7142 2964 Acked Read 3 Done Done Sent false
7143 2965 Acked Read 10 Done Done Sent false
7144 2966 Acked Write 3 Done Done Sent false
7145 2967 Acked Write 6 Done Done Sent false
7146 2968 Acked Flush 0 Done Done Sent false
7147 2969 Acked Write 10 Done Done Sent false
7148 2970 Acked Write 9 Done Done Sent false
7149 2971 Acked Write 3 Done Done Sent false
7150 2972 Acked Read 9 Done Done Sent false
7151 2973 Acked Read 5 Done Done Sent false
7152 2974 Acked Read 6 Done Done Sent false
7153 2975 Acked Read 9 Done Done Sent false
7154 2976 Acked Write 1 Done Done Sent false
7155 2977 Acked Read 8 Done Done Sent false
7156 2978 Acked Write 3 Done Done Sent false
7157 2979 Acked Read 3 Done Done Sent false
7158 2980 Acked Write 7 Done Done Sent false
7159 2981 Acked Read 9 Done Done Sent false
7160 2982 Acked Read 7 Done Done Sent false
7161 2983 Acked Read 9 Done Done Sent false
7162 2984 Acked Write 2 Done Done Sent false
7163 2985 Acked Read 10 Done Done Sent false
7164 2986 Acked Write 7 Done Done Sent false
7165 2987 Acked Read 7 Done Done Sent false
7166 2988 Acked Write 6 Done Done Sent false
7167 2989 Acked Write 1 Done Done Sent false
7168 2990 Acked Read 10 Done Done Sent false
7169 2991 Acked Write 6 Done Done Sent false
7170 2992 Acked Read 3 Done Done Sent false
7171 2993 Acked Write 1 Done Done Sent false
7172 2994 Acked Write 6 Done Done Sent false
7173 2995 Acked Read 6 Done Done Sent false
7174 2996 Acked Write 10 Done Done Sent false
7175 2997 Acked Write 3 Done Done Sent false
7176 2998 Acked Read 9 Done Done Sent false
7177 2999 Acked Write 4 Done Done Sent false
7178 3000 Acked Flush 0 Done Done Sent false
7179 3001 Acked Write 4 Done Done Sent false
7180 3002 Acked Read 10 Done Done Sent false
7181 3003 Acked Write 4 Done Done Sent false
7182 3004 Acked Read 10 Done Done Sent false
7183 3005 Acked Read 5 Done Done Sent false
7184 3006 Acked Write 1 Done Done Sent false
7185 3007 Acked Write 10 Done Done Sent false
7186 3008 Acked Read 7 Done Done Sent false
7187 3009 Acked Write 5 Done Done Sent false
7188 3010 Acked Write 7 Done Done Sent false
7189 3011 Acked Write 5 Done Done Sent false
7190 3012 Acked Read 7 Done Done Sent false
7191 3013 Acked Read 4 Done Done Sent false
7192 3014 Acked Flush 0 Done Done Sent false
7193 3015 Acked Write 9 Done Done Sent false
7194 3016 Acked Flush 0 Done Done Sent false
7195 3017 Acked Write 4 Done Done Sent false
7196 3018 Acked Write 2 Done Done Sent false
7197 3019 Acked Read 2 Done Done Sent false
7198 3020 Acked Write 2 Done Done Sent false
7199 3021 Acked Read 6 Done Done Sent false
7200 3022 Acked Read 2 Done Done Sent false
7201 3023 Acked Read 1 Done Done Sent false
7202 3024 Acked Write 1 Done Done Sent false
7203 3025 Acked Write 8 Done Done Sent false
7204 3026 Acked Write 8 Done Done Sent false
7205 3027 Acked Read 1 Done Done Sent false
7206 3028 Acked Write 7 Done Done Sent false
7207 3029 Acked Read 4 Done Done Sent false
7208 3030 Acked Flush 0 Done Done Sent false
7209 3031 Acked Read 4 Done Done Sent false
7210 3032 Acked Write 9 Done Done Sent false
7211 3033 Acked Flush 0 Done Done Sent false
7212 3034 Acked Read 3 Done Done Sent false
7213 3035 Acked Write 10 Done Done Sent false
7214 3036 Acked Read 8 Done Done Sent false
7215 3037 Acked Write 8 Done Done Sent false
7216 3038 Acked Read 4 Done Done Sent false
7217 3039 Acked Read 4 Done Done Sent false
7218 3040 Acked Read 9 Done Done Sent false
7219 3041 Acked Flush 0 Done Done Sent false
7220 3042 Acked Flush 0 Done Done Sent false
7221 3043 Acked Read 4 Done Done Sent false
7222 3044 Acked Read 5 Done Done Sent false
7223 3045 Acked Flush 0 Done Done Sent false
7224 3046 Acked Read 10 Done Done Sent false
7225 3047 Acked Write 8 Done Done Sent false
7226 3048 Acked Read 8 Done Done Sent false
7227 3049 Acked Write 6 Done Done Sent false
7228 3050 Acked Write 10 Done Done Sent false
7229 3051 Acked Write 3 Done Done Sent false
7230 3052 Acked Write 8 Done Done Sent false
7231 3053 Acked Write 6 Done Done Sent false
7232 3054 Acked Read 8 Done Done Sent false
7233 3055 Acked Read 5 Done Done Sent false
7234 3056 Acked Write 5 Done Done Sent false
7235 3057 Acked Read 4 Done Done Sent false
7236 3058 Acked Write 4 Done Done Sent false
7237 3059 Acked Write 8 Done Done Sent false
7238 3060 Acked Write 2 Done Done Sent false
7239 3061 Acked Write 10 Done Done Sent false
7240 3062 Acked Write 3 Done Done Sent false
7241 3063 Acked Read 4 Done Done Sent false
7242 3064 Acked Read 3 Done Done Sent false
7243 3065 Acked Write 5 Done Done Sent false
7244 3066 Acked Write 6 Done Done Sent false
7245 3067 Acked Write 7 Done Done Sent false
7246 3068 Acked Write 9 Done Done Sent false
7247 3069 Acked Write 2 Done Done Sent false
7248 3070 Acked Write 3 Done Done Sent false
7249 3071 Acked Write 2 Done Done Sent false
7250 3072 Acked Read 1 Done Done Sent false
7251 3073 Acked Write 6 Done Done Sent false
7252 3074 Acked Flush 0 Done Done Sent false
7253 3075 Acked Read 5 Done Done Sent false
7254 3076 Acked Read 8 Done Done Sent false
7255 3077 Acked Read 10 Done Done Sent false
7256 3078 Acked Write 8 Done Done Sent false
7257 3079 Acked Read 7 Done Done Sent false
7258 3080 Acked Read 8 Done Done Sent false
7259 3081 Acked Write 5 Done Done Sent false
7260 3082 Acked Write 3 Done Done Sent false
7261 3083 Acked Write 5 Done Done Sent false
7262 3084 Acked Read 5 Done Done Sent false
7263 3085 Acked Read 9 Done Done Sent false
7264 3086 Acked Write 3 Done Done Sent false
7265 3087 Acked Read 8 Done Done Sent false
7266 3088 Acked Read 9 Done Done Sent false
7267 3089 Acked Write 1 Done Done Sent false
7268 3090 Acked Flush 0 Done Done Sent false
7269 3091 Acked Read 9 Done Done Sent false
7270 3092 Acked Write 7 Done Done Sent false
7271 3093 Acked Flush 0 Done Done Sent false
7272 3094 Acked Read 3 Done Done Sent false
7273 3095 Acked Read 9 Done Done Sent false
7274 3096 Acked Write 6 Done Done Sent false
7275 3097 Acked Read 6 Done Done Sent false
7276 3098 Acked Write 4 Done Done Sent false
7277 3099 Acked Write 3 Done Done Sent false
7278 3100 Acked Read 7 Done Done Sent false
7279 3101 Acked Flush 0 Done Done Sent false
7280 3102 Acked Flush 0 Done Done Sent false
7281 3103 Acked Flush 0 Done Done Sent false
7282 3104 Acked Read 2 Done Done Sent false
7283 3105 Acked Write 1 Done Done Sent false
7284 3106 Acked Write 4 Done Done Sent false
7285 3107 Acked Read 7 Done Done Sent false
7286 3108 Acked Write 3 Done Done Sent false
7287 3109 Acked Read 8 Done Done Sent false
7288 3110 Acked Read 3 Done Done Sent false
7289 3111 Acked Read 9 Done Done Sent false
7290 3112 Acked Read 5 Done Done Sent false
7291 STATES DS:0 DS:1 DS:2 TOTAL
7292 Sent 0 0 301 301
7293 Done 301 301 0 602
7294 Skipped 0 0 0 0
7295 Error 0 0 0 0
7296 Last Flush: Some(JobId(3103)) Some(JobId(3103)) Some(JobId(2811))
7297 Downstairs last five completed: 2811 2810 2809 2808 2807
7298 Upstairs last five completed: 3112 3111 3110 3109 3108
7299 CLIENT: Up:0 ds:301 act:2
7300 [8830] got start action
7301 [8830] I am going to restart
7302 Starting downstairs at port 8830
7303 Make output file at "/tmp/dsc/downstairs-8830.txt"
7304 [8830][2] reports Starting
7305 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1212)
7306 [8830][2] reports Running
7307 ----------------------------------------------------------------
7308 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
7309 States: Active Active Offline
7310 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7311 2812 Acked Write 4 Done Done Sent false
7312 2813 Acked Read 1 Done Done Sent false
7313 2814 Acked Read 10 Done Done Sent false
7314 2815 Acked Read 10 Done Done Sent false
7315 2816 Acked Read 4 Done Done Sent false
7316 2817 Acked Read 5 Done Done Sent false
7317 2818 Acked Write 7 Done Done Sent false
7318 2819 Acked Read 10 Done Done Sent false
7319 2820 Acked Read 9 Done Done Sent false
7320 2821 Acked Read 7 Done Done Sent false
7321 2822 Acked Read 10 Done Done Sent false
7322 2823 Acked Write 5 Done Done Sent false
7323 2824 Acked Read 5 Done Done Sent false
7324 2825 Acked Read 8 Done Done Sent false
7325 2826 Acked Write 8 Done Done Sent false
7326 2827 Acked Write 4 Done Done Sent false
7327 2828 Acked Read 3 Done Done Sent false
7328 2829 Acked Write 2 Done Done Sent false
7329 2830 Acked Read 3 Done Done Sent false
7330 2831 Acked Read 2 Done Done Sent false
7331 2832 Acked Flush 0 Done Done Sent false
7332 2833 Acked Write 7 Done Done Sent false
7333 2834 Acked Read 5 Done Done Sent false
7334 2835 Acked Write 2 Done Done Sent false
7335 2836 Acked Read 5 Done Done Sent false
7336 2837 Acked Write 8 Done Done Sent false
7337 2838 Acked Read 6 Done Done Sent false
7338 2839 Acked Read 3 Done Done Sent false
7339 2840 Acked Flush 0 Done Done Sent false
7340 2841 Acked Read 4 Done Done Sent false
7341 2842 Acked Read 4 Done Done Sent false
7342 2843 Acked Read 8 Done Done Sent false
7343 2844 Acked Write 6 Done Done Sent false
7344 2845 Acked Write 5 Done Done Sent false
7345 2846 Acked Read 1 Done Done Sent false
7346 2847 Acked Read 9 Done Done Sent false
7347 2848 Acked Read 3 Done Done Sent false
7348 2849 Acked Write 6 Done Done Sent false
7349 2850 Acked Read 3 Done Done Sent false
7350 2851 Acked Read 2 Done Done Sent false
7351 2852 Acked Read 7 Done Done Sent false
7352 2853 Acked Read 2 Done Done Sent false
7353 2854 Acked Read 5 Done Done Sent false
7354 2855 Acked Read 6 Done Done Sent false
7355 2856 Acked Read 2 Done Done Sent false
7356 2857 Acked Flush 0 Done Done Sent false
7357 2858 Acked Read 4 Done Done Sent false
7358 2859 Acked Flush 0 Done Done Sent false
7359 2860 Acked Read 3 Done Done Sent false
7360 2861 Acked Flush 0 Done Done Sent false
7361 2862 Acked Read 1 Done Done Sent false
7362 2863 Acked Read 4 Done Done Sent false
7363 2864 Acked Read 2 Done Done Sent false
7364 2865 Acked Read 3 Done Done Sent false
7365 2866 Acked Read 7 Done Done Sent false
7366 2867 Acked Read 2 Done Done Sent false
7367 2868 Acked Read 5 Done Done Sent false
7368 2869 Acked Read 7 Done Done Sent false
7369 2870 Acked Write 3 Done Done Sent false
7370 2871 Acked Read 2 Done Done Sent false
7371 2872 Acked Read 8 Done Done Sent false
7372 2873 Acked Read 9 Done Done Sent false
7373 2874 Acked Write 1 Done Done Sent false
7374 2875 Acked Read 6 Done Done Sent false
7375 2876 Acked Read 6 Done Done Sent false
7376 2877 Acked Flush 0 Done Done Sent false
7377 2878 Acked Read 3 Done Done Sent false
7378 2879 Acked Write 4 Done Done Sent false
7379 2880 Acked Write 10 Done Done Sent false
7380 2881 Acked Read 1 Done Done Sent false
7381 2882 Acked Read 6 Done Done Sent false
7382 2883 Acked Read 8 Done Done Sent false
7383 2884 Acked Write 2 Done Done Sent false
7384 2885 Acked Flush 0 Done Done Sent false
7385 2886 Acked Flush 0 Done Done Sent false
7386 2887 Acked Write 3 Done Done Sent false
7387 2888 Acked Write 1 Done Done Sent false
7388 2889 Acked Write 3 Done Done Sent false
7389 2890 Acked Read 6 Done Done Sent false
7390 2891 Acked Read 10 Done Done Sent false
7391 2892 Acked Read 7 Done Done Sent false
7392 2893 Acked Write 8 Done Done Sent false
7393 2894 Acked Read 4 Done Done Sent false
7394 2895 Acked Write 2 Done Done Sent false
7395 2896 Acked Read 5 Done Done Sent false
7396 2897 Acked Write 7 Done Done Sent false
7397 2898 Acked Write 8 Done Done Sent false
7398 2899 Acked Read 4 Done Done Sent false
7399 2900 Acked Read 10 Done Done Sent false
7400 2901 Acked Read 9 Done Done Sent false
7401 2902 Acked Read 2 Done Done Sent false
7402 2903 Acked Read 4 Done Done Sent false
7403 2904 Acked Write 5 Done Done Sent false
7404 2905 Acked Flush 0 Done Done Sent false
7405 2906 Acked Read 5 Done Done Sent false
7406 2907 Acked Flush 0 Done Done Sent false
7407 2908 Acked Write 10 Done Done Sent false
7408 2909 Acked Flush 0 Done Done Sent false
7409 2910 Acked Read 7 Done Done Sent false
7410 2911 Acked Write 7 Done Done Sent false
7411 2912 Acked Write 2 Done Done Sent false
7412 2913 Acked Write 9 Done Done Sent false
7413 2914 Acked Read 3 Done Done Sent false
7414 2915 Acked Write 3 Done Done Sent false
7415 2916 Acked Read 8 Done Done Sent false
7416 2917 Acked Read 2 Done Done Sent false
7417 2918 Acked Flush 0 Done Done Sent false
7418 2919 Acked Write 1 Done Done Sent false
7419 2920 Acked Flush 0 Done Done Sent false
7420 2921 Acked Read 4 Done Done Sent false
7421 2922 Acked Read 8 Done Done Sent false
7422 2923 Acked Read 8 Done Done Sent false
7423 2924 Acked Read 7 Done Done Sent false
7424 2925 Acked Read 2 Done Done Sent false
7425 2926 Acked Read 3 Done Done Sent false
7426 2927 Acked Flush 0 Done Done Sent false
7427 2928 Acked Write 8 Done Done Sent false
7428 2929 Acked Write 5 Done Done Sent false
7429 2930 Acked Read 4 Done Done Sent false
7430 2931 Acked Read 2 Done Done Sent false
7431 2932 Acked Read 7 Done Done Sent false
7432 2933 Acked Write 8 Done Done Sent false
7433 2934 Acked Read 10 Done Done Sent false
7434 2935 Acked Read 7 Done Done Sent false
7435 2936 Acked Write 6 Done Done Sent false
7436 2937 Acked Write 4 Done Done Sent false
7437 2938 Acked Read 10 Done Done Sent false
7438 2939 Acked Read 9 Done Done Sent false
7439 2940 Acked Read 7 Done Done Sent false
7440 2941 Acked Read 9 Done Done Sent false
7441 2942 Acked Read 4 Done Done Sent false
7442 2943 Acked Write 1 Done Done Sent false
7443 2944 Acked Flush 0 Done Done Sent false
7444 2945 Acked Read 9 Done Done Sent false
7445 2946 Acked Read 4 Done Done Sent false
7446 2947 Acked Write 9 Done Done Sent false
7447 2948 Acked Write 7 Done Done Sent false
7448 2949 Acked Read 9 Done Done Sent false
7449 2950 Acked Read 3 Done Done Sent false
7450 2951 Acked Read 3 Done Done Sent false
7451 2952 Acked Write 6 Done Done Sent false
7452 2953 Acked Read 1 Done Done Sent false
7453 2954 Acked Read 3 Done Done Sent false
7454 2955 Acked Write 6 Done Done Sent false
7455 2956 Acked Write 7 Done Done Sent false
7456 2957 Acked Read 5 Done Done Sent false
7457 2958 Acked Write 3 Done Done Sent false
7458 2959 Acked Write 8 Done Done Sent false
7459 2960 Acked Read 1 Done Done Sent false
7460 2961 Acked Write 3 Done Done Sent false
7461 2962 Acked Write 4 Done Done Sent false
7462 2963 Acked Read 2 Done Done Sent false
7463 2964 Acked Read 3 Done Done Sent false
7464 2965 Acked Read 10 Done Done Sent false
7465 2966 Acked Write 3 Done Done Sent false
7466 2967 Acked Write 6 Done Done Sent false
7467 2968 Acked Flush 0 Done Done Sent false
7468 2969 Acked Write 10 Done Done Sent false
7469 2970 Acked Write 9 Done Done Sent false
7470 2971 Acked Write 3 Done Done Sent false
7471 2972 Acked Read 9 Done Done Sent false
7472 2973 Acked Read 5 Done Done Sent false
7473 2974 Acked Read 6 Done Done Sent false
7474 2975 Acked Read 9 Done Done Sent false
7475 2976 Acked Write 1 Done Done Sent false
7476 2977 Acked Read 8 Done Done Sent false
7477 2978 Acked Write 3 Done Done Sent false
7478 2979 Acked Read 3 Done Done Sent false
7479 2980 Acked Write 7 Done Done Sent false
7480 2981 Acked Read 9 Done Done Sent false
7481 2982 Acked Read 7 Done Done Sent false
7482 2983 Acked Read 9 Done Done Sent false
7483 2984 Acked Write 2 Done Done Sent false
7484 2985 Acked Read 10 Done Done Sent false
7485 2986 Acked Write 7 Done Done Sent false
7486 2987 Acked Read 7 Done Done Sent false
7487 2988 Acked Write 6 Done Done Sent false
7488 2989 Acked Write 1 Done Done Sent false
7489 2990 Acked Read 10 Done Done Sent false
7490 2991 Acked Write 6 Done Done Sent false
7491 2992 Acked Read 3 Done Done Sent false
7492 2993 Acked Write 1 Done Done Sent false
7493 2994 Acked Write 6 Done Done Sent false
7494 2995 Acked Read 6 Done Done Sent false
7495 2996 Acked Write 10 Done Done Sent false
7496 2997 Acked Write 3 Done Done Sent false
7497 2998 Acked Read 9 Done Done Sent false
7498 2999 Acked Write 4 Done Done Sent false
7499 3000 Acked Flush 0 Done Done Sent false
7500 3001 Acked Write 4 Done Done Sent false
7501 3002 Acked Read 10 Done Done Sent false
7502 3003 Acked Write 4 Done Done Sent false
7503 3004 Acked Read 10 Done Done Sent false
7504 3005 Acked Read 5 Done Done Sent false
7505 3006 Acked Write 1 Done Done Sent false
7506 3007 Acked Write 10 Done Done Sent false
7507 3008 Acked Read 7 Done Done Sent false
7508 3009 Acked Write 5 Done Done Sent false
7509 3010 Acked Write 7 Done Done Sent false
7510 3011 Acked Write 5 Done Done Sent false
7511 3012 Acked Read 7 Done Done Sent false
7512 3013 Acked Read 4 Done Done Sent false
7513 3014 Acked Flush 0 Done Done Sent false
7514 3015 Acked Write 9 Done Done Sent false
7515 3016 Acked Flush 0 Done Done Sent false
7516 3017 Acked Write 4 Done Done Sent false
7517 3018 Acked Write 2 Done Done Sent false
7518 3019 Acked Read 2 Done Done Sent false
7519 3020 Acked Write 2 Done Done Sent false
7520 3021 Acked Read 6 Done Done Sent false
7521 3022 Acked Read 2 Done Done Sent false
7522 3023 Acked Read 1 Done Done Sent false
7523 3024 Acked Write 1 Done Done Sent false
7524 3025 Acked Write 8 Done Done Sent false
7525 3026 Acked Write 8 Done Done Sent false
7526 3027 Acked Read 1 Done Done Sent false
7527 3028 Acked Write 7 Done Done Sent false
7528 3029 Acked Read 4 Done Done Sent false
7529 3030 Acked Flush 0 Done Done Sent false
7530 3031 Acked Read 4 Done Done Sent false
7531 3032 Acked Write 9 Done Done Sent false
7532 3033 Acked Flush 0 Done Done Sent false
7533 3034 Acked Read 3 Done Done Sent false
7534 3035 Acked Write 10 Done Done Sent false
7535 3036 Acked Read 8 Done Done Sent false
7536 3037 Acked Write 8 Done Done Sent false
7537 3038 Acked Read 4 Done Done Sent false
7538 3039 Acked Read 4 Done Done Sent false
7539 3040 Acked Read 9 Done Done Sent false
7540 3041 Acked Flush 0 Done Done Sent false
7541 3042 Acked Flush 0 Done Done Sent false
7542 3043 Acked Read 4 Done Done Sent false
7543 3044 Acked Read 5 Done Done Sent false
7544 3045 Acked Flush 0 Done Done Sent false
7545 3046 Acked Read 10 Done Done Sent false
7546 3047 Acked Write 8 Done Done Sent false
7547 3048 Acked Read 8 Done Done Sent false
7548 3049 Acked Write 6 Done Done Sent false
7549 3050 Acked Write 10 Done Done Sent false
7550 3051 Acked Write 3 Done Done Sent false
7551 3052 Acked Write 8 Done Done Sent false
7552 3053 Acked Write 6 Done Done Sent false
7553 3054 Acked Read 8 Done Done Sent false
7554 3055 Acked Read 5 Done Done Sent false
7555 3056 Acked Write 5 Done Done Sent false
7556 3057 Acked Read 4 Done Done Sent false
7557 3058 Acked Write 4 Done Done Sent false
7558 3059 Acked Write 8 Done Done Sent false
7559 3060 Acked Write 2 Done Done Sent false
7560 3061 Acked Write 10 Done Done Sent false
7561 3062 Acked Write 3 Done Done Sent false
7562 3063 Acked Read 4 Done Done Sent false
7563 3064 Acked Read 3 Done Done Sent false
7564 3065 Acked Write 5 Done Done Sent false
7565 3066 Acked Write 6 Done Done Sent false
7566 3067 Acked Write 7 Done Done Sent false
7567 3068 Acked Write 9 Done Done Sent false
7568 3069 Acked Write 2 Done Done Sent false
7569 3070 Acked Write 3 Done Done Sent false
7570 3071 Acked Write 2 Done Done Sent false
7571 3072 Acked Read 1 Done Done Sent false
7572 3073 Acked Write 6 Done Done Sent false
7573 3074 Acked Flush 0 Done Done Sent false
7574 3075 Acked Read 5 Done Done Sent false
7575 3076 Acked Read 8 Done Done Sent false
7576 3077 Acked Read 10 Done Done Sent false
7577 3078 Acked Write 8 Done Done Sent false
7578 3079 Acked Read 7 Done Done Sent false
7579 3080 Acked Read 8 Done Done Sent false
7580 3081 Acked Write 5 Done Done Sent false
7581 3082 Acked Write 3 Done Done Sent false
7582 3083 Acked Write 5 Done Done Sent false
7583 3084 Acked Read 5 Done Done Sent false
7584 3085 Acked Read 9 Done Done Sent false
7585 3086 Acked Write 3 Done Done Sent false
7586 3087 Acked Read 8 Done Done Sent false
7587 3088 Acked Read 9 Done Done Sent false
7588 3089 Acked Write 1 Done Done Sent false
7589 3090 Acked Flush 0 Done Done Sent false
7590 3091 Acked Read 9 Done Done Sent false
7591 3092 Acked Write 7 Done Done Sent false
7592 3093 Acked Flush 0 Done Done Sent false
7593 3094 Acked Read 3 Done Done Sent false
7594 3095 Acked Read 9 Done Done Sent false
7595 3096 Acked Write 6 Done Done Sent false
7596 3097 Acked Read 6 Done Done Sent false
7597 3098 Acked Write 4 Done Done Sent false
7598 3099 Acked Write 3 Done Done Sent false
7599 3100 Acked Read 7 Done Done Sent false
7600 3101 Acked Flush 0 Done Done Sent false
7601 3102 Acked Flush 0 Done Done Sent false
7602 3103 Acked Flush 0 Done Done Sent false
7603 3104 Acked Read 2 Done Done Sent false
7604 3105 Acked Write 1 Done Done Sent false
7605 3106 Acked Write 4 Done Done Sent false
7606 3107 Acked Read 7 Done Done Sent false
7607 3108 Acked Write 3 Done Done Sent false
7608 3109 Acked Read 8 Done Done Sent false
7609 3110 Acked Read 3 Done Done Sent false
7610 3111 Acked Read 9 Done Done Sent false
7611 3112 Acked Read 5 Done Done Sent false
7612 3113 Acked Flush 0 Done Done Sent false
7613 STATES DS:0 DS:1 DS:2 TOTAL
7614 Sent 0 0 302 302
7615 Done 302 302 0 604
7616 Skipped 0 0 0 0
7617 Error 0 0 0 0
7618 Last Flush: Some(JobId(3113)) Some(JobId(3113)) Some(JobId(2811))
7619 Downstairs last five completed: 2811 2810 2809 2808 2807
7620 Upstairs last five completed: 3113 3112 3111 3110 3109
7621 CLIENT: Up:0 ds:302 act:2
76222025-01-23T17:39:19.120ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
76232025-01-23T17:39:19.120ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50295) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
76242025-01-23T17:39:19.121ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
76252025-01-23T17:39:19.122ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
76262025-01-23T17:39:19.122ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
76272025-01-23T17:39:19.122ZINFOcrucible: send last flush ID to this DS: Some(JobId(2811)) = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
76282025-01-23T17:39:19.122ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
76292025-01-23T17:39:19.122ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(2811)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
76302025-01-23T17:39:19.122ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(2811)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
7631 ----------------------------------------------------------------
7632 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7633 Downstairs last five completed: 3113 3112 3111 3110 3109
7634 Upstairs last five completed: 3113 3112 3111 3110 3109
7635 CLIENT: Up:0 ds:0 act:3
7636 Replay: All jobs finished
7637 Main task has work to do, go find it
7638 got dsc Stop(0)
7639 stop 0
7640 Jan 23 17:39:21.371 INFO request completed[8810] Got stop action so:false kr:false
7641 , latency_us: 356, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 8bcf6ac9-baba-4061-a65a-63a4b9ffb9eb, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
7642 Jan 23 17:39:21.372 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:39:21.372095052Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
7643 INFO request completed, latency_us: 135, response_code: 200, uri: /state/cid/0, method: GET, req_id: 545369b1-a584-47ba-90d7-0bb4f240bc87, remote_addr: 127.0.0.1:42015, local_addr: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:39:21.372177484Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
7644 127.0.0.1:9998
76452025-01-23T17:39:21.372ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
76462025-01-23T17:39:21.372ZINFOcrucible: client task is exiting = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
7647 [8810] Exited with: ExitStatus(unix_wait_status(9))
7648 [8810][0] reports Exit
7649 Jan 23 17:39:25.374 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/0, method: GET, req_id: 7d249d03-2d8c-461a-be81-a0385fd5e9ae, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
7650 001/300 Read block 4526 len 32768
7651 002/300 Read block 2909 len 16384
7652 003/300 Write block 4470 len 32768 data: 2 2 2 2 2 2 2 2
7653 004/300 Read block 1116 len 36864
7654 005/300 Write block 276 len 28672 data: 2 2 2 3 2 2 3
7655 006/300 Write block 3098 len 36864 data: 4 4 4 4 3 3 3 3 4
7656 007/300 Write block 1625 len 32768 data: 3 3 3 3 2 2 2 2
7657 008/300 Read block 3199 len 4096
7658 009/300 Flush
7659 010/300 Read block 4152 len 28672
7660 011/300 Write block 1144 len 16384 data: 5 5 4 4
7661 012/300 Write block 2262 len 16384 data: 3 3 3 3
7662 013/300 Read block 3868 len 32768
7663 014/300 Write block 2048 len 28672 data: 3 2 2 3 3 3 3
7664 015/300 Read block 4541 len 20480
7665 016/300 Read block 2215 len 4096
7666 017/300 Write block 3758 len 16384 data: 2 2 2 2
7667 018/300 Read block 1315 len 36864
7668 019/300 Read block 4871 len 12288
7669 020/300 Write block 1423 len 16384 data: 3 3 3 3
7670 021/300 Read block 4718 len 40960
7671 022/300 Read block 3324 len 36864
7672 023/300 Write block 4604 len 16384 data: 4 4 4 4
7673 024/300 Write block 332 len 24576 data: 5 5 3 3 2 3
7674 025/300 Write block 1002 len 4096 data: 2
7675 026/300 Write block 632 len 40960 data: 3 2 2 2 2 2 2 2 4 3
7676 027/300 Write block 920 len 28672 data: 2 2 2 2 2 2 2
7677 028/300 Read block 344 len 32768
7678 029/300 Write block 1900 len 40960 data: 4 4 4 3 5 6 6 7 7 6
7679 030/300 Write block 4982 len 36864 data: 2 2 2 2 4 3 3 2 3
7680 031/300 Write block 3863 len 20480 data: 3 2 2 2 2
7681 032/300 Read block 3109 len 32768
7682 033/300 Read block 813 len 40960
7683 034/300 Read block 4980 len 12288
7684 035/300 Read block 3198 len 24576
7685 036/300 Read block 4626 len 16384
7686 037/300 Read block 136 len 28672
7687 038/300 Read block 2050 len 20480
7688 039/300 Write block 873 len 8192 data: 2 2
7689 040/300 Read block 1907 len 36864
7690 041/300 Write block 1391 len 32768 data: 3 3 3 3 2 2 2 2
7691 042/300 Read block 1679 len 24576
7692 043/300 Read block 624 len 16384
7693 044/300 Read block 4173 len 40960
7694 045/300 Write block 2854 len 32768 data: 3 3 3 5 4 5 4 4
7695 046/300 Write block 1640 len 16384 data: 3 3 4 4
7696 047/300 Read block 3117 len 28672
7697 048/300 Read block 2461 len 20480
7698 049/300 Read block 1990 len 8192
7699 050/300 Flush
7700 051/300 Read block 621 len 16384
7701 052/300 Read block 4974 len 28672
7702 053/300 Read block 3113 len 28672
7703 054/300 Read block 1816 len 20480
7704 055/300 Write block 1982 len 4096 data: 2
7705 056/300 Read block 1022 len 16384
7706 057/300 Write block 1659 len 32768 data: 2 2 2 2 2 2 3 2
7707 058/300 Write block 2612 len 28672 data: 4 3 4 4 4 3 2
7708 059/300 Read block 2475 len 12288
7709 060/300 Read block 4236 len 12288
7710 061/300 Read block 3683 len 32768
7711 062/300 Write block 4948 len 28672 data: 2 3 3 3 4 4 3
7712 063/300 Read block 1829 len 24576
7713 064/300 Read block 3332 len 28672
7714 065/300 Read block 4355 len 40960
7715 066/300 Read block 76 len 28672
7716 067/300 Write block 2262 len 4096 data: 4
7717 068/300 Read block 2808 len 40960
7718 069/300 Read block 1297 len 24576
7719 070/300 Read block 4213 len 40960
7720 071/300 Read block 60 len 8192
7721 072/300 Write block 1449 len 16384 data: 2 3 2 2
7722 073/300 Flush
7723 074/300 Read block 4427 len 40960
7724 075/300 Read block 3617 len 28672
7725 076/300 Read block 1865 len 40960
7726 077/300 Read block 4651 len 8192
7727 078/300 Write block 3429 len 32768 data: 6 6 6 6 7 6 6 4
7728 079/300 Write block 3722 len 12288 data: 3 3 3
7729 080/300 Read block 4496 len 12288
7730 081/300 Write block 4869 len 32768 data: 3 4 4 4 4 4 4 4
7731 082/300 Write block 3521 len 40960 data: 2 3 5 4 4 4 3 2 3 3
7732 083/300 Read block 4553 len 4096
7733 084/300 Write block 4555 len 16384 data: 3 4 3 3
7734 085/300 Read block 1927 len 28672
7735 086/300 Flush
7736 087/300 Flush
7737 088/300 Write block 4622 len 36864 data: 5 4 2 2 2 2 3 3 3
7738 089/300 Read block 3096 len 32768
7739 090/300 Read block 1192 len 24576
7740 091/300 Write block 3292 len 20480 data: 3 2 2 2 2
7741 092/300 Read block 4235 len 24576
7742 093/300 Read block 669 len 4096
7743 094/300 Read block 2376 len 32768
7744 095/300 Read block 2185 len 8192
7745 096/300 Write block 1612 len 40960 data: 3 3 2 3 4 4 5 6 5 4
7746 097/300 Read block 2255 len 8192
7747 098/300 Read block 1229 len 40960
7748 099/300 Read block 1024 len 40960
7749 100/300 Write block 1799 len 16384 data: 3 4 4 4
7750 101/300 Write block 4126 len 8192 data: 2 2
7751 102/300 Read block 2354 len 40960
7752 103/300 Write block 3397 len 28672 data: 3 3 3 2 2 2 2
7753 104/300 Read block 1672 len 16384
7754 105/300 Write block 1257 len 28672 data: 6 7 7 7 5 5 5
7755 106/300 Write block 3269 len 28672 data: 2 2 2 2 2 2 2
7756 107/300 Read block 2212 len 12288
7757 108/300 Read block 925 len 16384
7758 109/300 Read block 4463 len 20480
7759 110/300 Read block 640 len 16384
7760 111/300 Write block 4810 len 12288 data: 2 2 3
7761 112/300 Write block 2477 len 8192 data: 3 3
7762 113/300 Write block 3827 len 16384 data: 2 2 2 2
7763 114/300 Write block 510 len 20480 data: 2 2 2 2 2
7764 115/300 Read block 871 len 24576
7765 116/300 Flush
7766 117/300 Read block 3318 len 24576
7767 118/300 Read block 3409 len 8192
7768 119/300 Read block 4034 len 24576
7769 120/300 Write block 3872 len 8192 data: 3 4
7770 121/300 Read block 3588 len 24576
7771 122/300 Read block 2469 len 4096
7772 123/300 Read block 2024 len 36864
7773 124/300 Write block 2432 len 20480 data: 2 2 2 2 2
7774 125/300 Read block 3986 len 36864
7775 126/300 Write block 4174 len 40960 data: 2 2 3 4 4 5 5 5 4 4
7776 127/300 Write block 570 len 24576 data: 4 5 4 3 3 2
7777 128/300 Read block 3466 len 12288
7778 129/300 Write block 133 len 36864 data: 2 2 2 2 2 2 2 3 3
7779 130/300 Read block 3744 len 36864
7780 131/300 Read block 4246 len 40960
7781 132/300 Read block 4259 len 32768
7782 133/300 Read block 1484 len 40960
7783 134/300 Write block 2770 len 12288 data: 3 2 2
7784 135/300 Read block 2410 len 12288
7785 136/300 Read block 4340 len 4096
7786 137/300 Read block 369 len 20480
7787 138/300 Read block 4017 len 24576
7788 139/300 Write block 221 len 4096 data: 2
7789 140/300 Read block 622 len 32768
7790 141/300 Read block 2400 len 4096
7791 142/300 Flush
7792 143/300 Read block 4859 len 40960
7793 144/300 Write block 2696 len 36864 data: 2 2 2 2 2 2 2 2 2
7794 145/300 Flush
7795 146/300 Write block 2868 len 36864 data: 5 5 3 2 2 2 2 2 2
7796 147/300 Read block 3578 len 20480
7797 148/300 Read block 1916 len 4096
7798 149/300 Read block 2675 len 24576
7799 150/300 Read block 1080 len 12288
7800 151/300 Write block 1107 len 36864 data: 2 2 2 3 3 3 3 3 3
7801 152/300 Write block 2223 len 4096 data: 3
7802 153/300 Write block 4167 len 8192 data: 2 2
7803 154/300 Write block 3068 len 12288 data: 2 3 2
7804 155/300 Read block 4800 len 36864
7805 156/300 Write block 4352 len 24576 data: 3 2 2 2 2 2
7806 157/300 Write block 1865 len 20480 data: 5 7 7 6 5
7807 158/300 Read block 4958 len 16384
7808 159/300 Write block 2433 len 4096 data: 3
7809 160/300 Write block 881 len 28672 data: 3 2 2 2 2 3 3
7810 161/300 Flush
7811 162/300 Read block 2909 len 8192
7812 163/300 Read block 4522 len 36864
7813 164/300 Read block 4307 len 20480
7814 165/300 Read block 4154 len 28672
7815 166/300 Read block 3708 len 28672
7816 167/300 Read block 2350 len 4096
7817 168/300 Read block 4015 len 40960
7818 169/300 Read block 1902 len 28672
7819 170/300 Read block 1450 len 32768
7820 171/300 Flush
7821 172/300 Read block 682 len 12288
7822 173/300 Read block 1032 len 40960
7823 174/300 Write block 3132 len 12288 data: 2 2 2
7824 175/300 Write block 4917 len 40960 data: 2 2 2 2 2 2 3 3 3 3
7825 176/300 Read block 2105 len 4096
7826 177/300 Write block 1301 len 4096 data: 4
7827 178/300 Read block 2459 len 36864
7828 179/300 Read block 1846 len 8192
7829 180/300 Read block 1653 len 28672
7830 181/300 Flush
7831 182/300 Read block 1989 len 40960
7832 183/300 Read block 3457 len 24576
7833 184/300 Write block 1662 len 28672 data: 3 3 3 4 3 2 2
7834 185/300 Write block 2399 len 20480 data: 4 4 3 3 3
7835 186/300 Read block 3859 len 24576
7836 187/300 Read block 1987 len 32768
7837 188/300 Write block 259 len 36864 data: 2 3 3 3 2 2 2 2 2
7838 189/300 Read block 898 len 40960
7839 190/300 Read block 1163 len 16384
7840 191/300 Write block 3745 len 20480 data: 3 3 2 2 2
7841 192/300 Read block 1554 len 12288
7842 193/300 Write block 4223 len 32768 data: 2 2 2 2 2 2 2 2
7843 194/300 Write block 880 len 8192 data: 3 4
7844 195/300 Read block 664 len 24576
7845 196/300 Read block 1117 len 12288
7846 197/300 Flush
7847 198/300 Read block 2111 len 24576
7848 199/300 Read block 3291 len 8192
7849 200/300 Write block 1680 len 20480 data: 3 4 4 4 4
7850 201/300 Read block 3068 len 40960
7851 202/300 Flush
7852 203/300 Write block 3409 len 20480 data: 2 2 2 2 2
7853 204/300 Read block 160 len 24576
7854 205/300 Write block 2710 len 40960 data: 4 4 4 4 4 3 3 2 2 2
7855 206/300 Read block 204 len 36864
7856 207/300 Read block 3700 len 36864
7857 208/300 Write block 2684 len 8192 data: 3 3
7858 209/300 Write block 3108 len 20480 data: 4 4 5 6 5
7859 210/300 Read block 306 len 20480
7860 211/300 Read block 2467 len 16384
7861 212/300 Write block 4418 len 8192 data: 2 2
7862 213/300 Write block 995 len 16384 data: 4 4 4 3
7863 214/300 Write block 4975 len 20480 data: 2 2 2 2 2
7864 215/300 Read block 3820 len 8192
7865 216/300 Write block 2616 len 40960 data: 5 4 3 2 3 3 3 3 4 3
7866 217/300 Flush
7867 218/300 Write block 2933 len 28672 data: 2 2 2 2 2 3 3
7868 219/300 Read block 1776 len 12288
7869 220/300 Read block 1724 len 36864
7870 221/300 Read block 1003 len 28672
7871 222/300 Read block 1811 len 40960
7872 223/300 Write block 186 len 24576 data: 3 3 3 2 2 2
7873 224/300 Write block 4682 len 20480 data: 4 4 3 4 4
7874 225/300 Write block 1693 len 4096 data: 3
7875 226/300 Read block 1673 len 32768
7876 227/300 Write block 2529 len 28672 data: 4 4 4 4 3 4 4
7877 228/300 Write block 4029 len 12288 data: 3 3 3
7878 229/300 Write block 4670 len 16384 data: 4 4 4 4
7879 230/300 Read block 544 len 40960
7880 231/300 Write block 1919 len 32768 data: 3 3 3 4 4 6 4 4
7881 232/300 Read block 4708 len 36864
7882 233/300 Read block 736 len 40960
7883 234/300 Write block 2317 len 16384 data: 2 2 2 2
7884 235/300 Write block 361 len 4096 data: 3
7885 236/300 Read block 2825 len 16384
7886 237/300 Read block 1317 len 4096
7887 238/300 Read block 3679 len 4096
7888 239/300 Write block 2819 len 8192 data: 2 2
7889 240/300 Read block 3469 len 8192
7890 241/300 Read block 4749 len 16384
7891 242/300 Write block 252 len 20480 data: 4 4 4 2 3
7892 243/300 Read block 1347 len 40960
7893 244/300 Read block 4417 len 24576
7894 245/300 Read block 1885 len 20480
7895 246/300 Read block 4269 len 36864
7896 247/300 Write block 4054 len 8192 data: 3 2
7897 248/300 Read block 1471 len 12288
7898 249/300 Read block 4649 len 40960
7899 250/300 Flush
7900 251/300 Read block 1782 len 32768
7901 252/300 Write block 2606 len 12288 data: 4 4 4
7902 253/300 Flush
7903 254/300 Flush
7904 255/300 Read block 4636 len 20480
7905 256/300 Read block 164 len 4096
7906 257/300 Read block 853 len 32768
7907 258/300 Write block 1171 len 16384 data: 2 2 2 2
7908 259/300 Write block 4208 len 8192 data: 5 3
7909 260/300 Read block 1475 len 20480
7910 261/300 Read block 1323 len 28672
7911 262/300 Write block 2423 len 8192 data: 2 2
7912 263/300 Write block 1024 len 20480 data: 3 3 3 4 4
7913 264/300 Read block 183 len 16384
7914 265/300 Read block 2464 len 16384
7915 266/300 Write block 4556 len 12288 data: 5 4 4
7916 267/300 Read block 2004 len 24576
7917 268/300 Read block 3485 len 12288
7918 269/300 Flush
7919 270/300 Read block 4016 len 16384
7920 271/300 Write block 2146 len 16384 data: 3 3 3 3
7921 272/300 Read block 3713 len 24576
7922 273/300 Read block 2300 len 20480
7923 274/300 Read block 2571 len 36864
7924 275/300 Read block 4480 len 20480
7925 276/300 Read block 1645 len 8192
7926 277/300 Write block 279 len 8192 data: 4 3
7927 278/300 Read block 3410 len 36864
7928 279/300 Read block 4450 len 20480
7929 280/300 Read block 320 len 20480
7930 281/300 Write block 4573 len 24576 data: 3 3 3 4 4 5
7931 282/300 Flush
7932 283/300 Read block 2316 len 8192
7933 284/300 Write block 2386 len 40960 data: 3 2 2 2 2 2 2 3 3 3
7934 285/300 Read block 2373 len 32768
7935 286/300 Write block 1637 len 4096 data: 2
7936 287/300 Read block 3027 len 28672
7937 288/300 Read block 3719 len 36864
7938 289/300 Read block 2993 len 20480
7939 290/300 Write block 4834 len 16384 data: 2 2 2 2
7940 291/300 Read block 18 len 36864
7941 292/300 Read block 1797 len 20480
7942 293/300 Write block 2086 len 4096 data: 2
7943 294/300 Flush
7944 295/300 Read block 1703 len 32768
7945 296/300 Read block 4516 len 20480
7946 297/300 Write block 1774 len 4096 data: 2
7947 298/300 Write block 2380 len 28672 data: 5 5 5 5 5 5 4
7948 299/300 Read block 2267 len 32768
7949 300/300 Write block 72 len 40960 data: 2 2 2 2 2 3 3 3 4 3
7950 301/300 Read block 4202 len 4096
7951 Main task has work to do, go find it
7952 got dsc Start(0)
7953 start 0
7954 Jan 23 17:39:25.634 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/0, method: POST, req_id: 99d1b139-f67e-47b3-a831-588fdd908f96, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
7955 Replay: started 0, returned:Ok(())
7956 ----------------------------------------------------------------
7957 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7958 States: Offline Active Active
7959 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7960 3114 Acked Read 8 Sent Done Done false
7961 3115 Acked Read 4 Sent Done Done false
7962 3116 Acked Write 8 Sent Done Done false
7963 3117 Acked Read 9 Sent Done Done false
7964 3118 Acked Write 7 Sent Done Done false
7965 3119 Acked Write 9 Sent Done Done false
7966 3120 Acked Write 8 Sent Done Done false
7967 3121 Acked Read 1 Sent Done Done false
7968 3122 Acked Flush 0 Sent Done Done false
7969 3123 Acked Read 7 Sent Done Done false
7970 3124 Acked Write 4 Sent Done Done false
7971 3125 Acked Write 4 Sent Done Done false
7972 3126 Acked Read 8 Sent Done Done false
7973 3127 Acked Write 7 Sent Done Done false
7974 3128 Acked Read 5 Sent Done Done false
7975 3129 Acked Read 1 Sent Done Done false
7976 3130 Acked Write 4 Sent Done Done false
7977 3131 Acked Read 9 Sent Done Done false
7978 3132 Acked Read 3 Sent Done Done false
7979 3133 Acked Write 4 Sent Done Done false
7980 3134 Acked Read 10 Sent Done Done false
7981 3135 Acked Read 9 Sent Done Done false
7982 3136 Acked Write 4 Sent Done Done false
7983 3137 Acked Write 6 Sent Done Done false
7984 3138 Acked Write 1 Sent Done Done false
7985 3139 Acked Write 10 Sent Done Done false
7986 3140 Acked Write 7 Sent Done Done false
7987 3141 Acked Read 8 Sent Done Done false
7988 3142 Acked Write 10 Sent Done Done false
7989 3143 Acked Write 9 Sent Done Done false
7990 3144 Acked Write 5 Sent Done Done false
7991 3145 Acked Read 8 Sent Done Done false
7992 3146 Acked Read 10 Sent Done Done false
7993 3147 Acked Read 3 Sent Done Done false
7994 3148 Acked Read 6 Sent Done Done false
7995 3149 Acked Read 4 Sent Done Done false
7996 3150 Acked Read 7 Sent Done Done false
7997 3151 Acked Read 5 Sent Done Done false
7998 3152 Acked Write 2 Sent Done Done false
7999 3153 Acked Read 9 Sent Done Done false
8000 3154 Acked Write 8 Sent Done Done false
8001 3155 Acked Read 6 Sent Done Done false
8002 3156 Acked Read 4 Sent Done Done false
8003 3157 Acked Read 10 Sent Done Done false
8004 3158 Acked Write 8 Sent Done Done false
8005 3159 Acked Write 4 Sent Done Done false
8006 3160 Acked Read 7 Sent Done Done false
8007 3161 Acked Read 5 Sent Done Done false
8008 3162 Acked Read 2 Sent Done Done false
8009 3163 Acked Flush 0 Sent Done Done false
8010 3164 Acked Read 4 Sent Done Done false
8011 3165 Acked Read 7 Sent Done Done false
8012 3166 Acked Read 7 Sent Done Done false
8013 3167 Acked Read 5 Sent Done Done false
8014 3168 Acked Write 1 Sent Done Done false
8015 3169 Acked Read 4 Sent Done Done false
8016 3170 Acked Write 8 Sent Done Done false
8017 3171 Acked Write 7 Sent Done Done false
8018 3172 Acked Read 3 Sent Done Done false
8019 3173 Acked Read 3 Sent Done Done false
8020 3174 Acked Read 8 Sent Done Done false
8021 3175 Acked Write 7 Sent Done Done false
8022 3176 Acked Read 6 Sent Done Done false
8023 3177 Acked Read 7 Sent Done Done false
8024 3178 Acked Read 10 Sent Done Done false
8025 3179 Acked Read 7 Sent Done Done false
8026 3180 Acked Write 1 Sent Done Done false
8027 3181 Acked Read 10 Sent Done Done false
8028 3182 Acked Read 6 Sent Done Done false
8029 3183 Acked Read 10 Sent Done Done false
8030 3184 Acked Read 2 Sent Done Done false
8031 3185 Acked Write 4 Sent Done Done false
8032 3186 Acked Flush 0 Sent Done Done false
8033 3187 Acked Read 10 Sent Done Done false
8034 3188 Acked Read 7 Sent Done Done false
8035 3189 Acked Read 10 Sent Done Done false
8036 3190 Acked Read 2 Sent Done Done false
8037 3191 Acked Write 8 Sent Done Done false
8038 3192 Acked Write 3 Sent Done Done false
8039 3193 Acked Read 3 Sent Done Done false
8040 3194 Acked Write 8 Sent Done Done false
8041 3195 Acked Write 10 Sent Done Done false
8042 3196 Acked Read 1 Sent Done Done false
8043 3197 Acked Write 4 Sent Done Done false
8044 3198 Acked Read 7 Sent Done Done false
8045 3199 Acked Flush 0 Sent Done Done false
8046 3200 Acked Flush 0 Sent Done Done false
8047 3201 Acked Write 9 Sent Done Done false
8048 3202 Acked Read 8 Sent Done Done false
8049 3203 Acked Read 6 Sent Done Done false
8050 3204 Acked Write 5 Sent Done Done false
8051 3205 Acked Read 6 Sent Done Done false
8052 3206 Acked Read 1 Sent Done Done false
8053 3207 Acked Read 8 Sent Done Done false
8054 3208 Acked Read 2 Sent Done Done false
8055 3209 Acked Write 10 Sent Done Done false
8056 3210 Acked Read 2 Sent Done Done false
8057 3211 Acked Read 10 Sent Done Done false
8058 3212 Acked Read 10 Sent Done Done false
8059 3213 Acked Write 4 Sent Done Done false
8060 3214 Acked Write 2 Sent Done Done false
8061 3215 Acked Read 10 Sent Done Done false
8062 3216 Acked Write 7 Sent Done Done false
8063 3217 Acked Read 4 Sent Done Done false
8064 3218 Acked Write 7 Sent Done Done false
8065 3219 Acked Write 7 Sent Done Done false
8066 3220 Acked Read 3 Sent Done Done false
8067 3221 Acked Read 4 Sent Done Done false
8068 3222 Acked Read 5 Sent Done Done false
8069 3223 Acked Read 4 Sent Done Done false
8070 3224 Acked Write 3 Sent Done Done false
8071 3225 Acked Write 2 Sent Done Done false
8072 3226 Acked Write 4 Sent Done Done false
8073 3227 Acked Write 5 Sent Done Done false
8074 3228 Acked Read 6 Sent Done Done false
8075 3229 Acked Flush 0 Sent Done Done false
8076 3230 Acked Read 6 Sent Done Done false
8077 3231 Acked Read 2 Sent Done Done false
8078 3232 Acked Read 6 Sent Done Done false
8079 3233 Acked Write 2 Sent Done Done false
8080 3234 Acked Read 6 Sent Done Done false
8081 3235 Acked Read 1 Sent Done Done false
8082 3236 Acked Read 9 Sent Done Done false
8083 3237 Acked Write 5 Sent Done Done false
8084 3238 Acked Read 9 Sent Done Done false
8085 3239 Acked Write 10 Sent Done Done false
8086 3240 Acked Write 6 Sent Done Done false
8087 3241 Acked Read 3 Sent Done Done false
8088 3242 Acked Write 9 Sent Done Done false
8089 3243 Acked Read 9 Sent Done Done false
8090 3244 Acked Read 10 Sent Done Done false
8091 3245 Acked Read 8 Sent Done Done false
8092 3246 Acked Read 10 Sent Done Done false
8093 3247 Acked Write 3 Sent Done Done false
8094 3248 Acked Read 3 Sent Done Done false
8095 3249 Acked Read 1 Sent Done Done false
8096 3250 Acked Read 5 Sent Done Done false
8097 3251 Acked Read 6 Sent Done Done false
8098 3252 Acked Write 1 Sent Done Done false
8099 3253 Acked Read 8 Sent Done Done false
8100 3254 Acked Read 1 Sent Done Done false
8101 3255 Acked Flush 0 Sent Done Done false
8102 3256 Acked Read 10 Sent Done Done false
8103 3257 Acked Write 9 Sent Done Done false
8104 3258 Acked Flush 0 Sent Done Done false
8105 3259 Acked Write 9 Sent Done Done false
8106 3260 Acked Read 5 Sent Done Done false
8107 3261 Acked Read 1 Sent Done Done false
8108 3262 Acked Read 6 Sent Done Done false
8109 3263 Acked Read 3 Sent Done Done false
8110 3264 Acked Write 9 Sent Done Done false
8111 3265 Acked Write 1 Sent Done Done false
8112 3266 Acked Write 2 Sent Done Done false
8113 3267 Acked Write 3 Sent Done Done false
8114 3268 Acked Read 9 Sent Done Done false
8115 3269 Acked Write 6 Sent Done Done false
8116 3270 Acked Write 5 Sent Done Done false
8117 3271 Acked Read 4 Sent Done Done false
8118 3272 Acked Write 1 Sent Done Done false
8119 3273 Acked Write 7 Sent Done Done false
8120 3274 Acked Flush 0 Sent Done Done false
8121 3275 Acked Read 2 Sent Done Done false
8122 3276 Acked Read 9 Sent Done Done false
8123 3277 Acked Read 5 Sent Done Done false
8124 3278 Acked Read 7 Sent Done Done false
8125 3279 Acked Read 7 Sent Done Done false
8126 3280 Acked Read 1 Sent Done Done false
8127 3281 Acked Read 10 Sent Done Done false
8128 3282 Acked Read 7 Sent Done Done false
8129 3283 Acked Read 8 Sent Done Done false
8130 3284 Acked Flush 0 Sent Done Done false
8131 3285 Acked Read 3 Sent Done Done false
8132 3286 Acked Read 10 Sent Done Done false
8133 3287 Acked Write 3 Sent Done Done false
8134 3288 Acked Write 10 Sent Done Done false
8135 3289 Acked Read 1 Sent Done Done false
8136 3290 Acked Write 1 Sent Done Done false
8137 3291 Acked Read 9 Sent Done Done false
8138 3292 Acked Read 2 Sent Done Done false
8139 3293 Acked Read 7 Sent Done Done false
8140 3294 Acked Flush 0 Sent Done Done false
8141 3295 Acked Read 10 Sent Done Done false
8142 3296 Acked Read 6 Sent Done Done false
8143 3297 Acked Write 7 Sent Done Done false
8144 3298 Acked Write 5 Sent Done Done false
8145 3299 Acked Read 6 Sent Done Done false
8146 3300 Acked Read 8 Sent Done Done false
8147 3301 Acked Write 9 Sent Done Done false
8148 3302 Acked Read 10 Sent Done Done false
8149 3303 Acked Read 4 Sent Done Done false
8150 3304 Acked Write 5 Sent Done Done false
8151 3305 Acked Read 3 Sent Done Done false
8152 3306 Acked Write 8 Sent Done Done false
8153 3307 Acked Write 2 Sent Done Done false
8154 3308 Acked Read 6 Sent Done Done false
8155 3309 Acked Read 3 Sent Done Done false
8156 3310 Acked Flush 0 Sent Done Done false
8157 3311 Acked Read 6 Sent Done Done false
8158 3312 Acked Read 2 Sent Done Done false
8159 3313 Acked Write 5 Sent Done Done false
8160 3314 Acked Read 10 Sent Done Done false
8161 3315 Acked Flush 0 Sent Done Done false
8162 3316 Acked Write 5 Sent Done Done false
8163 3317 Acked Read 6 Sent Done Done false
8164 3318 Acked Write 10 Sent Done Done false
8165 3319 Acked Read 9 Sent Done Done false
8166 3320 Acked Read 9 Sent Done Done false
8167 3321 Acked Write 2 Sent Done Done false
8168 3322 Acked Write 5 Sent Done Done false
8169 3323 Acked Read 5 Sent Done Done false
8170 3324 Acked Read 4 Sent Done Done false
8171 3325 Acked Write 2 Sent Done Done false
8172 3326 Acked Write 4 Sent Done Done false
8173 3327 Acked Write 5 Sent Done Done false
8174 3328 Acked Read 2 Sent Done Done false
8175 3329 Acked Write 10 Sent Done Done false
8176 3330 Acked Flush 0 Sent Done Done false
8177 3331 Acked Write 7 Sent Done Done false
8178 3332 Acked Read 3 Sent Done Done false
8179 3333 Acked Read 9 Sent Done Done false
8180 3334 Acked Read 7 Sent Done Done false
8181 3335 Acked Read 10 Sent Done Done false
8182 3336 Acked Write 6 Sent Done Done false
8183 3337 Acked Write 5 Sent Done Done false
8184 3338 Acked Write 1 Sent Done Done false
8185 3339 Acked Read 8 Sent Done Done false
8186 3340 Acked Write 7 Sent Done Done false
8187 3341 Acked Write 3 Sent Done Done false
8188 3342 Acked Write 4 Sent Done Done false
8189 3343 Acked Read 10 Sent Done Done false
8190 3344 Acked Write 8 Sent Done Done false
8191 3345 Acked Read 9 Sent Done Done false
8192 3346 Acked Read 10 Sent Done Done false
8193 3347 Acked Write 4 Sent Done Done false
8194 3348 Acked Write 1 Sent Done Done false
8195 3349 Acked Read 4 Sent Done Done false
8196 3350 Acked Read 1 Sent Done Done false
8197 3351 Acked Read 1 Sent Done Done false
8198 3352 Acked Write 2 Sent Done Done false
8199 3353 Acked Read 2 Sent Done Done false
8200 3354 Acked Read 4 Sent Done Done false
8201 3355 Acked Write 5 Sent Done Done false
8202 3356 Acked Read 10 Sent Done Done false
8203 3357 Acked Read 6 Sent Done Done false
8204 3358 Acked Read 5 Sent Done Done false
8205 3359 Acked Read 9 Sent Done Done false
8206 3360 Acked Write 2 Sent Done Done false
8207 3361 Acked Read 3 Sent Done Done false
8208 3362 Acked Read 10 Sent Done Done false
8209 3363 Acked Flush 0 Sent Done Done false
8210 3364 Acked Read 8 Sent Done Done false
8211 3365 Acked Write 3 Sent Done Done false
8212 3366 Acked Flush 0 Sent Done Done false
8213 3367 Acked Flush 0 Sent Done Done false
8214 3368 Acked Read 5 Sent Done Done false
8215 3369 Acked Read 1 Sent Done Done false
8216 3370 Acked Read 8 Sent Done Done false
8217 3371 Acked Write 4 Sent Done Done false
8218 3372 Acked Write 2 Sent Done Done false
8219 3373 Acked Read 5 Sent Done Done false
8220 3374 Acked Read 7 Sent Done Done false
8221 3375 Acked Write 2 Sent Done Done false
8222 3376 Acked Write 5 Sent Done Done false
8223 3377 Acked Read 4 Sent Done Done false
8224 3378 Acked Read 4 Sent Done Done false
8225 3379 Acked Write 3 Sent Done Done false
8226 3380 Acked Read 6 Sent Done Done false
8227 3381 Acked Read 3 Sent Done Done false
8228 3382 Acked Flush 0 Sent Done Done false
8229 3383 Acked Read 4 Sent Done Done false
8230 3384 Acked Write 4 Sent Done Done false
8231 3385 Acked Read 6 Sent Done Done false
8232 3386 Acked Read 5 Sent Done Done false
8233 3387 Acked Read 9 Sent Done Done false
8234 3388 Acked Read 5 Sent Done Done false
8235 3389 Acked Read 2 Sent Done Done false
8236 3390 Acked Write 2 Sent Done Done false
8237 3391 Acked Read 9 Sent Done Done false
8238 3392 Acked Read 5 Sent Done Done false
8239 3393 Acked Read 5 Sent Done Done false
8240 3394 Acked Write 6 Sent Done Done false
8241 3395 Acked Flush 0 Sent Done Done false
8242 3396 Acked Read 2 Sent Done Done false
8243 3397 Acked Write 10 Sent Done Done false
8244 3398 Acked Read 8 Sent Done Done false
8245 3399 Acked Write 1 Sent Done Done false
8246 3400 Acked Read 7 Sent Done Done false
8247 3401 Acked Read 9 Sent Done Done false
8248 3402 Acked Read 5 Sent Done Done false
8249 3403 Acked Write 4 Sent Done Done false
8250 3404 Acked Read 9 Sent Done Done false
8251 3405 Acked Read 5 Sent Done Done false
8252 3406 Acked Write 1 Sent Done Done false
8253 3407 Acked Flush 0 Sent Done Done false
8254 3408 Acked Read 8 Sent Done Done false
8255 3409 Acked Read 5 Sent Done Done false
8256 3410 Acked Write 1 Sent Done Done false
8257 3411 Acked Write 7 Sent Done Done false
8258 3412 Acked Read 8 Sent Done Done false
8259 3413 Acked Write 10 Sent Done Done false
8260 3414 Acked Read 1 Sent Done Done false
8261 STATES DS:0 DS:1 DS:2 TOTAL
8262 Sent 301 0 0 301
8263 Done 0 301 301 602
8264 Skipped 0 0 0 0
8265 Error 0 0 0 0
8266 Last Flush: Some(JobId(3113)) Some(JobId(3407)) Some(JobId(3407))
8267 Downstairs last five completed: 3113 3112 3111 3110 3109
8268 Upstairs last five completed: 3414 3413 3412 3411 3410
8269 CLIENT: Up:0 ds:301 act:2
8270 ----------------------------------------------------------------
8271 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
8272 States: Offline Active Active
8273 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8274 3114 Acked Read 8 Sent Done Done false
8275 3115 Acked Read 4 Sent Done Done false
8276 3116 Acked Write 8 Sent Done Done false
8277 3117 Acked Read 9 Sent Done Done false
8278 3118 Acked Write 7 Sent Done Done false
8279 3119 Acked Write 9 Sent Done Done false
8280 3120 Acked Write 8 Sent Done Done false
8281 3121 Acked Read 1 Sent Done Done false
8282 3122 Acked Flush 0 Sent Done Done false
8283 3123 Acked Read 7 Sent Done Done false
8284 3124 Acked Write 4 Sent Done Done false
8285 3125 Acked Write 4 Sent Done Done false
8286 3126 Acked Read 8 Sent Done Done false
8287 3127 Acked Write 7 Sent Done Done false
8288 3128 Acked Read 5 Sent Done Done false
8289 3129 Acked Read 1 Sent Done Done false
8290 3130 Acked Write 4 Sent Done Done false
8291 3131 Acked Read 9 Sent Done Done false
8292 3132 Acked Read 3 Sent Done Done false
8293 3133 Acked Write 4 Sent Done Done false
8294 3134 Acked Read 10 Sent Done Done false
8295 3135 Acked Read 9 Sent Done Done false
8296 3136 Acked Write 4 Sent Done Done false
8297 3137 Acked Write 6 Sent Done Done false
8298 3138 Acked Write 1 Sent Done Done false
8299 3139 Acked Write 10 Sent Done Done false
8300 3140 Acked Write 7 Sent Done Done false
8301 3141 Acked Read 8 Sent Done Done false
8302 3142 Acked Write 10 Sent Done Done false
8303 3143 Acked Write 9 Sent Done Done false
8304 3144 Acked Write 5 Sent Done Done false
8305 3145 Acked Read 8 Sent Done Done false
8306 3146 Acked Read 10 Sent Done Done false
8307 3147 Acked Read 3 Sent Done Done false
8308 3148 Acked Read 6 Sent Done Done false
8309 3149 Acked Read 4 Sent Done Done false
8310 3150 Acked Read 7 Sent Done Done false
8311 3151 Acked Read 5 Sent Done Done false
8312 3152 Acked Write 2 Sent Done Done false
8313 3153 Acked Read 9 Sent Done Done false
8314 3154 Acked Write 8 Sent Done Done false
8315 3155 Acked Read 6 Sent Done Done false
8316 3156 Acked Read 4 Sent Done Done false
8317 3157 Acked Read 10 Sent Done Done false
8318 3158 Acked Write 8 Sent Done Done false
8319 3159 Acked Write 4 Sent Done Done false
8320 3160 Acked Read 7 Sent Done Done false
8321 3161 Acked Read 5 Sent Done Done false
8322 3162 Acked Read 2 Sent Done Done false
8323 3163 Acked Flush 0 Sent Done Done false
8324 3164 Acked Read 4 Sent Done Done false
8325 3165 Acked Read 7 Sent Done Done false
8326 3166 Acked Read 7 Sent Done Done false
8327 3167 Acked Read 5 Sent Done Done false
8328 3168 Acked Write 1 Sent Done Done false
8329 3169 Acked Read 4 Sent Done Done false
8330 3170 Acked Write 8 Sent Done Done false
8331 3171 Acked Write 7 Sent Done Done false
8332 3172 Acked Read 3 Sent Done Done false
8333 3173 Acked Read 3 Sent Done Done false
8334 3174 Acked Read 8 Sent Done Done false
8335 3175 Acked Write 7 Sent Done Done false
8336 3176 Acked Read 6 Sent Done Done false
8337 3177 Acked Read 7 Sent Done Done false
8338 3178 Acked Read 10 Sent Done Done false
8339 3179 Acked Read 7 Sent Done Done false
8340 3180 Acked Write 1 Sent Done Done false
8341 3181 Acked Read 10 Sent Done Done false
8342 3182 Acked Read 6 Sent Done Done false
8343 3183 Acked Read 10 Sent Done Done false
8344 3184 Acked Read 2 Sent Done Done false
8345 3185 Acked Write 4 Sent Done Done false
8346 3186 Acked Flush 0 Sent Done Done false
8347 3187 Acked Read 10 Sent Done Done false
8348 3188 Acked Read 7 Sent Done Done false
8349 3189 Acked Read 10 Sent Done Done false
8350 3190 Acked Read 2 Sent Done Done false
8351 3191 Acked Write 8 Sent Done Done false
8352 3192 Acked Write 3 Sent Done Done false
8353 3193 Acked Read 3 Sent Done Done false
8354 3194 Acked Write 8 Sent Done Done false
8355 3195 Acked Write 10 Sent Done Done false
8356 3196 Acked Read 1 Sent Done Done false
8357 3197 Acked Write 4 Sent Done Done false
8358 3198 Acked Read 7 Sent Done Done false
8359 3199 Acked Flush 0 Sent Done Done false
8360 3200 Acked Flush 0 Sent Done Done false
8361 3201 Acked Write 9 Sent Done Done false
8362 3202 Acked Read 8 Sent Done Done false
8363 3203 Acked Read 6 Sent Done Done false
8364 3204 Acked Write 5 Sent Done Done false
8365 3205 Acked Read 6 Sent Done Done false
8366 3206 Acked Read 1 Sent Done Done false
8367 3207 Acked Read 8 Sent Done Done false
8368 3208 Acked Read 2 Sent Done Done false
8369 3209 Acked Write 10 Sent Done Done false
8370 3210 Acked Read 2 Sent Done Done false
8371 3211 Acked Read 10 Sent Done Done false
8372 3212 Acked Read 10 Sent Done Done false
8373 3213 Acked Write 4 Sent Done Done false
8374 3214 Acked Write 2 Sent Done Done false
8375 3215 Acked Read 10 Sent Done Done false
8376 3216 Acked Write 7 Sent Done Done false
8377 3217 Acked Read 4 Sent Done Done false
8378 3218 Acked Write 7 Sent Done Done false
8379 3219 Acked Write 7 Sent Done Done false
8380 3220 Acked Read 3 Sent Done Done false
8381 3221 Acked Read 4 Sent Done Done false
8382 3222 Acked Read 5 Sent Done Done false
8383 3223 Acked Read 4 Sent Done Done false
8384 3224 Acked Write 3 Sent Done Done false
8385 3225 Acked Write 2 Sent Done Done false
8386 3226 Acked Write 4 Sent Done Done false
8387 3227 Acked Write 5 Sent Done Done false
8388 3228 Acked Read 6 Sent Done Done false
8389 3229 Acked Flush 0 Sent Done Done false
8390 3230 Acked Read 6 Sent Done Done false
8391 3231 Acked Read 2 Sent Done Done false
8392 3232 Acked Read 6 Sent Done Done false
8393 3233 Acked Write 2 Sent Done Done false
8394 3234 Acked Read 6 Sent Done Done false
8395 3235 Acked Read 1 Sent Done Done false
8396 3236 Acked Read 9 Sent Done Done false
8397 3237 Acked Write 5 Sent Done Done false
8398 3238 Acked Read 9 Sent Done Done false
8399 3239 Acked Write 10 Sent Done Done false
8400 3240 Acked Write 6 Sent Done Done false
8401 3241 Acked Read 3 Sent Done Done false
8402 3242 Acked Write 9 Sent Done Done false
8403 3243 Acked Read 9 Sent Done Done false
8404 3244 Acked Read 10 Sent Done Done false
8405 3245 Acked Read 8 Sent Done Done false
8406 3246 Acked Read 10 Sent Done Done false
8407 3247 Acked Write 3 Sent Done Done false
8408 3248 Acked Read 3 Sent Done Done false
8409 3249 Acked Read 1 Sent Done Done false
8410 3250 Acked Read 5 Sent Done Done false
8411 3251 Acked Read 6 Sent Done Done false
8412 3252 Acked Write 1 Sent Done Done false
8413 3253 Acked Read 8 Sent Done Done false
8414 3254 Acked Read 1 Sent Done Done false
8415 3255 Acked Flush 0 Sent Done Done false
8416 3256 Acked Read 10 Sent Done Done false
8417 3257 Acked Write 9 Sent Done Done false
8418 3258 Acked Flush 0 Sent Done Done false
8419 3259 Acked Write 9 Sent Done Done false
8420 3260 Acked Read 5 Sent Done Done false
8421 3261 Acked Read 1 Sent Done Done false
8422 3262 Acked Read 6 Sent Done Done false
8423 3263 Acked Read 3 Sent Done Done false
8424 3264 Acked Write 9 Sent Done Done false
8425 3265 Acked Write 1 Sent Done Done false
8426 3266 Acked Write 2 Sent Done Done false
8427 3267 Acked Write 3 Sent Done Done false
8428 3268 Acked Read 9 Sent Done Done false
8429 3269 Acked Write 6 Sent Done Done false
8430 3270 Acked Write 5 Sent Done Done false
8431 3271 Acked Read 4 Sent Done Done false
8432 3272 Acked Write 1 Sent Done Done false
8433 3273 Acked Write 7 Sent Done Done false
8434 3274 Acked Flush 0 Sent Done Done false
8435 3275 Acked Read 2 Sent Done Done false
8436 3276 Acked Read 9 Sent Done Done false
8437 3277 Acked Read 5 Sent Done Done false
8438 3278 Acked Read 7 Sent Done Done false
8439 3279 Acked Read 7 Sent Done Done false
8440 3280 Acked Read 1 Sent Done Done false
8441 3281 Acked Read 10 Sent Done Done false
8442 3282 Acked Read 7 Sent Done Done false
8443 3283 Acked Read 8 Sent Done Done false
8444 3284 Acked Flush 0 Sent Done Done false
8445 3285 Acked Read 3 Sent Done Done false
8446 3286 Acked Read 10 Sent Done Done false
8447 3287 Acked Write 3 Sent Done Done false
8448 3288 Acked Write 10 Sent Done Done false
8449 3289 Acked Read 1 Sent Done Done false
8450 3290 Acked Write 1 Sent Done Done false
8451 3291 Acked Read 9 Sent Done Done false
8452 3292 Acked Read 2 Sent Done Done false
8453 3293 Acked Read 7 Sent Done Done false
8454 3294 Acked Flush 0 Sent Done Done false
8455 3295 Acked Read 10 Sent Done Done false
8456 3296 Acked Read 6 Sent Done Done false
8457 3297 Acked Write 7 Sent Done Done false
8458 3298 Acked Write 5 Sent Done Done false
8459 3299 Acked Read 6 Sent Done Done false
8460 3300 Acked Read 8 Sent Done Done false
8461 3301 Acked Write 9 Sent Done Done false
8462 3302 Acked Read 10 Sent Done Done false
8463 3303 Acked Read 4 Sent Done Done false
8464 3304 Acked Write 5 Sent Done Done false
8465 3305 Acked Read 3 Sent Done Done false
8466 3306 Acked Write 8 Sent Done Done false
8467 3307 Acked Write 2 Sent Done Done false
8468 3308 Acked Read 6 Sent Done Done false
8469 3309 Acked Read 3 Sent Done Done false
8470 3310 Acked Flush 0 Sent Done Done false
8471 3311 Acked Read 6 Sent Done Done false
8472 3312 Acked Read 2 Sent Done Done false
8473 3313 Acked Write 5 Sent Done Done false
8474 3314 Acked Read 10 Sent Done Done false
8475 3315 Acked Flush 0 Sent Done Done false
8476 3316 Acked Write 5 Sent Done Done false
8477 3317 Acked Read 6 Sent Done Done false
8478 3318 Acked Write 10 Sent Done Done false
8479 3319 Acked Read 9 Sent Done Done false
8480 3320 Acked Read 9 Sent Done Done false
8481 3321 Acked Write 2 Sent Done Done false
8482 3322 Acked Write 5 Sent Done Done false
8483 3323 Acked Read 5 Sent Done Done false
8484 3324 Acked Read 4 Sent Done Done false
8485 3325 Acked Write 2 Sent Done Done false
8486 3326 Acked Write 4 Sent Done Done false
8487 3327 Acked Write 5 Sent Done Done false
8488 3328 Acked Read 2 Sent Done Done false
8489 3329 Acked Write 10 Sent Done Done false
8490 3330 Acked Flush 0 Sent Done Done false
8491 3331 Acked Write 7 Sent Done Done false
8492 3332 Acked Read 3 Sent Done Done false
8493 3333 Acked Read 9 Sent Done Done false
8494 3334 Acked Read 7 Sent Done Done false
8495 3335 Acked Read 10 Sent Done Done false
8496 3336 Acked Write 6 Sent Done Done false
8497 3337 Acked Write 5 Sent Done Done false
8498 3338 Acked Write 1 Sent Done Done false
8499 3339 Acked Read 8 Sent Done Done false
8500 3340 Acked Write 7 Sent Done Done false
8501 3341 Acked Write 3 Sent Done Done false
8502 3342 Acked Write 4 Sent Done Done false
8503 3343 Acked Read 10 Sent Done Done false
8504 3344 Acked Write 8 Sent Done Done false
8505 3345 Acked Read 9 Sent Done Done false
8506 3346 Acked Read 10 Sent Done Done false
8507 3347 Acked Write 4 Sent Done Done false
8508 3348 Acked Write 1 Sent Done Done false
8509 3349 Acked Read 4 Sent Done Done false
8510 3350 Acked Read 1 Sent Done Done false
8511 3351 Acked Read 1 Sent Done Done false
8512 3352 Acked Write 2 Sent Done Done false
8513 3353 Acked Read 2 Sent Done Done false
8514 3354 Acked Read 4 Sent Done Done false
8515 3355 Acked Write 5 Sent Done Done false
8516 3356 Acked Read 10 Sent Done Done false
8517 3357 Acked Read 6 Sent Done Done false
8518 3358 Acked Read 5 Sent Done Done false
8519 3359 Acked Read 9 Sent Done Done false
8520 3360 Acked Write 2 Sent Done Done false
8521 3361 Acked Read 3 Sent Done Done false
8522 3362 Acked Read 10 Sent Done Done false
8523 3363 Acked Flush 0 Sent Done Done false
8524 3364 Acked Read 8 Sent Done Done false
8525 3365 Acked Write 3 Sent Done Done false
8526 3366 Acked Flush 0 Sent Done Done false
8527 3367 Acked Flush 0 Sent Done Done false
8528 3368 Acked Read 5 Sent Done Done false
8529 3369 Acked Read 1 Sent Done Done false
8530 3370 Acked Read 8 Sent Done Done false
8531 3371 Acked Write 4 Sent Done Done false
8532 3372 Acked Write 2 Sent Done Done false
8533 3373 Acked Read 5 Sent Done Done false
8534 3374 Acked Read 7 Sent Done Done false
8535 3375 Acked Write 2 Sent Done Done false
8536 3376 Acked Write 5 Sent Done Done false
8537 3377 Acked Read 4 Sent Done Done false
8538 3378 Acked Read 4 Sent Done Done false
8539 3379 Acked Write 3 Sent Done Done false
8540 3380 Acked Read 6 Sent Done Done false
8541 3381 Acked Read 3 Sent Done Done false
8542 3382 Acked Flush 0 Sent Done Done false
8543 3383 Acked Read 4 Sent Done Done false
8544 3384 Acked Write 4 Sent Done Done false
8545 3385 Acked Read 6 Sent Done Done false
8546 3386 Acked Read 5 Sent Done Done false
8547 3387 Acked Read 9 Sent Done Done false
8548 3388 Acked Read 5 Sent Done Done false
8549 3389 Acked Read 2 Sent Done Done false
8550 3390 Acked Write 2 Sent Done Done false
8551 3391 Acked Read 9 Sent Done Done false
8552 3392 Acked Read 5 Sent Done Done false
8553 3393 Acked Read 5 Sent Done Done false
8554 3394 Acked Write 6 Sent Done Done false
8555 3395 Acked Flush 0 Sent Done Done false
8556 3396 Acked Read 2 Sent Done Done false
8557 3397 Acked Write 10 Sent Done Done false
8558 3398 Acked Read 8 Sent Done Done false
8559 3399 Acked Write 1 Sent Done Done false
8560 3400 Acked Read 7 Sent Done Done false
8561 3401 Acked Read 9 Sent Done Done false
8562 3402 Acked Read 5 Sent Done Done false
8563 3403 Acked Write 4 Sent Done Done false
8564 3404 Acked Read 9 Sent Done Done false
8565 3405 Acked Read 5 Sent Done Done false
8566 3406 Acked Write 1 Sent Done Done false
8567 3407 Acked Flush 0 Sent Done Done false
8568 3408 Acked Read 8 Sent Done Done false
8569 3409 Acked Read 5 Sent Done Done false
8570 3410 Acked Write 1 Sent Done Done false
8571 3411 Acked Write 7 Sent Done Done false
8572 3412 Acked Read 8 Sent Done Done false
8573 3413 Acked Write 10 Sent Done Done false
8574 3414 Acked Read 1 Sent Done Done false
8575 3415 Acked Flush 0 Sent Done Done false
8576 STATES DS:0 DS:1 DS:2 TOTAL
8577 Sent 302 0 0 302
8578 Done 0 302 302 604
8579 Skipped 0 0 0 0
8580 Error 0 0 0 0
8581 Last Flush: Some(JobId(3113)) Some(JobId(3415)) Some(JobId(3415))
8582 Downstairs last five completed: 3113 3112 3111 3110 3109
8583 Upstairs last five completed: 3415 3414 3413 3412 3411
8584 CLIENT: Up:0 ds:302 act:2
85852025-01-23T17:39:31.373ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
85862025-01-23T17:39:31.373ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
85872025-01-23T17:39:31.373ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
85882025-01-23T17:39:31.374ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
85892025-01-23T17:39:31.374ZINFOcrucible: client task is exiting = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
8590 [8810] got start action
8591 [8810] I am going to restart
8592 Starting downstairs at port 8810
8593 Make output file at "/tmp/dsc/downstairs-8810.txt"
8594 [8810][0] reports Starting
8595 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1213)
8596 [8810][0] reports Running
8597 ----------------------------------------------------------------
8598 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
8599 States: Offline Active Active
8600 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8601 3114 Acked Read 8 Sent Done Done false
8602 3115 Acked Read 4 Sent Done Done false
8603 3116 Acked Write 8 Sent Done Done false
8604 3117 Acked Read 9 Sent Done Done false
8605 3118 Acked Write 7 Sent Done Done false
8606 3119 Acked Write 9 Sent Done Done false
8607 3120 Acked Write 8 Sent Done Done false
8608 3121 Acked Read 1 Sent Done Done false
8609 3122 Acked Flush 0 Sent Done Done false
8610 3123 Acked Read 7 Sent Done Done false
8611 3124 Acked Write 4 Sent Done Done false
8612 3125 Acked Write 4 Sent Done Done false
8613 3126 Acked Read 8 Sent Done Done false
8614 3127 Acked Write 7 Sent Done Done false
8615 3128 Acked Read 5 Sent Done Done false
8616 3129 Acked Read 1 Sent Done Done false
8617 3130 Acked Write 4 Sent Done Done false
8618 3131 Acked Read 9 Sent Done Done false
8619 3132 Acked Read 3 Sent Done Done false
8620 3133 Acked Write 4 Sent Done Done false
8621 3134 Acked Read 10 Sent Done Done false
8622 3135 Acked Read 9 Sent Done Done false
8623 3136 Acked Write 4 Sent Done Done false
8624 3137 Acked Write 6 Sent Done Done false
8625 3138 Acked Write 1 Sent Done Done false
8626 3139 Acked Write 10 Sent Done Done false
8627 3140 Acked Write 7 Sent Done Done false
8628 3141 Acked Read 8 Sent Done Done false
8629 3142 Acked Write 10 Sent Done Done false
8630 3143 Acked Write 9 Sent Done Done false
8631 3144 Acked Write 5 Sent Done Done false
8632 3145 Acked Read 8 Sent Done Done false
8633 3146 Acked Read 10 Sent Done Done false
8634 3147 Acked Read 3 Sent Done Done false
8635 3148 Acked Read 6 Sent Done Done false
8636 3149 Acked Read 4 Sent Done Done false
8637 3150 Acked Read 7 Sent Done Done false
8638 3151 Acked Read 5 Sent Done Done false
8639 3152 Acked Write 2 Sent Done Done false
8640 3153 Acked Read 9 Sent Done Done false
8641 3154 Acked Write 8 Sent Done Done false
8642 3155 Acked Read 6 Sent Done Done false
8643 3156 Acked Read 4 Sent Done Done false
8644 3157 Acked Read 10 Sent Done Done false
8645 3158 Acked Write 8 Sent Done Done false
8646 3159 Acked Write 4 Sent Done Done false
8647 3160 Acked Read 7 Sent Done Done false
8648 3161 Acked Read 5 Sent Done Done false
8649 3162 Acked Read 2 Sent Done Done false
8650 3163 Acked Flush 0 Sent Done Done false
8651 3164 Acked Read 4 Sent Done Done false
8652 3165 Acked Read 7 Sent Done Done false
8653 3166 Acked Read 7 Sent Done Done false
8654 3167 Acked Read 5 Sent Done Done false
8655 3168 Acked Write 1 Sent Done Done false
8656 3169 Acked Read 4 Sent Done Done false
8657 3170 Acked Write 8 Sent Done Done false
8658 3171 Acked Write 7 Sent Done Done false
8659 3172 Acked Read 3 Sent Done Done false
8660 3173 Acked Read 3 Sent Done Done false
8661 3174 Acked Read 8 Sent Done Done false
8662 3175 Acked Write 7 Sent Done Done false
8663 3176 Acked Read 6 Sent Done Done false
8664 3177 Acked Read 7 Sent Done Done false
8665 3178 Acked Read 10 Sent Done Done false
8666 3179 Acked Read 7 Sent Done Done false
8667 3180 Acked Write 1 Sent Done Done false
8668 3181 Acked Read 10 Sent Done Done false
8669 3182 Acked Read 6 Sent Done Done false
8670 3183 Acked Read 10 Sent Done Done false
8671 3184 Acked Read 2 Sent Done Done false
8672 3185 Acked Write 4 Sent Done Done false
8673 3186 Acked Flush 0 Sent Done Done false
8674 3187 Acked Read 10 Sent Done Done false
8675 3188 Acked Read 7 Sent Done Done false
8676 3189 Acked Read 10 Sent Done Done false
8677 3190 Acked Read 2 Sent Done Done false
8678 3191 Acked Write 8 Sent Done Done false
8679 3192 Acked Write 3 Sent Done Done false
8680 3193 Acked Read 3 Sent Done Done false
8681 3194 Acked Write 8 Sent Done Done false
8682 3195 Acked Write 10 Sent Done Done false
8683 3196 Acked Read 1 Sent Done Done false
8684 3197 Acked Write 4 Sent Done Done false
8685 3198 Acked Read 7 Sent Done Done false
8686 3199 Acked Flush 0 Sent Done Done false
8687 3200 Acked Flush 0 Sent Done Done false
8688 3201 Acked Write 9 Sent Done Done false
8689 3202 Acked Read 8 Sent Done Done false
8690 3203 Acked Read 6 Sent Done Done false
8691 3204 Acked Write 5 Sent Done Done false
8692 3205 Acked Read 6 Sent Done Done false
8693 3206 Acked Read 1 Sent Done Done false
8694 3207 Acked Read 8 Sent Done Done false
8695 3208 Acked Read 2 Sent Done Done false
8696 3209 Acked Write 10 Sent Done Done false
8697 3210 Acked Read 2 Sent Done Done false
8698 3211 Acked Read 10 Sent Done Done false
8699 3212 Acked Read 10 Sent Done Done false
8700 3213 Acked Write 4 Sent Done Done false
8701 3214 Acked Write 2 Sent Done Done false
8702 3215 Acked Read 10 Sent Done Done false
8703 3216 Acked Write 7 Sent Done Done false
8704 3217 Acked Read 4 Sent Done Done false
8705 3218 Acked Write 7 Sent Done Done false
8706 3219 Acked Write 7 Sent Done Done false
8707 3220 Acked Read 3 Sent Done Done false
8708 3221 Acked Read 4 Sent Done Done false
8709 3222 Acked Read 5 Sent Done Done false
8710 3223 Acked Read 4 Sent Done Done false
8711 3224 Acked Write 3 Sent Done Done false
8712 3225 Acked Write 2 Sent Done Done false
8713 3226 Acked Write 4 Sent Done Done false
8714 3227 Acked Write 5 Sent Done Done false
8715 3228 Acked Read 6 Sent Done Done false
8716 3229 Acked Flush 0 Sent Done Done false
8717 3230 Acked Read 6 Sent Done Done false
8718 3231 Acked Read 2 Sent Done Done false
8719 3232 Acked Read 6 Sent Done Done false
8720 3233 Acked Write 2 Sent Done Done false
8721 3234 Acked Read 6 Sent Done Done false
8722 3235 Acked Read 1 Sent Done Done false
8723 3236 Acked Read 9 Sent Done Done false
8724 3237 Acked Write 5 Sent Done Done false
8725 3238 Acked Read 9 Sent Done Done false
8726 3239 Acked Write 10 Sent Done Done false
8727 3240 Acked Write 6 Sent Done Done false
8728 3241 Acked Read 3 Sent Done Done false
8729 3242 Acked Write 9 Sent Done Done false
8730 3243 Acked Read 9 Sent Done Done false
8731 3244 Acked Read 10 Sent Done Done false
8732 3245 Acked Read 8 Sent Done Done false
8733 3246 Acked Read 10 Sent Done Done false
8734 3247 Acked Write 3 Sent Done Done false
8735 3248 Acked Read 3 Sent Done Done false
8736 3249 Acked Read 1 Sent Done Done false
8737 3250 Acked Read 5 Sent Done Done false
8738 3251 Acked Read 6 Sent Done Done false
8739 3252 Acked Write 1 Sent Done Done false
8740 3253 Acked Read 8 Sent Done Done false
8741 3254 Acked Read 1 Sent Done Done false
8742 3255 Acked Flush 0 Sent Done Done false
8743 3256 Acked Read 10 Sent Done Done false
8744 3257 Acked Write 9 Sent Done Done false
8745 3258 Acked Flush 0 Sent Done Done false
8746 3259 Acked Write 9 Sent Done Done false
8747 3260 Acked Read 5 Sent Done Done false
8748 3261 Acked Read 1 Sent Done Done false
8749 3262 Acked Read 6 Sent Done Done false
8750 3263 Acked Read 3 Sent Done Done false
8751 3264 Acked Write 9 Sent Done Done false
8752 3265 Acked Write 1 Sent Done Done false
8753 3266 Acked Write 2 Sent Done Done false
8754 3267 Acked Write 3 Sent Done Done false
8755 3268 Acked Read 9 Sent Done Done false
8756 3269 Acked Write 6 Sent Done Done false
8757 3270 Acked Write 5 Sent Done Done false
8758 3271 Acked Read 4 Sent Done Done false
8759 3272 Acked Write 1 Sent Done Done false
8760 3273 Acked Write 7 Sent Done Done false
8761 3274 Acked Flush 0 Sent Done Done false
8762 3275 Acked Read 2 Sent Done Done false
8763 3276 Acked Read 9 Sent Done Done false
8764 3277 Acked Read 5 Sent Done Done false
8765 3278 Acked Read 7 Sent Done Done false
8766 3279 Acked Read 7 Sent Done Done false
8767 3280 Acked Read 1 Sent Done Done false
8768 3281 Acked Read 10 Sent Done Done false
8769 3282 Acked Read 7 Sent Done Done false
8770 3283 Acked Read 8 Sent Done Done false
8771 3284 Acked Flush 0 Sent Done Done false
8772 3285 Acked Read 3 Sent Done Done false
8773 3286 Acked Read 10 Sent Done Done false
8774 3287 Acked Write 3 Sent Done Done false
8775 3288 Acked Write 10 Sent Done Done false
8776 3289 Acked Read 1 Sent Done Done false
8777 3290 Acked Write 1 Sent Done Done false
8778 3291 Acked Read 9 Sent Done Done false
8779 3292 Acked Read 2 Sent Done Done false
8780 3293 Acked Read 7 Sent Done Done false
8781 3294 Acked Flush 0 Sent Done Done false
8782 3295 Acked Read 10 Sent Done Done false
8783 3296 Acked Read 6 Sent Done Done false
8784 3297 Acked Write 7 Sent Done Done false
8785 3298 Acked Write 5 Sent Done Done false
8786 3299 Acked Read 6 Sent Done Done false
8787 3300 Acked Read 8 Sent Done Done false
8788 3301 Acked Write 9 Sent Done Done false
8789 3302 Acked Read 10 Sent Done Done false
8790 3303 Acked Read 4 Sent Done Done false
8791 3304 Acked Write 5 Sent Done Done false
8792 3305 Acked Read 3 Sent Done Done false
8793 3306 Acked Write 8 Sent Done Done false
8794 3307 Acked Write 2 Sent Done Done false
8795 3308 Acked Read 6 Sent Done Done false
8796 3309 Acked Read 3 Sent Done Done false
8797 3310 Acked Flush 0 Sent Done Done false
8798 3311 Acked Read 6 Sent Done Done false
8799 3312 Acked Read 2 Sent Done Done false
8800 3313 Acked Write 5 Sent Done Done false
8801 3314 Acked Read 10 Sent Done Done false
8802 3315 Acked Flush 0 Sent Done Done false
8803 3316 Acked Write 5 Sent Done Done false
8804 3317 Acked Read 6 Sent Done Done false
8805 3318 Acked Write 10 Sent Done Done false
8806 3319 Acked Read 9 Sent Done Done false
8807 3320 Acked Read 9 Sent Done Done false
8808 3321 Acked Write 2 Sent Done Done false
8809 3322 Acked Write 5 Sent Done Done false
8810 3323 Acked Read 5 Sent Done Done false
8811 3324 Acked Read 4 Sent Done Done false
8812 3325 Acked Write 2 Sent Done Done false
8813 3326 Acked Write 4 Sent Done Done false
8814 3327 Acked Write 5 Sent Done Done false
8815 3328 Acked Read 2 Sent Done Done false
8816 3329 Acked Write 10 Sent Done Done false
8817 3330 Acked Flush 0 Sent Done Done false
8818 3331 Acked Write 7 Sent Done Done false
8819 3332 Acked Read 3 Sent Done Done false
8820 3333 Acked Read 9 Sent Done Done false
8821 3334 Acked Read 7 Sent Done Done false
8822 3335 Acked Read 10 Sent Done Done false
8823 3336 Acked Write 6 Sent Done Done false
8824 3337 Acked Write 5 Sent Done Done false
8825 3338 Acked Write 1 Sent Done Done false
8826 3339 Acked Read 8 Sent Done Done false
8827 3340 Acked Write 7 Sent Done Done false
8828 3341 Acked Write 3 Sent Done Done false
8829 3342 Acked Write 4 Sent Done Done false
8830 3343 Acked Read 10 Sent Done Done false
8831 3344 Acked Write 8 Sent Done Done false
8832 3345 Acked Read 9 Sent Done Done false
8833 3346 Acked Read 10 Sent Done Done false
8834 3347 Acked Write 4 Sent Done Done false
8835 3348 Acked Write 1 Sent Done Done false
8836 3349 Acked Read 4 Sent Done Done false
8837 3350 Acked Read 1 Sent Done Done false
8838 3351 Acked Read 1 Sent Done Done false
8839 3352 Acked Write 2 Sent Done Done false
8840 3353 Acked Read 2 Sent Done Done false
8841 3354 Acked Read 4 Sent Done Done false
8842 3355 Acked Write 5 Sent Done Done false
8843 3356 Acked Read 10 Sent Done Done false
8844 3357 Acked Read 6 Sent Done Done false
8845 3358 Acked Read 5 Sent Done Done false
8846 3359 Acked Read 9 Sent Done Done false
8847 3360 Acked Write 2 Sent Done Done false
8848 3361 Acked Read 3 Sent Done Done false
8849 3362 Acked Read 10 Sent Done Done false
8850 3363 Acked Flush 0 Sent Done Done false
8851 3364 Acked Read 8 Sent Done Done false
8852 3365 Acked Write 3 Sent Done Done false
8853 3366 Acked Flush 0 Sent Done Done false
8854 3367 Acked Flush 0 Sent Done Done false
8855 3368 Acked Read 5 Sent Done Done false
8856 3369 Acked Read 1 Sent Done Done false
8857 3370 Acked Read 8 Sent Done Done false
8858 3371 Acked Write 4 Sent Done Done false
8859 3372 Acked Write 2 Sent Done Done false
8860 3373 Acked Read 5 Sent Done Done false
8861 3374 Acked Read 7 Sent Done Done false
8862 3375 Acked Write 2 Sent Done Done false
8863 3376 Acked Write 5 Sent Done Done false
8864 3377 Acked Read 4 Sent Done Done false
8865 3378 Acked Read 4 Sent Done Done false
8866 3379 Acked Write 3 Sent Done Done false
8867 3380 Acked Read 6 Sent Done Done false
8868 3381 Acked Read 3 Sent Done Done false
8869 3382 Acked Flush 0 Sent Done Done false
8870 3383 Acked Read 4 Sent Done Done false
8871 3384 Acked Write 4 Sent Done Done false
8872 3385 Acked Read 6 Sent Done Done false
8873 3386 Acked Read 5 Sent Done Done false
8874 3387 Acked Read 9 Sent Done Done false
8875 3388 Acked Read 5 Sent Done Done false
8876 3389 Acked Read 2 Sent Done Done false
8877 3390 Acked Write 2 Sent Done Done false
8878 3391 Acked Read 9 Sent Done Done false
8879 3392 Acked Read 5 Sent Done Done false
8880 3393 Acked Read 5 Sent Done Done false
8881 3394 Acked Write 6 Sent Done Done false
8882 3395 Acked Flush 0 Sent Done Done false
8883 3396 Acked Read 2 Sent Done Done false
8884 3397 Acked Write 10 Sent Done Done false
8885 3398 Acked Read 8 Sent Done Done false
8886 3399 Acked Write 1 Sent Done Done false
8887 3400 Acked Read 7 Sent Done Done false
8888 3401 Acked Read 9 Sent Done Done false
8889 3402 Acked Read 5 Sent Done Done false
8890 3403 Acked Write 4 Sent Done Done false
8891 3404 Acked Read 9 Sent Done Done false
8892 3405 Acked Read 5 Sent Done Done false
8893 3406 Acked Write 1 Sent Done Done false
8894 3407 Acked Flush 0 Sent Done Done false
8895 3408 Acked Read 8 Sent Done Done false
8896 3409 Acked Read 5 Sent Done Done false
8897 3410 Acked Write 1 Sent Done Done false
8898 3411 Acked Write 7 Sent Done Done false
8899 3412 Acked Read 8 Sent Done Done false
8900 3413 Acked Write 10 Sent Done Done false
8901 3414 Acked Read 1 Sent Done Done false
8902 3415 Acked Flush 0 Sent Done Done false
8903 STATES DS:0 DS:1 DS:2 TOTAL
8904 Sent 302 0 0 302
8905 Done 0 302 302 604
8906 Skipped 0 0 0 0
8907 Error 0 0 0 0
8908 Last Flush: Some(JobId(3113)) Some(JobId(3415)) Some(JobId(3415))
8909 Downstairs last five completed: 3113 3112 3111 3110 3109
8910 Upstairs last five completed: 3415 3414 3413 3412 3411
8911 CLIENT: Up:0 ds:302 act:2
8912 ----------------------------------------------------------------
8913 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
8914 States: Offline Active Active
8915 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8916 3114 Acked Read 8 Sent Done Done false
8917 3115 Acked Read 4 Sent Done Done false
8918 3116 Acked Write 8 Sent Done Done false
8919 3117 Acked Read 9 Sent Done Done false
8920 3118 Acked Write 7 Sent Done Done false
8921 3119 Acked Write 9 Sent Done Done false
8922 3120 Acked Write 8 Sent Done Done false
8923 3121 Acked Read 1 Sent Done Done false
8924 3122 Acked Flush 0 Sent Done Done false
8925 3123 Acked Read 7 Sent Done Done false
8926 3124 Acked Write 4 Sent Done Done false
8927 3125 Acked Write 4 Sent Done Done false
8928 3126 Acked Read 8 Sent Done Done false
8929 3127 Acked Write 7 Sent Done Done false
8930 3128 Acked Read 5 Sent Done Done false
8931 3129 Acked Read 1 Sent Done Done false
8932 3130 Acked Write 4 Sent Done Done false
8933 3131 Acked Read 9 Sent Done Done false
8934 3132 Acked Read 3 Sent Done Done false
8935 3133 Acked Write 4 Sent Done Done false
8936 3134 Acked Read 10 Sent Done Done false
8937 3135 Acked Read 9 Sent Done Done false
8938 3136 Acked Write 4 Sent Done Done false
8939 3137 Acked Write 6 Sent Done Done false
8940 3138 Acked Write 1 Sent Done Done false
8941 3139 Acked Write 10 Sent Done Done false
8942 3140 Acked Write 7 Sent Done Done false
8943 3141 Acked Read 8 Sent Done Done false
8944 3142 Acked Write 10 Sent Done Done false
8945 3143 Acked Write 9 Sent Done Done false
8946 3144 Acked Write 5 Sent Done Done false
8947 3145 Acked Read 8 Sent Done Done false
8948 3146 Acked Read 10 Sent Done Done false
8949 3147 Acked Read 3 Sent Done Done false
8950 3148 Acked Read 6 Sent Done Done false
8951 3149 Acked Read 4 Sent Done Done false
8952 3150 Acked Read 7 Sent Done Done false
8953 3151 Acked Read 5 Sent Done Done false
8954 3152 Acked Write 2 Sent Done Done false
8955 3153 Acked Read 9 Sent Done Done false
8956 3154 Acked Write 8 Sent Done Done false
8957 3155 Acked Read 6 Sent Done Done false
8958 3156 Acked Read 4 Sent Done Done false
8959 3157 Acked Read 10 Sent Done Done false
8960 3158 Acked Write 8 Sent Done Done false
8961 3159 Acked Write 4 Sent Done Done false
8962 3160 Acked Read 7 Sent Done Done false
8963 3161 Acked Read 5 Sent Done Done false
8964 3162 Acked Read 2 Sent Done Done false
8965 3163 Acked Flush 0 Sent Done Done false
8966 3164 Acked Read 4 Sent Done Done false
8967 3165 Acked Read 7 Sent Done Done false
8968 3166 Acked Read 7 Sent Done Done false
8969 3167 Acked Read 5 Sent Done Done false
8970 3168 Acked Write 1 Sent Done Done false
8971 3169 Acked Read 4 Sent Done Done false
8972 3170 Acked Write 8 Sent Done Done false
8973 3171 Acked Write 7 Sent Done Done false
8974 3172 Acked Read 3 Sent Done Done false
8975 3173 Acked Read 3 Sent Done Done false
8976 3174 Acked Read 8 Sent Done Done false
8977 3175 Acked Write 7 Sent Done Done false
8978 3176 Acked Read 6 Sent Done Done false
8979 3177 Acked Read 7 Sent Done Done false
8980 3178 Acked Read 10 Sent Done Done false
8981 3179 Acked Read 7 Sent Done Done false
8982 3180 Acked Write 1 Sent Done Done false
8983 3181 Acked Read 10 Sent Done Done false
8984 3182 Acked Read 6 Sent Done Done false
8985 3183 Acked Read 10 Sent Done Done false
8986 3184 Acked Read 2 Sent Done Done false
8987 3185 Acked Write 4 Sent Done Done false
8988 3186 Acked Flush 0 Sent Done Done false
8989 3187 Acked Read 10 Sent Done Done false
8990 3188 Acked Read 7 Sent Done Done false
8991 3189 Acked Read 10 Sent Done Done false
8992 3190 Acked Read 2 Sent Done Done false
8993 3191 Acked Write 8 Sent Done Done false
8994 3192 Acked Write 3 Sent Done Done false
8995 3193 Acked Read 3 Sent Done Done false
8996 3194 Acked Write 8 Sent Done Done false
8997 3195 Acked Write 10 Sent Done Done false
8998 3196 Acked Read 1 Sent Done Done false
8999 3197 Acked Write 4 Sent Done Done false
9000 3198 Acked Read 7 Sent Done Done false
9001 3199 Acked Flush 0 Sent Done Done false
9002 3200 Acked Flush 0 Sent Done Done false
9003 3201 Acked Write 9 Sent Done Done false
9004 3202 Acked Read 8 Sent Done Done false
9005 3203 Acked Read 6 Sent Done Done false
9006 3204 Acked Write 5 Sent Done Done false
9007 3205 Acked Read 6 Sent Done Done false
9008 3206 Acked Read 1 Sent Done Done false
9009 3207 Acked Read 8 Sent Done Done false
9010 3208 Acked Read 2 Sent Done Done false
9011 3209 Acked Write 10 Sent Done Done false
9012 3210 Acked Read 2 Sent Done Done false
9013 3211 Acked Read 10 Sent Done Done false
9014 3212 Acked Read 10 Sent Done Done false
9015 3213 Acked Write 4 Sent Done Done false
9016 3214 Acked Write 2 Sent Done Done false
9017 3215 Acked Read 10 Sent Done Done false
9018 3216 Acked Write 7 Sent Done Done false
9019 3217 Acked Read 4 Sent Done Done false
9020 3218 Acked Write 7 Sent Done Done false
9021 3219 Acked Write 7 Sent Done Done false
9022 3220 Acked Read 3 Sent Done Done false
9023 3221 Acked Read 4 Sent Done Done false
9024 3222 Acked Read 5 Sent Done Done false
9025 3223 Acked Read 4 Sent Done Done false
9026 3224 Acked Write 3 Sent Done Done false
9027 3225 Acked Write 2 Sent Done Done false
9028 3226 Acked Write 4 Sent Done Done false
9029 3227 Acked Write 5 Sent Done Done false
9030 3228 Acked Read 6 Sent Done Done false
9031 3229 Acked Flush 0 Sent Done Done false
9032 3230 Acked Read 6 Sent Done Done false
9033 3231 Acked Read 2 Sent Done Done false
9034 3232 Acked Read 6 Sent Done Done false
9035 3233 Acked Write 2 Sent Done Done false
9036 3234 Acked Read 6 Sent Done Done false
9037 3235 Acked Read 1 Sent Done Done false
9038 3236 Acked Read 9 Sent Done Done false
9039 3237 Acked Write 5 Sent Done Done false
9040 3238 Acked Read 9 Sent Done Done false
9041 3239 Acked Write 10 Sent Done Done false
9042 3240 Acked Write 6 Sent Done Done false
9043 3241 Acked Read 3 Sent Done Done false
9044 3242 Acked Write 9 Sent Done Done false
9045 3243 Acked Read 9 Sent Done Done false
9046 3244 Acked Read 10 Sent Done Done false
9047 3245 Acked Read 8 Sent Done Done false
9048 3246 Acked Read 10 Sent Done Done false
9049 3247 Acked Write 3 Sent Done Done false
9050 3248 Acked Read 3 Sent Done Done false
9051 3249 Acked Read 1 Sent Done Done false
9052 3250 Acked Read 5 Sent Done Done false
9053 3251 Acked Read 6 Sent Done Done false
9054 3252 Acked Write 1 Sent Done Done false
9055 3253 Acked Read 8 Sent Done Done false
9056 3254 Acked Read 1 Sent Done Done false
9057 3255 Acked Flush 0 Sent Done Done false
9058 3256 Acked Read 10 Sent Done Done false
9059 3257 Acked Write 9 Sent Done Done false
9060 3258 Acked Flush 0 Sent Done Done false
9061 3259 Acked Write 9 Sent Done Done false
9062 3260 Acked Read 5 Sent Done Done false
9063 3261 Acked Read 1 Sent Done Done false
9064 3262 Acked Read 6 Sent Done Done false
9065 3263 Acked Read 3 Sent Done Done false
9066 3264 Acked Write 9 Sent Done Done false
9067 3265 Acked Write 1 Sent Done Done false
9068 3266 Acked Write 2 Sent Done Done false
9069 3267 Acked Write 3 Sent Done Done false
9070 3268 Acked Read 9 Sent Done Done false
9071 3269 Acked Write 6 Sent Done Done false
9072 3270 Acked Write 5 Sent Done Done false
9073 3271 Acked Read 4 Sent Done Done false
9074 3272 Acked Write 1 Sent Done Done false
9075 3273 Acked Write 7 Sent Done Done false
9076 3274 Acked Flush 0 Sent Done Done false
9077 3275 Acked Read 2 Sent Done Done false
9078 3276 Acked Read 9 Sent Done Done false
9079 3277 Acked Read 5 Sent Done Done false
9080 3278 Acked Read 7 Sent Done Done false
9081 3279 Acked Read 7 Sent Done Done false
9082 3280 Acked Read 1 Sent Done Done false
9083 3281 Acked Read 10 Sent Done Done false
9084 3282 Acked Read 7 Sent Done Done false
9085 3283 Acked Read 8 Sent Done Done false
9086 3284 Acked Flush 0 Sent Done Done false
9087 3285 Acked Read 3 Sent Done Done false
9088 3286 Acked Read 10 Sent Done Done false
9089 3287 Acked Write 3 Sent Done Done false
9090 3288 Acked Write 10 Sent Done Done false
9091 3289 Acked Read 1 Sent Done Done false
9092 3290 Acked Write 1 Sent Done Done false
9093 3291 Acked Read 9 Sent Done Done false
9094 3292 Acked Read 2 Sent Done Done false
9095 3293 Acked Read 7 Sent Done Done false
9096 3294 Acked Flush 0 Sent Done Done false
9097 3295 Acked Read 10 Sent Done Done false
9098 3296 Acked Read 6 Sent Done Done false
9099 3297 Acked Write 7 Sent Done Done false
9100 3298 Acked Write 5 Sent Done Done false
9101 3299 Acked Read 6 Sent Done Done false
9102 3300 Acked Read 8 Sent Done Done false
9103 3301 Acked Write 9 Sent Done Done false
9104 3302 Acked Read 10 Sent Done Done false
9105 3303 Acked Read 4 Sent Done Done false
9106 3304 Acked Write 5 Sent Done Done false
9107 3305 Acked Read 3 Sent Done Done false
9108 3306 Acked Write 8 Sent Done Done false
9109 3307 Acked Write 2 Sent Done Done false
9110 3308 Acked Read 6 Sent Done Done false
9111 3309 Acked Read 3 Sent Done Done false
9112 3310 Acked Flush 0 Sent Done Done false
9113 3311 Acked Read 6 Sent Done Done false
9114 3312 Acked Read 2 Sent Done Done false
9115 3313 Acked Write 5 Sent Done Done false
9116 3314 Acked Read 10 Sent Done Done false
9117 3315 Acked Flush 0 Sent Done Done false
9118 3316 Acked Write 5 Sent Done Done false
9119 3317 Acked Read 6 Sent Done Done false
9120 3318 Acked Write 10 Sent Done Done false
9121 3319 Acked Read 9 Sent Done Done false
9122 3320 Acked Read 9 Sent Done Done false
9123 3321 Acked Write 2 Sent Done Done false
9124 3322 Acked Write 5 Sent Done Done false
9125 3323 Acked Read 5 Sent Done Done false
9126 3324 Acked Read 4 Sent Done Done false
9127 3325 Acked Write 2 Sent Done Done false
9128 3326 Acked Write 4 Sent Done Done false
9129 3327 Acked Write 5 Sent Done Done false
9130 3328 Acked Read 2 Sent Done Done false
9131 3329 Acked Write 10 Sent Done Done false
9132 3330 Acked Flush 0 Sent Done Done false
9133 3331 Acked Write 7 Sent Done Done false
9134 3332 Acked Read 3 Sent Done Done false
9135 3333 Acked Read 9 Sent Done Done false
9136 3334 Acked Read 7 Sent Done Done false
9137 3335 Acked Read 10 Sent Done Done false
9138 3336 Acked Write 6 Sent Done Done false
9139 3337 Acked Write 5 Sent Done Done false
9140 3338 Acked Write 1 Sent Done Done false
9141 3339 Acked Read 8 Sent Done Done false
9142 3340 Acked Write 7 Sent Done Done false
9143 3341 Acked Write 3 Sent Done Done false
9144 3342 Acked Write 4 Sent Done Done false
9145 3343 Acked Read 10 Sent Done Done false
9146 3344 Acked Write 8 Sent Done Done false
9147 3345 Acked Read 9 Sent Done Done false
9148 3346 Acked Read 10 Sent Done Done false
9149 3347 Acked Write 4 Sent Done Done false
9150 3348 Acked Write 1 Sent Done Done false
9151 3349 Acked Read 4 Sent Done Done false
9152 3350 Acked Read 1 Sent Done Done false
9153 3351 Acked Read 1 Sent Done Done false
9154 3352 Acked Write 2 Sent Done Done false
9155 3353 Acked Read 2 Sent Done Done false
9156 3354 Acked Read 4 Sent Done Done false
9157 3355 Acked Write 5 Sent Done Done false
9158 3356 Acked Read 10 Sent Done Done false
9159 3357 Acked Read 6 Sent Done Done false
9160 3358 Acked Read 5 Sent Done Done false
9161 3359 Acked Read 9 Sent Done Done false
9162 3360 Acked Write 2 Sent Done Done false
9163 3361 Acked Read 3 Sent Done Done false
9164 3362 Acked Read 10 Sent Done Done false
9165 3363 Acked Flush 0 Sent Done Done false
9166 3364 Acked Read 8 Sent Done Done false
9167 3365 Acked Write 3 Sent Done Done false
9168 3366 Acked Flush 0 Sent Done Done false
9169 3367 Acked Flush 0 Sent Done Done false
9170 3368 Acked Read 5 Sent Done Done false
9171 3369 Acked Read 1 Sent Done Done false
9172 3370 Acked Read 8 Sent Done Done false
9173 3371 Acked Write 4 Sent Done Done false
9174 3372 Acked Write 2 Sent Done Done false
9175 3373 Acked Read 5 Sent Done Done false
9176 3374 Acked Read 7 Sent Done Done false
9177 3375 Acked Write 2 Sent Done Done false
9178 3376 Acked Write 5 Sent Done Done false
9179 3377 Acked Read 4 Sent Done Done false
9180 3378 Acked Read 4 Sent Done Done false
9181 3379 Acked Write 3 Sent Done Done false
9182 3380 Acked Read 6 Sent Done Done false
9183 3381 Acked Read 3 Sent Done Done false
9184 3382 Acked Flush 0 Sent Done Done false
9185 3383 Acked Read 4 Sent Done Done false
9186 3384 Acked Write 4 Sent Done Done false
9187 3385 Acked Read 6 Sent Done Done false
9188 3386 Acked Read 5 Sent Done Done false
9189 3387 Acked Read 9 Sent Done Done false
9190 3388 Acked Read 5 Sent Done Done false
9191 3389 Acked Read 2 Sent Done Done false
9192 3390 Acked Write 2 Sent Done Done false
9193 3391 Acked Read 9 Sent Done Done false
9194 3392 Acked Read 5 Sent Done Done false
9195 3393 Acked Read 5 Sent Done Done false
9196 3394 Acked Write 6 Sent Done Done false
9197 3395 Acked Flush 0 Sent Done Done false
9198 3396 Acked Read 2 Sent Done Done false
9199 3397 Acked Write 10 Sent Done Done false
9200 3398 Acked Read 8 Sent Done Done false
9201 3399 Acked Write 1 Sent Done Done false
9202 3400 Acked Read 7 Sent Done Done false
9203 3401 Acked Read 9 Sent Done Done false
9204 3402 Acked Read 5 Sent Done Done false
9205 3403 Acked Write 4 Sent Done Done false
9206 3404 Acked Read 9 Sent Done Done false
9207 3405 Acked Read 5 Sent Done Done false
9208 3406 Acked Write 1 Sent Done Done false
9209 3407 Acked Flush 0 Sent Done Done false
9210 3408 Acked Read 8 Sent Done Done false
9211 3409 Acked Read 5 Sent Done Done false
9212 3410 Acked Write 1 Sent Done Done false
9213 3411 Acked Write 7 Sent Done Done false
9214 3412 Acked Read 8 Sent Done Done false
9215 3413 Acked Write 10 Sent Done Done false
9216 3414 Acked Read 1 Sent Done Done false
9217 3415 Acked Flush 0 Sent Done Done false
9218 STATES DS:0 DS:1 DS:2 TOTAL
9219 Sent 302 0 0 302
9220 Done 0 302 302 604
9221 Skipped 0 0 0 0
9222 Error 0 0 0 0
9223 Last Flush: Some(JobId(3113)) Some(JobId(3415)) Some(JobId(3415))
9224 Downstairs last five completed: 3113 3112 3111 3110 3109
9225 Upstairs last five completed: 3415 3414 3413 3412 3411
9226 CLIENT: Up:0 ds:302 act:2
92272025-01-23T17:39:41.375ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
92282025-01-23T17:39:41.375ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52417) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
92292025-01-23T17:39:41.376ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
92302025-01-23T17:39:41.377ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
92312025-01-23T17:39:41.377ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
92322025-01-23T17:39:41.377ZINFOcrucible: send last flush ID to this DS: Some(JobId(3113)) = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
92332025-01-23T17:39:41.377ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
92342025-01-23T17:39:41.377ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(3113)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
92352025-01-23T17:39:41.377ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(3113)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
9236 ----------------------------------------------------------------
9237 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9238 Downstairs last five completed: 3415 3414 3413 3412 3411
9239 Upstairs last five completed: 3415 3414 3413 3412 3411
9240 CLIENT: Up:0 ds:0 act:3
9241 Replay: All jobs finished
9242 Main task has work to do, go find it
9243 got dsc Stop(2)
9244 stop 2
9245 Jan 23 17:39:41.643 INFO[8830] Got stop action so:false kr:false
9246 request completed, latency_us: 378, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 7dac0c77-f37a-4528-9ac3-944d0b53d463, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
9247 Jan 23 17:39:41.644 INFO{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:39:41.644529719Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
9248 request completed, latency_us: 133, response_code: 200, uri: /state/cid/2, method: GET, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:39:41.644581404Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
9249 req_id{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:39:41.644607731Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
9250 : 8effde20-70c9-4a0d-86fa-6e3f8eadd4ce, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
92512025-01-23T17:39:41.644ZINFOcrucible: client task is exiting = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
9252 [8830] Exited with: ExitStatus(unix_wait_status(9))
9253 [8830][2] reports Exit
9254 Jan 23 17:39:45.646 INFO request completed, latency_us: 199, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2336ba6d-d8d4-4d64-b23b-9c6435353f5c, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
9255 001/300 Read block 3642 len 4096
9256 002/300 Write block 1273 len 28672 data: 2 2 3 3 3 3 3
9257 003/300 Write block 2093 len 40960 data: 2 2 2 2 2 2 2 2 2 2
9258 004/300 Read block 545 len 16384
9259 005/300 Write block 2541 len 32768 data: 3 2 2 2 2 2 3 3
9260 006/300 Flush
9261 007/300 Read block 2212 len 32768
9262 008/300 Read block 3566 len 4096
9263 009/300 Write block 3914 len 4096 data: 4
9264 010/300 Read block 1497 len 32768
9265 011/300 Write block 4525 len 32768 data: 2 2 2 2 2 2 2 2
9266 012/300 Read block 1656 len 28672
9267 013/300 Read block 2872 len 32768
9268 014/300 Read block 685 len 32768
9269 015/300 Read block 1693 len 12288
9270 016/300 Write block 3501 len 24576 data: 3 3 4 4 4 4
9271 017/300 Read block 35 len 4096
9272 018/300 Write block 2202 len 12288 data: 2 3 3
9273 019/300 Read block 176 len 20480
9274 020/300 Read block 4096 len 20480
9275 021/300 Write block 3532 len 4096 data: 4
9276 022/300 Write block 3884 len 36864 data: 3 3 2 2 2 2 2 2 2
9277 023/300 Read block 1414 len 36864
9278 024/300 Read block 4058 len 40960
9279 025/300 Read block 519 len 40960
9280 026/300 Read block 2954 len 28672
9281 027/300 Write block 2477 len 36864 data: 4 4 5 3 3 3 3 3 3
9282 028/300 Flush
9283 029/300 Read block 4464 len 32768
9284 030/300 Write block 672 len 20480 data: 3 2 2 2 2
9285 031/300 Flush
9286 032/300 Read block 3032 len 8192
9287 033/300 Write block 4040 len 4096 data: 2
9288 034/300 Write block 3009 len 40960 data: 3 3 3 4 4 4 3 3 3 3
9289 035/300 Write block 754 len 28672 data: 2 2 2 2 2 3 4
9290 036/300 Write block 2397 len 8192 data: 3 4
9291 037/300 Read block 1013 len 20480
9292 038/300 Read block 435 len 28672
9293 039/300 Read block 1771 len 20480
9294 040/300 Write block 2020 len 4096 data: 2
9295 041/300 Flush
9296 042/300 Read block 4975 len 32768
9297 043/300 Write block 100 len 12288 data: 3 3 4
9298 044/300 Read block 2624 len 24576
9299 045/300 Write block 3459 len 40960 data: 4 4 4 4 3 2 2 2 2 3
9300 046/300 Read block 1344 len 36864
9301 047/300 Read block 2760 len 24576
9302 048/300 Read block 177 len 32768
9303 049/300 Read block 4639 len 8192
9304 050/300 Write block 2576 len 4096 data: 4
9305 051/300 Read block 625 len 28672
9306 052/300 Read block 4249 len 12288
9307 053/300 Read block 4350 len 40960
9308 054/300 Write block 430 len 40960 data: 2 3 2 2 2 2 2 2 2 2
9309 055/300 Read block 2022 len 8192
9310 056/300 Read block 4696 len 36864
9311 057/300 Read block 152 len 40960
9312 058/300 Flush
9313 059/300 Write block 201 len 20480 data: 4 3 5 5 5
9314 060/300 Read block 1948 len 8192
9315 061/300 Read block 2877 len 16384
9316 062/300 Read block 4567 len 24576
9317 063/300 Read block 2587 len 40960
9318 064/300 Write block 4332 len 8192 data: 3 2
9319 065/300 Flush
9320 066/300 Read block 928 len 40960
9321 067/300 Read block 626 len 24576
9322 068/300 Write block 2317 len 20480 data: 3 3 3 3 2
9323 069/300 Read block 2161 len 28672
9324 070/300 Write block 2964 len 8192 data: 4 4
9325 071/300 Write block 1917 len 8192 data: 2 3
9326 072/300 Read block 2089 len 20480
9327 073/300 Write block 1500 len 32768 data: 2 2 2 2 3 3 4 4
9328 074/300 Flush
9329 075/300 Read block 1154 len 28672
9330 076/300 Read block 3362 len 36864
9331 077/300 Read block 1155 len 24576
9332 078/300 Flush
9333 079/300 Read block 3800 len 12288
9334 080/300 Read block 2356 len 8192
9335 081/300 Read block 2931 len 4096
9336 082/300 Write block 3290 len 24576 data: 2 2 4 3 3 3
9337 083/300 Read block 1253 len 28672
9338 084/300 Read block 4451 len 24576
9339 085/300 Read block 1297 len 4096
9340 086/300 Read block 2695 len 28672
9341 087/300 Write block 2113 len 24576 data: 3 3 3 3 3 3
9342 088/300 Read block 1982 len 36864
9343 089/300 Flush
9344 090/300 Write block 187 len 40960 data: 4 4 3 3 3 3 3 3 3 3
9345 091/300 Read block 4344 len 20480
9346 092/300 Write block 2669 len 24576 data: 2 3 2 2 2 2
9347 093/300 Read block 4855 len 28672
9348 094/300 Read block 1032 len 32768
9349 095/300 Read block 3426 len 28672
9350 096/300 Read block 4847 len 28672
9351 097/300 Read block 3899 len 28672
9352 098/300 Write block 4920 len 40960 data: 3 3 3 4 4 4 4 3 3 2
9353 099/300 Read block 2600 len 20480
9354 100/300 Write block 4601 len 12288 data: 4 4 4
9355 101/300 Write block 4690 len 28672 data: 3 3 3 4 3 3 3
9356 102/300 Write block 1353 len 12288 data: 3 3 2
9357 103/300 Read block 3312 len 36864
9358 104/300 Write block 2617 len 4096 data: 5
9359 105/300 Read block 4551 len 20480
9360 106/300 Write block 3401 len 36864 data: 3 3 3 2 3 3 3 2 3
9361 107/300 Write block 2359 len 24576 data: 3 3 3 3 3 3
9362 108/300 Read block 4134 len 32768
9363 109/300 Write block 3748 len 32768 data: 3 3 3 3 3 3 3 3
9364 110/300 Read block 3859 len 4096
9365 111/300 Flush
9366 112/300 Read block 1715 len 12288
9367 113/300 Read block 354 len 16384
9368 114/300 Read block 997 len 12288
9369 115/300 Write block 3327 len 24576 data: 2 2 2 3 3 3
9370 116/300 Flush
9371 117/300 Read block 991 len 4096
9372 118/300 Read block 4254 len 32768
9373 119/300 Read block 804 len 32768
9374 120/300 Read block 2280 len 40960
9375 121/300 Read block 1550 len 8192
9376 122/300 Write block 1549 len 36864 data: 3 4 3 3 3 4 3 2 2
9377 123/300 Write block 440 len 8192 data: 2 3
9378 124/300 Write block 4509 len 28672 data: 2 2 2 2 2 2 3
9379 125/300 Read block 2685 len 32768
9380 126/300 Write block 3642 len 16384 data: 3 3 3 3
9381 127/300 Write block 4692 len 4096 data: 4
9382 128/300 Write block 1157 len 36864 data: 4 4 4 4 5 6 6 6 4
9383 129/300 Read block 3543 len 28672
9384 130/300 Read block 3745 len 36864
9385 131/300 Read block 2686 len 32768
9386 132/300 Write block 524 len 28672 data: 4 5 5 2 2 3 3
9387 133/300 Write block 1988 len 28672 data: 2 2 2 2 2 3 3
9388 134/300 Read block 434 len 20480
9389 135/300 Read block 2028 len 16384
9390 136/300 Write block 2248 len 40960 data: 2 2 2 3 4 4 5 5 5 5
9391 137/300 Read block 3256 len 28672
9392 138/300 Write block 3599 len 32768 data: 3 3 3 3 3 4 3 3
9393 139/300 Write block 527 len 20480 data: 3 3 4 4 3
9394 140/300 Read block 1700 len 20480
9395 141/300 Flush
9396 142/300 Read block 3176 len 40960
9397 143/300 Write block 712 len 4096 data: 2
9398 144/300 Flush
9399 145/300 Write block 3063 len 4096 data: 2
9400 146/300 Write block 639 len 16384 data: 3 5 4 3
9401 147/300 Read block 2002 len 8192
9402 148/300 Write block 4493 len 28672 data: 2 2 2 3 3 3 3
9403 149/300 Read block 4538 len 20480
9404 150/300 Write block 3691 len 40960 data: 2 3 3 3 3 3 2 2 2 3
9405 151/300 Read block 2604 len 40960
9406 152/300 Read block 2503 len 28672
9407 153/300 Write block 4589 len 24576 data: 2 2 2 2 2 2
9408 154/300 Read block 3704 len 40960
9409 155/300 Read block 324 len 32768
9410 156/300 Write block 1947 len 28672 data: 2 2 2 2 2 2 2
9411 157/300 Read block 4867 len 4096
9412 158/300 Read block 1053 len 40960
9413 159/300 Read block 4980 len 36864
9414 160/300 Write block 1896 len 40960 data: 3 3 3 3 5 5 5 4 6 7
9415 161/300 Read block 4275 len 24576
9416 162/300 Read block 1422 len 12288
9417 163/300 Read block 188 len 40960
9418 164/300 Write block 342 len 24576 data: 3 3 3 2 2 2
9419 165/300 Write block 3044 len 16384 data: 2 2 2 2
9420 166/300 Write block 1294 len 16384 data: 3 3 3 4
9421 167/300 Read block 650 len 40960
9422 168/300 Write block 1794 len 32768 data: 2 2 3 3 3 4 5 5
9423 169/300 Read block 3077 len 4096
9424 170/300 Read block 4049 len 40960
9425 171/300 Flush
9426 172/300 Write block 4613 len 32768 data: 4 4 3 4 4 6 6 5
9427 173/300 Write block 668 len 24576 data: 3 3 3 3 4 3
9428 174/300 Write block 3982 len 20480 data: 3 3 3 3 4
9429 175/300 Write block 995 len 24576 data: 5 5 5 4 3 3
9430 176/300 Read block 1224 len 40960
9431 177/300 Write block 4743 len 32768 data: 2 2 2 2 2 2 2 2
9432 178/300 Read block 646 len 36864
9433 179/300 Write block 3 len 36864 data: 2 3 3 2 2 2 2 2 2
9434 180/300 Read block 2254 len 4096
9435 181/300 Write block 2595 len 24576 data: 3 2 2 2 2 2
9436 182/300 Read block 3842 len 24576
9437 183/300 Write block 4761 len 4096 data: 4
9438 184/300 Write block 1505 len 36864 data: 4 5 5 4 4 5 4 4 3
9439 185/300 Read block 590 len 40960
9440 186/300 Write block 107 len 24576 data: 4 5 5 7 7 7
9441 187/300 Read block 250 len 16384
9442 188/300 Write block 1191 len 12288 data: 3 3 3
9443 189/300 Write block 609 len 24576 data: 2 2 2 2 2 2
9444 190/300 Flush
9445 191/300 Write block 535 len 24576 data: 4 4 4 5 4 4
9446 192/300 Read block 3941 len 28672
9447 193/300 Write block 4209 len 28672 data: 4 3 3 3 2 2 2
9448 194/300 Read block 1654 len 32768
9449 195/300 Write block 3541 len 8192 data: 3 4
9450 196/300 Read block 1415 len 36864
9451 197/300 Read block 77 len 36864
9452 198/300 Read block 4474 len 36864
9453 199/300 Write block 735 len 4096 data: 4
9454 200/300 Flush
9455 201/300 Read block 1143 len 12288
9456 202/300 Read block 4348 len 32768
9457 203/300 Write block 2455 len 40960 data: 3 3 4 3 3 3 3 3 3 4
9458 204/300 Read block 3807 len 20480
9459 205/300 Flush
9460 206/300 Read block 140 len 28672
9461 207/300 Read block 3094 len 40960
9462 208/300 Write block 1545 len 8192 data: 2 2
9463 209/300 Read block 639 len 20480
9464 210/300 Write block 1137 len 8192 data: 3 3
9465 211/300 Write block 4730 len 40960 data: 3 3 3 2 2 2 2 2 2 2
9466 212/300 Write block 2666 len 36864 data: 2 2 2 3 4 3 3 3 3
9467 213/300 Read block 4179 len 40960
9468 214/300 Read block 3782 len 28672
9469 215/300 Write block 658 len 36864 data: 4 3 3 2 2 2 2 2 2
9470 216/300 Write block 4468 len 4096 data: 2
9471 217/300 Read block 2172 len 4096
9472 218/300 Read block 3465 len 12288
9473 219/300 Write block 293 len 12288 data: 3 3 4
9474 220/300 Write block 4597 len 12288 data: 2 2 2
9475 221/300 Read block 3309 len 20480
9476 222/300 Write block 3307 len 40960 data: 4 4 4 5 4 4 4 4 4 3
9477 223/300 Write block 1258 len 8192 data: 8 8
9478 224/300 Write block 641 len 32768 data: 5 4 3 3 3 3 2 2
9479 225/300 Write block 923 len 8192 data: 3 3
9480 226/300 Read block 258 len 32768
9481 227/300 Read block 3728 len 12288
9482 228/300 Read block 610 len 28672
9483 229/300 Read block 4012 len 40960
9484 230/300 Read block 3635 len 4096
9485 231/300 Write block 4004 len 32768 data: 2 2 2 2 2 2 3 3
9486 232/300 Write block 583 len 20480 data: 2 3 3 3 3
9487 233/300 Write block 1474 len 28672 data: 4 3 3 5 5 4 4
9488 234/300 Read block 1540 len 32768
9489 235/300 Write block 4821 len 40960 data: 3 2 2 2 2 2 3 3 3 2
9490 236/300 Write block 4452 len 40960 data: 2 2 2 3 3 2 3 3 3 3
9491 237/300 Read block 1960 len 20480
9492 238/300 Write block 1911 len 40960 data: 5 4 4 3 3 3 3 4 4 4
9493 239/300 Write block 3592 len 40960 data: 3 4 4 4 4 3 3 4 4 4
9494 240/300 Write block 3414 len 36864 data: 3 3 4 4 3 3 4 3 3
9495 241/300 Read block 284 len 36864
9496 242/300 Read block 2996 len 16384
9497 243/300 Write block 2123 len 4096 data: 3
9498 244/300 Read block 192 len 32768
9499 245/300 Write block 3546 len 16384 data: 4 4 4 5
9500 246/300 Read block 4746 len 20480
9501 247/300 Write block 1114 len 16384 data: 4 4 2 2
9502 248/300 Flush
9503 249/300 Read block 3078 len 40960
9504 250/300 Flush
9505 251/300 Flush
9506 252/300 Write block 117 len 12288 data: 5 5 5
9507 253/300 Write block 981 len 28672 data: 3 4 3 5 5 5 5
9508 254/300 Read block 3737 len 32768
9509 255/300 Write block 322 len 8192 data: 2 2
9510 256/300 Write block 2876 len 36864 data: 3 2 2 2 2 2 2 2 2
9511 257/300 Write block 861 len 32768 data: 2 2 2 2 2 2 2 2
9512 258/300 Flush
9513 259/300 Write block 67 len 32768 data: 3 3 4 4 3 3 3 3
9514 260/300 Read block 1746 len 24576
9515 261/300 Read block 2224 len 4096
9516 262/300 Write block 4113 len 28672 data: 3 3 3 3 3 3 2
9517 263/300 Read block 1774 len 24576
9518 264/300 Read block 576 len 16384
9519 265/300 Write block 139 len 20480 data: 3 4 4 3 3
9520 266/300 Read block 1065 len 24576
9521 267/300 Read block 3476 len 32768
9522 268/300 Read block 2547 len 4096
9523 269/300 Write block 4365 len 20480 data: 4 4 4 4 4
9524 270/300 Read block 2099 len 20480
9525 271/300 Flush
9526 272/300 Write block 4504 len 28672 data: 2 2 2 2 2 3 3
9527 273/300 Read block 4697 len 24576
9528 274/300 Write block 778 len 36864 data: 4 4 4 2 2 3 4 5 3
9529 275/300 Read block 4530 len 36864
9530 276/300 Write block 2523 len 24576 data: 2 2 3 3 3 4
9531 277/300 Flush
9532 278/300 Write block 1331 len 32768 data: 5 5 4 3 3 2 2 2
9533 279/300 Write block 136 len 32768 data: 3 3 3 4 5 5 4 4
9534 280/300 Write block 3589 len 28672 data: 4 5 3 4 5 5 5
9535 281/300 Write block 1834 len 16384 data: 3 3 3 3
9536 282/300 Read block 3316 len 20480
9537 283/300 Read block 1573 len 32768
9538 284/300 Write block 4641 len 40960 data: 3 3 2 2 2 2 2 2 2 2
9539 285/300 Flush
9540 286/300 Read block 1060 len 32768
9541 287/300 Write block 3710 len 32768 data: 3 2 2 2 2 2 2 2
9542 288/300 Flush
9543 289/300 Read block 1500 len 28672
9544 290/300 Write block 1914 len 8192 data: 4 4
9545 291/300 Write block 293 len 24576 data: 4 4 5 3 3 4
9546 292/300 Read block 3015 len 32768
9547 293/300 Flush
9548 294/300 Write block 1308 len 40960 data: 4 4 5 5 3 3 3 3 3 2
9549 295/300 Write block 1157 len 16384 data: 5 5 5 5
9550 296/300 Read block 1433 len 4096
9551 297/300 Write block 2048 len 16384 data: 4 3 3 4
9552 298/300 Write block 2832 len 4096 data: 4
9553 299/300 Read block 184 len 36864
9554 300/300 Write block 1868 len 28672 data: 7 6 4 3 2 2 3
9555 301/300 Write block 4735 len 8192 data: 3 3
9556 Main task has work to do, go find it
9557 got dsc Start(2)
9558 start 2
9559 Jan 23 17:39:45.904 INFO request completed, latency_us: 299, response_code: 204, uri: /start/cid/2, method: POST, req_id: 78ee0750-d23b-4952-8b76-2e2e3212587b, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
9560 Replay: started 2, returned:Ok(())
9561 ----------------------------------------------------------------
9562 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9563 States: Active Active Offline
9564 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9565 3416 Acked Read 1 Done Done Sent false
9566 3417 Acked Write 7 Done Done Sent false
9567 3418 Acked Write 10 Done Done Sent false
9568 3419 Acked Read 4 Done Done Sent false
9569 3420 Acked Write 8 Done Done Sent false
9570 3421 Acked Flush 0 Done Done Sent false
9571 3422 Acked Read 8 Done Done Sent false
9572 3423 Acked Read 1 Done Done Sent false
9573 3424 Acked Write 1 Done Done Sent false
9574 3425 Acked Read 8 Done Done Sent false
9575 3426 Acked Write 8 Done Done Sent false
9576 3427 Acked Read 7 Done Done Sent false
9577 3428 Acked Read 8 Done Done Sent false
9578 3429 Acked Read 8 Done Done Sent false
9579 3430 Acked Read 3 Done Done Sent false
9580 3431 Acked Write 6 Done Done Sent false
9581 3432 Acked Read 1 Done Done Sent false
9582 3433 Acked Write 3 Done Done Sent false
9583 3434 Acked Read 5 Done Done Sent false
9584 3435 Acked Read 5 Done Done Sent false
9585 3436 Acked Write 1 Done Done Sent false
9586 3437 Acked Write 9 Done Done Sent false
9587 3438 Acked Read 9 Done Done Sent false
9588 3439 Acked Read 10 Done Done Sent false
9589 3440 Acked Read 10 Done Done Sent false
9590 3441 Acked Read 7 Done Done Sent false
9591 3442 Acked Write 9 Done Done Sent false
9592 3443 Acked Flush 0 Done Done Sent false
9593 3444 Acked Read 8 Done Done Sent false
9594 3445 Acked Write 5 Done Done Sent false
9595 3446 Acked Flush 0 Done Done Sent false
9596 3447 Acked Read 2 Done Done Sent false
9597 3448 Acked Write 1 Done Done Sent false
9598 3449 Acked Write 10 Done Done Sent false
9599 3450 Acked Write 7 Done Done Sent false
9600 3451 Acked Write 2 Done Done Sent false
9601 3452 Acked Read 5 Done Done Sent false
9602 3453 Acked Read 7 Done Done Sent false
9603 3454 Acked Read 5 Done Done Sent false
9604 3455 Acked Write 1 Done Done Sent false
9605 3456 Acked Flush 0 Done Done Sent false
9606 3457 Acked Read 8 Done Done Sent false
9607 3458 Acked Write 3 Done Done Sent false
9608 3459 Acked Read 6 Done Done Sent false
9609 3460 Acked Write 10 Done Done Sent false
9610 3461 Acked Read 9 Done Done Sent false
9611 3462 Acked Read 6 Done Done Sent false
9612 3463 Acked Read 8 Done Done Sent false
9613 3464 Acked Read 2 Done Done Sent false
9614 3465 Acked Write 1 Done Done Sent false
9615 3466 Acked Read 7 Done Done Sent false
9616 3467 Acked Read 3 Done Done Sent false
9617 3468 Acked Read 10 Done Done Sent false
9618 3469 Acked Write 10 Done Done Sent false
9619 3470 Acked Read 2 Done Done Sent false
9620 3471 Acked Read 9 Done Done Sent false
9621 3472 Acked Read 10 Done Done Sent false
9622 3473 Acked Flush 0 Done Done Sent false
9623 3474 Acked Write 5 Done Done Sent false
9624 3475 Acked Read 2 Done Done Sent false
9625 3476 Acked Read 4 Done Done Sent false
9626 3477 Acked Read 6 Done Done Sent false
9627 3478 Acked Read 10 Done Done Sent false
9628 3479 Acked Write 2 Done Done Sent false
9629 3480 Acked Flush 0 Done Done Sent false
9630 3481 Acked Read 10 Done Done Sent false
9631 3482 Acked Read 6 Done Done Sent false
9632 3483 Acked Write 5 Done Done Sent false
9633 3484 Acked Read 7 Done Done Sent false
9634 3485 Acked Write 2 Done Done Sent false
9635 3486 Acked Write 2 Done Done Sent false
9636 3487 Acked Read 5 Done Done Sent false
9637 3488 Acked Write 8 Done Done Sent false
9638 3489 Acked Flush 0 Done Done Sent false
9639 3490 Acked Read 7 Done Done Sent false
9640 3491 Acked Read 9 Done Done Sent false
9641 3492 Acked Read 6 Done Done Sent false
9642 3493 Acked Flush 0 Done Done Sent false
9643 3494 Acked Read 3 Done Done Sent false
9644 3495 Acked Read 2 Done Done Sent false
9645 3496 Acked Read 1 Done Done Sent false
9646 3497 Acked Write 6 Done Done Sent false
9647 3498 Acked Read 7 Done Done Sent false
9648 3499 Acked Read 6 Done Done Sent false
9649 3500 Acked Read 1 Done Done Sent false
9650 3501 Acked Read 7 Done Done Sent false
9651 3502 Acked Write 6 Done Done Sent false
9652 3503 Acked Read 9 Done Done Sent false
9653 3504 Acked Flush 0 Done Done Sent false
9654 3505 Acked Write 10 Done Done Sent false
9655 3506 Acked Read 5 Done Done Sent false
9656 3507 Acked Write 6 Done Done Sent false
9657 3508 Acked Read 7 Done Done Sent false
9658 3509 Acked Read 8 Done Done Sent false
9659 3510 Acked Read 7 Done Done Sent false
9660 3511 Acked Read 7 Done Done Sent false
9661 3512 Acked Read 7 Done Done Sent false
9662 3513 Acked Write 10 Done Done Sent false
9663 3514 Acked Read 5 Done Done Sent false
9664 3515 Acked Write 3 Done Done Sent false
9665 3516 Acked Write 7 Done Done Sent false
9666 3517 Acked Write 3 Done Done Sent false
9667 3518 Acked Read 9 Done Done Sent false
9668 3519 Acked Write 1 Done Done Sent false
9669 3520 Acked Read 5 Done Done Sent false
9670 3521 Acked Write 9 Done Done Sent false
9671 3522 Acked Write 6 Done Done Sent false
9672 3523 Acked Read 8 Done Done Sent false
9673 3524 Acked Write 8 Done Done Sent false
9674 3525 Acked Read 1 Done Done Sent false
9675 3526 Acked Flush 0 Done Done Sent false
9676 3527 Acked Read 3 Done Done Sent false
9677 3528 Acked Read 4 Done Done Sent false
9678 3529 Acked Read 3 Done Done Sent false
9679 3530 Acked Write 6 Done Done Sent false
9680 3531 Acked Flush 0 Done Done Sent false
9681 3532 Acked Read 1 Done Done Sent false
9682 3533 Acked Read 8 Done Done Sent false
9683 3534 Acked Read 8 Done Done Sent false
9684 3535 Acked Read 10 Done Done Sent false
9685 3536 Acked Read 2 Done Done Sent false
9686 3537 Acked Write 9 Done Done Sent false
9687 3538 Acked Write 2 Done Done Sent false
9688 3539 Acked Write 7 Done Done Sent false
9689 3540 Acked Read 8 Done Done Sent false
9690 3541 Acked Write 4 Done Done Sent false
9691 3542 Acked Write 1 Done Done Sent false
9692 3543 Acked Write 9 Done Done Sent false
9693 3544 Acked Read 7 Done Done Sent false
9694 3545 Acked Read 9 Done Done Sent false
9695 3546 Acked Read 8 Done Done Sent false
9696 3547 Acked Write 7 Done Done Sent false
9697 3548 Acked Write 7 Done Done Sent false
9698 3549 Acked Read 5 Done Done Sent false
9699 3550 Acked Read 4 Done Done Sent false
9700 3551 Acked Write 10 Done Done Sent false
9701 3552 Acked Read 7 Done Done Sent false
9702 3553 Acked Write 8 Done Done Sent false
9703 3554 Acked Write 5 Done Done Sent false
9704 3555 Acked Read 5 Done Done Sent false
9705 3556 Acked Flush 0 Done Done Sent false
9706 3557 Acked Read 10 Done Done Sent false
9707 3558 Acked Write 1 Done Done Sent false
9708 3559 Acked Flush 0 Done Done Sent false
9709 3560 Acked Write 1 Done Done Sent false
9710 3561 Acked Write 4 Done Done Sent false
9711 3562 Acked Read 2 Done Done Sent false
9712 3563 Acked Write 7 Done Done Sent false
9713 3564 Acked Read 5 Done Done Sent false
9714 3565 Acked Write 10 Done Done Sent false
9715 3566 Acked Read 10 Done Done Sent false
9716 3567 Acked Read 7 Done Done Sent false
9717 3568 Acked Write 6 Done Done Sent false
9718 3569 Acked Read 10 Done Done Sent false
9719 3570 Acked Read 8 Done Done Sent false
9720 3571 Acked Write 7 Done Done Sent false
9721 3572 Acked Read 1 Done Done Sent false
9722 3573 Acked Read 10 Done Done Sent false
9723 3574 Acked Read 9 Done Done Sent false
9724 3575 Acked Write 10 Done Done Sent false
9725 3576 Acked Read 6 Done Done Sent false
9726 3577 Acked Read 3 Done Done Sent false
9727 3578 Acked Read 10 Done Done Sent false
9728 3579 Acked Write 6 Done Done Sent false
9729 3580 Acked Write 4 Done Done Sent false
9730 3581 Acked Write 4 Done Done Sent false
9731 3582 Acked Read 10 Done Done Sent false
9732 3583 Acked Write 8 Done Done Sent false
9733 3584 Acked Read 1 Done Done Sent false
9734 3585 Acked Read 10 Done Done Sent false
9735 3586 Acked Flush 0 Done Done Sent false
9736 3587 Acked Write 8 Done Done Sent false
9737 3588 Acked Write 6 Done Done Sent false
9738 3589 Acked Write 5 Done Done Sent false
9739 3590 Acked Write 6 Done Done Sent false
9740 3591 Acked Read 10 Done Done Sent false
9741 3592 Acked Write 8 Done Done Sent false
9742 3593 Acked Read 9 Done Done Sent false
9743 3594 Acked Write 9 Done Done Sent false
9744 3595 Acked Read 1 Done Done Sent false
9745 3596 Acked Write 6 Done Done Sent false
9746 3597 Acked Read 6 Done Done Sent false
9747 3598 Acked Write 1 Done Done Sent false
9748 3599 Acked Write 9 Done Done Sent false
9749 3600 Acked Read 10 Done Done Sent false
9750 3601 Acked Write 6 Done Done Sent false
9751 3602 Acked Read 4 Done Done Sent false
9752 3603 Acked Write 3 Done Done Sent false
9753 3604 Acked Write 6 Done Done Sent false
9754 3605 Acked Flush 0 Done Done Sent false
9755 3606 Acked Write 6 Done Done Sent false
9756 3607 Acked Read 7 Done Done Sent false
9757 3608 Acked Write 7 Done Done Sent false
9758 3609 Acked Read 8 Done Done Sent false
9759 3610 Acked Write 2 Done Done Sent false
9760 3611 Acked Read 9 Done Done Sent false
9761 3612 Acked Read 9 Done Done Sent false
9762 3613 Acked Read 9 Done Done Sent false
9763 3614 Acked Write 1 Done Done Sent false
9764 3615 Acked Flush 0 Done Done Sent false
9765 3616 Acked Read 3 Done Done Sent false
9766 3617 Acked Read 8 Done Done Sent false
9767 3618 Acked Write 10 Done Done Sent false
9768 3619 Acked Read 5 Done Done Sent false
9769 3620 Acked Flush 0 Done Done Sent false
9770 3621 Acked Read 7 Done Done Sent false
9771 3622 Acked Read 10 Done Done Sent false
9772 3623 Acked Write 2 Done Done Sent false
9773 3624 Acked Read 5 Done Done Sent false
9774 3625 Acked Write 2 Done Done Sent false
9775 3626 Acked Write 10 Done Done Sent false
9776 3627 Acked Write 9 Done Done Sent false
9777 3628 Acked Read 10 Done Done Sent false
9778 3629 Acked Read 7 Done Done Sent false
9779 3630 Acked Write 9 Done Done Sent false
9780 3631 Acked Write 1 Done Done Sent false
9781 3632 Acked Read 1 Done Done Sent false
9782 3633 Acked Read 3 Done Done Sent false
9783 3634 Acked Write 3 Done Done Sent false
9784 3635 Acked Write 3 Done Done Sent false
9785 3636 Acked Read 5 Done Done Sent false
9786 3637 Acked Write 10 Done Done Sent false
9787 3638 Acked Write 2 Done Done Sent false
9788 3639 Acked Write 8 Done Done Sent false
9789 3640 Acked Write 2 Done Done Sent false
9790 3641 Acked Read 8 Done Done Sent false
9791 3642 Acked Read 3 Done Done Sent false
9792 3643 Acked Read 7 Done Done Sent false
9793 3644 Acked Read 10 Done Done Sent false
9794 3645 Acked Read 1 Done Done Sent false
9795 3646 Acked Write 8 Done Done Sent false
9796 3647 Acked Write 5 Done Done Sent false
9797 3648 Acked Write 7 Done Done Sent false
9798 3649 Acked Read 8 Done Done Sent false
9799 3650 Acked Write 10 Done Done Sent false
9800 3651 Acked Write 10 Done Done Sent false
9801 3652 Acked Read 5 Done Done Sent false
9802 3653 Acked Write 10 Done Done Sent false
9803 3654 Acked Write 10 Done Done Sent false
9804 3655 Acked Write 9 Done Done Sent false
9805 3656 Acked Read 9 Done Done Sent false
9806 3657 Acked Read 4 Done Done Sent false
9807 3658 Acked Write 1 Done Done Sent false
9808 3659 Acked Read 8 Done Done Sent false
9809 3660 Acked Write 4 Done Done Sent false
9810 3661 Acked Read 5 Done Done Sent false
9811 3662 Acked Write 4 Done Done Sent false
9812 3663 Acked Flush 0 Done Done Sent false
9813 3664 Acked Read 10 Done Done Sent false
9814 3665 Acked Flush 0 Done Done Sent false
9815 3666 Acked Flush 0 Done Done Sent false
9816 3667 Acked Write 3 Done Done Sent false
9817 3668 Acked Write 7 Done Done Sent false
9818 3669 Acked Read 8 Done Done Sent false
9819 3670 Acked Write 2 Done Done Sent false
9820 3671 Acked Write 9 Done Done Sent false
9821 3672 Acked Write 8 Done Done Sent false
9822 3673 Acked Flush 0 Done Done Sent false
9823 3674 Acked Write 8 Done Done Sent false
9824 3675 Acked Read 6 Done Done Sent false
9825 3676 Acked Read 1 Done Done Sent false
9826 3677 Acked Write 7 Done Done Sent false
9827 3678 Acked Read 6 Done Done Sent false
9828 3679 Acked Read 4 Done Done Sent false
9829 3680 Acked Write 5 Done Done Sent false
9830 3681 Acked Read 6 Done Done Sent false
9831 3682 Acked Read 8 Done Done Sent false
9832 3683 Acked Read 1 Done Done Sent false
9833 3684 Acked Write 5 Done Done Sent false
9834 3685 Acked Read 5 Done Done Sent false
9835 3686 Acked Flush 0 Done Done Sent false
9836 3687 Acked Write 7 Done Done Sent false
9837 3688 Acked Read 6 Done Done Sent false
9838 3689 Acked Write 9 Done Done Sent false
9839 3690 Acked Read 9 Done Done Sent false
9840 3691 Acked Write 6 Done Done Sent false
9841 3692 Acked Flush 0 Done Done Sent false
9842 3693 Acked Write 8 Done Done Sent false
9843 3694 Acked Write 8 Done Done Sent false
9844 3695 Acked Write 7 Done Done Sent false
9845 3696 Acked Write 4 Done Done Sent false
9846 3697 Acked Read 5 Done Done Sent false
9847 3698 Acked Read 8 Done Done Sent false
9848 3699 Acked Write 10 Done Done Sent false
9849 3700 Acked Flush 0 Done Done Sent false
9850 3701 Acked Read 8 Done Done Sent false
9851 3702 Acked Write 8 Done Done Sent false
9852 3703 Acked Flush 0 Done Done Sent false
9853 3704 Acked Read 7 Done Done Sent false
9854 3705 Acked Write 2 Done Done Sent false
9855 3706 Acked Write 6 Done Done Sent false
9856 3707 Acked Read 8 Done Done Sent false
9857 3708 Acked Flush 0 Done Done Sent false
9858 3709 Acked Write 10 Done Done Sent false
9859 3710 Acked Write 4 Done Done Sent false
9860 3711 Acked Read 1 Done Done Sent false
9861 3712 Acked Write 4 Done Done Sent false
9862 3713 Acked Write 1 Done Done Sent false
9863 3714 Acked Read 9 Done Done Sent false
9864 3715 Acked Write 7 Done Done Sent false
9865 3716 Acked Write 2 Done Done Sent false
9866 STATES DS:0 DS:1 DS:2 TOTAL
9867 Sent 0 0 301 301
9868 Done 301 301 0 602
9869 Skipped 0 0 0 0
9870 Error 0 0 0 0
9871 Last Flush: Some(JobId(3708)) Some(JobId(3708)) Some(JobId(3415))
9872 Downstairs last five completed: 3415 3414 3413 3412 3411
9873 Upstairs last five completed: 3716 3715 3714 3713 3712
9874 CLIENT: Up:0 ds:301 act:2
9875 [8830] got start action
9876 [8830] I am going to restart
9877 Starting downstairs at port 8830
9878 Make output file at "/tmp/dsc/downstairs-8830.txt"
9879 [8830][2] reports Starting
9880 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1214)
9881 [8830][2] reports Running
9882 ----------------------------------------------------------------
9883 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
9884 States: Active Active Offline
9885 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9886 3416 Acked Read 1 Done Done Sent false
9887 3417 Acked Write 7 Done Done Sent false
9888 3418 Acked Write 10 Done Done Sent false
9889 3419 Acked Read 4 Done Done Sent false
9890 3420 Acked Write 8 Done Done Sent false
9891 3421 Acked Flush 0 Done Done Sent false
9892 3422 Acked Read 8 Done Done Sent false
9893 3423 Acked Read 1 Done Done Sent false
9894 3424 Acked Write 1 Done Done Sent false
9895 3425 Acked Read 8 Done Done Sent false
9896 3426 Acked Write 8 Done Done Sent false
9897 3427 Acked Read 7 Done Done Sent false
9898 3428 Acked Read 8 Done Done Sent false
9899 3429 Acked Read 8 Done Done Sent false
9900 3430 Acked Read 3 Done Done Sent false
9901 3431 Acked Write 6 Done Done Sent false
9902 3432 Acked Read 1 Done Done Sent false
9903 3433 Acked Write 3 Done Done Sent false
9904 3434 Acked Read 5 Done Done Sent false
9905 3435 Acked Read 5 Done Done Sent false
9906 3436 Acked Write 1 Done Done Sent false
9907 3437 Acked Write 9 Done Done Sent false
9908 3438 Acked Read 9 Done Done Sent false
9909 3439 Acked Read 10 Done Done Sent false
9910 3440 Acked Read 10 Done Done Sent false
9911 3441 Acked Read 7 Done Done Sent false
9912 3442 Acked Write 9 Done Done Sent false
9913 3443 Acked Flush 0 Done Done Sent false
9914 3444 Acked Read 8 Done Done Sent false
9915 3445 Acked Write 5 Done Done Sent false
9916 3446 Acked Flush 0 Done Done Sent false
9917 3447 Acked Read 2 Done Done Sent false
9918 3448 Acked Write 1 Done Done Sent false
9919 3449 Acked Write 10 Done Done Sent false
9920 3450 Acked Write 7 Done Done Sent false
9921 3451 Acked Write 2 Done Done Sent false
9922 3452 Acked Read 5 Done Done Sent false
9923 3453 Acked Read 7 Done Done Sent false
9924 3454 Acked Read 5 Done Done Sent false
9925 3455 Acked Write 1 Done Done Sent false
9926 3456 Acked Flush 0 Done Done Sent false
9927 3457 Acked Read 8 Done Done Sent false
9928 3458 Acked Write 3 Done Done Sent false
9929 3459 Acked Read 6 Done Done Sent false
9930 3460 Acked Write 10 Done Done Sent false
9931 3461 Acked Read 9 Done Done Sent false
9932 3462 Acked Read 6 Done Done Sent false
9933 3463 Acked Read 8 Done Done Sent false
9934 3464 Acked Read 2 Done Done Sent false
9935 3465 Acked Write 1 Done Done Sent false
9936 3466 Acked Read 7 Done Done Sent false
9937 3467 Acked Read 3 Done Done Sent false
9938 3468 Acked Read 10 Done Done Sent false
9939 3469 Acked Write 10 Done Done Sent false
9940 3470 Acked Read 2 Done Done Sent false
9941 3471 Acked Read 9 Done Done Sent false
9942 3472 Acked Read 10 Done Done Sent false
9943 3473 Acked Flush 0 Done Done Sent false
9944 3474 Acked Write 5 Done Done Sent false
9945 3475 Acked Read 2 Done Done Sent false
9946 3476 Acked Read 4 Done Done Sent false
9947 3477 Acked Read 6 Done Done Sent false
9948 3478 Acked Read 10 Done Done Sent false
9949 3479 Acked Write 2 Done Done Sent false
9950 3480 Acked Flush 0 Done Done Sent false
9951 3481 Acked Read 10 Done Done Sent false
9952 3482 Acked Read 6 Done Done Sent false
9953 3483 Acked Write 5 Done Done Sent false
9954 3484 Acked Read 7 Done Done Sent false
9955 3485 Acked Write 2 Done Done Sent false
9956 3486 Acked Write 2 Done Done Sent false
9957 3487 Acked Read 5 Done Done Sent false
9958 3488 Acked Write 8 Done Done Sent false
9959 3489 Acked Flush 0 Done Done Sent false
9960 3490 Acked Read 7 Done Done Sent false
9961 3491 Acked Read 9 Done Done Sent false
9962 3492 Acked Read 6 Done Done Sent false
9963 3493 Acked Flush 0 Done Done Sent false
9964 3494 Acked Read 3 Done Done Sent false
9965 3495 Acked Read 2 Done Done Sent false
9966 3496 Acked Read 1 Done Done Sent false
9967 3497 Acked Write 6 Done Done Sent false
9968 3498 Acked Read 7 Done Done Sent false
9969 3499 Acked Read 6 Done Done Sent false
9970 3500 Acked Read 1 Done Done Sent false
9971 3501 Acked Read 7 Done Done Sent false
9972 3502 Acked Write 6 Done Done Sent false
9973 3503 Acked Read 9 Done Done Sent false
9974 3504 Acked Flush 0 Done Done Sent false
9975 3505 Acked Write 10 Done Done Sent false
9976 3506 Acked Read 5 Done Done Sent false
9977 3507 Acked Write 6 Done Done Sent false
9978 3508 Acked Read 7 Done Done Sent false
9979 3509 Acked Read 8 Done Done Sent false
9980 3510 Acked Read 7 Done Done Sent false
9981 3511 Acked Read 7 Done Done Sent false
9982 3512 Acked Read 7 Done Done Sent false
9983 3513 Acked Write 10 Done Done Sent false
9984 3514 Acked Read 5 Done Done Sent false
9985 3515 Acked Write 3 Done Done Sent false
9986 3516 Acked Write 7 Done Done Sent false
9987 3517 Acked Write 3 Done Done Sent false
9988 3518 Acked Read 9 Done Done Sent false
9989 3519 Acked Write 1 Done Done Sent false
9990 3520 Acked Read 5 Done Done Sent false
9991 3521 Acked Write 9 Done Done Sent false
9992 3522 Acked Write 6 Done Done Sent false
9993 3523 Acked Read 8 Done Done Sent false
9994 3524 Acked Write 8 Done Done Sent false
9995 3525 Acked Read 1 Done Done Sent false
9996 3526 Acked Flush 0 Done Done Sent false
9997 3527 Acked Read 3 Done Done Sent false
9998 3528 Acked Read 4 Done Done Sent false
9999 3529 Acked Read 3 Done Done Sent false
10000 3530 Acked Write 6 Done Done Sent false
10001 3531 Acked Flush 0 Done Done Sent false
10002 3532 Acked Read 1 Done Done Sent false
10003 3533 Acked Read 8 Done Done Sent false
10004 3534 Acked Read 8 Done Done Sent false
10005 3535 Acked Read 10 Done Done Sent false
10006 3536 Acked Read 2 Done Done Sent false
10007 3537 Acked Write 9 Done Done Sent false
10008 3538 Acked Write 2 Done Done Sent false
10009 3539 Acked Write 7 Done Done Sent false
10010 3540 Acked Read 8 Done Done Sent false
10011 3541 Acked Write 4 Done Done Sent false
10012 3542 Acked Write 1 Done Done Sent false
10013 3543 Acked Write 9 Done Done Sent false
10014 3544 Acked Read 7 Done Done Sent false
10015 3545 Acked Read 9 Done Done Sent false
10016 3546 Acked Read 8 Done Done Sent false
10017 3547 Acked Write 7 Done Done Sent false
10018 3548 Acked Write 7 Done Done Sent false
10019 3549 Acked Read 5 Done Done Sent false
10020 3550 Acked Read 4 Done Done Sent false
10021 3551 Acked Write 10 Done Done Sent false
10022 3552 Acked Read 7 Done Done Sent false
10023 3553 Acked Write 8 Done Done Sent false
10024 3554 Acked Write 5 Done Done Sent false
10025 3555 Acked Read 5 Done Done Sent false
10026 3556 Acked Flush 0 Done Done Sent false
10027 3557 Acked Read 10 Done Done Sent false
10028 3558 Acked Write 1 Done Done Sent false
10029 3559 Acked Flush 0 Done Done Sent false
10030 3560 Acked Write 1 Done Done Sent false
10031 3561 Acked Write 4 Done Done Sent false
10032 3562 Acked Read 2 Done Done Sent false
10033 3563 Acked Write 7 Done Done Sent false
10034 3564 Acked Read 5 Done Done Sent false
10035 3565 Acked Write 10 Done Done Sent false
10036 3566 Acked Read 10 Done Done Sent false
10037 3567 Acked Read 7 Done Done Sent false
10038 3568 Acked Write 6 Done Done Sent false
10039 3569 Acked Read 10 Done Done Sent false
10040 3570 Acked Read 8 Done Done Sent false
10041 3571 Acked Write 7 Done Done Sent false
10042 3572 Acked Read 1 Done Done Sent false
10043 3573 Acked Read 10 Done Done Sent false
10044 3574 Acked Read 9 Done Done Sent false
10045 3575 Acked Write 10 Done Done Sent false
10046 3576 Acked Read 6 Done Done Sent false
10047 3577 Acked Read 3 Done Done Sent false
10048 3578 Acked Read 10 Done Done Sent false
10049 3579 Acked Write 6 Done Done Sent false
10050 3580 Acked Write 4 Done Done Sent false
10051 3581 Acked Write 4 Done Done Sent false
10052 3582 Acked Read 10 Done Done Sent false
10053 3583 Acked Write 8 Done Done Sent false
10054 3584 Acked Read 1 Done Done Sent false
10055 3585 Acked Read 10 Done Done Sent false
10056 3586 Acked Flush 0 Done Done Sent false
10057 3587 Acked Write 8 Done Done Sent false
10058 3588 Acked Write 6 Done Done Sent false
10059 3589 Acked Write 5 Done Done Sent false
10060 3590 Acked Write 6 Done Done Sent false
10061 3591 Acked Read 10 Done Done Sent false
10062 3592 Acked Write 8 Done Done Sent false
10063 3593 Acked Read 9 Done Done Sent false
10064 3594 Acked Write 9 Done Done Sent false
10065 3595 Acked Read 1 Done Done Sent false
10066 3596 Acked Write 6 Done Done Sent false
10067 3597 Acked Read 6 Done Done Sent false
10068 3598 Acked Write 1 Done Done Sent false
10069 3599 Acked Write 9 Done Done Sent false
10070 3600 Acked Read 10 Done Done Sent false
10071 3601 Acked Write 6 Done Done Sent false
10072 3602 Acked Read 4 Done Done Sent false
10073 3603 Acked Write 3 Done Done Sent false
10074 3604 Acked Write 6 Done Done Sent false
10075 3605 Acked Flush 0 Done Done Sent false
10076 3606 Acked Write 6 Done Done Sent false
10077 3607 Acked Read 7 Done Done Sent false
10078 3608 Acked Write 7 Done Done Sent false
10079 3609 Acked Read 8 Done Done Sent false
10080 3610 Acked Write 2 Done Done Sent false
10081 3611 Acked Read 9 Done Done Sent false
10082 3612 Acked Read 9 Done Done Sent false
10083 3613 Acked Read 9 Done Done Sent false
10084 3614 Acked Write 1 Done Done Sent false
10085 3615 Acked Flush 0 Done Done Sent false
10086 3616 Acked Read 3 Done Done Sent false
10087 3617 Acked Read 8 Done Done Sent false
10088 3618 Acked Write 10 Done Done Sent false
10089 3619 Acked Read 5 Done Done Sent false
10090 3620 Acked Flush 0 Done Done Sent false
10091 3621 Acked Read 7 Done Done Sent false
10092 3622 Acked Read 10 Done Done Sent false
10093 3623 Acked Write 2 Done Done Sent false
10094 3624 Acked Read 5 Done Done Sent false
10095 3625 Acked Write 2 Done Done Sent false
10096 3626 Acked Write 10 Done Done Sent false
10097 3627 Acked Write 9 Done Done Sent false
10098 3628 Acked Read 10 Done Done Sent false
10099 3629 Acked Read 7 Done Done Sent false
10100 3630 Acked Write 9 Done Done Sent false
10101 3631 Acked Write 1 Done Done Sent false
10102 3632 Acked Read 1 Done Done Sent false
10103 3633 Acked Read 3 Done Done Sent false
10104 3634 Acked Write 3 Done Done Sent false
10105 3635 Acked Write 3 Done Done Sent false
10106 3636 Acked Read 5 Done Done Sent false
10107 3637 Acked Write 10 Done Done Sent false
10108 3638 Acked Write 2 Done Done Sent false
10109 3639 Acked Write 8 Done Done Sent false
10110 3640 Acked Write 2 Done Done Sent false
10111 3641 Acked Read 8 Done Done Sent false
10112 3642 Acked Read 3 Done Done Sent false
10113 3643 Acked Read 7 Done Done Sent false
10114 3644 Acked Read 10 Done Done Sent false
10115 3645 Acked Read 1 Done Done Sent false
10116 3646 Acked Write 8 Done Done Sent false
10117 3647 Acked Write 5 Done Done Sent false
10118 3648 Acked Write 7 Done Done Sent false
10119 3649 Acked Read 8 Done Done Sent false
10120 3650 Acked Write 10 Done Done Sent false
10121 3651 Acked Write 10 Done Done Sent false
10122 3652 Acked Read 5 Done Done Sent false
10123 3653 Acked Write 10 Done Done Sent false
10124 3654 Acked Write 10 Done Done Sent false
10125 3655 Acked Write 9 Done Done Sent false
10126 3656 Acked Read 9 Done Done Sent false
10127 3657 Acked Read 4 Done Done Sent false
10128 3658 Acked Write 1 Done Done Sent false
10129 3659 Acked Read 8 Done Done Sent false
10130 3660 Acked Write 4 Done Done Sent false
10131 3661 Acked Read 5 Done Done Sent false
10132 3662 Acked Write 4 Done Done Sent false
10133 3663 Acked Flush 0 Done Done Sent false
10134 3664 Acked Read 10 Done Done Sent false
10135 3665 Acked Flush 0 Done Done Sent false
10136 3666 Acked Flush 0 Done Done Sent false
10137 3667 Acked Write 3 Done Done Sent false
10138 3668 Acked Write 7 Done Done Sent false
10139 3669 Acked Read 8 Done Done Sent false
10140 3670 Acked Write 2 Done Done Sent false
10141 3671 Acked Write 9 Done Done Sent false
10142 3672 Acked Write 8 Done Done Sent false
10143 3673 Acked Flush 0 Done Done Sent false
10144 3674 Acked Write 8 Done Done Sent false
10145 3675 Acked Read 6 Done Done Sent false
10146 3676 Acked Read 1 Done Done Sent false
10147 3677 Acked Write 7 Done Done Sent false
10148 3678 Acked Read 6 Done Done Sent false
10149 3679 Acked Read 4 Done Done Sent false
10150 3680 Acked Write 5 Done Done Sent false
10151 3681 Acked Read 6 Done Done Sent false
10152 3682 Acked Read 8 Done Done Sent false
10153 3683 Acked Read 1 Done Done Sent false
10154 3684 Acked Write 5 Done Done Sent false
10155 3685 Acked Read 5 Done Done Sent false
10156 3686 Acked Flush 0 Done Done Sent false
10157 3687 Acked Write 7 Done Done Sent false
10158 3688 Acked Read 6 Done Done Sent false
10159 3689 Acked Write 9 Done Done Sent false
10160 3690 Acked Read 9 Done Done Sent false
10161 3691 Acked Write 6 Done Done Sent false
10162 3692 Acked Flush 0 Done Done Sent false
10163 3693 Acked Write 8 Done Done Sent false
10164 3694 Acked Write 8 Done Done Sent false
10165 3695 Acked Write 7 Done Done Sent false
10166 3696 Acked Write 4 Done Done Sent false
10167 3697 Acked Read 5 Done Done Sent false
10168 3698 Acked Read 8 Done Done Sent false
10169 3699 Acked Write 10 Done Done Sent false
10170 3700 Acked Flush 0 Done Done Sent false
10171 3701 Acked Read 8 Done Done Sent false
10172 3702 Acked Write 8 Done Done Sent false
10173 3703 Acked Flush 0 Done Done Sent false
10174 3704 Acked Read 7 Done Done Sent false
10175 3705 Acked Write 2 Done Done Sent false
10176 3706 Acked Write 6 Done Done Sent false
10177 3707 Acked Read 8 Done Done Sent false
10178 3708 Acked Flush 0 Done Done Sent false
10179 3709 Acked Write 10 Done Done Sent false
10180 3710 Acked Write 4 Done Done Sent false
10181 3711 Acked Read 1 Done Done Sent false
10182 3712 Acked Write 4 Done Done Sent false
10183 3713 Acked Write 1 Done Done Sent false
10184 3714 Acked Read 9 Done Done Sent false
10185 3715 Acked Write 7 Done Done Sent false
10186 3716 Acked Write 2 Done Done Sent false
10187 3717 Acked Flush 0 Done Done Sent false
10188 STATES DS:0 DS:1 DS:2 TOTAL
10189 Sent 0 0 302 302
10190 Done 302 302 0 604
10191 Skipped 0 0 0 0
10192 Error 0 0 0 0
10193 Last Flush: Some(JobId(3717)) Some(JobId(3717)) Some(JobId(3415))
10194 Downstairs last five completed: 3415 3414 3413 3412 3411
10195 Upstairs last five completed: 3717 3716 3715 3714 3713
10196 CLIENT: Up:0 ds:302 act:2
101972025-01-23T17:39:51.645ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
101982025-01-23T17:39:51.645ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56277) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
101992025-01-23T17:39:51.646ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
102002025-01-23T17:39:51.647ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
102012025-01-23T17:39:51.647ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
102022025-01-23T17:39:51.647ZINFOcrucible: send last flush ID to this DS: Some(JobId(3415)) = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
102032025-01-23T17:39:51.647ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
102042025-01-23T17:39:51.647ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(3415)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
102052025-01-23T17:39:51.647ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(3415)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
10206 ----------------------------------------------------------------
10207 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10208 Downstairs last five completed: 3717 3716 3715 3714 3713
10209 Upstairs last five completed: 3717 3716 3715 3714 3713
10210 CLIENT: Up:0 ds:0 act:3
10211 Replay: All jobs finished
10212 Main task has work to do, go find it
10213 got dsc Stop(1)
10214 stop 1
10215 [8820] Got stop action so:false kr:false
10216 Jan 23 17:39:53.908 INFO request completed, latency_us: 327, response_code: 204, uri: /stop/cid/1, method: POST, req_id: ac94715f-5020-4e0b-85e2-27825573dec3, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
102172025-01-23T17:39:53.909ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
102182025-01-23T17:39:53.909ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
102192025-01-23T17:39:53.909ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
102202025-01-23T17:39:53.909ZINFOcrucible: client task is exiting = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
10221 Jan 23 17:39:53.909 INFO request completed, latency_us: 156, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2a91fa07-a451-48e1-8096-2841ac1a184e, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
10222 [8820] Exited with: ExitStatus(unix_wait_status(9))
10223 [8820][1] reports Exit
10224 Jan 23 17:39:57.911 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/1, method: GET, req_id: e83578b2-edaf-4434-baf4-8d28fc03f33b, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
10225 001/300 Read block 4121 len 4096
10226 002/300 Read block 3920 len 12288
10227 003/300 Flush
10228 004/300 Write block 1675 len 32768 data: 2 2 2 2 2 4 5 5
10229 005/300 Read block 4313 len 24576
10230 006/300 Flush
10231 007/300 Read block 27 len 32768
10232 008/300 Read block 579 len 8192
10233 009/300 Write block 33 len 8192 data: 3 4
10234 010/300 Flush
10235 011/300 Read block 580 len 20480
10236 012/300 Read block 195 len 32768
10237 013/300 Read block 1484 len 28672
10238 014/300 Read block 1215 len 8192
10239 015/300 Read block 4366 len 24576
10240 016/300 Read block 1603 len 36864
10241 017/300 Read block 1927 len 32768
10242 018/300 Read block 3546 len 16384
10243 019/300 Flush
10244 020/300 Read block 4366 len 16384
10245 021/300 Write block 1002 len 4096 data: 3
10246 022/300 Read block 2319 len 20480
10247 023/300 Read block 1919 len 36864
10248 024/300 Read block 4453 len 12288
10249 025/300 Write block 4412 len 16384 data: 2 2 2 2
10250 026/300 Write block 2695 len 28672 data: 2 3 3 3 3 3 3
10251 027/300 Write block 2413 len 4096 data: 3
10252 028/300 Read block 783 len 28672
10253 029/300 Read block 2480 len 28672
10254 030/300 Read block 3215 len 28672
10255 031/300 Read block 1875 len 32768
10256 032/300 Read block 204 len 28672
10257 033/300 Read block 1538 len 12288
10258 034/300 Write block 929 len 24576 data: 2 2 2 2 2 2
10259 035/300 Read block 3328 len 4096
10260 036/300 Write block 493 len 8192 data: 2 2
10261 037/300 Read block 1285 len 36864
10262 038/300 Read block 3944 len 4096
10263 039/300 Write block 2728 len 40960 data: 3 3 3 3 3 3 2 2 2 3
10264 040/300 Write block 3879 len 32768 data: 3 2 2 2 2 4 4 3
10265 041/300 Write block 1531 len 32768 data: 2 2 3 3 3 3 3 2
10266 042/300 Read block 3224 len 8192
10267 043/300 Read block 2243 len 12288
10268 044/300 Read block 3970 len 24576
10269 045/300 Read block 4316 len 28672
10270 046/300 Write block 2921 len 32768 data: 3 3 3 3 3 3 3 3
10271 047/300 Write block 2293 len 12288 data: 3 3 2
10272 048/300 Write block 3161 len 16384 data: 3 3 2 2
10273 049/300 Read block 1559 len 12288
10274 050/300 Write block 2238 len 32768 data: 2 4 4 4 4 4 3 3
10275 051/300 Write block 784 len 4096 data: 5
10276 052/300 Read block 655 len 4096
10277 053/300 Write block 2080 len 4096 data: 3
10278 054/300 Read block 1662 len 20480
10279 055/300 Write block 2074 len 36864 data: 4 3 3 2 2 2 4 3 4
10280 056/300 Read block 3221 len 36864
10281 057/300 Read block 1874 len 20480
10282 058/300 Write block 1751 len 12288 data: 4 4 4
10283 059/300 Read block 3644 len 4096
10284 060/300 Write block 2501 len 12288 data: 2 2 2
10285 061/300 Read block 771 len 12288
10286 062/300 Read block 2788 len 12288
10287 063/300 Read block 2918 len 4096
10288 064/300 Write block 2226 len 28672 data: 2 2 3 3 4 4 4
10289 065/300 Read block 994 len 20480
10290 066/300 Write block 584 len 8192 data: 4 4
10291 067/300 Write block 940 len 32768 data: 2 2 2 2 2 2 2 2
10292 068/300 Write block 3133 len 32768 data: 3 3 2 2 2 2 2 3
10293 069/300 Read block 4867 len 36864
10294 070/300 Read block 1769 len 12288
10295 071/300 Write block 3880 len 24576 data: 3 3 3 3 5 5
10296 072/300 Write block 1532 len 8192 data: 3 4
10297 073/300 Read block 4375 len 16384
10298 074/300 Write block 1019 len 36864 data: 2 2 3 3 3 4 4 4 5
10299 075/300 Write block 4264 len 4096 data: 2
10300 076/300 Flush
10301 077/300 Read block 3999 len 20480
10302 078/300 Read block 2057 len 20480
10303 079/300 Read block 455 len 4096
10304 080/300 Write block 1059 len 32768 data: 2 2 3 3 3 3 3 2
10305 081/300 Read block 4602 len 40960
10306 082/300 Read block 2131 len 16384
10307 083/300 Flush
10308 084/300 Flush
10309 085/300 Read block 3690 len 24576
10310 086/300 Write block 3035 len 40960 data: 5 4 4 3 3 3 3 2 2 3
10311 087/300 Write block 4270 len 8192 data: 5 6
10312 088/300 Flush
10313 089/300 Read block 77 len 32768
10314 090/300 Read block 4494 len 20480
10315 091/300 Write block 4967 len 40960 data: 2 2 2 2 4 4 3 2 3 3
10316 092/300 Write block 4552 len 4096 data: 3
10317 093/300 Write block 2271 len 40960 data: 2 2 2 2 2 2 2 2 2 3
10318 094/300 Write block 74 len 16384 data: 4 3 3 4
10319 095/300 Read block 53 len 36864
10320 096/300 Read block 369 len 36864
10321 097/300 Write block 3850 len 12288 data: 4 5 5
10322 098/300 Write block 1009 len 16384 data: 3 3 3 3
10323 099/300 Read block 3112 len 40960
10324 100/300 Write block 2226 len 32768 data: 3 3 4 4 5 5 5 4
10325 101/300 Write block 2281 len 32768 data: 3 3 3 2 2 3 3 3
10326 102/300 Write block 3716 len 32768 data: 3 3 2 2 3 3 4 4
10327 103/300 Read block 2212 len 24576
10328 104/300 Write block 3315 len 36864 data: 5 4 3 3 3 2 2 2 2
10329 105/300 Flush
10330 106/300 Write block 4884 len 24576 data: 3 3 3 3 2 2
10331 107/300 Write block 4846 len 28672 data: 3 3 3 3 2 2 2
10332 108/300 Write block 294 len 40960 data: 5 6 4 4 5 5 5 4 3 3
10333 109/300 Flush
10334 110/300 Write block 3818 len 20480 data: 3 3 3 3 3
10335 111/300 Read block 4034 len 4096
10336 112/300 Read block 4072 len 36864
10337 113/300 Read block 773 len 8192
10338 114/300 Read block 425 len 28672
10339 115/300 Read block 59 len 36864
10340 116/300 Read block 468 len 32768
10341 117/300 Write block 2413 len 20480 data: 4 3 3 3 4
10342 118/300 Read block 2889 len 16384
10343 119/300 Read block 1757 len 12288
10344 120/300 Read block 1890 len 4096
10345 121/300 Read block 4626 len 4096
10346 122/300 Read block 3440 len 40960
10347 123/300 Write block 3670 len 24576 data: 3 3 3 3 3 3
10348 124/300 Write block 447 len 4096 data: 4
10349 125/300 Read block 3658 len 36864
10350 126/300 Read block 4200 len 28672
10351 127/300 Read block 1872 len 4096
10352 128/300 Write block 4117 len 20480 data: 4 4 3 2 2
10353 129/300 Read block 32 len 20480
10354 130/300 Write block 3944 len 4096 data: 3
10355 131/300 Read block 269 len 24576
10356 132/300 Write block 4029 len 4096 data: 4
10357 133/300 Read block 1343 len 16384
10358 134/300 Read block 2817 len 16384
10359 135/300 Read block 4488 len 32768
10360 136/300 Write block 3242 len 16384 data: 3 2 2 2
10361 137/300 Write block 565 len 28672 data: 2 2 2 2 3 5 6
10362 138/300 Read block 824 len 16384
10363 139/300 Read block 129 len 16384
10364 140/300 Read block 4235 len 40960
10365 141/300 Write block 1623 len 4096 data: 4
10366 142/300 Write block 4594 len 8192 data: 3 2
10367 143/300 Read block 4720 len 24576
10368 144/300 Read block 4116 len 4096
10369 145/300 Read block 2433 len 24576
10370 146/300 Read block 3761 len 4096
10371 147/300 Read block 3292 len 4096
10372 148/300 Read block 3059 len 36864
10373 149/300 Read block 2743 len 28672
10374 150/300 Flush
10375 151/300 Write block 4853 len 40960 data: 2 2 2 2 2 2 2 2 2 2
10376 152/300 Read block 4588 len 40960
10377 153/300 Write block 3706 len 16384 data: 4 4 4 3
10378 154/300 Write block 4278 len 32768 data: 4 4 5 5 5 5 6 5
10379 155/300 Write block 2052 len 20480 data: 4 4 4 3 3
10380 156/300 Read block 93 len 32768
10381 157/300 Read block 2316 len 4096
10382 158/300 Read block 3994 len 28672
10383 159/300 Read block 1378 len 16384
10384 160/300 Read block 2039 len 16384
10385 161/300 Read block 1307 len 4096
10386 162/300 Read block 1367 len 12288
10387 163/300 Write block 1337 len 4096 data: 3
10388 164/300 Write block 3959 len 32768 data: 3 4 5 5 5 4 4 3
10389 165/300 Write block 2055 len 32768 data: 4 4 4 4 4 4 4 5
10390 166/300 Read block 4595 len 36864
10391 167/300 Read block 2280 len 16384
10392 168/300 Write block 1116 len 28672 data: 3 3 2 2 2 2 2
10393 169/300 Write block 3627 len 12288 data: 3 3 3
10394 170/300 Read block 4634 len 16384
10395 171/300 Flush
10396 172/300 Write block 3489 len 16384 data: 3 3 3 3
10397 173/300 Read block 4596 len 4096
10398 174/300 Read block 1120 len 40960
10399 175/300 Read block 4548 len 32768
10400 176/300 Read block 4492 len 12288
10401 177/300 Read block 465 len 20480
10402 178/300 Write block 4150 len 36864 data: 2 2 2 2 2 2 2 2 2
10403 179/300 Write block 4080 len 12288 data: 3 3 3
10404 180/300 Write block 1687 len 20480 data: 3 3 2 2 3
10405 181/300 Write block 4622 len 32768 data: 6 5 3 3 3 3 4 4
10406 182/300 Write block 1196 len 24576 data: 2 2 2 2 3 3
10407 183/300 Write block 2658 len 20480 data: 2 2 2 3 3
10408 184/300 Read block 1189 len 32768
10409 185/300 Write block 1066 len 40960 data: 3 2 2 2 2 3 3 3 3 3
10410 186/300 Flush
10411 187/300 Read block 2336 len 40960
10412 188/300 Flush
10413 189/300 Write block 3013 len 40960 data: 5 5 4 4 4 4 3 2 2 2
10414 190/300 Read block 4586 len 28672
10415 191/300 Read block 2189 len 12288
10416 192/300 Write block 3948 len 12288 data: 3 4 3
10417 193/300 Read block 1010 len 16384
10418 194/300 Write block 4255 len 32768 data: 2 3 3 3 3 3 3 2
10419 195/300 Read block 1456 len 40960
10420 196/300 Read block 2341 len 20480
10421 197/300 Read block 3624 len 20480
10422 198/300 Read block 2302 len 12288
10423 199/300 Read block 3766 len 16384
10424 200/300 Read block 1720 len 28672
10425 201/300 Write block 1674 len 24576 data: 2 3 3 3 3 3
10426 202/300 Read block 284 len 16384
10427 203/300 Read block 1561 len 36864
10428 204/300 Read block 1209 len 8192
10429 205/300 Read block 4128 len 36864
10430 206/300 Read block 3328 len 4096
10431 207/300 Read block 2497 len 8192
10432 208/300 Read block 3763 len 24576
10433 209/300 Write block 2473 len 12288 data: 2 3 3
10434 210/300 Write block 1539 len 12288 data: 2 2 2
10435 211/300 Write block 3935 len 4096 data: 3
10436 212/300 Write block 2131 len 40960 data: 3 3 2 2 2 2 2 2 2 2
10437 213/300 Write block 3491 len 28672 data: 4 4 4 4 4 4 5
10438 214/300 Read block 366 len 32768
10439 215/300 Write block 1170 len 36864 data: 2 3 3 3 3 2 3 4 4
10440 216/300 Read block 157 len 36864
10441 217/300 Read block 4065 len 32768
10442 218/300 Read block 2687 len 16384
10443 219/300 Read block 1963 len 4096
10444 220/300 Write block 2815 len 36864 data: 2 2 2 2 3 3 2 2 2
10445 221/300 Write block 47 len 36864 data: 2 2 2 2 2 3 4 4 4
10446 222/300 Write block 1808 len 8192 data: 2 2
10447 223/300 Read block 522 len 4096
10448 224/300 Flush
10449 225/300 Read block 4926 len 8192
10450 226/300 Flush
10451 227/300 Read block 8 len 40960
10452 228/300 Write block 1400 len 16384 data: 2 3 3 3
10453 229/300 Write block 349 len 12288 data: 3 3 3
10454 230/300 Read block 3453 len 28672
10455 231/300 Read block 2013 len 20480
10456 232/300 Write block 479 len 28672 data: 2 3 3 3 4 4 4
10457 233/300 Read block 3248 len 12288
10458 234/300 Write block 1686 len 20480 data: 3 4 4 3 3
10459 235/300 Write block 4566 len 28672 data: 5 5 5 5 6 5 3
10460 236/300 Read block 3009 len 12288
10461 237/300 Read block 1983 len 32768
10462 238/300 Write block 3500 len 12288 data: 3 4 4
10463 239/300 Read block 4434 len 24576
10464 240/300 Write block 3456 len 28672 data: 3 4 4 5 5 5 5
10465 241/300 Read block 1202 len 40960
10466 242/300 Write block 3110 len 36864 data: 6 7 6 5 5 5 5 5 4
10467 243/300 Write block 2276 len 36864 data: 3 3 3 3 4 4 4 4 3
10468 244/300 Write block 4269 len 24576 data: 5 6 7 6 5 5
10469 245/300 Read block 1053 len 40960
10470 246/300 Write block 1694 len 8192 data: 2 2
10471 247/300 Read block 912 len 28672
10472 248/300 Write block 553 len 20480 data: 3 3 3 5 4
10473 249/300 Flush
10474 250/300 Write block 3558 len 24576 data: 3 3 2 2 2 2
10475 251/300 Read block 2844 len 8192
10476 252/300 Write block 4273 len 28672 data: 6 6 5 3 3 5 5
10477 253/300 Read block 143 len 4096
10478 254/300 Write block 2991 len 12288 data: 3 4 3
10479 255/300 Read block 3623 len 40960
10480 256/300 Write block 1967 len 4096 data: 4
10481 257/300 Flush
10482 258/300 Read block 3104 len 12288
10483 259/300 Read block 3 len 20480
10484 260/300 Flush
10485 261/300 Write block 4376 len 28672 data: 2 2 2 2 2 2 2
10486 262/300 Read block 2027 len 8192
10487 263/300 Write block 3224 len 12288 data: 2 3 3
10488 264/300 Write block 1834 len 16384 data: 4 4 4 4
10489 265/300 Read block 1616 len 32768
10490 266/300 Flush
10491 267/300 Read block 1089 len 4096
10492 268/300 Read block 3133 len 32768
10493 269/300 Read block 4895 len 40960
10494 270/300 Read block 1980 len 12288
10495 271/300 Write block 936 len 20480 data: 3 3 3 2 3
10496 272/300 Write block 1610 len 16384 data: 2 3 4 4
10497 273/300 Write block 4589 len 40960 data: 3 3 3 3 3 4 3 2 3 3
10498 274/300 Read block 3783 len 12288
10499 275/300 Write block 4290 len 20480 data: 3 3 2 2 2
10500 276/300 Read block 3309 len 28672
10501 277/300 Flush
10502 278/300 Write block 2268 len 16384 data: 2 2 2 3
10503 279/300 Write block 2944 len 12288 data: 4 3 3
10504 280/300 Read block 2356 len 40960
10505 281/300 Write block 3293 len 4096 data: 4
10506 282/300 Write block 3174 len 12288 data: 2 2 2
10507 283/300 Read block 1549 len 12288
10508 284/300 Read block 1742 len 36864
10509 285/300 Write block 73 len 32768 data: 4 5 4 4 5 4 4 5
10510 286/300 Read block 1222 len 4096
10511 287/300 Read block 1036 len 28672
10512 288/300 Read block 1908 len 40960
10513 289/300 Read block 3057 len 36864
10514 290/300 Write block 3811 len 24576 data: 3 3 2 3 2 2
10515 291/300 Read block 4683 len 4096
10516 292/300 Write block 1647 len 40960 data: 4 4 4 4 4 5 5 3 3 2
10517 293/300 Write block 3568 len 28672 data: 4 4 3 3 3 3 3
10518 294/300 Write block 1159 len 28672 data: 6 6 6 7 7 7 5
10519 295/300 Write block 214 len 4096 data: 3
10520 296/300 Flush
10521 297/300 Flush
10522 298/300 Write block 3943 len 20480 data: 4 4 3 4 4
10523 299/300 Read block 1636 len 16384
10524 300/300 Read block 4332 len 4096
10525 301/300 Read block 2145 len 20480
10526 Main task has work to do, go find it
10527 got dsc Start(1)
10528 start 1
10529 Jan 23 17:39:58.155 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/1, method: POST, req_id: 8d653bbd-f560-469e-8b48-d0662262ac4b, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
10530 Replay: started 1, returned:Ok(())
10531 ----------------------------------------------------------------
10532 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10533 States: Active Offline Active
10534 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10535 3718 Acked Read 1 Done Sent Done false
10536 3719 Acked Read 3 Done Sent Done false
10537 3720 Acked Flush 0 Done Sent Done false
10538 3721 Acked Write 8 Done Sent Done false
10539 3722 Acked Read 6 Done Sent Done false
10540 3723 Acked Flush 0 Done Sent Done false
10541 3724 Acked Read 8 Done Sent Done false
10542 3725 Acked Read 2 Done Sent Done false
10543 3726 Acked Write 2 Done Sent Done false
10544 3727 Acked Flush 0 Done Sent Done false
10545 3728 Acked Read 5 Done Sent Done false
10546 3729 Acked Read 8 Done Sent Done false
10547 3730 Acked Read 7 Done Sent Done false
10548 3731 Acked Read 2 Done Sent Done false
10549 3732 Acked Read 6 Done Sent Done false
10550 3733 Acked Read 9 Done Sent Done false
10551 3734 Acked Read 8 Done Sent Done false
10552 3735 Acked Read 4 Done Sent Done false
10553 3736 Acked Flush 0 Done Sent Done false
10554 3737 Acked Read 4 Done Sent Done false
10555 3738 Acked Write 1 Done Sent Done false
10556 3739 Acked Read 5 Done Sent Done false
10557 3740 Acked Read 9 Done Sent Done false
10558 3741 Acked Read 3 Done Sent Done false
10559 3742 Acked Write 4 Done Sent Done false
10560 3743 Acked Write 7 Done Sent Done false
10561 3744 Acked Write 1 Done Sent Done false
10562 3745 Acked Read 7 Done Sent Done false
10563 3746 Acked Read 7 Done Sent Done false
10564 3747 Acked Read 7 Done Sent Done false
10565 3748 Acked Read 8 Done Sent Done false
10566 3749 Acked Read 7 Done Sent Done false
10567 3750 Acked Read 3 Done Sent Done false
10568 3751 Acked Write 6 Done Sent Done false
10569 3752 Acked Read 1 Done Sent Done false
10570 3753 Acked Write 2 Done Sent Done false
10571 3754 Acked Read 9 Done Sent Done false
10572 3755 Acked Read 1 Done Sent Done false
10573 3756 Acked Write 10 Done Sent Done false
10574 3757 Acked Write 8 Done Sent Done false
10575 3758 Acked Write 8 Done Sent Done false
10576 3759 Acked Read 2 Done Sent Done false
10577 3760 Acked Read 3 Done Sent Done false
10578 3761 Acked Read 6 Done Sent Done false
10579 3762 Acked Read 7 Done Sent Done false
10580 3763 Acked Write 8 Done Sent Done false
10581 3764 Acked Write 3 Done Sent Done false
10582 3765 Acked Write 4 Done Sent Done false
10583 3766 Acked Read 3 Done Sent Done false
10584 3767 Acked Write 8 Done Sent Done false
10585 3768 Acked Write 1 Done Sent Done false
10586 3769 Acked Read 1 Done Sent Done false
10587 3770 Acked Write 1 Done Sent Done false
10588 3771 Acked Read 5 Done Sent Done false
10589 3772 Acked Write 9 Done Sent Done false
10590 3773 Acked Read 9 Done Sent Done false
10591 3774 Acked Read 5 Done Sent Done false
10592 3775 Acked Write 3 Done Sent Done false
10593 3776 Acked Read 1 Done Sent Done false
10594 3777 Acked Write 3 Done Sent Done false
10595 3778 Acked Read 3 Done Sent Done false
10596 3779 Acked Read 3 Done Sent Done false
10597 3780 Acked Read 1 Done Sent Done false
10598 3781 Acked Write 7 Done Sent Done false
10599 3782 Acked Read 5 Done Sent Done false
10600 3783 Acked Write 2 Done Sent Done false
10601 3784 Acked Write 8 Done Sent Done false
10602 3785 Acked Write 8 Done Sent Done false
10603 3786 Acked Read 9 Done Sent Done false
10604 3787 Acked Read 3 Done Sent Done false
10605 3788 Acked Write 6 Done Sent Done false
10606 3789 Acked Write 2 Done Sent Done false
10607 3790 Acked Read 4 Done Sent Done false
10608 3791 Acked Write 9 Done Sent Done false
10609 3792 Acked Write 1 Done Sent Done false
10610 3793 Acked Flush 0 Done Sent Done false
10611 3794 Acked Read 5 Done Sent Done false
10612 3795 Acked Read 5 Done Sent Done false
10613 3796 Acked Read 1 Done Sent Done false
10614 3797 Acked Write 8 Done Sent Done false
10615 3798 Acked Read 10 Done Sent Done false
10616 3799 Acked Read 4 Done Sent Done false
10617 3800 Acked Flush 0 Done Sent Done false
10618 3801 Acked Flush 0 Done Sent Done false
10619 3802 Acked Read 6 Done Sent Done false
10620 3803 Acked Write 10 Done Sent Done false
10621 3804 Acked Write 2 Done Sent Done false
10622 3805 Acked Flush 0 Done Sent Done false
10623 3806 Acked Read 8 Done Sent Done false
10624 3807 Acked Read 5 Done Sent Done false
10625 3808 Acked Write 10 Done Sent Done false
10626 3809 Acked Write 1 Done Sent Done false
10627 3810 Acked Write 10 Done Sent Done false
10628 3811 Acked Write 4 Done Sent Done false
10629 3812 Acked Read 9 Done Sent Done false
10630 3813 Acked Read 9 Done Sent Done false
10631 3814 Acked Write 3 Done Sent Done false
10632 3815 Acked Write 4 Done Sent Done false
10633 3816 Acked Read 10 Done Sent Done false
10634 3817 Acked Write 8 Done Sent Done false
10635 3818 Acked Write 8 Done Sent Done false
10636 3819 Acked Write 8 Done Sent Done false
10637 3820 Acked Read 6 Done Sent Done false
10638 3821 Acked Write 9 Done Sent Done false
10639 3822 Acked Flush 0 Done Sent Done false
10640 3823 Acked Write 6 Done Sent Done false
10641 3824 Acked Write 7 Done Sent Done false
10642 3825 Acked Write 10 Done Sent Done false
10643 3826 Acked Flush 0 Done Sent Done false
10644 3827 Acked Write 5 Done Sent Done false
10645 3828 Acked Read 1 Done Sent Done false
10646 3829 Acked Read 9 Done Sent Done false
10647 3830 Acked Read 2 Done Sent Done false
10648 3831 Acked Read 7 Done Sent Done false
10649 3832 Acked Read 9 Done Sent Done false
10650 3833 Acked Read 8 Done Sent Done false
10651 3834 Acked Write 5 Done Sent Done false
10652 3835 Acked Read 4 Done Sent Done false
10653 3836 Acked Read 3 Done Sent Done false
10654 3837 Acked Read 1 Done Sent Done false
10655 3838 Acked Read 1 Done Sent Done false
10656 3839 Acked Read 10 Done Sent Done false
10657 3840 Acked Write 6 Done Sent Done false
10658 3841 Acked Write 1 Done Sent Done false
10659 3842 Acked Read 9 Done Sent Done false
10660 3843 Acked Read 7 Done Sent Done false
10661 3844 Acked Read 1 Done Sent Done false
10662 3845 Acked Write 5 Done Sent Done false
10663 3846 Acked Read 5 Done Sent Done false
10664 3847 Acked Write 1 Done Sent Done false
10665 3848 Acked Read 6 Done Sent Done false
10666 3849 Acked Write 1 Done Sent Done false
10667 3850 Acked Read 4 Done Sent Done false
10668 3851 Acked Read 4 Done Sent Done false
10669 3852 Acked Read 8 Done Sent Done false
10670 3853 Acked Write 4 Done Sent Done false
10671 3854 Acked Write 7 Done Sent Done false
10672 3855 Acked Read 4 Done Sent Done false
10673 3856 Acked Read 4 Done Sent Done false
10674 3857 Acked Read 10 Done Sent Done false
10675 3858 Acked Write 1 Done Sent Done false
10676 3859 Acked Write 2 Done Sent Done false
10677 3860 Acked Read 6 Done Sent Done false
10678 3861 Acked Read 1 Done Sent Done false
10679 3862 Acked Read 6 Done Sent Done false
10680 3863 Acked Read 1 Done Sent Done false
10681 3864 Acked Read 1 Done Sent Done false
10682 3865 Acked Read 9 Done Sent Done false
10683 3866 Acked Read 7 Done Sent Done false
10684 3867 Acked Flush 0 Done Sent Done false
10685 3868 Acked Write 10 Done Sent Done false
10686 3869 Acked Read 10 Done Sent Done false
10687 3870 Acked Write 4 Done Sent Done false
10688 3871 Acked Write 8 Done Sent Done false
10689 3872 Acked Write 5 Done Sent Done false
10690 3873 Acked Read 8 Done Sent Done false
10691 3874 Acked Read 1 Done Sent Done false
10692 3875 Acked Read 7 Done Sent Done false
10693 3876 Acked Read 4 Done Sent Done false
10694 3877 Acked Read 4 Done Sent Done false
10695 3878 Acked Read 1 Done Sent Done false
10696 3879 Acked Read 3 Done Sent Done false
10697 3880 Acked Write 1 Done Sent Done false
10698 3881 Acked Write 8 Done Sent Done false
10699 3882 Acked Write 8 Done Sent Done false
10700 3883 Acked Read 9 Done Sent Done false
10701 3884 Acked Read 4 Done Sent Done false
10702 3885 Acked Write 7 Done Sent Done false
10703 3886 Acked Write 3 Done Sent Done false
10704 3887 Acked Read 4 Done Sent Done false
10705 3888 Acked Flush 0 Done Sent Done false
10706 3889 Acked Write 4 Done Sent Done false
10707 3890 Acked Read 1 Done Sent Done false
10708 3891 Acked Read 10 Done Sent Done false
10709 3892 Acked Read 8 Done Sent Done false
10710 3893 Acked Read 3 Done Sent Done false
10711 3894 Acked Read 5 Done Sent Done false
10712 3895 Acked Write 9 Done Sent Done false
10713 3896 Acked Write 3 Done Sent Done false
10714 3897 Acked Write 5 Done Sent Done false
10715 3898 Acked Write 8 Done Sent Done false
10716 3899 Acked Write 6 Done Sent Done false
10717 3900 Acked Write 5 Done Sent Done false
10718 3901 Acked Read 8 Done Sent Done false
10719 3902 Acked Write 10 Done Sent Done false
10720 3903 Acked Flush 0 Done Sent Done false
10721 3904 Acked Read 10 Done Sent Done false
10722 3905 Acked Flush 0 Done Sent Done false
10723 3906 Acked Write 10 Done Sent Done false
10724 3907 Acked Read 7 Done Sent Done false
10725 3908 Acked Read 3 Done Sent Done false
10726 3909 Acked Write 3 Done Sent Done false
10727 3910 Acked Read 4 Done Sent Done false
10728 3911 Acked Write 8 Done Sent Done false
10729 3912 Acked Read 10 Done Sent Done false
10730 3913 Acked Read 5 Done Sent Done false
10731 3914 Acked Read 5 Done Sent Done false
10732 3915 Acked Read 3 Done Sent Done false
10733 3916 Acked Read 4 Done Sent Done false
10734 3917 Acked Read 7 Done Sent Done false
10735 3918 Acked Write 6 Done Sent Done false
10736 3919 Acked Read 4 Done Sent Done false
10737 3920 Acked Read 9 Done Sent Done false
10738 3921 Acked Read 2 Done Sent Done false
10739 3922 Acked Read 9 Done Sent Done false
10740 3923 Acked Read 1 Done Sent Done false
10741 3924 Acked Read 2 Done Sent Done false
10742 3925 Acked Read 6 Done Sent Done false
10743 3926 Acked Write 3 Done Sent Done false
10744 3927 Acked Write 3 Done Sent Done false
10745 3928 Acked Write 1 Done Sent Done false
10746 3929 Acked Write 10 Done Sent Done false
10747 3930 Acked Write 7 Done Sent Done false
10748 3931 Acked Read 8 Done Sent Done false
10749 3932 Acked Write 9 Done Sent Done false
10750 3933 Acked Read 9 Done Sent Done false
10751 3934 Acked Read 8 Done Sent Done false
10752 3935 Acked Read 4 Done Sent Done false
10753 3936 Acked Read 1 Done Sent Done false
10754 3937 Acked Write 9 Done Sent Done false
10755 3938 Acked Write 9 Done Sent Done false
10756 3939 Acked Write 2 Done Sent Done false
10757 3940 Acked Read 1 Done Sent Done false
10758 3941 Acked Flush 0 Done Sent Done false
10759 3942 Acked Read 2 Done Sent Done false
10760 3943 Acked Flush 0 Done Sent Done false
10761 3944 Acked Read 10 Done Sent Done false
10762 3945 Acked Write 4 Done Sent Done false
10763 3946 Acked Write 3 Done Sent Done false
10764 3947 Acked Read 7 Done Sent Done false
10765 3948 Acked Read 5 Done Sent Done false
10766 3949 Acked Write 7 Done Sent Done false
10767 3950 Acked Read 3 Done Sent Done false
10768 3951 Acked Write 5 Done Sent Done false
10769 3952 Acked Write 7 Done Sent Done false
10770 3953 Acked Read 3 Done Sent Done false
10771 3954 Acked Read 8 Done Sent Done false
10772 3955 Acked Write 3 Done Sent Done false
10773 3956 Acked Read 6 Done Sent Done false
10774 3957 Acked Write 7 Done Sent Done false
10775 3958 Acked Read 10 Done Sent Done false
10776 3959 Acked Write 9 Done Sent Done false
10777 3960 Acked Write 9 Done Sent Done false
10778 3961 Acked Write 6 Done Sent Done false
10779 3962 Acked Read 10 Done Sent Done false
10780 3963 Acked Write 2 Done Sent Done false
10781 3964 Acked Read 7 Done Sent Done false
10782 3965 Acked Write 5 Done Sent Done false
10783 3966 Acked Flush 0 Done Sent Done false
10784 3967 Acked Write 6 Done Sent Done false
10785 3968 Acked Read 2 Done Sent Done false
10786 3969 Acked Write 7 Done Sent Done false
10787 3970 Acked Read 1 Done Sent Done false
10788 3971 Acked Write 3 Done Sent Done false
10789 3972 Acked Read 10 Done Sent Done false
10790 3973 Acked Write 1 Done Sent Done false
10791 3974 Acked Flush 0 Done Sent Done false
10792 3975 Acked Read 3 Done Sent Done false
10793 3976 Acked Read 5 Done Sent Done false
10794 3977 Acked Flush 0 Done Sent Done false
10795 3978 Acked Write 7 Done Sent Done false
10796 3979 Acked Read 2 Done Sent Done false
10797 3980 Acked Write 3 Done Sent Done false
10798 3981 Acked Write 4 Done Sent Done false
10799 3982 Acked Read 8 Done Sent Done false
10800 3983 Acked Flush 0 Done Sent Done false
10801 3984 Acked Read 1 Done Sent Done false
10802 3985 Acked Read 8 Done Sent Done false
10803 3986 Acked Read 10 Done Sent Done false
10804 3987 Acked Read 3 Done Sent Done false
10805 3988 Acked Write 5 Done Sent Done false
10806 3989 Acked Write 4 Done Sent Done false
10807 3990 Acked Write 10 Done Sent Done false
10808 3991 Acked Read 3 Done Sent Done false
10809 3992 Acked Write 5 Done Sent Done false
10810 3993 Acked Read 7 Done Sent Done false
10811 3994 Acked Flush 0 Done Sent Done false
10812 3995 Acked Write 4 Done Sent Done false
10813 3996 Acked Write 3 Done Sent Done false
10814 3997 Acked Read 10 Done Sent Done false
10815 3998 Acked Write 1 Done Sent Done false
10816 3999 Acked Write 3 Done Sent Done false
10817 4000 Acked Read 3 Done Sent Done false
10818 4001 Acked Read 9 Done Sent Done false
10819 4002 Acked Write 8 Done Sent Done false
10820 4003 Acked Read 1 Done Sent Done false
10821 4004 Acked Read 7 Done Sent Done false
10822 4005 Acked Read 10 Done Sent Done false
10823 4006 Acked Read 9 Done Sent Done false
10824 4007 Acked Write 6 Done Sent Done false
10825 4008 Acked Read 1 Done Sent Done false
10826 4009 Acked Write 10 Done Sent Done false
10827 4010 Acked Write 7 Done Sent Done false
10828 4011 Acked Write 7 Done Sent Done false
10829 4012 Acked Write 1 Done Sent Done false
10830 4013 Acked Flush 0 Done Sent Done false
10831 4014 Acked Flush 0 Done Sent Done false
10832 4015 Acked Write 5 Done Sent Done false
10833 4016 Acked Read 4 Done Sent Done false
10834 4017 Acked Read 1 Done Sent Done false
10835 4018 Acked Read 5 Done Sent Done false
10836 STATES DS:0 DS:1 DS:2 TOTAL
10837 Sent 0 301 0 301
10838 Done 301 0 301 602
10839 Skipped 0 0 0 0
10840 Error 0 0 0 0
10841 Last Flush: Some(JobId(4014)) Some(JobId(3717)) Some(JobId(4014))
10842 Downstairs last five completed: 3717 3716 3715 3714 3713
10843 Upstairs last five completed: 4018 4017 4016 4015 4014
10844 CLIENT: Up:0 ds:301 act:2
10845 [8820] got start action
10846 [8820] I am going to restart
10847 Starting downstairs at port 8820
10848 Make output file at "/tmp/dsc/downstairs-8820.txt"
10849 [8820][1] reports Starting
10850 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1215)
10851 [8820][1] reports Running
10852 ----------------------------------------------------------------
10853 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
10854 States: Active Offline Active
10855 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10856 3718 Acked Read 1 Done Sent Done false
10857 3719 Acked Read 3 Done Sent Done false
10858 3720 Acked Flush 0 Done Sent Done false
10859 3721 Acked Write 8 Done Sent Done false
10860 3722 Acked Read 6 Done Sent Done false
10861 3723 Acked Flush 0 Done Sent Done false
10862 3724 Acked Read 8 Done Sent Done false
10863 3725 Acked Read 2 Done Sent Done false
10864 3726 Acked Write 2 Done Sent Done false
10865 3727 Acked Flush 0 Done Sent Done false
10866 3728 Acked Read 5 Done Sent Done false
10867 3729 Acked Read 8 Done Sent Done false
10868 3730 Acked Read 7 Done Sent Done false
10869 3731 Acked Read 2 Done Sent Done false
10870 3732 Acked Read 6 Done Sent Done false
10871 3733 Acked Read 9 Done Sent Done false
10872 3734 Acked Read 8 Done Sent Done false
10873 3735 Acked Read 4 Done Sent Done false
10874 3736 Acked Flush 0 Done Sent Done false
10875 3737 Acked Read 4 Done Sent Done false
10876 3738 Acked Write 1 Done Sent Done false
10877 3739 Acked Read 5 Done Sent Done false
10878 3740 Acked Read 9 Done Sent Done false
10879 3741 Acked Read 3 Done Sent Done false
10880 3742 Acked Write 4 Done Sent Done false
10881 3743 Acked Write 7 Done Sent Done false
10882 3744 Acked Write 1 Done Sent Done false
10883 3745 Acked Read 7 Done Sent Done false
10884 3746 Acked Read 7 Done Sent Done false
10885 3747 Acked Read 7 Done Sent Done false
10886 3748 Acked Read 8 Done Sent Done false
10887 3749 Acked Read 7 Done Sent Done false
10888 3750 Acked Read 3 Done Sent Done false
10889 3751 Acked Write 6 Done Sent Done false
10890 3752 Acked Read 1 Done Sent Done false
10891 3753 Acked Write 2 Done Sent Done false
10892 3754 Acked Read 9 Done Sent Done false
10893 3755 Acked Read 1 Done Sent Done false
10894 3756 Acked Write 10 Done Sent Done false
10895 3757 Acked Write 8 Done Sent Done false
10896 3758 Acked Write 8 Done Sent Done false
10897 3759 Acked Read 2 Done Sent Done false
10898 3760 Acked Read 3 Done Sent Done false
10899 3761 Acked Read 6 Done Sent Done false
10900 3762 Acked Read 7 Done Sent Done false
10901 3763 Acked Write 8 Done Sent Done false
10902 3764 Acked Write 3 Done Sent Done false
10903 3765 Acked Write 4 Done Sent Done false
10904 3766 Acked Read 3 Done Sent Done false
10905 3767 Acked Write 8 Done Sent Done false
10906 3768 Acked Write 1 Done Sent Done false
10907 3769 Acked Read 1 Done Sent Done false
10908 3770 Acked Write 1 Done Sent Done false
10909 3771 Acked Read 5 Done Sent Done false
10910 3772 Acked Write 9 Done Sent Done false
10911 3773 Acked Read 9 Done Sent Done false
10912 3774 Acked Read 5 Done Sent Done false
10913 3775 Acked Write 3 Done Sent Done false
10914 3776 Acked Read 1 Done Sent Done false
10915 3777 Acked Write 3 Done Sent Done false
10916 3778 Acked Read 3 Done Sent Done false
10917 3779 Acked Read 3 Done Sent Done false
10918 3780 Acked Read 1 Done Sent Done false
10919 3781 Acked Write 7 Done Sent Done false
10920 3782 Acked Read 5 Done Sent Done false
10921 3783 Acked Write 2 Done Sent Done false
10922 3784 Acked Write 8 Done Sent Done false
10923 3785 Acked Write 8 Done Sent Done false
10924 3786 Acked Read 9 Done Sent Done false
10925 3787 Acked Read 3 Done Sent Done false
10926 3788 Acked Write 6 Done Sent Done false
10927 3789 Acked Write 2 Done Sent Done false
10928 3790 Acked Read 4 Done Sent Done false
10929 3791 Acked Write 9 Done Sent Done false
10930 3792 Acked Write 1 Done Sent Done false
10931 3793 Acked Flush 0 Done Sent Done false
10932 3794 Acked Read 5 Done Sent Done false
10933 3795 Acked Read 5 Done Sent Done false
10934 3796 Acked Read 1 Done Sent Done false
10935 3797 Acked Write 8 Done Sent Done false
10936 3798 Acked Read 10 Done Sent Done false
10937 3799 Acked Read 4 Done Sent Done false
10938 3800 Acked Flush 0 Done Sent Done false
10939 3801 Acked Flush 0 Done Sent Done false
10940 3802 Acked Read 6 Done Sent Done false
10941 3803 Acked Write 10 Done Sent Done false
10942 3804 Acked Write 2 Done Sent Done false
10943 3805 Acked Flush 0 Done Sent Done false
10944 3806 Acked Read 8 Done Sent Done false
10945 3807 Acked Read 5 Done Sent Done false
10946 3808 Acked Write 10 Done Sent Done false
10947 3809 Acked Write 1 Done Sent Done false
10948 3810 Acked Write 10 Done Sent Done false
10949 3811 Acked Write 4 Done Sent Done false
10950 3812 Acked Read 9 Done Sent Done false
10951 3813 Acked Read 9 Done Sent Done false
10952 3814 Acked Write 3 Done Sent Done false
10953 3815 Acked Write 4 Done Sent Done false
10954 3816 Acked Read 10 Done Sent Done false
10955 3817 Acked Write 8 Done Sent Done false
10956 3818 Acked Write 8 Done Sent Done false
10957 3819 Acked Write 8 Done Sent Done false
10958 3820 Acked Read 6 Done Sent Done false
10959 3821 Acked Write 9 Done Sent Done false
10960 3822 Acked Flush 0 Done Sent Done false
10961 3823 Acked Write 6 Done Sent Done false
10962 3824 Acked Write 7 Done Sent Done false
10963 3825 Acked Write 10 Done Sent Done false
10964 3826 Acked Flush 0 Done Sent Done false
10965 3827 Acked Write 5 Done Sent Done false
10966 3828 Acked Read 1 Done Sent Done false
10967 3829 Acked Read 9 Done Sent Done false
10968 3830 Acked Read 2 Done Sent Done false
10969 3831 Acked Read 7 Done Sent Done false
10970 3832 Acked Read 9 Done Sent Done false
10971 3833 Acked Read 8 Done Sent Done false
10972 3834 Acked Write 5 Done Sent Done false
10973 3835 Acked Read 4 Done Sent Done false
10974 3836 Acked Read 3 Done Sent Done false
10975 3837 Acked Read 1 Done Sent Done false
10976 3838 Acked Read 1 Done Sent Done false
10977 3839 Acked Read 10 Done Sent Done false
10978 3840 Acked Write 6 Done Sent Done false
10979 3841 Acked Write 1 Done Sent Done false
10980 3842 Acked Read 9 Done Sent Done false
10981 3843 Acked Read 7 Done Sent Done false
10982 3844 Acked Read 1 Done Sent Done false
10983 3845 Acked Write 5 Done Sent Done false
10984 3846 Acked Read 5 Done Sent Done false
10985 3847 Acked Write 1 Done Sent Done false
10986 3848 Acked Read 6 Done Sent Done false
10987 3849 Acked Write 1 Done Sent Done false
10988 3850 Acked Read 4 Done Sent Done false
10989 3851 Acked Read 4 Done Sent Done false
10990 3852 Acked Read 8 Done Sent Done false
10991 3853 Acked Write 4 Done Sent Done false
10992 3854 Acked Write 7 Done Sent Done false
10993 3855 Acked Read 4 Done Sent Done false
10994 3856 Acked Read 4 Done Sent Done false
10995 3857 Acked Read 10 Done Sent Done false
10996 3858 Acked Write 1 Done Sent Done false
10997 3859 Acked Write 2 Done Sent Done false
10998 3860 Acked Read 6 Done Sent Done false
10999 3861 Acked Read 1 Done Sent Done false
11000 3862 Acked Read 6 Done Sent Done false
11001 3863 Acked Read 1 Done Sent Done false
11002 3864 Acked Read 1 Done Sent Done false
11003 3865 Acked Read 9 Done Sent Done false
11004 3866 Acked Read 7 Done Sent Done false
11005 3867 Acked Flush 0 Done Sent Done false
11006 3868 Acked Write 10 Done Sent Done false
11007 3869 Acked Read 10 Done Sent Done false
11008 3870 Acked Write 4 Done Sent Done false
11009 3871 Acked Write 8 Done Sent Done false
11010 3872 Acked Write 5 Done Sent Done false
11011 3873 Acked Read 8 Done Sent Done false
11012 3874 Acked Read 1 Done Sent Done false
11013 3875 Acked Read 7 Done Sent Done false
11014 3876 Acked Read 4 Done Sent Done false
11015 3877 Acked Read 4 Done Sent Done false
11016 3878 Acked Read 1 Done Sent Done false
11017 3879 Acked Read 3 Done Sent Done false
11018 3880 Acked Write 1 Done Sent Done false
11019 3881 Acked Write 8 Done Sent Done false
11020 3882 Acked Write 8 Done Sent Done false
11021 3883 Acked Read 9 Done Sent Done false
11022 3884 Acked Read 4 Done Sent Done false
11023 3885 Acked Write 7 Done Sent Done false
11024 3886 Acked Write 3 Done Sent Done false
11025 3887 Acked Read 4 Done Sent Done false
11026 3888 Acked Flush 0 Done Sent Done false
11027 3889 Acked Write 4 Done Sent Done false
11028 3890 Acked Read 1 Done Sent Done false
11029 3891 Acked Read 10 Done Sent Done false
11030 3892 Acked Read 8 Done Sent Done false
11031 3893 Acked Read 3 Done Sent Done false
11032 3894 Acked Read 5 Done Sent Done false
11033 3895 Acked Write 9 Done Sent Done false
11034 3896 Acked Write 3 Done Sent Done false
11035 3897 Acked Write 5 Done Sent Done false
11036 3898 Acked Write 8 Done Sent Done false
11037 3899 Acked Write 6 Done Sent Done false
11038 3900 Acked Write 5 Done Sent Done false
11039 3901 Acked Read 8 Done Sent Done false
11040 3902 Acked Write 10 Done Sent Done false
11041 3903 Acked Flush 0 Done Sent Done false
11042 3904 Acked Read 10 Done Sent Done false
11043 3905 Acked Flush 0 Done Sent Done false
11044 3906 Acked Write 10 Done Sent Done false
11045 3907 Acked Read 7 Done Sent Done false
11046 3908 Acked Read 3 Done Sent Done false
11047 3909 Acked Write 3 Done Sent Done false
11048 3910 Acked Read 4 Done Sent Done false
11049 3911 Acked Write 8 Done Sent Done false
11050 3912 Acked Read 10 Done Sent Done false
11051 3913 Acked Read 5 Done Sent Done false
11052 3914 Acked Read 5 Done Sent Done false
11053 3915 Acked Read 3 Done Sent Done false
11054 3916 Acked Read 4 Done Sent Done false
11055 3917 Acked Read 7 Done Sent Done false
11056 3918 Acked Write 6 Done Sent Done false
11057 3919 Acked Read 4 Done Sent Done false
11058 3920 Acked Read 9 Done Sent Done false
11059 3921 Acked Read 2 Done Sent Done false
11060 3922 Acked Read 9 Done Sent Done false
11061 3923 Acked Read 1 Done Sent Done false
11062 3924 Acked Read 2 Done Sent Done false
11063 3925 Acked Read 6 Done Sent Done false
11064 3926 Acked Write 3 Done Sent Done false
11065 3927 Acked Write 3 Done Sent Done false
11066 3928 Acked Write 1 Done Sent Done false
11067 3929 Acked Write 10 Done Sent Done false
11068 3930 Acked Write 7 Done Sent Done false
11069 3931 Acked Read 8 Done Sent Done false
11070 3932 Acked Write 9 Done Sent Done false
11071 3933 Acked Read 9 Done Sent Done false
11072 3934 Acked Read 8 Done Sent Done false
11073 3935 Acked Read 4 Done Sent Done false
11074 3936 Acked Read 1 Done Sent Done false
11075 3937 Acked Write 9 Done Sent Done false
11076 3938 Acked Write 9 Done Sent Done false
11077 3939 Acked Write 2 Done Sent Done false
11078 3940 Acked Read 1 Done Sent Done false
11079 3941 Acked Flush 0 Done Sent Done false
11080 3942 Acked Read 2 Done Sent Done false
11081 3943 Acked Flush 0 Done Sent Done false
11082 3944 Acked Read 10 Done Sent Done false
11083 3945 Acked Write 4 Done Sent Done false
11084 3946 Acked Write 3 Done Sent Done false
11085 3947 Acked Read 7 Done Sent Done false
11086 3948 Acked Read 5 Done Sent Done false
11087 3949 Acked Write 7 Done Sent Done false
11088 3950 Acked Read 3 Done Sent Done false
11089 3951 Acked Write 5 Done Sent Done false
11090 3952 Acked Write 7 Done Sent Done false
11091 3953 Acked Read 3 Done Sent Done false
11092 3954 Acked Read 8 Done Sent Done false
11093 3955 Acked Write 3 Done Sent Done false
11094 3956 Acked Read 6 Done Sent Done false
11095 3957 Acked Write 7 Done Sent Done false
11096 3958 Acked Read 10 Done Sent Done false
11097 3959 Acked Write 9 Done Sent Done false
11098 3960 Acked Write 9 Done Sent Done false
11099 3961 Acked Write 6 Done Sent Done false
11100 3962 Acked Read 10 Done Sent Done false
11101 3963 Acked Write 2 Done Sent Done false
11102 3964 Acked Read 7 Done Sent Done false
11103 3965 Acked Write 5 Done Sent Done false
11104 3966 Acked Flush 0 Done Sent Done false
11105 3967 Acked Write 6 Done Sent Done false
11106 3968 Acked Read 2 Done Sent Done false
11107 3969 Acked Write 7 Done Sent Done false
11108 3970 Acked Read 1 Done Sent Done false
11109 3971 Acked Write 3 Done Sent Done false
11110 3972 Acked Read 10 Done Sent Done false
11111 3973 Acked Write 1 Done Sent Done false
11112 3974 Acked Flush 0 Done Sent Done false
11113 3975 Acked Read 3 Done Sent Done false
11114 3976 Acked Read 5 Done Sent Done false
11115 3977 Acked Flush 0 Done Sent Done false
11116 3978 Acked Write 7 Done Sent Done false
11117 3979 Acked Read 2 Done Sent Done false
11118 3980 Acked Write 3 Done Sent Done false
11119 3981 Acked Write 4 Done Sent Done false
11120 3982 Acked Read 8 Done Sent Done false
11121 3983 Acked Flush 0 Done Sent Done false
11122 3984 Acked Read 1 Done Sent Done false
11123 3985 Acked Read 8 Done Sent Done false
11124 3986 Acked Read 10 Done Sent Done false
11125 3987 Acked Read 3 Done Sent Done false
11126 3988 Acked Write 5 Done Sent Done false
11127 3989 Acked Write 4 Done Sent Done false
11128 3990 Acked Write 10 Done Sent Done false
11129 3991 Acked Read 3 Done Sent Done false
11130 3992 Acked Write 5 Done Sent Done false
11131 3993 Acked Read 7 Done Sent Done false
11132 3994 Acked Flush 0 Done Sent Done false
11133 3995 Acked Write 4 Done Sent Done false
11134 3996 Acked Write 3 Done Sent Done false
11135 3997 Acked Read 10 Done Sent Done false
11136 3998 Acked Write 1 Done Sent Done false
11137 3999 Acked Write 3 Done Sent Done false
11138 4000 Acked Read 3 Done Sent Done false
11139 4001 Acked Read 9 Done Sent Done false
11140 4002 Acked Write 8 Done Sent Done false
11141 4003 Acked Read 1 Done Sent Done false
11142 4004 Acked Read 7 Done Sent Done false
11143 4005 Acked Read 10 Done Sent Done false
11144 4006 Acked Read 9 Done Sent Done false
11145 4007 Acked Write 6 Done Sent Done false
11146 4008 Acked Read 1 Done Sent Done false
11147 4009 Acked Write 10 Done Sent Done false
11148 4010 Acked Write 7 Done Sent Done false
11149 4011 Acked Write 7 Done Sent Done false
11150 4012 Acked Write 1 Done Sent Done false
11151 4013 Acked Flush 0 Done Sent Done false
11152 4014 Acked Flush 0 Done Sent Done false
11153 4015 Acked Write 5 Done Sent Done false
11154 4016 Acked Read 4 Done Sent Done false
11155 4017 Acked Read 1 Done Sent Done false
11156 4018 Acked Read 5 Done Sent Done false
11157 4019 Acked Flush 0 Done Sent Done false
11158 STATES DS:0 DS:1 DS:2 TOTAL
11159 Sent 0 302 0 302
11160 Done 302 0 302 604
11161 Skipped 0 0 0 0
11162 Error 0 0 0 0
11163 Last Flush: Some(JobId(4019)) Some(JobId(3717)) Some(JobId(4019))
11164 Downstairs last five completed: 3717 3716 3715 3714 3713
11165 Upstairs last five completed: 4019 4018 4017 4016 4015
11166 CLIENT: Up:0 ds:302 act:2
111672025-01-23T17:40:03.910ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
111682025-01-23T17:40:03.910ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65306) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
111692025-01-23T17:40:03.911ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
111702025-01-23T17:40:03.912ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
111712025-01-23T17:40:03.912ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
111722025-01-23T17:40:03.912ZINFOcrucible: send last flush ID to this DS: Some(JobId(3717)) = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
111732025-01-23T17:40:03.912ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
111742025-01-23T17:40:03.912ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(3717)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
111752025-01-23T17:40:03.912ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(3717)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
11176 ----------------------------------------------------------------
11177 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11178 Downstairs last five completed: 4019 4018 4017 4016 4015
11179 Upstairs last five completed: 4019 4018 4017 4016 4015
11180 CLIENT: Up:0 ds:0 act:3
11181 Replay: All jobs finished
11182 Main task has work to do, go find it
11183 got dsc Stop(2)
11184 stop 2
11185 Jan 23 17:40:06.160 INFO request completed[8830] Got stop action so:false kr:false
11186 , latency_us: 344, response_code: 204, uri: /stop/cid/2, method: POST, req_id: a2ba9ae7-4be3-4985-8827-34f34f757707, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
11187 Jan 23 17:40:06.161 INFO request completed, latency_us: 157, response_code{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:40:06.161292173Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
11188 : 200, uri: /state/cid/2, method: GET, req_id: d15ef1c5-6202-4308-8cf9-323f4c1a56e1, remote_addr: 127.0.0.1{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:40:06.161428509Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
11189 :42015, local_addr: 127.0.0.1:9998
111902025-01-23T17:40:06.161ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
111912025-01-23T17:40:06.161ZINFOcrucible: client task is exiting = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
11192 [8830] Exited with: ExitStatus(unix_wait_status(9))
11193 [8830][2] reports Exit
11194 Jan 23 17:40:10.163 INFO request completed, latency_us: 194, response_code: 200, uri: /state/cid/2, method: GET, req_id: fc90b9f6-6246-4851-a1a4-265cbb20df0f, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
11195 001/300 Read block 3656 len 28672
11196 002/300 Read block 1197 len 8192
11197 003/300 Write block 230 len 16384 data: 2 2 2 2
11198 004/300 Flush
11199 005/300 Read block 1997 len 16384
11200 006/300 Write block 3514 len 8192 data: 4 4
11201 007/300 Write block 4135 len 40960 data: 3 3 3 3 3 5 5 4 4 4
11202 008/300 Write block 1292 len 4096 data: 5
11203 009/300 Read block 4050 len 32768
11204 010/300 Write block 753 len 4096 data: 2
11205 011/300 Read block 891 len 28672
11206 012/300 Read block 4407 len 16384
11207 013/300 Read block 4660 len 20480
11208 014/300 Write block 1956 len 4096 data: 2
11209 015/300 Read block 245 len 24576
11210 016/300 Read block 4538 len 40960
11211 017/300 Read block 1965 len 28672
11212 018/300 Read block 3894 len 16384
11213 019/300 Read block 4161 len 28672
11214 020/300 Write block 4274 len 16384 data: 7 6 4 4
11215 021/300 Read block 4689 len 12288
11216 022/300 Read block 690 len 4096
11217 023/300 Write block 98 len 12288 data: 2 3 4
11218 024/300 Read block 1005 len 40960
11219 025/300 Flush
11220 026/300 Flush
11221 027/300 Read block 2380 len 12288
11222 028/300 Read block 2096 len 36864
11223 029/300 Read block 897 len 4096
11224 030/300 Read block 2329 len 8192
11225 031/300 Read block 648 len 16384
11226 032/300 Read block 2253 len 12288
11227 033/300 Read block 643 len 20480
11228 034/300 Read block 4925 len 8192
11229 035/300 Read block 4744 len 20480
11230 036/300 Flush
11231 037/300 Write block 3421 len 12288 data: 4 4 3
11232 038/300 Read block 2886 len 12288
11233 039/300 Write block 909 len 40960 data: 2 2 2 3 3 2 2 2 2 2
11234 040/300 Write block 4857 len 32768 data: 3 3 3 3 3 3 2 2
11235 041/300 Write block 2765 len 20480 data: 3 3 3 4 3
11236 042/300 Flush
11237 043/300 Read block 4848 len 40960
11238 044/300 Write block 2832 len 40960 data: 5 4 3 3 3 5 4 4 4 5
11239 045/300 Read block 4047 len 4096
11240 046/300 Flush
11241 047/300 Read block 1412 len 28672
11242 048/300 Write block 3208 len 8192 data: 3 3
11243 049/300 Read block 2318 len 20480
11244 050/300 Read block 3346 len 8192
11245 051/300 Write block 1764 len 12288 data: 4 2 2
11246 052/300 Read block 431 len 8192
11247 053/300 Read block 2677 len 20480
11248 054/300 Flush
11249 055/300 Read block 658 len 16384
11250 056/300 Read block 2238 len 12288
11251 057/300 Write block 3516 len 36864 data: 4 4 4 2 2 3 4 6 5
11252 058/300 Flush
11253 059/300 Read block 2424 len 4096
11254 060/300 Write block 1185 len 20480 data: 2 2 2 2 3
11255 061/300 Write block 4361 len 12288 data: 4 4 4
11256 062/300 Read block 1523 len 20480
11257 063/300 Write block 2750 len 24576 data: 5 5 4 3 3 3
11258 064/300 Write block 2688 len 4096 data: 3
11259 065/300 Read block 4985 len 28672
11260 066/300 Read block 3892 len 32768
11261 067/300 Read block 2211 len 20480
11262 068/300 Flush
11263 069/300 Write block 1298 len 32768 data: 4 4 4 5 3 3 4 4
11264 070/300 Write block 518 len 20480 data: 2 2 2 3 4
11265 071/300 Write block 460 len 28672 data: 4 7 7 6 6 6 6
11266 072/300 Read block 172 len 12288
11267 073/300 Read block 2096 len 32768
11268 074/300 Write block 1734 len 20480 data: 4 3 3 4 4
11269 075/300 Write block 4966 len 28672 data: 2 3 3 3 3 5 5
11270 076/300 Write block 1920 len 24576 data: 5 4 5 5 7 5
11271 077/300 Read block 4516 len 4096
11272 078/300 Write block 346 len 4096 data: 3
11273 079/300 Read block 4738 len 40960
11274 080/300 Write block 633 len 24576 data: 3 3 3 3 3 3
11275 081/300 Write block 595 len 16384 data: 4 3 2 2
11276 082/300 Read block 619 len 20480
11277 083/300 Write block 4409 len 32768 data: 2 2 2 3 3 3 3 2
11278 084/300 Write block 1308 len 24576 data: 5 5 6 6 4 4
11279 085/300 Write block 746 len 20480 data: 3 3 3 3 3
11280 086/300 Write block 3176 len 24576 data: 3 2 2 2 2 2
11281 087/300 Write block 2651 len 12288 data: 3 3 2
11282 088/300 Read block 4357 len 16384
11283 089/300 Write block 2158 len 28672 data: 2 3 3 3 3 3 2
11284 090/300 Read block 102 len 4096
11285 091/300 Read block 1035 len 4096
11286 092/300 Read block 2478 len 4096
11287 093/300 Flush
11288 094/300 Read block 198 len 36864
11289 095/300 Read block 1904 len 36864
11290 096/300 Write block 1431 len 32768 data: 2 2 2 2 2 2 2 2
11291 097/300 Flush
11292 098/300 Write block 3253 len 28672 data: 3 3 3 2 2 2 2
11293 099/300 Read block 1586 len 16384
11294 100/300 Read block 4867 len 36864
11295 101/300 Read block 2425 len 40960
11296 102/300 Write block 3930 len 16384 data: 2 3 3 4
11297 103/300 Write block 4706 len 32768 data: 3 3 4 4 4 2 2 3
11298 104/300 Write block 3991 len 40960 data: 2 3 3 3 3 3 3 3 3 3
11299 105/300 Read block 3558 len 16384
11300 106/300 Read block 2613 len 4096
11301 107/300 Read block 406 len 20480
11302 108/300 Read block 3055 len 12288
11303 109/300 Write block 3868 len 8192 data: 2 3
11304 110/300 Read block 4566 len 24576
11305 111/300 Flush
11306 112/300 Write block 4311 len 24576 data: 2 2 2 2 2 3
11307 113/300 Read block 4083 len 32768
11308 114/300 Read block 21 len 28672
11309 115/300 Read block 2452 len 12288
11310 116/300 Write block 264 len 12288 data: 3 3 3
11311 117/300 Write block 2634 len 12288 data: 3 3 3
11312 118/300 Write block 1282 len 28672 data: 3 2 3 3 3 3 3
11313 119/300 Read block 2079 len 8192
11314 120/300 Write block 1994 len 4096 data: 4
11315 121/300 Write block 2222 len 28672 data: 4 4 3 2 4 4 5
11316 122/300 Flush
11317 123/300 Read block 1822 len 40960
11318 124/300 Write block 1908 len 12288 data: 8 7 5
11319 125/300 Write block 1939 len 24576 data: 3 3 4 4 4 4
11320 126/300 Read block 54 len 8192
11321 127/300 Flush
11322 128/300 Write block 1870 len 36864 data: 5 4 3 3 4 3 3 3 3
11323 129/300 Write block 157 len 20480 data: 5 4 4 5 4
11324 130/300 Write block 1021 len 24576 data: 4 4 4 5 5 5
11325 131/300 Write block 3397 len 4096 data: 4
11326 132/300 Write block 2839 len 20480 data: 5 5 6 5 5
11327 133/300 Read block 3575 len 8192
11328 134/300 Write block 513 len 4096 data: 3
11329 135/300 Write block 4426 len 24576 data: 4 3 2 2 2 3
11330 136/300 Write block 1758 len 28672 data: 4 4 4 3 3 4 5
11331 137/300 Read block 3593 len 24576
11332 138/300 Read block 1809 len 12288
11333 139/300 Write block 2041 len 4096 data: 2
11334 140/300 Write block 3653 len 32768 data: 3 4 5 4 4 3 3 4
11335 141/300 Write block 869 len 36864 data: 2 2 2 2 3 3 2 3 3
11336 142/300 Write block 1004 len 32768 data: 3 4 4 4 3 4 4 4
11337 143/300 Flush
11338 144/300 Write block 869 len 8192 data: 3 3
11339 145/300 Flush
11340 146/300 Read block 2452 len 12288
11341 147/300 Write block 2799 len 16384 data: 3 3 3 3
11342 148/300 Read block 3903 len 12288
11343 149/300 Read block 1841 len 12288
11344 150/300 Read block 4405 len 40960
11345 151/300 Write block 4907 len 32768 data: 2 2 2 3 3 3 3 4
11346 152/300 Write block 3551 len 20480 data: 5 5 4 4 3
11347 153/300 Read block 4171 len 8192
11348 154/300 Write block 1698 len 12288 data: 3 3 3
11349 155/300 Read block 2332 len 28672
11350 156/300 Read block 1926 len 4096
11351 157/300 Write block 4446 len 24576 data: 2 2 2 2 2 2
11352 158/300 Read block 2644 len 16384
11353 159/300 Read block 512 len 16384
11354 160/300 Write block 2490 len 8192 data: 3 4
11355 161/300 Read block 3703 len 28672
11356 162/300 Write block 1193 len 32768 data: 4 2 2 3 3 3 3 4
11357 163/300 Write block 3410 len 28672 data: 3 3 3 3 4 4 5
11358 164/300 Write block 1923 len 12288 data: 6 8 6
11359 165/300 Write block 1147 len 32768 data: 5 3 3 2 2 2 2 2
11360 166/300 Write block 1221 len 8192 data: 3 3
11361 167/300 Read block 3148 len 40960
11362 168/300 Flush
11363 169/300 Write block 2720 len 32768 data: 2 2 2 2 2 2 3 3
11364 170/300 Write block 4000 len 8192 data: 4 2
11365 171/300 Write block 3229 len 24576 data: 2 2 2 2 3 4
11366 172/300 Read block 1358 len 36864
11367 173/300 Write block 1239 len 16384 data: 3 3 2 2
11368 174/300 Write block 141 len 24576 data: 6 5 5 2 2 2
11369 175/300 Read block 756 len 8192
11370 176/300 Flush
11371 177/300 Read block 4356 len 16384
11372 178/300 Write block 3162 len 16384 data: 4 3 3 2
11373 179/300 Write block 961 len 8192 data: 2 3
11374 180/300 Write block 102 len 28672 data: 5 4 4 3 4 5 6
11375 181/300 Write block 437 len 12288 data: 3 3 3
11376 182/300 Flush
11377 183/300 Write block 812 len 24576 data: 4 3 3 3 3 2
11378 184/300 Read block 4909 len 36864
11379 185/300 Read block 1690 len 16384
11380 186/300 Write block 1218 len 12288 data: 2 3 3
11381 187/300 Read block 3223 len 12288
11382 188/300 Write block 3460 len 4096 data: 6
11383 189/300 Read block 844 len 24576
11384 190/300 Read block 1982 len 36864
11385 191/300 Read block 2123 len 24576
11386 192/300 Read block 1159 len 24576
11387 193/300 Read block 4120 len 36864
11388 194/300 Read block 640 len 40960
11389 195/300 Write block 464 len 32768 data: 7 7 7 5 4 3 3 2
11390 196/300 Flush
11391 197/300 Read block 3666 len 28672
11392 198/300 Read block 4026 len 20480
11393 199/300 Read block 765 len 24576
11394 200/300 Write block 3145 len 4096 data: 4
11395 201/300 Read block 3798 len 32768
11396 202/300 Flush
11397 203/300 Read block 705 len 4096
11398 204/300 Read block 2043 len 28672
11399 205/300 Read block 4538 len 4096
11400 206/300 Write block 3706 len 8192 data: 5 5
11401 207/300 Write block 4584 len 12288 data: 3 3 3
11402 208/300 Flush
11403 209/300 Write block 4759 len 40960 data: 3 4 5 4 4 4 3 3 4 4
11404 210/300 Read block 4783 len 24576
11405 211/300 Read block 3998 len 24576
11406 212/300 Read block 3190 len 36864
11407 213/300 Write block 3180 len 20480 data: 3 3 2 5 5
11408 214/300 Write block 4623 len 4096 data: 6
11409 215/300 Write block 2811 len 36864 data: 2 2 2 2 3 3 3 3 4
11410 216/300 Write block 3903 len 4096 data: 2
11411 217/300 Write block 1561 len 32768 data: 2 2 2 2 3 3 3 3
11412 218/300 Write block 2878 len 4096 data: 3
11413 219/300 Read block 107 len 40960
11414 220/300 Read block 3308 len 28672
11415 221/300 Write block 244 len 32768 data: 2 2 2 2 2 2 2 2
11416 222/300 Read block 3373 len 4096
11417 223/300 Write block 3289 len 20480 data: 2 3 3 5 5
11418 224/300 Write block 1843 len 28672 data: 2 2 2 2 2 2 3
11419 225/300 Write block 4685 len 12288 data: 5 5 4
11420 226/300 Read block 3760 len 16384
11421 227/300 Write block 890 len 4096 data: 3
11422 228/300 Flush
11423 229/300 Write block 2830 len 24576 data: 4 4 6 5 4 4
11424 230/300 Write block 2216 len 20480 data: 4 4 4 4 3
11425 231/300 Write block 3157 len 20480 data: 3 3 3 3 4
11426 232/300 Read block 3671 len 32768
11427 233/300 Read block 3951 len 4096
11428 234/300 Read block 4153 len 20480
11429 235/300 Read block 2886 len 32768
11430 236/300 Read block 418 len 36864
11431 237/300 Write block 4495 len 24576 data: 3 4 4 4 4 3
11432 238/300 Read block 582 len 36864
11433 239/300 Write block 1644 len 24576 data: 4 3 3 5 5 5
11434 240/300 Write block 2270 len 20480 data: 3 4 3 3 3
11435 241/300 Write block 4777 len 4096 data: 3
11436 242/300 Write block 3933 len 20480 data: 5 4 4 3 3
11437 243/300 Write block 1329 len 32768 data: 3 4 6 6 5 4 4 3
11438 244/300 Read block 4601 len 4096
11439 245/300 Write block 3857 len 4096 data: 5
11440 246/300 Write block 3499 len 28672 data: 3 4 5 5 5 5 5
11441 247/300 Write block 2134 len 32768 data: 3 3 3 3 3 3 3 2
11442 248/300 Read block 1502 len 28672
11443 249/300 Write block 3724 len 24576 data: 4 2 3 2 2 2
11444 250/300 Write block 4730 len 40960 data: 4 4 4 3 3 4 4 3 3 3
11445 251/300 Write block 1067 len 24576 data: 3 3 3 3 4 4
11446 252/300 Read block 2453 len 4096
11447 253/300 Read block 318 len 8192
11448 254/300 Write block 768 len 16384 data: 3 3 3 4
11449 255/300 Write block 139 len 8192 data: 5 6
11450 256/300 Read block 4225 len 20480
11451 257/300 Read block 2011 len 16384
11452 258/300 Write block 1502 len 32768 data: 3 3 4 5 6 6 5 5
11453 259/300 Read block 4325 len 40960
11454 260/300 Read block 1554 len 36864
11455 261/300 Write block 1721 len 4096 data: 2
11456 262/300 Flush
11457 263/300 Write block 1593 len 28672 data: 2 2 2 2 2 2 2
11458 264/300 Write block 3383 len 4096 data: 3
11459 265/300 Read block 2888 len 8192
11460 266/300 Read block 2602 len 36864
11461 267/300 Write block 3305 len 20480 data: 3 3 5 5 5
11462 268/300 Read block 3848 len 12288
11463 269/300 Write block 3611 len 4096 data: 6
11464 270/300 Read block 1199 len 16384
11465 271/300 Write block 2574 len 12288 data: 4 4 5
11466 272/300 Write block 3026 len 12288 data: 3 3 2
11467 273/300 Read block 4455 len 12288
11468 274/300 Write block 2262 len 12288 data: 5 4 4
11469 275/300 Write block 1619 len 16384 data: 7 6 5 3
11470 276/300 Read block 1535 len 20480
11471 277/300 Read block 1318 len 16384
11472 278/300 Write block 114 len 12288 data: 6 6 7
11473 279/300 Read block 1400 len 4096
11474 280/300 Read block 4588 len 40960
11475 281/300 Read block 396 len 8192
11476 282/300 Read block 3728 len 12288
11477 283/300 Read block 3171 len 28672
11478 284/300 Flush
11479 285/300 Flush
11480 286/300 Read block 2340 len 4096
11481 287/300 Read block 814 len 8192
11482 288/300 Write block 3423 len 16384 data: 4 3 4 4
11483 289/300 Write block 2174 len 32768 data: 3 3 2 2 2 2 2 3
11484 290/300 Write block 4044 len 24576 data: 2 2 3 3 3 4
11485 291/300 Write block 1088 len 16384 data: 3 3 3 3
11486 292/300 Write block 4696 len 36864 data: 4 3 3 3 3 3 3 3 3
11487 293/300 Read block 1246 len 4096
11488 294/300 Read block 1956 len 36864
11489 295/300 Read block 463 len 28672
11490 296/300 Read block 1176 len 20480
11491 297/300 Read block 4731 len 16384
11492 298/300 Write block 4501 len 20480 data: 2 2 2 3 3
11493 299/300 Write block 3011 len 28672 data: 4 5 6 6 5 5 5
11494 300/300 Read block 325 len 28672
11495 301/300 Write block 4793 len 16384 data: 2 2 2 2
11496 Main task has work to do, go find it
11497 got dsc Start(2)
11498 start 2
11499 Jan 23 17:40:10.385 INFO request completed, latency_us: 182, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1fe7848e-5dcb-475c-adbf-d231a61a3128, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
11500 Replay: started 2, returned:Ok(())
11501 ----------------------------------------------------------------
11502 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11503 States: Active Active Offline
11504 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11505 4020 Acked Read 7 Done Done Sent false
11506 4021 Acked Read 2 Done Done Sent false
11507 4022 Acked Write 4 Done Done Sent false
11508 4023 Acked Flush 0 Done Done Sent false
11509 4024 Acked Read 4 Done Done Sent false
11510 4025 Acked Write 2 Done Done Sent false
11511 4026 Acked Write 10 Done Done Sent false
11512 4027 Acked Write 1 Done Done Sent false
11513 4028 Acked Read 8 Done Done Sent false
11514 4029 Acked Write 1 Done Done Sent false
11515 4030 Acked Read 7 Done Done Sent false
11516 4031 Acked Read 4 Done Done Sent false
11517 4032 Acked Read 5 Done Done Sent false
11518 4033 Acked Write 1 Done Done Sent false
11519 4034 Acked Read 6 Done Done Sent false
11520 4035 Acked Read 10 Done Done Sent false
11521 4036 Acked Read 7 Done Done Sent false
11522 4037 Acked Read 4 Done Done Sent false
11523 4038 Acked Read 7 Done Done Sent false
11524 4039 Acked Write 4 Done Done Sent false
11525 4040 Acked Read 3 Done Done Sent false
11526 4041 Acked Read 1 Done Done Sent false
11527 4042 Acked Write 3 Done Done Sent false
11528 4043 Acked Read 10 Done Done Sent false
11529 4044 Acked Flush 0 Done Done Sent false
11530 4045 Acked Flush 0 Done Done Sent false
11531 4046 Acked Read 3 Done Done Sent false
11532 4047 Acked Read 9 Done Done Sent false
11533 4048 Acked Read 1 Done Done Sent false
11534 4049 Acked Read 2 Done Done Sent false
11535 4050 Acked Read 4 Done Done Sent false
11536 4051 Acked Read 3 Done Done Sent false
11537 4052 Acked Read 5 Done Done Sent false
11538 4053 Acked Read 2 Done Done Sent false
11539 4054 Acked Read 5 Done Done Sent false
11540 4055 Acked Flush 0 Done Done Sent false
11541 4056 Acked Write 3 Done Done Sent false
11542 4057 Acked Read 3 Done Done Sent false
11543 4058 Acked Write 10 Done Done Sent false
11544 4059 Acked Write 8 Done Done Sent false
11545 4060 Acked Write 5 Done Done Sent false
11546 4061 Acked Flush 0 Done Done Sent false
11547 4062 Acked Read 10 Done Done Sent false
11548 4063 Acked Write 10 Done Done Sent false
11549 4064 Acked Read 1 Done Done Sent false
11550 4065 Acked Flush 0 Done Done Sent false
11551 4066 Acked Read 7 Done Done Sent false
11552 4067 Acked Write 2 Done Done Sent false
11553 4068 Acked Read 5 Done Done Sent false
11554 4069 Acked Read 2 Done Done Sent false
11555 4070 Acked Write 3 Done Done Sent false
11556 4071 Acked Read 2 Done Done Sent false
11557 4072 Acked Read 5 Done Done Sent false
11558 4073 Acked Flush 0 Done Done Sent false
11559 4074 Acked Read 4 Done Done Sent false
11560 4075 Acked Read 3 Done Done Sent false
11561 4076 Acked Write 9 Done Done Sent false
11562 4077 Acked Flush 0 Done Done Sent false
11563 4078 Acked Read 1 Done Done Sent false
11564 4079 Acked Write 5 Done Done Sent false
11565 4080 Acked Write 3 Done Done Sent false
11566 4081 Acked Read 5 Done Done Sent false
11567 4082 Acked Write 6 Done Done Sent false
11568 4083 Acked Write 1 Done Done Sent false
11569 4084 Acked Read 7 Done Done Sent false
11570 4085 Acked Read 8 Done Done Sent false
11571 4086 Acked Read 5 Done Done Sent false
11572 4087 Acked Flush 0 Done Done Sent false
11573 4088 Acked Write 8 Done Done Sent false
11574 4089 Acked Write 5 Done Done Sent false
11575 4090 Acked Write 7 Done Done Sent false
11576 4091 Acked Read 3 Done Done Sent false
11577 4092 Acked Read 8 Done Done Sent false
11578 4093 Acked Write 5 Done Done Sent false
11579 4094 Acked Write 7 Done Done Sent false
11580 4095 Acked Write 6 Done Done Sent false
11581 4096 Acked Read 1 Done Done Sent false
11582 4097 Acked Write 1 Done Done Sent false
11583 4098 Acked Read 10 Done Done Sent false
11584 4099 Acked Write 6 Done Done Sent false
11585 4100 Acked Write 4 Done Done Sent false
11586 4101 Acked Read 5 Done Done Sent false
11587 4102 Acked Write 8 Done Done Sent false
11588 4103 Acked Write 6 Done Done Sent false
11589 4104 Acked Write 5 Done Done Sent false
11590 4105 Acked Write 6 Done Done Sent false
11591 4106 Acked Write 3 Done Done Sent false
11592 4107 Acked Read 4 Done Done Sent false
11593 4108 Acked Write 7 Done Done Sent false
11594 4109 Acked Read 1 Done Done Sent false
11595 4110 Acked Read 1 Done Done Sent false
11596 4111 Acked Read 1 Done Done Sent false
11597 4112 Acked Flush 0 Done Done Sent false
11598 4113 Acked Read 9 Done Done Sent false
11599 4114 Acked Read 9 Done Done Sent false
11600 4115 Acked Write 8 Done Done Sent false
11601 4116 Acked Flush 0 Done Done Sent false
11602 4117 Acked Write 7 Done Done Sent false
11603 4118 Acked Read 4 Done Done Sent false
11604 4119 Acked Read 9 Done Done Sent false
11605 4120 Acked Read 10 Done Done Sent false
11606 4121 Acked Write 4 Done Done Sent false
11607 4122 Acked Write 8 Done Done Sent false
11608 4123 Acked Write 10 Done Done Sent false
11609 4124 Acked Read 4 Done Done Sent false
11610 4125 Acked Read 1 Done Done Sent false
11611 4126 Acked Read 5 Done Done Sent false
11612 4127 Acked Read 3 Done Done Sent false
11613 4128 Acked Write 2 Done Done Sent false
11614 4129 Acked Read 6 Done Done Sent false
11615 4130 Acked Flush 0 Done Done Sent false
11616 4131 Acked Write 6 Done Done Sent false
11617 4132 Acked Read 8 Done Done Sent false
11618 4133 Acked Read 7 Done Done Sent false
11619 4134 Acked Read 3 Done Done Sent false
11620 4135 Acked Write 3 Done Done Sent false
11621 4136 Acked Write 3 Done Done Sent false
11622 4137 Acked Write 7 Done Done Sent false
11623 4138 Acked Read 2 Done Done Sent false
11624 4139 Acked Write 1 Done Done Sent false
11625 4140 Acked Write 7 Done Done Sent false
11626 4141 Acked Flush 0 Done Done Sent false
11627 4142 Acked Read 10 Done Done Sent false
11628 4143 Acked Write 3 Done Done Sent false
11629 4144 Acked Write 6 Done Done Sent false
11630 4145 Acked Read 2 Done Done Sent false
11631 4146 Acked Flush 0 Done Done Sent false
11632 4147 Acked Write 9 Done Done Sent false
11633 4148 Acked Write 5 Done Done Sent false
11634 4149 Acked Write 6 Done Done Sent false
11635 4150 Acked Write 1 Done Done Sent false
11636 4151 Acked Write 5 Done Done Sent false
11637 4152 Acked Read 2 Done Done Sent false
11638 4153 Acked Write 1 Done Done Sent false
11639 4154 Acked Write 6 Done Done Sent false
11640 4155 Acked Write 7 Done Done Sent false
11641 4156 Acked Read 6 Done Done Sent false
11642 4157 Acked Read 3 Done Done Sent false
11643 4158 Acked Write 1 Done Done Sent false
11644 4159 Acked Write 8 Done Done Sent false
11645 4160 Acked Write 9 Done Done Sent false
11646 4161 Acked Write 8 Done Done Sent false
11647 4162 Acked Flush 0 Done Done Sent false
11648 4163 Acked Write 2 Done Done Sent false
11649 4164 Acked Flush 0 Done Done Sent false
11650 4165 Acked Read 3 Done Done Sent false
11651 4166 Acked Write 4 Done Done Sent false
11652 4167 Acked Read 3 Done Done Sent false
11653 4168 Acked Read 3 Done Done Sent false
11654 4169 Acked Read 10 Done Done Sent false
11655 4170 Acked Write 8 Done Done Sent false
11656 4171 Acked Write 5 Done Done Sent false
11657 4172 Acked Read 2 Done Done Sent false
11658 4173 Acked Write 3 Done Done Sent false
11659 4174 Acked Read 7 Done Done Sent false
11660 4175 Acked Read 1 Done Done Sent false
11661 4176 Acked Write 6 Done Done Sent false
11662 4177 Acked Read 4 Done Done Sent false
11663 4178 Acked Read 4 Done Done Sent false
11664 4179 Acked Write 2 Done Done Sent false
11665 4180 Acked Read 7 Done Done Sent false
11666 4181 Acked Write 8 Done Done Sent false
11667 4182 Acked Write 7 Done Done Sent false
11668 4183 Acked Write 3 Done Done Sent false
11669 4184 Acked Write 8 Done Done Sent false
11670 4185 Acked Write 2 Done Done Sent false
11671 4186 Acked Read 10 Done Done Sent false
11672 4187 Acked Flush 0 Done Done Sent false
11673 4188 Acked Write 8 Done Done Sent false
11674 4189 Acked Write 2 Done Done Sent false
11675 4190 Acked Write 6 Done Done Sent false
11676 4191 Acked Read 9 Done Done Sent false
11677 4192 Acked Write 4 Done Done Sent false
11678 4193 Acked Write 6 Done Done Sent false
11679 4194 Acked Read 2 Done Done Sent false
11680 4195 Acked Flush 0 Done Done Sent false
11681 4196 Acked Read 4 Done Done Sent false
11682 4197 Acked Write 4 Done Done Sent false
11683 4198 Acked Write 2 Done Done Sent false
11684 4199 Acked Write 7 Done Done Sent false
11685 4200 Acked Write 3 Done Done Sent false
11686 4201 Acked Flush 0 Done Done Sent false
11687 4202 Acked Write 6 Done Done Sent false
11688 4203 Acked Read 9 Done Done Sent false
11689 4204 Acked Read 4 Done Done Sent false
11690 4205 Acked Write 3 Done Done Sent false
11691 4206 Acked Read 3 Done Done Sent false
11692 4207 Acked Write 1 Done Done Sent false
11693 4208 Acked Read 6 Done Done Sent false
11694 4209 Acked Read 9 Done Done Sent false
11695 4210 Acked Read 6 Done Done Sent false
11696 4211 Acked Read 6 Done Done Sent false
11697 4212 Acked Read 9 Done Done Sent false
11698 4213 Acked Read 10 Done Done Sent false
11699 4214 Acked Write 8 Done Done Sent false
11700 4215 Acked Flush 0 Done Done Sent false
11701 4216 Acked Read 7 Done Done Sent false
11702 4217 Acked Read 5 Done Done Sent false
11703 4218 Acked Read 6 Done Done Sent false
11704 4219 Acked Write 1 Done Done Sent false
11705 4220 Acked Read 8 Done Done Sent false
11706 4221 Acked Flush 0 Done Done Sent false
11707 4222 Acked Read 1 Done Done Sent false
11708 4223 Acked Read 7 Done Done Sent false
11709 4224 Acked Read 1 Done Done Sent false
11710 4225 Acked Write 2 Done Done Sent false
11711 4226 Acked Write 3 Done Done Sent false
11712 4227 Acked Flush 0 Done Done Sent false
11713 4228 Acked Write 10 Done Done Sent false
11714 4229 Acked Read 6 Done Done Sent false
11715 4230 Acked Read 6 Done Done Sent false
11716 4231 Acked Read 9 Done Done Sent false
11717 4232 Acked Write 5 Done Done Sent false
11718 4233 Acked Write 1 Done Done Sent false
11719 4234 Acked Write 9 Done Done Sent false
11720 4235 Acked Write 1 Done Done Sent false
11721 4236 Acked Write 8 Done Done Sent false
11722 4237 Acked Write 1 Done Done Sent false
11723 4238 Acked Read 10 Done Done Sent false
11724 4239 Acked Read 7 Done Done Sent false
11725 4240 Acked Write 8 Done Done Sent false
11726 4241 Acked Read 1 Done Done Sent false
11727 4242 Acked Write 5 Done Done Sent false
11728 4243 Acked Write 7 Done Done Sent false
11729 4244 Acked Write 3 Done Done Sent false
11730 4245 Acked Read 4 Done Done Sent false
11731 4246 Acked Write 1 Done Done Sent false
11732 4247 Acked Flush 0 Done Done Sent false
11733 4248 Acked Write 6 Done Done Sent false
11734 4249 Acked Write 5 Done Done Sent false
11735 4250 Acked Write 5 Done Done Sent false
11736 4251 Acked Read 8 Done Done Sent false
11737 4252 Acked Read 1 Done Done Sent false
11738 4253 Acked Read 5 Done Done Sent false
11739 4254 Acked Read 8 Done Done Sent false
11740 4255 Acked Read 9 Done Done Sent false
11741 4256 Acked Write 6 Done Done Sent false
11742 4257 Acked Read 9 Done Done Sent false
11743 4258 Acked Write 6 Done Done Sent false
11744 4259 Acked Write 5 Done Done Sent false
11745 4260 Acked Write 1 Done Done Sent false
11746 4261 Acked Write 5 Done Done Sent false
11747 4262 Acked Write 8 Done Done Sent false
11748 4263 Acked Read 1 Done Done Sent false
11749 4264 Acked Write 1 Done Done Sent false
11750 4265 Acked Write 7 Done Done Sent false
11751 4266 Acked Write 8 Done Done Sent false
11752 4267 Acked Read 7 Done Done Sent false
11753 4268 Acked Write 6 Done Done Sent false
11754 4269 Acked Write 10 Done Done Sent false
11755 4270 Acked Write 6 Done Done Sent false
11756 4271 Acked Read 1 Done Done Sent false
11757 4272 Acked Read 2 Done Done Sent false
11758 4273 Acked Write 4 Done Done Sent false
11759 4274 Acked Write 2 Done Done Sent false
11760 4275 Acked Read 5 Done Done Sent false
11761 4276 Acked Read 4 Done Done Sent false
11762 4277 Acked Write 8 Done Done Sent false
11763 4278 Acked Read 10 Done Done Sent false
11764 4279 Acked Read 9 Done Done Sent false
11765 4280 Acked Write 1 Done Done Sent false
11766 4281 Acked Flush 0 Done Done Sent false
11767 4282 Acked Write 7 Done Done Sent false
11768 4283 Acked Write 1 Done Done Sent false
11769 4284 Acked Read 2 Done Done Sent false
11770 4285 Acked Read 9 Done Done Sent false
11771 4286 Acked Write 5 Done Done Sent false
11772 4287 Acked Read 3 Done Done Sent false
11773 4288 Acked Write 1 Done Done Sent false
11774 4289 Acked Read 4 Done Done Sent false
11775 4290 Acked Write 3 Done Done Sent false
11776 4291 Acked Write 3 Done Done Sent false
11777 4292 Acked Read 3 Done Done Sent false
11778 4293 Acked Write 3 Done Done Sent false
11779 4294 Acked Write 4 Done Done Sent false
11780 4295 Acked Read 5 Done Done Sent false
11781 4296 Acked Read 4 Done Done Sent false
11782 4297 Acked Write 3 Done Done Sent false
11783 4298 Acked Read 1 Done Done Sent false
11784 4299 Acked Read 10 Done Done Sent false
11785 4300 Acked Read 2 Done Done Sent false
11786 4301 Acked Read 3 Done Done Sent false
11787 4302 Acked Read 7 Done Done Sent false
11788 4303 Acked Flush 0 Done Done Sent false
11789 4304 Acked Flush 0 Done Done Sent false
11790 4305 Acked Read 1 Done Done Sent false
11791 4306 Acked Read 2 Done Done Sent false
11792 4307 Acked Write 4 Done Done Sent false
11793 4308 Acked Write 8 Done Done Sent false
11794 4309 Acked Write 6 Done Done Sent false
11795 4310 Acked Write 4 Done Done Sent false
11796 4311 Acked Write 9 Done Done Sent false
11797 4312 Acked Read 1 Done Done Sent false
11798 4313 Acked Read 9 Done Done Sent false
11799 4314 Acked Read 7 Done Done Sent false
11800 4315 Acked Read 5 Done Done Sent false
11801 4316 Acked Read 4 Done Done Sent false
11802 4317 Acked Write 5 Done Done Sent false
11803 4318 Acked Write 7 Done Done Sent false
11804 4319 Acked Read 7 Done Done Sent false
11805 4320 Acked Write 4 Done Done Sent false
11806 STATES DS:0 DS:1 DS:2 TOTAL
11807 Sent 0 0 301 301
11808 Done 301 301 0 602
11809 Skipped 0 0 0 0
11810 Error 0 0 0 0
11811 Last Flush: Some(JobId(4304)) Some(JobId(4304)) Some(JobId(4019))
11812 Downstairs last five completed: 4019 4018 4017 4016 4015
11813 Upstairs last five completed: 4320 4319 4318 4317 4316
11814 CLIENT: Up:0 ds:301 act:2
11815 [8830] got start action
11816 [8830] I am going to restart
11817 Starting downstairs at port 8830
11818 Make output file at "/tmp/dsc/downstairs-8830.txt"
11819 [8830][2] reports Starting
11820 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1216)
11821 [8830][2] reports Running
11822 ----------------------------------------------------------------
11823 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
11824 States: Active Active Offline
11825 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11826 4020 Acked Read 7 Done Done Sent false
11827 4021 Acked Read 2 Done Done Sent false
11828 4022 Acked Write 4 Done Done Sent false
11829 4023 Acked Flush 0 Done Done Sent false
11830 4024 Acked Read 4 Done Done Sent false
11831 4025 Acked Write 2 Done Done Sent false
11832 4026 Acked Write 10 Done Done Sent false
11833 4027 Acked Write 1 Done Done Sent false
11834 4028 Acked Read 8 Done Done Sent false
11835 4029 Acked Write 1 Done Done Sent false
11836 4030 Acked Read 7 Done Done Sent false
11837 4031 Acked Read 4 Done Done Sent false
11838 4032 Acked Read 5 Done Done Sent false
11839 4033 Acked Write 1 Done Done Sent false
11840 4034 Acked Read 6 Done Done Sent false
11841 4035 Acked Read 10 Done Done Sent false
11842 4036 Acked Read 7 Done Done Sent false
11843 4037 Acked Read 4 Done Done Sent false
11844 4038 Acked Read 7 Done Done Sent false
11845 4039 Acked Write 4 Done Done Sent false
11846 4040 Acked Read 3 Done Done Sent false
11847 4041 Acked Read 1 Done Done Sent false
11848 4042 Acked Write 3 Done Done Sent false
11849 4043 Acked Read 10 Done Done Sent false
11850 4044 Acked Flush 0 Done Done Sent false
11851 4045 Acked Flush 0 Done Done Sent false
11852 4046 Acked Read 3 Done Done Sent false
11853 4047 Acked Read 9 Done Done Sent false
11854 4048 Acked Read 1 Done Done Sent false
11855 4049 Acked Read 2 Done Done Sent false
11856 4050 Acked Read 4 Done Done Sent false
11857 4051 Acked Read 3 Done Done Sent false
11858 4052 Acked Read 5 Done Done Sent false
11859 4053 Acked Read 2 Done Done Sent false
11860 4054 Acked Read 5 Done Done Sent false
11861 4055 Acked Flush 0 Done Done Sent false
11862 4056 Acked Write 3 Done Done Sent false
11863 4057 Acked Read 3 Done Done Sent false
11864 4058 Acked Write 10 Done Done Sent false
11865 4059 Acked Write 8 Done Done Sent false
11866 4060 Acked Write 5 Done Done Sent false
11867 4061 Acked Flush 0 Done Done Sent false
11868 4062 Acked Read 10 Done Done Sent false
11869 4063 Acked Write 10 Done Done Sent false
11870 4064 Acked Read 1 Done Done Sent false
11871 4065 Acked Flush 0 Done Done Sent false
11872 4066 Acked Read 7 Done Done Sent false
11873 4067 Acked Write 2 Done Done Sent false
11874 4068 Acked Read 5 Done Done Sent false
11875 4069 Acked Read 2 Done Done Sent false
11876 4070 Acked Write 3 Done Done Sent false
11877 4071 Acked Read 2 Done Done Sent false
11878 4072 Acked Read 5 Done Done Sent false
11879 4073 Acked Flush 0 Done Done Sent false
11880 4074 Acked Read 4 Done Done Sent false
11881 4075 Acked Read 3 Done Done Sent false
11882 4076 Acked Write 9 Done Done Sent false
11883 4077 Acked Flush 0 Done Done Sent false
11884 4078 Acked Read 1 Done Done Sent false
11885 4079 Acked Write 5 Done Done Sent false
11886 4080 Acked Write 3 Done Done Sent false
11887 4081 Acked Read 5 Done Done Sent false
11888 4082 Acked Write 6 Done Done Sent false
11889 4083 Acked Write 1 Done Done Sent false
11890 4084 Acked Read 7 Done Done Sent false
11891 4085 Acked Read 8 Done Done Sent false
11892 4086 Acked Read 5 Done Done Sent false
11893 4087 Acked Flush 0 Done Done Sent false
11894 4088 Acked Write 8 Done Done Sent false
11895 4089 Acked Write 5 Done Done Sent false
11896 4090 Acked Write 7 Done Done Sent false
11897 4091 Acked Read 3 Done Done Sent false
11898 4092 Acked Read 8 Done Done Sent false
11899 4093 Acked Write 5 Done Done Sent false
11900 4094 Acked Write 7 Done Done Sent false
11901 4095 Acked Write 6 Done Done Sent false
11902 4096 Acked Read 1 Done Done Sent false
11903 4097 Acked Write 1 Done Done Sent false
11904 4098 Acked Read 10 Done Done Sent false
11905 4099 Acked Write 6 Done Done Sent false
11906 4100 Acked Write 4 Done Done Sent false
11907 4101 Acked Read 5 Done Done Sent false
11908 4102 Acked Write 8 Done Done Sent false
11909 4103 Acked Write 6 Done Done Sent false
11910 4104 Acked Write 5 Done Done Sent false
11911 4105 Acked Write 6 Done Done Sent false
11912 4106 Acked Write 3 Done Done Sent false
11913 4107 Acked Read 4 Done Done Sent false
11914 4108 Acked Write 7 Done Done Sent false
11915 4109 Acked Read 1 Done Done Sent false
11916 4110 Acked Read 1 Done Done Sent false
11917 4111 Acked Read 1 Done Done Sent false
11918 4112 Acked Flush 0 Done Done Sent false
11919 4113 Acked Read 9 Done Done Sent false
11920 4114 Acked Read 9 Done Done Sent false
11921 4115 Acked Write 8 Done Done Sent false
11922 4116 Acked Flush 0 Done Done Sent false
11923 4117 Acked Write 7 Done Done Sent false
11924 4118 Acked Read 4 Done Done Sent false
11925 4119 Acked Read 9 Done Done Sent false
11926 4120 Acked Read 10 Done Done Sent false
11927 4121 Acked Write 4 Done Done Sent false
11928 4122 Acked Write 8 Done Done Sent false
11929 4123 Acked Write 10 Done Done Sent false
11930 4124 Acked Read 4 Done Done Sent false
11931 4125 Acked Read 1 Done Done Sent false
11932 4126 Acked Read 5 Done Done Sent false
11933 4127 Acked Read 3 Done Done Sent false
11934 4128 Acked Write 2 Done Done Sent false
11935 4129 Acked Read 6 Done Done Sent false
11936 4130 Acked Flush 0 Done Done Sent false
11937 4131 Acked Write 6 Done Done Sent false
11938 4132 Acked Read 8 Done Done Sent false
11939 4133 Acked Read 7 Done Done Sent false
11940 4134 Acked Read 3 Done Done Sent false
11941 4135 Acked Write 3 Done Done Sent false
11942 4136 Acked Write 3 Done Done Sent false
11943 4137 Acked Write 7 Done Done Sent false
11944 4138 Acked Read 2 Done Done Sent false
11945 4139 Acked Write 1 Done Done Sent false
11946 4140 Acked Write 7 Done Done Sent false
11947 4141 Acked Flush 0 Done Done Sent false
11948 4142 Acked Read 10 Done Done Sent false
11949 4143 Acked Write 3 Done Done Sent false
11950 4144 Acked Write 6 Done Done Sent false
11951 4145 Acked Read 2 Done Done Sent false
11952 4146 Acked Flush 0 Done Done Sent false
11953 4147 Acked Write 9 Done Done Sent false
11954 4148 Acked Write 5 Done Done Sent false
11955 4149 Acked Write 6 Done Done Sent false
11956 4150 Acked Write 1 Done Done Sent false
11957 4151 Acked Write 5 Done Done Sent false
11958 4152 Acked Read 2 Done Done Sent false
11959 4153 Acked Write 1 Done Done Sent false
11960 4154 Acked Write 6 Done Done Sent false
11961 4155 Acked Write 7 Done Done Sent false
11962 4156 Acked Read 6 Done Done Sent false
11963 4157 Acked Read 3 Done Done Sent false
11964 4158 Acked Write 1 Done Done Sent false
11965 4159 Acked Write 8 Done Done Sent false
11966 4160 Acked Write 9 Done Done Sent false
11967 4161 Acked Write 8 Done Done Sent false
11968 4162 Acked Flush 0 Done Done Sent false
11969 4163 Acked Write 2 Done Done Sent false
11970 4164 Acked Flush 0 Done Done Sent false
11971 4165 Acked Read 3 Done Done Sent false
11972 4166 Acked Write 4 Done Done Sent false
11973 4167 Acked Read 3 Done Done Sent false
11974 4168 Acked Read 3 Done Done Sent false
11975 4169 Acked Read 10 Done Done Sent false
11976 4170 Acked Write 8 Done Done Sent false
11977 4171 Acked Write 5 Done Done Sent false
11978 4172 Acked Read 2 Done Done Sent false
11979 4173 Acked Write 3 Done Done Sent false
11980 4174 Acked Read 7 Done Done Sent false
11981 4175 Acked Read 1 Done Done Sent false
11982 4176 Acked Write 6 Done Done Sent false
11983 4177 Acked Read 4 Done Done Sent false
11984 4178 Acked Read 4 Done Done Sent false
11985 4179 Acked Write 2 Done Done Sent false
11986 4180 Acked Read 7 Done Done Sent false
11987 4181 Acked Write 8 Done Done Sent false
11988 4182 Acked Write 7 Done Done Sent false
11989 4183 Acked Write 3 Done Done Sent false
11990 4184 Acked Write 8 Done Done Sent false
11991 4185 Acked Write 2 Done Done Sent false
11992 4186 Acked Read 10 Done Done Sent false
11993 4187 Acked Flush 0 Done Done Sent false
11994 4188 Acked Write 8 Done Done Sent false
11995 4189 Acked Write 2 Done Done Sent false
11996 4190 Acked Write 6 Done Done Sent false
11997 4191 Acked Read 9 Done Done Sent false
11998 4192 Acked Write 4 Done Done Sent false
11999 4193 Acked Write 6 Done Done Sent false
12000 4194 Acked Read 2 Done Done Sent false
12001 4195 Acked Flush 0 Done Done Sent false
12002 4196 Acked Read 4 Done Done Sent false
12003 4197 Acked Write 4 Done Done Sent false
12004 4198 Acked Write 2 Done Done Sent false
12005 4199 Acked Write 7 Done Done Sent false
12006 4200 Acked Write 3 Done Done Sent false
12007 4201 Acked Flush 0 Done Done Sent false
12008 4202 Acked Write 6 Done Done Sent false
12009 4203 Acked Read 9 Done Done Sent false
12010 4204 Acked Read 4 Done Done Sent false
12011 4205 Acked Write 3 Done Done Sent false
12012 4206 Acked Read 3 Done Done Sent false
12013 4207 Acked Write 1 Done Done Sent false
12014 4208 Acked Read 6 Done Done Sent false
12015 4209 Acked Read 9 Done Done Sent false
12016 4210 Acked Read 6 Done Done Sent false
12017 4211 Acked Read 6 Done Done Sent false
12018 4212 Acked Read 9 Done Done Sent false
12019 4213 Acked Read 10 Done Done Sent false
12020 4214 Acked Write 8 Done Done Sent false
12021 4215 Acked Flush 0 Done Done Sent false
12022 4216 Acked Read 7 Done Done Sent false
12023 4217 Acked Read 5 Done Done Sent false
12024 4218 Acked Read 6 Done Done Sent false
12025 4219 Acked Write 1 Done Done Sent false
12026 4220 Acked Read 8 Done Done Sent false
12027 4221 Acked Flush 0 Done Done Sent false
12028 4222 Acked Read 1 Done Done Sent false
12029 4223 Acked Read 7 Done Done Sent false
12030 4224 Acked Read 1 Done Done Sent false
12031 4225 Acked Write 2 Done Done Sent false
12032 4226 Acked Write 3 Done Done Sent false
12033 4227 Acked Flush 0 Done Done Sent false
12034 4228 Acked Write 10 Done Done Sent false
12035 4229 Acked Read 6 Done Done Sent false
12036 4230 Acked Read 6 Done Done Sent false
12037 4231 Acked Read 9 Done Done Sent false
12038 4232 Acked Write 5 Done Done Sent false
12039 4233 Acked Write 1 Done Done Sent false
12040 4234 Acked Write 9 Done Done Sent false
12041 4235 Acked Write 1 Done Done Sent false
12042 4236 Acked Write 8 Done Done Sent false
12043 4237 Acked Write 1 Done Done Sent false
12044 4238 Acked Read 10 Done Done Sent false
12045 4239 Acked Read 7 Done Done Sent false
12046 4240 Acked Write 8 Done Done Sent false
12047 4241 Acked Read 1 Done Done Sent false
12048 4242 Acked Write 5 Done Done Sent false
12049 4243 Acked Write 7 Done Done Sent false
12050 4244 Acked Write 3 Done Done Sent false
12051 4245 Acked Read 4 Done Done Sent false
12052 4246 Acked Write 1 Done Done Sent false
12053 4247 Acked Flush 0 Done Done Sent false
12054 4248 Acked Write 6 Done Done Sent false
12055 4249 Acked Write 5 Done Done Sent false
12056 4250 Acked Write 5 Done Done Sent false
12057 4251 Acked Read 8 Done Done Sent false
12058 4252 Acked Read 1 Done Done Sent false
12059 4253 Acked Read 5 Done Done Sent false
12060 4254 Acked Read 8 Done Done Sent false
12061 4255 Acked Read 9 Done Done Sent false
12062 4256 Acked Write 6 Done Done Sent false
12063 4257 Acked Read 9 Done Done Sent false
12064 4258 Acked Write 6 Done Done Sent false
12065 4259 Acked Write 5 Done Done Sent false
12066 4260 Acked Write 1 Done Done Sent false
12067 4261 Acked Write 5 Done Done Sent false
12068 4262 Acked Write 8 Done Done Sent false
12069 4263 Acked Read 1 Done Done Sent false
12070 4264 Acked Write 1 Done Done Sent false
12071 4265 Acked Write 7 Done Done Sent false
12072 4266 Acked Write 8 Done Done Sent false
12073 4267 Acked Read 7 Done Done Sent false
12074 4268 Acked Write 6 Done Done Sent false
12075 4269 Acked Write 10 Done Done Sent false
12076 4270 Acked Write 6 Done Done Sent false
12077 4271 Acked Read 1 Done Done Sent false
12078 4272 Acked Read 2 Done Done Sent false
12079 4273 Acked Write 4 Done Done Sent false
12080 4274 Acked Write 2 Done Done Sent false
12081 4275 Acked Read 5 Done Done Sent false
12082 4276 Acked Read 4 Done Done Sent false
12083 4277 Acked Write 8 Done Done Sent false
12084 4278 Acked Read 10 Done Done Sent false
12085 4279 Acked Read 9 Done Done Sent false
12086 4280 Acked Write 1 Done Done Sent false
12087 4281 Acked Flush 0 Done Done Sent false
12088 4282 Acked Write 7 Done Done Sent false
12089 4283 Acked Write 1 Done Done Sent false
12090 4284 Acked Read 2 Done Done Sent false
12091 4285 Acked Read 9 Done Done Sent false
12092 4286 Acked Write 5 Done Done Sent false
12093 4287 Acked Read 3 Done Done Sent false
12094 4288 Acked Write 1 Done Done Sent false
12095 4289 Acked Read 4 Done Done Sent false
12096 4290 Acked Write 3 Done Done Sent false
12097 4291 Acked Write 3 Done Done Sent false
12098 4292 Acked Read 3 Done Done Sent false
12099 4293 Acked Write 3 Done Done Sent false
12100 4294 Acked Write 4 Done Done Sent false
12101 4295 Acked Read 5 Done Done Sent false
12102 4296 Acked Read 4 Done Done Sent false
12103 4297 Acked Write 3 Done Done Sent false
12104 4298 Acked Read 1 Done Done Sent false
12105 4299 Acked Read 10 Done Done Sent false
12106 4300 Acked Read 2 Done Done Sent false
12107 4301 Acked Read 3 Done Done Sent false
12108 4302 Acked Read 7 Done Done Sent false
12109 4303 Acked Flush 0 Done Done Sent false
12110 4304 Acked Flush 0 Done Done Sent false
12111 4305 Acked Read 1 Done Done Sent false
12112 4306 Acked Read 2 Done Done Sent false
12113 4307 Acked Write 4 Done Done Sent false
12114 4308 Acked Write 8 Done Done Sent false
12115 4309 Acked Write 6 Done Done Sent false
12116 4310 Acked Write 4 Done Done Sent false
12117 4311 Acked Write 9 Done Done Sent false
12118 4312 Acked Read 1 Done Done Sent false
12119 4313 Acked Read 9 Done Done Sent false
12120 4314 Acked Read 7 Done Done Sent false
12121 4315 Acked Read 5 Done Done Sent false
12122 4316 Acked Read 4 Done Done Sent false
12123 4317 Acked Write 5 Done Done Sent false
12124 4318 Acked Write 7 Done Done Sent false
12125 4319 Acked Read 7 Done Done Sent false
12126 4320 Acked Write 4 Done Done Sent false
12127 4321 Acked Flush 0 Done Done Sent false
12128 STATES DS:0 DS:1 DS:2 TOTAL
12129 Sent 0 0 302 302
12130 Done 302 302 0 604
12131 Skipped 0 0 0 0
12132 Error 0 0 0 0
12133 Last Flush: Some(JobId(4321)) Some(JobId(4321)) Some(JobId(4019))
12134 Downstairs last five completed: 4019 4018 4017 4016 4015
12135 Upstairs last five completed: 4321 4320 4319 4318 4317
12136 CLIENT: Up:0 ds:302 act:2
121372025-01-23T17:40:16.163ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
121382025-01-23T17:40:16.163ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55509) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
121392025-01-23T17:40:16.164ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
121402025-01-23T17:40:16.165ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
121412025-01-23T17:40:16.165ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
121422025-01-23T17:40:16.165ZINFOcrucible: send last flush ID to this DS: Some(JobId(4019)) = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
121432025-01-23T17:40:16.165ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
121442025-01-23T17:40:16.165ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(4019)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
121452025-01-23T17:40:16.165ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(4019)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
12146 ----------------------------------------------------------------
12147 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12148 Downstairs last five completed: 4321 4320 4319 4318 4317
12149 Upstairs last five completed: 4321 4320 4319 4318 4317
12150 CLIENT: Up:0 ds:0 act:3
12151 Replay: All jobs finished
12152 Main task has work to do, go find it
12153 got dsc Stop(2)
12154 stop 2
12155 [8830] Got stop action so:false kr:false
12156 Jan 23 17:40:18.390 INFO request completed, latency_us: 380, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 20d9bc7f-fad7-450d-a7bf-f216b78c0f07, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
121572025-01-23T17:40:18.391ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
121582025-01-23T17:40:18.391ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
121592025-01-23T17:40:18.391ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
12160 Jan 23 17:40:18.391 INFO request completed, latency_us: 128, response_code: 200, uri: /state/cid/2, method: GET, req_id: 51c26e42-d255-456f-ba3b-12fdb3141f01, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T17:40:18.39155309Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
12161 
12162 [8830] Exited with: ExitStatus(unix_wait_status(9))
12163 [8830][2] reports Exit
12164 Jan 23 17:40:22.393 INFO request completed, latency_us: 272, response_code: 200, uri: /state/cid/2, method: GET, req_id: cbdb1696-290d-4421-9cbd-4c3b69773c40, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
12165 001/300 Flush
12166 002/300 Read block 264 len 36864
12167 003/300 Read block 713 len 40960
12168 004/300 Write block 746 len 32768 data: 4 4 4 4 4 2 2 3
12169 005/300 Write block 3418 len 40960 data: 4 4 5 5 5 5 4 5 5 4
12170 006/300 Write block 1162 len 12288 data: 8 8 8
12171 007/300 Read block 4006 len 8192
12172 008/300 Flush
12173 009/300 Write block 1144 len 4096 data: 6
12174 010/300 Write block 3162 len 28672 data: 5 4 4 3 2 2 2
12175 011/300 Write block 3019 len 4096 data: 4
12176 012/300 Read block 3521 len 12288
12177 013/300 Write block 1688 len 32768 data: 5 4 4 4 3 4 3 3
12178 014/300 Write block 4738 len 40960 data: 4 4 2 2 2 3 3 3 3 3
12179 015/300 Read block 4015 len 20480
12180 016/300 Read block 720 len 8192
12181 017/300 Write block 4234 len 40960 data: 4 3 3 3 2 3 3 3 3 3
12182 018/300 Write block 1400 len 4096 data: 3
12183 019/300 Read block 1202 len 4096
12184 020/300 Write block 1818 len 8192 data: 3 3
12185 021/300 Read block 4616 len 28672
12186 022/300 Read block 4795 len 32768
12187 023/300 Write block 2420 len 40960 data: 3 3 3 3 3 3 3 2 2 2
12188 024/300 Write block 3139 len 36864 data: 3 4 3 4 4 4 5 4 3
12189 025/300 Read block 1261 len 28672
12190 026/300 Read block 3851 len 28672
12191 027/300 Write block 1836 len 4096 data: 5
12192 028/300 Write block 1945 len 16384 data: 3 3 3 3
12193 029/300 Read block 3879 len 8192
12194 030/300 Flush
12195 031/300 Read block 2052 len 4096
12196 032/300 Read block 4238 len 36864
12197 033/300 Read block 1571 len 28672
12198 034/300 Read block 3150 len 12288
12199 035/300 Read block 816 len 40960
12200 036/300 Read block 1698 len 12288
12201 037/300 Read block 1102 len 4096
12202 038/300 Write block 1320 len 12288 data: 2 2 2
12203 039/300 Read block 3531 len 16384
12204 040/300 Write block 1268 len 12288 data: 2 2 3
12205 041/300 Write block 3229 len 36864 data: 3 3 3 3 4 5 4 4 4
12206 042/300 Write block 3005 len 36864 data: 2 2 2 3 4 4 5 6 7
12207 043/300 Write block 3734 len 40960 data: 2 2 2 2 2 3 3 3 2 2
12208 044/300 Write block 4556 len 28672 data: 6 5 5 2 2 2 2
12209 045/300 Read block 874 len 20480
12210 046/300 Write block 582 len 36864 data: 2 3 5 5 4 4 3 3 3
12211 047/300 Write block 38 len 12288 data: 3 3 3
12212 048/300 Write block 4563 len 20480 data: 3 5 5 6 6
12213 049/300 Read block 1211 len 16384
12214 050/300 Write block 2121 len 20480 data: 2 2 4 3 3
12215 051/300 Write block 96 len 12288 data: 2 2 3
12216 052/300 Write block 368 len 40960 data: 6 6 6 6 6 6 6 6 5 3
12217 053/300 Read block 1918 len 28672
12218 054/300 Read block 3464 len 24576
12219 055/300 Read block 4719 len 4096
12220 056/300 Write block 4928 len 20480 data: 4 3 2 3 3
12221 057/300 Write block 2798 len 12288 data: 3 4 4
12222 058/300 Write block 1086 len 20480 data: 3 3 4 4 4
12223 059/300 Write block 1614 len 40960 data: 3 4 5 5 6 8 7 6 4 5
12224 060/300 Write block 2152 len 32768 data: 4 3 3 3 2 2 3 4
12225 061/300 Write block 195 len 8192 data: 4 4
12226 062/300 Read block 3831 len 16384
12227 063/300 Write block 3223 len 16384 data: 2 3 4 4
12228 064/300 Write block 1865 len 8192 data: 6 8
12229 065/300 Read block 4933 len 28672
12230 066/300 Flush
12231 067/300 Flush
12232 068/300 Read block 674 len 20480
12233 069/300 Read block 2266 len 36864
12234 070/300 Read block 4481 len 36864
12235 071/300 Read block 4164 len 36864
12236 072/300 Write block 2371 len 8192 data: 4 4
12237 073/300 Write block 287 len 40960 data: 5 6 6 6 3 3 5 6 7 5
12238 074/300 Read block 1564 len 8192
12239 075/300 Write block 4909 len 4096 data: 3
12240 076/300 Write block 1834 len 32768 data: 5 5 6 5 3 3 3 2
12241 077/300 Read block 256 len 4096
12242 078/300 Read block 816 len 8192
12243 079/300 Write block 1775 len 32768 data: 2 3 3 3 3 2 2 3
12244 080/300 Write block 3041 len 16384 data: 4 3 3 4
12245 081/300 Read block 3318 len 28672
12246 082/300 Write block 1236 len 16384 data: 5 5 3 4
12247 083/300 Write block 3462 len 8192 data: 6 4
12248 084/300 Read block 1666 len 40960
12249 085/300 Read block 4971 len 4096
12250 086/300 Write block 3276 len 28672 data: 2 2 2 2 2 2 2
12251 087/300 Write block 3158 len 40960 data: 4 4 4 5 6 5 5 4 3 3
12252 088/300 Read block 2707 len 28672
12253 089/300 Write block 820 len 40960 data: 4 4 5 5 6 6 6 5 4 4
12254 090/300 Read block 2342 len 28672
12255 091/300 Read block 2103 len 12288
12256 092/300 Flush
12257 093/300 Read block 563 len 40960
12258 094/300 Write block 1827 len 4096 data: 5
12259 095/300 Write block 3437 len 40960 data: 3 2 3 3 4 4 4 3 2 2
12260 096/300 Write block 4561 len 40960 data: 3 3 4 6 6 7 7 6 6 7
12261 097/300 Write block 1559 len 16384 data: 3 3 3 3
12262 098/300 Flush
12263 099/300 Read block 3727 len 12288
12264 100/300 Read block 1644 len 28672
12265 101/300 Read block 4886 len 12288
12266 102/300 Write block 749 len 24576 data: 5 5 3 3 4 3
12267 103/300 Write block 3329 len 12288 data: 3 4 4
12268 104/300 Write block 2973 len 20480 data: 4 4 4 3 3
12269 105/300 Read block 4560 len 12288
12270 106/300 Read block 1114 len 16384
12271 107/300 Write block 4256 len 8192 data: 4 4
12272 108/300 Write block 1582 len 8192 data: 2 2
12273 109/300 Write block 4282 len 28672 data: 6 6 7 6 4 3 4
12274 110/300 Write block 2390 len 8192 data: 3 3
12275 111/300 Write block 4034 len 12288 data: 5 5 5
12276 112/300 Read block 4495 len 32768
12277 113/300 Read block 857 len 36864
12278 114/300 Flush
12279 115/300 Read block 4963 len 4096
12280 116/300 Write block 4864 len 28672 data: 3 2 2 2 2 4 5
12281 117/300 Write block 4630 len 40960 data: 4 3 3 2 2 2 2 3 3 3
12282 118/300 Write block 353 len 24576 data: 2 2 2 3 3 3
12283 119/300 Read block 2528 len 24576
12284 120/300 Read block 2524 len 32768
12285 121/300 Read block 3506 len 36864
12286 122/300 Flush
12287 123/300 Read block 2924 len 16384
12288 124/300 Flush
12289 125/300 Flush
12290 126/300 Read block 3456 len 40960
12291 127/300 Read block 4509 len 24576
12292 128/300 Read block 4990 len 28672
12293 129/300 Write block 2146 len 16384 data: 4 4 4 4
12294 130/300 Read block 2856 len 20480
12295 131/300 Write block 4971 len 16384 data: 6 6 4 3
12296 132/300 Flush
12297 133/300 Read block 1228 len 32768
12298 134/300 Flush
12299 135/300 Write block 773 len 24576 data: 5 5 4 4 4 5
12300 136/300 Read block 2322 len 32768
12301 137/300 Write block 3907 len 28672 data: 3 3 3 3 4 4 4
12302 138/300 Read block 12 len 32768
12303 139/300 Write block 4720 len 8192 data: 2 3
12304 140/300 Write block 3599 len 4096 data: 5
12305 141/300 Write block 4642 len 36864 data: 4 3 3 3 3 3 3 3 3
12306 142/300 Write block 4943 len 16384 data: 3 3 3 3
12307 143/300 Read block 2464 len 36864
12308 144/300 Write block 3353 len 4096 data: 3
12309 145/300 Read block 2639 len 8192
12310 146/300 Read block 4116 len 36864
12311 147/300 Write block 1499 len 32768 data: 2 3 3 4 4 5 6 7
12312 148/300 Read block 4698 len 32768
12313 149/300 Write block 4348 len 8192 data: 3 4
12314 150/300 Read block 3581 len 36864
12315 151/300 Write block 3760 len 4096 data: 3
12316 152/300 Write block 4895 len 24576 data: 4 3 3 2 3 3
12317 153/300 Read block 1593 len 16384
12318 154/300 Read block 611 len 4096
12319 155/300 Write block 2991 len 36864 data: 4 5 4 2 2 4 4 3 4
12320 156/300 Flush
12321 157/300 Read block 2636 len 24576
12322 158/300 Write block 4862 len 36864 data: 4 3 4 3 3 3 3 5 6
12323 159/300 Write block 551 len 36864 data: 3 3 4 4 4 6 5 3 2
12324 160/300 Read block 4447 len 12288
12325 161/300 Write block 2408 len 36864 data: 2 2 2 2 2 5 4 4 4
12326 162/300 Read block 4689 len 24576
12327 163/300 Read block 1898 len 20480
12328 164/300 Read block 3867 len 16384
12329 165/300 Read block 1765 len 24576
12330 166/300 Read block 512 len 4096
12331 167/300 Read block 113 len 28672
12332 168/300 Read block 4535 len 36864
12333 169/300 Write block 4623 len 32768 data: 7 4 4 4 4 5 5 5
12334 170/300 Flush
12335 171/300 Write block 4341 len 4096 data: 4
12336 172/300 Read block 2466 len 36864
12337 173/300 Write block 4315 len 40960 data: 3 4 2 3 3 3 3 3 3 3
12338 174/300 Write block 3397 len 24576 data: 5 4 4 3 4 4
12339 175/300 Read block 2063 len 28672
12340 176/300 Write block 2717 len 20480 data: 3 3 3 3 3
12341 177/300 Write block 1204 len 24576 data: 3 3 3 3 3 3
12342 178/300 Read block 732 len 28672
12343 179/300 Read block 1977 len 4096
12344 180/300 Flush
12345 181/300 Read block 1175 len 20480
12346 182/300 Read block 73 len 32768
12347 183/300 Flush
12348 184/300 Flush
12349 185/300 Read block 3859 len 4096
12350 186/300 Flush
12351 187/300 Write block 2393 len 32768 data: 4 4 4 3 4 5 5 5
12352 188/300 Read block 1648 len 32768
12353 189/300 Read block 1056 len 12288
12354 190/300 Read block 1805 len 20480
12355 191/300 Read block 4326 len 28672
12356 192/300 Read block 900 len 16384
12357 193/300 Write block 628 len 20480 data: 3 3 4 3 4
12358 194/300 Read block 3330 len 4096
12359 195/300 Write block 3442 len 24576 data: 5 5 4 3 3 2
12360 196/300 Read block 2889 len 40960
12361 197/300 Write block 2804 len 32768 data: 3 3 3 2 2 2 2 3
12362 198/300 Read block 4051 len 32768
12363 199/300 Read block 1520 len 4096
12364 200/300 Read block 1142 len 24576
12365 201/300 Write block 4728 len 36864 data: 4 3 5 5 5 4 4 5 5
12366 202/300 Write block 4373 len 20480 data: 5 4 3 3 3
12367 203/300 Read block 569 len 4096
12368 204/300 Read block 3019 len 20480
12369 205/300 Read block 1676 len 20480
12370 206/300 Read block 4529 len 16384
12371 207/300 Flush
12372 208/300 Flush
12373 209/300 Flush
12374 210/300 Write block 2746 len 28672 data: 4 4 5 5 6 6 5
12375 211/300 Write block 954 len 28672 data: 2 2 2 2 2 2 2
12376 212/300 Write block 3828 len 36864 data: 3 3 3 2 2 2 2 4 5
12377 213/300 Flush
12378 214/300 Read block 441 len 28672
12379 215/300 Flush
12380 216/300 Read block 4532 len 8192
12381 217/300 Write block 1833 len 8192 data: 3 6
12382 218/300 Write block 3474 len 40960 data: 3 4 4 3 3 3 3 2 2 2
12383 219/300 Flush
12384 220/300 Read block 3566 len 24576
12385 221/300 Read block 3289 len 8192
12386 222/300 Write block 3466 len 16384 data: 3 3 4 2
12387 223/300 Read block 2852 len 24576
12388 224/300 Read block 3028 len 28672
12389 225/300 Write block 3069 len 4096 data: 4
12390 226/300 Flush
12391 227/300 Read block 4091 len 12288
12392 228/300 Write block 2594 len 32768 data: 4 4 3 3 3 3 3 2
12393 229/300 Read block 3279 len 24576
12394 230/300 Write block 1509 len 8192 data: 6 6
12395 231/300 Read block 4031 len 20480
12396 232/300 Read block 3654 len 8192
12397 233/300 Write block 3665 len 36864 data: 3 3 4 4 5 4 4 4 4
12398 234/300 Read block 1357 len 40960
12399 235/300 Write block 682 len 12288 data: 2 2 2
12400 236/300 Flush
12401 237/300 Read block 715 len 16384
12402 238/300 Write block 3305 len 12288 data: 4 4 6
12403 239/300 Write block 2514 len 36864 data: 2 2 2 2 2 2 2 2 2
12404 240/300 Read block 2687 len 16384
12405 241/300 Write block 3703 len 40960 data: 4 4 3 6 6 5 4 4 3 3
12406 242/300 Read block 3600 len 20480
12407 243/300 Write block 1135 len 36864 data: 2 3 4 4 4 4 4 6 5
12408 244/300 Read block 1194 len 16384
12409 245/300 Read block 3718 len 4096
12410 246/300 Write block 2101 len 16384 data: 3 3 2 3
12411 247/300 Write block 506 len 16384 data: 2 2 2 2
12412 248/300 Read block 2694 len 20480
12413 249/300 Flush
12414 250/300 Read block 4605 len 36864
12415 251/300 Flush
12416 252/300 Write block 3735 len 16384 data: 3 3 3 3
12417 253/300 Flush
12418 254/300 Read block 3028 len 40960
12419 255/300 Write block 2353 len 4096 data: 3
12420 256/300 Read block 4111 len 40960
12421 257/300 Flush
12422 258/300 Write block 1487 len 32768 data: 3 3 3 3 2 2 3 2
12423 259/300 Write block 263 len 4096 data: 3
12424 260/300 Read block 2209 len 36864
12425 261/300 Read block 1152 len 20480
12426 262/300 Write block 3746 len 12288 data: 4 3 4
12427 263/300 Read block 3929 len 8192
12428 264/300 Write block 400 len 8192 data: 3 2
12429 265/300 Write block 3363 len 4096 data: 4
12430 266/300 Read block 262 len 20480
12431 267/300 Read block 1147 len 8192
12432 268/300 Write block 3755 len 8192 data: 4 3
12433 269/300 Flush
12434 270/300 Write block 4254 len 24576 data: 2 3 5 5 4 4
12435 271/300 Write block 210 len 36864 data: 4 4 4 4 4 3 3 4 4
12436 272/300 Write block 809 len 20480 data: 2 3 3 5 4
12437 273/300 Write block 1314 len 20480 data: 4 4 4 3 2
12438 274/300 Write block 2686 len 20480 data: 3 3 4 2 2
12439 275/300 Read block 691 len 28672
12440 276/300 Read block 953 len 12288
12441 277/300 Read block 3792 len 32768
12442 278/300 Write block 4968 len 20480 data: 4 4 4 7 7
12443 279/300 Write block 3400 len 4096 data: 4
12444 280/300 Read block 4443 len 16384
12445 281/300 Flush
12446 282/300 Write block 825 len 20480 data: 7 7 6 5 5
12447 283/300 Read block 693 len 36864
12448 284/300 Read block 3543 len 4096
12449 285/300 Flush
12450 286/300 Write block 1412 len 40960 data: 2 2 2 2 4 3 3 4 4 3
12451 287/300 Read block 471 len 28672
12452 288/300 Write block 4967 len 12288 data: 4 5 5
12453 289/300 Read block 3883 len 12288
12454 290/300 Read block 3105 len 4096
12455 291/300 Write block 4614 len 32768 data: 5 4 5 5 7 7 6 5
12456 292/300 Read block 4428 len 36864
12457 293/300 Read block 1360 len 12288
12458 294/300 Flush
12459 295/300 Write block 3758 len 8192 data: 3 3
12460 296/300 Read block 2986 len 4096
12461 297/300 Write block 581 len 40960 data: 2 3 4 6 6 5 5 4 4 4
12462 298/300 Flush
12463 299/300 Flush
12464 300/300 Read block 3111 len 12288
12465 301/300 Write block 667 len 36864 data: 2 4 4 4 4 5 4 3 3
12466 Main task has work to do, go find it
12467 got dsc Start(2)
12468 start 2
12469 Jan 23 17:40:22.625 INFO request completed, latency_us: 220, response_code: 204, uri: /start/cid/2, method: POST, req_id: a1de4af1-6d7a-4c14-99b6-fa7116977636, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
12470 Replay: started 2, returned:Ok(())
12471 ----------------------------------------------------------------
12472 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12473 States: Active Active Offline
12474 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12475 4322 Acked Flush 0 Done Done Sent false
12476 4323 Acked Read 9 Done Done Sent false
12477 4324 Acked Read 10 Done Done Sent false
12478 4325 Acked Write 8 Done Done Sent false
12479 4326 Acked Write 10 Done Done Sent false
12480 4327 Acked Write 3 Done Done Sent false
12481 4328 Acked Read 2 Done Done Sent false
12482 4329 Acked Flush 0 Done Done Sent false
12483 4330 Acked Write 1 Done Done Sent false
12484 4331 Acked Write 7 Done Done Sent false
12485 4332 Acked Write 1 Done Done Sent false
12486 4333 Acked Read 3 Done Done Sent false
12487 4334 Acked Write 8 Done Done Sent false
12488 4335 Acked Write 10 Done Done Sent false
12489 4336 Acked Read 5 Done Done Sent false
12490 4337 Acked Read 2 Done Done Sent false
12491 4338 Acked Write 10 Done Done Sent false
12492 4339 Acked Write 1 Done Done Sent false
12493 4340 Acked Read 1 Done Done Sent false
12494 4341 Acked Write 2 Done Done Sent false
12495 4342 Acked Read 7 Done Done Sent false
12496 4343 Acked Read 8 Done Done Sent false
12497 4344 Acked Write 10 Done Done Sent false
12498 4345 Acked Write 9 Done Done Sent false
12499 4346 Acked Read 7 Done Done Sent false
12500 4347 Acked Read 7 Done Done Sent false
12501 4348 Acked Write 1 Done Done Sent false
12502 4349 Acked Write 4 Done Done Sent false
12503 4350 Acked Read 2 Done Done Sent false
12504 4351 Acked Flush 0 Done Done Sent false
12505 4352 Acked Read 1 Done Done Sent false
12506 4353 Acked Read 9 Done Done Sent false
12507 4354 Acked Read 7 Done Done Sent false
12508 4355 Acked Read 3 Done Done Sent false
12509 4356 Acked Read 10 Done Done Sent false
12510 4357 Acked Read 3 Done Done Sent false
12511 4358 Acked Read 1 Done Done Sent false
12512 4359 Acked Write 3 Done Done Sent false
12513 4360 Acked Read 4 Done Done Sent false
12514 4361 Acked Write 3 Done Done Sent false
12515 4362 Acked Write 9 Done Done Sent false
12516 4363 Acked Write 9 Done Done Sent false
12517 4364 Acked Write 10 Done Done Sent false
12518 4365 Acked Write 7 Done Done Sent false
12519 4366 Acked Read 5 Done Done Sent false
12520 4367 Acked Write 9 Done Done Sent false
12521 4368 Acked Write 3 Done Done Sent false
12522 4369 Acked Write 5 Done Done Sent false
12523 4370 Acked Read 4 Done Done Sent false
12524 4371 Acked Write 5 Done Done Sent false
12525 4372 Acked Write 3 Done Done Sent false
12526 4373 Acked Write 10 Done Done Sent false
12527 4374 Acked Read 7 Done Done Sent false
12528 4375 Acked Read 6 Done Done Sent false
12529 4376 Acked Read 1 Done Done Sent false
12530 4377 Acked Write 5 Done Done Sent false
12531 4378 Acked Write 3 Done Done Sent false
12532 4379 Acked Write 5 Done Done Sent false
12533 4380 Acked Write 10 Done Done Sent false
12534 4381 Acked Write 8 Done Done Sent false
12535 4382 Acked Write 2 Done Done Sent false
12536 4383 Acked Read 4 Done Done Sent false
12537 4384 Acked Write 4 Done Done Sent false
12538 4385 Acked Write 2 Done Done Sent false
12539 4386 Acked Read 7 Done Done Sent false
12540 4387 Acked Flush 0 Done Done Sent false
12541 4388 Acked Flush 0 Done Done Sent false
12542 4389 Acked Read 5 Done Done Sent false
12543 4390 Acked Read 9 Done Done Sent false
12544 4391 Acked Read 9 Done Done Sent false
12545 4392 Acked Read 9 Done Done Sent false
12546 4393 Acked Write 2 Done Done Sent false
12547 4394 Acked Write 10 Done Done Sent false
12548 4395 Acked Read 2 Done Done Sent false
12549 4396 Acked Write 1 Done Done Sent false
12550 4397 Acked Write 8 Done Done Sent false
12551 4398 Acked Read 1 Done Done Sent false
12552 4399 Acked Read 2 Done Done Sent false
12553 4400 Acked Write 8 Done Done Sent false
12554 4401 Acked Write 4 Done Done Sent false
12555 4402 Acked Read 7 Done Done Sent false
12556 4403 Acked Write 4 Done Done Sent false
12557 4404 Acked Write 2 Done Done Sent false
12558 4405 Acked Read 10 Done Done Sent false
12559 4406 Acked Read 1 Done Done Sent false
12560 4407 Acked Write 7 Done Done Sent false
12561 4408 Acked Write 10 Done Done Sent false
12562 4409 Acked Read 7 Done Done Sent false
12563 4410 Acked Write 10 Done Done Sent false
12564 4411 Acked Read 7 Done Done Sent false
12565 4412 Acked Read 3 Done Done Sent false
12566 4413 Acked Flush 0 Done Done Sent false
12567 4414 Acked Read 10 Done Done Sent false
12568 4415 Acked Write 1 Done Done Sent false
12569 4416 Acked Write 10 Done Done Sent false
12570 4417 Acked Write 10 Done Done Sent false
12571 4418 Acked Write 4 Done Done Sent false
12572 4419 Acked Flush 0 Done Done Sent false
12573 4420 Acked Read 3 Done Done Sent false
12574 4421 Acked Read 7 Done Done Sent false
12575 4422 Acked Read 3 Done Done Sent false
12576 4423 Acked Write 6 Done Done Sent false
12577 4424 Acked Write 3 Done Done Sent false
12578 4425 Acked Write 5 Done Done Sent false
12579 4426 Acked Read 3 Done Done Sent false
12580 4427 Acked Read 4 Done Done Sent false
12581 4428 Acked Write 2 Done Done Sent false
12582 4429 Acked Write 2 Done Done Sent false
12583 4430 Acked Write 7 Done Done Sent false
12584 4431 Acked Write 2 Done Done Sent false
12585 4432 Acked Write 3 Done Done Sent false
12586 4433 Acked Read 8 Done Done Sent false
12587 4434 Acked Read 9 Done Done Sent false
12588 4435 Acked Flush 0 Done Done Sent false
12589 4436 Acked Read 1 Done Done Sent false
12590 4437 Acked Write 7 Done Done Sent false
12591 4438 Acked Write 10 Done Done Sent false
12592 4439 Acked Write 6 Done Done Sent false
12593 4440 Acked Read 6 Done Done Sent false
12594 4441 Acked Read 8 Done Done Sent false
12595 4442 Acked Read 9 Done Done Sent false
12596 4443 Acked Flush 0 Done Done Sent false
12597 4444 Acked Read 4 Done Done Sent false
12598 4445 Acked Flush 0 Done Done Sent false
12599 4446 Acked Flush 0 Done Done Sent false
12600 4447 Acked Read 10 Done Done Sent false
12601 4448 Acked Read 6 Done Done Sent false
12602 4449 Acked Read 7 Done Done Sent false
12603 4450 Acked Write 4 Done Done Sent false
12604 4451 Acked Read 5 Done Done Sent false
12605 4452 Acked Write 4 Done Done Sent false
12606 4453 Acked Flush 0 Done Done Sent false
12607 4454 Acked Read 8 Done Done Sent false
12608 4455 Acked Flush 0 Done Done Sent false
12609 4456 Acked Write 6 Done Done Sent false
12610 4457 Acked Read 8 Done Done Sent false
12611 4458 Acked Write 7 Done Done Sent false
12612 4459 Acked Read 8 Done Done Sent false
12613 4460 Acked Write 2 Done Done Sent false
12614 4461 Acked Write 1 Done Done Sent false
12615 4462 Acked Write 9 Done Done Sent false
12616 4463 Acked Write 4 Done Done Sent false
12617 4464 Acked Read 9 Done Done Sent false
12618 4465 Acked Write 1 Done Done Sent false
12619 4466 Acked Read 2 Done Done Sent false
12620 4467 Acked Read 9 Done Done Sent false
12621 4468 Acked Write 8 Done Done Sent false
12622 4469 Acked Read 8 Done Done Sent false
12623 4470 Acked Write 2 Done Done Sent false
12624 4471 Acked Read 9 Done Done Sent false
12625 4472 Acked Write 1 Done Done Sent false
12626 4473 Acked Write 6 Done Done Sent false
12627 4474 Acked Read 4 Done Done Sent false
12628 4475 Acked Read 1 Done Done Sent false
12629 4476 Acked Write 9 Done Done Sent false
12630 4477 Acked Flush 0 Done Done Sent false
12631 4478 Acked Read 6 Done Done Sent false
12632 4479 Acked Write 9 Done Done Sent false
12633 4480 Acked Write 9 Done Done Sent false
12634 4481 Acked Read 3 Done Done Sent false
12635 4482 Acked Write 9 Done Done Sent false
12636 4483 Acked Read 6 Done Done Sent false
12637 4484 Acked Read 5 Done Done Sent false
12638 4485 Acked Read 4 Done Done Sent false
12639 4486 Acked Read 6 Done Done Sent false
12640 4487 Acked Read 1 Done Done Sent false
12641 4488 Acked Read 7 Done Done Sent false
12642 4489 Acked Read 9 Done Done Sent false
12643 4490 Acked Write 8 Done Done Sent false
12644 4491 Acked Flush 0 Done Done Sent false
12645 4492 Acked Write 1 Done Done Sent false
12646 4493 Acked Read 9 Done Done Sent false
12647 4494 Acked Write 10 Done Done Sent false
12648 4495 Acked Write 6 Done Done Sent false
12649 4496 Acked Read 7 Done Done Sent false
12650 4497 Acked Write 5 Done Done Sent false
12651 4498 Acked Write 6 Done Done Sent false
12652 4499 Acked Read 7 Done Done Sent false
12653 4500 Acked Read 1 Done Done Sent false
12654 4501 Acked Flush 0 Done Done Sent false
12655 4502 Acked Read 5 Done Done Sent false
12656 4503 Acked Read 8 Done Done Sent false
12657 4504 Acked Flush 0 Done Done Sent false
12658 4505 Acked Flush 0 Done Done Sent false
12659 4506 Acked Read 1 Done Done Sent false
12660 4507 Acked Flush 0 Done Done Sent false
12661 4508 Acked Write 8 Done Done Sent false
12662 4509 Acked Read 8 Done Done Sent false
12663 4510 Acked Read 3 Done Done Sent false
12664 4511 Acked Read 5 Done Done Sent false
12665 4512 Acked Read 7 Done Done Sent false
12666 4513 Acked Read 4 Done Done Sent false
12667 4514 Acked Write 5 Done Done Sent false
12668 4515 Acked Read 1 Done Done Sent false
12669 4516 Acked Write 6 Done Done Sent false
12670 4517 Acked Read 10 Done Done Sent false
12671 4518 Acked Write 8 Done Done Sent false
12672 4519 Acked Read 8 Done Done Sent false
12673 4520 Acked Read 1 Done Done Sent false
12674 4521 Acked Read 6 Done Done Sent false
12675 4522 Acked Write 9 Done Done Sent false
12676 4523 Acked Write 5 Done Done Sent false
12677 4524 Acked Read 1 Done Done Sent false
12678 4525 Acked Read 5 Done Done Sent false
12679 4526 Acked Read 5 Done Done Sent false
12680 4527 Acked Read 4 Done Done Sent false
12681 4528 Acked Flush 0 Done Done Sent false
12682 4529 Acked Flush 0 Done Done Sent false
12683 4530 Acked Flush 0 Done Done Sent false
12684 4531 Acked Write 7 Done Done Sent false
12685 4532 Acked Write 7 Done Done Sent false
12686 4533 Acked Write 9 Done Done Sent false
12687 4534 Acked Flush 0 Done Done Sent false
12688 4535 Acked Read 7 Done Done Sent false
12689 4536 Acked Flush 0 Done Done Sent false
12690 4537 Acked Read 2 Done Done Sent false
12691 4538 Acked Write 2 Done Done Sent false
12692 4539 Acked Write 10 Done Done Sent false
12693 4540 Acked Flush 0 Done Done Sent false
12694 4541 Acked Read 6 Done Done Sent false
12695 4542 Acked Read 2 Done Done Sent false
12696 4543 Acked Write 4 Done Done Sent false
12697 4544 Acked Read 6 Done Done Sent false
12698 4545 Acked Read 7 Done Done Sent false
12699 4546 Acked Write 1 Done Done Sent false
12700 4547 Acked Flush 0 Done Done Sent false
12701 4548 Acked Read 3 Done Done Sent false
12702 4549 Acked Write 8 Done Done Sent false
12703 4550 Acked Read 6 Done Done Sent false
12704 4551 Acked Write 2 Done Done Sent false
12705 4552 Acked Read 5 Done Done Sent false
12706 4553 Acked Read 2 Done Done Sent false
12707 4554 Acked Write 9 Done Done Sent false
12708 4555 Acked Read 10 Done Done Sent false
12709 4556 Acked Write 3 Done Done Sent false
12710 4557 Acked Flush 0 Done Done Sent false
12711 4558 Acked Read 4 Done Done Sent false
12712 4559 Acked Write 3 Done Done Sent false
12713 4560 Acked Write 9 Done Done Sent false
12714 4561 Acked Read 4 Done Done Sent false
12715 4562 Acked Write 10 Done Done Sent false
12716 4563 Acked Read 5 Done Done Sent false
12717 4564 Acked Write 9 Done Done Sent false
12718 4565 Acked Read 4 Done Done Sent false
12719 4566 Acked Read 1 Done Done Sent false
12720 4567 Acked Write 4 Done Done Sent false
12721 4568 Acked Write 4 Done Done Sent false
12722 4569 Acked Read 5 Done Done Sent false
12723 4570 Acked Flush 0 Done Done Sent false
12724 4571 Acked Read 9 Done Done Sent false
12725 4572 Acked Flush 0 Done Done Sent false
12726 4573 Acked Write 4 Done Done Sent false
12727 4574 Acked Flush 0 Done Done Sent false
12728 4575 Acked Read 10 Done Done Sent false
12729 4576 Acked Write 1 Done Done Sent false
12730 4577 Acked Read 10 Done Done Sent false
12731 4578 Acked Flush 0 Done Done Sent false
12732 4579 Acked Write 8 Done Done Sent false
12733 4580 Acked Write 1 Done Done Sent false
12734 4581 Acked Read 9 Done Done Sent false
12735 4582 Acked Read 5 Done Done Sent false
12736 4583 Acked Write 3 Done Done Sent false
12737 4584 Acked Read 2 Done Done Sent false
12738 4585 Acked Write 2 Done Done Sent false
12739 4586 Acked Write 1 Done Done Sent false
12740 4587 Acked Read 5 Done Done Sent false
12741 4588 Acked Read 2 Done Done Sent false
12742 4589 Acked Write 2 Done Done Sent false
12743 4590 Acked Flush 0 Done Done Sent false
12744 4591 Acked Write 6 Done Done Sent false
12745 4592 Acked Write 9 Done Done Sent false
12746 4593 Acked Write 5 Done Done Sent false
12747 4594 Acked Write 5 Done Done Sent false
12748 4595 Acked Write 5 Done Done Sent false
12749 4596 Acked Read 7 Done Done Sent false
12750 4597 Acked Read 3 Done Done Sent false
12751 4598 Acked Read 8 Done Done Sent false
12752 4599 Acked Write 5 Done Done Sent false
12753 4600 Acked Write 1 Done Done Sent false
12754 4601 Acked Read 4 Done Done Sent false
12755 4602 Acked Flush 0 Done Done Sent false
12756 4603 Acked Write 5 Done Done Sent false
12757 4604 Acked Read 9 Done Done Sent false
12758 4605 Acked Read 1 Done Done Sent false
12759 4606 Acked Flush 0 Done Done Sent false
12760 4607 Acked Write 10 Done Done Sent false
12761 4608 Acked Read 7 Done Done Sent false
12762 4609 Acked Write 3 Done Done Sent false
12763 4610 Acked Read 3 Done Done Sent false
12764 4611 Acked Read 1 Done Done Sent false
12765 4612 Acked Write 8 Done Done Sent false
12766 4613 Acked Read 9 Done Done Sent false
12767 4614 Acked Read 3 Done Done Sent false
12768 4615 Acked Flush 0 Done Done Sent false
12769 4616 Acked Write 2 Done Done Sent false
12770 4617 Acked Read 1 Done Done Sent false
12771 4618 Acked Write 10 Done Done Sent false
12772 4619 Acked Flush 0 Done Done Sent false
12773 4620 Acked Flush 0 Done Done Sent false
12774 4621 Acked Read 3 Done Done Sent false
12775 4622 Acked Write 9 Done Done Sent false
12776 STATES DS:0 DS:1 DS:2 TOTAL
12777 Sent 0 0 301 301
12778 Done 301 301 0 602
12779 Skipped 0 0 0 0
12780 Error 0 0 0 0
12781 Last Flush: Some(JobId(4620)) Some(JobId(4620)) Some(JobId(4321))
12782 Downstairs last five completed: 4321 4320 4319 4318 4317
12783 Upstairs last five completed: 4622 4621 4620 4619 4618
12784 CLIENT: Up:0 ds:301 act:2
12785 [8830] got start action
12786 [8830] I am going to restart
12787 Starting downstairs at port 8830
12788 Make output file at "/tmp/dsc/downstairs-8830.txt"
12789 [8830][2] reports Starting
12790 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1217)
12791 [8830][2] reports Running
12792 ----------------------------------------------------------------
12793 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
12794 States: Active Active Offline
12795 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12796 4322 Acked Flush 0 Done Done Sent false
12797 4323 Acked Read 9 Done Done Sent false
12798 4324 Acked Read 10 Done Done Sent false
12799 4325 Acked Write 8 Done Done Sent false
12800 4326 Acked Write 10 Done Done Sent false
12801 4327 Acked Write 3 Done Done Sent false
12802 4328 Acked Read 2 Done Done Sent false
12803 4329 Acked Flush 0 Done Done Sent false
12804 4330 Acked Write 1 Done Done Sent false
12805 4331 Acked Write 7 Done Done Sent false
12806 4332 Acked Write 1 Done Done Sent false
12807 4333 Acked Read 3 Done Done Sent false
12808 4334 Acked Write 8 Done Done Sent false
12809 4335 Acked Write 10 Done Done Sent false
12810 4336 Acked Read 5 Done Done Sent false
12811 4337 Acked Read 2 Done Done Sent false
12812 4338 Acked Write 10 Done Done Sent false
12813 4339 Acked Write 1 Done Done Sent false
12814 4340 Acked Read 1 Done Done Sent false
12815 4341 Acked Write 2 Done Done Sent false
12816 4342 Acked Read 7 Done Done Sent false
12817 4343 Acked Read 8 Done Done Sent false
12818 4344 Acked Write 10 Done Done Sent false
12819 4345 Acked Write 9 Done Done Sent false
12820 4346 Acked Read 7 Done Done Sent false
12821 4347 Acked Read 7 Done Done Sent false
12822 4348 Acked Write 1 Done Done Sent false
12823 4349 Acked Write 4 Done Done Sent false
12824 4350 Acked Read 2 Done Done Sent false
12825 4351 Acked Flush 0 Done Done Sent false
12826 4352 Acked Read 1 Done Done Sent false
12827 4353 Acked Read 9 Done Done Sent false
12828 4354 Acked Read 7 Done Done Sent false
12829 4355 Acked Read 3 Done Done Sent false
12830 4356 Acked Read 10 Done Done Sent false
12831 4357 Acked Read 3 Done Done Sent false
12832 4358 Acked Read 1 Done Done Sent false
12833 4359 Acked Write 3 Done Done Sent false
12834 4360 Acked Read 4 Done Done Sent false
12835 4361 Acked Write 3 Done Done Sent false
12836 4362 Acked Write 9 Done Done Sent false
12837 4363 Acked Write 9 Done Done Sent false
12838 4364 Acked Write 10 Done Done Sent false
12839 4365 Acked Write 7 Done Done Sent false
12840 4366 Acked Read 5 Done Done Sent false
12841 4367 Acked Write 9 Done Done Sent false
12842 4368 Acked Write 3 Done Done Sent false
12843 4369 Acked Write 5 Done Done Sent false
12844 4370 Acked Read 4 Done Done Sent false
12845 4371 Acked Write 5 Done Done Sent false
12846 4372 Acked Write 3 Done Done Sent false
12847 4373 Acked Write 10 Done Done Sent false
12848 4374 Acked Read 7 Done Done Sent false
12849 4375 Acked Read 6 Done Done Sent false
12850 4376 Acked Read 1 Done Done Sent false
12851 4377 Acked Write 5 Done Done Sent false
12852 4378 Acked Write 3 Done Done Sent false
12853 4379 Acked Write 5 Done Done Sent false
12854 4380 Acked Write 10 Done Done Sent false
12855 4381 Acked Write 8 Done Done Sent false
12856 4382 Acked Write 2 Done Done Sent false
12857 4383 Acked Read 4 Done Done Sent false
12858 4384 Acked Write 4 Done Done Sent false
12859 4385 Acked Write 2 Done Done Sent false
12860 4386 Acked Read 7 Done Done Sent false
12861 4387 Acked Flush 0 Done Done Sent false
12862 4388 Acked Flush 0 Done Done Sent false
12863 4389 Acked Read 5 Done Done Sent false
12864 4390 Acked Read 9 Done Done Sent false
12865 4391 Acked Read 9 Done Done Sent false
12866 4392 Acked Read 9 Done Done Sent false
12867 4393 Acked Write 2 Done Done Sent false
12868 4394 Acked Write 10 Done Done Sent false
12869 4395 Acked Read 2 Done Done Sent false
12870 4396 Acked Write 1 Done Done Sent false
12871 4397 Acked Write 8 Done Done Sent false
12872 4398 Acked Read 1 Done Done Sent false
12873 4399 Acked Read 2 Done Done Sent false
12874 4400 Acked Write 8 Done Done Sent false
12875 4401 Acked Write 4 Done Done Sent false
12876 4402 Acked Read 7 Done Done Sent false
12877 4403 Acked Write 4 Done Done Sent false
12878 4404 Acked Write 2 Done Done Sent false
12879 4405 Acked Read 10 Done Done Sent false
12880 4406 Acked Read 1 Done Done Sent false
12881 4407 Acked Write 7 Done Done Sent false
12882 4408 Acked Write 10 Done Done Sent false
12883 4409 Acked Read 7 Done Done Sent false
12884 4410 Acked Write 10 Done Done Sent false
12885 4411 Acked Read 7 Done Done Sent false
12886 4412 Acked Read 3 Done Done Sent false
12887 4413 Acked Flush 0 Done Done Sent false
12888 4414 Acked Read 10 Done Done Sent false
12889 4415 Acked Write 1 Done Done Sent false
12890 4416 Acked Write 10 Done Done Sent false
12891 4417 Acked Write 10 Done Done Sent false
12892 4418 Acked Write 4 Done Done Sent false
12893 4419 Acked Flush 0 Done Done Sent false
12894 4420 Acked Read 3 Done Done Sent false
12895 4421 Acked Read 7 Done Done Sent false
12896 4422 Acked Read 3 Done Done Sent false
12897 4423 Acked Write 6 Done Done Sent false
12898 4424 Acked Write 3 Done Done Sent false
12899 4425 Acked Write 5 Done Done Sent false
12900 4426 Acked Read 3 Done Done Sent false
12901 4427 Acked Read 4 Done Done Sent false
12902 4428 Acked Write 2 Done Done Sent false
12903 4429 Acked Write 2 Done Done Sent false
12904 4430 Acked Write 7 Done Done Sent false
12905 4431 Acked Write 2 Done Done Sent false
12906 4432 Acked Write 3 Done Done Sent false
12907 4433 Acked Read 8 Done Done Sent false
12908 4434 Acked Read 9 Done Done Sent false
12909 4435 Acked Flush 0 Done Done Sent false
12910 4436 Acked Read 1 Done Done Sent false
12911 4437 Acked Write 7 Done Done Sent false
12912 4438 Acked Write 10 Done Done Sent false
12913 4439 Acked Write 6 Done Done Sent false
12914 4440 Acked Read 6 Done Done Sent false
12915 4441 Acked Read 8 Done Done Sent false
12916 4442 Acked Read 9 Done Done Sent false
12917 4443 Acked Flush 0 Done Done Sent false
12918 4444 Acked Read 4 Done Done Sent false
12919 4445 Acked Flush 0 Done Done Sent false
12920 4446 Acked Flush 0 Done Done Sent false
12921 4447 Acked Read 10 Done Done Sent false
12922 4448 Acked Read 6 Done Done Sent false
12923 4449 Acked Read 7 Done Done Sent false
12924 4450 Acked Write 4 Done Done Sent false
12925 4451 Acked Read 5 Done Done Sent false
12926 4452 Acked Write 4 Done Done Sent false
12927 4453 Acked Flush 0 Done Done Sent false
12928 4454 Acked Read 8 Done Done Sent false
12929 4455 Acked Flush 0 Done Done Sent false
12930 4456 Acked Write 6 Done Done Sent false
12931 4457 Acked Read 8 Done Done Sent false
12932 4458 Acked Write 7 Done Done Sent false
12933 4459 Acked Read 8 Done Done Sent false
12934 4460 Acked Write 2 Done Done Sent false
12935 4461 Acked Write 1 Done Done Sent false
12936 4462 Acked Write 9 Done Done Sent false
12937 4463 Acked Write 4 Done Done Sent false
12938 4464 Acked Read 9 Done Done Sent false
12939 4465 Acked Write 1 Done Done Sent false
12940 4466 Acked Read 2 Done Done Sent false
12941 4467 Acked Read 9 Done Done Sent false
12942 4468 Acked Write 8 Done Done Sent false
12943 4469 Acked Read 8 Done Done Sent false
12944 4470 Acked Write 2 Done Done Sent false
12945 4471 Acked Read 9 Done Done Sent false
12946 4472 Acked Write 1 Done Done Sent false
12947 4473 Acked Write 6 Done Done Sent false
12948 4474 Acked Read 4 Done Done Sent false
12949 4475 Acked Read 1 Done Done Sent false
12950 4476 Acked Write 9 Done Done Sent false
12951 4477 Acked Flush 0 Done Done Sent false
12952 4478 Acked Read 6 Done Done Sent false
12953 4479 Acked Write 9 Done Done Sent false
12954 4480 Acked Write 9 Done Done Sent false
12955 4481 Acked Read 3 Done Done Sent false
12956 4482 Acked Write 9 Done Done Sent false
12957 4483 Acked Read 6 Done Done Sent false
12958 4484 Acked Read 5 Done Done Sent false
12959 4485 Acked Read 4 Done Done Sent false
12960 4486 Acked Read 6 Done Done Sent false
12961 4487 Acked Read 1 Done Done Sent false
12962 4488 Acked Read 7 Done Done Sent false
12963 4489 Acked Read 9 Done Done Sent false
12964 4490 Acked Write 8 Done Done Sent false
12965 4491 Acked Flush 0 Done Done Sent false
12966 4492 Acked Write 1 Done Done Sent false
12967 4493 Acked Read 9 Done Done Sent false
12968 4494 Acked Write 10 Done Done Sent false
12969 4495 Acked Write 6 Done Done Sent false
12970 4496 Acked Read 7 Done Done Sent false
12971 4497 Acked Write 5 Done Done Sent false
12972 4498 Acked Write 6 Done Done Sent false
12973 4499 Acked Read 7 Done Done Sent false
12974 4500 Acked Read 1 Done Done Sent false
12975 4501 Acked Flush 0 Done Done Sent false
12976 4502 Acked Read 5 Done Done Sent false
12977 4503 Acked Read 8 Done Done Sent false
12978 4504 Acked Flush 0 Done Done Sent false
12979 4505 Acked Flush 0 Done Done Sent false
12980 4506 Acked Read 1 Done Done Sent false
12981 4507 Acked Flush 0 Done Done Sent false
12982 4508 Acked Write 8 Done Done Sent false
12983 4509 Acked Read 8 Done Done Sent false
12984 4510 Acked Read 3 Done Done Sent false
12985 4511 Acked Read 5 Done Done Sent false
12986 4512 Acked Read 7 Done Done Sent false
12987 4513 Acked Read 4 Done Done Sent false
12988 4514 Acked Write 5 Done Done Sent false
12989 4515 Acked Read 1 Done Done Sent false
12990 4516 Acked Write 6 Done Done Sent false
12991 4517 Acked Read 10 Done Done Sent false
12992 4518 Acked Write 8 Done Done Sent false
12993 4519 Acked Read 8 Done Done Sent false
12994 4520 Acked Read 1 Done Done Sent false
12995 4521 Acked Read 6 Done Done Sent false
12996 4522 Acked Write 9 Done Done Sent false
12997 4523 Acked Write 5 Done Done Sent false
12998 4524 Acked Read 1 Done Done Sent false
12999 4525 Acked Read 5 Done Done Sent false
13000 4526 Acked Read 5 Done Done Sent false
13001 4527 Acked Read 4 Done Done Sent false
13002 4528 Acked Flush 0 Done Done Sent false
13003 4529 Acked Flush 0 Done Done Sent false
13004 4530 Acked Flush 0 Done Done Sent false
13005 4531 Acked Write 7 Done Done Sent false
13006 4532 Acked Write 7 Done Done Sent false
13007 4533 Acked Write 9 Done Done Sent false
13008 4534 Acked Flush 0 Done Done Sent false
13009 4535 Acked Read 7 Done Done Sent false
13010 4536 Acked Flush 0 Done Done Sent false
13011 4537 Acked Read 2 Done Done Sent false
13012 4538 Acked Write 2 Done Done Sent false
13013 4539 Acked Write 10 Done Done Sent false
13014 4540 Acked Flush 0 Done Done Sent false
13015 4541 Acked Read 6 Done Done Sent false
13016 4542 Acked Read 2 Done Done Sent false
13017 4543 Acked Write 4 Done Done Sent false
13018 4544 Acked Read 6 Done Done Sent false
13019 4545 Acked Read 7 Done Done Sent false
13020 4546 Acked Write 1 Done Done Sent false
13021 4547 Acked Flush 0 Done Done Sent false
13022 4548 Acked Read 3 Done Done Sent false
13023 4549 Acked Write 8 Done Done Sent false
13024 4550 Acked Read 6 Done Done Sent false
13025 4551 Acked Write 2 Done Done Sent false
13026 4552 Acked Read 5 Done Done Sent false
13027 4553 Acked Read 2 Done Done Sent false
13028 4554 Acked Write 9 Done Done Sent false
13029 4555 Acked Read 10 Done Done Sent false
13030 4556 Acked Write 3 Done Done Sent false
13031 4557 Acked Flush 0 Done Done Sent false
13032 4558 Acked Read 4 Done Done Sent false
13033 4559 Acked Write 3 Done Done Sent false
13034 4560 Acked Write 9 Done Done Sent false
13035 4561 Acked Read 4 Done Done Sent false
13036 4562 Acked Write 10 Done Done Sent false
13037 4563 Acked Read 5 Done Done Sent false
13038 4564 Acked Write 9 Done Done Sent false
13039 4565 Acked Read 4 Done Done Sent false
13040 4566 Acked Read 1 Done Done Sent false
13041 4567 Acked Write 4 Done Done Sent false
13042 4568 Acked Write 4 Done Done Sent false
13043 4569 Acked Read 5 Done Done Sent false
13044 4570 Acked Flush 0 Done Done Sent false
13045 4571 Acked Read 9 Done Done Sent false
13046 4572 Acked Flush 0 Done Done Sent false
13047 4573 Acked Write 4 Done Done Sent false
13048 4574 Acked Flush 0 Done Done Sent false
13049 4575 Acked Read 10 Done Done Sent false
13050 4576 Acked Write 1 Done Done Sent false
13051 4577 Acked Read 10 Done Done Sent false
13052 4578 Acked Flush 0 Done Done Sent false
13053 4579 Acked Write 8 Done Done Sent false
13054 4580 Acked Write 1 Done Done Sent false
13055 4581 Acked Read 9 Done Done Sent false
13056 4582 Acked Read 5 Done Done Sent false
13057 4583 Acked Write 3 Done Done Sent false
13058 4584 Acked Read 2 Done Done Sent false
13059 4585 Acked Write 2 Done Done Sent false
13060 4586 Acked Write 1 Done Done Sent false
13061 4587 Acked Read 5 Done Done Sent false
13062 4588 Acked Read 2 Done Done Sent false
13063 4589 Acked Write 2 Done Done Sent false
13064 4590 Acked Flush 0 Done Done Sent false
13065 4591 Acked Write 6 Done Done Sent false
13066 4592 Acked Write 9 Done Done Sent false
13067 4593 Acked Write 5 Done Done Sent false
13068 4594 Acked Write 5 Done Done Sent false
13069 4595 Acked Write 5 Done Done Sent false
13070 4596 Acked Read 7 Done Done Sent false
13071 4597 Acked Read 3 Done Done Sent false
13072 4598 Acked Read 8 Done Done Sent false
13073 4599 Acked Write 5 Done Done Sent false
13074 4600 Acked Write 1 Done Done Sent false
13075 4601 Acked Read 4 Done Done Sent false
13076 4602 Acked Flush 0 Done Done Sent false
13077 4603 Acked Write 5 Done Done Sent false
13078 4604 Acked Read 9 Done Done Sent false
13079 4605 Acked Read 1 Done Done Sent false
13080 4606 Acked Flush 0 Done Done Sent false
13081 4607 Acked Write 10 Done Done Sent false
13082 4608 Acked Read 7 Done Done Sent false
13083 4609 Acked Write 3 Done Done Sent false
13084 4610 Acked Read 3 Done Done Sent false
13085 4611 Acked Read 1 Done Done Sent false
13086 4612 Acked Write 8 Done Done Sent false
13087 4613 Acked Read 9 Done Done Sent false
13088 4614 Acked Read 3 Done Done Sent false
13089 4615 Acked Flush 0 Done Done Sent false
13090 4616 Acked Write 2 Done Done Sent false
13091 4617 Acked Read 1 Done Done Sent false
13092 4618 Acked Write 10 Done Done Sent false
13093 4619 Acked Flush 0 Done Done Sent false
13094 4620 Acked Flush 0 Done Done Sent false
13095 4621 Acked Read 3 Done Done Sent false
13096 4622 Acked Write 9 Done Done Sent false
13097 4623 Acked Flush 0 Done Done Sent false
13098 STATES DS:0 DS:1 DS:2 TOTAL
13099 Sent 0 0 302 302
13100 Done 302 302 0 604
13101 Skipped 0 0 0 0
13102 Error 0 0 0 0
13103 Last Flush: Some(JobId(4623)) Some(JobId(4623)) Some(JobId(4321))
13104 Downstairs last five completed: 4321 4320 4319 4318 4317
13105 Upstairs last five completed: 4623 4622 4621 4620 4619
13106 CLIENT: Up:0 ds:302 act:2
131072025-01-23T17:40:28.392ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
131082025-01-23T17:40:28.392ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34011) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
131092025-01-23T17:40:28.393ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
131102025-01-23T17:40:28.394ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
131112025-01-23T17:40:28.394ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
131122025-01-23T17:40:28.394ZINFOcrucible: send last flush ID to this DS: Some(JobId(4321)) = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
131132025-01-23T17:40:28.394ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
131142025-01-23T17:40:28.394ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(4321)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
131152025-01-23T17:40:28.394ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(4321)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
13116 ----------------------------------------------------------------
13117 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13118 Downstairs last five completed: 4623 4622 4621 4620 4619
13119 Upstairs last five completed: 4623 4622 4621 4620 4619
13120 CLIENT: Up:0 ds:0 act:3
13121 Replay: All jobs finished
13122 Main task has work to do, go find it
13123 got dsc Stop(1)
13124 stop 1
13125 Jan 23 17:40:30.631[8820] Got stop action so:false kr:false
13126 INFO request completed, latency_us: 415, response_code: 204, uri: /stop/cid/1, method: POST, req_id: a83cb1dc-cb6b-46ea-b9d7-3c4f5f2fbfd2, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
131272025-01-23T17:40:30.631ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
13128 Jan 23 17:40:30.631 INFO request completed, latency_us: 149, response_code: 200, uri: /state/cid/1, method: GET, req_id: b57934a9-a445-4828-bece-d088e5193fc0, remote_addr: 127.0.0.1:42015, local_addr: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:40:30.631915419Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
13129 127.0.0.1:9998
131302025-01-23T17:40:30.631ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
131312025-01-23T17:40:30.632ZINFOcrucible: client task is exiting = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
13132 [8820] Exited with: ExitStatus(unix_wait_status(9))
13133 [8820][1] reports Exit
13134 Jan 23 17:40:34.634 INFO request completed, latency_us: 361, response_code: 200, uri: /state/cid/1, method: GET, req_id: f076102a-df0b-46c4-82f7-db443c74edd8, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
13135 001/300 Write block 1052 len 40960 data: 4 3 2 2 2 2 2 3 3 4
13136 002/300 Read block 186 len 24576
13137 003/300 Read block 1708 len 32768
13138 004/300 Read block 3754 len 20480
13139 005/300 Read block 4711 len 4096
13140 006/300 Write block 1272 len 36864 data: 2 3 3 4 4 4 4 4 3
13141 007/300 Write block 2745 len 16384 data: 4 5 5 6
13142 008/300 Read block 1529 len 36864
13143 009/300 Read block 4695 len 4096
13144 010/300 Write block 1272 len 32768 data: 3 4 4 5 5 5 5 5
13145 011/300 Write block 2412 len 40960 data: 3 6 5 5 5 5 4 4 4 4
13146 012/300 Read block 1599 len 36864
13147 013/300 Write block 41 len 40960 data: 3 3 3 3 2 2 3 3 3 3
13148 014/300 Read block 971 len 16384
13149 015/300 Write block 2589 len 12288 data: 3 4 4
13150 016/300 Flush
13151 017/300 Read block 4798 len 32768
13152 018/300 Write block 2181 len 20480 data: 4 3 3 3 2
13153 019/300 Flush
13154 020/300 Read block 619 len 8192
13155 021/300 Read block 4419 len 20480
13156 022/300 Read block 332 len 28672
13157 023/300 Read block 1516 len 8192
13158 024/300 Write block 3232 len 32768 data: 4 5 6 5 5 5 4 4
13159 025/300 Write block 3564 len 12288 data: 3 3 3
13160 026/300 Read block 4494 len 12288
13161 027/300 Write block 3955 len 32768 data: 4 4 4 3 4 5 6 6
13162 028/300 Flush
13163 029/300 Write block 854 len 8192 data: 2 2
13164 030/300 Write block 68 len 28672 data: 4 5 5 4 4 5 6
13165 031/300 Write block 2672 len 20480 data: 4 4 4 2 3
13166 032/300 Flush
13167 033/300 Read block 3063 len 36864
13168 034/300 Write block 4757 len 36864 data: 3 3 4 5 6 5 5 5 4
13169 035/300 Read block 779 len 40960
13170 036/300 Read block 1387 len 4096
13171 037/300 Read block 2800 len 12288
13172 038/300 Read block 166 len 12288
13173 039/300 Read block 249 len 8192
13174 040/300 Read block 2432 len 36864
13175 041/300 Read block 4915 len 24576
13176 042/300 Flush
13177 043/300 Write block 542 len 40960 data: 4 5 4 4 4 3 3 4 4 4
13178 044/300 Read block 4495 len 20480
13179 045/300 Write block 195 len 40960 data: 5 5 3 3 3 3 5 4 6 6
13180 046/300 Write block 1560 len 36864 data: 4 4 4 3 3 4 4 4 4
13181 047/300 Read block 2065 len 40960
13182 048/300 Read block 3481 len 16384
13183 049/300 Write block 1833 len 16384 data: 4 7 6 7
13184 050/300 Write block 3027 len 28672 data: 4 3 2 2 3 4 5
13185 051/300 Read block 253 len 36864
13186 052/300 Read block 755 len 28672
13187 053/300 Write block 3680 len 24576 data: 2 2 2 2 2 3
13188 054/300 Read block 4611 len 8192
13189 055/300 Write block 2124 len 20480 data: 4 4 4 4 3
13190 056/300 Read block 1163 len 36864
13191 057/300 Write block 3156 len 36864 data: 2 4 5 5 5 6 7 6 6
13192 058/300 Read block 462 len 16384
13193 059/300 Write block 1710 len 16384 data: 2 3 3 2
13194 060/300 Read block 2620 len 24576
13195 061/300 Read block 1894 len 28672
13196 062/300 Read block 4579 len 40960
13197 063/300 Read block 4829 len 40960
13198 064/300 Flush
13199 065/300 Read block 4304 len 12288
13200 066/300 Read block 2073 len 20480
13201 067/300 Read block 2187 len 16384
13202 068/300 Read block 4051 len 12288
13203 069/300 Write block 4670 len 24576 data: 5 5 5 5 4 4
13204 070/300 Flush
13205 071/300 Flush
13206 072/300 Write block 2618 len 36864 data: 4 3 4 4 4 4 5 4 4
13207 073/300 Write block 3501 len 12288 data: 6 6 6
13208 074/300 Write block 4945 len 32768 data: 4 4 3 3 4 4 4 5
13209 075/300 Flush
13210 076/300 Flush
13211 077/300 Read block 2371 len 16384
13212 078/300 Read block 2258 len 8192
13213 079/300 Read block 2057 len 32768
13214 080/300 Read block 767 len 36864
13215 081/300 Read block 3761 len 12288
13216 082/300 Read block 4705 len 24576
13217 083/300 Read block 3916 len 4096
13218 084/300 Read block 3122 len 4096
13219 085/300 Read block 1290 len 40960
13220 086/300 Write block 3677 len 36864 data: 3 3 2 3 3 3 3 3 4
13221 087/300 Write block 4095 len 16384 data: 3 2 2 2
13222 088/300 Read block 2450 len 8192
13223 089/300 Read block 4135 len 24576
13224 090/300 Read block 1283 len 12288
13225 091/300 Read block 3899 len 40960
13226 092/300 Flush
13227 093/300 Flush
13228 094/300 Write block 237 len 24576 data: 3 2 2 2 2 2
13229 095/300 Read block 3702 len 16384
13230 096/300 Write block 2095 len 20480 data: 3 3 3 3 3
13231 097/300 Read block 1001 len 24576
13232 098/300 Write block 2881 len 36864 data: 3 3 3 3 2 2 3 3 4
13233 099/300 Read block 501 len 8192
13234 100/300 Read block 4394 len 12288
13235 101/300 Write block 58 len 28672 data: 4 4 3 3 3 3 3
13236 102/300 Flush
13237 103/300 Flush
13238 104/300 Write block 2290 len 12288 data: 3 3 3
13239 105/300 Write block 2594 len 20480 data: 5 5 4 4 4
13240 106/300 Read block 2200 len 40960
13241 107/300 Flush
13242 108/300 Read block 3262 len 28672
13243 109/300 Read block 3143 len 20480
13244 110/300 Read block 3767 len 28672
13245 111/300 Read block 3050 len 20480
13246 112/300 Read block 4536 len 8192
13247 113/300 Read block 459 len 4096
13248 114/300 Write block 2677 len 8192 data: 2 2
13249 115/300 Flush
13250 116/300 Read block 2825 len 12288
13251 117/300 Flush
13252 118/300 Write block 888 len 12288 data: 3 4 4
13253 119/300 Flush
13254 120/300 Read block 3948 len 24576
13255 121/300 Read block 3437 len 40960
13256 122/300 Flush
13257 123/300 Write block 3727 len 28672 data: 3 3 3 2 2 4 2
13258 124/300 Write block 1837 len 8192 data: 6 4
13259 125/300 Read block 695 len 32768
13260 126/300 Read block 587 len 4096
13261 127/300 Read block 2034 len 36864
13262 128/300 Write block 797 len 12288 data: 3 3 3
13263 129/300 Write block 3663 len 36864 data: 4 3 4 4 5 5 6 5 5
13264 130/300 Write block 3702 len 32768 data: 4 5 5 4 7 7 6 5
13265 131/300 Read block 2693 len 36864
13266 132/300 Read block 4982 len 36864
13267 133/300 Read block 1092 len 36864
13268 134/300 Write block 1471 len 28672 data: 3 3 4 5 4 4 6
13269 135/300 Write block 2361 len 4096 data: 4
13270 136/300 Read block 777 len 36864
13271 137/300 Write block 2943 len 16384 data: 4 5 4 4
13272 138/300 Read block 1369 len 32768
13273 139/300 Write block 4679 len 28672 data: 3 3 3 5 5 4 6
13274 140/300 Write block 2651 len 32768 data: 4 4 3 2 2 2 2 3
13275 141/300 Flush
13276 142/300 Write block 909 len 40960 data: 3 3 3 4 4 3 3 3 3 3
13277 143/300 Write block 4633 len 12288 data: 3 3 3
13278 144/300 Write block 3821 len 4096 data: 4
13279 145/300 Read block 795 len 12288
13280 146/300 Read block 367 len 12288
13281 147/300 Read block 2599 len 40960
13282 148/300 Read block 1042 len 8192
13283 149/300 Write block 2695 len 24576 data: 3 4 4 4 4 4
13284 150/300 Write block 3461 len 12288 data: 6 7 5
13285 151/300 Write block 3219 len 40960 data: 2 2 2 2 3 4 5 5 2 2
13286 152/300 Read block 1556 len 16384
13287 153/300 Read block 3735 len 20480
13288 154/300 Flush
13289 155/300 Write block 1278 len 12288 data: 6 6 4
13290 156/300 Write block 2384 len 24576 data: 6 6 5 3 3 3
13291 157/300 Read block 1540 len 16384
13292 158/300 Write block 1198 len 4096 data: 4
13293 159/300 Read block 555 len 12288
13294 160/300 Write block 2618 len 24576 data: 5 4 5 5 5 5
13295 161/300 Read block 2193 len 24576
13296 162/300 Write block 4340 len 4096 data: 3
13297 163/300 Read block 1822 len 12288
13298 164/300 Write block 1758 len 40960 data: 5 5 5 4 4 5 6 3 3 2
13299 165/300 Read block 1524 len 16384
13300 166/300 Flush
13301 167/300 Read block 2475 len 8192
13302 168/300 Flush
13303 169/300 Read block 808 len 4096
13304 170/300 Read block 829 len 20480
13305 171/300 Write block 3488 len 8192 data: 2 4
13306 172/300 Write block 2735 len 4096 data: 3
13307 173/300 Read block 4101 len 24576
13308 174/300 Read block 3508 len 28672
13309 175/300 Write block 4239 len 8192 data: 4 4
13310 176/300 Read block 1048 len 28672
13311 177/300 Read block 4120 len 4096
13312 178/300 Read block 4973 len 4096
13313 179/300 Write block 3470 len 24576 data: 2 2 2 3 4 5
13314 180/300 Read block 2304 len 20480
13315 181/300 Write block 4779 len 36864 data: 3 3 2 2 2 2 2 2 2
13316 182/300 Write block 654 len 36864 data: 3 3 3 4 5 4 4 3 3
13317 183/300 Read block 4220 len 12288
13318 184/300 Read block 4780 len 12288
13319 185/300 Write block 241 len 32768 data: 3 3 2 3 3 3 3 3
13320 186/300 Write block 4501 len 40960 data: 3 3 3 4 4 3 3 3 4 4
13321 187/300 Read block 674 len 8192
13322 188/300 Read block 809 len 32768
13323 189/300 Write block 2074 len 40960 data: 5 4 4 3 3 3 5 4 5 3
13324 190/300 Flush
13325 191/300 Write block 4378 len 40960 data: 3 3 3 3 3 2 2 2 2 2
13326 192/300 Write block 639 len 4096 data: 4
13327 193/300 Write block 3558 len 12288 data: 4 4 3
13328 194/300 Flush
13329 195/300 Read block 4577 len 36864
13330 196/300 Read block 1205 len 20480
13331 197/300 Read block 321 len 20480
13332 198/300 Write block 159 len 4096 data: 5
13333 199/300 Read block 3131 len 16384
13334 200/300 Read block 4504 len 12288
13335 201/300 Flush
13336 202/300 Write block 3086 len 24576 data: 3 3 3 3 3 2
13337 203/300 Write block 3218 len 24576 data: 2 3 3 3 3 4
13338 204/300 Write block 4797 len 36864 data: 2 3 4 3 3 3 3 3 2
13339 205/300 Read block 4708 len 28672
13340 206/300 Write block 2595 len 12288 data: 6 5 5
13341 207/300 Write block 4324 len 8192 data: 4 2
13342 208/300 Write block 4101 len 32768 data: 2 2 2 2 2 2 2 2
13343 209/300 Flush
13344 210/300 Read block 1487 len 20480
13345 211/300 Write block 1710 len 32768 data: 3 4 4 3 2 2 2 2
13346 212/300 Flush
13347 213/300 Write block 3017 len 32768 data: 6 5 5 3 3 3 2 3
13348 214/300 Write block 541 len 32768 data: 5 5 6 5 5 5 4 4
13349 215/300 Write block 1625 len 16384 data: 4 4 4 4
13350 216/300 Write block 736 len 4096 data: 4
13351 217/300 Read block 1137 len 20480
13352 218/300 Write block 394 len 36864 data: 4 3 3 3 3 3 4 3 2
13353 219/300 Read block 1 len 32768
13354 220/300 Write block 1208 len 12288 data: 4 4 2
13355 221/300 Read block 1732 len 12288
13356 222/300 Read block 2741 len 20480
13357 223/300 Write block 3264 len 8192 data: 3 2
13358 224/300 Read block 4504 len 8192
13359 225/300 Write block 1896 len 32768 data: 4 4 4 4 6 6 6 5
13360 226/300 Read block 2945 len 8192
13361 227/300 Read block 1093 len 16384
13362 228/300 Read block 2386 len 4096
13363 229/300 Write block 2152 len 28672 data: 5 4 4 4 3 3 4
13364 230/300 Write block 2208 len 40960 data: 4 3 3 3 3 3 2 2 5 5
13365 231/300 Write block 3179 len 40960 data: 3 4 4 3 6 6 4 4 4 2
13366 232/300 Read block 953 len 16384
13367 233/300 Write block 3626 len 16384 data: 3 4 4 4
13368 234/300 Flush
13369 235/300 Read block 759 len 28672
13370 236/300 Write block 1446 len 36864 data: 2 2 2 3 4 3 3 2 2
13371 237/300 Write block 578 len 32768 data: 2 2 2 3 4 5 7 7
13372 238/300 Write block 231 len 36864 data: 3 3 3 2 2 2 4 3 3
13373 239/300 Read block 2437 len 16384
13374 240/300 Read block 2021 len 4096
13375 241/300 Write block 780 len 16384 data: 5 3 3 4
13376 242/300 Write block 916 len 28672 data: 4 4 4 2 3 3 3
13377 243/300 Read block 4233 len 32768
13378 244/300 Read block 3935 len 4096
13379 245/300 Read block 650 len 12288
13380 246/300 Write block 1412 len 4096 data: 3
13381 247/300 Write block 416 len 8192 data: 2 2
13382 248/300 Flush
13383 249/300 Read block 1394 len 16384
13384 250/300 Write block 2920 len 36864 data: 3 4 4 4 4 4 4 4 4
13385 251/300 Read block 14 len 28672
13386 252/300 Read block 1044 len 28672
13387 253/300 Read block 4932 len 16384
13388 254/300 Read block 2277 len 36864
13389 255/300 Write block 2897 len 40960 data: 2 3 3 3 3 3 3 3 3 3
13390 256/300 Read block 1229 len 16384
13391 257/300 Write block 2965 len 40960 data: 5 3 3 2 2 2 4 4 5 5
13392 258/300 Write block 4442 len 20480 data: 4 4 3 3 3
13393 259/300 Write block 4020 len 40960 data: 3 3 2 2 2 2 3 3 3 5
13394 260/300 Read block 581 len 12288
13395 261/300 Write block 4008 len 4096 data: 3
13396 262/300 Write block 2096 len 32768 data: 4 4 4 4 3 4 4 3
13397 263/300 Read block 1289 len 24576
13398 264/300 Write block 464 len 24576 data: 8 8 8 6 5 4
13399 265/300 Write block 4456 len 32768 data: 4 3 4 4 4 4 3 3
13400 266/300 Write block 2280 len 32768 data: 5 5 5 5 4 3 4 4
13401 267/300 Read block 2171 len 36864
13402 268/300 Read block 2834 len 32768
13403 269/300 Read block 3270 len 40960
13404 270/300 Read block 2135 len 8192
13405 271/300 Write block 2110 len 32768 data: 4 3 5 4 4 4 4 4
13406 272/300 Write block 2054 len 40960 data: 5 5 5 5 5 5 5 5 6 5
13407 273/300 Write block 1822 len 32768 data: 4 3 3 6 5 6 5 5
13408 274/300 Read block 978 len 36864
13409 275/300 Read block 3416 len 28672
13410 276/300 Flush
13411 277/300 Flush
13412 278/300 Read block 3356 len 12288
13413 279/300 Read block 4215 len 8192
13414 280/300 Read block 1393 len 12288
13415 281/300 Write block 1965 len 24576 data: 6 5 5 3 4 4
13416 282/300 Write block 4048 len 36864 data: 4 5 3 3 3 3 4 3 2
13417 283/300 Flush
13418 284/300 Read block 2640 len 40960
13419 285/300 Read block 740 len 40960
13420 286/300 Write block 3665 len 40960 data: 5 5 6 6 7 6 6 5 5 4
13421 287/300 Read block 2313 len 20480
13422 288/300 Write block 3887 len 32768 data: 3 3 3 3 3 3 3 3
13423 289/300 Write block 1481 len 28672 data: 3 2 2 2 2 2 4
13424 290/300 Read block 611 len 20480
13425 291/300 Write block 2293 len 24576 data: 4 4 3 2 2 2
13426 292/300 Read block 622 len 16384
13427 293/300 Read block 4916 len 12288
13428 294/300 Read block 993 len 24576
13429 295/300 Read block 1651 len 32768
13430 296/300 Read block 3221 len 12288
13431 297/300 Read block 2231 len 16384
13432 298/300 Read block 693 len 4096
13433 299/300 Read block 3870 len 40960
13434 300/300 Write block 3672 len 40960 data: 6 6 5 4 3 4 4 3 4 4
13435 301/300 Read block 32 len 20480
13436 Main task has work to do, go find it
13437 got dsc Start(1)
13438 start 1
13439 Jan 23 17:40:34.878 INFO request completed, latency_us: 197, response_code: 204, uri: /start/cid/1, method: POST, req_id: 3fa646ee-9ef4-4095-8f91-a546ebe7104b, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
13440 Replay: started 1, returned:Ok(())
13441 ----------------------------------------------------------------
13442 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13443 States: Active Offline Active
13444 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13445 4624 Acked Write 10 Done Sent Done false
13446 4625 Acked Read 6 Done Sent Done false
13447 4626 Acked Read 8 Done Sent Done false
13448 4627 Acked Read 5 Done Sent Done false
13449 4628 Acked Read 1 Done Sent Done false
13450 4629 Acked Write 9 Done Sent Done false
13451 4630 Acked Write 4 Done Sent Done false
13452 4631 Acked Read 9 Done Sent Done false
13453 4632 Acked Read 1 Done Sent Done false
13454 4633 Acked Write 8 Done Sent Done false
13455 4634 Acked Write 10 Done Sent Done false
13456 4635 Acked Read 9 Done Sent Done false
13457 4636 Acked Write 10 Done Sent Done false
13458 4637 Acked Read 4 Done Sent Done false
13459 4638 Acked Write 3 Done Sent Done false
13460 4639 Acked Flush 0 Done Sent Done false
13461 4640 Acked Read 8 Done Sent Done false
13462 4641 Acked Write 5 Done Sent Done false
13463 4642 Acked Flush 0 Done Sent Done false
13464 4643 Acked Read 2 Done Sent Done false
13465 4644 Acked Read 5 Done Sent Done false
13466 4645 Acked Read 7 Done Sent Done false
13467 4646 Acked Read 2 Done Sent Done false
13468 4647 Acked Write 8 Done Sent Done false
13469 4648 Acked Write 3 Done Sent Done false
13470 4649 Acked Read 3 Done Sent Done false
13471 4650 Acked Write 8 Done Sent Done false
13472 4651 Acked Flush 0 Done Sent Done false
13473 4652 Acked Write 2 Done Sent Done false
13474 4653 Acked Write 7 Done Sent Done false
13475 4654 Acked Write 5 Done Sent Done false
13476 4655 Acked Flush 0 Done Sent Done false
13477 4656 Acked Read 9 Done Sent Done false
13478 4657 Acked Write 9 Done Sent Done false
13479 4658 Acked Read 10 Done Sent Done false
13480 4659 Acked Read 1 Done Sent Done false
13481 4660 Acked Read 3 Done Sent Done false
13482 4661 Acked Read 3 Done Sent Done false
13483 4662 Acked Read 2 Done Sent Done false
13484 4663 Acked Read 9 Done Sent Done false
13485 4664 Acked Read 6 Done Sent Done false
13486 4665 Acked Flush 0 Done Sent Done false
13487 4666 Acked Write 10 Done Sent Done false
13488 4667 Acked Read 5 Done Sent Done false
13489 4668 Acked Write 10 Done Sent Done false
13490 4669 Acked Write 9 Done Sent Done false
13491 4670 Acked Read 10 Done Sent Done false
13492 4671 Acked Read 4 Done Sent Done false
13493 4672 Acked Write 4 Done Sent Done false
13494 4673 Acked Write 7 Done Sent Done false
13495 4674 Acked Read 9 Done Sent Done false
13496 4675 Acked Read 7 Done Sent Done false
13497 4676 Acked Write 6 Done Sent Done false
13498 4677 Acked Read 2 Done Sent Done false
13499 4678 Acked Write 5 Done Sent Done false
13500 4679 Acked Read 9 Done Sent Done false
13501 4680 Acked Write 9 Done Sent Done false
13502 4681 Acked Read 4 Done Sent Done false
13503 4682 Acked Write 4 Done Sent Done false
13504 4683 Acked Read 6 Done Sent Done false
13505 4684 Acked Read 7 Done Sent Done false
13506 4685 Acked Read 10 Done Sent Done false
13507 4686 Acked Read 10 Done Sent Done false
13508 4687 Acked Flush 0 Done Sent Done false
13509 4688 Acked Read 3 Done Sent Done false
13510 4689 Acked Read 5 Done Sent Done false
13511 4690 Acked Read 4 Done Sent Done false
13512 4691 Acked Read 3 Done Sent Done false
13513 4692 Acked Write 6 Done Sent Done false
13514 4693 Acked Flush 0 Done Sent Done false
13515 4694 Acked Flush 0 Done Sent Done false
13516 4695 Acked Write 9 Done Sent Done false
13517 4696 Acked Write 3 Done Sent Done false
13518 4697 Acked Write 8 Done Sent Done false
13519 4698 Acked Flush 0 Done Sent Done false
13520 4699 Acked Flush 0 Done Sent Done false
13521 4700 Acked Read 4 Done Sent Done false
13522 4701 Acked Read 2 Done Sent Done false
13523 4702 Acked Read 8 Done Sent Done false
13524 4703 Acked Read 9 Done Sent Done false
13525 4704 Acked Read 3 Done Sent Done false
13526 4705 Acked Read 6 Done Sent Done false
13527 4706 Acked Read 1 Done Sent Done false
13528 4707 Acked Read 1 Done Sent Done false
13529 4708 Acked Read 10 Done Sent Done false
13530 4709 Acked Write 9 Done Sent Done false
13531 4710 Acked Write 4 Done Sent Done false
13532 4711 Acked Read 2 Done Sent Done false
13533 4712 Acked Read 6 Done Sent Done false
13534 4713 Acked Read 3 Done Sent Done false
13535 4714 Acked Read 10 Done Sent Done false
13536 4715 Acked Flush 0 Done Sent Done false
13537 4716 Acked Flush 0 Done Sent Done false
13538 4717 Acked Write 6 Done Sent Done false
13539 4718 Acked Read 4 Done Sent Done false
13540 4719 Acked Write 5 Done Sent Done false
13541 4720 Acked Read 6 Done Sent Done false
13542 4721 Acked Write 9 Done Sent Done false
13543 4722 Acked Read 2 Done Sent Done false
13544 4723 Acked Read 3 Done Sent Done false
13545 4724 Acked Write 7 Done Sent Done false
13546 4725 Acked Flush 0 Done Sent Done false
13547 4726 Acked Flush 0 Done Sent Done false
13548 4727 Acked Write 3 Done Sent Done false
13549 4728 Acked Write 5 Done Sent Done false
13550 4729 Acked Read 10 Done Sent Done false
13551 4730 Acked Flush 0 Done Sent Done false
13552 4731 Acked Read 7 Done Sent Done false
13553 4732 Acked Read 5 Done Sent Done false
13554 4733 Acked Read 7 Done Sent Done false
13555 4734 Acked Read 5 Done Sent Done false
13556 4735 Acked Read 2 Done Sent Done false
13557 4736 Acked Read 1 Done Sent Done false
13558 4737 Acked Write 2 Done Sent Done false
13559 4738 Acked Flush 0 Done Sent Done false
13560 4739 Acked Read 3 Done Sent Done false
13561 4740 Acked Flush 0 Done Sent Done false
13562 4741 Acked Write 3 Done Sent Done false
13563 4742 Acked Flush 0 Done Sent Done false
13564 4743 Acked Read 6 Done Sent Done false
13565 4744 Acked Read 10 Done Sent Done false
13566 4745 Acked Flush 0 Done Sent Done false
13567 4746 Acked Write 7 Done Sent Done false
13568 4747 Acked Write 2 Done Sent Done false
13569 4748 Acked Read 8 Done Sent Done false
13570 4749 Acked Read 1 Done Sent Done false
13571 4750 Acked Read 9 Done Sent Done false
13572 4751 Acked Write 3 Done Sent Done false
13573 4752 Acked Write 9 Done Sent Done false
13574 4753 Acked Write 8 Done Sent Done false
13575 4754 Acked Read 9 Done Sent Done false
13576 4755 Acked Read 9 Done Sent Done false
13577 4756 Acked Read 9 Done Sent Done false
13578 4757 Acked Write 7 Done Sent Done false
13579 4758 Acked Write 1 Done Sent Done false
13580 4759 Acked Read 9 Done Sent Done false
13581 4760 Acked Write 4 Done Sent Done false
13582 4761 Acked Read 8 Done Sent Done false
13583 4762 Acked Write 7 Done Sent Done false
13584 4763 Acked Write 8 Done Sent Done false
13585 4764 Acked Flush 0 Done Sent Done false
13586 4765 Acked Write 10 Done Sent Done false
13587 4766 Acked Write 3 Done Sent Done false
13588 4767 Acked Write 1 Done Sent Done false
13589 4768 Acked Read 3 Done Sent Done false
13590 4769 Acked Read 3 Done Sent Done false
13591 4770 Acked Read 10 Done Sent Done false
13592 4771 Acked Read 2 Done Sent Done false
13593 4772 Acked Write 6 Done Sent Done false
13594 4773 Acked Write 3 Done Sent Done false
13595 4774 Acked Write 10 Done Sent Done false
13596 4775 Acked Read 4 Done Sent Done false
13597 4776 Acked Read 5 Done Sent Done false
13598 4777 Acked Flush 0 Done Sent Done false
13599 4778 Acked Write 3 Done Sent Done false
13600 4779 Acked Write 6 Done Sent Done false
13601 4780 Acked Read 4 Done Sent Done false
13602 4781 Acked Write 1 Done Sent Done false
13603 4782 Acked Read 3 Done Sent Done false
13604 4783 Acked Write 6 Done Sent Done false
13605 4784 Acked Read 6 Done Sent Done false
13606 4785 Acked Write 1 Done Sent Done false
13607 4786 Acked Read 3 Done Sent Done false
13608 4787 Acked Write 10 Done Sent Done false
13609 4788 Acked Read 4 Done Sent Done false
13610 4789 Acked Flush 0 Done Sent Done false
13611 4790 Acked Read 2 Done Sent Done false
13612 4791 Acked Flush 0 Done Sent Done false
13613 4792 Acked Read 1 Done Sent Done false
13614 4793 Acked Read 5 Done Sent Done false
13615 4794 Acked Write 2 Done Sent Done false
13616 4795 Acked Write 1 Done Sent Done false
13617 4796 Acked Read 6 Done Sent Done false
13618 4797 Acked Read 7 Done Sent Done false
13619 4798 Acked Write 2 Done Sent Done false
13620 4799 Acked Read 7 Done Sent Done false
13621 4800 Acked Read 1 Done Sent Done false
13622 4801 Acked Read 1 Done Sent Done false
13623 4802 Acked Write 6 Done Sent Done false
13624 4803 Acked Read 5 Done Sent Done false
13625 4804 Acked Write 9 Done Sent Done false
13626 4805 Acked Write 9 Done Sent Done false
13627 4806 Acked Read 3 Done Sent Done false
13628 4807 Acked Read 3 Done Sent Done false
13629 4808 Acked Write 8 Done Sent Done false
13630 4809 Acked Write 10 Done Sent Done false
13631 4810 Acked Read 2 Done Sent Done false
13632 4811 Acked Read 8 Done Sent Done false
13633 4812 Acked Write 10 Done Sent Done false
13634 4813 Acked Flush 0 Done Sent Done false
13635 4814 Acked Write 10 Done Sent Done false
13636 4815 Acked Write 1 Done Sent Done false
13637 4816 Acked Write 3 Done Sent Done false
13638 4817 Acked Flush 0 Done Sent Done false
13639 4818 Acked Read 9 Done Sent Done false
13640 4819 Acked Read 5 Done Sent Done false
13641 4820 Acked Read 5 Done Sent Done false
13642 4821 Acked Write 1 Done Sent Done false
13643 4822 Acked Read 4 Done Sent Done false
13644 4823 Acked Read 3 Done Sent Done false
13645 4824 Acked Flush 0 Done Sent Done false
13646 4825 Acked Write 6 Done Sent Done false
13647 4826 Acked Write 6 Done Sent Done false
13648 4827 Acked Write 9 Done Sent Done false
13649 4828 Acked Read 7 Done Sent Done false
13650 4829 Acked Write 3 Done Sent Done false
13651 4830 Acked Write 2 Done Sent Done false
13652 4831 Acked Write 8 Done Sent Done false
13653 4832 Acked Flush 0 Done Sent Done false
13654 4833 Acked Read 5 Done Sent Done false
13655 4834 Acked Write 8 Done Sent Done false
13656 4835 Acked Flush 0 Done Sent Done false
13657 4836 Acked Write 8 Done Sent Done false
13658 4837 Acked Write 8 Done Sent Done false
13659 4838 Acked Write 4 Done Sent Done false
13660 4839 Acked Write 1 Done Sent Done false
13661 4840 Acked Read 5 Done Sent Done false
13662 4841 Acked Write 9 Done Sent Done false
13663 4842 Acked Read 8 Done Sent Done false
13664 4843 Acked Write 3 Done Sent Done false
13665 4844 Acked Read 3 Done Sent Done false
13666 4845 Acked Read 5 Done Sent Done false
13667 4846 Acked Write 2 Done Sent Done false
13668 4847 Acked Read 2 Done Sent Done false
13669 4848 Acked Write 8 Done Sent Done false
13670 4849 Acked Read 2 Done Sent Done false
13671 4850 Acked Read 4 Done Sent Done false
13672 4851 Acked Read 1 Done Sent Done false
13673 4852 Acked Write 7 Done Sent Done false
13674 4853 Acked Write 10 Done Sent Done false
13675 4854 Acked Write 10 Done Sent Done false
13676 4855 Acked Read 4 Done Sent Done false
13677 4856 Acked Write 4 Done Sent Done false
13678 4857 Acked Flush 0 Done Sent Done false
13679 4858 Acked Read 7 Done Sent Done false
13680 4859 Acked Write 9 Done Sent Done false
13681 4860 Acked Write 8 Done Sent Done false
13682 4861 Acked Write 9 Done Sent Done false
13683 4862 Acked Read 4 Done Sent Done false
13684 4863 Acked Read 1 Done Sent Done false
13685 4864 Acked Write 4 Done Sent Done false
13686 4865 Acked Write 7 Done Sent Done false
13687 4866 Acked Read 8 Done Sent Done false
13688 4867 Acked Read 1 Done Sent Done false
13689 4868 Acked Read 3 Done Sent Done false
13690 4869 Acked Write 1 Done Sent Done false
13691 4870 Acked Write 2 Done Sent Done false
13692 4871 Acked Flush 0 Done Sent Done false
13693 4872 Acked Read 4 Done Sent Done false
13694 4873 Acked Write 9 Done Sent Done false
13695 4874 Acked Read 7 Done Sent Done false
13696 4875 Acked Read 7 Done Sent Done false
13697 4876 Acked Read 4 Done Sent Done false
13698 4877 Acked Read 9 Done Sent Done false
13699 4878 Acked Write 10 Done Sent Done false
13700 4879 Acked Read 4 Done Sent Done false
13701 4880 Acked Write 10 Done Sent Done false
13702 4881 Acked Write 5 Done Sent Done false
13703 4882 Acked Write 10 Done Sent Done false
13704 4883 Acked Read 3 Done Sent Done false
13705 4884 Acked Write 1 Done Sent Done false
13706 4885 Acked Write 8 Done Sent Done false
13707 4886 Acked Read 6 Done Sent Done false
13708 4887 Acked Write 6 Done Sent Done false
13709 4888 Acked Write 8 Done Sent Done false
13710 4889 Acked Write 8 Done Sent Done false
13711 4890 Acked Read 9 Done Sent Done false
13712 4891 Acked Read 8 Done Sent Done false
13713 4892 Acked Read 10 Done Sent Done false
13714 4893 Acked Read 2 Done Sent Done false
13715 4894 Acked Write 8 Done Sent Done false
13716 4895 Acked Write 10 Done Sent Done false
13717 4896 Acked Write 8 Done Sent Done false
13718 4897 Acked Read 9 Done Sent Done false
13719 4898 Acked Read 7 Done Sent Done false
13720 4899 Acked Flush 0 Done Sent Done false
13721 4900 Acked Flush 0 Done Sent Done false
13722 4901 Acked Read 3 Done Sent Done false
13723 4902 Acked Read 2 Done Sent Done false
13724 4903 Acked Read 3 Done Sent Done false
13725 4904 Acked Write 6 Done Sent Done false
13726 4905 Acked Write 9 Done Sent Done false
13727 4906 Acked Flush 0 Done Sent Done false
13728 4907 Acked Read 10 Done Sent Done false
13729 4908 Acked Read 10 Done Sent Done false
13730 4909 Acked Write 10 Done Sent Done false
13731 4910 Acked Read 5 Done Sent Done false
13732 4911 Acked Write 8 Done Sent Done false
13733 4912 Acked Write 7 Done Sent Done false
13734 4913 Acked Read 5 Done Sent Done false
13735 4914 Acked Write 6 Done Sent Done false
13736 4915 Acked Read 4 Done Sent Done false
13737 4916 Acked Read 3 Done Sent Done false
13738 4917 Acked Read 6 Done Sent Done false
13739 4918 Acked Read 8 Done Sent Done false
13740 4919 Acked Read 3 Done Sent Done false
13741 4920 Acked Read 4 Done Sent Done false
13742 4921 Acked Read 1 Done Sent Done false
13743 4922 Acked Read 10 Done Sent Done false
13744 4923 Acked Write 10 Done Sent Done false
13745 4924 Acked Read 5 Done Sent Done false
13746 STATES DS:0 DS:1 DS:2 TOTAL
13747 Sent 0 301 0 301
13748 Done 301 0 301 602
13749 Skipped 0 0 0 0
13750 Error 0 0 0 0
13751 Last Flush: Some(JobId(4906)) Some(JobId(4623)) Some(JobId(4906))
13752 Downstairs last five completed: 4623 4622 4621 4620 4619
13753 Upstairs last five completed: 4924 4923 4922 4921 4920
13754 CLIENT: Up:0 ds:301 act:2
13755 [8820] got start action
13756 [8820] I am going to restart
13757 Starting downstairs at port 8820
13758 Make output file at "/tmp/dsc/downstairs-8820.txt"
13759 [8820][1] reports Starting
13760 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1218)
13761 [8820][1] reports Running
13762 ----------------------------------------------------------------
13763 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
13764 States: Active Offline Active
13765 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13766 4624 Acked Write 10 Done Sent Done false
13767 4625 Acked Read 6 Done Sent Done false
13768 4626 Acked Read 8 Done Sent Done false
13769 4627 Acked Read 5 Done Sent Done false
13770 4628 Acked Read 1 Done Sent Done false
13771 4629 Acked Write 9 Done Sent Done false
13772 4630 Acked Write 4 Done Sent Done false
13773 4631 Acked Read 9 Done Sent Done false
13774 4632 Acked Read 1 Done Sent Done false
13775 4633 Acked Write 8 Done Sent Done false
13776 4634 Acked Write 10 Done Sent Done false
13777 4635 Acked Read 9 Done Sent Done false
13778 4636 Acked Write 10 Done Sent Done false
13779 4637 Acked Read 4 Done Sent Done false
13780 4638 Acked Write 3 Done Sent Done false
13781 4639 Acked Flush 0 Done Sent Done false
13782 4640 Acked Read 8 Done Sent Done false
13783 4641 Acked Write 5 Done Sent Done false
13784 4642 Acked Flush 0 Done Sent Done false
13785 4643 Acked Read 2 Done Sent Done false
13786 4644 Acked Read 5 Done Sent Done false
13787 4645 Acked Read 7 Done Sent Done false
13788 4646 Acked Read 2 Done Sent Done false
13789 4647 Acked Write 8 Done Sent Done false
13790 4648 Acked Write 3 Done Sent Done false
13791 4649 Acked Read 3 Done Sent Done false
13792 4650 Acked Write 8 Done Sent Done false
13793 4651 Acked Flush 0 Done Sent Done false
13794 4652 Acked Write 2 Done Sent Done false
13795 4653 Acked Write 7 Done Sent Done false
13796 4654 Acked Write 5 Done Sent Done false
13797 4655 Acked Flush 0 Done Sent Done false
13798 4656 Acked Read 9 Done Sent Done false
13799 4657 Acked Write 9 Done Sent Done false
13800 4658 Acked Read 10 Done Sent Done false
13801 4659 Acked Read 1 Done Sent Done false
13802 4660 Acked Read 3 Done Sent Done false
13803 4661 Acked Read 3 Done Sent Done false
13804 4662 Acked Read 2 Done Sent Done false
13805 4663 Acked Read 9 Done Sent Done false
13806 4664 Acked Read 6 Done Sent Done false
13807 4665 Acked Flush 0 Done Sent Done false
13808 4666 Acked Write 10 Done Sent Done false
13809 4667 Acked Read 5 Done Sent Done false
13810 4668 Acked Write 10 Done Sent Done false
13811 4669 Acked Write 9 Done Sent Done false
13812 4670 Acked Read 10 Done Sent Done false
13813 4671 Acked Read 4 Done Sent Done false
13814 4672 Acked Write 4 Done Sent Done false
13815 4673 Acked Write 7 Done Sent Done false
13816 4674 Acked Read 9 Done Sent Done false
13817 4675 Acked Read 7 Done Sent Done false
13818 4676 Acked Write 6 Done Sent Done false
13819 4677 Acked Read 2 Done Sent Done false
13820 4678 Acked Write 5 Done Sent Done false
13821 4679 Acked Read 9 Done Sent Done false
13822 4680 Acked Write 9 Done Sent Done false
13823 4681 Acked Read 4 Done Sent Done false
13824 4682 Acked Write 4 Done Sent Done false
13825 4683 Acked Read 6 Done Sent Done false
13826 4684 Acked Read 7 Done Sent Done false
13827 4685 Acked Read 10 Done Sent Done false
13828 4686 Acked Read 10 Done Sent Done false
13829 4687 Acked Flush 0 Done Sent Done false
13830 4688 Acked Read 3 Done Sent Done false
13831 4689 Acked Read 5 Done Sent Done false
13832 4690 Acked Read 4 Done Sent Done false
13833 4691 Acked Read 3 Done Sent Done false
13834 4692 Acked Write 6 Done Sent Done false
13835 4693 Acked Flush 0 Done Sent Done false
13836 4694 Acked Flush 0 Done Sent Done false
13837 4695 Acked Write 9 Done Sent Done false
13838 4696 Acked Write 3 Done Sent Done false
13839 4697 Acked Write 8 Done Sent Done false
13840 4698 Acked Flush 0 Done Sent Done false
13841 4699 Acked Flush 0 Done Sent Done false
13842 4700 Acked Read 4 Done Sent Done false
13843 4701 Acked Read 2 Done Sent Done false
13844 4702 Acked Read 8 Done Sent Done false
13845 4703 Acked Read 9 Done Sent Done false
13846 4704 Acked Read 3 Done Sent Done false
13847 4705 Acked Read 6 Done Sent Done false
13848 4706 Acked Read 1 Done Sent Done false
13849 4707 Acked Read 1 Done Sent Done false
13850 4708 Acked Read 10 Done Sent Done false
13851 4709 Acked Write 9 Done Sent Done false
13852 4710 Acked Write 4 Done Sent Done false
13853 4711 Acked Read 2 Done Sent Done false
13854 4712 Acked Read 6 Done Sent Done false
13855 4713 Acked Read 3 Done Sent Done false
13856 4714 Acked Read 10 Done Sent Done false
13857 4715 Acked Flush 0 Done Sent Done false
13858 4716 Acked Flush 0 Done Sent Done false
13859 4717 Acked Write 6 Done Sent Done false
13860 4718 Acked Read 4 Done Sent Done false
13861 4719 Acked Write 5 Done Sent Done false
13862 4720 Acked Read 6 Done Sent Done false
13863 4721 Acked Write 9 Done Sent Done false
13864 4722 Acked Read 2 Done Sent Done false
13865 4723 Acked Read 3 Done Sent Done false
13866 4724 Acked Write 7 Done Sent Done false
13867 4725 Acked Flush 0 Done Sent Done false
13868 4726 Acked Flush 0 Done Sent Done false
13869 4727 Acked Write 3 Done Sent Done false
13870 4728 Acked Write 5 Done Sent Done false
13871 4729 Acked Read 10 Done Sent Done false
13872 4730 Acked Flush 0 Done Sent Done false
13873 4731 Acked Read 7 Done Sent Done false
13874 4732 Acked Read 5 Done Sent Done false
13875 4733 Acked Read 7 Done Sent Done false
13876 4734 Acked Read 5 Done Sent Done false
13877 4735 Acked Read 2 Done Sent Done false
13878 4736 Acked Read 1 Done Sent Done false
13879 4737 Acked Write 2 Done Sent Done false
13880 4738 Acked Flush 0 Done Sent Done false
13881 4739 Acked Read 3 Done Sent Done false
13882 4740 Acked Flush 0 Done Sent Done false
13883 4741 Acked Write 3 Done Sent Done false
13884 4742 Acked Flush 0 Done Sent Done false
13885 4743 Acked Read 6 Done Sent Done false
13886 4744 Acked Read 10 Done Sent Done false
13887 4745 Acked Flush 0 Done Sent Done false
13888 4746 Acked Write 7 Done Sent Done false
13889 4747 Acked Write 2 Done Sent Done false
13890 4748 Acked Read 8 Done Sent Done false
13891 4749 Acked Read 1 Done Sent Done false
13892 4750 Acked Read 9 Done Sent Done false
13893 4751 Acked Write 3 Done Sent Done false
13894 4752 Acked Write 9 Done Sent Done false
13895 4753 Acked Write 8 Done Sent Done false
13896 4754 Acked Read 9 Done Sent Done false
13897 4755 Acked Read 9 Done Sent Done false
13898 4756 Acked Read 9 Done Sent Done false
13899 4757 Acked Write 7 Done Sent Done false
13900 4758 Acked Write 1 Done Sent Done false
13901 4759 Acked Read 9 Done Sent Done false
13902 4760 Acked Write 4 Done Sent Done false
13903 4761 Acked Read 8 Done Sent Done false
13904 4762 Acked Write 7 Done Sent Done false
13905 4763 Acked Write 8 Done Sent Done false
13906 4764 Acked Flush 0 Done Sent Done false
13907 4765 Acked Write 10 Done Sent Done false
13908 4766 Acked Write 3 Done Sent Done false
13909 4767 Acked Write 1 Done Sent Done false
13910 4768 Acked Read 3 Done Sent Done false
13911 4769 Acked Read 3 Done Sent Done false
13912 4770 Acked Read 10 Done Sent Done false
13913 4771 Acked Read 2 Done Sent Done false
13914 4772 Acked Write 6 Done Sent Done false
13915 4773 Acked Write 3 Done Sent Done false
13916 4774 Acked Write 10 Done Sent Done false
13917 4775 Acked Read 4 Done Sent Done false
13918 4776 Acked Read 5 Done Sent Done false
13919 4777 Acked Flush 0 Done Sent Done false
13920 4778 Acked Write 3 Done Sent Done false
13921 4779 Acked Write 6 Done Sent Done false
13922 4780 Acked Read 4 Done Sent Done false
13923 4781 Acked Write 1 Done Sent Done false
13924 4782 Acked Read 3 Done Sent Done false
13925 4783 Acked Write 6 Done Sent Done false
13926 4784 Acked Read 6 Done Sent Done false
13927 4785 Acked Write 1 Done Sent Done false
13928 4786 Acked Read 3 Done Sent Done false
13929 4787 Acked Write 10 Done Sent Done false
13930 4788 Acked Read 4 Done Sent Done false
13931 4789 Acked Flush 0 Done Sent Done false
13932 4790 Acked Read 2 Done Sent Done false
13933 4791 Acked Flush 0 Done Sent Done false
13934 4792 Acked Read 1 Done Sent Done false
13935 4793 Acked Read 5 Done Sent Done false
13936 4794 Acked Write 2 Done Sent Done false
13937 4795 Acked Write 1 Done Sent Done false
13938 4796 Acked Read 6 Done Sent Done false
13939 4797 Acked Read 7 Done Sent Done false
13940 4798 Acked Write 2 Done Sent Done false
13941 4799 Acked Read 7 Done Sent Done false
13942 4800 Acked Read 1 Done Sent Done false
13943 4801 Acked Read 1 Done Sent Done false
13944 4802 Acked Write 6 Done Sent Done false
13945 4803 Acked Read 5 Done Sent Done false
13946 4804 Acked Write 9 Done Sent Done false
13947 4805 Acked Write 9 Done Sent Done false
13948 4806 Acked Read 3 Done Sent Done false
13949 4807 Acked Read 3 Done Sent Done false
13950 4808 Acked Write 8 Done Sent Done false
13951 4809 Acked Write 10 Done Sent Done false
13952 4810 Acked Read 2 Done Sent Done false
13953 4811 Acked Read 8 Done Sent Done false
13954 4812 Acked Write 10 Done Sent Done false
13955 4813 Acked Flush 0 Done Sent Done false
13956 4814 Acked Write 10 Done Sent Done false
13957 4815 Acked Write 1 Done Sent Done false
13958 4816 Acked Write 3 Done Sent Done false
13959 4817 Acked Flush 0 Done Sent Done false
13960 4818 Acked Read 9 Done Sent Done false
13961 4819 Acked Read 5 Done Sent Done false
13962 4820 Acked Read 5 Done Sent Done false
13963 4821 Acked Write 1 Done Sent Done false
13964 4822 Acked Read 4 Done Sent Done false
13965 4823 Acked Read 3 Done Sent Done false
13966 4824 Acked Flush 0 Done Sent Done false
13967 4825 Acked Write 6 Done Sent Done false
13968 4826 Acked Write 6 Done Sent Done false
13969 4827 Acked Write 9 Done Sent Done false
13970 4828 Acked Read 7 Done Sent Done false
13971 4829 Acked Write 3 Done Sent Done false
13972 4830 Acked Write 2 Done Sent Done false
13973 4831 Acked Write 8 Done Sent Done false
13974 4832 Acked Flush 0 Done Sent Done false
13975 4833 Acked Read 5 Done Sent Done false
13976 4834 Acked Write 8 Done Sent Done false
13977 4835 Acked Flush 0 Done Sent Done false
13978 4836 Acked Write 8 Done Sent Done false
13979 4837 Acked Write 8 Done Sent Done false
13980 4838 Acked Write 4 Done Sent Done false
13981 4839 Acked Write 1 Done Sent Done false
13982 4840 Acked Read 5 Done Sent Done false
13983 4841 Acked Write 9 Done Sent Done false
13984 4842 Acked Read 8 Done Sent Done false
13985 4843 Acked Write 3 Done Sent Done false
13986 4844 Acked Read 3 Done Sent Done false
13987 4845 Acked Read 5 Done Sent Done false
13988 4846 Acked Write 2 Done Sent Done false
13989 4847 Acked Read 2 Done Sent Done false
13990 4848 Acked Write 8 Done Sent Done false
13991 4849 Acked Read 2 Done Sent Done false
13992 4850 Acked Read 4 Done Sent Done false
13993 4851 Acked Read 1 Done Sent Done false
13994 4852 Acked Write 7 Done Sent Done false
13995 4853 Acked Write 10 Done Sent Done false
13996 4854 Acked Write 10 Done Sent Done false
13997 4855 Acked Read 4 Done Sent Done false
13998 4856 Acked Write 4 Done Sent Done false
13999 4857 Acked Flush 0 Done Sent Done false
14000 4858 Acked Read 7 Done Sent Done false
14001 4859 Acked Write 9 Done Sent Done false
14002 4860 Acked Write 8 Done Sent Done false
14003 4861 Acked Write 9 Done Sent Done false
14004 4862 Acked Read 4 Done Sent Done false
14005 4863 Acked Read 1 Done Sent Done false
14006 4864 Acked Write 4 Done Sent Done false
14007 4865 Acked Write 7 Done Sent Done false
14008 4866 Acked Read 8 Done Sent Done false
14009 4867 Acked Read 1 Done Sent Done false
14010 4868 Acked Read 3 Done Sent Done false
14011 4869 Acked Write 1 Done Sent Done false
14012 4870 Acked Write 2 Done Sent Done false
14013 4871 Acked Flush 0 Done Sent Done false
14014 4872 Acked Read 4 Done Sent Done false
14015 4873 Acked Write 9 Done Sent Done false
14016 4874 Acked Read 7 Done Sent Done false
14017 4875 Acked Read 7 Done Sent Done false
14018 4876 Acked Read 4 Done Sent Done false
14019 4877 Acked Read 9 Done Sent Done false
14020 4878 Acked Write 10 Done Sent Done false
14021 4879 Acked Read 4 Done Sent Done false
14022 4880 Acked Write 10 Done Sent Done false
14023 4881 Acked Write 5 Done Sent Done false
14024 4882 Acked Write 10 Done Sent Done false
14025 4883 Acked Read 3 Done Sent Done false
14026 4884 Acked Write 1 Done Sent Done false
14027 4885 Acked Write 8 Done Sent Done false
14028 4886 Acked Read 6 Done Sent Done false
14029 4887 Acked Write 6 Done Sent Done false
14030 4888 Acked Write 8 Done Sent Done false
14031 4889 Acked Write 8 Done Sent Done false
14032 4890 Acked Read 9 Done Sent Done false
14033 4891 Acked Read 8 Done Sent Done false
14034 4892 Acked Read 10 Done Sent Done false
14035 4893 Acked Read 2 Done Sent Done false
14036 4894 Acked Write 8 Done Sent Done false
14037 4895 Acked Write 10 Done Sent Done false
14038 4896 Acked Write 8 Done Sent Done false
14039 4897 Acked Read 9 Done Sent Done false
14040 4898 Acked Read 7 Done Sent Done false
14041 4899 Acked Flush 0 Done Sent Done false
14042 4900 Acked Flush 0 Done Sent Done false
14043 4901 Acked Read 3 Done Sent Done false
14044 4902 Acked Read 2 Done Sent Done false
14045 4903 Acked Read 3 Done Sent Done false
14046 4904 Acked Write 6 Done Sent Done false
14047 4905 Acked Write 9 Done Sent Done false
14048 4906 Acked Flush 0 Done Sent Done false
14049 4907 Acked Read 10 Done Sent Done false
14050 4908 Acked Read 10 Done Sent Done false
14051 4909 Acked Write 10 Done Sent Done false
14052 4910 Acked Read 5 Done Sent Done false
14053 4911 Acked Write 8 Done Sent Done false
14054 4912 Acked Write 7 Done Sent Done false
14055 4913 Acked Read 5 Done Sent Done false
14056 4914 Acked Write 6 Done Sent Done false
14057 4915 Acked Read 4 Done Sent Done false
14058 4916 Acked Read 3 Done Sent Done false
14059 4917 Acked Read 6 Done Sent Done false
14060 4918 Acked Read 8 Done Sent Done false
14061 4919 Acked Read 3 Done Sent Done false
14062 4920 Acked Read 4 Done Sent Done false
14063 4921 Acked Read 1 Done Sent Done false
14064 4922 Acked Read 10 Done Sent Done false
14065 4923 Acked Write 10 Done Sent Done false
14066 4924 Acked Read 5 Done Sent Done false
14067 4925 Acked Flush 0 Done Sent Done false
14068 STATES DS:0 DS:1 DS:2 TOTAL
14069 Sent 0 302 0 302
14070 Done 302 0 302 604
14071 Skipped 0 0 0 0
14072 Error 0 0 0 0
14073 Last Flush: Some(JobId(4925)) Some(JobId(4623)) Some(JobId(4925))
14074 Downstairs last five completed: 4623 4622 4621 4620 4619
14075 Upstairs last five completed: 4925 4924 4923 4922 4921
14076 CLIENT: Up:0 ds:302 act:2
140772025-01-23T17:40:40.632ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
140782025-01-23T17:40:40.632ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48637) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
140792025-01-23T17:40:40.633ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
140802025-01-23T17:40:40.634ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
140812025-01-23T17:40:40.634ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
140822025-01-23T17:40:40.634ZINFOcrucible: send last flush ID to this DS: Some(JobId(4623)) = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
140832025-01-23T17:40:40.634ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
140842025-01-23T17:40:40.634ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(4623)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
140852025-01-23T17:40:40.634ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(4623)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
14086 ----------------------------------------------------------------
14087 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14088 Downstairs last five completed: 4925 4924 4923 4922 4921
14089 Upstairs last five completed: 4925 4924 4923 4922 4921
14090 CLIENT: Up:0 ds:0 act:3
14091 Replay: All jobs finished
14092 Main task has work to do, go find it
14093 got dsc Stop(2)
14094 stop 2
14095 Jan 23 17:40:42.885 INFO request completed, latency_us: [8830] Got stop action so:false kr:false
14096 371, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 5234068b-b530-46c6-a6ff-1c99ea13b062, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
140972025-01-23T17:40:42.885ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
140982025-01-23T17:40:42.885ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
140992025-01-23T17:40:42.885ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
14100 Jan 23 17:40:42.886 INFO request completed, latency_us: 143, response_code: 200, uri: /state/cid/2, method: GET, req_id: a7d29336-552c-4d6d-b783-e47ce41bcd0b, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
141012025-01-23T17:40:42.886ZINFOcrucible: client task is exiting = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
14102 [8830] Exited with: ExitStatus(unix_wait_status(9))
14103 [8830][2] reports Exit
14104 Jan 23 17:40:46.888 INFO request completed, latency_us: 216, response_code: 200, uri: /state/cid/2, method: GET, req_id: cec0be03-7294-40c1-9e56-abdb15cf2caa, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
14105 001/300 Write block 534 len 36864 data: 4 5 5 5 6 5 5 6 6
14106 002/300 Read block 257 len 28672
14107 003/300 Read block 498 len 36864
14108 004/300 Read block 4352 len 40960
14109 005/300 Read block 583 len 4096
14110 006/300 Read block 4167 len 4096
14111 007/300 Flush
14112 008/300 Read block 1204 len 32768
14113 009/300 Read block 4323 len 12288
14114 010/300 Read block 4640 len 36864
14115 011/300 Read block 441 len 20480
14116 012/300 Write block 3245 len 32768 data: 3 2 2 2 2 3 2 2
14117 013/300 Read block 3453 len 36864
14118 014/300 Read block 369 len 16384
14119 015/300 Read block 180 len 4096
14120 016/300 Flush
14121 017/300 Write block 4681 len 8192 data: 4 6
14122 018/300 Read block 3986 len 4096
14123 019/300 Write block 778 len 40960 data: 6 5 6 4 4 5 6 6 4 2
14124 020/300 Read block 2837 len 4096
14125 021/300 Write block 3702 len 16384 data: 5 6 6 5
14126 022/300 Write block 357 len 4096 data: 4
14127 023/300 Flush
14128 024/300 Read block 585 len 4096
14129 025/300 Write block 3352 len 28672 data: 3 4 3 2 2 2 2
14130 026/300 Read block 539 len 32768
14131 027/300 Read block 2800 len 36864
14132 028/300 Read block 718 len 36864
14133 029/300 Read block 512 len 40960
14134 030/300 Read block 1977 len 8192
14135 031/300 Write block 3447 len 28672 data: 3 2 2 2 2 2 2
14136 032/300 Write block 2804 len 20480 data: 4 4 4 3 3
14137 033/300 Read block 3185 len 28672
14138 034/300 Read block 1773 len 20480
14139 035/300 Read block 2875 len 36864
14140 036/300 Write block 2962 len 16384 data: 4 3 5 6
14141 037/300 Read block 232 len 20480
14142 038/300 Read block 56 len 32768
14143 039/300 Flush
14144 040/300 Flush
14145 041/300 Read block 3713 len 20480
14146 042/300 Read block 3838 len 36864
14147 043/300 Read block 1951 len 40960
14148 044/300 Read block 2851 len 4096
14149 045/300 Write block 1111 len 28672 data: 4 4 4 5 5 4 4
14150 046/300 Write block 2260 len 24576 data: 3 3 6 5 5 4
14151 047/300 Write block 4585 len 16384 data: 4 4 3 2
14152 048/300 Read block 3876 len 32768
14153 049/300 Read block 50 len 40960
14154 050/300 Write block 670 len 12288 data: 5 5 6
14155 051/300 Flush
14156 052/300 Write block 70 len 24576 data: 6 5 5 6 7 5
14157 053/300 Read block 3447 len 16384
14158 054/300 Read block 4178 len 32768
14159 055/300 Flush
14160 056/300 Read block 1281 len 40960
14161 057/300 Write block 3649 len 28672 data: 3 3 3 3 4 5 6
14162 058/300 Read block 1597 len 8192
14163 059/300 Read block 2752 len 40960
14164 060/300 Write block 224 len 24576 data: 2 2 2 2 2 2
14165 061/300 Read block 873 len 36864
14166 062/300 Write block 4652 len 32768 data: 2 2 2 2 2 2 2 2
14167 063/300 Write block 883 len 4096 data: 3
14168 064/300 Read block 2459 len 32768
14169 065/300 Write block 1568 len 24576 data: 5 4 4 3 3 3
14170 066/300 Write block 1791 len 24576 data: 3 3 3 3 3 4
14171 067/300 Flush
14172 068/300 Write block 909 len 20480 data: 4 4 4 5 5
14173 069/300 Read block 1254 len 12288
14174 070/300 Read block 3050 len 36864
14175 071/300 Read block 1097 len 4096
14176 072/300 Write block 3206 len 40960 data: 3 3 4 4 3 3 2 2 2 2
14177 073/300 Read block 4634 len 12288
14178 074/300 Flush
14179 075/300 Read block 150 len 36864
14180 076/300 Read block 369 len 20480
14181 077/300 Write block 3545 len 16384 data: 4 5 5 5
14182 078/300 Read block 7 len 16384
14183 079/300 Read block 865 len 16384
14184 080/300 Flush
14185 081/300 Read block 620 len 32768
14186 082/300 Write block 3685 len 32768 data: 5 3 3 3 3 2 3 4
14187 083/300 Read block 90 len 16384
14188 084/300 Read block 4177 len 16384
14189 085/300 Read block 360 len 24576
14190 086/300 Read block 4403 len 8192
14191 087/300 Read block 4044 len 32768
14192 088/300 Read block 3433 len 40960
14193 089/300 Read block 2060 len 16384
14194 090/300 Write block 4799 len 16384 data: 5 4 4 4
14195 091/300 Flush
14196 092/300 Write block 3878 len 16384 data: 3 4 4 4
14197 093/300 Write block 2958 len 8192 data: 3 3
14198 094/300 Read block 4026 len 28672
14199 095/300 Read block 1994 len 28672
14200 096/300 Read block 274 len 16384
14201 097/300 Read block 4354 len 40960
14202 098/300 Read block 2600 len 24576
14203 099/300 Write block 2383 len 8192 data: 6 7
14204 100/300 Read block 371 len 4096
14205 101/300 Read block 814 len 40960
14206 102/300 Read block 3907 len 28672
14207 103/300 Read block 2974 len 32768
14208 104/300 Read block 3195 len 40960
14209 105/300 Write block 1769 len 8192 data: 2 2
14210 106/300 Write block 2567 len 40960 data: 2 2 2 2 4 3 3 5 5 6
14211 107/300 Write block 2719 len 4096 data: 4
14212 108/300 Write block 2249 len 40960 data: 3 3 4 5 5 6 6 6 6 3
14213 109/300 Write block 4513 len 40960 data: 3 3 4 3 3 3 3 3 3 3
14214 110/300 Read block 2622 len 16384
14215 111/300 Write block 2365 len 20480 data: 2 2 3 3 3
14216 112/300 Flush
14217 113/300 Read block 699 len 8192
14218 114/300 Write block 2096 len 36864 data: 5 5 5 5 4 5 5 4 4
14219 115/300 Read block 1283 len 24576
14220 116/300 Read block 2070 len 12288
14221 117/300 Write block 1095 len 36864 data: 3 3 4 4 4 4 4 4 4
14222 118/300 Read block 1662 len 36864
14223 119/300 Write block 1857 len 8192 data: 4 3
14224 120/300 Write block 2218 len 36864 data: 5 5 4 3 5 5 4 3 5
14225 121/300 Write block 3520 len 4096 data: 3
14226 122/300 Flush
14227 123/300 Write block 3941 len 12288 data: 4 4 5
14228 124/300 Read block 4343 len 12288
14229 125/300 Read block 3067 len 12288
14230 126/300 Write block 240 len 28672 data: 3 4 4 3 4 4 4
14231 127/300 Write block 4874 len 4096 data: 5
14232 128/300 Flush
14233 129/300 Read block 4840 len 24576
14234 130/300 Read block 3314 len 32768
14235 131/300 Read block 893 len 28672
14236 132/300 Write block 2747 len 40960 data: 6 7 6 7 7 6 4 4 4 2
14237 133/300 Read block 1963 len 8192
14238 134/300 Write block 385 len 20480 data: 2 2 2 3 3
14239 135/300 Write block 1908 len 4096 data: 9
14240 136/300 Read block 601 len 12288
14241 137/300 Read block 3482 len 8192
14242 138/300 Read block 2058 len 24576
14243 139/300 Read block 3759 len 4096
14244 140/300 Read block 2035 len 28672
14245 141/300 Write block 12 len 20480 data: 2 3 3 3 3
14246 142/300 Read block 2695 len 20480
14247 143/300 Write block 161 len 12288 data: 5 4 3
14248 144/300 Write block 639 len 28672 data: 5 6 6 5 4 4 4
14249 145/300 Read block 2804 len 4096
14250 146/300 Read block 1929 len 16384
14251 147/300 Write block 1240 len 40960 data: 4 3 3 2 2 2 2 3 3 3
14252 148/300 Write block 2866 len 32768 data: 5 6 6 6 4 3 3 3
14253 149/300 Write block 3772 len 36864 data: 2 2 2 2 2 2 2 2 2
14254 150/300 Read block 3801 len 12288
14255 151/300 Write block 57 len 12288 data: 4 5 5
14256 152/300 Flush
14257 153/300 Flush
14258 154/300 Flush
14259 155/300 Write block 3043 len 24576 data: 4 5 3 3 3 3
14260 156/300 Read block 3721 len 8192
14261 157/300 Write block 1058 len 4096 data: 3
14262 158/300 Flush
14263 159/300 Read block 1801 len 20480
14264 160/300 Read block 1276 len 20480
14265 161/300 Read block 4869 len 16384
14266 162/300 Write block 2074 len 8192 data: 6 5
14267 163/300 Write block 610 len 28672 data: 3 3 3 3 3 3 4
14268 164/300 Write block 2380 len 4096 data: 6
14269 165/300 Read block 4755 len 32768
14270 166/300 Write block 41 len 8192 data: 4 4
14271 167/300 Read block 83 len 8192
14272 168/300 Flush
14273 169/300 Write block 2396 len 40960 data: 4 5 6 6 6 4 4 4 3 3
14274 170/300 Write block 1087 len 28672 data: 4 5 5 5 4 3 2
14275 171/300 Read block 3045 len 28672
14276 172/300 Write block 2494 len 8192 data: 3 3
14277 173/300 Read block 3489 len 16384
14278 174/300 Read block 3372 len 12288
14279 175/300 Read block 4611 len 32768
14280 176/300 Write block 1154 len 28672 data: 3 3 5 6 6 7 7
14281 177/300 Write block 2610 len 24576 data: 4 4 5 4 5 5
14282 178/300 Write block 1346 len 24576 data: 3 3 3 3 3 3
14283 179/300 Read block 2 len 32768
14284 180/300 Read block 1325 len 24576
14285 181/300 Read block 817 len 20480
14286 182/300 Read block 428 len 16384
14287 183/300 Read block 3272 len 36864
14288 184/300 Read block 3483 len 12288
14289 185/300 Write block 3709 len 16384 data: 6 5 4 4
14290 186/300 Read block 2784 len 28672
14291 187/300 Flush
14292 188/300 Read block 3976 len 12288
14293 189/300 Write block 1861 len 28672 data: 4 6 5 5 7 9 8
14294 190/300 Write block 2498 len 8192 data: 3 2
14295 191/300 Write block 1250 len 32768 data: 3 3 2 2 3 3 3 7
14296 192/300 Read block 884 len 12288
14297 193/300 Read block 1526 len 12288
14298 194/300 Write block 2141 len 24576 data: 3 2 2 2 3 5
14299 195/300 Write block 3901 len 24576 data: 2 2 3 2 2 2
14300 196/300 Write block 67 len 40960 data: 4 5 6 7 6 6 7 8 6 5
14301 197/300 Read block 554 len 16384
14302 198/300 Write block 2395 len 16384 data: 5 5 6 7
14303 199/300 Read block 2939 len 28672
14304 200/300 Read block 255 len 20480
14305 201/300 Flush
14306 202/300 Read block 1894 len 8192
14307 203/300 Read block 9 len 40960
14308 204/300 Write block 4937 len 36864 data: 4 4 3 3 2 2 4 4 5
14309 205/300 Write block 1952 len 16384 data: 3 3 2 3
14310 206/300 Write block 1023 len 24576 data: 5 6 6 6 6 5
14311 207/300 Write block 1920 len 36864 data: 6 5 6 7 9 7 5 4 3
14312 208/300 Write block 1319 len 36864 data: 2 3 3 3 2 3 3 3 3
14313 209/300 Write block 4303 len 16384 data: 2 2 2 2
14314 210/300 Read block 3617 len 36864
14315 211/300 Write block 3602 len 12288 data: 4 4 5
14316 212/300 Read block 204 len 32768
14317 213/300 Write block 3275 len 12288 data: 3 3 3
14318 214/300 Read block 2161 len 16384
14319 215/300 Read block 4157 len 16384
14320 216/300 Read block 2191 len 24576
14321 217/300 Read block 4662 len 36864
14322 218/300 Read block 2714 len 24576
14323 219/300 Read block 2152 len 4096
14324 220/300 Read block 868 len 36864
14325 221/300 Write block 1373 len 16384 data: 3 3 3 3
14326 222/300 Read block 3960 len 4096
14327 223/300 Flush
14328 224/300 Read block 4607 len 16384
14329 225/300 Write block 2366 len 16384 data: 3 4 4 4
14330 226/300 Read block 3384 len 4096
14331 227/300 Write block 4468 len 20480 data: 3 2 3 3 3
14332 228/300 Flush
14333 229/300 Write block 2699 len 4096 data: 5
14334 230/300 Write block 3266 len 8192 data: 2 2
14335 231/300 Flush
14336 232/300 Read block 1975 len 16384
14337 233/300 Read block 1319 len 40960
14338 234/300 Read block 4359 len 16384
14339 235/300 Read block 2694 len 32768
14340 236/300 Read block 1882 len 24576
14341 237/300 Write block 4437 len 20480 data: 4 4 4 4 4
14342 238/300 Write block 1340 len 20480 data: 2 2 2 2 2
14343 239/300 Read block 3392 len 4096
14344 240/300 Read block 1190 len 24576
14345 241/300 Write block 1600 len 4096 data: 2
14346 242/300 Read block 432 len 24576
14347 243/300 Read block 2465 len 28672
14348 244/300 Read block 2670 len 8192
14349 245/300 Read block 4191 len 28672
14350 246/300 Read block 932 len 36864
14351 247/300 Write block 1493 len 8192 data: 4 3
14352 248/300 Read block 2319 len 40960
14353 249/300 Write block 1420 len 32768 data: 5 4 3 4 4 4 4 4
14354 250/300 Read block 3104 len 24576
14355 251/300 Write block 2861 len 28672 data: 5 4 4 4 4 6 7
14356 252/300 Read block 3879 len 32768
14357 253/300 Read block 1882 len 32768
14358 254/300 Write block 4591 len 24576 data: 4 4 4 5 4 3
14359 255/300 Write block 1951 len 28672 data: 3 4 4 3 4 3 3
14360 256/300 Read block 4705 len 12288
14361 257/300 Read block 2452 len 32768
14362 258/300 Read block 4580 len 24576
14363 259/300 Write block 970 len 36864 data: 3 3 4 4 4 4 2 2 2
14364 260/300 Read block 1262 len 4096
14365 261/300 Flush
14366 262/300 Write block 886 len 16384 data: 4 4 4 5
14367 263/300 Flush
14368 264/300 Read block 1321 len 4096
14369 265/300 Read block 1377 len 36864
14370 266/300 Read block 4636 len 40960
14371 267/300 Read block 4846 len 12288
14372 268/300 Write block 3054 len 36864 data: 3 3 3 2 2 2 2 2 2
14373 269/300 Write block 4745 len 8192 data: 4 4
14374 270/300 Write block 3785 len 24576 data: 3 3 3 3 3 2
14375 271/300 Write block 4558 len 40960 data: 6 3 3 4 4 5 7 7 8 8
14376 272/300 Write block 4777 len 12288 data: 4 3 4
14377 273/300 Write block 763 len 8192 data: 2 3
14378 274/300 Read block 4437 len 40960
14379 275/300 Write block 2310 len 24576 data: 2 2 2 2 2 2
14380 276/300 Read block 1078 len 16384
14381 277/300 Write block 900 len 36864 data: 2 3 3 3 3 3 3 3 2
14382 278/300 Write block 3147 len 36864 data: 4 3 3 3 3 2 2 2 2
14383 279/300 Read block 3932 len 20480
14384 280/300 Read block 744 len 28672
14385 281/300 Write block 2808 len 40960 data: 4 3 3 4 3 3 3 4 4 4
14386 282/300 Read block 1383 len 16384
14387 283/300 Read block 63 len 12288
14388 284/300 Write block 3326 len 20480 data: 3 3 3 4 5
14389 285/300 Read block 3644 len 40960
14390 286/300 Read block 68 len 12288
14391 287/300 Write block 158 len 32768 data: 5 6 6 6 5 4 3 5
14392 288/300 Read block 4408 len 20480
14393 289/300 Read block 3105 len 24576
14394 290/300 Read block 1131 len 16384
14395 291/300 Read block 112 len 20480
14396 292/300 Read block 185 len 8192
14397 293/300 Read block 749 len 16384
14398 294/300 Read block 4757 len 16384
14399 295/300 Read block 2195 len 8192
14400 296/300 Write block 3764 len 12288 data: 3 3 3
14401 297/300 Read block 3172 len 28672
14402 298/300 Read block 1829 len 16384
14403 299/300 Write block 2202 len 4096 data: 3
14404 300/300 Write block 3502 len 8192 data: 7 7
14405 301/300 Write block 3304 len 12288 data: 4 5 5
14406 Main task has work to do, go find it
14407 got dsc Start(2)
14408 start 2
14409 Jan 23 17:40:47.136 INFO request completed, latency_us: 206, response_code: 204, uri: /start/cid/2, method: POST, req_id: 6c70fb49-b1b9-4f5b-8df6-5fe74b0c1148, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
14410 Replay: started 2, returned:Ok(())
14411 ----------------------------------------------------------------
14412 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14413 States: Active Active Offline
14414 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14415 4926 Acked Write 9 Done Done Sent false
14416 4927 Acked Read 7 Done Done Sent false
14417 4928 Acked Read 9 Done Done Sent false
14418 4929 Acked Read 10 Done Done Sent false
14419 4930 Acked Read 1 Done Done Sent false
14420 4931 Acked Read 1 Done Done Sent false
14421 4932 Acked Flush 0 Done Done Sent false
14422 4933 Acked Read 8 Done Done Sent false
14423 4934 Acked Read 3 Done Done Sent false
14424 4935 Acked Read 9 Done Done Sent false
14425 4936 Acked Read 5 Done Done Sent false
14426 4937 Acked Write 8 Done Done Sent false
14427 4938 Acked Read 9 Done Done Sent false
14428 4939 Acked Read 4 Done Done Sent false
14429 4940 Acked Read 1 Done Done Sent false
14430 4941 Acked Flush 0 Done Done Sent false
14431 4942 Acked Write 2 Done Done Sent false
14432 4943 Acked Read 1 Done Done Sent false
14433 4944 Acked Write 10 Done Done Sent false
14434 4945 Acked Read 1 Done Done Sent false
14435 4946 Acked Write 4 Done Done Sent false
14436 4947 Acked Write 1 Done Done Sent false
14437 4948 Acked Flush 0 Done Done Sent false
14438 4949 Acked Read 1 Done Done Sent false
14439 4950 Acked Write 7 Done Done Sent false
14440 4951 Acked Read 8 Done Done Sent false
14441 4952 Acked Read 9 Done Done Sent false
14442 4953 Acked Read 9 Done Done Sent false
14443 4954 Acked Read 10 Done Done Sent false
14444 4955 Acked Read 2 Done Done Sent false
14445 4956 Acked Write 7 Done Done Sent false
14446 4957 Acked Write 5 Done Done Sent false
14447 4958 Acked Read 7 Done Done Sent false
14448 4959 Acked Read 5 Done Done Sent false
14449 4960 Acked Read 9 Done Done Sent false
14450 4961 Acked Write 4 Done Done Sent false
14451 4962 Acked Read 5 Done Done Sent false
14452 4963 Acked Read 8 Done Done Sent false
14453 4964 Acked Flush 0 Done Done Sent false
14454 4965 Acked Flush 0 Done Done Sent false
14455 4966 Acked Read 5 Done Done Sent false
14456 4967 Acked Read 9 Done Done Sent false
14457 4968 Acked Read 10 Done Done Sent false
14458 4969 Acked Read 1 Done Done Sent false
14459 4970 Acked Write 7 Done Done Sent false
14460 4971 Acked Write 6 Done Done Sent false
14461 4972 Acked Write 4 Done Done Sent false
14462 4973 Acked Read 8 Done Done Sent false
14463 4974 Acked Read 10 Done Done Sent false
14464 4975 Acked Write 3 Done Done Sent false
14465 4976 Acked Flush 0 Done Done Sent false
14466 4977 Acked Write 6 Done Done Sent false
14467 4978 Acked Read 4 Done Done Sent false
14468 4979 Acked Read 8 Done Done Sent false
14469 4980 Acked Flush 0 Done Done Sent false
14470 4981 Acked Read 10 Done Done Sent false
14471 4982 Acked Write 7 Done Done Sent false
14472 4983 Acked Read 2 Done Done Sent false
14473 4984 Acked Read 10 Done Done Sent false
14474 4985 Acked Write 6 Done Done Sent false
14475 4986 Acked Read 9 Done Done Sent false
14476 4987 Acked Write 8 Done Done Sent false
14477 4988 Acked Write 1 Done Done Sent false
14478 4989 Acked Read 8 Done Done Sent false
14479 4990 Acked Write 6 Done Done Sent false
14480 4991 Acked Write 6 Done Done Sent false
14481 4992 Acked Flush 0 Done Done Sent false
14482 4993 Acked Write 5 Done Done Sent false
14483 4994 Acked Read 3 Done Done Sent false
14484 4995 Acked Read 9 Done Done Sent false
14485 4996 Acked Read 1 Done Done Sent false
14486 4997 Acked Write 10 Done Done Sent false
14487 4998 Acked Read 3 Done Done Sent false
14488 4999 Acked Flush 0 Done Done Sent false
14489 5000 Acked Read 9 Done Done Sent false
14490 5001 Acked Read 5 Done Done Sent false
14491 5002 Acked Write 4 Done Done Sent false
14492 5003 Acked Read 4 Done Done Sent false
14493 5004 Acked Read 4 Done Done Sent false
14494 5005 Acked Flush 0 Done Done Sent false
14495 5006 Acked Read 8 Done Done Sent false
14496 5007 Acked Write 8 Done Done Sent false
14497 5008 Acked Read 4 Done Done Sent false
14498 5009 Acked Read 4 Done Done Sent false
14499 5010 Acked Read 6 Done Done Sent false
14500 5011 Acked Read 2 Done Done Sent false
14501 5012 Acked Read 8 Done Done Sent false
14502 5013 Acked Read 10 Done Done Sent false
14503 5014 Acked Read 4 Done Done Sent false
14504 5015 Acked Write 4 Done Done Sent false
14505 5016 Acked Flush 0 Done Done Sent false
14506 5017 Acked Write 4 Done Done Sent false
14507 5018 Acked Write 2 Done Done Sent false
14508 5019 Acked Read 7 Done Done Sent false
14509 5020 Acked Read 7 Done Done Sent false
14510 5021 Acked Read 4 Done Done Sent false
14511 5022 Acked Read 10 Done Done Sent false
14512 5023 Acked Read 6 Done Done Sent false
14513 5024 Acked Write 2 Done Done Sent false
14514 5025 Acked Read 1 Done Done Sent false
14515 5026 Acked Read 10 Done Done Sent false
14516 5027 Acked Read 7 Done Done Sent false
14517 5028 Acked Read 8 Done Done Sent false
14518 5029 Acked Read 10 Done Done Sent false
14519 5030 Acked Write 2 Done Done Sent false
14520 5031 Acked Write 10 Done Done Sent false
14521 5032 Acked Write 1 Done Done Sent false
14522 5033 Acked Write 10 Done Done Sent false
14523 5034 Acked Write 10 Done Done Sent false
14524 5035 Acked Read 4 Done Done Sent false
14525 5036 Acked Write 5 Done Done Sent false
14526 5037 Acked Flush 0 Done Done Sent false
14527 5038 Acked Read 2 Done Done Sent false
14528 5039 Acked Write 9 Done Done Sent false
14529 5040 Acked Read 6 Done Done Sent false
14530 5041 Acked Read 3 Done Done Sent false
14531 5042 Acked Write 9 Done Done Sent false
14532 5043 Acked Read 9 Done Done Sent false
14533 5044 Acked Write 2 Done Done Sent false
14534 5045 Acked Write 9 Done Done Sent false
14535 5046 Acked Write 1 Done Done Sent false
14536 5047 Acked Flush 0 Done Done Sent false
14537 5048 Acked Write 3 Done Done Sent false
14538 5049 Acked Read 3 Done Done Sent false
14539 5050 Acked Read 3 Done Done Sent false
14540 5051 Acked Write 7 Done Done Sent false
14541 5052 Acked Write 1 Done Done Sent false
14542 5053 Acked Flush 0 Done Done Sent false
14543 5054 Acked Read 6 Done Done Sent false
14544 5055 Acked Read 8 Done Done Sent false
14545 5056 Acked Read 7 Done Done Sent false
14546 5057 Acked Write 10 Done Done Sent false
14547 5058 Acked Read 2 Done Done Sent false
14548 5059 Acked Write 5 Done Done Sent false
14549 5060 Acked Write 1 Done Done Sent false
14550 5061 Acked Read 3 Done Done Sent false
14551 5062 Acked Read 2 Done Done Sent false
14552 5063 Acked Read 6 Done Done Sent false
14553 5064 Acked Read 1 Done Done Sent false
14554 5065 Acked Read 7 Done Done Sent false
14555 5066 Acked Write 5 Done Done Sent false
14556 5067 Acked Read 5 Done Done Sent false
14557 5068 Acked Write 3 Done Done Sent false
14558 5069 Acked Write 7 Done Done Sent false
14559 5070 Acked Read 1 Done Done Sent false
14560 5071 Acked Read 4 Done Done Sent false
14561 5072 Acked Write 10 Done Done Sent false
14562 5073 Acked Write 8 Done Done Sent false
14563 5074 Acked Write 9 Done Done Sent false
14564 5075 Acked Read 3 Done Done Sent false
14565 5076 Acked Write 3 Done Done Sent false
14566 5077 Acked Flush 0 Done Done Sent false
14567 5078 Acked Flush 0 Done Done Sent false
14568 5079 Acked Flush 0 Done Done Sent false
14569 5080 Acked Write 6 Done Done Sent false
14570 5081 Acked Read 2 Done Done Sent false
14571 5082 Acked Write 1 Done Done Sent false
14572 5083 Acked Flush 0 Done Done Sent false
14573 5084 Acked Read 5 Done Done Sent false
14574 5085 Acked Read 5 Done Done Sent false
14575 5086 Acked Read 4 Done Done Sent false
14576 5087 Acked Write 2 Done Done Sent false
14577 5088 Acked Write 7 Done Done Sent false
14578 5089 Acked Write 1 Done Done Sent false
14579 5090 Acked Read 8 Done Done Sent false
14580 5091 Acked Write 2 Done Done Sent false
14581 5092 Acked Read 2 Done Done Sent false
14582 5093 Acked Flush 0 Done Done Sent false
14583 5094 Acked Write 10 Done Done Sent false
14584 5095 Acked Write 7 Done Done Sent false
14585 5096 Acked Read 7 Done Done Sent false
14586 5097 Acked Write 2 Done Done Sent false
14587 5098 Acked Read 4 Done Done Sent false
14588 5099 Acked Read 3 Done Done Sent false
14589 5100 Acked Read 8 Done Done Sent false
14590 5101 Acked Write 7 Done Done Sent false
14591 5102 Acked Write 6 Done Done Sent false
14592 5103 Acked Write 6 Done Done Sent false
14593 5104 Acked Read 8 Done Done Sent false
14594 5105 Acked Read 6 Done Done Sent false
14595 5106 Acked Read 5 Done Done Sent false
14596 5107 Acked Read 4 Done Done Sent false
14597 5108 Acked Read 9 Done Done Sent false
14598 5109 Acked Read 3 Done Done Sent false
14599 5110 Acked Write 4 Done Done Sent false
14600 5111 Acked Read 7 Done Done Sent false
14601 5112 Acked Flush 0 Done Done Sent false
14602 5113 Acked Read 3 Done Done Sent false
14603 5114 Acked Write 7 Done Done Sent false
14604 5115 Acked Write 2 Done Done Sent false
14605 5116 Acked Write 8 Done Done Sent false
14606 5117 Acked Read 3 Done Done Sent false
14607 5118 Acked Read 3 Done Done Sent false
14608 5119 Acked Write 6 Done Done Sent false
14609 5120 Acked Write 6 Done Done Sent false
14610 5121 Acked Write 10 Done Done Sent false
14611 5122 Acked Read 4 Done Done Sent false
14612 5123 Acked Write 4 Done Done Sent false
14613 5124 Acked Read 7 Done Done Sent false
14614 5125 Acked Read 5 Done Done Sent false
14615 5126 Acked Flush 0 Done Done Sent false
14616 5127 Acked Read 2 Done Done Sent false
14617 5128 Acked Read 10 Done Done Sent false
14618 5129 Acked Write 9 Done Done Sent false
14619 5130 Acked Write 4 Done Done Sent false
14620 5131 Acked Write 6 Done Done Sent false
14621 5132 Acked Write 9 Done Done Sent false
14622 5133 Acked Write 9 Done Done Sent false
14623 5134 Acked Write 4 Done Done Sent false
14624 5135 Acked Read 9 Done Done Sent false
14625 5136 Acked Write 3 Done Done Sent false
14626 5137 Acked Read 8 Done Done Sent false
14627 5138 Acked Write 3 Done Done Sent false
14628 5139 Acked Read 4 Done Done Sent false
14629 5140 Acked Read 4 Done Done Sent false
14630 5141 Acked Read 6 Done Done Sent false
14631 5142 Acked Read 9 Done Done Sent false
14632 5143 Acked Read 6 Done Done Sent false
14633 5144 Acked Read 1 Done Done Sent false
14634 5145 Acked Read 9 Done Done Sent false
14635 5146 Acked Write 4 Done Done Sent false
14636 5147 Acked Read 1 Done Done Sent false
14637 5148 Acked Flush 0 Done Done Sent false
14638 5149 Acked Read 4 Done Done Sent false
14639 5150 Acked Write 4 Done Done Sent false
14640 5151 Acked Read 1 Done Done Sent false
14641 5152 Acked Write 5 Done Done Sent false
14642 5153 Acked Flush 0 Done Done Sent false
14643 5154 Acked Write 1 Done Done Sent false
14644 5155 Acked Write 2 Done Done Sent false
14645 5156 Acked Flush 0 Done Done Sent false
14646 5157 Acked Read 4 Done Done Sent false
14647 5158 Acked Read 10 Done Done Sent false
14648 5159 Acked Read 4 Done Done Sent false
14649 5160 Acked Read 8 Done Done Sent false
14650 5161 Acked Read 6 Done Done Sent false
14651 5162 Acked Write 5 Done Done Sent false
14652 5163 Acked Write 5 Done Done Sent false
14653 5164 Acked Read 1 Done Done Sent false
14654 5165 Acked Read 6 Done Done Sent false
14655 5166 Acked Write 1 Done Done Sent false
14656 5167 Acked Read 6 Done Done Sent false
14657 5168 Acked Read 7 Done Done Sent false
14658 5169 Acked Read 2 Done Done Sent false
14659 5170 Acked Read 7 Done Done Sent false
14660 5171 Acked Read 9 Done Done Sent false
14661 5172 Acked Write 2 Done Done Sent false
14662 5173 Acked Read 10 Done Done Sent false
14663 5174 Acked Write 8 Done Done Sent false
14664 5175 Acked Read 6 Done Done Sent false
14665 5176 Acked Write 7 Done Done Sent false
14666 5177 Acked Read 8 Done Done Sent false
14667 5178 Acked Read 8 Done Done Sent false
14668 5179 Acked Write 6 Done Done Sent false
14669 5180 Acked Write 7 Done Done Sent false
14670 5181 Acked Read 3 Done Done Sent false
14671 5182 Acked Read 8 Done Done Sent false
14672 5183 Acked Read 6 Done Done Sent false
14673 5184 Acked Write 9 Done Done Sent false
14674 5185 Acked Read 1 Done Done Sent false
14675 5186 Acked Flush 0 Done Done Sent false
14676 5187 Acked Write 4 Done Done Sent false
14677 5188 Acked Flush 0 Done Done Sent false
14678 5189 Acked Read 1 Done Done Sent false
14679 5190 Acked Read 9 Done Done Sent false
14680 5191 Acked Read 10 Done Done Sent false
14681 5192 Acked Read 3 Done Done Sent false
14682 5193 Acked Write 9 Done Done Sent false
14683 5194 Acked Write 2 Done Done Sent false
14684 5195 Acked Write 6 Done Done Sent false
14685 5196 Acked Write 10 Done Done Sent false
14686 5197 Acked Write 3 Done Done Sent false
14687 5198 Acked Write 2 Done Done Sent false
14688 5199 Acked Read 10 Done Done Sent false
14689 5200 Acked Write 6 Done Done Sent false
14690 5201 Acked Read 4 Done Done Sent false
14691 5202 Acked Write 9 Done Done Sent false
14692 5203 Acked Write 9 Done Done Sent false
14693 5204 Acked Read 5 Done Done Sent false
14694 5205 Acked Read 7 Done Done Sent false
14695 5206 Acked Write 10 Done Done Sent false
14696 5207 Acked Read 4 Done Done Sent false
14697 5208 Acked Read 3 Done Done Sent false
14698 5209 Acked Write 5 Done Done Sent false
14699 5210 Acked Read 10 Done Done Sent false
14700 5211 Acked Read 3 Done Done Sent false
14701 5212 Acked Write 8 Done Done Sent false
14702 5213 Acked Read 5 Done Done Sent false
14703 5214 Acked Read 6 Done Done Sent false
14704 5215 Acked Read 4 Done Done Sent false
14705 5216 Acked Read 5 Done Done Sent false
14706 5217 Acked Read 2 Done Done Sent false
14707 5218 Acked Read 4 Done Done Sent false
14708 5219 Acked Read 4 Done Done Sent false
14709 5220 Acked Read 2 Done Done Sent false
14710 5221 Acked Write 3 Done Done Sent false
14711 5222 Acked Read 7 Done Done Sent false
14712 5223 Acked Read 4 Done Done Sent false
14713 5224 Acked Write 1 Done Done Sent false
14714 5225 Acked Write 2 Done Done Sent false
14715 5226 Acked Write 3 Done Done Sent false
14716 STATES DS:0 DS:1 DS:2 TOTAL
14717 Sent 0 0 301 301
14718 Done 301 301 0 602
14719 Skipped 0 0 0 0
14720 Error 0 0 0 0
14721 Last Flush: Some(JobId(5188)) Some(JobId(5188)) Some(JobId(4925))
14722 Downstairs last five completed: 4925 4924 4923 4922 4921
14723 Upstairs last five completed: 5226 5225 5224 5223 5222
14724 CLIENT: Up:0 ds:301 act:2
14725 [8830] got start action
14726 [8830] I am going to restart
14727 Starting downstairs at port 8830
14728 Make output file at "/tmp/dsc/downstairs-8830.txt"
14729 [8830][2] reports Starting
14730 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1219)
14731 [8830][2] reports Running
14732 ----------------------------------------------------------------
14733 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
14734 States: Active Active Offline
14735 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14736 4926 Acked Write 9 Done Done Sent false
14737 4927 Acked Read 7 Done Done Sent false
14738 4928 Acked Read 9 Done Done Sent false
14739 4929 Acked Read 10 Done Done Sent false
14740 4930 Acked Read 1 Done Done Sent false
14741 4931 Acked Read 1 Done Done Sent false
14742 4932 Acked Flush 0 Done Done Sent false
14743 4933 Acked Read 8 Done Done Sent false
14744 4934 Acked Read 3 Done Done Sent false
14745 4935 Acked Read 9 Done Done Sent false
14746 4936 Acked Read 5 Done Done Sent false
14747 4937 Acked Write 8 Done Done Sent false
14748 4938 Acked Read 9 Done Done Sent false
14749 4939 Acked Read 4 Done Done Sent false
14750 4940 Acked Read 1 Done Done Sent false
14751 4941 Acked Flush 0 Done Done Sent false
14752 4942 Acked Write 2 Done Done Sent false
14753 4943 Acked Read 1 Done Done Sent false
14754 4944 Acked Write 10 Done Done Sent false
14755 4945 Acked Read 1 Done Done Sent false
14756 4946 Acked Write 4 Done Done Sent false
14757 4947 Acked Write 1 Done Done Sent false
14758 4948 Acked Flush 0 Done Done Sent false
14759 4949 Acked Read 1 Done Done Sent false
14760 4950 Acked Write 7 Done Done Sent false
14761 4951 Acked Read 8 Done Done Sent false
14762 4952 Acked Read 9 Done Done Sent false
14763 4953 Acked Read 9 Done Done Sent false
14764 4954 Acked Read 10 Done Done Sent false
14765 4955 Acked Read 2 Done Done Sent false
14766 4956 Acked Write 7 Done Done Sent false
14767 4957 Acked Write 5 Done Done Sent false
14768 4958 Acked Read 7 Done Done Sent false
14769 4959 Acked Read 5 Done Done Sent false
14770 4960 Acked Read 9 Done Done Sent false
14771 4961 Acked Write 4 Done Done Sent false
14772 4962 Acked Read 5 Done Done Sent false
14773 4963 Acked Read 8 Done Done Sent false
14774 4964 Acked Flush 0 Done Done Sent false
14775 4965 Acked Flush 0 Done Done Sent false
14776 4966 Acked Read 5 Done Done Sent false
14777 4967 Acked Read 9 Done Done Sent false
14778 4968 Acked Read 10 Done Done Sent false
14779 4969 Acked Read 1 Done Done Sent false
14780 4970 Acked Write 7 Done Done Sent false
14781 4971 Acked Write 6 Done Done Sent false
14782 4972 Acked Write 4 Done Done Sent false
14783 4973 Acked Read 8 Done Done Sent false
14784 4974 Acked Read 10 Done Done Sent false
14785 4975 Acked Write 3 Done Done Sent false
14786 4976 Acked Flush 0 Done Done Sent false
14787 4977 Acked Write 6 Done Done Sent false
14788 4978 Acked Read 4 Done Done Sent false
14789 4979 Acked Read 8 Done Done Sent false
14790 4980 Acked Flush 0 Done Done Sent false
14791 4981 Acked Read 10 Done Done Sent false
14792 4982 Acked Write 7 Done Done Sent false
14793 4983 Acked Read 2 Done Done Sent false
14794 4984 Acked Read 10 Done Done Sent false
14795 4985 Acked Write 6 Done Done Sent false
14796 4986 Acked Read 9 Done Done Sent false
14797 4987 Acked Write 8 Done Done Sent false
14798 4988 Acked Write 1 Done Done Sent false
14799 4989 Acked Read 8 Done Done Sent false
14800 4990 Acked Write 6 Done Done Sent false
14801 4991 Acked Write 6 Done Done Sent false
14802 4992 Acked Flush 0 Done Done Sent false
14803 4993 Acked Write 5 Done Done Sent false
14804 4994 Acked Read 3 Done Done Sent false
14805 4995 Acked Read 9 Done Done Sent false
14806 4996 Acked Read 1 Done Done Sent false
14807 4997 Acked Write 10 Done Done Sent false
14808 4998 Acked Read 3 Done Done Sent false
14809 4999 Acked Flush 0 Done Done Sent false
14810 5000 Acked Read 9 Done Done Sent false
14811 5001 Acked Read 5 Done Done Sent false
14812 5002 Acked Write 4 Done Done Sent false
14813 5003 Acked Read 4 Done Done Sent false
14814 5004 Acked Read 4 Done Done Sent false
14815 5005 Acked Flush 0 Done Done Sent false
14816 5006 Acked Read 8 Done Done Sent false
14817 5007 Acked Write 8 Done Done Sent false
14818 5008 Acked Read 4 Done Done Sent false
14819 5009 Acked Read 4 Done Done Sent false
14820 5010 Acked Read 6 Done Done Sent false
14821 5011 Acked Read 2 Done Done Sent false
14822 5012 Acked Read 8 Done Done Sent false
14823 5013 Acked Read 10 Done Done Sent false
14824 5014 Acked Read 4 Done Done Sent false
14825 5015 Acked Write 4 Done Done Sent false
14826 5016 Acked Flush 0 Done Done Sent false
14827 5017 Acked Write 4 Done Done Sent false
14828 5018 Acked Write 2 Done Done Sent false
14829 5019 Acked Read 7 Done Done Sent false
14830 5020 Acked Read 7 Done Done Sent false
14831 5021 Acked Read 4 Done Done Sent false
14832 5022 Acked Read 10 Done Done Sent false
14833 5023 Acked Read 6 Done Done Sent false
14834 5024 Acked Write 2 Done Done Sent false
14835 5025 Acked Read 1 Done Done Sent false
14836 5026 Acked Read 10 Done Done Sent false
14837 5027 Acked Read 7 Done Done Sent false
14838 5028 Acked Read 8 Done Done Sent false
14839 5029 Acked Read 10 Done Done Sent false
14840 5030 Acked Write 2 Done Done Sent false
14841 5031 Acked Write 10 Done Done Sent false
14842 5032 Acked Write 1 Done Done Sent false
14843 5033 Acked Write 10 Done Done Sent false
14844 5034 Acked Write 10 Done Done Sent false
14845 5035 Acked Read 4 Done Done Sent false
14846 5036 Acked Write 5 Done Done Sent false
14847 5037 Acked Flush 0 Done Done Sent false
14848 5038 Acked Read 2 Done Done Sent false
14849 5039 Acked Write 9 Done Done Sent false
14850 5040 Acked Read 6 Done Done Sent false
14851 5041 Acked Read 3 Done Done Sent false
14852 5042 Acked Write 9 Done Done Sent false
14853 5043 Acked Read 9 Done Done Sent false
14854 5044 Acked Write 2 Done Done Sent false
14855 5045 Acked Write 9 Done Done Sent false
14856 5046 Acked Write 1 Done Done Sent false
14857 5047 Acked Flush 0 Done Done Sent false
14858 5048 Acked Write 3 Done Done Sent false
14859 5049 Acked Read 3 Done Done Sent false
14860 5050 Acked Read 3 Done Done Sent false
14861 5051 Acked Write 7 Done Done Sent false
14862 5052 Acked Write 1 Done Done Sent false
14863 5053 Acked Flush 0 Done Done Sent false
14864 5054 Acked Read 6 Done Done Sent false
14865 5055 Acked Read 8 Done Done Sent false
14866 5056 Acked Read 7 Done Done Sent false
14867 5057 Acked Write 10 Done Done Sent false
14868 5058 Acked Read 2 Done Done Sent false
14869 5059 Acked Write 5 Done Done Sent false
14870 5060 Acked Write 1 Done Done Sent false
14871 5061 Acked Read 3 Done Done Sent false
14872 5062 Acked Read 2 Done Done Sent false
14873 5063 Acked Read 6 Done Done Sent false
14874 5064 Acked Read 1 Done Done Sent false
14875 5065 Acked Read 7 Done Done Sent false
14876 5066 Acked Write 5 Done Done Sent false
14877 5067 Acked Read 5 Done Done Sent false
14878 5068 Acked Write 3 Done Done Sent false
14879 5069 Acked Write 7 Done Done Sent false
14880 5070 Acked Read 1 Done Done Sent false
14881 5071 Acked Read 4 Done Done Sent false
14882 5072 Acked Write 10 Done Done Sent false
14883 5073 Acked Write 8 Done Done Sent false
14884 5074 Acked Write 9 Done Done Sent false
14885 5075 Acked Read 3 Done Done Sent false
14886 5076 Acked Write 3 Done Done Sent false
14887 5077 Acked Flush 0 Done Done Sent false
14888 5078 Acked Flush 0 Done Done Sent false
14889 5079 Acked Flush 0 Done Done Sent false
14890 5080 Acked Write 6 Done Done Sent false
14891 5081 Acked Read 2 Done Done Sent false
14892 5082 Acked Write 1 Done Done Sent false
14893 5083 Acked Flush 0 Done Done Sent false
14894 5084 Acked Read 5 Done Done Sent false
14895 5085 Acked Read 5 Done Done Sent false
14896 5086 Acked Read 4 Done Done Sent false
14897 5087 Acked Write 2 Done Done Sent false
14898 5088 Acked Write 7 Done Done Sent false
14899 5089 Acked Write 1 Done Done Sent false
14900 5090 Acked Read 8 Done Done Sent false
14901 5091 Acked Write 2 Done Done Sent false
14902 5092 Acked Read 2 Done Done Sent false
14903 5093 Acked Flush 0 Done Done Sent false
14904 5094 Acked Write 10 Done Done Sent false
14905 5095 Acked Write 7 Done Done Sent false
14906 5096 Acked Read 7 Done Done Sent false
14907 5097 Acked Write 2 Done Done Sent false
14908 5098 Acked Read 4 Done Done Sent false
14909 5099 Acked Read 3 Done Done Sent false
14910 5100 Acked Read 8 Done Done Sent false
14911 5101 Acked Write 7 Done Done Sent false
14912 5102 Acked Write 6 Done Done Sent false
14913 5103 Acked Write 6 Done Done Sent false
14914 5104 Acked Read 8 Done Done Sent false
14915 5105 Acked Read 6 Done Done Sent false
14916 5106 Acked Read 5 Done Done Sent false
14917 5107 Acked Read 4 Done Done Sent false
14918 5108 Acked Read 9 Done Done Sent false
14919 5109 Acked Read 3 Done Done Sent false
14920 5110 Acked Write 4 Done Done Sent false
14921 5111 Acked Read 7 Done Done Sent false
14922 5112 Acked Flush 0 Done Done Sent false
14923 5113 Acked Read 3 Done Done Sent false
14924 5114 Acked Write 7 Done Done Sent false
14925 5115 Acked Write 2 Done Done Sent false
14926 5116 Acked Write 8 Done Done Sent false
14927 5117 Acked Read 3 Done Done Sent false
14928 5118 Acked Read 3 Done Done Sent false
14929 5119 Acked Write 6 Done Done Sent false
14930 5120 Acked Write 6 Done Done Sent false
14931 5121 Acked Write 10 Done Done Sent false
14932 5122 Acked Read 4 Done Done Sent false
14933 5123 Acked Write 4 Done Done Sent false
14934 5124 Acked Read 7 Done Done Sent false
14935 5125 Acked Read 5 Done Done Sent false
14936 5126 Acked Flush 0 Done Done Sent false
14937 5127 Acked Read 2 Done Done Sent false
14938 5128 Acked Read 10 Done Done Sent false
14939 5129 Acked Write 9 Done Done Sent false
14940 5130 Acked Write 4 Done Done Sent false
14941 5131 Acked Write 6 Done Done Sent false
14942 5132 Acked Write 9 Done Done Sent false
14943 5133 Acked Write 9 Done Done Sent false
14944 5134 Acked Write 4 Done Done Sent false
14945 5135 Acked Read 9 Done Done Sent false
14946 5136 Acked Write 3 Done Done Sent false
14947 5137 Acked Read 8 Done Done Sent false
14948 5138 Acked Write 3 Done Done Sent false
14949 5139 Acked Read 4 Done Done Sent false
14950 5140 Acked Read 4 Done Done Sent false
14951 5141 Acked Read 6 Done Done Sent false
14952 5142 Acked Read 9 Done Done Sent false
14953 5143 Acked Read 6 Done Done Sent false
14954 5144 Acked Read 1 Done Done Sent false
14955 5145 Acked Read 9 Done Done Sent false
14956 5146 Acked Write 4 Done Done Sent false
14957 5147 Acked Read 1 Done Done Sent false
14958 5148 Acked Flush 0 Done Done Sent false
14959 5149 Acked Read 4 Done Done Sent false
14960 5150 Acked Write 4 Done Done Sent false
14961 5151 Acked Read 1 Done Done Sent false
14962 5152 Acked Write 5 Done Done Sent false
14963 5153 Acked Flush 0 Done Done Sent false
14964 5154 Acked Write 1 Done Done Sent false
14965 5155 Acked Write 2 Done Done Sent false
14966 5156 Acked Flush 0 Done Done Sent false
14967 5157 Acked Read 4 Done Done Sent false
14968 5158 Acked Read 10 Done Done Sent false
14969 5159 Acked Read 4 Done Done Sent false
14970 5160 Acked Read 8 Done Done Sent false
14971 5161 Acked Read 6 Done Done Sent false
14972 5162 Acked Write 5 Done Done Sent false
14973 5163 Acked Write 5 Done Done Sent false
14974 5164 Acked Read 1 Done Done Sent false
14975 5165 Acked Read 6 Done Done Sent false
14976 5166 Acked Write 1 Done Done Sent false
14977 5167 Acked Read 6 Done Done Sent false
14978 5168 Acked Read 7 Done Done Sent false
14979 5169 Acked Read 2 Done Done Sent false
14980 5170 Acked Read 7 Done Done Sent false
14981 5171 Acked Read 9 Done Done Sent false
14982 5172 Acked Write 2 Done Done Sent false
14983 5173 Acked Read 10 Done Done Sent false
14984 5174 Acked Write 8 Done Done Sent false
14985 5175 Acked Read 6 Done Done Sent false
14986 5176 Acked Write 7 Done Done Sent false
14987 5177 Acked Read 8 Done Done Sent false
14988 5178 Acked Read 8 Done Done Sent false
14989 5179 Acked Write 6 Done Done Sent false
14990 5180 Acked Write 7 Done Done Sent false
14991 5181 Acked Read 3 Done Done Sent false
14992 5182 Acked Read 8 Done Done Sent false
14993 5183 Acked Read 6 Done Done Sent false
14994 5184 Acked Write 9 Done Done Sent false
14995 5185 Acked Read 1 Done Done Sent false
14996 5186 Acked Flush 0 Done Done Sent false
14997 5187 Acked Write 4 Done Done Sent false
14998 5188 Acked Flush 0 Done Done Sent false
14999 5189 Acked Read 1 Done Done Sent false
15000 5190 Acked Read 9 Done Done Sent false
15001 5191 Acked Read 10 Done Done Sent false
15002 5192 Acked Read 3 Done Done Sent false
15003 5193 Acked Write 9 Done Done Sent false
15004 5194 Acked Write 2 Done Done Sent false
15005 5195 Acked Write 6 Done Done Sent false
15006 5196 Acked Write 10 Done Done Sent false
15007 5197 Acked Write 3 Done Done Sent false
15008 5198 Acked Write 2 Done Done Sent false
15009 5199 Acked Read 10 Done Done Sent false
15010 5200 Acked Write 6 Done Done Sent false
15011 5201 Acked Read 4 Done Done Sent false
15012 5202 Acked Write 9 Done Done Sent false
15013 5203 Acked Write 9 Done Done Sent false
15014 5204 Acked Read 5 Done Done Sent false
15015 5205 Acked Read 7 Done Done Sent false
15016 5206 Acked Write 10 Done Done Sent false
15017 5207 Acked Read 4 Done Done Sent false
15018 5208 Acked Read 3 Done Done Sent false
15019 5209 Acked Write 5 Done Done Sent false
15020 5210 Acked Read 10 Done Done Sent false
15021 5211 Acked Read 3 Done Done Sent false
15022 5212 Acked Write 8 Done Done Sent false
15023 5213 Acked Read 5 Done Done Sent false
15024 5214 Acked Read 6 Done Done Sent false
15025 5215 Acked Read 4 Done Done Sent false
15026 5216 Acked Read 5 Done Done Sent false
15027 5217 Acked Read 2 Done Done Sent false
15028 5218 Acked Read 4 Done Done Sent false
15029 5219 Acked Read 4 Done Done Sent false
15030 5220 Acked Read 2 Done Done Sent false
15031 5221 Acked Write 3 Done Done Sent false
15032 5222 Acked Read 7 Done Done Sent false
15033 5223 Acked Read 4 Done Done Sent false
15034 5224 Acked Write 1 Done Done Sent false
15035 5225 Acked Write 2 Done Done Sent false
15036 5226 Acked Write 3 Done Done Sent false
15037 5227 Acked Flush 0 Done Done Sent false
15038 STATES DS:0 DS:1 DS:2 TOTAL
15039 Sent 0 0 302 302
15040 Done 302 302 0 604
15041 Skipped 0 0 0 0
15042 Error 0 0 0 0
15043 Last Flush: Some(JobId(5227)) Some(JobId(5227)) Some(JobId(4925))
15044 Downstairs last five completed: 4925 4924 4923 4922 4921
15045 Upstairs last five completed: 5227 5226 5225 5224 5223
15046 CLIENT: Up:0 ds:302 act:2
150472025-01-23T17:40:52.887ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
150482025-01-23T17:40:52.887ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44652) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
150492025-01-23T17:40:52.888ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
150502025-01-23T17:40:52.888ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
150512025-01-23T17:40:52.888ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
150522025-01-23T17:40:52.889ZINFOcrucible: send last flush ID to this DS: Some(JobId(4925)) = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
150532025-01-23T17:40:52.889ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
150542025-01-23T17:40:52.889ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(4925)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
150552025-01-23T17:40:52.889ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(4925)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
15056 ----------------------------------------------------------------
15057 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15058 Downstairs last five completed: 5227 5226 5225 5224 5223
15059 Upstairs last five completed: 5227 5226 5225 5224 5223
15060 CLIENT: Up:0 ds:0 act:3
15061 Replay: All jobs finished
15062 Main task has work to do, go find it
15063 got dsc Stop(2)
15064 stop 2
15065 Jan 23 17:40:55.142[8830] Got stop action so:false kr:false
15066 INFO request completed, latency_us: 389, response_code: 204, uri: /stop/cid/2, method: POST, req_id: aef6141e-02a7-4d07-89fb-6532ce608349, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
15067 Jan 23 17:40:55.143 INFO request completed, latency_us: 141, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9c078e7e-541e-4eaa-9b59-513adb313240, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
150682025-01-23T17:40:55.143ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
150692025-01-23T17:40:55.143ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
150702025-01-23T17:40:55.143ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
150712025-01-23T17:40:55.143ZINFOcrucible: client task is exiting = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
15072 [8830] Exited with: ExitStatus(unix_wait_status(9))
15073 [8830][2] reports Exit
15074 Jan 23 17:40:59.145 INFO request completed, latency_us: 215, response_code: 200, uri: /state/cid/2, method: GET, req_id: 17ed3515-c4eb-45b3-bb66-ff0ac2a02ce0, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
15075 001/300 Read block 4585 len 8192
15076 002/300 Flush
15077 003/300 Read block 962 len 28672
15078 004/300 Write block 4538 len 40960 data: 3 3 2 2 2 2 2 2 2 2
15079 005/300 Read block 4497 len 20480
15080 006/300 Write block 1740 len 32768 data: 2 2 2 2 2 2 2 2
15081 007/300 Write block 477 len 40960 data: 2 2 3 4 4 4 5 5 5 4
15082 008/300 Read block 1324 len 4096
15083 009/300 Flush
15084 010/300 Read block 3266 len 20480
15085 011/300 Write block 3757 len 8192 data: 3 4
15086 012/300 Read block 70 len 16384
15087 013/300 Read block 1858 len 16384
15088 014/300 Read block 281 len 28672
15089 015/300 Read block 4691 len 36864
15090 016/300 Flush
15091 017/300 Flush
15092 018/300 Read block 3902 len 4096
15093 019/300 Write block 3988 len 24576 data: 4 2 2 3 4 4
15094 020/300 Write block 2564 len 4096 data: 2
15095 021/300 Read block 989 len 20480
15096 022/300 Write block 2043 len 20480 data: 2 2 2 3 3
15097 023/300 Read block 4981 len 12288
15098 024/300 Read block 979 len 8192
15099 025/300 Read block 836 len 12288
15100 026/300 Write block 1391 len 8192 data: 4 4
15101 027/300 Read block 2330 len 40960
15102 028/300 Read block 1289 len 36864
15103 029/300 Flush
15104 030/300 Write block 2395 len 24576 data: 6 6 7 8 7 7
15105 031/300 Write block 1793 len 32768 data: 4 4 4 5 4 4 5 6
15106 032/300 Read block 3753 len 40960
15107 033/300 Write block 272 len 36864 data: 2 2 3 3 3 3 3 5 4
15108 034/300 Read block 4880 len 36864
15109 035/300 Read block 875 len 32768
15110 036/300 Read block 3216 len 12288
15111 037/300 Write block 1733 len 12288 data: 3 5 4
15112 038/300 Write block 2645 len 24576 data: 4 4 3 3 2 2
15113 039/300 Read block 928 len 24576
15114 040/300 Read block 2474 len 20480
15115 041/300 Read block 1752 len 40960
15116 042/300 Read block 3618 len 4096
15117 043/300 Write block 3352 len 32768 data: 4 5 4 3 3 3 3 2
15118 044/300 Write block 3847 len 36864 data: 2 4 4 5 6 6 6 6 6
15119 045/300 Read block 3431 len 16384
15120 046/300 Write block 2534 len 16384 data: 5 5 4 3
15121 047/300 Read block 3694 len 8192
15122 048/300 Read block 119 len 32768
15123 049/300 Write block 2268 len 32768 data: 3 3 4 5 4 4 4 3
15124 050/300 Flush
15125 051/300 Read block 3810 len 36864
15126 052/300 Write block 217 len 8192 data: 5 5
15127 053/300 Read block 5 len 12288
15128 054/300 Write block 3964 len 24576 data: 5 5 4 3 4 4
15129 055/300 Read block 3209 len 36864
15130 056/300 Read block 3971 len 12288
15131 057/300 Write block 1608 len 16384 data: 2 2 3 4
15132 058/300 Write block 2680 len 24576 data: 2 2 2 2 4 4
15133 059/300 Write block 1541 len 4096 data: 3
15134 060/300 Write block 1697 len 4096 data: 2
15135 061/300 Read block 3843 len 4096
15136 062/300 Write block 3724 len 12288 data: 5 3 4
15137 063/300 Read block 3878 len 32768
15138 064/300 Read block 2540 len 40960
15139 065/300 Write block 4254 len 12288 data: 3 4 6
15140 066/300 Write block 3466 len 28672 data: 4 4 5 3 3 3 3
15141 067/300 Read block 4918 len 12288
15142 068/300 Flush
15143 069/300 Read block 939 len 32768
15144 070/300 Write block 4648 len 40960 data: 4 4 4 2 3 3 3 3 3 3
15145 071/300 Write block 1272 len 12288 data: 4 5 5
15146 072/300 Flush
15147 073/300 Write block 748 len 36864 data: 5 6 6 4 4 5 4 3 3
15148 074/300 Read block 3151 len 40960
15149 075/300 Write block 2954 len 16384 data: 4 4 2 3
15150 076/300 Write block 4630 len 16384 data: 6 4 4 4
15151 077/300 Read block 4077 len 16384
15152 078/300 Read block 2977 len 8192
15153 079/300 Read block 3484 len 8192
15154 080/300 Read block 3034 len 24576
15155 081/300 Read block 124 len 20480
15156 082/300 Write block 1221 len 12288 data: 4 4 3
15157 083/300 Read block 4616 len 28672
15158 084/300 Flush
15159 085/300 Read block 94 len 4096
15160 086/300 Read block 311 len 16384
15161 087/300 Read block 4658 len 20480
15162 088/300 Write block 4487 len 8192 data: 2 2
15163 089/300 Write block 1842 len 4096 data: 2
15164 090/300 Write block 646 len 4096 data: 4
15165 091/300 Flush
15166 092/300 Write block 2972 len 32768 data: 5 6 6 5 4 4 2 2
15167 093/300 Write block 2536 len 36864 data: 5 4 3 3 3 4 3 3 3
15168 094/300 Read block 3507 len 16384
15169 095/300 Read block 982 len 40960
15170 096/300 Write block 2021 len 28672 data: 2 2 2 2 3 2 2
15171 097/300 Write block 2598 len 12288 data: 5 4 4
15172 098/300 Write block 2739 len 4096 data: 4
15173 099/300 Read block 2920 len 28672
15174 100/300 Read block 3966 len 28672
15175 101/300 Read block 2922 len 24576
15176 102/300 Write block 3774 len 12288 data: 3 3 3
15177 103/300 Write block 1005 len 28672 data: 5 5 5 4 5 5 5
15178 104/300 Flush
15179 105/300 Write block 756 len 20480 data: 4 3 3 4 5
15180 106/300 Read block 3165 len 28672
15181 107/300 Write block 321 len 12288 data: 2 3 3
15182 108/300 Flush
15183 109/300 Read block 1363 len 16384
15184 110/300 Write block 819 len 24576 data: 2 5 5 6 6 7
15185 111/300 Read block 2295 len 4096
15186 112/300 Read block 1097 len 16384
15187 113/300 Read block 4146 len 24576
15188 114/300 Flush
15189 115/300 Write block 170 len 8192 data: 5 6
15190 116/300 Write block 3811 len 24576 data: 4 4 3 4 3 3
15191 117/300 Flush
15192 118/300 Read block 2713 len 24576
15193 119/300 Write block 1968 len 28672 data: 4 5 5 4 4 3 4
15194 120/300 Write block 1915 len 28672 data: 5 4 4 5 5 7 6
15195 121/300 Write block 2749 len 16384 data: 7 8 8 7
15196 122/300 Read block 3009 len 12288
15197 123/300 Write block 2417 len 20480 data: 6 5 5 5 5
15198 124/300 Write block 2200 len 16384 data: 4 2 4 4
15199 125/300 Write block 1530 len 40960 data: 2 3 4 5 4 4 4 4 3 3
15200 126/300 Write block 1655 len 8192 data: 4 3
15201 127/300 Read block 762 len 28672
15202 128/300 Read block 125 len 28672
15203 129/300 Read block 2870 len 20480
15204 130/300 Write block 797 len 28672 data: 4 4 4 3 3 3 3
15205 131/300 Read block 1952 len 32768
15206 132/300 Read block 512 len 16384
15207 133/300 Write block 1330 len 12288 data: 5 7 7
15208 134/300 Flush
15209 135/300 Read block 500 len 20480
15210 136/300 Write block 3982 len 8192 data: 4 4
15211 137/300 Read block 1018 len 36864
15212 138/300 Write block 384 len 28672 data: 2 3 3 3 4 4 4
15213 139/300 Write block 996 len 20480 data: 6 6 5 4 4
15214 140/300 Read block 4605 len 24576
15215 141/300 Write block 3297 len 8192 data: 2 4
15216 142/300 Read block 3302 len 4096
15217 143/300 Write block 4863 len 32768 data: 4 5 4 4 4 4 6 7
15218 144/300 Read block 3433 len 28672
15219 145/300 Write block 1598 len 8192 data: 3 3
15220 146/300 Read block 1993 len 16384
15221 147/300 Write block 3344 len 8192 data: 3 3
15222 148/300 Read block 864 len 36864
15223 149/300 Read block 4502 len 32768
15224 150/300 Read block 2506 len 24576
15225 151/300 Read block 718 len 12288
15226 152/300 Read block 1064 len 24576
15227 153/300 Read block 3028 len 20480
15228 154/300 Read block 2177 len 12288
15229 155/300 Read block 3404 len 20480
15230 156/300 Write block 1975 len 32768 data: 4 4 4 4 3 3 2 3
15231 157/300 Flush
15232 158/300 Read block 874 len 20480
15233 159/300 Write block 1136 len 12288 data: 4 5 5
15234 160/300 Read block 3962 len 24576
15235 161/300 Read block 118 len 12288
15236 162/300 Write block 739 len 40960 data: 3 2 2 3 3 3 3 5 5 6
15237 163/300 Read block 1330 len 32768
15238 164/300 Write block 853 len 36864 data: 2 3 3 2 2 2 2 2 3
15239 165/300 Read block 3556 len 32768
15240 166/300 Read block 3954 len 8192
15241 167/300 Read block 1650 len 8192
15242 168/300 Read block 1076 len 16384
15243 169/300 Write block 3248 len 20480 data: 3 3 4 3 3
15244 170/300 Flush
15245 171/300 Flush
15246 172/300 Read block 4797 len 32768
15247 173/300 Write block 328 len 20480 data: 4 4 4 5 6
15248 174/300 Write block 4531 len 12288 data: 3 3 3
15249 175/300 Read block 3787 len 12288
15250 176/300 Flush
15251 177/300 Read block 3711 len 12288
15252 178/300 Read block 934 len 12288
15253 179/300 Write block 693 len 36864 data: 2 2 3 3 4 4 4 4 4
15254 180/300 Read block 260 len 20480
15255 181/300 Read block 2572 len 8192
15256 182/300 Read block 4795 len 16384
15257 183/300 Read block 3966 len 32768
15258 184/300 Write block 3211 len 8192 data: 4 3
15259 185/300 Read block 1569 len 36864
15260 186/300 Flush
15261 187/300 Read block 2475 len 20480
15262 188/300 Write block 865 len 40960 data: 3 3 3 3 4 4 3 3 4 4
15263 189/300 Write block 3467 len 16384 data: 5 6 4 4
15264 190/300 Read block 2930 len 40960
15265 191/300 Write block 4221 len 16384 data: 2 2 3 3
15266 192/300 Read block 63 len 28672
15267 193/300 Write block 2862 len 24576 data: 5 5 5 5 7 8
15268 194/300 Write block 3649 len 20480 data: 4 4 4 4 5
15269 195/300 Write block 4931 len 32768 data: 4 4 4 4 4 5 5 5
15270 196/300 Write block 3409 len 8192 data: 4 4
15271 197/300 Read block 2576 len 32768
15272 198/300 Read block 3715 len 36864
15273 199/300 Read block 2583 len 32768
15274 200/300 Read block 470 len 4096
15275 201/300 Flush
15276 202/300 Read block 2847 len 28672
15277 203/300 Write block 1961 len 4096 data: 4
15278 204/300 Write block 2369 len 40960 data: 5 3 5 5 5 5 5 6 6 6
15279 205/300 Flush
15280 206/300 Write block 4149 len 28672 data: 3 3 3 3 3 3 3
15281 207/300 Write block 2533 len 20480 data: 4 6 6 6 5
15282 208/300 Write block 2544 len 24576 data: 4 3 3 4 4 3
15283 209/300 Read block 2915 len 8192
15284 210/300 Read block 1836 len 28672
15285 211/300 Write block 4445 len 36864 data: 4 4 3 3 3 3 3 3 3
15286 212/300 Write block 3 len 36864 data: 3 4 4 3 3 3 3 3 3
15287 213/300 Write block 1740 len 16384 data: 3 3 3 3
15288 214/300 Write block 2165 len 24576 data: 2 2 2 2 2 2
15289 215/300 Write block 1213 len 12288 data: 2 2 2
15290 216/300 Flush
15291 217/300 Write block 4593 len 12288 data: 5 6 5
15292 218/300 Read block 3518 len 40960
15293 219/300 Read block 4030 len 36864
15294 220/300 Flush
15295 221/300 Flush
15296 222/300 Write block 2704 len 8192 data: 3 2
15297 223/300 Read block 1012 len 8192
15298 224/300 Write block 2490 len 40960 data: 4 5 4 4 4 4 3 3 4 3
15299 225/300 Read block 3286 len 24576
15300 226/300 Write block 4148 len 40960 data: 3 4 4 4 4 4 4 4 3 3
15301 227/300 Write block 2327 len 28672 data: 4 4 3 3 3 3 4
15302 228/300 Read block 4279 len 28672
15303 229/300 Read block 3905 len 28672
15304 230/300 Write block 4841 len 24576 data: 2 2 2 2 3 4
15305 231/300 Read block 3672 len 28672
15306 232/300 Write block 584 len 8192 data: 8 8
15307 233/300 Write block 3040 len 4096 data: 4
15308 234/300 Write block 228 len 4096 data: 3
15309 235/300 Write block 2061 len 24576 data: 6 7 6 5 3 3
15310 236/300 Write block 4703 len 28672 data: 4 4 3 4 4 5 5
15311 237/300 Flush
15312 238/300 Read block 4235 len 16384
15313 239/300 Read block 2169 len 16384
15314 240/300 Read block 2033 len 16384
15315 241/300 Write block 4809 len 32768 data: 2 3 3 4 3 3 3 4
15316 242/300 Flush
15317 243/300 Read block 2087 len 36864
15318 244/300 Read block 2080 len 16384
15319 245/300 Flush
15320 246/300 Write block 2238 len 12288 data: 3 5 5
15321 247/300 Flush
15322 248/300 Read block 2220 len 40960
15323 249/300 Write block 2236 len 36864 data: 3 2 4 6 6 5 5 5 4
15324 250/300 Flush
15325 251/300 Read block 4127 len 20480
15326 252/300 Read block 2801 len 36864
15327 253/300 Flush
15328 254/300 Read block 4717 len 24576
15329 255/300 Write block 1977 len 20480 data: 5 5 4 4 3
15330 256/300 Write block 3498 len 8192 data: 4 4
15331 257/300 Read block 1404 len 4096
15332 258/300 Write block 3255 len 4096 data: 4
15333 259/300 Read block 1844 len 12288
15334 260/300 Read block 3717 len 12288
15335 261/300 Read block 3255 len 12288
15336 262/300 Write block 1373 len 24576 data: 4 4 4 4 3 2
15337 263/300 Read block 3449 len 40960
15338 264/300 Read block 109 len 4096
15339 265/300 Write block 3733 len 4096 data: 3
15340 266/300 Read block 4762 len 20480
15341 267/300 Write block 2752 len 16384 data: 8 5 5 5
15342 268/300 Read block 1474 len 12288
15343 269/300 Read block 2621 len 16384
15344 270/300 Write block 566 len 32768 data: 3 3 3 4 6 7 5 4
15345 271/300 Write block 1837 len 12288 data: 7 5 4
15346 272/300 Write block 64 len 20480 data: 4 3 3 5 6
15347 273/300 Read block 2960 len 28672
15348 274/300 Write block 4495 len 40960 data: 4 5 5 5 5 4 4 4 4 5
15349 275/300 Flush
15350 276/300 Read block 2454 len 36864
15351 277/300 Read block 2502 len 28672
15352 278/300 Read block 2460 len 8192
15353 279/300 Write block 3332 len 40960 data: 4 3 3 3 3 4 4 4 3 2
15354 280/300 Read block 4477 len 12288
15355 281/300 Read block 560 len 24576
15356 282/300 Flush
15357 283/300 Read block 494 len 28672
15358 284/300 Read block 3377 len 8192
15359 285/300 Write block 1768 len 36864 data: 2 3 3 2 2 2 3 3 4
15360 286/300 Read block 4783 len 24576
15361 287/300 Read block 199 len 16384
15362 288/300 Read block 4297 len 28672
15363 289/300 Read block 4802 len 32768
15364 290/300 Write block 2222 len 16384 data: 6 6 5 4
15365 291/300 Read block 1315 len 28672
15366 292/300 Write block 1946 len 20480 data: 4 4 4 3 3
15367 293/300 Write block 4042 len 20480 data: 2 2 3 3 4
15368 294/300 Read block 367 len 12288
15369 295/300 Write block 852 len 24576 data: 2 3 4 4 3 3
15370 296/300 Write block 146 len 16384 data: 3 2 2 2
15371 297/300 Read block 3015 len 20480
15372 298/300 Read block 4293 len 8192
15373 299/300 Read block 2779 len 32768
15374 300/300 Read block 2479 len 20480
15375 301/300 Read block 336 len 28672
15376 Main task has work to do, go find it
15377 got dsc Start(2)
15378 start 2
15379 Jan 23 17:40:59.384 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/2, method: POST, req_id: 686cdee3-fe73-4655-85d7-ce91b450a820, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
15380 Replay: started 2, returned:Ok(())
15381 ----------------------------------------------------------------
15382 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15383 States: Active Active Offline
15384 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15385 5228 Acked Read 2 Done Done Sent false
15386 5229 Acked Flush 0 Done Done Sent false
15387 5230 Acked Read 7 Done Done Sent false
15388 5231 Acked Write 10 Done Done Sent false
15389 5232 Acked Read 5 Done Done Sent false
15390 5233 Acked Write 8 Done Done Sent false
15391 5234 Acked Write 10 Done Done Sent false
15392 5235 Acked Read 1 Done Done Sent false
15393 5236 Acked Flush 0 Done Done Sent false
15394 5237 Acked Read 5 Done Done Sent false
15395 5238 Acked Write 2 Done Done Sent false
15396 5239 Acked Read 4 Done Done Sent false
15397 5240 Acked Read 4 Done Done Sent false
15398 5241 Acked Read 7 Done Done Sent false
15399 5242 Acked Read 9 Done Done Sent false
15400 5243 Acked Flush 0 Done Done Sent false
15401 5244 Acked Flush 0 Done Done Sent false
15402 5245 Acked Read 1 Done Done Sent false
15403 5246 Acked Write 6 Done Done Sent false
15404 5247 Acked Write 1 Done Done Sent false
15405 5248 Acked Read 5 Done Done Sent false
15406 5249 Acked Write 5 Done Done Sent false
15407 5250 Acked Read 3 Done Done Sent false
15408 5251 Acked Read 2 Done Done Sent false
15409 5252 Acked Read 3 Done Done Sent false
15410 5253 Acked Write 2 Done Done Sent false
15411 5254 Acked Read 10 Done Done Sent false
15412 5255 Acked Read 9 Done Done Sent false
15413 5256 Acked Flush 0 Done Done Sent false
15414 5257 Acked Write 6 Done Done Sent false
15415 5258 Acked Write 8 Done Done Sent false
15416 5259 Acked Read 10 Done Done Sent false
15417 5260 Acked Write 9 Done Done Sent false
15418 5261 Acked Read 9 Done Done Sent false
15419 5262 Acked Read 8 Done Done Sent false
15420 5263 Acked Read 3 Done Done Sent false
15421 5264 Acked Write 3 Done Done Sent false
15422 5265 Acked Write 6 Done Done Sent false
15423 5266 Acked Read 6 Done Done Sent false
15424 5267 Acked Read 5 Done Done Sent false
15425 5268 Acked Read 10 Done Done Sent false
15426 5269 Acked Read 1 Done Done Sent false
15427 5270 Acked Write 8 Done Done Sent false
15428 5271 Acked Write 9 Done Done Sent false
15429 5272 Acked Read 4 Done Done Sent false
15430 5273 Acked Write 4 Done Done Sent false
15431 5274 Acked Read 2 Done Done Sent false
15432 5275 Acked Read 8 Done Done Sent false
15433 5276 Acked Write 8 Done Done Sent false
15434 5277 Acked Flush 0 Done Done Sent false
15435 5278 Acked Read 9 Done Done Sent false
15436 5279 Acked Write 2 Done Done Sent false
15437 5280 Acked Read 3 Done Done Sent false
15438 5281 Acked Write 6 Done Done Sent false
15439 5282 Acked Read 9 Done Done Sent false
15440 5283 Acked Read 3 Done Done Sent false
15441 5284 Acked Write 4 Done Done Sent false
15442 5285 Acked Write 6 Done Done Sent false
15443 5286 Acked Write 1 Done Done Sent false
15444 5287 Acked Write 1 Done Done Sent false
15445 5288 Acked Read 1 Done Done Sent false
15446 5289 Acked Write 3 Done Done Sent false
15447 5290 Acked Read 8 Done Done Sent false
15448 5291 Acked Read 10 Done Done Sent false
15449 5292 Acked Write 3 Done Done Sent false
15450 5293 Acked Write 7 Done Done Sent false
15451 5294 Acked Read 3 Done Done Sent false
15452 5295 Acked Flush 0 Done Done Sent false
15453 5296 Acked Read 8 Done Done Sent false
15454 5297 Acked Write 10 Done Done Sent false
15455 5298 Acked Write 3 Done Done Sent false
15456 5299 Acked Flush 0 Done Done Sent false
15457 5300 Acked Write 9 Done Done Sent false
15458 5301 Acked Read 10 Done Done Sent false
15459 5302 Acked Write 4 Done Done Sent false
15460 5303 Acked Write 4 Done Done Sent false
15461 5304 Acked Read 4 Done Done Sent false
15462 5305 Acked Read 2 Done Done Sent false
15463 5306 Acked Read 2 Done Done Sent false
15464 5307 Acked Read 6 Done Done Sent false
15465 5308 Acked Read 5 Done Done Sent false
15466 5309 Acked Write 3 Done Done Sent false
15467 5310 Acked Read 7 Done Done Sent false
15468 5311 Acked Flush 0 Done Done Sent false
15469 5312 Acked Read 1 Done Done Sent false
15470 5313 Acked Read 4 Done Done Sent false
15471 5314 Acked Read 5 Done Done Sent false
15472 5315 Acked Write 2 Done Done Sent false
15473 5316 Acked Write 1 Done Done Sent false
15474 5317 Acked Write 1 Done Done Sent false
15475 5318 Acked Flush 0 Done Done Sent false
15476 5319 Acked Write 8 Done Done Sent false
15477 5320 Acked Write 9 Done Done Sent false
15478 5321 Acked Read 4 Done Done Sent false
15479 5322 Acked Read 10 Done Done Sent false
15480 5323 Acked Write 7 Done Done Sent false
15481 5324 Acked Write 3 Done Done Sent false
15482 5325 Acked Write 1 Done Done Sent false
15483 5326 Acked Read 7 Done Done Sent false
15484 5327 Acked Read 7 Done Done Sent false
15485 5328 Acked Read 6 Done Done Sent false
15486 5329 Acked Write 3 Done Done Sent false
15487 5330 Acked Write 7 Done Done Sent false
15488 5331 Acked Flush 0 Done Done Sent false
15489 5332 Acked Write 5 Done Done Sent false
15490 5333 Acked Read 7 Done Done Sent false
15491 5334 Acked Write 3 Done Done Sent false
15492 5335 Acked Flush 0 Done Done Sent false
15493 5336 Acked Read 4 Done Done Sent false
15494 5337 Acked Write 6 Done Done Sent false
15495 5338 Acked Read 1 Done Done Sent false
15496 5339 Acked Read 4 Done Done Sent false
15497 5340 Acked Read 6 Done Done Sent false
15498 5341 Acked Flush 0 Done Done Sent false
15499 5342 Acked Write 2 Done Done Sent false
15500 5343 Acked Write 6 Done Done Sent false
15501 5344 Acked Flush 0 Done Done Sent false
15502 5345 Acked Read 6 Done Done Sent false
15503 5346 Acked Write 7 Done Done Sent false
15504 5347 Acked Write 7 Done Done Sent false
15505 5348 Acked Write 4 Done Done Sent false
15506 5349 Acked Read 3 Done Done Sent false
15507 5350 Acked Write 5 Done Done Sent false
15508 5351 Acked Write 4 Done Done Sent false
15509 5352 Acked Write 10 Done Done Sent false
15510 5353 Acked Write 2 Done Done Sent false
15511 5354 Acked Read 7 Done Done Sent false
15512 5355 Acked Read 7 Done Done Sent false
15513 5356 Acked Read 5 Done Done Sent false
15514 5357 Acked Write 7 Done Done Sent false
15515 5358 Acked Read 8 Done Done Sent false
15516 5359 Acked Read 4 Done Done Sent false
15517 5360 Acked Write 3 Done Done Sent false
15518 5361 Acked Flush 0 Done Done Sent false
15519 5362 Acked Read 5 Done Done Sent false
15520 5363 Acked Write 2 Done Done Sent false
15521 5364 Acked Read 9 Done Done Sent false
15522 5365 Acked Write 7 Done Done Sent false
15523 5366 Acked Write 5 Done Done Sent false
15524 5367 Acked Read 6 Done Done Sent false
15525 5368 Acked Write 2 Done Done Sent false
15526 5369 Acked Read 1 Done Done Sent false
15527 5370 Acked Write 8 Done Done Sent false
15528 5371 Acked Read 7 Done Done Sent false
15529 5372 Acked Write 2 Done Done Sent false
15530 5373 Acked Read 4 Done Done Sent false
15531 5374 Acked Write 2 Done Done Sent false
15532 5375 Acked Read 9 Done Done Sent false
15533 5376 Acked Read 8 Done Done Sent false
15534 5377 Acked Read 6 Done Done Sent false
15535 5378 Acked Read 3 Done Done Sent false
15536 5379 Acked Read 6 Done Done Sent false
15537 5380 Acked Read 5 Done Done Sent false
15538 5381 Acked Read 3 Done Done Sent false
15539 5382 Acked Read 5 Done Done Sent false
15540 5383 Acked Write 8 Done Done Sent false
15541 5384 Acked Flush 0 Done Done Sent false
15542 5385 Acked Read 5 Done Done Sent false
15543 5386 Acked Write 3 Done Done Sent false
15544 5387 Acked Read 6 Done Done Sent false
15545 5388 Acked Read 3 Done Done Sent false
15546 5389 Acked Write 10 Done Done Sent false
15547 5390 Acked Read 8 Done Done Sent false
15548 5391 Acked Write 9 Done Done Sent false
15549 5392 Acked Read 8 Done Done Sent false
15550 5393 Acked Read 2 Done Done Sent false
15551 5394 Acked Read 2 Done Done Sent false
15552 5395 Acked Read 4 Done Done Sent false
15553 5396 Acked Write 5 Done Done Sent false
15554 5397 Acked Flush 0 Done Done Sent false
15555 5398 Acked Flush 0 Done Done Sent false
15556 5399 Acked Read 8 Done Done Sent false
15557 5400 Acked Write 5 Done Done Sent false
15558 5401 Acked Write 3 Done Done Sent false
15559 5402 Acked Read 3 Done Done Sent false
15560 5403 Acked Flush 0 Done Done Sent false
15561 5404 Acked Read 3 Done Done Sent false
15562 5405 Acked Read 3 Done Done Sent false
15563 5406 Acked Write 9 Done Done Sent false
15564 5407 Acked Read 5 Done Done Sent false
15565 5408 Acked Read 2 Done Done Sent false
15566 5409 Acked Read 4 Done Done Sent false
15567 5410 Acked Read 8 Done Done Sent false
15568 5411 Acked Write 2 Done Done Sent false
15569 5412 Acked Read 9 Done Done Sent false
15570 5413 Acked Flush 0 Done Done Sent false
15571 5414 Acked Read 5 Done Done Sent false
15572 5415 Acked Write 10 Done Done Sent false
15573 5416 Acked Write 4 Done Done Sent false
15574 5417 Acked Read 10 Done Done Sent false
15575 5418 Acked Write 4 Done Done Sent false
15576 5419 Acked Read 7 Done Done Sent false
15577 5420 Acked Write 6 Done Done Sent false
15578 5421 Acked Write 5 Done Done Sent false
15579 5422 Acked Write 8 Done Done Sent false
15580 5423 Acked Write 2 Done Done Sent false
15581 5424 Acked Read 8 Done Done Sent false
15582 5425 Acked Read 9 Done Done Sent false
15583 5426 Acked Read 8 Done Done Sent false
15584 5427 Acked Read 1 Done Done Sent false
15585 5428 Acked Flush 0 Done Done Sent false
15586 5429 Acked Read 7 Done Done Sent false
15587 5430 Acked Write 1 Done Done Sent false
15588 5431 Acked Write 10 Done Done Sent false
15589 5432 Acked Flush 0 Done Done Sent false
15590 5433 Acked Write 7 Done Done Sent false
15591 5434 Acked Write 5 Done Done Sent false
15592 5435 Acked Write 6 Done Done Sent false
15593 5436 Acked Read 2 Done Done Sent false
15594 5437 Acked Read 7 Done Done Sent false
15595 5438 Acked Write 9 Done Done Sent false
15596 5439 Acked Write 9 Done Done Sent false
15597 5440 Acked Write 4 Done Done Sent false
15598 5441 Acked Write 6 Done Done Sent false
15599 5442 Acked Write 3 Done Done Sent false
15600 5443 Acked Flush 0 Done Done Sent false
15601 5444 Acked Write 3 Done Done Sent false
15602 5445 Acked Read 10 Done Done Sent false
15603 5446 Acked Read 9 Done Done Sent false
15604 5447 Acked Flush 0 Done Done Sent false
15605 5448 Acked Flush 0 Done Done Sent false
15606 5449 Acked Write 2 Done Done Sent false
15607 5450 Acked Read 2 Done Done Sent false
15608 5451 Acked Write 10 Done Done Sent false
15609 5452 Acked Read 6 Done Done Sent false
15610 5453 Acked Write 10 Done Done Sent false
15611 5454 Acked Write 7 Done Done Sent false
15612 5455 Acked Read 7 Done Done Sent false
15613 5456 Acked Read 7 Done Done Sent false
15614 5457 Acked Write 6 Done Done Sent false
15615 5458 Acked Read 7 Done Done Sent false
15616 5459 Acked Write 2 Done Done Sent false
15617 5460 Acked Write 1 Done Done Sent false
15618 5461 Acked Write 1 Done Done Sent false
15619 5462 Acked Write 6 Done Done Sent false
15620 5463 Acked Write 7 Done Done Sent false
15621 5464 Acked Flush 0 Done Done Sent false
15622 5465 Acked Read 4 Done Done Sent false
15623 5466 Acked Read 4 Done Done Sent false
15624 5467 Acked Read 4 Done Done Sent false
15625 5468 Acked Write 8 Done Done Sent false
15626 5469 Acked Flush 0 Done Done Sent false
15627 5470 Acked Read 9 Done Done Sent false
15628 5471 Acked Read 4 Done Done Sent false
15629 5472 Acked Flush 0 Done Done Sent false
15630 5473 Acked Write 3 Done Done Sent false
15631 5474 Acked Flush 0 Done Done Sent false
15632 5475 Acked Read 10 Done Done Sent false
15633 5476 Acked Write 9 Done Done Sent false
15634 5477 Acked Flush 0 Done Done Sent false
15635 5478 Acked Read 5 Done Done Sent false
15636 5479 Acked Read 9 Done Done Sent false
15637 5480 Acked Flush 0 Done Done Sent false
15638 5481 Acked Read 6 Done Done Sent false
15639 5482 Acked Write 5 Done Done Sent false
15640 5483 Acked Write 2 Done Done Sent false
15641 5484 Acked Read 1 Done Done Sent false
15642 5485 Acked Write 1 Done Done Sent false
15643 5486 Acked Read 3 Done Done Sent false
15644 5487 Acked Read 3 Done Done Sent false
15645 5488 Acked Read 3 Done Done Sent false
15646 5489 Acked Write 6 Done Done Sent false
15647 5490 Acked Read 10 Done Done Sent false
15648 5491 Acked Read 1 Done Done Sent false
15649 5492 Acked Write 1 Done Done Sent false
15650 5493 Acked Read 5 Done Done Sent false
15651 5494 Acked Write 4 Done Done Sent false
15652 5495 Acked Read 3 Done Done Sent false
15653 5496 Acked Read 4 Done Done Sent false
15654 5497 Acked Write 8 Done Done Sent false
15655 5498 Acked Write 3 Done Done Sent false
15656 5499 Acked Write 5 Done Done Sent false
15657 5500 Acked Read 7 Done Done Sent false
15658 5501 Acked Write 10 Done Done Sent false
15659 5502 Acked Flush 0 Done Done Sent false
15660 5503 Acked Read 9 Done Done Sent false
15661 5504 Acked Read 7 Done Done Sent false
15662 5505 Acked Read 2 Done Done Sent false
15663 5506 Acked Write 10 Done Done Sent false
15664 5507 Acked Read 3 Done Done Sent false
15665 5508 Acked Read 6 Done Done Sent false
15666 5509 Acked Flush 0 Done Done Sent false
15667 5510 Acked Read 7 Done Done Sent false
15668 5511 Acked Read 2 Done Done Sent false
15669 5512 Acked Write 9 Done Done Sent false
15670 5513 Acked Read 6 Done Done Sent false
15671 5514 Acked Read 4 Done Done Sent false
15672 5515 Acked Read 7 Done Done Sent false
15673 5516 Acked Read 8 Done Done Sent false
15674 5517 Acked Write 4 Done Done Sent false
15675 5518 Acked Read 7 Done Done Sent false
15676 5519 Acked Write 5 Done Done Sent false
15677 5520 Acked Write 5 Done Done Sent false
15678 5521 Acked Read 3 Done Done Sent false
15679 5522 Acked Write 6 Done Done Sent false
15680 5523 Acked Write 4 Done Done Sent false
15681 5524 Acked Read 5 Done Done Sent false
15682 5525 Acked Read 2 Done Done Sent false
15683 5526 Acked Read 8 Done Done Sent false
15684 5527 Acked Read 5 Done Done Sent false
15685 5528 Acked Read 7 Done Done Sent false
15686 STATES DS:0 DS:1 DS:2 TOTAL
15687 Sent 0 0 301 301
15688 Done 301 301 0 602
15689 Skipped 0 0 0 0
15690 Error 0 0 0 0
15691 Last Flush: Some(JobId(5509)) Some(JobId(5509)) Some(JobId(5227))
15692 Downstairs last five completed: 5227 5226 5225 5224 5223
15693 Upstairs last five completed: 5528 5527 5526 5525 5524
15694 CLIENT: Up:0 ds:301 act:2
15695 [8830] got start action
15696 [8830] I am going to restart
15697 Starting downstairs at port 8830
15698 Make output file at "/tmp/dsc/downstairs-8830.txt"
15699 [8830][2] reports Starting
15700 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1220)
15701 [8830][2] reports Running
15702 ----------------------------------------------------------------
15703 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
15704 States: Active Active Offline
15705 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15706 5228 Acked Read 2 Done Done Sent false
15707 5229 Acked Flush 0 Done Done Sent false
15708 5230 Acked Read 7 Done Done Sent false
15709 5231 Acked Write 10 Done Done Sent false
15710 5232 Acked Read 5 Done Done Sent false
15711 5233 Acked Write 8 Done Done Sent false
15712 5234 Acked Write 10 Done Done Sent false
15713 5235 Acked Read 1 Done Done Sent false
15714 5236 Acked Flush 0 Done Done Sent false
15715 5237 Acked Read 5 Done Done Sent false
15716 5238 Acked Write 2 Done Done Sent false
15717 5239 Acked Read 4 Done Done Sent false
15718 5240 Acked Read 4 Done Done Sent false
15719 5241 Acked Read 7 Done Done Sent false
15720 5242 Acked Read 9 Done Done Sent false
15721 5243 Acked Flush 0 Done Done Sent false
15722 5244 Acked Flush 0 Done Done Sent false
15723 5245 Acked Read 1 Done Done Sent false
15724 5246 Acked Write 6 Done Done Sent false
15725 5247 Acked Write 1 Done Done Sent false
15726 5248 Acked Read 5 Done Done Sent false
15727 5249 Acked Write 5 Done Done Sent false
15728 5250 Acked Read 3 Done Done Sent false
15729 5251 Acked Read 2 Done Done Sent false
15730 5252 Acked Read 3 Done Done Sent false
15731 5253 Acked Write 2 Done Done Sent false
15732 5254 Acked Read 10 Done Done Sent false
15733 5255 Acked Read 9 Done Done Sent false
15734 5256 Acked Flush 0 Done Done Sent false
15735 5257 Acked Write 6 Done Done Sent false
15736 5258 Acked Write 8 Done Done Sent false
15737 5259 Acked Read 10 Done Done Sent false
15738 5260 Acked Write 9 Done Done Sent false
15739 5261 Acked Read 9 Done Done Sent false
15740 5262 Acked Read 8 Done Done Sent false
15741 5263 Acked Read 3 Done Done Sent false
15742 5264 Acked Write 3 Done Done Sent false
15743 5265 Acked Write 6 Done Done Sent false
15744 5266 Acked Read 6 Done Done Sent false
15745 5267 Acked Read 5 Done Done Sent false
15746 5268 Acked Read 10 Done Done Sent false
15747 5269 Acked Read 1 Done Done Sent false
15748 5270 Acked Write 8 Done Done Sent false
15749 5271 Acked Write 9 Done Done Sent false
15750 5272 Acked Read 4 Done Done Sent false
15751 5273 Acked Write 4 Done Done Sent false
15752 5274 Acked Read 2 Done Done Sent false
15753 5275 Acked Read 8 Done Done Sent false
15754 5276 Acked Write 8 Done Done Sent false
15755 5277 Acked Flush 0 Done Done Sent false
15756 5278 Acked Read 9 Done Done Sent false
15757 5279 Acked Write 2 Done Done Sent false
15758 5280 Acked Read 3 Done Done Sent false
15759 5281 Acked Write 6 Done Done Sent false
15760 5282 Acked Read 9 Done Done Sent false
15761 5283 Acked Read 3 Done Done Sent false
15762 5284 Acked Write 4 Done Done Sent false
15763 5285 Acked Write 6 Done Done Sent false
15764 5286 Acked Write 1 Done Done Sent false
15765 5287 Acked Write 1 Done Done Sent false
15766 5288 Acked Read 1 Done Done Sent false
15767 5289 Acked Write 3 Done Done Sent false
15768 5290 Acked Read 8 Done Done Sent false
15769 5291 Acked Read 10 Done Done Sent false
15770 5292 Acked Write 3 Done Done Sent false
15771 5293 Acked Write 7 Done Done Sent false
15772 5294 Acked Read 3 Done Done Sent false
15773 5295 Acked Flush 0 Done Done Sent false
15774 5296 Acked Read 8 Done Done Sent false
15775 5297 Acked Write 10 Done Done Sent false
15776 5298 Acked Write 3 Done Done Sent false
15777 5299 Acked Flush 0 Done Done Sent false
15778 5300 Acked Write 9 Done Done Sent false
15779 5301 Acked Read 10 Done Done Sent false
15780 5302 Acked Write 4 Done Done Sent false
15781 5303 Acked Write 4 Done Done Sent false
15782 5304 Acked Read 4 Done Done Sent false
15783 5305 Acked Read 2 Done Done Sent false
15784 5306 Acked Read 2 Done Done Sent false
15785 5307 Acked Read 6 Done Done Sent false
15786 5308 Acked Read 5 Done Done Sent false
15787 5309 Acked Write 3 Done Done Sent false
15788 5310 Acked Read 7 Done Done Sent false
15789 5311 Acked Flush 0 Done Done Sent false
15790 5312 Acked Read 1 Done Done Sent false
15791 5313 Acked Read 4 Done Done Sent false
15792 5314 Acked Read 5 Done Done Sent false
15793 5315 Acked Write 2 Done Done Sent false
15794 5316 Acked Write 1 Done Done Sent false
15795 5317 Acked Write 1 Done Done Sent false
15796 5318 Acked Flush 0 Done Done Sent false
15797 5319 Acked Write 8 Done Done Sent false
15798 5320 Acked Write 9 Done Done Sent false
15799 5321 Acked Read 4 Done Done Sent false
15800 5322 Acked Read 10 Done Done Sent false
15801 5323 Acked Write 7 Done Done Sent false
15802 5324 Acked Write 3 Done Done Sent false
15803 5325 Acked Write 1 Done Done Sent false
15804 5326 Acked Read 7 Done Done Sent false
15805 5327 Acked Read 7 Done Done Sent false
15806 5328 Acked Read 6 Done Done Sent false
15807 5329 Acked Write 3 Done Done Sent false
15808 5330 Acked Write 7 Done Done Sent false
15809 5331 Acked Flush 0 Done Done Sent false
15810 5332 Acked Write 5 Done Done Sent false
15811 5333 Acked Read 7 Done Done Sent false
15812 5334 Acked Write 3 Done Done Sent false
15813 5335 Acked Flush 0 Done Done Sent false
15814 5336 Acked Read 4 Done Done Sent false
15815 5337 Acked Write 6 Done Done Sent false
15816 5338 Acked Read 1 Done Done Sent false
15817 5339 Acked Read 4 Done Done Sent false
15818 5340 Acked Read 6 Done Done Sent false
15819 5341 Acked Flush 0 Done Done Sent false
15820 5342 Acked Write 2 Done Done Sent false
15821 5343 Acked Write 6 Done Done Sent false
15822 5344 Acked Flush 0 Done Done Sent false
15823 5345 Acked Read 6 Done Done Sent false
15824 5346 Acked Write 7 Done Done Sent false
15825 5347 Acked Write 7 Done Done Sent false
15826 5348 Acked Write 4 Done Done Sent false
15827 5349 Acked Read 3 Done Done Sent false
15828 5350 Acked Write 5 Done Done Sent false
15829 5351 Acked Write 4 Done Done Sent false
15830 5352 Acked Write 10 Done Done Sent false
15831 5353 Acked Write 2 Done Done Sent false
15832 5354 Acked Read 7 Done Done Sent false
15833 5355 Acked Read 7 Done Done Sent false
15834 5356 Acked Read 5 Done Done Sent false
15835 5357 Acked Write 7 Done Done Sent false
15836 5358 Acked Read 8 Done Done Sent false
15837 5359 Acked Read 4 Done Done Sent false
15838 5360 Acked Write 3 Done Done Sent false
15839 5361 Acked Flush 0 Done Done Sent false
15840 5362 Acked Read 5 Done Done Sent false
15841 5363 Acked Write 2 Done Done Sent false
15842 5364 Acked Read 9 Done Done Sent false
15843 5365 Acked Write 7 Done Done Sent false
15844 5366 Acked Write 5 Done Done Sent false
15845 5367 Acked Read 6 Done Done Sent false
15846 5368 Acked Write 2 Done Done Sent false
15847 5369 Acked Read 1 Done Done Sent false
15848 5370 Acked Write 8 Done Done Sent false
15849 5371 Acked Read 7 Done Done Sent false
15850 5372 Acked Write 2 Done Done Sent false
15851 5373 Acked Read 4 Done Done Sent false
15852 5374 Acked Write 2 Done Done Sent false
15853 5375 Acked Read 9 Done Done Sent false
15854 5376 Acked Read 8 Done Done Sent false
15855 5377 Acked Read 6 Done Done Sent false
15856 5378 Acked Read 3 Done Done Sent false
15857 5379 Acked Read 6 Done Done Sent false
15858 5380 Acked Read 5 Done Done Sent false
15859 5381 Acked Read 3 Done Done Sent false
15860 5382 Acked Read 5 Done Done Sent false
15861 5383 Acked Write 8 Done Done Sent false
15862 5384 Acked Flush 0 Done Done Sent false
15863 5385 Acked Read 5 Done Done Sent false
15864 5386 Acked Write 3 Done Done Sent false
15865 5387 Acked Read 6 Done Done Sent false
15866 5388 Acked Read 3 Done Done Sent false
15867 5389 Acked Write 10 Done Done Sent false
15868 5390 Acked Read 8 Done Done Sent false
15869 5391 Acked Write 9 Done Done Sent false
15870 5392 Acked Read 8 Done Done Sent false
15871 5393 Acked Read 2 Done Done Sent false
15872 5394 Acked Read 2 Done Done Sent false
15873 5395 Acked Read 4 Done Done Sent false
15874 5396 Acked Write 5 Done Done Sent false
15875 5397 Acked Flush 0 Done Done Sent false
15876 5398 Acked Flush 0 Done Done Sent false
15877 5399 Acked Read 8 Done Done Sent false
15878 5400 Acked Write 5 Done Done Sent false
15879 5401 Acked Write 3 Done Done Sent false
15880 5402 Acked Read 3 Done Done Sent false
15881 5403 Acked Flush 0 Done Done Sent false
15882 5404 Acked Read 3 Done Done Sent false
15883 5405 Acked Read 3 Done Done Sent false
15884 5406 Acked Write 9 Done Done Sent false
15885 5407 Acked Read 5 Done Done Sent false
15886 5408 Acked Read 2 Done Done Sent false
15887 5409 Acked Read 4 Done Done Sent false
15888 5410 Acked Read 8 Done Done Sent false
15889 5411 Acked Write 2 Done Done Sent false
15890 5412 Acked Read 9 Done Done Sent false
15891 5413 Acked Flush 0 Done Done Sent false
15892 5414 Acked Read 5 Done Done Sent false
15893 5415 Acked Write 10 Done Done Sent false
15894 5416 Acked Write 4 Done Done Sent false
15895 5417 Acked Read 10 Done Done Sent false
15896 5418 Acked Write 4 Done Done Sent false
15897 5419 Acked Read 7 Done Done Sent false
15898 5420 Acked Write 6 Done Done Sent false
15899 5421 Acked Write 5 Done Done Sent false
15900 5422 Acked Write 8 Done Done Sent false
15901 5423 Acked Write 2 Done Done Sent false
15902 5424 Acked Read 8 Done Done Sent false
15903 5425 Acked Read 9 Done Done Sent false
15904 5426 Acked Read 8 Done Done Sent false
15905 5427 Acked Read 1 Done Done Sent false
15906 5428 Acked Flush 0 Done Done Sent false
15907 5429 Acked Read 7 Done Done Sent false
15908 5430 Acked Write 1 Done Done Sent false
15909 5431 Acked Write 10 Done Done Sent false
15910 5432 Acked Flush 0 Done Done Sent false
15911 5433 Acked Write 7 Done Done Sent false
15912 5434 Acked Write 5 Done Done Sent false
15913 5435 Acked Write 6 Done Done Sent false
15914 5436 Acked Read 2 Done Done Sent false
15915 5437 Acked Read 7 Done Done Sent false
15916 5438 Acked Write 9 Done Done Sent false
15917 5439 Acked Write 9 Done Done Sent false
15918 5440 Acked Write 4 Done Done Sent false
15919 5441 Acked Write 6 Done Done Sent false
15920 5442 Acked Write 3 Done Done Sent false
15921 5443 Acked Flush 0 Done Done Sent false
15922 5444 Acked Write 3 Done Done Sent false
15923 5445 Acked Read 10 Done Done Sent false
15924 5446 Acked Read 9 Done Done Sent false
15925 5447 Acked Flush 0 Done Done Sent false
15926 5448 Acked Flush 0 Done Done Sent false
15927 5449 Acked Write 2 Done Done Sent false
15928 5450 Acked Read 2 Done Done Sent false
15929 5451 Acked Write 10 Done Done Sent false
15930 5452 Acked Read 6 Done Done Sent false
15931 5453 Acked Write 10 Done Done Sent false
15932 5454 Acked Write 7 Done Done Sent false
15933 5455 Acked Read 7 Done Done Sent false
15934 5456 Acked Read 7 Done Done Sent false
15935 5457 Acked Write 6 Done Done Sent false
15936 5458 Acked Read 7 Done Done Sent false
15937 5459 Acked Write 2 Done Done Sent false
15938 5460 Acked Write 1 Done Done Sent false
15939 5461 Acked Write 1 Done Done Sent false
15940 5462 Acked Write 6 Done Done Sent false
15941 5463 Acked Write 7 Done Done Sent false
15942 5464 Acked Flush 0 Done Done Sent false
15943 5465 Acked Read 4 Done Done Sent false
15944 5466 Acked Read 4 Done Done Sent false
15945 5467 Acked Read 4 Done Done Sent false
15946 5468 Acked Write 8 Done Done Sent false
15947 5469 Acked Flush 0 Done Done Sent false
15948 5470 Acked Read 9 Done Done Sent false
15949 5471 Acked Read 4 Done Done Sent false
15950 5472 Acked Flush 0 Done Done Sent false
15951 5473 Acked Write 3 Done Done Sent false
15952 5474 Acked Flush 0 Done Done Sent false
15953 5475 Acked Read 10 Done Done Sent false
15954 5476 Acked Write 9 Done Done Sent false
15955 5477 Acked Flush 0 Done Done Sent false
15956 5478 Acked Read 5 Done Done Sent false
15957 5479 Acked Read 9 Done Done Sent false
15958 5480 Acked Flush 0 Done Done Sent false
15959 5481 Acked Read 6 Done Done Sent false
15960 5482 Acked Write 5 Done Done Sent false
15961 5483 Acked Write 2 Done Done Sent false
15962 5484 Acked Read 1 Done Done Sent false
15963 5485 Acked Write 1 Done Done Sent false
15964 5486 Acked Read 3 Done Done Sent false
15965 5487 Acked Read 3 Done Done Sent false
15966 5488 Acked Read 3 Done Done Sent false
15967 5489 Acked Write 6 Done Done Sent false
15968 5490 Acked Read 10 Done Done Sent false
15969 5491 Acked Read 1 Done Done Sent false
15970 5492 Acked Write 1 Done Done Sent false
15971 5493 Acked Read 5 Done Done Sent false
15972 5494 Acked Write 4 Done Done Sent false
15973 5495 Acked Read 3 Done Done Sent false
15974 5496 Acked Read 4 Done Done Sent false
15975 5497 Acked Write 8 Done Done Sent false
15976 5498 Acked Write 3 Done Done Sent false
15977 5499 Acked Write 5 Done Done Sent false
15978 5500 Acked Read 7 Done Done Sent false
15979 5501 Acked Write 10 Done Done Sent false
15980 5502 Acked Flush 0 Done Done Sent false
15981 5503 Acked Read 9 Done Done Sent false
15982 5504 Acked Read 7 Done Done Sent false
15983 5505 Acked Read 2 Done Done Sent false
15984 5506 Acked Write 10 Done Done Sent false
15985 5507 Acked Read 3 Done Done Sent false
15986 5508 Acked Read 6 Done Done Sent false
15987 5509 Acked Flush 0 Done Done Sent false
15988 5510 Acked Read 7 Done Done Sent false
15989 5511 Acked Read 2 Done Done Sent false
15990 5512 Acked Write 9 Done Done Sent false
15991 5513 Acked Read 6 Done Done Sent false
15992 5514 Acked Read 4 Done Done Sent false
15993 5515 Acked Read 7 Done Done Sent false
15994 5516 Acked Read 8 Done Done Sent false
15995 5517 Acked Write 4 Done Done Sent false
15996 5518 Acked Read 7 Done Done Sent false
15997 5519 Acked Write 5 Done Done Sent false
15998 5520 Acked Write 5 Done Done Sent false
15999 5521 Acked Read 3 Done Done Sent false
16000 5522 Acked Write 6 Done Done Sent false
16001 5523 Acked Write 4 Done Done Sent false
16002 5524 Acked Read 5 Done Done Sent false
16003 5525 Acked Read 2 Done Done Sent false
16004 5526 Acked Read 8 Done Done Sent false
16005 5527 Acked Read 5 Done Done Sent false
16006 5528 Acked Read 7 Done Done Sent false
16007 5529 Acked Flush 0 Done Done Sent false
16008 STATES DS:0 DS:1 DS:2 TOTAL
16009 Sent 0 0 302 302
16010 Done 302 302 0 604
16011 Skipped 0 0 0 0
16012 Error 0 0 0 0
16013 Last Flush: Some(JobId(5529)) Some(JobId(5529)) Some(JobId(5227))
16014 Downstairs last five completed: 5227 5226 5225 5224 5223
16015 Upstairs last five completed: 5529 5528 5527 5526 5525
16016 CLIENT: Up:0 ds:302 act:2
160172025-01-23T17:41:05.144ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
160182025-01-23T17:41:05.144ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53473) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
160192025-01-23T17:41:05.145ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
160202025-01-23T17:41:05.146ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
160212025-01-23T17:41:05.146ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
160222025-01-23T17:41:05.146ZINFOcrucible: send last flush ID to this DS: Some(JobId(5227)) = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
160232025-01-23T17:41:05.146ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
160242025-01-23T17:41:05.146ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(5227)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
160252025-01-23T17:41:05.146ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(5227)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
16026 ----------------------------------------------------------------
16027 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16028 Downstairs last five completed: 5529 5528 5527 5526 5525
16029 Upstairs last five completed: 5529 5528 5527 5526 5525
16030 CLIENT: Up:0 ds:0 act:3
16031 Replay: All jobs finished
16032 Main task has work to do, go find it
16033 got dsc Stop(2)
16034 stop 2
16035 [8830] Got stop action so:false kr:false
16036 Jan 23 17:41:07.390 INFO request completed, latency_us: 329, response_code: 204, uri: /stop/cid/2, method: POST, req_id: ba19383e-eddd-4808-9a10-8c480b46deae, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
160372025-01-23T17:41:07.390ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
160382025-01-23T17:41:07.390ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
16039 Jan 23 17:41:07.390 INFO request completed, latency_us: 148, {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:41:07.391005885Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
16040 response_code: 200, uri: /state/cid/2, method: GET, req_id: 37210785-89e0-4053-b456-0f191d4ea6d1, remote_addr: 127.0.0{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T17:41:07.391094717Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
16041 .1:42015, local_addr: 127.0.0.1:9998
16042 [8830] Exited with: ExitStatus(unix_wait_status(9))
16043 [8830][2] reports Exit
16044 Jan 23 17:41:11.393 INFO request completed, latency_us: 224, response_code: 200, uri: /state/cid/2, method: GET, req_id: c9a1eb87-11e8-4116-927f-f7411e91c0b9, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
16045 001/300 Read block 3721 len 8192
16046 002/300 Read block 1993 len 20480
16047 003/300 Write block 3912 len 40960 data: 5 5 5 4 3 3 2 2 4 4
16048 004/300 Read block 1675 len 4096
16049 005/300 Write block 1564 len 32768 data: 4 5 5 5 6 5 5 4
16050 006/300 Read block 2257 len 8192
16051 007/300 Read block 1254 len 32768
16052 008/300 Read block 3213 len 20480
16053 009/300 Read block 3083 len 20480
16054 010/300 Read block 3628 len 24576
16055 011/300 Read block 3315 len 32768
16056 012/300 Read block 4127 len 28672
16057 013/300 Write block 4747 len 8192 data: 4 3
16058 014/300 Read block 1905 len 16384
16059 015/300 Read block 3745 len 32768
16060 016/300 Write block 2350 len 32768 data: 4 5 3 4 3 4 4 5
16061 017/300 Write block 1961 len 12288 data: 5 4 4
16062 018/300 Write block 1349 len 40960 data: 4 4 4 3 4 4 3 2 2 2
16063 019/300 Read block 2312 len 8192
16064 020/300 Read block 2460 len 16384
16065 021/300 Read block 234 len 36864
16066 022/300 Write block 4170 len 28672 data: 2 2 2 2 3 3 4
16067 023/300 Write block 3579 len 20480 data: 5 5 5 5 5
16068 024/300 Write block 2824 len 4096 data: 3
16069 025/300 Read block 1498 len 4096
16070 026/300 Write block 2591 len 4096 data: 5
16071 027/300 Write block 640 len 24576 data: 7 7 6 5 5 5
16072 028/300 Write block 4520 len 32768 data: 4 4 4 3 3 3 3 3
16073 029/300 Write block 2160 len 16384 data: 4 4 4 4
16074 030/300 Write block 1032 len 16384 data: 2 2 2 3
16075 031/300 Write block 2627 len 24576 data: 3 4 5 5 5 4
16076 032/300 Write block 2674 len 24576 data: 5 3 4 3 3 2
16077 033/300 Write block 1999 len 28672 data: 3 4 4 4 3 3 3
16078 034/300 Write block 2744 len 20480 data: 3 5 6 7 8
16079 035/300 Read block 703 len 16384
16080 036/300 Read block 4165 len 16384
16081 037/300 Write block 4042 len 12288 data: 3 3 4
16082 038/300 Read block 3973 len 32768
16083 039/300 Read block 2059 len 8192
16084 040/300 Read block 379 len 12288
16085 041/300 Flush
16086 042/300 Write block 4449 len 12288 data: 4 4 4
16087 043/300 Read block 3164 len 32768
16088 044/300 Read block 3857 len 24576
16089 045/300 Read block 2088 len 12288
16090 046/300 Write block 78 len 32768 data: 5 5 6 4 4 4 4 3
16091 047/300 Read block 2608 len 32768
16092 048/300 Read block 3809 len 8192
16093 049/300 Write block 2937 len 36864 data: 3 4 4 3 4 4 5 6 5
16094 050/300 Write block 1068 len 16384 data: 4 4 4 5
16095 051/300 Read block 4560 len 8192
16096 052/300 Write block 169 len 12288 data: 4 6 7
16097 053/300 Read block 4174 len 12288
16098 054/300 Read block 3299 len 4096
16099 055/300 Read block 1206 len 40960
16100 056/300 Read block 1601 len 40960
16101 057/300 Write block 982 len 20480 data: 5 4 6 6 6
16102 058/300 Read block 20 len 20480
16103 059/300 Write block 3859 len 20480 data: 5 4 3 3 4
16104 060/300 Write block 2927 len 4096 data: 5
16105 061/300 Read block 826 len 16384
16106 062/300 Read block 1259 len 16384
16107 063/300 Read block 2622 len 20480
16108 064/300 Write block 2191 len 4096 data: 4
16109 065/300 Write block 817 len 12288 data: 3 2 3
16110 066/300 Write block 1311 len 12288 data: 7 5 5
16111 067/300 Read block 2449 len 12288
16112 068/300 Read block 3704 len 12288
16113 069/300 Write block 2205 len 24576 data: 3 3 4 5 4 4
16114 070/300 Read block 4680 len 40960
16115 071/300 Read block 3545 len 20480
16116 072/300 Write block 665 len 32768 data: 3 3 3 5 5 6 6 7
16117 073/300 Write block 1115 len 8192 data: 6 5
16118 074/300 Write block 329 len 4096 data: 5
16119 075/300 Write block 937 len 40960 data: 4 4 3 4 3 3 3 3 3 3
16120 076/300 Read block 4248 len 40960
16121 077/300 Read block 4085 len 4096
16122 078/300 Write block 66 len 28672 data: 4 6 7 7 8 7 7
16123 079/300 Read block 1096 len 16384
16124 080/300 Write block 3680 len 4096 data: 5
16125 081/300 Read block 717 len 20480
16126 082/300 Write block 3442 len 12288 data: 6 6 5
16127 083/300 Write block 4475 len 36864 data: 3 3 3 2 2 2 2 2 2
16128 084/300 Read block 898 len 32768
16129 085/300 Write block 4993 len 12288 data: 4 4 4
16130 086/300 Write block 1344 len 20480 data: 3 3 4 4 4
16131 087/300 Write block 707 len 40960 data: 2 2 2 2 2 3 3 3 3 4
16132 088/300 Write block 3426 len 16384 data: 6 5 4 7
16133 089/300 Write block 4654 len 32768 data: 4 4 4 4 3 3 2 2
16134 090/300 Write block 3044 len 28672 data: 6 4 4 4 4 3 3
16135 091/300 Flush
16136 092/300 Read block 3467 len 8192
16137 093/300 Read block 2918 len 24576
16138 094/300 Flush
16139 095/300 Write block 1873 len 12288 data: 4 5 4
16140 096/300 Write block 4920 len 32768 data: 4 4 4 5 5 5 5 4
16141 097/300 Write block 4135 len 36864 data: 4 4 4 4 4 6 6 5 5
16142 098/300 Write block 1942 len 28672 data: 5 5 5 4 5 5 5
16143 099/300 Read block 141 len 12288
16144 100/300 Read block 4298 len 24576
16145 101/300 Write block 38 len 12288 data: 4 4 4
16146 102/300 Write block 2036 len 32768 data: 2 2 2 2 2 3 2 3
16147 103/300 Flush
16148 104/300 Flush
16149 105/300 Read block 4074 len 8192
16150 106/300 Read block 4401 len 40960
16151 107/300 Read block 1128 len 40960
16152 108/300 Write block 1162 len 24576 data: 9 9 9 6 3 3
16153 109/300 Read block 1630 len 4096
16154 110/300 Read block 4579 len 16384
16155 111/300 Write block 979 len 24576 data: 2 3 4 6 5 7
16156 112/300 Flush
16157 113/300 Read block 3495 len 28672
16158 114/300 Write block 2009 len 24576 data: 4 4 5 5 4 4
16159 115/300 Read block 1342 len 4096
16160 116/300 Write block 4867 len 24576 data: 5 5 7 8 5 5
16161 117/300 Write block 4830 len 24576 data: 3 2 2 2 3 3
16162 118/300 Read block 4291 len 12288
16163 119/300 Write block 3746 len 8192 data: 5 4
16164 120/300 Write block 1317 len 32768 data: 4 3 3 4 4 4 3 4
16165 121/300 Read block 3902 len 4096
16166 122/300 Read block 1507 len 24576
16167 123/300 Write block 974 len 24576 data: 5 5 3 3 3 3
16168 124/300 Write block 2349 len 32768 data: 5 5 6 4 5 4 5 5
16169 125/300 Flush
16170 126/300 Write block 585 len 8192 data: 9 6
16171 127/300 Read block 1865 len 24576
16172 128/300 Read block 2649 len 24576
16173 129/300 Flush
16174 130/300 Write block 4618 len 12288 data: 8 8 7
16175 131/300 Write block 835 len 20480 data: 4 4 4 4 5
16176 132/300 Read block 582 len 40960
16177 133/300 Flush
16178 134/300 Flush
16179 135/300 Write block 3378 len 16384 data: 2 2 2 2
16180 136/300 Flush
16181 137/300 Read block 4670 len 24576
16182 138/300 Write block 2228 len 12288 data: 6 5 6
16183 139/300 Write block 3245 len 8192 data: 4 3
16184 140/300 Write block 4886 len 16384 data: 4 4 3 3
16185 141/300 Read block 2929 len 16384
16186 142/300 Read block 2119 len 28672
16187 143/300 Write block 4230 len 24576 data: 3 2 3 4 5 4
16188 144/300 Read block 3665 len 24576
16189 145/300 Read block 18 len 36864
16190 146/300 Write block 2396 len 24576 data: 7 8 9 8 8 5
16191 147/300 Read block 3357 len 24576
16192 148/300 Read block 2296 len 28672
16193 149/300 Read block 2515 len 36864
16194 150/300 Write block 2277 len 20480 data: 4 4 4 6 6
16195 151/300 Flush
16196 152/300 Write block 3115 len 28672 data: 6 6 6 5 4 3 2
16197 153/300 Flush
16198 154/300 Flush
16199 155/300 Read block 3973 len 20480
16200 156/300 Read block 4469 len 24576
16201 157/300 Read block 3994 len 8192
16202 158/300 Read block 13 len 16384
16203 159/300 Read block 2025 len 28672
16204 160/300 Write block 4511 len 24576 data: 3 3 4 4 5 4
16205 161/300 Write block 1347 len 4096 data: 5
16206 162/300 Read block 4597 len 28672
16207 163/300 Write block 3799 len 16384 data: 3 3 2 2
16208 164/300 Read block 1616 len 36864
16209 165/300 Flush
16210 166/300 Write block 2309 len 20480 data: 3 3 3 3 3
16211 167/300 Read block 1323 len 12288
16212 168/300 Write block 4258 len 28672 data: 5 5 4 4 3 2 3
16213 169/300 Flush
16214 170/300 Read block 3815 len 36864
16215 171/300 Read block 3027 len 20480
16216 172/300 Read block 4419 len 28672
16217 173/300 Read block 4418 len 32768
16218 174/300 Read block 345 len 4096
16219 175/300 Write block 2016 len 36864 data: 3 2 2 2 3 3 3 3 3
16220 176/300 Flush
16221 177/300 Read block 590 len 16384
16222 178/300 Read block 1654 len 36864
16223 179/300 Write block 809 len 24576 data: 3 4 4 6 5 4
16224 180/300 Read block 1102 len 24576
16225 181/300 Read block 4879 len 24576
16226 182/300 Read block 2000 len 24576
16227 183/300 Read block 2243 len 16384
16228 184/300 Flush
16229 185/300 Write block 3306 len 4096 data: 6
16230 186/300 Read block 3021 len 20480
16231 187/300 Read block 3682 len 8192
16232 188/300 Write block 3031 len 36864 data: 4 5 6 5 6 5 5 4 4
16233 189/300 Read block 2244 len 4096
16234 190/300 Write block 2482 len 36864 data: 4 4 4 4 3 3 3 3 5
16235 191/300 Flush
16236 192/300 Write block 4808 len 20480 data: 2 3 4 4 5
16237 193/300 Write block 1310 len 4096 data: 7
16238 194/300 Read block 560 len 40960
16239 195/300 Write block 4831 len 16384 data: 3 3 3 4
16240 196/300 Read block 1654 len 16384
16241 197/300 Flush
16242 198/300 Read block 2780 len 24576
16243 199/300 Read block 3401 len 20480
16244 200/300 Write block 3805 len 40960 data: 3 4 4 4 3 3 5 5 4 5
16245 201/300 Read block 1636 len 24576
16246 202/300 Read block 560 len 8192
16247 203/300 Read block 3382 len 20480
16248 204/300 Read block 2485 len 16384
16249 205/300 Write block 1499 len 16384 data: 3 4 4 5
16250 206/300 Read block 3508 len 16384
16251 207/300 Read block 79 len 36864
16252 208/300 Read block 4347 len 8192
16253 209/300 Read block 2830 len 24576
16254 210/300 Read block 2454 len 28672
16255 211/300 Read block 4601 len 28672
16256 212/300 Read block 2910 len 40960
16257 213/300 Write block 3997 len 28672 data: 4 4 4 5 3 2 2
16258 214/300 Write block 4192 len 36864 data: 2 3 3 3 3 3 3 3 4
16259 215/300 Read block 2908 len 24576
16260 216/300 Read block 51 len 36864
16261 217/300 Read block 1180 len 4096
16262 218/300 Read block 3970 len 28672
16263 219/300 Read block 3771 len 16384
16264 220/300 Write block 3701 len 4096 data: 4
16265 221/300 Read block 3899 len 32768
16266 222/300 Read block 735 len 32768
16267 223/300 Write block 2666 len 16384 data: 3 3 3 4
16268 224/300 Write block 1988 len 20480 data: 3 3 3 3 3
16269 225/300 Read block 3643 len 32768
16270 226/300 Read block 3214 len 28672
16271 227/300 Write block 211 len 36864 data: 5 5 5 5 4 4 6 6 3
16272 228/300 Read block 1784 len 12288
16273 229/300 Read block 1646 len 20480
16274 230/300 Read block 1538 len 4096
16275 231/300 Read block 3361 len 36864
16276 232/300 Flush
16277 233/300 Write block 67 len 16384 data: 7 8 8 9
16278 234/300 Flush
16279 235/300 Write block 3374 len 8192 data: 4 3
16280 236/300 Write block 1042 len 24576 data: 2 4 4 4 3 3
16281 237/300 Flush
16282 238/300 Read block 3643 len 20480
16283 239/300 Read block 500 len 28672
16284 240/300 Write block 3334 len 4096 data: 4
16285 241/300 Read block 3539 len 36864
16286 242/300 Write block 3102 len 40960 data: 4 4 4 4 5 4 5 5 7 8
16287 243/300 Read block 3233 len 24576
16288 244/300 Write block 4700 len 36864 data: 4 4 4 5 5 4 5 5 6
16289 245/300 Flush
16290 246/300 Write block 4102 len 24576 data: 3 3 3 3 3 3
16291 247/300 Flush
16292 248/300 Read block 249 len 8192
16293 249/300 Read block 2806 len 8192
16294 250/300 Flush
16295 251/300 Write block 1767 len 28672 data: 3 3 4 4 3 3 3
16296 252/300 Read block 2390 len 20480
16297 253/300 Write block 2180 len 16384 data: 3 5 4 4
16298 254/300 Write block 1488 len 4096 data: 4
16299 255/300 Read block 2729 len 40960
16300 256/300 Flush
16301 257/300 Flush
16302 258/300 Read block 2855 len 32768
16303 259/300 Read block 2162 len 16384
16304 260/300 Read block 2362 len 20480
16305 261/300 Read block 3991 len 12288
16306 262/300 Write block 1692 len 16384 data: 4 5 4 4
16307 263/300 Read block 851 len 16384
16308 264/300 Read block 3465 len 24576
16309 265/300 Write block 4126 len 4096 data: 3
16310 266/300 Read block 4296 len 16384
16311 267/300 Read block 3354 len 36864
16312 268/300 Read block 4496 len 36864
16313 269/300 Read block 4215 len 12288
16314 270/300 Write block 3998 len 20480 data: 5 5 6 4 3
16315 271/300 Read block 3517 len 8192
16316 272/300 Read block 3437 len 12288
16317 273/300 Write block 2281 len 40960 data: 7 6 6 5 4 5 5 4 3 4
16318 274/300 Write block 4571 len 24576 data: 6 4 4 4 4 5
16319 275/300 Write block 564 len 32768 data: 2 3 4 4 4 5 7 8
16320 276/300 Read block 2436 len 24576
16321 277/300 Read block 463 len 28672
16322 278/300 Write block 4745 len 8192 data: 5 5
16323 279/300 Read block 4800 len 16384
16324 280/300 Read block 2522 len 12288
16325 281/300 Write block 4945 len 12288 data: 6 5 4
16326 282/300 Read block 1945 len 4096
16327 283/300 Write block 1086 len 36864 data: 4 5 6 6 6 5 4 3 3
16328 284/300 Write block 4377 len 20480 data: 4 4 4 4 4
16329 285/300 Read block 4702 len 8192
16330 286/300 Read block 1785 len 12288
16331 287/300 Write block 4144 len 36864 data: 5 4 3 3 4 5 5 5 5
16332 288/300 Write block 2499 len 32768 data: 4 2 3 3 3 2 2 2
16333 289/300 Read block 66 len 16384
16334 290/300 Read block 1715 len 4096
16335 291/300 Write block 1908 len 36864 data: 10 8 6 6 5 5 5 6 5
16336 292/300 Flush
16337 293/300 Write block 4061 len 16384 data: 3 3 3 3
16338 294/300 Write block 1383 len 20480 data: 2 2 2 2 2
16339 295/300 Write block 3652 len 8192 data: 5 6
16340 296/300 Read block 394 len 16384
16341 297/300 Write block 4678 len 24576 data: 4 4 4 5 7 6
16342 298/300 Read block 1866 len 24576
16343 299/300 Read block 362 len 16384
16344 300/300 Flush
16345 301/300 Read block 4930 len 16384
16346 Main task has work to do, go find it
16347 got dsc Start(2)
16348 start 2
16349 Jan 23 17:41:11.632 INFO request completed, latency_us: 242, response_code: 204, uri: /start/cid/2, method: POST, req_id: e4059968-56b3-492c-805f-294f7f90390c, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
16350 Replay: started 2, returned:Ok(())
16351 ----------------------------------------------------------------
16352 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16353 States: Active Active Offline
16354 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16355 5530 Acked Read 2 Done Done Sent false
16356 5531 Acked Read 5 Done Done Sent false
16357 5532 Acked Write 10 Done Done Sent false
16358 5533 Acked Read 1 Done Done Sent false
16359 5534 Acked Write 8 Done Done Sent false
16360 5535 Acked Read 2 Done Done Sent false
16361 5536 Acked Read 8 Done Done Sent false
16362 5537 Acked Read 5 Done Done Sent false
16363 5538 Acked Read 5 Done Done Sent false
16364 5539 Acked Read 6 Done Done Sent false
16365 5540 Acked Read 8 Done Done Sent false
16366 5541 Acked Read 7 Done Done Sent false
16367 5542 Acked Write 2 Done Done Sent false
16368 5543 Acked Read 4 Done Done Sent false
16369 5544 Acked Read 8 Done Done Sent false
16370 5545 Acked Write 8 Done Done Sent false
16371 5546 Acked Write 3 Done Done Sent false
16372 5547 Acked Write 10 Done Done Sent false
16373 5548 Acked Read 2 Done Done Sent false
16374 5549 Acked Read 4 Done Done Sent false
16375 5550 Acked Read 9 Done Done Sent false
16376 5551 Acked Write 7 Done Done Sent false
16377 5552 Acked Write 5 Done Done Sent false
16378 5553 Acked Write 1 Done Done Sent false
16379 5554 Acked Read 1 Done Done Sent false
16380 5555 Acked Write 1 Done Done Sent false
16381 5556 Acked Write 6 Done Done Sent false
16382 5557 Acked Write 8 Done Done Sent false
16383 5558 Acked Write 4 Done Done Sent false
16384 5559 Acked Write 4 Done Done Sent false
16385 5560 Acked Write 6 Done Done Sent false
16386 5561 Acked Write 6 Done Done Sent false
16387 5562 Acked Write 7 Done Done Sent false
16388 5563 Acked Write 5 Done Done Sent false
16389 5564 Acked Read 4 Done Done Sent false
16390 5565 Acked Read 4 Done Done Sent false
16391 5566 Acked Write 3 Done Done Sent false
16392 5567 Acked Read 8 Done Done Sent false
16393 5568 Acked Read 2 Done Done Sent false
16394 5569 Acked Read 3 Done Done Sent false
16395 5570 Acked Flush 0 Done Done Sent false
16396 5571 Acked Write 3 Done Done Sent false
16397 5572 Acked Read 8 Done Done Sent false
16398 5573 Acked Read 6 Done Done Sent false
16399 5574 Acked Read 3 Done Done Sent false
16400 5575 Acked Write 8 Done Done Sent false
16401 5576 Acked Read 8 Done Done Sent false
16402 5577 Acked Read 2 Done Done Sent false
16403 5578 Acked Write 9 Done Done Sent false
16404 5579 Acked Write 4 Done Done Sent false
16405 5580 Acked Read 2 Done Done Sent false
16406 5581 Acked Write 3 Done Done Sent false
16407 5582 Acked Read 3 Done Done Sent false
16408 5583 Acked Read 1 Done Done Sent false
16409 5584 Acked Read 10 Done Done Sent false
16410 5585 Acked Read 10 Done Done Sent false
16411 5586 Acked Write 5 Done Done Sent false
16412 5587 Acked Read 5 Done Done Sent false
16413 5588 Acked Write 5 Done Done Sent false
16414 5589 Acked Write 1 Done Done Sent false
16415 5590 Acked Read 4 Done Done Sent false
16416 5591 Acked Read 4 Done Done Sent false
16417 5592 Acked Read 5 Done Done Sent false
16418 5593 Acked Write 1 Done Done Sent false
16419 5594 Acked Write 3 Done Done Sent false
16420 5595 Acked Write 3 Done Done Sent false
16421 5596 Acked Read 3 Done Done Sent false
16422 5597 Acked Read 3 Done Done Sent false
16423 5598 Acked Write 6 Done Done Sent false
16424 5599 Acked Read 10 Done Done Sent false
16425 5600 Acked Read 5 Done Done Sent false
16426 5601 Acked Write 8 Done Done Sent false
16427 5602 Acked Write 2 Done Done Sent false
16428 5603 Acked Write 1 Done Done Sent false
16429 5604 Acked Write 10 Done Done Sent false
16430 5605 Acked Read 10 Done Done Sent false
16431 5606 Acked Read 1 Done Done Sent false
16432 5607 Acked Write 7 Done Done Sent false
16433 5608 Acked Read 4 Done Done Sent false
16434 5609 Acked Write 1 Done Done Sent false
16435 5610 Acked Read 5 Done Done Sent false
16436 5611 Acked Write 3 Done Done Sent false
16437 5612 Acked Write 9 Done Done Sent false
16438 5613 Acked Read 8 Done Done Sent false
16439 5614 Acked Write 3 Done Done Sent false
16440 5615 Acked Write 5 Done Done Sent false
16441 5616 Acked Write 10 Done Done Sent false
16442 5617 Acked Write 4 Done Done Sent false
16443 5618 Acked Write 8 Done Done Sent false
16444 5619 Acked Write 7 Done Done Sent false
16445 5620 Acked Flush 0 Done Done Sent false
16446 5621 Acked Read 2 Done Done Sent false
16447 5622 Acked Read 6 Done Done Sent false
16448 5623 Acked Flush 0 Done Done Sent false
16449 5624 Acked Write 3 Done Done Sent false
16450 5625 Acked Write 8 Done Done Sent false
16451 5626 Acked Write 9 Done Done Sent false
16452 5627 Acked Write 7 Done Done Sent false
16453 5628 Acked Read 3 Done Done Sent false
16454 5629 Acked Read 6 Done Done Sent false
16455 5630 Acked Write 3 Done Done Sent false
16456 5631 Acked Write 8 Done Done Sent false
16457 5632 Acked Flush 0 Done Done Sent false
16458 5633 Acked Flush 0 Done Done Sent false
16459 5634 Acked Read 2 Done Done Sent false
16460 5635 Acked Read 10 Done Done Sent false
16461 5636 Acked Read 10 Done Done Sent false
16462 5637 Acked Write 6 Done Done Sent false
16463 5638 Acked Read 1 Done Done Sent false
16464 5639 Acked Read 4 Done Done Sent false
16465 5640 Acked Write 6 Done Done Sent false
16466 5641 Acked Flush 0 Done Done Sent false
16467 5642 Acked Read 7 Done Done Sent false
16468 5643 Acked Write 6 Done Done Sent false
16469 5644 Acked Read 1 Done Done Sent false
16470 5645 Acked Write 6 Done Done Sent false
16471 5646 Acked Write 6 Done Done Sent false
16472 5647 Acked Read 3 Done Done Sent false
16473 5648 Acked Write 2 Done Done Sent false
16474 5649 Acked Write 8 Done Done Sent false
16475 5650 Acked Read 1 Done Done Sent false
16476 5651 Acked Read 6 Done Done Sent false
16477 5652 Acked Write 6 Done Done Sent false
16478 5653 Acked Write 8 Done Done Sent false
16479 5654 Acked Flush 0 Done Done Sent false
16480 5655 Acked Write 2 Done Done Sent false
16481 5656 Acked Read 6 Done Done Sent false
16482 5657 Acked Read 6 Done Done Sent false
16483 5658 Acked Flush 0 Done Done Sent false
16484 5659 Acked Write 3 Done Done Sent false
16485 5660 Acked Write 5 Done Done Sent false
16486 5661 Acked Read 10 Done Done Sent false
16487 5662 Acked Flush 0 Done Done Sent false
16488 5663 Acked Flush 0 Done Done Sent false
16489 5664 Acked Write 4 Done Done Sent false
16490 5665 Acked Flush 0 Done Done Sent false
16491 5666 Acked Read 6 Done Done Sent false
16492 5667 Acked Write 3 Done Done Sent false
16493 5668 Acked Write 2 Done Done Sent false
16494 5669 Acked Write 4 Done Done Sent false
16495 5670 Acked Read 4 Done Done Sent false
16496 5671 Acked Read 7 Done Done Sent false
16497 5672 Acked Write 6 Done Done Sent false
16498 5673 Acked Read 6 Done Done Sent false
16499 5674 Acked Read 9 Done Done Sent false
16500 5675 Acked Write 6 Done Done Sent false
16501 5676 Acked Read 6 Done Done Sent false
16502 5677 Acked Read 7 Done Done Sent false
16503 5678 Acked Read 9 Done Done Sent false
16504 5679 Acked Write 5 Done Done Sent false
16505 5680 Acked Flush 0 Done Done Sent false
16506 5681 Acked Write 7 Done Done Sent false
16507 5682 Acked Flush 0 Done Done Sent false
16508 5683 Acked Flush 0 Done Done Sent false
16509 5684 Acked Read 5 Done Done Sent false
16510 5685 Acked Read 6 Done Done Sent false
16511 5686 Acked Read 2 Done Done Sent false
16512 5687 Acked Read 4 Done Done Sent false
16513 5688 Acked Read 7 Done Done Sent false
16514 5689 Acked Write 6 Done Done Sent false
16515 5690 Acked Write 1 Done Done Sent false
16516 5691 Acked Read 7 Done Done Sent false
16517 5692 Acked Write 4 Done Done Sent false
16518 5693 Acked Read 9 Done Done Sent false
16519 5694 Acked Flush 0 Done Done Sent false
16520 5695 Acked Write 5 Done Done Sent false
16521 5696 Acked Read 3 Done Done Sent false
16522 5697 Acked Write 7 Done Done Sent false
16523 5698 Acked Flush 0 Done Done Sent false
16524 5699 Acked Read 9 Done Done Sent false
16525 5700 Acked Read 5 Done Done Sent false
16526 5701 Acked Read 7 Done Done Sent false
16527 5702 Acked Read 8 Done Done Sent false
16528 5703 Acked Read 1 Done Done Sent false
16529 5704 Acked Write 9 Done Done Sent false
16530 5705 Acked Flush 0 Done Done Sent false
16531 5706 Acked Read 4 Done Done Sent false
16532 5707 Acked Read 9 Done Done Sent false
16533 5708 Acked Write 6 Done Done Sent false
16534 5709 Acked Read 6 Done Done Sent false
16535 5710 Acked Read 6 Done Done Sent false
16536 5711 Acked Read 6 Done Done Sent false
16537 5712 Acked Read 4 Done Done Sent false
16538 5713 Acked Flush 0 Done Done Sent false
16539 5714 Acked Write 1 Done Done Sent false
16540 5715 Acked Read 5 Done Done Sent false
16541 5716 Acked Read 2 Done Done Sent false
16542 5717 Acked Write 9 Done Done Sent false
16543 5718 Acked Read 1 Done Done Sent false
16544 5719 Acked Write 9 Done Done Sent false
16545 5720 Acked Flush 0 Done Done Sent false
16546 5721 Acked Write 5 Done Done Sent false
16547 5722 Acked Write 1 Done Done Sent false
16548 5723 Acked Read 10 Done Done Sent false
16549 5724 Acked Write 4 Done Done Sent false
16550 5725 Acked Read 4 Done Done Sent false
16551 5726 Acked Flush 0 Done Done Sent false
16552 5727 Acked Read 6 Done Done Sent false
16553 5728 Acked Read 5 Done Done Sent false
16554 5729 Acked Write 10 Done Done Sent false
16555 5730 Acked Read 6 Done Done Sent false
16556 5731 Acked Read 2 Done Done Sent false
16557 5732 Acked Read 5 Done Done Sent false
16558 5733 Acked Read 4 Done Done Sent false
16559 5734 Acked Write 4 Done Done Sent false
16560 5735 Acked Read 4 Done Done Sent false
16561 5736 Acked Read 9 Done Done Sent false
16562 5737 Acked Read 2 Done Done Sent false
16563 5738 Acked Read 6 Done Done Sent false
16564 5739 Acked Read 7 Done Done Sent false
16565 5740 Acked Read 7 Done Done Sent false
16566 5741 Acked Read 10 Done Done Sent false
16567 5742 Acked Write 7 Done Done Sent false
16568 5743 Acked Write 9 Done Done Sent false
16569 5744 Acked Read 6 Done Done Sent false
16570 5745 Acked Read 9 Done Done Sent false
16571 5746 Acked Read 1 Done Done Sent false
16572 5747 Acked Read 7 Done Done Sent false
16573 5748 Acked Read 4 Done Done Sent false
16574 5749 Acked Write 1 Done Done Sent false
16575 5750 Acked Read 8 Done Done Sent false
16576 5751 Acked Read 8 Done Done Sent false
16577 5752 Acked Write 4 Done Done Sent false
16578 5753 Acked Write 5 Done Done Sent false
16579 5754 Acked Read 8 Done Done Sent false
16580 5755 Acked Read 7 Done Done Sent false
16581 5756 Acked Write 9 Done Done Sent false
16582 5757 Acked Read 3 Done Done Sent false
16583 5758 Acked Read 5 Done Done Sent false
16584 5759 Acked Read 1 Done Done Sent false
16585 5760 Acked Read 9 Done Done Sent false
16586 5761 Acked Flush 0 Done Done Sent false
16587 5762 Acked Write 4 Done Done Sent false
16588 5763 Acked Flush 0 Done Done Sent false
16589 5764 Acked Write 2 Done Done Sent false
16590 5765 Acked Write 6 Done Done Sent false
16591 5766 Acked Flush 0 Done Done Sent false
16592 5767 Acked Read 5 Done Done Sent false
16593 5768 Acked Read 7 Done Done Sent false
16594 5769 Acked Write 1 Done Done Sent false
16595 5770 Acked Read 9 Done Done Sent false
16596 5771 Acked Write 10 Done Done Sent false
16597 5772 Acked Read 6 Done Done Sent false
16598 5773 Acked Write 9 Done Done Sent false
16599 5774 Acked Flush 0 Done Done Sent false
16600 5775 Acked Write 6 Done Done Sent false
16601 5776 Acked Flush 0 Done Done Sent false
16602 5777 Acked Read 2 Done Done Sent false
16603 5778 Acked Read 2 Done Done Sent false
16604 5779 Acked Flush 0 Done Done Sent false
16605 5780 Acked Write 7 Done Done Sent false
16606 5781 Acked Read 5 Done Done Sent false
16607 5782 Acked Write 4 Done Done Sent false
16608 5783 Acked Write 1 Done Done Sent false
16609 5784 Acked Read 10 Done Done Sent false
16610 5785 Acked Flush 0 Done Done Sent false
16611 5786 Acked Flush 0 Done Done Sent false
16612 5787 Acked Read 8 Done Done Sent false
16613 5788 Acked Read 4 Done Done Sent false
16614 5789 Acked Read 5 Done Done Sent false
16615 5790 Acked Read 3 Done Done Sent false
16616 5791 Acked Write 4 Done Done Sent false
16617 5792 Acked Read 4 Done Done Sent false
16618 5793 Acked Read 6 Done Done Sent false
16619 5794 Acked Write 1 Done Done Sent false
16620 5795 Acked Read 4 Done Done Sent false
16621 5796 Acked Read 9 Done Done Sent false
16622 5797 Acked Read 9 Done Done Sent false
16623 5798 Acked Read 3 Done Done Sent false
16624 5799 Acked Write 5 Done Done Sent false
16625 5800 Acked Read 2 Done Done Sent false
16626 5801 Acked Read 3 Done Done Sent false
16627 5802 Acked Write 10 Done Done Sent false
16628 5803 Acked Write 6 Done Done Sent false
16629 5804 Acked Write 8 Done Done Sent false
16630 5805 Acked Read 6 Done Done Sent false
16631 5806 Acked Read 7 Done Done Sent false
16632 5807 Acked Write 2 Done Done Sent false
16633 5808 Acked Read 4 Done Done Sent false
16634 5809 Acked Read 3 Done Done Sent false
16635 5810 Acked Write 3 Done Done Sent false
16636 5811 Acked Read 1 Done Done Sent false
16637 5812 Acked Write 9 Done Done Sent false
16638 5813 Acked Write 5 Done Done Sent false
16639 5814 Acked Read 2 Done Done Sent false
16640 5815 Acked Read 3 Done Done Sent false
16641 5816 Acked Write 9 Done Done Sent false
16642 5817 Acked Write 8 Done Done Sent false
16643 5818 Acked Read 4 Done Done Sent false
16644 5819 Acked Read 1 Done Done Sent false
16645 5820 Acked Write 9 Done Done Sent false
16646 5821 Acked Flush 0 Done Done Sent false
16647 5822 Acked Write 4 Done Done Sent false
16648 5823 Acked Write 5 Done Done Sent false
16649 5824 Acked Write 2 Done Done Sent false
16650 5825 Acked Read 4 Done Done Sent false
16651 5826 Acked Write 6 Done Done Sent false
16652 5827 Acked Read 6 Done Done Sent false
16653 5828 Acked Read 4 Done Done Sent false
16654 5829 Acked Flush 0 Done Done Sent false
16655 5830 Acked Read 4 Done Done Sent false
16656 STATES DS:0 DS:1 DS:2 TOTAL
16657 Sent 0 0 301 301
16658 Done 301 301 0 602
16659 Skipped 0 0 0 0
16660 Error 0 0 0 0
16661 Last Flush: Some(JobId(5829)) Some(JobId(5829)) Some(JobId(5529))
16662 Downstairs last five completed: 5529 5528 5527 5526 5525
16663 Upstairs last five completed: 5830 5829 5828 5827 5826
16664 CLIENT: Up:0 ds:301 act:2
16665 [8830] got start action
16666 [8830] I am going to restart
16667 Starting downstairs at port 8830
16668 Make output file at "/tmp/dsc/downstairs-8830.txt"
16669 [8830][2] reports Starting
16670 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1221)
16671 [8830][2] reports Running
16672 ----------------------------------------------------------------
16673 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
16674 States: Active Active Offline
16675 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16676 5530 Acked Read 2 Done Done Sent false
16677 5531 Acked Read 5 Done Done Sent false
16678 5532 Acked Write 10 Done Done Sent false
16679 5533 Acked Read 1 Done Done Sent false
16680 5534 Acked Write 8 Done Done Sent false
16681 5535 Acked Read 2 Done Done Sent false
16682 5536 Acked Read 8 Done Done Sent false
16683 5537 Acked Read 5 Done Done Sent false
16684 5538 Acked Read 5 Done Done Sent false
16685 5539 Acked Read 6 Done Done Sent false
16686 5540 Acked Read 8 Done Done Sent false
16687 5541 Acked Read 7 Done Done Sent false
16688 5542 Acked Write 2 Done Done Sent false
16689 5543 Acked Read 4 Done Done Sent false
16690 5544 Acked Read 8 Done Done Sent false
16691 5545 Acked Write 8 Done Done Sent false
16692 5546 Acked Write 3 Done Done Sent false
16693 5547 Acked Write 10 Done Done Sent false
16694 5548 Acked Read 2 Done Done Sent false
16695 5549 Acked Read 4 Done Done Sent false
16696 5550 Acked Read 9 Done Done Sent false
16697 5551 Acked Write 7 Done Done Sent false
16698 5552 Acked Write 5 Done Done Sent false
16699 5553 Acked Write 1 Done Done Sent false
16700 5554 Acked Read 1 Done Done Sent false
16701 5555 Acked Write 1 Done Done Sent false
16702 5556 Acked Write 6 Done Done Sent false
16703 5557 Acked Write 8 Done Done Sent false
16704 5558 Acked Write 4 Done Done Sent false
16705 5559 Acked Write 4 Done Done Sent false
16706 5560 Acked Write 6 Done Done Sent false
16707 5561 Acked Write 6 Done Done Sent false
16708 5562 Acked Write 7 Done Done Sent false
16709 5563 Acked Write 5 Done Done Sent false
16710 5564 Acked Read 4 Done Done Sent false
16711 5565 Acked Read 4 Done Done Sent false
16712 5566 Acked Write 3 Done Done Sent false
16713 5567 Acked Read 8 Done Done Sent false
16714 5568 Acked Read 2 Done Done Sent false
16715 5569 Acked Read 3 Done Done Sent false
16716 5570 Acked Flush 0 Done Done Sent false
16717 5571 Acked Write 3 Done Done Sent false
16718 5572 Acked Read 8 Done Done Sent false
16719 5573 Acked Read 6 Done Done Sent false
16720 5574 Acked Read 3 Done Done Sent false
16721 5575 Acked Write 8 Done Done Sent false
16722 5576 Acked Read 8 Done Done Sent false
16723 5577 Acked Read 2 Done Done Sent false
16724 5578 Acked Write 9 Done Done Sent false
16725 5579 Acked Write 4 Done Done Sent false
16726 5580 Acked Read 2 Done Done Sent false
16727 5581 Acked Write 3 Done Done Sent false
16728 5582 Acked Read 3 Done Done Sent false
16729 5583 Acked Read 1 Done Done Sent false
16730 5584 Acked Read 10 Done Done Sent false
16731 5585 Acked Read 10 Done Done Sent false
16732 5586 Acked Write 5 Done Done Sent false
16733 5587 Acked Read 5 Done Done Sent false
16734 5588 Acked Write 5 Done Done Sent false
16735 5589 Acked Write 1 Done Done Sent false
16736 5590 Acked Read 4 Done Done Sent false
16737 5591 Acked Read 4 Done Done Sent false
16738 5592 Acked Read 5 Done Done Sent false
16739 5593 Acked Write 1 Done Done Sent false
16740 5594 Acked Write 3 Done Done Sent false
16741 5595 Acked Write 3 Done Done Sent false
16742 5596 Acked Read 3 Done Done Sent false
16743 5597 Acked Read 3 Done Done Sent false
16744 5598 Acked Write 6 Done Done Sent false
16745 5599 Acked Read 10 Done Done Sent false
16746 5600 Acked Read 5 Done Done Sent false
16747 5601 Acked Write 8 Done Done Sent false
16748 5602 Acked Write 2 Done Done Sent false
16749 5603 Acked Write 1 Done Done Sent false
16750 5604 Acked Write 10 Done Done Sent false
16751 5605 Acked Read 10 Done Done Sent false
16752 5606 Acked Read 1 Done Done Sent false
16753 5607 Acked Write 7 Done Done Sent false
16754 5608 Acked Read 4 Done Done Sent false
16755 5609 Acked Write 1 Done Done Sent false
16756 5610 Acked Read 5 Done Done Sent false
16757 5611 Acked Write 3 Done Done Sent false
16758 5612 Acked Write 9 Done Done Sent false
16759 5613 Acked Read 8 Done Done Sent false
16760 5614 Acked Write 3 Done Done Sent false
16761 5615 Acked Write 5 Done Done Sent false
16762 5616 Acked Write 10 Done Done Sent false
16763 5617 Acked Write 4 Done Done Sent false
16764 5618 Acked Write 8 Done Done Sent false
16765 5619 Acked Write 7 Done Done Sent false
16766 5620 Acked Flush 0 Done Done Sent false
16767 5621 Acked Read 2 Done Done Sent false
16768 5622 Acked Read 6 Done Done Sent false
16769 5623 Acked Flush 0 Done Done Sent false
16770 5624 Acked Write 3 Done Done Sent false
16771 5625 Acked Write 8 Done Done Sent false
16772 5626 Acked Write 9 Done Done Sent false
16773 5627 Acked Write 7 Done Done Sent false
16774 5628 Acked Read 3 Done Done Sent false
16775 5629 Acked Read 6 Done Done Sent false
16776 5630 Acked Write 3 Done Done Sent false
16777 5631 Acked Write 8 Done Done Sent false
16778 5632 Acked Flush 0 Done Done Sent false
16779 5633 Acked Flush 0 Done Done Sent false
16780 5634 Acked Read 2 Done Done Sent false
16781 5635 Acked Read 10 Done Done Sent false
16782 5636 Acked Read 10 Done Done Sent false
16783 5637 Acked Write 6 Done Done Sent false
16784 5638 Acked Read 1 Done Done Sent false
16785 5639 Acked Read 4 Done Done Sent false
16786 5640 Acked Write 6 Done Done Sent false
16787 5641 Acked Flush 0 Done Done Sent false
16788 5642 Acked Read 7 Done Done Sent false
16789 5643 Acked Write 6 Done Done Sent false
16790 5644 Acked Read 1 Done Done Sent false
16791 5645 Acked Write 6 Done Done Sent false
16792 5646 Acked Write 6 Done Done Sent false
16793 5647 Acked Read 3 Done Done Sent false
16794 5648 Acked Write 2 Done Done Sent false
16795 5649 Acked Write 8 Done Done Sent false
16796 5650 Acked Read 1 Done Done Sent false
16797 5651 Acked Read 6 Done Done Sent false
16798 5652 Acked Write 6 Done Done Sent false
16799 5653 Acked Write 8 Done Done Sent false
16800 5654 Acked Flush 0 Done Done Sent false
16801 5655 Acked Write 2 Done Done Sent false
16802 5656 Acked Read 6 Done Done Sent false
16803 5657 Acked Read 6 Done Done Sent false
16804 5658 Acked Flush 0 Done Done Sent false
16805 5659 Acked Write 3 Done Done Sent false
16806 5660 Acked Write 5 Done Done Sent false
16807 5661 Acked Read 10 Done Done Sent false
16808 5662 Acked Flush 0 Done Done Sent false
16809 5663 Acked Flush 0 Done Done Sent false
16810 5664 Acked Write 4 Done Done Sent false
16811 5665 Acked Flush 0 Done Done Sent false
16812 5666 Acked Read 6 Done Done Sent false
16813 5667 Acked Write 3 Done Done Sent false
16814 5668 Acked Write 2 Done Done Sent false
16815 5669 Acked Write 4 Done Done Sent false
16816 5670 Acked Read 4 Done Done Sent false
16817 5671 Acked Read 7 Done Done Sent false
16818 5672 Acked Write 6 Done Done Sent false
16819 5673 Acked Read 6 Done Done Sent false
16820 5674 Acked Read 9 Done Done Sent false
16821 5675 Acked Write 6 Done Done Sent false
16822 5676 Acked Read 6 Done Done Sent false
16823 5677 Acked Read 7 Done Done Sent false
16824 5678 Acked Read 9 Done Done Sent false
16825 5679 Acked Write 5 Done Done Sent false
16826 5680 Acked Flush 0 Done Done Sent false
16827 5681 Acked Write 7 Done Done Sent false
16828 5682 Acked Flush 0 Done Done Sent false
16829 5683 Acked Flush 0 Done Done Sent false
16830 5684 Acked Read 5 Done Done Sent false
16831 5685 Acked Read 6 Done Done Sent false
16832 5686 Acked Read 2 Done Done Sent false
16833 5687 Acked Read 4 Done Done Sent false
16834 5688 Acked Read 7 Done Done Sent false
16835 5689 Acked Write 6 Done Done Sent false
16836 5690 Acked Write 1 Done Done Sent false
16837 5691 Acked Read 7 Done Done Sent false
16838 5692 Acked Write 4 Done Done Sent false
16839 5693 Acked Read 9 Done Done Sent false
16840 5694 Acked Flush 0 Done Done Sent false
16841 5695 Acked Write 5 Done Done Sent false
16842 5696 Acked Read 3 Done Done Sent false
16843 5697 Acked Write 7 Done Done Sent false
16844 5698 Acked Flush 0 Done Done Sent false
16845 5699 Acked Read 9 Done Done Sent false
16846 5700 Acked Read 5 Done Done Sent false
16847 5701 Acked Read 7 Done Done Sent false
16848 5702 Acked Read 8 Done Done Sent false
16849 5703 Acked Read 1 Done Done Sent false
16850 5704 Acked Write 9 Done Done Sent false
16851 5705 Acked Flush 0 Done Done Sent false
16852 5706 Acked Read 4 Done Done Sent false
16853 5707 Acked Read 9 Done Done Sent false
16854 5708 Acked Write 6 Done Done Sent false
16855 5709 Acked Read 6 Done Done Sent false
16856 5710 Acked Read 6 Done Done Sent false
16857 5711 Acked Read 6 Done Done Sent false
16858 5712 Acked Read 4 Done Done Sent false
16859 5713 Acked Flush 0 Done Done Sent false
16860 5714 Acked Write 1 Done Done Sent false
16861 5715 Acked Read 5 Done Done Sent false
16862 5716 Acked Read 2 Done Done Sent false
16863 5717 Acked Write 9 Done Done Sent false
16864 5718 Acked Read 1 Done Done Sent false
16865 5719 Acked Write 9 Done Done Sent false
16866 5720 Acked Flush 0 Done Done Sent false
16867 5721 Acked Write 5 Done Done Sent false
16868 5722 Acked Write 1 Done Done Sent false
16869 5723 Acked Read 10 Done Done Sent false
16870 5724 Acked Write 4 Done Done Sent false
16871 5725 Acked Read 4 Done Done Sent false
16872 5726 Acked Flush 0 Done Done Sent false
16873 5727 Acked Read 6 Done Done Sent false
16874 5728 Acked Read 5 Done Done Sent false
16875 5729 Acked Write 10 Done Done Sent false
16876 5730 Acked Read 6 Done Done Sent false
16877 5731 Acked Read 2 Done Done Sent false
16878 5732 Acked Read 5 Done Done Sent false
16879 5733 Acked Read 4 Done Done Sent false
16880 5734 Acked Write 4 Done Done Sent false
16881 5735 Acked Read 4 Done Done Sent false
16882 5736 Acked Read 9 Done Done Sent false
16883 5737 Acked Read 2 Done Done Sent false
16884 5738 Acked Read 6 Done Done Sent false
16885 5739 Acked Read 7 Done Done Sent false
16886 5740 Acked Read 7 Done Done Sent false
16887 5741 Acked Read 10 Done Done Sent false
16888 5742 Acked Write 7 Done Done Sent false
16889 5743 Acked Write 9 Done Done Sent false
16890 5744 Acked Read 6 Done Done Sent false
16891 5745 Acked Read 9 Done Done Sent false
16892 5746 Acked Read 1 Done Done Sent false
16893 5747 Acked Read 7 Done Done Sent false
16894 5748 Acked Read 4 Done Done Sent false
16895 5749 Acked Write 1 Done Done Sent false
16896 5750 Acked Read 8 Done Done Sent false
16897 5751 Acked Read 8 Done Done Sent false
16898 5752 Acked Write 4 Done Done Sent false
16899 5753 Acked Write 5 Done Done Sent false
16900 5754 Acked Read 8 Done Done Sent false
16901 5755 Acked Read 7 Done Done Sent false
16902 5756 Acked Write 9 Done Done Sent false
16903 5757 Acked Read 3 Done Done Sent false
16904 5758 Acked Read 5 Done Done Sent false
16905 5759 Acked Read 1 Done Done Sent false
16906 5760 Acked Read 9 Done Done Sent false
16907 5761 Acked Flush 0 Done Done Sent false
16908 5762 Acked Write 4 Done Done Sent false
16909 5763 Acked Flush 0 Done Done Sent false
16910 5764 Acked Write 2 Done Done Sent false
16911 5765 Acked Write 6 Done Done Sent false
16912 5766 Acked Flush 0 Done Done Sent false
16913 5767 Acked Read 5 Done Done Sent false
16914 5768 Acked Read 7 Done Done Sent false
16915 5769 Acked Write 1 Done Done Sent false
16916 5770 Acked Read 9 Done Done Sent false
16917 5771 Acked Write 10 Done Done Sent false
16918 5772 Acked Read 6 Done Done Sent false
16919 5773 Acked Write 9 Done Done Sent false
16920 5774 Acked Flush 0 Done Done Sent false
16921 5775 Acked Write 6 Done Done Sent false
16922 5776 Acked Flush 0 Done Done Sent false
16923 5777 Acked Read 2 Done Done Sent false
16924 5778 Acked Read 2 Done Done Sent false
16925 5779 Acked Flush 0 Done Done Sent false
16926 5780 Acked Write 7 Done Done Sent false
16927 5781 Acked Read 5 Done Done Sent false
16928 5782 Acked Write 4 Done Done Sent false
16929 5783 Acked Write 1 Done Done Sent false
16930 5784 Acked Read 10 Done Done Sent false
16931 5785 Acked Flush 0 Done Done Sent false
16932 5786 Acked Flush 0 Done Done Sent false
16933 5787 Acked Read 8 Done Done Sent false
16934 5788 Acked Read 4 Done Done Sent false
16935 5789 Acked Read 5 Done Done Sent false
16936 5790 Acked Read 3 Done Done Sent false
16937 5791 Acked Write 4 Done Done Sent false
16938 5792 Acked Read 4 Done Done Sent false
16939 5793 Acked Read 6 Done Done Sent false
16940 5794 Acked Write 1 Done Done Sent false
16941 5795 Acked Read 4 Done Done Sent false
16942 5796 Acked Read 9 Done Done Sent false
16943 5797 Acked Read 9 Done Done Sent false
16944 5798 Acked Read 3 Done Done Sent false
16945 5799 Acked Write 5 Done Done Sent false
16946 5800 Acked Read 2 Done Done Sent false
16947 5801 Acked Read 3 Done Done Sent false
16948 5802 Acked Write 10 Done Done Sent false
16949 5803 Acked Write 6 Done Done Sent false
16950 5804 Acked Write 8 Done Done Sent false
16951 5805 Acked Read 6 Done Done Sent false
16952 5806 Acked Read 7 Done Done Sent false
16953 5807 Acked Write 2 Done Done Sent false
16954 5808 Acked Read 4 Done Done Sent false
16955 5809 Acked Read 3 Done Done Sent false
16956 5810 Acked Write 3 Done Done Sent false
16957 5811 Acked Read 1 Done Done Sent false
16958 5812 Acked Write 9 Done Done Sent false
16959 5813 Acked Write 5 Done Done Sent false
16960 5814 Acked Read 2 Done Done Sent false
16961 5815 Acked Read 3 Done Done Sent false
16962 5816 Acked Write 9 Done Done Sent false
16963 5817 Acked Write 8 Done Done Sent false
16964 5818 Acked Read 4 Done Done Sent false
16965 5819 Acked Read 1 Done Done Sent false
16966 5820 Acked Write 9 Done Done Sent false
16967 5821 Acked Flush 0 Done Done Sent false
16968 5822 Acked Write 4 Done Done Sent false
16969 5823 Acked Write 5 Done Done Sent false
16970 5824 Acked Write 2 Done Done Sent false
16971 5825 Acked Read 4 Done Done Sent false
16972 5826 Acked Write 6 Done Done Sent false
16973 5827 Acked Read 6 Done Done Sent false
16974 5828 Acked Read 4 Done Done Sent false
16975 5829 Acked Flush 0 Done Done Sent false
16976 5830 Acked Read 4 Done Done Sent false
16977 5831 Acked Flush 0 Done Done Sent false
16978 STATES DS:0 DS:1 DS:2 TOTAL
16979 Sent 0 0 302 302
16980 Done 302 302 0 604
16981 Skipped 0 0 0 0
16982 Error 0 0 0 0
16983 Last Flush: Some(JobId(5831)) Some(JobId(5831)) Some(JobId(5529))
16984 Downstairs last five completed: 5529 5528 5527 5526 5525
16985 Upstairs last five completed: 5831 5830 5829 5828 5827
16986 CLIENT: Up:0 ds:302 act:2
169872025-01-23T17:41:17.391ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
169882025-01-23T17:41:17.391ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39758) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
169892025-01-23T17:41:17.392ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
169902025-01-23T17:41:17.393ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
169912025-01-23T17:41:17.393ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
169922025-01-23T17:41:17.393ZINFOcrucible: send last flush ID to this DS: Some(JobId(5529)) = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
169932025-01-23T17:41:17.393ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
169942025-01-23T17:41:17.393ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(5529)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
169952025-01-23T17:41:17.393ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(5529)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
16996 ----------------------------------------------------------------
16997 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16998 Downstairs last five completed: 5831 5830 5829 5828 5827
16999 Upstairs last five completed: 5831 5830 5829 5828 5827
17000 CLIENT: Up:0 ds:0 act:3
17001 Replay: All jobs finished
17002 Main task has work to do, go find it
17003 got dsc Stop(0)
17004 stop 0
17005 [8810] Got stop action so:false kr:false
17006 Jan 23 17:41:19.639 INFO request completed, latency_us: 356, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 0c5914be-d10b-4833-9cba-59292aea053a, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
17007 Jan 23 17:41:19.640 INFO request completed, latency_us: 115, response_code: 200, uri: /state/cid/0, method: GET, req_id: 5970a17b-cba1-4e9c-9134-339be8711a16, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
170082025-01-23T17:41:19.640ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
170092025-01-23T17:41:19.640ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
170102025-01-23T17:41:19.640ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
170112025-01-23T17:41:19.640ZINFOcrucible: client task is exiting = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
17012 [8810] Exited with: ExitStatus(unix_wait_status(9))
17013 [8810][0] reports Exit
17014 Jan 23 17:41:23.642 INFO request completed, latency_us: 248, response_code: 200, uri: /state/cid/0, method: GET, req_id: 0d392ac9-d336-4662-9e0b-11e7854619f0, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
17015 001/300 Read block 2917 len 16384
17016 002/300 Read block 4782 len 16384
17017 003/300 Read block 2328 len 24576
17018 004/300 Flush
17019 005/300 Read block 4522 len 32768
17020 006/300 Write block 3484 len 12288 data: 2 2 2
17021 007/300 Write block 328 len 28672 data: 5 6 5 6 7 6 4
17022 008/300 Write block 131 len 36864 data: 4 3 3 3 3 4 4 4 6
17023 009/300 Write block 648 len 4096 data: 3
17024 010/300 Write block 4697 len 16384 data: 4 4 4 5
17025 011/300 Write block 3383 len 24576 data: 4 3 4 3 3 3
17026 012/300 Read block 657 len 24576
17027 013/300 Read block 487 len 40960
17028 014/300 Write block 2901 len 20480 data: 4 4 4 4 4
17029 015/300 Write block 1970 len 12288 data: 6 5 5
17030 016/300 Write block 404 len 20480 data: 3 3 4 4 3
17031 017/300 Write block 2812 len 16384 data: 4 4 4 5
17032 018/300 Write block 2407 len 40960 data: 2 3 3 3 3 4 7 6 6 6
17033 019/300 Read block 530 len 8192
17034 020/300 Read block 4022 len 32768
17035 021/300 Read block 1683 len 36864
17036 022/300 Read block 363 len 32768
17037 023/300 Read block 4720 len 32768
17038 024/300 Write block 1418 len 24576 data: 4 5 6 5 4 5
17039 025/300 Read block 2882 len 4096
17040 026/300 Read block 2089 len 28672
17041 027/300 Read block 1117 len 20480
17042 028/300 Write block 452 len 20480 data: 2 2 2 3 3
17043 029/300 Write block 4817 len 28672 data: 4 4 3 3 4 3 3
17044 030/300 Write block 1801 len 24576 data: 6 5 4 4 4 2
17045 031/300 Flush
17046 032/300 Write block 3069 len 28672 data: 5 3 2 2 2 2 2
17047 033/300 Flush
17048 034/300 Write block 2023 len 4096 data: 4
17049 035/300 Write block 250 len 12288 data: 3 3 5
17050 036/300 Flush
17051 037/300 Write block 4578 len 20480 data: 6 5 4 4 4
17052 038/300 Write block 548 len 8192 data: 5 5
17053 039/300 Write block 1402 len 20480 data: 4 4 3 2 2
17054 040/300 Read block 1494 len 36864
17055 041/300 Read block 4747 len 16384
17056 042/300 Write block 3954 len 20480 data: 4 5 5 5 4
17057 043/300 Write block 3197 len 16384 data: 3 4 4 4
17058 044/300 Read block 2979 len 20480
17059 045/300 Flush
17060 046/300 Read block 4697 len 32768
17061 047/300 Read block 2559 len 24576
17062 048/300 Write block 1956 len 12288 data: 4 4 3
17063 049/300 Read block 3661 len 12288
17064 050/300 Write block 1859 len 32768 data: 3 3 5 7 6 6 8 10
17065 051/300 Read block 118 len 12288
17066 052/300 Read block 2358 len 28672
17067 053/300 Read block 3618 len 24576
17068 054/300 Read block 3635 len 12288
17069 055/300 Write block 4557 len 12288 data: 6 7 4
17070 056/300 Read block 549 len 28672
17071 057/300 Read block 87 len 36864
17072 058/300 Write block 649 len 36864 data: 2 2 3 3 3 4 4 4 5
17073 059/300 Read block 4828 len 24576
17074 060/300 Write block 1316 len 8192 data: 5 5
17075 061/300 Write block 3754 len 4096 data: 4
17076 062/300 Write block 1730 len 20480 data: 4 3 3 4 6
17077 063/300 Write block 3763 len 40960 data: 3 4 4 4 2 2 2 2 2 3
17078 064/300 Read block 3956 len 24576
17079 065/300 Read block 131 len 24576
17080 066/300 Read block 3575 len 28672
17081 067/300 Write block 1363 len 12288 data: 3 4 4
17082 068/300 Write block 706 len 12288 data: 3 3 3
17083 069/300 Read block 717 len 24576
17084 070/300 Write block 2112 len 36864 data: 6 5 5 5 5 5 4 3 3
17085 071/300 Write block 1215 len 40960 data: 3 2 2 3 4 4 5 5 4 3
17086 072/300 Write block 1845 len 4096 data: 3
17087 073/300 Flush
17088 074/300 Write block 426 len 8192 data: 2 2
17089 075/300 Write block 3699 len 12288 data: 3 4 5
17090 076/300 Write block 1771 len 40960 data: 4 4 4 4 4 5 4 4 4 3
17091 077/300 Flush
17092 078/300 Read block 2392 len 32768
17093 079/300 Read block 4555 len 40960
17094 080/300 Read block 3335 len 40960
17095 081/300 Read block 2552 len 36864
17096 082/300 Write block 71 len 24576 data: 8 8 8 9 7 6
17097 083/300 Read block 3411 len 20480
17098 084/300 Write block 1231 len 20480 data: 2 2 3 3 4
17099 085/300 Write block 811 len 24576 data: 5 7 6 5 4 4
17100 086/300 Write block 2283 len 40960 data: 7 6 5 6 6 5 4 5 4 4
17101 087/300 Read block 4267 len 12288
17102 088/300 Write block 753 len 24576 data: 6 5 4 5 4 4
17103 089/300 Write block 2386 len 4096 data: 6
17104 090/300 Write block 1520 len 28672 data: 2 2 2 2 2 2 2
17105 091/300 Read block 2726 len 20480
17106 092/300 Write block 278 len 32768 data: 4 6 5 3 4 3 3 4
17107 093/300 Read block 3228 len 32768
17108 094/300 Read block 4515 len 8192
17109 095/300 Write block 1612 len 40960 data: 5 5 4 5 6 6 7 9 8 7
17110 096/300 Write block 2502 len 20480 data: 4 4 3 3 3
17111 097/300 Read block 1071 len 40960
17112 098/300 Read block 4 len 4096
17113 099/300 Read block 4540 len 28672
17114 100/300 Write block 4972 len 24576 data: 8 5 4 4 4 3
17115 101/300 Read block 1442 len 28672
17116 102/300 Read block 408 len 20480
17117 103/300 Read block 1919 len 36864
17118 104/300 Flush
17119 105/300 Write block 4910 len 24576 data: 4 4 4 4 5 4
17120 106/300 Read block 2579 len 36864
17121 107/300 Read block 3096 len 16384
17122 108/300 Write block 1365 len 12288 data: 5 3 3
17123 109/300 Flush
17124 110/300 Read block 1528 len 32768
17125 111/300 Write block 3766 len 28672 data: 5 3 3 3 3 3 4
17126 112/300 Read block 954 len 20480
17127 113/300 Read block 3173 len 24576
17128 114/300 Read block 2555 len 20480
17129 115/300 Write block 2530 len 8192 data: 5 5
17130 116/300 Read block 2177 len 4096
17131 117/300 Flush
17132 118/300 Read block 1450 len 16384
17133 119/300 Read block 3593 len 20480
17134 120/300 Read block 4861 len 40960
17135 121/300 Flush
17136 122/300 Write block 3835 len 8192 data: 5 6
17137 123/300 Write block 2618 len 40960 data: 6 5 6 6 6 6 6 5 5 4
17138 124/300 Flush
17139 125/300 Read block 1204 len 4096
17140 126/300 Flush
17141 127/300 Write block 4444 len 12288 data: 4 5 5
17142 128/300 Read block 156 len 32768
17143 129/300 Write block 4683 len 16384 data: 7 5 7 6
17144 130/300 Flush
17145 131/300 Read block 2963 len 20480
17146 132/300 Read block 3720 len 28672
17147 133/300 Read block 3757 len 28672
17148 134/300 Write block 4534 len 24576 data: 4 4 3 4 4 4
17149 135/300 Read block 3320 len 12288
17150 136/300 Read block 2275 len 28672
17151 137/300 Read block 4624 len 28672
17152 138/300 Write block 4936 len 20480 data: 6 6 6 4 4
17153 139/300 Read block 200 len 20480
17154 140/300 Write block 3546 len 16384 data: 6 6 6 6
17155 141/300 Read block 4936 len 24576
17156 142/300 Read block 2666 len 40960
17157 143/300 Read block 1837 len 12288
17158 144/300 Read block 3792 len 16384
17159 145/300 Read block 3047 len 36864
17160 146/300 Write block 2083 len 12288 data: 4 3 2
17161 147/300 Read block 3234 len 36864
17162 148/300 Write block 3481 len 16384 data: 3 3 3 3
17163 149/300 Read block 4664 len 4096
17164 150/300 Write block 2499 len 28672 data: 5 3 4 5 5 4 4
17165 151/300 Read block 3077 len 28672
17166 152/300 Read block 2542 len 16384
17167 153/300 Write block 2987 len 24576 data: 4 4 3 4 5 6
17168 154/300 Read block 3879 len 16384
17169 155/300 Write block 4908 len 24576 data: 3 4 5 5 5 5
17170 156/300 Read block 2792 len 8192
17171 157/300 Write block 3166 len 4096 data: 4
17172 158/300 Write block 1318 len 16384 data: 4 4 5 5
17173 159/300 Write block 2178 len 20480 data: 3 3 4 6 5
17174 160/300 Read block 2584 len 36864
17175 161/300 Read block 1703 len 28672
17176 162/300 Read block 3035 len 16384
17177 163/300 Read block 885 len 36864
17178 164/300 Read block 2221 len 24576
17179 165/300 Write block 1953 len 28672 data: 5 4 5 5 5 4 3
17180 166/300 Read block 1150 len 20480
17181 167/300 Read block 2560 len 36864
17182 168/300 Read block 4422 len 40960
17183 169/300 Read block 425 len 24576
17184 170/300 Read block 587 len 20480
17185 171/300 Read block 2585 len 32768
17186 172/300 Read block 182 len 20480
17187 173/300 Write block 1530 len 36864 data: 3 4 5 6 5 5 5 5 4
17188 174/300 Write block 1069 len 24576 data: 5 5 6 5 4 4
17189 175/300 Write block 62 len 16384 data: 4 4 5 4
17190 176/300 Read block 1909 len 32768
17191 177/300 Write block 1129 len 12288 data: 2 2 2
17192 178/300 Write block 4329 len 28672 data: 3 3 3 4 3 4 5
17193 179/300 Write block 2765 len 4096 data: 4
17194 180/300 Read block 4258 len 20480
17195 181/300 Flush
17196 182/300 Write block 2746 len 20480 data: 7 8 9 8 9
17197 183/300 Write block 1271 len 16384 data: 3 5 6 6
17198 184/300 Write block 1510 len 36864 data: 7 5 5 4 4 4 3 3 3
17199 185/300 Write block 2997 len 12288 data: 5 4 5
17200 186/300 Read block 393 len 28672
17201 187/300 Write block 1965 len 8192 data: 7 6
17202 188/300 Flush
17203 189/300 Write block 583 len 12288 data: 6 9 10
17204 190/300 Read block 4635 len 8192
17205 191/300 Write block 4212 len 24576 data: 4 3 3 3 3 3
17206 192/300 Read block 3549 len 36864
17207 193/300 Read block 149 len 28672
17208 194/300 Read block 4891 len 40960
17209 195/300 Read block 2047 len 4096
17210 196/300 Read block 1862 len 32768
17211 197/300 Write block 3151 len 40960 data: 4 3 3 3 3 3 5 6 6 6
17212 198/300 Read block 4960 len 4096
17213 199/300 Flush
17214 200/300 Read block 4448 len 32768
17215 201/300 Read block 458 len 36864
17216 202/300 Write block 1802 len 24576 data: 6 5 5 5 3 2
17217 203/300 Write block 4947 len 16384 data: 5 4 5 5
17218 204/300 Write block 327 len 20480 data: 2 6 7 6 7
17219 205/300 Write block 3678 len 4096 data: 5
17220 206/300 Flush
17221 207/300 Flush
17222 208/300 Flush
17223 209/300 Read block 3392 len 8192
17224 210/300 Read block 3448 len 24576
17225 211/300 Read block 4912 len 32768
17226 212/300 Flush
17227 213/300 Read block 2611 len 20480
17228 214/300 Read block 1895 len 36864
17229 215/300 Write block 2727 len 20480 data: 4 4 4 4 4
17230 216/300 Write block 615 len 28672 data: 4 5 4 4 5 4 3
17231 217/300 Write block 3776 len 8192 data: 4 3
17232 218/300 Read block 1930 len 24576
17233 219/300 Write block 3698 len 32768 data: 3 4 5 6 6 7 7 6
17234 220/300 Write block 1455 len 40960 data: 2 2 2 2 2 2 2 3 3 3
17235 221/300 Read block 4109 len 24576
17236 222/300 Read block 854 len 4096
17237 223/300 Flush
17238 224/300 Read block 962 len 20480
17239 225/300 Write block 4299 len 28672 data: 2 2 2 2 3 3 3
17240 226/300 Read block 383 len 36864
17241 227/300 Read block 1548 len 32768
17242 228/300 Write block 2886 len 32768 data: 3 4 4 5 3 3 3 3
17243 229/300 Write block 4454 len 20480 data: 3 4 5 4 5
17244 230/300 Read block 905 len 20480
17245 231/300 Write block 2761 len 8192 data: 3 3
17246 232/300 Write block 3542 len 8192 data: 5 4
17247 233/300 Read block 2312 len 36864
17248 234/300 Write block 3651 len 16384 data: 5 6 7 6
17249 235/300 Read block 290 len 16384
17250 236/300 Read block 2353 len 40960
17251 237/300 Write block 576 len 32768 data: 2 2 3 3 3 4 5 7
17252 238/300 Read block 68 len 28672
17253 239/300 Flush
17254 240/300 Read block 2056 len 36864
17255 241/300 Write block 684 len 12288 data: 3 2 2
17256 242/300 Write block 3963 len 40960 data: 6 6 6 5 4 5 5 4 4 3
17257 243/300 Write block 922 len 24576 data: 4 4 4 3 3 2
17258 244/300 Write block 3588 len 24576 data: 4 5 6 4 5 6
17259 245/300 Write block 3912 len 24576 data: 6 6 6 5 4 4
17260 246/300 Read block 1210 len 40960
17261 247/300 Write block 1065 len 16384 data: 4 4 4 5
17262 248/300 Read block 2632 len 8192
17263 249/300 Write block 4928 len 36864 data: 5 4 3 5 5 5 5 5 7
17264 250/300 Read block 2131 len 36864
17265 251/300 Write block 2010 len 12288 data: 5 6 6
17266 252/300 Read block 122 len 4096
17267 253/300 Read block 774 len 28672
17268 254/300 Read block 2740 len 32768
17269 255/300 Flush
17270 256/300 Write block 4961 len 20480 data: 2 2 2 2 2
17271 257/300 Flush
17272 258/300 Read block 1365 len 32768
17273 259/300 Write block 2729 len 32768 data: 5 5 5 4 4 3 4 3
17274 260/300 Write block 163 len 32768 data: 5 4 6 5 5 4 5 7
17275 261/300 Read block 3252 len 40960
17276 262/300 Read block 2256 len 4096
17277 263/300 Flush
17278 264/300 Write block 3316 len 4096 data: 5
17279 265/300 Read block 564 len 28672
17280 266/300 Read block 2835 len 20480
17281 267/300 Write block 3099 len 40960 data: 5 5 5 5 5 5 5 6 5 6
17282 268/300 Flush
17283 269/300 Write block 2396 len 40960 data: 8 9 10 9 9 6 5 5 4 4
17284 270/300 Read block 3959 len 24576
17285 271/300 Flush
17286 272/300 Write block 4226 len 32768 data: 3 3 3 3 4 3 4 5
17287 273/300 Read block 919 len 12288
17288 274/300 Write block 2384 len 8192 data: 8 7
17289 275/300 Read block 1688 len 36864
17290 276/300 Read block 4594 len 32768
17291 277/300 Flush
17292 278/300 Read block 3073 len 4096
17293 279/300 Read block 2695 len 12288
17294 280/300 Read block 1703 len 12288
17295 281/300 Read block 923 len 16384
17296 282/300 Flush
17297 283/300 Write block 1199 len 8192 data: 4 5
17298 284/300 Read block 1102 len 4096
17299 285/300 Write block 3682 len 16384 data: 4 4 4 6
17300 286/300 Write block 793 len 20480 data: 2 2 2 3 5
17301 287/300 Flush
17302 288/300 Read block 3616 len 12288
17303 289/300 Read block 161 len 8192
17304 290/300 Read block 4979 len 40960
17305 291/300 Write block 3810 len 24576 data: 4 6 6 5 6 4
17306 292/300 Read block 2407 len 12288
17307 293/300 Write block 294 len 16384 data: 7 8 6 5
17308 294/300 Write block 2582 len 20480 data: 2 2 2 2 2
17309 295/300 Read block 4071 len 16384
17310 296/300 Read block 1874 len 24576
17311 297/300 Read block 4236 len 36864
17312 298/300 Read block 2733 len 36864
17313 299/300 Write block 3048 len 24576 data: 5 4 4 3 3 3
17314 300/300 Write block 50 len 40960 data: 4 3 4 5 5 5 4 5 6 6
17315 301/300 Read block 2796 len 24576
17316 Main task has work to do, go find it
17317 got dsc Start(0)
17318 start 0
17319 Jan 23 17:41:23.890 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/0, method: POST, req_id: 96a39e03-8f1f-427f-96b6-59ca360e9579, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
17320 Replay: started 0, returned:Ok(())
17321 ----------------------------------------------------------------
17322 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17323 States: Offline Active Active
17324 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17325 5832 Acked Read 4 Sent Done Done false
17326 5833 Acked Read 4 Sent Done Done false
17327 5834 Acked Read 6 Sent Done Done false
17328 5835 Acked Flush 0 Sent Done Done false
17329 5836 Acked Read 8 Sent Done Done false
17330 5837 Acked Write 3 Sent Done Done false
17331 5838 Acked Write 7 Sent Done Done false
17332 5839 Acked Write 9 Sent Done Done false
17333 5840 Acked Write 1 Sent Done Done false
17334 5841 Acked Write 4 Sent Done Done false
17335 5842 Acked Write 6 Sent Done Done false
17336 5843 Acked Read 6 Sent Done Done false
17337 5844 Acked Read 10 Sent Done Done false
17338 5845 Acked Write 5 Sent Done Done false
17339 5846 Acked Write 3 Sent Done Done false
17340 5847 Acked Write 5 Sent Done Done false
17341 5848 Acked Write 4 Sent Done Done false
17342 5849 Acked Write 10 Sent Done Done false
17343 5850 Acked Read 2 Sent Done Done false
17344 5851 Acked Read 8 Sent Done Done false
17345 5852 Acked Read 9 Sent Done Done false
17346 5853 Acked Read 8 Sent Done Done false
17347 5854 Acked Read 8 Sent Done Done false
17348 5855 Acked Write 6 Sent Done Done false
17349 5856 Acked Read 1 Sent Done Done false
17350 5857 Acked Read 7 Sent Done Done false
17351 5858 Acked Read 5 Sent Done Done false
17352 5859 Acked Write 5 Sent Done Done false
17353 5860 Acked Write 7 Sent Done Done false
17354 5861 Acked Write 6 Sent Done Done false
17355 5862 Acked Flush 0 Sent Done Done false
17356 5863 Acked Write 7 Sent Done Done false
17357 5864 Acked Flush 0 Sent Done Done false
17358 5865 Acked Write 1 Sent Done Done false
17359 5866 Acked Write 3 Sent Done Done false
17360 5867 Acked Flush 0 Sent Done Done false
17361 5868 Acked Write 5 Sent Done Done false
17362 5869 Acked Write 2 Sent Done Done false
17363 5870 Acked Write 5 Sent Done Done false
17364 5871 Acked Read 9 Sent Done Done false
17365 5872 Acked Read 4 Sent Done Done false
17366 5873 Acked Write 5 Sent Done Done false
17367 5874 Acked Write 4 Sent Done Done false
17368 5875 Acked Read 5 Sent Done Done false
17369 5876 Acked Flush 0 Sent Done Done false
17370 5877 Acked Read 8 Sent Done Done false
17371 5878 Acked Read 6 Sent Done Done false
17372 5879 Acked Write 3 Sent Done Done false
17373 5880 Acked Read 3 Sent Done Done false
17374 5881 Acked Write 8 Sent Done Done false
17375 5882 Acked Read 3 Sent Done Done false
17376 5883 Acked Read 7 Sent Done Done false
17377 5884 Acked Read 6 Sent Done Done false
17378 5885 Acked Read 3 Sent Done Done false
17379 5886 Acked Write 3 Sent Done Done false
17380 5887 Acked Read 7 Sent Done Done false
17381 5888 Acked Read 9 Sent Done Done false
17382 5889 Acked Write 9 Sent Done Done false
17383 5890 Acked Read 6 Sent Done Done false
17384 5891 Acked Write 2 Sent Done Done false
17385 5892 Acked Write 1 Sent Done Done false
17386 5893 Acked Write 5 Sent Done Done false
17387 5894 Acked Write 10 Sent Done Done false
17388 5895 Acked Read 6 Sent Done Done false
17389 5896 Acked Read 6 Sent Done Done false
17390 5897 Acked Read 7 Sent Done Done false
17391 5898 Acked Write 3 Sent Done Done false
17392 5899 Acked Write 3 Sent Done Done false
17393 5900 Acked Read 6 Sent Done Done false
17394 5901 Acked Write 9 Sent Done Done false
17395 5902 Acked Write 10 Sent Done Done false
17396 5903 Acked Write 1 Sent Done Done false
17397 5904 Acked Flush 0 Sent Done Done false
17398 5905 Acked Write 2 Sent Done Done false
17399 5906 Acked Write 3 Sent Done Done false
17400 5907 Acked Write 10 Sent Done Done false
17401 5908 Acked Flush 0 Sent Done Done false
17402 5909 Acked Read 8 Sent Done Done false
17403 5910 Acked Read 10 Sent Done Done false
17404 5911 Acked Read 10 Sent Done Done false
17405 5912 Acked Read 9 Sent Done Done false
17406 5913 Acked Write 6 Sent Done Done false
17407 5914 Acked Read 5 Sent Done Done false
17408 5915 Acked Write 5 Sent Done Done false
17409 5916 Acked Write 6 Sent Done Done false
17410 5917 Acked Write 10 Sent Done Done false
17411 5918 Acked Read 3 Sent Done Done false
17412 5919 Acked Write 6 Sent Done Done false
17413 5920 Acked Write 1 Sent Done Done false
17414 5921 Acked Write 7 Sent Done Done false
17415 5922 Acked Read 5 Sent Done Done false
17416 5923 Acked Write 8 Sent Done Done false
17417 5924 Acked Read 8 Sent Done Done false
17418 5925 Acked Read 2 Sent Done Done false
17419 5926 Acked Write 10 Sent Done Done false
17420 5927 Acked Write 5 Sent Done Done false
17421 5928 Acked Read 10 Sent Done Done false
17422 5929 Acked Read 1 Sent Done Done false
17423 5930 Acked Read 7 Sent Done Done false
17424 5931 Acked Write 6 Sent Done Done false
17425 5932 Acked Read 7 Sent Done Done false
17426 5933 Acked Read 5 Sent Done Done false
17427 5934 Acked Read 9 Sent Done Done false
17428 5935 Acked Flush 0 Sent Done Done false
17429 5936 Acked Write 6 Sent Done Done false
17430 5937 Acked Read 9 Sent Done Done false
17431 5938 Acked Read 4 Sent Done Done false
17432 5939 Acked Write 3 Sent Done Done false
17433 5940 Acked Flush 0 Sent Done Done false
17434 5941 Acked Read 8 Sent Done Done false
17435 5942 Acked Write 7 Sent Done Done false
17436 5943 Acked Read 5 Sent Done Done false
17437 5944 Acked Read 6 Sent Done Done false
17438 5945 Acked Read 5 Sent Done Done false
17439 5946 Acked Write 2 Sent Done Done false
17440 5947 Acked Read 1 Sent Done Done false
17441 5948 Acked Flush 0 Sent Done Done false
17442 5949 Acked Read 4 Sent Done Done false
17443 5950 Acked Read 5 Sent Done Done false
17444 5951 Acked Read 10 Sent Done Done false
17445 5952 Acked Flush 0 Sent Done Done false
17446 5953 Acked Write 2 Sent Done Done false
17447 5954 Acked Write 10 Sent Done Done false
17448 5955 Acked Flush 0 Sent Done Done false
17449 5956 Acked Read 1 Sent Done Done false
17450 5957 Acked Flush 0 Sent Done Done false
17451 5958 Acked Write 3 Sent Done Done false
17452 5959 Acked Read 8 Sent Done Done false
17453 5960 Acked Write 4 Sent Done Done false
17454 5961 Acked Flush 0 Sent Done Done false
17455 5962 Acked Read 5 Sent Done Done false
17456 5963 Acked Read 7 Sent Done Done false
17457 5964 Acked Read 7 Sent Done Done false
17458 5965 Acked Write 6 Sent Done Done false
17459 5966 Acked Read 3 Sent Done Done false
17460 5967 Acked Read 7 Sent Done Done false
17461 5968 Acked Read 7 Sent Done Done false
17462 5969 Acked Write 5 Sent Done Done false
17463 5970 Acked Read 5 Sent Done Done false
17464 5971 Acked Write 4 Sent Done Done false
17465 5972 Acked Read 6 Sent Done Done false
17466 5973 Acked Read 10 Sent Done Done false
17467 5974 Acked Read 3 Sent Done Done false
17468 5975 Acked Read 4 Sent Done Done false
17469 5976 Acked Read 9 Sent Done Done false
17470 5977 Acked Write 3 Sent Done Done false
17471 5978 Acked Read 9 Sent Done Done false
17472 5979 Acked Write 4 Sent Done Done false
17473 5980 Acked Read 1 Sent Done Done false
17474 5981 Acked Write 7 Sent Done Done false
17475 5982 Acked Read 7 Sent Done Done false
17476 5983 Acked Read 4 Sent Done Done false
17477 5984 Acked Write 6 Sent Done Done false
17478 5985 Acked Read 4 Sent Done Done false
17479 5986 Acked Write 6 Sent Done Done false
17480 5987 Acked Read 2 Sent Done Done false
17481 5988 Acked Write 1 Sent Done Done false
17482 5989 Acked Write 4 Sent Done Done false
17483 5990 Acked Write 5 Sent Done Done false
17484 5991 Acked Read 9 Sent Done Done false
17485 5992 Acked Read 7 Sent Done Done false
17486 5993 Acked Read 4 Sent Done Done false
17487 5994 Acked Read 9 Sent Done Done false
17488 5995 Acked Read 6 Sent Done Done false
17489 5996 Acked Write 7 Sent Done Done false
17490 5997 Acked Read 5 Sent Done Done false
17491 5998 Acked Read 9 Sent Done Done false
17492 5999 Acked Read 10 Sent Done Done false
17493 6000 Acked Read 6 Sent Done Done false
17494 6001 Acked Read 5 Sent Done Done false
17495 6002 Acked Read 8 Sent Done Done false
17496 6003 Acked Read 5 Sent Done Done false
17497 6004 Acked Write 9 Sent Done Done false
17498 6005 Acked Write 6 Sent Done Done false
17499 6006 Acked Write 4 Sent Done Done false
17500 6007 Acked Read 8 Sent Done Done false
17501 6008 Acked Write 3 Sent Done Done false
17502 6009 Acked Write 7 Sent Done Done false
17503 6010 Acked Write 1 Sent Done Done false
17504 6011 Acked Read 5 Sent Done Done false
17505 6012 Acked Flush 0 Sent Done Done false
17506 6013 Acked Write 5 Sent Done Done false
17507 6014 Acked Write 4 Sent Done Done false
17508 6015 Acked Write 9 Sent Done Done false
17509 6016 Acked Write 3 Sent Done Done false
17510 6017 Acked Read 7 Sent Done Done false
17511 6018 Acked Write 2 Sent Done Done false
17512 6019 Acked Flush 0 Sent Done Done false
17513 6020 Acked Write 3 Sent Done Done false
17514 6021 Acked Read 2 Sent Done Done false
17515 6022 Acked Write 6 Sent Done Done false
17516 6023 Acked Read 9 Sent Done Done false
17517 6024 Acked Read 7 Sent Done Done false
17518 6025 Acked Read 10 Sent Done Done false
17519 6026 Acked Read 1 Sent Done Done false
17520 6027 Acked Read 8 Sent Done Done false
17521 6028 Acked Write 10 Sent Done Done false
17522 6029 Acked Read 1 Sent Done Done false
17523 6030 Acked Flush 0 Sent Done Done false
17524 6031 Acked Read 8 Sent Done Done false
17525 6032 Acked Read 9 Sent Done Done false
17526 6033 Acked Write 6 Sent Done Done false
17527 6034 Acked Write 4 Sent Done Done false
17528 6035 Acked Write 5 Sent Done Done false
17529 6036 Acked Write 1 Sent Done Done false
17530 6037 Acked Flush 0 Sent Done Done false
17531 6038 Acked Flush 0 Sent Done Done false
17532 6039 Acked Flush 0 Sent Done Done false
17533 6040 Acked Read 2 Sent Done Done false
17534 6041 Acked Read 6 Sent Done Done false
17535 6042 Acked Read 8 Sent Done Done false
17536 6043 Acked Flush 0 Sent Done Done false
17537 6044 Acked Read 5 Sent Done Done false
17538 6045 Acked Read 9 Sent Done Done false
17539 6046 Acked Write 5 Sent Done Done false
17540 6047 Acked Write 7 Sent Done Done false
17541 6048 Acked Write 2 Sent Done Done false
17542 6049 Acked Read 6 Sent Done Done false
17543 6050 Acked Write 8 Sent Done Done false
17544 6051 Acked Write 10 Sent Done Done false
17545 6052 Acked Read 6 Sent Done Done false
17546 6053 Acked Read 1 Sent Done Done false
17547 6054 Acked Flush 0 Sent Done Done false
17548 6055 Acked Read 5 Sent Done Done false
17549 6056 Acked Write 7 Sent Done Done false
17550 6057 Acked Read 9 Sent Done Done false
17551 6058 Acked Read 8 Sent Done Done false
17552 6059 Acked Write 8 Sent Done Done false
17553 6060 Acked Write 5 Sent Done Done false
17554 6061 Acked Read 5 Sent Done Done false
17555 6062 Acked Write 2 Sent Done Done false
17556 6063 Acked Write 2 Sent Done Done false
17557 6064 Acked Read 9 Sent Done Done false
17558 6065 Acked Write 4 Sent Done Done false
17559 6066 Acked Read 4 Sent Done Done false
17560 6067 Acked Read 10 Sent Done Done false
17561 6068 Acked Write 8 Sent Done Done false
17562 6069 Acked Read 7 Sent Done Done false
17563 6070 Acked Flush 0 Sent Done Done false
17564 6071 Acked Read 9 Sent Done Done false
17565 6072 Acked Write 3 Sent Done Done false
17566 6073 Acked Write 10 Sent Done Done false
17567 6074 Acked Write 6 Sent Done Done false
17568 6075 Acked Write 6 Sent Done Done false
17569 6076 Acked Write 6 Sent Done Done false
17570 6077 Acked Read 10 Sent Done Done false
17571 6078 Acked Write 4 Sent Done Done false
17572 6079 Acked Read 2 Sent Done Done false
17573 6080 Acked Write 9 Sent Done Done false
17574 6081 Acked Read 9 Sent Done Done false
17575 6082 Acked Write 3 Sent Done Done false
17576 6083 Acked Read 1 Sent Done Done false
17577 6084 Acked Read 7 Sent Done Done false
17578 6085 Acked Read 8 Sent Done Done false
17579 6086 Acked Flush 0 Sent Done Done false
17580 6087 Acked Write 5 Sent Done Done false
17581 6088 Acked Flush 0 Sent Done Done false
17582 6089 Acked Read 8 Sent Done Done false
17583 6090 Acked Write 8 Sent Done Done false
17584 6091 Acked Write 8 Sent Done Done false
17585 6092 Acked Read 10 Sent Done Done false
17586 6093 Acked Read 1 Sent Done Done false
17587 6094 Acked Flush 0 Sent Done Done false
17588 6095 Acked Write 1 Sent Done Done false
17589 6096 Acked Read 7 Sent Done Done false
17590 6097 Acked Read 5 Sent Done Done false
17591 6098 Acked Write 10 Sent Done Done false
17592 6099 Acked Flush 0 Sent Done Done false
17593 6100 Acked Write 10 Sent Done Done false
17594 6101 Acked Read 6 Sent Done Done false
17595 6102 Acked Flush 0 Sent Done Done false
17596 6103 Acked Write 8 Sent Done Done false
17597 6104 Acked Read 3 Sent Done Done false
17598 6105 Acked Write 2 Sent Done Done false
17599 6106 Acked Read 9 Sent Done Done false
17600 6107 Acked Read 8 Sent Done Done false
17601 6108 Acked Flush 0 Sent Done Done false
17602 6109 Acked Read 1 Sent Done Done false
17603 6110 Acked Read 3 Sent Done Done false
17604 6111 Acked Read 3 Sent Done Done false
17605 6112 Acked Read 4 Sent Done Done false
17606 6113 Acked Flush 0 Sent Done Done false
17607 6114 Acked Write 2 Sent Done Done false
17608 6115 Acked Read 1 Sent Done Done false
17609 6116 Acked Write 4 Sent Done Done false
17610 6117 Acked Write 5 Sent Done Done false
17611 6118 Acked Flush 0 Sent Done Done false
17612 6119 Acked Read 3 Sent Done Done false
17613 6120 Acked Read 2 Sent Done Done false
17614 6121 Acked Read 10 Sent Done Done false
17615 6122 Acked Write 6 Sent Done Done false
17616 6123 Acked Read 3 Sent Done Done false
17617 6124 Acked Write 4 Sent Done Done false
17618 6125 Acked Write 5 Sent Done Done false
17619 6126 Acked Read 4 Sent Done Done false
17620 6127 Acked Read 6 Sent Done Done false
17621 6128 Acked Read 9 Sent Done Done false
17622 6129 Acked Read 9 Sent Done Done false
17623 6130 Acked Write 6 Sent Done Done false
17624 6131 Acked Write 10 Sent Done Done false
17625 6132 Acked Read 6 Sent Done Done false
17626 STATES DS:0 DS:1 DS:2 TOTAL
17627 Sent 301 0 0 301
17628 Done 0 301 301 602
17629 Skipped 0 0 0 0
17630 Error 0 0 0 0
17631 Last Flush: Some(JobId(5831)) Some(JobId(6118)) Some(JobId(6118))
17632 Downstairs last five completed: 5831 5830 5829 5828 5827
17633 Upstairs last five completed: 6132 6131 6130 6129 6128
17634 CLIENT: Up:0 ds:301 act:2
17635 ----------------------------------------------------------------
17636 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
17637 States: Offline Active Active
17638 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17639 5832 Acked Read 4 Sent Done Done false
17640 5833 Acked Read 4 Sent Done Done false
17641 5834 Acked Read 6 Sent Done Done false
17642 5835 Acked Flush 0 Sent Done Done false
17643 5836 Acked Read 8 Sent Done Done false
17644 5837 Acked Write 3 Sent Done Done false
17645 5838 Acked Write 7 Sent Done Done false
17646 5839 Acked Write 9 Sent Done Done false
17647 5840 Acked Write 1 Sent Done Done false
17648 5841 Acked Write 4 Sent Done Done false
17649 5842 Acked Write 6 Sent Done Done false
17650 5843 Acked Read 6 Sent Done Done false
17651 5844 Acked Read 10 Sent Done Done false
17652 5845 Acked Write 5 Sent Done Done false
17653 5846 Acked Write 3 Sent Done Done false
17654 5847 Acked Write 5 Sent Done Done false
17655 5848 Acked Write 4 Sent Done Done false
17656 5849 Acked Write 10 Sent Done Done false
17657 5850 Acked Read 2 Sent Done Done false
17658 5851 Acked Read 8 Sent Done Done false
17659 5852 Acked Read 9 Sent Done Done false
17660 5853 Acked Read 8 Sent Done Done false
17661 5854 Acked Read 8 Sent Done Done false
17662 5855 Acked Write 6 Sent Done Done false
17663 5856 Acked Read 1 Sent Done Done false
17664 5857 Acked Read 7 Sent Done Done false
17665 5858 Acked Read 5 Sent Done Done false
17666 5859 Acked Write 5 Sent Done Done false
17667 5860 Acked Write 7 Sent Done Done false
17668 5861 Acked Write 6 Sent Done Done false
17669 5862 Acked Flush 0 Sent Done Done false
17670 5863 Acked Write 7 Sent Done Done false
17671 5864 Acked Flush 0 Sent Done Done false
17672 5865 Acked Write 1 Sent Done Done false
17673 5866 Acked Write 3 Sent Done Done false
17674 5867 Acked Flush 0 Sent Done Done false
17675 5868 Acked Write 5 Sent Done Done false
17676 5869 Acked Write 2 Sent Done Done false
17677 5870 Acked Write 5 Sent Done Done false
17678 5871 Acked Read 9 Sent Done Done false
17679 5872 Acked Read 4 Sent Done Done false
17680 5873 Acked Write 5 Sent Done Done false
17681 5874 Acked Write 4 Sent Done Done false
17682 5875 Acked Read 5 Sent Done Done false
17683 5876 Acked Flush 0 Sent Done Done false
17684 5877 Acked Read 8 Sent Done Done false
17685 5878 Acked Read 6 Sent Done Done false
17686 5879 Acked Write 3 Sent Done Done false
17687 5880 Acked Read 3 Sent Done Done false
17688 5881 Acked Write 8 Sent Done Done false
17689 5882 Acked Read 3 Sent Done Done false
17690 5883 Acked Read 7 Sent Done Done false
17691 5884 Acked Read 6 Sent Done Done false
17692 5885 Acked Read 3 Sent Done Done false
17693 5886 Acked Write 3 Sent Done Done false
17694 5887 Acked Read 7 Sent Done Done false
17695 5888 Acked Read 9 Sent Done Done false
17696 5889 Acked Write 9 Sent Done Done false
17697 5890 Acked Read 6 Sent Done Done false
17698 5891 Acked Write 2 Sent Done Done false
17699 5892 Acked Write 1 Sent Done Done false
17700 5893 Acked Write 5 Sent Done Done false
17701 5894 Acked Write 10 Sent Done Done false
17702 5895 Acked Read 6 Sent Done Done false
17703 5896 Acked Read 6 Sent Done Done false
17704 5897 Acked Read 7 Sent Done Done false
17705 5898 Acked Write 3 Sent Done Done false
17706 5899 Acked Write 3 Sent Done Done false
17707 5900 Acked Read 6 Sent Done Done false
17708 5901 Acked Write 9 Sent Done Done false
17709 5902 Acked Write 10 Sent Done Done false
17710 5903 Acked Write 1 Sent Done Done false
17711 5904 Acked Flush 0 Sent Done Done false
17712 5905 Acked Write 2 Sent Done Done false
17713 5906 Acked Write 3 Sent Done Done false
17714 5907 Acked Write 10 Sent Done Done false
17715 5908 Acked Flush 0 Sent Done Done false
17716 5909 Acked Read 8 Sent Done Done false
17717 5910 Acked Read 10 Sent Done Done false
17718 5911 Acked Read 10 Sent Done Done false
17719 5912 Acked Read 9 Sent Done Done false
17720 5913 Acked Write 6 Sent Done Done false
17721 5914 Acked Read 5 Sent Done Done false
17722 5915 Acked Write 5 Sent Done Done false
17723 5916 Acked Write 6 Sent Done Done false
17724 5917 Acked Write 10 Sent Done Done false
17725 5918 Acked Read 3 Sent Done Done false
17726 5919 Acked Write 6 Sent Done Done false
17727 5920 Acked Write 1 Sent Done Done false
17728 5921 Acked Write 7 Sent Done Done false
17729 5922 Acked Read 5 Sent Done Done false
17730 5923 Acked Write 8 Sent Done Done false
17731 5924 Acked Read 8 Sent Done Done false
17732 5925 Acked Read 2 Sent Done Done false
17733 5926 Acked Write 10 Sent Done Done false
17734 5927 Acked Write 5 Sent Done Done false
17735 5928 Acked Read 10 Sent Done Done false
17736 5929 Acked Read 1 Sent Done Done false
17737 5930 Acked Read 7 Sent Done Done false
17738 5931 Acked Write 6 Sent Done Done false
17739 5932 Acked Read 7 Sent Done Done false
17740 5933 Acked Read 5 Sent Done Done false
17741 5934 Acked Read 9 Sent Done Done false
17742 5935 Acked Flush 0 Sent Done Done false
17743 5936 Acked Write 6 Sent Done Done false
17744 5937 Acked Read 9 Sent Done Done false
17745 5938 Acked Read 4 Sent Done Done false
17746 5939 Acked Write 3 Sent Done Done false
17747 5940 Acked Flush 0 Sent Done Done false
17748 5941 Acked Read 8 Sent Done Done false
17749 5942 Acked Write 7 Sent Done Done false
17750 5943 Acked Read 5 Sent Done Done false
17751 5944 Acked Read 6 Sent Done Done false
17752 5945 Acked Read 5 Sent Done Done false
17753 5946 Acked Write 2 Sent Done Done false
17754 5947 Acked Read 1 Sent Done Done false
17755 5948 Acked Flush 0 Sent Done Done false
17756 5949 Acked Read 4 Sent Done Done false
17757 5950 Acked Read 5 Sent Done Done false
17758 5951 Acked Read 10 Sent Done Done false
17759 5952 Acked Flush 0 Sent Done Done false
17760 5953 Acked Write 2 Sent Done Done false
17761 5954 Acked Write 10 Sent Done Done false
17762 5955 Acked Flush 0 Sent Done Done false
17763 5956 Acked Read 1 Sent Done Done false
17764 5957 Acked Flush 0 Sent Done Done false
17765 5958 Acked Write 3 Sent Done Done false
17766 5959 Acked Read 8 Sent Done Done false
17767 5960 Acked Write 4 Sent Done Done false
17768 5961 Acked Flush 0 Sent Done Done false
17769 5962 Acked Read 5 Sent Done Done false
17770 5963 Acked Read 7 Sent Done Done false
17771 5964 Acked Read 7 Sent Done Done false
17772 5965 Acked Write 6 Sent Done Done false
17773 5966 Acked Read 3 Sent Done Done false
17774 5967 Acked Read 7 Sent Done Done false
17775 5968 Acked Read 7 Sent Done Done false
17776 5969 Acked Write 5 Sent Done Done false
17777 5970 Acked Read 5 Sent Done Done false
17778 5971 Acked Write 4 Sent Done Done false
17779 5972 Acked Read 6 Sent Done Done false
17780 5973 Acked Read 10 Sent Done Done false
17781 5974 Acked Read 3 Sent Done Done false
17782 5975 Acked Read 4 Sent Done Done false
17783 5976 Acked Read 9 Sent Done Done false
17784 5977 Acked Write 3 Sent Done Done false
17785 5978 Acked Read 9 Sent Done Done false
17786 5979 Acked Write 4 Sent Done Done false
17787 5980 Acked Read 1 Sent Done Done false
17788 5981 Acked Write 7 Sent Done Done false
17789 5982 Acked Read 7 Sent Done Done false
17790 5983 Acked Read 4 Sent Done Done false
17791 5984 Acked Write 6 Sent Done Done false
17792 5985 Acked Read 4 Sent Done Done false
17793 5986 Acked Write 6 Sent Done Done false
17794 5987 Acked Read 2 Sent Done Done false
17795 5988 Acked Write 1 Sent Done Done false
17796 5989 Acked Write 4 Sent Done Done false
17797 5990 Acked Write 5 Sent Done Done false
17798 5991 Acked Read 9 Sent Done Done false
17799 5992 Acked Read 7 Sent Done Done false
17800 5993 Acked Read 4 Sent Done Done false
17801 5994 Acked Read 9 Sent Done Done false
17802 5995 Acked Read 6 Sent Done Done false
17803 5996 Acked Write 7 Sent Done Done false
17804 5997 Acked Read 5 Sent Done Done false
17805 5998 Acked Read 9 Sent Done Done false
17806 5999 Acked Read 10 Sent Done Done false
17807 6000 Acked Read 6 Sent Done Done false
17808 6001 Acked Read 5 Sent Done Done false
17809 6002 Acked Read 8 Sent Done Done false
17810 6003 Acked Read 5 Sent Done Done false
17811 6004 Acked Write 9 Sent Done Done false
17812 6005 Acked Write 6 Sent Done Done false
17813 6006 Acked Write 4 Sent Done Done false
17814 6007 Acked Read 8 Sent Done Done false
17815 6008 Acked Write 3 Sent Done Done false
17816 6009 Acked Write 7 Sent Done Done false
17817 6010 Acked Write 1 Sent Done Done false
17818 6011 Acked Read 5 Sent Done Done false
17819 6012 Acked Flush 0 Sent Done Done false
17820 6013 Acked Write 5 Sent Done Done false
17821 6014 Acked Write 4 Sent Done Done false
17822 6015 Acked Write 9 Sent Done Done false
17823 6016 Acked Write 3 Sent Done Done false
17824 6017 Acked Read 7 Sent Done Done false
17825 6018 Acked Write 2 Sent Done Done false
17826 6019 Acked Flush 0 Sent Done Done false
17827 6020 Acked Write 3 Sent Done Done false
17828 6021 Acked Read 2 Sent Done Done false
17829 6022 Acked Write 6 Sent Done Done false
17830 6023 Acked Read 9 Sent Done Done false
17831 6024 Acked Read 7 Sent Done Done false
17832 6025 Acked Read 10 Sent Done Done false
17833 6026 Acked Read 1 Sent Done Done false
17834 6027 Acked Read 8 Sent Done Done false
17835 6028 Acked Write 10 Sent Done Done false
17836 6029 Acked Read 1 Sent Done Done false
17837 6030 Acked Flush 0 Sent Done Done false
17838 6031 Acked Read 8 Sent Done Done false
17839 6032 Acked Read 9 Sent Done Done false
17840 6033 Acked Write 6 Sent Done Done false
17841 6034 Acked Write 4 Sent Done Done false
17842 6035 Acked Write 5 Sent Done Done false
17843 6036 Acked Write 1 Sent Done Done false
17844 6037 Acked Flush 0 Sent Done Done false
17845 6038 Acked Flush 0 Sent Done Done false
17846 6039 Acked Flush 0 Sent Done Done false
17847 6040 Acked Read 2 Sent Done Done false
17848 6041 Acked Read 6 Sent Done Done false
17849 6042 Acked Read 8 Sent Done Done false
17850 6043 Acked Flush 0 Sent Done Done false
17851 6044 Acked Read 5 Sent Done Done false
17852 6045 Acked Read 9 Sent Done Done false
17853 6046 Acked Write 5 Sent Done Done false
17854 6047 Acked Write 7 Sent Done Done false
17855 6048 Acked Write 2 Sent Done Done false
17856 6049 Acked Read 6 Sent Done Done false
17857 6050 Acked Write 8 Sent Done Done false
17858 6051 Acked Write 10 Sent Done Done false
17859 6052 Acked Read 6 Sent Done Done false
17860 6053 Acked Read 1 Sent Done Done false
17861 6054 Acked Flush 0 Sent Done Done false
17862 6055 Acked Read 5 Sent Done Done false
17863 6056 Acked Write 7 Sent Done Done false
17864 6057 Acked Read 9 Sent Done Done false
17865 6058 Acked Read 8 Sent Done Done false
17866 6059 Acked Write 8 Sent Done Done false
17867 6060 Acked Write 5 Sent Done Done false
17868 6061 Acked Read 5 Sent Done Done false
17869 6062 Acked Write 2 Sent Done Done false
17870 6063 Acked Write 2 Sent Done Done false
17871 6064 Acked Read 9 Sent Done Done false
17872 6065 Acked Write 4 Sent Done Done false
17873 6066 Acked Read 4 Sent Done Done false
17874 6067 Acked Read 10 Sent Done Done false
17875 6068 Acked Write 8 Sent Done Done false
17876 6069 Acked Read 7 Sent Done Done false
17877 6070 Acked Flush 0 Sent Done Done false
17878 6071 Acked Read 9 Sent Done Done false
17879 6072 Acked Write 3 Sent Done Done false
17880 6073 Acked Write 10 Sent Done Done false
17881 6074 Acked Write 6 Sent Done Done false
17882 6075 Acked Write 6 Sent Done Done false
17883 6076 Acked Write 6 Sent Done Done false
17884 6077 Acked Read 10 Sent Done Done false
17885 6078 Acked Write 4 Sent Done Done false
17886 6079 Acked Read 2 Sent Done Done false
17887 6080 Acked Write 9 Sent Done Done false
17888 6081 Acked Read 9 Sent Done Done false
17889 6082 Acked Write 3 Sent Done Done false
17890 6083 Acked Read 1 Sent Done Done false
17891 6084 Acked Read 7 Sent Done Done false
17892 6085 Acked Read 8 Sent Done Done false
17893 6086 Acked Flush 0 Sent Done Done false
17894 6087 Acked Write 5 Sent Done Done false
17895 6088 Acked Flush 0 Sent Done Done false
17896 6089 Acked Read 8 Sent Done Done false
17897 6090 Acked Write 8 Sent Done Done false
17898 6091 Acked Write 8 Sent Done Done false
17899 6092 Acked Read 10 Sent Done Done false
17900 6093 Acked Read 1 Sent Done Done false
17901 6094 Acked Flush 0 Sent Done Done false
17902 6095 Acked Write 1 Sent Done Done false
17903 6096 Acked Read 7 Sent Done Done false
17904 6097 Acked Read 5 Sent Done Done false
17905 6098 Acked Write 10 Sent Done Done false
17906 6099 Acked Flush 0 Sent Done Done false
17907 6100 Acked Write 10 Sent Done Done false
17908 6101 Acked Read 6 Sent Done Done false
17909 6102 Acked Flush 0 Sent Done Done false
17910 6103 Acked Write 8 Sent Done Done false
17911 6104 Acked Read 3 Sent Done Done false
17912 6105 Acked Write 2 Sent Done Done false
17913 6106 Acked Read 9 Sent Done Done false
17914 6107 Acked Read 8 Sent Done Done false
17915 6108 Acked Flush 0 Sent Done Done false
17916 6109 Acked Read 1 Sent Done Done false
17917 6110 Acked Read 3 Sent Done Done false
17918 6111 Acked Read 3 Sent Done Done false
17919 6112 Acked Read 4 Sent Done Done false
17920 6113 Acked Flush 0 Sent Done Done false
17921 6114 Acked Write 2 Sent Done Done false
17922 6115 Acked Read 1 Sent Done Done false
17923 6116 Acked Write 4 Sent Done Done false
17924 6117 Acked Write 5 Sent Done Done false
17925 6118 Acked Flush 0 Sent Done Done false
17926 6119 Acked Read 3 Sent Done Done false
17927 6120 Acked Read 2 Sent Done Done false
17928 6121 Acked Read 10 Sent Done Done false
17929 6122 Acked Write 6 Sent Done Done false
17930 6123 Acked Read 3 Sent Done Done false
17931 6124 Acked Write 4 Sent Done Done false
17932 6125 Acked Write 5 Sent Done Done false
17933 6126 Acked Read 4 Sent Done Done false
17934 6127 Acked Read 6 Sent Done Done false
17935 6128 Acked Read 9 Sent Done Done false
17936 6129 Acked Read 9 Sent Done Done false
17937 6130 Acked Write 6 Sent Done Done false
17938 6131 Acked Write 10 Sent Done Done false
17939 6132 Acked Read 6 Sent Done Done false
17940 6133 Acked Flush 0 Sent Done Done false
17941 STATES DS:0 DS:1 DS:2 TOTAL
17942 Sent 302 0 0 302
17943 Done 0 302 302 604
17944 Skipped 0 0 0 0
17945 Error 0 0 0 0
17946 Last Flush: Some(JobId(5831)) Some(JobId(6133)) Some(JobId(6133))
17947 Downstairs last five completed: 5831 5830 5829 5828 5827
17948 Upstairs last five completed: 6133 6132 6131 6130 6129
17949 CLIENT: Up:0 ds:302 act:2
179502025-01-23T17:41:29.642ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
179512025-01-23T17:41:29.642ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
179522025-01-23T17:41:29.642ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
179532025-01-23T17:41:29.642ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
179542025-01-23T17:41:29.642ZINFOcrucible: client task is exiting = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
17955 ----------------------------------------------------------------
17956 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
17957 States: Offline Active Active
17958 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17959 5832 Acked Read 4 Sent Done Done false
17960 5833 Acked Read 4 Sent Done Done false
17961 5834 Acked Read 6 Sent Done Done false
17962 5835 Acked Flush 0 Sent Done Done false
17963 5836 Acked Read 8 Sent Done Done false
17964 5837 Acked Write 3 Sent Done Done false
17965 5838 Acked Write 7 Sent Done Done false
17966 5839 Acked Write 9 Sent Done Done false
17967 5840 Acked Write 1 Sent Done Done false
17968 5841 Acked Write 4 Sent Done Done false
17969 5842 Acked Write 6 Sent Done Done false
17970 5843 Acked Read 6 Sent Done Done false
17971 5844 Acked Read 10 Sent Done Done false
17972 5845 Acked Write 5 Sent Done Done false
17973 5846 Acked Write 3 Sent Done Done false
17974 5847 Acked Write 5 Sent Done Done false
17975 5848 Acked Write 4 Sent Done Done false
17976 5849 Acked Write 10 Sent Done Done false
17977 5850 Acked Read 2 Sent Done Done false
17978 5851 Acked Read 8 Sent Done Done false
17979 5852 Acked Read 9 Sent Done Done false
17980 5853 Acked Read 8 Sent Done Done false
17981 5854 Acked Read 8 Sent Done Done false
17982 5855 Acked Write 6 Sent Done Done false
17983 5856 Acked Read 1 Sent Done Done false
17984 5857 Acked Read 7 Sent Done Done false
17985 5858 Acked Read 5 Sent Done Done false
17986 5859 Acked Write 5 Sent Done Done false
17987 5860 Acked Write 7 Sent Done Done false
17988 5861 Acked Write 6 Sent Done Done false
17989 5862 Acked Flush 0 Sent Done Done false
17990 5863 Acked Write 7 Sent Done Done false
17991 5864 Acked Flush 0 Sent Done Done false
17992 5865 Acked Write 1 Sent Done Done false
17993 5866 Acked Write 3 Sent Done Done false
17994 5867 Acked Flush 0 Sent Done Done false
17995 5868 Acked Write 5 Sent Done Done false
17996 5869 Acked Write 2 Sent Done Done false
17997 5870 Acked Write 5 Sent Done Done false
17998 5871 Acked Read 9 Sent Done Done false
17999 5872 Acked Read 4 Sent Done Done false
18000 5873 Acked Write 5 Sent Done Done false
18001 5874 Acked Write 4 Sent Done Done false
18002 5875 Acked Read 5 Sent Done Done false
18003 5876 Acked Flush 0 Sent Done Done false
18004 5877 Acked Read 8 Sent Done Done false
18005 5878 Acked Read 6 Sent Done Done false
18006 5879 Acked Write 3 Sent Done Done false
18007 5880 Acked Read 3 Sent Done Done false
18008 5881 Acked Write 8 Sent Done Done false
18009 5882 Acked Read 3 Sent Done Done false
18010 5883 Acked Read 7 Sent Done Done false
18011 5884 Acked Read 6 Sent Done Done false
18012 5885 Acked Read 3 Sent Done Done false
18013 5886 Acked Write 3 Sent Done Done false
18014 5887 Acked Read 7 Sent Done Done false
18015 5888 Acked Read 9 Sent Done Done false
18016 5889 Acked Write 9 Sent Done Done false
18017 5890 Acked Read 6 Sent Done Done false
18018 5891 Acked Write 2 Sent Done Done false
18019 5892 Acked Write 1 Sent Done Done false
18020 5893 Acked Write 5 Sent Done Done false
18021 5894 Acked Write 10 Sent Done Done false
18022 5895 Acked Read 6 Sent Done Done false
18023 5896 Acked Read 6 Sent Done Done false
18024 5897 Acked Read 7 Sent Done Done false
18025 5898 Acked Write 3 Sent Done Done false
18026 5899 Acked Write 3 Sent Done Done false
18027 5900 Acked Read 6 Sent Done Done false
18028 5901 Acked Write 9 Sent Done Done false
18029 5902 Acked Write 10 Sent Done Done false
18030 5903 Acked Write 1 Sent Done Done false
18031 5904 Acked Flush 0 Sent Done Done false
18032 5905 Acked Write 2 Sent Done Done false
18033 5906 Acked Write 3 Sent Done Done false
18034 5907 Acked Write 10 Sent Done Done false
18035 5908 Acked Flush 0 Sent Done Done false
18036 5909 Acked Read 8 Sent Done Done false
18037 5910 Acked Read 10 Sent Done Done false
18038 5911 Acked Read 10 Sent Done Done false
18039 5912 Acked Read 9 Sent Done Done false
18040 5913 Acked Write 6 Sent Done Done false
18041 5914 Acked Read 5 Sent Done Done false
18042 5915 Acked Write 5 Sent Done Done false
18043 5916 Acked Write 6 Sent Done Done false
18044 5917 Acked Write 10 Sent Done Done false
18045 5918 Acked Read 3 Sent Done Done false
18046 5919 Acked Write 6 Sent Done Done false
18047 5920 Acked Write 1 Sent Done Done false
18048 5921 Acked Write 7 Sent Done Done false
18049 5922 Acked Read 5 Sent Done Done false
18050 5923 Acked Write 8 Sent Done Done false
18051 5924 Acked Read 8 Sent Done Done false
18052 5925 Acked Read 2 Sent Done Done false
18053 5926 Acked Write 10 Sent Done Done false
18054 5927 Acked Write 5 Sent Done Done false
18055 5928 Acked Read 10 Sent Done Done false
18056 5929 Acked Read 1 Sent Done Done false
18057 5930 Acked Read 7 Sent Done Done false
18058 5931 Acked Write 6 Sent Done Done false
18059 5932 Acked Read 7 Sent Done Done false
18060 5933 Acked Read 5 Sent Done Done false
18061 5934 Acked Read 9 Sent Done Done false
18062 5935 Acked Flush 0 Sent Done Done false
18063 5936 Acked Write 6 Sent Done Done false
18064 5937 Acked Read 9 Sent Done Done false
18065 5938 Acked Read 4 Sent Done Done false
18066 5939 Acked Write 3 Sent Done Done false
18067 5940 Acked Flush 0 Sent Done Done false
18068 5941 Acked Read 8 Sent Done Done false
18069 5942 Acked Write 7 Sent Done Done false
18070 5943 Acked Read 5 Sent Done Done false
18071 5944 Acked Read 6 Sent Done Done false
18072 5945 Acked Read 5 Sent Done Done false
18073 5946 Acked Write 2 Sent Done Done false
18074 5947 Acked Read 1 Sent Done Done false
18075 5948 Acked Flush 0 Sent Done Done false
18076 5949 Acked Read 4 Sent Done Done false
18077 5950 Acked Read 5 Sent Done Done false
18078 5951 Acked Read 10 Sent Done Done false
18079 5952 Acked Flush 0 Sent Done Done false
18080 5953 Acked Write 2 Sent Done Done false
18081 5954 Acked Write 10 Sent Done Done false
18082 5955 Acked Flush 0 Sent Done Done false
18083 5956 Acked Read 1 Sent Done Done false
18084 5957 Acked Flush 0 Sent Done Done false
18085 5958 Acked Write 3 Sent Done Done false
18086 5959 Acked Read 8 Sent Done Done false
18087 5960 Acked Write 4 Sent Done Done false
18088 5961 Acked Flush 0 Sent Done Done false
18089 5962 Acked Read 5 Sent Done Done false
18090 5963 Acked Read 7 Sent Done Done false
18091 5964 Acked Read 7 Sent Done Done false
18092 5965 Acked Write 6 Sent Done Done false
18093 5966 Acked Read 3 Sent Done Done false
18094 5967 Acked Read 7 Sent Done Done false
18095 5968 Acked Read 7 Sent Done Done false
18096 5969 Acked Write 5 Sent Done Done false
18097 5970 Acked Read 5 Sent Done Done false
18098 5971 Acked Write 4 Sent Done Done false
18099 5972 Acked Read 6 Sent Done Done false
18100 5973 Acked Read 10 Sent Done Done false
18101 5974 Acked Read 3 Sent Done Done false
18102 5975 Acked Read 4 Sent Done Done false
18103 5976 Acked Read 9 Sent Done Done false
18104 5977 Acked Write 3 Sent Done Done false
18105 5978 Acked Read 9 Sent Done Done false
18106 5979 Acked Write 4 Sent Done Done false
18107 5980 Acked Read 1 Sent Done Done false
18108 5981 Acked Write 7 Sent Done Done false
18109 5982 Acked Read 7 Sent Done Done false
18110 5983 Acked Read 4 Sent Done Done false
18111 5984 Acked Write 6 Sent Done Done false
18112 5985 Acked Read 4 Sent Done Done false
18113 5986 Acked Write 6 Sent Done Done false
18114 5987 Acked Read 2 Sent Done Done false
18115 5988 Acked Write 1 Sent Done Done false
18116 5989 Acked Write 4 Sent Done Done false
18117 5990 Acked Write 5 Sent Done Done false
18118 5991 Acked Read 9 Sent Done Done false
18119 5992 Acked Read 7 Sent Done Done false
18120 5993 Acked Read 4 Sent Done Done false
18121 5994 Acked Read 9 Sent Done Done false
18122 5995 Acked Read 6 Sent Done Done false
18123 5996 Acked Write 7 Sent Done Done false
18124 5997 Acked Read 5 Sent Done Done false
18125 5998 Acked Read 9 Sent Done Done false
18126 5999 Acked Read 10 Sent Done Done false
18127 6000 Acked Read 6 Sent Done Done false
18128 6001 Acked Read 5 Sent Done Done false
18129 6002 Acked Read 8 Sent Done Done false
18130 6003 Acked Read 5 Sent Done Done false
18131 6004 Acked Write 9 Sent Done Done false
18132 6005 Acked Write 6 Sent Done Done false
18133 6006 Acked Write 4 Sent Done Done false
18134 6007 Acked Read 8 Sent Done Done false
18135 6008 Acked Write 3 Sent Done Done false
18136 6009 Acked Write 7 Sent Done Done false
18137 6010 Acked Write 1 Sent Done Done false
18138 6011 Acked Read 5 Sent Done Done false
18139 6012 Acked Flush 0 Sent Done Done false
18140 6013 Acked Write 5 Sent Done Done false
18141 6014 Acked Write 4 Sent Done Done false
18142 6015 Acked Write 9 Sent Done Done false
18143 6016 Acked Write 3 Sent Done Done false
18144 6017 Acked Read 7 Sent Done Done false
18145 6018 Acked Write 2 Sent Done Done false
18146 6019 Acked Flush 0 Sent Done Done false
18147 6020 Acked Write 3 Sent Done Done false
18148 6021 Acked Read 2 Sent Done Done false
18149 6022 Acked Write 6 Sent Done Done false
18150 6023 Acked Read 9 Sent Done Done false
18151 6024 Acked Read 7 Sent Done Done false
18152 6025 Acked Read 10 Sent Done Done false
18153 6026 Acked Read 1 Sent Done Done false
18154 6027 Acked Read 8 Sent Done Done false
18155 6028 Acked Write 10 Sent Done Done false
18156 6029 Acked Read 1 Sent Done Done false
18157 6030 Acked Flush 0 Sent Done Done false
18158 6031 Acked Read 8 Sent Done Done false
18159 6032 Acked Read 9 Sent Done Done false
18160 6033 Acked Write 6 Sent Done Done false
18161 6034 Acked Write 4 Sent Done Done false
18162 6035 Acked Write 5 Sent Done Done false
18163 6036 Acked Write 1 Sent Done Done false
18164 6037 Acked Flush 0 Sent Done Done false
18165 6038 Acked Flush 0 Sent Done Done false
18166 6039 Acked Flush 0 Sent Done Done false
18167 6040 Acked Read 2 Sent Done Done false
18168 6041 Acked Read 6 Sent Done Done false
18169 6042 Acked Read 8 Sent Done Done false
18170 6043 Acked Flush 0 Sent Done Done false
18171 6044 Acked Read 5 Sent Done Done false
18172 6045 Acked Read 9 Sent Done Done false
18173 6046 Acked Write 5 Sent Done Done false
18174 6047 Acked Write 7 Sent Done Done false
18175 6048 Acked Write 2 Sent Done Done false
18176 6049 Acked Read 6 Sent Done Done false
18177 6050 Acked Write 8 Sent Done Done false
18178 6051 Acked Write 10 Sent Done Done false
18179 6052 Acked Read 6 Sent Done Done false
18180 6053 Acked Read 1 Sent Done Done false
18181 6054 Acked Flush 0 Sent Done Done false
18182 6055 Acked Read 5 Sent Done Done false
18183 6056 Acked Write 7 Sent Done Done false
18184 6057 Acked Read 9 Sent Done Done false
18185 6058 Acked Read 8 Sent Done Done false
18186 6059 Acked Write 8 Sent Done Done false
18187 6060 Acked Write 5 Sent Done Done false
18188 6061 Acked Read 5 Sent Done Done false
18189 6062 Acked Write 2 Sent Done Done false
18190 6063 Acked Write 2 Sent Done Done false
18191 6064 Acked Read 9 Sent Done Done false
18192 6065 Acked Write 4 Sent Done Done false
18193 6066 Acked Read 4 Sent Done Done false
18194 6067 Acked Read 10 Sent Done Done false
18195 6068 Acked Write 8 Sent Done Done false
18196 6069 Acked Read 7 Sent Done Done false
18197 6070 Acked Flush 0 Sent Done Done false
18198 6071 Acked Read 9 Sent Done Done false
18199 6072 Acked Write 3 Sent Done Done false
18200 6073 Acked Write 10 Sent Done Done false
18201 6074 Acked Write 6 Sent Done Done false
18202 6075 Acked Write 6 Sent Done Done false
18203 6076 Acked Write 6 Sent Done Done false
18204 6077 Acked Read 10 Sent Done Done false
18205 6078 Acked Write 4 Sent Done Done false
18206 6079 Acked Read 2 Sent Done Done false
18207 6080 Acked Write 9 Sent Done Done false
18208 6081 Acked Read 9 Sent Done Done false
18209 6082 Acked Write 3 Sent Done Done false
18210 6083 Acked Read 1 Sent Done Done false
18211 6084 Acked Read 7 Sent Done Done false
18212 6085 Acked Read 8 Sent Done Done false
18213 6086 Acked Flush 0 Sent Done Done false
18214 6087 Acked Write 5 Sent Done Done false
18215 6088 Acked Flush 0 Sent Done Done false
18216 6089 Acked Read 8 Sent Done Done false
18217 6090 Acked Write 8 Sent Done Done false
18218 6091 Acked Write 8 Sent Done Done false
18219 6092 Acked Read 10 Sent Done Done false
18220 6093 Acked Read 1 Sent Done Done false
18221 6094 Acked Flush 0 Sent Done Done false
18222 6095 Acked Write 1 Sent Done Done false
18223 6096 Acked Read 7 Sent Done Done false
18224 6097 Acked Read 5 Sent Done Done false
18225 6098 Acked Write 10 Sent Done Done false
18226 6099 Acked Flush 0 Sent Done Done false
18227 6100 Acked Write 10 Sent Done Done false
18228 6101 Acked Read 6 Sent Done Done false
18229 6102 Acked Flush 0 Sent Done Done false
18230 6103 Acked Write 8 Sent Done Done false
18231 6104 Acked Read 3 Sent Done Done false
18232 6105 Acked Write 2 Sent Done Done false
18233 6106 Acked Read 9 Sent Done Done false
18234 6107 Acked Read 8 Sent Done Done false
18235 6108 Acked Flush 0 Sent Done Done false
18236 6109 Acked Read 1 Sent Done Done false
18237 6110 Acked Read 3 Sent Done Done false
18238 6111 Acked Read 3 Sent Done Done false
18239 6112 Acked Read 4 Sent Done Done false
18240 6113 Acked Flush 0 Sent Done Done false
18241 6114 Acked Write 2 Sent Done Done false
18242 6115 Acked Read 1 Sent Done Done false
18243 6116 Acked Write 4 Sent Done Done false
18244 6117 Acked Write 5 Sent Done Done false
18245 6118 Acked Flush 0 Sent Done Done false
18246 6119 Acked Read 3 Sent Done Done false
18247 6120 Acked Read 2 Sent Done Done false
18248 6121 Acked Read 10 Sent Done Done false
18249 6122 Acked Write 6 Sent Done Done false
18250 6123 Acked Read 3 Sent Done Done false
18251 6124 Acked Write 4 Sent Done Done false
18252 6125 Acked Write 5 Sent Done Done false
18253 6126 Acked Read 4 Sent Done Done false
18254 6127 Acked Read 6 Sent Done Done false
18255 6128 Acked Read 9 Sent Done Done false
18256 6129 Acked Read 9 Sent Done Done false
18257 6130 Acked Write 6 Sent Done Done false
18258 6131 Acked Write 10 Sent Done Done false
18259 6132 Acked Read 6 Sent Done Done false
18260 6133 Acked Flush 0 Sent Done Done false
18261 STATES DS:0 DS:1 DS:2 TOTAL
18262 Sent 302 0 0 302
18263 Done 0 302 302 604
18264 Skipped 0 0 0 0
18265 Error 0 0 0 0
18266 Last Flush: Some(JobId(5831)) Some(JobId(6133)) Some(JobId(6133))
18267 Downstairs last five completed: 5831 5830 5829 5828 5827
18268 Upstairs last five completed: 6133 6132 6131 6130 6129
18269 CLIENT: Up:0 ds:302 act:2
18270 [8810] got start action
18271 [8810] I am going to restart
18272 Starting downstairs at port 8810
18273 Make output file at "/tmp/dsc/downstairs-8810.txt"
18274 [8810][0] reports Starting
18275 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1222)
18276 [8810][0] reports Running
18277 ----------------------------------------------------------------
18278 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
18279 States: Offline Active Active
18280 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18281 5832 Acked Read 4 Sent Done Done false
18282 5833 Acked Read 4 Sent Done Done false
18283 5834 Acked Read 6 Sent Done Done false
18284 5835 Acked Flush 0 Sent Done Done false
18285 5836 Acked Read 8 Sent Done Done false
18286 5837 Acked Write 3 Sent Done Done false
18287 5838 Acked Write 7 Sent Done Done false
18288 5839 Acked Write 9 Sent Done Done false
18289 5840 Acked Write 1 Sent Done Done false
18290 5841 Acked Write 4 Sent Done Done false
18291 5842 Acked Write 6 Sent Done Done false
18292 5843 Acked Read 6 Sent Done Done false
18293 5844 Acked Read 10 Sent Done Done false
18294 5845 Acked Write 5 Sent Done Done false
18295 5846 Acked Write 3 Sent Done Done false
18296 5847 Acked Write 5 Sent Done Done false
18297 5848 Acked Write 4 Sent Done Done false
18298 5849 Acked Write 10 Sent Done Done false
18299 5850 Acked Read 2 Sent Done Done false
18300 5851 Acked Read 8 Sent Done Done false
18301 5852 Acked Read 9 Sent Done Done false
18302 5853 Acked Read 8 Sent Done Done false
18303 5854 Acked Read 8 Sent Done Done false
18304 5855 Acked Write 6 Sent Done Done false
18305 5856 Acked Read 1 Sent Done Done false
18306 5857 Acked Read 7 Sent Done Done false
18307 5858 Acked Read 5 Sent Done Done false
18308 5859 Acked Write 5 Sent Done Done false
18309 5860 Acked Write 7 Sent Done Done false
18310 5861 Acked Write 6 Sent Done Done false
18311 5862 Acked Flush 0 Sent Done Done false
18312 5863 Acked Write 7 Sent Done Done false
18313 5864 Acked Flush 0 Sent Done Done false
18314 5865 Acked Write 1 Sent Done Done false
18315 5866 Acked Write 3 Sent Done Done false
18316 5867 Acked Flush 0 Sent Done Done false
18317 5868 Acked Write 5 Sent Done Done false
18318 5869 Acked Write 2 Sent Done Done false
18319 5870 Acked Write 5 Sent Done Done false
18320 5871 Acked Read 9 Sent Done Done false
18321 5872 Acked Read 4 Sent Done Done false
18322 5873 Acked Write 5 Sent Done Done false
18323 5874 Acked Write 4 Sent Done Done false
18324 5875 Acked Read 5 Sent Done Done false
18325 5876 Acked Flush 0 Sent Done Done false
18326 5877 Acked Read 8 Sent Done Done false
18327 5878 Acked Read 6 Sent Done Done false
18328 5879 Acked Write 3 Sent Done Done false
18329 5880 Acked Read 3 Sent Done Done false
18330 5881 Acked Write 8 Sent Done Done false
18331 5882 Acked Read 3 Sent Done Done false
18332 5883 Acked Read 7 Sent Done Done false
18333 5884 Acked Read 6 Sent Done Done false
18334 5885 Acked Read 3 Sent Done Done false
18335 5886 Acked Write 3 Sent Done Done false
18336 5887 Acked Read 7 Sent Done Done false
18337 5888 Acked Read 9 Sent Done Done false
18338 5889 Acked Write 9 Sent Done Done false
18339 5890 Acked Read 6 Sent Done Done false
18340 5891 Acked Write 2 Sent Done Done false
18341 5892 Acked Write 1 Sent Done Done false
18342 5893 Acked Write 5 Sent Done Done false
18343 5894 Acked Write 10 Sent Done Done false
18344 5895 Acked Read 6 Sent Done Done false
18345 5896 Acked Read 6 Sent Done Done false
18346 5897 Acked Read 7 Sent Done Done false
18347 5898 Acked Write 3 Sent Done Done false
18348 5899 Acked Write 3 Sent Done Done false
18349 5900 Acked Read 6 Sent Done Done false
18350 5901 Acked Write 9 Sent Done Done false
18351 5902 Acked Write 10 Sent Done Done false
18352 5903 Acked Write 1 Sent Done Done false
18353 5904 Acked Flush 0 Sent Done Done false
18354 5905 Acked Write 2 Sent Done Done false
18355 5906 Acked Write 3 Sent Done Done false
18356 5907 Acked Write 10 Sent Done Done false
18357 5908 Acked Flush 0 Sent Done Done false
18358 5909 Acked Read 8 Sent Done Done false
18359 5910 Acked Read 10 Sent Done Done false
18360 5911 Acked Read 10 Sent Done Done false
18361 5912 Acked Read 9 Sent Done Done false
18362 5913 Acked Write 6 Sent Done Done false
18363 5914 Acked Read 5 Sent Done Done false
18364 5915 Acked Write 5 Sent Done Done false
18365 5916 Acked Write 6 Sent Done Done false
18366 5917 Acked Write 10 Sent Done Done false
18367 5918 Acked Read 3 Sent Done Done false
18368 5919 Acked Write 6 Sent Done Done false
18369 5920 Acked Write 1 Sent Done Done false
18370 5921 Acked Write 7 Sent Done Done false
18371 5922 Acked Read 5 Sent Done Done false
18372 5923 Acked Write 8 Sent Done Done false
18373 5924 Acked Read 8 Sent Done Done false
18374 5925 Acked Read 2 Sent Done Done false
18375 5926 Acked Write 10 Sent Done Done false
18376 5927 Acked Write 5 Sent Done Done false
18377 5928 Acked Read 10 Sent Done Done false
18378 5929 Acked Read 1 Sent Done Done false
18379 5930 Acked Read 7 Sent Done Done false
18380 5931 Acked Write 6 Sent Done Done false
18381 5932 Acked Read 7 Sent Done Done false
18382 5933 Acked Read 5 Sent Done Done false
18383 5934 Acked Read 9 Sent Done Done false
18384 5935 Acked Flush 0 Sent Done Done false
18385 5936 Acked Write 6 Sent Done Done false
18386 5937 Acked Read 9 Sent Done Done false
18387 5938 Acked Read 4 Sent Done Done false
18388 5939 Acked Write 3 Sent Done Done false
18389 5940 Acked Flush 0 Sent Done Done false
18390 5941 Acked Read 8 Sent Done Done false
18391 5942 Acked Write 7 Sent Done Done false
18392 5943 Acked Read 5 Sent Done Done false
18393 5944 Acked Read 6 Sent Done Done false
18394 5945 Acked Read 5 Sent Done Done false
18395 5946 Acked Write 2 Sent Done Done false
18396 5947 Acked Read 1 Sent Done Done false
18397 5948 Acked Flush 0 Sent Done Done false
18398 5949 Acked Read 4 Sent Done Done false
18399 5950 Acked Read 5 Sent Done Done false
18400 5951 Acked Read 10 Sent Done Done false
18401 5952 Acked Flush 0 Sent Done Done false
18402 5953 Acked Write 2 Sent Done Done false
18403 5954 Acked Write 10 Sent Done Done false
18404 5955 Acked Flush 0 Sent Done Done false
18405 5956 Acked Read 1 Sent Done Done false
18406 5957 Acked Flush 0 Sent Done Done false
18407 5958 Acked Write 3 Sent Done Done false
18408 5959 Acked Read 8 Sent Done Done false
18409 5960 Acked Write 4 Sent Done Done false
18410 5961 Acked Flush 0 Sent Done Done false
18411 5962 Acked Read 5 Sent Done Done false
18412 5963 Acked Read 7 Sent Done Done false
18413 5964 Acked Read 7 Sent Done Done false
18414 5965 Acked Write 6 Sent Done Done false
18415 5966 Acked Read 3 Sent Done Done false
18416 5967 Acked Read 7 Sent Done Done false
18417 5968 Acked Read 7 Sent Done Done false
18418 5969 Acked Write 5 Sent Done Done false
18419 5970 Acked Read 5 Sent Done Done false
18420 5971 Acked Write 4 Sent Done Done false
18421 5972 Acked Read 6 Sent Done Done false
18422 5973 Acked Read 10 Sent Done Done false
18423 5974 Acked Read 3 Sent Done Done false
18424 5975 Acked Read 4 Sent Done Done false
18425 5976 Acked Read 9 Sent Done Done false
18426 5977 Acked Write 3 Sent Done Done false
18427 5978 Acked Read 9 Sent Done Done false
18428 5979 Acked Write 4 Sent Done Done false
18429 5980 Acked Read 1 Sent Done Done false
18430 5981 Acked Write 7 Sent Done Done false
18431 5982 Acked Read 7 Sent Done Done false
18432 5983 Acked Read 4 Sent Done Done false
18433 5984 Acked Write 6 Sent Done Done false
18434 5985 Acked Read 4 Sent Done Done false
18435 5986 Acked Write 6 Sent Done Done false
18436 5987 Acked Read 2 Sent Done Done false
18437 5988 Acked Write 1 Sent Done Done false
18438 5989 Acked Write 4 Sent Done Done false
18439 5990 Acked Write 5 Sent Done Done false
18440 5991 Acked Read 9 Sent Done Done false
18441 5992 Acked Read 7 Sent Done Done false
18442 5993 Acked Read 4 Sent Done Done false
18443 5994 Acked Read 9 Sent Done Done false
18444 5995 Acked Read 6 Sent Done Done false
18445 5996 Acked Write 7 Sent Done Done false
18446 5997 Acked Read 5 Sent Done Done false
18447 5998 Acked Read 9 Sent Done Done false
18448 5999 Acked Read 10 Sent Done Done false
18449 6000 Acked Read 6 Sent Done Done false
18450 6001 Acked Read 5 Sent Done Done false
18451 6002 Acked Read 8 Sent Done Done false
18452 6003 Acked Read 5 Sent Done Done false
18453 6004 Acked Write 9 Sent Done Done false
18454 6005 Acked Write 6 Sent Done Done false
18455 6006 Acked Write 4 Sent Done Done false
18456 6007 Acked Read 8 Sent Done Done false
18457 6008 Acked Write 3 Sent Done Done false
18458 6009 Acked Write 7 Sent Done Done false
18459 6010 Acked Write 1 Sent Done Done false
18460 6011 Acked Read 5 Sent Done Done false
18461 6012 Acked Flush 0 Sent Done Done false
18462 6013 Acked Write 5 Sent Done Done false
18463 6014 Acked Write 4 Sent Done Done false
18464 6015 Acked Write 9 Sent Done Done false
18465 6016 Acked Write 3 Sent Done Done false
18466 6017 Acked Read 7 Sent Done Done false
18467 6018 Acked Write 2 Sent Done Done false
18468 6019 Acked Flush 0 Sent Done Done false
18469 6020 Acked Write 3 Sent Done Done false
18470 6021 Acked Read 2 Sent Done Done false
18471 6022 Acked Write 6 Sent Done Done false
18472 6023 Acked Read 9 Sent Done Done false
18473 6024 Acked Read 7 Sent Done Done false
18474 6025 Acked Read 10 Sent Done Done false
18475 6026 Acked Read 1 Sent Done Done false
18476 6027 Acked Read 8 Sent Done Done false
18477 6028 Acked Write 10 Sent Done Done false
18478 6029 Acked Read 1 Sent Done Done false
18479 6030 Acked Flush 0 Sent Done Done false
18480 6031 Acked Read 8 Sent Done Done false
18481 6032 Acked Read 9 Sent Done Done false
18482 6033 Acked Write 6 Sent Done Done false
18483 6034 Acked Write 4 Sent Done Done false
18484 6035 Acked Write 5 Sent Done Done false
18485 6036 Acked Write 1 Sent Done Done false
18486 6037 Acked Flush 0 Sent Done Done false
18487 6038 Acked Flush 0 Sent Done Done false
18488 6039 Acked Flush 0 Sent Done Done false
18489 6040 Acked Read 2 Sent Done Done false
18490 6041 Acked Read 6 Sent Done Done false
18491 6042 Acked Read 8 Sent Done Done false
18492 6043 Acked Flush 0 Sent Done Done false
18493 6044 Acked Read 5 Sent Done Done false
18494 6045 Acked Read 9 Sent Done Done false
18495 6046 Acked Write 5 Sent Done Done false
18496 6047 Acked Write 7 Sent Done Done false
18497 6048 Acked Write 2 Sent Done Done false
18498 6049 Acked Read 6 Sent Done Done false
18499 6050 Acked Write 8 Sent Done Done false
18500 6051 Acked Write 10 Sent Done Done false
18501 6052 Acked Read 6 Sent Done Done false
18502 6053 Acked Read 1 Sent Done Done false
18503 6054 Acked Flush 0 Sent Done Done false
18504 6055 Acked Read 5 Sent Done Done false
18505 6056 Acked Write 7 Sent Done Done false
18506 6057 Acked Read 9 Sent Done Done false
18507 6058 Acked Read 8 Sent Done Done false
18508 6059 Acked Write 8 Sent Done Done false
18509 6060 Acked Write 5 Sent Done Done false
18510 6061 Acked Read 5 Sent Done Done false
18511 6062 Acked Write 2 Sent Done Done false
18512 6063 Acked Write 2 Sent Done Done false
18513 6064 Acked Read 9 Sent Done Done false
18514 6065 Acked Write 4 Sent Done Done false
18515 6066 Acked Read 4 Sent Done Done false
18516 6067 Acked Read 10 Sent Done Done false
18517 6068 Acked Write 8 Sent Done Done false
18518 6069 Acked Read 7 Sent Done Done false
18519 6070 Acked Flush 0 Sent Done Done false
18520 6071 Acked Read 9 Sent Done Done false
18521 6072 Acked Write 3 Sent Done Done false
18522 6073 Acked Write 10 Sent Done Done false
18523 6074 Acked Write 6 Sent Done Done false
18524 6075 Acked Write 6 Sent Done Done false
18525 6076 Acked Write 6 Sent Done Done false
18526 6077 Acked Read 10 Sent Done Done false
18527 6078 Acked Write 4 Sent Done Done false
18528 6079 Acked Read 2 Sent Done Done false
18529 6080 Acked Write 9 Sent Done Done false
18530 6081 Acked Read 9 Sent Done Done false
18531 6082 Acked Write 3 Sent Done Done false
18532 6083 Acked Read 1 Sent Done Done false
18533 6084 Acked Read 7 Sent Done Done false
18534 6085 Acked Read 8 Sent Done Done false
18535 6086 Acked Flush 0 Sent Done Done false
18536 6087 Acked Write 5 Sent Done Done false
18537 6088 Acked Flush 0 Sent Done Done false
18538 6089 Acked Read 8 Sent Done Done false
18539 6090 Acked Write 8 Sent Done Done false
18540 6091 Acked Write 8 Sent Done Done false
18541 6092 Acked Read 10 Sent Done Done false
18542 6093 Acked Read 1 Sent Done Done false
18543 6094 Acked Flush 0 Sent Done Done false
18544 6095 Acked Write 1 Sent Done Done false
18545 6096 Acked Read 7 Sent Done Done false
18546 6097 Acked Read 5 Sent Done Done false
18547 6098 Acked Write 10 Sent Done Done false
18548 6099 Acked Flush 0 Sent Done Done false
18549 6100 Acked Write 10 Sent Done Done false
18550 6101 Acked Read 6 Sent Done Done false
18551 6102 Acked Flush 0 Sent Done Done false
18552 6103 Acked Write 8 Sent Done Done false
18553 6104 Acked Read 3 Sent Done Done false
18554 6105 Acked Write 2 Sent Done Done false
18555 6106 Acked Read 9 Sent Done Done false
18556 6107 Acked Read 8 Sent Done Done false
18557 6108 Acked Flush 0 Sent Done Done false
18558 6109 Acked Read 1 Sent Done Done false
18559 6110 Acked Read 3 Sent Done Done false
18560 6111 Acked Read 3 Sent Done Done false
18561 6112 Acked Read 4 Sent Done Done false
18562 6113 Acked Flush 0 Sent Done Done false
18563 6114 Acked Write 2 Sent Done Done false
18564 6115 Acked Read 1 Sent Done Done false
18565 6116 Acked Write 4 Sent Done Done false
18566 6117 Acked Write 5 Sent Done Done false
18567 6118 Acked Flush 0 Sent Done Done false
18568 6119 Acked Read 3 Sent Done Done false
18569 6120 Acked Read 2 Sent Done Done false
18570 6121 Acked Read 10 Sent Done Done false
18571 6122 Acked Write 6 Sent Done Done false
18572 6123 Acked Read 3 Sent Done Done false
18573 6124 Acked Write 4 Sent Done Done false
18574 6125 Acked Write 5 Sent Done Done false
18575 6126 Acked Read 4 Sent Done Done false
18576 6127 Acked Read 6 Sent Done Done false
18577 6128 Acked Read 9 Sent Done Done false
18578 6129 Acked Read 9 Sent Done Done false
18579 6130 Acked Write 6 Sent Done Done false
18580 6131 Acked Write 10 Sent Done Done false
18581 6132 Acked Read 6 Sent Done Done false
18582 6133 Acked Flush 0 Sent Done Done false
18583 STATES DS:0 DS:1 DS:2 TOTAL
18584 Sent 302 0 0 302
18585 Done 0 302 302 604
18586 Skipped 0 0 0 0
18587 Error 0 0 0 0
18588 Last Flush: Some(JobId(5831)) Some(JobId(6133)) Some(JobId(6133))
18589 Downstairs last five completed: 5831 5830 5829 5828 5827
18590 Upstairs last five completed: 6133 6132 6131 6130 6129
18591 CLIENT: Up:0 ds:302 act:2
185922025-01-23T17:41:39.644ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
185932025-01-23T17:41:39.644ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49067) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
185942025-01-23T17:41:39.645ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
185952025-01-23T17:41:39.646ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
185962025-01-23T17:41:39.646ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
185972025-01-23T17:41:39.646ZINFOcrucible: send last flush ID to this DS: Some(JobId(5831)) = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
185982025-01-23T17:41:39.646ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
185992025-01-23T17:41:39.646ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(5831)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
186002025-01-23T17:41:39.646ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(5831)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
18601 ----------------------------------------------------------------
18602 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18603 Downstairs last five completed: 6133 6132 6131 6130 6129
18604 Upstairs last five completed: 6133 6132 6131 6130 6129
18605 CLIENT: Up:0 ds:0 act:3
18606 Replay: All jobs finished
18607 Main task has work to do, go find it
18608 got dsc Stop(1)
18609 stop 1
18610 [8820] Got stop action so:false kr:false
18611 Jan 23 17:41:39.899 INFO request completed, latency_us: 389, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 5e9c51bd-2356-43fa-9c4b-86c9e4cd7c0c, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
186122025-01-23T17:41:39.899ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
186132025-01-23T17:41:39.899ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
186142025-01-23T17:41:39.899ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
186152025-01-23T17:41:39.900ZINFOcrucible: client task is exiting = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
18616 Jan 23 17:41:39.900 INFO request completed, latency_us: 194, response_code: 200, uri: /state/cid/1, method: GET, req_id: 31563e64-31dc-48c9-b94d-ad32acf06f9e, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
18617 [8820] Exited with: ExitStatus(unix_wait_status(9))
18618 [8820][1] reports Exit
18619 Jan 23 17:41:43.902 INFO request completed, latency_us: 226, response_code: 200, uri: /state/cid/1, method: GET, req_id: 983aa45a-b06c-465f-9b5f-60e292942f49, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
18620 001/300 Read block 499 len 36864
18621 002/300 Read block 1537 len 40960
18622 003/300 Read block 2411 len 8192
18623 004/300 Write block 1608 len 36864 data: 3 3 4 5 6 6 5 6 7
18624 005/300 Write block 989 len 40960 data: 6 5 5 5 4 4 6 7 7 6
18625 006/300 Write block 3057 len 32768 data: 3 3 3 3 3 3 3 2
18626 007/300 Read block 4271 len 16384
18627 008/300 Read block 767 len 28672
18628 009/300 Read block 942 len 36864
18629 010/300 Write block 2136 len 20480 data: 4 4 4 4 4
18630 011/300 Write block 1918 len 40960 data: 6 6 8 7 7 8 10 8 6 5
18631 012/300 Write block 948 len 32768 data: 2 2 2 2 2 2 3 3
18632 013/300 Flush
18633 014/300 Flush
18634 015/300 Write block 1652 len 16384 data: 6 6 4 5
18635 016/300 Flush
18636 017/300 Write block 3662 len 32768 data: 4 5 4 6 6 7 7 8
18637 018/300 Read block 4015 len 36864
18638 019/300 Write block 2779 len 20480 data: 2 2 2 3 3
18639 020/300 Write block 4155 len 24576 data: 5 4 4 3 2 2
18640 021/300 Write block 3649 len 8192 data: 5 5
18641 022/300 Write block 783 len 40960 data: 6 7 7 5 3 2 2 2 2 2
18642 023/300 Read block 4668 len 32768
18643 024/300 Write block 72 len 40960 data: 9 9 10 8 7 6 6 6 7 5
18644 025/300 Write block 1454 len 32768 data: 3 3 3 3 3 3 3 3
18645 026/300 Write block 875 len 32768 data: 3 4 4 3 3 4 5 3
18646 027/300 Read block 3165 len 4096
18647 028/300 Write block 3947 len 4096 data: 5
18648 029/300 Read block 1097 len 28672
18649 030/300 Read block 1484 len 24576
18650 031/300 Flush
18651 032/300 Read block 475 len 36864
18652 033/300 Write block 3262 len 4096 data: 2
18653 034/300 Read block 558 len 20480
18654 035/300 Write block 2298 len 16384 data: 3 2 2 3
18655 036/300 Write block 2962 len 24576 data: 5 4 6 7 4 4
18656 037/300 Read block 4755 len 20480
18657 038/300 Read block 3246 len 8192
18658 039/300 Read block 4590 len 28672
18659 040/300 Read block 1010 len 24576
18660 041/300 Read block 4988 len 4096
18661 042/300 Read block 2202 len 4096
18662 043/300 Write block 2585 len 4096 data: 3
18663 044/300 Read block 2490 len 16384
18664 045/300 Write block 4751 len 20480 data: 2 2 2 2 2
18665 046/300 Write block 4794 len 8192 data: 3 3
18666 047/300 Read block 2545 len 4096
18667 048/300 Read block 2962 len 12288
18668 049/300 Write block 2073 len 8192 data: 5 7
18669 050/300 Read block 3478 len 20480
18670 051/300 Read block 3969 len 4096
18671 052/300 Write block 4485 len 4096 data: 2
18672 053/300 Read block 809 len 4096
18673 054/300 Write block 3027 len 32768 data: 5 4 3 3 5 6 7 6
18674 055/300 Read block 772 len 4096
18675 056/300 Write block 702 len 40960 data: 3 3 2 3 4 4 4 3 3 3
18676 057/300 Write block 4970 len 20480 data: 5 8 9 6 5
18677 058/300 Read block 876 len 40960
18678 059/300 Read block 4622 len 40960
18679 060/300 Read block 1114 len 40960
18680 061/300 Write block 491 len 36864 data: 2 2 3 3 2 3 3 3 3
18681 062/300 Read block 2384 len 40960
18682 063/300 Write block 3386 len 4096 data: 4
18683 064/300 Read block 4299 len 40960
18684 065/300 Flush
18685 066/300 Write block 519 len 20480 data: 3 3 4 5 4
18686 067/300 Write block 784 len 28672 data: 8 8 6 4 3 3 3
18687 068/300 Read block 517 len 36864
18688 069/300 Write block 2251 len 8192 data: 5 6
18689 070/300 Write block 3480 len 16384 data: 4 4 4 4
18690 071/300 Read block 3017 len 36864
18691 072/300 Write block 990 len 16384 data: 6 6 6 5
18692 073/300 Write block 2611 len 28672 data: 5 6 5 6 6 6 6
18693 074/300 Write block 1919 len 32768 data: 7 9 8 8 9 11 9 7
18694 075/300 Read block 834 len 20480
18695 076/300 Write block 96 len 32768 data: 3 3 4 4 5 4 6 5
18696 077/300 Write block 1581 len 4096 data: 2
18697 078/300 Write block 4699 len 28672 data: 5 6 5 5 6 6 5
18698 079/300 Flush
18699 080/300 Read block 1551 len 36864
18700 081/300 Read block 2255 len 20480
18701 082/300 Read block 742 len 24576
18702 083/300 Write block 4528 len 12288 data: 3 3 3
18703 084/300 Read block 966 len 20480
18704 085/300 Write block 2409 len 20480 data: 4 4 4 5 8
18705 086/300 Read block 1128 len 12288
18706 087/300 Read block 4401 len 4096
18707 088/300 Read block 1998 len 24576
18708 089/300 Read block 4976 len 16384
18709 090/300 Read block 3232 len 4096
18710 091/300 Read block 3336 len 24576
18711 092/300 Read block 4476 len 16384
18712 093/300 Write block 649 len 36864 data: 3 3 4 4 4 5 5 5 6
18713 094/300 Write block 715 len 40960 data: 4 5 3 4 3 3 3 4 4 4
18714 095/300 Read block 2125 len 40960
18715 096/300 Write block 4480 len 36864 data: 3 3 3 3 2 3 2 3 3
18716 097/300 Write block 549 len 12288 data: 6 5 5
18717 098/300 Read block 2503 len 8192
18718 099/300 Write block 4301 len 20480 data: 3 3 4 4 4
18719 100/300 Flush
18720 101/300 Flush
18721 102/300 Read block 2543 len 20480
18722 103/300 Read block 3077 len 32768
18723 104/300 Flush
18724 105/300 Read block 3537 len 24576
18725 106/300 Write block 3916 len 36864 data: 5 5 3 3 5 5 4 4 4
18726 107/300 Write block 2069 len 24576 data: 6 5 5 5 6 8
18727 108/300 Read block 2932 len 40960
18728 109/300 Write block 4864 len 24576 data: 6 5 5 6 6 8
18729 110/300 Write block 2498 len 40960 data: 5 6 4 5 6 6 5 5 4 2
18730 111/300 Write block 3135 len 36864 data: 3 3 3 3 4 5 4 5 5
18731 112/300 Read block 2764 len 28672
18732 113/300 Read block 3292 len 24576
18733 114/300 Write block 1976 len 24576 data: 5 6 6 5 5 4
18734 115/300 Read block 2931 len 40960
18735 116/300 Read block 1853 len 12288
18736 117/300 Write block 1012 len 40960 data: 4 3 2 2 2 2 2 3 3 5
18737 118/300 Write block 2630 len 36864 data: 6 6 5 4 4 4 4 3 3
18738 119/300 Read block 4220 len 20480
18739 120/300 Write block 3662 len 16384 data: 5 6 5 7
18740 121/300 Write block 2931 len 32768 data: 3 3 3 3 3 3 4 5
18741 122/300 Read block 2839 len 28672
18742 123/300 Write block 2395 len 28672 data: 7 9 10 11 10 10 7
18743 124/300 Read block 4080 len 16384
18744 125/300 Write block 2734 len 4096 data: 4
18745 126/300 Read block 2043 len 16384
18746 127/300 Read block 304 len 28672
18747 128/300 Write block 3156 len 4096 data: 4
18748 129/300 Write block 3562 len 40960 data: 3 3 4 4 4 3 5 5 4 4
18749 130/300 Write block 1232 len 40960 data: 3 4 4 5 6 6 4 5 5 4
18750 131/300 Read block 1892 len 40960
18751 132/300 Write block 615 len 20480 data: 5 6 5 5 6
18752 133/300 Write block 1553 len 32768 data: 4 5 4 3 3 3 4 5
18753 134/300 Write block 609 len 40960 data: 3 4 4 4 4 4 6 7 6 6
18754 135/300 Read block 4551 len 16384
18755 136/300 Write block 3974 len 36864 data: 3 2 2 2 2 2 2 2 5
18756 137/300 Read block 2484 len 12288
18757 138/300 Read block 4744 len 20480
18758 139/300 Read block 1668 len 20480
18759 140/300 Read block 1270 len 12288
18760 141/300 Write block 2868 len 16384 data: 7 7 5 4
18761 142/300 Flush
18762 143/300 Flush
18763 144/300 Read block 2128 len 12288
18764 145/300 Read block 304 len 12288
18765 146/300 Write block 4031 len 12288 data: 4 5 5
18766 147/300 Read block 2848 len 20480
18767 148/300 Write block 4048 len 32768 data: 5 6 4 4 4 4 5 4
18768 149/300 Read block 4844 len 28672
18769 150/300 Read block 638 len 20480
18770 151/300 Read block 757 len 16384
18771 152/300 Read block 589 len 32768
18772 153/300 Flush
18773 154/300 Read block 2629 len 16384
18774 155/300 Flush
18775 156/300 Read block 952 len 40960
18776 157/300 Read block 3212 len 28672
18777 158/300 Read block 1797 len 16384
18778 159/300 Write block 4771 len 36864 data: 3 2 3 3 3 3 5 4 5
18779 160/300 Read block 2760 len 32768
18780 161/300 Write block 4162 len 16384 data: 2 2 2 2
18781 162/300 Write block 1105 len 4096 data: 2
18782 163/300 Read block 4563 len 24576
18783 164/300 Write block 1876 len 20480 data: 4 4 4 3 4
18784 165/300 Read block 2926 len 24576
18785 166/300 Read block 2351 len 24576
18786 167/300 Read block 3791 len 4096
18787 168/300 Write block 584 len 40960 data: 10 11 7 6 5 5 5 3 4 4
18788 169/300 Write block 2351 len 40960 data: 7 5 6 5 6 6 6 4 4 4
18789 170/300 Read block 30 len 12288
18790 171/300 Write block 1688 len 36864 data: 6 5 5 5 5 6 5 5 2
18791 172/300 Read block 4499 len 28672
18792 173/300 Write block 4575 len 24576 data: 5 6 5 7 6 5
18793 174/300 Write block 747 len 20480 data: 6 7 7 7 5
18794 175/300 Read block 2318 len 24576
18795 176/300 Flush
18796 177/300 Write block 4641 len 20480 data: 4 5 4 4 4
18797 178/300 Read block 2011 len 28672
18798 179/300 Read block 2436 len 20480
18799 180/300 Read block 413 len 32768
18800 181/300 Flush
18801 182/300 Read block 2222 len 36864
18802 183/300 Write block 3907 len 20480 data: 4 4 4 4 5
18803 184/300 Read block 3792 len 40960
18804 185/300 Write block 240 len 16384 data: 4 5 5 4
18805 186/300 Read block 1908 len 4096
18806 187/300 Read block 3642 len 36864
18807 188/300 Write block 3019 len 28672 data: 6 4 4 4 3 4 3
18808 189/300 Read block 1300 len 32768
18809 190/300 Write block 4605 len 20480 data: 5 5 5 3 4
18810 191/300 Write block 3402 len 40960 data: 5 4 3 4 4 4 3 5 5 4
18811 192/300 Write block 4213 len 40960 data: 4 4 4 4 4 2 2 2 3 3
18812 193/300 Write block 3600 len 8192 data: 5 5
18813 194/300 Read block 4693 len 24576
18814 195/300 Read block 1031 len 32768
18815 196/300 Read block 2922 len 20480
18816 197/300 Read block 1989 len 20480
18817 198/300 Read block 1930 len 12288
18818 199/300 Read block 1686 len 40960
18819 200/300 Write block 1077 len 12288 data: 2 2 2
18820 201/300 Read block 1070 len 36864
18821 202/300 Write block 3922 len 16384 data: 5 5 5 3
18822 203/300 Write block 4540 len 24576 data: 3 3 3 3 3 3
18823 204/300 Write block 1513 len 24576 data: 5 5 5 4 4 4
18824 205/300 Read block 1626 len 28672
18825 206/300 Read block 889 len 12288
18826 207/300 Flush
18827 208/300 Write block 4020 len 12288 data: 4 4 3
18828 209/300 Read block 3498 len 36864
18829 210/300 Read block 185 len 32768
18830 211/300 Read block 960 len 12288
18831 212/300 Write block 1706 len 36864 data: 3 2 2 2 4 5 5 4 3
18832 213/300 Read block 3410 len 24576
18833 214/300 Write block 3729 len 40960 data: 4 3 3 5 4 3 4 4 4 4
18834 215/300 Read block 3398 len 36864
18835 216/300 Read block 1234 len 24576
18836 217/300 Read block 878 len 4096
18837 218/300 Write block 383 len 24576 data: 2 3 4 4 4 5
18838 219/300 Read block 4624 len 24576
18839 220/300 Read block 2622 len 28672
18840 221/300 Write block 2988 len 20480 data: 5 4 5 6 7
18841 222/300 Read block 2134 len 28672
18842 223/300 Read block 1190 len 16384
18843 224/300 Read block 807 len 4096
18844 225/300 Read block 4467 len 28672
18845 226/300 Read block 622 len 16384
18846 227/300 Read block 278 len 28672
18847 228/300 Read block 1020 len 4096
18848 229/300 Read block 3982 len 4096
18849 230/300 Read block 2042 len 8192
18850 231/300 Write block 2739 len 32768 data: 5 4 4 4 4 4 6 8
18851 232/300 Read block 4474 len 40960
18852 233/300 Write block 1872 len 16384 data: 4 5 6 5
18853 234/300 Read block 2007 len 20480
18854 235/300 Read block 2388 len 32768
18855 236/300 Write block 4756 len 4096 data: 2
18856 237/300 Write block 833 len 40960 data: 5 5 5 5 5 5 6 3 2 3
18857 238/300 Read block 3242 len 16384
18858 239/300 Read block 407 len 28672
18859 240/300 Write block 892 len 40960 data: 3 3 2 2 2 2 2 2 3 4
18860 241/300 Write block 2805 len 28672 data: 5 5 4 5 4 4 5
18861 242/300 Read block 3582 len 32768
18862 243/300 Read block 827 len 32768
18863 244/300 Write block 3893 len 12288 data: 4 4 3
18864 245/300 Read block 2070 len 40960
18865 246/300 Read block 3031 len 12288
18866 247/300 Read block 1234 len 32768
18867 248/300 Read block 797 len 40960
18868 249/300 Read block 679 len 36864
18869 250/300 Write block 3635 len 36864 data: 2 3 3 2 2 2 2 4 4
18870 251/300 Flush
18871 252/300 Read block 2223 len 40960
18872 253/300 Read block 4460 len 16384
18873 254/300 Read block 1561 len 12288
18874 255/300 Flush
18875 256/300 Read block 4984 len 32768
18876 257/300 Flush
18877 258/300 Write block 2167 len 20480 data: 3 3 3 3 3
18878 259/300 Read block 2386 len 40960
18879 260/300 Write block 4533 len 36864 data: 4 5 5 4 5 5 5 4 4
18880 261/300 Write block 3726 len 16384 data: 5 4 4 5
18881 262/300 Write block 3577 len 8192 data: 3 4
18882 263/300 Write block 1276 len 20480 data: 6 6 7 7 5
18883 264/300 Read block 4968 len 8192
18884 265/300 Read block 3824 len 32768
18885 266/300 Read block 4388 len 32768
18886 267/300 Flush
18887 268/300 Write block 4710 len 40960 data: 5 3 3 4 3 2 2 2 2 2
18888 269/300 Read block 2787 len 20480
18889 270/300 Flush
18890 271/300 Read block 568 len 32768
18891 272/300 Read block 2040 len 12288
18892 273/300 Write block 4406 len 36864 data: 4 3 2 3 3 3 4 4 4
18893 274/300 Read block 2695 len 36864
18894 275/300 Write block 3638 len 4096 data: 3
18895 276/300 Read block 1610 len 28672
18896 277/300 Write block 3444 len 24576 data: 6 4 4 4 3 3
18897 278/300 Read block 812 len 24576
18898 279/300 Flush
18899 280/300 Write block 2810 len 36864 data: 5 6 5 5 5 6 5 5 4
18900 281/300 Read block 2450 len 40960
18901 282/300 Write block 429 len 36864 data: 2 3 4 3 3 3 3 3 4
18902 283/300 Read block 4566 len 40960
18903 284/300 Write block 1835 len 36864 data: 7 8 8 6 5 4 3 3 3
18904 285/300 Flush
18905 286/300 Read block 2347 len 12288
18906 287/300 Flush
18907 288/300 Read block 2211 len 20480
18908 289/300 Read block 4198 len 28672
18909 290/300 Write block 4249 len 24576 data: 3 3 3 2 2 4
18910 291/300 Read block 4558 len 8192
18911 292/300 Read block 595 len 24576
18912 293/300 Write block 4497 len 32768 data: 6 6 6 5 5 5 5 6
18913 294/300 Read block 3110 len 24576
18914 295/300 Write block 3578 len 32768 data: 5 6 6 6 6 6 6 6
18915 296/300 Read block 3588 len 24576
18916 297/300 Read block 399 len 8192
18917 298/300 Read block 4611 len 28672
18918 299/300 Write block 541 len 4096 data: 7
18919 300/300 Read block 1537 len 20480
18920 301/300 Write block 668 len 20480 data: 6 6 7 7 8
18921 Main task has work to do, go find it
18922 got dsc Start(1)
18923 start 1
18924 Jan 23 17:41:44.160 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/1, method: POST, req_id: caab8366-00a2-4919-92fa-934df654802a, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
18925 Replay: started 1, returned:Ok(())
18926 ----------------------------------------------------------------
18927 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18928 States: Active Offline Active
18929 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18930 6134 Acked Read 9 Done Sent Done false
18931 6135 Acked Read 10 Done Sent Done false
18932 6136 Acked Read 2 Done Sent Done false
18933 6137 Acked Write 9 Done Sent Done false
18934 6138 Acked Write 10 Done Sent Done false
18935 6139 Acked Write 8 Done Sent Done false
18936 6140 Acked Read 4 Done Sent Done false
18937 6141 Acked Read 7 Done Sent Done false
18938 6142 Acked Read 9 Done Sent Done false
18939 6143 Acked Write 5 Done Sent Done false
18940 6144 Acked Write 10 Done Sent Done false
18941 6145 Acked Write 8 Done Sent Done false
18942 6146 Acked Flush 0 Done Sent Done false
18943 6147 Acked Flush 0 Done Sent Done false
18944 6148 Acked Write 4 Done Sent Done false
18945 6149 Acked Flush 0 Done Sent Done false
18946 6150 Acked Write 8 Done Sent Done false
18947 6151 Acked Read 9 Done Sent Done false
18948 6152 Acked Write 5 Done Sent Done false
18949 6153 Acked Write 6 Done Sent Done false
18950 6154 Acked Write 2 Done Sent Done false
18951 6155 Acked Write 10 Done Sent Done false
18952 6156 Acked Read 8 Done Sent Done false
18953 6157 Acked Write 10 Done Sent Done false
18954 6158 Acked Write 8 Done Sent Done false
18955 6159 Acked Write 8 Done Sent Done false
18956 6160 Acked Read 1 Done Sent Done false
18957 6161 Acked Write 1 Done Sent Done false
18958 6162 Acked Read 7 Done Sent Done false
18959 6163 Acked Read 6 Done Sent Done false
18960 6164 Acked Flush 0 Done Sent Done false
18961 6165 Acked Read 9 Done Sent Done false
18962 6166 Acked Write 1 Done Sent Done false
18963 6167 Acked Read 5 Done Sent Done false
18964 6168 Acked Write 4 Done Sent Done false
18965 6169 Acked Write 6 Done Sent Done false
18966 6170 Acked Read 5 Done Sent Done false
18967 6171 Acked Read 2 Done Sent Done false
18968 6172 Acked Read 7 Done Sent Done false
18969 6173 Acked Read 6 Done Sent Done false
18970 6174 Acked Read 1 Done Sent Done false
18971 6175 Acked Read 1 Done Sent Done false
18972 6176 Acked Write 1 Done Sent Done false
18973 6177 Acked Read 4 Done Sent Done false
18974 6178 Acked Write 5 Done Sent Done false
18975 6179 Acked Write 2 Done Sent Done false
18976 6180 Acked Read 1 Done Sent Done false
18977 6181 Acked Read 3 Done Sent Done false
18978 6182 Acked Write 2 Done Sent Done false
18979 6183 Acked Read 5 Done Sent Done false
18980 6184 Acked Read 1 Done Sent Done false
18981 6185 Acked Write 1 Done Sent Done false
18982 6186 Acked Read 1 Done Sent Done false
18983 6187 Acked Write 8 Done Sent Done false
18984 6188 Acked Read 1 Done Sent Done false
18985 6189 Acked Write 10 Done Sent Done false
18986 6190 Acked Write 5 Done Sent Done false
18987 6191 Acked Read 10 Done Sent Done false
18988 6192 Acked Read 10 Done Sent Done false
18989 6193 Acked Read 10 Done Sent Done false
18990 6194 Acked Write 9 Done Sent Done false
18991 6195 Acked Read 10 Done Sent Done false
18992 6196 Acked Write 1 Done Sent Done false
18993 6197 Acked Read 10 Done Sent Done false
18994 6198 Acked Flush 0 Done Sent Done false
18995 6199 Acked Write 5 Done Sent Done false
18996 6200 Acked Write 7 Done Sent Done false
18997 6201 Acked Read 9 Done Sent Done false
18998 6202 Acked Write 2 Done Sent Done false
18999 6203 Acked Write 4 Done Sent Done false
19000 6204 Acked Read 9 Done Sent Done false
19001 6205 Acked Write 4 Done Sent Done false
19002 6206 Acked Write 7 Done Sent Done false
19003 6207 Acked Write 8 Done Sent Done false
19004 6208 Acked Read 5 Done Sent Done false
19005 6209 Acked Write 8 Done Sent Done false
19006 6210 Acked Write 1 Done Sent Done false
19007 6211 Acked Write 7 Done Sent Done false
19008 6212 Acked Flush 0 Done Sent Done false
19009 6213 Acked Read 9 Done Sent Done false
19010 6214 Acked Read 5 Done Sent Done false
19011 6215 Acked Read 6 Done Sent Done false
19012 6216 Acked Write 3 Done Sent Done false
19013 6217 Acked Read 5 Done Sent Done false
19014 6218 Acked Write 5 Done Sent Done false
19015 6219 Acked Read 3 Done Sent Done false
19016 6220 Acked Read 1 Done Sent Done false
19017 6221 Acked Read 6 Done Sent Done false
19018 6222 Acked Read 4 Done Sent Done false
19019 6223 Acked Read 1 Done Sent Done false
19020 6224 Acked Read 6 Done Sent Done false
19021 6225 Acked Read 4 Done Sent Done false
19022 6226 Acked Write 9 Done Sent Done false
19023 6227 Acked Write 10 Done Sent Done false
19024 6228 Acked Read 10 Done Sent Done false
19025 6229 Acked Write 9 Done Sent Done false
19026 6230 Acked Write 3 Done Sent Done false
19027 6231 Acked Read 2 Done Sent Done false
19028 6232 Acked Write 5 Done Sent Done false
19029 6233 Acked Flush 0 Done Sent Done false
19030 6234 Acked Flush 0 Done Sent Done false
19031 6235 Acked Read 5 Done Sent Done false
19032 6236 Acked Read 8 Done Sent Done false
19033 6237 Acked Flush 0 Done Sent Done false
19034 6238 Acked Read 6 Done Sent Done false
19035 6239 Acked Write 9 Done Sent Done false
19036 6240 Acked Write 6 Done Sent Done false
19037 6241 Acked Read 10 Done Sent Done false
19038 6242 Acked Write 6 Done Sent Done false
19039 6243 Acked Write 10 Done Sent Done false
19040 6244 Acked Write 9 Done Sent Done false
19041 6245 Acked Read 7 Done Sent Done false
19042 6246 Acked Read 6 Done Sent Done false
19043 6247 Acked Write 6 Done Sent Done false
19044 6248 Acked Read 10 Done Sent Done false
19045 6249 Acked Read 3 Done Sent Done false
19046 6250 Acked Write 10 Done Sent Done false
19047 6251 Acked Write 9 Done Sent Done false
19048 6252 Acked Read 5 Done Sent Done false
19049 6253 Acked Write 4 Done Sent Done false
19050 6254 Acked Write 8 Done Sent Done false
19051 6255 Acked Read 7 Done Sent Done false
19052 6256 Acked Write 7 Done Sent Done false
19053 6257 Acked Read 4 Done Sent Done false
19054 6258 Acked Write 1 Done Sent Done false
19055 6259 Acked Read 4 Done Sent Done false
19056 6260 Acked Read 7 Done Sent Done false
19057 6261 Acked Write 1 Done Sent Done false
19058 6262 Acked Write 10 Done Sent Done false
19059 6263 Acked Write 10 Done Sent Done false
19060 6264 Acked Read 10 Done Sent Done false
19061 6265 Acked Write 5 Done Sent Done false
19062 6266 Acked Write 8 Done Sent Done false
19063 6267 Acked Write 10 Done Sent Done false
19064 6268 Acked Read 4 Done Sent Done false
19065 6269 Acked Write 9 Done Sent Done false
19066 6270 Acked Read 3 Done Sent Done false
19067 6271 Acked Read 5 Done Sent Done false
19068 6272 Acked Read 5 Done Sent Done false
19069 6273 Acked Read 3 Done Sent Done false
19070 6274 Acked Write 4 Done Sent Done false
19071 6275 Acked Flush 0 Done Sent Done false
19072 6276 Acked Flush 0 Done Sent Done false
19073 6277 Acked Read 3 Done Sent Done false
19074 6278 Acked Read 3 Done Sent Done false
19075 6279 Acked Write 3 Done Sent Done false
19076 6280 Acked Read 5 Done Sent Done false
19077 6281 Acked Write 8 Done Sent Done false
19078 6282 Acked Read 7 Done Sent Done false
19079 6283 Acked Read 5 Done Sent Done false
19080 6284 Acked Read 4 Done Sent Done false
19081 6285 Acked Read 8 Done Sent Done false
19082 6286 Acked Flush 0 Done Sent Done false
19083 6287 Acked Read 4 Done Sent Done false
19084 6288 Acked Flush 0 Done Sent Done false
19085 6289 Acked Read 10 Done Sent Done false
19086 6290 Acked Read 7 Done Sent Done false
19087 6291 Acked Read 4 Done Sent Done false
19088 6292 Acked Write 9 Done Sent Done false
19089 6293 Acked Read 8 Done Sent Done false
19090 6294 Acked Write 4 Done Sent Done false
19091 6295 Acked Write 1 Done Sent Done false
19092 6296 Acked Read 6 Done Sent Done false
19093 6297 Acked Write 5 Done Sent Done false
19094 6298 Acked Read 6 Done Sent Done false
19095 6299 Acked Read 6 Done Sent Done false
19096 6300 Acked Read 1 Done Sent Done false
19097 6301 Acked Write 10 Done Sent Done false
19098 6302 Acked Write 10 Done Sent Done false
19099 6303 Acked Read 3 Done Sent Done false
19100 6304 Acked Write 9 Done Sent Done false
19101 6305 Acked Read 7 Done Sent Done false
19102 6306 Acked Write 6 Done Sent Done false
19103 6307 Acked Write 5 Done Sent Done false
19104 6308 Acked Read 6 Done Sent Done false
19105 6309 Acked Flush 0 Done Sent Done false
19106 6310 Acked Write 5 Done Sent Done false
19107 6311 Acked Read 7 Done Sent Done false
19108 6312 Acked Read 5 Done Sent Done false
19109 6313 Acked Read 8 Done Sent Done false
19110 6314 Acked Flush 0 Done Sent Done false
19111 6315 Acked Read 9 Done Sent Done false
19112 6316 Acked Write 5 Done Sent Done false
19113 6317 Acked Read 10 Done Sent Done false
19114 6318 Acked Write 4 Done Sent Done false
19115 6319 Acked Read 1 Done Sent Done false
19116 6320 Acked Read 9 Done Sent Done false
19117 6321 Acked Write 7 Done Sent Done false
19118 6322 Acked Read 8 Done Sent Done false
19119 6323 Acked Write 5 Done Sent Done false
19120 6324 Acked Write 10 Done Sent Done false
19121 6325 Acked Write 10 Done Sent Done false
19122 6326 Acked Write 2 Done Sent Done false
19123 6327 Acked Read 6 Done Sent Done false
19124 6328 Acked Read 8 Done Sent Done false
19125 6329 Acked Read 5 Done Sent Done false
19126 6330 Acked Read 5 Done Sent Done false
19127 6331 Acked Read 3 Done Sent Done false
19128 6332 Acked Read 10 Done Sent Done false
19129 6333 Acked Write 3 Done Sent Done false
19130 6334 Acked Read 9 Done Sent Done false
19131 6335 Acked Write 4 Done Sent Done false
19132 6336 Acked Write 6 Done Sent Done false
19133 6337 Acked Write 6 Done Sent Done false
19134 6338 Acked Read 7 Done Sent Done false
19135 6339 Acked Read 3 Done Sent Done false
19136 6340 Acked Flush 0 Done Sent Done false
19137 6341 Acked Write 3 Done Sent Done false
19138 6342 Acked Read 9 Done Sent Done false
19139 6343 Acked Read 8 Done Sent Done false
19140 6344 Acked Read 3 Done Sent Done false
19141 6345 Acked Write 9 Done Sent Done false
19142 6346 Acked Read 6 Done Sent Done false
19143 6347 Acked Write 10 Done Sent Done false
19144 6348 Acked Read 9 Done Sent Done false
19145 6349 Acked Read 6 Done Sent Done false
19146 6350 Acked Read 1 Done Sent Done false
19147 6351 Acked Write 6 Done Sent Done false
19148 6352 Acked Read 6 Done Sent Done false
19149 6353 Acked Read 7 Done Sent Done false
19150 6354 Acked Write 5 Done Sent Done false
19151 6355 Acked Read 7 Done Sent Done false
19152 6356 Acked Read 4 Done Sent Done false
19153 6357 Acked Read 1 Done Sent Done false
19154 6358 Acked Read 7 Done Sent Done false
19155 6359 Acked Read 4 Done Sent Done false
19156 6360 Acked Read 7 Done Sent Done false
19157 6361 Acked Read 1 Done Sent Done false
19158 6362 Acked Read 1 Done Sent Done false
19159 6363 Acked Read 2 Done Sent Done false
19160 6364 Acked Write 8 Done Sent Done false
19161 6365 Acked Read 10 Done Sent Done false
19162 6366 Acked Write 4 Done Sent Done false
19163 6367 Acked Read 5 Done Sent Done false
19164 6368 Acked Read 8 Done Sent Done false
19165 6369 Acked Write 1 Done Sent Done false
19166 6370 Acked Write 10 Done Sent Done false
19167 6371 Acked Read 4 Done Sent Done false
19168 6372 Acked Read 7 Done Sent Done false
19169 6373 Acked Write 10 Done Sent Done false
19170 6374 Acked Write 7 Done Sent Done false
19171 6375 Acked Read 8 Done Sent Done false
19172 6376 Acked Read 8 Done Sent Done false
19173 6377 Acked Write 3 Done Sent Done false
19174 6378 Acked Read 10 Done Sent Done false
19175 6379 Acked Read 3 Done Sent Done false
19176 6380 Acked Read 8 Done Sent Done false
19177 6381 Acked Read 10 Done Sent Done false
19178 6382 Acked Read 9 Done Sent Done false
19179 6383 Acked Write 9 Done Sent Done false
19180 6384 Acked Flush 0 Done Sent Done false
19181 6385 Acked Read 10 Done Sent Done false
19182 6386 Acked Read 4 Done Sent Done false
19183 6387 Acked Read 3 Done Sent Done false
19184 6388 Acked Flush 0 Done Sent Done false
19185 6389 Acked Read 8 Done Sent Done false
19186 6390 Acked Flush 0 Done Sent Done false
19187 6391 Acked Write 5 Done Sent Done false
19188 6392 Acked Read 10 Done Sent Done false
19189 6393 Acked Write 9 Done Sent Done false
19190 6394 Acked Write 4 Done Sent Done false
19191 6395 Acked Write 2 Done Sent Done false
19192 6396 Acked Write 5 Done Sent Done false
19193 6397 Acked Read 2 Done Sent Done false
19194 6398 Acked Read 8 Done Sent Done false
19195 6399 Acked Read 8 Done Sent Done false
19196 6400 Acked Flush 0 Done Sent Done false
19197 6401 Acked Write 10 Done Sent Done false
19198 6402 Acked Read 5 Done Sent Done false
19199 6403 Acked Flush 0 Done Sent Done false
19200 6404 Acked Read 8 Done Sent Done false
19201 6405 Acked Read 3 Done Sent Done false
19202 6406 Acked Write 9 Done Sent Done false
19203 6407 Acked Read 9 Done Sent Done false
19204 6408 Acked Write 1 Done Sent Done false
19205 6409 Acked Read 7 Done Sent Done false
19206 6410 Acked Write 6 Done Sent Done false
19207 6411 Acked Read 6 Done Sent Done false
19208 6412 Acked Flush 0 Done Sent Done false
19209 6413 Acked Write 9 Done Sent Done false
19210 6414 Acked Read 10 Done Sent Done false
19211 6415 Acked Write 9 Done Sent Done false
19212 6416 Acked Read 10 Done Sent Done false
19213 6417 Acked Write 9 Done Sent Done false
19214 6418 Acked Flush 0 Done Sent Done false
19215 6419 Acked Read 3 Done Sent Done false
19216 6420 Acked Flush 0 Done Sent Done false
19217 6421 Acked Read 5 Done Sent Done false
19218 6422 Acked Read 7 Done Sent Done false
19219 6423 Acked Write 6 Done Sent Done false
19220 6424 Acked Read 2 Done Sent Done false
19221 6425 Acked Read 6 Done Sent Done false
19222 6426 Acked Write 8 Done Sent Done false
19223 6427 Acked Read 6 Done Sent Done false
19224 6428 Acked Write 8 Done Sent Done false
19225 6429 Acked Read 6 Done Sent Done false
19226 6430 Acked Read 2 Done Sent Done false
19227 6431 Acked Read 7 Done Sent Done false
19228 6432 Acked Write 1 Done Sent Done false
19229 6433 Acked Read 5 Done Sent Done false
19230 6434 Acked Write 5 Done Sent Done false
19231 STATES DS:0 DS:1 DS:2 TOTAL
19232 Sent 0 301 0 301
19233 Done 301 0 301 602
19234 Skipped 0 0 0 0
19235 Error 0 0 0 0
19236 Last Flush: Some(JobId(6420)) Some(JobId(6133)) Some(JobId(6420))
19237 Downstairs last five completed: 6133 6132 6131 6130 6129
19238 Upstairs last five completed: 6434 6433 6432 6431 6430
19239 CLIENT: Up:0 ds:301 act:2
19240 [8820] got start action
19241 [8820] I am going to restart
19242 Starting downstairs at port 8820
19243 Make output file at "/tmp/dsc/downstairs-8820.txt"
19244 [8820][1] reports Starting
19245 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1223)
19246 [8820][1] reports Running
19247 ----------------------------------------------------------------
19248 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
19249 States: Active Offline Active
19250 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19251 6134 Acked Read 9 Done Sent Done false
19252 6135 Acked Read 10 Done Sent Done false
19253 6136 Acked Read 2 Done Sent Done false
19254 6137 Acked Write 9 Done Sent Done false
19255 6138 Acked Write 10 Done Sent Done false
19256 6139 Acked Write 8 Done Sent Done false
19257 6140 Acked Read 4 Done Sent Done false
19258 6141 Acked Read 7 Done Sent Done false
19259 6142 Acked Read 9 Done Sent Done false
19260 6143 Acked Write 5 Done Sent Done false
19261 6144 Acked Write 10 Done Sent Done false
19262 6145 Acked Write 8 Done Sent Done false
19263 6146 Acked Flush 0 Done Sent Done false
19264 6147 Acked Flush 0 Done Sent Done false
19265 6148 Acked Write 4 Done Sent Done false
19266 6149 Acked Flush 0 Done Sent Done false
19267 6150 Acked Write 8 Done Sent Done false
19268 6151 Acked Read 9 Done Sent Done false
19269 6152 Acked Write 5 Done Sent Done false
19270 6153 Acked Write 6 Done Sent Done false
19271 6154 Acked Write 2 Done Sent Done false
19272 6155 Acked Write 10 Done Sent Done false
19273 6156 Acked Read 8 Done Sent Done false
19274 6157 Acked Write 10 Done Sent Done false
19275 6158 Acked Write 8 Done Sent Done false
19276 6159 Acked Write 8 Done Sent Done false
19277 6160 Acked Read 1 Done Sent Done false
19278 6161 Acked Write 1 Done Sent Done false
19279 6162 Acked Read 7 Done Sent Done false
19280 6163 Acked Read 6 Done Sent Done false
19281 6164 Acked Flush 0 Done Sent Done false
19282 6165 Acked Read 9 Done Sent Done false
19283 6166 Acked Write 1 Done Sent Done false
19284 6167 Acked Read 5 Done Sent Done false
19285 6168 Acked Write 4 Done Sent Done false
19286 6169 Acked Write 6 Done Sent Done false
19287 6170 Acked Read 5 Done Sent Done false
19288 6171 Acked Read 2 Done Sent Done false
19289 6172 Acked Read 7 Done Sent Done false
19290 6173 Acked Read 6 Done Sent Done false
19291 6174 Acked Read 1 Done Sent Done false
19292 6175 Acked Read 1 Done Sent Done false
19293 6176 Acked Write 1 Done Sent Done false
19294 6177 Acked Read 4 Done Sent Done false
19295 6178 Acked Write 5 Done Sent Done false
19296 6179 Acked Write 2 Done Sent Done false
19297 6180 Acked Read 1 Done Sent Done false
19298 6181 Acked Read 3 Done Sent Done false
19299 6182 Acked Write 2 Done Sent Done false
19300 6183 Acked Read 5 Done Sent Done false
19301 6184 Acked Read 1 Done Sent Done false
19302 6185 Acked Write 1 Done Sent Done false
19303 6186 Acked Read 1 Done Sent Done false
19304 6187 Acked Write 8 Done Sent Done false
19305 6188 Acked Read 1 Done Sent Done false
19306 6189 Acked Write 10 Done Sent Done false
19307 6190 Acked Write 5 Done Sent Done false
19308 6191 Acked Read 10 Done Sent Done false
19309 6192 Acked Read 10 Done Sent Done false
19310 6193 Acked Read 10 Done Sent Done false
19311 6194 Acked Write 9 Done Sent Done false
19312 6195 Acked Read 10 Done Sent Done false
19313 6196 Acked Write 1 Done Sent Done false
19314 6197 Acked Read 10 Done Sent Done false
19315 6198 Acked Flush 0 Done Sent Done false
19316 6199 Acked Write 5 Done Sent Done false
19317 6200 Acked Write 7 Done Sent Done false
19318 6201 Acked Read 9 Done Sent Done false
19319 6202 Acked Write 2 Done Sent Done false
19320 6203 Acked Write 4 Done Sent Done false
19321 6204 Acked Read 9 Done Sent Done false
19322 6205 Acked Write 4 Done Sent Done false
19323 6206 Acked Write 7 Done Sent Done false
19324 6207 Acked Write 8 Done Sent Done false
19325 6208 Acked Read 5 Done Sent Done false
19326 6209 Acked Write 8 Done Sent Done false
19327 6210 Acked Write 1 Done Sent Done false
19328 6211 Acked Write 7 Done Sent Done false
19329 6212 Acked Flush 0 Done Sent Done false
19330 6213 Acked Read 9 Done Sent Done false
19331 6214 Acked Read 5 Done Sent Done false
19332 6215 Acked Read 6 Done Sent Done false
19333 6216 Acked Write 3 Done Sent Done false
19334 6217 Acked Read 5 Done Sent Done false
19335 6218 Acked Write 5 Done Sent Done false
19336 6219 Acked Read 3 Done Sent Done false
19337 6220 Acked Read 1 Done Sent Done false
19338 6221 Acked Read 6 Done Sent Done false
19339 6222 Acked Read 4 Done Sent Done false
19340 6223 Acked Read 1 Done Sent Done false
19341 6224 Acked Read 6 Done Sent Done false
19342 6225 Acked Read 4 Done Sent Done false
19343 6226 Acked Write 9 Done Sent Done false
19344 6227 Acked Write 10 Done Sent Done false
19345 6228 Acked Read 10 Done Sent Done false
19346 6229 Acked Write 9 Done Sent Done false
19347 6230 Acked Write 3 Done Sent Done false
19348 6231 Acked Read 2 Done Sent Done false
19349 6232 Acked Write 5 Done Sent Done false
19350 6233 Acked Flush 0 Done Sent Done false
19351 6234 Acked Flush 0 Done Sent Done false
19352 6235 Acked Read 5 Done Sent Done false
19353 6236 Acked Read 8 Done Sent Done false
19354 6237 Acked Flush 0 Done Sent Done false
19355 6238 Acked Read 6 Done Sent Done false
19356 6239 Acked Write 9 Done Sent Done false
19357 6240 Acked Write 6 Done Sent Done false
19358 6241 Acked Read 10 Done Sent Done false
19359 6242 Acked Write 6 Done Sent Done false
19360 6243 Acked Write 10 Done Sent Done false
19361 6244 Acked Write 9 Done Sent Done false
19362 6245 Acked Read 7 Done Sent Done false
19363 6246 Acked Read 6 Done Sent Done false
19364 6247 Acked Write 6 Done Sent Done false
19365 6248 Acked Read 10 Done Sent Done false
19366 6249 Acked Read 3 Done Sent Done false
19367 6250 Acked Write 10 Done Sent Done false
19368 6251 Acked Write 9 Done Sent Done false
19369 6252 Acked Read 5 Done Sent Done false
19370 6253 Acked Write 4 Done Sent Done false
19371 6254 Acked Write 8 Done Sent Done false
19372 6255 Acked Read 7 Done Sent Done false
19373 6256 Acked Write 7 Done Sent Done false
19374 6257 Acked Read 4 Done Sent Done false
19375 6258 Acked Write 1 Done Sent Done false
19376 6259 Acked Read 4 Done Sent Done false
19377 6260 Acked Read 7 Done Sent Done false
19378 6261 Acked Write 1 Done Sent Done false
19379 6262 Acked Write 10 Done Sent Done false
19380 6263 Acked Write 10 Done Sent Done false
19381 6264 Acked Read 10 Done Sent Done false
19382 6265 Acked Write 5 Done Sent Done false
19383 6266 Acked Write 8 Done Sent Done false
19384 6267 Acked Write 10 Done Sent Done false
19385 6268 Acked Read 4 Done Sent Done false
19386 6269 Acked Write 9 Done Sent Done false
19387 6270 Acked Read 3 Done Sent Done false
19388 6271 Acked Read 5 Done Sent Done false
19389 6272 Acked Read 5 Done Sent Done false
19390 6273 Acked Read 3 Done Sent Done false
19391 6274 Acked Write 4 Done Sent Done false
19392 6275 Acked Flush 0 Done Sent Done false
19393 6276 Acked Flush 0 Done Sent Done false
19394 6277 Acked Read 3 Done Sent Done false
19395 6278 Acked Read 3 Done Sent Done false
19396 6279 Acked Write 3 Done Sent Done false
19397 6280 Acked Read 5 Done Sent Done false
19398 6281 Acked Write 8 Done Sent Done false
19399 6282 Acked Read 7 Done Sent Done false
19400 6283 Acked Read 5 Done Sent Done false
19401 6284 Acked Read 4 Done Sent Done false
19402 6285 Acked Read 8 Done Sent Done false
19403 6286 Acked Flush 0 Done Sent Done false
19404 6287 Acked Read 4 Done Sent Done false
19405 6288 Acked Flush 0 Done Sent Done false
19406 6289 Acked Read 10 Done Sent Done false
19407 6290 Acked Read 7 Done Sent Done false
19408 6291 Acked Read 4 Done Sent Done false
19409 6292 Acked Write 9 Done Sent Done false
19410 6293 Acked Read 8 Done Sent Done false
19411 6294 Acked Write 4 Done Sent Done false
19412 6295 Acked Write 1 Done Sent Done false
19413 6296 Acked Read 6 Done Sent Done false
19414 6297 Acked Write 5 Done Sent Done false
19415 6298 Acked Read 6 Done Sent Done false
19416 6299 Acked Read 6 Done Sent Done false
19417 6300 Acked Read 1 Done Sent Done false
19418 6301 Acked Write 10 Done Sent Done false
19419 6302 Acked Write 10 Done Sent Done false
19420 6303 Acked Read 3 Done Sent Done false
19421 6304 Acked Write 9 Done Sent Done false
19422 6305 Acked Read 7 Done Sent Done false
19423 6306 Acked Write 6 Done Sent Done false
19424 6307 Acked Write 5 Done Sent Done false
19425 6308 Acked Read 6 Done Sent Done false
19426 6309 Acked Flush 0 Done Sent Done false
19427 6310 Acked Write 5 Done Sent Done false
19428 6311 Acked Read 7 Done Sent Done false
19429 6312 Acked Read 5 Done Sent Done false
19430 6313 Acked Read 8 Done Sent Done false
19431 6314 Acked Flush 0 Done Sent Done false
19432 6315 Acked Read 9 Done Sent Done false
19433 6316 Acked Write 5 Done Sent Done false
19434 6317 Acked Read 10 Done Sent Done false
19435 6318 Acked Write 4 Done Sent Done false
19436 6319 Acked Read 1 Done Sent Done false
19437 6320 Acked Read 9 Done Sent Done false
19438 6321 Acked Write 7 Done Sent Done false
19439 6322 Acked Read 8 Done Sent Done false
19440 6323 Acked Write 5 Done Sent Done false
19441 6324 Acked Write 10 Done Sent Done false
19442 6325 Acked Write 10 Done Sent Done false
19443 6326 Acked Write 2 Done Sent Done false
19444 6327 Acked Read 6 Done Sent Done false
19445 6328 Acked Read 8 Done Sent Done false
19446 6329 Acked Read 5 Done Sent Done false
19447 6330 Acked Read 5 Done Sent Done false
19448 6331 Acked Read 3 Done Sent Done false
19449 6332 Acked Read 10 Done Sent Done false
19450 6333 Acked Write 3 Done Sent Done false
19451 6334 Acked Read 9 Done Sent Done false
19452 6335 Acked Write 4 Done Sent Done false
19453 6336 Acked Write 6 Done Sent Done false
19454 6337 Acked Write 6 Done Sent Done false
19455 6338 Acked Read 7 Done Sent Done false
19456 6339 Acked Read 3 Done Sent Done false
19457 6340 Acked Flush 0 Done Sent Done false
19458 6341 Acked Write 3 Done Sent Done false
19459 6342 Acked Read 9 Done Sent Done false
19460 6343 Acked Read 8 Done Sent Done false
19461 6344 Acked Read 3 Done Sent Done false
19462 6345 Acked Write 9 Done Sent Done false
19463 6346 Acked Read 6 Done Sent Done false
19464 6347 Acked Write 10 Done Sent Done false
19465 6348 Acked Read 9 Done Sent Done false
19466 6349 Acked Read 6 Done Sent Done false
19467 6350 Acked Read 1 Done Sent Done false
19468 6351 Acked Write 6 Done Sent Done false
19469 6352 Acked Read 6 Done Sent Done false
19470 6353 Acked Read 7 Done Sent Done false
19471 6354 Acked Write 5 Done Sent Done false
19472 6355 Acked Read 7 Done Sent Done false
19473 6356 Acked Read 4 Done Sent Done false
19474 6357 Acked Read 1 Done Sent Done false
19475 6358 Acked Read 7 Done Sent Done false
19476 6359 Acked Read 4 Done Sent Done false
19477 6360 Acked Read 7 Done Sent Done false
19478 6361 Acked Read 1 Done Sent Done false
19479 6362 Acked Read 1 Done Sent Done false
19480 6363 Acked Read 2 Done Sent Done false
19481 6364 Acked Write 8 Done Sent Done false
19482 6365 Acked Read 10 Done Sent Done false
19483 6366 Acked Write 4 Done Sent Done false
19484 6367 Acked Read 5 Done Sent Done false
19485 6368 Acked Read 8 Done Sent Done false
19486 6369 Acked Write 1 Done Sent Done false
19487 6370 Acked Write 10 Done Sent Done false
19488 6371 Acked Read 4 Done Sent Done false
19489 6372 Acked Read 7 Done Sent Done false
19490 6373 Acked Write 10 Done Sent Done false
19491 6374 Acked Write 7 Done Sent Done false
19492 6375 Acked Read 8 Done Sent Done false
19493 6376 Acked Read 8 Done Sent Done false
19494 6377 Acked Write 3 Done Sent Done false
19495 6378 Acked Read 10 Done Sent Done false
19496 6379 Acked Read 3 Done Sent Done false
19497 6380 Acked Read 8 Done Sent Done false
19498 6381 Acked Read 10 Done Sent Done false
19499 6382 Acked Read 9 Done Sent Done false
19500 6383 Acked Write 9 Done Sent Done false
19501 6384 Acked Flush 0 Done Sent Done false
19502 6385 Acked Read 10 Done Sent Done false
19503 6386 Acked Read 4 Done Sent Done false
19504 6387 Acked Read 3 Done Sent Done false
19505 6388 Acked Flush 0 Done Sent Done false
19506 6389 Acked Read 8 Done Sent Done false
19507 6390 Acked Flush 0 Done Sent Done false
19508 6391 Acked Write 5 Done Sent Done false
19509 6392 Acked Read 10 Done Sent Done false
19510 6393 Acked Write 9 Done Sent Done false
19511 6394 Acked Write 4 Done Sent Done false
19512 6395 Acked Write 2 Done Sent Done false
19513 6396 Acked Write 5 Done Sent Done false
19514 6397 Acked Read 2 Done Sent Done false
19515 6398 Acked Read 8 Done Sent Done false
19516 6399 Acked Read 8 Done Sent Done false
19517 6400 Acked Flush 0 Done Sent Done false
19518 6401 Acked Write 10 Done Sent Done false
19519 6402 Acked Read 5 Done Sent Done false
19520 6403 Acked Flush 0 Done Sent Done false
19521 6404 Acked Read 8 Done Sent Done false
19522 6405 Acked Read 3 Done Sent Done false
19523 6406 Acked Write 9 Done Sent Done false
19524 6407 Acked Read 9 Done Sent Done false
19525 6408 Acked Write 1 Done Sent Done false
19526 6409 Acked Read 7 Done Sent Done false
19527 6410 Acked Write 6 Done Sent Done false
19528 6411 Acked Read 6 Done Sent Done false
19529 6412 Acked Flush 0 Done Sent Done false
19530 6413 Acked Write 9 Done Sent Done false
19531 6414 Acked Read 10 Done Sent Done false
19532 6415 Acked Write 9 Done Sent Done false
19533 6416 Acked Read 10 Done Sent Done false
19534 6417 Acked Write 9 Done Sent Done false
19535 6418 Acked Flush 0 Done Sent Done false
19536 6419 Acked Read 3 Done Sent Done false
19537 6420 Acked Flush 0 Done Sent Done false
19538 6421 Acked Read 5 Done Sent Done false
19539 6422 Acked Read 7 Done Sent Done false
19540 6423 Acked Write 6 Done Sent Done false
19541 6424 Acked Read 2 Done Sent Done false
19542 6425 Acked Read 6 Done Sent Done false
19543 6426 Acked Write 8 Done Sent Done false
19544 6427 Acked Read 6 Done Sent Done false
19545 6428 Acked Write 8 Done Sent Done false
19546 6429 Acked Read 6 Done Sent Done false
19547 6430 Acked Read 2 Done Sent Done false
19548 6431 Acked Read 7 Done Sent Done false
19549 6432 Acked Write 1 Done Sent Done false
19550 6433 Acked Read 5 Done Sent Done false
19551 6434 Acked Write 5 Done Sent Done false
19552 6435 Acked Flush 0 Done Sent Done false
19553 STATES DS:0 DS:1 DS:2 TOTAL
19554 Sent 0 302 0 302
19555 Done 302 0 302 604
19556 Skipped 0 0 0 0
19557 Error 0 0 0 0
19558 Last Flush: Some(JobId(6435)) Some(JobId(6133)) Some(JobId(6435))
19559 Downstairs last five completed: 6133 6132 6131 6130 6129
19560 Upstairs last five completed: 6435 6434 6433 6432 6431
19561 CLIENT: Up:0 ds:302 act:2
195622025-01-23T17:41:49.900ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
195632025-01-23T17:41:49.901ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62763) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
195642025-01-23T17:41:49.902ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
195652025-01-23T17:41:49.902ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
195662025-01-23T17:41:49.902ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
195672025-01-23T17:41:49.902ZINFOcrucible: send last flush ID to this DS: Some(JobId(6133)) = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
195682025-01-23T17:41:49.902ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
195692025-01-23T17:41:49.902ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(6133)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
195702025-01-23T17:41:49.903ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(6133)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
19571 ----------------------------------------------------------------
19572 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19573 Downstairs last five completed: 6435 6434 6433 6432 6431
19574 Upstairs last five completed: 6435 6434 6433 6432 6431
19575 CLIENT: Up:0 ds:0 act:3
19576 Replay: All jobs finished
19577 Main task has work to do, go find it
19578 got dsc Stop(0)
19579 stop 0
19580 [8810] Got stop action so:false kr:false
19581 Jan 23 17:41:52.167 INFO request completed, latency_us: 382, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 594b62e7-4608-4c2a-99d4-5ec9edacdbd7, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
195822025-01-23T17:41:52.168ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
195832025-01-23T17:41:52.168ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
195842025-01-23T17:41:52.168ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
195852025-01-23T17:41:52.168ZINFOcrucible: client task is exiting = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
19586 Jan 23 17:41:52.168 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2814b42a-7713-44cd-9ed0-07c5d476ee6c, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
19587 [8810] Exited with: ExitStatus(unix_wait_status(9))
19588 [8810][0] reports Exit
19589 Jan 23 17:41:56.171 INFO request completed, latency_us: 290, response_code: 200, uri: /state/cid/0, method: GET, req_id: 0e616d75-713c-44d9-88de-cc209cdc3b41, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
19590 001/300 Write block 4739 len 36864 data: 5 3 3 3 4 4 6 6 5
19591 002/300 Read block 2433 len 24576
19592 003/300 Read block 1442 len 8192
19593 004/300 Read block 4395 len 8192
19594 005/300 Read block 2288 len 20480
19595 006/300 Read block 2240 len 20480
19596 007/300 Write block 1794 len 28672 data: 5 5 6 5 5 6 7
19597 008/300 Read block 3128 len 12288
19598 009/300 Write block 4265 len 24576 data: 2 2 4 5 6 7
19599 010/300 Write block 4057 len 16384 data: 3 3 3 3
19600 011/300 Flush
19601 012/300 Read block 1893 len 4096
19602 013/300 Read block 748 len 16384
19603 014/300 Read block 2106 len 24576
19604 015/300 Write block 1451 len 28672 data: 4 4 3 4 4 4 4
19605 016/300 Read block 4161 len 36864
19606 017/300 Write block 2003 len 12288 data: 4 4 4
19607 018/300 Write block 1328 len 16384 data: 3 4 6 8
19608 019/300 Read block 3878 len 4096
19609 020/300 Write block 2848 len 32768 data: 2 2 2 3 3 3 4 4
19610 021/300 Read block 34 len 40960
19611 022/300 Flush
19612 023/300 Read block 2732 len 36864
19613 024/300 Write block 4262 len 8192 data: 4 3
19614 025/300 Flush
19615 026/300 Read block 3480 len 4096
19616 027/300 Read block 4576 len 4096
19617 028/300 Flush
19618 029/300 Write block 1406 len 28672 data: 3 3 3 3 3 3 4
19619 030/300 Read block 1983 len 4096
19620 031/300 Write block 3562 len 16384 data: 4 4 5 5
19621 032/300 Write block 3798 len 24576 data: 3 4 4 3 3 3
19622 033/300 Read block 470 len 12288
19623 034/300 Read block 4653 len 36864
19624 035/300 Write block 1516 len 36864 data: 5 5 5 2 3 3 3 3 3
19625 036/300 Write block 2898 len 24576 data: 4 4 4 5 5 5
19626 037/300 Read block 2230 len 20480
19627 038/300 Read block 1260 len 24576
19628 039/300 Write block 1404 len 32768 data: 4 3 4 4 4 4 4 4
19629 040/300 Read block 3907 len 24576
19630 041/300 Read block 2178 len 20480
19631 042/300 Read block 4752 len 40960
19632 043/300 Write block 4613 len 4096 data: 5
19633 044/300 Flush
19634 045/300 Read block 2493 len 16384
19635 046/300 Read block 4469 len 28672
19636 047/300 Write block 4687 len 12288 data: 5 4 4
19637 048/300 Write block 3884 len 28672 data: 6 6 4 4 4 4 4
19638 049/300 Write block 1082 len 16384 data: 3 4 4 4
19639 050/300 Flush
19640 051/300 Write block 2626 len 4096 data: 6
19641 052/300 Read block 1773 len 36864
19642 053/300 Write block 635 len 12288 data: 4 4 4
19643 054/300 Read block 3499 len 8192
19644 055/300 Write block 3131 len 24576 data: 2 3 4 4 4 4
19645 056/300 Read block 89 len 36864
19646 057/300 Write block 2745 len 4096 data: 7
19647 058/300 Write block 3917 len 8192 data: 6 4
19648 059/300 Read block 1246 len 32768
19649 060/300 Read block 4408 len 32768
19650 061/300 Write block 2065 len 28672 data: 4 4 5 5 7 6 6
19651 062/300 Read block 4333 len 8192
19652 063/300 Write block 4665 len 16384 data: 2 2 3 3
19653 064/300 Read block 4366 len 12288
19654 065/300 Read block 1999 len 4096
19655 066/300 Read block 519 len 40960
19656 067/300 Write block 4599 len 24576 data: 3 4 5 5 5 5
19657 068/300 Read block 2459 len 8192
19658 069/300 Write block 2545 len 24576 data: 4 4 5 5 4 3
19659 070/300 Read block 4048 len 28672
19660 071/300 Flush
19661 072/300 Flush
19662 073/300 Flush
19663 074/300 Write block 197 len 40960 data: 4 4 4 4 6 5 7 7 6 5
19664 075/300 Write block 1604 len 4096 data: 2
19665 076/300 Write block 228 len 28672 data: 4 3 3 4 4 4 3
19666 077/300 Write block 2480 len 36864 data: 4 4 5 5 5 5 4 4 4
19667 078/300 Write block 4461 len 4096 data: 5
19668 079/300 Write block 237 len 8192 data: 5 4
19669 080/300 Write block 2928 len 4096 data: 5
19670 081/300 Read block 2613 len 36864
19671 082/300 Read block 3030 len 36864
19672 083/300 Flush
19673 084/300 Read block 3504 len 4096
19674 085/300 Write block 988 len 20480 data: 5 7 7 7 7
19675 086/300 Write block 2955 len 12288 data: 5 3 4
19676 087/300 Read block 829 len 8192
19677 088/300 Write block 2669 len 8192 data: 5 5
19678 089/300 Write block 3468 len 40960 data: 7 5 5 4 4 4 5 6 5 4
19679 090/300 Read block 1057 len 12288
19680 091/300 Write block 2543 len 28672 data: 4 5 5 5 6 6 5
19681 092/300 Read block 3003 len 20480
19682 093/300 Read block 512 len 36864
19683 094/300 Read block 332 len 20480
19684 095/300 Write block 4031 len 20480 data: 5 6 6 6 6
19685 096/300 Write block 109 len 8192 data: 6 8
19686 097/300 Write block 964 len 40960 data: 2 2 2 2 2 2 4 4 5 5
19687 098/300 Write block 2794 len 40960 data: 2 2 2 3 4 5 5 4 4 3
19688 099/300 Write block 3189 len 36864 data: 3 2 2 2 2 2 2 2 4
19689 100/300 Read block 1376 len 40960
19690 101/300 Read block 1037 len 20480
19691 102/300 Read block 3297 len 12288
19692 103/300 Write block 4512 len 32768 data: 4 5 5 6 5 4 4 4
19693 104/300 Write block 2792 len 12288 data: 2 2 3
19694 105/300 Read block 1962 len 20480
19695 106/300 Write block 3599 len 4096 data: 6
19696 107/300 Write block 4178 len 32768 data: 5 6 6 6 5 5 2 2
19697 108/300 Read block 3205 len 20480
19698 109/300 Read block 1965 len 28672
19699 110/300 Read block 1740 len 24576
19700 111/300 Write block 3874 len 32768 data: 5 4 3 3 4 5 5 5
19701 112/300 Write block 3163 len 40960 data: 7 7 5 5 4 3 2 2 2 2
19702 113/300 Write block 3428 len 36864 data: 5 8 7 7 7 8 7 7 5
19703 114/300 Read block 1044 len 4096
19704 115/300 Read block 1830 len 36864
19705 116/300 Read block 1852 len 4096
19706 117/300 Read block 4738 len 32768
19707 118/300 Read block 4820 len 24576
19708 119/300 Write block 2453 len 12288 data: 4 4 4
19709 120/300 Write block 2128 len 8192 data: 4 3
19710 121/300 Write block 1497 len 16384 data: 2 2 4 5
19711 122/300 Read block 4476 len 8192
19712 123/300 Flush
19713 124/300 Write block 1587 len 20480 data: 2 2 2 2 2
19714 125/300 Read block 2915 len 36864
19715 126/300 Read block 2019 len 20480
19716 127/300 Flush
19717 128/300 Read block 4383 len 20480
19718 129/300 Write block 2523 len 8192 data: 3 3
19719 130/300 Read block 1256 len 8192
19720 131/300 Write block 4129 len 36864 data: 2 2 2 2 2 3 5 5 5
19721 132/300 Read block 1480 len 8192
19722 133/300 Flush
19723 134/300 Write block 2900 len 8192 data: 5 6
19724 135/300 Write block 1668 len 4096 data: 3
19725 136/300 Write block 4520 len 4096 data: 5
19726 137/300 Read block 388 len 40960
19727 138/300 Read block 4139 len 40960
19728 139/300 Read block 1974 len 20480
19729 140/300 Write block 2875 len 28672 data: 3 4 3 4 3 3 4
19730 141/300 Read block 1043 len 28672
19731 142/300 Write block 147 len 4096 data: 3
19732 143/300 Read block 993 len 24576
19733 144/300 Write block 1232 len 28672 data: 4 5 5 6 7 7 5
19734 145/300 Read block 278 len 36864
19735 146/300 Write block 4435 len 16384 data: 5 5 5 5
19736 147/300 Write block 566 len 32768 data: 5 5 5 6 8 9 6 5
19737 148/300 Read block 4597 len 12288
19738 149/300 Write block 1158 len 36864 data: 7 8 8 7 10 10 10 7 4
19739 150/300 Write block 1579 len 40960 data: 2 2 3 3 3 2 2 2 3 3
19740 151/300 Read block 4474 len 28672
19741 152/300 Read block 2323 len 4096
19742 153/300 Read block 1341 len 24576
19743 154/300 Write block 2520 len 4096 data: 3
19744 155/300 Write block 2833 len 40960 data: 6 5 5 4 6 5 6 6 7 6
19745 156/300 Read block 479 len 24576
19746 157/300 Read block 3298 len 36864
19747 158/300 Read block 2259 len 40960
19748 159/300 Write block 2320 len 32768 data: 4 3 3 3 3 3 3 5
19749 160/300 Read block 2129 len 16384
19750 161/300 Read block 4191 len 36864
19751 162/300 Write block 4602 len 36864 data: 6 6 6 6 6 6 4 5 4
19752 163/300 Flush
19753 164/300 Flush
19754 165/300 Read block 2317 len 16384
19755 166/300 Write block 2146 len 16384 data: 6 5 5 5
19756 167/300 Write block 4967 len 24576 data: 5 6 6 6 9 10
19757 168/300 Read block 1090 len 12288
19758 169/300 Write block 1670 len 8192 data: 2 2
19759 170/300 Read block 1439 len 16384
19760 171/300 Read block 1953 len 20480
19761 172/300 Read block 521 len 8192
19762 173/300 Read block 4088 len 28672
19763 174/300 Write block 2974 len 16384 data: 7 6 5 5
19764 175/300 Read block 3423 len 40960
19765 176/300 Read block 1838 len 36864
19766 177/300 Read block 995 len 24576
19767 178/300 Read block 551 len 28672
19768 179/300 Read block 4054 len 40960
19769 180/300 Write block 1468 len 16384 data: 2 2 3 4
19770 181/300 Read block 4414 len 12288
19771 182/300 Read block 1536 len 28672
19772 183/300 Read block 3840 len 32768
19773 184/300 Read block 1470 len 4096
19774 185/300 Write block 1697 len 20480 data: 3 4 4 4 3
19775 186/300 Read block 4486 len 36864
19776 187/300 Read block 4293 len 32768
19777 188/300 Read block 1134 len 20480
19778 189/300 Flush
19779 190/300 Write block 958 len 36864 data: 3 3 3 3 4 2 3 3 3
19780 191/300 Read block 2359 len 36864
19781 192/300 Write block 2668 len 28672 data: 4 6 6 4 5 5 6
19782 193/300 Read block 1481 len 36864
19783 194/300 Write block 2811 len 20480 data: 7 6 6 6 7
19784 195/300 Write block 1072 len 28672 data: 6 5 5 4 3 3 3
19785 196/300 Read block 2972 len 32768
19786 197/300 Write block 4363 len 12288 data: 5 4 5
19787 198/300 Flush
19788 199/300 Read block 4324 len 36864
19789 200/300 Write block 3469 len 36864 data: 6 6 5 5 5 6 7 6 5
19790 201/300 Read block 1077 len 40960
19791 202/300 Write block 4700 len 36864 data: 7 6 6 7 7 6 6 6 7
19792 203/300 Read block 1838 len 8192
19793 204/300 Read block 2081 len 28672
19794 205/300 Read block 236 len 24576
19795 206/300 Write block 4661 len 20480 data: 3 2 2 2 3
19796 207/300 Write block 868 len 8192 data: 4 5
19797 208/300 Read block 3153 len 8192
19798 209/300 Read block 1610 len 12288
19799 210/300 Write block 4704 len 20480 data: 8 7 7 7 8
19800 211/300 Read block 4041 len 20480
19801 212/300 Flush
19802 213/300 Write block 3229 len 4096 data: 4
19803 214/300 Flush
19804 215/300 Read block 3781 len 16384
19805 216/300 Read block 2264 len 40960
19806 217/300 Read block 3707 len 24576
19807 218/300 Read block 2964 len 24576
19808 219/300 Write block 4677 len 4096 data: 4
19809 220/300 Read block 1641 len 28672
19810 221/300 Read block 1608 len 16384
19811 222/300 Read block 2413 len 16384
19812 223/300 Read block 3241 len 36864
19813 224/300 Flush
19814 225/300 Read block 4794 len 32768
19815 226/300 Write block 1543 len 40960 data: 2 2 3 3 2 3 4 5 4 4
19816 227/300 Write block 3800 len 28672 data: 5 4 4 4 3 4 5
19817 228/300 Read block 221 len 16384
19818 229/300 Read block 521 len 4096
19819 230/300 Read block 4178 len 8192
19820 231/300 Write block 4366 len 36864 data: 5 5 5 5 5 5 5 6 5
19821 232/300 Write block 4716 len 4096 data: 3
19822 233/300 Write block 944 len 40960 data: 4 4 4 3 3 3 3 3 3 3
19823 234/300 Read block 4895 len 40960
19824 235/300 Flush
19825 236/300 Write block 2491 len 24576 data: 6 5 5 5 5 4
19826 237/300 Read block 4226 len 16384
19827 238/300 Read block 3917 len 12288
19828 239/300 Write block 3015 len 16384 data: 6 6 7 6
19829 240/300 Read block 470 len 36864
19830 241/300 Write block 2925 len 8192 data: 5 5
19831 242/300 Read block 3359 len 12288
19832 243/300 Read block 2452 len 28672
19833 244/300 Read block 2857 len 24576
19834 245/300 Read block 3592 len 40960
19835 246/300 Read block 3596 len 24576
19836 247/300 Write block 2811 len 40960 data: 8 7 7 7 8 6 6 5 5 4
19837 248/300 Read block 4621 len 20480
19838 249/300 Read block 21 len 4096
19839 250/300 Read block 1163 len 8192
19840 251/300 Write block 3131 len 32768 data: 3 4 5 5 5 5 4 4
19841 252/300 Write block 1736 len 24576 data: 4 5 5 4 4 4
19842 253/300 Write block 3968 len 28672 data: 6 6 5 5 4 3 4
19843 254/300 Read block 1659 len 16384
19844 255/300 Write block 2183 len 32768 data: 5 4 3 2 2 2 4 4
19845 256/300 Read block 997 len 12288
19846 257/300 Write block 3600 len 24576 data: 6 6 5 5 6 4
19847 258/300 Write block 4622 len 4096 data: 7
19848 259/300 Write block 2684 len 20480 data: 5 5 4 4 5
19849 260/300 Read block 1261 len 12288
19850 261/300 Flush
19851 262/300 Write block 168 len 4096 data: 5
19852 263/300 Write block 1283 len 20480 data: 3 4 4 4 4
19853 264/300 Flush
19854 265/300 Read block 3930 len 40960
19855 266/300 Write block 2684 len 8192 data: 6 6
19856 267/300 Write block 1439 len 16384 data: 2 2 2 2
19857 268/300 Read block 3248 len 4096
19858 269/300 Flush
19859 270/300 Read block 1306 len 40960
19860 271/300 Read block 3103 len 4096
19861 272/300 Flush
19862 273/300 Read block 2248 len 8192
19863 274/300 Read block 4207 len 8192
19864 275/300 Write block 3899 len 16384 data: 2 2 3 3
19865 276/300 Write block 3050 len 28672 data: 5 4 4 4 4 4 4
19866 277/300 Read block 4287 len 16384
19867 278/300 Write block 2137 len 8192 data: 5 5
19868 279/300 Read block 2093 len 36864
19869 280/300 Flush
19870 281/300 Read block 3121 len 32768
19871 282/300 Read block 4245 len 16384
19872 283/300 Read block 1647 len 12288
19873 284/300 Read block 2292 len 32768
19874 285/300 Read block 878 len 8192
19875 286/300 Read block 1699 len 32768
19876 287/300 Read block 1944 len 36864
19877 288/300 Read block 1241 len 4096
19878 289/300 Read block 2576 len 36864
19879 290/300 Flush
19880 291/300 Read block 489 len 8192
19881 292/300 Read block 3546 len 12288
19882 293/300 Read block 3086 len 40960
19883 294/300 Read block 4571 len 16384
19884 295/300 Read block 471 len 40960
19885 296/300 Read block 356 len 12288
19886 297/300 Read block 904 len 4096
19887 298/300 Write block 1780 len 32768 data: 4 3 4 3 3 2 2 3
19888 299/300 Read block 3901 len 12288
19889 300/300 Read block 60 len 20480
19890 301/300 Write block 1650 len 16384 data: 5 5 7 7
19891 Main task has work to do, go find it
19892 got dsc Start(0)
19893 start 0
19894 Jan 23 17:41:56.416 INFO request completed, latency_us: 227, response_code: 204, uri: /start/cid/0, method: POST, req_id: 32d4ca8f-48dd-488c-98d0-ea8e87f8df4b, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
19895 Replay: started 0, returned:Ok(())
19896 ----------------------------------------------------------------
19897 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19898 States: Offline Active Active
19899 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19900 6436 Acked Write 9 Sent Done Done false
19901 6437 Acked Read 6 Sent Done Done false
19902 6438 Acked Read 2 Sent Done Done false
19903 6439 Acked Read 2 Sent Done Done false
19904 6440 Acked Read 5 Sent Done Done false
19905 6441 Acked Read 5 Sent Done Done false
19906 6442 Acked Write 7 Sent Done Done false
19907 6443 Acked Read 3 Sent Done Done false
19908 6444 Acked Write 6 Sent Done Done false
19909 6445 Acked Write 4 Sent Done Done false
19910 6446 Acked Flush 0 Sent Done Done false
19911 6447 Acked Read 1 Sent Done Done false
19912 6448 Acked Read 4 Sent Done Done false
19913 6449 Acked Read 6 Sent Done Done false
19914 6450 Acked Write 7 Sent Done Done false
19915 6451 Acked Read 9 Sent Done Done false
19916 6452 Acked Write 3 Sent Done Done false
19917 6453 Acked Write 4 Sent Done Done false
19918 6454 Acked Read 1 Sent Done Done false
19919 6455 Acked Write 8 Sent Done Done false
19920 6456 Acked Read 10 Sent Done Done false
19921 6457 Acked Flush 0 Sent Done Done false
19922 6458 Acked Read 9 Sent Done Done false
19923 6459 Acked Write 2 Sent Done Done false
19924 6460 Acked Flush 0 Sent Done Done false
19925 6461 Acked Read 1 Sent Done Done false
19926 6462 Acked Read 1 Sent Done Done false
19927 6463 Acked Flush 0 Sent Done Done false
19928 6464 Acked Write 7 Sent Done Done false
19929 6465 Acked Read 1 Sent Done Done false
19930 6466 Acked Write 4 Sent Done Done false
19931 6467 Acked Write 6 Sent Done Done false
19932 6468 Acked Read 3 Sent Done Done false
19933 6469 Acked Read 9 Sent Done Done false
19934 6470 Acked Write 9 Sent Done Done false
19935 6471 Acked Write 6 Sent Done Done false
19936 6472 Acked Read 5 Sent Done Done false
19937 6473 Acked Read 6 Sent Done Done false
19938 6474 Acked Write 8 Sent Done Done false
19939 6475 Acked Read 6 Sent Done Done false
19940 6476 Acked Read 5 Sent Done Done false
19941 6477 Acked Read 10 Sent Done Done false
19942 6478 Acked Write 1 Sent Done Done false
19943 6479 Acked Flush 0 Sent Done Done false
19944 6480 Acked Read 4 Sent Done Done false
19945 6481 Acked Read 7 Sent Done Done false
19946 6482 Acked Write 3 Sent Done Done false
19947 6483 Acked Write 7 Sent Done Done false
19948 6484 Acked Write 4 Sent Done Done false
19949 6485 Acked Flush 0 Sent Done Done false
19950 6486 Acked Write 1 Sent Done Done false
19951 6487 Acked Read 9 Sent Done Done false
19952 6488 Acked Write 3 Sent Done Done false
19953 6489 Acked Read 2 Sent Done Done false
19954 6490 Acked Write 6 Sent Done Done false
19955 6491 Acked Read 9 Sent Done Done false
19956 6492 Acked Write 1 Sent Done Done false
19957 6493 Acked Write 2 Sent Done Done false
19958 6494 Acked Read 8 Sent Done Done false
19959 6495 Acked Read 8 Sent Done Done false
19960 6496 Acked Write 7 Sent Done Done false
19961 6497 Acked Read 2 Sent Done Done false
19962 6498 Acked Write 4 Sent Done Done false
19963 6499 Acked Read 3 Sent Done Done false
19964 6500 Acked Read 1 Sent Done Done false
19965 6501 Acked Read 10 Sent Done Done false
19966 6502 Acked Write 6 Sent Done Done false
19967 6503 Acked Read 2 Sent Done Done false
19968 6504 Acked Write 6 Sent Done Done false
19969 6505 Acked Read 7 Sent Done Done false
19970 6506 Acked Flush 0 Sent Done Done false
19971 6507 Acked Flush 0 Sent Done Done false
19972 6508 Acked Flush 0 Sent Done Done false
19973 6509 Acked Write 10 Sent Done Done false
19974 6510 Acked Write 1 Sent Done Done false
19975 6511 Acked Write 7 Sent Done Done false
19976 6512 Acked Write 9 Sent Done Done false
19977 6513 Acked Write 1 Sent Done Done false
19978 6514 Acked Write 2 Sent Done Done false
19979 6515 Acked Write 1 Sent Done Done false
19980 6516 Acked Read 9 Sent Done Done false
19981 6517 Acked Read 9 Sent Done Done false
19982 6518 Acked Flush 0 Sent Done Done false
19983 6519 Acked Read 1 Sent Done Done false
19984 6520 Acked Write 5 Sent Done Done false
19985 6521 Acked Write 3 Sent Done Done false
19986 6522 Acked Read 2 Sent Done Done false
19987 6523 Acked Write 2 Sent Done Done false
19988 6524 Acked Write 10 Sent Done Done false
19989 6525 Acked Read 3 Sent Done Done false
19990 6526 Acked Write 7 Sent Done Done false
19991 6527 Acked Read 5 Sent Done Done false
19992 6528 Acked Read 9 Sent Done Done false
19993 6529 Acked Read 5 Sent Done Done false
19994 6530 Acked Write 5 Sent Done Done false
19995 6531 Acked Write 2 Sent Done Done false
19996 6532 Acked Write 10 Sent Done Done false
19997 6533 Acked Write 10 Sent Done Done false
19998 6534 Acked Write 9 Sent Done Done false
19999 6535 Acked Read 10 Sent Done Done false
20000 6536 Acked Read 5 Sent Done Done false
20001 6537 Acked Read 3 Sent Done Done false
20002 6538 Acked Write 8 Sent Done Done false
20003 6539 Acked Write 3 Sent Done Done false
20004 6540 Acked Read 5 Sent Done Done false
20005 6541 Acked Write 1 Sent Done Done false
20006 6542 Acked Write 8 Sent Done Done false
20007 6543 Acked Read 5 Sent Done Done false
20008 6544 Acked Read 7 Sent Done Done false
20009 6545 Acked Read 6 Sent Done Done false
20010 6546 Acked Write 8 Sent Done Done false
20011 6547 Acked Write 10 Sent Done Done false
20012 6548 Acked Write 9 Sent Done Done false
20013 6549 Acked Read 1 Sent Done Done false
20014 6550 Acked Read 9 Sent Done Done false
20015 6551 Acked Read 1 Sent Done Done false
20016 6552 Acked Read 8 Sent Done Done false
20017 6553 Acked Read 6 Sent Done Done false
20018 6554 Acked Write 3 Sent Done Done false
20019 6555 Acked Write 2 Sent Done Done false
20020 6556 Acked Write 4 Sent Done Done false
20021 6557 Acked Read 2 Sent Done Done false
20022 6558 Acked Flush 0 Sent Done Done false
20023 6559 Acked Write 5 Sent Done Done false
20024 6560 Acked Read 9 Sent Done Done false
20025 6561 Acked Read 5 Sent Done Done false
20026 6562 Acked Flush 0 Sent Done Done false
20027 6563 Acked Read 5 Sent Done Done false
20028 6564 Acked Write 2 Sent Done Done false
20029 6565 Acked Read 2 Sent Done Done false
20030 6566 Acked Write 9 Sent Done Done false
20031 6567 Acked Read 2 Sent Done Done false
20032 6568 Acked Flush 0 Sent Done Done false
20033 6569 Acked Write 2 Sent Done Done false
20034 6570 Acked Write 1 Sent Done Done false
20035 6571 Acked Write 1 Sent Done Done false
20036 6572 Acked Read 10 Sent Done Done false
20037 6573 Acked Read 10 Sent Done Done false
20038 6574 Acked Read 5 Sent Done Done false
20039 6575 Acked Write 7 Sent Done Done false
20040 6576 Acked Read 7 Sent Done Done false
20041 6577 Acked Write 1 Sent Done Done false
20042 6578 Acked Read 6 Sent Done Done false
20043 6579 Acked Write 7 Sent Done Done false
20044 6580 Acked Read 9 Sent Done Done false
20045 6581 Acked Write 4 Sent Done Done false
20046 6582 Acked Write 8 Sent Done Done false
20047 6583 Acked Read 3 Sent Done Done false
20048 6584 Acked Write 9 Sent Done Done false
20049 6585 Acked Write 10 Sent Done Done false
20050 6586 Acked Read 7 Sent Done Done false
20051 6587 Acked Read 1 Sent Done Done false
20052 6588 Acked Read 6 Sent Done Done false
20053 6589 Acked Write 1 Sent Done Done false
20054 6590 Acked Write 10 Sent Done Done false
20055 6591 Acked Read 6 Sent Done Done false
20056 6592 Acked Read 9 Sent Done Done false
20057 6593 Acked Read 10 Sent Done Done false
20058 6594 Acked Write 8 Sent Done Done false
20059 6595 Acked Read 4 Sent Done Done false
20060 6596 Acked Read 9 Sent Done Done false
20061 6597 Acked Write 9 Sent Done Done false
20062 6598 Acked Flush 0 Sent Done Done false
20063 6599 Acked Flush 0 Sent Done Done false
20064 6600 Acked Read 4 Sent Done Done false
20065 6601 Acked Write 4 Sent Done Done false
20066 6602 Acked Write 6 Sent Done Done false
20067 6603 Acked Read 3 Sent Done Done false
20068 6604 Acked Write 2 Sent Done Done false
20069 6605 Acked Read 4 Sent Done Done false
20070 6606 Acked Read 5 Sent Done Done false
20071 6607 Acked Read 2 Sent Done Done false
20072 6608 Acked Read 7 Sent Done Done false
20073 6609 Acked Write 4 Sent Done Done false
20074 6610 Acked Read 10 Sent Done Done false
20075 6611 Acked Read 9 Sent Done Done false
20076 6612 Acked Read 6 Sent Done Done false
20077 6613 Acked Read 7 Sent Done Done false
20078 6614 Acked Read 10 Sent Done Done false
20079 6615 Acked Write 4 Sent Done Done false
20080 6616 Acked Read 3 Sent Done Done false
20081 6617 Acked Read 7 Sent Done Done false
20082 6618 Acked Read 8 Sent Done Done false
20083 6619 Acked Read 1 Sent Done Done false
20084 6620 Acked Write 5 Sent Done Done false
20085 6621 Acked Read 9 Sent Done Done false
20086 6622 Acked Read 8 Sent Done Done false
20087 6623 Acked Read 5 Sent Done Done false
20088 6624 Acked Flush 0 Sent Done Done false
20089 6625 Acked Write 9 Sent Done Done false
20090 6626 Acked Read 9 Sent Done Done false
20091 6627 Acked Write 7 Sent Done Done false
20092 6628 Acked Read 9 Sent Done Done false
20093 6629 Acked Write 5 Sent Done Done false
20094 6630 Acked Write 7 Sent Done Done false
20095 6631 Acked Read 8 Sent Done Done false
20096 6632 Acked Write 3 Sent Done Done false
20097 6633 Acked Flush 0 Sent Done Done false
20098 6634 Acked Read 9 Sent Done Done false
20099 6635 Acked Write 9 Sent Done Done false
20100 6636 Acked Read 10 Sent Done Done false
20101 6637 Acked Write 9 Sent Done Done false
20102 6638 Acked Read 2 Sent Done Done false
20103 6639 Acked Read 7 Sent Done Done false
20104 6640 Acked Read 6 Sent Done Done false
20105 6641 Acked Write 5 Sent Done Done false
20106 6642 Acked Write 2 Sent Done Done false
20107 6643 Acked Read 2 Sent Done Done false
20108 6644 Acked Read 3 Sent Done Done false
20109 6645 Acked Write 5 Sent Done Done false
20110 6646 Acked Read 5 Sent Done Done false
20111 6647 Acked Flush 0 Sent Done Done false
20112 6648 Acked Write 1 Sent Done Done false
20113 6649 Acked Flush 0 Sent Done Done false
20114 6650 Acked Read 4 Sent Done Done false
20115 6651 Acked Read 10 Sent Done Done false
20116 6652 Acked Read 6 Sent Done Done false
20117 6653 Acked Read 6 Sent Done Done false
20118 6654 Acked Write 1 Sent Done Done false
20119 6655 Acked Read 7 Sent Done Done false
20120 6656 Acked Read 4 Sent Done Done false
20121 6657 Acked Read 4 Sent Done Done false
20122 6658 Acked Read 9 Sent Done Done false
20123 6659 Acked Flush 0 Sent Done Done false
20124 6660 Acked Read 8 Sent Done Done false
20125 6661 Acked Write 10 Sent Done Done false
20126 6662 Acked Write 7 Sent Done Done false
20127 6663 Acked Read 4 Sent Done Done false
20128 6664 Acked Read 1 Sent Done Done false
20129 6665 Acked Read 2 Sent Done Done false
20130 6666 Acked Write 9 Sent Done Done false
20131 6667 Acked Write 1 Sent Done Done false
20132 6668 Acked Write 10 Sent Done Done false
20133 6669 Acked Read 10 Sent Done Done false
20134 6670 Acked Flush 0 Sent Done Done false
20135 6671 Acked Write 6 Sent Done Done false
20136 6672 Acked Read 4 Sent Done Done false
20137 6673 Acked Read 3 Sent Done Done false
20138 6674 Acked Write 4 Sent Done Done false
20139 6675 Acked Read 9 Sent Done Done false
20140 6676 Acked Write 2 Sent Done Done false
20141 6677 Acked Read 3 Sent Done Done false
20142 6678 Acked Read 7 Sent Done Done false
20143 6679 Acked Read 6 Sent Done Done false
20144 6680 Acked Read 10 Sent Done Done false
20145 6681 Acked Read 6 Sent Done Done false
20146 6682 Acked Write 10 Sent Done Done false
20147 6683 Acked Read 5 Sent Done Done false
20148 6684 Acked Read 1 Sent Done Done false
20149 6685 Acked Read 2 Sent Done Done false
20150 6686 Acked Write 8 Sent Done Done false
20151 6687 Acked Write 6 Sent Done Done false
20152 6688 Acked Write 7 Sent Done Done false
20153 6689 Acked Read 4 Sent Done Done false
20154 6690 Acked Write 8 Sent Done Done false
20155 6691 Acked Read 3 Sent Done Done false
20156 6692 Acked Write 6 Sent Done Done false
20157 6693 Acked Write 1 Sent Done Done false
20158 6694 Acked Write 5 Sent Done Done false
20159 6695 Acked Read 3 Sent Done Done false
20160 6696 Acked Flush 0 Sent Done Done false
20161 6697 Acked Write 1 Sent Done Done false
20162 6698 Acked Write 5 Sent Done Done false
20163 6699 Acked Flush 0 Sent Done Done false
20164 6700 Acked Read 10 Sent Done Done false
20165 6701 Acked Write 2 Sent Done Done false
20166 6702 Acked Write 4 Sent Done Done false
20167 6703 Acked Read 1 Sent Done Done false
20168 6704 Acked Flush 0 Sent Done Done false
20169 6705 Acked Read 10 Sent Done Done false
20170 6706 Acked Read 1 Sent Done Done false
20171 6707 Acked Flush 0 Sent Done Done false
20172 6708 Acked Read 2 Sent Done Done false
20173 6709 Acked Read 2 Sent Done Done false
20174 6710 Acked Write 4 Sent Done Done false
20175 6711 Acked Write 7 Sent Done Done false
20176 6712 Acked Read 4 Sent Done Done false
20177 6713 Acked Write 2 Sent Done Done false
20178 6714 Acked Read 9 Sent Done Done false
20179 6715 Acked Flush 0 Sent Done Done false
20180 6716 Acked Read 8 Sent Done Done false
20181 6717 Acked Read 4 Sent Done Done false
20182 6718 Acked Read 3 Sent Done Done false
20183 6719 Acked Read 8 Sent Done Done false
20184 6720 Acked Read 2 Sent Done Done false
20185 6721 Acked Read 8 Sent Done Done false
20186 6722 Acked Read 9 Sent Done Done false
20187 6723 Acked Read 1 Sent Done Done false
20188 6724 Acked Read 9 Sent Done Done false
20189 6725 Acked Flush 0 Sent Done Done false
20190 6726 Acked Read 2 Sent Done Done false
20191 6727 Acked Read 3 Sent Done Done false
20192 6728 Acked Read 10 Sent Done Done false
20193 6729 Acked Read 4 Sent Done Done false
20194 6730 Acked Read 10 Sent Done Done false
20195 6731 Acked Read 3 Sent Done Done false
20196 6732 Acked Read 1 Sent Done Done false
20197 6733 Acked Write 8 Sent Done Done false
20198 6734 Acked Read 3 Sent Done Done false
20199 6735 Acked Read 5 Sent Done Done false
20200 6736 Acked Write 4 Sent Done Done false
20201 STATES DS:0 DS:1 DS:2 TOTAL
20202 Sent 301 0 0 301
20203 Done 0 301 301 602
20204 Skipped 0 0 0 0
20205 Error 0 0 0 0
20206 Last Flush: Some(JobId(6435)) Some(JobId(6725)) Some(JobId(6725))
20207 Downstairs last five completed: 6435 6434 6433 6432 6431
20208 Upstairs last five completed: 6736 6735 6734 6733 6732
20209 CLIENT: Up:0 ds:301 act:2
20210 [8810] got start action
20211 [8810] I am going to restart
20212 Starting downstairs at port 8810
20213 Make output file at "/tmp/dsc/downstairs-8810.txt"
20214 [8810][0] reports Starting
20215 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1224)
20216 [8810][0] reports Running
20217 ----------------------------------------------------------------
20218 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
20219 States: Offline Active Active
20220 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20221 6436 Acked Write 9 Sent Done Done false
20222 6437 Acked Read 6 Sent Done Done false
20223 6438 Acked Read 2 Sent Done Done false
20224 6439 Acked Read 2 Sent Done Done false
20225 6440 Acked Read 5 Sent Done Done false
20226 6441 Acked Read 5 Sent Done Done false
20227 6442 Acked Write 7 Sent Done Done false
20228 6443 Acked Read 3 Sent Done Done false
20229 6444 Acked Write 6 Sent Done Done false
20230 6445 Acked Write 4 Sent Done Done false
20231 6446 Acked Flush 0 Sent Done Done false
20232 6447 Acked Read 1 Sent Done Done false
20233 6448 Acked Read 4 Sent Done Done false
20234 6449 Acked Read 6 Sent Done Done false
20235 6450 Acked Write 7 Sent Done Done false
20236 6451 Acked Read 9 Sent Done Done false
20237 6452 Acked Write 3 Sent Done Done false
20238 6453 Acked Write 4 Sent Done Done false
20239 6454 Acked Read 1 Sent Done Done false
20240 6455 Acked Write 8 Sent Done Done false
20241 6456 Acked Read 10 Sent Done Done false
20242 6457 Acked Flush 0 Sent Done Done false
20243 6458 Acked Read 9 Sent Done Done false
20244 6459 Acked Write 2 Sent Done Done false
20245 6460 Acked Flush 0 Sent Done Done false
20246 6461 Acked Read 1 Sent Done Done false
20247 6462 Acked Read 1 Sent Done Done false
20248 6463 Acked Flush 0 Sent Done Done false
20249 6464 Acked Write 7 Sent Done Done false
20250 6465 Acked Read 1 Sent Done Done false
20251 6466 Acked Write 4 Sent Done Done false
20252 6467 Acked Write 6 Sent Done Done false
20253 6468 Acked Read 3 Sent Done Done false
20254 6469 Acked Read 9 Sent Done Done false
20255 6470 Acked Write 9 Sent Done Done false
20256 6471 Acked Write 6 Sent Done Done false
20257 6472 Acked Read 5 Sent Done Done false
20258 6473 Acked Read 6 Sent Done Done false
20259 6474 Acked Write 8 Sent Done Done false
20260 6475 Acked Read 6 Sent Done Done false
20261 6476 Acked Read 5 Sent Done Done false
20262 6477 Acked Read 10 Sent Done Done false
20263 6478 Acked Write 1 Sent Done Done false
20264 6479 Acked Flush 0 Sent Done Done false
20265 6480 Acked Read 4 Sent Done Done false
20266 6481 Acked Read 7 Sent Done Done false
20267 6482 Acked Write 3 Sent Done Done false
20268 6483 Acked Write 7 Sent Done Done false
20269 6484 Acked Write 4 Sent Done Done false
20270 6485 Acked Flush 0 Sent Done Done false
20271 6486 Acked Write 1 Sent Done Done false
20272 6487 Acked Read 9 Sent Done Done false
20273 6488 Acked Write 3 Sent Done Done false
20274 6489 Acked Read 2 Sent Done Done false
20275 6490 Acked Write 6 Sent Done Done false
20276 6491 Acked Read 9 Sent Done Done false
20277 6492 Acked Write 1 Sent Done Done false
20278 6493 Acked Write 2 Sent Done Done false
20279 6494 Acked Read 8 Sent Done Done false
20280 6495 Acked Read 8 Sent Done Done false
20281 6496 Acked Write 7 Sent Done Done false
20282 6497 Acked Read 2 Sent Done Done false
20283 6498 Acked Write 4 Sent Done Done false
20284 6499 Acked Read 3 Sent Done Done false
20285 6500 Acked Read 1 Sent Done Done false
20286 6501 Acked Read 10 Sent Done Done false
20287 6502 Acked Write 6 Sent Done Done false
20288 6503 Acked Read 2 Sent Done Done false
20289 6504 Acked Write 6 Sent Done Done false
20290 6505 Acked Read 7 Sent Done Done false
20291 6506 Acked Flush 0 Sent Done Done false
20292 6507 Acked Flush 0 Sent Done Done false
20293 6508 Acked Flush 0 Sent Done Done false
20294 6509 Acked Write 10 Sent Done Done false
20295 6510 Acked Write 1 Sent Done Done false
20296 6511 Acked Write 7 Sent Done Done false
20297 6512 Acked Write 9 Sent Done Done false
20298 6513 Acked Write 1 Sent Done Done false
20299 6514 Acked Write 2 Sent Done Done false
20300 6515 Acked Write 1 Sent Done Done false
20301 6516 Acked Read 9 Sent Done Done false
20302 6517 Acked Read 9 Sent Done Done false
20303 6518 Acked Flush 0 Sent Done Done false
20304 6519 Acked Read 1 Sent Done Done false
20305 6520 Acked Write 5 Sent Done Done false
20306 6521 Acked Write 3 Sent Done Done false
20307 6522 Acked Read 2 Sent Done Done false
20308 6523 Acked Write 2 Sent Done Done false
20309 6524 Acked Write 10 Sent Done Done false
20310 6525 Acked Read 3 Sent Done Done false
20311 6526 Acked Write 7 Sent Done Done false
20312 6527 Acked Read 5 Sent Done Done false
20313 6528 Acked Read 9 Sent Done Done false
20314 6529 Acked Read 5 Sent Done Done false
20315 6530 Acked Write 5 Sent Done Done false
20316 6531 Acked Write 2 Sent Done Done false
20317 6532 Acked Write 10 Sent Done Done false
20318 6533 Acked Write 10 Sent Done Done false
20319 6534 Acked Write 9 Sent Done Done false
20320 6535 Acked Read 10 Sent Done Done false
20321 6536 Acked Read 5 Sent Done Done false
20322 6537 Acked Read 3 Sent Done Done false
20323 6538 Acked Write 8 Sent Done Done false
20324 6539 Acked Write 3 Sent Done Done false
20325 6540 Acked Read 5 Sent Done Done false
20326 6541 Acked Write 1 Sent Done Done false
20327 6542 Acked Write 8 Sent Done Done false
20328 6543 Acked Read 5 Sent Done Done false
20329 6544 Acked Read 7 Sent Done Done false
20330 6545 Acked Read 6 Sent Done Done false
20331 6546 Acked Write 8 Sent Done Done false
20332 6547 Acked Write 10 Sent Done Done false
20333 6548 Acked Write 9 Sent Done Done false
20334 6549 Acked Read 1 Sent Done Done false
20335 6550 Acked Read 9 Sent Done Done false
20336 6551 Acked Read 1 Sent Done Done false
20337 6552 Acked Read 8 Sent Done Done false
20338 6553 Acked Read 6 Sent Done Done false
20339 6554 Acked Write 3 Sent Done Done false
20340 6555 Acked Write 2 Sent Done Done false
20341 6556 Acked Write 4 Sent Done Done false
20342 6557 Acked Read 2 Sent Done Done false
20343 6558 Acked Flush 0 Sent Done Done false
20344 6559 Acked Write 5 Sent Done Done false
20345 6560 Acked Read 9 Sent Done Done false
20346 6561 Acked Read 5 Sent Done Done false
20347 6562 Acked Flush 0 Sent Done Done false
20348 6563 Acked Read 5 Sent Done Done false
20349 6564 Acked Write 2 Sent Done Done false
20350 6565 Acked Read 2 Sent Done Done false
20351 6566 Acked Write 9 Sent Done Done false
20352 6567 Acked Read 2 Sent Done Done false
20353 6568 Acked Flush 0 Sent Done Done false
20354 6569 Acked Write 2 Sent Done Done false
20355 6570 Acked Write 1 Sent Done Done false
20356 6571 Acked Write 1 Sent Done Done false
20357 6572 Acked Read 10 Sent Done Done false
20358 6573 Acked Read 10 Sent Done Done false
20359 6574 Acked Read 5 Sent Done Done false
20360 6575 Acked Write 7 Sent Done Done false
20361 6576 Acked Read 7 Sent Done Done false
20362 6577 Acked Write 1 Sent Done Done false
20363 6578 Acked Read 6 Sent Done Done false
20364 6579 Acked Write 7 Sent Done Done false
20365 6580 Acked Read 9 Sent Done Done false
20366 6581 Acked Write 4 Sent Done Done false
20367 6582 Acked Write 8 Sent Done Done false
20368 6583 Acked Read 3 Sent Done Done false
20369 6584 Acked Write 9 Sent Done Done false
20370 6585 Acked Write 10 Sent Done Done false
20371 6586 Acked Read 7 Sent Done Done false
20372 6587 Acked Read 1 Sent Done Done false
20373 6588 Acked Read 6 Sent Done Done false
20374 6589 Acked Write 1 Sent Done Done false
20375 6590 Acked Write 10 Sent Done Done false
20376 6591 Acked Read 6 Sent Done Done false
20377 6592 Acked Read 9 Sent Done Done false
20378 6593 Acked Read 10 Sent Done Done false
20379 6594 Acked Write 8 Sent Done Done false
20380 6595 Acked Read 4 Sent Done Done false
20381 6596 Acked Read 9 Sent Done Done false
20382 6597 Acked Write 9 Sent Done Done false
20383 6598 Acked Flush 0 Sent Done Done false
20384 6599 Acked Flush 0 Sent Done Done false
20385 6600 Acked Read 4 Sent Done Done false
20386 6601 Acked Write 4 Sent Done Done false
20387 6602 Acked Write 6 Sent Done Done false
20388 6603 Acked Read 3 Sent Done Done false
20389 6604 Acked Write 2 Sent Done Done false
20390 6605 Acked Read 4 Sent Done Done false
20391 6606 Acked Read 5 Sent Done Done false
20392 6607 Acked Read 2 Sent Done Done false
20393 6608 Acked Read 7 Sent Done Done false
20394 6609 Acked Write 4 Sent Done Done false
20395 6610 Acked Read 10 Sent Done Done false
20396 6611 Acked Read 9 Sent Done Done false
20397 6612 Acked Read 6 Sent Done Done false
20398 6613 Acked Read 7 Sent Done Done false
20399 6614 Acked Read 10 Sent Done Done false
20400 6615 Acked Write 4 Sent Done Done false
20401 6616 Acked Read 3 Sent Done Done false
20402 6617 Acked Read 7 Sent Done Done false
20403 6618 Acked Read 8 Sent Done Done false
20404 6619 Acked Read 1 Sent Done Done false
20405 6620 Acked Write 5 Sent Done Done false
20406 6621 Acked Read 9 Sent Done Done false
20407 6622 Acked Read 8 Sent Done Done false
20408 6623 Acked Read 5 Sent Done Done false
20409 6624 Acked Flush 0 Sent Done Done false
20410 6625 Acked Write 9 Sent Done Done false
20411 6626 Acked Read 9 Sent Done Done false
20412 6627 Acked Write 7 Sent Done Done false
20413 6628 Acked Read 9 Sent Done Done false
20414 6629 Acked Write 5 Sent Done Done false
20415 6630 Acked Write 7 Sent Done Done false
20416 6631 Acked Read 8 Sent Done Done false
20417 6632 Acked Write 3 Sent Done Done false
20418 6633 Acked Flush 0 Sent Done Done false
20419 6634 Acked Read 9 Sent Done Done false
20420 6635 Acked Write 9 Sent Done Done false
20421 6636 Acked Read 10 Sent Done Done false
20422 6637 Acked Write 9 Sent Done Done false
20423 6638 Acked Read 2 Sent Done Done false
20424 6639 Acked Read 7 Sent Done Done false
20425 6640 Acked Read 6 Sent Done Done false
20426 6641 Acked Write 5 Sent Done Done false
20427 6642 Acked Write 2 Sent Done Done false
20428 6643 Acked Read 2 Sent Done Done false
20429 6644 Acked Read 3 Sent Done Done false
20430 6645 Acked Write 5 Sent Done Done false
20431 6646 Acked Read 5 Sent Done Done false
20432 6647 Acked Flush 0 Sent Done Done false
20433 6648 Acked Write 1 Sent Done Done false
20434 6649 Acked Flush 0 Sent Done Done false
20435 6650 Acked Read 4 Sent Done Done false
20436 6651 Acked Read 10 Sent Done Done false
20437 6652 Acked Read 6 Sent Done Done false
20438 6653 Acked Read 6 Sent Done Done false
20439 6654 Acked Write 1 Sent Done Done false
20440 6655 Acked Read 7 Sent Done Done false
20441 6656 Acked Read 4 Sent Done Done false
20442 6657 Acked Read 4 Sent Done Done false
20443 6658 Acked Read 9 Sent Done Done false
20444 6659 Acked Flush 0 Sent Done Done false
20445 6660 Acked Read 8 Sent Done Done false
20446 6661 Acked Write 10 Sent Done Done false
20447 6662 Acked Write 7 Sent Done Done false
20448 6663 Acked Read 4 Sent Done Done false
20449 6664 Acked Read 1 Sent Done Done false
20450 6665 Acked Read 2 Sent Done Done false
20451 6666 Acked Write 9 Sent Done Done false
20452 6667 Acked Write 1 Sent Done Done false
20453 6668 Acked Write 10 Sent Done Done false
20454 6669 Acked Read 10 Sent Done Done false
20455 6670 Acked Flush 0 Sent Done Done false
20456 6671 Acked Write 6 Sent Done Done false
20457 6672 Acked Read 4 Sent Done Done false
20458 6673 Acked Read 3 Sent Done Done false
20459 6674 Acked Write 4 Sent Done Done false
20460 6675 Acked Read 9 Sent Done Done false
20461 6676 Acked Write 2 Sent Done Done false
20462 6677 Acked Read 3 Sent Done Done false
20463 6678 Acked Read 7 Sent Done Done false
20464 6679 Acked Read 6 Sent Done Done false
20465 6680 Acked Read 10 Sent Done Done false
20466 6681 Acked Read 6 Sent Done Done false
20467 6682 Acked Write 10 Sent Done Done false
20468 6683 Acked Read 5 Sent Done Done false
20469 6684 Acked Read 1 Sent Done Done false
20470 6685 Acked Read 2 Sent Done Done false
20471 6686 Acked Write 8 Sent Done Done false
20472 6687 Acked Write 6 Sent Done Done false
20473 6688 Acked Write 7 Sent Done Done false
20474 6689 Acked Read 4 Sent Done Done false
20475 6690 Acked Write 8 Sent Done Done false
20476 6691 Acked Read 3 Sent Done Done false
20477 6692 Acked Write 6 Sent Done Done false
20478 6693 Acked Write 1 Sent Done Done false
20479 6694 Acked Write 5 Sent Done Done false
20480 6695 Acked Read 3 Sent Done Done false
20481 6696 Acked Flush 0 Sent Done Done false
20482 6697 Acked Write 1 Sent Done Done false
20483 6698 Acked Write 5 Sent Done Done false
20484 6699 Acked Flush 0 Sent Done Done false
20485 6700 Acked Read 10 Sent Done Done false
20486 6701 Acked Write 2 Sent Done Done false
20487 6702 Acked Write 4 Sent Done Done false
20488 6703 Acked Read 1 Sent Done Done false
20489 6704 Acked Flush 0 Sent Done Done false
20490 6705 Acked Read 10 Sent Done Done false
20491 6706 Acked Read 1 Sent Done Done false
20492 6707 Acked Flush 0 Sent Done Done false
20493 6708 Acked Read 2 Sent Done Done false
20494 6709 Acked Read 2 Sent Done Done false
20495 6710 Acked Write 4 Sent Done Done false
20496 6711 Acked Write 7 Sent Done Done false
20497 6712 Acked Read 4 Sent Done Done false
20498 6713 Acked Write 2 Sent Done Done false
20499 6714 Acked Read 9 Sent Done Done false
20500 6715 Acked Flush 0 Sent Done Done false
20501 6716 Acked Read 8 Sent Done Done false
20502 6717 Acked Read 4 Sent Done Done false
20503 6718 Acked Read 3 Sent Done Done false
20504 6719 Acked Read 8 Sent Done Done false
20505 6720 Acked Read 2 Sent Done Done false
20506 6721 Acked Read 8 Sent Done Done false
20507 6722 Acked Read 9 Sent Done Done false
20508 6723 Acked Read 1 Sent Done Done false
20509 6724 Acked Read 9 Sent Done Done false
20510 6725 Acked Flush 0 Sent Done Done false
20511 6726 Acked Read 2 Sent Done Done false
20512 6727 Acked Read 3 Sent Done Done false
20513 6728 Acked Read 10 Sent Done Done false
20514 6729 Acked Read 4 Sent Done Done false
20515 6730 Acked Read 10 Sent Done Done false
20516 6731 Acked Read 3 Sent Done Done false
20517 6732 Acked Read 1 Sent Done Done false
20518 6733 Acked Write 8 Sent Done Done false
20519 6734 Acked Read 3 Sent Done Done false
20520 6735 Acked Read 5 Sent Done Done false
20521 6736 Acked Write 4 Sent Done Done false
20522 6737 Acked Flush 0 Sent Done Done false
20523 STATES DS:0 DS:1 DS:2 TOTAL
20524 Sent 302 0 0 302
20525 Done 0 302 302 604
20526 Skipped 0 0 0 0
20527 Error 0 0 0 0
20528 Last Flush: Some(JobId(6435)) Some(JobId(6737)) Some(JobId(6737))
20529 Downstairs last five completed: 6435 6434 6433 6432 6431
20530 Upstairs last five completed: 6737 6736 6735 6734 6733
20531 CLIENT: Up:0 ds:302 act:2
205322025-01-23T17:42:02.170ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
205332025-01-23T17:42:02.170ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50352) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
205342025-01-23T17:42:02.171ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
205352025-01-23T17:42:02.172ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
205362025-01-23T17:42:02.172ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
205372025-01-23T17:42:02.172ZINFOcrucible: send last flush ID to this DS: Some(JobId(6435)) = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
205382025-01-23T17:42:02.172ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
205392025-01-23T17:42:02.172ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(6435)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
205402025-01-23T17:42:02.172ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(6435)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
20541 ----------------------------------------------------------------
20542 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20543 Downstairs last five completed: 6737 6736 6735 6734 6733
20544 Upstairs last five completed: 6737 6736 6735 6734 6733
20545 CLIENT: Up:0 ds:0 act:3
20546 Replay: All jobs finished
20547 Main task has work to do, go find it
20548 got dsc Stop(2)
20549 stop 2
20550 Jan 23 17:42:04.422 INFO request completed, latency_us:[8830] Got stop action so:false kr:false
20551 378, response_code: 204, uri: /stop/cid/2, method: POST, req_id: c46ce32d-7f95-47e9-8f70-3e7ee39f3bfb, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
205522025-01-23T17:42:04.423ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
205532025-01-23T17:42:04.423ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
20554 Jan 23 17:42:04.423{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:42:04.423107724Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
20555 INFO request completed, latency_us: 162, response_code: 200, uri: /state/cid/2, method: GET, req_id: f0a2b4f7-9b02-471b-9e48-c9aec5fd93de, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
205562025-01-23T17:42:04.423ZINFOcrucible: client task is exiting = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
20557 [8830] Exited with: ExitStatus(unix_wait_status(9))
20558 [8830][2] reports Exit
20559 Jan 23 17:42:08.425 INFO request completed, latency_us: 263, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4d961032-e9ea-4756-b680-e4a3c73f23cf, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
20560 001/300 Write block 4969 len 8192 data: 7 7
20561 002/300 Read block 99 len 8192
20562 003/300 Read block 3097 len 36864
20563 004/300 Read block 2711 len 32768
20564 005/300 Read block 4572 len 32768
20565 006/300 Read block 4736 len 16384
20566 007/300 Flush
20567 008/300 Write block 2686 len 24576 data: 5 5 6 3 3 2
20568 009/300 Read block 3430 len 36864
20569 010/300 Flush
20570 011/300 Read block 3947 len 16384
20571 012/300 Read block 821 len 8192
20572 013/300 Write block 2740 len 24576 data: 5 5 5 5 5 8
20573 014/300 Write block 2417 len 8192 data: 7 6
20574 015/300 Flush
20575 016/300 Read block 4125 len 8192
20576 017/300 Read block 2641 len 28672
20577 018/300 Flush
20578 019/300 Read block 2918 len 24576
20579 020/300 Flush
20580 021/300 Flush
20581 022/300 Read block 3380 len 12288
20582 023/300 Flush
20583 024/300 Read block 3721 len 28672
20584 025/300 Read block 2603 len 32768
20585 026/300 Write block 773 len 28672 data: 6 6 5 5 5 7 6
20586 027/300 Flush
20587 028/300 Read block 3736 len 4096
20588 029/300 Read block 4303 len 20480
20589 030/300 Read block 1075 len 32768
20590 031/300 Read block 128 len 28672
20591 032/300 Write block 2836 len 32768 data: 5 7 6 7 7 8 7 6
20592 033/300 Write block 3421 len 20480 data: 6 6 6 5 6
20593 034/300 Flush
20594 035/300 Read block 4075 len 16384
20595 036/300 Read block 820 len 40960
20596 037/300 Write block 992 len 4096 data: 8
20597 038/300 Read block 2880 len 20480
20598 039/300 Read block 1415 len 36864
20599 040/300 Read block 4732 len 8192
20600 041/300 Write block 2686 len 40960 data: 6 6 7 4 4 3 2 2 2 4
20601 042/300 Read block 3231 len 16384
20602 043/300 Read block 2687 len 20480
20603 044/300 Write block 2757 len 28672 data: 2 3 3 3 4 4 3
20604 045/300 Write block 1189 len 12288 data: 4 3 4
20605 046/300 Write block 1598 len 4096 data: 4
20606 047/300 Read block 1675 len 24576
20607 048/300 Write block 4426 len 4096 data: 5
20608 049/300 Read block 3000 len 8192
20609 050/300 Read block 3278 len 24576
20610 051/300 Read block 2028 len 16384
20611 052/300 Write block 12 len 8192 data: 3 4
20612 053/300 Read block 756 len 16384
20613 054/300 Write block 1311 len 24576 data: 8 6 6 5 5 6
20614 055/300 Read block 2412 len 4096
20615 056/300 Write block 1616 len 16384 data: 8 7 8 10
20616 057/300 Write block 3241 len 12288 data: 3 4 3
20617 058/300 Read block 251 len 36864
20618 059/300 Read block 1791 len 20480
20619 060/300 Read block 3675 len 8192
20620 061/300 Read block 2431 len 8192
20621 062/300 Write block 3029 len 32768 data: 4 4 6 7 8 7 7 6
20622 063/300 Read block 801 len 20480
20623 064/300 Read block 1309 len 28672
20624 065/300 Read block 3925 len 40960
20625 066/300 Write block 1693 len 40960 data: 7 6 6 3 4 5 5 5 4 3
20626 067/300 Write block 2462 len 8192 data: 4 4
20627 068/300 Read block 982 len 20480
20628 069/300 Flush
20629 070/300 Write block 3255 len 28672 data: 5 3 3 3 3 2 2
20630 071/300 Flush
20631 072/300 Read block 4672 len 16384
20632 073/300 Write block 1374 len 28672 data: 5 5 5 4 3 2 2
20633 074/300 Flush
20634 075/300 Write block 555 len 16384 data: 5 7 6 4
20635 076/300 Write block 2892 len 16384 data: 4 4 4 4
20636 077/300 Read block 3178 len 36864
20637 078/300 Read block 4669 len 4096
20638 079/300 Read block 4088 len 16384
20639 080/300 Read block 1935 len 4096
20640 081/300 Write block 4350 len 16384 data: 3 3 4 3
20641 082/300 Read block 2780 len 32768
20642 083/300 Read block 702 len 20480
20643 084/300 Write block 1448 len 36864 data: 3 4 5 5 5 4 5 5 5
20644 085/300 Read block 338 len 20480
20645 086/300 Read block 470 len 20480
20646 087/300 Read block 586 len 40960
20647 088/300 Read block 4643 len 8192
20648 089/300 Write block 1848 len 40960 data: 3 4 3 3 3 3 4 4 4 5
20649 090/300 Flush
20650 091/300 Write block 1001 len 32768 data: 2 4 2 4 6 6 6 5
20651 092/300 Write block 4113 len 32768 data: 4 4 4 4 5 5 4 3
20652 093/300 Read block 512 len 36864
20653 094/300 Flush
20654 095/300 Write block 3042 len 4096 data: 4
20655 096/300 Read block 264 len 4096
20656 097/300 Read block 209 len 8192
20657 098/300 Read block 1471 len 32768
20658 099/300 Write block 946 len 24576 data: 5 4 4 4 4 4
20659 100/300 Read block 877 len 12288
20660 101/300 Read block 4192 len 8192
20661 102/300 Read block 505 len 32768
20662 103/300 Flush
20663 104/300 Write block 1199 len 8192 data: 5 6
20664 105/300 Read block 3452 len 40960
20665 106/300 Read block 1014 len 20480
20666 107/300 Read block 1790 len 12288
20667 108/300 Write block 2378 len 8192 data: 7 6
20668 109/300 Read block 3083 len 40960
20669 110/300 Write block 3089 len 4096 data: 4
20670 111/300 Write block 3728 len 12288 data: 5 6 4
20671 112/300 Read block 4494 len 40960
20672 113/300 Read block 3672 len 16384
20673 114/300 Read block 1586 len 4096
20674 115/300 Read block 3150 len 28672
20675 116/300 Write block 2725 len 36864 data: 3 4 5 5 6 6 6 5 5
20676 117/300 Write block 4061 len 36864 data: 4 4 4 4 2 3 2 2 2
20677 118/300 Read block 1597 len 32768
20678 119/300 Read block 2777 len 4096
20679 120/300 Write block 4554 len 32768 data: 3 4 7 7 8 5 4 5
20680 121/300 Read block 3766 len 4096
20681 122/300 Read block 3241 len 12288
20682 123/300 Write block 762 len 16384 data: 3 3 4 4
20683 124/300 Write block 2153 len 36864 data: 5 5 5 4 4 5 5 5 5
20684 125/300 Read block 994 len 36864
20685 126/300 Flush
20686 127/300 Write block 3940 len 28672 data: 4 5 5 6 5 4 5
20687 128/300 Read block 4960 len 28672
20688 129/300 Write block 1492 len 36864 data: 3 5 4 2 2 3 3 5 6
20689 130/300 Read block 4376 len 24576
20690 131/300 Flush
20691 132/300 Read block 3622 len 32768
20692 133/300 Write block 3027 len 32768 data: 6 5 5 5 7 8 9 8
20693 134/300 Flush
20694 135/300 Write block 3014 len 32768 data: 7 7 7 8 7 7 5 5
20695 136/300 Read block 3529 len 4096
20696 137/300 Write block 3193 len 8192 data: 3 3
20697 138/300 Write block 3371 len 4096 data: 3
20698 139/300 Read block 4948 len 4096
20699 140/300 Flush
20700 141/300 Read block 4494 len 8192
20701 142/300 Flush
20702 143/300 Read block 3733 len 4096
20703 144/300 Read block 626 len 32768
20704 145/300 Read block 2828 len 32768
20705 146/300 Read block 3859 len 28672
20706 147/300 Write block 735 len 12288 data: 5 5 3
20707 148/300 Write block 2813 len 4096 data: 8
20708 149/300 Flush
20709 150/300 Write block 293 len 16384 data: 6 8 9 7
20710 151/300 Read block 4443 len 16384
20711 152/300 Write block 784 len 40960 data: 9 9 7 5 4 4 4 3 3 3
20712 153/300 Read block 364 len 40960
20713 154/300 Read block 1851 len 36864
20714 155/300 Read block 4892 len 4096
20715 156/300 Write block 904 len 8192 data: 4 4
20716 157/300 Read block 138 len 40960
20717 158/300 Read block 2852 len 28672
20718 159/300 Read block 4498 len 20480
20719 160/300 Read block 1483 len 12288
20720 161/300 Write block 2561 len 8192 data: 2 2
20721 162/300 Write block 2560 len 36864 data: 2 3 3 2 3 2 2 3 3
20722 163/300 Write block 3685 len 36864 data: 7 4 4 4 4 3 4 5 4
20723 164/300 Write block 4391 len 40960 data: 3 3 3 4 4 3 3 4 4 4
20724 165/300 Read block 4590 len 40960
20725 166/300 Read block 2872 len 20480
20726 167/300 Read block 4544 len 40960
20727 168/300 Write block 4552 len 36864 data: 4 3 4 5 8 8 9 6 5
20728 169/300 Flush
20729 170/300 Read block 3961 len 4096
20730 171/300 Write block 4628 len 40960 data: 6 6 7 5 5 5 4 4 3 4
20731 172/300 Read block 539 len 24576
20732 173/300 Read block 4311 len 36864
20733 174/300 Read block 417 len 8192
20734 175/300 Read block 39 len 16384
20735 176/300 Write block 2019 len 40960 data: 3 4 4 4 5 4 4 3 3 2
20736 177/300 Read block 1691 len 16384
20737 178/300 Read block 375 len 16384
20738 179/300 Write block 2477 len 36864 data: 5 5 6 5 5 6 6 6 6
20739 180/300 Write block 1442 len 24576 data: 3 2 2 2 3 3
20740 181/300 Read block 3742 len 40960
20741 182/300 Write block 334 len 28672 data: 5 4 3 4 4 4 4
20742 183/300 Read block 2687 len 40960
20743 184/300 Read block 4110 len 24576
20744 185/300 Flush
20745 186/300 Read block 871 len 8192
20746 187/300 Write block 1081 len 32768 data: 3 4 5 5 5 5 6 7
20747 188/300 Read block 2739 len 40960
20748 189/300 Write block 3379 len 12288 data: 3 3 3
20749 190/300 Read block 3019 len 16384
20750 191/300 Write block 206 len 40960 data: 6 2 3 3 5 6 6 6 6 5
20751 192/300 Read block 1610 len 20480
20752 193/300 Read block 4752 len 12288
20753 194/300 Flush
20754 195/300 Read block 37 len 20480
20755 196/300 Read block 2294 len 12288
20756 197/300 Read block 4651 len 28672
20757 198/300 Write block 4158 len 12288 data: 4 3 3
20758 199/300 Read block 2725 len 8192
20759 200/300 Write block 1609 len 40960 data: 4 5 6 7 7 6 7 9 8 9
20760 201/300 Write block 2805 len 12288 data: 6 6 5
20761 202/300 Read block 323 len 28672
20762 203/300 Flush
20763 204/300 Flush
20764 205/300 Write block 2606 len 32768 data: 5 5 5 3 5 6 7 6
20765 206/300 Write block 1742 len 32768 data: 4 4 3 3 3 3 2 3
20766 207/300 Write block 279 len 36864 data: 7 6 4 5 4 4 5 4 6
20767 208/300 Write block 2203 len 36864 data: 5 4 4 4 5 6 5 5 4
20768 209/300 Read block 4694 len 32768
20769 210/300 Read block 3401 len 12288
20770 211/300 Write block 960 len 32768 data: 4 4 5 3 4 4 4 3
20771 212/300 Write block 3231 len 28672 data: 4 5 6 7 6 6 6
20772 213/300 Flush
20773 214/300 Read block 1868 len 36864
20774 215/300 Read block 2850 len 8192
20775 216/300 Write block 1152 len 36864 data: 3 3 4 4 6 7 8 9 9
20776 217/300 Read block 4693 len 32768
20777 218/300 Read block 4028 len 36864
20778 219/300 Read block 4958 len 12288
20779 220/300 Write block 3403 len 8192 data: 5 4
20780 221/300 Write block 4393 len 36864 data: 4 5 5 4 4 5 5 5 4
20781 222/300 Write block 1697 len 24576 data: 5 6 6 6 5 4
20782 223/300 Flush
20783 224/300 Write block 4900 len 16384 data: 4 3 3 3
20784 225/300 Read block 3829 len 8192
20785 226/300 Read block 2252 len 20480
20786 227/300 Write block 1435 len 20480 data: 3 3 3 3 3
20787 228/300 Read block 454 len 16384
20788 229/300 Read block 573 len 36864
20789 230/300 Read block 4416 len 24576
20790 231/300 Write block 2450 len 4096 data: 5
20791 232/300 Write block 1250 len 24576 data: 4 4 3 3 4 4
20792 233/300 Write block 504 len 8192 data: 2 2
20793 234/300 Read block 3766 len 24576
20794 235/300 Read block 1863 len 4096
20795 236/300 Read block 2886 len 36864
20796 237/300 Write block 2470 len 28672 data: 2 2 2 3 4 4 3
20797 238/300 Write block 3175 len 28672 data: 3 4 3 3 4 5 5
20798 239/300 Read block 3021 len 20480
20799 240/300 Write block 4229 len 12288 data: 4 5 4
20800 241/300 Write block 3890 len 32768 data: 5 4 4 5 5 4 3 3
20801 242/300 Flush
20802 243/300 Write block 1633 len 40960 data: 2 2 2 2 3 2 2 4 4 5
20803 244/300 Read block 960 len 16384
20804 245/300 Write block 3919 len 20480 data: 4 6 6 6 6
20805 246/300 Read block 3175 len 8192
20806 247/300 Write block 1896 len 20480 data: 5 5 5 5 7
20807 248/300 Write block 2727 len 20480 data: 6 6 7 7 7
20808 249/300 Write block 1650 len 24576 data: 6 6 8 8 5 6
20809 250/300 Read block 3727 len 32768
20810 251/300 Write block 3508 len 12288 data: 3 4 5
20811 252/300 Write block 4845 len 32768 data: 4 5 4 4 4 3 3 3
20812 253/300 Write block 3327 len 8192 data: 4 4
20813 254/300 Write block 712 len 16384 data: 4 4 4 5
20814 255/300 Read block 4659 len 24576
20815 256/300 Read block 3007 len 4096
20816 257/300 Write block 3875 len 28672 data: 5 4 4 5 6 6 6
20817 258/300 Write block 522 len 8192 data: 6 5
20818 259/300 Write block 3822 len 32768 data: 4 3 2 3 3 3 4 4
20819 260/300 Flush
20820 261/300 Read block 2815 len 8192
20821 262/300 Read block 2234 len 8192
20822 263/300 Read block 4988 len 16384
20823 264/300 Write block 4545 len 8192 data: 4 3
20824 265/300 Write block 3845 len 20480 data: 2 2 3 5 5
20825 266/300 Write block 347 len 36864 data: 3 3 4 4 4 2 3 3 3
20826 267/300 Read block 2521 len 36864
20827 268/300 Read block 2492 len 36864
20828 269/300 Write block 69 len 16384 data: 9 10 9 10
20829 270/300 Read block 4611 len 4096
20830 271/300 Read block 2982 len 36864
20831 272/300 Write block 3491 len 36864 data: 5 5 5 5 5 5 6 5 5
20832 273/300 Read block 1052 len 12288
20833 274/300 Read block 3017 len 36864
20834 275/300 Read block 1957 len 16384
20835 276/300 Read block 2482 len 8192
20836 277/300 Write block 236 len 24576 data: 3 6 5 4 5 6
20837 278/300 Read block 2369 len 12288
20838 279/300 Read block 3481 len 12288
20839 280/300 Read block 2037 len 28672
20840 281/300 Read block 1057 len 24576
20841 282/300 Write block 3625 len 36864 data: 3 4 5 5 5 3 3 3 2
20842 283/300 Read block 550 len 4096
20843 284/300 Write block 3565 len 20480 data: 6 5 4 6 6
20844 285/300 Read block 1372 len 4096
20845 286/300 Read block 3417 len 32768
20846 287/300 Read block 2084 len 8192
20847 288/300 Write block 4344 len 8192 data: 4 3
20848 289/300 Read block 983 len 32768
20849 290/300 Read block 4580 len 40960
20850 291/300 Write block 2200 len 8192 data: 5 3
20851 292/300 Write block 1820 len 20480 data: 3 4 5 4 4
20852 293/300 Write block 2229 len 32768 data: 6 7 6 6 5 5 4 4
20853 294/300 Write block 986 len 20480 data: 7 6 6 8 8
20854 295/300 Read block 2282 len 8192
20855 296/300 Read block 3903 len 28672
20856 297/300 Write block 1765 len 24576 data: 4 4 4 4 5 5
20857 298/300 Flush
20858 299/300 Write block 784 len 12288 data: 10 10 8
20859 300/300 Read block 2946 len 32768
20860 301/300 Read block 2384 len 16384
20861 Main task has work to do, go find it
20862 got dsc Start(2)
20863 start 2
20864 Jan 23 17:42:08.670 INFO request completed, latency_us: 247, response_code: 204, uri: /start/cid/2, method: POST, req_id: fc0a7c35-6b40-44da-9bed-c53273f6cfe1, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
20865 Replay: started 2, returned:Ok(())
20866 ----------------------------------------------------------------
20867 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20868 States: Active Active Offline
20869 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20870 6738 Acked Write 2 Done Done Sent false
20871 6739 Acked Read 2 Done Done Sent false
20872 6740 Acked Read 9 Done Done Sent false
20873 6741 Acked Read 8 Done Done Sent false
20874 6742 Acked Read 8 Done Done Sent false
20875 6743 Acked Read 4 Done Done Sent false
20876 6744 Acked Flush 0 Done Done Sent false
20877 6745 Acked Write 6 Done Done Sent false
20878 6746 Acked Read 9 Done Done Sent false
20879 6747 Acked Flush 0 Done Done Sent false
20880 6748 Acked Read 4 Done Done Sent false
20881 6749 Acked Read 2 Done Done Sent false
20882 6750 Acked Write 6 Done Done Sent false
20883 6751 Acked Write 2 Done Done Sent false
20884 6752 Acked Flush 0 Done Done Sent false
20885 6753 Acked Read 2 Done Done Sent false
20886 6754 Acked Read 7 Done Done Sent false
20887 6755 Acked Flush 0 Done Done Sent false
20888 6756 Acked Read 6 Done Done Sent false
20889 6757 Acked Flush 0 Done Done Sent false
20890 6758 Acked Flush 0 Done Done Sent false
20891 6759 Acked Read 3 Done Done Sent false
20892 6760 Acked Flush 0 Done Done Sent false
20893 6761 Acked Read 7 Done Done Sent false
20894 6762 Acked Read 8 Done Done Sent false
20895 6763 Acked Write 7 Done Done Sent false
20896 6764 Acked Flush 0 Done Done Sent false
20897 6765 Acked Read 1 Done Done Sent false
20898 6766 Acked Read 5 Done Done Sent false
20899 6767 Acked Read 8 Done Done Sent false
20900 6768 Acked Read 7 Done Done Sent false
20901 6769 Acked Write 8 Done Done Sent false
20902 6770 Acked Write 5 Done Done Sent false
20903 6771 Acked Flush 0 Done Done Sent false
20904 6772 Acked Read 4 Done Done Sent false
20905 6773 Acked Read 10 Done Done Sent false
20906 6774 Acked Write 1 Done Done Sent false
20907 6775 Acked Read 5 Done Done Sent false
20908 6776 Acked Read 9 Done Done Sent false
20909 6777 Acked Read 2 Done Done Sent false
20910 6778 Acked Write 10 Done Done Sent false
20911 6779 Acked Read 4 Done Done Sent false
20912 6780 Acked Read 5 Done Done Sent false
20913 6781 Acked Write 7 Done Done Sent false
20914 6782 Acked Write 3 Done Done Sent false
20915 6783 Acked Write 1 Done Done Sent false
20916 6784 Acked Read 6 Done Done Sent false
20917 6785 Acked Write 1 Done Done Sent false
20918 6786 Acked Read 2 Done Done Sent false
20919 6787 Acked Read 6 Done Done Sent false
20920 6788 Acked Read 4 Done Done Sent false
20921 6789 Acked Write 2 Done Done Sent false
20922 6790 Acked Read 4 Done Done Sent false
20923 6791 Acked Write 6 Done Done Sent false
20924 6792 Acked Read 1 Done Done Sent false
20925 6793 Acked Write 4 Done Done Sent false
20926 6794 Acked Write 3 Done Done Sent false
20927 6795 Acked Read 9 Done Done Sent false
20928 6796 Acked Read 5 Done Done Sent false
20929 6797 Acked Read 2 Done Done Sent false
20930 6798 Acked Read 2 Done Done Sent false
20931 6799 Acked Write 8 Done Done Sent false
20932 6800 Acked Read 5 Done Done Sent false
20933 6801 Acked Read 7 Done Done Sent false
20934 6802 Acked Read 10 Done Done Sent false
20935 6803 Acked Write 10 Done Done Sent false
20936 6804 Acked Write 2 Done Done Sent false
20937 6805 Acked Read 5 Done Done Sent false
20938 6806 Acked Flush 0 Done Done Sent false
20939 6807 Acked Write 7 Done Done Sent false
20940 6808 Acked Flush 0 Done Done Sent false
20941 6809 Acked Read 4 Done Done Sent false
20942 6810 Acked Write 7 Done Done Sent false
20943 6811 Acked Flush 0 Done Done Sent false
20944 6812 Acked Write 4 Done Done Sent false
20945 6813 Acked Write 4 Done Done Sent false
20946 6814 Acked Read 9 Done Done Sent false
20947 6815 Acked Read 1 Done Done Sent false
20948 6816 Acked Read 4 Done Done Sent false
20949 6817 Acked Read 1 Done Done Sent false
20950 6818 Acked Write 4 Done Done Sent false
20951 6819 Acked Read 8 Done Done Sent false
20952 6820 Acked Read 5 Done Done Sent false
20953 6821 Acked Write 9 Done Done Sent false
20954 6822 Acked Read 5 Done Done Sent false
20955 6823 Acked Read 5 Done Done Sent false
20956 6824 Acked Read 10 Done Done Sent false
20957 6825 Acked Read 2 Done Done Sent false
20958 6826 Acked Write 10 Done Done Sent false
20959 6827 Acked Flush 0 Done Done Sent false
20960 6828 Acked Write 8 Done Done Sent false
20961 6829 Acked Write 8 Done Done Sent false
20962 6830 Acked Read 9 Done Done Sent false
20963 6831 Acked Flush 0 Done Done Sent false
20964 6832 Acked Write 1 Done Done Sent false
20965 6833 Acked Read 1 Done Done Sent false
20966 6834 Acked Read 2 Done Done Sent false
20967 6835 Acked Read 8 Done Done Sent false
20968 6836 Acked Write 6 Done Done Sent false
20969 6837 Acked Read 3 Done Done Sent false
20970 6838 Acked Read 2 Done Done Sent false
20971 6839 Acked Read 8 Done Done Sent false
20972 6840 Acked Flush 0 Done Done Sent false
20973 6841 Acked Write 2 Done Done Sent false
20974 6842 Acked Read 10 Done Done Sent false
20975 6843 Acked Read 5 Done Done Sent false
20976 6844 Acked Read 3 Done Done Sent false
20977 6845 Acked Write 2 Done Done Sent false
20978 6846 Acked Read 10 Done Done Sent false
20979 6847 Acked Write 1 Done Done Sent false
20980 6848 Acked Write 3 Done Done Sent false
20981 6849 Acked Read 10 Done Done Sent false
20982 6850 Acked Read 4 Done Done Sent false
20983 6851 Acked Read 1 Done Done Sent false
20984 6852 Acked Read 7 Done Done Sent false
20985 6853 Acked Write 9 Done Done Sent false
20986 6854 Acked Write 9 Done Done Sent false
20987 6855 Acked Read 8 Done Done Sent false
20988 6856 Acked Read 1 Done Done Sent false
20989 6857 Acked Write 8 Done Done Sent false
20990 6858 Acked Read 1 Done Done Sent false
20991 6859 Acked Read 3 Done Done Sent false
20992 6860 Acked Write 4 Done Done Sent false
20993 6861 Acked Write 9 Done Done Sent false
20994 6862 Acked Read 9 Done Done Sent false
20995 6863 Acked Flush 0 Done Done Sent false
20996 6864 Acked Write 7 Done Done Sent false
20997 6865 Acked Read 7 Done Done Sent false
20998 6866 Acked Write 9 Done Done Sent false
20999 6867 Acked Read 6 Done Done Sent false
21000 6868 Acked Flush 0 Done Done Sent false
21001 6869 Acked Read 8 Done Done Sent false
21002 6870 Acked Write 8 Done Done Sent false
21003 6871 Acked Flush 0 Done Done Sent false
21004 6872 Acked Write 8 Done Done Sent false
21005 6873 Acked Read 1 Done Done Sent false
21006 6874 Acked Write 2 Done Done Sent false
21007 6875 Acked Write 1 Done Done Sent false
21008 6876 Acked Read 1 Done Done Sent false
21009 6877 Acked Flush 0 Done Done Sent false
21010 6878 Acked Read 2 Done Done Sent false
21011 6879 Acked Flush 0 Done Done Sent false
21012 6880 Acked Read 1 Done Done Sent false
21013 6881 Acked Read 8 Done Done Sent false
21014 6882 Acked Read 8 Done Done Sent false
21015 6883 Acked Read 7 Done Done Sent false
21016 6884 Acked Write 3 Done Done Sent false
21017 6885 Acked Write 1 Done Done Sent false
21018 6886 Acked Flush 0 Done Done Sent false
21019 6887 Acked Write 4 Done Done Sent false
21020 6888 Acked Read 4 Done Done Sent false
21021 6889 Acked Write 10 Done Done Sent false
21022 6890 Acked Read 10 Done Done Sent false
21023 6891 Acked Read 9 Done Done Sent false
21024 6892 Acked Read 1 Done Done Sent false
21025 6893 Acked Write 2 Done Done Sent false
21026 6894 Acked Read 10 Done Done Sent false
21027 6895 Acked Read 7 Done Done Sent false
21028 6896 Acked Read 5 Done Done Sent false
21029 6897 Acked Read 3 Done Done Sent false
21030 6898 Acked Write 2 Done Done Sent false
21031 6899 Acked Write 9 Done Done Sent false
21032 6900 Acked Write 9 Done Done Sent false
21033 6901 Acked Write 10 Done Done Sent false
21034 6902 Acked Read 10 Done Done Sent false
21035 6903 Acked Read 5 Done Done Sent false
21036 6904 Acked Read 10 Done Done Sent false
21037 6905 Acked Write 9 Done Done Sent false
21038 6906 Acked Flush 0 Done Done Sent false
21039 6907 Acked Read 1 Done Done Sent false
21040 6908 Acked Write 10 Done Done Sent false
21041 6909 Acked Read 6 Done Done Sent false
21042 6910 Acked Read 9 Done Done Sent false
21043 6911 Acked Read 2 Done Done Sent false
21044 6912 Acked Read 4 Done Done Sent false
21045 6913 Acked Write 10 Done Done Sent false
21046 6914 Acked Read 4 Done Done Sent false
21047 6915 Acked Read 4 Done Done Sent false
21048 6916 Acked Write 9 Done Done Sent false
21049 6917 Acked Write 6 Done Done Sent false
21050 6918 Acked Read 10 Done Done Sent false
21051 6919 Acked Write 7 Done Done Sent false
21052 6920 Acked Read 10 Done Done Sent false
21053 6921 Acked Read 6 Done Done Sent false
21054 6922 Acked Flush 0 Done Done Sent false
21055 6923 Acked Read 2 Done Done Sent false
21056 6924 Acked Write 8 Done Done Sent false
21057 6925 Acked Read 10 Done Done Sent false
21058 6926 Acked Write 3 Done Done Sent false
21059 6927 Acked Read 4 Done Done Sent false
21060 6928 Acked Write 10 Done Done Sent false
21061 6929 Acked Read 5 Done Done Sent false
21062 6930 Acked Read 3 Done Done Sent false
21063 6931 Acked Flush 0 Done Done Sent false
21064 6932 Acked Read 5 Done Done Sent false
21065 6933 Acked Read 3 Done Done Sent false
21066 6934 Acked Read 7 Done Done Sent false
21067 6935 Acked Write 3 Done Done Sent false
21068 6936 Acked Read 2 Done Done Sent false
21069 6937 Acked Write 10 Done Done Sent false
21070 6938 Acked Write 3 Done Done Sent false
21071 6939 Acked Read 7 Done Done Sent false
21072 6940 Acked Flush 0 Done Done Sent false
21073 6941 Acked Flush 0 Done Done Sent false
21074 6942 Acked Write 8 Done Done Sent false
21075 6943 Acked Write 8 Done Done Sent false
21076 6944 Acked Write 9 Done Done Sent false
21077 6945 Acked Write 9 Done Done Sent false
21078 6946 Acked Read 8 Done Done Sent false
21079 6947 Acked Read 3 Done Done Sent false
21080 6948 Acked Write 8 Done Done Sent false
21081 6949 Acked Write 7 Done Done Sent false
21082 6950 Acked Flush 0 Done Done Sent false
21083 6951 Acked Read 9 Done Done Sent false
21084 6952 Acked Read 2 Done Done Sent false
21085 6953 Acked Write 9 Done Done Sent false
21086 6954 Acked Read 8 Done Done Sent false
21087 6955 Acked Read 9 Done Done Sent false
21088 6956 Acked Read 3 Done Done Sent false
21089 6957 Acked Write 2 Done Done Sent false
21090 6958 Acked Write 9 Done Done Sent false
21091 6959 Acked Write 6 Done Done Sent false
21092 6960 Acked Flush 0 Done Done Sent false
21093 6961 Acked Write 4 Done Done Sent false
21094 6962 Acked Read 2 Done Done Sent false
21095 6963 Acked Read 5 Done Done Sent false
21096 6964 Acked Write 5 Done Done Sent false
21097 6965 Acked Read 4 Done Done Sent false
21098 6966 Acked Read 9 Done Done Sent false
21099 6967 Acked Read 6 Done Done Sent false
21100 6968 Acked Write 1 Done Done Sent false
21101 6969 Acked Write 6 Done Done Sent false
21102 6970 Acked Write 2 Done Done Sent false
21103 6971 Acked Read 6 Done Done Sent false
21104 6972 Acked Read 1 Done Done Sent false
21105 6973 Acked Read 9 Done Done Sent false
21106 6974 Acked Write 7 Done Done Sent false
21107 6975 Acked Write 7 Done Done Sent false
21108 6976 Acked Read 5 Done Done Sent false
21109 6977 Acked Write 3 Done Done Sent false
21110 6978 Acked Write 8 Done Done Sent false
21111 6979 Acked Flush 0 Done Done Sent false
21112 6980 Acked Write 10 Done Done Sent false
21113 6981 Acked Read 4 Done Done Sent false
21114 6982 Acked Write 5 Done Done Sent false
21115 6983 Acked Read 2 Done Done Sent false
21116 6984 Acked Write 5 Done Done Sent false
21117 6985 Acked Write 5 Done Done Sent false
21118 6986 Acked Write 6 Done Done Sent false
21119 6987 Acked Read 8 Done Done Sent false
21120 6988 Acked Write 3 Done Done Sent false
21121 6989 Acked Write 8 Done Done Sent false
21122 6990 Acked Write 2 Done Done Sent false
21123 6991 Acked Write 4 Done Done Sent false
21124 6992 Acked Read 6 Done Done Sent false
21125 6993 Acked Read 1 Done Done Sent false
21126 6994 Acked Write 7 Done Done Sent false
21127 6995 Acked Write 2 Done Done Sent false
21128 6996 Acked Write 8 Done Done Sent false
21129 6997 Acked Flush 0 Done Done Sent false
21130 6998 Acked Read 2 Done Done Sent false
21131 6999 Acked Read 2 Done Done Sent false
21132 7000 Acked Read 4 Done Done Sent false
21133 7001 Acked Write 2 Done Done Sent false
21134 7002 Acked Write 5 Done Done Sent false
21135 7003 Acked Write 9 Done Done Sent false
21136 7004 Acked Read 9 Done Done Sent false
21137 7005 Acked Read 9 Done Done Sent false
21138 7006 Acked Write 4 Done Done Sent false
21139 7007 Acked Read 1 Done Done Sent false
21140 7008 Acked Read 9 Done Done Sent false
21141 7009 Acked Write 9 Done Done Sent false
21142 7010 Acked Read 3 Done Done Sent false
21143 7011 Acked Read 9 Done Done Sent false
21144 7012 Acked Read 4 Done Done Sent false
21145 7013 Acked Read 2 Done Done Sent false
21146 7014 Acked Write 6 Done Done Sent false
21147 7015 Acked Read 3 Done Done Sent false
21148 7016 Acked Read 3 Done Done Sent false
21149 7017 Acked Read 7 Done Done Sent false
21150 7018 Acked Read 6 Done Done Sent false
21151 7019 Acked Write 9 Done Done Sent false
21152 7020 Acked Read 1 Done Done Sent false
21153 7021 Acked Write 5 Done Done Sent false
21154 7022 Acked Read 1 Done Done Sent false
21155 7023 Acked Read 8 Done Done Sent false
21156 7024 Acked Read 2 Done Done Sent false
21157 7025 Acked Write 2 Done Done Sent false
21158 7026 Acked Read 8 Done Done Sent false
21159 7027 Acked Read 10 Done Done Sent false
21160 7028 Acked Write 2 Done Done Sent false
21161 7029 Acked Write 5 Done Done Sent false
21162 7030 Acked Write 8 Done Done Sent false
21163 7031 Acked Write 5 Done Done Sent false
21164 7032 Acked Read 2 Done Done Sent false
21165 7033 Acked Read 7 Done Done Sent false
21166 7034 Acked Write 6 Done Done Sent false
21167 7035 Acked Flush 0 Done Done Sent false
21168 7036 Acked Write 3 Done Done Sent false
21169 7037 Acked Read 8 Done Done Sent false
21170 7038 Acked Read 4 Done Done Sent false
21171 STATES DS:0 DS:1 DS:2 TOTAL
21172 Sent 0 0 301 301
21173 Done 301 301 0 602
21174 Skipped 0 0 0 0
21175 Error 0 0 0 0
21176 Last Flush: Some(JobId(7035)) Some(JobId(7035)) Some(JobId(6737))
21177 Downstairs last five completed: 6737 6736 6735 6734 6733
21178 Upstairs last five completed: 7038 7037 7036 7035 7034
21179 CLIENT: Up:0 ds:301 act:2
21180 [8830] got start action
21181 [8830] I am going to restart
21182 Starting downstairs at port 8830
21183 Make output file at "/tmp/dsc/downstairs-8830.txt"
21184 [8830][2] reports Starting
21185 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1225)
21186 [8830][2] reports Running
21187 ----------------------------------------------------------------
21188 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
21189 States: Active Active Offline
21190 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21191 6738 Acked Write 2 Done Done Sent false
21192 6739 Acked Read 2 Done Done Sent false
21193 6740 Acked Read 9 Done Done Sent false
21194 6741 Acked Read 8 Done Done Sent false
21195 6742 Acked Read 8 Done Done Sent false
21196 6743 Acked Read 4 Done Done Sent false
21197 6744 Acked Flush 0 Done Done Sent false
21198 6745 Acked Write 6 Done Done Sent false
21199 6746 Acked Read 9 Done Done Sent false
21200 6747 Acked Flush 0 Done Done Sent false
21201 6748 Acked Read 4 Done Done Sent false
21202 6749 Acked Read 2 Done Done Sent false
21203 6750 Acked Write 6 Done Done Sent false
21204 6751 Acked Write 2 Done Done Sent false
21205 6752 Acked Flush 0 Done Done Sent false
21206 6753 Acked Read 2 Done Done Sent false
21207 6754 Acked Read 7 Done Done Sent false
21208 6755 Acked Flush 0 Done Done Sent false
21209 6756 Acked Read 6 Done Done Sent false
21210 6757 Acked Flush 0 Done Done Sent false
21211 6758 Acked Flush 0 Done Done Sent false
21212 6759 Acked Read 3 Done Done Sent false
21213 6760 Acked Flush 0 Done Done Sent false
21214 6761 Acked Read 7 Done Done Sent false
21215 6762 Acked Read 8 Done Done Sent false
21216 6763 Acked Write 7 Done Done Sent false
21217 6764 Acked Flush 0 Done Done Sent false
21218 6765 Acked Read 1 Done Done Sent false
21219 6766 Acked Read 5 Done Done Sent false
21220 6767 Acked Read 8 Done Done Sent false
21221 6768 Acked Read 7 Done Done Sent false
21222 6769 Acked Write 8 Done Done Sent false
21223 6770 Acked Write 5 Done Done Sent false
21224 6771 Acked Flush 0 Done Done Sent false
21225 6772 Acked Read 4 Done Done Sent false
21226 6773 Acked Read 10 Done Done Sent false
21227 6774 Acked Write 1 Done Done Sent false
21228 6775 Acked Read 5 Done Done Sent false
21229 6776 Acked Read 9 Done Done Sent false
21230 6777 Acked Read 2 Done Done Sent false
21231 6778 Acked Write 10 Done Done Sent false
21232 6779 Acked Read 4 Done Done Sent false
21233 6780 Acked Read 5 Done Done Sent false
21234 6781 Acked Write 7 Done Done Sent false
21235 6782 Acked Write 3 Done Done Sent false
21236 6783 Acked Write 1 Done Done Sent false
21237 6784 Acked Read 6 Done Done Sent false
21238 6785 Acked Write 1 Done Done Sent false
21239 6786 Acked Read 2 Done Done Sent false
21240 6787 Acked Read 6 Done Done Sent false
21241 6788 Acked Read 4 Done Done Sent false
21242 6789 Acked Write 2 Done Done Sent false
21243 6790 Acked Read 4 Done Done Sent false
21244 6791 Acked Write 6 Done Done Sent false
21245 6792 Acked Read 1 Done Done Sent false
21246 6793 Acked Write 4 Done Done Sent false
21247 6794 Acked Write 3 Done Done Sent false
21248 6795 Acked Read 9 Done Done Sent false
21249 6796 Acked Read 5 Done Done Sent false
21250 6797 Acked Read 2 Done Done Sent false
21251 6798 Acked Read 2 Done Done Sent false
21252 6799 Acked Write 8 Done Done Sent false
21253 6800 Acked Read 5 Done Done Sent false
21254 6801 Acked Read 7 Done Done Sent false
21255 6802 Acked Read 10 Done Done Sent false
21256 6803 Acked Write 10 Done Done Sent false
21257 6804 Acked Write 2 Done Done Sent false
21258 6805 Acked Read 5 Done Done Sent false
21259 6806 Acked Flush 0 Done Done Sent false
21260 6807 Acked Write 7 Done Done Sent false
21261 6808 Acked Flush 0 Done Done Sent false
21262 6809 Acked Read 4 Done Done Sent false
21263 6810 Acked Write 7 Done Done Sent false
21264 6811 Acked Flush 0 Done Done Sent false
21265 6812 Acked Write 4 Done Done Sent false
21266 6813 Acked Write 4 Done Done Sent false
21267 6814 Acked Read 9 Done Done Sent false
21268 6815 Acked Read 1 Done Done Sent false
21269 6816 Acked Read 4 Done Done Sent false
21270 6817 Acked Read 1 Done Done Sent false
21271 6818 Acked Write 4 Done Done Sent false
21272 6819 Acked Read 8 Done Done Sent false
21273 6820 Acked Read 5 Done Done Sent false
21274 6821 Acked Write 9 Done Done Sent false
21275 6822 Acked Read 5 Done Done Sent false
21276 6823 Acked Read 5 Done Done Sent false
21277 6824 Acked Read 10 Done Done Sent false
21278 6825 Acked Read 2 Done Done Sent false
21279 6826 Acked Write 10 Done Done Sent false
21280 6827 Acked Flush 0 Done Done Sent false
21281 6828 Acked Write 8 Done Done Sent false
21282 6829 Acked Write 8 Done Done Sent false
21283 6830 Acked Read 9 Done Done Sent false
21284 6831 Acked Flush 0 Done Done Sent false
21285 6832 Acked Write 1 Done Done Sent false
21286 6833 Acked Read 1 Done Done Sent false
21287 6834 Acked Read 2 Done Done Sent false
21288 6835 Acked Read 8 Done Done Sent false
21289 6836 Acked Write 6 Done Done Sent false
21290 6837 Acked Read 3 Done Done Sent false
21291 6838 Acked Read 2 Done Done Sent false
21292 6839 Acked Read 8 Done Done Sent false
21293 6840 Acked Flush 0 Done Done Sent false
21294 6841 Acked Write 2 Done Done Sent false
21295 6842 Acked Read 10 Done Done Sent false
21296 6843 Acked Read 5 Done Done Sent false
21297 6844 Acked Read 3 Done Done Sent false
21298 6845 Acked Write 2 Done Done Sent false
21299 6846 Acked Read 10 Done Done Sent false
21300 6847 Acked Write 1 Done Done Sent false
21301 6848 Acked Write 3 Done Done Sent false
21302 6849 Acked Read 10 Done Done Sent false
21303 6850 Acked Read 4 Done Done Sent false
21304 6851 Acked Read 1 Done Done Sent false
21305 6852 Acked Read 7 Done Done Sent false
21306 6853 Acked Write 9 Done Done Sent false
21307 6854 Acked Write 9 Done Done Sent false
21308 6855 Acked Read 8 Done Done Sent false
21309 6856 Acked Read 1 Done Done Sent false
21310 6857 Acked Write 8 Done Done Sent false
21311 6858 Acked Read 1 Done Done Sent false
21312 6859 Acked Read 3 Done Done Sent false
21313 6860 Acked Write 4 Done Done Sent false
21314 6861 Acked Write 9 Done Done Sent false
21315 6862 Acked Read 9 Done Done Sent false
21316 6863 Acked Flush 0 Done Done Sent false
21317 6864 Acked Write 7 Done Done Sent false
21318 6865 Acked Read 7 Done Done Sent false
21319 6866 Acked Write 9 Done Done Sent false
21320 6867 Acked Read 6 Done Done Sent false
21321 6868 Acked Flush 0 Done Done Sent false
21322 6869 Acked Read 8 Done Done Sent false
21323 6870 Acked Write 8 Done Done Sent false
21324 6871 Acked Flush 0 Done Done Sent false
21325 6872 Acked Write 8 Done Done Sent false
21326 6873 Acked Read 1 Done Done Sent false
21327 6874 Acked Write 2 Done Done Sent false
21328 6875 Acked Write 1 Done Done Sent false
21329 6876 Acked Read 1 Done Done Sent false
21330 6877 Acked Flush 0 Done Done Sent false
21331 6878 Acked Read 2 Done Done Sent false
21332 6879 Acked Flush 0 Done Done Sent false
21333 6880 Acked Read 1 Done Done Sent false
21334 6881 Acked Read 8 Done Done Sent false
21335 6882 Acked Read 8 Done Done Sent false
21336 6883 Acked Read 7 Done Done Sent false
21337 6884 Acked Write 3 Done Done Sent false
21338 6885 Acked Write 1 Done Done Sent false
21339 6886 Acked Flush 0 Done Done Sent false
21340 6887 Acked Write 4 Done Done Sent false
21341 6888 Acked Read 4 Done Done Sent false
21342 6889 Acked Write 10 Done Done Sent false
21343 6890 Acked Read 10 Done Done Sent false
21344 6891 Acked Read 9 Done Done Sent false
21345 6892 Acked Read 1 Done Done Sent false
21346 6893 Acked Write 2 Done Done Sent false
21347 6894 Acked Read 10 Done Done Sent false
21348 6895 Acked Read 7 Done Done Sent false
21349 6896 Acked Read 5 Done Done Sent false
21350 6897 Acked Read 3 Done Done Sent false
21351 6898 Acked Write 2 Done Done Sent false
21352 6899 Acked Write 9 Done Done Sent false
21353 6900 Acked Write 9 Done Done Sent false
21354 6901 Acked Write 10 Done Done Sent false
21355 6902 Acked Read 10 Done Done Sent false
21356 6903 Acked Read 5 Done Done Sent false
21357 6904 Acked Read 10 Done Done Sent false
21358 6905 Acked Write 9 Done Done Sent false
21359 6906 Acked Flush 0 Done Done Sent false
21360 6907 Acked Read 1 Done Done Sent false
21361 6908 Acked Write 10 Done Done Sent false
21362 6909 Acked Read 6 Done Done Sent false
21363 6910 Acked Read 9 Done Done Sent false
21364 6911 Acked Read 2 Done Done Sent false
21365 6912 Acked Read 4 Done Done Sent false
21366 6913 Acked Write 10 Done Done Sent false
21367 6914 Acked Read 4 Done Done Sent false
21368 6915 Acked Read 4 Done Done Sent false
21369 6916 Acked Write 9 Done Done Sent false
21370 6917 Acked Write 6 Done Done Sent false
21371 6918 Acked Read 10 Done Done Sent false
21372 6919 Acked Write 7 Done Done Sent false
21373 6920 Acked Read 10 Done Done Sent false
21374 6921 Acked Read 6 Done Done Sent false
21375 6922 Acked Flush 0 Done Done Sent false
21376 6923 Acked Read 2 Done Done Sent false
21377 6924 Acked Write 8 Done Done Sent false
21378 6925 Acked Read 10 Done Done Sent false
21379 6926 Acked Write 3 Done Done Sent false
21380 6927 Acked Read 4 Done Done Sent false
21381 6928 Acked Write 10 Done Done Sent false
21382 6929 Acked Read 5 Done Done Sent false
21383 6930 Acked Read 3 Done Done Sent false
21384 6931 Acked Flush 0 Done Done Sent false
21385 6932 Acked Read 5 Done Done Sent false
21386 6933 Acked Read 3 Done Done Sent false
21387 6934 Acked Read 7 Done Done Sent false
21388 6935 Acked Write 3 Done Done Sent false
21389 6936 Acked Read 2 Done Done Sent false
21390 6937 Acked Write 10 Done Done Sent false
21391 6938 Acked Write 3 Done Done Sent false
21392 6939 Acked Read 7 Done Done Sent false
21393 6940 Acked Flush 0 Done Done Sent false
21394 6941 Acked Flush 0 Done Done Sent false
21395 6942 Acked Write 8 Done Done Sent false
21396 6943 Acked Write 8 Done Done Sent false
21397 6944 Acked Write 9 Done Done Sent false
21398 6945 Acked Write 9 Done Done Sent false
21399 6946 Acked Read 8 Done Done Sent false
21400 6947 Acked Read 3 Done Done Sent false
21401 6948 Acked Write 8 Done Done Sent false
21402 6949 Acked Write 7 Done Done Sent false
21403 6950 Acked Flush 0 Done Done Sent false
21404 6951 Acked Read 9 Done Done Sent false
21405 6952 Acked Read 2 Done Done Sent false
21406 6953 Acked Write 9 Done Done Sent false
21407 6954 Acked Read 8 Done Done Sent false
21408 6955 Acked Read 9 Done Done Sent false
21409 6956 Acked Read 3 Done Done Sent false
21410 6957 Acked Write 2 Done Done Sent false
21411 6958 Acked Write 9 Done Done Sent false
21412 6959 Acked Write 6 Done Done Sent false
21413 6960 Acked Flush 0 Done Done Sent false
21414 6961 Acked Write 4 Done Done Sent false
21415 6962 Acked Read 2 Done Done Sent false
21416 6963 Acked Read 5 Done Done Sent false
21417 6964 Acked Write 5 Done Done Sent false
21418 6965 Acked Read 4 Done Done Sent false
21419 6966 Acked Read 9 Done Done Sent false
21420 6967 Acked Read 6 Done Done Sent false
21421 6968 Acked Write 1 Done Done Sent false
21422 6969 Acked Write 6 Done Done Sent false
21423 6970 Acked Write 2 Done Done Sent false
21424 6971 Acked Read 6 Done Done Sent false
21425 6972 Acked Read 1 Done Done Sent false
21426 6973 Acked Read 9 Done Done Sent false
21427 6974 Acked Write 7 Done Done Sent false
21428 6975 Acked Write 7 Done Done Sent false
21429 6976 Acked Read 5 Done Done Sent false
21430 6977 Acked Write 3 Done Done Sent false
21431 6978 Acked Write 8 Done Done Sent false
21432 6979 Acked Flush 0 Done Done Sent false
21433 6980 Acked Write 10 Done Done Sent false
21434 6981 Acked Read 4 Done Done Sent false
21435 6982 Acked Write 5 Done Done Sent false
21436 6983 Acked Read 2 Done Done Sent false
21437 6984 Acked Write 5 Done Done Sent false
21438 6985 Acked Write 5 Done Done Sent false
21439 6986 Acked Write 6 Done Done Sent false
21440 6987 Acked Read 8 Done Done Sent false
21441 6988 Acked Write 3 Done Done Sent false
21442 6989 Acked Write 8 Done Done Sent false
21443 6990 Acked Write 2 Done Done Sent false
21444 6991 Acked Write 4 Done Done Sent false
21445 6992 Acked Read 6 Done Done Sent false
21446 6993 Acked Read 1 Done Done Sent false
21447 6994 Acked Write 7 Done Done Sent false
21448 6995 Acked Write 2 Done Done Sent false
21449 6996 Acked Write 8 Done Done Sent false
21450 6997 Acked Flush 0 Done Done Sent false
21451 6998 Acked Read 2 Done Done Sent false
21452 6999 Acked Read 2 Done Done Sent false
21453 7000 Acked Read 4 Done Done Sent false
21454 7001 Acked Write 2 Done Done Sent false
21455 7002 Acked Write 5 Done Done Sent false
21456 7003 Acked Write 9 Done Done Sent false
21457 7004 Acked Read 9 Done Done Sent false
21458 7005 Acked Read 9 Done Done Sent false
21459 7006 Acked Write 4 Done Done Sent false
21460 7007 Acked Read 1 Done Done Sent false
21461 7008 Acked Read 9 Done Done Sent false
21462 7009 Acked Write 9 Done Done Sent false
21463 7010 Acked Read 3 Done Done Sent false
21464 7011 Acked Read 9 Done Done Sent false
21465 7012 Acked Read 4 Done Done Sent false
21466 7013 Acked Read 2 Done Done Sent false
21467 7014 Acked Write 6 Done Done Sent false
21468 7015 Acked Read 3 Done Done Sent false
21469 7016 Acked Read 3 Done Done Sent false
21470 7017 Acked Read 7 Done Done Sent false
21471 7018 Acked Read 6 Done Done Sent false
21472 7019 Acked Write 9 Done Done Sent false
21473 7020 Acked Read 1 Done Done Sent false
21474 7021 Acked Write 5 Done Done Sent false
21475 7022 Acked Read 1 Done Done Sent false
21476 7023 Acked Read 8 Done Done Sent false
21477 7024 Acked Read 2 Done Done Sent false
21478 7025 Acked Write 2 Done Done Sent false
21479 7026 Acked Read 8 Done Done Sent false
21480 7027 Acked Read 10 Done Done Sent false
21481 7028 Acked Write 2 Done Done Sent false
21482 7029 Acked Write 5 Done Done Sent false
21483 7030 Acked Write 8 Done Done Sent false
21484 7031 Acked Write 5 Done Done Sent false
21485 7032 Acked Read 2 Done Done Sent false
21486 7033 Acked Read 7 Done Done Sent false
21487 7034 Acked Write 6 Done Done Sent false
21488 7035 Acked Flush 0 Done Done Sent false
21489 7036 Acked Write 3 Done Done Sent false
21490 7037 Acked Read 8 Done Done Sent false
21491 7038 Acked Read 4 Done Done Sent false
21492 7039 Acked Flush 0 Done Done Sent false
21493 STATES DS:0 DS:1 DS:2 TOTAL
21494 Sent 0 0 302 302
21495 Done 302 302 0 604
21496 Skipped 0 0 0 0
21497 Error 0 0 0 0
21498 Last Flush: Some(JobId(7039)) Some(JobId(7039)) Some(JobId(6737))
21499 Downstairs last five completed: 6737 6736 6735 6734 6733
21500 Upstairs last five completed: 7039 7038 7037 7036 7035
21501 CLIENT: Up:0 ds:302 act:2
215022025-01-23T17:42:14.424ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
215032025-01-23T17:42:14.424ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65133) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
215042025-01-23T17:42:14.425ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
215052025-01-23T17:42:14.426ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
215062025-01-23T17:42:14.426ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
215072025-01-23T17:42:14.426ZINFOcrucible: send last flush ID to this DS: Some(JobId(6737)) = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
215082025-01-23T17:42:14.426ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
215092025-01-23T17:42:14.426ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(6737)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
215102025-01-23T17:42:14.426ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(6737)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
21511 ----------------------------------------------------------------
21512 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21513 Downstairs last five completed: 7039 7038 7037 7036 7035
21514 Upstairs last five completed: 7039 7038 7037 7036 7035
21515 CLIENT: Up:0 ds:0 act:3
21516 Replay: All jobs finished
21517 Main task has work to do, go find it
21518 got dsc Stop(1)
21519 stop 1
21520 Jan 23 17:42:16.675 [8820] Got stop action so:false kr:false
21521 INFO request completed, latency_us: 333, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 62eb53be-29d6-4833-8393-5a2223b0e757, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
215222025-01-23T17:42:16.676ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
215232025-01-23T17:42:16.676ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
215242025-01-23T17:42:16.676ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
215252025-01-23T17:42:16.676ZINFOcrucible: client task is exiting = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
21526 Jan 23 17:42:16.676 INFO request completed, latency_us: 122, response_code: 200, uri: /state/cid/1, method: GET, req_id: 557acd3c-8b87-4fbf-b57f-10f7be34a7e1, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
21527 [8820] Exited with: ExitStatus(unix_wait_status(9))
21528 [8820][1] reports Exit
21529 Jan 23 17:42:20.678 INFO request completed, latency_us: 196, response_code: 200, uri: /state/cid/1, method: GET, req_id: e7a6d452-1afd-4add-95f6-066a2df63e67, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
21530 001/300 Read block 2415 len 16384
21531 002/300 Write block 3369 len 12288 data: 3 3 4
21532 003/300 Read block 3233 len 28672
21533 004/300 Write block 918 len 20480 data: 5 3 4 4 5
21534 005/300 Read block 2644 len 40960
21535 006/300 Flush
21536 007/300 Read block 4884 len 8192
21537 008/300 Write block 4412 len 32768 data: 5 5 5 4 3 2 3 3
21538 009/300 Flush
21539 010/300 Flush
21540 011/300 Write block 826 len 16384 data: 8 7 6 6
21541 012/300 Write block 3685 len 12288 data: 8 5 5
21542 013/300 Read block 4619 len 12288
21543 014/300 Read block 1929 len 8192
21544 015/300 Read block 1487 len 24576
21545 016/300 Read block 4256 len 40960
21546 017/300 Write block 3691 len 12288 data: 5 6 5
21547 018/300 Flush
21548 019/300 Read block 17 len 36864
21549 020/300 Read block 1559 len 20480
21550 021/300 Flush
21551 022/300 Write block 177 len 24576 data: 4 4 3 3 3 3
21552 023/300 Write block 2050 len 4096 data: 4
21553 024/300 Flush
21554 025/300 Write block 3572 len 4096 data: 4
21555 026/300 Read block 2177 len 24576
21556 027/300 Read block 2325 len 12288
21557 028/300 Flush
21558 029/300 Write block 2956 len 40960 data: 4 5 4 4 4 4 6 5 7 8
21559 030/300 Read block 3880 len 4096
21560 031/300 Flush
21561 032/300 Read block 4368 len 20480
21562 033/300 Read block 3380 len 36864
21563 034/300 Read block 3434 len 20480
21564 035/300 Read block 1893 len 24576
21565 036/300 Write block 1171 len 20480 data: 4 4 4 4 3
21566 037/300 Write block 936 len 8192 data: 4 5
21567 038/300 Read block 413 len 8192
21568 039/300 Write block 4969 len 32768 data: 8 8 10 11 7 6 5 5
21569 040/300 Read block 4497 len 32768
21570 041/300 Write block 4952 len 32768 data: 6 5 4 3 2 2 3 3
21571 042/300 Read block 1972 len 4096
21572 043/300 Write block 1064 len 12288 data: 4 5 5
21573 044/300 Read block 4371 len 8192
21574 045/300 Flush
21575 046/300 Flush
21576 047/300 Read block 3161 len 20480
21577 048/300 Read block 1618 len 16384
21578 049/300 Write block 1330 len 36864 data: 7 9 8 6 5 5 4 4 3
21579 050/300 Write block 2140 len 16384 data: 5 4 3 3
21580 051/300 Write block 2398 len 20480 data: 12 11 11 8 6
21581 052/300 Write block 645 len 36864 data: 6 5 3 4 4 4 5 5 5
21582 053/300 Read block 1157 len 16384
21583 054/300 Read block 4604 len 12288
21584 055/300 Read block 2369 len 24576
21585 056/300 Write block 3267 len 28672 data: 3 2 3 3 3 3 3
21586 057/300 Read block 2710 len 32768
21587 058/300 Read block 3206 len 4096
21588 059/300 Read block 2819 len 36864
21589 060/300 Read block 1106 len 8192
21590 061/300 Read block 1285 len 32768
21591 062/300 Read block 1382 len 12288
21592 063/300 Write block 4759 len 28672 data: 5 6 7 6 6 6 5
21593 064/300 Read block 4281 len 12288
21594 065/300 Read block 1337 len 12288
21595 066/300 Read block 122 len 40960
21596 067/300 Read block 2770 len 24576
21597 068/300 Write block 1388 len 8192 data: 2 3
21598 069/300 Read block 4476 len 24576
21599 070/300 Flush
21600 071/300 Read block 4681 len 12288
21601 072/300 Flush
21602 073/300 Write block 4578 len 32768 data: 8 7 6 5 5 3 4 5
21603 074/300 Flush
21604 075/300 Write block 4084 len 12288 data: 3 3 2
21605 076/300 Read block 3247 len 32768
21606 077/300 Write block 1658 len 40960 data: 2 3 3 3 4 4 4 5 4 3
21607 078/300 Read block 3909 len 24576
21608 079/300 Write block 1838 len 8192 data: 7 6
21609 080/300 Write block 1602 len 4096 data: 2
21610 081/300 Flush
21611 082/300 Write block 369 len 32768 data: 7 7 7 7 7 7 7 6
21612 083/300 Read block 285 len 8192
21613 084/300 Flush
21614 085/300 Read block 2748 len 20480
21615 086/300 Write block 3203 len 16384 data: 3 4 3 4
21616 087/300 Read block 3045 len 24576
21617 088/300 Write block 2771 len 36864 data: 3 3 2 2 2 2 2 2 3
21618 089/300 Read block 2242 len 8192
21619 090/300 Read block 3373 len 36864
21620 091/300 Read block 4099 len 12288
21621 092/300 Write block 1982 len 4096 data: 4
21622 093/300 Read block 1441 len 32768
21623 094/300 Write block 1288 len 4096 data: 4
21624 095/300 Write block 4841 len 4096 data: 3
21625 096/300 Flush
21626 097/300 Write block 4388 len 16384 data: 2 2 2 4
21627 098/300 Write block 1398 len 12288 data: 3 2 4
21628 099/300 Read block 1512 len 24576
21629 100/300 Read block 315 len 4096
21630 101/300 Flush
21631 102/300 Write block 1286 len 20480 data: 5 5 5 3 3
21632 103/300 Read block 2977 len 40960
21633 104/300 Read block 422 len 16384
21634 105/300 Read block 3586 len 20480
21635 106/300 Read block 2193 len 20480
21636 107/300 Read block 4399 len 24576
21637 108/300 Write block 1206 len 12288 data: 4 4 5
21638 109/300 Write block 837 len 28672 data: 6 6 7 4 3 4 4
21639 110/300 Write block 4153 len 28672 data: 5 5 6 5 5 5 4
21640 111/300 Read block 2394 len 40960
21641 112/300 Flush
21642 113/300 Read block 1413 len 20480
21643 114/300 Read block 1160 len 32768
21644 115/300 Read block 4851 len 20480
21645 116/300 Read block 4940 len 24576
21646 117/300 Write block 4658 len 24576 data: 4 4 3 4 3 3
21647 118/300 Write block 4078 len 8192 data: 2 3
21648 119/300 Write block 2717 len 36864 data: 4 4 5 4 4 3 3 3 4
21649 120/300 Read block 4758 len 28672
21650 121/300 Read block 807 len 4096
21651 122/300 Read block 2912 len 24576
21652 123/300 Flush
21653 124/300 Write block 375 len 8192 data: 8 7
21654 125/300 Write block 652 len 28672 data: 6 6 6 6 6 7 6
21655 126/300 Write block 1023 len 16384 data: 6 7 7 7
21656 127/300 Read block 260 len 28672
21657 128/300 Flush
21658 129/300 Read block 1580 len 16384
21659 130/300 Read block 2774 len 12288
21660 131/300 Write block 1697 len 24576 data: 6 7 7 7 6 5
21661 132/300 Read block 1690 len 20480
21662 133/300 Write block 3809 len 4096 data: 4
21663 134/300 Flush
21664 135/300 Read block 2145 len 16384
21665 136/300 Write block 335 len 36864 data: 5 4 5 5 5 5 4 4 4
21666 137/300 Write block 1944 len 24576 data: 6 5 6 6 6 4
21667 138/300 Read block 3604 len 36864
21668 139/300 Write block 4131 len 28672 data: 3 3 3 4 6 6 6
21669 140/300 Write block 4249 len 8192 data: 4 4
21670 141/300 Write block 3918 len 20480 data: 5 5 7 7 7
21671 142/300 Write block 371 len 36864 data: 8 8 8 8 9 8 4 3 2
21672 143/300 Write block 2564 len 8192 data: 4 3
21673 144/300 Write block 4425 len 12288 data: 3 6 4
21674 145/300 Write block 3961 len 32768 data: 7 7 7 7 7 6 5 7
21675 146/300 Write block 1754 len 36864 data: 3 4 5 4 6 6 6 5 5
21676 147/300 Write block 2341 len 32768 data: 3 4 4 4 4 4 4 5
21677 148/300 Read block 1390 len 12288
21678 149/300 Read block 3185 len 12288
21679 150/300 Write block 3668 len 4096 data: 8
21680 151/300 Write block 1171 len 12288 data: 5 5 5
21681 152/300 Read block 1100 len 36864
21682 153/300 Read block 1495 len 16384
21683 154/300 Write block 4234 len 40960 data: 6 5 4 4 3 5 5 4 4 4
21684 155/300 Flush
21685 156/300 Read block 4010 len 12288
21686 157/300 Flush
21687 158/300 Flush
21688 159/300 Write block 3320 len 24576 data: 3 3 3 3 3 3
21689 160/300 Read block 2923 len 24576
21690 161/300 Read block 4597 len 4096
21691 162/300 Read block 4935 len 32768
21692 163/300 Read block 1747 len 28672
21693 164/300 Read block 1106 len 12288
21694 165/300 Write block 4660 len 8192 data: 4 5
21695 166/300 Write block 3908 len 4096 data: 5
21696 167/300 Read block 1511 len 20480
21697 168/300 Read block 4040 len 12288
21698 169/300 Write block 3450 len 36864 data: 3 3 3 3 2 3 4 5 5
21699 170/300 Read block 1127 len 16384
21700 171/300 Write block 4597 len 36864 data: 4 4 4 5 6 7 7 7 7
21701 172/300 Read block 2009 len 20480
21702 173/300 Read block 3778 len 28672
21703 174/300 Read block 2964 len 28672
21704 175/300 Read block 651 len 16384
21705 176/300 Read block 1464 len 8192
21706 177/300 Write block 643 len 16384 data: 6 6 7 6
21707 178/300 Write block 3492 len 36864 data: 6 6 6 6 6 7 6 6 5
21708 179/300 Read block 4666 len 32768
21709 180/300 Write block 8 len 24576 data: 4 4 4 4 4 5
21710 181/300 Read block 1153 len 36864
21711 182/300 Read block 3457 len 16384
21712 183/300 Write block 3762 len 28672 data: 3 4 5 5 6 4 4
21713 184/300 Read block 4058 len 16384
21714 185/300 Write block 2132 len 4096 data: 4
21715 186/300 Read block 3373 len 20480
21716 187/300 Read block 3231 len 32768
21717 188/300 Read block 2743 len 24576
21718 189/300 Flush
21719 190/300 Read block 4976 len 8192
21720 191/300 Write block 1491 len 24576 data: 3 4 6 5 3 3
21721 192/300 Write block 2234 len 16384 data: 6 5 5 3
21722 193/300 Write block 3374 len 12288 data: 5 4 2
21723 194/300 Read block 4517 len 40960
21724 195/300 Read block 2551 len 36864
21725 196/300 Read block 2454 len 40960
21726 197/300 Read block 3025 len 36864
21727 198/300 Read block 4766 len 16384
21728 199/300 Read block 3087 len 36864
21729 200/300 Write block 840 len 4096 data: 5
21730 201/300 Read block 3385 len 36864
21731 202/300 Read block 2345 len 36864
21732 203/300 Write block 3562 len 32768 data: 5 5 6 7 6 5 7 7
21733 204/300 Read block 3772 len 12288
21734 205/300 Read block 3884 len 12288
21735 206/300 Write block 1580 len 40960 data: 3 4 4 4 3 3 3 4 4 3
21736 207/300 Read block 3815 len 20480
21737 208/300 Read block 2683 len 8192
21738 209/300 Read block 2453 len 32768
21739 210/300 Flush
21740 211/300 Flush
21741 212/300 Write block 3950 len 16384 data: 4 3 3 4
21742 213/300 Read block 1952 len 40960
21743 214/300 Flush
21744 215/300 Write block 4240 len 28672 data: 6 5 5 5 3 3 3
21745 216/300 Flush
21746 217/300 Read block 2888 len 36864
21747 218/300 Read block 3606 len 16384
21748 219/300 Read block 2523 len 20480
21749 220/300 Write block 216 len 32768 data: 5 7 7 4 2 3 2 2
21750 221/300 Read block 2346 len 24576
21751 222/300 Write block 2205 len 20480 data: 5 5 6 7 6
21752 223/300 Read block 4519 len 4096
21753 224/300 Write block 2611 len 16384 data: 7 8 7 7
21754 225/300 Read block 3017 len 4096
21755 226/300 Flush
21756 227/300 Write block 2717 len 12288 data: 5 5 6
21757 228/300 Read block 4516 len 16384
21758 229/300 Write block 3827 len 4096 data: 4
21759 230/300 Read block 2711 len 36864
21760 231/300 Read block 649 len 24576
21761 232/300 Read block 3630 len 32768
21762 233/300 Write block 2374 len 4096 data: 6
21763 234/300 Read block 3900 len 28672
21764 235/300 Write block 4917 len 20480 data: 3 3 3 5 5
21765 236/300 Write block 1032 len 20480 data: 3 3 3 4 3
21766 237/300 Read block 392 len 36864
21767 238/300 Flush
21768 239/300 Read block 1365 len 32768
21769 240/300 Read block 4807 len 20480
21770 241/300 Read block 1428 len 12288
21771 242/300 Write block 537 len 36864 data: 6 7 6 6 8 7 7 6 6
21772 243/300 Read block 2255 len 36864
21773 244/300 Read block 1494 len 20480
21774 245/300 Flush
21775 246/300 Write block 4704 len 4096 data: 9
21776 247/300 Write block 3474 len 32768 data: 7 8 7 6 4 4 5 5
21777 248/300 Write block 3522 len 20480 data: 5 7 6 5 5
21778 249/300 Write block 126 len 8192 data: 6 6
21779 250/300 Flush
21780 251/300 Write block 1626 len 24576 data: 5 5 5 3 3 3
21781 252/300 Read block 1680 len 28672
21782 253/300 Read block 3243 len 8192
21783 254/300 Read block 2056 len 20480
21784 255/300 Write block 4465 len 4096 data: 2
21785 256/300 Read block 622 len 12288
21786 257/300 Write block 2375 len 24576 data: 6 7 7 8 7 7
21787 258/300 Write block 2223 len 32768 data: 7 6 5 6 5 7 7 8
21788 259/300 Read block 2047 len 12288
21789 260/300 Read block 2320 len 4096
21790 261/300 Write block 3905 len 20480 data: 3 3 5 6 5
21791 262/300 Read block 783 len 8192
21792 263/300 Write block 3952 len 20480 data: 4 5 5 6 6
21793 264/300 Read block 1044 len 28672
21794 265/300 Write block 1107 len 16384 data: 3 3 3 4
21795 266/300 Read block 4971 len 24576
21796 267/300 Write block 4172 len 24576 data: 3 3 4 4 5 5
21797 268/300 Read block 4510 len 12288
21798 269/300 Flush
21799 270/300 Write block 4167 len 8192 data: 3 3
21800 271/300 Write block 1681 len 36864 data: 6 6 5 5 3 4 5 7 6
21801 272/300 Write block 4894 len 32768 data: 4 5 4 4 3 4 5 4
21802 273/300 Read block 1208 len 32768
21803 274/300 Write block 22 len 8192 data: 3 3
21804 275/300 Write block 3720 len 36864 data: 4 4 5 5 6 4 6 5 6
21805 276/300 Read block 1440 len 32768
21806 277/300 Read block 3303 len 32768
21807 278/300 Write block 2321 len 8192 data: 4 4
21808 279/300 Write block 2396 len 8192 data: 10 11
21809 280/300 Flush
21810 281/300 Read block 690 len 16384
21811 282/300 Read block 1552 len 12288
21812 283/300 Flush
21813 284/300 Write block 179 len 8192 data: 4 4
21814 285/300 Read block 1572 len 40960
21815 286/300 Read block 3019 len 24576
21816 287/300 Read block 1021 len 12288
21817 288/300 Flush
21818 289/300 Read block 3384 len 4096
21819 290/300 Flush
21820 291/300 Write block 1426 len 8192 data: 5 5
21821 292/300 Read block 1220 len 4096
21822 293/300 Read block 3079 len 24576
21823 294/300 Write block 3932 len 16384 data: 4 6 5 5
21824 295/300 Read block 4163 len 24576
21825 296/300 Write block 250 len 20480 data: 4 4 6 5 5
21826 297/300 Flush
21827 298/300 Read block 2429 len 28672
21828 299/300 Write block 3714 len 12288 data: 3 3 4
21829 300/300 Read block 99 len 20480
21830 301/300 Flush
21831 Main task has work to do, go find it
21832 got dsc Start(1)
21833 start 1
21834 Jan 23 17:42:20.916 INFO request completed, latency_us: 206, response_code: 204, uri: /start/cid/1, method: POST, req_id: 4f938109-73ed-4c7e-be90-3285d3a53fc7, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
21835 Replay: started 1, returned:Ok(())
21836 ----------------------------------------------------------------
21837 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21838 States: Active Offline Active
21839 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21840 7040 Acked Read 4 Done Sent Done false
21841 7041 Acked Write 3 Done Sent Done false
21842 7042 Acked Read 7 Done Sent Done false
21843 7043 Acked Write 5 Done Sent Done false
21844 7044 Acked Read 10 Done Sent Done false
21845 7045 Acked Flush 0 Done Sent Done false
21846 7046 Acked Read 2 Done Sent Done false
21847 7047 Acked Write 8 Done Sent Done false
21848 7048 Acked Flush 0 Done Sent Done false
21849 7049 Acked Flush 0 Done Sent Done false
21850 7050 Acked Write 4 Done Sent Done false
21851 7051 Acked Write 3 Done Sent Done false
21852 7052 Acked Read 3 Done Sent Done false
21853 7053 Acked Read 2 Done Sent Done false
21854 7054 Acked Read 6 Done Sent Done false
21855 7055 Acked Read 10 Done Sent Done false
21856 7056 Acked Write 3 Done Sent Done false
21857 7057 Acked Flush 0 Done Sent Done false
21858 7058 Acked Read 9 Done Sent Done false
21859 7059 Acked Read 5 Done Sent Done false
21860 7060 Acked Flush 0 Done Sent Done false
21861 7061 Acked Write 6 Done Sent Done false
21862 7062 Acked Write 1 Done Sent Done false
21863 7063 Acked Flush 0 Done Sent Done false
21864 7064 Acked Write 1 Done Sent Done false
21865 7065 Acked Read 6 Done Sent Done false
21866 7066 Acked Read 3 Done Sent Done false
21867 7067 Acked Flush 0 Done Sent Done false
21868 7068 Acked Write 10 Done Sent Done false
21869 7069 Acked Read 1 Done Sent Done false
21870 7070 Acked Flush 0 Done Sent Done false
21871 7071 Acked Read 5 Done Sent Done false
21872 7072 Acked Read 9 Done Sent Done false
21873 7073 Acked Read 5 Done Sent Done false
21874 7074 Acked Read 6 Done Sent Done false
21875 7075 Acked Write 5 Done Sent Done false
21876 7076 Acked Write 2 Done Sent Done false
21877 7077 Acked Read 2 Done Sent Done false
21878 7078 Acked Write 8 Done Sent Done false
21879 7079 Acked Read 8 Done Sent Done false
21880 7080 Acked Write 8 Done Sent Done false
21881 7081 Acked Read 1 Done Sent Done false
21882 7082 Acked Write 3 Done Sent Done false
21883 7083 Acked Read 2 Done Sent Done false
21884 7084 Acked Flush 0 Done Sent Done false
21885 7085 Acked Flush 0 Done Sent Done false
21886 7086 Acked Read 5 Done Sent Done false
21887 7087 Acked Read 4 Done Sent Done false
21888 7088 Acked Write 9 Done Sent Done false
21889 7089 Acked Write 4 Done Sent Done false
21890 7090 Acked Write 5 Done Sent Done false
21891 7091 Acked Write 9 Done Sent Done false
21892 7092 Acked Read 4 Done Sent Done false
21893 7093 Acked Read 3 Done Sent Done false
21894 7094 Acked Read 6 Done Sent Done false
21895 7095 Acked Write 7 Done Sent Done false
21896 7096 Acked Read 8 Done Sent Done false
21897 7097 Acked Read 1 Done Sent Done false
21898 7098 Acked Read 9 Done Sent Done false
21899 7099 Acked Read 2 Done Sent Done false
21900 7100 Acked Read 8 Done Sent Done false
21901 7101 Acked Read 3 Done Sent Done false
21902 7102 Acked Write 7 Done Sent Done false
21903 7103 Acked Read 3 Done Sent Done false
21904 7104 Acked Read 3 Done Sent Done false
21905 7105 Acked Read 10 Done Sent Done false
21906 7106 Acked Read 6 Done Sent Done false
21907 7107 Acked Write 2 Done Sent Done false
21908 7108 Acked Read 6 Done Sent Done false
21909 7109 Acked Flush 0 Done Sent Done false
21910 7110 Acked Read 3 Done Sent Done false
21911 7111 Acked Flush 0 Done Sent Done false
21912 7112 Acked Write 8 Done Sent Done false
21913 7113 Acked Flush 0 Done Sent Done false
21914 7114 Acked Write 3 Done Sent Done false
21915 7115 Acked Read 8 Done Sent Done false
21916 7116 Acked Write 10 Done Sent Done false
21917 7117 Acked Read 6 Done Sent Done false
21918 7118 Acked Write 2 Done Sent Done false
21919 7119 Acked Write 1 Done Sent Done false
21920 7120 Acked Flush 0 Done Sent Done false
21921 7121 Acked Write 8 Done Sent Done false
21922 7122 Acked Read 2 Done Sent Done false
21923 7123 Acked Flush 0 Done Sent Done false
21924 7124 Acked Read 5 Done Sent Done false
21925 7125 Acked Write 4 Done Sent Done false
21926 7126 Acked Read 6 Done Sent Done false
21927 7127 Acked Write 9 Done Sent Done false
21928 7128 Acked Read 2 Done Sent Done false
21929 7129 Acked Read 9 Done Sent Done false
21930 7130 Acked Read 3 Done Sent Done false
21931 7131 Acked Write 1 Done Sent Done false
21932 7132 Acked Read 8 Done Sent Done false
21933 7133 Acked Write 1 Done Sent Done false
21934 7134 Acked Write 1 Done Sent Done false
21935 7135 Acked Flush 0 Done Sent Done false
21936 7136 Acked Write 4 Done Sent Done false
21937 7137 Acked Write 3 Done Sent Done false
21938 7138 Acked Read 6 Done Sent Done false
21939 7139 Acked Read 1 Done Sent Done false
21940 7140 Acked Flush 0 Done Sent Done false
21941 7141 Acked Write 5 Done Sent Done false
21942 7142 Acked Read 10 Done Sent Done false
21943 7143 Acked Read 4 Done Sent Done false
21944 7144 Acked Read 5 Done Sent Done false
21945 7145 Acked Read 5 Done Sent Done false
21946 7146 Acked Read 6 Done Sent Done false
21947 7147 Acked Write 3 Done Sent Done false
21948 7148 Acked Write 7 Done Sent Done false
21949 7149 Acked Write 7 Done Sent Done false
21950 7150 Acked Read 10 Done Sent Done false
21951 7151 Acked Flush 0 Done Sent Done false
21952 7152 Acked Read 5 Done Sent Done false
21953 7153 Acked Read 8 Done Sent Done false
21954 7154 Acked Read 5 Done Sent Done false
21955 7155 Acked Read 6 Done Sent Done false
21956 7156 Acked Write 6 Done Sent Done false
21957 7157 Acked Write 2 Done Sent Done false
21958 7158 Acked Write 9 Done Sent Done false
21959 7159 Acked Read 7 Done Sent Done false
21960 7160 Acked Read 1 Done Sent Done false
21961 7161 Acked Read 6 Done Sent Done false
21962 7162 Acked Flush 0 Done Sent Done false
21963 7163 Acked Write 2 Done Sent Done false
21964 7164 Acked Write 7 Done Sent Done false
21965 7165 Acked Write 4 Done Sent Done false
21966 7166 Acked Read 7 Done Sent Done false
21967 7167 Acked Flush 0 Done Sent Done false
21968 7168 Acked Read 4 Done Sent Done false
21969 7169 Acked Read 3 Done Sent Done false
21970 7170 Acked Write 6 Done Sent Done false
21971 7171 Acked Read 5 Done Sent Done false
21972 7172 Acked Write 1 Done Sent Done false
21973 7173 Acked Flush 0 Done Sent Done false
21974 7174 Acked Read 4 Done Sent Done false
21975 7175 Acked Write 9 Done Sent Done false
21976 7176 Acked Write 6 Done Sent Done false
21977 7177 Acked Read 9 Done Sent Done false
21978 7178 Acked Write 7 Done Sent Done false
21979 7179 Acked Write 2 Done Sent Done false
21980 7180 Acked Write 5 Done Sent Done false
21981 7181 Acked Write 9 Done Sent Done false
21982 7182 Acked Write 2 Done Sent Done false
21983 7183 Acked Write 3 Done Sent Done false
21984 7184 Acked Write 8 Done Sent Done false
21985 7185 Acked Write 9 Done Sent Done false
21986 7186 Acked Write 8 Done Sent Done false
21987 7187 Acked Read 3 Done Sent Done false
21988 7188 Acked Read 3 Done Sent Done false
21989 7189 Acked Write 1 Done Sent Done false
21990 7190 Acked Write 3 Done Sent Done false
21991 7191 Acked Read 9 Done Sent Done false
21992 7192 Acked Read 4 Done Sent Done false
21993 7193 Acked Write 10 Done Sent Done false
21994 7194 Acked Flush 0 Done Sent Done false
21995 7195 Acked Read 3 Done Sent Done false
21996 7196 Acked Flush 0 Done Sent Done false
21997 7197 Acked Flush 0 Done Sent Done false
21998 7198 Acked Write 6 Done Sent Done false
21999 7199 Acked Read 6 Done Sent Done false
22000 7200 Acked Read 1 Done Sent Done false
22001 7201 Acked Read 8 Done Sent Done false
22002 7202 Acked Read 7 Done Sent Done false
22003 7203 Acked Read 3 Done Sent Done false
22004 7204 Acked Write 2 Done Sent Done false
22005 7205 Acked Write 1 Done Sent Done false
22006 7206 Acked Read 5 Done Sent Done false
22007 7207 Acked Read 3 Done Sent Done false
22008 7208 Acked Write 9 Done Sent Done false
22009 7209 Acked Read 4 Done Sent Done false
22010 7210 Acked Write 9 Done Sent Done false
22011 7211 Acked Read 5 Done Sent Done false
22012 7212 Acked Read 7 Done Sent Done false
22013 7213 Acked Read 7 Done Sent Done false
22014 7214 Acked Read 4 Done Sent Done false
22015 7215 Acked Read 2 Done Sent Done false
22016 7216 Acked Write 4 Done Sent Done false
22017 7217 Acked Write 9 Done Sent Done false
22018 7218 Acked Read 8 Done Sent Done false
22019 7219 Acked Write 6 Done Sent Done false
22020 7220 Acked Read 9 Done Sent Done false
22021 7221 Acked Read 4 Done Sent Done false
22022 7222 Acked Write 7 Done Sent Done false
22023 7223 Acked Read 4 Done Sent Done false
22024 7224 Acked Write 1 Done Sent Done false
22025 7225 Acked Read 5 Done Sent Done false
22026 7226 Acked Read 8 Done Sent Done false
22027 7227 Acked Read 6 Done Sent Done false
22028 7228 Acked Flush 0 Done Sent Done false
22029 7229 Acked Read 2 Done Sent Done false
22030 7230 Acked Write 6 Done Sent Done false
22031 7231 Acked Write 4 Done Sent Done false
22032 7232 Acked Write 3 Done Sent Done false
22033 7233 Acked Read 10 Done Sent Done false
22034 7234 Acked Read 9 Done Sent Done false
22035 7235 Acked Read 10 Done Sent Done false
22036 7236 Acked Read 9 Done Sent Done false
22037 7237 Acked Read 4 Done Sent Done false
22038 7238 Acked Read 9 Done Sent Done false
22039 7239 Acked Write 1 Done Sent Done false
22040 7240 Acked Read 9 Done Sent Done false
22041 7241 Acked Read 9 Done Sent Done false
22042 7242 Acked Write 8 Done Sent Done false
22043 7243 Acked Read 3 Done Sent Done false
22044 7244 Acked Read 3 Done Sent Done false
22045 7245 Acked Write 10 Done Sent Done false
22046 7246 Acked Read 5 Done Sent Done false
22047 7247 Acked Read 2 Done Sent Done false
22048 7248 Acked Read 8 Done Sent Done false
22049 7249 Acked Flush 0 Done Sent Done false
22050 7250 Acked Flush 0 Done Sent Done false
22051 7251 Acked Write 4 Done Sent Done false
22052 7252 Acked Read 10 Done Sent Done false
22053 7253 Acked Flush 0 Done Sent Done false
22054 7254 Acked Write 7 Done Sent Done false
22055 7255 Acked Flush 0 Done Sent Done false
22056 7256 Acked Read 9 Done Sent Done false
22057 7257 Acked Read 4 Done Sent Done false
22058 7258 Acked Read 5 Done Sent Done false
22059 7259 Acked Write 8 Done Sent Done false
22060 7260 Acked Read 6 Done Sent Done false
22061 7261 Acked Write 5 Done Sent Done false
22062 7262 Acked Read 1 Done Sent Done false
22063 7263 Acked Write 4 Done Sent Done false
22064 7264 Acked Read 1 Done Sent Done false
22065 7265 Acked Flush 0 Done Sent Done false
22066 7266 Acked Write 3 Done Sent Done false
22067 7267 Acked Read 4 Done Sent Done false
22068 7268 Acked Write 1 Done Sent Done false
22069 7269 Acked Read 9 Done Sent Done false
22070 7270 Acked Read 6 Done Sent Done false
22071 7271 Acked Read 8 Done Sent Done false
22072 7272 Acked Write 1 Done Sent Done false
22073 7273 Acked Read 7 Done Sent Done false
22074 7274 Acked Write 5 Done Sent Done false
22075 7275 Acked Write 5 Done Sent Done false
22076 7276 Acked Read 9 Done Sent Done false
22077 7277 Acked Flush 0 Done Sent Done false
22078 7278 Acked Read 8 Done Sent Done false
22079 7279 Acked Read 5 Done Sent Done false
22080 7280 Acked Read 3 Done Sent Done false
22081 7281 Acked Write 9 Done Sent Done false
22082 7282 Acked Read 9 Done Sent Done false
22083 7283 Acked Read 5 Done Sent Done false
22084 7284 Acked Flush 0 Done Sent Done false
22085 7285 Acked Write 1 Done Sent Done false
22086 7286 Acked Write 8 Done Sent Done false
22087 7287 Acked Write 5 Done Sent Done false
22088 7288 Acked Write 2 Done Sent Done false
22089 7289 Acked Flush 0 Done Sent Done false
22090 7290 Acked Write 6 Done Sent Done false
22091 7291 Acked Read 7 Done Sent Done false
22092 7292 Acked Read 2 Done Sent Done false
22093 7293 Acked Read 5 Done Sent Done false
22094 7294 Acked Write 1 Done Sent Done false
22095 7295 Acked Read 3 Done Sent Done false
22096 7296 Acked Write 6 Done Sent Done false
22097 7297 Acked Write 8 Done Sent Done false
22098 7298 Acked Read 3 Done Sent Done false
22099 7299 Acked Read 1 Done Sent Done false
22100 7300 Acked Write 5 Done Sent Done false
22101 7301 Acked Read 2 Done Sent Done false
22102 7302 Acked Write 5 Done Sent Done false
22103 7303 Acked Read 7 Done Sent Done false
22104 7304 Acked Write 4 Done Sent Done false
22105 7305 Acked Read 6 Done Sent Done false
22106 7306 Acked Write 6 Done Sent Done false
22107 7307 Acked Read 3 Done Sent Done false
22108 7308 Acked Flush 0 Done Sent Done false
22109 7309 Acked Write 2 Done Sent Done false
22110 7310 Acked Write 9 Done Sent Done false
22111 7311 Acked Write 8 Done Sent Done false
22112 7312 Acked Read 8 Done Sent Done false
22113 7313 Acked Write 2 Done Sent Done false
22114 7314 Acked Write 9 Done Sent Done false
22115 7315 Acked Read 8 Done Sent Done false
22116 7316 Acked Read 8 Done Sent Done false
22117 7317 Acked Write 2 Done Sent Done false
22118 7318 Acked Write 2 Done Sent Done false
22119 7319 Acked Flush 0 Done Sent Done false
22120 7320 Acked Read 4 Done Sent Done false
22121 7321 Acked Read 3 Done Sent Done false
22122 7322 Acked Flush 0 Done Sent Done false
22123 7323 Acked Write 2 Done Sent Done false
22124 7324 Acked Read 10 Done Sent Done false
22125 7325 Acked Read 6 Done Sent Done false
22126 7326 Acked Read 3 Done Sent Done false
22127 7327 Acked Flush 0 Done Sent Done false
22128 7328 Acked Read 1 Done Sent Done false
22129 7329 Acked Flush 0 Done Sent Done false
22130 7330 Acked Write 2 Done Sent Done false
22131 7331 Acked Read 1 Done Sent Done false
22132 7332 Acked Read 6 Done Sent Done false
22133 7333 Acked Write 4 Done Sent Done false
22134 7334 Acked Read 6 Done Sent Done false
22135 7335 Acked Write 5 Done Sent Done false
22136 7336 Acked Flush 0 Done Sent Done false
22137 7337 Acked Read 7 Done Sent Done false
22138 7338 Acked Write 3 Done Sent Done false
22139 7339 Acked Read 5 Done Sent Done false
22140 7340 Acked Flush 0 Done Sent Done false
22141 STATES DS:0 DS:1 DS:2 TOTAL
22142 Sent 0 301 0 301
22143 Done 301 0 301 602
22144 Skipped 0 0 0 0
22145 Error 0 0 0 0
22146 Last Flush: Some(JobId(7340)) Some(JobId(7039)) Some(JobId(7340))
22147 Downstairs last five completed: 7039 7038 7037 7036 7035
22148 Upstairs last five completed: 7340 7339 7338 7337 7336
22149 CLIENT: Up:0 ds:301 act:2
22150 [8820] got start action
22151 [8820] I am going to restart
22152 Starting downstairs at port 8820
22153 Make output file at "/tmp/dsc/downstairs-8820.txt"
22154 [8820][1] reports Starting
22155 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1226)
22156 [8820][1] reports Running
22157 ----------------------------------------------------------------
22158 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22159 States: Active Offline Active
22160 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22161 7040 Acked Read 4 Done Sent Done false
22162 7041 Acked Write 3 Done Sent Done false
22163 7042 Acked Read 7 Done Sent Done false
22164 7043 Acked Write 5 Done Sent Done false
22165 7044 Acked Read 10 Done Sent Done false
22166 7045 Acked Flush 0 Done Sent Done false
22167 7046 Acked Read 2 Done Sent Done false
22168 7047 Acked Write 8 Done Sent Done false
22169 7048 Acked Flush 0 Done Sent Done false
22170 7049 Acked Flush 0 Done Sent Done false
22171 7050 Acked Write 4 Done Sent Done false
22172 7051 Acked Write 3 Done Sent Done false
22173 7052 Acked Read 3 Done Sent Done false
22174 7053 Acked Read 2 Done Sent Done false
22175 7054 Acked Read 6 Done Sent Done false
22176 7055 Acked Read 10 Done Sent Done false
22177 7056 Acked Write 3 Done Sent Done false
22178 7057 Acked Flush 0 Done Sent Done false
22179 7058 Acked Read 9 Done Sent Done false
22180 7059 Acked Read 5 Done Sent Done false
22181 7060 Acked Flush 0 Done Sent Done false
22182 7061 Acked Write 6 Done Sent Done false
22183 7062 Acked Write 1 Done Sent Done false
22184 7063 Acked Flush 0 Done Sent Done false
22185 7064 Acked Write 1 Done Sent Done false
22186 7065 Acked Read 6 Done Sent Done false
22187 7066 Acked Read 3 Done Sent Done false
22188 7067 Acked Flush 0 Done Sent Done false
22189 7068 Acked Write 10 Done Sent Done false
22190 7069 Acked Read 1 Done Sent Done false
22191 7070 Acked Flush 0 Done Sent Done false
22192 7071 Acked Read 5 Done Sent Done false
22193 7072 Acked Read 9 Done Sent Done false
22194 7073 Acked Read 5 Done Sent Done false
22195 7074 Acked Read 6 Done Sent Done false
22196 7075 Acked Write 5 Done Sent Done false
22197 7076 Acked Write 2 Done Sent Done false
22198 7077 Acked Read 2 Done Sent Done false
22199 7078 Acked Write 8 Done Sent Done false
22200 7079 Acked Read 8 Done Sent Done false
22201 7080 Acked Write 8 Done Sent Done false
22202 7081 Acked Read 1 Done Sent Done false
22203 7082 Acked Write 3 Done Sent Done false
22204 7083 Acked Read 2 Done Sent Done false
22205 7084 Acked Flush 0 Done Sent Done false
22206 7085 Acked Flush 0 Done Sent Done false
22207 7086 Acked Read 5 Done Sent Done false
22208 7087 Acked Read 4 Done Sent Done false
22209 7088 Acked Write 9 Done Sent Done false
22210 7089 Acked Write 4 Done Sent Done false
22211 7090 Acked Write 5 Done Sent Done false
22212 7091 Acked Write 9 Done Sent Done false
22213 7092 Acked Read 4 Done Sent Done false
22214 7093 Acked Read 3 Done Sent Done false
22215 7094 Acked Read 6 Done Sent Done false
22216 7095 Acked Write 7 Done Sent Done false
22217 7096 Acked Read 8 Done Sent Done false
22218 7097 Acked Read 1 Done Sent Done false
22219 7098 Acked Read 9 Done Sent Done false
22220 7099 Acked Read 2 Done Sent Done false
22221 7100 Acked Read 8 Done Sent Done false
22222 7101 Acked Read 3 Done Sent Done false
22223 7102 Acked Write 7 Done Sent Done false
22224 7103 Acked Read 3 Done Sent Done false
22225 7104 Acked Read 3 Done Sent Done false
22226 7105 Acked Read 10 Done Sent Done false
22227 7106 Acked Read 6 Done Sent Done false
22228 7107 Acked Write 2 Done Sent Done false
22229 7108 Acked Read 6 Done Sent Done false
22230 7109 Acked Flush 0 Done Sent Done false
22231 7110 Acked Read 3 Done Sent Done false
22232 7111 Acked Flush 0 Done Sent Done false
22233 7112 Acked Write 8 Done Sent Done false
22234 7113 Acked Flush 0 Done Sent Done false
22235 7114 Acked Write 3 Done Sent Done false
22236 7115 Acked Read 8 Done Sent Done false
22237 7116 Acked Write 10 Done Sent Done false
22238 7117 Acked Read 6 Done Sent Done false
22239 7118 Acked Write 2 Done Sent Done false
22240 7119 Acked Write 1 Done Sent Done false
22241 7120 Acked Flush 0 Done Sent Done false
22242 7121 Acked Write 8 Done Sent Done false
22243 7122 Acked Read 2 Done Sent Done false
22244 7123 Acked Flush 0 Done Sent Done false
22245 7124 Acked Read 5 Done Sent Done false
22246 7125 Acked Write 4 Done Sent Done false
22247 7126 Acked Read 6 Done Sent Done false
22248 7127 Acked Write 9 Done Sent Done false
22249 7128 Acked Read 2 Done Sent Done false
22250 7129 Acked Read 9 Done Sent Done false
22251 7130 Acked Read 3 Done Sent Done false
22252 7131 Acked Write 1 Done Sent Done false
22253 7132 Acked Read 8 Done Sent Done false
22254 7133 Acked Write 1 Done Sent Done false
22255 7134 Acked Write 1 Done Sent Done false
22256 7135 Acked Flush 0 Done Sent Done false
22257 7136 Acked Write 4 Done Sent Done false
22258 7137 Acked Write 3 Done Sent Done false
22259 7138 Acked Read 6 Done Sent Done false
22260 7139 Acked Read 1 Done Sent Done false
22261 7140 Acked Flush 0 Done Sent Done false
22262 7141 Acked Write 5 Done Sent Done false
22263 7142 Acked Read 10 Done Sent Done false
22264 7143 Acked Read 4 Done Sent Done false
22265 7144 Acked Read 5 Done Sent Done false
22266 7145 Acked Read 5 Done Sent Done false
22267 7146 Acked Read 6 Done Sent Done false
22268 7147 Acked Write 3 Done Sent Done false
22269 7148 Acked Write 7 Done Sent Done false
22270 7149 Acked Write 7 Done Sent Done false
22271 7150 Acked Read 10 Done Sent Done false
22272 7151 Acked Flush 0 Done Sent Done false
22273 7152 Acked Read 5 Done Sent Done false
22274 7153 Acked Read 8 Done Sent Done false
22275 7154 Acked Read 5 Done Sent Done false
22276 7155 Acked Read 6 Done Sent Done false
22277 7156 Acked Write 6 Done Sent Done false
22278 7157 Acked Write 2 Done Sent Done false
22279 7158 Acked Write 9 Done Sent Done false
22280 7159 Acked Read 7 Done Sent Done false
22281 7160 Acked Read 1 Done Sent Done false
22282 7161 Acked Read 6 Done Sent Done false
22283 7162 Acked Flush 0 Done Sent Done false
22284 7163 Acked Write 2 Done Sent Done false
22285 7164 Acked Write 7 Done Sent Done false
22286 7165 Acked Write 4 Done Sent Done false
22287 7166 Acked Read 7 Done Sent Done false
22288 7167 Acked Flush 0 Done Sent Done false
22289 7168 Acked Read 4 Done Sent Done false
22290 7169 Acked Read 3 Done Sent Done false
22291 7170 Acked Write 6 Done Sent Done false
22292 7171 Acked Read 5 Done Sent Done false
22293 7172 Acked Write 1 Done Sent Done false
22294 7173 Acked Flush 0 Done Sent Done false
22295 7174 Acked Read 4 Done Sent Done false
22296 7175 Acked Write 9 Done Sent Done false
22297 7176 Acked Write 6 Done Sent Done false
22298 7177 Acked Read 9 Done Sent Done false
22299 7178 Acked Write 7 Done Sent Done false
22300 7179 Acked Write 2 Done Sent Done false
22301 7180 Acked Write 5 Done Sent Done false
22302 7181 Acked Write 9 Done Sent Done false
22303 7182 Acked Write 2 Done Sent Done false
22304 7183 Acked Write 3 Done Sent Done false
22305 7184 Acked Write 8 Done Sent Done false
22306 7185 Acked Write 9 Done Sent Done false
22307 7186 Acked Write 8 Done Sent Done false
22308 7187 Acked Read 3 Done Sent Done false
22309 7188 Acked Read 3 Done Sent Done false
22310 7189 Acked Write 1 Done Sent Done false
22311 7190 Acked Write 3 Done Sent Done false
22312 7191 Acked Read 9 Done Sent Done false
22313 7192 Acked Read 4 Done Sent Done false
22314 7193 Acked Write 10 Done Sent Done false
22315 7194 Acked Flush 0 Done Sent Done false
22316 7195 Acked Read 3 Done Sent Done false
22317 7196 Acked Flush 0 Done Sent Done false
22318 7197 Acked Flush 0 Done Sent Done false
22319 7198 Acked Write 6 Done Sent Done false
22320 7199 Acked Read 6 Done Sent Done false
22321 7200 Acked Read 1 Done Sent Done false
22322 7201 Acked Read 8 Done Sent Done false
22323 7202 Acked Read 7 Done Sent Done false
22324 7203 Acked Read 3 Done Sent Done false
22325 7204 Acked Write 2 Done Sent Done false
22326 7205 Acked Write 1 Done Sent Done false
22327 7206 Acked Read 5 Done Sent Done false
22328 7207 Acked Read 3 Done Sent Done false
22329 7208 Acked Write 9 Done Sent Done false
22330 7209 Acked Read 4 Done Sent Done false
22331 7210 Acked Write 9 Done Sent Done false
22332 7211 Acked Read 5 Done Sent Done false
22333 7212 Acked Read 7 Done Sent Done false
22334 7213 Acked Read 7 Done Sent Done false
22335 7214 Acked Read 4 Done Sent Done false
22336 7215 Acked Read 2 Done Sent Done false
22337 7216 Acked Write 4 Done Sent Done false
22338 7217 Acked Write 9 Done Sent Done false
22339 7218 Acked Read 8 Done Sent Done false
22340 7219 Acked Write 6 Done Sent Done false
22341 7220 Acked Read 9 Done Sent Done false
22342 7221 Acked Read 4 Done Sent Done false
22343 7222 Acked Write 7 Done Sent Done false
22344 7223 Acked Read 4 Done Sent Done false
22345 7224 Acked Write 1 Done Sent Done false
22346 7225 Acked Read 5 Done Sent Done false
22347 7226 Acked Read 8 Done Sent Done false
22348 7227 Acked Read 6 Done Sent Done false
22349 7228 Acked Flush 0 Done Sent Done false
22350 7229 Acked Read 2 Done Sent Done false
22351 7230 Acked Write 6 Done Sent Done false
22352 7231 Acked Write 4 Done Sent Done false
22353 7232 Acked Write 3 Done Sent Done false
22354 7233 Acked Read 10 Done Sent Done false
22355 7234 Acked Read 9 Done Sent Done false
22356 7235 Acked Read 10 Done Sent Done false
22357 7236 Acked Read 9 Done Sent Done false
22358 7237 Acked Read 4 Done Sent Done false
22359 7238 Acked Read 9 Done Sent Done false
22360 7239 Acked Write 1 Done Sent Done false
22361 7240 Acked Read 9 Done Sent Done false
22362 7241 Acked Read 9 Done Sent Done false
22363 7242 Acked Write 8 Done Sent Done false
22364 7243 Acked Read 3 Done Sent Done false
22365 7244 Acked Read 3 Done Sent Done false
22366 7245 Acked Write 10 Done Sent Done false
22367 7246 Acked Read 5 Done Sent Done false
22368 7247 Acked Read 2 Done Sent Done false
22369 7248 Acked Read 8 Done Sent Done false
22370 7249 Acked Flush 0 Done Sent Done false
22371 7250 Acked Flush 0 Done Sent Done false
22372 7251 Acked Write 4 Done Sent Done false
22373 7252 Acked Read 10 Done Sent Done false
22374 7253 Acked Flush 0 Done Sent Done false
22375 7254 Acked Write 7 Done Sent Done false
22376 7255 Acked Flush 0 Done Sent Done false
22377 7256 Acked Read 9 Done Sent Done false
22378 7257 Acked Read 4 Done Sent Done false
22379 7258 Acked Read 5 Done Sent Done false
22380 7259 Acked Write 8 Done Sent Done false
22381 7260 Acked Read 6 Done Sent Done false
22382 7261 Acked Write 5 Done Sent Done false
22383 7262 Acked Read 1 Done Sent Done false
22384 7263 Acked Write 4 Done Sent Done false
22385 7264 Acked Read 1 Done Sent Done false
22386 7265 Acked Flush 0 Done Sent Done false
22387 7266 Acked Write 3 Done Sent Done false
22388 7267 Acked Read 4 Done Sent Done false
22389 7268 Acked Write 1 Done Sent Done false
22390 7269 Acked Read 9 Done Sent Done false
22391 7270 Acked Read 6 Done Sent Done false
22392 7271 Acked Read 8 Done Sent Done false
22393 7272 Acked Write 1 Done Sent Done false
22394 7273 Acked Read 7 Done Sent Done false
22395 7274 Acked Write 5 Done Sent Done false
22396 7275 Acked Write 5 Done Sent Done false
22397 7276 Acked Read 9 Done Sent Done false
22398 7277 Acked Flush 0 Done Sent Done false
22399 7278 Acked Read 8 Done Sent Done false
22400 7279 Acked Read 5 Done Sent Done false
22401 7280 Acked Read 3 Done Sent Done false
22402 7281 Acked Write 9 Done Sent Done false
22403 7282 Acked Read 9 Done Sent Done false
22404 7283 Acked Read 5 Done Sent Done false
22405 7284 Acked Flush 0 Done Sent Done false
22406 7285 Acked Write 1 Done Sent Done false
22407 7286 Acked Write 8 Done Sent Done false
22408 7287 Acked Write 5 Done Sent Done false
22409 7288 Acked Write 2 Done Sent Done false
22410 7289 Acked Flush 0 Done Sent Done false
22411 7290 Acked Write 6 Done Sent Done false
22412 7291 Acked Read 7 Done Sent Done false
22413 7292 Acked Read 2 Done Sent Done false
22414 7293 Acked Read 5 Done Sent Done false
22415 7294 Acked Write 1 Done Sent Done false
22416 7295 Acked Read 3 Done Sent Done false
22417 7296 Acked Write 6 Done Sent Done false
22418 7297 Acked Write 8 Done Sent Done false
22419 7298 Acked Read 3 Done Sent Done false
22420 7299 Acked Read 1 Done Sent Done false
22421 7300 Acked Write 5 Done Sent Done false
22422 7301 Acked Read 2 Done Sent Done false
22423 7302 Acked Write 5 Done Sent Done false
22424 7303 Acked Read 7 Done Sent Done false
22425 7304 Acked Write 4 Done Sent Done false
22426 7305 Acked Read 6 Done Sent Done false
22427 7306 Acked Write 6 Done Sent Done false
22428 7307 Acked Read 3 Done Sent Done false
22429 7308 Acked Flush 0 Done Sent Done false
22430 7309 Acked Write 2 Done Sent Done false
22431 7310 Acked Write 9 Done Sent Done false
22432 7311 Acked Write 8 Done Sent Done false
22433 7312 Acked Read 8 Done Sent Done false
22434 7313 Acked Write 2 Done Sent Done false
22435 7314 Acked Write 9 Done Sent Done false
22436 7315 Acked Read 8 Done Sent Done false
22437 7316 Acked Read 8 Done Sent Done false
22438 7317 Acked Write 2 Done Sent Done false
22439 7318 Acked Write 2 Done Sent Done false
22440 7319 Acked Flush 0 Done Sent Done false
22441 7320 Acked Read 4 Done Sent Done false
22442 7321 Acked Read 3 Done Sent Done false
22443 7322 Acked Flush 0 Done Sent Done false
22444 7323 Acked Write 2 Done Sent Done false
22445 7324 Acked Read 10 Done Sent Done false
22446 7325 Acked Read 6 Done Sent Done false
22447 7326 Acked Read 3 Done Sent Done false
22448 7327 Acked Flush 0 Done Sent Done false
22449 7328 Acked Read 1 Done Sent Done false
22450 7329 Acked Flush 0 Done Sent Done false
22451 7330 Acked Write 2 Done Sent Done false
22452 7331 Acked Read 1 Done Sent Done false
22453 7332 Acked Read 6 Done Sent Done false
22454 7333 Acked Write 4 Done Sent Done false
22455 7334 Acked Read 6 Done Sent Done false
22456 7335 Acked Write 5 Done Sent Done false
22457 7336 Acked Flush 0 Done Sent Done false
22458 7337 Acked Read 7 Done Sent Done false
22459 7338 Acked Write 3 Done Sent Done false
22460 7339 Acked Read 5 Done Sent Done false
22461 7340 Acked Flush 0 Done Sent Done false
22462 STATES DS:0 DS:1 DS:2 TOTAL
22463 Sent 0 301 0 301
22464 Done 301 0 301 602
22465 Skipped 0 0 0 0
22466 Error 0 0 0 0
22467 Last Flush: Some(JobId(7340)) Some(JobId(7039)) Some(JobId(7340))
22468 Downstairs last five completed: 7039 7038 7037 7036 7035
22469 Upstairs last five completed: 7340 7339 7338 7337 7336
22470 CLIENT: Up:0 ds:301 act:2
224712025-01-23T17:42:26.678ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
224722025-01-23T17:42:26.678ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49001) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
224732025-01-23T17:42:26.679ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
224742025-01-23T17:42:26.680ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
224752025-01-23T17:42:26.680ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
224762025-01-23T17:42:26.680ZINFOcrucible: send last flush ID to this DS: Some(JobId(7039)) = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
224772025-01-23T17:42:26.680ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
224782025-01-23T17:42:26.680ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7039)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
224792025-01-23T17:42:26.680ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7039)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
22480 ----------------------------------------------------------------
22481 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22482 Downstairs last five completed: 7340 7339 7338 7337 7336
22483 Upstairs last five completed: 7340 7339 7338 7337 7336
22484 CLIENT: Up:0 ds:0 act:3
22485 Replay: All jobs finished
22486 Main task has work to do, go find it
22487 got dsc Stop(1)
22488 stop 1
22489 Jan 23 17:42:28.920 INFO [8820] Got stop action so:false kr:false
22490 request completed, latency_us: 366, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 3a253fa5-e791-4a3f-a818-58620a807280, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
224912025-01-23T17:42:28.921ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
22492 Jan 23 17:42:28.921 INFO request completed, latency_us: 180, response_code: 200, uri: /state/cid/1, method: GET, req_id: 3c6a132d-a7ee-4a75-80ac-889e9ad4493b, remote_addr: 127.0.0{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:42:28.92185563Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
22493 .1:42015, local_addr: 127.0.0.1:9998
224942025-01-23T17:42:28.921ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
224952025-01-23T17:42:28.921ZINFOcrucible: client task is exiting = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
22496 [8820] Exited with: ExitStatus(unix_wait_status(9))
22497 [8820][1] reports Exit
22498 Jan 23 17:42:32.923 INFO request completed, latency_us: 329, response_code: 200, uri: /state/cid/1, method: GET, req_id: 002352ec-89ae-4a4b-a9d8-5c9549f71577, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
22499 001/300 Write block 4009 len 32768 data: 3 4 4 3 2 3 3 3
22500 002/300 Read block 96 len 8192
22501 003/300 Read block 3371 len 4096
22502 004/300 Write block 1401 len 36864 data: 4 5 5 5 4 5 5 5 5
22503 005/300 Read block 2877 len 12288
22504 006/300 Write block 4886 len 40960 data: 5 5 4 4 2 2 2 2 5 6
22505 007/300 Write block 3944 len 28672 data: 6 5 6 6 4 5 5
22506 008/300 Write block 3178 len 32768 data: 4 5 6 6 4 7 7 5
22507 009/300 Write block 4103 len 24576 data: 4 4 4 4 4 3
22508 010/300 Read block 597 len 28672
22509 011/300 Flush
22510 012/300 Flush
22511 013/300 Read block 810 len 16384
22512 014/300 Write block 3805 len 16384 data: 5 6 5 5
22513 015/300 Read block 502 len 12288
22514 016/300 Write block 758 len 36864 data: 5 5 6 4 4 4 5 5 4
22515 017/300 Write block 2127 len 36864 data: 5 5 4 3 4 5 3 4 4
22516 018/300 Read block 4968 len 4096
22517 019/300 Read block 3112 len 16384
22518 020/300 Flush
22519 021/300 Read block 809 len 24576
22520 022/300 Write block 2159 len 40960 data: 6 6 6 5 5 3 3 3 4 4
22521 023/300 Read block 4029 len 24576
22522 024/300 Read block 653 len 8192
22523 025/300 Write block 2296 len 4096 data: 3
22524 026/300 Write block 1811 len 24576 data: 2 2 2 2 4 4
22525 027/300 Read block 102 len 8192
22526 028/300 Read block 1200 len 20480
22527 029/300 Read block 3764 len 24576
22528 030/300 Write block 640 len 36864 data: 8 8 7 7 7 8 7 4 5
22529 031/300 Write block 2381 len 20480 data: 6 6 7 9 8
22530 032/300 Read block 4741 len 20480
22531 033/300 Write block 777 len 16384 data: 6 8 7 7
22532 034/300 Read block 3331 len 24576
22533 035/300 Write block 4134 len 8192 data: 5 7
22534 036/300 Write block 2128 len 32768 data: 6 5 4 5 6 4 5 5
22535 037/300 Read block 2755 len 12288
22536 038/300 Write block 37 len 4096 data: 3
22537 039/300 Read block 3465 len 32768
22538 040/300 Flush
22539 041/300 Read block 448 len 20480
22540 042/300 Read block 407 len 36864
22541 043/300 Write block 2477 len 12288 data: 6 6 7
22542 044/300 Write block 3713 len 24576 data: 3 4 4 5 4 3
22543 045/300 Write block 1993 len 28672 data: 4 5 3 3 3 3 4
22544 046/300 Write block 440 len 40960 data: 3 4 3 3 3 3 3 5 4 2
22545 047/300 Read block 4806 len 16384
22546 048/300 Read block 3967 len 4096
22547 049/300 Write block 4536 len 20480 data: 5 6 6 6 5
22548 050/300 Write block 4587 len 20480 data: 4 3 4 4 5
22549 051/300 Read block 1991 len 28672
22550 052/300 Write block 3196 len 8192 data: 3 5
22551 053/300 Read block 2902 len 20480
22552 054/300 Read block 2239 len 4096
22553 055/300 Read block 348 len 28672
22554 056/300 Read block 2422 len 40960
22555 057/300 Read block 2309 len 12288
22556 058/300 Read block 2571 len 16384
22557 059/300 Flush
22558 060/300 Read block 4619 len 28672
22559 061/300 Write block 538 len 28672 data: 8 7 7 9 8 8 7
22560 062/300 Read block 3830 len 40960
22561 063/300 Read block 4253 len 8192
22562 064/300 Read block 565 len 24576
22563 065/300 Flush
22564 066/300 Read block 304 len 4096
22565 067/300 Read block 1800 len 4096
22566 068/300 Read block 3374 len 20480
22567 069/300 Write block 3653 len 12288 data: 8 7 7
22568 070/300 Flush
22569 071/300 Read block 4345 len 40960
22570 072/300 Flush
22571 073/300 Write block 3581 len 8192 data: 7 7
22572 074/300 Read block 2826 len 20480
22573 075/300 Write block 4295 len 4096 data: 2
22574 076/300 Read block 3620 len 4096
22575 077/300 Read block 2480 len 28672
22576 078/300 Read block 3760 len 36864
22577 079/300 Write block 540 len 8192 data: 8 10
22578 080/300 Write block 2187 len 4096 data: 3
22579 081/300 Write block 4990 len 28672 data: 4 3 3 5 5 5 4
22580 082/300 Write block 1874 len 40960 data: 7 6 5 5 5 4 5 4 5 5
22581 083/300 Write block 975 len 28672 data: 6 4 4 4 4 4 5
22582 084/300 Write block 2001 len 24576 data: 5 5 5 5 5 3
22583 085/300 Read block 4410 len 28672
22584 086/300 Write block 4185 len 12288 data: 3 2 2
22585 087/300 Read block 2706 len 28672
22586 088/300 Read block 4449 len 20480
22587 089/300 Write block 3704 len 28672 data: 8 7 8 8 7 7 6
22588 090/300 Flush
22589 091/300 Flush
22590 092/300 Write block 3369 len 12288 data: 4 4 5
22591 093/300 Read block 2983 len 4096
22592 094/300 Write block 2171 len 40960 data: 4 3 3 4 4 3 3 4 4 5
22593 095/300 Read block 260 len 36864
22594 096/300 Read block 1800 len 20480
22595 097/300 Read block 3292 len 24576
22596 098/300 Flush
22597 099/300 Read block 939 len 16384
22598 100/300 Write block 2800 len 28672 data: 6 5 5 4 5 7 7
22599 101/300 Read block 868 len 16384
22600 102/300 Flush
22601 103/300 Read block 1138 len 16384
22602 104/300 Flush
22603 105/300 Write block 2659 len 40960 data: 3 3 4 4 2 2 2 4 4 5
22604 106/300 Read block 1359 len 40960
22605 107/300 Write block 633 len 24576 data: 4 4 5 5 5 4
22606 108/300 Read block 2682 len 12288
22607 109/300 Read block 462 len 32768
22608 110/300 Flush
22609 111/300 Read block 2663 len 40960
22610 112/300 Write block 4873 len 40960 data: 5 6 5 5 3 3 3 3 3 3
22611 113/300 Read block 3188 len 36864
22612 114/300 Write block 1920 len 12288 data: 10 9 9
22613 115/300 Read block 3958 len 4096
22614 116/300 Read block 4434 len 40960
22615 117/300 Read block 2472 len 36864
22616 118/300 Write block 4362 len 32768 data: 5 6 5 6 6 6 6 6
22617 119/300 Write block 2509 len 28672 data: 2 2 2 2 2 3 3
22618 120/300 Read block 859 len 20480
22619 121/300 Read block 924 len 28672
22620 122/300 Read block 4992 len 32768
22621 123/300 Flush
22622 124/300 Flush
22623 125/300 Flush
22624 126/300 Write block 2848 len 24576 data: 3 3 3 4 4 4
22625 127/300 Read block 1883 len 36864
22626 128/300 Write block 2989 len 8192 data: 5 6
22627 129/300 Write block 2685 len 28672 data: 7 7 7 8 5 5 4
22628 130/300 Write block 3033 len 36864 data: 10 9 8 7 6 5 5 5 5
22629 131/300 Read block 567 len 12288
22630 132/300 Read block 2944 len 12288
22631 133/300 Write block 1720 len 36864 data: 3 3 3 3 3 3 3 3 4
22632 134/300 Write block 2692 len 20480 data: 3 3 3 5 5
22633 135/300 Read block 4092 len 40960
22634 136/300 Read block 1481 len 28672
22635 137/300 Read block 2441 len 40960
22636 138/300 Write block 416 len 8192 data: 3 3
22637 139/300 Read block 2228 len 32768
22638 140/300 Read block 3277 len 16384
22639 141/300 Write block 2383 len 28672 data: 8 10 9 7 4 4 4
22640 142/300 Read block 1949 len 12288
22641 143/300 Flush
22642 144/300 Read block 2237 len 28672
22643 145/300 Read block 2432 len 16384
22644 146/300 Write block 4109 len 20480 data: 2 2 2 3 5
22645 147/300 Write block 2222 len 12288 data: 7 8 7
22646 148/300 Read block 1096 len 24576
22647 149/300 Write block 2665 len 12288 data: 3 5 5
22648 150/300 Flush
22649 151/300 Flush
22650 152/300 Write block 3267 len 12288 data: 4 3 4
22651 153/300 Write block 642 len 8192 data: 8 8
22652 154/300 Read block 1775 len 24576
22653 155/300 Read block 1783 len 24576
22654 156/300 Write block 3876 len 4096 data: 5
22655 157/300 Flush
22656 158/300 Write block 4528 len 4096 data: 4
22657 159/300 Write block 734 len 16384 data: 4 6 6 4
22658 160/300 Read block 2602 len 40960
22659 161/300 Write block 3154 len 28672 data: 4 4 5 6 7 7 7
22660 162/300 Read block 2708 len 16384
22661 163/300 Read block 1801 len 40960
22662 164/300 Write block 4780 len 36864 data: 4 3 3 3 3 3 3 3 2
22663 165/300 Read block 3209 len 12288
22664 166/300 Read block 4099 len 24576
22665 167/300 Read block 4157 len 20480
22666 168/300 Read block 3558 len 8192
22667 169/300 Write block 4548 len 20480 data: 2 2 3 3 5
22668 170/300 Flush
22669 171/300 Write block 233 len 20480 data: 5 4 3 4 7
22670 172/300 Write block 3542 len 24576 data: 6 5 5 5 7 7
22671 173/300 Read block 3711 len 16384
22672 174/300 Flush
22673 175/300 Read block 1281 len 32768
22674 176/300 Write block 3017 len 4096 data: 9
22675 177/300 Write block 512 len 40960 data: 3 4 3 2 2 2 3 4 4 5
22676 178/300 Read block 2393 len 4096
22677 179/300 Read block 3418 len 36864
22678 180/300 Write block 3984 len 20480 data: 4 4 5 4 5
22679 181/300 Read block 2888 len 24576
22680 182/300 Write block 2140 len 12288 data: 6 5 4
22681 183/300 Write block 3346 len 4096 data: 5
22682 184/300 Flush
22683 185/300 Write block 1294 len 8192 data: 4 4
22684 186/300 Write block 3386 len 4096 data: 5
22685 187/300 Read block 1836 len 32768
22686 188/300 Read block 1959 len 36864
22687 189/300 Read block 1672 len 28672
22688 190/300 Write block 2075 len 32768 data: 6 5 4 4 4 6 5 6
22689 191/300 Read block 982 len 16384
22690 192/300 Read block 1759 len 8192
22691 193/300 Read block 1845 len 8192
22692 194/300 Flush
22693 195/300 Write block 1544 len 32768 data: 3 4 4 3 4 5 6 5
22694 196/300 Write block 821 len 24576 data: 6 7 7 8 8 9
22695 197/300 Read block 511 len 8192
22696 198/300 Read block 753 len 16384
22697 199/300 Flush
22698 200/300 Write block 4778 len 28672 data: 5 6 5 4 4 4 4
22699 201/300 Read block 231 len 8192
22700 202/300 Read block 689 len 20480
22701 203/300 Write block 4514 len 28672 data: 6 7 6 5 5 5 6
22702 204/300 Write block 381 len 20480 data: 2 2 3 4 5
22703 205/300 Write block 453 len 20480 data: 3 3 4 4 3
22704 206/300 Write block 212 len 36864 data: 7 7 7 6 6 8 8 5 3
22705 207/300 Read block 58 len 4096
22706 208/300 Read block 1660 len 24576
22707 209/300 Read block 2046 len 28672
22708 210/300 Write block 2869 len 28672 data: 8 6 5 4 4 3 4
22709 211/300 Write block 3681 len 20480 data: 5 5 5 5 9
22710 212/300 Read block 3006 len 20480
22711 213/300 Write block 4061 len 28672 data: 5 5 5 5 3 4 3
22712 214/300 Read block 2216 len 24576
22713 215/300 Write block 4894 len 40960 data: 6 7 5 5 4 5 6 5 4 4
22714 216/300 Write block 2587 len 40960 data: 3 3 4 5 6 4 4 6 7 6
22715 217/300 Write block 3615 len 4096 data: 5
22716 218/300 Read block 539 len 12288
22717 219/300 Read block 2283 len 24576
22718 220/300 Write block 4544 len 36864 data: 4 5 4 3 3 3 4 4 6
22719 221/300 Read block 4879 len 4096
22720 222/300 Read block 4562 len 24576
22721 223/300 Flush
22722 224/300 Read block 4834 len 24576
22723 225/300 Write block 414 len 24576 data: 3 3 4 4 2 2
22724 226/300 Flush
22725 227/300 Read block 2701 len 32768
22726 228/300 Write block 2409 len 36864 data: 5 5 5 6 9 7 7 7 8
22727 229/300 Read block 2105 len 28672
22728 230/300 Flush
22729 231/300 Read block 1331 len 32768
22730 232/300 Write block 3308 len 36864 data: 6 6 6 5 5 5 5 6 6
22731 233/300 Write block 1466 len 24576 data: 3 3 3 3 4 5
22732 234/300 Write block 319 len 20480 data: 2 2 3 4 4
22733 235/300 Flush
22734 236/300 Flush
22735 237/300 Read block 1929 len 12288
22736 238/300 Read block 1173 len 4096
22737 239/300 Write block 1353 len 40960 data: 5 5 4 3 3 3 2 3 3 3
22738 240/300 Write block 3410 len 36864 data: 6 5 4 4 5 5 6 5 5
22739 241/300 Read block 938 len 28672
22740 242/300 Read block 1423 len 24576
22741 243/300 Read block 2060 len 8192
22742 244/300 Flush
22743 245/300 Flush
22744 246/300 Read block 4370 len 40960
22745 247/300 Write block 4103 len 16384 data: 5 5 5 5
22746 248/300 Write block 1169 len 4096 data: 2
22747 249/300 Flush
22748 250/300 Read block 1485 len 36864
22749 251/300 Read block 287 len 20480
22750 252/300 Read block 750 len 28672
22751 253/300 Write block 2816 len 8192 data: 7 7
22752 254/300 Flush
22753 255/300 Write block 746 len 32768 data: 6 7 8 8 8 6 5 7
22754 256/300 Write block 2920 len 24576 data: 4 5 5 5 5 6
22755 257/300 Write block 2792 len 32768 data: 3 3 4 3 3 4 5 6
22756 258/300 Read block 4935 len 16384
22757 259/300 Flush
22758 260/300 Read block 1306 len 20480
22759 261/300 Write block 2919 len 32768 data: 3 5 6 6 6 6 7 6
22760 262/300 Flush
22761 263/300 Read block 2698 len 28672
22762 264/300 Read block 4131 len 12288
22763 265/300 Write block 34 len 40960 data: 5 3 4 4 5 5 5 5 5 4
22764 266/300 Read block 2201 len 32768
22765 267/300 Write block 906 len 32768 data: 4 4 3 5 5 5 6 6
22766 268/300 Read block 1532 len 24576
22767 269/300 Write block 668 len 32768 data: 7 7 8 8 9 5 4 4
22768 270/300 Read block 3715 len 28672
22769 271/300 Write block 3706 len 36864 data: 9 9 8 8 7 5 5 4 5
22770 272/300 Read block 3464 len 16384
22771 273/300 Write block 1469 len 16384 data: 4 5 6 4
22772 274/300 Flush
22773 275/300 Flush
22774 276/300 Flush
22775 277/300 Flush
22776 278/300 Flush
22777 279/300 Write block 1114 len 32768 data: 6 7 6 5 3 3 3 3
22778 280/300 Read block 3096 len 8192
22779 281/300 Read block 9 len 24576
22780 282/300 Read block 4818 len 20480
22781 283/300 Read block 2418 len 36864
22782 284/300 Read block 2281 len 36864
22783 285/300 Read block 201 len 32768
22784 286/300 Flush
22785 287/300 Write block 4599 len 20480 data: 5 6 7 8 8
22786 288/300 Read block 2304 len 4096
22787 289/300 Read block 347 len 12288
22788 290/300 Read block 2853 len 28672
22789 291/300 Write block 3305 len 8192 data: 6 7
22790 292/300 Write block 4264 len 24576 data: 4 3 3 5 6 7
22791 293/300 Read block 4372 len 28672
22792 294/300 Write block 1346 len 28672 data: 5 6 5 5 5 5 4
22793 295/300 Read block 2685 len 8192
22794 296/300 Read block 2804 len 20480
22795 297/300 Write block 4303 len 4096 data: 5
22796 298/300 Write block 3646 len 32768 data: 3 4 3 6 6 6 7 9
22797 299/300 Write block 3903 len 4096 data: 4
22798 300/300 Write block 4506 len 40960 data: 4 4 4 5 5 4 5 6 7 8
22799 301/300 Read block 439 len 20480
22800 Main task has work to do, go find it
22801 got dsc Start(1)
22802 start 1
22803 Jan 23 17:42:33.161 INFO request completed, latency_us: 175, response_code: 204, uri: /start/cid/1, method: POST, req_id: 264dd681-38cd-426a-bb07-6e880a818248, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
22804 Replay: started 1, returned:Ok(())
22805 ----------------------------------------------------------------
22806 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22807 States: Active Offline Active
22808 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22809 7341 Acked Write 8 Done Sent Done false
22810 7342 Acked Read 2 Done Sent Done false
22811 7343 Acked Read 1 Done Sent Done false
22812 7344 Acked Write 9 Done Sent Done false
22813 7345 Acked Read 3 Done Sent Done false
22814 7346 Acked Write 10 Done Sent Done false
22815 7347 Acked Write 7 Done Sent Done false
22816 7348 Acked Write 8 Done Sent Done false
22817 7349 Acked Write 6 Done Sent Done false
22818 7350 Acked Read 7 Done Sent Done false
22819 7351 Acked Flush 0 Done Sent Done false
22820 7352 Acked Flush 0 Done Sent Done false
22821 7353 Acked Read 4 Done Sent Done false
22822 7354 Acked Write 4 Done Sent Done false
22823 7355 Acked Read 3 Done Sent Done false
22824 7356 Acked Write 9 Done Sent Done false
22825 7357 Acked Write 9 Done Sent Done false
22826 7358 Acked Read 1 Done Sent Done false
22827 7359 Acked Read 4 Done Sent Done false
22828 7360 Acked Flush 0 Done Sent Done false
22829 7361 Acked Read 6 Done Sent Done false
22830 7362 Acked Write 10 Done Sent Done false
22831 7363 Acked Read 6 Done Sent Done false
22832 7364 Acked Read 2 Done Sent Done false
22833 7365 Acked Write 1 Done Sent Done false
22834 7366 Acked Write 6 Done Sent Done false
22835 7367 Acked Read 2 Done Sent Done false
22836 7368 Acked Read 5 Done Sent Done false
22837 7369 Acked Read 6 Done Sent Done false
22838 7370 Acked Write 9 Done Sent Done false
22839 7371 Acked Write 5 Done Sent Done false
22840 7372 Acked Read 5 Done Sent Done false
22841 7373 Acked Write 4 Done Sent Done false
22842 7374 Acked Read 6 Done Sent Done false
22843 7375 Acked Write 2 Done Sent Done false
22844 7376 Acked Write 8 Done Sent Done false
22845 7377 Acked Read 3 Done Sent Done false
22846 7378 Acked Write 1 Done Sent Done false
22847 7379 Acked Read 8 Done Sent Done false
22848 7380 Acked Flush 0 Done Sent Done false
22849 7381 Acked Read 5 Done Sent Done false
22850 7382 Acked Read 9 Done Sent Done false
22851 7383 Acked Write 3 Done Sent Done false
22852 7384 Acked Write 6 Done Sent Done false
22853 7385 Acked Write 7 Done Sent Done false
22854 7386 Acked Write 10 Done Sent Done false
22855 7387 Acked Read 4 Done Sent Done false
22856 7388 Acked Read 1 Done Sent Done false
22857 7389 Acked Write 5 Done Sent Done false
22858 7390 Acked Write 5 Done Sent Done false
22859 7391 Acked Read 7 Done Sent Done false
22860 7392 Acked Write 2 Done Sent Done false
22861 7393 Acked Read 5 Done Sent Done false
22862 7394 Acked Read 1 Done Sent Done false
22863 7395 Acked Read 7 Done Sent Done false
22864 7396 Acked Read 10 Done Sent Done false
22865 7397 Acked Read 3 Done Sent Done false
22866 7398 Acked Read 4 Done Sent Done false
22867 7399 Acked Flush 0 Done Sent Done false
22868 7400 Acked Read 7 Done Sent Done false
22869 7401 Acked Write 7 Done Sent Done false
22870 7402 Acked Read 10 Done Sent Done false
22871 7403 Acked Read 2 Done Sent Done false
22872 7404 Acked Read 6 Done Sent Done false
22873 7405 Acked Flush 0 Done Sent Done false
22874 7406 Acked Read 1 Done Sent Done false
22875 7407 Acked Read 1 Done Sent Done false
22876 7408 Acked Read 5 Done Sent Done false
22877 7409 Acked Write 3 Done Sent Done false
22878 7410 Acked Flush 0 Done Sent Done false
22879 7411 Acked Read 10 Done Sent Done false
22880 7412 Acked Flush 0 Done Sent Done false
22881 7413 Acked Write 2 Done Sent Done false
22882 7414 Acked Read 5 Done Sent Done false
22883 7415 Acked Write 1 Done Sent Done false
22884 7416 Acked Read 1 Done Sent Done false
22885 7417 Acked Read 7 Done Sent Done false
22886 7418 Acked Read 9 Done Sent Done false
22887 7419 Acked Write 2 Done Sent Done false
22888 7420 Acked Write 1 Done Sent Done false
22889 7421 Acked Write 7 Done Sent Done false
22890 7422 Acked Write 10 Done Sent Done false
22891 7423 Acked Write 7 Done Sent Done false
22892 7424 Acked Write 6 Done Sent Done false
22893 7425 Acked Read 7 Done Sent Done false
22894 7426 Acked Write 3 Done Sent Done false
22895 7427 Acked Read 7 Done Sent Done false
22896 7428 Acked Read 5 Done Sent Done false
22897 7429 Acked Write 7 Done Sent Done false
22898 7430 Acked Flush 0 Done Sent Done false
22899 7431 Acked Flush 0 Done Sent Done false
22900 7432 Acked Write 3 Done Sent Done false
22901 7433 Acked Read 1 Done Sent Done false
22902 7434 Acked Write 10 Done Sent Done false
22903 7435 Acked Read 9 Done Sent Done false
22904 7436 Acked Read 5 Done Sent Done false
22905 7437 Acked Read 6 Done Sent Done false
22906 7438 Acked Flush 0 Done Sent Done false
22907 7439 Acked Read 4 Done Sent Done false
22908 7440 Acked Write 7 Done Sent Done false
22909 7441 Acked Read 4 Done Sent Done false
22910 7442 Acked Flush 0 Done Sent Done false
22911 7443 Acked Read 4 Done Sent Done false
22912 7444 Acked Flush 0 Done Sent Done false
22913 7445 Acked Write 10 Done Sent Done false
22914 7446 Acked Read 10 Done Sent Done false
22915 7447 Acked Write 6 Done Sent Done false
22916 7448 Acked Read 3 Done Sent Done false
22917 7449 Acked Read 8 Done Sent Done false
22918 7450 Acked Flush 0 Done Sent Done false
22919 7451 Acked Read 10 Done Sent Done false
22920 7452 Acked Write 10 Done Sent Done false
22921 7453 Acked Read 9 Done Sent Done false
22922 7454 Acked Write 3 Done Sent Done false
22923 7455 Acked Read 1 Done Sent Done false
22924 7456 Acked Read 10 Done Sent Done false
22925 7457 Acked Read 9 Done Sent Done false
22926 7458 Acked Write 8 Done Sent Done false
22927 7459 Acked Write 7 Done Sent Done false
22928 7460 Acked Read 5 Done Sent Done false
22929 7461 Acked Read 7 Done Sent Done false
22930 7462 Acked Read 8 Done Sent Done false
22931 7463 Acked Flush 0 Done Sent Done false
22932 7464 Acked Flush 0 Done Sent Done false
22933 7465 Acked Flush 0 Done Sent Done false
22934 7466 Acked Write 6 Done Sent Done false
22935 7467 Acked Read 9 Done Sent Done false
22936 7468 Acked Write 2 Done Sent Done false
22937 7469 Acked Write 7 Done Sent Done false
22938 7470 Acked Write 9 Done Sent Done false
22939 7471 Acked Read 3 Done Sent Done false
22940 7472 Acked Read 3 Done Sent Done false
22941 7473 Acked Write 9 Done Sent Done false
22942 7474 Acked Write 5 Done Sent Done false
22943 7475 Acked Read 10 Done Sent Done false
22944 7476 Acked Read 7 Done Sent Done false
22945 7477 Acked Read 10 Done Sent Done false
22946 7478 Acked Write 2 Done Sent Done false
22947 7479 Acked Read 8 Done Sent Done false
22948 7480 Acked Read 4 Done Sent Done false
22949 7481 Acked Write 7 Done Sent Done false
22950 7482 Acked Read 3 Done Sent Done false
22951 7483 Acked Flush 0 Done Sent Done false
22952 7484 Acked Read 7 Done Sent Done false
22953 7485 Acked Read 4 Done Sent Done false
22954 7486 Acked Write 5 Done Sent Done false
22955 7487 Acked Write 3 Done Sent Done false
22956 7488 Acked Read 6 Done Sent Done false
22957 7489 Acked Write 3 Done Sent Done false
22958 7490 Acked Flush 0 Done Sent Done false
22959 7491 Acked Flush 0 Done Sent Done false
22960 7492 Acked Write 3 Done Sent Done false
22961 7493 Acked Write 2 Done Sent Done false
22962 7494 Acked Read 6 Done Sent Done false
22963 7495 Acked Read 6 Done Sent Done false
22964 7496 Acked Write 1 Done Sent Done false
22965 7497 Acked Flush 0 Done Sent Done false
22966 7498 Acked Write 1 Done Sent Done false
22967 7499 Acked Write 4 Done Sent Done false
22968 7500 Acked Read 10 Done Sent Done false
22969 7501 Acked Write 7 Done Sent Done false
22970 7502 Acked Read 4 Done Sent Done false
22971 7503 Acked Read 10 Done Sent Done false
22972 7504 Acked Write 9 Done Sent Done false
22973 7505 Acked Read 3 Done Sent Done false
22974 7506 Acked Read 6 Done Sent Done false
22975 7507 Acked Read 5 Done Sent Done false
22976 7508 Acked Read 2 Done Sent Done false
22977 7509 Acked Write 5 Done Sent Done false
22978 7510 Acked Flush 0 Done Sent Done false
22979 7511 Acked Write 5 Done Sent Done false
22980 7512 Acked Write 6 Done Sent Done false
22981 7513 Acked Read 4 Done Sent Done false
22982 7514 Acked Flush 0 Done Sent Done false
22983 7515 Acked Read 8 Done Sent Done false
22984 7516 Acked Write 1 Done Sent Done false
22985 7517 Acked Write 10 Done Sent Done false
22986 7518 Acked Read 1 Done Sent Done false
22987 7519 Acked Read 9 Done Sent Done false
22988 7520 Acked Write 5 Done Sent Done false
22989 7521 Acked Read 6 Done Sent Done false
22990 7522 Acked Write 3 Done Sent Done false
22991 7523 Acked Write 1 Done Sent Done false
22992 7524 Acked Flush 0 Done Sent Done false
22993 7525 Acked Write 2 Done Sent Done false
22994 7526 Acked Write 1 Done Sent Done false
22995 7527 Acked Read 8 Done Sent Done false
22996 7528 Acked Read 9 Done Sent Done false
22997 7529 Acked Read 7 Done Sent Done false
22998 7530 Acked Write 8 Done Sent Done false
22999 7531 Acked Read 4 Done Sent Done false
23000 7532 Acked Read 2 Done Sent Done false
23001 7533 Acked Read 2 Done Sent Done false
23002 7534 Acked Flush 0 Done Sent Done false
23003 7535 Acked Write 8 Done Sent Done false
23004 7536 Acked Write 6 Done Sent Done false
23005 7537 Acked Read 2 Done Sent Done false
23006 7538 Acked Read 4 Done Sent Done false
23007 7539 Acked Flush 0 Done Sent Done false
23008 7540 Acked Write 7 Done Sent Done false
23009 7541 Acked Read 2 Done Sent Done false
23010 7542 Acked Read 5 Done Sent Done false
23011 7543 Acked Write 7 Done Sent Done false
23012 7544 Acked Write 5 Done Sent Done false
23013 7545 Acked Write 5 Done Sent Done false
23014 7546 Acked Write 9 Done Sent Done false
23015 7547 Acked Read 1 Done Sent Done false
23016 7548 Acked Read 6 Done Sent Done false
23017 7549 Acked Read 7 Done Sent Done false
23018 7550 Acked Write 7 Done Sent Done false
23019 7551 Acked Write 5 Done Sent Done false
23020 7552 Acked Read 5 Done Sent Done false
23021 7553 Acked Write 7 Done Sent Done false
23022 7554 Acked Read 6 Done Sent Done false
23023 7555 Acked Write 10 Done Sent Done false
23024 7556 Acked Write 10 Done Sent Done false
23025 7557 Acked Write 1 Done Sent Done false
23026 7558 Acked Read 3 Done Sent Done false
23027 7559 Acked Read 6 Done Sent Done false
23028 7560 Acked Write 9 Done Sent Done false
23029 7561 Acked Read 1 Done Sent Done false
23030 7562 Acked Read 6 Done Sent Done false
23031 7563 Acked Flush 0 Done Sent Done false
23032 7564 Acked Read 6 Done Sent Done false
23033 7565 Acked Write 6 Done Sent Done false
23034 7566 Acked Flush 0 Done Sent Done false
23035 7567 Acked Read 8 Done Sent Done false
23036 7568 Acked Write 9 Done Sent Done false
23037 7569 Acked Read 7 Done Sent Done false
23038 7570 Acked Flush 0 Done Sent Done false
23039 7571 Acked Read 8 Done Sent Done false
23040 7572 Acked Write 9 Done Sent Done false
23041 7573 Acked Write 6 Done Sent Done false
23042 7574 Acked Write 5 Done Sent Done false
23043 7575 Acked Flush 0 Done Sent Done false
23044 7576 Acked Flush 0 Done Sent Done false
23045 7577 Acked Read 3 Done Sent Done false
23046 7578 Acked Read 1 Done Sent Done false
23047 7579 Acked Write 10 Done Sent Done false
23048 7580 Acked Write 9 Done Sent Done false
23049 7581 Acked Read 7 Done Sent Done false
23050 7582 Acked Read 6 Done Sent Done false
23051 7583 Acked Read 2 Done Sent Done false
23052 7584 Acked Flush 0 Done Sent Done false
23053 7585 Acked Flush 0 Done Sent Done false
23054 7586 Acked Read 10 Done Sent Done false
23055 7587 Acked Write 4 Done Sent Done false
23056 7588 Acked Write 1 Done Sent Done false
23057 7589 Acked Flush 0 Done Sent Done false
23058 7590 Acked Read 9 Done Sent Done false
23059 7591 Acked Read 5 Done Sent Done false
23060 7592 Acked Read 7 Done Sent Done false
23061 7593 Acked Write 2 Done Sent Done false
23062 7594 Acked Flush 0 Done Sent Done false
23063 7595 Acked Write 8 Done Sent Done false
23064 7596 Acked Write 6 Done Sent Done false
23065 7597 Acked Write 8 Done Sent Done false
23066 7598 Acked Read 4 Done Sent Done false
23067 7599 Acked Flush 0 Done Sent Done false
23068 7600 Acked Read 5 Done Sent Done false
23069 7601 Acked Write 8 Done Sent Done false
23070 7602 Acked Flush 0 Done Sent Done false
23071 7603 Acked Read 7 Done Sent Done false
23072 7604 Acked Read 3 Done Sent Done false
23073 7605 Acked Write 10 Done Sent Done false
23074 7606 Acked Read 8 Done Sent Done false
23075 7607 Acked Write 8 Done Sent Done false
23076 7608 Acked Read 6 Done Sent Done false
23077 7609 Acked Write 8 Done Sent Done false
23078 7610 Acked Read 7 Done Sent Done false
23079 7611 Acked Write 9 Done Sent Done false
23080 7612 Acked Read 4 Done Sent Done false
23081 7613 Acked Write 4 Done Sent Done false
23082 7614 Acked Flush 0 Done Sent Done false
23083 7615 Acked Flush 0 Done Sent Done false
23084 7616 Acked Flush 0 Done Sent Done false
23085 7617 Acked Flush 0 Done Sent Done false
23086 7618 Acked Flush 0 Done Sent Done false
23087 7619 Acked Write 8 Done Sent Done false
23088 7620 Acked Read 2 Done Sent Done false
23089 7621 Acked Read 6 Done Sent Done false
23090 7622 Acked Read 5 Done Sent Done false
23091 7623 Acked Read 9 Done Sent Done false
23092 7624 Acked Read 9 Done Sent Done false
23093 7625 Acked Read 8 Done Sent Done false
23094 7626 Acked Flush 0 Done Sent Done false
23095 7627 Acked Write 5 Done Sent Done false
23096 7628 Acked Read 1 Done Sent Done false
23097 7629 Acked Read 3 Done Sent Done false
23098 7630 Acked Read 7 Done Sent Done false
23099 7631 Acked Write 2 Done Sent Done false
23100 7632 Acked Write 6 Done Sent Done false
23101 7633 Acked Read 7 Done Sent Done false
23102 7634 Acked Write 7 Done Sent Done false
23103 7635 Acked Read 2 Done Sent Done false
23104 7636 Acked Read 5 Done Sent Done false
23105 7637 Acked Write 1 Done Sent Done false
23106 7638 Acked Write 8 Done Sent Done false
23107 7639 Acked Write 1 Done Sent Done false
23108 7640 Acked Write 10 Done Sent Done false
23109 7641 Acked Read 5 Done Sent Done false
23110 STATES DS:0 DS:1 DS:2 TOTAL
23111 Sent 0 301 0 301
23112 Done 301 0 301 602
23113 Skipped 0 0 0 0
23114 Error 0 0 0 0
23115 Last Flush: Some(JobId(7626)) Some(JobId(7340)) Some(JobId(7626))
23116 Downstairs last five completed: 7340 7339 7338 7337 7336
23117 Upstairs last five completed: 7641 7640 7639 7638 7637
23118 CLIENT: Up:0 ds:301 act:2
23119 [8820] got start action
23120 [8820] I am going to restart
23121 Starting downstairs at port 8820
23122 Make output file at "/tmp/dsc/downstairs-8820.txt"
23123 [8820][1] reports Starting
23124 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1227)
23125 [8820][1] reports Running
23126 ----------------------------------------------------------------
23127 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
23128 States: Active Offline Active
23129 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23130 7341 Acked Write 8 Done Sent Done false
23131 7342 Acked Read 2 Done Sent Done false
23132 7343 Acked Read 1 Done Sent Done false
23133 7344 Acked Write 9 Done Sent Done false
23134 7345 Acked Read 3 Done Sent Done false
23135 7346 Acked Write 10 Done Sent Done false
23136 7347 Acked Write 7 Done Sent Done false
23137 7348 Acked Write 8 Done Sent Done false
23138 7349 Acked Write 6 Done Sent Done false
23139 7350 Acked Read 7 Done Sent Done false
23140 7351 Acked Flush 0 Done Sent Done false
23141 7352 Acked Flush 0 Done Sent Done false
23142 7353 Acked Read 4 Done Sent Done false
23143 7354 Acked Write 4 Done Sent Done false
23144 7355 Acked Read 3 Done Sent Done false
23145 7356 Acked Write 9 Done Sent Done false
23146 7357 Acked Write 9 Done Sent Done false
23147 7358 Acked Read 1 Done Sent Done false
23148 7359 Acked Read 4 Done Sent Done false
23149 7360 Acked Flush 0 Done Sent Done false
23150 7361 Acked Read 6 Done Sent Done false
23151 7362 Acked Write 10 Done Sent Done false
23152 7363 Acked Read 6 Done Sent Done false
23153 7364 Acked Read 2 Done Sent Done false
23154 7365 Acked Write 1 Done Sent Done false
23155 7366 Acked Write 6 Done Sent Done false
23156 7367 Acked Read 2 Done Sent Done false
23157 7368 Acked Read 5 Done Sent Done false
23158 7369 Acked Read 6 Done Sent Done false
23159 7370 Acked Write 9 Done Sent Done false
23160 7371 Acked Write 5 Done Sent Done false
23161 7372 Acked Read 5 Done Sent Done false
23162 7373 Acked Write 4 Done Sent Done false
23163 7374 Acked Read 6 Done Sent Done false
23164 7375 Acked Write 2 Done Sent Done false
23165 7376 Acked Write 8 Done Sent Done false
23166 7377 Acked Read 3 Done Sent Done false
23167 7378 Acked Write 1 Done Sent Done false
23168 7379 Acked Read 8 Done Sent Done false
23169 7380 Acked Flush 0 Done Sent Done false
23170 7381 Acked Read 5 Done Sent Done false
23171 7382 Acked Read 9 Done Sent Done false
23172 7383 Acked Write 3 Done Sent Done false
23173 7384 Acked Write 6 Done Sent Done false
23174 7385 Acked Write 7 Done Sent Done false
23175 7386 Acked Write 10 Done Sent Done false
23176 7387 Acked Read 4 Done Sent Done false
23177 7388 Acked Read 1 Done Sent Done false
23178 7389 Acked Write 5 Done Sent Done false
23179 7390 Acked Write 5 Done Sent Done false
23180 7391 Acked Read 7 Done Sent Done false
23181 7392 Acked Write 2 Done Sent Done false
23182 7393 Acked Read 5 Done Sent Done false
23183 7394 Acked Read 1 Done Sent Done false
23184 7395 Acked Read 7 Done Sent Done false
23185 7396 Acked Read 10 Done Sent Done false
23186 7397 Acked Read 3 Done Sent Done false
23187 7398 Acked Read 4 Done Sent Done false
23188 7399 Acked Flush 0 Done Sent Done false
23189 7400 Acked Read 7 Done Sent Done false
23190 7401 Acked Write 7 Done Sent Done false
23191 7402 Acked Read 10 Done Sent Done false
23192 7403 Acked Read 2 Done Sent Done false
23193 7404 Acked Read 6 Done Sent Done false
23194 7405 Acked Flush 0 Done Sent Done false
23195 7406 Acked Read 1 Done Sent Done false
23196 7407 Acked Read 1 Done Sent Done false
23197 7408 Acked Read 5 Done Sent Done false
23198 7409 Acked Write 3 Done Sent Done false
23199 7410 Acked Flush 0 Done Sent Done false
23200 7411 Acked Read 10 Done Sent Done false
23201 7412 Acked Flush 0 Done Sent Done false
23202 7413 Acked Write 2 Done Sent Done false
23203 7414 Acked Read 5 Done Sent Done false
23204 7415 Acked Write 1 Done Sent Done false
23205 7416 Acked Read 1 Done Sent Done false
23206 7417 Acked Read 7 Done Sent Done false
23207 7418 Acked Read 9 Done Sent Done false
23208 7419 Acked Write 2 Done Sent Done false
23209 7420 Acked Write 1 Done Sent Done false
23210 7421 Acked Write 7 Done Sent Done false
23211 7422 Acked Write 10 Done Sent Done false
23212 7423 Acked Write 7 Done Sent Done false
23213 7424 Acked Write 6 Done Sent Done false
23214 7425 Acked Read 7 Done Sent Done false
23215 7426 Acked Write 3 Done Sent Done false
23216 7427 Acked Read 7 Done Sent Done false
23217 7428 Acked Read 5 Done Sent Done false
23218 7429 Acked Write 7 Done Sent Done false
23219 7430 Acked Flush 0 Done Sent Done false
23220 7431 Acked Flush 0 Done Sent Done false
23221 7432 Acked Write 3 Done Sent Done false
23222 7433 Acked Read 1 Done Sent Done false
23223 7434 Acked Write 10 Done Sent Done false
23224 7435 Acked Read 9 Done Sent Done false
23225 7436 Acked Read 5 Done Sent Done false
23226 7437 Acked Read 6 Done Sent Done false
23227 7438 Acked Flush 0 Done Sent Done false
23228 7439 Acked Read 4 Done Sent Done false
23229 7440 Acked Write 7 Done Sent Done false
23230 7441 Acked Read 4 Done Sent Done false
23231 7442 Acked Flush 0 Done Sent Done false
23232 7443 Acked Read 4 Done Sent Done false
23233 7444 Acked Flush 0 Done Sent Done false
23234 7445 Acked Write 10 Done Sent Done false
23235 7446 Acked Read 10 Done Sent Done false
23236 7447 Acked Write 6 Done Sent Done false
23237 7448 Acked Read 3 Done Sent Done false
23238 7449 Acked Read 8 Done Sent Done false
23239 7450 Acked Flush 0 Done Sent Done false
23240 7451 Acked Read 10 Done Sent Done false
23241 7452 Acked Write 10 Done Sent Done false
23242 7453 Acked Read 9 Done Sent Done false
23243 7454 Acked Write 3 Done Sent Done false
23244 7455 Acked Read 1 Done Sent Done false
23245 7456 Acked Read 10 Done Sent Done false
23246 7457 Acked Read 9 Done Sent Done false
23247 7458 Acked Write 8 Done Sent Done false
23248 7459 Acked Write 7 Done Sent Done false
23249 7460 Acked Read 5 Done Sent Done false
23250 7461 Acked Read 7 Done Sent Done false
23251 7462 Acked Read 8 Done Sent Done false
23252 7463 Acked Flush 0 Done Sent Done false
23253 7464 Acked Flush 0 Done Sent Done false
23254 7465 Acked Flush 0 Done Sent Done false
23255 7466 Acked Write 6 Done Sent Done false
23256 7467 Acked Read 9 Done Sent Done false
23257 7468 Acked Write 2 Done Sent Done false
23258 7469 Acked Write 7 Done Sent Done false
23259 7470 Acked Write 9 Done Sent Done false
23260 7471 Acked Read 3 Done Sent Done false
23261 7472 Acked Read 3 Done Sent Done false
23262 7473 Acked Write 9 Done Sent Done false
23263 7474 Acked Write 5 Done Sent Done false
23264 7475 Acked Read 10 Done Sent Done false
23265 7476 Acked Read 7 Done Sent Done false
23266 7477 Acked Read 10 Done Sent Done false
23267 7478 Acked Write 2 Done Sent Done false
23268 7479 Acked Read 8 Done Sent Done false
23269 7480 Acked Read 4 Done Sent Done false
23270 7481 Acked Write 7 Done Sent Done false
23271 7482 Acked Read 3 Done Sent Done false
23272 7483 Acked Flush 0 Done Sent Done false
23273 7484 Acked Read 7 Done Sent Done false
23274 7485 Acked Read 4 Done Sent Done false
23275 7486 Acked Write 5 Done Sent Done false
23276 7487 Acked Write 3 Done Sent Done false
23277 7488 Acked Read 6 Done Sent Done false
23278 7489 Acked Write 3 Done Sent Done false
23279 7490 Acked Flush 0 Done Sent Done false
23280 7491 Acked Flush 0 Done Sent Done false
23281 7492 Acked Write 3 Done Sent Done false
23282 7493 Acked Write 2 Done Sent Done false
23283 7494 Acked Read 6 Done Sent Done false
23284 7495 Acked Read 6 Done Sent Done false
23285 7496 Acked Write 1 Done Sent Done false
23286 7497 Acked Flush 0 Done Sent Done false
23287 7498 Acked Write 1 Done Sent Done false
23288 7499 Acked Write 4 Done Sent Done false
23289 7500 Acked Read 10 Done Sent Done false
23290 7501 Acked Write 7 Done Sent Done false
23291 7502 Acked Read 4 Done Sent Done false
23292 7503 Acked Read 10 Done Sent Done false
23293 7504 Acked Write 9 Done Sent Done false
23294 7505 Acked Read 3 Done Sent Done false
23295 7506 Acked Read 6 Done Sent Done false
23296 7507 Acked Read 5 Done Sent Done false
23297 7508 Acked Read 2 Done Sent Done false
23298 7509 Acked Write 5 Done Sent Done false
23299 7510 Acked Flush 0 Done Sent Done false
23300 7511 Acked Write 5 Done Sent Done false
23301 7512 Acked Write 6 Done Sent Done false
23302 7513 Acked Read 4 Done Sent Done false
23303 7514 Acked Flush 0 Done Sent Done false
23304 7515 Acked Read 8 Done Sent Done false
23305 7516 Acked Write 1 Done Sent Done false
23306 7517 Acked Write 10 Done Sent Done false
23307 7518 Acked Read 1 Done Sent Done false
23308 7519 Acked Read 9 Done Sent Done false
23309 7520 Acked Write 5 Done Sent Done false
23310 7521 Acked Read 6 Done Sent Done false
23311 7522 Acked Write 3 Done Sent Done false
23312 7523 Acked Write 1 Done Sent Done false
23313 7524 Acked Flush 0 Done Sent Done false
23314 7525 Acked Write 2 Done Sent Done false
23315 7526 Acked Write 1 Done Sent Done false
23316 7527 Acked Read 8 Done Sent Done false
23317 7528 Acked Read 9 Done Sent Done false
23318 7529 Acked Read 7 Done Sent Done false
23319 7530 Acked Write 8 Done Sent Done false
23320 7531 Acked Read 4 Done Sent Done false
23321 7532 Acked Read 2 Done Sent Done false
23322 7533 Acked Read 2 Done Sent Done false
23323 7534 Acked Flush 0 Done Sent Done false
23324 7535 Acked Write 8 Done Sent Done false
23325 7536 Acked Write 6 Done Sent Done false
23326 7537 Acked Read 2 Done Sent Done false
23327 7538 Acked Read 4 Done Sent Done false
23328 7539 Acked Flush 0 Done Sent Done false
23329 7540 Acked Write 7 Done Sent Done false
23330 7541 Acked Read 2 Done Sent Done false
23331 7542 Acked Read 5 Done Sent Done false
23332 7543 Acked Write 7 Done Sent Done false
23333 7544 Acked Write 5 Done Sent Done false
23334 7545 Acked Write 5 Done Sent Done false
23335 7546 Acked Write 9 Done Sent Done false
23336 7547 Acked Read 1 Done Sent Done false
23337 7548 Acked Read 6 Done Sent Done false
23338 7549 Acked Read 7 Done Sent Done false
23339 7550 Acked Write 7 Done Sent Done false
23340 7551 Acked Write 5 Done Sent Done false
23341 7552 Acked Read 5 Done Sent Done false
23342 7553 Acked Write 7 Done Sent Done false
23343 7554 Acked Read 6 Done Sent Done false
23344 7555 Acked Write 10 Done Sent Done false
23345 7556 Acked Write 10 Done Sent Done false
23346 7557 Acked Write 1 Done Sent Done false
23347 7558 Acked Read 3 Done Sent Done false
23348 7559 Acked Read 6 Done Sent Done false
23349 7560 Acked Write 9 Done Sent Done false
23350 7561 Acked Read 1 Done Sent Done false
23351 7562 Acked Read 6 Done Sent Done false
23352 7563 Acked Flush 0 Done Sent Done false
23353 7564 Acked Read 6 Done Sent Done false
23354 7565 Acked Write 6 Done Sent Done false
23355 7566 Acked Flush 0 Done Sent Done false
23356 7567 Acked Read 8 Done Sent Done false
23357 7568 Acked Write 9 Done Sent Done false
23358 7569 Acked Read 7 Done Sent Done false
23359 7570 Acked Flush 0 Done Sent Done false
23360 7571 Acked Read 8 Done Sent Done false
23361 7572 Acked Write 9 Done Sent Done false
23362 7573 Acked Write 6 Done Sent Done false
23363 7574 Acked Write 5 Done Sent Done false
23364 7575 Acked Flush 0 Done Sent Done false
23365 7576 Acked Flush 0 Done Sent Done false
23366 7577 Acked Read 3 Done Sent Done false
23367 7578 Acked Read 1 Done Sent Done false
23368 7579 Acked Write 10 Done Sent Done false
23369 7580 Acked Write 9 Done Sent Done false
23370 7581 Acked Read 7 Done Sent Done false
23371 7582 Acked Read 6 Done Sent Done false
23372 7583 Acked Read 2 Done Sent Done false
23373 7584 Acked Flush 0 Done Sent Done false
23374 7585 Acked Flush 0 Done Sent Done false
23375 7586 Acked Read 10 Done Sent Done false
23376 7587 Acked Write 4 Done Sent Done false
23377 7588 Acked Write 1 Done Sent Done false
23378 7589 Acked Flush 0 Done Sent Done false
23379 7590 Acked Read 9 Done Sent Done false
23380 7591 Acked Read 5 Done Sent Done false
23381 7592 Acked Read 7 Done Sent Done false
23382 7593 Acked Write 2 Done Sent Done false
23383 7594 Acked Flush 0 Done Sent Done false
23384 7595 Acked Write 8 Done Sent Done false
23385 7596 Acked Write 6 Done Sent Done false
23386 7597 Acked Write 8 Done Sent Done false
23387 7598 Acked Read 4 Done Sent Done false
23388 7599 Acked Flush 0 Done Sent Done false
23389 7600 Acked Read 5 Done Sent Done false
23390 7601 Acked Write 8 Done Sent Done false
23391 7602 Acked Flush 0 Done Sent Done false
23392 7603 Acked Read 7 Done Sent Done false
23393 7604 Acked Read 3 Done Sent Done false
23394 7605 Acked Write 10 Done Sent Done false
23395 7606 Acked Read 8 Done Sent Done false
23396 7607 Acked Write 8 Done Sent Done false
23397 7608 Acked Read 6 Done Sent Done false
23398 7609 Acked Write 8 Done Sent Done false
23399 7610 Acked Read 7 Done Sent Done false
23400 7611 Acked Write 9 Done Sent Done false
23401 7612 Acked Read 4 Done Sent Done false
23402 7613 Acked Write 4 Done Sent Done false
23403 7614 Acked Flush 0 Done Sent Done false
23404 7615 Acked Flush 0 Done Sent Done false
23405 7616 Acked Flush 0 Done Sent Done false
23406 7617 Acked Flush 0 Done Sent Done false
23407 7618 Acked Flush 0 Done Sent Done false
23408 7619 Acked Write 8 Done Sent Done false
23409 7620 Acked Read 2 Done Sent Done false
23410 7621 Acked Read 6 Done Sent Done false
23411 7622 Acked Read 5 Done Sent Done false
23412 7623 Acked Read 9 Done Sent Done false
23413 7624 Acked Read 9 Done Sent Done false
23414 7625 Acked Read 8 Done Sent Done false
23415 7626 Acked Flush 0 Done Sent Done false
23416 7627 Acked Write 5 Done Sent Done false
23417 7628 Acked Read 1 Done Sent Done false
23418 7629 Acked Read 3 Done Sent Done false
23419 7630 Acked Read 7 Done Sent Done false
23420 7631 Acked Write 2 Done Sent Done false
23421 7632 Acked Write 6 Done Sent Done false
23422 7633 Acked Read 7 Done Sent Done false
23423 7634 Acked Write 7 Done Sent Done false
23424 7635 Acked Read 2 Done Sent Done false
23425 7636 Acked Read 5 Done Sent Done false
23426 7637 Acked Write 1 Done Sent Done false
23427 7638 Acked Write 8 Done Sent Done false
23428 7639 Acked Write 1 Done Sent Done false
23429 7640 Acked Write 10 Done Sent Done false
23430 7641 Acked Read 5 Done Sent Done false
23431 7642 Acked Flush 0 Done Sent Done false
23432 STATES DS:0 DS:1 DS:2 TOTAL
23433 Sent 0 302 0 302
23434 Done 302 0 302 604
23435 Skipped 0 0 0 0
23436 Error 0 0 0 0
23437 Last Flush: Some(JobId(7642)) Some(JobId(7340)) Some(JobId(7642))
23438 Downstairs last five completed: 7340 7339 7338 7337 7336
23439 Upstairs last five completed: 7642 7641 7640 7639 7638
23440 CLIENT: Up:0 ds:302 act:2
234412025-01-23T17:42:38.923ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
234422025-01-23T17:42:38.923ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37590) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
234432025-01-23T17:42:38.924ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
234442025-01-23T17:42:38.925ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
234452025-01-23T17:42:38.925ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
234462025-01-23T17:42:38.925ZINFOcrucible: send last flush ID to this DS: Some(JobId(7340)) = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
234472025-01-23T17:42:38.925ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
234482025-01-23T17:42:38.925ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(7340)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
234492025-01-23T17:42:38.925ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(7340)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
23450 ----------------------------------------------------------------
23451 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23452 Downstairs last five completed: 7642 7641 7640 7639 7638
23453 Upstairs last five completed: 7642 7641 7640 7639 7638
23454 CLIENT: Up:0 ds:0 act:3
23455 Replay: All jobs finished
23456 Main task has work to do, go find it
23457 got dsc Stop(1)
23458 stop 1
23459 [8820] Got stop action so:false kr:false
23460 Jan 23 17:42:41.166 INFO request completed, latency_us: 342, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 22dc386e-ced3-4eac-95ab-741ba01afe81, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
234612025-01-23T17:42:41.167ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
234622025-01-23T17:42:41.167ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
23463 Jan 23 17:42:41.167 INFO request completed, latency_us: 143, response_code:{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:42:41.167610419Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
23464 200, uri: /state/cid/1, method: GET, req_id: 73055bd0-12e5-4172-b202-a02f494e7ce1, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
234652025-01-23T17:42:41.167ZINFOcrucible: client task is exiting = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
23466 [8820] Exited with: ExitStatus(unix_wait_status(9))
23467 [8820][1] reports Exit
23468 Jan 23 17:42:45.170 INFO request completed, latency_us: 212, response_code: 200, uri: /state/cid/1, method: GET, req_id: 08783357-d8d6-422e-91f7-5561b33fe8bc, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
23469 001/300 Write block 4584 len 12288 data: 5 6 5
23470 002/300 Write block 2056 len 36864 data: 6 6 6 6 6 7 8 7 6
23471 003/300 Write block 2275 len 8192 data: 4 4
23472 004/300 Read block 384 len 28672
23473 005/300 Write block 2477 len 32768 data: 7 7 8 6 6 7 7 7
23474 006/300 Read block 4986 len 32768
23475 007/300 Read block 3751 len 40960
23476 008/300 Write block 516 len 4096 data: 3
23477 009/300 Read block 4769 len 16384
23478 010/300 Read block 1190 len 24576
23479 011/300 Read block 2482 len 24576
23480 012/300 Read block 2078 len 32768
23481 013/300 Read block 4878 len 8192
23482 014/300 Write block 2697 len 16384 data: 5 5 6 5
23483 015/300 Read block 2732 len 32768
23484 016/300 Write block 17 len 32768 data: 4 4 4 4 3 4 4 3
23485 017/300 Read block 4757 len 24576
23486 018/300 Flush
23487 019/300 Read block 4184 len 12288
23488 020/300 Write block 2 len 40960 data: 2 4 5 5 4 4 5 5 5 5
23489 021/300 Read block 1354 len 8192
23490 022/300 Read block 4165 len 24576
23491 023/300 Read block 1660 len 8192
23492 024/300 Read block 2770 len 20480
23493 025/300 Read block 177 len 40960
23494 026/300 Write block 3823 len 20480 data: 4 3 4 4 5
23495 027/300 Read block 2537 len 40960
23496 028/300 Write block 3869 len 28672 data: 4 3 3 4 5 6 6
23497 029/300 Read block 2696 len 24576
23498 030/300 Write block 3254 len 28672 data: 4 6 4 4 4 4 3
23499 031/300 Read block 327 len 20480
23500 032/300 Flush
23501 033/300 Read block 2187 len 32768
23502 034/300 Write block 1614 len 20480 data: 7 8 10 9 10
23503 035/300 Read block 1785 len 24576
23504 036/300 Read block 463 len 40960
23505 037/300 Write block 2926 len 12288 data: 7 6 6
23506 038/300 Read block 2794 len 4096
23507 039/300 Write block 764 len 20480 data: 6 6 5 3 4
23508 040/300 Write block 4471 len 36864 data: 4 4 3 3 4 4 4 3 3
23509 041/300 Write block 909 len 24576 data: 6 6 6 7 7 4
23510 042/300 Read block 2343 len 24576
23511 043/300 Read block 4016 len 28672
23512 044/300 Read block 2879 len 20480
23513 045/300 Flush
23514 046/300 Read block 4491 len 36864
23515 047/300 Read block 2154 len 4096
23516 048/300 Write block 2123 len 4096 data: 5
23517 049/300 Write block 594 len 8192 data: 3 5
23518 050/300 Write block 3291 len 36864 data: 4 6 6 4 4 3 3 5 4
23519 051/300 Write block 3111 len 36864 data: 9 7 6 6 7 7 7 6 5
23520 052/300 Write block 3893 len 16384 data: 6 6 5 4
23521 053/300 Write block 1428 len 24576 data: 3 2 2 3 3 3
23522 054/300 Write block 420 len 16384 data: 2 2 2 2
23523 055/300 Flush
23524 056/300 Flush
23525 057/300 Write block 1384 len 32768 data: 3 3 3 3 3 4 4 5
23526 058/300 Write block 2860 len 36864 data: 5 6 6 6 6 6 8 9 8
23527 059/300 Read block 2045 len 36864
23528 060/300 Read block 1950 len 32768
23529 061/300 Read block 1849 len 16384
23530 062/300 Read block 1664 len 4096
23531 063/300 Read block 3338 len 28672
23532 064/300 Read block 1505 len 40960
23533 065/300 Read block 2676 len 28672
23534 066/300 Read block 4866 len 28672
23535 067/300 Write block 1870 len 36864 data: 6 5 5 6 8 7 6 6 6
23536 068/300 Read block 3714 len 36864
23537 069/300 Read block 2536 len 40960
23538 070/300 Read block 4172 len 20480
23539 071/300 Write block 4498 len 12288 data: 7 7 6
23540 072/300 Read block 379 len 24576
23541 073/300 Write block 4514 len 36864 data: 8 9 7 6 6 6 7 5 5
23542 074/300 Write block 3958 len 12288 data: 5 5 6
23543 075/300 Read block 2503 len 4096
23544 076/300 Read block 4518 len 40960
23545 077/300 Write block 1810 len 28672 data: 2 3 3 3 3 5 5
23546 078/300 Flush
23547 079/300 Read block 4421 len 4096
23548 080/300 Write block 2720 len 8192 data: 5 5
23549 081/300 Read block 4566 len 20480
23550 082/300 Read block 2657 len 40960
23551 083/300 Read block 53 len 20480
23552 084/300 Read block 443 len 28672
23553 085/300 Write block 4033 len 28672 data: 7 7 7 6 5 5 4
23554 086/300 Read block 3818 len 40960
23555 087/300 Write block 1177 len 24576 data: 5 5 5 4 4 4
23556 088/300 Write block 3884 len 20480 data: 7 7 5 5 5
23557 089/300 Read block 4106 len 20480
23558 090/300 Write block 1923 len 36864 data: 10 12 10 8 6 4 3 3 2
23559 091/300 Read block 1672 len 20480
23560 092/300 Write block 2423 len 20480 data: 4 4 4 4 3
23561 093/300 Write block 3381 len 8192 data: 4 2
23562 094/300 Flush
23563 095/300 Write block 2332 len 40960 data: 4 5 4 4 4 4 4 3 4 4
23564 096/300 Read block 318 len 12288
23565 097/300 Write block 1557 len 24576 data: 4 4 5 6 5 5
23566 098/300 Read block 732 len 4096
23567 099/300 Write block 1003 len 28672 data: 3 5 7 7 7 6 6
23568 100/300 Read block 1768 len 32768
23569 101/300 Write block 4917 len 4096 data: 4
23570 102/300 Write block 2894 len 16384 data: 5 5 2 3
23571 103/300 Write block 4541 len 36864 data: 5 4 4 5 6 5 4 4 4
23572 104/300 Write block 3288 len 28672 data: 3 3 4 5 7 7 5
23573 105/300 Read block 1640 len 36864
23574 106/300 Write block 3666 len 32768 data: 7 8 9 9 7 7 7 7
23575 107/300 Read block 1142 len 24576
23576 108/300 Read block 4145 len 28672
23577 109/300 Read block 115 len 32768
23578 110/300 Read block 1253 len 8192
23579 111/300 Read block 1554 len 12288
23580 112/300 Write block 2923 len 40960 data: 7 7 8 8 7 7 3 3 4 4
23581 113/300 Write block 3759 len 8192 data: 4 4
23582 114/300 Read block 4880 len 24576
23583 115/300 Write block 1904 len 24576 data: 7 8 7 8 11 9
23584 116/300 Flush
23585 117/300 Write block 1961 len 40960 data: 6 5 5 6 8 7 6 5 6 7
23586 118/300 Read block 1113 len 24576
23587 119/300 Flush
23588 120/300 Write block 4715 len 32768 data: 3 4 3 3 3 3 4 3
23589 121/300 Read block 4936 len 40960
23590 122/300 Write block 3240 len 28672 data: 4 4 5 4 3 5 4
23591 123/300 Write block 3244 len 4096 data: 4
23592 124/300 Write block 792 len 40960 data: 4 4 3 3 4 6 5 5 4 4
23593 125/300 Read block 2822 len 40960
23594 126/300 Read block 3946 len 16384
23595 127/300 Read block 965 len 28672
23596 128/300 Read block 443 len 28672
23597 129/300 Write block 1066 len 36864 data: 6 5 6 6 6 7 7 6 6
23598 130/300 Write block 328 len 8192 data: 7 8
23599 131/300 Write block 936 len 28672 data: 5 6 5 4 5 4 4
23600 132/300 Read block 2792 len 40960
23601 133/300 Read block 3199 len 20480
23602 134/300 Read block 171 len 12288
23603 135/300 Read block 4846 len 12288
23604 136/300 Flush
23605 137/300 Read block 609 len 40960
23606 138/300 Write block 2982 len 32768 data: 3 3 3 3 4 5 6 6
23607 139/300 Read block 3251 len 12288
23608 140/300 Flush
23609 141/300 Read block 4500 len 8192
23610 142/300 Read block 3146 len 20480
23611 143/300 Read block 3907 len 12288
23612 144/300 Flush
23613 145/300 Flush
23614 146/300 Read block 3169 len 32768
23615 147/300 Read block 3455 len 40960
23616 148/300 Read block 2613 len 24576
23617 149/300 Write block 2930 len 8192 data: 4 5
23618 150/300 Write block 2327 len 4096 data: 6
23619 151/300 Write block 1844 len 36864 data: 3 4 3 3 4 5 4 4 4
23620 152/300 Write block 3634 len 32768 data: 2 3 4 4 4 3 3 3
23621 153/300 Write block 984 len 32768 data: 8 7 8 7 7 9 9 8
23622 154/300 Write block 4773 len 36864 data: 4 4 4 4 6 6 7 6 5
23623 155/300 Read block 3823 len 12288
23624 156/300 Write block 422 len 36864 data: 3 3 2 2 3 3 2 3 4
23625 157/300 Read block 601 len 8192
23626 158/300 Read block 2648 len 8192
23627 159/300 Write block 3803 len 36864 data: 5 4 6 7 6 6 5 5 7
23628 160/300 Read block 2279 len 24576
23629 161/300 Read block 3410 len 36864
23630 162/300 Write block 3576 len 36864 data: 4 4 6 7 7 8 8 7 7
23631 163/300 Read block 849 len 28672
23632 164/300 Read block 1226 len 32768
23633 165/300 Read block 3683 len 36864
23634 166/300 Read block 4694 len 16384
23635 167/300 Read block 939 len 40960
23636 168/300 Read block 2072 len 20480
23637 169/300 Write block 3662 len 40960 data: 6 7 6 8 8 9 10 10 8 8
23638 170/300 Write block 2987 len 4096 data: 6
23639 171/300 Read block 3467 len 36864
23640 172/300 Read block 3998 len 16384
23641 173/300 Write block 1767 len 40960 data: 5 5 6 6 5 5 5 5 5 6
23642 174/300 Write block 4265 len 28672 data: 4 4 6 7 8 8 8
23643 175/300 Read block 4262 len 28672
23644 176/300 Read block 3610 len 20480
23645 177/300 Write block 696 len 12288 data: 4 5 5
23646 178/300 Write block 3275 len 4096 data: 4
23647 179/300 Read block 179 len 24576
23648 180/300 Write block 2215 len 32768 data: 3 6 6 6 6 5 4 8
23649 181/300 Write block 3000 len 12288 data: 4 4 3
23650 182/300 Read block 4492 len 12288
23651 183/300 Read block 2786 len 16384
23652 184/300 Write block 409 len 32768 data: 2 2 2 2 4 4 4 5
23653 185/300 Read block 1847 len 40960
23654 186/300 Read block 1177 len 12288
23655 187/300 Read block 907 len 8192
23656 188/300 Read block 4154 len 8192
23657 189/300 Read block 844 len 40960
23658 190/300 Flush
23659 191/300 Write block 3409 len 40960 data: 6 7 6 5 5 6 6 7 6 6
23660 192/300 Read block 4718 len 4096
23661 193/300 Read block 889 len 36864
23662 194/300 Read block 509 len 4096
23663 195/300 Read block 2610 len 8192
23664 196/300 Read block 4106 len 36864
23665 197/300 Read block 3873 len 16384
23666 198/300 Write block 2429 len 36864 data: 3 2 2 3 4 3 3 3 2
23667 199/300 Read block 4297 len 32768
23668 200/300 Write block 1312 len 32768 data: 7 7 6 6 7 6 5 5
23669 201/300 Write block 4751 len 20480 data: 3 3 3 3 3
23670 202/300 Read block 589 len 36864
23671 203/300 Read block 2402 len 4096
23672 204/300 Read block 148 len 24576
23673 205/300 Write block 3343 len 40960 data: 3 4 4 6 5 5 4 3 3 5
23674 206/300 Write block 2434 len 40960 data: 4 4 4 3 2 2 2 4 4 5
23675 207/300 Read block 4937 len 32768
23676 208/300 Read block 3697 len 8192
23677 209/300 Write block 1905 len 32768 data: 9 8 9 12 10 7 7 6
23678 210/300 Read block 1025 len 24576
23679 211/300 Read block 3056 len 16384
23680 212/300 Read block 2981 len 16384
23681 213/300 Write block 3739 len 12288 data: 4 4 4
23682 214/300 Write block 3810 len 16384 data: 6 8 7 6
23683 215/300 Write block 3043 len 24576 data: 5 7 5 5 5 6
23684 216/300 Read block 4557 len 28672
23685 217/300 Read block 4038 len 28672
23686 218/300 Read block 2699 len 16384
23687 219/300 Flush
23688 220/300 Read block 4247 len 8192
23689 221/300 Write block 2083 len 12288 data: 5 4 3
23690 222/300 Write block 2563 len 4096 data: 3
23691 223/300 Read block 3388 len 20480
23692 224/300 Flush
23693 225/300 Flush
23694 226/300 Read block 4089 len 36864
23695 227/300 Read block 34 len 32768
23696 228/300 Read block 1295 len 16384
23697 229/300 Read block 2906 len 20480
23698 230/300 Write block 628 len 36864 data: 4 4 5 4 5 5 5 6 6
23699 231/300 Write block 1426 len 24576 data: 6 6 4 3 3 4
23700 232/300 Read block 3715 len 16384
23701 233/300 Write block 2042 len 16384 data: 3 4 3 3
23702 234/300 Write block 555 len 40960 data: 6 8 7 5 3 2 2 2 2 3
23703 235/300 Write block 794 len 24576 data: 4 4 5 7 6 6
23704 236/300 Read block 1953 len 4096
23705 237/300 Write block 2225 len 32768 data: 6 7 6 8 8 9 7 7
23706 238/300 Read block 2110 len 20480
23707 239/300 Flush
23708 240/300 Read block 439 len 8192
23709 241/300 Write block 2794 len 28672 data: 5 4 4 5 6 7 7
23710 242/300 Write block 2341 len 4096 data: 5
23711 243/300 Write block 2860 len 36864 data: 6 7 7 7 7 7 9 10 9
23712 244/300 Read block 1727 len 20480
23713 245/300 Read block 4095 len 32768
23714 246/300 Write block 3836 len 24576 data: 7 5 6 6 6 6
23715 247/300 Write block 595 len 28672 data: 6 4 3 3 4 5 5
23716 248/300 Write block 4710 len 32768 data: 6 4 4 5 4 4 5 4
23717 249/300 Write block 4353 len 32768 data: 4 3 3 3 3 2 2 2
23718 250/300 Read block 4537 len 32768
23719 251/300 Read block 1731 len 20480
23720 252/300 Read block 3219 len 4096
23721 253/300 Read block 1927 len 32768
23722 254/300 Read block 1365 len 36864
23723 255/300 Write block 4107 len 40960 data: 5 4 3 3 3 4 6 5 5 5
23724 256/300 Write block 4304 len 20480 data: 5 5 3 2 2
23725 257/300 Read block 2894 len 28672
23726 258/300 Flush
23727 259/300 Read block 3662 len 8192
23728 260/300 Read block 392 len 12288
23729 261/300 Read block 4810 len 40960
23730 262/300 Read block 2845 len 16384
23731 263/300 Write block 4589 len 16384 data: 5 5 6 5
23732 264/300 Read block 1134 len 24576
23733 265/300 Write block 2594 len 16384 data: 7 8 7 6
23734 266/300 Read block 3806 len 4096
23735 267/300 Read block 2967 len 16384
23736 268/300 Write block 3131 len 40960 data: 4 5 6 6 6 6 5 5 5 6
23737 269/300 Write block 2049 len 28672 data: 4 5 5 5 5 6 6
23738 270/300 Read block 2432 len 8192
23739 271/300 Write block 1794 len 20480 data: 6 6 7 6 6
23740 272/300 Read block 1118 len 32768
23741 273/300 Read block 2029 len 28672
23742 274/300 Write block 1823 len 12288 data: 5 5 7
23743 275/300 Read block 3495 len 12288
23744 276/300 Flush
23745 277/300 Read block 443 len 20480
23746 278/300 Write block 3794 len 20480 data: 3 3 3 3 4
23747 279/300 Read block 1498 len 32768
23748 280/300 Write block 430 len 4096 data: 5
23749 281/300 Flush
23750 282/300 Write block 4396 len 28672 data: 5 5 6 6 6 5 4
23751 283/300 Write block 141 len 32768 data: 7 6 6 3 3 4 4 3
23752 284/300 Read block 3330 len 20480
23753 285/300 Read block 2376 len 32768
23754 286/300 Write block 212 len 20480 data: 8 8 8 7 7
23755 287/300 Write block 1753 len 16384 data: 5 4 5 6
23756 288/300 Write block 624 len 16384 data: 2 2 2 3
23757 289/300 Write block 4211 len 32768 data: 4 5 5 5 5 5 5 3
23758 290/300 Write block 4804 len 12288 data: 4 3 2
23759 291/300 Write block 570 len 20480 data: 9 10 7 6 4
23760 292/300 Read block 127 len 4096
23761 293/300 Read block 4643 len 8192
23762 294/300 Flush
23763 295/300 Write block 167 len 16384 data: 6 6 6 8
23764 296/300 Write block 1628 len 28672 data: 6 4 4 4 3 3 3
23765 297/300 Write block 2727 len 28672 data: 7 7 8 8 8 6 6
23766 298/300 Read block 2853 len 40960
23767 299/300 Flush
23768 300/300 Read block 2631 len 28672
23769 301/300 Write block 4008 len 28672 data: 4 4 5 5 4 3 4
23770 Main task has work to do, go find it
23771 got dsc Start(1)
23772 start 1
23773 Jan 23 17:42:45.423 INFO request completed, latency_us: 243, response_code: 204, uri: /start/cid/1, method: POST, req_id: d7d7de85-d4c3-461b-b837-488fb6265f1d, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
23774 Replay: started 1, returned:Ok(())
23775 ----------------------------------------------------------------
23776 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23777 States: Active Offline Active
23778 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23779 7643 Acked Write 3 Done Sent Done false
23780 7644 Acked Write 9 Done Sent Done false
23781 7645 Acked Write 2 Done Sent Done false
23782 7646 Acked Read 7 Done Sent Done false
23783 7647 Acked Write 8 Done Sent Done false
23784 7648 Acked Read 8 Done Sent Done false
23785 7649 Acked Read 10 Done Sent Done false
23786 7650 Acked Write 1 Done Sent Done false
23787 7651 Acked Read 4 Done Sent Done false
23788 7652 Acked Read 6 Done Sent Done false
23789 7653 Acked Read 6 Done Sent Done false
23790 7654 Acked Read 8 Done Sent Done false
23791 7655 Acked Read 2 Done Sent Done false
23792 7656 Acked Write 4 Done Sent Done false
23793 7657 Acked Read 8 Done Sent Done false
23794 7658 Acked Write 8 Done Sent Done false
23795 7659 Acked Read 6 Done Sent Done false
23796 7660 Acked Flush 0 Done Sent Done false
23797 7661 Acked Read 3 Done Sent Done false
23798 7662 Acked Write 10 Done Sent Done false
23799 7663 Acked Read 2 Done Sent Done false
23800 7664 Acked Read 6 Done Sent Done false
23801 7665 Acked Read 2 Done Sent Done false
23802 7666 Acked Read 5 Done Sent Done false
23803 7667 Acked Read 10 Done Sent Done false
23804 7668 Acked Write 5 Done Sent Done false
23805 7669 Acked Read 10 Done Sent Done false
23806 7670 Acked Write 7 Done Sent Done false
23807 7671 Acked Read 6 Done Sent Done false
23808 7672 Acked Write 7 Done Sent Done false
23809 7673 Acked Read 5 Done Sent Done false
23810 7674 Acked Flush 0 Done Sent Done false
23811 7675 Acked Read 8 Done Sent Done false
23812 7676 Acked Write 5 Done Sent Done false
23813 7677 Acked Read 6 Done Sent Done false
23814 7678 Acked Read 10 Done Sent Done false
23815 7679 Acked Write 3 Done Sent Done false
23816 7680 Acked Read 1 Done Sent Done false
23817 7681 Acked Write 5 Done Sent Done false
23818 7682 Acked Write 9 Done Sent Done false
23819 7683 Acked Write 6 Done Sent Done false
23820 7684 Acked Read 6 Done Sent Done false
23821 7685 Acked Read 7 Done Sent Done false
23822 7686 Acked Read 5 Done Sent Done false
23823 7687 Acked Flush 0 Done Sent Done false
23824 7688 Acked Read 9 Done Sent Done false
23825 7689 Acked Read 1 Done Sent Done false
23826 7690 Acked Write 1 Done Sent Done false
23827 7691 Acked Write 2 Done Sent Done false
23828 7692 Acked Write 9 Done Sent Done false
23829 7693 Acked Write 9 Done Sent Done false
23830 7694 Acked Write 4 Done Sent Done false
23831 7695 Acked Write 6 Done Sent Done false
23832 7696 Acked Write 4 Done Sent Done false
23833 7697 Acked Flush 0 Done Sent Done false
23834 7698 Acked Flush 0 Done Sent Done false
23835 7699 Acked Write 8 Done Sent Done false
23836 7700 Acked Write 9 Done Sent Done false
23837 7701 Acked Read 9 Done Sent Done false
23838 7702 Acked Read 8 Done Sent Done false
23839 7703 Acked Read 4 Done Sent Done false
23840 7704 Acked Read 1 Done Sent Done false
23841 7705 Acked Read 7 Done Sent Done false
23842 7706 Acked Read 10 Done Sent Done false
23843 7707 Acked Read 7 Done Sent Done false
23844 7708 Acked Read 7 Done Sent Done false
23845 7709 Acked Write 9 Done Sent Done false
23846 7710 Acked Read 9 Done Sent Done false
23847 7711 Acked Read 10 Done Sent Done false
23848 7712 Acked Read 5 Done Sent Done false
23849 7713 Acked Write 3 Done Sent Done false
23850 7714 Acked Read 6 Done Sent Done false
23851 7715 Acked Write 9 Done Sent Done false
23852 7716 Acked Write 3 Done Sent Done false
23853 7717 Acked Read 1 Done Sent Done false
23854 7718 Acked Read 10 Done Sent Done false
23855 7719 Acked Write 7 Done Sent Done false
23856 7720 Acked Flush 0 Done Sent Done false
23857 7721 Acked Read 1 Done Sent Done false
23858 7722 Acked Write 2 Done Sent Done false
23859 7723 Acked Read 5 Done Sent Done false
23860 7724 Acked Read 10 Done Sent Done false
23861 7725 Acked Read 5 Done Sent Done false
23862 7726 Acked Read 7 Done Sent Done false
23863 7727 Acked Write 7 Done Sent Done false
23864 7728 Acked Read 10 Done Sent Done false
23865 7729 Acked Write 6 Done Sent Done false
23866 7730 Acked Write 5 Done Sent Done false
23867 7731 Acked Read 5 Done Sent Done false
23868 7732 Acked Write 9 Done Sent Done false
23869 7733 Acked Read 5 Done Sent Done false
23870 7734 Acked Write 5 Done Sent Done false
23871 7735 Acked Write 2 Done Sent Done false
23872 7736 Acked Flush 0 Done Sent Done false
23873 7737 Acked Write 10 Done Sent Done false
23874 7738 Acked Read 3 Done Sent Done false
23875 7739 Acked Write 6 Done Sent Done false
23876 7740 Acked Read 1 Done Sent Done false
23877 7741 Acked Write 7 Done Sent Done false
23878 7742 Acked Read 8 Done Sent Done false
23879 7743 Acked Write 1 Done Sent Done false
23880 7744 Acked Write 4 Done Sent Done false
23881 7745 Acked Write 9 Done Sent Done false
23882 7746 Acked Write 7 Done Sent Done false
23883 7747 Acked Read 9 Done Sent Done false
23884 7748 Acked Write 8 Done Sent Done false
23885 7749 Acked Read 6 Done Sent Done false
23886 7750 Acked Read 7 Done Sent Done false
23887 7751 Acked Read 8 Done Sent Done false
23888 7752 Acked Read 2 Done Sent Done false
23889 7753 Acked Read 3 Done Sent Done false
23890 7754 Acked Write 10 Done Sent Done false
23891 7755 Acked Write 2 Done Sent Done false
23892 7756 Acked Read 6 Done Sent Done false
23893 7757 Acked Write 6 Done Sent Done false
23894 7758 Acked Flush 0 Done Sent Done false
23895 7759 Acked Write 10 Done Sent Done false
23896 7760 Acked Read 6 Done Sent Done false
23897 7761 Acked Flush 0 Done Sent Done false
23898 7762 Acked Write 8 Done Sent Done false
23899 7763 Acked Read 10 Done Sent Done false
23900 7764 Acked Write 7 Done Sent Done false
23901 7765 Acked Write 1 Done Sent Done false
23902 7766 Acked Write 10 Done Sent Done false
23903 7767 Acked Read 10 Done Sent Done false
23904 7768 Acked Read 4 Done Sent Done false
23905 7769 Acked Read 7 Done Sent Done false
23906 7770 Acked Read 7 Done Sent Done false
23907 7771 Acked Write 9 Done Sent Done false
23908 7772 Acked Write 2 Done Sent Done false
23909 7773 Acked Write 7 Done Sent Done false
23910 7774 Acked Read 10 Done Sent Done false
23911 7775 Acked Read 5 Done Sent Done false
23912 7776 Acked Read 3 Done Sent Done false
23913 7777 Acked Read 3 Done Sent Done false
23914 7778 Acked Flush 0 Done Sent Done false
23915 7779 Acked Read 10 Done Sent Done false
23916 7780 Acked Write 8 Done Sent Done false
23917 7781 Acked Read 3 Done Sent Done false
23918 7782 Acked Flush 0 Done Sent Done false
23919 7783 Acked Read 2 Done Sent Done false
23920 7784 Acked Read 5 Done Sent Done false
23921 7785 Acked Read 3 Done Sent Done false
23922 7786 Acked Flush 0 Done Sent Done false
23923 7787 Acked Flush 0 Done Sent Done false
23924 7788 Acked Read 8 Done Sent Done false
23925 7789 Acked Read 10 Done Sent Done false
23926 7790 Acked Read 6 Done Sent Done false
23927 7791 Acked Write 2 Done Sent Done false
23928 7792 Acked Write 1 Done Sent Done false
23929 7793 Acked Write 9 Done Sent Done false
23930 7794 Acked Write 8 Done Sent Done false
23931 7795 Acked Write 8 Done Sent Done false
23932 7796 Acked Write 9 Done Sent Done false
23933 7797 Acked Read 3 Done Sent Done false
23934 7798 Acked Write 9 Done Sent Done false
23935 7799 Acked Read 2 Done Sent Done false
23936 7800 Acked Read 2 Done Sent Done false
23937 7801 Acked Write 9 Done Sent Done false
23938 7802 Acked Read 6 Done Sent Done false
23939 7803 Acked Read 9 Done Sent Done false
23940 7804 Acked Write 9 Done Sent Done false
23941 7805 Acked Read 7 Done Sent Done false
23942 7806 Acked Read 8 Done Sent Done false
23943 7807 Acked Read 9 Done Sent Done false
23944 7808 Acked Read 4 Done Sent Done false
23945 7809 Acked Read 10 Done Sent Done false
23946 7810 Acked Read 5 Done Sent Done false
23947 7811 Acked Write 10 Done Sent Done false
23948 7812 Acked Write 1 Done Sent Done false
23949 7813 Acked Read 9 Done Sent Done false
23950 7814 Acked Read 4 Done Sent Done false
23951 7815 Acked Write 10 Done Sent Done false
23952 7816 Acked Write 7 Done Sent Done false
23953 7817 Acked Read 7 Done Sent Done false
23954 7818 Acked Read 5 Done Sent Done false
23955 7819 Acked Write 3 Done Sent Done false
23956 7820 Acked Write 1 Done Sent Done false
23957 7821 Acked Read 6 Done Sent Done false
23958 7822 Acked Write 8 Done Sent Done false
23959 7823 Acked Write 3 Done Sent Done false
23960 7824 Acked Read 3 Done Sent Done false
23961 7825 Acked Read 4 Done Sent Done false
23962 7826 Acked Write 8 Done Sent Done false
23963 7827 Acked Read 10 Done Sent Done false
23964 7828 Acked Read 3 Done Sent Done false
23965 7829 Acked Read 2 Done Sent Done false
23966 7830 Acked Read 2 Done Sent Done false
23967 7831 Acked Read 10 Done Sent Done false
23968 7832 Acked Flush 0 Done Sent Done false
23969 7833 Acked Write 10 Done Sent Done false
23970 7834 Acked Read 1 Done Sent Done false
23971 7835 Acked Read 9 Done Sent Done false
23972 7836 Acked Read 1 Done Sent Done false
23973 7837 Acked Read 2 Done Sent Done false
23974 7838 Acked Read 9 Done Sent Done false
23975 7839 Acked Read 4 Done Sent Done false
23976 7840 Acked Write 9 Done Sent Done false
23977 7841 Acked Read 8 Done Sent Done false
23978 7842 Acked Write 8 Done Sent Done false
23979 7843 Acked Write 5 Done Sent Done false
23980 7844 Acked Read 9 Done Sent Done false
23981 7845 Acked Read 1 Done Sent Done false
23982 7846 Acked Read 6 Done Sent Done false
23983 7847 Acked Write 10 Done Sent Done false
23984 7848 Acked Write 10 Done Sent Done false
23985 7849 Acked Read 8 Done Sent Done false
23986 7850 Acked Read 2 Done Sent Done false
23987 7851 Acked Write 8 Done Sent Done false
23988 7852 Acked Read 6 Done Sent Done false
23989 7853 Acked Read 4 Done Sent Done false
23990 7854 Acked Read 4 Done Sent Done false
23991 7855 Acked Write 3 Done Sent Done false
23992 7856 Acked Write 4 Done Sent Done false
23993 7857 Acked Write 6 Done Sent Done false
23994 7858 Acked Read 7 Done Sent Done false
23995 7859 Acked Read 7 Done Sent Done false
23996 7860 Acked Read 4 Done Sent Done false
23997 7861 Acked Flush 0 Done Sent Done false
23998 7862 Acked Read 2 Done Sent Done false
23999 7863 Acked Write 3 Done Sent Done false
24000 7864 Acked Write 1 Done Sent Done false
24001 7865 Acked Read 5 Done Sent Done false
24002 7866 Acked Flush 0 Done Sent Done false
24003 7867 Acked Flush 0 Done Sent Done false
24004 7868 Acked Read 9 Done Sent Done false
24005 7869 Acked Read 8 Done Sent Done false
24006 7870 Acked Read 4 Done Sent Done false
24007 7871 Acked Read 5 Done Sent Done false
24008 7872 Acked Write 9 Done Sent Done false
24009 7873 Acked Write 6 Done Sent Done false
24010 7874 Acked Read 4 Done Sent Done false
24011 7875 Acked Write 4 Done Sent Done false
24012 7876 Acked Write 10 Done Sent Done false
24013 7877 Acked Write 6 Done Sent Done false
24014 7878 Acked Read 1 Done Sent Done false
24015 7879 Acked Write 8 Done Sent Done false
24016 7880 Acked Read 5 Done Sent Done false
24017 7881 Acked Flush 0 Done Sent Done false
24018 7882 Acked Read 2 Done Sent Done false
24019 7883 Acked Write 7 Done Sent Done false
24020 7884 Acked Write 1 Done Sent Done false
24021 7885 Acked Write 9 Done Sent Done false
24022 7886 Acked Read 5 Done Sent Done false
24023 7887 Acked Read 8 Done Sent Done false
24024 7888 Acked Write 6 Done Sent Done false
24025 7889 Acked Write 7 Done Sent Done false
24026 7890 Acked Write 8 Done Sent Done false
24027 7891 Acked Write 8 Done Sent Done false
24028 7892 Acked Read 8 Done Sent Done false
24029 7893 Acked Read 5 Done Sent Done false
24030 7894 Acked Read 1 Done Sent Done false
24031 7895 Acked Read 8 Done Sent Done false
24032 7896 Acked Read 9 Done Sent Done false
24033 7897 Acked Write 10 Done Sent Done false
24034 7898 Acked Write 5 Done Sent Done false
24035 7899 Acked Read 7 Done Sent Done false
24036 7900 Acked Flush 0 Done Sent Done false
24037 7901 Acked Read 2 Done Sent Done false
24038 7902 Acked Read 3 Done Sent Done false
24039 7903 Acked Read 10 Done Sent Done false
24040 7904 Acked Read 4 Done Sent Done false
24041 7905 Acked Write 4 Done Sent Done false
24042 7906 Acked Read 6 Done Sent Done false
24043 7907 Acked Write 4 Done Sent Done false
24044 7908 Acked Read 1 Done Sent Done false
24045 7909 Acked Read 4 Done Sent Done false
24046 7910 Acked Write 10 Done Sent Done false
24047 7911 Acked Write 7 Done Sent Done false
24048 7912 Acked Read 2 Done Sent Done false
24049 7913 Acked Write 5 Done Sent Done false
24050 7914 Acked Read 8 Done Sent Done false
24051 7915 Acked Read 7 Done Sent Done false
24052 7916 Acked Write 3 Done Sent Done false
24053 7917 Acked Read 3 Done Sent Done false
24054 7918 Acked Flush 0 Done Sent Done false
24055 7919 Acked Read 5 Done Sent Done false
24056 7920 Acked Write 5 Done Sent Done false
24057 7921 Acked Read 8 Done Sent Done false
24058 7922 Acked Write 1 Done Sent Done false
24059 7923 Acked Flush 0 Done Sent Done false
24060 7924 Acked Write 7 Done Sent Done false
24061 7925 Acked Write 8 Done Sent Done false
24062 7926 Acked Read 5 Done Sent Done false
24063 7927 Acked Read 8 Done Sent Done false
24064 7928 Acked Write 5 Done Sent Done false
24065 7929 Acked Write 4 Done Sent Done false
24066 7930 Acked Write 4 Done Sent Done false
24067 7931 Acked Write 8 Done Sent Done false
24068 7932 Acked Write 3 Done Sent Done false
24069 7933 Acked Write 5 Done Sent Done false
24070 7934 Acked Read 1 Done Sent Done false
24071 7935 Acked Read 2 Done Sent Done false
24072 7936 Acked Flush 0 Done Sent Done false
24073 7937 Acked Write 4 Done Sent Done false
24074 7938 Acked Write 7 Done Sent Done false
24075 7939 Acked Write 7 Done Sent Done false
24076 7940 Acked Read 10 Done Sent Done false
24077 7941 Acked Flush 0 Done Sent Done false
24078 7942 Acked Read 7 Done Sent Done false
24079 7943 Acked Write 7 Done Sent Done false
24080 STATES DS:0 DS:1 DS:2 TOTAL
24081 Sent 0 301 0 301
24082 Done 301 0 301 602
24083 Skipped 0 0 0 0
24084 Error 0 0 0 0
24085 Last Flush: Some(JobId(7941)) Some(JobId(7642)) Some(JobId(7941))
24086 Downstairs last five completed: 7642 7641 7640 7639 7638
24087 Upstairs last five completed: 7943 7942 7941 7940 7939
24088 CLIENT: Up:0 ds:301 act:2
24089 ----------------------------------------------------------------
24090 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24091 States: Active Offline Active
24092 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24093 7643 Acked Write 3 Done Sent Done false
24094 7644 Acked Write 9 Done Sent Done false
24095 7645 Acked Write 2 Done Sent Done false
24096 7646 Acked Read 7 Done Sent Done false
24097 7647 Acked Write 8 Done Sent Done false
24098 7648 Acked Read 8 Done Sent Done false
24099 7649 Acked Read 10 Done Sent Done false
24100 7650 Acked Write 1 Done Sent Done false
24101 7651 Acked Read 4 Done Sent Done false
24102 7652 Acked Read 6 Done Sent Done false
24103 7653 Acked Read 6 Done Sent Done false
24104 7654 Acked Read 8 Done Sent Done false
24105 7655 Acked Read 2 Done Sent Done false
24106 7656 Acked Write 4 Done Sent Done false
24107 7657 Acked Read 8 Done Sent Done false
24108 7658 Acked Write 8 Done Sent Done false
24109 7659 Acked Read 6 Done Sent Done false
24110 7660 Acked Flush 0 Done Sent Done false
24111 7661 Acked Read 3 Done Sent Done false
24112 7662 Acked Write 10 Done Sent Done false
24113 7663 Acked Read 2 Done Sent Done false
24114 7664 Acked Read 6 Done Sent Done false
24115 7665 Acked Read 2 Done Sent Done false
24116 7666 Acked Read 5 Done Sent Done false
24117 7667 Acked Read 10 Done Sent Done false
24118 7668 Acked Write 5 Done Sent Done false
24119 7669 Acked Read 10 Done Sent Done false
24120 7670 Acked Write 7 Done Sent Done false
24121 7671 Acked Read 6 Done Sent Done false
24122 7672 Acked Write 7 Done Sent Done false
24123 7673 Acked Read 5 Done Sent Done false
24124 7674 Acked Flush 0 Done Sent Done false
24125 7675 Acked Read 8 Done Sent Done false
24126 7676 Acked Write 5 Done Sent Done false
24127 7677 Acked Read 6 Done Sent Done false
24128 7678 Acked Read 10 Done Sent Done false
24129 7679 Acked Write 3 Done Sent Done false
24130 7680 Acked Read 1 Done Sent Done false
24131 7681 Acked Write 5 Done Sent Done false
24132 7682 Acked Write 9 Done Sent Done false
24133 7683 Acked Write 6 Done Sent Done false
24134 7684 Acked Read 6 Done Sent Done false
24135 7685 Acked Read 7 Done Sent Done false
24136 7686 Acked Read 5 Done Sent Done false
24137 7687 Acked Flush 0 Done Sent Done false
24138 7688 Acked Read 9 Done Sent Done false
24139 7689 Acked Read 1 Done Sent Done false
24140 7690 Acked Write 1 Done Sent Done false
24141 7691 Acked Write 2 Done Sent Done false
24142 7692 Acked Write 9 Done Sent Done false
24143 7693 Acked Write 9 Done Sent Done false
24144 7694 Acked Write 4 Done Sent Done false
24145 7695 Acked Write 6 Done Sent Done false
24146 7696 Acked Write 4 Done Sent Done false
24147 7697 Acked Flush 0 Done Sent Done false
24148 7698 Acked Flush 0 Done Sent Done false
24149 7699 Acked Write 8 Done Sent Done false
24150 7700 Acked Write 9 Done Sent Done false
24151 7701 Acked Read 9 Done Sent Done false
24152 7702 Acked Read 8 Done Sent Done false
24153 7703 Acked Read 4 Done Sent Done false
24154 7704 Acked Read 1 Done Sent Done false
24155 7705 Acked Read 7 Done Sent Done false
24156 7706 Acked Read 10 Done Sent Done false
24157 7707 Acked Read 7 Done Sent Done false
24158 7708 Acked Read 7 Done Sent Done false
24159 7709 Acked Write 9 Done Sent Done false
24160 7710 Acked Read 9 Done Sent Done false
24161 7711 Acked Read 10 Done Sent Done false
24162 7712 Acked Read 5 Done Sent Done false
24163 7713 Acked Write 3 Done Sent Done false
24164 7714 Acked Read 6 Done Sent Done false
24165 7715 Acked Write 9 Done Sent Done false
24166 7716 Acked Write 3 Done Sent Done false
24167 7717 Acked Read 1 Done Sent Done false
24168 7718 Acked Read 10 Done Sent Done false
24169 7719 Acked Write 7 Done Sent Done false
24170 7720 Acked Flush 0 Done Sent Done false
24171 7721 Acked Read 1 Done Sent Done false
24172 7722 Acked Write 2 Done Sent Done false
24173 7723 Acked Read 5 Done Sent Done false
24174 7724 Acked Read 10 Done Sent Done false
24175 7725 Acked Read 5 Done Sent Done false
24176 7726 Acked Read 7 Done Sent Done false
24177 7727 Acked Write 7 Done Sent Done false
24178 7728 Acked Read 10 Done Sent Done false
24179 7729 Acked Write 6 Done Sent Done false
24180 7730 Acked Write 5 Done Sent Done false
24181 7731 Acked Read 5 Done Sent Done false
24182 7732 Acked Write 9 Done Sent Done false
24183 7733 Acked Read 5 Done Sent Done false
24184 7734 Acked Write 5 Done Sent Done false
24185 7735 Acked Write 2 Done Sent Done false
24186 7736 Acked Flush 0 Done Sent Done false
24187 7737 Acked Write 10 Done Sent Done false
24188 7738 Acked Read 3 Done Sent Done false
24189 7739 Acked Write 6 Done Sent Done false
24190 7740 Acked Read 1 Done Sent Done false
24191 7741 Acked Write 7 Done Sent Done false
24192 7742 Acked Read 8 Done Sent Done false
24193 7743 Acked Write 1 Done Sent Done false
24194 7744 Acked Write 4 Done Sent Done false
24195 7745 Acked Write 9 Done Sent Done false
24196 7746 Acked Write 7 Done Sent Done false
24197 7747 Acked Read 9 Done Sent Done false
24198 7748 Acked Write 8 Done Sent Done false
24199 7749 Acked Read 6 Done Sent Done false
24200 7750 Acked Read 7 Done Sent Done false
24201 7751 Acked Read 8 Done Sent Done false
24202 7752 Acked Read 2 Done Sent Done false
24203 7753 Acked Read 3 Done Sent Done false
24204 7754 Acked Write 10 Done Sent Done false
24205 7755 Acked Write 2 Done Sent Done false
24206 7756 Acked Read 6 Done Sent Done false
24207 7757 Acked Write 6 Done Sent Done false
24208 7758 Acked Flush 0 Done Sent Done false
24209 7759 Acked Write 10 Done Sent Done false
24210 7760 Acked Read 6 Done Sent Done false
24211 7761 Acked Flush 0 Done Sent Done false
24212 7762 Acked Write 8 Done Sent Done false
24213 7763 Acked Read 10 Done Sent Done false
24214 7764 Acked Write 7 Done Sent Done false
24215 7765 Acked Write 1 Done Sent Done false
24216 7766 Acked Write 10 Done Sent Done false
24217 7767 Acked Read 10 Done Sent Done false
24218 7768 Acked Read 4 Done Sent Done false
24219 7769 Acked Read 7 Done Sent Done false
24220 7770 Acked Read 7 Done Sent Done false
24221 7771 Acked Write 9 Done Sent Done false
24222 7772 Acked Write 2 Done Sent Done false
24223 7773 Acked Write 7 Done Sent Done false
24224 7774 Acked Read 10 Done Sent Done false
24225 7775 Acked Read 5 Done Sent Done false
24226 7776 Acked Read 3 Done Sent Done false
24227 7777 Acked Read 3 Done Sent Done false
24228 7778 Acked Flush 0 Done Sent Done false
24229 7779 Acked Read 10 Done Sent Done false
24230 7780 Acked Write 8 Done Sent Done false
24231 7781 Acked Read 3 Done Sent Done false
24232 7782 Acked Flush 0 Done Sent Done false
24233 7783 Acked Read 2 Done Sent Done false
24234 7784 Acked Read 5 Done Sent Done false
24235 7785 Acked Read 3 Done Sent Done false
24236 7786 Acked Flush 0 Done Sent Done false
24237 7787 Acked Flush 0 Done Sent Done false
24238 7788 Acked Read 8 Done Sent Done false
24239 7789 Acked Read 10 Done Sent Done false
24240 7790 Acked Read 6 Done Sent Done false
24241 7791 Acked Write 2 Done Sent Done false
24242 7792 Acked Write 1 Done Sent Done false
24243 7793 Acked Write 9 Done Sent Done false
24244 7794 Acked Write 8 Done Sent Done false
24245 7795 Acked Write 8 Done Sent Done false
24246 7796 Acked Write 9 Done Sent Done false
24247 7797 Acked Read 3 Done Sent Done false
24248 7798 Acked Write 9 Done Sent Done false
24249 7799 Acked Read 2 Done Sent Done false
24250 7800 Acked Read 2 Done Sent Done false
24251 7801 Acked Write 9 Done Sent Done false
24252 7802 Acked Read 6 Done Sent Done false
24253 7803 Acked Read 9 Done Sent Done false
24254 7804 Acked Write 9 Done Sent Done false
24255 7805 Acked Read 7 Done Sent Done false
24256 7806 Acked Read 8 Done Sent Done false
24257 7807 Acked Read 9 Done Sent Done false
24258 7808 Acked Read 4 Done Sent Done false
24259 7809 Acked Read 10 Done Sent Done false
24260 7810 Acked Read 5 Done Sent Done false
24261 7811 Acked Write 10 Done Sent Done false
24262 7812 Acked Write 1 Done Sent Done false
24263 7813 Acked Read 9 Done Sent Done false
24264 7814 Acked Read 4 Done Sent Done false
24265 7815 Acked Write 10 Done Sent Done false
24266 7816 Acked Write 7 Done Sent Done false
24267 7817 Acked Read 7 Done Sent Done false
24268 7818 Acked Read 5 Done Sent Done false
24269 7819 Acked Write 3 Done Sent Done false
24270 7820 Acked Write 1 Done Sent Done false
24271 7821 Acked Read 6 Done Sent Done false
24272 7822 Acked Write 8 Done Sent Done false
24273 7823 Acked Write 3 Done Sent Done false
24274 7824 Acked Read 3 Done Sent Done false
24275 7825 Acked Read 4 Done Sent Done false
24276 7826 Acked Write 8 Done Sent Done false
24277 7827 Acked Read 10 Done Sent Done false
24278 7828 Acked Read 3 Done Sent Done false
24279 7829 Acked Read 2 Done Sent Done false
24280 7830 Acked Read 2 Done Sent Done false
24281 7831 Acked Read 10 Done Sent Done false
24282 7832 Acked Flush 0 Done Sent Done false
24283 7833 Acked Write 10 Done Sent Done false
24284 7834 Acked Read 1 Done Sent Done false
24285 7835 Acked Read 9 Done Sent Done false
24286 7836 Acked Read 1 Done Sent Done false
24287 7837 Acked Read 2 Done Sent Done false
24288 7838 Acked Read 9 Done Sent Done false
24289 7839 Acked Read 4 Done Sent Done false
24290 7840 Acked Write 9 Done Sent Done false
24291 7841 Acked Read 8 Done Sent Done false
24292 7842 Acked Write 8 Done Sent Done false
24293 7843 Acked Write 5 Done Sent Done false
24294 7844 Acked Read 9 Done Sent Done false
24295 7845 Acked Read 1 Done Sent Done false
24296 7846 Acked Read 6 Done Sent Done false
24297 7847 Acked Write 10 Done Sent Done false
24298 7848 Acked Write 10 Done Sent Done false
24299 7849 Acked Read 8 Done Sent Done false
24300 7850 Acked Read 2 Done Sent Done false
24301 7851 Acked Write 8 Done Sent Done false
24302 7852 Acked Read 6 Done Sent Done false
24303 7853 Acked Read 4 Done Sent Done false
24304 7854 Acked Read 4 Done Sent Done false
24305 7855 Acked Write 3 Done Sent Done false
24306 7856 Acked Write 4 Done Sent Done false
24307 7857 Acked Write 6 Done Sent Done false
24308 7858 Acked Read 7 Done Sent Done false
24309 7859 Acked Read 7 Done Sent Done false
24310 7860 Acked Read 4 Done Sent Done false
24311 7861 Acked Flush 0 Done Sent Done false
24312 7862 Acked Read 2 Done Sent Done false
24313 7863 Acked Write 3 Done Sent Done false
24314 7864 Acked Write 1 Done Sent Done false
24315 7865 Acked Read 5 Done Sent Done false
24316 7866 Acked Flush 0 Done Sent Done false
24317 7867 Acked Flush 0 Done Sent Done false
24318 7868 Acked Read 9 Done Sent Done false
24319 7869 Acked Read 8 Done Sent Done false
24320 7870 Acked Read 4 Done Sent Done false
24321 7871 Acked Read 5 Done Sent Done false
24322 7872 Acked Write 9 Done Sent Done false
24323 7873 Acked Write 6 Done Sent Done false
24324 7874 Acked Read 4 Done Sent Done false
24325 7875 Acked Write 4 Done Sent Done false
24326 7876 Acked Write 10 Done Sent Done false
24327 7877 Acked Write 6 Done Sent Done false
24328 7878 Acked Read 1 Done Sent Done false
24329 7879 Acked Write 8 Done Sent Done false
24330 7880 Acked Read 5 Done Sent Done false
24331 7881 Acked Flush 0 Done Sent Done false
24332 7882 Acked Read 2 Done Sent Done false
24333 7883 Acked Write 7 Done Sent Done false
24334 7884 Acked Write 1 Done Sent Done false
24335 7885 Acked Write 9 Done Sent Done false
24336 7886 Acked Read 5 Done Sent Done false
24337 7887 Acked Read 8 Done Sent Done false
24338 7888 Acked Write 6 Done Sent Done false
24339 7889 Acked Write 7 Done Sent Done false
24340 7890 Acked Write 8 Done Sent Done false
24341 7891 Acked Write 8 Done Sent Done false
24342 7892 Acked Read 8 Done Sent Done false
24343 7893 Acked Read 5 Done Sent Done false
24344 7894 Acked Read 1 Done Sent Done false
24345 7895 Acked Read 8 Done Sent Done false
24346 7896 Acked Read 9 Done Sent Done false
24347 7897 Acked Write 10 Done Sent Done false
24348 7898 Acked Write 5 Done Sent Done false
24349 7899 Acked Read 7 Done Sent Done false
24350 7900 Acked Flush 0 Done Sent Done false
24351 7901 Acked Read 2 Done Sent Done false
24352 7902 Acked Read 3 Done Sent Done false
24353 7903 Acked Read 10 Done Sent Done false
24354 7904 Acked Read 4 Done Sent Done false
24355 7905 Acked Write 4 Done Sent Done false
24356 7906 Acked Read 6 Done Sent Done false
24357 7907 Acked Write 4 Done Sent Done false
24358 7908 Acked Read 1 Done Sent Done false
24359 7909 Acked Read 4 Done Sent Done false
24360 7910 Acked Write 10 Done Sent Done false
24361 7911 Acked Write 7 Done Sent Done false
24362 7912 Acked Read 2 Done Sent Done false
24363 7913 Acked Write 5 Done Sent Done false
24364 7914 Acked Read 8 Done Sent Done false
24365 7915 Acked Read 7 Done Sent Done false
24366 7916 Acked Write 3 Done Sent Done false
24367 7917 Acked Read 3 Done Sent Done false
24368 7918 Acked Flush 0 Done Sent Done false
24369 7919 Acked Read 5 Done Sent Done false
24370 7920 Acked Write 5 Done Sent Done false
24371 7921 Acked Read 8 Done Sent Done false
24372 7922 Acked Write 1 Done Sent Done false
24373 7923 Acked Flush 0 Done Sent Done false
24374 7924 Acked Write 7 Done Sent Done false
24375 7925 Acked Write 8 Done Sent Done false
24376 7926 Acked Read 5 Done Sent Done false
24377 7927 Acked Read 8 Done Sent Done false
24378 7928 Acked Write 5 Done Sent Done false
24379 7929 Acked Write 4 Done Sent Done false
24380 7930 Acked Write 4 Done Sent Done false
24381 7931 Acked Write 8 Done Sent Done false
24382 7932 Acked Write 3 Done Sent Done false
24383 7933 Acked Write 5 Done Sent Done false
24384 7934 Acked Read 1 Done Sent Done false
24385 7935 Acked Read 2 Done Sent Done false
24386 7936 Acked Flush 0 Done Sent Done false
24387 7937 Acked Write 4 Done Sent Done false
24388 7938 Acked Write 7 Done Sent Done false
24389 7939 Acked Write 7 Done Sent Done false
24390 7940 Acked Read 10 Done Sent Done false
24391 7941 Acked Flush 0 Done Sent Done false
24392 7942 Acked Read 7 Done Sent Done false
24393 7943 Acked Write 7 Done Sent Done false
24394 7944 Acked Flush 0 Done Sent Done false
24395 STATES DS:0 DS:1 DS:2 TOTAL
24396 Sent 0 302 0 302
24397 Done 302 0 302 604
24398 Skipped 0 0 0 0
24399 Error 0 0 0 0
24400 Last Flush: Some(JobId(7944)) Some(JobId(7642)) Some(JobId(7944))
24401 Downstairs last five completed: 7642 7641 7640 7639 7638
24402 Upstairs last five completed: 7944 7943 7942 7941 7940
24403 CLIENT: Up:0 ds:302 act:2
244042025-01-23T17:42:51.169ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
244052025-01-23T17:42:51.169ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
244062025-01-23T17:42:51.169ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
244072025-01-23T17:42:51.169ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
244082025-01-23T17:42:51.169ZINFOcrucible: client task is exiting = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
24409 [8820] got start action
24410 [8820] I am going to restart
24411 Starting downstairs at port 8820
24412 Make output file at "/tmp/dsc/downstairs-8820.txt"
24413 [8820][1] reports Starting
24414 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1228)
24415 [8820][1] reports Running
24416 ----------------------------------------------------------------
24417 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24418 States: Active Offline Active
24419 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24420 7643 Acked Write 3 Done Sent Done false
24421 7644 Acked Write 9 Done Sent Done false
24422 7645 Acked Write 2 Done Sent Done false
24423 7646 Acked Read 7 Done Sent Done false
24424 7647 Acked Write 8 Done Sent Done false
24425 7648 Acked Read 8 Done Sent Done false
24426 7649 Acked Read 10 Done Sent Done false
24427 7650 Acked Write 1 Done Sent Done false
24428 7651 Acked Read 4 Done Sent Done false
24429 7652 Acked Read 6 Done Sent Done false
24430 7653 Acked Read 6 Done Sent Done false
24431 7654 Acked Read 8 Done Sent Done false
24432 7655 Acked Read 2 Done Sent Done false
24433 7656 Acked Write 4 Done Sent Done false
24434 7657 Acked Read 8 Done Sent Done false
24435 7658 Acked Write 8 Done Sent Done false
24436 7659 Acked Read 6 Done Sent Done false
24437 7660 Acked Flush 0 Done Sent Done false
24438 7661 Acked Read 3 Done Sent Done false
24439 7662 Acked Write 10 Done Sent Done false
24440 7663 Acked Read 2 Done Sent Done false
24441 7664 Acked Read 6 Done Sent Done false
24442 7665 Acked Read 2 Done Sent Done false
24443 7666 Acked Read 5 Done Sent Done false
24444 7667 Acked Read 10 Done Sent Done false
24445 7668 Acked Write 5 Done Sent Done false
24446 7669 Acked Read 10 Done Sent Done false
24447 7670 Acked Write 7 Done Sent Done false
24448 7671 Acked Read 6 Done Sent Done false
24449 7672 Acked Write 7 Done Sent Done false
24450 7673 Acked Read 5 Done Sent Done false
24451 7674 Acked Flush 0 Done Sent Done false
24452 7675 Acked Read 8 Done Sent Done false
24453 7676 Acked Write 5 Done Sent Done false
24454 7677 Acked Read 6 Done Sent Done false
24455 7678 Acked Read 10 Done Sent Done false
24456 7679 Acked Write 3 Done Sent Done false
24457 7680 Acked Read 1 Done Sent Done false
24458 7681 Acked Write 5 Done Sent Done false
24459 7682 Acked Write 9 Done Sent Done false
24460 7683 Acked Write 6 Done Sent Done false
24461 7684 Acked Read 6 Done Sent Done false
24462 7685 Acked Read 7 Done Sent Done false
24463 7686 Acked Read 5 Done Sent Done false
24464 7687 Acked Flush 0 Done Sent Done false
24465 7688 Acked Read 9 Done Sent Done false
24466 7689 Acked Read 1 Done Sent Done false
24467 7690 Acked Write 1 Done Sent Done false
24468 7691 Acked Write 2 Done Sent Done false
24469 7692 Acked Write 9 Done Sent Done false
24470 7693 Acked Write 9 Done Sent Done false
24471 7694 Acked Write 4 Done Sent Done false
24472 7695 Acked Write 6 Done Sent Done false
24473 7696 Acked Write 4 Done Sent Done false
24474 7697 Acked Flush 0 Done Sent Done false
24475 7698 Acked Flush 0 Done Sent Done false
24476 7699 Acked Write 8 Done Sent Done false
24477 7700 Acked Write 9 Done Sent Done false
24478 7701 Acked Read 9 Done Sent Done false
24479 7702 Acked Read 8 Done Sent Done false
24480 7703 Acked Read 4 Done Sent Done false
24481 7704 Acked Read 1 Done Sent Done false
24482 7705 Acked Read 7 Done Sent Done false
24483 7706 Acked Read 10 Done Sent Done false
24484 7707 Acked Read 7 Done Sent Done false
24485 7708 Acked Read 7 Done Sent Done false
24486 7709 Acked Write 9 Done Sent Done false
24487 7710 Acked Read 9 Done Sent Done false
24488 7711 Acked Read 10 Done Sent Done false
24489 7712 Acked Read 5 Done Sent Done false
24490 7713 Acked Write 3 Done Sent Done false
24491 7714 Acked Read 6 Done Sent Done false
24492 7715 Acked Write 9 Done Sent Done false
24493 7716 Acked Write 3 Done Sent Done false
24494 7717 Acked Read 1 Done Sent Done false
24495 7718 Acked Read 10 Done Sent Done false
24496 7719 Acked Write 7 Done Sent Done false
24497 7720 Acked Flush 0 Done Sent Done false
24498 7721 Acked Read 1 Done Sent Done false
24499 7722 Acked Write 2 Done Sent Done false
24500 7723 Acked Read 5 Done Sent Done false
24501 7724 Acked Read 10 Done Sent Done false
24502 7725 Acked Read 5 Done Sent Done false
24503 7726 Acked Read 7 Done Sent Done false
24504 7727 Acked Write 7 Done Sent Done false
24505 7728 Acked Read 10 Done Sent Done false
24506 7729 Acked Write 6 Done Sent Done false
24507 7730 Acked Write 5 Done Sent Done false
24508 7731 Acked Read 5 Done Sent Done false
24509 7732 Acked Write 9 Done Sent Done false
24510 7733 Acked Read 5 Done Sent Done false
24511 7734 Acked Write 5 Done Sent Done false
24512 7735 Acked Write 2 Done Sent Done false
24513 7736 Acked Flush 0 Done Sent Done false
24514 7737 Acked Write 10 Done Sent Done false
24515 7738 Acked Read 3 Done Sent Done false
24516 7739 Acked Write 6 Done Sent Done false
24517 7740 Acked Read 1 Done Sent Done false
24518 7741 Acked Write 7 Done Sent Done false
24519 7742 Acked Read 8 Done Sent Done false
24520 7743 Acked Write 1 Done Sent Done false
24521 7744 Acked Write 4 Done Sent Done false
24522 7745 Acked Write 9 Done Sent Done false
24523 7746 Acked Write 7 Done Sent Done false
24524 7747 Acked Read 9 Done Sent Done false
24525 7748 Acked Write 8 Done Sent Done false
24526 7749 Acked Read 6 Done Sent Done false
24527 7750 Acked Read 7 Done Sent Done false
24528 7751 Acked Read 8 Done Sent Done false
24529 7752 Acked Read 2 Done Sent Done false
24530 7753 Acked Read 3 Done Sent Done false
24531 7754 Acked Write 10 Done Sent Done false
24532 7755 Acked Write 2 Done Sent Done false
24533 7756 Acked Read 6 Done Sent Done false
24534 7757 Acked Write 6 Done Sent Done false
24535 7758 Acked Flush 0 Done Sent Done false
24536 7759 Acked Write 10 Done Sent Done false
24537 7760 Acked Read 6 Done Sent Done false
24538 7761 Acked Flush 0 Done Sent Done false
24539 7762 Acked Write 8 Done Sent Done false
24540 7763 Acked Read 10 Done Sent Done false
24541 7764 Acked Write 7 Done Sent Done false
24542 7765 Acked Write 1 Done Sent Done false
24543 7766 Acked Write 10 Done Sent Done false
24544 7767 Acked Read 10 Done Sent Done false
24545 7768 Acked Read 4 Done Sent Done false
24546 7769 Acked Read 7 Done Sent Done false
24547 7770 Acked Read 7 Done Sent Done false
24548 7771 Acked Write 9 Done Sent Done false
24549 7772 Acked Write 2 Done Sent Done false
24550 7773 Acked Write 7 Done Sent Done false
24551 7774 Acked Read 10 Done Sent Done false
24552 7775 Acked Read 5 Done Sent Done false
24553 7776 Acked Read 3 Done Sent Done false
24554 7777 Acked Read 3 Done Sent Done false
24555 7778 Acked Flush 0 Done Sent Done false
24556 7779 Acked Read 10 Done Sent Done false
24557 7780 Acked Write 8 Done Sent Done false
24558 7781 Acked Read 3 Done Sent Done false
24559 7782 Acked Flush 0 Done Sent Done false
24560 7783 Acked Read 2 Done Sent Done false
24561 7784 Acked Read 5 Done Sent Done false
24562 7785 Acked Read 3 Done Sent Done false
24563 7786 Acked Flush 0 Done Sent Done false
24564 7787 Acked Flush 0 Done Sent Done false
24565 7788 Acked Read 8 Done Sent Done false
24566 7789 Acked Read 10 Done Sent Done false
24567 7790 Acked Read 6 Done Sent Done false
24568 7791 Acked Write 2 Done Sent Done false
24569 7792 Acked Write 1 Done Sent Done false
24570 7793 Acked Write 9 Done Sent Done false
24571 7794 Acked Write 8 Done Sent Done false
24572 7795 Acked Write 8 Done Sent Done false
24573 7796 Acked Write 9 Done Sent Done false
24574 7797 Acked Read 3 Done Sent Done false
24575 7798 Acked Write 9 Done Sent Done false
24576 7799 Acked Read 2 Done Sent Done false
24577 7800 Acked Read 2 Done Sent Done false
24578 7801 Acked Write 9 Done Sent Done false
24579 7802 Acked Read 6 Done Sent Done false
24580 7803 Acked Read 9 Done Sent Done false
24581 7804 Acked Write 9 Done Sent Done false
24582 7805 Acked Read 7 Done Sent Done false
24583 7806 Acked Read 8 Done Sent Done false
24584 7807 Acked Read 9 Done Sent Done false
24585 7808 Acked Read 4 Done Sent Done false
24586 7809 Acked Read 10 Done Sent Done false
24587 7810 Acked Read 5 Done Sent Done false
24588 7811 Acked Write 10 Done Sent Done false
24589 7812 Acked Write 1 Done Sent Done false
24590 7813 Acked Read 9 Done Sent Done false
24591 7814 Acked Read 4 Done Sent Done false
24592 7815 Acked Write 10 Done Sent Done false
24593 7816 Acked Write 7 Done Sent Done false
24594 7817 Acked Read 7 Done Sent Done false
24595 7818 Acked Read 5 Done Sent Done false
24596 7819 Acked Write 3 Done Sent Done false
24597 7820 Acked Write 1 Done Sent Done false
24598 7821 Acked Read 6 Done Sent Done false
24599 7822 Acked Write 8 Done Sent Done false
24600 7823 Acked Write 3 Done Sent Done false
24601 7824 Acked Read 3 Done Sent Done false
24602 7825 Acked Read 4 Done Sent Done false
24603 7826 Acked Write 8 Done Sent Done false
24604 7827 Acked Read 10 Done Sent Done false
24605 7828 Acked Read 3 Done Sent Done false
24606 7829 Acked Read 2 Done Sent Done false
24607 7830 Acked Read 2 Done Sent Done false
24608 7831 Acked Read 10 Done Sent Done false
24609 7832 Acked Flush 0 Done Sent Done false
24610 7833 Acked Write 10 Done Sent Done false
24611 7834 Acked Read 1 Done Sent Done false
24612 7835 Acked Read 9 Done Sent Done false
24613 7836 Acked Read 1 Done Sent Done false
24614 7837 Acked Read 2 Done Sent Done false
24615 7838 Acked Read 9 Done Sent Done false
24616 7839 Acked Read 4 Done Sent Done false
24617 7840 Acked Write 9 Done Sent Done false
24618 7841 Acked Read 8 Done Sent Done false
24619 7842 Acked Write 8 Done Sent Done false
24620 7843 Acked Write 5 Done Sent Done false
24621 7844 Acked Read 9 Done Sent Done false
24622 7845 Acked Read 1 Done Sent Done false
24623 7846 Acked Read 6 Done Sent Done false
24624 7847 Acked Write 10 Done Sent Done false
24625 7848 Acked Write 10 Done Sent Done false
24626 7849 Acked Read 8 Done Sent Done false
24627 7850 Acked Read 2 Done Sent Done false
24628 7851 Acked Write 8 Done Sent Done false
24629 7852 Acked Read 6 Done Sent Done false
24630 7853 Acked Read 4 Done Sent Done false
24631 7854 Acked Read 4 Done Sent Done false
24632 7855 Acked Write 3 Done Sent Done false
24633 7856 Acked Write 4 Done Sent Done false
24634 7857 Acked Write 6 Done Sent Done false
24635 7858 Acked Read 7 Done Sent Done false
24636 7859 Acked Read 7 Done Sent Done false
24637 7860 Acked Read 4 Done Sent Done false
24638 7861 Acked Flush 0 Done Sent Done false
24639 7862 Acked Read 2 Done Sent Done false
24640 7863 Acked Write 3 Done Sent Done false
24641 7864 Acked Write 1 Done Sent Done false
24642 7865 Acked Read 5 Done Sent Done false
24643 7866 Acked Flush 0 Done Sent Done false
24644 7867 Acked Flush 0 Done Sent Done false
24645 7868 Acked Read 9 Done Sent Done false
24646 7869 Acked Read 8 Done Sent Done false
24647 7870 Acked Read 4 Done Sent Done false
24648 7871 Acked Read 5 Done Sent Done false
24649 7872 Acked Write 9 Done Sent Done false
24650 7873 Acked Write 6 Done Sent Done false
24651 7874 Acked Read 4 Done Sent Done false
24652 7875 Acked Write 4 Done Sent Done false
24653 7876 Acked Write 10 Done Sent Done false
24654 7877 Acked Write 6 Done Sent Done false
24655 7878 Acked Read 1 Done Sent Done false
24656 7879 Acked Write 8 Done Sent Done false
24657 7880 Acked Read 5 Done Sent Done false
24658 7881 Acked Flush 0 Done Sent Done false
24659 7882 Acked Read 2 Done Sent Done false
24660 7883 Acked Write 7 Done Sent Done false
24661 7884 Acked Write 1 Done Sent Done false
24662 7885 Acked Write 9 Done Sent Done false
24663 7886 Acked Read 5 Done Sent Done false
24664 7887 Acked Read 8 Done Sent Done false
24665 7888 Acked Write 6 Done Sent Done false
24666 7889 Acked Write 7 Done Sent Done false
24667 7890 Acked Write 8 Done Sent Done false
24668 7891 Acked Write 8 Done Sent Done false
24669 7892 Acked Read 8 Done Sent Done false
24670 7893 Acked Read 5 Done Sent Done false
24671 7894 Acked Read 1 Done Sent Done false
24672 7895 Acked Read 8 Done Sent Done false
24673 7896 Acked Read 9 Done Sent Done false
24674 7897 Acked Write 10 Done Sent Done false
24675 7898 Acked Write 5 Done Sent Done false
24676 7899 Acked Read 7 Done Sent Done false
24677 7900 Acked Flush 0 Done Sent Done false
24678 7901 Acked Read 2 Done Sent Done false
24679 7902 Acked Read 3 Done Sent Done false
24680 7903 Acked Read 10 Done Sent Done false
24681 7904 Acked Read 4 Done Sent Done false
24682 7905 Acked Write 4 Done Sent Done false
24683 7906 Acked Read 6 Done Sent Done false
24684 7907 Acked Write 4 Done Sent Done false
24685 7908 Acked Read 1 Done Sent Done false
24686 7909 Acked Read 4 Done Sent Done false
24687 7910 Acked Write 10 Done Sent Done false
24688 7911 Acked Write 7 Done Sent Done false
24689 7912 Acked Read 2 Done Sent Done false
24690 7913 Acked Write 5 Done Sent Done false
24691 7914 Acked Read 8 Done Sent Done false
24692 7915 Acked Read 7 Done Sent Done false
24693 7916 Acked Write 3 Done Sent Done false
24694 7917 Acked Read 3 Done Sent Done false
24695 7918 Acked Flush 0 Done Sent Done false
24696 7919 Acked Read 5 Done Sent Done false
24697 7920 Acked Write 5 Done Sent Done false
24698 7921 Acked Read 8 Done Sent Done false
24699 7922 Acked Write 1 Done Sent Done false
24700 7923 Acked Flush 0 Done Sent Done false
24701 7924 Acked Write 7 Done Sent Done false
24702 7925 Acked Write 8 Done Sent Done false
24703 7926 Acked Read 5 Done Sent Done false
24704 7927 Acked Read 8 Done Sent Done false
24705 7928 Acked Write 5 Done Sent Done false
24706 7929 Acked Write 4 Done Sent Done false
24707 7930 Acked Write 4 Done Sent Done false
24708 7931 Acked Write 8 Done Sent Done false
24709 7932 Acked Write 3 Done Sent Done false
24710 7933 Acked Write 5 Done Sent Done false
24711 7934 Acked Read 1 Done Sent Done false
24712 7935 Acked Read 2 Done Sent Done false
24713 7936 Acked Flush 0 Done Sent Done false
24714 7937 Acked Write 4 Done Sent Done false
24715 7938 Acked Write 7 Done Sent Done false
24716 7939 Acked Write 7 Done Sent Done false
24717 7940 Acked Read 10 Done Sent Done false
24718 7941 Acked Flush 0 Done Sent Done false
24719 7942 Acked Read 7 Done Sent Done false
24720 7943 Acked Write 7 Done Sent Done false
24721 7944 Acked Flush 0 Done Sent Done false
24722 STATES DS:0 DS:1 DS:2 TOTAL
24723 Sent 0 302 0 302
24724 Done 302 0 302 604
24725 Skipped 0 0 0 0
24726 Error 0 0 0 0
24727 Last Flush: Some(JobId(7944)) Some(JobId(7642)) Some(JobId(7944))
24728 Downstairs last five completed: 7642 7641 7640 7639 7638
24729 Upstairs last five completed: 7944 7943 7942 7941 7940
24730 CLIENT: Up:0 ds:302 act:2
24731 ----------------------------------------------------------------
24732 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24733 States: Active Offline Active
24734 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24735 7643 Acked Write 3 Done Sent Done false
24736 7644 Acked Write 9 Done Sent Done false
24737 7645 Acked Write 2 Done Sent Done false
24738 7646 Acked Read 7 Done Sent Done false
24739 7647 Acked Write 8 Done Sent Done false
24740 7648 Acked Read 8 Done Sent Done false
24741 7649 Acked Read 10 Done Sent Done false
24742 7650 Acked Write 1 Done Sent Done false
24743 7651 Acked Read 4 Done Sent Done false
24744 7652 Acked Read 6 Done Sent Done false
24745 7653 Acked Read 6 Done Sent Done false
24746 7654 Acked Read 8 Done Sent Done false
24747 7655 Acked Read 2 Done Sent Done false
24748 7656 Acked Write 4 Done Sent Done false
24749 7657 Acked Read 8 Done Sent Done false
24750 7658 Acked Write 8 Done Sent Done false
24751 7659 Acked Read 6 Done Sent Done false
24752 7660 Acked Flush 0 Done Sent Done false
24753 7661 Acked Read 3 Done Sent Done false
24754 7662 Acked Write 10 Done Sent Done false
24755 7663 Acked Read 2 Done Sent Done false
24756 7664 Acked Read 6 Done Sent Done false
24757 7665 Acked Read 2 Done Sent Done false
24758 7666 Acked Read 5 Done Sent Done false
24759 7667 Acked Read 10 Done Sent Done false
24760 7668 Acked Write 5 Done Sent Done false
24761 7669 Acked Read 10 Done Sent Done false
24762 7670 Acked Write 7 Done Sent Done false
24763 7671 Acked Read 6 Done Sent Done false
24764 7672 Acked Write 7 Done Sent Done false
24765 7673 Acked Read 5 Done Sent Done false
24766 7674 Acked Flush 0 Done Sent Done false
24767 7675 Acked Read 8 Done Sent Done false
24768 7676 Acked Write 5 Done Sent Done false
24769 7677 Acked Read 6 Done Sent Done false
24770 7678 Acked Read 10 Done Sent Done false
24771 7679 Acked Write 3 Done Sent Done false
24772 7680 Acked Read 1 Done Sent Done false
24773 7681 Acked Write 5 Done Sent Done false
24774 7682 Acked Write 9 Done Sent Done false
24775 7683 Acked Write 6 Done Sent Done false
24776 7684 Acked Read 6 Done Sent Done false
24777 7685 Acked Read 7 Done Sent Done false
24778 7686 Acked Read 5 Done Sent Done false
24779 7687 Acked Flush 0 Done Sent Done false
24780 7688 Acked Read 9 Done Sent Done false
24781 7689 Acked Read 1 Done Sent Done false
24782 7690 Acked Write 1 Done Sent Done false
24783 7691 Acked Write 2 Done Sent Done false
24784 7692 Acked Write 9 Done Sent Done false
24785 7693 Acked Write 9 Done Sent Done false
24786 7694 Acked Write 4 Done Sent Done false
24787 7695 Acked Write 6 Done Sent Done false
24788 7696 Acked Write 4 Done Sent Done false
24789 7697 Acked Flush 0 Done Sent Done false
24790 7698 Acked Flush 0 Done Sent Done false
24791 7699 Acked Write 8 Done Sent Done false
24792 7700 Acked Write 9 Done Sent Done false
24793 7701 Acked Read 9 Done Sent Done false
24794 7702 Acked Read 8 Done Sent Done false
24795 7703 Acked Read 4 Done Sent Done false
24796 7704 Acked Read 1 Done Sent Done false
24797 7705 Acked Read 7 Done Sent Done false
24798 7706 Acked Read 10 Done Sent Done false
24799 7707 Acked Read 7 Done Sent Done false
24800 7708 Acked Read 7 Done Sent Done false
24801 7709 Acked Write 9 Done Sent Done false
24802 7710 Acked Read 9 Done Sent Done false
24803 7711 Acked Read 10 Done Sent Done false
24804 7712 Acked Read 5 Done Sent Done false
24805 7713 Acked Write 3 Done Sent Done false
24806 7714 Acked Read 6 Done Sent Done false
24807 7715 Acked Write 9 Done Sent Done false
24808 7716 Acked Write 3 Done Sent Done false
24809 7717 Acked Read 1 Done Sent Done false
24810 7718 Acked Read 10 Done Sent Done false
24811 7719 Acked Write 7 Done Sent Done false
24812 7720 Acked Flush 0 Done Sent Done false
24813 7721 Acked Read 1 Done Sent Done false
24814 7722 Acked Write 2 Done Sent Done false
24815 7723 Acked Read 5 Done Sent Done false
24816 7724 Acked Read 10 Done Sent Done false
24817 7725 Acked Read 5 Done Sent Done false
24818 7726 Acked Read 7 Done Sent Done false
24819 7727 Acked Write 7 Done Sent Done false
24820 7728 Acked Read 10 Done Sent Done false
24821 7729 Acked Write 6 Done Sent Done false
24822 7730 Acked Write 5 Done Sent Done false
24823 7731 Acked Read 5 Done Sent Done false
24824 7732 Acked Write 9 Done Sent Done false
24825 7733 Acked Read 5 Done Sent Done false
24826 7734 Acked Write 5 Done Sent Done false
24827 7735 Acked Write 2 Done Sent Done false
24828 7736 Acked Flush 0 Done Sent Done false
24829 7737 Acked Write 10 Done Sent Done false
24830 7738 Acked Read 3 Done Sent Done false
24831 7739 Acked Write 6 Done Sent Done false
24832 7740 Acked Read 1 Done Sent Done false
24833 7741 Acked Write 7 Done Sent Done false
24834 7742 Acked Read 8 Done Sent Done false
24835 7743 Acked Write 1 Done Sent Done false
24836 7744 Acked Write 4 Done Sent Done false
24837 7745 Acked Write 9 Done Sent Done false
24838 7746 Acked Write 7 Done Sent Done false
24839 7747 Acked Read 9 Done Sent Done false
24840 7748 Acked Write 8 Done Sent Done false
24841 7749 Acked Read 6 Done Sent Done false
24842 7750 Acked Read 7 Done Sent Done false
24843 7751 Acked Read 8 Done Sent Done false
24844 7752 Acked Read 2 Done Sent Done false
24845 7753 Acked Read 3 Done Sent Done false
24846 7754 Acked Write 10 Done Sent Done false
24847 7755 Acked Write 2 Done Sent Done false
24848 7756 Acked Read 6 Done Sent Done false
24849 7757 Acked Write 6 Done Sent Done false
24850 7758 Acked Flush 0 Done Sent Done false
24851 7759 Acked Write 10 Done Sent Done false
24852 7760 Acked Read 6 Done Sent Done false
24853 7761 Acked Flush 0 Done Sent Done false
24854 7762 Acked Write 8 Done Sent Done false
24855 7763 Acked Read 10 Done Sent Done false
24856 7764 Acked Write 7 Done Sent Done false
24857 7765 Acked Write 1 Done Sent Done false
24858 7766 Acked Write 10 Done Sent Done false
24859 7767 Acked Read 10 Done Sent Done false
24860 7768 Acked Read 4 Done Sent Done false
24861 7769 Acked Read 7 Done Sent Done false
24862 7770 Acked Read 7 Done Sent Done false
24863 7771 Acked Write 9 Done Sent Done false
24864 7772 Acked Write 2 Done Sent Done false
24865 7773 Acked Write 7 Done Sent Done false
24866 7774 Acked Read 10 Done Sent Done false
24867 7775 Acked Read 5 Done Sent Done false
24868 7776 Acked Read 3 Done Sent Done false
24869 7777 Acked Read 3 Done Sent Done false
24870 7778 Acked Flush 0 Done Sent Done false
24871 7779 Acked Read 10 Done Sent Done false
24872 7780 Acked Write 8 Done Sent Done false
24873 7781 Acked Read 3 Done Sent Done false
24874 7782 Acked Flush 0 Done Sent Done false
24875 7783 Acked Read 2 Done Sent Done false
24876 7784 Acked Read 5 Done Sent Done false
24877 7785 Acked Read 3 Done Sent Done false
24878 7786 Acked Flush 0 Done Sent Done false
24879 7787 Acked Flush 0 Done Sent Done false
24880 7788 Acked Read 8 Done Sent Done false
24881 7789 Acked Read 10 Done Sent Done false
24882 7790 Acked Read 6 Done Sent Done false
24883 7791 Acked Write 2 Done Sent Done false
24884 7792 Acked Write 1 Done Sent Done false
24885 7793 Acked Write 9 Done Sent Done false
24886 7794 Acked Write 8 Done Sent Done false
24887 7795 Acked Write 8 Done Sent Done false
24888 7796 Acked Write 9 Done Sent Done false
24889 7797 Acked Read 3 Done Sent Done false
24890 7798 Acked Write 9 Done Sent Done false
24891 7799 Acked Read 2 Done Sent Done false
24892 7800 Acked Read 2 Done Sent Done false
24893 7801 Acked Write 9 Done Sent Done false
24894 7802 Acked Read 6 Done Sent Done false
24895 7803 Acked Read 9 Done Sent Done false
24896 7804 Acked Write 9 Done Sent Done false
24897 7805 Acked Read 7 Done Sent Done false
24898 7806 Acked Read 8 Done Sent Done false
24899 7807 Acked Read 9 Done Sent Done false
24900 7808 Acked Read 4 Done Sent Done false
24901 7809 Acked Read 10 Done Sent Done false
24902 7810 Acked Read 5 Done Sent Done false
24903 7811 Acked Write 10 Done Sent Done false
24904 7812 Acked Write 1 Done Sent Done false
24905 7813 Acked Read 9 Done Sent Done false
24906 7814 Acked Read 4 Done Sent Done false
24907 7815 Acked Write 10 Done Sent Done false
24908 7816 Acked Write 7 Done Sent Done false
24909 7817 Acked Read 7 Done Sent Done false
24910 7818 Acked Read 5 Done Sent Done false
24911 7819 Acked Write 3 Done Sent Done false
24912 7820 Acked Write 1 Done Sent Done false
24913 7821 Acked Read 6 Done Sent Done false
24914 7822 Acked Write 8 Done Sent Done false
24915 7823 Acked Write 3 Done Sent Done false
24916 7824 Acked Read 3 Done Sent Done false
24917 7825 Acked Read 4 Done Sent Done false
24918 7826 Acked Write 8 Done Sent Done false
24919 7827 Acked Read 10 Done Sent Done false
24920 7828 Acked Read 3 Done Sent Done false
24921 7829 Acked Read 2 Done Sent Done false
24922 7830 Acked Read 2 Done Sent Done false
24923 7831 Acked Read 10 Done Sent Done false
24924 7832 Acked Flush 0 Done Sent Done false
24925 7833 Acked Write 10 Done Sent Done false
24926 7834 Acked Read 1 Done Sent Done false
24927 7835 Acked Read 9 Done Sent Done false
24928 7836 Acked Read 1 Done Sent Done false
24929 7837 Acked Read 2 Done Sent Done false
24930 7838 Acked Read 9 Done Sent Done false
24931 7839 Acked Read 4 Done Sent Done false
24932 7840 Acked Write 9 Done Sent Done false
24933 7841 Acked Read 8 Done Sent Done false
24934 7842 Acked Write 8 Done Sent Done false
24935 7843 Acked Write 5 Done Sent Done false
24936 7844 Acked Read 9 Done Sent Done false
24937 7845 Acked Read 1 Done Sent Done false
24938 7846 Acked Read 6 Done Sent Done false
24939 7847 Acked Write 10 Done Sent Done false
24940 7848 Acked Write 10 Done Sent Done false
24941 7849 Acked Read 8 Done Sent Done false
24942 7850 Acked Read 2 Done Sent Done false
24943 7851 Acked Write 8 Done Sent Done false
24944 7852 Acked Read 6 Done Sent Done false
24945 7853 Acked Read 4 Done Sent Done false
24946 7854 Acked Read 4 Done Sent Done false
24947 7855 Acked Write 3 Done Sent Done false
24948 7856 Acked Write 4 Done Sent Done false
24949 7857 Acked Write 6 Done Sent Done false
24950 7858 Acked Read 7 Done Sent Done false
24951 7859 Acked Read 7 Done Sent Done false
24952 7860 Acked Read 4 Done Sent Done false
24953 7861 Acked Flush 0 Done Sent Done false
24954 7862 Acked Read 2 Done Sent Done false
24955 7863 Acked Write 3 Done Sent Done false
24956 7864 Acked Write 1 Done Sent Done false
24957 7865 Acked Read 5 Done Sent Done false
24958 7866 Acked Flush 0 Done Sent Done false
24959 7867 Acked Flush 0 Done Sent Done false
24960 7868 Acked Read 9 Done Sent Done false
24961 7869 Acked Read 8 Done Sent Done false
24962 7870 Acked Read 4 Done Sent Done false
24963 7871 Acked Read 5 Done Sent Done false
24964 7872 Acked Write 9 Done Sent Done false
24965 7873 Acked Write 6 Done Sent Done false
24966 7874 Acked Read 4 Done Sent Done false
24967 7875 Acked Write 4 Done Sent Done false
24968 7876 Acked Write 10 Done Sent Done false
24969 7877 Acked Write 6 Done Sent Done false
24970 7878 Acked Read 1 Done Sent Done false
24971 7879 Acked Write 8 Done Sent Done false
24972 7880 Acked Read 5 Done Sent Done false
24973 7881 Acked Flush 0 Done Sent Done false
24974 7882 Acked Read 2 Done Sent Done false
24975 7883 Acked Write 7 Done Sent Done false
24976 7884 Acked Write 1 Done Sent Done false
24977 7885 Acked Write 9 Done Sent Done false
24978 7886 Acked Read 5 Done Sent Done false
24979 7887 Acked Read 8 Done Sent Done false
24980 7888 Acked Write 6 Done Sent Done false
24981 7889 Acked Write 7 Done Sent Done false
24982 7890 Acked Write 8 Done Sent Done false
24983 7891 Acked Write 8 Done Sent Done false
24984 7892 Acked Read 8 Done Sent Done false
24985 7893 Acked Read 5 Done Sent Done false
24986 7894 Acked Read 1 Done Sent Done false
24987 7895 Acked Read 8 Done Sent Done false
24988 7896 Acked Read 9 Done Sent Done false
24989 7897 Acked Write 10 Done Sent Done false
24990 7898 Acked Write 5 Done Sent Done false
24991 7899 Acked Read 7 Done Sent Done false
24992 7900 Acked Flush 0 Done Sent Done false
24993 7901 Acked Read 2 Done Sent Done false
24994 7902 Acked Read 3 Done Sent Done false
24995 7903 Acked Read 10 Done Sent Done false
24996 7904 Acked Read 4 Done Sent Done false
24997 7905 Acked Write 4 Done Sent Done false
24998 7906 Acked Read 6 Done Sent Done false
24999 7907 Acked Write 4 Done Sent Done false
25000 7908 Acked Read 1 Done Sent Done false
25001 7909 Acked Read 4 Done Sent Done false
25002 7910 Acked Write 10 Done Sent Done false
25003 7911 Acked Write 7 Done Sent Done false
25004 7912 Acked Read 2 Done Sent Done false
25005 7913 Acked Write 5 Done Sent Done false
25006 7914 Acked Read 8 Done Sent Done false
25007 7915 Acked Read 7 Done Sent Done false
25008 7916 Acked Write 3 Done Sent Done false
25009 7917 Acked Read 3 Done Sent Done false
25010 7918 Acked Flush 0 Done Sent Done false
25011 7919 Acked Read 5 Done Sent Done false
25012 7920 Acked Write 5 Done Sent Done false
25013 7921 Acked Read 8 Done Sent Done false
25014 7922 Acked Write 1 Done Sent Done false
25015 7923 Acked Flush 0 Done Sent Done false
25016 7924 Acked Write 7 Done Sent Done false
25017 7925 Acked Write 8 Done Sent Done false
25018 7926 Acked Read 5 Done Sent Done false
25019 7927 Acked Read 8 Done Sent Done false
25020 7928 Acked Write 5 Done Sent Done false
25021 7929 Acked Write 4 Done Sent Done false
25022 7930 Acked Write 4 Done Sent Done false
25023 7931 Acked Write 8 Done Sent Done false
25024 7932 Acked Write 3 Done Sent Done false
25025 7933 Acked Write 5 Done Sent Done false
25026 7934 Acked Read 1 Done Sent Done false
25027 7935 Acked Read 2 Done Sent Done false
25028 7936 Acked Flush 0 Done Sent Done false
25029 7937 Acked Write 4 Done Sent Done false
25030 7938 Acked Write 7 Done Sent Done false
25031 7939 Acked Write 7 Done Sent Done false
25032 7940 Acked Read 10 Done Sent Done false
25033 7941 Acked Flush 0 Done Sent Done false
25034 7942 Acked Read 7 Done Sent Done false
25035 7943 Acked Write 7 Done Sent Done false
25036 7944 Acked Flush 0 Done Sent Done false
25037 STATES DS:0 DS:1 DS:2 TOTAL
25038 Sent 0 302 0 302
25039 Done 302 0 302 604
25040 Skipped 0 0 0 0
25041 Error 0 0 0 0
25042 Last Flush: Some(JobId(7944)) Some(JobId(7642)) Some(JobId(7944))
25043 Downstairs last five completed: 7642 7641 7640 7639 7638
25044 Upstairs last five completed: 7944 7943 7942 7941 7940
25045 CLIENT: Up:0 ds:302 act:2
250462025-01-23T17:43:01.170ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
250472025-01-23T17:43:01.170ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54733) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
250482025-01-23T17:43:01.171ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
250492025-01-23T17:43:01.172ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
250502025-01-23T17:43:01.172ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
250512025-01-23T17:43:01.172ZINFOcrucible: send last flush ID to this DS: Some(JobId(7642)) = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
250522025-01-23T17:43:01.172ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
250532025-01-23T17:43:01.172ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(7642)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
250542025-01-23T17:43:01.172ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(7642)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
25055 ----------------------------------------------------------------
25056 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25057 Downstairs last five completed: 7944 7943 7942 7941 7940
25058 Upstairs last five completed: 7944 7943 7942 7941 7940
25059 CLIENT: Up:0 ds:0 act:3
25060 Replay: All jobs finished
25061 Main task has work to do, go find it
25062 got dsc Stop(2)
25063 stop 2
25064 [8830] Got stop action so:false kr:false
25065 Jan 23 17:43:01.433 INFO request completed, latency_us: 355, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 3d631060-6a67-4c00-98ca-ee91a55102f5, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
250662025-01-23T17:43:01.433ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
250672025-01-23T17:43:01.433ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
250682025-01-23T17:43:01.433ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
25069 Jan 23 17:43:01.433 INFO request completed, latency_us: 167, response_code: 200, uri: /state/cid/2, method: GET, req_id: 53821678-2976-4e0d-891f-236daa06844c, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
250702025-01-23T17:43:01.434ZINFOcrucible: client task is exiting = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
25071 [8830] Exited with: ExitStatus(unix_wait_status(9))
25072 [8830][2] reports Exit
25073 Jan 23 17:43:05.436 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/2, method: GET, req_id: d3d1f2b1-f018-4f41-a282-5f73366cdeab, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
25074 001/300 Read block 2404 len 20480
25075 002/300 Flush
25076 003/300 Read block 2034 len 4096
25077 004/300 Read block 4795 len 16384
25078 005/300 Read block 1528 len 32768
25079 006/300 Write block 2675 len 20480 data: 4 5 4 4 3
25080 007/300 Read block 2760 len 40960
25081 008/300 Write block 2966 len 4096 data: 5
25082 009/300 Write block 1084 len 16384 data: 6 6 6 7
25083 010/300 Write block 4691 len 8192 data: 4 5
25084 011/300 Flush
25085 012/300 Write block 3231 len 16384 data: 5 6 7 8
25086 013/300 Read block 1274 len 16384
25087 014/300 Read block 185 len 24576
25088 015/300 Write block 2780 len 28672 data: 3 3 4 4 3 3 2
25089 016/300 Write block 1902 len 20480 data: 7 6 8 10 9
25090 017/300 Write block 2030 len 8192 data: 2 2
25091 018/300 Flush
25092 019/300 Read block 3510 len 40960
25093 020/300 Write block 2512 len 40960 data: 3 3 4 4 3 3 3 3 4 3
25094 021/300 Read block 1202 len 8192
25095 022/300 Write block 2252 len 20480 data: 7 6 7 7 7
25096 023/300 Write block 827 len 28672 data: 8 7 7 3 3 5 6
25097 024/300 Read block 4345 len 20480
25098 025/300 Read block 4636 len 16384
25099 026/300 Read block 3297 len 36864
25100 027/300 Read block 703 len 12288
25101 028/300 Write block 794 len 40960 data: 5 5 6 8 7 7 5 5 4 4
25102 029/300 Read block 4493 len 36864
25103 030/300 Read block 2604 len 40960
25104 031/300 Flush
25105 032/300 Write block 2663 len 20480 data: 3 3 4 6 6
25106 033/300 Read block 3910 len 36864
25107 034/300 Write block 4261 len 16384 data: 5 5 4 5
25108 035/300 Write block 1036 len 20480 data: 4 2 2 2 2
25109 036/300 Read block 807 len 12288
25110 037/300 Write block 633 len 8192 data: 6 6
25111 038/300 Read block 4577 len 28672
25112 039/300 Write block 12 len 40960 data: 5 6 4 4 4 5 5 5 5 4
25113 040/300 Flush
25114 041/300 Write block 1925 len 28672 data: 11 9 7 5 4 4 3
25115 042/300 Flush
25116 043/300 Write block 316 len 4096 data: 2
25117 044/300 Read block 4963 len 20480
25118 045/300 Write block 4216 len 20480 data: 6 6 4 3 3
25119 046/300 Write block 2448 len 16384 data: 5 5 6 4
25120 047/300 Flush
25121 048/300 Read block 579 len 12288
25122 049/300 Read block 706 len 32768
25123 050/300 Write block 4596 len 24576 data: 4 5 5 6 7 8
25124 051/300 Flush
25125 052/300 Read block 2975 len 8192
25126 053/300 Write block 3593 len 20480 data: 7 6 6 5 4
25127 054/300 Write block 4826 len 24576 data: 3 4 4 4 4 4
25128 055/300 Read block 2551 len 12288
25129 056/300 Flush
25130 057/300 Read block 2431 len 12288
25131 058/300 Read block 2329 len 8192
25132 059/300 Write block 2427 len 20480 data: 4 3 4 3 3
25133 060/300 Write block 4401 len 28672 data: 6 5 4 4 4 5 4
25134 061/300 Write block 3825 len 36864 data: 5 5 6 5 5 4 3 3 3
25135 062/300 Write block 3423 len 32768 data: 7 6 7 7 6 6 9 8
25136 063/300 Read block 4243 len 4096
25137 064/300 Write block 4494 len 8192 data: 3 5
25138 065/300 Write block 959 len 28672 data: 4 5 5 6 4 5 5
25139 066/300 Read block 2695 len 4096
25140 067/300 Write block 46 len 8192 data: 3 4
25141 068/300 Read block 2325 len 40960
25142 069/300 Write block 2413 len 36864 data: 10 8 8 8 9 7 6 6 6
25143 070/300 Read block 3814 len 12288
25144 071/300 Read block 4909 len 40960
25145 072/300 Read block 2874 len 4096
25146 073/300 Write block 460 len 36864 data: 5 8 8 7 9 9 9 7 6
25147 074/300 Write block 886 len 36864 data: 5 5 5 6 5 3 4 4 3
25148 075/300 Write block 1167 len 32768 data: 4 2 3 3 6 6 6 5
25149 076/300 Read block 3666 len 28672
25150 077/300 Read block 3264 len 4096
25151 078/300 Read block 3254 len 40960
25152 079/300 Write block 3651 len 40960 data: 7 8 10 8 8 5 5 4 4 5
25153 080/300 Write block 655 len 12288 data: 7 7 8
25154 081/300 Write block 1565 len 24576 data: 6 6 6 7 6 6
25155 082/300 Read block 4231 len 16384
25156 083/300 Write block 4809 len 28672 data: 4 5 5 6 4 4 4
25157 084/300 Read block 2943 len 12288
25158 085/300 Read block 4266 len 40960
25159 086/300 Read block 1823 len 32768
25160 087/300 Read block 1347 len 4096
25161 088/300 Write block 3121 len 16384 data: 3 2 2 2
25162 089/300 Read block 357 len 28672
25163 090/300 Read block 835 len 24576
25164 091/300 Write block 4904 len 28672 data: 3 3 2 3 4 5 6
25165 092/300 Write block 4649 len 24576 data: 5 5 3 4 4 5
25166 093/300 Write block 4398 len 24576 data: 7 7 7 7 6 5
25167 094/300 Read block 4802 len 36864
25168 095/300 Write block 1397 len 24576 data: 3 4 3 5 5 6
25169 096/300 Write block 3825 len 4096 data: 6
25170 097/300 Read block 4584 len 12288
25171 098/300 Read block 280 len 4096
25172 099/300 Write block 827 len 32768 data: 9 8 8 4 4 6 7 6
25173 100/300 Flush
25174 101/300 Read block 4024 len 24576
25175 102/300 Write block 922 len 24576 data: 6 5 5 4 4 3
25176 103/300 Read block 2839 len 40960
25177 104/300 Read block 4394 len 4096
25178 105/300 Read block 2584 len 16384
25179 106/300 Write block 1171 len 12288 data: 7 7 7
25180 107/300 Write block 1572 len 8192 data: 4 4
25181 108/300 Write block 3666 len 8192 data: 9 10
25182 109/300 Write block 506 len 40960 data: 3 3 3 3 3 3 4 5 4 3
25183 110/300 Read block 2892 len 8192
25184 111/300 Flush
25185 112/300 Write block 4459 len 12288 data: 5 5 6
25186 113/300 Read block 3957 len 20480
25187 114/300 Read block 2103 len 36864
25188 115/300 Flush
25189 116/300 Read block 4337 len 28672
25190 117/300 Read block 4434 len 32768
25191 118/300 Read block 3094 len 4096
25192 119/300 Read block 1986 len 12288
25193 120/300 Write block 690 len 8192 data: 2 2
25194 121/300 Write block 1279 len 32768 data: 8 6 3 4 4 5 5 6
25195 122/300 Read block 2489 len 36864
25196 123/300 Read block 895 len 28672
25197 124/300 Write block 4301 len 32768 data: 4 4 6 6 6 4 3 3
25198 125/300 Flush
25199 126/300 Read block 1146 len 24576
25200 127/300 Write block 3499 len 4096 data: 7
25201 128/300 Read block 4795 len 40960
25202 129/300 Flush
25203 130/300 Write block 3136 len 16384 data: 7 6 6 6
25204 131/300 Read block 2846 len 4096
25205 132/300 Read block 2002 len 16384
25206 133/300 Read block 829 len 20480
25207 134/300 Read block 2037 len 28672
25208 135/300 Flush
25209 136/300 Read block 503 len 4096
25210 137/300 Read block 4124 len 32768
25211 138/300 Write block 4621 len 40960 data: 6 8 8 5 5 5 5 7 7 8
25212 139/300 Write block 4465 len 8192 data: 3 2
25213 140/300 Write block 2997 len 32768 data: 6 5 6 5 5 4 3 2
25214 141/300 Write block 1531 len 24576 data: 5 6 7 6 6 6
25215 142/300 Flush
25216 143/300 Write block 2416 len 36864 data: 9 10 8 7 7 7 4 5 5
25217 144/300 Write block 4308 len 36864 data: 4 2 2 3 3 3 3 4 5
25218 145/300 Read block 4551 len 8192
25219 146/300 Write block 4625 len 24576 data: 6 6 6 8 8 9
25220 147/300 Write block 2056 len 20480 data: 7 7 7 7 7
25221 148/300 Flush
25222 149/300 Read block 2513 len 28672
25223 150/300 Read block 3557 len 12288
25224 151/300 Read block 2487 len 32768
25225 152/300 Read block 2493 len 12288
25226 153/300 Read block 4891 len 28672
25227 154/300 Read block 4564 len 36864
25228 155/300 Read block 3115 len 4096
25229 156/300 Write block 1120 len 16384 data: 4 4 3 2
25230 157/300 Read block 4175 len 8192
25231 158/300 Read block 1672 len 12288
25232 159/300 Write block 2567 len 40960 data: 4 4 3 3 5 4 4 6 6 7
25233 160/300 Write block 2805 len 40960 data: 8 8 6 6 5 6 9 8 9 8
25234 161/300 Read block 2125 len 20480
25235 162/300 Read block 792 len 40960
25236 163/300 Write block 3548 len 40960 data: 7 7 5 6 6 5 5 4 3 3
25237 164/300 Write block 942 len 28672 data: 5 4 5 5 6 5 5
25238 165/300 Write block 2509 len 40960 data: 3 3 3 4 4 5 5 4 4 4
25239 166/300 Write block 1852 len 4096 data: 5
25240 167/300 Read block 2634 len 36864
25241 168/300 Write block 1579 len 4096 data: 3
25242 169/300 Write block 1311 len 20480 data: 9 8 8 7 7
25243 170/300 Read block 4284 len 40960
25244 171/300 Write block 1846 len 28672 data: 4 4 5 6 5 5 6
25245 172/300 Write block 3558 len 40960 data: 5 5 4 3 6 6 7 8 7 6
25246 173/300 Read block 588 len 24576
25247 174/300 Read block 4284 len 36864
25248 175/300 Write block 73 len 4096 data: 10
25249 176/300 Read block 2422 len 28672
25250 177/300 Flush
25251 178/300 Write block 3786 len 28672 data: 4 4 4 4 3 2 3
25252 179/300 Read block 27 len 20480
25253 180/300 Read block 696 len 28672
25254 181/300 Read block 547 len 12288
25255 182/300 Write block 1654 len 12288 data: 6 7 4
25256 183/300 Read block 3460 len 24576
25257 184/300 Read block 760 len 12288
25258 185/300 Read block 3080 len 24576
25259 186/300 Read block 341 len 20480
25260 187/300 Write block 1474 len 24576 data: 6 5 5 7 6 5
25261 188/300 Read block 1281 len 8192
25262 189/300 Write block 2928 len 24576 data: 8 4 5 6 5 4
25263 190/300 Write block 2234 len 16384 data: 7 6 6 4
25264 191/300 Flush
25265 192/300 Read block 3651 len 28672
25266 193/300 Read block 4410 len 24576
25267 194/300 Write block 248 len 16384 data: 4 3 5 5
25268 195/300 Write block 890 len 24576 data: 6 4 5 5 4 3
25269 196/300 Read block 1240 len 8192
25270 197/300 Read block 2073 len 8192
25271 198/300 Write block 4552 len 24576 data: 7 4 5 6 9 9
25272 199/300 Write block 1958 len 24576 data: 5 4 4 7 6 6
25273 200/300 Write block 4939 len 36864 data: 5 5 3 3 5 5 7 6 6
25274 201/300 Write block 3148 len 16384 data: 4 4 4 5
25275 202/300 Read block 4087 len 16384
25276 203/300 Write block 1193 len 8192 data: 5 3
25277 204/300 Read block 1026 len 24576
25278 205/300 Write block 3250 len 8192 data: 5 4
25279 206/300 Read block 1285 len 32768
25280 207/300 Write block 1381 len 28672 data: 2 2 3 4 4 4 4
25281 208/300 Read block 3119 len 28672
25282 209/300 Write block 2466 len 36864 data: 3 3 2 2 3 3 3 4 5
25283 210/300 Read block 315 len 16384
25284 211/300 Read block 1305 len 40960
25285 212/300 Flush
25286 213/300 Write block 806 len 28672 data: 3 3 3 4 5 6 8
25287 214/300 Write block 2849 len 12288 data: 4 4 5
25288 215/300 Read block 3905 len 8192
25289 216/300 Read block 1977 len 36864
25290 217/300 Write block 3460 len 40960 data: 7 7 8 6 3 3 5 6 8 7
25291 218/300 Flush
25292 219/300 Read block 1468 len 28672
25293 220/300 Read block 4687 len 36864
25294 221/300 Read block 4404 len 4096
25295 222/300 Flush
25296 223/300 Write block 1728 len 36864 data: 5 4 5 4 4 5 7 5 5
25297 224/300 Write block 1829 len 8192 data: 6 5
25298 225/300 Flush
25299 226/300 Read block 3694 len 4096
25300 227/300 Read block 1519 len 12288
25301 228/300 Read block 957 len 8192
25302 229/300 Write block 1046 len 40960 data: 4 4 3 4 4 4 5 4 3 3
25303 230/300 Write block 4081 len 20480 data: 4 4 3 4 4
25304 231/300 Write block 1013 len 20480 data: 4 3 3 3 3
25305 232/300 Read block 405 len 4096
25306 233/300 Read block 3172 len 16384
25307 234/300 Read block 3551 len 36864
25308 235/300 Write block 2801 len 20480 data: 6 6 5 6 9
25309 236/300 Write block 4531 len 4096 data: 4
25310 237/300 Read block 4726 len 16384
25311 238/300 Read block 3209 len 12288
25312 239/300 Write block 2314 len 28672 data: 3 3 2 4 4 4 5
25313 240/300 Read block 2722 len 16384
25314 241/300 Read block 2377 len 16384
25315 242/300 Write block 617 len 20480 data: 7 7 7 5 4
25316 243/300 Write block 104 len 20480 data: 5 4 5 6 7
25317 244/300 Write block 1658 len 36864 data: 3 4 4 4 5 5 5 6 5
25318 245/300 Flush
25319 246/300 Write block 385 len 16384 data: 6 5 5 6
25320 247/300 Write block 2957 len 24576 data: 6 5 5 5 5 7
25321 248/300 Read block 2973 len 16384
25322 249/300 Read block 544 len 24576
25323 250/300 Write block 4009 len 4096 data: 5
25324 251/300 Read block 4030 len 16384
25325 252/300 Write block 2683 len 32768 data: 3 7 8 8 8 9 6 6
25326 253/300 Read block 1550 len 12288
25327 254/300 Write block 705 len 4096 data: 4
25328 255/300 Flush
25329 256/300 Write block 1564 len 8192 data: 5 7
25330 257/300 Flush
25331 258/300 Flush
25332 259/300 Read block 4663 len 16384
25333 260/300 Write block 4753 len 32768 data: 4 4 4 3 4 4 6 7
25334 261/300 Write block 1280 len 28672 data: 7 4 5 5 6 6 7
25335 262/300 Flush
25336 263/300 Read block 2051 len 16384
25337 264/300 Write block 3066 len 20480 data: 2 2 3 6 4
25338 265/300 Flush
25339 266/300 Flush
25340 267/300 Flush
25341 268/300 Read block 2612 len 24576
25342 269/300 Read block 3721 len 32768
25343 270/300 Write block 2707 len 32768 data: 4 4 4 5 5 5 5 5
25344 271/300 Read block 338 len 8192
25345 272/300 Read block 763 len 12288
25346 273/300 Write block 2876 len 36864 data: 5 4 5 4 4 5 4 4 4
25347 274/300 Read block 4622 len 20480
25348 275/300 Write block 1604 len 36864 data: 3 2 2 2 4 5 6 7 8
25349 276/300 Read block 4520 len 32768
25350 277/300 Read block 3688 len 36864
25351 278/300 Read block 4009 len 36864
25352 279/300 Write block 2991 len 32768 data: 7 8 5 3 3 5 7 6
25353 280/300 Read block 1877 len 32768
25354 281/300 Read block 2394 len 8192
25355 282/300 Read block 4852 len 32768
25356 283/300 Write block 18 len 8192 data: 6 6
25357 284/300 Write block 830 len 4096 data: 5
25358 285/300 Write block 578 len 16384 data: 4 4 4 5
25359 286/300 Read block 485 len 8192
25360 287/300 Write block 2631 len 4096 data: 7
25361 288/300 Read block 3828 len 20480
25362 289/300 Write block 1755 len 24576 data: 6 7 5 7 7 7
25363 290/300 Read block 473 len 8192
25364 291/300 Read block 1753 len 16384
25365 292/300 Write block 3103 len 32768 data: 6 6 6 7 6 7 6 8
25366 293/300 Read block 4585 len 16384
25367 294/300 Write block 4463 len 40960 data: 4 3 4 3 2 4 3 4 5 5
25368 295/300 Read block 3157 len 36864
25369 296/300 Write block 1633 len 12288 data: 4 4 3
25370 297/300 Write block 47 len 8192 data: 5 4
25371 298/300 Read block 2314 len 32768
25372 299/300 Read block 3231 len 36864
25373 300/300 Read block 245 len 20480
25374 301/300 Write block 733 len 8192 data: 4 5
25375 Main task has work to do, go find it
25376 got dsc Start(2)
25377 start 2
25378 Jan 23 17:43:05.674 INFO request completed, latency_us: 299, response_code: 204, uri: /start/cid/2, method: POST, req_id: b5024c01-8004-4cb5-93f2-f6828f2494e7, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
25379 Replay: started 2, returned:Ok(())
25380 ----------------------------------------------------------------
25381 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25382 States: Active Active Offline
25383 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25384 7945 Acked Read 5 Done Done Sent false
25385 7946 Acked Flush 0 Done Done Sent false
25386 7947 Acked Read 1 Done Done Sent false
25387 7948 Acked Read 4 Done Done Sent false
25388 7949 Acked Read 8 Done Done Sent false
25389 7950 Acked Write 5 Done Done Sent false
25390 7951 Acked Read 10 Done Done Sent false
25391 7952 Acked Write 1 Done Done Sent false
25392 7953 Acked Write 4 Done Done Sent false
25393 7954 Acked Write 2 Done Done Sent false
25394 7955 Acked Flush 0 Done Done Sent false
25395 7956 Acked Write 4 Done Done Sent false
25396 7957 Acked Read 4 Done Done Sent false
25397 7958 Acked Read 6 Done Done Sent false
25398 7959 Acked Write 7 Done Done Sent false
25399 7960 Acked Write 5 Done Done Sent false
25400 7961 Acked Write 2 Done Done Sent false
25401 7962 Acked Flush 0 Done Done Sent false
25402 7963 Acked Read 10 Done Done Sent false
25403 7964 Acked Write 10 Done Done Sent false
25404 7965 Acked Read 2 Done Done Sent false
25405 7966 Acked Write 5 Done Done Sent false
25406 7967 Acked Write 7 Done Done Sent false
25407 7968 Acked Read 5 Done Done Sent false
25408 7969 Acked Read 4 Done Done Sent false
25409 7970 Acked Read 9 Done Done Sent false
25410 7971 Acked Read 3 Done Done Sent false
25411 7972 Acked Write 10 Done Done Sent false
25412 7973 Acked Read 9 Done Done Sent false
25413 7974 Acked Read 10 Done Done Sent false
25414 7975 Acked Flush 0 Done Done Sent false
25415 7976 Acked Write 5 Done Done Sent false
25416 7977 Acked Read 9 Done Done Sent false
25417 7978 Acked Write 4 Done Done Sent false
25418 7979 Acked Write 5 Done Done Sent false
25419 7980 Acked Read 3 Done Done Sent false
25420 7981 Acked Write 2 Done Done Sent false
25421 7982 Acked Read 7 Done Done Sent false
25422 7983 Acked Write 10 Done Done Sent false
25423 7984 Acked Flush 0 Done Done Sent false
25424 7985 Acked Write 7 Done Done Sent false
25425 7986 Acked Flush 0 Done Done Sent false
25426 7987 Acked Write 1 Done Done Sent false
25427 7988 Acked Read 5 Done Done Sent false
25428 7989 Acked Write 5 Done Done Sent false
25429 7990 Acked Write 4 Done Done Sent false
25430 7991 Acked Flush 0 Done Done Sent false
25431 7992 Acked Read 3 Done Done Sent false
25432 7993 Acked Read 8 Done Done Sent false
25433 7994 Acked Write 6 Done Done Sent false
25434 7995 Acked Flush 0 Done Done Sent false
25435 7996 Acked Read 2 Done Done Sent false
25436 7997 Acked Write 5 Done Done Sent false
25437 7998 Acked Write 6 Done Done Sent false
25438 7999 Acked Read 3 Done Done Sent false
25439 8000 Acked Flush 0 Done Done Sent false
25440 8001 Acked Read 3 Done Done Sent false
25441 8002 Acked Read 2 Done Done Sent false
25442 8003 Acked Write 5 Done Done Sent false
25443 8004 Acked Write 7 Done Done Sent false
25444 8005 Acked Write 9 Done Done Sent false
25445 8006 Acked Write 8 Done Done Sent false
25446 8007 Acked Read 1 Done Done Sent false
25447 8008 Acked Write 2 Done Done Sent false
25448 8009 Acked Write 7 Done Done Sent false
25449 8010 Acked Read 1 Done Done Sent false
25450 8011 Acked Write 2 Done Done Sent false
25451 8012 Acked Read 10 Done Done Sent false
25452 8013 Acked Write 9 Done Done Sent false
25453 8014 Acked Read 3 Done Done Sent false
25454 8015 Acked Read 10 Done Done Sent false
25455 8016 Acked Read 1 Done Done Sent false
25456 8017 Acked Write 9 Done Done Sent false
25457 8018 Acked Write 9 Done Done Sent false
25458 8019 Acked Write 8 Done Done Sent false
25459 8020 Acked Read 7 Done Done Sent false
25460 8021 Acked Read 1 Done Done Sent false
25461 8022 Acked Read 10 Done Done Sent false
25462 8023 Acked Write 10 Done Done Sent false
25463 8024 Acked Write 3 Done Done Sent false
25464 8025 Acked Write 6 Done Done Sent false
25465 8026 Acked Read 4 Done Done Sent false
25466 8027 Acked Write 7 Done Done Sent false
25467 8028 Acked Read 3 Done Done Sent false
25468 8029 Acked Read 10 Done Done Sent false
25469 8030 Acked Read 8 Done Done Sent false
25470 8031 Acked Read 1 Done Done Sent false
25471 8032 Acked Write 4 Done Done Sent false
25472 8033 Acked Read 7 Done Done Sent false
25473 8034 Acked Read 6 Done Done Sent false
25474 8035 Acked Write 7 Done Done Sent false
25475 8036 Acked Write 6 Done Done Sent false
25476 8037 Acked Write 6 Done Done Sent false
25477 8038 Acked Read 9 Done Done Sent false
25478 8039 Acked Write 6 Done Done Sent false
25479 8040 Acked Write 1 Done Done Sent false
25480 8041 Acked Read 3 Done Done Sent false
25481 8042 Acked Read 1 Done Done Sent false
25482 8043 Acked Write 8 Done Done Sent false
25483 8044 Acked Flush 0 Done Done Sent false
25484 8045 Acked Read 6 Done Done Sent false
25485 8046 Acked Write 6 Done Done Sent false
25486 8047 Acked Read 10 Done Done Sent false
25487 8048 Acked Read 1 Done Done Sent false
25488 8049 Acked Read 4 Done Done Sent false
25489 8050 Acked Write 3 Done Done Sent false
25490 8051 Acked Write 2 Done Done Sent false
25491 8052 Acked Write 2 Done Done Sent false
25492 8053 Acked Write 10 Done Done Sent false
25493 8054 Acked Read 2 Done Done Sent false
25494 8055 Acked Flush 0 Done Done Sent false
25495 8056 Acked Write 3 Done Done Sent false
25496 8057 Acked Read 5 Done Done Sent false
25497 8058 Acked Read 9 Done Done Sent false
25498 8059 Acked Flush 0 Done Done Sent false
25499 8060 Acked Read 7 Done Done Sent false
25500 8061 Acked Read 8 Done Done Sent false
25501 8062 Acked Read 1 Done Done Sent false
25502 8063 Acked Read 3 Done Done Sent false
25503 8064 Acked Write 2 Done Done Sent false
25504 8065 Acked Write 8 Done Done Sent false
25505 8066 Acked Read 9 Done Done Sent false
25506 8067 Acked Read 7 Done Done Sent false
25507 8068 Acked Write 8 Done Done Sent false
25508 8069 Acked Flush 0 Done Done Sent false
25509 8070 Acked Read 6 Done Done Sent false
25510 8071 Acked Write 1 Done Done Sent false
25511 8072 Acked Read 10 Done Done Sent false
25512 8073 Acked Flush 0 Done Done Sent false
25513 8074 Acked Write 4 Done Done Sent false
25514 8075 Acked Read 1 Done Done Sent false
25515 8076 Acked Read 4 Done Done Sent false
25516 8077 Acked Read 5 Done Done Sent false
25517 8078 Acked Read 7 Done Done Sent false
25518 8079 Acked Flush 0 Done Done Sent false
25519 8080 Acked Read 1 Done Done Sent false
25520 8081 Acked Read 8 Done Done Sent false
25521 8082 Acked Write 10 Done Done Sent false
25522 8083 Acked Write 2 Done Done Sent false
25523 8084 Acked Write 8 Done Done Sent false
25524 8085 Acked Write 6 Done Done Sent false
25525 8086 Acked Flush 0 Done Done Sent false
25526 8087 Acked Write 9 Done Done Sent false
25527 8088 Acked Write 9 Done Done Sent false
25528 8089 Acked Read 2 Done Done Sent false
25529 8090 Acked Write 6 Done Done Sent false
25530 8091 Acked Write 5 Done Done Sent false
25531 8092 Acked Flush 0 Done Done Sent false
25532 8093 Acked Read 7 Done Done Sent false
25533 8094 Acked Read 3 Done Done Sent false
25534 8095 Acked Read 8 Done Done Sent false
25535 8096 Acked Read 3 Done Done Sent false
25536 8097 Acked Read 7 Done Done Sent false
25537 8098 Acked Read 9 Done Done Sent false
25538 8099 Acked Read 1 Done Done Sent false
25539 8100 Acked Write 4 Done Done Sent false
25540 8101 Acked Read 2 Done Done Sent false
25541 8102 Acked Read 3 Done Done Sent false
25542 8103 Acked Write 10 Done Done Sent false
25543 8104 Acked Write 10 Done Done Sent false
25544 8105 Acked Read 5 Done Done Sent false
25545 8106 Acked Read 10 Done Done Sent false
25546 8107 Acked Write 10 Done Done Sent false
25547 8108 Acked Write 7 Done Done Sent false
25548 8109 Acked Write 10 Done Done Sent false
25549 8110 Acked Write 1 Done Done Sent false
25550 8111 Acked Read 9 Done Done Sent false
25551 8112 Acked Write 1 Done Done Sent false
25552 8113 Acked Write 5 Done Done Sent false
25553 8114 Acked Read 10 Done Done Sent false
25554 8115 Acked Write 7 Done Done Sent false
25555 8116 Acked Write 10 Done Done Sent false
25556 8117 Acked Read 6 Done Done Sent false
25557 8118 Acked Read 9 Done Done Sent false
25558 8119 Acked Write 1 Done Done Sent false
25559 8120 Acked Read 7 Done Done Sent false
25560 8121 Acked Flush 0 Done Done Sent false
25561 8122 Acked Write 7 Done Done Sent false
25562 8123 Acked Read 5 Done Done Sent false
25563 8124 Acked Read 7 Done Done Sent false
25564 8125 Acked Read 3 Done Done Sent false
25565 8126 Acked Write 3 Done Done Sent false
25566 8127 Acked Read 6 Done Done Sent false
25567 8128 Acked Read 3 Done Done Sent false
25568 8129 Acked Read 6 Done Done Sent false
25569 8130 Acked Read 5 Done Done Sent false
25570 8131 Acked Write 6 Done Done Sent false
25571 8132 Acked Read 2 Done Done Sent false
25572 8133 Acked Write 6 Done Done Sent false
25573 8134 Acked Write 4 Done Done Sent false
25574 8135 Acked Flush 0 Done Done Sent false
25575 8136 Acked Read 7 Done Done Sent false
25576 8137 Acked Read 6 Done Done Sent false
25577 8138 Acked Write 4 Done Done Sent false
25578 8139 Acked Write 6 Done Done Sent false
25579 8140 Acked Read 2 Done Done Sent false
25580 8141 Acked Read 2 Done Done Sent false
25581 8142 Acked Write 6 Done Done Sent false
25582 8143 Acked Write 6 Done Done Sent false
25583 8144 Acked Write 9 Done Done Sent false
25584 8145 Acked Write 4 Done Done Sent false
25585 8146 Acked Read 4 Done Done Sent false
25586 8147 Acked Write 2 Done Done Sent false
25587 8148 Acked Read 6 Done Done Sent false
25588 8149 Acked Write 2 Done Done Sent false
25589 8150 Acked Read 8 Done Done Sent false
25590 8151 Acked Write 7 Done Done Sent false
25591 8152 Acked Read 7 Done Done Sent false
25592 8153 Acked Write 9 Done Done Sent false
25593 8154 Acked Read 4 Done Done Sent false
25594 8155 Acked Read 10 Done Done Sent false
25595 8156 Acked Flush 0 Done Done Sent false
25596 8157 Acked Write 7 Done Done Sent false
25597 8158 Acked Write 3 Done Done Sent false
25598 8159 Acked Read 2 Done Done Sent false
25599 8160 Acked Read 9 Done Done Sent false
25600 8161 Acked Write 10 Done Done Sent false
25601 8162 Acked Flush 0 Done Done Sent false
25602 8163 Acked Read 7 Done Done Sent false
25603 8164 Acked Read 9 Done Done Sent false
25604 8165 Acked Read 1 Done Done Sent false
25605 8166 Acked Flush 0 Done Done Sent false
25606 8167 Acked Write 9 Done Done Sent false
25607 8168 Acked Write 2 Done Done Sent false
25608 8169 Acked Flush 0 Done Done Sent false
25609 8170 Acked Read 1 Done Done Sent false
25610 8171 Acked Read 3 Done Done Sent false
25611 8172 Acked Read 2 Done Done Sent false
25612 8173 Acked Write 10 Done Done Sent false
25613 8174 Acked Write 5 Done Done Sent false
25614 8175 Acked Write 5 Done Done Sent false
25615 8176 Acked Read 1 Done Done Sent false
25616 8177 Acked Read 4 Done Done Sent false
25617 8178 Acked Read 9 Done Done Sent false
25618 8179 Acked Write 5 Done Done Sent false
25619 8180 Acked Write 1 Done Done Sent false
25620 8181 Acked Read 4 Done Done Sent false
25621 8182 Acked Read 3 Done Done Sent false
25622 8183 Acked Write 7 Done Done Sent false
25623 8184 Acked Read 4 Done Done Sent false
25624 8185 Acked Read 4 Done Done Sent false
25625 8186 Acked Write 5 Done Done Sent false
25626 8187 Acked Write 5 Done Done Sent false
25627 8188 Acked Write 9 Done Done Sent false
25628 8189 Acked Flush 0 Done Done Sent false
25629 8190 Acked Write 4 Done Done Sent false
25630 8191 Acked Write 6 Done Done Sent false
25631 8192 Acked Read 4 Done Done Sent false
25632 8193 Acked Read 6 Done Done Sent false
25633 8194 Acked Write 1 Done Done Sent false
25634 8195 Acked Read 4 Done Done Sent false
25635 8196 Acked Write 8 Done Done Sent false
25636 8197 Acked Read 3 Done Done Sent false
25637 8198 Acked Write 1 Done Done Sent false
25638 8199 Acked Flush 0 Done Done Sent false
25639 8200 Acked Write 2 Done Done Sent false
25640 8201 Acked Flush 0 Done Done Sent false
25641 8202 Acked Flush 0 Done Done Sent false
25642 8203 Acked Read 4 Done Done Sent false
25643 8204 Acked Write 8 Done Done Sent false
25644 8205 Acked Write 7 Done Done Sent false
25645 8206 Acked Flush 0 Done Done Sent false
25646 8207 Acked Read 4 Done Done Sent false
25647 8208 Acked Write 5 Done Done Sent false
25648 8209 Acked Flush 0 Done Done Sent false
25649 8210 Acked Flush 0 Done Done Sent false
25650 8211 Acked Flush 0 Done Done Sent false
25651 8212 Acked Read 6 Done Done Sent false
25652 8213 Acked Read 8 Done Done Sent false
25653 8214 Acked Write 8 Done Done Sent false
25654 8215 Acked Read 2 Done Done Sent false
25655 8216 Acked Read 3 Done Done Sent false
25656 8217 Acked Write 9 Done Done Sent false
25657 8218 Acked Read 5 Done Done Sent false
25658 8219 Acked Write 9 Done Done Sent false
25659 8220 Acked Read 8 Done Done Sent false
25660 8221 Acked Read 9 Done Done Sent false
25661 8222 Acked Read 9 Done Done Sent false
25662 8223 Acked Write 8 Done Done Sent false
25663 8224 Acked Read 8 Done Done Sent false
25664 8225 Acked Read 2 Done Done Sent false
25665 8226 Acked Read 8 Done Done Sent false
25666 8227 Acked Write 2 Done Done Sent false
25667 8228 Acked Write 1 Done Done Sent false
25668 8229 Acked Write 4 Done Done Sent false
25669 8230 Acked Read 2 Done Done Sent false
25670 8231 Acked Write 1 Done Done Sent false
25671 8232 Acked Read 5 Done Done Sent false
25672 8233 Acked Write 6 Done Done Sent false
25673 8234 Acked Read 2 Done Done Sent false
25674 8235 Acked Read 4 Done Done Sent false
25675 8236 Acked Write 8 Done Done Sent false
25676 8237 Acked Read 4 Done Done Sent false
25677 8238 Acked Write 10 Done Done Sent false
25678 8239 Acked Read 9 Done Done Sent false
25679 8240 Acked Write 3 Done Done Sent false
25680 8241 Acked Write 2 Done Done Sent false
25681 8242 Acked Read 8 Done Done Sent false
25682 8243 Acked Read 9 Done Done Sent false
25683 8244 Acked Read 5 Done Done Sent false
25684 8245 Acked Write 2 Done Done Sent false
25685 STATES DS:0 DS:1 DS:2 TOTAL
25686 Sent 0 0 301 301
25687 Done 301 301 0 602
25688 Skipped 0 0 0 0
25689 Error 0 0 0 0
25690 Last Flush: Some(JobId(8211)) Some(JobId(8211)) Some(JobId(7944))
25691 Downstairs last five completed: 7944 7943 7942 7941 7940
25692 Upstairs last five completed: 8245 8244 8243 8242 8241
25693 CLIENT: Up:0 ds:301 act:2
25694 [8830] got start action
25695 [8830] I am going to restart
25696 Starting downstairs at port 8830
25697 Make output file at "/tmp/dsc/downstairs-8830.txt"
25698 [8830][2] reports Starting
25699 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1229)
25700 [8830][2] reports Running
25701 ----------------------------------------------------------------
25702 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
25703 States: Active Active Offline
25704 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25705 7945 Acked Read 5 Done Done Sent false
25706 7946 Acked Flush 0 Done Done Sent false
25707 7947 Acked Read 1 Done Done Sent false
25708 7948 Acked Read 4 Done Done Sent false
25709 7949 Acked Read 8 Done Done Sent false
25710 7950 Acked Write 5 Done Done Sent false
25711 7951 Acked Read 10 Done Done Sent false
25712 7952 Acked Write 1 Done Done Sent false
25713 7953 Acked Write 4 Done Done Sent false
25714 7954 Acked Write 2 Done Done Sent false
25715 7955 Acked Flush 0 Done Done Sent false
25716 7956 Acked Write 4 Done Done Sent false
25717 7957 Acked Read 4 Done Done Sent false
25718 7958 Acked Read 6 Done Done Sent false
25719 7959 Acked Write 7 Done Done Sent false
25720 7960 Acked Write 5 Done Done Sent false
25721 7961 Acked Write 2 Done Done Sent false
25722 7962 Acked Flush 0 Done Done Sent false
25723 7963 Acked Read 10 Done Done Sent false
25724 7964 Acked Write 10 Done Done Sent false
25725 7965 Acked Read 2 Done Done Sent false
25726 7966 Acked Write 5 Done Done Sent false
25727 7967 Acked Write 7 Done Done Sent false
25728 7968 Acked Read 5 Done Done Sent false
25729 7969 Acked Read 4 Done Done Sent false
25730 7970 Acked Read 9 Done Done Sent false
25731 7971 Acked Read 3 Done Done Sent false
25732 7972 Acked Write 10 Done Done Sent false
25733 7973 Acked Read 9 Done Done Sent false
25734 7974 Acked Read 10 Done Done Sent false
25735 7975 Acked Flush 0 Done Done Sent false
25736 7976 Acked Write 5 Done Done Sent false
25737 7977 Acked Read 9 Done Done Sent false
25738 7978 Acked Write 4 Done Done Sent false
25739 7979 Acked Write 5 Done Done Sent false
25740 7980 Acked Read 3 Done Done Sent false
25741 7981 Acked Write 2 Done Done Sent false
25742 7982 Acked Read 7 Done Done Sent false
25743 7983 Acked Write 10 Done Done Sent false
25744 7984 Acked Flush 0 Done Done Sent false
25745 7985 Acked Write 7 Done Done Sent false
25746 7986 Acked Flush 0 Done Done Sent false
25747 7987 Acked Write 1 Done Done Sent false
25748 7988 Acked Read 5 Done Done Sent false
25749 7989 Acked Write 5 Done Done Sent false
25750 7990 Acked Write 4 Done Done Sent false
25751 7991 Acked Flush 0 Done Done Sent false
25752 7992 Acked Read 3 Done Done Sent false
25753 7993 Acked Read 8 Done Done Sent false
25754 7994 Acked Write 6 Done Done Sent false
25755 7995 Acked Flush 0 Done Done Sent false
25756 7996 Acked Read 2 Done Done Sent false
25757 7997 Acked Write 5 Done Done Sent false
25758 7998 Acked Write 6 Done Done Sent false
25759 7999 Acked Read 3 Done Done Sent false
25760 8000 Acked Flush 0 Done Done Sent false
25761 8001 Acked Read 3 Done Done Sent false
25762 8002 Acked Read 2 Done Done Sent false
25763 8003 Acked Write 5 Done Done Sent false
25764 8004 Acked Write 7 Done Done Sent false
25765 8005 Acked Write 9 Done Done Sent false
25766 8006 Acked Write 8 Done Done Sent false
25767 8007 Acked Read 1 Done Done Sent false
25768 8008 Acked Write 2 Done Done Sent false
25769 8009 Acked Write 7 Done Done Sent false
25770 8010 Acked Read 1 Done Done Sent false
25771 8011 Acked Write 2 Done Done Sent false
25772 8012 Acked Read 10 Done Done Sent false
25773 8013 Acked Write 9 Done Done Sent false
25774 8014 Acked Read 3 Done Done Sent false
25775 8015 Acked Read 10 Done Done Sent false
25776 8016 Acked Read 1 Done Done Sent false
25777 8017 Acked Write 9 Done Done Sent false
25778 8018 Acked Write 9 Done Done Sent false
25779 8019 Acked Write 8 Done Done Sent false
25780 8020 Acked Read 7 Done Done Sent false
25781 8021 Acked Read 1 Done Done Sent false
25782 8022 Acked Read 10 Done Done Sent false
25783 8023 Acked Write 10 Done Done Sent false
25784 8024 Acked Write 3 Done Done Sent false
25785 8025 Acked Write 6 Done Done Sent false
25786 8026 Acked Read 4 Done Done Sent false
25787 8027 Acked Write 7 Done Done Sent false
25788 8028 Acked Read 3 Done Done Sent false
25789 8029 Acked Read 10 Done Done Sent false
25790 8030 Acked Read 8 Done Done Sent false
25791 8031 Acked Read 1 Done Done Sent false
25792 8032 Acked Write 4 Done Done Sent false
25793 8033 Acked Read 7 Done Done Sent false
25794 8034 Acked Read 6 Done Done Sent false
25795 8035 Acked Write 7 Done Done Sent false
25796 8036 Acked Write 6 Done Done Sent false
25797 8037 Acked Write 6 Done Done Sent false
25798 8038 Acked Read 9 Done Done Sent false
25799 8039 Acked Write 6 Done Done Sent false
25800 8040 Acked Write 1 Done Done Sent false
25801 8041 Acked Read 3 Done Done Sent false
25802 8042 Acked Read 1 Done Done Sent false
25803 8043 Acked Write 8 Done Done Sent false
25804 8044 Acked Flush 0 Done Done Sent false
25805 8045 Acked Read 6 Done Done Sent false
25806 8046 Acked Write 6 Done Done Sent false
25807 8047 Acked Read 10 Done Done Sent false
25808 8048 Acked Read 1 Done Done Sent false
25809 8049 Acked Read 4 Done Done Sent false
25810 8050 Acked Write 3 Done Done Sent false
25811 8051 Acked Write 2 Done Done Sent false
25812 8052 Acked Write 2 Done Done Sent false
25813 8053 Acked Write 10 Done Done Sent false
25814 8054 Acked Read 2 Done Done Sent false
25815 8055 Acked Flush 0 Done Done Sent false
25816 8056 Acked Write 3 Done Done Sent false
25817 8057 Acked Read 5 Done Done Sent false
25818 8058 Acked Read 9 Done Done Sent false
25819 8059 Acked Flush 0 Done Done Sent false
25820 8060 Acked Read 7 Done Done Sent false
25821 8061 Acked Read 8 Done Done Sent false
25822 8062 Acked Read 1 Done Done Sent false
25823 8063 Acked Read 3 Done Done Sent false
25824 8064 Acked Write 2 Done Done Sent false
25825 8065 Acked Write 8 Done Done Sent false
25826 8066 Acked Read 9 Done Done Sent false
25827 8067 Acked Read 7 Done Done Sent false
25828 8068 Acked Write 8 Done Done Sent false
25829 8069 Acked Flush 0 Done Done Sent false
25830 8070 Acked Read 6 Done Done Sent false
25831 8071 Acked Write 1 Done Done Sent false
25832 8072 Acked Read 10 Done Done Sent false
25833 8073 Acked Flush 0 Done Done Sent false
25834 8074 Acked Write 4 Done Done Sent false
25835 8075 Acked Read 1 Done Done Sent false
25836 8076 Acked Read 4 Done Done Sent false
25837 8077 Acked Read 5 Done Done Sent false
25838 8078 Acked Read 7 Done Done Sent false
25839 8079 Acked Flush 0 Done Done Sent false
25840 8080 Acked Read 1 Done Done Sent false
25841 8081 Acked Read 8 Done Done Sent false
25842 8082 Acked Write 10 Done Done Sent false
25843 8083 Acked Write 2 Done Done Sent false
25844 8084 Acked Write 8 Done Done Sent false
25845 8085 Acked Write 6 Done Done Sent false
25846 8086 Acked Flush 0 Done Done Sent false
25847 8087 Acked Write 9 Done Done Sent false
25848 8088 Acked Write 9 Done Done Sent false
25849 8089 Acked Read 2 Done Done Sent false
25850 8090 Acked Write 6 Done Done Sent false
25851 8091 Acked Write 5 Done Done Sent false
25852 8092 Acked Flush 0 Done Done Sent false
25853 8093 Acked Read 7 Done Done Sent false
25854 8094 Acked Read 3 Done Done Sent false
25855 8095 Acked Read 8 Done Done Sent false
25856 8096 Acked Read 3 Done Done Sent false
25857 8097 Acked Read 7 Done Done Sent false
25858 8098 Acked Read 9 Done Done Sent false
25859 8099 Acked Read 1 Done Done Sent false
25860 8100 Acked Write 4 Done Done Sent false
25861 8101 Acked Read 2 Done Done Sent false
25862 8102 Acked Read 3 Done Done Sent false
25863 8103 Acked Write 10 Done Done Sent false
25864 8104 Acked Write 10 Done Done Sent false
25865 8105 Acked Read 5 Done Done Sent false
25866 8106 Acked Read 10 Done Done Sent false
25867 8107 Acked Write 10 Done Done Sent false
25868 8108 Acked Write 7 Done Done Sent false
25869 8109 Acked Write 10 Done Done Sent false
25870 8110 Acked Write 1 Done Done Sent false
25871 8111 Acked Read 9 Done Done Sent false
25872 8112 Acked Write 1 Done Done Sent false
25873 8113 Acked Write 5 Done Done Sent false
25874 8114 Acked Read 10 Done Done Sent false
25875 8115 Acked Write 7 Done Done Sent false
25876 8116 Acked Write 10 Done Done Sent false
25877 8117 Acked Read 6 Done Done Sent false
25878 8118 Acked Read 9 Done Done Sent false
25879 8119 Acked Write 1 Done Done Sent false
25880 8120 Acked Read 7 Done Done Sent false
25881 8121 Acked Flush 0 Done Done Sent false
25882 8122 Acked Write 7 Done Done Sent false
25883 8123 Acked Read 5 Done Done Sent false
25884 8124 Acked Read 7 Done Done Sent false
25885 8125 Acked Read 3 Done Done Sent false
25886 8126 Acked Write 3 Done Done Sent false
25887 8127 Acked Read 6 Done Done Sent false
25888 8128 Acked Read 3 Done Done Sent false
25889 8129 Acked Read 6 Done Done Sent false
25890 8130 Acked Read 5 Done Done Sent false
25891 8131 Acked Write 6 Done Done Sent false
25892 8132 Acked Read 2 Done Done Sent false
25893 8133 Acked Write 6 Done Done Sent false
25894 8134 Acked Write 4 Done Done Sent false
25895 8135 Acked Flush 0 Done Done Sent false
25896 8136 Acked Read 7 Done Done Sent false
25897 8137 Acked Read 6 Done Done Sent false
25898 8138 Acked Write 4 Done Done Sent false
25899 8139 Acked Write 6 Done Done Sent false
25900 8140 Acked Read 2 Done Done Sent false
25901 8141 Acked Read 2 Done Done Sent false
25902 8142 Acked Write 6 Done Done Sent false
25903 8143 Acked Write 6 Done Done Sent false
25904 8144 Acked Write 9 Done Done Sent false
25905 8145 Acked Write 4 Done Done Sent false
25906 8146 Acked Read 4 Done Done Sent false
25907 8147 Acked Write 2 Done Done Sent false
25908 8148 Acked Read 6 Done Done Sent false
25909 8149 Acked Write 2 Done Done Sent false
25910 8150 Acked Read 8 Done Done Sent false
25911 8151 Acked Write 7 Done Done Sent false
25912 8152 Acked Read 7 Done Done Sent false
25913 8153 Acked Write 9 Done Done Sent false
25914 8154 Acked Read 4 Done Done Sent false
25915 8155 Acked Read 10 Done Done Sent false
25916 8156 Acked Flush 0 Done Done Sent false
25917 8157 Acked Write 7 Done Done Sent false
25918 8158 Acked Write 3 Done Done Sent false
25919 8159 Acked Read 2 Done Done Sent false
25920 8160 Acked Read 9 Done Done Sent false
25921 8161 Acked Write 10 Done Done Sent false
25922 8162 Acked Flush 0 Done Done Sent false
25923 8163 Acked Read 7 Done Done Sent false
25924 8164 Acked Read 9 Done Done Sent false
25925 8165 Acked Read 1 Done Done Sent false
25926 8166 Acked Flush 0 Done Done Sent false
25927 8167 Acked Write 9 Done Done Sent false
25928 8168 Acked Write 2 Done Done Sent false
25929 8169 Acked Flush 0 Done Done Sent false
25930 8170 Acked Read 1 Done Done Sent false
25931 8171 Acked Read 3 Done Done Sent false
25932 8172 Acked Read 2 Done Done Sent false
25933 8173 Acked Write 10 Done Done Sent false
25934 8174 Acked Write 5 Done Done Sent false
25935 8175 Acked Write 5 Done Done Sent false
25936 8176 Acked Read 1 Done Done Sent false
25937 8177 Acked Read 4 Done Done Sent false
25938 8178 Acked Read 9 Done Done Sent false
25939 8179 Acked Write 5 Done Done Sent false
25940 8180 Acked Write 1 Done Done Sent false
25941 8181 Acked Read 4 Done Done Sent false
25942 8182 Acked Read 3 Done Done Sent false
25943 8183 Acked Write 7 Done Done Sent false
25944 8184 Acked Read 4 Done Done Sent false
25945 8185 Acked Read 4 Done Done Sent false
25946 8186 Acked Write 5 Done Done Sent false
25947 8187 Acked Write 5 Done Done Sent false
25948 8188 Acked Write 9 Done Done Sent false
25949 8189 Acked Flush 0 Done Done Sent false
25950 8190 Acked Write 4 Done Done Sent false
25951 8191 Acked Write 6 Done Done Sent false
25952 8192 Acked Read 4 Done Done Sent false
25953 8193 Acked Read 6 Done Done Sent false
25954 8194 Acked Write 1 Done Done Sent false
25955 8195 Acked Read 4 Done Done Sent false
25956 8196 Acked Write 8 Done Done Sent false
25957 8197 Acked Read 3 Done Done Sent false
25958 8198 Acked Write 1 Done Done Sent false
25959 8199 Acked Flush 0 Done Done Sent false
25960 8200 Acked Write 2 Done Done Sent false
25961 8201 Acked Flush 0 Done Done Sent false
25962 8202 Acked Flush 0 Done Done Sent false
25963 8203 Acked Read 4 Done Done Sent false
25964 8204 Acked Write 8 Done Done Sent false
25965 8205 Acked Write 7 Done Done Sent false
25966 8206 Acked Flush 0 Done Done Sent false
25967 8207 Acked Read 4 Done Done Sent false
25968 8208 Acked Write 5 Done Done Sent false
25969 8209 Acked Flush 0 Done Done Sent false
25970 8210 Acked Flush 0 Done Done Sent false
25971 8211 Acked Flush 0 Done Done Sent false
25972 8212 Acked Read 6 Done Done Sent false
25973 8213 Acked Read 8 Done Done Sent false
25974 8214 Acked Write 8 Done Done Sent false
25975 8215 Acked Read 2 Done Done Sent false
25976 8216 Acked Read 3 Done Done Sent false
25977 8217 Acked Write 9 Done Done Sent false
25978 8218 Acked Read 5 Done Done Sent false
25979 8219 Acked Write 9 Done Done Sent false
25980 8220 Acked Read 8 Done Done Sent false
25981 8221 Acked Read 9 Done Done Sent false
25982 8222 Acked Read 9 Done Done Sent false
25983 8223 Acked Write 8 Done Done Sent false
25984 8224 Acked Read 8 Done Done Sent false
25985 8225 Acked Read 2 Done Done Sent false
25986 8226 Acked Read 8 Done Done Sent false
25987 8227 Acked Write 2 Done Done Sent false
25988 8228 Acked Write 1 Done Done Sent false
25989 8229 Acked Write 4 Done Done Sent false
25990 8230 Acked Read 2 Done Done Sent false
25991 8231 Acked Write 1 Done Done Sent false
25992 8232 Acked Read 5 Done Done Sent false
25993 8233 Acked Write 6 Done Done Sent false
25994 8234 Acked Read 2 Done Done Sent false
25995 8235 Acked Read 4 Done Done Sent false
25996 8236 Acked Write 8 Done Done Sent false
25997 8237 Acked Read 4 Done Done Sent false
25998 8238 Acked Write 10 Done Done Sent false
25999 8239 Acked Read 9 Done Done Sent false
26000 8240 Acked Write 3 Done Done Sent false
26001 8241 Acked Write 2 Done Done Sent false
26002 8242 Acked Read 8 Done Done Sent false
26003 8243 Acked Read 9 Done Done Sent false
26004 8244 Acked Read 5 Done Done Sent false
26005 8245 Acked Write 2 Done Done Sent false
26006 8246 Acked Flush 0 Done Done Sent false
26007 STATES DS:0 DS:1 DS:2 TOTAL
26008 Sent 0 0 302 302
26009 Done 302 302 0 604
26010 Skipped 0 0 0 0
26011 Error 0 0 0 0
26012 Last Flush: Some(JobId(8246)) Some(JobId(8246)) Some(JobId(7944))
26013 Downstairs last five completed: 7944 7943 7942 7941 7940
26014 Upstairs last five completed: 8246 8245 8244 8243 8242
26015 CLIENT: Up:0 ds:302 act:2
260162025-01-23T17:43:11.435ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
260172025-01-23T17:43:11.435ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:45179) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
260182025-01-23T17:43:11.436ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
260192025-01-23T17:43:11.437ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
260202025-01-23T17:43:11.437ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
260212025-01-23T17:43:11.437ZINFOcrucible: send last flush ID to this DS: Some(JobId(7944)) = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
260222025-01-23T17:43:11.437ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
260232025-01-23T17:43:11.437ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(7944)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
260242025-01-23T17:43:11.437ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(7944)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
26025 ----------------------------------------------------------------
26026 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26027 Downstairs last five completed: 8246 8245 8244 8243 8242
26028 Upstairs last five completed: 8246 8245 8244 8243 8242
26029 CLIENT: Up:0 ds:0 act:3
26030 Replay: All jobs finished
26031 Main task has work to do, go find it
26032 got dsc Stop(0)
26033 stop 0
26034 Jan 23 17:43:13.679 [8810] Got stop action so:false kr:false
26035 INFO request completed, latency_us: 346, response_code: 204, uri: /stop/cid/0, method: POST, req_id: c87c3ce3-d78c-4b16-9164-25814d5ea92f, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
260362025-01-23T17:43:13.680ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
260372025-01-23T17:43:13.680ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
260382025-01-23T17:43:13.680ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
26039 Jan 23 17:43:13.680 INFO request completed, latency_us: 118, response_code: 200, uri: /state/cid/0, method: GET, req_id: 723c80d7-bee1-45bc-8245-504074ab3edf, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
260402025-01-23T17:43:13.680ZINFOcrucible: client task is exiting = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
26041 [8810] Exited with: ExitStatus(unix_wait_status(9))
26042 [8810][0] reports Exit
26043 Jan 23 17:43:17.682 INFO request completed, latency_us: 247, response_code: 200, uri: /state/cid/0, method: GET, req_id: f8fa477b-1d26-4728-a73f-6f804dde5258, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
26044 001/300 Read block 2132 len 32768
26045 002/300 Write block 3207 len 8192 data: 4 5
26046 003/300 Write block 2337 len 36864 data: 5 5 4 5 6 5 5 5 5
26047 004/300 Write block 555 len 8192 data: 7 9
26048 005/300 Read block 436 len 12288
26049 006/300 Read block 1740 len 16384
26050 007/300 Write block 575 len 24576 data: 3 3 3 5 5 5
26051 008/300 Read block 1006 len 24576
26052 009/300 Flush
26053 010/300 Write block 289 len 28672 data: 7 7 4 4 7 9 10
26054 011/300 Read block 2016 len 16384
26055 012/300 Read block 2722 len 12288
26056 013/300 Flush
26057 014/300 Flush
26058 015/300 Read block 3066 len 16384
26059 016/300 Write block 2118 len 4096 data: 5
26060 017/300 Read block 2075 len 40960
26061 018/300 Read block 2373 len 32768
26062 019/300 Read block 3333 len 4096
26063 020/300 Read block 4583 len 24576
26064 021/300 Read block 2863 len 36864
26065 022/300 Read block 3461 len 16384
26066 023/300 Write block 1799 len 28672 data: 7 8 7 7 6 6 6
26067 024/300 Read block 2339 len 16384
26068 025/300 Read block 4719 len 4096
26069 026/300 Read block 2084 len 40960
26070 027/300 Write block 4708 len 4096 data: 9
26071 028/300 Write block 613 len 40960 data: 5 5 7 8 8 8 8 6 5 3
26072 029/300 Write block 1601 len 32768 data: 2 3 2 4 3 3 3 5
26073 030/300 Read block 69 len 8192
26074 031/300 Read block 3040 len 40960
26075 032/300 Read block 3760 len 40960
26076 033/300 Read block 3580 len 4096
26077 034/300 Read block 4298 len 28672
26078 035/300 Write block 1184 len 40960 data: 3 3 3 3 3 5 4 5 4 6
26079 036/300 Write block 4587 len 36864 data: 5 4 6 6 7 6 6 7 6
26080 037/300 Write block 4441 len 32768 data: 5 5 5 5 6 6 4 4
26081 038/300 Read block 1945 len 8192
26082 039/300 Read block 3790 len 12288
26083 040/300 Flush
26084 041/300 Read block 1377 len 12288
26085 042/300 Write block 1059 len 16384 data: 4 4 5 4
26086 043/300 Write block 1991 len 12288 data: 4 4 5
26087 044/300 Write block 3537 len 20480 data: 2 2 2 3 4
26088 045/300 Write block 292 len 8192 data: 5 8
26089 046/300 Read block 4880 len 32768
26090 047/300 Read block 2865 len 24576
26091 048/300 Read block 3351 len 40960
26092 049/300 Read block 1211 len 24576
26093 050/300 Write block 2491 len 36864 data: 7 6 6 6 6 5 4 6 7
26094 051/300 Write block 2069 len 16384 data: 8 7 7 6
26095 052/300 Flush
26096 053/300 Read block 2013 len 4096
26097 054/300 Read block 1467 len 16384
26098 055/300 Write block 3256 len 4096 data: 5
26099 056/300 Read block 1608 len 8192
26100 057/300 Read block 1453 len 24576
26101 058/300 Read block 1952 len 8192
26102 059/300 Flush
26103 060/300 Write block 3104 len 32768 data: 7 7 8 7 8 7 9 10
26104 061/300 Read block 1764 len 8192
26105 062/300 Read block 1567 len 20480
26106 063/300 Write block 4097 len 20480 data: 3 3 2 2 3
26107 064/300 Write block 2199 len 28672 data: 4 6 4 5 6 5 6
26108 065/300 Read block 4984 len 24576
26109 066/300 Write block 2491 len 32768 data: 8 7 7 7 7 6 5 7
26110 067/300 Write block 2731 len 12288 data: 9 7 7
26111 068/300 Read block 2971 len 40960
26112 069/300 Write block 2016 len 40960 data: 4 3 3 4 5 5 5 6 5 5
26113 070/300 Read block 4591 len 40960
26114 071/300 Write block 2061 len 16384 data: 8 9 8 7
26115 072/300 Write block 3201 len 28672 data: 3 3 4 5 4 5 5
26116 073/300 Read block 1878 len 20480
26117 074/300 Read block 3477 len 8192
26118 075/300 Read block 2357 len 24576
26119 076/300 Read block 2022 len 36864
26120 077/300 Write block 2366 len 28672 data: 4 5 5 6 4 6 6
26121 078/300 Read block 664 len 4096
26122 079/300 Read block 856 len 36864
26123 080/300 Write block 1728 len 16384 data: 6 5 6 5
26124 081/300 Read block 1720 len 32768
26125 082/300 Write block 2760 len 20480 data: 4 5 5 4 3
26126 083/300 Read block 81 len 12288
26127 084/300 Write block 3997 len 36864 data: 5 6 6 7 5 4 3 3 3
26128 085/300 Read block 3479 len 12288
26129 086/300 Read block 2295 len 12288
26130 087/300 Write block 4264 len 32768 data: 6 5 5 7 8 9 9 9
26131 088/300 Write block 1162 len 12288 data: 11 11 11
26132 089/300 Write block 2181 len 4096 data: 7
26133 090/300 Write block 1592 len 16384 data: 2 3 3 3
26134 091/300 Flush
26135 092/300 Write block 3495 len 36864 data: 7 7 8 7 8 6 7 8 8
26136 093/300 Read block 3590 len 4096
26137 094/300 Read block 1937 len 16384
26138 095/300 Write block 4316 len 32768 data: 6 3 4 4 4 4 4 4
26139 096/300 Read block 1946 len 40960
26140 097/300 Write block 3095 len 8192 data: 2 2
26141 098/300 Write block 108 len 4096 data: 8
26142 099/300 Write block 525 len 36864 data: 6 6 4 4 5 5 4 3 4
26143 100/300 Read block 486 len 8192
26144 101/300 Write block 856 len 16384 data: 4 4 3 3
26145 102/300 Write block 3155 len 4096 data: 5
26146 103/300 Write block 3542 len 16384 data: 7 6 6 6
26147 104/300 Write block 325 len 8192 data: 2 2
26148 105/300 Write block 2533 len 12288 data: 5 7 7
26149 106/300 Read block 1090 len 12288
26150 107/300 Read block 118 len 36864
26151 108/300 Read block 222 len 12288
26152 109/300 Read block 4494 len 36864
26153 110/300 Write block 2016 len 8192 data: 5 4
26154 111/300 Read block 3333 len 32768
26155 112/300 Read block 1380 len 16384
26156 113/300 Write block 811 len 40960 data: 7 9 7 6 5 5 4 3 4 6
26157 114/300 Read block 1783 len 12288
26158 115/300 Read block 2023 len 12288
26159 116/300 Write block 1853 len 32768 data: 4 5 5 5 6 4 4 4
26160 117/300 Read block 1439 len 20480
26161 118/300 Read block 3461 len 4096
26162 119/300 Read block 4982 len 16384
26163 120/300 Read block 376 len 20480
26164 121/300 Write block 4909 len 20480 data: 6 7 6 6 6
26165 122/300 Write block 1297 len 12288 data: 5 5 5
26166 123/300 Write block 3450 len 28672 data: 4 4 4 4 3 4 5
26167 124/300 Read block 2558 len 12288
26168 125/300 Flush
26169 126/300 Read block 2598 len 28672
26170 127/300 Read block 1808 len 20480
26171 128/300 Flush
26172 129/300 Write block 4880 len 28672 data: 4 4 4 3 4 4 6
26173 130/300 Flush
26174 131/300 Read block 3710 len 12288
26175 132/300 Write block 1381 len 16384 data: 3 3 4 5
26176 133/300 Flush
26177 134/300 Read block 978 len 32768
26178 135/300 Read block 449 len 4096
26179 136/300 Write block 561 len 28672 data: 3 3 3 4 4 6 6
26180 137/300 Write block 3980 len 28672 data: 3 3 6 5 5 5 6
26181 138/300 Read block 3016 len 36864
26182 139/300 Read block 2517 len 32768
26183 140/300 Read block 2985 len 32768
26184 141/300 Read block 3669 len 8192
26185 142/300 Write block 3292 len 16384 data: 8 8 6 5
26186 143/300 Read block 724 len 20480
26187 144/300 Write block 3580 len 24576 data: 8 9 9 8 8 7
26188 145/300 Read block 17 len 16384
26189 146/300 Read block 2931 len 28672
26190 147/300 Read block 3143 len 20480
26191 148/300 Read block 947 len 4096
26192 149/300 Read block 416 len 36864
26193 150/300 Flush
26194 151/300 Write block 4833 len 40960 data: 4 5 4 3 3 2 2 2 4 3
26195 152/300 Write block 2129 len 24576 data: 6 5 6 7 5 6
26196 153/300 Read block 1174 len 16384
26197 154/300 Write block 4409 len 40960 data: 4 4 4 6 6 6 5 4 3 4
26198 155/300 Read block 4621 len 28672
26199 156/300 Read block 3865 len 24576
26200 157/300 Flush
26201 158/300 Read block 1687 len 36864
26202 159/300 Write block 3751 len 24576 data: 4 4 4 5 5 4
26203 160/300 Read block 2713 len 32768
26204 161/300 Flush
26205 162/300 Write block 4175 len 4096 data: 5
26206 163/300 Write block 2500 len 32768 data: 5 6 7 7 6 6 5 3
26207 164/300 Read block 506 len 20480
26208 165/300 Read block 1514 len 16384
26209 166/300 Write block 794 len 4096 data: 6
26210 167/300 Read block 2740 len 28672
26211 168/300 Write block 971 len 36864 data: 5 6 6 6 7 5 5 5 5
26212 169/300 Write block 1303 len 36864 data: 4 5 5 4 5 6 6 8 10
26213 170/300 Write block 2243 len 12288 data: 6 5 4
26214 171/300 Write block 518 len 4096 data: 4
26215 172/300 Write block 192 len 36864 data: 4 4 4 6 6 5 5 5 5
26216 173/300 Read block 1045 len 28672
26217 174/300 Write block 298 len 32768 data: 6 6 6 5 4 4 3 3
26218 175/300 Read block 3000 len 20480
26219 176/300 Read block 4269 len 32768
26220 177/300 Read block 2504 len 36864
26221 178/300 Read block 1358 len 16384
26222 179/300 Write block 492 len 20480 data: 3 4 4 3 4
26223 180/300 Write block 2360 len 28672 data: 5 5 4 4 4 3 5
26224 181/300 Write block 2473 len 8192 data: 5 6
26225 182/300 Read block 379 len 32768
26226 183/300 Read block 1924 len 28672
26227 184/300 Read block 4216 len 40960
26228 185/300 Write block 1212 len 20480 data: 2 3 3 4 3
26229 186/300 Write block 4312 len 8192 data: 4 4
26230 187/300 Write block 817 len 28672 data: 5 4 5 7 7 8 8
26231 188/300 Read block 2264 len 28672
26232 189/300 Write block 1456 len 12288 data: 6 5 4
26233 190/300 Write block 3330 len 40960 data: 6 5 5 4 5 4 4 5 5 5
26234 191/300 Read block 1685 len 36864
26235 192/300 Write block 45 len 12288 data: 3 4 6
26236 193/300 Read block 874 len 36864
26237 194/300 Read block 3794 len 16384
26238 195/300 Write block 635 len 16384 data: 7 7 6 5
26239 196/300 Read block 96 len 12288
26240 197/300 Read block 696 len 32768
26241 198/300 Flush
26242 199/300 Write block 732 len 20480 data: 4 5 6 7 7
26243 200/300 Write block 1703 len 36864 data: 3 3 3 4 3 3 3 5 6
26244 201/300 Flush
26245 202/300 Write block 552 len 28672 data: 4 5 5 8 10 8 6
26246 203/300 Read block 306 len 24576
26247 204/300 Read block 2361 len 28672
26248 205/300 Read block 2333 len 28672
26249 206/300 Read block 3888 len 40960
26250 207/300 Flush
26251 208/300 Write block 4481 len 28672 data: 4 4 4 3 4 3 4
26252 209/300 Write block 3650 len 40960 data: 7 8 9 11 9 9 6 6 5 5
26253 210/300 Write block 3656 len 20480 data: 7 7 6 6 6
26254 211/300 Flush
26255 212/300 Read block 77 len 24576
26256 213/300 Read block 217 len 8192
26257 214/300 Read block 1574 len 8192
26258 215/300 Read block 1852 len 36864
26259 216/300 Read block 3474 len 40960
26260 217/300 Write block 1933 len 12288 data: 3 3 3
26261 218/300 Read block 3784 len 24576
26262 219/300 Read block 1334 len 24576
26263 220/300 Write block 4314 len 28672 data: 4 5 7 4 5 5 5
26264 221/300 Write block 4472 len 12288 data: 6 4 4
26265 222/300 Write block 4402 len 8192 data: 7 6
26266 223/300 Read block 2436 len 28672
26267 224/300 Read block 4826 len 12288
26268 225/300 Write block 1978 len 36864 data: 7 6 6 5 5 2 2 2 2
26269 226/300 Write block 4633 len 8192 data: 6 5
26270 227/300 Write block 222 len 28672 data: 3 3 3 3 3 3 5
26271 228/300 Write block 4915 len 8192 data: 5 3
26272 229/300 Write block 2577 len 28672 data: 2 2 2 2 2 3 3
26273 230/300 Write block 4399 len 12288 data: 8 8 8
26274 231/300 Read block 978 len 40960
26275 232/300 Write block 4298 len 20480 data: 2 3 3 5 5
26276 233/300 Write block 1184 len 16384 data: 4 4 4 4
26277 234/300 Write block 2789 len 32768 data: 2 2 2 4 4 6 5 5
26278 235/300 Read block 634 len 24576
26279 236/300 Read block 687 len 36864
26280 237/300 Read block 3442 len 20480
26281 238/300 Read block 1393 len 8192
26282 239/300 Flush
26283 240/300 Write block 904 len 32768 data: 5 5 5 5 4 7 7 7
26284 241/300 Flush
26285 242/300 Read block 2020 len 16384
26286 243/300 Write block 1718 len 36864 data: 2 3 4 4 4 4 4 4 4
26287 244/300 Write block 2724 len 32768 data: 4 5 5 8 8 9 9 10
26288 245/300 Write block 4554 len 24576 data: 6 7 10 10 10 7
26289 246/300 Read block 100 len 24576
26290 247/300 Write block 4909 len 24576 data: 7 8 7 7 7 6
26291 248/300 Write block 2373 len 36864 data: 6 7 7 8 8 9 8 8 7
26292 249/300 Read block 865 len 36864
26293 250/300 Read block 3140 len 16384
26294 251/300 Flush
26295 252/300 Read block 2905 len 4096
26296 253/300 Read block 1523 len 20480
26297 254/300 Read block 1867 len 12288
26298 255/300 Write block 1938 len 20480 data: 3 4 4 5 6
26299 256/300 Read block 2779 len 4096
26300 257/300 Write block 631 len 36864 data: 5 6 7 7 8 8 7 6 6
26301 258/300 Write block 2350 len 24576 data: 6 8 6 7 6 7
26302 259/300 Write block 2701 len 4096 data: 4
26303 260/300 Write block 161 len 16384 data: 7 6 6 5
26304 261/300 Write block 3163 len 36864 data: 8 8 6 6 5 4 3 3 3
26305 262/300 Read block 3918 len 36864
26306 263/300 Read block 1527 len 36864
26307 264/300 Flush
26308 265/300 Read block 1007 len 20480
26309 266/300 Read block 2388 len 28672
26310 267/300 Write block 351 len 36864 data: 5 3 4 4 4 4 5 4 3
26311 268/300 Flush
26312 269/300 Read block 402 len 24576
26313 270/300 Read block 1684 len 12288
26314 271/300 Read block 1766 len 16384
26315 272/300 Read block 4377 len 28672
26316 273/300 Read block 1484 len 28672
26317 274/300 Flush
26318 275/300 Write block 2307 len 16384 data: 3 3 4 4
26319 276/300 Read block 4054 len 4096
26320 277/300 Write block 3306 len 36864 data: 8 7 7 7 7 6 6 6 6
26321 278/300 Write block 3302 len 12288 data: 3 3 5
26322 279/300 Read block 4977 len 40960
26323 280/300 Write block 3635 len 12288 data: 4 5 5
26324 281/300 Read block 2204 len 12288
26325 282/300 Write block 2200 len 24576 data: 7 5 6 7 6 7
26326 283/300 Write block 3650 len 16384 data: 8 9 10 12
26327 284/300 Read block 3357 len 28672
26328 285/300 Write block 2040 len 28672 data: 3 4 4 5 4 4 4
26329 286/300 Flush
26330 287/300 Flush
26331 288/300 Read block 2099 len 40960
26332 289/300 Read block 100 len 20480
26333 290/300 Write block 3674 len 28672 data: 6 5 4 5 6 4 6
26334 291/300 Read block 4877 len 28672
26335 292/300 Write block 4099 len 36864 data: 3 3 4 4 6 6 6 6 6
26336 293/300 Write block 4560 len 20480 data: 6 6 5 6 8
26337 294/300 Read block 3374 len 16384
26338 295/300 Read block 4873 len 40960
26339 296/300 Read block 3925 len 16384
26340 297/300 Flush
26341 298/300 Flush
26342 299/300 Read block 1754 len 28672
26343 300/300 Flush
26344 301/300 Write block 975 len 8192 data: 8 6
26345 Main task has work to do, go find it
26346 got dsc Start(0)
26347 start 0
26348 Jan 23 17:43:17.929 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/0, method: POST, req_id: 7f61ca37-a874-40ba-846a-657835ccbcd3, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
26349 Replay: started 0, returned:Ok(())
26350 ----------------------------------------------------------------
26351 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26352 States: Offline Active Active
26353 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26354 8247 Acked Read 8 Sent Done Done false
26355 8248 Acked Write 2 Sent Done Done false
26356 8249 Acked Write 9 Sent Done Done false
26357 8250 Acked Write 2 Sent Done Done false
26358 8251 Acked Read 3 Sent Done Done false
26359 8252 Acked Read 4 Sent Done Done false
26360 8253 Acked Write 6 Sent Done Done false
26361 8254 Acked Read 6 Sent Done Done false
26362 8255 Acked Flush 0 Sent Done Done false
26363 8256 Acked Write 7 Sent Done Done false
26364 8257 Acked Read 4 Sent Done Done false
26365 8258 Acked Read 3 Sent Done Done false
26366 8259 Acked Flush 0 Sent Done Done false
26367 8260 Acked Flush 0 Sent Done Done false
26368 8261 Acked Read 4 Sent Done Done false
26369 8262 Acked Write 1 Sent Done Done false
26370 8263 Acked Read 10 Sent Done Done false
26371 8264 Acked Read 8 Sent Done Done false
26372 8265 Acked Read 1 Sent Done Done false
26373 8266 Acked Read 6 Sent Done Done false
26374 8267 Acked Read 9 Sent Done Done false
26375 8268 Acked Read 4 Sent Done Done false
26376 8269 Acked Write 7 Sent Done Done false
26377 8270 Acked Read 4 Sent Done Done false
26378 8271 Acked Read 1 Sent Done Done false
26379 8272 Acked Read 10 Sent Done Done false
26380 8273 Acked Write 1 Sent Done Done false
26381 8274 Acked Write 10 Sent Done Done false
26382 8275 Acked Write 8 Sent Done Done false
26383 8276 Acked Read 2 Sent Done Done false
26384 8277 Acked Read 10 Sent Done Done false
26385 8278 Acked Read 10 Sent Done Done false
26386 8279 Acked Read 1 Sent Done Done false
26387 8280 Acked Read 7 Sent Done Done false
26388 8281 Acked Write 10 Sent Done Done false
26389 8282 Acked Write 9 Sent Done Done false
26390 8283 Acked Write 8 Sent Done Done false
26391 8284 Acked Read 2 Sent Done Done false
26392 8285 Acked Read 3 Sent Done Done false
26393 8286 Acked Flush 0 Sent Done Done false
26394 8287 Acked Read 3 Sent Done Done false
26395 8288 Acked Write 4 Sent Done Done false
26396 8289 Acked Write 3 Sent Done Done false
26397 8290 Acked Write 5 Sent Done Done false
26398 8291 Acked Write 2 Sent Done Done false
26399 8292 Acked Read 8 Sent Done Done false
26400 8293 Acked Read 6 Sent Done Done false
26401 8294 Acked Read 10 Sent Done Done false
26402 8295 Acked Read 6 Sent Done Done false
26403 8296 Acked Write 9 Sent Done Done false
26404 8297 Acked Write 4 Sent Done Done false
26405 8298 Acked Flush 0 Sent Done Done false
26406 8299 Acked Read 1 Sent Done Done false
26407 8300 Acked Read 4 Sent Done Done false
26408 8301 Acked Write 1 Sent Done Done false
26409 8302 Acked Read 2 Sent Done Done false
26410 8303 Acked Read 6 Sent Done Done false
26411 8304 Acked Read 2 Sent Done Done false
26412 8305 Acked Flush 0 Sent Done Done false
26413 8306 Acked Write 8 Sent Done Done false
26414 8307 Acked Read 2 Sent Done Done false
26415 8308 Acked Read 5 Sent Done Done false
26416 8309 Acked Write 5 Sent Done Done false
26417 8310 Acked Write 7 Sent Done Done false
26418 8311 Acked Read 6 Sent Done Done false
26419 8312 Acked Write 8 Sent Done Done false
26420 8313 Acked Write 3 Sent Done Done false
26421 8314 Acked Read 10 Sent Done Done false
26422 8315 Acked Write 10 Sent Done Done false
26423 8316 Acked Read 10 Sent Done Done false
26424 8317 Acked Write 4 Sent Done Done false
26425 8318 Acked Write 7 Sent Done Done false
26426 8319 Acked Read 5 Sent Done Done false
26427 8320 Acked Read 2 Sent Done Done false
26428 8321 Acked Read 6 Sent Done Done false
26429 8322 Acked Read 9 Sent Done Done false
26430 8323 Acked Write 7 Sent Done Done false
26431 8324 Acked Read 1 Sent Done Done false
26432 8325 Acked Read 9 Sent Done Done false
26433 8326 Acked Write 4 Sent Done Done false
26434 8327 Acked Read 8 Sent Done Done false
26435 8328 Acked Write 5 Sent Done Done false
26436 8329 Acked Read 3 Sent Done Done false
26437 8330 Acked Write 9 Sent Done Done false
26438 8331 Acked Read 3 Sent Done Done false
26439 8332 Acked Read 3 Sent Done Done false
26440 8333 Acked Write 8 Sent Done Done false
26441 8334 Acked Write 3 Sent Done Done false
26442 8335 Acked Write 1 Sent Done Done false
26443 8336 Acked Write 4 Sent Done Done false
26444 8337 Acked Flush 0 Sent Done Done false
26445 8338 Acked Write 9 Sent Done Done false
26446 8339 Acked Read 1 Sent Done Done false
26447 8340 Acked Read 4 Sent Done Done false
26448 8341 Acked Write 8 Sent Done Done false
26449 8342 Acked Read 10 Sent Done Done false
26450 8343 Acked Write 2 Sent Done Done false
26451 8344 Acked Write 1 Sent Done Done false
26452 8345 Acked Write 9 Sent Done Done false
26453 8346 Acked Read 2 Sent Done Done false
26454 8347 Acked Write 4 Sent Done Done false
26455 8348 Acked Write 1 Sent Done Done false
26456 8349 Acked Write 4 Sent Done Done false
26457 8350 Acked Write 2 Sent Done Done false
26458 8351 Acked Write 3 Sent Done Done false
26459 8352 Acked Read 3 Sent Done Done false
26460 8353 Acked Read 9 Sent Done Done false
26461 8354 Acked Read 3 Sent Done Done false
26462 8355 Acked Read 9 Sent Done Done false
26463 8356 Acked Write 2 Sent Done Done false
26464 8357 Acked Read 8 Sent Done Done false
26465 8358 Acked Read 4 Sent Done Done false
26466 8359 Acked Write 10 Sent Done Done false
26467 8360 Acked Read 3 Sent Done Done false
26468 8361 Acked Read 3 Sent Done Done false
26469 8362 Acked Write 8 Sent Done Done false
26470 8363 Acked Read 5 Sent Done Done false
26471 8364 Acked Read 1 Sent Done Done false
26472 8365 Acked Read 4 Sent Done Done false
26473 8366 Acked Read 5 Sent Done Done false
26474 8367 Acked Write 5 Sent Done Done false
26475 8368 Acked Write 3 Sent Done Done false
26476 8369 Acked Write 7 Sent Done Done false
26477 8370 Acked Read 3 Sent Done Done false
26478 8371 Acked Flush 0 Sent Done Done false
26479 8372 Acked Read 7 Sent Done Done false
26480 8373 Acked Read 5 Sent Done Done false
26481 8374 Acked Flush 0 Sent Done Done false
26482 8375 Acked Write 7 Sent Done Done false
26483 8376 Acked Flush 0 Sent Done Done false
26484 8377 Acked Read 3 Sent Done Done false
26485 8378 Acked Write 4 Sent Done Done false
26486 8379 Acked Flush 0 Sent Done Done false
26487 8380 Acked Read 8 Sent Done Done false
26488 8381 Acked Read 1 Sent Done Done false
26489 8382 Acked Write 7 Sent Done Done false
26490 8383 Acked Write 7 Sent Done Done false
26491 8384 Acked Read 9 Sent Done Done false
26492 8385 Acked Read 8 Sent Done Done false
26493 8386 Acked Read 8 Sent Done Done false
26494 8387 Acked Read 2 Sent Done Done false
26495 8388 Acked Write 4 Sent Done Done false
26496 8389 Acked Read 5 Sent Done Done false
26497 8390 Acked Write 6 Sent Done Done false
26498 8391 Acked Read 4 Sent Done Done false
26499 8392 Acked Read 7 Sent Done Done false
26500 8393 Acked Read 5 Sent Done Done false
26501 8394 Acked Read 1 Sent Done Done false
26502 8395 Acked Read 9 Sent Done Done false
26503 8396 Acked Flush 0 Sent Done Done false
26504 8397 Acked Write 10 Sent Done Done false
26505 8398 Acked Write 6 Sent Done Done false
26506 8399 Acked Read 4 Sent Done Done false
26507 8400 Acked Write 10 Sent Done Done false
26508 8401 Acked Read 7 Sent Done Done false
26509 8402 Acked Read 6 Sent Done Done false
26510 8403 Acked Flush 0 Sent Done Done false
26511 8404 Acked Read 9 Sent Done Done false
26512 8405 Acked Write 6 Sent Done Done false
26513 8406 Acked Read 8 Sent Done Done false
26514 8407 Acked Flush 0 Sent Done Done false
26515 8408 Acked Write 1 Sent Done Done false
26516 8409 Acked Write 8 Sent Done Done false
26517 8410 Acked Read 5 Sent Done Done false
26518 8411 Acked Read 4 Sent Done Done false
26519 8412 Acked Write 1 Sent Done Done false
26520 8413 Acked Read 7 Sent Done Done false
26521 8414 Acked Write 9 Sent Done Done false
26522 8415 Acked Write 9 Sent Done Done false
26523 8416 Acked Write 3 Sent Done Done false
26524 8417 Acked Write 1 Sent Done Done false
26525 8418 Acked Write 9 Sent Done Done false
26526 8419 Acked Read 7 Sent Done Done false
26527 8420 Acked Write 8 Sent Done Done false
26528 8421 Acked Read 5 Sent Done Done false
26529 8422 Acked Read 8 Sent Done Done false
26530 8423 Acked Read 9 Sent Done Done false
26531 8424 Acked Read 4 Sent Done Done false
26532 8425 Acked Write 5 Sent Done Done false
26533 8426 Acked Write 7 Sent Done Done false
26534 8427 Acked Write 2 Sent Done Done false
26535 8428 Acked Read 8 Sent Done Done false
26536 8429 Acked Read 7 Sent Done Done false
26537 8430 Acked Read 10 Sent Done Done false
26538 8431 Acked Write 5 Sent Done Done false
26539 8432 Acked Write 2 Sent Done Done false
26540 8433 Acked Write 7 Sent Done Done false
26541 8434 Acked Read 7 Sent Done Done false
26542 8435 Acked Write 3 Sent Done Done false
26543 8436 Acked Write 10 Sent Done Done false
26544 8437 Acked Read 9 Sent Done Done false
26545 8438 Acked Write 3 Sent Done Done false
26546 8439 Acked Read 9 Sent Done Done false
26547 8440 Acked Read 4 Sent Done Done false
26548 8441 Acked Write 4 Sent Done Done false
26549 8442 Acked Read 3 Sent Done Done false
26550 8443 Acked Read 8 Sent Done Done false
26551 8444 Acked Flush 0 Sent Done Done false
26552 8445 Acked Write 5 Sent Done Done false
26553 8446 Acked Write 9 Sent Done Done false
26554 8447 Acked Flush 0 Sent Done Done false
26555 8448 Acked Write 7 Sent Done Done false
26556 8449 Acked Read 6 Sent Done Done false
26557 8450 Acked Read 7 Sent Done Done false
26558 8451 Acked Read 7 Sent Done Done false
26559 8452 Acked Read 10 Sent Done Done false
26560 8453 Acked Flush 0 Sent Done Done false
26561 8454 Acked Write 7 Sent Done Done false
26562 8455 Acked Write 10 Sent Done Done false
26563 8456 Acked Write 5 Sent Done Done false
26564 8457 Acked Flush 0 Sent Done Done false
26565 8458 Acked Read 6 Sent Done Done false
26566 8459 Acked Read 2 Sent Done Done false
26567 8460 Acked Read 2 Sent Done Done false
26568 8461 Acked Read 9 Sent Done Done false
26569 8462 Acked Read 10 Sent Done Done false
26570 8463 Acked Write 3 Sent Done Done false
26571 8464 Acked Read 6 Sent Done Done false
26572 8465 Acked Read 6 Sent Done Done false
26573 8466 Acked Write 7 Sent Done Done false
26574 8467 Acked Write 3 Sent Done Done false
26575 8468 Acked Write 2 Sent Done Done false
26576 8469 Acked Read 7 Sent Done Done false
26577 8470 Acked Read 3 Sent Done Done false
26578 8471 Acked Write 9 Sent Done Done false
26579 8472 Acked Write 2 Sent Done Done false
26580 8473 Acked Write 7 Sent Done Done false
26581 8474 Acked Write 2 Sent Done Done false
26582 8475 Acked Write 7 Sent Done Done false
26583 8476 Acked Write 3 Sent Done Done false
26584 8477 Acked Read 10 Sent Done Done false
26585 8478 Acked Write 5 Sent Done Done false
26586 8479 Acked Write 4 Sent Done Done false
26587 8480 Acked Write 8 Sent Done Done false
26588 8481 Acked Read 6 Sent Done Done false
26589 8482 Acked Read 9 Sent Done Done false
26590 8483 Acked Read 5 Sent Done Done false
26591 8484 Acked Read 2 Sent Done Done false
26592 8485 Acked Flush 0 Sent Done Done false
26593 8486 Acked Write 8 Sent Done Done false
26594 8487 Acked Flush 0 Sent Done Done false
26595 8488 Acked Read 4 Sent Done Done false
26596 8489 Acked Write 9 Sent Done Done false
26597 8490 Acked Write 8 Sent Done Done false
26598 8491 Acked Write 6 Sent Done Done false
26599 8492 Acked Read 6 Sent Done Done false
26600 8493 Acked Write 6 Sent Done Done false
26601 8494 Acked Write 9 Sent Done Done false
26602 8495 Acked Read 9 Sent Done Done false
26603 8496 Acked Read 4 Sent Done Done false
26604 8497 Acked Flush 0 Sent Done Done false
26605 8498 Acked Read 1 Sent Done Done false
26606 8499 Acked Read 5 Sent Done Done false
26607 8500 Acked Read 3 Sent Done Done false
26608 8501 Acked Write 5 Sent Done Done false
26609 8502 Acked Read 1 Sent Done Done false
26610 8503 Acked Write 9 Sent Done Done false
26611 8504 Acked Write 6 Sent Done Done false
26612 8505 Acked Write 1 Sent Done Done false
26613 8506 Acked Write 4 Sent Done Done false
26614 8507 Acked Write 9 Sent Done Done false
26615 8508 Acked Read 9 Sent Done Done false
26616 8509 Acked Read 9 Sent Done Done false
26617 8510 Acked Flush 0 Sent Done Done false
26618 8511 Acked Read 5 Sent Done Done false
26619 8512 Acked Read 7 Sent Done Done false
26620 8513 Acked Write 9 Sent Done Done false
26621 8514 Acked Flush 0 Sent Done Done false
26622 8515 Acked Read 6 Sent Done Done false
26623 8516 Acked Read 3 Sent Done Done false
26624 8517 Acked Read 4 Sent Done Done false
26625 8518 Acked Read 7 Sent Done Done false
26626 8519 Acked Read 7 Sent Done Done false
26627 8520 Acked Flush 0 Sent Done Done false
26628 8521 Acked Write 4 Sent Done Done false
26629 8522 Acked Read 1 Sent Done Done false
26630 8523 Acked Write 9 Sent Done Done false
26631 8524 Acked Write 3 Sent Done Done false
26632 8525 Acked Read 10 Sent Done Done false
26633 8526 Acked Write 3 Sent Done Done false
26634 8527 Acked Read 3 Sent Done Done false
26635 8528 Acked Write 6 Sent Done Done false
26636 8529 Acked Write 4 Sent Done Done false
26637 8530 Acked Read 7 Sent Done Done false
26638 8531 Acked Write 7 Sent Done Done false
26639 8532 Acked Flush 0 Sent Done Done false
26640 8533 Acked Flush 0 Sent Done Done false
26641 8534 Acked Read 10 Sent Done Done false
26642 8535 Acked Read 5 Sent Done Done false
26643 8536 Acked Write 7 Sent Done Done false
26644 8537 Acked Read 7 Sent Done Done false
26645 8538 Acked Write 9 Sent Done Done false
26646 8539 Acked Write 5 Sent Done Done false
26647 8540 Acked Read 4 Sent Done Done false
26648 8541 Acked Read 10 Sent Done Done false
26649 8542 Acked Read 4 Sent Done Done false
26650 8543 Acked Flush 0 Sent Done Done false
26651 8544 Acked Flush 0 Sent Done Done false
26652 8545 Acked Read 7 Sent Done Done false
26653 8546 Acked Flush 0 Sent Done Done false
26654 8547 Acked Write 2 Sent Done Done false
26655 STATES DS:0 DS:1 DS:2 TOTAL
26656 Sent 301 0 0 301
26657 Done 0 301 301 602
26658 Skipped 0 0 0 0
26659 Error 0 0 0 0
26660 Last Flush: Some(JobId(8246)) Some(JobId(8546)) Some(JobId(8546))
26661 Downstairs last five completed: 8246 8245 8244 8243 8242
26662 Upstairs last five completed: 8547 8546 8545 8544 8543
26663 CLIENT: Up:0 ds:301 act:2
26664 [8810] got start action
26665 [8810] I am going to restart
26666 Starting downstairs at port 8810
26667 Make output file at "/tmp/dsc/downstairs-8810.txt"
26668 [8810][0] reports Starting
26669 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1230)
26670 [8810][0] reports Running
26671 ----------------------------------------------------------------
26672 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
26673 States: Offline Active Active
26674 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26675 8247 Acked Read 8 Sent Done Done false
26676 8248 Acked Write 2 Sent Done Done false
26677 8249 Acked Write 9 Sent Done Done false
26678 8250 Acked Write 2 Sent Done Done false
26679 8251 Acked Read 3 Sent Done Done false
26680 8252 Acked Read 4 Sent Done Done false
26681 8253 Acked Write 6 Sent Done Done false
26682 8254 Acked Read 6 Sent Done Done false
26683 8255 Acked Flush 0 Sent Done Done false
26684 8256 Acked Write 7 Sent Done Done false
26685 8257 Acked Read 4 Sent Done Done false
26686 8258 Acked Read 3 Sent Done Done false
26687 8259 Acked Flush 0 Sent Done Done false
26688 8260 Acked Flush 0 Sent Done Done false
26689 8261 Acked Read 4 Sent Done Done false
26690 8262 Acked Write 1 Sent Done Done false
26691 8263 Acked Read 10 Sent Done Done false
26692 8264 Acked Read 8 Sent Done Done false
26693 8265 Acked Read 1 Sent Done Done false
26694 8266 Acked Read 6 Sent Done Done false
26695 8267 Acked Read 9 Sent Done Done false
26696 8268 Acked Read 4 Sent Done Done false
26697 8269 Acked Write 7 Sent Done Done false
26698 8270 Acked Read 4 Sent Done Done false
26699 8271 Acked Read 1 Sent Done Done false
26700 8272 Acked Read 10 Sent Done Done false
26701 8273 Acked Write 1 Sent Done Done false
26702 8274 Acked Write 10 Sent Done Done false
26703 8275 Acked Write 8 Sent Done Done false
26704 8276 Acked Read 2 Sent Done Done false
26705 8277 Acked Read 10 Sent Done Done false
26706 8278 Acked Read 10 Sent Done Done false
26707 8279 Acked Read 1 Sent Done Done false
26708 8280 Acked Read 7 Sent Done Done false
26709 8281 Acked Write 10 Sent Done Done false
26710 8282 Acked Write 9 Sent Done Done false
26711 8283 Acked Write 8 Sent Done Done false
26712 8284 Acked Read 2 Sent Done Done false
26713 8285 Acked Read 3 Sent Done Done false
26714 8286 Acked Flush 0 Sent Done Done false
26715 8287 Acked Read 3 Sent Done Done false
26716 8288 Acked Write 4 Sent Done Done false
26717 8289 Acked Write 3 Sent Done Done false
26718 8290 Acked Write 5 Sent Done Done false
26719 8291 Acked Write 2 Sent Done Done false
26720 8292 Acked Read 8 Sent Done Done false
26721 8293 Acked Read 6 Sent Done Done false
26722 8294 Acked Read 10 Sent Done Done false
26723 8295 Acked Read 6 Sent Done Done false
26724 8296 Acked Write 9 Sent Done Done false
26725 8297 Acked Write 4 Sent Done Done false
26726 8298 Acked Flush 0 Sent Done Done false
26727 8299 Acked Read 1 Sent Done Done false
26728 8300 Acked Read 4 Sent Done Done false
26729 8301 Acked Write 1 Sent Done Done false
26730 8302 Acked Read 2 Sent Done Done false
26731 8303 Acked Read 6 Sent Done Done false
26732 8304 Acked Read 2 Sent Done Done false
26733 8305 Acked Flush 0 Sent Done Done false
26734 8306 Acked Write 8 Sent Done Done false
26735 8307 Acked Read 2 Sent Done Done false
26736 8308 Acked Read 5 Sent Done Done false
26737 8309 Acked Write 5 Sent Done Done false
26738 8310 Acked Write 7 Sent Done Done false
26739 8311 Acked Read 6 Sent Done Done false
26740 8312 Acked Write 8 Sent Done Done false
26741 8313 Acked Write 3 Sent Done Done false
26742 8314 Acked Read 10 Sent Done Done false
26743 8315 Acked Write 10 Sent Done Done false
26744 8316 Acked Read 10 Sent Done Done false
26745 8317 Acked Write 4 Sent Done Done false
26746 8318 Acked Write 7 Sent Done Done false
26747 8319 Acked Read 5 Sent Done Done false
26748 8320 Acked Read 2 Sent Done Done false
26749 8321 Acked Read 6 Sent Done Done false
26750 8322 Acked Read 9 Sent Done Done false
26751 8323 Acked Write 7 Sent Done Done false
26752 8324 Acked Read 1 Sent Done Done false
26753 8325 Acked Read 9 Sent Done Done false
26754 8326 Acked Write 4 Sent Done Done false
26755 8327 Acked Read 8 Sent Done Done false
26756 8328 Acked Write 5 Sent Done Done false
26757 8329 Acked Read 3 Sent Done Done false
26758 8330 Acked Write 9 Sent Done Done false
26759 8331 Acked Read 3 Sent Done Done false
26760 8332 Acked Read 3 Sent Done Done false
26761 8333 Acked Write 8 Sent Done Done false
26762 8334 Acked Write 3 Sent Done Done false
26763 8335 Acked Write 1 Sent Done Done false
26764 8336 Acked Write 4 Sent Done Done false
26765 8337 Acked Flush 0 Sent Done Done false
26766 8338 Acked Write 9 Sent Done Done false
26767 8339 Acked Read 1 Sent Done Done false
26768 8340 Acked Read 4 Sent Done Done false
26769 8341 Acked Write 8 Sent Done Done false
26770 8342 Acked Read 10 Sent Done Done false
26771 8343 Acked Write 2 Sent Done Done false
26772 8344 Acked Write 1 Sent Done Done false
26773 8345 Acked Write 9 Sent Done Done false
26774 8346 Acked Read 2 Sent Done Done false
26775 8347 Acked Write 4 Sent Done Done false
26776 8348 Acked Write 1 Sent Done Done false
26777 8349 Acked Write 4 Sent Done Done false
26778 8350 Acked Write 2 Sent Done Done false
26779 8351 Acked Write 3 Sent Done Done false
26780 8352 Acked Read 3 Sent Done Done false
26781 8353 Acked Read 9 Sent Done Done false
26782 8354 Acked Read 3 Sent Done Done false
26783 8355 Acked Read 9 Sent Done Done false
26784 8356 Acked Write 2 Sent Done Done false
26785 8357 Acked Read 8 Sent Done Done false
26786 8358 Acked Read 4 Sent Done Done false
26787 8359 Acked Write 10 Sent Done Done false
26788 8360 Acked Read 3 Sent Done Done false
26789 8361 Acked Read 3 Sent Done Done false
26790 8362 Acked Write 8 Sent Done Done false
26791 8363 Acked Read 5 Sent Done Done false
26792 8364 Acked Read 1 Sent Done Done false
26793 8365 Acked Read 4 Sent Done Done false
26794 8366 Acked Read 5 Sent Done Done false
26795 8367 Acked Write 5 Sent Done Done false
26796 8368 Acked Write 3 Sent Done Done false
26797 8369 Acked Write 7 Sent Done Done false
26798 8370 Acked Read 3 Sent Done Done false
26799 8371 Acked Flush 0 Sent Done Done false
26800 8372 Acked Read 7 Sent Done Done false
26801 8373 Acked Read 5 Sent Done Done false
26802 8374 Acked Flush 0 Sent Done Done false
26803 8375 Acked Write 7 Sent Done Done false
26804 8376 Acked Flush 0 Sent Done Done false
26805 8377 Acked Read 3 Sent Done Done false
26806 8378 Acked Write 4 Sent Done Done false
26807 8379 Acked Flush 0 Sent Done Done false
26808 8380 Acked Read 8 Sent Done Done false
26809 8381 Acked Read 1 Sent Done Done false
26810 8382 Acked Write 7 Sent Done Done false
26811 8383 Acked Write 7 Sent Done Done false
26812 8384 Acked Read 9 Sent Done Done false
26813 8385 Acked Read 8 Sent Done Done false
26814 8386 Acked Read 8 Sent Done Done false
26815 8387 Acked Read 2 Sent Done Done false
26816 8388 Acked Write 4 Sent Done Done false
26817 8389 Acked Read 5 Sent Done Done false
26818 8390 Acked Write 6 Sent Done Done false
26819 8391 Acked Read 4 Sent Done Done false
26820 8392 Acked Read 7 Sent Done Done false
26821 8393 Acked Read 5 Sent Done Done false
26822 8394 Acked Read 1 Sent Done Done false
26823 8395 Acked Read 9 Sent Done Done false
26824 8396 Acked Flush 0 Sent Done Done false
26825 8397 Acked Write 10 Sent Done Done false
26826 8398 Acked Write 6 Sent Done Done false
26827 8399 Acked Read 4 Sent Done Done false
26828 8400 Acked Write 10 Sent Done Done false
26829 8401 Acked Read 7 Sent Done Done false
26830 8402 Acked Read 6 Sent Done Done false
26831 8403 Acked Flush 0 Sent Done Done false
26832 8404 Acked Read 9 Sent Done Done false
26833 8405 Acked Write 6 Sent Done Done false
26834 8406 Acked Read 8 Sent Done Done false
26835 8407 Acked Flush 0 Sent Done Done false
26836 8408 Acked Write 1 Sent Done Done false
26837 8409 Acked Write 8 Sent Done Done false
26838 8410 Acked Read 5 Sent Done Done false
26839 8411 Acked Read 4 Sent Done Done false
26840 8412 Acked Write 1 Sent Done Done false
26841 8413 Acked Read 7 Sent Done Done false
26842 8414 Acked Write 9 Sent Done Done false
26843 8415 Acked Write 9 Sent Done Done false
26844 8416 Acked Write 3 Sent Done Done false
26845 8417 Acked Write 1 Sent Done Done false
26846 8418 Acked Write 9 Sent Done Done false
26847 8419 Acked Read 7 Sent Done Done false
26848 8420 Acked Write 8 Sent Done Done false
26849 8421 Acked Read 5 Sent Done Done false
26850 8422 Acked Read 8 Sent Done Done false
26851 8423 Acked Read 9 Sent Done Done false
26852 8424 Acked Read 4 Sent Done Done false
26853 8425 Acked Write 5 Sent Done Done false
26854 8426 Acked Write 7 Sent Done Done false
26855 8427 Acked Write 2 Sent Done Done false
26856 8428 Acked Read 8 Sent Done Done false
26857 8429 Acked Read 7 Sent Done Done false
26858 8430 Acked Read 10 Sent Done Done false
26859 8431 Acked Write 5 Sent Done Done false
26860 8432 Acked Write 2 Sent Done Done false
26861 8433 Acked Write 7 Sent Done Done false
26862 8434 Acked Read 7 Sent Done Done false
26863 8435 Acked Write 3 Sent Done Done false
26864 8436 Acked Write 10 Sent Done Done false
26865 8437 Acked Read 9 Sent Done Done false
26866 8438 Acked Write 3 Sent Done Done false
26867 8439 Acked Read 9 Sent Done Done false
26868 8440 Acked Read 4 Sent Done Done false
26869 8441 Acked Write 4 Sent Done Done false
26870 8442 Acked Read 3 Sent Done Done false
26871 8443 Acked Read 8 Sent Done Done false
26872 8444 Acked Flush 0 Sent Done Done false
26873 8445 Acked Write 5 Sent Done Done false
26874 8446 Acked Write 9 Sent Done Done false
26875 8447 Acked Flush 0 Sent Done Done false
26876 8448 Acked Write 7 Sent Done Done false
26877 8449 Acked Read 6 Sent Done Done false
26878 8450 Acked Read 7 Sent Done Done false
26879 8451 Acked Read 7 Sent Done Done false
26880 8452 Acked Read 10 Sent Done Done false
26881 8453 Acked Flush 0 Sent Done Done false
26882 8454 Acked Write 7 Sent Done Done false
26883 8455 Acked Write 10 Sent Done Done false
26884 8456 Acked Write 5 Sent Done Done false
26885 8457 Acked Flush 0 Sent Done Done false
26886 8458 Acked Read 6 Sent Done Done false
26887 8459 Acked Read 2 Sent Done Done false
26888 8460 Acked Read 2 Sent Done Done false
26889 8461 Acked Read 9 Sent Done Done false
26890 8462 Acked Read 10 Sent Done Done false
26891 8463 Acked Write 3 Sent Done Done false
26892 8464 Acked Read 6 Sent Done Done false
26893 8465 Acked Read 6 Sent Done Done false
26894 8466 Acked Write 7 Sent Done Done false
26895 8467 Acked Write 3 Sent Done Done false
26896 8468 Acked Write 2 Sent Done Done false
26897 8469 Acked Read 7 Sent Done Done false
26898 8470 Acked Read 3 Sent Done Done false
26899 8471 Acked Write 9 Sent Done Done false
26900 8472 Acked Write 2 Sent Done Done false
26901 8473 Acked Write 7 Sent Done Done false
26902 8474 Acked Write 2 Sent Done Done false
26903 8475 Acked Write 7 Sent Done Done false
26904 8476 Acked Write 3 Sent Done Done false
26905 8477 Acked Read 10 Sent Done Done false
26906 8478 Acked Write 5 Sent Done Done false
26907 8479 Acked Write 4 Sent Done Done false
26908 8480 Acked Write 8 Sent Done Done false
26909 8481 Acked Read 6 Sent Done Done false
26910 8482 Acked Read 9 Sent Done Done false
26911 8483 Acked Read 5 Sent Done Done false
26912 8484 Acked Read 2 Sent Done Done false
26913 8485 Acked Flush 0 Sent Done Done false
26914 8486 Acked Write 8 Sent Done Done false
26915 8487 Acked Flush 0 Sent Done Done false
26916 8488 Acked Read 4 Sent Done Done false
26917 8489 Acked Write 9 Sent Done Done false
26918 8490 Acked Write 8 Sent Done Done false
26919 8491 Acked Write 6 Sent Done Done false
26920 8492 Acked Read 6 Sent Done Done false
26921 8493 Acked Write 6 Sent Done Done false
26922 8494 Acked Write 9 Sent Done Done false
26923 8495 Acked Read 9 Sent Done Done false
26924 8496 Acked Read 4 Sent Done Done false
26925 8497 Acked Flush 0 Sent Done Done false
26926 8498 Acked Read 1 Sent Done Done false
26927 8499 Acked Read 5 Sent Done Done false
26928 8500 Acked Read 3 Sent Done Done false
26929 8501 Acked Write 5 Sent Done Done false
26930 8502 Acked Read 1 Sent Done Done false
26931 8503 Acked Write 9 Sent Done Done false
26932 8504 Acked Write 6 Sent Done Done false
26933 8505 Acked Write 1 Sent Done Done false
26934 8506 Acked Write 4 Sent Done Done false
26935 8507 Acked Write 9 Sent Done Done false
26936 8508 Acked Read 9 Sent Done Done false
26937 8509 Acked Read 9 Sent Done Done false
26938 8510 Acked Flush 0 Sent Done Done false
26939 8511 Acked Read 5 Sent Done Done false
26940 8512 Acked Read 7 Sent Done Done false
26941 8513 Acked Write 9 Sent Done Done false
26942 8514 Acked Flush 0 Sent Done Done false
26943 8515 Acked Read 6 Sent Done Done false
26944 8516 Acked Read 3 Sent Done Done false
26945 8517 Acked Read 4 Sent Done Done false
26946 8518 Acked Read 7 Sent Done Done false
26947 8519 Acked Read 7 Sent Done Done false
26948 8520 Acked Flush 0 Sent Done Done false
26949 8521 Acked Write 4 Sent Done Done false
26950 8522 Acked Read 1 Sent Done Done false
26951 8523 Acked Write 9 Sent Done Done false
26952 8524 Acked Write 3 Sent Done Done false
26953 8525 Acked Read 10 Sent Done Done false
26954 8526 Acked Write 3 Sent Done Done false
26955 8527 Acked Read 3 Sent Done Done false
26956 8528 Acked Write 6 Sent Done Done false
26957 8529 Acked Write 4 Sent Done Done false
26958 8530 Acked Read 7 Sent Done Done false
26959 8531 Acked Write 7 Sent Done Done false
26960 8532 Acked Flush 0 Sent Done Done false
26961 8533 Acked Flush 0 Sent Done Done false
26962 8534 Acked Read 10 Sent Done Done false
26963 8535 Acked Read 5 Sent Done Done false
26964 8536 Acked Write 7 Sent Done Done false
26965 8537 Acked Read 7 Sent Done Done false
26966 8538 Acked Write 9 Sent Done Done false
26967 8539 Acked Write 5 Sent Done Done false
26968 8540 Acked Read 4 Sent Done Done false
26969 8541 Acked Read 10 Sent Done Done false
26970 8542 Acked Read 4 Sent Done Done false
26971 8543 Acked Flush 0 Sent Done Done false
26972 8544 Acked Flush 0 Sent Done Done false
26973 8545 Acked Read 7 Sent Done Done false
26974 8546 Acked Flush 0 Sent Done Done false
26975 8547 Acked Write 2 Sent Done Done false
26976 8548 Acked Flush 0 Sent Done Done false
26977 STATES DS:0 DS:1 DS:2 TOTAL
26978 Sent 302 0 0 302
26979 Done 0 302 302 604
26980 Skipped 0 0 0 0
26981 Error 0 0 0 0
26982 Last Flush: Some(JobId(8246)) Some(JobId(8548)) Some(JobId(8548))
26983 Downstairs last five completed: 8246 8245 8244 8243 8242
26984 Upstairs last five completed: 8548 8547 8546 8545 8544
26985 CLIENT: Up:0 ds:302 act:2
269862025-01-23T17:43:23.682ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
269872025-01-23T17:43:23.682ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46024) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
269882025-01-23T17:43:23.683ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
269892025-01-23T17:43:23.684ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
269902025-01-23T17:43:23.684ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
269912025-01-23T17:43:23.684ZINFOcrucible: send last flush ID to this DS: Some(JobId(8246)) = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
269922025-01-23T17:43:23.684ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
269932025-01-23T17:43:23.684ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(8246)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
269942025-01-23T17:43:23.684ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(8246)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
26995 ----------------------------------------------------------------
26996 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26997 Downstairs last five completed: 8548 8547 8546 8545 8544
26998 Upstairs last five completed: 8548 8547 8546 8545 8544
26999 CLIENT: Up:0 ds:0 act:3
27000 Replay: All jobs finished
27001 Main task has work to do, go find it
27002 got dsc Stop(0)
27003 stop 0
27004 [8810] Got stop action so:false kr:false
27005 Jan 23 17:43:25.935 INFO request completed, latency_us: 397, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 09d9ba8c-8756-4d40-9963-d2083eda4049, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
270062025-01-23T17:43:25.936ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
27007 Jan 23 17:43:25.936 INFO request completed, latency_us: 117, response_code: 200, uri: /state/cid/0, method: GET, req_id: 08c14ad7-c16b-42f3-ac6f-663e0c743e94, remote_addr: 127.0{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:43:25.936350029Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
27008 .0.1:42015, local_addr: 127{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:43:25.936389985Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
27009 .0.0.1:9998
270102025-01-23T17:43:25.936ZINFOcrucible: client task is exiting = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
27011 [8810] Exited with: ExitStatus(unix_wait_status(9))
27012 [8810][0] reports Exit
27013 Jan 23 17:43:29.939 INFO request completed, latency_us: 238, response_code: 200, uri: /state/cid/0, method: GET, req_id: cfdf8880-f5ff-48a0-a14f-374995444d9f, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
27014 001/300 Read block 594 len 36864
27015 002/300 Write block 4464 len 24576 data: 4 5 4 3 5 4
27016 003/300 Read block 3758 len 12288
27017 004/300 Read block 3034 len 8192
27018 005/300 Write block 2884 len 20480 data: 5 3 4 5 5
27019 006/300 Read block 4932 len 24576
27020 007/300 Read block 325 len 8192
27021 008/300 Write block 2229 len 20480 data: 9 10 8 8 6
27022 009/300 Write block 1855 len 36864 data: 6 6 7 5 5 5 6 8 7
27023 010/300 Write block 3337 len 12288 data: 6 6 6
27024 011/300 Write block 1829 len 28672 data: 7 6 4 3 5 8 8
27025 012/300 Read block 106 len 24576
27026 013/300 Write block 3126 len 16384 data: 2 2 2 2
27027 014/300 Read block 3628 len 16384
27028 015/300 Read block 3558 len 36864
27029 016/300 Write block 2679 len 4096 data: 4
27030 017/300 Write block 4769 len 12288 data: 3 3 4
27031 018/300 Read block 1179 len 4096
27032 019/300 Write block 478 len 4096 data: 3
27033 020/300 Write block 1708 len 20480 data: 4 4 6 7 6
27034 021/300 Read block 1497 len 28672
27035 022/300 Write block 1866 len 36864 data: 11 9 8 7 7 6 6 7 9
27036 023/300 Flush
27037 024/300 Write block 2637 len 16384 data: 4 4 4 4
27038 025/300 Read block 3257 len 20480
27039 026/300 Read block 2457 len 12288
27040 027/300 Read block 4619 len 20480
27041 028/300 Write block 2901 len 36864 data: 7 6 6 5 5 4 3 2 2
27042 029/300 Read block 2293 len 20480
27043 030/300 Write block 1699 len 16384 data: 8 8 7 6
27044 031/300 Write block 2550 len 12288 data: 4 3 3
27045 032/300 Write block 990 len 12288 data: 10 9 9
27046 033/300 Write block 1974 len 32768 data: 5 5 6 7 8 7 7 6
27047 034/300 Write block 2276 len 28672 data: 5 5 5 5 7 8 7
27048 035/300 Read block 198 len 20480
27049 036/300 Write block 1467 len 12288 data: 4 4 5
27050 037/300 Read block 3283 len 12288
27051 038/300 Read block 2214 len 12288
27052 039/300 Write block 3727 len 16384 data: 6 7 7 5
27053 040/300 Read block 700 len 28672
27054 041/300 Read block 118 len 32768
27055 042/300 Write block 1049 len 28672 data: 5 5 5 6 5 4 4
27056 043/300 Write block 3398 len 36864 data: 5 5 5 5 6 6 5 5 5
27057 044/300 Read block 2940 len 36864
27058 045/300 Flush
27059 046/300 Write block 1166 len 12288 data: 5 5 3
27060 047/300 Read block 1372 len 28672
27061 048/300 Read block 2059 len 20480
27062 049/300 Read block 1572 len 32768
27063 050/300 Read block 209 len 4096
27064 051/300 Read block 393 len 40960
27065 052/300 Write block 3419 len 4096 data: 5
27066 053/300 Write block 815 len 12288 data: 6 6 6
27067 054/300 Read block 4275 len 20480
27068 055/300 Write block 3385 len 4096 data: 5
27069 056/300 Read block 2340 len 32768
27070 057/300 Write block 4162 len 28672 data: 3 3 3 3 2 4 4
27071 058/300 Flush
27072 059/300 Read block 1801 len 24576
27073 060/300 Write block 3230 len 16384 data: 4 6 7 8
27074 061/300 Read block 2903 len 20480
27075 062/300 Write block 3660 len 20480 data: 7 3 7 8 7
27076 063/300 Read block 1587 len 24576
27077 064/300 Read block 329 len 8192
27078 065/300 Read block 1850 len 20480
27079 066/300 Write block 2321 len 8192 data: 5 5
27080 067/300 Write block 4680 len 8192 data: 5 6
27081 068/300 Read block 4762 len 20480
27082 069/300 Read block 2225 len 28672
27083 070/300 Read block 2689 len 36864
27084 071/300 Write block 144 len 24576 data: 4 4 5 5 4 3
27085 072/300 Write block 3976 len 36864 data: 3 3 3 3 4 4 7 6 6
27086 073/300 Read block 4405 len 16384
27087 074/300 Write block 3740 len 40960 data: 5 5 3 3 3 4 6 5 5 4
27088 075/300 Read block 738 len 28672
27089 076/300 Read block 568 len 12288
27090 077/300 Write block 4663 len 16384 data: 4 3 4 3
27091 078/300 Read block 924 len 16384
27092 079/300 Write block 2461 len 40960 data: 4 5 5 5 4 4 4 3 3 4
27093 080/300 Read block 2264 len 16384
27094 081/300 Write block 1750 len 4096 data: 4
27095 082/300 Write block 3225 len 32768 data: 6 6 3 3 5 5 7 8
27096 083/300 Read block 742 len 24576
27097 084/300 Read block 4378 len 12288
27098 085/300 Read block 1961 len 16384
27099 086/300 Read block 230 len 32768
27100 087/300 Write block 4179 len 12288 data: 7 7 7
27101 088/300 Write block 134 len 36864 data: 4 4 5 5 5 7 7 8 7
27102 089/300 Read block 3143 len 36864
27103 090/300 Read block 3229 len 36864
27104 091/300 Read block 4235 len 28672
27105 092/300 Write block 647 len 36864 data: 5 6 5 5 6 7 7 7 8
27106 093/300 Write block 3478 len 4096 data: 5
27107 094/300 Write block 669 len 12288 data: 8 9 9
27108 095/300 Read block 3524 len 40960
27109 096/300 Read block 2768 len 24576
27110 097/300 Write block 2570 len 32768 data: 4 6 5 5 7 7 8 3
27111 098/300 Read block 1029 len 24576
27112 099/300 Flush
27113 100/300 Read block 1101 len 24576
27114 101/300 Flush
27115 102/300 Flush
27116 103/300 Read block 3142 len 40960
27117 104/300 Read block 745 len 20480
27118 105/300 Read block 183 len 24576
27119 106/300 Write block 4485 len 4096 data: 5
27120 107/300 Read block 2196 len 36864
27121 108/300 Read block 2841 len 24576
27122 109/300 Read block 4365 len 28672
27123 110/300 Read block 2834 len 16384
27124 111/300 Read block 3327 len 24576
27125 112/300 Write block 2240 len 36864 data: 7 6 6 7 6 5 2 2 3
27126 113/300 Write block 92 len 8192 data: 2 2
27127 114/300 Write block 3911 len 16384 data: 6 7 7 7
27128 115/300 Write block 4924 len 40960 data: 6 6 6 5 6 5 4 6 6 6
27129 116/300 Write block 1676 len 40960 data: 4 4 4 4 5 7 7 6 6 4
27130 117/300 Write block 973 len 36864 data: 7 7 9 7 6 6 6 5 6
27131 118/300 Read block 87 len 8192
27132 119/300 Read block 1448 len 20480
27133 120/300 Flush
27134 121/300 Write block 3121 len 12288 data: 4 3 3
27135 122/300 Write block 3029 len 36864 data: 6 6 8 9 11 10 9 8 7
27136 123/300 Read block 4278 len 36864
27137 124/300 Write block 57 len 36864 data: 6 7 7 4 4 5 5 6 5
27138 125/300 Read block 896 len 12288
27139 126/300 Write block 2301 len 36864 data: 4 3 3 3 3 3 4 4 5
27140 127/300 Flush
27141 128/300 Read block 2835 len 36864
27142 129/300 Write block 4968 len 28672 data: 7 9 9 11 12 8 7
27143 130/300 Read block 230 len 28672
27144 131/300 Read block 4108 len 24576
27145 132/300 Write block 974 len 20480 data: 8 10 8 7 7
27146 133/300 Write block 3292 len 20480 data: 9 9 7 6 4
27147 134/300 Read block 3346 len 36864
27148 135/300 Write block 3038 len 40960 data: 6 6 6 6 5 6 8 6 6 6
27149 136/300 Flush
27150 137/300 Write block 1050 len 28672 data: 6 6 7 6 5 5 3
27151 138/300 Write block 1501 len 28672 data: 5 6 5 6 7 8 7
27152 139/300 Write block 3639 len 20480 data: 4 4 4 5 5
27153 140/300 Read block 4613 len 20480
27154 141/300 Read block 581 len 12288
27155 142/300 Read block 1926 len 24576
27156 143/300 Read block 216 len 24576
27157 144/300 Write block 1350 len 28672 data: 6 6 5 6 6 5 4
27158 145/300 Write block 3526 len 24576 data: 6 4 3 4 4 3
27159 146/300 Flush
27160 147/300 Read block 1767 len 36864
27161 148/300 Write block 3045 len 24576 data: 7 7 7 7 5 6
27162 149/300 Write block 887 len 4096 data: 6
27163 150/300 Read block 1295 len 40960
27164 151/300 Write block 2428 len 36864 data: 4 5 4 4 4 5 5 5 5
27165 152/300 Read block 3542 len 8192
27166 153/300 Write block 4854 len 28672 data: 3 3 3 4 4 4 4
27167 154/300 Flush
27168 155/300 Write block 3577 len 12288 data: 5 7 8
27169 156/300 Read block 87 len 32768
27170 157/300 Read block 800 len 32768
27171 158/300 Read block 1565 len 40960
27172 159/300 Flush
27173 160/300 Read block 1453 len 8192
27174 161/300 Write block 3079 len 40960 data: 2 2 3 3 3 3 3 4 4 4
27175 162/300 Flush
27176 163/300 Write block 2554 len 8192 data: 4 4
27177 164/300 Read block 1380 len 8192
27178 165/300 Write block 4457 len 16384 data: 5 6 6 6
27179 166/300 Write block 1969 len 16384 data: 7 8 6 6
27180 167/300 Read block 4491 len 4096
27181 168/300 Flush
27182 169/300 Write block 1996 len 36864 data: 4 4 4 5 5 6 6 6 6
27183 170/300 Read block 3157 len 28672
27184 171/300 Write block 1380 len 28672 data: 3 4 4 5 6 5 5
27185 172/300 Write block 412 len 8192 data: 3 5
27186 173/300 Write block 1202 len 20480 data: 3 3 4 4 5
27187 174/300 Read block 889 len 12288
27188 175/300 Read block 1301 len 24576
27189 176/300 Read block 1055 len 4096
27190 177/300 Write block 3554 len 4096 data: 6
27191 178/300 Read block 158 len 8192
27192 179/300 Read block 700 len 4096
27193 180/300 Write block 949 len 32768 data: 5 5 5 4 4 4 4 3
27194 181/300 Read block 4171 len 24576
27195 182/300 Read block 245 len 28672
27196 183/300 Read block 1603 len 32768
27197 184/300 Read block 3824 len 24576
27198 185/300 Write block 3726 len 28672 data: 7 7 8 8 6 4 6
27199 186/300 Read block 378 len 16384
27200 187/300 Read block 1136 len 24576
27201 188/300 Write block 4519 len 12288 data: 7 8 6
27202 189/300 Write block 1044 len 20480 data: 5 5 5 5 4
27203 190/300 Read block 4143 len 12288
27204 191/300 Read block 1866 len 32768
27205 192/300 Write block 2554 len 32768 data: 5 5 4 4 4 2 3 4
27206 193/300 Read block 3241 len 8192
27207 194/300 Write block 2420 len 28672 data: 8 8 5 6 6 5 5
27208 195/300 Write block 2498 len 20480 data: 8 8 6 7 8
27209 196/300 Read block 3286 len 8192
27210 197/300 Write block 3875 len 28672 data: 7 6 5 6 7 7 7
27211 198/300 Write block 2044 len 24576 data: 5 5 5 4 5 5
27212 199/300 Read block 1150 len 4096
27213 200/300 Write block 4071 len 12288 data: 2 2 2
27214 201/300 Write block 4316 len 24576 data: 8 5 6 6 6 5
27215 202/300 Read block 3900 len 24576
27216 203/300 Write block 4263 len 40960 data: 5 7 6 6 8 9 10 10 10 7
27217 204/300 Read block 1067 len 28672
27218 205/300 Write block 1207 len 20480 data: 5 6 5 3 2
27219 206/300 Read block 3779 len 8192
27220 207/300 Read block 4719 len 20480
27221 208/300 Read block 738 len 24576
27222 209/300 Write block 3490 len 16384 data: 4 6 7 7
27223 210/300 Read block 759 len 32768
27224 211/300 Write block 3811 len 8192 data: 9 8
27225 212/300 Write block 85 len 40960 data: 4 3 3 3 2 2 2 3 3 2
27226 213/300 Write block 3454 len 28672 data: 4 5 6 6 6 6 8
27227 214/300 Read block 1887 len 40960
27228 215/300 Read block 2380 len 4096
27229 216/300 Write block 2591 len 12288 data: 7 5 5
27230 217/300 Flush
27231 218/300 Write block 4842 len 16384 data: 4 3 3 5
27232 219/300 Write block 4660 len 24576 data: 5 6 4 5 4 5
27233 220/300 Read block 3910 len 4096
27234 221/300 Write block 2564 len 28672 data: 5 4 3 5 5 4 5
27235 222/300 Read block 1423 len 8192
27236 223/300 Write block 4437 len 32768 data: 6 6 5 5 6 6 6 6
27237 224/300 Write block 2668 len 16384 data: 6 7 7 5
27238 225/300 Read block 41 len 28672
27239 226/300 Flush
27240 227/300 Write block 2689 len 24576 data: 7 7 5 4 4 4
27241 228/300 Read block 3895 len 36864
27242 229/300 Read block 1028 len 12288
27243 230/300 Flush
27244 231/300 Write block 3494 len 36864 data: 7 8 8 9 8 9 7 8 9
27245 232/300 Read block 647 len 40960
27246 233/300 Write block 3069 len 8192 data: 7 5
27247 234/300 Read block 4423 len 36864
27248 235/300 Write block 3223 len 4096 data: 5
27249 236/300 Read block 4736 len 12288
27250 237/300 Write block 2114 len 16384 data: 6 6 6 6
27251 238/300 Write block 593 len 24576 data: 5 4 7 5 4 4
27252 239/300 Write block 2847 len 4096 data: 4
27253 240/300 Write block 4847 len 36864 data: 5 5 5 4 4 4 3 4 4
27254 241/300 Read block 1445 len 32768
27255 242/300 Write block 4406 len 32768 data: 6 5 3 5 5 5 7 7
27256 243/300 Read block 4541 len 12288
27257 244/300 Write block 3851 len 4096 data: 7
27258 245/300 Read block 2429 len 24576
27259 246/300 Read block 3801 len 12288
27260 247/300 Read block 3156 len 16384
27261 248/300 Read block 4679 len 8192
27262 249/300 Flush
27263 250/300 Read block 4337 len 24576
27264 251/300 Write block 679 len 8192 data: 2 2
27265 252/300 Read block 4085 len 16384
27266 253/300 Write block 2455 len 20480 data: 5 4 5 4 4
27267 254/300 Read block 2768 len 24576
27268 255/300 Read block 3012 len 4096
27269 256/300 Read block 67 len 8192
27270 257/300 Write block 4508 len 8192 data: 5 6
27271 258/300 Write block 62 len 28672 data: 6 6 7 6 5 8 9
27272 259/300 Write block 3938 len 40960 data: 4 3 5 6 6 7 7 6 7 7
27273 260/300 Read block 1388 len 28672
27274 261/300 Read block 2177 len 28672
27275 262/300 Write block 1412 len 12288 data: 5 3 3
27276 263/300 Write block 2957 len 12288 data: 7 6 6
27277 264/300 Read block 840 len 16384
27278 265/300 Read block 3853 len 20480
27279 266/300 Flush
27280 267/300 Read block 1076 len 40960
27281 268/300 Read block 1945 len 20480
27282 269/300 Write block 969 len 4096 data: 3
27283 270/300 Write block 3530 len 4096 data: 5
27284 271/300 Write block 1230 len 12288 data: 2 3 5
27285 272/300 Read block 4422 len 40960
27286 273/300 Write block 1236 len 4096 data: 8
27287 274/300 Write block 3286 len 4096 data: 3
27288 275/300 Read block 379 len 40960
27289 276/300 Read block 3595 len 40960
27290 277/300 Read block 3613 len 8192
27291 278/300 Read block 2129 len 16384
27292 279/300 Read block 1620 len 28672
27293 280/300 Write block 4838 len 32768 data: 3 3 3 5 5 4 4 6
27294 281/300 Read block 4173 len 4096
27295 282/300 Read block 990 len 32768
27296 283/300 Read block 1931 len 12288
27297 284/300 Read block 428 len 16384
27298 285/300 Write block 2610 len 24576 data: 6 8 9 8 8 7
27299 286/300 Write block 892 len 24576 data: 6 6 5 4 3 3
27300 287/300 Write block 3446 len 16384 data: 5 5 4 4
27301 288/300 Write block 117 len 20480 data: 6 6 6 3 3
27302 289/300 Read block 671 len 24576
27303 290/300 Read block 3 len 36864
27304 291/300 Read block 811 len 32768
27305 292/300 Read block 971 len 8192
27306 293/300 Write block 2683 len 16384 data: 4 8 9 9
27307 294/300 Read block 1322 len 20480
27308 295/300 Write block 2340 len 4096 data: 6
27309 296/300 Write block 1723 len 12288 data: 5 5 5
27310 297/300 Read block 53 len 28672
27311 298/300 Read block 2008 len 12288
27312 299/300 Read block 4177 len 12288
27313 300/300 Read block 980 len 12288
27314 301/300 Write block 4228 len 4096 data: 4
27315 Main task has work to do, go find it
27316 got dsc Start(0)
27317 start 0
27318 Jan 23 17:43:30.178 INFO request completed, latency_us: 224, response_code: 204, uri: /start/cid/0, method: POST, req_id: 0add508f-692a-412c-adfc-7e143326093a, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
27319 Replay: started 0, returned:Ok(())
27320 ----------------------------------------------------------------
27321 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27322 States: Offline Active Active
27323 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27324 8549 Acked Read 9 Sent Done Done false
27325 8550 Acked Write 6 Sent Done Done false
27326 8551 Acked Read 3 Sent Done Done false
27327 8552 Acked Read 2 Sent Done Done false
27328 8553 Acked Write 5 Sent Done Done false
27329 8554 Acked Read 6 Sent Done Done false
27330 8555 Acked Read 2 Sent Done Done false
27331 8556 Acked Write 5 Sent Done Done false
27332 8557 Acked Write 9 Sent Done Done false
27333 8558 Acked Write 3 Sent Done Done false
27334 8559 Acked Write 7 Sent Done Done false
27335 8560 Acked Read 6 Sent Done Done false
27336 8561 Acked Write 4 Sent Done Done false
27337 8562 Acked Read 4 Sent Done Done false
27338 8563 Acked Read 9 Sent Done Done false
27339 8564 Acked Write 1 Sent Done Done false
27340 8565 Acked Write 3 Sent Done Done false
27341 8566 Acked Read 1 Sent Done Done false
27342 8567 Acked Write 1 Sent Done Done false
27343 8568 Acked Write 5 Sent Done Done false
27344 8569 Acked Read 7 Sent Done Done false
27345 8570 Acked Write 9 Sent Done Done false
27346 8571 Acked Flush 0 Sent Done Done false
27347 8572 Acked Write 4 Sent Done Done false
27348 8573 Acked Read 5 Sent Done Done false
27349 8574 Acked Read 3 Sent Done Done false
27350 8575 Acked Read 5 Sent Done Done false
27351 8576 Acked Write 9 Sent Done Done false
27352 8577 Acked Read 5 Sent Done Done false
27353 8578 Acked Write 4 Sent Done Done false
27354 8579 Acked Write 3 Sent Done Done false
27355 8580 Acked Write 3 Sent Done Done false
27356 8581 Acked Write 8 Sent Done Done false
27357 8582 Acked Write 7 Sent Done Done false
27358 8583 Acked Read 5 Sent Done Done false
27359 8584 Acked Write 3 Sent Done Done false
27360 8585 Acked Read 3 Sent Done Done false
27361 8586 Acked Read 3 Sent Done Done false
27362 8587 Acked Write 4 Sent Done Done false
27363 8588 Acked Read 7 Sent Done Done false
27364 8589 Acked Read 8 Sent Done Done false
27365 8590 Acked Write 7 Sent Done Done false
27366 8591 Acked Write 9 Sent Done Done false
27367 8592 Acked Read 9 Sent Done Done false
27368 8593 Acked Flush 0 Sent Done Done false
27369 8594 Acked Write 3 Sent Done Done false
27370 8595 Acked Read 7 Sent Done Done false
27371 8596 Acked Read 5 Sent Done Done false
27372 8597 Acked Read 8 Sent Done Done false
27373 8598 Acked Read 1 Sent Done Done false
27374 8599 Acked Read 10 Sent Done Done false
27375 8600 Acked Write 1 Sent Done Done false
27376 8601 Acked Write 3 Sent Done Done false
27377 8602 Acked Read 5 Sent Done Done false
27378 8603 Acked Write 1 Sent Done Done false
27379 8604 Acked Read 8 Sent Done Done false
27380 8605 Acked Write 7 Sent Done Done false
27381 8606 Acked Flush 0 Sent Done Done false
27382 8607 Acked Read 6 Sent Done Done false
27383 8608 Acked Write 4 Sent Done Done false
27384 8609 Acked Read 5 Sent Done Done false
27385 8610 Acked Write 5 Sent Done Done false
27386 8611 Acked Read 6 Sent Done Done false
27387 8612 Acked Read 2 Sent Done Done false
27388 8613 Acked Read 5 Sent Done Done false
27389 8614 Acked Write 2 Sent Done Done false
27390 8615 Acked Write 2 Sent Done Done false
27391 8616 Acked Read 5 Sent Done Done false
27392 8617 Acked Read 7 Sent Done Done false
27393 8618 Acked Read 9 Sent Done Done false
27394 8619 Acked Write 6 Sent Done Done false
27395 8620 Acked Write 9 Sent Done Done false
27396 8621 Acked Read 4 Sent Done Done false
27397 8622 Acked Write 10 Sent Done Done false
27398 8623 Acked Read 7 Sent Done Done false
27399 8624 Acked Read 3 Sent Done Done false
27400 8625 Acked Write 4 Sent Done Done false
27401 8626 Acked Read 4 Sent Done Done false
27402 8627 Acked Write 10 Sent Done Done false
27403 8628 Acked Read 4 Sent Done Done false
27404 8629 Acked Write 1 Sent Done Done false
27405 8630 Acked Write 8 Sent Done Done false
27406 8631 Acked Read 6 Sent Done Done false
27407 8632 Acked Read 3 Sent Done Done false
27408 8633 Acked Read 4 Sent Done Done false
27409 8634 Acked Read 8 Sent Done Done false
27410 8635 Acked Write 3 Sent Done Done false
27411 8636 Acked Write 9 Sent Done Done false
27412 8637 Acked Read 9 Sent Done Done false
27413 8638 Acked Read 9 Sent Done Done false
27414 8639 Acked Read 7 Sent Done Done false
27415 8640 Acked Write 9 Sent Done Done false
27416 8641 Acked Write 1 Sent Done Done false
27417 8642 Acked Write 3 Sent Done Done false
27418 8643 Acked Read 10 Sent Done Done false
27419 8644 Acked Read 6 Sent Done Done false
27420 8645 Acked Write 8 Sent Done Done false
27421 8646 Acked Read 6 Sent Done Done false
27422 8647 Acked Flush 0 Sent Done Done false
27423 8648 Acked Read 6 Sent Done Done false
27424 8649 Acked Flush 0 Sent Done Done false
27425 8650 Acked Flush 0 Sent Done Done false
27426 8651 Acked Read 10 Sent Done Done false
27427 8652 Acked Read 5 Sent Done Done false
27428 8653 Acked Read 6 Sent Done Done false
27429 8654 Acked Write 1 Sent Done Done false
27430 8655 Acked Read 9 Sent Done Done false
27431 8656 Acked Read 6 Sent Done Done false
27432 8657 Acked Read 7 Sent Done Done false
27433 8658 Acked Read 4 Sent Done Done false
27434 8659 Acked Read 6 Sent Done Done false
27435 8660 Acked Write 9 Sent Done Done false
27436 8661 Acked Write 2 Sent Done Done false
27437 8662 Acked Write 4 Sent Done Done false
27438 8663 Acked Write 10 Sent Done Done false
27439 8664 Acked Write 10 Sent Done Done false
27440 8665 Acked Write 9 Sent Done Done false
27441 8666 Acked Read 2 Sent Done Done false
27442 8667 Acked Read 5 Sent Done Done false
27443 8668 Acked Flush 0 Sent Done Done false
27444 8669 Acked Write 3 Sent Done Done false
27445 8670 Acked Write 9 Sent Done Done false
27446 8671 Acked Read 9 Sent Done Done false
27447 8672 Acked Write 9 Sent Done Done false
27448 8673 Acked Read 3 Sent Done Done false
27449 8674 Acked Write 9 Sent Done Done false
27450 8675 Acked Flush 0 Sent Done Done false
27451 8676 Acked Read 9 Sent Done Done false
27452 8677 Acked Write 7 Sent Done Done false
27453 8678 Acked Read 7 Sent Done Done false
27454 8679 Acked Read 6 Sent Done Done false
27455 8680 Acked Write 5 Sent Done Done false
27456 8681 Acked Write 5 Sent Done Done false
27457 8682 Acked Read 9 Sent Done Done false
27458 8683 Acked Write 10 Sent Done Done false
27459 8684 Acked Flush 0 Sent Done Done false
27460 8685 Acked Write 7 Sent Done Done false
27461 8686 Acked Write 7 Sent Done Done false
27462 8687 Acked Write 5 Sent Done Done false
27463 8688 Acked Read 5 Sent Done Done false
27464 8689 Acked Read 3 Sent Done Done false
27465 8690 Acked Read 6 Sent Done Done false
27466 8691 Acked Read 6 Sent Done Done false
27467 8692 Acked Write 7 Sent Done Done false
27468 8693 Acked Write 6 Sent Done Done false
27469 8694 Acked Flush 0 Sent Done Done false
27470 8695 Acked Read 9 Sent Done Done false
27471 8696 Acked Write 6 Sent Done Done false
27472 8697 Acked Write 1 Sent Done Done false
27473 8698 Acked Read 10 Sent Done Done false
27474 8699 Acked Write 9 Sent Done Done false
27475 8700 Acked Read 2 Sent Done Done false
27476 8701 Acked Write 7 Sent Done Done false
27477 8702 Acked Flush 0 Sent Done Done false
27478 8703 Acked Write 3 Sent Done Done false
27479 8704 Acked Read 8 Sent Done Done false
27480 8705 Acked Read 8 Sent Done Done false
27481 8706 Acked Read 10 Sent Done Done false
27482 8707 Acked Flush 0 Sent Done Done false
27483 8708 Acked Read 2 Sent Done Done false
27484 8709 Acked Write 10 Sent Done Done false
27485 8710 Acked Flush 0 Sent Done Done false
27486 8711 Acked Write 2 Sent Done Done false
27487 8712 Acked Read 2 Sent Done Done false
27488 8713 Acked Write 4 Sent Done Done false
27489 8714 Acked Write 4 Sent Done Done false
27490 8715 Acked Read 1 Sent Done Done false
27491 8716 Acked Flush 0 Sent Done Done false
27492 8717 Acked Write 9 Sent Done Done false
27493 8718 Acked Read 7 Sent Done Done false
27494 8719 Acked Write 7 Sent Done Done false
27495 8720 Acked Write 2 Sent Done Done false
27496 8721 Acked Write 5 Sent Done Done false
27497 8722 Acked Read 3 Sent Done Done false
27498 8723 Acked Read 6 Sent Done Done false
27499 8724 Acked Read 1 Sent Done Done false
27500 8725 Acked Write 1 Sent Done Done false
27501 8726 Acked Read 2 Sent Done Done false
27502 8727 Acked Read 1 Sent Done Done false
27503 8728 Acked Write 8 Sent Done Done false
27504 8729 Acked Read 6 Sent Done Done false
27505 8730 Acked Read 7 Sent Done Done false
27506 8731 Acked Read 8 Sent Done Done false
27507 8732 Acked Read 6 Sent Done Done false
27508 8733 Acked Write 7 Sent Done Done false
27509 8734 Acked Read 4 Sent Done Done false
27510 8735 Acked Read 6 Sent Done Done false
27511 8736 Acked Write 3 Sent Done Done false
27512 8737 Acked Write 5 Sent Done Done false
27513 8738 Acked Read 3 Sent Done Done false
27514 8739 Acked Read 8 Sent Done Done false
27515 8740 Acked Write 8 Sent Done Done false
27516 8741 Acked Read 2 Sent Done Done false
27517 8742 Acked Write 7 Sent Done Done false
27518 8743 Acked Write 5 Sent Done Done false
27519 8744 Acked Read 2 Sent Done Done false
27520 8745 Acked Write 7 Sent Done Done false
27521 8746 Acked Write 6 Sent Done Done false
27522 8747 Acked Read 1 Sent Done Done false
27523 8748 Acked Write 3 Sent Done Done false
27524 8749 Acked Write 6 Sent Done Done false
27525 8750 Acked Read 6 Sent Done Done false
27526 8751 Acked Write 10 Sent Done Done false
27527 8752 Acked Read 7 Sent Done Done false
27528 8753 Acked Write 5 Sent Done Done false
27529 8754 Acked Read 2 Sent Done Done false
27530 8755 Acked Read 5 Sent Done Done false
27531 8756 Acked Read 6 Sent Done Done false
27532 8757 Acked Write 4 Sent Done Done false
27533 8758 Acked Read 8 Sent Done Done false
27534 8759 Acked Write 2 Sent Done Done false
27535 8760 Acked Write 10 Sent Done Done false
27536 8761 Acked Write 7 Sent Done Done false
27537 8762 Acked Read 10 Sent Done Done false
27538 8763 Acked Read 1 Sent Done Done false
27539 8764 Acked Write 3 Sent Done Done false
27540 8765 Acked Flush 0 Sent Done Done false
27541 8766 Acked Write 4 Sent Done Done false
27542 8767 Acked Write 6 Sent Done Done false
27543 8768 Acked Read 1 Sent Done Done false
27544 8769 Acked Write 7 Sent Done Done false
27545 8770 Acked Read 2 Sent Done Done false
27546 8771 Acked Write 8 Sent Done Done false
27547 8772 Acked Write 4 Sent Done Done false
27548 8773 Acked Read 7 Sent Done Done false
27549 8774 Acked Flush 0 Sent Done Done false
27550 8775 Acked Write 6 Sent Done Done false
27551 8776 Acked Read 9 Sent Done Done false
27552 8777 Acked Read 3 Sent Done Done false
27553 8778 Acked Flush 0 Sent Done Done false
27554 8779 Acked Write 9 Sent Done Done false
27555 8780 Acked Read 10 Sent Done Done false
27556 8781 Acked Write 2 Sent Done Done false
27557 8782 Acked Read 9 Sent Done Done false
27558 8783 Acked Write 1 Sent Done Done false
27559 8784 Acked Read 3 Sent Done Done false
27560 8785 Acked Write 4 Sent Done Done false
27561 8786 Acked Write 6 Sent Done Done false
27562 8787 Acked Write 1 Sent Done Done false
27563 8788 Acked Write 9 Sent Done Done false
27564 8789 Acked Read 8 Sent Done Done false
27565 8790 Acked Write 8 Sent Done Done false
27566 8791 Acked Read 3 Sent Done Done false
27567 8792 Acked Write 1 Sent Done Done false
27568 8793 Acked Read 6 Sent Done Done false
27569 8794 Acked Read 3 Sent Done Done false
27570 8795 Acked Read 4 Sent Done Done false
27571 8796 Acked Read 2 Sent Done Done false
27572 8797 Acked Flush 0 Sent Done Done false
27573 8798 Acked Read 6 Sent Done Done false
27574 8799 Acked Write 2 Sent Done Done false
27575 8800 Acked Read 4 Sent Done Done false
27576 8801 Acked Write 5 Sent Done Done false
27577 8802 Acked Read 6 Sent Done Done false
27578 8803 Acked Read 1 Sent Done Done false
27579 8804 Acked Read 2 Sent Done Done false
27580 8805 Acked Write 2 Sent Done Done false
27581 8806 Acked Write 7 Sent Done Done false
27582 8807 Acked Write 10 Sent Done Done false
27583 8808 Acked Read 7 Sent Done Done false
27584 8809 Acked Read 7 Sent Done Done false
27585 8810 Acked Write 3 Sent Done Done false
27586 8811 Acked Write 3 Sent Done Done false
27587 8812 Acked Read 4 Sent Done Done false
27588 8813 Acked Read 5 Sent Done Done false
27589 8814 Acked Flush 0 Sent Done Done false
27590 8815 Acked Read 10 Sent Done Done false
27591 8816 Acked Read 5 Sent Done Done false
27592 8817 Acked Write 1 Sent Done Done false
27593 8818 Acked Write 1 Sent Done Done false
27594 8819 Acked Write 3 Sent Done Done false
27595 8820 Acked Read 10 Sent Done Done false
27596 8821 Acked Write 1 Sent Done Done false
27597 8822 Acked Write 1 Sent Done Done false
27598 8823 Acked Read 10 Sent Done Done false
27599 8824 Acked Read 10 Sent Done Done false
27600 8825 Acked Read 2 Sent Done Done false
27601 8826 Acked Read 4 Sent Done Done false
27602 8827 Acked Read 7 Sent Done Done false
27603 8828 Acked Write 8 Sent Done Done false
27604 8829 Acked Read 1 Sent Done Done false
27605 8830 Acked Read 8 Sent Done Done false
27606 8831 Acked Read 3 Sent Done Done false
27607 8832 Acked Read 4 Sent Done Done false
27608 8833 Acked Write 6 Sent Done Done false
27609 8834 Acked Write 6 Sent Done Done false
27610 8835 Acked Write 4 Sent Done Done false
27611 8836 Acked Write 5 Sent Done Done false
27612 8837 Acked Read 6 Sent Done Done false
27613 8838 Acked Read 9 Sent Done Done false
27614 8839 Acked Read 8 Sent Done Done false
27615 8840 Acked Read 2 Sent Done Done false
27616 8841 Acked Write 4 Sent Done Done false
27617 8842 Acked Read 5 Sent Done Done false
27618 8843 Acked Write 1 Sent Done Done false
27619 8844 Acked Write 3 Sent Done Done false
27620 8845 Acked Read 7 Sent Done Done false
27621 8846 Acked Read 3 Sent Done Done false
27622 8847 Acked Read 3 Sent Done Done false
27623 8848 Acked Read 3 Sent Done Done false
27624 8849 Acked Write 1 Sent Done Done false
27625 STATES DS:0 DS:1 DS:2 TOTAL
27626 Sent 301 0 0 301
27627 Done 0 301 301 602
27628 Skipped 0 0 0 0
27629 Error 0 0 0 0
27630 Last Flush: Some(JobId(8548)) Some(JobId(8814)) Some(JobId(8814))
27631 Downstairs last five completed: 8548 8547 8546 8545 8544
27632 Upstairs last five completed: 8849 8848 8847 8846 8845
27633 CLIENT: Up:0 ds:301 act:2
27634 [8810] got start action
27635 [8810] I am going to restart
27636 Starting downstairs at port 8810
27637 Make output file at "/tmp/dsc/downstairs-8810.txt"
27638 [8810][0] reports Starting
27639 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1231)
27640 [8810][0] reports Running
27641 ----------------------------------------------------------------
27642 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
27643 States: Offline Active Active
27644 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27645 8549 Acked Read 9 Sent Done Done false
27646 8550 Acked Write 6 Sent Done Done false
27647 8551 Acked Read 3 Sent Done Done false
27648 8552 Acked Read 2 Sent Done Done false
27649 8553 Acked Write 5 Sent Done Done false
27650 8554 Acked Read 6 Sent Done Done false
27651 8555 Acked Read 2 Sent Done Done false
27652 8556 Acked Write 5 Sent Done Done false
27653 8557 Acked Write 9 Sent Done Done false
27654 8558 Acked Write 3 Sent Done Done false
27655 8559 Acked Write 7 Sent Done Done false
27656 8560 Acked Read 6 Sent Done Done false
27657 8561 Acked Write 4 Sent Done Done false
27658 8562 Acked Read 4 Sent Done Done false
27659 8563 Acked Read 9 Sent Done Done false
27660 8564 Acked Write 1 Sent Done Done false
27661 8565 Acked Write 3 Sent Done Done false
27662 8566 Acked Read 1 Sent Done Done false
27663 8567 Acked Write 1 Sent Done Done false
27664 8568 Acked Write 5 Sent Done Done false
27665 8569 Acked Read 7 Sent Done Done false
27666 8570 Acked Write 9 Sent Done Done false
27667 8571 Acked Flush 0 Sent Done Done false
27668 8572 Acked Write 4 Sent Done Done false
27669 8573 Acked Read 5 Sent Done Done false
27670 8574 Acked Read 3 Sent Done Done false
27671 8575 Acked Read 5 Sent Done Done false
27672 8576 Acked Write 9 Sent Done Done false
27673 8577 Acked Read 5 Sent Done Done false
27674 8578 Acked Write 4 Sent Done Done false
27675 8579 Acked Write 3 Sent Done Done false
27676 8580 Acked Write 3 Sent Done Done false
27677 8581 Acked Write 8 Sent Done Done false
27678 8582 Acked Write 7 Sent Done Done false
27679 8583 Acked Read 5 Sent Done Done false
27680 8584 Acked Write 3 Sent Done Done false
27681 8585 Acked Read 3 Sent Done Done false
27682 8586 Acked Read 3 Sent Done Done false
27683 8587 Acked Write 4 Sent Done Done false
27684 8588 Acked Read 7 Sent Done Done false
27685 8589 Acked Read 8 Sent Done Done false
27686 8590 Acked Write 7 Sent Done Done false
27687 8591 Acked Write 9 Sent Done Done false
27688 8592 Acked Read 9 Sent Done Done false
27689 8593 Acked Flush 0 Sent Done Done false
27690 8594 Acked Write 3 Sent Done Done false
27691 8595 Acked Read 7 Sent Done Done false
27692 8596 Acked Read 5 Sent Done Done false
27693 8597 Acked Read 8 Sent Done Done false
27694 8598 Acked Read 1 Sent Done Done false
27695 8599 Acked Read 10 Sent Done Done false
27696 8600 Acked Write 1 Sent Done Done false
27697 8601 Acked Write 3 Sent Done Done false
27698 8602 Acked Read 5 Sent Done Done false
27699 8603 Acked Write 1 Sent Done Done false
27700 8604 Acked Read 8 Sent Done Done false
27701 8605 Acked Write 7 Sent Done Done false
27702 8606 Acked Flush 0 Sent Done Done false
27703 8607 Acked Read 6 Sent Done Done false
27704 8608 Acked Write 4 Sent Done Done false
27705 8609 Acked Read 5 Sent Done Done false
27706 8610 Acked Write 5 Sent Done Done false
27707 8611 Acked Read 6 Sent Done Done false
27708 8612 Acked Read 2 Sent Done Done false
27709 8613 Acked Read 5 Sent Done Done false
27710 8614 Acked Write 2 Sent Done Done false
27711 8615 Acked Write 2 Sent Done Done false
27712 8616 Acked Read 5 Sent Done Done false
27713 8617 Acked Read 7 Sent Done Done false
27714 8618 Acked Read 9 Sent Done Done false
27715 8619 Acked Write 6 Sent Done Done false
27716 8620 Acked Write 9 Sent Done Done false
27717 8621 Acked Read 4 Sent Done Done false
27718 8622 Acked Write 10 Sent Done Done false
27719 8623 Acked Read 7 Sent Done Done false
27720 8624 Acked Read 3 Sent Done Done false
27721 8625 Acked Write 4 Sent Done Done false
27722 8626 Acked Read 4 Sent Done Done false
27723 8627 Acked Write 10 Sent Done Done false
27724 8628 Acked Read 4 Sent Done Done false
27725 8629 Acked Write 1 Sent Done Done false
27726 8630 Acked Write 8 Sent Done Done false
27727 8631 Acked Read 6 Sent Done Done false
27728 8632 Acked Read 3 Sent Done Done false
27729 8633 Acked Read 4 Sent Done Done false
27730 8634 Acked Read 8 Sent Done Done false
27731 8635 Acked Write 3 Sent Done Done false
27732 8636 Acked Write 9 Sent Done Done false
27733 8637 Acked Read 9 Sent Done Done false
27734 8638 Acked Read 9 Sent Done Done false
27735 8639 Acked Read 7 Sent Done Done false
27736 8640 Acked Write 9 Sent Done Done false
27737 8641 Acked Write 1 Sent Done Done false
27738 8642 Acked Write 3 Sent Done Done false
27739 8643 Acked Read 10 Sent Done Done false
27740 8644 Acked Read 6 Sent Done Done false
27741 8645 Acked Write 8 Sent Done Done false
27742 8646 Acked Read 6 Sent Done Done false
27743 8647 Acked Flush 0 Sent Done Done false
27744 8648 Acked Read 6 Sent Done Done false
27745 8649 Acked Flush 0 Sent Done Done false
27746 8650 Acked Flush 0 Sent Done Done false
27747 8651 Acked Read 10 Sent Done Done false
27748 8652 Acked Read 5 Sent Done Done false
27749 8653 Acked Read 6 Sent Done Done false
27750 8654 Acked Write 1 Sent Done Done false
27751 8655 Acked Read 9 Sent Done Done false
27752 8656 Acked Read 6 Sent Done Done false
27753 8657 Acked Read 7 Sent Done Done false
27754 8658 Acked Read 4 Sent Done Done false
27755 8659 Acked Read 6 Sent Done Done false
27756 8660 Acked Write 9 Sent Done Done false
27757 8661 Acked Write 2 Sent Done Done false
27758 8662 Acked Write 4 Sent Done Done false
27759 8663 Acked Write 10 Sent Done Done false
27760 8664 Acked Write 10 Sent Done Done false
27761 8665 Acked Write 9 Sent Done Done false
27762 8666 Acked Read 2 Sent Done Done false
27763 8667 Acked Read 5 Sent Done Done false
27764 8668 Acked Flush 0 Sent Done Done false
27765 8669 Acked Write 3 Sent Done Done false
27766 8670 Acked Write 9 Sent Done Done false
27767 8671 Acked Read 9 Sent Done Done false
27768 8672 Acked Write 9 Sent Done Done false
27769 8673 Acked Read 3 Sent Done Done false
27770 8674 Acked Write 9 Sent Done Done false
27771 8675 Acked Flush 0 Sent Done Done false
27772 8676 Acked Read 9 Sent Done Done false
27773 8677 Acked Write 7 Sent Done Done false
27774 8678 Acked Read 7 Sent Done Done false
27775 8679 Acked Read 6 Sent Done Done false
27776 8680 Acked Write 5 Sent Done Done false
27777 8681 Acked Write 5 Sent Done Done false
27778 8682 Acked Read 9 Sent Done Done false
27779 8683 Acked Write 10 Sent Done Done false
27780 8684 Acked Flush 0 Sent Done Done false
27781 8685 Acked Write 7 Sent Done Done false
27782 8686 Acked Write 7 Sent Done Done false
27783 8687 Acked Write 5 Sent Done Done false
27784 8688 Acked Read 5 Sent Done Done false
27785 8689 Acked Read 3 Sent Done Done false
27786 8690 Acked Read 6 Sent Done Done false
27787 8691 Acked Read 6 Sent Done Done false
27788 8692 Acked Write 7 Sent Done Done false
27789 8693 Acked Write 6 Sent Done Done false
27790 8694 Acked Flush 0 Sent Done Done false
27791 8695 Acked Read 9 Sent Done Done false
27792 8696 Acked Write 6 Sent Done Done false
27793 8697 Acked Write 1 Sent Done Done false
27794 8698 Acked Read 10 Sent Done Done false
27795 8699 Acked Write 9 Sent Done Done false
27796 8700 Acked Read 2 Sent Done Done false
27797 8701 Acked Write 7 Sent Done Done false
27798 8702 Acked Flush 0 Sent Done Done false
27799 8703 Acked Write 3 Sent Done Done false
27800 8704 Acked Read 8 Sent Done Done false
27801 8705 Acked Read 8 Sent Done Done false
27802 8706 Acked Read 10 Sent Done Done false
27803 8707 Acked Flush 0 Sent Done Done false
27804 8708 Acked Read 2 Sent Done Done false
27805 8709 Acked Write 10 Sent Done Done false
27806 8710 Acked Flush 0 Sent Done Done false
27807 8711 Acked Write 2 Sent Done Done false
27808 8712 Acked Read 2 Sent Done Done false
27809 8713 Acked Write 4 Sent Done Done false
27810 8714 Acked Write 4 Sent Done Done false
27811 8715 Acked Read 1 Sent Done Done false
27812 8716 Acked Flush 0 Sent Done Done false
27813 8717 Acked Write 9 Sent Done Done false
27814 8718 Acked Read 7 Sent Done Done false
27815 8719 Acked Write 7 Sent Done Done false
27816 8720 Acked Write 2 Sent Done Done false
27817 8721 Acked Write 5 Sent Done Done false
27818 8722 Acked Read 3 Sent Done Done false
27819 8723 Acked Read 6 Sent Done Done false
27820 8724 Acked Read 1 Sent Done Done false
27821 8725 Acked Write 1 Sent Done Done false
27822 8726 Acked Read 2 Sent Done Done false
27823 8727 Acked Read 1 Sent Done Done false
27824 8728 Acked Write 8 Sent Done Done false
27825 8729 Acked Read 6 Sent Done Done false
27826 8730 Acked Read 7 Sent Done Done false
27827 8731 Acked Read 8 Sent Done Done false
27828 8732 Acked Read 6 Sent Done Done false
27829 8733 Acked Write 7 Sent Done Done false
27830 8734 Acked Read 4 Sent Done Done false
27831 8735 Acked Read 6 Sent Done Done false
27832 8736 Acked Write 3 Sent Done Done false
27833 8737 Acked Write 5 Sent Done Done false
27834 8738 Acked Read 3 Sent Done Done false
27835 8739 Acked Read 8 Sent Done Done false
27836 8740 Acked Write 8 Sent Done Done false
27837 8741 Acked Read 2 Sent Done Done false
27838 8742 Acked Write 7 Sent Done Done false
27839 8743 Acked Write 5 Sent Done Done false
27840 8744 Acked Read 2 Sent Done Done false
27841 8745 Acked Write 7 Sent Done Done false
27842 8746 Acked Write 6 Sent Done Done false
27843 8747 Acked Read 1 Sent Done Done false
27844 8748 Acked Write 3 Sent Done Done false
27845 8749 Acked Write 6 Sent Done Done false
27846 8750 Acked Read 6 Sent Done Done false
27847 8751 Acked Write 10 Sent Done Done false
27848 8752 Acked Read 7 Sent Done Done false
27849 8753 Acked Write 5 Sent Done Done false
27850 8754 Acked Read 2 Sent Done Done false
27851 8755 Acked Read 5 Sent Done Done false
27852 8756 Acked Read 6 Sent Done Done false
27853 8757 Acked Write 4 Sent Done Done false
27854 8758 Acked Read 8 Sent Done Done false
27855 8759 Acked Write 2 Sent Done Done false
27856 8760 Acked Write 10 Sent Done Done false
27857 8761 Acked Write 7 Sent Done Done false
27858 8762 Acked Read 10 Sent Done Done false
27859 8763 Acked Read 1 Sent Done Done false
27860 8764 Acked Write 3 Sent Done Done false
27861 8765 Acked Flush 0 Sent Done Done false
27862 8766 Acked Write 4 Sent Done Done false
27863 8767 Acked Write 6 Sent Done Done false
27864 8768 Acked Read 1 Sent Done Done false
27865 8769 Acked Write 7 Sent Done Done false
27866 8770 Acked Read 2 Sent Done Done false
27867 8771 Acked Write 8 Sent Done Done false
27868 8772 Acked Write 4 Sent Done Done false
27869 8773 Acked Read 7 Sent Done Done false
27870 8774 Acked Flush 0 Sent Done Done false
27871 8775 Acked Write 6 Sent Done Done false
27872 8776 Acked Read 9 Sent Done Done false
27873 8777 Acked Read 3 Sent Done Done false
27874 8778 Acked Flush 0 Sent Done Done false
27875 8779 Acked Write 9 Sent Done Done false
27876 8780 Acked Read 10 Sent Done Done false
27877 8781 Acked Write 2 Sent Done Done false
27878 8782 Acked Read 9 Sent Done Done false
27879 8783 Acked Write 1 Sent Done Done false
27880 8784 Acked Read 3 Sent Done Done false
27881 8785 Acked Write 4 Sent Done Done false
27882 8786 Acked Write 6 Sent Done Done false
27883 8787 Acked Write 1 Sent Done Done false
27884 8788 Acked Write 9 Sent Done Done false
27885 8789 Acked Read 8 Sent Done Done false
27886 8790 Acked Write 8 Sent Done Done false
27887 8791 Acked Read 3 Sent Done Done false
27888 8792 Acked Write 1 Sent Done Done false
27889 8793 Acked Read 6 Sent Done Done false
27890 8794 Acked Read 3 Sent Done Done false
27891 8795 Acked Read 4 Sent Done Done false
27892 8796 Acked Read 2 Sent Done Done false
27893 8797 Acked Flush 0 Sent Done Done false
27894 8798 Acked Read 6 Sent Done Done false
27895 8799 Acked Write 2 Sent Done Done false
27896 8800 Acked Read 4 Sent Done Done false
27897 8801 Acked Write 5 Sent Done Done false
27898 8802 Acked Read 6 Sent Done Done false
27899 8803 Acked Read 1 Sent Done Done false
27900 8804 Acked Read 2 Sent Done Done false
27901 8805 Acked Write 2 Sent Done Done false
27902 8806 Acked Write 7 Sent Done Done false
27903 8807 Acked Write 10 Sent Done Done false
27904 8808 Acked Read 7 Sent Done Done false
27905 8809 Acked Read 7 Sent Done Done false
27906 8810 Acked Write 3 Sent Done Done false
27907 8811 Acked Write 3 Sent Done Done false
27908 8812 Acked Read 4 Sent Done Done false
27909 8813 Acked Read 5 Sent Done Done false
27910 8814 Acked Flush 0 Sent Done Done false
27911 8815 Acked Read 10 Sent Done Done false
27912 8816 Acked Read 5 Sent Done Done false
27913 8817 Acked Write 1 Sent Done Done false
27914 8818 Acked Write 1 Sent Done Done false
27915 8819 Acked Write 3 Sent Done Done false
27916 8820 Acked Read 10 Sent Done Done false
27917 8821 Acked Write 1 Sent Done Done false
27918 8822 Acked Write 1 Sent Done Done false
27919 8823 Acked Read 10 Sent Done Done false
27920 8824 Acked Read 10 Sent Done Done false
27921 8825 Acked Read 2 Sent Done Done false
27922 8826 Acked Read 4 Sent Done Done false
27923 8827 Acked Read 7 Sent Done Done false
27924 8828 Acked Write 8 Sent Done Done false
27925 8829 Acked Read 1 Sent Done Done false
27926 8830 Acked Read 8 Sent Done Done false
27927 8831 Acked Read 3 Sent Done Done false
27928 8832 Acked Read 4 Sent Done Done false
27929 8833 Acked Write 6 Sent Done Done false
27930 8834 Acked Write 6 Sent Done Done false
27931 8835 Acked Write 4 Sent Done Done false
27932 8836 Acked Write 5 Sent Done Done false
27933 8837 Acked Read 6 Sent Done Done false
27934 8838 Acked Read 9 Sent Done Done false
27935 8839 Acked Read 8 Sent Done Done false
27936 8840 Acked Read 2 Sent Done Done false
27937 8841 Acked Write 4 Sent Done Done false
27938 8842 Acked Read 5 Sent Done Done false
27939 8843 Acked Write 1 Sent Done Done false
27940 8844 Acked Write 3 Sent Done Done false
27941 8845 Acked Read 7 Sent Done Done false
27942 8846 Acked Read 3 Sent Done Done false
27943 8847 Acked Read 3 Sent Done Done false
27944 8848 Acked Read 3 Sent Done Done false
27945 8849 Acked Write 1 Sent Done Done false
27946 8850 Acked Flush 0 Sent Done Done false
27947 STATES DS:0 DS:1 DS:2 TOTAL
27948 Sent 302 0 0 302
27949 Done 0 302 302 604
27950 Skipped 0 0 0 0
27951 Error 0 0 0 0
27952 Last Flush: Some(JobId(8548)) Some(JobId(8850)) Some(JobId(8850))
27953 Downstairs last five completed: 8548 8547 8546 8545 8544
27954 Upstairs last five completed: 8850 8849 8848 8847 8846
27955 CLIENT: Up:0 ds:302 act:2
279562025-01-23T17:43:35.937ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
279572025-01-23T17:43:35.937ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65084) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
279582025-01-23T17:43:35.939ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
279592025-01-23T17:43:35.939ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
279602025-01-23T17:43:35.939ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
279612025-01-23T17:43:35.939ZINFOcrucible: send last flush ID to this DS: Some(JobId(8548)) = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
279622025-01-23T17:43:35.939ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
279632025-01-23T17:43:35.939ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(8548)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
279642025-01-23T17:43:35.940ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(8548)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
27965 ----------------------------------------------------------------
27966 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
27967 Downstairs last five completed: 8850 8849 8848 8847 8846
27968 Upstairs last five completed: 8850 8849 8848 8847 8846
27969 CLIENT: Up:0 ds:0 act:3
27970 Replay: All jobs finished
27971 Main task has work to do, go find it
27972 got dsc Stop(2)
27973 stop 2
27974 Jan 23 17:43:38.184 INFO request completed, latency_us: 394[8830] Got stop action so:false kr:false
27975 , response_code: 204, uri: /stop/cid/2, method: POST, req_id: 93bee33a-6882-4bbf-b72c-3facebd6adb5, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
279762025-01-23T17:43:38.185ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
279772025-01-23T17:43:38.185ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
279782025-01-23T17:43:38.185ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
27979 Jan 23 17:43:38.185 INFO request completed, latency_us: 128, response_code: 200, uri: /state/cid/2, method: GET, req_id: 13d24e1d-c66d-42c8-9e9f-4064bed5e014, remote_addr: 127.0.0.1{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T17:43:38.185567097Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
27980 :42015, local_addr: 127.0.0.1:9998
27981 [8830] Exited with: ExitStatus(unix_wait_status(9))
27982 [8830][2] reports Exit
27983 Jan 23 17:43:42.188 INFO request completed, latency_us: 405, response_code: 200, uri: /state/cid/2, method: GET, req_id: 343b9e65-098f-4e45-8704-7f7af091a6c0, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
27984 001/300 Flush
27985 002/300 Write block 4280 len 24576 data: 6 6 7 7 8 7
27986 003/300 Flush
27987 004/300 Read block 2728 len 40960
27988 005/300 Read block 1939 len 4096
27989 006/300 Read block 4846 len 24576
27990 007/300 Read block 3975 len 16384
27991 008/300 Flush
27992 009/300 Write block 4607 len 16384 data: 7 5 6 5
27993 010/300 Read block 2759 len 8192
27994 011/300 Read block 1433 len 40960
27995 012/300 Write block 4737 len 40960 data: 4 5 6 4 4 4 5 5 7 7
27996 013/300 Flush
27997 014/300 Write block 3355 len 32768 data: 4 4 4 4 3 2 3 3
27998 015/300 Write block 3646 len 12288 data: 4 5 4
27999 016/300 Write block 2898 len 28672 data: 5 5 6 8 7 7 6
28000 017/300 Read block 4556 len 16384
28001 018/300 Write block 4449 len 36864 data: 5 5 5 4 4 4 5 6 6
28002 019/300 Read block 2702 len 24576
28003 020/300 Read block 3280 len 40960
28004 021/300 Write block 2411 len 8192 data: 6 7
28005 022/300 Read block 1246 len 4096
28006 023/300 Read block 3948 len 24576
28007 024/300 Write block 2435 len 36864 data: 6 6 4 3 3 3 5 5 6
28008 025/300 Write block 153 len 16384 data: 4 4 4 4
28009 026/300 Read block 797 len 28672
28010 027/300 Write block 2906 len 12288 data: 5 4 3
28011 028/300 Write block 1883 len 16384 data: 6 5 5 4
28012 029/300 Read block 2147 len 28672
28013 030/300 Flush
28014 031/300 Write block 4327 len 12288 data: 3 3 4
28015 032/300 Write block 3935 len 24576 data: 6 4 4 5 4 6
28016 033/300 Write block 2159 len 36864 data: 7 7 7 6 6 4 4 4 5
28017 034/300 Read block 1633 len 8192
28018 035/300 Write block 2917 len 40960 data: 2 3 4 6 7 7 8 8 9 9
28019 036/300 Read block 814 len 28672
28020 037/300 Read block 4728 len 12288
28021 038/300 Write block 1722 len 24576 data: 5 6 6 6 5 4
28022 039/300 Read block 2134 len 24576
28023 040/300 Flush
28024 041/300 Read block 1278 len 28672
28025 042/300 Write block 4895 len 12288 data: 8 6 6
28026 043/300 Read block 2707 len 32768
28027 044/300 Write block 1367 len 32768 data: 4 4 4 5 5 3 5 6
28028 045/300 Write block 3249 len 4096 data: 4
28029 046/300 Read block 3703 len 28672
28030 047/300 Write block 4032 len 12288 data: 7 8 8
28031 048/300 Flush
28032 049/300 Read block 4209 len 40960
28033 050/300 Write block 2063 len 28672 data: 9 8 5 5 6 6 9
28034 051/300 Read block 384 len 4096
28035 052/300 Read block 1633 len 24576
28036 053/300 Write block 3828 len 12288 data: 6 6 5
28037 054/300 Flush
28038 055/300 Write block 98 len 4096 data: 5
28039 056/300 Flush
28040 057/300 Read block 1834 len 32768
28041 058/300 Read block 3328 len 24576
28042 059/300 Write block 636 len 28672 data: 9 8 7 7 9 9 9
28043 060/300 Write block 224 len 40960 data: 4 4 4 4 6 4 4 5 5 6
28044 061/300 Write block 418 len 4096 data: 3
28045 062/300 Read block 4860 len 32768
28046 063/300 Write block 1369 len 32768 data: 5 6 6 4 6 7 6 6
28047 064/300 Write block 3918 len 28672 data: 6 6 8 8 8 7 6
28048 065/300 Read block 400 len 16384
28049 066/300 Read block 3876 len 8192
28050 067/300 Read block 4848 len 20480
28051 068/300 Write block 2702 len 40960 data: 3 3 4 3 2 5 5 5 6 6
28052 069/300 Read block 1022 len 28672
28053 070/300 Read block 258 len 32768
28054 071/300 Write block 4089 len 40960 data: 3 3 3 3 3 3 4 3 4 4
28055 072/300 Write block 2633 len 32768 data: 5 5 5 5 5 5 5 5
28056 073/300 Read block 4174 len 24576
28057 074/300 Read block 4430 len 8192
28058 075/300 Read block 3170 len 4096
28059 076/300 Write block 3187 len 32768 data: 5 3 4 3 3 3 4 4
28060 077/300 Write block 2356 len 28672 data: 7 7 5 5 6 6 5
28061 078/300 Read block 1284 len 20480
28062 079/300 Read block 1097 len 40960
28063 080/300 Read block 1009 len 32768
28064 081/300 Write block 1967 len 24576 data: 7 6 8 9 7 7
28065 082/300 Read block 1015 len 36864
28066 083/300 Read block 1638 len 36864
28067 084/300 Write block 1372 len 36864 data: 5 7 8 7 7 5 4 3 4
28068 085/300 Flush
28069 086/300 Write block 4587 len 24576 data: 6 5 7 7 8 7
28070 087/300 Write block 1774 len 8192 data: 6 6
28071 088/300 Write block 3098 len 40960 data: 5 6 6 6 6 7 8 8 9 8
28072 089/300 Write block 947 len 24576 data: 6 6 6 6 6 5
28073 090/300 Write block 2538 len 40960 data: 4 4 4 5 4 5 6 6 6 7
28074 091/300 Flush
28075 092/300 Write block 681 len 40960 data: 2 3 3 4 3 3 2 2 2 3
28076 093/300 Read block 1629 len 24576
28077 094/300 Read block 4415 len 4096
28078 095/300 Read block 1735 len 36864
28079 096/300 Write block 472 len 12288 data: 2 2 2
28080 097/300 Read block 4451 len 8192
28081 098/300 Write block 3626 len 4096 data: 5
28082 099/300 Read block 4309 len 8192
28083 100/300 Write block 3548 len 12288 data: 8 8 6
28084 101/300 Read block 2546 len 16384
28085 102/300 Write block 478 len 24576 data: 4 4 5 5 5 6
28086 103/300 Write block 2325 len 16384 data: 4 4 7 5
28087 104/300 Read block 2486 len 16384
28088 105/300 Flush
28089 106/300 Write block 2617 len 28672 data: 7 7 6 7 7 7 7
28090 107/300 Write block 1073 len 4096 data: 7
28091 108/300 Read block 1904 len 36864
28092 109/300 Read block 861 len 36864
28093 110/300 Read block 4148 len 16384
28094 111/300 Write block 2516 len 20480 data: 5 5 5 4 5
28095 112/300 Read block 2729 len 40960
28096 113/300 Read block 4277 len 12288
28097 114/300 Read block 2897 len 16384
28098 115/300 Flush
28099 116/300 Flush
28100 117/300 Write block 1651 len 32768 data: 7 9 9 7 8 5 2 4
28101 118/300 Flush
28102 119/300 Read block 3904 len 24576
28103 120/300 Write block 4392 len 12288 data: 4 5 6
28104 121/300 Write block 4269 len 12288 data: 11 11 11
28105 122/300 Flush
28106 123/300 Read block 117 len 4096
28107 124/300 Write block 2633 len 4096 data: 6
28108 125/300 Write block 640 len 32768 data: 10 10 10 9 8 9 8 6
28109 126/300 Flush
28110 127/300 Write block 1072 len 12288 data: 8 8 7
28111 128/300 Write block 4401 len 40960 data: 9 8 7 5 5 7 6 4 6 6
28112 129/300 Write block 4915 len 24576 data: 6 4 5 4 4 6
28113 130/300 Read block 953 len 20480
28114 131/300 Write block 4611 len 16384 data: 4 4 6 6
28115 132/300 Write block 2399 len 20480 data: 12 12 9 7 6
28116 133/300 Write block 2811 len 12288 data: 10 9 10
28117 134/300 Flush
28118 135/300 Read block 606 len 32768
28119 136/300 Read block 2824 len 24576
28120 137/300 Read block 4361 len 40960
28121 138/300 Write block 816 len 32768 data: 7 7 5 6 8 8 9 9
28122 139/300 Read block 401 len 28672
28123 140/300 Write block 3269 len 20480 data: 5 4 4 4 4
28124 141/300 Read block 2352 len 32768
28125 142/300 Write block 1371 len 32768 data: 7 6 8 9 8 8 6 5
28126 143/300 Flush
28127 144/300 Write block 3217 len 4096 data: 2
28128 145/300 Read block 1165 len 16384
28129 146/300 Write block 1897 len 12288 data: 6 6 6
28130 147/300 Read block 1670 len 40960
28131 148/300 Read block 308 len 20480
28132 149/300 Write block 4674 len 32768 data: 5 5 4 5 5 5 6 7
28133 150/300 Write block 2016 len 4096 data: 6
28134 151/300 Read block 2629 len 24576
28135 152/300 Read block 4729 len 12288
28136 153/300 Read block 4837 len 4096
28137 154/300 Read block 2258 len 28672
28138 155/300 Write block 366 len 28672 data: 2 6 7 8 8 9 9
28139 156/300 Read block 4498 len 28672
28140 157/300 Write block 2312 len 40960 data: 4 4 4 4 3 5 5 5 6 6
28141 158/300 Flush
28142 159/300 Write block 2071 len 32768 data: 8 7 7 9 7 6 5 5
28143 160/300 Write block 152 len 28672 data: 3 5 5 5 5 6 6
28144 161/300 Write block 2969 len 20480 data: 3 3 5 6 7
28145 162/300 Write block 3546 len 4096 data: 8
28146 163/300 Write block 277 len 28672 data: 4 5 8 7 5 6 5
28147 164/300 Write block 3311 len 4096 data: 7
28148 165/300 Write block 2622 len 12288 data: 8 8 7
28149 166/300 Write block 4971 len 32768 data: 12 13 9 8 6 6 4 3
28150 167/300 Read block 2643 len 16384
28151 168/300 Read block 4045 len 16384
28152 169/300 Write block 1594 len 28672 data: 4 4 3 3 5 4 3
28153 170/300 Write block 3197 len 20480 data: 6 5 5 5 4
28154 171/300 Write block 418 len 36864 data: 4 3 3 3 4 4 3 3 4
28155 172/300 Write block 4071 len 32768 data: 3 3 3 2 2 2 2 3
28156 173/300 Read block 2408 len 36864
28157 174/300 Read block 4273 len 12288
28158 175/300 Write block 1148 len 28672 data: 4 4 3 3 4 4 5
28159 176/300 Read block 2767 len 16384
28160 177/300 Read block 2094 len 20480
28161 178/300 Write block 666 len 12288 data: 4 4 8
28162 179/300 Write block 3981 len 20480 data: 5 8 7 7 6
28163 180/300 Write block 3917 len 4096 data: 7
28164 181/300 Write block 1555 len 32768 data: 5 4 5 5 6 7 6 6
28165 182/300 Read block 701 len 20480
28166 183/300 Read block 340 len 12288
28167 184/300 Write block 1784 len 8192 data: 4 3
28168 185/300 Read block 1147 len 36864
28169 186/300 Read block 3412 len 28672
28170 187/300 Write block 1411 len 40960 data: 5 6 4 4 3 5 4 5 6 7
28171 188/300 Read block 2983 len 40960
28172 189/300 Write block 4811 len 40960 data: 6 7 5 5 5 5 5 5 4 4
28173 190/300 Write block 1217 len 4096 data: 3
28174 191/300 Read block 2486 len 28672
28175 192/300 Write block 730 len 40960 data: 3 4 5 6 7 8 8 5 3 4
28176 193/300 Read block 4901 len 28672
28177 194/300 Read block 97 len 40960
28178 195/300 Read block 1225 len 16384
28179 196/300 Read block 4605 len 4096
28180 197/300 Read block 621 len 16384
28181 198/300 Write block 896 len 8192 data: 4 4
28182 199/300 Read block 4850 len 12288
28183 200/300 Write block 2141 len 16384 data: 6 5 4 3
28184 201/300 Read block 2096 len 8192
28185 202/300 Flush
28186 203/300 Write block 3047 len 8192 data: 8 8
28187 204/300 Read block 9 len 32768
28188 205/300 Flush
28189 206/300 Write block 3550 len 12288 data: 7 7 7
28190 207/300 Write block 3207 len 40960 data: 6 6 5 4 5 4 3 3 3 2
28191 208/300 Write block 4983 len 16384 data: 3 3 3 5
28192 209/300 Write block 4725 len 28672 data: 3 5 5 5 4 6 6
28193 210/300 Read block 19 len 36864
28194 211/300 Write block 4195 len 32768 data: 4 4 4 4 4 5 4 4
28195 212/300 Read block 1486 len 20480
28196 213/300 Read block 1126 len 32768
28197 214/300 Read block 4931 len 16384
28198 215/300 Read block 275 len 28672
28199 216/300 Write block 1711 len 20480 data: 8 7 5 4 3
28200 217/300 Read block 1848 len 4096
28201 218/300 Write block 721 len 28672 data: 4 5 5 5 4 4 3
28202 219/300 Read block 2045 len 32768
28203 220/300 Read block 1358 len 24576
28204 221/300 Read block 2079 len 36864
28205 222/300 Read block 1680 len 40960
28206 223/300 Write block 4311 len 24576 data: 4 5 5 5 6 9
28207 224/300 Flush
28208 225/300 Write block 4175 len 16384 data: 6 6 6 6
28209 226/300 Read block 299 len 24576
28210 227/300 Read block 1666 len 32768
28211 228/300 Read block 3732 len 40960
28212 229/300 Flush
28213 230/300 Read block 2276 len 40960
28214 231/300 Write block 1675 len 36864 data: 4 5 5 5 5 6 8 8 7
28215 232/300 Write block 4421 len 12288 data: 2 2 3
28216 233/300 Read block 2687 len 32768
28217 234/300 Write block 3806 len 24576 data: 8 7 7 6 7 10
28218 235/300 Read block 912 len 24576
28219 236/300 Write block 375 len 24576 data: 10 9 5 4 3 2
28220 237/300 Write block 390 len 8192 data: 5 4
28221 238/300 Write block 2590 len 28672 data: 6 8 6 6 8 9 8
28222 239/300 Read block 3201 len 24576
28223 240/300 Write block 4986 len 24576 data: 6 4 4 3 5 4
28224 241/300 Read block 4397 len 40960
28225 242/300 Read block 2953 len 36864
28226 243/300 Read block 18 len 16384
28227 244/300 Flush
28228 245/300 Read block 4831 len 24576
28229 246/300 Write block 3071 len 36864 data: 3 3 3 3 3 2 2 2 3
28230 247/300 Write block 3635 len 24576 data: 5 6 6 5 5 5
28231 248/300 Read block 3373 len 16384
28232 249/300 Read block 4042 len 32768
28233 250/300 Write block 4110 len 40960 data: 4 4 5 7 6 6 6 6 6 5
28234 251/300 Flush
28235 252/300 Write block 4645 len 8192 data: 5 4
28236 253/300 Read block 600 len 32768
28237 254/300 Read block 4065 len 4096
28238 255/300 Read block 1784 len 12288
28239 256/300 Flush
28240 257/300 Flush
28241 258/300 Write block 1297 len 12288 data: 6 6 6
28242 259/300 Read block 1420 len 32768
28243 260/300 Flush
28244 261/300 Write block 4507 len 4096 data: 5
28245 262/300 Flush
28246 263/300 Flush
28247 264/300 Read block 1796 len 24576
28248 265/300 Read block 2771 len 8192
28249 266/300 Flush
28250 267/300 Read block 3828 len 36864
28251 268/300 Flush
28252 269/300 Write block 4716 len 4096 data: 6
28253 270/300 Read block 4342 len 16384
28254 271/300 Read block 1538 len 40960
28255 272/300 Read block 794 len 4096
28256 273/300 Write block 787 len 28672 data: 6 5 5 5 4 5 5
28257 274/300 Write block 2429 len 12288 data: 6 5 5
28258 275/300 Write block 2258 len 40960 data: 4 3 4 4 7 6 6 5 3 3
28259 276/300 Write block 4613 len 12288 data: 7 7 5
28260 277/300 Write block 4667 len 24576 data: 4 4 3 6 6 6
28261 278/300 Write block 4545 len 20480 data: 7 6 5 5 5
28262 279/300 Read block 1029 len 40960
28263 280/300 Read block 3870 len 8192
28264 281/300 Read block 1543 len 4096
28265 282/300 Read block 2082 len 20480
28266 283/300 Read block 348 len 36864
28267 284/300 Write block 2856 len 16384 data: 4 6 5 6
28268 285/300 Read block 4910 len 32768
28269 286/300 Write block 3536 len 12288 data: 2 3 3
28270 287/300 Flush
28271 288/300 Write block 3952 len 12288 data: 5 6 6
28272 289/300 Write block 1420 len 16384 data: 8 6 5 6
28273 290/300 Write block 495 len 20480 data: 4 5 4 4 4
28274 291/300 Read block 2110 len 24576
28275 292/300 Write block 3222 len 16384 data: 4 6 5 7
28276 293/300 Read block 660 len 16384
28277 294/300 Write block 1753 len 16384 data: 6 5 7 8
28278 295/300 Write block 3071 len 20480 data: 4 4 4 4 4
28279 296/300 Read block 1315 len 16384
28280 297/300 Read block 2587 len 12288
28281 298/300 Read block 2567 len 24576
28282 299/300 Read block 4155 len 32768
28283 300/300 Write block 1198 len 24576 data: 5 6 7 4 4 4
28284 301/300 Read block 3696 len 12288
28285 Main task has work to do, go find it
28286 got dsc Start(2)
28287 start 2
28288 Jan 23 17:43:42.425 INFO request completed, latency_us: 188, response_code: 204, uri: /start/cid/2, method: POST, req_id: 93cdb8bc-18ac-4391-abe5-4f4bdc37ab8f, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
28289 Replay: started 2, returned:Ok(())
28290 ----------------------------------------------------------------
28291 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28292 States: Active Active Offline
28293 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28294 8851 Acked Flush 0 Done Done Sent false
28295 8852 Acked Write 6 Done Done Sent false
28296 8853 Acked Flush 0 Done Done Sent false
28297 8854 Acked Read 10 Done Done Sent false
28298 8855 Acked Read 1 Done Done Sent false
28299 8856 Acked Read 6 Done Done Sent false
28300 8857 Acked Read 4 Done Done Sent false
28301 8858 Acked Flush 0 Done Done Sent false
28302 8859 Acked Write 4 Done Done Sent false
28303 8860 Acked Read 2 Done Done Sent false
28304 8861 Acked Read 10 Done Done Sent false
28305 8862 Acked Write 10 Done Done Sent false
28306 8863 Acked Flush 0 Done Done Sent false
28307 8864 Acked Write 8 Done Done Sent false
28308 8865 Acked Write 3 Done Done Sent false
28309 8866 Acked Write 7 Done Done Sent false
28310 8867 Acked Read 4 Done Done Sent false
28311 8868 Acked Write 9 Done Done Sent false
28312 8869 Acked Read 6 Done Done Sent false
28313 8870 Acked Read 10 Done Done Sent false
28314 8871 Acked Write 2 Done Done Sent false
28315 8872 Acked Read 1 Done Done Sent false
28316 8873 Acked Read 6 Done Done Sent false
28317 8874 Acked Write 9 Done Done Sent false
28318 8875 Acked Write 4 Done Done Sent false
28319 8876 Acked Read 7 Done Done Sent false
28320 8877 Acked Write 3 Done Done Sent false
28321 8878 Acked Write 4 Done Done Sent false
28322 8879 Acked Read 7 Done Done Sent false
28323 8880 Acked Flush 0 Done Done Sent false
28324 8881 Acked Write 3 Done Done Sent false
28325 8882 Acked Write 6 Done Done Sent false
28326 8883 Acked Write 9 Done Done Sent false
28327 8884 Acked Read 2 Done Done Sent false
28328 8885 Acked Write 10 Done Done Sent false
28329 8886 Acked Read 7 Done Done Sent false
28330 8887 Acked Read 3 Done Done Sent false
28331 8888 Acked Write 6 Done Done Sent false
28332 8889 Acked Read 6 Done Done Sent false
28333 8890 Acked Flush 0 Done Done Sent false
28334 8891 Acked Read 7 Done Done Sent false
28335 8892 Acked Write 3 Done Done Sent false
28336 8893 Acked Read 8 Done Done Sent false
28337 8894 Acked Write 8 Done Done Sent false
28338 8895 Acked Write 1 Done Done Sent false
28339 8896 Acked Read 7 Done Done Sent false
28340 8897 Acked Write 3 Done Done Sent false
28341 8898 Acked Flush 0 Done Done Sent false
28342 8899 Acked Read 10 Done Done Sent false
28343 8900 Acked Write 7 Done Done Sent false
28344 8901 Acked Read 1 Done Done Sent false
28345 8902 Acked Read 6 Done Done Sent false
28346 8903 Acked Write 3 Done Done Sent false
28347 8904 Acked Flush 0 Done Done Sent false
28348 8905 Acked Write 1 Done Done Sent false
28349 8906 Acked Flush 0 Done Done Sent false
28350 8907 Acked Read 8 Done Done Sent false
28351 8908 Acked Read 6 Done Done Sent false
28352 8909 Acked Write 7 Done Done Sent false
28353 8910 Acked Write 10 Done Done Sent false
28354 8911 Acked Write 1 Done Done Sent false
28355 8912 Acked Read 8 Done Done Sent false
28356 8913 Acked Write 8 Done Done Sent false
28357 8914 Acked Write 7 Done Done Sent false
28358 8915 Acked Read 4 Done Done Sent false
28359 8916 Acked Read 2 Done Done Sent false
28360 8917 Acked Read 5 Done Done Sent false
28361 8918 Acked Write 10 Done Done Sent false
28362 8919 Acked Read 7 Done Done Sent false
28363 8920 Acked Read 8 Done Done Sent false
28364 8921 Acked Write 10 Done Done Sent false
28365 8922 Acked Write 8 Done Done Sent false
28366 8923 Acked Read 6 Done Done Sent false
28367 8924 Acked Read 2 Done Done Sent false
28368 8925 Acked Read 1 Done Done Sent false
28369 8926 Acked Write 8 Done Done Sent false
28370 8927 Acked Write 7 Done Done Sent false
28371 8928 Acked Read 5 Done Done Sent false
28372 8929 Acked Read 10 Done Done Sent false
28373 8930 Acked Read 8 Done Done Sent false
28374 8931 Acked Write 6 Done Done Sent false
28375 8932 Acked Read 9 Done Done Sent false
28376 8933 Acked Read 9 Done Done Sent false
28377 8934 Acked Write 9 Done Done Sent false
28378 8935 Acked Flush 0 Done Done Sent false
28379 8936 Acked Write 6 Done Done Sent false
28380 8937 Acked Write 2 Done Done Sent false
28381 8938 Acked Write 10 Done Done Sent false
28382 8939 Acked Write 6 Done Done Sent false
28383 8940 Acked Write 10 Done Done Sent false
28384 8941 Acked Flush 0 Done Done Sent false
28385 8942 Acked Write 10 Done Done Sent false
28386 8943 Acked Read 6 Done Done Sent false
28387 8944 Acked Read 1 Done Done Sent false
28388 8945 Acked Read 9 Done Done Sent false
28389 8946 Acked Write 3 Done Done Sent false
28390 8947 Acked Read 2 Done Done Sent false
28391 8948 Acked Write 1 Done Done Sent false
28392 8949 Acked Read 2 Done Done Sent false
28393 8950 Acked Write 3 Done Done Sent false
28394 8951 Acked Read 4 Done Done Sent false
28395 8952 Acked Write 6 Done Done Sent false
28396 8953 Acked Write 4 Done Done Sent false
28397 8954 Acked Read 4 Done Done Sent false
28398 8955 Acked Flush 0 Done Done Sent false
28399 8956 Acked Write 7 Done Done Sent false
28400 8957 Acked Write 1 Done Done Sent false
28401 8958 Acked Read 9 Done Done Sent false
28402 8959 Acked Read 9 Done Done Sent false
28403 8960 Acked Read 4 Done Done Sent false
28404 8961 Acked Write 5 Done Done Sent false
28405 8962 Acked Read 10 Done Done Sent false
28406 8963 Acked Read 3 Done Done Sent false
28407 8964 Acked Read 4 Done Done Sent false
28408 8965 Acked Flush 0 Done Done Sent false
28409 8966 Acked Flush 0 Done Done Sent false
28410 8967 Acked Write 8 Done Done Sent false
28411 8968 Acked Flush 0 Done Done Sent false
28412 8969 Acked Read 6 Done Done Sent false
28413 8970 Acked Write 3 Done Done Sent false
28414 8971 Acked Write 3 Done Done Sent false
28415 8972 Acked Flush 0 Done Done Sent false
28416 8973 Acked Read 1 Done Done Sent false
28417 8974 Acked Write 1 Done Done Sent false
28418 8975 Acked Write 8 Done Done Sent false
28419 8976 Acked Flush 0 Done Done Sent false
28420 8977 Acked Write 3 Done Done Sent false
28421 8978 Acked Write 10 Done Done Sent false
28422 8979 Acked Write 6 Done Done Sent false
28423 8980 Acked Read 5 Done Done Sent false
28424 8981 Acked Write 4 Done Done Sent false
28425 8982 Acked Write 5 Done Done Sent false
28426 8983 Acked Write 3 Done Done Sent false
28427 8984 Acked Flush 0 Done Done Sent false
28428 8985 Acked Read 8 Done Done Sent false
28429 8986 Acked Read 6 Done Done Sent false
28430 8987 Acked Read 10 Done Done Sent false
28431 8988 Acked Write 8 Done Done Sent false
28432 8989 Acked Read 7 Done Done Sent false
28433 8990 Acked Write 5 Done Done Sent false
28434 8991 Acked Read 8 Done Done Sent false
28435 8992 Acked Write 8 Done Done Sent false
28436 8993 Acked Flush 0 Done Done Sent false
28437 8994 Acked Write 1 Done Done Sent false
28438 8995 Acked Read 4 Done Done Sent false
28439 8996 Acked Write 3 Done Done Sent false
28440 8997 Acked Read 10 Done Done Sent false
28441 8998 Acked Read 5 Done Done Sent false
28442 8999 Acked Write 8 Done Done Sent false
28443 9000 Acked Write 1 Done Done Sent false
28444 9001 Acked Read 6 Done Done Sent false
28445 9002 Acked Read 3 Done Done Sent false
28446 9003 Acked Read 1 Done Done Sent false
28447 9004 Acked Read 7 Done Done Sent false
28448 9005 Acked Write 7 Done Done Sent false
28449 9006 Acked Read 7 Done Done Sent false
28450 9007 Acked Write 10 Done Done Sent false
28451 9008 Acked Flush 0 Done Done Sent false
28452 9009 Acked Write 8 Done Done Sent false
28453 9010 Acked Write 7 Done Done Sent false
28454 9011 Acked Write 5 Done Done Sent false
28455 9012 Acked Write 1 Done Done Sent false
28456 9013 Acked Write 7 Done Done Sent false
28457 9014 Acked Write 1 Done Done Sent false
28458 9015 Acked Write 3 Done Done Sent false
28459 9016 Acked Write 8 Done Done Sent false
28460 9017 Acked Read 4 Done Done Sent false
28461 9018 Acked Read 4 Done Done Sent false
28462 9019 Acked Write 7 Done Done Sent false
28463 9020 Acked Write 5 Done Done Sent false
28464 9021 Acked Write 9 Done Done Sent false
28465 9022 Acked Write 8 Done Done Sent false
28466 9023 Acked Read 9 Done Done Sent false
28467 9024 Acked Read 3 Done Done Sent false
28468 9025 Acked Write 7 Done Done Sent false
28469 9026 Acked Read 4 Done Done Sent false
28470 9027 Acked Read 5 Done Done Sent false
28471 9028 Acked Write 3 Done Done Sent false
28472 9029 Acked Write 5 Done Done Sent false
28473 9030 Acked Write 1 Done Done Sent false
28474 9031 Acked Write 8 Done Done Sent false
28475 9032 Acked Read 5 Done Done Sent false
28476 9033 Acked Read 3 Done Done Sent false
28477 9034 Acked Write 2 Done Done Sent false
28478 9035 Acked Read 9 Done Done Sent false
28479 9036 Acked Read 7 Done Done Sent false
28480 9037 Acked Write 10 Done Done Sent false
28481 9038 Acked Read 10 Done Done Sent false
28482 9039 Acked Write 10 Done Done Sent false
28483 9040 Acked Write 1 Done Done Sent false
28484 9041 Acked Read 7 Done Done Sent false
28485 9042 Acked Write 10 Done Done Sent false
28486 9043 Acked Read 7 Done Done Sent false
28487 9044 Acked Read 10 Done Done Sent false
28488 9045 Acked Read 4 Done Done Sent false
28489 9046 Acked Read 1 Done Done Sent false
28490 9047 Acked Read 4 Done Done Sent false
28491 9048 Acked Write 2 Done Done Sent false
28492 9049 Acked Read 3 Done Done Sent false
28493 9050 Acked Write 4 Done Done Sent false
28494 9051 Acked Read 2 Done Done Sent false
28495 9052 Acked Flush 0 Done Done Sent false
28496 9053 Acked Write 2 Done Done Sent false
28497 9054 Acked Read 8 Done Done Sent false
28498 9055 Acked Flush 0 Done Done Sent false
28499 9056 Acked Write 3 Done Done Sent false
28500 9057 Acked Write 10 Done Done Sent false
28501 9058 Acked Write 4 Done Done Sent false
28502 9059 Acked Write 7 Done Done Sent false
28503 9060 Acked Read 9 Done Done Sent false
28504 9061 Acked Write 8 Done Done Sent false
28505 9062 Acked Read 5 Done Done Sent false
28506 9063 Acked Read 8 Done Done Sent false
28507 9064 Acked Read 4 Done Done Sent false
28508 9065 Acked Read 7 Done Done Sent false
28509 9066 Acked Write 5 Done Done Sent false
28510 9067 Acked Read 1 Done Done Sent false
28511 9068 Acked Write 7 Done Done Sent false
28512 9069 Acked Read 8 Done Done Sent false
28513 9070 Acked Read 6 Done Done Sent false
28514 9071 Acked Read 9 Done Done Sent false
28515 9072 Acked Read 10 Done Done Sent false
28516 9073 Acked Write 6 Done Done Sent false
28517 9074 Acked Flush 0 Done Done Sent false
28518 9075 Acked Write 4 Done Done Sent false
28519 9076 Acked Read 6 Done Done Sent false
28520 9077 Acked Read 8 Done Done Sent false
28521 9078 Acked Read 10 Done Done Sent false
28522 9079 Acked Flush 0 Done Done Sent false
28523 9080 Acked Read 10 Done Done Sent false
28524 9081 Acked Write 9 Done Done Sent false
28525 9082 Acked Write 3 Done Done Sent false
28526 9083 Acked Read 8 Done Done Sent false
28527 9084 Acked Write 6 Done Done Sent false
28528 9085 Acked Read 6 Done Done Sent false
28529 9086 Acked Write 6 Done Done Sent false
28530 9087 Acked Write 2 Done Done Sent false
28531 9088 Acked Write 7 Done Done Sent false
28532 9089 Acked Read 6 Done Done Sent false
28533 9090 Acked Write 6 Done Done Sent false
28534 9091 Acked Read 10 Done Done Sent false
28535 9092 Acked Read 9 Done Done Sent false
28536 9093 Acked Read 4 Done Done Sent false
28537 9094 Acked Flush 0 Done Done Sent false
28538 9095 Acked Read 6 Done Done Sent false
28539 9096 Acked Write 9 Done Done Sent false
28540 9097 Acked Write 6 Done Done Sent false
28541 9098 Acked Read 4 Done Done Sent false
28542 9099 Acked Read 8 Done Done Sent false
28543 9100 Acked Write 10 Done Done Sent false
28544 9101 Acked Flush 0 Done Done Sent false
28545 9102 Acked Write 2 Done Done Sent false
28546 9103 Acked Read 8 Done Done Sent false
28547 9104 Acked Read 1 Done Done Sent false
28548 9105 Acked Read 3 Done Done Sent false
28549 9106 Acked Flush 0 Done Done Sent false
28550 9107 Acked Flush 0 Done Done Sent false
28551 9108 Acked Write 3 Done Done Sent false
28552 9109 Acked Read 8 Done Done Sent false
28553 9110 Acked Flush 0 Done Done Sent false
28554 9111 Acked Write 1 Done Done Sent false
28555 9112 Acked Flush 0 Done Done Sent false
28556 9113 Acked Flush 0 Done Done Sent false
28557 9114 Acked Read 6 Done Done Sent false
28558 9115 Acked Read 2 Done Done Sent false
28559 9116 Acked Flush 0 Done Done Sent false
28560 9117 Acked Read 9 Done Done Sent false
28561 9118 Acked Flush 0 Done Done Sent false
28562 9119 Acked Write 1 Done Done Sent false
28563 9120 Acked Read 4 Done Done Sent false
28564 9121 Acked Read 10 Done Done Sent false
28565 9122 Acked Read 1 Done Done Sent false
28566 9123 Acked Write 7 Done Done Sent false
28567 9124 Acked Write 3 Done Done Sent false
28568 9125 Acked Write 10 Done Done Sent false
28569 9126 Acked Write 3 Done Done Sent false
28570 9127 Acked Write 6 Done Done Sent false
28571 9128 Acked Write 5 Done Done Sent false
28572 9129 Acked Read 10 Done Done Sent false
28573 9130 Acked Read 2 Done Done Sent false
28574 9131 Acked Read 1 Done Done Sent false
28575 9132 Acked Read 5 Done Done Sent false
28576 9133 Acked Read 9 Done Done Sent false
28577 9134 Acked Write 4 Done Done Sent false
28578 9135 Acked Read 8 Done Done Sent false
28579 9136 Acked Write 3 Done Done Sent false
28580 9137 Acked Flush 0 Done Done Sent false
28581 9138 Acked Write 3 Done Done Sent false
28582 9139 Acked Write 4 Done Done Sent false
28583 9140 Acked Write 5 Done Done Sent false
28584 9141 Acked Read 6 Done Done Sent false
28585 9142 Acked Write 4 Done Done Sent false
28586 9143 Acked Read 4 Done Done Sent false
28587 9144 Acked Write 4 Done Done Sent false
28588 9145 Acked Write 5 Done Done Sent false
28589 9146 Acked Read 4 Done Done Sent false
28590 9147 Acked Read 3 Done Done Sent false
28591 9148 Acked Read 6 Done Done Sent false
28592 9149 Acked Read 8 Done Done Sent false
28593 9150 Acked Write 6 Done Done Sent false
28594 9151 Acked Read 3 Done Done Sent false
28595 STATES DS:0 DS:1 DS:2 TOTAL
28596 Sent 0 0 301 301
28597 Done 301 301 0 602
28598 Skipped 0 0 0 0
28599 Error 0 0 0 0
28600 Last Flush: Some(JobId(9137)) Some(JobId(9137)) Some(JobId(8850))
28601 Downstairs last five completed: 8850 8849 8848 8847 8846
28602 Upstairs last five completed: 9151 9150 9149 9148 9147
28603 CLIENT: Up:0 ds:301 act:2
28604 [8830] got start action
28605 [8830] I am going to restart
28606 Starting downstairs at port 8830
28607 Make output file at "/tmp/dsc/downstairs-8830.txt"
28608 [8830][2] reports Starting
28609 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1232)
28610 [8830][2] reports Running
28611 ----------------------------------------------------------------
28612 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
28613 States: Active Active Offline
28614 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28615 8851 Acked Flush 0 Done Done Sent false
28616 8852 Acked Write 6 Done Done Sent false
28617 8853 Acked Flush 0 Done Done Sent false
28618 8854 Acked Read 10 Done Done Sent false
28619 8855 Acked Read 1 Done Done Sent false
28620 8856 Acked Read 6 Done Done Sent false
28621 8857 Acked Read 4 Done Done Sent false
28622 8858 Acked Flush 0 Done Done Sent false
28623 8859 Acked Write 4 Done Done Sent false
28624 8860 Acked Read 2 Done Done Sent false
28625 8861 Acked Read 10 Done Done Sent false
28626 8862 Acked Write 10 Done Done Sent false
28627 8863 Acked Flush 0 Done Done Sent false
28628 8864 Acked Write 8 Done Done Sent false
28629 8865 Acked Write 3 Done Done Sent false
28630 8866 Acked Write 7 Done Done Sent false
28631 8867 Acked Read 4 Done Done Sent false
28632 8868 Acked Write 9 Done Done Sent false
28633 8869 Acked Read 6 Done Done Sent false
28634 8870 Acked Read 10 Done Done Sent false
28635 8871 Acked Write 2 Done Done Sent false
28636 8872 Acked Read 1 Done Done Sent false
28637 8873 Acked Read 6 Done Done Sent false
28638 8874 Acked Write 9 Done Done Sent false
28639 8875 Acked Write 4 Done Done Sent false
28640 8876 Acked Read 7 Done Done Sent false
28641 8877 Acked Write 3 Done Done Sent false
28642 8878 Acked Write 4 Done Done Sent false
28643 8879 Acked Read 7 Done Done Sent false
28644 8880 Acked Flush 0 Done Done Sent false
28645 8881 Acked Write 3 Done Done Sent false
28646 8882 Acked Write 6 Done Done Sent false
28647 8883 Acked Write 9 Done Done Sent false
28648 8884 Acked Read 2 Done Done Sent false
28649 8885 Acked Write 10 Done Done Sent false
28650 8886 Acked Read 7 Done Done Sent false
28651 8887 Acked Read 3 Done Done Sent false
28652 8888 Acked Write 6 Done Done Sent false
28653 8889 Acked Read 6 Done Done Sent false
28654 8890 Acked Flush 0 Done Done Sent false
28655 8891 Acked Read 7 Done Done Sent false
28656 8892 Acked Write 3 Done Done Sent false
28657 8893 Acked Read 8 Done Done Sent false
28658 8894 Acked Write 8 Done Done Sent false
28659 8895 Acked Write 1 Done Done Sent false
28660 8896 Acked Read 7 Done Done Sent false
28661 8897 Acked Write 3 Done Done Sent false
28662 8898 Acked Flush 0 Done Done Sent false
28663 8899 Acked Read 10 Done Done Sent false
28664 8900 Acked Write 7 Done Done Sent false
28665 8901 Acked Read 1 Done Done Sent false
28666 8902 Acked Read 6 Done Done Sent false
28667 8903 Acked Write 3 Done Done Sent false
28668 8904 Acked Flush 0 Done Done Sent false
28669 8905 Acked Write 1 Done Done Sent false
28670 8906 Acked Flush 0 Done Done Sent false
28671 8907 Acked Read 8 Done Done Sent false
28672 8908 Acked Read 6 Done Done Sent false
28673 8909 Acked Write 7 Done Done Sent false
28674 8910 Acked Write 10 Done Done Sent false
28675 8911 Acked Write 1 Done Done Sent false
28676 8912 Acked Read 8 Done Done Sent false
28677 8913 Acked Write 8 Done Done Sent false
28678 8914 Acked Write 7 Done Done Sent false
28679 8915 Acked Read 4 Done Done Sent false
28680 8916 Acked Read 2 Done Done Sent false
28681 8917 Acked Read 5 Done Done Sent false
28682 8918 Acked Write 10 Done Done Sent false
28683 8919 Acked Read 7 Done Done Sent false
28684 8920 Acked Read 8 Done Done Sent false
28685 8921 Acked Write 10 Done Done Sent false
28686 8922 Acked Write 8 Done Done Sent false
28687 8923 Acked Read 6 Done Done Sent false
28688 8924 Acked Read 2 Done Done Sent false
28689 8925 Acked Read 1 Done Done Sent false
28690 8926 Acked Write 8 Done Done Sent false
28691 8927 Acked Write 7 Done Done Sent false
28692 8928 Acked Read 5 Done Done Sent false
28693 8929 Acked Read 10 Done Done Sent false
28694 8930 Acked Read 8 Done Done Sent false
28695 8931 Acked Write 6 Done Done Sent false
28696 8932 Acked Read 9 Done Done Sent false
28697 8933 Acked Read 9 Done Done Sent false
28698 8934 Acked Write 9 Done Done Sent false
28699 8935 Acked Flush 0 Done Done Sent false
28700 8936 Acked Write 6 Done Done Sent false
28701 8937 Acked Write 2 Done Done Sent false
28702 8938 Acked Write 10 Done Done Sent false
28703 8939 Acked Write 6 Done Done Sent false
28704 8940 Acked Write 10 Done Done Sent false
28705 8941 Acked Flush 0 Done Done Sent false
28706 8942 Acked Write 10 Done Done Sent false
28707 8943 Acked Read 6 Done Done Sent false
28708 8944 Acked Read 1 Done Done Sent false
28709 8945 Acked Read 9 Done Done Sent false
28710 8946 Acked Write 3 Done Done Sent false
28711 8947 Acked Read 2 Done Done Sent false
28712 8948 Acked Write 1 Done Done Sent false
28713 8949 Acked Read 2 Done Done Sent false
28714 8950 Acked Write 3 Done Done Sent false
28715 8951 Acked Read 4 Done Done Sent false
28716 8952 Acked Write 6 Done Done Sent false
28717 8953 Acked Write 4 Done Done Sent false
28718 8954 Acked Read 4 Done Done Sent false
28719 8955 Acked Flush 0 Done Done Sent false
28720 8956 Acked Write 7 Done Done Sent false
28721 8957 Acked Write 1 Done Done Sent false
28722 8958 Acked Read 9 Done Done Sent false
28723 8959 Acked Read 9 Done Done Sent false
28724 8960 Acked Read 4 Done Done Sent false
28725 8961 Acked Write 5 Done Done Sent false
28726 8962 Acked Read 10 Done Done Sent false
28727 8963 Acked Read 3 Done Done Sent false
28728 8964 Acked Read 4 Done Done Sent false
28729 8965 Acked Flush 0 Done Done Sent false
28730 8966 Acked Flush 0 Done Done Sent false
28731 8967 Acked Write 8 Done Done Sent false
28732 8968 Acked Flush 0 Done Done Sent false
28733 8969 Acked Read 6 Done Done Sent false
28734 8970 Acked Write 3 Done Done Sent false
28735 8971 Acked Write 3 Done Done Sent false
28736 8972 Acked Flush 0 Done Done Sent false
28737 8973 Acked Read 1 Done Done Sent false
28738 8974 Acked Write 1 Done Done Sent false
28739 8975 Acked Write 8 Done Done Sent false
28740 8976 Acked Flush 0 Done Done Sent false
28741 8977 Acked Write 3 Done Done Sent false
28742 8978 Acked Write 10 Done Done Sent false
28743 8979 Acked Write 6 Done Done Sent false
28744 8980 Acked Read 5 Done Done Sent false
28745 8981 Acked Write 4 Done Done Sent false
28746 8982 Acked Write 5 Done Done Sent false
28747 8983 Acked Write 3 Done Done Sent false
28748 8984 Acked Flush 0 Done Done Sent false
28749 8985 Acked Read 8 Done Done Sent false
28750 8986 Acked Read 6 Done Done Sent false
28751 8987 Acked Read 10 Done Done Sent false
28752 8988 Acked Write 8 Done Done Sent false
28753 8989 Acked Read 7 Done Done Sent false
28754 8990 Acked Write 5 Done Done Sent false
28755 8991 Acked Read 8 Done Done Sent false
28756 8992 Acked Write 8 Done Done Sent false
28757 8993 Acked Flush 0 Done Done Sent false
28758 8994 Acked Write 1 Done Done Sent false
28759 8995 Acked Read 4 Done Done Sent false
28760 8996 Acked Write 3 Done Done Sent false
28761 8997 Acked Read 10 Done Done Sent false
28762 8998 Acked Read 5 Done Done Sent false
28763 8999 Acked Write 8 Done Done Sent false
28764 9000 Acked Write 1 Done Done Sent false
28765 9001 Acked Read 6 Done Done Sent false
28766 9002 Acked Read 3 Done Done Sent false
28767 9003 Acked Read 1 Done Done Sent false
28768 9004 Acked Read 7 Done Done Sent false
28769 9005 Acked Write 7 Done Done Sent false
28770 9006 Acked Read 7 Done Done Sent false
28771 9007 Acked Write 10 Done Done Sent false
28772 9008 Acked Flush 0 Done Done Sent false
28773 9009 Acked Write 8 Done Done Sent false
28774 9010 Acked Write 7 Done Done Sent false
28775 9011 Acked Write 5 Done Done Sent false
28776 9012 Acked Write 1 Done Done Sent false
28777 9013 Acked Write 7 Done Done Sent false
28778 9014 Acked Write 1 Done Done Sent false
28779 9015 Acked Write 3 Done Done Sent false
28780 9016 Acked Write 8 Done Done Sent false
28781 9017 Acked Read 4 Done Done Sent false
28782 9018 Acked Read 4 Done Done Sent false
28783 9019 Acked Write 7 Done Done Sent false
28784 9020 Acked Write 5 Done Done Sent false
28785 9021 Acked Write 9 Done Done Sent false
28786 9022 Acked Write 8 Done Done Sent false
28787 9023 Acked Read 9 Done Done Sent false
28788 9024 Acked Read 3 Done Done Sent false
28789 9025 Acked Write 7 Done Done Sent false
28790 9026 Acked Read 4 Done Done Sent false
28791 9027 Acked Read 5 Done Done Sent false
28792 9028 Acked Write 3 Done Done Sent false
28793 9029 Acked Write 5 Done Done Sent false
28794 9030 Acked Write 1 Done Done Sent false
28795 9031 Acked Write 8 Done Done Sent false
28796 9032 Acked Read 5 Done Done Sent false
28797 9033 Acked Read 3 Done Done Sent false
28798 9034 Acked Write 2 Done Done Sent false
28799 9035 Acked Read 9 Done Done Sent false
28800 9036 Acked Read 7 Done Done Sent false
28801 9037 Acked Write 10 Done Done Sent false
28802 9038 Acked Read 10 Done Done Sent false
28803 9039 Acked Write 10 Done Done Sent false
28804 9040 Acked Write 1 Done Done Sent false
28805 9041 Acked Read 7 Done Done Sent false
28806 9042 Acked Write 10 Done Done Sent false
28807 9043 Acked Read 7 Done Done Sent false
28808 9044 Acked Read 10 Done Done Sent false
28809 9045 Acked Read 4 Done Done Sent false
28810 9046 Acked Read 1 Done Done Sent false
28811 9047 Acked Read 4 Done Done Sent false
28812 9048 Acked Write 2 Done Done Sent false
28813 9049 Acked Read 3 Done Done Sent false
28814 9050 Acked Write 4 Done Done Sent false
28815 9051 Acked Read 2 Done Done Sent false
28816 9052 Acked Flush 0 Done Done Sent false
28817 9053 Acked Write 2 Done Done Sent false
28818 9054 Acked Read 8 Done Done Sent false
28819 9055 Acked Flush 0 Done Done Sent false
28820 9056 Acked Write 3 Done Done Sent false
28821 9057 Acked Write 10 Done Done Sent false
28822 9058 Acked Write 4 Done Done Sent false
28823 9059 Acked Write 7 Done Done Sent false
28824 9060 Acked Read 9 Done Done Sent false
28825 9061 Acked Write 8 Done Done Sent false
28826 9062 Acked Read 5 Done Done Sent false
28827 9063 Acked Read 8 Done Done Sent false
28828 9064 Acked Read 4 Done Done Sent false
28829 9065 Acked Read 7 Done Done Sent false
28830 9066 Acked Write 5 Done Done Sent false
28831 9067 Acked Read 1 Done Done Sent false
28832 9068 Acked Write 7 Done Done Sent false
28833 9069 Acked Read 8 Done Done Sent false
28834 9070 Acked Read 6 Done Done Sent false
28835 9071 Acked Read 9 Done Done Sent false
28836 9072 Acked Read 10 Done Done Sent false
28837 9073 Acked Write 6 Done Done Sent false
28838 9074 Acked Flush 0 Done Done Sent false
28839 9075 Acked Write 4 Done Done Sent false
28840 9076 Acked Read 6 Done Done Sent false
28841 9077 Acked Read 8 Done Done Sent false
28842 9078 Acked Read 10 Done Done Sent false
28843 9079 Acked Flush 0 Done Done Sent false
28844 9080 Acked Read 10 Done Done Sent false
28845 9081 Acked Write 9 Done Done Sent false
28846 9082 Acked Write 3 Done Done Sent false
28847 9083 Acked Read 8 Done Done Sent false
28848 9084 Acked Write 6 Done Done Sent false
28849 9085 Acked Read 6 Done Done Sent false
28850 9086 Acked Write 6 Done Done Sent false
28851 9087 Acked Write 2 Done Done Sent false
28852 9088 Acked Write 7 Done Done Sent false
28853 9089 Acked Read 6 Done Done Sent false
28854 9090 Acked Write 6 Done Done Sent false
28855 9091 Acked Read 10 Done Done Sent false
28856 9092 Acked Read 9 Done Done Sent false
28857 9093 Acked Read 4 Done Done Sent false
28858 9094 Acked Flush 0 Done Done Sent false
28859 9095 Acked Read 6 Done Done Sent false
28860 9096 Acked Write 9 Done Done Sent false
28861 9097 Acked Write 6 Done Done Sent false
28862 9098 Acked Read 4 Done Done Sent false
28863 9099 Acked Read 8 Done Done Sent false
28864 9100 Acked Write 10 Done Done Sent false
28865 9101 Acked Flush 0 Done Done Sent false
28866 9102 Acked Write 2 Done Done Sent false
28867 9103 Acked Read 8 Done Done Sent false
28868 9104 Acked Read 1 Done Done Sent false
28869 9105 Acked Read 3 Done Done Sent false
28870 9106 Acked Flush 0 Done Done Sent false
28871 9107 Acked Flush 0 Done Done Sent false
28872 9108 Acked Write 3 Done Done Sent false
28873 9109 Acked Read 8 Done Done Sent false
28874 9110 Acked Flush 0 Done Done Sent false
28875 9111 Acked Write 1 Done Done Sent false
28876 9112 Acked Flush 0 Done Done Sent false
28877 9113 Acked Flush 0 Done Done Sent false
28878 9114 Acked Read 6 Done Done Sent false
28879 9115 Acked Read 2 Done Done Sent false
28880 9116 Acked Flush 0 Done Done Sent false
28881 9117 Acked Read 9 Done Done Sent false
28882 9118 Acked Flush 0 Done Done Sent false
28883 9119 Acked Write 1 Done Done Sent false
28884 9120 Acked Read 4 Done Done Sent false
28885 9121 Acked Read 10 Done Done Sent false
28886 9122 Acked Read 1 Done Done Sent false
28887 9123 Acked Write 7 Done Done Sent false
28888 9124 Acked Write 3 Done Done Sent false
28889 9125 Acked Write 10 Done Done Sent false
28890 9126 Acked Write 3 Done Done Sent false
28891 9127 Acked Write 6 Done Done Sent false
28892 9128 Acked Write 5 Done Done Sent false
28893 9129 Acked Read 10 Done Done Sent false
28894 9130 Acked Read 2 Done Done Sent false
28895 9131 Acked Read 1 Done Done Sent false
28896 9132 Acked Read 5 Done Done Sent false
28897 9133 Acked Read 9 Done Done Sent false
28898 9134 Acked Write 4 Done Done Sent false
28899 9135 Acked Read 8 Done Done Sent false
28900 9136 Acked Write 3 Done Done Sent false
28901 9137 Acked Flush 0 Done Done Sent false
28902 9138 Acked Write 3 Done Done Sent false
28903 9139 Acked Write 4 Done Done Sent false
28904 9140 Acked Write 5 Done Done Sent false
28905 9141 Acked Read 6 Done Done Sent false
28906 9142 Acked Write 4 Done Done Sent false
28907 9143 Acked Read 4 Done Done Sent false
28908 9144 Acked Write 4 Done Done Sent false
28909 9145 Acked Write 5 Done Done Sent false
28910 9146 Acked Read 4 Done Done Sent false
28911 9147 Acked Read 3 Done Done Sent false
28912 9148 Acked Read 6 Done Done Sent false
28913 9149 Acked Read 8 Done Done Sent false
28914 9150 Acked Write 6 Done Done Sent false
28915 9151 Acked Read 3 Done Done Sent false
28916 9152 Acked Flush 0 Done Done Sent false
28917 STATES DS:0 DS:1 DS:2 TOTAL
28918 Sent 0 0 302 302
28919 Done 302 302 0 604
28920 Skipped 0 0 0 0
28921 Error 0 0 0 0
28922 Last Flush: Some(JobId(9152)) Some(JobId(9152)) Some(JobId(8850))
28923 Downstairs last five completed: 8850 8849 8848 8847 8846
28924 Upstairs last five completed: 9152 9151 9150 9149 9148
28925 CLIENT: Up:0 ds:302 act:2
289262025-01-23T17:43:48.186ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
289272025-01-23T17:43:48.186ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41219) = io task client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
289282025-01-23T17:43:48.188ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
289292025-01-23T17:43:48.188ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
289302025-01-23T17:43:48.188ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
289312025-01-23T17:43:48.188ZINFOcrucible: send last flush ID to this DS: Some(JobId(8850)) = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
289322025-01-23T17:43:48.188ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
289332025-01-23T17:43:48.188ZINFOcrucible: [2] client re-new 302 jobs since flush Some(JobId(8850)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
289342025-01-23T17:43:48.189ZINFOcrucible: [2] Replayed 302 jobs since flush: Some(JobId(8850)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
28935 ----------------------------------------------------------------
28936 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28937 Downstairs last five completed: 9152 9151 9150 9149 9148
28938 Upstairs last five completed: 9152 9151 9150 9149 9148
28939 CLIENT: Up:0 ds:0 act:3
28940 Replay: All jobs finished
28941 Main task has work to do, go find it
28942 got dsc Stop(0)
28943 stop 0
28944 Jan 23 17:43:50.430 INFO [8810] Got stop action so:false kr:false
28945 request completed, latency_us: 408, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 50f994bf-27f4-4ae1-8e24-b64ca7c2589c, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
289462025-01-23T17:43:50.431ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
28947 Jan 23 17:43:50.431 INFO request completed, latency_us: 126, response_code: 200, uri{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:43:50.431519845Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
28948 : /state/cid/0, method: GET, req_id: 6af887a8-d9ff-4664-9c30-4c3e5508ae90, remote_addr:{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:43:50.431554292Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
28949 127.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-23T17:43:50.431595408Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
28950 0.0.1:42015, local_addr: 127.0.0.1:9998
28951 [8810] Exited with: ExitStatus(unix_wait_status(9))
28952 [8810][0] reports Exit
28953 Jan 23 17:43:54.434 INFO request completed, latency_us: 233, response_code: 200, uri: /state/cid/0, method: GET, req_id: 53c778b6-1ab3-47bc-b0d1-bf8bc7361b18, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
28954 001/300 Write block 2858 len 32768 data: 6 7 7 8 8 8 8 8
28955 002/300 Flush
28956 003/300 Write block 2457 len 32768 data: 6 5 5 4 5 6 6 6
28957 004/300 Write block 3905 len 24576 data: 4 4 6 7 6 5
28958 005/300 Write block 1139 len 36864 data: 5 5 5 7 6 7 6 5 6
28959 006/300 Read block 3655 len 4096
28960 007/300 Read block 4741 len 40960
28961 008/300 Write block 204 len 16384 data: 8 7 7 3
28962 009/300 Write block 1441 len 16384 data: 3 4 3 3
28963 010/300 Read block 1118 len 12288
28964 011/300 Flush
28965 012/300 Read block 4595 len 16384
28966 013/300 Read block 3740 len 4096
28967 014/300 Write block 1791 len 20480 data: 4 4 5 7 7
28968 015/300 Write block 4417 len 40960 data: 4 5 4 2 3 3 4 3 4 7
28969 016/300 Flush
28970 017/300 Write block 3203 len 12288 data: 5 6 5
28971 018/300 Write block 3884 len 20480 data: 8 8 6 6 6
28972 019/300 Write block 3927 len 40960 data: 2 2 2 3 4 5 7 6 7 5
28973 020/300 Read block 1919 len 40960
28974 021/300 Write block 3198 len 12288 data: 6 6 6
28975 022/300 Write block 4545 len 4096 data: 8
28976 023/300 Write block 3581 len 40960 data: 10 10 9 9 8 5 4 5 6 7
28977 024/300 Read block 3295 len 4096
28978 025/300 Write block 3813 len 36864 data: 7 7 5 4 3 4 4 4 5
28979 026/300 Write block 999 len 12288 data: 5 5 3
28980 027/300 Write block 1856 len 12288 data: 7 8 6
28981 028/300 Read block 3110 len 12288
28982 029/300 Write block 1055 len 4096 data: 6
28983 030/300 Read block 4342 len 12288
28984 031/300 Read block 1538 len 28672
28985 032/300 Read block 1211 len 20480
28986 033/300 Write block 1070 len 40960 data: 7 8 9 9 8 5 4 4 4 3
28987 034/300 Read block 1672 len 20480
28988 035/300 Read block 814 len 16384
28989 036/300 Write block 4532 len 8192 data: 4 5
28990 037/300 Write block 4447 len 28672 data: 5 5 6 6 6 5 5
28991 038/300 Write block 3788 len 4096 data: 5
28992 039/300 Read block 1156 len 36864
28993 040/300 Write block 2643 len 4096 data: 4
28994 041/300 Read block 4550 len 4096
28995 042/300 Write block 4706 len 40960 data: 8 8 10 6 7 5 5 6 5 5
28996 043/300 Read block 2433 len 24576
28997 044/300 Write block 4841 len 20480 data: 6 6 5 5 7
28998 045/300 Read block 1838 len 24576
28999 046/300 Write block 3855 len 8192 data: 7 4
29000 047/300 Read block 4518 len 12288
29001 048/300 Write block 4343 len 36864 data: 4 5 4 3 3 4 5 4 4
29002 049/300 Read block 102 len 40960
29003 050/300 Write block 1112 len 36864 data: 5 5 7 8 7 6 4 4 5
29004 051/300 Write block 1403 len 40960 data: 6 6 5 6 6 6 6 5 6 7
29005 052/300 Write block 3629 len 12288 data: 6 4 4
29006 053/300 Write block 1933 len 16384 data: 4 4 4 3
29007 054/300 Read block 2093 len 40960
29008 055/300 Read block 3413 len 24576
29009 056/300 Write block 3107 len 8192 data: 9 9
29010 057/300 Read block 4134 len 4096
29011 058/300 Write block 2930 len 28672 data: 6 7 6 5 4 4 4
29012 059/300 Flush
29013 060/300 Write block 3893 len 40960 data: 7 7 6 5 4 2 3 3 4 4
29014 061/300 Read block 4096 len 40960
29015 062/300 Write block 1169 len 36864 data: 4 4 8 8 8 6 4 4 6
29016 063/300 Write block 1681 len 4096 data: 9
29017 064/300 Read block 3500 len 20480
29018 065/300 Write block 3471 len 16384 data: 6 6 6 8
29019 066/300 Write block 4336 len 40960 data: 5 4 3 3 4 5 4 5 6 5
29020 067/300 Read block 1609 len 4096
29021 068/300 Write block 2060 len 28672 data: 8 9 10 10 9 6 6
29022 069/300 Read block 570 len 28672
29023 070/300 Read block 2944 len 32768
29024 071/300 Write block 3845 len 20480 data: 3 3 4 6 6
29025 072/300 Write block 4676 len 12288 data: 5 6 6
29026 073/300 Read block 140 len 8192
29027 074/300 Read block 3099 len 28672
29028 075/300 Read block 4354 len 28672
29029 076/300 Write block 4768 len 28672 data: 5 4 4 5 3 5 5
29030 077/300 Read block 1526 len 12288
29031 078/300 Write block 3159 len 20480 data: 8 8 7 8 9
29032 079/300 Read block 1271 len 24576
29033 080/300 Flush
29034 081/300 Read block 2293 len 24576
29035 082/300 Read block 4019 len 32768
29036 083/300 Flush
29037 084/300 Read block 4606 len 32768
29038 085/300 Write block 2865 len 16384 data: 9 10 11 10
29039 086/300 Read block 270 len 20480
29040 087/300 Write block 4464 len 4096 data: 5
29041 088/300 Flush
29042 089/300 Read block 698 len 16384
29043 090/300 Read block 3404 len 12288
29044 091/300 Read block 2161 len 28672
29045 092/300 Write block 3373 len 4096 data: 2
29046 093/300 Read block 825 len 32768
29047 094/300 Read block 2583 len 8192
29048 095/300 Write block 1179 len 4096 data: 6
29049 096/300 Write block 1902 len 32768 data: 8 7 9 11 10 10 13 11
29050 097/300 Write block 3745 len 12288 data: 5 7 6
29051 098/300 Read block 2639 len 20480
29052 099/300 Write block 1895 len 20480 data: 4 6 7 7 7
29053 100/300 Write block 1708 len 20480 data: 5 5 7 9 8
29054 101/300 Read block 470 len 20480
29055 102/300 Write block 4970 len 20480 data: 10 13 14 10 9
29056 103/300 Write block 486 len 16384 data: 5 4 3 3
29057 104/300 Write block 491 len 16384 data: 3 4 5 5
29058 105/300 Write block 1279 len 12288 data: 9 8 5
29059 106/300 Read block 581 len 12288
29060 107/300 Read block 2872 len 36864
29061 108/300 Write block 2608 len 36864 data: 6 4 7 9 10 9 9 8 7
29062 109/300 Read block 3446 len 4096
29063 110/300 Write block 1116 len 4096 data: 8
29064 111/300 Read block 2549 len 8192
29065 112/300 Write block 4997 len 12288 data: 4 4 3
29066 113/300 Read block 3465 len 40960
29067 114/300 Read block 4062 len 32768
29068 115/300 Read block 3661 len 32768
29069 116/300 Write block 3175 len 8192 data: 4 5
29070 117/300 Read block 2209 len 32768
29071 118/300 Read block 1965 len 8192
29072 119/300 Write block 1112 len 32768 data: 6 6 8 9 9 7 5 5
29073 120/300 Read block 3633 len 36864
29074 121/300 Write block 3982 len 28672 data: 9 8 8 7 7 5 6
29075 122/300 Write block 1085 len 20480 data: 7 7 8 8 7
29076 123/300 Read block 3891 len 32768
29077 124/300 Write block 1147 len 40960 data: 7 5 5 4 4 5 5 6 5 7
29078 125/300 Write block 755 len 12288 data: 5 6 5
29079 126/300 Write block 2544 len 40960 data: 7 7 7 8 7 6 5 4 4 3
29080 127/300 Write block 1435 len 40960 data: 4 4 4 4 4 3 4 5 4 4
29081 128/300 Write block 3434 len 40960 data: 8 8 6 4 3 4 4 5 7 7
29082 129/300 Write block 134 len 40960 data: 5 5 6 6 6 8 8 9 8 7
29083 130/300 Write block 3081 len 12288 data: 4 4 4
29084 131/300 Read block 2535 len 40960
29085 132/300 Write block 3767 len 24576 data: 5 5 4 4 4 5
29086 133/300 Write block 3893 len 24576 data: 8 8 7 6 5 3
29087 134/300 Flush
29088 135/300 Flush
29089 136/300 Write block 974 len 20480 data: 9 11 9 8 8
29090 137/300 Write block 1717 len 32768 data: 3 3 4 5 5 6 7 7
29091 138/300 Read block 3983 len 8192
29092 139/300 Read block 50 len 36864
29093 140/300 Flush
29094 141/300 Write block 2014 len 20480 data: 5 3 7 5 4
29095 142/300 Write block 3505 len 8192 data: 6 5
29096 143/300 Flush
29097 144/300 Write block 2608 len 36864 data: 7 5 8 10 11 10 10 9 8
29098 145/300 Write block 2852 len 8192 data: 5 5
29099 146/300 Read block 3997 len 32768
29100 147/300 Write block 3374 len 16384 data: 6 5 3 2
29101 148/300 Read block 1330 len 24576
29102 149/300 Write block 2909 len 20480 data: 3 2 3 3 3
29103 150/300 Read block 2065 len 8192
29104 151/300 Write block 4512 len 20480 data: 6 7 9 10 8
29105 152/300 Flush
29106 153/300 Write block 1963 len 32768 data: 7 7 9 8 8 7 9 10
29107 154/300 Write block 2691 len 32768 data: 6 5 5 5 6 6 6 6
29108 155/300 Write block 3469 len 28672 data: 8 7 7 7 7 9 9
29109 156/300 Read block 1942 len 36864
29110 157/300 Read block 4727 len 20480
29111 158/300 Read block 3435 len 40960
29112 159/300 Read block 3373 len 16384
29113 160/300 Write block 2597 len 28672 data: 7 6 5 5 3 2 4
29114 161/300 Write block 267 len 24576 data: 3 2 2 2 2 3
29115 162/300 Flush
29116 163/300 Flush
29117 164/300 Write block 1603 len 32768 data: 3 5 4 4 4 6 6 7
29118 165/300 Write block 3931 len 28672 data: 5 6 8 7 8 6 5
29119 166/300 Write block 723 len 8192 data: 6 6
29120 167/300 Read block 1486 len 4096
29121 168/300 Read block 886 len 36864
29122 169/300 Write block 277 len 20480 data: 5 6 9 8 6
29123 170/300 Read block 2231 len 36864
29124 171/300 Read block 3546 len 24576
29125 172/300 Read block 1289 len 4096
29126 173/300 Read block 2946 len 4096
29127 174/300 Read block 991 len 28672
29128 175/300 Write block 1984 len 24576 data: 3 3 3 2 4 4
29129 176/300 Read block 2559 len 40960
29130 177/300 Flush
29131 178/300 Read block 824 len 36864
29132 179/300 Read block 3717 len 32768
29133 180/300 Flush
29134 181/300 Write block 2388 len 40960 data: 5 5 4 4 3 5 5 8 11 12
29135 182/300 Write block 643 len 36864 data: 10 9 10 9 7 7 6 6 7
29136 183/300 Read block 4444 len 12288
29137 184/300 Write block 699 len 40960 data: 5 5 5 4 4 3 5 5 5 5
29138 185/300 Read block 3369 len 20480
29139 186/300 Read block 77 len 4096
29140 187/300 Read block 4436 len 4096
29141 188/300 Read block 1273 len 32768
29142 189/300 Write block 3782 len 12288 data: 2 3 3
29143 190/300 Read block 2091 len 32768
29144 191/300 Write block 4769 len 12288 data: 5 5 6
29145 192/300 Write block 3049 len 16384 data: 6 7 5 5
29146 193/300 Flush
29147 194/300 Read block 1510 len 24576
29148 195/300 Read block 462 len 24576
29149 196/300 Flush
29150 197/300 Read block 580 len 20480
29151 198/300 Read block 55 len 16384
29152 199/300 Read block 4734 len 40960
29153 200/300 Read block 1072 len 20480
29154 201/300 Read block 2845 len 16384
29155 202/300 Read block 1146 len 8192
29156 203/300 Write block 1375 len 20480 data: 9 9 7 6 4
29157 204/300 Write block 1488 len 8192 data: 5 4
29158 205/300 Write block 1129 len 32768 data: 3 3 3 2 2 2 3 5
29159 206/300 Read block 3459 len 40960
29160 207/300 Write block 1084 len 16384 data: 7 8 8 9
29161 208/300 Read block 1466 len 16384
29162 209/300 Read block 2341 len 8192
29163 210/300 Write block 2685 len 28672 data: 10 10 9 10 8 8 7
29164 211/300 Read block 1806 len 24576
29165 212/300 Write block 1971 len 16384 data: 8 8 4 6
29166 213/300 Read block 2424 len 40960
29167 214/300 Write block 4330 len 32768 data: 4 4 5 4 5 6 6 5
29168 215/300 Read block 3066 len 16384
29169 216/300 Read block 4841 len 36864
29170 217/300 Write block 3040 len 12288 data: 7 7 6
29171 218/300 Read block 3866 len 40960
29172 219/300 Read block 4387 len 12288
29173 220/300 Write block 2905 len 4096 data: 6
29174 221/300 Read block 1385 len 36864
29175 222/300 Write block 2262 len 36864 data: 8 7 7 6 4 4 4 4 5
29176 223/300 Read block 1136 len 20480
29177 224/300 Read block 1859 len 28672
29178 225/300 Write block 3670 len 40960 data: 9 9 8 8 7 6 5 6 7 5
29179 226/300 Read block 2858 len 4096
29180 227/300 Write block 2021 len 36864 data: 6 6 7 6 6 4 4 3 2
29181 228/300 Read block 2179 len 24576
29182 229/300 Write block 3579 len 8192 data: 9 9
29183 230/300 Read block 4873 len 20480
29184 231/300 Read block 4028 len 24576
29185 232/300 Read block 4104 len 16384
29186 233/300 Write block 3418 len 4096 data: 7
29187 234/300 Read block 4783 len 40960
29188 235/300 Read block 4736 len 36864
29189 236/300 Write block 2297 len 20480 data: 3 4 3 3 5
29190 237/300 Write block 607 len 28672 data: 5 4 4 5 5 5 6
29191 238/300 Write block 450 len 8192 data: 2 2
29192 239/300 Flush
29193 240/300 Read block 3030 len 40960
29194 241/300 Flush
29195 242/300 Flush
29196 243/300 Flush
29197 244/300 Read block 1036 len 32768
29198 245/300 Read block 3667 len 4096
29199 246/300 Read block 1053 len 12288
29200 247/300 Read block 3787 len 8192
29201 248/300 Write block 3092 len 24576 data: 3 3 2 3 3 2
29202 249/300 Read block 2942 len 16384
29203 250/300 Read block 1846 len 32768
29204 251/300 Write block 2426 len 36864 data: 6 5 5 7 6 6 5 6 6
29205 252/300 Read block 464 len 32768
29206 253/300 Read block 2229 len 8192
29207 254/300 Write block 4826 len 12288 data: 4 5 5
29208 255/300 Read block 550 len 28672
29209 256/300 Read block 619 len 20480
29210 257/300 Write block 4926 len 16384 data: 7 6 7 6
29211 258/300 Write block 444 len 28672 data: 4 4 4 6 5 3 3
29212 259/300 Flush
29213 260/300 Write block 2075 len 8192 data: 8 7
29214 261/300 Read block 1217 len 36864
29215 262/300 Write block 2338 len 20480 data: 6 5 7 7 6
29216 263/300 Read block 1585 len 28672
29217 264/300 Write block 3445 len 40960 data: 5 6 6 5 5 5 5 5 5 5
29218 265/300 Read block 2649 len 8192
29219 266/300 Read block 394 len 4096
29220 267/300 Read block 1327 len 16384
29221 268/300 Read block 1165 len 24576
29222 269/300 Write block 3992 len 36864 data: 5 5 4 4 4 6 7 7 8
29223 270/300 Write block 3221 len 20480 data: 4 5 7 6 8
29224 271/300 Write block 4666 len 32768 data: 4 5 5 4 7 7 7 6
29225 272/300 Read block 532 len 16384
29226 273/300 Write block 375 len 24576 data: 11 10 6 5 4 3
29227 274/300 Write block 4199 len 4096 data: 5
29228 275/300 Write block 2782 len 8192 data: 5 5
29229 276/300 Write block 4984 len 4096 data: 4
29230 277/300 Flush
29231 278/300 Flush
29232 279/300 Read block 4147 len 20480
29233 280/300 Flush
29234 281/300 Read block 4782 len 28672
29235 282/300 Write block 2047 len 12288 data: 5 6 6
29236 283/300 Read block 635 len 8192
29237 284/300 Write block 3366 len 4096 data: 3
29238 285/300 Write block 426 len 16384 data: 5 4 3 4
29239 286/300 Read block 290 len 16384
29240 287/300 Read block 1957 len 36864
29241 288/300 Read block 3344 len 8192
29242 289/300 Read block 282 len 32768
29243 290/300 Read block 1821 len 32768
29244 291/300 Read block 3288 len 16384
29245 292/300 Write block 4179 len 12288 data: 8 8 8
29246 293/300 Read block 3356 len 24576
29247 294/300 Read block 738 len 32768
29248 295/300 Read block 2855 len 20480
29249 296/300 Write block 314 len 36864 data: 3 3 3 2 2 3 3 4 5
29250 297/300 Flush
29251 298/300 Read block 924 len 8192
29252 299/300 Write block 4503 len 28672 data: 6 7 5 5 6 6 7
29253 300/300 Write block 3720 len 12288 data: 5 5 6
29254 301/300 Read block 4213 len 4096
29255 Main task has work to do, go find it
29256 got dsc Start(0)
29257 start 0
29258 Jan 23 17:43:54.671 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/0, method: POST, req_id: ef229750-2b54-428a-a7cf-1a2fb260f40d, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
29259 Replay: started 0, returned:Ok(())
29260 ----------------------------------------------------------------
29261 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29262 States: Offline Active Active
29263 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29264 9153 Acked Write 8 Sent Done Done false
29265 9154 Acked Flush 0 Sent Done Done false
29266 9155 Acked Write 8 Sent Done Done false
29267 9156 Acked Write 6 Sent Done Done false
29268 9157 Acked Write 9 Sent Done Done false
29269 9158 Acked Read 1 Sent Done Done false
29270 9159 Acked Read 10 Sent Done Done false
29271 9160 Acked Write 4 Sent Done Done false
29272 9161 Acked Write 4 Sent Done Done false
29273 9162 Acked Read 3 Sent Done Done false
29274 9163 Acked Flush 0 Sent Done Done false
29275 9164 Acked Read 4 Sent Done Done false
29276 9165 Acked Read 1 Sent Done Done false
29277 9166 Acked Write 5 Sent Done Done false
29278 9167 Acked Write 10 Sent Done Done false
29279 9168 Acked Flush 0 Sent Done Done false
29280 9169 Acked Write 3 Sent Done Done false
29281 9170 Acked Write 5 Sent Done Done false
29282 9171 Acked Write 10 Sent Done Done false
29283 9172 Acked Read 10 Sent Done Done false
29284 9173 Acked Write 3 Sent Done Done false
29285 9174 Acked Write 1 Sent Done Done false
29286 9175 Acked Write 10 Sent Done Done false
29287 9176 Acked Read 1 Sent Done Done false
29288 9177 Acked Write 9 Sent Done Done false
29289 9178 Acked Write 3 Sent Done Done false
29290 9179 Acked Write 3 Sent Done Done false
29291 9180 Acked Read 3 Sent Done Done false
29292 9181 Acked Write 1 Sent Done Done false
29293 9182 Acked Read 3 Sent Done Done false
29294 9183 Acked Read 7 Sent Done Done false
29295 9184 Acked Read 5 Sent Done Done false
29296 9185 Acked Write 10 Sent Done Done false
29297 9186 Acked Read 5 Sent Done Done false
29298 9187 Acked Read 4 Sent Done Done false
29299 9188 Acked Write 2 Sent Done Done false
29300 9189 Acked Write 7 Sent Done Done false
29301 9190 Acked Write 1 Sent Done Done false
29302 9191 Acked Read 9 Sent Done Done false
29303 9192 Acked Write 1 Sent Done Done false
29304 9193 Acked Read 1 Sent Done Done false
29305 9194 Acked Write 10 Sent Done Done false
29306 9195 Acked Read 6 Sent Done Done false
29307 9196 Acked Write 5 Sent Done Done false
29308 9197 Acked Read 6 Sent Done Done false
29309 9198 Acked Write 2 Sent Done Done false
29310 9199 Acked Read 3 Sent Done Done false
29311 9200 Acked Write 9 Sent Done Done false
29312 9201 Acked Read 10 Sent Done Done false
29313 9202 Acked Write 9 Sent Done Done false
29314 9203 Acked Write 10 Sent Done Done false
29315 9204 Acked Write 3 Sent Done Done false
29316 9205 Acked Write 4 Sent Done Done false
29317 9206 Acked Read 10 Sent Done Done false
29318 9207 Acked Read 6 Sent Done Done false
29319 9208 Acked Write 2 Sent Done Done false
29320 9209 Acked Read 1 Sent Done Done false
29321 9210 Acked Write 7 Sent Done Done false
29322 9211 Acked Flush 0 Sent Done Done false
29323 9212 Acked Write 10 Sent Done Done false
29324 9213 Acked Read 10 Sent Done Done false
29325 9214 Acked Write 9 Sent Done Done false
29326 9215 Acked Write 1 Sent Done Done false
29327 9216 Acked Read 5 Sent Done Done false
29328 9217 Acked Write 4 Sent Done Done false
29329 9218 Acked Write 10 Sent Done Done false
29330 9219 Acked Read 1 Sent Done Done false
29331 9220 Acked Write 7 Sent Done Done false
29332 9221 Acked Read 7 Sent Done Done false
29333 9222 Acked Read 8 Sent Done Done false
29334 9223 Acked Write 5 Sent Done Done false
29335 9224 Acked Write 3 Sent Done Done false
29336 9225 Acked Read 2 Sent Done Done false
29337 9226 Acked Read 7 Sent Done Done false
29338 9227 Acked Read 7 Sent Done Done false
29339 9228 Acked Write 7 Sent Done Done false
29340 9229 Acked Read 3 Sent Done Done false
29341 9230 Acked Write 5 Sent Done Done false
29342 9231 Acked Read 6 Sent Done Done false
29343 9232 Acked Flush 0 Sent Done Done false
29344 9233 Acked Read 6 Sent Done Done false
29345 9234 Acked Read 8 Sent Done Done false
29346 9235 Acked Flush 0 Sent Done Done false
29347 9236 Acked Read 8 Sent Done Done false
29348 9237 Acked Write 4 Sent Done Done false
29349 9238 Acked Read 5 Sent Done Done false
29350 9239 Acked Write 1 Sent Done Done false
29351 9240 Acked Flush 0 Sent Done Done false
29352 9241 Acked Read 4 Sent Done Done false
29353 9242 Acked Read 3 Sent Done Done false
29354 9243 Acked Read 7 Sent Done Done false
29355 9244 Acked Write 1 Sent Done Done false
29356 9245 Acked Read 8 Sent Done Done false
29357 9246 Acked Read 2 Sent Done Done false
29358 9247 Acked Write 1 Sent Done Done false
29359 9248 Acked Write 8 Sent Done Done false
29360 9249 Acked Write 3 Sent Done Done false
29361 9250 Acked Read 5 Sent Done Done false
29362 9251 Acked Write 5 Sent Done Done false
29363 9252 Acked Write 5 Sent Done Done false
29364 9253 Acked Read 5 Sent Done Done false
29365 9254 Acked Write 5 Sent Done Done false
29366 9255 Acked Write 4 Sent Done Done false
29367 9256 Acked Write 4 Sent Done Done false
29368 9257 Acked Write 3 Sent Done Done false
29369 9258 Acked Read 3 Sent Done Done false
29370 9259 Acked Read 9 Sent Done Done false
29371 9260 Acked Write 9 Sent Done Done false
29372 9261 Acked Read 1 Sent Done Done false
29373 9262 Acked Write 1 Sent Done Done false
29374 9263 Acked Read 2 Sent Done Done false
29375 9264 Acked Write 3 Sent Done Done false
29376 9265 Acked Read 10 Sent Done Done false
29377 9266 Acked Read 8 Sent Done Done false
29378 9267 Acked Read 8 Sent Done Done false
29379 9268 Acked Write 2 Sent Done Done false
29380 9269 Acked Read 8 Sent Done Done false
29381 9270 Acked Read 2 Sent Done Done false
29382 9271 Acked Write 8 Sent Done Done false
29383 9272 Acked Read 9 Sent Done Done false
29384 9273 Acked Write 7 Sent Done Done false
29385 9274 Acked Write 5 Sent Done Done false
29386 9275 Acked Read 8 Sent Done Done false
29387 9276 Acked Write 10 Sent Done Done false
29388 9277 Acked Write 3 Sent Done Done false
29389 9278 Acked Write 10 Sent Done Done false
29390 9279 Acked Write 10 Sent Done Done false
29391 9280 Acked Write 10 Sent Done Done false
29392 9281 Acked Write 10 Sent Done Done false
29393 9282 Acked Write 3 Sent Done Done false
29394 9283 Acked Read 10 Sent Done Done false
29395 9284 Acked Write 6 Sent Done Done false
29396 9285 Acked Write 6 Sent Done Done false
29397 9286 Acked Flush 0 Sent Done Done false
29398 9287 Acked Flush 0 Sent Done Done false
29399 9288 Acked Write 5 Sent Done Done false
29400 9289 Acked Write 8 Sent Done Done false
29401 9290 Acked Read 2 Sent Done Done false
29402 9291 Acked Read 9 Sent Done Done false
29403 9292 Acked Flush 0 Sent Done Done false
29404 9293 Acked Write 5 Sent Done Done false
29405 9294 Acked Write 2 Sent Done Done false
29406 9295 Acked Flush 0 Sent Done Done false
29407 9296 Acked Write 9 Sent Done Done false
29408 9297 Acked Write 2 Sent Done Done false
29409 9298 Acked Read 8 Sent Done Done false
29410 9299 Acked Write 4 Sent Done Done false
29411 9300 Acked Read 6 Sent Done Done false
29412 9301 Acked Write 5 Sent Done Done false
29413 9302 Acked Read 2 Sent Done Done false
29414 9303 Acked Write 5 Sent Done Done false
29415 9304 Acked Flush 0 Sent Done Done false
29416 9305 Acked Write 8 Sent Done Done false
29417 9306 Acked Write 8 Sent Done Done false
29418 9307 Acked Write 7 Sent Done Done false
29419 9308 Acked Read 9 Sent Done Done false
29420 9309 Acked Read 5 Sent Done Done false
29421 9310 Acked Read 10 Sent Done Done false
29422 9311 Acked Read 4 Sent Done Done false
29423 9312 Acked Write 7 Sent Done Done false
29424 9313 Acked Write 6 Sent Done Done false
29425 9314 Acked Flush 0 Sent Done Done false
29426 9315 Acked Flush 0 Sent Done Done false
29427 9316 Acked Write 8 Sent Done Done false
29428 9317 Acked Write 7 Sent Done Done false
29429 9318 Acked Write 2 Sent Done Done false
29430 9319 Acked Read 1 Sent Done Done false
29431 9320 Acked Read 9 Sent Done Done false
29432 9321 Acked Write 5 Sent Done Done false
29433 9322 Acked Read 9 Sent Done Done false
29434 9323 Acked Read 6 Sent Done Done false
29435 9324 Acked Read 1 Sent Done Done false
29436 9325 Acked Read 1 Sent Done Done false
29437 9326 Acked Read 7 Sent Done Done false
29438 9327 Acked Write 6 Sent Done Done false
29439 9328 Acked Read 10 Sent Done Done false
29440 9329 Acked Flush 0 Sent Done Done false
29441 9330 Acked Read 9 Sent Done Done false
29442 9331 Acked Read 8 Sent Done Done false
29443 9332 Acked Flush 0 Sent Done Done false
29444 9333 Acked Write 10 Sent Done Done false
29445 9334 Acked Write 9 Sent Done Done false
29446 9335 Acked Read 3 Sent Done Done false
29447 9336 Acked Write 10 Sent Done Done false
29448 9337 Acked Read 5 Sent Done Done false
29449 9338 Acked Read 1 Sent Done Done false
29450 9339 Acked Read 1 Sent Done Done false
29451 9340 Acked Read 8 Sent Done Done false
29452 9341 Acked Write 3 Sent Done Done false
29453 9342 Acked Read 8 Sent Done Done false
29454 9343 Acked Write 3 Sent Done Done false
29455 9344 Acked Write 4 Sent Done Done false
29456 9345 Acked Flush 0 Sent Done Done false
29457 9346 Acked Read 6 Sent Done Done false
29458 9347 Acked Read 6 Sent Done Done false
29459 9348 Acked Flush 0 Sent Done Done false
29460 9349 Acked Read 5 Sent Done Done false
29461 9350 Acked Read 4 Sent Done Done false
29462 9351 Acked Read 10 Sent Done Done false
29463 9352 Acked Read 5 Sent Done Done false
29464 9353 Acked Read 4 Sent Done Done false
29465 9354 Acked Read 2 Sent Done Done false
29466 9355 Acked Write 5 Sent Done Done false
29467 9356 Acked Write 2 Sent Done Done false
29468 9357 Acked Write 8 Sent Done Done false
29469 9358 Acked Read 10 Sent Done Done false
29470 9359 Acked Write 4 Sent Done Done false
29471 9360 Acked Read 4 Sent Done Done false
29472 9361 Acked Read 2 Sent Done Done false
29473 9362 Acked Write 7 Sent Done Done false
29474 9363 Acked Read 6 Sent Done Done false
29475 9364 Acked Write 4 Sent Done Done false
29476 9365 Acked Read 10 Sent Done Done false
29477 9366 Acked Write 8 Sent Done Done false
29478 9367 Acked Read 4 Sent Done Done false
29479 9368 Acked Read 9 Sent Done Done false
29480 9369 Acked Write 3 Sent Done Done false
29481 9370 Acked Read 10 Sent Done Done false
29482 9371 Acked Read 3 Sent Done Done false
29483 9372 Acked Write 1 Sent Done Done false
29484 9373 Acked Read 9 Sent Done Done false
29485 9374 Acked Write 9 Sent Done Done false
29486 9375 Acked Read 5 Sent Done Done false
29487 9376 Acked Read 7 Sent Done Done false
29488 9377 Acked Write 10 Sent Done Done false
29489 9378 Acked Read 1 Sent Done Done false
29490 9379 Acked Write 9 Sent Done Done false
29491 9380 Acked Read 6 Sent Done Done false
29492 9381 Acked Write 2 Sent Done Done false
29493 9382 Acked Read 5 Sent Done Done false
29494 9383 Acked Read 6 Sent Done Done false
29495 9384 Acked Read 4 Sent Done Done false
29496 9385 Acked Write 1 Sent Done Done false
29497 9386 Acked Read 10 Sent Done Done false
29498 9387 Acked Read 9 Sent Done Done false
29499 9388 Acked Write 5 Sent Done Done false
29500 9389 Acked Write 7 Sent Done Done false
29501 9390 Acked Write 2 Sent Done Done false
29502 9391 Acked Flush 0 Sent Done Done false
29503 9392 Acked Read 10 Sent Done Done false
29504 9393 Acked Flush 0 Sent Done Done false
29505 9394 Acked Flush 0 Sent Done Done false
29506 9395 Acked Flush 0 Sent Done Done false
29507 9396 Acked Read 8 Sent Done Done false
29508 9397 Acked Read 1 Sent Done Done false
29509 9398 Acked Read 3 Sent Done Done false
29510 9399 Acked Read 2 Sent Done Done false
29511 9400 Acked Write 6 Sent Done Done false
29512 9401 Acked Read 4 Sent Done Done false
29513 9402 Acked Read 8 Sent Done Done false
29514 9403 Acked Write 9 Sent Done Done false
29515 9404 Acked Read 8 Sent Done Done false
29516 9405 Acked Read 2 Sent Done Done false
29517 9406 Acked Write 3 Sent Done Done false
29518 9407 Acked Read 7 Sent Done Done false
29519 9408 Acked Read 5 Sent Done Done false
29520 9409 Acked Write 4 Sent Done Done false
29521 9410 Acked Write 7 Sent Done Done false
29522 9411 Acked Flush 0 Sent Done Done false
29523 9412 Acked Write 2 Sent Done Done false
29524 9413 Acked Read 9 Sent Done Done false
29525 9414 Acked Write 5 Sent Done Done false
29526 9415 Acked Read 7 Sent Done Done false
29527 9416 Acked Write 10 Sent Done Done false
29528 9417 Acked Read 2 Sent Done Done false
29529 9418 Acked Read 1 Sent Done Done false
29530 9419 Acked Read 4 Sent Done Done false
29531 9420 Acked Read 6 Sent Done Done false
29532 9421 Acked Write 9 Sent Done Done false
29533 9422 Acked Write 5 Sent Done Done false
29534 9423 Acked Write 8 Sent Done Done false
29535 9424 Acked Read 4 Sent Done Done false
29536 9425 Acked Write 6 Sent Done Done false
29537 9426 Acked Write 1 Sent Done Done false
29538 9427 Acked Write 2 Sent Done Done false
29539 9428 Acked Write 1 Sent Done Done false
29540 9429 Acked Flush 0 Sent Done Done false
29541 9430 Acked Flush 0 Sent Done Done false
29542 9431 Acked Read 5 Sent Done Done false
29543 9432 Acked Flush 0 Sent Done Done false
29544 9433 Acked Read 7 Sent Done Done false
29545 9434 Acked Write 3 Sent Done Done false
29546 9435 Acked Read 2 Sent Done Done false
29547 9436 Acked Write 1 Sent Done Done false
29548 9437 Acked Write 4 Sent Done Done false
29549 9438 Acked Read 4 Sent Done Done false
29550 9439 Acked Read 9 Sent Done Done false
29551 9440 Acked Read 2 Sent Done Done false
29552 9441 Acked Read 8 Sent Done Done false
29553 9442 Acked Read 8 Sent Done Done false
29554 9443 Acked Read 4 Sent Done Done false
29555 9444 Acked Write 3 Sent Done Done false
29556 9445 Acked Read 6 Sent Done Done false
29557 9446 Acked Read 8 Sent Done Done false
29558 9447 Acked Read 5 Sent Done Done false
29559 9448 Acked Write 9 Sent Done Done false
29560 9449 Acked Flush 0 Sent Done Done false
29561 9450 Acked Read 2 Sent Done Done false
29562 9451 Acked Write 7 Sent Done Done false
29563 9452 Acked Write 3 Sent Done Done false
29564 9453 Acked Read 1 Sent Done Done false
29565 STATES DS:0 DS:1 DS:2 TOTAL
29566 Sent 301 0 0 301
29567 Done 0 301 301 602
29568 Skipped 0 0 0 0
29569 Error 0 0 0 0
29570 Last Flush: Some(JobId(9152)) Some(JobId(9449)) Some(JobId(9449))
29571 Downstairs last five completed: 9152 9151 9150 9149 9148
29572 Upstairs last five completed: 9453 9452 9451 9450 9449
29573 CLIENT: Up:0 ds:301 act:2
29574 [8810] got start action
29575 [8810] I am going to restart
29576 Starting downstairs at port 8810
29577 Make output file at "/tmp/dsc/downstairs-8810.txt"
29578 [8810][0] reports Starting
29579 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1233)
29580 [8810][0] reports Running
29581 ----------------------------------------------------------------
29582 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
29583 States: Offline Active Active
29584 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29585 9153 Acked Write 8 Sent Done Done false
29586 9154 Acked Flush 0 Sent Done Done false
29587 9155 Acked Write 8 Sent Done Done false
29588 9156 Acked Write 6 Sent Done Done false
29589 9157 Acked Write 9 Sent Done Done false
29590 9158 Acked Read 1 Sent Done Done false
29591 9159 Acked Read 10 Sent Done Done false
29592 9160 Acked Write 4 Sent Done Done false
29593 9161 Acked Write 4 Sent Done Done false
29594 9162 Acked Read 3 Sent Done Done false
29595 9163 Acked Flush 0 Sent Done Done false
29596 9164 Acked Read 4 Sent Done Done false
29597 9165 Acked Read 1 Sent Done Done false
29598 9166 Acked Write 5 Sent Done Done false
29599 9167 Acked Write 10 Sent Done Done false
29600 9168 Acked Flush 0 Sent Done Done false
29601 9169 Acked Write 3 Sent Done Done false
29602 9170 Acked Write 5 Sent Done Done false
29603 9171 Acked Write 10 Sent Done Done false
29604 9172 Acked Read 10 Sent Done Done false
29605 9173 Acked Write 3 Sent Done Done false
29606 9174 Acked Write 1 Sent Done Done false
29607 9175 Acked Write 10 Sent Done Done false
29608 9176 Acked Read 1 Sent Done Done false
29609 9177 Acked Write 9 Sent Done Done false
29610 9178 Acked Write 3 Sent Done Done false
29611 9179 Acked Write 3 Sent Done Done false
29612 9180 Acked Read 3 Sent Done Done false
29613 9181 Acked Write 1 Sent Done Done false
29614 9182 Acked Read 3 Sent Done Done false
29615 9183 Acked Read 7 Sent Done Done false
29616 9184 Acked Read 5 Sent Done Done false
29617 9185 Acked Write 10 Sent Done Done false
29618 9186 Acked Read 5 Sent Done Done false
29619 9187 Acked Read 4 Sent Done Done false
29620 9188 Acked Write 2 Sent Done Done false
29621 9189 Acked Write 7 Sent Done Done false
29622 9190 Acked Write 1 Sent Done Done false
29623 9191 Acked Read 9 Sent Done Done false
29624 9192 Acked Write 1 Sent Done Done false
29625 9193 Acked Read 1 Sent Done Done false
29626 9194 Acked Write 10 Sent Done Done false
29627 9195 Acked Read 6 Sent Done Done false
29628 9196 Acked Write 5 Sent Done Done false
29629 9197 Acked Read 6 Sent Done Done false
29630 9198 Acked Write 2 Sent Done Done false
29631 9199 Acked Read 3 Sent Done Done false
29632 9200 Acked Write 9 Sent Done Done false
29633 9201 Acked Read 10 Sent Done Done false
29634 9202 Acked Write 9 Sent Done Done false
29635 9203 Acked Write 10 Sent Done Done false
29636 9204 Acked Write 3 Sent Done Done false
29637 9205 Acked Write 4 Sent Done Done false
29638 9206 Acked Read 10 Sent Done Done false
29639 9207 Acked Read 6 Sent Done Done false
29640 9208 Acked Write 2 Sent Done Done false
29641 9209 Acked Read 1 Sent Done Done false
29642 9210 Acked Write 7 Sent Done Done false
29643 9211 Acked Flush 0 Sent Done Done false
29644 9212 Acked Write 10 Sent Done Done false
29645 9213 Acked Read 10 Sent Done Done false
29646 9214 Acked Write 9 Sent Done Done false
29647 9215 Acked Write 1 Sent Done Done false
29648 9216 Acked Read 5 Sent Done Done false
29649 9217 Acked Write 4 Sent Done Done false
29650 9218 Acked Write 10 Sent Done Done false
29651 9219 Acked Read 1 Sent Done Done false
29652 9220 Acked Write 7 Sent Done Done false
29653 9221 Acked Read 7 Sent Done Done false
29654 9222 Acked Read 8 Sent Done Done false
29655 9223 Acked Write 5 Sent Done Done false
29656 9224 Acked Write 3 Sent Done Done false
29657 9225 Acked Read 2 Sent Done Done false
29658 9226 Acked Read 7 Sent Done Done false
29659 9227 Acked Read 7 Sent Done Done false
29660 9228 Acked Write 7 Sent Done Done false
29661 9229 Acked Read 3 Sent Done Done false
29662 9230 Acked Write 5 Sent Done Done false
29663 9231 Acked Read 6 Sent Done Done false
29664 9232 Acked Flush 0 Sent Done Done false
29665 9233 Acked Read 6 Sent Done Done false
29666 9234 Acked Read 8 Sent Done Done false
29667 9235 Acked Flush 0 Sent Done Done false
29668 9236 Acked Read 8 Sent Done Done false
29669 9237 Acked Write 4 Sent Done Done false
29670 9238 Acked Read 5 Sent Done Done false
29671 9239 Acked Write 1 Sent Done Done false
29672 9240 Acked Flush 0 Sent Done Done false
29673 9241 Acked Read 4 Sent Done Done false
29674 9242 Acked Read 3 Sent Done Done false
29675 9243 Acked Read 7 Sent Done Done false
29676 9244 Acked Write 1 Sent Done Done false
29677 9245 Acked Read 8 Sent Done Done false
29678 9246 Acked Read 2 Sent Done Done false
29679 9247 Acked Write 1 Sent Done Done false
29680 9248 Acked Write 8 Sent Done Done false
29681 9249 Acked Write 3 Sent Done Done false
29682 9250 Acked Read 5 Sent Done Done false
29683 9251 Acked Write 5 Sent Done Done false
29684 9252 Acked Write 5 Sent Done Done false
29685 9253 Acked Read 5 Sent Done Done false
29686 9254 Acked Write 5 Sent Done Done false
29687 9255 Acked Write 4 Sent Done Done false
29688 9256 Acked Write 4 Sent Done Done false
29689 9257 Acked Write 3 Sent Done Done false
29690 9258 Acked Read 3 Sent Done Done false
29691 9259 Acked Read 9 Sent Done Done false
29692 9260 Acked Write 9 Sent Done Done false
29693 9261 Acked Read 1 Sent Done Done false
29694 9262 Acked Write 1 Sent Done Done false
29695 9263 Acked Read 2 Sent Done Done false
29696 9264 Acked Write 3 Sent Done Done false
29697 9265 Acked Read 10 Sent Done Done false
29698 9266 Acked Read 8 Sent Done Done false
29699 9267 Acked Read 8 Sent Done Done false
29700 9268 Acked Write 2 Sent Done Done false
29701 9269 Acked Read 8 Sent Done Done false
29702 9270 Acked Read 2 Sent Done Done false
29703 9271 Acked Write 8 Sent Done Done false
29704 9272 Acked Read 9 Sent Done Done false
29705 9273 Acked Write 7 Sent Done Done false
29706 9274 Acked Write 5 Sent Done Done false
29707 9275 Acked Read 8 Sent Done Done false
29708 9276 Acked Write 10 Sent Done Done false
29709 9277 Acked Write 3 Sent Done Done false
29710 9278 Acked Write 10 Sent Done Done false
29711 9279 Acked Write 10 Sent Done Done false
29712 9280 Acked Write 10 Sent Done Done false
29713 9281 Acked Write 10 Sent Done Done false
29714 9282 Acked Write 3 Sent Done Done false
29715 9283 Acked Read 10 Sent Done Done false
29716 9284 Acked Write 6 Sent Done Done false
29717 9285 Acked Write 6 Sent Done Done false
29718 9286 Acked Flush 0 Sent Done Done false
29719 9287 Acked Flush 0 Sent Done Done false
29720 9288 Acked Write 5 Sent Done Done false
29721 9289 Acked Write 8 Sent Done Done false
29722 9290 Acked Read 2 Sent Done Done false
29723 9291 Acked Read 9 Sent Done Done false
29724 9292 Acked Flush 0 Sent Done Done false
29725 9293 Acked Write 5 Sent Done Done false
29726 9294 Acked Write 2 Sent Done Done false
29727 9295 Acked Flush 0 Sent Done Done false
29728 9296 Acked Write 9 Sent Done Done false
29729 9297 Acked Write 2 Sent Done Done false
29730 9298 Acked Read 8 Sent Done Done false
29731 9299 Acked Write 4 Sent Done Done false
29732 9300 Acked Read 6 Sent Done Done false
29733 9301 Acked Write 5 Sent Done Done false
29734 9302 Acked Read 2 Sent Done Done false
29735 9303 Acked Write 5 Sent Done Done false
29736 9304 Acked Flush 0 Sent Done Done false
29737 9305 Acked Write 8 Sent Done Done false
29738 9306 Acked Write 8 Sent Done Done false
29739 9307 Acked Write 7 Sent Done Done false
29740 9308 Acked Read 9 Sent Done Done false
29741 9309 Acked Read 5 Sent Done Done false
29742 9310 Acked Read 10 Sent Done Done false
29743 9311 Acked Read 4 Sent Done Done false
29744 9312 Acked Write 7 Sent Done Done false
29745 9313 Acked Write 6 Sent Done Done false
29746 9314 Acked Flush 0 Sent Done Done false
29747 9315 Acked Flush 0 Sent Done Done false
29748 9316 Acked Write 8 Sent Done Done false
29749 9317 Acked Write 7 Sent Done Done false
29750 9318 Acked Write 2 Sent Done Done false
29751 9319 Acked Read 1 Sent Done Done false
29752 9320 Acked Read 9 Sent Done Done false
29753 9321 Acked Write 5 Sent Done Done false
29754 9322 Acked Read 9 Sent Done Done false
29755 9323 Acked Read 6 Sent Done Done false
29756 9324 Acked Read 1 Sent Done Done false
29757 9325 Acked Read 1 Sent Done Done false
29758 9326 Acked Read 7 Sent Done Done false
29759 9327 Acked Write 6 Sent Done Done false
29760 9328 Acked Read 10 Sent Done Done false
29761 9329 Acked Flush 0 Sent Done Done false
29762 9330 Acked Read 9 Sent Done Done false
29763 9331 Acked Read 8 Sent Done Done false
29764 9332 Acked Flush 0 Sent Done Done false
29765 9333 Acked Write 10 Sent Done Done false
29766 9334 Acked Write 9 Sent Done Done false
29767 9335 Acked Read 3 Sent Done Done false
29768 9336 Acked Write 10 Sent Done Done false
29769 9337 Acked Read 5 Sent Done Done false
29770 9338 Acked Read 1 Sent Done Done false
29771 9339 Acked Read 1 Sent Done Done false
29772 9340 Acked Read 8 Sent Done Done false
29773 9341 Acked Write 3 Sent Done Done false
29774 9342 Acked Read 8 Sent Done Done false
29775 9343 Acked Write 3 Sent Done Done false
29776 9344 Acked Write 4 Sent Done Done false
29777 9345 Acked Flush 0 Sent Done Done false
29778 9346 Acked Read 6 Sent Done Done false
29779 9347 Acked Read 6 Sent Done Done false
29780 9348 Acked Flush 0 Sent Done Done false
29781 9349 Acked Read 5 Sent Done Done false
29782 9350 Acked Read 4 Sent Done Done false
29783 9351 Acked Read 10 Sent Done Done false
29784 9352 Acked Read 5 Sent Done Done false
29785 9353 Acked Read 4 Sent Done Done false
29786 9354 Acked Read 2 Sent Done Done false
29787 9355 Acked Write 5 Sent Done Done false
29788 9356 Acked Write 2 Sent Done Done false
29789 9357 Acked Write 8 Sent Done Done false
29790 9358 Acked Read 10 Sent Done Done false
29791 9359 Acked Write 4 Sent Done Done false
29792 9360 Acked Read 4 Sent Done Done false
29793 9361 Acked Read 2 Sent Done Done false
29794 9362 Acked Write 7 Sent Done Done false
29795 9363 Acked Read 6 Sent Done Done false
29796 9364 Acked Write 4 Sent Done Done false
29797 9365 Acked Read 10 Sent Done Done false
29798 9366 Acked Write 8 Sent Done Done false
29799 9367 Acked Read 4 Sent Done Done false
29800 9368 Acked Read 9 Sent Done Done false
29801 9369 Acked Write 3 Sent Done Done false
29802 9370 Acked Read 10 Sent Done Done false
29803 9371 Acked Read 3 Sent Done Done false
29804 9372 Acked Write 1 Sent Done Done false
29805 9373 Acked Read 9 Sent Done Done false
29806 9374 Acked Write 9 Sent Done Done false
29807 9375 Acked Read 5 Sent Done Done false
29808 9376 Acked Read 7 Sent Done Done false
29809 9377 Acked Write 10 Sent Done Done false
29810 9378 Acked Read 1 Sent Done Done false
29811 9379 Acked Write 9 Sent Done Done false
29812 9380 Acked Read 6 Sent Done Done false
29813 9381 Acked Write 2 Sent Done Done false
29814 9382 Acked Read 5 Sent Done Done false
29815 9383 Acked Read 6 Sent Done Done false
29816 9384 Acked Read 4 Sent Done Done false
29817 9385 Acked Write 1 Sent Done Done false
29818 9386 Acked Read 10 Sent Done Done false
29819 9387 Acked Read 9 Sent Done Done false
29820 9388 Acked Write 5 Sent Done Done false
29821 9389 Acked Write 7 Sent Done Done false
29822 9390 Acked Write 2 Sent Done Done false
29823 9391 Acked Flush 0 Sent Done Done false
29824 9392 Acked Read 10 Sent Done Done false
29825 9393 Acked Flush 0 Sent Done Done false
29826 9394 Acked Flush 0 Sent Done Done false
29827 9395 Acked Flush 0 Sent Done Done false
29828 9396 Acked Read 8 Sent Done Done false
29829 9397 Acked Read 1 Sent Done Done false
29830 9398 Acked Read 3 Sent Done Done false
29831 9399 Acked Read 2 Sent Done Done false
29832 9400 Acked Write 6 Sent Done Done false
29833 9401 Acked Read 4 Sent Done Done false
29834 9402 Acked Read 8 Sent Done Done false
29835 9403 Acked Write 9 Sent Done Done false
29836 9404 Acked Read 8 Sent Done Done false
29837 9405 Acked Read 2 Sent Done Done false
29838 9406 Acked Write 3 Sent Done Done false
29839 9407 Acked Read 7 Sent Done Done false
29840 9408 Acked Read 5 Sent Done Done false
29841 9409 Acked Write 4 Sent Done Done false
29842 9410 Acked Write 7 Sent Done Done false
29843 9411 Acked Flush 0 Sent Done Done false
29844 9412 Acked Write 2 Sent Done Done false
29845 9413 Acked Read 9 Sent Done Done false
29846 9414 Acked Write 5 Sent Done Done false
29847 9415 Acked Read 7 Sent Done Done false
29848 9416 Acked Write 10 Sent Done Done false
29849 9417 Acked Read 2 Sent Done Done false
29850 9418 Acked Read 1 Sent Done Done false
29851 9419 Acked Read 4 Sent Done Done false
29852 9420 Acked Read 6 Sent Done Done false
29853 9421 Acked Write 9 Sent Done Done false
29854 9422 Acked Write 5 Sent Done Done false
29855 9423 Acked Write 8 Sent Done Done false
29856 9424 Acked Read 4 Sent Done Done false
29857 9425 Acked Write 6 Sent Done Done false
29858 9426 Acked Write 1 Sent Done Done false
29859 9427 Acked Write 2 Sent Done Done false
29860 9428 Acked Write 1 Sent Done Done false
29861 9429 Acked Flush 0 Sent Done Done false
29862 9430 Acked Flush 0 Sent Done Done false
29863 9431 Acked Read 5 Sent Done Done false
29864 9432 Acked Flush 0 Sent Done Done false
29865 9433 Acked Read 7 Sent Done Done false
29866 9434 Acked Write 3 Sent Done Done false
29867 9435 Acked Read 2 Sent Done Done false
29868 9436 Acked Write 1 Sent Done Done false
29869 9437 Acked Write 4 Sent Done Done false
29870 9438 Acked Read 4 Sent Done Done false
29871 9439 Acked Read 9 Sent Done Done false
29872 9440 Acked Read 2 Sent Done Done false
29873 9441 Acked Read 8 Sent Done Done false
29874 9442 Acked Read 8 Sent Done Done false
29875 9443 Acked Read 4 Sent Done Done false
29876 9444 Acked Write 3 Sent Done Done false
29877 9445 Acked Read 6 Sent Done Done false
29878 9446 Acked Read 8 Sent Done Done false
29879 9447 Acked Read 5 Sent Done Done false
29880 9448 Acked Write 9 Sent Done Done false
29881 9449 Acked Flush 0 Sent Done Done false
29882 9450 Acked Read 2 Sent Done Done false
29883 9451 Acked Write 7 Sent Done Done false
29884 9452 Acked Write 3 Sent Done Done false
29885 9453 Acked Read 1 Sent Done Done false
29886 9454 Acked Flush 0 Sent Done Done false
29887 STATES DS:0 DS:1 DS:2 TOTAL
29888 Sent 302 0 0 302
29889 Done 0 302 302 604
29890 Skipped 0 0 0 0
29891 Error 0 0 0 0
29892 Last Flush: Some(JobId(9152)) Some(JobId(9454)) Some(JobId(9454))
29893 Downstairs last five completed: 9152 9151 9150 9149 9148
29894 Upstairs last five completed: 9454 9453 9452 9451 9450
29895 CLIENT: Up:0 ds:302 act:2
298962025-01-23T17:44:00.432ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
298972025-01-23T17:44:00.432ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42550) = io task client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
298982025-01-23T17:44:00.433ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
298992025-01-23T17:44:00.434ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
299002025-01-23T17:44:00.434ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
299012025-01-23T17:44:00.434ZINFOcrucible: send last flush ID to this DS: Some(JobId(9152)) = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
299022025-01-23T17:44:00.434ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
299032025-01-23T17:44:00.434ZINFOcrucible: [0] client re-new 302 jobs since flush Some(JobId(9152)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
299042025-01-23T17:44:00.434ZINFOcrucible: [0] Replayed 302 jobs since flush: Some(JobId(9152)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
29905 ----------------------------------------------------------------
29906 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
29907 Downstairs last five completed: 9454 9453 9452 9451 9450
29908 Upstairs last five completed: 9454 9453 9452 9451 9450
29909 CLIENT: Up:0 ds:0 act:3
29910 Replay: All jobs finished
29911 Main task has work to do, go find it
29912 got dsc Stop(1)
29913 stop 1
29914 Jan 23 17:44:02.677[8820] Got stop action so:false kr:false
29915 INFO request completed, latency_us: 369, response_code: 204, uri: /stop/cid/1, method: POST, req_id: b2d3adfa-54e6-4f44-a99d-c3d9ad2ee68d, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
29916 Jan 23 17:44:02.678 INFO request completed, latency_us: 121, response_code: 200, uri: /state/cid/1, method: GET, req_id: a9cc013e-a1b0-4708-a832-cd366674a796, remote_addr: 127.0.0.1:42015, local_addr{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:44:02.677995663Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
29917 : 127.0.0.1:9998
299182025-01-23T17:44:02.678ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
299192025-01-23T17:44:02.678ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
299202025-01-23T17:44:02.678ZINFOcrucible: client task is exiting = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
29921 [8820] Exited with: ExitStatus(unix_wait_status(9))
29922 [8820][1] reports Exit
29923 Jan 23 17:44:06.680 INFO request completed, latency_us: 225, response_code: 200, uri: /state/cid/1, method: GET, req_id: fb445382-6a59-4365-93f8-602ba4423d86, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
29924 001/300 Write block 2348 len 16384 data: 6 6 7 9
29925 002/300 Write block 1914 len 4096 data: 6
29926 003/300 Flush
29927 004/300 Read block 1886 len 24576
29928 005/300 Flush
29929 006/300 Flush
29930 007/300 Write block 3359 len 8192 data: 4 3
29931 008/300 Write block 1338 len 12288 data: 4 3 3
29932 009/300 Write block 1051 len 40960 data: 7 8 7 6 7 4 3 4 5 5
29933 010/300 Write block 1311 len 36864 data: 11 9 9 8 8 8 7 6 6
29934 011/300 Read block 3449 len 24576
29935 012/300 Flush
29936 013/300 Flush
29937 014/300 Flush
29938 015/300 Write block 1368 len 4096 data: 5
29939 016/300 Write block 1697 len 16384 data: 7 8 9 9
29940 017/300 Write block 1243 len 32768 data: 3 3 3 3 4 4 4 5
29941 018/300 Read block 2496 len 36864
29942 019/300 Read block 4604 len 28672
29943 020/300 Write block 3640 len 32768 data: 6 5 6 6 4 4 5 6
29944 021/300 Read block 2635 len 36864
29945 022/300 Read block 2652 len 16384
29946 023/300 Write block 2276 len 32768 data: 6 6 6 6 8 9 8 8
29947 024/300 Read block 639 len 12288
29948 025/300 Flush
29949 026/300 Write block 4685 len 36864 data: 8 7 6 5 5 4 5 6 5
29950 027/300 Write block 4929 len 28672 data: 7 5 7 7 7 6 6
29951 028/300 Flush
29952 029/300 Write block 1664 len 4096 data: 6
29953 030/300 Read block 4104 len 36864
29954 031/300 Read block 438 len 28672
29955 032/300 Read block 2596 len 28672
29956 033/300 Read block 2124 len 20480
29957 034/300 Read block 4195 len 40960
29958 035/300 Write block 696 len 16384 data: 5 6 6 6
29959 036/300 Read block 3400 len 16384
29960 037/300 Write block 3824 len 36864 data: 4 7 6 7 7 7 6 4 4
29961 038/300 Read block 3297 len 40960
29962 039/300 Read block 2286 len 36864
29963 040/300 Write block 2484 len 20480 data: 8 7 5 5 5
29964 041/300 Read block 4447 len 28672
29965 042/300 Read block 3775 len 4096
29966 043/300 Write block 2315 len 12288 data: 5 4 6
29967 044/300 Read block 2978 len 32768
29968 045/300 Write block 2660 len 20480 data: 4 5 5 4 4
29969 046/300 Read block 3196 len 12288
29970 047/300 Read block 972 len 40960
29971 048/300 Read block 515 len 12288
29972 049/300 Read block 515 len 32768
29973 050/300 Read block 189 len 24576
29974 051/300 Write block 1089 len 16384 data: 8 7 6 5
29975 052/300 Flush
29976 053/300 Read block 3706 len 40960
29977 054/300 Write block 3424 len 24576 data: 7 8 8 7 7 10
29978 055/300 Read block 3003 len 28672
29979 056/300 Write block 2513 len 36864 data: 5 6 6 6 6 6 5 6 4
29980 057/300 Write block 3145 len 24576 data: 6 5 5 5 5 5
29981 058/300 Read block 377 len 8192
29982 059/300 Read block 3433 len 40960
29983 060/300 Write block 1967 len 8192 data: 9 8
29984 061/300 Read block 1767 len 12288
29985 062/300 Read block 3652 len 20480
29986 063/300 Write block 3294 len 8192 data: 8 7
29987 064/300 Read block 511 len 16384
29988 065/300 Write block 1805 len 32768 data: 7 4 3 3 3 3 4 4
29989 066/300 Flush
29990 067/300 Read block 4105 len 40960
29991 068/300 Write block 2111 len 36864 data: 4 7 6 7 7 7 7 6 4
29992 069/300 Read block 4246 len 40960
29993 070/300 Write block 4532 len 40960 data: 5 6 6 6 6 7 7 7 6 6
29994 071/300 Read block 4314 len 4096
29995 072/300 Read block 2869 len 4096
29996 073/300 Read block 1087 len 32768
29997 074/300 Read block 4065 len 40960
29998 075/300 Read block 4584 len 12288
29999 076/300 Write block 4772 len 20480 data: 4 6 6 5 5
30000 077/300 Write block 3194 len 28672 data: 5 3 4 7 7 7 7
30001 078/300 Write block 451 len 20480 data: 3 3 4 4 5
30002 079/300 Read block 1565 len 36864
30003 080/300 Read block 2897 len 40960
30004 081/300 Read block 849 len 28672
30005 082/300 Read block 728 len 28672
30006 083/300 Read block 2737 len 24576
30007 084/300 Read block 3938 len 12288
30008 085/300 Read block 1588 len 20480
30009 086/300 Read block 4444 len 4096
30010 087/300 Read block 2360 len 16384
30011 088/300 Read block 4441 len 4096
30012 089/300 Flush
30013 090/300 Write block 4028 len 8192 data: 4 6
30014 091/300 Write block 3610 len 28672 data: 5 7 6 6 5 6 5
30015 092/300 Read block 1108 len 20480
30016 093/300 Write block 3224 len 8192 data: 7 9
30017 094/300 Write block 147 len 40960 data: 6 5 4 2 3 4 6 6 6 6
30018 095/300 Flush
30019 096/300 Write block 727 len 28672 data: 4 4 3 4 5 6 7
30020 097/300 Read block 2832 len 36864
30021 098/300 Flush
30022 099/300 Write block 1816 len 24576 data: 6 4 4 4 4 5
30023 100/300 Write block 4772 len 24576 data: 5 7 7 6 6 7
30024 101/300 Read block 2041 len 32768
30025 102/300 Write block 4429 len 20480 data: 3 3 4 3 3
30026 103/300 Read block 2843 len 24576
30027 104/300 Flush
30028 105/300 Write block 4443 len 40960 data: 7 7 7 7 6 6 7 7 7 6
30029 106/300 Flush
30030 107/300 Read block 2268 len 20480
30031 108/300 Read block 3013 len 12288
30032 109/300 Read block 3458 len 16384
30033 110/300 Write block 3008 len 4096 data: 4
30034 111/300 Read block 988 len 20480
30035 112/300 Flush
30036 113/300 Write block 1913 len 40960 data: 6 7 7 6 5 7 8 11 10 10
30037 114/300 Write block 4015 len 20480 data: 4 4 3 3 2
30038 115/300 Write block 69 len 12288 data: 10 11 10
30039 116/300 Read block 133 len 4096
30040 117/300 Read block 1405 len 40960
30041 118/300 Flush
30042 119/300 Read block 2761 len 4096
30043 120/300 Write block 1311 len 8192 data: 12 10
30044 121/300 Write block 3701 len 16384 data: 7 7 8 9
30045 122/300 Read block 4955 len 24576
30046 123/300 Read block 143 len 40960
30047 124/300 Write block 1488 len 4096 data: 6
30048 125/300 Write block 3744 len 36864 data: 4 6 8 7 6 5 4 5 5
30049 126/300 Write block 1409 len 12288 data: 7 6 7
30050 127/300 Read block 3950 len 36864
30051 128/300 Write block 2911 len 32768 data: 4 4 4 3 3 2 3 4
30052 129/300 Read block 2905 len 32768
30053 130/300 Write block 4729 len 12288 data: 5 7 7
30054 131/300 Read block 3189 len 20480
30055 132/300 Read block 198 len 12288
30056 133/300 Read block 3793 len 24576
30057 134/300 Read block 3214 len 36864
30058 135/300 Read block 3993 len 24576
30059 136/300 Read block 2084 len 32768
30060 137/300 Write block 1451 len 36864 data: 6 6 5 6 6 7 6 5 4
30061 138/300 Write block 2278 len 8192 data: 7 7
30062 139/300 Read block 2464 len 8192
30063 140/300 Read block 1524 len 36864
30064 141/300 Flush
30065 142/300 Flush
30066 143/300 Read block 4575 len 24576
30067 144/300 Flush
30068 145/300 Read block 4416 len 36864
30069 146/300 Write block 3938 len 32768 data: 6 5 7 7 7 8 8 7
30070 147/300 Read block 4513 len 16384
30071 148/300 Write block 3160 len 4096 data: 9
30072 149/300 Read block 4386 len 28672
30073 150/300 Flush
30074 151/300 Read block 2856 len 28672
30075 152/300 Read block 1307 len 40960
30076 153/300 Read block 2469 len 40960
30077 154/300 Read block 2311 len 24576
30078 155/300 Read block 2382 len 32768
30079 156/300 Read block 1581 len 24576
30080 157/300 Flush
30081 158/300 Write block 3073 len 32768 data: 5 5 5 3 3 3 4 3
30082 159/300 Write block 484 len 40960 data: 6 6 6 5 4 4 3 4 5 6
30083 160/300 Read block 682 len 40960
30084 161/300 Write block 3188 len 28672 data: 4 5 4 4 4 5 6
30085 162/300 Write block 1848 len 40960 data: 6 7 6 6 7 5 6 7 8 9
30086 163/300 Write block 748 len 24576 data: 9 9 9 7 6 8
30087 164/300 Read block 2232 len 20480
30088 165/300 Read block 2455 len 28672
30089 166/300 Read block 2623 len 36864
30090 167/300 Read block 1578 len 40960
30091 168/300 Read block 1870 len 36864
30092 169/300 Write block 2969 len 16384 data: 4 4 6 7
30093 170/300 Read block 382 len 8192
30094 171/300 Flush
30095 172/300 Flush
30096 173/300 Read block 3182 len 32768
30097 174/300 Read block 717 len 24576
30098 175/300 Read block 4952 len 8192
30099 176/300 Read block 3740 len 40960
30100 177/300 Read block 2855 len 16384
30101 178/300 Read block 2078 len 32768
30102 179/300 Write block 3353 len 16384 data: 6 5 5 5
30103 180/300 Flush
30104 181/300 Write block 1995 len 16384 data: 4 5 5 5
30105 182/300 Read block 1230 len 40960
30106 183/300 Read block 2785 len 36864
30107 184/300 Read block 91 len 28672
30108 185/300 Flush
30109 186/300 Write block 1893 len 8192 data: 3 4
30110 187/300 Write block 1457 len 24576 data: 7 6 5 4 4 4
30111 188/300 Read block 1099 len 4096
30112 189/300 Write block 4391 len 24576 data: 5 5 6 7 6 6
30113 190/300 Write block 1509 len 16384 data: 7 8 6 6
30114 191/300 Write block 2006 len 12288 data: 4 3 3
30115 192/300 Read block 883 len 12288
30116 193/300 Read block 953 len 40960
30117 194/300 Read block 3899 len 20480
30118 195/300 Write block 4378 len 24576 data: 5 5 5 5 4 3
30119 196/300 Read block 3437 len 20480
30120 197/300 Read block 152 len 16384
30121 198/300 Flush
30122 199/300 Read block 2655 len 36864
30123 200/300 Write block 1518 len 24576 data: 6 3 4 4 4 4
30124 201/300 Read block 2897 len 28672
30125 202/300 Read block 4665 len 24576
30126 203/300 Read block 518 len 32768
30127 204/300 Read block 1432 len 24576
30128 205/300 Write block 4564 len 16384 data: 9 8 9 9
30129 206/300 Read block 2786 len 32768
30130 207/300 Write block 2804 len 20480 data: 7 10 9 7 7
30131 208/300 Read block 1464 len 28672
30132 209/300 Read block 1985 len 12288
30133 210/300 Read block 1203 len 36864
30134 211/300 Read block 488 len 4096
30135 212/300 Flush
30136 213/300 Read block 3554 len 40960
30137 214/300 Flush
30138 215/300 Write block 4240 len 28672 data: 7 6 6 6 4 4 4
30139 216/300 Flush
30140 217/300 Flush
30141 218/300 Read block 966 len 24576
30142 219/300 Flush
30143 220/300 Flush
30144 221/300 Write block 4461 len 16384 data: 7 4 5 6
30145 222/300 Read block 1910 len 32768
30146 223/300 Write block 1036 len 28672 data: 5 3 3 3 3 2 3
30147 224/300 Read block 4899 len 32768
30148 225/300 Write block 1505 len 32768 data: 8 9 8 6 8 9 7 7
30149 226/300 Write block 4693 len 36864 data: 6 4 4 5 5 5 6 8 7
30150 227/300 Flush
30151 228/300 Write block 3764 len 36864 data: 6 6 7 6 6 5 5 5 6
30152 229/300 Read block 1504 len 8192
30153 230/300 Write block 4046 len 36864 data: 5 4 6 7 5 5 5 5 6
30154 231/300 Read block 2960 len 4096
30155 232/300 Write block 2001 len 4096 data: 7
30156 233/300 Read block 2308 len 28672
30157 234/300 Read block 4152 len 16384
30158 235/300 Write block 3004 len 32768 data: 3 3 3 3 5 5 5 6
30159 236/300 Read block 4756 len 4096
30160 237/300 Write block 4409 len 36864 data: 7 7 6 8 8 7 6 5 5
30161 238/300 Read block 4615 len 16384
30162 239/300 Read block 264 len 40960
30163 240/300 Write block 306 len 12288 data: 2 2 2
30164 241/300 Read block 1787 len 8192
30165 242/300 Write block 4720 len 20480 data: 4 5 4 4 3
30166 243/300 Write block 3686 len 8192 data: 6 6
30167 244/300 Write block 2100 len 4096 data: 5
30168 245/300 Read block 428 len 40960
30169 246/300 Write block 1610 len 4096 data: 8
30170 247/300 Read block 436 len 4096
30171 248/300 Write block 1228 len 8192 data: 2 2
30172 249/300 Write block 4725 len 40960 data: 4 6 6 6 6 8 8 6 5 5
30173 250/300 Write block 2455 len 16384 data: 6 5 7 6
30174 251/300 Flush
30175 252/300 Read block 2354 len 24576
30176 253/300 Write block 685 len 8192 data: 4 4
30177 254/300 Read block 4242 len 40960
30178 255/300 Flush
30179 256/300 Read block 1466 len 8192
30180 257/300 Write block 4127 len 20480 data: 3 2 3 3 4
30181 258/300 Read block 4914 len 4096
30182 259/300 Write block 4391 len 36864 data: 6 6 7 8 7 7 6 8 9
30183 260/300 Write block 3039 len 16384 data: 7 8 8 7
30184 261/300 Write block 3321 len 40960 data: 4 4 4 4 4 4 5 5 5 7
30185 262/300 Read block 4906 len 32768
30186 263/300 Flush
30187 264/300 Read block 4538 len 20480
30188 265/300 Read block 4924 len 4096
30189 266/300 Read block 249 len 16384
30190 267/300 Read block 4030 len 28672
30191 268/300 Read block 2848 len 8192
30192 269/300 Write block 1126 len 36864 data: 2 2 2 4 4 4 3 3 3
30193 270/300 Write block 826 len 32768 data: 10 10 9 9 6 5 7 8
30194 271/300 Read block 3419 len 12288
30195 272/300 Read block 2591 len 40960
30196 273/300 Read block 311 len 32768
30197 274/300 Write block 1910 len 36864 data: 8 8 7 7 8 8 7 6 8
30198 275/300 Read block 2482 len 36864
30199 276/300 Write block 4954 len 36864 data: 5 4 3 3 4 4 2 3 3
30200 277/300 Write block 2850 len 20480 data: 5 6 6 6 5
30201 278/300 Read block 712 len 28672
30202 279/300 Read block 498 len 20480
30203 280/300 Write block 859 len 40960 data: 4 3 4 3 3 3 4 4 4 5
30204 281/300 Read block 2313 len 40960
30205 282/300 Write block 503 len 32768 data: 2 3 3 4 4 4 4 4
30206 283/300 Read block 3536 len 28672
30207 284/300 Write block 641 len 20480 data: 11 11 11 10 11
30208 285/300 Read block 60 len 24576
30209 286/300 Write block 2603 len 4096 data: 5
30210 287/300 Read block 4072 len 32768
30211 288/300 Read block 2290 len 24576
30212 289/300 Read block 2852 len 36864
30213 290/300 Write block 1591 len 40960 data: 3 3 4 5 5 4 4 6 5 4
30214 291/300 Read block 472 len 28672
30215 292/300 Write block 1478 len 36864 data: 7 6 5 4 3 3 3 3 3
30216 293/300 Read block 2445 len 32768
30217 294/300 Write block 2830 len 40960 data: 5 5 7 7 6 6 6 8 7 8
30218 295/300 Write block 1097 len 32768 data: 5 5 5 5 5 5 5 2
30219 296/300 Flush
30220 297/300 Write block 3162 len 24576 data: 9 10 9 7 7 6
30221 298/300 Read block 2855 len 12288
30222 299/300 Read block 3852 len 8192
30223 300/300 Write block 1670 len 36864 data: 3 3 2 2 3 5 6 6 6
30224 301/300 Read block 1918 len 12288
30225 Main task has work to do, go find it
30226 got dsc Start(1)
30227 start 1
30228 Jan 23 17:44:06.943 INFO request completed, latency_us: 237, response_code: 204, uri: /start/cid/1, method: POST, req_id: 92193468-d590-4641-a85b-ad6c77298329, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
30229 Replay: started 1, returned:Ok(())
30230 ----------------------------------------------------------------
30231 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30232 States: Active Offline Active
30233 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30234 9455 Acked Write 4 Done Sent Done false
30235 9456 Acked Write 1 Done Sent Done false
30236 9457 Acked Flush 0 Done Sent Done false
30237 9458 Acked Read 6 Done Sent Done false
30238 9459 Acked Flush 0 Done Sent Done false
30239 9460 Acked Flush 0 Done Sent Done false
30240 9461 Acked Write 2 Done Sent Done false
30241 9462 Acked Write 3 Done Sent Done false
30242 9463 Acked Write 10 Done Sent Done false
30243 9464 Acked Write 9 Done Sent Done false
30244 9465 Acked Read 6 Done Sent Done false
30245 9466 Acked Flush 0 Done Sent Done false
30246 9467 Acked Flush 0 Done Sent Done false
30247 9468 Acked Flush 0 Done Sent Done false
30248 9469 Acked Write 1 Done Sent Done false
30249 9470 Acked Write 4 Done Sent Done false
30250 9471 Acked Write 8 Done Sent Done false
30251 9472 Acked Read 9 Done Sent Done false
30252 9473 Acked Read 7 Done Sent Done false
30253 9474 Acked Write 8 Done Sent Done false
30254 9475 Acked Read 9 Done Sent Done false
30255 9476 Acked Read 4 Done Sent Done false
30256 9477 Acked Write 8 Done Sent Done false
30257 9478 Acked Read 3 Done Sent Done false
30258 9479 Acked Flush 0 Done Sent Done false
30259 9480 Acked Write 9 Done Sent Done false
30260 9481 Acked Write 7 Done Sent Done false
30261 9482 Acked Flush 0 Done Sent Done false
30262 9483 Acked Write 1 Done Sent Done false
30263 9484 Acked Read 9 Done Sent Done false
30264 9485 Acked Read 7 Done Sent Done false
30265 9486 Acked Read 7 Done Sent Done false
30266 9487 Acked Read 5 Done Sent Done false
30267 9488 Acked Read 10 Done Sent Done false
30268 9489 Acked Write 4 Done Sent Done false
30269 9490 Acked Read 4 Done Sent Done false
30270 9491 Acked Write 9 Done Sent Done false
30271 9492 Acked Read 10 Done Sent Done false
30272 9493 Acked Read 9 Done Sent Done false
30273 9494 Acked Write 5 Done Sent Done false
30274 9495 Acked Read 7 Done Sent Done false
30275 9496 Acked Read 1 Done Sent Done false
30276 9497 Acked Write 3 Done Sent Done false
30277 9498 Acked Read 8 Done Sent Done false
30278 9499 Acked Write 5 Done Sent Done false
30279 9500 Acked Read 3 Done Sent Done false
30280 9501 Acked Read 10 Done Sent Done false
30281 9502 Acked Read 3 Done Sent Done false
30282 9503 Acked Read 8 Done Sent Done false
30283 9504 Acked Read 6 Done Sent Done false
30284 9505 Acked Write 4 Done Sent Done false
30285 9506 Acked Flush 0 Done Sent Done false
30286 9507 Acked Read 10 Done Sent Done false
30287 9508 Acked Write 6 Done Sent Done false
30288 9509 Acked Read 7 Done Sent Done false
30289 9510 Acked Write 9 Done Sent Done false
30290 9511 Acked Write 6 Done Sent Done false
30291 9512 Acked Read 2 Done Sent Done false
30292 9513 Acked Read 10 Done Sent Done false
30293 9514 Acked Write 2 Done Sent Done false
30294 9515 Acked Read 3 Done Sent Done false
30295 9516 Acked Read 5 Done Sent Done false
30296 9517 Acked Write 2 Done Sent Done false
30297 9518 Acked Read 4 Done Sent Done false
30298 9519 Acked Write 8 Done Sent Done false
30299 9520 Acked Flush 0 Done Sent Done false
30300 9521 Acked Read 10 Done Sent Done false
30301 9522 Acked Write 9 Done Sent Done false
30302 9523 Acked Read 10 Done Sent Done false
30303 9524 Acked Write 10 Done Sent Done false
30304 9525 Acked Read 1 Done Sent Done false
30305 9526 Acked Read 1 Done Sent Done false
30306 9527 Acked Read 8 Done Sent Done false
30307 9528 Acked Read 10 Done Sent Done false
30308 9529 Acked Read 3 Done Sent Done false
30309 9530 Acked Write 5 Done Sent Done false
30310 9531 Acked Write 7 Done Sent Done false
30311 9532 Acked Write 5 Done Sent Done false
30312 9533 Acked Read 9 Done Sent Done false
30313 9534 Acked Read 10 Done Sent Done false
30314 9535 Acked Read 7 Done Sent Done false
30315 9536 Acked Read 7 Done Sent Done false
30316 9537 Acked Read 6 Done Sent Done false
30317 9538 Acked Read 3 Done Sent Done false
30318 9539 Acked Read 5 Done Sent Done false
30319 9540 Acked Read 1 Done Sent Done false
30320 9541 Acked Read 4 Done Sent Done false
30321 9542 Acked Read 1 Done Sent Done false
30322 9543 Acked Flush 0 Done Sent Done false
30323 9544 Acked Write 2 Done Sent Done false
30324 9545 Acked Write 7 Done Sent Done false
30325 9546 Acked Read 5 Done Sent Done false
30326 9547 Acked Write 2 Done Sent Done false
30327 9548 Acked Write 10 Done Sent Done false
30328 9549 Acked Flush 0 Done Sent Done false
30329 9550 Acked Write 7 Done Sent Done false
30330 9551 Acked Read 9 Done Sent Done false
30331 9552 Acked Flush 0 Done Sent Done false
30332 9553 Acked Write 6 Done Sent Done false
30333 9554 Acked Write 6 Done Sent Done false
30334 9555 Acked Read 8 Done Sent Done false
30335 9556 Acked Write 5 Done Sent Done false
30336 9557 Acked Read 6 Done Sent Done false
30337 9558 Acked Flush 0 Done Sent Done false
30338 9559 Acked Write 10 Done Sent Done false
30339 9560 Acked Flush 0 Done Sent Done false
30340 9561 Acked Read 5 Done Sent Done false
30341 9562 Acked Read 3 Done Sent Done false
30342 9563 Acked Read 4 Done Sent Done false
30343 9564 Acked Write 1 Done Sent Done false
30344 9565 Acked Read 5 Done Sent Done false
30345 9566 Acked Flush 0 Done Sent Done false
30346 9567 Acked Write 10 Done Sent Done false
30347 9568 Acked Write 5 Done Sent Done false
30348 9569 Acked Write 3 Done Sent Done false
30349 9570 Acked Read 1 Done Sent Done false
30350 9571 Acked Read 10 Done Sent Done false
30351 9572 Acked Flush 0 Done Sent Done false
30352 9573 Acked Read 1 Done Sent Done false
30353 9574 Acked Write 2 Done Sent Done false
30354 9575 Acked Write 4 Done Sent Done false
30355 9576 Acked Read 6 Done Sent Done false
30356 9577 Acked Read 10 Done Sent Done false
30357 9578 Acked Write 1 Done Sent Done false
30358 9579 Acked Write 9 Done Sent Done false
30359 9580 Acked Write 3 Done Sent Done false
30360 9581 Acked Read 9 Done Sent Done false
30361 9582 Acked Write 8 Done Sent Done false
30362 9583 Acked Read 8 Done Sent Done false
30363 9584 Acked Write 3 Done Sent Done false
30364 9585 Acked Read 5 Done Sent Done false
30365 9586 Acked Read 3 Done Sent Done false
30366 9587 Acked Read 6 Done Sent Done false
30367 9588 Acked Read 9 Done Sent Done false
30368 9589 Acked Read 6 Done Sent Done false
30369 9590 Acked Read 8 Done Sent Done false
30370 9591 Acked Write 9 Done Sent Done false
30371 9592 Acked Write 2 Done Sent Done false
30372 9593 Acked Read 2 Done Sent Done false
30373 9594 Acked Read 9 Done Sent Done false
30374 9595 Acked Flush 0 Done Sent Done false
30375 9596 Acked Flush 0 Done Sent Done false
30376 9597 Acked Read 6 Done Sent Done false
30377 9598 Acked Flush 0 Done Sent Done false
30378 9599 Acked Read 9 Done Sent Done false
30379 9600 Acked Write 8 Done Sent Done false
30380 9601 Acked Read 4 Done Sent Done false
30381 9602 Acked Write 1 Done Sent Done false
30382 9603 Acked Read 7 Done Sent Done false
30383 9604 Acked Flush 0 Done Sent Done false
30384 9605 Acked Read 7 Done Sent Done false
30385 9606 Acked Read 10 Done Sent Done false
30386 9607 Acked Read 10 Done Sent Done false
30387 9608 Acked Read 6 Done Sent Done false
30388 9609 Acked Read 8 Done Sent Done false
30389 9610 Acked Read 6 Done Sent Done false
30390 9611 Acked Flush 0 Done Sent Done false
30391 9612 Acked Write 8 Done Sent Done false
30392 9613 Acked Write 10 Done Sent Done false
30393 9614 Acked Read 10 Done Sent Done false
30394 9615 Acked Write 7 Done Sent Done false
30395 9616 Acked Write 10 Done Sent Done false
30396 9617 Acked Write 6 Done Sent Done false
30397 9618 Acked Read 5 Done Sent Done false
30398 9619 Acked Read 7 Done Sent Done false
30399 9620 Acked Read 9 Done Sent Done false
30400 9621 Acked Read 10 Done Sent Done false
30401 9622 Acked Read 9 Done Sent Done false
30402 9623 Acked Write 4 Done Sent Done false
30403 9624 Acked Read 2 Done Sent Done false
30404 9625 Acked Flush 0 Done Sent Done false
30405 9626 Acked Flush 0 Done Sent Done false
30406 9627 Acked Read 8 Done Sent Done false
30407 9628 Acked Read 6 Done Sent Done false
30408 9629 Acked Read 2 Done Sent Done false
30409 9630 Acked Read 10 Done Sent Done false
30410 9631 Acked Read 4 Done Sent Done false
30411 9632 Acked Read 8 Done Sent Done false
30412 9633 Acked Write 4 Done Sent Done false
30413 9634 Acked Flush 0 Done Sent Done false
30414 9635 Acked Write 4 Done Sent Done false
30415 9636 Acked Read 10 Done Sent Done false
30416 9637 Acked Read 9 Done Sent Done false
30417 9638 Acked Read 7 Done Sent Done false
30418 9639 Acked Flush 0 Done Sent Done false
30419 9640 Acked Write 2 Done Sent Done false
30420 9641 Acked Write 6 Done Sent Done false
30421 9642 Acked Read 1 Done Sent Done false
30422 9643 Acked Write 6 Done Sent Done false
30423 9644 Acked Write 4 Done Sent Done false
30424 9645 Acked Write 3 Done Sent Done false
30425 9646 Acked Read 3 Done Sent Done false
30426 9647 Acked Read 10 Done Sent Done false
30427 9648 Acked Read 5 Done Sent Done false
30428 9649 Acked Write 6 Done Sent Done false
30429 9650 Acked Read 5 Done Sent Done false
30430 9651 Acked Read 4 Done Sent Done false
30431 9652 Acked Flush 0 Done Sent Done false
30432 9653 Acked Read 9 Done Sent Done false
30433 9654 Acked Write 6 Done Sent Done false
30434 9655 Acked Read 7 Done Sent Done false
30435 9656 Acked Read 6 Done Sent Done false
30436 9657 Acked Read 8 Done Sent Done false
30437 9658 Acked Read 6 Done Sent Done false
30438 9659 Acked Write 4 Done Sent Done false
30439 9660 Acked Read 8 Done Sent Done false
30440 9661 Acked Write 5 Done Sent Done false
30441 9662 Acked Read 7 Done Sent Done false
30442 9663 Acked Read 3 Done Sent Done false
30443 9664 Acked Read 9 Done Sent Done false
30444 9665 Acked Read 1 Done Sent Done false
30445 9666 Acked Flush 0 Done Sent Done false
30446 9667 Acked Read 10 Done Sent Done false
30447 9668 Acked Flush 0 Done Sent Done false
30448 9669 Acked Write 7 Done Sent Done false
30449 9670 Acked Flush 0 Done Sent Done false
30450 9671 Acked Flush 0 Done Sent Done false
30451 9672 Acked Read 6 Done Sent Done false
30452 9673 Acked Flush 0 Done Sent Done false
30453 9674 Acked Flush 0 Done Sent Done false
30454 9675 Acked Write 4 Done Sent Done false
30455 9676 Acked Read 8 Done Sent Done false
30456 9677 Acked Write 7 Done Sent Done false
30457 9678 Acked Read 8 Done Sent Done false
30458 9679 Acked Write 8 Done Sent Done false
30459 9680 Acked Write 9 Done Sent Done false
30460 9681 Acked Flush 0 Done Sent Done false
30461 9682 Acked Write 9 Done Sent Done false
30462 9683 Acked Read 2 Done Sent Done false
30463 9684 Acked Write 9 Done Sent Done false
30464 9685 Acked Read 1 Done Sent Done false
30465 9686 Acked Write 1 Done Sent Done false
30466 9687 Acked Read 7 Done Sent Done false
30467 9688 Acked Read 4 Done Sent Done false
30468 9689 Acked Write 8 Done Sent Done false
30469 9690 Acked Read 1 Done Sent Done false
30470 9691 Acked Write 9 Done Sent Done false
30471 9692 Acked Read 4 Done Sent Done false
30472 9693 Acked Read 10 Done Sent Done false
30473 9694 Acked Write 3 Done Sent Done false
30474 9695 Acked Read 2 Done Sent Done false
30475 9696 Acked Write 5 Done Sent Done false
30476 9697 Acked Write 2 Done Sent Done false
30477 9698 Acked Write 1 Done Sent Done false
30478 9699 Acked Read 10 Done Sent Done false
30479 9700 Acked Write 1 Done Sent Done false
30480 9701 Acked Read 1 Done Sent Done false
30481 9702 Acked Write 2 Done Sent Done false
30482 9703 Acked Write 10 Done Sent Done false
30483 9704 Acked Write 4 Done Sent Done false
30484 9705 Acked Flush 0 Done Sent Done false
30485 9706 Acked Read 6 Done Sent Done false
30486 9707 Acked Write 2 Done Sent Done false
30487 9708 Acked Read 10 Done Sent Done false
30488 9709 Acked Flush 0 Done Sent Done false
30489 9710 Acked Read 2 Done Sent Done false
30490 9711 Acked Write 5 Done Sent Done false
30491 9712 Acked Read 1 Done Sent Done false
30492 9713 Acked Write 9 Done Sent Done false
30493 9714 Acked Write 4 Done Sent Done false
30494 9715 Acked Write 10 Done Sent Done false
30495 9716 Acked Read 8 Done Sent Done false
30496 9717 Acked Flush 0 Done Sent Done false
30497 9718 Acked Read 5 Done Sent Done false
30498 9719 Acked Read 1 Done Sent Done false
30499 9720 Acked Read 4 Done Sent Done false
30500 9721 Acked Read 7 Done Sent Done false
30501 9722 Acked Read 2 Done Sent Done false
30502 9723 Acked Write 9 Done Sent Done false
30503 9724 Acked Write 8 Done Sent Done false
30504 9725 Acked Read 3 Done Sent Done false
30505 9726 Acked Read 10 Done Sent Done false
30506 9727 Acked Read 8 Done Sent Done false
30507 9728 Acked Write 9 Done Sent Done false
30508 9729 Acked Read 9 Done Sent Done false
30509 9730 Acked Write 9 Done Sent Done false
30510 9731 Acked Write 5 Done Sent Done false
30511 9732 Acked Read 7 Done Sent Done false
30512 9733 Acked Read 5 Done Sent Done false
30513 9734 Acked Write 10 Done Sent Done false
30514 9735 Acked Read 10 Done Sent Done false
30515 9736 Acked Write 8 Done Sent Done false
30516 9737 Acked Read 7 Done Sent Done false
30517 9738 Acked Write 5 Done Sent Done false
30518 9739 Acked Read 6 Done Sent Done false
30519 9740 Acked Write 1 Done Sent Done false
30520 9741 Acked Read 8 Done Sent Done false
30521 9742 Acked Read 6 Done Sent Done false
30522 9743 Acked Read 9 Done Sent Done false
30523 9744 Acked Write 10 Done Sent Done false
30524 9745 Acked Read 7 Done Sent Done false
30525 9746 Acked Write 9 Done Sent Done false
30526 9747 Acked Read 8 Done Sent Done false
30527 9748 Acked Write 10 Done Sent Done false
30528 9749 Acked Write 8 Done Sent Done false
30529 9750 Acked Flush 0 Done Sent Done false
30530 9751 Acked Write 6 Done Sent Done false
30531 9752 Acked Read 3 Done Sent Done false
30532 9753 Acked Read 2 Done Sent Done false
30533 9754 Acked Write 9 Done Sent Done false
30534 9755 Acked Read 3 Done Sent Done false
30535 STATES DS:0 DS:1 DS:2 TOTAL
30536 Sent 0 301 0 301
30537 Done 301 0 301 602
30538 Skipped 0 0 0 0
30539 Error 0 0 0 0
30540 Last Flush: Some(JobId(9750)) Some(JobId(9454)) Some(JobId(9750))
30541 Downstairs last five completed: 9454 9453 9452 9451 9450
30542 Upstairs last five completed: 9755 9754 9753 9752 9751
30543 CLIENT: Up:0 ds:301 act:2
30544 [8820] got start action
30545 [8820] I am going to restart
30546 Starting downstairs at port 8820
30547 Make output file at "/tmp/dsc/downstairs-8820.txt"
30548 [8820][1] reports Starting
30549 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1234)
30550 [8820][1] reports Running
30551 ----------------------------------------------------------------
30552 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
30553 States: Active Offline Active
30554 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30555 9455 Acked Write 4 Done Sent Done false
30556 9456 Acked Write 1 Done Sent Done false
30557 9457 Acked Flush 0 Done Sent Done false
30558 9458 Acked Read 6 Done Sent Done false
30559 9459 Acked Flush 0 Done Sent Done false
30560 9460 Acked Flush 0 Done Sent Done false
30561 9461 Acked Write 2 Done Sent Done false
30562 9462 Acked Write 3 Done Sent Done false
30563 9463 Acked Write 10 Done Sent Done false
30564 9464 Acked Write 9 Done Sent Done false
30565 9465 Acked Read 6 Done Sent Done false
30566 9466 Acked Flush 0 Done Sent Done false
30567 9467 Acked Flush 0 Done Sent Done false
30568 9468 Acked Flush 0 Done Sent Done false
30569 9469 Acked Write 1 Done Sent Done false
30570 9470 Acked Write 4 Done Sent Done false
30571 9471 Acked Write 8 Done Sent Done false
30572 9472 Acked Read 9 Done Sent Done false
30573 9473 Acked Read 7 Done Sent Done false
30574 9474 Acked Write 8 Done Sent Done false
30575 9475 Acked Read 9 Done Sent Done false
30576 9476 Acked Read 4 Done Sent Done false
30577 9477 Acked Write 8 Done Sent Done false
30578 9478 Acked Read 3 Done Sent Done false
30579 9479 Acked Flush 0 Done Sent Done false
30580 9480 Acked Write 9 Done Sent Done false
30581 9481 Acked Write 7 Done Sent Done false
30582 9482 Acked Flush 0 Done Sent Done false
30583 9483 Acked Write 1 Done Sent Done false
30584 9484 Acked Read 9 Done Sent Done false
30585 9485 Acked Read 7 Done Sent Done false
30586 9486 Acked Read 7 Done Sent Done false
30587 9487 Acked Read 5 Done Sent Done false
30588 9488 Acked Read 10 Done Sent Done false
30589 9489 Acked Write 4 Done Sent Done false
30590 9490 Acked Read 4 Done Sent Done false
30591 9491 Acked Write 9 Done Sent Done false
30592 9492 Acked Read 10 Done Sent Done false
30593 9493 Acked Read 9 Done Sent Done false
30594 9494 Acked Write 5 Done Sent Done false
30595 9495 Acked Read 7 Done Sent Done false
30596 9496 Acked Read 1 Done Sent Done false
30597 9497 Acked Write 3 Done Sent Done false
30598 9498 Acked Read 8 Done Sent Done false
30599 9499 Acked Write 5 Done Sent Done false
30600 9500 Acked Read 3 Done Sent Done false
30601 9501 Acked Read 10 Done Sent Done false
30602 9502 Acked Read 3 Done Sent Done false
30603 9503 Acked Read 8 Done Sent Done false
30604 9504 Acked Read 6 Done Sent Done false
30605 9505 Acked Write 4 Done Sent Done false
30606 9506 Acked Flush 0 Done Sent Done false
30607 9507 Acked Read 10 Done Sent Done false
30608 9508 Acked Write 6 Done Sent Done false
30609 9509 Acked Read 7 Done Sent Done false
30610 9510 Acked Write 9 Done Sent Done false
30611 9511 Acked Write 6 Done Sent Done false
30612 9512 Acked Read 2 Done Sent Done false
30613 9513 Acked Read 10 Done Sent Done false
30614 9514 Acked Write 2 Done Sent Done false
30615 9515 Acked Read 3 Done Sent Done false
30616 9516 Acked Read 5 Done Sent Done false
30617 9517 Acked Write 2 Done Sent Done false
30618 9518 Acked Read 4 Done Sent Done false
30619 9519 Acked Write 8 Done Sent Done false
30620 9520 Acked Flush 0 Done Sent Done false
30621 9521 Acked Read 10 Done Sent Done false
30622 9522 Acked Write 9 Done Sent Done false
30623 9523 Acked Read 10 Done Sent Done false
30624 9524 Acked Write 10 Done Sent Done false
30625 9525 Acked Read 1 Done Sent Done false
30626 9526 Acked Read 1 Done Sent Done false
30627 9527 Acked Read 8 Done Sent Done false
30628 9528 Acked Read 10 Done Sent Done false
30629 9529 Acked Read 3 Done Sent Done false
30630 9530 Acked Write 5 Done Sent Done false
30631 9531 Acked Write 7 Done Sent Done false
30632 9532 Acked Write 5 Done Sent Done false
30633 9533 Acked Read 9 Done Sent Done false
30634 9534 Acked Read 10 Done Sent Done false
30635 9535 Acked Read 7 Done Sent Done false
30636 9536 Acked Read 7 Done Sent Done false
30637 9537 Acked Read 6 Done Sent Done false
30638 9538 Acked Read 3 Done Sent Done false
30639 9539 Acked Read 5 Done Sent Done false
30640 9540 Acked Read 1 Done Sent Done false
30641 9541 Acked Read 4 Done Sent Done false
30642 9542 Acked Read 1 Done Sent Done false
30643 9543 Acked Flush 0 Done Sent Done false
30644 9544 Acked Write 2 Done Sent Done false
30645 9545 Acked Write 7 Done Sent Done false
30646 9546 Acked Read 5 Done Sent Done false
30647 9547 Acked Write 2 Done Sent Done false
30648 9548 Acked Write 10 Done Sent Done false
30649 9549 Acked Flush 0 Done Sent Done false
30650 9550 Acked Write 7 Done Sent Done false
30651 9551 Acked Read 9 Done Sent Done false
30652 9552 Acked Flush 0 Done Sent Done false
30653 9553 Acked Write 6 Done Sent Done false
30654 9554 Acked Write 6 Done Sent Done false
30655 9555 Acked Read 8 Done Sent Done false
30656 9556 Acked Write 5 Done Sent Done false
30657 9557 Acked Read 6 Done Sent Done false
30658 9558 Acked Flush 0 Done Sent Done false
30659 9559 Acked Write 10 Done Sent Done false
30660 9560 Acked Flush 0 Done Sent Done false
30661 9561 Acked Read 5 Done Sent Done false
30662 9562 Acked Read 3 Done Sent Done false
30663 9563 Acked Read 4 Done Sent Done false
30664 9564 Acked Write 1 Done Sent Done false
30665 9565 Acked Read 5 Done Sent Done false
30666 9566 Acked Flush 0 Done Sent Done false
30667 9567 Acked Write 10 Done Sent Done false
30668 9568 Acked Write 5 Done Sent Done false
30669 9569 Acked Write 3 Done Sent Done false
30670 9570 Acked Read 1 Done Sent Done false
30671 9571 Acked Read 10 Done Sent Done false
30672 9572 Acked Flush 0 Done Sent Done false
30673 9573 Acked Read 1 Done Sent Done false
30674 9574 Acked Write 2 Done Sent Done false
30675 9575 Acked Write 4 Done Sent Done false
30676 9576 Acked Read 6 Done Sent Done false
30677 9577 Acked Read 10 Done Sent Done false
30678 9578 Acked Write 1 Done Sent Done false
30679 9579 Acked Write 9 Done Sent Done false
30680 9580 Acked Write 3 Done Sent Done false
30681 9581 Acked Read 9 Done Sent Done false
30682 9582 Acked Write 8 Done Sent Done false
30683 9583 Acked Read 8 Done Sent Done false
30684 9584 Acked Write 3 Done Sent Done false
30685 9585 Acked Read 5 Done Sent Done false
30686 9586 Acked Read 3 Done Sent Done false
30687 9587 Acked Read 6 Done Sent Done false
30688 9588 Acked Read 9 Done Sent Done false
30689 9589 Acked Read 6 Done Sent Done false
30690 9590 Acked Read 8 Done Sent Done false
30691 9591 Acked Write 9 Done Sent Done false
30692 9592 Acked Write 2 Done Sent Done false
30693 9593 Acked Read 2 Done Sent Done false
30694 9594 Acked Read 9 Done Sent Done false
30695 9595 Acked Flush 0 Done Sent Done false
30696 9596 Acked Flush 0 Done Sent Done false
30697 9597 Acked Read 6 Done Sent Done false
30698 9598 Acked Flush 0 Done Sent Done false
30699 9599 Acked Read 9 Done Sent Done false
30700 9600 Acked Write 8 Done Sent Done false
30701 9601 Acked Read 4 Done Sent Done false
30702 9602 Acked Write 1 Done Sent Done false
30703 9603 Acked Read 7 Done Sent Done false
30704 9604 Acked Flush 0 Done Sent Done false
30705 9605 Acked Read 7 Done Sent Done false
30706 9606 Acked Read 10 Done Sent Done false
30707 9607 Acked Read 10 Done Sent Done false
30708 9608 Acked Read 6 Done Sent Done false
30709 9609 Acked Read 8 Done Sent Done false
30710 9610 Acked Read 6 Done Sent Done false
30711 9611 Acked Flush 0 Done Sent Done false
30712 9612 Acked Write 8 Done Sent Done false
30713 9613 Acked Write 10 Done Sent Done false
30714 9614 Acked Read 10 Done Sent Done false
30715 9615 Acked Write 7 Done Sent Done false
30716 9616 Acked Write 10 Done Sent Done false
30717 9617 Acked Write 6 Done Sent Done false
30718 9618 Acked Read 5 Done Sent Done false
30719 9619 Acked Read 7 Done Sent Done false
30720 9620 Acked Read 9 Done Sent Done false
30721 9621 Acked Read 10 Done Sent Done false
30722 9622 Acked Read 9 Done Sent Done false
30723 9623 Acked Write 4 Done Sent Done false
30724 9624 Acked Read 2 Done Sent Done false
30725 9625 Acked Flush 0 Done Sent Done false
30726 9626 Acked Flush 0 Done Sent Done false
30727 9627 Acked Read 8 Done Sent Done false
30728 9628 Acked Read 6 Done Sent Done false
30729 9629 Acked Read 2 Done Sent Done false
30730 9630 Acked Read 10 Done Sent Done false
30731 9631 Acked Read 4 Done Sent Done false
30732 9632 Acked Read 8 Done Sent Done false
30733 9633 Acked Write 4 Done Sent Done false
30734 9634 Acked Flush 0 Done Sent Done false
30735 9635 Acked Write 4 Done Sent Done false
30736 9636 Acked Read 10 Done Sent Done false
30737 9637 Acked Read 9 Done Sent Done false
30738 9638 Acked Read 7 Done Sent Done false
30739 9639 Acked Flush 0 Done Sent Done false
30740 9640 Acked Write 2 Done Sent Done false
30741 9641 Acked Write 6 Done Sent Done false
30742 9642 Acked Read 1 Done Sent Done false
30743 9643 Acked Write 6 Done Sent Done false
30744 9644 Acked Write 4 Done Sent Done false
30745 9645 Acked Write 3 Done Sent Done false
30746 9646 Acked Read 3 Done Sent Done false
30747 9647 Acked Read 10 Done Sent Done false
30748 9648 Acked Read 5 Done Sent Done false
30749 9649 Acked Write 6 Done Sent Done false
30750 9650 Acked Read 5 Done Sent Done false
30751 9651 Acked Read 4 Done Sent Done false
30752 9652 Acked Flush 0 Done Sent Done false
30753 9653 Acked Read 9 Done Sent Done false
30754 9654 Acked Write 6 Done Sent Done false
30755 9655 Acked Read 7 Done Sent Done false
30756 9656 Acked Read 6 Done Sent Done false
30757 9657 Acked Read 8 Done Sent Done false
30758 9658 Acked Read 6 Done Sent Done false
30759 9659 Acked Write 4 Done Sent Done false
30760 9660 Acked Read 8 Done Sent Done false
30761 9661 Acked Write 5 Done Sent Done false
30762 9662 Acked Read 7 Done Sent Done false
30763 9663 Acked Read 3 Done Sent Done false
30764 9664 Acked Read 9 Done Sent Done false
30765 9665 Acked Read 1 Done Sent Done false
30766 9666 Acked Flush 0 Done Sent Done false
30767 9667 Acked Read 10 Done Sent Done false
30768 9668 Acked Flush 0 Done Sent Done false
30769 9669 Acked Write 7 Done Sent Done false
30770 9670 Acked Flush 0 Done Sent Done false
30771 9671 Acked Flush 0 Done Sent Done false
30772 9672 Acked Read 6 Done Sent Done false
30773 9673 Acked Flush 0 Done Sent Done false
30774 9674 Acked Flush 0 Done Sent Done false
30775 9675 Acked Write 4 Done Sent Done false
30776 9676 Acked Read 8 Done Sent Done false
30777 9677 Acked Write 7 Done Sent Done false
30778 9678 Acked Read 8 Done Sent Done false
30779 9679 Acked Write 8 Done Sent Done false
30780 9680 Acked Write 9 Done Sent Done false
30781 9681 Acked Flush 0 Done Sent Done false
30782 9682 Acked Write 9 Done Sent Done false
30783 9683 Acked Read 2 Done Sent Done false
30784 9684 Acked Write 9 Done Sent Done false
30785 9685 Acked Read 1 Done Sent Done false
30786 9686 Acked Write 1 Done Sent Done false
30787 9687 Acked Read 7 Done Sent Done false
30788 9688 Acked Read 4 Done Sent Done false
30789 9689 Acked Write 8 Done Sent Done false
30790 9690 Acked Read 1 Done Sent Done false
30791 9691 Acked Write 9 Done Sent Done false
30792 9692 Acked Read 4 Done Sent Done false
30793 9693 Acked Read 10 Done Sent Done false
30794 9694 Acked Write 3 Done Sent Done false
30795 9695 Acked Read 2 Done Sent Done false
30796 9696 Acked Write 5 Done Sent Done false
30797 9697 Acked Write 2 Done Sent Done false
30798 9698 Acked Write 1 Done Sent Done false
30799 9699 Acked Read 10 Done Sent Done false
30800 9700 Acked Write 1 Done Sent Done false
30801 9701 Acked Read 1 Done Sent Done false
30802 9702 Acked Write 2 Done Sent Done false
30803 9703 Acked Write 10 Done Sent Done false
30804 9704 Acked Write 4 Done Sent Done false
30805 9705 Acked Flush 0 Done Sent Done false
30806 9706 Acked Read 6 Done Sent Done false
30807 9707 Acked Write 2 Done Sent Done false
30808 9708 Acked Read 10 Done Sent Done false
30809 9709 Acked Flush 0 Done Sent Done false
30810 9710 Acked Read 2 Done Sent Done false
30811 9711 Acked Write 5 Done Sent Done false
30812 9712 Acked Read 1 Done Sent Done false
30813 9713 Acked Write 9 Done Sent Done false
30814 9714 Acked Write 4 Done Sent Done false
30815 9715 Acked Write 10 Done Sent Done false
30816 9716 Acked Read 8 Done Sent Done false
30817 9717 Acked Flush 0 Done Sent Done false
30818 9718 Acked Read 5 Done Sent Done false
30819 9719 Acked Read 1 Done Sent Done false
30820 9720 Acked Read 4 Done Sent Done false
30821 9721 Acked Read 7 Done Sent Done false
30822 9722 Acked Read 2 Done Sent Done false
30823 9723 Acked Write 9 Done Sent Done false
30824 9724 Acked Write 8 Done Sent Done false
30825 9725 Acked Read 3 Done Sent Done false
30826 9726 Acked Read 10 Done Sent Done false
30827 9727 Acked Read 8 Done Sent Done false
30828 9728 Acked Write 9 Done Sent Done false
30829 9729 Acked Read 9 Done Sent Done false
30830 9730 Acked Write 9 Done Sent Done false
30831 9731 Acked Write 5 Done Sent Done false
30832 9732 Acked Read 7 Done Sent Done false
30833 9733 Acked Read 5 Done Sent Done false
30834 9734 Acked Write 10 Done Sent Done false
30835 9735 Acked Read 10 Done Sent Done false
30836 9736 Acked Write 8 Done Sent Done false
30837 9737 Acked Read 7 Done Sent Done false
30838 9738 Acked Write 5 Done Sent Done false
30839 9739 Acked Read 6 Done Sent Done false
30840 9740 Acked Write 1 Done Sent Done false
30841 9741 Acked Read 8 Done Sent Done false
30842 9742 Acked Read 6 Done Sent Done false
30843 9743 Acked Read 9 Done Sent Done false
30844 9744 Acked Write 10 Done Sent Done false
30845 9745 Acked Read 7 Done Sent Done false
30846 9746 Acked Write 9 Done Sent Done false
30847 9747 Acked Read 8 Done Sent Done false
30848 9748 Acked Write 10 Done Sent Done false
30849 9749 Acked Write 8 Done Sent Done false
30850 9750 Acked Flush 0 Done Sent Done false
30851 9751 Acked Write 6 Done Sent Done false
30852 9752 Acked Read 3 Done Sent Done false
30853 9753 Acked Read 2 Done Sent Done false
30854 9754 Acked Write 9 Done Sent Done false
30855 9755 Acked Read 3 Done Sent Done false
30856 9756 Acked Flush 0 Done Sent Done false
30857 STATES DS:0 DS:1 DS:2 TOTAL
30858 Sent 0 302 0 302
30859 Done 302 0 302 604
30860 Skipped 0 0 0 0
30861 Error 0 0 0 0
30862 Last Flush: Some(JobId(9756)) Some(JobId(9454)) Some(JobId(9756))
30863 Downstairs last five completed: 9454 9453 9452 9451 9450
30864 Upstairs last five completed: 9756 9755 9754 9753 9752
30865 CLIENT: Up:0 ds:302 act:2
308662025-01-23T17:44:12.679ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
308672025-01-23T17:44:12.679ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55169) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
308682025-01-23T17:44:12.681ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
308692025-01-23T17:44:12.681ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
308702025-01-23T17:44:12.681ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
308712025-01-23T17:44:12.681ZINFOcrucible: send last flush ID to this DS: Some(JobId(9454)) = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
308722025-01-23T17:44:12.681ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
308732025-01-23T17:44:12.681ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(9454)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
308742025-01-23T17:44:12.681ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(9454)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
30875 ----------------------------------------------------------------
30876 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30877 Downstairs last five completed: 9756 9755 9754 9753 9752
30878 Upstairs last five completed: 9756 9755 9754 9753 9752
30879 CLIENT: Up:0 ds:0 act:3
30880 Replay: All jobs finished
30881 Main task has work to do, go find it
30882 got dsc Stop(1)
30883 stop 1
30884 Jan 23 17:44:14.948 INFO [8820] Got stop action so:false kr:false
30885 request completed, latency_us: 380, response_code: 204, uri: /stop/cid/1, method: POST, req_id: a23924d7-b38c-4570-a71d-245328814f94, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
308862025-01-23T17:44:14.949ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
308872025-01-23T17:44:14.949ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
308882025-01-23T17:44:14.949ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
308892025-01-23T17:44:14.949ZINFOcrucible: client task is exiting = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
30890 Jan 23 17:44:14.949 INFO request completed, latency_us: 147, response_code: 200, uri: /state/cid/1, method: GET, req_id: 7b6cb021-f3fb-4f10-9ab7-7b26cee6165c, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
30891 [8820] Exited with: ExitStatus(unix_wait_status(9))
30892 [8820][1] reports Exit
30893 Jan 23 17:44:18.952 INFO request completed, latency_us: 341, response_code: 200, uri: /state/cid/1, method: GET, req_id: 370d3c9c-ae5f-4f1d-a0a8-d33172c0affa, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
30894 001/300 Read block 307 len 36864
30895 002/300 Read block 524 len 28672
30896 003/300 Read block 12 len 12288
30897 004/300 Read block 3301 len 40960
30898 005/300 Read block 1521 len 12288
30899 006/300 Read block 777 len 8192
30900 007/300 Flush
30901 008/300 Flush
30902 009/300 Read block 2307 len 16384
30903 010/300 Write block 3777 len 8192 data: 4 3
30904 011/300 Write block 2141 len 28672 data: 7 6 5 4 4 7 6
30905 012/300 Write block 344 len 32768 data: 4 3 4 4 4 5 5 6
30906 013/300 Read block 2483 len 4096
30907 014/300 Read block 1983 len 16384
30908 015/300 Write block 761 len 20480 data: 5 5 5 7 7
30909 016/300 Write block 2875 len 20480 data: 5 6 5 6 5
30910 017/300 Read block 3434 len 40960
30911 018/300 Write block 1255 len 16384 data: 5 4 8 9
30912 019/300 Write block 834 len 20480 data: 7 6 6 7 7
30913 020/300 Write block 1971 len 8192 data: 9 9
30914 021/300 Read block 684 len 16384
30915 022/300 Read block 4525 len 32768
30916 023/300 Read block 4790 len 12288
30917 024/300 Write block 498 len 4096 data: 5
30918 025/300 Flush
30919 026/300 Read block 672 len 16384
30920 027/300 Read block 4803 len 28672
30921 028/300 Write block 2999 len 24576 data: 7 6 6 5 4 4
30922 029/300 Read block 4793 len 36864
30923 030/300 Read block 340 len 28672
30924 031/300 Read block 4069 len 4096
30925 032/300 Write block 1931 len 28672 data: 4 3 5 5 5 4 3
30926 033/300 Write block 4022 len 36864 data: 4 3 3 3 4 4 5 7 4
30927 034/300 Read block 362 len 40960
30928 035/300 Read block 782 len 8192
30929 036/300 Flush
30930 037/300 Read block 3641 len 32768
30931 038/300 Write block 2381 len 24576 data: 8 7 9 11 10 8
30932 039/300 Write block 2818 len 24576 data: 6 6 5 3 3 3
30933 040/300 Write block 1985 len 20480 data: 4 4 3 5 5
30934 041/300 Flush
30935 042/300 Flush
30936 043/300 Read block 2650 len 20480
30937 044/300 Read block 2537 len 32768
30938 045/300 Read block 4453 len 40960
30939 046/300 Read block 3146 len 28672
30940 047/300 Read block 4260 len 20480
30941 048/300 Write block 1201 len 12288 data: 5 5 5
30942 049/300 Read block 290 len 40960
30943 050/300 Write block 3941 len 28672 data: 8 8 9 9 8 8 8
30944 051/300 Write block 3562 len 24576 data: 7 7 8 9 8 7
30945 052/300 Read block 3064 len 24576
30946 053/300 Read block 4977 len 24576
30947 054/300 Write block 2520 len 36864 data: 7 5 3 4 4 4 4 4 5
30948 055/300 Write block 388 len 24576 data: 7 5 6 5 4 4
30949 056/300 Write block 2337 len 4096 data: 6
30950 057/300 Read block 284 len 8192
30951 058/300 Read block 4635 len 32768
30952 059/300 Read block 4032 len 32768
30953 060/300 Write block 3200 len 24576 data: 8 5 4 6 7 6
30954 061/300 Read block 3167 len 32768
30955 062/300 Read block 1318 len 12288
30956 063/300 Write block 4778 len 8192 data: 7 8
30957 064/300 Read block 406 len 28672
30958 065/300 Flush
30959 066/300 Read block 1247 len 20480
30960 067/300 Write block 546 len 4096 data: 6
30961 068/300 Write block 824 len 20480 data: 9 9 11 11 10
30962 069/300 Read block 2946 len 24576
30963 070/300 Write block 3504 len 8192 data: 6 7
30964 071/300 Write block 2163 len 32768 data: 7 5 5 5 6 5 4 4
30965 072/300 Write block 2058 len 28672 data: 8 8 9 10 11 11 10
30966 073/300 Write block 787 len 12288 data: 7 6 6
30967 074/300 Read block 3723 len 36864
30968 075/300 Read block 734 len 12288
30969 076/300 Read block 309 len 28672
30970 077/300 Read block 4334 len 28672
30971 078/300 Read block 4163 len 4096
30972 079/300 Write block 4341 len 8192 data: 6 5
30973 080/300 Write block 3297 len 4096 data: 4
30974 081/300 Read block 2596 len 4096
30975 082/300 Flush
30976 083/300 Write block 2195 len 36864 data: 6 5 5 5 5 8 6 7 8
30977 084/300 Read block 1527 len 16384
30978 085/300 Read block 3557 len 40960
30979 086/300 Write block 4482 len 28672 data: 5 5 4 6 4 5 4
30980 087/300 Read block 4790 len 36864
30981 088/300 Read block 3542 len 28672
30982 089/300 Write block 1043 len 12288 data: 5 6 6
30983 090/300 Read block 12 len 8192
30984 091/300 Read block 4506 len 20480
30985 092/300 Write block 768 len 8192 data: 5 4
30986 093/300 Read block 141 len 40960
30987 094/300 Flush
30988 095/300 Write block 3102 len 36864 data: 7 8 9 9 10 10 10 8 10
30989 096/300 Write block 2512 len 32768 data: 5 6 7 7 7 7 7 6
30990 097/300 Read block 4679 len 24576
30991 098/300 Read block 1057 len 12288
30992 099/300 Write block 2955 len 28672 data: 6 5 8 7 7 6 6
30993 100/300 Read block 242 len 8192
30994 101/300 Read block 2222 len 20480
30995 102/300 Flush
30996 103/300 Write block 4410 len 12288 data: 8 7 9
30997 104/300 Write block 4046 len 4096 data: 6
30998 105/300 Write block 2266 len 16384 data: 5 5 5 5
30999 106/300 Read block 2651 len 8192
31000 107/300 Write block 2346 len 24576 data: 5 5 7 7 8 10
31001 108/300 Write block 4571 len 40960 data: 7 5 5 5 6 7 6 9 8 7
31002 109/300 Write block 890 len 28672 data: 7 5 7 7 6 5 5
31003 110/300 Read block 4796 len 4096
31004 111/300 Read block 1599 len 28672
31005 112/300 Write block 110 len 16384 data: 9 8 8 6
31006 113/300 Read block 3527 len 20480
31007 114/300 Read block 4726 len 4096
31008 115/300 Read block 1299 len 36864
31009 116/300 Read block 626 len 16384
31010 117/300 Read block 3892 len 28672
31011 118/300 Write block 4816 len 36864 data: 6 6 6 5 5 5 4 4 3
31012 119/300 Write block 187 len 28672 data: 5 5 4 4 4 5 5
31013 120/300 Write block 732 len 36864 data: 7 8 8 9 9 6 4 5 3
31014 121/300 Write block 4328 len 32768 data: 4 5 5 5 6 5 6 7
31015 122/300 Read block 2114 len 4096
31016 123/300 Write block 4552 len 28672 data: 8 5 7 8 11 11 11
31017 124/300 Write block 4150 len 4096 data: 6
31018 125/300 Read block 4286 len 16384
31019 126/300 Flush
31020 127/300 Write block 3449 len 32768 data: 6 6 6 6 6 6 6 7
31021 128/300 Read block 1839 len 16384
31022 129/300 Write block 2036 len 12288 data: 3 3 3
31023 130/300 Write block 3830 len 16384 data: 7 5 5 4
31024 131/300 Write block 1590 len 16384 data: 3 4 4 5
31025 132/300 Read block 2195 len 36864
31026 133/300 Write block 4298 len 28672 data: 3 4 4 6 6 7 7
31027 134/300 Flush
31028 135/300 Read block 610 len 32768
31029 136/300 Read block 1264 len 36864
31030 137/300 Write block 4735 len 40960 data: 6 6 5 6 7 5 5 5 6 6
31031 138/300 Write block 2647 len 4096 data: 4
31032 139/300 Write block 1004 len 8192 data: 6 8
31033 140/300 Read block 1466 len 40960
31034 141/300 Read block 447 len 24576
31035 142/300 Read block 3959 len 4096
31036 143/300 Write block 3322 len 20480 data: 5 5 5 5 5
31037 144/300 Read block 2230 len 28672
31038 145/300 Write block 1311 len 20480 data: 13 11 10 9 9
31039 146/300 Read block 4020 len 28672
31040 147/300 Read block 4683 len 32768
31041 148/300 Read block 3278 len 32768
31042 149/300 Read block 1733 len 32768
31043 150/300 Write block 3676 len 36864 data: 6 7 8 6 7 6 6 6 6
31044 151/300 Read block 2512 len 40960
31045 152/300 Read block 2586 len 4096
31046 153/300 Flush
31047 154/300 Read block 2375 len 40960
31048 155/300 Read block 510 len 36864
31049 156/300 Write block 803 len 20480 data: 5 4 3 4 4
31050 157/300 Write block 4745 len 28672 data: 8 8 6 4 3 3 4
31051 158/300 Read block 3839 len 8192
31052 159/300 Write block 2435 len 24576 data: 7 7 5 4 4 4
31053 160/300 Read block 909 len 36864
31054 161/300 Write block 4182 len 36864 data: 6 6 3 4 3 3 2 2 2
31055 162/300 Read block 2995 len 36864
31056 163/300 Write block 4023 len 16384 data: 4 4 4 5
31057 164/300 Flush
31058 165/300 Write block 1678 len 36864 data: 7 6 7 10 9 8 7 5 5
31059 166/300 Write block 1603 len 8192 data: 4 6
31060 167/300 Write block 4479 len 36864 data: 4 4 5 6 6 5 7 5 6
31061 168/300 Read block 2165 len 20480
31062 169/300 Write block 1528 len 12288 data: 2 2 4
31063 170/300 Write block 2788 len 36864 data: 2 3 3 3 5 5 7 6 6
31064 171/300 Write block 1182 len 16384 data: 5 4 5 5
31065 172/300 Read block 1997 len 28672
31066 173/300 Write block 4316 len 28672 data: 10 6 7 7 7 6 5
31067 174/300 Write block 990 len 20480 data: 11 10 10 6 5
31068 175/300 Read block 3964 len 8192
31069 176/300 Write block 1718 len 40960 data: 4 5 6 6 7 8 8 7 6 5
31070 177/300 Read block 2163 len 8192
31071 178/300 Write block 1104 len 40960 data: 3 3 2 4 4 4 5 5 7 7
31072 179/300 Write block 1653 len 4096 data: 10
31073 180/300 Read block 1185 len 36864
31074 181/300 Read block 1144 len 40960
31075 182/300 Read block 2151 len 12288
31076 183/300 Read block 3110 len 12288
31077 184/300 Write block 2562 len 12288 data: 4 4 6
31078 185/300 Read block 885 len 28672
31079 186/300 Read block 2576 len 8192
31080 187/300 Read block 1647 len 32768
31081 188/300 Write block 3483 len 8192 data: 5 4
31082 189/300 Read block 4153 len 36864
31083 190/300 Read block 3754 len 16384
31084 191/300 Write block 2822 len 28672 data: 4 4 4 2 2 3 3
31085 192/300 Read block 881 len 28672
31086 193/300 Write block 579 len 8192 data: 6 6
31087 194/300 Write block 3632 len 24576 data: 4 3 3 6 7 7
31088 195/300 Read block 4480 len 32768
31089 196/300 Read block 4078 len 24576
31090 197/300 Read block 2828 len 36864
31091 198/300 Read block 3009 len 28672
31092 199/300 Write block 3370 len 24576 data: 5 6 3 3 7 6
31093 200/300 Write block 3190 len 24576 data: 5 5 5 6 7 4
31094 201/300 Read block 283 len 24576
31095 202/300 Write block 566 len 24576 data: 7 7 6 7 10 11
31096 203/300 Read block 4672 len 24576
31097 204/300 Read block 38 len 20480
31098 205/300 Write block 1265 len 24576 data: 4 3 2 3 3 4
31099 206/300 Read block 3098 len 40960
31100 207/300 Read block 835 len 12288
31101 208/300 Flush
31102 209/300 Read block 4530 len 12288
31103 210/300 Read block 4859 len 40960
31104 211/300 Flush
31105 212/300 Write block 4181 len 16384 data: 9 7 7 4
31106 213/300 Write block 2247 len 32768 data: 3 4 4 4 6 8 7 8
31107 214/300 Write block 2409 len 24576 data: 6 6 7 8 11 9
31108 215/300 Read block 2502 len 16384
31109 216/300 Read block 1090 len 32768
31110 217/300 Read block 1243 len 12288
31111 218/300 Read block 2661 len 28672
31112 219/300 Write block 241 len 28672 data: 7 6 5 5 5 5 4
31113 220/300 Read block 825 len 24576
31114 221/300 Write block 2134 len 20480 data: 7 6 5 6 6
31115 222/300 Flush
31116 223/300 Write block 1218 len 24576 data: 4 5 5 6 6 5
31117 224/300 Read block 704 len 36864
31118 225/300 Write block 1256 len 8192 data: 5 9
31119 226/300 Read block 3019 len 32768
31120 227/300 Read block 1407 len 16384
31121 228/300 Read block 1767 len 36864
31122 229/300 Read block 1456 len 32768
31123 230/300 Write block 4528 len 20480 data: 5 4 4 5 6
31124 231/300 Read block 4427 len 20480
31125 232/300 Read block 1982 len 16384
31126 233/300 Read block 3891 len 24576
31127 234/300 Read block 4129 len 36864
31128 235/300 Read block 1396 len 28672
31129 236/300 Write block 660 len 20480 data: 5 4 4 3 3
31130 237/300 Read block 1461 len 32768
31131 238/300 Write block 257 len 36864 data: 3 2 3 4 4 4 4 4 4
31132 239/300 Write block 1417 len 32768 data: 5 6 7 9 7 6 7 5
31133 240/300 Write block 1246 len 16384 data: 4 5 5 5
31134 241/300 Read block 3423 len 12288
31135 242/300 Flush
31136 243/300 Write block 1544 len 36864 data: 4 5 5 4 5 6 7 6 5
31137 244/300 Flush
31138 245/300 Read block 3548 len 28672
31139 246/300 Flush
31140 247/300 Write block 1962 len 36864 data: 7 8 8 10 9 10 9 10 11
31141 248/300 Read block 473 len 32768
31142 249/300 Read block 4395 len 28672
31143 250/300 Read block 4920 len 20480
31144 251/300 Read block 790 len 16384
31145 252/300 Flush
31146 253/300 Read block 3593 len 40960
31147 254/300 Read block 50 len 12288
31148 255/300 Write block 2554 len 12288 data: 6 6 5
31149 256/300 Read block 2854 len 36864
31150 257/300 Flush
31151 258/300 Flush
31152 259/300 Read block 3524 len 24576
31153 260/300 Write block 386 len 16384 data: 6 6 8 6
31154 261/300 Read block 993 len 40960
31155 262/300 Read block 372 len 4096
31156 263/300 Read block 4984 len 36864
31157 264/300 Read block 1618 len 20480
31158 265/300 Write block 3863 len 32768 data: 5 3 3 3 3 3 5 4
31159 266/300 Read block 4422 len 4096
31160 267/300 Read block 1531 len 12288
31161 268/300 Read block 667 len 16384
31162 269/300 Read block 1278 len 24576
31163 270/300 Read block 2119 len 20480
31164 271/300 Read block 3991 len 40960
31165 272/300 Write block 4376 len 4096 data: 4
31166 273/300 Write block 3586 len 16384 data: 6 5 6 7
31167 274/300 Write block 181 len 8192 data: 4 4
31168 275/300 Write block 1372 len 20480 data: 7 9 10 10 10
31169 276/300 Write block 2422 len 4096 data: 6
31170 277/300 Read block 2322 len 32768
31171 278/300 Read block 2167 len 36864
31172 279/300 Read block 38 len 32768
31173 280/300 Flush
31174 281/300 Read block 3951 len 40960
31175 282/300 Write block 1210 len 16384 data: 4 3 3 4
31176 283/300 Read block 1860 len 4096
31177 284/300 Flush
31178 285/300 Write block 4757 len 32768 data: 5 5 7 8 8 7 7 7
31179 286/300 Flush
31180 287/300 Read block 2319 len 32768
31181 288/300 Write block 2926 len 24576 data: 10 8 9 5 7 8
31182 289/300 Read block 632 len 12288
31183 290/300 Read block 4368 len 40960
31184 291/300 Read block 1972 len 20480
31185 292/300 Read block 4535 len 32768
31186 293/300 Read block 616 len 16384
31187 294/300 Write block 3809 len 36864 data: 7 8 11 9 8 8 6 5 4
31188 295/300 Read block 1080 len 16384
31189 296/300 Write block 740 len 36864 data: 4 3 4 4 4 4 7 8 10
31190 297/300 Read block 936 len 8192
31191 298/300 Write block 4011 len 24576 data: 6 5 4 5 5 5
31192 299/300 Read block 4079 len 8192
31193 300/300 Write block 4440 len 28672 data: 6 7 7 8 8 8 8
31194 301/300 Write block 858 len 4096 data: 4
31195 Main task has work to do, go find it
31196 got dsc Start(1)
31197 start 1
31198 Jan 23 17:44:19.206 INFO request completed, latency_us: 237, response_code: 204, uri: /start/cid/1, method: POST, req_id: 4022f080-642c-4e66-9573-74176f25b5f8, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
31199 Replay: started 1, returned:Ok(())
31200 ----------------------------------------------------------------
31201 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31202 States: Active Offline Active
31203 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31204 9757 Acked Read 9 Done Sent Done false
31205 9758 Acked Read 7 Done Sent Done false
31206 9759 Acked Read 3 Done Sent Done false
31207 9760 Acked Read 10 Done Sent Done false
31208 9761 Acked Read 3 Done Sent Done false
31209 9762 Acked Read 2 Done Sent Done false
31210 9763 Acked Flush 0 Done Sent Done false
31211 9764 Acked Flush 0 Done Sent Done false
31212 9765 Acked Read 4 Done Sent Done false
31213 9766 Acked Write 2 Done Sent Done false
31214 9767 Acked Write 7 Done Sent Done false
31215 9768 Acked Write 8 Done Sent Done false
31216 9769 Acked Read 1 Done Sent Done false
31217 9770 Acked Read 4 Done Sent Done false
31218 9771 Acked Write 5 Done Sent Done false
31219 9772 Acked Write 5 Done Sent Done false
31220 9773 Acked Read 10 Done Sent Done false
31221 9774 Acked Write 4 Done Sent Done false
31222 9775 Acked Write 5 Done Sent Done false
31223 9776 Acked Write 2 Done Sent Done false
31224 9777 Acked Read 4 Done Sent Done false
31225 9778 Acked Read 8 Done Sent Done false
31226 9779 Acked Read 3 Done Sent Done false
31227 9780 Acked Write 1 Done Sent Done false
31228 9781 Acked Flush 0 Done Sent Done false
31229 9782 Acked Read 4 Done Sent Done false
31230 9783 Acked Read 7 Done Sent Done false
31231 9784 Acked Write 6 Done Sent Done false
31232 9785 Acked Read 9 Done Sent Done false
31233 9786 Acked Read 7 Done Sent Done false
31234 9787 Acked Read 1 Done Sent Done false
31235 9788 Acked Write 7 Done Sent Done false
31236 9789 Acked Write 9 Done Sent Done false
31237 9790 Acked Read 10 Done Sent Done false
31238 9791 Acked Read 2 Done Sent Done false
31239 9792 Acked Flush 0 Done Sent Done false
31240 9793 Acked Read 8 Done Sent Done false
31241 9794 Acked Write 6 Done Sent Done false
31242 9795 Acked Write 6 Done Sent Done false
31243 9796 Acked Write 5 Done Sent Done false
31244 9797 Acked Flush 0 Done Sent Done false
31245 9798 Acked Flush 0 Done Sent Done false
31246 9799 Acked Read 5 Done Sent Done false
31247 9800 Acked Read 8 Done Sent Done false
31248 9801 Acked Read 10 Done Sent Done false
31249 9802 Acked Read 7 Done Sent Done false
31250 9803 Acked Read 5 Done Sent Done false
31251 9804 Acked Write 3 Done Sent Done false
31252 9805 Acked Read 10 Done Sent Done false
31253 9806 Acked Write 7 Done Sent Done false
31254 9807 Acked Write 6 Done Sent Done false
31255 9808 Acked Read 6 Done Sent Done false
31256 9809 Acked Read 6 Done Sent Done false
31257 9810 Acked Write 9 Done Sent Done false
31258 9811 Acked Write 6 Done Sent Done false
31259 9812 Acked Write 1 Done Sent Done false
31260 9813 Acked Read 2 Done Sent Done false
31261 9814 Acked Read 8 Done Sent Done false
31262 9815 Acked Read 8 Done Sent Done false
31263 9816 Acked Write 6 Done Sent Done false
31264 9817 Acked Read 8 Done Sent Done false
31265 9818 Acked Read 3 Done Sent Done false
31266 9819 Acked Write 2 Done Sent Done false
31267 9820 Acked Read 7 Done Sent Done false
31268 9821 Acked Flush 0 Done Sent Done false
31269 9822 Acked Read 5 Done Sent Done false
31270 9823 Acked Write 1 Done Sent Done false
31271 9824 Acked Write 5 Done Sent Done false
31272 9825 Acked Read 6 Done Sent Done false
31273 9826 Acked Write 2 Done Sent Done false
31274 9827 Acked Write 8 Done Sent Done false
31275 9828 Acked Write 7 Done Sent Done false
31276 9829 Acked Write 3 Done Sent Done false
31277 9830 Acked Read 9 Done Sent Done false
31278 9831 Acked Read 3 Done Sent Done false
31279 9832 Acked Read 7 Done Sent Done false
31280 9833 Acked Read 7 Done Sent Done false
31281 9834 Acked Read 1 Done Sent Done false
31282 9835 Acked Write 2 Done Sent Done false
31283 9836 Acked Write 1 Done Sent Done false
31284 9837 Acked Read 1 Done Sent Done false
31285 9838 Acked Flush 0 Done Sent Done false
31286 9839 Acked Write 9 Done Sent Done false
31287 9840 Acked Read 4 Done Sent Done false
31288 9841 Acked Read 10 Done Sent Done false
31289 9842 Acked Write 7 Done Sent Done false
31290 9843 Acked Read 9 Done Sent Done false
31291 9844 Acked Read 7 Done Sent Done false
31292 9845 Acked Write 3 Done Sent Done false
31293 9846 Acked Read 2 Done Sent Done false
31294 9847 Acked Read 5 Done Sent Done false
31295 9848 Acked Write 2 Done Sent Done false
31296 9849 Acked Read 10 Done Sent Done false
31297 9850 Acked Flush 0 Done Sent Done false
31298 9851 Acked Write 9 Done Sent Done false
31299 9852 Acked Write 8 Done Sent Done false
31300 9853 Acked Read 6 Done Sent Done false
31301 9854 Acked Read 3 Done Sent Done false
31302 9855 Acked Write 7 Done Sent Done false
31303 9856 Acked Read 2 Done Sent Done false
31304 9857 Acked Read 5 Done Sent Done false
31305 9858 Acked Flush 0 Done Sent Done false
31306 9859 Acked Write 3 Done Sent Done false
31307 9860 Acked Write 1 Done Sent Done false
31308 9861 Acked Write 4 Done Sent Done false
31309 9862 Acked Read 2 Done Sent Done false
31310 9863 Acked Write 6 Done Sent Done false
31311 9864 Acked Write 10 Done Sent Done false
31312 9865 Acked Write 7 Done Sent Done false
31313 9866 Acked Read 1 Done Sent Done false
31314 9867 Acked Read 7 Done Sent Done false
31315 9868 Acked Write 4 Done Sent Done false
31316 9869 Acked Read 5 Done Sent Done false
31317 9870 Acked Read 1 Done Sent Done false
31318 9871 Acked Read 9 Done Sent Done false
31319 9872 Acked Read 4 Done Sent Done false
31320 9873 Acked Read 7 Done Sent Done false
31321 9874 Acked Write 9 Done Sent Done false
31322 9875 Acked Write 7 Done Sent Done false
31323 9876 Acked Write 9 Done Sent Done false
31324 9877 Acked Write 8 Done Sent Done false
31325 9878 Acked Read 1 Done Sent Done false
31326 9879 Acked Write 7 Done Sent Done false
31327 9880 Acked Write 1 Done Sent Done false
31328 9881 Acked Read 4 Done Sent Done false
31329 9882 Acked Flush 0 Done Sent Done false
31330 9883 Acked Write 8 Done Sent Done false
31331 9884 Acked Read 4 Done Sent Done false
31332 9885 Acked Write 3 Done Sent Done false
31333 9886 Acked Write 4 Done Sent Done false
31334 9887 Acked Write 4 Done Sent Done false
31335 9888 Acked Read 9 Done Sent Done false
31336 9889 Acked Write 7 Done Sent Done false
31337 9890 Acked Flush 0 Done Sent Done false
31338 9891 Acked Read 8 Done Sent Done false
31339 9892 Acked Read 9 Done Sent Done false
31340 9893 Acked Write 10 Done Sent Done false
31341 9894 Acked Write 1 Done Sent Done false
31342 9895 Acked Write 2 Done Sent Done false
31343 9896 Acked Read 10 Done Sent Done false
31344 9897 Acked Read 6 Done Sent Done false
31345 9898 Acked Read 1 Done Sent Done false
31346 9899 Acked Write 5 Done Sent Done false
31347 9900 Acked Read 7 Done Sent Done false
31348 9901 Acked Write 5 Done Sent Done false
31349 9902 Acked Read 7 Done Sent Done false
31350 9903 Acked Read 8 Done Sent Done false
31351 9904 Acked Read 8 Done Sent Done false
31352 9905 Acked Read 8 Done Sent Done false
31353 9906 Acked Write 9 Done Sent Done false
31354 9907 Acked Read 10 Done Sent Done false
31355 9908 Acked Read 1 Done Sent Done false
31356 9909 Acked Flush 0 Done Sent Done false
31357 9910 Acked Read 10 Done Sent Done false
31358 9911 Acked Read 9 Done Sent Done false
31359 9912 Acked Write 5 Done Sent Done false
31360 9913 Acked Write 7 Done Sent Done false
31361 9914 Acked Read 2 Done Sent Done false
31362 9915 Acked Write 6 Done Sent Done false
31363 9916 Acked Read 9 Done Sent Done false
31364 9917 Acked Write 9 Done Sent Done false
31365 9918 Acked Read 9 Done Sent Done false
31366 9919 Acked Write 4 Done Sent Done false
31367 9920 Acked Flush 0 Done Sent Done false
31368 9921 Acked Write 9 Done Sent Done false
31369 9922 Acked Write 2 Done Sent Done false
31370 9923 Acked Write 9 Done Sent Done false
31371 9924 Acked Read 5 Done Sent Done false
31372 9925 Acked Write 3 Done Sent Done false
31373 9926 Acked Write 9 Done Sent Done false
31374 9927 Acked Write 4 Done Sent Done false
31375 9928 Acked Read 7 Done Sent Done false
31376 9929 Acked Write 7 Done Sent Done false
31377 9930 Acked Write 5 Done Sent Done false
31378 9931 Acked Read 2 Done Sent Done false
31379 9932 Acked Write 10 Done Sent Done false
31380 9933 Acked Read 2 Done Sent Done false
31381 9934 Acked Write 10 Done Sent Done false
31382 9935 Acked Write 1 Done Sent Done false
31383 9936 Acked Read 9 Done Sent Done false
31384 9937 Acked Read 10 Done Sent Done false
31385 9938 Acked Read 3 Done Sent Done false
31386 9939 Acked Read 3 Done Sent Done false
31387 9940 Acked Write 3 Done Sent Done false
31388 9941 Acked Read 7 Done Sent Done false
31389 9942 Acked Read 2 Done Sent Done false
31390 9943 Acked Read 8 Done Sent Done false
31391 9944 Acked Write 2 Done Sent Done false
31392 9945 Acked Read 9 Done Sent Done false
31393 9946 Acked Read 4 Done Sent Done false
31394 9947 Acked Write 7 Done Sent Done false
31395 9948 Acked Read 7 Done Sent Done false
31396 9949 Acked Write 2 Done Sent Done false
31397 9950 Acked Write 6 Done Sent Done false
31398 9951 Acked Read 8 Done Sent Done false
31399 9952 Acked Read 6 Done Sent Done false
31400 9953 Acked Read 9 Done Sent Done false
31401 9954 Acked Read 7 Done Sent Done false
31402 9955 Acked Write 6 Done Sent Done false
31403 9956 Acked Write 6 Done Sent Done false
31404 9957 Acked Read 6 Done Sent Done false
31405 9958 Acked Write 6 Done Sent Done false
31406 9959 Acked Read 6 Done Sent Done false
31407 9960 Acked Read 5 Done Sent Done false
31408 9961 Acked Write 6 Done Sent Done false
31409 9962 Acked Read 10 Done Sent Done false
31410 9963 Acked Read 3 Done Sent Done false
31411 9964 Acked Flush 0 Done Sent Done false
31412 9965 Acked Read 3 Done Sent Done false
31413 9966 Acked Read 10 Done Sent Done false
31414 9967 Acked Flush 0 Done Sent Done false
31415 9968 Acked Write 4 Done Sent Done false
31416 9969 Acked Write 8 Done Sent Done false
31417 9970 Acked Write 6 Done Sent Done false
31418 9971 Acked Read 4 Done Sent Done false
31419 9972 Acked Read 8 Done Sent Done false
31420 9973 Acked Read 3 Done Sent Done false
31421 9974 Acked Read 7 Done Sent Done false
31422 9975 Acked Write 7 Done Sent Done false
31423 9976 Acked Read 6 Done Sent Done false
31424 9977 Acked Write 5 Done Sent Done false
31425 9978 Acked Flush 0 Done Sent Done false
31426 9979 Acked Write 6 Done Sent Done false
31427 9980 Acked Read 9 Done Sent Done false
31428 9981 Acked Write 2 Done Sent Done false
31429 9982 Acked Read 8 Done Sent Done false
31430 9983 Acked Read 4 Done Sent Done false
31431 9984 Acked Read 9 Done Sent Done false
31432 9985 Acked Read 8 Done Sent Done false
31433 9986 Acked Write 5 Done Sent Done false
31434 9987 Acked Read 5 Done Sent Done false
31435 9988 Acked Read 4 Done Sent Done false
31436 9989 Acked Read 6 Done Sent Done false
31437 9990 Acked Read 9 Done Sent Done false
31438 9991 Acked Read 7 Done Sent Done false
31439 9992 Acked Write 5 Done Sent Done false
31440 9993 Acked Read 8 Done Sent Done false
31441 9994 Acked Write 9 Done Sent Done false
31442 9995 Acked Write 8 Done Sent Done false
31443 9996 Acked Write 4 Done Sent Done false
31444 9997 Acked Read 3 Done Sent Done false
31445 9998 Acked Flush 0 Done Sent Done false
31446 9999 Acked Write 9 Done Sent Done false
31447 10000 Acked Flush 0 Done Sent Done false
31448 10001 Acked Read 7 Done Sent Done false
31449 10002 Acked Flush 0 Done Sent Done false
31450 10003 Acked Write 9 Done Sent Done false
31451 10004 Acked Read 8 Done Sent Done false
31452 10005 Acked Read 7 Done Sent Done false
31453 10006 Acked Read 5 Done Sent Done false
31454 10007 Acked Read 4 Done Sent Done false
31455 10008 Acked Flush 0 Done Sent Done false
31456 10009 Acked Read 10 Done Sent Done false
31457 10010 Acked Read 3 Done Sent Done false
31458 10011 Acked Write 3 Done Sent Done false
31459 10012 Acked Read 9 Done Sent Done false
31460 10013 Acked Flush 0 Done Sent Done false
31461 10014 Acked Flush 0 Done Sent Done false
31462 10015 Acked Read 6 Done Sent Done false
31463 10016 Acked Write 4 Done Sent Done false
31464 10017 Acked Read 10 Done Sent Done false
31465 10018 Acked Read 1 Done Sent Done false
31466 10019 Acked Read 9 Done Sent Done false
31467 10020 Acked Read 5 Done Sent Done false
31468 10021 Acked Write 8 Done Sent Done false
31469 10022 Acked Read 1 Done Sent Done false
31470 10023 Acked Read 3 Done Sent Done false
31471 10024 Acked Read 4 Done Sent Done false
31472 10025 Acked Read 6 Done Sent Done false
31473 10026 Acked Read 5 Done Sent Done false
31474 10027 Acked Read 10 Done Sent Done false
31475 10028 Acked Write 1 Done Sent Done false
31476 10029 Acked Write 4 Done Sent Done false
31477 10030 Acked Write 2 Done Sent Done false
31478 10031 Acked Write 5 Done Sent Done false
31479 10032 Acked Write 1 Done Sent Done false
31480 10033 Acked Read 8 Done Sent Done false
31481 10034 Acked Read 9 Done Sent Done false
31482 10035 Acked Read 8 Done Sent Done false
31483 10036 Acked Flush 0 Done Sent Done false
31484 10037 Acked Read 10 Done Sent Done false
31485 10038 Acked Write 4 Done Sent Done false
31486 10039 Acked Read 1 Done Sent Done false
31487 10040 Acked Flush 0 Done Sent Done false
31488 10041 Acked Write 8 Done Sent Done false
31489 10042 Acked Flush 0 Done Sent Done false
31490 10043 Acked Read 8 Done Sent Done false
31491 10044 Acked Write 6 Done Sent Done false
31492 10045 Acked Read 3 Done Sent Done false
31493 10046 Acked Read 10 Done Sent Done false
31494 10047 Acked Read 5 Done Sent Done false
31495 10048 Acked Read 8 Done Sent Done false
31496 10049 Acked Read 4 Done Sent Done false
31497 10050 Acked Write 9 Done Sent Done false
31498 10051 Acked Read 4 Done Sent Done false
31499 10052 Acked Write 9 Done Sent Done false
31500 10053 Acked Read 2 Done Sent Done false
31501 10054 Acked Write 6 Done Sent Done false
31502 10055 Acked Read 2 Done Sent Done false
31503 10056 Acked Write 7 Done Sent Done false
31504 10057 Acked Write 1 Done Sent Done false
31505 STATES DS:0 DS:1 DS:2 TOTAL
31506 Sent 0 301 0 301
31507 Done 301 0 301 602
31508 Skipped 0 0 0 0
31509 Error 0 0 0 0
31510 Last Flush: Some(JobId(10042)) Some(JobId(9756)) Some(JobId(10042))
31511 Downstairs last five completed: 9756 9755 9754 9753 9752
31512 Upstairs last five completed: 10057 10056 10055 10054 10053
31513 CLIENT: Up:0 ds:301 act:2
31514 [8820] got start action
31515 [8820] I am going to restart
31516 Starting downstairs at port 8820
31517 Make output file at "/tmp/dsc/downstairs-8820.txt"
31518 [8820][1] reports Starting
31519 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1235)
31520 [8820][1] reports Running
31521 ----------------------------------------------------------------
31522 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
31523 States: Active Offline Active
31524 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31525 9757 Acked Read 9 Done Sent Done false
31526 9758 Acked Read 7 Done Sent Done false
31527 9759 Acked Read 3 Done Sent Done false
31528 9760 Acked Read 10 Done Sent Done false
31529 9761 Acked Read 3 Done Sent Done false
31530 9762 Acked Read 2 Done Sent Done false
31531 9763 Acked Flush 0 Done Sent Done false
31532 9764 Acked Flush 0 Done Sent Done false
31533 9765 Acked Read 4 Done Sent Done false
31534 9766 Acked Write 2 Done Sent Done false
31535 9767 Acked Write 7 Done Sent Done false
31536 9768 Acked Write 8 Done Sent Done false
31537 9769 Acked Read 1 Done Sent Done false
31538 9770 Acked Read 4 Done Sent Done false
31539 9771 Acked Write 5 Done Sent Done false
31540 9772 Acked Write 5 Done Sent Done false
31541 9773 Acked Read 10 Done Sent Done false
31542 9774 Acked Write 4 Done Sent Done false
31543 9775 Acked Write 5 Done Sent Done false
31544 9776 Acked Write 2 Done Sent Done false
31545 9777 Acked Read 4 Done Sent Done false
31546 9778 Acked Read 8 Done Sent Done false
31547 9779 Acked Read 3 Done Sent Done false
31548 9780 Acked Write 1 Done Sent Done false
31549 9781 Acked Flush 0 Done Sent Done false
31550 9782 Acked Read 4 Done Sent Done false
31551 9783 Acked Read 7 Done Sent Done false
31552 9784 Acked Write 6 Done Sent Done false
31553 9785 Acked Read 9 Done Sent Done false
31554 9786 Acked Read 7 Done Sent Done false
31555 9787 Acked Read 1 Done Sent Done false
31556 9788 Acked Write 7 Done Sent Done false
31557 9789 Acked Write 9 Done Sent Done false
31558 9790 Acked Read 10 Done Sent Done false
31559 9791 Acked Read 2 Done Sent Done false
31560 9792 Acked Flush 0 Done Sent Done false
31561 9793 Acked Read 8 Done Sent Done false
31562 9794 Acked Write 6 Done Sent Done false
31563 9795 Acked Write 6 Done Sent Done false
31564 9796 Acked Write 5 Done Sent Done false
31565 9797 Acked Flush 0 Done Sent Done false
31566 9798 Acked Flush 0 Done Sent Done false
31567 9799 Acked Read 5 Done Sent Done false
31568 9800 Acked Read 8 Done Sent Done false
31569 9801 Acked Read 10 Done Sent Done false
31570 9802 Acked Read 7 Done Sent Done false
31571 9803 Acked Read 5 Done Sent Done false
31572 9804 Acked Write 3 Done Sent Done false
31573 9805 Acked Read 10 Done Sent Done false
31574 9806 Acked Write 7 Done Sent Done false
31575 9807 Acked Write 6 Done Sent Done false
31576 9808 Acked Read 6 Done Sent Done false
31577 9809 Acked Read 6 Done Sent Done false
31578 9810 Acked Write 9 Done Sent Done false
31579 9811 Acked Write 6 Done Sent Done false
31580 9812 Acked Write 1 Done Sent Done false
31581 9813 Acked Read 2 Done Sent Done false
31582 9814 Acked Read 8 Done Sent Done false
31583 9815 Acked Read 8 Done Sent Done false
31584 9816 Acked Write 6 Done Sent Done false
31585 9817 Acked Read 8 Done Sent Done false
31586 9818 Acked Read 3 Done Sent Done false
31587 9819 Acked Write 2 Done Sent Done false
31588 9820 Acked Read 7 Done Sent Done false
31589 9821 Acked Flush 0 Done Sent Done false
31590 9822 Acked Read 5 Done Sent Done false
31591 9823 Acked Write 1 Done Sent Done false
31592 9824 Acked Write 5 Done Sent Done false
31593 9825 Acked Read 6 Done Sent Done false
31594 9826 Acked Write 2 Done Sent Done false
31595 9827 Acked Write 8 Done Sent Done false
31596 9828 Acked Write 7 Done Sent Done false
31597 9829 Acked Write 3 Done Sent Done false
31598 9830 Acked Read 9 Done Sent Done false
31599 9831 Acked Read 3 Done Sent Done false
31600 9832 Acked Read 7 Done Sent Done false
31601 9833 Acked Read 7 Done Sent Done false
31602 9834 Acked Read 1 Done Sent Done false
31603 9835 Acked Write 2 Done Sent Done false
31604 9836 Acked Write 1 Done Sent Done false
31605 9837 Acked Read 1 Done Sent Done false
31606 9838 Acked Flush 0 Done Sent Done false
31607 9839 Acked Write 9 Done Sent Done false
31608 9840 Acked Read 4 Done Sent Done false
31609 9841 Acked Read 10 Done Sent Done false
31610 9842 Acked Write 7 Done Sent Done false
31611 9843 Acked Read 9 Done Sent Done false
31612 9844 Acked Read 7 Done Sent Done false
31613 9845 Acked Write 3 Done Sent Done false
31614 9846 Acked Read 2 Done Sent Done false
31615 9847 Acked Read 5 Done Sent Done false
31616 9848 Acked Write 2 Done Sent Done false
31617 9849 Acked Read 10 Done Sent Done false
31618 9850 Acked Flush 0 Done Sent Done false
31619 9851 Acked Write 9 Done Sent Done false
31620 9852 Acked Write 8 Done Sent Done false
31621 9853 Acked Read 6 Done Sent Done false
31622 9854 Acked Read 3 Done Sent Done false
31623 9855 Acked Write 7 Done Sent Done false
31624 9856 Acked Read 2 Done Sent Done false
31625 9857 Acked Read 5 Done Sent Done false
31626 9858 Acked Flush 0 Done Sent Done false
31627 9859 Acked Write 3 Done Sent Done false
31628 9860 Acked Write 1 Done Sent Done false
31629 9861 Acked Write 4 Done Sent Done false
31630 9862 Acked Read 2 Done Sent Done false
31631 9863 Acked Write 6 Done Sent Done false
31632 9864 Acked Write 10 Done Sent Done false
31633 9865 Acked Write 7 Done Sent Done false
31634 9866 Acked Read 1 Done Sent Done false
31635 9867 Acked Read 7 Done Sent Done false
31636 9868 Acked Write 4 Done Sent Done false
31637 9869 Acked Read 5 Done Sent Done false
31638 9870 Acked Read 1 Done Sent Done false
31639 9871 Acked Read 9 Done Sent Done false
31640 9872 Acked Read 4 Done Sent Done false
31641 9873 Acked Read 7 Done Sent Done false
31642 9874 Acked Write 9 Done Sent Done false
31643 9875 Acked Write 7 Done Sent Done false
31644 9876 Acked Write 9 Done Sent Done false
31645 9877 Acked Write 8 Done Sent Done false
31646 9878 Acked Read 1 Done Sent Done false
31647 9879 Acked Write 7 Done Sent Done false
31648 9880 Acked Write 1 Done Sent Done false
31649 9881 Acked Read 4 Done Sent Done false
31650 9882 Acked Flush 0 Done Sent Done false
31651 9883 Acked Write 8 Done Sent Done false
31652 9884 Acked Read 4 Done Sent Done false
31653 9885 Acked Write 3 Done Sent Done false
31654 9886 Acked Write 4 Done Sent Done false
31655 9887 Acked Write 4 Done Sent Done false
31656 9888 Acked Read 9 Done Sent Done false
31657 9889 Acked Write 7 Done Sent Done false
31658 9890 Acked Flush 0 Done Sent Done false
31659 9891 Acked Read 8 Done Sent Done false
31660 9892 Acked Read 9 Done Sent Done false
31661 9893 Acked Write 10 Done Sent Done false
31662 9894 Acked Write 1 Done Sent Done false
31663 9895 Acked Write 2 Done Sent Done false
31664 9896 Acked Read 10 Done Sent Done false
31665 9897 Acked Read 6 Done Sent Done false
31666 9898 Acked Read 1 Done Sent Done false
31667 9899 Acked Write 5 Done Sent Done false
31668 9900 Acked Read 7 Done Sent Done false
31669 9901 Acked Write 5 Done Sent Done false
31670 9902 Acked Read 7 Done Sent Done false
31671 9903 Acked Read 8 Done Sent Done false
31672 9904 Acked Read 8 Done Sent Done false
31673 9905 Acked Read 8 Done Sent Done false
31674 9906 Acked Write 9 Done Sent Done false
31675 9907 Acked Read 10 Done Sent Done false
31676 9908 Acked Read 1 Done Sent Done false
31677 9909 Acked Flush 0 Done Sent Done false
31678 9910 Acked Read 10 Done Sent Done false
31679 9911 Acked Read 9 Done Sent Done false
31680 9912 Acked Write 5 Done Sent Done false
31681 9913 Acked Write 7 Done Sent Done false
31682 9914 Acked Read 2 Done Sent Done false
31683 9915 Acked Write 6 Done Sent Done false
31684 9916 Acked Read 9 Done Sent Done false
31685 9917 Acked Write 9 Done Sent Done false
31686 9918 Acked Read 9 Done Sent Done false
31687 9919 Acked Write 4 Done Sent Done false
31688 9920 Acked Flush 0 Done Sent Done false
31689 9921 Acked Write 9 Done Sent Done false
31690 9922 Acked Write 2 Done Sent Done false
31691 9923 Acked Write 9 Done Sent Done false
31692 9924 Acked Read 5 Done Sent Done false
31693 9925 Acked Write 3 Done Sent Done false
31694 9926 Acked Write 9 Done Sent Done false
31695 9927 Acked Write 4 Done Sent Done false
31696 9928 Acked Read 7 Done Sent Done false
31697 9929 Acked Write 7 Done Sent Done false
31698 9930 Acked Write 5 Done Sent Done false
31699 9931 Acked Read 2 Done Sent Done false
31700 9932 Acked Write 10 Done Sent Done false
31701 9933 Acked Read 2 Done Sent Done false
31702 9934 Acked Write 10 Done Sent Done false
31703 9935 Acked Write 1 Done Sent Done false
31704 9936 Acked Read 9 Done Sent Done false
31705 9937 Acked Read 10 Done Sent Done false
31706 9938 Acked Read 3 Done Sent Done false
31707 9939 Acked Read 3 Done Sent Done false
31708 9940 Acked Write 3 Done Sent Done false
31709 9941 Acked Read 7 Done Sent Done false
31710 9942 Acked Read 2 Done Sent Done false
31711 9943 Acked Read 8 Done Sent Done false
31712 9944 Acked Write 2 Done Sent Done false
31713 9945 Acked Read 9 Done Sent Done false
31714 9946 Acked Read 4 Done Sent Done false
31715 9947 Acked Write 7 Done Sent Done false
31716 9948 Acked Read 7 Done Sent Done false
31717 9949 Acked Write 2 Done Sent Done false
31718 9950 Acked Write 6 Done Sent Done false
31719 9951 Acked Read 8 Done Sent Done false
31720 9952 Acked Read 6 Done Sent Done false
31721 9953 Acked Read 9 Done Sent Done false
31722 9954 Acked Read 7 Done Sent Done false
31723 9955 Acked Write 6 Done Sent Done false
31724 9956 Acked Write 6 Done Sent Done false
31725 9957 Acked Read 6 Done Sent Done false
31726 9958 Acked Write 6 Done Sent Done false
31727 9959 Acked Read 6 Done Sent Done false
31728 9960 Acked Read 5 Done Sent Done false
31729 9961 Acked Write 6 Done Sent Done false
31730 9962 Acked Read 10 Done Sent Done false
31731 9963 Acked Read 3 Done Sent Done false
31732 9964 Acked Flush 0 Done Sent Done false
31733 9965 Acked Read 3 Done Sent Done false
31734 9966 Acked Read 10 Done Sent Done false
31735 9967 Acked Flush 0 Done Sent Done false
31736 9968 Acked Write 4 Done Sent Done false
31737 9969 Acked Write 8 Done Sent Done false
31738 9970 Acked Write 6 Done Sent Done false
31739 9971 Acked Read 4 Done Sent Done false
31740 9972 Acked Read 8 Done Sent Done false
31741 9973 Acked Read 3 Done Sent Done false
31742 9974 Acked Read 7 Done Sent Done false
31743 9975 Acked Write 7 Done Sent Done false
31744 9976 Acked Read 6 Done Sent Done false
31745 9977 Acked Write 5 Done Sent Done false
31746 9978 Acked Flush 0 Done Sent Done false
31747 9979 Acked Write 6 Done Sent Done false
31748 9980 Acked Read 9 Done Sent Done false
31749 9981 Acked Write 2 Done Sent Done false
31750 9982 Acked Read 8 Done Sent Done false
31751 9983 Acked Read 4 Done Sent Done false
31752 9984 Acked Read 9 Done Sent Done false
31753 9985 Acked Read 8 Done Sent Done false
31754 9986 Acked Write 5 Done Sent Done false
31755 9987 Acked Read 5 Done Sent Done false
31756 9988 Acked Read 4 Done Sent Done false
31757 9989 Acked Read 6 Done Sent Done false
31758 9990 Acked Read 9 Done Sent Done false
31759 9991 Acked Read 7 Done Sent Done false
31760 9992 Acked Write 5 Done Sent Done false
31761 9993 Acked Read 8 Done Sent Done false
31762 9994 Acked Write 9 Done Sent Done false
31763 9995 Acked Write 8 Done Sent Done false
31764 9996 Acked Write 4 Done Sent Done false
31765 9997 Acked Read 3 Done Sent Done false
31766 9998 Acked Flush 0 Done Sent Done false
31767 9999 Acked Write 9 Done Sent Done false
31768 10000 Acked Flush 0 Done Sent Done false
31769 10001 Acked Read 7 Done Sent Done false
31770 10002 Acked Flush 0 Done Sent Done false
31771 10003 Acked Write 9 Done Sent Done false
31772 10004 Acked Read 8 Done Sent Done false
31773 10005 Acked Read 7 Done Sent Done false
31774 10006 Acked Read 5 Done Sent Done false
31775 10007 Acked Read 4 Done Sent Done false
31776 10008 Acked Flush 0 Done Sent Done false
31777 10009 Acked Read 10 Done Sent Done false
31778 10010 Acked Read 3 Done Sent Done false
31779 10011 Acked Write 3 Done Sent Done false
31780 10012 Acked Read 9 Done Sent Done false
31781 10013 Acked Flush 0 Done Sent Done false
31782 10014 Acked Flush 0 Done Sent Done false
31783 10015 Acked Read 6 Done Sent Done false
31784 10016 Acked Write 4 Done Sent Done false
31785 10017 Acked Read 10 Done Sent Done false
31786 10018 Acked Read 1 Done Sent Done false
31787 10019 Acked Read 9 Done Sent Done false
31788 10020 Acked Read 5 Done Sent Done false
31789 10021 Acked Write 8 Done Sent Done false
31790 10022 Acked Read 1 Done Sent Done false
31791 10023 Acked Read 3 Done Sent Done false
31792 10024 Acked Read 4 Done Sent Done false
31793 10025 Acked Read 6 Done Sent Done false
31794 10026 Acked Read 5 Done Sent Done false
31795 10027 Acked Read 10 Done Sent Done false
31796 10028 Acked Write 1 Done Sent Done false
31797 10029 Acked Write 4 Done Sent Done false
31798 10030 Acked Write 2 Done Sent Done false
31799 10031 Acked Write 5 Done Sent Done false
31800 10032 Acked Write 1 Done Sent Done false
31801 10033 Acked Read 8 Done Sent Done false
31802 10034 Acked Read 9 Done Sent Done false
31803 10035 Acked Read 8 Done Sent Done false
31804 10036 Acked Flush 0 Done Sent Done false
31805 10037 Acked Read 10 Done Sent Done false
31806 10038 Acked Write 4 Done Sent Done false
31807 10039 Acked Read 1 Done Sent Done false
31808 10040 Acked Flush 0 Done Sent Done false
31809 10041 Acked Write 8 Done Sent Done false
31810 10042 Acked Flush 0 Done Sent Done false
31811 10043 Acked Read 8 Done Sent Done false
31812 10044 Acked Write 6 Done Sent Done false
31813 10045 Acked Read 3 Done Sent Done false
31814 10046 Acked Read 10 Done Sent Done false
31815 10047 Acked Read 5 Done Sent Done false
31816 10048 Acked Read 8 Done Sent Done false
31817 10049 Acked Read 4 Done Sent Done false
31818 10050 Acked Write 9 Done Sent Done false
31819 10051 Acked Read 4 Done Sent Done false
31820 10052 Acked Write 9 Done Sent Done false
31821 10053 Acked Read 2 Done Sent Done false
31822 10054 Acked Write 6 Done Sent Done false
31823 10055 Acked Read 2 Done Sent Done false
31824 10056 Acked Write 7 Done Sent Done false
31825 10057 Acked Write 1 Done Sent Done false
31826 10058 Acked Flush 0 Done Sent Done false
31827 STATES DS:0 DS:1 DS:2 TOTAL
31828 Sent 0 302 0 302
31829 Done 302 0 302 604
31830 Skipped 0 0 0 0
31831 Error 0 0 0 0
31832 Last Flush: Some(JobId(10058)) Some(JobId(9756)) Some(JobId(10058))
31833 Downstairs last five completed: 9756 9755 9754 9753 9752
31834 Upstairs last five completed: 10058 10057 10056 10055 10054
31835 CLIENT: Up:0 ds:302 act:2
318362025-01-23T17:44:24.950ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
318372025-01-23T17:44:24.951ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41137) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
318382025-01-23T17:44:24.952ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
318392025-01-23T17:44:24.952ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
318402025-01-23T17:44:24.953ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
318412025-01-23T17:44:24.953ZINFOcrucible: send last flush ID to this DS: Some(JobId(9756)) = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
318422025-01-23T17:44:24.953ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
318432025-01-23T17:44:24.953ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(9756)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
318442025-01-23T17:44:24.953ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(9756)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
31845 ----------------------------------------------------------------
31846 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
31847 Downstairs last five completed: 10058 10057 10056 10055 10054
31848 Upstairs last five completed: 10058 10057 10056 10055 10054
31849 CLIENT: Up:0 ds:0 act:3
31850 Replay: All jobs finished
31851 Main task has work to do, go find it
31852 got dsc Stop(1)
31853 stop 1
31854 [8820] Got stop action so:false kr:false
31855 Jan 23 17:44:27.212 INFO request completed, latency_us: 338, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 37c4407c-9a49-4e88-9db4-c09b1ae4acd0, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
31856 Jan 23 17:44:27.213{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:44:27.213227434Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
31857 INFO request completed, latency_us: 124, response_code: 200, uri: /state/cid/1, method: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:44:27.213326904Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
31858 GET, req_id: 671b92c5-f03c-427e-91c0-a7cf9d445af8, remote_addr: 127.0{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-23T17:44:27.213397797Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1205,"session_id":"14e5020b-a7a5-48d4-836f-67e5604bd35c"}
31859 .0.1:42015, local_addr: 127.0.0.1:9998
318602025-01-23T17:44:27.213ZINFOcrucible: client task is exiting = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
31861 [8820] Exited with: ExitStatus(unix_wait_status(9))
31862 [8820][1] reports Exit
31863 Jan 23 17:44:31.216 INFO request completed, latency_us: 335, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6a39eefd-6a3b-43f3-9657-6ced17511808, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
31864 001/300 Write block 4253 len 36864 data: 3 5 5 7 6 6 6 5 6
31865 002/300 Flush
31866 003/300 Write block 4770 len 28672 data: 6 7 6 8 8 7 7
31867 004/300 Read block 968 len 36864
31868 005/300 Read block 3454 len 12288
31869 006/300 Write block 64 len 16384 data: 8 7 6 9
31870 007/300 Write block 1618 len 40960 data: 11 11 9 8 5 6 3 5 6 6
31871 008/300 Read block 2509 len 4096
31872 009/300 Write block 2118 len 20480 data: 7 5 4 3 3
31873 010/300 Write block 3701 len 20480 data: 8 8 9 10 8
31874 011/300 Flush
31875 012/300 Write block 2424 len 32768 data: 7 6 7 6 6 8 7 7
31876 013/300 Write block 4578 len 8192 data: 10 9
31877 014/300 Flush
31878 015/300 Read block 831 len 4096
31879 016/300 Flush
31880 017/300 Write block 2677 len 32768 data: 5 5 5 3 3 3 5 9
31881 018/300 Read block 1812 len 12288
31882 019/300 Read block 2239 len 40960
31883 020/300 Read block 4615 len 4096
31884 021/300 Read block 2612 len 16384
31885 022/300 Write block 3527 len 28672 data: 5 4 5 6 4 5 4
31886 023/300 Read block 3241 len 24576
31887 024/300 Read block 3674 len 20480
31888 025/300 Write block 1575 len 28672 data: 2 2 4 2 4 4 5
31889 026/300 Read block 2704 len 12288
31890 027/300 Read block 1141 len 28672
31891 028/300 Read block 3272 len 16384
31892 029/300 Write block 3761 len 8192 data: 3 4
31893 030/300 Write block 4087 len 20480 data: 2 2 4 4 4
31894 031/300 Write block 3851 len 32768 data: 8 7 7 7 8 5 6 5
31895 032/300 Read block 3013 len 40960
31896 033/300 Write block 831 len 16384 data: 6 8 9 8
31897 034/300 Write block 1422 len 20480 data: 7 8 6 5 7
31898 035/300 Write block 4805 len 20480 data: 4 3 2 3 5
31899 036/300 Read block 3830 len 20480
31900 037/300 Read block 2006 len 24576
31901 038/300 Read block 647 len 40960
31902 039/300 Read block 1109 len 8192
31903 040/300 Read block 3251 len 8192
31904 041/300 Write block 2730 len 28672 data: 10 11 8 8 5 5 4
31905 042/300 Read block 1619 len 36864
31906 043/300 Read block 2951 len 32768
31907 044/300 Read block 4439 len 20480
31908 045/300 Read block 2270 len 40960
31909 046/300 Write block 1044 len 4096 data: 7
31910 047/300 Write block 688 len 20480 data: 3 3 4 3 2
31911 048/300 Read block 922 len 12288
31912 049/300 Read block 57 len 24576
31913 050/300 Write block 1155 len 20480 data: 6 8 8 9 10
31914 051/300 Read block 2544 len 16384
31915 052/300 Write block 3349 len 24576 data: 5 4 4 6 7 6
31916 053/300 Read block 3963 len 12288
31917 054/300 Read block 2220 len 40960
31918 055/300 Read block 808 len 16384
31919 056/300 Write block 1437 len 8192 data: 5 5
31920 057/300 Write block 4656 len 16384 data: 5 5 5 5
31921 058/300 Write block 1651 len 4096 data: 8
31922 059/300 Write block 198 len 20480 data: 6 6 6 7 6
31923 060/300 Read block 4863 len 8192
31924 061/300 Write block 4985 len 36864 data: 4 7 5 5 4 6 5 4 6
31925 062/300 Flush
31926 063/300 Read block 284 len 24576
31927 064/300 Read block 3149 len 8192
31928 065/300 Read block 337 len 16384
31929 066/300 Read block 1896 len 32768
31930 067/300 Write block 4584 len 8192 data: 6 7
31931 068/300 Write block 694 len 4096 data: 3
31932 069/300 Write block 4634 len 8192 data: 6 5
31933 070/300 Read block 682 len 4096
31934 071/300 Write block 2076 len 4096 data: 8
31935 072/300 Write block 1479 len 28672 data: 7 6 5 4 4 4 4
31936 073/300 Write block 3149 len 4096 data: 6
31937 074/300 Read block 3886 len 32768
31938 075/300 Read block 2138 len 40960
31939 076/300 Write block 159 len 36864 data: 7 7 8 7 7 6 7 6 7
31940 077/300 Flush
31941 078/300 Write block 1592 len 4096 data: 5
31942 079/300 Write block 4000 len 40960 data: 9 6 5 4 4 4 3 3 5 6
31943 080/300 Read block 2244 len 24576
31944 081/300 Read block 4156 len 28672
31945 082/300 Flush
31946 083/300 Read block 2337 len 24576
31947 084/300 Flush
31948 085/300 Write block 3226 len 8192 data: 7 4
31949 086/300 Read block 3710 len 32768
31950 087/300 Write block 3214 len 8192 data: 4 4
31951 088/300 Read block 304 len 28672
31952 089/300 Flush
31953 090/300 Read block 4900 len 28672
31954 091/300 Flush
31955 092/300 Read block 1436 len 8192
31956 093/300 Read block 2090 len 28672
31957 094/300 Write block 4487 len 28672 data: 7 5 2 2 2 2 3
31958 095/300 Read block 2179 len 40960
31959 096/300 Read block 4674 len 36864
31960 097/300 Write block 2004 len 24576 data: 7 6 5 4 4 5
31961 098/300 Read block 601 len 4096
31962 099/300 Read block 4064 len 12288
31963 100/300 Read block 3126 len 16384
31964 101/300 Read block 2732 len 24576
31965 102/300 Write block 985 len 40960 data: 8 9 8 8 10 12 11 11 7 6
31966 103/300 Flush
31967 104/300 Write block 2748 len 28672 data: 10 9 10 9 9 6 6
31968 105/300 Read block 4059 len 4096
31969 106/300 Read block 622 len 32768
31970 107/300 Write block 4128 len 16384 data: 3 4 4 5
31971 108/300 Write block 2748 len 32768 data: 11 10 11 10 10 7 7 6
31972 109/300 Read block 1028 len 16384
31973 110/300 Write block 1232 len 32768 data: 6 6 6 7 9 8 6 6
31974 111/300 Write block 32 len 32768 data: 3 4 6 4 5 5 6 6
31975 112/300 Read block 311 len 36864
31976 113/300 Write block 2639 len 20480 data: 6 6 4 4 5
31977 114/300 Write block 3902 len 8192 data: 5 5
31978 115/300 Read block 2368 len 20480
31979 116/300 Write block 650 len 4096 data: 7
31980 117/300 Read block 892 len 20480
31981 118/300 Read block 2682 len 20480
31982 119/300 Write block 2969 len 32768 data: 5 5 7 8 8 8 7 6
31983 120/300 Flush
31984 121/300 Read block 2023 len 28672
31985 122/300 Write block 942 len 36864 data: 6 5 6 6 7 7 7 7 7
31986 123/300 Read block 3195 len 36864
31987 124/300 Read block 3859 len 32768
31988 125/300 Read block 4578 len 40960
31989 126/300 Read block 2925 len 32768
31990 127/300 Write block 4009 len 36864 data: 7 6 7 6 5 6 6 6 4
31991 128/300 Read block 1016 len 16384
31992 129/300 Write block 4957 len 32768 data: 4 5 5 3 4 4 3 3
31993 130/300 Write block 605 len 40960 data: 5 5 6 5 5 6 6 6 7 6
31994 131/300 Read block 1724 len 36864
31995 132/300 Write block 4310 len 40960 data: 3 5 6 6 6 7 11 7 8 8
31996 133/300 Flush
31997 134/300 Read block 394 len 16384
31998 135/300 Read block 2121 len 36864
31999 136/300 Write block 3128 len 8192 data: 3 3
32000 137/300 Write block 4438 len 16384 data: 7 6 7 8
32001 138/300 Read block 926 len 28672
32002 139/300 Read block 2847 len 4096
32003 140/300 Read block 2811 len 16384
32004 141/300 Read block 2090 len 12288
32005 142/300 Write block 3352 len 24576 data: 7 8 7 6 6 5
32006 143/300 Read block 1891 len 4096
32007 144/300 Read block 4514 len 32768
32008 145/300 Read block 2054 len 8192
32009 146/300 Write block 1519 len 32768 data: 4 5 5 5 5 4 3 3
32010 147/300 Read block 205 len 12288
32011 148/300 Read block 2709 len 28672
32012 149/300 Write block 3653 len 16384 data: 13 10 10 8
32013 150/300 Flush
32014 151/300 Write block 685 len 4096 data: 5
32015 152/300 Write block 690 len 32768 data: 5 4 3 3 4 4 6 7
32016 153/300 Write block 61 len 20480 data: 5 7 7 9 8
32017 154/300 Read block 3100 len 36864
32018 155/300 Read block 1957 len 8192
32019 156/300 Write block 1000 len 4096 data: 6
32020 157/300 Write block 3368 len 20480 data: 3 5 6 7 4
32021 158/300 Read block 802 len 28672
32022 159/300 Read block 4062 len 24576
32023 160/300 Write block 3411 len 12288 data: 7 6 6
32024 161/300 Read block 854 len 8192
32025 162/300 Write block 1262 len 40960 data: 6 6 5 5 4 3 4 4 5 4
32026 163/300 Read block 2505 len 32768
32027 164/300 Read block 2817 len 32768
32028 165/300 Flush
32029 166/300 Write block 4904 len 20480 data: 4 4 3 4 5
32030 167/300 Write block 2452 len 8192 data: 3 5
32031 168/300 Write block 4655 len 24576 data: 5 6 6 6 6 6
32032 169/300 Flush
32033 170/300 Read block 3629 len 24576
32034 171/300 Flush
32035 172/300 Read block 4779 len 24576
32036 173/300 Flush
32037 174/300 Read block 3889 len 40960
32038 175/300 Read block 2990 len 28672
32039 176/300 Read block 1157 len 4096
32040 177/300 Read block 4890 len 36864
32041 178/300 Write block 3271 len 24576 data: 5 5 5 3 5 4
32042 179/300 Read block 3276 len 20480
32043 180/300 Read block 1596 len 12288
32044 181/300 Write block 1666 len 16384 data: 6 4 4 2
32045 182/300 Write block 152 len 16384 data: 5 7 7 7
32046 183/300 Flush
32047 184/300 Read block 3883 len 16384
32048 185/300 Flush
32049 186/300 Flush
32050 187/300 Write block 1218 len 40960 data: 5 6 6 7 7 6 4 3 2 3
32051 188/300 Write block 1897 len 16384 data: 8 8 8 8
32052 189/300 Read block 1417 len 24576
32053 190/300 Write block 4965 len 8192 data: 3 3
32054 191/300 Read block 4236 len 32768
32055 192/300 Read block 13 len 8192
32056 193/300 Write block 2124 len 20480 data: 5 5 5 6 7
32057 194/300 Write block 1945 len 16384 data: 6 7 7 7
32058 195/300 Read block 2703 len 8192
32059 196/300 Flush
32060 197/300 Write block 4063 len 12288 data: 6 6 4
32061 198/300 Read block 4676 len 8192
32062 199/300 Read block 1784 len 28672
32063 200/300 Read block 1619 len 24576
32064 201/300 Read block 221 len 12288
32065 202/300 Write block 2808 len 32768 data: 8 6 7 11 10 11 9 9
32066 203/300 Flush
32067 204/300 Read block 2316 len 32768
32068 205/300 Read block 246 len 16384
32069 206/300 Read block 1227 len 12288
32070 207/300 Read block 4262 len 32768
32071 208/300 Read block 428 len 28672
32072 209/300 Flush
32073 210/300 Write block 3577 len 40960 data: 6 8 10 10 11 11 10 10 9 7
32074 211/300 Read block 1291 len 16384
32075 212/300 Write block 2344 len 40960 data: 6 6 6 6 8 8 9 11 7 8
32076 213/300 Read block 1520 len 16384
32077 214/300 Read block 4986 len 24576
32078 215/300 Flush
32079 216/300 Read block 3017 len 40960
32080 217/300 Flush
32081 218/300 Read block 3873 len 16384
32082 219/300 Read block 4626 len 32768
32083 220/300 Read block 4541 len 28672
32084 221/300 Write block 3895 len 16384 data: 8 7 6 4
32085 222/300 Write block 401 len 8192 data: 4 3
32086 223/300 Read block 669 len 36864
32087 224/300 Write block 642 len 16384 data: 12 12 11 12
32088 225/300 Flush
32089 226/300 Read block 4787 len 36864
32090 227/300 Read block 2686 len 36864
32091 228/300 Write block 941 len 24576 data: 5 7 6 7 7 8
32092 229/300 Flush
32093 230/300 Read block 3499 len 16384
32094 231/300 Read block 3354 len 40960
32095 232/300 Write block 1342 len 24576 data: 3 3 4 4 6 7
32096 233/300 Write block 4517 len 4096 data: 7
32097 234/300 Read block 1326 len 24576
32098 235/300 Write block 3896 len 16384 data: 8 7 5 4
32099 236/300 Read block 4419 len 16384
32100 237/300 Read block 4325 len 24576
32101 238/300 Read block 730 len 12288
32102 239/300 Write block 3913 len 32768 data: 8 8 6 6 8 7 7 9
32103 240/300 Read block 3210 len 20480
32104 241/300 Read block 1012 len 8192
32105 242/300 Read block 1320 len 4096
32106 243/300 Read block 2890 len 40960
32107 244/300 Write block 4652 len 24576 data: 5 5 6 6 7 7
32108 245/300 Read block 4020 len 12288
32109 246/300 Read block 2927 len 36864
32110 247/300 Flush
32111 248/300 Read block 3170 len 12288
32112 249/300 Read block 2447 len 16384
32113 250/300 Write block 110 len 24576 data: 10 9 9 7 7 7
32114 251/300 Read block 821 len 36864
32115 252/300 Read block 1752 len 28672
32116 253/300 Write block 1132 len 4096 data: 4
32117 254/300 Read block 313 len 12288
32118 255/300 Read block 3752 len 32768
32119 256/300 Write block 1831 len 20480 data: 5 4 6 9 9
32120 257/300 Read block 1719 len 8192
32121 258/300 Read block 4641 len 40960
32122 259/300 Read block 4427 len 4096
32123 260/300 Write block 4571 len 16384 data: 8 6 6 6
32124 261/300 Read block 2836 len 8192
32125 262/300 Read block 3245 len 16384
32126 263/300 Read block 4073 len 28672
32127 264/300 Write block 2175 len 24576 data: 5 4 4 5 5 6
32128 265/300 Read block 4170 len 20480
32129 266/300 Read block 2382 len 12288
32130 267/300 Read block 4160 len 40960
32131 268/300 Read block 2019 len 28672
32132 269/300 Write block 3994 len 24576 data: 5 5 5 7 8 8
32133 270/300 Read block 1626 len 8192
32134 271/300 Read block 896 len 36864
32135 272/300 Write block 3639 len 20480 data: 6 7 6 7 7
32136 273/300 Read block 3787 len 40960
32137 274/300 Write block 4663 len 40960 data: 6 5 6 5 6 6 5 8 8 8
32138 275/300 Read block 489 len 32768
32139 276/300 Read block 2097 len 36864
32140 277/300 Write block 2887 len 36864 data: 6 6 6 4 4 5 5 6 6
32141 278/300 Flush
32142 279/300 Read block 4213 len 32768
32143 280/300 Read block 2663 len 32768
32144 281/300 Read block 1772 len 24576
32145 282/300 Write block 4508 len 8192 data: 7 8
32146 283/300 Read block 2486 len 4096
32147 284/300 Flush
32148 285/300 Write block 2647 len 16384 data: 5 4 3 3
32149 286/300 Read block 3847 len 28672
32150 287/300 Read block 817 len 4096
32151 288/300 Write block 3133 len 12288 data: 7 7 7
32152 289/300 Read block 3758 len 24576
32153 290/300 Read block 1954 len 32768
32154 291/300 Write block 3382 len 32768 data: 3 5 4 6 6 4 4 3
32155 292/300 Write block 3706 len 12288 data: 10 10 9
32156 293/300 Read block 170 len 32768
32157 294/300 Flush
32158 295/300 Write block 556 len 12288 data: 11 9 7
32159 296/300 Read block 708 len 16384
32160 297/300 Read block 688 len 4096
32161 298/300 Read block 1547 len 32768
32162 299/300 Read block 2763 len 4096
32163 300/300 Read block 144 len 36864
32164 301/300 Read block 724 len 36864
32165 Main task has work to do, go find it
32166 got dsc Start(1)
32167 start 1
32168 Jan 23 17:44:31.472 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/1, method: POST, req_id: 6fa8a908-3e69-4260-a1b4-4b1352133c1f, remote_addr: 127.0.0.1:42015, local_addr: 127.0.0.1:9998
32169 Replay: started 1, returned:Ok(())
32170 ----------------------------------------------------------------
32171 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32172 States: Active Offline Active
32173 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32174 10059 Acked Write 9 Done Sent Done false
32175 10060 Acked Flush 0 Done Sent Done false
32176 10061 Acked Write 7 Done Sent Done false
32177 10062 Acked Read 9 Done Sent Done false
32178 10063 Acked Read 3 Done Sent Done false
32179 10064 Acked Write 4 Done Sent Done false
32180 10065 Acked Write 10 Done Sent Done false
32181 10066 Acked Read 1 Done Sent Done false
32182 10067 Acked Write 5 Done Sent Done false
32183 10068 Acked Write 5 Done Sent Done false
32184 10069 Acked Flush 0 Done Sent Done false
32185 10070 Acked Write 8 Done Sent Done false
32186 10071 Acked Write 2 Done Sent Done false
32187 10072 Acked Flush 0 Done Sent Done false
32188 10073 Acked Read 1 Done Sent Done false
32189 10074 Acked Flush 0 Done Sent Done false
32190 10075 Acked Write 8 Done Sent Done false
32191 10076 Acked Read 3 Done Sent Done false
32192 10077 Acked Read 10 Done Sent Done false
32193 10078 Acked Read 1 Done Sent Done false
32194 10079 Acked Read 4 Done Sent Done false
32195 10080 Acked Write 7 Done Sent Done false
32196 10081 Acked Read 6 Done Sent Done false
32197 10082 Acked Read 5 Done Sent Done false
32198 10083 Acked Write 7 Done Sent Done false
32199 10084 Acked Read 3 Done Sent Done false
32200 10085 Acked Read 7 Done Sent Done false
32201 10086 Acked Read 4 Done Sent Done false
32202 10087 Acked Write 2 Done Sent Done false
32203 10088 Acked Write 5 Done Sent Done false
32204 10089 Acked Write 8 Done Sent Done false
32205 10090 Acked Read 10 Done Sent Done false
32206 10091 Acked Write 4 Done Sent Done false
32207 10092 Acked Write 5 Done Sent Done false
32208 10093 Acked Write 5 Done Sent Done false
32209 10094 Acked Read 5 Done Sent Done false
32210 10095 Acked Read 6 Done Sent Done false
32211 10096 Acked Read 10 Done Sent Done false
32212 10097 Acked Read 2 Done Sent Done false
32213 10098 Acked Read 2 Done Sent Done false
32214 10099 Acked Write 7 Done Sent Done false
32215 10100 Acked Read 9 Done Sent Done false
32216 10101 Acked Read 8 Done Sent Done false
32217 10102 Acked Read 5 Done Sent Done false
32218 10103 Acked Read 10 Done Sent Done false
32219 10104 Acked Write 1 Done Sent Done false
32220 10105 Acked Write 5 Done Sent Done false
32221 10106 Acked Read 3 Done Sent Done false
32222 10107 Acked Read 6 Done Sent Done false
32223 10108 Acked Write 5 Done Sent Done false
32224 10109 Acked Read 4 Done Sent Done false
32225 10110 Acked Write 6 Done Sent Done false
32226 10111 Acked Read 3 Done Sent Done false
32227 10112 Acked Read 10 Done Sent Done false
32228 10113 Acked Read 4 Done Sent Done false
32229 10114 Acked Write 2 Done Sent Done false
32230 10115 Acked Write 4 Done Sent Done false
32231 10116 Acked Write 1 Done Sent Done false
32232 10117 Acked Write 5 Done Sent Done false
32233 10118 Acked Read 2 Done Sent Done false
32234 10119 Acked Write 9 Done Sent Done false
32235 10120 Acked Flush 0 Done Sent Done false
32236 10121 Acked Read 6 Done Sent Done false
32237 10122 Acked Read 2 Done Sent Done false
32238 10123 Acked Read 4 Done Sent Done false
32239 10124 Acked Read 8 Done Sent Done false
32240 10125 Acked Write 2 Done Sent Done false
32241 10126 Acked Write 1 Done Sent Done false
32242 10127 Acked Write 2 Done Sent Done false
32243 10128 Acked Read 1 Done Sent Done false
32244 10129 Acked Write 1 Done Sent Done false
32245 10130 Acked Write 7 Done Sent Done false
32246 10131 Acked Write 1 Done Sent Done false
32247 10132 Acked Read 8 Done Sent Done false
32248 10133 Acked Read 10 Done Sent Done false
32249 10134 Acked Write 9 Done Sent Done false
32250 10135 Acked Flush 0 Done Sent Done false
32251 10136 Acked Write 1 Done Sent Done false
32252 10137 Acked Write 10 Done Sent Done false
32253 10138 Acked Read 6 Done Sent Done false
32254 10139 Acked Read 7 Done Sent Done false
32255 10140 Acked Flush 0 Done Sent Done false
32256 10141 Acked Read 6 Done Sent Done false
32257 10142 Acked Flush 0 Done Sent Done false
32258 10143 Acked Write 2 Done Sent Done false
32259 10144 Acked Read 8 Done Sent Done false
32260 10145 Acked Write 2 Done Sent Done false
32261 10146 Acked Read 7 Done Sent Done false
32262 10147 Acked Flush 0 Done Sent Done false
32263 10148 Acked Read 7 Done Sent Done false
32264 10149 Acked Flush 0 Done Sent Done false
32265 10150 Acked Read 2 Done Sent Done false
32266 10151 Acked Read 7 Done Sent Done false
32267 10152 Acked Write 7 Done Sent Done false
32268 10153 Acked Read 10 Done Sent Done false
32269 10154 Acked Read 9 Done Sent Done false
32270 10155 Acked Write 6 Done Sent Done false
32271 10156 Acked Read 1 Done Sent Done false
32272 10157 Acked Read 3 Done Sent Done false
32273 10158 Acked Read 4 Done Sent Done false
32274 10159 Acked Read 6 Done Sent Done false
32275 10160 Acked Write 10 Done Sent Done false
32276 10161 Acked Flush 0 Done Sent Done false
32277 10162 Acked Write 7 Done Sent Done false
32278 10163 Acked Read 1 Done Sent Done false
32279 10164 Acked Read 8 Done Sent Done false
32280 10165 Acked Write 4 Done Sent Done false
32281 10166 Acked Write 8 Done Sent Done false
32282 10167 Acked Read 4 Done Sent Done false
32283 10168 Acked Write 8 Done Sent Done false
32284 10169 Acked Write 8 Done Sent Done false
32285 10170 Acked Read 9 Done Sent Done false
32286 10171 Acked Write 5 Done Sent Done false
32287 10172 Acked Write 2 Done Sent Done false
32288 10173 Acked Read 5 Done Sent Done false
32289 10174 Acked Write 1 Done Sent Done false
32290 10175 Acked Read 5 Done Sent Done false
32291 10176 Acked Read 5 Done Sent Done false
32292 10177 Acked Write 8 Done Sent Done false
32293 10178 Acked Flush 0 Done Sent Done false
32294 10179 Acked Read 7 Done Sent Done false
32295 10180 Acked Write 9 Done Sent Done false
32296 10181 Acked Read 9 Done Sent Done false
32297 10182 Acked Read 8 Done Sent Done false
32298 10183 Acked Read 10 Done Sent Done false
32299 10184 Acked Read 8 Done Sent Done false
32300 10185 Acked Write 9 Done Sent Done false
32301 10186 Acked Read 4 Done Sent Done false
32302 10187 Acked Write 8 Done Sent Done false
32303 10188 Acked Write 10 Done Sent Done false
32304 10189 Acked Read 9 Done Sent Done false
32305 10190 Acked Write 10 Done Sent Done false
32306 10191 Acked Flush 0 Done Sent Done false
32307 10192 Acked Read 4 Done Sent Done false
32308 10193 Acked Read 9 Done Sent Done false
32309 10194 Acked Write 2 Done Sent Done false
32310 10195 Acked Write 4 Done Sent Done false
32311 10196 Acked Read 7 Done Sent Done false
32312 10197 Acked Read 1 Done Sent Done false
32313 10198 Acked Read 4 Done Sent Done false
32314 10199 Acked Read 3 Done Sent Done false
32315 10200 Acked Write 6 Done Sent Done false
32316 10201 Acked Read 1 Done Sent Done false
32317 10202 Acked Read 8 Done Sent Done false
32318 10203 Acked Read 2 Done Sent Done false
32319 10204 Acked Write 8 Done Sent Done false
32320 10205 Acked Read 3 Done Sent Done false
32321 10206 Acked Read 7 Done Sent Done false
32322 10207 Acked Write 4 Done Sent Done false
32323 10208 Acked Flush 0 Done Sent Done false
32324 10209 Acked Write 1 Done Sent Done false
32325 10210 Acked Write 8 Done Sent Done false
32326 10211 Acked Write 5 Done Sent Done false
32327 10212 Acked Read 9 Done Sent Done false
32328 10213 Acked Read 2 Done Sent Done false
32329 10214 Acked Write 1 Done Sent Done false
32330 10215 Acked Write 5 Done Sent Done false
32331 10216 Acked Read 7 Done Sent Done false
32332 10217 Acked Read 6 Done Sent Done false
32333 10218 Acked Write 3 Done Sent Done false
32334 10219 Acked Read 2 Done Sent Done false
32335 10220 Acked Write 10 Done Sent Done false
32336 10221 Acked Read 8 Done Sent Done false
32337 10222 Acked Read 8 Done Sent Done false
32338 10223 Acked Flush 0 Done Sent Done false
32339 10224 Acked Write 5 Done Sent Done false
32340 10225 Acked Write 2 Done Sent Done false
32341 10226 Acked Write 6 Done Sent Done false
32342 10227 Acked Flush 0 Done Sent Done false
32343 10228 Acked Read 6 Done Sent Done false
32344 10229 Acked Flush 0 Done Sent Done false
32345 10230 Acked Read 6 Done Sent Done false
32346 10231 Acked Flush 0 Done Sent Done false
32347 10232 Acked Read 10 Done Sent Done false
32348 10233 Acked Read 7 Done Sent Done false
32349 10234 Acked Read 1 Done Sent Done false
32350 10235 Acked Read 9 Done Sent Done false
32351 10236 Acked Write 6 Done Sent Done false
32352 10237 Acked Read 5 Done Sent Done false
32353 10238 Acked Read 3 Done Sent Done false
32354 10239 Acked Write 4 Done Sent Done false
32355 10240 Acked Write 4 Done Sent Done false
32356 10241 Acked Flush 0 Done Sent Done false
32357 10242 Acked Read 4 Done Sent Done false
32358 10243 Acked Flush 0 Done Sent Done false
32359 10244 Acked Flush 0 Done Sent Done false
32360 10245 Acked Write 10 Done Sent Done false
32361 10246 Acked Write 4 Done Sent Done false
32362 10247 Acked Read 6 Done Sent Done false
32363 10248 Acked Write 2 Done Sent Done false
32364 10249 Acked Read 8 Done Sent Done false
32365 10250 Acked Read 2 Done Sent Done false
32366 10251 Acked Write 5 Done Sent Done false
32367 10252 Acked Write 4 Done Sent Done false
32368 10253 Acked Read 2 Done Sent Done false
32369 10254 Acked Flush 0 Done Sent Done false
32370 10255 Acked Write 3 Done Sent Done false
32371 10256 Acked Read 2 Done Sent Done false
32372 10257 Acked Read 7 Done Sent Done false
32373 10258 Acked Read 6 Done Sent Done false
32374 10259 Acked Read 3 Done Sent Done false
32375 10260 Acked Write 8 Done Sent Done false
32376 10261 Acked Flush 0 Done Sent Done false
32377 10262 Acked Read 8 Done Sent Done false
32378 10263 Acked Read 4 Done Sent Done false
32379 10264 Acked Read 3 Done Sent Done false
32380 10265 Acked Read 8 Done Sent Done false
32381 10266 Acked Read 7 Done Sent Done false
32382 10267 Acked Flush 0 Done Sent Done false
32383 10268 Acked Write 10 Done Sent Done false
32384 10269 Acked Read 4 Done Sent Done false
32385 10270 Acked Write 10 Done Sent Done false
32386 10271 Acked Read 4 Done Sent Done false
32387 10272 Acked Read 6 Done Sent Done false
32388 10273 Acked Flush 0 Done Sent Done false
32389 10274 Acked Read 10 Done Sent Done false
32390 10275 Acked Flush 0 Done Sent Done false
32391 10276 Acked Read 4 Done Sent Done false
32392 10277 Acked Read 8 Done Sent Done false
32393 10278 Acked Read 7 Done Sent Done false
32394 10279 Acked Write 4 Done Sent Done false
32395 10280 Acked Write 2 Done Sent Done false
32396 10281 Acked Read 9 Done Sent Done false
32397 10282 Acked Write 4 Done Sent Done false
32398 10283 Acked Flush 0 Done Sent Done false
32399 10284 Acked Read 9 Done Sent Done false
32400 10285 Acked Read 9 Done Sent Done false
32401 10286 Acked Write 6 Done Sent Done false
32402 10287 Acked Flush 0 Done Sent Done false
32403 10288 Acked Read 4 Done Sent Done false
32404 10289 Acked Read 10 Done Sent Done false
32405 10290 Acked Write 6 Done Sent Done false
32406 10291 Acked Write 1 Done Sent Done false
32407 10292 Acked Read 6 Done Sent Done false
32408 10293 Acked Write 4 Done Sent Done false
32409 10294 Acked Read 4 Done Sent Done false
32410 10295 Acked Read 6 Done Sent Done false
32411 10296 Acked Read 3 Done Sent Done false
32412 10297 Acked Write 8 Done Sent Done false
32413 10298 Acked Read 5 Done Sent Done false
32414 10299 Acked Read 2 Done Sent Done false
32415 10300 Acked Read 1 Done Sent Done false
32416 10301 Acked Read 10 Done Sent Done false
32417 10302 Acked Write 6 Done Sent Done false
32418 10303 Acked Read 3 Done Sent Done false
32419 10304 Acked Read 9 Done Sent Done false
32420 10305 Acked Flush 0 Done Sent Done false
32421 10306 Acked Read 3 Done Sent Done false
32422 10307 Acked Read 4 Done Sent Done false
32423 10308 Acked Write 6 Done Sent Done false
32424 10309 Acked Read 9 Done Sent Done false
32425 10310 Acked Read 7 Done Sent Done false
32426 10311 Acked Write 1 Done Sent Done false
32427 10312 Acked Read 3 Done Sent Done false
32428 10313 Acked Read 8 Done Sent Done false
32429 10314 Acked Write 5 Done Sent Done false
32430 10315 Acked Read 2 Done Sent Done false
32431 10316 Acked Read 10 Done Sent Done false
32432 10317 Acked Read 1 Done Sent Done false
32433 10318 Acked Write 4 Done Sent Done false
32434 10319 Acked Read 2 Done Sent Done false
32435 10320 Acked Read 4 Done Sent Done false
32436 10321 Acked Read 7 Done Sent Done false
32437 10322 Acked Write 6 Done Sent Done false
32438 10323 Acked Read 5 Done Sent Done false
32439 10324 Acked Read 3 Done Sent Done false
32440 10325 Acked Read 10 Done Sent Done false
32441 10326 Acked Read 7 Done Sent Done false
32442 10327 Acked Write 6 Done Sent Done false
32443 10328 Acked Read 2 Done Sent Done false
32444 10329 Acked Read 9 Done Sent Done false
32445 10330 Acked Write 5 Done Sent Done false
32446 10331 Acked Read 10 Done Sent Done false
32447 10332 Acked Write 10 Done Sent Done false
32448 10333 Acked Read 8 Done Sent Done false
32449 10334 Acked Read 9 Done Sent Done false
32450 10335 Acked Write 9 Done Sent Done false
32451 10336 Acked Flush 0 Done Sent Done false
32452 10337 Acked Read 8 Done Sent Done false
32453 10338 Acked Read 8 Done Sent Done false
32454 10339 Acked Read 6 Done Sent Done false
32455 10340 Acked Write 2 Done Sent Done false
32456 10341 Acked Read 1 Done Sent Done false
32457 10342 Acked Flush 0 Done Sent Done false
32458 10343 Acked Write 4 Done Sent Done false
32459 10344 Acked Read 7 Done Sent Done false
32460 10345 Acked Read 1 Done Sent Done false
32461 10346 Acked Write 3 Done Sent Done false
32462 10347 Acked Read 6 Done Sent Done false
32463 10348 Acked Read 8 Done Sent Done false
32464 10349 Acked Write 8 Done Sent Done false
32465 10350 Acked Write 3 Done Sent Done false
32466 10351 Acked Read 8 Done Sent Done false
32467 10352 Acked Flush 0 Done Sent Done false
32468 10353 Acked Write 3 Done Sent Done false
32469 10354 Acked Read 4 Done Sent Done false
32470 10355 Acked Read 1 Done Sent Done false
32471 10356 Acked Read 8 Done Sent Done false
32472 10357 Acked Read 1 Done Sent Done false
32473 10358 Acked Read 9 Done Sent Done false
32474 10359 Acked Read 9 Done Sent Done false
32475 STATES DS:0 DS:1 DS:2 TOTAL
32476 Sent 0 301 0 301
32477 Done 301 0 301 602
32478 Skipped 0 0 0 0
32479 Error 0 0 0 0
32480 Last Flush: Some(JobId(10352)) Some(JobId(10058)) Some(JobId(10352))
32481 Downstairs last five completed: 10058 10057 10056 10055 10054
32482 Upstairs last five completed: 10359 10358 10357 10356 10355
32483 CLIENT: Up:0 ds:301 act:2
32484 ----------------------------------------------------------------
32485 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
32486 States: Active Offline Active
32487 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32488 10059 Acked Write 9 Done Sent Done false
32489 10060 Acked Flush 0 Done Sent Done false
32490 10061 Acked Write 7 Done Sent Done false
32491 10062 Acked Read 9 Done Sent Done false
32492 10063 Acked Read 3 Done Sent Done false
32493 10064 Acked Write 4 Done Sent Done false
32494 10065 Acked Write 10 Done Sent Done false
32495 10066 Acked Read 1 Done Sent Done false
32496 10067 Acked Write 5 Done Sent Done false
32497 10068 Acked Write 5 Done Sent Done false
32498 10069 Acked Flush 0 Done Sent Done false
32499 10070 Acked Write 8 Done Sent Done false
32500 10071 Acked Write 2 Done Sent Done false
32501 10072 Acked Flush 0 Done Sent Done false
32502 10073 Acked Read 1 Done Sent Done false
32503 10074 Acked Flush 0 Done Sent Done false
32504 10075 Acked Write 8 Done Sent Done false
32505 10076 Acked Read 3 Done Sent Done false
32506 10077 Acked Read 10 Done Sent Done false
32507 10078 Acked Read 1 Done Sent Done false
32508 10079 Acked Read 4 Done Sent Done false
32509 10080 Acked Write 7 Done Sent Done false
32510 10081 Acked Read 6 Done Sent Done false
32511 10082 Acked Read 5 Done Sent Done false
32512 10083 Acked Write 7 Done Sent Done false
32513 10084 Acked Read 3 Done Sent Done false
32514 10085 Acked Read 7 Done Sent Done false
32515 10086 Acked Read 4 Done Sent Done false
32516 10087 Acked Write 2 Done Sent Done false
32517 10088 Acked Write 5 Done Sent Done false
32518 10089 Acked Write 8 Done Sent Done false
32519 10090 Acked Read 10 Done Sent Done false
32520 10091 Acked Write 4 Done Sent Done false
32521 10092 Acked Write 5 Done Sent Done false
32522 10093 Acked Write 5 Done Sent Done false
32523 10094 Acked Read 5 Done Sent Done false
32524 10095 Acked Read 6 Done Sent Done false
32525 10096 Acked Read 10 Done Sent Done false
32526 10097 Acked Read 2 Done Sent Done false
32527 10098 Acked Read 2 Done Sent Done false
32528 10099 Acked Write 7 Done Sent Done false
32529 10100 Acked Read 9 Done Sent Done false
32530 10101 Acked Read 8 Done Sent Done false
32531 10102 Acked Read 5 Done Sent Done false
32532 10103 Acked Read 10 Done Sent Done false
32533 10104 Acked Write 1 Done Sent Done false
32534 10105 Acked Write 5 Done Sent Done false
32535 10106 Acked Read 3 Done Sent Done false
32536 10107 Acked Read 6 Done Sent Done false
32537 10108 Acked Write 5 Done Sent Done false
32538 10109 Acked Read 4 Done Sent Done false
32539 10110 Acked Write 6 Done Sent Done false
32540 10111 Acked Read 3 Done Sent Done false
32541 10112 Acked Read 10 Done Sent Done false
32542 10113 Acked Read 4 Done Sent Done false
32543 10114 Acked Write 2 Done Sent Done false
32544 10115 Acked Write 4 Done Sent Done false
32545 10116 Acked Write 1 Done Sent Done false
32546 10117 Acked Write 5 Done Sent Done false
32547 10118 Acked Read 2 Done Sent Done false
32548 10119 Acked Write 9 Done Sent Done false
32549 10120 Acked Flush 0 Done Sent Done false
32550 10121 Acked Read 6 Done Sent Done false
32551 10122 Acked Read 2 Done Sent Done false
32552 10123 Acked Read 4 Done Sent Done false
32553 10124 Acked Read 8 Done Sent Done false
32554 10125 Acked Write 2 Done Sent Done false
32555 10126 Acked Write 1 Done Sent Done false
32556 10127 Acked Write 2 Done Sent Done false
32557 10128 Acked Read 1 Done Sent Done false
32558 10129 Acked Write 1 Done Sent Done false
32559 10130 Acked Write 7 Done Sent Done false
32560 10131 Acked Write 1 Done Sent Done false
32561 10132 Acked Read 8 Done Sent Done false
32562 10133 Acked Read 10 Done Sent Done false
32563 10134 Acked Write 9 Done Sent Done false
32564 10135 Acked Flush 0 Done Sent Done false
32565 10136 Acked Write 1 Done Sent Done false
32566 10137 Acked Write 10 Done Sent Done false
32567 10138 Acked Read 6 Done Sent Done false
32568 10139 Acked Read 7 Done Sent Done false
32569 10140 Acked Flush 0 Done Sent Done false
32570 10141 Acked Read 6 Done Sent Done false
32571 10142 Acked Flush 0 Done Sent Done false
32572 10143 Acked Write 2 Done Sent Done false
32573 10144 Acked Read 8 Done Sent Done false
32574 10145 Acked Write 2 Done Sent Done false
32575 10146 Acked Read 7 Done Sent Done false
32576 10147 Acked Flush 0 Done Sent Done false
32577 10148 Acked Read 7 Done Sent Done false
32578 10149 Acked Flush 0 Done Sent Done false
32579 10150 Acked Read 2 Done Sent Done false
32580 10151 Acked Read 7 Done Sent Done false
32581 10152 Acked Write 7 Done Sent Done false
32582 10153 Acked Read 10 Done Sent Done false
32583 10154 Acked Read 9 Done Sent Done false
32584 10155 Acked Write 6 Done Sent Done false
32585 10156 Acked Read 1 Done Sent Done false
32586 10157 Acked Read 3 Done Sent Done false
32587 10158 Acked Read 4 Done Sent Done false
32588 10159 Acked Read 6 Done Sent Done false
32589 10160 Acked Write 10 Done Sent Done false
32590 10161 Acked Flush 0 Done Sent Done false
32591 10162 Acked Write 7 Done Sent Done false
32592 10163 Acked Read 1 Done Sent Done false
32593 10164 Acked Read 8 Done Sent Done false
32594 10165 Acked Write 4 Done Sent Done false
32595 10166 Acked Write 8 Done Sent Done false
32596 10167 Acked Read 4 Done Sent Done false
32597 10168 Acked Write 8 Done Sent Done false
32598 10169 Acked Write 8 Done Sent Done false
32599 10170 Acked Read 9 Done Sent Done false
32600 10171 Acked Write 5 Done Sent Done false
32601 10172 Acked Write 2 Done Sent Done false
32602 10173 Acked Read 5 Done Sent Done false
32603 10174 Acked Write 1 Done Sent Done false
32604 10175 Acked Read 5 Done Sent Done false
32605 10176 Acked Read 5 Done Sent Done false
32606 10177 Acked Write 8 Done Sent Done false
32607 10178 Acked Flush 0 Done Sent Done false
32608 10179 Acked Read 7 Done Sent Done false
32609 10180 Acked Write 9 Done Sent Done false
32610 10181 Acked Read 9 Done Sent Done false
32611 10182 Acked Read 8 Done Sent Done false
32612 10183 Acked Read 10 Done Sent Done false
32613 10184 Acked Read 8 Done Sent Done false
32614 10185 Acked Write 9 Done Sent Done false
32615 10186 Acked Read 4 Done Sent Done false
32616 10187 Acked Write 8 Done Sent Done false
32617 10188 Acked Write 10 Done Sent Done false
32618 10189 Acked Read 9 Done Sent Done false
32619 10190 Acked Write 10 Done Sent Done false
32620 10191 Acked Flush 0 Done Sent Done false
32621 10192 Acked Read 4 Done Sent Done false
32622 10193 Acked Read 9 Done Sent Done false
32623 10194 Acked Write 2 Done Sent Done false
32624 10195 Acked Write 4 Done Sent Done false
32625 10196 Acked Read 7 Done Sent Done false
32626 10197 Acked Read 1 Done Sent Done false
32627 10198 Acked Read 4 Done Sent Done false
32628 10199 Acked Read 3 Done Sent Done false
32629 10200 Acked Write 6 Done Sent Done false
32630 10201 Acked Read 1 Done Sent Done false
32631 10202 Acked Read 8 Done Sent Done false
32632 10203 Acked Read 2 Done Sent Done false
32633 10204 Acked Write 8 Done Sent Done false
32634 10205 Acked Read 3 Done Sent Done false
32635 10206 Acked Read 7 Done Sent Done false
32636 10207 Acked Write 4 Done Sent Done false
32637 10208 Acked Flush 0 Done Sent Done false
32638 10209 Acked Write 1 Done Sent Done false
32639 10210 Acked Write 8 Done Sent Done false
32640 10211 Acked Write 5 Done Sent Done false
32641 10212 Acked Read 9 Done Sent Done false
32642 10213 Acked Read 2 Done Sent Done false
32643 10214 Acked Write 1 Done Sent Done false
32644 10215 Acked Write 5 Done Sent Done false
32645 10216 Acked Read 7 Done Sent Done false
32646 10217 Acked Read 6 Done Sent Done false
32647 10218 Acked Write 3 Done Sent Done false
32648 10219 Acked Read 2 Done Sent Done false
32649 10220 Acked Write 10 Done Sent Done false
32650 10221 Acked Read 8 Done Sent Done false
32651 10222 Acked Read 8 Done Sent Done false
32652 10223 Acked Flush 0 Done Sent Done false
32653 10224 Acked Write 5 Done Sent Done false
32654 10225 Acked Write 2 Done Sent Done false
32655 10226 Acked Write 6 Done Sent Done false
32656 10227 Acked Flush 0 Done Sent Done false
32657 10228 Acked Read 6 Done Sent Done false
32658 10229 Acked Flush 0 Done Sent Done false
32659 10230 Acked Read 6 Done Sent Done false
32660 10231 Acked Flush 0 Done Sent Done false
32661 10232 Acked Read 10 Done Sent Done false
32662 10233 Acked Read 7 Done Sent Done false
32663 10234 Acked Read 1 Done Sent Done false
32664 10235 Acked Read 9 Done Sent Done false
32665 10236 Acked Write 6 Done Sent Done false
32666 10237 Acked Read 5 Done Sent Done false
32667 10238 Acked Read 3 Done Sent Done false
32668 10239 Acked Write 4 Done Sent Done false
32669 10240 Acked Write 4 Done Sent Done false
32670 10241 Acked Flush 0 Done Sent Done false
32671 10242 Acked Read 4 Done Sent Done false
32672 10243 Acked Flush 0 Done Sent Done false
32673 10244 Acked Flush 0 Done Sent Done false
32674 10245 Acked Write 10 Done Sent Done false
32675 10246 Acked Write 4 Done Sent Done false
32676 10247 Acked Read 6 Done Sent Done false
32677 10248 Acked Write 2 Done Sent Done false
32678 10249 Acked Read 8 Done Sent Done false
32679 10250 Acked Read 2 Done Sent Done false
32680 10251 Acked Write 5 Done Sent Done false
32681 10252 Acked Write 4 Done Sent Done false
32682 10253 Acked Read 2 Done Sent Done false
32683 10254 Acked Flush 0 Done Sent Done false
32684 10255 Acked Write 3 Done Sent Done false
32685 10256 Acked Read 2 Done Sent Done false
32686 10257 Acked Read 7 Done Sent Done false
32687 10258 Acked Read 6 Done Sent Done false
32688 10259 Acked Read 3 Done Sent Done false
32689 10260 Acked Write 8 Done Sent Done false
32690 10261 Acked Flush 0 Done Sent Done false
32691 10262 Acked Read 8 Done Sent Done false
32692 10263 Acked Read 4 Done Sent Done false
32693 10264 Acked Read 3 Done Sent Done false
32694 10265 Acked Read 8 Done Sent Done false
32695 10266 Acked Read 7 Done Sent Done false
32696 10267 Acked Flush 0 Done Sent Done false
32697 10268 Acked Write 10 Done Sent Done false
32698 10269 Acked Read 4 Done Sent Done false
32699 10270 Acked Write 10 Done Sent Done false
32700 10271 Acked Read 4 Done Sent Done false
32701 10272 Acked Read 6 Done Sent Done false
32702 10273 Acked Flush 0 Done Sent Done false
32703 10274 Acked Read 10 Done Sent Done false
32704 10275 Acked Flush 0 Done Sent Done false
32705 10276 Acked Read 4 Done Sent Done false
32706 10277 Acked Read 8 Done Sent Done false
32707 10278 Acked Read 7 Done Sent Done false
32708 10279 Acked Write 4 Done Sent Done false
32709 10280 Acked Write 2 Done Sent Done false
32710 10281 Acked Read 9 Done Sent Done false
32711 10282 Acked Write 4 Done Sent Done false
32712 10283 Acked Flush 0 Done Sent Done false
32713 10284 Acked Read 9 Done Sent Done false
32714 10285 Acked Read 9 Done Sent Done false
32715 10286 Acked Write 6 Done Sent Done false
32716 10287 Acked Flush 0 Done Sent Done false
32717 10288 Acked Read 4 Done Sent Done false
32718 10289 Acked Read 10 Done Sent Done false
32719 10290 Acked Write 6 Done Sent Done false
32720 10291 Acked Write 1 Done Sent Done false
32721 10292 Acked Read 6 Done Sent Done false
32722 10293 Acked Write 4 Done Sent Done false
32723 10294 Acked Read 4 Done Sent Done false
32724 10295 Acked Read 6 Done Sent Done false
32725 10296 Acked Read 3 Done Sent Done false
32726 10297 Acked Write 8 Done Sent Done false
32727 10298 Acked Read 5 Done Sent Done false
32728 10299 Acked Read 2 Done Sent Done false
32729 10300 Acked Read 1 Done Sent Done false
32730 10301 Acked Read 10 Done Sent Done false
32731 10302 Acked Write 6 Done Sent Done false
32732 10303 Acked Read 3 Done Sent Done false
32733 10304 Acked Read 9 Done Sent Done false
32734 10305 Acked Flush 0 Done Sent Done false
32735 10306 Acked Read 3 Done Sent Done false
32736 10307 Acked Read 4 Done Sent Done false
32737 10308 Acked Write 6 Done Sent Done false
32738 10309 Acked Read 9 Done Sent Done false
32739 10310 Acked Read 7 Done Sent Done false
32740 10311 Acked Write 1 Done Sent Done false
32741 10312 Acked Read 3 Done Sent Done false
32742 10313 Acked Read 8 Done Sent Done false
32743 10314 Acked Write 5 Done Sent Done false
32744 10315 Acked Read 2 Done Sent Done false
32745 10316 Acked Read 10 Done Sent Done false
32746 10317 Acked Read 1 Done Sent Done false
32747 10318 Acked Write 4 Done Sent Done false
32748 10319 Acked Read 2 Done Sent Done false
32749 10320 Acked Read 4 Done Sent Done false
32750 10321 Acked Read 7 Done Sent Done false
32751 10322 Acked Write 6 Done Sent Done false
32752 10323 Acked Read 5 Done Sent Done false
32753 10324 Acked Read 3 Done Sent Done false
32754 10325 Acked Read 10 Done Sent Done false
32755 10326 Acked Read 7 Done Sent Done false
32756 10327 Acked Write 6 Done Sent Done false
32757 10328 Acked Read 2 Done Sent Done false
32758 10329 Acked Read 9 Done Sent Done false
32759 10330 Acked Write 5 Done Sent Done false
32760 10331 Acked Read 10 Done Sent Done false
32761 10332 Acked Write 10 Done Sent Done false
32762 10333 Acked Read 8 Done Sent Done false
32763 10334 Acked Read 9 Done Sent Done false
32764 10335 Acked Write 9 Done Sent Done false
32765 10336 Acked Flush 0 Done Sent Done false
32766 10337 Acked Read 8 Done Sent Done false
32767 10338 Acked Read 8 Done Sent Done false
32768 10339 Acked Read 6 Done Sent Done false
32769 10340 Acked Write 2 Done Sent Done false
32770 10341 Acked Read 1 Done Sent Done false
32771 10342 Acked Flush 0 Done Sent Done false
32772 10343 Acked Write 4 Done Sent Done false
32773 10344 Acked Read 7 Done Sent Done false
32774 10345 Acked Read 1 Done Sent Done false
32775 10346 Acked Write 3 Done Sent Done false
32776 10347 Acked Read 6 Done Sent Done false
32777 10348 Acked Read 8 Done Sent Done false
32778 10349 Acked Write 8 Done Sent Done false
32779 10350 Acked Write 3 Done Sent Done false
32780 10351 Acked Read 8 Done Sent Done false
32781 10352 Acked Flush 0 Done Sent Done false
32782 10353 Acked Write 3 Done Sent Done false
32783 10354 Acked Read 4 Done Sent Done false
32784 10355 Acked Read 1 Done Sent Done false
32785 10356 Acked Read 8 Done Sent Done false
32786 10357 Acked Read 1 Done Sent Done false
32787 10358 Acked Read 9 Done Sent Done false
32788 10359 Acked Read 9 Done Sent Done false
32789 10360 Acked Flush 0 Done Sent Done false
32790 STATES DS:0 DS:1 DS:2 TOTAL
32791 Sent 0 302 0 302
32792 Done 302 0 302 604
32793 Skipped 0 0 0 0
32794 Error 0 0 0 0
32795 Last Flush: Some(JobId(10360)) Some(JobId(10058)) Some(JobId(10360))
32796 Downstairs last five completed: 10058 10057 10056 10055 10054
32797 Upstairs last five completed: 10360 10359 10358 10357 10356
32798 CLIENT: Up:0 ds:302 act:2
327992025-01-23T17:44:37.214ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
328002025-01-23T17:44:37.214ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
328012025-01-23T17:44:37.214ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
328022025-01-23T17:44:37.214ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
328032025-01-23T17:44:37.214ZINFOcrucible: client task is exiting = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
32804 ----------------------------------------------------------------
32805 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
32806 States: Active Offline Active
32807 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32808 10059 Acked Write 9 Done Sent Done false
32809 10060 Acked Flush 0 Done Sent Done false
32810 10061 Acked Write 7 Done Sent Done false
32811 10062 Acked Read 9 Done Sent Done false
32812 10063 Acked Read 3 Done Sent Done false
32813 10064 Acked Write 4 Done Sent Done false
32814 10065 Acked Write 10 Done Sent Done false
32815 10066 Acked Read 1 Done Sent Done false
32816 10067 Acked Write 5 Done Sent Done false
32817 10068 Acked Write 5 Done Sent Done false
32818 10069 Acked Flush 0 Done Sent Done false
32819 10070 Acked Write 8 Done Sent Done false
32820 10071 Acked Write 2 Done Sent Done false
32821 10072 Acked Flush 0 Done Sent Done false
32822 10073 Acked Read 1 Done Sent Done false
32823 10074 Acked Flush 0 Done Sent Done false
32824 10075 Acked Write 8 Done Sent Done false
32825 10076 Acked Read 3 Done Sent Done false
32826 10077 Acked Read 10 Done Sent Done false
32827 10078 Acked Read 1 Done Sent Done false
32828 10079 Acked Read 4 Done Sent Done false
32829 10080 Acked Write 7 Done Sent Done false
32830 10081 Acked Read 6 Done Sent Done false
32831 10082 Acked Read 5 Done Sent Done false
32832 10083 Acked Write 7 Done Sent Done false
32833 10084 Acked Read 3 Done Sent Done false
32834 10085 Acked Read 7 Done Sent Done false
32835 10086 Acked Read 4 Done Sent Done false
32836 10087 Acked Write 2 Done Sent Done false
32837 10088 Acked Write 5 Done Sent Done false
32838 10089 Acked Write 8 Done Sent Done false
32839 10090 Acked Read 10 Done Sent Done false
32840 10091 Acked Write 4 Done Sent Done false
32841 10092 Acked Write 5 Done Sent Done false
32842 10093 Acked Write 5 Done Sent Done false
32843 10094 Acked Read 5 Done Sent Done false
32844 10095 Acked Read 6 Done Sent Done false
32845 10096 Acked Read 10 Done Sent Done false
32846 10097 Acked Read 2 Done Sent Done false
32847 10098 Acked Read 2 Done Sent Done false
32848 10099 Acked Write 7 Done Sent Done false
32849 10100 Acked Read 9 Done Sent Done false
32850 10101 Acked Read 8 Done Sent Done false
32851 10102 Acked Read 5 Done Sent Done false
32852 10103 Acked Read 10 Done Sent Done false
32853 10104 Acked Write 1 Done Sent Done false
32854 10105 Acked Write 5 Done Sent Done false
32855 10106 Acked Read 3 Done Sent Done false
32856 10107 Acked Read 6 Done Sent Done false
32857 10108 Acked Write 5 Done Sent Done false
32858 10109 Acked Read 4 Done Sent Done false
32859 10110 Acked Write 6 Done Sent Done false
32860 10111 Acked Read 3 Done Sent Done false
32861 10112 Acked Read 10 Done Sent Done false
32862 10113 Acked Read 4 Done Sent Done false
32863 10114 Acked Write 2 Done Sent Done false
32864 10115 Acked Write 4 Done Sent Done false
32865 10116 Acked Write 1 Done Sent Done false
32866 10117 Acked Write 5 Done Sent Done false
32867 10118 Acked Read 2 Done Sent Done false
32868 10119 Acked Write 9 Done Sent Done false
32869 10120 Acked Flush 0 Done Sent Done false
32870 10121 Acked Read 6 Done Sent Done false
32871 10122 Acked Read 2 Done Sent Done false
32872 10123 Acked Read 4 Done Sent Done false
32873 10124 Acked Read 8 Done Sent Done false
32874 10125 Acked Write 2 Done Sent Done false
32875 10126 Acked Write 1 Done Sent Done false
32876 10127 Acked Write 2 Done Sent Done false
32877 10128 Acked Read 1 Done Sent Done false
32878 10129 Acked Write 1 Done Sent Done false
32879 10130 Acked Write 7 Done Sent Done false
32880 10131 Acked Write 1 Done Sent Done false
32881 10132 Acked Read 8 Done Sent Done false
32882 10133 Acked Read 10 Done Sent Done false
32883 10134 Acked Write 9 Done Sent Done false
32884 10135 Acked Flush 0 Done Sent Done false
32885 10136 Acked Write 1 Done Sent Done false
32886 10137 Acked Write 10 Done Sent Done false
32887 10138 Acked Read 6 Done Sent Done false
32888 10139 Acked Read 7 Done Sent Done false
32889 10140 Acked Flush 0 Done Sent Done false
32890 10141 Acked Read 6 Done Sent Done false
32891 10142 Acked Flush 0 Done Sent Done false
32892 10143 Acked Write 2 Done Sent Done false
32893 10144 Acked Read 8 Done Sent Done false
32894 10145 Acked Write 2 Done Sent Done false
32895 10146 Acked Read 7 Done Sent Done false
32896 10147 Acked Flush 0 Done Sent Done false
32897 10148 Acked Read 7 Done Sent Done false
32898 10149 Acked Flush 0 Done Sent Done false
32899 10150 Acked Read 2 Done Sent Done false
32900 10151 Acked Read 7 Done Sent Done false
32901 10152 Acked Write 7 Done Sent Done false
32902 10153 Acked Read 10 Done Sent Done false
32903 10154 Acked Read 9 Done Sent Done false
32904 10155 Acked Write 6 Done Sent Done false
32905 10156 Acked Read 1 Done Sent Done false
32906 10157 Acked Read 3 Done Sent Done false
32907 10158 Acked Read 4 Done Sent Done false
32908 10159 Acked Read 6 Done Sent Done false
32909 10160 Acked Write 10 Done Sent Done false
32910 10161 Acked Flush 0 Done Sent Done false
32911 10162 Acked Write 7 Done Sent Done false
32912 10163 Acked Read 1 Done Sent Done false
32913 10164 Acked Read 8 Done Sent Done false
32914 10165 Acked Write 4 Done Sent Done false
32915 10166 Acked Write 8 Done Sent Done false
32916 10167 Acked Read 4 Done Sent Done false
32917 10168 Acked Write 8 Done Sent Done false
32918 10169 Acked Write 8 Done Sent Done false
32919 10170 Acked Read 9 Done Sent Done false
32920 10171 Acked Write 5 Done Sent Done false
32921 10172 Acked Write 2 Done Sent Done false
32922 10173 Acked Read 5 Done Sent Done false
32923 10174 Acked Write 1 Done Sent Done false
32924 10175 Acked Read 5 Done Sent Done false
32925 10176 Acked Read 5 Done Sent Done false
32926 10177 Acked Write 8 Done Sent Done false
32927 10178 Acked Flush 0 Done Sent Done false
32928 10179 Acked Read 7 Done Sent Done false
32929 10180 Acked Write 9 Done Sent Done false
32930 10181 Acked Read 9 Done Sent Done false
32931 10182 Acked Read 8 Done Sent Done false
32932 10183 Acked Read 10 Done Sent Done false
32933 10184 Acked Read 8 Done Sent Done false
32934 10185 Acked Write 9 Done Sent Done false
32935 10186 Acked Read 4 Done Sent Done false
32936 10187 Acked Write 8 Done Sent Done false
32937 10188 Acked Write 10 Done Sent Done false
32938 10189 Acked Read 9 Done Sent Done false
32939 10190 Acked Write 10 Done Sent Done false
32940 10191 Acked Flush 0 Done Sent Done false
32941 10192 Acked Read 4 Done Sent Done false
32942 10193 Acked Read 9 Done Sent Done false
32943 10194 Acked Write 2 Done Sent Done false
32944 10195 Acked Write 4 Done Sent Done false
32945 10196 Acked Read 7 Done Sent Done false
32946 10197 Acked Read 1 Done Sent Done false
32947 10198 Acked Read 4 Done Sent Done false
32948 10199 Acked Read 3 Done Sent Done false
32949 10200 Acked Write 6 Done Sent Done false
32950 10201 Acked Read 1 Done Sent Done false
32951 10202 Acked Read 8 Done Sent Done false
32952 10203 Acked Read 2 Done Sent Done false
32953 10204 Acked Write 8 Done Sent Done false
32954 10205 Acked Read 3 Done Sent Done false
32955 10206 Acked Read 7 Done Sent Done false
32956 10207 Acked Write 4 Done Sent Done false
32957 10208 Acked Flush 0 Done Sent Done false
32958 10209 Acked Write 1 Done Sent Done false
32959 10210 Acked Write 8 Done Sent Done false
32960 10211 Acked Write 5 Done Sent Done false
32961 10212 Acked Read 9 Done Sent Done false
32962 10213 Acked Read 2 Done Sent Done false
32963 10214 Acked Write 1 Done Sent Done false
32964 10215 Acked Write 5 Done Sent Done false
32965 10216 Acked Read 7 Done Sent Done false
32966 10217 Acked Read 6 Done Sent Done false
32967 10218 Acked Write 3 Done Sent Done false
32968 10219 Acked Read 2 Done Sent Done false
32969 10220 Acked Write 10 Done Sent Done false
32970 10221 Acked Read 8 Done Sent Done false
32971 10222 Acked Read 8 Done Sent Done false
32972 10223 Acked Flush 0 Done Sent Done false
32973 10224 Acked Write 5 Done Sent Done false
32974 10225 Acked Write 2 Done Sent Done false
32975 10226 Acked Write 6 Done Sent Done false
32976 10227 Acked Flush 0 Done Sent Done false
32977 10228 Acked Read 6 Done Sent Done false
32978 10229 Acked Flush 0 Done Sent Done false
32979 10230 Acked Read 6 Done Sent Done false
32980 10231 Acked Flush 0 Done Sent Done false
32981 10232 Acked Read 10 Done Sent Done false
32982 10233 Acked Read 7 Done Sent Done false
32983 10234 Acked Read 1 Done Sent Done false
32984 10235 Acked Read 9 Done Sent Done false
32985 10236 Acked Write 6 Done Sent Done false
32986 10237 Acked Read 5 Done Sent Done false
32987 10238 Acked Read 3 Done Sent Done false
32988 10239 Acked Write 4 Done Sent Done false
32989 10240 Acked Write 4 Done Sent Done false
32990 10241 Acked Flush 0 Done Sent Done false
32991 10242 Acked Read 4 Done Sent Done false
32992 10243 Acked Flush 0 Done Sent Done false
32993 10244 Acked Flush 0 Done Sent Done false
32994 10245 Acked Write 10 Done Sent Done false
32995 10246 Acked Write 4 Done Sent Done false
32996 10247 Acked Read 6 Done Sent Done false
32997 10248 Acked Write 2 Done Sent Done false
32998 10249 Acked Read 8 Done Sent Done false
32999 10250 Acked Read 2 Done Sent Done false
33000 10251 Acked Write 5 Done Sent Done false
33001 10252 Acked Write 4 Done Sent Done false
33002 10253 Acked Read 2 Done Sent Done false
33003 10254 Acked Flush 0 Done Sent Done false
33004 10255 Acked Write 3 Done Sent Done false
33005 10256 Acked Read 2 Done Sent Done false
33006 10257 Acked Read 7 Done Sent Done false
33007 10258 Acked Read 6 Done Sent Done false
33008 10259 Acked Read 3 Done Sent Done false
33009 10260 Acked Write 8 Done Sent Done false
33010 10261 Acked Flush 0 Done Sent Done false
33011 10262 Acked Read 8 Done Sent Done false
33012 10263 Acked Read 4 Done Sent Done false
33013 10264 Acked Read 3 Done Sent Done false
33014 10265 Acked Read 8 Done Sent Done false
33015 10266 Acked Read 7 Done Sent Done false
33016 10267 Acked Flush 0 Done Sent Done false
33017 10268 Acked Write 10 Done Sent Done false
33018 10269 Acked Read 4 Done Sent Done false
33019 10270 Acked Write 10 Done Sent Done false
33020 10271 Acked Read 4 Done Sent Done false
33021 10272 Acked Read 6 Done Sent Done false
33022 10273 Acked Flush 0 Done Sent Done false
33023 10274 Acked Read 10 Done Sent Done false
33024 10275 Acked Flush 0 Done Sent Done false
33025 10276 Acked Read 4 Done Sent Done false
33026 10277 Acked Read 8 Done Sent Done false
33027 10278 Acked Read 7 Done Sent Done false
33028 10279 Acked Write 4 Done Sent Done false
33029 10280 Acked Write 2 Done Sent Done false
33030 10281 Acked Read 9 Done Sent Done false
33031 10282 Acked Write 4 Done Sent Done false
33032 10283 Acked Flush 0 Done Sent Done false
33033 10284 Acked Read 9 Done Sent Done false
33034 10285 Acked Read 9 Done Sent Done false
33035 10286 Acked Write 6 Done Sent Done false
33036 10287 Acked Flush 0 Done Sent Done false
33037 10288 Acked Read 4 Done Sent Done false
33038 10289 Acked Read 10 Done Sent Done false
33039 10290 Acked Write 6 Done Sent Done false
33040 10291 Acked Write 1 Done Sent Done false
33041 10292 Acked Read 6 Done Sent Done false
33042 10293 Acked Write 4 Done Sent Done false
33043 10294 Acked Read 4 Done Sent Done false
33044 10295 Acked Read 6 Done Sent Done false
33045 10296 Acked Read 3 Done Sent Done false
33046 10297 Acked Write 8 Done Sent Done false
33047 10298 Acked Read 5 Done Sent Done false
33048 10299 Acked Read 2 Done Sent Done false
33049 10300 Acked Read 1 Done Sent Done false
33050 10301 Acked Read 10 Done Sent Done false
33051 10302 Acked Write 6 Done Sent Done false
33052 10303 Acked Read 3 Done Sent Done false
33053 10304 Acked Read 9 Done Sent Done false
33054 10305 Acked Flush 0 Done Sent Done false
33055 10306 Acked Read 3 Done Sent Done false
33056 10307 Acked Read 4 Done Sent Done false
33057 10308 Acked Write 6 Done Sent Done false
33058 10309 Acked Read 9 Done Sent Done false
33059 10310 Acked Read 7 Done Sent Done false
33060 10311 Acked Write 1 Done Sent Done false
33061 10312 Acked Read 3 Done Sent Done false
33062 10313 Acked Read 8 Done Sent Done false
33063 10314 Acked Write 5 Done Sent Done false
33064 10315 Acked Read 2 Done Sent Done false
33065 10316 Acked Read 10 Done Sent Done false
33066 10317 Acked Read 1 Done Sent Done false
33067 10318 Acked Write 4 Done Sent Done false
33068 10319 Acked Read 2 Done Sent Done false
33069 10320 Acked Read 4 Done Sent Done false
33070 10321 Acked Read 7 Done Sent Done false
33071 10322 Acked Write 6 Done Sent Done false
33072 10323 Acked Read 5 Done Sent Done false
33073 10324 Acked Read 3 Done Sent Done false
33074 10325 Acked Read 10 Done Sent Done false
33075 10326 Acked Read 7 Done Sent Done false
33076 10327 Acked Write 6 Done Sent Done false
33077 10328 Acked Read 2 Done Sent Done false
33078 10329 Acked Read 9 Done Sent Done false
33079 10330 Acked Write 5 Done Sent Done false
33080 10331 Acked Read 10 Done Sent Done false
33081 10332 Acked Write 10 Done Sent Done false
33082 10333 Acked Read 8 Done Sent Done false
33083 10334 Acked Read 9 Done Sent Done false
33084 10335 Acked Write 9 Done Sent Done false
33085 10336 Acked Flush 0 Done Sent Done false
33086 10337 Acked Read 8 Done Sent Done false
33087 10338 Acked Read 8 Done Sent Done false
33088 10339 Acked Read 6 Done Sent Done false
33089 10340 Acked Write 2 Done Sent Done false
33090 10341 Acked Read 1 Done Sent Done false
33091 10342 Acked Flush 0 Done Sent Done false
33092 10343 Acked Write 4 Done Sent Done false
33093 10344 Acked Read 7 Done Sent Done false
33094 10345 Acked Read 1 Done Sent Done false
33095 10346 Acked Write 3 Done Sent Done false
33096 10347 Acked Read 6 Done Sent Done false
33097 10348 Acked Read 8 Done Sent Done false
33098 10349 Acked Write 8 Done Sent Done false
33099 10350 Acked Write 3 Done Sent Done false
33100 10351 Acked Read 8 Done Sent Done false
33101 10352 Acked Flush 0 Done Sent Done false
33102 10353 Acked Write 3 Done Sent Done false
33103 10354 Acked Read 4 Done Sent Done false
33104 10355 Acked Read 1 Done Sent Done false
33105 10356 Acked Read 8 Done Sent Done false
33106 10357 Acked Read 1 Done Sent Done false
33107 10358 Acked Read 9 Done Sent Done false
33108 10359 Acked Read 9 Done Sent Done false
33109 10360 Acked Flush 0 Done Sent Done false
33110 STATES DS:0 DS:1 DS:2 TOTAL
33111 Sent 0 302 0 302
33112 Done 302 0 302 604
33113 Skipped 0 0 0 0
33114 Error 0 0 0 0
33115 Last Flush: Some(JobId(10360)) Some(JobId(10058)) Some(JobId(10360))
33116 Downstairs last five completed: 10058 10057 10056 10055 10054
33117 Upstairs last five completed: 10360 10359 10358 10357 10356
33118 CLIENT: Up:0 ds:302 act:2
33119 [8820] got start action
33120 [8820] I am going to restart
33121 Starting downstairs at port 8820
33122 Make output file at "/tmp/dsc/downstairs-8820.txt"
33123 [8820][1] reports Starting
33124 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1236)
33125 [8820][1] reports Running
33126 ----------------------------------------------------------------
33127 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
33128 States: Active Offline Active
33129 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33130 10059 Acked Write 9 Done Sent Done false
33131 10060 Acked Flush 0 Done Sent Done false
33132 10061 Acked Write 7 Done Sent Done false
33133 10062 Acked Read 9 Done Sent Done false
33134 10063 Acked Read 3 Done Sent Done false
33135 10064 Acked Write 4 Done Sent Done false
33136 10065 Acked Write 10 Done Sent Done false
33137 10066 Acked Read 1 Done Sent Done false
33138 10067 Acked Write 5 Done Sent Done false
33139 10068 Acked Write 5 Done Sent Done false
33140 10069 Acked Flush 0 Done Sent Done false
33141 10070 Acked Write 8 Done Sent Done false
33142 10071 Acked Write 2 Done Sent Done false
33143 10072 Acked Flush 0 Done Sent Done false
33144 10073 Acked Read 1 Done Sent Done false
33145 10074 Acked Flush 0 Done Sent Done false
33146 10075 Acked Write 8 Done Sent Done false
33147 10076 Acked Read 3 Done Sent Done false
33148 10077 Acked Read 10 Done Sent Done false
33149 10078 Acked Read 1 Done Sent Done false
33150 10079 Acked Read 4 Done Sent Done false
33151 10080 Acked Write 7 Done Sent Done false
33152 10081 Acked Read 6 Done Sent Done false
33153 10082 Acked Read 5 Done Sent Done false
33154 10083 Acked Write 7 Done Sent Done false
33155 10084 Acked Read 3 Done Sent Done false
33156 10085 Acked Read 7 Done Sent Done false
33157 10086 Acked Read 4 Done Sent Done false
33158 10087 Acked Write 2 Done Sent Done false
33159 10088 Acked Write 5 Done Sent Done false
33160 10089 Acked Write 8 Done Sent Done false
33161 10090 Acked Read 10 Done Sent Done false
33162 10091 Acked Write 4 Done Sent Done false
33163 10092 Acked Write 5 Done Sent Done false
33164 10093 Acked Write 5 Done Sent Done false
33165 10094 Acked Read 5 Done Sent Done false
33166 10095 Acked Read 6 Done Sent Done false
33167 10096 Acked Read 10 Done Sent Done false
33168 10097 Acked Read 2 Done Sent Done false
33169 10098 Acked Read 2 Done Sent Done false
33170 10099 Acked Write 7 Done Sent Done false
33171 10100 Acked Read 9 Done Sent Done false
33172 10101 Acked Read 8 Done Sent Done false
33173 10102 Acked Read 5 Done Sent Done false
33174 10103 Acked Read 10 Done Sent Done false
33175 10104 Acked Write 1 Done Sent Done false
33176 10105 Acked Write 5 Done Sent Done false
33177 10106 Acked Read 3 Done Sent Done false
33178 10107 Acked Read 6 Done Sent Done false
33179 10108 Acked Write 5 Done Sent Done false
33180 10109 Acked Read 4 Done Sent Done false
33181 10110 Acked Write 6 Done Sent Done false
33182 10111 Acked Read 3 Done Sent Done false
33183 10112 Acked Read 10 Done Sent Done false
33184 10113 Acked Read 4 Done Sent Done false
33185 10114 Acked Write 2 Done Sent Done false
33186 10115 Acked Write 4 Done Sent Done false
33187 10116 Acked Write 1 Done Sent Done false
33188 10117 Acked Write 5 Done Sent Done false
33189 10118 Acked Read 2 Done Sent Done false
33190 10119 Acked Write 9 Done Sent Done false
33191 10120 Acked Flush 0 Done Sent Done false
33192 10121 Acked Read 6 Done Sent Done false
33193 10122 Acked Read 2 Done Sent Done false
33194 10123 Acked Read 4 Done Sent Done false
33195 10124 Acked Read 8 Done Sent Done false
33196 10125 Acked Write 2 Done Sent Done false
33197 10126 Acked Write 1 Done Sent Done false
33198 10127 Acked Write 2 Done Sent Done false
33199 10128 Acked Read 1 Done Sent Done false
33200 10129 Acked Write 1 Done Sent Done false
33201 10130 Acked Write 7 Done Sent Done false
33202 10131 Acked Write 1 Done Sent Done false
33203 10132 Acked Read 8 Done Sent Done false
33204 10133 Acked Read 10 Done Sent Done false
33205 10134 Acked Write 9 Done Sent Done false
33206 10135 Acked Flush 0 Done Sent Done false
33207 10136 Acked Write 1 Done Sent Done false
33208 10137 Acked Write 10 Done Sent Done false
33209 10138 Acked Read 6 Done Sent Done false
33210 10139 Acked Read 7 Done Sent Done false
33211 10140 Acked Flush 0 Done Sent Done false
33212 10141 Acked Read 6 Done Sent Done false
33213 10142 Acked Flush 0 Done Sent Done false
33214 10143 Acked Write 2 Done Sent Done false
33215 10144 Acked Read 8 Done Sent Done false
33216 10145 Acked Write 2 Done Sent Done false
33217 10146 Acked Read 7 Done Sent Done false
33218 10147 Acked Flush 0 Done Sent Done false
33219 10148 Acked Read 7 Done Sent Done false
33220 10149 Acked Flush 0 Done Sent Done false
33221 10150 Acked Read 2 Done Sent Done false
33222 10151 Acked Read 7 Done Sent Done false
33223 10152 Acked Write 7 Done Sent Done false
33224 10153 Acked Read 10 Done Sent Done false
33225 10154 Acked Read 9 Done Sent Done false
33226 10155 Acked Write 6 Done Sent Done false
33227 10156 Acked Read 1 Done Sent Done false
33228 10157 Acked Read 3 Done Sent Done false
33229 10158 Acked Read 4 Done Sent Done false
33230 10159 Acked Read 6 Done Sent Done false
33231 10160 Acked Write 10 Done Sent Done false
33232 10161 Acked Flush 0 Done Sent Done false
33233 10162 Acked Write 7 Done Sent Done false
33234 10163 Acked Read 1 Done Sent Done false
33235 10164 Acked Read 8 Done Sent Done false
33236 10165 Acked Write 4 Done Sent Done false
33237 10166 Acked Write 8 Done Sent Done false
33238 10167 Acked Read 4 Done Sent Done false
33239 10168 Acked Write 8 Done Sent Done false
33240 10169 Acked Write 8 Done Sent Done false
33241 10170 Acked Read 9 Done Sent Done false
33242 10171 Acked Write 5 Done Sent Done false
33243 10172 Acked Write 2 Done Sent Done false
33244 10173 Acked Read 5 Done Sent Done false
33245 10174 Acked Write 1 Done Sent Done false
33246 10175 Acked Read 5 Done Sent Done false
33247 10176 Acked Read 5 Done Sent Done false
33248 10177 Acked Write 8 Done Sent Done false
33249 10178 Acked Flush 0 Done Sent Done false
33250 10179 Acked Read 7 Done Sent Done false
33251 10180 Acked Write 9 Done Sent Done false
33252 10181 Acked Read 9 Done Sent Done false
33253 10182 Acked Read 8 Done Sent Done false
33254 10183 Acked Read 10 Done Sent Done false
33255 10184 Acked Read 8 Done Sent Done false
33256 10185 Acked Write 9 Done Sent Done false
33257 10186 Acked Read 4 Done Sent Done false
33258 10187 Acked Write 8 Done Sent Done false
33259 10188 Acked Write 10 Done Sent Done false
33260 10189 Acked Read 9 Done Sent Done false
33261 10190 Acked Write 10 Done Sent Done false
33262 10191 Acked Flush 0 Done Sent Done false
33263 10192 Acked Read 4 Done Sent Done false
33264 10193 Acked Read 9 Done Sent Done false
33265 10194 Acked Write 2 Done Sent Done false
33266 10195 Acked Write 4 Done Sent Done false
33267 10196 Acked Read 7 Done Sent Done false
33268 10197 Acked Read 1 Done Sent Done false
33269 10198 Acked Read 4 Done Sent Done false
33270 10199 Acked Read 3 Done Sent Done false
33271 10200 Acked Write 6 Done Sent Done false
33272 10201 Acked Read 1 Done Sent Done false
33273 10202 Acked Read 8 Done Sent Done false
33274 10203 Acked Read 2 Done Sent Done false
33275 10204 Acked Write 8 Done Sent Done false
33276 10205 Acked Read 3 Done Sent Done false
33277 10206 Acked Read 7 Done Sent Done false
33278 10207 Acked Write 4 Done Sent Done false
33279 10208 Acked Flush 0 Done Sent Done false
33280 10209 Acked Write 1 Done Sent Done false
33281 10210 Acked Write 8 Done Sent Done false
33282 10211 Acked Write 5 Done Sent Done false
33283 10212 Acked Read 9 Done Sent Done false
33284 10213 Acked Read 2 Done Sent Done false
33285 10214 Acked Write 1 Done Sent Done false
33286 10215 Acked Write 5 Done Sent Done false
33287 10216 Acked Read 7 Done Sent Done false
33288 10217 Acked Read 6 Done Sent Done false
33289 10218 Acked Write 3 Done Sent Done false
33290 10219 Acked Read 2 Done Sent Done false
33291 10220 Acked Write 10 Done Sent Done false
33292 10221 Acked Read 8 Done Sent Done false
33293 10222 Acked Read 8 Done Sent Done false
33294 10223 Acked Flush 0 Done Sent Done false
33295 10224 Acked Write 5 Done Sent Done false
33296 10225 Acked Write 2 Done Sent Done false
33297 10226 Acked Write 6 Done Sent Done false
33298 10227 Acked Flush 0 Done Sent Done false
33299 10228 Acked Read 6 Done Sent Done false
33300 10229 Acked Flush 0 Done Sent Done false
33301 10230 Acked Read 6 Done Sent Done false
33302 10231 Acked Flush 0 Done Sent Done false
33303 10232 Acked Read 10 Done Sent Done false
33304 10233 Acked Read 7 Done Sent Done false
33305 10234 Acked Read 1 Done Sent Done false
33306 10235 Acked Read 9 Done Sent Done false
33307 10236 Acked Write 6 Done Sent Done false
33308 10237 Acked Read 5 Done Sent Done false
33309 10238 Acked Read 3 Done Sent Done false
33310 10239 Acked Write 4 Done Sent Done false
33311 10240 Acked Write 4 Done Sent Done false
33312 10241 Acked Flush 0 Done Sent Done false
33313 10242 Acked Read 4 Done Sent Done false
33314 10243 Acked Flush 0 Done Sent Done false
33315 10244 Acked Flush 0 Done Sent Done false
33316 10245 Acked Write 10 Done Sent Done false
33317 10246 Acked Write 4 Done Sent Done false
33318 10247 Acked Read 6 Done Sent Done false
33319 10248 Acked Write 2 Done Sent Done false
33320 10249 Acked Read 8 Done Sent Done false
33321 10250 Acked Read 2 Done Sent Done false
33322 10251 Acked Write 5 Done Sent Done false
33323 10252 Acked Write 4 Done Sent Done false
33324 10253 Acked Read 2 Done Sent Done false
33325 10254 Acked Flush 0 Done Sent Done false
33326 10255 Acked Write 3 Done Sent Done false
33327 10256 Acked Read 2 Done Sent Done false
33328 10257 Acked Read 7 Done Sent Done false
33329 10258 Acked Read 6 Done Sent Done false
33330 10259 Acked Read 3 Done Sent Done false
33331 10260 Acked Write 8 Done Sent Done false
33332 10261 Acked Flush 0 Done Sent Done false
33333 10262 Acked Read 8 Done Sent Done false
33334 10263 Acked Read 4 Done Sent Done false
33335 10264 Acked Read 3 Done Sent Done false
33336 10265 Acked Read 8 Done Sent Done false
33337 10266 Acked Read 7 Done Sent Done false
33338 10267 Acked Flush 0 Done Sent Done false
33339 10268 Acked Write 10 Done Sent Done false
33340 10269 Acked Read 4 Done Sent Done false
33341 10270 Acked Write 10 Done Sent Done false
33342 10271 Acked Read 4 Done Sent Done false
33343 10272 Acked Read 6 Done Sent Done false
33344 10273 Acked Flush 0 Done Sent Done false
33345 10274 Acked Read 10 Done Sent Done false
33346 10275 Acked Flush 0 Done Sent Done false
33347 10276 Acked Read 4 Done Sent Done false
33348 10277 Acked Read 8 Done Sent Done false
33349 10278 Acked Read 7 Done Sent Done false
33350 10279 Acked Write 4 Done Sent Done false
33351 10280 Acked Write 2 Done Sent Done false
33352 10281 Acked Read 9 Done Sent Done false
33353 10282 Acked Write 4 Done Sent Done false
33354 10283 Acked Flush 0 Done Sent Done false
33355 10284 Acked Read 9 Done Sent Done false
33356 10285 Acked Read 9 Done Sent Done false
33357 10286 Acked Write 6 Done Sent Done false
33358 10287 Acked Flush 0 Done Sent Done false
33359 10288 Acked Read 4 Done Sent Done false
33360 10289 Acked Read 10 Done Sent Done false
33361 10290 Acked Write 6 Done Sent Done false
33362 10291 Acked Write 1 Done Sent Done false
33363 10292 Acked Read 6 Done Sent Done false
33364 10293 Acked Write 4 Done Sent Done false
33365 10294 Acked Read 4 Done Sent Done false
33366 10295 Acked Read 6 Done Sent Done false
33367 10296 Acked Read 3 Done Sent Done false
33368 10297 Acked Write 8 Done Sent Done false
33369 10298 Acked Read 5 Done Sent Done false
33370 10299 Acked Read 2 Done Sent Done false
33371 10300 Acked Read 1 Done Sent Done false
33372 10301 Acked Read 10 Done Sent Done false
33373 10302 Acked Write 6 Done Sent Done false
33374 10303 Acked Read 3 Done Sent Done false
33375 10304 Acked Read 9 Done Sent Done false
33376 10305 Acked Flush 0 Done Sent Done false
33377 10306 Acked Read 3 Done Sent Done false
33378 10307 Acked Read 4 Done Sent Done false
33379 10308 Acked Write 6 Done Sent Done false
33380 10309 Acked Read 9 Done Sent Done false
33381 10310 Acked Read 7 Done Sent Done false
33382 10311 Acked Write 1 Done Sent Done false
33383 10312 Acked Read 3 Done Sent Done false
33384 10313 Acked Read 8 Done Sent Done false
33385 10314 Acked Write 5 Done Sent Done false
33386 10315 Acked Read 2 Done Sent Done false
33387 10316 Acked Read 10 Done Sent Done false
33388 10317 Acked Read 1 Done Sent Done false
33389 10318 Acked Write 4 Done Sent Done false
33390 10319 Acked Read 2 Done Sent Done false
33391 10320 Acked Read 4 Done Sent Done false
33392 10321 Acked Read 7 Done Sent Done false
33393 10322 Acked Write 6 Done Sent Done false
33394 10323 Acked Read 5 Done Sent Done false
33395 10324 Acked Read 3 Done Sent Done false
33396 10325 Acked Read 10 Done Sent Done false
33397 10326 Acked Read 7 Done Sent Done false
33398 10327 Acked Write 6 Done Sent Done false
33399 10328 Acked Read 2 Done Sent Done false
33400 10329 Acked Read 9 Done Sent Done false
33401 10330 Acked Write 5 Done Sent Done false
33402 10331 Acked Read 10 Done Sent Done false
33403 10332 Acked Write 10 Done Sent Done false
33404 10333 Acked Read 8 Done Sent Done false
33405 10334 Acked Read 9 Done Sent Done false
33406 10335 Acked Write 9 Done Sent Done false
33407 10336 Acked Flush 0 Done Sent Done false
33408 10337 Acked Read 8 Done Sent Done false
33409 10338 Acked Read 8 Done Sent Done false
33410 10339 Acked Read 6 Done Sent Done false
33411 10340 Acked Write 2 Done Sent Done false
33412 10341 Acked Read 1 Done Sent Done false
33413 10342 Acked Flush 0 Done Sent Done false
33414 10343 Acked Write 4 Done Sent Done false
33415 10344 Acked Read 7 Done Sent Done false
33416 10345 Acked Read 1 Done Sent Done false
33417 10346 Acked Write 3 Done Sent Done false
33418 10347 Acked Read 6 Done Sent Done false
33419 10348 Acked Read 8 Done Sent Done false
33420 10349 Acked Write 8 Done Sent Done false
33421 10350 Acked Write 3 Done Sent Done false
33422 10351 Acked Read 8 Done Sent Done false
33423 10352 Acked Flush 0 Done Sent Done false
33424 10353 Acked Write 3 Done Sent Done false
33425 10354 Acked Read 4 Done Sent Done false
33426 10355 Acked Read 1 Done Sent Done false
33427 10356 Acked Read 8 Done Sent Done false
33428 10357 Acked Read 1 Done Sent Done false
33429 10358 Acked Read 9 Done Sent Done false
33430 10359 Acked Read 9 Done Sent Done false
33431 10360 Acked Flush 0 Done Sent Done false
33432 STATES DS:0 DS:1 DS:2 TOTAL
33433 Sent 0 302 0 302
33434 Done 302 0 302 604
33435 Skipped 0 0 0 0
33436 Error 0 0 0 0
33437 Last Flush: Some(JobId(10360)) Some(JobId(10058)) Some(JobId(10360))
33438 Downstairs last five completed: 10058 10057 10056 10055 10054
33439 Upstairs last five completed: 10360 10359 10358 10357 10356
33440 CLIENT: Up:0 ds:302 act:2
334412025-01-23T17:44:47.215ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
334422025-01-23T17:44:47.216ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44681) = io task client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
334432025-01-23T17:44:47.217ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
334442025-01-23T17:44:47.218ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
334452025-01-23T17:44:47.218ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
334462025-01-23T17:44:47.218ZINFOcrucible: send last flush ID to this DS: Some(JobId(10058)) = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
334472025-01-23T17:44:47.218ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
334482025-01-23T17:44:47.218ZINFOcrucible: [1] client re-new 302 jobs since flush Some(JobId(10058)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
334492025-01-23T17:44:47.218ZINFOcrucible: [1] Replayed 302 jobs since flush: Some(JobId(10058)) = downstairs session_id = 14e5020b-a7a5-48d4-836f-67e5604bd35c
33450 ----------------------------------------------------------------
33451 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33452 Downstairs last five completed: 10360 10359 10358 10357 10356
33453 Upstairs last five completed: 10360 10359 10358 10357 10356
33454 CLIENT: Up:0 ds:0 act:3
33455 Replay: All jobs finished
33456 Test replay has completed
33457 Wrote out file "/tmp/test_replay_verify.log"
33458 CLIENT: Tests done. All submitted work has been ACK'd
33459 ----------------------------------------------------------------
33460 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33461 Downstairs last five completed: 10360 10359 10358 10357 10356
33462 Upstairs last five completed: 10360 10359 10358 10357 10356
33463 CLIENT: Up:0 ds:0 act:3
33464 CLIENT: All jobs finished, all DS active.
334652025-01-23T17:44:47.484ZWARNcrucible: Guest handle has been dropped
33466 Replays:30 time: 7:00
33467 Do final verify
33468 Jan 23 17:44:47.557 INFO accepted connection, remote_addr: 127.0.0.1:34531, local_addr: 127.0.0.1:9998
33469 Jan 23 17:44:47.558 INFO request completed, latency_us: 197, response_code: 200, uri: /regioninfo, method: GET, req_id: fc2812f2-9c19-4de1-917f-2e4286cbe522, remote_addr: 127.0.0.1:34531, local_addr: 127.0.0.1:9998
334702025-01-23T17:44:47.559ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
33471 Jan 23 17:44:47.559 INFO request completed, latency_us: 110, response_code: 200, uri: /regioncount, method: GET, req_id: 7dcf4b65-1d9e-4c78-af74-72a58cc44bdc, remote_addr: 127.0.0.1:34531, local_addr: 127.0.0.1:9998
334722025-01-23T17:44:47.559ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
33473 Jan 23 17:44:47.560 INFO request completed, latency_us: 127, response_code: 200, uri: /port/cid/0, method: GET, req_id: ee2cf899-1cf1-4734-96d0-ded78ced709e, remote_addr: 127.0.0.1:34531, local_addr: 127.0.0.1:9998
33474 Jan 23 17:44:47.560 INFO request completed, latency_us: 92, response_code: 200, uri: /port/cid/1, method: GET, req_id: 97c94036-a216-468c-819f-59911e6d6304, remote_addr: 127.0.0.1:34531, local_addr: 127.0.0.1:9998
33475 Jan 23 17:44:47.561 INFO request completed, latency_us: 101, response_code: 200, uri: /port/cid/2, method: GET, req_id: 3133d147-4bf3-4c16-8ef8-eed2e23a092f, remote_addr: 127.0.0.1:34531, local_addr: 127.0.0.1:9998
334762025-01-23T17:44:47.561ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
334772025-01-23T17:44:47.561ZINFOcrucible: Upstairs starts
334782025-01-23T17:44:47.561ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "2d334f767f4e70887c1c804fc50babae5e22bede", git_commit_timestamp: "2025-01-23T17:21:44.000000000Z", git_branch: "renovate/tokio-util-0.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
334792025-01-23T17:44:47.561ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
334802025-01-23T17:44:47.561ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
334812025-01-23T17:44:47.562ZINFOcrucible: Crucible 5f650144-10d6-44ae-bb88-93af60736fe4 has session id: 446a9b2a-b92b-4914-9cf8-f6e98750ef83 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334822025-01-23T17:44:47.562ZINFOcrucible: Upstairs opts: Upstairs UUID: 5f650144-10d6-44ae-bb88-93af60736fe4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334832025-01-23T17:44:47.562ZINFOcrucible: Crucible stats registered with UUID: 5f650144-10d6-44ae-bb88-93af60736fe4 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334842025-01-23T17:44:47.562ZWARNcrucible: could not find Downstairs address for Nexus session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334852025-01-23T17:44:47.563ZINFOcrucible: Downstairs in volume = 3 task = crutest
334862025-01-23T17:44:47.563ZINFOcrucible: The guest has requested activation with gen:3
334872025-01-23T17:44:47.563ZINFOcrucible: 5f650144-10d6-44ae-bb88-93af60736fe4 active request set session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334882025-01-23T17:44:47.563ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334892025-01-23T17:44:47.563ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334902025-01-23T17:44:47.563ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334912025-01-23T17:44:47.563ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334922025-01-23T17:44:47.563ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334932025-01-23T17:44:47.563ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334942025-01-23T17:44:47.563ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334952025-01-23T17:44:47.563ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334962025-01-23T17:44:47.563ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334972025-01-23T17:44:47.563ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59021) = io task client = 2 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334982025-01-23T17:44:47.563ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:57169) = io task client = 0 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
334992025-01-23T17:44:47.563ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51882) = io task client = 1 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335002025-01-23T17:44:47.564ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335012025-01-23T17:44:47.564ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335022025-01-23T17:44:47.564ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335032025-01-23T17:44:47.564ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335042025-01-23T17:44:47.565ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335052025-01-23T17:44:47.565ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335062025-01-23T17:44:47.565ZINFOcrucible: Waiting for more clients to be ready session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335072025-01-23T17:44:47.565ZINFOcrucible: Waiting for more clients to be ready session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335082025-01-23T17:44:47.565ZINFOcrucible: [0]R flush_numbers[0..12]: [915, 930, 905, 897, 927, 933, 927, 900, 905, 928, 915, 930] = downstairs session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335092025-01-23T17:44:47.565ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335102025-01-23T17:44:47.565ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335112025-01-23T17:44:47.565ZINFOcrucible: [1]R flush_numbers[0..12]: [915, 930, 905, 897, 927, 933, 927, 900, 905, 928, 915, 930] = downstairs session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335122025-01-23T17:44:47.565ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335132025-01-23T17:44:47.566ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335142025-01-23T17:44:47.566ZINFOcrucible: [2]R flush_numbers[0..12]: [915, 930, 905, 897, 927, 933, 927, 900, 905, 928, 915, 930] = downstairs session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335152025-01-23T17:44:47.566ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335162025-01-23T17:44:47.566ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
33517 {"msg":"Max found gen is 3","v":0,"name":"crucible","level":30,"time":"2025-01-23T17:44:47.566074766Z","hostname":"w-01JJA3RYZ2J0869YVMXJYHKQVA","pid":1241Wait for a query_work_queue command to finish before sending IO
33518 ,"":"downstairs","":"downstairs","session_id":"446a9b2a-b92b-4914-9cf8-f6e98750ef83"}
335192025-01-23T17:44:47.566ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335202025-01-23T17:44:47.566ZINFOcrucible: Next flush: 934 = downstairs session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335212025-01-23T17:44:47.566ZINFOcrucible: All extents match = downstairs session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335222025-01-23T17:44:47.566ZINFOcrucible: No downstairs reconciliation required session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335232025-01-23T17:44:47.566ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335242025-01-23T17:44:47.566ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335252025-01-23T17:44:47.566ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335262025-01-23T17:44:47.566ZINFOcrucible: All required reconciliation work is completed session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335272025-01-23T17:44:47.566ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335282025-01-23T17:44:47.566ZINFOcrucible: 5f650144-10d6-44ae-bb88-93af60736fe4 is now active with session: 446a9b2a-b92b-4914-9cf8-f6e98750ef83 session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
33529 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
335302025-01-23T17:44:47.566ZINFOcrucible: Set Active after no reconciliation session_id = 446a9b2a-b92b-4914-9cf8-f6e98750ef83
335312025-01-23T17:44:47.566ZINFOcrucible: The guest has finished waiting for activation with:3
33532 Loading write count information from file "/tmp/test_replay_verify.log"
33533 Read and Verify all blocks (0..5000 range:false)
33534 Wrote out file "/tmp/test_replay_verify.log"
33535 Verify test completed
33536 Wrote out file "/tmp/test_replay_verify.log"
33537 CLIENT: Tests done. All submitted work has been ACK'd
33538 ----------------------------------------------------------------
33539 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
33540 States: Active Active Active
33541 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33542 1000 Acked Read 256 Done Done Done false
33543 1001 Acked Read 256 Done Done Done false
33544 1002 Acked Read 256 Done Done Done false
33545 1003 Acked Read 256 Done Done Done false
33546 1004 Acked Read 256 Done Done Done false
33547 1005 Acked Read 256 Done Done Done false
33548 1006 Acked Read 256 Done Done Done false
33549 1007 Acked Read 256 Done Done Done false
33550 1008 Acked Read 256 Done Done Done false
33551 1009 Acked Read 256 Done Done Done false
33552 1010 Acked Read 256 Done Done Done false
33553 1011 Acked Read 256 Done Done Done false
33554 1012 Acked Read 256 Done Done Done false
33555 1013 Acked Read 256 Done Done Done false
33556 1014 Acked Read 256 Done Done Done false
33557 1015 Acked Read 256 Done Done Done false
33558 1016 Acked Read 256 Done Done Done false
33559 1017 Acked Read 256 Done Done Done false
33560 1018 Acked Read 136 Done Done Done false
33561 1019 Acked Read 256 Done Done Done false
33562 STATES DS:0 DS:1 DS:2 TOTAL
33563 Sent 0 0 0 0
33564 Done 20 20 20 60
33565 Skipped 0 0 0 0
33566 Error 0 0 0 0
33567 Last Flush: None None None
33568 Downstairs last five completed:
33569 Upstairs last five completed: 1019 1018 1017 1016 1015
33570 CLIENT: Up:0 ds:20 act:3
33571 ----------------------------------------------------------------
33572 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
33573 Downstairs last five completed: 1020 1019 1018 1017 1016
33574 Upstairs last five completed: 1020 1019 1018 1017 1016
33575 CLIENT: Up:0 ds:0 act:3
33576 CLIENT: All crucible jobs finished, exiting program
335772025-01-23T17:44:51.964ZWARNcrucible: Guest handle has been dropped
33578 Jan 23 17:44:52.031 INFO accepted connection, remote_addr: 127.0.0.1:48665, local_addr: 127.0.0.1:9998
33579 Main task has work to do, go find it
33580 got dsc Shutdown
33581 Shutdown
33582 Jan 23 17:44:52.032 INFO request completed, latency_us: 185, response_code: 204, Shut it down
33583 uri[8810] Disable keep_running
33584 :[8810] Got stop action so:false kr:false
33585 /shutdown, method: [8820] Disable keep_running
33586 POST, req_id: c26959e4-2ff0-4595-97b4-2844535cfea8, remote_addr: 127.0.0.1:48665, local_addr: 127.0.0.1:9998
33587 [8820] Got stop action so:false kr:false
33588 [8830] Disable keep_running
33589 [8830] Got stop action so:false kr:false
33590 [8820] Exited with: ExitStatus(unix_wait_status(9))
33591 [8820][1] reports Exit
33592 [8830] Exited with: ExitStatus(unix_wait_status(9))
33593 [8830][2] reports Exit
33594 [8810] Exited with: ExitStatus(unix_wait_status(9))
33595 [8810][0] reports Exit
33596 January 23, 2025 at 05:44:57 PM UTC Test ends with 0